2024-04-02T07:19:26.618 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-02T07:19:26.618 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} --name vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi -- /home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459/orig.config.yaml 2024-04-02T07:19:26.666 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T07:19:26.752 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459 branch: wip-vshankar-testing-20240326.105515-reef description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} email: vshankar@redhat.com first_in_suite: false job_id: '7635459' kernel: client: branch: testing flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-vshankar-testing-20240326.105515-reef ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: d0e392d9aee635f2b65768c460c7a151ead7a5e2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d0e392d9aee635f2b65768c460c7a151ead7a5e2 kclient: syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240326.105515-reef sha1: d46289a69430c98a5ee030c7aba50d0c7af4ad4b owner: scheduled_vshankar@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 7327 sha1: d0e392d9aee635f2b65768c460c7a151ead7a5e2 sleep_before_teardown: 0 subset: 990/1024 suite: fs suite_branch: wip-vshankar-testing-20240326.105515-reef suite_path: /home/teuthworker/src/github.com_vshankar_ceph_d46289a69430c98a5ee030c7aba50d0c7af4ad4b/qa suite_relpath: qa suite_repo: https://github.com/vshankar/ceph.git suite_sha1: d46289a69430c98a5ee030c7aba50d0c7af4ad4b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGtwTkQ7YIBikeL384i+WuovQQ9VOrg7Jr01f4YTXbCtcED02BXdNA6Vxc692Plr1mrtlShqlFiikUvYOzF/3U= smithi113.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNNQGxHVHDjN0qBn7n38dK/3CdgI7Sj/i3cmkwTSGB4mRRhXktG01i0oglnwBx99f4xOW6VGFSlX6tXmYuVKlWE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-02_06:51:51 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-02T07:19:26.752 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_vshankar_ceph_d46289a69430c98a5ee030c7aba50d0c7af4ad4b/qa; will attempt to use it 2024-04-02T07:19:26.753 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_vshankar_ceph_d46289a69430c98a5ee030c7aba50d0c7af4ad4b/qa/tasks 2024-04-02T07:19:26.753 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-02T07:19:26.755 INFO:teuthology.task.internal:Checking packages... 2024-04-02T07:19:26.778 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd0e392d9aee635f2b65768c460c7a151ead7a5e2' 2024-04-02T07:19:26.778 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-02T07:19:26.778 INFO:teuthology.packaging:ref: None 2024-04-02T07:19:26.778 INFO:teuthology.packaging:tag: None 2024-04-02T07:19:26.778 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240326.105515-reef 2024-04-02T07:19:26.778 INFO:teuthology.packaging:sha1: d0e392d9aee635f2b65768c460c7a151ead7a5e2 2024-04-02T07:19:26.778 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing-20240326.105515-reef 2024-04-02T07:19:26.996 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1015.gd0e392d9 2024-04-02T07:19:26.997 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-02T07:19:27.004 INFO:teuthology.task.internal:no buildpackages task found 2024-04-02T07:19:27.004 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-02T07:19:27.024 INFO:teuthology.task.internal:Saving configuration 2024-04-02T07:19:27.042 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-02T07:19:27.048 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T07:19:27.075 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 07:13:45.433852', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGtwTkQ7YIBikeL384i+WuovQQ9VOrg7Jr01f4YTXbCtcED02BXdNA6Vxc692Plr1mrtlShqlFiikUvYOzF/3U='} 2024-04-02T07:19:27.104 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 07:13:45.431954', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNNQGxHVHDjN0qBn7n38dK/3CdgI7Sj/i3cmkwTSGB4mRRhXktG01i0oglnwBx99f4xOW6VGFSlX6tXmYuVKlWE='} 2024-04-02T07:19:27.104 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-02T07:19:27.110 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-02T07:19:27.111 INFO:teuthology.task.internal:roles: ubuntu@smithi113.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-02T07:19:27.111 INFO:teuthology.run_tasks:Running task console_log... 2024-04-02T07:19:27.183 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4e99f9f0d0>, signals=[15]) 2024-04-02T07:19:27.184 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-02T07:19:27.190 INFO:teuthology.task.internal:Opening connections... 2024-04-02T07:19:27.190 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:19:27.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T07:19:27.267 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi113.front.sepia.ceph.com 2024-04-02T07:19:27.268 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T07:19:27.339 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-02T07:19:27.346 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2024-04-02T07:19:27.368 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2024-04-02T07:19:27.369 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2024-04-02T07:19:27.429 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="CentOS Stream" 2024-04-02T07:19:27.430 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="8" 2024-04-02T07:19:27.430 INFO:teuthology.orchestra.run.smithi089.stdout:ID="centos" 2024-04-02T07:19:27.430 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="rhel fedora" 2024-04-02T07:19:27.430 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="8" 2024-04-02T07:19:27.430 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el8" 2024-04-02T07:19:27.430 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-02T07:19:27.430 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2024-04-02T07:19:27.430 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-02T07:19:27.430 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://centos.org/" 2024-04-02T07:19:27.430 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-02T07:19:27.430 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-02T07:19:27.430 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-02T07:19:27.431 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2024-04-02T07:19:27.456 DEBUG:teuthology.orchestra.run.smithi113:> uname -m 2024-04-02T07:19:27.479 INFO:teuthology.orchestra.run.smithi113.stdout:x86_64 2024-04-02T07:19:27.479 DEBUG:teuthology.orchestra.run.smithi113:> cat /etc/os-release 2024-04-02T07:19:27.539 INFO:teuthology.orchestra.run.smithi113.stdout:NAME="CentOS Stream" 2024-04-02T07:19:27.540 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION="8" 2024-04-02T07:19:27.540 INFO:teuthology.orchestra.run.smithi113.stdout:ID="centos" 2024-04-02T07:19:27.540 INFO:teuthology.orchestra.run.smithi113.stdout:ID_LIKE="rhel fedora" 2024-04-02T07:19:27.540 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION_ID="8" 2024-04-02T07:19:27.540 INFO:teuthology.orchestra.run.smithi113.stdout:PLATFORM_ID="platform:el8" 2024-04-02T07:19:27.540 INFO:teuthology.orchestra.run.smithi113.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-02T07:19:27.540 INFO:teuthology.orchestra.run.smithi113.stdout:ANSI_COLOR="0;31" 2024-04-02T07:19:27.540 INFO:teuthology.orchestra.run.smithi113.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-02T07:19:27.540 INFO:teuthology.orchestra.run.smithi113.stdout:HOME_URL="https://centos.org/" 2024-04-02T07:19:27.540 INFO:teuthology.orchestra.run.smithi113.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-02T07:19:27.540 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-02T07:19:27.540 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-02T07:19:27.541 INFO:teuthology.lock.ops:Updating smithi113.front.sepia.ceph.com on lock server 2024-04-02T07:19:27.564 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-02T07:19:27.574 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-02T07:19:27.579 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-02T07:19:27.579 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2024-04-02T07:19:27.582 DEBUG:teuthology.orchestra.run.smithi113:> test '!' -e /home/ubuntu/cephtest 2024-04-02T07:19:27.601 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-02T07:19:27.608 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-02T07:19:27.608 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2024-04-02T07:19:27.642 DEBUG:teuthology.orchestra.run.smithi113:> test -z $(ls -A /var/lib/ceph) 2024-04-02T07:19:27.670 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-02T07:19:27.726 INFO:teuthology.run_tasks:Running task kernel... 2024-04-02T07:19:27.748 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-04-02T07:19:27.748 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-04-02T07:19:27.748 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-04-02T07:19:27.749 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-02T07:19:27.749 DEBUG:teuthology.orchestra.run.smithi113:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-02T07:19:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T07:19:27.769 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2024-04-02T07:19:27.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T07:19:27.772 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2024-04-02T07:19:28.022 DEBUG:teuthology.task.kernel:sha1 for client.0 is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-04-02T07:19:28.022 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-04-02T07:19:28.022 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-04-02T07:19:28.042 DEBUG:teuthology.task.kernel:sha1 for client.1 is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-04-02T07:19:28.042 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-04-02T07:19:28.042 DEBUG:teuthology.orchestra.run.smithi113:> uname -r 2024-04-02T07:19:28.043 INFO:teuthology.orchestra.run.smithi089.stdout:4.18.0-489.el8.x86_64 2024-04-02T07:19:28.044 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-489.el8.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-04-02T07:19:28.044 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-02T07:19:28.044 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum upgrade -y linux-firmware 2024-04-02T07:19:28.065 INFO:teuthology.orchestra.run.smithi113.stdout:4.18.0-489.el8.x86_64 2024-04-02T07:19:28.065 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-489.el8.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-04-02T07:19:28.065 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-02T07:19:28.066 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum upgrade -y linux-firmware 2024-04-02T07:19:32.016 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - AppStream 21 MB/s | 28 MB 00:01 2024-04-02T07:19:33.299 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - AppStream 10 MB/s | 28 MB 00:02 2024-04-02T07:19:37.990 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - BaseOS 10 MB/s | 10 MB 00:01 2024-04-02T07:19:39.291 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - BaseOS 9.7 MB/s | 10 MB 00:01 2024-04-02T07:19:39.982 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - Extras 47 kB/s | 18 kB 00:00 2024-04-02T07:19:40.299 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - Extras common packages 36 kB/s | 7.5 kB 00:00 2024-04-02T07:19:40.567 INFO:teuthology.orchestra.run.smithi113.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-04-02T07:19:41.183 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - Extras 66 kB/s | 18 kB 00:00 2024-04-02T07:19:41.592 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - Extras common packages 25 kB/s | 7.5 kB 00:00 2024-04-02T07:19:41.795 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-04-02T07:19:41.819 INFO:teuthology.orchestra.run.smithi089.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-04-02T07:19:43.097 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-04-02T07:19:45.559 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 587 kB/s | 24 kB 00:00 2024-04-02T07:19:46.870 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 466 kB/s | 24 kB 00:00 2024-04-02T07:19:48.679 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-02T07:19:48.679 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-02T07:19:48.679 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repo Size 2024-04-02T07:19:48.679 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-02T07:19:48.679 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrading: 2024-04-02T07:19:48.679 INFO:teuthology.orchestra.run.smithi113.stdout: linux-firmware noarch 20240111-121.gitb3132c18.el8 baseos 355 M 2024-04-02T07:19:48.679 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:19:48.680 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-02T07:19:48.680 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-02T07:19:48.680 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrade 1 Package 2024-04-02T07:19:48.680 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:19:48.680 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 355 M 2024-04-02T07:19:48.680 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-02T07:19:50.007 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-02T07:19:50.008 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-02T07:19:50.008 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-04-02T07:19:50.008 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-02T07:19:50.008 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2024-04-02T07:19:50.008 INFO:teuthology.orchestra.run.smithi089.stdout: linux-firmware noarch 20240111-121.gitb3132c18.el8 baseos 355 M 2024-04-02T07:19:50.008 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:19:50.008 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-02T07:19:50.008 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-02T07:19:50.008 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 1 Package 2024-04-02T07:19:50.009 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:19:50.009 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 355 M 2024-04-02T07:19:50.009 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-02T07:20:01.267 INFO:teuthology.orchestra.run.smithi113.stdout:linux-firmware-20240111-121.gitb3132c18.el8.noa 28 MB/s | 355 MB 00:12 2024-04-02T07:20:01.267 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-02T07:20:01.267 INFO:teuthology.orchestra.run.smithi113.stdout:Total 28 MB/s | 355 MB 00:12 2024-04-02T07:20:02.553 INFO:teuthology.orchestra.run.smithi089.stdout:linux-firmware-20240111-121.gitb3132c18.el8.noa 28 MB/s | 355 MB 00:12 2024-04-02T07:20:02.554 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-02T07:20:02.554 INFO:teuthology.orchestra.run.smithi089.stdout:Total 28 MB/s | 355 MB 00:12 2024-04-02T07:20:03.366 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-02T07:20:03.435 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-02T07:20:03.436 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-02T07:20:04.651 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-02T07:20:04.713 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-02T07:20:04.713 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-02T07:20:05.602 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-02T07:20:05.636 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-02T07:20:06.882 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-02T07:20:06.908 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-02T07:20:28.662 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-02T07:20:29.027 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-04-02T07:20:29.290 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-04-02T07:20:29.534 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-04-02T07:20:29.534 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-04-02T07:20:29.788 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-02T07:20:29.964 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-04-02T07:20:29.964 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:20:29.964 INFO:teuthology.orchestra.run.smithi113.stdout:Upgraded: 2024-04-02T07:20:29.964 INFO:teuthology.orchestra.run.smithi113.stdout: linux-firmware-20240111-121.gitb3132c18.el8.noarch 2024-04-02T07:20:29.964 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:20:29.964 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-02T07:20:30.160 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-04-02T07:20:30.164 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.1... 2024-04-02T07:20:30.165 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-04-02T07:20:30.398 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ 2024-04-02T07:20:30.399 DEBUG:teuthology.orchestra.run.smithi113:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ --input-file=- 2024-04-02T07:20:30.401 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi113.front.sepia.ceph.com to complete... 2024-04-02T07:20:30.406 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-04-02T07:20:30.629 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-04-02T07:20:30.629 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-04-02T07:20:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-04-02T07:20:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:20:31.092 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2024-04-02T07:20:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: linux-firmware-20240111-121.gitb3132c18.el8.noarch 2024-04-02T07:20:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:20:31.092 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-02T07:20:31.276 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-04-02T07:20:31.276 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-04-02T07:20:31.505 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ 2024-04-02T07:20:31.506 DEBUG:teuthology.orchestra.run.smithi089:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ --input-file=- 2024-04-02T07:20:31.508 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi089.front.sepia.ceph.com to complete... 2024-04-02T07:20:50.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02 07:20:50 URL:https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [401601944/401601944] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-02T07:20:50.269 INFO:teuthology.orchestra.run.smithi089.stderr:FINISHED --2024-04-02 07:20:50-- 2024-04-02T07:20:50.269 INFO:teuthology.orchestra.run.smithi089.stderr:Total wall clock time: 19s 2024-04-02T07:20:50.269 INFO:teuthology.orchestra.run.smithi089.stderr:Downloaded: 1 files, 383M in 18s (20.7 MB/s) 2024-04-02T07:20:50.273 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-04-02T07:20:50.273 DEBUG:teuthology.orchestra.run.smithi089:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-02T07:20:52.025 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying... ######################################## 2024-04-02T07:20:52.070 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing... ######################################## 2024-04-02T07:20:52.178 INFO:teuthology.orchestra.run.smithi089.stdout:Updating / installing... 2024-04-02T07:21:32.490 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-04-02T07:21:32.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02 07:21:32 URL:https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [401601944/401601944] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-02T07:21:32.561 INFO:teuthology.orchestra.run.smithi113.stderr:FINISHED --2024-04-02 07:21:32-- 2024-04-02T07:21:32.561 INFO:teuthology.orchestra.run.smithi113.stderr:Total wall clock time: 1m 2s 2024-04-02T07:21:32.561 INFO:teuthology.orchestra.run.smithi113.stderr:Downloaded: 1 files, 383M in 1m 2s (6.20 MB/s) 2024-04-02T07:21:32.564 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.1... 2024-04-02T07:21:32.564 DEBUG:teuthology.orchestra.run.smithi113:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-02T07:21:33.811 INFO:teuthology.orchestra.run.smithi089.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-02T07:21:34.298 INFO:teuthology.orchestra.run.smithi113.stdout:Verifying... ######################################## 2024-04-02T07:21:34.341 INFO:teuthology.orchestra.run.smithi113.stdout:Preparing... ######################################## 2024-04-02T07:21:34.449 INFO:teuthology.orchestra.run.smithi113.stdout:Updating / installing... 2024-04-02T07:21:59.421 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi089.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-02T07:21:59.422 DEBUG:teuthology.orchestra.run.smithi089:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-02T07:21:59.453 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-04-02T07:21:59.453 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-04-02T07:21:59.453 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-04-02T07:21:59.453 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-04-02T07:21:59.453 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-04-02T07:21:59.453 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-04-02T07:21:59.453 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-04-02T07:21:59.453 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-02T07:21:59.454 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-02T07:21:59.454 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-02T07:21:59.454 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-02T07:21:59.454 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-02T07:21:59.454 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-02T07:21:59.454 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-02T07:21:59.454 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-02T07:21:59.454 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-02T07:21:59.454 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-02T07:21:59.454 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-02T07:21:59.454 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-02T07:21:59.454 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-02T07:21:59.455 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-02T07:21:59.455 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-02T07:21:59.455 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-02T07:21:59.455 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-02T07:21:59.455 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-02T07:21:59.455 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-02T07:21:59.455 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-02T07:21:59.455 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-04-02T07:21:59.455 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-04-02T07:21:59.455 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-04-02T07:21:59.455 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-02T07:21:59.455 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-04-02T07:21:59.455 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-02T07:21:59.455 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-02T07:21:59.456 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-04-02T07:21:59.456 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-04-02T07:21:59.456 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-04-02T07:21:59.456 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-04-02T07:21:59.456 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-04-02T07:21:59.456 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-04-02T07:21:59.456 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-04-02T07:21:59.456 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-04-02T07:21:59.456 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-02T07:21:59.456 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-04-02T07:21:59.456 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-04-02T07:21:59.456 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-04-02T07:21:59.456 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-04-02T07:21:59.457 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-04-02T07:21:59.457 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-04-02T07:21:59.457 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-04-02T07:21:59.457 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-04-02T07:21:59.457 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-04-02T07:21:59.457 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-04-02T07:21:59.457 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-04-02T07:21:59.457 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-04-02T07:21:59.457 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-04-02T07:21:59.457 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-04-02T07:21:59.457 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-04-02T07:21:59.457 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-04-02T07:21:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-04-02T07:21:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-04-02T07:21:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-04-02T07:21:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-04-02T07:21:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-04-02T07:21:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-04-02T07:21:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-04-02T07:21:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-04-02T07:21:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-04-02T07:21:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-04-02T07:21:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-04-02T07:21:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-04-02T07:21:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-04-02T07:21:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-04-02T07:21:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-04-02T07:21:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-04-02T07:21:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-04-02T07:21:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-04-02T07:21:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-04-02T07:21:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-04-02T07:21:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-04-02T07:21:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-04-02T07:21:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-04-02T07:21:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-04-02T07:21:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-04-02T07:21:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-04-02T07:21:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-04-02T07:21:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-04-02T07:21:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-02T07:21:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-04-02T07:21:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-04-02T07:21:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-04-02T07:21:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-04-02T07:21:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-04-02T07:21:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-04-02T07:21:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-04-02T07:21:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-04-02T07:21:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-04-02T07:21:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-04-02T07:21:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-04-02T07:21:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-02T07:21:59.461 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-04-02T07:21:59.462 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-04-02T07:21:59.462 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-04-02T07:21:59.462 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-04-02T07:21:59.462 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-04-02T07:21:59.462 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-04-02T07:21:59.462 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-04-02T07:21:59.462 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-04-02T07:21:59.462 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-04-02T07:21:59.462 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-04-02T07:21:59.462 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-04-02T07:21:59.462 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-04-02T07:21:59.462 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-04-02T07:21:59.462 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-02T07:21:59.463 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-02T07:21:59.463 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-04-02T07:21:59.463 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-04-02T07:21:59.463 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-04-02T07:21:59.463 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-04-02T07:21:59.463 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-04-02T07:21:59.463 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-04-02T07:21:59.463 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-04-02T07:21:59.463 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-04-02T07:21:59.482 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-04-02T07:21:59.482 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-04-02T07:21:59.482 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-04-02T07:21:59.482 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-04-02T07:21:59.482 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-04-02T07:21:59.482 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-04-02T07:21:59.482 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-04-02T07:21:59.482 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-04-02T07:21:59.482 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-04-02T07:21:59.483 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-04-02T07:21:59.483 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-04-02T07:21:59.483 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-04-02T07:21:59.483 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-04-02T07:21:59.483 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-04-02T07:21:59.483 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-04-02T07:21:59.483 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-04-02T07:21:59.483 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-04-02T07:21:59.483 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-04-02T07:21:59.483 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-04-02T07:21:59.483 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-02T07:21:59.483 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-02T07:21:59.483 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-04-02T07:21:59.483 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-04-02T07:21:59.484 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-04-02T07:21:59.484 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-04-02T07:21:59.484 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-04-02T07:21:59.484 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-04-02T07:21:59.484 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-04-02T07:21:59.484 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-04-02T07:21:59.484 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-04-02T07:21:59.484 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-04-02T07:21:59.484 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-04-02T07:21:59.484 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-02T07:21:59.484 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-04-02T07:21:59.484 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-04-02T07:21:59.484 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-04-02T07:21:59.485 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-04-02T07:21:59.485 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-04-02T07:21:59.485 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-04-02T07:21:59.485 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-04-02T07:21:59.485 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-02T07:21:59.485 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-04-02T07:21:59.485 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-04-02T07:21:59.485 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-04-02T07:21:59.485 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-04-02T07:21:59.485 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-04-02T07:21:59.485 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-04-02T07:21:59.485 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-02T07:21:59.485 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-02T07:21:59.485 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-02T07:21:59.486 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-04-02T07:21:59.486 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-02T07:21:59.486 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-04-02T07:21:59.486 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-02T07:21:59.486 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-02T07:21:59.486 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-02T07:21:59.486 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-02T07:21:59.486 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-02T07:21:59.486 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-02T07:21:59.486 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-04-02T07:21:59.486 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-04-02T07:21:59.486 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-04-02T07:21:59.486 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-04-02T07:21:59.486 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-04-02T07:21:59.487 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-04-02T07:21:59.487 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-04-02T07:21:59.487 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-02T07:21:59.487 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-02T07:21:59.487 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-02T07:21:59.487 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-02T07:21:59.487 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-02T07:21:59.487 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-04-02T07:21:59.487 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-04-02T07:21:59.487 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-04-02T07:21:59.487 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-02T07:21:59.487 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-02T07:21:59.487 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-02T07:21:59.487 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-02T07:21:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-02T07:21:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-02T07:21:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-04-02T07:21:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-04-02T07:21:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-02T07:21:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-04-02T07:21:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-04-02T07:21:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-04-02T07:21:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-04-02T07:21:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-02T07:21:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-02T07:21:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-02T07:21:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-04-02T07:21:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-04-02T07:21:59.489 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-04-02T07:21:59.490 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-04-02T07:21:59.490 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-04-02T07:21:59.490 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-04-02T07:21:59.490 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-04-02T07:21:59.490 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-04-02T07:21:59.490 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-04-02T07:21:59.490 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-04-02T07:21:59.490 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-04-02T07:21:59.490 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-04-02T07:21:59.490 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-04-02T07:21:59.490 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-04-02T07:21:59.490 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-04-02T07:21:59.490 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-02T07:21:59.491 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-04-02T07:21:59.491 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-04-02T07:21:59.491 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-02T07:21:59.491 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-04-02T07:21:59.491 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-04-02T07:21:59.491 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-04-02T07:21:59.491 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-04-02T07:21:59.491 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-04-02T07:21:59.491 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-04-02T07:21:59.491 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-04-02T07:21:59.491 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-04-02T07:21:59.491 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-04-02T07:21:59.491 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-04-02T07:21:59.491 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-04-02T07:21:59.492 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-02T07:21:59.492 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-02T07:21:59.492 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-04-02T07:21:59.492 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-04-02T07:21:59.492 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-04-02T07:21:59.492 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-04-02T07:21:59.492 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-04-02T07:21:59.492 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-04-02T07:21:59.492 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-04-02T07:21:59.492 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-04-02T07:21:59.492 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-02T07:21:59.492 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-02T07:21:59.492 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-04-02T07:21:59.493 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-04-02T07:21:59.493 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-04-02T07:21:59.493 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-04-02T07:21:59.493 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-04-02T07:21:59.493 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-04-02T07:21:59.493 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-04-02T07:21:59.493 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-04-02T07:21:59.493 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-04-02T07:21:59.493 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-02T07:21:59.493 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-04-02T07:21:59.493 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-02T07:21:59.493 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-02T07:21:59.493 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-02T07:21:59.493 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-02T07:21:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-02T07:21:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-02T07:21:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-02T07:21:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-02T07:21:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-02T07:21:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-04-02T07:21:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-04-02T07:21:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-04-02T07:21:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-02T07:21:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-04-02T07:21:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-04-02T07:21:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-04-02T07:21:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-04-02T07:21:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-04-02T07:21:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-02T07:21:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-04-02T07:21:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-04-02T07:21:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-04-02T07:21:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-04-02T07:21:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-04-02T07:21:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-04-02T07:21:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-04-02T07:21:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-04-02T07:21:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-04-02T07:21:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-04-02T07:21:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-04-02T07:21:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-04-02T07:21:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-04-02T07:21:59.496 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-04-02T07:21:59.496 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-04-02T07:21:59.496 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-04-02T07:21:59.496 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-04-02T07:21:59.496 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-04-02T07:21:59.496 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-04-02T07:21:59.496 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-02T07:21:59.496 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-04-02T07:21:59.496 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-04-02T07:21:59.496 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-04-02T07:21:59.496 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-04-02T07:21:59.496 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-04-02T07:21:59.496 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-02T07:21:59.497 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-04-02T07:21:59.498 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-04-02T07:21:59.498 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-02T07:21:59.498 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-04-02T07:21:59.498 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-04-02T07:21:59.498 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-04-02T07:21:59.498 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-04-02T07:21:59.498 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-04-02T07:21:59.498 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-04-02T07:21:59.498 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-04-02T07:21:59.498 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-04-02T07:21:59.498 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-04-02T07:21:59.498 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-04-02T07:21:59.498 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-04-02T07:21:59.498 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-04-02T07:21:59.499 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-04-02T07:21:59.499 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-04-02T07:21:59.499 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-02T07:21:59.499 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-04-02T07:21:59.499 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-04-02T07:21:59.499 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-04-02T07:21:59.499 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-04-02T07:21:59.499 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-04-02T07:21:59.499 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-04-02T07:21:59.499 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-04-02T07:21:59.499 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-04-02T07:21:59.500 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-04-02T07:21:59.500 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-04-02T07:21:59.500 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-04-02T07:21:59.500 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-04-02T07:21:59.500 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-04-02T07:21:59.500 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-04-02T07:21:59.500 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-04-02T07:21:59.500 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-04-02T07:21:59.500 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-04-02T07:21:59.500 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-04-02T07:21:59.500 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-04-02T07:21:59.500 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-04-02T07:21:59.500 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-04-02T07:21:59.501 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-04-02T07:21:59.501 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-04-02T07:21:59.501 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-04-02T07:21:59.501 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-04-02T07:21:59.501 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-04-02T07:21:59.501 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-04-02T07:21:59.501 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-04-02T07:21:59.501 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-04-02T07:21:59.501 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-04-02T07:21:59.501 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-04-02T07:21:59.501 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-04-02T07:21:59.501 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-04-02T07:21:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-04-02T07:21:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-04-02T07:21:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-04-02T07:21:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-04-02T07:21:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-04-02T07:21:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-04-02T07:21:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-04-02T07:21:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-04-02T07:21:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-04-02T07:21:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-04-02T07:21:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-04-02T07:21:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-04-02T07:21:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-02T07:21:59.503 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-04-02T07:21:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-04-02T07:21:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-04-02T07:21:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-02T07:21:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-04-02T07:21:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-04-02T07:21:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-04-02T07:21:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-04-02T07:21:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-04-02T07:21:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-04-02T07:21:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-04-02T07:21:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-04-02T07:21:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-04-02T07:21:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-04-02T07:21:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-04-02T07:21:59.505 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-04-02T07:21:59.505 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-04-02T07:21:59.505 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-04-02T07:21:59.505 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-04-02T07:21:59.505 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-04-02T07:21:59.505 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-04-02T07:21:59.505 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-04-02T07:21:59.505 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-04-02T07:21:59.505 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-04-02T07:21:59.505 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-04-02T07:21:59.505 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-04-02T07:21:59.505 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-04-02T07:21:59.505 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-04-02T07:21:59.505 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-04-02T07:21:59.506 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-04-02T07:21:59.506 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-04-02T07:21:59.506 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-02T07:21:59.506 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-02T07:21:59.506 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-04-02T07:21:59.506 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-02T07:21:59.506 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-02T07:21:59.506 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-02T07:21:59.506 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-02T07:21:59.506 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-02T07:21:59.506 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-02T07:21:59.506 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-02T07:21:59.506 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-02T07:21:59.506 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-02T07:21:59.510 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-02T07:21:59.510 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-02T07:21:59.510 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-02T07:21:59.510 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-02T07:21:59.510 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-02T07:21:59.510 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-02T07:21:59.510 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-02T07:21:59.511 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-02T07:21:59.511 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-02T07:21:59.511 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-02T07:21:59.511 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-02T07:21:59.511 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-02T07:21:59.511 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-02T07:21:59.511 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-02T07:21:59.511 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-04-02T07:21:59.511 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-04-02T07:21:59.511 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-02T07:21:59.511 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-04-02T07:21:59.511 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-04-02T07:21:59.511 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-04-02T07:21:59.512 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-04-02T07:21:59.512 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-02T07:21:59.512 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-04-02T07:21:59.512 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-02T07:21:59.512 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-02T07:21:59.512 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-02T07:21:59.512 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-02T07:21:59.512 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-02T07:21:59.512 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-04-02T07:21:59.512 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-04-02T07:21:59.512 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-02T07:21:59.512 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-04-02T07:21:59.512 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-02T07:21:59.513 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-04-02T07:21:59.513 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-02T07:21:59.513 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-04-02T07:21:59.513 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-02T07:21:59.513 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-04-02T07:21:59.513 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-04-02T07:21:59.513 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-02T07:21:59.513 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-04-02T07:21:59.513 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-02T07:21:59.513 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-04-02T07:21:59.513 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-02T07:21:59.513 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-04-02T07:21:59.513 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-02T07:21:59.514 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-04-02T07:21:59.514 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-02T07:21:59.514 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-04-02T07:21:59.514 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-04-02T07:21:59.514 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-02T07:21:59.514 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-02T07:21:59.514 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-04-02T07:21:59.514 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-04-02T07:21:59.514 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-04-02T07:21:59.514 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-02T07:21:59.514 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-04-02T07:21:59.514 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-04-02T07:21:59.514 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-04-02T07:21:59.514 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-04-02T07:21:59.515 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-02T07:21:59.515 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-04-02T07:21:59.515 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-02T07:21:59.515 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-02T07:21:59.515 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-02T07:21:59.515 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-04-02T07:21:59.515 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-02T07:21:59.515 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-04-02T07:21:59.515 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-04-02T07:21:59.515 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-04-02T07:21:59.515 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-02T07:21:59.515 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-04-02T07:21:59.515 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-04-02T07:21:59.515 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-04-02T07:21:59.516 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-04-02T07:21:59.516 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-04-02T07:21:59.516 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-04-02T07:21:59.516 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-04-02T07:21:59.516 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-04-02T07:21:59.516 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-02T07:21:59.516 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-02T07:21:59.516 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-02T07:21:59.516 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-02T07:21:59.516 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-02T07:21:59.516 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-02T07:21:59.516 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-02T07:21:59.516 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-02T07:21:59.517 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-02T07:21:59.517 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-04-02T07:21:59.517 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-04-02T07:21:59.517 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-04-02T07:21:59.517 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-04-02T07:21:59.517 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-04-02T07:21:59.517 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-04-02T07:21:59.517 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-04-02T07:21:59.517 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-04-02T07:21:59.517 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-04-02T07:21:59.517 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-04-02T07:21:59.517 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-04-02T07:21:59.517 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-04-02T07:21:59.518 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-04-02T07:21:59.518 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-04-02T07:21:59.518 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-04-02T07:21:59.518 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-04-02T07:21:59.518 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-04-02T07:21:59.518 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-04-02T07:21:59.518 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-04-02T07:21:59.518 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-02T07:21:59.518 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-04-02T07:21:59.518 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-04-02T07:21:59.518 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-04-02T07:21:59.518 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-04-02T07:21:59.518 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-04-02T07:21:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-04-02T07:21:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-04-02T07:21:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-04-02T07:21:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-04-02T07:21:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-04-02T07:21:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-04-02T07:21:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-04-02T07:21:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-04-02T07:21:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-04-02T07:21:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-04-02T07:21:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-04-02T07:21:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-04-02T07:21:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-04-02T07:21:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-04-02T07:21:59.520 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-04-02T07:21:59.520 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-04-02T07:21:59.520 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-04-02T07:21:59.520 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-04-02T07:21:59.520 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-02T07:21:59.520 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-04-02T07:21:59.520 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-04-02T07:21:59.520 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-04-02T07:21:59.520 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-04-02T07:21:59.520 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-04-02T07:21:59.520 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-04-02T07:21:59.520 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-04-02T07:21:59.520 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-02T07:21:59.520 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-04-02T07:21:59.521 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-04-02T07:21:59.521 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-04-02T07:21:59.521 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-04-02T07:21:59.521 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-04-02T07:21:59.521 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-04-02T07:21:59.521 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-02T07:21:59.521 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-02T07:21:59.521 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-04-02T07:21:59.521 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-04-02T07:21:59.521 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-04-02T07:21:59.521 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-04-02T07:21:59.521 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-04-02T07:21:59.522 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-02T07:21:59.522 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-04-02T07:21:59.522 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-02T07:21:59.522 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-02T07:21:59.522 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-02T07:21:59.522 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-04-02T07:21:59.522 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-04-02T07:21:59.522 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-04-02T07:21:59.522 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-04-02T07:21:59.522 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-04-02T07:21:59.522 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-02T07:21:59.523 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-04-02T07:21:59.523 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-04-02T07:21:59.523 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-04-02T07:21:59.523 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-04-02T07:21:59.523 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-04-02T07:21:59.523 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-02T07:21:59.523 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-02T07:21:59.523 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-02T07:21:59.523 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-04-02T07:21:59.523 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-02T07:21:59.523 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-04-02T07:21:59.523 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-04-02T07:21:59.523 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-04-02T07:21:59.523 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-04-02T07:21:59.524 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-02T07:21:59.524 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-02T07:21:59.524 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-02T07:21:59.524 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-02T07:21:59.524 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-04-02T07:21:59.524 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-02T07:21:59.524 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-02T07:21:59.524 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-02T07:21:59.524 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-02T07:21:59.524 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-04-02T07:21:59.524 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-02T07:21:59.524 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-04-02T07:21:59.524 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-02T07:21:59.524 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-04-02T07:21:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-02T07:21:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-04-02T07:21:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-02T07:21:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-04-02T07:21:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-02T07:21:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-02T07:21:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-02T07:21:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-04-02T07:21:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-04-02T07:21:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-02T07:21:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-02T07:21:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-02T07:21:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-02T07:21:59.525 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-02T07:21:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-04-02T07:21:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-02T07:21:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-02T07:21:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-04-02T07:21:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-02T07:21:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-02T07:21:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-02T07:21:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-02T07:21:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-02T07:21:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-04-02T07:21:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-02T07:21:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-02T07:21:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-04-02T07:21:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-04-02T07:21:59.527 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-02T07:21:59.527 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-04-02T07:21:59.527 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-02T07:21:59.527 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-02T07:21:59.527 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-02T07:21:59.527 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-02T07:21:59.527 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-02T07:21:59.527 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-02T07:21:59.527 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-02T07:21:59.527 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-02T07:21:59.527 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-02T07:21:59.527 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-02T07:21:59.527 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-04-02T07:21:59.527 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-04-02T07:21:59.528 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-02T07:21:59.528 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-04-02T07:21:59.528 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-04-02T07:21:59.528 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-02T07:21:59.528 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-02T07:21:59.528 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-02T07:21:59.528 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-02T07:21:59.528 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-02T07:21:59.528 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-02T07:21:59.528 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-04-02T07:21:59.528 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-02T07:21:59.528 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-02T07:21:59.528 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-04-02T07:21:59.528 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-04-02T07:21:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-04-02T07:21:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-02T07:21:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-04-02T07:21:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-04-02T07:21:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-04-02T07:21:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-02T07:21:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-04-02T07:21:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-02T07:21:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-04-02T07:21:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-02T07:21:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-04-02T07:21:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-02T07:21:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-04-02T07:21:59.529 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-02T07:21:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-04-02T07:21:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-02T07:21:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-04-02T07:21:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-02T07:21:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-04-02T07:21:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-02T07:21:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-02T07:21:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-02T07:21:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-02T07:21:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-02T07:21:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-04-02T07:21:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-04-02T07:21:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-02T07:21:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-02T07:21:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-02T07:21:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-04-02T07:21:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-02T07:21:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-02T07:21:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-02T07:21:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-02T07:21:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-02T07:21:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-02T07:21:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-04-02T07:21:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-02T07:21:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-04-02T07:21:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-02T07:21:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-02T07:21:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-04-02T07:21:59.532 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-02T07:21:59.532 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-02T07:21:59.532 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-04-02T07:21:59.532 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-02T07:21:59.532 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-04-02T07:21:59.532 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-02T07:21:59.532 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-04-02T07:21:59.532 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-02T07:21:59.532 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-02T07:21:59.532 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-04-02T07:21:59.532 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-02T07:21:59.532 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-02T07:21:59.532 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-02T07:21:59.532 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-02T07:21:59.533 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-02T07:21:59.533 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-04-02T07:21:59.533 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-02T07:21:59.533 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-04-02T07:21:59.533 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-02T07:21:59.533 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-02T07:21:59.533 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-02T07:21:59.533 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-02T07:21:59.533 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-04-02T07:21:59.533 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-02T07:21:59.533 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-04-02T07:21:59.533 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-02T07:21:59.533 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-04-02T07:21:59.534 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-02T07:21:59.534 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-04-02T07:21:59.534 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-02T07:21:59.534 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-02T07:21:59.534 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-04-02T07:21:59.534 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-02T07:21:59.534 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-02T07:21:59.534 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-02T07:21:59.534 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-02T07:21:59.534 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-04-02T07:21:59.534 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-02T07:21:59.534 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-02T07:21:59.534 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-02T07:21:59.534 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-04-02T07:21:59.535 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-02T07:21:59.535 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-02T07:21:59.535 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-02T07:21:59.535 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-02T07:21:59.535 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-04-02T07:21:59.535 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-02T07:21:59.535 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-04-02T07:21:59.535 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-02T07:21:59.535 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-04-02T07:21:59.535 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-02T07:21:59.535 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-02T07:21:59.535 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-04-02T07:21:59.535 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-02T07:21:59.535 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-02T07:21:59.536 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-04-02T07:21:59.536 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-02T07:21:59.536 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-04-02T07:21:59.536 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-04-02T07:21:59.536 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-02T07:21:59.536 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-04-02T07:21:59.536 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-02T07:21:59.536 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-04-02T07:21:59.536 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-04-02T07:21:59.536 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-02T07:21:59.536 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-04-02T07:21:59.536 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-04-02T07:21:59.536 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-04-02T07:21:59.536 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-02T07:21:59.537 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-02T07:21:59.538 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-04-02T07:21:59.538 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-04-02T07:21:59.538 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-04-02T07:21:59.538 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-04-02T07:21:59.538 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-04-02T07:21:59.538 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-04-02T07:21:59.538 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-04-02T07:21:59.538 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-04-02T07:21:59.538 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-04-02T07:21:59.538 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-04-02T07:21:59.538 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-04-02T07:21:59.538 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-04-02T07:21:59.538 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-04-02T07:21:59.539 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-04-02T07:21:59.539 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-04-02T07:21:59.539 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-04-02T07:21:59.539 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-04-02T07:21:59.539 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-04-02T07:21:59.539 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-02T07:21:59.539 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-04-02T07:21:59.539 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-02T07:21:59.539 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-02T07:21:59.539 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-02T07:21:59.539 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-02T07:21:59.539 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-04-02T07:21:59.539 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-04-02T07:21:59.539 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-04-02T07:21:59.540 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-04-02T07:21:59.541 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-04-02T07:21:59.541 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-02T07:21:59.541 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-04-02T07:21:59.541 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-04-02T07:21:59.541 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-04-02T07:21:59.541 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-02T07:21:59.541 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-04-02T07:21:59.541 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-04-02T07:21:59.541 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-04-02T07:21:59.541 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-04-02T07:21:59.541 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-04-02T07:21:59.541 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-04-02T07:21:59.541 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-04-02T07:21:59.541 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-04-02T07:21:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-04-02T07:21:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-04-02T07:21:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-04-02T07:21:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-04-02T07:21:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-04-02T07:21:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-04-02T07:21:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-02T07:21:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-04-02T07:21:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-02T07:21:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-04-02T07:21:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-04-02T07:21:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-04-02T07:21:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-04-02T07:21:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-02T07:21:59.543 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-04-02T07:21:59.543 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-04-02T07:21:59.543 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-02T07:21:59.543 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-04-02T07:21:59.543 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-04-02T07:21:59.543 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-02T07:21:59.543 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-04-02T07:21:59.543 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-02T07:21:59.543 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-02T07:21:59.543 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-02T07:21:59.543 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-02T07:21:59.543 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-02T07:21:59.543 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-02T07:21:59.544 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-02T07:21:59.544 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-02T07:21:59.544 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-02T07:21:59.544 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-04-02T07:21:59.544 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-04-02T07:21:59.544 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-02T07:21:59.544 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-04-02T07:21:59.544 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-02T07:21:59.544 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-02T07:21:59.544 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-02T07:21:59.544 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-02T07:21:59.544 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-02T07:21:59.544 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-04-02T07:21:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-04-02T07:21:59.546 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-04-02T07:21:59.546 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-04-02T07:21:59.546 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-02T07:21:59.546 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-02T07:21:59.546 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-02T07:21:59.546 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-02T07:21:59.546 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-02T07:21:59.546 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-02T07:21:59.546 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-02T07:21:59.546 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-02T07:21:59.546 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-02T07:21:59.546 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-02T07:21:59.546 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-02T07:21:59.547 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-02T07:21:59.548 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-02T07:21:59.548 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-02T07:21:59.548 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-02T07:21:59.548 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-02T07:21:59.548 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-04-02T07:21:59.548 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-04-02T07:21:59.548 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-04-02T07:21:59.548 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-04-02T07:21:59.548 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-04-02T07:21:59.548 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-04-02T07:21:59.548 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-04-02T07:21:59.548 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-04-02T07:21:59.548 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-02T07:21:59.548 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-02T07:21:59.549 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-04-02T07:21:59.550 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-04-02T07:21:59.550 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-02T07:21:59.550 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-04-02T07:21:59.550 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-02T07:21:59.550 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-02T07:21:59.550 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-02T07:21:59.550 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-02T07:21:59.550 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-02T07:21:59.550 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-04-02T07:21:59.550 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-04-02T07:21:59.550 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-02T07:21:59.550 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-02T07:21:59.550 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-02T07:21:59.550 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-04-02T07:21:59.551 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-04-02T07:21:59.552 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-04-02T07:21:59.552 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-04-02T07:21:59.552 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-02T07:21:59.552 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-02T07:21:59.552 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-04-02T07:21:59.552 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-04-02T07:21:59.552 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-04-02T07:21:59.552 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-02T07:21:59.552 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-04-02T07:21:59.552 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-04-02T07:21:59.552 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-04-02T07:21:59.552 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-02T07:21:59.553 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-04-02T07:21:59.553 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-04-02T07:21:59.553 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-02T07:21:59.553 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-02T07:21:59.553 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-02T07:21:59.553 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-04-02T07:21:59.553 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-02T07:21:59.553 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-04-02T07:21:59.553 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-02T07:21:59.553 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-04-02T07:21:59.553 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-04-02T07:21:59.553 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-02T07:21:59.554 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-04-02T07:21:59.554 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-04-02T07:21:59.554 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-04-02T07:21:59.554 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-02T07:21:59.554 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-02T07:21:59.554 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-04-02T07:21:59.554 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-02T07:21:59.554 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-04-02T07:21:59.554 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-04-02T07:21:59.554 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-02T07:21:59.554 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-02T07:21:59.554 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-02T07:21:59.554 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-02T07:21:59.555 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-02T07:21:59.555 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-04-02T07:21:59.555 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-04-02T07:21:59.555 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-04-02T07:21:59.555 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-04-02T07:21:59.555 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-02T07:21:59.555 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-04-02T07:21:59.555 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-02T07:21:59.555 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-04-02T07:21:59.555 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-04-02T07:21:59.555 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-04-02T07:21:59.555 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-04-02T07:21:59.555 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-04-02T07:21:59.555 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-02T07:21:59.556 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-04-02T07:21:59.556 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-02T07:21:59.556 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-04-02T07:21:59.556 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-04-02T07:21:59.556 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-04-02T07:21:59.556 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-04-02T07:21:59.556 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-04-02T07:21:59.556 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-04-02T07:21:59.556 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-02T07:21:59.556 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-04-02T07:21:59.556 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-04-02T07:21:59.556 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-04-02T07:21:59.556 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-02T07:21:59.556 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-02T07:21:59.557 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-02T07:21:59.557 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-02T07:21:59.557 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-02T07:21:59.557 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-02T07:21:59.557 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-02T07:21:59.557 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-02T07:21:59.557 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-02T07:21:59.557 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-02T07:21:59.557 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-02T07:21:59.557 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-02T07:21:59.557 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-02T07:21:59.557 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-02T07:21:59.557 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-02T07:21:59.557 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-02T07:21:59.558 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-02T07:21:59.558 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-02T07:21:59.558 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-04-02T07:21:59.558 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-04-02T07:21:59.558 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-02T07:21:59.558 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-04-02T07:21:59.558 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-04-02T07:21:59.558 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-04-02T07:21:59.558 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-04-02T07:21:59.558 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-04-02T07:21:59.558 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-02T07:21:59.558 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-04-02T07:21:59.558 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-04-02T07:21:59.558 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-02T07:21:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-04-02T07:21:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-04-02T07:21:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-02T07:21:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-04-02T07:21:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-02T07:21:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-04-02T07:21:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-04-02T07:21:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-04-02T07:21:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-04-02T07:21:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-04-02T07:21:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-04-02T07:21:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-04-02T07:21:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-04-02T07:21:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-04-02T07:21:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-04-02T07:21:59.561 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-04-02T07:21:59.561 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-02T07:21:59.561 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-04-02T07:21:59.561 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-04-02T07:21:59.561 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-04-02T07:21:59.561 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-02T07:21:59.561 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-02T07:21:59.561 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-04-02T07:21:59.561 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-04-02T07:21:59.561 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-02T07:21:59.561 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-04-02T07:21:59.561 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-04-02T07:21:59.561 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-04-02T07:21:59.561 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-04-02T07:21:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-04-02T07:21:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-04-02T07:21:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-04-02T07:21:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-02T07:21:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-04-02T07:21:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-04-02T07:21:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-04-02T07:21:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-02T07:21:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-02T07:21:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-04-02T07:21:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-02T07:21:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-02T07:21:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-02T07:21:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-04-02T07:21:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-04-02T07:21:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-04-02T07:21:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-04-02T07:21:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-02T07:21:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-04-02T07:21:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-02T07:21:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-02T07:21:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-02T07:21:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-04-02T07:21:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-04-02T07:21:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-04-02T07:21:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-04-02T07:21:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-04-02T07:21:59.563 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-04-02T07:21:59.564 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-02T07:21:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-04-02T07:21:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-04-02T07:21:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-02T07:21:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-04-02T07:21:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-04-02T07:21:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-02T07:21:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-02T07:21:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-02T07:21:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-02T07:21:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-02T07:21:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-02T07:21:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-02T07:21:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-04-02T07:21:59.565 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-02T07:21:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-02T07:21:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-02T07:21:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-02T07:21:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-02T07:21:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-04-02T07:21:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-04-02T07:21:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-02T07:21:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-02T07:21:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-02T07:21:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-04-02T07:21:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-04-02T07:21:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-02T07:21:59.566 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-02T07:21:59.567 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-04-02T07:21:59.567 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-02T07:21:59.567 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-04-02T07:21:59.567 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-04-02T07:21:59.567 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-02T07:21:59.567 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-04-02T07:21:59.567 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-04-02T07:21:59.567 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-04-02T07:21:59.567 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-04-02T07:21:59.567 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-04-02T07:21:59.567 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-04-02T07:21:59.567 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-02T07:21:59.567 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-04-02T07:21:59.567 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-02T07:21:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-02T07:21:59.569 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-02T07:21:59.569 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-02T07:21:59.569 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-02T07:21:59.569 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-02T07:21:59.569 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-02T07:21:59.569 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-02T07:21:59.569 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-02T07:21:59.569 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-04-02T07:21:59.569 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-04-02T07:21:59.569 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-02T07:21:59.569 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-02T07:21:59.569 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-02T07:21:59.569 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-04-02T07:21:59.569 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-04-02T07:21:59.570 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-02T07:21:59.570 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-02T07:21:59.570 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-04-02T07:21:59.570 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-04-02T07:21:59.570 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-02T07:21:59.570 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-02T07:21:59.570 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-04-02T07:21:59.570 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-04-02T07:21:59.570 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-04-02T07:21:59.570 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-02T07:21:59.570 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-02T07:21:59.570 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-02T07:21:59.570 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-02T07:21:59.570 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-02T07:21:59.571 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-04-02T07:21:59.572 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-04-02T07:21:59.572 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-02T07:21:59.572 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-04-02T07:21:59.572 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-04-02T07:21:59.572 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-04-02T07:21:59.572 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-04-02T07:21:59.572 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-04-02T07:21:59.572 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-04-02T07:21:59.572 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-04-02T07:21:59.572 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-04-02T07:21:59.572 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-04-02T07:21:59.572 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-04-02T07:21:59.572 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-04-02T07:21:59.572 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-04-02T07:21:59.573 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-04-02T07:21:59.574 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-04-02T07:21:59.574 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-04-02T07:21:59.574 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-04-02T07:21:59.574 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-04-02T07:21:59.574 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-04-02T07:21:59.574 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-04-02T07:21:59.574 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-04-02T07:21:59.574 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-04-02T07:21:59.574 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-04-02T07:21:59.574 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-04-02T07:21:59.574 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-04-02T07:21:59.574 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-04-02T07:21:59.574 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-02T07:21:59.575 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-04-02T07:21:59.576 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-04-02T07:21:59.576 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-04-02T07:21:59.576 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-04-02T07:21:59.576 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-04-02T07:21:59.576 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-04-02T07:21:59.576 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-04-02T07:21:59.576 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-04-02T07:21:59.576 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-04-02T07:21:59.576 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-04-02T07:21:59.576 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-04-02T07:21:59.576 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-04-02T07:21:59.577 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-04-02T07:21:59.577 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-04-02T07:21:59.577 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-04-02T07:21:59.577 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-04-02T07:21:59.577 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-04-02T07:21:59.577 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-04-02T07:21:59.577 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-04-02T07:21:59.577 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-04-02T07:21:59.577 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-04-02T07:21:59.577 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-04-02T07:21:59.577 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-04-02T07:21:59.577 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-04-02T07:21:59.577 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-04-02T07:21:59.577 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-04-02T07:21:59.578 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-04-02T07:21:59.578 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-04-02T07:21:59.578 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-04-02T07:21:59.578 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-04-02T07:21:59.578 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-04-02T07:21:59.578 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-04-02T07:21:59.578 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-04-02T07:21:59.578 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-04-02T07:21:59.578 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-04-02T07:21:59.578 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-04-02T07:21:59.578 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-04-02T07:21:59.578 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-04-02T07:21:59.578 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-04-02T07:21:59.578 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-04-02T07:21:59.579 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-04-02T07:21:59.579 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-04-02T07:21:59.579 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-04-02T07:21:59.579 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-04-02T07:21:59.579 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-04-02T07:21:59.579 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-04-02T07:21:59.579 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-04-02T07:21:59.579 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-04-02T07:21:59.579 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-04-02T07:21:59.579 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-04-02T07:21:59.579 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-04-02T07:21:59.579 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-04-02T07:21:59.579 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-04-02T07:21:59.579 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-04-02T07:21:59.580 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-04-02T07:21:59.580 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-04-02T07:21:59.580 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-04-02T07:21:59.580 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-04-02T07:21:59.580 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-04-02T07:21:59.580 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-02T07:21:59.580 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-04-02T07:21:59.580 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-04-02T07:21:59.580 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-04-02T07:21:59.580 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-02T07:21:59.580 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-04-02T07:21:59.580 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-02T07:21:59.580 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-04-02T07:21:59.581 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-02T07:21:59.581 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-04-02T07:21:59.581 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-02T07:21:59.581 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-02T07:21:59.581 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-02T07:21:59.581 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-04-02T07:21:59.581 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-04-02T07:21:59.581 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-04-02T07:21:59.581 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-04-02T07:21:59.581 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-04-02T07:21:59.581 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-04-02T07:21:59.581 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-04-02T07:21:59.581 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-04-02T07:21:59.581 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-04-02T07:21:59.582 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-04-02T07:21:59.582 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-04-02T07:21:59.582 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-04-02T07:21:59.582 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-04-02T07:21:59.582 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-04-02T07:21:59.582 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-04-02T07:21:59.582 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-04-02T07:21:59.582 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-04-02T07:21:59.582 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-04-02T07:21:59.582 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-04-02T07:21:59.582 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-04-02T07:21:59.582 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-04-02T07:21:59.582 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-04-02T07:21:59.582 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-04-02T07:21:59.583 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-04-02T07:21:59.583 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-04-02T07:21:59.583 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-04-02T07:21:59.583 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-04-02T07:21:59.583 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-04-02T07:21:59.583 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-04-02T07:21:59.583 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-02T07:21:59.583 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-04-02T07:21:59.583 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-04-02T07:21:59.583 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-04-02T07:21:59.583 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-04-02T07:21:59.583 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-04-02T07:21:59.583 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-04-02T07:21:59.583 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-04-02T07:21:59.584 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-04-02T07:21:59.585 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-04-02T07:21:59.585 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-04-02T07:21:59.585 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-04-02T07:21:59.585 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-04-02T07:21:59.585 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-04-02T07:21:59.585 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-04-02T07:21:59.585 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-04-02T07:21:59.585 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-04-02T07:21:59.585 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-04-02T07:21:59.585 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-02T07:21:59.585 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-04-02T07:21:59.585 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-04-02T07:21:59.585 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-04-02T07:21:59.585 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-04-02T07:21:59.586 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-04-02T07:21:59.586 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-04-02T07:21:59.586 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-04-02T07:21:59.586 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-04-02T07:21:59.586 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-04-02T07:21:59.586 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-04-02T07:21:59.586 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-04-02T07:21:59.586 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-04-02T07:21:59.586 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-04-02T07:21:59.586 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-04-02T07:21:59.586 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-04-02T07:21:59.586 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-04-02T07:21:59.586 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-04-02T07:21:59.586 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-04-02T07:21:59.587 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-04-02T07:21:59.587 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-04-02T07:21:59.587 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-04-02T07:21:59.587 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-04-02T07:21:59.587 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-02T07:21:59.587 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-02T07:21:59.587 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-02T07:21:59.587 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-02T07:21:59.587 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-02T07:21:59.587 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-02T07:21:59.587 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-02T07:21:59.587 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-02T07:21:59.587 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-02T07:21:59.587 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-02T07:21:59.588 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-02T07:21:59.588 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-02T07:21:59.588 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-02T07:21:59.588 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-02T07:21:59.588 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-02T07:21:59.588 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-02T07:21:59.588 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-02T07:21:59.588 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-02T07:21:59.588 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-02T07:21:59.588 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-02T07:21:59.588 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-04-02T07:21:59.588 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-02T07:21:59.588 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-02T07:21:59.588 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-04-02T07:21:59.589 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-04-02T07:21:59.590 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-04-02T07:21:59.590 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-04-02T07:21:59.590 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-04-02T07:21:59.590 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-04-02T07:21:59.590 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-04-02T07:21:59.590 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-02T07:21:59.590 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-04-02T07:21:59.590 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-02T07:21:59.590 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-04-02T07:21:59.590 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-04-02T07:21:59.590 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-04-02T07:21:59.590 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-04-02T07:21:59.590 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-04-02T07:21:59.590 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-04-02T07:21:59.591 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-04-02T07:21:59.591 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-04-02T07:21:59.591 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-04-02T07:21:59.591 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-04-02T07:21:59.591 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-04-02T07:21:59.591 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-04-02T07:21:59.591 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-02T07:21:59.591 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-02T07:21:59.591 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-02T07:21:59.591 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-02T07:21:59.591 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-02T07:21:59.591 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-02T07:21:59.591 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-02T07:21:59.592 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-02T07:21:59.592 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-02T07:21:59.592 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-02T07:21:59.592 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-02T07:21:59.592 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-02T07:21:59.592 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-02T07:21:59.592 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-02T07:21:59.592 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-02T07:21:59.592 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-02T07:21:59.592 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-02T07:21:59.592 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-02T07:21:59.592 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-04-02T07:21:59.593 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-04-02T07:21:59.594 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-04-02T07:21:59.594 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-04-02T07:21:59.594 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-04-02T07:21:59.594 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-02T07:21:59.594 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-04-02T07:21:59.594 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-04-02T07:21:59.594 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-04-02T07:21:59.594 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-04-02T07:21:59.597 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-04-02T07:21:59.597 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-02T07:21:59.597 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-04-02T07:21:59.597 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-04-02T07:21:59.597 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-04-02T07:21:59.597 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-02T07:21:59.597 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-02T07:21:59.597 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-02T07:21:59.597 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-02T07:21:59.597 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-02T07:21:59.597 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-02T07:21:59.597 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-02T07:21:59.597 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-02T07:21:59.598 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-02T07:21:59.598 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-02T07:21:59.598 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-02T07:21:59.598 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-02T07:21:59.598 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-02T07:21:59.598 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-02T07:21:59.598 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-02T07:21:59.598 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-02T07:21:59.598 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-02T07:21:59.598 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-02T07:21:59.598 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-02T07:21:59.598 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-02T07:21:59.598 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-02T07:21:59.598 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-02T07:21:59.599 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-02T07:21:59.599 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-02T07:21:59.599 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-04-02T07:21:59.599 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-04-02T07:21:59.599 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-02T07:21:59.599 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-04-02T07:21:59.599 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-04-02T07:21:59.599 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-04-02T07:21:59.599 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-04-02T07:21:59.599 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-02T07:21:59.599 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-04-02T07:21:59.599 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-04-02T07:21:59.599 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-04-02T07:21:59.600 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-04-02T07:21:59.600 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-04-02T07:21:59.600 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-04-02T07:21:59.600 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-04-02T07:21:59.600 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-04-02T07:21:59.600 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-04-02T07:21:59.600 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-04-02T07:21:59.600 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-04-02T07:21:59.600 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-04-02T07:21:59.600 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-04-02T07:21:59.600 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-04-02T07:21:59.600 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-02T07:21:59.600 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-02T07:21:59.601 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-02T07:21:59.601 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-02T07:21:59.601 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-02T07:21:59.601 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-02T07:21:59.601 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-02T07:21:59.601 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-02T07:21:59.601 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-02T07:21:59.601 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-02T07:21:59.601 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-02T07:21:59.601 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-02T07:21:59.601 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-02T07:21:59.601 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-02T07:21:59.601 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-02T07:21:59.601 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-04-02T07:21:59.602 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-02T07:21:59.602 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-02T07:21:59.602 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-02T07:21:59.602 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-02T07:21:59.602 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-02T07:21:59.602 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-02T07:21:59.602 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-02T07:21:59.602 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-02T07:21:59.602 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-02T07:21:59.602 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-02T07:21:59.602 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-02T07:21:59.602 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-02T07:21:59.602 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-02T07:21:59.602 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-02T07:21:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-04-02T07:21:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-04-02T07:21:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-02T07:21:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-02T07:21:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-02T07:21:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-02T07:21:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-02T07:21:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-02T07:21:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-02T07:21:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-02T07:21:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-02T07:21:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-02T07:21:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-02T07:21:59.603 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-02T07:21:59.604 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-04-02T07:21:59.605 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-02T07:21:59.605 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-02T07:21:59.605 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-04-02T07:21:59.605 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-04-02T07:21:59.605 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-04-02T07:21:59.605 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-02T07:21:59.605 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-04-02T07:21:59.605 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-02T07:21:59.605 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-04-02T07:21:59.605 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-04-02T07:21:59.605 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-04-02T07:21:59.605 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-04-02T07:21:59.605 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-04-02T07:21:59.605 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-04-02T07:21:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-04-02T07:21:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-04-02T07:21:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-04-02T07:21:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-04-02T07:21:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-04-02T07:21:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-04-02T07:21:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-04-02T07:21:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-04-02T07:21:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-02T07:21:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-02T07:21:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-02T07:21:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-04-02T07:21:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-04-02T07:21:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-04-02T07:21:59.607 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-04-02T07:21:59.607 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-02T07:21:59.607 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-04-02T07:21:59.607 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-04-02T07:21:59.607 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-02T07:21:59.607 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-04-02T07:21:59.607 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-04-02T07:21:59.607 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-02T07:21:59.607 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-04-02T07:21:59.607 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-02T07:21:59.607 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-04-02T07:21:59.607 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-02T07:21:59.607 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-04-02T07:21:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-04-02T07:21:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-04-02T07:21:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-04-02T07:21:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-04-02T07:21:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-04-02T07:21:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-04-02T07:21:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-04-02T07:21:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-04-02T07:21:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-04-02T07:21:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-04-02T07:21:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-04-02T07:21:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-04-02T07:21:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-04-02T07:21:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-04-02T07:21:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-04-02T07:21:59.610 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-04-02T07:21:59.610 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-04-02T07:21:59.610 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-04-02T07:21:59.610 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-04-02T07:21:59.610 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-04-02T07:21:59.610 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-04-02T07:21:59.610 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-04-02T07:21:59.610 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-04-02T07:21:59.610 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-04-02T07:21:59.610 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-04-02T07:21:59.610 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-04-02T07:21:59.610 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-04-02T07:21:59.611 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-04-02T07:21:59.611 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-04-02T07:21:59.611 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-04-02T07:21:59.611 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-04-02T07:21:59.611 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-04-02T07:21:59.611 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-04-02T07:21:59.611 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-04-02T07:21:59.611 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-04-02T07:21:59.611 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-04-02T07:21:59.611 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-04-02T07:21:59.611 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-04-02T07:21:59.611 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-04-02T07:21:59.611 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-04-02T07:21:59.611 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-04-02T07:21:59.612 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-04-02T07:21:59.612 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-04-02T07:21:59.612 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-04-02T07:21:59.612 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-04-02T07:21:59.612 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-04-02T07:21:59.612 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-04-02T07:21:59.612 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-04-02T07:21:59.612 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-04-02T07:21:59.612 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-04-02T07:21:59.612 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-04-02T07:21:59.612 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-04-02T07:21:59.612 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-04-02T07:21:59.612 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-04-02T07:21:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-04-02T07:21:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-04-02T07:21:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-04-02T07:21:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-04-02T07:21:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-04-02T07:21:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-04-02T07:21:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-04-02T07:21:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-04-02T07:21:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-04-02T07:21:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-04-02T07:21:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-04-02T07:21:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-04-02T07:21:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-04-02T07:21:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-04-02T07:21:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-04-02T07:21:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-04-02T07:21:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-04-02T07:21:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-04-02T07:21:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-04-02T07:21:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-04-02T07:21:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-04-02T07:21:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-04-02T07:21:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-04-02T07:21:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-04-02T07:21:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-04-02T07:21:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-04-02T07:21:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-04-02T07:21:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-04-02T07:21:59.616 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-04-02T07:21:59.617 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-04-02T07:21:59.617 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-02T07:21:59.617 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-02T07:21:59.617 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-04-02T07:21:59.617 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-02T07:21:59.617 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-02T07:21:59.617 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-04-02T07:21:59.617 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-02T07:21:59.617 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-04-02T07:21:59.617 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-04-02T07:21:59.617 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-04-02T07:21:59.617 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-02T07:21:59.617 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-04-02T07:21:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-04-02T07:21:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-04-02T07:21:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-04-02T07:21:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-04-02T07:21:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-04-02T07:21:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-04-02T07:21:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-04-02T07:21:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-04-02T07:21:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-04-02T07:21:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-04-02T07:21:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-04-02T07:21:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-04-02T07:21:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-04-02T07:21:59.619 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-04-02T07:21:59.619 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-04-02T07:21:59.619 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-04-02T07:21:59.619 INFO:teuthology.orchestra.run.smithi089.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-04-02T07:21:59.622 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-04-02T07:21:59.623 DEBUG:teuthology.orchestra.run.smithi089:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-02T07:21:59.665 INFO:teuthology.orchestra.run.smithi089.stdout:postinstall scriptlet (using /bin/sh): 2024-04-02T07:21:59.666 INFO:teuthology.orchestra.run.smithi089.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-02T07:21:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-04-02T07:21:59.666 INFO:teuthology.orchestra.run.smithi089.stdout:fi 2024-04-02T07:21:59.666 INFO:teuthology.orchestra.run.smithi089.stdout:for file in vmlinuz System.map config; do 2024-04-02T07:21:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-04-02T07:21:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-04-02T07:21:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: fi 2024-04-02T07:21:59.666 INFO:teuthology.orchestra.run.smithi089.stdout:done 2024-04-02T07:21:59.666 INFO:teuthology.orchestra.run.smithi089.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-02T07:21:59.666 INFO:teuthology.orchestra.run.smithi089.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-04-02T07:21:59.666 INFO:teuthology.orchestra.run.smithi089.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-04-02T07:21:59.666 INFO:teuthology.orchestra.run.smithi089.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-04-02T07:21:59.666 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-04-02T07:21:59.667 INFO:teuthology.orchestra.run.smithi089.stdout:fi 2024-04-02T07:21:59.667 INFO:teuthology.orchestra.run.smithi089.stdout:postuninstall scriptlet (using /bin/sh): 2024-04-02T07:21:59.667 INFO:teuthology.orchestra.run.smithi089.stdout:if [ -x /sbin/update-bootloader ]; then 2024-04-02T07:21:59.667 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-04-02T07:21:59.667 INFO:teuthology.orchestra.run.smithi089.stdout:fi 2024-04-02T07:21:59.668 DEBUG:teuthology.orchestra.run.smithi089:> sudo rpm -qi grub2-tools 2024-04-02T07:21:59.765 INFO:teuthology.orchestra.run.smithi089.stdout:Name : grub2-tools 2024-04-02T07:21:59.766 INFO:teuthology.orchestra.run.smithi089.stdout:Epoch : 1 2024-04-02T07:21:59.766 INFO:teuthology.orchestra.run.smithi089.stdout:Version : 2.02 2024-04-02T07:21:59.766 INFO:teuthology.orchestra.run.smithi089.stdout:Release : 129.el8 2024-04-02T07:21:59.766 INFO:teuthology.orchestra.run.smithi089.stdout:Architecture: x86_64 2024-04-02T07:21:59.766 INFO:teuthology.orchestra.run.smithi089.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-02T07:21:59.766 INFO:teuthology.orchestra.run.smithi089.stdout:Group : System Environment/Base 2024-04-02T07:21:59.766 INFO:teuthology.orchestra.run.smithi089.stdout:Size : 9265586 2024-04-02T07:21:59.766 INFO:teuthology.orchestra.run.smithi089.stdout:License : GPLv3+ 2024-04-02T07:21:59.766 INFO:teuthology.orchestra.run.smithi089.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-02T07:21:59.766 INFO:teuthology.orchestra.run.smithi089.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-02T07:21:59.766 INFO:teuthology.orchestra.run.smithi089.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-02T07:21:59.766 INFO:teuthology.orchestra.run.smithi089.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-02T07:21:59.766 INFO:teuthology.orchestra.run.smithi089.stdout:Relocations : (not relocatable) 2024-04-02T07:21:59.767 INFO:teuthology.orchestra.run.smithi089.stdout:Packager : CentOS BuildSystem 2024-04-02T07:21:59.767 INFO:teuthology.orchestra.run.smithi089.stdout:Vendor : CentOS 2024-04-02T07:21:59.767 INFO:teuthology.orchestra.run.smithi089.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-02T07:21:59.767 INFO:teuthology.orchestra.run.smithi089.stdout:Summary : Support tools for GRUB. 2024-04-02T07:21:59.767 INFO:teuthology.orchestra.run.smithi089.stdout:Description : 2024-04-02T07:21:59.767 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:21:59.767 INFO:teuthology.orchestra.run.smithi089.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-02T07:21:59.767 INFO:teuthology.orchestra.run.smithi089.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-02T07:21:59.767 INFO:teuthology.orchestra.run.smithi089.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-02T07:21:59.767 INFO:teuthology.orchestra.run.smithi089.stdout:hardware devices. 2024-04-02T07:21:59.767 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:21:59.767 INFO:teuthology.orchestra.run.smithi089.stdout:This subpackage provides tools for support of all platforms. 2024-04-02T07:21:59.768 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-02T07:21:59.768 INFO:teuthology.task.kernel:Updating grub on smithi089 to boot 6.8.0-g1df0c7e8a235 2024-04-02T07:21:59.769 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-02T07:22:02.216 INFO:teuthology.orchestra.run.smithi089.stderr:Generating grub configuration file ... 2024-04-02T07:22:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:done 2024-04-02T07:22:04.015 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-04-02T07:22:04.058 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.dI2MSdvOJq 2024-04-02T07:22:04.059 DEBUG:teuthology.orchestra.run.smithi089:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dI2MSdvOJq 2024-04-02T07:22:04.132 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.dI2MSdvOJq 2024-04-02T07:22:04.283 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.dI2MSdvOJq is 6KB 2024-04-02T07:22:04.334 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.dI2MSdvOJq 2024-04-02T07:22:04.352 DEBUG:teuthology.orchestra.run.smithi089:> sudo /bin/ls /boot/loader/entries || true 2024-04-02T07:22:04.429 INFO:teuthology.orchestra.run.smithi089.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-02T07:22:04.430 INFO:teuthology.orchestra.run.smithi089.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-02T07:22:04.430 INFO:teuthology.orchestra.run.smithi089.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-02T07:22:04.430 INFO:teuthology.orchestra.run.smithi089.stdout:a1b29d6c289747bdb22930598d39874a-6.8.0-g1df0c7e8a235.conf 2024-04-02T07:22:04.431 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-6.8.0-g1df0c7e8a235 2024-04-02T07:22:04.581 DEBUG:teuthology.orchestra.run.smithi089:> sudo shutdown -r now 2024-04-02T07:22:14.700 INFO:teuthology.orchestra.run.smithi113.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-04-02T07:22:16.072 INFO:teuthology.orchestra.run.smithi113.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-02T07:22:34.583 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-02T07:22:34.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-04-02T07:22:34.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T07:22:39.344 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi113.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-02T07:22:39.344 DEBUG:teuthology.orchestra.run.smithi113:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-02T07:22:39.405 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-04-02T07:22:39.405 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-04-02T07:22:39.405 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-04-02T07:22:39.406 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-04-02T07:22:39.406 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-04-02T07:22:39.406 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-04-02T07:22:39.406 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-04-02T07:22:39.406 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-02T07:22:39.406 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-02T07:22:39.406 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-02T07:22:39.406 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-02T07:22:39.406 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-02T07:22:39.406 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-02T07:22:39.406 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-02T07:22:39.406 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-02T07:22:39.406 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-02T07:22:39.407 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-02T07:22:39.407 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-02T07:22:39.407 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-02T07:22:39.407 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-02T07:22:39.407 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-02T07:22:39.407 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-02T07:22:39.407 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-02T07:22:39.407 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-02T07:22:39.407 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-02T07:22:39.407 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-02T07:22:39.407 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-02T07:22:39.407 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-04-02T07:22:39.407 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-04-02T07:22:39.407 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-04-02T07:22:39.408 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-02T07:22:39.408 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-04-02T07:22:39.408 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-02T07:22:39.408 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-02T07:22:39.408 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-04-02T07:22:39.408 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-04-02T07:22:39.408 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-04-02T07:22:39.409 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-04-02T07:22:39.409 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-04-02T07:22:39.409 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-04-02T07:22:39.409 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-04-02T07:22:39.409 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-04-02T07:22:39.409 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-02T07:22:39.409 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-04-02T07:22:39.409 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-04-02T07:22:39.409 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-04-02T07:22:39.409 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-04-02T07:22:39.409 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-04-02T07:22:39.409 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-04-02T07:22:39.409 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-04-02T07:22:39.409 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-04-02T07:22:39.410 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-04-02T07:22:39.410 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-04-02T07:22:39.410 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-04-02T07:22:39.410 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-04-02T07:22:39.410 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-04-02T07:22:39.410 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-04-02T07:22:39.410 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-04-02T07:22:39.410 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-04-02T07:22:39.410 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-04-02T07:22:39.410 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-04-02T07:22:39.410 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-04-02T07:22:39.410 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-04-02T07:22:39.410 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-04-02T07:22:39.410 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-04-02T07:22:39.411 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-04-02T07:22:39.412 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-04-02T07:22:39.413 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-04-02T07:22:39.414 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-04-02T07:22:39.415 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-04-02T07:22:39.415 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-04-02T07:22:39.415 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-02T07:22:39.415 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-02T07:22:39.415 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-04-02T07:22:39.415 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-04-02T07:22:39.415 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-04-02T07:22:39.415 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-04-02T07:22:39.415 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-04-02T07:22:39.415 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-04-02T07:22:39.415 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-04-02T07:22:39.415 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-04-02T07:22:39.415 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-04-02T07:22:39.415 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-04-02T07:22:39.416 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-04-02T07:22:39.416 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-04-02T07:22:39.416 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-04-02T07:22:39.416 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-04-02T07:22:39.416 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-04-02T07:22:39.416 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-04-02T07:22:39.416 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-04-02T07:22:39.416 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-04-02T07:22:39.416 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-04-02T07:22:39.417 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-04-02T07:22:39.417 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-04-02T07:22:39.417 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-04-02T07:22:39.417 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-04-02T07:22:39.417 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-04-02T07:22:39.417 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-04-02T07:22:39.417 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-04-02T07:22:39.417 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-04-02T07:22:39.417 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-02T07:22:39.418 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-02T07:22:39.418 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-04-02T07:22:39.418 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-04-02T07:22:39.418 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-04-02T07:22:39.418 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-04-02T07:22:39.418 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-04-02T07:22:39.418 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-04-02T07:22:39.418 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-04-02T07:22:39.418 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-04-02T07:22:39.418 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-04-02T07:22:39.418 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-04-02T07:22:39.418 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-04-02T07:22:39.419 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-02T07:22:39.419 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-04-02T07:22:39.419 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-04-02T07:22:39.420 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-04-02T07:22:39.420 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-04-02T07:22:39.420 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-04-02T07:22:39.420 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-04-02T07:22:39.420 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-04-02T07:22:39.420 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-02T07:22:39.420 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-04-02T07:22:39.420 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-04-02T07:22:39.420 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-04-02T07:22:39.420 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-04-02T07:22:39.420 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-04-02T07:22:39.420 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-04-02T07:22:39.421 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-02T07:22:39.421 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-02T07:22:39.421 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-02T07:22:39.421 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-04-02T07:22:39.421 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-02T07:22:39.421 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-04-02T07:22:39.421 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-02T07:22:39.421 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-02T07:22:39.421 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-02T07:22:39.421 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-02T07:22:39.421 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-02T07:22:39.421 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-02T07:22:39.421 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-04-02T07:22:39.421 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-04-02T07:22:39.422 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-04-02T07:22:39.422 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-04-02T07:22:39.422 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-04-02T07:22:39.422 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-04-02T07:22:39.422 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-04-02T07:22:39.422 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-02T07:22:39.422 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-02T07:22:39.422 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-02T07:22:39.423 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-02T07:22:39.423 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-02T07:22:39.423 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-04-02T07:22:39.423 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-04-02T07:22:39.423 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-04-02T07:22:39.423 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-02T07:22:39.423 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-02T07:22:39.423 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-02T07:22:39.423 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-02T07:22:39.423 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-02T07:22:39.423 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-02T07:22:39.423 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-04-02T07:22:39.423 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-04-02T07:22:39.423 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-04-02T07:22:39.424 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-04-02T07:22:39.425 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-04-02T07:22:39.426 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-04-02T07:22:39.426 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-04-02T07:22:39.426 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-04-02T07:22:39.426 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-04-02T07:22:39.426 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-04-02T07:22:39.426 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-04-02T07:22:39.426 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-02T07:22:39.426 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-04-02T07:22:39.426 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-04-02T07:22:39.426 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-02T07:22:39.426 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-04-02T07:22:39.426 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-04-02T07:22:39.427 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-04-02T07:22:39.428 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-04-02T07:22:39.428 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-04-02T07:22:39.428 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-04-02T07:22:39.428 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-04-02T07:22:39.428 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-02T07:22:39.428 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-02T07:22:39.428 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-04-02T07:22:39.428 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-04-02T07:22:39.428 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-04-02T07:22:39.428 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-04-02T07:22:39.428 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-04-02T07:22:39.428 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-04-02T07:22:39.428 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-04-02T07:22:39.429 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-04-02T07:22:39.429 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-04-02T07:22:39.429 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-02T07:22:39.429 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-04-02T07:22:39.429 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-02T07:22:39.429 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-02T07:22:39.429 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-02T07:22:39.429 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-02T07:22:39.429 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-02T07:22:39.429 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-02T07:22:39.429 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-02T07:22:39.429 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-02T07:22:39.429 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-02T07:22:39.429 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-04-02T07:22:39.430 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-04-02T07:22:39.430 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-04-02T07:22:39.430 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-02T07:22:39.430 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-04-02T07:22:39.430 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-04-02T07:22:39.430 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-04-02T07:22:39.430 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-04-02T07:22:39.430 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-04-02T07:22:39.430 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-02T07:22:39.430 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-04-02T07:22:39.430 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-04-02T07:22:39.430 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-04-02T07:22:39.430 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-04-02T07:22:39.431 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-02T07:22:39.432 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-04-02T07:22:39.432 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-04-02T07:22:39.432 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-04-02T07:22:39.432 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-04-02T07:22:39.432 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-04-02T07:22:39.432 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-04-02T07:22:39.432 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-04-02T07:22:39.432 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-04-02T07:22:39.433 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-04-02T07:22:39.433 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-04-02T07:22:39.433 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-04-02T07:22:39.433 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-04-02T07:22:39.433 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-04-02T07:22:39.433 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-04-02T07:22:39.433 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-04-02T07:22:39.433 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-04-02T07:22:39.433 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-04-02T07:22:39.433 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-04-02T07:22:39.433 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-04-02T07:22:39.433 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-02T07:22:39.433 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-04-02T07:22:39.433 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-04-02T07:22:39.434 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-02T07:22:39.434 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-04-02T07:22:39.434 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-04-02T07:22:39.434 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-04-02T07:22:39.434 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-04-02T07:22:39.434 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-04-02T07:22:39.434 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-04-02T07:22:39.434 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-04-02T07:22:39.434 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-04-02T07:22:39.434 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-04-02T07:22:39.434 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-04-02T07:22:39.434 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-04-02T07:22:39.434 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-04-02T07:22:39.434 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-04-02T07:22:39.435 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-04-02T07:22:39.435 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-02T07:22:39.435 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-04-02T07:22:39.435 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-04-02T07:22:39.435 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-04-02T07:22:39.435 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-04-02T07:22:39.435 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-04-02T07:22:39.435 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-04-02T07:22:39.435 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-04-02T07:22:39.435 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-04-02T07:22:39.435 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-04-02T07:22:39.435 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-04-02T07:22:39.435 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-04-02T07:22:39.436 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-04-02T07:22:39.436 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-04-02T07:22:39.436 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-04-02T07:22:39.436 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-04-02T07:22:39.436 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-04-02T07:22:39.436 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-04-02T07:22:39.436 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-04-02T07:22:39.436 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-04-02T07:22:39.436 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-04-02T07:22:39.436 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-04-02T07:22:39.436 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-04-02T07:22:39.436 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-04-02T07:22:39.436 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-04-02T07:22:39.436 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-04-02T07:22:39.437 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-04-02T07:22:39.438 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-04-02T07:22:39.438 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-04-02T07:22:39.438 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-04-02T07:22:39.438 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-04-02T07:22:39.438 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-04-02T07:22:39.438 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-04-02T07:22:39.438 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-04-02T07:22:39.438 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-04-02T07:22:39.438 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-02T07:22:39.438 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-02T07:22:39.438 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-02T07:22:39.438 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-02T07:22:39.438 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-02T07:22:39.439 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-02T07:22:39.439 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-02T07:22:39.439 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-02T07:22:39.439 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-02T07:22:39.439 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-02T07:22:39.439 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-02T07:22:39.439 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-02T07:22:39.439 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-04-02T07:22:39.439 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-04-02T07:22:39.439 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-04-02T07:22:39.439 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-02T07:22:39.439 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-04-02T07:22:39.439 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-04-02T07:22:39.439 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-04-02T07:22:39.440 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-04-02T07:22:39.440 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-04-02T07:22:39.440 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-04-02T07:22:39.440 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-04-02T07:22:39.440 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-04-02T07:22:39.440 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-04-02T07:22:39.440 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-04-02T07:22:39.440 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-04-02T07:22:39.440 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-04-02T07:22:39.440 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-04-02T07:22:39.440 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-04-02T07:22:39.440 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-04-02T07:22:39.440 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-04-02T07:22:39.441 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-04-02T07:22:39.441 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-04-02T07:22:39.441 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-04-02T07:22:39.441 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-04-02T07:22:39.441 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-04-02T07:22:39.441 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-04-02T07:22:39.441 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-04-02T07:22:39.441 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-04-02T07:22:39.441 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-04-02T07:22:39.441 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-04-02T07:22:39.441 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-04-02T07:22:39.441 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-02T07:22:39.441 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-02T07:22:39.441 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-04-02T07:22:39.442 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-02T07:22:39.442 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-02T07:22:39.442 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-02T07:22:39.442 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-02T07:22:39.442 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-02T07:22:39.442 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-02T07:22:39.442 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-02T07:22:39.442 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-02T07:22:39.442 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-02T07:22:39.442 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-02T07:22:39.442 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-02T07:22:39.442 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-02T07:22:39.442 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-04-02T07:22:39.443 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-04-02T07:22:39.444 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-04-02T07:22:39.444 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-02T07:22:39.444 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-04-02T07:22:39.444 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-02T07:22:39.444 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-02T07:22:39.444 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-02T07:22:39.444 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-02T07:22:39.444 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-02T07:22:39.444 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-04-02T07:22:39.444 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-04-02T07:22:39.444 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-02T07:22:39.444 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-04-02T07:22:39.444 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-02T07:22:39.444 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-04-02T07:22:39.445 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-02T07:22:39.445 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-04-02T07:22:39.445 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-02T07:22:39.445 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-04-02T07:22:39.445 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-04-02T07:22:39.445 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-02T07:22:39.445 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-04-02T07:22:39.445 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-02T07:22:39.445 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-04-02T07:22:39.445 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-02T07:22:39.445 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-04-02T07:22:39.445 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-02T07:22:39.445 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-04-02T07:22:39.445 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-02T07:22:39.446 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-04-02T07:22:39.446 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-04-02T07:22:39.446 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-02T07:22:39.446 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-02T07:22:39.446 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-04-02T07:22:39.446 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-04-02T07:22:39.446 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-04-02T07:22:39.446 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-02T07:22:39.446 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-04-02T07:22:39.446 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-04-02T07:22:39.446 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-04-02T07:22:39.446 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-04-02T07:22:39.446 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-02T07:22:39.446 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-04-02T07:22:39.447 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-02T07:22:39.447 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-02T07:22:39.447 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-02T07:22:39.447 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-04-02T07:22:39.447 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-02T07:22:39.447 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-04-02T07:22:39.447 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-04-02T07:22:39.447 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-04-02T07:22:39.447 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-02T07:22:39.447 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-04-02T07:22:39.447 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-04-02T07:22:39.447 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-04-02T07:22:39.447 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-04-02T07:22:39.447 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-04-02T07:22:39.448 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-04-02T07:22:39.449 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-04-02T07:22:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-04-02T07:22:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-02T07:22:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-04-02T07:22:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-04-02T07:22:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-04-02T07:22:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-04-02T07:22:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-04-02T07:22:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-04-02T07:22:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-04-02T07:22:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-04-02T07:22:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-04-02T07:22:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-04-02T07:22:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-04-02T07:22:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-04-02T07:22:39.451 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-04-02T07:22:39.451 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-04-02T07:22:39.451 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-04-02T07:22:39.451 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-04-02T07:22:39.451 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-04-02T07:22:39.451 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-04-02T07:22:39.451 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-04-02T07:22:39.451 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-04-02T07:22:39.451 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-04-02T07:22:39.451 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-04-02T07:22:39.451 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-04-02T07:22:39.451 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-02T07:22:39.451 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-04-02T07:22:39.451 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-04-02T07:22:39.452 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-04-02T07:22:39.452 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-04-02T07:22:39.452 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-04-02T07:22:39.452 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-04-02T07:22:39.452 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-04-02T07:22:39.452 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-02T07:22:39.452 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-04-02T07:22:39.452 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-04-02T07:22:39.452 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-04-02T07:22:39.452 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-04-02T07:22:39.452 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-04-02T07:22:39.452 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-04-02T07:22:39.452 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-02T07:22:39.452 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-04-02T07:22:39.453 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-04-02T07:22:39.454 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-02T07:22:39.454 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-04-02T07:22:39.454 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-04-02T07:22:39.454 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-04-02T07:22:39.454 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-04-02T07:22:39.454 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-04-02T07:22:39.454 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-02T07:22:39.454 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-02T07:22:39.454 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-02T07:22:39.454 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-04-02T07:22:39.454 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-02T07:22:39.454 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-04-02T07:22:39.454 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-04-02T07:22:39.455 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-02T07:22:39.456 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-02T07:22:39.457 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-04-02T07:22:39.457 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-02T07:22:39.457 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-02T07:22:39.457 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-04-02T07:22:39.457 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-02T07:22:39.457 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-02T07:22:39.457 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-02T07:22:39.457 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-02T07:22:39.457 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-02T07:22:39.457 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-04-02T07:22:39.457 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-02T07:22:39.457 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-02T07:22:39.457 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-04-02T07:22:39.457 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-04-02T07:22:39.458 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-02T07:22:39.459 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-04-02T07:22:39.459 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-04-02T07:22:39.459 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-02T07:22:39.459 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-02T07:22:39.459 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-02T07:22:39.459 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-02T07:22:39.459 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-02T07:22:39.459 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-02T07:22:39.459 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-04-02T07:22:39.459 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-02T07:22:39.459 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-02T07:22:39.459 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-04-02T07:22:39.459 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-04-02T07:22:39.459 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-04-02T07:22:39.460 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-02T07:22:39.460 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-04-02T07:22:39.460 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-04-02T07:22:39.460 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-04-02T07:22:39.460 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-02T07:22:39.460 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-04-02T07:22:39.460 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-02T07:22:39.460 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-04-02T07:22:39.460 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-02T07:22:39.460 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-04-02T07:22:39.460 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-02T07:22:39.460 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-04-02T07:22:39.460 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-02T07:22:39.460 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-02T07:22:39.461 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-04-02T07:22:39.462 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-02T07:22:39.462 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-02T07:22:39.462 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-02T07:22:39.462 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-02T07:22:39.462 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-02T07:22:39.462 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-02T07:22:39.462 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-04-02T07:22:39.462 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-02T07:22:39.462 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-04-02T07:22:39.462 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-02T07:22:39.462 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-02T07:22:39.462 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-04-02T07:22:39.462 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-02T07:22:39.462 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-02T07:22:39.463 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-04-02T07:22:39.463 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-02T07:22:39.463 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-04-02T07:22:39.463 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-02T07:22:39.463 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-04-02T07:22:39.463 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-02T07:22:39.463 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-02T07:22:39.463 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-04-02T07:22:39.463 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-02T07:22:39.463 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-02T07:22:39.463 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-02T07:22:39.463 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-02T07:22:39.464 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-02T07:22:39.465 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-02T07:22:39.466 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-02T07:22:39.466 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-04-02T07:22:39.466 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-02T07:22:39.466 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-04-02T07:22:39.466 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-02T07:22:39.466 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-04-02T07:22:39.466 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-02T07:22:39.466 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-02T07:22:39.466 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-04-02T07:22:39.466 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-02T07:22:39.466 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-02T07:22:39.466 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-04-02T07:22:39.466 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-02T07:22:39.466 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-04-02T07:22:39.467 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-04-02T07:22:39.467 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-02T07:22:39.467 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-04-02T07:22:39.467 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-02T07:22:39.467 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-04-02T07:22:39.467 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-04-02T07:22:39.467 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-02T07:22:39.467 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-04-02T07:22:39.467 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-04-02T07:22:39.467 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-04-02T07:22:39.467 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-04-02T07:22:39.467 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-02T07:22:39.467 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-02T07:22:39.467 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-02T07:22:39.468 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-04-02T07:22:39.468 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-04-02T07:22:39.468 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-04-02T07:22:39.468 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-04-02T07:22:39.468 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-04-02T07:22:39.468 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-02T07:22:39.468 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-02T07:22:39.468 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-04-02T07:22:39.468 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-04-02T07:22:39.468 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-02T07:22:39.468 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-02T07:22:39.468 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-02T07:22:39.468 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-04-02T07:22:39.468 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-04-02T07:22:39.469 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-04-02T07:22:39.470 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-04-02T07:22:39.470 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-02T07:22:39.470 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-04-02T07:22:39.470 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-02T07:22:39.470 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-02T07:22:39.470 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-02T07:22:39.470 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-02T07:22:39.470 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-04-02T07:22:39.470 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-04-02T07:22:39.470 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-04-02T07:22:39.470 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-04-02T07:22:39.470 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-04-02T07:22:39.470 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-04-02T07:22:39.470 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-04-02T07:22:39.471 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-04-02T07:22:39.472 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-04-02T07:22:39.473 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-02T07:22:39.474 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-04-02T07:22:39.474 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-02T07:22:39.474 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-02T07:22:39.474 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-02T07:22:39.474 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-02T07:22:39.474 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-02T07:22:39.474 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-02T07:22:39.474 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-02T07:22:39.474 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-02T07:22:39.474 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-02T07:22:39.474 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-04-02T07:22:39.474 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-04-02T07:22:39.474 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-02T07:22:39.474 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-04-02T07:22:39.475 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-04-02T07:22:39.476 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-02T07:22:39.476 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-02T07:22:39.476 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-04-02T07:22:39.476 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-04-02T07:22:39.476 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-04-02T07:22:39.476 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-04-02T07:22:39.476 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-04-02T07:22:39.476 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-04-02T07:22:39.476 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-02T07:22:39.476 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-02T07:22:39.476 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-02T07:22:39.476 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-02T07:22:39.476 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-02T07:22:39.476 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-02T07:22:39.477 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-02T07:22:39.478 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-02T07:22:39.478 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-02T07:22:39.478 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-02T07:22:39.478 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-02T07:22:39.478 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-02T07:22:39.478 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-02T07:22:39.478 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-02T07:22:39.478 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-02T07:22:39.478 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-02T07:22:39.478 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-04-02T07:22:39.478 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-04-02T07:22:39.478 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-04-02T07:22:39.478 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-04-02T07:22:39.478 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-04-02T07:22:39.479 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-04-02T07:22:39.479 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-04-02T07:22:39.479 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-04-02T07:22:39.479 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-02T07:22:39.479 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-04-02T07:22:39.479 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-04-02T07:22:39.479 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-02T07:22:39.479 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-02T07:22:39.479 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-04-02T07:22:39.479 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-04-02T07:22:39.479 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-02T07:22:39.479 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-04-02T07:22:39.480 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-04-02T07:22:39.480 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-02T07:22:39.480 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-04-02T07:22:39.480 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-04-02T07:22:39.480 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-04-02T07:22:39.480 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-02T07:22:39.480 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-04-02T07:22:39.480 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-02T07:22:39.480 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-04-02T07:22:39.480 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-04-02T07:22:39.480 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-02T07:22:39.480 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-04-02T07:22:39.480 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-02T07:22:39.480 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-04-02T07:22:39.481 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-02T07:22:39.482 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-02T07:22:39.482 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-04-02T07:22:39.482 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-04-02T07:22:39.482 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-04-02T07:22:39.482 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-04-02T07:22:39.482 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-04-02T07:22:39.482 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-04-02T07:22:39.482 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-04-02T07:22:39.482 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-04-02T07:22:39.482 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-04-02T07:22:39.482 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-04-02T07:22:39.482 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-02T07:22:39.482 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-02T07:22:39.482 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-04-02T07:22:39.483 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-04-02T07:22:39.483 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-04-02T07:22:39.483 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-02T07:22:39.483 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-04-02T07:22:39.483 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-04-02T07:22:39.483 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-04-02T07:22:39.483 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-02T07:22:39.483 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-04-02T07:22:39.483 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-04-02T07:22:39.483 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-02T07:22:39.483 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-02T07:22:39.483 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-02T07:22:39.483 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-04-02T07:22:39.483 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-02T07:22:39.484 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-02T07:22:39.485 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-02T07:22:39.485 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-02T07:22:39.485 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-02T07:22:39.485 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-04-02T07:22:39.485 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-04-02T07:22:39.485 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-04-02T07:22:39.485 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-04-02T07:22:39.485 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-02T07:22:39.485 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-04-02T07:22:39.485 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-02T07:22:39.485 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-04-02T07:22:39.485 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-04-02T07:22:39.485 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-04-02T07:22:39.485 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-04-02T07:22:39.486 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-04-02T07:22:39.486 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-02T07:22:39.486 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-04-02T07:22:39.486 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-02T07:22:39.486 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-04-02T07:22:39.486 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-04-02T07:22:39.486 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-04-02T07:22:39.486 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-04-02T07:22:39.486 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-04-02T07:22:39.486 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-04-02T07:22:39.486 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-02T07:22:39.486 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-04-02T07:22:39.486 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-04-02T07:22:39.486 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-04-02T07:22:39.487 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-02T07:22:39.487 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-02T07:22:39.487 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-02T07:22:39.487 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-02T07:22:39.487 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-02T07:22:39.487 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-02T07:22:39.487 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-02T07:22:39.487 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-02T07:22:39.487 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-02T07:22:39.487 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-02T07:22:39.487 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-02T07:22:39.487 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-02T07:22:39.487 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-02T07:22:39.487 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-02T07:22:39.488 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-02T07:22:39.488 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-02T07:22:39.488 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-02T07:22:39.488 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-02T07:22:39.488 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-04-02T07:22:39.488 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-04-02T07:22:39.488 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-02T07:22:39.488 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-04-02T07:22:39.488 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-04-02T07:22:39.488 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-04-02T07:22:39.488 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-04-02T07:22:39.488 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-04-02T07:22:39.488 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-02T07:22:39.488 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-04-02T07:22:39.489 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-04-02T07:22:39.490 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-04-02T07:22:39.490 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-02T07:22:39.490 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-04-02T07:22:39.490 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-04-02T07:22:39.490 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-04-02T07:22:39.490 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-02T07:22:39.490 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-04-02T07:22:39.490 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-04-02T07:22:39.490 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-04-02T07:22:39.490 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-04-02T07:22:39.490 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-04-02T07:22:39.490 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-04-02T07:22:39.490 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-04-02T07:22:39.490 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-04-02T07:22:39.491 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-04-02T07:22:39.491 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-04-02T07:22:39.491 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-04-02T07:22:39.491 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-02T07:22:39.491 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-04-02T07:22:39.491 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-04-02T07:22:39.491 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-04-02T07:22:39.491 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-02T07:22:39.491 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-02T07:22:39.491 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-04-02T07:22:39.491 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-04-02T07:22:39.491 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-02T07:22:39.491 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-04-02T07:22:39.491 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-02T07:22:39.492 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-02T07:22:39.493 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-04-02T07:22:39.493 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-04-02T07:22:39.493 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-04-02T07:22:39.493 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-04-02T07:22:39.493 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-02T07:22:39.493 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-04-02T07:22:39.493 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-02T07:22:39.493 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-02T07:22:39.493 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-02T07:22:39.493 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-04-02T07:22:39.493 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-04-02T07:22:39.493 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-04-02T07:22:39.493 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-04-02T07:22:39.493 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-04-02T07:22:39.494 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-04-02T07:22:39.494 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-04-02T07:22:39.494 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-04-02T07:22:39.494 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-04-02T07:22:39.494 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-04-02T07:22:39.494 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-04-02T07:22:39.494 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-04-02T07:22:39.494 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-04-02T07:22:39.494 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-04-02T07:22:39.494 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-02T07:22:39.494 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-04-02T07:22:39.494 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-04-02T07:22:39.494 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-02T07:22:39.494 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-02T07:22:39.495 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-04-02T07:22:39.495 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-02T07:22:39.495 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-04-02T07:22:39.495 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-04-02T07:22:39.495 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-02T07:22:39.495 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-04-02T07:22:39.495 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-04-02T07:22:39.495 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-02T07:22:39.495 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-02T07:22:39.495 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-02T07:22:39.495 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-02T07:22:39.495 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-02T07:22:39.495 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-02T07:22:39.495 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-02T07:22:39.496 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-02T07:22:39.497 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-04-02T07:22:39.497 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-02T07:22:39.497 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-04-02T07:22:39.497 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-04-02T07:22:39.497 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-02T07:22:39.497 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-04-02T07:22:39.497 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-04-02T07:22:39.497 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-04-02T07:22:39.497 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-04-02T07:22:39.497 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-04-02T07:22:39.497 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-04-02T07:22:39.497 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-02T07:22:39.497 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-04-02T07:22:39.497 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-04-02T07:22:39.498 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-04-02T07:22:39.498 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-04-02T07:22:39.498 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-04-02T07:22:39.498 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-02T07:22:39.498 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-02T07:22:39.498 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-04-02T07:22:39.498 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-04-02T07:22:39.498 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-04-02T07:22:39.498 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-04-02T07:22:39.498 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-04-02T07:22:39.498 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-04-02T07:22:39.498 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-02T07:22:39.498 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-02T07:22:39.499 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-04-02T07:22:39.500 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-04-02T07:22:39.500 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-02T07:22:39.500 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-02T07:22:39.500 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-04-02T07:22:39.500 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-04-02T07:22:39.500 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-02T07:22:39.500 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-02T07:22:39.500 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-04-02T07:22:39.500 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-04-02T07:22:39.500 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-04-02T07:22:39.500 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-02T07:22:39.500 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-02T07:22:39.500 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-02T07:22:39.500 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-02T07:22:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-04-02T07:22:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-02T07:22:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-04-02T07:22:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-02T07:22:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-02T07:22:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-02T07:22:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-04-02T07:22:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-04-02T07:22:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-04-02T07:22:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-02T07:22:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-04-02T07:22:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-04-02T07:22:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-04-02T07:22:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-04-02T07:22:39.502 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-02T07:22:39.502 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-04-02T07:22:39.502 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-04-02T07:22:39.502 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-02T07:22:39.502 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-04-02T07:22:39.502 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-04-02T07:22:39.502 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-04-02T07:22:39.502 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-04-02T07:22:39.502 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-04-02T07:22:39.502 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-04-02T07:22:39.502 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-04-02T07:22:39.502 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-04-02T07:22:39.502 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-04-02T07:22:39.502 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-04-02T07:22:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-04-02T07:22:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-04-02T07:22:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-04-02T07:22:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-04-02T07:22:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-04-02T07:22:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-04-02T07:22:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-04-02T07:22:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-04-02T07:22:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-04-02T07:22:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-04-02T07:22:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-04-02T07:22:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-02T07:22:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-04-02T07:22:39.503 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-04-02T07:22:39.504 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-04-02T07:22:39.504 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-04-02T07:22:39.504 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-04-02T07:22:39.504 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-04-02T07:22:39.504 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-04-02T07:22:39.504 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-04-02T07:22:39.504 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-04-02T07:22:39.504 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-04-02T07:22:39.504 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-04-02T07:22:39.504 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-04-02T07:22:39.504 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-04-02T07:22:39.504 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-04-02T07:22:39.504 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-04-02T07:22:39.504 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-02T07:22:39.505 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-04-02T07:22:39.506 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-02T07:22:39.506 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-04-02T07:22:39.506 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-04-02T07:22:39.506 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-04-02T07:22:39.506 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-04-02T07:22:39.506 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-04-02T07:22:39.506 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-04-02T07:22:39.506 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-04-02T07:22:39.506 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-04-02T07:22:39.506 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-04-02T07:22:39.506 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-04-02T07:22:39.506 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-04-02T07:22:39.506 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-04-02T07:22:39.506 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-04-02T07:22:39.507 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-04-02T07:22:39.508 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-04-02T07:22:39.508 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-04-02T07:22:39.508 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-04-02T07:22:39.508 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-04-02T07:22:39.508 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-04-02T07:22:39.508 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-04-02T07:22:39.508 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-04-02T07:22:39.508 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-04-02T07:22:39.508 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-04-02T07:22:39.508 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-04-02T07:22:39.508 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-04-02T07:22:39.508 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-04-02T07:22:39.508 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-04-02T07:22:39.508 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-04-02T07:22:39.509 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-04-02T07:22:39.510 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-04-02T07:22:39.510 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-04-02T07:22:39.510 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-02T07:22:39.510 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-04-02T07:22:39.510 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-04-02T07:22:39.510 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-04-02T07:22:39.510 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-02T07:22:39.510 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-04-02T07:22:39.510 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-02T07:22:39.510 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-04-02T07:22:39.510 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-02T07:22:39.510 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-04-02T07:22:39.510 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-02T07:22:39.510 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-02T07:22:39.511 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-02T07:22:39.511 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-04-02T07:22:39.511 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-04-02T07:22:39.511 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-04-02T07:22:39.511 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-04-02T07:22:39.511 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-04-02T07:22:39.511 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-04-02T07:22:39.511 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-04-02T07:22:39.511 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-04-02T07:22:39.511 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-04-02T07:22:39.511 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-04-02T07:22:39.511 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-04-02T07:22:39.511 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-04-02T07:22:39.511 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-04-02T07:22:39.512 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-04-02T07:22:39.513 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-04-02T07:22:39.514 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-04-02T07:22:39.514 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-04-02T07:22:39.514 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-04-02T07:22:39.514 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-04-02T07:22:39.514 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-02T07:22:39.514 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-04-02T07:22:39.514 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-04-02T07:22:39.514 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-04-02T07:22:39.514 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-04-02T07:22:39.514 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-04-02T07:22:39.514 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-04-02T07:22:39.514 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-04-02T07:22:39.514 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-04-02T07:22:39.514 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-04-02T07:22:39.515 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-04-02T07:22:39.516 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-04-02T07:22:39.516 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-04-02T07:22:39.516 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-04-02T07:22:39.516 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-04-02T07:22:39.516 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-04-02T07:22:39.516 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-04-02T07:22:39.516 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-04-02T07:22:39.516 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-04-02T07:22:39.516 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-04-02T07:22:39.516 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-04-02T07:22:39.516 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-04-02T07:22:39.516 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-04-02T07:22:39.516 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-02T07:22:39.516 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-02T07:22:39.517 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-02T07:22:39.517 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-02T07:22:39.517 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-02T07:22:39.517 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-02T07:22:39.517 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-02T07:22:39.517 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-02T07:22:39.517 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-02T07:22:39.517 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-02T07:22:39.517 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-02T07:22:39.517 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-02T07:22:39.517 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-02T07:22:39.517 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-02T07:22:39.517 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-02T07:22:39.517 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-02T07:22:39.518 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-02T07:22:39.518 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-02T07:22:39.518 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-02T07:22:39.518 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-02T07:22:39.518 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-04-02T07:22:39.518 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-02T07:22:39.518 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-02T07:22:39.518 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-02T07:22:39.518 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-04-02T07:22:39.518 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-04-02T07:22:39.518 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-04-02T07:22:39.518 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-04-02T07:22:39.518 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-04-02T07:22:39.518 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-04-02T07:22:39.519 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-04-02T07:22:39.519 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-04-02T07:22:39.519 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-04-02T07:22:39.519 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-04-02T07:22:39.519 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-04-02T07:22:39.519 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-04-02T07:22:39.519 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-04-02T07:22:39.519 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-04-02T07:22:39.519 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-04-02T07:22:39.519 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-04-02T07:22:39.519 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-04-02T07:22:39.519 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-04-02T07:22:39.519 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-04-02T07:22:39.519 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-04-02T07:22:39.520 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-04-02T07:22:39.521 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-02T07:22:39.521 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-02T07:22:39.521 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-02T07:22:39.521 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-02T07:22:39.521 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-02T07:22:39.521 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-02T07:22:39.521 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-02T07:22:39.521 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-02T07:22:39.521 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-02T07:22:39.521 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-02T07:22:39.521 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-02T07:22:39.521 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-02T07:22:39.521 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-02T07:22:39.521 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-02T07:22:39.522 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-02T07:22:39.522 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-02T07:22:39.522 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-02T07:22:39.522 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-02T07:22:39.522 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-02T07:22:39.522 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-02T07:22:39.522 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-02T07:22:39.522 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-04-02T07:22:39.522 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-04-02T07:22:39.522 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-04-02T07:22:39.522 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-04-02T07:22:39.522 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-04-02T07:22:39.522 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-04-02T07:22:39.522 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-04-02T07:22:39.523 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-04-02T07:22:39.523 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-04-02T07:22:39.523 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-04-02T07:22:39.523 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-04-02T07:22:39.523 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-04-02T07:22:39.523 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-04-02T07:22:39.523 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-04-02T07:22:39.523 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-04-02T07:22:39.523 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-04-02T07:22:39.523 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-02T07:22:39.523 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-04-02T07:22:39.523 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-04-02T07:22:39.523 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-04-02T07:22:39.523 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-04-02T07:22:39.524 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-04-02T07:22:39.524 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-02T07:22:39.524 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-04-02T07:22:39.524 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-04-02T07:22:39.524 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-04-02T07:22:39.524 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-02T07:22:39.524 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-02T07:22:39.524 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-02T07:22:39.524 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-02T07:22:39.524 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-02T07:22:39.524 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-02T07:22:39.524 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-02T07:22:39.524 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-02T07:22:39.524 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-02T07:22:39.525 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-02T07:22:39.525 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-02T07:22:39.525 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-02T07:22:39.525 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-02T07:22:39.525 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-02T07:22:39.525 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-02T07:22:39.525 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-02T07:22:39.525 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-02T07:22:39.525 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-02T07:22:39.525 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-02T07:22:39.525 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-02T07:22:39.525 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-02T07:22:39.525 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-02T07:22:39.525 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-02T07:22:39.526 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-02T07:22:39.526 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-04-02T07:22:39.526 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-04-02T07:22:39.526 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-02T07:22:39.526 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-04-02T07:22:39.526 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-04-02T07:22:39.526 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-04-02T07:22:39.526 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-04-02T07:22:39.526 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-02T07:22:39.526 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-04-02T07:22:39.526 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-04-02T07:22:39.526 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-04-02T07:22:39.526 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-04-02T07:22:39.526 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-02T07:22:39.527 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-02T07:22:39.528 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-02T07:22:39.528 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-02T07:22:39.528 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-02T07:22:39.528 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-02T07:22:39.528 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-02T07:22:39.528 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-02T07:22:39.528 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-02T07:22:39.528 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-02T07:22:39.528 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-02T07:22:39.528 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-04-02T07:22:39.528 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-02T07:22:39.528 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-02T07:22:39.528 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-02T07:22:39.528 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-02T07:22:39.529 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-02T07:22:39.529 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-02T07:22:39.529 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-02T07:22:39.529 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-02T07:22:39.529 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-02T07:22:39.529 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-02T07:22:39.529 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-02T07:22:39.529 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-02T07:22:39.529 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-02T07:22:39.529 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-02T07:22:39.529 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-04-02T07:22:39.529 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-04-02T07:22:39.529 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-02T07:22:39.529 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-04-02T07:22:39.530 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-04-02T07:22:39.531 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-02T07:22:39.531 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-02T07:22:39.531 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-04-02T07:22:39.531 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-04-02T07:22:39.531 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-02T07:22:39.531 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-02T07:22:39.531 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-04-02T07:22:39.531 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-04-02T07:22:39.531 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-02T07:22:39.531 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-04-02T07:22:39.531 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-02T07:22:39.531 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-02T07:22:39.531 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-04-02T07:22:39.531 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-04-02T07:22:39.532 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-04-02T07:22:39.533 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-04-02T07:22:39.533 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-04-02T07:22:39.533 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-04-02T07:22:39.533 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-02T07:22:39.533 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-02T07:22:39.533 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-02T07:22:39.533 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-04-02T07:22:39.533 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-04-02T07:22:39.533 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-04-02T07:22:39.533 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-04-02T07:22:39.533 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-02T07:22:39.533 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-04-02T07:22:39.533 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-04-02T07:22:39.533 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-02T07:22:39.534 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-04-02T07:22:39.534 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-04-02T07:22:39.534 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-02T07:22:39.534 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-04-02T07:22:39.534 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-02T07:22:39.534 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-04-02T07:22:39.534 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-02T07:22:39.534 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-02T07:22:39.534 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-04-02T07:22:39.534 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-04-02T07:22:39.534 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-04-02T07:22:39.534 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-04-02T07:22:39.534 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-04-02T07:22:39.534 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-04-02T07:22:39.535 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-04-02T07:22:39.536 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-04-02T07:22:39.537 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-04-02T07:22:39.537 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-04-02T07:22:39.537 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-04-02T07:22:39.537 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-04-02T07:22:39.537 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-04-02T07:22:39.537 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-04-02T07:22:39.537 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-04-02T07:22:39.537 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-04-02T07:22:39.537 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-04-02T07:22:39.537 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-04-02T07:22:39.537 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-04-02T07:22:39.537 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-04-02T07:22:39.537 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-04-02T07:22:39.537 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-04-02T07:22:39.538 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-04-02T07:22:39.538 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-04-02T07:22:39.538 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-04-02T07:22:39.538 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-04-02T07:22:39.538 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-04-02T07:22:39.538 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-04-02T07:22:39.538 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-04-02T07:22:39.538 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-04-02T07:22:39.538 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-04-02T07:22:39.538 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-04-02T07:22:39.538 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-04-02T07:22:39.538 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-04-02T07:22:39.538 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-04-02T07:22:39.538 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-04-02T07:22:39.539 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-04-02T07:22:39.539 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-04-02T07:22:39.539 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-04-02T07:22:39.539 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-04-02T07:22:39.539 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-04-02T07:22:39.539 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-04-02T07:22:39.539 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-04-02T07:22:39.539 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-04-02T07:22:39.539 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-04-02T07:22:39.539 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-04-02T07:22:39.539 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-04-02T07:22:39.539 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-04-02T07:22:39.539 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-04-02T07:22:39.540 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-04-02T07:22:39.540 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-04-02T07:22:39.540 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-04-02T07:22:39.540 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-04-02T07:22:39.540 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-04-02T07:22:39.540 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-04-02T07:22:39.540 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-04-02T07:22:39.540 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-04-02T07:22:39.540 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-04-02T07:22:39.540 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-04-02T07:22:39.540 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-04-02T07:22:39.540 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-04-02T07:22:39.540 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-04-02T07:22:39.540 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-04-02T07:22:39.541 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-04-02T07:22:39.542 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-04-02T07:22:39.542 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-04-02T07:22:39.542 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-04-02T07:22:39.542 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-04-02T07:22:39.542 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-04-02T07:22:39.542 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-04-02T07:22:39.542 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-04-02T07:22:39.542 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-04-02T07:22:39.542 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-04-02T07:22:39.542 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-02T07:22:39.542 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-02T07:22:39.542 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-04-02T07:22:39.542 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-04-02T07:22:39.542 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-04-02T07:22:39.543 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-04-02T07:22:39.544 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-04-02T07:22:39.545 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-04-02T07:22:39.545 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-04-02T07:22:39.545 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-04-02T07:22:39.545 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-04-02T07:22:39.545 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-04-02T07:22:39.548 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-04-02T07:22:39.549 DEBUG:teuthology.orchestra.run.smithi113:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-02T07:22:39.573 INFO:teuthology.orchestra.run.smithi113.stdout:postinstall scriptlet (using /bin/sh): 2024-04-02T07:22:39.573 INFO:teuthology.orchestra.run.smithi113.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-02T07:22:39.573 INFO:teuthology.orchestra.run.smithi113.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-04-02T07:22:39.573 INFO:teuthology.orchestra.run.smithi113.stdout:fi 2024-04-02T07:22:39.573 INFO:teuthology.orchestra.run.smithi113.stdout:for file in vmlinuz System.map config; do 2024-04-02T07:22:39.573 INFO:teuthology.orchestra.run.smithi113.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-04-02T07:22:39.574 INFO:teuthology.orchestra.run.smithi113.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-04-02T07:22:39.574 INFO:teuthology.orchestra.run.smithi113.stdout: fi 2024-04-02T07:22:39.574 INFO:teuthology.orchestra.run.smithi113.stdout:done 2024-04-02T07:22:39.574 INFO:teuthology.orchestra.run.smithi113.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-02T07:22:39.574 INFO:teuthology.orchestra.run.smithi113.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-04-02T07:22:39.574 INFO:teuthology.orchestra.run.smithi113.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-04-02T07:22:39.574 INFO:teuthology.orchestra.run.smithi113.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-04-02T07:22:39.574 INFO:teuthology.orchestra.run.smithi113.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-04-02T07:22:39.574 INFO:teuthology.orchestra.run.smithi113.stdout:fi 2024-04-02T07:22:39.574 INFO:teuthology.orchestra.run.smithi113.stdout:postuninstall scriptlet (using /bin/sh): 2024-04-02T07:22:39.574 INFO:teuthology.orchestra.run.smithi113.stdout:if [ -x /sbin/update-bootloader ]; then 2024-04-02T07:22:39.574 INFO:teuthology.orchestra.run.smithi113.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-04-02T07:22:39.574 INFO:teuthology.orchestra.run.smithi113.stdout:fi 2024-04-02T07:22:39.575 DEBUG:teuthology.orchestra.run.smithi113:> sudo rpm -qi grub2-tools 2024-04-02T07:22:39.650 INFO:teuthology.orchestra.run.smithi113.stdout:Name : grub2-tools 2024-04-02T07:22:39.650 INFO:teuthology.orchestra.run.smithi113.stdout:Epoch : 1 2024-04-02T07:22:39.650 INFO:teuthology.orchestra.run.smithi113.stdout:Version : 2.02 2024-04-02T07:22:39.650 INFO:teuthology.orchestra.run.smithi113.stdout:Release : 129.el8 2024-04-02T07:22:39.650 INFO:teuthology.orchestra.run.smithi113.stdout:Architecture: x86_64 2024-04-02T07:22:39.650 INFO:teuthology.orchestra.run.smithi113.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-02T07:22:39.650 INFO:teuthology.orchestra.run.smithi113.stdout:Group : System Environment/Base 2024-04-02T07:22:39.650 INFO:teuthology.orchestra.run.smithi113.stdout:Size : 9265586 2024-04-02T07:22:39.650 INFO:teuthology.orchestra.run.smithi113.stdout:License : GPLv3+ 2024-04-02T07:22:39.651 INFO:teuthology.orchestra.run.smithi113.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-02T07:22:39.651 INFO:teuthology.orchestra.run.smithi113.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-02T07:22:39.651 INFO:teuthology.orchestra.run.smithi113.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-02T07:22:39.651 INFO:teuthology.orchestra.run.smithi113.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-02T07:22:39.651 INFO:teuthology.orchestra.run.smithi113.stdout:Relocations : (not relocatable) 2024-04-02T07:22:39.651 INFO:teuthology.orchestra.run.smithi113.stdout:Packager : CentOS BuildSystem 2024-04-02T07:22:39.651 INFO:teuthology.orchestra.run.smithi113.stdout:Vendor : CentOS 2024-04-02T07:22:39.651 INFO:teuthology.orchestra.run.smithi113.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-02T07:22:39.651 INFO:teuthology.orchestra.run.smithi113.stdout:Summary : Support tools for GRUB. 2024-04-02T07:22:39.651 INFO:teuthology.orchestra.run.smithi113.stdout:Description : 2024-04-02T07:22:39.651 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:22:39.651 INFO:teuthology.orchestra.run.smithi113.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-02T07:22:39.651 INFO:teuthology.orchestra.run.smithi113.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-02T07:22:39.651 INFO:teuthology.orchestra.run.smithi113.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-02T07:22:39.652 INFO:teuthology.orchestra.run.smithi113.stdout:hardware devices. 2024-04-02T07:22:39.652 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:22:39.652 INFO:teuthology.orchestra.run.smithi113.stdout:This subpackage provides tools for support of all platforms. 2024-04-02T07:22:39.653 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-02T07:22:39.653 INFO:teuthology.task.kernel:Updating grub on smithi113 to boot 6.8.0-g1df0c7e8a235 2024-04-02T07:22:39.653 DEBUG:teuthology.orchestra.run.smithi113:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-02T07:22:43.329 INFO:teuthology.orchestra.run.smithi113.stderr:Generating grub configuration file ... 2024-04-02T07:22:45.382 INFO:teuthology.orchestra.run.smithi113.stderr:done 2024-04-02T07:22:45.384 DEBUG:teuthology.orchestra.run.smithi113:> mktemp 2024-04-02T07:22:45.412 INFO:teuthology.orchestra.run.smithi113.stdout:/tmp/tmp.QrU0OcwEAV 2024-04-02T07:22:45.412 DEBUG:teuthology.orchestra.run.smithi113:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QrU0OcwEAV 2024-04-02T07:22:45.501 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0666 /tmp/tmp.QrU0OcwEAV 2024-04-02T07:22:45.642 DEBUG:teuthology.orchestra.remote:smithi113:/tmp/tmp.QrU0OcwEAV is 6KB 2024-04-02T07:22:45.689 DEBUG:teuthology.orchestra.run.smithi113:> rm -fr /tmp/tmp.QrU0OcwEAV 2024-04-02T07:22:45.707 DEBUG:teuthology.orchestra.run.smithi113:> sudo /bin/ls /boot/loader/entries || true 2024-04-02T07:22:45.784 INFO:teuthology.orchestra.run.smithi113.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-02T07:22:45.785 INFO:teuthology.orchestra.run.smithi113.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-02T07:22:45.785 INFO:teuthology.orchestra.run.smithi113.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-02T07:22:45.785 INFO:teuthology.orchestra.run.smithi113.stdout:a1b29d6c289747bdb22930598d39874a-6.8.0-g1df0c7e8a235.conf 2024-04-02T07:22:45.786 DEBUG:teuthology.orchestra.run.smithi113:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-6.8.0-g1df0c7e8a235 2024-04-02T07:22:45.998 DEBUG:teuthology.orchestra.run.smithi113:> sudo shutdown -r now 2024-04-02T07:22:53.090 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2024-04-02T07:23:02.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-04-02T07:23:02.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T07:23:05.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2024-04-02T07:23:16.010 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-04-02T07:23:16.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-02T07:23:16.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T07:23:17.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-04-02T07:23:17.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T07:23:34.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-04-02T07:23:43.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-02T07:23:43.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T07:23:46.626 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-04-02T07:23:48.944 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-04-02T07:23:49.330 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-04-02T07:23:49.330 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g1df0c7e8a235"... 2024-04-02T07:23:49.330 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-04-02T07:23:49.373 INFO:teuthology.orchestra.run.smithi089.stdout:6.8.0-g1df0c7e8a235 2024-04-02T07:23:49.373 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-04-02T07:23:49.374 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-02T07:23:49.374 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-02T07:23:50.374 DEBUG:teuthology.parallel:result is None 2024-04-02T07:23:58.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-02T07:23:58.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T07:24:01.694 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-04-02T07:24:16.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-02T07:24:16.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T07:24:35.230 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-04-02T07:24:53.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-02T07:24:53.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T07:24:53.611 DEBUG:teuthology.orchestra.run.smithi113:> true 2024-04-02T07:24:54.010 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-02T07:24:54.010 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.8.0-g1df0c7e8a235"... 2024-04-02T07:24:54.010 DEBUG:teuthology.orchestra.run.smithi113:> uname -r 2024-04-02T07:24:54.053 INFO:teuthology.orchestra.run.smithi113.stdout:6.8.0-g1df0c7e8a235 2024-04-02T07:24:54.053 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-04-02T07:24:54.053 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-02T07:24:54.054 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-02T07:24:55.055 DEBUG:teuthology.parallel:result is None 2024-04-02T07:24:55.055 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-02T07:24:55.065 INFO:teuthology.task.internal:Creating test directory... 2024-04-02T07:24:55.065 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-02T07:24:55.071 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-02T07:24:55.118 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-02T07:24:55.125 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-02T07:24:55.143 INFO:teuthology.task.internal:Creating archive directory... 2024-04-02T07:24:55.143 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-02T07:24:55.147 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-02T07:24:55.219 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-02T07:24:55.227 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-02T07:24:55.228 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-02T07:24:55.231 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-02T07:24:55.340 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T07:24:55.347 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T07:24:55.369 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T07:24:55.379 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T07:24:55.382 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-02T07:24:55.393 INFO:teuthology.task.internal:Configuring sudo... 2024-04-02T07:24:55.394 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-02T07:24:55.397 DEBUG:teuthology.orchestra.run.smithi113:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-02T07:24:55.465 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-02T07:24:55.474 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-02T07:24:55.474 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-02T07:24:55.478 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-02T07:24:55.522 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T07:24:55.631 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T07:24:55.706 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T07:24:55.776 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T07:24:55.841 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:24:55.841 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-02T07:24:55.908 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T07:24:55.983 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T07:24:56.051 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T07:24:56.119 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T07:24:56.184 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:24:56.185 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-02T07:24:56.250 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2024-04-02T07:24:56.254 DEBUG:teuthology.orchestra.run.smithi113:> sudo service rsyslog restart 2024-04-02T07:24:56.363 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T07:24:56.394 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T07:24:56.871 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-02T07:24:56.879 INFO:teuthology.task.internal:Starting timer... 2024-04-02T07:24:56.880 INFO:teuthology.run_tasks:Running task pcp... 2024-04-02T07:24:56.891 INFO:teuthology.run_tasks:Running task selinux... 2024-04-02T07:24:56.901 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-02T07:24:56.901 DEBUG:teuthology.orchestra.run.smithi089:> sudo service auditd rotate 2024-04-02T07:24:57.018 INFO:teuthology.orchestra.run.smithi089.stdout:Rotating logs: 2024-04-02T07:24:57.021 DEBUG:teuthology.orchestra.run.smithi113:> sudo service auditd rotate 2024-04-02T07:24:57.141 INFO:teuthology.orchestra.run.smithi113.stdout:Rotating logs: 2024-04-02T07:24:57.144 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-02T07:24:57.144 DEBUG:teuthology.orchestra.run.smithi089:> /usr/sbin/getenforce 2024-04-02T07:24:57.190 INFO:teuthology.orchestra.run.smithi089.stdout:Permissive 2024-04-02T07:24:57.191 DEBUG:teuthology.orchestra.run.smithi113:> /usr/sbin/getenforce 2024-04-02T07:24:57.241 INFO:teuthology.orchestra.run.smithi113.stdout:Permissive 2024-04-02T07:24:57.242 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi089.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi113.front.sepia.ceph.com': 'permissive'} 2024-04-02T07:24:57.242 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-02T07:24:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T07:24:57.315 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-02T07:24:57.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T07:24:57.382 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-02T07:24:57.382 DEBUG:teuthology.orchestra.run.smithi089:> sudo /usr/sbin/setenforce permissive 2024-04-02T07:24:57.449 DEBUG:teuthology.orchestra.run.smithi113:> sudo /usr/sbin/setenforce permissive 2024-04-02T07:24:57.515 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-02T07:24:57.526 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-02T07:24:57.542 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-02T07:24:57.544 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi089.front.sepia.ceph.com,smithi113.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-02T07:32:03.275 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi089.front.sepia.ceph.com'), Remote(name='ubuntu@smithi113.front.sepia.ceph.com')] 2024-04-02T07:32:03.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-04-02T07:32:03.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T07:32:03.368 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-04-02T07:32:03.482 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-04-02T07:32:03.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-02T07:32:03.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T07:32:03.574 DEBUG:teuthology.orchestra.run.smithi113:> true 2024-04-02T07:32:03.687 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-02T07:32:03.687 INFO:teuthology.run_tasks:Running task clock... 2024-04-02T07:32:03.697 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-02T07:32:03.697 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-02T07:32:03.698 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T07:32:03.701 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-02T07:32:03.702 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T07:32:03.769 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-02T07:32:03.799 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-02T07:32:03.800 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-02T07:32:03.831 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-02T07:32:03.889 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: ntpd: command not found 2024-04-02T07:32:03.916 INFO:teuthology.orchestra.run.smithi089.stdout:506 Cannot talk to daemon 2024-04-02T07:32:03.917 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: ntpd: command not found 2024-04-02T07:32:03.942 INFO:teuthology.orchestra.run.smithi113.stdout:506 Cannot talk to daemon 2024-04-02T07:32:03.946 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-02T07:32:03.972 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-02T07:32:03.977 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-02T07:32:04.002 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-02T07:32:04.061 INFO:teuthology.orchestra.run.smithi089.stderr:bash: ntpq: command not found 2024-04-02T07:32:04.068 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T07:32:04.069 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-04-02T07:32:04.069 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T07:32:04.069 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T07:32:04.069 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T07:32:04.069 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T07:32:04.083 INFO:teuthology.orchestra.run.smithi113.stderr:bash: ntpq: command not found 2024-04-02T07:32:04.090 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T07:32:04.090 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2024-04-02T07:32:04.090 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T07:32:04.090 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T07:32:04.090 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T07:32:04.090 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T07:32:04.091 INFO:teuthology.run_tasks:Running task install... 2024-04-02T07:32:04.101 DEBUG:teuthology.task.install:project ceph 2024-04-02T07:32:04.101 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2'}} 2024-04-02T07:32:04.101 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2'} 2024-04-02T07:32:04.101 INFO:teuthology.task.install:Using flavor: default 2024-04-02T07:32:04.107 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-02T07:32:04.107 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-04-02T07:32:04.107 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2', 'tag': None, 'wait_for_package': False} 2024-04-02T07:32:04.108 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d0e392d9aee635f2b65768c460c7a151ead7a5e2 2024-04-02T07:32:04.109 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2', 'tag': None, 'wait_for_package': False} 2024-04-02T07:32:04.110 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d0e392d9aee635f2b65768c460c7a151ead7a5e2 2024-04-02T07:32:04.264 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/ 2024-04-02T07:32:04.264 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1015.gd0e392d9 2024-04-02T07:32:04.268 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/ 2024-04-02T07:32:04.268 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1015.gd0e392d9 2024-04-02T07:32:04.378 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-02T07:32:04.378 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:32:04.378 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-02T07:32:04.437 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240326.105515-reef/d0e392d9aee635f2b65768c460c7a151ead7a5e2/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-02T07:32:04.438 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:32:04.438 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-02T07:32:04.445 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-02T07:32:04.446 DEBUG:teuthology.orchestra.run.smithi113:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d0e392d9aee635f2b65768c460c7a151ead7a5e2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-02T07:32:04.503 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-02T07:32:04.503 DEBUG:teuthology.orchestra.run.smithi089:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d0e392d9aee635f2b65768c460c7a151ead7a5e2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-02T07:32:04.553 DEBUG:teuthology.orchestra.run.smithi113:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-02T07:32:04.610 DEBUG:teuthology.orchestra.run.smithi089:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-02T07:32:04.646 DEBUG:teuthology.orchestra.run.smithi113:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-02T07:32:04.700 DEBUG:teuthology.orchestra.run.smithi089:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-02T07:32:04.711 INFO:teuthology.orchestra.run.smithi113.stdout:check_obsoletes = 1 2024-04-02T07:32:04.715 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum clean all 2024-04-02T07:32:04.765 INFO:teuthology.orchestra.run.smithi089.stdout:check_obsoletes = 1 2024-04-02T07:32:04.769 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum clean all 2024-04-02T07:32:05.056 INFO:teuthology.orchestra.run.smithi113.stdout:76 files removed 2024-04-02T07:32:05.092 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-02T07:32:05.105 INFO:teuthology.orchestra.run.smithi089.stdout:76 files removed 2024-04-02T07:32:05.141 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-02T07:32:06.045 INFO:teuthology.orchestra.run.smithi113.stdout:ceph packages for x86_64 191 kB/s | 80 kB 00:00 2024-04-02T07:32:06.133 INFO:teuthology.orchestra.run.smithi089.stdout:ceph packages for x86_64 177 kB/s | 80 kB 00:00 2024-04-02T07:32:06.413 INFO:teuthology.orchestra.run.smithi113.stdout:ceph noarch packages 30 kB/s | 9.9 kB 00:00 2024-04-02T07:32:06.551 INFO:teuthology.orchestra.run.smithi089.stdout:ceph noarch packages 26 kB/s | 9.9 kB 00:00 2024-04-02T07:32:06.847 INFO:teuthology.orchestra.run.smithi113.stdout:ceph source packages 3.7 kB/s | 1.5 kB 00:00 2024-04-02T07:32:06.877 INFO:teuthology.orchestra.run.smithi089.stdout:ceph source packages 5.0 kB/s | 1.5 kB 00:00 2024-04-02T07:32:10.213 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - AppStream 8.3 MB/s | 28 MB 00:03 2024-04-02T07:32:10.485 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - AppStream 7.8 MB/s | 28 MB 00:03 2024-04-02T07:32:16.730 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - BaseOS 9.2 MB/s | 10 MB 00:01 2024-04-02T07:32:16.844 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - BaseOS 9.9 MB/s | 10 MB 00:01 2024-04-02T07:32:18.744 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - Extras 76 kB/s | 18 kB 00:00 2024-04-02T07:32:18.780 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - Extras 49 kB/s | 18 kB 00:00 2024-04-02T07:32:19.145 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 7.5 kB 00:00 2024-04-02T07:32:19.173 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - Extras common packages 27 kB/s | 7.5 kB 00:00 2024-04-02T07:32:21.180 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 8 - PowerTools 2.5 MB/s | 4.9 MB 00:01 2024-04-02T07:32:21.274 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - PowerTools 2.4 MB/s | 4.9 MB 00:02 2024-04-02T07:32:22.331 INFO:teuthology.orchestra.run.smithi089.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-04-02T07:32:22.342 INFO:teuthology.orchestra.run.smithi113.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-04-02T07:32:23.370 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 18 MB/s | 16 MB 00:00 2024-04-02T07:32:23.932 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-04-02T07:32:27.952 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 103 kB/s | 24 kB 00:00 2024-04-02T07:32:28.316 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 403 kB/s | 24 kB 00:00 2024-04-02T07:32:31.620 INFO:teuthology.orchestra.run.smithi113.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-02T07:32:31.620 INFO:teuthology.orchestra.run.smithi113.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-02T07:32:31.630 INFO:teuthology.orchestra.run.smithi113.stdout:Package NetworkManager-1:1.40.16-4.el8.x86_64 is already installed. 2024-04-02T07:32:31.630 INFO:teuthology.orchestra.run.smithi113.stdout:Package iproute-6.2.0-1.el8.x86_64 is already installed. 2024-04-02T07:32:31.631 INFO:teuthology.orchestra.run.smithi113.stdout:Package util-linux-2.32.1-41.el8.x86_64 is already installed. 2024-04-02T07:32:31.632 INFO:teuthology.orchestra.run.smithi113.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-04-02T07:32:31.634 INFO:teuthology.orchestra.run.smithi113.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-04-02T07:32:31.636 INFO:teuthology.orchestra.run.smithi113.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-04-02T07:32:31.637 INFO:teuthology.orchestra.run.smithi113.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-04-02T07:32:31.720 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-02T07:32:31.735 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-04-02T07:32:31.735 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-04-02T07:32:31.735 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-04-02T07:32:31.735 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-02T07:32:31.735 INFO:teuthology.orchestra.run.smithi113.stdout: bison x86_64 3.0.4-10.el8 appstream 688 k 2024-04-02T07:32:31.735 INFO:teuthology.orchestra.run.smithi113.stdout: ceph x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 6.4 k 2024-04-02T07:32:31.735 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 5.5 M 2024-04-02T07:32:31.736 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 854 k 2024-04-02T07:32:31.736 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 166 k 2024-04-02T07:32:31.736 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 1.6 M 2024-04-02T07:32:31.736 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 138 k 2024-04-02T07:32:31.736 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 1.8 M 2024-04-02T07:32:31.736 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 7.4 M 2024-04-02T07:32:31.736 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 49 k 2024-04-02T07:32:31.736 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 13 M 2024-04-02T07:32:31.736 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 47 M 2024-04-02T07:32:31.736 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-volume noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 263 k 2024-04-02T07:32:31.736 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 173 k 2024-04-02T07:32:31.736 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-mirror x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 261 k 2024-04-02T07:32:31.736 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-top noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 18 k 2024-04-02T07:32:31.736 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-devel x86_64 0.190-2.el8 baseos 62 k 2024-04-02T07:32:31.737 INFO:teuthology.orchestra.run.smithi113.stdout: flex x86_64 2.6.1-9.el8 appstream 320 k 2024-04-02T07:32:31.737 INFO:teuthology.orchestra.run.smithi113.stdout: libaio-devel x86_64 0.3.112-1.el8 baseos 19 k 2024-04-02T07:32:31.737 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 31 k 2024-04-02T07:32:31.737 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 778 k 2024-04-02T07:32:31.737 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 139 k 2024-04-02T07:32:31.737 INFO:teuthology.orchestra.run.smithi113.stdout: libtool x86_64 2.4.6-25.el8 appstream 709 k 2024-04-02T07:32:31.737 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-devel x86_64 2.32.1-46.el8 baseos 100 k 2024-04-02T07:32:31.737 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-devel x86_64 1:1.1.1k-12.el8 baseos 3.2 M 2024-04-02T07:32:31.737 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql x86_64 10.23-2.module_el8+572+929c87ac appstream 1.6 M 2024-04-02T07:32:31.737 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-contrib x86_64 10.23-2.module_el8+572+929c87ac appstream 901 k 2024-04-02T07:32:31.737 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-server x86_64 10.23-2.module_el8+572+929c87ac appstream 5.1 M 2024-04-02T07:32:31.737 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 197 k 2024-04-02T07:32:31.737 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 371 k 2024-04-02T07:32:31.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 364 k 2024-04-02T07:32:31.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 106 k 2024-04-02T07:32:31.738 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 95 k 2024-04-02T07:32:31.738 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 3.5 M 2024-04-02T07:32:31.738 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 181 k 2024-04-02T07:32:31.738 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 baseos 74 k 2024-04-02T07:32:31.738 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrading: 2024-04-02T07:32:31.738 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager x86_64 1:1.40.16-15.el8 baseos 2.3 M 2024-04-02T07:32:31.738 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-libnm x86_64 1:1.40.16-15.el8 baseos 1.9 M 2024-04-02T07:32:31.738 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-team x86_64 1:1.40.16-15.el8 baseos 161 k 2024-04-02T07:32:31.738 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-tui x86_64 1:1.40.16-15.el8 baseos 356 k 2024-04-02T07:32:31.738 INFO:teuthology.orchestra.run.smithi113.stdout: acl x86_64 2.2.53-3.el8 baseos 83 k 2024-04-02T07:32:31.738 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-debuginfod-client x86_64 0.190-2.el8 baseos 77 k 2024-04-02T07:32:31.739 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf x86_64 0.190-2.el8 baseos 233 k 2024-04-02T07:32:31.739 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libs x86_64 0.190-2.el8 baseos 305 k 2024-04-02T07:32:31.739 INFO:teuthology.orchestra.run.smithi113.stdout: iproute x86_64 6.2.0-5.el8 baseos 854 k 2024-04-02T07:32:31.739 INFO:teuthology.orchestra.run.smithi113.stdout: libacl x86_64 2.2.53-3.el8 baseos 35 k 2024-04-02T07:32:31.739 INFO:teuthology.orchestra.run.smithi113.stdout: libacl-devel x86_64 2.2.53-3.el8 baseos 85 k 2024-04-02T07:32:31.739 INFO:teuthology.orchestra.run.smithi113.stdout: libblkid x86_64 2.32.1-46.el8 baseos 221 k 2024-04-02T07:32:31.739 INFO:teuthology.orchestra.run.smithi113.stdout: libfdisk x86_64 2.32.1-46.el8 baseos 254 k 2024-04-02T07:32:31.739 INFO:teuthology.orchestra.run.smithi113.stdout: libmount x86_64 2.32.1-46.el8 baseos 237 k 2024-04-02T07:32:31.739 INFO:teuthology.orchestra.run.smithi113.stdout: librados2 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 3.7 M 2024-04-02T07:32:31.739 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 3.6 M 2024-04-02T07:32:31.739 INFO:teuthology.orchestra.run.smithi113.stdout: libsmartcols x86_64 2.32.1-46.el8 baseos 180 k 2024-04-02T07:32:31.739 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid x86_64 2.32.1-46.el8 baseos 99 k 2024-04-02T07:32:31.739 INFO:teuthology.orchestra.run.smithi113.stdout: openssl x86_64 1:1.1.1k-12.el8 baseos 738 k 2024-04-02T07:32:31.740 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-libs x86_64 1:1.1.1k-12.el8 baseos 1.5 M 2024-04-02T07:32:31.740 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux x86_64 2.32.1-46.el8 baseos 2.6 M 2024-04-02T07:32:31.740 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-user x86_64 2.32.1-46.el8 baseos 103 k 2024-04-02T07:32:31.740 INFO:teuthology.orchestra.run.smithi113.stdout: xfsdump x86_64 3.1.8-7.el8 baseos 334 k 2024-04-02T07:32:31.740 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs x86_64 5.0.0-12.el8 baseos 1.1 M 2024-04-02T07:32:31.740 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-04-02T07:32:31.740 INFO:teuthology.orchestra.run.smithi113.stdout: automake noarch 1.16.1-8.el8 appstream 740 k 2024-04-02T07:32:31.740 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 22 M 2024-04-02T07:32:31.740 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 23 k 2024-04-02T07:32:31.740 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 2.3 M 2024-04-02T07:32:31.740 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 245 k 2024-04-02T07:32:31.741 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 4.8 M 2024-04-02T07:32:31.741 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 18 M 2024-04-02T07:32:31.741 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 15 k 2024-04-02T07:32:31.741 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 24 k 2024-04-02T07:32:31.741 INFO:teuthology.orchestra.run.smithi113.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-02T07:32:31.741 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-02T07:32:31.741 INFO:teuthology.orchestra.run.smithi113.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-02T07:32:31.741 INFO:teuthology.orchestra.run.smithi113.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 baseos 48 k 2024-04-02T07:32:31.741 INFO:teuthology.orchestra.run.smithi113.stdout: krb5-devel x86_64 1.18.2-22.el8 baseos 561 k 2024-04-02T07:32:31.741 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 142 k 2024-04-02T07:32:31.741 INFO:teuthology.orchestra.run.smithi113.stdout: libcom_err-devel x86_64 1.45.6-5.el8 baseos 39 k 2024-04-02T07:32:31.741 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-02T07:32:31.741 INFO:teuthology.orchestra.run.smithi113.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-02T07:32:31.741 INFO:teuthology.orchestra.run.smithi113.stdout: libpq x86_64 13.5-1.el8 appstream 198 k 2024-04-02T07:32:31.742 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-02T07:32:31.742 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-02T07:32:31.742 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 474 k 2024-04-02T07:32:31.742 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-02T07:32:31.742 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 6.5 M 2024-04-02T07:32:31.742 INFO:teuthology.orchestra.run.smithi113.stdout: libselinux-devel x86_64 2.9-8.el8 baseos 200 k 2024-04-02T07:32:31.742 INFO:teuthology.orchestra.run.smithi113.stdout: libsepol-devel x86_64 2.9-3.el8 baseos 87 k 2024-04-02T07:32:31.742 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-02T07:32:31.742 INFO:teuthology.orchestra.run.smithi113.stdout: libverto-devel x86_64 0.3.2-2.el8 baseos 18 k 2024-04-02T07:32:31.742 INFO:teuthology.orchestra.run.smithi113.stdout: libzstd-devel x86_64 1.4.4-1.el8 baseos 44 k 2024-04-02T07:32:31.742 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-02T07:32:31.742 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-02T07:32:31.742 INFO:teuthology.orchestra.run.smithi113.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-02T07:32:31.743 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-02T07:32:31.743 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-devel x86_64 10.32-3.el8 baseos 605 k 2024-04-02T07:32:31.743 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-utf16 x86_64 10.32-3.el8 baseos 229 k 2024-04-02T07:32:31.743 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-utf32 x86_64 10.32-3.el8 baseos 220 k 2024-04-02T07:32:31.743 INFO:teuthology.orchestra.run.smithi113.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-02T07:32:31.743 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-02T07:32:31.743 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-02T07:32:31.743 INFO:teuthology.orchestra.run.smithi113.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-02T07:32:31.743 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-02T07:32:31.743 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 45 k 2024-04-02T07:32:31.743 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 129 k 2024-04-02T07:32:31.743 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-02T07:32:31.743 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-02T07:32:31.743 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-02T07:32:31.744 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-02T07:32:31.744 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-02T07:32:31.744 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-02T07:32:31.744 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-02T07:32:31.744 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-02T07:32:31.744 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-02T07:32:31.744 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-02T07:32:31.744 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-02T07:32:31.744 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-02T07:32:31.744 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-02T07:32:31.744 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-02T07:32:31.744 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-02T07:32:31.744 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-02T07:32:31.744 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-02T07:32:31.745 INFO:teuthology.orchestra.run.smithi113.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-02T07:32:31.745 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-02T07:32:31.745 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-02T07:32:31.745 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-02T07:32:31.745 INFO:teuthology.orchestra.run.smithi113.stdout: python3-packaging noarch 16.8-9.el8 powertools 48 k 2024-04-02T07:32:31.745 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-02T07:32:31.745 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-02T07:32:31.745 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-02T07:32:31.745 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-02T07:32:31.745 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-02T07:32:31.745 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-02T07:32:31.745 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-02T07:32:31.745 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-02T07:32:31.745 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-02T07:32:31.746 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-02T07:32:31.746 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-02T07:32:31.746 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-02T07:32:31.746 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-02T07:32:31.746 INFO:teuthology.orchestra.run.smithi113.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-02T07:32:31.746 INFO:teuthology.orchestra.run.smithi113.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-02T07:32:31.746 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-02T07:32:31.746 INFO:teuthology.orchestra.run.smithi113.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-02T07:32:31.746 INFO:teuthology.orchestra.run.smithi113.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-02T07:32:31.746 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-02T07:32:31.746 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-02T07:32:31.746 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-02T07:32:31.746 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-02T07:32:31.746 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-02T07:32:31.747 INFO:teuthology.orchestra.run.smithi113.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-02T07:32:31.747 INFO:teuthology.orchestra.run.smithi113.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-02T07:32:31.747 INFO:teuthology.orchestra.run.smithi113.stdout: uuid x86_64 1.6.2-43.el8 appstream 64 k 2024-04-02T07:32:31.747 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-02T07:32:31.747 INFO:teuthology.orchestra.run.smithi113.stdout: zlib-devel x86_64 1.2.11-21.el8 baseos 58 k 2024-04-02T07:32:31.747 INFO:teuthology.orchestra.run.smithi113.stdout:Enabling module streams: 2024-04-02T07:32:31.747 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql 10 2024-04-02T07:32:31.747 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:32:31.747 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-02T07:32:31.747 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-04-02T07:32:31.747 INFO:teuthology.orchestra.run.smithi113.stdout:Install 129 Packages 2024-04-02T07:32:31.747 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrade 24 Packages 2024-04-02T07:32:31.747 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:32:31.748 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 220 M 2024-04-02T07:32:31.748 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-02T07:32:31.993 INFO:teuthology.orchestra.run.smithi089.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-02T07:32:31.993 INFO:teuthology.orchestra.run.smithi089.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-02T07:32:32.003 INFO:teuthology.orchestra.run.smithi089.stdout:Package NetworkManager-1:1.40.16-4.el8.x86_64 is already installed. 2024-04-02T07:32:32.004 INFO:teuthology.orchestra.run.smithi089.stdout:Package iproute-6.2.0-1.el8.x86_64 is already installed. 2024-04-02T07:32:32.005 INFO:teuthology.orchestra.run.smithi089.stdout:Package util-linux-2.32.1-41.el8.x86_64 is already installed. 2024-04-02T07:32:32.005 INFO:teuthology.orchestra.run.smithi089.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-04-02T07:32:32.007 INFO:teuthology.orchestra.run.smithi089.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-04-02T07:32:32.009 INFO:teuthology.orchestra.run.smithi089.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-04-02T07:32:32.010 INFO:teuthology.orchestra.run.smithi089.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-04-02T07:32:32.094 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-02T07:32:32.109 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-04-02T07:32:32.109 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-04-02T07:32:32.109 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-04-02T07:32:32.109 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-02T07:32:32.109 INFO:teuthology.orchestra.run.smithi089.stdout: bison x86_64 3.0.4-10.el8 appstream 688 k 2024-04-02T07:32:32.109 INFO:teuthology.orchestra.run.smithi089.stdout: ceph x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 6.4 k 2024-04-02T07:32:32.109 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 5.5 M 2024-04-02T07:32:32.109 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 854 k 2024-04-02T07:32:32.109 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 166 k 2024-04-02T07:32:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 1.6 M 2024-04-02T07:32:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 138 k 2024-04-02T07:32:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 1.8 M 2024-04-02T07:32:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 7.4 M 2024-04-02T07:32:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 49 k 2024-04-02T07:32:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 13 M 2024-04-02T07:32:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 47 M 2024-04-02T07:32:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 263 k 2024-04-02T07:32:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 173 k 2024-04-02T07:32:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-mirror x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 261 k 2024-04-02T07:32:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-top noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 18 k 2024-04-02T07:32:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-devel x86_64 0.190-2.el8 baseos 62 k 2024-04-02T07:32:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: flex x86_64 2.6.1-9.el8 appstream 320 k 2024-04-02T07:32:32.111 INFO:teuthology.orchestra.run.smithi089.stdout: libaio-devel x86_64 0.3.112-1.el8 baseos 19 k 2024-04-02T07:32:32.111 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 31 k 2024-04-02T07:32:32.111 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 778 k 2024-04-02T07:32:32.111 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 139 k 2024-04-02T07:32:32.111 INFO:teuthology.orchestra.run.smithi089.stdout: libtool x86_64 2.4.6-25.el8 appstream 709 k 2024-04-02T07:32:32.111 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-devel x86_64 2.32.1-46.el8 baseos 100 k 2024-04-02T07:32:32.111 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-devel x86_64 1:1.1.1k-12.el8 baseos 3.2 M 2024-04-02T07:32:32.111 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql x86_64 10.23-2.module_el8+572+929c87ac appstream 1.6 M 2024-04-02T07:32:32.111 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-contrib x86_64 10.23-2.module_el8+572+929c87ac appstream 901 k 2024-04-02T07:32:32.111 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-server x86_64 10.23-2.module_el8+572+929c87ac appstream 5.1 M 2024-04-02T07:32:32.111 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 197 k 2024-04-02T07:32:32.111 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 371 k 2024-04-02T07:32:32.111 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 364 k 2024-04-02T07:32:32.111 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 106 k 2024-04-02T07:32:32.112 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 95 k 2024-04-02T07:32:32.112 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 3.5 M 2024-04-02T07:32:32.112 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 181 k 2024-04-02T07:32:32.112 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 baseos 74 k 2024-04-02T07:32:32.112 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2024-04-02T07:32:32.112 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager x86_64 1:1.40.16-15.el8 baseos 2.3 M 2024-04-02T07:32:32.112 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-libnm x86_64 1:1.40.16-15.el8 baseos 1.9 M 2024-04-02T07:32:32.112 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-team x86_64 1:1.40.16-15.el8 baseos 161 k 2024-04-02T07:32:32.112 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-tui x86_64 1:1.40.16-15.el8 baseos 356 k 2024-04-02T07:32:32.112 INFO:teuthology.orchestra.run.smithi089.stdout: acl x86_64 2.2.53-3.el8 baseos 83 k 2024-04-02T07:32:32.112 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-debuginfod-client x86_64 0.190-2.el8 baseos 77 k 2024-04-02T07:32:32.112 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf x86_64 0.190-2.el8 baseos 233 k 2024-04-02T07:32:32.112 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libs x86_64 0.190-2.el8 baseos 305 k 2024-04-02T07:32:32.112 INFO:teuthology.orchestra.run.smithi089.stdout: iproute x86_64 6.2.0-5.el8 baseos 854 k 2024-04-02T07:32:32.113 INFO:teuthology.orchestra.run.smithi089.stdout: libacl x86_64 2.2.53-3.el8 baseos 35 k 2024-04-02T07:32:32.113 INFO:teuthology.orchestra.run.smithi089.stdout: libacl-devel x86_64 2.2.53-3.el8 baseos 85 k 2024-04-02T07:32:32.113 INFO:teuthology.orchestra.run.smithi089.stdout: libblkid x86_64 2.32.1-46.el8 baseos 221 k 2024-04-02T07:32:32.113 INFO:teuthology.orchestra.run.smithi089.stdout: libfdisk x86_64 2.32.1-46.el8 baseos 254 k 2024-04-02T07:32:32.113 INFO:teuthology.orchestra.run.smithi089.stdout: libmount x86_64 2.32.1-46.el8 baseos 237 k 2024-04-02T07:32:32.113 INFO:teuthology.orchestra.run.smithi089.stdout: librados2 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 3.7 M 2024-04-02T07:32:32.113 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 3.6 M 2024-04-02T07:32:32.113 INFO:teuthology.orchestra.run.smithi089.stdout: libsmartcols x86_64 2.32.1-46.el8 baseos 180 k 2024-04-02T07:32:32.113 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid x86_64 2.32.1-46.el8 baseos 99 k 2024-04-02T07:32:32.113 INFO:teuthology.orchestra.run.smithi089.stdout: openssl x86_64 1:1.1.1k-12.el8 baseos 738 k 2024-04-02T07:32:32.113 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-libs x86_64 1:1.1.1k-12.el8 baseos 1.5 M 2024-04-02T07:32:32.113 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux x86_64 2.32.1-46.el8 baseos 2.6 M 2024-04-02T07:32:32.113 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux-user x86_64 2.32.1-46.el8 baseos 103 k 2024-04-02T07:32:32.113 INFO:teuthology.orchestra.run.smithi089.stdout: xfsdump x86_64 3.1.8-7.el8 baseos 334 k 2024-04-02T07:32:32.114 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs x86_64 5.0.0-12.el8 baseos 1.1 M 2024-04-02T07:32:32.114 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-04-02T07:32:32.114 INFO:teuthology.orchestra.run.smithi089.stdout: automake noarch 1.16.1-8.el8 appstream 740 k 2024-04-02T07:32:32.114 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 22 M 2024-04-02T07:32:32.114 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 23 k 2024-04-02T07:32:32.114 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 2.3 M 2024-04-02T07:32:32.114 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 245 k 2024-04-02T07:32:32.114 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 4.8 M 2024-04-02T07:32:32.114 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 18 M 2024-04-02T07:32:32.114 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1015.gd0e392d9.el8 ceph-noarch 15 k 2024-04-02T07:32:32.114 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 24 k 2024-04-02T07:32:32.114 INFO:teuthology.orchestra.run.smithi089.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-02T07:32:32.114 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-02T07:32:32.114 INFO:teuthology.orchestra.run.smithi089.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-02T07:32:32.115 INFO:teuthology.orchestra.run.smithi089.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 baseos 48 k 2024-04-02T07:32:32.115 INFO:teuthology.orchestra.run.smithi089.stdout: krb5-devel x86_64 1.18.2-22.el8 baseos 561 k 2024-04-02T07:32:32.115 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 142 k 2024-04-02T07:32:32.115 INFO:teuthology.orchestra.run.smithi089.stdout: libcom_err-devel x86_64 1.45.6-5.el8 baseos 39 k 2024-04-02T07:32:32.115 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-02T07:32:32.115 INFO:teuthology.orchestra.run.smithi089.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-02T07:32:32.115 INFO:teuthology.orchestra.run.smithi089.stdout: libpq x86_64 13.5-1.el8 appstream 198 k 2024-04-02T07:32:32.115 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-02T07:32:32.115 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-02T07:32:32.115 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 474 k 2024-04-02T07:32:32.115 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-02T07:32:32.115 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2 x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 6.5 M 2024-04-02T07:32:32.115 INFO:teuthology.orchestra.run.smithi089.stdout: libselinux-devel x86_64 2.9-8.el8 baseos 200 k 2024-04-02T07:32:32.115 INFO:teuthology.orchestra.run.smithi089.stdout: libsepol-devel x86_64 2.9-3.el8 baseos 87 k 2024-04-02T07:32:32.116 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-02T07:32:32.116 INFO:teuthology.orchestra.run.smithi089.stdout: libverto-devel x86_64 0.3.2-2.el8 baseos 18 k 2024-04-02T07:32:32.116 INFO:teuthology.orchestra.run.smithi089.stdout: libzstd-devel x86_64 1.4.4-1.el8 baseos 44 k 2024-04-02T07:32:32.116 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-02T07:32:32.116 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-02T07:32:32.116 INFO:teuthology.orchestra.run.smithi089.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-02T07:32:32.116 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-02T07:32:32.116 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-devel x86_64 10.32-3.el8 baseos 605 k 2024-04-02T07:32:32.116 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-utf16 x86_64 10.32-3.el8 baseos 229 k 2024-04-02T07:32:32.116 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-utf32 x86_64 10.32-3.el8 baseos 220 k 2024-04-02T07:32:32.116 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-02T07:32:32.116 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-02T07:32:32.116 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-02T07:32:32.116 INFO:teuthology.orchestra.run.smithi089.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 45 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common x86_64 2:18.2.2-1015.gd0e392d9.el8 ceph 129 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-02T07:32:32.117 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-02T07:32:32.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-02T07:32:32.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-02T07:32:32.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-02T07:32:32.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-02T07:32:32.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-02T07:32:32.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-02T07:32:32.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-02T07:32:32.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-02T07:32:32.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-02T07:32:32.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging noarch 16.8-9.el8 powertools 48 k 2024-04-02T07:32:32.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-02T07:32:32.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-02T07:32:32.118 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-02T07:32:32.119 INFO:teuthology.orchestra.run.smithi089.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout: uuid x86_64 1.6.2-43.el8 appstream 64 k 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout: zlib-devel x86_64 1.2.11-21.el8 baseos 58 k 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling module streams: 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql 10 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-02T07:32:32.120 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-04-02T07:32:32.121 INFO:teuthology.orchestra.run.smithi089.stdout:Install 129 Packages 2024-04-02T07:32:32.121 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 24 Packages 2024-04-02T07:32:32.121 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:32:32.121 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 220 M 2024-04-02T07:32:32.121 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-02T07:32:32.378 INFO:teuthology.orchestra.run.smithi113.stdout:(1/153): ceph-18.2.2-1015.gd0e392d9.el8.x86_64. 51 kB/s | 6.4 kB 00:00 2024-04-02T07:32:32.595 INFO:teuthology.orchestra.run.smithi113.stdout:(2/153): ceph-fuse-18.2.2-1015.gd0e392d9.el8.x8 3.8 MB/s | 854 kB 00:00 2024-04-02T07:32:32.654 INFO:teuthology.orchestra.run.smithi113.stdout:(3/153): ceph-immutable-object-cache-18.2.2-101 2.8 MB/s | 166 kB 00:00 2024-04-02T07:32:32.954 INFO:teuthology.orchestra.run.smithi113.stdout:(4/153): ceph-mds-18.2.2-1015.gd0e392d9.el8.x86 7.6 MB/s | 2.3 MB 00:00 2024-04-02T07:32:33.134 INFO:teuthology.orchestra.run.smithi089.stdout:(1/153): ceph-18.2.2-1015.gd0e392d9.el8.x86_64. 25 kB/s | 6.4 kB 00:00 2024-04-02T07:32:33.171 INFO:teuthology.orchestra.run.smithi113.stdout:(5/153): ceph-mgr-18.2.2-1015.gd0e392d9.el8.x86 7.2 MB/s | 1.6 MB 00:00 2024-04-02T07:32:33.522 INFO:teuthology.orchestra.run.smithi113.stdout:(6/153): ceph-base-18.2.2-1015.gd0e392d9.el8.x8 4.3 MB/s | 5.5 MB 00:01 2024-04-02T07:32:33.847 INFO:teuthology.orchestra.run.smithi113.stdout:(7/153): ceph-mon-18.2.2-1015.gd0e392d9.el8.x86 7.2 MB/s | 4.8 MB 00:00 2024-04-02T07:32:34.388 INFO:teuthology.orchestra.run.smithi113.stdout:(8/153): ceph-common-18.2.2-1015.gd0e392d9.el8. 10 MB/s | 22 MB 00:02 2024-04-02T07:32:34.464 INFO:teuthology.orchestra.run.smithi113.stdout:(9/153): ceph-selinux-18.2.2-1015.gd0e392d9.el8 325 kB/s | 24 kB 00:00 2024-04-02T07:32:34.518 INFO:teuthology.orchestra.run.smithi089.stdout:(2/153): ceph-fuse-18.2.2-1015.gd0e392d9.el8.x8 617 kB/s | 854 kB 00:01 2024-04-02T07:32:34.876 INFO:teuthology.orchestra.run.smithi089.stdout:(3/153): ceph-immutable-object-cache-18.2.2-101 463 kB/s | 166 kB 00:00 2024-04-02T07:32:35.714 INFO:teuthology.orchestra.run.smithi113.stdout:(10/153): ceph-radosgw-18.2.2-1015.gd0e392d9.el 7.0 MB/s | 13 MB 00:01 2024-04-02T07:32:35.797 INFO:teuthology.orchestra.run.smithi113.stdout:(11/153): cephfs-mirror-18.2.2-1015.gd0e392d9.e 3.1 MB/s | 261 kB 00:00 2024-04-02T07:32:35.873 INFO:teuthology.orchestra.run.smithi113.stdout:(12/153): libcephfs-devel-18.2.2-1015.gd0e392d9 412 kB/s | 31 kB 00:00 2024-04-02T07:32:36.023 INFO:teuthology.orchestra.run.smithi113.stdout:(13/153): libcephfs2-18.2.2-1015.gd0e392d9.el8. 5.1 MB/s | 778 kB 00:00 2024-04-02T07:32:36.098 INFO:teuthology.orchestra.run.smithi113.stdout:(14/153): libcephsqlite-18.2.2-1015.gd0e392d9.e 1.8 MB/s | 142 kB 00:00 2024-04-02T07:32:36.235 INFO:teuthology.orchestra.run.smithi089.stdout:(4/153): ceph-base-18.2.2-1015.gd0e392d9.el8.x8 1.6 MB/s | 5.5 MB 00:03 2024-04-02T07:32:36.299 INFO:teuthology.orchestra.run.smithi113.stdout:(15/153): ceph-osd-18.2.2-1015.gd0e392d9.el8.x8 6.6 MB/s | 18 MB 00:02 2024-04-02T07:32:36.332 INFO:teuthology.orchestra.run.smithi113.stdout:(16/153): librados-devel-18.2.2-1015.gd0e392d9. 594 kB/s | 139 kB 00:00 2024-04-02T07:32:36.374 INFO:teuthology.orchestra.run.smithi113.stdout:(17/153): libradosstriper1-18.2.2-1015.gd0e392d 6.2 MB/s | 474 kB 00:00 2024-04-02T07:32:36.458 INFO:teuthology.orchestra.run.smithi113.stdout:(18/153): python3-ceph-argparse-18.2.2-1015.gd0 537 kB/s | 45 kB 00:00 2024-04-02T07:32:36.558 INFO:teuthology.orchestra.run.smithi113.stdout:(19/153): python3-ceph-common-18.2.2-1015.gd0e3 1.3 MB/s | 129 kB 00:00 2024-04-02T07:32:36.667 INFO:teuthology.orchestra.run.smithi113.stdout:(20/153): python3-cephfs-18.2.2-1015.gd0e392d9. 1.8 MB/s | 197 kB 00:00 2024-04-02T07:32:36.775 INFO:teuthology.orchestra.run.smithi113.stdout:(21/153): python3-rados-18.2.2-1015.gd0e392d9.e 3.4 MB/s | 371 kB 00:00 2024-04-02T07:32:36.884 INFO:teuthology.orchestra.run.smithi113.stdout:(22/153): python3-rbd-18.2.2-1015.gd0e392d9.el8 3.3 MB/s | 364 kB 00:00 2024-04-02T07:32:36.885 INFO:teuthology.orchestra.run.smithi089.stdout:(5/153): ceph-mgr-18.2.2-1015.gd0e392d9.el8.x86 2.4 MB/s | 1.6 MB 00:00 2024-04-02T07:32:36.976 INFO:teuthology.orchestra.run.smithi113.stdout:(23/153): python3-rgw-18.2.2-1015.gd0e392d9.el8 1.1 MB/s | 106 kB 00:00 2024-04-02T07:32:37.060 INFO:teuthology.orchestra.run.smithi113.stdout:(24/153): librgw2-18.2.2-1015.gd0e392d9.el8.x86 9.0 MB/s | 6.5 MB 00:00 2024-04-02T07:32:37.076 INFO:teuthology.orchestra.run.smithi113.stdout:(25/153): rbd-fuse-18.2.2-1015.gd0e392d9.el8.x8 950 kB/s | 95 kB 00:00 2024-04-02T07:32:37.168 INFO:teuthology.orchestra.run.smithi113.stdout:(26/153): rbd-nbd-18.2.2-1015.gd0e392d9.el8.x86 1.9 MB/s | 181 kB 00:00 2024-04-02T07:32:37.260 INFO:teuthology.orchestra.run.smithi113.stdout:(27/153): ceph-grafana-dashboards-18.2.2-1015.g 254 kB/s | 23 kB 00:00 2024-04-02T07:32:37.310 INFO:teuthology.orchestra.run.smithi089.stdout:(6/153): ceph-mds-18.2.2-1015.gd0e392d9.el8.x86 964 kB/s | 2.3 MB 00:02 2024-04-02T07:32:37.353 INFO:teuthology.orchestra.run.smithi113.stdout:(28/153): ceph-mgr-cephadm-18.2.2-1015.gd0e392d 1.5 MB/s | 138 kB 00:00 2024-04-02T07:32:37.436 INFO:teuthology.orchestra.run.smithi113.stdout:(29/153): rbd-mirror-18.2.2-1015.gd0e392d9.el8. 9.2 MB/s | 3.5 MB 00:00 2024-04-02T07:32:37.620 INFO:teuthology.orchestra.run.smithi113.stdout:(30/153): ceph-mgr-dashboard-18.2.2-1015.gd0e39 6.7 MB/s | 1.8 MB 00:00 2024-04-02T07:32:37.728 INFO:teuthology.orchestra.run.smithi113.stdout:(31/153): ceph-mgr-modules-core-18.2.2-1015.gd0 2.2 MB/s | 245 kB 00:00 2024-04-02T07:32:37.829 INFO:teuthology.orchestra.run.smithi113.stdout:(32/153): ceph-mgr-rook-18.2.2-1015.gd0e392d9.e 492 kB/s | 49 kB 00:00 2024-04-02T07:32:37.921 INFO:teuthology.orchestra.run.smithi113.stdout:(33/153): ceph-prometheus-alerts-18.2.2-1015.gd 161 kB/s | 15 kB 00:00 2024-04-02T07:32:38.021 INFO:teuthology.orchestra.run.smithi113.stdout:(34/153): ceph-volume-18.2.2-1015.gd0e392d9.el8 2.6 MB/s | 263 kB 00:00 2024-04-02T07:32:38.130 INFO:teuthology.orchestra.run.smithi113.stdout:(35/153): cephadm-18.2.2-1015.gd0e392d9.el8.noa 1.6 MB/s | 173 kB 00:00 2024-04-02T07:32:38.263 INFO:teuthology.orchestra.run.smithi113.stdout:(36/153): ceph-mgr-diskprediction-local-18.2.2- 8.9 MB/s | 7.4 MB 00:00 2024-04-02T07:32:38.288 INFO:teuthology.orchestra.run.smithi113.stdout:(37/153): cephfs-top-18.2.2-1015.gd0e392d9.el8. 114 kB/s | 18 kB 00:00 2024-04-02T07:32:38.497 INFO:teuthology.orchestra.run.smithi113.stdout:(38/153): bison-3.0.4-10.el8.x86_64.rpm 3.2 MB/s | 688 kB 00:00 2024-04-02T07:32:38.522 INFO:teuthology.orchestra.run.smithi113.stdout:(39/153): automake-1.16.1-8.el8.noarch.rpm 2.8 MB/s | 740 kB 00:00 2024-04-02T07:32:38.586 INFO:teuthology.orchestra.run.smithi089.stdout:(7/153): ceph-mon-18.2.2-1015.gd0e392d9.el8.x86 2.8 MB/s | 4.8 MB 00:01 2024-04-02T07:32:38.998 INFO:teuthology.orchestra.run.smithi113.stdout:(40/153): ceph-test-18.2.2-1015.gd0e392d9.el8.x 10 MB/s | 47 MB 00:04 2024-04-02T07:32:39.015 INFO:teuthology.orchestra.run.smithi113.stdout:(41/153): jq-1.6-8.el8.x86_64.rpm 413 kB/s | 203 kB 00:00 2024-04-02T07:32:39.032 INFO:teuthology.orchestra.run.smithi113.stdout:(42/153): flex-2.6.1-9.el8.x86_64.rpm 598 kB/s | 320 kB 00:00 2024-04-02T07:32:39.124 INFO:teuthology.orchestra.run.smithi113.stdout:(43/153): librdkafka-1.6.1-1.el8.x86_64.rpm 6.1 MB/s | 677 kB 00:00 2024-04-02T07:32:39.157 INFO:teuthology.orchestra.run.smithi113.stdout:(44/153): libtool-2.4.6-25.el8.x86_64.rpm 5.5 MB/s | 709 kB 00:00 2024-04-02T07:32:39.250 INFO:teuthology.orchestra.run.smithi113.stdout:(45/153): libpq-13.5-1.el8.x86_64.rpm 788 kB/s | 198 kB 00:00 2024-04-02T07:32:39.266 INFO:teuthology.orchestra.run.smithi113.stdout:(46/153): oniguruma-6.8.2-3.el8.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-02T07:32:39.292 INFO:teuthology.orchestra.run.smithi113.stdout:(47/153): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2024-04-02T07:32:39.709 INFO:teuthology.orchestra.run.smithi113.stdout:(48/153): postgresql-10.23-2.module_el8+572+929 3.8 MB/s | 1.6 MB 00:00 2024-04-02T07:32:39.809 INFO:teuthology.orchestra.run.smithi113.stdout:(49/153): openblas-0.3.15-4.el8.x86_64.rpm 8.5 MB/s | 4.7 MB 00:00 2024-04-02T07:32:39.893 INFO:teuthology.orchestra.run.smithi113.stdout:(50/153): postgresql-contrib-10.23-2.module_el8 4.8 MB/s | 901 kB 00:00 2024-04-02T07:32:39.960 INFO:teuthology.orchestra.run.smithi113.stdout:(51/153): openblas-threads-0.3.15-4.el8.x86_64. 7.0 MB/s | 4.9 MB 00:00 2024-04-02T07:32:40.076 INFO:teuthology.orchestra.run.smithi113.stdout:(52/153): python3-jinja2-2.10.1-4.el8.noarch.rp 4.9 MB/s | 578 kB 00:00 2024-04-02T07:32:40.152 INFO:teuthology.orchestra.run.smithi113.stdout:(53/153): postgresql-server-10.23-2.module_el8+ 15 MB/s | 5.1 MB 00:00 2024-04-02T07:32:40.186 INFO:teuthology.orchestra.run.smithi113.stdout:(54/153): python3-jsonpatch-1.21-2.el8.noarch.r 244 kB/s | 27 kB 00:00 2024-04-02T07:32:40.211 INFO:teuthology.orchestra.run.smithi113.stdout:(55/153): python3-jsonpointer-1.10-11.el8.noarc 338 kB/s | 20 kB 00:00 2024-04-02T07:32:40.236 INFO:teuthology.orchestra.run.smithi113.stdout:(56/153): python3-mako-1.0.6-14.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2024-04-02T07:32:40.278 INFO:teuthology.orchestra.run.smithi113.stdout:(57/153): python3-markupsafe-0.23-19.el8.x86_64 583 kB/s | 39 kB 00:00 2024-04-02T07:32:40.320 INFO:teuthology.orchestra.run.smithi113.stdout:(58/153): python3-numpy-f2py-1.14.3-10.el8.x86_ 5.3 MB/s | 225 kB 00:00 2024-04-02T07:32:40.335 INFO:teuthology.orchestra.run.smithi089.stdout:(8/153): ceph-radosgw-18.2.2-1015.gd0e392d9.el8 7.5 MB/s | 13 MB 00:01 2024-04-02T07:32:40.386 INFO:teuthology.orchestra.run.smithi089.stdout:(9/153): ceph-selinux-18.2.2-1015.gd0e392d9.el8 491 kB/s | 24 kB 00:00 2024-04-02T07:32:40.396 INFO:teuthology.orchestra.run.smithi113.stdout:(59/153): python3-prettytable-0.7.2-14.el8.noar 583 kB/s | 44 kB 00:00 2024-04-02T07:32:40.446 INFO:teuthology.orchestra.run.smithi113.stdout:(60/153): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2024-04-02T07:32:40.554 INFO:teuthology.orchestra.run.smithi113.stdout:(61/153): python3-numpy-1.14.3-10.el8.x86_64.rp 12 MB/s | 3.7 MB 00:00 2024-04-02T07:32:40.621 INFO:teuthology.orchestra.run.smithi113.stdout:(62/153): python3-pyasn1-0.3.7-6.el8.noarch.rpm 722 kB/s | 126 kB 00:00 2024-04-02T07:32:40.638 INFO:teuthology.orchestra.run.smithi113.stdout:(63/153): python3-pyasn1-modules-0.3.7-6.el8.no 1.3 MB/s | 110 kB 00:00 2024-04-02T07:32:40.664 INFO:teuthology.orchestra.run.smithi113.stdout:(64/153): python3-pytz-2017.2-11.el8.noarch.rpm 1.3 MB/s | 57 kB 00:00 2024-04-02T07:32:40.731 INFO:teuthology.orchestra.run.smithi113.stdout:(65/153): python3-babel-2.5.1-7.el8.noarch.rpm 5.7 MB/s | 4.8 MB 00:00 2024-04-02T07:32:40.765 INFO:teuthology.orchestra.run.smithi113.stdout:(66/153): python3-werkzeug-0.12.2-4.el8.noarch. 4.5 MB/s | 457 kB 00:00 2024-04-02T07:32:40.798 INFO:teuthology.orchestra.run.smithi113.stdout:(67/153): socat-1.7.4.1-1.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2024-04-02T07:32:40.832 INFO:teuthology.orchestra.run.smithi113.stdout:(68/153): uuid-1.6.2-43.el8.x86_64.rpm 955 kB/s | 64 kB 00:00 2024-04-02T07:32:40.857 INFO:teuthology.orchestra.run.smithi113.stdout:(69/153): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-04-02T07:32:41.166 INFO:teuthology.orchestra.run.smithi113.stdout:(70/153): elfutils-libelf-devel-0.190-2.el8.x86 186 kB/s | 62 kB 00:00 2024-04-02T07:32:41.350 INFO:teuthology.orchestra.run.smithi113.stdout:(71/153): python3-scipy-1.0.0-21.module_el8+762 20 MB/s | 14 MB 00:00 2024-04-02T07:32:41.374 INFO:teuthology.orchestra.run.smithi113.stdout:(72/153): fuse-2.9.7-19.el8.x86_64.rpm 161 kB/s | 83 kB 00:00 2024-04-02T07:32:41.391 INFO:teuthology.orchestra.run.smithi113.stdout:(73/153): keyutils-libs-devel-1.5.10-9.el8.x86_ 213 kB/s | 48 kB 00:00 2024-04-02T07:32:41.484 INFO:teuthology.orchestra.run.smithi113.stdout:(74/153): libcom_err-devel-1.45.6-5.el8.x86_64. 422 kB/s | 39 kB 00:00 2024-04-02T07:32:41.600 INFO:teuthology.orchestra.run.smithi113.stdout:(75/153): libaio-devel-0.3.112-1.el8.x86_64.rpm 83 kB/s | 19 kB 00:00 2024-04-02T07:32:41.651 INFO:teuthology.orchestra.run.smithi113.stdout:(76/153): krb5-devel-1.18.2-22.el8.x86_64.rpm 1.8 MB/s | 561 kB 00:00 2024-04-02T07:32:41.718 INFO:teuthology.orchestra.run.smithi113.stdout:(77/153): libgfortran-8.5.0-21.el8.x86_64.rpm 2.7 MB/s | 645 kB 00:00 2024-04-02T07:32:41.734 INFO:teuthology.orchestra.run.smithi113.stdout:(78/153): librabbitmq-0.9.0-5.el8.x86_64.rpm 563 kB/s | 47 kB 00:00 2024-04-02T07:32:41.810 INFO:teuthology.orchestra.run.smithi113.stdout:(79/153): libselinux-devel-2.9-8.el8.x86_64.rpm 2.1 MB/s | 200 kB 00:00 2024-04-02T07:32:41.836 INFO:teuthology.orchestra.run.smithi113.stdout:(80/153): libquadmath-8.5.0-21.el8.x86_64.rpm 735 kB/s | 172 kB 00:00 2024-04-02T07:32:41.861 INFO:teuthology.orchestra.run.smithi113.stdout:(81/153): libsepol-devel-2.9-3.el8.x86_64.rpm 691 kB/s | 87 kB 00:00 2024-04-02T07:32:41.902 INFO:teuthology.orchestra.run.smithi113.stdout:(82/153): libuuid-devel-2.32.1-46.el8.x86_64.rp 1.1 MB/s | 100 kB 00:00 2024-04-02T07:32:41.944 INFO:teuthology.orchestra.run.smithi113.stdout:(83/153): libverto-devel-0.3.2-2.el8.x86_64.rpm 217 kB/s | 18 kB 00:00 2024-04-02T07:32:41.962 INFO:teuthology.orchestra.run.smithi113.stdout:(84/153): libzstd-devel-1.4.4-1.el8.x86_64.rpm 436 kB/s | 44 kB 00:00 2024-04-02T07:32:42.112 INFO:teuthology.orchestra.run.smithi113.stdout:(85/153): pcre2-devel-10.32-3.el8.x86_64.rpm 3.5 MB/s | 605 kB 00:00 2024-04-02T07:32:42.137 INFO:teuthology.orchestra.run.smithi113.stdout:(86/153): pcre2-utf16-10.32-3.el8.x86_64.rpm 1.3 MB/s | 229 kB 00:00 2024-04-02T07:32:42.196 INFO:teuthology.orchestra.run.smithi113.stdout:(87/153): openssl-devel-1.1.1k-12.el8.x86_64.rp 11 MB/s | 3.2 MB 00:00 2024-04-02T07:32:42.213 INFO:teuthology.orchestra.run.smithi113.stdout:(88/153): pcre2-utf32-10.32-3.el8.x86_64.rpm 2.1 MB/s | 220 kB 00:00 2024-04-02T07:32:42.246 INFO:teuthology.orchestra.run.smithi113.stdout:(89/153): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.2 MB/s | 249 kB 00:00 2024-04-02T07:32:42.297 INFO:teuthology.orchestra.run.smithi113.stdout:(90/153): python3-cryptography-3.2.1-7.el8.x86_ 6.2 MB/s | 641 kB 00:00 2024-04-02T07:32:42.322 INFO:teuthology.orchestra.run.smithi113.stdout:(91/153): python3-jwt-1.6.1-2.el8.noarch.rpm 393 kB/s | 43 kB 00:00 2024-04-02T07:32:42.347 INFO:teuthology.orchestra.run.smithi113.stdout:(92/153): python3-oauthlib-2.1.0-1.el8.noarch.r 1.5 MB/s | 155 kB 00:00 2024-04-02T07:32:42.377 INFO:teuthology.orchestra.run.smithi089.stdout:(10/153): ceph-common-18.2.2-1015.gd0e392d9.el8 2.3 MB/s | 22 MB 00:09 2024-04-02T07:32:42.389 INFO:teuthology.orchestra.run.smithi113.stdout:(93/153): python3-pycparser-2.14-14.el8.noarch. 1.2 MB/s | 109 kB 00:00 2024-04-02T07:32:42.414 INFO:teuthology.orchestra.run.smithi113.stdout:(94/153): python3-requests-oauthlib-1.0.0-1.el8 462 kB/s | 43 kB 00:00 2024-04-02T07:32:42.440 INFO:teuthology.orchestra.run.smithi113.stdout:(95/153): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 808 kB/s | 74 kB 00:00 2024-04-02T07:32:42.482 INFO:teuthology.orchestra.run.smithi113.stdout:(96/153): zlib-devel-1.2.11-21.el8.x86_64.rpm 629 kB/s | 58 kB 00:00 2024-04-02T07:32:42.511 INFO:teuthology.orchestra.run.smithi089.stdout:(11/153): cephfs-mirror-18.2.2-1015.gd0e392d9.e 1.9 MB/s | 261 kB 00:00 2024-04-02T07:32:42.570 INFO:teuthology.orchestra.run.smithi089.stdout:(12/153): libcephfs-devel-18.2.2-1015.gd0e392d9 531 kB/s | 31 kB 00:00 2024-04-02T07:32:42.590 INFO:teuthology.orchestra.run.smithi113.stdout:(97/153): python3-packaging-16.8-9.el8.noarch.r 274 kB/s | 48 kB 00:00 2024-04-02T07:32:42.633 INFO:teuthology.orchestra.run.smithi113.stdout:(98/153): liboath-2.6.2-3.el8.x86_64.rpm 391 kB/s | 59 kB 00:00 2024-04-02T07:32:42.716 INFO:teuthology.orchestra.run.smithi113.stdout:(99/153): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2024-04-02T07:32:42.774 INFO:teuthology.orchestra.run.smithi113.stdout:(100/153): python3-bcrypt-3.1.6-2.el8.1.x86_64. 748 kB/s | 44 kB 00:00 2024-04-02T07:32:42.787 INFO:teuthology.orchestra.run.smithi089.stdout:(13/153): libcephfs2-18.2.2-1015.gd0e392d9.el8. 3.5 MB/s | 778 kB 00:00 2024-04-02T07:32:42.800 INFO:teuthology.orchestra.run.smithi113.stdout:(101/153): libunwind-1.3.1-3.el8.x86_64.rpm 359 kB/s | 75 kB 00:00 2024-04-02T07:32:42.826 INFO:teuthology.orchestra.run.smithi113.stdout:(102/153): python3-asyncssh-2.7.0-2.el8.noarch. 2.4 MB/s | 476 kB 00:00 2024-04-02T07:32:42.850 INFO:teuthology.orchestra.run.smithi113.stdout:(103/153): python3-beautifulsoup4-4.6.3-2.el8.1 2.4 MB/s | 185 kB 00:00 2024-04-02T07:32:42.862 INFO:teuthology.orchestra.run.smithi089.stdout:(14/153): libcephsqlite-18.2.2-1015.gd0e392d9.e 1.8 MB/s | 142 kB 00:00 2024-04-02T07:32:42.876 INFO:teuthology.orchestra.run.smithi113.stdout:(104/153): python3-cachetools-3.1.1-4.el8.noarc 437 kB/s | 33 kB 00:00 2024-04-02T07:32:42.901 INFO:teuthology.orchestra.run.smithi113.stdout:(105/153): python3-certifi-2018.10.15-7.el8.noa 216 kB/s | 16 kB 00:00 2024-04-02T07:32:42.926 INFO:teuthology.orchestra.run.smithi113.stdout:(106/153): python3-cheroot-8.5.2-1.el8.noarch.r 2.2 MB/s | 173 kB 00:00 2024-04-02T07:32:42.960 INFO:teuthology.orchestra.run.smithi113.stdout:(107/153): python3-google-auth-1.1.1-10.el8.noa 1.4 MB/s | 81 kB 00:00 2024-04-02T07:32:42.963 INFO:teuthology.orchestra.run.smithi089.stdout:(15/153): librados-devel-18.2.2-1015.gd0e392d9. 1.4 MB/s | 139 kB 00:00 2024-04-02T07:32:42.985 INFO:teuthology.orchestra.run.smithi113.stdout:(108/153): python3-jaraco-6.2-6.el8.noarch.rpm 188 kB/s | 11 kB 00:00 2024-04-02T07:32:43.011 INFO:teuthology.orchestra.run.smithi113.stdout:(109/153): python3-jaraco-functools-2.0-4.el8.n 349 kB/s | 18 kB 00:00 2024-04-02T07:32:43.036 INFO:teuthology.orchestra.run.smithi113.stdout:(110/153): python3-cherrypy-18.4.0-1.el8.noarch 2.3 MB/s | 384 kB 00:00 2024-04-02T07:32:43.061 INFO:teuthology.orchestra.run.smithi113.stdout:(111/153): python3-logutils-0.3.5-11.el8.noarch 971 kB/s | 49 kB 00:00 2024-04-02T07:32:43.095 INFO:teuthology.orchestra.run.smithi113.stdout:(112/153): python3-more-itertools-7.2.0-3.el8.n 1.0 MB/s | 59 kB 00:00 2024-04-02T07:32:43.120 INFO:teuthology.orchestra.run.smithi113.stdout:(113/153): python3-natsort-7.1.1-2.el8.noarch.r 1.0 MB/s | 60 kB 00:00 2024-04-02T07:32:43.137 INFO:teuthology.orchestra.run.smithi089.stdout:(16/153): libradosstriper1-18.2.2-1015.gd0e392d 2.7 MB/s | 474 kB 00:00 2024-04-02T07:32:43.162 INFO:teuthology.orchestra.run.smithi113.stdout:(114/153): python3-kubernetes-11.0.0-6.el8.noar 6.6 MB/s | 1.2 MB 00:00 2024-04-02T07:32:43.179 INFO:teuthology.orchestra.run.smithi113.stdout:(115/153): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2024-04-02T07:32:43.204 INFO:teuthology.orchestra.run.smithi113.stdout:(116/153): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2024-04-02T07:32:43.222 INFO:teuthology.orchestra.run.smithi113.stdout:(117/153): python3-repoze-lru-0.7-6.el8.noarch. 572 kB/s | 34 kB 00:00 2024-04-02T07:32:43.247 INFO:teuthology.orchestra.run.smithi113.stdout:(118/153): python3-routes-2.4.1-12.el8.noarch.r 2.9 MB/s | 196 kB 00:00 2024-04-02T07:32:43.272 INFO:teuthology.orchestra.run.smithi113.stdout:(119/153): python3-rsa-4.9-2.el8.noarch.rpm 954 kB/s | 64 kB 00:00 2024-04-02T07:32:43.289 INFO:teuthology.orchestra.run.smithi113.stdout:(120/153): python3-simplegeneric-0.8.1-17.el8.n 279 kB/s | 19 kB 00:00 2024-04-02T07:32:43.306 INFO:teuthology.orchestra.run.smithi113.stdout:(121/153): python3-singledispatch-3.4.0.3-18.el 408 kB/s | 24 kB 00:00 2024-04-02T07:32:43.323 INFO:teuthology.orchestra.run.smithi113.stdout:(122/153): python3-tempora-1.14.1-5.el8.noarch. 582 kB/s | 29 kB 00:00 2024-04-02T07:32:43.340 INFO:teuthology.orchestra.run.smithi113.stdout:(123/153): python3-trustme-0.6.0-4.el8.noarch.r 542 kB/s | 27 kB 00:00 2024-04-02T07:32:43.365 INFO:teuthology.orchestra.run.smithi113.stdout:(124/153): python3-waitress-1.4.3-1.el8.noarch. 4.2 MB/s | 255 kB 00:00 2024-04-02T07:32:43.399 INFO:teuthology.orchestra.run.smithi113.stdout:(125/153): python3-webob-1.8.5-1.el8.1.noarch.r 3.2 MB/s | 251 kB 00:00 2024-04-02T07:32:43.424 INFO:teuthology.orchestra.run.smithi113.stdout:(126/153): python3-websocket-client-0.56.0-5.el 731 kB/s | 61 kB 00:00 2024-04-02T07:32:43.449 INFO:teuthology.orchestra.run.smithi113.stdout:(127/153): python3-webtest-2.0.33-1.el8.noarch. 1.0 MB/s | 85 kB 00:00 2024-04-02T07:32:43.475 INFO:teuthology.orchestra.run.smithi113.stdout:(128/153): python3-zc-lockfile-2.0-2.el8.noarch 305 kB/s | 23 kB 00:00 2024-04-02T07:32:43.575 INFO:teuthology.orchestra.run.smithi113.stdout:(129/153): thrift-0.13.0-2.el8.x86_64.rpm 12 MB/s | 1.7 MB 00:00 2024-04-02T07:32:43.755 INFO:teuthology.orchestra.run.smithi089.stdout:(17/153): ceph-osd-18.2.2-1015.gd0e392d9.el8.x8 2.9 MB/s | 18 MB 00:06 2024-04-02T07:32:43.838 INFO:teuthology.orchestra.run.smithi089.stdout:(18/153): python3-ceph-argparse-18.2.2-1015.gd0 538 kB/s | 45 kB 00:00 2024-04-02T07:32:43.847 INFO:teuthology.orchestra.run.smithi113.stdout:(130/153): librados2-18.2.2-1015.gd0e392d9.el8. 9.3 MB/s | 3.7 MB 00:00 2024-04-02T07:32:43.914 INFO:teuthology.orchestra.run.smithi089.stdout:(19/153): python3-ceph-common-18.2.2-1015.gd0e3 1.7 MB/s | 129 kB 00:00 2024-04-02T07:32:44.009 INFO:teuthology.orchestra.run.smithi113.stdout:(131/153): NetworkManager-1.40.16-15.el8.x86_64 5.3 MB/s | 2.3 MB 00:00 2024-04-02T07:32:44.030 INFO:teuthology.orchestra.run.smithi089.stdout:(20/153): python3-cephfs-18.2.2-1015.gd0e392d9. 1.6 MB/s | 197 kB 00:00 2024-04-02T07:32:44.076 INFO:teuthology.orchestra.run.smithi113.stdout:(132/153): librbd1-18.2.2-1015.gd0e392d9.el8.x8 5.9 MB/s | 3.6 MB 00:00 2024-04-02T07:32:44.101 INFO:teuthology.orchestra.run.smithi113.stdout:(133/153): NetworkManager-team-1.40.16-15.el8.x 1.7 MB/s | 161 kB 00:00 2024-04-02T07:32:44.172 INFO:teuthology.orchestra.run.smithi089.stdout:(21/153): python3-rados-18.2.2-1015.gd0e392d9.e 2.6 MB/s | 371 kB 00:00 2024-04-02T07:32:44.202 INFO:teuthology.orchestra.run.smithi113.stdout:(134/153): acl-2.2.53-3.el8.x86_64.rpm 828 kB/s | 83 kB 00:00 2024-04-02T07:32:44.302 INFO:teuthology.orchestra.run.smithi113.stdout:(135/153): elfutils-debuginfod-client-0.190-2.e 764 kB/s | 77 kB 00:00 2024-04-02T07:32:44.331 INFO:teuthology.orchestra.run.smithi089.stdout:(22/153): python3-rbd-18.2.2-1015.gd0e392d9.el8 2.3 MB/s | 364 kB 00:00 2024-04-02T07:32:44.344 INFO:teuthology.orchestra.run.smithi113.stdout:(136/153): NetworkManager-libnm-1.40.16-15.el8. 3.8 MB/s | 1.9 MB 00:00 2024-04-02T07:32:44.406 INFO:teuthology.orchestra.run.smithi089.stdout:(23/153): python3-rgw-18.2.2-1015.gd0e392d9.el8 1.4 MB/s | 106 kB 00:00 2024-04-02T07:32:44.411 INFO:teuthology.orchestra.run.smithi113.stdout:(137/153): elfutils-libelf-0.190-2.el8.x86_64.r 2.1 MB/s | 233 kB 00:00 2024-04-02T07:32:44.445 INFO:teuthology.orchestra.run.smithi113.stdout:(138/153): elfutils-libs-0.190-2.el8.x86_64.rpm 3.0 MB/s | 305 kB 00:00 2024-04-02T07:32:44.470 INFO:teuthology.orchestra.run.smithi113.stdout:(139/153): NetworkManager-tui-1.40.16-15.el8.x8 905 kB/s | 356 kB 00:00 2024-04-02T07:32:44.490 INFO:teuthology.orchestra.run.smithi089.stdout:(24/153): rbd-fuse-18.2.2-1015.gd0e392d9.el8.x8 1.1 MB/s | 95 kB 00:00 2024-04-02T07:32:44.512 INFO:teuthology.orchestra.run.smithi113.stdout:(140/153): iproute-6.2.0-5.el8.x86_64.rpm 8.3 MB/s | 854 kB 00:00 2024-04-02T07:32:44.537 INFO:teuthology.orchestra.run.smithi113.stdout:(141/153): libacl-2.2.53-3.el8.x86_64.rpm 378 kB/s | 35 kB 00:00 2024-04-02T07:32:44.554 INFO:teuthology.orchestra.run.smithi113.stdout:(142/153): libacl-devel-2.2.53-3.el8.x86_64.rpm 1.0 MB/s | 85 kB 00:00 2024-04-02T07:32:44.604 INFO:teuthology.orchestra.run.smithi113.stdout:(143/153): libblkid-2.32.1-46.el8.x86_64.rpm 2.3 MB/s | 221 kB 00:00 2024-04-02T07:32:44.638 INFO:teuthology.orchestra.run.smithi113.stdout:(144/153): libfdisk-2.32.1-46.el8.x86_64.rpm 2.5 MB/s | 254 kB 00:00 2024-04-02T07:32:44.663 INFO:teuthology.orchestra.run.smithi113.stdout:(145/153): libmount-2.32.1-46.el8.x86_64.rpm 2.1 MB/s | 237 kB 00:00 2024-04-02T07:32:44.697 INFO:teuthology.orchestra.run.smithi113.stdout:(146/153): libsmartcols-2.32.1-46.el8.x86_64.rp 1.9 MB/s | 180 kB 00:00 2024-04-02T07:32:44.730 INFO:teuthology.orchestra.run.smithi113.stdout:(147/153): libuuid-2.32.1-46.el8.x86_64.rpm 1.1 MB/s | 99 kB 00:00 2024-04-02T07:32:44.814 INFO:teuthology.orchestra.run.smithi113.stdout:(148/153): openssl-libs-1.1.1k-12.el8.x86_64.rp 13 MB/s | 1.5 MB 00:00 2024-04-02T07:32:44.848 INFO:teuthology.orchestra.run.smithi113.stdout:(149/153): openssl-1.1.1k-12.el8.x86_64.rpm 3.9 MB/s | 738 kB 00:00 2024-04-02T07:32:44.906 INFO:teuthology.orchestra.run.smithi113.stdout:(150/153): util-linux-user-2.32.1-46.el8.x86_64 1.1 MB/s | 103 kB 00:00 2024-04-02T07:32:44.965 INFO:teuthology.orchestra.run.smithi113.stdout:(151/153): util-linux-2.32.1-46.el8.x86_64.rpm 11 MB/s | 2.6 MB 00:00 2024-04-02T07:32:44.990 INFO:teuthology.orchestra.run.smithi113.stdout:(152/153): xfsdump-3.1.8-7.el8.x86_64.rpm 2.3 MB/s | 334 kB 00:00 2024-04-02T07:32:45.050 INFO:teuthology.orchestra.run.smithi113.stdout:(153/153): xfsprogs-5.0.0-12.el8.x86_64.rpm 7.6 MB/s | 1.1 MB 00:00 2024-04-02T07:32:45.053 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-02T07:32:45.053 INFO:teuthology.orchestra.run.smithi113.stdout:Total 17 MB/s | 220 MB 00:13 2024-04-02T07:32:45.174 INFO:teuthology.orchestra.run.smithi089.stdout:(25/153): ceph-test-18.2.2-1015.gd0e392d9.el8.x 9.9 MB/s | 47 MB 00:04 2024-04-02T07:32:45.266 INFO:teuthology.orchestra.run.smithi089.stdout:(26/153): librgw2-18.2.2-1015.gd0e392d9.el8.x86 3.1 MB/s | 6.5 MB 00:02 2024-04-02T07:32:45.291 INFO:teuthology.orchestra.run.smithi089.stdout:(27/153): rbd-nbd-18.2.2-1015.gd0e392d9.el8.x86 1.5 MB/s | 181 kB 00:00 2024-04-02T07:32:45.316 INFO:teuthology.orchestra.run.smithi089.stdout:(28/153): ceph-grafana-dashboards-18.2.2-1015.g 465 kB/s | 23 kB 00:00 2024-04-02T07:32:45.333 INFO:teuthology.orchestra.run.smithi089.stdout:(29/153): ceph-mgr-cephadm-18.2.2-1015.gd0e392d 3.2 MB/s | 138 kB 00:00 2024-04-02T07:32:45.575 INFO:teuthology.orchestra.run.smithi089.stdout:(30/153): rbd-mirror-18.2.2-1015.gd0e392d9.el8. 3.2 MB/s | 3.5 MB 00:01 2024-04-02T07:32:45.676 INFO:teuthology.orchestra.run.smithi089.stdout:(31/153): ceph-mgr-modules-core-18.2.2-1015.gd0 2.4 MB/s | 245 kB 00:00 2024-04-02T07:32:45.717 INFO:teuthology.orchestra.run.smithi089.stdout:(32/153): ceph-mgr-dashboard-18.2.2-1015.gd0e39 4.5 MB/s | 1.8 MB 00:00 2024-04-02T07:32:45.743 INFO:teuthology.orchestra.run.smithi089.stdout:(33/153): ceph-mgr-rook-18.2.2-1015.gd0e392d9.e 738 kB/s | 49 kB 00:00 2024-04-02T07:32:45.776 INFO:teuthology.orchestra.run.smithi089.stdout:(34/153): ceph-prometheus-alerts-18.2.2-1015.gd 252 kB/s | 15 kB 00:00 2024-04-02T07:32:45.843 INFO:teuthology.orchestra.run.smithi089.stdout:(35/153): cephadm-18.2.2-1015.gd0e392d9.el8.noa 2.5 MB/s | 173 kB 00:00 2024-04-02T07:32:45.868 INFO:teuthology.orchestra.run.smithi089.stdout:(36/153): ceph-volume-18.2.2-1015.gd0e392d9.el8 2.0 MB/s | 263 kB 00:00 2024-04-02T07:32:45.902 INFO:teuthology.orchestra.run.smithi089.stdout:(37/153): cephfs-top-18.2.2-1015.gd0e392d9.el8. 310 kB/s | 18 kB 00:00 2024-04-02T07:32:46.019 INFO:teuthology.orchestra.run.smithi089.stdout:(38/153): ceph-mgr-diskprediction-local-18.2.2- 11 MB/s | 7.4 MB 00:00 2024-04-02T07:32:46.153 INFO:teuthology.orchestra.run.smithi089.stdout:(39/153): automake-1.16.1-8.el8.noarch.rpm 2.5 MB/s | 740 kB 00:00 2024-04-02T07:32:46.412 INFO:teuthology.orchestra.run.smithi089.stdout:(40/153): bison-3.0.4-10.el8.x86_64.rpm 1.3 MB/s | 688 kB 00:00 2024-04-02T07:32:46.454 INFO:teuthology.orchestra.run.smithi089.stdout:(41/153): flex-2.6.1-9.el8.x86_64.rpm 736 kB/s | 320 kB 00:00 2024-04-02T07:32:46.479 INFO:teuthology.orchestra.run.smithi089.stdout:(42/153): jq-1.6-8.el8.x86_64.rpm 623 kB/s | 203 kB 00:00 2024-04-02T07:32:46.521 INFO:teuthology.orchestra.run.smithi089.stdout:(43/153): libpq-13.5-1.el8.x86_64.rpm 1.8 MB/s | 198 kB 00:00 2024-04-02T07:32:46.580 INFO:teuthology.orchestra.run.smithi089.stdout:(44/153): librdkafka-1.6.1-1.el8.x86_64.rpm 5.3 MB/s | 677 kB 00:00 2024-04-02T07:32:46.605 INFO:teuthology.orchestra.run.smithi089.stdout:(45/153): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.0 MB/s | 259 kB 00:00 2024-04-02T07:32:46.630 INFO:teuthology.orchestra.run.smithi089.stdout:(46/153): libtool-2.4.6-25.el8.x86_64.rpm 4.6 MB/s | 709 kB 00:00 2024-04-02T07:32:46.731 INFO:teuthology.orchestra.run.smithi089.stdout:(47/153): oniguruma-6.8.2-3.el8.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-04-02T07:32:46.906 INFO:teuthology.orchestra.run.smithi089.stdout:(48/153): postgresql-10.23-2.module_el8+572+929 8.9 MB/s | 1.6 MB 00:00 2024-04-02T07:32:47.014 INFO:teuthology.orchestra.run.smithi089.stdout:(49/153): postgresql-contrib-10.23-2.module_el8 8.2 MB/s | 901 kB 00:00 2024-04-02T07:32:47.019 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-02T07:32:47.148 INFO:teuthology.orchestra.run.smithi089.stdout:(50/153): openblas-0.3.15-4.el8.x86_64.rpm 8.7 MB/s | 4.7 MB 00:00 2024-04-02T07:32:47.357 INFO:teuthology.orchestra.run.smithi089.stdout:(51/153): postgresql-server-10.23-2.module_el8+ 15 MB/s | 5.1 MB 00:00 2024-04-02T07:32:47.481 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-02T07:32:47.481 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-02T07:32:47.482 INFO:teuthology.orchestra.run.smithi089.stdout:(52/153): python3-babel-2.5.1-7.el8.noarch.rpm 14 MB/s | 4.8 MB 00:00 2024-04-02T07:32:47.507 INFO:teuthology.orchestra.run.smithi089.stdout:(53/153): python3-jinja2-2.10.1-4.el8.noarch.rp 3.8 MB/s | 578 kB 00:00 2024-04-02T07:32:47.534 INFO:teuthology.orchestra.run.smithi089.stdout:(54/153): python3-jsonpatch-1.21-2.el8.noarch.r 530 kB/s | 27 kB 00:00 2024-04-02T07:32:47.591 INFO:teuthology.orchestra.run.smithi089.stdout:(55/153): python3-jsonpointer-1.10-11.el8.noarc 239 kB/s | 20 kB 00:00 2024-04-02T07:32:47.616 INFO:teuthology.orchestra.run.smithi089.stdout:(56/153): python3-mako-1.0.6-14.el8.noarch.rpm 1.8 MB/s | 157 kB 00:00 2024-04-02T07:32:47.642 INFO:teuthology.orchestra.run.smithi089.stdout:(57/153): python3-markupsafe-0.23-19.el8.x86_64 773 kB/s | 39 kB 00:00 2024-04-02T07:32:47.701 INFO:teuthology.orchestra.run.smithi089.stdout:(58/153): python3-numpy-f2py-1.14.3-10.el8.x86_ 3.8 MB/s | 225 kB 00:00 2024-04-02T07:32:47.751 INFO:teuthology.orchestra.run.smithi089.stdout:(59/153): python3-prettytable-0.7.2-14.el8.noar 877 kB/s | 44 kB 00:00 2024-04-02T07:32:47.801 INFO:teuthology.orchestra.run.smithi089.stdout:(60/153): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2024-04-02T07:32:47.876 INFO:teuthology.orchestra.run.smithi089.stdout:(61/153): python3-numpy-1.14.3-10.el8.x86_64.rp 14 MB/s | 3.7 MB 00:00 2024-04-02T07:32:47.902 INFO:teuthology.orchestra.run.smithi089.stdout:(62/153): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2024-04-02T07:32:47.927 INFO:teuthology.orchestra.run.smithi089.stdout:(63/153): python3-pyasn1-modules-0.3.7-6.el8.no 2.1 MB/s | 110 kB 00:00 2024-04-02T07:32:47.952 INFO:teuthology.orchestra.run.smithi089.stdout:(64/153): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2024-04-02T07:32:48.044 INFO:teuthology.orchestra.run.smithi089.stdout:(65/153): openblas-threads-0.3.15-4.el8.x86_64. 3.4 MB/s | 4.9 MB 00:01 2024-04-02T07:32:48.078 INFO:teuthology.orchestra.run.smithi089.stdout:(66/153): python3-werkzeug-0.12.2-4.el8.noarch. 3.6 MB/s | 457 kB 00:00 2024-04-02T07:32:48.153 INFO:teuthology.orchestra.run.smithi089.stdout:(67/153): socat-1.7.4.1-1.el8.x86_64.rpm 2.9 MB/s | 323 kB 00:00 2024-04-02T07:32:48.170 INFO:teuthology.orchestra.run.smithi089.stdout:(68/153): uuid-1.6.2-43.el8.x86_64.rpm 696 kB/s | 64 kB 00:00 2024-04-02T07:32:48.204 INFO:teuthology.orchestra.run.smithi089.stdout:(69/153): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-04-02T07:32:48.504 INFO:teuthology.orchestra.run.smithi089.stdout:(70/153): python3-scipy-1.0.0-21.module_el8+762 25 MB/s | 14 MB 00:00 2024-04-02T07:32:48.530 INFO:teuthology.orchestra.run.smithi089.stdout:(71/153): elfutils-libelf-devel-0.190-2.el8.x86 173 kB/s | 62 kB 00:00 2024-04-02T07:32:48.572 INFO:teuthology.orchestra.run.smithi089.stdout:(72/153): fuse-2.9.7-19.el8.x86_64.rpm 227 kB/s | 83 kB 00:00 2024-04-02T07:32:48.664 INFO:teuthology.orchestra.run.smithi089.stdout:(73/153): libaio-devel-0.3.112-1.el8.x86_64.rpm 203 kB/s | 19 kB 00:00 2024-04-02T07:32:48.764 INFO:teuthology.orchestra.run.smithi089.stdout:(74/153): libcom_err-devel-1.45.6-5.el8.x86_64. 387 kB/s | 39 kB 00:00 2024-04-02T07:32:48.806 INFO:teuthology.orchestra.run.smithi089.stdout:(75/153): keyutils-libs-devel-1.5.10-9.el8.x86_ 160 kB/s | 48 kB 00:00 2024-04-02T07:32:48.831 INFO:teuthology.orchestra.run.smithi089.stdout:(76/153): krb5-devel-1.18.2-22.el8.x86_64.rpm 1.8 MB/s | 561 kB 00:00 2024-04-02T07:32:48.923 INFO:teuthology.orchestra.run.smithi089.stdout:(77/153): librabbitmq-0.9.0-5.el8.x86_64.rpm 513 kB/s | 47 kB 00:00 2024-04-02T07:32:49.007 INFO:teuthology.orchestra.run.smithi089.stdout:(78/153): libselinux-devel-2.9-8.el8.x86_64.rpm 2.4 MB/s | 200 kB 00:00 2024-04-02T07:32:49.032 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-02T07:32:49.036 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-02T07:32:49.040 INFO:teuthology.orchestra.run.smithi089.stdout:(79/153): libquadmath-8.5.0-21.el8.x86_64.rpm 736 kB/s | 172 kB 00:00 2024-04-02T07:32:49.066 INFO:teuthology.orchestra.run.smithi089.stdout:(80/153): libgfortran-8.5.0-21.el8.x86_64.rpm 2.1 MB/s | 645 kB 00:00 2024-04-02T07:32:49.099 INFO:teuthology.orchestra.run.smithi089.stdout:(81/153): libsepol-devel-2.9-3.el8.x86_64.rpm 942 kB/s | 87 kB 00:00 2024-04-02T07:32:49.125 INFO:teuthology.orchestra.run.smithi089.stdout:(82/153): libuuid-devel-2.32.1-46.el8.x86_64.rp 1.2 MB/s | 100 kB 00:00 2024-04-02T07:32:49.158 INFO:teuthology.orchestra.run.smithi089.stdout:(83/153): libverto-devel-0.3.2-2.el8.x86_64.rpm 196 kB/s | 18 kB 00:00 2024-04-02T07:32:49.217 INFO:teuthology.orchestra.run.smithi089.stdout:(84/153): libzstd-devel-1.4.4-1.el8.x86_64.rpm 374 kB/s | 44 kB 00:00 2024-04-02T07:32:49.259 INFO:teuthology.orchestra.run.smithi089.stdout:(85/153): pcre2-devel-10.32-3.el8.x86_64.rpm 5.9 MB/s | 605 kB 00:00 2024-04-02T07:32:49.309 INFO:teuthology.orchestra.run.smithi089.stdout:(86/153): pcre2-utf16-10.32-3.el8.x86_64.rpm 2.4 MB/s | 229 kB 00:00 2024-04-02T07:32:49.351 INFO:teuthology.orchestra.run.smithi089.stdout:(87/153): pcre2-utf32-10.32-3.el8.x86_64.rpm 2.3 MB/s | 220 kB 00:00 2024-04-02T07:32:49.401 INFO:teuthology.orchestra.run.smithi089.stdout:(88/153): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.6 MB/s | 249 kB 00:00 2024-04-02T07:32:49.526 INFO:teuthology.orchestra.run.smithi089.stdout:(89/153): openssl-devel-1.1.1k-12.el8.x86_64.rp 7.9 MB/s | 3.2 MB 00:00 2024-04-02T07:32:49.552 INFO:teuthology.orchestra.run.smithi089.stdout:(90/153): python3-jwt-1.6.1-2.el8.noarch.rpm 283 kB/s | 43 kB 00:00 2024-04-02T07:32:49.577 INFO:teuthology.orchestra.run.smithi089.stdout:(91/153): python3-cryptography-3.2.1-7.el8.x86_ 2.8 MB/s | 641 kB 00:00 2024-04-02T07:32:49.619 INFO:teuthology.orchestra.run.smithi089.stdout:(92/153): python3-oauthlib-2.1.0-1.el8.noarch.r 1.6 MB/s | 155 kB 00:00 2024-04-02T07:32:49.644 INFO:teuthology.orchestra.run.smithi089.stdout:(93/153): python3-pycparser-2.14-14.el8.noarch. 1.2 MB/s | 109 kB 00:00 2024-04-02T07:32:49.670 INFO:teuthology.orchestra.run.smithi089.stdout:(94/153): python3-requests-oauthlib-1.0.0-1.el8 462 kB/s | 43 kB 00:00 2024-04-02T07:32:49.712 INFO:teuthology.orchestra.run.smithi089.stdout:(95/153): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 804 kB/s | 74 kB 00:00 2024-04-02T07:32:49.737 INFO:teuthology.orchestra.run.smithi089.stdout:(96/153): zlib-devel-1.2.11-21.el8.x86_64.rpm 630 kB/s | 58 kB 00:00 2024-04-02T07:32:49.854 INFO:teuthology.orchestra.run.smithi089.stdout:(97/153): python3-packaging-16.8-9.el8.noarch.r 262 kB/s | 48 kB 00:00 2024-04-02T07:32:49.904 INFO:teuthology.orchestra.run.smithi089.stdout:(98/153): liboath-2.6.2-3.el8.x86_64.rpm 363 kB/s | 59 kB 00:00 2024-04-02T07:32:49.946 INFO:teuthology.orchestra.run.smithi089.stdout:(99/153): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2024-04-02T07:32:50.005 INFO:teuthology.orchestra.run.smithi089.stdout:(100/153): python3-bcrypt-3.1.6-2.el8.1.x86_64. 752 kB/s | 44 kB 00:00 2024-04-02T07:32:50.030 INFO:teuthology.orchestra.run.smithi089.stdout:(101/153): libunwind-1.3.1-3.el8.x86_64.rpm 427 kB/s | 75 kB 00:00 2024-04-02T07:32:50.056 INFO:teuthology.orchestra.run.smithi089.stdout:(102/153): python3-beautifulsoup4-4.6.3-2.el8.1 3.6 MB/s | 185 kB 00:00 2024-04-02T07:32:50.114 INFO:teuthology.orchestra.run.smithi089.stdout:(103/153): python3-cachetools-3.1.1-4.el8.noarc 396 kB/s | 33 kB 00:00 2024-04-02T07:32:50.139 INFO:teuthology.orchestra.run.smithi089.stdout:(104/153): python3-certifi-2018.10.15-7.el8.noa 196 kB/s | 16 kB 00:00 2024-04-02T07:32:50.165 INFO:teuthology.orchestra.run.smithi089.stdout:(105/153): python3-asyncssh-2.7.0-2.el8.noarch. 1.8 MB/s | 476 kB 00:00 2024-04-02T07:32:50.207 INFO:teuthology.orchestra.run.smithi089.stdout:(106/153): python3-cherrypy-18.4.0-1.el8.noarch 5.6 MB/s | 384 kB 00:00 2024-04-02T07:32:50.232 INFO:teuthology.orchestra.run.smithi089.stdout:(107/153): python3-cheroot-8.5.2-1.el8.noarch.r 1.4 MB/s | 173 kB 00:00 2024-04-02T07:32:50.249 INFO:teuthology.orchestra.run.smithi089.stdout:(108/153): python3-jaraco-6.2-6.el8.noarch.rpm 266 kB/s | 11 kB 00:00 2024-04-02T07:32:50.274 INFO:teuthology.orchestra.run.smithi089.stdout:(109/153): python3-google-auth-1.1.1-10.el8.noa 746 kB/s | 81 kB 00:00 2024-04-02T07:32:50.291 INFO:teuthology.orchestra.run.smithi089.stdout:(110/153): python3-jaraco-functools-2.0-4.el8.n 300 kB/s | 18 kB 00:00 2024-04-02T07:32:50.375 INFO:teuthology.orchestra.run.smithi089.stdout:(111/153): python3-more-itertools-7.2.0-3.el8.n 707 kB/s | 59 kB 00:00 2024-04-02T07:32:50.408 INFO:teuthology.orchestra.run.smithi089.stdout:(112/153): python3-logutils-0.3.5-11.el8.noarch 364 kB/s | 49 kB 00:00 2024-04-02T07:32:50.450 INFO:teuthology.orchestra.run.smithi089.stdout:(113/153): python3-kubernetes-11.0.0-6.el8.noar 5.8 MB/s | 1.2 MB 00:00 2024-04-02T07:32:50.476 INFO:teuthology.orchestra.run.smithi089.stdout:(114/153): python3-natsort-7.1.1-2.el8.noarch.r 600 kB/s | 60 kB 00:00 2024-04-02T07:32:50.534 INFO:teuthology.orchestra.run.smithi089.stdout:(115/153): python3-repoze-lru-0.7-6.el8.noarch. 577 kB/s | 34 kB 00:00 2024-04-02T07:32:50.585 INFO:teuthology.orchestra.run.smithi089.stdout:(116/153): python3-routes-2.4.1-12.el8.noarch.r 3.8 MB/s | 196 kB 00:00 2024-04-02T07:32:50.610 INFO:teuthology.orchestra.run.smithi089.stdout:(117/153): python3-portend-2.6-1.el8.noarch.rpm 98 kB/s | 16 kB 00:00 2024-04-02T07:32:50.644 INFO:teuthology.orchestra.run.smithi089.stdout:(118/153): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2024-04-02T07:32:50.669 INFO:teuthology.orchestra.run.smithi089.stdout:(119/153): python3-rsa-4.9-2.el8.noarch.rpm 769 kB/s | 64 kB 00:00 2024-04-02T07:32:50.694 INFO:teuthology.orchestra.run.smithi089.stdout:(120/153): python3-simplegeneric-0.8.1-17.el8.n 224 kB/s | 19 kB 00:00 2024-04-02T07:32:50.719 INFO:teuthology.orchestra.run.smithi089.stdout:(121/153): python3-singledispatch-3.4.0.3-18.el 319 kB/s | 24 kB 00:00 2024-04-02T07:32:50.761 INFO:teuthology.orchestra.run.smithi089.stdout:(122/153): python3-tempora-1.14.1-5.el8.noarch. 439 kB/s | 29 kB 00:00 2024-04-02T07:32:50.786 INFO:teuthology.orchestra.run.smithi089.stdout:(123/153): python3-trustme-0.6.0-4.el8.noarch.r 299 kB/s | 27 kB 00:00 2024-04-02T07:32:50.804 INFO:teuthology.orchestra.run.smithi089.stdout:(124/153): python3-waitress-1.4.3-1.el8.noarch. 3.0 MB/s | 255 kB 00:00 2024-04-02T07:32:50.829 INFO:teuthology.orchestra.run.smithi089.stdout:(125/153): python3-websocket-client-0.56.0-5.el 1.4 MB/s | 61 kB 00:00 2024-04-02T07:32:50.854 INFO:teuthology.orchestra.run.smithi089.stdout:(126/153): python3-webtest-2.0.33-1.el8.noarch. 1.7 MB/s | 85 kB 00:00 2024-04-02T07:32:50.879 INFO:teuthology.orchestra.run.smithi089.stdout:(127/153): python3-webob-1.8.5-1.el8.1.noarch.r 2.1 MB/s | 251 kB 00:00 2024-04-02T07:32:50.905 INFO:teuthology.orchestra.run.smithi089.stdout:(128/153): python3-zc-lockfile-2.0-2.el8.noarch 304 kB/s | 23 kB 00:00 2024-04-02T07:32:51.197 INFO:teuthology.orchestra.run.smithi089.stdout:(129/153): thrift-0.13.0-2.el8.x86_64.rpm 5.1 MB/s | 1.7 MB 00:00 2024-04-02T07:32:51.739 INFO:teuthology.orchestra.run.smithi089.stdout:(130/153): NetworkManager-1.40.16-15.el8.x86_64 4.2 MB/s | 2.3 MB 00:00 2024-04-02T07:32:51.893 INFO:teuthology.orchestra.run.smithi089.stdout:(131/153): NetworkManager-libnm-1.40.16-15.el8. 12 MB/s | 1.9 MB 00:00 2024-04-02T07:32:51.989 INFO:teuthology.orchestra.run.smithi089.stdout:(132/153): NetworkManager-team-1.40.16-15.el8.x 1.6 MB/s | 161 kB 00:00 2024-04-02T07:32:52.089 INFO:teuthology.orchestra.run.smithi089.stdout:(133/153): NetworkManager-tui-1.40.16-15.el8.x8 3.5 MB/s | 356 kB 00:00 2024-04-02T07:32:52.181 INFO:teuthology.orchestra.run.smithi089.stdout:(134/153): acl-2.2.53-3.el8.x86_64.rpm 905 kB/s | 83 kB 00:00 2024-04-02T07:32:52.273 INFO:teuthology.orchestra.run.smithi089.stdout:(135/153): elfutils-debuginfod-client-0.190-2.e 838 kB/s | 77 kB 00:00 2024-04-02T07:32:52.332 INFO:teuthology.orchestra.run.smithi089.stdout:(136/153): librbd1-18.2.2-1015.gd0e392d9.el8.x8 2.5 MB/s | 3.6 MB 00:01 2024-04-02T07:32:52.391 INFO:teuthology.orchestra.run.smithi089.stdout:(137/153): librados2-18.2.2-1015.gd0e392d9.el8. 2.4 MB/s | 3.7 MB 00:01 2024-04-02T07:32:52.416 INFO:teuthology.orchestra.run.smithi089.stdout:(138/153): elfutils-libelf-0.190-2.el8.x86_64.r 1.6 MB/s | 233 kB 00:00 2024-04-02T07:32:52.508 INFO:teuthology.orchestra.run.smithi089.stdout:(139/153): libacl-2.2.53-3.el8.x86_64.rpm 381 kB/s | 35 kB 00:00 2024-04-02T07:32:52.600 INFO:teuthology.orchestra.run.smithi089.stdout:(140/153): libacl-devel-2.2.53-3.el8.x86_64.rpm 925 kB/s | 85 kB 00:00 2024-04-02T07:32:52.633 INFO:teuthology.orchestra.run.smithi089.stdout:(141/153): elfutils-libs-0.190-2.el8.x86_64.rpm 1.0 MB/s | 305 kB 00:00 2024-04-02T07:32:52.700 INFO:teuthology.orchestra.run.smithi089.stdout:(142/153): libblkid-2.32.1-46.el8.x86_64.rpm 2.2 MB/s | 221 kB 00:00 2024-04-02T07:32:52.726 INFO:teuthology.orchestra.run.smithi089.stdout:(143/153): libfdisk-2.32.1-46.el8.x86_64.rpm 2.7 MB/s | 254 kB 00:00 2024-04-02T07:32:52.776 INFO:teuthology.orchestra.run.smithi089.stdout:(144/153): iproute-6.2.0-5.el8.x86_64.rpm 2.2 MB/s | 854 kB 00:00 2024-04-02T07:32:52.801 INFO:teuthology.orchestra.run.smithi089.stdout:(145/153): libmount-2.32.1-46.el8.x86_64.rpm 2.3 MB/s | 237 kB 00:00 2024-04-02T07:32:52.843 INFO:teuthology.orchestra.run.smithi089.stdout:(146/153): libsmartcols-2.32.1-46.el8.x86_64.rp 1.5 MB/s | 180 kB 00:00 2024-04-02T07:32:52.869 INFO:teuthology.orchestra.run.smithi089.stdout:(147/153): libuuid-2.32.1-46.el8.x86_64.rpm 1.1 MB/s | 99 kB 00:00 2024-04-02T07:32:52.910 INFO:teuthology.orchestra.run.smithi089.stdout:(148/153): openssl-1.1.1k-12.el8.x86_64.rpm 6.6 MB/s | 738 kB 00:00 2024-04-02T07:32:53.002 INFO:teuthology.orchestra.run.smithi089.stdout:(149/153): util-linux-user-2.32.1-46.el8.x86_64 1.1 MB/s | 103 kB 00:00 2024-04-02T07:32:53.045 INFO:teuthology.orchestra.run.smithi089.stdout:(150/153): openssl-libs-1.1.1k-12.el8.x86_64.rp 7.3 MB/s | 1.5 MB 00:00 2024-04-02T07:32:53.103 INFO:teuthology.orchestra.run.smithi089.stdout:(151/153): xfsdump-3.1.8-7.el8.x86_64.rpm 3.3 MB/s | 334 kB 00:00 2024-04-02T07:32:53.153 INFO:teuthology.orchestra.run.smithi089.stdout:(152/153): util-linux-2.32.1-46.el8.x86_64.rpm 9.1 MB/s | 2.6 MB 00:00 2024-04-02T07:32:53.187 INFO:teuthology.orchestra.run.smithi089.stdout:(153/153): xfsprogs-5.0.0-12.el8.x86_64.rpm 7.7 MB/s | 1.1 MB 00:00 2024-04-02T07:32:53.192 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-02T07:32:53.193 INFO:teuthology.orchestra.run.smithi089.stdout:Total 10 MB/s | 220 MB 00:21 2024-04-02T07:32:55.004 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-02T07:32:55.150 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-02T07:32:55.226 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8.x86_64 1/1 2024-04-02T07:32:55.430 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8.x86_64 1/177 2024-04-02T07:32:55.612 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-02T07:32:55.612 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-02T07:32:55.691 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8.x86_64 1/177 2024-04-02T07:32:55.896 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/177 2024-04-02T07:32:56.059 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libuuid-2.32.1-46.el8.x86_64 3/177 2024-04-02T07:32:56.176 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libuuid-2.32.1-46.el8.x86_64 3/177 2024-04-02T07:32:56.601 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libblkid-2.32.1-46.el8.x86_64 4/177 2024-04-02T07:32:56.700 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libblkid-2.32.1-46.el8.x86_64 4/177 2024-04-02T07:32:56.924 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/177 2024-04-02T07:32:57.124 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 6/177 2024-04-02T07:32:57.208 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-02T07:32:57.212 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-02T07:32:57.332 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/177 2024-04-02T07:32:57.628 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-libelf-0.190-2.el8.x86_64 8/177 2024-04-02T07:32:57.859 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 9/177 2024-04-02T07:32:58.321 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 9/177 2024-04-02T07:32:58.509 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 10/177 2024-04-02T07:32:58.946 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 10/177 2024-04-02T07:32:59.125 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 11/177 2024-04-02T07:32:59.259 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 11/177 2024-04-02T07:32:59.450 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 12/177 2024-04-02T07:32:59.561 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 12/177 2024-04-02T07:32:59.898 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rados-2:18.2.2-1015.gd0e392d9.el8.x86_64 13/177 2024-04-02T07:33:00.142 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/177 2024-04-02T07:33:00.243 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/177 2024-04-02T07:33:00.447 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libpq-13.5-1.el8.x86_64 15/177 2024-04-02T07:33:00.681 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 16/177 2024-04-02T07:33:01.098 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 17/177 2024-04-02T07:33:01.463 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 18/177 2024-04-02T07:33:01.666 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 19/177 2024-04-02T07:33:01.787 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 19/177 2024-04-02T07:33:01.983 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86 20/177 2024-04-02T07:33:02.062 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86 20/177 2024-04-02T07:33:02.575 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libacl-2.2.53-3.el8.x86_64 21/177 2024-04-02T07:33:02.883 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-libnm-1:1.40.16-15.el8.x86_64 22/177 2024-04-02T07:33:02.955 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-libnm-1:1.40.16-15.el8.x86_64 22/177 2024-04-02T07:33:03.110 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-02T07:33:03.233 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-04-02T07:33:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8.x86_64 1/1 2024-04-02T07:33:03.496 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-04-02T07:33:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8.x86_64 1/177 2024-04-02T07:33:03.773 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8.x86_64 1/177 2024-04-02T07:33:03.967 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/177 2024-04-02T07:33:03.969 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-04-02T07:33:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libuuid-2.32.1-46.el8.x86_64 3/177 2024-04-02T07:33:04.198 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 24/177 2024-04-02T07:33:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libuuid-2.32.1-46.el8.x86_64 3/177 2024-04-02T07:33:04.398 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 25/177 2024-04-02T07:33:04.447 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libblkid-2.32.1-46.el8.x86_64 4/177 2024-04-02T07:33:04.547 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libblkid-2.32.1-46.el8.x86_64 4/177 2024-04-02T07:33:04.591 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 26/177 2024-04-02T07:33:04.746 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 26/177 2024-04-02T07:33:04.820 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/177 2024-04-02T07:33:04.965 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 27/177 2024-04-02T07:33:04.995 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 6/177 2024-04-02T07:33:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/177 2024-04-02T07:33:05.464 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : elfutils-libelf-0.190-2.el8.x86_64 8/177 2024-04-02T07:33:05.619 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 27/177 2024-04-02T07:33:05.663 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 9/177 2024-04-02T07:33:05.832 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-0.3.15-4.el8.x86_64 28/177 2024-04-02T07:33:06.118 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 9/177 2024-04-02T07:33:06.313 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 10/177 2024-04-02T07:33:06.539 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 28/177 2024-04-02T07:33:06.736 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 10/177 2024-04-02T07:33:06.757 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 29/177 2024-04-02T07:33:06.954 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 11/177 2024-04-02T07:33:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 11/177 2024-04-02T07:33:07.263 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 12/177 2024-04-02T07:33:07.403 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 12/177 2024-04-02T07:33:07.553 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 29/177 2024-04-02T07:33:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rados-2:18.2.2-1015.gd0e392d9.el8.x86_64 13/177 2024-04-02T07:33:07.929 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 30/177 2024-04-02T07:33:07.974 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/177 2024-04-02T07:33:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/177 2024-04-02T07:33:08.156 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 31/177 2024-04-02T07:33:08.304 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libpq-13.5-1.el8.x86_64 15/177 2024-04-02T07:33:08.372 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 32/177 2024-04-02T07:33:08.535 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 16/177 2024-04-02T07:33:08.591 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 33/177 2024-04-02T07:33:08.855 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-common-2:18.2.2-1015.gd0e392d9.el8. 34/177 2024-04-02T07:33:08.958 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 17/177 2024-04-02T07:33:09.028 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-argparse-2:18.2.2-1015.gd0e392d9.el 35/177 2024-04-02T07:33:09.218 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 18/177 2024-04-02T07:33:09.280 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cephfs-2:18.2.2-1015.gd0e392d9.el8.x86_6 36/177 2024-04-02T07:33:09.396 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 19/177 2024-04-02T07:33:09.520 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 19/177 2024-04-02T07:33:09.696 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86 20/177 2024-04-02T07:33:09.779 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86 20/177 2024-04-02T07:33:10.263 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libacl-2.2.53-3.el8.x86_64 21/177 2024-04-02T07:33:10.405 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 37/177 2024-04-02T07:33:10.504 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-libnm-1:1.40.16-15.el8.x86_64 22/177 2024-04-02T07:33:10.564 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-libnm-1:1.40.16-15.el8.x86_64 22/177 2024-04-02T07:33:10.834 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 38/177 2024-04-02T07:33:10.840 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-04-02T07:33:11.104 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-04-02T07:33:11.172 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 39/177 2024-04-02T07:33:11.381 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 40/177 2024-04-02T07:33:11.561 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-04-02T07:33:11.785 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 24/177 2024-04-02T07:33:11.969 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 25/177 2024-04-02T07:33:12.137 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 26/177 2024-04-02T07:33:12.296 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 26/177 2024-04-02T07:33:12.462 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 27/177 2024-04-02T07:33:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 27/177 2024-04-02T07:33:13.304 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-0.3.15-4.el8.x86_64 28/177 2024-04-02T07:33:13.398 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 40/177 2024-04-02T07:33:13.733 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 41/177 2024-04-02T07:33:13.953 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 42/177 2024-04-02T07:33:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 28/177 2024-04-02T07:33:14.195 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 29/177 2024-04-02T07:33:14.226 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rsa-4.9-2.el8.noarch 43/177 2024-04-02T07:33:14.445 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 44/177 2024-04-02T07:33:14.655 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librados-devel-2:18.2.2-1015.gd0e392d9.el8.x86_6 45/177 2024-04-02T07:33:14.878 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-libs-0.190-2.el8.x86_64 46/177 2024-04-02T07:33:15.008 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 29/177 2024-04-02T07:33:15.105 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el8.x86_64 47/177 2024-04-02T07:33:15.353 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 30/177 2024-04-02T07:33:15.372 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libfdisk-2.32.1-46.el8.x86_64 48/177 2024-04-02T07:33:15.463 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libfdisk-2.32.1-46.el8.x86_64 48/177 2024-04-02T07:33:15.578 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 31/177 2024-04-02T07:33:15.688 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libmount-2.32.1-46.el8.x86_64 49/177 2024-04-02T07:33:15.743 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 32/177 2024-04-02T07:33:15.755 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libmount-2.32.1-46.el8.x86_64 49/177 2024-04-02T07:33:15.956 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 33/177 2024-04-02T07:33:15.956 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libuuid-devel-2.32.1-46.el8.x86_64 50/177 2024-04-02T07:33:16.161 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-common-2:18.2.2-1015.gd0e392d9.el8. 34/177 2024-04-02T07:33:16.172 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 51/177 2024-04-02T07:33:16.299 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-argparse-2:18.2.2-1015.gd0e392d9.el 35/177 2024-04-02T07:33:16.363 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libsmartcols-2.32.1-46.el8.x86_64 52/177 2024-04-02T07:33:16.566 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cephfs-2:18.2.2-1015.gd0e392d9.el8.x86_6 36/177 2024-04-02T07:33:16.926 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libsmartcols-2.32.1-46.el8.x86_64 52/177 2024-04-02T07:33:17.222 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : util-linux-2.32.1-46.el8.x86_64 53/177 2024-04-02T07:33:17.249 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: util-linux-2.32.1-46.el8.x86_64 53/177 2024-04-02T07:33:17.478 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 54/177 2024-04-02T07:33:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 37/177 2024-04-02T07:33:17.776 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 55/177 2024-04-02T07:33:17.995 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 56/177 2024-04-02T07:33:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 38/177 2024-04-02T07:33:18.176 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 57/177 2024-04-02T07:33:18.338 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 39/177 2024-04-02T07:33:18.350 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 58/177 2024-04-02T07:33:18.544 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 40/177 2024-04-02T07:33:18.575 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 59/177 2024-04-02T07:33:18.836 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 60/177 2024-04-02T07:33:19.067 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 61/177 2024-04-02T07:33:19.308 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 62/177 2024-04-02T07:33:19.503 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 63/177 2024-04-02T07:33:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 64/177 2024-04-02T07:33:19.970 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 65/177 2024-04-02T07:33:20.173 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-portend-2.6-1.el8.noarch 66/177 2024-04-02T07:33:20.373 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 67/177 2024-04-02T07:33:20.566 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 40/177 2024-04-02T07:33:20.646 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 68/177 2024-04-02T07:33:20.871 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 41/177 2024-04-02T07:33:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 69/177 2024-04-02T07:33:21.092 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 42/177 2024-04-02T07:33:21.186 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 70/177 2024-04-02T07:33:21.298 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rsa-4.9-2.el8.noarch 43/177 2024-04-02T07:33:21.484 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 44/177 2024-04-02T07:33:21.553 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 71/177 2024-04-02T07:33:21.660 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librados-devel-2:18.2.2-1015.gd0e392d9.el8.x86_6 45/177 2024-04-02T07:33:21.840 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 72/177 2024-04-02T07:33:21.851 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : elfutils-libs-0.190-2.el8.x86_64 46/177 2024-04-02T07:33:22.088 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 73/177 2024-04-02T07:33:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el8.x86_64 47/177 2024-04-02T07:33:22.710 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libfdisk-2.32.1-46.el8.x86_64 48/177 2024-04-02T07:33:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libfdisk-2.32.1-46.el8.x86_64 48/177 2024-04-02T07:33:22.933 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 74/177 2024-04-02T07:33:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libmount-2.32.1-46.el8.x86_64 49/177 2024-04-02T07:33:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libmount-2.32.1-46.el8.x86_64 49/177 2024-04-02T07:33:23.170 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 75/177 2024-04-02T07:33:23.260 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 75/177 2024-04-02T07:33:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libuuid-devel-2.32.1-46.el8.x86_64 50/177 2024-04-02T07:33:23.453 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 51/177 2024-04-02T07:33:23.517 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 76/177 2024-04-02T07:33:23.627 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libsmartcols-2.32.1-46.el8.x86_64 52/177 2024-04-02T07:33:23.772 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-packaging-16.8-9.el8.noarch 77/177 2024-04-02T07:33:24.012 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 78/177 2024-04-02T07:33:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libsmartcols-2.32.1-46.el8.x86_64 52/177 2024-04-02T07:33:24.445 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 79/177 2024-04-02T07:33:24.494 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : util-linux-2.32.1-46.el8.x86_64 53/177 2024-04-02T07:33:24.522 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: util-linux-2.32.1-46.el8.x86_64 53/177 2024-04-02T07:33:24.742 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 54/177 2024-04-02T07:33:24.759 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 80/177 2024-04-02T07:33:24.998 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 55/177 2024-04-02T07:33:25.006 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 81/177 2024-04-02T07:33:25.209 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 56/177 2024-04-02T07:33:25.305 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 82/177 2024-04-02T07:33:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 57/177 2024-04-02T07:33:25.522 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 58/177 2024-04-02T07:33:25.553 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 83/177 2024-04-02T07:33:25.747 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 59/177 2024-04-02T07:33:25.949 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 60/177 2024-04-02T07:33:26.181 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 61/177 2024-04-02T07:33:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 62/177 2024-04-02T07:33:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 63/177 2024-04-02T07:33:26.681 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 64/177 2024-04-02T07:33:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 65/177 2024-04-02T07:33:26.850 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 84/177 2024-04-02T07:33:27.019 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-portend-2.6-1.el8.noarch 66/177 2024-04-02T07:33:27.194 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 67/177 2024-04-02T07:33:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 68/177 2024-04-02T07:33:27.664 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 69/177 2024-04-02T07:33:27.826 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 85/177 2024-04-02T07:33:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 70/177 2024-04-02T07:33:28.045 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 86/177 2024-04-02T07:33:28.240 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 71/177 2024-04-02T07:33:28.362 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 87/177 2024-04-02T07:33:28.487 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 72/177 2024-04-02T07:33:28.689 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 88/177 2024-04-02T07:33:28.705 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 73/177 2024-04-02T07:33:28.946 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 89/177 2024-04-02T07:33:29.214 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 90/177 2024-04-02T07:33:29.482 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 91/177 2024-04-02T07:33:29.498 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 74/177 2024-04-02T07:33:29.718 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 75/177 2024-04-02T07:33:29.807 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 75/177 2024-04-02T07:33:29.838 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 92/177 2024-04-02T07:33:30.016 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 76/177 2024-04-02T07:33:30.132 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 93/177 2024-04-02T07:33:30.286 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-packaging-16.8-9.el8.noarch 77/177 2024-04-02T07:33:30.329 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 94/177 2024-04-02T07:33:30.510 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 78/177 2024-04-02T07:33:30.577 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 95/177 2024-04-02T07:33:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 79/177 2024-04-02T07:33:30.934 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 96/177 2024-04-02T07:33:31.148 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libselinux-devel-2.9-8.el8.x86_64 97/177 2024-04-02T07:33:31.224 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 80/177 2024-04-02T07:33:31.324 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 98/177 2024-04-02T07:33:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 81/177 2024-04-02T07:33:31.564 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 99/177 2024-04-02T07:33:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 82/177 2024-04-02T07:33:31.837 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 100/177 2024-04-02T07:33:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 83/177 2024-04-02T07:33:32.087 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : fuse-2.9.7-19.el8.x86_64 101/177 2024-04-02T07:33:32.337 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 102/177 2024-04-02T07:33:32.535 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : uuid-1.6.2-43.el8.x86_64 103/177 2024-04-02T07:33:33.051 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 103/177 2024-04-02T07:33:33.216 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 84/177 2024-04-02T07:33:33.227 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 104/177 2024-04-02T07:33:33.590 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 85/177 2024-04-02T07:33:33.793 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 86/177 2024-04-02T07:33:34.068 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 87/177 2024-04-02T07:33:34.397 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 88/177 2024-04-02T07:33:34.662 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 89/177 2024-04-02T07:33:34.930 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 90/177 2024-04-02T07:33:35.130 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 91/177 2024-04-02T07:33:35.435 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 92/177 2024-04-02T07:33:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 93/177 2024-04-02T07:33:35.836 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 94/177 2024-04-02T07:33:35.957 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 105/177 2024-04-02T07:33:36.035 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 95/177 2024-04-02T07:33:36.252 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 105/177 2024-04-02T07:33:36.342 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 96/177 2024-04-02T07:33:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libselinux-devel-2.9-8.el8.x86_64 97/177 2024-04-02T07:33:36.698 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 98/177 2024-04-02T07:33:36.897 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 99/177 2024-04-02T07:33:36.958 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 105/177 2024-04-02T07:33:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 100/177 2024-04-02T07:33:37.305 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 106/177 2024-04-02T07:33:37.369 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fuse-2.9.7-19.el8.x86_64 101/177 2024-04-02T07:33:37.495 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 106/177 2024-04-02T07:33:37.602 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 102/177 2024-04-02T07:33:37.783 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : uuid-1.6.2-43.el8.x86_64 103/177 2024-04-02T07:33:37.854 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 103/177 2024-04-02T07:33:38.177 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 107/177 2024-04-02T07:33:38.384 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 104/177 2024-04-02T07:33:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 105/177 2024-04-02T07:33:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 105/177 2024-04-02T07:33:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 105/177 2024-04-02T07:33:42.465 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 106/177 2024-04-02T07:33:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 106/177 2024-04-02T07:33:42.882 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 107/177 2024-04-02T07:33:59.058 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 107/177 2024-04-02T07:33:59.058 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /sys 2024-04-02T07:33:59.058 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /proc 2024-04-02T07:33:59.058 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /dev 2024-04-02T07:33:59.059 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /run 2024-04-02T07:33:59.059 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /mnt 2024-04-02T07:33:59.059 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /var/tmp 2024-04-02T07:33:59.059 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /home 2024-04-02T07:33:59.059 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /tmp 2024-04-02T07:33:59.059 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /dev 2024-04-02T07:33:59.059 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:33:59.454 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 108/177 2024-04-02T07:34:01.198 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 108/177 2024-04-02T07:34:01.458 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 109/177 2024-04-02T07:34:01.721 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 109/177 2024-04-02T07:34:02.003 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 110/177 2024-04-02T07:34:02.538 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 110/177 2024-04-02T07:34:02.778 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 111/177 2024-04-02T07:34:02.827 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 111/177 2024-04-02T07:34:03.086 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 112/177 2024-04-02T07:34:03.316 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 113/177 2024-04-02T07:34:03.494 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 114/177 2024-04-02T07:34:03.588 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 114/177 2024-04-02T07:34:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 107/177 2024-04-02T07:34:03.644 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /sys 2024-04-02T07:34:03.645 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /proc 2024-04-02T07:34:03.645 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /dev 2024-04-02T07:34:03.645 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /run 2024-04-02T07:34:03.645 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /mnt 2024-04-02T07:34:03.645 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /var/tmp 2024-04-02T07:34:03.645 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /home 2024-04-02T07:34:03.645 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /tmp 2024-04-02T07:34:03.645 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /dev 2024-04-02T07:34:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:03.946 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : jq-1.6-8.el8.x86_64 115/177 2024-04-02T07:34:04.005 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 108/177 2024-04-02T07:34:04.311 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : automake-1.16.1-8.el8.noarch 116/177 2024-04-02T07:34:04.939 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 117/177 2024-04-02T07:34:05.168 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 117/177 2024-04-02T07:34:05.348 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1015.gd0e392d9.e 118/177 2024-04-02T07:34:05.592 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1015.gd0e392d9. 119/177 2024-04-02T07:34:05.753 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 108/177 2024-04-02T07:34:05.852 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noa 120/177 2024-04-02T07:34:06.012 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 109/177 2024-04-02T07:34:06.112 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noa 120/177 2024-04-02T07:34:06.274 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 109/177 2024-04-02T07:34:06.398 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.n 121/177 2024-04-02T07:34:06.454 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 110/177 2024-04-02T07:34:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 110/177 2024-04-02T07:34:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 111/177 2024-04-02T07:34:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 111/177 2024-04-02T07:34:07.438 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 112/177 2024-04-02T07:34:07.472 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.n 121/177 2024-04-02T07:34:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 113/177 2024-04-02T07:34:07.710 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e 122/177 2024-04-02T07:34:07.812 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 114/177 2024-04-02T07:34:07.837 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e 122/177 2024-04-02T07:34:07.891 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 114/177 2024-04-02T07:34:08.214 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : jq-1.6-8.el8.x86_64 115/177 2024-04-02T07:34:08.287 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1015.gd0e392d9.el 123/177 2024-04-02T07:34:08.437 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : automake-1.16.1-8.el8.noarch 116/177 2024-04-02T07:34:08.523 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 124/177 2024-04-02T07:34:08.589 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 124/177 2024-04-02T07:34:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 125/177 2024-04-02T07:34:08.819 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 125/177 2024-04-02T07:34:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 117/177 2024-04-02T07:34:09.082 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-2:18.2.2-1015.gd0e392d9.el8.x86_64 126/177 2024-04-02T07:34:09.256 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 117/177 2024-04-02T07:34:09.285 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libtool-2.4.6-25.el8.x86_64 127/177 2024-04-02T07:34:09.418 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1015.gd0e392d9.e 118/177 2024-04-02T07:34:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1015.gd0e392d9. 119/177 2024-04-02T07:34:09.837 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noa 120/177 2024-04-02T07:34:10.097 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noa 120/177 2024-04-02T07:34:10.310 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.n 121/177 2024-04-02T07:34:11.387 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.n 121/177 2024-04-02T07:34:11.553 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e 122/177 2024-04-02T07:34:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e 122/177 2024-04-02T07:34:12.091 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1015.gd0e392d9.el 123/177 2024-04-02T07:34:12.320 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 124/177 2024-04-02T07:34:12.385 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 124/177 2024-04-02T07:34:12.569 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 125/177 2024-04-02T07:34:12.596 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 125/177 2024-04-02T07:34:12.852 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-2:18.2.2-1015.gd0e392d9.el8.x86_64 126/177 2024-04-02T07:34:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libtool-2.4.6-25.el8.x86_64 127/177 2024-04-02T07:34:14.106 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 127/177 2024-04-02T07:34:15.621 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-test-2:18.2.2-1015.gd0e392d9.el8.x86_64 128/177 2024-04-02T07:34:15.922 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 129/177 2024-04-02T07:34:15.980 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 129/177 2024-04-02T07:34:16.250 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1015.gd0e39 130/177 2024-04-02T07:34:16.318 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1015.gd0e39 130/177 2024-04-02T07:34:16.583 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 131/177 2024-04-02T07:34:16.982 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 131/177 2024-04-02T07:34:17.259 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 132/177 2024-04-02T07:34:17.539 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 132/177 2024-04-02T07:34:17.842 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 127/177 2024-04-02T07:34:17.903 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-contrib-10.23-2.module_el8+572+929c87 133/177 2024-04-02T07:34:19.199 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 134/177 2024-04-02T07:34:19.379 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-test-2:18.2.2-1015.gd0e392d9.el8.x86_64 128/177 2024-04-02T07:34:19.572 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openssl-devel-1:1.1.1k-12.el8.x86_64 135/177 2024-04-02T07:34:19.627 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 129/177 2024-04-02T07:34:19.686 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 129/177 2024-04-02T07:34:19.773 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : elfutils-libelf-devel-0.190-2.el8.x86_64 136/177 2024-04-02T07:34:19.902 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1015.gd0e39 130/177 2024-04-02T07:34:19.971 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1015.gd0e39 130/177 2024-04-02T07:34:19.988 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : util-linux-user-2.32.1-46.el8.x86_64 137/177 2024-04-02T07:34:20.179 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 138/177 2024-04-02T07:34:20.185 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 131/177 2024-04-02T07:34:20.377 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs-devel-2:18.2.2-1015.gd0e392d9.el8.x86_ 139/177 2024-04-02T07:34:20.584 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 131/177 2024-04-02T07:34:20.631 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-team-1:1.40.16-15.el8.x86_64 140/177 2024-04-02T07:34:20.819 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 132/177 2024-04-02T07:34:20.851 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-tui-1:1.40.16-15.el8.x86_64 141/177 2024-04-02T07:34:21.104 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 132/177 2024-04-02T07:34:21.118 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : acl-2.2.53-3.el8.x86_64 142/177 2024-04-02T07:34:21.300 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libacl-devel-2.2.53-3.el8.x86_64 143/177 2024-04-02T07:34:21.439 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-contrib-10.23-2.module_el8+572+929c87 133/177 2024-04-02T07:34:22.481 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-04-02T07:34:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 134/177 2024-04-02T07:34:22.766 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-04-02T07:34:22.856 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-04-02T07:34:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openssl-devel-1:1.1.1k-12.el8.x86_64 135/177 2024-04-02T07:34:23.120 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : xfsdump-3.1.8-7.el8.x86_64 145/177 2024-04-02T07:34:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : elfutils-libelf-devel-0.190-2.el8.x86_64 136/177 2024-04-02T07:34:23.324 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephfs-top-2:18.2.2-1015.gd0e392d9.el8.noarch 146/177 2024-04-02T07:34:23.382 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : util-linux-user-2.32.1-46.el8.x86_64 137/177 2024-04-02T07:34:23.574 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 138/177 2024-04-02T07:34:23.574 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 147/177 2024-04-02T07:34:23.822 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs-devel-2:18.2.2-1015.gd0e392d9.el8.x86_ 139/177 2024-04-02T07:34:23.958 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-nbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 148/177 2024-04-02T07:34:24.042 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-team-1:1.40.16-15.el8.x86_64 140/177 2024-04-02T07:34:24.278 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-tui-1:1.40.16-15.el8.x86_64 141/177 2024-04-02T07:34:24.403 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : iproute-6.2.0-5.el8.x86_64 149/177 2024-04-02T07:34:24.586 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : acl-2.2.53-3.el8.x86_64 142/177 2024-04-02T07:34:24.666 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : openssl-1:1.1.1k-12.el8.x86_64 150/177 2024-04-02T07:34:24.752 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libacl-devel-2.2.53-3.el8.x86_64 143/177 2024-04-02T07:34:24.861 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 151/177 2024-04-02T07:34:25.073 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flex-2.6.1-9.el8.x86_64 152/177 2024-04-02T07:34:25.178 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 152/177 2024-04-02T07:34:25.406 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : bison-3.0.4-10.el8.x86_64 153/177 2024-04-02T07:34:25.432 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 153/177 2024-04-02T07:34:25.575 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-tui-1:1.40.16-4.el8.x86_64 154/177 2024-04-02T07:34:25.766 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-debuginfod-client-0.189-1.el8.x86_64 155/177 2024-04-02T07:34:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-04-02T07:34:25.948 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-libs-0.189-1.el8.x86_64 156/177 2024-04-02T07:34:26.159 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : openssl-1:1.1.1k-9.el8.x86_64 157/177 2024-04-02T07:34:26.194 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-04-02T07:34:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-04-02T07:34:26.364 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 158/177 2024-04-02T07:34:26.364 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 159/177 2024-04-02T07:34:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : xfsdump-3.1.8-7.el8.x86_64 145/177 2024-04-02T07:34:26.619 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 159/177 2024-04-02T07:34:26.620 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 160/177 2024-04-02T07:34:26.694 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-top-2:18.2.2-1015.gd0e392d9.el8.noarch 146/177 2024-04-02T07:34:26.888 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 160/177 2024-04-02T07:34:26.888 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 161/177 2024-04-02T07:34:26.935 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 147/177 2024-04-02T07:34:27.177 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 161/177 2024-04-02T07:34:27.292 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-nbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 148/177 2024-04-02T07:34:27.359 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : acl-2.2.53-1.el8.x86_64 162/177 2024-04-02T07:34:27.546 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libacl-devel-2.2.53-1.el8.x86_64 163/177 2024-04-02T07:34:27.732 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : iproute-6.2.0-5.el8.x86_64 149/177 2024-04-02T07:34:27.741 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : iproute-6.2.0-1.el8.x86_64 164/177 2024-04-02T07:34:27.947 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : util-linux-user-2.32.1-41.el8.x86_64 165/177 2024-04-02T07:34:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : openssl-1:1.1.1k-12.el8.x86_64 150/177 2024-04-02T07:34:28.143 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 151/177 2024-04-02T07:34:28.183 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : util-linux-2.32.1-41.el8.x86_64 166/177 2024-04-02T07:34:28.183 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libfdisk-2.32.1-41.el8.x86_64 167/177 2024-04-02T07:34:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flex-2.6.1-9.el8.x86_64 152/177 2024-04-02T07:34:28.426 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libfdisk-2.32.1-41.el8.x86_64 167/177 2024-04-02T07:34:28.426 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libmount-2.32.1-41.el8.x86_64 168/177 2024-04-02T07:34:28.502 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 152/177 2024-04-02T07:34:28.685 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libmount-2.32.1-41.el8.x86_64 168/177 2024-04-02T07:34:28.686 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libblkid-2.32.1-41.el8.x86_64 169/177 2024-04-02T07:34:28.718 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : bison-3.0.4-10.el8.x86_64 153/177 2024-04-02T07:34:28.744 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 153/177 2024-04-02T07:34:28.902 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-tui-1:1.40.16-4.el8.x86_64 154/177 2024-04-02T07:34:28.943 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libblkid-2.32.1-41.el8.x86_64 169/177 2024-04-02T07:34:29.086 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : elfutils-debuginfod-client-0.189-1.el8.x86_64 155/177 2024-04-02T07:34:29.091 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-team-1:1.40.16-4.el8.x86_64 170/177 2024-04-02T07:34:29.113 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-04-02T07:34:29.113 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-04-02T07:34:29.304 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : elfutils-libs-0.189-1.el8.x86_64 156/177 2024-04-02T07:34:29.527 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : openssl-1:1.1.1k-9.el8.x86_64 157/177 2024-04-02T07:34:29.665 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-04-02T07:34:29.665 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-libnm-1:1.40.16-4.el8.x86_64 172/177 2024-04-02T07:34:29.710 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 158/177 2024-04-02T07:34:29.710 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 159/177 2024-04-02T07:34:29.933 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-libnm-1:1.40.16-4.el8.x86_64 172/177 2024-04-02T07:34:29.934 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libuuid-2.32.1-41.el8.x86_64 173/177 2024-04-02T07:34:30.014 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 159/177 2024-04-02T07:34:30.014 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 160/177 2024-04-02T07:34:30.167 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libuuid-2.32.1-41.el8.x86_64 173/177 2024-04-02T07:34:30.167 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libsmartcols-2.32.1-41.el8.x86_64 174/177 2024-04-02T07:34:30.308 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 160/177 2024-04-02T07:34:30.308 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 161/177 2024-04-02T07:34:30.383 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libsmartcols-2.32.1-41.el8.x86_64 174/177 2024-04-02T07:34:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 161/177 2024-04-02T07:34:30.565 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-libelf-0.189-1.el8.x86_64 175/177 2024-04-02T07:34:30.708 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libacl-2.2.53-1.el8.x86_64 176/177 2024-04-02T07:34:30.709 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : openssl-libs-1:1.1.1k-9.el8.x86_64 177/177 2024-04-02T07:34:30.737 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : acl-2.2.53-1.el8.x86_64 162/177 2024-04-02T07:34:30.865 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libacl-devel-2.2.53-1.el8.x86_64 163/177 2024-04-02T07:34:31.094 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : iproute-6.2.0-1.el8.x86_64 164/177 2024-04-02T07:34:31.224 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openssl-libs-1:1.1.1k-9.el8.x86_64 177/177 2024-04-02T07:34:31.224 INFO:teuthology.orchestra.run.smithi113.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-02T07:34:31.224 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : util-linux-user-2.32.1-41.el8.x86_64 165/177 2024-04-02T07:34:31.519 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : util-linux-2.32.1-41.el8.x86_64 166/177 2024-04-02T07:34:31.519 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libfdisk-2.32.1-41.el8.x86_64 167/177 2024-04-02T07:34:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libfdisk-2.32.1-41.el8.x86_64 167/177 2024-04-02T07:34:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libmount-2.32.1-41.el8.x86_64 168/177 2024-04-02T07:34:31.955 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-2:18.2.2-1015.gd0e392d9.el8.x86_64 1/177 2024-04-02T07:34:31.955 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 2/177 2024-04-02T07:34:31.955 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 3/177 2024-04-02T07:34:31.955 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 4/177 2024-04-02T07:34:31.955 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1015.gd0e39 5/177 2024-04-02T07:34:31.955 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 6/177 2024-04-02T07:34:31.955 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 7/177 2024-04-02T07:34:31.955 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 8/177 2024-04-02T07:34:31.956 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 9/177 2024-04-02T07:34:31.956 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 10/177 2024-04-02T07:34:31.956 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 11/177 2024-04-02T07:34:31.956 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-test-2:18.2.2-1015.gd0e392d9.el8.x86_64 12/177 2024-04-02T07:34:31.956 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 13/177 2024-04-02T07:34:31.956 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs-devel-2:18.2.2-1015.gd0e392d9.el8.x86_ 14/177 2024-04-02T07:34:31.956 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 15/177 2024-04-02T07:34:31.956 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 16/177 2024-04-02T07:34:31.956 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados-devel-2:18.2.2-1015.gd0e392d9.el8.x86_6 17/177 2024-04-02T07:34:31.956 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86 18/177 2024-04-02T07:34:31.956 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 19/177 2024-04-02T07:34:31.956 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1015.gd0e392d9.el 20/177 2024-04-02T07:34:31.956 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-common-2:18.2.2-1015.gd0e392d9.el8. 21/177 2024-04-02T07:34:31.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cephfs-2:18.2.2-1015.gd0e392d9.el8.x86_6 22/177 2024-04-02T07:34:31.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rados-2:18.2.2-1015.gd0e392d9.el8.x86_64 23/177 2024-04-02T07:34:31.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 24/177 2024-04-02T07:34:31.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 25/177 2024-04-02T07:34:31.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 26/177 2024-04-02T07:34:31.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 27/177 2024-04-02T07:34:31.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-nbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 28/177 2024-04-02T07:34:31.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1015.gd0e392d9. 29/177 2024-04-02T07:34:31.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noa 30/177 2024-04-02T07:34:31.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.n 31/177 2024-04-02T07:34:31.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e 32/177 2024-04-02T07:34:31.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1015.gd0e392d9.el 33/177 2024-04-02T07:34:31.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 34/177 2024-04-02T07:34:31.957 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1015.gd0e392d9.e 35/177 2024-04-02T07:34:31.958 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 36/177 2024-04-02T07:34:31.958 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 37/177 2024-04-02T07:34:31.958 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephfs-top-2:18.2.2-1015.gd0e392d9.el8.noarch 38/177 2024-04-02T07:34:31.958 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : automake-1.16.1-8.el8.noarch 39/177 2024-04-02T07:34:31.958 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/177 2024-04-02T07:34:31.958 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/177 2024-04-02T07:34:31.958 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : jq-1.6-8.el8.x86_64 42/177 2024-04-02T07:34:31.958 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libpq-13.5-1.el8.x86_64 43/177 2024-04-02T07:34:31.958 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 44/177 2024-04-02T07:34:31.958 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 45/177 2024-04-02T07:34:31.958 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 46/177 2024-04-02T07:34:31.958 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 47/177 2024-04-02T07:34:31.958 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 48/177 2024-04-02T07:34:31.959 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 49/177 2024-04-02T07:34:31.959 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 50/177 2024-04-02T07:34:31.959 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-contrib-10.23-2.module_el8+572+929c87 51/177 2024-04-02T07:34:31.959 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-server-10.23-2.module_el8+572+929c87a 52/177 2024-04-02T07:34:31.959 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 53/177 2024-04-02T07:34:31.959 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 54/177 2024-04-02T07:34:31.959 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 55/177 2024-04-02T07:34:31.959 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 56/177 2024-04-02T07:34:31.959 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 57/177 2024-04-02T07:34:31.959 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 58/177 2024-04-02T07:34:31.959 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 59/177 2024-04-02T07:34:31.959 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 60/177 2024-04-02T07:34:31.959 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 61/177 2024-04-02T07:34:31.959 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/177 2024-04-02T07:34:31.960 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 63/177 2024-04-02T07:34:31.960 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 64/177 2024-04-02T07:34:31.960 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 65/177 2024-04-02T07:34:31.960 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 66/177 2024-04-02T07:34:31.960 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 67/177 2024-04-02T07:34:31.960 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 68/177 2024-04-02T07:34:31.960 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 69/177 2024-04-02T07:34:31.960 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/177 2024-04-02T07:34:31.960 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-devel-0.190-2.el8.x86_64 71/177 2024-04-02T07:34:31.960 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 72/177 2024-04-02T07:34:31.960 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 73/177 2024-04-02T07:34:31.960 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 74/177 2024-04-02T07:34:31.960 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 75/177 2024-04-02T07:34:31.960 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 76/177 2024-04-02T07:34:31.961 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 77/177 2024-04-02T07:34:31.961 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 78/177 2024-04-02T07:34:31.961 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 79/177 2024-04-02T07:34:31.961 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libselinux-devel-2.9-8.el8.x86_64 80/177 2024-04-02T07:34:31.961 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 81/177 2024-04-02T07:34:31.961 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-devel-2.32.1-46.el8.x86_64 82/177 2024-04-02T07:34:31.961 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 83/177 2024-04-02T07:34:31.961 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 84/177 2024-04-02T07:34:31.961 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8.x86_64 85/177 2024-04-02T07:34:31.961 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 86/177 2024-04-02T07:34:31.961 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 87/177 2024-04-02T07:34:31.961 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 88/177 2024-04-02T07:34:31.961 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 89/177 2024-04-02T07:34:31.962 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 90/177 2024-04-02T07:34:31.962 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 91/177 2024-04-02T07:34:31.962 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 92/177 2024-04-02T07:34:31.962 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 93/177 2024-04-02T07:34:31.962 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 94/177 2024-04-02T07:34:31.962 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 95/177 2024-04-02T07:34:31.962 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 96/177 2024-04-02T07:34:31.962 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 97/177 2024-04-02T07:34:31.962 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 98/177 2024-04-02T07:34:31.962 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 99/177 2024-04-02T07:34:31.962 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 100/177 2024-04-02T07:34:31.962 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 101/177 2024-04-02T07:34:31.962 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 102/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 103/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 104/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 105/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 106/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 107/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 108/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 109/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 110/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 111/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 112/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 113/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 114/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 115/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-portend-2.6-1.el8.noarch 116/177 2024-04-02T07:34:31.963 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 117/177 2024-04-02T07:34:31.964 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 118/177 2024-04-02T07:34:31.964 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 119/177 2024-04-02T07:34:31.964 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 120/177 2024-04-02T07:34:31.964 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 121/177 2024-04-02T07:34:31.964 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 122/177 2024-04-02T07:34:31.964 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 123/177 2024-04-02T07:34:31.964 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 124/177 2024-04-02T07:34:31.964 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 125/177 2024-04-02T07:34:31.964 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 126/177 2024-04-02T07:34:31.964 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 127/177 2024-04-02T07:34:31.964 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 128/177 2024-04-02T07:34:31.964 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 129/177 2024-04-02T07:34:31.964 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 130/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 131/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 132/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 133/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-1:1.40.16-15.el8.x86_64 134/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-1:1.40.16-4.el8.x86_64 135/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-libnm-1:1.40.16-15.el8.x86_64 136/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-libnm-1:1.40.16-4.el8.x86_64 137/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-team-1:1.40.16-15.el8.x86_64 138/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-team-1:1.40.16-4.el8.x86_64 139/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-tui-1:1.40.16-15.el8.x86_64 140/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-tui-1:1.40.16-4.el8.x86_64 141/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : acl-2.2.53-3.el8.x86_64 142/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : acl-2.2.53-1.el8.x86_64 143/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el8.x86_64 144/177 2024-04-02T07:34:31.965 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-debuginfod-client-0.189-1.el8.x86_64 145/177 2024-04-02T07:34:31.966 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-0.190-2.el8.x86_64 146/177 2024-04-02T07:34:31.966 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-0.189-1.el8.x86_64 147/177 2024-04-02T07:34:31.966 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libs-0.190-2.el8.x86_64 148/177 2024-04-02T07:34:31.966 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libs-0.189-1.el8.x86_64 149/177 2024-04-02T07:34:31.966 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : iproute-6.2.0-5.el8.x86_64 150/177 2024-04-02T07:34:31.966 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : iproute-6.2.0-1.el8.x86_64 151/177 2024-04-02T07:34:31.966 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-2.2.53-3.el8.x86_64 152/177 2024-04-02T07:34:31.966 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-2.2.53-1.el8.x86_64 153/177 2024-04-02T07:34:31.966 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-devel-2.2.53-3.el8.x86_64 154/177 2024-04-02T07:34:31.966 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-devel-2.2.53-1.el8.x86_64 155/177 2024-04-02T07:34:31.966 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libblkid-2.32.1-46.el8.x86_64 156/177 2024-04-02T07:34:31.966 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libblkid-2.32.1-41.el8.x86_64 157/177 2024-04-02T07:34:31.967 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libfdisk-2.32.1-46.el8.x86_64 158/177 2024-04-02T07:34:31.967 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libfdisk-2.32.1-41.el8.x86_64 159/177 2024-04-02T07:34:31.967 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libmount-2.32.1-46.el8.x86_64 160/177 2024-04-02T07:34:31.967 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libmount-2.32.1-41.el8.x86_64 161/177 2024-04-02T07:34:31.967 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libsmartcols-2.32.1-46.el8.x86_64 162/177 2024-04-02T07:34:31.967 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libsmartcols-2.32.1-41.el8.x86_64 163/177 2024-04-02T07:34:31.967 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-2.32.1-46.el8.x86_64 164/177 2024-04-02T07:34:31.967 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-2.32.1-41.el8.x86_64 165/177 2024-04-02T07:34:31.967 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-1:1.1.1k-12.el8.x86_64 166/177 2024-04-02T07:34:31.967 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-1:1.1.1k-9.el8.x86_64 167/177 2024-04-02T07:34:31.967 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8.x86_64 168/177 2024-04-02T07:34:31.967 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-libs-1:1.1.1k-9.el8.x86_64 169/177 2024-04-02T07:34:31.967 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-2.32.1-46.el8.x86_64 170/177 2024-04-02T07:34:31.967 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-2.32.1-41.el8.x86_64 171/177 2024-04-02T07:34:31.968 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-user-2.32.1-46.el8.x86_64 172/177 2024-04-02T07:34:31.968 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-user-2.32.1-41.el8.x86_64 173/177 2024-04-02T07:34:31.968 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsdump-3.1.8-7.el8.x86_64 174/177 2024-04-02T07:34:31.968 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 175/177 2024-04-02T07:34:31.968 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 176/177 2024-04-02T07:34:32.045 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libmount-2.32.1-41.el8.x86_64 168/177 2024-04-02T07:34:32.046 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libblkid-2.32.1-41.el8.x86_64 169/177 2024-04-02T07:34:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libblkid-2.32.1-41.el8.x86_64 169/177 2024-04-02T07:34:32.411 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-team-1:1.40.16-4.el8.x86_64 170/177 2024-04-02T07:34:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-04-02T07:34:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-04-02T07:34:32.970 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-04-02T07:34:32.970 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-libnm-1:1.40.16-4.el8.x86_64 172/177 2024-04-02T07:34:33.220 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-libnm-1:1.40.16-4.el8.x86_64 172/177 2024-04-02T07:34:33.220 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libuuid-2.32.1-41.el8.x86_64 173/177 2024-04-02T07:34:33.436 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libuuid-2.32.1-41.el8.x86_64 173/177 2024-04-02T07:34:33.437 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libsmartcols-2.32.1-41.el8.x86_64 174/177 2024-04-02T07:34:33.661 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libsmartcols-2.32.1-41.el8.x86_64 174/177 2024-04-02T07:34:33.827 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : elfutils-libelf-0.189-1.el8.x86_64 175/177 2024-04-02T07:34:33.995 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libacl-2.2.53-1.el8.x86_64 176/177 2024-04-02T07:34:33.995 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : openssl-libs-1:1.1.1k-9.el8.x86_64 177/177 2024-04-02T07:34:34.491 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: openssl-libs-1:1.1.1k-9.el8.x86_64 177/177 2024-04-02T07:34:34.491 INFO:teuthology.orchestra.run.smithi089.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-02T07:34:34.492 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:35.253 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-2:18.2.2-1015.gd0e392d9.el8.x86_64 1/177 2024-04-02T07:34:35.254 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 2/177 2024-04-02T07:34:35.254 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 3/177 2024-04-02T07:34:35.254 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 4/177 2024-04-02T07:34:35.254 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1015.gd0e39 5/177 2024-04-02T07:34:35.254 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 6/177 2024-04-02T07:34:35.254 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 7/177 2024-04-02T07:34:35.255 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 8/177 2024-04-02T07:34:35.255 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 9/177 2024-04-02T07:34:35.255 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 10/177 2024-04-02T07:34:35.255 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 11/177 2024-04-02T07:34:35.255 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-test-2:18.2.2-1015.gd0e392d9.el8.x86_64 12/177 2024-04-02T07:34:35.255 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 13/177 2024-04-02T07:34:35.255 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs-devel-2:18.2.2-1015.gd0e392d9.el8.x86_ 14/177 2024-04-02T07:34:35.255 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 15/177 2024-04-02T07:34:35.256 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 16/177 2024-04-02T07:34:35.256 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados-devel-2:18.2.2-1015.gd0e392d9.el8.x86_6 17/177 2024-04-02T07:34:35.256 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86 18/177 2024-04-02T07:34:35.256 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 19/177 2024-04-02T07:34:35.256 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1015.gd0e392d9.el 20/177 2024-04-02T07:34:35.256 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-common-2:18.2.2-1015.gd0e392d9.el8. 21/177 2024-04-02T07:34:35.256 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cephfs-2:18.2.2-1015.gd0e392d9.el8.x86_6 22/177 2024-04-02T07:34:35.257 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rados-2:18.2.2-1015.gd0e392d9.el8.x86_64 23/177 2024-04-02T07:34:35.257 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 24/177 2024-04-02T07:34:35.257 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 25/177 2024-04-02T07:34:35.257 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 26/177 2024-04-02T07:34:35.257 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 27/177 2024-04-02T07:34:35.257 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-nbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 28/177 2024-04-02T07:34:35.257 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1015.gd0e392d9. 29/177 2024-04-02T07:34:35.257 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noa 30/177 2024-04-02T07:34:35.258 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.n 31/177 2024-04-02T07:34:35.258 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e 32/177 2024-04-02T07:34:35.258 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1015.gd0e392d9.el 33/177 2024-04-02T07:34:35.258 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 34/177 2024-04-02T07:34:35.258 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1015.gd0e392d9.e 35/177 2024-04-02T07:34:35.258 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 36/177 2024-04-02T07:34:35.258 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 37/177 2024-04-02T07:34:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-top-2:18.2.2-1015.gd0e392d9.el8.noarch 38/177 2024-04-02T07:34:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : automake-1.16.1-8.el8.noarch 39/177 2024-04-02T07:34:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/177 2024-04-02T07:34:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/177 2024-04-02T07:34:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : jq-1.6-8.el8.x86_64 42/177 2024-04-02T07:34:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libpq-13.5-1.el8.x86_64 43/177 2024-04-02T07:34:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 44/177 2024-04-02T07:34:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 45/177 2024-04-02T07:34:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 46/177 2024-04-02T07:34:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 47/177 2024-04-02T07:34:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 48/177 2024-04-02T07:34:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 49/177 2024-04-02T07:34:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 50/177 2024-04-02T07:34:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-contrib-10.23-2.module_el8+572+929c87 51/177 2024-04-02T07:34:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-server-10.23-2.module_el8+572+929c87a 52/177 2024-04-02T07:34:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 53/177 2024-04-02T07:34:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 54/177 2024-04-02T07:34:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 55/177 2024-04-02T07:34:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 56/177 2024-04-02T07:34:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 57/177 2024-04-02T07:34:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 58/177 2024-04-02T07:34:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 59/177 2024-04-02T07:34:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 60/177 2024-04-02T07:34:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 61/177 2024-04-02T07:34:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/177 2024-04-02T07:34:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 63/177 2024-04-02T07:34:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 64/177 2024-04-02T07:34:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 65/177 2024-04-02T07:34:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 66/177 2024-04-02T07:34:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 67/177 2024-04-02T07:34:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 68/177 2024-04-02T07:34:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 69/177 2024-04-02T07:34:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/177 2024-04-02T07:34:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libelf-devel-0.190-2.el8.x86_64 71/177 2024-04-02T07:34:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 72/177 2024-04-02T07:34:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 73/177 2024-04-02T07:34:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 74/177 2024-04-02T07:34:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 75/177 2024-04-02T07:34:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 76/177 2024-04-02T07:34:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 77/177 2024-04-02T07:34:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 78/177 2024-04-02T07:34:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 79/177 2024-04-02T07:34:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libselinux-devel-2.9-8.el8.x86_64 80/177 2024-04-02T07:34:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 81/177 2024-04-02T07:34:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libuuid-devel-2.32.1-46.el8.x86_64 82/177 2024-04-02T07:34:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 83/177 2024-04-02T07:34:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 84/177 2024-04-02T07:34:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8.x86_64 85/177 2024-04-02T07:34:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 86/177 2024-04-02T07:34:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 87/177 2024-04-02T07:34:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 88/177 2024-04-02T07:34:35.265 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 89/177 2024-04-02T07:34:35.265 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 90/177 2024-04-02T07:34:35.265 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 91/177 2024-04-02T07:34:35.265 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 92/177 2024-04-02T07:34:35.265 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 93/177 2024-04-02T07:34:35.265 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 94/177 2024-04-02T07:34:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 95/177 2024-04-02T07:34:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 96/177 2024-04-02T07:34:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 97/177 2024-04-02T07:34:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 98/177 2024-04-02T07:34:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 99/177 2024-04-02T07:34:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 100/177 2024-04-02T07:34:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 101/177 2024-04-02T07:34:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 102/177 2024-04-02T07:34:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 103/177 2024-04-02T07:34:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 104/177 2024-04-02T07:34:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 105/177 2024-04-02T07:34:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 106/177 2024-04-02T07:34:35.266 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 107/177 2024-04-02T07:34:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 108/177 2024-04-02T07:34:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 109/177 2024-04-02T07:34:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 110/177 2024-04-02T07:34:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 111/177 2024-04-02T07:34:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 112/177 2024-04-02T07:34:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 113/177 2024-04-02T07:34:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 114/177 2024-04-02T07:34:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 115/177 2024-04-02T07:34:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-portend-2.6-1.el8.noarch 116/177 2024-04-02T07:34:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 117/177 2024-04-02T07:34:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 118/177 2024-04-02T07:34:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 119/177 2024-04-02T07:34:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 120/177 2024-04-02T07:34:35.267 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 121/177 2024-04-02T07:34:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 122/177 2024-04-02T07:34:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 123/177 2024-04-02T07:34:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 124/177 2024-04-02T07:34:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 125/177 2024-04-02T07:34:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 126/177 2024-04-02T07:34:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 127/177 2024-04-02T07:34:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 128/177 2024-04-02T07:34:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 129/177 2024-04-02T07:34:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 130/177 2024-04-02T07:34:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 131/177 2024-04-02T07:34:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 132/177 2024-04-02T07:34:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 133/177 2024-04-02T07:34:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-1:1.40.16-15.el8.x86_64 134/177 2024-04-02T07:34:35.268 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-1:1.40.16-4.el8.x86_64 135/177 2024-04-02T07:34:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-libnm-1:1.40.16-15.el8.x86_64 136/177 2024-04-02T07:34:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-libnm-1:1.40.16-4.el8.x86_64 137/177 2024-04-02T07:34:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-team-1:1.40.16-15.el8.x86_64 138/177 2024-04-02T07:34:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-team-1:1.40.16-4.el8.x86_64 139/177 2024-04-02T07:34:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-tui-1:1.40.16-15.el8.x86_64 140/177 2024-04-02T07:34:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-tui-1:1.40.16-4.el8.x86_64 141/177 2024-04-02T07:34:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : acl-2.2.53-3.el8.x86_64 142/177 2024-04-02T07:34:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : acl-2.2.53-1.el8.x86_64 143/177 2024-04-02T07:34:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el8.x86_64 144/177 2024-04-02T07:34:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-debuginfod-client-0.189-1.el8.x86_64 145/177 2024-04-02T07:34:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libelf-0.190-2.el8.x86_64 146/177 2024-04-02T07:34:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libelf-0.189-1.el8.x86_64 147/177 2024-04-02T07:34:35.269 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libs-0.190-2.el8.x86_64 148/177 2024-04-02T07:34:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libs-0.189-1.el8.x86_64 149/177 2024-04-02T07:34:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : iproute-6.2.0-5.el8.x86_64 150/177 2024-04-02T07:34:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : iproute-6.2.0-1.el8.x86_64 151/177 2024-04-02T07:34:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libacl-2.2.53-3.el8.x86_64 152/177 2024-04-02T07:34:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libacl-2.2.53-1.el8.x86_64 153/177 2024-04-02T07:34:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libacl-devel-2.2.53-3.el8.x86_64 154/177 2024-04-02T07:34:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libacl-devel-2.2.53-1.el8.x86_64 155/177 2024-04-02T07:34:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libblkid-2.32.1-46.el8.x86_64 156/177 2024-04-02T07:34:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libblkid-2.32.1-41.el8.x86_64 157/177 2024-04-02T07:34:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libfdisk-2.32.1-46.el8.x86_64 158/177 2024-04-02T07:34:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libfdisk-2.32.1-41.el8.x86_64 159/177 2024-04-02T07:34:35.270 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libmount-2.32.1-46.el8.x86_64 160/177 2024-04-02T07:34:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libmount-2.32.1-41.el8.x86_64 161/177 2024-04-02T07:34:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libsmartcols-2.32.1-46.el8.x86_64 162/177 2024-04-02T07:34:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libsmartcols-2.32.1-41.el8.x86_64 163/177 2024-04-02T07:34:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libuuid-2.32.1-46.el8.x86_64 164/177 2024-04-02T07:34:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libuuid-2.32.1-41.el8.x86_64 165/177 2024-04-02T07:34:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-1:1.1.1k-12.el8.x86_64 166/177 2024-04-02T07:34:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-1:1.1.1k-9.el8.x86_64 167/177 2024-04-02T07:34:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8.x86_64 168/177 2024-04-02T07:34:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-libs-1:1.1.1k-9.el8.x86_64 169/177 2024-04-02T07:34:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-2.32.1-46.el8.x86_64 170/177 2024-04-02T07:34:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-2.32.1-41.el8.x86_64 171/177 2024-04-02T07:34:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-user-2.32.1-46.el8.x86_64 172/177 2024-04-02T07:34:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-user-2.32.1-41.el8.x86_64 173/177 2024-04-02T07:34:35.271 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsdump-3.1.8-7.el8.x86_64 174/177 2024-04-02T07:34:35.272 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 175/177 2024-04-02T07:34:35.272 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 176/177 2024-04-02T07:34:35.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 177/177 2024-04-02T07:34:35.375 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:35.375 INFO:teuthology.orchestra.run.smithi113.stdout:Upgraded: 2024-04-02T07:34:35.375 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-1:1.40.16-15.el8.x86_64 2024-04-02T07:34:35.375 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-libnm-1:1.40.16-15.el8.x86_64 2024-04-02T07:34:35.375 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-team-1:1.40.16-15.el8.x86_64 2024-04-02T07:34:35.375 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-tui-1:1.40.16-15.el8.x86_64 2024-04-02T07:34:35.375 INFO:teuthology.orchestra.run.smithi113.stdout: acl-2.2.53-3.el8.x86_64 2024-04-02T07:34:35.375 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-debuginfod-client-0.190-2.el8.x86_64 2024-04-02T07:34:35.375 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-0.190-2.el8.x86_64 2024-04-02T07:34:35.376 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libs-0.190-2.el8.x86_64 2024-04-02T07:34:35.376 INFO:teuthology.orchestra.run.smithi113.stdout: iproute-6.2.0-5.el8.x86_64 2024-04-02T07:34:35.376 INFO:teuthology.orchestra.run.smithi113.stdout: libacl-2.2.53-3.el8.x86_64 2024-04-02T07:34:35.376 INFO:teuthology.orchestra.run.smithi113.stdout: libacl-devel-2.2.53-3.el8.x86_64 2024-04-02T07:34:35.376 INFO:teuthology.orchestra.run.smithi113.stdout: libblkid-2.32.1-46.el8.x86_64 2024-04-02T07:34:35.376 INFO:teuthology.orchestra.run.smithi113.stdout: libfdisk-2.32.1-46.el8.x86_64 2024-04-02T07:34:35.376 INFO:teuthology.orchestra.run.smithi113.stdout: libmount-2.32.1-46.el8.x86_64 2024-04-02T07:34:35.376 INFO:teuthology.orchestra.run.smithi113.stdout: librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.376 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.377 INFO:teuthology.orchestra.run.smithi113.stdout: libsmartcols-2.32.1-46.el8.x86_64 2024-04-02T07:34:35.377 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-2.32.1-46.el8.x86_64 2024-04-02T07:34:35.377 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-1:1.1.1k-12.el8.x86_64 2024-04-02T07:34:35.377 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-libs-1:1.1.1k-12.el8.x86_64 2024-04-02T07:34:35.377 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-2.32.1-46.el8.x86_64 2024-04-02T07:34:35.377 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-user-2.32.1-46.el8.x86_64 2024-04-02T07:34:35.377 INFO:teuthology.orchestra.run.smithi113.stdout: xfsdump-3.1.8-7.el8.x86_64 2024-04-02T07:34:35.377 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-04-02T07:34:35.377 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-02T07:34:35.377 INFO:teuthology.orchestra.run.smithi113.stdout: automake-1.16.1-8.el8.noarch 2024-04-02T07:34:35.377 INFO:teuthology.orchestra.run.smithi113.stdout: bison-3.0.4-10.el8.x86_64 2024-04-02T07:34:35.377 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.378 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.378 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.378 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.378 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:35.378 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.378 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.378 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.378 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:35.378 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:35.378 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:35.378 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:35.378 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:35.378 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.379 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.379 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:35.379 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.379 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.379 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.379 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:35.379 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:35.379 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.379 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-top-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:35.379 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-devel-0.190-2.el8.x86_64 2024-04-02T07:34:35.379 INFO:teuthology.orchestra.run.smithi113.stdout: flex-2.6.1-9.el8.x86_64 2024-04-02T07:34:35.379 INFO:teuthology.orchestra.run.smithi113.stdout: fuse-2.9.7-19.el8.x86_64 2024-04-02T07:34:35.379 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-02T07:34:35.380 INFO:teuthology.orchestra.run.smithi113.stdout: jq-1.6-8.el8.x86_64 2024-04-02T07:34:35.380 INFO:teuthology.orchestra.run.smithi113.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-04-02T07:34:35.380 INFO:teuthology.orchestra.run.smithi113.stdout: krb5-devel-1.18.2-22.el8.x86_64 2024-04-02T07:34:35.380 INFO:teuthology.orchestra.run.smithi113.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-02T07:34:35.380 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.380 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.380 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.380 INFO:teuthology.orchestra.run.smithi113.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2024-04-02T07:34:35.380 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-02T07:34:35.380 INFO:teuthology.orchestra.run.smithi113.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-02T07:34:35.380 INFO:teuthology.orchestra.run.smithi113.stdout: libpq-13.5-1.el8.x86_64 2024-04-02T07:34:35.380 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-02T07:34:35.380 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-02T07:34:35.380 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.381 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.381 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-02T07:34:35.381 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.381 INFO:teuthology.orchestra.run.smithi113.stdout: libselinux-devel-2.9-8.el8.x86_64 2024-04-02T07:34:35.381 INFO:teuthology.orchestra.run.smithi113.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-04-02T07:34:35.381 INFO:teuthology.orchestra.run.smithi113.stdout: libtool-2.4.6-25.el8.x86_64 2024-04-02T07:34:35.381 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-02T07:34:35.381 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-devel-2.32.1-46.el8.x86_64 2024-04-02T07:34:35.381 INFO:teuthology.orchestra.run.smithi113.stdout: libverto-devel-0.3.2-2.el8.x86_64 2024-04-02T07:34:35.381 INFO:teuthology.orchestra.run.smithi113.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-04-02T07:34:35.381 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-02T07:34:35.381 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-04-02T07:34:35.381 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-02T07:34:35.382 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-02T07:34:35.382 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-devel-1:1.1.1k-12.el8.x86_64 2024-04-02T07:34:35.382 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-devel-10.32-3.el8.x86_64 2024-04-02T07:34:35.382 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-utf16-10.32-3.el8.x86_64 2024-04-02T07:34:35.382 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-utf32-10.32-3.el8.x86_64 2024-04-02T07:34:35.382 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-10.23-2.module_el8+572+929c87ac.x86_64 2024-04-02T07:34:35.382 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-contrib-10.23-2.module_el8+572+929c87ac.x86_64 2024-04-02T07:34:35.382 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-server-10.23-2.module_el8+572+929c87ac.x86_64 2024-04-02T07:34:35.382 INFO:teuthology.orchestra.run.smithi113.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-02T07:34:35.382 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-02T07:34:35.382 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-02T07:34:35.382 INFO:teuthology.orchestra.run.smithi113.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-02T07:34:35.382 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-02T07:34:35.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-02T07:34:35.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-02T07:34:35.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-02T07:34:35.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-02T07:34:35.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-02T07:34:35.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-02T07:34:35.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-02T07:34:35.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-02T07:34:35.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-04-02T07:34:35.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-02T07:34:35.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-02T07:34:35.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-02T07:34:35.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-02T07:34:35.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-02T07:34:35.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-02T07:34:35.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-02T07:34:35.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-02T07:34:35.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-02T07:34:35.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-02T07:34:35.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-02T07:34:35.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-02T07:34:35.385 INFO:teuthology.orchestra.run.smithi113.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-02T07:34:35.385 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-02T07:34:35.385 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend-2.6-1.el8.noarch 2024-04-02T07:34:35.385 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-02T07:34:35.385 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-02T07:34:35.385 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-02T07:34:35.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-02T07:34:35.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-02T07:34:35.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-02T07:34:35.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-02T07:34:35.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-02T07:34:35.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-02T07:34:35.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-02T07:34:35.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-02T07:34:35.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-02T07:34:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-02T07:34:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-02T07:34:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-02T07:34:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-02T07:34:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-02T07:34:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-02T07:34:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-02T07:34:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-02T07:34:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-02T07:34:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-02T07:34:35.387 INFO:teuthology.orchestra.run.smithi113.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-02T07:34:35.388 INFO:teuthology.orchestra.run.smithi113.stdout: uuid-1.6.2-43.el8.x86_64 2024-04-02T07:34:35.388 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-04-02T07:34:35.388 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-02T07:34:35.388 INFO:teuthology.orchestra.run.smithi113.stdout: zlib-devel-1.2.11-21.el8.x86_64 2024-04-02T07:34:35.388 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:35.388 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-02T07:34:35.763 DEBUG:teuthology.parallel:result is None 2024-04-02T07:34:38.169 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 177/177 2024-04-02T07:34:38.169 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:38.169 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2024-04-02T07:34:38.169 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-1:1.40.16-15.el8.x86_64 2024-04-02T07:34:38.169 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-libnm-1:1.40.16-15.el8.x86_64 2024-04-02T07:34:38.169 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-team-1:1.40.16-15.el8.x86_64 2024-04-02T07:34:38.169 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-tui-1:1.40.16-15.el8.x86_64 2024-04-02T07:34:38.169 INFO:teuthology.orchestra.run.smithi089.stdout: acl-2.2.53-3.el8.x86_64 2024-04-02T07:34:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-debuginfod-client-0.190-2.el8.x86_64 2024-04-02T07:34:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-0.190-2.el8.x86_64 2024-04-02T07:34:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libs-0.190-2.el8.x86_64 2024-04-02T07:34:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: iproute-6.2.0-5.el8.x86_64 2024-04-02T07:34:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: libacl-2.2.53-3.el8.x86_64 2024-04-02T07:34:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: libacl-devel-2.2.53-3.el8.x86_64 2024-04-02T07:34:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: libblkid-2.32.1-46.el8.x86_64 2024-04-02T07:34:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: libfdisk-2.32.1-46.el8.x86_64 2024-04-02T07:34:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: libmount-2.32.1-46.el8.x86_64 2024-04-02T07:34:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: librados2-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: libsmartcols-2.32.1-46.el8.x86_64 2024-04-02T07:34:38.170 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-2.32.1-46.el8.x86_64 2024-04-02T07:34:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-1:1.1.1k-12.el8.x86_64 2024-04-02T07:34:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-libs-1:1.1.1k-12.el8.x86_64 2024-04-02T07:34:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux-2.32.1-46.el8.x86_64 2024-04-02T07:34:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux-user-2.32.1-46.el8.x86_64 2024-04-02T07:34:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: xfsdump-3.1.8-7.el8.x86_64 2024-04-02T07:34:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-04-02T07:34:38.171 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-02T07:34:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: automake-1.16.1-8.el8.noarch 2024-04-02T07:34:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: bison-3.0.4-10.el8.x86_64 2024-04-02T07:34:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.171 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-top-2:18.2.2-1015.gd0e392d9.el8.noarch 2024-04-02T07:34:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-devel-0.190-2.el8.x86_64 2024-04-02T07:34:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: flex-2.6.1-9.el8.x86_64 2024-04-02T07:34:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: fuse-2.9.7-19.el8.x86_64 2024-04-02T07:34:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-02T07:34:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: jq-1.6-8.el8.x86_64 2024-04-02T07:34:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-04-02T07:34:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: krb5-devel-1.18.2-22.el8.x86_64 2024-04-02T07:34:38.173 INFO:teuthology.orchestra.run.smithi089.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-02T07:34:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2024-04-02T07:34:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-02T07:34:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-02T07:34:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: libpq-13.5-1.el8.x86_64 2024-04-02T07:34:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-02T07:34:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-02T07:34:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-02T07:34:38.174 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: libselinux-devel-2.9-8.el8.x86_64 2024-04-02T07:34:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-04-02T07:34:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: libtool-2.4.6-25.el8.x86_64 2024-04-02T07:34:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-02T07:34:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-devel-2.32.1-46.el8.x86_64 2024-04-02T07:34:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: libverto-devel-0.3.2-2.el8.x86_64 2024-04-02T07:34:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-04-02T07:34:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-02T07:34:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-04-02T07:34:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-02T07:34:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-02T07:34:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-devel-1:1.1.1k-12.el8.x86_64 2024-04-02T07:34:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-devel-10.32-3.el8.x86_64 2024-04-02T07:34:38.175 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-utf16-10.32-3.el8.x86_64 2024-04-02T07:34:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: pcre2-utf32-10.32-3.el8.x86_64 2024-04-02T07:34:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-10.23-2.module_el8+572+929c87ac.x86_64 2024-04-02T07:34:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-contrib-10.23-2.module_el8+572+929c87ac.x86_64 2024-04-02T07:34:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-server-10.23-2.module_el8+572+929c87ac.x86_64 2024-04-02T07:34:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-02T07:34:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-02T07:34:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-02T07:34:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-02T07:34:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-02T07:34:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-02T07:34:38.176 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-02T07:34:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-02T07:34:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-02T07:34:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-02T07:34:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-02T07:34:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-02T07:34:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-02T07:34:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-04-02T07:34:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-02T07:34:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-02T07:34:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-02T07:34:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-02T07:34:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-02T07:34:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-02T07:34:38.177 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-02T07:34:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-02T07:34:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-02T07:34:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-02T07:34:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-02T07:34:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-02T07:34:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-02T07:34:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-02T07:34:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend-2.6-1.el8.noarch 2024-04-02T07:34:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-02T07:34:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-02T07:34:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-02T07:34:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-02T07:34:38.178 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-02T07:34:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-02T07:34:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-02T07:34:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-02T07:34:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-02T07:34:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-02T07:34:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-02T07:34:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-02T07:34:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-02T07:34:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-02T07:34:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-02T07:34:38.179 INFO:teuthology.orchestra.run.smithi089.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-02T07:34:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-02T07:34:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-02T07:34:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-02T07:34:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-02T07:34:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-02T07:34:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd-2:18.2.2-1015.gd0e392d9.el8.x86_64 2024-04-02T07:34:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-02T07:34:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-02T07:34:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: uuid-1.6.2-43.el8.x86_64 2024-04-02T07:34:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-04-02T07:34:38.180 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-02T07:34:38.181 INFO:teuthology.orchestra.run.smithi089.stdout: zlib-devel-1.2.11-21.el8.x86_64 2024-04-02T07:34:38.181 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:38.181 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-02T07:34:38.489 DEBUG:teuthology.parallel:result is None 2024-04-02T07:34:38.490 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d0e392d9aee635f2b65768c460c7a151ead7a5e2 2024-04-02T07:34:38.632 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-02T07:34:38.688 INFO:teuthology.orchestra.run.smithi089.stdout:18.2.2-1015.gd0e392d9.el8 2024-04-02T07:34:38.689 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1015.gd0e392d9.el8 2024-04-02T07:34:38.689 INFO:teuthology.task.install:The correct ceph version 18.2.2-1015.gd0e392d9 is installed. 2024-04-02T07:34:38.690 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d0e392d9aee635f2b65768c460c7a151ead7a5e2 2024-04-02T07:34:38.910 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-02T07:34:38.964 INFO:teuthology.orchestra.run.smithi113.stdout:18.2.2-1015.gd0e392d9.el8 2024-04-02T07:34:38.965 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1015.gd0e392d9.el8 2024-04-02T07:34:38.965 INFO:teuthology.task.install:The correct ceph version 18.2.2-1015.gd0e392d9 is installed. 2024-04-02T07:34:38.966 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-02T07:34:38.967 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:34:38.967 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-02T07:34:39.043 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:34:39.043 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-02T07:34:39.113 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-02T07:34:39.113 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:34:39.113 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2024-04-02T07:34:39.184 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-02T07:34:39.291 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:34:39.291 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/daemon-helper 2024-04-02T07:34:39.359 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-02T07:34:39.468 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-02T07:34:39.468 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:34:39.468 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-02T07:34:39.534 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-02T07:34:39.642 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:34:39.643 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-02T07:34:39.710 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-02T07:34:39.816 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-02T07:34:39.816 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:34:39.816 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2024-04-02T07:34:39.887 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-02T07:34:39.994 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:34:39.994 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/stdin-killer 2024-04-02T07:34:40.061 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-02T07:34:40.171 INFO:teuthology.run_tasks:Running task ceph... 2024-04-02T07:34:40.239 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-02T07:34:40.239 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /var/log/ceph 2024-04-02T07:34:40.243 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 777 /var/log/ceph 2024-04-02T07:34:40.311 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-02T07:34:40.312 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-02T07:34:40.315 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-02T07:34:40.382 INFO:tasks.ceph:Creating extra log directories... 2024-04-02T07:34:40.383 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-02T07:34:40.387 DEBUG:teuthology.orchestra.run.smithi113:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-02T07:34:40.460 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-02T07:34:40.460 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-02T07:34:40.460 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-02T07:34:40.460 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:34:40.460 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-02T07:34:40.505 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-02T07:34:40.712 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-02T07:34:40.779 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:34:40.779 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-02T07:34:40.823 DEBUG:teuthology.orchestra.run.smithi113:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-02T07:34:41.021 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-02T07:34:41.096 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-02T07:34:41.096 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-02T07:34:41.097 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459', 'branch': 'wip-vshankar-testing-20240326.105515-reef', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7635459', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20240326.105515-reef'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20240326.105515-reef', 'sha1': 'd46289a69430c98a5ee030c7aba50d0c7af4ad4b'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 7327, 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2', 'sleep_before_teardown': 0, 'subset': '990/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20240326.105515-reef', 'suite_path': '/home/teuthworker/src/github.com_vshankar_ceph_d46289a69430c98a5ee030c7aba50d0c7af4ad4b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/vshankar/ceph.git', 'suite_sha1': 'd46289a69430c98a5ee030c7aba50d0c7af4ad4b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi089.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGtwTkQ7YIBikeL384i+WuovQQ9VOrg7Jr01f4YTXbCtcED02BXdNA6Vxc692Plr1mrtlShqlFiikUvYOzF/3U=', 'smithi113.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNNQGxHVHDjN0qBn7n38dK/3CdgI7Sj/i3cmkwTSGB4mRRhXktG01i0oglnwBx99f4xOW6VGFSlX6tXmYuVKlWE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'd0e392d9aee635f2b65768c460c7a151ead7a5e2', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-04-02_06:51:51', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-02T07:34:41.097 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-02T07:34:41.101 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-02T07:34:41.145 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/run/ceph 2024-04-02T07:34:41.186 DEBUG:teuthology.orchestra.run.smithi113:> sudo install -d -m0777 -- /var/run/ceph 2024-04-02T07:34:41.253 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:34:41.253 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2024-04-02T07:34:41.297 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-02T07:34:41.298 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2024-04-02T07:34:41.384 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-02T07:34:41.385 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T07:34:41.385 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 444 Links: 1 2024-04-02T07:34:41.385 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T07:34:41.385 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T07:34:41.385 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-04-02 07:34:34.691082684 +0000 2024-04-02T07:34:41.385 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-04-02 07:31:25.789637588 +0000 2024-04-02T07:34:41.385 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-04-02 07:31:25.789637588 +0000 2024-04-02T07:34:41.385 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-04-02 07:31:25.765638113 +0000 2024-04-02T07:34:41.386 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-02T07:34:41.489 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-04-02T07:34:41.490 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-04-02T07:34:41.490 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000294774 s, 1.7 MB/s 2024-04-02T07:34:41.492 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-02T07:34:41.540 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2024-04-02T07:34:41.585 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-02T07:34:41.585 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T07:34:41.585 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 450 Links: 1 2024-04-02T07:34:41.585 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T07:34:41.585 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T07:34:41.585 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-04-02 07:34:34.691082684 +0000 2024-04-02T07:34:41.585 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-04-02 07:31:26.255627386 +0000 2024-04-02T07:34:41.585 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-04-02 07:31:26.255627386 +0000 2024-04-02T07:34:41.586 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-04-02 07:31:26.230627933 +0000 2024-04-02T07:34:41.586 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-02T07:34:41.692 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-04-02T07:34:41.692 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-04-02T07:34:41.692 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000281819 s, 1.8 MB/s 2024-04-02T07:34:41.694 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-02T07:34:41.742 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2024-04-02T07:34:41.828 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-02T07:34:41.829 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T07:34:41.829 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 456 Links: 1 2024-04-02T07:34:41.829 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T07:34:41.829 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T07:34:41.829 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-04-02 07:34:34.692082662 +0000 2024-04-02T07:34:41.829 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-04-02 07:31:26.720617206 +0000 2024-04-02T07:34:41.829 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-04-02 07:31:26.720617206 +0000 2024-04-02T07:34:41.829 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-04-02 07:31:26.695617754 +0000 2024-04-02T07:34:41.829 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-02T07:34:41.932 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-04-02T07:34:41.932 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-04-02T07:34:41.932 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000295736 s, 1.7 MB/s 2024-04-02T07:34:41.935 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-02T07:34:41.982 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2024-04-02T07:34:42.069 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-02T07:34:42.069 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T07:34:42.069 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 462 Links: 1 2024-04-02T07:34:42.069 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T07:34:42.069 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T07:34:42.069 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-04-02 07:34:34.692082662 +0000 2024-04-02T07:34:42.069 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-04-02 07:31:27.204606611 +0000 2024-04-02T07:34:42.069 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-04-02 07:31:27.204606611 +0000 2024-04-02T07:34:42.070 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-04-02 07:31:27.181607114 +0000 2024-04-02T07:34:42.070 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-02T07:34:42.176 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-04-02T07:34:42.177 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-04-02T07:34:42.177 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000376665 s, 1.4 MB/s 2024-04-02T07:34:42.179 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-02T07:34:42.227 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-02T07:34:42.228 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:34:42.228 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/scratch_devs of=/dev/stdout 2024-04-02T07:34:42.275 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-02T07:34:42.275 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_1 2024-04-02T07:34:42.362 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-02T07:34:42.362 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T07:34:42.362 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 445 Links: 1 2024-04-02T07:34:42.362 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T07:34:42.362 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T07:34:42.362 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-04-02 07:34:31.424258749 +0000 2024-04-02T07:34:42.362 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-04-02 07:31:25.369897019 +0000 2024-04-02T07:34:42.362 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-04-02 07:31:25.369897019 +0000 2024-04-02T07:34:42.362 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-04-02 07:31:25.342897551 +0000 2024-04-02T07:34:42.363 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-02T07:34:42.466 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-04-02T07:34:42.467 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-04-02T07:34:42.467 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000316738 s, 1.6 MB/s 2024-04-02T07:34:42.468 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-02T07:34:42.514 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_2 2024-04-02T07:34:42.599 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-02T07:34:42.600 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T07:34:42.600 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 451 Links: 1 2024-04-02T07:34:42.600 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T07:34:42.600 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T07:34:42.600 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-04-02 07:34:31.424258749 +0000 2024-04-02T07:34:42.601 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-04-02 07:31:25.845887651 +0000 2024-04-02T07:34:42.601 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-04-02 07:31:25.845887651 +0000 2024-04-02T07:34:42.601 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-04-02 07:31:25.821888123 +0000 2024-04-02T07:34:42.601 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-02T07:34:42.703 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-04-02T07:34:42.703 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-04-02T07:34:42.703 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000291393 s, 1.8 MB/s 2024-04-02T07:34:42.706 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-02T07:34:42.751 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_3 2024-04-02T07:34:42.835 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-02T07:34:42.835 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T07:34:42.835 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 457 Links: 1 2024-04-02T07:34:42.835 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T07:34:42.835 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T07:34:42.835 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-04-02 07:34:31.425258729 +0000 2024-04-02T07:34:42.835 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-04-02 07:31:26.322878262 +0000 2024-04-02T07:34:42.835 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-04-02 07:31:26.322878262 +0000 2024-04-02T07:34:42.835 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-04-02 07:31:26.294878813 +0000 2024-04-02T07:34:42.835 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-02T07:34:42.937 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-04-02T07:34:42.937 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-04-02T07:34:42.938 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.00045946 s, 1.1 MB/s 2024-04-02T07:34:42.940 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-02T07:34:42.986 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_4 2024-04-02T07:34:43.029 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-02T07:34:43.029 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T07:34:43.030 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 463 Links: 1 2024-04-02T07:34:43.030 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T07:34:43.030 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T07:34:43.030 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-04-02 07:34:31.425258729 +0000 2024-04-02T07:34:43.030 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-04-02 07:31:26.805868756 +0000 2024-04-02T07:34:43.030 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-04-02 07:31:26.805868756 +0000 2024-04-02T07:34:43.030 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-04-02 07:31:26.779869267 +0000 2024-04-02T07:34:43.030 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-02T07:34:43.132 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-04-02T07:34:43.133 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-04-02T07:34:43.133 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000473478 s, 1.1 MB/s 2024-04-02T07:34:43.135 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-02T07:34:43.181 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-02T07:34:43.181 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi113.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-02T07:34:43.181 INFO:tasks.ceph:Generating config... 2024-04-02T07:34:43.183 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-02T07:34:43.183 INFO:tasks.ceph:[client] debug client = 20 2024-04-02T07:34:43.183 INFO:tasks.ceph:[client] debug ms = 1 2024-04-02T07:34:43.184 INFO:tasks.ceph:[client] pid file = 2024-04-02T07:34:43.184 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-02T07:34:43.184 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-02T07:34:43.184 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-02T07:34:43.184 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-02T07:34:43.184 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-02T07:34:43.184 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-02T07:34:43.185 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-02T07:34:43.185 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-02T07:34:43.185 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-02T07:34:43.185 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-02T07:34:43.185 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-02T07:34:43.185 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-02T07:34:43.185 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-02T07:34:43.185 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-02T07:34:43.185 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-04-02T07:34:43.185 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-02T07:34:43.185 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-02T07:34:43.186 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-02T07:34:43.186 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-02T07:34:43.186 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-02T07:34:43.186 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-02T07:34:43.186 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-02T07:34:43.186 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-02T07:34:43.186 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-02T07:34:43.186 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-02T07:34:43.186 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-02T07:34:43.186 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-02T07:34:43.186 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-02T07:34:43.186 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-02T07:34:43.186 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-02T07:34:43.186 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-02T07:34:43.186 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-02T07:34:43.187 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-02T07:34:43.187 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-02T07:34:43.187 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-02T07:34:43.187 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-02T07:34:43.187 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-02T07:34:43.187 INFO:tasks.ceph:Setting up mon.a... 2024-04-02T07:34:43.187 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-02T07:34:43.319 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.keyring 2024-04-02T07:34:43.325 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-02T07:34:43.425 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-02T07:34:43.493 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.89'), ('mon.b', '172.21.15.113'), ('mon.c', '[v2:172.21.15.113:3301,v1:172.21.15.113:6790]')] 2024-04-02T07:34:43.494 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.89,172.21.15.113,[v2:172.21.15.113:3301,v1:172.21.15.113:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-02T07:34:43.495 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:34:43.495 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-02T07:34:43.540 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.89 --add b 172.21.15.113 --addv c '[v2:172.21.15.113:3301,v1:172.21.15.113:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-02T07:34:43.669 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-02T07:34:43.669 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: generated fsid ddf7474f-4630-4a1e-ada2-3982714d822b 2024-04-02T07:34:43.669 INFO:teuthology.orchestra.run.smithi089.stdout:setting min_mon_release = pacific 2024-04-02T07:34:43.669 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 0 2024-04-02T07:34:43.670 INFO:teuthology.orchestra.run.smithi089.stdout:fsid ddf7474f-4630-4a1e-ada2-3982714d822b 2024-04-02T07:34:43.670 INFO:teuthology.orchestra.run.smithi089.stdout:last_changed 2024-04-02T07:34:43.668348+0000 2024-04-02T07:34:43.670 INFO:teuthology.orchestra.run.smithi089.stdout:created 2024-04-02T07:34:43.668348+0000 2024-04-02T07:34:43.670 INFO:teuthology.orchestra.run.smithi089.stdout:min_mon_release 16 (pacific) 2024-04-02T07:34:43.670 INFO:teuthology.orchestra.run.smithi089.stdout:election_strategy: 1 2024-04-02T07:34:43.670 INFO:teuthology.orchestra.run.smithi089.stdout:0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2024-04-02T07:34:43.670 INFO:teuthology.orchestra.run.smithi089.stdout:1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.b 2024-04-02T07:34:43.670 INFO:teuthology.orchestra.run.smithi089.stdout:2: [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] mon.c 2024-04-02T07:34:43.670 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-02T07:34:43.672 DEBUG:teuthology.orchestra.run.smithi089:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-02T07:34:43.716 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ddf7474f-4630-4a1e-ada2-3982714d822b... 2024-04-02T07:34:43.717 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T07:34:43.761 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T07:34:43.874 INFO:teuthology.orchestra.run.smithi113.stdout:[global] 2024-04-02T07:34:43.874 INFO:teuthology.orchestra.run.smithi113.stdout: chdir = "" 2024-04-02T07:34:43.874 INFO:teuthology.orchestra.run.smithi113.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T07:34:43.874 INFO:teuthology.orchestra.run.smithi113.stdout: auth supported = cephx 2024-04-02T07:34:43.874 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.874 INFO:teuthology.orchestra.run.smithi113.stdout: filestore xattr use omap = true 2024-04-02T07:34:43.874 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.874 INFO:teuthology.orchestra.run.smithi113.stdout: mon clock drift allowed = 1.000 2024-04-02T07:34:43.874 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.874 INFO:teuthology.orchestra.run.smithi113.stdout: osd crush chooseleaf type = 0 2024-04-02T07:34:43.874 INFO:teuthology.orchestra.run.smithi113.stdout: auth debug = true 2024-04-02T07:34:43.874 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.875 INFO:teuthology.orchestra.run.smithi113.stdout: ms die on old message = true 2024-04-02T07:34:43.875 INFO:teuthology.orchestra.run.smithi113.stdout: ms die on bug = true 2024-04-02T07:34:43.875 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.875 INFO:teuthology.orchestra.run.smithi113.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T07:34:43.875 INFO:teuthology.orchestra.run.smithi113.stdout: mon pg warn max object skew = 0 2024-04-02T07:34:43.875 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.875 INFO:teuthology.orchestra.run.smithi113.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T07:34:43.875 INFO:teuthology.orchestra.run.smithi113.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T07:34:43.875 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.875 INFO:teuthology.orchestra.run.smithi113.stdout: osd pool default size = 2 2024-04-02T07:34:43.875 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.875 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd allow primary affinity = true 2024-04-02T07:34:43.875 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd allow pg remap = true 2024-04-02T07:34:43.875 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on legacy crush tunables = false 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on crush straw calc version zero = false 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on no sortbitwise = false 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on osd down out interval zero = false 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on too few osds = false 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: mon_allow_pool_size_one = true 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: osd default data pool replay window = 5 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: mon allow pool delete = true 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.876 INFO:teuthology.orchestra.run.smithi113.stdout: mon cluster log file level = debug 2024-04-02T07:34:43.877 INFO:teuthology.orchestra.run.smithi113.stdout: debug asserts on shutdown = true 2024-04-02T07:34:43.877 INFO:teuthology.orchestra.run.smithi113.stdout: mon health detail to clog = false 2024-04-02T07:34:43.877 INFO:teuthology.orchestra.run.smithi113.stdout: mon host = "172.21.15.89,172.21.15.113,[v2:172.21.15.113:3301,v1:172.21.15.113:6790]" 2024-04-02T07:34:43.877 INFO:teuthology.orchestra.run.smithi113.stdout: ms die on skipped message = False 2024-04-02T07:34:43.877 INFO:teuthology.orchestra.run.smithi113.stdout: fsid = ddf7474f-4630-4a1e-ada2-3982714d822b 2024-04-02T07:34:43.877 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.877 INFO:teuthology.orchestra.run.smithi113.stdout:[osd] 2024-04-02T07:34:43.877 INFO:teuthology.orchestra.run.smithi113.stdout: osd journal size = 100 2024-04-02T07:34:43.877 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.877 INFO:teuthology.orchestra.run.smithi113.stdout: osd scrub load threshold = 5.0 2024-04-02T07:34:43.877 INFO:teuthology.orchestra.run.smithi113.stdout: osd scrub max interval = 600 2024-04-02T07:34:43.877 INFO:teuthology.orchestra.run.smithi113.stdout: osd mclock profile = high_recovery_ops 2024-04-02T07:34:43.877 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.877 INFO:teuthology.orchestra.run.smithi113.stdout: osd recover clone overlap = true 2024-04-02T07:34:43.878 INFO:teuthology.orchestra.run.smithi113.stdout: osd recovery max chunk = 1048576 2024-04-02T07:34:43.878 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.878 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug shutdown = true 2024-04-02T07:34:43.878 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug op order = true 2024-04-02T07:34:43.878 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug verify stray on activate = true 2024-04-02T07:34:43.878 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.878 INFO:teuthology.orchestra.run.smithi113.stdout: osd open classes on start = true 2024-04-02T07:34:43.878 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug pg log writeout = true 2024-04-02T07:34:43.878 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.878 INFO:teuthology.orchestra.run.smithi113.stdout: osd deep scrub update digest min age = 30 2024-04-02T07:34:43.878 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.878 INFO:teuthology.orchestra.run.smithi113.stdout: osd map max advance = 10 2024-04-02T07:34:43.878 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.878 INFO:teuthology.orchestra.run.smithi113.stdout: journal zero on create = true 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: filestore ondisk finisher threads = 3 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: filestore apply finisher threads = 3 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: bdev debug aio = true 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug misdirected ops = true 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: bdev async discard = True 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: bdev enable discard = True 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: bluestore allocator = bitmap 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: bluestore block size = 96636764160 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: bluestore fsck on mount = True 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: debug bluefs = 1/20 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: debug bluestore = 1/20 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-04-02T07:34:43.879 INFO:teuthology.orchestra.run.smithi113.stdout: debug osd = 20 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout: debug rocksdb = 4/10 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd full ratio = 0.9 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd nearfull ratio = 0.8 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout: osd failsafe full ratio = 0.95 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout: osd objectstore = bluestore 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout: osd op complaint time = 180 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout:[mgr] 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout: debug mgr = 20 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout: debug mon = 20 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout: debug auth = 20 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min pgs per osd = 4 2024-04-02T07:34:43.880 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min bytes per osd = 10 2024-04-02T07:34:43.881 INFO:teuthology.orchestra.run.smithi113.stdout: mgr/telemetry/nag = false 2024-04-02T07:34:43.881 INFO:teuthology.orchestra.run.smithi113.stdout: debug client = 20 2024-04-02T07:34:43.881 INFO:teuthology.orchestra.run.smithi113.stdout: debug finisher = 20 2024-04-02T07:34:43.881 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.881 INFO:teuthology.orchestra.run.smithi113.stdout:[mon] 2024-04-02T07:34:43.881 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-04-02T07:34:43.881 INFO:teuthology.orchestra.run.smithi113.stdout: debug mon = 20 2024-04-02T07:34:43.881 INFO:teuthology.orchestra.run.smithi113.stdout: debug paxos = 20 2024-04-02T07:34:43.881 INFO:teuthology.orchestra.run.smithi113.stdout: debug auth = 20 2024-04-02T07:34:43.881 INFO:teuthology.orchestra.run.smithi113.stdout: mon data avail warn = 5 2024-04-02T07:34:43.881 INFO:teuthology.orchestra.run.smithi113.stdout: mon mgr mkfs grace = 240 2024-04-02T07:34:43.881 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min pgs per osd = 4 2024-04-02T07:34:43.881 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd reporter subtree level = osd 2024-04-02T07:34:43.881 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd prime pg temp = true 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min bytes per osd = 10 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: # 1m isn't quite enough 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: mon_down_mkfs_grace = 2m 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_filestore_osds = false 2024-04-02T07:34:43.882 INFO:teuthology.orchestra.run.smithi113.stdout: mon op complaint time = 120 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout:[client] 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout: rgw cache enabled = true 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout: rgw enable ops log = true 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout: rgw enable usage log = true 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout: client mount timeout = 600 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout: debug client = 20 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout: pid file = "" 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout: rados mon op timeout = 900 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout: rados osd op timeout = 900 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout:[mon.a] 2024-04-02T07:34:43.883 INFO:teuthology.orchestra.run.smithi113.stdout:[mon.b] 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout:[mon.c] 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.a] 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.c] 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.b] 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.d] 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout:[mds] 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout: debug mds = 20 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout: debug mds balancer = 20 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout: mds debug frag = True 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout: mds debug scatterstat = True 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout: mds op complaint time = 180 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout: mds verify scatter = True 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout: osd op complaint time = 180 2024-04-02T07:34:43.884 INFO:teuthology.orchestra.run.smithi113.stdout: rados mon op timeout = 900 2024-04-02T07:34:43.885 INFO:teuthology.orchestra.run.smithi113.stdout: rados osd op timeout = 900 2024-04-02T07:34:43.885 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2024-04-02T07:34:43.885 INFO:teuthology.orchestra.run.smithi089.stdout: chdir = "" 2024-04-02T07:34:43.885 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T07:34:43.885 INFO:teuthology.orchestra.run.smithi089.stdout: auth supported = cephx 2024-04-02T07:34:43.885 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.885 INFO:teuthology.orchestra.run.smithi089.stdout: filestore xattr use omap = true 2024-04-02T07:34:43.885 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.886 INFO:teuthology.orchestra.run.smithi089.stdout: mon clock drift allowed = 1.000 2024-04-02T07:34:43.886 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.886 INFO:teuthology.orchestra.run.smithi089.stdout: osd crush chooseleaf type = 0 2024-04-02T07:34:43.886 INFO:teuthology.orchestra.run.smithi089.stdout: auth debug = true 2024-04-02T07:34:43.886 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.886 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on old message = true 2024-04-02T07:34:43.886 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on bug = true 2024-04-02T07:34:43.886 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.886 INFO:teuthology.orchestra.run.smithi089.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T07:34:43.886 INFO:teuthology.orchestra.run.smithi089.stdout: mon pg warn max object skew = 0 2024-04-02T07:34:43.886 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.886 INFO:teuthology.orchestra.run.smithi089.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T07:34:43.886 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T07:34:43.886 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default size = 2 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow primary affinity = true 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow pg remap = true 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on legacy crush tunables = false 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on crush straw calc version zero = false 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on no sortbitwise = false 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on osd down out interval zero = false 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on too few osds = false 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: mon_allow_pool_size_one = true 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T07:34:43.887 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.888 INFO:teuthology.orchestra.run.smithi089.stdout: osd default data pool replay window = 5 2024-04-02T07:34:43.888 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.888 INFO:teuthology.orchestra.run.smithi089.stdout: mon allow pool delete = true 2024-04-02T07:34:43.888 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.888 INFO:teuthology.orchestra.run.smithi089.stdout: mon cluster log file level = debug 2024-04-02T07:34:43.888 INFO:teuthology.orchestra.run.smithi089.stdout: debug asserts on shutdown = true 2024-04-02T07:34:43.888 INFO:teuthology.orchestra.run.smithi089.stdout: mon health detail to clog = false 2024-04-02T07:34:43.888 INFO:teuthology.orchestra.run.smithi089.stdout: mon host = "172.21.15.89,172.21.15.113,[v2:172.21.15.113:3301,v1:172.21.15.113:6790]" 2024-04-02T07:34:43.888 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on skipped message = False 2024-04-02T07:34:43.888 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = ddf7474f-4630-4a1e-ada2-3982714d822b 2024-04-02T07:34:43.888 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.888 INFO:teuthology.orchestra.run.smithi089.stdout:[osd] 2024-04-02T07:34:43.888 INFO:teuthology.orchestra.run.smithi089.stdout: osd journal size = 100 2024-04-02T07:34:43.888 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub load threshold = 5.0 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub max interval = 600 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock profile = high_recovery_ops 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: osd recover clone overlap = true 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: osd recovery max chunk = 1048576 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug shutdown = true 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug op order = true 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify stray on activate = true 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: osd open classes on start = true 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug pg log writeout = true 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.889 INFO:teuthology.orchestra.run.smithi089.stdout: osd deep scrub update digest min age = 30 2024-04-02T07:34:43.890 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.890 INFO:teuthology.orchestra.run.smithi089.stdout: osd map max advance = 10 2024-04-02T07:34:43.890 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.890 INFO:teuthology.orchestra.run.smithi089.stdout: journal zero on create = true 2024-04-02T07:34:43.890 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.890 INFO:teuthology.orchestra.run.smithi089.stdout: filestore ondisk finisher threads = 3 2024-04-02T07:34:43.890 INFO:teuthology.orchestra.run.smithi089.stdout: filestore apply finisher threads = 3 2024-04-02T07:34:43.890 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.890 INFO:teuthology.orchestra.run.smithi089.stdout: bdev debug aio = true 2024-04-02T07:34:43.890 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug misdirected ops = true 2024-04-02T07:34:43.890 INFO:teuthology.orchestra.run.smithi089.stdout: bdev async discard = True 2024-04-02T07:34:43.890 INFO:teuthology.orchestra.run.smithi089.stdout: bdev enable discard = True 2024-04-02T07:34:43.890 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore allocator = bitmap 2024-04-02T07:34:43.890 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore block size = 96636764160 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore fsck on mount = True 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluefs = 1/20 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluestore = 1/20 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: debug osd = 20 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: debug rocksdb = 4/10 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd full ratio = 0.9 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd nearfull ratio = 0.8 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: osd failsafe full ratio = 0.95 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: osd objectstore = bluestore 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr] 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-04-02T07:34:43.891 INFO:teuthology.orchestra.run.smithi089.stdout: debug mgr = 20 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: mgr/telemetry/nag = false 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: debug finisher = 20 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout:[mon] 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: debug paxos = 20 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: mon data avail warn = 5 2024-04-02T07:34:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: mon mgr mkfs grace = 240 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd reporter subtree level = osd 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd prime pg temp = true 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: # 1m isn't quite enough 2024-04-02T07:34:43.893 INFO:teuthology.orchestra.run.smithi089.stdout: mon_down_mkfs_grace = 2m 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_filestore_osds = false 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout: mon op complaint time = 120 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout:[client] 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout: rgw cache enabled = true 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable ops log = true 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable usage log = true 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout: client mount timeout = 600 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = "" 2024-04-02T07:34:43.894 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.a] 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.b] 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.c] 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.a] 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.c] 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.b] 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.d] 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout:[mds] 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds = 20 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds balancer = 20 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug frag = True 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug scatterstat = True 2024-04-02T07:34:43.895 INFO:teuthology.orchestra.run.smithi089.stdout: mds op complaint time = 180 2024-04-02T07:34:43.896 INFO:teuthology.orchestra.run.smithi089.stdout: mds verify scatter = True 2024-04-02T07:34:43.896 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2024-04-02T07:34:43.896 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2024-04-02T07:34:43.896 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2024-04-02T07:34:43.901 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-02T07:34:43.901 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-02T07:34:44.012 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-02T07:34:44.012 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:34:44.013 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-02T07:34:44.060 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:34:44.060 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-02T07:34:44.148 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:34:44.148 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:34:44.148 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-02T07:34:44.148 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-02T07:34:44.282 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:34:44.282 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-02T07:34:44.366 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi113.front.sepia.ceph.com 2024-04-02T07:34:44.366 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:34:44.367 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-02T07:34:44.367 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-02T07:34:44.456 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:34:44.457 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-02T07:34:44.541 INFO:tasks.ceph:Setting up mon nodes... 2024-04-02T07:34:44.541 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-02T07:34:44.541 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-02T07:34:44.662 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-02T07:34:44.671 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-02T07:34:45.288 INFO:teuthology.orchestra.run.smithi113.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-02T07:34:45.295 INFO:tasks.ceph:Setting up mds nodes... 2024-04-02T07:34:45.296 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-02T07:34:45.413 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-02T07:34:45.418 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-02T07:34:45.485 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-02T07:34:45.600 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-02T07:34:45.605 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-02T07:34:45.670 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-02T07:34:45.794 INFO:teuthology.orchestra.run.smithi113.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-02T07:34:45.800 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-02T07:34:45.865 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-02T07:34:45.981 INFO:teuthology.orchestra.run.smithi113.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-02T07:34:45.986 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-02T07:34:46.052 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-02T07:34:46.052 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-02T07:34:46.145 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-02T07:34:46.174 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-02T07:34:46.266 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-02T07:34:46.295 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-02T07:34:46.392 INFO:teuthology.orchestra.run.smithi113.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-02T07:34:46.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-02T07:34:46.514 INFO:teuthology.orchestra.run.smithi113.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-02T07:34:46.543 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-02T07:34:46.543 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi113.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-02T07:34:46.543 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-02T07:34:46.608 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-02T07:34:46.609 INFO:tasks.ceph:role: osd.0 2024-04-02T07:34:46.609 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:34:46.609 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-02T07:34:46.681 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T07:34:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T07:34:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T07:34:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-02T07:34:46.681 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T07:34:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-04-02T07:34:46.681 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T07:34:46.682 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-02T07:34:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T07:34:46.682 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T07:34:47.116 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-04-02T07:34:47.119 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-04-02T07:34:47.119 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-02T07:34:47.325 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-02T07:34:47.399 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-02T07:34:47.464 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-02T07:34:47.464 INFO:tasks.ceph:role: osd.1 2024-04-02T07:34:47.464 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:34:47.464 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-02T07:34:47.537 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T07:34:47.537 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T07:34:47.537 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T07:34:47.537 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-02T07:34:47.537 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T07:34:47.537 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-04-02T07:34:47.538 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T07:34:47.538 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-02T07:34:47.538 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T07:34:47.538 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T07:34:47.945 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-04-02T07:34:47.948 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-04-02T07:34:47.949 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-02T07:34:48.080 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-02T07:34:48.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-02T07:34:48.217 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-02T07:34:48.217 INFO:tasks.ceph:role: osd.2 2024-04-02T07:34:48.217 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:34:48.217 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-02T07:34:48.290 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T07:34:48.290 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T07:34:48.291 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T07:34:48.291 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-02T07:34:48.291 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T07:34:48.291 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-04-02T07:34:48.291 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T07:34:48.291 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-02T07:34:48.291 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T07:34:48.291 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T07:34:48.752 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-04-02T07:34:48.756 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-04-02T07:34:48.756 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-02T07:34:48.941 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-02T07:34:49.014 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-02T07:34:49.077 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-02T07:34:49.078 INFO:tasks.ceph:role: osd.3 2024-04-02T07:34:49.078 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:34:49.078 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-02T07:34:49.149 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T07:34:49.150 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T07:34:49.150 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T07:34:49.150 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-02T07:34:49.150 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T07:34:49.150 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-04-02T07:34:49.150 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T07:34:49.150 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-02T07:34:49.150 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T07:34:49.150 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T07:34:49.580 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-04-02T07:34:49.584 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-04-02T07:34:49.584 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-02T07:34:49.734 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-02T07:34:49.805 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T07:34:50.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:50.170+0000 7eff23a98680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-02T07:34:50.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:50.170+0000 7eff23a98680 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-02T07:34:50.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:50.170+0000 7eff23a98680 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-02T07:34:50.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:50.170+0000 7eff23a98680 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-02T07:34:50.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:50.170+0000 7eff23a98680 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-02T07:34:50.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:50.172+0000 7eff23a98680 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-02T07:34:53.070 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-02T07:34:53.141 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T07:34:53.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:53.240+0000 7fd60186b680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-02T07:34:53.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:53.240+0000 7fd60186b680 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-02T07:34:53.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:53.240+0000 7fd60186b680 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-02T07:34:53.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:53.240+0000 7fd60186b680 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-02T07:34:53.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:53.240+0000 7fd60186b680 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-02T07:34:53.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:53.243+0000 7fd60186b680 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-02T07:34:56.129 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-02T07:34:56.197 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T07:34:56.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:56.297+0000 7f02dfafa680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-02T07:34:56.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:56.298+0000 7f02dfafa680 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-02T07:34:56.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:56.298+0000 7f02dfafa680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-02T07:34:56.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:56.298+0000 7f02dfafa680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-02T07:34:56.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:56.298+0000 7f02dfafa680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-02T07:34:56.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:56.299+0000 7f02dfafa680 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-02T07:34:59.188 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-02T07:34:59.257 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T07:34:59.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:59.356+0000 7f5426468680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-02T07:34:59.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:59.356+0000 7f5426468680 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-02T07:34:59.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:59.356+0000 7f5426468680 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-02T07:34:59.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:59.356+0000 7f5426468680 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-02T07:34:59.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:59.356+0000 7f5426468680 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-02T07:34:59.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:34:59.358+0000 7f5426468680 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-02T07:35:02.239 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-02T07:35:02.306 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-02T07:35:02.379 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-02T07:35:02.379 INFO:tasks.ceph:role: osd.4 2024-04-02T07:35:02.380 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi113.front.sepia.ceph.com 2024-04-02T07:35:02.380 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-02T07:35:02.454 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T07:35:02.454 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T07:35:02.454 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T07:35:02.455 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-02T07:35:02.455 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T07:35:02.455 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2024-04-02T07:35:02.455 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T07:35:02.455 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-02T07:35:02.455 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T07:35:02.455 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T07:35:02.938 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2024-04-02T07:35:02.942 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2024-04-02T07:35:02.942 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-02T07:35:03.109 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-02T07:35:03.185 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-02T07:35:03.252 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-02T07:35:03.252 INFO:tasks.ceph:role: osd.5 2024-04-02T07:35:03.252 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi113.front.sepia.ceph.com 2024-04-02T07:35:03.252 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-02T07:35:03.325 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T07:35:03.325 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T07:35:03.325 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T07:35:03.325 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-02T07:35:03.325 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T07:35:03.325 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2024-04-02T07:35:03.325 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T07:35:03.325 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-02T07:35:03.326 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T07:35:03.326 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T07:35:03.789 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2024-04-02T07:35:03.793 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2024-04-02T07:35:03.793 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-02T07:35:03.923 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-02T07:35:03.996 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-02T07:35:04.061 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-02T07:35:04.061 INFO:tasks.ceph:role: osd.6 2024-04-02T07:35:04.061 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi113.front.sepia.ceph.com 2024-04-02T07:35:04.062 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-02T07:35:04.135 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T07:35:04.135 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T07:35:04.135 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T07:35:04.135 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-02T07:35:04.135 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T07:35:04.136 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2024-04-02T07:35:04.136 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T07:35:04.136 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-02T07:35:04.136 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T07:35:04.136 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T07:35:04.631 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2024-04-02T07:35:04.636 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2024-04-02T07:35:04.636 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-02T07:35:04.796 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-02T07:35:04.871 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-02T07:35:04.937 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-02T07:35:04.937 INFO:tasks.ceph:role: osd.7 2024-04-02T07:35:04.937 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi113.front.sepia.ceph.com 2024-04-02T07:35:04.938 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-02T07:35:05.010 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T07:35:05.011 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T07:35:05.011 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T07:35:05.011 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-02T07:35:05.011 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T07:35:05.011 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2024-04-02T07:35:05.011 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T07:35:05.011 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-02T07:35:05.011 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T07:35:05.011 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T07:35:05.503 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2024-04-02T07:35:05.507 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2024-04-02T07:35:05.507 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-02T07:35:05.650 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-02T07:35:05.724 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T07:35:05.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:05.838+0000 7f86c8a3f680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-02T07:35:05.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:05.840+0000 7f86c8a3f680 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-02T07:35:05.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:05.840+0000 7f86c8a3f680 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-02T07:35:05.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:05.840+0000 7f86c8a3f680 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-02T07:35:05.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:05.840+0000 7f86c8a3f680 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-02T07:35:05.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:05.850+0000 7f86c8a3f680 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-02T07:35:08.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-02T07:35:08.812 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T07:35:08.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:08.911+0000 7fa6d1765680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-02T07:35:08.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:08.912+0000 7fa6d1765680 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-02T07:35:08.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:08.912+0000 7fa6d1765680 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-02T07:35:08.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:08.912+0000 7fa6d1765680 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-02T07:35:08.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:08.912+0000 7fa6d1765680 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-02T07:35:08.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:08.913+0000 7fa6d1765680 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-02T07:35:11.098 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:35:11.103 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:35:11.805 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-02T07:35:11.871 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T07:35:11.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:11.969+0000 7f5c699bf680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-02T07:35:11.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:11.969+0000 7f5c699bf680 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-02T07:35:11.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:11.969+0000 7f5c699bf680 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-02T07:35:11.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:11.969+0000 7f5c699bf680 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-02T07:35:11.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:11.969+0000 7f5c699bf680 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-02T07:35:11.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:11.971+0000 7f5c699bf680 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-02T07:35:14.852 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-02T07:35:14.919 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T07:35:15.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:15.020+0000 7fd4a0f3d680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-02T07:35:15.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:15.020+0000 7fd4a0f3d680 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-02T07:35:15.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:15.020+0000 7fd4a0f3d680 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-02T07:35:15.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:15.020+0000 7fd4a0f3d680 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-02T07:35:15.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:15.020+0000 7fd4a0f3d680 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-02T07:35:15.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:15.022+0000 7fd4a0f3d680 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-02T07:35:17.903 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-02T07:35:17.971 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-02T07:35:17.971 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:35:17.971 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-02T07:35:18.044 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:35:18.044 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-02T07:35:18.112 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:35:18.112 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-02T07:35:18.219 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:35:18.219 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-02T07:35:18.324 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:35:18.324 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-02T07:35:18.430 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:35:18.430 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-02T07:35:18.535 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:35:18.535 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-02T07:35:18.640 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:35:18.640 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-02T07:35:18.708 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:35:18.708 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-02T07:35:18.814 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:35:18.814 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-02T07:35:18.919 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:35:18.919 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-02T07:35:19.024 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:35:19.024 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-02T07:35:19.131 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:35:19.131 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-02T07:35:19.235 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:35:19.235 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-02T07:35:19.340 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:35:19.340 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-02T07:35:19.385 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:35:19.385 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-02T07:35:19.469 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:35:19.469 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-02T07:35:19.513 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:35:19.513 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-02T07:35:19.596 INFO:tasks.ceph:Adding keys to all mons... 2024-04-02T07:35:19.596 DEBUG:teuthology.orchestra.run.smithi089:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-02T07:35:19.600 DEBUG:teuthology.orchestra.run.smithi113:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-02T07:35:19.658 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.y] 2024-04-02T07:35:19.658 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA== 2024-04-02T07:35:19.658 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.a] 2024-04-02T07:35:19.658 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q== 2024-04-02T07:35:19.658 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.c] 2024-04-02T07:35:19.658 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g== 2024-04-02T07:35:19.659 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.0] 2024-04-02T07:35:19.659 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg== 2024-04-02T07:35:19.659 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.1] 2024-04-02T07:35:19.659 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w== 2024-04-02T07:35:19.659 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.2] 2024-04-02T07:35:19.659 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA== 2024-04-02T07:35:19.659 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.3] 2024-04-02T07:35:19.659 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg== 2024-04-02T07:35:19.659 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.x] 2024-04-02T07:35:19.659 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog== 2024-04-02T07:35:19.659 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.b] 2024-04-02T07:35:19.659 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw== 2024-04-02T07:35:19.659 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.d] 2024-04-02T07:35:19.660 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA== 2024-04-02T07:35:19.660 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.4] 2024-04-02T07:35:19.660 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg== 2024-04-02T07:35:19.660 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.5] 2024-04-02T07:35:19.660 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q== 2024-04-02T07:35:19.660 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.6] 2024-04-02T07:35:19.660 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw== 2024-04-02T07:35:19.660 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.7] 2024-04-02T07:35:19.660 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q== 2024-04-02T07:35:19.660 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2024-04-02T07:35:19.660 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ== 2024-04-02T07:35:19.660 INFO:teuthology.orchestra.run.smithi089.stdout:[client.2] 2024-04-02T07:35:19.661 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg== 2024-04-02T07:35:19.661 INFO:teuthology.orchestra.run.smithi089.stdout:[client.1] 2024-04-02T07:35:19.661 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A== 2024-04-02T07:35:19.661 INFO:teuthology.orchestra.run.smithi089.stdout:[client.3] 2024-04-02T07:35:19.661 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ== 2024-04-02T07:35:19.698 INFO:teuthology.orchestra.run.smithi113.stdout:[mgr.y] 2024-04-02T07:35:19.699 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA== 2024-04-02T07:35:19.699 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.a] 2024-04-02T07:35:19.699 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q== 2024-04-02T07:35:19.699 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.c] 2024-04-02T07:35:19.699 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g== 2024-04-02T07:35:19.699 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.0] 2024-04-02T07:35:19.699 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg== 2024-04-02T07:35:19.699 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.1] 2024-04-02T07:35:19.699 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w== 2024-04-02T07:35:19.699 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.2] 2024-04-02T07:35:19.699 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA== 2024-04-02T07:35:19.699 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.3] 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg== 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout:[mgr.x] 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog== 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.b] 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw== 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.d] 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA== 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.4] 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg== 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.5] 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q== 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.6] 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw== 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.7] 2024-04-02T07:35:19.700 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q== 2024-04-02T07:35:19.701 INFO:teuthology.orchestra.run.smithi113.stdout:[client.0] 2024-04-02T07:35:19.701 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ== 2024-04-02T07:35:19.701 INFO:teuthology.orchestra.run.smithi113.stdout:[client.2] 2024-04-02T07:35:19.701 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg== 2024-04-02T07:35:19.701 INFO:teuthology.orchestra.run.smithi113.stdout:[client.1] 2024-04-02T07:35:19.701 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A== 2024-04-02T07:35:19.701 INFO:teuthology.orchestra.run.smithi113.stdout:[client.3] 2024-04-02T07:35:19.701 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ== 2024-04-02T07:35:19.702 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-02T07:35:19.706 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-02T07:35:19.846 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T07:35:19.850 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T07:35:19.959 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T07:35:19.963 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T07:35:20.070 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:20.074 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:20.178 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:20.182 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:20.287 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:20.291 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:20.394 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:20.398 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:20.502 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-02T07:35:20.506 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-02T07:35:20.610 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T07:35:20.613 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T07:35:20.720 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T07:35:20.724 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-02T07:35:20.830 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:20.834 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:20.938 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:20.943 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:21.046 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:21.050 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:21.158 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:21.162 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T07:35:21.266 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T07:35:21.269 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T07:35:21.377 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T07:35:21.381 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T07:35:21.487 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T07:35:21.491 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T07:35:21.598 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T07:35:21.602 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T07:35:21.709 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-02T07:35:21.709 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-02T07:35:21.775 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-02T07:35:21.894 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-02T07:35:21.960 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-02T07:35:22.026 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-02T07:35:22.145 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-02T07:35:22.211 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-02T07:35:22.276 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-02T07:35:22.394 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-02T07:35:22.460 DEBUG:teuthology.orchestra.run.smithi089:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-02T07:35:22.464 DEBUG:teuthology.orchestra.run.smithi113:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-02T07:35:22.507 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-02T07:35:22.507 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-02T07:35:22.508 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-02T07:35:22.549 INFO:tasks.ceph.mon.a:Started 2024-04-02T07:35:22.550 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-02T07:35:22.550 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-02T07:35:22.554 INFO:tasks.ceph.mon.b:Started 2024-04-02T07:35:22.554 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-02T07:35:22.554 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-02T07:35:22.557 INFO:tasks.ceph.mon.c:Started 2024-04-02T07:35:22.558 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-02T07:35:22.558 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-02T07:35:22.558 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-02T07:35:22.561 INFO:tasks.ceph.mgr.y:Started 2024-04-02T07:35:22.561 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-02T07:35:22.562 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-02T07:35:22.565 INFO:tasks.ceph.mgr.x:Started 2024-04-02T07:35:22.566 INFO:tasks.ceph:Setting crush tunables to default 2024-04-02T07:35:22.566 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd crush tunables default 2024-04-02T07:35:22.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.753+0000 7fcffe600700 1 Processor -- start 2024-04-02T07:35:22.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.753+0000 7fcffe600700 1 -- start start 2024-04-02T07:35:22.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.754+0000 7fcffe600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 0x7fcff81687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:22.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.754+0000 7fcffe600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff8161420 con 0x7fcff80d16a0 2024-04-02T07:35:22.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.754+0000 7fcffe600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff8161700 con 0x7fcff815d9d0 2024-04-02T07:35:22.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.755+0000 7fcffe600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcff80d4340 con 0x7fcff80d2540 2024-04-02T07:35:22.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.755+0000 7fcff6a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcff80d16a0 0x7fcff80d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:37580/0 (socket says 172.21.15.89:37580) 2024-04-02T07:35:22.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.755+0000 7fcff6a00700 1 -- 172.21.15.89:0/3077075939 learned_addr learned my addr 172.21.15.89:0/3077075939 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:22.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.755+0000 7fcff7e00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 0x7fcff81687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:22.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.755+0000 7fcff7e00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 msgr2=0x7fcff81687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-02T07:35:22.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.755+0000 7fcff7e00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 msgr2=0x7fcff81687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T07:35:22.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.755+0000 7fcff7e00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 0x7fcff81687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T07:35:22.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.755+0000 7fcff7e00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 0x7fcff81687b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:35:22.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.956+0000 7fcff7e00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 0x7fcff81687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:22.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.956+0000 7fcff7e00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 msgr2=0x7fcff81687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-02T07:35:22.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.956+0000 7fcff7e00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 msgr2=0x7fcff81687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T07:35:22.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.956+0000 7fcff7e00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 0x7fcff81687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T07:35:22.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:22.956+0000 7fcff7e00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 0x7fcff81687b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:35:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:23.357+0000 7fcff7e00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 0x7fcff81687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:23.357+0000 7fcff7e00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 msgr2=0x7fcff81687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-02T07:35:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:23.357+0000 7fcff7e00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 msgr2=0x7fcff81687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T07:35:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:23.357+0000 7fcff7e00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 0x7fcff81687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T07:35:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:23.357+0000 7fcff7e00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 0x7fcff81687b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T07:35:24.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:24.158+0000 7fcff7e00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 0x7fcff81687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:24.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:24.158+0000 7fcff7e00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 msgr2=0x7fcff81687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-02T07:35:24.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:24.158+0000 7fcff7e00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 msgr2=0x7fcff81687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T07:35:24.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:24.158+0000 7fcff7e00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 0x7fcff81687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T07:35:24.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:24.158+0000 7fcff7e00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 0x7fcff81687b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T07:35:25.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.754+0000 7fcff4c00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 msgr2=0x7fcff81687b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T07:35:25.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.754+0000 7fcff4c00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.89:3300/0 conn(0x7fcff80d2540 0x7fcff81687b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:25.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.754+0000 7fcff4c00700 1 -- 172.21.15.89:0/3077075939 >> v1:172.21.15.113:6789/0 conn(0x7fcff815d9d0 legacy=0x7fcff80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:25.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.754+0000 7fcff4c00700 1 -- 172.21.15.89:0/3077075939 >> v1:172.21.15.89:6789/0 conn(0x7fcff80d16a0 legacy=0x7fcff80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:25.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.754+0000 7fcff4c00700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 0x7fcfd8005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:25.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.754+0000 7fcff4c00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 0x7fcfd8008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:25.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.754+0000 7fcff4c00700 1 -- 172.21.15.89:0/3077075939 --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd8008db0 con 0x7fcfd8006280 2024-04-02T07:35:25.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.754+0000 7fcff4c00700 1 -- 172.21.15.89:0/3077075939 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd8008f20 con 0x7fcfd80037a0 2024-04-02T07:35:25.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.755+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 0x7fcfd8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:25.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.755+0000 7fcff6a00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 0x7fcfd8008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:25.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.755+0000 7fcff7400700 1 -- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 msgr2=0x7fcfd8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-02T07:35:25.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.755+0000 7fcff7400700 1 -- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 msgr2=0x7fcfd8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T07:35:25.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.755+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 0x7fcfd8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T07:35:25.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.755+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 0x7fcfd8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:35:25.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.755+0000 7fcff6a00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 msgr2=0x7fcfd8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-02T07:35:25.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.755+0000 7fcff6a00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 msgr2=0x7fcfd8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T07:35:25.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.755+0000 7fcff6a00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 0x7fcfd8008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T07:35:25.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.755+0000 7fcff6a00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 0x7fcfd8008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:35:25.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.956+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 0x7fcfd8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:25.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.956+0000 7fcff6a00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 0x7fcfd8008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:25.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.956+0000 7fcff7400700 1 -- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 msgr2=0x7fcfd8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-02T07:35:25.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.956+0000 7fcff7400700 1 -- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 msgr2=0x7fcfd8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T07:35:25.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.956+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 0x7fcfd8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T07:35:25.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.956+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 0x7fcfd8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:35:25.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.956+0000 7fcff6a00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 msgr2=0x7fcfd8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-02T07:35:25.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.956+0000 7fcff6a00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 msgr2=0x7fcfd8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T07:35:25.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.956+0000 7fcff6a00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 0x7fcfd8008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T07:35:25.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:25.956+0000 7fcff6a00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 0x7fcfd8008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:35:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:26.357+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 0x7fcfd8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:26.357+0000 7fcff6a00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 0x7fcfd8008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:26.357+0000 7fcff7400700 1 -- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 msgr2=0x7fcfd8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-02T07:35:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:26.357+0000 7fcff7400700 1 -- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 msgr2=0x7fcfd8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T07:35:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:26.357+0000 7fcff6a00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 msgr2=0x7fcfd8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-02T07:35:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:26.357+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 0x7fcfd8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T07:35:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:26.357+0000 7fcff6a00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 msgr2=0x7fcfd8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T07:35:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:26.357+0000 7fcff6a00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 0x7fcfd8008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T07:35:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:26.357+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 0x7fcfd8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T07:35:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:26.357+0000 7fcff6a00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 0x7fcfd8008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T07:35:27.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:27.159+0000 7fcff6a00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 0x7fcfd8008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:27.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:27.159+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 0x7fcfd8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:27.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:27.159+0000 7fcff6a00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 msgr2=0x7fcfd8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-02T07:35:27.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:27.159+0000 7fcff7400700 1 -- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 msgr2=0x7fcfd8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-02T07:35:27.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:27.159+0000 7fcff6a00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 msgr2=0x7fcfd8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T07:35:27.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:27.159+0000 7fcff7400700 1 -- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 msgr2=0x7fcfd8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-02T07:35:27.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:27.159+0000 7fcff6a00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 0x7fcfd8008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T07:35:27.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:27.159+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 0x7fcfd8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T07:35:27.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:27.159+0000 7fcff6a00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 0x7fcfd8008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T07:35:27.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:27.159+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 0x7fcfd8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T07:35:28.135 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:28.132+0000 7f826b9b7240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:35:28.135 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:28.133+0000 7f016525e240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:35:28.340 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:28.338+0000 7f016525e240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:35:28.344 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:28.341+0000 7f826b9b7240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:35:28.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.754+0000 7fcff4c00700 1 -- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 msgr2=0x7fcfd8005bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T07:35:28.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.754+0000 7fcff4c00700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd80037a0 0x7fcfd8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:28.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.754+0000 7fcff4c00700 1 -- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 msgr2=0x7fcfd8008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T07:35:28.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.754+0000 7fcff4c00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8006280 0x7fcfd8008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:28.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.754+0000 7fcff4c00700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8009320 0x7fcfd800b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:28.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.754+0000 7fcff7e00700 1 -- 172.21.15.89:0/3077075939 reap_dead start 2024-04-02T07:35:28.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.754+0000 7fcff4c00700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd8007e40 0x7fcfd80082d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:28.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.754+0000 7fcff4c00700 1 -- 172.21.15.89:0/3077075939 --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff81619e0 con 0x7fcfd8003be0 2024-04-02T07:35:28.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.754+0000 7fcff4c00700 1 -- 172.21.15.89:0/3077075939 --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd8008db0 con 0x7fcfd8009320 2024-04-02T07:35:28.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.754+0000 7fcff4c00700 1 -- 172.21.15.89:0/3077075939 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd8008f20 con 0x7fcfd8007e40 2024-04-02T07:35:28.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.755+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> v2:172.21.15.113:3300/0 conn(0x7fcfd8009320 0x7fcfd800b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:28.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.755+0000 7fcff7e00700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd8007e40 0x7fcfd80082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:28.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.755+0000 7fcff7400700 1 -- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd8007e40 msgr2=0x7fcfd80082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:28.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.755+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd8007e40 0x7fcfd80082d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:28.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.755+0000 7fcff7400700 1 -- 172.21.15.89:0/3077075939 >> v1:172.21.15.113:6789/0 conn(0x7fcfd8003be0 legacy=0x7fcfd80066c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:28.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.755+0000 7fcff7400700 1 -- 172.21.15.89:0/3077075939 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd800ed10 con 0x7fcfd8009320 2024-04-02T07:35:28.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.755+0000 7fcff6000700 1 -- 172.21.15.89:0/3077075939 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411951962 0 0) 0x7fcff81619e0 con 0x7fcfd8003be0 2024-04-02T07:35:28.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.755+0000 7fcff7400700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfd8009320 0x7fcfd800b710 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8007cc0 tx=0x7fcfe8005610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9b7497f599b3f3a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:28.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.755+0000 7fcff6000700 1 -- 172.21.15.89:0/3077075939 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8016450 con 0x7fcfd8009320 2024-04-02T07:35:28.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.755+0000 7fcff6000700 1 -- 172.21.15.89:0/3077075939 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe80165d0 con 0x7fcfd8009320 2024-04-02T07:35:28.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.755+0000 7fcff6000700 1 -- 172.21.15.89:0/3077075939 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8016980 con 0x7fcfd8009320 2024-04-02T07:35:28.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.756+0000 7fcffe600700 1 -- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfd8009320 msgr2=0x7fcfd800b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:28.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.756+0000 7fcffe600700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfd8009320 0x7fcfd800b710 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8007cc0 tx=0x7fcfe8005610 comp rx=0 tx=0).stop 2024-04-02T07:35:28.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.756+0000 7fcffe600700 1 -- 172.21.15.89:0/3077075939 shutdown_connections 2024-04-02T07:35:28.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.756+0000 7fcffe600700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd8007e40 0x7fcfd80082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:28.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.756+0000 7fcffe600700 1 --2- 172.21.15.89:0/3077075939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfd8009320 0x7fcfd800b710 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:28.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.756+0000 7fcffe600700 1 -- 172.21.15.89:0/3077075939 >> 172.21.15.89:0/3077075939 conn(0x7fcff808aff0 msgr2=0x7fcff80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:28.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.756+0000 7fcffe600700 1 -- 172.21.15.89:0/3077075939 shutdown_connections 2024-04-02T07:35:28.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.756+0000 7fcffe600700 1 -- 172.21.15.89:0/3077075939 wait complete. 2024-04-02T07:35:28.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.758+0000 7fcffe600700 1 Processor -- start 2024-04-02T07:35:28.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.758+0000 7fcffe600700 1 -- start start 2024-04-02T07:35:28.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.759+0000 7fcffe600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd8003be0 0x7fcff814f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:28.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.759+0000 7fcffe600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd8007e40 0x7fcff815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.759+0000 7fcffe600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfd8009320 0x7fcff80b8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.759+0000 7fcffe600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff80cc720 con 0x7fcfd8007e40 2024-04-02T07:35:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.759+0000 7fcffe600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff815e600 con 0x7fcfd8009320 2024-04-02T07:35:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.759+0000 7fcffe600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff815e8e0 con 0x7fcfd8003be0 2024-04-02T07:35:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.759+0000 7fcff6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd8007e40 0x7fcff815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.760+0000 7fcff6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd8007e40 0x7fcff815ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54534/0 (socket says 172.21.15.89:54534) 2024-04-02T07:35:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.760+0000 7fcff6a00700 1 -- 172.21.15.89:0/1708256169 learned_addr learned my addr 172.21.15.89:0/1708256169 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.760+0000 7fcff7400700 1 --2- 172.21.15.89:0/1708256169 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd8003be0 0x7fcff814f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.760+0000 7fcff7e00700 1 --2- 172.21.15.89:0/1708256169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfd8009320 0x7fcff80b8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.760+0000 7fcff6a00700 1 -- 172.21.15.89:0/1708256169 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd8003be0 msgr2=0x7fcff814f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.760+0000 7fcff6a00700 1 --2- 172.21.15.89:0/1708256169 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd8003be0 0x7fcff814f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.760+0000 7fcff6a00700 1 -- 172.21.15.89:0/1708256169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfd8009320 msgr2=0x7fcff80b8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.760+0000 7fcff6a00700 1 --2- 172.21.15.89:0/1708256169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfd8009320 0x7fcff80b8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:28.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.760+0000 7fcff6a00700 1 -- 172.21.15.89:0/1708256169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe800f040 con 0x7fcfd8007e40 2024-04-02T07:35:28.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.760+0000 7fcff6a00700 1 --2- 172.21.15.89:0/1708256169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd8007e40 0x7fcff815ddb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40125c0 tx=0x7fcfe40128d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:28.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.760+0000 7fcfef400700 1 -- 172.21.15.89:0/1708256169 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4012e30 con 0x7fcfd8007e40 2024-04-02T07:35:28.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.761+0000 7fcfef400700 1 -- 172.21.15.89:0/1708256169 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe40072e0 con 0x7fcfd8007e40 2024-04-02T07:35:28.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.761+0000 7fcffe600700 1 -- 172.21.15.89:0/1708256169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff80cc0d0 con 0x7fcfd8007e40 2024-04-02T07:35:28.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.761+0000 7fcfef400700 1 -- 172.21.15.89:0/1708256169 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4007690 con 0x7fcfd8007e40 2024-04-02T07:35:28.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.761+0000 7fcffe600700 1 -- 172.21.15.89:0/1708256169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff815e2f0 con 0x7fcfd8007e40 2024-04-02T07:35:28.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.761+0000 7fcfef400700 1 -- 172.21.15.89:0/1708256169 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcfe4005b30 con 0x7fcfd8007e40 2024-04-02T07:35:28.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.761+0000 7fcfef400700 1 -- 172.21.15.89:0/1708256169 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcfe4015440 con 0x7fcfd8007e40 2024-04-02T07:35:28.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.762+0000 7fcffe600700 1 -- 172.21.15.89:0/1708256169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff8067720 con 0x7fcfd8007e40 2024-04-02T07:35:28.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.765+0000 7fcfef400700 1 -- 172.21.15.89:0/1708256169 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fcfe403db50 con 0x7fcfd8007e40 2024-04-02T07:35:28.777 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:28.776+0000 7f016525e240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:35:28.794 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:28.791+0000 7f826b9b7240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:35:28.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.843+0000 7fcffe600700 1 -- 172.21.15.89:0/1708256169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fcff80d3620 con 0x7fcfd8007e40 2024-04-02T07:35:28.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.847+0000 7fcfef400700 1 -- 172.21.15.89:0/1708256169 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fcfe402c070 con 0x7fcfd8007e40 2024-04-02T07:35:28.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.848+0000 7fcffe600700 1 -- 172.21.15.89:0/1708256169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd8007e40 msgr2=0x7fcff815ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:28.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.848+0000 7fcffe600700 1 --2- 172.21.15.89:0/1708256169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd8007e40 0x7fcff815ddb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40125c0 tx=0x7fcfe40128d0 comp rx=0 tx=0).stop 2024-04-02T07:35:28.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.848+0000 7fcffe600700 1 -- 172.21.15.89:0/1708256169 shutdown_connections 2024-04-02T07:35:28.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.848+0000 7fcffe600700 1 --2- 172.21.15.89:0/1708256169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfd8009320 0x7fcff80b8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:28.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.848+0000 7fcffe600700 1 --2- 172.21.15.89:0/1708256169 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfd8003be0 0x7fcff814f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:28.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.848+0000 7fcffe600700 1 --2- 172.21.15.89:0/1708256169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd8007e40 0x7fcff815ddb0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:28.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.848+0000 7fcffe600700 1 -- 172.21.15.89:0/1708256169 >> 172.21.15.89:0/1708256169 conn(0x7fcff808aff0 msgr2=0x7fcff80b1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:28.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.848+0000 7fcffe600700 1 -- 172.21.15.89:0/1708256169 shutdown_connections 2024-04-02T07:35:28.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:28.848+0000 7fcffe600700 1 -- 172.21.15.89:0/1708256169 wait complete. 2024-04-02T07:35:28.852 INFO:teuthology.orchestra.run.smithi089.stderr:adjusted tunables profile to default 2024-04-02T07:35:28.868 INFO:tasks.ceph:check_enable_crimson: False 2024-04-02T07:35:28.868 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-02T07:35:28.868 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:35:28.869 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-02T07:35:28.935 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:35:28.935 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-02T07:35:29.040 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:35:29.040 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-02T07:35:29.149 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:35:29.149 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-02T07:35:29.256 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:35:29.256 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-02T07:35:29.321 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:35:29.321 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-02T07:35:29.390 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:29.389+0000 7f016525e240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:35:29.410 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:29.408+0000 7f826b9b7240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:35:29.427 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:35:29.427 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-02T07:35:29.533 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:35:29.533 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-02T07:35:29.640 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new 9c3756e3-ac90-4364-90bf-287137eff24e 0 2024-04-02T07:35:29.656 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:29.654+0000 7f016525e240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:35:29.660 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:29.657+0000 7f826b9b7240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:35:29.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.852+0000 7f3bb4600700 1 Processor -- start 2024-04-02T07:35:29.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.852+0000 7f3bb4600700 1 -- start start 2024-04-02T07:35:29.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.853+0000 7f3bb4600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3bac0b4420 0x7f3bac0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:29.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.853+0000 7f3bb4600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3bac0af600 0x7f3bac06dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:29.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.853+0000 7f3bb4600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bac0ade10 con 0x7f3bac15a680 2024-04-02T07:35:29.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.853+0000 7f3bb4600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bac0ae040 con 0x7f3bac0af600 2024-04-02T07:35:29.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.853+0000 7f3bb4600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bac0ae320 con 0x7f3bac0b4420 2024-04-02T07:35:29.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.853+0000 7f3bb1200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3bac0b4420 0x7f3bac0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:29.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.853+0000 7f3bb1200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3bac0b4420 0x7f3bac0ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36684/0 (socket says 172.21.15.113:36684) 2024-04-02T07:35:29.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.853+0000 7f3bb1200700 1 -- 172.21.15.113:0/2141508718 learned_addr learned my addr 172.21.15.113:0/2141508718 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-02T07:35:29.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.853+0000 7f3bb2600700 1 --2- 172.21.15.113:0/2141508718 >> v2:172.21.15.89:3300/0 conn(0x7f3bac0af600 0x7f3bac06dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:29.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.854+0000 7f3bb1200700 1 -- 172.21.15.113:0/2141508718 >> v2:172.21.15.89:3300/0 conn(0x7f3bac0af600 msgr2=0x7f3bac06dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:29.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.854+0000 7f3bb1200700 1 --2- 172.21.15.113:0/2141508718 >> v2:172.21.15.89:3300/0 conn(0x7f3bac0af600 0x7f3bac06dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:29.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.854+0000 7f3bb1200700 1 -- 172.21.15.113:0/2141508718 >> v1:172.21.15.113:6789/0 conn(0x7f3bac15a680 legacy=0x7f3bac0b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:29.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.854+0000 7f3bb1200700 1 -- 172.21.15.113:0/2141508718 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac0ae600 con 0x7f3bac0b4420 2024-04-02T07:35:29.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.854+0000 7f3bb2600700 1 --2- 172.21.15.113:0/2141508718 >> v2:172.21.15.89:3300/0 conn(0x7f3bac0af600 0x7f3bac06dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:35:29.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.854+0000 7f3bb1200700 1 --2- 172.21.15.113:0/2141508718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bac0b4420 0x7f3bac0ad8d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3bac071350 tx=0x7f3b9800be70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b518e6a0175c317 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:29.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.854+0000 7f3babe00700 1 -- 172.21.15.113:0/2141508718 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9800dd30 con 0x7f3bac0b4420 2024-04-02T07:35:29.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.855+0000 7f3babe00700 1 -- 172.21.15.113:0/2141508718 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9800de90 con 0x7f3bac0b4420 2024-04-02T07:35:29.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.855+0000 7f3babe00700 1 -- 172.21.15.113:0/2141508718 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b98010690 con 0x7f3bac0b4420 2024-04-02T07:35:29.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.855+0000 7f3bb4600700 1 -- 172.21.15.113:0/2141508718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bac0b4420 msgr2=0x7f3bac0ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:29.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.855+0000 7f3bb4600700 1 --2- 172.21.15.113:0/2141508718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bac0b4420 0x7f3bac0ad8d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3bac071350 tx=0x7f3b9800be70 comp rx=0 tx=0).stop 2024-04-02T07:35:29.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.855+0000 7f3bb4600700 1 -- 172.21.15.113:0/2141508718 shutdown_connections 2024-04-02T07:35:29.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.855+0000 7f3bb4600700 1 --2- 172.21.15.113:0/2141508718 >> v2:172.21.15.89:3300/0 conn(0x7f3bac0af600 0x7f3bac06dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:29.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.855+0000 7f3bb4600700 1 --2- 172.21.15.113:0/2141508718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bac0b4420 0x7f3bac0ad8d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:29.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.855+0000 7f3bb4600700 1 -- 172.21.15.113:0/2141508718 >> 172.21.15.113:0/2141508718 conn(0x7f3bac0b5720 msgr2=0x7f3bac0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:29.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.855+0000 7f3bb4600700 1 -- 172.21.15.113:0/2141508718 shutdown_connections 2024-04-02T07:35:29.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.855+0000 7f3bb4600700 1 -- 172.21.15.113:0/2141508718 wait complete. 2024-04-02T07:35:29.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.857+0000 7f3bb4600700 1 Processor -- start 2024-04-02T07:35:29.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.857+0000 7f3bb4600700 1 -- start start 2024-04-02T07:35:29.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.858+0000 7f3bb4600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bac0af600 0x7f3bac09dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:29.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.858+0000 7f3bb4600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bac0b4420 0x7f3bac152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:29.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.858+0000 7f3bb4600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bac15a680 0x7f3bac0cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:29.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.858+0000 7f3bb4600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac09e320 con 0x7f3bac0b4420 2024-04-02T07:35:29.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.858+0000 7f3bb4600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac0cf830 con 0x7f3bac0af600 2024-04-02T07:35:29.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.858+0000 7f3bb4600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac09b350 con 0x7f3bac15a680 2024-04-02T07:35:29.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.858+0000 7f3bb1c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bac0af600 0x7f3bac09dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:29.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.858+0000 7f3bb1c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bac0af600 0x7f3bac09dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36690/0 (socket says 172.21.15.113:36690) 2024-04-02T07:35:29.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.858+0000 7f3bb2600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bac15a680 0x7f3bac0cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:29.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.858+0000 7f3bb1c00700 1 -- 172.21.15.113:0/578506423 learned_addr learned my addr 172.21.15.113:0/578506423 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-02T07:35:29.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.858+0000 7f3bb1200700 1 --2- 172.21.15.113:0/578506423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bac0b4420 0x7f3bac152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:29.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.859+0000 7f3bb1c00700 1 -- 172.21.15.113:0/578506423 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bac15a680 msgr2=0x7f3bac0cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:29.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.859+0000 7f3bb1c00700 1 --2- 172.21.15.113:0/578506423 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bac15a680 0x7f3bac0cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:29.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.859+0000 7f3bb1c00700 1 -- 172.21.15.113:0/578506423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bac0b4420 msgr2=0x7f3bac152190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:29.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.859+0000 7f3bb1c00700 1 --2- 172.21.15.113:0/578506423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bac0b4420 0x7f3bac152190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:29.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.859+0000 7f3bb1c00700 1 -- 172.21.15.113:0/578506423 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9800b7f0 con 0x7f3bac0af600 2024-04-02T07:35:29.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.859+0000 7f3bb1c00700 1 --2- 172.21.15.113:0/578506423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bac0af600 0x7f3bac09dde0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3ba000c450 tx=0x7f3ba000c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:29.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.859+0000 7f3ba9600700 1 -- 172.21.15.113:0/578506423 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba00052d0 con 0x7f3bac0af600 2024-04-02T07:35:29.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.859+0000 7f3ba9600700 1 -- 172.21.15.113:0/578506423 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba0005430 con 0x7f3bac0af600 2024-04-02T07:35:29.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.859+0000 7f3ba9600700 1 -- 172.21.15.113:0/578506423 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba00038c0 con 0x7f3bac0af600 2024-04-02T07:35:29.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.859+0000 7f3bb4600700 1 -- 172.21.15.113:0/578506423 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bac09b070 con 0x7f3bac0af600 2024-04-02T07:35:29.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.859+0000 7f3bb4600700 1 -- 172.21.15.113:0/578506423 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bac0cf4c0 con 0x7f3bac0af600 2024-04-02T07:35:29.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.860+0000 7f3ba9600700 1 -- 172.21.15.113:0/578506423 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3ba0003d20 con 0x7f3bac0af600 2024-04-02T07:35:29.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.860+0000 7f3ba9600700 1 -- 172.21.15.113:0/578506423 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3ba001c780 con 0x7f3bac0af600 2024-04-02T07:35:29.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.860+0000 7f3bb4600700 1 -- 172.21.15.113:0/578506423 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bac0cf040 con 0x7f3bac0af600 2024-04-02T07:35:29.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.863+0000 7f3ba9600700 1 -- 172.21.15.113:0/578506423 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f3bac0cf040 con 0x7f3bac0af600 2024-04-02T07:35:29.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.950+0000 7f3bb4600700 1 -- 172.21.15.113:0/578506423 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9c3756e3-ac90-4364-90bf-287137eff24e", "id": 0} v 0) v1 -- 0x7f3bac0cfb10 con 0x7f3bac0af600 2024-04-02T07:35:29.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.956+0000 7f3ba9600700 1 -- 172.21.15.113:0/578506423 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9c3756e3-ac90-4364-90bf-287137eff24e", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3ba0017070 con 0x7f3bac0af600 2024-04-02T07:35:29.958 INFO:teuthology.orchestra.run.smithi113.stdout:0 2024-04-02T07:35:29.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.957+0000 7f3bb4600700 1 -- 172.21.15.113:0/578506423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bac0af600 msgr2=0x7f3bac09dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:29.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.957+0000 7f3bb4600700 1 --2- 172.21.15.113:0/578506423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bac0af600 0x7f3bac09dde0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3ba000c450 tx=0x7f3ba000c760 comp rx=0 tx=0).stop 2024-04-02T07:35:29.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.957+0000 7f3bb4600700 1 -- 172.21.15.113:0/578506423 shutdown_connections 2024-04-02T07:35:29.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.957+0000 7f3bb4600700 1 --2- 172.21.15.113:0/578506423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bac0af600 0x7f3bac09dde0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:29.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.957+0000 7f3bb4600700 1 --2- 172.21.15.113:0/578506423 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bac15a680 0x7f3bac0cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:29.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.957+0000 7f3bb4600700 1 --2- 172.21.15.113:0/578506423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bac0b4420 0x7f3bac152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:29.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.957+0000 7f3bb4600700 1 -- 172.21.15.113:0/578506423 >> 172.21.15.113:0/578506423 conn(0x7f3bac0b5720 msgr2=0x7f3bac0b3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:29.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.958+0000 7f3bb4600700 1 -- 172.21.15.113:0/578506423 shutdown_connections 2024-04-02T07:35:29.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:29.958+0000 7f3bb4600700 1 -- 172.21.15.113:0/578506423 wait complete. 2024-04-02T07:35:29.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc 1 2024-04-02T07:35:30.129 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:30.126+0000 7f826b9b7240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:35:30.152 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:30.151+0000 7f016525e240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:35:30.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.167+0000 7fcd2de00700 1 Processor -- start 2024-04-02T07:35:30.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.167+0000 7fcd2de00700 1 -- start start 2024-04-02T07:35:30.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.168+0000 7fcd2de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcd281611b0 0x7fcd2815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.168+0000 7fcd2de00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcd280cee10 0x7fcd28161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.168+0000 7fcd2de00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd280ce270 con 0x7fcd2815f4d0 2024-04-02T07:35:30.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.168+0000 7fcd2de00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd2815b600 con 0x7fcd280cee10 2024-04-02T07:35:30.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.168+0000 7fcd2de00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd280d2760 con 0x7fcd281611b0 2024-04-02T07:35:30.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.168+0000 7fcd27400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcd280cee10 0x7fcd28161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.168+0000 7fcd26a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcd281611b0 0x7fcd2815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.168+0000 7fcd27400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcd280cee10 0x7fcd28161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36706/0 (socket says 172.21.15.113:36706) 2024-04-02T07:35:30.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.168+0000 7fcd27400700 1 -- 172.21.15.113:0/350579582 learned_addr learned my addr 172.21.15.113:0/350579582 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-02T07:35:30.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.169+0000 7fcd26a00700 1 -- 172.21.15.113:0/350579582 >> v2:172.21.15.113:3300/0 conn(0x7fcd280cee10 msgr2=0x7fcd28161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:30.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.169+0000 7fcd26a00700 1 --2- 172.21.15.113:0/350579582 >> v2:172.21.15.113:3300/0 conn(0x7fcd280cee10 0x7fcd28161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.169+0000 7fcd26a00700 1 -- 172.21.15.113:0/350579582 >> v1:172.21.15.89:6789/0 conn(0x7fcd2815f4d0 legacy=0x7fcd280ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:30.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.169+0000 7fcd26a00700 1 -- 172.21.15.113:0/350579582 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd280d2a40 con 0x7fcd281611b0 2024-04-02T07:35:30.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.169+0000 7fcd27400700 1 --2- 172.21.15.113:0/350579582 >> v2:172.21.15.113:3300/0 conn(0x7fcd280cee10 0x7fcd28161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:30.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.169+0000 7fcd26a00700 1 --2- 172.21.15.113:0/350579582 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcd281611b0 0x7fcd2815ef90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009840 tx=0x7fcd18009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7be96a12c2c229d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:30.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.169+0000 7fcd26000700 1 -- 172.21.15.113:0/350579582 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18011830 con 0x7fcd281611b0 2024-04-02T07:35:30.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.169+0000 7fcd26000700 1 -- 172.21.15.113:0/350579582 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd18011990 con 0x7fcd281611b0 2024-04-02T07:35:30.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.169+0000 7fcd26000700 1 -- 172.21.15.113:0/350579582 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18011d20 con 0x7fcd281611b0 2024-04-02T07:35:30.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.170+0000 7fcd2de00700 1 -- 172.21.15.113:0/350579582 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcd281611b0 msgr2=0x7fcd2815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:30.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.170+0000 7fcd2de00700 1 --2- 172.21.15.113:0/350579582 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcd281611b0 0x7fcd2815ef90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009840 tx=0x7fcd18009b50 comp rx=0 tx=0).stop 2024-04-02T07:35:30.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.170+0000 7fcd2de00700 1 -- 172.21.15.113:0/350579582 shutdown_connections 2024-04-02T07:35:30.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.170+0000 7fcd2de00700 1 --2- 172.21.15.113:0/350579582 >> v2:172.21.15.113:3300/0 conn(0x7fcd280cee10 0x7fcd28161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.170+0000 7fcd2de00700 1 --2- 172.21.15.113:0/350579582 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcd281611b0 0x7fcd2815ef90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.170+0000 7fcd2de00700 1 -- 172.21.15.113:0/350579582 >> 172.21.15.113:0/350579582 conn(0x7fcd2808aff0 msgr2=0x7fcd280d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:30.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.170+0000 7fcd2de00700 1 -- 172.21.15.113:0/350579582 shutdown_connections 2024-04-02T07:35:30.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.170+0000 7fcd2de00700 1 -- 172.21.15.113:0/350579582 wait complete. 2024-04-02T07:35:30.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.172+0000 7fcd2de00700 1 Processor -- start 2024-04-02T07:35:30.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.173+0000 7fcd2de00700 1 -- start start 2024-04-02T07:35:30.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd2de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd280cee10 0x7fcd281e97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd2de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcd2815f4d0 0x7fcd280ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd2de00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd281611b0 0x7fcd28157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd2de00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd280cab40 con 0x7fcd280cee10 2024-04-02T07:35:30.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd2de00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd28158180 con 0x7fcd281611b0 2024-04-02T07:35:30.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd2de00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd28157f50 con 0x7fcd2815f4d0 2024-04-02T07:35:30.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd27400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd281611b0 0x7fcd28157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd1fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcd2815f4d0 0x7fcd280ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd27400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd281611b0 0x7fcd28157a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36708/0 (socket says 172.21.15.113:36708) 2024-04-02T07:35:30.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd27400700 1 -- 172.21.15.113:0/2602024713 learned_addr learned my addr 172.21.15.113:0/2602024713 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-02T07:35:30.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd26a00700 1 --2- 172.21.15.113:0/2602024713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd280cee10 0x7fcd281e97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd27400700 1 -- 172.21.15.113:0/2602024713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcd2815f4d0 msgr2=0x7fcd280ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:30.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd27400700 1 --2- 172.21.15.113:0/2602024713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcd2815f4d0 0x7fcd280ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd27400700 1 -- 172.21.15.113:0/2602024713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd280cee10 msgr2=0x7fcd281e97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:30.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd27400700 1 --2- 172.21.15.113:0/2602024713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd280cee10 0x7fcd281e97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.174+0000 7fcd27400700 1 -- 172.21.15.113:0/2602024713 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd20009b30 con 0x7fcd281611b0 2024-04-02T07:35:30.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.175+0000 7fcd26a00700 1 --2- 172.21.15.113:0/2602024713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd280cee10 0x7fcd281e97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:30.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.175+0000 7fcd1fe00700 1 --2- 172.21.15.113:0/2602024713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcd2815f4d0 0x7fcd280ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:30.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.175+0000 7fcd27400700 1 --2- 172.21.15.113:0/2602024713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd281611b0 0x7fcd28157a10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcd20009f10 tx=0x7fcd2000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:30.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.175+0000 7fcd1f400700 1 -- 172.21.15.113:0/2602024713 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd20018070 con 0x7fcd281611b0 2024-04-02T07:35:30.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.175+0000 7fcd1f400700 1 -- 172.21.15.113:0/2602024713 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2000f040 con 0x7fcd281611b0 2024-04-02T07:35:30.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.175+0000 7fcd1f400700 1 -- 172.21.15.113:0/2602024713 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd20003c40 con 0x7fcd281611b0 2024-04-02T07:35:30.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.175+0000 7fcd2de00700 1 -- 172.21.15.113:0/2602024713 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd180094a0 con 0x7fcd281611b0 2024-04-02T07:35:30.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.175+0000 7fcd2de00700 1 -- 172.21.15.113:0/2602024713 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd280cb050 con 0x7fcd281611b0 2024-04-02T07:35:30.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.175+0000 7fcd1f400700 1 -- 172.21.15.113:0/2602024713 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcd2000cc40 con 0x7fcd281611b0 2024-04-02T07:35:30.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.176+0000 7fcd1f400700 1 -- 172.21.15.113:0/2602024713 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fcd2001d7c0 con 0x7fcd281611b0 2024-04-02T07:35:30.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.176+0000 7fcd2de00700 1 -- 172.21.15.113:0/2602024713 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd28067720 con 0x7fcd281611b0 2024-04-02T07:35:30.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.179+0000 7fcd1f400700 1 -- 172.21.15.113:0/2602024713 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fcd20014ca0 con 0x7fcd281611b0 2024-04-02T07:35:30.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.265+0000 7fcd2de00700 1 -- 172.21.15.113:0/2602024713 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc", "id": 1} v 0) v1 -- 0x7fcd281e9e00 con 0x7fcd281611b0 2024-04-02T07:35:30.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.273+0000 7fcd1f400700 1 -- 172.21.15.113:0/2602024713 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcd2002c0b0 con 0x7fcd281611b0 2024-04-02T07:35:30.274 INFO:teuthology.orchestra.run.smithi113.stdout:1 2024-04-02T07:35:30.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.274+0000 7fcd2de00700 1 -- 172.21.15.113:0/2602024713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd281611b0 msgr2=0x7fcd28157a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:30.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.274+0000 7fcd2de00700 1 --2- 172.21.15.113:0/2602024713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd281611b0 0x7fcd28157a10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcd20009f10 tx=0x7fcd2000c3a0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.274+0000 7fcd2de00700 1 -- 172.21.15.113:0/2602024713 shutdown_connections 2024-04-02T07:35:30.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.274+0000 7fcd2de00700 1 --2- 172.21.15.113:0/2602024713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd281611b0 0x7fcd28157a10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.274+0000 7fcd2de00700 1 --2- 172.21.15.113:0/2602024713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd280cee10 0x7fcd281e97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.274+0000 7fcd2de00700 1 --2- 172.21.15.113:0/2602024713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcd2815f4d0 0x7fcd280ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.274+0000 7fcd2de00700 1 -- 172.21.15.113:0/2602024713 >> 172.21.15.113:0/2602024713 conn(0x7fcd2808aff0 msgr2=0x7fcd28161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:30.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.274+0000 7fcd2de00700 1 -- 172.21.15.113:0/2602024713 shutdown_connections 2024-04-02T07:35:30.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.274+0000 7fcd2de00700 1 -- 172.21.15.113:0/2602024713 wait complete. 2024-04-02T07:35:30.292 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new 5dcd8a58-660d-4cfd-9d05-6109d2d0c378 2 2024-04-02T07:35:30.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.482+0000 7fe1e3000700 1 Processor -- start 2024-04-02T07:35:30.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.483+0000 7fe1e3000700 1 -- start start 2024-04-02T07:35:30.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.484+0000 7fe1e3000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc15d9d0 0x7fe1dc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.484+0000 7fe1e3000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe1dc0d16a0 0x7fe1dc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.484+0000 7fe1e3000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc161420 con 0x7fe1dc0d2370 2024-04-02T07:35:30.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.484+0000 7fe1e3000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1dc0d2820 con 0x7fe1dc0d16a0 2024-04-02T07:35:30.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.484+0000 7fe1e3000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc161700 con 0x7fe1dc15d9d0 2024-04-02T07:35:30.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.484+0000 7fe1db400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe1dc0d16a0 0x7fe1dc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.484+0000 7fe1dbe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc15d9d0 0x7fe1dc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.484+0000 7fe1e1000700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fe1dc0d2370 0x7fe1dc15ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:60610/0 (socket says 172.21.15.113:60610) 2024-04-02T07:35:30.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.484+0000 7fe1e1000700 1 -- 172.21.15.113:0/358363760 learned_addr learned my addr 172.21.15.113:0/358363760 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-04-02T07:35:30.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.484+0000 7fe1dbe00700 1 -- 172.21.15.113:0/358363760 >> v2:172.21.15.113:3300/0 conn(0x7fe1dc0d16a0 msgr2=0x7fe1dc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:30.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.484+0000 7fe1dbe00700 1 --2- 172.21.15.113:0/358363760 >> v2:172.21.15.113:3300/0 conn(0x7fe1dc0d16a0 0x7fe1dc0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.484+0000 7fe1dbe00700 1 -- 172.21.15.113:0/358363760 >> v1:172.21.15.113:6789/0 conn(0x7fe1dc0d2370 legacy=0x7fe1dc15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:30.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.484+0000 7fe1dbe00700 1 -- 172.21.15.113:0/358363760 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc1619e0 con 0x7fe1dc15d9d0 2024-04-02T07:35:30.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.484+0000 7fe1db400700 1 --2- 172.21.15.113:0/358363760 >> v2:172.21.15.113:3300/0 conn(0x7fe1dc0d16a0 0x7fe1dc0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:30.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.485+0000 7fe1dbe00700 1 --2- 172.21.15.113:0/358363760 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc15d9d0 0x7fe1dc0d3c30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009840 tx=0x7fe1cc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f7ae82cfe57ce4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:30.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.485+0000 7fe1daa00700 1 -- 172.21.15.113:0/358363760 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc011830 con 0x7fe1dc15d9d0 2024-04-02T07:35:30.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.485+0000 7fe1daa00700 1 -- 172.21.15.113:0/358363760 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1cc011990 con 0x7fe1dc15d9d0 2024-04-02T07:35:30.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.485+0000 7fe1daa00700 1 -- 172.21.15.113:0/358363760 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc011d20 con 0x7fe1dc15d9d0 2024-04-02T07:35:30.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.486+0000 7fe1e3000700 1 -- 172.21.15.113:0/358363760 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc15d9d0 msgr2=0x7fe1dc0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:30.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.486+0000 7fe1e3000700 1 --2- 172.21.15.113:0/358363760 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc15d9d0 0x7fe1dc0d3c30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009840 tx=0x7fe1cc009b50 comp rx=0 tx=0).stop 2024-04-02T07:35:30.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.486+0000 7fe1e3000700 1 -- 172.21.15.113:0/358363760 shutdown_connections 2024-04-02T07:35:30.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.486+0000 7fe1e3000700 1 --2- 172.21.15.113:0/358363760 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc15d9d0 0x7fe1dc0d3c30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.486+0000 7fe1e3000700 1 --2- 172.21.15.113:0/358363760 >> v2:172.21.15.113:3300/0 conn(0x7fe1dc0d16a0 0x7fe1dc0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.486+0000 7fe1e3000700 1 -- 172.21.15.113:0/358363760 >> 172.21.15.113:0/358363760 conn(0x7fe1dc08aff0 msgr2=0x7fe1dc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:30.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.486+0000 7fe1e3000700 1 -- 172.21.15.113:0/358363760 shutdown_connections 2024-04-02T07:35:30.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.486+0000 7fe1e3000700 1 -- 172.21.15.113:0/358363760 wait complete. 2024-04-02T07:35:30.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.487+0000 7fe1e3000700 1 Processor -- start 2024-04-02T07:35:30.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.488+0000 7fe1e3000700 1 -- start start 2024-04-02T07:35:30.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.488+0000 7fe1e3000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1dc0d16a0 0x7fe1dc1556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1e3000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc0d2370 0x7fe1dc0b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1e3000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc15d9d0 0x7fe1dc0b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1e3000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc155bf0 con 0x7fe1dc0d2370 2024-04-02T07:35:30.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1e3000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0b8050 con 0x7fe1dc0d16a0 2024-04-02T07:35:30.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1e3000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0b73d0 con 0x7fe1dc15d9d0 2024-04-02T07:35:30.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1dbe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1dc0d16a0 0x7fe1dc1556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1dbe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1dc0d16a0 0x7fe1dc1556b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36716/0 (socket says 172.21.15.113:36716) 2024-04-02T07:35:30.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1dbe00700 1 -- 172.21.15.113:0/726500508 learned_addr learned my addr 172.21.15.113:0/726500508 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-02T07:35:30.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1e1000700 1 --2- 172.21.15.113:0/726500508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc15d9d0 0x7fe1dc0b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1db400700 1 --2- 172.21.15.113:0/726500508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc0d2370 0x7fe1dc0b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1dbe00700 1 -- 172.21.15.113:0/726500508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc15d9d0 msgr2=0x7fe1dc0b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:30.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1dbe00700 1 --2- 172.21.15.113:0/726500508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc15d9d0 0x7fe1dc0b4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1dbe00700 1 -- 172.21.15.113:0/726500508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc0d2370 msgr2=0x7fe1dc0b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:30.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1dbe00700 1 --2- 172.21.15.113:0/726500508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc0d2370 0x7fe1dc0b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1dbe00700 1 -- 172.21.15.113:0/726500508 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c8009430 con 0x7fe1dc0d16a0 2024-04-02T07:35:30.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1db400700 1 --2- 172.21.15.113:0/726500508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc0d2370 0x7fe1dc0b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:35:30.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.489+0000 7fe1dbe00700 1 --2- 172.21.15.113:0/726500508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1dc0d16a0 0x7fe1dc1556b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009fd0 tx=0x7fe1cc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:30.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.490+0000 7fe1d3e00700 1 -- 172.21.15.113:0/726500508 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc0036a0 con 0x7fe1dc0d16a0 2024-04-02T07:35:30.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.490+0000 7fe1d3e00700 1 -- 172.21.15.113:0/726500508 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1cc01e070 con 0x7fe1dc0d16a0 2024-04-02T07:35:30.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.490+0000 7fe1d3e00700 1 -- 172.21.15.113:0/726500508 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc003a40 con 0x7fe1dc0d16a0 2024-04-02T07:35:30.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.490+0000 7fe1e3000700 1 -- 172.21.15.113:0/726500508 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1cc0094a0 con 0x7fe1dc0d16a0 2024-04-02T07:35:30.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.490+0000 7fe1e3000700 1 -- 172.21.15.113:0/726500508 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc0b7990 con 0x7fe1dc0d16a0 2024-04-02T07:35:30.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.490+0000 7fe1d3e00700 1 -- 172.21.15.113:0/726500508 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe1cc02d450 con 0x7fe1dc0d16a0 2024-04-02T07:35:30.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.490+0000 7fe1d3e00700 1 -- 172.21.15.113:0/726500508 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fe1cc01a840 con 0x7fe1dc0d16a0 2024-04-02T07:35:30.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.491+0000 7fe1e3000700 1 -- 172.21.15.113:0/726500508 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc0d1ca0 con 0x7fe1dc0d16a0 2024-04-02T07:35:30.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.493+0000 7fe1d3e00700 1 -- 172.21.15.113:0/726500508 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fe1cc020070 con 0x7fe1dc0d16a0 2024-04-02T07:35:30.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.576+0000 7fe1e3000700 1 -- 172.21.15.113:0/726500508 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5dcd8a58-660d-4cfd-9d05-6109d2d0c378", "id": 2} v 0) v1 -- 0x7fe1dc067720 con 0x7fe1dc0d16a0 2024-04-02T07:35:30.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.584+0000 7fe1d3e00700 1 -- 172.21.15.113:0/726500508 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5dcd8a58-660d-4cfd-9d05-6109d2d0c378", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe1cc015020 con 0x7fe1dc0d16a0 2024-04-02T07:35:30.585 INFO:teuthology.orchestra.run.smithi113.stdout:2 2024-04-02T07:35:30.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.585+0000 7fe1e3000700 1 -- 172.21.15.113:0/726500508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1dc0d16a0 msgr2=0x7fe1dc1556b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:30.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.585+0000 7fe1e3000700 1 --2- 172.21.15.113:0/726500508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1dc0d16a0 0x7fe1dc1556b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009fd0 tx=0x7fe1cc004750 comp rx=0 tx=0).stop 2024-04-02T07:35:30.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.585+0000 7fe1e3000700 1 -- 172.21.15.113:0/726500508 shutdown_connections 2024-04-02T07:35:30.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.585+0000 7fe1e3000700 1 --2- 172.21.15.113:0/726500508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1dc0d16a0 0x7fe1dc1556b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.585+0000 7fe1e3000700 1 --2- 172.21.15.113:0/726500508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc15d9d0 0x7fe1dc0b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.585+0000 7fe1e3000700 1 --2- 172.21.15.113:0/726500508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc0d2370 0x7fe1dc0b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.585+0000 7fe1e3000700 1 -- 172.21.15.113:0/726500508 >> 172.21.15.113:0/726500508 conn(0x7fe1dc08aff0 msgr2=0x7fe1dc15bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:30.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.585+0000 7fe1e3000700 1 -- 172.21.15.113:0/726500508 shutdown_connections 2024-04-02T07:35:30.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.585+0000 7fe1e3000700 1 -- 172.21.15.113:0/726500508 wait complete. 2024-04-02T07:35:30.602 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new e8e4e352-db2d-4b96-9282-b2a4883c8f66 3 2024-04-02T07:35:30.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.788+0000 7f539ca00700 1 Processor -- start 2024-04-02T07:35:30.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.788+0000 7f539ca00700 1 -- start start 2024-04-02T07:35:30.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.789+0000 7f539ca00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f539815d9d0 0x7f53980d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.789+0000 7f539ca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53980d16a0 0x7f53980d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.789+0000 7f539ca00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f53980d2370 0x7f539815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.789+0000 7f539ca00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53980d2820 con 0x7f53980d2370 2024-04-02T07:35:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.789+0000 7f539ca00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5398161420 con 0x7f539815d9d0 2024-04-02T07:35:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.789+0000 7f539ca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5398161700 con 0x7f53980d16a0 2024-04-02T07:35:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.789+0000 7f5395400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53980d16a0 0x7f53980d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.789+0000 7f5395e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f539815d9d0 0x7f53980d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.789+0000 7f5395e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f539815d9d0 0x7f53980d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36722/0 (socket says 172.21.15.113:36722) 2024-04-02T07:35:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.789+0000 7f5395e00700 1 -- 172.21.15.113:0/611404555 learned_addr learned my addr 172.21.15.113:0/611404555 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-02T07:35:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.789+0000 7f5395400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53980d16a0 0x7f53980d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.113:34806/0 (socket says 172.21.15.113:34806) 2024-04-02T07:35:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.789+0000 7f5396800700 1 --2- 172.21.15.113:0/611404555 >> v2:172.21.15.89:3300/0 conn(0x7f53980d2370 0x7f539815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.790+0000 7f5395e00700 1 -- 172.21.15.113:0/611404555 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53980d16a0 msgr2=0x7f53980d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.790+0000 7f5395e00700 1 --2- 172.21.15.113:0/611404555 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53980d16a0 0x7f53980d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.790+0000 7f5395e00700 1 -- 172.21.15.113:0/611404555 >> v2:172.21.15.89:3300/0 conn(0x7f53980d2370 msgr2=0x7f539815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:30.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.790+0000 7f5395e00700 1 --2- 172.21.15.113:0/611404555 >> v2:172.21.15.89:3300/0 conn(0x7f53980d2370 0x7f539815ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.790+0000 7f5395e00700 1 -- 172.21.15.113:0/611404555 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53981619e0 con 0x7f539815d9d0 2024-04-02T07:35:30.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.790+0000 7f5395e00700 1 --2- 172.21.15.113:0/611404555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f539815d9d0 0x7f53980d3c30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f53980d0ff0 tx=0x7f5388009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80996221a98fd1a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:30.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.790+0000 7f5394a00700 1 -- 172.21.15.113:0/611404555 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388011830 con 0x7f539815d9d0 2024-04-02T07:35:30.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.790+0000 7f5394a00700 1 -- 172.21.15.113:0/611404555 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5388011990 con 0x7f539815d9d0 2024-04-02T07:35:30.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.790+0000 7f5394a00700 1 -- 172.21.15.113:0/611404555 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388011d20 con 0x7f539815d9d0 2024-04-02T07:35:30.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.791+0000 7f539ca00700 1 -- 172.21.15.113:0/611404555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f539815d9d0 msgr2=0x7f53980d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:30.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.791+0000 7f539ca00700 1 --2- 172.21.15.113:0/611404555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f539815d9d0 0x7f53980d3c30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f53980d0ff0 tx=0x7f5388009bd0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.791+0000 7f539ca00700 1 -- 172.21.15.113:0/611404555 shutdown_connections 2024-04-02T07:35:30.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.791+0000 7f539ca00700 1 --2- 172.21.15.113:0/611404555 >> v2:172.21.15.89:3300/0 conn(0x7f53980d2370 0x7f539815ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.791+0000 7f539ca00700 1 --2- 172.21.15.113:0/611404555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f539815d9d0 0x7f53980d3c30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.791+0000 7f539ca00700 1 --2- 172.21.15.113:0/611404555 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53980d16a0 0x7f53980d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.791+0000 7f539ca00700 1 -- 172.21.15.113:0/611404555 >> 172.21.15.113:0/611404555 conn(0x7f539808aff0 msgr2=0x7f53980d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:30.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.791+0000 7f539ca00700 1 -- 172.21.15.113:0/611404555 shutdown_connections 2024-04-02T07:35:30.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.791+0000 7f539ca00700 1 -- 172.21.15.113:0/611404555 wait complete. 2024-04-02T07:35:30.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.793+0000 7f539ca00700 1 Processor -- start 2024-04-02T07:35:30.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.793+0000 7f539ca00700 1 -- start start 2024-04-02T07:35:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.794+0000 7f539ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53980d16a0 0x7f539814fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.794+0000 7f539ca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53980d2370 0x7f53980b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.794+0000 7f539ca00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f539815d9d0 0x7f53980b7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.794+0000 7f539ca00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53980b75d0 con 0x7f53980d16a0 2024-04-02T07:35:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.794+0000 7f539ca00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5398150170 con 0x7f539815d9d0 2024-04-02T07:35:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.794+0000 7f539ca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53981e7a70 con 0x7f53980d2370 2024-04-02T07:35:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.794+0000 7f5396800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f539815d9d0 0x7f53980b7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.794+0000 7f5395400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53980d2370 0x7f53980b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.794+0000 7f5396800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f539815d9d0 0x7f53980b7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36726/0 (socket says 172.21.15.113:36726) 2024-04-02T07:35:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.794+0000 7f5395e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53980d16a0 0x7f539814fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.794+0000 7f5396800700 1 -- 172.21.15.113:0/989189750 learned_addr learned my addr 172.21.15.113:0/989189750 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-02T07:35:30.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.795+0000 7f5395e00700 1 -- 172.21.15.113:0/989189750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53980d2370 msgr2=0x7f53980b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:30.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.795+0000 7f5395e00700 1 --2- 172.21.15.113:0/989189750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53980d2370 0x7f53980b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.795+0000 7f5395e00700 1 -- 172.21.15.113:0/989189750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f539815d9d0 msgr2=0x7f53980b7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:30.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.795+0000 7f5395e00700 1 --2- 172.21.15.113:0/989189750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f539815d9d0 0x7f53980b7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.795+0000 7f5395e00700 1 -- 172.21.15.113:0/989189750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5388009550 con 0x7f53980d16a0 2024-04-02T07:35:30.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.795+0000 7f5396800700 1 --2- 172.21.15.113:0/989189750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f539815d9d0 0x7f53980b7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:30.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.795+0000 7f5395e00700 1 --2- 172.21.15.113:0/989189750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53980d16a0 0x7f539814fc30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5388016000 tx=0x7f5388004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:30.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.795+0000 7f538e000700 1 -- 172.21.15.113:0/989189750 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388004450 con 0x7f53980d16a0 2024-04-02T07:35:30.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.795+0000 7f538e000700 1 -- 172.21.15.113:0/989189750 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538801e070 con 0x7f53980d16a0 2024-04-02T07:35:30.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.795+0000 7f539ca00700 1 -- 172.21.15.113:0/989189750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53980b72f0 con 0x7f53980d16a0 2024-04-02T07:35:30.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.795+0000 7f538e000700 1 -- 172.21.15.113:0/989189750 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53880038e0 con 0x7f53980d16a0 2024-04-02T07:35:30.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.795+0000 7f539ca00700 1 -- 172.21.15.113:0/989189750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53981e7700 con 0x7f53980d16a0 2024-04-02T07:35:30.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.796+0000 7f538e000700 1 -- 172.21.15.113:0/989189750 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5388003d40 con 0x7f53980d16a0 2024-04-02T07:35:30.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.796+0000 7f538e000700 1 -- 172.21.15.113:0/989189750 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f538802d450 con 0x7f53980d16a0 2024-04-02T07:35:30.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.796+0000 7f539ca00700 1 -- 172.21.15.113:0/989189750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53980d1ca0 con 0x7f53980d16a0 2024-04-02T07:35:30.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.801+0000 7f538e000700 1 -- 172.21.15.113:0/989189750 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f5388020070 con 0x7f53980d16a0 2024-04-02T07:35:30.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.885+0000 7f539ca00700 1 -- 172.21.15.113:0/989189750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e8e4e352-db2d-4b96-9282-b2a4883c8f66", "id": 3} v 0) v1 -- 0x7f53981e7280 con 0x7f53980d16a0 2024-04-02T07:35:30.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.891+0000 7f538e000700 1 -- 172.21.15.113:0/989189750 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e8e4e352-db2d-4b96-9282-b2a4883c8f66", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f53981e7280 con 0x7f53980d16a0 2024-04-02T07:35:30.892 INFO:teuthology.orchestra.run.smithi113.stdout:3 2024-04-02T07:35:30.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.892+0000 7f539ca00700 1 -- 172.21.15.113:0/989189750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53980d16a0 msgr2=0x7f539814fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:30.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.892+0000 7f539ca00700 1 --2- 172.21.15.113:0/989189750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53980d16a0 0x7f539814fc30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5388016000 tx=0x7f5388004750 comp rx=0 tx=0).stop 2024-04-02T07:35:30.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.892+0000 7f539ca00700 1 -- 172.21.15.113:0/989189750 shutdown_connections 2024-04-02T07:35:30.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.892+0000 7f539ca00700 1 --2- 172.21.15.113:0/989189750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f539815d9d0 0x7f53980b7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.892+0000 7f539ca00700 1 --2- 172.21.15.113:0/989189750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53980d16a0 0x7f539814fc30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.892+0000 7f539ca00700 1 --2- 172.21.15.113:0/989189750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53980d2370 0x7f53980b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:30.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.892+0000 7f539ca00700 1 -- 172.21.15.113:0/989189750 >> 172.21.15.113:0/989189750 conn(0x7f539808aff0 msgr2=0x7f53980d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:30.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.892+0000 7f539ca00700 1 -- 172.21.15.113:0/989189750 shutdown_connections 2024-04-02T07:35:30.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:30.892+0000 7f539ca00700 1 -- 172.21.15.113:0/989189750 wait complete. 2024-04-02T07:35:30.910 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new cb9001a3-5ee2-4c09-afcf-881626477a6e 4 2024-04-02T07:35:30.944 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:30.942+0000 7f826b9b7240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:35:31.036 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:31.035+0000 7f016525e240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:35:31.036 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:31.034+0000 7f826b9b7240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:35:31.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.094+0000 7ff62f400700 1 Processor -- start 2024-04-02T07:35:31.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.095+0000 7ff62f400700 1 -- start start 2024-04-02T07:35:31.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.096+0000 7ff62f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff6280d16a0 0x7ff6280d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:31.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.096+0000 7ff62f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6280d2370 0x7ff6280ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:31.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.096+0000 7ff62f400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff628161420 con 0x7ff62815d9d0 2024-04-02T07:35:31.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.096+0000 7ff62f400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6280d2820 con 0x7ff6280d16a0 2024-04-02T07:35:31.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.096+0000 7ff62f400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6280d4340 con 0x7ff6280d2370 2024-04-02T07:35:31.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.096+0000 7ff62d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6280d2370 0x7ff6280ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:31.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.096+0000 7ff62d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6280d2370 0x7ff6280ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.113:34828/0 (socket says 172.21.15.113:34828) 2024-04-02T07:35:31.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.096+0000 7ff62d400700 1 -- 172.21.15.113:0/303861956 learned_addr learned my addr 172.21.15.113:0/303861956 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-02T07:35:31.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.096+0000 7ff627e00700 1 --2- 172.21.15.113:0/303861956 >> v2:172.21.15.89:3300/0 conn(0x7ff6280d16a0 0x7ff6280d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:31.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.096+0000 7ff62d400700 1 -- 172.21.15.113:0/303861956 >> v2:172.21.15.89:3300/0 conn(0x7ff6280d16a0 msgr2=0x7ff6280d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:31.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.096+0000 7ff62d400700 1 --2- 172.21.15.113:0/303861956 >> v2:172.21.15.89:3300/0 conn(0x7ff6280d16a0 0x7ff6280d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.096+0000 7ff62d400700 1 -- 172.21.15.113:0/303861956 >> v1:172.21.15.89:6789/0 conn(0x7ff62815d9d0 legacy=0x7ff6280d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.096+0000 7ff62d400700 1 -- 172.21.15.113:0/303861956 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff628161700 con 0x7ff6280d2370 2024-04-02T07:35:31.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.097+0000 7ff62d400700 1 --2- 172.21.15.113:0/303861956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6280d2370 0x7ff6280ce360 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff620009e50 tx=0x7ff62000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=707dc236debe0488 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:31.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.097+0000 7ff627400700 1 -- 172.21.15.113:0/303861956 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62000e070 con 0x7ff6280d2370 2024-04-02T07:35:31.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.097+0000 7ff627400700 1 -- 172.21.15.113:0/303861956 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62000b910 con 0x7ff6280d2370 2024-04-02T07:35:31.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.097+0000 7ff627400700 1 -- 172.21.15.113:0/303861956 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62000bcc0 con 0x7ff6280d2370 2024-04-02T07:35:31.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.097+0000 7ff62f400700 1 -- 172.21.15.113:0/303861956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6280d2370 msgr2=0x7ff6280ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.097+0000 7ff62f400700 1 --2- 172.21.15.113:0/303861956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6280d2370 0x7ff6280ce360 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff620009e50 tx=0x7ff62000b330 comp rx=0 tx=0).stop 2024-04-02T07:35:31.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.098+0000 7ff62f400700 1 -- 172.21.15.113:0/303861956 shutdown_connections 2024-04-02T07:35:31.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.098+0000 7ff62f400700 1 --2- 172.21.15.113:0/303861956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6280d2370 0x7ff6280ce360 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.098+0000 7ff62f400700 1 --2- 172.21.15.113:0/303861956 >> v2:172.21.15.89:3300/0 conn(0x7ff6280d16a0 0x7ff6280d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.098+0000 7ff62f400700 1 -- 172.21.15.113:0/303861956 >> 172.21.15.113:0/303861956 conn(0x7ff62808aff0 msgr2=0x7ff6280d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:31.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.098+0000 7ff62f400700 1 -- 172.21.15.113:0/303861956 shutdown_connections 2024-04-02T07:35:31.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.098+0000 7ff62f400700 1 -- 172.21.15.113:0/303861956 wait complete. 2024-04-02T07:35:31.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.100+0000 7ff62f400700 1 Processor -- start 2024-04-02T07:35:31.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.100+0000 7ff62f400700 1 -- start start 2024-04-02T07:35:31.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.101+0000 7ff62f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6280d16a0 0x7ff628156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:31.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.101+0000 7ff62f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6280d2370 0x7ff6280cb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:31.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.101+0000 7ff62f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff62815d9d0 0x7ff6280ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:31.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.101+0000 7ff62f400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6280cc990 con 0x7ff62815d9d0 2024-04-02T07:35:31.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.101+0000 7ff62f400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6280cc3d0 con 0x7ff6280d2370 2024-04-02T07:35:31.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.101+0000 7ff62f400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6280cc6b0 con 0x7ff6280d16a0 2024-04-02T07:35:31.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.102+0000 7ff62ca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6280d16a0 0x7ff628156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:31.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.102+0000 7ff627e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6280d2370 0x7ff6280cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:31.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.102+0000 7ff62ca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6280d16a0 0x7ff628156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.113:34844/0 (socket says 172.21.15.113:34844) 2024-04-02T07:35:31.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.102+0000 7ff62ca00700 1 -- 172.21.15.113:0/1883248152 learned_addr learned my addr 172.21.15.113:0/1883248152 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-02T07:35:31.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.102+0000 7ff62d400700 1 --2- 172.21.15.113:0/1883248152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff62815d9d0 0x7ff6280ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:31.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.102+0000 7ff62ca00700 1 -- 172.21.15.113:0/1883248152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6280d2370 msgr2=0x7ff6280cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.102+0000 7ff62ca00700 1 --2- 172.21.15.113:0/1883248152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6280d2370 0x7ff6280cb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.102+0000 7ff62ca00700 1 -- 172.21.15.113:0/1883248152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff62815d9d0 msgr2=0x7ff6280ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.102+0000 7ff62ca00700 1 --2- 172.21.15.113:0/1883248152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff62815d9d0 0x7ff6280ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.102+0000 7ff62ca00700 1 -- 172.21.15.113:0/1883248152 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff620009b30 con 0x7ff6280d16a0 2024-04-02T07:35:31.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.102+0000 7ff62d400700 1 --2- 172.21.15.113:0/1883248152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff62815d9d0 0x7ff6280ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:31.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.102+0000 7ff62ca00700 1 --2- 172.21.15.113:0/1883248152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6280d16a0 0x7ff628156c80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00c450 tx=0x7ff61c00c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:31.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.103+0000 7ff624c00700 1 -- 172.21.15.113:0/1883248152 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c0052d0 con 0x7ff6280d16a0 2024-04-02T07:35:31.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.103+0000 7ff624c00700 1 -- 172.21.15.113:0/1883248152 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61c005430 con 0x7ff6280d16a0 2024-04-02T07:35:31.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.103+0000 7ff624c00700 1 -- 172.21.15.113:0/1883248152 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c0038c0 con 0x7ff6280d16a0 2024-04-02T07:35:31.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.103+0000 7ff62f400700 1 -- 172.21.15.113:0/1883248152 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62815df50 con 0x7ff6280d16a0 2024-04-02T07:35:31.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.103+0000 7ff62f400700 1 -- 172.21.15.113:0/1883248152 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6281571c0 con 0x7ff6280d16a0 2024-04-02T07:35:31.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.103+0000 7ff624c00700 1 -- 172.21.15.113:0/1883248152 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff61c003d20 con 0x7ff6280d16a0 2024-04-02T07:35:31.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.103+0000 7ff624c00700 1 -- 172.21.15.113:0/1883248152 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7ff61c011d90 con 0x7ff6280d16a0 2024-04-02T07:35:31.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.104+0000 7ff62f400700 1 -- 172.21.15.113:0/1883248152 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6280d1ca0 con 0x7ff6280d16a0 2024-04-02T07:35:31.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.107+0000 7ff624c00700 1 -- 172.21.15.113:0/1883248152 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7ff61c017070 con 0x7ff6280d16a0 2024-04-02T07:35:31.133 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:31.132+0000 7f016525e240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:35:31.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.191+0000 7ff62f400700 1 -- 172.21.15.113:0/1883248152 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd new", "uuid": "cb9001a3-5ee2-4c09-afcf-881626477a6e", "id": 4} v 0) v1 -- 0x7ff62815ddb0 con 0x7ff6280d16a0 2024-04-02T07:35:31.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.195+0000 7ff624c00700 1 -- 172.21.15.113:0/1883248152 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cb9001a3-5ee2-4c09-afcf-881626477a6e", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff62815ddb0 con 0x7ff6280d16a0 2024-04-02T07:35:31.197 INFO:teuthology.orchestra.run.smithi113.stdout:4 2024-04-02T07:35:31.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.196+0000 7ff62f400700 1 -- 172.21.15.113:0/1883248152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6280d16a0 msgr2=0x7ff628156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.196+0000 7ff62f400700 1 --2- 172.21.15.113:0/1883248152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6280d16a0 0x7ff628156c80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00c450 tx=0x7ff61c00c760 comp rx=0 tx=0).stop 2024-04-02T07:35:31.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.197+0000 7ff62f400700 1 -- 172.21.15.113:0/1883248152 shutdown_connections 2024-04-02T07:35:31.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.197+0000 7ff62f400700 1 --2- 172.21.15.113:0/1883248152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff62815d9d0 0x7ff6280ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.197+0000 7ff62f400700 1 --2- 172.21.15.113:0/1883248152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6280d16a0 0x7ff628156c80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.197+0000 7ff62f400700 1 --2- 172.21.15.113:0/1883248152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6280d2370 0x7ff6280cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.197+0000 7ff62f400700 1 -- 172.21.15.113:0/1883248152 >> 172.21.15.113:0/1883248152 conn(0x7ff62808aff0 msgr2=0x7ff6280b1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:31.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.197+0000 7ff62f400700 1 -- 172.21.15.113:0/1883248152 shutdown_connections 2024-04-02T07:35:31.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.197+0000 7ff62f400700 1 -- 172.21.15.113:0/1883248152 wait complete. 2024-04-02T07:35:31.203 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:31.201+0000 7f826b9b7240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:35:31.214 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new 9678135c-a55d-4a00-b68f-4605313c9756 5 2024-04-02T07:35:31.312 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:31.311+0000 7f016525e240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:35:31.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.400+0000 7f5024400700 1 Processor -- start 2024-04-02T07:35:31.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.400+0000 7f5024400700 1 -- start start 2024-04-02T07:35:31.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.401+0000 7f5024400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f501c0d16a0 0x7f501c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:31.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.401+0000 7f5024400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501c161420 con 0x7f501c15d9d0 2024-04-02T07:35:31.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.401+0000 7f5024400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501c161700 con 0x7f501c0d2370 2024-04-02T07:35:31.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.401+0000 7f5024400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501c0d2820 con 0x7f501c0d16a0 2024-04-02T07:35:31.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.401+0000 7f5022400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f501c0d2370 0x7f501c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:60618/0 (socket says 172.21.15.113:60618) 2024-04-02T07:35:31.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.401+0000 7f5021000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f501c0d16a0 0x7f501c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:31.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.401+0000 7f5022400700 1 -- 172.21.15.113:0/2465790384 learned_addr learned my addr 172.21.15.113:0/2465790384 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-04-02T07:35:31.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.402+0000 7f501be00700 1 -- 172.21.15.113:0/2465790384 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473804374 0 0) 0x7f501c161700 con 0x7f501c0d2370 2024-04-02T07:35:31.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.402+0000 7f501be00700 1 -- 172.21.15.113:0/2465790384 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50040035b0 con 0x7f501c0d2370 2024-04-02T07:35:31.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.402+0000 7f501be00700 1 -- 172.21.15.113:0/2465790384 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096768569 0 0) 0x7f501c161420 con 0x7f501c15d9d0 2024-04-02T07:35:31.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.402+0000 7f5021000700 1 -- 172.21.15.113:0/2465790384 >> v1:172.21.15.113:6789/0 conn(0x7f501c0d2370 legacy=0x7f501c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.402+0000 7f5021000700 1 -- 172.21.15.113:0/2465790384 >> v1:172.21.15.89:6789/0 conn(0x7f501c15d9d0 legacy=0x7f501c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.402+0000 7f5021000700 1 -- 172.21.15.113:0/2465790384 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501c1619e0 con 0x7f501c0d16a0 2024-04-02T07:35:31.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.402+0000 7f5021000700 1 --2- 172.21.15.113:0/2465790384 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f501c0d16a0 0x7f501c0d1e30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5008009850 tx=0x7f5008009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c40e4d9afddc1966 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:31.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.402+0000 7f501be00700 1 -- 172.21.15.113:0/2465790384 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5008005e30 con 0x7f501c0d16a0 2024-04-02T07:35:31.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.402+0000 7f501be00700 1 -- 172.21.15.113:0/2465790384 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5008006030 con 0x7f501c0d16a0 2024-04-02T07:35:31.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.402+0000 7f501be00700 1 -- 172.21.15.113:0/2465790384 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50080063a0 con 0x7f501c0d16a0 2024-04-02T07:35:31.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.403+0000 7f5024400700 1 -- 172.21.15.113:0/2465790384 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f501c0d16a0 msgr2=0x7f501c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.403+0000 7f5024400700 1 --2- 172.21.15.113:0/2465790384 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f501c0d16a0 0x7f501c0d1e30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5008009850 tx=0x7f5008009b60 comp rx=0 tx=0).stop 2024-04-02T07:35:31.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.403+0000 7f5024400700 1 -- 172.21.15.113:0/2465790384 shutdown_connections 2024-04-02T07:35:31.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.403+0000 7f5024400700 1 --2- 172.21.15.113:0/2465790384 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f501c0d16a0 0x7f501c0d1e30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.403+0000 7f5024400700 1 -- 172.21.15.113:0/2465790384 >> 172.21.15.113:0/2465790384 conn(0x7f501c08aff0 msgr2=0x7f501c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:31.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.403+0000 7f5024400700 1 -- 172.21.15.113:0/2465790384 shutdown_connections 2024-04-02T07:35:31.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.403+0000 7f5024400700 1 -- 172.21.15.113:0/2465790384 wait complete. 2024-04-02T07:35:31.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.405+0000 7f5024400700 1 Processor -- start 2024-04-02T07:35:31.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.405+0000 7f5024400700 1 -- start start 2024-04-02T07:35:31.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.406+0000 7f5024400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f501c0d16a0 0x7f501c14fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:31.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.406+0000 7f5024400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f501c0d2370 0x7f501c15f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:31.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.406+0000 7f5024400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f501c15d9d0 0x7f501c0b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:31.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.406+0000 7f5024400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c14ff50 con 0x7f501c0d2370 2024-04-02T07:35:31.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.406+0000 7f5024400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c15ddb0 con 0x7f501c15d9d0 2024-04-02T07:35:31.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.406+0000 7f5024400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501c15e000 con 0x7f501c0d16a0 2024-04-02T07:35:31.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.406+0000 7f5021000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f501c0d2370 0x7f501c15f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:31.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.406+0000 7f5022400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f501c15d9d0 0x7f501c0b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:31.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.406+0000 7f5021a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f501c0d16a0 0x7f501c14fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:31.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.406+0000 7f5021000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f501c0d2370 0x7f501c15f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.113:35078/0 (socket says 172.21.15.113:35078) 2024-04-02T07:35:31.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.406+0000 7f5021000700 1 -- 172.21.15.113:0/130421625 learned_addr learned my addr 172.21.15.113:0/130421625 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-02T07:35:31.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.407+0000 7f5021000700 1 -- 172.21.15.113:0/130421625 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f501c0d16a0 msgr2=0x7f501c14fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.407+0000 7f5021000700 1 --2- 172.21.15.113:0/130421625 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f501c0d16a0 0x7f501c14fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.407+0000 7f5021000700 1 -- 172.21.15.113:0/130421625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f501c15d9d0 msgr2=0x7f501c0b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.407+0000 7f5021000700 1 --2- 172.21.15.113:0/130421625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f501c15d9d0 0x7f501c0b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.407+0000 7f5021000700 1 -- 172.21.15.113:0/130421625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5008009430 con 0x7f501c0d2370 2024-04-02T07:35:31.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.407+0000 7f5021a00700 1 --2- 172.21.15.113:0/130421625 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f501c0d16a0 0x7f501c14fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:31.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.407+0000 7f5022400700 1 --2- 172.21.15.113:0/130421625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f501c15d9d0 0x7f501c0b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:31.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.407+0000 7f5021000700 1 --2- 172.21.15.113:0/130421625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f501c0d2370 0x7f501c15f0a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5008012000 tx=0x7f5008004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:31.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.407+0000 7f5019600700 1 -- 172.21.15.113:0/130421625 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5008006890 con 0x7f501c0d2370 2024-04-02T07:35:31.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.407+0000 7f5019600700 1 -- 172.21.15.113:0/130421625 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5008020070 con 0x7f501c0d2370 2024-04-02T07:35:31.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.407+0000 7f5019600700 1 -- 172.21.15.113:0/130421625 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5008004240 con 0x7f501c0d2370 2024-04-02T07:35:31.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.407+0000 7f5024400700 1 -- 172.21.15.113:0/130421625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f501c15e2e0 con 0x7f501c0d2370 2024-04-02T07:35:31.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.408+0000 7f5024400700 1 -- 172.21.15.113:0/130421625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501c15e8a0 con 0x7f501c0d2370 2024-04-02T07:35:31.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.408+0000 7f5019600700 1 -- 172.21.15.113:0/130421625 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5008022030 con 0x7f501c0d2370 2024-04-02T07:35:31.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.408+0000 7f5019600700 1 -- 172.21.15.113:0/130421625 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f5008025430 con 0x7f501c0d2370 2024-04-02T07:35:31.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.409+0000 7f5024400700 1 -- 172.21.15.113:0/130421625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501c0d1ca0 con 0x7f501c0d2370 2024-04-02T07:35:31.413 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:31.410+0000 7f826b9b7240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:35:31.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.413+0000 7f5019600700 1 -- 172.21.15.113:0/130421625 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f50080036a0 con 0x7f501c0d2370 2024-04-02T07:35:31.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.500+0000 7f5024400700 1 -- 172.21.15.113:0/130421625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9678135c-a55d-4a00-b68f-4605313c9756", "id": 5} v 0) v1 -- 0x7f501c067720 con 0x7f501c0d2370 2024-04-02T07:35:31.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.504+0000 7f5019600700 1 -- 172.21.15.113:0/130421625 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9678135c-a55d-4a00-b68f-4605313c9756", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5008003840 con 0x7f501c0d2370 2024-04-02T07:35:31.505 INFO:teuthology.orchestra.run.smithi113.stdout:5 2024-04-02T07:35:31.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.505+0000 7f5024400700 1 -- 172.21.15.113:0/130421625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f501c0d2370 msgr2=0x7f501c15f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.505+0000 7f5024400700 1 --2- 172.21.15.113:0/130421625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f501c0d2370 0x7f501c15f0a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5008012000 tx=0x7f5008004750 comp rx=0 tx=0).stop 2024-04-02T07:35:31.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.505+0000 7f5024400700 1 -- 172.21.15.113:0/130421625 shutdown_connections 2024-04-02T07:35:31.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.505+0000 7f5024400700 1 --2- 172.21.15.113:0/130421625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f501c15d9d0 0x7f501c0b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.505+0000 7f5024400700 1 --2- 172.21.15.113:0/130421625 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f501c0d16a0 0x7f501c14fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.505+0000 7f5024400700 1 --2- 172.21.15.113:0/130421625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f501c0d2370 0x7f501c15f0a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.505+0000 7f5024400700 1 -- 172.21.15.113:0/130421625 >> 172.21.15.113:0/130421625 conn(0x7f501c08aff0 msgr2=0x7f501c0cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.505+0000 7f5024400700 1 -- 172.21.15.113:0/130421625 shutdown_connections 2024-04-02T07:35:31.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.505+0000 7f5024400700 1 -- 172.21.15.113:0/130421625 wait complete. 2024-04-02T07:35:31.507 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:31.505+0000 7f826b9b7240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:35:31.523 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new 6c67a32e-252d-4449-b039-35b67c16f38e 6 2024-04-02T07:35:31.529 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:31.528+0000 7f016525e240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:35:31.604 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:31.602+0000 7f826b9b7240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:35:31.629 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:31.628+0000 7f016525e240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:35:31.701 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:31.698+0000 7f826b9b7240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:35:31.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.710+0000 7f27b1800700 1 Processor -- start 2024-04-02T07:35:31.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.710+0000 7f27b1800700 1 -- start start 2024-04-02T07:35:31.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.711+0000 7f27b1800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f27ac15d230 0x7f27ac0d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:31.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.711+0000 7f27b1800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27ac15d610 0x7f27ac0d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:31.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.711+0000 7f27b1800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ac0b6a40 con 0x7f27ac0b8fe0 2024-04-02T07:35:31.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.711+0000 7f27b1800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27ac0b6cf0 con 0x7f27ac15d230 2024-04-02T07:35:31.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.711+0000 7f27b1800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27ac0b6fd0 con 0x7f27ac15d610 2024-04-02T07:35:31.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.712+0000 7f27abe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f27ac15d230 0x7f27ac0d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:31.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.712+0000 7f27b0e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27ac15d610 0x7f27ac0d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:31.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.712+0000 7f27b0e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27ac15d610 0x7f27ac0d02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.113:34878/0 (socket says 172.21.15.113:34878) 2024-04-02T07:35:31.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.712+0000 7f27b0e00700 1 -- 172.21.15.113:0/1848284624 learned_addr learned my addr 172.21.15.113:0/1848284624 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-02T07:35:31.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.712+0000 7f27abe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f27ac15d230 0x7f27ac0d1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36754/0 (socket says 172.21.15.113:36754) 2024-04-02T07:35:31.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.712+0000 7f27a3e00700 1 --1- 172.21.15.113:0/1848284624 >> v1:172.21.15.113:6789/0 conn(0x7f27ac0b8fe0 0x7f27ac0b6330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:60622/0 (socket says 172.21.15.113:60622) 2024-04-02T07:35:31.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.712+0000 7f27b0e00700 1 -- 172.21.15.113:0/1848284624 >> v2:172.21.15.113:3300/0 conn(0x7f27ac15d230 msgr2=0x7f27ac0d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:31.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.712+0000 7f27b0e00700 1 --2- 172.21.15.113:0/1848284624 >> v2:172.21.15.113:3300/0 conn(0x7f27ac15d230 0x7f27ac0d1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.712+0000 7f27b0e00700 1 -- 172.21.15.113:0/1848284624 >> v1:172.21.15.113:6789/0 conn(0x7f27ac0b8fe0 legacy=0x7f27ac0b6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.712+0000 7f27b0e00700 1 -- 172.21.15.113:0/1848284624 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ac0b72b0 con 0x7f27ac15d610 2024-04-02T07:35:31.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.712+0000 7f27b0e00700 1 --2- 172.21.15.113:0/1848284624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27ac15d610 0x7f27ac0d02e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f27a4009e80 tx=0x7f27a400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=257915d74b2dfee2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:31.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.713+0000 7f27ab400700 1 -- 172.21.15.113:0/1848284624 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a400e070 con 0x7f27ac15d610 2024-04-02T07:35:31.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.713+0000 7f27ab400700 1 -- 172.21.15.113:0/1848284624 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a400b8d0 con 0x7f27ac15d610 2024-04-02T07:35:31.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.713+0000 7f27ab400700 1 -- 172.21.15.113:0/1848284624 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a400bc80 con 0x7f27ac15d610 2024-04-02T07:35:31.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.713+0000 7f27b1800700 1 -- 172.21.15.113:0/1848284624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27ac15d610 msgr2=0x7f27ac0d02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.713+0000 7f27b1800700 1 --2- 172.21.15.113:0/1848284624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27ac15d610 0x7f27ac0d02e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f27a4009e80 tx=0x7f27a400b330 comp rx=0 tx=0).stop 2024-04-02T07:35:31.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.713+0000 7f27b1800700 1 -- 172.21.15.113:0/1848284624 shutdown_connections 2024-04-02T07:35:31.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.713+0000 7f27b1800700 1 --2- 172.21.15.113:0/1848284624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27ac15d610 0x7f27ac0d02e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.713+0000 7f27b1800700 1 --2- 172.21.15.113:0/1848284624 >> v2:172.21.15.113:3300/0 conn(0x7f27ac15d230 0x7f27ac0d1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.713+0000 7f27b1800700 1 -- 172.21.15.113:0/1848284624 >> 172.21.15.113:0/1848284624 conn(0x7f27ac0668e0 msgr2=0x7f27ac066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:31.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.713+0000 7f27b1800700 1 -- 172.21.15.113:0/1848284624 shutdown_connections 2024-04-02T07:35:31.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.713+0000 7f27b1800700 1 -- 172.21.15.113:0/1848284624 wait complete. 2024-04-02T07:35:31.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.715+0000 7f27b1800700 1 Processor -- start 2024-04-02T07:35:31.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.715+0000 7f27b1800700 1 -- start start 2024-04-02T07:35:31.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.716+0000 7f27b1800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27ac0b8fe0 0x7f27ac160720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:31.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.716+0000 7f27b1800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27ac15d230 0x7f27ac153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:31.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.716+0000 7f27b1800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27ac15d610 0x7f27ac0c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:31.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.716+0000 7f27b1800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ac15dad0 con 0x7f27ac15d230 2024-04-02T07:35:31.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.716+0000 7f27b1800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ac15e650 con 0x7f27ac0b8fe0 2024-04-02T07:35:31.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.716+0000 7f27b1800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27ac15dd20 con 0x7f27ac15d610 2024-04-02T07:35:31.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27b0e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27ac15d610 0x7f27ac0c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:31.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27abe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27ac0b8fe0 0x7f27ac160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:31.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27b0e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27ac15d610 0x7f27ac0c7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.113:34892/0 (socket says 172.21.15.113:34892) 2024-04-02T07:35:31.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27b0e00700 1 -- 172.21.15.113:0/1034159867 learned_addr learned my addr 172.21.15.113:0/1034159867 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-02T07:35:31.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27a3e00700 1 --2- 172.21.15.113:0/1034159867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27ac15d230 0x7f27ac153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:31.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27b0e00700 1 -- 172.21.15.113:0/1034159867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27ac0b8fe0 msgr2=0x7f27ac160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27b0e00700 1 --2- 172.21.15.113:0/1034159867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27ac0b8fe0 0x7f27ac160720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27b0e00700 1 -- 172.21.15.113:0/1034159867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27ac15d230 msgr2=0x7f27ac153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27b0e00700 1 --2- 172.21.15.113:0/1034159867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27ac15d230 0x7f27ac153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27b0e00700 1 -- 172.21.15.113:0/1034159867 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a4009b30 con 0x7f27ac15d610 2024-04-02T07:35:31.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27a3e00700 1 --2- 172.21.15.113:0/1034159867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27ac15d230 0x7f27ac153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:35:31.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27b0e00700 1 --2- 172.21.15.113:0/1034159867 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27ac15d610 0x7f27ac0c7ad0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f27ac0b7110 tx=0x7f27a4004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:31.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27a8c00700 1 -- 172.21.15.113:0/1034159867 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a40048b0 con 0x7f27ac15d610 2024-04-02T07:35:31.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27a8c00700 1 -- 172.21.15.113:0/1034159867 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a40099b0 con 0x7f27ac15d610 2024-04-02T07:35:31.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27b1800700 1 -- 172.21.15.113:0/1034159867 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ac15e000 con 0x7f27ac15d610 2024-04-02T07:35:31.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.717+0000 7f27a8c00700 1 -- 172.21.15.113:0/1034159867 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a401b670 con 0x7f27ac15d610 2024-04-02T07:35:31.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.718+0000 7f27b1800700 1 -- 172.21.15.113:0/1034159867 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ac154a80 con 0x7f27ac15d610 2024-04-02T07:35:31.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.718+0000 7f27a8c00700 1 -- 172.21.15.113:0/1034159867 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f27a401bad0 con 0x7f27ac15d610 2024-04-02T07:35:31.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.718+0000 7f27a8c00700 1 -- 172.21.15.113:0/1034159867 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f27a401bd80 con 0x7f27ac15d610 2024-04-02T07:35:31.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.719+0000 7f27b1800700 1 -- 172.21.15.113:0/1034159867 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ac04ea80 con 0x7f27ac15d610 2024-04-02T07:35:31.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.721+0000 7f27a8c00700 1 -- 172.21.15.113:0/1034159867 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f27a401f020 con 0x7f27ac15d610 2024-04-02T07:35:31.734 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:31.733+0000 7f016525e240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:35:31.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.808+0000 7f27b1800700 1 -- 172.21.15.113:0/1034159867 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6c67a32e-252d-4449-b039-35b67c16f38e", "id": 6} v 0) v1 -- 0x7f27ac15e790 con 0x7f27ac15d610 2024-04-02T07:35:31.810 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:31.808+0000 7f826b9b7240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:35:31.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.812+0000 7f27a8c00700 1 -- 172.21.15.113:0/1034159867 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6c67a32e-252d-4449-b039-35b67c16f38e", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f27a4024410 con 0x7f27ac15d610 2024-04-02T07:35:31.813 INFO:teuthology.orchestra.run.smithi113.stdout:6 2024-04-02T07:35:31.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.813+0000 7f27b1800700 1 -- 172.21.15.113:0/1034159867 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27ac15d610 msgr2=0x7f27ac0c7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:31.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.813+0000 7f27b1800700 1 --2- 172.21.15.113:0/1034159867 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27ac15d610 0x7f27ac0c7ad0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f27ac0b7110 tx=0x7f27a4004ad0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.813+0000 7f27b1800700 1 -- 172.21.15.113:0/1034159867 shutdown_connections 2024-04-02T07:35:31.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.813+0000 7f27b1800700 1 --2- 172.21.15.113:0/1034159867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27ac0b8fe0 0x7f27ac160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.813+0000 7f27b1800700 1 --2- 172.21.15.113:0/1034159867 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27ac15d610 0x7f27ac0c7ad0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.813+0000 7f27b1800700 1 --2- 172.21.15.113:0/1034159867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27ac15d230 0x7f27ac153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:31.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.813+0000 7f27b1800700 1 -- 172.21.15.113:0/1034159867 >> 172.21.15.113:0/1034159867 conn(0x7f27ac0668e0 msgr2=0x7f27ac0b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:31.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.813+0000 7f27b1800700 1 -- 172.21.15.113:0/1034159867 shutdown_connections 2024-04-02T07:35:31.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:31.813+0000 7f27b1800700 1 -- 172.21.15.113:0/1034159867 wait complete. 2024-04-02T07:35:31.832 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new c4564c2a-1b9d-4ed2-a54b-19158f3dc757 7 2024-04-02T07:35:31.836 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:31.835+0000 7f016525e240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:35:31.958 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:31.956+0000 7f016525e240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:35:32.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.022+0000 7fb7d0600700 1 Processor -- start 2024-04-02T07:35:32.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.022+0000 7fb7d0600700 1 -- start start 2024-04-02T07:35:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.023+0000 7fb7d0600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb7c80d16a0 0x7fb7c80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.023+0000 7fb7d0600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7c80d2370 0x7fb7c80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.023+0000 7fb7d0600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c8161420 con 0x7fb7c815d9d0 2024-04-02T07:35:32.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.023+0000 7fb7d0600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c80d4340 con 0x7fb7c80d2370 2024-04-02T07:35:32.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.023+0000 7fb7d0600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c815edc0 con 0x7fb7c80d16a0 2024-04-02T07:35:32.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.024+0000 7fb7cd200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb7c80d16a0 0x7fb7c80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:32.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.024+0000 7fb7cd200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb7c80d16a0 0x7fb7c80d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36786/0 (socket says 172.21.15.113:36786) 2024-04-02T07:35:32.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.024+0000 7fb7cd200700 1 -- 172.21.15.113:0/1177295347 learned_addr learned my addr 172.21.15.113:0/1177295347 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-02T07:35:32.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.024+0000 7fb7ce600700 1 --2- 172.21.15.113:0/1177295347 >> v2:172.21.15.89:3300/0 conn(0x7fb7c80d2370 0x7fb7c80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:32.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.024+0000 7fb7cd200700 1 -- 172.21.15.113:0/1177295347 >> v2:172.21.15.89:3300/0 conn(0x7fb7c80d2370 msgr2=0x7fb7c80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:32.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.024+0000 7fb7cd200700 1 --2- 172.21.15.113:0/1177295347 >> v2:172.21.15.89:3300/0 conn(0x7fb7c80d2370 0x7fb7c80ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.024+0000 7fb7cd200700 1 -- 172.21.15.113:0/1177295347 >> v1:172.21.15.89:6789/0 conn(0x7fb7c815d9d0 legacy=0x7fb7c80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:32.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.024+0000 7fb7c7e00700 1 -- 172.21.15.113:0/1177295347 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917799833 0 0) 0x7fb7c8161420 con 0x7fb7c815d9d0 2024-04-02T07:35:32.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.024+0000 7fb7cd200700 1 -- 172.21.15.113:0/1177295347 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c8161700 con 0x7fb7c80d16a0 2024-04-02T07:35:32.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.024+0000 7fb7cd200700 1 --2- 172.21.15.113:0/1177295347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d16a0 0x7fb7c80d1e30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80d12f0 tx=0x7fb7b4009bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bcba865200bb08a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:32.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.025+0000 7fb7c7e00700 1 -- 172.21.15.113:0/1177295347 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4011830 con 0x7fb7c80d16a0 2024-04-02T07:35:32.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.025+0000 7fb7c7e00700 1 -- 172.21.15.113:0/1177295347 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b4011990 con 0x7fb7c80d16a0 2024-04-02T07:35:32.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.025+0000 7fb7c7e00700 1 -- 172.21.15.113:0/1177295347 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4011d40 con 0x7fb7c80d16a0 2024-04-02T07:35:32.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.025+0000 7fb7d0600700 1 -- 172.21.15.113:0/1177295347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d16a0 msgr2=0x7fb7c80d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:32.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.025+0000 7fb7d0600700 1 --2- 172.21.15.113:0/1177295347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d16a0 0x7fb7c80d1e30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80d12f0 tx=0x7fb7b4009bc0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.025+0000 7fb7d0600700 1 -- 172.21.15.113:0/1177295347 shutdown_connections 2024-04-02T07:35:32.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.025+0000 7fb7d0600700 1 --2- 172.21.15.113:0/1177295347 >> v2:172.21.15.89:3300/0 conn(0x7fb7c80d2370 0x7fb7c80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.025+0000 7fb7d0600700 1 --2- 172.21.15.113:0/1177295347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d16a0 0x7fb7c80d1e30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.025+0000 7fb7d0600700 1 -- 172.21.15.113:0/1177295347 >> 172.21.15.113:0/1177295347 conn(0x7fb7c808aff0 msgr2=0x7fb7c80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:32.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.026+0000 7fb7d0600700 1 -- 172.21.15.113:0/1177295347 shutdown_connections 2024-04-02T07:35:32.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.026+0000 7fb7d0600700 1 -- 172.21.15.113:0/1177295347 wait complete. 2024-04-02T07:35:32.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.027+0000 7fb7d0600700 1 Processor -- start 2024-04-02T07:35:32.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.028+0000 7fb7d0600700 1 -- start start 2024-04-02T07:35:32.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.028+0000 7fb7d0600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d16a0 0x7fb7c80cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:32.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.028+0000 7fb7d0600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c80d2370 0x7fb7c80b1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.028+0000 7fb7d0600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c815d9d0 0x7fb7c814b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.028+0000 7fb7d0600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80b7710 con 0x7fb7c815d9d0 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.028+0000 7fb7d0600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c81f6ef0 con 0x7fb7c80d16a0 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.029+0000 7fb7d0600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c81f71d0 con 0x7fb7c80d2370 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.029+0000 7fb7cdc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d16a0 0x7fb7c80cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.029+0000 7fb7cd200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c80d2370 0x7fb7c80b1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.029+0000 7fb7cd200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c80d2370 0x7fb7c80b1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.113:34896/0 (socket says 172.21.15.113:34896) 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.029+0000 7fb7cd200700 1 -- 172.21.15.113:0/3172773520 learned_addr learned my addr 172.21.15.113:0/3172773520 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.029+0000 7fb7cdc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d16a0 0x7fb7c80cb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36792/0 (socket says 172.21.15.113:36792) 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.029+0000 7fb7ce600700 1 --2- 172.21.15.113:0/3172773520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c815d9d0 0x7fb7c814b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.029+0000 7fb7cd200700 1 -- 172.21.15.113:0/3172773520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d16a0 msgr2=0x7fb7c80cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.029+0000 7fb7cd200700 1 --2- 172.21.15.113:0/3172773520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d16a0 0x7fb7c80cb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.029+0000 7fb7cd200700 1 -- 172.21.15.113:0/3172773520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c815d9d0 msgr2=0x7fb7c814b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.029+0000 7fb7cd200700 1 --2- 172.21.15.113:0/3172773520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c815d9d0 0x7fb7c814b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.029+0000 7fb7cd200700 1 -- 172.21.15.113:0/3172773520 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b40094f0 con 0x7fb7c80d2370 2024-04-02T07:35:32.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.029+0000 7fb7cdc00700 1 --2- 172.21.15.113:0/3172773520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d16a0 0x7fb7c80cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:32.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.029+0000 7fb7cd200700 1 --2- 172.21.15.113:0/3172773520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c80d2370 0x7fb7c80b1860 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009130 tx=0x7fb7b4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:32.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.030+0000 7fb7c5600700 1 -- 172.21.15.113:0/3172773520 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4004440 con 0x7fb7c80d2370 2024-04-02T07:35:32.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.030+0000 7fb7c5600700 1 -- 172.21.15.113:0/3172773520 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b40045a0 con 0x7fb7c80d2370 2024-04-02T07:35:32.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.030+0000 7fb7c5600700 1 -- 172.21.15.113:0/3172773520 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40038a0 con 0x7fb7c80d2370 2024-04-02T07:35:32.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.030+0000 7fb7d0600700 1 -- 172.21.15.113:0/3172773520 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c81f74b0 con 0x7fb7c80d2370 2024-04-02T07:35:32.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.030+0000 7fb7d0600700 1 -- 172.21.15.113:0/3172773520 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c80cbe90 con 0x7fb7c80d2370 2024-04-02T07:35:32.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.030+0000 7fb7c5600700 1 -- 172.21.15.113:0/3172773520 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb7b4003d00 con 0x7fb7c80d2370 2024-04-02T07:35:32.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.030+0000 7fb7c5600700 1 -- 172.21.15.113:0/3172773520 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fb7b4021030 con 0x7fb7c80d2370 2024-04-02T07:35:32.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.031+0000 7fb7d0600700 1 -- 172.21.15.113:0/3172773520 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c80d1ca0 con 0x7fb7c80d2370 2024-04-02T07:35:32.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.033+0000 7fb7c5600700 1 -- 172.21.15.113:0/3172773520 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fb7b402b060 con 0x7fb7c80d2370 2024-04-02T07:35:32.055 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:32.052+0000 7f826b9b7240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:35:32.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.120+0000 7fb7d0600700 1 -- 172.21.15.113:0/3172773520 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c4564c2a-1b9d-4ed2-a54b-19158f3dc757", "id": 7} v 0) v1 -- 0x7fb7c80b7880 con 0x7fb7c80d2370 2024-04-02T07:35:32.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.128+0000 7fb7c5600700 1 -- 172.21.15.113:0/3172773520 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c4564c2a-1b9d-4ed2-a54b-19158f3dc757", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb7b4015020 con 0x7fb7c80d2370 2024-04-02T07:35:32.129 INFO:teuthology.orchestra.run.smithi113.stdout:7 2024-04-02T07:35:32.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.128+0000 7fb7d0600700 1 -- 172.21.15.113:0/3172773520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c80d2370 msgr2=0x7fb7c80b1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:32.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.128+0000 7fb7d0600700 1 --2- 172.21.15.113:0/3172773520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c80d2370 0x7fb7c80b1860 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009130 tx=0x7fb7b4004750 comp rx=0 tx=0).stop 2024-04-02T07:35:32.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.129+0000 7fb7d0600700 1 -- 172.21.15.113:0/3172773520 shutdown_connections 2024-04-02T07:35:32.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.129+0000 7fb7d0600700 1 --2- 172.21.15.113:0/3172773520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d16a0 0x7fb7c80cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.129+0000 7fb7d0600700 1 --2- 172.21.15.113:0/3172773520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c815d9d0 0x7fb7c814b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.129+0000 7fb7d0600700 1 --2- 172.21.15.113:0/3172773520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c80d2370 0x7fb7c80b1860 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.129+0000 7fb7d0600700 1 -- 172.21.15.113:0/3172773520 >> 172.21.15.113:0/3172773520 conn(0x7fb7c808aff0 msgr2=0x7fb7c80b01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:32.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.129+0000 7fb7d0600700 1 -- 172.21.15.113:0/3172773520 shutdown_connections 2024-04-02T07:35:32.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-02T07:35:32.129+0000 7fb7d0600700 1 -- 172.21.15.113:0/3172773520 wait complete. 2024-04-02T07:35:32.146 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-02T07:35:32.147 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-02T07:35:32.151 INFO:tasks.ceph.osd.0:Started 2024-04-02T07:35:32.151 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-02T07:35:32.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-02T07:35:32.155 INFO:tasks.ceph.osd.1:Started 2024-04-02T07:35:32.155 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-02T07:35:32.155 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-02T07:35:32.159 INFO:tasks.ceph.osd.2:Started 2024-04-02T07:35:32.159 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-02T07:35:32.160 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-02T07:35:32.164 INFO:tasks.ceph.osd.3:Started 2024-04-02T07:35:32.164 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-02T07:35:32.165 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-02T07:35:32.168 INFO:tasks.ceph.osd.4:Started 2024-04-02T07:35:32.168 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-02T07:35:32.169 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-02T07:35:32.172 INFO:tasks.ceph.osd.5:Started 2024-04-02T07:35:32.172 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-02T07:35:32.172 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-02T07:35:32.176 INFO:tasks.ceph.osd.6:Started 2024-04-02T07:35:32.176 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-02T07:35:32.177 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-02T07:35:32.181 INFO:tasks.ceph.osd.7:Started 2024-04-02T07:35:32.181 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:35:32.230 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:32.229+0000 7f016525e240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:35:32.369 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:32.366+0000 7f826b9b7240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:35:32.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.447+0000 7fb954c00700 1 Processor -- start 2024-04-02T07:35:32.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.447+0000 7fb954c00700 1 -- start start 2024-04-02T07:35:32.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.449+0000 7fb954c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb9500d2870 0x7fb9500d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:32.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.449+0000 7fb954c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9500d2dc0 con 0x7fb9500d0a00 2024-04-02T07:35:32.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.449+0000 7fb954c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9500d30a0 con 0x7fb95015d5d0 2024-04-02T07:35:32.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.449+0000 7fb954c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95015d9b0 con 0x7fb9500d2870 2024-04-02T07:35:32.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.449+0000 7fb94ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb9500d0a00 0x7fb9500d0f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46210/0 (socket says 172.21.15.89:46210) 2024-04-02T07:35:32.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.449+0000 7fb94ea00700 1 -- 172.21.15.89:0/2156253365 learned_addr learned my addr 172.21.15.89:0/2156253365 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:32.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.450+0000 7fb94fe00700 1 --2- 172.21.15.89:0/2156253365 >> v2:172.21.15.113:3300/0 conn(0x7fb9500d2870 0x7fb9500d3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:32.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.450+0000 7fb94e000700 1 -- 172.21.15.89:0/2156253365 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633294777 0 0) 0x7fb9500d2dc0 con 0x7fb9500d0a00 2024-04-02T07:35:32.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.450+0000 7fb94e000700 1 -- 172.21.15.89:0/2156253365 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9340035b0 con 0x7fb9500d0a00 2024-04-02T07:35:32.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.450+0000 7fb94fe00700 1 -- 172.21.15.89:0/2156253365 >> v1:172.21.15.113:6789/0 conn(0x7fb95015d5d0 legacy=0x7fb9500d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:32.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.450+0000 7fb94fe00700 1 -- 172.21.15.89:0/2156253365 >> v1:172.21.15.89:6789/0 conn(0x7fb9500d0a00 legacy=0x7fb9500d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.450+0000 7fb94fe00700 1 -- 172.21.15.89:0/2156253365 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9500d3380 con 0x7fb9500d2870 2024-04-02T07:35:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.450+0000 7fb94fe00700 1 --2- 172.21.15.89:0/2156253365 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9500d2870 0x7fb9500d3e60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb948009d90 tx=0x7fb94800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0ea6b1978937279 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.451+0000 7fb94e000700 1 -- 172.21.15.89:0/2156253365 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94800e070 con 0x7fb9500d2870 2024-04-02T07:35:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.451+0000 7fb94e000700 1 -- 172.21.15.89:0/2156253365 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94800bb20 con 0x7fb9500d2870 2024-04-02T07:35:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.451+0000 7fb94e000700 1 -- 172.21.15.89:0/2156253365 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb948012430 con 0x7fb9500d2870 2024-04-02T07:35:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.451+0000 7fb954c00700 1 -- 172.21.15.89:0/2156253365 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9500d2870 msgr2=0x7fb9500d3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.451+0000 7fb954c00700 1 --2- 172.21.15.89:0/2156253365 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9500d2870 0x7fb9500d3e60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb948009d90 tx=0x7fb94800b330 comp rx=0 tx=0).stop 2024-04-02T07:35:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.451+0000 7fb954c00700 1 -- 172.21.15.89:0/2156253365 shutdown_connections 2024-04-02T07:35:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.451+0000 7fb954c00700 1 --2- 172.21.15.89:0/2156253365 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9500d2870 0x7fb9500d3e60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.451+0000 7fb954c00700 1 -- 172.21.15.89:0/2156253365 >> 172.21.15.89:0/2156253365 conn(0x7fb9500668e0 msgr2=0x7fb950066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.452+0000 7fb954c00700 1 -- 172.21.15.89:0/2156253365 shutdown_connections 2024-04-02T07:35:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.452+0000 7fb954c00700 1 -- 172.21.15.89:0/2156253365 wait complete. 2024-04-02T07:35:32.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.454+0000 7fb954c00700 1 Processor -- start 2024-04-02T07:35:32.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.454+0000 7fb954c00700 1 -- start start 2024-04-02T07:35:32.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.455+0000 7fb954c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9500d0a00 0x7fb95015ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:32.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.455+0000 7fb954c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9500d2870 0x7fb95014dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:32.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.455+0000 7fb954c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95015d5d0 0x7fb95015dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:32.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.455+0000 7fb954c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95015faf0 con 0x7fb9500d2870 2024-04-02T07:35:32.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.455+0000 7fb954c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95015ee70 con 0x7fb95015d5d0 2024-04-02T07:35:32.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.455+0000 7fb954c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95015f150 con 0x7fb9500d0a00 2024-04-02T07:35:32.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.455+0000 7fb94ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9500d2870 0x7fb95014dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:32.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.456+0000 7fb94fe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95015d5d0 0x7fb95015dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:32.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.456+0000 7fb94ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9500d2870 0x7fb95014dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60264/0 (socket says 172.21.15.89:60264) 2024-04-02T07:35:32.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.456+0000 7fb94ea00700 1 -- 172.21.15.89:0/2707697176 learned_addr learned my addr 172.21.15.89:0/2707697176 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:32.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.456+0000 7fb94f400700 1 --2- 172.21.15.89:0/2707697176 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9500d0a00 0x7fb95015ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:32.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.456+0000 7fb94ea00700 1 -- 172.21.15.89:0/2707697176 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9500d0a00 msgr2=0x7fb95015ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:32.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.456+0000 7fb94ea00700 1 --2- 172.21.15.89:0/2707697176 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9500d0a00 0x7fb95015ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.456+0000 7fb94ea00700 1 -- 172.21.15.89:0/2707697176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95015d5d0 msgr2=0x7fb95015dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:32.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.456+0000 7fb94ea00700 1 --2- 172.21.15.89:0/2707697176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95015d5d0 0x7fb95015dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.456+0000 7fb94ea00700 1 -- 172.21.15.89:0/2707697176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb948009a70 con 0x7fb9500d2870 2024-04-02T07:35:32.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.456+0000 7fb94fe00700 1 --2- 172.21.15.89:0/2707697176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95015d5d0 0x7fb95015dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:32.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.456+0000 7fb94ea00700 1 --2- 172.21.15.89:0/2707697176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9500d2870 0x7fb95014dbf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb93c001eb0 tx=0x7fb93c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:32.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.457+0000 7fb947400700 1 -- 172.21.15.89:0/2707697176 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c005d70 con 0x7fb9500d2870 2024-04-02T07:35:32.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.457+0000 7fb947400700 1 -- 172.21.15.89:0/2707697176 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c0070d0 con 0x7fb9500d2870 2024-04-02T07:35:32.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.457+0000 7fb947400700 1 -- 172.21.15.89:0/2707697176 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c007460 con 0x7fb9500d2870 2024-04-02T07:35:32.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.457+0000 7fb954c00700 1 -- 172.21.15.89:0/2707697176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95015f430 con 0x7fb9500d2870 2024-04-02T07:35:32.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.457+0000 7fb954c00700 1 -- 172.21.15.89:0/2707697176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb950160450 con 0x7fb9500d2870 2024-04-02T07:35:32.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.457+0000 7fb947400700 1 -- 172.21.15.89:0/2707697176 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb93c012400 con 0x7fb9500d2870 2024-04-02T07:35:32.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.457+0000 7fb947400700 1 -- 172.21.15.89:0/2707697176 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb93c012d80 con 0x7fb9500d2870 2024-04-02T07:35:32.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.458+0000 7fb954c00700 1 -- 172.21.15.89:0/2707697176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95015fc30 con 0x7fb9500d2870 2024-04-02T07:35:32.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.460+0000 7fb947400700 1 -- 172.21.15.89:0/2707697176 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fb95015fc30 con 0x7fb9500d2870 2024-04-02T07:35:32.527 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:32.526+0000 7f016525e240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:35:32.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.533+0000 7fb954c00700 1 -- 172.21.15.89:0/2707697176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9500b2860 con 0x7fb9500d2870 2024-04-02T07:35:32.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.533+0000 7fb947400700 1 -- 172.21.15.89:0/2707697176 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fb93c021060 con 0x7fb9500d2870 2024-04-02T07:35:32.536 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:35:32.536 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":10,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:32.121142+0000","last_up_change":"0.000000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:35:32.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.534+0000 7fb954c00700 1 -- 172.21.15.89:0/2707697176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9500d2870 msgr2=0x7fb95014dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:32.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.534+0000 7fb954c00700 1 --2- 172.21.15.89:0/2707697176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9500d2870 0x7fb95014dbf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb93c001eb0 tx=0x7fb93c00c7b0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.535+0000 7fb954c00700 1 -- 172.21.15.89:0/2707697176 shutdown_connections 2024-04-02T07:35:32.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.535+0000 7fb954c00700 1 --2- 172.21.15.89:0/2707697176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95015d5d0 0x7fb95015dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.535+0000 7fb954c00700 1 --2- 172.21.15.89:0/2707697176 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9500d0a00 0x7fb95015ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.535+0000 7fb954c00700 1 --2- 172.21.15.89:0/2707697176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9500d2870 0x7fb95014dbf0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:32.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.535+0000 7fb954c00700 1 -- 172.21.15.89:0/2707697176 >> 172.21.15.89:0/2707697176 conn(0x7fb9500668e0 msgr2=0x7fb9500b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:32.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.535+0000 7fb954c00700 1 -- 172.21.15.89:0/2707697176 shutdown_connections 2024-04-02T07:35:32.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:32.535+0000 7fb954c00700 1 -- 172.21.15.89:0/2707697176 wait complete. 2024-04-02T07:35:32.555 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-02T07:35:32.555 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-02T07:35:32.576 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:32.574+0000 7f826b9b7240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:35:32.668 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:32.665+0000 7f826b9b7240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:35:32.728 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:32.727+0000 7f016525e240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:35:32.818 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:32.817+0000 7f016525e240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:35:32.844 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-02T07:35:32.841+0000 7f064217d680 -1 Falling back to public interface 2024-04-02T07:35:32.850 INFO:tasks.ceph.osd.5.smithi113.stderr:2024-04-02T07:35:32.849+0000 7f71f9f16680 -1 Falling back to public interface 2024-04-02T07:35:32.864 INFO:tasks.ceph.osd.4.smithi113.stderr:2024-04-02T07:35:32.863+0000 7f5a24fd5680 -1 Falling back to public interface 2024-04-02T07:35:32.874 INFO:tasks.ceph.osd.7.smithi113.stderr:2024-04-02T07:35:32.873+0000 7f915a25b680 -1 Falling back to public interface 2024-04-02T07:35:32.875 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-02T07:35:32.873+0000 7f3485087680 -1 Falling back to public interface 2024-04-02T07:35:32.890 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-02T07:35:32.888+0000 7f2f9a538680 -1 Falling back to public interface 2024-04-02T07:35:32.891 INFO:tasks.ceph.osd.6.smithi113.stderr:2024-04-02T07:35:32.890+0000 7f37a5312680 -1 Falling back to public interface 2024-04-02T07:35:32.898 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-02T07:35:32.896+0000 7f54e1668680 -1 Falling back to public interface 2024-04-02T07:35:33.361 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-02T07:35:33.481 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:33.478+0000 7f826b9b7240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:35:33.540 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.534+0000 7ffa62400700 1 Processor -- start 2024-04-02T07:35:33.540 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.534+0000 7ffa62400700 1 -- start start 2024-04-02T07:35:33.540 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.535+0000 7ffa62400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffa5c0d2ad0 0x7ffa5c0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:33.541 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.535+0000 7ffa62400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c15f030 con 0x7ffa5c06c200 2024-04-02T07:35:33.541 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.535+0000 7ffa62400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c0d3bf0 con 0x7ffa5c15ec50 2024-04-02T07:35:33.541 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.535+0000 7ffa62400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5c1615d0 con 0x7ffa5c0d2ad0 2024-04-02T07:35:33.541 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.536+0000 7ffa5be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ffa5c06c200 0x7ffa5c1619f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46224/0 (socket says 172.21.15.89:46224) 2024-04-02T07:35:33.541 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.536+0000 7ffa5be00700 1 -- 172.21.15.89:0/3130217446 learned_addr learned my addr 172.21.15.89:0/3130217446 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:33.541 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.536+0000 7ffa5aa00700 1 --2- 172.21.15.89:0/3130217446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffa5c0d2ad0 0x7ffa5c0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:33.541 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.536+0000 7ffa5a000700 1 -- 172.21.15.89:0/3130217446 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832477730 0 0) 0x7ffa5c15f030 con 0x7ffa5c06c200 2024-04-02T07:35:33.541 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.536+0000 7ffa5a000700 1 -- 172.21.15.89:0/3130217446 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa400035b0 con 0x7ffa5c06c200 2024-04-02T07:35:33.541 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.536+0000 7ffa5a000700 1 -- 172.21.15.89:0/3130217446 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98942977 0 0) 0x7ffa5c0d3bf0 con 0x7ffa5c15ec50 2024-04-02T07:35:33.541 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.536+0000 7ffa5a000700 1 -- 172.21.15.89:0/3130217446 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa5c15f030 con 0x7ffa5c15ec50 2024-04-02T07:35:33.541 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.536+0000 7ffa5aa00700 1 -- 172.21.15.89:0/3130217446 >> v1:172.21.15.113:6789/0 conn(0x7ffa5c15ec50 legacy=0x7ffa5c0ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:33.541 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.536+0000 7ffa5aa00700 1 -- 172.21.15.89:0/3130217446 >> v1:172.21.15.89:6789/0 conn(0x7ffa5c06c200 legacy=0x7ffa5c1619f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:33.541 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.536+0000 7ffa5aa00700 1 -- 172.21.15.89:0/3130217446 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5c0ce9e0 con 0x7ffa5c0d2ad0 2024-04-02T07:35:33.542 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.536+0000 7ffa5aa00700 1 --2- 172.21.15.89:0/3130217446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffa5c0d2ad0 0x7ffa5c0d2460 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009850 tx=0x7ffa48009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=118715d8313a2c11 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:33.542 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.537+0000 7ffa5a000700 1 -- 172.21.15.89:0/3130217446 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa48005e30 con 0x7ffa5c0d2ad0 2024-04-02T07:35:33.542 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.537+0000 7ffa5a000700 1 -- 172.21.15.89:0/3130217446 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa48006030 con 0x7ffa5c0d2ad0 2024-04-02T07:35:33.542 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.537+0000 7ffa5a000700 1 -- 172.21.15.89:0/3130217446 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa480063e0 con 0x7ffa5c0d2ad0 2024-04-02T07:35:33.542 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.537+0000 7ffa62400700 1 -- 172.21.15.89:0/3130217446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffa5c0d2ad0 msgr2=0x7ffa5c0d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:33.542 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.537+0000 7ffa62400700 1 --2- 172.21.15.89:0/3130217446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffa5c0d2ad0 0x7ffa5c0d2460 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009850 tx=0x7ffa48009b60 comp rx=0 tx=0).stop 2024-04-02T07:35:33.542 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.537+0000 7ffa62400700 1 -- 172.21.15.89:0/3130217446 shutdown_connections 2024-04-02T07:35:33.542 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.537+0000 7ffa62400700 1 --2- 172.21.15.89:0/3130217446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffa5c0d2ad0 0x7ffa5c0d2460 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:33.542 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.537+0000 7ffa62400700 1 -- 172.21.15.89:0/3130217446 >> 172.21.15.89:0/3130217446 conn(0x7ffa5c08aff0 msgr2=0x7ffa5c0d34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:33.542 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.538+0000 7ffa62400700 1 -- 172.21.15.89:0/3130217446 shutdown_connections 2024-04-02T07:35:33.542 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.538+0000 7ffa62400700 1 -- 172.21.15.89:0/3130217446 wait complete. 2024-04-02T07:35:33.545 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.539+0000 7ffa62400700 1 Processor -- start 2024-04-02T07:35:33.545 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.539+0000 7ffa62400700 1 -- start start 2024-04-02T07:35:33.545 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.540+0000 7ffa62400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa5c06c200 0x7ffa5c15ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:33.545 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.540+0000 7ffa62400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffa5c0d2ad0 0x7ffa5c0ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:33.545 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.540+0000 7ffa62400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa5c15ec50 0x7ffa5c0c7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:33.545 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.540+0000 7ffa62400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5c0c7a90 con 0x7ffa5c06c200 2024-04-02T07:35:33.545 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.540+0000 7ffa62400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5c0b7190 con 0x7ffa5c15ec50 2024-04-02T07:35:33.545 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.540+0000 7ffa62400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5c0b7470 con 0x7ffa5c0d2ad0 2024-04-02T07:35:33.545 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.540+0000 7ffa5b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa5c06c200 0x7ffa5c15ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:33.545 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa5b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa5c06c200 0x7ffa5c15ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60276/0 (socket says 172.21.15.89:60276) 2024-04-02T07:35:33.545 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa5b400700 1 -- 172.21.15.89:0/3250614815 learned_addr learned my addr 172.21.15.89:0/3250614815 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:33.546 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa5aa00700 1 --2- 172.21.15.89:0/3250614815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffa5c0d2ad0 0x7ffa5c0ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:33.546 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa5be00700 1 --2- 172.21.15.89:0/3250614815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa5c15ec50 0x7ffa5c0c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:33.546 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa5b400700 1 -- 172.21.15.89:0/3250614815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffa5c0d2ad0 msgr2=0x7ffa5c0ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:33.546 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa5b400700 1 --2- 172.21.15.89:0/3250614815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffa5c0d2ad0 0x7ffa5c0ca5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:33.546 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa5b400700 1 -- 172.21.15.89:0/3250614815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa5c15ec50 msgr2=0x7ffa5c0c7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:33.546 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa5b400700 1 --2- 172.21.15.89:0/3250614815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa5c15ec50 0x7ffa5c0c7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:33.546 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa5b400700 1 -- 172.21.15.89:0/3250614815 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa48009430 con 0x7ffa5c06c200 2024-04-02T07:35:33.546 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa5be00700 1 --2- 172.21.15.89:0/3250614815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa5c15ec50 0x7ffa5c0c7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:35:33.546 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa5b400700 1 --2- 172.21.15.89:0/3250614815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa5c06c200 0x7ffa5c15ab60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c00c500 tx=0x7ffa4c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:33.546 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa53400700 1 -- 172.21.15.89:0/3250614815 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c0052d0 con 0x7ffa5c06c200 2024-04-02T07:35:33.546 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa53400700 1 -- 172.21.15.89:0/3250614815 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4c005450 con 0x7ffa5c06c200 2024-04-02T07:35:33.546 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa53400700 1 -- 172.21.15.89:0/3250614815 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c0038c0 con 0x7ffa5c06c200 2024-04-02T07:35:33.546 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.541+0000 7ffa62400700 1 -- 172.21.15.89:0/3250614815 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa5c0b7750 con 0x7ffa5c06c200 2024-04-02T07:35:33.547 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.542+0000 7ffa62400700 1 -- 172.21.15.89:0/3250614815 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa5c15b0a0 con 0x7ffa5c06c200 2024-04-02T07:35:33.547 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.542+0000 7ffa53400700 1 -- 172.21.15.89:0/3250614815 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ffa4c003d20 con 0x7ffa5c06c200 2024-04-02T07:35:33.547 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.542+0000 7ffa53400700 1 -- 172.21.15.89:0/3250614815 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7ffa4c011e00 con 0x7ffa5c06c200 2024-04-02T07:35:33.547 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.543+0000 7ffa62400700 1 -- 172.21.15.89:0/3250614815 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa5c160430 con 0x7ffa5c06c200 2024-04-02T07:35:33.547 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.545+0000 7ffa53400700 1 -- 172.21.15.89:0/3250614815 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7ffa4c017070 con 0x7ffa5c06c200 2024-04-02T07:35:33.632 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.629+0000 7ffa62400700 1 -- 172.21.15.89:0/3250614815 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa5c0b7940 con 0x7ffa5c06c200 2024-04-02T07:35:33.633 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.630+0000 7ffa53400700 1 -- 172.21.15.89:0/3250614815 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7ffa4c011590 con 0x7ffa5c06c200 2024-04-02T07:35:33.633 INFO:teuthology.misc.health.smithi089.stdout: 2024-04-02T07:35:33.633 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":10,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:32.121142+0000","last_up_change":"0.000000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:35:33.634 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.631+0000 7ffa62400700 1 -- 172.21.15.89:0/3250614815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa5c06c200 msgr2=0x7ffa5c15ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:33.634 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.631+0000 7ffa62400700 1 --2- 172.21.15.89:0/3250614815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa5c06c200 0x7ffa5c15ab60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c00c500 tx=0x7ffa4c00c810 comp rx=0 tx=0).stop 2024-04-02T07:35:33.634 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.631+0000 7ffa62400700 1 -- 172.21.15.89:0/3250614815 shutdown_connections 2024-04-02T07:35:33.634 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.631+0000 7ffa62400700 1 --2- 172.21.15.89:0/3250614815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa5c15ec50 0x7ffa5c0c7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:33.634 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.631+0000 7ffa62400700 1 --2- 172.21.15.89:0/3250614815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa5c06c200 0x7ffa5c15ab60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:33.635 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.631+0000 7ffa62400700 1 --2- 172.21.15.89:0/3250614815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffa5c0d2ad0 0x7ffa5c0ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:33.635 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.631+0000 7ffa62400700 1 -- 172.21.15.89:0/3250614815 >> 172.21.15.89:0/3250614815 conn(0x7ffa5c08aff0 msgr2=0x7ffa5c15cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:33.635 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.631+0000 7ffa62400700 1 -- 172.21.15.89:0/3250614815 shutdown_connections 2024-04-02T07:35:33.635 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:33.631+0000 7ffa62400700 1 -- 172.21.15.89:0/3250614815 wait complete. 2024-04-02T07:35:33.642 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:33.641+0000 7f016525e240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:35:33.649 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-02T07:35:33.704 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:33.701+0000 7f826b9b7240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:35:33.851 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:33.849+0000 7f016525e240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:35:33.905 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:33.902+0000 7f826b9b7240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:35:34.053 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:34.052+0000 7f016525e240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:35:34.397 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:34.394+0000 7f826b9b7240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:35:34.545 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:34.544+0000 7f016525e240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:35:34.601 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:34.599+0000 7f826b9b7240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:35:34.749 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:34.748+0000 7f016525e240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:35:35.039 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:35.036+0000 7f826b9b7240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:35:35.166 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:35.163+0000 7f826b9b7240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:35:35.194 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:35.192+0000 7f016525e240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:35:35.331 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:35:35.329+0000 7f016525e240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:35:36.190 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:36.184+0000 7f8222000700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-02T07:35:36.190 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:35:36.184+0000 7f8222000700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-02T07:35:36.345 INFO:tasks.ceph.osd.5.smithi113.stderr:2024-04-02T07:35:36.343+0000 7f71f9f16680 -1 osd.5 0 log_to_monitors true 2024-04-02T07:35:36.352 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-02T07:35:36.349+0000 7f064217d680 -1 osd.0 0 log_to_monitors true 2024-04-02T07:35:36.395 INFO:tasks.ceph.osd.4.smithi113.stderr:2024-04-02T07:35:36.393+0000 7f5a24fd5680 -1 osd.4 0 log_to_monitors true 2024-04-02T07:35:36.423 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-02T07:35:36.421+0000 7f3485087680 -1 osd.1 0 log_to_monitors true 2024-04-02T07:35:36.450 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-02T07:35:36.448+0000 7f54e1668680 -1 osd.2 0 log_to_monitors true 2024-04-02T07:35:36.475 INFO:tasks.ceph.osd.7.smithi113.stderr:2024-04-02T07:35:36.473+0000 7f915a25b680 -1 osd.7 0 log_to_monitors true 2024-04-02T07:35:36.488 INFO:tasks.ceph.osd.6.smithi113.stderr:2024-04-02T07:35:36.486+0000 7f37a5312680 -1 osd.6 0 log_to_monitors true 2024-04-02T07:35:36.507 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-02T07:35:36.504+0000 7f2f9a538680 -1 osd.3 0 log_to_monitors true 2024-04-02T07:35:39.937 INFO:tasks.ceph.osd.7.smithi113.stderr:2024-04-02T07:35:39.934+0000 7f9147600700 -1 osd.7 0 waiting for initial osdmap 2024-04-02T07:35:39.945 INFO:tasks.ceph.osd.7.smithi113.stderr:2024-04-02T07:35:39.942+0000 7f913d800700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T07:35:40.010 INFO:tasks.ceph.osd.5.smithi113.stderr:2024-04-02T07:35:40.008+0000 7f71e7200700 -1 osd.5 0 waiting for initial osdmap 2024-04-02T07:35:40.020 INFO:tasks.ceph.osd.5.smithi113.stderr:2024-04-02T07:35:40.018+0000 7f71dd400700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T07:35:40.035 INFO:tasks.ceph.osd.4.smithi113.stderr:2024-04-02T07:35:40.033+0000 7f5a12200700 -1 osd.4 0 waiting for initial osdmap 2024-04-02T07:35:40.042 INFO:tasks.ceph.osd.4.smithi113.stderr:2024-04-02T07:35:40.040+0000 7f5a08400700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T07:35:40.043 INFO:tasks.ceph.osd.6.smithi113.stderr:2024-04-02T07:35:40.041+0000 7f3792600700 -1 osd.6 0 waiting for initial osdmap 2024-04-02T07:35:40.052 INFO:tasks.ceph.osd.6.smithi113.stderr:2024-04-02T07:35:40.050+0000 7f3788800700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T07:35:40.455 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-02T07:35:40.584 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-02T07:35:40.581+0000 7f2f87800700 -1 osd.3 0 waiting for initial osdmap 2024-04-02T07:35:40.590 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-02T07:35:40.589+0000 7f2f7da00700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T07:35:40.612 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-02T07:35:40.610+0000 7f3472400700 -1 osd.1 0 waiting for initial osdmap 2024-04-02T07:35:40.620 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-02T07:35:40.619+0000 7f062f400700 -1 osd.0 0 waiting for initial osdmap 2024-04-02T07:35:40.624 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-02T07:35:40.623+0000 7f3468600700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T07:35:40.637 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-02T07:35:40.636+0000 7f0625600700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T07:35:40.647 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-02T07:35:40.646+0000 7f54cea00700 -1 osd.2 0 waiting for initial osdmap 2024-04-02T07:35:40.664 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-02T07:35:40.663+0000 7f54c4c00700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T07:35:40.700 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.699+0000 7fb5a0800700 1 Processor -- start 2024-04-02T07:35:40.700 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.699+0000 7fb5a0800700 1 -- start start 2024-04-02T07:35:40.703 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.701+0000 7fb5a0800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb59815d9e0 0x7fb5980cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:40.703 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.701+0000 7fb5a0800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5980d0910 con 0x7fb59806eed0 2024-04-02T07:35:40.703 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.701+0000 7fb5a0800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59809a250 con 0x7fb59806f520 2024-04-02T07:35:40.703 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.701+0000 7fb5a0800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb59809adf0 con 0x7fb59815d9e0 2024-04-02T07:35:40.703 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.701+0000 7fb59de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb59815d9e0 0x7fb5980cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:40.703 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.701+0000 7fb59de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb59815d9e0 0x7fb5980cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53622/0 (socket says 172.21.15.89:53622) 2024-04-02T07:35:40.703 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.701+0000 7fb59de00700 1 -- 172.21.15.89:0/2921011254 learned_addr learned my addr 172.21.15.89:0/2921011254 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:40.703 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.701+0000 7fb59d400700 1 --1- 172.21.15.89:0/2921011254 >> v1:172.21.15.89:6789/0 conn(0x7fb59806eed0 0x7fb59809a6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43984/0 (socket says 172.21.15.89:43984) 2024-04-02T07:35:40.704 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.701+0000 7fb59de00700 1 -- 172.21.15.89:0/2921011254 >> v1:172.21.15.113:6789/0 conn(0x7fb59806f520 legacy=0x7fb5980cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:40.704 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.701+0000 7fb59de00700 1 -- 172.21.15.89:0/2921011254 >> v1:172.21.15.89:6789/0 conn(0x7fb59806eed0 legacy=0x7fb59809a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:40.704 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.701+0000 7fb59de00700 1 -- 172.21.15.89:0/2921011254 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5980d0630 con 0x7fb59815d9e0 2024-04-02T07:35:40.704 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.701+0000 7fb59ca00700 1 -- 172.21.15.89:0/2921011254 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854985480 0 0) 0x7fb5980d0910 con 0x7fb59806eed0 2024-04-02T07:35:40.704 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.702+0000 7fb59de00700 1 --2- 172.21.15.89:0/2921011254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb59815d9e0 0x7fb5980cffe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb58c0098f0 tx=0x7fb58c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aea1a45531de852 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:40.705 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.702+0000 7fb59ca00700 1 -- 172.21.15.89:0/2921011254 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c00b8a0 con 0x7fb59815d9e0 2024-04-02T07:35:40.705 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.702+0000 7fb59ca00700 1 -- 172.21.15.89:0/2921011254 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58c00ba00 con 0x7fb59815d9e0 2024-04-02T07:35:40.705 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.702+0000 7fb59ca00700 1 -- 172.21.15.89:0/2921011254 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c00bd90 con 0x7fb59815d9e0 2024-04-02T07:35:40.705 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.702+0000 7fb5a0800700 1 -- 172.21.15.89:0/2921011254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb59815d9e0 msgr2=0x7fb5980cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:40.705 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.702+0000 7fb5a0800700 1 --2- 172.21.15.89:0/2921011254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb59815d9e0 0x7fb5980cffe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb58c0098f0 tx=0x7fb58c009c00 comp rx=0 tx=0).stop 2024-04-02T07:35:40.705 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.702+0000 7fb5a0800700 1 -- 172.21.15.89:0/2921011254 shutdown_connections 2024-04-02T07:35:40.705 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.702+0000 7fb5a0800700 1 --2- 172.21.15.89:0/2921011254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb59815d9e0 0x7fb5980cffe0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:40.705 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.703+0000 7fb5a0800700 1 -- 172.21.15.89:0/2921011254 >> 172.21.15.89:0/2921011254 conn(0x7fb5980d1360 msgr2=0x7fb5980d1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:40.706 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.703+0000 7fb5a0800700 1 -- 172.21.15.89:0/2921011254 shutdown_connections 2024-04-02T07:35:40.706 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.703+0000 7fb5a0800700 1 -- 172.21.15.89:0/2921011254 wait complete. 2024-04-02T07:35:40.706 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.704+0000 7fb5a0800700 1 Processor -- start 2024-04-02T07:35:40.706 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.704+0000 7fb5a0800700 1 -- start start 2024-04-02T07:35:40.708 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.705+0000 7fb5a0800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb59806eed0 0x7fb598156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:40.708 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.705+0000 7fb5a0800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb59806f520 0x7fb5980c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:40.708 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.705+0000 7fb5a0800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb59815d9e0 0x7fb5980af220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:40.708 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.705+0000 7fb5a0800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5980c9540 con 0x7fb59815d9e0 2024-04-02T07:35:40.708 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.705+0000 7fb5a0800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5980c8930 con 0x7fb59806f520 2024-04-02T07:35:40.708 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.705+0000 7fb5a0800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5980c8c10 con 0x7fb59806eed0 2024-04-02T07:35:40.709 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.706+0000 7fb59e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb59815d9e0 0x7fb5980af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:40.709 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.706+0000 7fb59e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb59815d9e0 0x7fb5980af220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53628/0 (socket says 172.21.15.89:53628) 2024-04-02T07:35:40.709 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.706+0000 7fb59e800700 1 -- 172.21.15.89:0/1388829588 learned_addr learned my addr 172.21.15.89:0/1388829588 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:40.709 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.706+0000 7fb59d400700 1 --2- 172.21.15.89:0/1388829588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb59806f520 0x7fb5980c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:40.709 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.706+0000 7fb59de00700 1 --2- 172.21.15.89:0/1388829588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb59806eed0 0x7fb598156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:40.709 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.706+0000 7fb59e800700 1 -- 172.21.15.89:0/1388829588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb59806eed0 msgr2=0x7fb598156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:40.709 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.706+0000 7fb59e800700 1 --2- 172.21.15.89:0/1388829588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb59806eed0 0x7fb598156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:40.709 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.706+0000 7fb59e800700 1 -- 172.21.15.89:0/1388829588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb59806f520 msgr2=0x7fb5980c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:40.709 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.706+0000 7fb59e800700 1 --2- 172.21.15.89:0/1388829588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb59806f520 0x7fb5980c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:40.709 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.706+0000 7fb59e800700 1 -- 172.21.15.89:0/1388829588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58c009550 con 0x7fb59815d9e0 2024-04-02T07:35:40.709 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.706+0000 7fb59e800700 1 --2- 172.21.15.89:0/1388829588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb59815d9e0 0x7fb5980af220 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb59400cc50 tx=0x7fb59400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:40.709 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.706+0000 7fb592000700 1 -- 172.21.15.89:0/1388829588 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb594003c00 con 0x7fb59815d9e0 2024-04-02T07:35:40.709 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.706+0000 7fb592000700 1 -- 172.21.15.89:0/1388829588 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb594003d60 con 0x7fb59815d9e0 2024-04-02T07:35:40.709 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.706+0000 7fb592000700 1 -- 172.21.15.89:0/1388829588 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59400aab0 con 0x7fb59815d9e0 2024-04-02T07:35:40.710 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.707+0000 7fb5a0800700 1 -- 172.21.15.89:0/1388829588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5980c8ef0 con 0x7fb59815d9e0 2024-04-02T07:35:40.710 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.707+0000 7fb5a0800700 1 -- 172.21.15.89:0/1388829588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb598156e70 con 0x7fb59815d9e0 2024-04-02T07:35:40.710 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.708+0000 7fb5a0800700 1 -- 172.21.15.89:0/1388829588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb598157030 con 0x7fb59815d9e0 2024-04-02T07:35:40.710 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.708+0000 7fb592000700 1 -- 172.21.15.89:0/1388829588 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7fb59400ac10 con 0x7fb59815d9e0 2024-04-02T07:35:40.711 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.709+0000 7fb592000700 1 --2- 172.21.15.89:0/1388829588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb57c06ac00 0x7fb57c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:40.711 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.709+0000 7fb592000700 1 -- 172.21.15.89:0/1388829588 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fb5940884f0 con 0x7fb59815d9e0 2024-04-02T07:35:40.711 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.710+0000 7fb59de00700 1 --2- 172.21.15.89:0/1388829588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb57c06ac00 0x7fb57c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:40.712 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.710+0000 7fb59de00700 1 --2- 172.21.15.89:0/1388829588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb57c06ac00 0x7fb57c06d0c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb58c000c00 tx=0x7fb58c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:40.716 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.714+0000 7fb592000700 1 -- 172.21.15.89:0/1388829588 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb5940544a0 con 0x7fb59815d9e0 2024-04-02T07:35:40.919 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.917+0000 7fb5a0800700 1 -- 172.21.15.89:0/1388829588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb598067040 con 0x7fb59815d9e0 2024-04-02T07:35:40.919 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.918+0000 7fb592000700 1 -- 172.21.15.89:0/1388829588 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fb594057ac0 con 0x7fb59815d9e0 2024-04-02T07:35:40.920 INFO:teuthology.misc.health.smithi089.stdout: 2024-04-02T07:35:40.920 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":12,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:38.236808+0000","last_up_change":"0.000000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:35:40.922 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.920+0000 7fb5a0800700 1 -- 172.21.15.89:0/1388829588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb57c06ac00 msgr2=0x7fb57c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:40.922 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.920+0000 7fb5a0800700 1 --2- 172.21.15.89:0/1388829588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb57c06ac00 0x7fb57c06d0c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb58c000c00 tx=0x7fb58c008030 comp rx=0 tx=0).stop 2024-04-02T07:35:40.922 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.920+0000 7fb5a0800700 1 -- 172.21.15.89:0/1388829588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb59815d9e0 msgr2=0x7fb5980af220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:40.923 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.920+0000 7fb5a0800700 1 --2- 172.21.15.89:0/1388829588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb59815d9e0 0x7fb5980af220 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb59400cc50 tx=0x7fb59400cf60 comp rx=0 tx=0).stop 2024-04-02T07:35:40.923 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.920+0000 7fb5a0800700 1 -- 172.21.15.89:0/1388829588 shutdown_connections 2024-04-02T07:35:40.923 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.920+0000 7fb5a0800700 1 --2- 172.21.15.89:0/1388829588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb57c06ac00 0x7fb57c06d0c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:40.923 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.920+0000 7fb5a0800700 1 --2- 172.21.15.89:0/1388829588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb59815d9e0 0x7fb5980af220 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:40.923 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.920+0000 7fb5a0800700 1 --2- 172.21.15.89:0/1388829588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb59806eed0 0x7fb598156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:40.923 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.920+0000 7fb5a0800700 1 --2- 172.21.15.89:0/1388829588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb59806f520 0x7fb5980c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:40.923 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.920+0000 7fb5a0800700 1 -- 172.21.15.89:0/1388829588 >> 172.21.15.89:0/1388829588 conn(0x7fb5980d1360 msgr2=0x7fb5980b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:40.923 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.921+0000 7fb5a0800700 1 -- 172.21.15.89:0/1388829588 shutdown_connections 2024-04-02T07:35:40.923 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:40.921+0000 7fb5a0800700 1 -- 172.21.15.89:0/1388829588 wait complete. 2024-04-02T07:35:40.937 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-02T07:35:41.228 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:35:41.232 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:35:47.742 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-02T07:35:47.923 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.921+0000 7fa84e400700 1 Processor -- start 2024-04-02T07:35:47.923 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.921+0000 7fa84e400700 1 -- start start 2024-04-02T07:35:47.924 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.922+0000 7fa84e400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa84815e9f0 0x7fa8480b4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:47.924 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.922+0000 7fa84e400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa84815edd0 0x7fa8480b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:47.924 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.922+0000 7fa84e400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8480b4140 con 0x7fa8480cef80 2024-04-02T07:35:47.924 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.922+0000 7fa84e400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8480cdef0 con 0x7fa84815edd0 2024-04-02T07:35:47.924 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.922+0000 7fa84e400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8480ce1d0 con 0x7fa84815e9f0 2024-04-02T07:35:47.925 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.923+0000 7fa847e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa8480cef80 0x7fa848161600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43994/0 (socket says 172.21.15.89:43994) 2024-04-02T07:35:47.925 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.923+0000 7fa847e00700 1 -- 172.21.15.89:0/3219572559 learned_addr learned my addr 172.21.15.89:0/3219572559 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:47.925 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.923+0000 7fa846a00700 1 --2- 172.21.15.89:0/3219572559 >> v2:172.21.15.113:3300/0 conn(0x7fa84815edd0 0x7fa8480b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:47.925 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.923+0000 7fa847400700 1 --2- 172.21.15.89:0/3219572559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa84815e9f0 0x7fa8480b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:47.927 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.923+0000 7fa846000700 1 -- 172.21.15.89:0/3219572559 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597010170 0 0) 0x7fa8480b4140 con 0x7fa8480cef80 2024-04-02T07:35:47.927 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.923+0000 7fa846000700 1 -- 172.21.15.89:0/3219572559 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa82c0035b0 con 0x7fa8480cef80 2024-04-02T07:35:47.927 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.923+0000 7fa846a00700 1 -- 172.21.15.89:0/3219572559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa84815e9f0 msgr2=0x7fa8480b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:47.927 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.923+0000 7fa846a00700 1 --2- 172.21.15.89:0/3219572559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa84815e9f0 0x7fa8480b4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:47.927 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.923+0000 7fa846a00700 1 -- 172.21.15.89:0/3219572559 >> v1:172.21.15.89:6789/0 conn(0x7fa8480cef80 legacy=0x7fa848161600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:47.927 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.923+0000 7fa846a00700 1 -- 172.21.15.89:0/3219572559 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8480d17f0 con 0x7fa84815edd0 2024-04-02T07:35:47.927 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.923+0000 7fa847400700 1 --2- 172.21.15.89:0/3219572559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa84815e9f0 0x7fa8480b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:47.928 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.923+0000 7fa846a00700 1 --2- 172.21.15.89:0/3219572559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa84815edd0 0x7fa8480b4f60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa834009840 tx=0x7fa834009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13b2a2d51af6666d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:47.928 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.924+0000 7fa846000700 1 -- 172.21.15.89:0/3219572559 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834011830 con 0x7fa84815edd0 2024-04-02T07:35:47.928 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.924+0000 7fa846000700 1 -- 172.21.15.89:0/3219572559 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa834011990 con 0x7fa84815edd0 2024-04-02T07:35:47.928 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.924+0000 7fa846000700 1 -- 172.21.15.89:0/3219572559 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834011d20 con 0x7fa84815edd0 2024-04-02T07:35:47.928 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.924+0000 7fa84e400700 1 -- 172.21.15.89:0/3219572559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa84815edd0 msgr2=0x7fa8480b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:47.928 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.924+0000 7fa84e400700 1 --2- 172.21.15.89:0/3219572559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa84815edd0 0x7fa8480b4f60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa834009840 tx=0x7fa834009b50 comp rx=0 tx=0).stop 2024-04-02T07:35:47.928 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.924+0000 7fa84e400700 1 -- 172.21.15.89:0/3219572559 shutdown_connections 2024-04-02T07:35:47.928 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.924+0000 7fa84e400700 1 --2- 172.21.15.89:0/3219572559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa84815e9f0 0x7fa8480b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:47.928 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.924+0000 7fa84e400700 1 --2- 172.21.15.89:0/3219572559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa84815edd0 0x7fa8480b4f60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:47.928 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.924+0000 7fa84e400700 1 -- 172.21.15.89:0/3219572559 >> 172.21.15.89:0/3219572559 conn(0x7fa84808aff0 msgr2=0x7fa8480d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:47.928 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.924+0000 7fa84e400700 1 -- 172.21.15.89:0/3219572559 shutdown_connections 2024-04-02T07:35:47.928 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.924+0000 7fa84e400700 1 -- 172.21.15.89:0/3219572559 wait complete. 2024-04-02T07:35:47.931 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.926+0000 7fa84e400700 1 Processor -- start 2024-04-02T07:35:47.931 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.926+0000 7fa84e400700 1 -- start start 2024-04-02T07:35:47.931 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.927+0000 7fa84e400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8480cef80 0x7fa8481f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:47.931 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.927+0000 7fa84e400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa84815e9f0 0x7fa8480cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:47.931 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.927+0000 7fa84e400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa84815edd0 0x7fa8480cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:47.931 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.927+0000 7fa84e400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8481f4060 con 0x7fa8480cef80 2024-04-02T07:35:47.931 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.927+0000 7fa84e400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8480b0200 con 0x7fa84815edd0 2024-04-02T07:35:47.931 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.927+0000 7fa84e400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8480b04e0 con 0x7fa84815e9f0 2024-04-02T07:35:47.931 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.927+0000 7fa847400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8480cef80 0x7fa8481f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:47.931 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.927+0000 7fa847400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8480cef80 0x7fa8481f3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53686/0 (socket says 172.21.15.89:53686) 2024-04-02T07:35:47.931 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.927+0000 7fa847400700 1 -- 172.21.15.89:0/3235542020 learned_addr learned my addr 172.21.15.89:0/3235542020 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:47.932 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.927+0000 7fa846a00700 1 --2- 172.21.15.89:0/3235542020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa84815e9f0 0x7fa8480cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:47.932 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.927+0000 7fa847e00700 1 --2- 172.21.15.89:0/3235542020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa84815edd0 0x7fa8480cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:47.932 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.928+0000 7fa847400700 1 -- 172.21.15.89:0/3235542020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa84815e9f0 msgr2=0x7fa8480cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:47.932 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.928+0000 7fa847400700 1 --2- 172.21.15.89:0/3235542020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa84815e9f0 0x7fa8480cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:47.932 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.928+0000 7fa847400700 1 -- 172.21.15.89:0/3235542020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa84815edd0 msgr2=0x7fa8480cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:47.932 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.928+0000 7fa847400700 1 --2- 172.21.15.89:0/3235542020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa84815edd0 0x7fa8480cc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:47.932 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.928+0000 7fa847400700 1 -- 172.21.15.89:0/3235542020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8380094a0 con 0x7fa8480cef80 2024-04-02T07:35:47.932 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.928+0000 7fa847400700 1 --2- 172.21.15.89:0/3235542020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8480cef80 0x7fa8481f3b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa83800ebe0 tx=0x7fa83800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:47.932 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.928+0000 7fa83f400700 1 -- 172.21.15.89:0/3235542020 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838009760 con 0x7fa8480cef80 2024-04-02T07:35:47.932 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.928+0000 7fa83f400700 1 -- 172.21.15.89:0/3235542020 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83800c850 con 0x7fa8480cef80 2024-04-02T07:35:47.932 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.928+0000 7fa84e400700 1 -- 172.21.15.89:0/3235542020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8340094a0 con 0x7fa8480cef80 2024-04-02T07:35:47.932 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.928+0000 7fa83f400700 1 -- 172.21.15.89:0/3235542020 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83800cbe0 con 0x7fa8480cef80 2024-04-02T07:35:47.932 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.929+0000 7fa84e400700 1 -- 172.21.15.89:0/3235542020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8480b0aa0 con 0x7fa8480cef80 2024-04-02T07:35:47.933 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.930+0000 7fa84e400700 1 -- 172.21.15.89:0/3235542020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8480d4420 con 0x7fa8480cef80 2024-04-02T07:35:47.933 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.930+0000 7fa83f400700 1 -- 172.21.15.89:0/3235542020 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa83800cd40 con 0x7fa8480cef80 2024-04-02T07:35:47.934 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.931+0000 7fa83f400700 1 --2- 172.21.15.89:0/3235542020 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa82806ac90 0x7fa82806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:47.934 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.931+0000 7fa83f400700 1 -- 172.21.15.89:0/3235542020 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa838088fe0 con 0x7fa8480cef80 2024-04-02T07:35:47.934 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.931+0000 7fa846a00700 1 --2- 172.21.15.89:0/3235542020 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa82806ac90 0x7fa82806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:47.934 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.932+0000 7fa846a00700 1 --2- 172.21.15.89:0/3235542020 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa82806ac90 0x7fa82806d150 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa834000c00 tx=0x7fa834008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:47.938 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:47.936+0000 7fa83f400700 1 -- 172.21.15.89:0/3235542020 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa838054500 con 0x7fa8480cef80 2024-04-02T07:35:48.141 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:48.139+0000 7fa84e400700 1 -- 172.21.15.89:0/3235542020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa84804ea80 con 0x7fa8480cef80 2024-04-02T07:35:48.142 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:48.140+0000 7fa83f400700 1 -- 172.21.15.89:0/3235542020 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12382 (secure 0 0 0) 0x7fa838057b20 con 0x7fa8480cef80 2024-04-02T07:35:48.142 INFO:teuthology.misc.health.smithi089.stdout: 2024-04-02T07:35:48.142 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":16,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:43.948311+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:35:48.144 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:48.142+0000 7fa84e400700 1 -- 172.21.15.89:0/3235542020 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa82806ac90 msgr2=0x7fa82806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:48.144 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:48.142+0000 7fa84e400700 1 --2- 172.21.15.89:0/3235542020 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa82806ac90 0x7fa82806d150 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa834000c00 tx=0x7fa834008030 comp rx=0 tx=0).stop 2024-04-02T07:35:48.144 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:48.142+0000 7fa84e400700 1 -- 172.21.15.89:0/3235542020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8480cef80 msgr2=0x7fa8481f3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:48.144 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:48.142+0000 7fa84e400700 1 --2- 172.21.15.89:0/3235542020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8480cef80 0x7fa8481f3b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa83800ebe0 tx=0x7fa83800eef0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.145 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:48.142+0000 7fa84e400700 1 -- 172.21.15.89:0/3235542020 shutdown_connections 2024-04-02T07:35:48.145 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:48.142+0000 7fa84e400700 1 --2- 172.21.15.89:0/3235542020 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa82806ac90 0x7fa82806d150 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.145 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:48.142+0000 7fa84e400700 1 --2- 172.21.15.89:0/3235542020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa84815edd0 0x7fa8480cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.145 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:48.142+0000 7fa84e400700 1 --2- 172.21.15.89:0/3235542020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8480cef80 0x7fa8481f3b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.145 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:48.142+0000 7fa84e400700 1 --2- 172.21.15.89:0/3235542020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa84815e9f0 0x7fa8480cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.145 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:48.142+0000 7fa84e400700 1 -- 172.21.15.89:0/3235542020 >> 172.21.15.89:0/3235542020 conn(0x7fa84808aff0 msgr2=0x7fa84806c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:48.145 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:48.142+0000 7fa84e400700 1 -- 172.21.15.89:0/3235542020 shutdown_connections 2024-04-02T07:35:48.145 INFO:teuthology.misc.health.smithi089.stderr:2024-04-02T07:35:48.142+0000 7fa84e400700 1 -- 172.21.15.89:0/3235542020 wait complete. 2024-04-02T07:35:48.159 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-02T07:35:48.160 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-02T07:35:48.160 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T07:35:48.160 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T07:35:48.164 INFO:tasks.ceph.mds.a:Started 2024-04-02T07:35:48.164 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T07:35:48.164 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T07:35:48.168 INFO:tasks.ceph.mds.c:Started 2024-04-02T07:35:48.168 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T07:35:48.168 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T07:35:48.172 INFO:tasks.ceph.mds.b:Started 2024-04-02T07:35:48.172 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T07:35:48.173 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T07:35:48.176 INFO:tasks.ceph.mds.d:Started 2024-04-02T07:35:48.176 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-02T07:35:48.177 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:35:48.309 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T07:35:48.309 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T07:35:48.323 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T07:35:48.323 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T07:35:48.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.377+0000 7f4bbf000700 1 Processor -- start 2024-04-02T07:35:48.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.377+0000 7f4bbf000700 1 -- start start 2024-04-02T07:35:48.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.378+0000 7f4bbf000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bb80d18b0 0x7f4bb80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:48.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.378+0000 7f4bbf000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb80ceda0 con 0x7f4bb8161590 2024-04-02T07:35:48.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bbf000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb81612f0 con 0x7f4bb8160f10 2024-04-02T07:35:48.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bbf000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb80d1d30 con 0x7f4bb80d18b0 2024-04-02T07:35:48.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bbd000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4bb8161590 0x7f4bb80b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44028/0 (socket says 172.21.15.89:44028) 2024-04-02T07:35:48.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bbd000700 1 -- 172.21.15.89:0/2991223677 learned_addr learned my addr 172.21.15.89:0/2991223677 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:48.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bb7400700 1 --2- 172.21.15.89:0/2991223677 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bb80d18b0 0x7f4bb80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:48.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238746364 0 0) 0x7f4bb80ceda0 con 0x7f4bb8161590 2024-04-02T07:35:48.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b9c0035b0 con 0x7f4bb8161590 2024-04-02T07:35:48.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706801424 0 0) 0x7f4bb81612f0 con 0x7f4bb8160f10 2024-04-02T07:35:48.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb80ceda0 con 0x7f4bb8160f10 2024-04-02T07:35:48.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2557176655 0 0) 0x7f4b9c0035b0 con 0x7f4bb8161590 2024-04-02T07:35:48.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bb80d18b0 msgr2=0x7f4bb80cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:48.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bb6a00700 1 --2- 172.21.15.89:0/2991223677 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bb80d18b0 0x7f4bb80cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 >> v1:172.21.15.113:6789/0 conn(0x7f4bb8160f10 legacy=0x7f4bb80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:48.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb80ce470 con 0x7f4bb8161590 2024-04-02T07:35:48.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.379+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f4bb00019f0 con 0x7f4bb8161590 2024-04-02T07:35:48.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 >> v1:172.21.15.89:6789/0 conn(0x7f4bb8161590 legacy=0x7f4bb80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:48.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bb6a00700 1 --2- 172.21.15.89:0/2991223677 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b9c003bc0 0x7f4b9c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:48.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bb6a00700 1 --2- 172.21.15.89:0/2991223677 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bb80d18b0 0x7f4b9c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:48.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bb6a00700 1 --2- 172.21.15.89:0/2991223677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9c004d90 0x7f4b9c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:48.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb80d1d30 con 0x7f4b9c004d90 2024-04-02T07:35:48.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c00bba0 con 0x7f4b9c003bc0 2024-04-02T07:35:48.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c00bd30 con 0x7f4bb80d18b0 2024-04-02T07:35:48.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bbd000700 1 --2- 172.21.15.89:0/2991223677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9c004d90 0x7f4b9c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:48.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bb7400700 1 --2- 172.21.15.89:0/2991223677 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bb80d18b0 0x7f4b9c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:48.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bb7e00700 1 --2- 172.21.15.89:0/2991223677 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b9c003bc0 0x7f4b9c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:48.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bb7400700 1 -- 172.21.15.89:0/2991223677 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b9c003bc0 msgr2=0x7f4b9c003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:48.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bb7400700 1 --2- 172.21.15.89:0/2991223677 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b9c003bc0 0x7f4b9c003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bb7400700 1 -- 172.21.15.89:0/2991223677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9c004d90 msgr2=0x7f4b9c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:48.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bb7400700 1 --2- 172.21.15.89:0/2991223677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9c004d90 0x7f4b9c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.380+0000 7f4bb7400700 1 -- 172.21.15.89:0/2991223677 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9c0035b0 con 0x7f4bb80d18b0 2024-04-02T07:35:48.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.381+0000 7f4bb7400700 1 --2- 172.21.15.89:0/2991223677 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bb80d18b0 0x7f4b9c004850 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009b90 tx=0x7f4b9c005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c9631733eb8d546 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:48.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.381+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba400e6c0 con 0x7f4bb80d18b0 2024-04-02T07:35:48.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.381+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba400e820 con 0x7f4bb80d18b0 2024-04-02T07:35:48.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.381+0000 7f4bb6a00700 1 -- 172.21.15.89:0/2991223677 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba400eb90 con 0x7f4bb80d18b0 2024-04-02T07:35:48.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.381+0000 7f4bbf000700 1 -- 172.21.15.89:0/2991223677 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bb80d18b0 msgr2=0x7f4b9c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:48.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.381+0000 7f4bbf000700 1 --2- 172.21.15.89:0/2991223677 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bb80d18b0 0x7f4b9c004850 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009b90 tx=0x7f4b9c005880 comp rx=0 tx=0).stop 2024-04-02T07:35:48.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.381+0000 7f4bbd000700 1 -- 172.21.15.89:0/2991223677 reap_dead start 2024-04-02T07:35:48.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.381+0000 7f4bbf000700 1 -- 172.21.15.89:0/2991223677 shutdown_connections 2024-04-02T07:35:48.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.381+0000 7f4bbf000700 1 -- 172.21.15.89:0/2991223677 >> 172.21.15.89:0/2991223677 conn(0x7f4bb808aff0 msgr2=0x7f4bb80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:48.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.381+0000 7f4bbf000700 1 -- 172.21.15.89:0/2991223677 shutdown_connections 2024-04-02T07:35:48.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.382+0000 7f4bbf000700 1 -- 172.21.15.89:0/2991223677 wait complete. 2024-04-02T07:35:48.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.384+0000 7f4bbf000700 1 Processor -- start 2024-04-02T07:35:48.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.384+0000 7f4bbf000700 1 -- start start 2024-04-02T07:35:48.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.385+0000 7f4bbf000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bb80d18b0 0x7f4bb815e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:48.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.385+0000 7f4bbf000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bb8158620 0x7f4bb809def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:48.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.385+0000 7f4bbf000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bb81e8510 0x7f4bb81e89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:48.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.385+0000 7f4bbf000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb815f980 con 0x7f4bb81e8510 2024-04-02T07:35:48.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.385+0000 7f4bbf000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb81e8f00 con 0x7f4bb8158620 2024-04-02T07:35:48.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.385+0000 7f4bbf000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb815fda0 con 0x7f4bb80d18b0 2024-04-02T07:35:48.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.385+0000 7f4bbd000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bb81e8510 0x7f4bb81e89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:48.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.385+0000 7f4bbd000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bb81e8510 0x7f4bb81e89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53722/0 (socket says 172.21.15.89:53722) 2024-04-02T07:35:48.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.385+0000 7f4bbd000700 1 -- 172.21.15.89:0/1515664670 learned_addr learned my addr 172.21.15.89:0/1515664670 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:48.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.385+0000 7f4bb7e00700 1 --2- 172.21.15.89:0/1515664670 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bb80d18b0 0x7f4bb815e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:48.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.386+0000 7f4bb7400700 1 --2- 172.21.15.89:0/1515664670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bb8158620 0x7f4bb809def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:48.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.386+0000 7f4bbd000700 1 -- 172.21.15.89:0/1515664670 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bb80d18b0 msgr2=0x7f4bb815e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:48.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.386+0000 7f4bbd000700 1 --2- 172.21.15.89:0/1515664670 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bb80d18b0 0x7f4bb815e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.386+0000 7f4bbd000700 1 -- 172.21.15.89:0/1515664670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bb8158620 msgr2=0x7f4bb809def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:48.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.386+0000 7f4bbd000700 1 --2- 172.21.15.89:0/1515664670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bb8158620 0x7f4bb809def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.386+0000 7f4bbd000700 1 -- 172.21.15.89:0/1515664670 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb815ed00 con 0x7f4bb81e8510 2024-04-02T07:35:48.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.386+0000 7f4bbd000700 1 --2- 172.21.15.89:0/1515664670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bb81e8510 0x7f4bb81e89c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4bb808d550 tx=0x7f4bb000a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:48.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.386+0000 7f4bafe00700 1 -- 172.21.15.89:0/1515664670 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb000ab70 con 0x7f4bb81e8510 2024-04-02T07:35:48.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.386+0000 7f4bafe00700 1 -- 172.21.15.89:0/1515664670 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb000acf0 con 0x7f4bb81e8510 2024-04-02T07:35:48.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.386+0000 7f4bafe00700 1 -- 172.21.15.89:0/1515664670 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0013630 con 0x7f4bb81e8510 2024-04-02T07:35:48.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.386+0000 7f4bbf000700 1 -- 172.21.15.89:0/1515664670 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb815efe0 con 0x7f4bb81e8510 2024-04-02T07:35:48.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.387+0000 7f4bbf000700 1 -- 172.21.15.89:0/1515664670 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb815f5a0 con 0x7f4bb81e8510 2024-04-02T07:35:48.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.387+0000 7f4bbf000700 1 -- 172.21.15.89:0/1515664670 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb815fac0 con 0x7f4bb81e8510 2024-04-02T07:35:48.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.388+0000 7f4bafe00700 1 -- 172.21.15.89:0/1515664670 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4bb0013790 con 0x7f4bb81e8510 2024-04-02T07:35:48.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.390+0000 7f4bafe00700 1 --2- 172.21.15.89:0/1515664670 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4b9806ac90 0x7f4b9806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:48.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.390+0000 7f4bafe00700 1 -- 172.21.15.89:0/1515664670 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4bb0089ae0 con 0x7f4bb81e8510 2024-04-02T07:35:48.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.390+0000 7f4bb7e00700 1 --2- 172.21.15.89:0/1515664670 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4b9806ac90 0x7f4b9806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:48.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.390+0000 7f4bb7e00700 1 --2- 172.21.15.89:0/1515664670 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4b9806ac90 0x7f4b9806d150 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8007fd0 tx=0x7f4ba8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:48.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.394+0000 7f4bafe00700 1 -- 172.21.15.89:0/1515664670 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4bb815fac0 con 0x7f4bb81e8510 2024-04-02T07:35:48.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.600+0000 7f4bbf000700 1 -- 172.21.15.89:0/1515664670 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bb815fac0 con 0x7f4bb81e8510 2024-04-02T07:35:48.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.601+0000 7f4bafe00700 1 -- 172.21.15.89:0/1515664670 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12382 (secure 0 0 0) 0x7f4bb815fac0 con 0x7f4bb81e8510 2024-04-02T07:35:48.603 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:35:48.603 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":16,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:43.948311+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:35:48.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.603+0000 7f4bbf000700 1 -- 172.21.15.89:0/1515664670 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4b9806ac90 msgr2=0x7f4b9806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:48.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.603+0000 7f4bbf000700 1 --2- 172.21.15.89:0/1515664670 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4b9806ac90 0x7f4b9806d150 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8007fd0 tx=0x7f4ba8009040 comp rx=0 tx=0).stop 2024-04-02T07:35:48.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.603+0000 7f4bbf000700 1 -- 172.21.15.89:0/1515664670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bb81e8510 msgr2=0x7f4bb81e89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:48.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.603+0000 7f4bbf000700 1 --2- 172.21.15.89:0/1515664670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bb81e8510 0x7f4bb81e89c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4bb808d550 tx=0x7f4bb000a510 comp rx=0 tx=0).stop 2024-04-02T07:35:48.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.603+0000 7f4bbf000700 1 -- 172.21.15.89:0/1515664670 shutdown_connections 2024-04-02T07:35:48.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.603+0000 7f4bbf000700 1 --2- 172.21.15.89:0/1515664670 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4b9806ac90 0x7f4b9806d150 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.603+0000 7f4bbf000700 1 --2- 172.21.15.89:0/1515664670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bb81e8510 0x7f4bb81e89c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.603+0000 7f4bbf000700 1 --2- 172.21.15.89:0/1515664670 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bb80d18b0 0x7f4bb815e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.603+0000 7f4bbf000700 1 --2- 172.21.15.89:0/1515664670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bb8158620 0x7f4bb809def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.603+0000 7f4bbf000700 1 -- 172.21.15.89:0/1515664670 >> 172.21.15.89:0/1515664670 conn(0x7f4bb808aff0 msgr2=0x7f4bb815bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:48.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.603+0000 7f4bbf000700 1 -- 172.21.15.89:0/1515664670 shutdown_connections 2024-04-02T07:35:48.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.603+0000 7f4bbf000700 1 -- 172.21.15.89:0/1515664670 wait complete. 2024-04-02T07:35:48.623 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:35:48.623 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:35:48.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.817+0000 7f52a6400700 1 Processor -- start 2024-04-02T07:35:48.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.817+0000 7f52a6400700 1 -- start start 2024-04-02T07:35:48.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.819+0000 7f52a6400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f52a015d800 0x7f52a00725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:48.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.819+0000 7f52a6400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52a00ce230 0x7f52a00ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:48.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.819+0000 7f52a6400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a00709c0 con 0x7f52a00b2070 2024-04-02T07:35:48.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.819+0000 7f52a6400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a00b40f0 con 0x7f52a015d800 2024-04-02T07:35:48.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.819+0000 7f52a6400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0070c80 con 0x7f52a00ce230 2024-04-02T07:35:48.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.819+0000 7f529f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f52a015d800 0x7f52a00725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:48.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.819+0000 7f529f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f52a015d800 0x7f52a00725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57700/0 (socket says 172.21.15.89:57700) 2024-04-02T07:35:48.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.819+0000 7f529f400700 1 -- 172.21.15.89:0/4191025001 learned_addr learned my addr 172.21.15.89:0/4191025001 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:48.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.820+0000 7f5297e00700 1 --2- 172.21.15.89:0/4191025001 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52a00ce230 0x7f52a00ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:48.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.820+0000 7f529f400700 1 -- 172.21.15.89:0/4191025001 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52a00ce230 msgr2=0x7f52a00ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:48.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.820+0000 7f529f400700 1 --2- 172.21.15.89:0/4191025001 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52a00ce230 0x7f52a00ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.820+0000 7f529f400700 1 -- 172.21.15.89:0/4191025001 >> v1:172.21.15.89:6789/0 conn(0x7f52a00b2070 legacy=0x7f52a015dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:48.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.820+0000 7f529f400700 1 -- 172.21.15.89:0/4191025001 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a0070f60 con 0x7f52a015d800 2024-04-02T07:35:48.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.820+0000 7f529ea00700 1 -- 172.21.15.89:0/4191025001 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384468295 0 0) 0x7f52a00709c0 con 0x7f52a00b2070 2024-04-02T07:35:48.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.820+0000 7f529f400700 1 --2- 172.21.15.89:0/4191025001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52a015d800 0x7f52a00725c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f52900098f0 tx=0x7f5290009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad74bd6960ae7c1c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:48.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.820+0000 7f529ea00700 1 -- 172.21.15.89:0/4191025001 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529000b8a0 con 0x7f52a015d800 2024-04-02T07:35:48.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.820+0000 7f529ea00700 1 -- 172.21.15.89:0/4191025001 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529000ba00 con 0x7f52a015d800 2024-04-02T07:35:48.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.820+0000 7f529ea00700 1 -- 172.21.15.89:0/4191025001 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529000bd90 con 0x7f52a015d800 2024-04-02T07:35:48.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.821+0000 7f52a6400700 1 -- 172.21.15.89:0/4191025001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52a015d800 msgr2=0x7f52a00725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:48.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.821+0000 7f52a6400700 1 --2- 172.21.15.89:0/4191025001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52a015d800 0x7f52a00725c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f52900098f0 tx=0x7f5290009c00 comp rx=0 tx=0).stop 2024-04-02T07:35:48.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.821+0000 7f52a6400700 1 -- 172.21.15.89:0/4191025001 shutdown_connections 2024-04-02T07:35:48.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.821+0000 7f52a6400700 1 --2- 172.21.15.89:0/4191025001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52a015d800 0x7f52a00725c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.821+0000 7f52a6400700 1 --2- 172.21.15.89:0/4191025001 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52a00ce230 0x7f52a00ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.821+0000 7f52a6400700 1 -- 172.21.15.89:0/4191025001 >> 172.21.15.89:0/4191025001 conn(0x7f52a00b0d70 msgr2=0x7f52a00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:48.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.821+0000 7f52a6400700 1 -- 172.21.15.89:0/4191025001 shutdown_connections 2024-04-02T07:35:48.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.821+0000 7f52a6400700 1 -- 172.21.15.89:0/4191025001 wait complete. 2024-04-02T07:35:48.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.822+0000 7f52a6400700 1 Processor -- start 2024-04-02T07:35:48.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.823+0000 7f52a6400700 1 -- start start 2024-04-02T07:35:48.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.823+0000 7f52a6400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52a00b2070 0x7f52a0146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:48.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.823+0000 7f52a6400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52a00ce230 0x7f52a006e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:48.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.823+0000 7f52a6400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52a015d800 0x7f52a015a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:48.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.824+0000 7f52a6400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a01f2d40 con 0x7f52a015d800 2024-04-02T07:35:48.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.824+0000 7f52a6400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a01472e0 con 0x7f52a00ce230 2024-04-02T07:35:48.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.824+0000 7f52a6400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a01556c0 con 0x7f52a00b2070 2024-04-02T07:35:48.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.824+0000 7f529fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52a015d800 0x7f52a015a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:48.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.824+0000 7f529fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52a015d800 0x7f52a015a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57716/0 (socket says 172.21.15.89:57716) 2024-04-02T07:35:48.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.824+0000 7f529fe00700 1 -- 172.21.15.89:0/254174664 learned_addr learned my addr 172.21.15.89:0/254174664 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:48.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.824+0000 7f529f400700 1 --2- 172.21.15.89:0/254174664 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52a00b2070 0x7f52a0146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:48.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.824+0000 7f5297e00700 1 --2- 172.21.15.89:0/254174664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52a00ce230 0x7f52a006e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:48.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.824+0000 7f529fe00700 1 -- 172.21.15.89:0/254174664 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52a00b2070 msgr2=0x7f52a0146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:48.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.824+0000 7f529fe00700 1 --2- 172.21.15.89:0/254174664 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52a00b2070 0x7f52a0146da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.824+0000 7f529fe00700 1 -- 172.21.15.89:0/254174664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52a00ce230 msgr2=0x7f52a006e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:48.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.824+0000 7f529fe00700 1 --2- 172.21.15.89:0/254174664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52a00ce230 0x7f52a006e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:48.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.824+0000 7f529fe00700 1 -- 172.21.15.89:0/254174664 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5290009550 con 0x7f52a015d800 2024-04-02T07:35:48.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.824+0000 7f529fe00700 1 --2- 172.21.15.89:0/254174664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52a015d800 0x7f52a015a530 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f529800c760 tx=0x7f529800ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:48.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.825+0000 7f5297400700 1 -- 172.21.15.89:0/254174664 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298003e30 con 0x7f52a015d800 2024-04-02T07:35:48.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.825+0000 7f5297400700 1 -- 172.21.15.89:0/254174664 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5298005070 con 0x7f52a015d800 2024-04-02T07:35:48.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.825+0000 7f52a6400700 1 -- 172.21.15.89:0/254174664 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a0155850 con 0x7f52a015d800 2024-04-02T07:35:48.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.825+0000 7f5297400700 1 -- 172.21.15.89:0/254174664 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298005400 con 0x7f52a015d800 2024-04-02T07:35:48.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.825+0000 7f52a6400700 1 -- 172.21.15.89:0/254174664 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a015c3a0 con 0x7f52a015d800 2024-04-02T07:35:48.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.826+0000 7f52a6400700 1 -- 172.21.15.89:0/254174664 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a01559e0 con 0x7f52a015d800 2024-04-02T07:35:48.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.827+0000 7f5297400700 1 -- 172.21.15.89:0/254174664 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5298005560 con 0x7f52a015d800 2024-04-02T07:35:48.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.827+0000 7f5297400700 1 --2- 172.21.15.89:0/254174664 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f527c06ac90 0x7f527c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:48.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.828+0000 7f5297400700 1 -- 172.21.15.89:0/254174664 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5298088bd0 con 0x7f52a015d800 2024-04-02T07:35:48.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.828+0000 7f529f400700 1 --2- 172.21.15.89:0/254174664 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f527c06ac90 0x7f527c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:48.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.828+0000 7f529f400700 1 --2- 172.21.15.89:0/254174664 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f527c06ac90 0x7f527c06d150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5290000c00 tx=0x7f5290008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:48.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:48.832+0000 7f5297400700 1 -- 172.21.15.89:0/254174664 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f52980540f0 con 0x7f52a015d800 2024-04-02T07:35:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.037+0000 7f52a6400700 1 -- 172.21.15.89:0/254174664 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52a0067720 con 0x7f52a015d800 2024-04-02T07:35:49.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.037+0000 7f5297400700 1 -- 172.21.15.89:0/254174664 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5298057710 con 0x7f52a015d800 2024-04-02T07:35:49.075 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:35:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.039+0000 7f52a6400700 1 -- 172.21.15.89:0/254174664 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f527c06ac90 msgr2=0x7f527c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.040+0000 7f52a6400700 1 --2- 172.21.15.89:0/254174664 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f527c06ac90 0x7f527c06d150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5290000c00 tx=0x7f5290008030 comp rx=0 tx=0).stop 2024-04-02T07:35:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.040+0000 7f52a6400700 1 -- 172.21.15.89:0/254174664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52a015d800 msgr2=0x7f52a015a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.040+0000 7f52a6400700 1 --2- 172.21.15.89:0/254174664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52a015d800 0x7f52a015a530 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f529800c760 tx=0x7f529800ca70 comp rx=0 tx=0).stop 2024-04-02T07:35:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.040+0000 7f52a6400700 1 -- 172.21.15.89:0/254174664 shutdown_connections 2024-04-02T07:35:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.040+0000 7f52a6400700 1 --2- 172.21.15.89:0/254174664 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f527c06ac90 0x7f527c06d150 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.040+0000 7f52a6400700 1 --2- 172.21.15.89:0/254174664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52a015d800 0x7f52a015a530 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.040+0000 7f52a6400700 1 --2- 172.21.15.89:0/254174664 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52a00b2070 0x7f52a0146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.040+0000 7f52a6400700 1 --2- 172.21.15.89:0/254174664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52a00ce230 0x7f52a006e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.040+0000 7f52a6400700 1 -- 172.21.15.89:0/254174664 >> 172.21.15.89:0/254174664 conn(0x7f52a00b0d70 msgr2=0x7f52a00afbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:49.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.040+0000 7f52a6400700 1 -- 172.21.15.89:0/254174664 shutdown_connections 2024-04-02T07:35:49.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.040+0000 7f52a6400700 1 -- 172.21.15.89:0/254174664 wait complete. 2024-04-02T07:35:49.077 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:35:49.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.266+0000 7fc168800700 1 Processor -- start 2024-04-02T07:35:49.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.266+0000 7fc168800700 1 -- start start 2024-04-02T07:35:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.267+0000 7fc168800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc1600d18b0 0x7fc1600cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.267+0000 7fc168800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc160161590 0x7fc160161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.267+0000 7fc168800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1600ceda0 con 0x7fc160160f10 2024-04-02T07:35:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.267+0000 7fc168800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1600d1d30 con 0x7fc1600d18b0 2024-04-02T07:35:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.267+0000 7fc168800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc160161f60 con 0x7fc160161590 2024-04-02T07:35:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.267+0000 7fc165400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc1600d18b0 0x7fc1600cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.267+0000 7fc165400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc1600d18b0 0x7fc1600cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57728/0 (socket says 172.21.15.89:57728) 2024-04-02T07:35:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.267+0000 7fc165400700 1 -- 172.21.15.89:0/3785925796 learned_addr learned my addr 172.21.15.89:0/3785925796 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.267+0000 7fc166800700 1 --2- 172.21.15.89:0/3785925796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc160161590 0x7fc160161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:49.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.268+0000 7fc165400700 1 -- 172.21.15.89:0/3785925796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc160161590 msgr2=0x7fc160161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:49.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.268+0000 7fc165400700 1 --2- 172.21.15.89:0/3785925796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc160161590 0x7fc160161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.268+0000 7fc165400700 1 -- 172.21.15.89:0/3785925796 >> v1:172.21.15.113:6789/0 conn(0x7fc160160f10 legacy=0x7fc1600d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.268+0000 7fc165400700 1 -- 172.21.15.89:0/3785925796 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1601612f0 con 0x7fc1600d18b0 2024-04-02T07:35:49.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.268+0000 7fc164a00700 1 -- 172.21.15.89:0/3785925796 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163593297 0 0) 0x7fc1600ceda0 con 0x7fc160160f10 2024-04-02T07:35:49.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.268+0000 7fc165400700 1 --2- 172.21.15.89:0/3785925796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1600d18b0 0x7fc1600cf230 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc150009920 tx=0x7fc150009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafddac8f459ecb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:49.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.268+0000 7fc164a00700 1 -- 172.21.15.89:0/3785925796 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150011830 con 0x7fc1600d18b0 2024-04-02T07:35:49.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.268+0000 7fc164a00700 1 -- 172.21.15.89:0/3785925796 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc150011990 con 0x7fc1600d18b0 2024-04-02T07:35:49.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.268+0000 7fc164a00700 1 -- 172.21.15.89:0/3785925796 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150011d20 con 0x7fc1600d18b0 2024-04-02T07:35:49.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.269+0000 7fc168800700 1 -- 172.21.15.89:0/3785925796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1600d18b0 msgr2=0x7fc1600cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.269+0000 7fc168800700 1 --2- 172.21.15.89:0/3785925796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1600d18b0 0x7fc1600cf230 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc150009920 tx=0x7fc150009c30 comp rx=0 tx=0).stop 2024-04-02T07:35:49.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.269+0000 7fc168800700 1 -- 172.21.15.89:0/3785925796 shutdown_connections 2024-04-02T07:35:49.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.269+0000 7fc168800700 1 --2- 172.21.15.89:0/3785925796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc160161590 0x7fc160161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.269+0000 7fc168800700 1 --2- 172.21.15.89:0/3785925796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1600d18b0 0x7fc1600cf230 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.269+0000 7fc168800700 1 -- 172.21.15.89:0/3785925796 >> 172.21.15.89:0/3785925796 conn(0x7fc16008aff0 msgr2=0x7fc1600d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:49.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.269+0000 7fc168800700 1 -- 172.21.15.89:0/3785925796 shutdown_connections 2024-04-02T07:35:49.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.269+0000 7fc168800700 1 -- 172.21.15.89:0/3785925796 wait complete. 2024-04-02T07:35:49.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.270+0000 7fc168800700 1 Processor -- start 2024-04-02T07:35:49.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.271+0000 7fc168800700 1 -- start start 2024-04-02T07:35:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.271+0000 7fc168800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1600d18b0 0x7fc16015dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.271+0000 7fc168800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc160160f10 0x7fc16015fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.271+0000 7fc168800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc160161590 0x7fc16014b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc168800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16014c2b0 con 0x7fc1600d18b0 2024-04-02T07:35:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc168800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1601493c0 con 0x7fc160160f10 2024-04-02T07:35:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc168800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16015e450 con 0x7fc160161590 2024-04-02T07:35:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc165e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1600d18b0 0x7fc16015dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc165e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1600d18b0 0x7fc16015dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57730/0 (socket says 172.21.15.89:57730) 2024-04-02T07:35:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc165e00700 1 -- 172.21.15.89:0/1248566535 learned_addr learned my addr 172.21.15.89:0/1248566535 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc165400700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc160160f10 0x7fc16015fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc166800700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc160161590 0x7fc16014b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc165e00700 1 -- 172.21.15.89:0/1248566535 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc160161590 msgr2=0x7fc16014b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc165e00700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc160161590 0x7fc16014b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc165e00700 1 -- 172.21.15.89:0/1248566535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc160160f10 msgr2=0x7fc16015fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc165e00700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc160160f10 0x7fc16015fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc165e00700 1 -- 172.21.15.89:0/1248566535 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc150009550 con 0x7fc1600d18b0 2024-04-02T07:35:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc166800700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc160161590 0x7fc16014b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:35:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc165e00700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1600d18b0 0x7fc16015dcf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc15400c580 tx=0x7fc15400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc15a000700 1 -- 172.21.15.89:0/1248566535 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1540054e0 con 0x7fc1600d18b0 2024-04-02T07:35:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.272+0000 7fc15a000700 1 -- 172.21.15.89:0/1248566535 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc154003680 con 0x7fc1600d18b0 2024-04-02T07:35:49.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.273+0000 7fc15a000700 1 -- 172.21.15.89:0/1248566535 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154003a30 con 0x7fc1600d18b0 2024-04-02T07:35:49.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.273+0000 7fc168800700 1 -- 172.21.15.89:0/1248566535 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16015e230 con 0x7fc1600d18b0 2024-04-02T07:35:49.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.273+0000 7fc168800700 1 -- 172.21.15.89:0/1248566535 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc160149610 con 0x7fc1600d18b0 2024-04-02T07:35:49.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.274+0000 7fc168800700 1 -- 172.21.15.89:0/1248566535 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16008b3d0 con 0x7fc1600d18b0 2024-04-02T07:35:49.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.275+0000 7fc15a000700 1 -- 172.21.15.89:0/1248566535 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc154003b90 con 0x7fc1600d18b0 2024-04-02T07:35:49.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.275+0000 7fc15a000700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc14406ac90 0x7fc14406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:49.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.275+0000 7fc15a000700 1 -- 172.21.15.89:0/1248566535 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc1540884a0 con 0x7fc1600d18b0 2024-04-02T07:35:49.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.276+0000 7fc165400700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc14406ac90 0x7fc14406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:49.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.276+0000 7fc165400700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc14406ac90 0x7fc14406d150 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc150014040 tx=0x7fc150008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:49.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.280+0000 7fc15a000700 1 -- 172.21.15.89:0/1248566535 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc154053930 con 0x7fc1600d18b0 2024-04-02T07:35:49.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.527+0000 7fc168800700 1 -- 172.21.15.89:0/1248566535 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc160067720 con 0x7fc1600d18b0 2024-04-02T07:35:49.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.527+0000 7fc15a000700 1 -- 172.21.15.89:0/1248566535 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3527 (secure 0 0 0) 0x7fc154017020 con 0x7fc1600d18b0 2024-04-02T07:35:49.529 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:35:49.529 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2427593953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2427593953},{"type":"v1","addr":"172.21.15.113:6835","nonce":2427593953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/926246816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":926246816},{"type":"v1","addr":"172.21.15.89:6836","nonce":926246816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2701685411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2701685411},{"type":"v1","addr":"172.21.15.89:6837","nonce":2701685411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3962490902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3962490902},{"type":"v1","addr":"172.21.15.113:6834","nonce":3962490902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-04-02T07:35:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.530+0000 7fc168800700 1 -- 172.21.15.89:0/1248566535 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc14406ac90 msgr2=0x7fc14406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.530+0000 7fc168800700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc14406ac90 0x7fc14406d150 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc150014040 tx=0x7fc150008030 comp rx=0 tx=0).stop 2024-04-02T07:35:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.530+0000 7fc168800700 1 -- 172.21.15.89:0/1248566535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1600d18b0 msgr2=0x7fc16015dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.530+0000 7fc168800700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1600d18b0 0x7fc16015dcf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc15400c580 tx=0x7fc15400c890 comp rx=0 tx=0).stop 2024-04-02T07:35:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.530+0000 7fc168800700 1 -- 172.21.15.89:0/1248566535 shutdown_connections 2024-04-02T07:35:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.530+0000 7fc168800700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc14406ac90 0x7fc14406d150 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.530+0000 7fc168800700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc160161590 0x7fc16014b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.530+0000 7fc168800700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1600d18b0 0x7fc16015dcf0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.530+0000 7fc168800700 1 --2- 172.21.15.89:0/1248566535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc160160f10 0x7fc16015fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.530+0000 7fc168800700 1 -- 172.21.15.89:0/1248566535 >> 172.21.15.89:0/1248566535 conn(0x7fc16008aff0 msgr2=0x7fc1600caae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.530+0000 7fc168800700 1 -- 172.21.15.89:0/1248566535 shutdown_connections 2024-04-02T07:35:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.530+0000 7fc168800700 1 -- 172.21.15.89:0/1248566535 wait complete. 2024-04-02T07:35:49.533 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 3 2024-04-02T07:35:49.550 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:35:49.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.739+0000 7f10a6800700 1 Processor -- start 2024-04-02T07:35:49.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.739+0000 7f10a6800700 1 -- start start 2024-04-02T07:35:49.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.740+0000 7f10a6800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f10a0072580 0x7f10a00b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:49.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.740+0000 7f10a6800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a00b2180 con 0x7f10a00b2e50 2024-04-02T07:35:49.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.740+0000 7f10a6800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a015b100 con 0x7f10a00b3af0 2024-04-02T07:35:49.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.740+0000 7f10a6800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a015b3e0 con 0x7f10a0072580 2024-04-02T07:35:49.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.740+0000 7f109fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f10a00b2e50 0x7f10a00709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36894/0 (socket says 172.21.15.89:36894) 2024-04-02T07:35:49.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.740+0000 7f109fe00700 1 -- 172.21.15.89:0/3685739609 learned_addr learned my addr 172.21.15.89:0/3685739609 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:49.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.741+0000 7f109ea00700 1 --2- 172.21.15.89:0/3685739609 >> v2:172.21.15.113:3300/0 conn(0x7f10a0072580 0x7f10a00b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:49.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.741+0000 7f109e000700 1 -- 172.21.15.89:0/3685739609 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657791288 0 0) 0x7f10a00b2180 con 0x7f10a00b2e50 2024-04-02T07:35:49.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.741+0000 7f109e000700 1 -- 172.21.15.89:0/3685739609 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10840035b0 con 0x7f10a00b2e50 2024-04-02T07:35:49.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.741+0000 7f109e000700 1 -- 172.21.15.89:0/3685739609 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405664773 0 0) 0x7f10a015b100 con 0x7f10a00b3af0 2024-04-02T07:35:49.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.741+0000 7f109e000700 1 -- 172.21.15.89:0/3685739609 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a00b2180 con 0x7f10a00b3af0 2024-04-02T07:35:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.741+0000 7f109ea00700 1 -- 172.21.15.89:0/3685739609 >> v1:172.21.15.113:6789/0 conn(0x7f10a00b3af0 legacy=0x7f10a015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.741+0000 7f109ea00700 1 -- 172.21.15.89:0/3685739609 >> v1:172.21.15.89:6789/0 conn(0x7f10a00b2e50 legacy=0x7f10a00709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.741+0000 7f109e000700 1 -- 172.21.15.89:0/3685739609 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3676007841 0 0) 0x7f10840035b0 con 0x7f10a00b2e50 2024-04-02T07:35:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.741+0000 7f109ea00700 1 -- 172.21.15.89:0/3685739609 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a00b23f0 con 0x7f10a0072580 2024-04-02T07:35:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.741+0000 7f109ea00700 1 --2- 172.21.15.89:0/3685739609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10a0072580 0x7f10a00b2910 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f108c009840 tx=0x7f108c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebae2393bc8e8705 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.742+0000 7f109e000700 1 -- 172.21.15.89:0/3685739609 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10980019f0 con 0x7f10a0072580 2024-04-02T07:35:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.742+0000 7f109e000700 1 -- 172.21.15.89:0/3685739609 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108c004030 con 0x7f10a0072580 2024-04-02T07:35:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.742+0000 7f109e000700 1 -- 172.21.15.89:0/3685739609 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108c0043c0 con 0x7f10a0072580 2024-04-02T07:35:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.742+0000 7f10a6800700 1 -- 172.21.15.89:0/3685739609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10a0072580 msgr2=0x7f10a00b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.742+0000 7f10a6800700 1 --2- 172.21.15.89:0/3685739609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10a0072580 0x7f10a00b2910 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f108c009840 tx=0x7f108c009b50 comp rx=0 tx=0).stop 2024-04-02T07:35:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.742+0000 7f10a6800700 1 -- 172.21.15.89:0/3685739609 shutdown_connections 2024-04-02T07:35:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.742+0000 7f10a6800700 1 --2- 172.21.15.89:0/3685739609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10a0072580 0x7f10a00b2910 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.742+0000 7f10a6800700 1 -- 172.21.15.89:0/3685739609 >> 172.21.15.89:0/3685739609 conn(0x7f10a00cf190 msgr2=0x7f10a00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:49.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.742+0000 7f10a6800700 1 -- 172.21.15.89:0/3685739609 shutdown_connections 2024-04-02T07:35:49.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.742+0000 7f10a6800700 1 -- 172.21.15.89:0/3685739609 wait complete. 2024-04-02T07:35:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.744+0000 7f10a6800700 1 Processor -- start 2024-04-02T07:35:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.744+0000 7f10a6800700 1 -- start start 2024-04-02T07:35:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.745+0000 7f10a6800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10a0072580 0x7f10a01543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.745+0000 7f10a6800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10a00b2e50 0x7f10a0152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.745+0000 7f10a6800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10a00b3af0 0x7f10a015cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.745+0000 7f10a6800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a00c94a0 con 0x7f10a00b2e50 2024-04-02T07:35:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.745+0000 7f10a6800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a0153420 con 0x7f10a00b3af0 2024-04-02T07:35:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.745+0000 7f10a6800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a01528d0 con 0x7f10a0072580 2024-04-02T07:35:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.745+0000 7f109ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10a00b2e50 0x7f10a0152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.745+0000 7f109ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10a00b2e50 0x7f10a0152390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57738/0 (socket says 172.21.15.89:57738) 2024-04-02T07:35:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.745+0000 7f109ea00700 1 -- 172.21.15.89:0/3641607496 learned_addr learned my addr 172.21.15.89:0/3641607496 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.745+0000 7f109fe00700 1 --2- 172.21.15.89:0/3641607496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10a00b3af0 0x7f10a015cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.745+0000 7f109f400700 1 --2- 172.21.15.89:0/3641607496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10a0072580 0x7f10a01543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.746+0000 7f109ea00700 1 -- 172.21.15.89:0/3641607496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10a0072580 msgr2=0x7f10a01543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.746+0000 7f109ea00700 1 --2- 172.21.15.89:0/3641607496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10a0072580 0x7f10a01543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.746+0000 7f109ea00700 1 -- 172.21.15.89:0/3641607496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10a00b3af0 msgr2=0x7f10a015cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.746+0000 7f109ea00700 1 --2- 172.21.15.89:0/3641607496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10a00b3af0 0x7f10a015cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.746+0000 7f109ea00700 1 -- 172.21.15.89:0/3641607496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108c0094a0 con 0x7f10a00b2e50 2024-04-02T07:35:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.746+0000 7f109ea00700 1 --2- 172.21.15.89:0/3641607496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10a00b2e50 0x7f10a0152390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f108c0090b0 tx=0x7f108c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.746+0000 7f1097400700 1 -- 172.21.15.89:0/3641607496 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108c004570 con 0x7f10a00b2e50 2024-04-02T07:35:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.746+0000 7f1097400700 1 -- 172.21.15.89:0/3641607496 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108c01f070 con 0x7f10a00b2e50 2024-04-02T07:35:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.746+0000 7f10a6800700 1 -- 172.21.15.89:0/3641607496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a0152be0 con 0x7f10a00b2e50 2024-04-02T07:35:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.746+0000 7f1097400700 1 -- 172.21.15.89:0/3641607496 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108c011870 con 0x7f10a00b2e50 2024-04-02T07:35:49.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.746+0000 7f10a6800700 1 -- 172.21.15.89:0/3641607496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a00c96d0 con 0x7f10a00b2e50 2024-04-02T07:35:49.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.747+0000 7f10a6800700 1 -- 172.21.15.89:0/3641607496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a0152f70 con 0x7f10a00b2e50 2024-04-02T07:35:49.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.748+0000 7f1097400700 1 -- 172.21.15.89:0/3641607496 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f108c0119d0 con 0x7f10a00b2e50 2024-04-02T07:35:49.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.749+0000 7f1097400700 1 --2- 172.21.15.89:0/3641607496 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f108006ac90 0x7f108006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:49.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.749+0000 7f1097400700 1 -- 172.21.15.89:0/3641607496 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f108c021070 con 0x7f10a00b2e50 2024-04-02T07:35:49.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.749+0000 7f109f400700 1 --2- 172.21.15.89:0/3641607496 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f108006ac90 0x7f108006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:49.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.750+0000 7f109f400700 1 --2- 172.21.15.89:0/3641607496 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f108006ac90 0x7f108006d150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f10900011e0 tx=0x7f1090009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:49.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.753+0000 7f1097400700 1 -- 172.21.15.89:0/3641607496 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f108c055d90 con 0x7f10a00b2e50 2024-04-02T07:35:49.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.958+0000 7f10a6800700 1 -- 172.21.15.89:0/3641607496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10a0067720 con 0x7f10a00b2e50 2024-04-02T07:35:49.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.958+0000 7f1097400700 1 -- 172.21.15.89:0/3641607496 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12382 (secure 0 0 0) 0x7f108c02b090 con 0x7f10a00b2e50 2024-04-02T07:35:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:35:49.960 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":16,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:43.948311+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:35:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.961+0000 7f10a6800700 1 -- 172.21.15.89:0/3641607496 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f108006ac90 msgr2=0x7f108006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.961+0000 7f10a6800700 1 --2- 172.21.15.89:0/3641607496 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f108006ac90 0x7f108006d150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f10900011e0 tx=0x7f1090009040 comp rx=0 tx=0).stop 2024-04-02T07:35:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.961+0000 7f10a6800700 1 -- 172.21.15.89:0/3641607496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10a00b2e50 msgr2=0x7f10a0152390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.961+0000 7f10a6800700 1 --2- 172.21.15.89:0/3641607496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10a00b2e50 0x7f10a0152390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f108c0090b0 tx=0x7f108c004750 comp rx=0 tx=0).stop 2024-04-02T07:35:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.961+0000 7f10a6800700 1 -- 172.21.15.89:0/3641607496 shutdown_connections 2024-04-02T07:35:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.961+0000 7f10a6800700 1 --2- 172.21.15.89:0/3641607496 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f108006ac90 0x7f108006d150 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.961+0000 7f10a6800700 1 --2- 172.21.15.89:0/3641607496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10a00b3af0 0x7f10a015cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.961+0000 7f10a6800700 1 --2- 172.21.15.89:0/3641607496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10a0072580 0x7f10a01543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.961+0000 7f10a6800700 1 --2- 172.21.15.89:0/3641607496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10a00b2e50 0x7f10a0152390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.961+0000 7f10a6800700 1 -- 172.21.15.89:0/3641607496 >> 172.21.15.89:0/3641607496 conn(0x7f10a00cf190 msgr2=0x7f10a00acde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.961+0000 7f10a6800700 1 -- 172.21.15.89:0/3641607496 shutdown_connections 2024-04-02T07:35:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:49.961+0000 7f10a6800700 1 -- 172.21.15.89:0/3641607496 wait complete. 2024-04-02T07:35:49.980 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:35:49.981 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:35:50.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.171+0000 7f17ee600700 1 Processor -- start 2024-04-02T07:35:50.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.171+0000 7f17ee600700 1 -- start start 2024-04-02T07:35:50.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17ee600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f17e815d9d0 0x7f17e80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:50.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17ee600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17e80d2540 0x7f17e80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:50.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17ee600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e8161420 con 0x7f17e80d16a0 2024-04-02T07:35:50.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17ee600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e815edc0 con 0x7f17e815d9d0 2024-04-02T07:35:50.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17ee600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e8161700 con 0x7f17e80d2540 2024-04-02T07:35:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17e7e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17e80d2540 0x7f17e80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17e7e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17e80d2540 0x7f17e80ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:57274/0 (socket says 172.21.15.89:57274) 2024-04-02T07:35:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17e7400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f17e815d9d0 0x7f17e80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17e7e00700 1 -- 172.21.15.89:0/2732081275 learned_addr learned my addr 172.21.15.89:0/2732081275 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17dfe00700 1 --1- 172.21.15.89:0/2732081275 >> v1:172.21.15.113:6789/0 conn(0x7f17e80d16a0 0x7f17e80d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:38616/0 (socket says 172.21.15.89:38616) 2024-04-02T07:35:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17e7e00700 1 -- 172.21.15.89:0/2732081275 >> v2:172.21.15.113:3300/0 conn(0x7f17e815d9d0 msgr2=0x7f17e80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17e7e00700 1 --2- 172.21.15.89:0/2732081275 >> v2:172.21.15.113:3300/0 conn(0x7f17e815d9d0 0x7f17e80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17e7e00700 1 -- 172.21.15.89:0/2732081275 >> v1:172.21.15.113:6789/0 conn(0x7f17e80d16a0 legacy=0x7f17e80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17e7e00700 1 -- 172.21.15.89:0/2732081275 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e81619e0 con 0x7f17e80d2540 2024-04-02T07:35:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.173+0000 7f17e7400700 1 --2- 172.21.15.89:0/2732081275 >> v2:172.21.15.113:3300/0 conn(0x7f17e815d9d0 0x7f17e80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.174+0000 7f17e7e00700 1 --2- 172.21.15.89:0/2732081275 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17e80d2540 0x7f17e80ce360 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009e50 tx=0x7f17e000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d72407eba08640a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.174+0000 7f17e6a00700 1 -- 172.21.15.89:0/2732081275 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e000e070 con 0x7f17e80d2540 2024-04-02T07:35:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.174+0000 7f17e6a00700 1 -- 172.21.15.89:0/2732081275 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e000b910 con 0x7f17e80d2540 2024-04-02T07:35:50.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.174+0000 7f17e6a00700 1 -- 172.21.15.89:0/2732081275 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e000bcc0 con 0x7f17e80d2540 2024-04-02T07:35:50.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.174+0000 7f17ee600700 1 -- 172.21.15.89:0/2732081275 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17e80d2540 msgr2=0x7f17e80ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:50.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.174+0000 7f17ee600700 1 --2- 172.21.15.89:0/2732081275 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17e80d2540 0x7f17e80ce360 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009e50 tx=0x7f17e000b330 comp rx=0 tx=0).stop 2024-04-02T07:35:50.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.175+0000 7f17ee600700 1 -- 172.21.15.89:0/2732081275 shutdown_connections 2024-04-02T07:35:50.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.175+0000 7f17ee600700 1 --2- 172.21.15.89:0/2732081275 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17e80d2540 0x7f17e80ce360 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.175+0000 7f17ee600700 1 --2- 172.21.15.89:0/2732081275 >> v2:172.21.15.113:3300/0 conn(0x7f17e815d9d0 0x7f17e80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.175+0000 7f17ee600700 1 -- 172.21.15.89:0/2732081275 >> 172.21.15.89:0/2732081275 conn(0x7f17e808aff0 msgr2=0x7f17e80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:50.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.175+0000 7f17ee600700 1 -- 172.21.15.89:0/2732081275 shutdown_connections 2024-04-02T07:35:50.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.175+0000 7f17ee600700 1 -- 172.21.15.89:0/2732081275 wait complete. 2024-04-02T07:35:50.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.176+0000 7f17ee600700 1 Processor -- start 2024-04-02T07:35:50.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.176+0000 7f17ee600700 1 -- start start 2024-04-02T07:35:50.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.177+0000 7f17ee600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17e80d16a0 0x7f17e814f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:50.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.177+0000 7f17ee600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17e80d2540 0x7f17e80cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:50.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.177+0000 7f17ee600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17e815d9d0 0x7f17e80d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:50.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.177+0000 7f17ee600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80cc5a0 con 0x7f17e80d16a0 2024-04-02T07:35:50.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.177+0000 7f17ee600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80cc840 con 0x7f17e80d2540 2024-04-02T07:35:50.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.177+0000 7f17ee600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80ccb20 con 0x7f17e815d9d0 2024-04-02T07:35:50.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17e7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17e80d16a0 0x7f17e814f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:50.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17e7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17e80d16a0 0x7f17e814f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57748/0 (socket says 172.21.15.89:57748) 2024-04-02T07:35:50.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17e7400700 1 -- 172.21.15.89:0/1702044463 learned_addr learned my addr 172.21.15.89:0/1702044463 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:50.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17dfe00700 1 --2- 172.21.15.89:0/1702044463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17e80d2540 0x7f17e80cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:50.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17e7e00700 1 --2- 172.21.15.89:0/1702044463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17e815d9d0 0x7f17e80d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:50.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17e7400700 1 -- 172.21.15.89:0/1702044463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17e815d9d0 msgr2=0x7f17e80d04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:50.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17e7400700 1 --2- 172.21.15.89:0/1702044463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17e815d9d0 0x7f17e80d04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17e7400700 1 -- 172.21.15.89:0/1702044463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17e80d2540 msgr2=0x7f17e80cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:50.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17e7400700 1 --2- 172.21.15.89:0/1702044463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17e80d2540 0x7f17e80cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17e7400700 1 -- 172.21.15.89:0/1702044463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d80094a0 con 0x7f17e80d16a0 2024-04-02T07:35:50.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17e7400700 1 --2- 172.21.15.89:0/1702044463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17e80d16a0 0x7f17e814f7c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f17d800ebe0 tx=0x7f17d800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:50.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17df400700 1 -- 172.21.15.89:0/1702044463 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d8009760 con 0x7f17e80d16a0 2024-04-02T07:35:50.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17df400700 1 -- 172.21.15.89:0/1702044463 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d800c850 con 0x7f17e80d16a0 2024-04-02T07:35:50.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17df400700 1 -- 172.21.15.89:0/1702044463 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d800cbe0 con 0x7f17e80d16a0 2024-04-02T07:35:50.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.178+0000 7f17ee600700 1 -- 172.21.15.89:0/1702044463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e0009b30 con 0x7f17e80d16a0 2024-04-02T07:35:50.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.179+0000 7f17ee600700 1 -- 172.21.15.89:0/1702044463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e80cd0e0 con 0x7f17e80d16a0 2024-04-02T07:35:50.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.179+0000 7f17ee600700 1 -- 172.21.15.89:0/1702044463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e80cd450 con 0x7f17e80d16a0 2024-04-02T07:35:50.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.180+0000 7f17df400700 1 -- 172.21.15.89:0/1702044463 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f17d800cd40 con 0x7f17e80d16a0 2024-04-02T07:35:50.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.181+0000 7f17df400700 1 --2- 172.21.15.89:0/1702044463 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f17c406ac90 0x7f17c406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:50.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.181+0000 7f17df400700 1 -- 172.21.15.89:0/1702044463 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f17d8088fd0 con 0x7f17e80d16a0 2024-04-02T07:35:50.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.181+0000 7f17dfe00700 1 --2- 172.21.15.89:0/1702044463 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f17c406ac90 0x7f17c406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:50.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.182+0000 7f17dfe00700 1 --2- 172.21.15.89:0/1702044463 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f17c406ac90 0x7f17c406d150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f17e815d240 tx=0x7f17d0007ec0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:50.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.186+0000 7f17df400700 1 -- 172.21.15.89:0/1702044463 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f17d8054500 con 0x7f17e80d16a0 2024-04-02T07:35:50.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.394+0000 7f17ee600700 1 -- 172.21.15.89:0/1702044463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f17e80d1ca0 con 0x7f17e80d16a0 2024-04-02T07:35:50.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.395+0000 7f17df400700 1 -- 172.21.15.89:0/1702044463 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f17d8057b20 con 0x7f17e80d16a0 2024-04-02T07:35:50.397 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:35:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.397+0000 7f17ee600700 1 -- 172.21.15.89:0/1702044463 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f17c406ac90 msgr2=0x7f17c406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.397+0000 7f17ee600700 1 --2- 172.21.15.89:0/1702044463 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f17c406ac90 0x7f17c406d150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f17e815d240 tx=0x7f17d0007ec0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.397+0000 7f17ee600700 1 -- 172.21.15.89:0/1702044463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17e80d16a0 msgr2=0x7f17e814f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.397+0000 7f17ee600700 1 --2- 172.21.15.89:0/1702044463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17e80d16a0 0x7f17e814f7c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f17d800ebe0 tx=0x7f17d800eef0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.397+0000 7f17ee600700 1 -- 172.21.15.89:0/1702044463 shutdown_connections 2024-04-02T07:35:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.397+0000 7f17ee600700 1 --2- 172.21.15.89:0/1702044463 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f17c406ac90 0x7f17c406d150 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.397+0000 7f17ee600700 1 --2- 172.21.15.89:0/1702044463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17e815d9d0 0x7f17e80d04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.397+0000 7f17ee600700 1 --2- 172.21.15.89:0/1702044463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17e80d16a0 0x7f17e814f7c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.397+0000 7f17ee600700 1 --2- 172.21.15.89:0/1702044463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17e80d2540 0x7f17e80cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.397+0000 7f17ee600700 1 -- 172.21.15.89:0/1702044463 >> 172.21.15.89:0/1702044463 conn(0x7f17e808aff0 msgr2=0x7f17e80b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.398+0000 7f17ee600700 1 -- 172.21.15.89:0/1702044463 shutdown_connections 2024-04-02T07:35:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.398+0000 7f17ee600700 1 -- 172.21.15.89:0/1702044463 wait complete. 2024-04-02T07:35:50.417 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T07:35:50.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.606+0000 7f924b400700 1 Processor -- start 2024-04-02T07:35:50.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.606+0000 7f924b400700 1 -- start start 2024-04-02T07:35:50.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.607+0000 7f924b400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9244160f10 0x7f92440d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:50.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.607+0000 7f924b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92440cf230 0x7f92440d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:50.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.607+0000 7f924b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9244161590 0x7f9244161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:50.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.607+0000 7f924b400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9244161f60 con 0x7f9244161590 2024-04-02T07:35:50.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.607+0000 7f924b400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92440ceda0 con 0x7f9244160f10 2024-04-02T07:35:50.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.607+0000 7f924b400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92440d23d0 con 0x7f92440cf230 2024-04-02T07:35:50.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.608+0000 7f9249400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9244161590 0x7f9244161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:50.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.608+0000 7f9249400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9244161590 0x7f9244161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57754/0 (socket says 172.21.15.89:57754) 2024-04-02T07:35:50.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.608+0000 7f9249400700 1 -- 172.21.15.89:0/3278987527 learned_addr learned my addr 172.21.15.89:0/3278987527 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:50.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.608+0000 7f9248a00700 1 --2- 172.21.15.89:0/3278987527 >> v2:172.21.15.113:3300/0 conn(0x7f9244160f10 0x7f92440d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:50.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.608+0000 7f9243e00700 1 --2- 172.21.15.89:0/3278987527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92440cf230 0x7f92440d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:50.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.608+0000 7f9249400700 1 -- 172.21.15.89:0/3278987527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92440cf230 msgr2=0x7f92440d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:50.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.608+0000 7f9249400700 1 --2- 172.21.15.89:0/3278987527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92440cf230 0x7f92440d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.608+0000 7f9249400700 1 -- 172.21.15.89:0/3278987527 >> v2:172.21.15.113:3300/0 conn(0x7f9244160f10 msgr2=0x7f92440d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:50.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.608+0000 7f9249400700 1 --2- 172.21.15.89:0/3278987527 >> v2:172.21.15.113:3300/0 conn(0x7f9244160f10 0x7f92440d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.608+0000 7f9249400700 1 -- 172.21.15.89:0/3278987527 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92441612f0 con 0x7f9244161590 2024-04-02T07:35:50.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.608+0000 7f9243e00700 1 --2- 172.21.15.89:0/3278987527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92440cf230 0x7f92440d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:35:50.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.608+0000 7f9249400700 1 --2- 172.21.15.89:0/3278987527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9244161590 0x7f9244161a20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f92440d1080 tx=0x7f923c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8301d947d0bab717 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:50.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.608+0000 7f9243400700 1 -- 172.21.15.89:0/3278987527 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c00e070 con 0x7f9244161590 2024-04-02T07:35:50.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.608+0000 7f9243400700 1 -- 172.21.15.89:0/3278987527 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923c00b910 con 0x7f9244161590 2024-04-02T07:35:50.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.609+0000 7f9243400700 1 -- 172.21.15.89:0/3278987527 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c00bcc0 con 0x7f9244161590 2024-04-02T07:35:50.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.609+0000 7f924b400700 1 -- 172.21.15.89:0/3278987527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9244161590 msgr2=0x7f9244161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:50.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.609+0000 7f924b400700 1 --2- 172.21.15.89:0/3278987527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9244161590 0x7f9244161a20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f92440d1080 tx=0x7f923c00b330 comp rx=0 tx=0).stop 2024-04-02T07:35:50.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.609+0000 7f924b400700 1 -- 172.21.15.89:0/3278987527 shutdown_connections 2024-04-02T07:35:50.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.609+0000 7f924b400700 1 --2- 172.21.15.89:0/3278987527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9244161590 0x7f9244161a20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.609+0000 7f924b400700 1 --2- 172.21.15.89:0/3278987527 >> v2:172.21.15.113:3300/0 conn(0x7f9244160f10 0x7f92440d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.609+0000 7f924b400700 1 --2- 172.21.15.89:0/3278987527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92440cf230 0x7f92440d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.609+0000 7f924b400700 1 -- 172.21.15.89:0/3278987527 >> 172.21.15.89:0/3278987527 conn(0x7f924408aff0 msgr2=0x7f92440d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:50.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.609+0000 7f924b400700 1 -- 172.21.15.89:0/3278987527 shutdown_connections 2024-04-02T07:35:50.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.609+0000 7f924b400700 1 -- 172.21.15.89:0/3278987527 wait complete. 2024-04-02T07:35:50.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.611+0000 7f924b400700 1 Processor -- start 2024-04-02T07:35:50.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.611+0000 7f924b400700 1 -- start start 2024-04-02T07:35:50.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.612+0000 7f924b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92440cf230 0x7f92440b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:50.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.612+0000 7f924b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9244160f10 0x7f924415fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:50.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.612+0000 7f924b400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9244161590 0x7f92441f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:50.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.612+0000 7f924b400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92440b89d0 con 0x7f9244160f10 2024-04-02T07:35:50.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.612+0000 7f924b400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92441f6550 con 0x7f9244161590 2024-04-02T07:35:50.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.612+0000 7f924b400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9244160110 con 0x7f92440cf230 2024-04-02T07:35:50.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.612+0000 7f9243e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9244160f10 0x7f924415fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:50.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.612+0000 7f9243e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9244160f10 0x7f924415fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57762/0 (socket says 172.21.15.89:57762) 2024-04-02T07:35:50.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.612+0000 7f9243e00700 1 -- 172.21.15.89:0/1696316509 learned_addr learned my addr 172.21.15.89:0/1696316509 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:50.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.612+0000 7f9248a00700 1 --2- 172.21.15.89:0/1696316509 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92440cf230 0x7f92440b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:50.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.612+0000 7f9249400700 1 --2- 172.21.15.89:0/1696316509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9244161590 0x7f92441f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:50.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.612+0000 7f9243e00700 1 -- 172.21.15.89:0/1696316509 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92440cf230 msgr2=0x7f92440b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:50.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.612+0000 7f9243e00700 1 --2- 172.21.15.89:0/1696316509 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92440cf230 0x7f92440b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.613+0000 7f9243e00700 1 -- 172.21.15.89:0/1696316509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9244161590 msgr2=0x7f92441f2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:50.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.613+0000 7f9243e00700 1 --2- 172.21.15.89:0/1696316509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9244161590 0x7f92441f2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.613+0000 7f9243e00700 1 -- 172.21.15.89:0/1696316509 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923c009b30 con 0x7f9244160f10 2024-04-02T07:35:50.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.613+0000 7f9243e00700 1 --2- 172.21.15.89:0/1696316509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9244160f10 0x7f924415fbd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f923000e810 tx=0x7f923000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:50.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.613+0000 7f9240c00700 1 -- 172.21.15.89:0/1696316509 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923000c020 con 0x7f9244160f10 2024-04-02T07:35:50.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.613+0000 7f9240c00700 1 -- 172.21.15.89:0/1696316509 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9230004250 con 0x7f9244160f10 2024-04-02T07:35:50.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.613+0000 7f924b400700 1 -- 172.21.15.89:0/1696316509 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f924415e460 con 0x7f9244160f10 2024-04-02T07:35:50.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.613+0000 7f9240c00700 1 -- 172.21.15.89:0/1696316509 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92300036a0 con 0x7f9244160f10 2024-04-02T07:35:50.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.613+0000 7f924b400700 1 -- 172.21.15.89:0/1696316509 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f924415ea20 con 0x7f9244160f10 2024-04-02T07:35:50.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.614+0000 7f924b400700 1 -- 172.21.15.89:0/1696316509 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9244067720 con 0x7f9244160f10 2024-04-02T07:35:50.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.615+0000 7f9240c00700 1 -- 172.21.15.89:0/1696316509 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9230003800 con 0x7f9244160f10 2024-04-02T07:35:50.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.616+0000 7f9240c00700 1 --2- 172.21.15.89:0/1696316509 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f922406ac90 0x7f922406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:50.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.616+0000 7f9240c00700 1 -- 172.21.15.89:0/1696316509 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f92300889d0 con 0x7f9244160f10 2024-04-02T07:35:50.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.616+0000 7f9248a00700 1 --2- 172.21.15.89:0/1696316509 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f922406ac90 0x7f922406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:50.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.617+0000 7f9248a00700 1 --2- 172.21.15.89:0/1696316509 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f922406ac90 0x7f922406d150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9238009ac0 tx=0x7f9238009210 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:50.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.620+0000 7f9240c00700 1 -- 172.21.15.89:0/1696316509 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9230053e60 con 0x7f9244160f10 2024-04-02T07:35:50.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.824+0000 7f924b400700 1 -- 172.21.15.89:0/1696316509 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f92440d1de0 con 0x7f9244160f10 2024-04-02T07:35:50.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.825+0000 7f9240c00700 1 -- 172.21.15.89:0/1696316509 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9230057480 con 0x7f9244160f10 2024-04-02T07:35:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:35:50.827 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:35:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:35:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T07:35:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T07:35:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:35:50.827 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:35:50.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.827+0000 7f924b400700 1 -- 172.21.15.89:0/1696316509 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f922406ac90 msgr2=0x7f922406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:50.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.827+0000 7f924b400700 1 --2- 172.21.15.89:0/1696316509 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f922406ac90 0x7f922406d150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9238009ac0 tx=0x7f9238009210 comp rx=0 tx=0).stop 2024-04-02T07:35:50.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.827+0000 7f924b400700 1 -- 172.21.15.89:0/1696316509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9244160f10 msgr2=0x7f924415fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:50.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.827+0000 7f924b400700 1 --2- 172.21.15.89:0/1696316509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9244160f10 0x7f924415fbd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f923000e810 tx=0x7f923000eb20 comp rx=0 tx=0).stop 2024-04-02T07:35:50.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.827+0000 7f924b400700 1 -- 172.21.15.89:0/1696316509 shutdown_connections 2024-04-02T07:35:50.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.827+0000 7f924b400700 1 --2- 172.21.15.89:0/1696316509 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f922406ac90 0x7f922406d150 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.827+0000 7f924b400700 1 --2- 172.21.15.89:0/1696316509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9244161590 0x7f92441f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.827+0000 7f924b400700 1 --2- 172.21.15.89:0/1696316509 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92440cf230 0x7f92440b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.827+0000 7f924b400700 1 --2- 172.21.15.89:0/1696316509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9244160f10 0x7f924415fbd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:50.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.827+0000 7f924b400700 1 -- 172.21.15.89:0/1696316509 >> 172.21.15.89:0/1696316509 conn(0x7f924408aff0 msgr2=0x7f92440cad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:50.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.827+0000 7f924b400700 1 -- 172.21.15.89:0/1696316509 shutdown_connections 2024-04-02T07:35:50.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:50.827+0000 7f924b400700 1 -- 172.21.15.89:0/1696316509 wait complete. 2024-04-02T07:35:50.847 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T07:35:50.847 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T07:35:51.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.040+0000 7f7a21200700 1 Processor -- start 2024-04-02T07:35:51.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.040+0000 7f7a21200700 1 -- start start 2024-04-02T07:35:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.041+0000 7f7a21200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7a1c0ced30 0x7f7a1c0d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.041+0000 7f7a21200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a1c0d2710 0x7f7a1c15f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.041+0000 7f7a21200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1c0ce8a0 con 0x7f7a1c15ddf0 2024-04-02T07:35:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.041+0000 7f7a21200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1c15ec60 con 0x7f7a1c0ced30 2024-04-02T07:35:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.041+0000 7f7a21200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0cdc90 con 0x7f7a1c0d2710 2024-04-02T07:35:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.042+0000 7f7a1a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a1c0d2710 0x7f7a1c15f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.042+0000 7f7a19400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7a1c0ced30 0x7f7a1c0d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.042+0000 7f7a1a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a1c0d2710 0x7f7a1c15f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:57326/0 (socket says 172.21.15.89:57326) 2024-04-02T07:35:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.042+0000 7f7a19e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f7a1c15ddf0 0x7f7a1c15e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:38630/0 (socket says 172.21.15.89:38630) 2024-04-02T07:35:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.042+0000 7f7a1a800700 1 -- 172.21.15.89:0/2374546207 learned_addr learned my addr 172.21.15.89:0/2374546207 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:51.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.042+0000 7f7a19400700 1 -- 172.21.15.89:0/2374546207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a1c0d2710 msgr2=0x7f7a1c15f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:51.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.042+0000 7f7a19400700 1 --2- 172.21.15.89:0/2374546207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a1c0d2710 0x7f7a1c15f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:51.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.042+0000 7f7a19400700 1 -- 172.21.15.89:0/2374546207 >> v1:172.21.15.113:6789/0 conn(0x7f7a1c15ddf0 legacy=0x7f7a1c15e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:51.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.042+0000 7f7a19400700 1 -- 172.21.15.89:0/2374546207 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c0cdf70 con 0x7f7a1c0ced30 2024-04-02T07:35:51.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.042+0000 7f7a1a800700 1 --2- 172.21.15.89:0/2374546207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a1c0d2710 0x7f7a1c15f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:51.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.043+0000 7f7a19400700 1 --2- 172.21.15.89:0/2374546207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a1c0ced30 0x7f7a1c0d21d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009840 tx=0x7f7a08009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44052216c9eeaa5f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:51.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.043+0000 7f7a18a00700 1 -- 172.21.15.89:0/2374546207 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a08011830 con 0x7f7a1c0ced30 2024-04-02T07:35:51.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.043+0000 7f7a18a00700 1 -- 172.21.15.89:0/2374546207 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a08011990 con 0x7f7a1c0ced30 2024-04-02T07:35:51.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.043+0000 7f7a18a00700 1 -- 172.21.15.89:0/2374546207 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a08011d20 con 0x7f7a1c0ced30 2024-04-02T07:35:51.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.043+0000 7f7a21200700 1 -- 172.21.15.89:0/2374546207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a1c0ced30 msgr2=0x7f7a1c0d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:51.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.043+0000 7f7a21200700 1 --2- 172.21.15.89:0/2374546207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a1c0ced30 0x7f7a1c0d21d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009840 tx=0x7f7a08009b50 comp rx=0 tx=0).stop 2024-04-02T07:35:51.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.044+0000 7f7a21200700 1 -- 172.21.15.89:0/2374546207 shutdown_connections 2024-04-02T07:35:51.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.044+0000 7f7a21200700 1 --2- 172.21.15.89:0/2374546207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a1c0d2710 0x7f7a1c15f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:51.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.044+0000 7f7a21200700 1 --2- 172.21.15.89:0/2374546207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a1c0ced30 0x7f7a1c0d21d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:51.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.044+0000 7f7a21200700 1 -- 172.21.15.89:0/2374546207 >> 172.21.15.89:0/2374546207 conn(0x7f7a1c08aff0 msgr2=0x7f7a1c0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:51.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.044+0000 7f7a21200700 1 -- 172.21.15.89:0/2374546207 shutdown_connections 2024-04-02T07:35:51.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.044+0000 7f7a21200700 1 -- 172.21.15.89:0/2374546207 wait complete. 2024-04-02T07:35:51.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.045+0000 7f7a21200700 1 Processor -- start 2024-04-02T07:35:51.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.045+0000 7f7a21200700 1 -- start start 2024-04-02T07:35:51.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.046+0000 7f7a21200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1c0ced30 0x7f7a1c0b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:51.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.046+0000 7f7a21200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a1c0d2710 0x7f7a1c15c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:51.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.046+0000 7f7a21200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a1c15ddf0 0x7f7a1c15cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:51.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.046+0000 7f7a21200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c15f410 con 0x7f7a1c0ced30 2024-04-02T07:35:51.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.046+0000 7f7a21200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0b36e0 con 0x7f7a1c15ddf0 2024-04-02T07:35:51.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.046+0000 7f7a21200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0b39c0 con 0x7f7a1c0d2710 2024-04-02T07:35:51.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.047+0000 7f7a19e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1c0ced30 0x7f7a1c0b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:51.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.047+0000 7f7a19e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1c0ced30 0x7f7a1c0b4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57774/0 (socket says 172.21.15.89:57774) 2024-04-02T07:35:51.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.047+0000 7f7a19e00700 1 -- 172.21.15.89:0/539396019 learned_addr learned my addr 172.21.15.89:0/539396019 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:51.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.047+0000 7f7a1a800700 1 --2- 172.21.15.89:0/539396019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a1c15ddf0 0x7f7a1c15cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:51.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.047+0000 7f7a19400700 1 --2- 172.21.15.89:0/539396019 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a1c0d2710 0x7f7a1c15c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:51.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.047+0000 7f7a19e00700 1 -- 172.21.15.89:0/539396019 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a1c0d2710 msgr2=0x7f7a1c15c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:51.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.047+0000 7f7a19e00700 1 --2- 172.21.15.89:0/539396019 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a1c0d2710 0x7f7a1c15c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:51.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.047+0000 7f7a19e00700 1 -- 172.21.15.89:0/539396019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a1c15ddf0 msgr2=0x7f7a1c15cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:51.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.047+0000 7f7a19e00700 1 --2- 172.21.15.89:0/539396019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a1c15ddf0 0x7f7a1c15cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:51.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.047+0000 7f7a19e00700 1 -- 172.21.15.89:0/539396019 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a14009b30 con 0x7f7a1c0ced30 2024-04-02T07:35:51.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.047+0000 7f7a19e00700 1 --2- 172.21.15.89:0/539396019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1c0ced30 0x7f7a1c0b4d70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00c500 tx=0x7f7a0c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:51.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.047+0000 7f7a12000700 1 -- 172.21.15.89:0/539396019 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c005cb0 con 0x7f7a1c0ced30 2024-04-02T07:35:51.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.048+0000 7f7a12000700 1 -- 172.21.15.89:0/539396019 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c005e10 con 0x7f7a1c0ced30 2024-04-02T07:35:51.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.048+0000 7f7a21200700 1 -- 172.21.15.89:0/539396019 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a080094a0 con 0x7f7a1c0ced30 2024-04-02T07:35:51.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.048+0000 7f7a12000700 1 -- 172.21.15.89:0/539396019 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c0072d0 con 0x7f7a1c0ced30 2024-04-02T07:35:51.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.048+0000 7f7a21200700 1 -- 172.21.15.89:0/539396019 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1c0b3f80 con 0x7f7a1c0ced30 2024-04-02T07:35:51.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.049+0000 7f7a21200700 1 -- 172.21.15.89:0/539396019 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1c0723b0 con 0x7f7a1c0ced30 2024-04-02T07:35:51.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.050+0000 7f7a12000700 1 -- 172.21.15.89:0/539396019 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7a0c007430 con 0x7f7a1c0ced30 2024-04-02T07:35:51.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.051+0000 7f7a12000700 1 --2- 172.21.15.89:0/539396019 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f79fc06ac90 0x7f79fc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:51.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.051+0000 7f7a12000700 1 -- 172.21.15.89:0/539396019 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7a0c088210 con 0x7f7a1c0ced30 2024-04-02T07:35:51.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.051+0000 7f7a19400700 1 --2- 172.21.15.89:0/539396019 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f79fc06ac90 0x7f79fc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:51.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.052+0000 7f7a19400700 1 --2- 172.21.15.89:0/539396019 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f79fc06ac90 0x7f79fc06d150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7a08000c00 tx=0x7f7a08008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:51.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.055+0000 7f7a12000700 1 -- 172.21.15.89:0/539396019 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7a0c0537e0 con 0x7f7a1c0ced30 2024-04-02T07:35:51.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.256+0000 7f7a21200700 1 -- 172.21.15.89:0/539396019 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7a1c067720 con 0x7f7a1c0ced30 2024-04-02T07:35:51.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.732+0000 7f7a12000700 1 -- 172.21.15.89:0/539396019 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7a0c056e00 con 0x7f7a1c0ced30 2024-04-02T07:35:51.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.735+0000 7f7a21200700 1 -- 172.21.15.89:0/539396019 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f79fc06ac90 msgr2=0x7f79fc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:51.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.735+0000 7f7a21200700 1 --2- 172.21.15.89:0/539396019 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f79fc06ac90 0x7f79fc06d150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7a08000c00 tx=0x7f7a08008030 comp rx=0 tx=0).stop 2024-04-02T07:35:51.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.735+0000 7f7a21200700 1 -- 172.21.15.89:0/539396019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1c0ced30 msgr2=0x7f7a1c0b4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:51.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.735+0000 7f7a21200700 1 --2- 172.21.15.89:0/539396019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1c0ced30 0x7f7a1c0b4d70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00c500 tx=0x7f7a0c00c810 comp rx=0 tx=0).stop 2024-04-02T07:35:51.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.735+0000 7f7a21200700 1 -- 172.21.15.89:0/539396019 shutdown_connections 2024-04-02T07:35:51.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.735+0000 7f7a21200700 1 --2- 172.21.15.89:0/539396019 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f79fc06ac90 0x7f79fc06d150 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:51.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.735+0000 7f7a21200700 1 --2- 172.21.15.89:0/539396019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a1c15ddf0 0x7f7a1c15cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:51.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.735+0000 7f7a21200700 1 --2- 172.21.15.89:0/539396019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1c0ced30 0x7f7a1c0b4d70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:51.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.735+0000 7f7a21200700 1 --2- 172.21.15.89:0/539396019 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a1c0d2710 0x7f7a1c15c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:51.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.735+0000 7f7a21200700 1 -- 172.21.15.89:0/539396019 >> 172.21.15.89:0/539396019 conn(0x7f7a1c08aff0 msgr2=0x7f7a1c0b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:51.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.735+0000 7f7a21200700 1 -- 172.21.15.89:0/539396019 shutdown_connections 2024-04-02T07:35:51.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.736+0000 7f7a21200700 1 -- 172.21.15.89:0/539396019 wait complete. 2024-04-02T07:35:51.742 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T07:35:51.762 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T07:35:51.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.957+0000 7f62f3a00700 1 Processor -- start 2024-04-02T07:35:51.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.957+0000 7f62f3a00700 1 -- start start 2024-04-02T07:35:51.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.958+0000 7f62f3a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62ec15d9d0 0x7f62ec0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:51.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.958+0000 7f62f3a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f62ec0d2540 0x7f62ec0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:51.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.958+0000 7f62f3a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62ec161420 con 0x7f62ec0d16a0 2024-04-02T07:35:51.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.958+0000 7f62f3a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62ec15edc0 con 0x7f62ec0d2540 2024-04-02T07:35:51.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.958+0000 7f62f3a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec161700 con 0x7f62ec15d9d0 2024-04-02T07:35:51.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.958+0000 7f62f1000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62ec15d9d0 0x7f62ec0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:51.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.958+0000 7f62f1a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f62ec0d2540 0x7f62ec0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:51.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.958+0000 7f62f1000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62ec15d9d0 0x7f62ec0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:57354/0 (socket says 172.21.15.89:57354) 2024-04-02T07:35:51.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.958+0000 7f62f1000700 1 -- 172.21.15.89:0/1064873512 learned_addr learned my addr 172.21.15.89:0/1064873512 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:51.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.958+0000 7f62ebe00700 1 --1- 172.21.15.89:0/1064873512 >> v1:172.21.15.113:6789/0 conn(0x7f62ec0d16a0 0x7f62ec0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:38632/0 (socket says 172.21.15.89:38632) 2024-04-02T07:35:51.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.959+0000 7f62f1000700 1 -- 172.21.15.89:0/1064873512 >> v2:172.21.15.113:3300/0 conn(0x7f62ec0d2540 msgr2=0x7f62ec0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:51.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.959+0000 7f62f1000700 1 --2- 172.21.15.89:0/1064873512 >> v2:172.21.15.113:3300/0 conn(0x7f62ec0d2540 0x7f62ec0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:51.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.959+0000 7f62f1000700 1 -- 172.21.15.89:0/1064873512 >> v1:172.21.15.113:6789/0 conn(0x7f62ec0d16a0 legacy=0x7f62ec0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:51.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.959+0000 7f62f1000700 1 -- 172.21.15.89:0/1064873512 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec1619e0 con 0x7f62ec15d9d0 2024-04-02T07:35:51.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.959+0000 7f62f1a00700 1 --2- 172.21.15.89:0/1064873512 >> v2:172.21.15.113:3300/0 conn(0x7f62ec0d2540 0x7f62ec0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:51.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.959+0000 7f62f1000700 1 --2- 172.21.15.89:0/1064873512 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62ec15d9d0 0x7f62ec0d3c30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009920 tx=0x7f62e0009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afe4bf2d65f64d44 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:51.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.959+0000 7f62eb400700 1 -- 172.21.15.89:0/1064873512 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e0011830 con 0x7f62ec15d9d0 2024-04-02T07:35:51.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.959+0000 7f62eb400700 1 -- 172.21.15.89:0/1064873512 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e0011990 con 0x7f62ec15d9d0 2024-04-02T07:35:51.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.959+0000 7f62eb400700 1 -- 172.21.15.89:0/1064873512 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e0011d20 con 0x7f62ec15d9d0 2024-04-02T07:35:51.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.960+0000 7f62f3a00700 1 -- 172.21.15.89:0/1064873512 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62ec15d9d0 msgr2=0x7f62ec0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:51.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.960+0000 7f62f3a00700 1 --2- 172.21.15.89:0/1064873512 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62ec15d9d0 0x7f62ec0d3c30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009920 tx=0x7f62e0009c30 comp rx=0 tx=0).stop 2024-04-02T07:35:51.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.960+0000 7f62f3a00700 1 -- 172.21.15.89:0/1064873512 shutdown_connections 2024-04-02T07:35:51.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.960+0000 7f62f3a00700 1 --2- 172.21.15.89:0/1064873512 >> v2:172.21.15.113:3300/0 conn(0x7f62ec0d2540 0x7f62ec0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:51.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.960+0000 7f62f3a00700 1 --2- 172.21.15.89:0/1064873512 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62ec15d9d0 0x7f62ec0d3c30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:51.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.960+0000 7f62f3a00700 1 -- 172.21.15.89:0/1064873512 >> 172.21.15.89:0/1064873512 conn(0x7f62ec08aff0 msgr2=0x7f62ec0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:51.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.960+0000 7f62f3a00700 1 -- 172.21.15.89:0/1064873512 shutdown_connections 2024-04-02T07:35:51.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.960+0000 7f62f3a00700 1 -- 172.21.15.89:0/1064873512 wait complete. 2024-04-02T07:35:51.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.962+0000 7f62f3a00700 1 Processor -- start 2024-04-02T07:35:51.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.962+0000 7f62f3a00700 1 -- start start 2024-04-02T07:35:51.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62f3a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62ec0d16a0 0x7f62ec0b7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62f3a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec0d2540 0x7f62ec06d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62f3a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62ec15d9d0 0x7f62ec14b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62f3a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec06d5b0 con 0x7f62ec0d2540 2024-04-02T07:35:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62f3a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec14be10 con 0x7f62ec0d16a0 2024-04-02T07:35:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62f3a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec06d7c0 con 0x7f62ec15d9d0 2024-04-02T07:35:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62ebe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec0d2540 0x7f62ec06d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62ebe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec0d2540 0x7f62ec06d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57784/0 (socket says 172.21.15.89:57784) 2024-04-02T07:35:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62ebe00700 1 -- 172.21.15.89:0/1857359938 learned_addr learned my addr 172.21.15.89:0/1857359938 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62f1a00700 1 --2- 172.21.15.89:0/1857359938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62ec15d9d0 0x7f62ec14b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62f1000700 1 --2- 172.21.15.89:0/1857359938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62ec0d16a0 0x7f62ec0b7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62ebe00700 1 -- 172.21.15.89:0/1857359938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62ec15d9d0 msgr2=0x7f62ec14b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62ebe00700 1 --2- 172.21.15.89:0/1857359938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62ec15d9d0 0x7f62ec14b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62ebe00700 1 -- 172.21.15.89:0/1857359938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62ec0d16a0 msgr2=0x7f62ec0b7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:51.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62ebe00700 1 --2- 172.21.15.89:0/1857359938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62ec0d16a0 0x7f62ec0b7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:51.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.963+0000 7f62ebe00700 1 -- 172.21.15.89:0/1857359938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e4009a70 con 0x7f62ec0d2540 2024-04-02T07:35:51.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.964+0000 7f62ebe00700 1 --2- 172.21.15.89:0/1857359938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec0d2540 0x7f62ec06d070 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f62d800c4d0 tx=0x7f62d800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:51.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.964+0000 7f62e8c00700 1 -- 172.21.15.89:0/1857359938 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d8005d00 con 0x7f62ec0d2540 2024-04-02T07:35:51.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.964+0000 7f62e8c00700 1 -- 172.21.15.89:0/1857359938 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d8005e60 con 0x7f62ec0d2540 2024-04-02T07:35:51.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.964+0000 7f62f3a00700 1 -- 172.21.15.89:0/1857359938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e0009550 con 0x7f62ec0d2540 2024-04-02T07:35:51.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.964+0000 7f62e8c00700 1 -- 172.21.15.89:0/1857359938 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d8007310 con 0x7f62ec0d2540 2024-04-02T07:35:51.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.964+0000 7f62f3a00700 1 -- 172.21.15.89:0/1857359938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ec0b0580 con 0x7f62ec0d2540 2024-04-02T07:35:51.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.965+0000 7f62f3a00700 1 -- 172.21.15.89:0/1857359938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ec0d1ca0 con 0x7f62ec0d2540 2024-04-02T07:35:51.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.966+0000 7f62e8c00700 1 -- 172.21.15.89:0/1857359938 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f62d8007470 con 0x7f62ec0d2540 2024-04-02T07:35:51.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.967+0000 7f62e8c00700 1 --2- 172.21.15.89:0/1857359938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f62cc06ac90 0x7f62cc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:51.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.967+0000 7f62e8c00700 1 -- 172.21.15.89:0/1857359938 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f62d8088510 con 0x7f62ec0d2540 2024-04-02T07:35:51.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.967+0000 7f62f1000700 1 --2- 172.21.15.89:0/1857359938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f62cc06ac90 0x7f62cc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:51.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.968+0000 7f62f1000700 1 --2- 172.21.15.89:0/1857359938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f62cc06ac90 0x7f62cc06d150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f62e0000c00 tx=0x7f62e0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:51.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:51.972+0000 7f62e8c00700 1 -- 172.21.15.89:0/1857359938 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f62d80538c0 con 0x7f62ec0d2540 2024-04-02T07:35:52.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.193+0000 7f62f3a00700 1 -- 172.21.15.89:0/1857359938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f62ec067720 con 0x7f62ec0d2540 2024-04-02T07:35:52.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.740+0000 7f62e8c00700 1 -- 172.21.15.89:0/1857359938 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f62d8001260 con 0x7f62ec0d2540 2024-04-02T07:35:52.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.745+0000 7f62f3a00700 1 -- 172.21.15.89:0/1857359938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f62cc06ac90 msgr2=0x7f62cc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:52.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.745+0000 7f62f3a00700 1 --2- 172.21.15.89:0/1857359938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f62cc06ac90 0x7f62cc06d150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f62e0000c00 tx=0x7f62e0008030 comp rx=0 tx=0).stop 2024-04-02T07:35:52.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.745+0000 7f62f3a00700 1 -- 172.21.15.89:0/1857359938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec0d2540 msgr2=0x7f62ec06d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:52.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.745+0000 7f62f3a00700 1 --2- 172.21.15.89:0/1857359938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec0d2540 0x7f62ec06d070 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f62d800c4d0 tx=0x7f62d800c7e0 comp rx=0 tx=0).stop 2024-04-02T07:35:52.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.746+0000 7f62f3a00700 1 -- 172.21.15.89:0/1857359938 shutdown_connections 2024-04-02T07:35:52.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.746+0000 7f62f3a00700 1 --2- 172.21.15.89:0/1857359938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f62cc06ac90 0x7f62cc06d150 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:52.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.746+0000 7f62f3a00700 1 --2- 172.21.15.89:0/1857359938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62ec0d16a0 0x7f62ec0b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:52.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.746+0000 7f62f3a00700 1 --2- 172.21.15.89:0/1857359938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62ec15d9d0 0x7f62ec14b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:52.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.746+0000 7f62f3a00700 1 --2- 172.21.15.89:0/1857359938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec0d2540 0x7f62ec06d070 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f62d800c4d0 tx=0x7f62d800c7e0 comp rx=0 tx=0).stop 2024-04-02T07:35:52.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.746+0000 7f62f3a00700 1 -- 172.21.15.89:0/1857359938 >> 172.21.15.89:0/1857359938 conn(0x7f62ec08aff0 msgr2=0x7f62ec15c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:52.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.750+0000 7f62f3a00700 1 -- 172.21.15.89:0/1857359938 shutdown_connections 2024-04-02T07:35:52.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.750+0000 7f62f3a00700 1 -- 172.21.15.89:0/1857359938 wait complete. 2024-04-02T07:35:52.755 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T07:35:52.781 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T07:35:52.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.973+0000 7f53de000700 1 Processor -- start 2024-04-02T07:35:52.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.973+0000 7f53de000700 1 -- start start 2024-04-02T07:35:52.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.974+0000 7f53de000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f53d815d9d0 0x7f53d80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:52.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.974+0000 7f53de000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53d80d2540 0x7f53d80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:52.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.974+0000 7f53de000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d8161420 con 0x7f53d80d16a0 2024-04-02T07:35:52.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.974+0000 7f53de000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53d815edc0 con 0x7f53d815d9d0 2024-04-02T07:35:52.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.974+0000 7f53de000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d8161700 con 0x7f53d80d2540 2024-04-02T07:35:52.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.975+0000 7f53d7e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53d80d2540 0x7f53d80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:52.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.975+0000 7f53d7400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f53d815d9d0 0x7f53d80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:52.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.975+0000 7f53d6a00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f53d80d16a0 0x7f53d80d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:38642/0 (socket says 172.21.15.89:38642) 2024-04-02T07:35:52.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.975+0000 7f53d6a00700 1 -- 172.21.15.89:0/2225121635 learned_addr learned my addr 172.21.15.89:0/2225121635 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:52.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.975+0000 7f53d6000700 1 -- 172.21.15.89:0/2225121635 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450832927 0 0) 0x7f53d8161420 con 0x7f53d80d16a0 2024-04-02T07:35:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.975+0000 7f53d6000700 1 -- 172.21.15.89:0/2225121635 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53bc0035b0 con 0x7f53d80d16a0 2024-04-02T07:35:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.975+0000 7f53d7400700 1 -- 172.21.15.89:0/2225121635 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53d80d2540 msgr2=0x7f53d80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.975+0000 7f53d7400700 1 --2- 172.21.15.89:0/2225121635 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53d80d2540 0x7f53d80ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.975+0000 7f53d7400700 1 -- 172.21.15.89:0/2225121635 >> v1:172.21.15.113:6789/0 conn(0x7f53d80d16a0 legacy=0x7f53d80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.975+0000 7f53d7400700 1 -- 172.21.15.89:0/2225121635 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d81619e0 con 0x7f53d815d9d0 2024-04-02T07:35:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.975+0000 7f53d7e00700 1 --2- 172.21.15.89:0/2225121635 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53d80d2540 0x7f53d80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.975+0000 7f53d7400700 1 --2- 172.21.15.89:0/2225121635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53d815d9d0 0x7f53d80d3c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009840 tx=0x7f53c8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4bd775192d6eed3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.976+0000 7f53d6000700 1 -- 172.21.15.89:0/2225121635 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c8011830 con 0x7f53d815d9d0 2024-04-02T07:35:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.976+0000 7f53d6000700 1 -- 172.21.15.89:0/2225121635 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c8011990 con 0x7f53d815d9d0 2024-04-02T07:35:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.976+0000 7f53d6000700 1 -- 172.21.15.89:0/2225121635 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c8011d20 con 0x7f53d815d9d0 2024-04-02T07:35:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.976+0000 7f53de000700 1 -- 172.21.15.89:0/2225121635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53d815d9d0 msgr2=0x7f53d80d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.976+0000 7f53de000700 1 --2- 172.21.15.89:0/2225121635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53d815d9d0 0x7f53d80d3c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009840 tx=0x7f53c8009b50 comp rx=0 tx=0).stop 2024-04-02T07:35:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.976+0000 7f53de000700 1 -- 172.21.15.89:0/2225121635 shutdown_connections 2024-04-02T07:35:52.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.976+0000 7f53de000700 1 --2- 172.21.15.89:0/2225121635 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53d80d2540 0x7f53d80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:52.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.976+0000 7f53de000700 1 --2- 172.21.15.89:0/2225121635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53d815d9d0 0x7f53d80d3c30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:52.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.976+0000 7f53de000700 1 -- 172.21.15.89:0/2225121635 >> 172.21.15.89:0/2225121635 conn(0x7f53d808aff0 msgr2=0x7f53d80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:52.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.976+0000 7f53de000700 1 -- 172.21.15.89:0/2225121635 shutdown_connections 2024-04-02T07:35:52.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.976+0000 7f53de000700 1 -- 172.21.15.89:0/2225121635 wait complete. 2024-04-02T07:35:52.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.978+0000 7f53de000700 1 Processor -- start 2024-04-02T07:35:52.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.978+0000 7f53de000700 1 -- start start 2024-04-02T07:35:52.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.979+0000 7f53de000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53d80d16a0 0x7f53d806c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:52.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.979+0000 7f53de000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53d80d2540 0x7f53d815c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:52.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.979+0000 7f53de000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53d815d9d0 0x7f53d815ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:52.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.979+0000 7f53de000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d81e8fd0 con 0x7f53d815d9d0 2024-04-02T07:35:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.979+0000 7f53de000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d80b71a0 con 0x7f53d80d16a0 2024-04-02T07:35:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.979+0000 7f53de000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d80b7480 con 0x7f53d80d2540 2024-04-02T07:35:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.979+0000 7f53d7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53d815d9d0 0x7f53d815ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.979+0000 7f53d7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53d815d9d0 0x7f53d815ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57794/0 (socket says 172.21.15.89:57794) 2024-04-02T07:35:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.979+0000 7f53d7e00700 1 -- 172.21.15.89:0/2918085850 learned_addr learned my addr 172.21.15.89:0/2918085850 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.979+0000 7f53d7400700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53d80d16a0 0x7f53d806c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.979+0000 7f53d6a00700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53d80d2540 0x7f53d815c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.980+0000 7f53d7e00700 1 -- 172.21.15.89:0/2918085850 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53d80d2540 msgr2=0x7f53d815c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.980+0000 7f53d7e00700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53d80d2540 0x7f53d815c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.980+0000 7f53d7e00700 1 -- 172.21.15.89:0/2918085850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53d80d16a0 msgr2=0x7f53d806c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.980+0000 7f53d7e00700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53d80d16a0 0x7f53d806c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.980+0000 7f53d7e00700 1 -- 172.21.15.89:0/2918085850 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d0009a70 con 0x7f53d815d9d0 2024-04-02T07:35:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.980+0000 7f53d7400700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53d80d16a0 0x7f53d806c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.980+0000 7f53d7e00700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53d815d9d0 0x7f53d815ba60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f53d000ef90 tx=0x7f53d000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.980+0000 7f53cf400700 1 -- 172.21.15.89:0/2918085850 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d000cc90 con 0x7f53d815d9d0 2024-04-02T07:35:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.980+0000 7f53cf400700 1 -- 172.21.15.89:0/2918085850 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d000f040 con 0x7f53d815d9d0 2024-04-02T07:35:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.980+0000 7f53cf400700 1 -- 172.21.15.89:0/2918085850 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d00146c0 con 0x7f53d815d9d0 2024-04-02T07:35:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.980+0000 7f53de000700 1 -- 172.21.15.89:0/2918085850 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c80094a0 con 0x7f53d815d9d0 2024-04-02T07:35:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.980+0000 7f53de000700 1 -- 172.21.15.89:0/2918085850 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d80b7a40 con 0x7f53d815d9d0 2024-04-02T07:35:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.981+0000 7f53de000700 1 -- 172.21.15.89:0/2918085850 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d80d1ca0 con 0x7f53d815d9d0 2024-04-02T07:35:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.983+0000 7f53cf400700 1 -- 172.21.15.89:0/2918085850 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f53d0014820 con 0x7f53d815d9d0 2024-04-02T07:35:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.983+0000 7f53cf400700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f53b806ac90 0x7f53b806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.984+0000 7f53cf400700 1 -- 172.21.15.89:0/2918085850 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f53d0089f50 con 0x7f53d815d9d0 2024-04-02T07:35:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.984+0000 7f53d7400700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f53b806ac90 0x7f53b806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:52.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.984+0000 7f53d7400700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f53b806ac90 0x7f53b806d150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f53c8000c00 tx=0x7f53c801c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:52.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:52.989+0000 7f53cf400700 1 -- 172.21.15.89:0/2918085850 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f53d0055100 con 0x7f53d815d9d0 2024-04-02T07:35:53.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.233+0000 7f53de000700 1 -- 172.21.15.89:0/2918085850 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f53d8067720 con 0x7f53d815d9d0 2024-04-02T07:35:53.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.244+0000 7f53cf400700 1 -- 172.21.15.89:0/2918085850 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f53d0058720 con 0x7f53d815d9d0 2024-04-02T07:35:53.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.249+0000 7f53de000700 1 -- 172.21.15.89:0/2918085850 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f53b806ac90 msgr2=0x7f53b806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:53.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.249+0000 7f53de000700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f53b806ac90 0x7f53b806d150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f53c8000c00 tx=0x7f53c801c040 comp rx=0 tx=0).stop 2024-04-02T07:35:53.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.249+0000 7f53de000700 1 -- 172.21.15.89:0/2918085850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53d815d9d0 msgr2=0x7f53d815ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:53.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.249+0000 7f53de000700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53d815d9d0 0x7f53d815ba60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f53d000ef90 tx=0x7f53d000c3a0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.249+0000 7f53de000700 1 -- 172.21.15.89:0/2918085850 shutdown_connections 2024-04-02T07:35:53.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.249+0000 7f53de000700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f53b806ac90 0x7f53b806d150 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.249+0000 7f53de000700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53d80d16a0 0x7f53d806c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.249+0000 7f53de000700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53d815d9d0 0x7f53d815ba60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.249+0000 7f53de000700 1 --2- 172.21.15.89:0/2918085850 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53d80d2540 0x7f53d815c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.249+0000 7f53de000700 1 -- 172.21.15.89:0/2918085850 >> 172.21.15.89:0/2918085850 conn(0x7f53d808aff0 msgr2=0x7f53d80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:53.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.250+0000 7f53de000700 1 -- 172.21.15.89:0/2918085850 shutdown_connections 2024-04-02T07:35:53.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.250+0000 7f53de000700 1 -- 172.21.15.89:0/2918085850 wait complete. 2024-04-02T07:35:53.255 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 2 and data pool 3 2024-04-02T07:35:53.287 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:35:53.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.476+0000 7f2392800700 1 Processor -- start 2024-04-02T07:35:53.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.476+0000 7f2392800700 1 -- start start 2024-04-02T07:35:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.478+0000 7f2392800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f238c160f10 0x7f238c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.478+0000 7f2392800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f238c0cf230 0x7f238c0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.478+0000 7f2392800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c0ceda0 con 0x7f238c161590 2024-04-02T07:35:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.478+0000 7f2392800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f238c0d23d0 con 0x7f238c160f10 2024-04-02T07:35:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.478+0000 7f2392800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238c1612f0 con 0x7f238c0cf230 2024-04-02T07:35:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.479+0000 7f2383e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f238c0cf230 0x7f238c0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.479+0000 7f238b400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f238c160f10 0x7f238c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.479+0000 7f238be00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f238c161590 0x7f238c0b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:38654/0 (socket says 172.21.15.89:38654) 2024-04-02T07:35:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.479+0000 7f238be00700 1 -- 172.21.15.89:0/315341562 learned_addr learned my addr 172.21.15.89:0/315341562 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.479+0000 7f238b400700 1 -- 172.21.15.89:0/315341562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f238c0cf230 msgr2=0x7f238c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:53.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.479+0000 7f238b400700 1 --2- 172.21.15.89:0/315341562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f238c0cf230 0x7f238c0d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.479+0000 7f238b400700 1 -- 172.21.15.89:0/315341562 >> v1:172.21.15.113:6789/0 conn(0x7f238c161590 legacy=0x7f238c0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:53.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.479+0000 7f238b400700 1 -- 172.21.15.89:0/315341562 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c0ce470 con 0x7f238c160f10 2024-04-02T07:35:53.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.479+0000 7f2383e00700 1 --2- 172.21.15.89:0/315341562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f238c0cf230 0x7f238c0d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:53.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.479+0000 7f238b400700 1 --2- 172.21.15.89:0/315341562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f238c160f10 0x7f238c0d18b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f237c009840 tx=0x7f237c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b0deda6d2a43936 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:53.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.480+0000 7f238aa00700 1 -- 172.21.15.89:0/315341562 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c011830 con 0x7f238c160f10 2024-04-02T07:35:53.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.480+0000 7f238aa00700 1 -- 172.21.15.89:0/315341562 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237c011990 con 0x7f238c160f10 2024-04-02T07:35:53.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.480+0000 7f238aa00700 1 -- 172.21.15.89:0/315341562 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c011d20 con 0x7f238c160f10 2024-04-02T07:35:53.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.480+0000 7f2392800700 1 -- 172.21.15.89:0/315341562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f238c160f10 msgr2=0x7f238c0d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:53.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.480+0000 7f2392800700 1 --2- 172.21.15.89:0/315341562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f238c160f10 0x7f238c0d18b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f237c009840 tx=0x7f237c009b50 comp rx=0 tx=0).stop 2024-04-02T07:35:53.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.480+0000 7f2392800700 1 -- 172.21.15.89:0/315341562 shutdown_connections 2024-04-02T07:35:53.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.480+0000 7f2392800700 1 --2- 172.21.15.89:0/315341562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f238c160f10 0x7f238c0d18b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.480+0000 7f2392800700 1 --2- 172.21.15.89:0/315341562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f238c0cf230 0x7f238c0d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.480+0000 7f2392800700 1 -- 172.21.15.89:0/315341562 >> 172.21.15.89:0/315341562 conn(0x7f238c08aff0 msgr2=0x7f238c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:53.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.480+0000 7f2392800700 1 -- 172.21.15.89:0/315341562 shutdown_connections 2024-04-02T07:35:53.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.480+0000 7f2392800700 1 -- 172.21.15.89:0/315341562 wait complete. 2024-04-02T07:35:53.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.482+0000 7f2392800700 1 Processor -- start 2024-04-02T07:35:53.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.482+0000 7f2392800700 1 -- start start 2024-04-02T07:35:53.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.483+0000 7f2392800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f238c0cf230 0x7f238c0b8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:53.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.483+0000 7f2392800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c160f10 0x7f238c0b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:53.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.483+0000 7f2392800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f238c161590 0x7f238c0b76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:53.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.483+0000 7f2392800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0b8b60 con 0x7f238c160f10 2024-04-02T07:35:53.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.483+0000 7f2392800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c157900 con 0x7f238c0cf230 2024-04-02T07:35:53.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.483+0000 7f2392800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238c156470 con 0x7f238c161590 2024-04-02T07:35:53.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.483+0000 7f2383e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c160f10 0x7f238c0b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:53.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.483+0000 7f2383e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c160f10 0x7f238c0b7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57808/0 (socket says 172.21.15.89:57808) 2024-04-02T07:35:53.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.483+0000 7f2383e00700 1 -- 172.21.15.89:0/2244731167 learned_addr learned my addr 172.21.15.89:0/2244731167 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:53.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.483+0000 7f238b400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f238c0cf230 0x7f238c0b8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:53.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.484+0000 7f238be00700 1 --2- 172.21.15.89:0/2244731167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f238c161590 0x7f238c0b76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:53.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.484+0000 7f238b400700 1 -- 172.21.15.89:0/2244731167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f238c161590 msgr2=0x7f238c0b76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:53.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.484+0000 7f238b400700 1 --2- 172.21.15.89:0/2244731167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f238c161590 0x7f238c0b76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.484+0000 7f238b400700 1 -- 172.21.15.89:0/2244731167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c160f10 msgr2=0x7f238c0b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:53.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.484+0000 7f238b400700 1 --2- 172.21.15.89:0/2244731167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c160f10 0x7f238c0b7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.484+0000 7f238b400700 1 -- 172.21.15.89:0/2244731167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23740094a0 con 0x7f238c0cf230 2024-04-02T07:35:53.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.484+0000 7f2383e00700 1 --2- 172.21.15.89:0/2244731167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c160f10 0x7f238c0b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:53.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.484+0000 7f238b400700 1 --2- 172.21.15.89:0/2244731167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f238c0cf230 0x7f238c0b8620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f237c016000 tx=0x7f237c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:53.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.484+0000 7f2383400700 1 -- 172.21.15.89:0/2244731167 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c0036a0 con 0x7f238c0cf230 2024-04-02T07:35:53.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.484+0000 7f2383400700 1 -- 172.21.15.89:0/2244731167 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237c01e070 con 0x7f238c0cf230 2024-04-02T07:35:53.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.484+0000 7f2383400700 1 -- 172.21.15.89:0/2244731167 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c003a20 con 0x7f238c0cf230 2024-04-02T07:35:53.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.484+0000 7f2392800700 1 -- 172.21.15.89:0/2244731167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237c0094a0 con 0x7f238c0cf230 2024-04-02T07:35:53.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.485+0000 7f2392800700 1 -- 172.21.15.89:0/2244731167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c1572b0 con 0x7f238c0cf230 2024-04-02T07:35:53.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.485+0000 7f2392800700 1 -- 172.21.15.89:0/2244731167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c067720 con 0x7f238c0cf230 2024-04-02T07:35:53.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.487+0000 7f2383400700 1 -- 172.21.15.89:0/2244731167 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f237c003b80 con 0x7f238c0cf230 2024-04-02T07:35:53.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.487+0000 7f2383400700 1 --2- 172.21.15.89:0/2244731167 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f236806ac90 0x7f236806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:53.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.487+0000 7f2383400700 1 -- 172.21.15.89:0/2244731167 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f237c08a790 con 0x7f238c0cf230 2024-04-02T07:35:53.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.488+0000 7f2383e00700 1 --2- 172.21.15.89:0/2244731167 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f236806ac90 0x7f236806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:53.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.488+0000 7f2383e00700 1 --2- 172.21.15.89:0/2244731167 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f236806ac90 0x7f236806d150 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f238c1f6880 tx=0x7f2374009210 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:53.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.492+0000 7f2383400700 1 -- 172.21.15.89:0/2244731167 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f237c055960 con 0x7f238c0cf230 2024-04-02T07:35:53.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.695+0000 7f2392800700 1 -- 172.21.15.89:0/2244731167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f238c06c160 con 0x7f238c0cf230 2024-04-02T07:35:53.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.696+0000 7f2383400700 1 -- 172.21.15.89:0/2244731167 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f237c020090 con 0x7f238c0cf230 2024-04-02T07:35:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:35:53.699 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":19,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:53.234652+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:35:53.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.699+0000 7f2392800700 1 -- 172.21.15.89:0/2244731167 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f236806ac90 msgr2=0x7f236806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:53.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.699+0000 7f2392800700 1 --2- 172.21.15.89:0/2244731167 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f236806ac90 0x7f236806d150 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f238c1f6880 tx=0x7f2374009210 comp rx=0 tx=0).stop 2024-04-02T07:35:53.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.699+0000 7f2392800700 1 -- 172.21.15.89:0/2244731167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f238c0cf230 msgr2=0x7f238c0b8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:53.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.699+0000 7f2392800700 1 --2- 172.21.15.89:0/2244731167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f238c0cf230 0x7f238c0b8620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f237c016000 tx=0x7f237c004750 comp rx=0 tx=0).stop 2024-04-02T07:35:53.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.699+0000 7f2392800700 1 -- 172.21.15.89:0/2244731167 shutdown_connections 2024-04-02T07:35:53.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.699+0000 7f2392800700 1 --2- 172.21.15.89:0/2244731167 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f236806ac90 0x7f236806d150 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.699+0000 7f2392800700 1 --2- 172.21.15.89:0/2244731167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f238c0cf230 0x7f238c0b8620 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.699+0000 7f2392800700 1 --2- 172.21.15.89:0/2244731167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f238c161590 0x7f238c0b76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.699+0000 7f2392800700 1 --2- 172.21.15.89:0/2244731167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c160f10 0x7f238c0b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.699+0000 7f2392800700 1 -- 172.21.15.89:0/2244731167 >> 172.21.15.89:0/2244731167 conn(0x7f238c08aff0 msgr2=0x7f238c0caf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:53.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.699+0000 7f2392800700 1 -- 172.21.15.89:0/2244731167 shutdown_connections 2024-04-02T07:35:53.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.699+0000 7f2392800700 1 -- 172.21.15.89:0/2244731167 wait complete. 2024-04-02T07:35:53.719 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:35:53.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.909+0000 7f7613200700 1 Processor -- start 2024-04-02T07:35:53.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.909+0000 7f7613200700 1 -- start start 2024-04-02T07:35:53.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.910+0000 7f7613200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f760c160f10 0x7f760c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:53.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.910+0000 7f7613200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f760c161590 0x7f760c161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:53.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.910+0000 7f7613200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f760c0ceda0 con 0x7f760c0cf230 2024-04-02T07:35:53.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.910+0000 7f7613200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f760c161f80 con 0x7f760c160f10 2024-04-02T07:35:53.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.910+0000 7f7613200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f760c1612f0 con 0x7f760c161590 2024-04-02T07:35:53.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.911+0000 7f760be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f760c160f10 0x7f760c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:53.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.911+0000 7f760be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f760c160f10 0x7f760c0d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57812/0 (socket says 172.21.15.89:57812) 2024-04-02T07:35:53.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.911+0000 7f760be00700 1 -- 172.21.15.89:0/3084226854 learned_addr learned my addr 172.21.15.89:0/3084226854 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:53.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.911+0000 7f7611200700 1 --2- 172.21.15.89:0/3084226854 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f760c161590 0x7f760c161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:53.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.911+0000 7f760be00700 1 -- 172.21.15.89:0/3084226854 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f760c161590 msgr2=0x7f760c161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:53.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.911+0000 7f760be00700 1 --2- 172.21.15.89:0/3084226854 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f760c161590 0x7f760c161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.911+0000 7f760be00700 1 -- 172.21.15.89:0/3084226854 >> v1:172.21.15.113:6789/0 conn(0x7f760c0cf230 legacy=0x7f760c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:53.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.911+0000 7f760be00700 1 -- 172.21.15.89:0/3084226854 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760c0ce470 con 0x7f760c160f10 2024-04-02T07:35:53.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.911+0000 7f760be00700 1 --2- 172.21.15.89:0/3084226854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c160f10 0x7f760c0d18b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f75fc0098f0 tx=0x7f75fc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c194b5670f3666 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:53.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.912+0000 7f760aa00700 1 -- 172.21.15.89:0/3084226854 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc00b8a0 con 0x7f760c160f10 2024-04-02T07:35:53.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.912+0000 7f760aa00700 1 -- 172.21.15.89:0/3084226854 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75fc00ba00 con 0x7f760c160f10 2024-04-02T07:35:53.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.912+0000 7f760aa00700 1 -- 172.21.15.89:0/3084226854 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc00bd90 con 0x7f760c160f10 2024-04-02T07:35:53.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.912+0000 7f7613200700 1 -- 172.21.15.89:0/3084226854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c160f10 msgr2=0x7f760c0d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:53.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.912+0000 7f7613200700 1 --2- 172.21.15.89:0/3084226854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c160f10 0x7f760c0d18b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f75fc0098f0 tx=0x7f75fc009c00 comp rx=0 tx=0).stop 2024-04-02T07:35:53.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.912+0000 7f7613200700 1 -- 172.21.15.89:0/3084226854 shutdown_connections 2024-04-02T07:35:53.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.912+0000 7f7613200700 1 --2- 172.21.15.89:0/3084226854 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f760c161590 0x7f760c161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.912+0000 7f7613200700 1 --2- 172.21.15.89:0/3084226854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c160f10 0x7f760c0d18b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.912+0000 7f7613200700 1 -- 172.21.15.89:0/3084226854 >> 172.21.15.89:0/3084226854 conn(0x7f760c08aff0 msgr2=0x7f760c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:53.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.912+0000 7f7613200700 1 -- 172.21.15.89:0/3084226854 shutdown_connections 2024-04-02T07:35:53.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.912+0000 7f7613200700 1 -- 172.21.15.89:0/3084226854 wait complete. 2024-04-02T07:35:53.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.914+0000 7f7613200700 1 Processor -- start 2024-04-02T07:35:53.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.914+0000 7f7613200700 1 -- start start 2024-04-02T07:35:53.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.915+0000 7f7613200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f760c0cf230 0x7f760c155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:53.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.915+0000 7f7613200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f760c160f10 0x7f760c156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:53.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.915+0000 7f7613200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c161590 0x7f760c158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:53.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.915+0000 7f7613200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c157dd0 con 0x7f760c161590 2024-04-02T07:35:53.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.915+0000 7f7613200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c157150 con 0x7f760c160f10 2024-04-02T07:35:53.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.915+0000 7f7613200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f760c157430 con 0x7f760c0cf230 2024-04-02T07:35:53.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.915+0000 7f7611200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c161590 0x7f760c158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:53.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.915+0000 7f7611200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c161590 0x7f760c158240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57826/0 (socket says 172.21.15.89:57826) 2024-04-02T07:35:53.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.915+0000 7f7611200700 1 -- 172.21.15.89:0/3162014499 learned_addr learned my addr 172.21.15.89:0/3162014499 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:53.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.915+0000 7f760b400700 1 --2- 172.21.15.89:0/3162014499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f760c160f10 0x7f760c156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:53.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.915+0000 7f760be00700 1 --2- 172.21.15.89:0/3162014499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f760c0cf230 0x7f760c155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:53.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.915+0000 7f7611200700 1 -- 172.21.15.89:0/3162014499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f760c0cf230 msgr2=0x7f760c155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:53.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.915+0000 7f7611200700 1 --2- 172.21.15.89:0/3162014499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f760c0cf230 0x7f760c155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.916+0000 7f7611200700 1 -- 172.21.15.89:0/3162014499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f760c160f10 msgr2=0x7f760c156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:53.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.916+0000 7f7611200700 1 --2- 172.21.15.89:0/3162014499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f760c160f10 0x7f760c156650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:53.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.916+0000 7f7611200700 1 -- 172.21.15.89:0/3162014499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75fc009550 con 0x7f760c161590 2024-04-02T07:35:53.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.916+0000 7f7611200700 1 --2- 172.21.15.89:0/3162014499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c161590 0x7f760c158240 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7604007e60 tx=0x7f760400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:53.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.916+0000 7f7603e00700 1 -- 172.21.15.89:0/3162014499 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760400ed30 con 0x7f760c161590 2024-04-02T07:35:53.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.916+0000 7f7603e00700 1 -- 172.21.15.89:0/3162014499 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7604014420 con 0x7f760c161590 2024-04-02T07:35:53.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.916+0000 7f7603e00700 1 -- 172.21.15.89:0/3162014499 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76040147d0 con 0x7f760c161590 2024-04-02T07:35:53.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.916+0000 7f7613200700 1 -- 172.21.15.89:0/3162014499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760c157710 con 0x7f760c161590 2024-04-02T07:35:53.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.916+0000 7f7613200700 1 -- 172.21.15.89:0/3162014499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f760c15fcf0 con 0x7f760c161590 2024-04-02T07:35:53.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.917+0000 7f7613200700 1 -- 172.21.15.89:0/3162014499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760c067720 con 0x7f760c161590 2024-04-02T07:35:53.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.918+0000 7f7603e00700 1 -- 172.21.15.89:0/3162014499 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7604014930 con 0x7f760c161590 2024-04-02T07:35:53.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.919+0000 7f7603e00700 1 --2- 172.21.15.89:0/3162014499 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f75ec06ac90 0x7f75ec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:53.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.919+0000 7f7603e00700 1 -- 172.21.15.89:0/3162014499 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76040899e0 con 0x7f760c161590 2024-04-02T07:35:53.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.919+0000 7f760be00700 1 --2- 172.21.15.89:0/3162014499 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f75ec06ac90 0x7f75ec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:53.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.920+0000 7f760be00700 1 --2- 172.21.15.89:0/3162014499 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f75ec06ac90 0x7f75ec06d150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f75fc014040 tx=0x7f75fc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:53.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:53.924+0000 7f7603e00700 1 -- 172.21.15.89:0/3162014499 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7604054bd0 con 0x7f760c161590 2024-04-02T07:35:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.129+0000 7f7613200700 1 -- 172.21.15.89:0/3162014499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f760c155fd0 con 0x7f760c161590 2024-04-02T07:35:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.130+0000 7f7603e00700 1 -- 172.21.15.89:0/3162014499 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f76040581f0 con 0x7f760c161590 2024-04-02T07:35:54.132 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:35:54.132 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":19,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:53.234652+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:35:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.132+0000 7f7613200700 1 -- 172.21.15.89:0/3162014499 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f75ec06ac90 msgr2=0x7f75ec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.132+0000 7f7613200700 1 --2- 172.21.15.89:0/3162014499 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f75ec06ac90 0x7f75ec06d150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f75fc014040 tx=0x7f75fc008030 comp rx=0 tx=0).stop 2024-04-02T07:35:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.132+0000 7f7613200700 1 -- 172.21.15.89:0/3162014499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c161590 msgr2=0x7f760c158240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.132+0000 7f7613200700 1 --2- 172.21.15.89:0/3162014499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c161590 0x7f760c158240 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7604007e60 tx=0x7f760400e3c0 comp rx=0 tx=0).stop 2024-04-02T07:35:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.132+0000 7f7613200700 1 -- 172.21.15.89:0/3162014499 shutdown_connections 2024-04-02T07:35:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.132+0000 7f7613200700 1 --2- 172.21.15.89:0/3162014499 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f75ec06ac90 0x7f75ec06d150 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.132+0000 7f7613200700 1 --2- 172.21.15.89:0/3162014499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f760c161590 0x7f760c158240 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.132+0000 7f7613200700 1 --2- 172.21.15.89:0/3162014499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f760c0cf230 0x7f760c155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.132+0000 7f7613200700 1 --2- 172.21.15.89:0/3162014499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f760c160f10 0x7f760c156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.132+0000 7f7613200700 1 -- 172.21.15.89:0/3162014499 >> 172.21.15.89:0/3162014499 conn(0x7f760c08aff0 msgr2=0x7f760c0ca9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.132+0000 7f7613200700 1 -- 172.21.15.89:0/3162014499 shutdown_connections 2024-04-02T07:35:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.133+0000 7f7613200700 1 -- 172.21.15.89:0/3162014499 wait complete. 2024-04-02T07:35:54.152 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T07:35:54.327 INFO:tasks.ceph.mds.b.smithi113.stderr:2024-04-02T07:35:54.325+0000 7f02a4200700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T07:35:54.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.347+0000 7f62aa400700 1 Processor -- start 2024-04-02T07:35:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.347+0000 7f62aa400700 1 -- start start 2024-04-02T07:35:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.348+0000 7f62aa400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f62a40d16a0 0x7f62a40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.348+0000 7f62aa400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62a40d2370 0x7f62a40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.348+0000 7f62aa400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a4161420 con 0x7f62a415d9d0 2024-04-02T07:35:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.348+0000 7f62aa400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40d2820 con 0x7f62a40d16a0 2024-04-02T07:35:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.348+0000 7f62aa400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40d4340 con 0x7f62a40d2370 2024-04-02T07:35:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.349+0000 7f62a2a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f62a40d16a0 0x7f62a40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.349+0000 7f62a3400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f62a415d9d0 0x7f62a40d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:38668/0 (socket says 172.21.15.89:38668) 2024-04-02T07:35:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.349+0000 7f62a3400700 1 -- 172.21.15.89:0/4252981500 learned_addr learned my addr 172.21.15.89:0/4252981500 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.349+0000 7f62a3e00700 1 --2- 172.21.15.89:0/4252981500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62a40d2370 0x7f62a40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.349+0000 7f62a2000700 1 -- 172.21.15.89:0/4252981500 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258093153 0 0) 0x7f62a4161420 con 0x7f62a415d9d0 2024-04-02T07:35:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.349+0000 7f62a2000700 1 -- 172.21.15.89:0/4252981500 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62880035b0 con 0x7f62a415d9d0 2024-04-02T07:35:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.349+0000 7f62a3e00700 1 -- 172.21.15.89:0/4252981500 >> v2:172.21.15.113:3300/0 conn(0x7f62a40d16a0 msgr2=0x7f62a40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.349+0000 7f62a3e00700 1 --2- 172.21.15.89:0/4252981500 >> v2:172.21.15.113:3300/0 conn(0x7f62a40d16a0 0x7f62a40d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.349+0000 7f62a3e00700 1 -- 172.21.15.89:0/4252981500 >> v1:172.21.15.113:6789/0 conn(0x7f62a415d9d0 legacy=0x7f62a40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.349+0000 7f62a3e00700 1 -- 172.21.15.89:0/4252981500 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a4161700 con 0x7f62a40d2370 2024-04-02T07:35:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.349+0000 7f62a2a00700 1 --2- 172.21.15.89:0/4252981500 >> v2:172.21.15.113:3300/0 conn(0x7f62a40d16a0 0x7f62a40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.350+0000 7f62a3e00700 1 --2- 172.21.15.89:0/4252981500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62a40d2370 0x7f62a40ce360 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f629c009d90 tx=0x7f629c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b87ec3a932c715ef server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.350+0000 7f62a2000700 1 -- 172.21.15.89:0/4252981500 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c00e070 con 0x7f62a40d2370 2024-04-02T07:35:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.350+0000 7f62a2000700 1 -- 172.21.15.89:0/4252981500 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629c00bb20 con 0x7f62a40d2370 2024-04-02T07:35:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.350+0000 7f62a2000700 1 -- 172.21.15.89:0/4252981500 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c00be90 con 0x7f62a40d2370 2024-04-02T07:35:54.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.350+0000 7f62aa400700 1 -- 172.21.15.89:0/4252981500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62a40d2370 msgr2=0x7f62a40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:54.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.350+0000 7f62aa400700 1 --2- 172.21.15.89:0/4252981500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62a40d2370 0x7f62a40ce360 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f629c009d90 tx=0x7f629c00b330 comp rx=0 tx=0).stop 2024-04-02T07:35:54.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.351+0000 7f62aa400700 1 -- 172.21.15.89:0/4252981500 shutdown_connections 2024-04-02T07:35:54.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.351+0000 7f62aa400700 1 --2- 172.21.15.89:0/4252981500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62a40d2370 0x7f62a40ce360 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:54.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.351+0000 7f62aa400700 1 --2- 172.21.15.89:0/4252981500 >> v2:172.21.15.113:3300/0 conn(0x7f62a40d16a0 0x7f62a40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:54.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.351+0000 7f62aa400700 1 -- 172.21.15.89:0/4252981500 >> 172.21.15.89:0/4252981500 conn(0x7f62a408aff0 msgr2=0x7f62a40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:54.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.351+0000 7f62aa400700 1 -- 172.21.15.89:0/4252981500 shutdown_connections 2024-04-02T07:35:54.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.351+0000 7f62aa400700 1 -- 172.21.15.89:0/4252981500 wait complete. 2024-04-02T07:35:54.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.352+0000 7f62aa400700 1 Processor -- start 2024-04-02T07:35:54.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.352+0000 7f62aa400700 1 -- start start 2024-04-02T07:35:54.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.353+0000 7f62aa400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62a40d16a0 0x7f62a41f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:54.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.353+0000 7f62aa400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40d2370 0x7f62a40cb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:54.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.353+0000 7f62aa400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62a415d9d0 0x7f62a40cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:54.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.353+0000 7f62aa400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40cbf80 con 0x7f62a40d2370 2024-04-02T07:35:54.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.353+0000 7f62aa400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a41e8f10 con 0x7f62a415d9d0 2024-04-02T07:35:54.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.353+0000 7f62aa400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40b6150 con 0x7f62a40d16a0 2024-04-02T07:35:54.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f62a2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40d2370 0x7f62a40cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:54.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f62a2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40d2370 0x7f62a40cb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57828/0 (socket says 172.21.15.89:57828) 2024-04-02T07:35:54.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f62a2a00700 1 -- 172.21.15.89:0/2273009648 learned_addr learned my addr 172.21.15.89:0/2273009648 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:54.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f62a3400700 1 --2- 172.21.15.89:0/2273009648 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62a40d16a0 0x7f62a41f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:54.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f62a3e00700 1 --2- 172.21.15.89:0/2273009648 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62a415d9d0 0x7f62a40cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:54.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f62a2a00700 1 -- 172.21.15.89:0/2273009648 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62a40d16a0 msgr2=0x7f62a41f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:54.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f62a2a00700 1 --2- 172.21.15.89:0/2273009648 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62a40d16a0 0x7f62a41f2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:54.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f62a2a00700 1 -- 172.21.15.89:0/2273009648 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62a415d9d0 msgr2=0x7f62a40cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:54.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f62a2a00700 1 --2- 172.21.15.89:0/2273009648 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62a415d9d0 0x7f62a40cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:54.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f62a2a00700 1 -- 172.21.15.89:0/2273009648 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6290009430 con 0x7f62a40d2370 2024-04-02T07:35:54.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f62a2a00700 1 --2- 172.21.15.89:0/2273009648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40d2370 0x7f62a40cb500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f629000ebe0 tx=0x7f629000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:54.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f629b400700 1 -- 172.21.15.89:0/2273009648 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290009780 con 0x7f62a40d2370 2024-04-02T07:35:54.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f629b400700 1 -- 172.21.15.89:0/2273009648 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6290005a20 con 0x7f62a40d2370 2024-04-02T07:35:54.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f629b400700 1 -- 172.21.15.89:0/2273009648 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290005dd0 con 0x7f62a40d2370 2024-04-02T07:35:54.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.354+0000 7f62aa400700 1 -- 172.21.15.89:0/2273009648 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629c009a70 con 0x7f62a40d2370 2024-04-02T07:35:54.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.355+0000 7f62aa400700 1 -- 172.21.15.89:0/2273009648 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a40b6710 con 0x7f62a40d2370 2024-04-02T07:35:54.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.356+0000 7f62aa400700 1 -- 172.21.15.89:0/2273009648 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a40d1ca0 con 0x7f62a40d2370 2024-04-02T07:35:54.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.356+0000 7f629b400700 1 -- 172.21.15.89:0/2273009648 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f629000c850 con 0x7f62a40d2370 2024-04-02T07:35:54.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.357+0000 7f629b400700 1 --2- 172.21.15.89:0/2273009648 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f628406ac90 0x7f628406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:54.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.357+0000 7f629b400700 1 -- 172.21.15.89:0/2273009648 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6290089520 con 0x7f62a40d2370 2024-04-02T07:35:54.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.358+0000 7f62a3400700 1 --2- 172.21.15.89:0/2273009648 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f628406ac90 0x7f628406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:54.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.358+0000 7f62a3400700 1 --2- 172.21.15.89:0/2273009648 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f628406ac90 0x7f628406d150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6294007fd0 tx=0x7f6294009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:54.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.362+0000 7f629b400700 1 -- 172.21.15.89:0/2273009648 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f62900546f0 con 0x7f62a40d2370 2024-04-02T07:35:54.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:54.608+0000 7f62aa400700 1 -- 172.21.15.89:0/2273009648 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f62a4067720 con 0x7f62a40d2370 2024-04-02T07:35:55.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.246+0000 7f629b400700 1 -- 172.21.15.89:0/2273009648 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6290057d10 con 0x7f62a40d2370 2024-04-02T07:35:55.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.249+0000 7f62aa400700 1 -- 172.21.15.89:0/2273009648 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f628406ac90 msgr2=0x7f628406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:55.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.249+0000 7f62aa400700 1 --2- 172.21.15.89:0/2273009648 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f628406ac90 0x7f628406d150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6294007fd0 tx=0x7f6294009040 comp rx=0 tx=0).stop 2024-04-02T07:35:55.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.249+0000 7f62aa400700 1 -- 172.21.15.89:0/2273009648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40d2370 msgr2=0x7f62a40cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:55.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.249+0000 7f62aa400700 1 --2- 172.21.15.89:0/2273009648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40d2370 0x7f62a40cb500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f629000ebe0 tx=0x7f629000eef0 comp rx=0 tx=0).stop 2024-04-02T07:35:55.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.249+0000 7f62aa400700 1 -- 172.21.15.89:0/2273009648 shutdown_connections 2024-04-02T07:35:55.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.249+0000 7f62aa400700 1 --2- 172.21.15.89:0/2273009648 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f628406ac90 0x7f628406d150 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:55.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.249+0000 7f62aa400700 1 --2- 172.21.15.89:0/2273009648 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62a415d9d0 0x7f62a40cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:55.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.249+0000 7f62aa400700 1 --2- 172.21.15.89:0/2273009648 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62a40d16a0 0x7f62a41f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:55.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.249+0000 7f62aa400700 1 --2- 172.21.15.89:0/2273009648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40d2370 0x7f62a40cb500 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:55.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.249+0000 7f62aa400700 1 -- 172.21.15.89:0/2273009648 >> 172.21.15.89:0/2273009648 conn(0x7f62a408aff0 msgr2=0x7f62a406d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:55.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.250+0000 7f62aa400700 1 -- 172.21.15.89:0/2273009648 shutdown_connections 2024-04-02T07:35:55.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.250+0000 7f62aa400700 1 -- 172.21.15.89:0/2273009648 wait complete. 2024-04-02T07:35:55.271 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T07:35:55.271 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T07:35:55.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.466+0000 7f594a600700 1 Processor -- start 2024-04-02T07:35:55.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.466+0000 7f594a600700 1 -- start start 2024-04-02T07:35:55.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.467+0000 7f594a600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f594415d9d0 0x7f59440d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:55.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.467+0000 7f594a600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5944161420 con 0x7f59440d2540 2024-04-02T07:35:55.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.467+0000 7f594a600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5944161700 con 0x7f59440d16a0 2024-04-02T07:35:55.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.467+0000 7f594a600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594415edc0 con 0x7f594415d9d0 2024-04-02T07:35:55.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.467+0000 7f5943400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f594415d9d0 0x7f59440d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:55.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.467+0000 7f5943400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f594415d9d0 0x7f59440d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57844/0 (socket says 172.21.15.89:57844) 2024-04-02T07:35:55.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.467+0000 7f5943400700 1 -- 172.21.15.89:0/2018809557 learned_addr learned my addr 172.21.15.89:0/2018809557 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:55.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.467+0000 7f5943e00700 1 --1- 172.21.15.89:0/2018809557 >> v1:172.21.15.89:6789/0 conn(0x7f59440d2540 0x7f59440ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36904/0 (socket says 172.21.15.89:36904) 2024-04-02T07:35:55.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.468+0000 7f5943400700 1 -- 172.21.15.89:0/2018809557 >> v1:172.21.15.113:6789/0 conn(0x7f59440d16a0 legacy=0x7f59440d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:55.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.468+0000 7f5943400700 1 -- 172.21.15.89:0/2018809557 >> v1:172.21.15.89:6789/0 conn(0x7f59440d2540 legacy=0x7f59440ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.468+0000 7f5943400700 1 -- 172.21.15.89:0/2018809557 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59441619e0 con 0x7f594415d9d0 2024-04-02T07:35:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.468+0000 7f5942000700 1 -- 172.21.15.89:0/2018809557 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042543475 0 0) 0x7f5944161420 con 0x7f59440d2540 2024-04-02T07:35:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.468+0000 7f5943400700 1 --2- 172.21.15.89:0/2018809557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f594415d9d0 0x7f59440d3c30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f59340098f0 tx=0x7f5934009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b9e244d088341a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.468+0000 7f5942000700 1 -- 172.21.15.89:0/2018809557 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593400b8a0 con 0x7f594415d9d0 2024-04-02T07:35:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.468+0000 7f5942000700 1 -- 172.21.15.89:0/2018809557 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593400ba00 con 0x7f594415d9d0 2024-04-02T07:35:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.468+0000 7f5942000700 1 -- 172.21.15.89:0/2018809557 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593400bd90 con 0x7f594415d9d0 2024-04-02T07:35:55.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.469+0000 7f594a600700 1 -- 172.21.15.89:0/2018809557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f594415d9d0 msgr2=0x7f59440d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:55.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.469+0000 7f594a600700 1 --2- 172.21.15.89:0/2018809557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f594415d9d0 0x7f59440d3c30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f59340098f0 tx=0x7f5934009c00 comp rx=0 tx=0).stop 2024-04-02T07:35:55.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.469+0000 7f594a600700 1 -- 172.21.15.89:0/2018809557 shutdown_connections 2024-04-02T07:35:55.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.469+0000 7f594a600700 1 --2- 172.21.15.89:0/2018809557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f594415d9d0 0x7f59440d3c30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:55.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.469+0000 7f594a600700 1 -- 172.21.15.89:0/2018809557 >> 172.21.15.89:0/2018809557 conn(0x7f594408aff0 msgr2=0x7f59440d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:55.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.469+0000 7f594a600700 1 -- 172.21.15.89:0/2018809557 shutdown_connections 2024-04-02T07:35:55.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.469+0000 7f594a600700 1 -- 172.21.15.89:0/2018809557 wait complete. 2024-04-02T07:35:55.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.471+0000 7f594a600700 1 Processor -- start 2024-04-02T07:35:55.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.471+0000 7f594a600700 1 -- start start 2024-04-02T07:35:55.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f594a600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59440d16a0 0x7f59440caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:55.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f594a600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59440d2540 0x7f59440b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:55.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f594a600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f594415d9d0 0x7f59440b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:55.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f594a600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59441e8270 con 0x7f594415d9d0 2024-04-02T07:35:55.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f594a600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59440caff0 con 0x7f59440d16a0 2024-04-02T07:35:55.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f594a600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59441e7660 con 0x7f59440d2540 2024-04-02T07:35:55.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f5943e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f594415d9d0 0x7f59440b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:55.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f5943e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f594415d9d0 0x7f59440b85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57860/0 (socket says 172.21.15.89:57860) 2024-04-02T07:35:55.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f5943e00700 1 -- 172.21.15.89:0/1468310947 learned_addr learned my addr 172.21.15.89:0/1468310947 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:55.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f5943400700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59440d16a0 0x7f59440caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:55.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f5942a00700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59440d2540 0x7f59440b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:55.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f5943e00700 1 -- 172.21.15.89:0/1468310947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59440d2540 msgr2=0x7f59440b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:55.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f5943e00700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59440d2540 0x7f59440b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:55.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f5943e00700 1 -- 172.21.15.89:0/1468310947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59440d16a0 msgr2=0x7f59440caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:55.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f5943e00700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59440d16a0 0x7f59440caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:55.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f5943e00700 1 -- 172.21.15.89:0/1468310947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5934009550 con 0x7f594415d9d0 2024-04-02T07:35:55.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.472+0000 7f5942a00700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59440d2540 0x7f59440b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:35:55.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.473+0000 7f5943e00700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f594415d9d0 0x7f59440b85a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f593c00cc70 tx=0x7f593c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:55.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.473+0000 7f593b400700 1 -- 172.21.15.89:0/1468310947 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c00ab50 con 0x7f594415d9d0 2024-04-02T07:35:55.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.473+0000 7f593b400700 1 -- 172.21.15.89:0/1468310947 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c00acd0 con 0x7f594415d9d0 2024-04-02T07:35:55.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.473+0000 7f594a600700 1 -- 172.21.15.89:0/1468310947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59441e7940 con 0x7f594415d9d0 2024-04-02T07:35:55.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.473+0000 7f593b400700 1 -- 172.21.15.89:0/1468310947 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c007310 con 0x7f594415d9d0 2024-04-02T07:35:55.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.473+0000 7f594a600700 1 -- 172.21.15.89:0/1468310947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59441e7f00 con 0x7f594415d9d0 2024-04-02T07:35:55.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.474+0000 7f594a600700 1 -- 172.21.15.89:0/1468310947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59440d1ca0 con 0x7f594415d9d0 2024-04-02T07:35:55.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.475+0000 7f593b400700 1 -- 172.21.15.89:0/1468310947 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f593c007470 con 0x7f594415d9d0 2024-04-02T07:35:55.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.476+0000 7f593b400700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f592406ac90 0x7f592406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:55.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.476+0000 7f593b400700 1 -- 172.21.15.89:0/1468310947 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f593c0891d0 con 0x7f594415d9d0 2024-04-02T07:35:55.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.476+0000 7f5943400700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f592406ac90 0x7f592406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:55.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.476+0000 7f5943400700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f592406ac90 0x7f592406d150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5934014040 tx=0x7f593401c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:55.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.480+0000 7f593b400700 1 -- 172.21.15.89:0/1468310947 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f593c054310 con 0x7f594415d9d0 2024-04-02T07:35:55.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:55.725+0000 7f594a600700 1 -- 172.21.15.89:0/1468310947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f59440d3c30 con 0x7f594415d9d0 2024-04-02T07:35:56.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.256+0000 7f593b400700 1 -- 172.21.15.89:0/1468310947 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f593c057930 con 0x7f594415d9d0 2024-04-02T07:35:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.259+0000 7f594a600700 1 -- 172.21.15.89:0/1468310947 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f592406ac90 msgr2=0x7f592406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.259+0000 7f594a600700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f592406ac90 0x7f592406d150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5934014040 tx=0x7f593401c040 comp rx=0 tx=0).stop 2024-04-02T07:35:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.259+0000 7f594a600700 1 -- 172.21.15.89:0/1468310947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f594415d9d0 msgr2=0x7f59440b85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.259+0000 7f594a600700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f594415d9d0 0x7f59440b85a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f593c00cc70 tx=0x7f593c00cf80 comp rx=0 tx=0).stop 2024-04-02T07:35:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.259+0000 7f594a600700 1 -- 172.21.15.89:0/1468310947 shutdown_connections 2024-04-02T07:35:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.259+0000 7f594a600700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f592406ac90 0x7f592406d150 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.259+0000 7f594a600700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59440d16a0 0x7f59440caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.259+0000 7f594a600700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f594415d9d0 0x7f59440b85a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.259+0000 7f594a600700 1 --2- 172.21.15.89:0/1468310947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59440d2540 0x7f59440b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.259+0000 7f594a600700 1 -- 172.21.15.89:0/1468310947 >> 172.21.15.89:0/1468310947 conn(0x7f594408aff0 msgr2=0x7f59440ceca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:56.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.259+0000 7f594a600700 1 -- 172.21.15.89:0/1468310947 shutdown_connections 2024-04-02T07:35:56.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.259+0000 7f594a600700 1 -- 172.21.15.89:0/1468310947 wait complete. 2024-04-02T07:35:56.280 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T07:35:56.280 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:35:56.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.470+0000 7f314aa00700 1 Processor -- start 2024-04-02T07:35:56.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.470+0000 7f314aa00700 1 -- start start 2024-04-02T07:35:56.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.471+0000 7f314aa00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f31440d18b0 0x7f31440cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:56.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.471+0000 7f314aa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3144161590 0x7f3144161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:56.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.471+0000 7f314aa00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31440ceda0 con 0x7f3144160f10 2024-04-02T07:35:56.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.471+0000 7f314aa00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31440d1d30 con 0x7f31440d18b0 2024-04-02T07:35:56.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.471+0000 7f314aa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3144161f60 con 0x7f3144161590 2024-04-02T07:35:56.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.472+0000 7f3148a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3144161590 0x7f3144161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:56.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.472+0000 7f3148a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3144161590 0x7f3144161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:57444/0 (socket says 172.21.15.89:57444) 2024-04-02T07:35:56.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.472+0000 7f3143400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f31440d18b0 0x7f31440cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:56.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.472+0000 7f3148a00700 1 -- 172.21.15.89:0/1656153136 learned_addr learned my addr 172.21.15.89:0/1656153136 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:56.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.472+0000 7f3143e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f3144160f10 0x7f31440d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:38686/0 (socket says 172.21.15.89:38686) 2024-04-02T07:35:56.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.472+0000 7f3148a00700 1 -- 172.21.15.89:0/1656153136 >> v2:172.21.15.113:3300/0 conn(0x7f31440d18b0 msgr2=0x7f31440cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:56.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.472+0000 7f3148a00700 1 --2- 172.21.15.89:0/1656153136 >> v2:172.21.15.113:3300/0 conn(0x7f31440d18b0 0x7f31440cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.472+0000 7f3148a00700 1 -- 172.21.15.89:0/1656153136 >> v1:172.21.15.113:6789/0 conn(0x7f3144160f10 legacy=0x7f31440d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:56.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.472+0000 7f3148a00700 1 -- 172.21.15.89:0/1656153136 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31441612f0 con 0x7f3144161590 2024-04-02T07:35:56.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.472+0000 7f3142a00700 1 -- 172.21.15.89:0/1656153136 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563028135 0 0) 0x7f31440ceda0 con 0x7f3144160f10 2024-04-02T07:35:56.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.472+0000 7f3143400700 1 --2- 172.21.15.89:0/1656153136 >> v2:172.21.15.113:3300/0 conn(0x7f31440d18b0 0x7f31440cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.472+0000 7f3148a00700 1 --2- 172.21.15.89:0/1656153136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3144161590 0x7f3144161a20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f313c009e50 tx=0x7f313c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ae16413fb5e7d30 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.473+0000 7f3142a00700 1 -- 172.21.15.89:0/1656153136 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313c00e070 con 0x7f3144161590 2024-04-02T07:35:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.473+0000 7f3142a00700 1 -- 172.21.15.89:0/1656153136 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313c00b910 con 0x7f3144161590 2024-04-02T07:35:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.473+0000 7f3142a00700 1 -- 172.21.15.89:0/1656153136 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313c00bcc0 con 0x7f3144161590 2024-04-02T07:35:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.473+0000 7f314aa00700 1 -- 172.21.15.89:0/1656153136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3144161590 msgr2=0x7f3144161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.473+0000 7f314aa00700 1 --2- 172.21.15.89:0/1656153136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3144161590 0x7f3144161a20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f313c009e50 tx=0x7f313c00b330 comp rx=0 tx=0).stop 2024-04-02T07:35:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.473+0000 7f314aa00700 1 -- 172.21.15.89:0/1656153136 shutdown_connections 2024-04-02T07:35:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.473+0000 7f314aa00700 1 --2- 172.21.15.89:0/1656153136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3144161590 0x7f3144161a20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.473+0000 7f314aa00700 1 --2- 172.21.15.89:0/1656153136 >> v2:172.21.15.113:3300/0 conn(0x7f31440d18b0 0x7f31440cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.473+0000 7f314aa00700 1 -- 172.21.15.89:0/1656153136 >> 172.21.15.89:0/1656153136 conn(0x7f314408aff0 msgr2=0x7f31440d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.473+0000 7f314aa00700 1 -- 172.21.15.89:0/1656153136 shutdown_connections 2024-04-02T07:35:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.473+0000 7f314aa00700 1 -- 172.21.15.89:0/1656153136 wait complete. 2024-04-02T07:35:56.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.476+0000 7f314aa00700 1 Processor -- start 2024-04-02T07:35:56.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.476+0000 7f314aa00700 1 -- start start 2024-04-02T07:35:56.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.477+0000 7f314aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31440d18b0 0x7f31440b8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:56.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.477+0000 7f314aa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3144160f10 0x7f31441587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:56.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.477+0000 7f314aa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3144161590 0x7f31440b7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:56.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.477+0000 7f314aa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31440b94e0 con 0x7f31440d18b0 2024-04-02T07:35:56.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.477+0000 7f314aa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314409e150 con 0x7f3144161590 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.477+0000 7f314aa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3144158cf0 con 0x7f3144160f10 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.477+0000 7f3143e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31440d18b0 0x7f31440b8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.477+0000 7f3143e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31440d18b0 0x7f31440b8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57866/0 (socket says 172.21.15.89:57866) 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.477+0000 7f3143e00700 1 -- 172.21.15.89:0/3478786382 learned_addr learned my addr 172.21.15.89:0/3478786382 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.477+0000 7f3143400700 1 --2- 172.21.15.89:0/3478786382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3144160f10 0x7f31441587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.477+0000 7f3148a00700 1 --2- 172.21.15.89:0/3478786382 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3144161590 0x7f31440b7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.478+0000 7f3143e00700 1 -- 172.21.15.89:0/3478786382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3144160f10 msgr2=0x7f31441587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.478+0000 7f3143e00700 1 --2- 172.21.15.89:0/3478786382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3144160f10 0x7f31441587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.478+0000 7f3143e00700 1 -- 172.21.15.89:0/3478786382 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3144161590 msgr2=0x7f31440b7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.478+0000 7f3143e00700 1 --2- 172.21.15.89:0/3478786382 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3144161590 0x7f31440b7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.478+0000 7f3143e00700 1 -- 172.21.15.89:0/3478786382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31300094a0 con 0x7f31440d18b0 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.478+0000 7f3143e00700 1 --2- 172.21.15.89:0/3478786382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31440d18b0 0x7f31440b8fa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f313400c550 tx=0x7f313400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.478+0000 7f313be00700 1 -- 172.21.15.89:0/3478786382 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3134004c80 con 0x7f31440d18b0 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.478+0000 7f313be00700 1 -- 172.21.15.89:0/3478786382 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3134004e00 con 0x7f31440d18b0 2024-04-02T07:35:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.478+0000 7f314aa00700 1 -- 172.21.15.89:0/3478786382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313c009b30 con 0x7f31440d18b0 2024-04-02T07:35:56.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.478+0000 7f313be00700 1 -- 172.21.15.89:0/3478786382 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3134007310 con 0x7f31440d18b0 2024-04-02T07:35:56.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.478+0000 7f314aa00700 1 -- 172.21.15.89:0/3478786382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31440b84c0 con 0x7f31440d18b0 2024-04-02T07:35:56.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.479+0000 7f314aa00700 1 -- 172.21.15.89:0/3478786382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3144067720 con 0x7f31440d18b0 2024-04-02T07:35:56.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.480+0000 7f313be00700 1 -- 172.21.15.89:0/3478786382 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3134007470 con 0x7f31440d18b0 2024-04-02T07:35:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.481+0000 7f313be00700 1 --2- 172.21.15.89:0/3478786382 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f312406ac90 0x7f312406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.482+0000 7f313be00700 1 -- 172.21.15.89:0/3478786382 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3134088ea0 con 0x7f31440d18b0 2024-04-02T07:35:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.482+0000 7f3143400700 1 --2- 172.21.15.89:0/3478786382 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f312406ac90 0x7f312406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:56.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.482+0000 7f3143400700 1 --2- 172.21.15.89:0/3478786382 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f312406ac90 0x7f312406d150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f314415fd90 tx=0x7f3130009210 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:56.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.486+0000 7f313be00700 1 -- 172.21.15.89:0/3478786382 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3134054080 con 0x7f31440d18b0 2024-04-02T07:35:56.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.732+0000 7f314aa00700 1 -- 172.21.15.89:0/3478786382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f314415b4c0 con 0x7f31440d18b0 2024-04-02T07:35:56.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.733+0000 7f313be00700 1 -- 172.21.15.89:0/3478786382 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4731 (secure 0 0 0) 0x7f31340576a0 con 0x7f31440d18b0 2024-04-02T07:35:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:35:56.735 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2427593953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2427593953},{"type":"v1","addr":"172.21.15.113:6835","nonce":2427593953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/926246816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":926246816},{"type":"v1","addr":"172.21.15.89:6836","nonce":926246816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2701685411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2701685411},{"type":"v1","addr":"172.21.15.89:6837","nonce":2701685411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:35:53.235047+0000","modified":"2024-04-02T07:35:56.245811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6834/3962490902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3962490902},{"type":"v1","addr":"172.21.15.113:6834","nonce":3962490902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-02T07:35:56.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.735+0000 7f314aa00700 1 -- 172.21.15.89:0/3478786382 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f312406ac90 msgr2=0x7f312406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:56.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.735+0000 7f314aa00700 1 --2- 172.21.15.89:0/3478786382 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f312406ac90 0x7f312406d150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f314415fd90 tx=0x7f3130009210 comp rx=0 tx=0).stop 2024-04-02T07:35:56.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.735+0000 7f314aa00700 1 -- 172.21.15.89:0/3478786382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31440d18b0 msgr2=0x7f31440b8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:56.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.735+0000 7f314aa00700 1 --2- 172.21.15.89:0/3478786382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31440d18b0 0x7f31440b8fa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f313400c550 tx=0x7f313400c860 comp rx=0 tx=0).stop 2024-04-02T07:35:56.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.735+0000 7f314aa00700 1 -- 172.21.15.89:0/3478786382 shutdown_connections 2024-04-02T07:35:56.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.735+0000 7f314aa00700 1 --2- 172.21.15.89:0/3478786382 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f312406ac90 0x7f312406d150 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.735+0000 7f314aa00700 1 --2- 172.21.15.89:0/3478786382 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3144161590 0x7f31440b7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.735+0000 7f314aa00700 1 --2- 172.21.15.89:0/3478786382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31440d18b0 0x7f31440b8fa0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.735+0000 7f314aa00700 1 --2- 172.21.15.89:0/3478786382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3144160f10 0x7f31441587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.735+0000 7f314aa00700 1 -- 172.21.15.89:0/3478786382 >> 172.21.15.89:0/3478786382 conn(0x7f314408aff0 msgr2=0x7f31440ca930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:56.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.736+0000 7f314aa00700 1 -- 172.21.15.89:0/3478786382 shutdown_connections 2024-04-02T07:35:56.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.736+0000 7f314aa00700 1 -- 172.21.15.89:0/3478786382 wait complete. 2024-04-02T07:35:56.738 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2024-04-02T07:35:56.755 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:35:56.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.944+0000 7f86cb000700 1 Processor -- start 2024-04-02T07:35:56.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.944+0000 7f86cb000700 1 -- start start 2024-04-02T07:35:56.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.945+0000 7f86cb000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40d18b0 0x7f86c40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:56.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.945+0000 7f86cb000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f86c4161590 0x7f86c4161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:56.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.945+0000 7f86cb000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c40ceda0 con 0x7f86c4160f10 2024-04-02T07:35:56.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.945+0000 7f86cb000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c40d1d30 con 0x7f86c4161590 2024-04-02T07:35:56.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.946+0000 7f86cb000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c4161f60 con 0x7f86c40d18b0 2024-04-02T07:35:56.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.946+0000 7f86c3e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f86c4160f10 0x7f86c40d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36906/0 (socket says 172.21.15.89:36906) 2024-04-02T07:35:56.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.946+0000 7f86c3e00700 1 -- 172.21.15.89:0/4127033418 learned_addr learned my addr 172.21.15.89:0/4127033418 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:56.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.946+0000 7f86c9000700 1 --2- 172.21.15.89:0/4127033418 >> v2:172.21.15.89:3300/0 conn(0x7f86c4161590 0x7f86c4161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:56.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.946+0000 7f86c3400700 1 --2- 172.21.15.89:0/4127033418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40d18b0 0x7f86c40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:56.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.946+0000 7f86c9000700 1 -- 172.21.15.89:0/4127033418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40d18b0 msgr2=0x7f86c40cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:56.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.946+0000 7f86c9000700 1 --2- 172.21.15.89:0/4127033418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40d18b0 0x7f86c40cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.946+0000 7f86c9000700 1 -- 172.21.15.89:0/4127033418 >> v1:172.21.15.89:6789/0 conn(0x7f86c4160f10 legacy=0x7f86c40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:56.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.946+0000 7f86c9000700 1 -- 172.21.15.89:0/4127033418 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c41612f0 con 0x7f86c4161590 2024-04-02T07:35:56.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.946+0000 7f86c2a00700 1 -- 172.21.15.89:0/4127033418 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753004821 0 0) 0x7f86c40ceda0 con 0x7f86c4160f10 2024-04-02T07:35:56.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.946+0000 7f86c9000700 1 --2- 172.21.15.89:0/4127033418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c4161590 0x7f86c4161a20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f86bc009dc0 tx=0x7f86bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e578e78933987f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:56.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.947+0000 7f86c2a00700 1 -- 172.21.15.89:0/4127033418 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86bc00e070 con 0x7f86c4161590 2024-04-02T07:35:56.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.947+0000 7f86c2a00700 1 -- 172.21.15.89:0/4127033418 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86bc00bb90 con 0x7f86c4161590 2024-04-02T07:35:56.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.947+0000 7f86c2a00700 1 -- 172.21.15.89:0/4127033418 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86bc012430 con 0x7f86c4161590 2024-04-02T07:35:56.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.947+0000 7f86cb000700 1 -- 172.21.15.89:0/4127033418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c4161590 msgr2=0x7f86c4161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:56.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.947+0000 7f86cb000700 1 --2- 172.21.15.89:0/4127033418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c4161590 0x7f86c4161a20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f86bc009dc0 tx=0x7f86bc00b330 comp rx=0 tx=0).stop 2024-04-02T07:35:56.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.947+0000 7f86cb000700 1 -- 172.21.15.89:0/4127033418 shutdown_connections 2024-04-02T07:35:56.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.947+0000 7f86cb000700 1 --2- 172.21.15.89:0/4127033418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c4161590 0x7f86c4161a20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.947+0000 7f86cb000700 1 --2- 172.21.15.89:0/4127033418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40d18b0 0x7f86c40cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.947+0000 7f86cb000700 1 -- 172.21.15.89:0/4127033418 >> 172.21.15.89:0/4127033418 conn(0x7f86c408aff0 msgr2=0x7f86c40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:56.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.947+0000 7f86cb000700 1 -- 172.21.15.89:0/4127033418 shutdown_connections 2024-04-02T07:35:56.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.947+0000 7f86cb000700 1 -- 172.21.15.89:0/4127033418 wait complete. 2024-04-02T07:35:56.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.949+0000 7f86cb000700 1 Processor -- start 2024-04-02T07:35:56.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.949+0000 7f86cb000700 1 -- start start 2024-04-02T07:35:56.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.950+0000 7f86cb000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40d18b0 0x7f86c4158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:56.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.950+0000 7f86cb000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c4160f10 0x7f86c415df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:56.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.950+0000 7f86cb000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c4161590 0x7f86c40b8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:56.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.950+0000 7f86cb000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40b6bd0 con 0x7f86c4160f10 2024-04-02T07:35:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.950+0000 7f86cb000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40b9270 con 0x7f86c4161590 2024-04-02T07:35:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.950+0000 7f86cb000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40b62a0 con 0x7f86c40d18b0 2024-04-02T07:35:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.950+0000 7f86c3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c4160f10 0x7f86c415df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.950+0000 7f86c3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c4160f10 0x7f86c415df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57884/0 (socket says 172.21.15.89:57884) 2024-04-02T07:35:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.950+0000 7f86c3400700 1 -- 172.21.15.89:0/2112967821 learned_addr learned my addr 172.21.15.89:0/2112967821 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.950+0000 7f86c3e00700 1 --2- 172.21.15.89:0/2112967821 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40d18b0 0x7f86c4158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.950+0000 7f86c9000700 1 --2- 172.21.15.89:0/2112967821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c4161590 0x7f86c40b8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.951+0000 7f86c3400700 1 -- 172.21.15.89:0/2112967821 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40d18b0 msgr2=0x7f86c4158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.951+0000 7f86c3400700 1 --2- 172.21.15.89:0/2112967821 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40d18b0 0x7f86c4158c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.951+0000 7f86c3400700 1 -- 172.21.15.89:0/2112967821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c4161590 msgr2=0x7f86c40b8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.951+0000 7f86c3400700 1 --2- 172.21.15.89:0/2112967821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c4161590 0x7f86c40b8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.951+0000 7f86c3400700 1 -- 172.21.15.89:0/2112967821 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86bc009a70 con 0x7f86c4160f10 2024-04-02T07:35:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.951+0000 7f86c3400700 1 --2- 172.21.15.89:0/2112967821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c4160f10 0x7f86c415df60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f86b000d800 tx=0x7f86b000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:56.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.951+0000 7f86bbe00700 1 -- 172.21.15.89:0/2112967821 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b00125f0 con 0x7f86c4160f10 2024-04-02T07:35:56.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.951+0000 7f86bbe00700 1 -- 172.21.15.89:0/2112967821 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b0012750 con 0x7f86c4160f10 2024-04-02T07:35:56.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.951+0000 7f86cb000700 1 -- 172.21.15.89:0/2112967821 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c40b6580 con 0x7f86c4160f10 2024-04-02T07:35:56.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.951+0000 7f86bbe00700 1 -- 172.21.15.89:0/2112967821 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b0012b00 con 0x7f86c4160f10 2024-04-02T07:35:56.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.951+0000 7f86cb000700 1 -- 172.21.15.89:0/2112967821 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c41f2ee0 con 0x7f86c4160f10 2024-04-02T07:35:56.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.952+0000 7f86cb000700 1 -- 172.21.15.89:0/2112967821 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c40b6d40 con 0x7f86c4160f10 2024-04-02T07:35:56.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.953+0000 7f86bbe00700 1 -- 172.21.15.89:0/2112967821 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f86b0012c60 con 0x7f86c4160f10 2024-04-02T07:35:56.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.955+0000 7f86bbe00700 1 --2- 172.21.15.89:0/2112967821 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f86a406ac90 0x7f86a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:56.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.955+0000 7f86bbe00700 1 -- 172.21.15.89:0/2112967821 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86b00893f0 con 0x7f86c4160f10 2024-04-02T07:35:56.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.955+0000 7f86c3e00700 1 --2- 172.21.15.89:0/2112967821 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f86a406ac90 0x7f86a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:56.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.955+0000 7f86c3e00700 1 --2- 172.21.15.89:0/2112967821 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f86a406ac90 0x7f86a406d150 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f86b4000f60 tx=0x7f86b4007e70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:56.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:56.959+0000 7f86bbe00700 1 -- 172.21.15.89:0/2112967821 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f86b00545e0 con 0x7f86c4160f10 2024-04-02T07:35:57.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.164+0000 7f86cb000700 1 -- 172.21.15.89:0/2112967821 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86c4067720 con 0x7f86c4160f10 2024-04-02T07:35:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.165+0000 7f86bbe00700 1 -- 172.21.15.89:0/2112967821 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f86b0057c00 con 0x7f86c4160f10 2024-04-02T07:35:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:35:57.167 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:54.238913+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:35:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.168+0000 7f86cb000700 1 -- 172.21.15.89:0/2112967821 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f86a406ac90 msgr2=0x7f86a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.168+0000 7f86cb000700 1 --2- 172.21.15.89:0/2112967821 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f86a406ac90 0x7f86a406d150 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f86b4000f60 tx=0x7f86b4007e70 comp rx=0 tx=0).stop 2024-04-02T07:35:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.168+0000 7f86cb000700 1 -- 172.21.15.89:0/2112967821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c4160f10 msgr2=0x7f86c415df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.168+0000 7f86cb000700 1 --2- 172.21.15.89:0/2112967821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c4160f10 0x7f86c415df60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f86b000d800 tx=0x7f86b000db10 comp rx=0 tx=0).stop 2024-04-02T07:35:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.168+0000 7f86cb000700 1 -- 172.21.15.89:0/2112967821 shutdown_connections 2024-04-02T07:35:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.168+0000 7f86cb000700 1 --2- 172.21.15.89:0/2112967821 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f86a406ac90 0x7f86a406d150 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.168+0000 7f86cb000700 1 --2- 172.21.15.89:0/2112967821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c4161590 0x7f86c40b8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.168+0000 7f86cb000700 1 --2- 172.21.15.89:0/2112967821 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40d18b0 0x7f86c4158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.168+0000 7f86cb000700 1 --2- 172.21.15.89:0/2112967821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c4160f10 0x7f86c415df60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.168+0000 7f86cb000700 1 -- 172.21.15.89:0/2112967821 >> 172.21.15.89:0/2112967821 conn(0x7f86c408aff0 msgr2=0x7f86c40cf8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.168+0000 7f86cb000700 1 -- 172.21.15.89:0/2112967821 shutdown_connections 2024-04-02T07:35:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.168+0000 7f86cb000700 1 -- 172.21.15.89:0/2112967821 wait complete. 2024-04-02T07:35:57.188 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T07:35:57.188 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:35:57.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.377+0000 7f154f600700 1 Processor -- start 2024-04-02T07:35:57.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.377+0000 7f154f600700 1 -- start start 2024-04-02T07:35:57.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.378+0000 7f154f600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f154815b5d0 0x7f15480ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:57.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.378+0000 7f154f600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f15480ac7d0 0x7f154806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:57.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.378+0000 7f154f600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f154815fd50 con 0x7f15480cb3f0 2024-04-02T07:35:57.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.378+0000 7f154f600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154815b9b0 con 0x7f15480ac7d0 2024-04-02T07:35:57.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.378+0000 7f154f600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154815f140 con 0x7f154815b5d0 2024-04-02T07:35:57.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.379+0000 7f1547e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f15480cb3f0 0x7f15480ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36914/0 (socket says 172.21.15.89:36914) 2024-04-02T07:35:57.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.379+0000 7f1547e00700 1 -- 172.21.15.89:0/2871924916 learned_addr learned my addr 172.21.15.89:0/2871924916 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:57.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.379+0000 7f154d600700 1 --2- 172.21.15.89:0/2871924916 >> v2:172.21.15.89:3300/0 conn(0x7f15480ac7d0 0x7f154806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:57.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.379+0000 7f154cc00700 1 --2- 172.21.15.89:0/2871924916 >> v2:172.21.15.113:3300/0 conn(0x7f154815b5d0 0x7f15480ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.379+0000 7f1547400700 1 -- 172.21.15.89:0/2871924916 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431864503 0 0) 0x7f154815fd50 con 0x7f15480cb3f0 2024-04-02T07:35:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.379+0000 7f1547400700 1 -- 172.21.15.89:0/2871924916 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15300035b0 con 0x7f15480cb3f0 2024-04-02T07:35:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.379+0000 7f154d600700 1 -- 172.21.15.89:0/2871924916 >> v2:172.21.15.113:3300/0 conn(0x7f154815b5d0 msgr2=0x7f15480ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.379+0000 7f154d600700 1 --2- 172.21.15.89:0/2871924916 >> v2:172.21.15.113:3300/0 conn(0x7f154815b5d0 0x7f15480ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.379+0000 7f154d600700 1 -- 172.21.15.89:0/2871924916 >> v1:172.21.15.89:6789/0 conn(0x7f15480cb3f0 legacy=0x7f15480ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.379+0000 7f154d600700 1 -- 172.21.15.89:0/2871924916 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154815f420 con 0x7f15480ac7d0 2024-04-02T07:35:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.379+0000 7f154d600700 1 --2- 172.21.15.89:0/2871924916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15480ac7d0 0x7f154806c9d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f154806b8b0 tx=0x7f154000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb60988817fcbe5a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.380+0000 7f1547400700 1 -- 172.21.15.89:0/2871924916 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154000e070 con 0x7f15480ac7d0 2024-04-02T07:35:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.380+0000 7f1547400700 1 -- 172.21.15.89:0/2871924916 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154000bb20 con 0x7f15480ac7d0 2024-04-02T07:35:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.380+0000 7f1547400700 1 -- 172.21.15.89:0/2871924916 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1540012430 con 0x7f15480ac7d0 2024-04-02T07:35:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.380+0000 7f154f600700 1 -- 172.21.15.89:0/2871924916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15480ac7d0 msgr2=0x7f154806c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.380+0000 7f154f600700 1 --2- 172.21.15.89:0/2871924916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15480ac7d0 0x7f154806c9d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f154806b8b0 tx=0x7f154000b330 comp rx=0 tx=0).stop 2024-04-02T07:35:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.380+0000 7f154f600700 1 -- 172.21.15.89:0/2871924916 shutdown_connections 2024-04-02T07:35:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.380+0000 7f154f600700 1 --2- 172.21.15.89:0/2871924916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15480ac7d0 0x7f154806c9d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.380+0000 7f154f600700 1 --2- 172.21.15.89:0/2871924916 >> v2:172.21.15.113:3300/0 conn(0x7f154815b5d0 0x7f15480ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.380+0000 7f154f600700 1 -- 172.21.15.89:0/2871924916 >> 172.21.15.89:0/2871924916 conn(0x7f15480cf190 msgr2=0x7f15480cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.380+0000 7f154f600700 1 -- 172.21.15.89:0/2871924916 shutdown_connections 2024-04-02T07:35:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.380+0000 7f154f600700 1 -- 172.21.15.89:0/2871924916 wait complete. 2024-04-02T07:35:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.382+0000 7f154f600700 1 Processor -- start 2024-04-02T07:35:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.382+0000 7f154f600700 1 -- start start 2024-04-02T07:35:57.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.383+0000 7f154f600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15480ac7d0 0x7f1548155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:57.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.383+0000 7f154f600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f154815b5d0 0x7f15481f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:57.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.383+0000 7f154f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15481f45e0 0x7f15481f4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.383+0000 7f154f600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15481f4fd0 con 0x7f154815b5d0 2024-04-02T07:35:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.383+0000 7f154f600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154806e220 con 0x7f15480ac7d0 2024-04-02T07:35:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.383+0000 7f154f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154809cdf0 con 0x7f15481f45e0 2024-04-02T07:35:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f1547e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f154815b5d0 0x7f15481f40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f1547e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f154815b5d0 0x7f15481f40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57902/0 (socket says 172.21.15.89:57902) 2024-04-02T07:35:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f1547e00700 1 -- 172.21.15.89:0/3117507271 learned_addr learned my addr 172.21.15.89:0/3117507271 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f154cc00700 1 --2- 172.21.15.89:0/3117507271 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15480ac7d0 0x7f1548155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f154d600700 1 --2- 172.21.15.89:0/3117507271 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15481f45e0 0x7f15481f4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f1547e00700 1 -- 172.21.15.89:0/3117507271 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15481f45e0 msgr2=0x7f15481f4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f1547e00700 1 --2- 172.21.15.89:0/3117507271 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15481f45e0 0x7f15481f4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f1547e00700 1 -- 172.21.15.89:0/3117507271 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15480ac7d0 msgr2=0x7f1548155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f1547e00700 1 --2- 172.21.15.89:0/3117507271 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15480ac7d0 0x7f1548155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f1547e00700 1 -- 172.21.15.89:0/3117507271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1540009a70 con 0x7f154815b5d0 2024-04-02T07:35:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f1547e00700 1 --2- 172.21.15.89:0/3117507271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f154815b5d0 0x7f15481f40a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f153400c730 tx=0x7f153400ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:57.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f1544c00700 1 -- 172.21.15.89:0/3117507271 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1534005d70 con 0x7f154815b5d0 2024-04-02T07:35:57.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f1544c00700 1 -- 172.21.15.89:0/3117507271 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15340070d0 con 0x7f154815b5d0 2024-04-02T07:35:57.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f154f600700 1 -- 172.21.15.89:0/3117507271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154809d0d0 con 0x7f154815b5d0 2024-04-02T07:35:57.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f1544c00700 1 -- 172.21.15.89:0/3117507271 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1534007440 con 0x7f154815b5d0 2024-04-02T07:35:57.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.384+0000 7f154f600700 1 -- 172.21.15.89:0/3117507271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154806dd00 con 0x7f154815b5d0 2024-04-02T07:35:57.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.385+0000 7f154f600700 1 -- 172.21.15.89:0/3117507271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15481581e0 con 0x7f154815b5d0 2024-04-02T07:35:57.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.386+0000 7f1544c00700 1 -- 172.21.15.89:0/3117507271 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f15340075a0 con 0x7f154815b5d0 2024-04-02T07:35:57.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.388+0000 7f1544c00700 1 --2- 172.21.15.89:0/3117507271 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f152806ac90 0x7f152806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:57.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.388+0000 7f1544c00700 1 -- 172.21.15.89:0/3117507271 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15340886c0 con 0x7f154815b5d0 2024-04-02T07:35:57.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.388+0000 7f154cc00700 1 --2- 172.21.15.89:0/3117507271 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f152806ac90 0x7f152806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:57.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.388+0000 7f154cc00700 1 --2- 172.21.15.89:0/3117507271 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f152806ac90 0x7f152806d150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f153c006b20 tx=0x7f153c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:57.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.392+0000 7f1544c00700 1 -- 172.21.15.89:0/3117507271 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15340538b0 con 0x7f154815b5d0 2024-04-02T07:35:57.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.590+0000 7f154f600700 1 -- 172.21.15.89:0/3117507271 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f154809cc70 con 0x7f152806ac90 2024-04-02T07:35:57.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.599+0000 7f1544c00700 1 -- 172.21.15.89:0/3117507271 <== mgr.4108 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264698 (secure 0 0 0) 0x7f153c0076b0 con 0x7f152806ac90 2024-04-02T07:35:57.604 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:35:57.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.603+0000 7f154f600700 1 -- 172.21.15.89:0/3117507271 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f152806ac90 msgr2=0x7f152806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:57.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.603+0000 7f154f600700 1 --2- 172.21.15.89:0/3117507271 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f152806ac90 0x7f152806d150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f153c006b20 tx=0x7f153c008040 comp rx=0 tx=0).stop 2024-04-02T07:35:57.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.603+0000 7f154f600700 1 -- 172.21.15.89:0/3117507271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f154815b5d0 msgr2=0x7f15481f40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:57.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.603+0000 7f154f600700 1 --2- 172.21.15.89:0/3117507271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f154815b5d0 0x7f15481f40a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f153400c730 tx=0x7f153400ca40 comp rx=0 tx=0).stop 2024-04-02T07:35:57.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.603+0000 7f154f600700 1 -- 172.21.15.89:0/3117507271 shutdown_connections 2024-04-02T07:35:57.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.603+0000 7f154f600700 1 --2- 172.21.15.89:0/3117507271 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f152806ac90 0x7f152806d150 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.603+0000 7f154f600700 1 --2- 172.21.15.89:0/3117507271 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15480ac7d0 0x7f1548155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.603+0000 7f154f600700 1 --2- 172.21.15.89:0/3117507271 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15481f45e0 0x7f15481f4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.603+0000 7f154f600700 1 --2- 172.21.15.89:0/3117507271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f154815b5d0 0x7f15481f40a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.603+0000 7f154f600700 1 -- 172.21.15.89:0/3117507271 >> 172.21.15.89:0/3117507271 conn(0x7f15480cf190 msgr2=0x7f15480b3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:57.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.603+0000 7f154f600700 1 -- 172.21.15.89:0/3117507271 shutdown_connections 2024-04-02T07:35:57.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.603+0000 7f154f600700 1 -- 172.21.15.89:0/3117507271 wait complete. 2024-04-02T07:35:57.607 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:35:57.622 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-04-02T07:35:57.183599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219000,"kb_used_data":2816,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755720,"statfs":{"total":773094113280,"available":772869857280,"internally_reserved":0,"allocated":2883584,"data_stored":1750606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.452801"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615693+0000","last_change":"2024-04-02T07:35:53.278052+0000","last_active":"2024-04-02T07:35:54.615693+0000","last_peered":"2024-04-02T07:35:54.615693+0000","last_clean":"2024-04-02T07:35:54.615693+0000","last_became_active":"2024-04-02T07:35:53.277648+0000","last_became_peered":"2024-04-02T07:35:53.277648+0000","last_unstale":"2024-04-02T07:35:54.615693+0000","last_undegraded":"2024-04-02T07:35:54.615693+0000","last_fullsized":"2024-04-02T07:35:54.615693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.380856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617752+0000","last_change":"2024-04-02T07:35:53.271744+0000","last_active":"2024-04-02T07:35:54.617752+0000","last_peered":"2024-04-02T07:35:54.617752+0000","last_clean":"2024-04-02T07:35:54.617752+0000","last_became_active":"2024-04-02T07:35:53.271453+0000","last_became_peered":"2024-04-02T07:35:53.271453+0000","last_unstale":"2024-04-02T07:35:54.617752+0000","last_undegraded":"2024-04-02T07:35:54.617752+0000","last_fullsized":"2024-04-02T07:35:54.617752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:41:21.403396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617763+0000","last_change":"2024-04-02T07:35:53.257746+0000","last_active":"2024-04-02T07:35:54.617763+0000","last_peered":"2024-04-02T07:35:54.617763+0000","last_clean":"2024-04-02T07:35:54.617763+0000","last_became_active":"2024-04-02T07:35:53.257428+0000","last_became_peered":"2024-04-02T07:35:53.257428+0000","last_unstale":"2024-04-02T07:35:54.617763+0000","last_undegraded":"2024-04-02T07:35:54.617763+0000","last_fullsized":"2024-04-02T07:35:54.617763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:13:04.549460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557285+0000","last_change":"2024-04-02T07:35:53.269314+0000","last_active":"2024-04-02T07:35:54.557285+0000","last_peered":"2024-04-02T07:35:54.557285+0000","last_clean":"2024-04-02T07:35:54.557285+0000","last_became_active":"2024-04-02T07:35:53.269081+0000","last_became_peered":"2024-04-02T07:35:53.269081+0000","last_unstale":"2024-04-02T07:35:54.557285+0000","last_undegraded":"2024-04-02T07:35:54.557285+0000","last_fullsized":"2024-04-02T07:35:54.557285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:24.367224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615439+0000","last_change":"2024-04-02T07:35:53.257006+0000","last_active":"2024-04-02T07:35:54.615439+0000","last_peered":"2024-04-02T07:35:54.615439+0000","last_clean":"2024-04-02T07:35:54.615439+0000","last_became_active":"2024-04-02T07:35:53.256623+0000","last_became_peered":"2024-04-02T07:35:53.256623+0000","last_unstale":"2024-04-02T07:35:54.615439+0000","last_undegraded":"2024-04-02T07:35:54.615439+0000","last_fullsized":"2024-04-02T07:35:54.615439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:13:04.549460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557080+0000","last_change":"2024-04-02T07:35:53.256730+0000","last_active":"2024-04-02T07:35:54.557080+0000","last_peered":"2024-04-02T07:35:54.557080+0000","last_clean":"2024-04-02T07:35:54.557080+0000","last_became_active":"2024-04-02T07:35:53.256395+0000","last_became_peered":"2024-04-02T07:35:53.256395+0000","last_unstale":"2024-04-02T07:35:54.557080+0000","last_undegraded":"2024-04-02T07:35:54.557080+0000","last_fullsized":"2024-04-02T07:35:54.557080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:55.943715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615284+0000","last_change":"2024-04-02T07:35:53.267364+0000","last_active":"2024-04-02T07:35:54.615284+0000","last_peered":"2024-04-02T07:35:54.615284+0000","last_clean":"2024-04-02T07:35:54.615284+0000","last_became_active":"2024-04-02T07:35:53.267076+0000","last_became_peered":"2024-04-02T07:35:53.267076+0000","last_unstale":"2024-04-02T07:35:54.615284+0000","last_undegraded":"2024-04-02T07:35:54.615284+0000","last_fullsized":"2024-04-02T07:35:54.615284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:53.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617144+0000","last_change":"2024-04-02T07:35:53.275758+0000","last_active":"2024-04-02T07:35:54.617144+0000","last_peered":"2024-04-02T07:35:54.617144+0000","last_clean":"2024-04-02T07:35:54.617144+0000","last_became_active":"2024-04-02T07:35:53.264243+0000","last_became_peered":"2024-04-02T07:35:53.264243+0000","last_unstale":"2024-04-02T07:35:54.617144+0000","last_undegraded":"2024-04-02T07:35:54.617144+0000","last_fullsized":"2024-04-02T07:35:54.617144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:13.487258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246979+0000","last_change":"2024-04-02T07:35:53.280476+0000","last_active":"2024-04-02T07:35:54.246979+0000","last_peered":"2024-04-02T07:35:54.246979+0000","last_clean":"2024-04-02T07:35:54.246979+0000","last_became_active":"2024-04-02T07:35:53.280209+0000","last_became_peered":"2024-04-02T07:35:53.280209+0000","last_unstale":"2024-04-02T07:35:54.246979+0000","last_undegraded":"2024-04-02T07:35:54.246979+0000","last_fullsized":"2024-04-02T07:35:54.246979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:06.923152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615751+0000","last_change":"2024-04-02T07:35:53.258035+0000","last_active":"2024-04-02T07:35:54.615751+0000","last_peered":"2024-04-02T07:35:54.615751+0000","last_clean":"2024-04-02T07:35:54.615751+0000","last_became_active":"2024-04-02T07:35:53.257721+0000","last_became_peered":"2024-04-02T07:35:53.257721+0000","last_unstale":"2024-04-02T07:35:54.615751+0000","last_undegraded":"2024-04-02T07:35:54.615751+0000","last_fullsized":"2024-04-02T07:35:54.615751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:18.703428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615499+0000","last_change":"2024-04-02T07:35:53.275062+0000","last_active":"2024-04-02T07:35:54.615499+0000","last_peered":"2024-04-02T07:35:54.615499+0000","last_clean":"2024-04-02T07:35:54.615499+0000","last_became_active":"2024-04-02T07:35:53.274852+0000","last_became_peered":"2024-04-02T07:35:53.274852+0000","last_unstale":"2024-04-02T07:35:54.615499+0000","last_undegraded":"2024-04-02T07:35:54.615499+0000","last_fullsized":"2024-04-02T07:35:54.615499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:13.487258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615560+0000","last_change":"2024-04-02T07:35:53.269911+0000","last_active":"2024-04-02T07:35:54.615560+0000","last_peered":"2024-04-02T07:35:54.615560+0000","last_clean":"2024-04-02T07:35:54.615560+0000","last_became_active":"2024-04-02T07:35:53.265651+0000","last_became_peered":"2024-04-02T07:35:53.265651+0000","last_unstale":"2024-04-02T07:35:54.615560+0000","last_undegraded":"2024-04-02T07:35:54.615560+0000","last_fullsized":"2024-04-02T07:35:54.615560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:16.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617009+0000","last_change":"2024-04-02T07:35:53.273591+0000","last_active":"2024-04-02T07:35:54.617009+0000","last_peered":"2024-04-02T07:35:54.617009+0000","last_clean":"2024-04-02T07:35:54.617009+0000","last_became_active":"2024-04-02T07:35:53.273289+0000","last_became_peered":"2024-04-02T07:35:53.273289+0000","last_unstale":"2024-04-02T07:35:54.617009+0000","last_undegraded":"2024-04-02T07:35:54.617009+0000","last_fullsized":"2024-04-02T07:35:54.617009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:00:20.419421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616704+0000","last_change":"2024-04-02T07:35:53.281809+0000","last_active":"2024-04-02T07:35:54.616704+0000","last_peered":"2024-04-02T07:35:54.616704+0000","last_clean":"2024-04-02T07:35:54.616704+0000","last_became_active":"2024-04-02T07:35:53.277985+0000","last_became_peered":"2024-04-02T07:35:53.277985+0000","last_unstale":"2024-04-02T07:35:54.616704+0000","last_undegraded":"2024-04-02T07:35:54.616704+0000","last_fullsized":"2024-04-02T07:35:54.616704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:42.518534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617297+0000","last_change":"2024-04-02T07:35:53.268138+0000","last_active":"2024-04-02T07:35:54.617297+0000","last_peered":"2024-04-02T07:35:54.617297+0000","last_clean":"2024-04-02T07:35:54.617297+0000","last_became_active":"2024-04-02T07:35:53.264989+0000","last_became_peered":"2024-04-02T07:35:53.264989+0000","last_unstale":"2024-04-02T07:35:54.617297+0000","last_undegraded":"2024-04-02T07:35:54.617297+0000","last_fullsized":"2024-04-02T07:35:54.617297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.380856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245556+0000","last_change":"2024-04-02T07:35:53.276292+0000","last_active":"2024-04-02T07:35:54.245556+0000","last_peered":"2024-04-02T07:35:54.245556+0000","last_clean":"2024-04-02T07:35:54.245556+0000","last_became_active":"2024-04-02T07:35:53.276003+0000","last_became_peered":"2024-04-02T07:35:53.276003+0000","last_unstale":"2024-04-02T07:35:54.245556+0000","last_undegraded":"2024-04-02T07:35:54.245556+0000","last_fullsized":"2024-04-02T07:35:54.245556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:33.477903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247301+0000","last_change":"2024-04-02T07:35:53.269437+0000","last_active":"2024-04-02T07:35:54.247301+0000","last_peered":"2024-04-02T07:35:54.247301+0000","last_clean":"2024-04-02T07:35:54.247301+0000","last_became_active":"2024-04-02T07:35:53.269192+0000","last_became_peered":"2024-04-02T07:35:53.269192+0000","last_unstale":"2024-04-02T07:35:54.247301+0000","last_undegraded":"2024-04-02T07:35:54.247301+0000","last_fullsized":"2024-04-02T07:35:54.247301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:35.381030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617797+0000","last_change":"2024-04-02T07:35:53.259373+0000","last_active":"2024-04-02T07:35:54.617797+0000","last_peered":"2024-04-02T07:35:54.617797+0000","last_clean":"2024-04-02T07:35:54.617797+0000","last_became_active":"2024-04-02T07:35:53.258986+0000","last_became_peered":"2024-04-02T07:35:53.258986+0000","last_unstale":"2024-04-02T07:35:54.617797+0000","last_undegraded":"2024-04-02T07:35:54.617797+0000","last_fullsized":"2024-04-02T07:35:54.617797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:27.051276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246472+0000","last_change":"2024-04-02T07:35:53.277181+0000","last_active":"2024-04-02T07:35:54.246472+0000","last_peered":"2024-04-02T07:35:54.246472+0000","last_clean":"2024-04-02T07:35:54.246472+0000","last_became_active":"2024-04-02T07:35:53.276207+0000","last_became_peered":"2024-04-02T07:35:53.276207+0000","last_unstale":"2024-04-02T07:35:54.246472+0000","last_undegraded":"2024-04-02T07:35:54.246472+0000","last_fullsized":"2024-04-02T07:35:54.246472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:13.487258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247380+0000","last_change":"2024-04-02T07:35:52.756587+0000","last_active":"2024-04-02T07:35:54.247380+0000","last_peered":"2024-04-02T07:35:54.247380+0000","last_clean":"2024-04-02T07:35:54.247380+0000","last_became_active":"2024-04-02T07:35:52.755981+0000","last_became_peered":"2024-04-02T07:35:52.755981+0000","last_unstale":"2024-04-02T07:35:54.247380+0000","last_undegraded":"2024-04-02T07:35:54.247380+0000","last_fullsized":"2024-04-02T07:35:54.247380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:07.324155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617519+0000","last_change":"2024-04-02T07:35:53.257824+0000","last_active":"2024-04-02T07:35:54.617519+0000","last_peered":"2024-04-02T07:35:54.617519+0000","last_clean":"2024-04-02T07:35:54.617519+0000","last_became_active":"2024-04-02T07:35:53.257500+0000","last_became_peered":"2024-04-02T07:35:53.257500+0000","last_unstale":"2024-04-02T07:35:54.617519+0000","last_undegraded":"2024-04-02T07:35:54.617519+0000","last_fullsized":"2024-04-02T07:35:54.617519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:35.709321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617620+0000","last_change":"2024-04-02T07:35:52.761793+0000","last_active":"2024-04-02T07:35:54.617620+0000","last_peered":"2024-04-02T07:35:54.617620+0000","last_clean":"2024-04-02T07:35:54.617620+0000","last_became_active":"2024-04-02T07:35:52.761510+0000","last_became_peered":"2024-04-02T07:35:52.761510+0000","last_unstale":"2024-04-02T07:35:54.617620+0000","last_undegraded":"2024-04-02T07:35:54.617620+0000","last_fullsized":"2024-04-02T07:35:54.617620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617652+0000","last_change":"2024-04-02T07:35:53.262736+0000","last_active":"2024-04-02T07:35:54.617652+0000","last_peered":"2024-04-02T07:35:54.617652+0000","last_clean":"2024-04-02T07:35:54.617652+0000","last_became_active":"2024-04-02T07:35:53.262405+0000","last_became_peered":"2024-04-02T07:35:53.262405+0000","last_unstale":"2024-04-02T07:35:54.617652+0000","last_undegraded":"2024-04-02T07:35:54.617652+0000","last_fullsized":"2024-04-02T07:35:54.617652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:52:59.688399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615557+0000","last_change":"2024-04-02T07:35:52.765725+0000","last_active":"2024-04-02T07:35:54.615557+0000","last_peered":"2024-04-02T07:35:54.615557+0000","last_clean":"2024-04-02T07:35:54.615557+0000","last_became_active":"2024-04-02T07:35:52.765393+0000","last_became_peered":"2024-04-02T07:35:52.765393+0000","last_unstale":"2024-04-02T07:35:54.615557+0000","last_undegraded":"2024-04-02T07:35:54.615557+0000","last_fullsized":"2024-04-02T07:35:54.615557+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:31.244191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246863+0000","last_change":"2024-04-02T07:35:53.255693+0000","last_active":"2024-04-02T07:35:54.246863+0000","last_peered":"2024-04-02T07:35:54.246863+0000","last_clean":"2024-04-02T07:35:54.246863+0000","last_became_active":"2024-04-02T07:35:53.254627+0000","last_became_peered":"2024-04-02T07:35:53.254627+0000","last_unstale":"2024-04-02T07:35:54.246863+0000","last_undegraded":"2024-04-02T07:35:54.246863+0000","last_fullsized":"2024-04-02T07:35:54.246863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:42.518534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615455+0000","last_change":"2024-04-02T07:35:52.762790+0000","last_active":"2024-04-02T07:35:54.615455+0000","last_peered":"2024-04-02T07:35:54.615455+0000","last_clean":"2024-04-02T07:35:54.615455+0000","last_became_active":"2024-04-02T07:35:52.762416+0000","last_became_peered":"2024-04-02T07:35:52.762416+0000","last_unstale":"2024-04-02T07:35:54.615455+0000","last_undegraded":"2024-04-02T07:35:54.615455+0000","last_fullsized":"2024-04-02T07:35:54.615455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:05.916519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246530+0000","last_change":"2024-04-02T07:35:53.274608+0000","last_active":"2024-04-02T07:35:54.246530+0000","last_peered":"2024-04-02T07:35:54.246530+0000","last_clean":"2024-04-02T07:35:54.246530+0000","last_became_active":"2024-04-02T07:35:53.274308+0000","last_became_peered":"2024-04-02T07:35:53.274308+0000","last_unstale":"2024-04-02T07:35:54.246530+0000","last_undegraded":"2024-04-02T07:35:54.246530+0000","last_fullsized":"2024-04-02T07:35:54.246530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:28:26.191159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617955+0000","last_change":"2024-04-02T07:35:52.761806+0000","last_active":"2024-04-02T07:35:54.617955+0000","last_peered":"2024-04-02T07:35:54.617955+0000","last_clean":"2024-04-02T07:35:54.617955+0000","last_became_active":"2024-04-02T07:35:52.761556+0000","last_became_peered":"2024-04-02T07:35:52.761556+0000","last_unstale":"2024-04-02T07:35:54.617955+0000","last_undegraded":"2024-04-02T07:35:54.617955+0000","last_fullsized":"2024-04-02T07:35:54.617955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246589+0000","last_change":"2024-04-02T07:35:53.262036+0000","last_active":"2024-04-02T07:35:54.246589+0000","last_peered":"2024-04-02T07:35:54.246589+0000","last_clean":"2024-04-02T07:35:54.246589+0000","last_became_active":"2024-04-02T07:35:53.259860+0000","last_became_peered":"2024-04-02T07:35:53.259860+0000","last_unstale":"2024-04-02T07:35:54.246589+0000","last_undegraded":"2024-04-02T07:35:54.246589+0000","last_fullsized":"2024-04-02T07:35:54.246589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:27.051276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245921+0000","last_change":"2024-04-02T07:35:52.773232+0000","last_active":"2024-04-02T07:35:54.245921+0000","last_peered":"2024-04-02T07:35:54.245921+0000","last_clean":"2024-04-02T07:35:54.245921+0000","last_became_active":"2024-04-02T07:35:52.772864+0000","last_became_peered":"2024-04-02T07:35:52.772864+0000","last_unstale":"2024-04-02T07:35:54.245921+0000","last_undegraded":"2024-04-02T07:35:54.245921+0000","last_fullsized":"2024-04-02T07:35:54.245921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:26.044643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247078+0000","last_change":"2024-04-02T07:35:53.274460+0000","last_active":"2024-04-02T07:35:54.247078+0000","last_peered":"2024-04-02T07:35:54.247078+0000","last_clean":"2024-04-02T07:35:54.247078+0000","last_became_active":"2024-04-02T07:35:53.274227+0000","last_became_peered":"2024-04-02T07:35:53.274227+0000","last_unstale":"2024-04-02T07:35:54.247078+0000","last_undegraded":"2024-04-02T07:35:54.247078+0000","last_fullsized":"2024-04-02T07:35:54.247078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:31.170936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616577+0000","last_change":"2024-04-02T07:35:52.769404+0000","last_active":"2024-04-02T07:35:54.616577+0000","last_peered":"2024-04-02T07:35:54.616577+0000","last_clean":"2024-04-02T07:35:54.616577+0000","last_became_active":"2024-04-02T07:35:52.769091+0000","last_became_peered":"2024-04-02T07:35:52.769091+0000","last_unstale":"2024-04-02T07:35:54.616577+0000","last_undegraded":"2024-04-02T07:35:54.616577+0000","last_fullsized":"2024-04-02T07:35:54.616577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617272+0000","last_change":"2024-04-02T07:35:53.261740+0000","last_active":"2024-04-02T07:35:54.617272+0000","last_peered":"2024-04-02T07:35:54.617272+0000","last_clean":"2024-04-02T07:35:54.617272+0000","last_became_active":"2024-04-02T07:35:53.261475+0000","last_became_peered":"2024-04-02T07:35:53.261475+0000","last_unstale":"2024-04-02T07:35:54.617272+0000","last_undegraded":"2024-04-02T07:35:54.617272+0000","last_fullsized":"2024-04-02T07:35:54.617272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:31.170936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245738+0000","last_change":"2024-04-02T07:35:52.765814+0000","last_active":"2024-04-02T07:35:54.245738+0000","last_peered":"2024-04-02T07:35:54.245738+0000","last_clean":"2024-04-02T07:35:54.245738+0000","last_became_active":"2024-04-02T07:35:52.764956+0000","last_became_peered":"2024-04-02T07:35:52.764956+0000","last_unstale":"2024-04-02T07:35:54.245738+0000","last_undegraded":"2024-04-02T07:35:54.245738+0000","last_fullsized":"2024-04-02T07:35:54.245738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:49.544958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247082+0000","last_change":"2024-04-02T07:35:53.278155+0000","last_active":"2024-04-02T07:35:54.247082+0000","last_peered":"2024-04-02T07:35:54.247082+0000","last_clean":"2024-04-02T07:35:54.247082+0000","last_became_active":"2024-04-02T07:35:53.274007+0000","last_became_peered":"2024-04-02T07:35:53.274007+0000","last_unstale":"2024-04-02T07:35:54.247082+0000","last_undegraded":"2024-04-02T07:35:54.247082+0000","last_fullsized":"2024-04-02T07:35:54.247082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:05:41.161422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246958+0000","last_change":"2024-04-02T07:35:52.759611+0000","last_active":"2024-04-02T07:35:54.246958+0000","last_peered":"2024-04-02T07:35:54.246958+0000","last_clean":"2024-04-02T07:35:54.246958+0000","last_became_active":"2024-04-02T07:35:52.759164+0000","last_became_peered":"2024-04-02T07:35:52.759164+0000","last_unstale":"2024-04-02T07:35:54.246958+0000","last_undegraded":"2024-04-02T07:35:54.246958+0000","last_fullsized":"2024-04-02T07:35:54.246958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557483+0000","last_change":"2024-04-02T07:35:53.269405+0000","last_active":"2024-04-02T07:35:54.557483+0000","last_peered":"2024-04-02T07:35:54.557483+0000","last_clean":"2024-04-02T07:35:54.557483+0000","last_became_active":"2024-04-02T07:35:53.267870+0000","last_became_peered":"2024-04-02T07:35:53.267870+0000","last_unstale":"2024-04-02T07:35:54.557483+0000","last_undegraded":"2024-04-02T07:35:54.557483+0000","last_fullsized":"2024-04-02T07:35:54.557483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:08.722219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615626+0000","last_change":"2024-04-02T07:35:52.765581+0000","last_active":"2024-04-02T07:35:54.615626+0000","last_peered":"2024-04-02T07:35:54.615626+0000","last_clean":"2024-04-02T07:35:54.615626+0000","last_became_active":"2024-04-02T07:35:52.764766+0000","last_became_peered":"2024-04-02T07:35:52.764766+0000","last_unstale":"2024-04-02T07:35:54.615626+0000","last_undegraded":"2024-04-02T07:35:54.615626+0000","last_fullsized":"2024-04-02T07:35:54.615626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:07.715585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617760+0000","last_change":"2024-04-02T07:35:53.275646+0000","last_active":"2024-04-02T07:35:54.617760+0000","last_peered":"2024-04-02T07:35:54.617760+0000","last_clean":"2024-04-02T07:35:54.617760+0000","last_became_active":"2024-04-02T07:35:53.275296+0000","last_became_peered":"2024-04-02T07:35:53.275296+0000","last_unstale":"2024-04-02T07:35:54.617760+0000","last_undegraded":"2024-04-02T07:35:54.617760+0000","last_fullsized":"2024-04-02T07:35:54.617760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:17.569969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617746+0000","last_change":"2024-04-02T07:35:52.751184+0000","last_active":"2024-04-02T07:35:54.617746+0000","last_peered":"2024-04-02T07:35:54.617746+0000","last_clean":"2024-04-02T07:35:54.617746+0000","last_became_active":"2024-04-02T07:35:52.750436+0000","last_became_peered":"2024-04-02T07:35:52.750436+0000","last_unstale":"2024-04-02T07:35:54.617746+0000","last_undegraded":"2024-04-02T07:35:54.617746+0000","last_fullsized":"2024-04-02T07:35:54.617746+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246276+0000","last_change":"2024-04-02T07:35:53.277457+0000","last_active":"2024-04-02T07:35:54.246276+0000","last_peered":"2024-04-02T07:35:54.246276+0000","last_clean":"2024-04-02T07:35:54.246276+0000","last_became_active":"2024-04-02T07:35:53.277059+0000","last_became_peered":"2024-04-02T07:35:53.277059+0000","last_unstale":"2024-04-02T07:35:54.246276+0000","last_undegraded":"2024-04-02T07:35:54.246276+0000","last_fullsized":"2024-04-02T07:35:54.246276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:18:02.329099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557196+0000","last_change":"2024-04-02T07:35:52.757551+0000","last_active":"2024-04-02T07:35:54.557196+0000","last_peered":"2024-04-02T07:35:54.557196+0000","last_clean":"2024-04-02T07:35:54.557196+0000","last_became_active":"2024-04-02T07:35:52.756930+0000","last_became_peered":"2024-04-02T07:35:52.756930+0000","last_unstale":"2024-04-02T07:35:54.557196+0000","last_undegraded":"2024-04-02T07:35:54.557196+0000","last_fullsized":"2024-04-02T07:35:54.557196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:40:29.501936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615608+0000","last_change":"2024-04-02T07:35:53.265162+0000","last_active":"2024-04-02T07:35:54.615608+0000","last_peered":"2024-04-02T07:35:54.615608+0000","last_clean":"2024-04-02T07:35:54.615608+0000","last_became_active":"2024-04-02T07:35:53.264897+0000","last_became_peered":"2024-04-02T07:35:53.264897+0000","last_unstale":"2024-04-02T07:35:54.615608+0000","last_undegraded":"2024-04-02T07:35:54.615608+0000","last_fullsized":"2024-04-02T07:35:54.615608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:35.709321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616800+0000","last_change":"2024-04-02T07:35:52.767385+0000","last_active":"2024-04-02T07:35:54.616800+0000","last_peered":"2024-04-02T07:35:54.616800+0000","last_clean":"2024-04-02T07:35:54.616800+0000","last_became_active":"2024-04-02T07:35:52.767027+0000","last_became_peered":"2024-04-02T07:35:52.767027+0000","last_unstale":"2024-04-02T07:35:54.616800+0000","last_undegraded":"2024-04-02T07:35:54.616800+0000","last_fullsized":"2024-04-02T07:35:54.616800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616462+0000","last_change":"2024-04-02T07:35:53.283307+0000","last_active":"2024-04-02T07:35:54.616462+0000","last_peered":"2024-04-02T07:35:54.616462+0000","last_clean":"2024-04-02T07:35:54.616462+0000","last_became_active":"2024-04-02T07:35:53.282062+0000","last_became_peered":"2024-04-02T07:35:53.282062+0000","last_unstale":"2024-04-02T07:35:54.616462+0000","last_undegraded":"2024-04-02T07:35:54.616462+0000","last_fullsized":"2024-04-02T07:35:54.616462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:24.367224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246052+0000","last_change":"2024-04-02T07:35:52.753164+0000","last_active":"2024-04-02T07:35:54.246052+0000","last_peered":"2024-04-02T07:35:54.246052+0000","last_clean":"2024-04-02T07:35:54.246052+0000","last_became_active":"2024-04-02T07:35:52.752227+0000","last_became_peered":"2024-04-02T07:35:52.752227+0000","last_unstale":"2024-04-02T07:35:54.246052+0000","last_undegraded":"2024-04-02T07:35:54.246052+0000","last_fullsized":"2024-04-02T07:35:54.246052+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:07.715585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246205+0000","last_change":"2024-04-02T07:35:53.269115+0000","last_active":"2024-04-02T07:35:54.246205+0000","last_peered":"2024-04-02T07:35:54.246205+0000","last_clean":"2024-04-02T07:35:54.246205+0000","last_became_active":"2024-04-02T07:35:53.268815+0000","last_became_peered":"2024-04-02T07:35:53.268815+0000","last_unstale":"2024-04-02T07:35:54.246205+0000","last_undegraded":"2024-04-02T07:35:54.246205+0000","last_fullsized":"2024-04-02T07:35:54.246205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:46:59.080243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615336+0000","last_change":"2024-04-02T07:35:52.764457+0000","last_active":"2024-04-02T07:35:54.615336+0000","last_peered":"2024-04-02T07:35:54.615336+0000","last_clean":"2024-04-02T07:35:54.615336+0000","last_became_active":"2024-04-02T07:35:52.764002+0000","last_became_peered":"2024-04-02T07:35:52.764002+0000","last_unstale":"2024-04-02T07:35:54.615336+0000","last_undegraded":"2024-04-02T07:35:54.615336+0000","last_fullsized":"2024-04-02T07:35:54.615336+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:38:44.437274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246412+0000","last_change":"2024-04-02T07:35:53.262284+0000","last_active":"2024-04-02T07:35:54.246412+0000","last_peered":"2024-04-02T07:35:54.246412+0000","last_clean":"2024-04-02T07:35:54.246412+0000","last_became_active":"2024-04-02T07:35:53.260682+0000","last_became_peered":"2024-04-02T07:35:53.260682+0000","last_unstale":"2024-04-02T07:35:54.246412+0000","last_undegraded":"2024-04-02T07:35:54.246412+0000","last_fullsized":"2024-04-02T07:35:54.246412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:55.943715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246893+0000","last_change":"2024-04-02T07:35:52.754933+0000","last_active":"2024-04-02T07:35:54.246893+0000","last_peered":"2024-04-02T07:35:54.246893+0000","last_clean":"2024-04-02T07:35:54.246893+0000","last_became_active":"2024-04-02T07:35:52.750245+0000","last_became_peered":"2024-04-02T07:35:52.750245+0000","last_unstale":"2024-04-02T07:35:54.246893+0000","last_undegraded":"2024-04-02T07:35:54.246893+0000","last_fullsized":"2024-04-02T07:35:54.246893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245859+0000","last_change":"2024-04-02T07:35:53.266981+0000","last_active":"2024-04-02T07:35:54.245859+0000","last_peered":"2024-04-02T07:35:54.245859+0000","last_clean":"2024-04-02T07:35:54.245859+0000","last_became_active":"2024-04-02T07:35:53.266194+0000","last_became_peered":"2024-04-02T07:35:53.266194+0000","last_unstale":"2024-04-02T07:35:54.245859+0000","last_undegraded":"2024-04-02T07:35:54.245859+0000","last_fullsized":"2024-04-02T07:35:54.245859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:22:27.488539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615714+0000","last_change":"2024-04-02T07:35:52.771817+0000","last_active":"2024-04-02T07:35:54.615714+0000","last_peered":"2024-04-02T07:35:54.615714+0000","last_clean":"2024-04-02T07:35:54.615714+0000","last_became_active":"2024-04-02T07:35:52.771315+0000","last_became_peered":"2024-04-02T07:35:52.771315+0000","last_unstale":"2024-04-02T07:35:54.615714+0000","last_undegraded":"2024-04-02T07:35:54.615714+0000","last_fullsized":"2024-04-02T07:35:54.615714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:07.324155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557469+0000","last_change":"2024-04-02T07:35:53.254431+0000","last_active":"2024-04-02T07:35:54.557469+0000","last_peered":"2024-04-02T07:35:54.557469+0000","last_clean":"2024-04-02T07:35:54.557469+0000","last_became_active":"2024-04-02T07:35:53.253582+0000","last_became_peered":"2024-04-02T07:35:53.253582+0000","last_unstale":"2024-04-02T07:35:54.557469+0000","last_undegraded":"2024-04-02T07:35:54.557469+0000","last_fullsized":"2024-04-02T07:35:54.557469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:32.250824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617486+0000","last_change":"2024-04-02T07:35:52.761017+0000","last_active":"2024-04-02T07:35:54.617486+0000","last_peered":"2024-04-02T07:35:54.617486+0000","last_clean":"2024-04-02T07:35:54.617486+0000","last_became_active":"2024-04-02T07:35:52.760290+0000","last_became_peered":"2024-04-02T07:35:52.760290+0000","last_unstale":"2024-04-02T07:35:54.617486+0000","last_undegraded":"2024-04-02T07:35:54.617486+0000","last_fullsized":"2024-04-02T07:35:54.617486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:45.255736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557174+0000","last_change":"2024-04-02T07:35:53.263108+0000","last_active":"2024-04-02T07:35:54.557174+0000","last_peered":"2024-04-02T07:35:54.557174+0000","last_clean":"2024-04-02T07:35:54.557174+0000","last_became_active":"2024-04-02T07:35:53.262111+0000","last_became_peered":"2024-04-02T07:35:53.262111+0000","last_unstale":"2024-04-02T07:35:54.557174+0000","last_undegraded":"2024-04-02T07:35:54.557174+0000","last_fullsized":"2024-04-02T07:35:54.557174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:27.051276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245974+0000","last_change":"2024-04-02T07:35:52.763712+0000","last_active":"2024-04-02T07:35:54.245974+0000","last_peered":"2024-04-02T07:35:54.245974+0000","last_clean":"2024-04-02T07:35:54.245974+0000","last_became_active":"2024-04-02T07:35:52.763457+0000","last_became_peered":"2024-04-02T07:35:52.763457+0000","last_unstale":"2024-04-02T07:35:54.245974+0000","last_undegraded":"2024-04-02T07:35:54.245974+0000","last_fullsized":"2024-04-02T07:35:54.245974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:41.511901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615482+0000","last_change":"2024-04-02T07:35:53.254132+0000","last_active":"2024-04-02T07:35:54.615482+0000","last_peered":"2024-04-02T07:35:54.615482+0000","last_clean":"2024-04-02T07:35:54.615482+0000","last_became_active":"2024-04-02T07:35:53.253705+0000","last_became_peered":"2024-04-02T07:35:53.253705+0000","last_unstale":"2024-04-02T07:35:54.615482+0000","last_undegraded":"2024-04-02T07:35:54.615482+0000","last_fullsized":"2024-04-02T07:35:54.615482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:17.569969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617123+0000","last_change":"2024-04-02T07:35:53.267551+0000","last_active":"2024-04-02T07:35:54.617123+0000","last_peered":"2024-04-02T07:35:54.617123+0000","last_clean":"2024-04-02T07:35:54.617123+0000","last_became_active":"2024-04-02T07:35:53.263849+0000","last_became_peered":"2024-04-02T07:35:53.263849+0000","last_unstale":"2024-04-02T07:35:54.617123+0000","last_undegraded":"2024-04-02T07:35:54.617123+0000","last_fullsized":"2024-04-02T07:35:54.617123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:54:04.724188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616708+0000","last_change":"2024-04-02T07:35:52.764685+0000","last_active":"2024-04-02T07:35:54.616708+0000","last_peered":"2024-04-02T07:35:54.616708+0000","last_clean":"2024-04-02T07:35:54.616708+0000","last_became_active":"2024-04-02T07:35:52.764421+0000","last_became_peered":"2024-04-02T07:35:52.764421+0000","last_unstale":"2024-04-02T07:35:54.616708+0000","last_undegraded":"2024-04-02T07:35:54.616708+0000","last_fullsized":"2024-04-02T07:35:54.616708+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247020+0000","last_change":"2024-04-02T07:35:53.264778+0000","last_active":"2024-04-02T07:35:54.247020+0000","last_peered":"2024-04-02T07:35:54.247020+0000","last_clean":"2024-04-02T07:35:54.247020+0000","last_became_active":"2024-04-02T07:35:53.264445+0000","last_became_peered":"2024-04-02T07:35:53.264445+0000","last_unstale":"2024-04-02T07:35:54.247020+0000","last_undegraded":"2024-04-02T07:35:54.247020+0000","last_fullsized":"2024-04-02T07:35:54.247020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:00:20.419421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246196+0000","last_change":"2024-04-02T07:35:52.767065+0000","last_active":"2024-04-02T07:35:54.246196+0000","last_peered":"2024-04-02T07:35:54.246196+0000","last_clean":"2024-04-02T07:35:54.246196+0000","last_became_active":"2024-04-02T07:35:52.764931+0000","last_became_peered":"2024-04-02T07:35:52.764931+0000","last_unstale":"2024-04-02T07:35:54.246196+0000","last_undegraded":"2024-04-02T07:35:54.246196+0000","last_fullsized":"2024-04-02T07:35:54.246196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:05.916519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-02T07:35:53.250580+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:35:53.250580+0000","last_peered":"2024-04-02T07:35:53.250580+0000","last_clean":"2024-04-02T07:35:53.250580+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:35:53.250580+0000","last_undegraded":"2024-04-02T07:35:53.250580+0000","last_fullsized":"2024-04-02T07:35:53.250580+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616550+0000","last_change":"2024-04-02T07:35:53.283251+0000","last_active":"2024-04-02T07:35:54.616550+0000","last_peered":"2024-04-02T07:35:54.616550+0000","last_clean":"2024-04-02T07:35:54.616550+0000","last_became_active":"2024-04-02T07:35:53.276591+0000","last_became_peered":"2024-04-02T07:35:53.276591+0000","last_unstale":"2024-04-02T07:35:54.616550+0000","last_undegraded":"2024-04-02T07:35:54.616550+0000","last_fullsized":"2024-04-02T07:35:54.616550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:07.528924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245695+0000","last_change":"2024-04-02T07:35:52.764279+0000","last_active":"2024-04-02T07:35:54.245695+0000","last_peered":"2024-04-02T07:35:54.245695+0000","last_clean":"2024-04-02T07:35:54.245695+0000","last_became_active":"2024-04-02T07:35:52.759649+0000","last_became_peered":"2024-04-02T07:35:52.759649+0000","last_unstale":"2024-04-02T07:35:54.245695+0000","last_undegraded":"2024-04-02T07:35:54.245695+0000","last_fullsized":"2024-04-02T07:35:54.245695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:54.937081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246537+0000","last_change":"2024-04-02T07:35:52.756573+0000","last_active":"2024-04-02T07:35:54.246537+0000","last_peered":"2024-04-02T07:35:54.246537+0000","last_clean":"2024-04-02T07:35:54.246537+0000","last_became_active":"2024-04-02T07:35:52.756192+0000","last_became_peered":"2024-04-02T07:35:52.756192+0000","last_unstale":"2024-04-02T07:35:54.246537+0000","last_undegraded":"2024-04-02T07:35:54.246537+0000","last_fullsized":"2024-04-02T07:35:54.246537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616866+0000","last_change":"2024-04-02T07:35:53.269008+0000","last_active":"2024-04-02T07:35:54.616866+0000","last_peered":"2024-04-02T07:35:54.616866+0000","last_clean":"2024-04-02T07:35:54.616866+0000","last_became_active":"2024-04-02T07:35:53.268663+0000","last_became_peered":"2024-04-02T07:35:53.268663+0000","last_unstale":"2024-04-02T07:35:54.616866+0000","last_undegraded":"2024-04-02T07:35:54.616866+0000","last_fullsized":"2024-04-02T07:35:54.616866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:05:41.161422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247184+0000","last_change":"2024-04-02T07:35:52.769059+0000","last_active":"2024-04-02T07:35:54.247184+0000","last_peered":"2024-04-02T07:35:54.247184+0000","last_clean":"2024-04-02T07:35:54.247184+0000","last_became_active":"2024-04-02T07:35:52.768711+0000","last_became_peered":"2024-04-02T07:35:52.768711+0000","last_unstale":"2024-04-02T07:35:54.247184+0000","last_undegraded":"2024-04-02T07:35:54.247184+0000","last_fullsized":"2024-04-02T07:35:54.247184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247358+0000","last_change":"2024-04-02T07:35:53.275607+0000","last_active":"2024-04-02T07:35:54.247358+0000","last_peered":"2024-04-02T07:35:54.247358+0000","last_clean":"2024-04-02T07:35:54.247358+0000","last_became_active":"2024-04-02T07:35:53.274591+0000","last_became_peered":"2024-04-02T07:35:53.274591+0000","last_unstale":"2024-04-02T07:35:54.247358+0000","last_undegraded":"2024-04-02T07:35:54.247358+0000","last_fullsized":"2024-04-02T07:35:54.247358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:13:58.887392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247393+0000","last_change":"2024-04-02T07:35:52.771999+0000","last_active":"2024-04-02T07:35:54.247393+0000","last_peered":"2024-04-02T07:35:54.247393+0000","last_clean":"2024-04-02T07:35:54.247393+0000","last_became_active":"2024-04-02T07:35:52.771720+0000","last_became_peered":"2024-04-02T07:35:52.771720+0000","last_unstale":"2024-04-02T07:35:54.247393+0000","last_undegraded":"2024-04-02T07:35:54.247393+0000","last_fullsized":"2024-04-02T07:35:54.247393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:31.244191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247153+0000","last_change":"2024-04-02T07:35:53.258654+0000","last_active":"2024-04-02T07:35:54.247153+0000","last_peered":"2024-04-02T07:35:54.247153+0000","last_clean":"2024-04-02T07:35:54.247153+0000","last_became_active":"2024-04-02T07:35:53.257658+0000","last_became_peered":"2024-04-02T07:35:53.257658+0000","last_unstale":"2024-04-02T07:35:54.247153+0000","last_undegraded":"2024-04-02T07:35:54.247153+0000","last_fullsized":"2024-04-02T07:35:54.247153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:07.528924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615282+0000","last_change":"2024-04-02T07:35:52.764308+0000","last_active":"2024-04-02T07:35:54.615282+0000","last_peered":"2024-04-02T07:35:54.615282+0000","last_clean":"2024-04-02T07:35:54.615282+0000","last_became_active":"2024-04-02T07:35:52.763178+0000","last_became_peered":"2024-04-02T07:35:52.763178+0000","last_unstale":"2024-04-02T07:35:54.615282+0000","last_undegraded":"2024-04-02T07:35:54.615282+0000","last_fullsized":"2024-04-02T07:35:54.615282+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:45.255736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245826+0000","last_change":"2024-04-02T07:35:53.258575+0000","last_active":"2024-04-02T07:35:54.245826+0000","last_peered":"2024-04-02T07:35:54.245826+0000","last_clean":"2024-04-02T07:35:54.245826+0000","last_became_active":"2024-04-02T07:35:53.258326+0000","last_became_peered":"2024-04-02T07:35:53.258326+0000","last_unstale":"2024-04-02T07:35:54.245826+0000","last_undegraded":"2024-04-02T07:35:54.245826+0000","last_fullsized":"2024-04-02T07:35:54.245826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:16.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247051+0000","last_change":"2024-04-02T07:35:52.771988+0000","last_active":"2024-04-02T07:35:54.247051+0000","last_peered":"2024-04-02T07:35:54.247051+0000","last_clean":"2024-04-02T07:35:54.247051+0000","last_became_active":"2024-04-02T07:35:52.771616+0000","last_became_peered":"2024-04-02T07:35:52.771616+0000","last_unstale":"2024-04-02T07:35:54.247051+0000","last_undegraded":"2024-04-02T07:35:54.247051+0000","last_fullsized":"2024-04-02T07:35:54.247051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:45.255736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246904+0000","last_change":"2024-04-02T07:35:53.258346+0000","last_active":"2024-04-02T07:35:54.246904+0000","last_peered":"2024-04-02T07:35:54.246904+0000","last_clean":"2024-04-02T07:35:54.246904+0000","last_became_active":"2024-04-02T07:35:53.258044+0000","last_became_peered":"2024-04-02T07:35:53.258044+0000","last_unstale":"2024-04-02T07:35:54.246904+0000","last_undegraded":"2024-04-02T07:35:54.246904+0000","last_fullsized":"2024-04-02T07:35:54.246904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:50.551591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616574+0000","last_change":"2024-04-02T07:35:52.758632+0000","last_active":"2024-04-02T07:35:54.616574+0000","last_peered":"2024-04-02T07:35:54.616574+0000","last_clean":"2024-04-02T07:35:54.616574+0000","last_became_active":"2024-04-02T07:35:52.758347+0000","last_became_peered":"2024-04-02T07:35:52.758347+0000","last_unstale":"2024-04-02T07:35:54.616574+0000","last_undegraded":"2024-04-02T07:35:54.616574+0000","last_fullsized":"2024-04-02T07:35:54.616574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617044+0000","last_change":"2024-04-02T07:35:53.271892+0000","last_active":"2024-04-02T07:35:54.617044+0000","last_peered":"2024-04-02T07:35:54.617044+0000","last_clean":"2024-04-02T07:35:54.617044+0000","last_became_active":"2024-04-02T07:35:53.271472+0000","last_became_peered":"2024-04-02T07:35:53.271472+0000","last_unstale":"2024-04-02T07:35:54.617044+0000","last_undegraded":"2024-04-02T07:35:54.617044+0000","last_fullsized":"2024-04-02T07:35:54.617044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:06.923152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246267+0000","last_change":"2024-04-02T07:35:52.758627+0000","last_active":"2024-04-02T07:35:54.246267+0000","last_peered":"2024-04-02T07:35:54.246267+0000","last_clean":"2024-04-02T07:35:54.246267+0000","last_became_active":"2024-04-02T07:35:52.758330+0000","last_became_peered":"2024-04-02T07:35:52.758330+0000","last_unstale":"2024-04-02T07:35:54.246267+0000","last_undegraded":"2024-04-02T07:35:54.246267+0000","last_fullsized":"2024-04-02T07:35:54.246267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:23.360591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245991+0000","last_change":"2024-04-02T07:35:53.266441+0000","last_active":"2024-04-02T07:35:54.245991+0000","last_peered":"2024-04-02T07:35:54.245991+0000","last_clean":"2024-04-02T07:35:54.245991+0000","last_became_active":"2024-04-02T07:35:53.265569+0000","last_became_peered":"2024-04-02T07:35:53.265569+0000","last_unstale":"2024-04-02T07:35:54.245991+0000","last_undegraded":"2024-04-02T07:35:54.245991+0000","last_fullsized":"2024-04-02T07:35:54.245991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:53.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616939+0000","last_change":"2024-04-02T07:35:52.762509+0000","last_active":"2024-04-02T07:35:54.616939+0000","last_peered":"2024-04-02T07:35:54.616939+0000","last_clean":"2024-04-02T07:35:54.616939+0000","last_became_active":"2024-04-02T07:35:52.762190+0000","last_became_peered":"2024-04-02T07:35:52.762190+0000","last_unstale":"2024-04-02T07:35:54.616939+0000","last_undegraded":"2024-04-02T07:35:54.616939+0000","last_fullsized":"2024-04-02T07:35:54.616939+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246626+0000","last_change":"2024-04-02T07:35:53.272664+0000","last_active":"2024-04-02T07:35:54.246626+0000","last_peered":"2024-04-02T07:35:54.246626+0000","last_clean":"2024-04-02T07:35:54.246626+0000","last_became_active":"2024-04-02T07:35:53.272408+0000","last_became_peered":"2024-04-02T07:35:53.272408+0000","last_unstale":"2024-04-02T07:35:54.246626+0000","last_undegraded":"2024-04-02T07:35:54.246626+0000","last_fullsized":"2024-04-02T07:35:54.246626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:18.703428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617830+0000","last_change":"2024-04-02T07:35:52.760241+0000","last_active":"2024-04-02T07:35:54.617830+0000","last_peered":"2024-04-02T07:35:54.617830+0000","last_clean":"2024-04-02T07:35:54.617830+0000","last_became_active":"2024-04-02T07:35:52.759523+0000","last_became_peered":"2024-04-02T07:35:52.759523+0000","last_unstale":"2024-04-02T07:35:54.617830+0000","last_undegraded":"2024-04-02T07:35:54.617830+0000","last_fullsized":"2024-04-02T07:35:54.617830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:07.324155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615417+0000","last_change":"2024-04-02T07:35:53.252444+0000","last_active":"2024-04-02T07:35:54.615417+0000","last_peered":"2024-04-02T07:35:54.615417+0000","last_clean":"2024-04-02T07:35:54.615417+0000","last_became_active":"2024-04-02T07:35:53.252160+0000","last_became_peered":"2024-04-02T07:35:53.252160+0000","last_unstale":"2024-04-02T07:35:54.615417+0000","last_undegraded":"2024-04-02T07:35:54.615417+0000","last_fullsized":"2024-04-02T07:35:54.615417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:18:02.329099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247287+0000","last_change":"2024-04-02T07:35:52.760163+0000","last_active":"2024-04-02T07:35:54.247287+0000","last_peered":"2024-04-02T07:35:54.247287+0000","last_clean":"2024-04-02T07:35:54.247287+0000","last_became_active":"2024-04-02T07:35:52.759785+0000","last_became_peered":"2024-04-02T07:35:52.759785+0000","last_unstale":"2024-04-02T07:35:54.247287+0000","last_undegraded":"2024-04-02T07:35:54.247287+0000","last_fullsized":"2024-04-02T07:35:54.247287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615799+0000","last_change":"2024-04-02T07:35:53.259830+0000","last_active":"2024-04-02T07:35:54.615799+0000","last_peered":"2024-04-02T07:35:54.615799+0000","last_clean":"2024-04-02T07:35:54.615799+0000","last_became_active":"2024-04-02T07:35:53.259518+0000","last_became_peered":"2024-04-02T07:35:53.259518+0000","last_unstale":"2024-04-02T07:35:54.615799+0000","last_undegraded":"2024-04-02T07:35:54.615799+0000","last_fullsized":"2024-04-02T07:35:54.615799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:54:41.930328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.556975+0000","last_change":"2024-04-02T07:35:53.263050+0000","last_active":"2024-04-02T07:35:54.556975+0000","last_peered":"2024-04-02T07:35:54.556975+0000","last_clean":"2024-04-02T07:35:54.556975+0000","last_became_active":"2024-04-02T07:35:53.262790+0000","last_became_peered":"2024-04-02T07:35:53.262790+0000","last_unstale":"2024-04-02T07:35:54.556975+0000","last_undegraded":"2024-04-02T07:35:54.556975+0000","last_fullsized":"2024-04-02T07:35:54.556975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:42.518534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616668+0000","last_change":"2024-04-02T07:35:53.269409+0000","last_active":"2024-04-02T07:35:54.616668+0000","last_peered":"2024-04-02T07:35:54.616668+0000","last_clean":"2024-04-02T07:35:54.616668+0000","last_became_active":"2024-04-02T07:35:53.267892+0000","last_became_peered":"2024-04-02T07:35:53.267892+0000","last_unstale":"2024-04-02T07:35:54.616668+0000","last_undegraded":"2024-04-02T07:35:54.616668+0000","last_fullsized":"2024-04-02T07:35:54.616668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:08.722219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245810+0000","last_change":"2024-04-02T07:35:53.260570+0000","last_active":"2024-04-02T07:35:54.245810+0000","last_peered":"2024-04-02T07:35:54.245810+0000","last_clean":"2024-04-02T07:35:54.245810+0000","last_became_active":"2024-04-02T07:35:53.260265+0000","last_became_peered":"2024-04-02T07:35:53.260265+0000","last_unstale":"2024-04-02T07:35:54.245810+0000","last_undegraded":"2024-04-02T07:35:54.245810+0000","last_fullsized":"2024-04-02T07:35:54.245810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:13:55.860837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617586+0000","last_change":"2024-04-02T07:35:53.260933+0000","last_active":"2024-04-02T07:35:54.617586+0000","last_peered":"2024-04-02T07:35:54.617586+0000","last_clean":"2024-04-02T07:35:54.617586+0000","last_became_active":"2024-04-02T07:35:53.260662+0000","last_became_peered":"2024-04-02T07:35:53.260662+0000","last_unstale":"2024-04-02T07:35:54.617586+0000","last_undegraded":"2024-04-02T07:35:54.617586+0000","last_fullsized":"2024-04-02T07:35:54.617586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:16.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616780+0000","last_change":"2024-04-02T07:35:53.274324+0000","last_active":"2024-04-02T07:35:54.616780+0000","last_peered":"2024-04-02T07:35:54.616780+0000","last_clean":"2024-04-02T07:35:54.616780+0000","last_became_active":"2024-04-02T07:35:53.274049+0000","last_became_peered":"2024-04-02T07:35:53.274049+0000","last_unstale":"2024-04-02T07:35:54.616780+0000","last_undegraded":"2024-04-02T07:35:54.616780+0000","last_fullsized":"2024-04-02T07:35:54.616780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:54:04.724188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616584+0000","last_change":"2024-04-02T07:35:53.272852+0000","last_active":"2024-04-02T07:35:54.616584+0000","last_peered":"2024-04-02T07:35:54.616584+0000","last_clean":"2024-04-02T07:35:54.616584+0000","last_became_active":"2024-04-02T07:35:53.272036+0000","last_became_peered":"2024-04-02T07:35:53.272036+0000","last_unstale":"2024-04-02T07:35:54.616584+0000","last_undegraded":"2024-04-02T07:35:54.616584+0000","last_fullsized":"2024-04-02T07:35:54.616584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:50.551591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557432+0000","last_change":"2024-04-02T07:35:53.275573+0000","last_active":"2024-04-02T07:35:54.557432+0000","last_peered":"2024-04-02T07:35:54.557432+0000","last_clean":"2024-04-02T07:35:54.557432+0000","last_became_active":"2024-04-02T07:35:53.272553+0000","last_became_peered":"2024-04-02T07:35:53.272553+0000","last_unstale":"2024-04-02T07:35:54.557432+0000","last_undegraded":"2024-04-02T07:35:54.557432+0000","last_fullsized":"2024-04-02T07:35:54.557432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:06.923152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617508+0000","last_change":"2024-04-02T07:35:53.262359+0000","last_active":"2024-04-02T07:35:54.617508+0000","last_peered":"2024-04-02T07:35:54.617508+0000","last_clean":"2024-04-02T07:35:54.617508+0000","last_became_active":"2024-04-02T07:35:53.262038+0000","last_became_peered":"2024-04-02T07:35:53.262038+0000","last_unstale":"2024-04-02T07:35:54.617508+0000","last_undegraded":"2024-04-02T07:35:54.617508+0000","last_fullsized":"2024-04-02T07:35:54.617508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:18.703428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245590+0000","last_change":"2024-04-02T07:35:53.284523+0000","last_active":"2024-04-02T07:35:54.245590+0000","last_peered":"2024-04-02T07:35:54.245590+0000","last_clean":"2024-04-02T07:35:54.245590+0000","last_became_active":"2024-04-02T07:35:53.284200+0000","last_became_peered":"2024-04-02T07:35:53.284200+0000","last_unstale":"2024-04-02T07:35:54.245590+0000","last_undegraded":"2024-04-02T07:35:54.245590+0000","last_fullsized":"2024-04-02T07:35:54.245590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:14:43.088676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617379+0000","last_change":"2024-04-02T07:35:53.259326+0000","last_active":"2024-04-02T07:35:54.617379+0000","last_peered":"2024-04-02T07:35:54.617379+0000","last_clean":"2024-04-02T07:35:54.617379+0000","last_became_active":"2024-04-02T07:35:53.259023+0000","last_became_peered":"2024-04-02T07:35:53.259023+0000","last_unstale":"2024-04-02T07:35:54.617379+0000","last_undegraded":"2024-04-02T07:35:54.617379+0000","last_fullsized":"2024-04-02T07:35:54.617379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:54:41.930328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246691+0000","last_change":"2024-04-02T07:35:53.261005+0000","last_active":"2024-04-02T07:35:54.246691+0000","last_peered":"2024-04-02T07:35:54.246691+0000","last_clean":"2024-04-02T07:35:54.246691+0000","last_became_active":"2024-04-02T07:35:53.259618+0000","last_became_peered":"2024-04-02T07:35:53.259618+0000","last_unstale":"2024-04-02T07:35:54.246691+0000","last_undegraded":"2024-04-02T07:35:54.246691+0000","last_fullsized":"2024-04-02T07:35:54.246691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:24.367224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247226+0000","last_change":"2024-04-02T07:35:53.262531+0000","last_active":"2024-04-02T07:35:54.247226+0000","last_peered":"2024-04-02T07:35:54.247226+0000","last_clean":"2024-04-02T07:35:54.247226+0000","last_became_active":"2024-04-02T07:35:53.262196+0000","last_became_peered":"2024-04-02T07:35:53.262196+0000","last_unstale":"2024-04-02T07:35:54.247226+0000","last_undegraded":"2024-04-02T07:35:54.247226+0000","last_fullsized":"2024-04-02T07:35:54.247226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:54:04.724188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615537+0000","last_change":"2024-04-02T07:35:53.266761+0000","last_active":"2024-04-02T07:35:54.615537+0000","last_peered":"2024-04-02T07:35:54.615537+0000","last_clean":"2024-04-02T07:35:54.615537+0000","last_became_active":"2024-04-02T07:35:53.264360+0000","last_became_peered":"2024-04-02T07:35:53.264360+0000","last_unstale":"2024-04-02T07:35:54.615537+0000","last_undegraded":"2024-04-02T07:35:54.615537+0000","last_fullsized":"2024-04-02T07:35:54.615537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:31.170936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":71480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":73134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":660339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:35:57.626 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:35:57.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.851+0000 7f88bb200700 1 Processor -- start 2024-04-02T07:35:57.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.851+0000 7f88bb200700 1 -- start start 2024-04-02T07:35:57.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.852+0000 7f88bb200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88b40ceec0 0x7f88b41612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:57.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.852+0000 7f88bb200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f88b4161800 0x7f88b40b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:57.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.852+0000 7f88bb200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b40d30d0 con 0x7f88b415d640 2024-04-02T07:35:57.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.852+0000 7f88bb200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b4161c90 con 0x7f88b4161800 2024-04-02T07:35:57.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.852+0000 7f88bb200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b40b6840 con 0x7f88b40ceec0 2024-04-02T07:35:57.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.852+0000 7f88b9200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f88b4161800 0x7f88b40b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:57.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.853+0000 7f88b9200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f88b4161800 0x7f88b40b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57914/0 (socket says 172.21.15.89:57914) 2024-04-02T07:35:57.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.853+0000 7f88b9200700 1 -- 172.21.15.89:0/873749270 learned_addr learned my addr 172.21.15.89:0/873749270 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:57.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.853+0000 7f88b3400700 1 --2- 172.21.15.89:0/873749270 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88b40ceec0 0x7f88b41612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:57.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.853+0000 7f88b9200700 1 -- 172.21.15.89:0/873749270 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88b40ceec0 msgr2=0x7f88b41612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:57.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.853+0000 7f88b9200700 1 --2- 172.21.15.89:0/873749270 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88b40ceec0 0x7f88b41612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.853+0000 7f88b9200700 1 -- 172.21.15.89:0/873749270 >> v1:172.21.15.113:6789/0 conn(0x7f88b415d640 legacy=0x7f88b40d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:57.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.853+0000 7f88b9200700 1 -- 172.21.15.89:0/873749270 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b40d2b30 con 0x7f88b4161800 2024-04-02T07:35:57.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.853+0000 7f88b3400700 1 --2- 172.21.15.89:0/873749270 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88b40ceec0 0x7f88b41612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:35:57.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.853+0000 7f88b9200700 1 --2- 172.21.15.89:0/873749270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4161800 0x7f88b40b6cd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009e50 tx=0x7f88ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afef48ad4d5e8c4f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:57.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.853+0000 7f88b2a00700 1 -- 172.21.15.89:0/873749270 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac00e070 con 0x7f88b4161800 2024-04-02T07:35:57.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.854+0000 7f88b2a00700 1 -- 172.21.15.89:0/873749270 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ac00b9c0 con 0x7f88b4161800 2024-04-02T07:35:57.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.854+0000 7f88b2a00700 1 -- 172.21.15.89:0/873749270 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac00bd70 con 0x7f88b4161800 2024-04-02T07:35:57.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.854+0000 7f88bb200700 1 -- 172.21.15.89:0/873749270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4161800 msgr2=0x7f88b40b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:57.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.854+0000 7f88bb200700 1 --2- 172.21.15.89:0/873749270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4161800 0x7f88b40b6cd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009e50 tx=0x7f88ac00b330 comp rx=0 tx=0).stop 2024-04-02T07:35:57.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.854+0000 7f88bb200700 1 -- 172.21.15.89:0/873749270 shutdown_connections 2024-04-02T07:35:57.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.854+0000 7f88bb200700 1 --2- 172.21.15.89:0/873749270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4161800 0x7f88b40b6cd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.854+0000 7f88bb200700 1 --2- 172.21.15.89:0/873749270 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88b40ceec0 0x7f88b41612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.854+0000 7f88bb200700 1 -- 172.21.15.89:0/873749270 >> 172.21.15.89:0/873749270 conn(0x7f88b408aff0 msgr2=0x7f88b40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:57.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.854+0000 7f88bb200700 1 -- 172.21.15.89:0/873749270 shutdown_connections 2024-04-02T07:35:57.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.854+0000 7f88bb200700 1 -- 172.21.15.89:0/873749270 wait complete. 2024-04-02T07:35:57.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.856+0000 7f88bb200700 1 Processor -- start 2024-04-02T07:35:57.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.856+0000 7f88bb200700 1 -- start start 2024-04-02T07:35:57.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.857+0000 7f88bb200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88b40ceec0 0x7f88b415ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:57.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.857+0000 7f88bb200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88b415d640 0x7f88b40ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:57.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.857+0000 7f88bb200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4161800 0x7f88b40cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:57.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.857+0000 7f88bb200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b415f670 con 0x7f88b4161800 2024-04-02T07:35:57.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.857+0000 7f88bb200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b40ce7e0 con 0x7f88b415d640 2024-04-02T07:35:57.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.857+0000 7f88bb200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b40cad20 con 0x7f88b40ceec0 2024-04-02T07:35:57.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.857+0000 7f88b9200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4161800 0x7f88b40cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:57.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.857+0000 7f88b9200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4161800 0x7f88b40cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57916/0 (socket says 172.21.15.89:57916) 2024-04-02T07:35:57.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.857+0000 7f88b9200700 1 -- 172.21.15.89:0/2356971910 learned_addr learned my addr 172.21.15.89:0/2356971910 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:57.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.857+0000 7f88b3e00700 1 --2- 172.21.15.89:0/2356971910 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88b40ceec0 0x7f88b415ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:57.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.857+0000 7f88b3400700 1 --2- 172.21.15.89:0/2356971910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88b415d640 0x7f88b40ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:57.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.857+0000 7f88b9200700 1 -- 172.21.15.89:0/2356971910 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88b40ceec0 msgr2=0x7f88b415ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:57.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.858+0000 7f88b9200700 1 --2- 172.21.15.89:0/2356971910 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88b40ceec0 0x7f88b415ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.858+0000 7f88b9200700 1 -- 172.21.15.89:0/2356971910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88b415d640 msgr2=0x7f88b40ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:57.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.858+0000 7f88b9200700 1 --2- 172.21.15.89:0/2356971910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88b415d640 0x7f88b40ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:57.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.858+0000 7f88b9200700 1 -- 172.21.15.89:0/2356971910 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ac009b30 con 0x7f88b4161800 2024-04-02T07:35:57.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.858+0000 7f88b9200700 1 --2- 172.21.15.89:0/2356971910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4161800 0x7f88b40cc180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f88b408c490 tx=0x7f88ac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:57.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.858+0000 7f88abe00700 1 -- 172.21.15.89:0/2356971910 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac004840 con 0x7f88b4161800 2024-04-02T07:35:57.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.858+0000 7f88abe00700 1 -- 172.21.15.89:0/2356971910 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ac0092e0 con 0x7f88b4161800 2024-04-02T07:35:57.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.858+0000 7f88bb200700 1 -- 172.21.15.89:0/2356971910 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b40cde40 con 0x7f88b4161800 2024-04-02T07:35:57.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.858+0000 7f88abe00700 1 -- 172.21.15.89:0/2356971910 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac01c660 con 0x7f88b4161800 2024-04-02T07:35:57.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.858+0000 7f88bb200700 1 -- 172.21.15.89:0/2356971910 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b40ce400 con 0x7f88b4161800 2024-04-02T07:35:57.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.859+0000 7f88bb200700 1 -- 172.21.15.89:0/2356971910 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b40ce920 con 0x7f88b4161800 2024-04-02T07:35:57.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.860+0000 7f88abe00700 1 -- 172.21.15.89:0/2356971910 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f88ac01c7c0 con 0x7f88b4161800 2024-04-02T07:35:57.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.861+0000 7f88abe00700 1 --2- 172.21.15.89:0/2356971910 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f889406ac90 0x7f889406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:57.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.861+0000 7f88abe00700 1 -- 172.21.15.89:0/2356971910 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88ac08af70 con 0x7f88b4161800 2024-04-02T07:35:57.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.861+0000 7f88b3e00700 1 --2- 172.21.15.89:0/2356971910 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f889406ac90 0x7f889406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:57.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.861+0000 7f88b3e00700 1 --2- 172.21.15.89:0/2356971910 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f889406ac90 0x7f889406d150 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f88a4007fd0 tx=0x7f88a4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:57.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:57.866+0000 7f88abe00700 1 -- 172.21.15.89:0/2356971910 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f88b40ce920 con 0x7f88b4161800 2024-04-02T07:35:58.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.062+0000 7f88bb200700 1 -- 172.21.15.89:0/2356971910 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f88b40b64d0 con 0x7f889406ac90 2024-04-02T07:35:58.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.070+0000 7f88abe00700 1 -- 172.21.15.89:0/2356971910 <== mgr.4108 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264698 (secure 0 0 0) 0x7f88a4001560 con 0x7f889406ac90 2024-04-02T07:35:58.075 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:35:58.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.074+0000 7f88bb200700 1 -- 172.21.15.89:0/2356971910 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f889406ac90 msgr2=0x7f889406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:58.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.074+0000 7f88bb200700 1 --2- 172.21.15.89:0/2356971910 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f889406ac90 0x7f889406d150 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f88a4007fd0 tx=0x7f88a4009040 comp rx=0 tx=0).stop 2024-04-02T07:35:58.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.074+0000 7f88bb200700 1 -- 172.21.15.89:0/2356971910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4161800 msgr2=0x7f88b40cc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:58.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.074+0000 7f88bb200700 1 --2- 172.21.15.89:0/2356971910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4161800 0x7f88b40cc180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f88b408c490 tx=0x7f88ac004ad0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.074+0000 7f88bb200700 1 -- 172.21.15.89:0/2356971910 shutdown_connections 2024-04-02T07:35:58.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.074+0000 7f88bb200700 1 --2- 172.21.15.89:0/2356971910 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f889406ac90 0x7f889406d150 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.074+0000 7f88bb200700 1 --2- 172.21.15.89:0/2356971910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88b4161800 0x7f88b40cc180 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.074+0000 7f88bb200700 1 --2- 172.21.15.89:0/2356971910 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88b40ceec0 0x7f88b415ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.074+0000 7f88bb200700 1 --2- 172.21.15.89:0/2356971910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88b415d640 0x7f88b40ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.074+0000 7f88bb200700 1 -- 172.21.15.89:0/2356971910 >> 172.21.15.89:0/2356971910 conn(0x7f88b408aff0 msgr2=0x7f88b406cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:58.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.074+0000 7f88bb200700 1 -- 172.21.15.89:0/2356971910 shutdown_connections 2024-04-02T07:35:58.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.074+0000 7f88bb200700 1 -- 172.21.15.89:0/2356971910 wait complete. 2024-04-02T07:35:58.078 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:35:58.093 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-04-02T07:35:57.183599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219000,"kb_used_data":2816,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755720,"statfs":{"total":773094113280,"available":772869857280,"internally_reserved":0,"allocated":2883584,"data_stored":1750606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.452801"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615693+0000","last_change":"2024-04-02T07:35:53.278052+0000","last_active":"2024-04-02T07:35:54.615693+0000","last_peered":"2024-04-02T07:35:54.615693+0000","last_clean":"2024-04-02T07:35:54.615693+0000","last_became_active":"2024-04-02T07:35:53.277648+0000","last_became_peered":"2024-04-02T07:35:53.277648+0000","last_unstale":"2024-04-02T07:35:54.615693+0000","last_undegraded":"2024-04-02T07:35:54.615693+0000","last_fullsized":"2024-04-02T07:35:54.615693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.380856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617752+0000","last_change":"2024-04-02T07:35:53.271744+0000","last_active":"2024-04-02T07:35:54.617752+0000","last_peered":"2024-04-02T07:35:54.617752+0000","last_clean":"2024-04-02T07:35:54.617752+0000","last_became_active":"2024-04-02T07:35:53.271453+0000","last_became_peered":"2024-04-02T07:35:53.271453+0000","last_unstale":"2024-04-02T07:35:54.617752+0000","last_undegraded":"2024-04-02T07:35:54.617752+0000","last_fullsized":"2024-04-02T07:35:54.617752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:41:21.403396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617763+0000","last_change":"2024-04-02T07:35:53.257746+0000","last_active":"2024-04-02T07:35:54.617763+0000","last_peered":"2024-04-02T07:35:54.617763+0000","last_clean":"2024-04-02T07:35:54.617763+0000","last_became_active":"2024-04-02T07:35:53.257428+0000","last_became_peered":"2024-04-02T07:35:53.257428+0000","last_unstale":"2024-04-02T07:35:54.617763+0000","last_undegraded":"2024-04-02T07:35:54.617763+0000","last_fullsized":"2024-04-02T07:35:54.617763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:13:04.549460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557285+0000","last_change":"2024-04-02T07:35:53.269314+0000","last_active":"2024-04-02T07:35:54.557285+0000","last_peered":"2024-04-02T07:35:54.557285+0000","last_clean":"2024-04-02T07:35:54.557285+0000","last_became_active":"2024-04-02T07:35:53.269081+0000","last_became_peered":"2024-04-02T07:35:53.269081+0000","last_unstale":"2024-04-02T07:35:54.557285+0000","last_undegraded":"2024-04-02T07:35:54.557285+0000","last_fullsized":"2024-04-02T07:35:54.557285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:24.367224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615439+0000","last_change":"2024-04-02T07:35:53.257006+0000","last_active":"2024-04-02T07:35:54.615439+0000","last_peered":"2024-04-02T07:35:54.615439+0000","last_clean":"2024-04-02T07:35:54.615439+0000","last_became_active":"2024-04-02T07:35:53.256623+0000","last_became_peered":"2024-04-02T07:35:53.256623+0000","last_unstale":"2024-04-02T07:35:54.615439+0000","last_undegraded":"2024-04-02T07:35:54.615439+0000","last_fullsized":"2024-04-02T07:35:54.615439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:13:04.549460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557080+0000","last_change":"2024-04-02T07:35:53.256730+0000","last_active":"2024-04-02T07:35:54.557080+0000","last_peered":"2024-04-02T07:35:54.557080+0000","last_clean":"2024-04-02T07:35:54.557080+0000","last_became_active":"2024-04-02T07:35:53.256395+0000","last_became_peered":"2024-04-02T07:35:53.256395+0000","last_unstale":"2024-04-02T07:35:54.557080+0000","last_undegraded":"2024-04-02T07:35:54.557080+0000","last_fullsized":"2024-04-02T07:35:54.557080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:55.943715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615284+0000","last_change":"2024-04-02T07:35:53.267364+0000","last_active":"2024-04-02T07:35:54.615284+0000","last_peered":"2024-04-02T07:35:54.615284+0000","last_clean":"2024-04-02T07:35:54.615284+0000","last_became_active":"2024-04-02T07:35:53.267076+0000","last_became_peered":"2024-04-02T07:35:53.267076+0000","last_unstale":"2024-04-02T07:35:54.615284+0000","last_undegraded":"2024-04-02T07:35:54.615284+0000","last_fullsized":"2024-04-02T07:35:54.615284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:53.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617144+0000","last_change":"2024-04-02T07:35:53.275758+0000","last_active":"2024-04-02T07:35:54.617144+0000","last_peered":"2024-04-02T07:35:54.617144+0000","last_clean":"2024-04-02T07:35:54.617144+0000","last_became_active":"2024-04-02T07:35:53.264243+0000","last_became_peered":"2024-04-02T07:35:53.264243+0000","last_unstale":"2024-04-02T07:35:54.617144+0000","last_undegraded":"2024-04-02T07:35:54.617144+0000","last_fullsized":"2024-04-02T07:35:54.617144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:13.487258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246979+0000","last_change":"2024-04-02T07:35:53.280476+0000","last_active":"2024-04-02T07:35:54.246979+0000","last_peered":"2024-04-02T07:35:54.246979+0000","last_clean":"2024-04-02T07:35:54.246979+0000","last_became_active":"2024-04-02T07:35:53.280209+0000","last_became_peered":"2024-04-02T07:35:53.280209+0000","last_unstale":"2024-04-02T07:35:54.246979+0000","last_undegraded":"2024-04-02T07:35:54.246979+0000","last_fullsized":"2024-04-02T07:35:54.246979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:06.923152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615751+0000","last_change":"2024-04-02T07:35:53.258035+0000","last_active":"2024-04-02T07:35:54.615751+0000","last_peered":"2024-04-02T07:35:54.615751+0000","last_clean":"2024-04-02T07:35:54.615751+0000","last_became_active":"2024-04-02T07:35:53.257721+0000","last_became_peered":"2024-04-02T07:35:53.257721+0000","last_unstale":"2024-04-02T07:35:54.615751+0000","last_undegraded":"2024-04-02T07:35:54.615751+0000","last_fullsized":"2024-04-02T07:35:54.615751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:18.703428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615499+0000","last_change":"2024-04-02T07:35:53.275062+0000","last_active":"2024-04-02T07:35:54.615499+0000","last_peered":"2024-04-02T07:35:54.615499+0000","last_clean":"2024-04-02T07:35:54.615499+0000","last_became_active":"2024-04-02T07:35:53.274852+0000","last_became_peered":"2024-04-02T07:35:53.274852+0000","last_unstale":"2024-04-02T07:35:54.615499+0000","last_undegraded":"2024-04-02T07:35:54.615499+0000","last_fullsized":"2024-04-02T07:35:54.615499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:13.487258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615560+0000","last_change":"2024-04-02T07:35:53.269911+0000","last_active":"2024-04-02T07:35:54.615560+0000","last_peered":"2024-04-02T07:35:54.615560+0000","last_clean":"2024-04-02T07:35:54.615560+0000","last_became_active":"2024-04-02T07:35:53.265651+0000","last_became_peered":"2024-04-02T07:35:53.265651+0000","last_unstale":"2024-04-02T07:35:54.615560+0000","last_undegraded":"2024-04-02T07:35:54.615560+0000","last_fullsized":"2024-04-02T07:35:54.615560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:16.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617009+0000","last_change":"2024-04-02T07:35:53.273591+0000","last_active":"2024-04-02T07:35:54.617009+0000","last_peered":"2024-04-02T07:35:54.617009+0000","last_clean":"2024-04-02T07:35:54.617009+0000","last_became_active":"2024-04-02T07:35:53.273289+0000","last_became_peered":"2024-04-02T07:35:53.273289+0000","last_unstale":"2024-04-02T07:35:54.617009+0000","last_undegraded":"2024-04-02T07:35:54.617009+0000","last_fullsized":"2024-04-02T07:35:54.617009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:00:20.419421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616704+0000","last_change":"2024-04-02T07:35:53.281809+0000","last_active":"2024-04-02T07:35:54.616704+0000","last_peered":"2024-04-02T07:35:54.616704+0000","last_clean":"2024-04-02T07:35:54.616704+0000","last_became_active":"2024-04-02T07:35:53.277985+0000","last_became_peered":"2024-04-02T07:35:53.277985+0000","last_unstale":"2024-04-02T07:35:54.616704+0000","last_undegraded":"2024-04-02T07:35:54.616704+0000","last_fullsized":"2024-04-02T07:35:54.616704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:42.518534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617297+0000","last_change":"2024-04-02T07:35:53.268138+0000","last_active":"2024-04-02T07:35:54.617297+0000","last_peered":"2024-04-02T07:35:54.617297+0000","last_clean":"2024-04-02T07:35:54.617297+0000","last_became_active":"2024-04-02T07:35:53.264989+0000","last_became_peered":"2024-04-02T07:35:53.264989+0000","last_unstale":"2024-04-02T07:35:54.617297+0000","last_undegraded":"2024-04-02T07:35:54.617297+0000","last_fullsized":"2024-04-02T07:35:54.617297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.380856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245556+0000","last_change":"2024-04-02T07:35:53.276292+0000","last_active":"2024-04-02T07:35:54.245556+0000","last_peered":"2024-04-02T07:35:54.245556+0000","last_clean":"2024-04-02T07:35:54.245556+0000","last_became_active":"2024-04-02T07:35:53.276003+0000","last_became_peered":"2024-04-02T07:35:53.276003+0000","last_unstale":"2024-04-02T07:35:54.245556+0000","last_undegraded":"2024-04-02T07:35:54.245556+0000","last_fullsized":"2024-04-02T07:35:54.245556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:33.477903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247301+0000","last_change":"2024-04-02T07:35:53.269437+0000","last_active":"2024-04-02T07:35:54.247301+0000","last_peered":"2024-04-02T07:35:54.247301+0000","last_clean":"2024-04-02T07:35:54.247301+0000","last_became_active":"2024-04-02T07:35:53.269192+0000","last_became_peered":"2024-04-02T07:35:53.269192+0000","last_unstale":"2024-04-02T07:35:54.247301+0000","last_undegraded":"2024-04-02T07:35:54.247301+0000","last_fullsized":"2024-04-02T07:35:54.247301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:35.381030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617797+0000","last_change":"2024-04-02T07:35:53.259373+0000","last_active":"2024-04-02T07:35:54.617797+0000","last_peered":"2024-04-02T07:35:54.617797+0000","last_clean":"2024-04-02T07:35:54.617797+0000","last_became_active":"2024-04-02T07:35:53.258986+0000","last_became_peered":"2024-04-02T07:35:53.258986+0000","last_unstale":"2024-04-02T07:35:54.617797+0000","last_undegraded":"2024-04-02T07:35:54.617797+0000","last_fullsized":"2024-04-02T07:35:54.617797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:27.051276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246472+0000","last_change":"2024-04-02T07:35:53.277181+0000","last_active":"2024-04-02T07:35:54.246472+0000","last_peered":"2024-04-02T07:35:54.246472+0000","last_clean":"2024-04-02T07:35:54.246472+0000","last_became_active":"2024-04-02T07:35:53.276207+0000","last_became_peered":"2024-04-02T07:35:53.276207+0000","last_unstale":"2024-04-02T07:35:54.246472+0000","last_undegraded":"2024-04-02T07:35:54.246472+0000","last_fullsized":"2024-04-02T07:35:54.246472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:13.487258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247380+0000","last_change":"2024-04-02T07:35:52.756587+0000","last_active":"2024-04-02T07:35:54.247380+0000","last_peered":"2024-04-02T07:35:54.247380+0000","last_clean":"2024-04-02T07:35:54.247380+0000","last_became_active":"2024-04-02T07:35:52.755981+0000","last_became_peered":"2024-04-02T07:35:52.755981+0000","last_unstale":"2024-04-02T07:35:54.247380+0000","last_undegraded":"2024-04-02T07:35:54.247380+0000","last_fullsized":"2024-04-02T07:35:54.247380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:07.324155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617519+0000","last_change":"2024-04-02T07:35:53.257824+0000","last_active":"2024-04-02T07:35:54.617519+0000","last_peered":"2024-04-02T07:35:54.617519+0000","last_clean":"2024-04-02T07:35:54.617519+0000","last_became_active":"2024-04-02T07:35:53.257500+0000","last_became_peered":"2024-04-02T07:35:53.257500+0000","last_unstale":"2024-04-02T07:35:54.617519+0000","last_undegraded":"2024-04-02T07:35:54.617519+0000","last_fullsized":"2024-04-02T07:35:54.617519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:35.709321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617620+0000","last_change":"2024-04-02T07:35:52.761793+0000","last_active":"2024-04-02T07:35:54.617620+0000","last_peered":"2024-04-02T07:35:54.617620+0000","last_clean":"2024-04-02T07:35:54.617620+0000","last_became_active":"2024-04-02T07:35:52.761510+0000","last_became_peered":"2024-04-02T07:35:52.761510+0000","last_unstale":"2024-04-02T07:35:54.617620+0000","last_undegraded":"2024-04-02T07:35:54.617620+0000","last_fullsized":"2024-04-02T07:35:54.617620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617652+0000","last_change":"2024-04-02T07:35:53.262736+0000","last_active":"2024-04-02T07:35:54.617652+0000","last_peered":"2024-04-02T07:35:54.617652+0000","last_clean":"2024-04-02T07:35:54.617652+0000","last_became_active":"2024-04-02T07:35:53.262405+0000","last_became_peered":"2024-04-02T07:35:53.262405+0000","last_unstale":"2024-04-02T07:35:54.617652+0000","last_undegraded":"2024-04-02T07:35:54.617652+0000","last_fullsized":"2024-04-02T07:35:54.617652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:52:59.688399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615557+0000","last_change":"2024-04-02T07:35:52.765725+0000","last_active":"2024-04-02T07:35:54.615557+0000","last_peered":"2024-04-02T07:35:54.615557+0000","last_clean":"2024-04-02T07:35:54.615557+0000","last_became_active":"2024-04-02T07:35:52.765393+0000","last_became_peered":"2024-04-02T07:35:52.765393+0000","last_unstale":"2024-04-02T07:35:54.615557+0000","last_undegraded":"2024-04-02T07:35:54.615557+0000","last_fullsized":"2024-04-02T07:35:54.615557+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:31.244191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246863+0000","last_change":"2024-04-02T07:35:53.255693+0000","last_active":"2024-04-02T07:35:54.246863+0000","last_peered":"2024-04-02T07:35:54.246863+0000","last_clean":"2024-04-02T07:35:54.246863+0000","last_became_active":"2024-04-02T07:35:53.254627+0000","last_became_peered":"2024-04-02T07:35:53.254627+0000","last_unstale":"2024-04-02T07:35:54.246863+0000","last_undegraded":"2024-04-02T07:35:54.246863+0000","last_fullsized":"2024-04-02T07:35:54.246863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:42.518534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615455+0000","last_change":"2024-04-02T07:35:52.762790+0000","last_active":"2024-04-02T07:35:54.615455+0000","last_peered":"2024-04-02T07:35:54.615455+0000","last_clean":"2024-04-02T07:35:54.615455+0000","last_became_active":"2024-04-02T07:35:52.762416+0000","last_became_peered":"2024-04-02T07:35:52.762416+0000","last_unstale":"2024-04-02T07:35:54.615455+0000","last_undegraded":"2024-04-02T07:35:54.615455+0000","last_fullsized":"2024-04-02T07:35:54.615455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:05.916519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246530+0000","last_change":"2024-04-02T07:35:53.274608+0000","last_active":"2024-04-02T07:35:54.246530+0000","last_peered":"2024-04-02T07:35:54.246530+0000","last_clean":"2024-04-02T07:35:54.246530+0000","last_became_active":"2024-04-02T07:35:53.274308+0000","last_became_peered":"2024-04-02T07:35:53.274308+0000","last_unstale":"2024-04-02T07:35:54.246530+0000","last_undegraded":"2024-04-02T07:35:54.246530+0000","last_fullsized":"2024-04-02T07:35:54.246530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:28:26.191159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617955+0000","last_change":"2024-04-02T07:35:52.761806+0000","last_active":"2024-04-02T07:35:54.617955+0000","last_peered":"2024-04-02T07:35:54.617955+0000","last_clean":"2024-04-02T07:35:54.617955+0000","last_became_active":"2024-04-02T07:35:52.761556+0000","last_became_peered":"2024-04-02T07:35:52.761556+0000","last_unstale":"2024-04-02T07:35:54.617955+0000","last_undegraded":"2024-04-02T07:35:54.617955+0000","last_fullsized":"2024-04-02T07:35:54.617955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246589+0000","last_change":"2024-04-02T07:35:53.262036+0000","last_active":"2024-04-02T07:35:54.246589+0000","last_peered":"2024-04-02T07:35:54.246589+0000","last_clean":"2024-04-02T07:35:54.246589+0000","last_became_active":"2024-04-02T07:35:53.259860+0000","last_became_peered":"2024-04-02T07:35:53.259860+0000","last_unstale":"2024-04-02T07:35:54.246589+0000","last_undegraded":"2024-04-02T07:35:54.246589+0000","last_fullsized":"2024-04-02T07:35:54.246589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:27.051276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245921+0000","last_change":"2024-04-02T07:35:52.773232+0000","last_active":"2024-04-02T07:35:54.245921+0000","last_peered":"2024-04-02T07:35:54.245921+0000","last_clean":"2024-04-02T07:35:54.245921+0000","last_became_active":"2024-04-02T07:35:52.772864+0000","last_became_peered":"2024-04-02T07:35:52.772864+0000","last_unstale":"2024-04-02T07:35:54.245921+0000","last_undegraded":"2024-04-02T07:35:54.245921+0000","last_fullsized":"2024-04-02T07:35:54.245921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:26.044643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247078+0000","last_change":"2024-04-02T07:35:53.274460+0000","last_active":"2024-04-02T07:35:54.247078+0000","last_peered":"2024-04-02T07:35:54.247078+0000","last_clean":"2024-04-02T07:35:54.247078+0000","last_became_active":"2024-04-02T07:35:53.274227+0000","last_became_peered":"2024-04-02T07:35:53.274227+0000","last_unstale":"2024-04-02T07:35:54.247078+0000","last_undegraded":"2024-04-02T07:35:54.247078+0000","last_fullsized":"2024-04-02T07:35:54.247078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:31.170936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616577+0000","last_change":"2024-04-02T07:35:52.769404+0000","last_active":"2024-04-02T07:35:54.616577+0000","last_peered":"2024-04-02T07:35:54.616577+0000","last_clean":"2024-04-02T07:35:54.616577+0000","last_became_active":"2024-04-02T07:35:52.769091+0000","last_became_peered":"2024-04-02T07:35:52.769091+0000","last_unstale":"2024-04-02T07:35:54.616577+0000","last_undegraded":"2024-04-02T07:35:54.616577+0000","last_fullsized":"2024-04-02T07:35:54.616577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617272+0000","last_change":"2024-04-02T07:35:53.261740+0000","last_active":"2024-04-02T07:35:54.617272+0000","last_peered":"2024-04-02T07:35:54.617272+0000","last_clean":"2024-04-02T07:35:54.617272+0000","last_became_active":"2024-04-02T07:35:53.261475+0000","last_became_peered":"2024-04-02T07:35:53.261475+0000","last_unstale":"2024-04-02T07:35:54.617272+0000","last_undegraded":"2024-04-02T07:35:54.617272+0000","last_fullsized":"2024-04-02T07:35:54.617272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:31.170936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245738+0000","last_change":"2024-04-02T07:35:52.765814+0000","last_active":"2024-04-02T07:35:54.245738+0000","last_peered":"2024-04-02T07:35:54.245738+0000","last_clean":"2024-04-02T07:35:54.245738+0000","last_became_active":"2024-04-02T07:35:52.764956+0000","last_became_peered":"2024-04-02T07:35:52.764956+0000","last_unstale":"2024-04-02T07:35:54.245738+0000","last_undegraded":"2024-04-02T07:35:54.245738+0000","last_fullsized":"2024-04-02T07:35:54.245738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:49.544958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247082+0000","last_change":"2024-04-02T07:35:53.278155+0000","last_active":"2024-04-02T07:35:54.247082+0000","last_peered":"2024-04-02T07:35:54.247082+0000","last_clean":"2024-04-02T07:35:54.247082+0000","last_became_active":"2024-04-02T07:35:53.274007+0000","last_became_peered":"2024-04-02T07:35:53.274007+0000","last_unstale":"2024-04-02T07:35:54.247082+0000","last_undegraded":"2024-04-02T07:35:54.247082+0000","last_fullsized":"2024-04-02T07:35:54.247082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:05:41.161422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246958+0000","last_change":"2024-04-02T07:35:52.759611+0000","last_active":"2024-04-02T07:35:54.246958+0000","last_peered":"2024-04-02T07:35:54.246958+0000","last_clean":"2024-04-02T07:35:54.246958+0000","last_became_active":"2024-04-02T07:35:52.759164+0000","last_became_peered":"2024-04-02T07:35:52.759164+0000","last_unstale":"2024-04-02T07:35:54.246958+0000","last_undegraded":"2024-04-02T07:35:54.246958+0000","last_fullsized":"2024-04-02T07:35:54.246958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557483+0000","last_change":"2024-04-02T07:35:53.269405+0000","last_active":"2024-04-02T07:35:54.557483+0000","last_peered":"2024-04-02T07:35:54.557483+0000","last_clean":"2024-04-02T07:35:54.557483+0000","last_became_active":"2024-04-02T07:35:53.267870+0000","last_became_peered":"2024-04-02T07:35:53.267870+0000","last_unstale":"2024-04-02T07:35:54.557483+0000","last_undegraded":"2024-04-02T07:35:54.557483+0000","last_fullsized":"2024-04-02T07:35:54.557483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:08.722219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615626+0000","last_change":"2024-04-02T07:35:52.765581+0000","last_active":"2024-04-02T07:35:54.615626+0000","last_peered":"2024-04-02T07:35:54.615626+0000","last_clean":"2024-04-02T07:35:54.615626+0000","last_became_active":"2024-04-02T07:35:52.764766+0000","last_became_peered":"2024-04-02T07:35:52.764766+0000","last_unstale":"2024-04-02T07:35:54.615626+0000","last_undegraded":"2024-04-02T07:35:54.615626+0000","last_fullsized":"2024-04-02T07:35:54.615626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:07.715585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617760+0000","last_change":"2024-04-02T07:35:53.275646+0000","last_active":"2024-04-02T07:35:54.617760+0000","last_peered":"2024-04-02T07:35:54.617760+0000","last_clean":"2024-04-02T07:35:54.617760+0000","last_became_active":"2024-04-02T07:35:53.275296+0000","last_became_peered":"2024-04-02T07:35:53.275296+0000","last_unstale":"2024-04-02T07:35:54.617760+0000","last_undegraded":"2024-04-02T07:35:54.617760+0000","last_fullsized":"2024-04-02T07:35:54.617760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:17.569969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617746+0000","last_change":"2024-04-02T07:35:52.751184+0000","last_active":"2024-04-02T07:35:54.617746+0000","last_peered":"2024-04-02T07:35:54.617746+0000","last_clean":"2024-04-02T07:35:54.617746+0000","last_became_active":"2024-04-02T07:35:52.750436+0000","last_became_peered":"2024-04-02T07:35:52.750436+0000","last_unstale":"2024-04-02T07:35:54.617746+0000","last_undegraded":"2024-04-02T07:35:54.617746+0000","last_fullsized":"2024-04-02T07:35:54.617746+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246276+0000","last_change":"2024-04-02T07:35:53.277457+0000","last_active":"2024-04-02T07:35:54.246276+0000","last_peered":"2024-04-02T07:35:54.246276+0000","last_clean":"2024-04-02T07:35:54.246276+0000","last_became_active":"2024-04-02T07:35:53.277059+0000","last_became_peered":"2024-04-02T07:35:53.277059+0000","last_unstale":"2024-04-02T07:35:54.246276+0000","last_undegraded":"2024-04-02T07:35:54.246276+0000","last_fullsized":"2024-04-02T07:35:54.246276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:18:02.329099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557196+0000","last_change":"2024-04-02T07:35:52.757551+0000","last_active":"2024-04-02T07:35:54.557196+0000","last_peered":"2024-04-02T07:35:54.557196+0000","last_clean":"2024-04-02T07:35:54.557196+0000","last_became_active":"2024-04-02T07:35:52.756930+0000","last_became_peered":"2024-04-02T07:35:52.756930+0000","last_unstale":"2024-04-02T07:35:54.557196+0000","last_undegraded":"2024-04-02T07:35:54.557196+0000","last_fullsized":"2024-04-02T07:35:54.557196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:40:29.501936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615608+0000","last_change":"2024-04-02T07:35:53.265162+0000","last_active":"2024-04-02T07:35:54.615608+0000","last_peered":"2024-04-02T07:35:54.615608+0000","last_clean":"2024-04-02T07:35:54.615608+0000","last_became_active":"2024-04-02T07:35:53.264897+0000","last_became_peered":"2024-04-02T07:35:53.264897+0000","last_unstale":"2024-04-02T07:35:54.615608+0000","last_undegraded":"2024-04-02T07:35:54.615608+0000","last_fullsized":"2024-04-02T07:35:54.615608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:35.709321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616800+0000","last_change":"2024-04-02T07:35:52.767385+0000","last_active":"2024-04-02T07:35:54.616800+0000","last_peered":"2024-04-02T07:35:54.616800+0000","last_clean":"2024-04-02T07:35:54.616800+0000","last_became_active":"2024-04-02T07:35:52.767027+0000","last_became_peered":"2024-04-02T07:35:52.767027+0000","last_unstale":"2024-04-02T07:35:54.616800+0000","last_undegraded":"2024-04-02T07:35:54.616800+0000","last_fullsized":"2024-04-02T07:35:54.616800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616462+0000","last_change":"2024-04-02T07:35:53.283307+0000","last_active":"2024-04-02T07:35:54.616462+0000","last_peered":"2024-04-02T07:35:54.616462+0000","last_clean":"2024-04-02T07:35:54.616462+0000","last_became_active":"2024-04-02T07:35:53.282062+0000","last_became_peered":"2024-04-02T07:35:53.282062+0000","last_unstale":"2024-04-02T07:35:54.616462+0000","last_undegraded":"2024-04-02T07:35:54.616462+0000","last_fullsized":"2024-04-02T07:35:54.616462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:24.367224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246052+0000","last_change":"2024-04-02T07:35:52.753164+0000","last_active":"2024-04-02T07:35:54.246052+0000","last_peered":"2024-04-02T07:35:54.246052+0000","last_clean":"2024-04-02T07:35:54.246052+0000","last_became_active":"2024-04-02T07:35:52.752227+0000","last_became_peered":"2024-04-02T07:35:52.752227+0000","last_unstale":"2024-04-02T07:35:54.246052+0000","last_undegraded":"2024-04-02T07:35:54.246052+0000","last_fullsized":"2024-04-02T07:35:54.246052+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:07.715585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246205+0000","last_change":"2024-04-02T07:35:53.269115+0000","last_active":"2024-04-02T07:35:54.246205+0000","last_peered":"2024-04-02T07:35:54.246205+0000","last_clean":"2024-04-02T07:35:54.246205+0000","last_became_active":"2024-04-02T07:35:53.268815+0000","last_became_peered":"2024-04-02T07:35:53.268815+0000","last_unstale":"2024-04-02T07:35:54.246205+0000","last_undegraded":"2024-04-02T07:35:54.246205+0000","last_fullsized":"2024-04-02T07:35:54.246205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:46:59.080243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615336+0000","last_change":"2024-04-02T07:35:52.764457+0000","last_active":"2024-04-02T07:35:54.615336+0000","last_peered":"2024-04-02T07:35:54.615336+0000","last_clean":"2024-04-02T07:35:54.615336+0000","last_became_active":"2024-04-02T07:35:52.764002+0000","last_became_peered":"2024-04-02T07:35:52.764002+0000","last_unstale":"2024-04-02T07:35:54.615336+0000","last_undegraded":"2024-04-02T07:35:54.615336+0000","last_fullsized":"2024-04-02T07:35:54.615336+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:38:44.437274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246412+0000","last_change":"2024-04-02T07:35:53.262284+0000","last_active":"2024-04-02T07:35:54.246412+0000","last_peered":"2024-04-02T07:35:54.246412+0000","last_clean":"2024-04-02T07:35:54.246412+0000","last_became_active":"2024-04-02T07:35:53.260682+0000","last_became_peered":"2024-04-02T07:35:53.260682+0000","last_unstale":"2024-04-02T07:35:54.246412+0000","last_undegraded":"2024-04-02T07:35:54.246412+0000","last_fullsized":"2024-04-02T07:35:54.246412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:55.943715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246893+0000","last_change":"2024-04-02T07:35:52.754933+0000","last_active":"2024-04-02T07:35:54.246893+0000","last_peered":"2024-04-02T07:35:54.246893+0000","last_clean":"2024-04-02T07:35:54.246893+0000","last_became_active":"2024-04-02T07:35:52.750245+0000","last_became_peered":"2024-04-02T07:35:52.750245+0000","last_unstale":"2024-04-02T07:35:54.246893+0000","last_undegraded":"2024-04-02T07:35:54.246893+0000","last_fullsized":"2024-04-02T07:35:54.246893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245859+0000","last_change":"2024-04-02T07:35:53.266981+0000","last_active":"2024-04-02T07:35:54.245859+0000","last_peered":"2024-04-02T07:35:54.245859+0000","last_clean":"2024-04-02T07:35:54.245859+0000","last_became_active":"2024-04-02T07:35:53.266194+0000","last_became_peered":"2024-04-02T07:35:53.266194+0000","last_unstale":"2024-04-02T07:35:54.245859+0000","last_undegraded":"2024-04-02T07:35:54.245859+0000","last_fullsized":"2024-04-02T07:35:54.245859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:22:27.488539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615714+0000","last_change":"2024-04-02T07:35:52.771817+0000","last_active":"2024-04-02T07:35:54.615714+0000","last_peered":"2024-04-02T07:35:54.615714+0000","last_clean":"2024-04-02T07:35:54.615714+0000","last_became_active":"2024-04-02T07:35:52.771315+0000","last_became_peered":"2024-04-02T07:35:52.771315+0000","last_unstale":"2024-04-02T07:35:54.615714+0000","last_undegraded":"2024-04-02T07:35:54.615714+0000","last_fullsized":"2024-04-02T07:35:54.615714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:07.324155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557469+0000","last_change":"2024-04-02T07:35:53.254431+0000","last_active":"2024-04-02T07:35:54.557469+0000","last_peered":"2024-04-02T07:35:54.557469+0000","last_clean":"2024-04-02T07:35:54.557469+0000","last_became_active":"2024-04-02T07:35:53.253582+0000","last_became_peered":"2024-04-02T07:35:53.253582+0000","last_unstale":"2024-04-02T07:35:54.557469+0000","last_undegraded":"2024-04-02T07:35:54.557469+0000","last_fullsized":"2024-04-02T07:35:54.557469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:32.250824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617486+0000","last_change":"2024-04-02T07:35:52.761017+0000","last_active":"2024-04-02T07:35:54.617486+0000","last_peered":"2024-04-02T07:35:54.617486+0000","last_clean":"2024-04-02T07:35:54.617486+0000","last_became_active":"2024-04-02T07:35:52.760290+0000","last_became_peered":"2024-04-02T07:35:52.760290+0000","last_unstale":"2024-04-02T07:35:54.617486+0000","last_undegraded":"2024-04-02T07:35:54.617486+0000","last_fullsized":"2024-04-02T07:35:54.617486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:45.255736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557174+0000","last_change":"2024-04-02T07:35:53.263108+0000","last_active":"2024-04-02T07:35:54.557174+0000","last_peered":"2024-04-02T07:35:54.557174+0000","last_clean":"2024-04-02T07:35:54.557174+0000","last_became_active":"2024-04-02T07:35:53.262111+0000","last_became_peered":"2024-04-02T07:35:53.262111+0000","last_unstale":"2024-04-02T07:35:54.557174+0000","last_undegraded":"2024-04-02T07:35:54.557174+0000","last_fullsized":"2024-04-02T07:35:54.557174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:27.051276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245974+0000","last_change":"2024-04-02T07:35:52.763712+0000","last_active":"2024-04-02T07:35:54.245974+0000","last_peered":"2024-04-02T07:35:54.245974+0000","last_clean":"2024-04-02T07:35:54.245974+0000","last_became_active":"2024-04-02T07:35:52.763457+0000","last_became_peered":"2024-04-02T07:35:52.763457+0000","last_unstale":"2024-04-02T07:35:54.245974+0000","last_undegraded":"2024-04-02T07:35:54.245974+0000","last_fullsized":"2024-04-02T07:35:54.245974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:41.511901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615482+0000","last_change":"2024-04-02T07:35:53.254132+0000","last_active":"2024-04-02T07:35:54.615482+0000","last_peered":"2024-04-02T07:35:54.615482+0000","last_clean":"2024-04-02T07:35:54.615482+0000","last_became_active":"2024-04-02T07:35:53.253705+0000","last_became_peered":"2024-04-02T07:35:53.253705+0000","last_unstale":"2024-04-02T07:35:54.615482+0000","last_undegraded":"2024-04-02T07:35:54.615482+0000","last_fullsized":"2024-04-02T07:35:54.615482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:17.569969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617123+0000","last_change":"2024-04-02T07:35:53.267551+0000","last_active":"2024-04-02T07:35:54.617123+0000","last_peered":"2024-04-02T07:35:54.617123+0000","last_clean":"2024-04-02T07:35:54.617123+0000","last_became_active":"2024-04-02T07:35:53.263849+0000","last_became_peered":"2024-04-02T07:35:53.263849+0000","last_unstale":"2024-04-02T07:35:54.617123+0000","last_undegraded":"2024-04-02T07:35:54.617123+0000","last_fullsized":"2024-04-02T07:35:54.617123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:54:04.724188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616708+0000","last_change":"2024-04-02T07:35:52.764685+0000","last_active":"2024-04-02T07:35:54.616708+0000","last_peered":"2024-04-02T07:35:54.616708+0000","last_clean":"2024-04-02T07:35:54.616708+0000","last_became_active":"2024-04-02T07:35:52.764421+0000","last_became_peered":"2024-04-02T07:35:52.764421+0000","last_unstale":"2024-04-02T07:35:54.616708+0000","last_undegraded":"2024-04-02T07:35:54.616708+0000","last_fullsized":"2024-04-02T07:35:54.616708+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247020+0000","last_change":"2024-04-02T07:35:53.264778+0000","last_active":"2024-04-02T07:35:54.247020+0000","last_peered":"2024-04-02T07:35:54.247020+0000","last_clean":"2024-04-02T07:35:54.247020+0000","last_became_active":"2024-04-02T07:35:53.264445+0000","last_became_peered":"2024-04-02T07:35:53.264445+0000","last_unstale":"2024-04-02T07:35:54.247020+0000","last_undegraded":"2024-04-02T07:35:54.247020+0000","last_fullsized":"2024-04-02T07:35:54.247020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:00:20.419421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246196+0000","last_change":"2024-04-02T07:35:52.767065+0000","last_active":"2024-04-02T07:35:54.246196+0000","last_peered":"2024-04-02T07:35:54.246196+0000","last_clean":"2024-04-02T07:35:54.246196+0000","last_became_active":"2024-04-02T07:35:52.764931+0000","last_became_peered":"2024-04-02T07:35:52.764931+0000","last_unstale":"2024-04-02T07:35:54.246196+0000","last_undegraded":"2024-04-02T07:35:54.246196+0000","last_fullsized":"2024-04-02T07:35:54.246196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:05.916519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-02T07:35:53.250580+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:35:53.250580+0000","last_peered":"2024-04-02T07:35:53.250580+0000","last_clean":"2024-04-02T07:35:53.250580+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:35:53.250580+0000","last_undegraded":"2024-04-02T07:35:53.250580+0000","last_fullsized":"2024-04-02T07:35:53.250580+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616550+0000","last_change":"2024-04-02T07:35:53.283251+0000","last_active":"2024-04-02T07:35:54.616550+0000","last_peered":"2024-04-02T07:35:54.616550+0000","last_clean":"2024-04-02T07:35:54.616550+0000","last_became_active":"2024-04-02T07:35:53.276591+0000","last_became_peered":"2024-04-02T07:35:53.276591+0000","last_unstale":"2024-04-02T07:35:54.616550+0000","last_undegraded":"2024-04-02T07:35:54.616550+0000","last_fullsized":"2024-04-02T07:35:54.616550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:07.528924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245695+0000","last_change":"2024-04-02T07:35:52.764279+0000","last_active":"2024-04-02T07:35:54.245695+0000","last_peered":"2024-04-02T07:35:54.245695+0000","last_clean":"2024-04-02T07:35:54.245695+0000","last_became_active":"2024-04-02T07:35:52.759649+0000","last_became_peered":"2024-04-02T07:35:52.759649+0000","last_unstale":"2024-04-02T07:35:54.245695+0000","last_undegraded":"2024-04-02T07:35:54.245695+0000","last_fullsized":"2024-04-02T07:35:54.245695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:54.937081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246537+0000","last_change":"2024-04-02T07:35:52.756573+0000","last_active":"2024-04-02T07:35:54.246537+0000","last_peered":"2024-04-02T07:35:54.246537+0000","last_clean":"2024-04-02T07:35:54.246537+0000","last_became_active":"2024-04-02T07:35:52.756192+0000","last_became_peered":"2024-04-02T07:35:52.756192+0000","last_unstale":"2024-04-02T07:35:54.246537+0000","last_undegraded":"2024-04-02T07:35:54.246537+0000","last_fullsized":"2024-04-02T07:35:54.246537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616866+0000","last_change":"2024-04-02T07:35:53.269008+0000","last_active":"2024-04-02T07:35:54.616866+0000","last_peered":"2024-04-02T07:35:54.616866+0000","last_clean":"2024-04-02T07:35:54.616866+0000","last_became_active":"2024-04-02T07:35:53.268663+0000","last_became_peered":"2024-04-02T07:35:53.268663+0000","last_unstale":"2024-04-02T07:35:54.616866+0000","last_undegraded":"2024-04-02T07:35:54.616866+0000","last_fullsized":"2024-04-02T07:35:54.616866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:05:41.161422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247184+0000","last_change":"2024-04-02T07:35:52.769059+0000","last_active":"2024-04-02T07:35:54.247184+0000","last_peered":"2024-04-02T07:35:54.247184+0000","last_clean":"2024-04-02T07:35:54.247184+0000","last_became_active":"2024-04-02T07:35:52.768711+0000","last_became_peered":"2024-04-02T07:35:52.768711+0000","last_unstale":"2024-04-02T07:35:54.247184+0000","last_undegraded":"2024-04-02T07:35:54.247184+0000","last_fullsized":"2024-04-02T07:35:54.247184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247358+0000","last_change":"2024-04-02T07:35:53.275607+0000","last_active":"2024-04-02T07:35:54.247358+0000","last_peered":"2024-04-02T07:35:54.247358+0000","last_clean":"2024-04-02T07:35:54.247358+0000","last_became_active":"2024-04-02T07:35:53.274591+0000","last_became_peered":"2024-04-02T07:35:53.274591+0000","last_unstale":"2024-04-02T07:35:54.247358+0000","last_undegraded":"2024-04-02T07:35:54.247358+0000","last_fullsized":"2024-04-02T07:35:54.247358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:13:58.887392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247393+0000","last_change":"2024-04-02T07:35:52.771999+0000","last_active":"2024-04-02T07:35:54.247393+0000","last_peered":"2024-04-02T07:35:54.247393+0000","last_clean":"2024-04-02T07:35:54.247393+0000","last_became_active":"2024-04-02T07:35:52.771720+0000","last_became_peered":"2024-04-02T07:35:52.771720+0000","last_unstale":"2024-04-02T07:35:54.247393+0000","last_undegraded":"2024-04-02T07:35:54.247393+0000","last_fullsized":"2024-04-02T07:35:54.247393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:31.244191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247153+0000","last_change":"2024-04-02T07:35:53.258654+0000","last_active":"2024-04-02T07:35:54.247153+0000","last_peered":"2024-04-02T07:35:54.247153+0000","last_clean":"2024-04-02T07:35:54.247153+0000","last_became_active":"2024-04-02T07:35:53.257658+0000","last_became_peered":"2024-04-02T07:35:53.257658+0000","last_unstale":"2024-04-02T07:35:54.247153+0000","last_undegraded":"2024-04-02T07:35:54.247153+0000","last_fullsized":"2024-04-02T07:35:54.247153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:07.528924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615282+0000","last_change":"2024-04-02T07:35:52.764308+0000","last_active":"2024-04-02T07:35:54.615282+0000","last_peered":"2024-04-02T07:35:54.615282+0000","last_clean":"2024-04-02T07:35:54.615282+0000","last_became_active":"2024-04-02T07:35:52.763178+0000","last_became_peered":"2024-04-02T07:35:52.763178+0000","last_unstale":"2024-04-02T07:35:54.615282+0000","last_undegraded":"2024-04-02T07:35:54.615282+0000","last_fullsized":"2024-04-02T07:35:54.615282+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:45.255736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245826+0000","last_change":"2024-04-02T07:35:53.258575+0000","last_active":"2024-04-02T07:35:54.245826+0000","last_peered":"2024-04-02T07:35:54.245826+0000","last_clean":"2024-04-02T07:35:54.245826+0000","last_became_active":"2024-04-02T07:35:53.258326+0000","last_became_peered":"2024-04-02T07:35:53.258326+0000","last_unstale":"2024-04-02T07:35:54.245826+0000","last_undegraded":"2024-04-02T07:35:54.245826+0000","last_fullsized":"2024-04-02T07:35:54.245826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:16.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247051+0000","last_change":"2024-04-02T07:35:52.771988+0000","last_active":"2024-04-02T07:35:54.247051+0000","last_peered":"2024-04-02T07:35:54.247051+0000","last_clean":"2024-04-02T07:35:54.247051+0000","last_became_active":"2024-04-02T07:35:52.771616+0000","last_became_peered":"2024-04-02T07:35:52.771616+0000","last_unstale":"2024-04-02T07:35:54.247051+0000","last_undegraded":"2024-04-02T07:35:54.247051+0000","last_fullsized":"2024-04-02T07:35:54.247051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:45.255736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246904+0000","last_change":"2024-04-02T07:35:53.258346+0000","last_active":"2024-04-02T07:35:54.246904+0000","last_peered":"2024-04-02T07:35:54.246904+0000","last_clean":"2024-04-02T07:35:54.246904+0000","last_became_active":"2024-04-02T07:35:53.258044+0000","last_became_peered":"2024-04-02T07:35:53.258044+0000","last_unstale":"2024-04-02T07:35:54.246904+0000","last_undegraded":"2024-04-02T07:35:54.246904+0000","last_fullsized":"2024-04-02T07:35:54.246904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:50.551591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616574+0000","last_change":"2024-04-02T07:35:52.758632+0000","last_active":"2024-04-02T07:35:54.616574+0000","last_peered":"2024-04-02T07:35:54.616574+0000","last_clean":"2024-04-02T07:35:54.616574+0000","last_became_active":"2024-04-02T07:35:52.758347+0000","last_became_peered":"2024-04-02T07:35:52.758347+0000","last_unstale":"2024-04-02T07:35:54.616574+0000","last_undegraded":"2024-04-02T07:35:54.616574+0000","last_fullsized":"2024-04-02T07:35:54.616574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617044+0000","last_change":"2024-04-02T07:35:53.271892+0000","last_active":"2024-04-02T07:35:54.617044+0000","last_peered":"2024-04-02T07:35:54.617044+0000","last_clean":"2024-04-02T07:35:54.617044+0000","last_became_active":"2024-04-02T07:35:53.271472+0000","last_became_peered":"2024-04-02T07:35:53.271472+0000","last_unstale":"2024-04-02T07:35:54.617044+0000","last_undegraded":"2024-04-02T07:35:54.617044+0000","last_fullsized":"2024-04-02T07:35:54.617044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:06.923152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246267+0000","last_change":"2024-04-02T07:35:52.758627+0000","last_active":"2024-04-02T07:35:54.246267+0000","last_peered":"2024-04-02T07:35:54.246267+0000","last_clean":"2024-04-02T07:35:54.246267+0000","last_became_active":"2024-04-02T07:35:52.758330+0000","last_became_peered":"2024-04-02T07:35:52.758330+0000","last_unstale":"2024-04-02T07:35:54.246267+0000","last_undegraded":"2024-04-02T07:35:54.246267+0000","last_fullsized":"2024-04-02T07:35:54.246267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:23.360591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245991+0000","last_change":"2024-04-02T07:35:53.266441+0000","last_active":"2024-04-02T07:35:54.245991+0000","last_peered":"2024-04-02T07:35:54.245991+0000","last_clean":"2024-04-02T07:35:54.245991+0000","last_became_active":"2024-04-02T07:35:53.265569+0000","last_became_peered":"2024-04-02T07:35:53.265569+0000","last_unstale":"2024-04-02T07:35:54.245991+0000","last_undegraded":"2024-04-02T07:35:54.245991+0000","last_fullsized":"2024-04-02T07:35:54.245991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:53.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616939+0000","last_change":"2024-04-02T07:35:52.762509+0000","last_active":"2024-04-02T07:35:54.616939+0000","last_peered":"2024-04-02T07:35:54.616939+0000","last_clean":"2024-04-02T07:35:54.616939+0000","last_became_active":"2024-04-02T07:35:52.762190+0000","last_became_peered":"2024-04-02T07:35:52.762190+0000","last_unstale":"2024-04-02T07:35:54.616939+0000","last_undegraded":"2024-04-02T07:35:54.616939+0000","last_fullsized":"2024-04-02T07:35:54.616939+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246626+0000","last_change":"2024-04-02T07:35:53.272664+0000","last_active":"2024-04-02T07:35:54.246626+0000","last_peered":"2024-04-02T07:35:54.246626+0000","last_clean":"2024-04-02T07:35:54.246626+0000","last_became_active":"2024-04-02T07:35:53.272408+0000","last_became_peered":"2024-04-02T07:35:53.272408+0000","last_unstale":"2024-04-02T07:35:54.246626+0000","last_undegraded":"2024-04-02T07:35:54.246626+0000","last_fullsized":"2024-04-02T07:35:54.246626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:18.703428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617830+0000","last_change":"2024-04-02T07:35:52.760241+0000","last_active":"2024-04-02T07:35:54.617830+0000","last_peered":"2024-04-02T07:35:54.617830+0000","last_clean":"2024-04-02T07:35:54.617830+0000","last_became_active":"2024-04-02T07:35:52.759523+0000","last_became_peered":"2024-04-02T07:35:52.759523+0000","last_unstale":"2024-04-02T07:35:54.617830+0000","last_undegraded":"2024-04-02T07:35:54.617830+0000","last_fullsized":"2024-04-02T07:35:54.617830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:07.324155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615417+0000","last_change":"2024-04-02T07:35:53.252444+0000","last_active":"2024-04-02T07:35:54.615417+0000","last_peered":"2024-04-02T07:35:54.615417+0000","last_clean":"2024-04-02T07:35:54.615417+0000","last_became_active":"2024-04-02T07:35:53.252160+0000","last_became_peered":"2024-04-02T07:35:53.252160+0000","last_unstale":"2024-04-02T07:35:54.615417+0000","last_undegraded":"2024-04-02T07:35:54.615417+0000","last_fullsized":"2024-04-02T07:35:54.615417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:18:02.329099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247287+0000","last_change":"2024-04-02T07:35:52.760163+0000","last_active":"2024-04-02T07:35:54.247287+0000","last_peered":"2024-04-02T07:35:54.247287+0000","last_clean":"2024-04-02T07:35:54.247287+0000","last_became_active":"2024-04-02T07:35:52.759785+0000","last_became_peered":"2024-04-02T07:35:52.759785+0000","last_unstale":"2024-04-02T07:35:54.247287+0000","last_undegraded":"2024-04-02T07:35:54.247287+0000","last_fullsized":"2024-04-02T07:35:54.247287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615799+0000","last_change":"2024-04-02T07:35:53.259830+0000","last_active":"2024-04-02T07:35:54.615799+0000","last_peered":"2024-04-02T07:35:54.615799+0000","last_clean":"2024-04-02T07:35:54.615799+0000","last_became_active":"2024-04-02T07:35:53.259518+0000","last_became_peered":"2024-04-02T07:35:53.259518+0000","last_unstale":"2024-04-02T07:35:54.615799+0000","last_undegraded":"2024-04-02T07:35:54.615799+0000","last_fullsized":"2024-04-02T07:35:54.615799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:54:41.930328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.556975+0000","last_change":"2024-04-02T07:35:53.263050+0000","last_active":"2024-04-02T07:35:54.556975+0000","last_peered":"2024-04-02T07:35:54.556975+0000","last_clean":"2024-04-02T07:35:54.556975+0000","last_became_active":"2024-04-02T07:35:53.262790+0000","last_became_peered":"2024-04-02T07:35:53.262790+0000","last_unstale":"2024-04-02T07:35:54.556975+0000","last_undegraded":"2024-04-02T07:35:54.556975+0000","last_fullsized":"2024-04-02T07:35:54.556975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:42.518534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616668+0000","last_change":"2024-04-02T07:35:53.269409+0000","last_active":"2024-04-02T07:35:54.616668+0000","last_peered":"2024-04-02T07:35:54.616668+0000","last_clean":"2024-04-02T07:35:54.616668+0000","last_became_active":"2024-04-02T07:35:53.267892+0000","last_became_peered":"2024-04-02T07:35:53.267892+0000","last_unstale":"2024-04-02T07:35:54.616668+0000","last_undegraded":"2024-04-02T07:35:54.616668+0000","last_fullsized":"2024-04-02T07:35:54.616668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:08.722219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245810+0000","last_change":"2024-04-02T07:35:53.260570+0000","last_active":"2024-04-02T07:35:54.245810+0000","last_peered":"2024-04-02T07:35:54.245810+0000","last_clean":"2024-04-02T07:35:54.245810+0000","last_became_active":"2024-04-02T07:35:53.260265+0000","last_became_peered":"2024-04-02T07:35:53.260265+0000","last_unstale":"2024-04-02T07:35:54.245810+0000","last_undegraded":"2024-04-02T07:35:54.245810+0000","last_fullsized":"2024-04-02T07:35:54.245810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:13:55.860837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617586+0000","last_change":"2024-04-02T07:35:53.260933+0000","last_active":"2024-04-02T07:35:54.617586+0000","last_peered":"2024-04-02T07:35:54.617586+0000","last_clean":"2024-04-02T07:35:54.617586+0000","last_became_active":"2024-04-02T07:35:53.260662+0000","last_became_peered":"2024-04-02T07:35:53.260662+0000","last_unstale":"2024-04-02T07:35:54.617586+0000","last_undegraded":"2024-04-02T07:35:54.617586+0000","last_fullsized":"2024-04-02T07:35:54.617586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:16.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616780+0000","last_change":"2024-04-02T07:35:53.274324+0000","last_active":"2024-04-02T07:35:54.616780+0000","last_peered":"2024-04-02T07:35:54.616780+0000","last_clean":"2024-04-02T07:35:54.616780+0000","last_became_active":"2024-04-02T07:35:53.274049+0000","last_became_peered":"2024-04-02T07:35:53.274049+0000","last_unstale":"2024-04-02T07:35:54.616780+0000","last_undegraded":"2024-04-02T07:35:54.616780+0000","last_fullsized":"2024-04-02T07:35:54.616780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:54:04.724188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616584+0000","last_change":"2024-04-02T07:35:53.272852+0000","last_active":"2024-04-02T07:35:54.616584+0000","last_peered":"2024-04-02T07:35:54.616584+0000","last_clean":"2024-04-02T07:35:54.616584+0000","last_became_active":"2024-04-02T07:35:53.272036+0000","last_became_peered":"2024-04-02T07:35:53.272036+0000","last_unstale":"2024-04-02T07:35:54.616584+0000","last_undegraded":"2024-04-02T07:35:54.616584+0000","last_fullsized":"2024-04-02T07:35:54.616584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:50.551591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557432+0000","last_change":"2024-04-02T07:35:53.275573+0000","last_active":"2024-04-02T07:35:54.557432+0000","last_peered":"2024-04-02T07:35:54.557432+0000","last_clean":"2024-04-02T07:35:54.557432+0000","last_became_active":"2024-04-02T07:35:53.272553+0000","last_became_peered":"2024-04-02T07:35:53.272553+0000","last_unstale":"2024-04-02T07:35:54.557432+0000","last_undegraded":"2024-04-02T07:35:54.557432+0000","last_fullsized":"2024-04-02T07:35:54.557432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:06.923152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617508+0000","last_change":"2024-04-02T07:35:53.262359+0000","last_active":"2024-04-02T07:35:54.617508+0000","last_peered":"2024-04-02T07:35:54.617508+0000","last_clean":"2024-04-02T07:35:54.617508+0000","last_became_active":"2024-04-02T07:35:53.262038+0000","last_became_peered":"2024-04-02T07:35:53.262038+0000","last_unstale":"2024-04-02T07:35:54.617508+0000","last_undegraded":"2024-04-02T07:35:54.617508+0000","last_fullsized":"2024-04-02T07:35:54.617508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:18.703428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245590+0000","last_change":"2024-04-02T07:35:53.284523+0000","last_active":"2024-04-02T07:35:54.245590+0000","last_peered":"2024-04-02T07:35:54.245590+0000","last_clean":"2024-04-02T07:35:54.245590+0000","last_became_active":"2024-04-02T07:35:53.284200+0000","last_became_peered":"2024-04-02T07:35:53.284200+0000","last_unstale":"2024-04-02T07:35:54.245590+0000","last_undegraded":"2024-04-02T07:35:54.245590+0000","last_fullsized":"2024-04-02T07:35:54.245590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:14:43.088676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617379+0000","last_change":"2024-04-02T07:35:53.259326+0000","last_active":"2024-04-02T07:35:54.617379+0000","last_peered":"2024-04-02T07:35:54.617379+0000","last_clean":"2024-04-02T07:35:54.617379+0000","last_became_active":"2024-04-02T07:35:53.259023+0000","last_became_peered":"2024-04-02T07:35:53.259023+0000","last_unstale":"2024-04-02T07:35:54.617379+0000","last_undegraded":"2024-04-02T07:35:54.617379+0000","last_fullsized":"2024-04-02T07:35:54.617379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:54:41.930328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246691+0000","last_change":"2024-04-02T07:35:53.261005+0000","last_active":"2024-04-02T07:35:54.246691+0000","last_peered":"2024-04-02T07:35:54.246691+0000","last_clean":"2024-04-02T07:35:54.246691+0000","last_became_active":"2024-04-02T07:35:53.259618+0000","last_became_peered":"2024-04-02T07:35:53.259618+0000","last_unstale":"2024-04-02T07:35:54.246691+0000","last_undegraded":"2024-04-02T07:35:54.246691+0000","last_fullsized":"2024-04-02T07:35:54.246691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:24.367224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247226+0000","last_change":"2024-04-02T07:35:53.262531+0000","last_active":"2024-04-02T07:35:54.247226+0000","last_peered":"2024-04-02T07:35:54.247226+0000","last_clean":"2024-04-02T07:35:54.247226+0000","last_became_active":"2024-04-02T07:35:53.262196+0000","last_became_peered":"2024-04-02T07:35:53.262196+0000","last_unstale":"2024-04-02T07:35:54.247226+0000","last_undegraded":"2024-04-02T07:35:54.247226+0000","last_fullsized":"2024-04-02T07:35:54.247226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:54:04.724188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615537+0000","last_change":"2024-04-02T07:35:53.266761+0000","last_active":"2024-04-02T07:35:54.615537+0000","last_peered":"2024-04-02T07:35:54.615537+0000","last_clean":"2024-04-02T07:35:54.615537+0000","last_became_active":"2024-04-02T07:35:53.264360+0000","last_became_peered":"2024-04-02T07:35:53.264360+0000","last_unstale":"2024-04-02T07:35:54.615537+0000","last_undegraded":"2024-04-02T07:35:54.615537+0000","last_fullsized":"2024-04-02T07:35:54.615537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:31.170936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":71480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":73134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":660339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:35:58.097 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T07:35:58.097 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-02T07:35:58.098 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-02T07:35:58.098 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-02T07:35:58.098 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-02T07:35:58.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.322+0000 7fe2dba00700 1 Processor -- start 2024-04-02T07:35:58.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.322+0000 7fe2dba00700 1 -- start start 2024-04-02T07:35:58.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.323+0000 7fe2dba00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe2d4160f10 0x7fe2d40d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.323+0000 7fe2dba00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d40ceda0 con 0x7fe2d4161590 2024-04-02T07:35:58.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.323+0000 7fe2dba00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d41612f0 con 0x7fe2d40cf230 2024-04-02T07:35:58.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.323+0000 7fe2dba00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2d40ce470 con 0x7fe2d4160f10 2024-04-02T07:35:58.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.324+0000 7fe2d9a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe2d4161590 0x7fe2d40b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36930/0 (socket says 172.21.15.89:36930) 2024-04-02T07:35:58.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.324+0000 7fe2d9a00700 1 -- 172.21.15.89:0/3625150095 learned_addr learned my addr 172.21.15.89:0/3625150095 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.324+0000 7fe2d9000700 1 --2- 172.21.15.89:0/3625150095 >> v2:172.21.15.113:3300/0 conn(0x7fe2d4160f10 0x7fe2d40d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.324+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441919682 0 0) 0x7fe2d40ceda0 con 0x7fe2d4161590 2024-04-02T07:35:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.324+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2bc0035b0 con 0x7fe2d4161590 2024-04-02T07:35:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.324+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741222997 0 0) 0x7fe2d41612f0 con 0x7fe2d40cf230 2024-04-02T07:35:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.324+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d40ceda0 con 0x7fe2d40cf230 2024-04-02T07:35:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.324+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 49811301 0 0) 0x7fe2bc0035b0 con 0x7fe2d4161590 2024-04-02T07:35:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.324+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 >> v2:172.21.15.113:3300/0 conn(0x7fe2d4160f10 msgr2=0x7fe2d40d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.324+0000 7fe2d3400700 1 --2- 172.21.15.89:0/3625150095 >> v2:172.21.15.113:3300/0 conn(0x7fe2d4160f10 0x7fe2d40d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.324+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 >> v1:172.21.15.113:6789/0 conn(0x7fe2d40cf230 legacy=0x7fe2d40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.324+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d40ce750 con 0x7fe2d4161590 2024-04-02T07:35:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fe2cc0019f0 con 0x7fe2d4161590 2024-04-02T07:35:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 >> v1:172.21.15.89:6789/0 conn(0x7fe2d4161590 legacy=0x7fe2d40b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d3400700 1 --2- 172.21.15.89:0/3625150095 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2bc003be0 0x7fe2bc003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d3400700 1 --2- 172.21.15.89:0/3625150095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2bc004850 0x7fe2bc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d3400700 1 --2- 172.21.15.89:0/3625150095 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2bc005260 0x7fe2bc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d40ce470 con 0x7fe2bc004850 2024-04-02T07:35:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc00c190 con 0x7fe2bc003be0 2024-04-02T07:35:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc00c350 con 0x7fe2bc005260 2024-04-02T07:35:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d3e00700 1 --2- 172.21.15.89:0/3625150095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2bc004850 0x7fe2bc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d9000700 1 --2- 172.21.15.89:0/3625150095 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2bc003be0 0x7fe2bc003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d9a00700 1 --2- 172.21.15.89:0/3625150095 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2bc005260 0x7fe2bc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d3e00700 1 -- 172.21.15.89:0/3625150095 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2bc005260 msgr2=0x7fe2bc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d3e00700 1 --2- 172.21.15.89:0/3625150095 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2bc005260 0x7fe2bc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d3e00700 1 -- 172.21.15.89:0/3625150095 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2bc003be0 msgr2=0x7fe2bc003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d3e00700 1 --2- 172.21.15.89:0/3625150095 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2bc003be0 0x7fe2bc003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d3e00700 1 -- 172.21.15.89:0/3625150095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc0035b0 con 0x7fe2bc004850 2024-04-02T07:35:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.325+0000 7fe2d9a00700 1 -- 172.21.15.89:0/3625150095 reap_dead start 2024-04-02T07:35:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.326+0000 7fe2d3e00700 1 --2- 172.21.15.89:0/3625150095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2bc004850 0x7fe2bc004cd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe2d40d25a0 tx=0x7fe2bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f2174c4fe6e6b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.326+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0001430 con 0x7fe2bc004850 2024-04-02T07:35:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.326+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c00052f0 con 0x7fe2bc004850 2024-04-02T07:35:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.326+0000 7fe2d3400700 1 -- 172.21.15.89:0/3625150095 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0001d40 con 0x7fe2bc004850 2024-04-02T07:35:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.326+0000 7fe2dba00700 1 -- 172.21.15.89:0/3625150095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2bc004850 msgr2=0x7fe2bc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.326+0000 7fe2dba00700 1 --2- 172.21.15.89:0/3625150095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2bc004850 0x7fe2bc004cd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe2d40d25a0 tx=0x7fe2bc005880 comp rx=0 tx=0).stop 2024-04-02T07:35:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.326+0000 7fe2dba00700 1 -- 172.21.15.89:0/3625150095 shutdown_connections 2024-04-02T07:35:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.326+0000 7fe2dba00700 1 --2- 172.21.15.89:0/3625150095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2bc004850 0x7fe2bc004cd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.326+0000 7fe2dba00700 1 -- 172.21.15.89:0/3625150095 >> 172.21.15.89:0/3625150095 conn(0x7fe2d408aff0 msgr2=0x7fe2d40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.326+0000 7fe2dba00700 1 -- 172.21.15.89:0/3625150095 shutdown_connections 2024-04-02T07:35:58.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.326+0000 7fe2dba00700 1 -- 172.21.15.89:0/3625150095 wait complete. 2024-04-02T07:35:58.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.328+0000 7fe2dba00700 1 Processor -- start 2024-04-02T07:35:58.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.328+0000 7fe2dba00700 1 -- start start 2024-04-02T07:35:58.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.329+0000 7fe2dba00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2bc004850 0x7fe2d40a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.329+0000 7fe2dba00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415f550 0x7fe2d406ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.329+0000 7fe2dba00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d415f9d0 0x7fe2d406bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.329+0000 7fe2dba00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d40a13c0 con 0x7fe2d415f550 2024-04-02T07:35:58.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.329+0000 7fe2dba00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d415e550 con 0x7fe2bc004850 2024-04-02T07:35:58.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.329+0000 7fe2dba00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d415e830 con 0x7fe2d415f9d0 2024-04-02T07:35:58.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.329+0000 7fe2d3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415f550 0x7fe2d406ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.329+0000 7fe2d3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415f550 0x7fe2d406ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57934/0 (socket says 172.21.15.89:57934) 2024-04-02T07:35:58.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.330+0000 7fe2d3e00700 1 -- 172.21.15.89:0/3158017405 learned_addr learned my addr 172.21.15.89:0/3158017405 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:58.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.330+0000 7fe2d9a00700 1 --2- 172.21.15.89:0/3158017405 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d415f9d0 0x7fe2d406bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.330+0000 7fe2d3e00700 1 -- 172.21.15.89:0/3158017405 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d415f9d0 msgr2=0x7fe2d406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:58.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.330+0000 7fe2d3e00700 1 --2- 172.21.15.89:0/3158017405 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d415f9d0 0x7fe2d406bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.330+0000 7fe2d3e00700 1 -- 172.21.15.89:0/3158017405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2bc004850 msgr2=0x7fe2d40a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:58.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.330+0000 7fe2d3e00700 1 --2- 172.21.15.89:0/3158017405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2bc004850 0x7fe2d40a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.330+0000 7fe2d3e00700 1 -- 172.21.15.89:0/3158017405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d415eb10 con 0x7fe2d415f550 2024-04-02T07:35:58.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.330+0000 7fe2d3e00700 1 --2- 172.21.15.89:0/3158017405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415f550 0x7fe2d406ba00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc00c160 tx=0x7fe2c00012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:58.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.330+0000 7fe2d0c00700 1 -- 172.21.15.89:0/3158017405 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0006a70 con 0x7fe2d415f550 2024-04-02T07:35:58.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.330+0000 7fe2d0c00700 1 -- 172.21.15.89:0/3158017405 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c0006bf0 con 0x7fe2d415f550 2024-04-02T07:35:58.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.330+0000 7fe2dba00700 1 -- 172.21.15.89:0/3158017405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d415edf0 con 0x7fe2d415f550 2024-04-02T07:35:58.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.330+0000 7fe2d0c00700 1 -- 172.21.15.89:0/3158017405 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0007040 con 0x7fe2d415f550 2024-04-02T07:35:58.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.331+0000 7fe2dba00700 1 -- 172.21.15.89:0/3158017405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d415b210 con 0x7fe2d415f550 2024-04-02T07:35:58.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.331+0000 7fe2dba00700 1 -- 172.21.15.89:0/3158017405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d415b3a0 con 0x7fe2d415f550 2024-04-02T07:35:58.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.332+0000 7fe2d0c00700 1 -- 172.21.15.89:0/3158017405 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe2c00071a0 con 0x7fe2d415f550 2024-04-02T07:35:58.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.333+0000 7fe2d0c00700 1 --2- 172.21.15.89:0/3158017405 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe2b406ac90 0x7fe2b406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.333+0000 7fe2d0c00700 1 -- 172.21.15.89:0/3158017405 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2c0089e70 con 0x7fe2d415f550 2024-04-02T07:35:58.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.333+0000 7fe2d9000700 1 --2- 172.21.15.89:0/3158017405 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe2b406ac90 0x7fe2b406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.334+0000 7fe2d9000700 1 --2- 172.21.15.89:0/3158017405 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe2b406ac90 0x7fe2b406d150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8006f10 tx=0x7fe2c8006e80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:58.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.337+0000 7fe2d0c00700 1 -- 172.21.15.89:0/3158017405 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe2c0058bf0 con 0x7fe2d415f550 2024-04-02T07:35:58.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.591+0000 7fe2dba00700 1 -- 172.21.15.89:0/3158017405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe2d415f310 con 0x7fe2d415f550 2024-04-02T07:35:58.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.595+0000 7fe2d0c00700 1 -- 172.21.15.89:0/3158017405 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170925 (secure 0 0 0) 0x7fe2c0021090 con 0x7fe2d415f550 2024-04-02T07:35:58.599 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:35:58.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.598+0000 7fe2dba00700 1 -- 172.21.15.89:0/3158017405 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe2b406ac90 msgr2=0x7fe2b406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:58.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.598+0000 7fe2dba00700 1 --2- 172.21.15.89:0/3158017405 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe2b406ac90 0x7fe2b406d150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8006f10 tx=0x7fe2c8006e80 comp rx=0 tx=0).stop 2024-04-02T07:35:58.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.598+0000 7fe2dba00700 1 -- 172.21.15.89:0/3158017405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415f550 msgr2=0x7fe2d406ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:58.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.598+0000 7fe2dba00700 1 --2- 172.21.15.89:0/3158017405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415f550 0x7fe2d406ba00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc00c160 tx=0x7fe2c00012d0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.598+0000 7fe2dba00700 1 -- 172.21.15.89:0/3158017405 shutdown_connections 2024-04-02T07:35:58.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.598+0000 7fe2dba00700 1 --2- 172.21.15.89:0/3158017405 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe2b406ac90 0x7fe2b406d150 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.598+0000 7fe2dba00700 1 --2- 172.21.15.89:0/3158017405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2bc004850 0x7fe2d40a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.598+0000 7fe2dba00700 1 --2- 172.21.15.89:0/3158017405 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d415f9d0 0x7fe2d406bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.598+0000 7fe2dba00700 1 --2- 172.21.15.89:0/3158017405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415f550 0x7fe2d406ba00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.598+0000 7fe2dba00700 1 -- 172.21.15.89:0/3158017405 >> 172.21.15.89:0/3158017405 conn(0x7fe2d408aff0 msgr2=0x7fe2d40cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:58.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.598+0000 7fe2dba00700 1 -- 172.21.15.89:0/3158017405 shutdown_connections 2024-04-02T07:35:58.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.598+0000 7fe2dba00700 1 -- 172.21.15.89:0/3158017405 wait complete. 2024-04-02T07:35:58.618 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":5,"active_gid":4108,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":43357},{"type":"v1","addr":"172.21.15.89:6833","nonce":43357}]},"active_addr":"172.21.15.89:6833/43357","active_change":"2024-04-02T07:35:35.169009+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4101,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":2201396539}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":2645815547}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":350807093}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":399385398}]}]} 2024-04-02T07:35:58.622 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-02T07:35:58.622 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-02T07:35:58.622 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:35:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.813+0000 7efd6d400700 1 Processor -- start 2024-04-02T07:35:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.814+0000 7efd6d400700 1 -- start start 2024-04-02T07:35:58.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.815+0000 7efd6d400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efd68161ca0 0x7efd680cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.815+0000 7efd6d400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd680ceda0 con 0x7efd680d18b0 2024-04-02T07:35:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.815+0000 7efd6d400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd681612f0 con 0x7efd68160f10 2024-04-02T07:35:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.815+0000 7efd6d400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd680d1d30 con 0x7efd68161ca0 2024-04-02T07:35:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.815+0000 7efd65e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efd680d18b0 0x7efd68161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34780/0 (socket says 172.21.15.89:34780) 2024-04-02T07:35:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.815+0000 7efd65e00700 1 -- 172.21.15.89:0/2381226767 learned_addr learned my addr 172.21.15.89:0/2381226767 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:58.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.815+0000 7efd67200700 1 --2- 172.21.15.89:0/2381226767 >> v2:172.21.15.113:3300/0 conn(0x7efd68161ca0 0x7efd680cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.815+0000 7efd65400700 1 -- 172.21.15.89:0/2381226767 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257861252 0 0) 0x7efd680ceda0 con 0x7efd680d18b0 2024-04-02T07:35:58.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.815+0000 7efd65400700 1 -- 172.21.15.89:0/2381226767 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd4c0035b0 con 0x7efd680d18b0 2024-04-02T07:35:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd65400700 1 -- 172.21.15.89:0/2381226767 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1505537249 0 0) 0x7efd4c0035b0 con 0x7efd680d18b0 2024-04-02T07:35:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd65400700 1 -- 172.21.15.89:0/2381226767 >> v2:172.21.15.113:3300/0 conn(0x7efd68161ca0 msgr2=0x7efd680cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd65400700 1 --2- 172.21.15.89:0/2381226767 >> v2:172.21.15.113:3300/0 conn(0x7efd68161ca0 0x7efd680cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd65400700 1 -- 172.21.15.89:0/2381226767 >> v1:172.21.15.113:6789/0 conn(0x7efd68160f10 legacy=0x7efd680d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd65400700 1 -- 172.21.15.89:0/2381226767 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd680ce470 con 0x7efd680d18b0 2024-04-02T07:35:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd65400700 1 -- 172.21.15.89:0/2381226767 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7efd54001670 con 0x7efd680d18b0 2024-04-02T07:35:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd65400700 1 -- 172.21.15.89:0/2381226767 >> v1:172.21.15.89:6789/0 conn(0x7efd680d18b0 legacy=0x7efd68161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd65400700 1 --2- 172.21.15.89:0/2381226767 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd4c003a00 0x7efd4c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd65400700 1 --2- 172.21.15.89:0/2381226767 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4c004400 0x7efd4c0048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd65400700 1 --2- 172.21.15.89:0/2381226767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd4c005170 0x7efd4c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd65400700 1 -- 172.21.15.89:0/2381226767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd680d1d30 con 0x7efd4c005170 2024-04-02T07:35:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd65400700 1 -- 172.21.15.89:0/2381226767 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4c00c0b0 con 0x7efd4c003a00 2024-04-02T07:35:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd65400700 1 -- 172.21.15.89:0/2381226767 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd4c00c270 con 0x7efd4c004400 2024-04-02T07:35:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd67200700 1 --2- 172.21.15.89:0/2381226767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd4c005170 0x7efd4c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd66800700 1 --2- 172.21.15.89:0/2381226767 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd4c003a00 0x7efd4c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd65e00700 1 --2- 172.21.15.89:0/2381226767 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4c004400 0x7efd4c0048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd67200700 1 -- 172.21.15.89:0/2381226767 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4c004400 msgr2=0x7efd4c0048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd67200700 1 --2- 172.21.15.89:0/2381226767 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4c004400 0x7efd4c0048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd67200700 1 -- 172.21.15.89:0/2381226767 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd4c003a00 msgr2=0x7efd4c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd67200700 1 --2- 172.21.15.89:0/2381226767 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd4c003a00 0x7efd4c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.816+0000 7efd67200700 1 -- 172.21.15.89:0/2381226767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4c0035b0 con 0x7efd4c005170 2024-04-02T07:35:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.817+0000 7efd67200700 1 -- 172.21.15.89:0/2381226767 reap_dead start 2024-04-02T07:35:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.817+0000 7efd67200700 1 --2- 172.21.15.89:0/2381226767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd4c005170 0x7efd4c00ba10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efd60009fc0 tx=0x7efd4c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c3657b136a92a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.817+0000 7efd65400700 1 -- 172.21.15.89:0/2381226767 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd60003a20 con 0x7efd4c005170 2024-04-02T07:35:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.817+0000 7efd65400700 1 -- 172.21.15.89:0/2381226767 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6000f040 con 0x7efd4c005170 2024-04-02T07:35:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.817+0000 7efd65400700 1 -- 172.21.15.89:0/2381226767 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd60003da0 con 0x7efd4c005170 2024-04-02T07:35:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.817+0000 7efd6d400700 1 -- 172.21.15.89:0/2381226767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd4c005170 msgr2=0x7efd4c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.817+0000 7efd6d400700 1 --2- 172.21.15.89:0/2381226767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd4c005170 0x7efd4c00ba10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efd60009fc0 tx=0x7efd4c005880 comp rx=0 tx=0).stop 2024-04-02T07:35:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.817+0000 7efd6d400700 1 -- 172.21.15.89:0/2381226767 shutdown_connections 2024-04-02T07:35:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.817+0000 7efd6d400700 1 --2- 172.21.15.89:0/2381226767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd4c005170 0x7efd4c00ba10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.817+0000 7efd6d400700 1 -- 172.21.15.89:0/2381226767 >> 172.21.15.89:0/2381226767 conn(0x7efd6808aff0 msgr2=0x7efd680d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.818+0000 7efd6d400700 1 -- 172.21.15.89:0/2381226767 shutdown_connections 2024-04-02T07:35:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.818+0000 7efd6d400700 1 -- 172.21.15.89:0/2381226767 wait complete. 2024-04-02T07:35:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.819+0000 7efd6d400700 1 Processor -- start 2024-04-02T07:35:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.819+0000 7efd6d400700 1 -- start start 2024-04-02T07:35:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.820+0000 7efd6d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd4c005170 0x7efd6815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.820+0000 7efd6d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd681588c0 0x7efd6815e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.820+0000 7efd6d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd6815f5e0 0x7efd6814e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.820+0000 7efd6d400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6815e170 con 0x7efd6815f5e0 2024-04-02T07:35:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.820+0000 7efd6d400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6815d560 con 0x7efd4c005170 2024-04-02T07:35:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.820+0000 7efd6d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd6815d840 con 0x7efd681588c0 2024-04-02T07:35:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.821+0000 7efd66800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd4c005170 0x7efd6815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.821+0000 7efd67200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd6815f5e0 0x7efd6814e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.821+0000 7efd66800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd4c005170 0x7efd6815cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33372/0 (socket says 172.21.15.89:33372) 2024-04-02T07:35:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.821+0000 7efd66800700 1 -- 172.21.15.89:0/4267611947 learned_addr learned my addr 172.21.15.89:0/4267611947 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.821+0000 7efd65e00700 1 --2- 172.21.15.89:0/4267611947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd681588c0 0x7efd6815e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.821+0000 7efd66800700 1 -- 172.21.15.89:0/4267611947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd681588c0 msgr2=0x7efd6815e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.821+0000 7efd66800700 1 --2- 172.21.15.89:0/4267611947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd681588c0 0x7efd6815e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.821+0000 7efd66800700 1 -- 172.21.15.89:0/4267611947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd6815f5e0 msgr2=0x7efd6814e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.821+0000 7efd66800700 1 --2- 172.21.15.89:0/4267611947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd6815f5e0 0x7efd6814e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.821+0000 7efd66800700 1 -- 172.21.15.89:0/4267611947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6815db20 con 0x7efd4c005170 2024-04-02T07:35:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.821+0000 7efd67200700 1 --2- 172.21.15.89:0/4267611947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd6815f5e0 0x7efd6814e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.821+0000 7efd66800700 1 --2- 172.21.15.89:0/4267611947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd4c005170 0x7efd6815cef0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efd5800f780 tx=0x7efd5800fa90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.821+0000 7efd5ea00700 1 -- 172.21.15.89:0/4267611947 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58017070 con 0x7efd4c005170 2024-04-02T07:35:58.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.822+0000 7efd5ea00700 1 -- 172.21.15.89:0/4267611947 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd580052e0 con 0x7efd4c005170 2024-04-02T07:35:58.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.822+0000 7efd6d400700 1 -- 172.21.15.89:0/4267611947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd6815de00 con 0x7efd4c005170 2024-04-02T07:35:58.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.822+0000 7efd5ea00700 1 -- 172.21.15.89:0/4267611947 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58005470 con 0x7efd4c005170 2024-04-02T07:35:58.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.822+0000 7efd6d400700 1 -- 172.21.15.89:0/4267611947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd6815fa90 con 0x7efd4c005170 2024-04-02T07:35:58.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.823+0000 7efd6d400700 1 -- 172.21.15.89:0/4267611947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd680cd890 con 0x7efd4c005170 2024-04-02T07:35:58.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.824+0000 7efd5ea00700 1 -- 172.21.15.89:0/4267611947 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efd58006860 con 0x7efd4c005170 2024-04-02T07:35:58.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.824+0000 7efd5ea00700 1 --2- 172.21.15.89:0/4267611947 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efd4806ac90 0x7efd4806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:58.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.825+0000 7efd5ea00700 1 -- 172.21.15.89:0/4267611947 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd58088a30 con 0x7efd4c005170 2024-04-02T07:35:58.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.825+0000 7efd65e00700 1 --2- 172.21.15.89:0/4267611947 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efd4806ac90 0x7efd4806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:58.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.825+0000 7efd65e00700 1 --2- 172.21.15.89:0/4267611947 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efd4806ac90 0x7efd4806d150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd6815cc80 tx=0x7efd54007040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:58.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:58.830+0000 7efd5ea00700 1 -- 172.21.15.89:0/4267611947 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efd58053c00 con 0x7efd4c005170 2024-04-02T07:35:59.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.034+0000 7efd6d400700 1 -- 172.21.15.89:0/4267611947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd680cf860 con 0x7efd4c005170 2024-04-02T07:35:59.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.036+0000 7efd5ea00700 1 -- 172.21.15.89:0/4267611947 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7efd58057220 con 0x7efd4c005170 2024-04-02T07:35:59.038 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:35:59.038 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:54.238913+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:35:59.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.038+0000 7efd6d400700 1 -- 172.21.15.89:0/4267611947 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efd4806ac90 msgr2=0x7efd4806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.038+0000 7efd6d400700 1 --2- 172.21.15.89:0/4267611947 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efd4806ac90 0x7efd4806d150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd6815cc80 tx=0x7efd54007040 comp rx=0 tx=0).stop 2024-04-02T07:35:59.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.038+0000 7efd6d400700 1 -- 172.21.15.89:0/4267611947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd4c005170 msgr2=0x7efd6815cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.038+0000 7efd6d400700 1 --2- 172.21.15.89:0/4267611947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd4c005170 0x7efd6815cef0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efd5800f780 tx=0x7efd5800fa90 comp rx=0 tx=0).stop 2024-04-02T07:35:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.038+0000 7efd6d400700 1 -- 172.21.15.89:0/4267611947 shutdown_connections 2024-04-02T07:35:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.038+0000 7efd6d400700 1 --2- 172.21.15.89:0/4267611947 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efd4806ac90 0x7efd4806d150 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.038+0000 7efd6d400700 1 --2- 172.21.15.89:0/4267611947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd4c005170 0x7efd6815cef0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.038+0000 7efd6d400700 1 --2- 172.21.15.89:0/4267611947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd6815f5e0 0x7efd6814e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.038+0000 7efd6d400700 1 --2- 172.21.15.89:0/4267611947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd681588c0 0x7efd6815e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.038+0000 7efd6d400700 1 -- 172.21.15.89:0/4267611947 >> 172.21.15.89:0/4267611947 conn(0x7efd6808aff0 msgr2=0x7efd680d2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.038+0000 7efd6d400700 1 -- 172.21.15.89:0/4267611947 shutdown_connections 2024-04-02T07:35:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.038+0000 7efd6d400700 1 -- 172.21.15.89:0/4267611947 wait complete. 2024-04-02T07:35:59.058 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-02T07:35:59.058 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:35:59.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.247+0000 7fcb42200700 1 Processor -- start 2024-04-02T07:35:59.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.248+0000 7fcb42200700 1 -- start start 2024-04-02T07:35:59.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.249+0000 7fcb42200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcb3c0d18b0 0x7fcb3c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.249+0000 7fcb42200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3c161590 0x7fcb3c161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.249+0000 7fcb42200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3c0ceda0 con 0x7fcb3c160f10 2024-04-02T07:35:59.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.249+0000 7fcb42200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb3c0d1d30 con 0x7fcb3c0d18b0 2024-04-02T07:35:59.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.249+0000 7fcb42200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c161f60 con 0x7fcb3c161590 2024-04-02T07:35:59.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.249+0000 7fcb3b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcb3c160f10 0x7fcb3c0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34794/0 (socket says 172.21.15.89:34794) 2024-04-02T07:35:59.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.249+0000 7fcb3b400700 1 -- 172.21.15.89:0/2058375550 learned_addr learned my addr 172.21.15.89:0/2058375550 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:59.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.249+0000 7fcb3aa00700 1 --2- 172.21.15.89:0/2058375550 >> v2:172.21.15.113:3300/0 conn(0x7fcb3c0d18b0 0x7fcb3c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.249+0000 7fcb3be00700 1 --2- 172.21.15.89:0/2058375550 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3c161590 0x7fcb3c161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.249+0000 7fcb3a000700 1 -- 172.21.15.89:0/2058375550 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625783322 0 0) 0x7fcb3c0ceda0 con 0x7fcb3c160f10 2024-04-02T07:35:59.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.249+0000 7fcb3a000700 1 -- 172.21.15.89:0/2058375550 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb200035b0 con 0x7fcb3c160f10 2024-04-02T07:35:59.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.250+0000 7fcb3aa00700 1 -- 172.21.15.89:0/2058375550 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3c161590 msgr2=0x7fcb3c161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:59.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.250+0000 7fcb3aa00700 1 --2- 172.21.15.89:0/2058375550 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3c161590 0x7fcb3c161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.250+0000 7fcb3aa00700 1 -- 172.21.15.89:0/2058375550 >> v1:172.21.15.89:6789/0 conn(0x7fcb3c160f10 legacy=0x7fcb3c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.250+0000 7fcb3aa00700 1 -- 172.21.15.89:0/2058375550 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3c1612f0 con 0x7fcb3c0d18b0 2024-04-02T07:35:59.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.250+0000 7fcb3be00700 1 --2- 172.21.15.89:0/2058375550 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3c161590 0x7fcb3c161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:59.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.250+0000 7fcb3aa00700 1 --2- 172.21.15.89:0/2058375550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb3c0d18b0 0x7fcb3c0cf230 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcb28009870 tx=0x7fcb28009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d5a33e544a2e79b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.250+0000 7fcb3a000700 1 -- 172.21.15.89:0/2058375550 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb28011830 con 0x7fcb3c0d18b0 2024-04-02T07:35:59.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.250+0000 7fcb3a000700 1 -- 172.21.15.89:0/2058375550 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb28011990 con 0x7fcb3c0d18b0 2024-04-02T07:35:59.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.250+0000 7fcb3a000700 1 -- 172.21.15.89:0/2058375550 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb28011d20 con 0x7fcb3c0d18b0 2024-04-02T07:35:59.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.251+0000 7fcb42200700 1 -- 172.21.15.89:0/2058375550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb3c0d18b0 msgr2=0x7fcb3c0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.251+0000 7fcb42200700 1 --2- 172.21.15.89:0/2058375550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb3c0d18b0 0x7fcb3c0cf230 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcb28009870 tx=0x7fcb28009b80 comp rx=0 tx=0).stop 2024-04-02T07:35:59.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.251+0000 7fcb42200700 1 -- 172.21.15.89:0/2058375550 shutdown_connections 2024-04-02T07:35:59.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.251+0000 7fcb42200700 1 --2- 172.21.15.89:0/2058375550 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3c161590 0x7fcb3c161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.251+0000 7fcb42200700 1 --2- 172.21.15.89:0/2058375550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb3c0d18b0 0x7fcb3c0cf230 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.251+0000 7fcb42200700 1 -- 172.21.15.89:0/2058375550 >> 172.21.15.89:0/2058375550 conn(0x7fcb3c08aff0 msgr2=0x7fcb3c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.251+0000 7fcb42200700 1 -- 172.21.15.89:0/2058375550 shutdown_connections 2024-04-02T07:35:59.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.251+0000 7fcb42200700 1 -- 172.21.15.89:0/2058375550 wait complete. 2024-04-02T07:35:59.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.253+0000 7fcb42200700 1 Processor -- start 2024-04-02T07:35:59.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.253+0000 7fcb42200700 1 -- start start 2024-04-02T07:35:59.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.253+0000 7fcb42200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3c0d18b0 0x7fcb3c155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb42200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb3c160f10 0x7fcb3c0b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb42200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb3c161590 0x7fcb3c158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb42200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c0b8c70 con 0x7fcb3c161590 2024-04-02T07:35:59.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb42200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c15e0f0 con 0x7fcb3c160f10 2024-04-02T07:35:59.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb42200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c15ddc0 con 0x7fcb3c0d18b0 2024-04-02T07:35:59.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb3be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb3c161590 0x7fcb3c158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb3be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb3c161590 0x7fcb3c158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52416/0 (socket says 172.21.15.89:52416) 2024-04-02T07:35:59.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb3be00700 1 -- 172.21.15.89:0/2935377084 learned_addr learned my addr 172.21.15.89:0/2935377084 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:59.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb3b400700 1 --2- 172.21.15.89:0/2935377084 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3c0d18b0 0x7fcb3c155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb3aa00700 1 --2- 172.21.15.89:0/2935377084 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb3c160f10 0x7fcb3c0b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb3be00700 1 -- 172.21.15.89:0/2935377084 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3c0d18b0 msgr2=0x7fcb3c155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb3be00700 1 --2- 172.21.15.89:0/2935377084 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3c0d18b0 0x7fcb3c155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb3be00700 1 -- 172.21.15.89:0/2935377084 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb3c160f10 msgr2=0x7fcb3c0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb3be00700 1 --2- 172.21.15.89:0/2935377084 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb3c160f10 0x7fcb3c0b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb3be00700 1 -- 172.21.15.89:0/2935377084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb34009a70 con 0x7fcb3c161590 2024-04-02T07:35:59.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.254+0000 7fcb3be00700 1 --2- 172.21.15.89:0/2935377084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb3c161590 0x7fcb3c158100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400efc0 tx=0x7fcb3400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.255+0000 7fcb33400700 1 -- 172.21.15.89:0/2935377084 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34009cd0 con 0x7fcb3c161590 2024-04-02T07:35:59.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.255+0000 7fcb33400700 1 -- 172.21.15.89:0/2935377084 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3400f040 con 0x7fcb3c161590 2024-04-02T07:35:59.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.255+0000 7fcb33400700 1 -- 172.21.15.89:0/2935377084 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3400cd60 con 0x7fcb3c161590 2024-04-02T07:35:59.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.255+0000 7fcb42200700 1 -- 172.21.15.89:0/2935377084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb280094a0 con 0x7fcb3c161590 2024-04-02T07:35:59.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.255+0000 7fcb42200700 1 -- 172.21.15.89:0/2935377084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3c0b86b0 con 0x7fcb3c161590 2024-04-02T07:35:59.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.256+0000 7fcb42200700 1 -- 172.21.15.89:0/2935377084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3c15e260 con 0x7fcb3c161590 2024-04-02T07:35:59.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.257+0000 7fcb33400700 1 -- 172.21.15.89:0/2935377084 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcb34014480 con 0x7fcb3c161590 2024-04-02T07:35:59.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.258+0000 7fcb33400700 1 --2- 172.21.15.89:0/2935377084 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb1c06acc0 0x7fcb1c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.258+0000 7fcb33400700 1 -- 172.21.15.89:0/2935377084 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb34089b00 con 0x7fcb3c161590 2024-04-02T07:35:59.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.258+0000 7fcb3b400700 1 --2- 172.21.15.89:0/2935377084 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb1c06acc0 0x7fcb1c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.258+0000 7fcb3b400700 1 --2- 172.21.15.89:0/2935377084 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb1c06acc0 0x7fcb1c06d180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c005910 tx=0x7fcb2c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.262+0000 7fcb33400700 1 -- 172.21.15.89:0/2935377084 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcb3408be60 con 0x7fcb3c161590 2024-04-02T07:35:59.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.464+0000 7fcb42200700 1 -- 172.21.15.89:0/2935377084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb3c067720 con 0x7fcb3c161590 2024-04-02T07:35:59.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.465+0000 7fcb33400700 1 -- 172.21.15.89:0/2935377084 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7fcb34018090 con 0x7fcb3c161590 2024-04-02T07:35:59.467 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:35:59.467 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:54.238913+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:35:59.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.468+0000 7fcb42200700 1 -- 172.21.15.89:0/2935377084 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb1c06acc0 msgr2=0x7fcb1c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.468+0000 7fcb42200700 1 --2- 172.21.15.89:0/2935377084 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb1c06acc0 0x7fcb1c06d180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c005910 tx=0x7fcb2c009040 comp rx=0 tx=0).stop 2024-04-02T07:35:59.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.468+0000 7fcb42200700 1 -- 172.21.15.89:0/2935377084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb3c161590 msgr2=0x7fcb3c158100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.468+0000 7fcb42200700 1 --2- 172.21.15.89:0/2935377084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb3c161590 0x7fcb3c158100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400efc0 tx=0x7fcb3400c3a0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.468+0000 7fcb42200700 1 -- 172.21.15.89:0/2935377084 shutdown_connections 2024-04-02T07:35:59.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.468+0000 7fcb42200700 1 --2- 172.21.15.89:0/2935377084 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb1c06acc0 0x7fcb1c06d180 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.468+0000 7fcb42200700 1 --2- 172.21.15.89:0/2935377084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb3c161590 0x7fcb3c158100 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.468+0000 7fcb42200700 1 --2- 172.21.15.89:0/2935377084 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3c0d18b0 0x7fcb3c155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.468+0000 7fcb42200700 1 --2- 172.21.15.89:0/2935377084 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb3c160f10 0x7fcb3c0b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.468+0000 7fcb42200700 1 -- 172.21.15.89:0/2935377084 >> 172.21.15.89:0/2935377084 conn(0x7fcb3c08aff0 msgr2=0x7fcb3c0ca8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.468+0000 7fcb42200700 1 -- 172.21.15.89:0/2935377084 shutdown_connections 2024-04-02T07:35:59.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.468+0000 7fcb42200700 1 -- 172.21.15.89:0/2935377084 wait complete. 2024-04-02T07:35:59.487 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-02T07:35:59.488 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-02T07:35:59.488 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-02T07:35:59.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-02T07:35:59.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-02T07:35:59.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-02T07:35:59.490 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-02T07:35:59.490 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-02T07:35:59.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.836+0000 7f127fe00700 1 Processor -- start 2024-04-02T07:35:59.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.837+0000 7f127fe00700 1 -- start start 2024-04-02T07:35:59.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.840+0000 7f127fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1280158f80 0x7f12800d1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.840+0000 7f127fe00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12800d1180 con 0x7f128015f4f0 2024-04-02T07:35:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.840+0000 7f127fe00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128015f9f0 con 0x7f12800cdb60 2024-04-02T07:35:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.840+0000 7f127fe00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f128015bc10 con 0x7f1280158f80 2024-04-02T07:35:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.842+0000 7f127e000700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f12800cdb60 0x7f128015ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:39656/0 (socket says 172.21.15.89:39656) 2024-04-02T07:35:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.842+0000 7f127e000700 1 -- 172.21.15.89:0/1252969823 learned_addr learned my addr 172.21.15.89:0/1252969823 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.842+0000 7f127ea00700 1 --2- 172.21.15.89:0/1252969823 >> v2:172.21.15.89:3300/0 conn(0x7f1280158f80 0x7f12800d1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.843+0000 7f8155000700 1 Processor -- start 2024-04-02T07:35:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.843+0000 7f8155000700 1 -- start start 2024-04-02T07:35:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.844+0000 7f127ea00700 1 -- 172.21.15.89:0/1252969823 >> v1:172.21.15.113:6789/0 conn(0x7f12800cdb60 legacy=0x7f128015ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.844+0000 7f127d600700 1 -- 172.21.15.89:0/1252969823 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409391747 0 0) 0x7f12800d1180 con 0x7f128015f4f0 2024-04-02T07:35:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.844+0000 7f127ea00700 1 -- 172.21.15.89:0/1252969823 >> v1:172.21.15.89:6789/0 conn(0x7f128015f4f0 legacy=0x7f12800cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.844+0000 7f127ea00700 1 -- 172.21.15.89:0/1252969823 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12800d1420 con 0x7f1280158f80 2024-04-02T07:35:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.844+0000 7f8155000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8150160df0 0x7f815006c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.844+0000 7f8155000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f81500b2150 0x7f81501611d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.844+0000 7f8155000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81500d43d0 con 0x7f81500b09d0 2024-04-02T07:35:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.844+0000 7f8155000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81500d3e10 con 0x7f81500b2150 2024-04-02T07:35:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.844+0000 7f8155000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81500d40f0 con 0x7f8150160df0 2024-04-02T07:35:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.845+0000 7f127ea00700 1 --2- 172.21.15.89:0/1252969823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1280158f80 0x7f12800d1c20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f12700097c0 tx=0x7f1270009ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef24a4cce113f6e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.845+0000 7f814fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f81500b2150 0x7f81501611d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.845+0000 7f814fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f81500b2150 0x7f81501611d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52426/0 (socket says 172.21.15.89:52426) 2024-04-02T07:35:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.845+0000 7f814fe00700 1 -- 172.21.15.89:0/3894980349 learned_addr learned my addr 172.21.15.89:0/3894980349 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.846+0000 7f814fe00700 1 -- 172.21.15.89:0/3894980349 >> v2:172.21.15.113:3300/0 conn(0x7f8150160df0 msgr2=0x7f815006c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.846+0000 7f814fe00700 1 --2- 172.21.15.89:0/3894980349 >> v2:172.21.15.113:3300/0 conn(0x7f8150160df0 0x7f815006c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.846+0000 7f814fe00700 1 -- 172.21.15.89:0/3894980349 >> v1:172.21.15.89:6789/0 conn(0x7f81500b09d0 legacy=0x7f81500d1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.844+0000 7f4c92000700 1 Processor -- start 2024-04-02T07:35:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.844+0000 7f4c92000700 1 -- start start 2024-04-02T07:35:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.846+0000 7f4c92000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c8c0724c0 0x7f4c8c0988d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.846+0000 7f4c92000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8c0cba40 con 0x7f4c8c0ae510 2024-04-02T07:35:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.846+0000 7f4c92000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8c0cc9f0 con 0x7f4c8c0ceeb0 2024-04-02T07:35:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.846+0000 7f4c92000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c8c0cece0 con 0x7f4c8c0724c0 2024-04-02T07:35:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.847+0000 7f814fe00700 1 -- 172.21.15.89:0/3894980349 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815006d0f0 con 0x7f81500b2150 2024-04-02T07:35:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.847+0000 7f4c8be00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c8c0724c0 0x7f4c8c0988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.847+0000 7f4c8be00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c8c0724c0 0x7f4c8c0988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33406/0 (socket says 172.21.15.89:33406) 2024-04-02T07:35:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.847+0000 7f4c8be00700 1 -- 172.21.15.89:0/1834440581 learned_addr learned my addr 172.21.15.89:0/1834440581 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.847+0000 7f4c8be00700 1 -- 172.21.15.89:0/1834440581 >> v1:172.21.15.113:6789/0 conn(0x7f4c8c0ceeb0 legacy=0x7f4c8c0cf290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T07:35:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.847+0000 7f814fe00700 1 --2- 172.21.15.89:0/3894980349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81500b2150 0x7f81501611d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f815008cd90 tx=0x7f8148009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f81bff100e5c8e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.847+0000 7f4c8be00700 1 -- 172.21.15.89:0/1834440581 >> v1:172.21.15.89:6789/0 conn(0x7f4c8c0ae510 legacy=0x7f4c8c0981c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T07:35:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.847+0000 7f4c8be00700 1 -- 172.21.15.89:0/1834440581 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c072930 con 0x7f4c8c0724c0 2024-04-02T07:35:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.848+0000 7f127d600700 1 -- 172.21.15.89:0/1252969823 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270004230 con 0x7f1280158f80 2024-04-02T07:35:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.848+0000 7f4c8be00700 1 --2- 172.21.15.89:0/1834440581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c8c0724c0 0x7f4c8c0988d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4c84007810 tx=0x7f4c84007b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c993d6536b6c0de5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.848+0000 7f814ea00700 1 -- 172.21.15.89:0/3894980349 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8148004230 con 0x7f81500b2150 2024-04-02T07:35:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.848+0000 7f37a4c00700 1 Processor -- start 2024-04-02T07:35:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.849+0000 7f814ea00700 1 -- 172.21.15.89:0/3894980349 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8148004390 con 0x7f81500b2150 2024-04-02T07:35:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.849+0000 7f814ea00700 1 -- 172.21.15.89:0/3894980349 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81480036a0 con 0x7f81500b2150 2024-04-02T07:35:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.848+0000 7f127d600700 1 -- 172.21.15.89:0/1252969823 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1270004390 con 0x7f1280158f80 2024-04-02T07:35:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.849+0000 7fbda8e00700 1 Processor -- start 2024-04-02T07:35:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.849+0000 7f4c8a000700 1 -- 172.21.15.89:0/1834440581 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c84011070 con 0x7f4c8c0724c0 2024-04-02T07:35:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.849+0000 7f127d600700 1 -- 172.21.15.89:0/1252969823 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270004230 con 0x7f1280158f80 2024-04-02T07:35:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.849+0000 7f4c8a000700 1 -- 172.21.15.89:0/1834440581 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8400cbd0 con 0x7f4c8c0724c0 2024-04-02T07:35:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.849+0000 7f4c8a000700 1 -- 172.21.15.89:0/1834440581 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c84003a20 con 0x7f4c8c0724c0 2024-04-02T07:35:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.849+0000 7f37a4c00700 1 -- start start 2024-04-02T07:35:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.850+0000 7f127fe00700 1 -- 172.21.15.89:0/1252969823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1280158f80 msgr2=0x7f12800d1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.850+0000 7f127fe00700 1 --2- 172.21.15.89:0/1252969823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1280158f80 0x7f12800d1c20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f12700097c0 tx=0x7f1270009ad0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.850+0000 7f4c92000700 1 -- 172.21.15.89:0/1834440581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c8c0724c0 msgr2=0x7f4c8c0988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.850+0000 7f4c92000700 1 --2- 172.21.15.89:0/1834440581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c8c0724c0 0x7f4c8c0988d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4c84007810 tx=0x7f4c84007b20 comp rx=0 tx=0).stop 2024-04-02T07:35:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.850+0000 7f4c92000700 1 -- 172.21.15.89:0/1834440581 shutdown_connections 2024-04-02T07:35:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.850+0000 7f4c92000700 1 --2- 172.21.15.89:0/1834440581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c8c0724c0 0x7f4c8c0988d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.850+0000 7f4c92000700 1 -- 172.21.15.89:0/1834440581 >> 172.21.15.89:0/1834440581 conn(0x7f4c8c070f00 msgr2=0x7f4c8c071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.850+0000 7f127fe00700 1 -- 172.21.15.89:0/1252969823 shutdown_connections 2024-04-02T07:35:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.850+0000 7f127fe00700 1 --2- 172.21.15.89:0/1252969823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1280158f80 0x7f12800d1c20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.850+0000 7f4c92000700 1 -- 172.21.15.89:0/1834440581 shutdown_connections 2024-04-02T07:35:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7f127fe00700 1 -- 172.21.15.89:0/1252969823 >> 172.21.15.89:0/1252969823 conn(0x7f12800b2e60 msgr2=0x7f12800b3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7f37a4c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f37a015d810 0x7f37a00b4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7f37a4c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f37a006e810 0x7f37a00cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7f37a4c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a00cc140 con 0x7f37a0072910 2024-04-02T07:35:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7f379d400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f37a0072910 0x7f37a006e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:39660/0 (socket says 172.21.15.89:39660) 2024-04-02T07:35:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7f379d400700 1 -- 172.21.15.89:0/205316231 learned_addr learned my addr 172.21.15.89:0/205316231 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7f37a4c00700 1 -- 172.21.15.89:0/205316231 --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a015df80 con 0x7f37a015d810 2024-04-02T07:35:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7f37a4c00700 1 -- 172.21.15.89:0/205316231 --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a015e260 con 0x7f37a006e810 2024-04-02T07:35:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.852+0000 7f379e800700 1 --2- 172.21.15.89:0/205316231 >> v2:172.21.15.113:3300/0 conn(0x7f37a006e810 0x7f37a00cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.852+0000 7f8155000700 1 -- 172.21.15.89:0/3894980349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81500b2150 msgr2=0x7f81501611d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.852+0000 7f8155000700 1 --2- 172.21.15.89:0/3894980349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81500b2150 0x7f81501611d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f815008cd90 tx=0x7f8148009bd0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7f4c92000700 1 -- 172.21.15.89:0/1834440581 wait complete. 2024-04-02T07:35:59.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.849+0000 7fbda8e00700 1 -- start start 2024-04-02T07:35:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7fbda8e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbda415d810 0x7fbda40b4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7fbda8e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbda4072910 0x7fbda40cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7fbda8e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda40cc140 con 0x7fbda406e100 2024-04-02T07:35:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7fbda8e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda415df80 con 0x7fbda415d810 2024-04-02T07:35:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7fbda8e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda415e260 con 0x7fbda4072910 2024-04-02T07:35:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7fbda1e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbda415d810 0x7fbda40b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7fbda1e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbda415d810 0x7fbda40b4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33410/0 (socket says 172.21.15.89:33410) 2024-04-02T07:35:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.851+0000 7fbda1e00700 1 -- 172.21.15.89:0/4266868446 learned_addr learned my addr 172.21.15.89:0/4266868446 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.852+0000 7fbda1e00700 1 -- 172.21.15.89:0/4266868446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbda4072910 msgr2=0x7fbda40cc5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T07:35:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.852+0000 7f379e800700 1 -- 172.21.15.89:0/205316231 >> v2:172.21.15.89:3300/0 conn(0x7f37a015d810 msgr2=0x7f37a00b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:59.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.852+0000 7f8155000700 1 -- 172.21.15.89:0/3894980349 shutdown_connections 2024-04-02T07:35:59.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.852+0000 7f8155000700 1 --2- 172.21.15.89:0/3894980349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81500b2150 0x7f81501611d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.852+0000 7f8155000700 1 --2- 172.21.15.89:0/3894980349 >> v2:172.21.15.113:3300/0 conn(0x7f8150160df0 0x7f815006c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.852+0000 7f8155000700 1 -- 172.21.15.89:0/3894980349 >> 172.21.15.89:0/3894980349 conn(0x7f815009c720 msgr2=0x7f81500d3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.852+0000 7f379e800700 1 --2- 172.21.15.89:0/205316231 >> v2:172.21.15.89:3300/0 conn(0x7f37a015d810 0x7f37a00b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.852+0000 7fbda1e00700 1 --2- 172.21.15.89:0/4266868446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbda4072910 0x7fbda40cc5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.853+0000 7f127fe00700 1 -- 172.21.15.89:0/1252969823 shutdown_connections 2024-04-02T07:35:59.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.853+0000 7fbda1e00700 1 -- 172.21.15.89:0/4266868446 >> v1:172.21.15.89:6789/0 conn(0x7fbda406e100 legacy=0x7fbda40ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:59.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.853+0000 7f8155000700 1 -- 172.21.15.89:0/3894980349 shutdown_connections 2024-04-02T07:35:59.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.852+0000 7f379e800700 1 -- 172.21.15.89:0/205316231 >> v1:172.21.15.113:6789/0 conn(0x7f37a0072910 legacy=0x7f37a006e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:59.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.852+0000 7f379e800700 1 -- 172.21.15.89:0/205316231 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a00cbdd0 con 0x7f37a006e810 2024-04-02T07:35:59.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.853+0000 7f8155000700 1 -- 172.21.15.89:0/3894980349 wait complete. 2024-04-02T07:35:59.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.853+0000 7fbda1e00700 1 -- 172.21.15.89:0/4266868446 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda40cbdd0 con 0x7fbda415d810 2024-04-02T07:35:59.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.853+0000 7f379e800700 1 --2- 172.21.15.89:0/205316231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37a006e810 0x7f37a00cc5d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f379800bfa0 tx=0x7f379800d330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73cca283b0520389 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.853+0000 7fbda1e00700 1 --2- 172.21.15.89:0/4266868446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbda415d810 0x7fbda40b4240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbd940098f0 tx=0x7fbd94009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=292e3fc211ff619e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.854+0000 7fbda0a00700 1 -- 172.21.15.89:0/4266868446 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94004230 con 0x7fbda415d810 2024-04-02T07:35:59.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.854+0000 7fbda0a00700 1 -- 172.21.15.89:0/4266868446 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd94004390 con 0x7fbda415d810 2024-04-02T07:35:59.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.854+0000 7fbda0a00700 1 -- 172.21.15.89:0/4266868446 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd940036a0 con 0x7fbda415d810 2024-04-02T07:35:59.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.855+0000 7f127fe00700 1 -- 172.21.15.89:0/1252969823 wait complete. 2024-04-02T07:35:59.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.855+0000 7f4c92000700 1 Processor -- start 2024-04-02T07:35:59.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.855+0000 7f4c92000700 1 -- start start 2024-04-02T07:35:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.855+0000 7f379ca00700 1 -- 172.21.15.89:0/205316231 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3798010070 con 0x7f37a006e810 2024-04-02T07:35:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.855+0000 7f379ca00700 1 -- 172.21.15.89:0/205316231 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379800b2e0 con 0x7f37a006e810 2024-04-02T07:35:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.855+0000 7f379ca00700 1 -- 172.21.15.89:0/205316231 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379800dc70 con 0x7f37a006e810 2024-04-02T07:35:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.856+0000 7f37a4c00700 1 -- 172.21.15.89:0/205316231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37a006e810 msgr2=0x7f37a00cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.856+0000 7f37a4c00700 1 --2- 172.21.15.89:0/205316231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37a006e810 0x7f37a00cc5d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f379800bfa0 tx=0x7f379800d330 comp rx=0 tx=0).stop 2024-04-02T07:35:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.857+0000 7f37a4c00700 1 -- 172.21.15.89:0/205316231 shutdown_connections 2024-04-02T07:35:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.857+0000 7f37a4c00700 1 --2- 172.21.15.89:0/205316231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37a006e810 0x7f37a00cc5d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.857+0000 7f37a4c00700 1 --2- 172.21.15.89:0/205316231 >> v2:172.21.15.89:3300/0 conn(0x7f37a015d810 0x7f37a00b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.857+0000 7f37a4c00700 1 -- 172.21.15.89:0/205316231 >> 172.21.15.89:0/205316231 conn(0x7f37a0070f00 msgr2=0x7f37a0071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.858+0000 7f127fe00700 1 Processor -- start 2024-04-02T07:35:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.856+0000 7f4c92000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c8c0724c0 0x7f4c8c094f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.856+0000 7f4c92000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c8c0ae510 0x7f4c8c0954c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.856+0000 7f4c92000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c8c0ceeb0 0x7f4c8c20b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.856+0000 7f4c92000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c2089c0 con 0x7f4c8c0724c0 2024-04-02T07:35:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.856+0000 7f4c92000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c208ca0 con 0x7f4c8c0ae510 2024-04-02T07:35:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.856+0000 7f4c92000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c208f80 con 0x7f4c8c0ceeb0 2024-04-02T07:35:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.857+0000 7f4c8aa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c8c0ae510 0x7f4c8c0954c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.857+0000 7f4c8aa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c8c0ae510 0x7f4c8c0954c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33418/0 (socket says 172.21.15.89:33418) 2024-04-02T07:35:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.857+0000 7f4c8aa00700 1 -- 172.21.15.89:0/4219615741 learned_addr learned my addr 172.21.15.89:0/4219615741 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.857+0000 7f4c8be00700 1 --2- 172.21.15.89:0/4219615741 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c8c0ceeb0 0x7f4c8c20b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.857+0000 7f4c8b400700 1 --2- 172.21.15.89:0/4219615741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c8c0724c0 0x7f4c8c094f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.858+0000 7f127fe00700 1 -- start start 2024-04-02T07:35:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.858+0000 7fbda8e00700 1 -- 172.21.15.89:0/4266868446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbda415d810 msgr2=0x7fbda40b4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.858+0000 7fbda8e00700 1 --2- 172.21.15.89:0/4266868446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbda415d810 0x7fbda40b4240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbd940098f0 tx=0x7fbd94009c00 comp rx=0 tx=0).stop 2024-04-02T07:35:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.858+0000 7fbda8e00700 1 -- 172.21.15.89:0/4266868446 shutdown_connections 2024-04-02T07:35:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.858+0000 7fbda8e00700 1 --2- 172.21.15.89:0/4266868446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbda415d810 0x7fbda40b4240 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.858+0000 7fbda8e00700 1 --2- 172.21.15.89:0/4266868446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbda4072910 0x7fbda40cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.858+0000 7fbda8e00700 1 -- 172.21.15.89:0/4266868446 >> 172.21.15.89:0/4266868446 conn(0x7fbda4070f00 msgr2=0x7fbda4071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.857+0000 7f8155000700 1 Processor -- start 2024-04-02T07:35:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.858+0000 7fbda8e00700 1 -- 172.21.15.89:0/4266868446 shutdown_connections 2024-04-02T07:35:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.858+0000 7fbda8e00700 1 -- 172.21.15.89:0/4266868446 wait complete. 2024-04-02T07:35:59.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.858+0000 7f4c8b400700 1 -- 172.21.15.89:0/4219615741 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c8c0ceeb0 msgr2=0x7f4c8c20b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f4c8b400700 1 --2- 172.21.15.89:0/4219615741 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c8c0ceeb0 0x7f4c8c20b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f4c8b400700 1 -- 172.21.15.89:0/4219615741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c8c0ae510 msgr2=0x7f4c8c0954c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f4c8b400700 1 --2- 172.21.15.89:0/4219615741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c8c0ae510 0x7f4c8c0954c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f4c8b400700 1 -- 172.21.15.89:0/4219615741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c840074f0 con 0x7f4c8c0724c0 2024-04-02T07:35:59.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f8155000700 1 -- start start 2024-04-02T07:35:59.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f127fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12800cdb60 0x7f1280148e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f127fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1280158f80 0x7f128015bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f127fe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f128015f4f0 0x7f128015c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f127fe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1280156560 con 0x7f1280158f80 2024-04-02T07:35:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f127fe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1280156840 con 0x7f128015f4f0 2024-04-02T07:35:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f127fe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1280156b20 con 0x7f12800cdb60 2024-04-02T07:35:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f127f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f128015f4f0 0x7f128015c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f127f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f128015f4f0 0x7f128015c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33432/0 (socket says 172.21.15.89:33432) 2024-04-02T07:35:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f127f400700 1 -- 172.21.15.89:0/542272977 learned_addr learned my addr 172.21.15.89:0/542272977 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f127ea00700 1 --2- 172.21.15.89:0/542272977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12800cdb60 0x7f1280148e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f4c8b400700 1 --2- 172.21.15.89:0/4219615741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c8c0724c0 0x7f4c8c094f80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c006fd0 tx=0x7f4c7c00b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.860+0000 7f127ea00700 1 -- 172.21.15.89:0/542272977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f128015f4f0 msgr2=0x7f128015c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.860+0000 7f127ea00700 1 --2- 172.21.15.89:0/542272977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f128015f4f0 0x7f128015c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.860+0000 7f127ea00700 1 -- 172.21.15.89:0/542272977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1280158f80 msgr2=0x7f128015bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.860+0000 7f127ea00700 1 --2- 172.21.15.89:0/542272977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1280158f80 0x7f128015bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.860+0000 7f127ea00700 1 -- 172.21.15.89:0/542272977 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12700094a0 con 0x7f12800cdb60 2024-04-02T07:35:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.860+0000 7f127ea00700 1 --2- 172.21.15.89:0/542272977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12800cdb60 0x7f1280148e80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1270015000 tx=0x7f1270004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7f37a4c00700 1 -- 172.21.15.89:0/205316231 shutdown_connections 2024-04-02T07:35:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7f37a4c00700 1 -- 172.21.15.89:0/205316231 wait complete. 2024-04-02T07:35:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7f8155000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81500b2150 0x7f81500b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.860+0000 7f4c83400700 1 -- 172.21.15.89:0/4219615741 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c011830 con 0x7f4c8c0724c0 2024-04-02T07:35:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.860+0000 7f4c83400700 1 -- 172.21.15.89:0/4219615741 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7c0119b0 con 0x7f4c8c0724c0 2024-04-02T07:35:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.860+0000 7f4c83400700 1 -- 172.21.15.89:0/4219615741 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c011d60 con 0x7f4c8c0724c0 2024-04-02T07:35:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7f1276a00700 1 -- 172.21.15.89:0/542272977 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270011830 con 0x7f12800cdb60 2024-04-02T07:35:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7f1276a00700 1 -- 172.21.15.89:0/542272977 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127001f070 con 0x7f12800cdb60 2024-04-02T07:35:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7f1276a00700 1 -- 172.21.15.89:0/542272977 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270011bd0 con 0x7f12800cdb60 2024-04-02T07:35:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7fbda8e00700 1 Processor -- start 2024-04-02T07:35:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7f8155000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8150160df0 0x7f81500b8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7f814f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81500b2150 0x7f81500b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7fbda8e00700 1 -- start start 2024-04-02T07:35:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.862+0000 7f127fe00700 1 -- 172.21.15.89:0/542272977 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1280156e00 con 0x7f12800cdb60 2024-04-02T07:35:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.862+0000 7f814f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81500b2150 0x7f81500b1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52456/0 (socket says 172.21.15.89:52456) 2024-04-02T07:35:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.862+0000 7f8155000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81500c6f80 0x7f81500c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.862+0000 7f8155000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81500b1ee0 con 0x7f81500b2150 2024-04-02T07:35:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.862+0000 7f8155000700 1 -- 172.21.15.89:0/2085192572 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81500b1c00 con 0x7f8150160df0 2024-04-02T07:35:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.862+0000 7f814f400700 1 -- 172.21.15.89:0/2085192572 learned_addr learned my addr 172.21.15.89:0/2085192572 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.862+0000 7f8155000700 1 -- 172.21.15.89:0/2085192572 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81501482c0 con 0x7f81500c6f80 2024-04-02T07:35:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.862+0000 7f127fe00700 1 -- 172.21.15.89:0/542272977 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1280157600 con 0x7f12800cdb60 2024-04-02T07:35:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7f4c92000700 1 -- 172.21.15.89:0/4219615741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8c09b730 con 0x7f4c8c0724c0 2024-04-02T07:35:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7f4c92000700 1 -- 172.21.15.89:0/4219615741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8c1699c0 con 0x7f4c8c0724c0 2024-04-02T07:35:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.862+0000 7f8146a00700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8150160df0 0x7f81500b8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.863+0000 7f4c92000700 1 -- 172.21.15.89:0/4219615741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4c8c06dfc0 con 0x7f4c8c0724c0 2024-04-02T07:35:59.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.863+0000 7f127fe00700 1 -- 172.21.15.89:0/542272977 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f128015fcd0 con 0x7f12800cdb60 2024-04-02T07:35:59.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.863+0000 7f814fe00700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81500c6f80 0x7f81500c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.863+0000 7f8146a00700 1 -- 172.21.15.89:0/2085192572 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81500c6f80 msgr2=0x7f81500c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.863+0000 7f8146a00700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81500c6f80 0x7f81500c7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.863+0000 7f8146a00700 1 -- 172.21.15.89:0/2085192572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81500b2150 msgr2=0x7f81500b1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.863+0000 7f8146a00700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81500b2150 0x7f81500b1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.863+0000 7f8146a00700 1 -- 172.21.15.89:0/2085192572 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8148009550 con 0x7f8150160df0 2024-04-02T07:35:59.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.863+0000 7f814f400700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81500b2150 0x7f81500b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:35:59.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.864+0000 7f8146a00700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8150160df0 0x7f81500b8900 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f813800c750 tx=0x7f813800ca60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.864+0000 7f8147e00700 1 -- 172.21.15.89:0/2085192572 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81380054e0 con 0x7f8150160df0 2024-04-02T07:35:59.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.864+0000 7f8147e00700 1 -- 172.21.15.89:0/2085192572 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8138003680 con 0x7f8150160df0 2024-04-02T07:35:59.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.864+0000 7f8147e00700 1 -- 172.21.15.89:0/2085192572 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8138003a30 con 0x7f8150160df0 2024-04-02T07:35:59.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.864+0000 7fbda8e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbda406e100 0x7fbda4147380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.864+0000 7fbda8e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbda4072910 0x7fbda4150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.865+0000 7fbda8e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbda415d810 0x7fbda4166df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.865+0000 7fbda8e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda40b0520 con 0x7fbda415d810 2024-04-02T07:35:59.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.865+0000 7fbda8e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda41478c0 con 0x7fbda406e100 2024-04-02T07:35:59.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.865+0000 7fbda8e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda414fc30 con 0x7fbda4072910 2024-04-02T07:35:59.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.864+0000 7f8155000700 1 -- 172.21.15.89:0/2085192572 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81501485a0 con 0x7f8150160df0 2024-04-02T07:35:59.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.865+0000 7f37a4c00700 1 Processor -- start 2024-04-02T07:35:59.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.865+0000 7f8155000700 1 -- 172.21.15.89:0/2085192572 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81500cd990 con 0x7f8150160df0 2024-04-02T07:35:59.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.857+0000 7f0455400700 1 Processor -- start 2024-04-02T07:35:59.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.858+0000 7f0455400700 1 -- start start 2024-04-02T07:35:59.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f0455400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f04501657c0 0x7f0450165c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f0455400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04500d1a00 0x7f04500d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.859+0000 7f0455400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04500d1570 con 0x7f045015f830 2024-04-02T07:35:59.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.860+0000 7f0455400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04501661b0 con 0x7f04501657c0 2024-04-02T07:35:59.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.860+0000 7f0455400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045015fc10 con 0x7f04500d1a00 2024-04-02T07:35:59.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.860+0000 7f0447e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f04501657c0 0x7f0450165c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.860+0000 7f0447e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f04501657c0 0x7f0450165c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33444/0 (socket says 172.21.15.89:33444) 2024-04-02T07:35:59.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.860+0000 7f0447e00700 1 -- 172.21.15.89:0/1416295668 learned_addr learned my addr 172.21.15.89:0/1416295668 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:59.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7f0447e00700 1 -- 172.21.15.89:0/1416295668 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04500d1a00 msgr2=0x7f04500d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:59.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7f0447e00700 1 --2- 172.21.15.89:0/1416295668 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04500d1a00 0x7f04500d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7f0447e00700 1 -- 172.21.15.89:0/1416295668 >> v1:172.21.15.89:6789/0 conn(0x7f045015f830 legacy=0x7f04500cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:59.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.861+0000 7f0447e00700 1 -- 172.21.15.89:0/1416295668 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04500d06b0 con 0x7f04501657c0 2024-04-02T07:35:59.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.862+0000 7f0447e00700 1 --2- 172.21.15.89:0/1416295668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04501657c0 0x7f0450165c70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0438006fd0 tx=0x7f0438009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f34b3577d03060c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.862+0000 7f044de00700 1 -- 172.21.15.89:0/1416295668 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0438005e30 con 0x7f04501657c0 2024-04-02T07:35:59.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.862+0000 7f044de00700 1 -- 172.21.15.89:0/1416295668 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0438006030 con 0x7f04501657c0 2024-04-02T07:35:59.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.865+0000 7f37a4c00700 1 -- start start 2024-04-02T07:35:59.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.862+0000 7f044de00700 1 -- 172.21.15.89:0/1416295668 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04380063c0 con 0x7f04501657c0 2024-04-02T07:35:59.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.864+0000 7f0455400700 1 -- 172.21.15.89:0/1416295668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04501657c0 msgr2=0x7f0450165c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.864+0000 7f0455400700 1 --2- 172.21.15.89:0/1416295668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04501657c0 0x7f0450165c70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0438006fd0 tx=0x7f0438009bd0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.865+0000 7f1276a00700 1 -- 172.21.15.89:0/542272977 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1270011d30 con 0x7f12800cdb60 2024-04-02T07:35:59.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.865+0000 7fbda1e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbda406e100 0x7fbda4147380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.865+0000 7fbda1400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbda4072910 0x7fbda4150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.866+0000 7fbda1e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbda406e100 0x7fbda4147380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33466/0 (socket says 172.21.15.89:33466) 2024-04-02T07:35:59.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.866+0000 7fbda1e00700 1 -- 172.21.15.89:0/2974408706 learned_addr learned my addr 172.21.15.89:0/2974408706 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:59.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.866+0000 7f4c83400700 1 -- 172.21.15.89:0/4219615741 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4c7c01a400 con 0x7f4c8c0724c0 2024-04-02T07:35:59.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.866+0000 7f1276a00700 1 --2- 172.21.15.89:0/542272977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f126006ad60 0x7f126006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.866+0000 7f1276a00700 1 -- 172.21.15.89:0/542272977 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1270021070 con 0x7f12800cdb60 2024-04-02T07:35:59.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.866+0000 7fbda2800700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbda415d810 0x7fbda4166df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.866+0000 7f1276a00700 1 --2- 172.21.15.89:0/542272977 >> [v2:172.21.15.89:6808/43703,v1:172.21.15.89:6809/43703] conn(0x7f1260072f80 0x7f12600753a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.866+0000 7f1276a00700 1 -- 172.21.15.89:0/542272977 --> [v2:172.21.15.89:6808/43703,v1:172.21.15.89:6809/43703] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1260075a50 con 0x7f1260072f80 2024-04-02T07:35:59.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.866+0000 7f1276a00700 1 -- 172.21.15.89:0/542272977 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f127008b490 con 0x7f12800cdb60 2024-04-02T07:35:59.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7fbda1e00700 1 -- 172.21.15.89:0/2974408706 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbda4072910 msgr2=0x7fbda4150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7fbda1e00700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbda4072910 0x7fbda4150110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7fbda1e00700 1 -- 172.21.15.89:0/2974408706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbda415d810 msgr2=0x7fbda4166df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7f0455400700 1 -- 172.21.15.89:0/1416295668 shutdown_connections 2024-04-02T07:35:59.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7fbda1e00700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbda415d810 0x7fbda4166df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7f0455400700 1 --2- 172.21.15.89:0/1416295668 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04500d1a00 0x7f04500d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7f0455400700 1 --2- 172.21.15.89:0/1416295668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04501657c0 0x7f0450165c70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7f0455400700 1 -- 172.21.15.89:0/1416295668 >> 172.21.15.89:0/1416295668 conn(0x7f045008aff0 msgr2=0x7f04500d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7fbda1e00700 1 -- 172.21.15.89:0/2974408706 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd94009550 con 0x7fbda406e100 2024-04-02T07:35:59.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7f2f47e00700 1 Processor -- start 2024-04-02T07:35:59.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7f37a4c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f37a006e810 0x7f37a01f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7f127f400700 1 --2- 172.21.15.89:0/542272977 >> [v2:172.21.15.89:6808/43703,v1:172.21.15.89:6809/43703] conn(0x7f1260072f80 0x7f12600753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7f0455400700 1 -- 172.21.15.89:0/1416295668 shutdown_connections 2024-04-02T07:35:59.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7fbda1400700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbda4072910 0x7fbda4150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:59.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7f37a4c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37a0072910 0x7f37a00c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7f379de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f37a006e810 0x7f37a01f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7f379de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f37a006e810 0x7f37a01f28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:60952/0 (socket says 172.21.15.89:60952) 2024-04-02T07:35:59.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7f379de00700 1 -- 172.21.15.89:0/2884055378 learned_addr learned my addr 172.21.15.89:0/2884055378 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:59.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7f37a4c00700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37a015d810 0x7f37a0151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7fa924a00700 1 Processor -- start 2024-04-02T07:35:59.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7fbda2800700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbda415d810 0x7fbda4166df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:35:59.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7f4c83400700 1 --2- 172.21.15.89:0/4219615741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4c6c06ad60 0x7f4c6c06d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7fa924a00700 1 -- start start 2024-04-02T07:35:59.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7f37a4c00700 1 -- 172.21.15.89:0/2884055378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a01f2e00 con 0x7f37a015d810 2024-04-02T07:35:59.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7f37a4c00700 1 -- 172.21.15.89:0/2884055378 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a0150bc0 con 0x7f37a0072910 2024-04-02T07:35:59.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7f37a4c00700 1 -- 172.21.15.89:0/2884055378 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a014ffb0 con 0x7f37a006e810 2024-04-02T07:35:59.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7f4c83400700 1 -- 172.21.15.89:0/4219615741 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c7c0897e0 con 0x7f4c8c0724c0 2024-04-02T07:35:59.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7f4c8aa00700 1 --2- 172.21.15.89:0/4219615741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4c6c06ad60 0x7f4c6c06d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7f4c83400700 1 --2- 172.21.15.89:0/4219615741 >> [v2:172.21.15.89:6816/43716,v1:172.21.15.89:6817/43716] conn(0x7f4c6c072f80 0x7f4c6c0753a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7f4c83400700 1 -- 172.21.15.89:0/4219615741 --> [v2:172.21.15.89:6816/43716,v1:172.21.15.89:6817/43716] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c6c075a50 con 0x7f4c6c072f80 2024-04-02T07:35:59.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7f4c83400700 1 -- 172.21.15.89:0/4219615741 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c7c089b90 con 0x7f4c8c0724c0 2024-04-02T07:35:59.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7f8155000700 1 -- 172.21.15.89:0/2085192572 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f815006cdb0 con 0x7f8150160df0 2024-04-02T07:35:59.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.867+0000 7f2f47e00700 1 -- start start 2024-04-02T07:35:59.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.869+0000 7f127f400700 1 --2- 172.21.15.89:0/542272977 >> [v2:172.21.15.89:6808/43703,v1:172.21.15.89:6809/43703] conn(0x7f1260072f80 0x7f12600753a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.869+0000 7f379e800700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37a015d810 0x7f37a0151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.869+0000 7f379d400700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37a0072910 0x7f37a00c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7f0455400700 1 -- 172.21.15.89:0/1416295668 wait complete. 2024-04-02T07:35:59.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.869+0000 7f2f47e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f400ce500 0x7f2f400b2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.869+0000 7f2f47e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2f400b3760 0x7f2f400b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7f2f44a00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f2f400cc370 0x7f2f400ce8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:39662/0 (socket says 172.21.15.89:39662) 2024-04-02T07:35:59.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7f8147e00700 1 -- 172.21.15.89:0/2085192572 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8138003b90 con 0x7f8150160df0 2024-04-02T07:35:59.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7f379de00700 1 -- 172.21.15.89:0/2884055378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37a0072910 msgr2=0x7f37a00c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7f379de00700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37a0072910 0x7f37a00c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7f4c8be00700 1 --2- 172.21.15.89:0/4219615741 >> [v2:172.21.15.89:6816/43716,v1:172.21.15.89:6817/43716] conn(0x7f4c6c072f80 0x7f4c6c0753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7fa924a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa9180e10b0 0x7fa918054e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7fa924a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa91803ca70 0x7fa918051b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7f379de00700 1 -- 172.21.15.89:0/2884055378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37a015d810 msgr2=0x7f37a0151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7f379de00700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37a015d810 0x7f37a0151050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.868+0000 7fbda1e00700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbda406e100 0x7fbda4147380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009130 tx=0x7fbd94004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7f379de00700 1 -- 172.21.15.89:0/2884055378 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379800bc80 con 0x7f37a006e810 2024-04-02T07:35:59.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7f1276a00700 1 -- 172.21.15.89:0/542272977 <== osd.1 v2:172.21.15.89:6808/43703 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1260075a50 con 0x7f1260072f80 2024-04-02T07:35:59.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7fa924a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa918056380 con 0x7fa918056810 2024-04-02T07:35:59.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7fa924a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9180e1490 con 0x7fa91803ca70 2024-04-02T07:35:59.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.870+0000 7fa924a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa918055770 con 0x7fa9180e10b0 2024-04-02T07:35:59.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7f379e800700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37a015d810 0x7f37a0151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:35:59.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7f2f47e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f40163190 con 0x7f2f400cc370 2024-04-02T07:35:59.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7f2f47e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f400b40d0 con 0x7f2f400b3760 2024-04-02T07:35:59.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7f2f47e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40162580 con 0x7f2f400ce500 2024-04-02T07:35:59.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7f2f44a00700 1 -- 172.21.15.89:0/2244615442 learned_addr learned my addr 172.21.15.89:0/2244615442 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:59.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7f8147e00700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f812c06ace0 0x7f812c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7f8147e00700 1 -- 172.21.15.89:0/2085192572 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8138088ac0 con 0x7f8150160df0 2024-04-02T07:35:59.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7f8147e00700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.89:6800/43701,v1:172.21.15.89:6801/43701] conn(0x7f812c072f00 0x7f812c075320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7f8147e00700 1 -- 172.21.15.89:0/2085192572 --> [v2:172.21.15.89:6800/43701,v1:172.21.15.89:6801/43701] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f812c0759d0 con 0x7f812c072f00 2024-04-02T07:35:59.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7f8147e00700 1 -- 172.21.15.89:0/2085192572 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8138088e30 con 0x7f8150160df0 2024-04-02T07:35:59.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7f127e000700 1 --2- 172.21.15.89:0/542272977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f126006ad60 0x7f126006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.873+0000 7f127e000700 1 --2- 172.21.15.89:0/542272977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f126006ad60 0x7f126006d220 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f126c000c00 tx=0x7f126c007a10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7fbd9a000700 1 -- 172.21.15.89:0/2974408706 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9400b660 con 0x7fbda406e100 2024-04-02T07:35:59.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7fbd9a000700 1 -- 172.21.15.89:0/2974408706 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9400b7c0 con 0x7fbda406e100 2024-04-02T07:35:59.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7fbd9a000700 1 -- 172.21.15.89:0/2974408706 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9400bb70 con 0x7fbda406e100 2024-04-02T07:35:59.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7fbda8e00700 1 -- 172.21.15.89:0/2974408706 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda40af8a0 con 0x7fbda406e100 2024-04-02T07:35:59.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7fbda8e00700 1 -- 172.21.15.89:0/2974408706 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda40afe60 con 0x7fbda406e100 2024-04-02T07:35:59.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.873+0000 7fbda8e00700 1 -- 172.21.15.89:0/2974408706 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbd7c000fc0 con 0x7fbda406e100 2024-04-02T07:35:59.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.872+0000 7f379de00700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f37a006e810 0x7f37a01f28c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f378c00b4a0 tx=0x7f378c00b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.872+0000 7f3796000700 1 -- 172.21.15.89:0/2884055378 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c011830 con 0x7f37a006e810 2024-04-02T07:35:59.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.872+0000 7f0455400700 1 Processor -- start 2024-04-02T07:35:59.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.872+0000 7f0455400700 1 -- start start 2024-04-02T07:35:59.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7f2f45400700 1 --2- 172.21.15.89:0/2244615442 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f400ce500 0x7f2f400b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.872+0000 7f2f45400700 1 -- 172.21.15.89:0/2244615442 >> v2:172.21.15.113:3300/0 conn(0x7f2f400b3760 msgr2=0x7f2f400b3b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T07:35:59.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.872+0000 7f2f45400700 1 --2- 172.21.15.89:0/2244615442 >> v2:172.21.15.113:3300/0 conn(0x7f2f400b3760 0x7f2f400b3b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.872+0000 7f2f45400700 1 -- 172.21.15.89:0/2244615442 >> v1:172.21.15.113:6789/0 conn(0x7f2f400cc370 legacy=0x7f2f400ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.872+0000 7f2f45400700 1 -- 172.21.15.89:0/2244615442 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f40162860 con 0x7f2f400ce500 2024-04-02T07:35:59.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.872+0000 7f2f45400700 1 --2- 172.21.15.89:0/2244615442 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f400ce500 0x7f2f400b2070 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f34009840 tx=0x7f2f34009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=182614a6f793c7b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.873+0000 7f2f3fe00700 1 -- 172.21.15.89:0/2244615442 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34011830 con 0x7f2f400ce500 2024-04-02T07:35:59.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.873+0000 7f2f3fe00700 1 -- 172.21.15.89:0/2244615442 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f34011990 con 0x7f2f400ce500 2024-04-02T07:35:59.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.873+0000 7f2f3fe00700 1 -- 172.21.15.89:0/2244615442 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34011d00 con 0x7f2f400ce500 2024-04-02T07:35:59.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7fa91e800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa9180e10b0 0x7fa918054e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7fa91f200700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fa918056810 0x7fa91800dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:39666/0 (socket says 172.21.15.89:39666) 2024-04-02T07:35:59.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.871+0000 7fa91f200700 1 -- 172.21.15.89:0/146272211 learned_addr learned my addr 172.21.15.89:0/146272211 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:35:59.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.873+0000 7f37a4c00700 1 -- 172.21.15.89:0/2884055378 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a0150290 con 0x7f37a006e810 2024-04-02T07:35:59.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7f2f47e00700 1 -- 172.21.15.89:0/2244615442 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f400ce500 msgr2=0x7f2f400b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7f2f47e00700 1 --2- 172.21.15.89:0/2244615442 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f400ce500 0x7f2f400b2070 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f34009840 tx=0x7f2f34009b50 comp rx=0 tx=0).stop 2024-04-02T07:35:59.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.872+0000 7fa91de00700 1 --2- 172.21.15.89:0/146272211 >> v2:172.21.15.89:3300/0 conn(0x7fa91803ca70 0x7fa918051b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.872+0000 7fa91d400700 1 -- 172.21.15.89:0/146272211 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499748422 0 0) 0x7fa918056380 con 0x7fa918056810 2024-04-02T07:35:59.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.872+0000 7fa91d400700 1 -- 172.21.15.89:0/146272211 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9040035b0 con 0x7fa918056810 2024-04-02T07:35:59.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.873+0000 7fa91d400700 1 -- 172.21.15.89:0/146272211 <== mon.1 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3682105323 0 0) 0x7fa9040035b0 con 0x7fa918056810 2024-04-02T07:35:59.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.873+0000 7fa91d400700 1 -- 172.21.15.89:0/146272211 >> v2:172.21.15.113:3300/0 conn(0x7fa9180e10b0 msgr2=0x7fa918054e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:59.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7fa91d400700 1 --2- 172.21.15.89:0/146272211 >> v2:172.21.15.113:3300/0 conn(0x7fa9180e10b0 0x7fa918054e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7fa91d400700 1 -- 172.21.15.89:0/146272211 >> v2:172.21.15.89:3300/0 conn(0x7fa91803ca70 msgr2=0x7fa918051b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:59.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7fa91d400700 1 --2- 172.21.15.89:0/146272211 >> v2:172.21.15.89:3300/0 conn(0x7fa91803ca70 0x7fa918051b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7fa91d400700 1 -- 172.21.15.89:0/146272211 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa918055a50 con 0x7fa918056810 2024-04-02T07:35:59.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.873+0000 7f37a4c00700 1 -- 172.21.15.89:0/2884055378 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a0150850 con 0x7f37a006e810 2024-04-02T07:35:59.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7fa91d400700 1 -- 172.21.15.89:0/146272211 <== mon.1 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fa9200686c0 con 0x7fa918056810 2024-04-02T07:35:59.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7fa91de00700 1 --2- 172.21.15.89:0/146272211 >> v2:172.21.15.89:3300/0 conn(0x7fa91803ca70 0x7fa918051b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:35:59.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7f3796000700 1 -- 172.21.15.89:0/2884055378 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378c011990 con 0x7f37a006e810 2024-04-02T07:35:59.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7f3796000700 1 -- 172.21.15.89:0/2884055378 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c011d40 con 0x7f37a006e810 2024-04-02T07:35:59.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7fa91e800700 1 --2- 172.21.15.89:0/146272211 >> v2:172.21.15.113:3300/0 conn(0x7fa9180e10b0 0x7fa918054e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:35:59.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7f814f400700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f812c06ace0 0x7f812c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7fa91d400700 1 -- 172.21.15.89:0/146272211 >> v1:172.21.15.113:6789/0 conn(0x7fa918056810 legacy=0x7fa91800dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7fa91d400700 1 --2- 172.21.15.89:0/146272211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa904003a00 0x7fa904003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7f814fe00700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.89:6800/43701,v1:172.21.15.89:6801/43701] conn(0x7f812c072f00 0x7f812c075320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f4c8aa00700 1 --2- 172.21.15.89:0/4219615741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4c6c06ad60 0x7f4c6c06d220 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c068120 tx=0x7f4c78009210 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f2f47e00700 1 -- 172.21.15.89:0/2244615442 shutdown_connections 2024-04-02T07:35:59.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7f0455400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04500d1a00 0x7f04501623b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7f0455400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f045015f830 0x7f045009e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7f0455400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04501657c0 0x7f0450156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7f044e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04500d1a00 0x7f04501623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f044e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04500d1a00 0x7f04501623b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52500/0 (socket says 172.21.15.89:52500) 2024-04-02T07:35:59.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f044e800700 1 -- 172.21.15.89:0/716504323 learned_addr learned my addr 172.21.15.89:0/716504323 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:59.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f0447e00700 1 --2- 172.21.15.89:0/716504323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f045015f830 0x7f045009e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f0455400700 1 -- 172.21.15.89:0/716504323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045009ec00 con 0x7f04500d1a00 2024-04-02T07:35:59.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f0455400700 1 -- 172.21.15.89:0/716504323 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045009eee0 con 0x7f045015f830 2024-04-02T07:35:59.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f0455400700 1 -- 172.21.15.89:0/716504323 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045009f1c0 con 0x7f04501657c0 2024-04-02T07:35:59.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f044f200700 1 --2- 172.21.15.89:0/716504323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04501657c0 0x7f0450156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f37a4c00700 1 -- 172.21.15.89:0/2884055378 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3778000fc0 con 0x7f37a006e810 2024-04-02T07:35:59.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f044f200700 1 -- 172.21.15.89:0/716504323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f045015f830 msgr2=0x7f045009e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f044f200700 1 --2- 172.21.15.89:0/716504323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f045015f830 0x7f045009e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f044f200700 1 -- 172.21.15.89:0/716504323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04500d1a00 msgr2=0x7f04501623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f044f200700 1 --2- 172.21.15.89:0/716504323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04500d1a00 0x7f04501623b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f2f47e00700 1 --2- 172.21.15.89:0/2244615442 >> v2:172.21.15.113:3300/0 conn(0x7f2f400b3760 0x7f2f400b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f2f47e00700 1 --2- 172.21.15.89:0/2244615442 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f400ce500 0x7f2f400b2070 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f2f47e00700 1 -- 172.21.15.89:0/2244615442 >> 172.21.15.89:0/2244615442 conn(0x7f2f400b0d70 msgr2=0x7f2f400b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.876+0000 7f2f47e00700 1 -- 172.21.15.89:0/2244615442 shutdown_connections 2024-04-02T07:35:59.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.876+0000 7f2f47e00700 1 -- 172.21.15.89:0/2244615442 wait complete. 2024-04-02T07:35:59.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.878+0000 7fbd9a000700 1 -- 172.21.15.89:0/2974408706 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbd94021070 con 0x7fbda406e100 2024-04-02T07:35:59.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7fa91d400700 1 --2- 172.21.15.89:0/146272211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa904004850 0x7fa904004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7fa91d400700 1 --2- 172.21.15.89:0/146272211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa904005210 0x7fa90400fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7fa91d400700 1 -- 172.21.15.89:0/146272211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9180e1490 con 0x7fa904003a00 2024-04-02T07:35:59.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7fa91d400700 1 -- 172.21.15.89:0/146272211 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa918055770 con 0x7fa904004850 2024-04-02T07:35:59.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.874+0000 7fa91d400700 1 -- 172.21.15.89:0/146272211 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa904010580 con 0x7fa904005210 2024-04-02T07:35:59.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7fa91de00700 1 --2- 172.21.15.89:0/146272211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa904004850 0x7fa904004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.876+0000 7fa91f200700 1 --2- 172.21.15.89:0/146272211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa904005210 0x7fa90400fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.876+0000 7fa91f200700 1 -- 172.21.15.89:0/146272211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa904004850 msgr2=0x7fa904004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:35:59.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.876+0000 7fa91f200700 1 --2- 172.21.15.89:0/146272211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa904004850 0x7fa904004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.876+0000 7fa91f200700 1 -- 172.21.15.89:0/146272211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa904003a00 msgr2=0x7fa904003de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T07:35:59.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.878+0000 7f2f47e00700 1 Processor -- start 2024-04-02T07:35:59.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f4c8be00700 1 --2- 172.21.15.89:0/4219615741 >> [v2:172.21.15.89:6816/43716,v1:172.21.15.89:6817/43716] conn(0x7f4c6c072f80 0x7f4c6c0753a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.877+0000 7f4c83400700 1 -- 172.21.15.89:0/4219615741 <== osd.3 v2:172.21.15.89:6816/43716 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4c6c075a50 con 0x7f4c6c072f80 2024-04-02T07:35:59.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.879+0000 7f2f47e00700 1 -- start start 2024-04-02T07:35:59.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.879+0000 7fa91e800700 1 --2- 172.21.15.89:0/146272211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa904003a00 0x7fa904003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.879+0000 7fa91f200700 1 --2- 172.21.15.89:0/146272211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa904003a00 0x7fa904003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.879+0000 7fa91f200700 1 -- 172.21.15.89:0/146272211 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9040035b0 con 0x7fa904005210 2024-04-02T07:35:59.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.879+0000 7fa91f200700 1 -- 172.21.15.89:0/146272211 reap_dead start 2024-04-02T07:35:59.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.875+0000 7f044f200700 1 -- 172.21.15.89:0/716504323 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0438009500 con 0x7f04501657c0 2024-04-02T07:35:59.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.876+0000 7f044f200700 1 --2- 172.21.15.89:0/716504323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04501657c0 0x7f0450156ea0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f044800b3f0 tx=0x7f044800b700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.877+0000 7f0446a00700 1 -- 172.21.15.89:0/716504323 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0448011630 con 0x7f04501657c0 2024-04-02T07:35:59.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.877+0000 7f0446a00700 1 -- 172.21.15.89:0/716504323 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0448011790 con 0x7f04501657c0 2024-04-02T07:35:59.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.877+0000 7f0455400700 1 -- 172.21.15.89:0/716504323 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045009f4a0 con 0x7f04501657c0 2024-04-02T07:35:59.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.877+0000 7f0455400700 1 -- 172.21.15.89:0/716504323 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f045009e8a0 con 0x7f04501657c0 2024-04-02T07:35:59.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.877+0000 7f0446a00700 1 -- 172.21.15.89:0/716504323 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0448011b40 con 0x7f04501657c0 2024-04-02T07:35:59.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.879+0000 7f043fe00700 1 -- 172.21.15.89:0/716504323 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0420000fc0 con 0x7f04501657c0 2024-04-02T07:35:59.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.879+0000 7fa91f200700 1 --2- 172.21.15.89:0/146272211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa904005210 0x7fa90400fe20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa920062370 tx=0x7fa904005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8da8394c08212e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.880+0000 7f814fe00700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.89:6800/43701,v1:172.21.15.89:6801/43701] conn(0x7f812c072f00 0x7f812c075320 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.880+0000 7fa91d400700 1 -- 172.21.15.89:0/146272211 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92008e070 con 0x7fa904005210 2024-04-02T07:35:59.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.880+0000 7fa91d400700 1 -- 172.21.15.89:0/146272211 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92006bc40 con 0x7fa904005210 2024-04-02T07:35:59.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.880+0000 7fa91d400700 1 -- 172.21.15.89:0/146272211 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa920093420 con 0x7fa904005210 2024-04-02T07:35:59.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.880+0000 7f814f400700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f812c06ace0 0x7f812c06d1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8140009fd0 tx=0x7f8140009630 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.881+0000 7fbd9a000700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbd8406ace0 0x7fbd8406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.881+0000 7fa924a00700 1 -- 172.21.15.89:0/146272211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa904005210 msgr2=0x7fa90400fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.881+0000 7fa924a00700 1 --2- 172.21.15.89:0/146272211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa904005210 0x7fa90400fe20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa920062370 tx=0x7fa904005880 comp rx=0 tx=0).stop 2024-04-02T07:35:59.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.880+0000 7f2f47e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f400b3760 0x7f2f40159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.880+0000 7f2f47e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f400cc370 0x7f2f4015ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.880+0000 7f2f47e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f400ce500 0x7f2f401506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.881+0000 7f2f47e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f401f7f40 con 0x7f2f400cc370 2024-04-02T07:35:59.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.881+0000 7f2f47e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400ac050 con 0x7f2f400b3760 2024-04-02T07:35:59.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.881+0000 7f2f47e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400ac610 con 0x7f2f400ce500 2024-04-02T07:35:59.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.881+0000 7f2f45400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f400b3760 0x7f2f40159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.881+0000 7f2f45400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f400b3760 0x7f2f40159f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33514/0 (socket says 172.21.15.89:33514) 2024-04-02T07:35:59.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.881+0000 7f2f45400700 1 -- 172.21.15.89:0/3603225041 learned_addr learned my addr 172.21.15.89:0/3603225041 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:59.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.881+0000 7f2f45e00700 1 --2- 172.21.15.89:0/3603225041 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f400ce500 0x7f2f401506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.881+0000 7fa924a00700 1 -- 172.21.15.89:0/146272211 shutdown_connections 2024-04-02T07:35:59.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.881+0000 7fa924a00700 1 --2- 172.21.15.89:0/146272211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa904005210 0x7fa90400fe20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.881+0000 7fa924a00700 1 -- 172.21.15.89:0/146272211 >> 172.21.15.89:0/146272211 conn(0x7fa918056d30 msgr2=0x7fa918057140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.879+0000 7f3796000700 1 -- 172.21.15.89:0/2884055378 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f378c01a400 con 0x7f37a006e810 2024-04-02T07:35:59.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.882+0000 7f8147e00700 1 -- 172.21.15.89:0/2085192572 <== osd.0 v2:172.21.15.89:6800/43701 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f812c0759d0 con 0x7f812c072f00 2024-04-02T07:35:59.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.882+0000 7fbd9a000700 1 -- 172.21.15.89:0/2974408706 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd9408aa70 con 0x7fbda406e100 2024-04-02T07:35:59.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.882+0000 7fbda1400700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbd8406ace0 0x7fbd8406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.882+0000 7fbd9a000700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.89:6824/43713,v1:172.21.15.89:6825/43713] conn(0x7fbd84072f00 0x7fbd84075320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.883+0000 7fbd9a000700 1 -- 172.21.15.89:0/2974408706 --> [v2:172.21.15.89:6824/43713,v1:172.21.15.89:6825/43713] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd840759d0 con 0x7fbd84072f00 2024-04-02T07:35:59.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.883+0000 7fbd9a000700 1 -- 172.21.15.89:0/2974408706 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd9408f2c0 con 0x7fbda406e100 2024-04-02T07:35:59.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.883+0000 7fbda2800700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.89:6824/43713,v1:172.21.15.89:6825/43713] conn(0x7fbd84072f00 0x7fbd84075320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.884+0000 7fbda2800700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.89:6824/43713,v1:172.21.15.89:6825/43713] conn(0x7fbd84072f00 0x7fbd84075320 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.884+0000 7f3796000700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f378006ad60 0x7f378006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.884+0000 7f3796000700 1 -- 172.21.15.89:0/2884055378 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f378c0897b0 con 0x7f37a006e810 2024-04-02T07:35:59.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.885+0000 7f3796000700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.113:6808/43940,v1:172.21.15.113:6809/43940] conn(0x7f3780072f80 0x7f37800753a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.885+0000 7f3796000700 1 -- 172.21.15.89:0/2884055378 --> [v2:172.21.15.113:6808/43940,v1:172.21.15.113:6809/43940] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3780075a50 con 0x7f3780072f80 2024-04-02T07:35:59.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.885+0000 7f3796000700 1 -- 172.21.15.89:0/2884055378 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f378c089b60 con 0x7f37a006e810 2024-04-02T07:35:59.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.881+0000 7f0446a00700 1 -- 172.21.15.89:0/716504323 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0448011ca0 con 0x7f04501657c0 2024-04-02T07:35:59.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.882+0000 7f0446a00700 1 --2- 172.21.15.89:0/716504323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f042c06ace0 0x7f042c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.883+0000 7f0446a00700 1 -- 172.21.15.89:0/716504323 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04480172c0 con 0x7f04501657c0 2024-04-02T07:35:59.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.883+0000 7f044e800700 1 --2- 172.21.15.89:0/716504323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f042c06ace0 0x7f042c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.883+0000 7f0446a00700 1 --2- 172.21.15.89:0/716504323 >> [v2:172.21.15.113:6800/43939,v1:172.21.15.113:6801/43939] conn(0x7f042c073be0 0x7f042c076080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.883+0000 7f0446a00700 1 -- 172.21.15.89:0/716504323 --> [v2:172.21.15.113:6800/43939,v1:172.21.15.113:6801/43939] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f042c076750 con 0x7f042c073be0 2024-04-02T07:35:59.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.883+0000 7f0446a00700 1 -- 172.21.15.89:0/716504323 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0448057dd0 con 0x7f04501657c0 2024-04-02T07:35:59.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.882+0000 7f2f45e00700 1 -- 172.21.15.89:0/3603225041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f400b3760 msgr2=0x7f2f40159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.882+0000 7f2f45e00700 1 --2- 172.21.15.89:0/3603225041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f400b3760 0x7f2f40159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.882+0000 7f2f45e00700 1 -- 172.21.15.89:0/3603225041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f400cc370 msgr2=0x7f2f4015ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.882+0000 7f2f45e00700 1 --2- 172.21.15.89:0/3603225041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f400cc370 0x7f2f4015ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.882+0000 7f2f45e00700 1 -- 172.21.15.89:0/3603225041 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f340094a0 con 0x7f2f400ce500 2024-04-02T07:35:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.882+0000 7f2f45e00700 1 --2- 172.21.15.89:0/3603225041 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f400ce500 0x7f2f401506d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800be00 tx=0x7f2f3800bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.882+0000 7f2f3d600700 1 -- 172.21.15.89:0/3603225041 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3800cc90 con 0x7f2f400ce500 2024-04-02T07:35:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.883+0000 7f2f3d600700 1 -- 172.21.15.89:0/3603225041 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3800ce10 con 0x7f2f400ce500 2024-04-02T07:35:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.883+0000 7f2f3d600700 1 -- 172.21.15.89:0/3603225041 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38014680 con 0x7f2f400ce500 2024-04-02T07:35:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.883+0000 7f2f47e00700 1 -- 172.21.15.89:0/3603225041 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f4015d230 con 0x7f2f400ce500 2024-04-02T07:35:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.883+0000 7f2f47e00700 1 -- 172.21.15.89:0/3603225041 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f4015a840 con 0x7f2f400ce500 2024-04-02T07:35:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.882+0000 7fa924a00700 1 -- 172.21.15.89:0/146272211 shutdown_connections 2024-04-02T07:35:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.882+0000 7fa924a00700 1 -- 172.21.15.89:0/146272211 wait complete. 2024-04-02T07:35:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.884+0000 7fa924a00700 1 Processor -- start 2024-04-02T07:35:59.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.885+0000 7fa924a00700 1 -- start start 2024-04-02T07:35:59.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.884+0000 7f0447e00700 1 --2- 172.21.15.89:0/716504323 >> [v2:172.21.15.113:6800/43939,v1:172.21.15.113:6801/43939] conn(0x7f042c073be0 0x7f042c076080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.885+0000 7fa924a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa904005210 0x7fa918036790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7fa924a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa918050860 0x7fa9180e2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7fa924a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9180091a0 0x7fa918009630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7fbd9a000700 1 -- 172.21.15.89:0/2974408706 <== osd.2 v2:172.21.15.89:6824/43713 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fbd840759d0 con 0x7fbd84072f00 2024-04-02T07:35:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7fa924a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa918009b70 con 0x7fa904005210 2024-04-02T07:35:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7f044e800700 1 --2- 172.21.15.89:0/716504323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f042c06ace0 0x7f042c06d1a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f044000f3c0 tx=0x7f0440004380 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7f379d400700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f378006ad60 0x7f378006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7fa924a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa918036d70 con 0x7fa9180091a0 2024-04-02T07:35:59.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7fa924a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa918037050 con 0x7fa918050860 2024-04-02T07:35:59.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7fa91de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa918050860 0x7fa9180e2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7fa91e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa904005210 0x7fa918036790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7f379e800700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.113:6808/43940,v1:172.21.15.113:6809/43940] conn(0x7f3780072f80 0x7f37800753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7fa91f200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9180091a0 0x7fa918009630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7fa91de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa918050860 0x7fa9180e2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:60982/0 (socket says 172.21.15.89:60982) 2024-04-02T07:35:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7fa91de00700 1 -- 172.21.15.89:0/3807770588 learned_addr learned my addr 172.21.15.89:0/3807770588 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:35:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7fa91de00700 1 -- 172.21.15.89:0/3807770588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9180091a0 msgr2=0x7fa918009630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7fa91de00700 1 --2- 172.21.15.89:0/3807770588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9180091a0 0x7fa918009630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.886+0000 7fa91de00700 1 -- 172.21.15.89:0/3807770588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa904005210 msgr2=0x7fa918036790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.887+0000 7f379d400700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f378006ad60 0x7f378006d220 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3790007fd0 tx=0x7f3790009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.887+0000 7fbda1400700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbd8406ace0 0x7fbd8406d1a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbda4071cc0 tx=0x7fbd90009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.888+0000 7f379e800700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.113:6808/43940,v1:172.21.15.113:6809/43940] conn(0x7f3780072f80 0x7f37800753a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.888+0000 7fa91de00700 1 --2- 172.21.15.89:0/3807770588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa904005210 0x7fa918036790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.888+0000 7fa91de00700 1 -- 172.21.15.89:0/3807770588 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa918037330 con 0x7fa918050860 2024-04-02T07:35:59.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.888+0000 7fa91e800700 1 --2- 172.21.15.89:0/3807770588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa904005210 0x7fa918036790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:35:59.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.889+0000 7fa91de00700 1 --2- 172.21.15.89:0/3807770588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa918050860 0x7fa9180e2510 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa91000f8b0 tx=0x7fa91000fbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.889+0000 7fa916a00700 1 -- 172.21.15.89:0/3807770588 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa910013850 con 0x7fa918050860 2024-04-02T07:35:59.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.889+0000 7fa916a00700 1 -- 172.21.15.89:0/3807770588 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91000c040 con 0x7fa918050860 2024-04-02T07:35:59.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.889+0000 7fa916a00700 1 -- 172.21.15.89:0/3807770588 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa910013bf0 con 0x7fa918050860 2024-04-02T07:35:59.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.889+0000 7fa924a00700 1 -- 172.21.15.89:0/3807770588 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa918037610 con 0x7fa918050860 2024-04-02T07:35:59.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.889+0000 7fa924a00700 1 -- 172.21.15.89:0/3807770588 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91804de00 con 0x7fa918050860 2024-04-02T07:35:59.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.888+0000 7f0447e00700 1 --2- 172.21.15.89:0/716504323 >> [v2:172.21.15.113:6800/43939,v1:172.21.15.113:6801/43939] conn(0x7f042c073be0 0x7f042c076080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.889+0000 7f0446a00700 1 -- 172.21.15.89:0/716504323 <== osd.5 v2:172.21.15.113:6800/43939 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f042c076750 con 0x7f042c073be0 2024-04-02T07:35:59.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.891+0000 7f2f32a00700 1 -- 172.21.15.89:0/3603225041 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2f40162e20 con 0x7f2f400ce500 2024-04-02T07:35:59.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.891+0000 7fa924a00700 1 -- 172.21.15.89:0/3807770588 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa918056010 con 0x7fa918050860 2024-04-02T07:35:59.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.892+0000 7f3796000700 1 -- 172.21.15.89:0/2884055378 <== osd.4 v2:172.21.15.113:6808/43940 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f3780075a50 con 0x7f3780072f80 2024-04-02T07:35:59.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.892+0000 7f2f3d600700 1 -- 172.21.15.89:0/3603225041 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2f3801e440 con 0x7f2f400ce500 2024-04-02T07:35:59.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.892+0000 7fa916a00700 1 -- 172.21.15.89:0/3807770588 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa910013d50 con 0x7fa918050860 2024-04-02T07:35:59.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.893+0000 7fa916a00700 1 --2- 172.21.15.89:0/3807770588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa90006ac90 0x7fa90006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.893+0000 7fa916a00700 1 -- 172.21.15.89:0/3807770588 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa910089ad0 con 0x7fa918050860 2024-04-02T07:35:59.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.893+0000 7fa916a00700 1 --2- 172.21.15.89:0/3807770588 >> [v2:172.21.15.113:6816/43941,v1:172.21.15.113:6817/43941] conn(0x7fa900072e30 0x7fa900075250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.893+0000 7fa916a00700 1 -- 172.21.15.89:0/3807770588 --> [v2:172.21.15.113:6816/43941,v1:172.21.15.113:6817/43941] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa900075900 con 0x7fa900072e30 2024-04-02T07:35:59.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.893+0000 7fa916a00700 1 -- 172.21.15.89:0/3807770588 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa910089f20 con 0x7fa918050860 2024-04-02T07:35:59.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.893+0000 7f2f3d600700 1 --2- 172.21.15.89:0/3603225041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f2006ac90 0x7f2f2006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.894+0000 7fa91e800700 1 --2- 172.21.15.89:0/3807770588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa90006ac90 0x7fa90006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.894+0000 7fa91f200700 1 --2- 172.21.15.89:0/3807770588 >> [v2:172.21.15.113:6816/43941,v1:172.21.15.113:6817/43941] conn(0x7fa900072e30 0x7fa900075250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.894+0000 7f2f3d600700 1 -- 172.21.15.89:0/3603225041 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f38089370 con 0x7f2f400ce500 2024-04-02T07:35:59.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.894+0000 7f2f3d600700 1 --2- 172.21.15.89:0/3603225041 >> [v2:172.21.15.113:6824/43958,v1:172.21.15.113:6825/43958] conn(0x7f2f20072eb0 0x7f2f200752d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:35:59.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.894+0000 7f2f3d600700 1 -- 172.21.15.89:0/3603225041 --> [v2:172.21.15.113:6824/43958,v1:172.21.15.113:6825/43958] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f20075980 con 0x7f2f20072eb0 2024-04-02T07:35:59.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.894+0000 7f2f3d600700 1 -- 172.21.15.89:0/3603225041 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f3808b990 con 0x7f2f400ce500 2024-04-02T07:35:59.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.895+0000 7fa91f200700 1 --2- 172.21.15.89:0/3807770588 >> [v2:172.21.15.113:6816/43941,v1:172.21.15.113:6817/43941] conn(0x7fa900072e30 0x7fa900075250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.895+0000 7f2f45400700 1 --2- 172.21.15.89:0/3603225041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f2006ac90 0x7f2f2006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.895+0000 7fa91e800700 1 --2- 172.21.15.89:0/3807770588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa90006ac90 0x7fa90006d150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa90c006ba0 tx=0x7fa90c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.896+0000 7f2f44a00700 1 --2- 172.21.15.89:0/3603225041 >> [v2:172.21.15.113:6824/43958,v1:172.21.15.113:6825/43958] conn(0x7f2f20072eb0 0x7f2f200752d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:35:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.896+0000 7fa916a00700 1 -- 172.21.15.89:0/3807770588 <== osd.7 v2:172.21.15.113:6816/43941 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa900075900 con 0x7fa900072e30 2024-04-02T07:35:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.896+0000 7f2f45400700 1 --2- 172.21.15.89:0/3603225041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f2006ac90 0x7f2f2006d150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2f34009b90 tx=0x7f2f34008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.896+0000 7f2f44a00700 1 --2- 172.21.15.89:0/3603225041 >> [v2:172.21.15.113:6824/43958,v1:172.21.15.113:6825/43958] conn(0x7f2f20072eb0 0x7f2f200752d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:35:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.898+0000 7f2f3d600700 1 -- 172.21.15.89:0/3603225041 <== osd.6 v2:172.21.15.113:6824/43958 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2f20075980 con 0x7f2f20072eb0 2024-04-02T07:35:59.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.910+0000 7f127fe00700 1 -- 172.21.15.89:0/542272977 --> [v2:172.21.15.89:6808/43703,v1:172.21.15.89:6809/43703] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1280157320 con 0x7f1260072f80 2024-04-02T07:35:59.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.911+0000 7f1276a00700 1 -- 172.21.15.89:0/542272977 <== osd.1 v2:172.21.15.89:6808/43703 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1280157320 con 0x7f1260072f80 2024-04-02T07:35:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.913+0000 7f127fe00700 1 -- 172.21.15.89:0/542272977 >> [v2:172.21.15.89:6808/43703,v1:172.21.15.89:6809/43703] conn(0x7f1260072f80 msgr2=0x7f12600753a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.913+0000 7f127fe00700 1 --2- 172.21.15.89:0/542272977 >> [v2:172.21.15.89:6808/43703,v1:172.21.15.89:6809/43703] conn(0x7f1260072f80 0x7f12600753a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.913+0000 7f127fe00700 1 -- 172.21.15.89:0/542272977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f126006ad60 msgr2=0x7f126006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.913+0000 7f127fe00700 1 --2- 172.21.15.89:0/542272977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f126006ad60 0x7f126006d220 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f126c000c00 tx=0x7f126c007a10 comp rx=0 tx=0).stop 2024-04-02T07:35:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.913+0000 7f127fe00700 1 -- 172.21.15.89:0/542272977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12800cdb60 msgr2=0x7f1280148e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.913+0000 7f127fe00700 1 --2- 172.21.15.89:0/542272977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12800cdb60 0x7f1280148e80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1270015000 tx=0x7f1270004750 comp rx=0 tx=0).stop 2024-04-02T07:35:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.913+0000 7f127f400700 1 -- 172.21.15.89:0/542272977 reap_dead start 2024-04-02T07:35:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.915+0000 7fbda8e00700 1 -- 172.21.15.89:0/2974408706 --> [v2:172.21.15.89:6824/43713,v1:172.21.15.89:6825/43713] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbd7c002d40 con 0x7fbd84072f00 2024-04-02T07:35:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.916+0000 7fbd9a000700 1 -- 172.21.15.89:0/2974408706 <== osd.2 v2:172.21.15.89:6824/43713 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbd7c002d40 con 0x7fbd84072f00 2024-04-02T07:35:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.913+0000 7f127fe00700 1 -- 172.21.15.89:0/542272977 shutdown_connections 2024-04-02T07:35:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.913+0000 7f127fe00700 1 -- 172.21.15.89:0/542272977 >> 172.21.15.89:0/542272977 conn(0x7f12800b2e60 msgr2=0x7f12800cfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.914+0000 7f127fe00700 1 -- 172.21.15.89:0/542272977 shutdown_connections 2024-04-02T07:35:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.914+0000 7f127fe00700 1 -- 172.21.15.89:0/542272977 wait complete. 2024-04-02T07:35:59.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.916+0000 7fbda8e00700 1 -- 172.21.15.89:0/2974408706 >> [v2:172.21.15.89:6824/43713,v1:172.21.15.89:6825/43713] conn(0x7fbd84072f00 msgr2=0x7fbd84075320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.916+0000 7fbda8e00700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.89:6824/43713,v1:172.21.15.89:6825/43713] conn(0x7fbd84072f00 0x7fbd84075320 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.917+0000 7fbda8e00700 1 -- 172.21.15.89:0/2974408706 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbd8406ace0 msgr2=0x7fbd8406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.917+0000 7fbda8e00700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbd8406ace0 0x7fbd8406d1a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbda4071cc0 tx=0x7fbd90009190 comp rx=0 tx=0).stop 2024-04-02T07:35:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.917+0000 7fbda8e00700 1 -- 172.21.15.89:0/2974408706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbda406e100 msgr2=0x7fbda4147380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.917+0000 7fbda8e00700 1 --2- 172.21.15.89:0/2974408706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbda406e100 0x7fbda4147380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009130 tx=0x7fbd94004750 comp rx=0 tx=0).stop 2024-04-02T07:35:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.917+0000 7fbda2800700 1 -- 172.21.15.89:0/2974408706 reap_dead start 2024-04-02T07:35:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.917+0000 7fbda8e00700 1 -- 172.21.15.89:0/2974408706 shutdown_connections 2024-04-02T07:35:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.917+0000 7fbda8e00700 1 -- 172.21.15.89:0/2974408706 >> 172.21.15.89:0/2974408706 conn(0x7fbda4070f00 msgr2=0x7fbda40b23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.917+0000 7fbda8e00700 1 -- 172.21.15.89:0/2974408706 shutdown_connections 2024-04-02T07:35:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.918+0000 7fbda8e00700 1 -- 172.21.15.89:0/2974408706 wait complete. 2024-04-02T07:35:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.923+0000 7f4c92000700 1 -- 172.21.15.89:0/4219615741 --> [v2:172.21.15.89:6816/43716,v1:172.21.15.89:6817/43716] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4c8c06dd20 con 0x7f4c6c072f80 2024-04-02T07:35:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.923+0000 7f8155000700 1 -- 172.21.15.89:0/2085192572 --> [v2:172.21.15.89:6800/43701,v1:172.21.15.89:6801/43701] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f81500d2140 con 0x7f812c072f00 2024-04-02T07:35:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.923+0000 7fa924a00700 1 -- 172.21.15.89:0/3807770588 --> [v2:172.21.15.113:6816/43941,v1:172.21.15.113:6817/43941] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa918005830 con 0x7fa900072e30 2024-04-02T07:35:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.923+0000 7f4c83400700 1 -- 172.21.15.89:0/4219615741 <== osd.3 v2:172.21.15.89:6816/43716 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4c84007dc0 con 0x7f4c6c072f80 2024-04-02T07:35:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.924+0000 7f8147e00700 1 -- 172.21.15.89:0/2085192572 <== osd.0 v2:172.21.15.89:6800/43701 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8148006c60 con 0x7f812c072f00 2024-04-02T07:35:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.924+0000 7fa916a00700 1 -- 172.21.15.89:0/3807770588 <== osd.7 v2:172.21.15.113:6816/43941 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa918005830 con 0x7fa900072e30 2024-04-02T07:35:59.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.924+0000 7f4c92000700 1 -- 172.21.15.89:0/4219615741 >> [v2:172.21.15.89:6816/43716,v1:172.21.15.89:6817/43716] conn(0x7f4c6c072f80 msgr2=0x7f4c6c0753a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.924+0000 7f4c92000700 1 --2- 172.21.15.89:0/4219615741 >> [v2:172.21.15.89:6816/43716,v1:172.21.15.89:6817/43716] conn(0x7f4c6c072f80 0x7f4c6c0753a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7f8155000700 1 -- 172.21.15.89:0/2085192572 >> [v2:172.21.15.89:6800/43701,v1:172.21.15.89:6801/43701] conn(0x7f812c072f00 msgr2=0x7f812c075320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7f8155000700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.89:6800/43701,v1:172.21.15.89:6801/43701] conn(0x7f812c072f00 0x7f812c075320 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7f8155000700 1 -- 172.21.15.89:0/2085192572 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f812c06ace0 msgr2=0x7f812c06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7f8155000700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f812c06ace0 0x7f812c06d1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8140009fd0 tx=0x7f8140009630 comp rx=0 tx=0).stop 2024-04-02T07:35:59.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7f8155000700 1 -- 172.21.15.89:0/2085192572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8150160df0 msgr2=0x7f81500b8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7f8155000700 1 --2- 172.21.15.89:0/2085192572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8150160df0 0x7f81500b8900 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f813800c750 tx=0x7f813800ca60 comp rx=0 tx=0).stop 2024-04-02T07:35:59.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7f4c92000700 1 -- 172.21.15.89:0/4219615741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4c6c06ad60 msgr2=0x7f4c6c06d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7f4c92000700 1 --2- 172.21.15.89:0/4219615741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4c6c06ad60 0x7f4c6c06d220 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c068120 tx=0x7f4c78009210 comp rx=0 tx=0).stop 2024-04-02T07:35:59.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7f4c92000700 1 -- 172.21.15.89:0/4219615741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c8c0724c0 msgr2=0x7f4c8c094f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7f4c92000700 1 --2- 172.21.15.89:0/4219615741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c8c0724c0 0x7f4c8c094f80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c006fd0 tx=0x7f4c7c00b730 comp rx=0 tx=0).stop 2024-04-02T07:35:59.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7f4c8be00700 1 -- 172.21.15.89:0/4219615741 reap_dead start 2024-04-02T07:35:59.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7fa924a00700 1 -- 172.21.15.89:0/3807770588 >> [v2:172.21.15.113:6816/43941,v1:172.21.15.113:6817/43941] conn(0x7fa900072e30 msgr2=0x7fa900075250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7fa924a00700 1 --2- 172.21.15.89:0/3807770588 >> [v2:172.21.15.113:6816/43941,v1:172.21.15.113:6817/43941] conn(0x7fa900072e30 0x7fa900075250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7fa924a00700 1 -- 172.21.15.89:0/3807770588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa90006ac90 msgr2=0x7fa90006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7fa924a00700 1 --2- 172.21.15.89:0/3807770588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa90006ac90 0x7fa90006d150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa90c006ba0 tx=0x7fa90c008040 comp rx=0 tx=0).stop 2024-04-02T07:35:59.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7fa924a00700 1 -- 172.21.15.89:0/3807770588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa918050860 msgr2=0x7fa9180e2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7fa924a00700 1 --2- 172.21.15.89:0/3807770588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa918050860 0x7fa9180e2510 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa91000f8b0 tx=0x7fa91000fbc0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7fa91f200700 1 -- 172.21.15.89:0/3807770588 reap_dead start 2024-04-02T07:35:59.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7f4c92000700 1 -- 172.21.15.89:0/4219615741 shutdown_connections 2024-04-02T07:35:59.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.925+0000 7f4c92000700 1 -- 172.21.15.89:0/4219615741 >> 172.21.15.89:0/4219615741 conn(0x7f4c8c070f00 msgr2=0x7f4c8c0ad640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.926+0000 7fa924a00700 1 -- 172.21.15.89:0/3807770588 shutdown_connections 2024-04-02T07:35:59.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.926+0000 7fa924a00700 1 -- 172.21.15.89:0/3807770588 >> 172.21.15.89:0/3807770588 conn(0x7fa918056d30 msgr2=0x7fa9180dd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.926+0000 7f4c92000700 1 -- 172.21.15.89:0/4219615741 shutdown_connections 2024-04-02T07:35:59.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.926+0000 7f4c92000700 1 -- 172.21.15.89:0/4219615741 wait complete. 2024-04-02T07:35:59.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.926+0000 7fa924a00700 1 -- 172.21.15.89:0/3807770588 shutdown_connections 2024-04-02T07:35:59.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.926+0000 7f814fe00700 1 -- 172.21.15.89:0/2085192572 reap_dead start 2024-04-02T07:35:59.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.926+0000 7f8155000700 1 -- 172.21.15.89:0/2085192572 shutdown_connections 2024-04-02T07:35:59.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.926+0000 7f8155000700 1 -- 172.21.15.89:0/2085192572 >> 172.21.15.89:0/2085192572 conn(0x7f815009c720 msgr2=0x7f81500cddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.926+0000 7fa924a00700 1 -- 172.21.15.89:0/3807770588 wait complete. 2024-04-02T07:35:59.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.927+0000 7f8155000700 1 -- 172.21.15.89:0/2085192572 shutdown_connections 2024-04-02T07:35:59.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.927+0000 7f8155000700 1 -- 172.21.15.89:0/2085192572 wait complete. 2024-04-02T07:35:59.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.928+0000 7f37a4c00700 1 -- 172.21.15.89:0/2884055378 --> [v2:172.21.15.113:6808/43940,v1:172.21.15.113:6809/43940] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3778002d40 con 0x7f3780072f80 2024-04-02T07:35:59.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.929+0000 7f3796000700 1 -- 172.21.15.89:0/2884055378 <== osd.4 v2:172.21.15.113:6808/43940 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3778002d40 con 0x7f3780072f80 2024-04-02T07:35:59.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.930+0000 7f37a4c00700 1 -- 172.21.15.89:0/2884055378 >> [v2:172.21.15.113:6808/43940,v1:172.21.15.113:6809/43940] conn(0x7f3780072f80 msgr2=0x7f37800753a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.930+0000 7f043fe00700 1 -- 172.21.15.89:0/716504323 --> [v2:172.21.15.113:6800/43939,v1:172.21.15.113:6801/43939] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0420002d40 con 0x7f042c073be0 2024-04-02T07:35:59.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.930+0000 7f37a4c00700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.113:6808/43940,v1:172.21.15.113:6809/43940] conn(0x7f3780072f80 0x7f37800753a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.930+0000 7f37a4c00700 1 -- 172.21.15.89:0/2884055378 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f378006ad60 msgr2=0x7f378006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.930+0000 7f37a4c00700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f378006ad60 0x7f378006d220 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3790007fd0 tx=0x7f3790009040 comp rx=0 tx=0).stop 2024-04-02T07:35:59.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.930+0000 7f37a4c00700 1 -- 172.21.15.89:0/2884055378 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f37a006e810 msgr2=0x7f37a01f28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.930+0000 7f37a4c00700 1 --2- 172.21.15.89:0/2884055378 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f37a006e810 0x7f37a01f28c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f378c00b4a0 tx=0x7f378c00b860 comp rx=0 tx=0).stop 2024-04-02T07:35:59.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.931+0000 7f379e800700 1 -- 172.21.15.89:0/2884055378 reap_dead start 2024-04-02T07:35:59.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.931+0000 7f37a4c00700 1 -- 172.21.15.89:0/2884055378 shutdown_connections 2024-04-02T07:35:59.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.930+0000 7f0446a00700 1 -- 172.21.15.89:0/716504323 <== osd.5 v2:172.21.15.113:6800/43939 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0420002d40 con 0x7f042c073be0 2024-04-02T07:35:59.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.931+0000 7f37a4c00700 1 -- 172.21.15.89:0/2884055378 >> 172.21.15.89:0/2884055378 conn(0x7f37a0070f00 msgr2=0x7f37a014de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.931+0000 7f37a4c00700 1 -- 172.21.15.89:0/2884055378 shutdown_connections 2024-04-02T07:35:59.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.931+0000 7f37a4c00700 1 -- 172.21.15.89:0/2884055378 wait complete. 2024-04-02T07:35:59.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.931+0000 7f043fe00700 1 -- 172.21.15.89:0/716504323 >> [v2:172.21.15.113:6800/43939,v1:172.21.15.113:6801/43939] conn(0x7f042c073be0 msgr2=0x7f042c076080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.932+0000 7f043fe00700 1 --2- 172.21.15.89:0/716504323 >> [v2:172.21.15.113:6800/43939,v1:172.21.15.113:6801/43939] conn(0x7f042c073be0 0x7f042c076080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.932+0000 7f043fe00700 1 -- 172.21.15.89:0/716504323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f042c06ace0 msgr2=0x7f042c06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.932+0000 7f043fe00700 1 --2- 172.21.15.89:0/716504323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f042c06ace0 0x7f042c06d1a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f044000f3c0 tx=0x7f0440004380 comp rx=0 tx=0).stop 2024-04-02T07:35:59.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.932+0000 7f043fe00700 1 -- 172.21.15.89:0/716504323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04501657c0 msgr2=0x7f0450156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.932+0000 7f043fe00700 1 --2- 172.21.15.89:0/716504323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04501657c0 0x7f0450156ea0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f044800b3f0 tx=0x7f044800b700 comp rx=0 tx=0).stop 2024-04-02T07:35:59.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.932+0000 7f044f200700 1 -- 172.21.15.89:0/716504323 reap_dead start 2024-04-02T07:35:59.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.932+0000 7f043fe00700 1 -- 172.21.15.89:0/716504323 shutdown_connections 2024-04-02T07:35:59.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.932+0000 7f043fe00700 1 -- 172.21.15.89:0/716504323 >> 172.21.15.89:0/716504323 conn(0x7f045008aff0 msgr2=0x7f04500b63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.933+0000 7f043fe00700 1 -- 172.21.15.89:0/716504323 shutdown_connections 2024-04-02T07:35:59.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.933+0000 7f043fe00700 1 -- 172.21.15.89:0/716504323 wait complete. 2024-04-02T07:35:59.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.933+0000 7f2f32a00700 1 -- 172.21.15.89:0/3603225041 --> [v2:172.21.15.113:6824/43958,v1:172.21.15.113:6825/43958] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2f4015aa60 con 0x7f2f20072eb0 2024-04-02T07:35:59.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.935+0000 7f2f3d600700 1 -- 172.21.15.89:0/3603225041 <== osd.6 v2:172.21.15.113:6824/43958 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2f4015aa60 con 0x7f2f20072eb0 2024-04-02T07:35:59.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.937+0000 7f2f32a00700 1 -- 172.21.15.89:0/3603225041 >> [v2:172.21.15.113:6824/43958,v1:172.21.15.113:6825/43958] conn(0x7f2f20072eb0 msgr2=0x7f2f200752d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.937+0000 7f2f32a00700 1 --2- 172.21.15.89:0/3603225041 >> [v2:172.21.15.113:6824/43958,v1:172.21.15.113:6825/43958] conn(0x7f2f20072eb0 0x7f2f200752d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.937+0000 7f2f32a00700 1 -- 172.21.15.89:0/3603225041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f2006ac90 msgr2=0x7f2f2006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.938+0000 7f2f32a00700 1 --2- 172.21.15.89:0/3603225041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f2006ac90 0x7f2f2006d150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2f34009b90 tx=0x7f2f34008030 comp rx=0 tx=0).stop 2024-04-02T07:35:59.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.938+0000 7f2f32a00700 1 -- 172.21.15.89:0/3603225041 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f400ce500 msgr2=0x7f2f401506d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:35:59.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.938+0000 7f2f32a00700 1 --2- 172.21.15.89:0/3603225041 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f400ce500 0x7f2f401506d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800be00 tx=0x7f2f3800bee0 comp rx=0 tx=0).stop 2024-04-02T07:35:59.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.938+0000 7f2f45e00700 1 -- 172.21.15.89:0/3603225041 reap_dead start 2024-04-02T07:35:59.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.938+0000 7f2f32a00700 1 -- 172.21.15.89:0/3603225041 shutdown_connections 2024-04-02T07:35:59.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.938+0000 7f2f32a00700 1 -- 172.21.15.89:0/3603225041 >> 172.21.15.89:0/3603225041 conn(0x7f2f400b0d70 msgr2=0x7f2f400ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:35:59.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.939+0000 7f2f32a00700 1 -- 172.21.15.89:0/3603225041 shutdown_connections 2024-04-02T07:35:59.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:35:59.939+0000 7f2f32a00700 1 -- 172.21.15.89:0/3603225041 wait complete. 2024-04-02T07:35:59.960 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-04-02T07:35:59.961 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T07:35:59.962 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2024-04-02T07:35:59.962 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T07:35:59.963 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2024-04-02T07:35:59.963 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-02T07:35:59.964 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-04-02T07:35:59.965 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-02T07:35:59.966 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2024-04-02T07:35:59.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T07:35:59.972 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-04-02T07:35:59.973 INFO:teuthology.orchestra.run.smithi089.stdout:55834574856 2024-04-02T07:35:59.973 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-02T07:35:59.974 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-04-02T07:35:59.974 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-02T07:35:59.975 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-02T07:36:00.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.314+0000 7f1f34c00700 1 Processor -- start 2024-04-02T07:36:00.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.316+0000 7f2ddce00700 1 Processor -- start 2024-04-02T07:36:00.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.316+0000 7f2ddce00700 1 -- start start 2024-04-02T07:36:00.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.317+0000 7f1f34c00700 1 -- start start 2024-04-02T07:36:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.318+0000 7f2ddce00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd815d710 0x7f2dd80d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.318+0000 7f2ddce00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2dd80d1b70 0x7f2dd80d1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.318+0000 7f2ddce00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80cf640 con 0x7f2dd80cff10 2024-04-02T07:36:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.318+0000 7f2ddce00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd80cf220 con 0x7f2dd80d1b70 2024-04-02T07:36:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.318+0000 7f2ddce00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd80d2530 con 0x7f2dd815d710 2024-04-02T07:36:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.319+0000 7f2dd6800700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f2dd80cff10 0x7f2dd80d0380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:39670/0 (socket says 172.21.15.89:39670) 2024-04-02T07:36:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.319+0000 7f2dd6800700 1 -- 172.21.15.89:0/1646865906 learned_addr learned my addr 172.21.15.89:0/1646865906 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.319+0000 7f2dd4a00700 1 -- 172.21.15.89:0/1646865906 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635524688 0 0) 0x7f2dd80cf640 con 0x7f2dd80cff10 2024-04-02T07:36:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.319+0000 7f2dd5400700 1 --2- 172.21.15.89:0/1646865906 >> v2:172.21.15.89:3300/0 conn(0x7f2dd80d1b70 0x7f2dd80d1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.319+0000 7f2dd4a00700 1 -- 172.21.15.89:0/1646865906 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dbc0035b0 con 0x7f2dd80cff10 2024-04-02T07:36:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd5e00700 1 --2- 172.21.15.89:0/1646865906 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd815d710 0x7f2dd80d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.319+0000 7f1f34c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1f300d21f0 0x7f1f300ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.319+0000 7f1f34c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f300ce270 con 0x7f1f300cec40 2024-04-02T07:36:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.319+0000 7f1f34c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3015b600 con 0x7f1f301611b0 2024-04-02T07:36:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.319+0000 7f1f34c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f300d2760 con 0x7f1f300d21f0 2024-04-02T07:36:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd4a00700 1 -- 172.21.15.89:0/1646865906 <== mon.1 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1926510760 0 0) 0x7f2dbc0035b0 con 0x7f2dd80cff10 2024-04-02T07:36:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd4a00700 1 -- 172.21.15.89:0/1646865906 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd815d710 msgr2=0x7f2dd80d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd4a00700 1 --2- 172.21.15.89:0/1646865906 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd815d710 0x7f2dd80d12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd4a00700 1 -- 172.21.15.89:0/1646865906 >> v2:172.21.15.89:3300/0 conn(0x7f2dd80d1b70 msgr2=0x7f2dd80d1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd4a00700 1 --2- 172.21.15.89:0/1646865906 >> v2:172.21.15.89:3300/0 conn(0x7f2dd80d1b70 0x7f2dd80d1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd4a00700 1 -- 172.21.15.89:0/1646865906 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd80ce5a0 con 0x7f2dd80cff10 2024-04-02T07:36:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd4a00700 1 -- 172.21.15.89:0/1646865906 <== mon.1 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f2dd00019f0 con 0x7f2dd80cff10 2024-04-02T07:36:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd4a00700 1 -- 172.21.15.89:0/1646865906 >> v1:172.21.15.113:6789/0 conn(0x7f2dd80cff10 legacy=0x7f2dd80d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd4a00700 1 --2- 172.21.15.89:0/1646865906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dbc003a00 0x7f2dbc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd4a00700 1 --2- 172.21.15.89:0/1646865906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dbc004850 0x7f2dbc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd4a00700 1 --2- 172.21.15.89:0/1646865906 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd815d710 0x7f2dbc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd4a00700 1 -- 172.21.15.89:0/1646865906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd80cf220 con 0x7f2dbc004850 2024-04-02T07:36:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd4a00700 1 -- 172.21.15.89:0/1646865906 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd80d2530 con 0x7f2dbc003a00 2024-04-02T07:36:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f2dd4a00700 1 -- 172.21.15.89:0/1646865906 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc014220 con 0x7f2dd815d710 2024-04-02T07:36:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.321+0000 7f2dd5e00700 1 --2- 172.21.15.89:0/1646865906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dbc003a00 0x7f2dbc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.321+0000 7f2dd5400700 1 --2- 172.21.15.89:0/1646865906 >> v2:172.21.15.89:3300/0 conn(0x7f2dd80d1b70 0x7f2dd80d1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.321+0000 7f2dd6800700 1 --2- 172.21.15.89:0/1646865906 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd815d710 0x7f2dbc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f1f2e800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1f300cec40 0x7f1f30161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34822/0 (socket says 172.21.15.89:34822) 2024-04-02T07:36:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f1f2e800700 1 -- 172.21.15.89:0/3164571729 learned_addr learned my addr 172.21.15.89:0/3164571729 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f1f2d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1f300d21f0 0x7f1f300ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.320+0000 7f1f2de00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f1f301611b0 0x7f1f3015ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:39686/0 (socket says 172.21.15.89:39686) 2024-04-02T07:36:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.322+0000 7f2dd6800700 1 -- 172.21.15.89:0/1646865906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dbc003a00 msgr2=0x7f2dbc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.322+0000 7f2dd6800700 1 --2- 172.21.15.89:0/1646865906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dbc003a00 0x7f2dbc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.322+0000 7f2dd6800700 1 -- 172.21.15.89:0/1646865906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dbc004850 msgr2=0x7f2dbc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.322+0000 7f2dd5400700 1 --2- 172.21.15.89:0/1646865906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dbc004850 0x7f2dbc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.322+0000 7f2dd6800700 1 --2- 172.21.15.89:0/1646865906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dbc004850 0x7f2dbc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.322+0000 7f1f2ca00700 1 -- 172.21.15.89:0/3164571729 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239268075 0 0) 0x7f1f300ce270 con 0x7f1f300cec40 2024-04-02T07:36:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.322+0000 7f1f2ca00700 1 -- 172.21.15.89:0/3164571729 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f140035b0 con 0x7f1f300cec40 2024-04-02T07:36:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.322+0000 7f2dd6800700 1 -- 172.21.15.89:0/1646865906 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd80ce5a0 con 0x7f2dd815d710 2024-04-02T07:36:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.322+0000 7f1f2d400700 1 -- 172.21.15.89:0/3164571729 >> v1:172.21.15.113:6789/0 conn(0x7f1f301611b0 legacy=0x7f1f3015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.322+0000 7f1f2d400700 1 -- 172.21.15.89:0/3164571729 >> v1:172.21.15.89:6789/0 conn(0x7f1f300cec40 legacy=0x7f1f30161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.323+0000 7f2dd5400700 1 --2- 172.21.15.89:0/1646865906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dbc004850 0x7f2dbc004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.323+0000 7f1f2d400700 1 -- 172.21.15.89:0/3164571729 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f300d2a40 con 0x7f1f300d21f0 2024-04-02T07:36:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.323+0000 7f1f2d400700 1 --2- 172.21.15.89:0/3164571729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f300d21f0 0x7f1f300ce700 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c009840 tx=0x7f1f1c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343f03e8d938153e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.324+0000 7f1f2ca00700 1 -- 172.21.15.89:0/3164571729 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c011830 con 0x7f1f300d21f0 2024-04-02T07:36:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.324+0000 7f1f2ca00700 1 -- 172.21.15.89:0/3164571729 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c011990 con 0x7f1f300d21f0 2024-04-02T07:36:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.324+0000 7fca46600700 1 Processor -- start 2024-04-02T07:36:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.325+0000 7fca46600700 1 -- start start 2024-04-02T07:36:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.323+0000 7fbf02000700 1 Processor -- start 2024-04-02T07:36:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.324+0000 7fbf02000700 1 -- start start 2024-04-02T07:36:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.325+0000 7f1f34c00700 1 -- 172.21.15.89:0/3164571729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f300d21f0 msgr2=0x7f1f300ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.325+0000 7f2dd6800700 1 --2- 172.21.15.89:0/1646865906 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd815d710 0x7f2dbc005210 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000a900 tx=0x7f2dbc005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62112982a135740f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.325+0000 7f1f34c00700 1 --2- 172.21.15.89:0/3164571729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f300d21f0 0x7f1f300ce700 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c009840 tx=0x7f1f1c009b50 comp rx=0 tx=0).stop 2024-04-02T07:36:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.325+0000 7ffb5fe00700 1 Processor -- start 2024-04-02T07:36:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.325+0000 7ffb5fe00700 1 -- start start 2024-04-02T07:36:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.326+0000 7f2dd4a00700 1 -- 172.21.15.89:0/1646865906 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0012600 con 0x7f2dd815d710 2024-04-02T07:36:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.326+0000 7fbf02000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbef4038140 0x7fbef4006200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.326+0000 7fbf02000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef4056c30 con 0x7fbef4057cd0 2024-04-02T07:36:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.326+0000 7fbf02000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef4056f10 con 0x7fbef4053610 2024-04-02T07:36:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.326+0000 7fbf02000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef40e1fc0 con 0x7fbef4038140 2024-04-02T07:36:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.326+0000 7f1f34c00700 1 -- 172.21.15.89:0/3164571729 shutdown_connections 2024-04-02T07:36:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.326+0000 7f1f34c00700 1 --2- 172.21.15.89:0/3164571729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f300d21f0 0x7f1f300ce700 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.326+0000 7f1f34c00700 1 -- 172.21.15.89:0/3164571729 >> 172.21.15.89:0/3164571729 conn(0x7f1f3008aff0 msgr2=0x7f1f300d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.326+0000 7f1f34c00700 1 -- 172.21.15.89:0/3164571729 shutdown_connections 2024-04-02T07:36:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.326+0000 7f1f34c00700 1 -- 172.21.15.89:0/3164571729 wait complete. 2024-04-02T07:36:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.327+0000 7f43f5600700 1 Processor -- start 2024-04-02T07:36:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.327+0000 7fbefbe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbef4057cd0 0x7fbef40e84a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34824/0 (socket says 172.21.15.89:34824) 2024-04-02T07:36:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.327+0000 7fbefbe00700 1 -- 172.21.15.89:0/1070448509 learned_addr learned my addr 172.21.15.89:0/1070448509 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.327+0000 7f43f5600700 1 -- start start 2024-04-02T07:36:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.327+0000 7fbf01600700 1 --2- 172.21.15.89:0/1070448509 >> v2:172.21.15.113:3300/0 conn(0x7fbef4038140 0x7fbef4006200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.326+0000 7f2dd4a00700 1 -- 172.21.15.89:0/1646865906 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd0012760 con 0x7f2dd815d710 2024-04-02T07:36:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.326+0000 7f2dd4a00700 1 -- 172.21.15.89:0/1646865906 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0012b10 con 0x7f2dd815d710 2024-04-02T07:36:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.327+0000 7f2ddce00700 1 -- 172.21.15.89:0/1646865906 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd815d710 msgr2=0x7f2dbc005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.327+0000 7f2ddce00700 1 --2- 172.21.15.89:0/1646865906 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd815d710 0x7f2dbc005210 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000a900 tx=0x7f2dbc005880 comp rx=0 tx=0).stop 2024-04-02T07:36:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.327+0000 7ffb5fe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ffb5804f7a0 0x7ffb580df410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.327+0000 7ffb5fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb580532f0 0x7ffb580061d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.327+0000 7ffb5fe00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb580543e0 con 0x7ffb5804fb80 2024-04-02T07:36:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.327+0000 7ffb5fe00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb580537d0 con 0x7ffb580532f0 2024-04-02T07:36:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.327+0000 7ffb5fe00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb58053ab0 con 0x7ffb5804f7a0 2024-04-02T07:36:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.327+0000 7f2dd6800700 1 -- 172.21.15.89:0/1646865906 reap_dead start 2024-04-02T07:36:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.328+0000 7f2ddce00700 1 -- 172.21.15.89:0/1646865906 shutdown_connections 2024-04-02T07:36:00.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.328+0000 7ffb5ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ffb5804fb80 0x7ffb5804ff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34836/0 (socket says 172.21.15.89:34836) 2024-04-02T07:36:00.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.328+0000 7ffb5ea00700 1 -- 172.21.15.89:0/1876708998 learned_addr learned my addr 172.21.15.89:0/1876708998 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:00.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.328+0000 7ffb5e000700 1 --2- 172.21.15.89:0/1876708998 >> v2:172.21.15.113:3300/0 conn(0x7ffb5804f7a0 0x7ffb580df410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.328+0000 7f2ddce00700 1 -- 172.21.15.89:0/1646865906 >> 172.21.15.89:0/1646865906 conn(0x7f2dd808aff0 msgr2=0x7f2dd80d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.328+0000 7fbf01600700 1 -- 172.21.15.89:0/1070448509 >> v1:172.21.15.113:6789/0 conn(0x7fbef4053610 legacy=0x7fbef40539f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T07:36:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.328+0000 7fbf01600700 1 -- 172.21.15.89:0/1070448509 >> v1:172.21.15.89:6789/0 conn(0x7fbef4057cd0 legacy=0x7fbef40e84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.328+0000 7fbf01600700 1 -- 172.21.15.89:0/1070448509 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef40571f0 con 0x7fbef4038140 2024-04-02T07:36:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.328+0000 7fbf01600700 1 --2- 172.21.15.89:0/1070448509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef4038140 0x7fbef4006200 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009840 tx=0x7fbee8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e32e5b8f64445f80 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.328+0000 7f2ddce00700 1 -- 172.21.15.89:0/1646865906 shutdown_connections 2024-04-02T07:36:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7f2ddce00700 1 -- 172.21.15.89:0/1646865906 wait complete. 2024-04-02T07:36:00.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7fbefb400700 1 -- 172.21.15.89:0/1070448509 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8004230 con 0x7fbef4038140 2024-04-02T07:36:00.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7fbefb400700 1 -- 172.21.15.89:0/1070448509 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee8004390 con 0x7fbef4038140 2024-04-02T07:36:00.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7fbefb400700 1 -- 172.21.15.89:0/1070448509 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee80036a0 con 0x7fbef4038140 2024-04-02T07:36:00.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.330+0000 7fbf02000700 1 -- 172.21.15.89:0/1070448509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef4038140 msgr2=0x7fbef4006200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.330+0000 7fbf02000700 1 --2- 172.21.15.89:0/1070448509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef4038140 0x7fbef4006200 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009840 tx=0x7fbee8009b50 comp rx=0 tx=0).stop 2024-04-02T07:36:00.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.330+0000 7fbf02000700 1 -- 172.21.15.89:0/1070448509 shutdown_connections 2024-04-02T07:36:00.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.330+0000 7fbf02000700 1 --2- 172.21.15.89:0/1070448509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef4038140 0x7fbef4006200 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.330+0000 7fbf02000700 1 -- 172.21.15.89:0/1070448509 >> 172.21.15.89:0/1070448509 conn(0x7fbef4055a40 msgr2=0x7fbef4055e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.330+0000 7f1f34c00700 1 Processor -- start 2024-04-02T07:36:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.330+0000 7fca46600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca4015d710 0x7fca400d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.330+0000 7fca46600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fca400d1b70 0x7fca400d1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.330+0000 7fca46600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca400cf640 con 0x7fca400cff10 2024-04-02T07:36:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.330+0000 7fca46600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca400cf220 con 0x7fca400d1b70 2024-04-02T07:36:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.330+0000 7fca46600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca400d2530 con 0x7fca4015d710 2024-04-02T07:36:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.331+0000 7fca3ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fca400d1b70 0x7fca400d1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.331+0000 7fca3ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fca400d1b70 0x7fca400d1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52572/0 (socket says 172.21.15.89:52572) 2024-04-02T07:36:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.331+0000 7fca3ea00700 1 -- 172.21.15.89:0/1124698432 learned_addr learned my addr 172.21.15.89:0/1124698432 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.331+0000 7fca3f400700 1 --2- 172.21.15.89:0/1124698432 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca4015d710 0x7fca400d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.331+0000 7fca3ea00700 1 -- 172.21.15.89:0/1124698432 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca4015d710 msgr2=0x7fca400d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7fca3ea00700 1 --2- 172.21.15.89:0/1124698432 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca4015d710 0x7fca400d12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7fca3ea00700 1 -- 172.21.15.89:0/1124698432 >> v1:172.21.15.89:6789/0 conn(0x7fca400cff10 legacy=0x7fca400d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7f2ddce00700 1 Processor -- start 2024-04-02T07:36:00.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7f2ddce00700 1 -- start start 2024-04-02T07:36:00.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7fca3ea00700 1 -- 172.21.15.89:0/1124698432 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca400ce5a0 con 0x7fca400d1b70 2024-04-02T07:36:00.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.328+0000 7ffb5f400700 1 --2- 172.21.15.89:0/1876708998 >> v2:172.21.15.89:3300/0 conn(0x7ffb580532f0 0x7ffb580061d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7ffb5e000700 1 -- 172.21.15.89:0/1876708998 >> v2:172.21.15.89:3300/0 conn(0x7ffb580532f0 msgr2=0x7ffb580061d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:00.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7ffb5e000700 1 --2- 172.21.15.89:0/1876708998 >> v2:172.21.15.89:3300/0 conn(0x7ffb580532f0 0x7ffb580061d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7ffb5e000700 1 -- 172.21.15.89:0/1876708998 >> v1:172.21.15.89:6789/0 conn(0x7ffb5804fb80 legacy=0x7ffb5804ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7ffb5e000700 1 -- 172.21.15.89:0/1876708998 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb58053d90 con 0x7ffb5804f7a0 2024-04-02T07:36:00.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7ffb5f400700 1 --2- 172.21.15.89:0/1876708998 >> v2:172.21.15.89:3300/0 conn(0x7ffb580532f0 0x7ffb580061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7ffb5e000700 1 --2- 172.21.15.89:0/1876708998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5804f7a0 0x7ffb580df410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800d9a0 tx=0x7ffb48009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae962d425094dcff server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.331+0000 7ffb5d600700 1 -- 172.21.15.89:0/1876708998 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb48011830 con 0x7ffb5804f7a0 2024-04-02T07:36:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.331+0000 7ffb5d600700 1 -- 172.21.15.89:0/1876708998 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb48011990 con 0x7ffb5804f7a0 2024-04-02T07:36:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.331+0000 7ffb5d600700 1 -- 172.21.15.89:0/1876708998 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb48011d20 con 0x7ffb5804f7a0 2024-04-02T07:36:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1876708998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5804f7a0 msgr2=0x7ffb580df410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7ffb5fe00700 1 --2- 172.21.15.89:0/1876708998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5804f7a0 0x7ffb580df410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800d9a0 tx=0x7ffb48009b20 comp rx=0 tx=0).stop 2024-04-02T07:36:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1876708998 shutdown_connections 2024-04-02T07:36:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7ffb5fe00700 1 --2- 172.21.15.89:0/1876708998 >> v2:172.21.15.89:3300/0 conn(0x7ffb580532f0 0x7ffb580061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7ffb5fe00700 1 --2- 172.21.15.89:0/1876708998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5804f7a0 0x7ffb580df410 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1876708998 >> 172.21.15.89:0/1876708998 conn(0x7ffb58054b20 msgr2=0x7ffb58054f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7f43f5600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43f015d7a0 0x7f43f00cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7f43f5600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f43f00724c0 0x7f43f00cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7f43f5600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f006df90 con 0x7f43f00cf4f0 2024-04-02T07:36:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7f43f5600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f006eba0 con 0x7f43f00724c0 2024-04-02T07:36:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.329+0000 7f43f5600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f006e270 con 0x7f43f015d7a0 2024-04-02T07:36:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.333+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1876708998 shutdown_connections 2024-04-02T07:36:00.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.333+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1876708998 wait complete. 2024-04-02T07:36:00.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.333+0000 7fca3ea00700 1 --2- 172.21.15.89:0/1124698432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca400d1b70 0x7fca400d1ff0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fca2c006fd0 tx=0x7fca2c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c9daea6f60a1c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.333+0000 7fbf02000700 1 -- 172.21.15.89:0/1070448509 shutdown_connections 2024-04-02T07:36:00.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.331+0000 7f1f34c00700 1 -- start start 2024-04-02T07:36:00.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7f1f34c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f300cec40 0x7f1f300ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7f1f34c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f300d21f0 0x7f1f301f6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7f1f34c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f301611b0 0x7f1f301f6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7f1f34c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f300c7420 con 0x7f1f301611b0 2024-04-02T07:36:00.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7f1f34c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f300cc750 con 0x7f1f300d21f0 2024-04-02T07:36:00.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.332+0000 7f1f34c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f300b8f30 con 0x7f1f300cec40 2024-04-02T07:36:00.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.333+0000 7fbf02000700 1 -- 172.21.15.89:0/1070448509 wait complete. 2024-04-02T07:36:00.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.333+0000 7f43e7e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f43f00724c0 0x7f43f00cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7f1f2d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f300d21f0 0x7f1f301f6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7f1f2d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f300d21f0 0x7f1f301f6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33566/0 (socket says 172.21.15.89:33566) 2024-04-02T07:36:00.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7f1f2d400700 1 -- 172.21.15.89:0/1789013898 learned_addr learned my addr 172.21.15.89:0/1789013898 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:00.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7f43eea00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43f015d7a0 0x7f43f00cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7f43eea00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43f015d7a0 0x7f43f00cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:32796/0 (socket says 172.21.15.89:32796) 2024-04-02T07:36:00.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7f43ef400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f43f00cf4f0 0x7f43f00cd020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34864/0 (socket says 172.21.15.89:34864) 2024-04-02T07:36:00.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7f43ef400700 1 -- 172.21.15.89:0/1685187747 learned_addr learned my addr 172.21.15.89:0/1685187747 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:00.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.333+0000 7fca3e000700 1 -- 172.21.15.89:0/1124698432 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c005e30 con 0x7fca400d1b70 2024-04-02T07:36:00.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7fca3e000700 1 -- 172.21.15.89:0/1124698432 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2c004030 con 0x7fca400d1b70 2024-04-02T07:36:00.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7fca3e000700 1 -- 172.21.15.89:0/1124698432 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c0043e0 con 0x7fca400d1b70 2024-04-02T07:36:00.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7f2ddce00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dd815d710 0x7f2dd80b16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7f2ddce00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd814dad0 0x7f2dd815e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7f1f2de00700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f300cec40 0x7f1f300ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.335+0000 7f1f2e800700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f301611b0 0x7f1f301f6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.335+0000 7f43e7e00700 1 -- 172.21.15.89:0/1685187747 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43f015d7a0 msgr2=0x7f43f00cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:00.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.335+0000 7f43ee000700 1 -- 172.21.15.89:0/1685187747 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929568638 0 0) 0x7f43f006df90 con 0x7f43f00cf4f0 2024-04-02T07:36:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.335+0000 7f43e7e00700 1 --2- 172.21.15.89:0/1685187747 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43f015d7a0 0x7f43f00cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.335+0000 7f43e7e00700 1 -- 172.21.15.89:0/1685187747 >> v1:172.21.15.89:6789/0 conn(0x7f43f00cf4f0 legacy=0x7f43f00cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.335+0000 7f43e7e00700 1 -- 172.21.15.89:0/1685187747 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f006e550 con 0x7f43f00724c0 2024-04-02T07:36:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.336+0000 7fca46600700 1 -- 172.21.15.89:0/1124698432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca400d1b70 msgr2=0x7fca400d1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.335+0000 7ffb5fe00700 1 Processor -- start 2024-04-02T07:36:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.335+0000 7f131a200700 1 Processor -- start 2024-04-02T07:36:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.336+0000 7f131a200700 1 -- start start 2024-04-02T07:36:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.336+0000 7fca46600700 1 --2- 172.21.15.89:0/1124698432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca400d1b70 0x7fca400d1ff0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fca2c006fd0 tx=0x7fca2c009bd0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.336+0000 7f43e7e00700 1 --2- 172.21.15.89:0/1685187747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43f00724c0 0x7f43f00cefb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f43e0009840 tx=0x7f43e0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e89aadea63b90604 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.336+0000 7f43ee000700 1 -- 172.21.15.89:0/1685187747 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0011830 con 0x7f43f00724c0 2024-04-02T07:36:00.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.336+0000 7f43ee000700 1 -- 172.21.15.89:0/1685187747 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e0011990 con 0x7f43f00724c0 2024-04-02T07:36:00.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.336+0000 7f43ee000700 1 -- 172.21.15.89:0/1685187747 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0011d20 con 0x7f43f00724c0 2024-04-02T07:36:00.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7f2ddce00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dd80b3c70 0x7f2dd80b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7f2ddce00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd80b4630 con 0x7f2dd815d710 2024-04-02T07:36:00.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7f2ddce00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd815eef0 con 0x7f2dd80b3c70 2024-04-02T07:36:00.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.334+0000 7f2ddce00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd814df50 con 0x7f2dd814dad0 2024-04-02T07:36:00.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.335+0000 7f2dd6800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dd80b3c70 0x7f2dd80b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.335+0000 7f2dd5400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd814dad0 0x7f2dd815e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.335+0000 7f2dd5400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd814dad0 0x7f2dd815e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:32812/0 (socket says 172.21.15.89:32812) 2024-04-02T07:36:00.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.335+0000 7f2dd5400700 1 -- 172.21.15.89:0/2584468145 learned_addr learned my addr 172.21.15.89:0/2584468145 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:00.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.335+0000 7f2dd5e00700 1 --2- 172.21.15.89:0/2584468145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dd815d710 0x7f2dd80b16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.336+0000 7f2dd5e00700 1 -- 172.21.15.89:0/2584468145 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd814dad0 msgr2=0x7f2dd815e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.336+0000 7f2dd5e00700 1 --2- 172.21.15.89:0/2584468145 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd814dad0 0x7f2dd815e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.336+0000 7f2dd5e00700 1 -- 172.21.15.89:0/2584468145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dd80b3c70 msgr2=0x7f2dd80b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.336+0000 7f2dd5e00700 1 --2- 172.21.15.89:0/2584468145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dd80b3c70 0x7f2dd80b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.336+0000 7f2dd5e00700 1 -- 172.21.15.89:0/2584468145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd80b0630 con 0x7f2dd815d710 2024-04-02T07:36:00.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7fca46600700 1 -- 172.21.15.89:0/1124698432 shutdown_connections 2024-04-02T07:36:00.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7fca46600700 1 --2- 172.21.15.89:0/1124698432 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca4015d710 0x7fca400d12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7fca46600700 1 --2- 172.21.15.89:0/1124698432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca400d1b70 0x7fca400d1ff0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7fca46600700 1 -- 172.21.15.89:0/1124698432 >> 172.21.15.89:0/1124698432 conn(0x7fca4008aff0 msgr2=0x7fca400d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7fbf02000700 1 Processor -- start 2024-04-02T07:36:00.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.336+0000 7ffb5fe00700 1 -- start start 2024-04-02T07:36:00.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7ffb5fe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5804f7a0 0x7ffb580cf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7ffb5fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb5804fb80 0x7ffb5804dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7ffb5fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb580532f0 0x7ffb580d4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7ffb5fe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb58049940 con 0x7ffb580532f0 2024-04-02T07:36:00.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7ffb5fe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5804e1a0 con 0x7ffb5804f7a0 2024-04-02T07:36:00.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7ffb5fe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5804e470 con 0x7ffb5804fb80 2024-04-02T07:36:00.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7f43f5600700 1 -- 172.21.15.89:0/1685187747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43f00724c0 msgr2=0x7f43f00cefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7f43f5600700 1 --2- 172.21.15.89:0/1685187747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43f00724c0 0x7f43f00cefb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f43e0009840 tx=0x7f43e0009b50 comp rx=0 tx=0).stop 2024-04-02T07:36:00.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7f1f2d400700 1 -- 172.21.15.89:0/1789013898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f300cec40 msgr2=0x7f1f300ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7f1f2d400700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f300cec40 0x7f1f300ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7f1f2d400700 1 -- 172.21.15.89:0/1789013898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f301611b0 msgr2=0x7f1f301f6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7f1f2d400700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f301611b0 0x7f1f301f6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7f1f2d400700 1 -- 172.21.15.89:0/1789013898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c0094a0 con 0x7f1f300d21f0 2024-04-02T07:36:00.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7f43f5600700 1 -- 172.21.15.89:0/1685187747 shutdown_connections 2024-04-02T07:36:00.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7f43f5600700 1 --2- 172.21.15.89:0/1685187747 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43f015d7a0 0x7f43f00cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7f43f5600700 1 --2- 172.21.15.89:0/1685187747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43f00724c0 0x7f43f00cefb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7f43f5600700 1 -- 172.21.15.89:0/1685187747 >> 172.21.15.89:0/1685187747 conn(0x7f43f0070f00 msgr2=0x7f43f0071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7f43f5600700 1 -- 172.21.15.89:0/1685187747 shutdown_connections 2024-04-02T07:36:00.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7ffb5e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb5804fb80 0x7ffb5804dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7f43f5600700 1 -- 172.21.15.89:0/1685187747 wait complete. 2024-04-02T07:36:00.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7fca46600700 1 -- 172.21.15.89:0/1124698432 shutdown_connections 2024-04-02T07:36:00.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7ffb5ea00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5804f7a0 0x7ffb580cf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7ffb5ea00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5804f7a0 0x7ffb580cf4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33572/0 (socket says 172.21.15.89:33572) 2024-04-02T07:36:00.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7ffb5ea00700 1 -- 172.21.15.89:0/1645032900 learned_addr learned my addr 172.21.15.89:0/1645032900 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:00.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7f2dd5e00700 1 --2- 172.21.15.89:0/2584468145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dd815d710 0x7f2dd80b16d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8011960 tx=0x7f2dc8011c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.337+0000 7fca46600700 1 -- 172.21.15.89:0/1124698432 wait complete. 2024-04-02T07:36:00.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7f131a200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f13141597f0 0x7f13140b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7f131a200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f131415d4d0 0x7f13140b2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7f131a200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13140b2970 con 0x7f131406dc60 2024-04-02T07:36:00.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7f131a200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13140b2c50 con 0x7f13141597f0 2024-04-02T07:36:00.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7f131a200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13140b2f30 con 0x7f131415d4d0 2024-04-02T07:36:00.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.338+0000 7f1f2e800700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f301611b0 0x7f1f301f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:00.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7f1f2d400700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f300d21f0 0x7f1f301f6430 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1f3008ce10 tx=0x7f1f1c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7ffb5e000700 1 -- 172.21.15.89:0/1645032900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5804f7a0 msgr2=0x7ffb580cf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7ffb5e000700 1 --2- 172.21.15.89:0/1645032900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5804f7a0 0x7ffb580cf4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7ffb5e000700 1 -- 172.21.15.89:0/1645032900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb580532f0 msgr2=0x7ffb580d4550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:36:00.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7ffb5e000700 1 --2- 172.21.15.89:0/1645032900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb580532f0 0x7ffb580d4550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7ffb5e000700 1 -- 172.21.15.89:0/1645032900 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb480094a0 con 0x7ffb5804fb80 2024-04-02T07:36:00.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7f1313400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f13141597f0 0x7f13140b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7f1f26000700 1 -- 172.21.15.89:0/1789013898 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c01aaa0 con 0x7f1f300d21f0 2024-04-02T07:36:00.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7f1313400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f13141597f0 0x7f13140b3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52600/0 (socket says 172.21.15.89:52600) 2024-04-02T07:36:00.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.340+0000 7f1313400700 1 -- 172.21.15.89:0/2025594184 learned_addr learned my addr 172.21.15.89:0/2025594184 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:00.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7fca46600700 1 Processor -- start 2024-04-02T07:36:00.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7f1f26000700 1 -- 172.21.15.89:0/1789013898 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c01ac00 con 0x7f1f300d21f0 2024-04-02T07:36:00.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7f2dce000700 1 -- 172.21.15.89:0/2584468145 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc80136a0 con 0x7f2dd815d710 2024-04-02T07:36:00.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7f2dce000700 1 -- 172.21.15.89:0/2584468145 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc8013800 con 0x7f2dd815d710 2024-04-02T07:36:00.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.339+0000 7f2dce000700 1 -- 172.21.15.89:0/2584468145 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc80136a0 con 0x7f2dd815d710 2024-04-02T07:36:00.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.340+0000 7f2ddce00700 1 -- 172.21.15.89:0/2584468145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd80b0910 con 0x7f2dd815d710 2024-04-02T07:36:00.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.340+0000 7f2ddce00700 1 -- 172.21.15.89:0/2584468145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd80b0ed0 con 0x7f2dd815d710 2024-04-02T07:36:00.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.340+0000 7ffb5e000700 1 --2- 172.21.15.89:0/1645032900 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb5804fb80 0x7ffb5804dc60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffb48016000 tx=0x7ffb48004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.340+0000 7f1f26000700 1 -- 172.21.15.89:0/1789013898 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c024420 con 0x7f1f300d21f0 2024-04-02T07:36:00.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.340+0000 7fca46600700 1 -- start start 2024-04-02T07:36:00.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.340+0000 7f130aa00700 1 --2- 172.21.15.89:0/2025594184 >> v2:172.21.15.113:3300/0 conn(0x7f131415d4d0 0x7f13140b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.340+0000 7f1f34c00700 1 -- 172.21.15.89:0/1789013898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f300b8c50 con 0x7f1f300d21f0 2024-04-02T07:36:00.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.340+0000 7ffb56a00700 1 -- 172.21.15.89:0/1645032900 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb480036a0 con 0x7ffb5804fb80 2024-04-02T07:36:00.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.340+0000 7f1f34c00700 1 -- 172.21.15.89:0/1789013898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f300cc3e0 con 0x7f1f300d21f0 2024-04-02T07:36:00.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.341+0000 7f1312a00700 1 -- 172.21.15.89:0/2025594184 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658139803 0 0) 0x7f13140b2970 con 0x7f131406dc60 2024-04-02T07:36:00.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.341+0000 7ffb56a00700 1 -- 172.21.15.89:0/1645032900 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4801f070 con 0x7ffb5804fb80 2024-04-02T07:36:00.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.341+0000 7ffb56a00700 1 -- 172.21.15.89:0/1645032900 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb48003a40 con 0x7ffb5804fb80 2024-04-02T07:36:00.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.341+0000 7f1312a00700 1 -- 172.21.15.89:0/2025594184 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12f80035b0 con 0x7f131406dc60 2024-04-02T07:36:00.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.341+0000 7fbf02000700 1 -- start start 2024-04-02T07:36:00.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.342+0000 7f43f5600700 1 Processor -- start 2024-04-02T07:36:00.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.341+0000 7f1313400700 1 -- 172.21.15.89:0/2025594184 >> v2:172.21.15.113:3300/0 conn(0x7f131415d4d0 msgr2=0x7f13140b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:00.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.341+0000 7f1313400700 1 --2- 172.21.15.89:0/2025594184 >> v2:172.21.15.113:3300/0 conn(0x7f131415d4d0 0x7f13140b2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.341+0000 7f1313400700 1 -- 172.21.15.89:0/2025594184 >> v1:172.21.15.113:6789/0 conn(0x7f131406dc60 legacy=0x7f131406e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.341+0000 7f1313400700 1 -- 172.21.15.89:0/2025594184 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13140b3210 con 0x7f13141597f0 2024-04-02T07:36:00.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.342+0000 7fca46600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca400cff10 0x7fca40151390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.342+0000 7fca46600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca400d1b70 0x7fca400b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.342+0000 7fca46600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca4015d710 0x7fca4015e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.342+0000 7fca46600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca400c8fd0 con 0x7fca400cff10 2024-04-02T07:36:00.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.342+0000 7f1313400700 1 --2- 172.21.15.89:0/2025594184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13141597f0 0x7f13140b3a10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f13140ccab0 tx=0x7f130400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac872b610a50a34 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.342+0000 7fca46600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca401518d0 con 0x7fca400d1b70 2024-04-02T07:36:00.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.342+0000 7fca46600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca400c83c0 con 0x7fca4015d710 2024-04-02T07:36:00.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.342+0000 7f43f5600700 1 -- start start 2024-04-02T07:36:00.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.342+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1645032900 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb58049bf0 con 0x7ffb5804fb80 2024-04-02T07:36:00.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.342+0000 7fca3fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca4015d710 0x7fca4015e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.342+0000 7fca3fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca4015d710 0x7fca4015e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:32842/0 (socket says 172.21.15.89:32842) 2024-04-02T07:36:00.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.342+0000 7f130aa00700 1 --2- 172.21.15.89:0/2025594184 >> v2:172.21.15.113:3300/0 conn(0x7f131415d4d0 0x7f13140b2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:00.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.342+0000 7fca3fe00700 1 -- 172.21.15.89:0/1705899622 learned_addr learned my addr 172.21.15.89:0/1705899622 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:00.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.343+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1645032900 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5804a1b0 con 0x7ffb5804fb80 2024-04-02T07:36:00.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.343+0000 7fca3ea00700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca400d1b70 0x7fca400b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.343+0000 7fca3f400700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca400cff10 0x7fca40151390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.343+0000 7f2ddce00700 1 -- 172.21.15.89:0/2584468145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd80ce210 con 0x7f2dd815d710 2024-04-02T07:36:00.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.344+0000 7f1f26000700 1 -- 172.21.15.89:0/1789013898 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1f1c022070 con 0x7f1f300d21f0 2024-04-02T07:36:00.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.344+0000 7f43f5600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f00724c0 0x7f43f0140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.344+0000 7fca3ea00700 1 -- 172.21.15.89:0/1705899622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca4015d710 msgr2=0x7fca4015e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.344+0000 7fca3ea00700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca4015d710 0x7fca4015e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.344+0000 7f1312a00700 1 -- 172.21.15.89:0/2025594184 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130400e070 con 0x7f13141597f0 2024-04-02T07:36:00.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.344+0000 7f1312a00700 1 -- 172.21.15.89:0/2025594184 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130400b910 con 0x7f13141597f0 2024-04-02T07:36:00.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.344+0000 7f1312a00700 1 -- 172.21.15.89:0/2025594184 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130400e070 con 0x7f13141597f0 2024-04-02T07:36:00.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.344+0000 7fca3ea00700 1 -- 172.21.15.89:0/1705899622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca400cff10 msgr2=0x7fca40151390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.344+0000 7f43f5600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43f00cf4f0 0x7f43f00989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.344+0000 7fca3ea00700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca400cff10 0x7fca40151390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.344+0000 7fca3ea00700 1 -- 172.21.15.89:0/1705899622 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c009500 con 0x7fca400d1b70 2024-04-02T07:36:00.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.345+0000 7f43f5600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43f015d7a0 0x7f43f01605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.345+0000 7fca3fe00700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca4015d710 0x7fca4015e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:00.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.345+0000 7fca3f400700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca400cff10 0x7fca40151390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:00.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.345+0000 7fca3ea00700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca400d1b70 0x7fca400b61b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fca2c013000 tx=0x7fca2c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.345+0000 7f131a200700 1 -- 172.21.15.89:0/2025594184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13141597f0 msgr2=0x7f13140b3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.345+0000 7f131a200700 1 --2- 172.21.15.89:0/2025594184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13141597f0 0x7f13140b3a10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f13140ccab0 tx=0x7f130400b330 comp rx=0 tx=0).stop 2024-04-02T07:36:00.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.346+0000 7f131a200700 1 -- 172.21.15.89:0/2025594184 shutdown_connections 2024-04-02T07:36:00.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.345+0000 7f43f5600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f0156cc0 con 0x7f43f00724c0 2024-04-02T07:36:00.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.345+0000 7f43f5600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f0141460 con 0x7f43f00cf4f0 2024-04-02T07:36:00.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.345+0000 7f43f5600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f0098ef0 con 0x7f43f015d7a0 2024-04-02T07:36:00.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.345+0000 7f43eea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f00724c0 0x7f43f0140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.346+0000 7f131a200700 1 --2- 172.21.15.89:0/2025594184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13141597f0 0x7f13140b3a10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.346+0000 7f1f34c00700 1 -- 172.21.15.89:0/1789013898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f30067720 con 0x7f1f300d21f0 2024-04-02T07:36:00.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.346+0000 7f131a200700 1 --2- 172.21.15.89:0/2025594184 >> v2:172.21.15.113:3300/0 conn(0x7f131415d4d0 0x7f13140b2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.346+0000 7f131a200700 1 -- 172.21.15.89:0/2025594184 >> 172.21.15.89:0/2025594184 conn(0x7f13140b0d70 msgr2=0x7f13140b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.346+0000 7fca37400700 1 -- 172.21.15.89:0/1705899622 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c003c30 con 0x7fca400d1b70 2024-04-02T07:36:00.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.346+0000 7fca37400700 1 -- 172.21.15.89:0/1705899622 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2c003d90 con 0x7fca400d1b70 2024-04-02T07:36:00.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.346+0000 7fca37400700 1 -- 172.21.15.89:0/1705899622 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c011660 con 0x7fca400d1b70 2024-04-02T07:36:00.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.346+0000 7fca46600700 1 -- 172.21.15.89:0/1705899622 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca400c86a0 con 0x7fca400d1b70 2024-04-02T07:36:00.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.345+0000 7f43eea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f00724c0 0x7f43f0140f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52616/0 (socket says 172.21.15.89:52616) 2024-04-02T07:36:00.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.345+0000 7f43eea00700 1 -- 172.21.15.89:0/2190103672 learned_addr learned my addr 172.21.15.89:0/2190103672 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:00.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.345+0000 7f43ef400700 1 --2- 172.21.15.89:0/2190103672 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43f015d7a0 0x7f43f01605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.347+0000 7f131a200700 1 -- 172.21.15.89:0/2025594184 shutdown_connections 2024-04-02T07:36:00.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.347+0000 7f131a200700 1 -- 172.21.15.89:0/2025594184 wait complete. 2024-04-02T07:36:00.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.347+0000 7ffb56a00700 1 -- 172.21.15.89:0/1645032900 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffb48003ba0 con 0x7ffb5804fb80 2024-04-02T07:36:00.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.347+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1645032900 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb580053f0 con 0x7ffb5804fb80 2024-04-02T07:36:00.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.345+0000 7f43e7e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43f00cf4f0 0x7f43f00989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.346+0000 7f43ef400700 1 -- 172.21.15.89:0/2190103672 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43f00cf4f0 msgr2=0x7f43f00989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.346+0000 7f43ef400700 1 --2- 172.21.15.89:0/2190103672 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43f00cf4f0 0x7f43f00989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.346+0000 7f43ef400700 1 -- 172.21.15.89:0/2190103672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f00724c0 msgr2=0x7f43f0140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.346+0000 7f43ef400700 1 --2- 172.21.15.89:0/2190103672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f00724c0 0x7f43f0140f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.346+0000 7f43ef400700 1 -- 172.21.15.89:0/2190103672 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e00094a0 con 0x7f43f015d7a0 2024-04-02T07:36:00.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.347+0000 7f43ef400700 1 --2- 172.21.15.89:0/2190103672 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43f015d7a0 0x7f43f01605c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f43e8014670 tx=0x7f43e8014980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.347+0000 7f1f26000700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f1006ac90 0x7f1f1006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.347+0000 7f1f26000700 1 -- 172.21.15.89:0/1789013898 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f1c08a460 con 0x7f1f300d21f0 2024-04-02T07:36:00.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.347+0000 7fca46600700 1 -- 172.21.15.89:0/1705899622 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca400c8c60 con 0x7fca400d1b70 2024-04-02T07:36:00.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.348+0000 7fbf02000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbef4038140 0x7fbef40e2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.349+0000 7fbf02000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef4057cd0 0x7fbef40eaf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.349+0000 7fbf02000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbef40eb450 0x7fbef40e43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.349+0000 7fbf02000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef40eb8e0 con 0x7fbef4038140 2024-04-02T07:36:00.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.349+0000 7fbf02000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef40e3350 con 0x7fbef4057cd0 2024-04-02T07:36:00.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.349+0000 7fbefbe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef4057cd0 0x7fbef40eaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.348+0000 7f1f2de00700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f1006ac90 0x7f1f1006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.347+0000 7f43e6a00700 1 -- 172.21.15.89:0/2190103672 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e8017070 con 0x7f43f015d7a0 2024-04-02T07:36:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.348+0000 7f43f5600700 1 -- 172.21.15.89:0/2190103672 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f0156320 con 0x7f43f015d7a0 2024-04-02T07:36:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.348+0000 7f43f5600700 1 -- 172.21.15.89:0/2190103672 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f01568e0 con 0x7f43f015d7a0 2024-04-02T07:36:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.348+0000 7f43e6a00700 1 -- 172.21.15.89:0/2190103672 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e80020e0 con 0x7f43f015d7a0 2024-04-02T07:36:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.349+0000 7f048d800700 1 Processor -- start 2024-04-02T07:36:00.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.348+0000 7ffb56a00700 1 --2- 172.21.15.89:0/1645032900 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb3c06ace0 0x7ffb3c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.349+0000 7ffb56a00700 1 -- 172.21.15.89:0/1645032900 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb48021070 con 0x7ffb5804fb80 2024-04-02T07:36:00.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.349+0000 7ffb5ea00700 1 --2- 172.21.15.89:0/1645032900 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb3c06ace0 0x7ffb3c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.349+0000 7fbf02000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef40e2740 con 0x7fbef40eb450 2024-04-02T07:36:00.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.349+0000 7fbefbe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef4057cd0 0x7fbef40eaf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33598/0 (socket says 172.21.15.89:33598) 2024-04-02T07:36:00.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.349+0000 7fbefbe00700 1 -- 172.21.15.89:0/3727365948 learned_addr learned my addr 172.21.15.89:0/3727365948 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:00.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.349+0000 7f43e6a00700 1 -- 172.21.15.89:0/2190103672 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e8017070 con 0x7f43f015d7a0 2024-04-02T07:36:00.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.349+0000 7f048d800700 1 -- start start 2024-04-02T07:36:00.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.350+0000 7f43f5600700 1 -- 172.21.15.89:0/2190103672 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c80052f0 con 0x7f43f015d7a0 2024-04-02T07:36:00.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.350+0000 7fbf01600700 1 --2- 172.21.15.89:0/3727365948 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbef40eb450 0x7fbef40e43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7fbefbe00700 1 -- 172.21.15.89:0/3727365948 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbef40eb450 msgr2=0x7fbef40e43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7fbefbe00700 1 --2- 172.21.15.89:0/3727365948 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbef40eb450 0x7fbef40e43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7fbefbe00700 1 -- 172.21.15.89:0/3727365948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbef4038140 msgr2=0x7fbef40e2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7fbefbe00700 1 --2- 172.21.15.89:0/3727365948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbef4038140 0x7fbef40e2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7fbefbe00700 1 -- 172.21.15.89:0/3727365948 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee80094a0 con 0x7fbef4057cd0 2024-04-02T07:36:00.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7f2dce000700 1 -- 172.21.15.89:0/2584468145 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2dc8013970 con 0x7f2dd815d710 2024-04-02T07:36:00.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7f131a200700 1 Processor -- start 2024-04-02T07:36:00.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7f1f2de00700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f1006ac90 0x7f1f1006d150 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1f20000f60 tx=0x7f1f20007eb0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7f048d800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f04880ce500 0x7f04880b2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7f048d800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04880cc370 0x7f04880ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7f048d800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0488163190 con 0x7f04880b3760 2024-04-02T07:36:00.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7f048d800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04880cee20 con 0x7f04880ce500 2024-04-02T07:36:00.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7f048d800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0488162580 con 0x7f04880cc370 2024-04-02T07:36:00.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7ffb5ea00700 1 --2- 172.21.15.89:0/1645032900 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb3c06ace0 0x7ffb3c06d1a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffb500017a0 tx=0x7ffb50009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.351+0000 7f131a200700 1 -- start start 2024-04-02T07:36:00.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.352+0000 7f0486000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04880cc370 0x7f04880ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.352+0000 7f0487400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f04880b3760 0x7f0488165ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34872/0 (socket says 172.21.15.89:34872) 2024-04-02T07:36:00.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.352+0000 7f0487400700 1 -- 172.21.15.89:0/3523197142 learned_addr learned my addr 172.21.15.89:0/3523197142 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:00.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.352+0000 7fbefbe00700 1 --2- 172.21.15.89:0/3727365948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef4057cd0 0x7fbef40eaf10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbef000c4b0 tx=0x7fbef000c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.352+0000 7fbef8c00700 1 -- 172.21.15.89:0/3727365948 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef0005d70 con 0x7fbef4057cd0 2024-04-02T07:36:00.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.353+0000 7fbef8c00700 1 -- 172.21.15.89:0/3727365948 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef00070d0 con 0x7fbef4057cd0 2024-04-02T07:36:00.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.353+0000 7fbef8c00700 1 -- 172.21.15.89:0/3727365948 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef0007440 con 0x7fbef4057cd0 2024-04-02T07:36:00.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.352+0000 7f2dce000700 1 --2- 172.21.15.89:0/2584468145 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2db806acf0 0x7f2db806d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.353+0000 7f1f26000700 1 -- 172.21.15.89:0/1789013898 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1f1c055750 con 0x7f1f300d21f0 2024-04-02T07:36:00.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.353+0000 7f131a200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f131406dc60 0x7f13140c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.353+0000 7f0486000700 1 -- 172.21.15.89:0/3523197142 >> v2:172.21.15.113:3300/0 conn(0x7f04880ce500 msgr2=0x7f04880b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:00.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.353+0000 7f0486000700 1 --2- 172.21.15.89:0/3523197142 >> v2:172.21.15.113:3300/0 conn(0x7f04880ce500 0x7f04880b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.353+0000 7f0486000700 1 -- 172.21.15.89:0/3523197142 >> v1:172.21.15.89:6789/0 conn(0x7f04880b3760 legacy=0x7f0488165ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.353+0000 7fca46600700 1 -- 172.21.15.89:0/1705899622 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca400c8df0 con 0x7fca400d1b70 2024-04-02T07:36:00.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.353+0000 7f0486000700 1 -- 172.21.15.89:0/3523197142 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0488162860 con 0x7f04880cc370 2024-04-02T07:36:00.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.353+0000 7f1313400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f131406dc60 0x7f13140c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.353+0000 7f0486000700 1 --2- 172.21.15.89:0/3523197142 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04880cc370 0x7f04880ce8e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0478009840 tx=0x7f0478009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3b6d6146268d2e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.353+0000 7f1313400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f131406dc60 0x7f13140c49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52646/0 (socket says 172.21.15.89:52646) 2024-04-02T07:36:00.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.353+0000 7f131a200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13141597f0 0x7f13141559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.354+0000 7f131a200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f131415d4d0 0x7f13140c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.354+0000 7f131a200700 1 -- 172.21.15.89:0/1797134703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1314159bd0 con 0x7f131406dc60 2024-04-02T07:36:00.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.354+0000 7f1313400700 1 -- 172.21.15.89:0/1797134703 learned_addr learned my addr 172.21.15.89:0/1797134703 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:00.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.354+0000 7f131a200700 1 -- 172.21.15.89:0/1797134703 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1314159eb0 con 0x7f131415d4d0 2024-04-02T07:36:00.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.354+0000 7f131a200700 1 -- 172.21.15.89:0/1797134703 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f131415a190 con 0x7f13141597f0 2024-04-02T07:36:00.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.354+0000 7f130aa00700 1 --2- 172.21.15.89:0/1797134703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13141597f0 0x7f13141559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.354+0000 7fca37400700 1 -- 172.21.15.89:0/1705899622 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fca2c0117c0 con 0x7fca400d1b70 2024-04-02T07:36:00.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.354+0000 7f1313e00700 1 --2- 172.21.15.89:0/1797134703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f131415d4d0 0x7f13140c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.355+0000 7f130aa00700 1 -- 172.21.15.89:0/1797134703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f131415d4d0 msgr2=0x7f13140c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.355+0000 7f130aa00700 1 --2- 172.21.15.89:0/1797134703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f131415d4d0 0x7f13140c5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.355+0000 7f130aa00700 1 -- 172.21.15.89:0/1797134703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f131406dc60 msgr2=0x7f13140c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.355+0000 7f130aa00700 1 --2- 172.21.15.89:0/1797134703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f131406dc60 0x7f13140c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.355+0000 7f130aa00700 1 -- 172.21.15.89:0/1797134703 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc009dc0 con 0x7f13141597f0 2024-04-02T07:36:00.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.355+0000 7f130aa00700 1 --2- 172.21.15.89:0/1797134703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13141597f0 0x7f13141559b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00e890 tx=0x7f12fc00eba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.355+0000 7fbf02000700 1 -- 172.21.15.89:0/3727365948 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef40e2a20 con 0x7fbef4057cd0 2024-04-02T07:36:00.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.355+0000 7fca37400700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fca2006ace0 0x7fca2006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.356+0000 7fca37400700 1 -- 172.21.15.89:0/1705899622 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca2c08a290 con 0x7fca400d1b70 2024-04-02T07:36:00.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.356+0000 7fbf02000700 1 -- 172.21.15.89:0/3727365948 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef40e2fe0 con 0x7fbef4057cd0 2024-04-02T07:36:00.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.356+0000 7f0485600700 1 -- 172.21.15.89:0/3523197142 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0478011830 con 0x7f04880cc370 2024-04-02T07:36:00.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.356+0000 7f0485600700 1 -- 172.21.15.89:0/3523197142 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0478011990 con 0x7f04880cc370 2024-04-02T07:36:00.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.356+0000 7f0485600700 1 -- 172.21.15.89:0/3523197142 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0478011d20 con 0x7f04880cc370 2024-04-02T07:36:00.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.356+0000 7ffb56a00700 1 -- 172.21.15.89:0/1645032900 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffb48055dc0 con 0x7ffb5804fb80 2024-04-02T07:36:00.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.355+0000 7f2dce000700 1 -- 172.21.15.89:0/2584468145 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2dc8089460 con 0x7f2dd815d710 2024-04-02T07:36:00.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.356+0000 7f130be00700 1 -- 172.21.15.89:0/1797134703 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc014890 con 0x7f13141597f0 2024-04-02T07:36:00.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.356+0000 7f130be00700 1 -- 172.21.15.89:0/1797134703 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12fc00f040 con 0x7f13141597f0 2024-04-02T07:36:00.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.356+0000 7f130be00700 1 -- 172.21.15.89:0/1797134703 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc014c30 con 0x7f13141597f0 2024-04-02T07:36:00.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.357+0000 7f131a200700 1 -- 172.21.15.89:0/1797134703 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1304009b30 con 0x7f13141597f0 2024-04-02T07:36:00.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.357+0000 7f131a200700 1 -- 172.21.15.89:0/1797134703 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13140b40d0 con 0x7f13141597f0 2024-04-02T07:36:00.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.357+0000 7f048d800700 1 -- 172.21.15.89:0/3523197142 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04880cc370 msgr2=0x7f04880ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.357+0000 7f048d800700 1 --2- 172.21.15.89:0/3523197142 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04880cc370 0x7f04880ce8e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0478009840 tx=0x7f0478009b50 comp rx=0 tx=0).stop 2024-04-02T07:36:00.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.357+0000 7f2dd5400700 1 --2- 172.21.15.89:0/2584468145 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2db806acf0 0x7f2db806d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.357+0000 7f048d800700 1 -- 172.21.15.89:0/3523197142 shutdown_connections 2024-04-02T07:36:00.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.357+0000 7f048d800700 1 --2- 172.21.15.89:0/3523197142 >> v2:172.21.15.113:3300/0 conn(0x7f04880ce500 0x7f04880b2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.357+0000 7f048d800700 1 --2- 172.21.15.89:0/3523197142 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04880cc370 0x7f04880ce8e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.357+0000 7f048d800700 1 -- 172.21.15.89:0/3523197142 >> 172.21.15.89:0/3523197142 conn(0x7f04880b0d70 msgr2=0x7f04880b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.358+0000 7f131a200700 1 -- 172.21.15.89:0/1797134703 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1314067720 con 0x7f13141597f0 2024-04-02T07:36:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.357+0000 7fca3f400700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fca2006ace0 0x7fca2006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.358+0000 7fca3f400700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fca2006ace0 0x7fca2006d1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fca4015cc90 tx=0x7fca30008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.359+0000 7f048d800700 1 -- 172.21.15.89:0/3523197142 shutdown_connections 2024-04-02T07:36:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.359+0000 7f048d800700 1 -- 172.21.15.89:0/3523197142 wait complete. 2024-04-02T07:36:00.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.359+0000 7f2dd5400700 1 --2- 172.21.15.89:0/2584468145 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2db806acf0 0x7f2db806d1b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2dd81f1ec0 tx=0x7f2dc4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.359+0000 7f43e6a00700 1 -- 172.21.15.89:0/2190103672 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f43e8012320 con 0x7f43f015d7a0 2024-04-02T07:36:00.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.359+0000 7f2dce000700 1 -- 172.21.15.89:0/2584468145 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2dc80545a0 con 0x7f2dd815d710 2024-04-02T07:36:00.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.359+0000 7fbf02000700 1 -- 172.21.15.89:0/3727365948 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef4005840 con 0x7fbef4057cd0 2024-04-02T07:36:00.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.360+0000 7f43e6a00700 1 --2- 172.21.15.89:0/2190103672 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f43cc06ad50 0x7f43cc06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.360+0000 7f43e6a00700 1 -- 172.21.15.89:0/2190103672 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43e8091830 con 0x7f43f015d7a0 2024-04-02T07:36:00.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.360+0000 7f43eea00700 1 --2- 172.21.15.89:0/2190103672 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f43cc06ad50 0x7f43cc06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.361+0000 7fca37400700 1 -- 172.21.15.89:0/1705899622 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fca2c0555e0 con 0x7fca400d1b70 2024-04-02T07:36:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.361+0000 7f048d800700 1 Processor -- start 2024-04-02T07:36:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.362+0000 7f048d800700 1 -- start start 2024-04-02T07:36:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.362+0000 7f43e6a00700 1 -- 172.21.15.89:0/2190103672 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f43e805ca00 con 0x7f43f015d7a0 2024-04-02T07:36:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.363+0000 7f43eea00700 1 --2- 172.21.15.89:0/2190103672 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f43cc06ad50 0x7f43cc06d210 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f43d8009b90 tx=0x7f43d8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.363+0000 7fbef8c00700 1 -- 172.21.15.89:0/3727365948 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbef00075a0 con 0x7fbef4057cd0 2024-04-02T07:36:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.364+0000 7f130be00700 1 -- 172.21.15.89:0/1797134703 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f12fc026430 con 0x7f13141597f0 2024-04-02T07:36:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.364+0000 7f048d800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04880b3760 0x7f0488093800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.364+0000 7f048d800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04880cc370 0x7f0488150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.364+0000 7f048d800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04880ce500 0x7f048806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.364+0000 7f048d800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0488099270 con 0x7f04880cc370 2024-04-02T07:36:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.364+0000 7f048d800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0488099550 con 0x7f04880b3760 2024-04-02T07:36:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.364+0000 7f048d800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0488099830 con 0x7f04880ce500 2024-04-02T07:36:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.364+0000 7f0486a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04880b3760 0x7f0488093800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.364+0000 7f0486a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04880b3760 0x7f0488093800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33638/0 (socket says 172.21.15.89:33638) 2024-04-02T07:36:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.364+0000 7f0486a00700 1 -- 172.21.15.89:0/2880528197 learned_addr learned my addr 172.21.15.89:0/2880528197 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.364+0000 7f0487400700 1 --2- 172.21.15.89:0/2880528197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04880ce500 0x7f048806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.364+0000 7fbef8c00700 1 --2- 172.21.15.89:0/3727365948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbedc06ad60 0x7fbedc06d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.365+0000 7fbef8c00700 1 -- 172.21.15.89:0/3727365948 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbef0088f40 con 0x7fbef4057cd0 2024-04-02T07:36:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.365+0000 7fbf00c00700 1 --2- 172.21.15.89:0/3727365948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbedc06ad60 0x7fbedc06d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.366+0000 7fbf00c00700 1 --2- 172.21.15.89:0/3727365948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbedc06ad60 0x7fbedc06d220 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbefc0644a0 tx=0x7fbefc068df0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.367+0000 7fbef8c00700 1 -- 172.21.15.89:0/3727365948 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbef00541b0 con 0x7fbef4057cd0 2024-04-02T07:36:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.367+0000 7f0486a00700 1 -- 172.21.15.89:0/2880528197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04880ce500 msgr2=0x7f048806d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.367+0000 7f0486a00700 1 --2- 172.21.15.89:0/2880528197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04880ce500 0x7f048806d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.367+0000 7f0486a00700 1 -- 172.21.15.89:0/2880528197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04880cc370 msgr2=0x7f0488150cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.367+0000 7f0486a00700 1 --2- 172.21.15.89:0/2880528197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04880cc370 0x7f0488150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.367+0000 7f0486a00700 1 -- 172.21.15.89:0/2880528197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04780094a0 con 0x7f04880b3760 2024-04-02T07:36:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.365+0000 7f130be00700 1 --2- 172.21.15.89:0/1797134703 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f12f006ad60 0x7f12f006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.365+0000 7f130be00700 1 -- 172.21.15.89:0/1797134703 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12fc089880 con 0x7f13141597f0 2024-04-02T07:36:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.367+0000 7f130be00700 1 -- 172.21.15.89:0/1797134703 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f12fc054ad0 con 0x7f13141597f0 2024-04-02T07:36:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.368+0000 7f0486a00700 1 --2- 172.21.15.89:0/2880528197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04880b3760 0x7f0488093800 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f047000b4a0 tx=0x7f047000b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.369+0000 7f1313400700 1 --2- 172.21.15.89:0/1797134703 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f12f006ad60 0x7f12f006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.370+0000 7f1313400700 1 --2- 172.21.15.89:0/1797134703 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f12f006ad60 0x7f12f006d220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1304009440 tx=0x7f1304009000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.371+0000 7f047ea00700 1 -- 172.21.15.89:0/2880528197 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0470011830 con 0x7f04880b3760 2024-04-02T07:36:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.371+0000 7f047ea00700 1 -- 172.21.15.89:0/2880528197 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0470011990 con 0x7f04880b3760 2024-04-02T07:36:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.371+0000 7f048d800700 1 -- 172.21.15.89:0/2880528197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0488099b10 con 0x7f04880b3760 2024-04-02T07:36:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.371+0000 7f047ea00700 1 -- 172.21.15.89:0/2880528197 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0470011d40 con 0x7f04880b3760 2024-04-02T07:36:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.371+0000 7f048d800700 1 -- 172.21.15.89:0/2880528197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0488093d40 con 0x7f04880b3760 2024-04-02T07:36:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.373+0000 7f048d800700 1 -- 172.21.15.89:0/2880528197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0488067720 con 0x7f04880b3760 2024-04-02T07:36:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.375+0000 7f047ea00700 1 -- 172.21.15.89:0/2880528197 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f047001a400 con 0x7f04880b3760 2024-04-02T07:36:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.377+0000 7f047ea00700 1 --2- 172.21.15.89:0/2880528197 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f046406ad60 0x7f046406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.377+0000 7f047ea00700 1 -- 172.21.15.89:0/2880528197 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04700892f0 con 0x7f04880b3760 2024-04-02T07:36:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.378+0000 7f0486000700 1 --2- 172.21.15.89:0/2880528197 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f046406ad60 0x7f046406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.379+0000 7f0486000700 1 --2- 172.21.15.89:0/2880528197 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f046406ad60 0x7f046406d220 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f04880cc750 tx=0x7f0478010000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.381+0000 7f047ea00700 1 -- 172.21.15.89:0/2880528197 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f04700544c0 con 0x7f04880b3760 2024-04-02T07:36:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.820+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1645032900 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ffb58052c90 con 0x7ffb5804fb80 2024-04-02T07:36:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.821+0000 7ffb56a00700 1 -- 172.21.15.89:0/1645032900 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffb48015070 con 0x7ffb5804fb80 2024-04-02T07:36:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.825+0000 7f1f34c00700 1 -- 172.21.15.89:0/1789013898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1f300b5d00 con 0x7f1f300d21f0 2024-04-02T07:36:00.827 INFO:teuthology.orchestra.run.smithi089.stdout:55834574852 2024-04-02T07:36:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.826+0000 7f1f26000700 1 -- 172.21.15.89:0/1789013898 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1f1c015020 con 0x7f1f300d21f0 2024-04-02T07:36:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.826+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1645032900 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb3c06ace0 msgr2=0x7ffb3c06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.826+0000 7ffb5fe00700 1 --2- 172.21.15.89:0/1645032900 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb3c06ace0 0x7ffb3c06d1a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffb500017a0 tx=0x7ffb50009040 comp rx=0 tx=0).stop 2024-04-02T07:36:00.828 INFO:teuthology.orchestra.run.smithi089.stdout:55834574852 2024-04-02T07:36:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.827+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1645032900 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb5804fb80 msgr2=0x7ffb5804dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.827+0000 7ffb5fe00700 1 --2- 172.21.15.89:0/1645032900 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb5804fb80 0x7ffb5804dc60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffb48016000 tx=0x7ffb48004750 comp rx=0 tx=0).stop 2024-04-02T07:36:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.827+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1645032900 shutdown_connections 2024-04-02T07:36:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.827+0000 7ffb5fe00700 1 --2- 172.21.15.89:0/1645032900 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb3c06ace0 0x7ffb3c06d1a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.827+0000 7ffb5fe00700 1 --2- 172.21.15.89:0/1645032900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5804f7a0 0x7ffb580cf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.827+0000 7ffb5fe00700 1 --2- 172.21.15.89:0/1645032900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb580532f0 0x7ffb580d4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.827+0000 7ffb5fe00700 1 --2- 172.21.15.89:0/1645032900 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb5804fb80 0x7ffb5804dc60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.827+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1645032900 >> 172.21.15.89:0/1645032900 conn(0x7ffb58054b20 msgr2=0x7ffb580e08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.828+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1645032900 shutdown_connections 2024-04-02T07:36:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.828+0000 7ffb5fe00700 1 -- 172.21.15.89:0/1645032900 wait complete. 2024-04-02T07:36:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.829+0000 7f2ddce00700 1 -- 172.21.15.89:0/2584468145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2dd80b1520 con 0x7f2dd815d710 2024-04-02T07:36:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.830+0000 7fca46600700 1 -- 172.21.15.89:0/1705899622 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fca40067e30 con 0x7fca400d1b70 2024-04-02T07:36:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.830+0000 7f2dce000700 1 -- 172.21.15.89:0/2584468145 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2dc80545a0 con 0x7f2dd815d710 2024-04-02T07:36:00.832 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-04-02T07:36:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.831+0000 7f1f34c00700 1 -- 172.21.15.89:0/1789013898 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f1006ac90 msgr2=0x7f1f1006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.831+0000 7f1f34c00700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f1006ac90 0x7f1f1006d150 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1f20000f60 tx=0x7f1f20007eb0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.831+0000 7f1f34c00700 1 -- 172.21.15.89:0/1789013898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f300d21f0 msgr2=0x7f1f301f6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.831+0000 7f1f34c00700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f300d21f0 0x7f1f301f6430 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1f3008ce10 tx=0x7f1f1c004750 comp rx=0 tx=0).stop 2024-04-02T07:36:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.831+0000 7fca37400700 1 -- 172.21.15.89:0/1705899622 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fca2c058c00 con 0x7fca400d1b70 2024-04-02T07:36:00.834 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2024-04-02T07:36:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.833+0000 7f1f34c00700 1 -- 172.21.15.89:0/1789013898 shutdown_connections 2024-04-02T07:36:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.833+0000 7f1f34c00700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f1006ac90 0x7f1f1006d150 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.833+0000 7f1f34c00700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f301611b0 0x7f1f301f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.833+0000 7f1f34c00700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f300cec40 0x7f1f300ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.833+0000 7f1f34c00700 1 --2- 172.21.15.89:0/1789013898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f300d21f0 0x7f1f301f6430 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.833+0000 7f1f34c00700 1 -- 172.21.15.89:0/1789013898 >> 172.21.15.89:0/1789013898 conn(0x7f1f3008aff0 msgr2=0x7f1f3006c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.833+0000 7f1f34c00700 1 -- 172.21.15.89:0/1789013898 shutdown_connections 2024-04-02T07:36:00.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.833+0000 7f2ddce00700 1 -- 172.21.15.89:0/2584468145 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2db806acf0 msgr2=0x7f2db806d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.833+0000 7f2ddce00700 1 --2- 172.21.15.89:0/2584468145 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2db806acf0 0x7f2db806d1b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2dd81f1ec0 tx=0x7f2dc4008040 comp rx=0 tx=0).stop 2024-04-02T07:36:00.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.834+0000 7f2ddce00700 1 -- 172.21.15.89:0/2584468145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dd815d710 msgr2=0x7f2dd80b16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.834+0000 7f2ddce00700 1 --2- 172.21.15.89:0/2584468145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dd815d710 0x7f2dd80b16d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8011960 tx=0x7f2dc8011c70 comp rx=0 tx=0).stop 2024-04-02T07:36:00.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.833+0000 7f1f34c00700 1 -- 172.21.15.89:0/1789013898 wait complete. 2024-04-02T07:36:00.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.834+0000 7f2ddce00700 1 -- 172.21.15.89:0/2584468145 shutdown_connections 2024-04-02T07:36:00.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.834+0000 7f2ddce00700 1 --2- 172.21.15.89:0/2584468145 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2db806acf0 0x7f2db806d1b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.834+0000 7f2ddce00700 1 --2- 172.21.15.89:0/2584468145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dd80b3c70 0x7f2dd80b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.834+0000 7f2ddce00700 1 --2- 172.21.15.89:0/2584468145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dd815d710 0x7f2dd80b16d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.834+0000 7f2ddce00700 1 --2- 172.21.15.89:0/2584468145 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2dd814dad0 0x7f2dd815e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.834+0000 7f2ddce00700 1 -- 172.21.15.89:0/2584468145 >> 172.21.15.89:0/2584468145 conn(0x7f2dd808aff0 msgr2=0x7f2dd80d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.835+0000 7f2ddce00700 1 -- 172.21.15.89:0/2584468145 shutdown_connections 2024-04-02T07:36:00.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.835+0000 7f2ddce00700 1 -- 172.21.15.89:0/2584468145 wait complete. 2024-04-02T07:36:00.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.836+0000 7fca46600700 1 -- 172.21.15.89:0/1705899622 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fca2006ace0 msgr2=0x7fca2006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.836+0000 7fca46600700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fca2006ace0 0x7fca2006d1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fca4015cc90 tx=0x7fca30008040 comp rx=0 tx=0).stop 2024-04-02T07:36:00.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.836+0000 7fca46600700 1 -- 172.21.15.89:0/1705899622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca400d1b70 msgr2=0x7fca400b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.836+0000 7fca46600700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca400d1b70 0x7fca400b61b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fca2c013000 tx=0x7fca2c004750 comp rx=0 tx=0).stop 2024-04-02T07:36:00.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.837+0000 7fca46600700 1 -- 172.21.15.89:0/1705899622 shutdown_connections 2024-04-02T07:36:00.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.837+0000 7fca46600700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fca2006ace0 0x7fca2006d1a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.837+0000 7fca46600700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca4015d710 0x7fca4015e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.837+0000 7fca46600700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca400cff10 0x7fca40151390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.837+0000 7fca46600700 1 --2- 172.21.15.89:0/1705899622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca400d1b70 0x7fca400b61b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.837+0000 7fca46600700 1 -- 172.21.15.89:0/1705899622 >> 172.21.15.89:0/1705899622 conn(0x7fca4008aff0 msgr2=0x7fca400b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.841+0000 7fbf02000700 1 -- 172.21.15.89:0/3727365948 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fbef40e1730 con 0x7fbef4057cd0 2024-04-02T07:36:00.843 INFO:teuthology.orchestra.run.smithi089.stdout:55834574852 2024-04-02T07:36:00.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.840+0000 7f43f5600700 1 -- 172.21.15.89:0/2190103672 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f43c8005870 con 0x7f43f015d7a0 2024-04-02T07:36:00.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.841+0000 7f43e6a00700 1 -- 172.21.15.89:0/2190103672 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f43e8022090 con 0x7f43f015d7a0 2024-04-02T07:36:00.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.837+0000 7fca46600700 1 -- 172.21.15.89:0/1705899622 shutdown_connections 2024-04-02T07:36:00.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.841+0000 7fca46600700 1 -- 172.21.15.89:0/1705899622 wait complete. 2024-04-02T07:36:00.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.841+0000 7fbef8c00700 1 -- 172.21.15.89:0/3727365948 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbef0017090 con 0x7fbef4057cd0 2024-04-02T07:36:00.844 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-04-02T07:36:00.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.844+0000 7f131a200700 1 -- 172.21.15.89:0/1797134703 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f131415a2d0 con 0x7f13141597f0 2024-04-02T07:36:00.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.845+0000 7f43f5600700 1 -- 172.21.15.89:0/2190103672 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f43cc06ad50 msgr2=0x7f43cc06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.845+0000 7f130be00700 1 -- 172.21.15.89:0/1797134703 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f131415a2d0 con 0x7f13141597f0 2024-04-02T07:36:00.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.845+0000 7fbf02000700 1 -- 172.21.15.89:0/3727365948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbedc06ad60 msgr2=0x7fbedc06d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.848 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2024-04-02T07:36:00.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.845+0000 7fbf02000700 1 --2- 172.21.15.89:0/3727365948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbedc06ad60 0x7fbedc06d220 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbefc0644a0 tx=0x7fbefc068df0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.845+0000 7fbf02000700 1 -- 172.21.15.89:0/3727365948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef4057cd0 msgr2=0x7fbef40eaf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.845+0000 7fbf02000700 1 --2- 172.21.15.89:0/3727365948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef4057cd0 0x7fbef40eaf10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbef000c4b0 tx=0x7fbef000c870 comp rx=0 tx=0).stop 2024-04-02T07:36:00.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.845+0000 7f43f5600700 1 --2- 172.21.15.89:0/2190103672 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f43cc06ad50 0x7f43cc06d210 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f43d8009b90 tx=0x7f43d8008040 comp rx=0 tx=0).stop 2024-04-02T07:36:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.846+0000 7fbf02000700 1 -- 172.21.15.89:0/3727365948 shutdown_connections 2024-04-02T07:36:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.846+0000 7fbf02000700 1 --2- 172.21.15.89:0/3727365948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbedc06ad60 0x7fbedc06d220 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.846+0000 7fbf02000700 1 --2- 172.21.15.89:0/3727365948 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbef40eb450 0x7fbef40e43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.846+0000 7fbf02000700 1 --2- 172.21.15.89:0/3727365948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbef4038140 0x7fbef40e2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.846+0000 7fbf02000700 1 --2- 172.21.15.89:0/3727365948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef4057cd0 0x7fbef40eaf10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.846+0000 7fbf02000700 1 -- 172.21.15.89:0/3727365948 >> 172.21.15.89:0/3727365948 conn(0x7fbef4055a40 msgr2=0x7fbef4055e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.846+0000 7fbf02000700 1 -- 172.21.15.89:0/3727365948 shutdown_connections 2024-04-02T07:36:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.846+0000 7fbf02000700 1 -- 172.21.15.89:0/3727365948 wait complete. 2024-04-02T07:36:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.846+0000 7f43f5600700 1 -- 172.21.15.89:0/2190103672 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43f015d7a0 msgr2=0x7f43f01605c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.846+0000 7f43f5600700 1 --2- 172.21.15.89:0/2190103672 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43f015d7a0 0x7f43f01605c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f43e8014670 tx=0x7f43e8014980 comp rx=0 tx=0).stop 2024-04-02T07:36:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.847+0000 7f43f5600700 1 -- 172.21.15.89:0/2190103672 shutdown_connections 2024-04-02T07:36:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.847+0000 7f43f5600700 1 --2- 172.21.15.89:0/2190103672 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f43cc06ad50 0x7f43cc06d210 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.847+0000 7f43f5600700 1 --2- 172.21.15.89:0/2190103672 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43f015d7a0 0x7f43f01605c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.847+0000 7f43f5600700 1 --2- 172.21.15.89:0/2190103672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f00724c0 0x7f43f0140f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.847+0000 7f43f5600700 1 --2- 172.21.15.89:0/2190103672 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43f00cf4f0 0x7f43f00989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.847+0000 7f43f5600700 1 -- 172.21.15.89:0/2190103672 >> 172.21.15.89:0/2190103672 conn(0x7f43f0070f00 msgr2=0x7f43f0071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.847+0000 7f43f5600700 1 -- 172.21.15.89:0/2190103672 shutdown_connections 2024-04-02T07:36:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.847+0000 7f43f5600700 1 -- 172.21.15.89:0/2190103672 wait complete. 2024-04-02T07:36:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.849+0000 7f048d800700 1 -- 172.21.15.89:0/2880528197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f0488099690 con 0x7f04880b3760 2024-04-02T07:36:00.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.850+0000 7f131a200700 1 -- 172.21.15.89:0/1797134703 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f12f006ad60 msgr2=0x7f12f006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.850+0000 7f131a200700 1 --2- 172.21.15.89:0/1797134703 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f12f006ad60 0x7f12f006d220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1304009440 tx=0x7f1304009000 comp rx=0 tx=0).stop 2024-04-02T07:36:00.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.850+0000 7f131a200700 1 -- 172.21.15.89:0/1797134703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13141597f0 msgr2=0x7f13141559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.850+0000 7f131a200700 1 --2- 172.21.15.89:0/1797134703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13141597f0 0x7f13141559b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00e890 tx=0x7f12fc00eba0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.850+0000 7f131a200700 1 -- 172.21.15.89:0/1797134703 shutdown_connections 2024-04-02T07:36:00.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.850+0000 7f131a200700 1 --2- 172.21.15.89:0/1797134703 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f12f006ad60 0x7f12f006d220 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.850+0000 7f131a200700 1 --2- 172.21.15.89:0/1797134703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f131415d4d0 0x7f13140c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.850+0000 7f131a200700 1 --2- 172.21.15.89:0/1797134703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f131406dc60 0x7f13140c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.850+0000 7f131a200700 1 --2- 172.21.15.89:0/1797134703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13141597f0 0x7f13141559b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.850+0000 7f131a200700 1 -- 172.21.15.89:0/1797134703 >> 172.21.15.89:0/1797134703 conn(0x7f13140b0d70 msgr2=0x7f13140b03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.851+0000 7f131a200700 1 -- 172.21.15.89:0/1797134703 shutdown_connections 2024-04-02T07:36:00.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.851+0000 7f131a200700 1 -- 172.21.15.89:0/1797134703 wait complete. 2024-04-02T07:36:00.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.851+0000 7f047ea00700 1 -- 172.21.15.89:0/2880528197 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0488099690 con 0x7f04880b3760 2024-04-02T07:36:00.859 INFO:teuthology.orchestra.run.smithi089.stdout:55834574852 2024-04-02T07:36:00.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.858+0000 7f048d800700 1 -- 172.21.15.89:0/2880528197 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f046406ad60 msgr2=0x7f046406d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.858+0000 7f048d800700 1 --2- 172.21.15.89:0/2880528197 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f046406ad60 0x7f046406d220 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f04880cc750 tx=0x7f0478010000 comp rx=0 tx=0).stop 2024-04-02T07:36:00.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.858+0000 7f048d800700 1 -- 172.21.15.89:0/2880528197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04880b3760 msgr2=0x7f0488093800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:00.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.858+0000 7f048d800700 1 --2- 172.21.15.89:0/2880528197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04880b3760 0x7f0488093800 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f047000b4a0 tx=0x7f047000b860 comp rx=0 tx=0).stop 2024-04-02T07:36:00.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.859+0000 7f048d800700 1 -- 172.21.15.89:0/2880528197 shutdown_connections 2024-04-02T07:36:00.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.859+0000 7f048d800700 1 --2- 172.21.15.89:0/2880528197 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f046406ad60 0x7f046406d220 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.859+0000 7f048d800700 1 --2- 172.21.15.89:0/2880528197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04880b3760 0x7f0488093800 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f047000b4a0 tx=0x7f047000b860 comp rx=0 tx=0).stop 2024-04-02T07:36:00.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.859+0000 7f048d800700 1 --2- 172.21.15.89:0/2880528197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04880ce500 0x7f048806d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.859+0000 7f048d800700 1 --2- 172.21.15.89:0/2880528197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04880cc370 0x7f0488150cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:00.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.859+0000 7f048d800700 1 -- 172.21.15.89:0/2880528197 >> 172.21.15.89:0/2880528197 conn(0x7f04880b0d70 msgr2=0x7f0488163f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:00.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.860+0000 7f048d800700 1 -- 172.21.15.89:0/2880528197 shutdown_connections 2024-04-02T07:36:00.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:00.860+0000 7f048d800700 1 -- 172.21.15.89:0/2880528197 wait complete. 2024-04-02T07:36:00.868 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.3 2024-04-02T07:36:00.873 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.2 2024-04-02T07:36:00.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2024-04-02T07:36:00.880 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2024-04-02T07:36:00.881 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2024-04-02T07:36:00.890 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2024-04-02T07:36:00.892 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.6 2024-04-02T07:36:00.894 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.5 2024-04-02T07:36:01.869 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-02T07:36:01.873 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T07:36:01.877 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T07:36:01.880 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-02T07:36:01.881 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-02T07:36:01.891 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T07:36:01.892 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-02T07:36:01.895 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-02T07:36:02.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.178+0000 7f2f4dc00700 1 Processor -- start 2024-04-02T07:36:02.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.178+0000 7f575f600700 1 Processor -- start 2024-04-02T07:36:02.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.178+0000 7f2f4dc00700 1 -- start start 2024-04-02T07:36:02.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.181+0000 7f2f4dc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f480cc370 0x7f2f480b2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.178+0000 7f575f600700 1 -- start start 2024-04-02T07:36:02.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.180+0000 7f575f600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f57580cef40 0x7f5758099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.180+0000 7f575f600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57580cf3f0 0x7f5758099fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.180+0000 7f575f600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57580ccdd0 con 0x7f575815f1e0 2024-04-02T07:36:02.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.180+0000 7f575f600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57580cd260 con 0x7f57580cf3f0 2024-04-02T07:36:02.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.180+0000 7f575f600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57581600c0 con 0x7f57580cef40 2024-04-02T07:36:02.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.182+0000 7f5757e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f57580cef40 0x7f5758099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.182+0000 7f5757e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f57580cef40 0x7f5758099a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33652/0 (socket says 172.21.15.89:33652) 2024-04-02T07:36:02.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.182+0000 7f5757e00700 1 -- 172.21.15.89:0/4051123417 learned_addr learned my addr 172.21.15.89:0/4051123417 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:02.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.182+0000 7f2f4dc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f48163190 con 0x7f2f480ce500 2024-04-02T07:36:02.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.183+0000 7f575d600700 1 --2- 172.21.15.89:0/4051123417 >> v2:172.21.15.89:3300/0 conn(0x7f57580cf3f0 0x7f5758099fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.183+0000 7f2f47400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f2f480b3760 0x7f2f48165ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:39718/0 (socket says 172.21.15.89:39718) 2024-04-02T07:36:02.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.183+0000 7f2f47400700 1 -- 172.21.15.89:0/101897120 learned_addr learned my addr 172.21.15.89:0/101897120 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:02.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.183+0000 7f2f4dc00700 1 -- 172.21.15.89:0/101897120 --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f48162580 con 0x7f2f480b3760 2024-04-02T07:36:02.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.184+0000 7f2f46000700 1 --2- 172.21.15.89:0/101897120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f480cc370 0x7f2f480b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.184+0000 7f575d600700 1 -- 172.21.15.89:0/4051123417 >> v2:172.21.15.113:3300/0 conn(0x7f57580cef40 msgr2=0x7f5758099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:02.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.184+0000 7f575d600700 1 --2- 172.21.15.89:0/4051123417 >> v2:172.21.15.113:3300/0 conn(0x7f57580cef40 0x7f5758099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.184+0000 7f575d600700 1 -- 172.21.15.89:0/4051123417 >> v1:172.21.15.89:6789/0 conn(0x7f575815f1e0 legacy=0x7f575815f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.184+0000 7f575d600700 1 -- 172.21.15.89:0/4051123417 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575815cae0 con 0x7f57580cf3f0 2024-04-02T07:36:02.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.185+0000 7f2f4dc00700 1 -- 172.21.15.89:0/101897120 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f48162860 con 0x7f2f480cc370 2024-04-02T07:36:02.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.185+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986055489 0 0) 0x7f2f48162580 con 0x7f2f480b3760 2024-04-02T07:36:02.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.185+0000 7f575d600700 1 --2- 172.21.15.89:0/4051123417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57580cf3f0 0x7f5758099fa0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5744009840 tx=0x7f5744009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a86df1c6f35a58 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.186+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f2c0035b0 con 0x7f2f480b3760 2024-04-02T07:36:02.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.186+0000 7f5757400700 1 -- 172.21.15.89:0/4051123417 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744011830 con 0x7f57580cf3f0 2024-04-02T07:36:02.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.186+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885868877 0 0) 0x7f2f48163190 con 0x7f2f480ce500 2024-04-02T07:36:02.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.186+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f48162580 con 0x7f2f480ce500 2024-04-02T07:36:02.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.186+0000 7f5757400700 1 -- 172.21.15.89:0/4051123417 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5744011990 con 0x7f57580cf3f0 2024-04-02T07:36:02.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.186+0000 7f5757400700 1 -- 172.21.15.89:0/4051123417 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744011d20 con 0x7f57580cf3f0 2024-04-02T07:36:02.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.187+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1741706592 0 0) 0x7f2f48162580 con 0x7f2f480ce500 2024-04-02T07:36:02.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.187+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f480cc370 msgr2=0x7f2f480b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:02.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.187+0000 7f2f45600700 1 --2- 172.21.15.89:0/101897120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f480cc370 0x7f2f480b2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.187+0000 7f575f600700 1 -- 172.21.15.89:0/4051123417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57580cf3f0 msgr2=0x7f5758099fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.187+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 >> v1:172.21.15.113:6789/0 conn(0x7f2f480b3760 legacy=0x7f2f48165ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.187+0000 7f2f46000700 1 --2- 172.21.15.89:0/101897120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f480cc370 0x7f2f480b2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:02.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.188+0000 7f575f600700 1 --2- 172.21.15.89:0/4051123417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57580cf3f0 0x7f5758099fa0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5744009840 tx=0x7f5744009b50 comp rx=0 tx=0).stop 2024-04-02T07:36:02.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.188+0000 7f77eae00700 1 Processor -- start 2024-04-02T07:36:02.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.188+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f48162b40 con 0x7f2f480ce500 2024-04-02T07:36:02.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.188+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f2f38001520 con 0x7f2f480ce500 2024-04-02T07:36:02.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.188+0000 7f575f600700 1 -- 172.21.15.89:0/4051123417 shutdown_connections 2024-04-02T07:36:02.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.188+0000 7f575f600700 1 --2- 172.21.15.89:0/4051123417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57580cf3f0 0x7f5758099fa0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.188+0000 7f575f600700 1 --2- 172.21.15.89:0/4051123417 >> v2:172.21.15.113:3300/0 conn(0x7f57580cef40 0x7f5758099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.188+0000 7f575f600700 1 -- 172.21.15.89:0/4051123417 >> 172.21.15.89:0/4051123417 conn(0x7f57580d1370 msgr2=0x7f57580d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.188+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 >> v1:172.21.15.89:6789/0 conn(0x7f2f480ce500 legacy=0x7f2f480ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.188+0000 7f77eae00700 1 -- start start 2024-04-02T07:36:02.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.189+0000 7f2f45600700 1 --2- 172.21.15.89:0/101897120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c004c50 0x7f2f2c005030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.189+0000 7f575f600700 1 -- 172.21.15.89:0/4051123417 shutdown_connections 2024-04-02T07:36:02.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.189+0000 7f2f45600700 1 --2- 172.21.15.89:0/101897120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f2c0037f0 0x7f2f2c003c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.189+0000 7f575f600700 1 -- 172.21.15.89:0/4051123417 wait complete. 2024-04-02T07:36:02.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.189+0000 7f2f45600700 1 --2- 172.21.15.89:0/101897120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f480cc370 0x7f2f2c0041b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.190+0000 7f2f46000700 1 --2- 172.21.15.89:0/101897120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f2c0037f0 0x7f2f2c003c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.190+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400015b0 con 0x7f2f2c004c50 2024-04-02T07:36:02.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.190+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f48162860 con 0x7f2f2c0037f0 2024-04-02T07:36:02.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.190+0000 7f2f47400700 1 --2- 172.21.15.89:0/101897120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f480cc370 0x7f2f2c0041b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.190+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c00bb70 con 0x7f2f480cc370 2024-04-02T07:36:02.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.191+0000 7f2f46a00700 1 --2- 172.21.15.89:0/101897120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c004c50 0x7f2f2c005030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.191+0000 7f2f47400700 1 -- 172.21.15.89:0/101897120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f2c0037f0 msgr2=0x7f2f2c003c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:02.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.191+0000 7f2f47400700 1 --2- 172.21.15.89:0/101897120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f2c0037f0 0x7f2f2c003c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.191+0000 7f2f47400700 1 -- 172.21.15.89:0/101897120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c004c50 msgr2=0x7f2f2c005030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:02.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.192+0000 7f2f47400700 1 --2- 172.21.15.89:0/101897120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c004c50 0x7f2f2c005030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.192+0000 7f2f47400700 1 -- 172.21.15.89:0/101897120 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f48162580 con 0x7f2f480cc370 2024-04-02T07:36:02.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.192+0000 7f2f46a00700 1 --2- 172.21.15.89:0/101897120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c004c50 0x7f2f2c005030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:02.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.192+0000 7f2f47400700 1 --2- 172.21.15.89:0/101897120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f480cc370 0x7f2f2c0041b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000a770 tx=0x7f2f2c005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27ebb13325cefdfd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.193+0000 7f575f600700 1 Processor -- start 2024-04-02T07:36:02.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.193+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4000a210 con 0x7f2f480cc370 2024-04-02T07:36:02.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.193+0000 7f575f600700 1 -- start start 2024-04-02T07:36:02.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.193+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f40007370 con 0x7f2f480cc370 2024-04-02T07:36:02.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.193+0000 7f2f45600700 1 -- 172.21.15.89:0/101897120 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f40007720 con 0x7f2f480cc370 2024-04-02T07:36:02.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.194+0000 7f77eae00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f77e4160df0 0x7f77e406c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.194+0000 7f2f4dc00700 1 -- 172.21.15.89:0/101897120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f480cc370 msgr2=0x7f2f2c0041b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.194+0000 7f2f4dc00700 1 --2- 172.21.15.89:0/101897120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f480cc370 0x7f2f2c0041b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000a770 tx=0x7f2f2c005880 comp rx=0 tx=0).stop 2024-04-02T07:36:02.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.194+0000 7f77eae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77e40b09d0 0x7f77e40d1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.195+0000 7f2f47400700 1 -- 172.21.15.89:0/101897120 reap_dead start 2024-04-02T07:36:02.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.194+0000 7f77eae00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e40d43d0 con 0x7f77e40d1f70 2024-04-02T07:36:02.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.195+0000 7f575f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f57580cef40 0x7f5758159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.195+0000 7f77eae00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e40d3e10 con 0x7f77e4160df0 2024-04-02T07:36:02.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.195+0000 7f2f4dc00700 1 -- 172.21.15.89:0/101897120 shutdown_connections 2024-04-02T07:36:02.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.195+0000 7f2f4dc00700 1 -- 172.21.15.89:0/101897120 >> 172.21.15.89:0/101897120 conn(0x7f2f480b0d70 msgr2=0x7f2f480b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.195+0000 7f77e9000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77e40b09d0 0x7f77e40d1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.195+0000 7f2f4dc00700 1 -- 172.21.15.89:0/101897120 shutdown_connections 2024-04-02T07:36:02.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.195+0000 7f77e9000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77e40b09d0 0x7f77e40d1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:32890/0 (socket says 172.21.15.89:32890) 2024-04-02T07:36:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.195+0000 7f2f4dc00700 1 -- 172.21.15.89:0/101897120 wait complete. 2024-04-02T07:36:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.195+0000 7f77e9000700 1 -- 172.21.15.89:0/3320974461 learned_addr learned my addr 172.21.15.89:0/3320974461 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.195+0000 7f77eae00700 1 -- 172.21.15.89:0/3320974461 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40d40f0 con 0x7f77e40b09d0 2024-04-02T07:36:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.195+0000 7f575f600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57580cf3f0 0x7f57581537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.196+0000 7f575f600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f575815f1e0 0x7f5758099a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.196+0000 7f77e9a00700 1 --2- 172.21.15.89:0/3320974461 >> v2:172.21.15.113:3300/0 conn(0x7f77e4160df0 0x7f77e406c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.196+0000 7f575f600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5758159540 con 0x7f575815f1e0 2024-04-02T07:36:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.196+0000 7f575f600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575815e430 con 0x7f57580cf3f0 2024-04-02T07:36:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.196+0000 7f575f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575815e710 con 0x7f57580cef40 2024-04-02T07:36:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.197+0000 7f575d600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f575815f1e0 0x7f5758099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.197+0000 7f575d600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f575815f1e0 0x7f5758099a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52662/0 (socket says 172.21.15.89:52662) 2024-04-02T07:36:02.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.197+0000 7f575d600700 1 -- 172.21.15.89:0/3574365830 learned_addr learned my addr 172.21.15.89:0/3574365830 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:02.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.197+0000 7f77e9a00700 1 -- 172.21.15.89:0/3320974461 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77e40b09d0 msgr2=0x7f77e40d1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:02.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.197+0000 7f575cc00700 1 --2- 172.21.15.89:0/3574365830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f57580cef40 0x7f5758159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.197+0000 7f77e9a00700 1 --2- 172.21.15.89:0/3320974461 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77e40b09d0 0x7f77e40d1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.197+0000 7f77e9a00700 1 -- 172.21.15.89:0/3320974461 >> v1:172.21.15.89:6789/0 conn(0x7f77e40d1f70 legacy=0x7f77e406b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.197+0000 7f5757e00700 1 --2- 172.21.15.89:0/3574365830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57580cf3f0 0x7f57581537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.197+0000 7f77e9a00700 1 -- 172.21.15.89:0/3320974461 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e406d0f0 con 0x7f77e4160df0 2024-04-02T07:36:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.198+0000 7f575cc00700 1 -- 172.21.15.89:0/3574365830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57580cf3f0 msgr2=0x7f57581537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.198+0000 7f77e9a00700 1 --2- 172.21.15.89:0/3320974461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e4160df0 0x7f77e406c870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009d90 tx=0x7f77dc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f97acdfff4ed5e0a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.198+0000 7f575cc00700 1 --2- 172.21.15.89:0/3574365830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57580cf3f0 0x7f57581537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.198+0000 7f575cc00700 1 -- 172.21.15.89:0/3574365830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f575815f1e0 msgr2=0x7f5758099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.198+0000 7f575cc00700 1 --2- 172.21.15.89:0/3574365830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f575815f1e0 0x7f5758099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.198+0000 7f575cc00700 1 -- 172.21.15.89:0/3574365830 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57440094a0 con 0x7f57580cef40 2024-04-02T07:36:02.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.199+0000 7f77e3e00700 1 -- 172.21.15.89:0/3320974461 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc00e070 con 0x7f77e4160df0 2024-04-02T07:36:02.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.199+0000 7f77e3e00700 1 -- 172.21.15.89:0/3320974461 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77dc00bbd0 con 0x7f77e4160df0 2024-04-02T07:36:02.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.199+0000 7f575cc00700 1 --2- 172.21.15.89:0/3574365830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f57580cef40 0x7f5758159000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f574800c440 tx=0x7f574800c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.199+0000 7f5754c00700 1 -- 172.21.15.89:0/3574365830 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57480052d0 con 0x7f57580cef40 2024-04-02T07:36:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.199+0000 7f77e3e00700 1 -- 172.21.15.89:0/3320974461 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc00e070 con 0x7f77e4160df0 2024-04-02T07:36:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.200+0000 7f77eae00700 1 -- 172.21.15.89:0/3320974461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e4160df0 msgr2=0x7f77e406c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.200+0000 7f77eae00700 1 --2- 172.21.15.89:0/3320974461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e4160df0 0x7f77e406c870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009d90 tx=0x7f77dc00b330 comp rx=0 tx=0).stop 2024-04-02T07:36:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.201+0000 7f77eae00700 1 -- 172.21.15.89:0/3320974461 shutdown_connections 2024-04-02T07:36:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.201+0000 7f77eae00700 1 --2- 172.21.15.89:0/3320974461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e4160df0 0x7f77e406c870 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.201+0000 7f77eae00700 1 --2- 172.21.15.89:0/3320974461 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77e40b09d0 0x7f77e40d1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.201+0000 7f77eae00700 1 -- 172.21.15.89:0/3320974461 >> 172.21.15.89:0/3320974461 conn(0x7f77e409c720 msgr2=0x7f77e40d3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.200+0000 7f575f600700 1 -- 172.21.15.89:0/3574365830 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575815e9f0 con 0x7f57580cef40 2024-04-02T07:36:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.200+0000 7f5754c00700 1 -- 172.21.15.89:0/3574365830 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5748005430 con 0x7f57580cef40 2024-04-02T07:36:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.200+0000 7f5754c00700 1 -- 172.21.15.89:0/3574365830 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57480052d0 con 0x7f57580cef40 2024-04-02T07:36:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.201+0000 7f575f600700 1 -- 172.21.15.89:0/3574365830 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57580b4260 con 0x7f57580cef40 2024-04-02T07:36:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.200+0000 7f2f4dc00700 1 Processor -- start 2024-04-02T07:36:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.202+0000 7f2f4dc00700 1 -- start start 2024-04-02T07:36:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.203+0000 7f77eae00700 1 -- 172.21.15.89:0/3320974461 shutdown_connections 2024-04-02T07:36:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.203+0000 7f575f600700 1 -- 172.21.15.89:0/3574365830 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5758067720 con 0x7f57580cef40 2024-04-02T07:36:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.203+0000 7f77eae00700 1 -- 172.21.15.89:0/3320974461 wait complete. 2024-04-02T07:36:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.206+0000 7f77eae00700 1 Processor -- start 2024-04-02T07:36:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.206+0000 7f5754c00700 1 -- 172.21.15.89:0/3574365830 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5748006860 con 0x7f57580cef40 2024-04-02T07:36:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.206+0000 7f77eae00700 1 -- start start 2024-04-02T07:36:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.207+0000 7f2f4dc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f480cc370 0x7f2f480aca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.208+0000 7f2f4dc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f48156950 0x7f2f48155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.208+0000 7f2f4dc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f48156280 0x7f2f4809b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.208+0000 7f2f46a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f480cc370 0x7f2f480aca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.208+0000 7f5754c00700 1 --2- 172.21.15.89:0/3574365830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f573806f100 0x7f57380715c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.208+0000 7f5754c00700 1 -- 172.21.15.89:0/3574365830 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5748087f90 con 0x7f57580cef40 2024-04-02T07:36:02.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.208+0000 7f2f46a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f480cc370 0x7f2f480aca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52664/0 (socket says 172.21.15.89:52664) 2024-04-02T07:36:02.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.208+0000 7f2f46000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f48156950 0x7f2f48155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.208+0000 7f2f46000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f48156950 0x7f2f48155d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33680/0 (socket says 172.21.15.89:33680) 2024-04-02T07:36:02.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.209+0000 7f2f47400700 1 --2- 172.21.15.89:0/1405987689 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f48156280 0x7f2f4809b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.209+0000 7f77eae00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e40d1f70 0x7f77e40ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.209+0000 7f2f4dc00700 1 -- 172.21.15.89:0/1405987689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f480ad990 con 0x7f2f480cc370 2024-04-02T07:36:02.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.209+0000 7f2f4dc00700 1 -- 172.21.15.89:0/1405987689 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f480ad3d0 con 0x7f2f48156950 2024-04-02T07:36:02.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.209+0000 7f77eae00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77e4160df0 0x7f77e415c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.209+0000 7f5757e00700 1 --2- 172.21.15.89:0/3574365830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f573806f100 0x7f57380715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.209+0000 7f2f4dc00700 1 -- 172.21.15.89:0/1405987689 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f480ad6b0 con 0x7f2f48156280 2024-04-02T07:36:02.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.209+0000 7f77eae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77e41f4f80 0x7f77e40b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.210+0000 7f77e9a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e40d1f70 0x7f77e40ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.210+0000 7f77e9a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e40d1f70 0x7f77e40ccf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33692/0 (socket says 172.21.15.89:33692) 2024-04-02T07:36:02.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.210+0000 7f77e9a00700 1 -- 172.21.15.89:0/803088132 learned_addr learned my addr 172.21.15.89:0/803088132 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:02.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.210+0000 7f77eae00700 1 -- 172.21.15.89:0/803088132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40cd440 con 0x7f77e4160df0 2024-04-02T07:36:02.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.210+0000 7f77eae00700 1 -- 172.21.15.89:0/803088132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40b6070 con 0x7f77e40d1f70 2024-04-02T07:36:02.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.210+0000 7f77ea400700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77e41f4f80 0x7f77e40b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.210+0000 7f77e9000700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77e4160df0 0x7f77e415c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.210+0000 7f77eae00700 1 -- 172.21.15.89:0/803088132 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e415d1f0 con 0x7f77e41f4f80 2024-04-02T07:36:02.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.210+0000 7f5757e00700 1 --2- 172.21.15.89:0/3574365830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f573806f100 0x7f57380715c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f57580c83a0 tx=0x7f575000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.209+0000 7f2f46a00700 1 -- 172.21.15.89:0/1405987689 learned_addr learned my addr 172.21.15.89:0/1405987689 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.211+0000 7f77ea400700 1 -- 172.21.15.89:0/803088132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e40d1f70 msgr2=0x7f77e40ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.211+0000 7f77ea400700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e40d1f70 0x7f77e40ccf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.212+0000 7f77ea400700 1 -- 172.21.15.89:0/803088132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77e4160df0 msgr2=0x7f77e415c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.212+0000 7f77ea400700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77e4160df0 0x7f77e415c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.212+0000 7f77ea400700 1 -- 172.21.15.89:0/803088132 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77dc009a70 con 0x7f77e41f4f80 2024-04-02T07:36:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.211+0000 7f2f47400700 1 -- 172.21.15.89:0/1405987689 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f48156950 msgr2=0x7f2f48155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.211+0000 7f2f47400700 1 --2- 172.21.15.89:0/1405987689 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f48156950 0x7f2f48155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.211+0000 7f2f47400700 1 -- 172.21.15.89:0/1405987689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f480cc370 msgr2=0x7f2f480aca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.211+0000 7f2f47400700 1 --2- 172.21.15.89:0/1405987689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f480cc370 0x7f2f480aca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.212+0000 7f2f47400700 1 -- 172.21.15.89:0/1405987689 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f48099690 con 0x7f2f48156280 2024-04-02T07:36:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.212+0000 7f77e9a00700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e40d1f70 0x7f77e40ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.213+0000 7f77e9000700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77e4160df0 0x7f77e415c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.213+0000 7f5754c00700 1 -- 172.21.15.89:0/3574365830 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5748053160 con 0x7f57580cef40 2024-04-02T07:36:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.213+0000 7f2f47400700 1 --2- 172.21.15.89:0/1405987689 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f48156280 0x7f2f4809b9f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0047d0 tx=0x7f2f40001ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.213+0000 7f77ea400700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77e41f4f80 0x7f77e40b5080 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f77d0000f60 tx=0x7f77d000c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.213+0000 7f2f3ea00700 1 -- 172.21.15.89:0/1405987689 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4000a210 con 0x7f2f48156280 2024-04-02T07:36:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.213+0000 7f77e1600700 1 -- 172.21.15.89:0/803088132 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d00052d0 con 0x7f77e41f4f80 2024-04-02T07:36:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.214+0000 7f77e1600700 1 -- 172.21.15.89:0/803088132 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d0005430 con 0x7f77e41f4f80 2024-04-02T07:36:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.213+0000 7f2f3ea00700 1 -- 172.21.15.89:0/1405987689 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f40007b20 con 0x7f2f48156280 2024-04-02T07:36:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.214+0000 7f2f3ea00700 1 -- 172.21.15.89:0/1405987689 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4001f860 con 0x7f2f48156280 2024-04-02T07:36:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.214+0000 7f2f4dc00700 1 -- 172.21.15.89:0/1405987689 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f48099970 con 0x7f2f48156280 2024-04-02T07:36:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.214+0000 7f2f4dc00700 1 -- 172.21.15.89:0/1405987689 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f480ad180 con 0x7f2f48156280 2024-04-02T07:36:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.214+0000 7f77eae00700 1 -- 172.21.15.89:0/803088132 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e40b2940 con 0x7f77e41f4f80 2024-04-02T07:36:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.214+0000 7f77e1600700 1 -- 172.21.15.89:0/803088132 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d00052d0 con 0x7f77e41f4f80 2024-04-02T07:36:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.216+0000 7f77eae00700 1 -- 172.21.15.89:0/803088132 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e41f5760 con 0x7f77e41f4f80 2024-04-02T07:36:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.217+0000 7f77eae00700 1 -- 172.21.15.89:0/803088132 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e404ea80 con 0x7f77e41f4f80 2024-04-02T07:36:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.219+0000 7f2f4dc00700 1 -- 172.21.15.89:0/1405987689 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f200052f0 con 0x7f2f48156280 2024-04-02T07:36:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.221+0000 7f2f3ea00700 1 -- 172.21.15.89:0/1405987689 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2f4001f9c0 con 0x7f2f48156280 2024-04-02T07:36:02.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.219+0000 7f77e1600700 1 -- 172.21.15.89:0/803088132 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f77d0003680 con 0x7f77e41f4f80 2024-04-02T07:36:02.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.221+0000 7f77e1600700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f77c406acf0 0x7f77c406d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.221+0000 7f77e1600700 1 -- 172.21.15.89:0/803088132 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77d0088610 con 0x7f77e41f4f80 2024-04-02T07:36:02.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.222+0000 7f77e9a00700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f77c406acf0 0x7f77c406d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.222+0000 7f2f3ea00700 1 --2- 172.21.15.89:0/1405987689 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f2406ac90 0x7f2f2406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.223+0000 7f2f3ea00700 1 -- 172.21.15.89:0/1405987689 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f4008b5a0 con 0x7f2f48156280 2024-04-02T07:36:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.223+0000 7f2f46a00700 1 --2- 172.21.15.89:0/1405987689 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f2406ac90 0x7f2f2406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.223+0000 7f77e9a00700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f77c406acf0 0x7f77c406d1b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f77dc000f80 tx=0x7f77dc011000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.224+0000 7f2f46a00700 1 --2- 172.21.15.89:0/1405987689 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f2406ac90 0x7f2f2406d150 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2f480b3080 tx=0x7f2f380018a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.224+0000 7f2f3ea00700 1 -- 172.21.15.89:0/1405987689 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2f400568b0 con 0x7f2f48156280 2024-04-02T07:36:02.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.229+0000 7f77e1600700 1 -- 172.21.15.89:0/803088132 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f77d0053820 con 0x7f77e41f4f80 2024-04-02T07:36:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.281+0000 7fa0bd200700 1 Processor -- start 2024-04-02T07:36:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.281+0000 7fa0bd200700 1 -- start start 2024-04-02T07:36:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.283+0000 7fa0bd200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa0b80d2020 0x7fa0b80d0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.283+0000 7f5a27600700 1 Processor -- start 2024-04-02T07:36:02.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.283+0000 7fa0bd200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b80cf640 con 0x7fa0b80d1b70 2024-04-02T07:36:02.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.283+0000 7fa0bd200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b80ce5a0 con 0x7fa0b815d710 2024-04-02T07:36:02.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.283+0000 7fa0bd200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b80cf220 con 0x7fa0b80d2020 2024-04-02T07:36:02.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.283+0000 7fa0b5e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fa0b815d710 0x7fa0b80d12f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:39734/0 (socket says 172.21.15.89:39734) 2024-04-02T07:36:02.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.283+0000 7fa0b5e00700 1 -- 172.21.15.89:0/3012049079 learned_addr learned my addr 172.21.15.89:0/3012049079 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:02.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.284+0000 7fb38fe00700 1 Processor -- start 2024-04-02T07:36:02.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.284+0000 7fb38fe00700 1 -- start start 2024-04-02T07:36:02.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.284+0000 7f5a27600700 1 -- start start 2024-04-02T07:36:02.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.286+0000 7fb38fe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb38804fb70 0x7fb38804ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.286+0000 7fb38fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb38804f790 0x7fb3880df400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.286+0000 7fb38fe00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3880543d0 con 0x7fb3880532e0 2024-04-02T07:36:02.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.286+0000 7fb38fe00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb388050490 con 0x7fb38804f790 2024-04-02T07:36:02.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.286+0000 7fb38fe00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3880537c0 con 0x7fb38804fb70 2024-04-02T07:36:02.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.286+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941307800 0 0) 0x7fa0b80ce5a0 con 0x7fa0b815d710 2024-04-02T07:36:02.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.286+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa09c0035b0 con 0x7fa0b815d710 2024-04-02T07:36:02.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.286+0000 7fa0b6800700 1 --2- 172.21.15.89:0/3012049079 >> v2:172.21.15.89:3300/0 conn(0x7fa0b80d2020 0x7fa0b80d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.286+0000 7fb38ea00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb38804fb70 0x7fb38804ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.286+0000 7fb38ea00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb38804fb70 0x7fb38804ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33698/0 (socket says 172.21.15.89:33698) 2024-04-02T07:36:02.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.286+0000 7fb38ea00700 1 -- 172.21.15.89:0/3099749337 learned_addr learned my addr 172.21.15.89:0/3099749337 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:02.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.286+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1649565623 0 0) 0x7fa0b80cf640 con 0x7fa0b80d1b70 2024-04-02T07:36:02.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.286+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b80ce5a0 con 0x7fa0b80d1b70 2024-04-02T07:36:02.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.287+0000 7fb38e000700 1 --2- 172.21.15.89:0/3099749337 >> v2:172.21.15.89:3300/0 conn(0x7fb38804f790 0x7fb3880df400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.287+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 <== mon.1 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2939201831 0 0) 0x7fa09c0035b0 con 0x7fa0b815d710 2024-04-02T07:36:02.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.287+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 >> v2:172.21.15.89:3300/0 conn(0x7fa0b80d2020 msgr2=0x7fa0b80d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:02.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.287+0000 7f5a27600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5a2015d810 0x7f5a200b4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.287+0000 7fb38e000700 1 -- 172.21.15.89:0/3099749337 >> v2:172.21.15.113:3300/0 conn(0x7fb38804fb70 msgr2=0x7fb38804ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:02.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.287+0000 7fb38e000700 1 --2- 172.21.15.89:0/3099749337 >> v2:172.21.15.113:3300/0 conn(0x7fb38804fb70 0x7fb38804ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.287+0000 7fb38e000700 1 -- 172.21.15.89:0/3099749337 >> v1:172.21.15.113:6789/0 conn(0x7fb3880532e0 legacy=0x7fb3880e2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:02.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.287+0000 7fb38e000700 1 -- 172.21.15.89:0/3099749337 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb388053aa0 con 0x7fb38804f790 2024-04-02T07:36:02.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.287+0000 7f5a27600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5a20072910 0x7f5a200cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.288+0000 7f5a27600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a200cc140 con 0x7f5a2006e100 2024-04-02T07:36:02.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.288+0000 7f5a27600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a2015df80 con 0x7f5a2015d810 2024-04-02T07:36:02.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.288+0000 7f5a27600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2015e260 con 0x7f5a20072910 2024-04-02T07:36:02.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.288+0000 7fa972400700 1 Processor -- start 2024-04-02T07:36:02.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.288+0000 7fa972400700 1 -- start start 2024-04-02T07:36:02.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.288+0000 7fb38e000700 1 --2- 172.21.15.89:0/3099749337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb38804f790 0x7fb3880df400 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb38800d870 tx=0x7fb37c009d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f02b6369a2346e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.289+0000 7fb38d600700 1 -- 172.21.15.89:0/3099749337 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c004230 con 0x7fb38804f790 2024-04-02T07:36:02.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.289+0000 7fb38d600700 1 -- 172.21.15.89:0/3099749337 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37c004390 con 0x7fb38804f790 2024-04-02T07:36:02.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.289+0000 7fb38d600700 1 -- 172.21.15.89:0/3099749337 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c0036a0 con 0x7fb38804f790 2024-04-02T07:36:02.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.290+0000 7f5a25600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5a2006e100 0x7f5a200ce470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34918/0 (socket says 172.21.15.89:34918) 2024-04-02T07:36:02.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.290+0000 7fb38fe00700 1 -- 172.21.15.89:0/3099749337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb38804f790 msgr2=0x7fb3880df400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.290+0000 7fb38fe00700 1 --2- 172.21.15.89:0/3099749337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb38804f790 0x7fb3880df400 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb38800d870 tx=0x7fb37c009d80 comp rx=0 tx=0).stop 2024-04-02T07:36:02.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.290+0000 7f5a25600700 1 -- 172.21.15.89:0/2236741231 learned_addr learned my addr 172.21.15.89:0/2236741231 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:02.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.290+0000 7fb38fe00700 1 -- 172.21.15.89:0/3099749337 shutdown_connections 2024-04-02T07:36:02.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.290+0000 7fa972400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa96c098060 0x7fa96c0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.290+0000 7fa972400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa96c0cbd50 con 0x7fa96c072760 2024-04-02T07:36:02.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.290+0000 7fa972400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa96c0ce7d0 con 0x7fa96c0ceee0 2024-04-02T07:36:02.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.290+0000 7fa972400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa96c098440 con 0x7fa96c098060 2024-04-02T07:36:02.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.290+0000 7f5a24c00700 1 --2- 172.21.15.89:0/2236741231 >> v2:172.21.15.113:3300/0 conn(0x7f5a2015d810 0x7f5a200b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.290+0000 7fb38fe00700 1 --2- 172.21.15.89:0/3099749337 >> v2:172.21.15.113:3300/0 conn(0x7fb38804fb70 0x7fb38804ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.290+0000 7fb38fe00700 1 --2- 172.21.15.89:0/3099749337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb38804f790 0x7fb3880df400 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.290+0000 7fb38fe00700 1 -- 172.21.15.89:0/3099749337 >> 172.21.15.89:0/3099749337 conn(0x7fb388054b10 msgr2=0x7fb388054f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fb38fe00700 1 -- 172.21.15.89:0/3099749337 shutdown_connections 2024-04-02T07:36:02.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7f5a24c00700 1 -- 172.21.15.89:0/2236741231 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5a20072910 msgr2=0x7f5a200cc5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T07:36:02.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa96b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa96c098060 0x7fa96c0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa96aa00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa96c072760 0x7fa96c0cc1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34924/0 (socket says 172.21.15.89:34924) 2024-04-02T07:36:02.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa96aa00700 1 -- 172.21.15.89:0/4137833352 learned_addr learned my addr 172.21.15.89:0/4137833352 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:02.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7f5a17e00700 1 --2- 172.21.15.89:0/2236741231 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5a20072910 0x7f5a200cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa0b4a00700 1 --2- 172.21.15.89:0/3012049079 >> v2:172.21.15.89:3300/0 conn(0x7fa0b80d2020 0x7fa0b80d0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 >> v1:172.21.15.89:6789/0 conn(0x7fa0b80d1b70 legacy=0x7fa0b80cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b80ce880 con 0x7fa0b815d710 2024-04-02T07:36:02.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 <== mon.1 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fa0b0001670 con 0x7fa0b815d710 2024-04-02T07:36:02.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 >> v1:172.21.15.113:6789/0 conn(0x7fa0b815d710 legacy=0x7fa0b80d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa0b4a00700 1 --2- 172.21.15.89:0/3012049079 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa09c003b90 0x7fa09c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa0b4a00700 1 --2- 172.21.15.89:0/3012049079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa09c004850 0x7fa09c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa0b4a00700 1 --2- 172.21.15.89:0/3012049079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa09c005260 0x7fa09c014220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a0001280 con 0x7fa09c005260 2024-04-02T07:36:02.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b80cf220 con 0x7fa09c004850 2024-04-02T07:36:02.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c014a20 con 0x7fa09c003b90 2024-04-02T07:36:02.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa0b5e00700 1 --2- 172.21.15.89:0/3012049079 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa09c003b90 0x7fa09c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7fa0b5400700 1 --2- 172.21.15.89:0/3012049079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa09c004850 0x7fa09c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.292+0000 7fb38fe00700 1 -- 172.21.15.89:0/3099749337 wait complete. 2024-04-02T07:36:02.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7f5a24c00700 1 --2- 172.21.15.89:0/2236741231 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5a20072910 0x7f5a200cc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7f5a24c00700 1 -- 172.21.15.89:0/2236741231 >> v1:172.21.15.89:6789/0 conn(0x7f5a2006e100 legacy=0x7f5a200ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.291+0000 7f5a24c00700 1 -- 172.21.15.89:0/2236741231 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a200cbdd0 con 0x7f5a2015d810 2024-04-02T07:36:02.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.292+0000 7f5a17e00700 1 --2- 172.21.15.89:0/2236741231 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5a20072910 0x7f5a200cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:02.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.293+0000 7f5a24c00700 1 --2- 172.21.15.89:0/2236741231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a2015d810 0x7f5a200b4240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000aea0 tx=0x7f5a1000c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aedf8ed572b94ac8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.292+0000 7fa96a000700 1 -- 172.21.15.89:0/4137833352 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693944191 0 0) 0x7fa96c0cbd50 con 0x7fa96c072760 2024-04-02T07:36:02.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.292+0000 7fa96a000700 1 -- 172.21.15.89:0/4137833352 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9500035b0 con 0x7fa96c072760 2024-04-02T07:36:02.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.293+0000 7fa96a000700 1 -- 172.21.15.89:0/4137833352 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894976778 0 0) 0x7fa96c0ce7d0 con 0x7fa96c0ceee0 2024-04-02T07:36:02.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.293+0000 7fa96a000700 1 -- 172.21.15.89:0/4137833352 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa96c0cbd50 con 0x7fa96c0ceee0 2024-04-02T07:36:02.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.292+0000 7fa0b5e00700 1 -- 172.21.15.89:0/3012049079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa09c004850 msgr2=0x7fa09c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:02.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.292+0000 7fa0b5e00700 1 --2- 172.21.15.89:0/3012049079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa09c004850 0x7fa09c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.292+0000 7fa0b5e00700 1 -- 172.21.15.89:0/3012049079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa09c005260 msgr2=0x7fa09c014220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T07:36:02.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.292+0000 7fa0b5e00700 1 --2- 172.21.15.89:0/3012049079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa09c005260 0x7fa09c014220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.292+0000 7fa0b5e00700 1 -- 172.21.15.89:0/3012049079 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b80ce880 con 0x7fa09c003b90 2024-04-02T07:36:02.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.292+0000 7fa0b5400700 1 --2- 172.21.15.89:0/3012049079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa09c004850 0x7fa09c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:02.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.292+0000 7fa0b6800700 1 --2- 172.21.15.89:0/3012049079 >> v2:172.21.15.89:3300/0 conn(0x7fa0b80d2020 0x7fa0b80d0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:02.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.292+0000 7fa0b6800700 1 -- 172.21.15.89:0/3012049079 reap_dead start 2024-04-02T07:36:02.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.293+0000 7fa96b400700 1 -- 172.21.15.89:0/4137833352 >> v1:172.21.15.113:6789/0 conn(0x7fa96c0ceee0 legacy=0x7fa96c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.293+0000 7fa96b400700 1 -- 172.21.15.89:0/4137833352 >> v1:172.21.15.89:6789/0 conn(0x7fa96c072760 legacy=0x7fa96c0cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.293+0000 7fa96b400700 1 -- 172.21.15.89:0/4137833352 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c0ceab0 con 0x7fa96c098060 2024-04-02T07:36:02.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.293+0000 7fa0b5e00700 1 --2- 172.21.15.89:0/3012049079 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa09c003b90 0x7fa09c003f70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000c2f0 tx=0x7fa09c005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4216736f17907579 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.293+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0007c30 con 0x7fa09c003b90 2024-04-02T07:36:02.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.293+0000 7f5a1fe00700 1 -- 172.21.15.89:0/2236741231 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1000f070 con 0x7f5a2015d810 2024-04-02T07:36:02.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.293+0000 7fa0b4a00700 1 -- 172.21.15.89:0/3012049079 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b000cdd0 con 0x7fa09c003b90 2024-04-02T07:36:02.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.294+0000 7f5a1fe00700 1 -- 172.21.15.89:0/2236741231 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1000a2e0 con 0x7f5a2015d810 2024-04-02T07:36:02.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.294+0000 7f5a1fe00700 1 -- 172.21.15.89:0/2236741231 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1000cc00 con 0x7f5a2015d810 2024-04-02T07:36:02.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.293+0000 7fa96b400700 1 --2- 172.21.15.89:0/4137833352 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa96c098060 0x7fa96c0ce0d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa95c0098a0 tx=0x7fa95c009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6890d54257001e94 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.294+0000 7fa96a000700 1 -- 172.21.15.89:0/4137833352 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c011830 con 0x7fa96c098060 2024-04-02T07:36:02.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.294+0000 7fa96a000700 1 -- 172.21.15.89:0/4137833352 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95c011990 con 0x7fa96c098060 2024-04-02T07:36:02.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.294+0000 7fa96a000700 1 -- 172.21.15.89:0/4137833352 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c011830 con 0x7fa96c098060 2024-04-02T07:36:02.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.294+0000 7fa0bd200700 1 -- 172.21.15.89:0/3012049079 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa09c003b90 msgr2=0x7fa09c003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.294+0000 7fa0bd200700 1 --2- 172.21.15.89:0/3012049079 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa09c003b90 0x7fa09c003f70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000c2f0 tx=0x7fa09c005880 comp rx=0 tx=0).stop 2024-04-02T07:36:02.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.295+0000 7fb38fe00700 1 Processor -- start 2024-04-02T07:36:02.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.295+0000 7f5a27600700 1 -- 172.21.15.89:0/2236741231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a2015d810 msgr2=0x7f5a200b4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.295+0000 7f5a27600700 1 --2- 172.21.15.89:0/2236741231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a2015d810 0x7f5a200b4240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000aea0 tx=0x7f5a1000c330 comp rx=0 tx=0).stop 2024-04-02T07:36:02.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.296+0000 7fa0bd200700 1 -- 172.21.15.89:0/3012049079 shutdown_connections 2024-04-02T07:36:02.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.296+0000 7fa0bd200700 1 --2- 172.21.15.89:0/3012049079 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa09c003b90 0x7fa09c003f70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.296+0000 7fa0bd200700 1 -- 172.21.15.89:0/3012049079 >> 172.21.15.89:0/3012049079 conn(0x7fa0b808aff0 msgr2=0x7fa0b80d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.296+0000 7fb38fe00700 1 -- start start 2024-04-02T07:36:02.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.296+0000 7fa0bd200700 1 -- 172.21.15.89:0/3012049079 shutdown_connections 2024-04-02T07:36:02.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.296+0000 7fa0bd200700 1 -- 172.21.15.89:0/3012049079 wait complete. 2024-04-02T07:36:02.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.297+0000 7f5a27600700 1 -- 172.21.15.89:0/2236741231 shutdown_connections 2024-04-02T07:36:02.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.297+0000 7f5a27600700 1 --2- 172.21.15.89:0/2236741231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a2015d810 0x7f5a200b4240 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.297+0000 7f5a27600700 1 --2- 172.21.15.89:0/2236741231 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5a20072910 0x7f5a200cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.297+0000 7f5a27600700 1 -- 172.21.15.89:0/2236741231 >> 172.21.15.89:0/2236741231 conn(0x7f5a20070f00 msgr2=0x7f5a20071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.295+0000 7fa972400700 1 -- 172.21.15.89:0/4137833352 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa96c098060 msgr2=0x7fa96c0ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.295+0000 7fa972400700 1 --2- 172.21.15.89:0/4137833352 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa96c098060 0x7fa96c0ce0d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa95c0098a0 tx=0x7fa95c009bb0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.296+0000 7fa972400700 1 -- 172.21.15.89:0/4137833352 shutdown_connections 2024-04-02T07:36:02.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.296+0000 7fa972400700 1 --2- 172.21.15.89:0/4137833352 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa96c098060 0x7fa96c0ce0d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.296+0000 7fa972400700 1 -- 172.21.15.89:0/4137833352 >> 172.21.15.89:0/4137833352 conn(0x7fa96c070f00 msgr2=0x7fa96c071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.296+0000 7fa972400700 1 -- 172.21.15.89:0/4137833352 shutdown_connections 2024-04-02T07:36:02.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.296+0000 7fa972400700 1 -- 172.21.15.89:0/4137833352 wait complete. 2024-04-02T07:36:02.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.298+0000 7efd48800700 1 Processor -- start 2024-04-02T07:36:02.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.298+0000 7fb38fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb38804f790 0x7fb38817ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.298+0000 7efd48800700 1 -- start start 2024-04-02T07:36:02.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.298+0000 7fb38fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb38804fb70 0x7fb3880daee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.298+0000 7fb38fe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3880532e0 0x7fb38804d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.299+0000 7fb38fe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38817cf90 con 0x7fb38804f790 2024-04-02T07:36:02.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.299+0000 7fb38fe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38804e350 con 0x7fb3880532e0 2024-04-02T07:36:02.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.299+0000 7fb38fe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38804e630 con 0x7fb38804fb70 2024-04-02T07:36:02.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.299+0000 7fb38f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3880532e0 0x7fb38804d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.299+0000 7fb38f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3880532e0 0x7fb38804d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33730/0 (socket says 172.21.15.89:33730) 2024-04-02T07:36:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.299+0000 7fb38f400700 1 -- 172.21.15.89:0/2616683871 learned_addr learned my addr 172.21.15.89:0/2616683871 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.297+0000 7f5a27600700 1 -- 172.21.15.89:0/2236741231 shutdown_connections 2024-04-02T07:36:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.297+0000 7f5a27600700 1 -- 172.21.15.89:0/2236741231 wait complete. 2024-04-02T07:36:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.299+0000 7fa0bd200700 1 Processor -- start 2024-04-02T07:36:02.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.299+0000 7fb38e000700 1 --2- 172.21.15.89:0/2616683871 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb38804fb70 0x7fb3880daee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.299+0000 7fa0bd200700 1 -- start start 2024-04-02T07:36:02.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.299+0000 7fb38ea00700 1 --2- 172.21.15.89:0/2616683871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb38804f790 0x7fb38817ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.299+0000 7fb38f400700 1 -- 172.21.15.89:0/2616683871 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb38804fb70 msgr2=0x7fb3880daee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.299+0000 7fa972400700 1 Processor -- start 2024-04-02T07:36:02.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.300+0000 7efd48800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4015a5c0 0x7efd400709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.300+0000 7efd48800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efd400b2d50 0x7efd400cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.300+0000 7efd48800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd400cb310 con 0x7efd4015dfe0 2024-04-02T07:36:02.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.300+0000 7efd48800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd40070f00 con 0x7efd400b2d50 2024-04-02T07:36:02.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.300+0000 7efd48800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd400cca90 con 0x7efd4015a5c0 2024-04-02T07:36:02.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.300+0000 7fa972400700 1 -- start start 2024-04-02T07:36:02.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.299+0000 7fb38f400700 1 --2- 172.21.15.89:0/2616683871 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb38804fb70 0x7fb3880daee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.299+0000 7fb38f400700 1 -- 172.21.15.89:0/2616683871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb38804f790 msgr2=0x7fb38817ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.300+0000 7fb38f400700 1 --2- 172.21.15.89:0/2616683871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb38804f790 0x7fb38817ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.300+0000 7fb38f400700 1 -- 172.21.15.89:0/2616683871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37c0094a0 con 0x7fb3880532e0 2024-04-02T07:36:02.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.300+0000 7fb38f400700 1 --2- 172.21.15.89:0/2616683871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3880532e0 0x7fb38804d1d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb390062370 tx=0x7fb390068110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.300+0000 7fb387400700 1 -- 172.21.15.89:0/2616683871 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39008e070 con 0x7fb3880532e0 2024-04-02T07:36:02.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.300+0000 7fb387400700 1 -- 172.21.15.89:0/2616683871 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39006bcc0 con 0x7fb3880532e0 2024-04-02T07:36:02.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.300+0000 7fb387400700 1 -- 172.21.15.89:0/2616683871 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb390093680 con 0x7fb3880532e0 2024-04-02T07:36:02.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7fa0bd200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa09c003b90 0x7fa0b8157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7fa0bd200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0b815f030 0x7fa0b80b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7fa0bd200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0b815f4b0 0x7fa0b80b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7fa0bd200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b8157fb0 con 0x7fa0b815f030 2024-04-02T07:36:02.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7fa0bd200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b8160ab0 con 0x7fa0b815f4b0 2024-04-02T07:36:02.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7fa0bd200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b815fa10 con 0x7fa09c003b90 2024-04-02T07:36:02.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7efd45e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4015a5c0 0x7efd400709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7fb38fe00700 1 -- 172.21.15.89:0/2616683871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38804e910 con 0x7fb3880532e0 2024-04-02T07:36:02.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7fb38fe00700 1 -- 172.21.15.89:0/2616683871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3880489c0 con 0x7fb3880532e0 2024-04-02T07:36:02.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7f5a27600700 1 Processor -- start 2024-04-02T07:36:02.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7efd46800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efd400b2d50 0x7efd400cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7efd46800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efd400b2d50 0x7efd400cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33732/0 (socket says 172.21.15.89:33732) 2024-04-02T07:36:02.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7efd46800700 1 -- 172.21.15.89:0/1435785815 learned_addr learned my addr 172.21.15.89:0/1435785815 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:02.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7f5a27600700 1 -- start start 2024-04-02T07:36:02.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.301+0000 7fa0b5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0b815f030 0x7fa0b80b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.302+0000 7fa0b5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0b815f030 0x7fa0b80b3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52700/0 (socket says 172.21.15.89:52700) 2024-04-02T07:36:02.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.302+0000 7fa0b5400700 1 -- 172.21.15.89:0/3683713856 learned_addr learned my addr 172.21.15.89:0/3683713856 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:02.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.302+0000 7efd46800700 1 -- 172.21.15.89:0/1435785815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4015a5c0 msgr2=0x7efd400709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:02.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.302+0000 7efd46800700 1 --2- 172.21.15.89:0/1435785815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4015a5c0 0x7efd400709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.302+0000 7efd46800700 1 -- 172.21.15.89:0/1435785815 >> v1:172.21.15.113:6789/0 conn(0x7efd4015dfe0 legacy=0x7efd400b2640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T07:36:02.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.302+0000 7efd46800700 1 -- 172.21.15.89:0/1435785815 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd400711a0 con 0x7efd400b2d50 2024-04-02T07:36:02.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.302+0000 7efd45e00700 1 --2- 172.21.15.89:0/1435785815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4015a5c0 0x7efd400709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:02.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7efd46800700 1 --2- 172.21.15.89:0/1435785815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd400b2d50 0x7efd400cd020 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efd340098f0 tx=0x7efd34009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a48734c73f19cc7a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa0b5e00700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa09c003b90 0x7fa0b8157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa0b6800700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0b815f4b0 0x7fa0b80b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7f5a27600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a2006e100 0x7f5a2014eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.304+0000 7f5a27600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a20072910 0x7f5a200b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa972400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa96c072760 0x7fa96c164b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa972400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c098060 0x7fa96c085320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa972400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa96c0ceee0 0x7fa96c085860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa972400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96c1646f0 con 0x7fa96c098060 2024-04-02T07:36:02.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa972400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96c085da0 con 0x7fa96c0ceee0 2024-04-02T07:36:02.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa972400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa96c163ae0 con 0x7fa96c072760 2024-04-02T07:36:02.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa96aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c098060 0x7fa96c085320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.304+0000 7f5a27600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5a2015d810 0x7f5a201504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa0b5400700 1 -- 172.21.15.89:0/3683713856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa09c003b90 msgr2=0x7fa0b8157a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa0b5400700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa09c003b90 0x7fa0b8157a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa0b5400700 1 -- 172.21.15.89:0/3683713856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0b815f4b0 msgr2=0x7fa0b80b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa0b5400700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0b815f4b0 0x7fa0b80b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa0b5400700 1 -- 172.21.15.89:0/3683713856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b815fcf0 con 0x7fa0b815f030 2024-04-02T07:36:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa0b5e00700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa09c003b90 0x7fa0b8157a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.303+0000 7fa0b6800700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0b815f4b0 0x7fa0b80b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.304+0000 7efd44a00700 1 -- 172.21.15.89:0/1435785815 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd34004230 con 0x7efd400b2d50 2024-04-02T07:36:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.304+0000 7efd44a00700 1 -- 172.21.15.89:0/1435785815 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd34004390 con 0x7efd400b2d50 2024-04-02T07:36:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.304+0000 7efd44a00700 1 -- 172.21.15.89:0/1435785815 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340036a0 con 0x7efd400b2d50 2024-04-02T07:36:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.304+0000 7f5a27600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a20142b80 con 0x7f5a2006e100 2024-04-02T07:36:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.304+0000 7f5a27600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a20150010 con 0x7f5a20072910 2024-04-02T07:36:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.304+0000 7f5a27600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2014f400 con 0x7f5a2015d810 2024-04-02T07:36:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.304+0000 7f5a17e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a20072910 0x7f5a200b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7f5a24c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a2006e100 0x7f5a2014eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7f5a24c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a2006e100 0x7f5a2014eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52720/0 (socket says 172.21.15.89:52720) 2024-04-02T07:36:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7f5a17e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a20072910 0x7f5a200b0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33756/0 (socket says 172.21.15.89:33756) 2024-04-02T07:36:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7f5a17e00700 1 -- 172.21.15.89:0/11724399 learned_addr learned my addr 172.21.15.89:0/11724399 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7f5a25600700 1 --2- 172.21.15.89:0/11724399 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5a2015d810 0x7f5a201504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.304+0000 7fa96aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c098060 0x7fa96c085320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52716/0 (socket says 172.21.15.89:52716) 2024-04-02T07:36:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.304+0000 7fa96aa00700 1 -- 172.21.15.89:0/3899091133 learned_addr learned my addr 172.21.15.89:0/3899091133 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.304+0000 7fa96b400700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa96c072760 0x7fa96c164b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.304+0000 7fa96be00700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa96c0ceee0 0x7fa96c085860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7fa96aa00700 1 -- 172.21.15.89:0/3899091133 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa96c072760 msgr2=0x7fa96c164b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7fa96aa00700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa96c072760 0x7fa96c164b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7fa96aa00700 1 -- 172.21.15.89:0/3899091133 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa96c0ceee0 msgr2=0x7fa96c085860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7fa96aa00700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa96c0ceee0 0x7fa96c085860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7fa96aa00700 1 -- 172.21.15.89:0/3899091133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c0094a0 con 0x7fa96c098060 2024-04-02T07:36:02.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7efd48800700 1 -- 172.21.15.89:0/1435785815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd400b2d50 msgr2=0x7efd400cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7efd48800700 1 --2- 172.21.15.89:0/1435785815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd400b2d50 0x7efd400cd020 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efd340098f0 tx=0x7efd34009c00 comp rx=0 tx=0).stop 2024-04-02T07:36:02.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7fa96b400700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa96c072760 0x7fa96c164b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7efd48800700 1 -- 172.21.15.89:0/1435785815 shutdown_connections 2024-04-02T07:36:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7efd48800700 1 --2- 172.21.15.89:0/1435785815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd400b2d50 0x7efd400cd020 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7efd48800700 1 --2- 172.21.15.89:0/1435785815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4015a5c0 0x7efd400709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7fa96aa00700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c098060 0x7fa96c085320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa95800c700 tx=0x7fa95800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7fa0b5400700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0b815f030 0x7fa0b80b3f10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0000fb0 tx=0x7fa0a0007f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7efd48800700 1 -- 172.21.15.89:0/1435785815 >> 172.21.15.89:0/1435785815 conn(0x7efd400b0d70 msgr2=0x7efd400b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.306+0000 7fa0ae000700 1 -- 172.21.15.89:0/3683713856 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a00052e0 con 0x7fa0b815f030 2024-04-02T07:36:02.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.306+0000 7efd48800700 1 -- 172.21.15.89:0/1435785815 shutdown_connections 2024-04-02T07:36:02.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.306+0000 7fa0bd200700 1 -- 172.21.15.89:0/3683713856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b815ffd0 con 0x7fa0b815f030 2024-04-02T07:36:02.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.306+0000 7efd48800700 1 -- 172.21.15.89:0/1435785815 wait complete. 2024-04-02T07:36:02.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.306+0000 7fa0bd200700 1 -- 172.21.15.89:0/3683713856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b8160590 con 0x7fa0b815f030 2024-04-02T07:36:02.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.306+0000 7fa0ae000700 1 -- 172.21.15.89:0/3683713856 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a0005440 con 0x7fa0b815f030 2024-04-02T07:36:02.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.306+0000 7fa963400700 1 -- 172.21.15.89:0/3899091133 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa958005d70 con 0x7fa96c098060 2024-04-02T07:36:02.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.307+0000 7fa963400700 1 -- 172.21.15.89:0/3899091133 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9580070d0 con 0x7fa96c098060 2024-04-02T07:36:02.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.307+0000 7fa963400700 1 -- 172.21.15.89:0/3899091133 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa958007460 con 0x7fa96c098060 2024-04-02T07:36:02.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.305+0000 7fb38fe00700 1 -- 172.21.15.89:0/2616683871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3880053e0 con 0x7fb3880532e0 2024-04-02T07:36:02.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.307+0000 7fb387400700 1 -- 172.21.15.89:0/2616683871 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb3900937e0 con 0x7fb3880532e0 2024-04-02T07:36:02.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.306+0000 7f5a17e00700 1 -- 172.21.15.89:0/11724399 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5a2015d810 msgr2=0x7f5a201504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.306+0000 7f5a17e00700 1 --2- 172.21.15.89:0/11724399 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5a2015d810 0x7f5a201504a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.306+0000 7f5a17e00700 1 -- 172.21.15.89:0/11724399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a2006e100 msgr2=0x7f5a2014eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.306+0000 7f5a17e00700 1 --2- 172.21.15.89:0/11724399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a2006e100 0x7f5a2014eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.306+0000 7f5a17e00700 1 -- 172.21.15.89:0/11724399 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1000ab80 con 0x7f5a20072910 2024-04-02T07:36:02.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.309+0000 7f5a17e00700 1 --2- 172.21.15.89:0/11724399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a20072910 0x7f5a200b0e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800c750 tx=0x7f5a0800ca60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.309+0000 7efd48800700 1 Processor -- start 2024-04-02T07:36:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.309+0000 7fb387400700 1 --2- 172.21.15.89:0/2616683871 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb37006ad60 0x7fb37006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.309+0000 7fb387400700 1 -- 172.21.15.89:0/2616683871 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb390102b00 con 0x7fb3880532e0 2024-04-02T07:36:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.309+0000 7efd48800700 1 -- start start 2024-04-02T07:36:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.308+0000 7fa972400700 1 -- 172.21.15.89:0/3899091133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa96c163dc0 con 0x7fa96c098060 2024-04-02T07:36:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.310+0000 7fb38ea00700 1 --2- 172.21.15.89:0/2616683871 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb37006ad60 0x7fb37006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.312+0000 7f5a1d600700 1 -- 172.21.15.89:0/11724399 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08011800 con 0x7f5a20072910 2024-04-02T07:36:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.311+0000 7efd48800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd400b2d50 0x7efd401fac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.311+0000 7efd48800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd4015a5c0 0x7efd40149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.311+0000 7efd48800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4015dfe0 0x7efd40149980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.311+0000 7efd48800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd400ac1a0 con 0x7efd4015a5c0 2024-04-02T07:36:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.311+0000 7efd48800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd400ac440 con 0x7efd400b2d50 2024-04-02T07:36:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.311+0000 7efd48800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd400ac720 con 0x7efd4015dfe0 2024-04-02T07:36:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.311+0000 7efd45e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd400b2d50 0x7efd401fac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.311+0000 7efd45e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd400b2d50 0x7efd401fac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33772/0 (socket says 172.21.15.89:33772) 2024-04-02T07:36:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.311+0000 7efd45e00700 1 -- 172.21.15.89:0/25884785 learned_addr learned my addr 172.21.15.89:0/25884785 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.311+0000 7efd46800700 1 --2- 172.21.15.89:0/25884785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4015dfe0 0x7efd40149980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.312+0000 7efd45400700 1 --2- 172.21.15.89:0/25884785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd4015a5c0 0x7efd40149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.312+0000 7fb38ea00700 1 --2- 172.21.15.89:0/2616683871 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb37006ad60 0x7fb37006d220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb38000b240 tx=0x7fb38000b4d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.313+0000 7fb387400700 1 -- 172.21.15.89:0/2616683871 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb3900cdd50 con 0x7fb3880532e0 2024-04-02T07:36:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.312+0000 7f5a1d600700 1 -- 172.21.15.89:0/11724399 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a08011960 con 0x7f5a20072910 2024-04-02T07:36:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.312+0000 7f5a1d600700 1 -- 172.21.15.89:0/11724399 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08011cd0 con 0x7f5a20072910 2024-04-02T07:36:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.312+0000 7f5a27600700 1 -- 172.21.15.89:0/11724399 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2014f740 con 0x7f5a20072910 2024-04-02T07:36:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.312+0000 7f5a27600700 1 -- 172.21.15.89:0/11724399 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2014f060 con 0x7f5a20072910 2024-04-02T07:36:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.308+0000 7fa972400700 1 -- 172.21.15.89:0/3899091133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa96c164380 con 0x7fa96c098060 2024-04-02T07:36:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.311+0000 7fa972400700 1 -- 172.21.15.89:0/3899091133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa96c06b7d0 con 0x7fa96c098060 2024-04-02T07:36:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.308+0000 7fa0ae000700 1 -- 172.21.15.89:0/3683713856 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a000d960 con 0x7fa0b815f030 2024-04-02T07:36:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.310+0000 7fa0ae000700 1 -- 172.21.15.89:0/3683713856 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa0a000dac0 con 0x7fa0b815f030 2024-04-02T07:36:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.312+0000 7fa0ae000700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa09406ac90 0x7fa09406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.312+0000 7fa0ae000700 1 -- 172.21.15.89:0/3683713856 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0a0089680 con 0x7fa0b815f030 2024-04-02T07:36:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.313+0000 7fa0b5e00700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa09406ac90 0x7fa09406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.313+0000 7fa0bd200700 1 -- 172.21.15.89:0/3683713856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0900052f0 con 0x7fa0b815f030 2024-04-02T07:36:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.312+0000 7efd45e00700 1 -- 172.21.15.89:0/25884785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4015dfe0 msgr2=0x7efd40149980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.312+0000 7efd45e00700 1 --2- 172.21.15.89:0/25884785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4015dfe0 0x7efd40149980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.312+0000 7efd45e00700 1 -- 172.21.15.89:0/25884785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd4015a5c0 msgr2=0x7efd40149440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.312+0000 7efd45e00700 1 --2- 172.21.15.89:0/25884785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd4015a5c0 0x7efd40149440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.312+0000 7efd45e00700 1 -- 172.21.15.89:0/25884785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd300094a0 con 0x7efd400b2d50 2024-04-02T07:36:02.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.314+0000 7fa0b5e00700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa09406ac90 0x7fa09406d150 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0001120 tx=0x7fa0b0007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.314+0000 7fa963400700 1 -- 172.21.15.89:0/3899091133 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa9580075c0 con 0x7fa96c098060 2024-04-02T07:36:02.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.314+0000 7f5a27600700 1 -- 172.21.15.89:0/11724399 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f80052f0 con 0x7f5a20072910 2024-04-02T07:36:02.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.314+0000 7efd45e00700 1 --2- 172.21.15.89:0/25884785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd400b2d50 0x7efd401fac00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efd3000ebe0 tx=0x7efd3000eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.314+0000 7efd3a000700 1 -- 172.21.15.89:0/25884785 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3000c850 con 0x7efd400b2d50 2024-04-02T07:36:02.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.314+0000 7efd3a000700 1 -- 172.21.15.89:0/25884785 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3000c9b0 con 0x7efd400b2d50 2024-04-02T07:36:02.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.314+0000 7efd3a000700 1 -- 172.21.15.89:0/25884785 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3000cd40 con 0x7efd400b2d50 2024-04-02T07:36:02.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.315+0000 7fa963400700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa94c06ace0 0x7fa94c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.315+0000 7fa963400700 1 -- 172.21.15.89:0/3899091133 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9580887e0 con 0x7fa96c098060 2024-04-02T07:36:02.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.315+0000 7efd48800700 1 -- 172.21.15.89:0/25884785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd34009550 con 0x7efd400b2d50 2024-04-02T07:36:02.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.316+0000 7f5a1d600700 1 -- 172.21.15.89:0/11724399 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5a08021350 con 0x7f5a20072910 2024-04-02T07:36:02.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.317+0000 7efd48800700 1 -- 172.21.15.89:0/25884785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd400acce0 con 0x7efd400b2d50 2024-04-02T07:36:02.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.317+0000 7f5a1d600700 1 --2- 172.21.15.89:0/11724399 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f59fc06ace0 0x7f59fc06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.318+0000 7fa96b400700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa94c06ace0 0x7fa94c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.318+0000 7f5a1d600700 1 -- 172.21.15.89:0/11724399 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a08089150 con 0x7f5a20072910 2024-04-02T07:36:02.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.318+0000 7fa96b400700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa94c06ace0 0x7fa94c06d1a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009870 tx=0x7fa95c01c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.319+0000 7efd2f400700 1 -- 172.21.15.89:0/25884785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd40067720 con 0x7efd400b2d50 2024-04-02T07:36:02.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.319+0000 7f5a24c00700 1 --2- 172.21.15.89:0/11724399 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f59fc06ace0 0x7f59fc06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.320+0000 7f5a24c00700 1 --2- 172.21.15.89:0/11724399 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f59fc06ace0 0x7f59fc06d1a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5a10000f80 tx=0x7f5a1000a000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.320+0000 7efd3a000700 1 -- 172.21.15.89:0/25884785 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efd30013470 con 0x7efd400b2d50 2024-04-02T07:36:02.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.321+0000 7efd3a000700 1 --2- 172.21.15.89:0/25884785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efd2406ace0 0x7efd2406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:02.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.322+0000 7efd45400700 1 --2- 172.21.15.89:0/25884785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efd2406ace0 0x7efd2406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:02.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.322+0000 7efd3a000700 1 -- 172.21.15.89:0/25884785 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd300588e0 con 0x7efd400b2d50 2024-04-02T07:36:02.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.323+0000 7efd45400700 1 --2- 172.21.15.89:0/25884785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efd2406ace0 0x7efd2406d1a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efd400ce9e0 tx=0x7efd3c00d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:02.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.323+0000 7f5a1d600700 1 -- 172.21.15.89:0/11724399 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5a08054310 con 0x7f5a20072910 2024-04-02T07:36:02.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.327+0000 7efd3a000700 1 -- 172.21.15.89:0/25884785 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efd30054520 con 0x7efd400b2d50 2024-04-02T07:36:02.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.326+0000 7fa0ae000700 1 -- 172.21.15.89:0/3683713856 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa0a0054740 con 0x7fa0b815f030 2024-04-02T07:36:02.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.335+0000 7fa963400700 1 -- 172.21.15.89:0/3899091133 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa958053aa0 con 0x7fa96c098060 2024-04-02T07:36:02.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.668+0000 7f575f600700 1 -- 172.21.15.89:0/3574365830 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f57580d20b0 con 0x7f57580cef40 2024-04-02T07:36:02.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.669+0000 7f5754c00700 1 -- 172.21.15.89:0/3574365830 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5748056780 con 0x7f57580cef40 2024-04-02T07:36:02.672 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-04-02T07:36:02.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.673+0000 7f575f600700 1 -- 172.21.15.89:0/3574365830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f573806f100 msgr2=0x7f57380715c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.673+0000 7f575f600700 1 --2- 172.21.15.89:0/3574365830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f573806f100 0x7f57380715c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f57580c83a0 tx=0x7f575000b040 comp rx=0 tx=0).stop 2024-04-02T07:36:02.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.674+0000 7f575f600700 1 -- 172.21.15.89:0/3574365830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f57580cef40 msgr2=0x7f5758159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.674+0000 7f575f600700 1 --2- 172.21.15.89:0/3574365830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f57580cef40 0x7f5758159000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f574800c440 tx=0x7f574800c750 comp rx=0 tx=0).stop 2024-04-02T07:36:02.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.675+0000 7f575f600700 1 -- 172.21.15.89:0/3574365830 shutdown_connections 2024-04-02T07:36:02.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.675+0000 7f575f600700 1 --2- 172.21.15.89:0/3574365830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f573806f100 0x7f57380715c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.675+0000 7f575f600700 1 --2- 172.21.15.89:0/3574365830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f575815f1e0 0x7f5758099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.675+0000 7f575f600700 1 --2- 172.21.15.89:0/3574365830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f57580cef40 0x7f5758159000 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.675+0000 7f575f600700 1 --2- 172.21.15.89:0/3574365830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57580cf3f0 0x7f57581537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.675+0000 7f575f600700 1 -- 172.21.15.89:0/3574365830 >> 172.21.15.89:0/3574365830 conn(0x7f57580d1370 msgr2=0x7f575809a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.676+0000 7f575f600700 1 -- 172.21.15.89:0/3574365830 shutdown_connections 2024-04-02T07:36:02.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.676+0000 7f575f600700 1 -- 172.21.15.89:0/3574365830 wait complete. 2024-04-02T07:36:02.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.695+0000 7f2f4dc00700 1 -- 172.21.15.89:0/1405987689 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2f20005870 con 0x7f2f48156280 2024-04-02T07:36:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.696+0000 7f2f3ea00700 1 -- 172.21.15.89:0/1405987689 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2f40017020 con 0x7f2f48156280 2024-04-02T07:36:02.698 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-04-02T07:36:02.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.700+0000 7f2f4dc00700 1 -- 172.21.15.89:0/1405987689 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f2406ac90 msgr2=0x7f2f2406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.700+0000 7f2f4dc00700 1 --2- 172.21.15.89:0/1405987689 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f2406ac90 0x7f2f2406d150 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2f480b3080 tx=0x7f2f380018a0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.700+0000 7f2f4dc00700 1 -- 172.21.15.89:0/1405987689 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f48156280 msgr2=0x7f2f4809b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.700+0000 7f2f4dc00700 1 --2- 172.21.15.89:0/1405987689 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f48156280 0x7f2f4809b9f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0047d0 tx=0x7f2f40001ad0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.702+0000 7f2f4dc00700 1 -- 172.21.15.89:0/1405987689 shutdown_connections 2024-04-02T07:36:02.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.702+0000 7f2f4dc00700 1 --2- 172.21.15.89:0/1405987689 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f2406ac90 0x7f2f2406d150 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.702+0000 7f2f4dc00700 1 --2- 172.21.15.89:0/1405987689 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f48156280 0x7f2f4809b9f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.702+0000 7f2f4dc00700 1 --2- 172.21.15.89:0/1405987689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f480cc370 0x7f2f480aca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.702+0000 7f2f4dc00700 1 --2- 172.21.15.89:0/1405987689 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f48156950 0x7f2f48155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.702+0000 7f2f4dc00700 1 -- 172.21.15.89:0/1405987689 >> 172.21.15.89:0/1405987689 conn(0x7f2f480b0d70 msgr2=0x7f2f480ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.702+0000 7f2f4dc00700 1 -- 172.21.15.89:0/1405987689 shutdown_connections 2024-04-02T07:36:02.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.702+0000 7f2f4dc00700 1 -- 172.21.15.89:0/1405987689 wait complete. 2024-04-02T07:36:02.711 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.2 2024-04-02T07:36:02.711 DEBUG:teuthology.parallel:result is None 2024-04-02T07:36:02.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.710+0000 7f77eae00700 1 -- 172.21.15.89:0/803088132 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f77e40b2790 con 0x7f77e41f4f80 2024-04-02T07:36:02.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.711+0000 7f77e1600700 1 -- 172.21.15.89:0/803088132 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f77d0056e40 con 0x7f77e41f4f80 2024-04-02T07:36:02.714 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2024-04-02T07:36:02.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.716+0000 7f77eae00700 1 -- 172.21.15.89:0/803088132 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f77c406acf0 msgr2=0x7f77c406d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.716+0000 7f77eae00700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f77c406acf0 0x7f77c406d1b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f77dc000f80 tx=0x7f77dc011000 comp rx=0 tx=0).stop 2024-04-02T07:36:02.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.716+0000 7f77eae00700 1 -- 172.21.15.89:0/803088132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77e41f4f80 msgr2=0x7f77e40b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.716+0000 7f77eae00700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77e41f4f80 0x7f77e40b5080 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f77d0000f60 tx=0x7f77d000c6e0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.717+0000 7f77eae00700 1 -- 172.21.15.89:0/803088132 shutdown_connections 2024-04-02T07:36:02.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.717+0000 7f77eae00700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f77c406acf0 0x7f77c406d1b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.717+0000 7f77eae00700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e40d1f70 0x7f77e40ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.717+0000 7f77eae00700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77e41f4f80 0x7f77e40b5080 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.717+0000 7f77eae00700 1 --2- 172.21.15.89:0/803088132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77e4160df0 0x7f77e415c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.717+0000 7f77eae00700 1 -- 172.21.15.89:0/803088132 >> 172.21.15.89:0/803088132 conn(0x7f77e409c720 msgr2=0x7f77e40d33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.717+0000 7f77eae00700 1 -- 172.21.15.89:0/803088132 shutdown_connections 2024-04-02T07:36:02.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.717+0000 7f77eae00700 1 -- 172.21.15.89:0/803088132 wait complete. 2024-04-02T07:36:02.730 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.3 2024-04-02T07:36:02.730 DEBUG:teuthology.parallel:result is None 2024-04-02T07:36:02.747 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2024-04-02T07:36:02.748 DEBUG:teuthology.parallel:result is None 2024-04-02T07:36:02.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.758+0000 7fa972400700 1 -- 172.21.15.89:0/3899091133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa96c0b2050 con 0x7fa96c098060 2024-04-02T07:36:02.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.758+0000 7fa963400700 1 -- 172.21.15.89:0/3899091133 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa9580570c0 con 0x7fa96c098060 2024-04-02T07:36:02.760 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2024-04-02T07:36:02.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.761+0000 7fa972400700 1 -- 172.21.15.89:0/3899091133 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa94c06ace0 msgr2=0x7fa94c06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.761+0000 7fa972400700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa94c06ace0 0x7fa94c06d1a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009870 tx=0x7fa95c01c040 comp rx=0 tx=0).stop 2024-04-02T07:36:02.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.761+0000 7fa972400700 1 -- 172.21.15.89:0/3899091133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c098060 msgr2=0x7fa96c085320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.761+0000 7fa972400700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c098060 0x7fa96c085320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa95800c700 tx=0x7fa95800ca10 comp rx=0 tx=0).stop 2024-04-02T07:36:02.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.761+0000 7fa972400700 1 -- 172.21.15.89:0/3899091133 shutdown_connections 2024-04-02T07:36:02.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.761+0000 7fa972400700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa94c06ace0 0x7fa94c06d1a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.761+0000 7fa972400700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa96c0ceee0 0x7fa96c085860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.761+0000 7fa972400700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa96c072760 0x7fa96c164b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.761+0000 7fa972400700 1 --2- 172.21.15.89:0/3899091133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c098060 0x7fa96c085320 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.761+0000 7fa972400700 1 -- 172.21.15.89:0/3899091133 >> 172.21.15.89:0/3899091133 conn(0x7fa96c070f00 msgr2=0x7fa96c0961d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.761+0000 7fa972400700 1 -- 172.21.15.89:0/3899091133 shutdown_connections 2024-04-02T07:36:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.762+0000 7fa972400700 1 -- 172.21.15.89:0/3899091133 wait complete. 2024-04-02T07:36:02.781 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2024-04-02T07:36:02.781 DEBUG:teuthology.parallel:result is None 2024-04-02T07:36:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.780+0000 7fb38fe00700 1 -- 172.21.15.89:0/2616683871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb388052c80 con 0x7fb3880532e0 2024-04-02T07:36:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.780+0000 7fb387400700 1 -- 172.21.15.89:0/2616683871 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb3900cdd50 con 0x7fb3880532e0 2024-04-02T07:36:02.782 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2024-04-02T07:36:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.783+0000 7fb38fe00700 1 -- 172.21.15.89:0/2616683871 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb37006ad60 msgr2=0x7fb37006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.783+0000 7fb38fe00700 1 --2- 172.21.15.89:0/2616683871 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb37006ad60 0x7fb37006d220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb38000b240 tx=0x7fb38000b4d0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.783+0000 7fb38fe00700 1 -- 172.21.15.89:0/2616683871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3880532e0 msgr2=0x7fb38804d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.783+0000 7fb38fe00700 1 --2- 172.21.15.89:0/2616683871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3880532e0 0x7fb38804d1d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb390062370 tx=0x7fb390068110 comp rx=0 tx=0).stop 2024-04-02T07:36:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.783+0000 7fb38fe00700 1 -- 172.21.15.89:0/2616683871 shutdown_connections 2024-04-02T07:36:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.783+0000 7fb38fe00700 1 --2- 172.21.15.89:0/2616683871 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb37006ad60 0x7fb37006d220 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.783+0000 7fb38fe00700 1 --2- 172.21.15.89:0/2616683871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3880532e0 0x7fb38804d1d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.783+0000 7fb38fe00700 1 --2- 172.21.15.89:0/2616683871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb38804f790 0x7fb38817ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.783+0000 7fb38fe00700 1 --2- 172.21.15.89:0/2616683871 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb38804fb70 0x7fb3880daee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.783+0000 7fb38fe00700 1 -- 172.21.15.89:0/2616683871 >> 172.21.15.89:0/2616683871 conn(0x7fb388054b10 msgr2=0x7fb3880e1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.783+0000 7fb38fe00700 1 -- 172.21.15.89:0/2616683871 shutdown_connections 2024-04-02T07:36:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.783+0000 7fb38fe00700 1 -- 172.21.15.89:0/2616683871 wait complete. 2024-04-02T07:36:02.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.789+0000 7f5a27600700 1 -- 172.21.15.89:0/11724399 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f59f80055a0 con 0x7f5a20072910 2024-04-02T07:36:02.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.790+0000 7f5a1d600700 1 -- 172.21.15.89:0/11724399 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5a08057930 con 0x7f5a20072910 2024-04-02T07:36:02.791 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-04-02T07:36:02.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.790+0000 7fa0bd200700 1 -- 172.21.15.89:0/3683713856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa0900055a0 con 0x7fa0b815f030 2024-04-02T07:36:02.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.791+0000 7fa0ae000700 1 -- 172.21.15.89:0/3683713856 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa0a0057d60 con 0x7fa0b815f030 2024-04-02T07:36:02.792 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-04-02T07:36:02.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.793+0000 7f5a27600700 1 -- 172.21.15.89:0/11724399 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f59fc06ace0 msgr2=0x7f59fc06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.793+0000 7f5a27600700 1 --2- 172.21.15.89:0/11724399 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f59fc06ace0 0x7f59fc06d1a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5a10000f80 tx=0x7f5a1000a000 comp rx=0 tx=0).stop 2024-04-02T07:36:02.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.793+0000 7f5a27600700 1 -- 172.21.15.89:0/11724399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a20072910 msgr2=0x7f5a200b0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.793+0000 7f5a27600700 1 --2- 172.21.15.89:0/11724399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a20072910 0x7f5a200b0e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800c750 tx=0x7f5a0800ca60 comp rx=0 tx=0).stop 2024-04-02T07:36:02.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.793+0000 7fa0bd200700 1 -- 172.21.15.89:0/3683713856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa09406ac90 msgr2=0x7fa09406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.793+0000 7fa0bd200700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa09406ac90 0x7fa09406d150 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0001120 tx=0x7fa0b0007000 comp rx=0 tx=0).stop 2024-04-02T07:36:02.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.793+0000 7fa0bd200700 1 -- 172.21.15.89:0/3683713856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0b815f030 msgr2=0x7fa0b80b3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.793+0000 7fa0bd200700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0b815f030 0x7fa0b80b3f10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0000fb0 tx=0x7fa0a0007f80 comp rx=0 tx=0).stop 2024-04-02T07:36:02.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7f5a27600700 1 -- 172.21.15.89:0/11724399 shutdown_connections 2024-04-02T07:36:02.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7fa0bd200700 1 -- 172.21.15.89:0/3683713856 shutdown_connections 2024-04-02T07:36:02.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7fa0bd200700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa09406ac90 0x7fa09406d150 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7fa0bd200700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0b815f4b0 0x7fa0b80b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7fa0bd200700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa09c003b90 0x7fa0b8157a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7fa0bd200700 1 --2- 172.21.15.89:0/3683713856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0b815f030 0x7fa0b80b3f10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7fa0bd200700 1 -- 172.21.15.89:0/3683713856 >> 172.21.15.89:0/3683713856 conn(0x7fa0b808aff0 msgr2=0x7fa0b80b3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7f5a27600700 1 --2- 172.21.15.89:0/11724399 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f59fc06ace0 0x7f59fc06d1a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7fa0bd200700 1 -- 172.21.15.89:0/3683713856 shutdown_connections 2024-04-02T07:36:02.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7f5a27600700 1 --2- 172.21.15.89:0/11724399 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5a2015d810 0x7f5a201504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7f5a27600700 1 --2- 172.21.15.89:0/11724399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a2006e100 0x7f5a2014eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7f5a27600700 1 --2- 172.21.15.89:0/11724399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a20072910 0x7f5a200b0e50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7f5a27600700 1 -- 172.21.15.89:0/11724399 >> 172.21.15.89:0/11724399 conn(0x7f5a20070f00 msgr2=0x7f5a200b3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7f5a27600700 1 -- 172.21.15.89:0/11724399 shutdown_connections 2024-04-02T07:36:02.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7f5a27600700 1 -- 172.21.15.89:0/11724399 wait complete. 2024-04-02T07:36:02.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.794+0000 7fa0bd200700 1 -- 172.21.15.89:0/3683713856 wait complete. 2024-04-02T07:36:02.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.798+0000 7efd2f400700 1 -- 172.21.15.89:0/25884785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7efd400ac580 con 0x7efd400b2d50 2024-04-02T07:36:02.801 INFO:teuthology.orchestra.run.smithi089.stdout:55834574856 2024-04-02T07:36:02.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.799+0000 7efd3a000700 1 -- 172.21.15.89:0/25884785 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efd30021020 con 0x7efd400b2d50 2024-04-02T07:36:02.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.803+0000 7efd2f400700 1 -- 172.21.15.89:0/25884785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efd2406ace0 msgr2=0x7efd2406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.803+0000 7efd2f400700 1 --2- 172.21.15.89:0/25884785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efd2406ace0 0x7efd2406d1a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efd400ce9e0 tx=0x7efd3c00d040 comp rx=0 tx=0).stop 2024-04-02T07:36:02.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.803+0000 7efd2f400700 1 -- 172.21.15.89:0/25884785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd400b2d50 msgr2=0x7efd401fac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:02.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.803+0000 7efd2f400700 1 --2- 172.21.15.89:0/25884785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd400b2d50 0x7efd401fac00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efd3000ebe0 tx=0x7efd3000eef0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.803+0000 7efd2f400700 1 -- 172.21.15.89:0/25884785 shutdown_connections 2024-04-02T07:36:02.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.803+0000 7efd2f400700 1 --2- 172.21.15.89:0/25884785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efd2406ace0 0x7efd2406d1a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.803+0000 7efd2f400700 1 --2- 172.21.15.89:0/25884785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd400b2d50 0x7efd401fac00 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efd3000ebe0 tx=0x7efd3000eef0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.803+0000 7efd2f400700 1 --2- 172.21.15.89:0/25884785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd4015dfe0 0x7efd40149980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.803+0000 7efd2f400700 1 --2- 172.21.15.89:0/25884785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd4015a5c0 0x7efd40149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:02.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.803+0000 7efd2f400700 1 -- 172.21.15.89:0/25884785 >> 172.21.15.89:0/25884785 conn(0x7efd400b0d70 msgr2=0x7efd4006d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:02.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.803+0000 7efd2f400700 1 -- 172.21.15.89:0/25884785 shutdown_connections 2024-04-02T07:36:02.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:02.803+0000 7efd2f400700 1 -- 172.21.15.89:0/25884785 wait complete. 2024-04-02T07:36:02.810 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2024-04-02T07:36:02.810 DEBUG:teuthology.parallel:result is None 2024-04-02T07:36:02.820 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.6 2024-04-02T07:36:02.820 DEBUG:teuthology.parallel:result is None 2024-04-02T07:36:02.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2024-04-02T07:36:02.826 DEBUG:teuthology.parallel:result is None 2024-04-02T07:36:02.831 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2024-04-02T07:36:02.831 DEBUG:teuthology.parallel:result is None 2024-04-02T07:36:02.831 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-02T07:36:02.831 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:36:03.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.060+0000 7fd532e00700 1 Processor -- start 2024-04-02T07:36:03.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.060+0000 7fd532e00700 1 -- start start 2024-04-02T07:36:03.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.061+0000 7fd532e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd52c15d640 0x7fd52c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:03.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.061+0000 7fd532e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd52c0d30d0 con 0x7fd52c0ceec0 2024-04-02T07:36:03.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.061+0000 7fd532e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd52c0b6840 con 0x7fd52c1619d0 2024-04-02T07:36:03.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.061+0000 7fd532e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd52c0d2880 con 0x7fd52c15d640 2024-04-02T07:36:03.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.061+0000 7fd52be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd52c15d640 0x7fd52c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:03.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.061+0000 7fd52be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd52c15d640 0x7fd52c0d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52726/0 (socket says 172.21.15.89:52726) 2024-04-02T07:36:03.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.061+0000 7fd52be00700 1 -- 172.21.15.89:0/578982810 learned_addr learned my addr 172.21.15.89:0/578982810 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:03.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.061+0000 7fd52b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd52c0ceec0 0x7fd52c1612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34936/0 (socket says 172.21.15.89:34936) 2024-04-02T07:36:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.062+0000 7fd52be00700 1 -- 172.21.15.89:0/578982810 >> v1:172.21.15.113:6789/0 conn(0x7fd52c1619d0 legacy=0x7fd52c0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.062+0000 7fd52be00700 1 -- 172.21.15.89:0/578982810 >> v1:172.21.15.89:6789/0 conn(0x7fd52c0ceec0 legacy=0x7fd52c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.062+0000 7fd52be00700 1 -- 172.21.15.89:0/578982810 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52c0d2b30 con 0x7fd52c15d640 2024-04-02T07:36:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.062+0000 7fd52be00700 1 --2- 172.21.15.89:0/578982810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd52c15d640 0x7fd52c0d2340 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd51c0098f0 tx=0x7fd51c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54950dceb96449f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.062+0000 7fd52aa00700 1 -- 172.21.15.89:0/578982810 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c004230 con 0x7fd52c15d640 2024-04-02T07:36:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.062+0000 7fd52aa00700 1 -- 172.21.15.89:0/578982810 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51c004390 con 0x7fd52c15d640 2024-04-02T07:36:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.062+0000 7fd52aa00700 1 -- 172.21.15.89:0/578982810 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c0036a0 con 0x7fd52c15d640 2024-04-02T07:36:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.063+0000 7fd532e00700 1 -- 172.21.15.89:0/578982810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd52c15d640 msgr2=0x7fd52c0d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.063+0000 7fd532e00700 1 --2- 172.21.15.89:0/578982810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd52c15d640 0x7fd52c0d2340 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd51c0098f0 tx=0x7fd51c009c00 comp rx=0 tx=0).stop 2024-04-02T07:36:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.063+0000 7fd532e00700 1 -- 172.21.15.89:0/578982810 shutdown_connections 2024-04-02T07:36:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.063+0000 7fd532e00700 1 --2- 172.21.15.89:0/578982810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd52c15d640 0x7fd52c0d2340 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.063+0000 7fd532e00700 1 -- 172.21.15.89:0/578982810 >> 172.21.15.89:0/578982810 conn(0x7fd52c08aff0 msgr2=0x7fd52c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.063+0000 7fd532e00700 1 -- 172.21.15.89:0/578982810 shutdown_connections 2024-04-02T07:36:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.063+0000 7fd532e00700 1 -- 172.21.15.89:0/578982810 wait complete. 2024-04-02T07:36:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.065+0000 7fd532e00700 1 Processor -- start 2024-04-02T07:36:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.065+0000 7fd532e00700 1 -- start start 2024-04-02T07:36:03.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd532e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd52c0ceec0 0x7fd52c14a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:03.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd532e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd52c15d640 0x7fd52c14d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd532e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd52c1619d0 0x7fd52c1f65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd532e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c14ecf0 con 0x7fd52c0ceec0 2024-04-02T07:36:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd532e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c14e0e0 con 0x7fd52c1619d0 2024-04-02T07:36:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd532e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c14e3c0 con 0x7fd52c15d640 2024-04-02T07:36:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd52be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd52c0ceec0 0x7fd52c14a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd52be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd52c0ceec0 0x7fd52c14a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52736/0 (socket says 172.21.15.89:52736) 2024-04-02T07:36:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd52be00700 1 -- 172.21.15.89:0/1921971680 learned_addr learned my addr 172.21.15.89:0/1921971680 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd52be00700 1 -- 172.21.15.89:0/1921971680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd52c15d640 msgr2=0x7fd52c14d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd52be00700 1 --2- 172.21.15.89:0/1921971680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd52c15d640 0x7fd52c14d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd52be00700 1 -- 172.21.15.89:0/1921971680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd52c1619d0 msgr2=0x7fd52c1f65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd52be00700 1 --2- 172.21.15.89:0/1921971680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd52c1619d0 0x7fd52c1f65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd52be00700 1 -- 172.21.15.89:0/1921971680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c009550 con 0x7fd52c0ceec0 2024-04-02T07:36:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.066+0000 7fd52be00700 1 --2- 172.21.15.89:0/1921971680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd52c0ceec0 0x7fd52c14a3a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009130 tx=0x7fd51c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.067+0000 7fd523e00700 1 -- 172.21.15.89:0/1921971680 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c011630 con 0x7fd52c0ceec0 2024-04-02T07:36:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.067+0000 7fd523e00700 1 -- 172.21.15.89:0/1921971680 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51c0117b0 con 0x7fd52c0ceec0 2024-04-02T07:36:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.067+0000 7fd532e00700 1 -- 172.21.15.89:0/1921971680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52c14e6a0 con 0x7fd52c0ceec0 2024-04-02T07:36:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.067+0000 7fd523e00700 1 -- 172.21.15.89:0/1921971680 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c011b60 con 0x7fd52c0ceec0 2024-04-02T07:36:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.067+0000 7fd532e00700 1 -- 172.21.15.89:0/1921971680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd52c14a8e0 con 0x7fd52c0ceec0 2024-04-02T07:36:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.068+0000 7fd532e00700 1 -- 172.21.15.89:0/1921971680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd52c067720 con 0x7fd52c0ceec0 2024-04-02T07:36:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.069+0000 7fd523e00700 1 -- 172.21.15.89:0/1921971680 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd51c022070 con 0x7fd52c0ceec0 2024-04-02T07:36:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.070+0000 7fd523e00700 1 --2- 172.21.15.89:0/1921971680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd50c06ac90 0x7fd50c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.070+0000 7fd523e00700 1 -- 172.21.15.89:0/1921971680 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd51c08b370 con 0x7fd52c0ceec0 2024-04-02T07:36:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.070+0000 7fd52b400700 1 --2- 172.21.15.89:0/1921971680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd50c06ac90 0x7fd50c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.071+0000 7fd52b400700 1 --2- 172.21.15.89:0/1921971680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd50c06ac90 0x7fd50c06d150 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd52c14d4c0 tx=0x7fd518007a30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.075+0000 7fd523e00700 1 -- 172.21.15.89:0/1921971680 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd51c056700 con 0x7fd52c0ceec0 2024-04-02T07:36:03.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.272+0000 7fd532e00700 1 -- 172.21.15.89:0/1921971680 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd52c0b64d0 con 0x7fd50c06ac90 2024-04-02T07:36:03.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.280+0000 7fd523e00700 1 -- 172.21.15.89:0/1921971680 <== mgr.4108 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264490 (secure 0 0 0) 0x7fd5180014a0 con 0x7fd50c06ac90 2024-04-02T07:36:03.287 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:03.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.284+0000 7fd532e00700 1 -- 172.21.15.89:0/1921971680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd50c06ac90 msgr2=0x7fd50c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:03.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.284+0000 7fd532e00700 1 --2- 172.21.15.89:0/1921971680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd50c06ac90 0x7fd50c06d150 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd52c14d4c0 tx=0x7fd518007a30 comp rx=0 tx=0).stop 2024-04-02T07:36:03.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.284+0000 7fd532e00700 1 -- 172.21.15.89:0/1921971680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd52c0ceec0 msgr2=0x7fd52c14a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.284+0000 7fd532e00700 1 --2- 172.21.15.89:0/1921971680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd52c0ceec0 0x7fd52c14a3a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009130 tx=0x7fd51c004750 comp rx=0 tx=0).stop 2024-04-02T07:36:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.284+0000 7fd532e00700 1 -- 172.21.15.89:0/1921971680 shutdown_connections 2024-04-02T07:36:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.284+0000 7fd532e00700 1 --2- 172.21.15.89:0/1921971680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd50c06ac90 0x7fd50c06d150 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.284+0000 7fd532e00700 1 --2- 172.21.15.89:0/1921971680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd52c1619d0 0x7fd52c1f65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.284+0000 7fd532e00700 1 --2- 172.21.15.89:0/1921971680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd52c0ceec0 0x7fd52c14a3a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.284+0000 7fd532e00700 1 --2- 172.21.15.89:0/1921971680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd52c15d640 0x7fd52c14d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.284+0000 7fd532e00700 1 -- 172.21.15.89:0/1921971680 >> 172.21.15.89:0/1921971680 conn(0x7fd52c08aff0 msgr2=0x7fd52c15b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.284+0000 7fd532e00700 1 -- 172.21.15.89:0/1921971680 shutdown_connections 2024-04-02T07:36:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.284+0000 7fd532e00700 1 -- 172.21.15.89:0/1921971680 wait complete. 2024-04-02T07:36:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:36:03.304 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T07:36:03.184870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219000,"kb_used_data":2816,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755720,"statfs":{"total":773094113280,"available":772869857280,"internally_reserved":0,"allocated":2883584,"data_stored":1750606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.944052"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615693+0000","last_change":"2024-04-02T07:35:53.278052+0000","last_active":"2024-04-02T07:35:54.615693+0000","last_peered":"2024-04-02T07:35:54.615693+0000","last_clean":"2024-04-02T07:35:54.615693+0000","last_became_active":"2024-04-02T07:35:53.277648+0000","last_became_peered":"2024-04-02T07:35:53.277648+0000","last_unstale":"2024-04-02T07:35:54.615693+0000","last_undegraded":"2024-04-02T07:35:54.615693+0000","last_fullsized":"2024-04-02T07:35:54.615693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.380856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617752+0000","last_change":"2024-04-02T07:35:53.271744+0000","last_active":"2024-04-02T07:35:54.617752+0000","last_peered":"2024-04-02T07:35:54.617752+0000","last_clean":"2024-04-02T07:35:54.617752+0000","last_became_active":"2024-04-02T07:35:53.271453+0000","last_became_peered":"2024-04-02T07:35:53.271453+0000","last_unstale":"2024-04-02T07:35:54.617752+0000","last_undegraded":"2024-04-02T07:35:54.617752+0000","last_fullsized":"2024-04-02T07:35:54.617752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:41:21.403396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617763+0000","last_change":"2024-04-02T07:35:53.257746+0000","last_active":"2024-04-02T07:35:54.617763+0000","last_peered":"2024-04-02T07:35:54.617763+0000","last_clean":"2024-04-02T07:35:54.617763+0000","last_became_active":"2024-04-02T07:35:53.257428+0000","last_became_peered":"2024-04-02T07:35:53.257428+0000","last_unstale":"2024-04-02T07:35:54.617763+0000","last_undegraded":"2024-04-02T07:35:54.617763+0000","last_fullsized":"2024-04-02T07:35:54.617763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:13:04.549460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557285+0000","last_change":"2024-04-02T07:35:53.269314+0000","last_active":"2024-04-02T07:35:54.557285+0000","last_peered":"2024-04-02T07:35:54.557285+0000","last_clean":"2024-04-02T07:35:54.557285+0000","last_became_active":"2024-04-02T07:35:53.269081+0000","last_became_peered":"2024-04-02T07:35:53.269081+0000","last_unstale":"2024-04-02T07:35:54.557285+0000","last_undegraded":"2024-04-02T07:35:54.557285+0000","last_fullsized":"2024-04-02T07:35:54.557285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:24.367224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615439+0000","last_change":"2024-04-02T07:35:53.257006+0000","last_active":"2024-04-02T07:35:54.615439+0000","last_peered":"2024-04-02T07:35:54.615439+0000","last_clean":"2024-04-02T07:35:54.615439+0000","last_became_active":"2024-04-02T07:35:53.256623+0000","last_became_peered":"2024-04-02T07:35:53.256623+0000","last_unstale":"2024-04-02T07:35:54.615439+0000","last_undegraded":"2024-04-02T07:35:54.615439+0000","last_fullsized":"2024-04-02T07:35:54.615439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:13:04.549460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557080+0000","last_change":"2024-04-02T07:35:53.256730+0000","last_active":"2024-04-02T07:35:54.557080+0000","last_peered":"2024-04-02T07:35:54.557080+0000","last_clean":"2024-04-02T07:35:54.557080+0000","last_became_active":"2024-04-02T07:35:53.256395+0000","last_became_peered":"2024-04-02T07:35:53.256395+0000","last_unstale":"2024-04-02T07:35:54.557080+0000","last_undegraded":"2024-04-02T07:35:54.557080+0000","last_fullsized":"2024-04-02T07:35:54.557080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:55.943715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615284+0000","last_change":"2024-04-02T07:35:53.267364+0000","last_active":"2024-04-02T07:35:54.615284+0000","last_peered":"2024-04-02T07:35:54.615284+0000","last_clean":"2024-04-02T07:35:54.615284+0000","last_became_active":"2024-04-02T07:35:53.267076+0000","last_became_peered":"2024-04-02T07:35:53.267076+0000","last_unstale":"2024-04-02T07:35:54.615284+0000","last_undegraded":"2024-04-02T07:35:54.615284+0000","last_fullsized":"2024-04-02T07:35:54.615284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:53.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617144+0000","last_change":"2024-04-02T07:35:53.275758+0000","last_active":"2024-04-02T07:35:54.617144+0000","last_peered":"2024-04-02T07:35:54.617144+0000","last_clean":"2024-04-02T07:35:54.617144+0000","last_became_active":"2024-04-02T07:35:53.264243+0000","last_became_peered":"2024-04-02T07:35:53.264243+0000","last_unstale":"2024-04-02T07:35:54.617144+0000","last_undegraded":"2024-04-02T07:35:54.617144+0000","last_fullsized":"2024-04-02T07:35:54.617144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:13.487258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246979+0000","last_change":"2024-04-02T07:35:53.280476+0000","last_active":"2024-04-02T07:35:54.246979+0000","last_peered":"2024-04-02T07:35:54.246979+0000","last_clean":"2024-04-02T07:35:54.246979+0000","last_became_active":"2024-04-02T07:35:53.280209+0000","last_became_peered":"2024-04-02T07:35:53.280209+0000","last_unstale":"2024-04-02T07:35:54.246979+0000","last_undegraded":"2024-04-02T07:35:54.246979+0000","last_fullsized":"2024-04-02T07:35:54.246979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:06.923152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615751+0000","last_change":"2024-04-02T07:35:53.258035+0000","last_active":"2024-04-02T07:35:54.615751+0000","last_peered":"2024-04-02T07:35:54.615751+0000","last_clean":"2024-04-02T07:35:54.615751+0000","last_became_active":"2024-04-02T07:35:53.257721+0000","last_became_peered":"2024-04-02T07:35:53.257721+0000","last_unstale":"2024-04-02T07:35:54.615751+0000","last_undegraded":"2024-04-02T07:35:54.615751+0000","last_fullsized":"2024-04-02T07:35:54.615751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:18.703428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615499+0000","last_change":"2024-04-02T07:35:53.275062+0000","last_active":"2024-04-02T07:35:54.615499+0000","last_peered":"2024-04-02T07:35:54.615499+0000","last_clean":"2024-04-02T07:35:54.615499+0000","last_became_active":"2024-04-02T07:35:53.274852+0000","last_became_peered":"2024-04-02T07:35:53.274852+0000","last_unstale":"2024-04-02T07:35:54.615499+0000","last_undegraded":"2024-04-02T07:35:54.615499+0000","last_fullsized":"2024-04-02T07:35:54.615499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:13.487258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615560+0000","last_change":"2024-04-02T07:35:53.269911+0000","last_active":"2024-04-02T07:35:54.615560+0000","last_peered":"2024-04-02T07:35:54.615560+0000","last_clean":"2024-04-02T07:35:54.615560+0000","last_became_active":"2024-04-02T07:35:53.265651+0000","last_became_peered":"2024-04-02T07:35:53.265651+0000","last_unstale":"2024-04-02T07:35:54.615560+0000","last_undegraded":"2024-04-02T07:35:54.615560+0000","last_fullsized":"2024-04-02T07:35:54.615560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:16.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617009+0000","last_change":"2024-04-02T07:35:53.273591+0000","last_active":"2024-04-02T07:35:54.617009+0000","last_peered":"2024-04-02T07:35:54.617009+0000","last_clean":"2024-04-02T07:35:54.617009+0000","last_became_active":"2024-04-02T07:35:53.273289+0000","last_became_peered":"2024-04-02T07:35:53.273289+0000","last_unstale":"2024-04-02T07:35:54.617009+0000","last_undegraded":"2024-04-02T07:35:54.617009+0000","last_fullsized":"2024-04-02T07:35:54.617009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:00:20.419421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616704+0000","last_change":"2024-04-02T07:35:53.281809+0000","last_active":"2024-04-02T07:35:54.616704+0000","last_peered":"2024-04-02T07:35:54.616704+0000","last_clean":"2024-04-02T07:35:54.616704+0000","last_became_active":"2024-04-02T07:35:53.277985+0000","last_became_peered":"2024-04-02T07:35:53.277985+0000","last_unstale":"2024-04-02T07:35:54.616704+0000","last_undegraded":"2024-04-02T07:35:54.616704+0000","last_fullsized":"2024-04-02T07:35:54.616704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:42.518534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617297+0000","last_change":"2024-04-02T07:35:53.268138+0000","last_active":"2024-04-02T07:35:54.617297+0000","last_peered":"2024-04-02T07:35:54.617297+0000","last_clean":"2024-04-02T07:35:54.617297+0000","last_became_active":"2024-04-02T07:35:53.264989+0000","last_became_peered":"2024-04-02T07:35:53.264989+0000","last_unstale":"2024-04-02T07:35:54.617297+0000","last_undegraded":"2024-04-02T07:35:54.617297+0000","last_fullsized":"2024-04-02T07:35:54.617297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.380856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245556+0000","last_change":"2024-04-02T07:35:53.276292+0000","last_active":"2024-04-02T07:35:54.245556+0000","last_peered":"2024-04-02T07:35:54.245556+0000","last_clean":"2024-04-02T07:35:54.245556+0000","last_became_active":"2024-04-02T07:35:53.276003+0000","last_became_peered":"2024-04-02T07:35:53.276003+0000","last_unstale":"2024-04-02T07:35:54.245556+0000","last_undegraded":"2024-04-02T07:35:54.245556+0000","last_fullsized":"2024-04-02T07:35:54.245556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:33.477903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247301+0000","last_change":"2024-04-02T07:35:53.269437+0000","last_active":"2024-04-02T07:35:54.247301+0000","last_peered":"2024-04-02T07:35:54.247301+0000","last_clean":"2024-04-02T07:35:54.247301+0000","last_became_active":"2024-04-02T07:35:53.269192+0000","last_became_peered":"2024-04-02T07:35:53.269192+0000","last_unstale":"2024-04-02T07:35:54.247301+0000","last_undegraded":"2024-04-02T07:35:54.247301+0000","last_fullsized":"2024-04-02T07:35:54.247301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:35.381030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617797+0000","last_change":"2024-04-02T07:35:53.259373+0000","last_active":"2024-04-02T07:35:54.617797+0000","last_peered":"2024-04-02T07:35:54.617797+0000","last_clean":"2024-04-02T07:35:54.617797+0000","last_became_active":"2024-04-02T07:35:53.258986+0000","last_became_peered":"2024-04-02T07:35:53.258986+0000","last_unstale":"2024-04-02T07:35:54.617797+0000","last_undegraded":"2024-04-02T07:35:54.617797+0000","last_fullsized":"2024-04-02T07:35:54.617797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:27.051276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246472+0000","last_change":"2024-04-02T07:35:53.277181+0000","last_active":"2024-04-02T07:35:54.246472+0000","last_peered":"2024-04-02T07:35:54.246472+0000","last_clean":"2024-04-02T07:35:54.246472+0000","last_became_active":"2024-04-02T07:35:53.276207+0000","last_became_peered":"2024-04-02T07:35:53.276207+0000","last_unstale":"2024-04-02T07:35:54.246472+0000","last_undegraded":"2024-04-02T07:35:54.246472+0000","last_fullsized":"2024-04-02T07:35:54.246472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:13.487258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247380+0000","last_change":"2024-04-02T07:35:52.756587+0000","last_active":"2024-04-02T07:35:54.247380+0000","last_peered":"2024-04-02T07:35:54.247380+0000","last_clean":"2024-04-02T07:35:54.247380+0000","last_became_active":"2024-04-02T07:35:52.755981+0000","last_became_peered":"2024-04-02T07:35:52.755981+0000","last_unstale":"2024-04-02T07:35:54.247380+0000","last_undegraded":"2024-04-02T07:35:54.247380+0000","last_fullsized":"2024-04-02T07:35:54.247380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:07.324155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617519+0000","last_change":"2024-04-02T07:35:53.257824+0000","last_active":"2024-04-02T07:35:54.617519+0000","last_peered":"2024-04-02T07:35:54.617519+0000","last_clean":"2024-04-02T07:35:54.617519+0000","last_became_active":"2024-04-02T07:35:53.257500+0000","last_became_peered":"2024-04-02T07:35:53.257500+0000","last_unstale":"2024-04-02T07:35:54.617519+0000","last_undegraded":"2024-04-02T07:35:54.617519+0000","last_fullsized":"2024-04-02T07:35:54.617519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:35.709321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617620+0000","last_change":"2024-04-02T07:35:52.761793+0000","last_active":"2024-04-02T07:35:54.617620+0000","last_peered":"2024-04-02T07:35:54.617620+0000","last_clean":"2024-04-02T07:35:54.617620+0000","last_became_active":"2024-04-02T07:35:52.761510+0000","last_became_peered":"2024-04-02T07:35:52.761510+0000","last_unstale":"2024-04-02T07:35:54.617620+0000","last_undegraded":"2024-04-02T07:35:54.617620+0000","last_fullsized":"2024-04-02T07:35:54.617620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617652+0000","last_change":"2024-04-02T07:35:53.262736+0000","last_active":"2024-04-02T07:35:54.617652+0000","last_peered":"2024-04-02T07:35:54.617652+0000","last_clean":"2024-04-02T07:35:54.617652+0000","last_became_active":"2024-04-02T07:35:53.262405+0000","last_became_peered":"2024-04-02T07:35:53.262405+0000","last_unstale":"2024-04-02T07:35:54.617652+0000","last_undegraded":"2024-04-02T07:35:54.617652+0000","last_fullsized":"2024-04-02T07:35:54.617652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:52:59.688399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615557+0000","last_change":"2024-04-02T07:35:52.765725+0000","last_active":"2024-04-02T07:35:54.615557+0000","last_peered":"2024-04-02T07:35:54.615557+0000","last_clean":"2024-04-02T07:35:54.615557+0000","last_became_active":"2024-04-02T07:35:52.765393+0000","last_became_peered":"2024-04-02T07:35:52.765393+0000","last_unstale":"2024-04-02T07:35:54.615557+0000","last_undegraded":"2024-04-02T07:35:54.615557+0000","last_fullsized":"2024-04-02T07:35:54.615557+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:31.244191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246863+0000","last_change":"2024-04-02T07:35:53.255693+0000","last_active":"2024-04-02T07:35:54.246863+0000","last_peered":"2024-04-02T07:35:54.246863+0000","last_clean":"2024-04-02T07:35:54.246863+0000","last_became_active":"2024-04-02T07:35:53.254627+0000","last_became_peered":"2024-04-02T07:35:53.254627+0000","last_unstale":"2024-04-02T07:35:54.246863+0000","last_undegraded":"2024-04-02T07:35:54.246863+0000","last_fullsized":"2024-04-02T07:35:54.246863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:42.518534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615455+0000","last_change":"2024-04-02T07:35:52.762790+0000","last_active":"2024-04-02T07:35:54.615455+0000","last_peered":"2024-04-02T07:35:54.615455+0000","last_clean":"2024-04-02T07:35:54.615455+0000","last_became_active":"2024-04-02T07:35:52.762416+0000","last_became_peered":"2024-04-02T07:35:52.762416+0000","last_unstale":"2024-04-02T07:35:54.615455+0000","last_undegraded":"2024-04-02T07:35:54.615455+0000","last_fullsized":"2024-04-02T07:35:54.615455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:05.916519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246530+0000","last_change":"2024-04-02T07:35:53.274608+0000","last_active":"2024-04-02T07:35:54.246530+0000","last_peered":"2024-04-02T07:35:54.246530+0000","last_clean":"2024-04-02T07:35:54.246530+0000","last_became_active":"2024-04-02T07:35:53.274308+0000","last_became_peered":"2024-04-02T07:35:53.274308+0000","last_unstale":"2024-04-02T07:35:54.246530+0000","last_undegraded":"2024-04-02T07:35:54.246530+0000","last_fullsized":"2024-04-02T07:35:54.246530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:28:26.191159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617955+0000","last_change":"2024-04-02T07:35:52.761806+0000","last_active":"2024-04-02T07:35:54.617955+0000","last_peered":"2024-04-02T07:35:54.617955+0000","last_clean":"2024-04-02T07:35:54.617955+0000","last_became_active":"2024-04-02T07:35:52.761556+0000","last_became_peered":"2024-04-02T07:35:52.761556+0000","last_unstale":"2024-04-02T07:35:54.617955+0000","last_undegraded":"2024-04-02T07:35:54.617955+0000","last_fullsized":"2024-04-02T07:35:54.617955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246589+0000","last_change":"2024-04-02T07:35:53.262036+0000","last_active":"2024-04-02T07:35:54.246589+0000","last_peered":"2024-04-02T07:35:54.246589+0000","last_clean":"2024-04-02T07:35:54.246589+0000","last_became_active":"2024-04-02T07:35:53.259860+0000","last_became_peered":"2024-04-02T07:35:53.259860+0000","last_unstale":"2024-04-02T07:35:54.246589+0000","last_undegraded":"2024-04-02T07:35:54.246589+0000","last_fullsized":"2024-04-02T07:35:54.246589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:27.051276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245921+0000","last_change":"2024-04-02T07:35:52.773232+0000","last_active":"2024-04-02T07:35:54.245921+0000","last_peered":"2024-04-02T07:35:54.245921+0000","last_clean":"2024-04-02T07:35:54.245921+0000","last_became_active":"2024-04-02T07:35:52.772864+0000","last_became_peered":"2024-04-02T07:35:52.772864+0000","last_unstale":"2024-04-02T07:35:54.245921+0000","last_undegraded":"2024-04-02T07:35:54.245921+0000","last_fullsized":"2024-04-02T07:35:54.245921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:26.044643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247078+0000","last_change":"2024-04-02T07:35:53.274460+0000","last_active":"2024-04-02T07:35:54.247078+0000","last_peered":"2024-04-02T07:35:54.247078+0000","last_clean":"2024-04-02T07:35:54.247078+0000","last_became_active":"2024-04-02T07:35:53.274227+0000","last_became_peered":"2024-04-02T07:35:53.274227+0000","last_unstale":"2024-04-02T07:35:54.247078+0000","last_undegraded":"2024-04-02T07:35:54.247078+0000","last_fullsized":"2024-04-02T07:35:54.247078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:31.170936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616577+0000","last_change":"2024-04-02T07:35:52.769404+0000","last_active":"2024-04-02T07:35:54.616577+0000","last_peered":"2024-04-02T07:35:54.616577+0000","last_clean":"2024-04-02T07:35:54.616577+0000","last_became_active":"2024-04-02T07:35:52.769091+0000","last_became_peered":"2024-04-02T07:35:52.769091+0000","last_unstale":"2024-04-02T07:35:54.616577+0000","last_undegraded":"2024-04-02T07:35:54.616577+0000","last_fullsized":"2024-04-02T07:35:54.616577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617272+0000","last_change":"2024-04-02T07:35:53.261740+0000","last_active":"2024-04-02T07:35:54.617272+0000","last_peered":"2024-04-02T07:35:54.617272+0000","last_clean":"2024-04-02T07:35:54.617272+0000","last_became_active":"2024-04-02T07:35:53.261475+0000","last_became_peered":"2024-04-02T07:35:53.261475+0000","last_unstale":"2024-04-02T07:35:54.617272+0000","last_undegraded":"2024-04-02T07:35:54.617272+0000","last_fullsized":"2024-04-02T07:35:54.617272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:31.170936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245738+0000","last_change":"2024-04-02T07:35:52.765814+0000","last_active":"2024-04-02T07:35:54.245738+0000","last_peered":"2024-04-02T07:35:54.245738+0000","last_clean":"2024-04-02T07:35:54.245738+0000","last_became_active":"2024-04-02T07:35:52.764956+0000","last_became_peered":"2024-04-02T07:35:52.764956+0000","last_unstale":"2024-04-02T07:35:54.245738+0000","last_undegraded":"2024-04-02T07:35:54.245738+0000","last_fullsized":"2024-04-02T07:35:54.245738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:49.544958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247082+0000","last_change":"2024-04-02T07:35:53.278155+0000","last_active":"2024-04-02T07:35:54.247082+0000","last_peered":"2024-04-02T07:35:54.247082+0000","last_clean":"2024-04-02T07:35:54.247082+0000","last_became_active":"2024-04-02T07:35:53.274007+0000","last_became_peered":"2024-04-02T07:35:53.274007+0000","last_unstale":"2024-04-02T07:35:54.247082+0000","last_undegraded":"2024-04-02T07:35:54.247082+0000","last_fullsized":"2024-04-02T07:35:54.247082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:05:41.161422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246958+0000","last_change":"2024-04-02T07:35:52.759611+0000","last_active":"2024-04-02T07:35:54.246958+0000","last_peered":"2024-04-02T07:35:54.246958+0000","last_clean":"2024-04-02T07:35:54.246958+0000","last_became_active":"2024-04-02T07:35:52.759164+0000","last_became_peered":"2024-04-02T07:35:52.759164+0000","last_unstale":"2024-04-02T07:35:54.246958+0000","last_undegraded":"2024-04-02T07:35:54.246958+0000","last_fullsized":"2024-04-02T07:35:54.246958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557483+0000","last_change":"2024-04-02T07:35:53.269405+0000","last_active":"2024-04-02T07:35:54.557483+0000","last_peered":"2024-04-02T07:35:54.557483+0000","last_clean":"2024-04-02T07:35:54.557483+0000","last_became_active":"2024-04-02T07:35:53.267870+0000","last_became_peered":"2024-04-02T07:35:53.267870+0000","last_unstale":"2024-04-02T07:35:54.557483+0000","last_undegraded":"2024-04-02T07:35:54.557483+0000","last_fullsized":"2024-04-02T07:35:54.557483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:08.722219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615626+0000","last_change":"2024-04-02T07:35:52.765581+0000","last_active":"2024-04-02T07:35:54.615626+0000","last_peered":"2024-04-02T07:35:54.615626+0000","last_clean":"2024-04-02T07:35:54.615626+0000","last_became_active":"2024-04-02T07:35:52.764766+0000","last_became_peered":"2024-04-02T07:35:52.764766+0000","last_unstale":"2024-04-02T07:35:54.615626+0000","last_undegraded":"2024-04-02T07:35:54.615626+0000","last_fullsized":"2024-04-02T07:35:54.615626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:07.715585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617760+0000","last_change":"2024-04-02T07:35:53.275646+0000","last_active":"2024-04-02T07:35:54.617760+0000","last_peered":"2024-04-02T07:35:54.617760+0000","last_clean":"2024-04-02T07:35:54.617760+0000","last_became_active":"2024-04-02T07:35:53.275296+0000","last_became_peered":"2024-04-02T07:35:53.275296+0000","last_unstale":"2024-04-02T07:35:54.617760+0000","last_undegraded":"2024-04-02T07:35:54.617760+0000","last_fullsized":"2024-04-02T07:35:54.617760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:17.569969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617746+0000","last_change":"2024-04-02T07:35:52.751184+0000","last_active":"2024-04-02T07:35:54.617746+0000","last_peered":"2024-04-02T07:35:54.617746+0000","last_clean":"2024-04-02T07:35:54.617746+0000","last_became_active":"2024-04-02T07:35:52.750436+0000","last_became_peered":"2024-04-02T07:35:52.750436+0000","last_unstale":"2024-04-02T07:35:54.617746+0000","last_undegraded":"2024-04-02T07:35:54.617746+0000","last_fullsized":"2024-04-02T07:35:54.617746+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246276+0000","last_change":"2024-04-02T07:35:53.277457+0000","last_active":"2024-04-02T07:35:54.246276+0000","last_peered":"2024-04-02T07:35:54.246276+0000","last_clean":"2024-04-02T07:35:54.246276+0000","last_became_active":"2024-04-02T07:35:53.277059+0000","last_became_peered":"2024-04-02T07:35:53.277059+0000","last_unstale":"2024-04-02T07:35:54.246276+0000","last_undegraded":"2024-04-02T07:35:54.246276+0000","last_fullsized":"2024-04-02T07:35:54.246276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:18:02.329099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557196+0000","last_change":"2024-04-02T07:35:52.757551+0000","last_active":"2024-04-02T07:35:54.557196+0000","last_peered":"2024-04-02T07:35:54.557196+0000","last_clean":"2024-04-02T07:35:54.557196+0000","last_became_active":"2024-04-02T07:35:52.756930+0000","last_became_peered":"2024-04-02T07:35:52.756930+0000","last_unstale":"2024-04-02T07:35:54.557196+0000","last_undegraded":"2024-04-02T07:35:54.557196+0000","last_fullsized":"2024-04-02T07:35:54.557196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:40:29.501936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615608+0000","last_change":"2024-04-02T07:35:53.265162+0000","last_active":"2024-04-02T07:35:54.615608+0000","last_peered":"2024-04-02T07:35:54.615608+0000","last_clean":"2024-04-02T07:35:54.615608+0000","last_became_active":"2024-04-02T07:35:53.264897+0000","last_became_peered":"2024-04-02T07:35:53.264897+0000","last_unstale":"2024-04-02T07:35:54.615608+0000","last_undegraded":"2024-04-02T07:35:54.615608+0000","last_fullsized":"2024-04-02T07:35:54.615608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:35.709321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616800+0000","last_change":"2024-04-02T07:35:52.767385+0000","last_active":"2024-04-02T07:35:54.616800+0000","last_peered":"2024-04-02T07:35:54.616800+0000","last_clean":"2024-04-02T07:35:54.616800+0000","last_became_active":"2024-04-02T07:35:52.767027+0000","last_became_peered":"2024-04-02T07:35:52.767027+0000","last_unstale":"2024-04-02T07:35:54.616800+0000","last_undegraded":"2024-04-02T07:35:54.616800+0000","last_fullsized":"2024-04-02T07:35:54.616800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616462+0000","last_change":"2024-04-02T07:35:53.283307+0000","last_active":"2024-04-02T07:35:54.616462+0000","last_peered":"2024-04-02T07:35:54.616462+0000","last_clean":"2024-04-02T07:35:54.616462+0000","last_became_active":"2024-04-02T07:35:53.282062+0000","last_became_peered":"2024-04-02T07:35:53.282062+0000","last_unstale":"2024-04-02T07:35:54.616462+0000","last_undegraded":"2024-04-02T07:35:54.616462+0000","last_fullsized":"2024-04-02T07:35:54.616462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:24.367224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246052+0000","last_change":"2024-04-02T07:35:52.753164+0000","last_active":"2024-04-02T07:35:54.246052+0000","last_peered":"2024-04-02T07:35:54.246052+0000","last_clean":"2024-04-02T07:35:54.246052+0000","last_became_active":"2024-04-02T07:35:52.752227+0000","last_became_peered":"2024-04-02T07:35:52.752227+0000","last_unstale":"2024-04-02T07:35:54.246052+0000","last_undegraded":"2024-04-02T07:35:54.246052+0000","last_fullsized":"2024-04-02T07:35:54.246052+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:07.715585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246205+0000","last_change":"2024-04-02T07:35:53.269115+0000","last_active":"2024-04-02T07:35:54.246205+0000","last_peered":"2024-04-02T07:35:54.246205+0000","last_clean":"2024-04-02T07:35:54.246205+0000","last_became_active":"2024-04-02T07:35:53.268815+0000","last_became_peered":"2024-04-02T07:35:53.268815+0000","last_unstale":"2024-04-02T07:35:54.246205+0000","last_undegraded":"2024-04-02T07:35:54.246205+0000","last_fullsized":"2024-04-02T07:35:54.246205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:46:59.080243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615336+0000","last_change":"2024-04-02T07:35:52.764457+0000","last_active":"2024-04-02T07:35:54.615336+0000","last_peered":"2024-04-02T07:35:54.615336+0000","last_clean":"2024-04-02T07:35:54.615336+0000","last_became_active":"2024-04-02T07:35:52.764002+0000","last_became_peered":"2024-04-02T07:35:52.764002+0000","last_unstale":"2024-04-02T07:35:54.615336+0000","last_undegraded":"2024-04-02T07:35:54.615336+0000","last_fullsized":"2024-04-02T07:35:54.615336+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:38:44.437274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246412+0000","last_change":"2024-04-02T07:35:53.262284+0000","last_active":"2024-04-02T07:35:54.246412+0000","last_peered":"2024-04-02T07:35:54.246412+0000","last_clean":"2024-04-02T07:35:54.246412+0000","last_became_active":"2024-04-02T07:35:53.260682+0000","last_became_peered":"2024-04-02T07:35:53.260682+0000","last_unstale":"2024-04-02T07:35:54.246412+0000","last_undegraded":"2024-04-02T07:35:54.246412+0000","last_fullsized":"2024-04-02T07:35:54.246412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:55.943715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246893+0000","last_change":"2024-04-02T07:35:52.754933+0000","last_active":"2024-04-02T07:35:54.246893+0000","last_peered":"2024-04-02T07:35:54.246893+0000","last_clean":"2024-04-02T07:35:54.246893+0000","last_became_active":"2024-04-02T07:35:52.750245+0000","last_became_peered":"2024-04-02T07:35:52.750245+0000","last_unstale":"2024-04-02T07:35:54.246893+0000","last_undegraded":"2024-04-02T07:35:54.246893+0000","last_fullsized":"2024-04-02T07:35:54.246893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245859+0000","last_change":"2024-04-02T07:35:53.266981+0000","last_active":"2024-04-02T07:35:54.245859+0000","last_peered":"2024-04-02T07:35:54.245859+0000","last_clean":"2024-04-02T07:35:54.245859+0000","last_became_active":"2024-04-02T07:35:53.266194+0000","last_became_peered":"2024-04-02T07:35:53.266194+0000","last_unstale":"2024-04-02T07:35:54.245859+0000","last_undegraded":"2024-04-02T07:35:54.245859+0000","last_fullsized":"2024-04-02T07:35:54.245859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:22:27.488539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615714+0000","last_change":"2024-04-02T07:35:52.771817+0000","last_active":"2024-04-02T07:35:54.615714+0000","last_peered":"2024-04-02T07:35:54.615714+0000","last_clean":"2024-04-02T07:35:54.615714+0000","last_became_active":"2024-04-02T07:35:52.771315+0000","last_became_peered":"2024-04-02T07:35:52.771315+0000","last_unstale":"2024-04-02T07:35:54.615714+0000","last_undegraded":"2024-04-02T07:35:54.615714+0000","last_fullsized":"2024-04-02T07:35:54.615714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:07.324155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557469+0000","last_change":"2024-04-02T07:35:53.254431+0000","last_active":"2024-04-02T07:35:54.557469+0000","last_peered":"2024-04-02T07:35:54.557469+0000","last_clean":"2024-04-02T07:35:54.557469+0000","last_became_active":"2024-04-02T07:35:53.253582+0000","last_became_peered":"2024-04-02T07:35:53.253582+0000","last_unstale":"2024-04-02T07:35:54.557469+0000","last_undegraded":"2024-04-02T07:35:54.557469+0000","last_fullsized":"2024-04-02T07:35:54.557469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:32.250824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617486+0000","last_change":"2024-04-02T07:35:52.761017+0000","last_active":"2024-04-02T07:35:54.617486+0000","last_peered":"2024-04-02T07:35:54.617486+0000","last_clean":"2024-04-02T07:35:54.617486+0000","last_became_active":"2024-04-02T07:35:52.760290+0000","last_became_peered":"2024-04-02T07:35:52.760290+0000","last_unstale":"2024-04-02T07:35:54.617486+0000","last_undegraded":"2024-04-02T07:35:54.617486+0000","last_fullsized":"2024-04-02T07:35:54.617486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:45.255736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557174+0000","last_change":"2024-04-02T07:35:53.263108+0000","last_active":"2024-04-02T07:35:54.557174+0000","last_peered":"2024-04-02T07:35:54.557174+0000","last_clean":"2024-04-02T07:35:54.557174+0000","last_became_active":"2024-04-02T07:35:53.262111+0000","last_became_peered":"2024-04-02T07:35:53.262111+0000","last_unstale":"2024-04-02T07:35:54.557174+0000","last_undegraded":"2024-04-02T07:35:54.557174+0000","last_fullsized":"2024-04-02T07:35:54.557174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:27.051276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245974+0000","last_change":"2024-04-02T07:35:52.763712+0000","last_active":"2024-04-02T07:35:54.245974+0000","last_peered":"2024-04-02T07:35:54.245974+0000","last_clean":"2024-04-02T07:35:54.245974+0000","last_became_active":"2024-04-02T07:35:52.763457+0000","last_became_peered":"2024-04-02T07:35:52.763457+0000","last_unstale":"2024-04-02T07:35:54.245974+0000","last_undegraded":"2024-04-02T07:35:54.245974+0000","last_fullsized":"2024-04-02T07:35:54.245974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:41.511901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615482+0000","last_change":"2024-04-02T07:35:53.254132+0000","last_active":"2024-04-02T07:35:54.615482+0000","last_peered":"2024-04-02T07:35:54.615482+0000","last_clean":"2024-04-02T07:35:54.615482+0000","last_became_active":"2024-04-02T07:35:53.253705+0000","last_became_peered":"2024-04-02T07:35:53.253705+0000","last_unstale":"2024-04-02T07:35:54.615482+0000","last_undegraded":"2024-04-02T07:35:54.615482+0000","last_fullsized":"2024-04-02T07:35:54.615482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:17.569969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617123+0000","last_change":"2024-04-02T07:35:53.267551+0000","last_active":"2024-04-02T07:35:54.617123+0000","last_peered":"2024-04-02T07:35:54.617123+0000","last_clean":"2024-04-02T07:35:54.617123+0000","last_became_active":"2024-04-02T07:35:53.263849+0000","last_became_peered":"2024-04-02T07:35:53.263849+0000","last_unstale":"2024-04-02T07:35:54.617123+0000","last_undegraded":"2024-04-02T07:35:54.617123+0000","last_fullsized":"2024-04-02T07:35:54.617123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:54:04.724188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616708+0000","last_change":"2024-04-02T07:35:52.764685+0000","last_active":"2024-04-02T07:35:54.616708+0000","last_peered":"2024-04-02T07:35:54.616708+0000","last_clean":"2024-04-02T07:35:54.616708+0000","last_became_active":"2024-04-02T07:35:52.764421+0000","last_became_peered":"2024-04-02T07:35:52.764421+0000","last_unstale":"2024-04-02T07:35:54.616708+0000","last_undegraded":"2024-04-02T07:35:54.616708+0000","last_fullsized":"2024-04-02T07:35:54.616708+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247020+0000","last_change":"2024-04-02T07:35:53.264778+0000","last_active":"2024-04-02T07:35:54.247020+0000","last_peered":"2024-04-02T07:35:54.247020+0000","last_clean":"2024-04-02T07:35:54.247020+0000","last_became_active":"2024-04-02T07:35:53.264445+0000","last_became_peered":"2024-04-02T07:35:53.264445+0000","last_unstale":"2024-04-02T07:35:54.247020+0000","last_undegraded":"2024-04-02T07:35:54.247020+0000","last_fullsized":"2024-04-02T07:35:54.247020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:00:20.419421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246196+0000","last_change":"2024-04-02T07:35:52.767065+0000","last_active":"2024-04-02T07:35:54.246196+0000","last_peered":"2024-04-02T07:35:54.246196+0000","last_clean":"2024-04-02T07:35:54.246196+0000","last_became_active":"2024-04-02T07:35:52.764931+0000","last_became_peered":"2024-04-02T07:35:52.764931+0000","last_unstale":"2024-04-02T07:35:54.246196+0000","last_undegraded":"2024-04-02T07:35:54.246196+0000","last_fullsized":"2024-04-02T07:35:54.246196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:05.916519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-02T07:35:53.250580+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:35:53.250580+0000","last_peered":"2024-04-02T07:35:53.250580+0000","last_clean":"2024-04-02T07:35:53.250580+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:35:53.250580+0000","last_undegraded":"2024-04-02T07:35:53.250580+0000","last_fullsized":"2024-04-02T07:35:53.250580+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616550+0000","last_change":"2024-04-02T07:35:53.283251+0000","last_active":"2024-04-02T07:35:54.616550+0000","last_peered":"2024-04-02T07:35:54.616550+0000","last_clean":"2024-04-02T07:35:54.616550+0000","last_became_active":"2024-04-02T07:35:53.276591+0000","last_became_peered":"2024-04-02T07:35:53.276591+0000","last_unstale":"2024-04-02T07:35:54.616550+0000","last_undegraded":"2024-04-02T07:35:54.616550+0000","last_fullsized":"2024-04-02T07:35:54.616550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:07.528924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245695+0000","last_change":"2024-04-02T07:35:52.764279+0000","last_active":"2024-04-02T07:35:54.245695+0000","last_peered":"2024-04-02T07:35:54.245695+0000","last_clean":"2024-04-02T07:35:54.245695+0000","last_became_active":"2024-04-02T07:35:52.759649+0000","last_became_peered":"2024-04-02T07:35:52.759649+0000","last_unstale":"2024-04-02T07:35:54.245695+0000","last_undegraded":"2024-04-02T07:35:54.245695+0000","last_fullsized":"2024-04-02T07:35:54.245695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:54.937081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246537+0000","last_change":"2024-04-02T07:35:52.756573+0000","last_active":"2024-04-02T07:35:54.246537+0000","last_peered":"2024-04-02T07:35:54.246537+0000","last_clean":"2024-04-02T07:35:54.246537+0000","last_became_active":"2024-04-02T07:35:52.756192+0000","last_became_peered":"2024-04-02T07:35:52.756192+0000","last_unstale":"2024-04-02T07:35:54.246537+0000","last_undegraded":"2024-04-02T07:35:54.246537+0000","last_fullsized":"2024-04-02T07:35:54.246537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616866+0000","last_change":"2024-04-02T07:35:53.269008+0000","last_active":"2024-04-02T07:35:54.616866+0000","last_peered":"2024-04-02T07:35:54.616866+0000","last_clean":"2024-04-02T07:35:54.616866+0000","last_became_active":"2024-04-02T07:35:53.268663+0000","last_became_peered":"2024-04-02T07:35:53.268663+0000","last_unstale":"2024-04-02T07:35:54.616866+0000","last_undegraded":"2024-04-02T07:35:54.616866+0000","last_fullsized":"2024-04-02T07:35:54.616866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:05:41.161422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:56.254933+0000","last_change":"2024-04-02T07:35:52.769059+0000","last_active":"2024-04-02T07:35:56.254933+0000","last_peered":"2024-04-02T07:35:56.254933+0000","last_clean":"2024-04-02T07:35:56.254933+0000","last_became_active":"2024-04-02T07:35:52.768711+0000","last_became_peered":"2024-04-02T07:35:52.768711+0000","last_unstale":"2024-04-02T07:35:56.254933+0000","last_undegraded":"2024-04-02T07:35:56.254933+0000","last_fullsized":"2024-04-02T07:35:56.254933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247358+0000","last_change":"2024-04-02T07:35:53.275607+0000","last_active":"2024-04-02T07:35:54.247358+0000","last_peered":"2024-04-02T07:35:54.247358+0000","last_clean":"2024-04-02T07:35:54.247358+0000","last_became_active":"2024-04-02T07:35:53.274591+0000","last_became_peered":"2024-04-02T07:35:53.274591+0000","last_unstale":"2024-04-02T07:35:54.247358+0000","last_undegraded":"2024-04-02T07:35:54.247358+0000","last_fullsized":"2024-04-02T07:35:54.247358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:13:58.887392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247393+0000","last_change":"2024-04-02T07:35:52.771999+0000","last_active":"2024-04-02T07:35:54.247393+0000","last_peered":"2024-04-02T07:35:54.247393+0000","last_clean":"2024-04-02T07:35:54.247393+0000","last_became_active":"2024-04-02T07:35:52.771720+0000","last_became_peered":"2024-04-02T07:35:52.771720+0000","last_unstale":"2024-04-02T07:35:54.247393+0000","last_undegraded":"2024-04-02T07:35:54.247393+0000","last_fullsized":"2024-04-02T07:35:54.247393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:31.244191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247153+0000","last_change":"2024-04-02T07:35:53.258654+0000","last_active":"2024-04-02T07:35:54.247153+0000","last_peered":"2024-04-02T07:35:54.247153+0000","last_clean":"2024-04-02T07:35:54.247153+0000","last_became_active":"2024-04-02T07:35:53.257658+0000","last_became_peered":"2024-04-02T07:35:53.257658+0000","last_unstale":"2024-04-02T07:35:54.247153+0000","last_undegraded":"2024-04-02T07:35:54.247153+0000","last_fullsized":"2024-04-02T07:35:54.247153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:07.528924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615282+0000","last_change":"2024-04-02T07:35:52.764308+0000","last_active":"2024-04-02T07:35:54.615282+0000","last_peered":"2024-04-02T07:35:54.615282+0000","last_clean":"2024-04-02T07:35:54.615282+0000","last_became_active":"2024-04-02T07:35:52.763178+0000","last_became_peered":"2024-04-02T07:35:52.763178+0000","last_unstale":"2024-04-02T07:35:54.615282+0000","last_undegraded":"2024-04-02T07:35:54.615282+0000","last_fullsized":"2024-04-02T07:35:54.615282+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:45.255736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245826+0000","last_change":"2024-04-02T07:35:53.258575+0000","last_active":"2024-04-02T07:35:54.245826+0000","last_peered":"2024-04-02T07:35:54.245826+0000","last_clean":"2024-04-02T07:35:54.245826+0000","last_became_active":"2024-04-02T07:35:53.258326+0000","last_became_peered":"2024-04-02T07:35:53.258326+0000","last_unstale":"2024-04-02T07:35:54.245826+0000","last_undegraded":"2024-04-02T07:35:54.245826+0000","last_fullsized":"2024-04-02T07:35:54.245826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:16.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247051+0000","last_change":"2024-04-02T07:35:52.771988+0000","last_active":"2024-04-02T07:35:54.247051+0000","last_peered":"2024-04-02T07:35:54.247051+0000","last_clean":"2024-04-02T07:35:54.247051+0000","last_became_active":"2024-04-02T07:35:52.771616+0000","last_became_peered":"2024-04-02T07:35:52.771616+0000","last_unstale":"2024-04-02T07:35:54.247051+0000","last_undegraded":"2024-04-02T07:35:54.247051+0000","last_fullsized":"2024-04-02T07:35:54.247051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:45.255736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246904+0000","last_change":"2024-04-02T07:35:53.258346+0000","last_active":"2024-04-02T07:35:54.246904+0000","last_peered":"2024-04-02T07:35:54.246904+0000","last_clean":"2024-04-02T07:35:54.246904+0000","last_became_active":"2024-04-02T07:35:53.258044+0000","last_became_peered":"2024-04-02T07:35:53.258044+0000","last_unstale":"2024-04-02T07:35:54.246904+0000","last_undegraded":"2024-04-02T07:35:54.246904+0000","last_fullsized":"2024-04-02T07:35:54.246904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:50.551591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616574+0000","last_change":"2024-04-02T07:35:52.758632+0000","last_active":"2024-04-02T07:35:54.616574+0000","last_peered":"2024-04-02T07:35:54.616574+0000","last_clean":"2024-04-02T07:35:54.616574+0000","last_became_active":"2024-04-02T07:35:52.758347+0000","last_became_peered":"2024-04-02T07:35:52.758347+0000","last_unstale":"2024-04-02T07:35:54.616574+0000","last_undegraded":"2024-04-02T07:35:54.616574+0000","last_fullsized":"2024-04-02T07:35:54.616574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617044+0000","last_change":"2024-04-02T07:35:53.271892+0000","last_active":"2024-04-02T07:35:54.617044+0000","last_peered":"2024-04-02T07:35:54.617044+0000","last_clean":"2024-04-02T07:35:54.617044+0000","last_became_active":"2024-04-02T07:35:53.271472+0000","last_became_peered":"2024-04-02T07:35:53.271472+0000","last_unstale":"2024-04-02T07:35:54.617044+0000","last_undegraded":"2024-04-02T07:35:54.617044+0000","last_fullsized":"2024-04-02T07:35:54.617044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:06.923152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246267+0000","last_change":"2024-04-02T07:35:52.758627+0000","last_active":"2024-04-02T07:35:54.246267+0000","last_peered":"2024-04-02T07:35:54.246267+0000","last_clean":"2024-04-02T07:35:54.246267+0000","last_became_active":"2024-04-02T07:35:52.758330+0000","last_became_peered":"2024-04-02T07:35:52.758330+0000","last_unstale":"2024-04-02T07:35:54.246267+0000","last_undegraded":"2024-04-02T07:35:54.246267+0000","last_fullsized":"2024-04-02T07:35:54.246267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:23.360591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245991+0000","last_change":"2024-04-02T07:35:53.266441+0000","last_active":"2024-04-02T07:35:54.245991+0000","last_peered":"2024-04-02T07:35:54.245991+0000","last_clean":"2024-04-02T07:35:54.245991+0000","last_became_active":"2024-04-02T07:35:53.265569+0000","last_became_peered":"2024-04-02T07:35:53.265569+0000","last_unstale":"2024-04-02T07:35:54.245991+0000","last_undegraded":"2024-04-02T07:35:54.245991+0000","last_fullsized":"2024-04-02T07:35:54.245991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:53.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616939+0000","last_change":"2024-04-02T07:35:52.762509+0000","last_active":"2024-04-02T07:35:54.616939+0000","last_peered":"2024-04-02T07:35:54.616939+0000","last_clean":"2024-04-02T07:35:54.616939+0000","last_became_active":"2024-04-02T07:35:52.762190+0000","last_became_peered":"2024-04-02T07:35:52.762190+0000","last_unstale":"2024-04-02T07:35:54.616939+0000","last_undegraded":"2024-04-02T07:35:54.616939+0000","last_fullsized":"2024-04-02T07:35:54.616939+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246626+0000","last_change":"2024-04-02T07:35:53.272664+0000","last_active":"2024-04-02T07:35:54.246626+0000","last_peered":"2024-04-02T07:35:54.246626+0000","last_clean":"2024-04-02T07:35:54.246626+0000","last_became_active":"2024-04-02T07:35:53.272408+0000","last_became_peered":"2024-04-02T07:35:53.272408+0000","last_unstale":"2024-04-02T07:35:54.246626+0000","last_undegraded":"2024-04-02T07:35:54.246626+0000","last_fullsized":"2024-04-02T07:35:54.246626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:18.703428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617830+0000","last_change":"2024-04-02T07:35:52.760241+0000","last_active":"2024-04-02T07:35:54.617830+0000","last_peered":"2024-04-02T07:35:54.617830+0000","last_clean":"2024-04-02T07:35:54.617830+0000","last_became_active":"2024-04-02T07:35:52.759523+0000","last_became_peered":"2024-04-02T07:35:52.759523+0000","last_unstale":"2024-04-02T07:35:54.617830+0000","last_undegraded":"2024-04-02T07:35:54.617830+0000","last_fullsized":"2024-04-02T07:35:54.617830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:07.324155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615417+0000","last_change":"2024-04-02T07:35:53.252444+0000","last_active":"2024-04-02T07:35:54.615417+0000","last_peered":"2024-04-02T07:35:54.615417+0000","last_clean":"2024-04-02T07:35:54.615417+0000","last_became_active":"2024-04-02T07:35:53.252160+0000","last_became_peered":"2024-04-02T07:35:53.252160+0000","last_unstale":"2024-04-02T07:35:54.615417+0000","last_undegraded":"2024-04-02T07:35:54.615417+0000","last_fullsized":"2024-04-02T07:35:54.615417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:18:02.329099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247287+0000","last_change":"2024-04-02T07:35:52.760163+0000","last_active":"2024-04-02T07:35:54.247287+0000","last_peered":"2024-04-02T07:35:54.247287+0000","last_clean":"2024-04-02T07:35:54.247287+0000","last_became_active":"2024-04-02T07:35:52.759785+0000","last_became_peered":"2024-04-02T07:35:52.759785+0000","last_unstale":"2024-04-02T07:35:54.247287+0000","last_undegraded":"2024-04-02T07:35:54.247287+0000","last_fullsized":"2024-04-02T07:35:54.247287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615799+0000","last_change":"2024-04-02T07:35:53.259830+0000","last_active":"2024-04-02T07:35:54.615799+0000","last_peered":"2024-04-02T07:35:54.615799+0000","last_clean":"2024-04-02T07:35:54.615799+0000","last_became_active":"2024-04-02T07:35:53.259518+0000","last_became_peered":"2024-04-02T07:35:53.259518+0000","last_unstale":"2024-04-02T07:35:54.615799+0000","last_undegraded":"2024-04-02T07:35:54.615799+0000","last_fullsized":"2024-04-02T07:35:54.615799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:54:41.930328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.556975+0000","last_change":"2024-04-02T07:35:53.263050+0000","last_active":"2024-04-02T07:35:54.556975+0000","last_peered":"2024-04-02T07:35:54.556975+0000","last_clean":"2024-04-02T07:35:54.556975+0000","last_became_active":"2024-04-02T07:35:53.262790+0000","last_became_peered":"2024-04-02T07:35:53.262790+0000","last_unstale":"2024-04-02T07:35:54.556975+0000","last_undegraded":"2024-04-02T07:35:54.556975+0000","last_fullsized":"2024-04-02T07:35:54.556975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:42.518534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616668+0000","last_change":"2024-04-02T07:35:53.269409+0000","last_active":"2024-04-02T07:35:54.616668+0000","last_peered":"2024-04-02T07:35:54.616668+0000","last_clean":"2024-04-02T07:35:54.616668+0000","last_became_active":"2024-04-02T07:35:53.267892+0000","last_became_peered":"2024-04-02T07:35:53.267892+0000","last_unstale":"2024-04-02T07:35:54.616668+0000","last_undegraded":"2024-04-02T07:35:54.616668+0000","last_fullsized":"2024-04-02T07:35:54.616668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:08.722219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245810+0000","last_change":"2024-04-02T07:35:53.260570+0000","last_active":"2024-04-02T07:35:54.245810+0000","last_peered":"2024-04-02T07:35:54.245810+0000","last_clean":"2024-04-02T07:35:54.245810+0000","last_became_active":"2024-04-02T07:35:53.260265+0000","last_became_peered":"2024-04-02T07:35:53.260265+0000","last_unstale":"2024-04-02T07:35:54.245810+0000","last_undegraded":"2024-04-02T07:35:54.245810+0000","last_fullsized":"2024-04-02T07:35:54.245810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:13:55.860837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617586+0000","last_change":"2024-04-02T07:35:53.260933+0000","last_active":"2024-04-02T07:35:54.617586+0000","last_peered":"2024-04-02T07:35:54.617586+0000","last_clean":"2024-04-02T07:35:54.617586+0000","last_became_active":"2024-04-02T07:35:53.260662+0000","last_became_peered":"2024-04-02T07:35:53.260662+0000","last_unstale":"2024-04-02T07:35:54.617586+0000","last_undegraded":"2024-04-02T07:35:54.617586+0000","last_fullsized":"2024-04-02T07:35:54.617586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:16.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616780+0000","last_change":"2024-04-02T07:35:53.274324+0000","last_active":"2024-04-02T07:35:54.616780+0000","last_peered":"2024-04-02T07:35:54.616780+0000","last_clean":"2024-04-02T07:35:54.616780+0000","last_became_active":"2024-04-02T07:35:53.274049+0000","last_became_peered":"2024-04-02T07:35:53.274049+0000","last_unstale":"2024-04-02T07:35:54.616780+0000","last_undegraded":"2024-04-02T07:35:54.616780+0000","last_fullsized":"2024-04-02T07:35:54.616780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:54:04.724188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616584+0000","last_change":"2024-04-02T07:35:53.272852+0000","last_active":"2024-04-02T07:35:54.616584+0000","last_peered":"2024-04-02T07:35:54.616584+0000","last_clean":"2024-04-02T07:35:54.616584+0000","last_became_active":"2024-04-02T07:35:53.272036+0000","last_became_peered":"2024-04-02T07:35:53.272036+0000","last_unstale":"2024-04-02T07:35:54.616584+0000","last_undegraded":"2024-04-02T07:35:54.616584+0000","last_fullsized":"2024-04-02T07:35:54.616584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:50.551591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557432+0000","last_change":"2024-04-02T07:35:53.275573+0000","last_active":"2024-04-02T07:35:54.557432+0000","last_peered":"2024-04-02T07:35:54.557432+0000","last_clean":"2024-04-02T07:35:54.557432+0000","last_became_active":"2024-04-02T07:35:53.272553+0000","last_became_peered":"2024-04-02T07:35:53.272553+0000","last_unstale":"2024-04-02T07:35:54.557432+0000","last_undegraded":"2024-04-02T07:35:54.557432+0000","last_fullsized":"2024-04-02T07:35:54.557432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:06.923152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617508+0000","last_change":"2024-04-02T07:35:53.262359+0000","last_active":"2024-04-02T07:35:54.617508+0000","last_peered":"2024-04-02T07:35:54.617508+0000","last_clean":"2024-04-02T07:35:54.617508+0000","last_became_active":"2024-04-02T07:35:53.262038+0000","last_became_peered":"2024-04-02T07:35:53.262038+0000","last_unstale":"2024-04-02T07:35:54.617508+0000","last_undegraded":"2024-04-02T07:35:54.617508+0000","last_fullsized":"2024-04-02T07:35:54.617508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:18.703428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245590+0000","last_change":"2024-04-02T07:35:53.284523+0000","last_active":"2024-04-02T07:35:54.245590+0000","last_peered":"2024-04-02T07:35:54.245590+0000","last_clean":"2024-04-02T07:35:54.245590+0000","last_became_active":"2024-04-02T07:35:53.284200+0000","last_became_peered":"2024-04-02T07:35:53.284200+0000","last_unstale":"2024-04-02T07:35:54.245590+0000","last_undegraded":"2024-04-02T07:35:54.245590+0000","last_fullsized":"2024-04-02T07:35:54.245590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:14:43.088676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617379+0000","last_change":"2024-04-02T07:35:53.259326+0000","last_active":"2024-04-02T07:35:54.617379+0000","last_peered":"2024-04-02T07:35:54.617379+0000","last_clean":"2024-04-02T07:35:54.617379+0000","last_became_active":"2024-04-02T07:35:53.259023+0000","last_became_peered":"2024-04-02T07:35:53.259023+0000","last_unstale":"2024-04-02T07:35:54.617379+0000","last_undegraded":"2024-04-02T07:35:54.617379+0000","last_fullsized":"2024-04-02T07:35:54.617379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:54:41.930328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246691+0000","last_change":"2024-04-02T07:35:53.261005+0000","last_active":"2024-04-02T07:35:54.246691+0000","last_peered":"2024-04-02T07:35:54.246691+0000","last_clean":"2024-04-02T07:35:54.246691+0000","last_became_active":"2024-04-02T07:35:53.259618+0000","last_became_peered":"2024-04-02T07:35:53.259618+0000","last_unstale":"2024-04-02T07:35:54.246691+0000","last_undegraded":"2024-04-02T07:35:54.246691+0000","last_fullsized":"2024-04-02T07:35:54.246691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:24.367224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247226+0000","last_change":"2024-04-02T07:35:53.262531+0000","last_active":"2024-04-02T07:35:54.247226+0000","last_peered":"2024-04-02T07:35:54.247226+0000","last_clean":"2024-04-02T07:35:54.247226+0000","last_became_active":"2024-04-02T07:35:53.262196+0000","last_became_peered":"2024-04-02T07:35:53.262196+0000","last_unstale":"2024-04-02T07:35:54.247226+0000","last_undegraded":"2024-04-02T07:35:54.247226+0000","last_fullsized":"2024-04-02T07:35:54.247226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:54:04.724188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615537+0000","last_change":"2024-04-02T07:35:53.266761+0000","last_active":"2024-04-02T07:35:54.615537+0000","last_peered":"2024-04-02T07:35:54.615537+0000","last_clean":"2024-04-02T07:35:54.615537+0000","last_became_active":"2024-04-02T07:35:53.264360+0000","last_became_peered":"2024-04-02T07:35:53.264360+0000","last_unstale":"2024-04-02T07:35:54.615537+0000","last_undegraded":"2024-04-02T07:35:54.615537+0000","last_fullsized":"2024-04-02T07:35:54.615537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:31.170936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9079999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0750000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9430000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7969999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9690000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.764}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.706}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1640000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.125}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.4800000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.0899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7539999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.931}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1580000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":71480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.476}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2839999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7310000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4470000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2909999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":73134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":660339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:36:03.308 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:36:03.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.533+0000 7f7b36400700 1 Processor -- start 2024-04-02T07:36:03.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.533+0000 7f7b36400700 1 -- start start 2024-04-02T07:36:03.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b36400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b30160f10 0x7f7b300d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:03.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b36400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b30161590 0x7f7b30161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:03.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b36400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b300ceda0 con 0x7f7b300cf230 2024-04-02T07:36:03.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b36400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b30161f80 con 0x7f7b30161590 2024-04-02T07:36:03.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b36400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b301612f0 con 0x7f7b30160f10 2024-04-02T07:36:03.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b2fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b30161590 0x7f7b30161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:03.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b2ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7b300cf230 0x7f7b300d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34940/0 (socket says 172.21.15.89:34940) 2024-04-02T07:36:03.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b2ea00700 1 -- 172.21.15.89:0/2065426042 learned_addr learned my addr 172.21.15.89:0/2065426042 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:03.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b2fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b30161590 0x7f7b30161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52752/0 (socket says 172.21.15.89:52752) 2024-04-02T07:36:03.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b2f400700 1 --2- 172.21.15.89:0/2065426042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b30160f10 0x7f7b300d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:03.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b2fe00700 1 -- 172.21.15.89:0/2065426042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b30160f10 msgr2=0x7f7b300d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:03.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b2fe00700 1 --2- 172.21.15.89:0/2065426042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b30160f10 0x7f7b300d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b2fe00700 1 -- 172.21.15.89:0/2065426042 >> v1:172.21.15.89:6789/0 conn(0x7f7b300cf230 legacy=0x7f7b300d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:03.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b2fe00700 1 -- 172.21.15.89:0/2065426042 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b300ce470 con 0x7f7b30161590 2024-04-02T07:36:03.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.535+0000 7f7b2e000700 1 -- 172.21.15.89:0/2065426042 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680550419 0 0) 0x7f7b300ceda0 con 0x7f7b300cf230 2024-04-02T07:36:03.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.536+0000 7f7b2fe00700 1 --2- 172.21.15.89:0/2065426042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b30161590 0x7f7b30161a40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b28009dc0 tx=0x7f7b2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7a9eadc4a85bf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:03.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.536+0000 7f7b2e000700 1 -- 172.21.15.89:0/2065426042 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2800e070 con 0x7f7b30161590 2024-04-02T07:36:03.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.536+0000 7f7b2e000700 1 -- 172.21.15.89:0/2065426042 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2800bb90 con 0x7f7b30161590 2024-04-02T07:36:03.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.536+0000 7f7b2e000700 1 -- 172.21.15.89:0/2065426042 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b28012430 con 0x7f7b30161590 2024-04-02T07:36:03.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.536+0000 7f7b36400700 1 -- 172.21.15.89:0/2065426042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b30161590 msgr2=0x7f7b30161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:03.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.536+0000 7f7b36400700 1 --2- 172.21.15.89:0/2065426042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b30161590 0x7f7b30161a40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b28009dc0 tx=0x7f7b2800b330 comp rx=0 tx=0).stop 2024-04-02T07:36:03.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.536+0000 7f7b36400700 1 -- 172.21.15.89:0/2065426042 shutdown_connections 2024-04-02T07:36:03.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.536+0000 7f7b36400700 1 --2- 172.21.15.89:0/2065426042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b30161590 0x7f7b30161a40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.536+0000 7f7b36400700 1 --2- 172.21.15.89:0/2065426042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b30160f10 0x7f7b300d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.536+0000 7f7b36400700 1 -- 172.21.15.89:0/2065426042 >> 172.21.15.89:0/2065426042 conn(0x7f7b3008aff0 msgr2=0x7f7b300d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:03.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.536+0000 7f7b36400700 1 -- 172.21.15.89:0/2065426042 shutdown_connections 2024-04-02T07:36:03.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.537+0000 7f7b36400700 1 -- 172.21.15.89:0/2065426042 wait complete. 2024-04-02T07:36:03.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.538+0000 7f7b36400700 1 Processor -- start 2024-04-02T07:36:03.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.538+0000 7f7b36400700 1 -- start start 2024-04-02T07:36:03.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.539+0000 7f7b36400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b300cf230 0x7f7b3014eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:03.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.539+0000 7f7b36400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b30160f10 0x7f7b3015dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:03.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.539+0000 7f7b36400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b30161590 0x7f7b3009e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:03.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.539+0000 7f7b36400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3014f080 con 0x7f7b300cf230 2024-04-02T07:36:03.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.539+0000 7f7b36400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3015e5f0 con 0x7f7b30160f10 2024-04-02T07:36:03.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.539+0000 7f7b36400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3015e8d0 con 0x7f7b30161590 2024-04-02T07:36:03.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.539+0000 7f7b2f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b300cf230 0x7f7b3014eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:03.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.539+0000 7f7b2f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b300cf230 0x7f7b3014eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52758/0 (socket says 172.21.15.89:52758) 2024-04-02T07:36:03.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.539+0000 7f7b2f400700 1 -- 172.21.15.89:0/3487005967 learned_addr learned my addr 172.21.15.89:0/3487005967 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:03.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.540+0000 7f7b2fe00700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b30161590 0x7f7b3009e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:03.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.540+0000 7f7b2ea00700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b30160f10 0x7f7b3015dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:03.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.540+0000 7f7b2f400700 1 -- 172.21.15.89:0/3487005967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b30161590 msgr2=0x7f7b3009e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:03.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.540+0000 7f7b2f400700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b30161590 0x7f7b3009e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.540+0000 7f7b2f400700 1 -- 172.21.15.89:0/3487005967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b30160f10 msgr2=0x7f7b3015dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:03.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.540+0000 7f7b2f400700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b30160f10 0x7f7b3015dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.540+0000 7f7b2f400700 1 -- 172.21.15.89:0/3487005967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b28009a70 con 0x7f7b300cf230 2024-04-02T07:36:03.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.540+0000 7f7b2ea00700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b30160f10 0x7f7b3015dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:03.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.540+0000 7f7b2f400700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b300cf230 0x7f7b3014eb40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000d800 tx=0x7f7b2000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:03.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.540+0000 7f7b27400700 1 -- 172.21.15.89:0/3487005967 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b200125f0 con 0x7f7b300cf230 2024-04-02T07:36:03.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.540+0000 7f7b27400700 1 -- 172.21.15.89:0/3487005967 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b20012750 con 0x7f7b300cf230 2024-04-02T07:36:03.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.540+0000 7f7b36400700 1 -- 172.21.15.89:0/3487005967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3015ebb0 con 0x7f7b300cf230 2024-04-02T07:36:03.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.540+0000 7f7b27400700 1 -- 172.21.15.89:0/3487005967 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20012b00 con 0x7f7b300cf230 2024-04-02T07:36:03.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.540+0000 7f7b36400700 1 -- 172.21.15.89:0/3487005967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3015f170 con 0x7f7b300cf230 2024-04-02T07:36:03.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.541+0000 7f7b36400700 1 -- 172.21.15.89:0/3487005967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3015f330 con 0x7f7b300cf230 2024-04-02T07:36:03.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.542+0000 7f7b27400700 1 -- 172.21.15.89:0/3487005967 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7b20012c60 con 0x7f7b300cf230 2024-04-02T07:36:03.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.543+0000 7f7b27400700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7b1006ac90 0x7f7b1006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:03.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.543+0000 7f7b27400700 1 -- 172.21.15.89:0/3487005967 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b200896e0 con 0x7f7b300cf230 2024-04-02T07:36:03.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.543+0000 7f7b2ea00700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7b1006ac90 0x7f7b1006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:03.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.544+0000 7f7b2ea00700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7b1006ac90 0x7f7b1006d150 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c007fd0 tx=0x7f7b1c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:03.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.548+0000 7f7b27400700 1 -- 172.21.15.89:0/3487005967 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7b200548b0 con 0x7f7b300cf230 2024-04-02T07:36:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.744+0000 7f7b36400700 1 -- 172.21.15.89:0/3487005967 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7b300cea30 con 0x7f7b1006ac90 2024-04-02T07:36:03.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.753+0000 7f7b27400700 1 -- 172.21.15.89:0/3487005967 <== mgr.4108 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264490 (secure 0 0 0) 0x7f7b1c001420 con 0x7f7b1006ac90 2024-04-02T07:36:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:03.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.757+0000 7f7b36400700 1 -- 172.21.15.89:0/3487005967 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7b1006ac90 msgr2=0x7f7b1006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:03.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.757+0000 7f7b36400700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7b1006ac90 0x7f7b1006d150 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c007fd0 tx=0x7f7b1c009040 comp rx=0 tx=0).stop 2024-04-02T07:36:03.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.757+0000 7f7b36400700 1 -- 172.21.15.89:0/3487005967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b300cf230 msgr2=0x7f7b3014eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:03.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.757+0000 7f7b36400700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b300cf230 0x7f7b3014eb40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000d800 tx=0x7f7b2000dbc0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.757+0000 7f7b36400700 1 -- 172.21.15.89:0/3487005967 shutdown_connections 2024-04-02T07:36:03.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.757+0000 7f7b36400700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7b1006ac90 0x7f7b1006d150 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.757+0000 7f7b36400700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b30161590 0x7f7b3009e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.757+0000 7f7b36400700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b300cf230 0x7f7b3014eb40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.757+0000 7f7b36400700 1 --2- 172.21.15.89:0/3487005967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b30160f10 0x7f7b3015dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:03.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.757+0000 7f7b36400700 1 -- 172.21.15.89:0/3487005967 >> 172.21.15.89:0/3487005967 conn(0x7f7b3008aff0 msgr2=0x7f7b300cba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:03.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.757+0000 7f7b36400700 1 -- 172.21.15.89:0/3487005967 shutdown_connections 2024-04-02T07:36:03.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:03.757+0000 7f7b36400700 1 -- 172.21.15.89:0/3487005967 wait complete. 2024-04-02T07:36:03.761 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:36:03.776 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T07:36:03.184870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219000,"kb_used_data":2816,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755720,"statfs":{"total":773094113280,"available":772869857280,"internally_reserved":0,"allocated":2883584,"data_stored":1750606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.944052"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615693+0000","last_change":"2024-04-02T07:35:53.278052+0000","last_active":"2024-04-02T07:35:54.615693+0000","last_peered":"2024-04-02T07:35:54.615693+0000","last_clean":"2024-04-02T07:35:54.615693+0000","last_became_active":"2024-04-02T07:35:53.277648+0000","last_became_peered":"2024-04-02T07:35:53.277648+0000","last_unstale":"2024-04-02T07:35:54.615693+0000","last_undegraded":"2024-04-02T07:35:54.615693+0000","last_fullsized":"2024-04-02T07:35:54.615693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.380856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617752+0000","last_change":"2024-04-02T07:35:53.271744+0000","last_active":"2024-04-02T07:35:54.617752+0000","last_peered":"2024-04-02T07:35:54.617752+0000","last_clean":"2024-04-02T07:35:54.617752+0000","last_became_active":"2024-04-02T07:35:53.271453+0000","last_became_peered":"2024-04-02T07:35:53.271453+0000","last_unstale":"2024-04-02T07:35:54.617752+0000","last_undegraded":"2024-04-02T07:35:54.617752+0000","last_fullsized":"2024-04-02T07:35:54.617752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:41:21.403396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617763+0000","last_change":"2024-04-02T07:35:53.257746+0000","last_active":"2024-04-02T07:35:54.617763+0000","last_peered":"2024-04-02T07:35:54.617763+0000","last_clean":"2024-04-02T07:35:54.617763+0000","last_became_active":"2024-04-02T07:35:53.257428+0000","last_became_peered":"2024-04-02T07:35:53.257428+0000","last_unstale":"2024-04-02T07:35:54.617763+0000","last_undegraded":"2024-04-02T07:35:54.617763+0000","last_fullsized":"2024-04-02T07:35:54.617763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:13:04.549460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557285+0000","last_change":"2024-04-02T07:35:53.269314+0000","last_active":"2024-04-02T07:35:54.557285+0000","last_peered":"2024-04-02T07:35:54.557285+0000","last_clean":"2024-04-02T07:35:54.557285+0000","last_became_active":"2024-04-02T07:35:53.269081+0000","last_became_peered":"2024-04-02T07:35:53.269081+0000","last_unstale":"2024-04-02T07:35:54.557285+0000","last_undegraded":"2024-04-02T07:35:54.557285+0000","last_fullsized":"2024-04-02T07:35:54.557285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:24.367224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615439+0000","last_change":"2024-04-02T07:35:53.257006+0000","last_active":"2024-04-02T07:35:54.615439+0000","last_peered":"2024-04-02T07:35:54.615439+0000","last_clean":"2024-04-02T07:35:54.615439+0000","last_became_active":"2024-04-02T07:35:53.256623+0000","last_became_peered":"2024-04-02T07:35:53.256623+0000","last_unstale":"2024-04-02T07:35:54.615439+0000","last_undegraded":"2024-04-02T07:35:54.615439+0000","last_fullsized":"2024-04-02T07:35:54.615439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:13:04.549460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557080+0000","last_change":"2024-04-02T07:35:53.256730+0000","last_active":"2024-04-02T07:35:54.557080+0000","last_peered":"2024-04-02T07:35:54.557080+0000","last_clean":"2024-04-02T07:35:54.557080+0000","last_became_active":"2024-04-02T07:35:53.256395+0000","last_became_peered":"2024-04-02T07:35:53.256395+0000","last_unstale":"2024-04-02T07:35:54.557080+0000","last_undegraded":"2024-04-02T07:35:54.557080+0000","last_fullsized":"2024-04-02T07:35:54.557080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:55.943715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615284+0000","last_change":"2024-04-02T07:35:53.267364+0000","last_active":"2024-04-02T07:35:54.615284+0000","last_peered":"2024-04-02T07:35:54.615284+0000","last_clean":"2024-04-02T07:35:54.615284+0000","last_became_active":"2024-04-02T07:35:53.267076+0000","last_became_peered":"2024-04-02T07:35:53.267076+0000","last_unstale":"2024-04-02T07:35:54.615284+0000","last_undegraded":"2024-04-02T07:35:54.615284+0000","last_fullsized":"2024-04-02T07:35:54.615284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:53.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617144+0000","last_change":"2024-04-02T07:35:53.275758+0000","last_active":"2024-04-02T07:35:54.617144+0000","last_peered":"2024-04-02T07:35:54.617144+0000","last_clean":"2024-04-02T07:35:54.617144+0000","last_became_active":"2024-04-02T07:35:53.264243+0000","last_became_peered":"2024-04-02T07:35:53.264243+0000","last_unstale":"2024-04-02T07:35:54.617144+0000","last_undegraded":"2024-04-02T07:35:54.617144+0000","last_fullsized":"2024-04-02T07:35:54.617144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:13.487258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246979+0000","last_change":"2024-04-02T07:35:53.280476+0000","last_active":"2024-04-02T07:35:54.246979+0000","last_peered":"2024-04-02T07:35:54.246979+0000","last_clean":"2024-04-02T07:35:54.246979+0000","last_became_active":"2024-04-02T07:35:53.280209+0000","last_became_peered":"2024-04-02T07:35:53.280209+0000","last_unstale":"2024-04-02T07:35:54.246979+0000","last_undegraded":"2024-04-02T07:35:54.246979+0000","last_fullsized":"2024-04-02T07:35:54.246979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:06.923152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615751+0000","last_change":"2024-04-02T07:35:53.258035+0000","last_active":"2024-04-02T07:35:54.615751+0000","last_peered":"2024-04-02T07:35:54.615751+0000","last_clean":"2024-04-02T07:35:54.615751+0000","last_became_active":"2024-04-02T07:35:53.257721+0000","last_became_peered":"2024-04-02T07:35:53.257721+0000","last_unstale":"2024-04-02T07:35:54.615751+0000","last_undegraded":"2024-04-02T07:35:54.615751+0000","last_fullsized":"2024-04-02T07:35:54.615751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:18.703428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615499+0000","last_change":"2024-04-02T07:35:53.275062+0000","last_active":"2024-04-02T07:35:54.615499+0000","last_peered":"2024-04-02T07:35:54.615499+0000","last_clean":"2024-04-02T07:35:54.615499+0000","last_became_active":"2024-04-02T07:35:53.274852+0000","last_became_peered":"2024-04-02T07:35:53.274852+0000","last_unstale":"2024-04-02T07:35:54.615499+0000","last_undegraded":"2024-04-02T07:35:54.615499+0000","last_fullsized":"2024-04-02T07:35:54.615499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:13.487258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615560+0000","last_change":"2024-04-02T07:35:53.269911+0000","last_active":"2024-04-02T07:35:54.615560+0000","last_peered":"2024-04-02T07:35:54.615560+0000","last_clean":"2024-04-02T07:35:54.615560+0000","last_became_active":"2024-04-02T07:35:53.265651+0000","last_became_peered":"2024-04-02T07:35:53.265651+0000","last_unstale":"2024-04-02T07:35:54.615560+0000","last_undegraded":"2024-04-02T07:35:54.615560+0000","last_fullsized":"2024-04-02T07:35:54.615560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:16.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617009+0000","last_change":"2024-04-02T07:35:53.273591+0000","last_active":"2024-04-02T07:35:54.617009+0000","last_peered":"2024-04-02T07:35:54.617009+0000","last_clean":"2024-04-02T07:35:54.617009+0000","last_became_active":"2024-04-02T07:35:53.273289+0000","last_became_peered":"2024-04-02T07:35:53.273289+0000","last_unstale":"2024-04-02T07:35:54.617009+0000","last_undegraded":"2024-04-02T07:35:54.617009+0000","last_fullsized":"2024-04-02T07:35:54.617009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:00:20.419421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616704+0000","last_change":"2024-04-02T07:35:53.281809+0000","last_active":"2024-04-02T07:35:54.616704+0000","last_peered":"2024-04-02T07:35:54.616704+0000","last_clean":"2024-04-02T07:35:54.616704+0000","last_became_active":"2024-04-02T07:35:53.277985+0000","last_became_peered":"2024-04-02T07:35:53.277985+0000","last_unstale":"2024-04-02T07:35:54.616704+0000","last_undegraded":"2024-04-02T07:35:54.616704+0000","last_fullsized":"2024-04-02T07:35:54.616704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:42.518534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617297+0000","last_change":"2024-04-02T07:35:53.268138+0000","last_active":"2024-04-02T07:35:54.617297+0000","last_peered":"2024-04-02T07:35:54.617297+0000","last_clean":"2024-04-02T07:35:54.617297+0000","last_became_active":"2024-04-02T07:35:53.264989+0000","last_became_peered":"2024-04-02T07:35:53.264989+0000","last_unstale":"2024-04-02T07:35:54.617297+0000","last_undegraded":"2024-04-02T07:35:54.617297+0000","last_fullsized":"2024-04-02T07:35:54.617297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.380856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245556+0000","last_change":"2024-04-02T07:35:53.276292+0000","last_active":"2024-04-02T07:35:54.245556+0000","last_peered":"2024-04-02T07:35:54.245556+0000","last_clean":"2024-04-02T07:35:54.245556+0000","last_became_active":"2024-04-02T07:35:53.276003+0000","last_became_peered":"2024-04-02T07:35:53.276003+0000","last_unstale":"2024-04-02T07:35:54.245556+0000","last_undegraded":"2024-04-02T07:35:54.245556+0000","last_fullsized":"2024-04-02T07:35:54.245556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:33.477903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247301+0000","last_change":"2024-04-02T07:35:53.269437+0000","last_active":"2024-04-02T07:35:54.247301+0000","last_peered":"2024-04-02T07:35:54.247301+0000","last_clean":"2024-04-02T07:35:54.247301+0000","last_became_active":"2024-04-02T07:35:53.269192+0000","last_became_peered":"2024-04-02T07:35:53.269192+0000","last_unstale":"2024-04-02T07:35:54.247301+0000","last_undegraded":"2024-04-02T07:35:54.247301+0000","last_fullsized":"2024-04-02T07:35:54.247301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:35.381030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617797+0000","last_change":"2024-04-02T07:35:53.259373+0000","last_active":"2024-04-02T07:35:54.617797+0000","last_peered":"2024-04-02T07:35:54.617797+0000","last_clean":"2024-04-02T07:35:54.617797+0000","last_became_active":"2024-04-02T07:35:53.258986+0000","last_became_peered":"2024-04-02T07:35:53.258986+0000","last_unstale":"2024-04-02T07:35:54.617797+0000","last_undegraded":"2024-04-02T07:35:54.617797+0000","last_fullsized":"2024-04-02T07:35:54.617797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:27.051276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246472+0000","last_change":"2024-04-02T07:35:53.277181+0000","last_active":"2024-04-02T07:35:54.246472+0000","last_peered":"2024-04-02T07:35:54.246472+0000","last_clean":"2024-04-02T07:35:54.246472+0000","last_became_active":"2024-04-02T07:35:53.276207+0000","last_became_peered":"2024-04-02T07:35:53.276207+0000","last_unstale":"2024-04-02T07:35:54.246472+0000","last_undegraded":"2024-04-02T07:35:54.246472+0000","last_fullsized":"2024-04-02T07:35:54.246472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:13.487258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247380+0000","last_change":"2024-04-02T07:35:52.756587+0000","last_active":"2024-04-02T07:35:54.247380+0000","last_peered":"2024-04-02T07:35:54.247380+0000","last_clean":"2024-04-02T07:35:54.247380+0000","last_became_active":"2024-04-02T07:35:52.755981+0000","last_became_peered":"2024-04-02T07:35:52.755981+0000","last_unstale":"2024-04-02T07:35:54.247380+0000","last_undegraded":"2024-04-02T07:35:54.247380+0000","last_fullsized":"2024-04-02T07:35:54.247380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:07.324155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617519+0000","last_change":"2024-04-02T07:35:53.257824+0000","last_active":"2024-04-02T07:35:54.617519+0000","last_peered":"2024-04-02T07:35:54.617519+0000","last_clean":"2024-04-02T07:35:54.617519+0000","last_became_active":"2024-04-02T07:35:53.257500+0000","last_became_peered":"2024-04-02T07:35:53.257500+0000","last_unstale":"2024-04-02T07:35:54.617519+0000","last_undegraded":"2024-04-02T07:35:54.617519+0000","last_fullsized":"2024-04-02T07:35:54.617519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:35.709321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617620+0000","last_change":"2024-04-02T07:35:52.761793+0000","last_active":"2024-04-02T07:35:54.617620+0000","last_peered":"2024-04-02T07:35:54.617620+0000","last_clean":"2024-04-02T07:35:54.617620+0000","last_became_active":"2024-04-02T07:35:52.761510+0000","last_became_peered":"2024-04-02T07:35:52.761510+0000","last_unstale":"2024-04-02T07:35:54.617620+0000","last_undegraded":"2024-04-02T07:35:54.617620+0000","last_fullsized":"2024-04-02T07:35:54.617620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617652+0000","last_change":"2024-04-02T07:35:53.262736+0000","last_active":"2024-04-02T07:35:54.617652+0000","last_peered":"2024-04-02T07:35:54.617652+0000","last_clean":"2024-04-02T07:35:54.617652+0000","last_became_active":"2024-04-02T07:35:53.262405+0000","last_became_peered":"2024-04-02T07:35:53.262405+0000","last_unstale":"2024-04-02T07:35:54.617652+0000","last_undegraded":"2024-04-02T07:35:54.617652+0000","last_fullsized":"2024-04-02T07:35:54.617652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:52:59.688399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615557+0000","last_change":"2024-04-02T07:35:52.765725+0000","last_active":"2024-04-02T07:35:54.615557+0000","last_peered":"2024-04-02T07:35:54.615557+0000","last_clean":"2024-04-02T07:35:54.615557+0000","last_became_active":"2024-04-02T07:35:52.765393+0000","last_became_peered":"2024-04-02T07:35:52.765393+0000","last_unstale":"2024-04-02T07:35:54.615557+0000","last_undegraded":"2024-04-02T07:35:54.615557+0000","last_fullsized":"2024-04-02T07:35:54.615557+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:31.244191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246863+0000","last_change":"2024-04-02T07:35:53.255693+0000","last_active":"2024-04-02T07:35:54.246863+0000","last_peered":"2024-04-02T07:35:54.246863+0000","last_clean":"2024-04-02T07:35:54.246863+0000","last_became_active":"2024-04-02T07:35:53.254627+0000","last_became_peered":"2024-04-02T07:35:53.254627+0000","last_unstale":"2024-04-02T07:35:54.246863+0000","last_undegraded":"2024-04-02T07:35:54.246863+0000","last_fullsized":"2024-04-02T07:35:54.246863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:42.518534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615455+0000","last_change":"2024-04-02T07:35:52.762790+0000","last_active":"2024-04-02T07:35:54.615455+0000","last_peered":"2024-04-02T07:35:54.615455+0000","last_clean":"2024-04-02T07:35:54.615455+0000","last_became_active":"2024-04-02T07:35:52.762416+0000","last_became_peered":"2024-04-02T07:35:52.762416+0000","last_unstale":"2024-04-02T07:35:54.615455+0000","last_undegraded":"2024-04-02T07:35:54.615455+0000","last_fullsized":"2024-04-02T07:35:54.615455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:05.916519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246530+0000","last_change":"2024-04-02T07:35:53.274608+0000","last_active":"2024-04-02T07:35:54.246530+0000","last_peered":"2024-04-02T07:35:54.246530+0000","last_clean":"2024-04-02T07:35:54.246530+0000","last_became_active":"2024-04-02T07:35:53.274308+0000","last_became_peered":"2024-04-02T07:35:53.274308+0000","last_unstale":"2024-04-02T07:35:54.246530+0000","last_undegraded":"2024-04-02T07:35:54.246530+0000","last_fullsized":"2024-04-02T07:35:54.246530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:28:26.191159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617955+0000","last_change":"2024-04-02T07:35:52.761806+0000","last_active":"2024-04-02T07:35:54.617955+0000","last_peered":"2024-04-02T07:35:54.617955+0000","last_clean":"2024-04-02T07:35:54.617955+0000","last_became_active":"2024-04-02T07:35:52.761556+0000","last_became_peered":"2024-04-02T07:35:52.761556+0000","last_unstale":"2024-04-02T07:35:54.617955+0000","last_undegraded":"2024-04-02T07:35:54.617955+0000","last_fullsized":"2024-04-02T07:35:54.617955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246589+0000","last_change":"2024-04-02T07:35:53.262036+0000","last_active":"2024-04-02T07:35:54.246589+0000","last_peered":"2024-04-02T07:35:54.246589+0000","last_clean":"2024-04-02T07:35:54.246589+0000","last_became_active":"2024-04-02T07:35:53.259860+0000","last_became_peered":"2024-04-02T07:35:53.259860+0000","last_unstale":"2024-04-02T07:35:54.246589+0000","last_undegraded":"2024-04-02T07:35:54.246589+0000","last_fullsized":"2024-04-02T07:35:54.246589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:27.051276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245921+0000","last_change":"2024-04-02T07:35:52.773232+0000","last_active":"2024-04-02T07:35:54.245921+0000","last_peered":"2024-04-02T07:35:54.245921+0000","last_clean":"2024-04-02T07:35:54.245921+0000","last_became_active":"2024-04-02T07:35:52.772864+0000","last_became_peered":"2024-04-02T07:35:52.772864+0000","last_unstale":"2024-04-02T07:35:54.245921+0000","last_undegraded":"2024-04-02T07:35:54.245921+0000","last_fullsized":"2024-04-02T07:35:54.245921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:26.044643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247078+0000","last_change":"2024-04-02T07:35:53.274460+0000","last_active":"2024-04-02T07:35:54.247078+0000","last_peered":"2024-04-02T07:35:54.247078+0000","last_clean":"2024-04-02T07:35:54.247078+0000","last_became_active":"2024-04-02T07:35:53.274227+0000","last_became_peered":"2024-04-02T07:35:53.274227+0000","last_unstale":"2024-04-02T07:35:54.247078+0000","last_undegraded":"2024-04-02T07:35:54.247078+0000","last_fullsized":"2024-04-02T07:35:54.247078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:31.170936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616577+0000","last_change":"2024-04-02T07:35:52.769404+0000","last_active":"2024-04-02T07:35:54.616577+0000","last_peered":"2024-04-02T07:35:54.616577+0000","last_clean":"2024-04-02T07:35:54.616577+0000","last_became_active":"2024-04-02T07:35:52.769091+0000","last_became_peered":"2024-04-02T07:35:52.769091+0000","last_unstale":"2024-04-02T07:35:54.616577+0000","last_undegraded":"2024-04-02T07:35:54.616577+0000","last_fullsized":"2024-04-02T07:35:54.616577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617272+0000","last_change":"2024-04-02T07:35:53.261740+0000","last_active":"2024-04-02T07:35:54.617272+0000","last_peered":"2024-04-02T07:35:54.617272+0000","last_clean":"2024-04-02T07:35:54.617272+0000","last_became_active":"2024-04-02T07:35:53.261475+0000","last_became_peered":"2024-04-02T07:35:53.261475+0000","last_unstale":"2024-04-02T07:35:54.617272+0000","last_undegraded":"2024-04-02T07:35:54.617272+0000","last_fullsized":"2024-04-02T07:35:54.617272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:31.170936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245738+0000","last_change":"2024-04-02T07:35:52.765814+0000","last_active":"2024-04-02T07:35:54.245738+0000","last_peered":"2024-04-02T07:35:54.245738+0000","last_clean":"2024-04-02T07:35:54.245738+0000","last_became_active":"2024-04-02T07:35:52.764956+0000","last_became_peered":"2024-04-02T07:35:52.764956+0000","last_unstale":"2024-04-02T07:35:54.245738+0000","last_undegraded":"2024-04-02T07:35:54.245738+0000","last_fullsized":"2024-04-02T07:35:54.245738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:49.544958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247082+0000","last_change":"2024-04-02T07:35:53.278155+0000","last_active":"2024-04-02T07:35:54.247082+0000","last_peered":"2024-04-02T07:35:54.247082+0000","last_clean":"2024-04-02T07:35:54.247082+0000","last_became_active":"2024-04-02T07:35:53.274007+0000","last_became_peered":"2024-04-02T07:35:53.274007+0000","last_unstale":"2024-04-02T07:35:54.247082+0000","last_undegraded":"2024-04-02T07:35:54.247082+0000","last_fullsized":"2024-04-02T07:35:54.247082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:05:41.161422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246958+0000","last_change":"2024-04-02T07:35:52.759611+0000","last_active":"2024-04-02T07:35:54.246958+0000","last_peered":"2024-04-02T07:35:54.246958+0000","last_clean":"2024-04-02T07:35:54.246958+0000","last_became_active":"2024-04-02T07:35:52.759164+0000","last_became_peered":"2024-04-02T07:35:52.759164+0000","last_unstale":"2024-04-02T07:35:54.246958+0000","last_undegraded":"2024-04-02T07:35:54.246958+0000","last_fullsized":"2024-04-02T07:35:54.246958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557483+0000","last_change":"2024-04-02T07:35:53.269405+0000","last_active":"2024-04-02T07:35:54.557483+0000","last_peered":"2024-04-02T07:35:54.557483+0000","last_clean":"2024-04-02T07:35:54.557483+0000","last_became_active":"2024-04-02T07:35:53.267870+0000","last_became_peered":"2024-04-02T07:35:53.267870+0000","last_unstale":"2024-04-02T07:35:54.557483+0000","last_undegraded":"2024-04-02T07:35:54.557483+0000","last_fullsized":"2024-04-02T07:35:54.557483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:08.722219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615626+0000","last_change":"2024-04-02T07:35:52.765581+0000","last_active":"2024-04-02T07:35:54.615626+0000","last_peered":"2024-04-02T07:35:54.615626+0000","last_clean":"2024-04-02T07:35:54.615626+0000","last_became_active":"2024-04-02T07:35:52.764766+0000","last_became_peered":"2024-04-02T07:35:52.764766+0000","last_unstale":"2024-04-02T07:35:54.615626+0000","last_undegraded":"2024-04-02T07:35:54.615626+0000","last_fullsized":"2024-04-02T07:35:54.615626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:07.715585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617760+0000","last_change":"2024-04-02T07:35:53.275646+0000","last_active":"2024-04-02T07:35:54.617760+0000","last_peered":"2024-04-02T07:35:54.617760+0000","last_clean":"2024-04-02T07:35:54.617760+0000","last_became_active":"2024-04-02T07:35:53.275296+0000","last_became_peered":"2024-04-02T07:35:53.275296+0000","last_unstale":"2024-04-02T07:35:54.617760+0000","last_undegraded":"2024-04-02T07:35:54.617760+0000","last_fullsized":"2024-04-02T07:35:54.617760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:17.569969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617746+0000","last_change":"2024-04-02T07:35:52.751184+0000","last_active":"2024-04-02T07:35:54.617746+0000","last_peered":"2024-04-02T07:35:54.617746+0000","last_clean":"2024-04-02T07:35:54.617746+0000","last_became_active":"2024-04-02T07:35:52.750436+0000","last_became_peered":"2024-04-02T07:35:52.750436+0000","last_unstale":"2024-04-02T07:35:54.617746+0000","last_undegraded":"2024-04-02T07:35:54.617746+0000","last_fullsized":"2024-04-02T07:35:54.617746+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246276+0000","last_change":"2024-04-02T07:35:53.277457+0000","last_active":"2024-04-02T07:35:54.246276+0000","last_peered":"2024-04-02T07:35:54.246276+0000","last_clean":"2024-04-02T07:35:54.246276+0000","last_became_active":"2024-04-02T07:35:53.277059+0000","last_became_peered":"2024-04-02T07:35:53.277059+0000","last_unstale":"2024-04-02T07:35:54.246276+0000","last_undegraded":"2024-04-02T07:35:54.246276+0000","last_fullsized":"2024-04-02T07:35:54.246276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:18:02.329099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557196+0000","last_change":"2024-04-02T07:35:52.757551+0000","last_active":"2024-04-02T07:35:54.557196+0000","last_peered":"2024-04-02T07:35:54.557196+0000","last_clean":"2024-04-02T07:35:54.557196+0000","last_became_active":"2024-04-02T07:35:52.756930+0000","last_became_peered":"2024-04-02T07:35:52.756930+0000","last_unstale":"2024-04-02T07:35:54.557196+0000","last_undegraded":"2024-04-02T07:35:54.557196+0000","last_fullsized":"2024-04-02T07:35:54.557196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:40:29.501936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615608+0000","last_change":"2024-04-02T07:35:53.265162+0000","last_active":"2024-04-02T07:35:54.615608+0000","last_peered":"2024-04-02T07:35:54.615608+0000","last_clean":"2024-04-02T07:35:54.615608+0000","last_became_active":"2024-04-02T07:35:53.264897+0000","last_became_peered":"2024-04-02T07:35:53.264897+0000","last_unstale":"2024-04-02T07:35:54.615608+0000","last_undegraded":"2024-04-02T07:35:54.615608+0000","last_fullsized":"2024-04-02T07:35:54.615608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:35.709321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616800+0000","last_change":"2024-04-02T07:35:52.767385+0000","last_active":"2024-04-02T07:35:54.616800+0000","last_peered":"2024-04-02T07:35:54.616800+0000","last_clean":"2024-04-02T07:35:54.616800+0000","last_became_active":"2024-04-02T07:35:52.767027+0000","last_became_peered":"2024-04-02T07:35:52.767027+0000","last_unstale":"2024-04-02T07:35:54.616800+0000","last_undegraded":"2024-04-02T07:35:54.616800+0000","last_fullsized":"2024-04-02T07:35:54.616800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616462+0000","last_change":"2024-04-02T07:35:53.283307+0000","last_active":"2024-04-02T07:35:54.616462+0000","last_peered":"2024-04-02T07:35:54.616462+0000","last_clean":"2024-04-02T07:35:54.616462+0000","last_became_active":"2024-04-02T07:35:53.282062+0000","last_became_peered":"2024-04-02T07:35:53.282062+0000","last_unstale":"2024-04-02T07:35:54.616462+0000","last_undegraded":"2024-04-02T07:35:54.616462+0000","last_fullsized":"2024-04-02T07:35:54.616462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:24.367224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246052+0000","last_change":"2024-04-02T07:35:52.753164+0000","last_active":"2024-04-02T07:35:54.246052+0000","last_peered":"2024-04-02T07:35:54.246052+0000","last_clean":"2024-04-02T07:35:54.246052+0000","last_became_active":"2024-04-02T07:35:52.752227+0000","last_became_peered":"2024-04-02T07:35:52.752227+0000","last_unstale":"2024-04-02T07:35:54.246052+0000","last_undegraded":"2024-04-02T07:35:54.246052+0000","last_fullsized":"2024-04-02T07:35:54.246052+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:07.715585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246205+0000","last_change":"2024-04-02T07:35:53.269115+0000","last_active":"2024-04-02T07:35:54.246205+0000","last_peered":"2024-04-02T07:35:54.246205+0000","last_clean":"2024-04-02T07:35:54.246205+0000","last_became_active":"2024-04-02T07:35:53.268815+0000","last_became_peered":"2024-04-02T07:35:53.268815+0000","last_unstale":"2024-04-02T07:35:54.246205+0000","last_undegraded":"2024-04-02T07:35:54.246205+0000","last_fullsized":"2024-04-02T07:35:54.246205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:46:59.080243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615336+0000","last_change":"2024-04-02T07:35:52.764457+0000","last_active":"2024-04-02T07:35:54.615336+0000","last_peered":"2024-04-02T07:35:54.615336+0000","last_clean":"2024-04-02T07:35:54.615336+0000","last_became_active":"2024-04-02T07:35:52.764002+0000","last_became_peered":"2024-04-02T07:35:52.764002+0000","last_unstale":"2024-04-02T07:35:54.615336+0000","last_undegraded":"2024-04-02T07:35:54.615336+0000","last_fullsized":"2024-04-02T07:35:54.615336+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:38:44.437274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246412+0000","last_change":"2024-04-02T07:35:53.262284+0000","last_active":"2024-04-02T07:35:54.246412+0000","last_peered":"2024-04-02T07:35:54.246412+0000","last_clean":"2024-04-02T07:35:54.246412+0000","last_became_active":"2024-04-02T07:35:53.260682+0000","last_became_peered":"2024-04-02T07:35:53.260682+0000","last_unstale":"2024-04-02T07:35:54.246412+0000","last_undegraded":"2024-04-02T07:35:54.246412+0000","last_fullsized":"2024-04-02T07:35:54.246412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:55.943715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246893+0000","last_change":"2024-04-02T07:35:52.754933+0000","last_active":"2024-04-02T07:35:54.246893+0000","last_peered":"2024-04-02T07:35:54.246893+0000","last_clean":"2024-04-02T07:35:54.246893+0000","last_became_active":"2024-04-02T07:35:52.750245+0000","last_became_peered":"2024-04-02T07:35:52.750245+0000","last_unstale":"2024-04-02T07:35:54.246893+0000","last_undegraded":"2024-04-02T07:35:54.246893+0000","last_fullsized":"2024-04-02T07:35:54.246893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245859+0000","last_change":"2024-04-02T07:35:53.266981+0000","last_active":"2024-04-02T07:35:54.245859+0000","last_peered":"2024-04-02T07:35:54.245859+0000","last_clean":"2024-04-02T07:35:54.245859+0000","last_became_active":"2024-04-02T07:35:53.266194+0000","last_became_peered":"2024-04-02T07:35:53.266194+0000","last_unstale":"2024-04-02T07:35:54.245859+0000","last_undegraded":"2024-04-02T07:35:54.245859+0000","last_fullsized":"2024-04-02T07:35:54.245859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:22:27.488539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615714+0000","last_change":"2024-04-02T07:35:52.771817+0000","last_active":"2024-04-02T07:35:54.615714+0000","last_peered":"2024-04-02T07:35:54.615714+0000","last_clean":"2024-04-02T07:35:54.615714+0000","last_became_active":"2024-04-02T07:35:52.771315+0000","last_became_peered":"2024-04-02T07:35:52.771315+0000","last_unstale":"2024-04-02T07:35:54.615714+0000","last_undegraded":"2024-04-02T07:35:54.615714+0000","last_fullsized":"2024-04-02T07:35:54.615714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:07.324155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557469+0000","last_change":"2024-04-02T07:35:53.254431+0000","last_active":"2024-04-02T07:35:54.557469+0000","last_peered":"2024-04-02T07:35:54.557469+0000","last_clean":"2024-04-02T07:35:54.557469+0000","last_became_active":"2024-04-02T07:35:53.253582+0000","last_became_peered":"2024-04-02T07:35:53.253582+0000","last_unstale":"2024-04-02T07:35:54.557469+0000","last_undegraded":"2024-04-02T07:35:54.557469+0000","last_fullsized":"2024-04-02T07:35:54.557469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:32.250824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617486+0000","last_change":"2024-04-02T07:35:52.761017+0000","last_active":"2024-04-02T07:35:54.617486+0000","last_peered":"2024-04-02T07:35:54.617486+0000","last_clean":"2024-04-02T07:35:54.617486+0000","last_became_active":"2024-04-02T07:35:52.760290+0000","last_became_peered":"2024-04-02T07:35:52.760290+0000","last_unstale":"2024-04-02T07:35:54.617486+0000","last_undegraded":"2024-04-02T07:35:54.617486+0000","last_fullsized":"2024-04-02T07:35:54.617486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:45.255736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557174+0000","last_change":"2024-04-02T07:35:53.263108+0000","last_active":"2024-04-02T07:35:54.557174+0000","last_peered":"2024-04-02T07:35:54.557174+0000","last_clean":"2024-04-02T07:35:54.557174+0000","last_became_active":"2024-04-02T07:35:53.262111+0000","last_became_peered":"2024-04-02T07:35:53.262111+0000","last_unstale":"2024-04-02T07:35:54.557174+0000","last_undegraded":"2024-04-02T07:35:54.557174+0000","last_fullsized":"2024-04-02T07:35:54.557174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:27.051276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245974+0000","last_change":"2024-04-02T07:35:52.763712+0000","last_active":"2024-04-02T07:35:54.245974+0000","last_peered":"2024-04-02T07:35:54.245974+0000","last_clean":"2024-04-02T07:35:54.245974+0000","last_became_active":"2024-04-02T07:35:52.763457+0000","last_became_peered":"2024-04-02T07:35:52.763457+0000","last_unstale":"2024-04-02T07:35:54.245974+0000","last_undegraded":"2024-04-02T07:35:54.245974+0000","last_fullsized":"2024-04-02T07:35:54.245974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:41.511901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615482+0000","last_change":"2024-04-02T07:35:53.254132+0000","last_active":"2024-04-02T07:35:54.615482+0000","last_peered":"2024-04-02T07:35:54.615482+0000","last_clean":"2024-04-02T07:35:54.615482+0000","last_became_active":"2024-04-02T07:35:53.253705+0000","last_became_peered":"2024-04-02T07:35:53.253705+0000","last_unstale":"2024-04-02T07:35:54.615482+0000","last_undegraded":"2024-04-02T07:35:54.615482+0000","last_fullsized":"2024-04-02T07:35:54.615482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:17.569969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617123+0000","last_change":"2024-04-02T07:35:53.267551+0000","last_active":"2024-04-02T07:35:54.617123+0000","last_peered":"2024-04-02T07:35:54.617123+0000","last_clean":"2024-04-02T07:35:54.617123+0000","last_became_active":"2024-04-02T07:35:53.263849+0000","last_became_peered":"2024-04-02T07:35:53.263849+0000","last_unstale":"2024-04-02T07:35:54.617123+0000","last_undegraded":"2024-04-02T07:35:54.617123+0000","last_fullsized":"2024-04-02T07:35:54.617123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:54:04.724188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616708+0000","last_change":"2024-04-02T07:35:52.764685+0000","last_active":"2024-04-02T07:35:54.616708+0000","last_peered":"2024-04-02T07:35:54.616708+0000","last_clean":"2024-04-02T07:35:54.616708+0000","last_became_active":"2024-04-02T07:35:52.764421+0000","last_became_peered":"2024-04-02T07:35:52.764421+0000","last_unstale":"2024-04-02T07:35:54.616708+0000","last_undegraded":"2024-04-02T07:35:54.616708+0000","last_fullsized":"2024-04-02T07:35:54.616708+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247020+0000","last_change":"2024-04-02T07:35:53.264778+0000","last_active":"2024-04-02T07:35:54.247020+0000","last_peered":"2024-04-02T07:35:54.247020+0000","last_clean":"2024-04-02T07:35:54.247020+0000","last_became_active":"2024-04-02T07:35:53.264445+0000","last_became_peered":"2024-04-02T07:35:53.264445+0000","last_unstale":"2024-04-02T07:35:54.247020+0000","last_undegraded":"2024-04-02T07:35:54.247020+0000","last_fullsized":"2024-04-02T07:35:54.247020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:00:20.419421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246196+0000","last_change":"2024-04-02T07:35:52.767065+0000","last_active":"2024-04-02T07:35:54.246196+0000","last_peered":"2024-04-02T07:35:54.246196+0000","last_clean":"2024-04-02T07:35:54.246196+0000","last_became_active":"2024-04-02T07:35:52.764931+0000","last_became_peered":"2024-04-02T07:35:52.764931+0000","last_unstale":"2024-04-02T07:35:54.246196+0000","last_undegraded":"2024-04-02T07:35:54.246196+0000","last_fullsized":"2024-04-02T07:35:54.246196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:05.916519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-02T07:35:53.250580+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:35:53.250580+0000","last_peered":"2024-04-02T07:35:53.250580+0000","last_clean":"2024-04-02T07:35:53.250580+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:35:53.250580+0000","last_undegraded":"2024-04-02T07:35:53.250580+0000","last_fullsized":"2024-04-02T07:35:53.250580+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616550+0000","last_change":"2024-04-02T07:35:53.283251+0000","last_active":"2024-04-02T07:35:54.616550+0000","last_peered":"2024-04-02T07:35:54.616550+0000","last_clean":"2024-04-02T07:35:54.616550+0000","last_became_active":"2024-04-02T07:35:53.276591+0000","last_became_peered":"2024-04-02T07:35:53.276591+0000","last_unstale":"2024-04-02T07:35:54.616550+0000","last_undegraded":"2024-04-02T07:35:54.616550+0000","last_fullsized":"2024-04-02T07:35:54.616550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:07.528924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245695+0000","last_change":"2024-04-02T07:35:52.764279+0000","last_active":"2024-04-02T07:35:54.245695+0000","last_peered":"2024-04-02T07:35:54.245695+0000","last_clean":"2024-04-02T07:35:54.245695+0000","last_became_active":"2024-04-02T07:35:52.759649+0000","last_became_peered":"2024-04-02T07:35:52.759649+0000","last_unstale":"2024-04-02T07:35:54.245695+0000","last_undegraded":"2024-04-02T07:35:54.245695+0000","last_fullsized":"2024-04-02T07:35:54.245695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:54.937081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246537+0000","last_change":"2024-04-02T07:35:52.756573+0000","last_active":"2024-04-02T07:35:54.246537+0000","last_peered":"2024-04-02T07:35:54.246537+0000","last_clean":"2024-04-02T07:35:54.246537+0000","last_became_active":"2024-04-02T07:35:52.756192+0000","last_became_peered":"2024-04-02T07:35:52.756192+0000","last_unstale":"2024-04-02T07:35:54.246537+0000","last_undegraded":"2024-04-02T07:35:54.246537+0000","last_fullsized":"2024-04-02T07:35:54.246537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616866+0000","last_change":"2024-04-02T07:35:53.269008+0000","last_active":"2024-04-02T07:35:54.616866+0000","last_peered":"2024-04-02T07:35:54.616866+0000","last_clean":"2024-04-02T07:35:54.616866+0000","last_became_active":"2024-04-02T07:35:53.268663+0000","last_became_peered":"2024-04-02T07:35:53.268663+0000","last_unstale":"2024-04-02T07:35:54.616866+0000","last_undegraded":"2024-04-02T07:35:54.616866+0000","last_fullsized":"2024-04-02T07:35:54.616866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:05:41.161422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:56.254933+0000","last_change":"2024-04-02T07:35:52.769059+0000","last_active":"2024-04-02T07:35:56.254933+0000","last_peered":"2024-04-02T07:35:56.254933+0000","last_clean":"2024-04-02T07:35:56.254933+0000","last_became_active":"2024-04-02T07:35:52.768711+0000","last_became_peered":"2024-04-02T07:35:52.768711+0000","last_unstale":"2024-04-02T07:35:56.254933+0000","last_undegraded":"2024-04-02T07:35:56.254933+0000","last_fullsized":"2024-04-02T07:35:56.254933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247358+0000","last_change":"2024-04-02T07:35:53.275607+0000","last_active":"2024-04-02T07:35:54.247358+0000","last_peered":"2024-04-02T07:35:54.247358+0000","last_clean":"2024-04-02T07:35:54.247358+0000","last_became_active":"2024-04-02T07:35:53.274591+0000","last_became_peered":"2024-04-02T07:35:53.274591+0000","last_unstale":"2024-04-02T07:35:54.247358+0000","last_undegraded":"2024-04-02T07:35:54.247358+0000","last_fullsized":"2024-04-02T07:35:54.247358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:13:58.887392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247393+0000","last_change":"2024-04-02T07:35:52.771999+0000","last_active":"2024-04-02T07:35:54.247393+0000","last_peered":"2024-04-02T07:35:54.247393+0000","last_clean":"2024-04-02T07:35:54.247393+0000","last_became_active":"2024-04-02T07:35:52.771720+0000","last_became_peered":"2024-04-02T07:35:52.771720+0000","last_unstale":"2024-04-02T07:35:54.247393+0000","last_undegraded":"2024-04-02T07:35:54.247393+0000","last_fullsized":"2024-04-02T07:35:54.247393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:31.244191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247153+0000","last_change":"2024-04-02T07:35:53.258654+0000","last_active":"2024-04-02T07:35:54.247153+0000","last_peered":"2024-04-02T07:35:54.247153+0000","last_clean":"2024-04-02T07:35:54.247153+0000","last_became_active":"2024-04-02T07:35:53.257658+0000","last_became_peered":"2024-04-02T07:35:53.257658+0000","last_unstale":"2024-04-02T07:35:54.247153+0000","last_undegraded":"2024-04-02T07:35:54.247153+0000","last_fullsized":"2024-04-02T07:35:54.247153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:07.528924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615282+0000","last_change":"2024-04-02T07:35:52.764308+0000","last_active":"2024-04-02T07:35:54.615282+0000","last_peered":"2024-04-02T07:35:54.615282+0000","last_clean":"2024-04-02T07:35:54.615282+0000","last_became_active":"2024-04-02T07:35:52.763178+0000","last_became_peered":"2024-04-02T07:35:52.763178+0000","last_unstale":"2024-04-02T07:35:54.615282+0000","last_undegraded":"2024-04-02T07:35:54.615282+0000","last_fullsized":"2024-04-02T07:35:54.615282+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:45.255736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245826+0000","last_change":"2024-04-02T07:35:53.258575+0000","last_active":"2024-04-02T07:35:54.245826+0000","last_peered":"2024-04-02T07:35:54.245826+0000","last_clean":"2024-04-02T07:35:54.245826+0000","last_became_active":"2024-04-02T07:35:53.258326+0000","last_became_peered":"2024-04-02T07:35:53.258326+0000","last_unstale":"2024-04-02T07:35:54.245826+0000","last_undegraded":"2024-04-02T07:35:54.245826+0000","last_fullsized":"2024-04-02T07:35:54.245826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:16.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247051+0000","last_change":"2024-04-02T07:35:52.771988+0000","last_active":"2024-04-02T07:35:54.247051+0000","last_peered":"2024-04-02T07:35:54.247051+0000","last_clean":"2024-04-02T07:35:54.247051+0000","last_became_active":"2024-04-02T07:35:52.771616+0000","last_became_peered":"2024-04-02T07:35:52.771616+0000","last_unstale":"2024-04-02T07:35:54.247051+0000","last_undegraded":"2024-04-02T07:35:54.247051+0000","last_fullsized":"2024-04-02T07:35:54.247051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:45.255736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246904+0000","last_change":"2024-04-02T07:35:53.258346+0000","last_active":"2024-04-02T07:35:54.246904+0000","last_peered":"2024-04-02T07:35:54.246904+0000","last_clean":"2024-04-02T07:35:54.246904+0000","last_became_active":"2024-04-02T07:35:53.258044+0000","last_became_peered":"2024-04-02T07:35:53.258044+0000","last_unstale":"2024-04-02T07:35:54.246904+0000","last_undegraded":"2024-04-02T07:35:54.246904+0000","last_fullsized":"2024-04-02T07:35:54.246904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:50.551591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616574+0000","last_change":"2024-04-02T07:35:52.758632+0000","last_active":"2024-04-02T07:35:54.616574+0000","last_peered":"2024-04-02T07:35:54.616574+0000","last_clean":"2024-04-02T07:35:54.616574+0000","last_became_active":"2024-04-02T07:35:52.758347+0000","last_became_peered":"2024-04-02T07:35:52.758347+0000","last_unstale":"2024-04-02T07:35:54.616574+0000","last_undegraded":"2024-04-02T07:35:54.616574+0000","last_fullsized":"2024-04-02T07:35:54.616574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.565859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617044+0000","last_change":"2024-04-02T07:35:53.271892+0000","last_active":"2024-04-02T07:35:54.617044+0000","last_peered":"2024-04-02T07:35:54.617044+0000","last_clean":"2024-04-02T07:35:54.617044+0000","last_became_active":"2024-04-02T07:35:53.271472+0000","last_became_peered":"2024-04-02T07:35:53.271472+0000","last_unstale":"2024-04-02T07:35:54.617044+0000","last_undegraded":"2024-04-02T07:35:54.617044+0000","last_fullsized":"2024-04-02T07:35:54.617044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:06.923152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246267+0000","last_change":"2024-04-02T07:35:52.758627+0000","last_active":"2024-04-02T07:35:54.246267+0000","last_peered":"2024-04-02T07:35:54.246267+0000","last_clean":"2024-04-02T07:35:54.246267+0000","last_became_active":"2024-04-02T07:35:52.758330+0000","last_became_peered":"2024-04-02T07:35:52.758330+0000","last_unstale":"2024-04-02T07:35:54.246267+0000","last_undegraded":"2024-04-02T07:35:54.246267+0000","last_fullsized":"2024-04-02T07:35:54.246267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:23.360591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245991+0000","last_change":"2024-04-02T07:35:53.266441+0000","last_active":"2024-04-02T07:35:54.245991+0000","last_peered":"2024-04-02T07:35:54.245991+0000","last_clean":"2024-04-02T07:35:54.245991+0000","last_became_active":"2024-04-02T07:35:53.265569+0000","last_became_peered":"2024-04-02T07:35:53.265569+0000","last_unstale":"2024-04-02T07:35:54.245991+0000","last_undegraded":"2024-04-02T07:35:54.245991+0000","last_fullsized":"2024-04-02T07:35:54.245991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:53.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616939+0000","last_change":"2024-04-02T07:35:52.762509+0000","last_active":"2024-04-02T07:35:54.616939+0000","last_peered":"2024-04-02T07:35:54.616939+0000","last_clean":"2024-04-02T07:35:54.616939+0000","last_became_active":"2024-04-02T07:35:52.762190+0000","last_became_peered":"2024-04-02T07:35:52.762190+0000","last_unstale":"2024-04-02T07:35:54.616939+0000","last_undegraded":"2024-04-02T07:35:54.616939+0000","last_fullsized":"2024-04-02T07:35:54.616939+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:07.831701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246626+0000","last_change":"2024-04-02T07:35:53.272664+0000","last_active":"2024-04-02T07:35:54.246626+0000","last_peered":"2024-04-02T07:35:54.246626+0000","last_clean":"2024-04-02T07:35:54.246626+0000","last_became_active":"2024-04-02T07:35:53.272408+0000","last_became_peered":"2024-04-02T07:35:53.272408+0000","last_unstale":"2024-04-02T07:35:54.246626+0000","last_undegraded":"2024-04-02T07:35:54.246626+0000","last_fullsized":"2024-04-02T07:35:54.246626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:18.703428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617830+0000","last_change":"2024-04-02T07:35:52.760241+0000","last_active":"2024-04-02T07:35:54.617830+0000","last_peered":"2024-04-02T07:35:54.617830+0000","last_clean":"2024-04-02T07:35:54.617830+0000","last_became_active":"2024-04-02T07:35:52.759523+0000","last_became_peered":"2024-04-02T07:35:52.759523+0000","last_unstale":"2024-04-02T07:35:54.617830+0000","last_undegraded":"2024-04-02T07:35:54.617830+0000","last_fullsized":"2024-04-02T07:35:54.617830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:07.324155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615417+0000","last_change":"2024-04-02T07:35:53.252444+0000","last_active":"2024-04-02T07:35:54.615417+0000","last_peered":"2024-04-02T07:35:54.615417+0000","last_clean":"2024-04-02T07:35:54.615417+0000","last_became_active":"2024-04-02T07:35:53.252160+0000","last_became_peered":"2024-04-02T07:35:53.252160+0000","last_unstale":"2024-04-02T07:35:54.615417+0000","last_undegraded":"2024-04-02T07:35:54.615417+0000","last_fullsized":"2024-04-02T07:35:54.615417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:18:02.329099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247287+0000","last_change":"2024-04-02T07:35:52.760163+0000","last_active":"2024-04-02T07:35:54.247287+0000","last_peered":"2024-04-02T07:35:54.247287+0000","last_clean":"2024-04-02T07:35:54.247287+0000","last_became_active":"2024-04-02T07:35:52.759785+0000","last_became_peered":"2024-04-02T07:35:52.759785+0000","last_unstale":"2024-04-02T07:35:54.247287+0000","last_undegraded":"2024-04-02T07:35:54.247287+0000","last_fullsized":"2024-04-02T07:35:54.247287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:51.723998+0000","last_clean_scrub_stamp":"2024-04-02T07:35:51.723998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:06:31.778694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615799+0000","last_change":"2024-04-02T07:35:53.259830+0000","last_active":"2024-04-02T07:35:54.615799+0000","last_peered":"2024-04-02T07:35:54.615799+0000","last_clean":"2024-04-02T07:35:54.615799+0000","last_became_active":"2024-04-02T07:35:53.259518+0000","last_became_peered":"2024-04-02T07:35:53.259518+0000","last_unstale":"2024-04-02T07:35:54.615799+0000","last_undegraded":"2024-04-02T07:35:54.615799+0000","last_fullsized":"2024-04-02T07:35:54.615799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:54:41.930328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.556975+0000","last_change":"2024-04-02T07:35:53.263050+0000","last_active":"2024-04-02T07:35:54.556975+0000","last_peered":"2024-04-02T07:35:54.556975+0000","last_clean":"2024-04-02T07:35:54.556975+0000","last_became_active":"2024-04-02T07:35:53.262790+0000","last_became_peered":"2024-04-02T07:35:53.262790+0000","last_unstale":"2024-04-02T07:35:54.556975+0000","last_undegraded":"2024-04-02T07:35:54.556975+0000","last_fullsized":"2024-04-02T07:35:54.556975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:42.518534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616668+0000","last_change":"2024-04-02T07:35:53.269409+0000","last_active":"2024-04-02T07:35:54.616668+0000","last_peered":"2024-04-02T07:35:54.616668+0000","last_clean":"2024-04-02T07:35:54.616668+0000","last_became_active":"2024-04-02T07:35:53.267892+0000","last_became_peered":"2024-04-02T07:35:53.267892+0000","last_unstale":"2024-04-02T07:35:54.616668+0000","last_undegraded":"2024-04-02T07:35:54.616668+0000","last_fullsized":"2024-04-02T07:35:54.616668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:08.722219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245810+0000","last_change":"2024-04-02T07:35:53.260570+0000","last_active":"2024-04-02T07:35:54.245810+0000","last_peered":"2024-04-02T07:35:54.245810+0000","last_clean":"2024-04-02T07:35:54.245810+0000","last_became_active":"2024-04-02T07:35:53.260265+0000","last_became_peered":"2024-04-02T07:35:53.260265+0000","last_unstale":"2024-04-02T07:35:54.245810+0000","last_undegraded":"2024-04-02T07:35:54.245810+0000","last_fullsized":"2024-04-02T07:35:54.245810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:13:55.860837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617586+0000","last_change":"2024-04-02T07:35:53.260933+0000","last_active":"2024-04-02T07:35:54.617586+0000","last_peered":"2024-04-02T07:35:54.617586+0000","last_clean":"2024-04-02T07:35:54.617586+0000","last_became_active":"2024-04-02T07:35:53.260662+0000","last_became_peered":"2024-04-02T07:35:53.260662+0000","last_unstale":"2024-04-02T07:35:54.617586+0000","last_undegraded":"2024-04-02T07:35:54.617586+0000","last_fullsized":"2024-04-02T07:35:54.617586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:16.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616780+0000","last_change":"2024-04-02T07:35:53.274324+0000","last_active":"2024-04-02T07:35:54.616780+0000","last_peered":"2024-04-02T07:35:54.616780+0000","last_clean":"2024-04-02T07:35:54.616780+0000","last_became_active":"2024-04-02T07:35:53.274049+0000","last_became_peered":"2024-04-02T07:35:53.274049+0000","last_unstale":"2024-04-02T07:35:54.616780+0000","last_undegraded":"2024-04-02T07:35:54.616780+0000","last_fullsized":"2024-04-02T07:35:54.616780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:54:04.724188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.616584+0000","last_change":"2024-04-02T07:35:53.272852+0000","last_active":"2024-04-02T07:35:54.616584+0000","last_peered":"2024-04-02T07:35:54.616584+0000","last_clean":"2024-04-02T07:35:54.616584+0000","last_became_active":"2024-04-02T07:35:53.272036+0000","last_became_peered":"2024-04-02T07:35:53.272036+0000","last_unstale":"2024-04-02T07:35:54.616584+0000","last_undegraded":"2024-04-02T07:35:54.616584+0000","last_fullsized":"2024-04-02T07:35:54.616584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:50.551591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.557432+0000","last_change":"2024-04-02T07:35:53.275573+0000","last_active":"2024-04-02T07:35:54.557432+0000","last_peered":"2024-04-02T07:35:54.557432+0000","last_clean":"2024-04-02T07:35:54.557432+0000","last_became_active":"2024-04-02T07:35:53.272553+0000","last_became_peered":"2024-04-02T07:35:53.272553+0000","last_unstale":"2024-04-02T07:35:54.557432+0000","last_undegraded":"2024-04-02T07:35:54.557432+0000","last_fullsized":"2024-04-02T07:35:54.557432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:06.923152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617508+0000","last_change":"2024-04-02T07:35:53.262359+0000","last_active":"2024-04-02T07:35:54.617508+0000","last_peered":"2024-04-02T07:35:54.617508+0000","last_clean":"2024-04-02T07:35:54.617508+0000","last_became_active":"2024-04-02T07:35:53.262038+0000","last_became_peered":"2024-04-02T07:35:53.262038+0000","last_unstale":"2024-04-02T07:35:54.617508+0000","last_undegraded":"2024-04-02T07:35:54.617508+0000","last_fullsized":"2024-04-02T07:35:54.617508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:18.703428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.245590+0000","last_change":"2024-04-02T07:35:53.284523+0000","last_active":"2024-04-02T07:35:54.245590+0000","last_peered":"2024-04-02T07:35:54.245590+0000","last_clean":"2024-04-02T07:35:54.245590+0000","last_became_active":"2024-04-02T07:35:53.284200+0000","last_became_peered":"2024-04-02T07:35:53.284200+0000","last_unstale":"2024-04-02T07:35:54.245590+0000","last_undegraded":"2024-04-02T07:35:54.245590+0000","last_fullsized":"2024-04-02T07:35:54.245590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:14:43.088676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.617379+0000","last_change":"2024-04-02T07:35:53.259326+0000","last_active":"2024-04-02T07:35:54.617379+0000","last_peered":"2024-04-02T07:35:54.617379+0000","last_clean":"2024-04-02T07:35:54.617379+0000","last_became_active":"2024-04-02T07:35:53.259023+0000","last_became_peered":"2024-04-02T07:35:53.259023+0000","last_unstale":"2024-04-02T07:35:54.617379+0000","last_undegraded":"2024-04-02T07:35:54.617379+0000","last_fullsized":"2024-04-02T07:35:54.617379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:54:41.930328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.246691+0000","last_change":"2024-04-02T07:35:53.261005+0000","last_active":"2024-04-02T07:35:54.246691+0000","last_peered":"2024-04-02T07:35:54.246691+0000","last_clean":"2024-04-02T07:35:54.246691+0000","last_became_active":"2024-04-02T07:35:53.259618+0000","last_became_peered":"2024-04-02T07:35:53.259618+0000","last_unstale":"2024-04-02T07:35:54.246691+0000","last_undegraded":"2024-04-02T07:35:54.246691+0000","last_fullsized":"2024-04-02T07:35:54.246691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:24.367224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.247226+0000","last_change":"2024-04-02T07:35:53.262531+0000","last_active":"2024-04-02T07:35:54.247226+0000","last_peered":"2024-04-02T07:35:54.247226+0000","last_clean":"2024-04-02T07:35:54.247226+0000","last_became_active":"2024-04-02T07:35:53.262196+0000","last_became_peered":"2024-04-02T07:35:53.262196+0000","last_unstale":"2024-04-02T07:35:54.247226+0000","last_undegraded":"2024-04-02T07:35:54.247226+0000","last_fullsized":"2024-04-02T07:35:54.247226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:54:04.724188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-02T07:35:54.615537+0000","last_change":"2024-04-02T07:35:53.266761+0000","last_active":"2024-04-02T07:35:54.615537+0000","last_peered":"2024-04-02T07:35:54.615537+0000","last_clean":"2024-04-02T07:35:54.615537+0000","last_became_active":"2024-04-02T07:35:53.264360+0000","last_became_peered":"2024-04-02T07:35:53.264360+0000","last_unstale":"2024-04-02T07:35:54.615537+0000","last_undegraded":"2024-04-02T07:35:54.615537+0000","last_fullsized":"2024-04-02T07:35:54.615537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:52.730631+0000","last_clean_scrub_stamp":"2024-04-02T07:35:52.730631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:01:31.170936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9079999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0750000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9430000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7969999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9690000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.764}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.706}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1640000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.125}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.4800000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.0899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7539999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.931}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1580000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":71480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.476}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2839999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7310000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4470000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2909999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":73134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":660339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:36:03.782 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-02T07:36:03.782 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-02T07:36:03.782 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-02T07:36:03.782 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-02T07:36:04.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.004+0000 7f411fa00700 1 Processor -- start 2024-04-02T07:36:04.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.004+0000 7f411fa00700 1 -- start start 2024-04-02T07:36:04.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.005+0000 7f411fa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4118160f10 0x7f41180d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.005+0000 7f411fa00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4118161590 0x7f4118161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.005+0000 7f411fa00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41180ceda0 con 0x7f41180cf230 2024-04-02T07:36:04.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.005+0000 7f411fa00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4118161f80 con 0x7f4118161590 2024-04-02T07:36:04.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.005+0000 7f411fa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41181612f0 con 0x7f4118160f10 2024-04-02T07:36:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.005+0000 7f411d000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4118160f10 0x7f41180d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.005+0000 7f411da00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4118161590 0x7f4118161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.005+0000 7f4117e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f41180cf230 0x7f41180d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:39776/0 (socket says 172.21.15.89:39776) 2024-04-02T07:36:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.005+0000 7f4117e00700 1 -- 172.21.15.89:0/4277976770 learned_addr learned my addr 172.21.15.89:0/4277976770 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.006+0000 7f4117400700 1 -- 172.21.15.89:0/4277976770 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408906744 0 0) 0x7f41180ceda0 con 0x7f41180cf230 2024-04-02T07:36:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.006+0000 7f411da00700 1 -- 172.21.15.89:0/4277976770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4118160f10 msgr2=0x7f41180d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.006+0000 7f411da00700 1 --2- 172.21.15.89:0/4277976770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4118160f10 0x7f41180d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.006+0000 7f411da00700 1 -- 172.21.15.89:0/4277976770 >> v1:172.21.15.113:6789/0 conn(0x7f41180cf230 legacy=0x7f41180d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.006+0000 7f411da00700 1 -- 172.21.15.89:0/4277976770 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41180ce470 con 0x7f4118161590 2024-04-02T07:36:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.006+0000 7f411d000700 1 --2- 172.21.15.89:0/4277976770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4118160f10 0x7f41180d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.006+0000 7f411da00700 1 --2- 172.21.15.89:0/4277976770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4118161590 0x7f4118161a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4110009dc0 tx=0x7f411000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e81805b761f4413 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.006+0000 7f4117400700 1 -- 172.21.15.89:0/4277976770 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411000e070 con 0x7f4118161590 2024-04-02T07:36:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.007+0000 7f4117400700 1 -- 172.21.15.89:0/4277976770 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411000bae0 con 0x7f4118161590 2024-04-02T07:36:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.007+0000 7f4117400700 1 -- 172.21.15.89:0/4277976770 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411000be90 con 0x7f4118161590 2024-04-02T07:36:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.007+0000 7f411fa00700 1 -- 172.21.15.89:0/4277976770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4118161590 msgr2=0x7f4118161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.007+0000 7f411fa00700 1 --2- 172.21.15.89:0/4277976770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4118161590 0x7f4118161a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4110009dc0 tx=0x7f411000b330 comp rx=0 tx=0).stop 2024-04-02T07:36:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.007+0000 7f411fa00700 1 -- 172.21.15.89:0/4277976770 shutdown_connections 2024-04-02T07:36:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.007+0000 7f411fa00700 1 --2- 172.21.15.89:0/4277976770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4118161590 0x7f4118161a40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.007+0000 7f411fa00700 1 --2- 172.21.15.89:0/4277976770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4118160f10 0x7f41180d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.007+0000 7f411fa00700 1 -- 172.21.15.89:0/4277976770 >> 172.21.15.89:0/4277976770 conn(0x7f411808aff0 msgr2=0x7f41180d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.007+0000 7f411fa00700 1 -- 172.21.15.89:0/4277976770 shutdown_connections 2024-04-02T07:36:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.007+0000 7f411fa00700 1 -- 172.21.15.89:0/4277976770 wait complete. 2024-04-02T07:36:04.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.009+0000 7f411fa00700 1 Processor -- start 2024-04-02T07:36:04.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.009+0000 7f411fa00700 1 -- start start 2024-04-02T07:36:04.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f411fa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41180cf230 0x7f411814eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f411fa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4118160f10 0x7f411815dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f411fa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4118161590 0x7f411809e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f411fa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411814f080 con 0x7f4118160f10 2024-04-02T07:36:04.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f411fa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411815e5f0 con 0x7f4118161590 2024-04-02T07:36:04.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f411fa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411815e8d0 con 0x7f41180cf230 2024-04-02T07:36:04.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f4117e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4118160f10 0x7f411815dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f4117e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4118160f10 0x7f411815dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52768/0 (socket says 172.21.15.89:52768) 2024-04-02T07:36:04.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f4117e00700 1 -- 172.21.15.89:0/4170540042 learned_addr learned my addr 172.21.15.89:0/4170540042 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:04.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f411d000700 1 --2- 172.21.15.89:0/4170540042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41180cf230 0x7f411814eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f411da00700 1 --2- 172.21.15.89:0/4170540042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4118161590 0x7f411809e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f4117e00700 1 -- 172.21.15.89:0/4170540042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41180cf230 msgr2=0x7f411814eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f4117e00700 1 --2- 172.21.15.89:0/4170540042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41180cf230 0x7f411814eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f4117e00700 1 -- 172.21.15.89:0/4170540042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4118161590 msgr2=0x7f411809e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.010+0000 7f4117e00700 1 --2- 172.21.15.89:0/4170540042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4118161590 0x7f411809e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.011+0000 7f4117e00700 1 -- 172.21.15.89:0/4170540042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c0094a0 con 0x7f4118160f10 2024-04-02T07:36:04.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.011+0000 7f4117e00700 1 --2- 172.21.15.89:0/4170540042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4118160f10 0x7f411815dd30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f410400c520 tx=0x7f410400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:04.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.011+0000 7f4114c00700 1 -- 172.21.15.89:0/4170540042 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104004c80 con 0x7f4118160f10 2024-04-02T07:36:04.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.011+0000 7f4114c00700 1 -- 172.21.15.89:0/4170540042 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4104004de0 con 0x7f4118160f10 2024-04-02T07:36:04.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.011+0000 7f4114c00700 1 -- 172.21.15.89:0/4170540042 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040072d0 con 0x7f4118160f10 2024-04-02T07:36:04.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.011+0000 7f411fa00700 1 -- 172.21.15.89:0/4170540042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4110009a70 con 0x7f4118160f10 2024-04-02T07:36:04.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.011+0000 7f411fa00700 1 -- 172.21.15.89:0/4170540042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f411815ee90 con 0x7f4118160f10 2024-04-02T07:36:04.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.012+0000 7f411fa00700 1 -- 172.21.15.89:0/4170540042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4118067720 con 0x7f4118160f10 2024-04-02T07:36:04.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.013+0000 7f4114c00700 1 -- 172.21.15.89:0/4170540042 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4104007430 con 0x7f4118160f10 2024-04-02T07:36:04.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.014+0000 7f4114c00700 1 --2- 172.21.15.89:0/4170540042 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40f806ac90 0x7f40f806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.014+0000 7f4114c00700 1 -- 172.21.15.89:0/4170540042 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4104088eb0 con 0x7f4118160f10 2024-04-02T07:36:04.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.015+0000 7f411d000700 1 --2- 172.21.15.89:0/4170540042 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40f806ac90 0x7f40f806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.015+0000 7f411d000700 1 --2- 172.21.15.89:0/4170540042 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40f806ac90 0x7f40f806d150 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f41180c9a20 tx=0x7f410c009210 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:04.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.019+0000 7f4114c00700 1 -- 172.21.15.89:0/4170540042 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4104054080 con 0x7f4118160f10 2024-04-02T07:36:04.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.259+0000 7f411fa00700 1 -- 172.21.15.89:0/4170540042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f411815db10 con 0x7f4118160f10 2024-04-02T07:36:04.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.260+0000 7f4114c00700 1 -- 172.21.15.89:0/4170540042 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f41040576a0 con 0x7f4118160f10 2024-04-02T07:36:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:04.262 INFO:teuthology.orchestra.run.smithi089.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-02T07:36:04.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.262+0000 7f411fa00700 1 -- 172.21.15.89:0/4170540042 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40f806ac90 msgr2=0x7f40f806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.262+0000 7f411fa00700 1 --2- 172.21.15.89:0/4170540042 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40f806ac90 0x7f40f806d150 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f41180c9a20 tx=0x7f410c009210 comp rx=0 tx=0).stop 2024-04-02T07:36:04.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.262+0000 7f411fa00700 1 -- 172.21.15.89:0/4170540042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4118160f10 msgr2=0x7f411815dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.262+0000 7f411fa00700 1 --2- 172.21.15.89:0/4170540042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4118160f10 0x7f411815dd30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f410400c520 tx=0x7f410400c830 comp rx=0 tx=0).stop 2024-04-02T07:36:04.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.262+0000 7f411fa00700 1 -- 172.21.15.89:0/4170540042 shutdown_connections 2024-04-02T07:36:04.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.262+0000 7f411fa00700 1 --2- 172.21.15.89:0/4170540042 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40f806ac90 0x7f40f806d150 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.262+0000 7f411fa00700 1 --2- 172.21.15.89:0/4170540042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4118161590 0x7f411809e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.262+0000 7f411fa00700 1 --2- 172.21.15.89:0/4170540042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41180cf230 0x7f411814eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.262+0000 7f411fa00700 1 --2- 172.21.15.89:0/4170540042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4118160f10 0x7f411815dd30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.262+0000 7f411fa00700 1 -- 172.21.15.89:0/4170540042 >> 172.21.15.89:0/4170540042 conn(0x7f411808aff0 msgr2=0x7f41180cba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:04.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.262+0000 7f411fa00700 1 -- 172.21.15.89:0/4170540042 shutdown_connections 2024-04-02T07:36:04.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.262+0000 7f411fa00700 1 -- 172.21.15.89:0/4170540042 wait complete. 2024-04-02T07:36:04.282 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-02T07:36:04.282 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:36:04.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.472+0000 7f853b000700 1 Processor -- start 2024-04-02T07:36:04.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.472+0000 7f853b000700 1 -- start start 2024-04-02T07:36:04.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.473+0000 7f853b000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f853415d640 0x7f85340d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.473+0000 7f853b000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85341619d0 0x7f85340b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.473+0000 7f853b000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85340d30d0 con 0x7f85340ceec0 2024-04-02T07:36:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.473+0000 7f853b000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85340b6840 con 0x7f85341619d0 2024-04-02T07:36:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.473+0000 7f853b000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85340d2880 con 0x7f853415d640 2024-04-02T07:36:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.474+0000 7f8539000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85341619d0 0x7f85340b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.474+0000 7f8539000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85341619d0 0x7f85340b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52776/0 (socket says 172.21.15.89:52776) 2024-04-02T07:36:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.474+0000 7f8539000700 1 -- 172.21.15.89:0/3119466870 learned_addr learned my addr 172.21.15.89:0/3119466870 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.474+0000 7f8533400700 1 --1- 172.21.15.89:0/3119466870 >> v1:172.21.15.89:6789/0 conn(0x7f85340ceec0 0x7f85341612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34954/0 (socket says 172.21.15.89:34954) 2024-04-02T07:36:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.474+0000 7f8533e00700 1 --2- 172.21.15.89:0/3119466870 >> v2:172.21.15.113:3300/0 conn(0x7f853415d640 0x7f85340d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.474+0000 7f8539000700 1 -- 172.21.15.89:0/3119466870 >> v2:172.21.15.113:3300/0 conn(0x7f853415d640 msgr2=0x7f85340d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.474+0000 7f8539000700 1 --2- 172.21.15.89:0/3119466870 >> v2:172.21.15.113:3300/0 conn(0x7f853415d640 0x7f85340d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.474+0000 7f8539000700 1 -- 172.21.15.89:0/3119466870 >> v1:172.21.15.89:6789/0 conn(0x7f85340ceec0 legacy=0x7f85341612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.474+0000 7f8539000700 1 -- 172.21.15.89:0/3119466870 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85340d2b30 con 0x7f85341619d0 2024-04-02T07:36:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.474+0000 7f8532a00700 1 -- 172.21.15.89:0/3119466870 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305655057 0 0) 0x7f85340d30d0 con 0x7f85340ceec0 2024-04-02T07:36:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.474+0000 7f8539000700 1 --2- 172.21.15.89:0/3119466870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85341619d0 0x7f85340b6cd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f85340b5c20 tx=0x7f852c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c51aa8dd9f8d67 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:04.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.475+0000 7f8532a00700 1 -- 172.21.15.89:0/3119466870 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c00e070 con 0x7f85341619d0 2024-04-02T07:36:04.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.475+0000 7f8532a00700 1 -- 172.21.15.89:0/3119466870 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c00b9c0 con 0x7f85341619d0 2024-04-02T07:36:04.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.475+0000 7f8532a00700 1 -- 172.21.15.89:0/3119466870 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c00bd70 con 0x7f85341619d0 2024-04-02T07:36:04.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.475+0000 7f853b000700 1 -- 172.21.15.89:0/3119466870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85341619d0 msgr2=0x7f85340b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.475+0000 7f853b000700 1 --2- 172.21.15.89:0/3119466870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85341619d0 0x7f85340b6cd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f85340b5c20 tx=0x7f852c00b330 comp rx=0 tx=0).stop 2024-04-02T07:36:04.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.475+0000 7f853b000700 1 -- 172.21.15.89:0/3119466870 shutdown_connections 2024-04-02T07:36:04.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.475+0000 7f853b000700 1 --2- 172.21.15.89:0/3119466870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85341619d0 0x7f85340b6cd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.475+0000 7f853b000700 1 --2- 172.21.15.89:0/3119466870 >> v2:172.21.15.113:3300/0 conn(0x7f853415d640 0x7f85340d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.475+0000 7f853b000700 1 -- 172.21.15.89:0/3119466870 >> 172.21.15.89:0/3119466870 conn(0x7f853408aff0 msgr2=0x7f85340d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:04.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.475+0000 7f853b000700 1 -- 172.21.15.89:0/3119466870 shutdown_connections 2024-04-02T07:36:04.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.475+0000 7f853b000700 1 -- 172.21.15.89:0/3119466870 wait complete. 2024-04-02T07:36:04.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.477+0000 7f853b000700 1 Processor -- start 2024-04-02T07:36:04.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.477+0000 7f853b000700 1 -- start start 2024-04-02T07:36:04.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.478+0000 7f853b000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85340ceec0 0x7f8534151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.478+0000 7f853b000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f853415d640 0x7f8534152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.478+0000 7f853b000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85341619d0 0x7f85341f6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.478+0000 7f853b000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85341606f0 con 0x7f85341619d0 2024-04-02T07:36:04.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.478+0000 7f853b000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853415f820 con 0x7f85340ceec0 2024-04-02T07:36:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.478+0000 7f853b000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85341f6e90 con 0x7f853415d640 2024-04-02T07:36:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.478+0000 7f8539000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85341619d0 0x7f85341f6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.478+0000 7f8539000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85341619d0 0x7f85341f6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52790/0 (socket says 172.21.15.89:52790) 2024-04-02T07:36:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.478+0000 7f8539000700 1 -- 172.21.15.89:0/678213919 learned_addr learned my addr 172.21.15.89:0/678213919 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.478+0000 7f8533e00700 1 --2- 172.21.15.89:0/678213919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85340ceec0 0x7f8534151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.478+0000 7f8533400700 1 --2- 172.21.15.89:0/678213919 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f853415d640 0x7f8534152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.479+0000 7f8539000700 1 -- 172.21.15.89:0/678213919 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f853415d640 msgr2=0x7f8534152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.479+0000 7f8539000700 1 --2- 172.21.15.89:0/678213919 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f853415d640 0x7f8534152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.479+0000 7f8539000700 1 -- 172.21.15.89:0/678213919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85340ceec0 msgr2=0x7f8534151860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.479+0000 7f8539000700 1 --2- 172.21.15.89:0/678213919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85340ceec0 0x7f8534151860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.479+0000 7f8539000700 1 -- 172.21.15.89:0/678213919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c009b30 con 0x7f85341619d0 2024-04-02T07:36:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.479+0000 7f8539000700 1 --2- 172.21.15.89:0/678213919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85341619d0 0x7f85341f6060 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f853408c490 tx=0x7f852c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.479+0000 7f852be00700 1 -- 172.21.15.89:0/678213919 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c004840 con 0x7f85341619d0 2024-04-02T07:36:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.479+0000 7f852be00700 1 -- 172.21.15.89:0/678213919 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c0092e0 con 0x7f85341619d0 2024-04-02T07:36:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.479+0000 7f852be00700 1 -- 172.21.15.89:0/678213919 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c01c660 con 0x7f85341619d0 2024-04-02T07:36:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.479+0000 7f853b000700 1 -- 172.21.15.89:0/678213919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853415fa70 con 0x7f85341619d0 2024-04-02T07:36:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.479+0000 7f853b000700 1 -- 172.21.15.89:0/678213919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8534160030 con 0x7f85341619d0 2024-04-02T07:36:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.480+0000 7f853b000700 1 -- 172.21.15.89:0/678213919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8534067720 con 0x7f85341619d0 2024-04-02T07:36:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.481+0000 7f852be00700 1 -- 172.21.15.89:0/678213919 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f852c01c7c0 con 0x7f85341619d0 2024-04-02T07:36:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.482+0000 7f852be00700 1 --2- 172.21.15.89:0/678213919 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f851406ac90 0x7f851406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.482+0000 7f852be00700 1 -- 172.21.15.89:0/678213919 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f852c08af30 con 0x7f85341619d0 2024-04-02T07:36:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.482+0000 7f8533e00700 1 --2- 172.21.15.89:0/678213919 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f851406ac90 0x7f851406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.483+0000 7f8533e00700 1 --2- 172.21.15.89:0/678213919 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f851406ac90 0x7f851406d150 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8524006b20 tx=0x7f8524008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:04.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.486+0000 7f852be00700 1 -- 172.21.15.89:0/678213919 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f852c056100 con 0x7f85341619d0 2024-04-02T07:36:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.691+0000 7f853b000700 1 -- 172.21.15.89:0/678213919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85340ceb00 con 0x7f85341619d0 2024-04-02T07:36:04.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.692+0000 7f852be00700 1 -- 172.21.15.89:0/678213919 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f852c059720 con 0x7f85341619d0 2024-04-02T07:36:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:04.694 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:54.238913+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:36:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.695+0000 7f853b000700 1 -- 172.21.15.89:0/678213919 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f851406ac90 msgr2=0x7f851406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.695+0000 7f853b000700 1 --2- 172.21.15.89:0/678213919 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f851406ac90 0x7f851406d150 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8524006b20 tx=0x7f8524008040 comp rx=0 tx=0).stop 2024-04-02T07:36:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.695+0000 7f853b000700 1 -- 172.21.15.89:0/678213919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85341619d0 msgr2=0x7f85341f6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.695+0000 7f853b000700 1 --2- 172.21.15.89:0/678213919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85341619d0 0x7f85341f6060 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f853408c490 tx=0x7f852c004ad0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.695+0000 7f853b000700 1 -- 172.21.15.89:0/678213919 shutdown_connections 2024-04-02T07:36:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.695+0000 7f853b000700 1 --2- 172.21.15.89:0/678213919 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f851406ac90 0x7f851406d150 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.695+0000 7f853b000700 1 --2- 172.21.15.89:0/678213919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85340ceec0 0x7f8534151860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.695+0000 7f853b000700 1 --2- 172.21.15.89:0/678213919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85341619d0 0x7f85341f6060 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.695+0000 7f853b000700 1 --2- 172.21.15.89:0/678213919 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f853415d640 0x7f8534152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.695+0000 7f853b000700 1 -- 172.21.15.89:0/678213919 >> 172.21.15.89:0/678213919 conn(0x7f853408aff0 msgr2=0x7f853415b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.695+0000 7f853b000700 1 -- 172.21.15.89:0/678213919 shutdown_connections 2024-04-02T07:36:04.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.695+0000 7f853b000700 1 -- 172.21.15.89:0/678213919 wait complete. 2024-04-02T07:36:04.715 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:35:51.296561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:35:52.232882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:36:04.715 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:36:04.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.907+0000 7f43aa600700 1 Processor -- start 2024-04-02T07:36:04.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.907+0000 7f43aa600700 1 -- start start 2024-04-02T07:36:04.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.909+0000 7f43aa600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f43a415d9d0 0x7f43a40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.909+0000 7f43aa600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f43a40d2540 0x7f43a40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.909+0000 7f43aa600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a4161420 con 0x7f43a40d16a0 2024-04-02T07:36:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.909+0000 7f43aa600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43a415edc0 con 0x7f43a415d9d0 2024-04-02T07:36:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.909+0000 7f43aa600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43a4161700 con 0x7f43a40d2540 2024-04-02T07:36:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.909+0000 7f43a3400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f43a415d9d0 0x7f43a40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.909+0000 7f43a3400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f43a415d9d0 0x7f43a40d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52796/0 (socket says 172.21.15.89:52796) 2024-04-02T07:36:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.909+0000 7f43a3400700 1 -- 172.21.15.89:0/1254440783 learned_addr learned my addr 172.21.15.89:0/1254440783 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:04.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.909+0000 7f43a3400700 1 -- 172.21.15.89:0/1254440783 >> v2:172.21.15.113:3300/0 conn(0x7f43a40d2540 msgr2=0x7f43a40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:04.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.909+0000 7f43a3400700 1 --2- 172.21.15.89:0/1254440783 >> v2:172.21.15.113:3300/0 conn(0x7f43a40d2540 0x7f43a40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.909+0000 7f43a3400700 1 -- 172.21.15.89:0/1254440783 >> v1:172.21.15.113:6789/0 conn(0x7f43a40d16a0 legacy=0x7f43a40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:04.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.909+0000 7f43a3400700 1 -- 172.21.15.89:0/1254440783 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a41619e0 con 0x7f43a415d9d0 2024-04-02T07:36:04.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.909+0000 7f43a3400700 1 --2- 172.21.15.89:0/1254440783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a415d9d0 0x7f43a40d3c30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f43a408ca10 tx=0x7f4394009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d32ce48c03f11d92 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.910+0000 7f43a2000700 1 -- 172.21.15.89:0/1254440783 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4394004230 con 0x7f43a415d9d0 2024-04-02T07:36:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.910+0000 7f43a2000700 1 -- 172.21.15.89:0/1254440783 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4394004390 con 0x7f43a415d9d0 2024-04-02T07:36:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.910+0000 7f43a2000700 1 -- 172.21.15.89:0/1254440783 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43940036a0 con 0x7f43a415d9d0 2024-04-02T07:36:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.910+0000 7f43aa600700 1 -- 172.21.15.89:0/1254440783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a415d9d0 msgr2=0x7f43a40d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.910+0000 7f43aa600700 1 --2- 172.21.15.89:0/1254440783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a415d9d0 0x7f43a40d3c30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f43a408ca10 tx=0x7f4394009bd0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.910+0000 7f43aa600700 1 -- 172.21.15.89:0/1254440783 shutdown_connections 2024-04-02T07:36:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.910+0000 7f43aa600700 1 --2- 172.21.15.89:0/1254440783 >> v2:172.21.15.113:3300/0 conn(0x7f43a40d2540 0x7f43a40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.910+0000 7f43aa600700 1 --2- 172.21.15.89:0/1254440783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a415d9d0 0x7f43a40d3c30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.910+0000 7f43aa600700 1 -- 172.21.15.89:0/1254440783 >> 172.21.15.89:0/1254440783 conn(0x7f43a408aff0 msgr2=0x7f43a40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.910+0000 7f43aa600700 1 -- 172.21.15.89:0/1254440783 shutdown_connections 2024-04-02T07:36:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.910+0000 7f43aa600700 1 -- 172.21.15.89:0/1254440783 wait complete. 2024-04-02T07:36:04.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.912+0000 7f43aa600700 1 Processor -- start 2024-04-02T07:36:04.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.913+0000 7f43aa600700 1 -- start start 2024-04-02T07:36:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43aa600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a40d16a0 0x7f43a41581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43aa600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43a40d2540 0x7f43a40b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43aa600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43a415d9d0 0x7f43a4155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43aa600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a4159a80 con 0x7f43a40d16a0 2024-04-02T07:36:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43aa600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40b82b0 con 0x7f43a40d2540 2024-04-02T07:36:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43aa600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40b8590 con 0x7f43a415d9d0 2024-04-02T07:36:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43a3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a40d16a0 0x7f43a41581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43a3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a40d16a0 0x7f43a41581f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52810/0 (socket says 172.21.15.89:52810) 2024-04-02T07:36:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43a3400700 1 -- 172.21.15.89:0/2652739632 learned_addr learned my addr 172.21.15.89:0/2652739632 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43a2a00700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43a40d2540 0x7f43a40b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43a3400700 1 -- 172.21.15.89:0/2652739632 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43a415d9d0 msgr2=0x7f43a4155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43a3e00700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43a415d9d0 0x7f43a4155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43a3400700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43a415d9d0 0x7f43a4155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43a3400700 1 -- 172.21.15.89:0/2652739632 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43a40d2540 msgr2=0x7f43a40b7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43a3400700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43a40d2540 0x7f43a40b7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.914+0000 7f43a3400700 1 -- 172.21.15.89:0/2652739632 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4394009550 con 0x7f43a40d16a0 2024-04-02T07:36:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.915+0000 7f43a2a00700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43a40d2540 0x7f43a40b7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.915+0000 7f43a3e00700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43a415d9d0 0x7f43a4155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.915+0000 7f43a3400700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a40d16a0 0x7f43a41581f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f43a408c230 tx=0x7f4394004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.915+0000 7f439b400700 1 -- 172.21.15.89:0/2652739632 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4394011630 con 0x7f43a40d16a0 2024-04-02T07:36:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.915+0000 7f439b400700 1 -- 172.21.15.89:0/2652739632 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4394011790 con 0x7f43a40d16a0 2024-04-02T07:36:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.915+0000 7f439b400700 1 -- 172.21.15.89:0/2652739632 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4394011b40 con 0x7f43a40d16a0 2024-04-02T07:36:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.915+0000 7f43aa600700 1 -- 172.21.15.89:0/2652739632 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a40b8870 con 0x7f43a40d16a0 2024-04-02T07:36:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.915+0000 7f43aa600700 1 -- 172.21.15.89:0/2652739632 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a40b7fc0 con 0x7f43a40d16a0 2024-04-02T07:36:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.916+0000 7f43aa600700 1 -- 172.21.15.89:0/2652739632 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a40d1ca0 con 0x7f43a40d16a0 2024-04-02T07:36:04.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.917+0000 7f439b400700 1 -- 172.21.15.89:0/2652739632 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4394011ca0 con 0x7f43a40d16a0 2024-04-02T07:36:04.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.918+0000 7f439b400700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f438406ac90 0x7f438406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:04.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.918+0000 7f439b400700 1 -- 172.21.15.89:0/2652739632 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f439408aa30 con 0x7f43a40d16a0 2024-04-02T07:36:04.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.918+0000 7f43a2a00700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f438406ac90 0x7f438406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:04.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.918+0000 7f43a2a00700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f438406ac90 0x7f438406d150 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f43a415f920 tx=0x7f4390009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:04.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:04.922+0000 7f439b400700 1 -- 172.21.15.89:0/2652739632 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4394055cb0 con 0x7f43a40d16a0 2024-04-02T07:36:05.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.123+0000 7f43aa600700 1 -- 172.21.15.89:0/2652739632 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f43a4067720 con 0x7f43a40d16a0 2024-04-02T07:36:05.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.123+0000 7f439b400700 1 -- 172.21.15.89:0/2652739632 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f439401f020 con 0x7f43a40d16a0 2024-04-02T07:36:05.125 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:36:05.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.126+0000 7f43aa600700 1 -- 172.21.15.89:0/2652739632 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f438406ac90 msgr2=0x7f438406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:05.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.126+0000 7f43aa600700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f438406ac90 0x7f438406d150 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f43a415f920 tx=0x7f4390009040 comp rx=0 tx=0).stop 2024-04-02T07:36:05.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.126+0000 7f43aa600700 1 -- 172.21.15.89:0/2652739632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a40d16a0 msgr2=0x7f43a41581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:05.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.126+0000 7f43aa600700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a40d16a0 0x7f43a41581f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f43a408c230 tx=0x7f4394004750 comp rx=0 tx=0).stop 2024-04-02T07:36:05.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.126+0000 7f43aa600700 1 -- 172.21.15.89:0/2652739632 shutdown_connections 2024-04-02T07:36:05.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.126+0000 7f43aa600700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f438406ac90 0x7f438406d150 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.126+0000 7f43aa600700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43a415d9d0 0x7f43a4155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.126+0000 7f43aa600700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a40d16a0 0x7f43a41581f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.126+0000 7f43aa600700 1 --2- 172.21.15.89:0/2652739632 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43a40d2540 0x7f43a40b7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.126+0000 7f43aa600700 1 -- 172.21.15.89:0/2652739632 >> 172.21.15.89:0/2652739632 conn(0x7f43a408aff0 msgr2=0x7f43a40b04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:05.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.126+0000 7f43aa600700 1 -- 172.21.15.89:0/2652739632 shutdown_connections 2024-04-02T07:36:05.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.126+0000 7f43aa600700 1 -- 172.21.15.89:0/2652739632 wait complete. 2024-04-02T07:36:05.146 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:36:05.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.339+0000 7f8373400700 1 Processor -- start 2024-04-02T07:36:05.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.339+0000 7f8373400700 1 -- start start 2024-04-02T07:36:05.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.340+0000 7f8373400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f836c15d9d0 0x7f836c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:05.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.340+0000 7f8373400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f836c0d16a0 0x7f836c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:05.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.340+0000 7f8373400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f836c0d2370 0x7f836c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:05.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.340+0000 7f8373400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f836c0d2820 con 0x7f836c0d16a0 2024-04-02T07:36:05.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.340+0000 7f8373400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f836c161420 con 0x7f836c0d2370 2024-04-02T07:36:05.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.340+0000 7f8373400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f836c161700 con 0x7f836c15d9d0 2024-04-02T07:36:05.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.340+0000 7f836be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f836c0d16a0 0x7f836c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:05.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.340+0000 7f836be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f836c0d16a0 0x7f836c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52824/0 (socket says 172.21.15.89:52824) 2024-04-02T07:36:05.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.340+0000 7f836be00700 1 -- 172.21.15.89:0/2576463507 learned_addr learned my addr 172.21.15.89:0/2576463507 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:05.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.341+0000 7f8370a00700 1 --2- 172.21.15.89:0/2576463507 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f836c15d9d0 0x7f836c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:05.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.341+0000 7f8371400700 1 --2- 172.21.15.89:0/2576463507 >> v2:172.21.15.113:3300/0 conn(0x7f836c0d2370 0x7f836c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:05.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.341+0000 7f836be00700 1 -- 172.21.15.89:0/2576463507 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f836c15d9d0 msgr2=0x7f836c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:05.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.341+0000 7f836be00700 1 --2- 172.21.15.89:0/2576463507 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f836c15d9d0 0x7f836c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.341+0000 7f836be00700 1 -- 172.21.15.89:0/2576463507 >> v2:172.21.15.113:3300/0 conn(0x7f836c0d2370 msgr2=0x7f836c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:05.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.341+0000 7f836be00700 1 --2- 172.21.15.89:0/2576463507 >> v2:172.21.15.113:3300/0 conn(0x7f836c0d2370 0x7f836c15ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.341+0000 7f836be00700 1 -- 172.21.15.89:0/2576463507 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836c1619e0 con 0x7f836c0d16a0 2024-04-02T07:36:05.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.341+0000 7f836be00700 1 --2- 172.21.15.89:0/2576463507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f836c0d16a0 0x7f836c0d1e30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f836c0cdc00 tx=0x7f8358009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2662cca42747be14 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:05.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.341+0000 7f836b400700 1 -- 172.21.15.89:0/2576463507 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8358005e30 con 0x7f836c0d16a0 2024-04-02T07:36:05.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.341+0000 7f836b400700 1 -- 172.21.15.89:0/2576463507 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8358004030 con 0x7f836c0d16a0 2024-04-02T07:36:05.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.341+0000 7f836b400700 1 -- 172.21.15.89:0/2576463507 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83580043e0 con 0x7f836c0d16a0 2024-04-02T07:36:05.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.342+0000 7f8373400700 1 -- 172.21.15.89:0/2576463507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f836c0d16a0 msgr2=0x7f836c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:05.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.342+0000 7f8373400700 1 --2- 172.21.15.89:0/2576463507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f836c0d16a0 0x7f836c0d1e30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f836c0cdc00 tx=0x7f8358009bd0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.342+0000 7f8373400700 1 -- 172.21.15.89:0/2576463507 shutdown_connections 2024-04-02T07:36:05.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.342+0000 7f8373400700 1 --2- 172.21.15.89:0/2576463507 >> v2:172.21.15.113:3300/0 conn(0x7f836c0d2370 0x7f836c15ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.342+0000 7f8373400700 1 --2- 172.21.15.89:0/2576463507 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f836c15d9d0 0x7f836c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.342+0000 7f8373400700 1 --2- 172.21.15.89:0/2576463507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f836c0d16a0 0x7f836c0d1e30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.342+0000 7f8373400700 1 -- 172.21.15.89:0/2576463507 >> 172.21.15.89:0/2576463507 conn(0x7f836c08aff0 msgr2=0x7f836c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:05.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.342+0000 7f8373400700 1 -- 172.21.15.89:0/2576463507 shutdown_connections 2024-04-02T07:36:05.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.342+0000 7f8373400700 1 -- 172.21.15.89:0/2576463507 wait complete. 2024-04-02T07:36:05.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.344+0000 7f8373400700 1 Processor -- start 2024-04-02T07:36:05.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.344+0000 7f8373400700 1 -- start start 2024-04-02T07:36:05.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8373400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f836c0d16a0 0x7f836c0cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:05.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8373400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f836c0d2370 0x7f836c0b86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:05.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8373400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f836c15d9d0 0x7f836c152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:05.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8373400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836c0b8e90 con 0x7f836c15d9d0 2024-04-02T07:36:05.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8373400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836c0b8c10 con 0x7f836c0d2370 2024-04-02T07:36:05.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8373400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f836c0b6b60 con 0x7f836c0d16a0 2024-04-02T07:36:05.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8371400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f836c15d9d0 0x7f836c152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:05.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8371400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f836c15d9d0 0x7f836c152810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52840/0 (socket says 172.21.15.89:52840) 2024-04-02T07:36:05.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8371400700 1 -- 172.21.15.89:0/3695938954 learned_addr learned my addr 172.21.15.89:0/3695938954 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8370a00700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f836c0d16a0 0x7f836c0cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f836be00700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f836c0d2370 0x7f836c0b86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8371400700 1 -- 172.21.15.89:0/3695938954 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f836c0d16a0 msgr2=0x7f836c0cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8371400700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f836c0d16a0 0x7f836c0cb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8371400700 1 -- 172.21.15.89:0/3695938954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f836c0d2370 msgr2=0x7f836c0b86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8371400700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f836c0d2370 0x7f836c0b86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8371400700 1 -- 172.21.15.89:0/3695938954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8358009500 con 0x7f836c15d9d0 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f8370a00700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f836c0d16a0 0x7f836c0cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.345+0000 7f836be00700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f836c0d2370 0x7f836c0b86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.346+0000 7f8371400700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f836c15d9d0 0x7f836c152810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8364007e60 tx=0x7f836400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.346+0000 7f8368c00700 1 -- 172.21.15.89:0/3695938954 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836400ede0 con 0x7f836c15d9d0 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.346+0000 7f8368c00700 1 -- 172.21.15.89:0/3695938954 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8364014420 con 0x7f836c15d9d0 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.346+0000 7f8373400700 1 -- 172.21.15.89:0/3695938954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f836c0b6e40 con 0x7f836c15d9d0 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.346+0000 7f8368c00700 1 -- 172.21.15.89:0/3695938954 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83640147d0 con 0x7f836c15d9d0 2024-04-02T07:36:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.346+0000 7f8373400700 1 -- 172.21.15.89:0/3695938954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f836c0b7400 con 0x7f836c15d9d0 2024-04-02T07:36:05.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.347+0000 7f8373400700 1 -- 172.21.15.89:0/3695938954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f836c0d1ca0 con 0x7f836c15d9d0 2024-04-02T07:36:05.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.348+0000 7f8368c00700 1 -- 172.21.15.89:0/3695938954 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8364014930 con 0x7f836c15d9d0 2024-04-02T07:36:05.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.349+0000 7f8368c00700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f834c06ac90 0x7f834c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:05.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.349+0000 7f8368c00700 1 -- 172.21.15.89:0/3695938954 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83640899e0 con 0x7f836c15d9d0 2024-04-02T07:36:05.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.349+0000 7f8370a00700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f834c06ac90 0x7f834c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:05.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.350+0000 7f8370a00700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f834c06ac90 0x7f834c06d150 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8360007a50 tx=0x7f8360008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:05.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.353+0000 7f8368c00700 1 -- 172.21.15.89:0/3695938954 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8364054c50 con 0x7f836c15d9d0 2024-04-02T07:36:05.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.555+0000 7f8373400700 1 -- 172.21.15.89:0/3695938954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f836c0b75f0 con 0x7f836c15d9d0 2024-04-02T07:36:05.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.555+0000 7f8368c00700 1 -- 172.21.15.89:0/3695938954 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8364058270 con 0x7f836c15d9d0 2024-04-02T07:36:05.557 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:36:05.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.558+0000 7f8373400700 1 -- 172.21.15.89:0/3695938954 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f834c06ac90 msgr2=0x7f834c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:05.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.558+0000 7f8373400700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f834c06ac90 0x7f834c06d150 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8360007a50 tx=0x7f8360008040 comp rx=0 tx=0).stop 2024-04-02T07:36:05.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.558+0000 7f8373400700 1 -- 172.21.15.89:0/3695938954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f836c15d9d0 msgr2=0x7f836c152810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:05.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.558+0000 7f8373400700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f836c15d9d0 0x7f836c152810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8364007e60 tx=0x7f836400e3c0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.558+0000 7f8373400700 1 -- 172.21.15.89:0/3695938954 shutdown_connections 2024-04-02T07:36:05.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.558+0000 7f8373400700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f834c06ac90 0x7f834c06d150 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.558+0000 7f8373400700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f836c15d9d0 0x7f836c152810 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.558+0000 7f8373400700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f836c0d16a0 0x7f836c0cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.558+0000 7f8373400700 1 --2- 172.21.15.89:0/3695938954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f836c0d2370 0x7f836c0b86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.558+0000 7f8373400700 1 -- 172.21.15.89:0/3695938954 >> 172.21.15.89:0/3695938954 conn(0x7f836c08aff0 msgr2=0x7f836c06d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:05.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.558+0000 7f8373400700 1 -- 172.21.15.89:0/3695938954 shutdown_connections 2024-04-02T07:36:05.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.558+0000 7f8373400700 1 -- 172.21.15.89:0/3695938954 wait complete. 2024-04-02T07:36:05.577 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:36:05.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.770+0000 7fcb53e00700 1 Processor -- start 2024-04-02T07:36:05.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.770+0000 7fcb53e00700 1 -- start start 2024-04-02T07:36:05.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.771+0000 7fcb53e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb4c0d2540 0x7fcb4c1687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:05.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.771+0000 7fcb53e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4c161420 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.771+0000 7fcb53e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4c161700 con 0x7fcb4c15d9d0 2024-04-02T07:36:05.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.771+0000 7fcb53e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4c0d4340 con 0x7fcb4c0d2540 2024-04-02T07:36:05.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.772+0000 7fcb51e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb4c0d2540 0x7fcb4c1687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:05.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.772+0000 7fcb50a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcb4c0d16a0 0x7fcb4c0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34962/0 (socket says 172.21.15.89:34962) 2024-04-02T07:36:05.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.772+0000 7fcb50a00700 1 -- 172.21.15.89:0/1583695146 learned_addr learned my addr 172.21.15.89:0/1583695146 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:05.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.772+0000 7fcb51e00700 1 -- 172.21.15.89:0/1583695146 >> v1:172.21.15.113:6789/0 conn(0x7fcb4c15d9d0 legacy=0x7fcb4c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:05.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.772+0000 7fcb51e00700 1 -- 172.21.15.89:0/1583695146 >> v1:172.21.15.89:6789/0 conn(0x7fcb4c0d16a0 legacy=0x7fcb4c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:05.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.772+0000 7fcb51e00700 1 -- 172.21.15.89:0/1583695146 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c1619e0 con 0x7fcb4c0d2540 2024-04-02T07:36:05.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.772+0000 7fcb4be00700 1 -- 172.21.15.89:0/1583695146 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890920763 0 0) 0x7fcb4c161420 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.772+0000 7fcb51e00700 1 --2- 172.21.15.89:0/1583695146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0d2540 0x7fcb4c1687b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009d90 tx=0x7fcb4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=311544a1924530d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:05.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.773+0000 7fcb4be00700 1 -- 172.21.15.89:0/1583695146 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4400e070 con 0x7fcb4c0d2540 2024-04-02T07:36:05.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.773+0000 7fcb4be00700 1 -- 172.21.15.89:0/1583695146 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4400bbd0 con 0x7fcb4c0d2540 2024-04-02T07:36:05.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.773+0000 7fcb4be00700 1 -- 172.21.15.89:0/1583695146 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb44012430 con 0x7fcb4c0d2540 2024-04-02T07:36:05.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.773+0000 7fcb53e00700 1 -- 172.21.15.89:0/1583695146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0d2540 msgr2=0x7fcb4c1687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:05.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.773+0000 7fcb53e00700 1 --2- 172.21.15.89:0/1583695146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0d2540 0x7fcb4c1687b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009d90 tx=0x7fcb4400b330 comp rx=0 tx=0).stop 2024-04-02T07:36:05.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.773+0000 7fcb53e00700 1 -- 172.21.15.89:0/1583695146 shutdown_connections 2024-04-02T07:36:05.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.773+0000 7fcb53e00700 1 --2- 172.21.15.89:0/1583695146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0d2540 0x7fcb4c1687b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.773+0000 7fcb53e00700 1 -- 172.21.15.89:0/1583695146 >> 172.21.15.89:0/1583695146 conn(0x7fcb4c08aff0 msgr2=0x7fcb4c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:05.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.773+0000 7fcb53e00700 1 -- 172.21.15.89:0/1583695146 shutdown_connections 2024-04-02T07:36:05.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.773+0000 7fcb53e00700 1 -- 172.21.15.89:0/1583695146 wait complete. 2024-04-02T07:36:05.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.775+0000 7fcb53e00700 1 Processor -- start 2024-04-02T07:36:05.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.775+0000 7fcb53e00700 1 -- start start 2024-04-02T07:36:05.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.776+0000 7fcb53e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0d16a0 0x7fcb4c15c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:05.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.776+0000 7fcb53e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0d2540 0x7fcb4c0cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:05.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.776+0000 7fcb53e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb4c15d9d0 0x7fcb4c0cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:05.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.776+0000 7fcb53e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c0b6d30 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.776+0000 7fcb53e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c0b7010 con 0x7fcb4c15d9d0 2024-04-02T07:36:05.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.776+0000 7fcb53e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c0b72f0 con 0x7fcb4c0d2540 2024-04-02T07:36:05.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.776+0000 7fcb51400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0d16a0 0x7fcb4c15c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:05.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.776+0000 7fcb51400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0d16a0 0x7fcb4c15c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52862/0 (socket says 172.21.15.89:52862) 2024-04-02T07:36:05.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.776+0000 7fcb51400700 1 -- 172.21.15.89:0/3164458207 learned_addr learned my addr 172.21.15.89:0/3164458207 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:05.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.776+0000 7fcb51e00700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb4c15d9d0 0x7fcb4c0cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:05.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.776+0000 7fcb50a00700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0d2540 0x7fcb4c0cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:05.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.776+0000 7fcb51400700 1 -- 172.21.15.89:0/3164458207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0d2540 msgr2=0x7fcb4c0cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:05.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.776+0000 7fcb51400700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0d2540 0x7fcb4c0cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.777+0000 7fcb51400700 1 -- 172.21.15.89:0/3164458207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb4c15d9d0 msgr2=0x7fcb4c0cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:05.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.777+0000 7fcb51400700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb4c15d9d0 0x7fcb4c0cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.777+0000 7fcb51400700 1 -- 172.21.15.89:0/3164458207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb44009a70 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.777+0000 7fcb51e00700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb4c15d9d0 0x7fcb4c0cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:05.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.777+0000 7fcb51400700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0d16a0 0x7fcb4c15c700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000c450 tx=0x7fcb4000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:05.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.777+0000 7fcb49600700 1 -- 172.21.15.89:0/3164458207 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb400052d0 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.777+0000 7fcb49600700 1 -- 172.21.15.89:0/3164458207 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb40005430 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.777+0000 7fcb53e00700 1 -- 172.21.15.89:0/3164458207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4c0b75d0 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.777+0000 7fcb49600700 1 -- 172.21.15.89:0/3164458207 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb400038c0 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.777+0000 7fcb53e00700 1 -- 172.21.15.89:0/3164458207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4c0b6020 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.778+0000 7fcb53e00700 1 -- 172.21.15.89:0/3164458207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4c0d1ca0 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.779+0000 7fcb49600700 1 -- 172.21.15.89:0/3164458207 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcb40003a20 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.780+0000 7fcb49600700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb2c06ac90 0x7fcb2c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:05.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.780+0000 7fcb49600700 1 -- 172.21.15.89:0/3164458207 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb400886d0 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.780+0000 7fcb50a00700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb2c06ac90 0x7fcb2c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:05.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.781+0000 7fcb50a00700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb2c06ac90 0x7fcb2c06d150 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c1611e0 tx=0x7fcb38009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:05.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.784+0000 7fcb49600700 1 -- 172.21.15.89:0/3164458207 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcb400538b0 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.985+0000 7fcb53e00700 1 -- 172.21.15.89:0/3164458207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb4c0b7af0 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.986+0000 7fcb49600700 1 -- 172.21.15.89:0/3164458207 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcb4c0b7af0 con 0x7fcb4c0d16a0 2024-04-02T07:36:05.988 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:36:05.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.988+0000 7fcb53e00700 1 -- 172.21.15.89:0/3164458207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb2c06ac90 msgr2=0x7fcb2c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:05.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.988+0000 7fcb53e00700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb2c06ac90 0x7fcb2c06d150 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c1611e0 tx=0x7fcb38009040 comp rx=0 tx=0).stop 2024-04-02T07:36:05.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.988+0000 7fcb53e00700 1 -- 172.21.15.89:0/3164458207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0d16a0 msgr2=0x7fcb4c15c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:05.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.988+0000 7fcb53e00700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0d16a0 0x7fcb4c15c700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000c450 tx=0x7fcb4000c810 comp rx=0 tx=0).stop 2024-04-02T07:36:05.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.988+0000 7fcb53e00700 1 -- 172.21.15.89:0/3164458207 shutdown_connections 2024-04-02T07:36:05.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.988+0000 7fcb53e00700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb2c06ac90 0x7fcb2c06d150 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.988+0000 7fcb53e00700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb4c15d9d0 0x7fcb4c0cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.988+0000 7fcb53e00700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0d16a0 0x7fcb4c15c700 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.988+0000 7fcb53e00700 1 --2- 172.21.15.89:0/3164458207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0d2540 0x7fcb4c0cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:05.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.988+0000 7fcb53e00700 1 -- 172.21.15.89:0/3164458207 >> 172.21.15.89:0/3164458207 conn(0x7fcb4c08aff0 msgr2=0x7fcb4c06cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:05.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.988+0000 7fcb53e00700 1 -- 172.21.15.89:0/3164458207 shutdown_connections 2024-04-02T07:36:05.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:05.988+0000 7fcb53e00700 1 -- 172.21.15.89:0/3164458207 wait complete. 2024-04-02T07:36:06.008 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:36:06.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.197+0000 7f6fdc800700 1 Processor -- start 2024-04-02T07:36:06.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.197+0000 7f6fdc800700 1 -- start start 2024-04-02T07:36:06.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.198+0000 7f6fdc800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6fd40d18b0 0x7f6fd40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:06.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.198+0000 7f6fdc800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fd4161590 0x7f6fd4161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:06.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.198+0000 7f6fdc800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd40ceda0 con 0x7f6fd4160f10 2024-04-02T07:36:06.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.198+0000 7f6fdc800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd40d1d30 con 0x7f6fd40d18b0 2024-04-02T07:36:06.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.198+0000 7f6fdc800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd4161f60 con 0x7f6fd4161590 2024-04-02T07:36:06.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.198+0000 7f6fd9400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6fd40d18b0 0x7f6fd40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:06.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.198+0000 7f6fd9400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6fd40d18b0 0x7f6fd40cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52878/0 (socket says 172.21.15.89:52878) 2024-04-02T07:36:06.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.198+0000 7f6fd9e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6fd4160f10 0x7f6fd40d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34978/0 (socket says 172.21.15.89:34978) 2024-04-02T07:36:06.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.198+0000 7f6fd9400700 1 -- 172.21.15.89:0/3494533795 learned_addr learned my addr 172.21.15.89:0/3494533795 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:06.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.198+0000 7f6fda800700 1 --2- 172.21.15.89:0/3494533795 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fd4161590 0x7f6fd4161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:06.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.199+0000 7f6fd9400700 1 -- 172.21.15.89:0/3494533795 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fd4161590 msgr2=0x7f6fd4161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:06.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.199+0000 7f6fd9400700 1 --2- 172.21.15.89:0/3494533795 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fd4161590 0x7f6fd4161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.199+0000 7f6fd9400700 1 -- 172.21.15.89:0/3494533795 >> v1:172.21.15.89:6789/0 conn(0x7f6fd4160f10 legacy=0x7f6fd40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:06.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.199+0000 7f6fd9400700 1 -- 172.21.15.89:0/3494533795 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd41612f0 con 0x7f6fd40d18b0 2024-04-02T07:36:06.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.199+0000 7f6fd9400700 1 --2- 172.21.15.89:0/3494533795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd40d18b0 0x7f6fd40cf230 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4009920 tx=0x7f6fc4009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151a6f36208d1ee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:06.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.199+0000 7f6fd8a00700 1 -- 172.21.15.89:0/3494533795 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4011830 con 0x7f6fd40d18b0 2024-04-02T07:36:06.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.199+0000 7f6fd8a00700 1 -- 172.21.15.89:0/3494533795 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc4011990 con 0x7f6fd40d18b0 2024-04-02T07:36:06.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.199+0000 7f6fd8a00700 1 -- 172.21.15.89:0/3494533795 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4011d20 con 0x7f6fd40d18b0 2024-04-02T07:36:06.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.200+0000 7f6fdc800700 1 -- 172.21.15.89:0/3494533795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd40d18b0 msgr2=0x7f6fd40cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:06.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.200+0000 7f6fdc800700 1 --2- 172.21.15.89:0/3494533795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd40d18b0 0x7f6fd40cf230 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4009920 tx=0x7f6fc4009c30 comp rx=0 tx=0).stop 2024-04-02T07:36:06.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.200+0000 7f6fdc800700 1 -- 172.21.15.89:0/3494533795 shutdown_connections 2024-04-02T07:36:06.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.200+0000 7f6fdc800700 1 --2- 172.21.15.89:0/3494533795 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fd4161590 0x7f6fd4161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.200+0000 7f6fdc800700 1 --2- 172.21.15.89:0/3494533795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd40d18b0 0x7f6fd40cf230 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.200+0000 7f6fdc800700 1 -- 172.21.15.89:0/3494533795 >> 172.21.15.89:0/3494533795 conn(0x7f6fd408aff0 msgr2=0x7f6fd40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:06.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.200+0000 7f6fdc800700 1 -- 172.21.15.89:0/3494533795 shutdown_connections 2024-04-02T07:36:06.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.200+0000 7f6fdc800700 1 -- 172.21.15.89:0/3494533795 wait complete. 2024-04-02T07:36:06.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.202+0000 7f6fdc800700 1 Processor -- start 2024-04-02T07:36:06.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.202+0000 7f6fdc800700 1 -- start start 2024-04-02T07:36:06.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fdc800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fd40d18b0 0x7f6fd40b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:06.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fdc800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fd4160f10 0x7f6fd40b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:06.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fdc800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd4161590 0x7f6fd415e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:06.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fdc800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40b7650 con 0x7f6fd4161590 2024-04-02T07:36:06.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fdc800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40b7930 con 0x7f6fd4160f10 2024-04-02T07:36:06.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fdc800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40b7c10 con 0x7f6fd40d18b0 2024-04-02T07:36:06.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fda800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd4161590 0x7f6fd415e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:06.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fda800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd4161590 0x7f6fd415e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52894/0 (socket says 172.21.15.89:52894) 2024-04-02T07:36:06.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fda800700 1 -- 172.21.15.89:0/1676644968 learned_addr learned my addr 172.21.15.89:0/1676644968 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fd9e00700 1 --2- 172.21.15.89:0/1676644968 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fd40d18b0 0x7f6fd40b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fd9400700 1 --2- 172.21.15.89:0/1676644968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fd4160f10 0x7f6fd40b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fda800700 1 -- 172.21.15.89:0/1676644968 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fd40d18b0 msgr2=0x7f6fd40b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fda800700 1 --2- 172.21.15.89:0/1676644968 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fd40d18b0 0x7f6fd40b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fda800700 1 -- 172.21.15.89:0/1676644968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fd4160f10 msgr2=0x7f6fd40b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fda800700 1 --2- 172.21.15.89:0/1676644968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fd4160f10 0x7f6fd40b2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.203+0000 7f6fda800700 1 -- 172.21.15.89:0/1676644968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc4009550 con 0x7f6fd4161590 2024-04-02T07:36:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.204+0000 7f6fda800700 1 --2- 172.21.15.89:0/1676644968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd4161590 0x7f6fd415e9d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6fd000efc0 tx=0x7f6fd000c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.204+0000 7f6fce000700 1 -- 172.21.15.89:0/1676644968 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd000cad0 con 0x7f6fd4161590 2024-04-02T07:36:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.204+0000 7f6fce000700 1 -- 172.21.15.89:0/1676644968 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd000f040 con 0x7f6fd4161590 2024-04-02T07:36:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.204+0000 7f6fce000700 1 -- 172.21.15.89:0/1676644968 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd000ce70 con 0x7f6fd4161590 2024-04-02T07:36:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.204+0000 7f6fdc800700 1 -- 172.21.15.89:0/1676644968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd40b7ef0 con 0x7f6fd4161590 2024-04-02T07:36:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.204+0000 7f6fdc800700 1 -- 172.21.15.89:0/1676644968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd40b8c30 con 0x7f6fd4161590 2024-04-02T07:36:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.205+0000 7f6fdc800700 1 -- 172.21.15.89:0/1676644968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd40b8410 con 0x7f6fd4161590 2024-04-02T07:36:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.206+0000 7f6fce000700 1 -- 172.21.15.89:0/1676644968 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6fd0014480 con 0x7f6fd4161590 2024-04-02T07:36:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.207+0000 7f6fce000700 1 --2- 172.21.15.89:0/1676644968 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6fb806ac90 0x7f6fb806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:06.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.207+0000 7f6fce000700 1 -- 172.21.15.89:0/1676644968 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fd0089a60 con 0x7f6fd4161590 2024-04-02T07:36:06.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.207+0000 7f6fd9e00700 1 --2- 172.21.15.89:0/1676644968 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6fb806ac90 0x7f6fb806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:06.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.208+0000 7f6fd9e00700 1 --2- 172.21.15.89:0/1676644968 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6fb806ac90 0x7f6fb806d150 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8000f60 tx=0x7f6fc8007ed0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:06.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.211+0000 7f6fce000700 1 -- 172.21.15.89:0/1676644968 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6fd0054c40 con 0x7f6fd4161590 2024-04-02T07:36:06.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.456+0000 7f6fdc800700 1 -- 172.21.15.89:0/1676644968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fd4067720 con 0x7f6fd4161590 2024-04-02T07:36:06.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.457+0000 7f6fce000700 1 -- 172.21.15.89:0/1676644968 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4731 (secure 0 0 0) 0x7f6fd0018090 con 0x7f6fd4161590 2024-04-02T07:36:06.459 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:06.459 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2427593953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2427593953},{"type":"v1","addr":"172.21.15.113:6835","nonce":2427593953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/926246816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":926246816},{"type":"v1","addr":"172.21.15.89:6836","nonce":926246816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2701685411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2701685411},{"type":"v1","addr":"172.21.15.89:6837","nonce":2701685411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:35:53.235047+0000","modified":"2024-04-02T07:35:56.245811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6834/3962490902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3962490902},{"type":"v1","addr":"172.21.15.113:6834","nonce":3962490902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-02T07:36:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.459+0000 7f6fdc800700 1 -- 172.21.15.89:0/1676644968 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6fb806ac90 msgr2=0x7f6fb806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.459+0000 7f6fdc800700 1 --2- 172.21.15.89:0/1676644968 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6fb806ac90 0x7f6fb806d150 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8000f60 tx=0x7f6fc8007ed0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.459+0000 7f6fdc800700 1 -- 172.21.15.89:0/1676644968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd4161590 msgr2=0x7f6fd415e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.459+0000 7f6fdc800700 1 --2- 172.21.15.89:0/1676644968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd4161590 0x7f6fd415e9d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6fd000efc0 tx=0x7f6fd000c0c0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.459+0000 7f6fdc800700 1 -- 172.21.15.89:0/1676644968 shutdown_connections 2024-04-02T07:36:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.459+0000 7f6fdc800700 1 --2- 172.21.15.89:0/1676644968 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6fb806ac90 0x7f6fb806d150 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.459+0000 7f6fdc800700 1 --2- 172.21.15.89:0/1676644968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd4161590 0x7f6fd415e9d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.459+0000 7f6fdc800700 1 --2- 172.21.15.89:0/1676644968 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fd40d18b0 0x7f6fd40b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.459+0000 7f6fdc800700 1 --2- 172.21.15.89:0/1676644968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fd4160f10 0x7f6fd40b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.459+0000 7f6fdc800700 1 -- 172.21.15.89:0/1676644968 >> 172.21.15.89:0/1676644968 conn(0x7f6fd408aff0 msgr2=0x7f6fd41e8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.459+0000 7f6fdc800700 1 -- 172.21.15.89:0/1676644968 shutdown_connections 2024-04-02T07:36:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.459+0000 7f6fdc800700 1 -- 172.21.15.89:0/1676644968 wait complete. 2024-04-02T07:36:06.462 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2024-04-02T07:36:06.478 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:36:06.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.666+0000 7fe63dc00700 1 Processor -- start 2024-04-02T07:36:06.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.667+0000 7fe63dc00700 1 -- start start 2024-04-02T07:36:06.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe63dc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe6380d18b0 0x7fe6380cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:06.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe63dc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe638161590 0x7fe638161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:06.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe63dc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6380ceda0 con 0x7fe638160f10 2024-04-02T07:36:06.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe63dc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6380d1d30 con 0x7fe638161590 2024-04-02T07:36:06.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe63dc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe638161f60 con 0x7fe6380d18b0 2024-04-02T07:36:06.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe637400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe638161590 0x7fe638161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:06.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe636a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe638160f10 0x7fe6380d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34988/0 (socket says 172.21.15.89:34988) 2024-04-02T07:36:06.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe636a00700 1 -- 172.21.15.89:0/263483810 learned_addr learned my addr 172.21.15.89:0/263483810 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:06.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe636000700 1 --2- 172.21.15.89:0/263483810 >> v2:172.21.15.113:3300/0 conn(0x7fe6380d18b0 0x7fe6380cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:06.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe637400700 1 -- 172.21.15.89:0/263483810 >> v2:172.21.15.113:3300/0 conn(0x7fe6380d18b0 msgr2=0x7fe6380cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:06.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe637400700 1 --2- 172.21.15.89:0/263483810 >> v2:172.21.15.113:3300/0 conn(0x7fe6380d18b0 0x7fe6380cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe637400700 1 -- 172.21.15.89:0/263483810 >> v1:172.21.15.89:6789/0 conn(0x7fe638160f10 legacy=0x7fe6380d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:06.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe637400700 1 -- 172.21.15.89:0/263483810 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6381612f0 con 0x7fe638161590 2024-04-02T07:36:06.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe636000700 1 --2- 172.21.15.89:0/263483810 >> v2:172.21.15.113:3300/0 conn(0x7fe6380d18b0 0x7fe6380cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:06.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.668+0000 7fe635600700 1 -- 172.21.15.89:0/263483810 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544318638 0 0) 0x7fe6380ceda0 con 0x7fe638160f10 2024-04-02T07:36:06.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.669+0000 7fe637400700 1 --2- 172.21.15.89:0/263483810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe638161590 0x7fe638161a20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe63808ca30 tx=0x7fe63000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e27573519005905 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:06.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.669+0000 7fe635600700 1 -- 172.21.15.89:0/263483810 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63000e070 con 0x7fe638161590 2024-04-02T07:36:06.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.669+0000 7fe635600700 1 -- 172.21.15.89:0/263483810 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63000bbd0 con 0x7fe638161590 2024-04-02T07:36:06.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.669+0000 7fe635600700 1 -- 172.21.15.89:0/263483810 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630012430 con 0x7fe638161590 2024-04-02T07:36:06.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.669+0000 7fe63dc00700 1 -- 172.21.15.89:0/263483810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe638161590 msgr2=0x7fe638161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:06.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.669+0000 7fe63dc00700 1 --2- 172.21.15.89:0/263483810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe638161590 0x7fe638161a20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe63808ca30 tx=0x7fe63000b330 comp rx=0 tx=0).stop 2024-04-02T07:36:06.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.670+0000 7fe63dc00700 1 -- 172.21.15.89:0/263483810 shutdown_connections 2024-04-02T07:36:06.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.670+0000 7fe63dc00700 1 --2- 172.21.15.89:0/263483810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe638161590 0x7fe638161a20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.670+0000 7fe63dc00700 1 --2- 172.21.15.89:0/263483810 >> v2:172.21.15.113:3300/0 conn(0x7fe6380d18b0 0x7fe6380cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.670+0000 7fe63dc00700 1 -- 172.21.15.89:0/263483810 >> 172.21.15.89:0/263483810 conn(0x7fe63808aff0 msgr2=0x7fe6380d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:06.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.670+0000 7fe63dc00700 1 -- 172.21.15.89:0/263483810 shutdown_connections 2024-04-02T07:36:06.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.670+0000 7fe63dc00700 1 -- 172.21.15.89:0/263483810 wait complete. 2024-04-02T07:36:06.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.671+0000 7fe63dc00700 1 Processor -- start 2024-04-02T07:36:06.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.671+0000 7fe63dc00700 1 -- start start 2024-04-02T07:36:06.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.672+0000 7fe63dc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6380d18b0 0x7fe6381553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:06.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.672+0000 7fe63dc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe638160f10 0x7fe6381e97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:06.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.672+0000 7fe63dc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe638161590 0x7fe638154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:06.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.672+0000 7fe63dc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe638155cf0 con 0x7fe638161590 2024-04-02T07:36:06.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.672+0000 7fe63dc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6380b72d0 con 0x7fe638160f10 2024-04-02T07:36:06.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.672+0000 7fe63dc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6380b75b0 con 0x7fe6380d18b0 2024-04-02T07:36:06.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.673+0000 7fe637400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe638161590 0x7fe638154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:06.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.673+0000 7fe637400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe638161590 0x7fe638154d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52912/0 (socket says 172.21.15.89:52912) 2024-04-02T07:36:06.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.673+0000 7fe637400700 1 -- 172.21.15.89:0/4040294964 learned_addr learned my addr 172.21.15.89:0/4040294964 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:06.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.673+0000 7fe636a00700 1 --2- 172.21.15.89:0/4040294964 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6380d18b0 0x7fe6381553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:06.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.673+0000 7fe636000700 1 --2- 172.21.15.89:0/4040294964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe638160f10 0x7fe6381e97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:06.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.673+0000 7fe637400700 1 -- 172.21.15.89:0/4040294964 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6380d18b0 msgr2=0x7fe6381553e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:06.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.673+0000 7fe637400700 1 --2- 172.21.15.89:0/4040294964 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6380d18b0 0x7fe6381553e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.673+0000 7fe637400700 1 -- 172.21.15.89:0/4040294964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe638160f10 msgr2=0x7fe6381e97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:06.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.673+0000 7fe637400700 1 --2- 172.21.15.89:0/4040294964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe638160f10 0x7fe6381e97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.673+0000 7fe637400700 1 -- 172.21.15.89:0/4040294964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe630009a70 con 0x7fe638161590 2024-04-02T07:36:06.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.673+0000 7fe637400700 1 --2- 172.21.15.89:0/4040294964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe638161590 0x7fe638154d50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe630014040 tx=0x7fe630004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:06.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.673+0000 7fe62ea00700 1 -- 172.21.15.89:0/4040294964 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630012430 con 0x7fe638161590 2024-04-02T07:36:06.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.674+0000 7fe62ea00700 1 -- 172.21.15.89:0/4040294964 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe630020070 con 0x7fe638161590 2024-04-02T07:36:06.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.674+0000 7fe62ea00700 1 -- 172.21.15.89:0/4040294964 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630004280 con 0x7fe638161590 2024-04-02T07:36:06.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.674+0000 7fe63dc00700 1 -- 172.21.15.89:0/4040294964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6381e7960 con 0x7fe638161590 2024-04-02T07:36:06.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.674+0000 7fe63dc00700 1 -- 172.21.15.89:0/4040294964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6381e8160 con 0x7fe638161590 2024-04-02T07:36:06.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.675+0000 7fe63dc00700 1 -- 172.21.15.89:0/4040294964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6381e77c0 con 0x7fe638161590 2024-04-02T07:36:06.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.676+0000 7fe62ea00700 1 -- 172.21.15.89:0/4040294964 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe6300043e0 con 0x7fe638161590 2024-04-02T07:36:06.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.676+0000 7fe62ea00700 1 --2- 172.21.15.89:0/4040294964 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe61806ac90 0x7fe61806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:06.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.676+0000 7fe62ea00700 1 -- 172.21.15.89:0/4040294964 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe630022070 con 0x7fe638161590 2024-04-02T07:36:06.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.677+0000 7fe636a00700 1 --2- 172.21.15.89:0/4040294964 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe61806ac90 0x7fe61806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:06.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.677+0000 7fe636a00700 1 --2- 172.21.15.89:0/4040294964 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe61806ac90 0x7fe61806d150 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe628000f60 tx=0x7fe628007e70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:06.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.680+0000 7fe62ea00700 1 -- 172.21.15.89:0/4040294964 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6381e77c0 con 0x7fe638161590 2024-04-02T07:36:06.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.885+0000 7fe63dc00700 1 -- 172.21.15.89:0/4040294964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6381e7e80 con 0x7fe638161590 2024-04-02T07:36:06.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.886+0000 7fe62ea00700 1 -- 172.21.15.89:0/4040294964 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7fe6381e7e80 con 0x7fe638161590 2024-04-02T07:36:06.888 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:06.888 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:54.238913+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:36:06.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.888+0000 7fe63dc00700 1 -- 172.21.15.89:0/4040294964 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe61806ac90 msgr2=0x7fe61806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:06.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.888+0000 7fe63dc00700 1 --2- 172.21.15.89:0/4040294964 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe61806ac90 0x7fe61806d150 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe628000f60 tx=0x7fe628007e70 comp rx=0 tx=0).stop 2024-04-02T07:36:06.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.888+0000 7fe63dc00700 1 -- 172.21.15.89:0/4040294964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe638161590 msgr2=0x7fe638154d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:06.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.888+0000 7fe63dc00700 1 --2- 172.21.15.89:0/4040294964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe638161590 0x7fe638154d50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe630014040 tx=0x7fe630004ad0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.888+0000 7fe63dc00700 1 -- 172.21.15.89:0/4040294964 shutdown_connections 2024-04-02T07:36:06.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.888+0000 7fe63dc00700 1 --2- 172.21.15.89:0/4040294964 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe61806ac90 0x7fe61806d150 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.888+0000 7fe63dc00700 1 --2- 172.21.15.89:0/4040294964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe638161590 0x7fe638154d50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.888+0000 7fe63dc00700 1 --2- 172.21.15.89:0/4040294964 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6380d18b0 0x7fe6381553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.888+0000 7fe63dc00700 1 --2- 172.21.15.89:0/4040294964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe638160f10 0x7fe6381e97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:06.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.888+0000 7fe63dc00700 1 -- 172.21.15.89:0/4040294964 >> 172.21.15.89:0/4040294964 conn(0x7fe63808aff0 msgr2=0x7fe6380cbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:06.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.888+0000 7fe63dc00700 1 -- 172.21.15.89:0/4040294964 shutdown_connections 2024-04-02T07:36:06.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:06.888+0000 7fe63dc00700 1 -- 172.21.15.89:0/4040294964 wait complete. 2024-04-02T07:36:06.908 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:35:51.296561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:35:52.232882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:36:06.909 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:36:07.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.100+0000 7f3195000700 1 Processor -- start 2024-04-02T07:36:07.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.101+0000 7f3195000700 1 -- start start 2024-04-02T07:36:07.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.102+0000 7f3195000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31900d16a0 0x7f31900d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.102+0000 7f3195000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3190161420 con 0x7f31900d2370 2024-04-02T07:36:07.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.102+0000 7f3195000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3190161700 con 0x7f319015d9d0 2024-04-02T07:36:07.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.102+0000 7f3195000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31900d2820 con 0x7f31900d16a0 2024-04-02T07:36:07.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.102+0000 7f318e800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f31900d2370 0x7f31900ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34992/0 (socket says 172.21.15.89:34992) 2024-04-02T07:36:07.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.102+0000 7f318e800700 1 -- 172.21.15.89:0/3734078856 learned_addr learned my addr 172.21.15.89:0/3734078856 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:07.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.102+0000 7f318d400700 1 --2- 172.21.15.89:0/3734078856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31900d16a0 0x7f31900d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.102+0000 7f318ca00700 1 -- 172.21.15.89:0/3734078856 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561575692 0 0) 0x7f3190161420 con 0x7f31900d2370 2024-04-02T07:36:07.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.102+0000 7f318ca00700 1 -- 172.21.15.89:0/3734078856 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31740035b0 con 0x7f31900d2370 2024-04-02T07:36:07.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.103+0000 7f318ca00700 1 -- 172.21.15.89:0/3734078856 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582200511 0 0) 0x7f3190161700 con 0x7f319015d9d0 2024-04-02T07:36:07.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.103+0000 7f318ca00700 1 -- 172.21.15.89:0/3734078856 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3190161420 con 0x7f319015d9d0 2024-04-02T07:36:07.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.103+0000 7f318d400700 1 -- 172.21.15.89:0/3734078856 >> v1:172.21.15.113:6789/0 conn(0x7f319015d9d0 legacy=0x7f31900d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.103+0000 7f318d400700 1 -- 172.21.15.89:0/3734078856 >> v1:172.21.15.89:6789/0 conn(0x7f31900d2370 legacy=0x7f31900ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.103+0000 7f318ca00700 1 -- 172.21.15.89:0/3734078856 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1072900006 0 0) 0x7f31740035b0 con 0x7f31900d2370 2024-04-02T07:36:07.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.103+0000 7f318d400700 1 -- 172.21.15.89:0/3734078856 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31901619e0 con 0x7f31900d16a0 2024-04-02T07:36:07.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.103+0000 7f318d400700 1 --2- 172.21.15.89:0/3734078856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31900d16a0 0x7f31900d1e30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f317c009850 tx=0x7f317c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bcde4870576dad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:07.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.103+0000 7f318ca00700 1 -- 172.21.15.89:0/3734078856 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31880019f0 con 0x7f31900d16a0 2024-04-02T07:36:07.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.103+0000 7f318ca00700 1 -- 172.21.15.89:0/3734078856 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317c005c30 con 0x7f31900d16a0 2024-04-02T07:36:07.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.103+0000 7f318ca00700 1 -- 172.21.15.89:0/3734078856 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317c004030 con 0x7f31900d16a0 2024-04-02T07:36:07.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.104+0000 7f3195000700 1 -- 172.21.15.89:0/3734078856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31900d16a0 msgr2=0x7f31900d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.104+0000 7f3195000700 1 --2- 172.21.15.89:0/3734078856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31900d16a0 0x7f31900d1e30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f317c009850 tx=0x7f317c009b60 comp rx=0 tx=0).stop 2024-04-02T07:36:07.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.104+0000 7f3195000700 1 -- 172.21.15.89:0/3734078856 shutdown_connections 2024-04-02T07:36:07.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.104+0000 7f3195000700 1 --2- 172.21.15.89:0/3734078856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31900d16a0 0x7f31900d1e30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.104+0000 7f3195000700 1 -- 172.21.15.89:0/3734078856 >> 172.21.15.89:0/3734078856 conn(0x7f319008aff0 msgr2=0x7f31900d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:07.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.104+0000 7f3195000700 1 -- 172.21.15.89:0/3734078856 shutdown_connections 2024-04-02T07:36:07.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.104+0000 7f3195000700 1 -- 172.21.15.89:0/3734078856 wait complete. 2024-04-02T07:36:07.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.106+0000 7f3195000700 1 Processor -- start 2024-04-02T07:36:07.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.106+0000 7f3195000700 1 -- start start 2024-04-02T07:36:07.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f3195000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31900d16a0 0x7f31901569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f3195000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31900d2370 0x7f319015fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f3195000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f319015d9d0 0x7f31901602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f3195000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319015eaa0 con 0x7f319015d9d0 2024-04-02T07:36:07.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f3195000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319015ed80 con 0x7f31900d16a0 2024-04-02T07:36:07.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f3195000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f319015f060 con 0x7f31900d2370 2024-04-02T07:36:07.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f318e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f319015d9d0 0x7f31901602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f318e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f319015d9d0 0x7f31901602d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52918/0 (socket says 172.21.15.89:52918) 2024-04-02T07:36:07.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f318e800700 1 -- 172.21.15.89:0/1766788746 learned_addr learned my addr 172.21.15.89:0/1766788746 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:07.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f318d400700 1 --2- 172.21.15.89:0/1766788746 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31900d2370 0x7f319015fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f318de00700 1 --2- 172.21.15.89:0/1766788746 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31900d16a0 0x7f31901569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f318e800700 1 -- 172.21.15.89:0/1766788746 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31900d2370 msgr2=0x7f319015fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f318e800700 1 --2- 172.21.15.89:0/1766788746 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31900d2370 0x7f319015fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f318e800700 1 -- 172.21.15.89:0/1766788746 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31900d16a0 msgr2=0x7f31901569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f318e800700 1 --2- 172.21.15.89:0/1766788746 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31900d16a0 0x7f31901569f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.108+0000 7f318e800700 1 -- 172.21.15.89:0/1766788746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317c009430 con 0x7f319015d9d0 2024-04-02T07:36:07.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.109+0000 7f318e800700 1 --2- 172.21.15.89:0/1766788746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f319015d9d0 0x7f31901602d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f31880017f0 tx=0x7f318800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:07.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.109+0000 7f3186000700 1 -- 172.21.15.89:0/1766788746 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318800ab70 con 0x7f319015d9d0 2024-04-02T07:36:07.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.109+0000 7f3186000700 1 -- 172.21.15.89:0/1766788746 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318800acf0 con 0x7f319015d9d0 2024-04-02T07:36:07.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.109+0000 7f3195000700 1 -- 172.21.15.89:0/1766788746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319015f340 con 0x7f319015d9d0 2024-04-02T07:36:07.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.109+0000 7f3186000700 1 -- 172.21.15.89:0/1766788746 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3188007310 con 0x7f319015d9d0 2024-04-02T07:36:07.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.109+0000 7f3195000700 1 -- 172.21.15.89:0/1766788746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319015ddb0 con 0x7f319015d9d0 2024-04-02T07:36:07.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.110+0000 7f3195000700 1 -- 172.21.15.89:0/1766788746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31900d1ca0 con 0x7f319015d9d0 2024-04-02T07:36:07.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.111+0000 7f3186000700 1 -- 172.21.15.89:0/1766788746 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3188007470 con 0x7f319015d9d0 2024-04-02T07:36:07.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.113+0000 7f3186000700 1 --2- 172.21.15.89:0/1766788746 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f317006ac90 0x7f317006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.113+0000 7f3186000700 1 -- 172.21.15.89:0/1766788746 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3188089290 con 0x7f319015d9d0 2024-04-02T07:36:07.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.113+0000 7f318de00700 1 --2- 172.21.15.89:0/1766788746 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f317006ac90 0x7f317006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.113+0000 7f318de00700 1 --2- 172.21.15.89:0/1766788746 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f317006ac90 0x7f317006d150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f31800011e0 tx=0x7f3180009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:07.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.117+0000 7f3186000700 1 -- 172.21.15.89:0/1766788746 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f31880543d0 con 0x7f319015d9d0 2024-04-02T07:36:07.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.317+0000 7f3195000700 1 -- 172.21.15.89:0/1766788746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3190067720 con 0x7f319015d9d0 2024-04-02T07:36:07.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.318+0000 7f3186000700 1 -- 172.21.15.89:0/1766788746 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f318800ae60 con 0x7f319015d9d0 2024-04-02T07:36:07.320 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:36:07.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.320+0000 7f3195000700 1 -- 172.21.15.89:0/1766788746 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f317006ac90 msgr2=0x7f317006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.320+0000 7f3195000700 1 --2- 172.21.15.89:0/1766788746 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f317006ac90 0x7f317006d150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f31800011e0 tx=0x7f3180009040 comp rx=0 tx=0).stop 2024-04-02T07:36:07.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.320+0000 7f3195000700 1 -- 172.21.15.89:0/1766788746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f319015d9d0 msgr2=0x7f31901602d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.320+0000 7f3195000700 1 --2- 172.21.15.89:0/1766788746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f319015d9d0 0x7f31901602d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f31880017f0 tx=0x7f318800a060 comp rx=0 tx=0).stop 2024-04-02T07:36:07.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.320+0000 7f3195000700 1 -- 172.21.15.89:0/1766788746 shutdown_connections 2024-04-02T07:36:07.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.320+0000 7f3195000700 1 --2- 172.21.15.89:0/1766788746 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f317006ac90 0x7f317006d150 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.320+0000 7f3195000700 1 --2- 172.21.15.89:0/1766788746 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31900d16a0 0x7f31901569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.320+0000 7f3195000700 1 --2- 172.21.15.89:0/1766788746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f319015d9d0 0x7f31901602d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.320+0000 7f3195000700 1 --2- 172.21.15.89:0/1766788746 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31900d2370 0x7f319015fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.320+0000 7f3195000700 1 -- 172.21.15.89:0/1766788746 >> 172.21.15.89:0/1766788746 conn(0x7f319008aff0 msgr2=0x7f31900d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:07.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.320+0000 7f3195000700 1 -- 172.21.15.89:0/1766788746 shutdown_connections 2024-04-02T07:36:07.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.320+0000 7f3195000700 1 -- 172.21.15.89:0/1766788746 wait complete. 2024-04-02T07:36:07.340 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:36:07.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.532+0000 7ff5c8a00700 1 Processor -- start 2024-04-02T07:36:07.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.532+0000 7ff5c8a00700 1 -- start start 2024-04-02T07:36:07.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.533+0000 7ff5c8a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff5c40d16a0 0x7ff5c40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.534+0000 7ff5c8a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c4161420 con 0x7ff5c415d9d0 2024-04-02T07:36:07.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.534+0000 7ff5c8a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c4161700 con 0x7ff5c40d2370 2024-04-02T07:36:07.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.534+0000 7ff5c8a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c40d4340 con 0x7ff5c40d16a0 2024-04-02T07:36:07.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.534+0000 7ff5c1e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff5c415d9d0 0x7ff5c40d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34994/0 (socket says 172.21.15.89:34994) 2024-04-02T07:36:07.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.534+0000 7ff5c1e00700 1 -- 172.21.15.89:0/4275463079 learned_addr learned my addr 172.21.15.89:0/4275463079 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:07.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.534+0000 7ff5c1400700 1 --2- 172.21.15.89:0/4275463079 >> v2:172.21.15.113:3300/0 conn(0x7ff5c40d16a0 0x7ff5c40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.534+0000 7ff5c0a00700 1 -- 172.21.15.89:0/4275463079 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871364755 0 0) 0x7ff5c4161420 con 0x7ff5c415d9d0 2024-04-02T07:36:07.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.534+0000 7ff5c0a00700 1 -- 172.21.15.89:0/4275463079 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a80035b0 con 0x7ff5c415d9d0 2024-04-02T07:36:07.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.534+0000 7ff5c0a00700 1 -- 172.21.15.89:0/4275463079 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630723496 0 0) 0x7ff5c4161700 con 0x7ff5c40d2370 2024-04-02T07:36:07.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.534+0000 7ff5c0a00700 1 -- 172.21.15.89:0/4275463079 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c4161420 con 0x7ff5c40d2370 2024-04-02T07:36:07.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.534+0000 7ff5c1400700 1 -- 172.21.15.89:0/4275463079 >> v1:172.21.15.113:6789/0 conn(0x7ff5c40d2370 legacy=0x7ff5c40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.534+0000 7ff5c1400700 1 -- 172.21.15.89:0/4275463079 >> v1:172.21.15.89:6789/0 conn(0x7ff5c415d9d0 legacy=0x7ff5c40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.534+0000 7ff5c1400700 1 -- 172.21.15.89:0/4275463079 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c41619e0 con 0x7ff5c40d16a0 2024-04-02T07:36:07.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.535+0000 7ff5c1400700 1 --2- 172.21.15.89:0/4275463079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5c40d16a0 0x7ff5c40d1e30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009820 tx=0x7ff5b0009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97eb56a95763bad server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:07.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.535+0000 7ff5c0a00700 1 -- 172.21.15.89:0/4275463079 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0005e30 con 0x7ff5c40d16a0 2024-04-02T07:36:07.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.535+0000 7ff5c0a00700 1 -- 172.21.15.89:0/4275463079 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b0006030 con 0x7ff5c40d16a0 2024-04-02T07:36:07.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.535+0000 7ff5c0a00700 1 -- 172.21.15.89:0/4275463079 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b00063a0 con 0x7ff5c40d16a0 2024-04-02T07:36:07.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.535+0000 7ff5c8a00700 1 -- 172.21.15.89:0/4275463079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5c40d16a0 msgr2=0x7ff5c40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.535+0000 7ff5c8a00700 1 --2- 172.21.15.89:0/4275463079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5c40d16a0 0x7ff5c40d1e30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009820 tx=0x7ff5b0009b30 comp rx=0 tx=0).stop 2024-04-02T07:36:07.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.536+0000 7ff5c8a00700 1 -- 172.21.15.89:0/4275463079 shutdown_connections 2024-04-02T07:36:07.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.536+0000 7ff5c8a00700 1 --2- 172.21.15.89:0/4275463079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5c40d16a0 0x7ff5c40d1e30 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.536+0000 7ff5c8a00700 1 -- 172.21.15.89:0/4275463079 >> 172.21.15.89:0/4275463079 conn(0x7ff5c408aff0 msgr2=0x7ff5c40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:07.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.536+0000 7ff5c8a00700 1 -- 172.21.15.89:0/4275463079 shutdown_connections 2024-04-02T07:36:07.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.536+0000 7ff5c8a00700 1 -- 172.21.15.89:0/4275463079 wait complete. 2024-04-02T07:36:07.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.537+0000 7ff5c8a00700 1 Processor -- start 2024-04-02T07:36:07.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.537+0000 7ff5c8a00700 1 -- start start 2024-04-02T07:36:07.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.538+0000 7ff5c8a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5c40d16a0 0x7ff5c4156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.538+0000 7ff5c8a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5c40d2370 0x7ff5c40ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.538+0000 7ff5c8a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5c415d9d0 0x7ff5c4158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.538+0000 7ff5c8a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c40ca5f0 con 0x7ff5c415d9d0 2024-04-02T07:36:07.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.538+0000 7ff5c8a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c40cb170 con 0x7ff5c40d2370 2024-04-02T07:36:07.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.538+0000 7ff5c8a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c4157030 con 0x7ff5c40d16a0 2024-04-02T07:36:07.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.539+0000 7ff5c2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5c415d9d0 0x7ff5c4158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.539+0000 7ff5c1400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5c40d2370 0x7ff5c40ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.539+0000 7ff5c1e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5c40d16a0 0x7ff5c4156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.539+0000 7ff5c2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5c415d9d0 0x7ff5c4158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52930/0 (socket says 172.21.15.89:52930) 2024-04-02T07:36:07.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.539+0000 7ff5c1400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5c40d2370 0x7ff5c40ccb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33912/0 (socket says 172.21.15.89:33912) 2024-04-02T07:36:07.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.539+0000 7ff5c2800700 1 -- 172.21.15.89:0/569187207 learned_addr learned my addr 172.21.15.89:0/569187207 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:07.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.539+0000 7ff5c2800700 1 -- 172.21.15.89:0/569187207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5c40d16a0 msgr2=0x7ff5c4156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.539+0000 7ff5c2800700 1 --2- 172.21.15.89:0/569187207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5c40d16a0 0x7ff5c4156af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.539+0000 7ff5c2800700 1 -- 172.21.15.89:0/569187207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5c40d2370 msgr2=0x7ff5c40ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.539+0000 7ff5c2800700 1 --2- 172.21.15.89:0/569187207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5c40d2370 0x7ff5c40ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.539+0000 7ff5c2800700 1 -- 172.21.15.89:0/569187207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b0009430 con 0x7ff5c415d9d0 2024-04-02T07:36:07.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.539+0000 7ff5c2800700 1 --2- 172.21.15.89:0/569187207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5c415d9d0 0x7ff5c4158140 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00cc50 tx=0x7ff5bc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:07.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.540+0000 7ff5ba000700 1 -- 172.21.15.89:0/569187207 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc003c00 con 0x7ff5c415d9d0 2024-04-02T07:36:07.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.540+0000 7ff5ba000700 1 -- 172.21.15.89:0/569187207 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5bc003d60 con 0x7ff5c415d9d0 2024-04-02T07:36:07.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.540+0000 7ff5c8a00700 1 -- 172.21.15.89:0/569187207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c40ca840 con 0x7ff5c415d9d0 2024-04-02T07:36:07.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.540+0000 7ff5ba000700 1 -- 172.21.15.89:0/569187207 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc00aad0 con 0x7ff5c415d9d0 2024-04-02T07:36:07.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.540+0000 7ff5c8a00700 1 -- 172.21.15.89:0/569187207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c40cae00 con 0x7ff5c415d9d0 2024-04-02T07:36:07.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.541+0000 7ff5c8a00700 1 -- 172.21.15.89:0/569187207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c40d1ca0 con 0x7ff5c415d9d0 2024-04-02T07:36:07.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.542+0000 7ff5ba000700 1 -- 172.21.15.89:0/569187207 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff5bc00ac30 con 0x7ff5c415d9d0 2024-04-02T07:36:07.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.543+0000 7ff5ba000700 1 --2- 172.21.15.89:0/569187207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff5a406ac90 0x7ff5a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.543+0000 7ff5ba000700 1 -- 172.21.15.89:0/569187207 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5bc089400 con 0x7ff5c415d9d0 2024-04-02T07:36:07.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.543+0000 7ff5c1e00700 1 --2- 172.21.15.89:0/569187207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff5a406ac90 0x7ff5a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.544+0000 7ff5c1e00700 1 --2- 172.21.15.89:0/569187207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff5a406ac90 0x7ff5a406d150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4007fd0 tx=0x7ff5b4007de0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:07.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.547+0000 7ff5ba000700 1 -- 172.21.15.89:0/569187207 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff5bc054540 con 0x7ff5c415d9d0 2024-04-02T07:36:07.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.747+0000 7ff5c8a00700 1 -- 172.21.15.89:0/569187207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff5c40d3310 con 0x7ff5c415d9d0 2024-04-02T07:36:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.747+0000 7ff5ba000700 1 -- 172.21.15.89:0/569187207 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff5bc057b60 con 0x7ff5c415d9d0 2024-04-02T07:36:07.749 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:36:07.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.750+0000 7ff5c8a00700 1 -- 172.21.15.89:0/569187207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff5a406ac90 msgr2=0x7ff5a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.750+0000 7ff5c8a00700 1 --2- 172.21.15.89:0/569187207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff5a406ac90 0x7ff5a406d150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4007fd0 tx=0x7ff5b4007de0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.750+0000 7ff5c8a00700 1 -- 172.21.15.89:0/569187207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5c415d9d0 msgr2=0x7ff5c4158140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.750+0000 7ff5c8a00700 1 --2- 172.21.15.89:0/569187207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5c415d9d0 0x7ff5c4158140 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00cc50 tx=0x7ff5bc00cf60 comp rx=0 tx=0).stop 2024-04-02T07:36:07.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.750+0000 7ff5c8a00700 1 -- 172.21.15.89:0/569187207 shutdown_connections 2024-04-02T07:36:07.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.750+0000 7ff5c8a00700 1 --2- 172.21.15.89:0/569187207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff5a406ac90 0x7ff5a406d150 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.750+0000 7ff5c8a00700 1 --2- 172.21.15.89:0/569187207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5c415d9d0 0x7ff5c4158140 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.750+0000 7ff5c8a00700 1 --2- 172.21.15.89:0/569187207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5c40d16a0 0x7ff5c4156af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.750+0000 7ff5c8a00700 1 --2- 172.21.15.89:0/569187207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5c40d2370 0x7ff5c40ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.750+0000 7ff5c8a00700 1 -- 172.21.15.89:0/569187207 >> 172.21.15.89:0/569187207 conn(0x7ff5c408aff0 msgr2=0x7ff5c40b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:07.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.750+0000 7ff5c8a00700 1 -- 172.21.15.89:0/569187207 shutdown_connections 2024-04-02T07:36:07.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.750+0000 7ff5c8a00700 1 -- 172.21.15.89:0/569187207 wait complete. 2024-04-02T07:36:07.771 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:36:07.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.961+0000 7ff341e00700 1 Processor -- start 2024-04-02T07:36:07.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.962+0000 7ff341e00700 1 -- start start 2024-04-02T07:36:07.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff341e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff33c15d9d0 0x7ff33c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff341e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff33c0d16a0 0x7ff33c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff341e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff33c0d2370 0x7ff33c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff341e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33c0d2820 con 0x7ff33c0d16a0 2024-04-02T07:36:07.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff341e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33c161420 con 0x7ff33c0d2370 2024-04-02T07:36:07.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff341e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c161700 con 0x7ff33c15d9d0 2024-04-02T07:36:07.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff33a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff33c0d16a0 0x7ff33c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff33a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff33c0d16a0 0x7ff33c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52942/0 (socket says 172.21.15.89:52942) 2024-04-02T07:36:07.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff33a000700 1 -- 172.21.15.89:0/3946776770 learned_addr learned my addr 172.21.15.89:0/3946776770 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:07.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff33b400700 1 --2- 172.21.15.89:0/3946776770 >> v2:172.21.15.113:3300/0 conn(0x7ff33c0d2370 0x7ff33c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff33aa00700 1 --2- 172.21.15.89:0/3946776770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff33c15d9d0 0x7ff33c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff33a000700 1 -- 172.21.15.89:0/3946776770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff33c15d9d0 msgr2=0x7ff33c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:07.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff33a000700 1 --2- 172.21.15.89:0/3946776770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff33c15d9d0 0x7ff33c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff33a000700 1 -- 172.21.15.89:0/3946776770 >> v2:172.21.15.113:3300/0 conn(0x7ff33c0d2370 msgr2=0x7ff33c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:07.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff33a000700 1 --2- 172.21.15.89:0/3946776770 >> v2:172.21.15.113:3300/0 conn(0x7ff33c0d2370 0x7ff33c15ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff33a000700 1 -- 172.21.15.89:0/3946776770 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33c1619e0 con 0x7ff33c0d16a0 2024-04-02T07:36:07.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.963+0000 7ff33aa00700 1 --2- 172.21.15.89:0/3946776770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff33c15d9d0 0x7ff33c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:07.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.964+0000 7ff33a000700 1 --2- 172.21.15.89:0/3946776770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff33c0d16a0 0x7ff33c0d1e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff33c0cdc00 tx=0x7ff328009bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca031109f6d02a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:07.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.964+0000 7ff339600700 1 -- 172.21.15.89:0/3946776770 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32800b8a0 con 0x7ff33c0d16a0 2024-04-02T07:36:07.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.964+0000 7ff339600700 1 -- 172.21.15.89:0/3946776770 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32800ba00 con 0x7ff33c0d16a0 2024-04-02T07:36:07.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.964+0000 7ff339600700 1 -- 172.21.15.89:0/3946776770 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32800bdb0 con 0x7ff33c0d16a0 2024-04-02T07:36:07.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.964+0000 7ff341e00700 1 -- 172.21.15.89:0/3946776770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff33c0d16a0 msgr2=0x7ff33c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.964+0000 7ff341e00700 1 --2- 172.21.15.89:0/3946776770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff33c0d16a0 0x7ff33c0d1e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff33c0cdc00 tx=0x7ff328009bc0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.964+0000 7ff341e00700 1 -- 172.21.15.89:0/3946776770 shutdown_connections 2024-04-02T07:36:07.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.964+0000 7ff341e00700 1 --2- 172.21.15.89:0/3946776770 >> v2:172.21.15.113:3300/0 conn(0x7ff33c0d2370 0x7ff33c15ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.964+0000 7ff341e00700 1 --2- 172.21.15.89:0/3946776770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff33c15d9d0 0x7ff33c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.964+0000 7ff341e00700 1 --2- 172.21.15.89:0/3946776770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff33c0d16a0 0x7ff33c0d1e30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.965+0000 7ff341e00700 1 -- 172.21.15.89:0/3946776770 >> 172.21.15.89:0/3946776770 conn(0x7ff33c08aff0 msgr2=0x7ff33c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:07.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.965+0000 7ff341e00700 1 -- 172.21.15.89:0/3946776770 shutdown_connections 2024-04-02T07:36:07.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.965+0000 7ff341e00700 1 -- 172.21.15.89:0/3946776770 wait complete. 2024-04-02T07:36:07.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.966+0000 7ff341e00700 1 Processor -- start 2024-04-02T07:36:07.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.966+0000 7ff341e00700 1 -- start start 2024-04-02T07:36:07.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.967+0000 7ff341e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff33c0d16a0 0x7ff33c0cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.967+0000 7ff341e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff33c0d2370 0x7ff33c0b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.967+0000 7ff341e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c15d9d0 0x7ff33c0b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.967+0000 7ff341e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c0b7450 con 0x7ff33c0d2370 2024-04-02T07:36:07.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.967+0000 7ff341e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c0cab60 con 0x7ff33c15d9d0 2024-04-02T07:36:07.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.967+0000 7ff341e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c0cae40 con 0x7ff33c0d16a0 2024-04-02T07:36:07.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff33a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff33c0d2370 0x7ff33c0b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff33a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff33c0d2370 0x7ff33c0b6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52948/0 (socket says 172.21.15.89:52948) 2024-04-02T07:36:07.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff33a000700 1 -- 172.21.15.89:0/3518620381 learned_addr learned my addr 172.21.15.89:0/3518620381 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:07.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff33b400700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c15d9d0 0x7ff33c0b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff33aa00700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff33c0d16a0 0x7ff33c0cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff33a000700 1 -- 172.21.15.89:0/3518620381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff33c0d16a0 msgr2=0x7ff33c0cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff33a000700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff33c0d16a0 0x7ff33c0cbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff33a000700 1 -- 172.21.15.89:0/3518620381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c15d9d0 msgr2=0x7ff33c0b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:07.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff33a000700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c15d9d0 0x7ff33c0b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:07.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff33a000700 1 -- 172.21.15.89:0/3518620381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3280094f0 con 0x7ff33c0d2370 2024-04-02T07:36:07.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff33aa00700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff33c0d16a0 0x7ff33c0cbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:07.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff33a000700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff33c0d2370 0x7ff33c0b6f10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff328000c00 tx=0x7ff328004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:07.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff333400700 1 -- 172.21.15.89:0/3518620381 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3280118f0 con 0x7ff33c0d2370 2024-04-02T07:36:07.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff333400700 1 -- 172.21.15.89:0/3518620381 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff328011a50 con 0x7ff33c0d2370 2024-04-02T07:36:07.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff333400700 1 -- 172.21.15.89:0/3518620381 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff328011e00 con 0x7ff33c0d2370 2024-04-02T07:36:07.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.968+0000 7ff341e00700 1 -- 172.21.15.89:0/3518620381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33c0cb120 con 0x7ff33c0d2370 2024-04-02T07:36:07.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.969+0000 7ff341e00700 1 -- 172.21.15.89:0/3518620381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33c0cb6e0 con 0x7ff33c0d2370 2024-04-02T07:36:07.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.969+0000 7ff341e00700 1 -- 172.21.15.89:0/3518620381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff33c0d1ca0 con 0x7ff33c0d2370 2024-04-02T07:36:07.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.970+0000 7ff333400700 1 -- 172.21.15.89:0/3518620381 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff32802d440 con 0x7ff33c0d2370 2024-04-02T07:36:07.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.971+0000 7ff333400700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff31c06ac90 0x7ff31c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:07.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.971+0000 7ff333400700 1 -- 172.21.15.89:0/3518620381 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff32808aa80 con 0x7ff33c0d2370 2024-04-02T07:36:07.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.971+0000 7ff33aa00700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff31c06ac90 0x7ff31c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:07.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.972+0000 7ff33aa00700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff31c06ac90 0x7ff31c06d150 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff32c008910 tx=0x7ff32c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:07.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:07.976+0000 7ff333400700 1 -- 172.21.15.89:0/3518620381 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff328055c50 con 0x7ff33c0d2370 2024-04-02T07:36:08.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.176+0000 7ff341e00700 1 -- 172.21.15.89:0/3518620381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff33c06c910 con 0x7ff33c0d2370 2024-04-02T07:36:08.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.176+0000 7ff333400700 1 -- 172.21.15.89:0/3518620381 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff328059270 con 0x7ff33c0d2370 2024-04-02T07:36:08.178 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:36:08.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.179+0000 7ff341e00700 1 -- 172.21.15.89:0/3518620381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff31c06ac90 msgr2=0x7ff31c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:08.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.179+0000 7ff341e00700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff31c06ac90 0x7ff31c06d150 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff32c008910 tx=0x7ff32c008040 comp rx=0 tx=0).stop 2024-04-02T07:36:08.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.179+0000 7ff341e00700 1 -- 172.21.15.89:0/3518620381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff33c0d2370 msgr2=0x7ff33c0b6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:08.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.179+0000 7ff341e00700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff33c0d2370 0x7ff33c0b6f10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff328000c00 tx=0x7ff328004750 comp rx=0 tx=0).stop 2024-04-02T07:36:08.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.179+0000 7ff341e00700 1 -- 172.21.15.89:0/3518620381 shutdown_connections 2024-04-02T07:36:08.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.179+0000 7ff341e00700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff31c06ac90 0x7ff31c06d150 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.179+0000 7ff341e00700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c15d9d0 0x7ff33c0b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.179+0000 7ff341e00700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff33c0d16a0 0x7ff33c0cbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.179+0000 7ff341e00700 1 --2- 172.21.15.89:0/3518620381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff33c0d2370 0x7ff33c0b6f10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.179+0000 7ff341e00700 1 -- 172.21.15.89:0/3518620381 >> 172.21.15.89:0/3518620381 conn(0x7ff33c08aff0 msgr2=0x7ff33c0b0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:08.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.179+0000 7ff341e00700 1 -- 172.21.15.89:0/3518620381 shutdown_connections 2024-04-02T07:36:08.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.179+0000 7ff341e00700 1 -- 172.21.15.89:0/3518620381 wait complete. 2024-04-02T07:36:08.199 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:36:08.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.387+0000 7f31ae600700 1 Processor -- start 2024-04-02T07:36:08.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.387+0000 7f31ae600700 1 -- start start 2024-04-02T07:36:08.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.388+0000 7f31ae600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a815d670 0x7f31a80ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:08.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.388+0000 7f31ae600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f31a815da50 0x7f31a80b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:08.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.388+0000 7f31ae600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a80b3890 con 0x7f31a80b3000 2024-04-02T07:36:08.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.388+0000 7f31ae600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a80acac0 con 0x7f31a815da50 2024-04-02T07:36:08.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.388+0000 7f31ae600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a8159f50 con 0x7f31a815d670 2024-04-02T07:36:08.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a7e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f31a80b3000 0x7f31a806dee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35004/0 (socket says 172.21.15.89:35004) 2024-04-02T07:36:08.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a7e00700 1 -- 172.21.15.89:0/1564363370 learned_addr learned my addr 172.21.15.89:0/1564363370 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:08.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a6a00700 1 --2- 172.21.15.89:0/1564363370 >> v2:172.21.15.113:3300/0 conn(0x7f31a815da50 0x7f31a80b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:08.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a7400700 1 --2- 172.21.15.89:0/1564363370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a815d670 0x7f31a80ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:08.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a6000700 1 -- 172.21.15.89:0/1564363370 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169206206 0 0) 0x7f31a80b3890 con 0x7f31a80b3000 2024-04-02T07:36:08.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a6000700 1 -- 172.21.15.89:0/1564363370 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f318c0035b0 con 0x7f31a80b3000 2024-04-02T07:36:08.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a6000700 1 -- 172.21.15.89:0/1564363370 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1386682829 0 0) 0x7f318c0035b0 con 0x7f31a80b3000 2024-04-02T07:36:08.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a6000700 1 -- 172.21.15.89:0/1564363370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a815d670 msgr2=0x7f31a80ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:08.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a6000700 1 --2- 172.21.15.89:0/1564363370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a815d670 0x7f31a80ac580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a6000700 1 -- 172.21.15.89:0/1564363370 >> v2:172.21.15.113:3300/0 conn(0x7f31a815da50 msgr2=0x7f31a80b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a6000700 1 --2- 172.21.15.89:0/1564363370 >> v2:172.21.15.113:3300/0 conn(0x7f31a815da50 0x7f31a80b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a6000700 1 -- 172.21.15.89:0/1564363370 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a80b35b0 con 0x7f31a80b3000 2024-04-02T07:36:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a6000700 1 -- 172.21.15.89:0/1564363370 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f31a00019f0 con 0x7f31a80b3000 2024-04-02T07:36:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a6a00700 1 --2- 172.21.15.89:0/1564363370 >> v2:172.21.15.113:3300/0 conn(0x7f31a815da50 0x7f31a80b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a6000700 1 -- 172.21.15.89:0/1564363370 >> v1:172.21.15.89:6789/0 conn(0x7f31a80b3000 legacy=0x7f31a806dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a6000700 1 --2- 172.21.15.89:0/1564363370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f318c003a00 0x7f318c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.389+0000 7f31a6000700 1 --2- 172.21.15.89:0/1564363370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f318c004850 0x7f318c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a6000700 1 --2- 172.21.15.89:0/1564363370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a815d670 0x7f318c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a6000700 1 -- 172.21.15.89:0/1564363370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80acac0 con 0x7f318c003a00 2024-04-02T07:36:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a6000700 1 -- 172.21.15.89:0/1564363370 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a8159f50 con 0x7f318c004850 2024-04-02T07:36:08.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a6000700 1 -- 172.21.15.89:0/1564363370 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f318c00dc20 con 0x7f31a815d670 2024-04-02T07:36:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a7400700 1 --2- 172.21.15.89:0/1564363370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f318c003a00 0x7f318c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a7e00700 1 --2- 172.21.15.89:0/1564363370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a815d670 0x7f318c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a6a00700 1 --2- 172.21.15.89:0/1564363370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f318c004850 0x7f318c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a7400700 1 -- 172.21.15.89:0/1564363370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a815d670 msgr2=0x7f318c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a7400700 1 --2- 172.21.15.89:0/1564363370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a815d670 0x7f318c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a7400700 1 -- 172.21.15.89:0/1564363370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f318c004850 msgr2=0x7f318c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a7400700 1 --2- 172.21.15.89:0/1564363370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f318c004850 0x7f318c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a7400700 1 -- 172.21.15.89:0/1564363370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318c0035b0 con 0x7f318c003a00 2024-04-02T07:36:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a6a00700 1 --2- 172.21.15.89:0/1564363370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f318c004850 0x7f318c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a7e00700 1 --2- 172.21.15.89:0/1564363370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a815d670 0x7f318c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a7400700 1 --2- 172.21.15.89:0/1564363370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f318c003a00 0x7f318c003e20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3198009b60 tx=0x7f318c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e332d352aef3ba9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:08.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a6000700 1 -- 172.21.15.89:0/1564363370 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319800c070 con 0x7f318c003a00 2024-04-02T07:36:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a6000700 1 -- 172.21.15.89:0/1564363370 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319800f970 con 0x7f318c003a00 2024-04-02T07:36:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.390+0000 7f31a6000700 1 -- 172.21.15.89:0/1564363370 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319800fce0 con 0x7f318c003a00 2024-04-02T07:36:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.391+0000 7f31ae600700 1 -- 172.21.15.89:0/1564363370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f318c003a00 msgr2=0x7f318c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.391+0000 7f31ae600700 1 --2- 172.21.15.89:0/1564363370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f318c003a00 0x7f318c003e20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3198009b60 tx=0x7f318c005880 comp rx=0 tx=0).stop 2024-04-02T07:36:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.391+0000 7f31a7e00700 1 -- 172.21.15.89:0/1564363370 reap_dead start 2024-04-02T07:36:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.391+0000 7f31ae600700 1 -- 172.21.15.89:0/1564363370 shutdown_connections 2024-04-02T07:36:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.391+0000 7f31ae600700 1 -- 172.21.15.89:0/1564363370 >> 172.21.15.89:0/1564363370 conn(0x7f31a80cf190 msgr2=0x7f31a80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.391+0000 7f31ae600700 1 -- 172.21.15.89:0/1564363370 shutdown_connections 2024-04-02T07:36:08.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.391+0000 7f31ae600700 1 -- 172.21.15.89:0/1564363370 wait complete. 2024-04-02T07:36:08.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.393+0000 7f31ae600700 1 Processor -- start 2024-04-02T07:36:08.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.393+0000 7f31ae600700 1 -- start start 2024-04-02T07:36:08.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31ae600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f318c003a00 0x7f31a80c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31ae600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a80cd950 0x7f31a81f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31ae600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a80afb30 0x7f31a80affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31ae600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80ca3d0 con 0x7f31a80cd950 2024-04-02T07:36:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31ae600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80b0520 con 0x7f318c003a00 2024-04-02T07:36:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31ae600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80c9750 con 0x7f31a80afb30 2024-04-02T07:36:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31a6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a80cd950 0x7f31a81f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31a6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a80cd950 0x7f31a81f2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52962/0 (socket says 172.21.15.89:52962) 2024-04-02T07:36:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31a6a00700 1 -- 172.21.15.89:0/3667359485 learned_addr learned my addr 172.21.15.89:0/3667359485 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31a7400700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f318c003a00 0x7f31a80c8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31a7e00700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a80afb30 0x7f31a80affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31a6a00700 1 -- 172.21.15.89:0/3667359485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a80afb30 msgr2=0x7f31a80affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31a6a00700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a80afb30 0x7f31a80affe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31a6a00700 1 -- 172.21.15.89:0/3667359485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f318c003a00 msgr2=0x7f31a80c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:08.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31a6a00700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f318c003a00 0x7f31a80c8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31a6a00700 1 -- 172.21.15.89:0/3667359485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a80c9a30 con 0x7f31a80cd950 2024-04-02T07:36:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.394+0000 7f31a7e00700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a80afb30 0x7f31a80affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.395+0000 7f31a6a00700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a80cd950 0x7f31a81f2e20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3194011940 tx=0x7f3194011d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.395+0000 7f319f400700 1 -- 172.21.15.89:0/3667359485 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319400fae0 con 0x7f31a80cd950 2024-04-02T07:36:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.395+0000 7f319f400700 1 -- 172.21.15.89:0/3667359485 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319400fc40 con 0x7f31a80cd950 2024-04-02T07:36:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.395+0000 7f31ae600700 1 -- 172.21.15.89:0/3667359485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a80c9d10 con 0x7f31a80cd950 2024-04-02T07:36:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.395+0000 7f319f400700 1 -- 172.21.15.89:0/3667359485 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3194015460 con 0x7f31a80cd950 2024-04-02T07:36:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.395+0000 7f31ae600700 1 -- 172.21.15.89:0/3667359485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a80c9380 con 0x7f31a80cd950 2024-04-02T07:36:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.396+0000 7f31ae600700 1 -- 172.21.15.89:0/3667359485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a80ca510 con 0x7f31a80cd950 2024-04-02T07:36:08.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.397+0000 7f319f400700 1 -- 172.21.15.89:0/3667359485 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f31940155c0 con 0x7f31a80cd950 2024-04-02T07:36:08.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.398+0000 7f319f400700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f318806ac90 0x7f318806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:08.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.398+0000 7f319f400700 1 -- 172.21.15.89:0/3667359485 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3194089cc0 con 0x7f31a80cd950 2024-04-02T07:36:08.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.398+0000 7f31a7400700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f318806ac90 0x7f318806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:08.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.399+0000 7f31a7400700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f318806ac90 0x7f318806d150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f31a81f6b10 tx=0x7f3198014000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:08.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.403+0000 7f319f400700 1 -- 172.21.15.89:0/3667359485 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f31a80ca510 con 0x7f31a80cd950 2024-04-02T07:36:08.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.647+0000 7f31ae600700 1 -- 172.21.15.89:0/3667359485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31a80ca510 con 0x7f31a80cd950 2024-04-02T07:36:08.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.648+0000 7f319f400700 1 -- 172.21.15.89:0/3667359485 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4731 (secure 0 0 0) 0x7f31a80ca510 con 0x7f31a80cd950 2024-04-02T07:36:08.650 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:08.650 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2427593953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2427593953},{"type":"v1","addr":"172.21.15.113:6835","nonce":2427593953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/926246816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":926246816},{"type":"v1","addr":"172.21.15.89:6836","nonce":926246816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2701685411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2701685411},{"type":"v1","addr":"172.21.15.89:6837","nonce":2701685411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:35:53.235047+0000","modified":"2024-04-02T07:35:56.245811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6834/3962490902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3962490902},{"type":"v1","addr":"172.21.15.113:6834","nonce":3962490902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-02T07:36:08.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.650+0000 7f31ae600700 1 -- 172.21.15.89:0/3667359485 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f318806ac90 msgr2=0x7f318806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:08.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.650+0000 7f31ae600700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f318806ac90 0x7f318806d150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f31a81f6b10 tx=0x7f3198014000 comp rx=0 tx=0).stop 2024-04-02T07:36:08.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.650+0000 7f31ae600700 1 -- 172.21.15.89:0/3667359485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a80cd950 msgr2=0x7f31a81f2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:08.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.650+0000 7f31ae600700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a80cd950 0x7f31a81f2e20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3194011940 tx=0x7f3194011d00 comp rx=0 tx=0).stop 2024-04-02T07:36:08.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.650+0000 7f31ae600700 1 -- 172.21.15.89:0/3667359485 shutdown_connections 2024-04-02T07:36:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.650+0000 7f31ae600700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f318806ac90 0x7f318806d150 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.650+0000 7f31ae600700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f318c003a00 0x7f31a80c8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.650+0000 7f31ae600700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31a80afb30 0x7f31a80affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.650+0000 7f31ae600700 1 --2- 172.21.15.89:0/3667359485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a80cd950 0x7f31a81f2e20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.650+0000 7f31ae600700 1 -- 172.21.15.89:0/3667359485 >> 172.21.15.89:0/3667359485 conn(0x7f31a80cf190 msgr2=0x7f31a8157e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.650+0000 7f31ae600700 1 -- 172.21.15.89:0/3667359485 shutdown_connections 2024-04-02T07:36:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.650+0000 7f31ae600700 1 -- 172.21.15.89:0/3667359485 wait complete. 2024-04-02T07:36:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2024-04-02T07:36:08.670 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:36:08.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.863+0000 7f6160400700 1 Processor -- start 2024-04-02T07:36:08.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.863+0000 7f6160400700 1 -- start start 2024-04-02T07:36:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.864+0000 7f6160400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f61580d18b0 0x7f61580cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.864+0000 7f6160400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61580ceda0 con 0x7f6158160f10 2024-04-02T07:36:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.864+0000 7f6160400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61581612f0 con 0x7f6158161590 2024-04-02T07:36:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.864+0000 7f6160400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61580d1d30 con 0x7f61580d18b0 2024-04-02T07:36:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.865+0000 7f615d000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f61580d18b0 0x7f61580cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.865+0000 7f615d000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f61580d18b0 0x7f61580cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47096/0 (socket says 172.21.15.89:47096) 2024-04-02T07:36:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.865+0000 7f615d000700 1 -- 172.21.15.89:0/99272254 learned_addr learned my addr 172.21.15.89:0/99272254 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.865+0000 7f615da00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6158160f10 0x7f61580d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56068/0 (socket says 172.21.15.89:56068) 2024-04-02T07:36:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.865+0000 7f615d000700 1 -- 172.21.15.89:0/99272254 >> v1:172.21.15.113:6789/0 conn(0x7f6158161590 legacy=0x7f61580b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:08.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.865+0000 7f615d000700 1 -- 172.21.15.89:0/99272254 >> v1:172.21.15.89:6789/0 conn(0x7f6158160f10 legacy=0x7f61580d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:08.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.865+0000 7f615d000700 1 -- 172.21.15.89:0/99272254 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61580ce470 con 0x7f61580d18b0 2024-04-02T07:36:08.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.865+0000 7f6157e00700 1 -- 172.21.15.89:0/99272254 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477303827 0 0) 0x7f61580ceda0 con 0x7f6158160f10 2024-04-02T07:36:08.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.865+0000 7f615d000700 1 --2- 172.21.15.89:0/99272254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61580d18b0 0x7f61580cf230 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f61440098f0 tx=0x7f6144009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67fc4462a5ebdee1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:08.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.865+0000 7f6157e00700 1 -- 172.21.15.89:0/99272254 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614400b8a0 con 0x7f61580d18b0 2024-04-02T07:36:08.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.865+0000 7f6157e00700 1 -- 172.21.15.89:0/99272254 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614400ba00 con 0x7f61580d18b0 2024-04-02T07:36:08.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.866+0000 7f6157e00700 1 -- 172.21.15.89:0/99272254 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614400bd90 con 0x7f61580d18b0 2024-04-02T07:36:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.866+0000 7f6160400700 1 -- 172.21.15.89:0/99272254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61580d18b0 msgr2=0x7f61580cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.866+0000 7f6160400700 1 --2- 172.21.15.89:0/99272254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61580d18b0 0x7f61580cf230 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f61440098f0 tx=0x7f6144009c00 comp rx=0 tx=0).stop 2024-04-02T07:36:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.866+0000 7f6160400700 1 -- 172.21.15.89:0/99272254 shutdown_connections 2024-04-02T07:36:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.866+0000 7f6160400700 1 --2- 172.21.15.89:0/99272254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61580d18b0 0x7f61580cf230 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.866+0000 7f6160400700 1 -- 172.21.15.89:0/99272254 >> 172.21.15.89:0/99272254 conn(0x7f615808aff0 msgr2=0x7f61580d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.866+0000 7f6160400700 1 -- 172.21.15.89:0/99272254 shutdown_connections 2024-04-02T07:36:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.866+0000 7f6160400700 1 -- 172.21.15.89:0/99272254 wait complete. 2024-04-02T07:36:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.868+0000 7f6160400700 1 Processor -- start 2024-04-02T07:36:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.869+0000 7f6160400700 1 -- start start 2024-04-02T07:36:08.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f6160400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61580d18b0 0x7f615815ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:08.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f6160400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6158160f10 0x7f615815e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:08.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f6160400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6158161590 0x7f61581f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:08.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f6160400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615815fe90 con 0x7f6158161590 2024-04-02T07:36:08.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f6160400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615815e760 con 0x7f6158160f10 2024-04-02T07:36:08.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f6160400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f615815f280 con 0x7f61580d18b0 2024-04-02T07:36:08.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f615e400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6158161590 0x7f61581f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:08.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f615e400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6158161590 0x7f61581f6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47112/0 (socket says 172.21.15.89:47112) 2024-04-02T07:36:08.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f615e400700 1 -- 172.21.15.89:0/1386905044 learned_addr learned my addr 172.21.15.89:0/1386905044 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f615da00700 1 --2- 172.21.15.89:0/1386905044 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61580d18b0 0x7f615815ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f615d000700 1 --2- 172.21.15.89:0/1386905044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6158160f10 0x7f615815e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f615e400700 1 -- 172.21.15.89:0/1386905044 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61580d18b0 msgr2=0x7f615815ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f615e400700 1 --2- 172.21.15.89:0/1386905044 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61580d18b0 0x7f615815ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f615e400700 1 -- 172.21.15.89:0/1386905044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6158160f10 msgr2=0x7f615815e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f615e400700 1 --2- 172.21.15.89:0/1386905044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6158160f10 0x7f615815e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.870+0000 7f615e400700 1 -- 172.21.15.89:0/1386905044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6144009550 con 0x7f6158161590 2024-04-02T07:36:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.871+0000 7f615e400700 1 --2- 172.21.15.89:0/1386905044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6158161590 0x7f61581f6a10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f615000cb50 tx=0x7f615000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.871+0000 7f6155600700 1 -- 172.21.15.89:0/1386905044 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6150003c00 con 0x7f6158161590 2024-04-02T07:36:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.871+0000 7f6155600700 1 -- 172.21.15.89:0/1386905044 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6150003d60 con 0x7f6158161590 2024-04-02T07:36:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.871+0000 7f6155600700 1 -- 172.21.15.89:0/1386905044 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615000aa20 con 0x7f6158161590 2024-04-02T07:36:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.871+0000 7f6160400700 1 -- 172.21.15.89:0/1386905044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f615815f560 con 0x7f6158161590 2024-04-02T07:36:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.871+0000 7f6160400700 1 -- 172.21.15.89:0/1386905044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f615815fb20 con 0x7f6158161590 2024-04-02T07:36:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.872+0000 7f6160400700 1 -- 172.21.15.89:0/1386905044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6158160000 con 0x7f6158161590 2024-04-02T07:36:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.873+0000 7f6155600700 1 -- 172.21.15.89:0/1386905044 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f615000ab80 con 0x7f6158161590 2024-04-02T07:36:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.874+0000 7f6155600700 1 --2- 172.21.15.89:0/1386905044 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f613806ac90 0x7f613806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.874+0000 7f6155600700 1 -- 172.21.15.89:0/1386905044 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6150089350 con 0x7f6158161590 2024-04-02T07:36:08.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.874+0000 7f615da00700 1 --2- 172.21.15.89:0/1386905044 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f613806ac90 0x7f613806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:08.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.875+0000 7f615da00700 1 --2- 172.21.15.89:0/1386905044 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f613806ac90 0x7f613806d150 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f614c000f60 tx=0x7f614c004a50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:08.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:08.878+0000 7f6155600700 1 -- 172.21.15.89:0/1386905044 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6150054520 con 0x7f6158161590 2024-04-02T07:36:09.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.081+0000 7f6160400700 1 -- 172.21.15.89:0/1386905044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6158067720 con 0x7f6158161590 2024-04-02T07:36:09.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.082+0000 7f6155600700 1 -- 172.21.15.89:0/1386905044 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f6150057b40 con 0x7f6158161590 2024-04-02T07:36:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:09.084 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:54.238913+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:36:09.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.085+0000 7f6160400700 1 -- 172.21.15.89:0/1386905044 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f613806ac90 msgr2=0x7f613806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:09.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.085+0000 7f6160400700 1 --2- 172.21.15.89:0/1386905044 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f613806ac90 0x7f613806d150 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f614c000f60 tx=0x7f614c004a50 comp rx=0 tx=0).stop 2024-04-02T07:36:09.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.085+0000 7f6160400700 1 -- 172.21.15.89:0/1386905044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6158161590 msgr2=0x7f61581f6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:09.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.085+0000 7f6160400700 1 --2- 172.21.15.89:0/1386905044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6158161590 0x7f61581f6a10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f615000cb50 tx=0x7f615000ce60 comp rx=0 tx=0).stop 2024-04-02T07:36:09.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.085+0000 7f6160400700 1 -- 172.21.15.89:0/1386905044 shutdown_connections 2024-04-02T07:36:09.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.085+0000 7f6160400700 1 --2- 172.21.15.89:0/1386905044 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f613806ac90 0x7f613806d150 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:09.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.085+0000 7f6160400700 1 --2- 172.21.15.89:0/1386905044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6158161590 0x7f61581f6a10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:09.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.085+0000 7f6160400700 1 --2- 172.21.15.89:0/1386905044 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61580d18b0 0x7f615815ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:09.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.085+0000 7f6160400700 1 --2- 172.21.15.89:0/1386905044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6158160f10 0x7f615815e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:09.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.085+0000 7f6160400700 1 -- 172.21.15.89:0/1386905044 >> 172.21.15.89:0/1386905044 conn(0x7f615808aff0 msgr2=0x7f61580cb020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:09.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.085+0000 7f6160400700 1 -- 172.21.15.89:0/1386905044 shutdown_connections 2024-04-02T07:36:09.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.085+0000 7f6160400700 1 -- 172.21.15.89:0/1386905044 wait complete. 2024-04-02T07:36:09.105 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:36:09.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.296+0000 7f777c800700 1 Processor -- start 2024-04-02T07:36:09.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.296+0000 7f777c800700 1 -- start start 2024-04-02T07:36:09.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.297+0000 7f777c800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f77740d18b0 0x7f77740cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:09.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.297+0000 7f777c800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7774161590 0x7f7774161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:09.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.297+0000 7f777c800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77740ceda0 con 0x7f7774160f10 2024-04-02T07:36:09.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.297+0000 7f777c800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77740d1d30 con 0x7f77740d18b0 2024-04-02T07:36:09.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.297+0000 7f777c800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7774161f60 con 0x7f7774161590 2024-04-02T07:36:09.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.298+0000 7f7779400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f77740d18b0 0x7f77740cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:09.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.298+0000 7f7779400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f77740d18b0 0x7f77740cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47118/0 (socket says 172.21.15.89:47118) 2024-04-02T07:36:09.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.298+0000 7f7779e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7774160f10 0x7f77740d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56084/0 (socket says 172.21.15.89:56084) 2024-04-02T07:36:09.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.298+0000 7f7779400700 1 -- 172.21.15.89:0/4105016209 learned_addr learned my addr 172.21.15.89:0/4105016209 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:09.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.298+0000 7f777a800700 1 --2- 172.21.15.89:0/4105016209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7774161590 0x7f7774161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:09.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.298+0000 7f7779400700 1 -- 172.21.15.89:0/4105016209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7774161590 msgr2=0x7f7774161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:09.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.298+0000 7f7779400700 1 --2- 172.21.15.89:0/4105016209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7774161590 0x7f7774161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:09.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.298+0000 7f7779400700 1 -- 172.21.15.89:0/4105016209 >> v1:172.21.15.89:6789/0 conn(0x7f7774160f10 legacy=0x7f77740d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:09.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.298+0000 7f7779400700 1 -- 172.21.15.89:0/4105016209 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77741612f0 con 0x7f77740d18b0 2024-04-02T07:36:09.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.298+0000 7f7778a00700 1 -- 172.21.15.89:0/4105016209 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189473335 0 0) 0x7f77740ceda0 con 0x7f7774160f10 2024-04-02T07:36:09.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.298+0000 7f777a800700 1 --2- 172.21.15.89:0/4105016209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7774161590 0x7f7774161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:09.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.298+0000 7f7779400700 1 --2- 172.21.15.89:0/4105016209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77740d18b0 0x7f77740cf230 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7764009920 tx=0x7f7764009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ee58b4764d71b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:09.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.299+0000 7f7778a00700 1 -- 172.21.15.89:0/4105016209 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776400ba20 con 0x7f77740d18b0 2024-04-02T07:36:09.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.299+0000 7f7778a00700 1 -- 172.21.15.89:0/4105016209 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776400bb80 con 0x7f77740d18b0 2024-04-02T07:36:09.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.299+0000 7f7778a00700 1 -- 172.21.15.89:0/4105016209 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7764009e70 con 0x7f77740d18b0 2024-04-02T07:36:09.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.299+0000 7f777c800700 1 -- 172.21.15.89:0/4105016209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77740d18b0 msgr2=0x7f77740cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:09.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.299+0000 7f777c800700 1 --2- 172.21.15.89:0/4105016209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77740d18b0 0x7f77740cf230 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7764009920 tx=0x7f7764009c30 comp rx=0 tx=0).stop 2024-04-02T07:36:09.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.299+0000 7f777c800700 1 -- 172.21.15.89:0/4105016209 shutdown_connections 2024-04-02T07:36:09.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.299+0000 7f777c800700 1 --2- 172.21.15.89:0/4105016209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7774161590 0x7f7774161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:09.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.299+0000 7f777c800700 1 --2- 172.21.15.89:0/4105016209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77740d18b0 0x7f77740cf230 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:09.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.299+0000 7f777c800700 1 -- 172.21.15.89:0/4105016209 >> 172.21.15.89:0/4105016209 conn(0x7f777408aff0 msgr2=0x7f77740d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:09.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.299+0000 7f777c800700 1 -- 172.21.15.89:0/4105016209 shutdown_connections 2024-04-02T07:36:09.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.299+0000 7f777c800700 1 -- 172.21.15.89:0/4105016209 wait complete. 2024-04-02T07:36:09.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.302+0000 7f777c800700 1 Processor -- start 2024-04-02T07:36:09.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.302+0000 7f777c800700 1 -- start start 2024-04-02T07:36:09.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.303+0000 7f777c800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77740d18b0 0x7f77741e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:09.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.303+0000 7f777c800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7774160f10 0x7f77741483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:09.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.303+0000 7f777c800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7774161590 0x7f77741488f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:09.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.303+0000 7f777c800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7774148e30 con 0x7f77740d18b0 2024-04-02T07:36:09.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.303+0000 7f777c800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77741e8630 con 0x7f7774161590 2024-04-02T07:36:09.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.303+0000 7f777c800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77741e8910 con 0x7f7774160f10 2024-04-02T07:36:09.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.303+0000 7f7779e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77740d18b0 0x7f77741e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.303+0000 7f7779e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77740d18b0 0x7f77741e7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47134/0 (socket says 172.21.15.89:47134) 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.303+0000 7f7779e00700 1 -- 172.21.15.89:0/2421489045 learned_addr learned my addr 172.21.15.89:0/2421489045 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.303+0000 7f7779400700 1 --2- 172.21.15.89:0/2421489045 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7774160f10 0x7f77741483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.303+0000 7f777a800700 1 --2- 172.21.15.89:0/2421489045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7774161590 0x7f77741488f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.304+0000 7f7779e00700 1 -- 172.21.15.89:0/2421489045 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7774160f10 msgr2=0x7f77741483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.304+0000 7f7779e00700 1 --2- 172.21.15.89:0/2421489045 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7774160f10 0x7f77741483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.304+0000 7f7779e00700 1 -- 172.21.15.89:0/2421489045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7774161590 msgr2=0x7f77741488f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.304+0000 7f7779e00700 1 --2- 172.21.15.89:0/2421489045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7774161590 0x7f77741488f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.304+0000 7f7779e00700 1 -- 172.21.15.89:0/2421489045 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7764009550 con 0x7f77740d18b0 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.304+0000 7f7779e00700 1 --2- 172.21.15.89:0/2421489045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77740d18b0 0x7f77741e7e00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f776800c550 tx=0x7f776800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.304+0000 7f776e000700 1 -- 172.21.15.89:0/2421489045 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768004c80 con 0x7f77740d18b0 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.304+0000 7f776e000700 1 -- 172.21.15.89:0/2421489045 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7768004e00 con 0x7f77740d18b0 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.304+0000 7f776e000700 1 -- 172.21.15.89:0/2421489045 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768007310 con 0x7f77740d18b0 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.304+0000 7f777c800700 1 -- 172.21.15.89:0/2421489045 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77741e8bf0 con 0x7f77740d18b0 2024-04-02T07:36:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.304+0000 7f777c800700 1 -- 172.21.15.89:0/2421489045 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77741e91b0 con 0x7f77740d18b0 2024-04-02T07:36:09.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.305+0000 7f777c800700 1 -- 172.21.15.89:0/2421489045 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7774067e30 con 0x7f77740d18b0 2024-04-02T07:36:09.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.306+0000 7f776e000700 1 -- 172.21.15.89:0/2421489045 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7768007470 con 0x7f77740d18b0 2024-04-02T07:36:09.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.307+0000 7f776e000700 1 --2- 172.21.15.89:0/2421489045 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f775806ac90 0x7f775806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:09.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.307+0000 7f776e000700 1 -- 172.21.15.89:0/2421489045 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77680890d0 con 0x7f77740d18b0 2024-04-02T07:36:09.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.307+0000 7f7779400700 1 --2- 172.21.15.89:0/2421489045 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f775806ac90 0x7f775806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:09.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.308+0000 7f7779400700 1 --2- 172.21.15.89:0/2421489045 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f775806ac90 0x7f775806d150 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7764014040 tx=0x7f7764008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:09.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.311+0000 7f776e000700 1 -- 172.21.15.89:0/2421489045 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7768054180 con 0x7f77740d18b0 2024-04-02T07:36:09.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.556+0000 7f777c800700 1 -- 172.21.15.89:0/2421489045 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77740d4450 con 0x7f77740d18b0 2024-04-02T07:36:09.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.556+0000 7f776e000700 1 -- 172.21.15.89:0/2421489045 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4731 (secure 0 0 0) 0x7f77680577a0 con 0x7f77740d18b0 2024-04-02T07:36:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:09.559 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2427593953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2427593953},{"type":"v1","addr":"172.21.15.113:6835","nonce":2427593953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/926246816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":926246816},{"type":"v1","addr":"172.21.15.89:6836","nonce":926246816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2701685411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2701685411},{"type":"v1","addr":"172.21.15.89:6837","nonce":2701685411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:35:53.235047+0000","modified":"2024-04-02T07:35:56.245811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6834/3962490902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3962490902},{"type":"v1","addr":"172.21.15.113:6834","nonce":3962490902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-02T07:36:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.559+0000 7f777c800700 1 -- 172.21.15.89:0/2421489045 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f775806ac90 msgr2=0x7f775806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.559+0000 7f777c800700 1 --2- 172.21.15.89:0/2421489045 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f775806ac90 0x7f775806d150 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7764014040 tx=0x7f7764008030 comp rx=0 tx=0).stop 2024-04-02T07:36:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.559+0000 7f777c800700 1 -- 172.21.15.89:0/2421489045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77740d18b0 msgr2=0x7f77741e7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.559+0000 7f777c800700 1 --2- 172.21.15.89:0/2421489045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77740d18b0 0x7f77741e7e00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f776800c550 tx=0x7f776800c860 comp rx=0 tx=0).stop 2024-04-02T07:36:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.559+0000 7f777c800700 1 -- 172.21.15.89:0/2421489045 shutdown_connections 2024-04-02T07:36:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.559+0000 7f777c800700 1 --2- 172.21.15.89:0/2421489045 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f775806ac90 0x7f775806d150 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.559+0000 7f777c800700 1 --2- 172.21.15.89:0/2421489045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7774161590 0x7f77741488f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.559+0000 7f777c800700 1 --2- 172.21.15.89:0/2421489045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77740d18b0 0x7f77741e7e00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.559+0000 7f777c800700 1 --2- 172.21.15.89:0/2421489045 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7774160f10 0x7f77741483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:09.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.559+0000 7f777c800700 1 -- 172.21.15.89:0/2421489045 >> 172.21.15.89:0/2421489045 conn(0x7f777408aff0 msgr2=0x7f77740ca650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:09.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.559+0000 7f777c800700 1 -- 172.21.15.89:0/2421489045 shutdown_connections 2024-04-02T07:36:09.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:09.559+0000 7f777c800700 1 -- 172.21.15.89:0/2421489045 wait complete. 2024-04-02T07:36:09.563 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2024-04-02T07:36:09.982 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T07:35:53.235047+0000', 'modified': '2024-04-02T07:35:56.245811+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4297}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4297': {'gid': 4297, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.113:6834/3962490902', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6832', 'nonce': 3962490902}, {'type': 'v1', 'addr': '172.21.15.113:6834', 'nonce': 3962490902}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T07:36:09.982 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T07:36:09.982 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-02T07:36:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.171+0000 7fe684800700 1 Processor -- start 2024-04-02T07:36:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.171+0000 7fe684800700 1 -- start start 2024-04-02T07:36:10.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe684800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c15af30 0x7fe67c15b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe684800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c06eae0 con 0x7fe67c06da80 2024-04-02T07:36:10.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe684800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c06f660 con 0x7fe67c0b4a50 2024-04-02T07:36:10.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe684800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c06faf0 con 0x7fe67c15af30 2024-04-02T07:36:10.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe682800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe67c06da80 0x7fe67c0b2c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56086/0 (socket says 172.21.15.89:56086) 2024-04-02T07:36:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe682800700 1 -- 172.21.15.89:0/1548087132 learned_addr learned my addr 172.21.15.89:0/1548087132 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe681e00700 1 --2- 172.21.15.89:0/1548087132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c15af30 0x7fe67c15b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649376371 0 0) 0x7fe67c06eae0 con 0x7fe67c06da80 2024-04-02T07:36:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6640035b0 con 0x7fe67c06da80 2024-04-02T07:36:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316252859 0 0) 0x7fe67c06f660 con 0x7fe67c0b4a50 2024-04-02T07:36:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe67c06eae0 con 0x7fe67c0b4a50 2024-04-02T07:36:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3675231253 0 0) 0x7fe6640035b0 con 0x7fe67c06da80 2024-04-02T07:36:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c15af30 msgr2=0x7fe67c15b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe680a00700 1 --2- 172.21.15.89:0/1548087132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c15af30 0x7fe67c15b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.173+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 >> v1:172.21.15.113:6789/0 conn(0x7fe67c0b4a50 legacy=0x7fe67c06d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c06e150 con 0x7fe67c06da80 2024-04-02T07:36:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fe6780019f0 con 0x7fe67c06da80 2024-04-02T07:36:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 >> v1:172.21.15.89:6789/0 conn(0x7fe67c06da80 legacy=0x7fe67c0b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe680a00700 1 --2- 172.21.15.89:0/1548087132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c15af30 0x7fe664003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe680a00700 1 --2- 172.21.15.89:0/1548087132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe664004120 0x7fe664004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe680a00700 1 --2- 172.21.15.89:0/1548087132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe664004d90 0x7fe664005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c06faf0 con 0x7fe664004d90 2024-04-02T07:36:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66400bba0 con 0x7fe664004120 2024-04-02T07:36:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66400bd30 con 0x7fe67c15af30 2024-04-02T07:36:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe681e00700 1 --2- 172.21.15.89:0/1548087132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c15af30 0x7fe664003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe682800700 1 --2- 172.21.15.89:0/1548087132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe664004d90 0x7fe664005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe681400700 1 --2- 172.21.15.89:0/1548087132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe664004120 0x7fe664004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe681e00700 1 -- 172.21.15.89:0/1548087132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe664004120 msgr2=0x7fe664004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe681e00700 1 --2- 172.21.15.89:0/1548087132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe664004120 0x7fe664004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe681e00700 1 -- 172.21.15.89:0/1548087132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe664004d90 msgr2=0x7fe664005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe681e00700 1 --2- 172.21.15.89:0/1548087132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe664004d90 0x7fe664005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe681e00700 1 -- 172.21.15.89:0/1548087132 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6640035b0 con 0x7fe67c15af30 2024-04-02T07:36:10.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.174+0000 7fe682800700 1 --2- 172.21.15.89:0/1548087132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe664004d90 0x7fe664005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:10.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.175+0000 7fe681e00700 1 --2- 172.21.15.89:0/1548087132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c15af30 0x7fe664003bc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe670009b60 tx=0x7fe664005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1bf8c9ee657cd12 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:10.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.175+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67000e6c0 con 0x7fe67c15af30 2024-04-02T07:36:10.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.175+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67000e820 con 0x7fe67c15af30 2024-04-02T07:36:10.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.175+0000 7fe680a00700 1 -- 172.21.15.89:0/1548087132 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67000eb90 con 0x7fe67c15af30 2024-04-02T07:36:10.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.175+0000 7fe684800700 1 -- 172.21.15.89:0/1548087132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c15af30 msgr2=0x7fe664003bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:10.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.175+0000 7fe684800700 1 --2- 172.21.15.89:0/1548087132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c15af30 0x7fe664003bc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe670009b60 tx=0x7fe664005880 comp rx=0 tx=0).stop 2024-04-02T07:36:10.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.175+0000 7fe682800700 1 -- 172.21.15.89:0/1548087132 reap_dead start 2024-04-02T07:36:10.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.175+0000 7fe684800700 1 -- 172.21.15.89:0/1548087132 shutdown_connections 2024-04-02T07:36:10.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.175+0000 7fe684800700 1 -- 172.21.15.89:0/1548087132 >> 172.21.15.89:0/1548087132 conn(0x7fe67c071fb0 msgr2=0x7fe67c06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:10.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.176+0000 7fe684800700 1 -- 172.21.15.89:0/1548087132 shutdown_connections 2024-04-02T07:36:10.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.176+0000 7fe684800700 1 -- 172.21.15.89:0/1548087132 wait complete. 2024-04-02T07:36:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.177+0000 7fe684800700 1 Processor -- start 2024-04-02T07:36:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.177+0000 7fe684800700 1 -- start start 2024-04-02T07:36:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.178+0000 7fe684800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c15af30 0x7fe67c0c4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.178+0000 7fe684800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c14e580 0x7fe67c14b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.178+0000 7fe684800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c1f71d0 0x7fe67c0b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.178+0000 7fe684800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c0cddb0 con 0x7fe67c15af30 2024-04-02T07:36:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.178+0000 7fe684800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c0c50b0 con 0x7fe67c14e580 2024-04-02T07:36:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.178+0000 7fe684800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c14ba30 con 0x7fe67c1f71d0 2024-04-02T07:36:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.178+0000 7fe681e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c15af30 0x7fe67c0c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.179+0000 7fe681e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c15af30 0x7fe67c0c4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47152/0 (socket says 172.21.15.89:47152) 2024-04-02T07:36:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.179+0000 7fe681e00700 1 -- 172.21.15.89:0/1564503126 learned_addr learned my addr 172.21.15.89:0/1564503126 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.179+0000 7fe681400700 1 --2- 172.21.15.89:0/1564503126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c14e580 0x7fe67c14b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.179+0000 7fe682800700 1 --2- 172.21.15.89:0/1564503126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c1f71d0 0x7fe67c0b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.179+0000 7fe681e00700 1 -- 172.21.15.89:0/1564503126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c1f71d0 msgr2=0x7fe67c0b2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.179+0000 7fe681e00700 1 --2- 172.21.15.89:0/1564503126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c1f71d0 0x7fe67c0b2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.179+0000 7fe681e00700 1 -- 172.21.15.89:0/1564503126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c14e580 msgr2=0x7fe67c14b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.179+0000 7fe681e00700 1 --2- 172.21.15.89:0/1564503126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c14e580 0x7fe67c14b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.179+0000 7fe681e00700 1 -- 172.21.15.89:0/1564503126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c0cd7f0 con 0x7fe67c15af30 2024-04-02T07:36:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.179+0000 7fe681e00700 1 --2- 172.21.15.89:0/1564503126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c15af30 0x7fe67c0c4b70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe6700090b0 tx=0x7fe6700047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.179+0000 7fe676000700 1 -- 172.21.15.89:0/1564503126 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670021070 con 0x7fe67c15af30 2024-04-02T07:36:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.179+0000 7fe676000700 1 -- 172.21.15.89:0/1564503126 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe670004510 con 0x7fe67c15af30 2024-04-02T07:36:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.179+0000 7fe684800700 1 -- 172.21.15.89:0/1564503126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67c0cdad0 con 0x7fe67c15af30 2024-04-02T07:36:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.179+0000 7fe676000700 1 -- 172.21.15.89:0/1564503126 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67000e6c0 con 0x7fe67c15af30 2024-04-02T07:36:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.180+0000 7fe684800700 1 -- 172.21.15.89:0/1564503126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67c1f6cb0 con 0x7fe67c15af30 2024-04-02T07:36:10.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.181+0000 7fe676000700 1 -- 172.21.15.89:0/1564503126 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe670025070 con 0x7fe67c15af30 2024-04-02T07:36:10.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.182+0000 7fe676000700 1 --2- 172.21.15.89:0/1564503126 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe66006ac90 0x7fe66006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.182+0000 7fe676000700 1 -- 172.21.15.89:0/1564503126 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe67008a320 con 0x7fe67c15af30 2024-04-02T07:36:10.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.182+0000 7fe681400700 1 --2- 172.21.15.89:0/1564503126 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe66006ac90 0x7fe66006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.183+0000 7fe681400700 1 --2- 172.21.15.89:0/1564503126 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe66006ac90 0x7fe66006d150 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe66c000c00 tx=0x7fe66c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:10.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.184+0000 7fe6982eeb80 1 Processor -- start 2024-04-02T07:36:10.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.184+0000 7fe6982eeb80 1 -- start start 2024-04-02T07:36:10.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.185+0000 7fe6982eeb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c240714830 0x55c240714c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.185+0000 7fe6982eeb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e7be0 0x55c2406eaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.185+0000 7fe6982eeb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e5b60 0x55c240711a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.185+0000 7fe6982eeb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c24065f650 con 0x55c2406e7be0 2024-04-02T07:36:10.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.185+0000 7fe6982eeb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c240640c30 con 0x55c2406e5b60 2024-04-02T07:36:10.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.185+0000 7fe6982eeb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c24063e430 con 0x55c240714830 2024-04-02T07:36:10.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.185+0000 7fe681400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e7be0 0x55c2406eaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.185+0000 7fe681400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e7be0 0x55c2406eaa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47158/0 (socket says 172.21.15.89:47158) 2024-04-02T07:36:10.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.185+0000 7fe681400700 1 -- 172.21.15.89:0/4282163862 learned_addr learned my addr 172.21.15.89:0/4282163862 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:10.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.185+0000 7fe682800700 1 --2- 172.21.15.89:0/4282163862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e5b60 0x55c240711a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.185+0000 7fe681e00700 1 --2- 172.21.15.89:0/4282163862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c240714830 0x55c240714c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.186+0000 7fe681400700 1 -- 172.21.15.89:0/4282163862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c240714830 msgr2=0x55c240714c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:10.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.186+0000 7fe681400700 1 --2- 172.21.15.89:0/4282163862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c240714830 0x55c240714c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.186+0000 7fe681400700 1 -- 172.21.15.89:0/4282163862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e5b60 msgr2=0x55c240711a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:10.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.186+0000 7fe681400700 1 --2- 172.21.15.89:0/4282163862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e5b60 0x55c240711a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.186+0000 7fe681400700 1 -- 172.21.15.89:0/4282163862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c24042d790 con 0x55c2406e7be0 2024-04-02T07:36:10.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.186+0000 7fe681400700 1 --2- 172.21.15.89:0/4282163862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e7be0 0x55c2406eaa10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe66c001eb0 tx=0x7fe66c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49f5d6c42e353ec server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:10.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.186+0000 7fe66a600700 1 -- 172.21.15.89:0/4282163862 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c015750 con 0x55c2406e7be0 2024-04-02T07:36:10.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.186+0000 7fe66a600700 1 -- 172.21.15.89:0/4282163862 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66c0158b0 con 0x55c2406e7be0 2024-04-02T07:36:10.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.186+0000 7fe66a600700 1 -- 172.21.15.89:0/4282163862 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66c015c20 con 0x55c2406e7be0 2024-04-02T07:36:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.187+0000 7fe6982eeb80 1 -- 172.21.15.89:0/4282163862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e7be0 msgr2=0x55c2406eaa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.187+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/4282163862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e7be0 0x55c2406eaa10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe66c001eb0 tx=0x7fe66c015000 comp rx=0 tx=0).stop 2024-04-02T07:36:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.187+0000 7fe6982eeb80 1 -- 172.21.15.89:0/4282163862 shutdown_connections 2024-04-02T07:36:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.187+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/4282163862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e5b60 0x55c240711a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.187+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/4282163862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c240714830 0x55c240714c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.187+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/4282163862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e7be0 0x55c2406eaa10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.187+0000 7fe6982eeb80 1 -- 172.21.15.89:0/4282163862 >> 172.21.15.89:0/4282163862 conn(0x55c2406486a0 msgr2=0x55c24072f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.187+0000 7fe6982eeb80 1 -- 172.21.15.89:0/4282163862 shutdown_connections 2024-04-02T07:36:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.187+0000 7fe6982eeb80 1 -- 172.21.15.89:0/4282163862 wait complete. 2024-04-02T07:36:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.188+0000 7fe6982eeb80 1 -- start start 2024-04-02T07:36:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.188+0000 7fe6982eeb80 1 Processor -- start 2024-04-02T07:36:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe6982eeb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e5b60 0x55c2407314d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe6982eeb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e7be0 0x55c24072be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe6982eeb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c240714830 0x55c2407117f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe6982eeb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c240639d20 con 0x55c2406e5b60 2024-04-02T07:36:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe6982eeb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2405d5770 con 0x55c2406e7be0 2024-04-02T07:36:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe6982eeb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2405f86f0 con 0x55c240714830 2024-04-02T07:36:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe6982eeb80 10 client.0 _ll_register_callbacks cb 0x55c2405df5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:36:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe681e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e5b60 0x55c2407314d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe681e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e5b60 0x55c2407314d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47172/0 (socket says 172.21.15.89:47172) 2024-04-02T07:36:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe681e00700 1 -- 172.21.15.89:0/653603770 learned_addr learned my addr 172.21.15.89:0/653603770 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe681e00700 1 -- 172.21.15.89:0/653603770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c240714830 msgr2=0x55c2407117f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe681e00700 1 --2- 172.21.15.89:0/653603770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c240714830 0x55c2407117f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe681400700 1 --2- 172.21.15.89:0/653603770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e7be0 0x55c24072be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe681e00700 1 -- 172.21.15.89:0/653603770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e7be0 msgr2=0x55c24072be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.189+0000 7fe681e00700 1 --2- 172.21.15.89:0/653603770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e7be0 0x55c24072be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe681e00700 1 -- 172.21.15.89:0/653603770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67c06eae0 con 0x55c2406e5b60 2024-04-02T07:36:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe681400700 1 --2- 172.21.15.89:0/653603770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e7be0 0x55c24072be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe681e00700 1 --2- 172.21.15.89:0/653603770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e5b60 0x55c2407314d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe670088840 tx=0x7fe670009210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82c77fdfe457a1f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe65ea00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:36:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe65ea00700 1 -- 172.21.15.89:0/653603770 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670061660 con 0x55c2406e5b60 2024-04-02T07:36:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe65ea00700 1 -- 172.21.15.89:0/653603770 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe670009730 con 0x55c2406e5b60 2024-04-02T07:36:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe6982eeb80 1 -- 172.21.15.89:0/653603770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c2406dc920 con 0x55c2406e5b60 2024-04-02T07:36:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe65ea00700 1 -- 172.21.15.89:0/653603770 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe670058810 con 0x55c2406e5b60 2024-04-02T07:36:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe6982eeb80 10 client.4644 fetch_fsmap learned FSMap version 8 2024-04-02T07:36:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe6982eeb80 1 -- 172.21.15.89:0/653603770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55c24051a0a0 con 0x55c2406e5b60 2024-04-02T07:36:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe65ea00700 1 -- 172.21.15.89:0/653603770 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c2406dc920 con 0x55c2406e5b60 2024-04-02T07:36:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe65ea00700 1 -- 172.21.15.89:0/653603770 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 8) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe6700611d0 con 0x55c2406e5b60 2024-04-02T07:36:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe6982eeb80 10 client.4644 fetch_fsmap finished waiting for FSMap version 8 2024-04-02T07:36:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe6982eeb80 10 client.4644 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-02T07:36:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe6982eeb80 20 client.4644 populate_metadata read hostname 'smithi089' 2024-04-02T07:36:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/653603770 >> [v2:172.21.15.113:6832/3962490902,v1:172.21.15.113:6834/3962490902] conn(0x55c240732e10 0x55c2407331f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe6982eeb80 4 client.4644 mds_command: new command op to 4297 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T07:36:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.190+0000 7fe6982eeb80 1 -- 172.21.15.89:0/653603770 --> [v2:172.21.15.113:6832/3962490902,v1:172.21.15.113:6834/3962490902] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2406b96f0 con 0x55c240732e10 2024-04-02T07:36:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.191+0000 7fe681400700 1 --2- 172.21.15.89:0/653603770 >> [v2:172.21.15.113:6832/3962490902,v1:172.21.15.113:6834/3962490902] conn(0x55c240732e10 0x55c2407331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.191+0000 7fe681400700 1 --2- 172.21.15.89:0/653603770 >> [v2:172.21.15.113:6832/3962490902,v1:172.21.15.113:6834/3962490902] conn(0x55c240732e10 0x55c2407331f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=333359e413d85fb9 server_cookie=e0ea78567f5ec559 in_seq=0 out_seq=0 2024-04-02T07:36:10.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.191+0000 7fe65ea00700 10 client.4644 ms_handle_connect on v2:172.21.15.113:6832/3962490902 2024-04-02T07:36:10.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.192+0000 7fe65ea00700 1 -- 172.21.15.89:0/653603770 <== mds.0 v2:172.21.15.113:6832/3962490902 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c2406b96f0 con 0x55c240732e10 2024-04-02T07:36:10.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.192+0000 7fe65ea00700 10 client.4644 handle_command_reply: tid=0 2024-04-02T07:36:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.192+0000 7fe6982eeb80 1 client.4644 shutdown 2024-04-02T07:36:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.193+0000 7fe6982eeb80 1 -- 172.21.15.89:0/653603770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e5b60 msgr2=0x55c2407314d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.193+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/653603770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e5b60 0x55c2407314d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe670088840 tx=0x7fe670009210 comp rx=0 tx=0).stop 2024-04-02T07:36:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.193+0000 7fe6982eeb80 1 -- 172.21.15.89:0/653603770 shutdown_connections 2024-04-02T07:36:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.193+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/653603770 >> [v2:172.21.15.113:6832/3962490902,v1:172.21.15.113:6834/3962490902] conn(0x55c240732e10 0x55c2407331f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.193+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/653603770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c240714830 0x55c2407117f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.193+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/653603770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e5b60 0x55c2407314d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.193+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/653603770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e7be0 0x55c24072be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.193+0000 7fe6982eeb80 1 -- 172.21.15.89:0/653603770 >> 172.21.15.89:0/653603770 conn(0x55c2406486a0 msgr2=0x55c240710ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.193+0000 7fe65ea00700 0 client.4644 ms_handle_reset on v2:172.21.15.113:6832/3962490902 2024-04-02T07:36:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.193+0000 7fe6982eeb80 1 -- 172.21.15.89:0/653603770 shutdown_connections 2024-04-02T07:36:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.193+0000 7fe6982eeb80 1 -- 172.21.15.89:0/653603770 wait complete. 2024-04-02T07:36:10.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.193+0000 7fe6982eeb80 20 client.4644 trim_cache size 0 max 16384 2024-04-02T07:36:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.211+0000 7fe6982eeb80 1 Processor -- start 2024-04-02T07:36:10.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.211+0000 7fe6982eeb80 1 -- start start 2024-04-02T07:36:10.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe6982eeb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c2406e5b60 0x55c2407156b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe6982eeb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e7be0 0x55c2406eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe6982eeb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c240714830 0x55c240712970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe6982eeb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c24042e390 con 0x55c240714830 2024-04-02T07:36:10.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe6982eeb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2406eb4b0 con 0x55c2406e7be0 2024-04-02T07:36:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe6982eeb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c240715bf0 con 0x55c2406e5b60 2024-04-02T07:36:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe682800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c240714830 0x55c240712970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe682800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c240714830 0x55c240712970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47188/0 (socket says 172.21.15.89:47188) 2024-04-02T07:36:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe682800700 1 -- 172.21.15.89:0/3570589221 learned_addr learned my addr 172.21.15.89:0/3570589221 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe681400700 1 --2- 172.21.15.89:0/3570589221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e7be0 0x55c2406eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe681e00700 1 --2- 172.21.15.89:0/3570589221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c2406e5b60 0x55c2407156b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe682800700 1 -- 172.21.15.89:0/3570589221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c2406e5b60 msgr2=0x55c2407156b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe682800700 1 --2- 172.21.15.89:0/3570589221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c2406e5b60 0x55c2407156b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe682800700 1 -- 172.21.15.89:0/3570589221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e7be0 msgr2=0x55c2406eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe682800700 1 --2- 172.21.15.89:0/3570589221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e7be0 0x55c2406eb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.212+0000 7fe682800700 1 -- 172.21.15.89:0/3570589221 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c240751570 con 0x55c240714830 2024-04-02T07:36:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.213+0000 7fe682800700 1 --2- 172.21.15.89:0/3570589221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c240714830 0x55c240712970 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe67c0711a0 tx=0x7fe67800a870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d3ac248a471b2d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.213+0000 7fe66b000700 1 -- 172.21.15.89:0/3570589221 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe678020430 con 0x55c240714830 2024-04-02T07:36:10.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.213+0000 7fe66b000700 1 -- 172.21.15.89:0/3570589221 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe678020590 con 0x55c240714830 2024-04-02T07:36:10.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.213+0000 7fe66b000700 1 -- 172.21.15.89:0/3570589221 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe678020900 con 0x55c240714830 2024-04-02T07:36:10.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.213+0000 7fe6982eeb80 1 -- 172.21.15.89:0/3570589221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c240714830 msgr2=0x55c240712970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:10.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.213+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/3570589221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c240714830 0x55c240712970 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe67c0711a0 tx=0x7fe67800a870 comp rx=0 tx=0).stop 2024-04-02T07:36:10.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.213+0000 7fe6982eeb80 1 -- 172.21.15.89:0/3570589221 shutdown_connections 2024-04-02T07:36:10.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.213+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/3570589221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c240714830 0x55c240712970 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.213+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/3570589221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c2406e5b60 0x55c2407156b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.213+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/3570589221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2406e7be0 0x55c2406eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.213+0000 7fe6982eeb80 1 -- 172.21.15.89:0/3570589221 >> 172.21.15.89:0/3570589221 conn(0x55c2406486a0 msgr2=0x55c240732010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:10.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.213+0000 7fe6982eeb80 1 -- 172.21.15.89:0/3570589221 shutdown_connections 2024-04-02T07:36:10.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.213+0000 7fe6982eeb80 1 -- 172.21.15.89:0/3570589221 wait complete. 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.214+0000 7fe6982eeb80 1 -- start start 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.214+0000 7fe6982eeb80 1 Processor -- start 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.215+0000 7fe6982eeb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e5b60 0x55c2406e4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.215+0000 7fe6982eeb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c2406e7be0 0x55c2406e4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.215+0000 7fe6982eeb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c240714830 0x55c240740c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.215+0000 7fe6982eeb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2406e6750 con 0x55c2406e5b60 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.215+0000 7fe6982eeb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c24070f380 con 0x55c240714830 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.215+0000 7fe6982eeb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2407140c0 con 0x55c2406e7be0 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.215+0000 7fe6982eeb80 10 client.0 _ll_register_callbacks cb 0x55c2405df5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.215+0000 7fe681e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e5b60 0x55c2406e4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.215+0000 7fe681e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e5b60 0x55c2406e4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47198/0 (socket says 172.21.15.89:47198) 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.215+0000 7fe681e00700 1 -- 172.21.15.89:0/1010563821 learned_addr learned my addr 172.21.15.89:0/1010563821 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe682800700 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c240714830 0x55c240740c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe681400700 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c2406e7be0 0x55c2406e4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe681e00700 1 -- 172.21.15.89:0/1010563821 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c2406e7be0 msgr2=0x55c2406e4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:10.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe681e00700 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c2406e7be0 0x55c2406e4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe681e00700 1 -- 172.21.15.89:0/1010563821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c240714830 msgr2=0x55c240740c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:10.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe681e00700 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c240714830 0x55c240740c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe681e00700 1 -- 172.21.15.89:0/1010563821 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67c06e150 con 0x55c2406e5b60 2024-04-02T07:36:10.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe681400700 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c2406e7be0 0x55c2406e4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:10.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe682800700 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c240714830 0x55c240740c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:10.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe681e00700 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e5b60 0x55c2406e4240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe670055d40 tx=0x7fe670088710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf2a689495d4210 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:10.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe65ea00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:36:10.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe65ea00700 1 -- 172.21.15.89:0/1010563821 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6700590f0 con 0x55c2406e5b60 2024-04-02T07:36:10.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe65ea00700 1 -- 172.21.15.89:0/1010563821 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe670061660 con 0x55c2406e5b60 2024-04-02T07:36:10.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe6982eeb80 1 -- 172.21.15.89:0/1010563821 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe66c0158b0 con 0x55c2406e5b60 2024-04-02T07:36:10.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe65ea00700 1 -- 172.21.15.89:0/1010563821 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6700611d0 con 0x55c2406e5b60 2024-04-02T07:36:10.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe6982eeb80 10 client.4650 fetch_fsmap learned FSMap version 8 2024-04-02T07:36:10.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe6982eeb80 1 -- 172.21.15.89:0/1010563821 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55c240600010 con 0x55c2406e5b60 2024-04-02T07:36:10.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.216+0000 7fe65ea00700 1 -- 172.21.15.89:0/1010563821 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66c0158b0 con 0x55c2406e5b60 2024-04-02T07:36:10.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.217+0000 7fe65ea00700 1 -- 172.21.15.89:0/1010563821 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 8) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe670058810 con 0x55c2406e5b60 2024-04-02T07:36:10.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.217+0000 7fe6982eeb80 10 client.4650 fetch_fsmap finished waiting for FSMap version 8 2024-04-02T07:36:10.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.217+0000 7fe6982eeb80 10 client.4650 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-02T07:36:10.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.217+0000 7fe6982eeb80 20 client.4650 populate_metadata read hostname 'smithi089' 2024-04-02T07:36:10.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.217+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.113:6832/3962490902,v1:172.21.15.113:6834/3962490902] conn(0x55c24077ed90 0x55c2407411d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:10.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.217+0000 7fe6982eeb80 4 client.4650 mds_command: new command op to 4297 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T07:36:10.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.217+0000 7fe6982eeb80 1 -- 172.21.15.89:0/1010563821 --> [v2:172.21.15.113:6832/3962490902,v1:172.21.15.113:6834/3962490902] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c240473040 con 0x55c24077ed90 2024-04-02T07:36:10.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.217+0000 7fe681400700 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.113:6832/3962490902,v1:172.21.15.113:6834/3962490902] conn(0x55c24077ed90 0x55c2407411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:10.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe681400700 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.113:6832/3962490902,v1:172.21.15.113:6834/3962490902] conn(0x55c24077ed90 0x55c2407411d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70500b13ecd02b3b server_cookie=63546f2feb4f205c in_seq=0 out_seq=0 2024-04-02T07:36:10.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe65ea00700 10 client.4650 ms_handle_connect on v2:172.21.15.113:6832/3962490902 2024-04-02T07:36:10.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe65ea00700 1 -- 172.21.15.89:0/1010563821 <== mds.0 v2:172.21.15.113:6832/3962490902 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55c240473040 con 0x55c24077ed90 2024-04-02T07:36:10.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe65ea00700 10 client.4650 handle_command_reply: tid=0 2024-04-02T07:36:10.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe6982eeb80 1 client.4650 shutdown 2024-04-02T07:36:10.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe6982eeb80 1 -- 172.21.15.89:0/1010563821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e5b60 msgr2=0x55c2406e4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:10.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e5b60 0x55c2406e4240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe670055d40 tx=0x7fe670088710 comp rx=0 tx=0).stop 2024-04-02T07:36:10.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe6982eeb80 1 -- 172.21.15.89:0/1010563821 shutdown_connections 2024-04-02T07:36:10.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.113:6832/3962490902,v1:172.21.15.113:6834/3962490902] conn(0x55c24077ed90 0x55c2407411d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c240714830 0x55c240740c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2406e5b60 0x55c2406e4240 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe6982eeb80 1 --2- 172.21.15.89:0/1010563821 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c2406e7be0 0x55c2406e4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe65ea00700 0 client.4650 ms_handle_reset on v2:172.21.15.113:6832/3962490902 2024-04-02T07:36:10.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe6982eeb80 1 -- 172.21.15.89:0/1010563821 >> 172.21.15.89:0/1010563821 conn(0x55c2406486a0 msgr2=0x55c240731ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:10.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe6982eeb80 1 -- 172.21.15.89:0/1010563821 shutdown_connections 2024-04-02T07:36:10.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe6982eeb80 1 -- 172.21.15.89:0/1010563821 wait complete. 2024-04-02T07:36:10.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.218+0000 7fe6982eeb80 20 client.4650 trim_cache size 0 max 16384 2024-04-02T07:36:10.226 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:36:10.226 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T07:36:10.226 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T07:36:10.226 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 4297, 2024-04-02T07:36:10.226 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T07:36:10.226 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T07:36:10.226 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T07:36:10.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 16.979784809000002, 2024-04-02T07:36:10.226 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 8, 2024-04-02T07:36:10.226 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 20, 2024-04-02T07:36:10.226 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 20, 2024-04-02T07:36:10.227 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 21.894232042999999 2024-04-02T07:36:10.227 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:36:10.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.219+0000 7fe684800700 1 -- 172.21.15.89:0/1564503126 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe66006ac90 msgr2=0x7fe66006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:10.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.219+0000 7fe684800700 1 --2- 172.21.15.89:0/1564503126 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe66006ac90 0x7fe66006d150 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe66c000c00 tx=0x7fe66c009040 comp rx=0 tx=0).stop 2024-04-02T07:36:10.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.219+0000 7fe684800700 1 -- 172.21.15.89:0/1564503126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c15af30 msgr2=0x7fe67c0c4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:10.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.219+0000 7fe684800700 1 --2- 172.21.15.89:0/1564503126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c15af30 0x7fe67c0c4b70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe6700090b0 tx=0x7fe6700047c0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.219+0000 7fe684800700 1 -- 172.21.15.89:0/1564503126 shutdown_connections 2024-04-02T07:36:10.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.219+0000 7fe684800700 1 --2- 172.21.15.89:0/1564503126 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe66006ac90 0x7fe66006d150 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.219+0000 7fe684800700 1 --2- 172.21.15.89:0/1564503126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c1f71d0 0x7fe67c0b2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.219+0000 7fe684800700 1 --2- 172.21.15.89:0/1564503126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c15af30 0x7fe67c0c4b70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.219+0000 7fe684800700 1 --2- 172.21.15.89:0/1564503126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c14e580 0x7fe67c14b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:10.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.219+0000 7fe684800700 1 -- 172.21.15.89:0/1564503126 >> 172.21.15.89:0/1564503126 conn(0x7fe67c071fb0 msgr2=0x7fe67c0adac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:10.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.219+0000 7fe684800700 1 -- 172.21.15.89:0/1564503126 shutdown_connections 2024-04-02T07:36:10.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:10.219+0000 7fe684800700 1 -- 172.21.15.89:0/1564503126 wait complete. 2024-04-02T07:36:10.239 INFO:teuthology.run_tasks:Running task sequential... 2024-04-02T07:36:10.251 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-02T07:36:10.251 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-02T07:36:10.251 INFO:teuthology.task.print:Enabling mgr modules 2024-04-02T07:36:10.251 INFO:teuthology.run_tasks:Running task kclient... 2024-04-02T07:36:10.272 INFO:tasks.kclient:Mounting kernel clients... 2024-04-02T07:36:10.272 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-02T07:36:10.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:10.272 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T07:36:10.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:10.318 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2024-04-02T07:36:10.428 INFO:teuthology.orchestra.run.smithi089.stderr:Cannot find device "ceph-brx" 2024-04-02T07:36:10.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T07:36:10.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:10.431 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T07:36:10.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:10.476 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2024-04-02T07:36:10.583 INFO:teuthology.orchestra.run.smithi089.stderr:Cannot find device "ceph-brx" 2024-04-02T07:36:10.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T07:36:10.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:10.587 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-04-02T07:36:10.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:10.637 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link delete ceph-brx 2024-04-02T07:36:10.710 INFO:teuthology.orchestra.run.smithi113.stderr:Cannot find device "ceph-brx" 2024-04-02T07:36:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T07:36:10.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:10.714 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-04-02T07:36:10.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:10.759 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link delete ceph-brx 2024-04-02T07:36:10.827 INFO:teuthology.orchestra.run.smithi113.stderr:Cannot find device "ceph-brx" 2024-04-02T07:36:10.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T07:36:10.830 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-04-02T07:36:10.830 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-02T07:36:10.830 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T07:36:10.830 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T07:36:11.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.018+0000 7f0778e00700 1 Processor -- start 2024-04-02T07:36:11.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.018+0000 7f0778e00700 1 -- start start 2024-04-02T07:36:11.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.020+0000 7f0778e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0774160f10 0x7f07740d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:11.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.020+0000 7f0778e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07740cf230 0x7f07740d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:11.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.020+0000 7f0778e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07740ceda0 con 0x7f0774161590 2024-04-02T07:36:11.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.020+0000 7f0778e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07740d23d0 con 0x7f0774160f10 2024-04-02T07:36:11.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.020+0000 7f0778e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07741612f0 con 0x7f07740cf230 2024-04-02T07:36:11.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.020+0000 7f0772800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0774161590 0x7f07740b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56098/0 (socket says 172.21.15.89:56098) 2024-04-02T07:36:11.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.020+0000 7f0772800700 1 -- 172.21.15.89:0/2031405916 learned_addr learned my addr 172.21.15.89:0/2031405916 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:11.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.020+0000 7f0771e00700 1 --2- 172.21.15.89:0/2031405916 >> v2:172.21.15.113:3300/0 conn(0x7f0774160f10 0x7f07740d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:11.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.020+0000 7f0771400700 1 --2- 172.21.15.89:0/2031405916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07740cf230 0x7f07740d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.021+0000 7f0770a00700 1 -- 172.21.15.89:0/2031405916 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711721657 0 0) 0x7f07740ceda0 con 0x7f0774161590 2024-04-02T07:36:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.021+0000 7f0770a00700 1 -- 172.21.15.89:0/2031405916 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07580035b0 con 0x7f0774161590 2024-04-02T07:36:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.021+0000 7f0771e00700 1 -- 172.21.15.89:0/2031405916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07740cf230 msgr2=0x7f07740d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.021+0000 7f0771e00700 1 --2- 172.21.15.89:0/2031405916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07740cf230 0x7f07740d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.021+0000 7f0771e00700 1 -- 172.21.15.89:0/2031405916 >> v1:172.21.15.89:6789/0 conn(0x7f0774161590 legacy=0x7f07740b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.021+0000 7f0771e00700 1 -- 172.21.15.89:0/2031405916 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07740ce470 con 0x7f0774160f10 2024-04-02T07:36:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.021+0000 7f0771400700 1 --2- 172.21.15.89:0/2031405916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07740cf230 0x7f07740d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.021+0000 7f0771e00700 1 --2- 172.21.15.89:0/2031405916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0774160f10 0x7f07740d18b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0764009840 tx=0x7f0764009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad798cb7582752ad server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.021+0000 7f0770a00700 1 -- 172.21.15.89:0/2031405916 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764011830 con 0x7f0774160f10 2024-04-02T07:36:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.021+0000 7f0770a00700 1 -- 172.21.15.89:0/2031405916 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0764011990 con 0x7f0774160f10 2024-04-02T07:36:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.021+0000 7f0770a00700 1 -- 172.21.15.89:0/2031405916 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764011d20 con 0x7f0774160f10 2024-04-02T07:36:11.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.022+0000 7f0778e00700 1 -- 172.21.15.89:0/2031405916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0774160f10 msgr2=0x7f07740d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:11.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.022+0000 7f0778e00700 1 --2- 172.21.15.89:0/2031405916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0774160f10 0x7f07740d18b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0764009840 tx=0x7f0764009b50 comp rx=0 tx=0).stop 2024-04-02T07:36:11.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.022+0000 7f0778e00700 1 -- 172.21.15.89:0/2031405916 shutdown_connections 2024-04-02T07:36:11.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.022+0000 7f0778e00700 1 --2- 172.21.15.89:0/2031405916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0774160f10 0x7f07740d18b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:11.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.022+0000 7f0778e00700 1 --2- 172.21.15.89:0/2031405916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07740cf230 0x7f07740d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:11.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.022+0000 7f0778e00700 1 -- 172.21.15.89:0/2031405916 >> 172.21.15.89:0/2031405916 conn(0x7f077408aff0 msgr2=0x7f07740d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:11.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.022+0000 7f0778e00700 1 -- 172.21.15.89:0/2031405916 shutdown_connections 2024-04-02T07:36:11.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.022+0000 7f0778e00700 1 -- 172.21.15.89:0/2031405916 wait complete. 2024-04-02T07:36:11.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.024+0000 7f0778e00700 1 Processor -- start 2024-04-02T07:36:11.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.024+0000 7f0778e00700 1 -- start start 2024-04-02T07:36:11.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0778e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07740cf230 0x7f07740b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:11.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0778e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0774160f10 0x7f077409efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:11.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0778e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0774161590 0x7f07741f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:11.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0778e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077414c250 con 0x7f0774160f10 2024-04-02T07:36:11.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0778e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07741f3580 con 0x7f0774161590 2024-04-02T07:36:11.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0778e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f077409f5f0 con 0x7f07740cf230 2024-04-02T07:36:11.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0771e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07740cf230 0x7f07740b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:11.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0771400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0774160f10 0x7f077409efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:11.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0771e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07740cf230 0x7f07740b14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:48562/0 (socket says 172.21.15.89:48562) 2024-04-02T07:36:11.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0771e00700 1 -- 172.21.15.89:0/2388052033 learned_addr learned my addr 172.21.15.89:0/2388052033 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:11.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0772800700 1 --2- 172.21.15.89:0/2388052033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0774161590 0x7f07741f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0771e00700 1 -- 172.21.15.89:0/2388052033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0774161590 msgr2=0x7f07741f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0771e00700 1 --2- 172.21.15.89:0/2388052033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0774161590 0x7f07741f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0771e00700 1 -- 172.21.15.89:0/2388052033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0774160f10 msgr2=0x7f077409efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0771e00700 1 --2- 172.21.15.89:0/2388052033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0774160f10 0x7f077409efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0771e00700 1 -- 172.21.15.89:0/2388052033 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07600094a0 con 0x7f07740cf230 2024-04-02T07:36:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.026+0000 7f0771400700 1 --2- 172.21.15.89:0/2388052033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0774160f10 0x7f077409efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.027+0000 7f0771e00700 1 --2- 172.21.15.89:0/2388052033 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07740cf230 0x7f07740b14c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0764009fd0 tx=0x7f0764004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.027+0000 7f076a000700 1 -- 172.21.15.89:0/2388052033 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07640036a0 con 0x7f07740cf230 2024-04-02T07:36:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.027+0000 7f076a000700 1 -- 172.21.15.89:0/2388052033 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076401e070 con 0x7f07740cf230 2024-04-02T07:36:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.027+0000 7f0778e00700 1 -- 172.21.15.89:0/2388052033 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07640094a0 con 0x7f07740cf230 2024-04-02T07:36:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.027+0000 7f076a000700 1 -- 172.21.15.89:0/2388052033 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764003a60 con 0x7f07740cf230 2024-04-02T07:36:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.027+0000 7f0778e00700 1 -- 172.21.15.89:0/2388052033 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077409fbb0 con 0x7f07740cf230 2024-04-02T07:36:11.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.028+0000 7f0778e00700 1 -- 172.21.15.89:0/2388052033 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0774067720 con 0x7f07740cf230 2024-04-02T07:36:11.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.029+0000 7f076a000700 1 -- 172.21.15.89:0/2388052033 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0764003bc0 con 0x7f07740cf230 2024-04-02T07:36:11.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.030+0000 7f076a000700 1 --2- 172.21.15.89:0/2388052033 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f075406ac90 0x7f075406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:11.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.030+0000 7f076a000700 1 -- 172.21.15.89:0/2388052033 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f076408a1b0 con 0x7f07740cf230 2024-04-02T07:36:11.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.030+0000 7f0771400700 1 --2- 172.21.15.89:0/2388052033 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f075406ac90 0x7f075406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:11.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.031+0000 7f0771400700 1 --2- 172.21.15.89:0/2388052033 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f075406ac90 0x7f075406d150 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f07740b3eb0 tx=0x7f0760009210 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:11.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.035+0000 7f076a000700 1 -- 172.21.15.89:0/2388052033 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0764055380 con 0x7f07740cf230 2024-04-02T07:36:11.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.283+0000 7f0778e00700 1 -- 172.21.15.89:0/2388052033 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f07740c9ee0 con 0x7f07740cf230 2024-04-02T07:36:11.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.283+0000 7f076a000700 1 -- 172.21.15.89:0/2388052033 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f07640589a0 con 0x7f07740cf230 2024-04-02T07:36:11.286 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T07:36:11.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.286+0000 7f0778e00700 1 -- 172.21.15.89:0/2388052033 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f075406ac90 msgr2=0x7f075406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:11.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.286+0000 7f0778e00700 1 --2- 172.21.15.89:0/2388052033 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f075406ac90 0x7f075406d150 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f07740b3eb0 tx=0x7f0760009210 comp rx=0 tx=0).stop 2024-04-02T07:36:11.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.286+0000 7f0778e00700 1 -- 172.21.15.89:0/2388052033 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07740cf230 msgr2=0x7f07740b14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:11.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.286+0000 7f0778e00700 1 --2- 172.21.15.89:0/2388052033 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07740cf230 0x7f07740b14c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0764009fd0 tx=0x7f0764004750 comp rx=0 tx=0).stop 2024-04-02T07:36:11.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.286+0000 7f0778e00700 1 -- 172.21.15.89:0/2388052033 shutdown_connections 2024-04-02T07:36:11.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.286+0000 7f0778e00700 1 --2- 172.21.15.89:0/2388052033 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f075406ac90 0x7f075406d150 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:11.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.286+0000 7f0778e00700 1 --2- 172.21.15.89:0/2388052033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0774161590 0x7f07741f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:11.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.286+0000 7f0778e00700 1 --2- 172.21.15.89:0/2388052033 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07740cf230 0x7f07740b14c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:11.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.286+0000 7f0778e00700 1 --2- 172.21.15.89:0/2388052033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0774160f10 0x7f077409efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:11.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.286+0000 7f0778e00700 1 -- 172.21.15.89:0/2388052033 >> 172.21.15.89:0/2388052033 conn(0x7f077408aff0 msgr2=0x7f077415b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:11.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.286+0000 7f0778e00700 1 -- 172.21.15.89:0/2388052033 shutdown_connections 2024-04-02T07:36:11.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:11.286+0000 7f0778e00700 1 -- 172.21.15.89:0/2388052033 wait complete. 2024-04-02T07:36:11.307 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T07:36:11.307 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T07:36:11.307 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T07:36:11.308 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T07:36:11.308 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T07:36:11.308 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T07:36:11.308 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T07:36:11.308 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T07:36:11.308 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:36:11.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:11.308 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T07:36:11.346 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:36:11.353 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:36:11.353 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:36:11.353 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:36:11.353 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:11.353 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T07:36:11.354 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:11.354 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:36:11.354 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:11.354 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T07:36:11.354 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:36:11.354 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:11.354 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T07:36:11.354 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:36:11.354 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:11.354 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T07:36:11.354 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:11.354 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T07:36:11.354 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42457sec preferred_lft 42457sec 2024-04-02T07:36:11.355 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T07:36:11.355 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:11.355 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-02T07:36:11.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:11.355 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:36:11.355 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:36:11.355 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add name ceph-brx type bridge 2024-04-02T07:36:11.356 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr flush dev ceph-brx 2024-04-02T07:36:11.356 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set ceph-brx up 2024-04-02T07:36:11.356 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-02T07:36:11.356 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:36:11.356 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:36:11.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:11.793 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T07:36:11.902 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T07:36:11.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:11.905 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T07:36:11.991 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T07:36:11.991 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T07:36:11.991 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T07:36:11.991 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T07:36:11.991 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T07:36:11.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:11.992 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:36:11.992 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:36:11.992 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T07:36:11.993 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T07:36:11.993 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T07:36:11.993 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:36:12.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:12.746 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T07:36:12.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:12.791 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list-id 2024-04-02T07:36:12.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:12.877 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:36:12.877 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:36:12.877 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T07:36:12.877 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-02T07:36:12.877 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:36:13.052 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-02T07:36:13.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:13.053 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:36:13.053 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:36:13.053 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-02T07:36:13.053 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-02T07:36:13.053 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-02T07:36:13.053 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-02T07:36:13.053 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-02T07:36:13.053 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:36:13.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:13.317 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:36:13.317 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:36:13.317 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T07:36:13.317 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set dev brx.0 master ceph-brx 2024-04-02T07:36:13.317 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:36:13.482 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:36:13.482 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T07:36:13.526 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:36:13.527 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:36:13.527 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:36:13.610 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T07:36:13.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:13.610 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:36:13.750 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:36:13.750 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:36:14.375 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:36:14.375 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:36:14.375 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T07:36:14.375 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T07:36:14.375 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T07:36:14.380 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T07:36:14.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:14.380 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:36:14.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:14.447 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:36:14.559 INFO:teuthology.orchestra.run.smithi089.stdout:client4659 2024-04-02T07:36:14.559 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:36:14.560 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:14.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:14.604 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:36:14.750 INFO:teuthology.orchestra.run.smithi089.stdout:client4659 2024-04-02T07:36:14.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:14.750 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:36:14.858 INFO:teuthology.orchestra.run.smithi089.stdout:client4659 2024-04-02T07:36:14.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:14.858 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ddf7474f-4630-4a1e-ada2-3982714d822b.client4659/status') 2024-04-02T07:36:14.980 INFO:teuthology.orchestra.run.smithi089.stdout:instance: client.4659 (3)192.168.0.1:0/345709953 2024-04-02T07:36:14.980 INFO:teuthology.orchestra.run.smithi089.stdout:blocklisted: false 2024-04-02T07:36:14.980 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records in 2024-04-02T07:36:14.980 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records out 2024-04-02T07:36:14.980 INFO:teuthology.orchestra.run.smithi089.stderr:68 bytes copied, 5.8841e-05 s, 1.2 MB/s 2024-04-02T07:36:14.983 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-04-02T07:36:14.983 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-02T07:36:14.983 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-02T07:36:14.983 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T07:36:15.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.170+0000 7f1d1bc00700 1 Processor -- start 2024-04-02T07:36:15.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.170+0000 7f1d1bc00700 1 -- start start 2024-04-02T07:36:15.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.171+0000 7f1d1bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d140ce590 0x7f1d140ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:15.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.171+0000 7f1d1bc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d140ceef0 con 0x7f1d140b6a20 2024-04-02T07:36:15.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.171+0000 7f1d1bc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d14160fc0 con 0x7f1d14160be0 2024-04-02T07:36:15.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.172+0000 7f1d1bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d140b65d0 con 0x7f1d140ce590 2024-04-02T07:36:15.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.172+0000 7f1d19c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1d140b6a20 0x7f1d140b6eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56112/0 (socket says 172.21.15.89:56112) 2024-04-02T07:36:15.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.172+0000 7f1d19c00700 1 -- 172.21.15.89:0/3931445366 learned_addr learned my addr 172.21.15.89:0/3931445366 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:15.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.172+0000 7f1d13e00700 1 --2- 172.21.15.89:0/3931445366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d140ce590 0x7f1d140ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:15.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.172+0000 7f1d13400700 1 -- 172.21.15.89:0/3931445366 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109948539 0 0) 0x7f1d140ceef0 con 0x7f1d140b6a20 2024-04-02T07:36:15.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.172+0000 7f1d13400700 1 -- 172.21.15.89:0/3931445366 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cfc0035b0 con 0x7f1d140b6a20 2024-04-02T07:36:15.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.172+0000 7f1d13400700 1 -- 172.21.15.89:0/3931445366 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1366424550 0 0) 0x7f1cfc0035b0 con 0x7f1d140b6a20 2024-04-02T07:36:15.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.172+0000 7f1d13400700 1 -- 172.21.15.89:0/3931445366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d140ce590 msgr2=0x7f1d140ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:15.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.172+0000 7f1d13400700 1 --2- 172.21.15.89:0/3931445366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d140ce590 0x7f1d140ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:15.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.172+0000 7f1d13400700 1 -- 172.21.15.89:0/3931445366 >> v1:172.21.15.113:6789/0 conn(0x7f1d14160be0 legacy=0x7f1d140cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:15.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.172+0000 7f1d13400700 1 -- 172.21.15.89:0/3931445366 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d140d21e0 con 0x7f1d140b6a20 2024-04-02T07:36:15.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.172+0000 7f1d13400700 1 -- 172.21.15.89:0/3931445366 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f1d0c0019f0 con 0x7f1d140b6a20 2024-04-02T07:36:15.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d13400700 1 -- 172.21.15.89:0/3931445366 >> v1:172.21.15.89:6789/0 conn(0x7f1d140b6a20 legacy=0x7f1d140b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:15.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d13400700 1 --2- 172.21.15.89:0/3931445366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d140ce590 0x7f1cfc003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:15.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d13400700 1 --2- 172.21.15.89:0/3931445366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cfc003f70 0x7f1cfc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:15.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d13400700 1 --2- 172.21.15.89:0/3931445366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cfc004d90 0x7f1cfc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:15.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d13400700 1 -- 172.21.15.89:0/3931445366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d140b65d0 con 0x7f1cfc004d90 2024-04-02T07:36:15.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d13400700 1 -- 172.21.15.89:0/3931445366 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc00ba70 con 0x7f1cfc003f70 2024-04-02T07:36:15.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d13400700 1 -- 172.21.15.89:0/3931445366 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc00bc30 con 0x7f1d140ce590 2024-04-02T07:36:15.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d19c00700 1 --2- 172.21.15.89:0/3931445366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cfc004d90 0x7f1cfc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:15.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d19200700 1 --2- 172.21.15.89:0/3931445366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d140ce590 0x7f1cfc003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:15.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d13e00700 1 --2- 172.21.15.89:0/3931445366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cfc003f70 0x7f1cfc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:15.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d19c00700 1 -- 172.21.15.89:0/3931445366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d140ce590 msgr2=0x7f1cfc003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:15.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d19c00700 1 --2- 172.21.15.89:0/3931445366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d140ce590 0x7f1cfc003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:15.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d19c00700 1 -- 172.21.15.89:0/3931445366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cfc003f70 msgr2=0x7f1cfc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:15.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d19c00700 1 --2- 172.21.15.89:0/3931445366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cfc003f70 0x7f1cfc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:15.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d19c00700 1 -- 172.21.15.89:0/3931445366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cfc0035b0 con 0x7f1cfc004d90 2024-04-02T07:36:15.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.173+0000 7f1d13e00700 1 --2- 172.21.15.89:0/3931445366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cfc003f70 0x7f1cfc004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:15.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.174+0000 7f1d19c00700 1 --2- 172.21.15.89:0/3931445366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cfc004d90 0x7f1cfc005210 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00a830 tx=0x7f1cfc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec687673cafeb4c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:15.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.174+0000 7f1d13400700 1 -- 172.21.15.89:0/3931445366 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c012600 con 0x7f1cfc004d90 2024-04-02T07:36:15.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.174+0000 7f1d13400700 1 -- 172.21.15.89:0/3931445366 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0c0127a0 con 0x7f1cfc004d90 2024-04-02T07:36:15.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.174+0000 7f1d13400700 1 -- 172.21.15.89:0/3931445366 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c012b50 con 0x7f1cfc004d90 2024-04-02T07:36:15.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.174+0000 7f1d1bc00700 1 -- 172.21.15.89:0/3931445366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cfc004d90 msgr2=0x7f1cfc005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:15.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.174+0000 7f1d1bc00700 1 --2- 172.21.15.89:0/3931445366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cfc004d90 0x7f1cfc005210 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00a830 tx=0x7f1cfc005880 comp rx=0 tx=0).stop 2024-04-02T07:36:15.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.174+0000 7f1d19c00700 1 -- 172.21.15.89:0/3931445366 reap_dead start 2024-04-02T07:36:15.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.174+0000 7f1d1bc00700 1 -- 172.21.15.89:0/3931445366 shutdown_connections 2024-04-02T07:36:15.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.174+0000 7f1d1bc00700 1 -- 172.21.15.89:0/3931445366 >> 172.21.15.89:0/3931445366 conn(0x7f1d1408aff0 msgr2=0x7f1d140d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:15.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.174+0000 7f1d1bc00700 1 -- 172.21.15.89:0/3931445366 shutdown_connections 2024-04-02T07:36:15.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.174+0000 7f1d1bc00700 1 -- 172.21.15.89:0/3931445366 wait complete. 2024-04-02T07:36:15.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.176+0000 7f1d1bc00700 1 Processor -- start 2024-04-02T07:36:15.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.176+0000 7f1d1bc00700 1 -- start start 2024-04-02T07:36:15.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.177+0000 7f1d1bc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cfc004d90 0x7f1d1415ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:15.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.177+0000 7f1d1bc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1406caf0 0x7f1d1406cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:15.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.177+0000 7f1d1bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d140cce70 0x7f1d140cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:15.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.177+0000 7f1d1bc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1406d4b0 con 0x7f1cfc004d90 2024-04-02T07:36:15.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.177+0000 7f1d1bc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d141f68a0 con 0x7f1d1406caf0 2024-04-02T07:36:15.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.177+0000 7f1d1bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d141f6b60 con 0x7f1d140cce70 2024-04-02T07:36:15.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.177+0000 7f1d19200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cfc004d90 0x7f1d1415ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:15.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.177+0000 7f1d19200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cfc004d90 0x7f1d1415ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47228/0 (socket says 172.21.15.89:47228) 2024-04-02T07:36:15.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.177+0000 7f1d19200700 1 -- 172.21.15.89:0/2436364224 learned_addr learned my addr 172.21.15.89:0/2436364224 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:15.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.178+0000 7f1d19c00700 1 --2- 172.21.15.89:0/2436364224 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d140cce70 0x7f1d140cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:15.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.178+0000 7f1d13e00700 1 --2- 172.21.15.89:0/2436364224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1406caf0 0x7f1d1406cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:15.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.178+0000 7f1d19200700 1 -- 172.21.15.89:0/2436364224 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d140cce70 msgr2=0x7f1d140cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:15.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.178+0000 7f1d19200700 1 --2- 172.21.15.89:0/2436364224 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d140cce70 0x7f1d140cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:15.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.178+0000 7f1d19200700 1 -- 172.21.15.89:0/2436364224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1406caf0 msgr2=0x7f1d1406cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:15.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.178+0000 7f1d19200700 1 --2- 172.21.15.89:0/2436364224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1406caf0 0x7f1d1406cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:15.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.178+0000 7f1d19200700 1 -- 172.21.15.89:0/2436364224 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d141f6e40 con 0x7f1cfc004d90 2024-04-02T07:36:15.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.178+0000 7f1d19200700 1 --2- 172.21.15.89:0/2436364224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cfc004d90 0x7f1d1415ba50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800f860 tx=0x7f1d0800fb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:15.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.178+0000 7f1d10c00700 1 -- 172.21.15.89:0/2436364224 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d080052e0 con 0x7f1cfc004d90 2024-04-02T07:36:15.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.178+0000 7f1d10c00700 1 -- 172.21.15.89:0/2436364224 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d08005460 con 0x7f1cfc004d90 2024-04-02T07:36:15.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.178+0000 7f1d1bc00700 1 -- 172.21.15.89:0/2436364224 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d141f7120 con 0x7f1cfc004d90 2024-04-02T07:36:15.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.178+0000 7f1d10c00700 1 -- 172.21.15.89:0/2436364224 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d080038c0 con 0x7f1cfc004d90 2024-04-02T07:36:15.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.179+0000 7f1d1bc00700 1 -- 172.21.15.89:0/2436364224 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1415bf90 con 0x7f1cfc004d90 2024-04-02T07:36:15.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.180+0000 7f1d1bc00700 1 -- 172.21.15.89:0/2436364224 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d14067720 con 0x7f1cfc004d90 2024-04-02T07:36:15.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.180+0000 7f1d10c00700 1 -- 172.21.15.89:0/2436364224 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1d08003a20 con 0x7f1cfc004d90 2024-04-02T07:36:15.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.182+0000 7f1d10c00700 1 --2- 172.21.15.89:0/2436364224 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1cf406ac90 0x7f1cf406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:15.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.182+0000 7f1d10c00700 1 -- 172.21.15.89:0/2436364224 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d080891e0 con 0x7f1cfc004d90 2024-04-02T07:36:15.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.182+0000 7f1d13e00700 1 --2- 172.21.15.89:0/2436364224 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1cf406ac90 0x7f1cf406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:15.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.182+0000 7f1d13e00700 1 --2- 172.21.15.89:0/2436364224 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1cf406ac90 0x7f1cf406d150 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1d140c9770 tx=0x7f1d00008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:15.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.187+0000 7f1d10c00700 1 -- 172.21.15.89:0/2436364224 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1d08054330 con 0x7f1cfc004d90 2024-04-02T07:36:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.433+0000 7f1d1bc00700 1 -- 172.21.15.89:0/2436364224 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1d140b5cb0 con 0x7f1cfc004d90 2024-04-02T07:36:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.433+0000 7f1d10c00700 1 -- 172.21.15.89:0/2436364224 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1d08057950 con 0x7f1cfc004d90 2024-04-02T07:36:15.436 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T07:36:15.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.435+0000 7f1d1bc00700 1 -- 172.21.15.89:0/2436364224 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1cf406ac90 msgr2=0x7f1cf406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:15.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.435+0000 7f1d1bc00700 1 --2- 172.21.15.89:0/2436364224 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1cf406ac90 0x7f1cf406d150 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1d140c9770 tx=0x7f1d00008040 comp rx=0 tx=0).stop 2024-04-02T07:36:15.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.435+0000 7f1d1bc00700 1 -- 172.21.15.89:0/2436364224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cfc004d90 msgr2=0x7f1d1415ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:15.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.435+0000 7f1d1bc00700 1 --2- 172.21.15.89:0/2436364224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cfc004d90 0x7f1d1415ba50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800f860 tx=0x7f1d0800fb70 comp rx=0 tx=0).stop 2024-04-02T07:36:15.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.436+0000 7f1d1bc00700 1 -- 172.21.15.89:0/2436364224 shutdown_connections 2024-04-02T07:36:15.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.436+0000 7f1d1bc00700 1 --2- 172.21.15.89:0/2436364224 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1cf406ac90 0x7f1cf406d150 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:15.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.436+0000 7f1d1bc00700 1 --2- 172.21.15.89:0/2436364224 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d140cce70 0x7f1d140cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:15.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.436+0000 7f1d1bc00700 1 --2- 172.21.15.89:0/2436364224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cfc004d90 0x7f1d1415ba50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:15.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.436+0000 7f1d1bc00700 1 --2- 172.21.15.89:0/2436364224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1406caf0 0x7f1d1406cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:15.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.436+0000 7f1d1bc00700 1 -- 172.21.15.89:0/2436364224 >> 172.21.15.89:0/2436364224 conn(0x7f1d1408aff0 msgr2=0x7f1d140b7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:15.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.436+0000 7f1d1bc00700 1 -- 172.21.15.89:0/2436364224 shutdown_connections 2024-04-02T07:36:15.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:15.436+0000 7f1d1bc00700 1 -- 172.21.15.89:0/2436364224 wait complete. 2024-04-02T07:36:15.456 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T07:36:15.456 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T07:36:15.456 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T07:36:15.456 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-02T07:36:15.457 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-02T07:36:15.457 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T07:36:15.457 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T07:36:15.457 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T07:36:15.457 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-02T07:36:15.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:15.457 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T07:36:15.503 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:36:15.503 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:36:15.503 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:36:15.504 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:15.504 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T07:36:15.504 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:15.504 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:36:15.504 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:15.504 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T07:36:15.504 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:36:15.504 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:15.504 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T07:36:15.504 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:36:15.504 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:15.505 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T07:36:15.505 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:15.505 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T07:36:15.505 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42453sec preferred_lft 42453sec 2024-04-02T07:36:15.505 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T07:36:15.505 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:15.505 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T07:36:15.505 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:15.505 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T07:36:15.505 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:15.505 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T07:36:15.505 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:15.505 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:36:15.506 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T07:36:15.506 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T07:36:15.506 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:15.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:15.506 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T07:36:15.607 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T07:36:15.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:15.610 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T07:36:15.658 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T07:36:15.658 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T07:36:15.659 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T07:36:15.659 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T07:36:15.659 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T07:36:15.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:15.659 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:36:15.659 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:36:15.659 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T07:36:15.659 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T07:36:15.660 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T07:36:15.660 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:36:15.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:15.822 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T07:36:15.866 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T07:36:15.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:15.867 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list-id 2024-04-02T07:36:15.952 INFO:teuthology.orchestra.run.smithi089.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-02T07:36:15.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:15.953 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:36:15.953 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:36:15.953 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T07:36:15.953 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-02T07:36:15.953 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:36:16.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:16.128 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-02T07:36:16.194 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:36:16.195 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:36:16.195 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:36:16.195 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:16.195 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T07:36:16.195 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:16.195 INFO:teuthology.orchestra.run.smithi089.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T07:36:16.195 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 8e:b4:0b:48:c2:24 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-02T07:36:16.195 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-02T07:36:16.195 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:16.195 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::8cb4:bff:fe48:c224/64 scope link 2024-04-02T07:36:16.195 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:16.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:16.197 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-02T07:36:16.265 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:36:16.265 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:36:16.266 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:36:16.266 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:16.266 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T07:36:16.266 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:16.266 INFO:teuthology.orchestra.run.smithi089.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T07:36:16.266 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 8e:b4:0b:48:c2:24 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-02T07:36:16.266 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-02T07:36:16.266 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:16.266 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::8cb4:bff:fe48:c224/64 scope link 2024-04-02T07:36:16.266 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:16.268 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-02T07:36:16.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:16.269 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:36:16.269 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:36:16.269 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-02T07:36:16.269 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-02T07:36:16.269 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-02T07:36:16.269 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-02T07:36:16.269 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-02T07:36:16.269 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:36:16.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:16.501 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:36:16.501 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:36:16.501 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.1 up 2024-04-02T07:36:16.501 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set dev brx.1 master ceph-brx 2024-04-02T07:36:16.501 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:36:16.665 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:36:16.665 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-02T07:36:16.709 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-02T07:36:16.710 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:36:16.710 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-02T07:36:16.793 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-04-02T07:36:16.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:16.793 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:36:16.933 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:36:16.934 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:36:16.962 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:36:16.962 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:36:16.962 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T07:36:16.962 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 2@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T07:36:16.962 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T07:36:16.967 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T07:36:16.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:16.967 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-02T07:36:17.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:17.033 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:36:17.116 INFO:teuthology.orchestra.run.smithi089.stdout:client4526 2024-04-02T07:36:17.117 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:36:17.117 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:17.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:17.161 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:36:17.289 INFO:teuthology.orchestra.run.smithi089.stdout:client4526 2024-04-02T07:36:17.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:17.289 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:36:17.374 INFO:teuthology.orchestra.run.smithi089.stdout:client4526 2024-04-02T07:36:17.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:17.375 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ddf7474f-4630-4a1e-ada2-3982714d822b.client4526/status') 2024-04-02T07:36:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:instance: client.4526 (3)172.21.15.89:0/321417395 2024-04-02T07:36:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:blocklisted: false 2024-04-02T07:36:17.477 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records in 2024-04-02T07:36:17.477 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records out 2024-04-02T07:36:17.477 INFO:teuthology.orchestra.run.smithi089.stderr:69 bytes copied, 5.4027e-05 s, 1.3 MB/s 2024-04-02T07:36:17.479 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-04-02T07:36:17.480 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-02T07:36:17.480 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-02T07:36:17.480 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T07:36:17.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.667+0000 7fabbf600700 1 Processor -- start 2024-04-02T07:36:17.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.667+0000 7fabbf600700 1 -- start start 2024-04-02T07:36:17.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.668+0000 7fabbf600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabb8160f10 0x7fabb80d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:17.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.668+0000 7fabbf600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fabb80cf230 0x7fabb80d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:17.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.668+0000 7fabbf600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fabb8161590 0x7fabb8161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:17.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.668+0000 7fabbf600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb8161f60 con 0x7fabb8161590 2024-04-02T07:36:17.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.668+0000 7fabbf600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb80ceda0 con 0x7fabb80cf230 2024-04-02T07:36:17.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.668+0000 7fabbf600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb80d23d0 con 0x7fabb8160f10 2024-04-02T07:36:17.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.668+0000 7fabbd600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fabb8161590 0x7fabb8161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:17.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.668+0000 7fabbd600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fabb8161590 0x7fabb8161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47238/0 (socket says 172.21.15.89:47238) 2024-04-02T07:36:17.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.668+0000 7fabbd600700 1 -- 172.21.15.89:0/3811817500 learned_addr learned my addr 172.21.15.89:0/3811817500 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:17.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.669+0000 7fabbcc00700 1 --2- 172.21.15.89:0/3811817500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabb8160f10 0x7fabb80d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:17.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.669+0000 7fabb7e00700 1 --2- 172.21.15.89:0/3811817500 >> v2:172.21.15.113:3300/0 conn(0x7fabb80cf230 0x7fabb80d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:17.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.669+0000 7fabbd600700 1 -- 172.21.15.89:0/3811817500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabb8160f10 msgr2=0x7fabb80d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:17.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.669+0000 7fabbd600700 1 --2- 172.21.15.89:0/3811817500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabb8160f10 0x7fabb80d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:17.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.669+0000 7fabbd600700 1 -- 172.21.15.89:0/3811817500 >> v2:172.21.15.113:3300/0 conn(0x7fabb80cf230 msgr2=0x7fabb80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:17.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.669+0000 7fabbd600700 1 --2- 172.21.15.89:0/3811817500 >> v2:172.21.15.113:3300/0 conn(0x7fabb80cf230 0x7fabb80d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.669+0000 7fabbd600700 1 -- 172.21.15.89:0/3811817500 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb81612f0 con 0x7fabb8161590 2024-04-02T07:36:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.669+0000 7fabb7e00700 1 --2- 172.21.15.89:0/3811817500 >> v2:172.21.15.113:3300/0 conn(0x7fabb80cf230 0x7fabb80d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.669+0000 7fabbd600700 1 --2- 172.21.15.89:0/3811817500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb8161590 0x7fabb8161a20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fabb808c1d0 tx=0x7fabb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58968a69a459a21c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.669+0000 7fabb7400700 1 -- 172.21.15.89:0/3811817500 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb000e070 con 0x7fabb8161590 2024-04-02T07:36:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.669+0000 7fabb7400700 1 -- 172.21.15.89:0/3811817500 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb000b9c0 con 0x7fabb8161590 2024-04-02T07:36:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.669+0000 7fabb7400700 1 -- 172.21.15.89:0/3811817500 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb000bd70 con 0x7fabb8161590 2024-04-02T07:36:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.670+0000 7fabbf600700 1 -- 172.21.15.89:0/3811817500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb8161590 msgr2=0x7fabb8161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.670+0000 7fabbf600700 1 --2- 172.21.15.89:0/3811817500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb8161590 0x7fabb8161a20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fabb808c1d0 tx=0x7fabb000b330 comp rx=0 tx=0).stop 2024-04-02T07:36:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.670+0000 7fabbf600700 1 -- 172.21.15.89:0/3811817500 shutdown_connections 2024-04-02T07:36:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.670+0000 7fabbf600700 1 --2- 172.21.15.89:0/3811817500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb8161590 0x7fabb8161a20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.670+0000 7fabbf600700 1 --2- 172.21.15.89:0/3811817500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabb8160f10 0x7fabb80d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.670+0000 7fabbf600700 1 --2- 172.21.15.89:0/3811817500 >> v2:172.21.15.113:3300/0 conn(0x7fabb80cf230 0x7fabb80d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:17.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.670+0000 7fabbf600700 1 -- 172.21.15.89:0/3811817500 >> 172.21.15.89:0/3811817500 conn(0x7fabb808aff0 msgr2=0x7fabb80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:17.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.670+0000 7fabbf600700 1 -- 172.21.15.89:0/3811817500 shutdown_connections 2024-04-02T07:36:17.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.670+0000 7fabbf600700 1 -- 172.21.15.89:0/3811817500 wait complete. 2024-04-02T07:36:17.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.672+0000 7fabbf600700 1 Processor -- start 2024-04-02T07:36:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.672+0000 7fabbf600700 1 -- start start 2024-04-02T07:36:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.673+0000 7fabbf600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabb80cf230 0x7fabb80b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.673+0000 7fabbf600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabb8160f10 0x7fabb81e7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.673+0000 7fabbf600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb8161590 0x7fabb81e7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.673+0000 7fabbf600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb815fcd0 con 0x7fabb8161590 2024-04-02T07:36:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.673+0000 7fabbf600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb8159000 con 0x7fabb80cf230 2024-04-02T07:36:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.673+0000 7fabbf600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb80b6320 con 0x7fabb8160f10 2024-04-02T07:36:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.673+0000 7fabbd600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb8161590 0x7fabb81e7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.673+0000 7fabbd600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb8161590 0x7fabb81e7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47240/0 (socket says 172.21.15.89:47240) 2024-04-02T07:36:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.673+0000 7fabbd600700 1 -- 172.21.15.89:0/2033514155 learned_addr learned my addr 172.21.15.89:0/2033514155 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.673+0000 7fabb7e00700 1 --2- 172.21.15.89:0/2033514155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabb8160f10 0x7fabb81e7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.673+0000 7fabbcc00700 1 --2- 172.21.15.89:0/2033514155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabb80cf230 0x7fabb80b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.674+0000 7fabbd600700 1 -- 172.21.15.89:0/2033514155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabb8160f10 msgr2=0x7fabb81e7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.674+0000 7fabbd600700 1 --2- 172.21.15.89:0/2033514155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabb8160f10 0x7fabb81e7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.674+0000 7fabbd600700 1 -- 172.21.15.89:0/2033514155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabb80cf230 msgr2=0x7fabb80b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.674+0000 7fabbd600700 1 --2- 172.21.15.89:0/2033514155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabb80cf230 0x7fabb80b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.674+0000 7fabbd600700 1 -- 172.21.15.89:0/2033514155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb0009b30 con 0x7fabb8161590 2024-04-02T07:36:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.674+0000 7fabbd600700 1 --2- 172.21.15.89:0/2033514155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb8161590 0x7fabb81e7550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fabb808c570 tx=0x7fabb0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.674+0000 7fabb4c00700 1 -- 172.21.15.89:0/2033514155 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb00099b0 con 0x7fabb8161590 2024-04-02T07:36:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.674+0000 7fabb4c00700 1 -- 172.21.15.89:0/2033514155 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb0004250 con 0x7fabb8161590 2024-04-02T07:36:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.674+0000 7fabbf600700 1 -- 172.21.15.89:0/2033514155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb80b6600 con 0x7fabb8161590 2024-04-02T07:36:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.674+0000 7fabb4c00700 1 -- 172.21.15.89:0/2033514155 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb0004600 con 0x7fabb8161590 2024-04-02T07:36:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.674+0000 7fabbf600700 1 -- 172.21.15.89:0/2033514155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb80b6bc0 con 0x7fabb8161590 2024-04-02T07:36:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.675+0000 7fabbf600700 1 -- 172.21.15.89:0/2033514155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb8067720 con 0x7fabb8161590 2024-04-02T07:36:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.676+0000 7fabb4c00700 1 -- 172.21.15.89:0/2033514155 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fabb0004760 con 0x7fabb8161590 2024-04-02T07:36:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.677+0000 7fabb4c00700 1 --2- 172.21.15.89:0/2033514155 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fab9806ac90 0x7fab9806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:17.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.677+0000 7fabb4c00700 1 -- 172.21.15.89:0/2033514155 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fabb008a730 con 0x7fabb8161590 2024-04-02T07:36:17.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.677+0000 7fabbcc00700 1 --2- 172.21.15.89:0/2033514155 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fab9806ac90 0x7fab9806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:17.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.678+0000 7fabbcc00700 1 --2- 172.21.15.89:0/2033514155 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fab9806ac90 0x7fab9806d150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fabac006b20 tx=0x7fabac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:17.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.681+0000 7fabb4c00700 1 -- 172.21.15.89:0/2033514155 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fabb0055900 con 0x7fabb8161590 2024-04-02T07:36:17.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.928+0000 7fabbf600700 1 -- 172.21.15.89:0/2033514155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fabb815d140 con 0x7fabb8161590 2024-04-02T07:36:17.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.928+0000 7fabb4c00700 1 -- 172.21.15.89:0/2033514155 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fabb0058f20 con 0x7fabb8161590 2024-04-02T07:36:17.931 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T07:36:17.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.930+0000 7fabbf600700 1 -- 172.21.15.89:0/2033514155 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fab9806ac90 msgr2=0x7fab9806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:17.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.930+0000 7fabbf600700 1 --2- 172.21.15.89:0/2033514155 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fab9806ac90 0x7fab9806d150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fabac006b20 tx=0x7fabac008040 comp rx=0 tx=0).stop 2024-04-02T07:36:17.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.930+0000 7fabbf600700 1 -- 172.21.15.89:0/2033514155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb8161590 msgr2=0x7fabb81e7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:17.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.930+0000 7fabbf600700 1 --2- 172.21.15.89:0/2033514155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb8161590 0x7fabb81e7550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fabb808c570 tx=0x7fabb0004ad0 comp rx=0 tx=0).stop 2024-04-02T07:36:17.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.930+0000 7fabbf600700 1 -- 172.21.15.89:0/2033514155 shutdown_connections 2024-04-02T07:36:17.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.930+0000 7fabbf600700 1 --2- 172.21.15.89:0/2033514155 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fab9806ac90 0x7fab9806d150 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:17.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.930+0000 7fabbf600700 1 --2- 172.21.15.89:0/2033514155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabb80cf230 0x7fabb80b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:17.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.930+0000 7fabbf600700 1 --2- 172.21.15.89:0/2033514155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabb8161590 0x7fabb81e7550 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:17.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.930+0000 7fabbf600700 1 --2- 172.21.15.89:0/2033514155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabb8160f10 0x7fabb81e7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:17.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.931+0000 7fabbf600700 1 -- 172.21.15.89:0/2033514155 >> 172.21.15.89:0/2033514155 conn(0x7fabb808aff0 msgr2=0x7fabb80ca9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:17.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.931+0000 7fabbf600700 1 -- 172.21.15.89:0/2033514155 shutdown_connections 2024-04-02T07:36:17.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:17.931+0000 7fabbf600700 1 -- 172.21.15.89:0/2033514155 wait complete. 2024-04-02T07:36:17.951 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T07:36:17.951 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T07:36:17.951 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi113.front.sepia.ceph.com 2024-04-02T07:36:17.951 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-02T07:36:17.951 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-02T07:36:17.952 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T07:36:17.952 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T07:36:17.952 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T07:36:17.952 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-02T07:36:17.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:17.952 DEBUG:teuthology.orchestra.run.smithi113:> ip addr 2024-04-02T07:36:18.001 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:36:18.001 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:36:18.001 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:36:18.001 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:18.002 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-04-02T07:36:18.002 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:18.002 INFO:teuthology.orchestra.run.smithi113.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:36:18.002 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:44 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:18.002 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f0 2024-04-02T07:36:18.002 INFO:teuthology.orchestra.run.smithi113.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:36:18.002 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:45 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:18.002 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f1 2024-04-02T07:36:18.002 INFO:teuthology.orchestra.run.smithi113.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:36:18.002 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c0 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:18.002 INFO:teuthology.orchestra.run.smithi113.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T07:36:18.002 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c1 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:18.002 INFO:teuthology.orchestra.run.smithi113.stdout: inet 172.21.15.113/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T07:36:18.002 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft 42507sec preferred_lft 42507sec 2024-04-02T07:36:18.003 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::ec4:7aff:fe8f:d7c1/64 scope link 2024-04-02T07:36:18.003 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:18.004 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-02T07:36:18.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:18.004 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-04-02T07:36:18.004 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-04-02T07:36:18.004 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link add name ceph-brx type bridge 2024-04-02T07:36:18.004 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip addr flush dev ceph-brx 2024-04-02T07:36:18.005 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set ceph-brx up 2024-04-02T07:36:18.005 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-02T07:36:18.005 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-04-02T07:36:18.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:18.467 DEBUG:teuthology.orchestra.run.smithi113:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T07:36:18.570 INFO:teuthology.orchestra.run.smithi113.stdout:1 2024-04-02T07:36:18.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:18.573 DEBUG:teuthology.orchestra.run.smithi113:> route 2024-04-02T07:36:18.647 INFO:teuthology.orchestra.run.smithi113.stdout:Kernel IP routing table 2024-04-02T07:36:18.647 INFO:teuthology.orchestra.run.smithi113.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T07:36:18.647 INFO:teuthology.orchestra.run.smithi113.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T07:36:18.647 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T07:36:18.647 INFO:teuthology.orchestra.run.smithi113.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T07:36:18.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:18.648 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-04-02T07:36:18.648 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-04-02T07:36:18.648 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T07:36:18.648 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T07:36:18.649 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T07:36:18.649 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-04-02T07:36:19.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:19.366 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-04-02T07:36:19.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:19.411 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list-id 2024-04-02T07:36:19.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:19.496 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-04-02T07:36:19.496 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-04-02T07:36:19.496 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-02T07:36:19.496 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-02T07:36:19.496 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-04-02T07:36:19.672 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-02T07:36:19.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:19.673 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-04-02T07:36:19.673 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-04-02T07:36:19.673 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-02T07:36:19.673 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-02T07:36:19.673 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-02T07:36:19.673 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-02T07:36:19.673 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-02T07:36:19.673 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-04-02T07:36:19.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:19.927 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-04-02T07:36:19.927 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-04-02T07:36:19.927 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set brx.0 up 2024-04-02T07:36:19.927 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set dev brx.0 master ceph-brx 2024-04-02T07:36:19.927 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-04-02T07:36:20.092 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:36:20.092 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-02T07:36:20.136 INFO:teuthology.orchestra.run.smithi113.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-02T07:36:20.136 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:36:20.137 DEBUG:teuthology.orchestra.run.smithi113:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-02T07:36:20.221 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-02T07:36:20.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:20.222 DEBUG:teuthology.orchestra.run.smithi113:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:36:20.362 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:36:20.362 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:36:20.971 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:36:20.971 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:36:20.971 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T07:36:20.971 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: trying mount with new device syntax: 1@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T07:36:20.971 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T07:36:20.976 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T07:36:20.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:20.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-02T07:36:21.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:21.044 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:36:21.132 INFO:teuthology.orchestra.run.smithi113.stdout:client4678 2024-04-02T07:36:21.132 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:36:21.133 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:21.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:21.176 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:36:21.301 INFO:teuthology.orchestra.run.smithi113.stdout:client4678 2024-04-02T07:36:21.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:21.301 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:36:21.384 INFO:teuthology.orchestra.run.smithi113.stdout:client4678 2024-04-02T07:36:21.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:21.385 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ddf7474f-4630-4a1e-ada2-3982714d822b.client4678/status') 2024-04-02T07:36:21.487 INFO:teuthology.orchestra.run.smithi113.stdout:instance: client.4678 (3)192.168.0.1:0/1108564974 2024-04-02T07:36:21.487 INFO:teuthology.orchestra.run.smithi113.stdout:blocklisted: false 2024-04-02T07:36:21.487 INFO:teuthology.orchestra.run.smithi113.stderr:0+1 records in 2024-04-02T07:36:21.487 INFO:teuthology.orchestra.run.smithi113.stderr:0+1 records out 2024-04-02T07:36:21.487 INFO:teuthology.orchestra.run.smithi113.stderr:69 bytes copied, 5.165e-05 s, 1.3 MB/s 2024-04-02T07:36:21.489 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-04-02T07:36:21.490 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-02T07:36:21.490 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-02T07:36:21.490 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T07:36:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.688+0000 7fc88c000700 1 Processor -- start 2024-04-02T07:36:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.688+0000 7fc88c000700 1 -- start start 2024-04-02T07:36:21.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.689+0000 7fc88c000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8840d18b0 0x7fc8840cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:21.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.689+0000 7fc88c000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc884161590 0x7fc884161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:21.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.689+0000 7fc88c000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8840ceda0 con 0x7fc884160f10 2024-04-02T07:36:21.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.689+0000 7fc88c000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8840d1d30 con 0x7fc8840d18b0 2024-04-02T07:36:21.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.689+0000 7fc88c000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc884161f60 con 0x7fc884161590 2024-04-02T07:36:21.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.690+0000 7fc888c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8840d18b0 0x7fc8840cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:21.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.690+0000 7fc888c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8840d18b0 0x7fc8840cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50798/0 (socket says 172.21.15.89:50798) 2024-04-02T07:36:21.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.690+0000 7fc888c00700 1 -- 172.21.15.89:0/1987704107 learned_addr learned my addr 172.21.15.89:0/1987704107 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:21.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.690+0000 7fc88a000700 1 --2- 172.21.15.89:0/1987704107 >> v2:172.21.15.113:3300/0 conn(0x7fc884161590 0x7fc884161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:21.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.690+0000 7fc888c00700 1 -- 172.21.15.89:0/1987704107 >> v2:172.21.15.113:3300/0 conn(0x7fc884161590 msgr2=0x7fc884161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:21.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.690+0000 7fc888c00700 1 --2- 172.21.15.89:0/1987704107 >> v2:172.21.15.113:3300/0 conn(0x7fc884161590 0x7fc884161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:21.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.690+0000 7fc888c00700 1 -- 172.21.15.89:0/1987704107 >> v1:172.21.15.89:6789/0 conn(0x7fc884160f10 legacy=0x7fc8840d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:21.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.690+0000 7fc888c00700 1 -- 172.21.15.89:0/1987704107 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8841612f0 con 0x7fc8840d18b0 2024-04-02T07:36:21.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.690+0000 7fc883e00700 1 -- 172.21.15.89:0/1987704107 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645700237 0 0) 0x7fc8840ceda0 con 0x7fc884160f10 2024-04-02T07:36:21.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.690+0000 7fc888c00700 1 --2- 172.21.15.89:0/1987704107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8840d18b0 0x7fc8840cf230 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc8840d1030 tx=0x7fc870009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78588f721b369f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:21.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.691+0000 7fc883e00700 1 -- 172.21.15.89:0/1987704107 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc870011830 con 0x7fc8840d18b0 2024-04-02T07:36:21.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.691+0000 7fc883e00700 1 -- 172.21.15.89:0/1987704107 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc870011990 con 0x7fc8840d18b0 2024-04-02T07:36:21.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.691+0000 7fc883e00700 1 -- 172.21.15.89:0/1987704107 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc870011d20 con 0x7fc8840d18b0 2024-04-02T07:36:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.691+0000 7fc88c000700 1 -- 172.21.15.89:0/1987704107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8840d18b0 msgr2=0x7fc8840cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.691+0000 7fc88c000700 1 --2- 172.21.15.89:0/1987704107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8840d18b0 0x7fc8840cf230 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc8840d1030 tx=0x7fc870009bd0 comp rx=0 tx=0).stop 2024-04-02T07:36:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.691+0000 7fc88c000700 1 -- 172.21.15.89:0/1987704107 shutdown_connections 2024-04-02T07:36:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.691+0000 7fc88c000700 1 --2- 172.21.15.89:0/1987704107 >> v2:172.21.15.113:3300/0 conn(0x7fc884161590 0x7fc884161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.691+0000 7fc88c000700 1 --2- 172.21.15.89:0/1987704107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8840d18b0 0x7fc8840cf230 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.691+0000 7fc88c000700 1 -- 172.21.15.89:0/1987704107 >> 172.21.15.89:0/1987704107 conn(0x7fc88408aff0 msgr2=0x7fc8840d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.691+0000 7fc88c000700 1 -- 172.21.15.89:0/1987704107 shutdown_connections 2024-04-02T07:36:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.691+0000 7fc88c000700 1 -- 172.21.15.89:0/1987704107 wait complete. 2024-04-02T07:36:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.693+0000 7fc88c000700 1 Processor -- start 2024-04-02T07:36:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.693+0000 7fc88c000700 1 -- start start 2024-04-02T07:36:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.694+0000 7fc88c000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8840d18b0 0x7fc884154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.694+0000 7fc88c000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc884160f10 0x7fc88415aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.694+0000 7fc88c000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc884161590 0x7fc8840b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.694+0000 7fc88c000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88415af50 con 0x7fc8840d18b0 2024-04-02T07:36:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.694+0000 7fc88c000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88415a580 con 0x7fc884160f10 2024-04-02T07:36:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.694+0000 7fc88c000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8840b6b70 con 0x7fc884161590 2024-04-02T07:36:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.694+0000 7fc889600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8840d18b0 0x7fc884154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.694+0000 7fc889600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8840d18b0 0x7fc884154ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50814/0 (socket says 172.21.15.89:50814) 2024-04-02T07:36:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.694+0000 7fc889600700 1 -- 172.21.15.89:0/2953341097 learned_addr learned my addr 172.21.15.89:0/2953341097 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.694+0000 7fc888c00700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc884160f10 0x7fc88415aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.694+0000 7fc88a000700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc884161590 0x7fc8840b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.695+0000 7fc889600700 1 -- 172.21.15.89:0/2953341097 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc884161590 msgr2=0x7fc8840b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.695+0000 7fc889600700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc884161590 0x7fc8840b6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.695+0000 7fc889600700 1 -- 172.21.15.89:0/2953341097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc884160f10 msgr2=0x7fc88415aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.695+0000 7fc889600700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc884160f10 0x7fc88415aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:21.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.695+0000 7fc889600700 1 -- 172.21.15.89:0/2953341097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc870009550 con 0x7fc8840d18b0 2024-04-02T07:36:21.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.695+0000 7fc888c00700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc884160f10 0x7fc88415aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:21.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.695+0000 7fc889600700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8840d18b0 0x7fc884154ce0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc87800c580 tx=0x7fc87800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:21.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.695+0000 7fc881600700 1 -- 172.21.15.89:0/2953341097 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8780054e0 con 0x7fc8840d18b0 2024-04-02T07:36:21.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.695+0000 7fc881600700 1 -- 172.21.15.89:0/2953341097 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc878003680 con 0x7fc8840d18b0 2024-04-02T07:36:21.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.695+0000 7fc88c000700 1 -- 172.21.15.89:0/2953341097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8840b6e50 con 0x7fc8840d18b0 2024-04-02T07:36:21.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.695+0000 7fc881600700 1 -- 172.21.15.89:0/2953341097 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc878003a30 con 0x7fc8840d18b0 2024-04-02T07:36:21.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.695+0000 7fc88c000700 1 -- 172.21.15.89:0/2953341097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88415a210 con 0x7fc8840d18b0 2024-04-02T07:36:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.696+0000 7fc88c000700 1 -- 172.21.15.89:0/2953341097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8840b9480 con 0x7fc8840d18b0 2024-04-02T07:36:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.697+0000 7fc881600700 1 -- 172.21.15.89:0/2953341097 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc878003b90 con 0x7fc8840d18b0 2024-04-02T07:36:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.698+0000 7fc881600700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc86406ac90 0x7fc86406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.698+0000 7fc881600700 1 -- 172.21.15.89:0/2953341097 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8780886f0 con 0x7fc8840d18b0 2024-04-02T07:36:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.698+0000 7fc888c00700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc86406ac90 0x7fc86406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:21.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.699+0000 7fc888c00700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc86406ac90 0x7fc86406d150 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc870014040 tx=0x7fc870008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:21.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.703+0000 7fc881600700 1 -- 172.21.15.89:0/2953341097 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc878053830 con 0x7fc8840d18b0 2024-04-02T07:36:21.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.952+0000 7fc88c000700 1 -- 172.21.15.89:0/2953341097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc8840ce1b0 con 0x7fc8840d18b0 2024-04-02T07:36:21.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.953+0000 7fc881600700 1 -- 172.21.15.89:0/2953341097 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc878017070 con 0x7fc8840d18b0 2024-04-02T07:36:21.956 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T07:36:21.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.955+0000 7fc88c000700 1 -- 172.21.15.89:0/2953341097 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc86406ac90 msgr2=0x7fc86406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:21.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.955+0000 7fc88c000700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc86406ac90 0x7fc86406d150 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc870014040 tx=0x7fc870008030 comp rx=0 tx=0).stop 2024-04-02T07:36:21.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.955+0000 7fc88c000700 1 -- 172.21.15.89:0/2953341097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8840d18b0 msgr2=0x7fc884154ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:21.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.955+0000 7fc88c000700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8840d18b0 0x7fc884154ce0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc87800c580 tx=0x7fc87800c890 comp rx=0 tx=0).stop 2024-04-02T07:36:21.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.955+0000 7fc88c000700 1 -- 172.21.15.89:0/2953341097 shutdown_connections 2024-04-02T07:36:21.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.955+0000 7fc88c000700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc86406ac90 0x7fc86406d150 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:21.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.955+0000 7fc88c000700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc884161590 0x7fc8840b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:21.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.955+0000 7fc88c000700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8840d18b0 0x7fc884154ce0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:21.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.955+0000 7fc88c000700 1 --2- 172.21.15.89:0/2953341097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc884160f10 0x7fc88415aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:21.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.955+0000 7fc88c000700 1 -- 172.21.15.89:0/2953341097 >> 172.21.15.89:0/2953341097 conn(0x7fc88408aff0 msgr2=0x7fc88415f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:21.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.956+0000 7fc88c000700 1 -- 172.21.15.89:0/2953341097 shutdown_connections 2024-04-02T07:36:21.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:21.956+0000 7fc88c000700 1 -- 172.21.15.89:0/2953341097 wait complete. 2024-04-02T07:36:21.977 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T07:36:21.977 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T07:36:21.977 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi113.front.sepia.ceph.com 2024-04-02T07:36:21.977 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-02T07:36:21.977 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-02T07:36:21.977 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T07:36:21.977 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T07:36:21.977 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T07:36:21.978 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-02T07:36:21.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:21.978 DEBUG:teuthology.orchestra.run.smithi113:> ip addr 2024-04-02T07:36:22.025 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:36:22.025 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:36:22.025 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:36:22.025 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:22.025 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-04-02T07:36:22.025 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:22.025 INFO:teuthology.orchestra.run.smithi113.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:36:22.025 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:44 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:22.025 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f0 2024-04-02T07:36:22.025 INFO:teuthology.orchestra.run.smithi113.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:36:22.025 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:45 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:22.025 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f1 2024-04-02T07:36:22.026 INFO:teuthology.orchestra.run.smithi113.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:36:22.026 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c0 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:22.026 INFO:teuthology.orchestra.run.smithi113.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T07:36:22.026 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c1 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:22.026 INFO:teuthology.orchestra.run.smithi113.stdout: inet 172.21.15.113/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T07:36:22.026 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft 42503sec preferred_lft 42503sec 2024-04-02T07:36:22.026 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::ec4:7aff:fe8f:d7c1/64 scope link 2024-04-02T07:36:22.026 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:22.026 INFO:teuthology.orchestra.run.smithi113.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T07:36:22.026 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 76:a3:e0:71:10:ee brd ff:ff:ff:ff:ff:ff 2024-04-02T07:36:22.026 INFO:teuthology.orchestra.run.smithi113.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T07:36:22.026 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:22.026 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::1484:9dff:fe93:e4b3/64 scope link 2024-04-02T07:36:22.027 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:22.027 INFO:teuthology.orchestra.run.smithi113.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:36:22.027 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 76:a3:e0:71:10:ee brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-02T07:36:22.027 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::74a3:e0ff:fe71:10ee/64 scope link 2024-04-02T07:36:22.027 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:22.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:22.028 DEBUG:teuthology.orchestra.run.smithi113:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T07:36:22.091 INFO:teuthology.orchestra.run.smithi113.stdout:1 2024-04-02T07:36:22.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:22.094 DEBUG:teuthology.orchestra.run.smithi113:> route 2024-04-02T07:36:22.141 INFO:teuthology.orchestra.run.smithi113.stdout:Kernel IP routing table 2024-04-02T07:36:22.141 INFO:teuthology.orchestra.run.smithi113.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T07:36:22.141 INFO:teuthology.orchestra.run.smithi113.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T07:36:22.141 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T07:36:22.141 INFO:teuthology.orchestra.run.smithi113.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T07:36:22.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:22.142 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-04-02T07:36:22.142 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-04-02T07:36:22.142 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T07:36:22.142 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T07:36:22.142 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T07:36:22.142 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-04-02T07:36:22.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:22.301 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-04-02T07:36:22.345 INFO:teuthology.orchestra.run.smithi113.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-02T07:36:22.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:22.347 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list-id 2024-04-02T07:36:22.431 INFO:teuthology.orchestra.run.smithi113.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-02T07:36:22.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:22.432 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-04-02T07:36:22.432 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-04-02T07:36:22.432 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-02T07:36:22.432 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-02T07:36:22.432 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-04-02T07:36:22.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:22.605 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-02T07:36:22.672 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:36:22.672 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:36:22.672 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:36:22.672 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:22.672 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-04-02T07:36:22.672 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:22.672 INFO:teuthology.orchestra.run.smithi113.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T07:36:22.672 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 92:c4:8e:94:20:47 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-02T07:36:22.672 INFO:teuthology.orchestra.run.smithi113.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-02T07:36:22.672 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:22.673 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::90c4:8eff:fe94:2047/64 scope link 2024-04-02T07:36:22.673 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:22.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:22.675 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-02T07:36:22.742 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:36:22.742 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:36:22.742 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:36:22.742 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:22.742 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-04-02T07:36:22.742 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:22.742 INFO:teuthology.orchestra.run.smithi113.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T07:36:22.742 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 92:c4:8e:94:20:47 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-02T07:36:22.742 INFO:teuthology.orchestra.run.smithi113.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-02T07:36:22.742 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:22.742 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::90c4:8eff:fe94:2047/64 scope link 2024-04-02T07:36:22.743 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:36:22.744 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-02T07:36:22.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:22.745 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-04-02T07:36:22.745 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-04-02T07:36:22.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-02T07:36:22.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-02T07:36:22.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-02T07:36:22.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-02T07:36:22.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-02T07:36:22.745 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-04-02T07:36:22.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:22.977 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-04-02T07:36:22.977 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-04-02T07:36:22.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set brx.1 up 2024-04-02T07:36:22.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set dev brx.1 master ceph-brx 2024-04-02T07:36:22.977 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-04-02T07:36:23.143 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:36:23.143 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-02T07:36:23.187 INFO:teuthology.orchestra.run.smithi113.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-02T07:36:23.187 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:36:23.188 DEBUG:teuthology.orchestra.run.smithi113:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-02T07:36:23.271 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-04-02T07:36:23.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:23.271 DEBUG:teuthology.orchestra.run.smithi113:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:36:23.410 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:36:23.410 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:36:23.438 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:36:23.438 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:36:23.438 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T07:36:23.438 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: trying mount with new device syntax: 3@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T07:36:23.438 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T07:36:23.442 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T07:36:23.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:23.442 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-02T07:36:23.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:23.508 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:36:23.592 INFO:teuthology.orchestra.run.smithi113.stdout:client4683 2024-04-02T07:36:23.593 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:36:23.593 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:23.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:23.637 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:36:23.761 INFO:teuthology.orchestra.run.smithi113.stdout:client4683 2024-04-02T07:36:23.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:23.761 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:36:23.845 INFO:teuthology.orchestra.run.smithi113.stdout:client4683 2024-04-02T07:36:23.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:23.846 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ddf7474f-4630-4a1e-ada2-3982714d822b.client4683/status') 2024-04-02T07:36:23.947 INFO:teuthology.orchestra.run.smithi113.stdout:instance: client.4683 (3)172.21.15.113:0/1165763071 2024-04-02T07:36:23.947 INFO:teuthology.orchestra.run.smithi113.stdout:blocklisted: false 2024-04-02T07:36:23.947 INFO:teuthology.orchestra.run.smithi113.stderr:0+1 records in 2024-04-02T07:36:23.948 INFO:teuthology.orchestra.run.smithi113.stderr:0+1 records out 2024-04-02T07:36:23.948 INFO:teuthology.orchestra.run.smithi113.stderr:71 bytes copied, 5.0139e-05 s, 1.4 MB/s 2024-04-02T07:36:23.950 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-02T07:36:23.985 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:36:24.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.185+0000 7fcdb2800700 1 Processor -- start 2024-04-02T07:36:24.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.185+0000 7fcdb2800700 1 -- start start 2024-04-02T07:36:24.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.186+0000 7fcdb2800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdac160f10 0x7fcdac0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:24.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.186+0000 7fcdb2800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcdac161590 0x7fcdac161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:24.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.186+0000 7fcdb2800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdac0ceda0 con 0x7fcdac0cf230 2024-04-02T07:36:24.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.186+0000 7fcdb2800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdac161f80 con 0x7fcdac161590 2024-04-02T07:36:24.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.186+0000 7fcdb2800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac1612f0 con 0x7fcdac160f10 2024-04-02T07:36:24.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.187+0000 7fcdabe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcdac161590 0x7fcdac161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:24.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.187+0000 7fcdabe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcdac161590 0x7fcdac161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50830/0 (socket says 172.21.15.89:50830) 2024-04-02T07:36:24.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.187+0000 7fcdabe00700 1 -- 172.21.15.89:0/1040964253 learned_addr learned my addr 172.21.15.89:0/1040964253 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:24.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.187+0000 7fcdab400700 1 --2- 172.21.15.89:0/1040964253 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdac160f10 0x7fcdac0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:24.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.187+0000 7fcdabe00700 1 -- 172.21.15.89:0/1040964253 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdac160f10 msgr2=0x7fcdac0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:24.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.187+0000 7fcdabe00700 1 --2- 172.21.15.89:0/1040964253 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdac160f10 0x7fcdac0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.187+0000 7fcdabe00700 1 -- 172.21.15.89:0/1040964253 >> v1:172.21.15.113:6789/0 conn(0x7fcdac0cf230 legacy=0x7fcdac0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:24.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.187+0000 7fcdabe00700 1 -- 172.21.15.89:0/1040964253 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac0ce470 con 0x7fcdac161590 2024-04-02T07:36:24.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.187+0000 7fcdabe00700 1 --2- 172.21.15.89:0/1040964253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac161590 0x7fcdac161a40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009e80 tx=0x7fcda400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fdecd9f79a1da03 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:24.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.188+0000 7fcdaa000700 1 -- 172.21.15.89:0/1040964253 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda400e070 con 0x7fcdac161590 2024-04-02T07:36:24.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.188+0000 7fcdaa000700 1 -- 172.21.15.89:0/1040964253 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda400b8d0 con 0x7fcdac161590 2024-04-02T07:36:24.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.188+0000 7fcdaa000700 1 -- 172.21.15.89:0/1040964253 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda400bc80 con 0x7fcdac161590 2024-04-02T07:36:24.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.188+0000 7fcdb2800700 1 -- 172.21.15.89:0/1040964253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac161590 msgr2=0x7fcdac161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:24.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.188+0000 7fcdb2800700 1 --2- 172.21.15.89:0/1040964253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac161590 0x7fcdac161a40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009e80 tx=0x7fcda400b330 comp rx=0 tx=0).stop 2024-04-02T07:36:24.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.188+0000 7fcdb2800700 1 -- 172.21.15.89:0/1040964253 shutdown_connections 2024-04-02T07:36:24.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.188+0000 7fcdb2800700 1 --2- 172.21.15.89:0/1040964253 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdac160f10 0x7fcdac0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.188+0000 7fcdb2800700 1 --2- 172.21.15.89:0/1040964253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac161590 0x7fcdac161a40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.188+0000 7fcdb2800700 1 -- 172.21.15.89:0/1040964253 >> 172.21.15.89:0/1040964253 conn(0x7fcdac08aff0 msgr2=0x7fcdac0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:24.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.188+0000 7fcdb2800700 1 -- 172.21.15.89:0/1040964253 shutdown_connections 2024-04-02T07:36:24.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.188+0000 7fcdb2800700 1 -- 172.21.15.89:0/1040964253 wait complete. 2024-04-02T07:36:24.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.190+0000 7fcdb2800700 1 Processor -- start 2024-04-02T07:36:24.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.190+0000 7fcdb2800700 1 -- start start 2024-04-02T07:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.191+0000 7fcdb2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac0cf230 0x7fcdac0b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.191+0000 7fcdb2800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdac160f10 0x7fcdac160030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.191+0000 7fcdb2800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdac161590 0x7fcdac0a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.191+0000 7fcdb2800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac14efb0 con 0x7fcdac0cf230 2024-04-02T07:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.191+0000 7fcdb2800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac0b6de0 con 0x7fcdac160f10 2024-04-02T07:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.191+0000 7fcdb2800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac0b70b0 con 0x7fcdac161590 2024-04-02T07:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.191+0000 7fcdab400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac0cf230 0x7fcdac0b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.191+0000 7fcdab400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac0cf230 0x7fcdac0b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50844/0 (socket says 172.21.15.89:50844) 2024-04-02T07:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.191+0000 7fcdab400700 1 -- 172.21.15.89:0/1009142381 learned_addr learned my addr 172.21.15.89:0/1009142381 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcdaaa00700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdac160f10 0x7fcdac160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcdabe00700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdac161590 0x7fcdac0a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcdab400700 1 -- 172.21.15.89:0/1009142381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdac161590 msgr2=0x7fcdac0a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcdab400700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdac161590 0x7fcdac0a08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcdab400700 1 -- 172.21.15.89:0/1009142381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdac160f10 msgr2=0x7fcdac160030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcdab400700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdac160f10 0x7fcdac160030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcdab400700 1 -- 172.21.15.89:0/1009142381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda4009b30 con 0x7fcdac0cf230 2024-04-02T07:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcdabe00700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdac161590 0x7fcdac0a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcdaaa00700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdac160f10 0x7fcdac160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcdab400700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac0cf230 0x7fcdac0b68a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00e870 tx=0x7fcd9c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcda3400700 1 -- 172.21.15.89:0/1009142381 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c009590 con 0x7fcdac0cf230 2024-04-02T07:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcda3400700 1 -- 172.21.15.89:0/1009142381 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9c00ee70 con 0x7fcdac0cf230 2024-04-02T07:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcdb2800700 1 -- 172.21.15.89:0/1009142381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdac0b7390 con 0x7fcdac0cf230 2024-04-02T07:36:24.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcda3400700 1 -- 172.21.15.89:0/1009142381 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c00ca50 con 0x7fcdac0cf230 2024-04-02T07:36:24.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.192+0000 7fcdb2800700 1 -- 172.21.15.89:0/1009142381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdac0b7950 con 0x7fcdac0cf230 2024-04-02T07:36:24.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.193+0000 7fcdb2800700 1 -- 172.21.15.89:0/1009142381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdac0729f0 con 0x7fcdac0cf230 2024-04-02T07:36:24.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.194+0000 7fcda3400700 1 -- 172.21.15.89:0/1009142381 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcd9c00cbb0 con 0x7fcdac0cf230 2024-04-02T07:36:24.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.195+0000 7fcda3400700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcd8c06ac90 0x7fcd8c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:24.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.195+0000 7fcda3400700 1 -- 172.21.15.89:0/1009142381 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd9c089720 con 0x7fcdac0cf230 2024-04-02T07:36:24.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.195+0000 7fcdaaa00700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcd8c06ac90 0x7fcd8c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:24.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.196+0000 7fcdaaa00700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcd8c06ac90 0x7fcd8c06d150 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcdac159110 tx=0x7fcd98009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.199+0000 7fcda3400700 1 -- 172.21.15.89:0/1009142381 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcd9c054860 con 0x7fcdac0cf230 2024-04-02T07:36:24.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.403+0000 7fcdb2800700 1 -- 172.21.15.89:0/1009142381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdac067720 con 0x7fcdac0cf230 2024-04-02T07:36:24.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.404+0000 7fcda3400700 1 -- 172.21.15.89:0/1009142381 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7fcd9c00ce60 con 0x7fcdac0cf230 2024-04-02T07:36:24.406 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:24.406 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:54.238913+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:36:24.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.406+0000 7fcdb2800700 1 -- 172.21.15.89:0/1009142381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcd8c06ac90 msgr2=0x7fcd8c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:24.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.406+0000 7fcdb2800700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcd8c06ac90 0x7fcd8c06d150 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcdac159110 tx=0x7fcd98009040 comp rx=0 tx=0).stop 2024-04-02T07:36:24.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.406+0000 7fcdb2800700 1 -- 172.21.15.89:0/1009142381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac0cf230 msgr2=0x7fcdac0b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:24.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.406+0000 7fcdb2800700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac0cf230 0x7fcdac0b68a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00e870 tx=0x7fcd9c00ec30 comp rx=0 tx=0).stop 2024-04-02T07:36:24.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.406+0000 7fcdb2800700 1 -- 172.21.15.89:0/1009142381 shutdown_connections 2024-04-02T07:36:24.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.406+0000 7fcdb2800700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcd8c06ac90 0x7fcd8c06d150 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.406+0000 7fcdb2800700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdac161590 0x7fcdac0a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.406+0000 7fcdb2800700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac0cf230 0x7fcdac0b68a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.406+0000 7fcdb2800700 1 --2- 172.21.15.89:0/1009142381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdac160f10 0x7fcdac160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.406+0000 7fcdb2800700 1 -- 172.21.15.89:0/1009142381 >> 172.21.15.89:0/1009142381 conn(0x7fcdac08aff0 msgr2=0x7fcdac06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:24.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.406+0000 7fcdb2800700 1 -- 172.21.15.89:0/1009142381 shutdown_connections 2024-04-02T07:36:24.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.406+0000 7fcdb2800700 1 -- 172.21.15.89:0/1009142381 wait complete. 2024-04-02T07:36:24.426 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:35:51.296561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:35:52.232882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:36:24.426 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:36:24.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.617+0000 7f3939a00700 1 Processor -- start 2024-04-02T07:36:24.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.618+0000 7f3939a00700 1 -- start start 2024-04-02T07:36:24.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.619+0000 7f3939a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f39340d16a0 0x7f39340d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:24.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.619+0000 7f3939a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3934161420 con 0x7f39340d2370 2024-04-02T07:36:24.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.619+0000 7f3939a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3934161700 con 0x7f393415d9d0 2024-04-02T07:36:24.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.619+0000 7f3939a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39340d4340 con 0x7f39340d16a0 2024-04-02T07:36:24.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.619+0000 7f392be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f39340d16a0 0x7f39340d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:24.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.619+0000 7f3933400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f39340d2370 0x7f39340ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59842/0 (socket says 172.21.15.89:59842) 2024-04-02T07:36:24.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.619+0000 7f3933400700 1 -- 172.21.15.89:0/316542596 learned_addr learned my addr 172.21.15.89:0/316542596 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:24.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.619+0000 7f3932000700 1 -- 172.21.15.89:0/316542596 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693758152 0 0) 0x7f3934161420 con 0x7f39340d2370 2024-04-02T07:36:24.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.620+0000 7f3932000700 1 -- 172.21.15.89:0/316542596 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39180035b0 con 0x7f39340d2370 2024-04-02T07:36:24.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.620+0000 7f3932000700 1 -- 172.21.15.89:0/316542596 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668957335 0 0) 0x7f3934161700 con 0x7f393415d9d0 2024-04-02T07:36:24.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.620+0000 7f392be00700 1 -- 172.21.15.89:0/316542596 >> v1:172.21.15.113:6789/0 conn(0x7f393415d9d0 legacy=0x7f39340d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:24.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.620+0000 7f392be00700 1 -- 172.21.15.89:0/316542596 >> v1:172.21.15.89:6789/0 conn(0x7f39340d2370 legacy=0x7f39340ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:24.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.620+0000 7f392be00700 1 -- 172.21.15.89:0/316542596 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39341619e0 con 0x7f39340d16a0 2024-04-02T07:36:24.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.620+0000 7f392be00700 1 --2- 172.21.15.89:0/316542596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39340d16a0 0x7f39340d1e30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f391c009820 tx=0x7f391c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6278a5ac363ffe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:24.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.620+0000 7f3932000700 1 -- 172.21.15.89:0/316542596 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c005e30 con 0x7f39340d16a0 2024-04-02T07:36:24.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.620+0000 7f3932000700 1 -- 172.21.15.89:0/316542596 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391c006030 con 0x7f39340d16a0 2024-04-02T07:36:24.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.620+0000 7f3932000700 1 -- 172.21.15.89:0/316542596 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c0063c0 con 0x7f39340d16a0 2024-04-02T07:36:24.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.621+0000 7f3939a00700 1 -- 172.21.15.89:0/316542596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39340d16a0 msgr2=0x7f39340d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:24.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.621+0000 7f3939a00700 1 --2- 172.21.15.89:0/316542596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39340d16a0 0x7f39340d1e30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f391c009820 tx=0x7f391c009b30 comp rx=0 tx=0).stop 2024-04-02T07:36:24.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.621+0000 7f3939a00700 1 -- 172.21.15.89:0/316542596 shutdown_connections 2024-04-02T07:36:24.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.621+0000 7f3939a00700 1 --2- 172.21.15.89:0/316542596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39340d16a0 0x7f39340d1e30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.621+0000 7f3939a00700 1 -- 172.21.15.89:0/316542596 >> 172.21.15.89:0/316542596 conn(0x7f393408aff0 msgr2=0x7f39340d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:24.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.621+0000 7f3939a00700 1 -- 172.21.15.89:0/316542596 shutdown_connections 2024-04-02T07:36:24.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.621+0000 7f3939a00700 1 -- 172.21.15.89:0/316542596 wait complete. 2024-04-02T07:36:24.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.622+0000 7f3939a00700 1 Processor -- start 2024-04-02T07:36:24.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.623+0000 7f3939a00700 1 -- start start 2024-04-02T07:36:24.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.623+0000 7f3939a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39340d16a0 0x7f393415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:24.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.623+0000 7f3939a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39340d2370 0x7f3934158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:24.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.623+0000 7f3939a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f393415d9d0 0x7f393415ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:24.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.623+0000 7f3939a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39340cd7f0 con 0x7f39340d16a0 2024-04-02T07:36:24.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.623+0000 7f3939a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39340cd510 con 0x7f393415d9d0 2024-04-02T07:36:24.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.623+0000 7f3939a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3934159800 con 0x7f39340d2370 2024-04-02T07:36:24.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.624+0000 7f3932a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39340d16a0 0x7f393415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:24.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.624+0000 7f3932a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39340d16a0 0x7f393415a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50852/0 (socket says 172.21.15.89:50852) 2024-04-02T07:36:24.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.624+0000 7f3932a00700 1 -- 172.21.15.89:0/3614169914 learned_addr learned my addr 172.21.15.89:0/3614169914 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:24.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.624+0000 7f392be00700 1 --2- 172.21.15.89:0/3614169914 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39340d2370 0x7f3934158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:24.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.624+0000 7f3933400700 1 --2- 172.21.15.89:0/3614169914 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f393415d9d0 0x7f393415ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:24.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.624+0000 7f3932a00700 1 -- 172.21.15.89:0/3614169914 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39340d2370 msgr2=0x7f3934158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:24.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.624+0000 7f3932a00700 1 --2- 172.21.15.89:0/3614169914 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39340d2370 0x7f3934158410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.624+0000 7f3932a00700 1 -- 172.21.15.89:0/3614169914 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f393415d9d0 msgr2=0x7f393415ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:24.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.624+0000 7f3932a00700 1 --2- 172.21.15.89:0/3614169914 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f393415d9d0 0x7f393415ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.624+0000 7f3932a00700 1 -- 172.21.15.89:0/3614169914 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391c009430 con 0x7f39340d16a0 2024-04-02T07:36:24.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.624+0000 7f3932a00700 1 --2- 172.21.15.89:0/3614169914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39340d16a0 0x7f393415a2e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f392400c550 tx=0x7f392400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:24.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.625+0000 7f392aa00700 1 -- 172.21.15.89:0/3614169914 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39240054e0 con 0x7f39340d16a0 2024-04-02T07:36:24.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.625+0000 7f392aa00700 1 -- 172.21.15.89:0/3614169914 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3924003680 con 0x7f39340d16a0 2024-04-02T07:36:24.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.625+0000 7f3939a00700 1 -- 172.21.15.89:0/3614169914 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3934159ae0 con 0x7f39340d16a0 2024-04-02T07:36:24.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.625+0000 7f392aa00700 1 -- 172.21.15.89:0/3614169914 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924003a30 con 0x7f39340d16a0 2024-04-02T07:36:24.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.625+0000 7f3939a00700 1 -- 172.21.15.89:0/3614169914 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3934158950 con 0x7f39340d16a0 2024-04-02T07:36:24.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.626+0000 7f3939a00700 1 -- 172.21.15.89:0/3614169914 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39340d1ca0 con 0x7f39340d16a0 2024-04-02T07:36:24.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.627+0000 7f392aa00700 1 -- 172.21.15.89:0/3614169914 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3924003b90 con 0x7f39340d16a0 2024-04-02T07:36:24.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.627+0000 7f392aa00700 1 --2- 172.21.15.89:0/3614169914 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f391006ac90 0x7f391006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:24.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.628+0000 7f392aa00700 1 -- 172.21.15.89:0/3614169914 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39240887e0 con 0x7f39340d16a0 2024-04-02T07:36:24.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.628+0000 7f392be00700 1 --2- 172.21.15.89:0/3614169914 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f391006ac90 0x7f391006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:24.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.628+0000 7f392be00700 1 --2- 172.21.15.89:0/3614169914 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f391006ac90 0x7f391006d150 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f39340d1ae0 tx=0x7f391c008070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:24.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.632+0000 7f392aa00700 1 -- 172.21.15.89:0/3614169914 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f39240539b0 con 0x7f39340d16a0 2024-04-02T07:36:24.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.837+0000 7f3939a00700 1 -- 172.21.15.89:0/3614169914 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3934067720 con 0x7f39340d16a0 2024-04-02T07:36:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.837+0000 7f392aa00700 1 -- 172.21.15.89:0/3614169914 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3924017020 con 0x7f39340d16a0 2024-04-02T07:36:24.839 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:36:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.840+0000 7f3939a00700 1 -- 172.21.15.89:0/3614169914 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f391006ac90 msgr2=0x7f391006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.840+0000 7f3939a00700 1 --2- 172.21.15.89:0/3614169914 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f391006ac90 0x7f391006d150 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f39340d1ae0 tx=0x7f391c008070 comp rx=0 tx=0).stop 2024-04-02T07:36:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.840+0000 7f3939a00700 1 -- 172.21.15.89:0/3614169914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39340d16a0 msgr2=0x7f393415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.840+0000 7f3939a00700 1 --2- 172.21.15.89:0/3614169914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39340d16a0 0x7f393415a2e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f392400c550 tx=0x7f392400c910 comp rx=0 tx=0).stop 2024-04-02T07:36:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.840+0000 7f3939a00700 1 -- 172.21.15.89:0/3614169914 shutdown_connections 2024-04-02T07:36:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.840+0000 7f3939a00700 1 --2- 172.21.15.89:0/3614169914 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f391006ac90 0x7f391006d150 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.840+0000 7f3939a00700 1 --2- 172.21.15.89:0/3614169914 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f393415d9d0 0x7f393415ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.840+0000 7f3939a00700 1 --2- 172.21.15.89:0/3614169914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39340d16a0 0x7f393415a2e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.840+0000 7f3939a00700 1 --2- 172.21.15.89:0/3614169914 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39340d2370 0x7f3934158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:24.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.840+0000 7f3939a00700 1 -- 172.21.15.89:0/3614169914 >> 172.21.15.89:0/3614169914 conn(0x7f393408aff0 msgr2=0x7f39340d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:24.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.840+0000 7f3939a00700 1 -- 172.21.15.89:0/3614169914 shutdown_connections 2024-04-02T07:36:24.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:24.840+0000 7f3939a00700 1 -- 172.21.15.89:0/3614169914 wait complete. 2024-04-02T07:36:24.859 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:36:25.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.051+0000 7f28e0600700 1 Processor -- start 2024-04-02T07:36:25.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.052+0000 7f28e0600700 1 -- start start 2024-04-02T07:36:25.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.053+0000 7f28e0600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f28d815d9d0 0x7f28d80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.053+0000 7f28e0600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d8161420 con 0x7f28d80d2540 2024-04-02T07:36:25.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.053+0000 7f28e0600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d8161700 con 0x7f28d80d16a0 2024-04-02T07:36:25.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.053+0000 7f28e0600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28d815edc0 con 0x7f28d815d9d0 2024-04-02T07:36:25.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.053+0000 7f28de600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f28d80d2540 0x7f28d80ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59856/0 (socket says 172.21.15.89:59856) 2024-04-02T07:36:25.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.053+0000 7f28de600700 1 -- 172.21.15.89:0/2406165025 learned_addr learned my addr 172.21.15.89:0/2406165025 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:25.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.053+0000 7f28ddc00700 1 --2- 172.21.15.89:0/2406165025 >> v2:172.21.15.113:3300/0 conn(0x7f28d815d9d0 0x7f28d80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.053+0000 7f28d7e00700 1 -- 172.21.15.89:0/2406165025 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404317333 0 0) 0x7f28d8161420 con 0x7f28d80d2540 2024-04-02T07:36:25.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.054+0000 7f28d7e00700 1 -- 172.21.15.89:0/2406165025 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c00035b0 con 0x7f28d80d2540 2024-04-02T07:36:25.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.054+0000 7f28ddc00700 1 -- 172.21.15.89:0/2406165025 >> v1:172.21.15.113:6789/0 conn(0x7f28d80d16a0 legacy=0x7f28d80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.054+0000 7f28ddc00700 1 -- 172.21.15.89:0/2406165025 >> v1:172.21.15.89:6789/0 conn(0x7f28d80d2540 legacy=0x7f28d80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.054+0000 7f28ddc00700 1 -- 172.21.15.89:0/2406165025 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d81619e0 con 0x7f28d815d9d0 2024-04-02T07:36:25.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.054+0000 7f28ddc00700 1 --2- 172.21.15.89:0/2406165025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28d815d9d0 0x7f28d80d3c30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009840 tx=0x7f28cc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0d8e6817caa9eb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:25.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.054+0000 7f28d7e00700 1 -- 172.21.15.89:0/2406165025 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc011830 con 0x7f28d815d9d0 2024-04-02T07:36:25.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.054+0000 7f28d7e00700 1 -- 172.21.15.89:0/2406165025 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28cc011990 con 0x7f28d815d9d0 2024-04-02T07:36:25.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.054+0000 7f28d7e00700 1 -- 172.21.15.89:0/2406165025 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc011d00 con 0x7f28d815d9d0 2024-04-02T07:36:25.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.055+0000 7f28e0600700 1 -- 172.21.15.89:0/2406165025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28d815d9d0 msgr2=0x7f28d80d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.055+0000 7f28e0600700 1 --2- 172.21.15.89:0/2406165025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28d815d9d0 0x7f28d80d3c30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009840 tx=0x7f28cc009b50 comp rx=0 tx=0).stop 2024-04-02T07:36:25.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.055+0000 7f28e0600700 1 -- 172.21.15.89:0/2406165025 shutdown_connections 2024-04-02T07:36:25.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.055+0000 7f28e0600700 1 --2- 172.21.15.89:0/2406165025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28d815d9d0 0x7f28d80d3c30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.055+0000 7f28e0600700 1 -- 172.21.15.89:0/2406165025 >> 172.21.15.89:0/2406165025 conn(0x7f28d808aff0 msgr2=0x7f28d80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:25.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.055+0000 7f28e0600700 1 -- 172.21.15.89:0/2406165025 shutdown_connections 2024-04-02T07:36:25.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.055+0000 7f28e0600700 1 -- 172.21.15.89:0/2406165025 wait complete. 2024-04-02T07:36:25.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.057+0000 7f28e0600700 1 Processor -- start 2024-04-02T07:36:25.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.057+0000 7f28e0600700 1 -- start start 2024-04-02T07:36:25.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28e0600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28d80d16a0 0x7f28d81f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28e0600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d80d2540 0x7f28d815bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28e0600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28d815d9d0 0x7f28d80ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28e0600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d815c120 con 0x7f28d80d2540 2024-04-02T07:36:25.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28e0600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d81f7410 con 0x7f28d80d16a0 2024-04-02T07:36:25.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28e0600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d81f76f0 con 0x7f28d815d9d0 2024-04-02T07:36:25.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28dd200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d80d2540 0x7f28d815bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28dd200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d80d2540 0x7f28d815bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50868/0 (socket says 172.21.15.89:50868) 2024-04-02T07:36:25.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28dd200700 1 -- 172.21.15.89:0/2987301522 learned_addr learned my addr 172.21.15.89:0/2987301522 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:25.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28de600700 1 --2- 172.21.15.89:0/2987301522 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28d815d9d0 0x7f28d80ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28ddc00700 1 --2- 172.21.15.89:0/2987301522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28d80d16a0 0x7f28d81f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28dd200700 1 -- 172.21.15.89:0/2987301522 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28d815d9d0 msgr2=0x7f28d80ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28dd200700 1 --2- 172.21.15.89:0/2987301522 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28d815d9d0 0x7f28d80ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28dd200700 1 -- 172.21.15.89:0/2987301522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28d80d16a0 msgr2=0x7f28d81f6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28dd200700 1 --2- 172.21.15.89:0/2987301522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28d80d16a0 0x7f28d81f6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.058+0000 7f28dd200700 1 -- 172.21.15.89:0/2987301522 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28cc0094a0 con 0x7f28d80d2540 2024-04-02T07:36:25.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.059+0000 7f28dd200700 1 --2- 172.21.15.89:0/2987301522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d80d2540 0x7f28d815bbe0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f28c400c450 tx=0x7f28c400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:25.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.059+0000 7f28d5600700 1 -- 172.21.15.89:0/2987301522 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c40052d0 con 0x7f28d80d2540 2024-04-02T07:36:25.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.059+0000 7f28d5600700 1 -- 172.21.15.89:0/2987301522 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c4005450 con 0x7f28d80d2540 2024-04-02T07:36:25.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.059+0000 7f28e0600700 1 -- 172.21.15.89:0/2987301522 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d81f79d0 con 0x7f28d80d2540 2024-04-02T07:36:25.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.059+0000 7f28d5600700 1 -- 172.21.15.89:0/2987301522 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c4006af0 con 0x7f28d80d2540 2024-04-02T07:36:25.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.059+0000 7f28e0600700 1 -- 172.21.15.89:0/2987301522 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d81f7cb0 con 0x7f28d80d2540 2024-04-02T07:36:25.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.060+0000 7f28e0600700 1 -- 172.21.15.89:0/2987301522 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d80d1ca0 con 0x7f28d80d2540 2024-04-02T07:36:25.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.061+0000 7f28d5600700 1 -- 172.21.15.89:0/2987301522 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f28c40036d0 con 0x7f28d80d2540 2024-04-02T07:36:25.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.062+0000 7f28d5600700 1 --2- 172.21.15.89:0/2987301522 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f28b806ac90 0x7f28b806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.062+0000 7f28d5600700 1 -- 172.21.15.89:0/2987301522 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28c4088600 con 0x7f28d80d2540 2024-04-02T07:36:25.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.062+0000 7f28ddc00700 1 --2- 172.21.15.89:0/2987301522 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f28b806ac90 0x7f28b806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.063+0000 7f28ddc00700 1 --2- 172.21.15.89:0/2987301522 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f28b806ac90 0x7f28b806d150 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0090b0 tx=0x7f28cc01c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:25.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.066+0000 7f28d5600700 1 -- 172.21.15.89:0/2987301522 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f28c40537d0 con 0x7f28d80d2540 2024-04-02T07:36:25.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.270+0000 7f28e0600700 1 -- 172.21.15.89:0/2987301522 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f28d80b8480 con 0x7f28d80d2540 2024-04-02T07:36:25.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.271+0000 7f28d5600700 1 -- 172.21.15.89:0/2987301522 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f28d80b8480 con 0x7f28d80d2540 2024-04-02T07:36:25.273 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:36:25.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.274+0000 7f28e0600700 1 -- 172.21.15.89:0/2987301522 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f28b806ac90 msgr2=0x7f28b806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.274+0000 7f28e0600700 1 --2- 172.21.15.89:0/2987301522 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f28b806ac90 0x7f28b806d150 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0090b0 tx=0x7f28cc01c040 comp rx=0 tx=0).stop 2024-04-02T07:36:25.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.274+0000 7f28e0600700 1 -- 172.21.15.89:0/2987301522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d80d2540 msgr2=0x7f28d815bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.274+0000 7f28e0600700 1 --2- 172.21.15.89:0/2987301522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d80d2540 0x7f28d815bbe0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f28c400c450 tx=0x7f28c400c810 comp rx=0 tx=0).stop 2024-04-02T07:36:25.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.274+0000 7f28e0600700 1 -- 172.21.15.89:0/2987301522 shutdown_connections 2024-04-02T07:36:25.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.274+0000 7f28e0600700 1 --2- 172.21.15.89:0/2987301522 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f28b806ac90 0x7f28b806d150 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.274+0000 7f28e0600700 1 --2- 172.21.15.89:0/2987301522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28d80d16a0 0x7f28d81f6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.274+0000 7f28e0600700 1 --2- 172.21.15.89:0/2987301522 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28d815d9d0 0x7f28d80ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.274+0000 7f28e0600700 1 --2- 172.21.15.89:0/2987301522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28d80d2540 0x7f28d815bbe0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.274+0000 7f28e0600700 1 -- 172.21.15.89:0/2987301522 >> 172.21.15.89:0/2987301522 conn(0x7f28d808aff0 msgr2=0x7f28d806c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:25.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.274+0000 7f28e0600700 1 -- 172.21.15.89:0/2987301522 shutdown_connections 2024-04-02T07:36:25.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.274+0000 7f28e0600700 1 -- 172.21.15.89:0/2987301522 wait complete. 2024-04-02T07:36:25.294 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:36:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.484+0000 7f22ec400700 1 Processor -- start 2024-04-02T07:36:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.484+0000 7f22ec400700 1 -- start start 2024-04-02T07:36:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.485+0000 7f22ec400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f22e40d16a0 0x7f22e40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.485+0000 7f22ec400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22e40d2370 0x7f22e40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.485+0000 7f22ec400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e4161420 con 0x7f22e415d9d0 2024-04-02T07:36:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.485+0000 7f22ec400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e40d2820 con 0x7f22e40d16a0 2024-04-02T07:36:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.485+0000 7f22ec400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e40d4340 con 0x7f22e40d2370 2024-04-02T07:36:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.486+0000 7f22e9a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f22e415d9d0 0x7f22e40d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59868/0 (socket says 172.21.15.89:59868) 2024-04-02T07:36:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.486+0000 7f22e9a00700 1 -- 172.21.15.89:0/1568426558 learned_addr learned my addr 172.21.15.89:0/1568426558 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.486+0000 7f22e9000700 1 --2- 172.21.15.89:0/1568426558 >> v2:172.21.15.113:3300/0 conn(0x7f22e40d16a0 0x7f22e40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.486+0000 7f22ea400700 1 --2- 172.21.15.89:0/1568426558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22e40d2370 0x7f22e40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.486+0000 7f22e3e00700 1 -- 172.21.15.89:0/1568426558 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561454864 0 0) 0x7f22e4161420 con 0x7f22e415d9d0 2024-04-02T07:36:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.486+0000 7f22e3e00700 1 -- 172.21.15.89:0/1568426558 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22cc0035b0 con 0x7f22e415d9d0 2024-04-02T07:36:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.486+0000 7f22e9000700 1 -- 172.21.15.89:0/1568426558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22e40d2370 msgr2=0x7f22e40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.486+0000 7f22e9000700 1 --2- 172.21.15.89:0/1568426558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22e40d2370 0x7f22e40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.486+0000 7f22e9000700 1 -- 172.21.15.89:0/1568426558 >> v1:172.21.15.89:6789/0 conn(0x7f22e415d9d0 legacy=0x7f22e40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.486+0000 7f22e9000700 1 -- 172.21.15.89:0/1568426558 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e4161700 con 0x7f22e40d16a0 2024-04-02T07:36:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.486+0000 7f22ea400700 1 --2- 172.21.15.89:0/1568426558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22e40d2370 0x7f22e40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.486+0000 7f22e9000700 1 --2- 172.21.15.89:0/1568426558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22e40d16a0 0x7f22e40d1e30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f22d0009820 tx=0x7f22d0009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd7cf896f9589454 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:25.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.487+0000 7f22e3e00700 1 -- 172.21.15.89:0/1568426558 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d0005e30 con 0x7f22e40d16a0 2024-04-02T07:36:25.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.487+0000 7f22e3e00700 1 -- 172.21.15.89:0/1568426558 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d0006030 con 0x7f22e40d16a0 2024-04-02T07:36:25.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.487+0000 7f22e3e00700 1 -- 172.21.15.89:0/1568426558 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d00063c0 con 0x7f22e40d16a0 2024-04-02T07:36:25.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.487+0000 7f22ec400700 1 -- 172.21.15.89:0/1568426558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22e40d16a0 msgr2=0x7f22e40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.487+0000 7f22ec400700 1 --2- 172.21.15.89:0/1568426558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22e40d16a0 0x7f22e40d1e30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f22d0009820 tx=0x7f22d0009b30 comp rx=0 tx=0).stop 2024-04-02T07:36:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.487+0000 7f22ec400700 1 -- 172.21.15.89:0/1568426558 shutdown_connections 2024-04-02T07:36:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.487+0000 7f22ec400700 1 --2- 172.21.15.89:0/1568426558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22e40d2370 0x7f22e40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.487+0000 7f22ec400700 1 --2- 172.21.15.89:0/1568426558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22e40d16a0 0x7f22e40d1e30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.487+0000 7f22ec400700 1 -- 172.21.15.89:0/1568426558 >> 172.21.15.89:0/1568426558 conn(0x7f22e408aff0 msgr2=0x7f22e40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.487+0000 7f22ec400700 1 -- 172.21.15.89:0/1568426558 shutdown_connections 2024-04-02T07:36:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.487+0000 7f22ec400700 1 -- 172.21.15.89:0/1568426558 wait complete. 2024-04-02T07:36:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.489+0000 7f22ec400700 1 Processor -- start 2024-04-02T07:36:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.489+0000 7f22ec400700 1 -- start start 2024-04-02T07:36:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.490+0000 7f22ec400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22e40d16a0 0x7f22e414ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.490+0000 7f22ec400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22e40d2370 0x7f22e40b89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.490+0000 7f22ec400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22e415d9d0 0x7f22e40ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.490+0000 7f22ec400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e41f77e0 con 0x7f22e40d2370 2024-04-02T07:36:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.490+0000 7f22ec400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e40b8560 con 0x7f22e40d16a0 2024-04-02T07:36:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.490+0000 7f22ec400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e40b7c30 con 0x7f22e415d9d0 2024-04-02T07:36:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.490+0000 7f22e9000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22e40d2370 0x7f22e40b89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.490+0000 7f22e9000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22e40d2370 0x7f22e40b89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50872/0 (socket says 172.21.15.89:50872) 2024-04-02T07:36:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.490+0000 7f22e9000700 1 -- 172.21.15.89:0/2497963799 learned_addr learned my addr 172.21.15.89:0/2497963799 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.491+0000 7f22e9a00700 1 --2- 172.21.15.89:0/2497963799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22e40d16a0 0x7f22e414ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.491+0000 7f22e9000700 1 -- 172.21.15.89:0/2497963799 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22e415d9d0 msgr2=0x7f22e40ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.491+0000 7f22e9000700 1 --2- 172.21.15.89:0/2497963799 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22e415d9d0 0x7f22e40ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.491+0000 7f22e9000700 1 -- 172.21.15.89:0/2497963799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22e40d16a0 msgr2=0x7f22e414ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.491+0000 7f22e9000700 1 --2- 172.21.15.89:0/2497963799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22e40d16a0 0x7f22e414ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.491+0000 7f22e9000700 1 -- 172.21.15.89:0/2497963799 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22dc009a70 con 0x7f22e40d2370 2024-04-02T07:36:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.491+0000 7f22e9000700 1 --2- 172.21.15.89:0/2497963799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22e40d2370 0x7f22e40b89f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f22d0000c00 tx=0x7f22d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.491+0000 7f22e1600700 1 -- 172.21.15.89:0/2497963799 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d00063a0 con 0x7f22e40d2370 2024-04-02T07:36:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.491+0000 7f22e1600700 1 -- 172.21.15.89:0/2497963799 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d0020070 con 0x7f22e40d2370 2024-04-02T07:36:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.491+0000 7f22ec400700 1 -- 172.21.15.89:0/2497963799 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d0009430 con 0x7f22e40d2370 2024-04-02T07:36:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.491+0000 7f22e1600700 1 -- 172.21.15.89:0/2497963799 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d0006760 con 0x7f22e40d2370 2024-04-02T07:36:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.491+0000 7f22ec400700 1 -- 172.21.15.89:0/2497963799 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e40b81f0 con 0x7f22e40d2370 2024-04-02T07:36:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.493+0000 7f22ec400700 1 -- 172.21.15.89:0/2497963799 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e40d1ca0 con 0x7f22e40d2370 2024-04-02T07:36:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.494+0000 7f22e1600700 1 -- 172.21.15.89:0/2497963799 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f22d0022030 con 0x7f22e40d2370 2024-04-02T07:36:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.495+0000 7f22e1600700 1 --2- 172.21.15.89:0/2497963799 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22c406ac90 0x7f22c406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.495+0000 7f22e1600700 1 -- 172.21.15.89:0/2497963799 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f22d008abc0 con 0x7f22e40d2370 2024-04-02T07:36:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.495+0000 7f22e9a00700 1 --2- 172.21.15.89:0/2497963799 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22c406ac90 0x7f22c406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.495+0000 7f22e9a00700 1 --2- 172.21.15.89:0/2497963799 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22c406ac90 0x7f22c406d150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f22d80011e0 tx=0x7f22d8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.498+0000 7f22e1600700 1 -- 172.21.15.89:0/2497963799 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f22d0055e10 con 0x7f22e40d2370 2024-04-02T07:36:25.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.699+0000 7f22ec400700 1 -- 172.21.15.89:0/2497963799 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f22e4067720 con 0x7f22e40d2370 2024-04-02T07:36:25.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.699+0000 7f22e1600700 1 -- 172.21.15.89:0/2497963799 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f22d0059430 con 0x7f22e40d2370 2024-04-02T07:36:25.701 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:36:25.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.702+0000 7f22ec400700 1 -- 172.21.15.89:0/2497963799 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22c406ac90 msgr2=0x7f22c406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.702+0000 7f22ec400700 1 --2- 172.21.15.89:0/2497963799 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22c406ac90 0x7f22c406d150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f22d80011e0 tx=0x7f22d8009040 comp rx=0 tx=0).stop 2024-04-02T07:36:25.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.702+0000 7f22ec400700 1 -- 172.21.15.89:0/2497963799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22e40d2370 msgr2=0x7f22e40b89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.702+0000 7f22ec400700 1 --2- 172.21.15.89:0/2497963799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22e40d2370 0x7f22e40b89f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f22d0000c00 tx=0x7f22d0004750 comp rx=0 tx=0).stop 2024-04-02T07:36:25.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.702+0000 7f22ec400700 1 -- 172.21.15.89:0/2497963799 shutdown_connections 2024-04-02T07:36:25.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.702+0000 7f22ec400700 1 --2- 172.21.15.89:0/2497963799 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22c406ac90 0x7f22c406d150 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.702+0000 7f22ec400700 1 --2- 172.21.15.89:0/2497963799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22e40d16a0 0x7f22e414ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.702+0000 7f22ec400700 1 --2- 172.21.15.89:0/2497963799 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22e415d9d0 0x7f22e40ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.702+0000 7f22ec400700 1 --2- 172.21.15.89:0/2497963799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22e40d2370 0x7f22e40b89f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.702+0000 7f22ec400700 1 -- 172.21.15.89:0/2497963799 >> 172.21.15.89:0/2497963799 conn(0x7f22e408aff0 msgr2=0x7f22e40b5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:25.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.702+0000 7f22ec400700 1 -- 172.21.15.89:0/2497963799 shutdown_connections 2024-04-02T07:36:25.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.702+0000 7f22ec400700 1 -- 172.21.15.89:0/2497963799 wait complete. 2024-04-02T07:36:25.722 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:36:25.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.914+0000 7f3313800700 1 Processor -- start 2024-04-02T07:36:25.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.914+0000 7f3313800700 1 -- start start 2024-04-02T07:36:25.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.915+0000 7f3313800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f330c160f10 0x7f330c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.915+0000 7f3313800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f330c161590 0x7f330c161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.915+0000 7f3313800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f330c0ceda0 con 0x7f330c0cf230 2024-04-02T07:36:25.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.915+0000 7f3313800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f330c161f80 con 0x7f330c160f10 2024-04-02T07:36:25.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.915+0000 7f3313800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f330c1612f0 con 0x7f330c161590 2024-04-02T07:36:25.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.915+0000 7f3310e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f330c160f10 0x7f330c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.915+0000 7f3310e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f330c160f10 0x7f330c0d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50874/0 (socket says 172.21.15.89:50874) 2024-04-02T07:36:25.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.915+0000 7f3310e00700 1 -- 172.21.15.89:0/3158569190 learned_addr learned my addr 172.21.15.89:0/3158569190 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.916+0000 7f3310e00700 1 -- 172.21.15.89:0/3158569190 >> v2:172.21.15.113:3300/0 conn(0x7f330c161590 msgr2=0x7f330c161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.916+0000 7f3310e00700 1 --2- 172.21.15.89:0/3158569190 >> v2:172.21.15.113:3300/0 conn(0x7f330c161590 0x7f330c161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.916+0000 7f3310e00700 1 -- 172.21.15.89:0/3158569190 >> v1:172.21.15.113:6789/0 conn(0x7f330c0cf230 legacy=0x7f330c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.916+0000 7f3310e00700 1 -- 172.21.15.89:0/3158569190 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c0ce470 con 0x7f330c160f10 2024-04-02T07:36:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.916+0000 7f3310e00700 1 --2- 172.21.15.89:0/3158569190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c160f10 0x7f330c0d18b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f330c0d1030 tx=0x7f3300009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b5e55bd8f2cc01 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.916+0000 7f330b400700 1 -- 172.21.15.89:0/3158569190 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3300004230 con 0x7f330c160f10 2024-04-02T07:36:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.916+0000 7f330b400700 1 -- 172.21.15.89:0/3158569190 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3300004390 con 0x7f330c160f10 2024-04-02T07:36:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.916+0000 7f330b400700 1 -- 172.21.15.89:0/3158569190 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33000036a0 con 0x7f330c160f10 2024-04-02T07:36:25.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.917+0000 7f3313800700 1 -- 172.21.15.89:0/3158569190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c160f10 msgr2=0x7f330c0d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.917+0000 7f3313800700 1 --2- 172.21.15.89:0/3158569190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c160f10 0x7f330c0d18b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f330c0d1030 tx=0x7f3300009bd0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.917+0000 7f3313800700 1 -- 172.21.15.89:0/3158569190 shutdown_connections 2024-04-02T07:36:25.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.917+0000 7f3313800700 1 --2- 172.21.15.89:0/3158569190 >> v2:172.21.15.113:3300/0 conn(0x7f330c161590 0x7f330c161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.917+0000 7f3313800700 1 --2- 172.21.15.89:0/3158569190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c160f10 0x7f330c0d18b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.917+0000 7f3313800700 1 -- 172.21.15.89:0/3158569190 >> 172.21.15.89:0/3158569190 conn(0x7f330c08aff0 msgr2=0x7f330c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:25.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.917+0000 7f3313800700 1 -- 172.21.15.89:0/3158569190 shutdown_connections 2024-04-02T07:36:25.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.917+0000 7f3313800700 1 -- 172.21.15.89:0/3158569190 wait complete. 2024-04-02T07:36:25.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.919+0000 7f3313800700 1 Processor -- start 2024-04-02T07:36:25.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.919+0000 7f3313800700 1 -- start start 2024-04-02T07:36:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.919+0000 7f3313800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f330c0cf230 0x7f330c14b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.919+0000 7f3313800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f330c160f10 0x7f330c15ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.920+0000 7f3313800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c161590 0x7f330c15f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.920+0000 7f3313800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c14ba90 con 0x7f330c161590 2024-04-02T07:36:25.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.920+0000 7f3313800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c1562f0 con 0x7f330c0cf230 2024-04-02T07:36:25.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.920+0000 7f3313800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330c15f830 con 0x7f330c160f10 2024-04-02T07:36:25.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.920+0000 7f3311800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c161590 0x7f330c15f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.920+0000 7f3311800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c161590 0x7f330c15f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50884/0 (socket says 172.21.15.89:50884) 2024-04-02T07:36:25.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.920+0000 7f3311800700 1 -- 172.21.15.89:0/420945178 learned_addr learned my addr 172.21.15.89:0/420945178 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:25.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.920+0000 7f3310e00700 1 --2- 172.21.15.89:0/420945178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f330c0cf230 0x7f330c14b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.920+0000 7f330be00700 1 --2- 172.21.15.89:0/420945178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f330c160f10 0x7f330c15ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.920+0000 7f3311800700 1 -- 172.21.15.89:0/420945178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f330c160f10 msgr2=0x7f330c15ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.920+0000 7f3311800700 1 --2- 172.21.15.89:0/420945178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f330c160f10 0x7f330c15ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.920+0000 7f3311800700 1 -- 172.21.15.89:0/420945178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f330c0cf230 msgr2=0x7f330c14b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:25.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.920+0000 7f3311800700 1 --2- 172.21.15.89:0/420945178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f330c0cf230 0x7f330c14b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:25.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.920+0000 7f3311800700 1 -- 172.21.15.89:0/420945178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3300009550 con 0x7f330c161590 2024-04-02T07:36:25.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.921+0000 7f3311800700 1 --2- 172.21.15.89:0/420945178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c161590 0x7f330c15f210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f330400b970 tx=0x7f330400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:25.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.921+0000 7f3308c00700 1 -- 172.21.15.89:0/420945178 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330400d860 con 0x7f330c161590 2024-04-02T07:36:25.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.921+0000 7f3308c00700 1 -- 172.21.15.89:0/420945178 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330400d9e0 con 0x7f330c161590 2024-04-02T07:36:25.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.921+0000 7f3313800700 1 -- 172.21.15.89:0/420945178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f330c15fb10 con 0x7f330c161590 2024-04-02T07:36:25.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.921+0000 7f3308c00700 1 -- 172.21.15.89:0/420945178 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330400dd90 con 0x7f330c161590 2024-04-02T07:36:25.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.921+0000 7f3313800700 1 -- 172.21.15.89:0/420945178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330c156540 con 0x7f330c161590 2024-04-02T07:36:25.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.922+0000 7f3313800700 1 -- 172.21.15.89:0/420945178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330c156760 con 0x7f330c161590 2024-04-02T07:36:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.923+0000 7f3308c00700 1 -- 172.21.15.89:0/420945178 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f330400e430 con 0x7f330c161590 2024-04-02T07:36:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.924+0000 7f3308c00700 1 --2- 172.21.15.89:0/420945178 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32ec06ac90 0x7f32ec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.924+0000 7f3308c00700 1 -- 172.21.15.89:0/420945178 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3304089520 con 0x7f330c161590 2024-04-02T07:36:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.924+0000 7f3310e00700 1 --2- 172.21.15.89:0/420945178 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32ec06ac90 0x7f32ec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.925+0000 7f3310e00700 1 --2- 172.21.15.89:0/420945178 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32ec06ac90 0x7f32ec06d150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3300009130 tx=0x7f3300008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:25.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:25.928+0000 7f3308c00700 1 -- 172.21.15.89:0/420945178 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3304054670 con 0x7f330c161590 2024-04-02T07:36:26.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.133+0000 7f3313800700 1 -- 172.21.15.89:0/420945178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f330c067720 con 0x7f330c161590 2024-04-02T07:36:26.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.134+0000 7f3308c00700 1 -- 172.21.15.89:0/420945178 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f330401c090 con 0x7f330c161590 2024-04-02T07:36:26.136 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:26.136 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:54.238913+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:36:26.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.136+0000 7f3313800700 1 -- 172.21.15.89:0/420945178 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32ec06ac90 msgr2=0x7f32ec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:26.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.136+0000 7f3313800700 1 --2- 172.21.15.89:0/420945178 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32ec06ac90 0x7f32ec06d150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3300009130 tx=0x7f3300008030 comp rx=0 tx=0).stop 2024-04-02T07:36:26.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.136+0000 7f3313800700 1 -- 172.21.15.89:0/420945178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c161590 msgr2=0x7f330c15f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:26.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.136+0000 7f3313800700 1 --2- 172.21.15.89:0/420945178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c161590 0x7f330c15f210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f330400b970 tx=0x7f330400bc80 comp rx=0 tx=0).stop 2024-04-02T07:36:26.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.136+0000 7f3313800700 1 -- 172.21.15.89:0/420945178 shutdown_connections 2024-04-02T07:36:26.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.136+0000 7f3313800700 1 --2- 172.21.15.89:0/420945178 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32ec06ac90 0x7f32ec06d150 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.136+0000 7f3313800700 1 --2- 172.21.15.89:0/420945178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f330c0cf230 0x7f330c14b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.136+0000 7f3313800700 1 --2- 172.21.15.89:0/420945178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f330c161590 0x7f330c15f210 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.136+0000 7f3313800700 1 --2- 172.21.15.89:0/420945178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f330c160f10 0x7f330c15ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.137+0000 7f3313800700 1 -- 172.21.15.89:0/420945178 >> 172.21.15.89:0/420945178 conn(0x7f330c08aff0 msgr2=0x7f330c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:26.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.137+0000 7f3313800700 1 -- 172.21.15.89:0/420945178 shutdown_connections 2024-04-02T07:36:26.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.137+0000 7f3313800700 1 -- 172.21.15.89:0/420945178 wait complete. 2024-04-02T07:36:26.156 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:35:51.296561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:35:52.232882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:36:26.157 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:36:26.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.349+0000 7fce15600700 1 Processor -- start 2024-04-02T07:36:26.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.349+0000 7fce15600700 1 -- start start 2024-04-02T07:36:26.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.350+0000 7fce15600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce100d16a0 0x7fce100d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:26.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.350+0000 7fce15600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce10161420 con 0x7fce1015d9d0 2024-04-02T07:36:26.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.350+0000 7fce15600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce10161700 con 0x7fce100d2370 2024-04-02T07:36:26.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.350+0000 7fce15600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce100d2820 con 0x7fce100d16a0 2024-04-02T07:36:26.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.351+0000 7fce0ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fce1015d9d0 0x7fce100d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59870/0 (socket says 172.21.15.89:59870) 2024-04-02T07:36:26.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.351+0000 7fce0ea00700 1 -- 172.21.15.89:0/686240601 learned_addr learned my addr 172.21.15.89:0/686240601 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:26.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.351+0000 7fce0e000700 1 --2- 172.21.15.89:0/686240601 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce100d16a0 0x7fce100d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:26.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.351+0000 7fce0d600700 1 -- 172.21.15.89:0/686240601 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878067227 0 0) 0x7fce10161420 con 0x7fce1015d9d0 2024-04-02T07:36:26.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.351+0000 7fce0d600700 1 -- 172.21.15.89:0/686240601 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdf40035b0 con 0x7fce1015d9d0 2024-04-02T07:36:26.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.351+0000 7fce0d600700 1 -- 172.21.15.89:0/686240601 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332612684 0 0) 0x7fce10161700 con 0x7fce100d2370 2024-04-02T07:36:26.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.351+0000 7fce0d600700 1 -- 172.21.15.89:0/686240601 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce10161420 con 0x7fce100d2370 2024-04-02T07:36:26.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.351+0000 7fce0e000700 1 -- 172.21.15.89:0/686240601 >> v1:172.21.15.113:6789/0 conn(0x7fce100d2370 legacy=0x7fce100ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:26.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.351+0000 7fce0e000700 1 -- 172.21.15.89:0/686240601 >> v1:172.21.15.89:6789/0 conn(0x7fce1015d9d0 legacy=0x7fce100d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:26.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.351+0000 7fce0e000700 1 -- 172.21.15.89:0/686240601 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce101619e0 con 0x7fce100d16a0 2024-04-02T07:36:26.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.352+0000 7fce0e000700 1 --2- 172.21.15.89:0/686240601 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce100d16a0 0x7fce100d1e30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc009850 tx=0x7fcdfc009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba25bb329db601e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:26.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.352+0000 7fce0d600700 1 -- 172.21.15.89:0/686240601 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc005e30 con 0x7fce100d16a0 2024-04-02T07:36:26.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.352+0000 7fce0d600700 1 -- 172.21.15.89:0/686240601 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc006030 con 0x7fce100d16a0 2024-04-02T07:36:26.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.352+0000 7fce0d600700 1 -- 172.21.15.89:0/686240601 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc0063a0 con 0x7fce100d16a0 2024-04-02T07:36:26.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.352+0000 7fce15600700 1 -- 172.21.15.89:0/686240601 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce100d16a0 msgr2=0x7fce100d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:26.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.352+0000 7fce15600700 1 --2- 172.21.15.89:0/686240601 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce100d16a0 0x7fce100d1e30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc009850 tx=0x7fcdfc009b60 comp rx=0 tx=0).stop 2024-04-02T07:36:26.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.353+0000 7fce15600700 1 -- 172.21.15.89:0/686240601 shutdown_connections 2024-04-02T07:36:26.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.353+0000 7fce15600700 1 --2- 172.21.15.89:0/686240601 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce100d16a0 0x7fce100d1e30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.353+0000 7fce15600700 1 -- 172.21.15.89:0/686240601 >> 172.21.15.89:0/686240601 conn(0x7fce1008aff0 msgr2=0x7fce100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:26.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.353+0000 7fce15600700 1 -- 172.21.15.89:0/686240601 shutdown_connections 2024-04-02T07:36:26.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.353+0000 7fce15600700 1 -- 172.21.15.89:0/686240601 wait complete. 2024-04-02T07:36:26.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.354+0000 7fce15600700 1 Processor -- start 2024-04-02T07:36:26.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.354+0000 7fce15600700 1 -- start start 2024-04-02T07:36:26.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.355+0000 7fce15600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce100d16a0 0x7fce1015f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.355+0000 7fce15600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce100d2370 0x7fce1014e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.355+0000 7fce15600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce1015d9d0 0x7fce1014eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.355+0000 7fce15600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce101f6a20 con 0x7fce1015d9d0 2024-04-02T07:36:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.355+0000 7fce15600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce101f6cc0 con 0x7fce100d16a0 2024-04-02T07:36:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.355+0000 7fce15600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce101f6fa0 con 0x7fce100d2370 2024-04-02T07:36:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.356+0000 7fce0f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce1015d9d0 0x7fce1014eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.356+0000 7fce0f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce1015d9d0 0x7fce1014eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50886/0 (socket says 172.21.15.89:50886) 2024-04-02T07:36:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.356+0000 7fce0f400700 1 -- 172.21.15.89:0/4103213334 learned_addr learned my addr 172.21.15.89:0/4103213334 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.356+0000 7fce0ea00700 1 --2- 172.21.15.89:0/4103213334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce100d16a0 0x7fce1015f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.356+0000 7fce0e000700 1 --2- 172.21.15.89:0/4103213334 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce100d2370 0x7fce1014e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.356+0000 7fce0f400700 1 -- 172.21.15.89:0/4103213334 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce100d2370 msgr2=0x7fce1014e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.356+0000 7fce0f400700 1 --2- 172.21.15.89:0/4103213334 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce100d2370 0x7fce1014e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.356+0000 7fce0f400700 1 -- 172.21.15.89:0/4103213334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce100d16a0 msgr2=0x7fce1015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.356+0000 7fce0f400700 1 --2- 172.21.15.89:0/4103213334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce100d16a0 0x7fce1015f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.356+0000 7fce0f400700 1 -- 172.21.15.89:0/4103213334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc009430 con 0x7fce1015d9d0 2024-04-02T07:36:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.356+0000 7fce0f400700 1 --2- 172.21.15.89:0/4103213334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce1015d9d0 0x7fce1014eaa0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fce0800cc20 tx=0x7fce0800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.356+0000 7fce06a00700 1 -- 172.21.15.89:0/4103213334 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce08003c00 con 0x7fce1015d9d0 2024-04-02T07:36:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.357+0000 7fce06a00700 1 -- 172.21.15.89:0/4103213334 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce08003d60 con 0x7fce1015d9d0 2024-04-02T07:36:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.357+0000 7fce15600700 1 -- 172.21.15.89:0/4103213334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce101f7280 con 0x7fce1015d9d0 2024-04-02T07:36:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.357+0000 7fce06a00700 1 -- 172.21.15.89:0/4103213334 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0800aae0 con 0x7fce1015d9d0 2024-04-02T07:36:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.357+0000 7fce15600700 1 -- 172.21.15.89:0/4103213334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1015fe00 con 0x7fce1015d9d0 2024-04-02T07:36:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.358+0000 7fce15600700 1 -- 172.21.15.89:0/4103213334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1015fff0 con 0x7fce1015d9d0 2024-04-02T07:36:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.359+0000 7fce06a00700 1 -- 172.21.15.89:0/4103213334 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fce0800ac40 con 0x7fce1015d9d0 2024-04-02T07:36:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.359+0000 7fce06a00700 1 --2- 172.21.15.89:0/4103213334 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcdf006ac90 0x7fcdf006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.360+0000 7fce06a00700 1 -- 172.21.15.89:0/4103213334 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce08089410 con 0x7fce1015d9d0 2024-04-02T07:36:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.360+0000 7fce0ea00700 1 --2- 172.21.15.89:0/4103213334 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcdf006ac90 0x7fcdf006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:26.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.360+0000 7fce0ea00700 1 --2- 172.21.15.89:0/4103213334 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcdf006ac90 0x7fcdf006d150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fce00007fd0 tx=0x7fce00007dc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:26.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.364+0000 7fce06a00700 1 -- 172.21.15.89:0/4103213334 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fce08054550 con 0x7fce1015d9d0 2024-04-02T07:36:26.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.565+0000 7fce15600700 1 -- 172.21.15.89:0/4103213334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fce100d1ca0 con 0x7fce1015d9d0 2024-04-02T07:36:26.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.566+0000 7fce06a00700 1 -- 172.21.15.89:0/4103213334 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fce08057b70 con 0x7fce1015d9d0 2024-04-02T07:36:26.567 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:36:26.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.568+0000 7fce15600700 1 -- 172.21.15.89:0/4103213334 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcdf006ac90 msgr2=0x7fcdf006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:26.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.568+0000 7fce15600700 1 --2- 172.21.15.89:0/4103213334 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcdf006ac90 0x7fcdf006d150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fce00007fd0 tx=0x7fce00007dc0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.568+0000 7fce15600700 1 -- 172.21.15.89:0/4103213334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce1015d9d0 msgr2=0x7fce1014eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:26.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.568+0000 7fce15600700 1 --2- 172.21.15.89:0/4103213334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce1015d9d0 0x7fce1014eaa0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fce0800cc20 tx=0x7fce0800cf30 comp rx=0 tx=0).stop 2024-04-02T07:36:26.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.568+0000 7fce15600700 1 -- 172.21.15.89:0/4103213334 shutdown_connections 2024-04-02T07:36:26.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.568+0000 7fce15600700 1 --2- 172.21.15.89:0/4103213334 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcdf006ac90 0x7fcdf006d150 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.568+0000 7fce15600700 1 --2- 172.21.15.89:0/4103213334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce100d16a0 0x7fce1015f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.568+0000 7fce15600700 1 --2- 172.21.15.89:0/4103213334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce1015d9d0 0x7fce1014eaa0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.568+0000 7fce15600700 1 --2- 172.21.15.89:0/4103213334 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce100d2370 0x7fce1014e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.568+0000 7fce15600700 1 -- 172.21.15.89:0/4103213334 >> 172.21.15.89:0/4103213334 conn(0x7fce1008aff0 msgr2=0x7fce100d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:26.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.568+0000 7fce15600700 1 -- 172.21.15.89:0/4103213334 shutdown_connections 2024-04-02T07:36:26.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.568+0000 7fce15600700 1 -- 172.21.15.89:0/4103213334 wait complete. 2024-04-02T07:36:26.588 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:36:26.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.781+0000 7f809a800700 1 Processor -- start 2024-04-02T07:36:26.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.781+0000 7f809a800700 1 -- start start 2024-04-02T07:36:26.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.782+0000 7f809a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f809415d9d0 0x7f80940d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:26.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.782+0000 7f809a800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f80940d2540 0x7f80940ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:26.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.782+0000 7f809a800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8094161420 con 0x7f80940d16a0 2024-04-02T07:36:26.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.782+0000 7f809a800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f809415edc0 con 0x7f80940d2540 2024-04-02T07:36:26.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.782+0000 7f809a800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8094161700 con 0x7f809415d9d0 2024-04-02T07:36:26.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.783+0000 7f8093e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f80940d2540 0x7f80940ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:26.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.783+0000 7f8093400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f809415d9d0 0x7f80940d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:26.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.783+0000 7f8093400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f809415d9d0 0x7f80940d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:60238/0 (socket says 172.21.15.89:60238) 2024-04-02T07:36:26.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.783+0000 7f8093e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f80940d2540 0x7f80940ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:34934/0 (socket says 172.21.15.89:34934) 2024-04-02T07:36:26.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.783+0000 7f8093400700 1 -- 172.21.15.89:0/719532377 learned_addr learned my addr 172.21.15.89:0/719532377 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:26.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.783+0000 7f8093400700 1 -- 172.21.15.89:0/719532377 >> v2:172.21.15.113:3300/0 conn(0x7f80940d2540 msgr2=0x7f80940ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:26.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.783+0000 7f8093400700 1 --2- 172.21.15.89:0/719532377 >> v2:172.21.15.113:3300/0 conn(0x7f80940d2540 0x7f80940ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.783+0000 7f8093400700 1 -- 172.21.15.89:0/719532377 >> v1:172.21.15.113:6789/0 conn(0x7f80940d16a0 legacy=0x7f80940d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:26.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.783+0000 7f8093400700 1 -- 172.21.15.89:0/719532377 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80941619e0 con 0x7f809415d9d0 2024-04-02T07:36:26.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.783+0000 7f8092000700 1 -- 172.21.15.89:0/719532377 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735819800 0 0) 0x7f8094161420 con 0x7f80940d16a0 2024-04-02T07:36:26.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.783+0000 7f8093400700 1 --2- 172.21.15.89:0/719532377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f809415d9d0 0x7f80940d3c30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8084009920 tx=0x7f8084009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62802ea010eb0788 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:26.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.784+0000 7f8092000700 1 -- 172.21.15.89:0/719532377 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8084011830 con 0x7f809415d9d0 2024-04-02T07:36:26.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.784+0000 7f8092000700 1 -- 172.21.15.89:0/719532377 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8084011990 con 0x7f809415d9d0 2024-04-02T07:36:26.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.784+0000 7f8092000700 1 -- 172.21.15.89:0/719532377 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8084011d20 con 0x7f809415d9d0 2024-04-02T07:36:26.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.784+0000 7f809a800700 1 -- 172.21.15.89:0/719532377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f809415d9d0 msgr2=0x7f80940d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:26.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.784+0000 7f809a800700 1 --2- 172.21.15.89:0/719532377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f809415d9d0 0x7f80940d3c30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8084009920 tx=0x7f8084009c30 comp rx=0 tx=0).stop 2024-04-02T07:36:26.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.784+0000 7f809a800700 1 -- 172.21.15.89:0/719532377 shutdown_connections 2024-04-02T07:36:26.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.784+0000 7f809a800700 1 --2- 172.21.15.89:0/719532377 >> v2:172.21.15.113:3300/0 conn(0x7f80940d2540 0x7f80940ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.784+0000 7f809a800700 1 --2- 172.21.15.89:0/719532377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f809415d9d0 0x7f80940d3c30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.784+0000 7f809a800700 1 -- 172.21.15.89:0/719532377 >> 172.21.15.89:0/719532377 conn(0x7f809408aff0 msgr2=0x7f80940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:26.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.784+0000 7f809a800700 1 -- 172.21.15.89:0/719532377 shutdown_connections 2024-04-02T07:36:26.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.784+0000 7f809a800700 1 -- 172.21.15.89:0/719532377 wait complete. 2024-04-02T07:36:26.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.786+0000 7f809a800700 1 Processor -- start 2024-04-02T07:36:26.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.786+0000 7f809a800700 1 -- start start 2024-04-02T07:36:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.787+0000 7f809a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80940d16a0 0x7f80940b6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.787+0000 7f809a800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80940d2540 0x7f80941e9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.787+0000 7f809a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f809415d9d0 0x7f809414f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.787+0000 7f809a800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80941e9690 con 0x7f809415d9d0 2024-04-02T07:36:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.787+0000 7f809a800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80940b6ff0 con 0x7f80940d2540 2024-04-02T07:36:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.787+0000 7f809a800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80940b72d0 con 0x7f80940d16a0 2024-04-02T07:36:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.787+0000 7f8093e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f809415d9d0 0x7f809414f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.787+0000 7f8093400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80940d16a0 0x7f80940b6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.787+0000 7f8092a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80940d2540 0x7f80941e9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.787+0000 7f8093e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f809415d9d0 0x7f809414f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50898/0 (socket says 172.21.15.89:50898) 2024-04-02T07:36:26.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.787+0000 7f8093e00700 1 -- 172.21.15.89:0/89405061 learned_addr learned my addr 172.21.15.89:0/89405061 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:26.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.788+0000 7f8093400700 1 -- 172.21.15.89:0/89405061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80940d2540 msgr2=0x7f80941e9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:26.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.788+0000 7f8093400700 1 --2- 172.21.15.89:0/89405061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80940d2540 0x7f80941e9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.788+0000 7f8093400700 1 -- 172.21.15.89:0/89405061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f809415d9d0 msgr2=0x7f809414f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:26.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.788+0000 7f8093400700 1 --2- 172.21.15.89:0/89405061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f809415d9d0 0x7f809414f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:26.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.788+0000 7f8093400700 1 -- 172.21.15.89:0/89405061 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8084009550 con 0x7f80940d16a0 2024-04-02T07:36:26.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.788+0000 7f8093e00700 1 --2- 172.21.15.89:0/89405061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f809415d9d0 0x7f809414f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:26.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.788+0000 7f8093400700 1 --2- 172.21.15.89:0/89405061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80940d16a0 0x7f80940b6a00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8084016000 tx=0x7f8084004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:26.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.788+0000 7f808b400700 1 -- 172.21.15.89:0/89405061 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8084004510 con 0x7f80940d16a0 2024-04-02T07:36:26.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.788+0000 7f808b400700 1 -- 172.21.15.89:0/89405061 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808401e070 con 0x7f80940d16a0 2024-04-02T07:36:26.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.788+0000 7f809a800700 1 -- 172.21.15.89:0/89405061 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80940b75b0 con 0x7f80940d16a0 2024-04-02T07:36:26.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.789+0000 7f808b400700 1 -- 172.21.15.89:0/89405061 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80840038c0 con 0x7f80940d16a0 2024-04-02T07:36:26.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.789+0000 7f809a800700 1 -- 172.21.15.89:0/89405061 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80940b7b70 con 0x7f80940d16a0 2024-04-02T07:36:26.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.790+0000 7f809a800700 1 -- 172.21.15.89:0/89405061 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80940d1ca0 con 0x7f80940d16a0 2024-04-02T07:36:26.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.791+0000 7f808b400700 1 -- 172.21.15.89:0/89405061 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8084003a20 con 0x7f80940d16a0 2024-04-02T07:36:26.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.792+0000 7f808b400700 1 --2- 172.21.15.89:0/89405061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f807406ac90 0x7f807406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:26.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.792+0000 7f808b400700 1 -- 172.21.15.89:0/89405061 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f808408a2c0 con 0x7f80940d16a0 2024-04-02T07:36:26.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.792+0000 7f8092a00700 1 --2- 172.21.15.89:0/89405061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f807406ac90 0x7f807406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:26.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.792+0000 7f8092a00700 1 --2- 172.21.15.89:0/89405061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f807406ac90 0x7f807406d150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f809406cfe0 tx=0x7f8080009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:26.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.797+0000 7f808b400700 1 -- 172.21.15.89:0/89405061 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8084055400 con 0x7f80940d16a0 2024-04-02T07:36:27.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.998+0000 7f809a800700 1 -- 172.21.15.89:0/89405061 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f80940d14c0 con 0x7f80940d16a0 2024-04-02T07:36:27.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:26.999+0000 7f808b400700 1 -- 172.21.15.89:0/89405061 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8084058a20 con 0x7f80940d16a0 2024-04-02T07:36:27.001 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:36:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.002+0000 7f809a800700 1 -- 172.21.15.89:0/89405061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f807406ac90 msgr2=0x7f807406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.002+0000 7f809a800700 1 --2- 172.21.15.89:0/89405061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f807406ac90 0x7f807406d150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f809406cfe0 tx=0x7f8080009040 comp rx=0 tx=0).stop 2024-04-02T07:36:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.002+0000 7f809a800700 1 -- 172.21.15.89:0/89405061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80940d16a0 msgr2=0x7f80940b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.002+0000 7f809a800700 1 --2- 172.21.15.89:0/89405061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80940d16a0 0x7f80940b6a00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8084016000 tx=0x7f8084004750 comp rx=0 tx=0).stop 2024-04-02T07:36:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.002+0000 7f809a800700 1 -- 172.21.15.89:0/89405061 shutdown_connections 2024-04-02T07:36:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.002+0000 7f809a800700 1 --2- 172.21.15.89:0/89405061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f807406ac90 0x7f807406d150 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.002+0000 7f809a800700 1 --2- 172.21.15.89:0/89405061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f809415d9d0 0x7f809414f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.002+0000 7f809a800700 1 --2- 172.21.15.89:0/89405061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80940d16a0 0x7f80940b6a00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.002+0000 7f809a800700 1 --2- 172.21.15.89:0/89405061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80940d2540 0x7f80941e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.002+0000 7f809a800700 1 -- 172.21.15.89:0/89405061 >> 172.21.15.89:0/89405061 conn(0x7f809408aff0 msgr2=0x7f80940b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.002+0000 7f809a800700 1 -- 172.21.15.89:0/89405061 shutdown_connections 2024-04-02T07:36:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.002+0000 7f809a800700 1 -- 172.21.15.89:0/89405061 wait complete. 2024-04-02T07:36:27.022 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:36:27.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.213+0000 7f4256a00700 1 Processor -- start 2024-04-02T07:36:27.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.213+0000 7f4256a00700 1 -- start start 2024-04-02T07:36:27.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.214+0000 7f4256a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42501523b0 0x7f425006ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.215+0000 7f4256a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4250152790 0x7f425006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.215+0000 7f4256a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4250072c00 con 0x7f42500ade80 2024-04-02T07:36:27.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.215+0000 7f4256a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f425015c560 con 0x7f4250152790 2024-04-02T07:36:27.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.215+0000 7f4256a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425015bc70 con 0x7f42501523b0 2024-04-02T07:36:27.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.215+0000 7f424f400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4250152790 0x7f425006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:27.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.215+0000 7f424fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42501523b0 0x7f425006ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:27.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.215+0000 7f424fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42501523b0 0x7f425006ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:60244/0 (socket says 172.21.15.89:60244) 2024-04-02T07:36:27.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.215+0000 7f4254a00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f42500ade80 0x7f42500ae310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:33896/0 (socket says 172.21.15.89:33896) 2024-04-02T07:36:27.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.215+0000 7f424fe00700 1 -- 172.21.15.89:0/2891024250 learned_addr learned my addr 172.21.15.89:0/2891024250 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:27.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.216+0000 7f424ea00700 1 -- 172.21.15.89:0/2891024250 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014931470 0 0) 0x7f4250072c00 con 0x7f42500ade80 2024-04-02T07:36:27.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.216+0000 7f424fe00700 1 -- 172.21.15.89:0/2891024250 >> v2:172.21.15.113:3300/0 conn(0x7f4250152790 msgr2=0x7f425006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:27.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.216+0000 7f424fe00700 1 --2- 172.21.15.89:0/2891024250 >> v2:172.21.15.113:3300/0 conn(0x7f4250152790 0x7f425006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.216+0000 7f424fe00700 1 -- 172.21.15.89:0/2891024250 >> v1:172.21.15.113:6789/0 conn(0x7f42500ade80 legacy=0x7f42500ae310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:27.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.216+0000 7f424fe00700 1 -- 172.21.15.89:0/2891024250 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4250070bd0 con 0x7f42501523b0 2024-04-02T07:36:27.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.216+0000 7f424fe00700 1 --2- 172.21.15.89:0/2891024250 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42501523b0 0x7f425006ab80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f42400098f0 tx=0x7f4240009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff23ab1580df4c8f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:27.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.216+0000 7f424ea00700 1 -- 172.21.15.89:0/2891024250 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4240011830 con 0x7f42501523b0 2024-04-02T07:36:27.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.216+0000 7f424ea00700 1 -- 172.21.15.89:0/2891024250 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4240011990 con 0x7f42501523b0 2024-04-02T07:36:27.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.216+0000 7f424ea00700 1 -- 172.21.15.89:0/2891024250 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4240011d20 con 0x7f42501523b0 2024-04-02T07:36:27.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.217+0000 7f4256a00700 1 -- 172.21.15.89:0/2891024250 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42501523b0 msgr2=0x7f425006ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:27.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.217+0000 7f4256a00700 1 --2- 172.21.15.89:0/2891024250 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42501523b0 0x7f425006ab80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f42400098f0 tx=0x7f4240009c00 comp rx=0 tx=0).stop 2024-04-02T07:36:27.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.217+0000 7f4256a00700 1 -- 172.21.15.89:0/2891024250 shutdown_connections 2024-04-02T07:36:27.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.217+0000 7f4256a00700 1 --2- 172.21.15.89:0/2891024250 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42501523b0 0x7f425006ab80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.217+0000 7f4256a00700 1 --2- 172.21.15.89:0/2891024250 >> v2:172.21.15.113:3300/0 conn(0x7f4250152790 0x7f425006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.217+0000 7f4256a00700 1 -- 172.21.15.89:0/2891024250 >> 172.21.15.89:0/2891024250 conn(0x7f42500b1230 msgr2=0x7f42500b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:27.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.217+0000 7f4256a00700 1 -- 172.21.15.89:0/2891024250 shutdown_connections 2024-04-02T07:36:27.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.217+0000 7f4256a00700 1 -- 172.21.15.89:0/2891024250 wait complete. 2024-04-02T07:36:27.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.219+0000 7f4256a00700 1 Processor -- start 2024-04-02T07:36:27.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.219+0000 7f4256a00700 1 -- start start 2024-04-02T07:36:27.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.220+0000 7f4256a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42500ade80 0x7f425009ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.220+0000 7f4256a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42501523b0 0x7f42500992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f4256a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4250152790 0x7f42501f2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f4256a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425009a780 con 0x7f42500ade80 2024-04-02T07:36:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f4256a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4250099b70 con 0x7f42501523b0 2024-04-02T07:36:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f4256a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4250099e50 con 0x7f4250152790 2024-04-02T07:36:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f424fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42500ade80 0x7f425009ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f424fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42500ade80 0x7f425009ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50902/0 (socket says 172.21.15.89:50902) 2024-04-02T07:36:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f424fe00700 1 -- 172.21.15.89:0/679527782 learned_addr learned my addr 172.21.15.89:0/679527782 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f424f400700 1 --2- 172.21.15.89:0/679527782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42501523b0 0x7f42500992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f4254a00700 1 --2- 172.21.15.89:0/679527782 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4250152790 0x7f42501f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f424fe00700 1 -- 172.21.15.89:0/679527782 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4250152790 msgr2=0x7f42501f2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f424fe00700 1 --2- 172.21.15.89:0/679527782 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4250152790 0x7f42501f2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f424fe00700 1 -- 172.21.15.89:0/679527782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42501523b0 msgr2=0x7f42500992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f424fe00700 1 --2- 172.21.15.89:0/679527782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42501523b0 0x7f42500992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f424fe00700 1 -- 172.21.15.89:0/679527782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4240009550 con 0x7f42500ade80 2024-04-02T07:36:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.221+0000 7f424fe00700 1 --2- 172.21.15.89:0/679527782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42500ade80 0x7f425009ac10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4240009c40 tx=0x7f4240004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.222+0000 7f4247e00700 1 -- 172.21.15.89:0/679527782 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42400045d0 con 0x7f42500ade80 2024-04-02T07:36:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.222+0000 7f4247e00700 1 -- 172.21.15.89:0/679527782 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f424001f070 con 0x7f42500ade80 2024-04-02T07:36:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.222+0000 7f4256a00700 1 -- 172.21.15.89:0/679527782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425009a130 con 0x7f42500ade80 2024-04-02T07:36:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.222+0000 7f4247e00700 1 -- 172.21.15.89:0/679527782 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42400038c0 con 0x7f42500ade80 2024-04-02T07:36:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.222+0000 7f4256a00700 1 -- 172.21.15.89:0/679527782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4250099820 con 0x7f42500ade80 2024-04-02T07:36:27.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.223+0000 7f4256a00700 1 -- 172.21.15.89:0/679527782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425009aa60 con 0x7f42500ade80 2024-04-02T07:36:27.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.224+0000 7f4247e00700 1 -- 172.21.15.89:0/679527782 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4240003a20 con 0x7f42500ade80 2024-04-02T07:36:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.225+0000 7f4247e00700 1 --2- 172.21.15.89:0/679527782 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f423006ac90 0x7f423006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.225+0000 7f4247e00700 1 -- 172.21.15.89:0/679527782 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4240021070 con 0x7f42500ade80 2024-04-02T07:36:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.225+0000 7f424f400700 1 --2- 172.21.15.89:0/679527782 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f423006ac90 0x7f423006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:27.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.226+0000 7f424f400700 1 --2- 172.21.15.89:0/679527782 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f423006ac90 0x7f423006d150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f425014f8b0 tx=0x7f423c009210 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:27.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.230+0000 7f4247e00700 1 -- 172.21.15.89:0/679527782 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4240055e10 con 0x7f42500ade80 2024-04-02T07:36:27.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.430+0000 7f4256a00700 1 -- 172.21.15.89:0/679527782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4250067720 con 0x7f42500ade80 2024-04-02T07:36:27.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.430+0000 7f4247e00700 1 -- 172.21.15.89:0/679527782 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f424002b020 con 0x7f42500ade80 2024-04-02T07:36:27.432 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:36:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.433+0000 7f4256a00700 1 -- 172.21.15.89:0/679527782 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f423006ac90 msgr2=0x7f423006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.433+0000 7f4256a00700 1 --2- 172.21.15.89:0/679527782 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f423006ac90 0x7f423006d150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f425014f8b0 tx=0x7f423c009210 comp rx=0 tx=0).stop 2024-04-02T07:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.433+0000 7f4256a00700 1 -- 172.21.15.89:0/679527782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42500ade80 msgr2=0x7f425009ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.433+0000 7f4256a00700 1 --2- 172.21.15.89:0/679527782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42500ade80 0x7f425009ac10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4240009c40 tx=0x7f4240004750 comp rx=0 tx=0).stop 2024-04-02T07:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.433+0000 7f4256a00700 1 -- 172.21.15.89:0/679527782 shutdown_connections 2024-04-02T07:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.433+0000 7f4256a00700 1 --2- 172.21.15.89:0/679527782 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f423006ac90 0x7f423006d150 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.433+0000 7f4256a00700 1 --2- 172.21.15.89:0/679527782 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4250152790 0x7f42501f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.433+0000 7f4256a00700 1 --2- 172.21.15.89:0/679527782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42500ade80 0x7f425009ac10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.433+0000 7f4256a00700 1 --2- 172.21.15.89:0/679527782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42501523b0 0x7f42500992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.433+0000 7f4256a00700 1 -- 172.21.15.89:0/679527782 >> 172.21.15.89:0/679527782 conn(0x7f42500b1230 msgr2=0x7f4250071f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.433+0000 7f4256a00700 1 -- 172.21.15.89:0/679527782 shutdown_connections 2024-04-02T07:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.433+0000 7f4256a00700 1 -- 172.21.15.89:0/679527782 wait complete. 2024-04-02T07:36:27.452 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:36:27.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.641+0000 7f956ae00700 1 Processor -- start 2024-04-02T07:36:27.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.641+0000 7f956ae00700 1 -- start start 2024-04-02T07:36:27.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.642+0000 7f956ae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95640d18b0 0x7f95640cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.642+0000 7f956ae00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9564161590 0x7f9564161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.643+0000 7f956ae00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95640ceda0 con 0x7f9564160f10 2024-04-02T07:36:27.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.643+0000 7f956ae00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95640d1d30 con 0x7f9564161590 2024-04-02T07:36:27.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.643+0000 7f956ae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9564161f60 con 0x7f95640d18b0 2024-04-02T07:36:27.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.643+0000 7f9563e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9564160f10 0x7f95640d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59876/0 (socket says 172.21.15.89:59876) 2024-04-02T07:36:27.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.643+0000 7f9563e00700 1 -- 172.21.15.89:0/1316218519 learned_addr learned my addr 172.21.15.89:0/1316218519 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:27.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.643+0000 7f9563400700 1 -- 172.21.15.89:0/1316218519 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712494223 0 0) 0x7f95640ceda0 con 0x7f9564160f10 2024-04-02T07:36:27.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.643+0000 7f9563400700 1 -- 172.21.15.89:0/1316218519 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95480035b0 con 0x7f9564160f10 2024-04-02T07:36:27.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.643+0000 7f9568e00700 1 --2- 172.21.15.89:0/1316218519 >> v2:172.21.15.113:3300/0 conn(0x7f9564161590 0x7f9564161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:27.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.643+0000 7f955be00700 1 --2- 172.21.15.89:0/1316218519 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95640d18b0 0x7f95640cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.643+0000 7f9563400700 1 -- 172.21.15.89:0/1316218519 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3176347553 0 0) 0x7f95480035b0 con 0x7f9564160f10 2024-04-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.643+0000 7f9563400700 1 -- 172.21.15.89:0/1316218519 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95640d18b0 msgr2=0x7f95640cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9563400700 1 --2- 172.21.15.89:0/1316218519 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95640d18b0 0x7f95640cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9563400700 1 -- 172.21.15.89:0/1316218519 >> v2:172.21.15.113:3300/0 conn(0x7f9564161590 msgr2=0x7f9564161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9563400700 1 --2- 172.21.15.89:0/1316218519 >> v2:172.21.15.113:3300/0 conn(0x7f9564161590 0x7f9564161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9563400700 1 -- 172.21.15.89:0/1316218519 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95641612f0 con 0x7f9564160f10 2024-04-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9563400700 1 -- 172.21.15.89:0/1316218519 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f9554001670 con 0x7f9564160f10 2024-04-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f955be00700 1 --2- 172.21.15.89:0/1316218519 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95640d18b0 0x7f95640cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9568e00700 1 --2- 172.21.15.89:0/1316218519 >> v2:172.21.15.113:3300/0 conn(0x7f9564161590 0x7f9564161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9563400700 1 -- 172.21.15.89:0/1316218519 >> v1:172.21.15.89:6789/0 conn(0x7f9564160f10 legacy=0x7f95640d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9563400700 1 --2- 172.21.15.89:0/1316218519 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9548003a00 0x7f9548003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9563400700 1 --2- 172.21.15.89:0/1316218519 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95640d18b0 0x7f9548004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9563400700 1 --2- 172.21.15.89:0/1316218519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9548004d90 0x7f9548005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9563400700 1 -- 172.21.15.89:0/1316218519 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95640d1d30 con 0x7f9548004d90 2024-04-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9563400700 1 -- 172.21.15.89:0/1316218519 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9564161f60 con 0x7f9548003a00 2024-04-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9563400700 1 -- 172.21.15.89:0/1316218519 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954800bb70 con 0x7f95640d18b0 2024-04-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9568e00700 1 --2- 172.21.15.89:0/1316218519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9548004d90 0x7f9548005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9563e00700 1 --2- 172.21.15.89:0/1316218519 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9548003a00 0x7f9548003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f955be00700 1 --2- 172.21.15.89:0/1316218519 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95640d18b0 0x7f9548004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9568e00700 1 -- 172.21.15.89:0/1316218519 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95640d18b0 msgr2=0x7f9548004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9568e00700 1 --2- 172.21.15.89:0/1316218519 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95640d18b0 0x7f9548004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9568e00700 1 -- 172.21.15.89:0/1316218519 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9548003a00 msgr2=0x7f9548003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9568e00700 1 --2- 172.21.15.89:0/1316218519 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9548003a00 0x7f9548003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.644+0000 7f9568e00700 1 -- 172.21.15.89:0/1316218519 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95480035b0 con 0x7f9548004d90 2024-04-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.645+0000 7f9568e00700 1 --2- 172.21.15.89:0/1316218519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9548004d90 0x7f9548005210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f955c0073b0 tx=0x7f9548005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8ea42e5f3f66710 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.645+0000 7f9563400700 1 -- 172.21.15.89:0/1316218519 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c014400 con 0x7f9548004d90 2024-04-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.645+0000 7f9563400700 1 -- 172.21.15.89:0/1316218519 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955c014560 con 0x7f9548004d90 2024-04-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.645+0000 7f9563400700 1 -- 172.21.15.89:0/1316218519 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c0148f0 con 0x7f9548004d90 2024-04-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.645+0000 7f956ae00700 1 -- 172.21.15.89:0/1316218519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9548004d90 msgr2=0x7f9548005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.645+0000 7f956ae00700 1 --2- 172.21.15.89:0/1316218519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9548004d90 0x7f9548005210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f955c0073b0 tx=0x7f9548005880 comp rx=0 tx=0).stop 2024-04-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.645+0000 7f9568e00700 1 -- 172.21.15.89:0/1316218519 reap_dead start 2024-04-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.646+0000 7f956ae00700 1 -- 172.21.15.89:0/1316218519 shutdown_connections 2024-04-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.646+0000 7f956ae00700 1 -- 172.21.15.89:0/1316218519 >> 172.21.15.89:0/1316218519 conn(0x7f956408aff0 msgr2=0x7f95640d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.646+0000 7f956ae00700 1 -- 172.21.15.89:0/1316218519 shutdown_connections 2024-04-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.646+0000 7f956ae00700 1 -- 172.21.15.89:0/1316218519 wait complete. 2024-04-02T07:36:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.647+0000 7f956ae00700 1 Processor -- start 2024-04-02T07:36:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.647+0000 7f956ae00700 1 -- start start 2024-04-02T07:36:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.648+0000 7f956ae00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9548004d90 0x7f95641f5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.648+0000 7f956ae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95641f3e70 0x7f956415d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.648+0000 7f956ae00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f956415d560 0x7f956415f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.648+0000 7f956ae00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956415dae0 con 0x7f9548004d90 2024-04-02T07:36:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.648+0000 7f956ae00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956415ddc0 con 0x7f956415d560 2024-04-02T07:36:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.648+0000 7f956ae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956415e0a0 con 0x7f95641f3e70 2024-04-02T07:36:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.649+0000 7f9563e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9548004d90 0x7f95641f5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.649+0000 7f9563e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9548004d90 0x7f95641f5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50930/0 (socket says 172.21.15.89:50930) 2024-04-02T07:36:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.649+0000 7f9563e00700 1 -- 172.21.15.89:0/4266330793 learned_addr learned my addr 172.21.15.89:0/4266330793 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.649+0000 7f9563e00700 1 -- 172.21.15.89:0/4266330793 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95641f3e70 msgr2=0x7f956415d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.649+0000 7f9563e00700 1 --2- 172.21.15.89:0/4266330793 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95641f3e70 0x7f956415d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.649+0000 7f9563e00700 1 -- 172.21.15.89:0/4266330793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f956415d560 msgr2=0x7f956415f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.649+0000 7f9563e00700 1 --2- 172.21.15.89:0/4266330793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f956415d560 0x7f956415f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.649+0000 7f9563e00700 1 -- 172.21.15.89:0/4266330793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956415e380 con 0x7f9548004d90 2024-04-02T07:36:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.649+0000 7f9563e00700 1 --2- 172.21.15.89:0/4266330793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9548004d90 0x7f95641f5ce0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9554001d70 tx=0x7f9554007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.649+0000 7f9560c00700 1 -- 172.21.15.89:0/4266330793 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955400f690 con 0x7f9548004d90 2024-04-02T07:36:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.649+0000 7f9560c00700 1 -- 172.21.15.89:0/4266330793 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955400f7f0 con 0x7f9548004d90 2024-04-02T07:36:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.649+0000 7f9560c00700 1 -- 172.21.15.89:0/4266330793 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955400fba0 con 0x7f9548004d90 2024-04-02T07:36:27.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.649+0000 7f956ae00700 1 -- 172.21.15.89:0/4266330793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956415e660 con 0x7f9548004d90 2024-04-02T07:36:27.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.650+0000 7f956ae00700 1 -- 172.21.15.89:0/4266330793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956415fa90 con 0x7f9548004d90 2024-04-02T07:36:27.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.651+0000 7f956ae00700 1 -- 172.21.15.89:0/4266330793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9564067720 con 0x7f9548004d90 2024-04-02T07:36:27.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.651+0000 7f9560c00700 1 -- 172.21.15.89:0/4266330793 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f955400fd00 con 0x7f9548004d90 2024-04-02T07:36:27.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.653+0000 7f9560c00700 1 --2- 172.21.15.89:0/4266330793 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f954006acc0 0x7f954006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:27.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.653+0000 7f9560c00700 1 -- 172.21.15.89:0/4266330793 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9554089350 con 0x7f9548004d90 2024-04-02T07:36:27.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.653+0000 7f955be00700 1 --2- 172.21.15.89:0/4266330793 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f954006acc0 0x7f954006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:27.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.654+0000 7f955be00700 1 --2- 172.21.15.89:0/4266330793 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f954006acc0 0x7f954006d180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f956415ce10 tx=0x7f954c004040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:27.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.657+0000 7f9560c00700 1 -- 172.21.15.89:0/4266330793 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9554054520 con 0x7f9548004d90 2024-04-02T07:36:27.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.861+0000 7f956ae00700 1 -- 172.21.15.89:0/4266330793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f956408b760 con 0x7f9548004d90 2024-04-02T07:36:27.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.862+0000 7f9560c00700 1 -- 172.21.15.89:0/4266330793 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f9554057b40 con 0x7f9548004d90 2024-04-02T07:36:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:27.864 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:54.238913+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:36:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.864+0000 7f956ae00700 1 -- 172.21.15.89:0/4266330793 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f954006acc0 msgr2=0x7f954006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.864+0000 7f956ae00700 1 --2- 172.21.15.89:0/4266330793 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f954006acc0 0x7f954006d180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f956415ce10 tx=0x7f954c004040 comp rx=0 tx=0).stop 2024-04-02T07:36:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.864+0000 7f956ae00700 1 -- 172.21.15.89:0/4266330793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9548004d90 msgr2=0x7f95641f5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.864+0000 7f956ae00700 1 --2- 172.21.15.89:0/4266330793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9548004d90 0x7f95641f5ce0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9554001d70 tx=0x7f9554007620 comp rx=0 tx=0).stop 2024-04-02T07:36:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.864+0000 7f956ae00700 1 -- 172.21.15.89:0/4266330793 shutdown_connections 2024-04-02T07:36:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.864+0000 7f956ae00700 1 --2- 172.21.15.89:0/4266330793 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f954006acc0 0x7f954006d180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.864+0000 7f956ae00700 1 --2- 172.21.15.89:0/4266330793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f956415d560 0x7f956415f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.864+0000 7f956ae00700 1 --2- 172.21.15.89:0/4266330793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9548004d90 0x7f95641f5ce0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.864+0000 7f956ae00700 1 --2- 172.21.15.89:0/4266330793 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95641f3e70 0x7f956415d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.864+0000 7f956ae00700 1 -- 172.21.15.89:0/4266330793 >> 172.21.15.89:0/4266330793 conn(0x7f956408aff0 msgr2=0x7f95640caeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.865+0000 7f956ae00700 1 -- 172.21.15.89:0/4266330793 shutdown_connections 2024-04-02T07:36:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:27.865+0000 7f956ae00700 1 -- 172.21.15.89:0/4266330793 wait complete. 2024-04-02T07:36:27.885 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:35:51.296561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:35:52.232882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:36:27.886 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:36:28.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.078+0000 7fb415c00700 1 Processor -- start 2024-04-02T07:36:28.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.078+0000 7fb415c00700 1 -- start start 2024-04-02T07:36:28.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.079+0000 7fb415c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4100d16a0 0x7fb4100d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.080+0000 7fb415c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb410161420 con 0x7fb41015d9d0 2024-04-02T07:36:28.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.080+0000 7fb415c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb410161700 con 0x7fb4100d2370 2024-04-02T07:36:28.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.080+0000 7fb415c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4100d2820 con 0x7fb4100d16a0 2024-04-02T07:36:28.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.080+0000 7fb40ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb41015d9d0 0x7fb4100d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59886/0 (socket says 172.21.15.89:59886) 2024-04-02T07:36:28.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.080+0000 7fb40ea00700 1 -- 172.21.15.89:0/1866418241 learned_addr learned my addr 172.21.15.89:0/1866418241 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:28.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.080+0000 7fb40d600700 1 -- 172.21.15.89:0/1866418241 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531329438 0 0) 0x7fb410161420 con 0x7fb41015d9d0 2024-04-02T07:36:28.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.080+0000 7fb40d600700 1 -- 172.21.15.89:0/1866418241 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3f40035b0 con 0x7fb41015d9d0 2024-04-02T07:36:28.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.080+0000 7fb40e000700 1 --2- 172.21.15.89:0/1866418241 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4100d16a0 0x7fb4100d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.080+0000 7fb40d600700 1 -- 172.21.15.89:0/1866418241 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 668212886 0 0) 0x7fb3f40035b0 con 0x7fb41015d9d0 2024-04-02T07:36:28.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40d600700 1 -- 172.21.15.89:0/1866418241 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4100d16a0 msgr2=0x7fb4100d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:28.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40d600700 1 --2- 172.21.15.89:0/1866418241 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4100d16a0 0x7fb4100d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40d600700 1 -- 172.21.15.89:0/1866418241 >> v1:172.21.15.113:6789/0 conn(0x7fb4100d2370 legacy=0x7fb4100ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40d600700 1 -- 172.21.15.89:0/1866418241 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4101619e0 con 0x7fb41015d9d0 2024-04-02T07:36:28.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40d600700 1 -- 172.21.15.89:0/1866418241 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fb400001670 con 0x7fb41015d9d0 2024-04-02T07:36:28.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40d600700 1 -- 172.21.15.89:0/1866418241 >> v1:172.21.15.89:6789/0 conn(0x7fb41015d9d0 legacy=0x7fb4100d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40d600700 1 --2- 172.21.15.89:0/1866418241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f4003a30 0x7fb3f4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40d600700 1 --2- 172.21.15.89:0/1866418241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3f4004850 0x7fb3f4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40d600700 1 --2- 172.21.15.89:0/1866418241 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4100d16a0 0x7fb3f4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40d600700 1 -- 172.21.15.89:0/1866418241 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4100d2820 con 0x7fb3f4003a30 2024-04-02T07:36:28.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40d600700 1 -- 172.21.15.89:0/1866418241 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f400bac0 con 0x7fb3f4004850 2024-04-02T07:36:28.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40d600700 1 -- 172.21.15.89:0/1866418241 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f400bc80 con 0x7fb4100d16a0 2024-04-02T07:36:28.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40ea00700 1 --2- 172.21.15.89:0/1866418241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f4003a30 0x7fb3f4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40f400700 1 --2- 172.21.15.89:0/1866418241 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4100d16a0 0x7fb3f4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.081+0000 7fb40e000700 1 --2- 172.21.15.89:0/1866418241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3f4004850 0x7fb3f4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.082+0000 7fb40ea00700 1 -- 172.21.15.89:0/1866418241 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4100d16a0 msgr2=0x7fb3f4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:28.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.082+0000 7fb40ea00700 1 --2- 172.21.15.89:0/1866418241 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4100d16a0 0x7fb3f4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.082+0000 7fb40ea00700 1 -- 172.21.15.89:0/1866418241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3f4004850 msgr2=0x7fb3f4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:28.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.082+0000 7fb40ea00700 1 --2- 172.21.15.89:0/1866418241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3f4004850 0x7fb3f4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.082+0000 7fb40ea00700 1 -- 172.21.15.89:0/1866418241 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f40035b0 con 0x7fb3f4003a30 2024-04-02T07:36:28.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.082+0000 7fb40e000700 1 --2- 172.21.15.89:0/1866418241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3f4004850 0x7fb3f4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:28.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.082+0000 7fb40ea00700 1 --2- 172.21.15.89:0/1866418241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f4003a30 0x7fb3f4003e50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb40000c210 tx=0x7fb3f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8dca3cd97d82da server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:28.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.082+0000 7fb40d600700 1 -- 172.21.15.89:0/1866418241 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb400007c10 con 0x7fb3f4003a30 2024-04-02T07:36:28.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.082+0000 7fb40d600700 1 -- 172.21.15.89:0/1866418241 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40000cdb0 con 0x7fb3f4003a30 2024-04-02T07:36:28.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.082+0000 7fb40d600700 1 -- 172.21.15.89:0/1866418241 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb400005b80 con 0x7fb3f4003a30 2024-04-02T07:36:28.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.082+0000 7fb415c00700 1 -- 172.21.15.89:0/1866418241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f4003a30 msgr2=0x7fb3f4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.082+0000 7fb415c00700 1 --2- 172.21.15.89:0/1866418241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f4003a30 0x7fb3f4003e50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb40000c210 tx=0x7fb3f4005880 comp rx=0 tx=0).stop 2024-04-02T07:36:28.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.083+0000 7fb40f400700 1 -- 172.21.15.89:0/1866418241 reap_dead start 2024-04-02T07:36:28.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.083+0000 7fb415c00700 1 -- 172.21.15.89:0/1866418241 shutdown_connections 2024-04-02T07:36:28.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.083+0000 7fb415c00700 1 -- 172.21.15.89:0/1866418241 >> 172.21.15.89:0/1866418241 conn(0x7fb41008aff0 msgr2=0x7fb4100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:28.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.083+0000 7fb415c00700 1 -- 172.21.15.89:0/1866418241 shutdown_connections 2024-04-02T07:36:28.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.083+0000 7fb415c00700 1 -- 172.21.15.89:0/1866418241 wait complete. 2024-04-02T07:36:28.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.084+0000 7fb415c00700 1 Processor -- start 2024-04-02T07:36:28.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.085+0000 7fb415c00700 1 -- start start 2024-04-02T07:36:28.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.085+0000 7fb415c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3f4003a30 0x7fb410155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.085+0000 7fb415c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb41015df40 0x7fb4100cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.085+0000 7fb415c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4100b45c0 0x7fb4100b4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.085+0000 7fb415c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4100b4fb0 con 0x7fb4100b45c0 2024-04-02T07:36:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.085+0000 7fb415c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41015e3d0 con 0x7fb3f4003a30 2024-04-02T07:36:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.085+0000 7fb415c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41015e6b0 con 0x7fb41015df40 2024-04-02T07:36:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.086+0000 7fb40f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4100b45c0 0x7fb4100b4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.086+0000 7fb40f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4100b45c0 0x7fb4100b4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50952/0 (socket says 172.21.15.89:50952) 2024-04-02T07:36:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.086+0000 7fb40f400700 1 -- 172.21.15.89:0/3834200961 learned_addr learned my addr 172.21.15.89:0/3834200961 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.086+0000 7fb40ea00700 1 --2- 172.21.15.89:0/3834200961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3f4003a30 0x7fb410155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.086+0000 7fb40e000700 1 --2- 172.21.15.89:0/3834200961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb41015df40 0x7fb4100cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.086+0000 7fb40f400700 1 -- 172.21.15.89:0/3834200961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb41015df40 msgr2=0x7fb4100cbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.086+0000 7fb40f400700 1 --2- 172.21.15.89:0/3834200961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb41015df40 0x7fb4100cbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.086+0000 7fb40f400700 1 -- 172.21.15.89:0/3834200961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3f4003a30 msgr2=0x7fb410155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.086+0000 7fb40f400700 1 --2- 172.21.15.89:0/3834200961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3f4003a30 0x7fb410155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.086+0000 7fb40f400700 1 -- 172.21.15.89:0/3834200961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41015e990 con 0x7fb4100b45c0 2024-04-02T07:36:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.086+0000 7fb40f400700 1 --2- 172.21.15.89:0/3834200961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4100b45c0 0x7fb4100b4a70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb4100d34f0 tx=0x7fb40800d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:28.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.087+0000 7fb406a00700 1 -- 172.21.15.89:0/3834200961 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408003c10 con 0x7fb4100b45c0 2024-04-02T07:36:28.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.087+0000 7fb406a00700 1 -- 172.21.15.89:0/3834200961 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb408003d90 con 0x7fb4100b45c0 2024-04-02T07:36:28.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.087+0000 7fb406a00700 1 -- 172.21.15.89:0/3834200961 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40800d950 con 0x7fb4100b45c0 2024-04-02T07:36:28.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.087+0000 7fb415c00700 1 -- 172.21.15.89:0/3834200961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41015ec70 con 0x7fb4100b45c0 2024-04-02T07:36:28.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.087+0000 7fb415c00700 1 -- 172.21.15.89:0/3834200961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb410156290 con 0x7fb4100b45c0 2024-04-02T07:36:28.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.088+0000 7fb415c00700 1 -- 172.21.15.89:0/3834200961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb41015f190 con 0x7fb4100b45c0 2024-04-02T07:36:28.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.089+0000 7fb406a00700 1 -- 172.21.15.89:0/3834200961 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb40800dab0 con 0x7fb4100b45c0 2024-04-02T07:36:28.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.089+0000 7fb406a00700 1 --2- 172.21.15.89:0/3834200961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb3f006ac90 0x7fb3f006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.090+0000 7fb406a00700 1 -- 172.21.15.89:0/3834200961 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb408089310 con 0x7fb4100b45c0 2024-04-02T07:36:28.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.090+0000 7fb40ea00700 1 --2- 172.21.15.89:0/3834200961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb3f006ac90 0x7fb3f006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.090+0000 7fb40ea00700 1 --2- 172.21.15.89:0/3834200961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb3f006ac90 0x7fb3f006d150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb400000c00 tx=0x7fb400007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:28.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.094+0000 7fb406a00700 1 -- 172.21.15.89:0/3834200961 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb41015f190 con 0x7fb4100b45c0 2024-04-02T07:36:28.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.294+0000 7fb415c00700 1 -- 172.21.15.89:0/3834200961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb41015f190 con 0x7fb4100b45c0 2024-04-02T07:36:28.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.295+0000 7fb406a00700 1 -- 172.21.15.89:0/3834200961 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb41015f190 con 0x7fb4100b45c0 2024-04-02T07:36:28.297 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:36:28.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.297+0000 7fb415c00700 1 -- 172.21.15.89:0/3834200961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb3f006ac90 msgr2=0x7fb3f006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.297+0000 7fb415c00700 1 --2- 172.21.15.89:0/3834200961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb3f006ac90 0x7fb3f006d150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb400000c00 tx=0x7fb400007000 comp rx=0 tx=0).stop 2024-04-02T07:36:28.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.297+0000 7fb415c00700 1 -- 172.21.15.89:0/3834200961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4100b45c0 msgr2=0x7fb4100b4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.297+0000 7fb415c00700 1 --2- 172.21.15.89:0/3834200961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4100b45c0 0x7fb4100b4a70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb4100d34f0 tx=0x7fb40800d150 comp rx=0 tx=0).stop 2024-04-02T07:36:28.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.297+0000 7fb415c00700 1 -- 172.21.15.89:0/3834200961 shutdown_connections 2024-04-02T07:36:28.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.297+0000 7fb415c00700 1 --2- 172.21.15.89:0/3834200961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb3f006ac90 0x7fb3f006d150 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.297+0000 7fb415c00700 1 --2- 172.21.15.89:0/3834200961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3f4003a30 0x7fb410155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.297+0000 7fb415c00700 1 --2- 172.21.15.89:0/3834200961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4100b45c0 0x7fb4100b4a70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.297+0000 7fb415c00700 1 --2- 172.21.15.89:0/3834200961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb41015df40 0x7fb4100cbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.297+0000 7fb415c00700 1 -- 172.21.15.89:0/3834200961 >> 172.21.15.89:0/3834200961 conn(0x7fb41008aff0 msgr2=0x7fb4100b2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:28.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.297+0000 7fb415c00700 1 -- 172.21.15.89:0/3834200961 shutdown_connections 2024-04-02T07:36:28.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.297+0000 7fb415c00700 1 -- 172.21.15.89:0/3834200961 wait complete. 2024-04-02T07:36:28.317 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:36:28.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.508+0000 7f1662200700 1 Processor -- start 2024-04-02T07:36:28.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.508+0000 7f1662200700 1 -- start start 2024-04-02T07:36:28.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.509+0000 7f1662200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f165c15d9d0 0x7f165c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.509+0000 7f1662200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165c161420 con 0x7f165c0d2540 2024-04-02T07:36:28.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.509+0000 7f1662200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165c161700 con 0x7f165c0d16a0 2024-04-02T07:36:28.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.509+0000 7f1662200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f165c15edc0 con 0x7f165c15d9d0 2024-04-02T07:36:28.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.510+0000 7f165be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f165c0d2540 0x7f165c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59896/0 (socket says 172.21.15.89:59896) 2024-04-02T07:36:28.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.510+0000 7f165be00700 1 -- 172.21.15.89:0/124493744 learned_addr learned my addr 172.21.15.89:0/124493744 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:28.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.510+0000 7f165b400700 1 --2- 172.21.15.89:0/124493744 >> v2:172.21.15.113:3300/0 conn(0x7f165c15d9d0 0x7f165c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.510+0000 7f165a000700 1 -- 172.21.15.89:0/124493744 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995299870 0 0) 0x7f165c161420 con 0x7f165c0d2540 2024-04-02T07:36:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.510+0000 7f165a000700 1 -- 172.21.15.89:0/124493744 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16400035b0 con 0x7f165c0d2540 2024-04-02T07:36:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.510+0000 7f165a000700 1 -- 172.21.15.89:0/124493744 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173910234 0 0) 0x7f165c161700 con 0x7f165c0d16a0 2024-04-02T07:36:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.510+0000 7f165a000700 1 -- 172.21.15.89:0/124493744 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f165c161420 con 0x7f165c0d16a0 2024-04-02T07:36:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.510+0000 7f165b400700 1 -- 172.21.15.89:0/124493744 >> v1:172.21.15.113:6789/0 conn(0x7f165c0d16a0 legacy=0x7f165c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.510+0000 7f165b400700 1 -- 172.21.15.89:0/124493744 >> v1:172.21.15.89:6789/0 conn(0x7f165c0d2540 legacy=0x7f165c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.510+0000 7f165b400700 1 -- 172.21.15.89:0/124493744 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165c1619e0 con 0x7f165c15d9d0 2024-04-02T07:36:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.511+0000 7f165b400700 1 --2- 172.21.15.89:0/124493744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f165c15d9d0 0x7f165c0d3c30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f164c009840 tx=0x7f164c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=534eba01ddf226a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.511+0000 7f165a000700 1 -- 172.21.15.89:0/124493744 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c011830 con 0x7f165c15d9d0 2024-04-02T07:36:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.511+0000 7f165a000700 1 -- 172.21.15.89:0/124493744 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164c011990 con 0x7f165c15d9d0 2024-04-02T07:36:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.511+0000 7f165a000700 1 -- 172.21.15.89:0/124493744 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c011d20 con 0x7f165c15d9d0 2024-04-02T07:36:28.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.511+0000 7f1662200700 1 -- 172.21.15.89:0/124493744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f165c15d9d0 msgr2=0x7f165c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.511+0000 7f1662200700 1 --2- 172.21.15.89:0/124493744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f165c15d9d0 0x7f165c0d3c30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f164c009840 tx=0x7f164c009b50 comp rx=0 tx=0).stop 2024-04-02T07:36:28.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.512+0000 7f1662200700 1 -- 172.21.15.89:0/124493744 shutdown_connections 2024-04-02T07:36:28.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.512+0000 7f1662200700 1 --2- 172.21.15.89:0/124493744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f165c15d9d0 0x7f165c0d3c30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.512+0000 7f1662200700 1 -- 172.21.15.89:0/124493744 >> 172.21.15.89:0/124493744 conn(0x7f165c08aff0 msgr2=0x7f165c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:28.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.512+0000 7f1662200700 1 -- 172.21.15.89:0/124493744 shutdown_connections 2024-04-02T07:36:28.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.512+0000 7f1662200700 1 -- 172.21.15.89:0/124493744 wait complete. 2024-04-02T07:36:28.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.514+0000 7f1662200700 1 Processor -- start 2024-04-02T07:36:28.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.514+0000 7f1662200700 1 -- start start 2024-04-02T07:36:28.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.515+0000 7f1662200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f165c0d16a0 0x7f165c159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.515+0000 7f1662200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f165c0d2540 0x7f165c1e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.515+0000 7f1662200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f165c15d9d0 0x7f165c0b8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.515+0000 7f1662200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c0b6020 con 0x7f165c0d16a0 2024-04-02T07:36:28.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.515+0000 7f1662200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c0b6300 con 0x7f165c0d2540 2024-04-02T07:36:28.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.515+0000 7f1662200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f165c1e7b80 con 0x7f165c15d9d0 2024-04-02T07:36:28.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.516+0000 7f165b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f165c0d16a0 0x7f165c159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.516+0000 7f165b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f165c0d16a0 0x7f165c159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50962/0 (socket says 172.21.15.89:50962) 2024-04-02T07:36:28.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.516+0000 7f165b400700 1 -- 172.21.15.89:0/1585498873 learned_addr learned my addr 172.21.15.89:0/1585498873 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:28.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.516+0000 7f165aa00700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f165c0d2540 0x7f165c1e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.516+0000 7f165b400700 1 -- 172.21.15.89:0/1585498873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f165c15d9d0 msgr2=0x7f165c0b8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.516+0000 7f165be00700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f165c15d9d0 0x7f165c0b8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.516+0000 7f165b400700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f165c15d9d0 0x7f165c0b8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.516+0000 7f165b400700 1 -- 172.21.15.89:0/1585498873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f165c0d2540 msgr2=0x7f165c1e7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.516+0000 7f165b400700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f165c0d2540 0x7f165c1e7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.516+0000 7f165b400700 1 -- 172.21.15.89:0/1585498873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c0094a0 con 0x7f165c0d16a0 2024-04-02T07:36:28.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.516+0000 7f165be00700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f165c15d9d0 0x7f165c0b8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:28.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.516+0000 7f165aa00700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f165c0d2540 0x7f165c1e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:28.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.516+0000 7f165b400700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f165c0d16a0 0x7f165c159a90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f164c0090b0 tx=0x7f164c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:28.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.517+0000 7f1653400700 1 -- 172.21.15.89:0/1585498873 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c01a9f0 con 0x7f165c0d16a0 2024-04-02T07:36:28.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.517+0000 7f1653400700 1 -- 172.21.15.89:0/1585498873 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164c01ab50 con 0x7f165c0d16a0 2024-04-02T07:36:28.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.517+0000 7f1653400700 1 -- 172.21.15.89:0/1585498873 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c0243f0 con 0x7f165c0d16a0 2024-04-02T07:36:28.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.517+0000 7f1662200700 1 -- 172.21.15.89:0/1585498873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165c1e7e60 con 0x7f165c0d16a0 2024-04-02T07:36:28.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.517+0000 7f1662200700 1 -- 172.21.15.89:0/1585498873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165c159fd0 con 0x7f165c0d16a0 2024-04-02T07:36:28.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.518+0000 7f1662200700 1 -- 172.21.15.89:0/1585498873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f165c0d1ca0 con 0x7f165c0d16a0 2024-04-02T07:36:28.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.518+0000 7f1653400700 1 -- 172.21.15.89:0/1585498873 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f164c02d450 con 0x7f165c0d16a0 2024-04-02T07:36:28.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.519+0000 7f1653400700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f163c06ac90 0x7f163c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.519+0000 7f1653400700 1 -- 172.21.15.89:0/1585498873 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f164c08a770 con 0x7f165c0d16a0 2024-04-02T07:36:28.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.520+0000 7f165aa00700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f163c06ac90 0x7f163c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.520+0000 7f165aa00700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f163c06ac90 0x7f163c06d150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f165c15d1b0 tx=0x7f1648009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:28.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.523+0000 7f1653400700 1 -- 172.21.15.89:0/1585498873 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f164c0558c0 con 0x7f165c0d16a0 2024-04-02T07:36:28.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.723+0000 7f1662200700 1 -- 172.21.15.89:0/1585498873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f165c161660 con 0x7f165c0d16a0 2024-04-02T07:36:28.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.724+0000 7f1653400700 1 -- 172.21.15.89:0/1585498873 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f164c006cc0 con 0x7f165c0d16a0 2024-04-02T07:36:28.726 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:36:28.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.726+0000 7f1662200700 1 -- 172.21.15.89:0/1585498873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f163c06ac90 msgr2=0x7f163c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.726+0000 7f1662200700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f163c06ac90 0x7f163c06d150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f165c15d1b0 tx=0x7f1648009040 comp rx=0 tx=0).stop 2024-04-02T07:36:28.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.726+0000 7f1662200700 1 -- 172.21.15.89:0/1585498873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f165c0d16a0 msgr2=0x7f165c159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.726+0000 7f1662200700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f165c0d16a0 0x7f165c159a90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f164c0090b0 tx=0x7f164c004750 comp rx=0 tx=0).stop 2024-04-02T07:36:28.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.726+0000 7f1662200700 1 -- 172.21.15.89:0/1585498873 shutdown_connections 2024-04-02T07:36:28.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.726+0000 7f1662200700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f163c06ac90 0x7f163c06d150 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.726+0000 7f1662200700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f165c15d9d0 0x7f165c0b8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.726+0000 7f1662200700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f165c0d16a0 0x7f165c159a90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.726+0000 7f1662200700 1 --2- 172.21.15.89:0/1585498873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f165c0d2540 0x7f165c1e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.726+0000 7f1662200700 1 -- 172.21.15.89:0/1585498873 >> 172.21.15.89:0/1585498873 conn(0x7f165c08aff0 msgr2=0x7f165c06bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:28.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.726+0000 7f1662200700 1 -- 172.21.15.89:0/1585498873 shutdown_connections 2024-04-02T07:36:28.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.727+0000 7f1662200700 1 -- 172.21.15.89:0/1585498873 wait complete. 2024-04-02T07:36:28.747 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:36:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.941+0000 7fe8c4200700 1 Processor -- start 2024-04-02T07:36:28.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.942+0000 7fe8c4200700 1 -- start start 2024-04-02T07:36:28.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.943+0000 7fe8c4200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe8bc0cee10 0x7fe8bc161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.943+0000 7fe8c4200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8bc0ce270 con 0x7fe8bc1611b0 2024-04-02T07:36:28.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.943+0000 7fe8c4200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8bc15b600 con 0x7fe8bc0d21f0 2024-04-02T07:36:28.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.943+0000 7fe8c4200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8bc0d2760 con 0x7fe8bc0cee10 2024-04-02T07:36:28.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.943+0000 7fe8c1800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe8bc1611b0 0x7fe8bc15ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46854/0 (socket says 172.21.15.89:46854) 2024-04-02T07:36:28.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.943+0000 7fe8c1800700 1 -- 172.21.15.89:0/3904065270 learned_addr learned my addr 172.21.15.89:0/3904065270 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:28.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.943+0000 7fe8c2200700 1 --2- 172.21.15.89:0/3904065270 >> v2:172.21.15.113:3300/0 conn(0x7fe8bc0cee10 0x7fe8bc161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.943+0000 7fe8bbe00700 1 -- 172.21.15.89:0/3904065270 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621517284 0 0) 0x7fe8bc0ce270 con 0x7fe8bc1611b0 2024-04-02T07:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.943+0000 7fe8bbe00700 1 -- 172.21.15.89:0/3904065270 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8a40035b0 con 0x7fe8bc1611b0 2024-04-02T07:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.944+0000 7fe8c2200700 1 -- 172.21.15.89:0/3904065270 >> v1:172.21.15.113:6789/0 conn(0x7fe8bc0d21f0 legacy=0x7fe8bc0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.944+0000 7fe8c2200700 1 -- 172.21.15.89:0/3904065270 >> v1:172.21.15.89:6789/0 conn(0x7fe8bc1611b0 legacy=0x7fe8bc15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.944+0000 7fe8c2200700 1 -- 172.21.15.89:0/3904065270 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8bc0d2a40 con 0x7fe8bc0cee10 2024-04-02T07:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.944+0000 7fe8c2200700 1 --2- 172.21.15.89:0/3904065270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8bc0cee10 0x7fe8bc161590 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009d90 tx=0x7fe8b400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3e9a53451170889 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.944+0000 7fe8bbe00700 1 -- 172.21.15.89:0/3904065270 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b400e070 con 0x7fe8bc0cee10 2024-04-02T07:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.944+0000 7fe8bbe00700 1 -- 172.21.15.89:0/3904065270 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b400bb20 con 0x7fe8bc0cee10 2024-04-02T07:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.944+0000 7fe8bbe00700 1 -- 172.21.15.89:0/3904065270 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4012430 con 0x7fe8bc0cee10 2024-04-02T07:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.945+0000 7fe8c4200700 1 -- 172.21.15.89:0/3904065270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8bc0cee10 msgr2=0x7fe8bc161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.945+0000 7fe8c4200700 1 --2- 172.21.15.89:0/3904065270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8bc0cee10 0x7fe8bc161590 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009d90 tx=0x7fe8b400b330 comp rx=0 tx=0).stop 2024-04-02T07:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.945+0000 7fe8c4200700 1 -- 172.21.15.89:0/3904065270 shutdown_connections 2024-04-02T07:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.945+0000 7fe8c4200700 1 --2- 172.21.15.89:0/3904065270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8bc0cee10 0x7fe8bc161590 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.945+0000 7fe8c4200700 1 -- 172.21.15.89:0/3904065270 >> 172.21.15.89:0/3904065270 conn(0x7fe8bc08aff0 msgr2=0x7fe8bc0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.945+0000 7fe8c4200700 1 -- 172.21.15.89:0/3904065270 shutdown_connections 2024-04-02T07:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.945+0000 7fe8c4200700 1 -- 172.21.15.89:0/3904065270 wait complete. 2024-04-02T07:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.947+0000 7fe8c4200700 1 Processor -- start 2024-04-02T07:36:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.947+0000 7fe8c4200700 1 -- start start 2024-04-02T07:36:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.947+0000 7fe8c4200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8bc0cee10 0x7fe8bc0b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c4200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8bc0d21f0 0x7fe8bc154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c4200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8bc1611b0 0x7fe8bc0ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c4200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc0c7850 con 0x7fe8bc1611b0 2024-04-02T07:36:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c4200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc0ccbf0 con 0x7fe8bc0d21f0 2024-04-02T07:36:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c4200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc0cc910 con 0x7fe8bc0cee10 2024-04-02T07:36:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c2200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8bc1611b0 0x7fe8bc0ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c2200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8bc1611b0 0x7fe8bc0ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40612/0 (socket says 172.21.15.89:40612) 2024-04-02T07:36:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c2200700 1 -- 172.21.15.89:0/2499556750 learned_addr learned my addr 172.21.15.89:0/2499556750 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c1800700 1 --2- 172.21.15.89:0/2499556750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8bc0cee10 0x7fe8bc0b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c0e00700 1 --2- 172.21.15.89:0/2499556750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8bc0d21f0 0x7fe8bc154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c2200700 1 -- 172.21.15.89:0/2499556750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8bc0cee10 msgr2=0x7fe8bc0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c2200700 1 --2- 172.21.15.89:0/2499556750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8bc0cee10 0x7fe8bc0b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c2200700 1 -- 172.21.15.89:0/2499556750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8bc0d21f0 msgr2=0x7fe8bc154ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c2200700 1 --2- 172.21.15.89:0/2499556750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8bc0d21f0 0x7fe8bc154ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.948+0000 7fe8c2200700 1 -- 172.21.15.89:0/2499556750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b4009a70 con 0x7fe8bc1611b0 2024-04-02T07:36:28.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.949+0000 7fe8c2200700 1 --2- 172.21.15.89:0/2499556750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8bc1611b0 0x7fe8bc0ca600 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc08ce10 tx=0x7fe8b4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:28.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.949+0000 7fe8b9600700 1 -- 172.21.15.89:0/2499556750 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4012430 con 0x7fe8bc1611b0 2024-04-02T07:36:28.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.949+0000 7fe8b9600700 1 -- 172.21.15.89:0/2499556750 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b4020070 con 0x7fe8bc1611b0 2024-04-02T07:36:28.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.949+0000 7fe8b9600700 1 -- 172.21.15.89:0/2499556750 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4004280 con 0x7fe8bc1611b0 2024-04-02T07:36:28.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.949+0000 7fe8c4200700 1 -- 172.21.15.89:0/2499556750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8bc0c7200 con 0x7fe8bc1611b0 2024-04-02T07:36:28.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.949+0000 7fe8c4200700 1 -- 172.21.15.89:0/2499556750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8bc155020 con 0x7fe8bc1611b0 2024-04-02T07:36:28.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.950+0000 7fe8c4200700 1 -- 172.21.15.89:0/2499556750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8bc0c7060 con 0x7fe8bc1611b0 2024-04-02T07:36:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.951+0000 7fe8b9600700 1 -- 172.21.15.89:0/2499556750 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe8b40043e0 con 0x7fe8bc1611b0 2024-04-02T07:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.952+0000 7fe8b9600700 1 --2- 172.21.15.89:0/2499556750 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe89c06ac90 0x7fe89c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.952+0000 7fe8b9600700 1 -- 172.21.15.89:0/2499556750 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8b4022070 con 0x7fe8bc1611b0 2024-04-02T07:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.952+0000 7fe8c1800700 1 --2- 172.21.15.89:0/2499556750 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe89c06ac90 0x7fe89c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.953+0000 7fe8c1800700 1 --2- 172.21.15.89:0/2499556750 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe89c06ac90 0x7fe89c06d150 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0007fd0 tx=0x7fe8b0007dc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:28.957+0000 7fe8b9600700 1 -- 172.21.15.89:0/2499556750 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe8bc0c7060 con 0x7fe8bc1611b0 2024-04-02T07:36:29.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.157+0000 7fe8c4200700 1 -- 172.21.15.89:0/2499556750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe8bc0c7060 con 0x7fe8bc1611b0 2024-04-02T07:36:29.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.157+0000 7fe8b9600700 1 -- 172.21.15.89:0/2499556750 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe8b402c020 con 0x7fe8bc1611b0 2024-04-02T07:36:29.159 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:36:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.160+0000 7fe8c4200700 1 -- 172.21.15.89:0/2499556750 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe89c06ac90 msgr2=0x7fe89c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.160+0000 7fe8c4200700 1 --2- 172.21.15.89:0/2499556750 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe89c06ac90 0x7fe89c06d150 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0007fd0 tx=0x7fe8b0007dc0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.160+0000 7fe8c4200700 1 -- 172.21.15.89:0/2499556750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8bc1611b0 msgr2=0x7fe8bc0ca600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.160+0000 7fe8c4200700 1 --2- 172.21.15.89:0/2499556750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8bc1611b0 0x7fe8bc0ca600 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc08ce10 tx=0x7fe8b4004ad0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.160+0000 7fe8c4200700 1 -- 172.21.15.89:0/2499556750 shutdown_connections 2024-04-02T07:36:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.160+0000 7fe8c4200700 1 --2- 172.21.15.89:0/2499556750 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe89c06ac90 0x7fe89c06d150 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.160+0000 7fe8c4200700 1 --2- 172.21.15.89:0/2499556750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8bc1611b0 0x7fe8bc0ca600 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.160+0000 7fe8c4200700 1 --2- 172.21.15.89:0/2499556750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8bc0cee10 0x7fe8bc0b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.160+0000 7fe8c4200700 1 --2- 172.21.15.89:0/2499556750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8bc0d21f0 0x7fe8bc154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.160+0000 7fe8c4200700 1 -- 172.21.15.89:0/2499556750 >> 172.21.15.89:0/2499556750 conn(0x7fe8bc08aff0 msgr2=0x7fe8bc0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.160+0000 7fe8c4200700 1 -- 172.21.15.89:0/2499556750 shutdown_connections 2024-04-02T07:36:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.160+0000 7fe8c4200700 1 -- 172.21.15.89:0/2499556750 wait complete. 2024-04-02T07:36:29.180 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:36:29.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.371+0000 7f6f05a00700 1 Processor -- start 2024-04-02T07:36:29.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.371+0000 7f6f05a00700 1 -- start start 2024-04-02T07:36:29.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.372+0000 7f6f05a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f00160f10 0x7f6f000d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:29.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.372+0000 7f6f05a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f000ceda0 con 0x7f6f00161590 2024-04-02T07:36:29.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.372+0000 7f6f05a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f001612f0 con 0x7f6f000cf230 2024-04-02T07:36:29.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.372+0000 7f6f05a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f000ce470 con 0x7f6f00160f10 2024-04-02T07:36:29.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.372+0000 7f6eff400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6f00161590 0x7f6f000b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46868/0 (socket says 172.21.15.89:46868) 2024-04-02T07:36:29.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.372+0000 7f6eff400700 1 -- 172.21.15.89:0/2349116865 learned_addr learned my addr 172.21.15.89:0/2349116865 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:29.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.373+0000 7f6efea00700 1 --2- 172.21.15.89:0/2349116865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f00160f10 0x7f6f000d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:29.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.373+0000 7f6efd600700 1 -- 172.21.15.89:0/2349116865 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968084277 0 0) 0x7f6f000ceda0 con 0x7f6f00161590 2024-04-02T07:36:29.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.373+0000 7f6efd600700 1 -- 172.21.15.89:0/2349116865 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee40035b0 con 0x7f6f00161590 2024-04-02T07:36:29.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.373+0000 7f6efea00700 1 -- 172.21.15.89:0/2349116865 >> v1:172.21.15.113:6789/0 conn(0x7f6f000cf230 legacy=0x7f6f000d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:29.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.373+0000 7f6efea00700 1 -- 172.21.15.89:0/2349116865 >> v1:172.21.15.89:6789/0 conn(0x7f6f00161590 legacy=0x7f6f000b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:29.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.373+0000 7f6efea00700 1 -- 172.21.15.89:0/2349116865 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f000ce750 con 0x7f6f00160f10 2024-04-02T07:36:29.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.373+0000 7f6efea00700 1 --2- 172.21.15.89:0/2349116865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f00160f10 0x7f6f000d18b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6ef00098a0 tx=0x7f6ef0009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61d92922e8d43815 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:29.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.374+0000 7f6efd600700 1 -- 172.21.15.89:0/2349116865 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0011830 con 0x7f6f00160f10 2024-04-02T07:36:29.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.374+0000 7f6efd600700 1 -- 172.21.15.89:0/2349116865 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef0011990 con 0x7f6f00160f10 2024-04-02T07:36:29.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.374+0000 7f6efd600700 1 -- 172.21.15.89:0/2349116865 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0011d20 con 0x7f6f00160f10 2024-04-02T07:36:29.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.374+0000 7f6f05a00700 1 -- 172.21.15.89:0/2349116865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f00160f10 msgr2=0x7f6f000d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:29.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.374+0000 7f6f05a00700 1 --2- 172.21.15.89:0/2349116865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f00160f10 0x7f6f000d18b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6ef00098a0 tx=0x7f6ef0009bb0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.374+0000 7f6f05a00700 1 -- 172.21.15.89:0/2349116865 shutdown_connections 2024-04-02T07:36:29.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.374+0000 7f6f05a00700 1 --2- 172.21.15.89:0/2349116865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f00160f10 0x7f6f000d18b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.374+0000 7f6f05a00700 1 -- 172.21.15.89:0/2349116865 >> 172.21.15.89:0/2349116865 conn(0x7f6f0008aff0 msgr2=0x7f6f000d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:29.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.374+0000 7f6f05a00700 1 -- 172.21.15.89:0/2349116865 shutdown_connections 2024-04-02T07:36:29.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.374+0000 7f6f05a00700 1 -- 172.21.15.89:0/2349116865 wait complete. 2024-04-02T07:36:29.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.376+0000 7f6f05a00700 1 Processor -- start 2024-04-02T07:36:29.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.376+0000 7f6f05a00700 1 -- start start 2024-04-02T07:36:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.377+0000 7f6f05a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f000cf230 0x7f6f0015aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.377+0000 7f6f05a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f00160f10 0x7f6f0015c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.377+0000 7f6f05a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f00161590 0x7f6f0015f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.377+0000 7f6f05a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0015a610 con 0x7f6f00160f10 2024-04-02T07:36:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.377+0000 7f6f05a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f00159a00 con 0x7f6f00161590 2024-04-02T07:36:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.377+0000 7f6f05a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f00159ce0 con 0x7f6f000cf230 2024-04-02T07:36:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.377+0000 7f6efe000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f00160f10 0x7f6f0015c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.377+0000 7f6efea00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f000cf230 0x7f6f0015aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.377+0000 7f6efe000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f00160f10 0x7f6f0015c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40626/0 (socket says 172.21.15.89:40626) 2024-04-02T07:36:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.377+0000 7f6efe000700 1 -- 172.21.15.89:0/2830451194 learned_addr learned my addr 172.21.15.89:0/2830451194 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.378+0000 7f6eff400700 1 --2- 172.21.15.89:0/2830451194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f00161590 0x7f6f0015f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.378+0000 7f6efe000700 1 -- 172.21.15.89:0/2830451194 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f000cf230 msgr2=0x7f6f0015aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.378+0000 7f6efe000700 1 --2- 172.21.15.89:0/2830451194 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f000cf230 0x7f6f0015aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.378+0000 7f6efe000700 1 -- 172.21.15.89:0/2830451194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f00161590 msgr2=0x7f6f0015f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:29.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.378+0000 7f6efe000700 1 --2- 172.21.15.89:0/2830451194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f00161590 0x7f6f0015f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.378+0000 7f6efe000700 1 -- 172.21.15.89:0/2830451194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef00094a0 con 0x7f6f00160f10 2024-04-02T07:36:29.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.378+0000 7f6efe000700 1 --2- 172.21.15.89:0/2830451194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f00160f10 0x7f6f0015c2a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6eec00c4b0 tx=0x7f6eec00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:29.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.378+0000 7f6ef6a00700 1 -- 172.21.15.89:0/2830451194 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec0052d0 con 0x7f6f00160f10 2024-04-02T07:36:29.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.378+0000 7f6ef6a00700 1 -- 172.21.15.89:0/2830451194 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eec005430 con 0x7f6f00160f10 2024-04-02T07:36:29.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.378+0000 7f6ef6a00700 1 -- 172.21.15.89:0/2830451194 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec0038c0 con 0x7f6f00160f10 2024-04-02T07:36:29.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.378+0000 7f6f05a00700 1 -- 172.21.15.89:0/2830451194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f00159fc0 con 0x7f6f00160f10 2024-04-02T07:36:29.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.379+0000 7f6f05a00700 1 -- 172.21.15.89:0/2830451194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0015c7e0 con 0x7f6f00160f10 2024-04-02T07:36:29.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.379+0000 7f6f05a00700 1 -- 172.21.15.89:0/2830451194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0015a7c0 con 0x7f6f00160f10 2024-04-02T07:36:29.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.380+0000 7f6ef6a00700 1 -- 172.21.15.89:0/2830451194 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6eec003a20 con 0x7f6f00160f10 2024-04-02T07:36:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.382+0000 7f6ef6a00700 1 --2- 172.21.15.89:0/2830451194 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ee006ac90 0x7f6ee006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.382+0000 7f6ef6a00700 1 -- 172.21.15.89:0/2830451194 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6eec088790 con 0x7f6f00160f10 2024-04-02T07:36:29.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.382+0000 7f6efea00700 1 --2- 172.21.15.89:0/2830451194 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ee006ac90 0x7f6ee006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:29.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.383+0000 7f6efea00700 1 --2- 172.21.15.89:0/2830451194 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ee006ac90 0x7f6ee006d150 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6ef00090b0 tx=0x7f6ef0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:29.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.386+0000 7f6ef6a00700 1 -- 172.21.15.89:0/2830451194 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6f0015a7c0 con 0x7f6f00160f10 2024-04-02T07:36:29.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.588+0000 7f6f05a00700 1 -- 172.21.15.89:0/2830451194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f00067720 con 0x7f6f00160f10 2024-04-02T07:36:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.590+0000 7f6ef6a00700 1 -- 172.21.15.89:0/2830451194 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f6eec017090 con 0x7f6f00160f10 2024-04-02T07:36:29.591 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:29.592 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:54.238913+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:36:29.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.592+0000 7f6f05a00700 1 -- 172.21.15.89:0/2830451194 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ee006ac90 msgr2=0x7f6ee006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:29.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.592+0000 7f6f05a00700 1 --2- 172.21.15.89:0/2830451194 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ee006ac90 0x7f6ee006d150 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6ef00090b0 tx=0x7f6ef0008030 comp rx=0 tx=0).stop 2024-04-02T07:36:29.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.592+0000 7f6f05a00700 1 -- 172.21.15.89:0/2830451194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f00160f10 msgr2=0x7f6f0015c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:29.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.592+0000 7f6f05a00700 1 --2- 172.21.15.89:0/2830451194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f00160f10 0x7f6f0015c2a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6eec00c4b0 tx=0x7f6eec00c7c0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.592+0000 7f6f05a00700 1 -- 172.21.15.89:0/2830451194 shutdown_connections 2024-04-02T07:36:29.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.592+0000 7f6f05a00700 1 --2- 172.21.15.89:0/2830451194 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ee006ac90 0x7f6ee006d150 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.592+0000 7f6f05a00700 1 --2- 172.21.15.89:0/2830451194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f00161590 0x7f6f0015f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.592+0000 7f6f05a00700 1 --2- 172.21.15.89:0/2830451194 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f000cf230 0x7f6f0015aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.592+0000 7f6f05a00700 1 --2- 172.21.15.89:0/2830451194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f00160f10 0x7f6f0015c2a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.592+0000 7f6f05a00700 1 -- 172.21.15.89:0/2830451194 >> 172.21.15.89:0/2830451194 conn(0x7f6f0008aff0 msgr2=0x7f6f000b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:29.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.592+0000 7f6f05a00700 1 -- 172.21.15.89:0/2830451194 shutdown_connections 2024-04-02T07:36:29.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.592+0000 7f6f05a00700 1 -- 172.21.15.89:0/2830451194 wait complete. 2024-04-02T07:36:29.612 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:35:51.296561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:35:52.232882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:36:29.612 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:36:29.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.803+0000 7f2e38600700 1 Processor -- start 2024-04-02T07:36:29.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.803+0000 7f2e38600700 1 -- start start 2024-04-02T07:36:29.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.804+0000 7f2e38600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2e300d16a0 0x7f2e300d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:29.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.804+0000 7f2e38600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2e300d2370 0x7f2e300ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:29.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.804+0000 7f2e38600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e30161420 con 0x7f2e3015d9d0 2024-04-02T07:36:29.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.804+0000 7f2e38600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e300d4340 con 0x7f2e300d16a0 2024-04-02T07:36:29.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.804+0000 7f2e38600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e3015edc0 con 0x7f2e300d2370 2024-04-02T07:36:29.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.804+0000 7f2e35200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2e300d16a0 0x7f2e300d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:29.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.804+0000 7f2e35200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2e300d16a0 0x7f2e300d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40632/0 (socket says 172.21.15.89:40632) 2024-04-02T07:36:29.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.804+0000 7f2e35200700 1 -- 172.21.15.89:0/2584677506 learned_addr learned my addr 172.21.15.89:0/2584677506 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:29.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.804+0000 7f2e36600700 1 --2- 172.21.15.89:0/2584677506 >> v2:172.21.15.113:3300/0 conn(0x7f2e300d2370 0x7f2e300ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:29.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.805+0000 7f2e35200700 1 -- 172.21.15.89:0/2584677506 >> v2:172.21.15.113:3300/0 conn(0x7f2e300d2370 msgr2=0x7f2e300ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:29.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.805+0000 7f2e35200700 1 --2- 172.21.15.89:0/2584677506 >> v2:172.21.15.113:3300/0 conn(0x7f2e300d2370 0x7f2e300ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.805+0000 7f2e35200700 1 -- 172.21.15.89:0/2584677506 >> v1:172.21.15.113:6789/0 conn(0x7f2e3015d9d0 legacy=0x7f2e300d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:29.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.805+0000 7f2e35200700 1 -- 172.21.15.89:0/2584677506 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e30161700 con 0x7f2e300d16a0 2024-04-02T07:36:29.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.805+0000 7f2e35200700 1 --2- 172.21.15.89:0/2584677506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e300d16a0 0x7f2e300d1e30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2e3008ca10 tx=0x7f2e1c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1188b99608afbc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:29.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.805+0000 7f2e2fe00700 1 -- 172.21.15.89:0/2584677506 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c005e30 con 0x7f2e300d16a0 2024-04-02T07:36:29.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.805+0000 7f2e2fe00700 1 -- 172.21.15.89:0/2584677506 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1c004030 con 0x7f2e300d16a0 2024-04-02T07:36:29.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.805+0000 7f2e2fe00700 1 -- 172.21.15.89:0/2584677506 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c0043e0 con 0x7f2e300d16a0 2024-04-02T07:36:29.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.806+0000 7f2e38600700 1 -- 172.21.15.89:0/2584677506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e300d16a0 msgr2=0x7f2e300d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:29.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.806+0000 7f2e38600700 1 --2- 172.21.15.89:0/2584677506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e300d16a0 0x7f2e300d1e30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2e3008ca10 tx=0x7f2e1c009bd0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.806+0000 7f2e38600700 1 -- 172.21.15.89:0/2584677506 shutdown_connections 2024-04-02T07:36:29.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.806+0000 7f2e38600700 1 --2- 172.21.15.89:0/2584677506 >> v2:172.21.15.113:3300/0 conn(0x7f2e300d2370 0x7f2e300ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.806+0000 7f2e38600700 1 --2- 172.21.15.89:0/2584677506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e300d16a0 0x7f2e300d1e30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.806+0000 7f2e38600700 1 -- 172.21.15.89:0/2584677506 >> 172.21.15.89:0/2584677506 conn(0x7f2e3008aff0 msgr2=0x7f2e300d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:29.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.806+0000 7f2e38600700 1 -- 172.21.15.89:0/2584677506 shutdown_connections 2024-04-02T07:36:29.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.806+0000 7f2e38600700 1 -- 172.21.15.89:0/2584677506 wait complete. 2024-04-02T07:36:29.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.808+0000 7f2e38600700 1 Processor -- start 2024-04-02T07:36:29.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.809+0000 7f2e38600700 1 -- start start 2024-04-02T07:36:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e38600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e300d16a0 0x7f2e301501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e38600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e300d2370 0x7f2e300b3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e38600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e3015d9d0 0x7f2e300b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e38600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300b8ec0 con 0x7f2e3015d9d0 2024-04-02T07:36:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e38600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30150e50 con 0x7f2e300d2370 2024-04-02T07:36:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e38600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300b8be0 con 0x7f2e300d16a0 2024-04-02T07:36:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e36600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e3015d9d0 0x7f2e300b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e36600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e3015d9d0 0x7f2e300b4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40644/0 (socket says 172.21.15.89:40644) 2024-04-02T07:36:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e36600700 1 -- 172.21.15.89:0/1388439137 learned_addr learned my addr 172.21.15.89:0/1388439137 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e35c00700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e300d16a0 0x7f2e301501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e35200700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e300d2370 0x7f2e300b3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e36600700 1 -- 172.21.15.89:0/1388439137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e300d16a0 msgr2=0x7f2e301501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e36600700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e300d16a0 0x7f2e301501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e36600700 1 -- 172.21.15.89:0/1388439137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e300d2370 msgr2=0x7f2e300b3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:29.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e36600700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e300d2370 0x7f2e300b3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:29.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e36600700 1 -- 172.21.15.89:0/1388439137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1c009500 con 0x7f2e3015d9d0 2024-04-02T07:36:29.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.810+0000 7f2e35200700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e300d2370 0x7f2e300b3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:29.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.811+0000 7f2e36600700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e3015d9d0 0x7f2e300b4070 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800efc0 tx=0x7f2e2800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:29.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.811+0000 7f2e2d600700 1 -- 172.21.15.89:0/1388439137 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28009ea0 con 0x7f2e3015d9d0 2024-04-02T07:36:29.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.811+0000 7f2e2d600700 1 -- 172.21.15.89:0/1388439137 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2800f040 con 0x7f2e3015d9d0 2024-04-02T07:36:29.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.811+0000 7f2e38600700 1 -- 172.21.15.89:0/1388439137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e300b3050 con 0x7f2e3015d9d0 2024-04-02T07:36:29.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.811+0000 7f2e2d600700 1 -- 172.21.15.89:0/1388439137 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28014480 con 0x7f2e3015d9d0 2024-04-02T07:36:29.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.811+0000 7f2e38600700 1 -- 172.21.15.89:0/1388439137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e300b3610 con 0x7f2e3015d9d0 2024-04-02T07:36:29.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.812+0000 7f2e38600700 1 -- 172.21.15.89:0/1388439137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e300d1ca0 con 0x7f2e3015d9d0 2024-04-02T07:36:29.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.813+0000 7f2e2d600700 1 -- 172.21.15.89:0/1388439137 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2e280145e0 con 0x7f2e3015d9d0 2024-04-02T07:36:29.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.814+0000 7f2e2d600700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2e1006ac90 0x7f2e1006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:29.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.814+0000 7f2e2d600700 1 -- 172.21.15.89:0/1388439137 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e28089c10 con 0x7f2e3015d9d0 2024-04-02T07:36:29.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.814+0000 7f2e35c00700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2e1006ac90 0x7f2e1006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:29.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.814+0000 7f2e35c00700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2e1006ac90 0x7f2e1006d150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2e24007fd0 tx=0x7f2e24009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:29.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:29.818+0000 7f2e2d600700 1 -- 172.21.15.89:0/1388439137 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e28054d50 con 0x7f2e3015d9d0 2024-04-02T07:36:30.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.019+0000 7f2e38600700 1 -- 172.21.15.89:0/1388439137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2e300b2eb0 con 0x7f2e3015d9d0 2024-04-02T07:36:30.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.019+0000 7f2e2d600700 1 -- 172.21.15.89:0/1388439137 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2e300b2eb0 con 0x7f2e3015d9d0 2024-04-02T07:36:30.021 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:36:30.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.021+0000 7f2e38600700 1 -- 172.21.15.89:0/1388439137 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2e1006ac90 msgr2=0x7f2e1006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:30.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.021+0000 7f2e38600700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2e1006ac90 0x7f2e1006d150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2e24007fd0 tx=0x7f2e24009040 comp rx=0 tx=0).stop 2024-04-02T07:36:30.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.021+0000 7f2e38600700 1 -- 172.21.15.89:0/1388439137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e3015d9d0 msgr2=0x7f2e300b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:30.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.021+0000 7f2e38600700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e3015d9d0 0x7f2e300b4070 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800efc0 tx=0x7f2e2800c3a0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.021+0000 7f2e38600700 1 -- 172.21.15.89:0/1388439137 shutdown_connections 2024-04-02T07:36:30.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.021+0000 7f2e38600700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2e1006ac90 0x7f2e1006d150 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.021+0000 7f2e38600700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e3015d9d0 0x7f2e300b4070 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.021+0000 7f2e38600700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e300d16a0 0x7f2e301501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.021+0000 7f2e38600700 1 --2- 172.21.15.89:0/1388439137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e300d2370 0x7f2e300b3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.021+0000 7f2e38600700 1 -- 172.21.15.89:0/1388439137 >> 172.21.15.89:0/1388439137 conn(0x7f2e3008aff0 msgr2=0x7f2e300d39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:30.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.022+0000 7f2e38600700 1 -- 172.21.15.89:0/1388439137 shutdown_connections 2024-04-02T07:36:30.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.022+0000 7f2e38600700 1 -- 172.21.15.89:0/1388439137 wait complete. 2024-04-02T07:36:30.043 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:36:30.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.235+0000 7fc32fe00700 1 Processor -- start 2024-04-02T07:36:30.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.235+0000 7fc32fe00700 1 -- start start 2024-04-02T07:36:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.236+0000 7fc32fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc32815d9d0 0x7fc3280d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.236+0000 7fc32fe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc3280d16a0 0x7fc3280d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.236+0000 7fc32fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3280d2370 0x7fc32815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.236+0000 7fc32fe00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3280d2820 con 0x7fc32815d9d0 2024-04-02T07:36:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.236+0000 7fc32fe00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc328161420 con 0x7fc3280d16a0 2024-04-02T07:36:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.236+0000 7fc32fe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc328161700 con 0x7fc3280d2370 2024-04-02T07:36:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.236+0000 7fc32d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc32815d9d0 0x7fc3280d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.236+0000 7fc32d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc32815d9d0 0x7fc3280d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40648/0 (socket says 172.21.15.89:40648) 2024-04-02T07:36:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.236+0000 7fc32d400700 1 -- 172.21.15.89:0/3360386633 learned_addr learned my addr 172.21.15.89:0/3360386633 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.236+0000 7fc32ca00700 1 --2- 172.21.15.89:0/3360386633 >> v2:172.21.15.113:3300/0 conn(0x7fc3280d16a0 0x7fc3280d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.236+0000 7fc32de00700 1 --2- 172.21.15.89:0/3360386633 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3280d2370 0x7fc32815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:30.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.237+0000 7fc32d400700 1 -- 172.21.15.89:0/3360386633 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3280d2370 msgr2=0x7fc32815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:30.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.237+0000 7fc32d400700 1 --2- 172.21.15.89:0/3360386633 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3280d2370 0x7fc32815ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.237+0000 7fc32d400700 1 -- 172.21.15.89:0/3360386633 >> v2:172.21.15.113:3300/0 conn(0x7fc3280d16a0 msgr2=0x7fc3280d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:30.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.237+0000 7fc32d400700 1 --2- 172.21.15.89:0/3360386633 >> v2:172.21.15.113:3300/0 conn(0x7fc3280d16a0 0x7fc3280d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.237+0000 7fc32d400700 1 -- 172.21.15.89:0/3360386633 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3281619e0 con 0x7fc32815d9d0 2024-04-02T07:36:30.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.237+0000 7fc32ca00700 1 --2- 172.21.15.89:0/3360386633 >> v2:172.21.15.113:3300/0 conn(0x7fc3280d16a0 0x7fc3280d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:30.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.237+0000 7fc32d400700 1 --2- 172.21.15.89:0/3360386633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc32815d9d0 0x7fc3280d3c30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc3280d12f0 tx=0x7fc31c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c2d2b8122764e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:30.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.237+0000 7fc327e00700 1 -- 172.21.15.89:0/3360386633 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31c00b870 con 0x7fc32815d9d0 2024-04-02T07:36:30.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.237+0000 7fc327e00700 1 -- 172.21.15.89:0/3360386633 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31c00b9d0 con 0x7fc32815d9d0 2024-04-02T07:36:30.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.237+0000 7fc327e00700 1 -- 172.21.15.89:0/3360386633 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31c00bd60 con 0x7fc32815d9d0 2024-04-02T07:36:30.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.238+0000 7fc32fe00700 1 -- 172.21.15.89:0/3360386633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc32815d9d0 msgr2=0x7fc3280d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:30.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.238+0000 7fc32fe00700 1 --2- 172.21.15.89:0/3360386633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc32815d9d0 0x7fc3280d3c30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc3280d12f0 tx=0x7fc31c009bd0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.238+0000 7fc32fe00700 1 -- 172.21.15.89:0/3360386633 shutdown_connections 2024-04-02T07:36:30.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.238+0000 7fc32fe00700 1 --2- 172.21.15.89:0/3360386633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc32815d9d0 0x7fc3280d3c30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.238+0000 7fc32fe00700 1 --2- 172.21.15.89:0/3360386633 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3280d2370 0x7fc32815ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.238+0000 7fc32fe00700 1 --2- 172.21.15.89:0/3360386633 >> v2:172.21.15.113:3300/0 conn(0x7fc3280d16a0 0x7fc3280d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.238+0000 7fc32fe00700 1 -- 172.21.15.89:0/3360386633 >> 172.21.15.89:0/3360386633 conn(0x7fc32808aff0 msgr2=0x7fc3280d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:30.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.238+0000 7fc32fe00700 1 -- 172.21.15.89:0/3360386633 shutdown_connections 2024-04-02T07:36:30.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.238+0000 7fc32fe00700 1 -- 172.21.15.89:0/3360386633 wait complete. 2024-04-02T07:36:30.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.240+0000 7fc32fe00700 1 Processor -- start 2024-04-02T07:36:30.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.240+0000 7fc32fe00700 1 -- start start 2024-04-02T07:36:30.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.241+0000 7fc32fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3280d16a0 0x7fc3280ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:30.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.241+0000 7fc32fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280d2370 0x7fc32814a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:30.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.242+0000 7fc32fe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc32815d9d0 0x7fc32814ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:30.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.242+0000 7fc32fe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3280cac80 con 0x7fc3280d2370 2024-04-02T07:36:30.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.242+0000 7fc32fe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3280cd390 con 0x7fc32815d9d0 2024-04-02T07:36:30.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.242+0000 7fc32fe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3280cd670 con 0x7fc3280d16a0 2024-04-02T07:36:30.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.242+0000 7fc32ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280d2370 0x7fc32814a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:30.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.242+0000 7fc32ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280d2370 0x7fc32814a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40664/0 (socket says 172.21.15.89:40664) 2024-04-02T07:36:30.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.242+0000 7fc32ca00700 1 -- 172.21.15.89:0/796330541 learned_addr learned my addr 172.21.15.89:0/796330541 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:30.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.242+0000 7fc32d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3280d16a0 0x7fc3280ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:30.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.242+0000 7fc32de00700 1 --2- 172.21.15.89:0/796330541 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc32815d9d0 0x7fc32814ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:30.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.242+0000 7fc32ca00700 1 -- 172.21.15.89:0/796330541 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3280d16a0 msgr2=0x7fc3280ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:30.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.242+0000 7fc32ca00700 1 --2- 172.21.15.89:0/796330541 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3280d16a0 0x7fc3280ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.242+0000 7fc32ca00700 1 -- 172.21.15.89:0/796330541 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc32815d9d0 msgr2=0x7fc32814ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:30.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.242+0000 7fc32ca00700 1 --2- 172.21.15.89:0/796330541 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc32815d9d0 0x7fc32814ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.242+0000 7fc32ca00700 1 -- 172.21.15.89:0/796330541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31c009550 con 0x7fc3280d2370 2024-04-02T07:36:30.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.243+0000 7fc32ca00700 1 --2- 172.21.15.89:0/796330541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280d2370 0x7fc32814a2b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc31400e7e0 tx=0x7fc31400eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:30.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.243+0000 7fc325600700 1 -- 172.21.15.89:0/796330541 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31400c070 con 0x7fc3280d2370 2024-04-02T07:36:30.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.243+0000 7fc325600700 1 -- 172.21.15.89:0/796330541 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc314005c20 con 0x7fc3280d2370 2024-04-02T07:36:30.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.243+0000 7fc32fe00700 1 -- 172.21.15.89:0/796330541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3280cd950 con 0x7fc3280d2370 2024-04-02T07:36:30.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.243+0000 7fc325600700 1 -- 172.21.15.89:0/796330541 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc314004030 con 0x7fc3280d2370 2024-04-02T07:36:30.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.243+0000 7fc32fe00700 1 -- 172.21.15.89:0/796330541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc32814a7f0 con 0x7fc3280d2370 2024-04-02T07:36:30.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.244+0000 7fc32fe00700 1 -- 172.21.15.89:0/796330541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3280d1ca0 con 0x7fc3280d2370 2024-04-02T07:36:30.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.245+0000 7fc325600700 1 -- 172.21.15.89:0/796330541 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc314004190 con 0x7fc3280d2370 2024-04-02T07:36:30.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.246+0000 7fc325600700 1 --2- 172.21.15.89:0/796330541 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc30806ac90 0x7fc30806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:30.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.246+0000 7fc325600700 1 -- 172.21.15.89:0/796330541 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc314088a60 con 0x7fc3280d2370 2024-04-02T07:36:30.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.246+0000 7fc32d400700 1 --2- 172.21.15.89:0/796330541 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc30806ac90 0x7fc30806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:30.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.246+0000 7fc32d400700 1 --2- 172.21.15.89:0/796330541 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc30806ac90 0x7fc30806d150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc31c014040 tx=0x7fc31c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:30.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.250+0000 7fc325600700 1 -- 172.21.15.89:0/796330541 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc314053bb0 con 0x7fc3280d2370 2024-04-02T07:36:30.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.451+0000 7fc32fe00700 1 -- 172.21.15.89:0/796330541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc3280d3310 con 0x7fc3280d2370 2024-04-02T07:36:30.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.452+0000 7fc325600700 1 -- 172.21.15.89:0/796330541 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc3140571d0 con 0x7fc3280d2370 2024-04-02T07:36:30.454 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:36:30.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.454+0000 7fc32fe00700 1 -- 172.21.15.89:0/796330541 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc30806ac90 msgr2=0x7fc30806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:30.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.454+0000 7fc32fe00700 1 --2- 172.21.15.89:0/796330541 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc30806ac90 0x7fc30806d150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc31c014040 tx=0x7fc31c008030 comp rx=0 tx=0).stop 2024-04-02T07:36:30.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.454+0000 7fc32fe00700 1 -- 172.21.15.89:0/796330541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280d2370 msgr2=0x7fc32814a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:30.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.454+0000 7fc32fe00700 1 --2- 172.21.15.89:0/796330541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280d2370 0x7fc32814a2b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc31400e7e0 tx=0x7fc31400eaf0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.454+0000 7fc32fe00700 1 -- 172.21.15.89:0/796330541 shutdown_connections 2024-04-02T07:36:30.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.454+0000 7fc32fe00700 1 --2- 172.21.15.89:0/796330541 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc30806ac90 0x7fc30806d150 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.454+0000 7fc32fe00700 1 --2- 172.21.15.89:0/796330541 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc32815d9d0 0x7fc32814ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.454+0000 7fc32fe00700 1 --2- 172.21.15.89:0/796330541 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3280d16a0 0x7fc3280ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.454+0000 7fc32fe00700 1 --2- 172.21.15.89:0/796330541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3280d2370 0x7fc32814a2b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.454+0000 7fc32fe00700 1 -- 172.21.15.89:0/796330541 >> 172.21.15.89:0/796330541 conn(0x7fc32808aff0 msgr2=0x7fc3280ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:30.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.455+0000 7fc32fe00700 1 -- 172.21.15.89:0/796330541 shutdown_connections 2024-04-02T07:36:30.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.455+0000 7fc32fe00700 1 -- 172.21.15.89:0/796330541 wait complete. 2024-04-02T07:36:30.474 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:36:30.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.665+0000 7fa798800700 1 Processor -- start 2024-04-02T07:36:30.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.665+0000 7fa798800700 1 -- start start 2024-04-02T07:36:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.666+0000 7fa798800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7900d16a0 0x7fa7900d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.666+0000 7fa798800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7900d2370 0x7fa7900ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.666+0000 7fa798800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa790161420 con 0x7fa79015d9d0 2024-04-02T07:36:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.666+0000 7fa798800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7900d2820 con 0x7fa7900d2370 2024-04-02T07:36:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.666+0000 7fa798800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7900d4340 con 0x7fa7900d16a0 2024-04-02T07:36:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.667+0000 7fa796800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7900d2370 0x7fa7900ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.667+0000 7fa795e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa79015d9d0 0x7fa7900d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46870/0 (socket says 172.21.15.89:46870) 2024-04-02T07:36:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.667+0000 7fa795e00700 1 -- 172.21.15.89:0/3788852767 learned_addr learned my addr 172.21.15.89:0/3788852767 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.667+0000 7fa795400700 1 --2- 172.21.15.89:0/3788852767 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7900d16a0 0x7fa7900d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.667+0000 7fa794a00700 1 -- 172.21.15.89:0/3788852767 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783896601 0 0) 0x7fa790161420 con 0x7fa79015d9d0 2024-04-02T07:36:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.667+0000 7fa796800700 1 -- 172.21.15.89:0/3788852767 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7900d16a0 msgr2=0x7fa7900d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.667+0000 7fa796800700 1 --2- 172.21.15.89:0/3788852767 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7900d16a0 0x7fa7900d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.667+0000 7fa796800700 1 -- 172.21.15.89:0/3788852767 >> v1:172.21.15.89:6789/0 conn(0x7fa79015d9d0 legacy=0x7fa7900d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.667+0000 7fa796800700 1 -- 172.21.15.89:0/3788852767 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa790161700 con 0x7fa7900d2370 2024-04-02T07:36:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.668+0000 7fa796800700 1 --2- 172.21.15.89:0/3788852767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d2370 0x7fa7900ce360 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa78c009dc0 tx=0x7fa78c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209ee157bb3d8e8d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.668+0000 7fa794a00700 1 -- 172.21.15.89:0/3788852767 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c00e070 con 0x7fa7900d2370 2024-04-02T07:36:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.668+0000 7fa794a00700 1 -- 172.21.15.89:0/3788852767 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78c00bb90 con 0x7fa7900d2370 2024-04-02T07:36:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.668+0000 7fa794a00700 1 -- 172.21.15.89:0/3788852767 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c012430 con 0x7fa7900d2370 2024-04-02T07:36:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.668+0000 7fa798800700 1 -- 172.21.15.89:0/3788852767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d2370 msgr2=0x7fa7900ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.668+0000 7fa798800700 1 --2- 172.21.15.89:0/3788852767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d2370 0x7fa7900ce360 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa78c009dc0 tx=0x7fa78c00b330 comp rx=0 tx=0).stop 2024-04-02T07:36:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.668+0000 7fa798800700 1 -- 172.21.15.89:0/3788852767 shutdown_connections 2024-04-02T07:36:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.668+0000 7fa798800700 1 --2- 172.21.15.89:0/3788852767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d2370 0x7fa7900ce360 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.668+0000 7fa798800700 1 --2- 172.21.15.89:0/3788852767 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7900d16a0 0x7fa7900d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.668+0000 7fa798800700 1 -- 172.21.15.89:0/3788852767 >> 172.21.15.89:0/3788852767 conn(0x7fa79008aff0 msgr2=0x7fa7900d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.668+0000 7fa798800700 1 -- 172.21.15.89:0/3788852767 shutdown_connections 2024-04-02T07:36:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.668+0000 7fa798800700 1 -- 172.21.15.89:0/3788852767 wait complete. 2024-04-02T07:36:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.670+0000 7fa798800700 1 Processor -- start 2024-04-02T07:36:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.670+0000 7fa798800700 1 -- start start 2024-04-02T07:36:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.671+0000 7fa798800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7900d16a0 0x7fa7901f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.671+0000 7fa798800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d2370 0x7fa79015f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.671+0000 7fa798800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa79015d9d0 0x7fa7900b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.671+0000 7fa798800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa790160a80 con 0x7fa7900d2370 2024-04-02T07:36:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.671+0000 7fa798800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901f64e0 con 0x7fa79015d9d0 2024-04-02T07:36:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.671+0000 7fa798800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa79015fe00 con 0x7fa7900d16a0 2024-04-02T07:36:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.671+0000 7fa795400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d2370 0x7fa79015f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.671+0000 7fa795400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d2370 0x7fa79015f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40686/0 (socket says 172.21.15.89:40686) 2024-04-02T07:36:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.671+0000 7fa795400700 1 -- 172.21.15.89:0/1084424100 learned_addr learned my addr 172.21.15.89:0/1084424100 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.672+0000 7fa796800700 1 --2- 172.21.15.89:0/1084424100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa79015d9d0 0x7fa7900b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.672+0000 7fa795e00700 1 --2- 172.21.15.89:0/1084424100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7900d16a0 0x7fa7901f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.672+0000 7fa795400700 1 -- 172.21.15.89:0/1084424100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7900d16a0 msgr2=0x7fa7901f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.672+0000 7fa795400700 1 --2- 172.21.15.89:0/1084424100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7900d16a0 0x7fa7901f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.672+0000 7fa795400700 1 -- 172.21.15.89:0/1084424100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa79015d9d0 msgr2=0x7fa7900b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.672+0000 7fa795400700 1 --2- 172.21.15.89:0/1084424100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa79015d9d0 0x7fa7900b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.672+0000 7fa795400700 1 -- 172.21.15.89:0/1084424100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c009a70 con 0x7fa7900d2370 2024-04-02T07:36:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.672+0000 7fa795400700 1 --2- 172.21.15.89:0/1084424100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d2370 0x7fa79015f420 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa78000e810 tx=0x7fa78000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.672+0000 7fa78a000700 1 -- 172.21.15.89:0/1084424100 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780013610 con 0x7fa7900d2370 2024-04-02T07:36:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.672+0000 7fa78a000700 1 -- 172.21.15.89:0/1084424100 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa780013770 con 0x7fa7900d2370 2024-04-02T07:36:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.672+0000 7fa78a000700 1 -- 172.21.15.89:0/1084424100 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780013b20 con 0x7fa7900d2370 2024-04-02T07:36:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.672+0000 7fa798800700 1 -- 172.21.15.89:0/1084424100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7901600e0 con 0x7fa7900d2370 2024-04-02T07:36:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.672+0000 7fa798800700 1 -- 172.21.15.89:0/1084424100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7901606a0 con 0x7fa7900d2370 2024-04-02T07:36:30.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.673+0000 7fa798800700 1 -- 172.21.15.89:0/1084424100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7900d1ca0 con 0x7fa7900d2370 2024-04-02T07:36:30.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.674+0000 7fa78a000700 1 -- 172.21.15.89:0/1084424100 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa780013c80 con 0x7fa7900d2370 2024-04-02T07:36:30.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.675+0000 7fa78a000700 1 --2- 172.21.15.89:0/1084424100 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa77406ac90 0x7fa77406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:30.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.675+0000 7fa78a000700 1 -- 172.21.15.89:0/1084424100 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7800890c0 con 0x7fa7900d2370 2024-04-02T07:36:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.675+0000 7fa795e00700 1 --2- 172.21.15.89:0/1084424100 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa77406ac90 0x7fa77406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.676+0000 7fa795e00700 1 --2- 172.21.15.89:0/1084424100 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa77406ac90 0x7fa77406d150 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa784000f60 tx=0x7fa784007c90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:30.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.679+0000 7fa78a000700 1 -- 172.21.15.89:0/1084424100 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa780054210 con 0x7fa7900d2370 2024-04-02T07:36:30.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.880+0000 7fa798800700 1 -- 172.21.15.89:0/1084424100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa7900ce870 con 0x7fa7900d2370 2024-04-02T07:36:30.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.881+0000 7fa78a000700 1 -- 172.21.15.89:0/1084424100 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa780057830 con 0x7fa7900d2370 2024-04-02T07:36:30.883 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:36:30.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.883+0000 7fa798800700 1 -- 172.21.15.89:0/1084424100 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa77406ac90 msgr2=0x7fa77406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:30.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.883+0000 7fa798800700 1 --2- 172.21.15.89:0/1084424100 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa77406ac90 0x7fa77406d150 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa784000f60 tx=0x7fa784007c90 comp rx=0 tx=0).stop 2024-04-02T07:36:30.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.883+0000 7fa798800700 1 -- 172.21.15.89:0/1084424100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d2370 msgr2=0x7fa79015f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:30.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.883+0000 7fa798800700 1 --2- 172.21.15.89:0/1084424100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d2370 0x7fa79015f420 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa78000e810 tx=0x7fa78000ebd0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.884+0000 7fa798800700 1 -- 172.21.15.89:0/1084424100 shutdown_connections 2024-04-02T07:36:30.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.884+0000 7fa798800700 1 --2- 172.21.15.89:0/1084424100 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa77406ac90 0x7fa77406d150 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.884+0000 7fa798800700 1 --2- 172.21.15.89:0/1084424100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa79015d9d0 0x7fa7900b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.884+0000 7fa798800700 1 --2- 172.21.15.89:0/1084424100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7900d16a0 0x7fa7901f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.884+0000 7fa798800700 1 --2- 172.21.15.89:0/1084424100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7900d2370 0x7fa79015f420 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:30.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.884+0000 7fa798800700 1 -- 172.21.15.89:0/1084424100 >> 172.21.15.89:0/1084424100 conn(0x7fa79008aff0 msgr2=0x7fa79015ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:30.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.884+0000 7fa798800700 1 -- 172.21.15.89:0/1084424100 shutdown_connections 2024-04-02T07:36:30.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:30.884+0000 7fa798800700 1 -- 172.21.15.89:0/1084424100 wait complete. 2024-04-02T07:36:30.937 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2024-04-02T07:36:30.937 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-02T07:36:30.937 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-04-02T07:36:31.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.125+0000 7fe36e800700 1 Processor -- start 2024-04-02T07:36:31.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.125+0000 7fe36e800700 1 -- start start 2024-04-02T07:36:31.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.126+0000 7fe36e800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe368160f10 0x7fe3680d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:31.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.126+0000 7fe36e800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe3680cf230 0x7fe3680d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:31.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.126+0000 7fe36e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe368161590 0x7fe368161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:31.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.126+0000 7fe36e800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe368161f60 con 0x7fe368160f10 2024-04-02T07:36:31.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.126+0000 7fe36e800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3680ceda0 con 0x7fe3680cf230 2024-04-02T07:36:31.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.126+0000 7fe36e800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3680d23d0 con 0x7fe368161590 2024-04-02T07:36:31.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.127+0000 7fe367400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe368160f10 0x7fe3680d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:31.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.127+0000 7fe367400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe368160f10 0x7fe3680d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40696/0 (socket says 172.21.15.89:40696) 2024-04-02T07:36:31.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.127+0000 7fe367400700 1 -- 172.21.15.89:0/4255401278 learned_addr learned my addr 172.21.15.89:0/4255401278 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:31.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.127+0000 7fe366a00700 1 --2- 172.21.15.89:0/4255401278 >> v2:172.21.15.113:3300/0 conn(0x7fe3680cf230 0x7fe3680d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:31.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.127+0000 7fe367e00700 1 --2- 172.21.15.89:0/4255401278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe368161590 0x7fe368161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:31.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.127+0000 7fe367400700 1 -- 172.21.15.89:0/4255401278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe368161590 msgr2=0x7fe368161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:31.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.127+0000 7fe367400700 1 --2- 172.21.15.89:0/4255401278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe368161590 0x7fe368161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:31.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.127+0000 7fe367400700 1 -- 172.21.15.89:0/4255401278 >> v2:172.21.15.113:3300/0 conn(0x7fe3680cf230 msgr2=0x7fe3680d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:31.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.127+0000 7fe367400700 1 --2- 172.21.15.89:0/4255401278 >> v2:172.21.15.113:3300/0 conn(0x7fe3680cf230 0x7fe3680d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:31.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.127+0000 7fe367400700 1 -- 172.21.15.89:0/4255401278 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3681612f0 con 0x7fe368160f10 2024-04-02T07:36:31.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.127+0000 7fe366a00700 1 --2- 172.21.15.89:0/4255401278 >> v2:172.21.15.113:3300/0 conn(0x7fe3680cf230 0x7fe3680d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:31.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.127+0000 7fe367e00700 1 --2- 172.21.15.89:0/4255401278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe368161590 0x7fe368161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:31.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.127+0000 7fe367400700 1 --2- 172.21.15.89:0/4255401278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368160f10 0x7fe3680d18b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe36808cdb0 tx=0x7fe358009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=777bf4aef80f5fe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:31.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.128+0000 7fe366000700 1 -- 172.21.15.89:0/4255401278 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35800b870 con 0x7fe368160f10 2024-04-02T07:36:31.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.128+0000 7fe366000700 1 -- 172.21.15.89:0/4255401278 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35800b9d0 con 0x7fe368160f10 2024-04-02T07:36:31.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.128+0000 7fe366000700 1 -- 172.21.15.89:0/4255401278 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35800bd60 con 0x7fe368160f10 2024-04-02T07:36:31.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.128+0000 7fe36e800700 1 -- 172.21.15.89:0/4255401278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368160f10 msgr2=0x7fe3680d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:31.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.128+0000 7fe36e800700 1 --2- 172.21.15.89:0/4255401278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368160f10 0x7fe3680d18b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe36808cdb0 tx=0x7fe358009bd0 comp rx=0 tx=0).stop 2024-04-02T07:36:31.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.128+0000 7fe36e800700 1 -- 172.21.15.89:0/4255401278 shutdown_connections 2024-04-02T07:36:31.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.128+0000 7fe36e800700 1 --2- 172.21.15.89:0/4255401278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368160f10 0x7fe3680d18b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:31.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.128+0000 7fe36e800700 1 --2- 172.21.15.89:0/4255401278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe368161590 0x7fe368161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:31.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.128+0000 7fe36e800700 1 --2- 172.21.15.89:0/4255401278 >> v2:172.21.15.113:3300/0 conn(0x7fe3680cf230 0x7fe3680d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:31.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.128+0000 7fe36e800700 1 -- 172.21.15.89:0/4255401278 >> 172.21.15.89:0/4255401278 conn(0x7fe36808aff0 msgr2=0x7fe3680d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:31.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.128+0000 7fe36e800700 1 -- 172.21.15.89:0/4255401278 shutdown_connections 2024-04-02T07:36:31.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.128+0000 7fe36e800700 1 -- 172.21.15.89:0/4255401278 wait complete. 2024-04-02T07:36:31.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.130+0000 7fe36e800700 1 Processor -- start 2024-04-02T07:36:31.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.130+0000 7fe36e800700 1 -- start start 2024-04-02T07:36:31.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.131+0000 7fe36e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3680cf230 0x7fe36814ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:31.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.131+0000 7fe36e800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe368160f10 0x7fe3680b8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:31.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.131+0000 7fe36e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368161590 0x7fe36815ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:31.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.131+0000 7fe36e800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36814efa0 con 0x7fe368161590 2024-04-02T07:36:31.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.131+0000 7fe36e800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3680b9480 con 0x7fe368160f10 2024-04-02T07:36:31.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.131+0000 7fe36e800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36815e7c0 con 0x7fe3680cf230 2024-04-02T07:36:31.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.131+0000 7fe367e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368161590 0x7fe36815ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:31.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.131+0000 7fe367e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368161590 0x7fe36815ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40712/0 (socket says 172.21.15.89:40712) 2024-04-02T07:36:31.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.131+0000 7fe367e00700 1 -- 172.21.15.89:0/3050903433 learned_addr learned my addr 172.21.15.89:0/3050903433 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:31.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.131+0000 7fe366a00700 1 --2- 172.21.15.89:0/3050903433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe368160f10 0x7fe3680b8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:31.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.131+0000 7fe367400700 1 --2- 172.21.15.89:0/3050903433 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3680cf230 0x7fe36814ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:31.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.132+0000 7fe367e00700 1 -- 172.21.15.89:0/3050903433 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3680cf230 msgr2=0x7fe36814ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:31.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.132+0000 7fe367e00700 1 --2- 172.21.15.89:0/3050903433 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3680cf230 0x7fe36814ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:31.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.132+0000 7fe367e00700 1 -- 172.21.15.89:0/3050903433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe368160f10 msgr2=0x7fe3680b8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:31.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.132+0000 7fe367e00700 1 --2- 172.21.15.89:0/3050903433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe368160f10 0x7fe3680b8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:31.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.132+0000 7fe367e00700 1 -- 172.21.15.89:0/3050903433 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe358009550 con 0x7fe368161590 2024-04-02T07:36:31.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.132+0000 7fe367e00700 1 --2- 172.21.15.89:0/3050903433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368161590 0x7fe36815ef70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe36000cd50 tx=0x7fe36000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:31.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.132+0000 7fe35f400700 1 -- 172.21.15.89:0/3050903433 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36000cea0 con 0x7fe368161590 2024-04-02T07:36:31.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.132+0000 7fe35f400700 1 -- 172.21.15.89:0/3050903433 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36000f480 con 0x7fe368161590 2024-04-02T07:36:31.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.132+0000 7fe36e800700 1 -- 172.21.15.89:0/3050903433 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3680b6c60 con 0x7fe368161590 2024-04-02T07:36:31.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.132+0000 7fe35f400700 1 -- 172.21.15.89:0/3050903433 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36000f830 con 0x7fe368161590 2024-04-02T07:36:31.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.132+0000 7fe36e800700 1 -- 172.21.15.89:0/3050903433 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3680b6f00 con 0x7fe368161590 2024-04-02T07:36:31.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.133+0000 7fe36e800700 1 -- 172.21.15.89:0/3050903433 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe368067720 con 0x7fe368161590 2024-04-02T07:36:31.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.134+0000 7fe35f400700 1 -- 172.21.15.89:0/3050903433 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe36000b040 con 0x7fe368161590 2024-04-02T07:36:31.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.135+0000 7fe35f400700 1 --2- 172.21.15.89:0/3050903433 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe34806ac90 0x7fe34806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:31.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.135+0000 7fe35f400700 1 -- 172.21.15.89:0/3050903433 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe360089950 con 0x7fe368161590 2024-04-02T07:36:31.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.135+0000 7fe367400700 1 --2- 172.21.15.89:0/3050903433 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe34806ac90 0x7fe34806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:31.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.136+0000 7fe367400700 1 --2- 172.21.15.89:0/3050903433 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe34806ac90 0x7fe34806d150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe358014040 tx=0x7fe358008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:31.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.139+0000 7fe35f400700 1 -- 172.21.15.89:0/3050903433 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe360054b20 con 0x7fe368161590 2024-04-02T07:36:31.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.419+0000 7fe36e800700 1 -- 172.21.15.89:0/3050903433 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7fe3680cd390 con 0x7fe368161590 2024-04-02T07:36:31.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.843+0000 7fe35f400700 1 -- 172.21.15.89:0/3050903433 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v58) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe360058140 con 0x7fe368161590 2024-04-02T07:36:31.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.845+0000 7fe36e800700 1 -- 172.21.15.89:0/3050903433 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe34806ac90 msgr2=0x7fe34806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.845+0000 7fe36e800700 1 --2- 172.21.15.89:0/3050903433 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe34806ac90 0x7fe34806d150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe358014040 tx=0x7fe358008030 comp rx=0 tx=0).stop 2024-04-02T07:36:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.845+0000 7fe36e800700 1 -- 172.21.15.89:0/3050903433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368161590 msgr2=0x7fe36815ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.845+0000 7fe36e800700 1 --2- 172.21.15.89:0/3050903433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368161590 0x7fe36815ef70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe36000cd50 tx=0x7fe36000a560 comp rx=0 tx=0).stop 2024-04-02T07:36:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.845+0000 7fe36e800700 1 -- 172.21.15.89:0/3050903433 shutdown_connections 2024-04-02T07:36:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.845+0000 7fe36e800700 1 --2- 172.21.15.89:0/3050903433 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe34806ac90 0x7fe34806d150 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.845+0000 7fe36e800700 1 --2- 172.21.15.89:0/3050903433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368161590 0x7fe36815ef70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.845+0000 7fe36e800700 1 --2- 172.21.15.89:0/3050903433 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3680cf230 0x7fe36814ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.845+0000 7fe36e800700 1 --2- 172.21.15.89:0/3050903433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe368160f10 0x7fe3680b8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.845+0000 7fe36e800700 1 -- 172.21.15.89:0/3050903433 >> 172.21.15.89:0/3050903433 conn(0x7fe36808aff0 msgr2=0x7fe3680d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.846+0000 7fe36e800700 1 -- 172.21.15.89:0/3050903433 shutdown_connections 2024-04-02T07:36:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:31.846+0000 7fe36e800700 1 -- 172.21.15.89:0/3050903433 wait complete. 2024-04-02T07:36:31.866 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T07:36:32.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.059+0000 7f6973400700 1 Processor -- start 2024-04-02T07:36:32.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.059+0000 7f6973400700 1 -- start start 2024-04-02T07:36:32.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.060+0000 7f6973400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f696c15d9d0 0x7f696c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.061+0000 7f6973400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f696c0d2540 0x7f696c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.061+0000 7f6973400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c161420 con 0x7f696c0d16a0 2024-04-02T07:36:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.061+0000 7f6973400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696c15edc0 con 0x7f696c15d9d0 2024-04-02T07:36:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.061+0000 7f6973400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c161700 con 0x7f696c0d2540 2024-04-02T07:36:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.061+0000 7f696be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f696c0d16a0 0x7f696c0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46874/0 (socket says 172.21.15.89:46874) 2024-04-02T07:36:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.061+0000 7f696be00700 1 -- 172.21.15.89:0/3276021904 learned_addr learned my addr 172.21.15.89:0/3276021904 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:32.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.061+0000 7f6970a00700 1 --2- 172.21.15.89:0/3276021904 >> v2:172.21.15.113:3300/0 conn(0x7f696c15d9d0 0x7f696c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.061+0000 7f6971400700 1 --2- 172.21.15.89:0/3276021904 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f696c0d2540 0x7f696c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.061+0000 7f696b400700 1 -- 172.21.15.89:0/3276021904 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317966713 0 0) 0x7f696c161420 con 0x7f696c0d16a0 2024-04-02T07:36:32.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.061+0000 7f696b400700 1 -- 172.21.15.89:0/3276021904 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69540035b0 con 0x7f696c0d16a0 2024-04-02T07:36:32.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696b400700 1 -- 172.21.15.89:0/3276021904 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1755533703 0 0) 0x7f69540035b0 con 0x7f696c0d16a0 2024-04-02T07:36:32.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696b400700 1 -- 172.21.15.89:0/3276021904 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f696c0d2540 msgr2=0x7f696c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:32.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696b400700 1 --2- 172.21.15.89:0/3276021904 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f696c0d2540 0x7f696c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696b400700 1 -- 172.21.15.89:0/3276021904 >> v2:172.21.15.113:3300/0 conn(0x7f696c15d9d0 msgr2=0x7f696c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:32.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696b400700 1 --2- 172.21.15.89:0/3276021904 >> v2:172.21.15.113:3300/0 conn(0x7f696c15d9d0 0x7f696c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696b400700 1 -- 172.21.15.89:0/3276021904 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c1619e0 con 0x7f696c0d16a0 2024-04-02T07:36:32.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696b400700 1 -- 172.21.15.89:0/3276021904 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f69580017f0 con 0x7f696c0d16a0 2024-04-02T07:36:32.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696b400700 1 -- 172.21.15.89:0/3276021904 >> v1:172.21.15.89:6789/0 conn(0x7f696c0d16a0 legacy=0x7f696c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:32.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696b400700 1 --2- 172.21.15.89:0/3276021904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6954003a00 0x7f6954003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696b400700 1 --2- 172.21.15.89:0/3276021904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6954004850 0x7f6954004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696b400700 1 --2- 172.21.15.89:0/3276021904 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f696c0d2540 0x7f6954005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696b400700 1 -- 172.21.15.89:0/3276021904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c161700 con 0x7f6954004850 2024-04-02T07:36:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696b400700 1 -- 172.21.15.89:0/3276021904 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695400ba10 con 0x7f6954003a00 2024-04-02T07:36:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696b400700 1 -- 172.21.15.89:0/3276021904 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695400bb50 con 0x7f696c0d2540 2024-04-02T07:36:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696be00700 1 --2- 172.21.15.89:0/3276021904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6954004850 0x7f6954004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f6970a00700 1 --2- 172.21.15.89:0/3276021904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6954003a00 0x7f6954003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f6971400700 1 --2- 172.21.15.89:0/3276021904 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f696c0d2540 0x7f6954005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696be00700 1 -- 172.21.15.89:0/3276021904 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f696c0d2540 msgr2=0x7f6954005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696be00700 1 --2- 172.21.15.89:0/3276021904 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f696c0d2540 0x7f6954005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696be00700 1 -- 172.21.15.89:0/3276021904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6954003a00 msgr2=0x7f6954003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696be00700 1 --2- 172.21.15.89:0/3276021904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6954003a00 0x7f6954003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f696be00700 1 -- 172.21.15.89:0/3276021904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69540035b0 con 0x7f6954004850 2024-04-02T07:36:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.062+0000 7f6971400700 1 --2- 172.21.15.89:0/3276021904 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f696c0d2540 0x7f6954005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.063+0000 7f696be00700 1 --2- 172.21.15.89:0/3276021904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6954004850 0x7f6954004cd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f695800c2f0 tx=0x7f6954005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aba2d88ff1fa843 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.063+0000 7f696b400700 1 -- 172.21.15.89:0/3276021904 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695800ce30 con 0x7f6954004850 2024-04-02T07:36:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.063+0000 7f696b400700 1 -- 172.21.15.89:0/3276021904 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6958007bf0 con 0x7f6954004850 2024-04-02T07:36:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.063+0000 7f696b400700 1 -- 172.21.15.89:0/3276021904 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6958007310 con 0x7f6954004850 2024-04-02T07:36:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.063+0000 7f6973400700 1 -- 172.21.15.89:0/3276021904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6954004850 msgr2=0x7f6954004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.063+0000 7f6973400700 1 --2- 172.21.15.89:0/3276021904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6954004850 0x7f6954004cd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f695800c2f0 tx=0x7f6954005880 comp rx=0 tx=0).stop 2024-04-02T07:36:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.063+0000 7f6971400700 1 -- 172.21.15.89:0/3276021904 reap_dead start 2024-04-02T07:36:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.063+0000 7f6973400700 1 -- 172.21.15.89:0/3276021904 shutdown_connections 2024-04-02T07:36:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.063+0000 7f6973400700 1 -- 172.21.15.89:0/3276021904 >> 172.21.15.89:0/3276021904 conn(0x7f696c08aff0 msgr2=0x7f696c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.063+0000 7f6973400700 1 -- 172.21.15.89:0/3276021904 shutdown_connections 2024-04-02T07:36:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.063+0000 7f6973400700 1 -- 172.21.15.89:0/3276021904 wait complete. 2024-04-02T07:36:32.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.065+0000 7f6973400700 1 Processor -- start 2024-04-02T07:36:32.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.065+0000 7f6973400700 1 -- start start 2024-04-02T07:36:32.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.066+0000 7f6973400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6954004850 0x7f696c155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.066+0000 7f6973400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f696c15f5d0 0x7f696c15fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.066+0000 7f6973400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f696c0b4ce0 0x7f696c0b5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.066+0000 7f6973400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0ccf70 con 0x7f6954004850 2024-04-02T07:36:32.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.066+0000 7f6973400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0b56a0 con 0x7f696c15f5d0 2024-04-02T07:36:32.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.066+0000 7f6973400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0cb080 con 0x7f696c0b4ce0 2024-04-02T07:36:32.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.066+0000 7f6970a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6954004850 0x7f696c155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f6970a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6954004850 0x7f696c155360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40726/0 (socket says 172.21.15.89:40726) 2024-04-02T07:36:32.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f6970a00700 1 -- 172.21.15.89:0/2100770881 learned_addr learned my addr 172.21.15.89:0/2100770881 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:32.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f6971400700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f696c0b4ce0 0x7f696c0b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f696be00700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f696c15f5d0 0x7f696c15fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f6970a00700 1 -- 172.21.15.89:0/2100770881 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f696c0b4ce0 msgr2=0x7f696c0b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:32.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f6970a00700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f696c0b4ce0 0x7f696c0b5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f6970a00700 1 -- 172.21.15.89:0/2100770881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f696c15f5d0 msgr2=0x7f696c15fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:32.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f6970a00700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f696c15f5d0 0x7f696c15fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f6970a00700 1 -- 172.21.15.89:0/2100770881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c1558a0 con 0x7f6954004850 2024-04-02T07:36:32.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f696be00700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f696c15f5d0 0x7f696c15fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:32.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f6970a00700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6954004850 0x7f696c155360 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6960011aa0 tx=0x7f6960011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:32.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f6968c00700 1 -- 172.21.15.89:0/2100770881 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696000f980 con 0x7f6954004850 2024-04-02T07:36:32.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f6968c00700 1 -- 172.21.15.89:0/2100770881 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696000fb00 con 0x7f6954004850 2024-04-02T07:36:32.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f6973400700 1 -- 172.21.15.89:0/2100770881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c0ca790 con 0x7f6954004850 2024-04-02T07:36:32.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f6968c00700 1 -- 172.21.15.89:0/2100770881 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696000feb0 con 0x7f6954004850 2024-04-02T07:36:32.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.067+0000 7f6973400700 1 -- 172.21.15.89:0/2100770881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c0caa30 con 0x7f6954004850 2024-04-02T07:36:32.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.068+0000 7f6973400700 1 -- 172.21.15.89:0/2100770881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c0cb360 con 0x7f6954004850 2024-04-02T07:36:32.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.069+0000 7f6968c00700 1 -- 172.21.15.89:0/2100770881 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6960014440 con 0x7f6954004850 2024-04-02T07:36:32.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.070+0000 7f6968c00700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f694c06ac90 0x7f694c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.070+0000 7f6968c00700 1 -- 172.21.15.89:0/2100770881 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6960089b30 con 0x7f6954004850 2024-04-02T07:36:32.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.070+0000 7f696be00700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f694c06ac90 0x7f694c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.071+0000 7f696be00700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f694c06ac90 0x7f694c06d150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6958015040 tx=0x7f695800a000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:32.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.074+0000 7f6968c00700 1 -- 172.21.15.89:0/2100770881 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6960054dd0 con 0x7f6954004850 2024-04-02T07:36:32.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.279+0000 7f6973400700 1 -- 172.21.15.89:0/2100770881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f696c0d1ca0 con 0x7f6954004850 2024-04-02T07:36:32.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.283+0000 7f6968c00700 1 -- 172.21.15.89:0/2100770881 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f69600583f0 con 0x7f6954004850 2024-04-02T07:36:32.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.285+0000 7f6973400700 1 -- 172.21.15.89:0/2100770881 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f694c06ac90 msgr2=0x7f694c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:32.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.285+0000 7f6973400700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f694c06ac90 0x7f694c06d150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6958015040 tx=0x7f695800a000 comp rx=0 tx=0).stop 2024-04-02T07:36:32.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.285+0000 7f6973400700 1 -- 172.21.15.89:0/2100770881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6954004850 msgr2=0x7f696c155360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:32.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.285+0000 7f6973400700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6954004850 0x7f696c155360 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6960011aa0 tx=0x7f6960011e60 comp rx=0 tx=0).stop 2024-04-02T07:36:32.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.286+0000 7f6973400700 1 -- 172.21.15.89:0/2100770881 shutdown_connections 2024-04-02T07:36:32.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.286+0000 7f6973400700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f694c06ac90 0x7f694c06d150 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.286+0000 7f6973400700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f696c0b4ce0 0x7f696c0b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.286+0000 7f6973400700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6954004850 0x7f696c155360 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.286+0000 7f6973400700 1 --2- 172.21.15.89:0/2100770881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f696c15f5d0 0x7f696c15fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.286+0000 7f6973400700 1 -- 172.21.15.89:0/2100770881 >> 172.21.15.89:0/2100770881 conn(0x7f696c08aff0 msgr2=0x7f696c0b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:32.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.286+0000 7f6973400700 1 -- 172.21.15.89:0/2100770881 shutdown_connections 2024-04-02T07:36:32.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.286+0000 7f6973400700 1 -- 172.21.15.89:0/2100770881 wait complete. 2024-04-02T07:36:32.306 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:36:32.306 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:36:32.390 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:36:32.482 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:36:32.482 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:36:32.482 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:32.527 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:36:32.527 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:32.612 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:36:32.612 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:32.697 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:36:32.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.927+0000 7fb126a00700 1 Processor -- start 2024-04-02T07:36:32.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.927+0000 7fb126a00700 1 -- start start 2024-04-02T07:36:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.928+0000 7fb126a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1200d18b0 0x7fb1200cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.928+0000 7fb126a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb120161590 0x7fb120161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.928+0000 7fb126a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1200ceda0 con 0x7fb120160f10 2024-04-02T07:36:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.928+0000 7fb126a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1200d1d30 con 0x7fb120161590 2024-04-02T07:36:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.928+0000 7fb126a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb120161f60 con 0x7fb1200d18b0 2024-04-02T07:36:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.929+0000 7fb11fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb120160f10 0x7fb1200d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46878/0 (socket says 172.21.15.89:46878) 2024-04-02T07:36:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.929+0000 7fb11fe00700 1 -- 172.21.15.89:0/532419892 learned_addr learned my addr 172.21.15.89:0/532419892 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.929+0000 7fb124a00700 1 --2- 172.21.15.89:0/532419892 >> v2:172.21.15.89:3300/0 conn(0x7fb120161590 0x7fb120161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.929+0000 7fb11f400700 1 --2- 172.21.15.89:0/532419892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1200d18b0 0x7fb1200cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.929+0000 7fb11ea00700 1 -- 172.21.15.89:0/532419892 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189878545 0 0) 0x7fb1200ceda0 con 0x7fb120160f10 2024-04-02T07:36:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.929+0000 7fb11ea00700 1 -- 172.21.15.89:0/532419892 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1040035b0 con 0x7fb120160f10 2024-04-02T07:36:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.929+0000 7fb124a00700 1 -- 172.21.15.89:0/532419892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1200d18b0 msgr2=0x7fb1200cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.929+0000 7fb124a00700 1 --2- 172.21.15.89:0/532419892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1200d18b0 0x7fb1200cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.929+0000 7fb124a00700 1 -- 172.21.15.89:0/532419892 >> v1:172.21.15.89:6789/0 conn(0x7fb120160f10 legacy=0x7fb1200d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.929+0000 7fb124a00700 1 -- 172.21.15.89:0/532419892 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1201612f0 con 0x7fb120161590 2024-04-02T07:36:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.929+0000 7fb124a00700 1 --2- 172.21.15.89:0/532419892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb120161590 0x7fb120161a20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb118009dc0 tx=0x7fb11800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7504d734cd881499 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:32.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.930+0000 7fb11ea00700 1 -- 172.21.15.89:0/532419892 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11800e070 con 0x7fb120161590 2024-04-02T07:36:32.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.930+0000 7fb11ea00700 1 -- 172.21.15.89:0/532419892 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11800bae0 con 0x7fb120161590 2024-04-02T07:36:32.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.930+0000 7fb11ea00700 1 -- 172.21.15.89:0/532419892 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11800be90 con 0x7fb120161590 2024-04-02T07:36:32.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.930+0000 7fb126a00700 1 -- 172.21.15.89:0/532419892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb120161590 msgr2=0x7fb120161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:32.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.930+0000 7fb126a00700 1 --2- 172.21.15.89:0/532419892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb120161590 0x7fb120161a20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb118009dc0 tx=0x7fb11800b330 comp rx=0 tx=0).stop 2024-04-02T07:36:32.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.930+0000 7fb126a00700 1 -- 172.21.15.89:0/532419892 shutdown_connections 2024-04-02T07:36:32.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.930+0000 7fb126a00700 1 --2- 172.21.15.89:0/532419892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb120161590 0x7fb120161a20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.930+0000 7fb126a00700 1 --2- 172.21.15.89:0/532419892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1200d18b0 0x7fb1200cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.930+0000 7fb126a00700 1 -- 172.21.15.89:0/532419892 >> 172.21.15.89:0/532419892 conn(0x7fb12008aff0 msgr2=0x7fb1200d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:32.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.930+0000 7fb126a00700 1 -- 172.21.15.89:0/532419892 shutdown_connections 2024-04-02T07:36:32.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.930+0000 7fb126a00700 1 -- 172.21.15.89:0/532419892 wait complete. 2024-04-02T07:36:32.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.932+0000 7fb126a00700 1 Processor -- start 2024-04-02T07:36:32.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.932+0000 7fb126a00700 1 -- start start 2024-04-02T07:36:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.933+0000 7fb126a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1200d18b0 0x7fb120158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.933+0000 7fb126a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb120160f10 0x7fb1200b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.933+0000 7fb126a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb120161590 0x7fb12006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.933+0000 7fb126a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb120158fb0 con 0x7fb120161590 2024-04-02T07:36:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.933+0000 7fb126a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1200b7170 con 0x7fb1200d18b0 2024-04-02T07:36:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.933+0000 7fb126a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1201f3e80 con 0x7fb120160f10 2024-04-02T07:36:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.933+0000 7fb124a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb120161590 0x7fb12006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.933+0000 7fb124a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb120161590 0x7fb12006c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40744/0 (socket says 172.21.15.89:40744) 2024-04-02T07:36:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.933+0000 7fb124a00700 1 -- 172.21.15.89:0/1130637587 learned_addr learned my addr 172.21.15.89:0/1130637587 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.933+0000 7fb11fe00700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1200d18b0 0x7fb120158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.933+0000 7fb11f400700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb120160f10 0x7fb1200b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.934+0000 7fb124a00700 1 -- 172.21.15.89:0/1130637587 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb120160f10 msgr2=0x7fb1200b7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.934+0000 7fb124a00700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb120160f10 0x7fb1200b7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.934+0000 7fb124a00700 1 -- 172.21.15.89:0/1130637587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1200d18b0 msgr2=0x7fb120158a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.934+0000 7fb124a00700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1200d18b0 0x7fb120158a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.934+0000 7fb124a00700 1 -- 172.21.15.89:0/1130637587 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb118009a70 con 0x7fb120161590 2024-04-02T07:36:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.934+0000 7fb11f400700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb120160f10 0x7fb1200b7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.934+0000 7fb124a00700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb120161590 0x7fb12006c540 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb11800b370 tx=0x7fb118004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.934+0000 7fb117e00700 1 -- 172.21.15.89:0/1130637587 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb118004550 con 0x7fb120161590 2024-04-02T07:36:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.934+0000 7fb117e00700 1 -- 172.21.15.89:0/1130637587 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb118020070 con 0x7fb120161590 2024-04-02T07:36:32.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.934+0000 7fb126a00700 1 -- 172.21.15.89:0/1130637587 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1201f4160 con 0x7fb120161590 2024-04-02T07:36:32.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.934+0000 7fb117e00700 1 -- 172.21.15.89:0/1130637587 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1180048f0 con 0x7fb120161590 2024-04-02T07:36:32.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.934+0000 7fb126a00700 1 -- 172.21.15.89:0/1130637587 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1200b7710 con 0x7fb120161590 2024-04-02T07:36:32.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.936+0000 7fb126a00700 1 -- 172.21.15.89:0/1130637587 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1200b7900 con 0x7fb120161590 2024-04-02T07:36:32.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.936+0000 7fb117e00700 1 -- 172.21.15.89:0/1130637587 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb118012430 con 0x7fb120161590 2024-04-02T07:36:32.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.937+0000 7fb117e00700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb10006ac90 0x7fb10006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:32.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.937+0000 7fb117e00700 1 -- 172.21.15.89:0/1130637587 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb118022070 con 0x7fb120161590 2024-04-02T07:36:32.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.937+0000 7fb11fe00700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb10006ac90 0x7fb10006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:32.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.938+0000 7fb11fe00700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb10006ac90 0x7fb10006d150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb110007fd0 tx=0x7fb110007dc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:32.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:32.942+0000 7fb117e00700 1 -- 172.21.15.89:0/1130637587 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb118059ea0 con 0x7fb120161590 2024-04-02T07:36:33.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.145+0000 7fb126a00700 1 -- 172.21.15.89:0/1130637587 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb120067720 con 0x7fb120161590 2024-04-02T07:36:33.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.146+0000 7fb117e00700 1 -- 172.21.15.89:0/1130637587 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7fb118059a30 con 0x7fb120161590 2024-04-02T07:36:33.148 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:33.148 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:54.238913+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:36:33.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.148+0000 7fb126a00700 1 -- 172.21.15.89:0/1130637587 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb10006ac90 msgr2=0x7fb10006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:33.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.148+0000 7fb126a00700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb10006ac90 0x7fb10006d150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb110007fd0 tx=0x7fb110007dc0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.148+0000 7fb126a00700 1 -- 172.21.15.89:0/1130637587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb120161590 msgr2=0x7fb12006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.148+0000 7fb126a00700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb120161590 0x7fb12006c540 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb11800b370 tx=0x7fb118004ad0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.148+0000 7fb126a00700 1 -- 172.21.15.89:0/1130637587 shutdown_connections 2024-04-02T07:36:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.148+0000 7fb126a00700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb10006ac90 0x7fb10006d150 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.148+0000 7fb126a00700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1200d18b0 0x7fb120158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.148+0000 7fb126a00700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb120161590 0x7fb12006c540 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.148+0000 7fb126a00700 1 --2- 172.21.15.89:0/1130637587 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb120160f10 0x7fb1200b7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.148+0000 7fb126a00700 1 -- 172.21.15.89:0/1130637587 >> 172.21.15.89:0/1130637587 conn(0x7fb12008aff0 msgr2=0x7fb1200cab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.149+0000 7fb126a00700 1 -- 172.21.15.89:0/1130637587 shutdown_connections 2024-04-02T07:36:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.149+0000 7fb126a00700 1 -- 172.21.15.89:0/1130637587 wait complete. 2024-04-02T07:36:33.168 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:35:51.296561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:35:52.232882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:36:33.169 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:36:33.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.364+0000 7f1adfa00700 1 Processor -- start 2024-04-02T07:36:33.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.365+0000 7f1adfa00700 1 -- start start 2024-04-02T07:36:33.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.366+0000 7f1adfa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ad815af30 0x7f1ad815b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:33.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.366+0000 7f1adfa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1ad80b4a50 0x7f1ad815e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:33.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.366+0000 7f1adfa00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad806eae0 con 0x7f1ad806d370 2024-04-02T07:36:33.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.366+0000 7f1adfa00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad806f660 con 0x7f1ad80b4a50 2024-04-02T07:36:33.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.366+0000 7f1adfa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad806faf0 con 0x7f1ad815af30 2024-04-02T07:36:33.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.366+0000 7f1adda00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1ad806d370 0x7f1ad8161260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46892/0 (socket says 172.21.15.89:46892) 2024-04-02T07:36:33.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.366+0000 7f1adda00700 1 -- 172.21.15.89:0/297266631 learned_addr learned my addr 172.21.15.89:0/297266631 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:33.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.366+0000 7f1ad7e00700 1 --2- 172.21.15.89:0/297266631 >> v2:172.21.15.89:3300/0 conn(0x7f1ad80b4a50 0x7f1ad815e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:33.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.366+0000 7f1add000700 1 --2- 172.21.15.89:0/297266631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ad815af30 0x7f1ad815b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:33.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.367+0000 7f1ad7e00700 1 -- 172.21.15.89:0/297266631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ad815af30 msgr2=0x7f1ad815b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:33.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.367+0000 7f1ad7e00700 1 --2- 172.21.15.89:0/297266631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ad815af30 0x7f1ad815b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.367+0000 7f1ad7e00700 1 -- 172.21.15.89:0/297266631 >> v1:172.21.15.89:6789/0 conn(0x7f1ad806d370 legacy=0x7f1ad8161260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:33.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.367+0000 7f1ad7e00700 1 -- 172.21.15.89:0/297266631 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad806e150 con 0x7f1ad80b4a50 2024-04-02T07:36:33.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.367+0000 7f1add000700 1 --2- 172.21.15.89:0/297266631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ad815af30 0x7f1ad815b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:33.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.367+0000 7f1ad7400700 1 -- 172.21.15.89:0/297266631 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191358545 0 0) 0x7f1ad806eae0 con 0x7f1ad806d370 2024-04-02T07:36:33.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.367+0000 7f1ad7e00700 1 --2- 172.21.15.89:0/297266631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad80b4a50 0x7f1ad815e6f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009840 tx=0x7f1ac4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5668f766b0f29802 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:33.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.367+0000 7f1ad7400700 1 -- 172.21.15.89:0/297266631 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4011830 con 0x7f1ad80b4a50 2024-04-02T07:36:33.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.367+0000 7f1ad7400700 1 -- 172.21.15.89:0/297266631 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac4011990 con 0x7f1ad80b4a50 2024-04-02T07:36:33.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.367+0000 7f1ad7400700 1 -- 172.21.15.89:0/297266631 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4011d20 con 0x7f1ad80b4a50 2024-04-02T07:36:33.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.368+0000 7f1adfa00700 1 -- 172.21.15.89:0/297266631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad80b4a50 msgr2=0x7f1ad815e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:33.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.368+0000 7f1adfa00700 1 --2- 172.21.15.89:0/297266631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad80b4a50 0x7f1ad815e6f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009840 tx=0x7f1ac4009b50 comp rx=0 tx=0).stop 2024-04-02T07:36:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.368+0000 7f1adfa00700 1 -- 172.21.15.89:0/297266631 shutdown_connections 2024-04-02T07:36:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.368+0000 7f1adfa00700 1 --2- 172.21.15.89:0/297266631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ad815af30 0x7f1ad815b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.368+0000 7f1adfa00700 1 --2- 172.21.15.89:0/297266631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad80b4a50 0x7f1ad815e6f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.368+0000 7f1adfa00700 1 -- 172.21.15.89:0/297266631 >> 172.21.15.89:0/297266631 conn(0x7f1ad8071fb0 msgr2=0x7f1ad806a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.368+0000 7f1adfa00700 1 -- 172.21.15.89:0/297266631 shutdown_connections 2024-04-02T07:36:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.368+0000 7f1adfa00700 1 -- 172.21.15.89:0/297266631 wait complete. 2024-04-02T07:36:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.369+0000 7f1adfa00700 1 Processor -- start 2024-04-02T07:36:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.370+0000 7f1adfa00700 1 -- start start 2024-04-02T07:36:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.370+0000 7f1adfa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad806d370 0x7f1ad8155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.370+0000 7f1adfa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ad80b4a50 0x7f1ad80b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.370+0000 7f1adfa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ad815af30 0x7f1ad80b3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.370+0000 7f1adfa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80ce450 con 0x7f1ad806d370 2024-04-02T07:36:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.370+0000 7f1adfa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80cf490 con 0x7f1ad815af30 2024-04-02T07:36:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.370+0000 7f1adfa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80ce6d0 con 0x7f1ad80b4a50 2024-04-02T07:36:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.371+0000 7f1add000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad806d370 0x7f1ad8155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.371+0000 7f1add000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad806d370 0x7f1ad8155670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40760/0 (socket says 172.21.15.89:40760) 2024-04-02T07:36:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.371+0000 7f1add000700 1 -- 172.21.15.89:0/4259268414 learned_addr learned my addr 172.21.15.89:0/4259268414 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.371+0000 7f1ad7e00700 1 --2- 172.21.15.89:0/4259268414 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ad80b4a50 0x7f1ad80b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.371+0000 7f1adda00700 1 --2- 172.21.15.89:0/4259268414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ad815af30 0x7f1ad80b3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.371+0000 7f1add000700 1 -- 172.21.15.89:0/4259268414 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ad80b4a50 msgr2=0x7f1ad80b2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.371+0000 7f1add000700 1 --2- 172.21.15.89:0/4259268414 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ad80b4a50 0x7f1ad80b2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.371+0000 7f1add000700 1 -- 172.21.15.89:0/4259268414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ad815af30 msgr2=0x7f1ad80b3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.371+0000 7f1add000700 1 --2- 172.21.15.89:0/4259268414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ad815af30 0x7f1ad80b3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.371+0000 7f1add000700 1 -- 172.21.15.89:0/4259268414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40094a0 con 0x7f1ad806d370 2024-04-02T07:36:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.371+0000 7f1add000700 1 --2- 172.21.15.89:0/4259268414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad806d370 0x7f1ad8155670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00e810 tx=0x7f1acc00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.372+0000 7f1ad4c00700 1 -- 172.21.15.89:0/4259268414 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1acc013610 con 0x7f1ad806d370 2024-04-02T07:36:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.372+0000 7f1ad4c00700 1 -- 172.21.15.89:0/4259268414 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1acc013770 con 0x7f1ad806d370 2024-04-02T07:36:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.372+0000 7f1adfa00700 1 -- 172.21.15.89:0/4259268414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad80ce9b0 con 0x7f1ad806d370 2024-04-02T07:36:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.372+0000 7f1ad4c00700 1 -- 172.21.15.89:0/4259268414 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1acc013b20 con 0x7f1ad806d370 2024-04-02T07:36:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.372+0000 7f1adfa00700 1 -- 172.21.15.89:0/4259268414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad80cef70 con 0x7f1ad806d370 2024-04-02T07:36:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.373+0000 7f1adfa00700 1 -- 172.21.15.89:0/4259268414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad8067720 con 0x7f1ad806d370 2024-04-02T07:36:33.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.374+0000 7f1ad4c00700 1 -- 172.21.15.89:0/4259268414 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1acc013c80 con 0x7f1ad806d370 2024-04-02T07:36:33.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.375+0000 7f1ad4c00700 1 --2- 172.21.15.89:0/4259268414 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1ab806ac90 0x7f1ab806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:33.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.375+0000 7f1ad4c00700 1 -- 172.21.15.89:0/4259268414 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1acc0890d0 con 0x7f1ad806d370 2024-04-02T07:36:33.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.375+0000 7f1ad7e00700 1 --2- 172.21.15.89:0/4259268414 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1ab806ac90 0x7f1ab806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:33.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.376+0000 7f1ad7e00700 1 --2- 172.21.15.89:0/4259268414 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1ab806ac90 0x7f1ab806d150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40090b0 tx=0x7f1ac4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:33.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.379+0000 7f1ad4c00700 1 -- 172.21.15.89:0/4259268414 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1acc054210 con 0x7f1ad806d370 2024-04-02T07:36:33.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.579+0000 7f1adfa00700 1 -- 172.21.15.89:0/4259268414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1ad806e550 con 0x7f1ad806d370 2024-04-02T07:36:33.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.579+0000 7f1ad4c00700 1 -- 172.21.15.89:0/4259268414 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1acc057830 con 0x7f1ad806d370 2024-04-02T07:36:33.581 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:36:33.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.582+0000 7f1adfa00700 1 -- 172.21.15.89:0/4259268414 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1ab806ac90 msgr2=0x7f1ab806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.582+0000 7f1adfa00700 1 --2- 172.21.15.89:0/4259268414 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1ab806ac90 0x7f1ab806d150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40090b0 tx=0x7f1ac4008030 comp rx=0 tx=0).stop 2024-04-02T07:36:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.582+0000 7f1adfa00700 1 -- 172.21.15.89:0/4259268414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad806d370 msgr2=0x7f1ad8155670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.582+0000 7f1adfa00700 1 --2- 172.21.15.89:0/4259268414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad806d370 0x7f1ad8155670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00e810 tx=0x7f1acc00ebd0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.582+0000 7f1adfa00700 1 -- 172.21.15.89:0/4259268414 shutdown_connections 2024-04-02T07:36:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.582+0000 7f1adfa00700 1 --2- 172.21.15.89:0/4259268414 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1ab806ac90 0x7f1ab806d150 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.582+0000 7f1adfa00700 1 --2- 172.21.15.89:0/4259268414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ad815af30 0x7f1ad80b3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.582+0000 7f1adfa00700 1 --2- 172.21.15.89:0/4259268414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ad806d370 0x7f1ad8155670 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.582+0000 7f1adfa00700 1 --2- 172.21.15.89:0/4259268414 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ad80b4a50 0x7f1ad80b2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.582+0000 7f1adfa00700 1 -- 172.21.15.89:0/4259268414 >> 172.21.15.89:0/4259268414 conn(0x7f1ad8071fb0 msgr2=0x7f1ad8158800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.582+0000 7f1adfa00700 1 -- 172.21.15.89:0/4259268414 shutdown_connections 2024-04-02T07:36:33.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.582+0000 7f1adfa00700 1 -- 172.21.15.89:0/4259268414 wait complete. 2024-04-02T07:36:33.602 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:36:33.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.794+0000 7f68cee00700 1 Processor -- start 2024-04-02T07:36:33.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.795+0000 7f68cee00700 1 -- start start 2024-04-02T07:36:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68cee00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f68c81611b0 0x7f68c815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68cee00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f68c815f4d0 0x7f68c80ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68cee00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68c80d21f0 0x7f68c80cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68cee00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68c80cf180 con 0x7f68c81611b0 2024-04-02T07:36:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68cee00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68c80ce270 con 0x7f68c815f4d0 2024-04-02T07:36:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68cee00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c815b600 con 0x7f68c80d21f0 2024-04-02T07:36:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68c7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f68c81611b0 0x7f68c815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68c7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f68c81611b0 0x7f68c815ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40766/0 (socket says 172.21.15.89:40766) 2024-04-02T07:36:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68c7e00700 1 -- 172.21.15.89:0/2258253914 learned_addr learned my addr 172.21.15.89:0/2258253914 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68cce00700 1 --2- 172.21.15.89:0/2258253914 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68c80d21f0 0x7f68c80cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68c7400700 1 --2- 172.21.15.89:0/2258253914 >> v2:172.21.15.113:3300/0 conn(0x7f68c815f4d0 0x7f68c80ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68c7e00700 1 -- 172.21.15.89:0/2258253914 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68c80d21f0 msgr2=0x7f68c80cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:33.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68c7e00700 1 --2- 172.21.15.89:0/2258253914 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68c80d21f0 0x7f68c80cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68c7e00700 1 -- 172.21.15.89:0/2258253914 >> v2:172.21.15.113:3300/0 conn(0x7f68c815f4d0 msgr2=0x7f68c80ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:33.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68c7e00700 1 --2- 172.21.15.89:0/2258253914 >> v2:172.21.15.113:3300/0 conn(0x7f68c815f4d0 0x7f68c80ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.796+0000 7f68c7e00700 1 -- 172.21.15.89:0/2258253914 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c80d2760 con 0x7f68c81611b0 2024-04-02T07:36:33.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.797+0000 7f68c7e00700 1 --2- 172.21.15.89:0/2258253914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68c81611b0 0x7f68c815ef90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f68c808c030 tx=0x7f68b8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb7721f406d7cc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:33.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.797+0000 7f68c6a00700 1 -- 172.21.15.89:0/2258253914 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b800b870 con 0x7f68c81611b0 2024-04-02T07:36:33.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.797+0000 7f68c6a00700 1 -- 172.21.15.89:0/2258253914 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b800b9d0 con 0x7f68c81611b0 2024-04-02T07:36:33.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.797+0000 7f68c6a00700 1 -- 172.21.15.89:0/2258253914 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b800bd60 con 0x7f68c81611b0 2024-04-02T07:36:33.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.797+0000 7f68cee00700 1 -- 172.21.15.89:0/2258253914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68c81611b0 msgr2=0x7f68c815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:33.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.797+0000 7f68cee00700 1 --2- 172.21.15.89:0/2258253914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68c81611b0 0x7f68c815ef90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f68c808c030 tx=0x7f68b8009bd0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.798+0000 7f68cee00700 1 -- 172.21.15.89:0/2258253914 shutdown_connections 2024-04-02T07:36:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.798+0000 7f68cee00700 1 --2- 172.21.15.89:0/2258253914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68c81611b0 0x7f68c815ef90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.798+0000 7f68cee00700 1 --2- 172.21.15.89:0/2258253914 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68c80d21f0 0x7f68c80cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.798+0000 7f68cee00700 1 --2- 172.21.15.89:0/2258253914 >> v2:172.21.15.113:3300/0 conn(0x7f68c815f4d0 0x7f68c80ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.798+0000 7f68cee00700 1 -- 172.21.15.89:0/2258253914 >> 172.21.15.89:0/2258253914 conn(0x7f68c808aff0 msgr2=0x7f68c80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.798+0000 7f68cee00700 1 -- 172.21.15.89:0/2258253914 shutdown_connections 2024-04-02T07:36:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.798+0000 7f68cee00700 1 -- 172.21.15.89:0/2258253914 wait complete. 2024-04-02T07:36:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.799+0000 7f68cee00700 1 Processor -- start 2024-04-02T07:36:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.800+0000 7f68cee00700 1 -- start start 2024-04-02T07:36:33.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.800+0000 7f68cee00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68c80d21f0 0x7f68c81f7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:33.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.800+0000 7f68cee00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68c815f4d0 0x7f68c80a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:33.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.800+0000 7f68cee00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68c81611b0 0x7f68c809ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:33.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.800+0000 7f68cee00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c81572b0 con 0x7f68c80d21f0 2024-04-02T07:36:33.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.800+0000 7f68cee00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c809d220 con 0x7f68c815f4d0 2024-04-02T07:36:33.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.800+0000 7f68cee00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c809d500 con 0x7f68c81611b0 2024-04-02T07:36:33.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.801+0000 7f68c7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68c80d21f0 0x7f68c81f7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:33.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.801+0000 7f68c7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68c80d21f0 0x7f68c81f7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40778/0 (socket says 172.21.15.89:40778) 2024-04-02T07:36:33.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.801+0000 7f68c7e00700 1 -- 172.21.15.89:0/4240220021 learned_addr learned my addr 172.21.15.89:0/4240220021 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.801+0000 7f68c7400700 1 --2- 172.21.15.89:0/4240220021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68c815f4d0 0x7f68c80a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.801+0000 7f68cce00700 1 --2- 172.21.15.89:0/4240220021 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68c81611b0 0x7f68c809ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.801+0000 7f68c7e00700 1 -- 172.21.15.89:0/4240220021 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68c81611b0 msgr2=0x7f68c809ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.801+0000 7f68c7e00700 1 --2- 172.21.15.89:0/4240220021 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68c81611b0 0x7f68c809ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.801+0000 7f68c7e00700 1 -- 172.21.15.89:0/4240220021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68c815f4d0 msgr2=0x7f68c80a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.801+0000 7f68c7e00700 1 --2- 172.21.15.89:0/4240220021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68c815f4d0 0x7f68c80a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.801+0000 7f68c7e00700 1 -- 172.21.15.89:0/4240220021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b8009550 con 0x7f68c80d21f0 2024-04-02T07:36:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.801+0000 7f68c7e00700 1 --2- 172.21.15.89:0/4240220021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68c80d21f0 0x7f68c81f7110 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f68b8000c00 tx=0x7f68b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.802+0000 7f68bfe00700 1 -- 172.21.15.89:0/4240220021 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b8004520 con 0x7f68c80d21f0 2024-04-02T07:36:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.802+0000 7f68bfe00700 1 -- 172.21.15.89:0/4240220021 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b801e070 con 0x7f68c80d21f0 2024-04-02T07:36:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.802+0000 7f68cee00700 1 -- 172.21.15.89:0/4240220021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c809d7e0 con 0x7f68c80d21f0 2024-04-02T07:36:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.802+0000 7f68bfe00700 1 -- 172.21.15.89:0/4240220021 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b80038c0 con 0x7f68c80d21f0 2024-04-02T07:36:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.802+0000 7f68cee00700 1 -- 172.21.15.89:0/4240220021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c809dda0 con 0x7f68c80d21f0 2024-04-02T07:36:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.803+0000 7f68cee00700 1 -- 172.21.15.89:0/4240220021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c80a1270 con 0x7f68c80d21f0 2024-04-02T07:36:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.804+0000 7f68bfe00700 1 -- 172.21.15.89:0/4240220021 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f68b8003a20 con 0x7f68c80d21f0 2024-04-02T07:36:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.805+0000 7f68bfe00700 1 --2- 172.21.15.89:0/4240220021 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f68a806ac90 0x7f68a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.805+0000 7f68bfe00700 1 -- 172.21.15.89:0/4240220021 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68b8020070 con 0x7f68c80d21f0 2024-04-02T07:36:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.805+0000 7f68c7400700 1 --2- 172.21.15.89:0/4240220021 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f68a806ac90 0x7f68a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.805+0000 7f68c7400700 1 --2- 172.21.15.89:0/4240220021 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f68a806ac90 0x7f68a806d150 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f68c80cc2a0 tx=0x7f68b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:33.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:33.809+0000 7f68bfe00700 1 -- 172.21.15.89:0/4240220021 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f68b80559b0 con 0x7f68c80d21f0 2024-04-02T07:36:34.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.010+0000 7f68cee00700 1 -- 172.21.15.89:0/4240220021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f68c809df90 con 0x7f68c80d21f0 2024-04-02T07:36:34.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.011+0000 7f68bfe00700 1 -- 172.21.15.89:0/4240220021 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f68b8015020 con 0x7f68c80d21f0 2024-04-02T07:36:34.013 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:36:34.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.013+0000 7f68cee00700 1 -- 172.21.15.89:0/4240220021 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f68a806ac90 msgr2=0x7f68a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:34.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.013+0000 7f68cee00700 1 --2- 172.21.15.89:0/4240220021 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f68a806ac90 0x7f68a806d150 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f68c80cc2a0 tx=0x7f68b4008040 comp rx=0 tx=0).stop 2024-04-02T07:36:34.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.013+0000 7f68cee00700 1 -- 172.21.15.89:0/4240220021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68c80d21f0 msgr2=0x7f68c81f7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:34.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.013+0000 7f68cee00700 1 --2- 172.21.15.89:0/4240220021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68c80d21f0 0x7f68c81f7110 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f68b8000c00 tx=0x7f68b8004750 comp rx=0 tx=0).stop 2024-04-02T07:36:34.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.013+0000 7f68cee00700 1 -- 172.21.15.89:0/4240220021 shutdown_connections 2024-04-02T07:36:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.013+0000 7f68cee00700 1 --2- 172.21.15.89:0/4240220021 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f68a806ac90 0x7f68a806d150 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.013+0000 7f68cee00700 1 --2- 172.21.15.89:0/4240220021 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68c81611b0 0x7f68c809ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.013+0000 7f68cee00700 1 --2- 172.21.15.89:0/4240220021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68c80d21f0 0x7f68c81f7110 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.013+0000 7f68cee00700 1 --2- 172.21.15.89:0/4240220021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68c815f4d0 0x7f68c80a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.013+0000 7f68cee00700 1 -- 172.21.15.89:0/4240220021 >> 172.21.15.89:0/4240220021 conn(0x7f68c808aff0 msgr2=0x7f68c8161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.013+0000 7f68cee00700 1 -- 172.21.15.89:0/4240220021 shutdown_connections 2024-04-02T07:36:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.013+0000 7f68cee00700 1 -- 172.21.15.89:0/4240220021 wait complete. 2024-04-02T07:36:34.035 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:36:34.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.227+0000 7f7c61800700 1 Processor -- start 2024-04-02T07:36:34.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.228+0000 7f7c61800700 1 -- start start 2024-04-02T07:36:34.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.229+0000 7f7c61800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7c5c0d16a0 0x7f7c5c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.229+0000 7f7c61800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c5c161420 con 0x7f7c5c0d2370 2024-04-02T07:36:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.229+0000 7f7c61800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c5c161700 con 0x7f7c5c15d9d0 2024-04-02T07:36:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.229+0000 7f7c61800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c5c0d4340 con 0x7f7c5c0d16a0 2024-04-02T07:36:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.229+0000 7f7c5b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7c5c0d2370 0x7f7c5c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46896/0 (socket says 172.21.15.89:46896) 2024-04-02T07:36:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.229+0000 7f7c5b400700 1 -- 172.21.15.89:0/3780039871 learned_addr learned my addr 172.21.15.89:0/3780039871 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.229+0000 7f7c53e00700 1 --2- 172.21.15.89:0/3780039871 >> v2:172.21.15.113:3300/0 conn(0x7f7c5c0d16a0 0x7f7c5c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.229+0000 7f7c5a000700 1 -- 172.21.15.89:0/3780039871 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485882511 0 0) 0x7f7c5c161420 con 0x7f7c5c0d2370 2024-04-02T07:36:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.229+0000 7f7c5a000700 1 -- 172.21.15.89:0/3780039871 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c400035b0 con 0x7f7c5c0d2370 2024-04-02T07:36:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.230+0000 7f7c53e00700 1 -- 172.21.15.89:0/3780039871 >> v1:172.21.15.113:6789/0 conn(0x7f7c5c15d9d0 legacy=0x7f7c5c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.230+0000 7f7c53e00700 1 -- 172.21.15.89:0/3780039871 >> v1:172.21.15.89:6789/0 conn(0x7f7c5c0d2370 legacy=0x7f7c5c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.230+0000 7f7c53e00700 1 -- 172.21.15.89:0/3780039871 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c1619e0 con 0x7f7c5c0d16a0 2024-04-02T07:36:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.230+0000 7f7c53e00700 1 --2- 172.21.15.89:0/3780039871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c5c0d16a0 0x7f7c5c0d1e30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009820 tx=0x7f7c44009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95276fcccb9e9ccc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.230+0000 7f7c5a000700 1 -- 172.21.15.89:0/3780039871 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c44005e30 con 0x7f7c5c0d16a0 2024-04-02T07:36:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.230+0000 7f7c5a000700 1 -- 172.21.15.89:0/3780039871 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c44006030 con 0x7f7c5c0d16a0 2024-04-02T07:36:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.230+0000 7f7c5a000700 1 -- 172.21.15.89:0/3780039871 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c440063c0 con 0x7f7c5c0d16a0 2024-04-02T07:36:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.231+0000 7f7c61800700 1 -- 172.21.15.89:0/3780039871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c5c0d16a0 msgr2=0x7f7c5c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.231+0000 7f7c61800700 1 --2- 172.21.15.89:0/3780039871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c5c0d16a0 0x7f7c5c0d1e30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009820 tx=0x7f7c44009b30 comp rx=0 tx=0).stop 2024-04-02T07:36:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.231+0000 7f7c61800700 1 -- 172.21.15.89:0/3780039871 shutdown_connections 2024-04-02T07:36:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.231+0000 7f7c61800700 1 --2- 172.21.15.89:0/3780039871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c5c0d16a0 0x7f7c5c0d1e30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.231+0000 7f7c61800700 1 -- 172.21.15.89:0/3780039871 >> 172.21.15.89:0/3780039871 conn(0x7f7c5c08aff0 msgr2=0x7f7c5c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:34.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.231+0000 7f7c61800700 1 -- 172.21.15.89:0/3780039871 shutdown_connections 2024-04-02T07:36:34.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.231+0000 7f7c61800700 1 -- 172.21.15.89:0/3780039871 wait complete. 2024-04-02T07:36:34.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.232+0000 7f7c61800700 1 Processor -- start 2024-04-02T07:36:34.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.233+0000 7f7c61800700 1 -- start start 2024-04-02T07:36:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.233+0000 7f7c61800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c5c0d16a0 0x7f7c5c0b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.233+0000 7f7c61800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c5c0d2370 0x7f7c5c0b8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c61800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c5c15d9d0 0x7f7c5c1607f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c61800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0b76d0 con 0x7f7c5c0d2370 2024-04-02T07:36:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c61800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0b7110 con 0x7f7c5c0d16a0 2024-04-02T07:36:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c61800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0b73f0 con 0x7f7c5c15d9d0 2024-04-02T07:36:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c53e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c5c0d2370 0x7f7c5c0b8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c53e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c5c0d2370 0x7f7c5c0b8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40784/0 (socket says 172.21.15.89:40784) 2024-04-02T07:36:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c53e00700 1 -- 172.21.15.89:0/650573750 learned_addr learned my addr 172.21.15.89:0/650573750 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c5b400700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c5c15d9d0 0x7f7c5c1607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c5aa00700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c5c0d16a0 0x7f7c5c0b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c53e00700 1 -- 172.21.15.89:0/650573750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c5c15d9d0 msgr2=0x7f7c5c1607f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c53e00700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c5c15d9d0 0x7f7c5c1607f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c53e00700 1 -- 172.21.15.89:0/650573750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c5c0d16a0 msgr2=0x7f7c5c0b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c53e00700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c5c0d16a0 0x7f7c5c0b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c53e00700 1 -- 172.21.15.89:0/650573750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c44009430 con 0x7f7c5c0d2370 2024-04-02T07:36:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c5aa00700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c5c0d16a0 0x7f7c5c0b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.234+0000 7f7c53e00700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c5c0d2370 0x7f7c5c0b8440 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7c44000c00 tx=0x7f7c44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.235+0000 7f7c52a00700 1 -- 172.21.15.89:0/650573750 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c44004240 con 0x7f7c5c0d2370 2024-04-02T07:36:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.235+0000 7f7c52a00700 1 -- 172.21.15.89:0/650573750 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c440043a0 con 0x7f7c5c0d2370 2024-04-02T07:36:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.235+0000 7f7c52a00700 1 -- 172.21.15.89:0/650573750 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c44006940 con 0x7f7c5c0d2370 2024-04-02T07:36:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.235+0000 7f7c61800700 1 -- 172.21.15.89:0/650573750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5c15df50 con 0x7f7c5c0d2370 2024-04-02T07:36:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.235+0000 7f7c61800700 1 -- 172.21.15.89:0/650573750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5c0b8980 con 0x7f7c5c0d2370 2024-04-02T07:36:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.236+0000 7f7c61800700 1 -- 172.21.15.89:0/650573750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5c0d1ca0 con 0x7f7c5c0d2370 2024-04-02T07:36:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.237+0000 7f7c52a00700 1 -- 172.21.15.89:0/650573750 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7c44022070 con 0x7f7c5c0d2370 2024-04-02T07:36:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.238+0000 7f7c52a00700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c3806ac90 0x7f7c3806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:34.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.238+0000 7f7c52a00700 1 -- 172.21.15.89:0/650573750 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c4408aa40 con 0x7f7c5c0d2370 2024-04-02T07:36:34.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.238+0000 7f7c5aa00700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c3806ac90 0x7f7c3806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:34.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.238+0000 7f7c5aa00700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c3806ac90 0x7f7c3806d150 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c000f60 tx=0x7f7c4c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:34.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.242+0000 7f7c52a00700 1 -- 172.21.15.89:0/650573750 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7c4408ce60 con 0x7f7c5c0d2370 2024-04-02T07:36:34.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.441+0000 7f7c61800700 1 -- 172.21.15.89:0/650573750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7c5c15e470 con 0x7f7c5c0d2370 2024-04-02T07:36:34.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.442+0000 7f7c52a00700 1 -- 172.21.15.89:0/650573750 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7c5c15e470 con 0x7f7c5c0d2370 2024-04-02T07:36:34.444 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:36:34.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.444+0000 7f7c61800700 1 -- 172.21.15.89:0/650573750 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c3806ac90 msgr2=0x7f7c3806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:34.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.444+0000 7f7c61800700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c3806ac90 0x7f7c3806d150 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c000f60 tx=0x7f7c4c009040 comp rx=0 tx=0).stop 2024-04-02T07:36:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.444+0000 7f7c61800700 1 -- 172.21.15.89:0/650573750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c5c0d2370 msgr2=0x7f7c5c0b8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.444+0000 7f7c61800700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c5c0d2370 0x7f7c5c0b8440 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7c44000c00 tx=0x7f7c44004750 comp rx=0 tx=0).stop 2024-04-02T07:36:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.444+0000 7f7c61800700 1 -- 172.21.15.89:0/650573750 shutdown_connections 2024-04-02T07:36:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.444+0000 7f7c61800700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c3806ac90 0x7f7c3806d150 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.444+0000 7f7c61800700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c5c0d16a0 0x7f7c5c0b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.444+0000 7f7c61800700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c5c15d9d0 0x7f7c5c1607f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.444+0000 7f7c61800700 1 --2- 172.21.15.89:0/650573750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c5c0d2370 0x7f7c5c0b8440 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.444+0000 7f7c61800700 1 -- 172.21.15.89:0/650573750 >> 172.21.15.89:0/650573750 conn(0x7f7c5c08aff0 msgr2=0x7f7c5c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.445+0000 7f7c61800700 1 -- 172.21.15.89:0/650573750 shutdown_connections 2024-04-02T07:36:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.445+0000 7f7c61800700 1 -- 172.21.15.89:0/650573750 wait complete. 2024-04-02T07:36:34.465 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T07:36:34.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.654+0000 7fd308000700 1 Processor -- start 2024-04-02T07:36:34.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.655+0000 7fd308000700 1 -- start start 2024-04-02T07:36:34.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.656+0000 7fd308000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd3000d21f0 0x7fd3000ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:34.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.656+0000 7fd308000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd3000cec40 0x7fd300161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:34.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.656+0000 7fd308000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3000ce270 con 0x7fd3001611b0 2024-04-02T07:36:34.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.656+0000 7fd308000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd30015b600 con 0x7fd3000d21f0 2024-04-02T07:36:34.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.656+0000 7fd308000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3000d2760 con 0x7fd3000cec40 2024-04-02T07:36:34.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.656+0000 7fd304c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd3000d21f0 0x7fd3000ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:34.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.656+0000 7fd304c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd3000d21f0 0x7fd3000ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40786/0 (socket says 172.21.15.89:40786) 2024-04-02T07:36:34.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.656+0000 7fd304c00700 1 -- 172.21.15.89:0/4016476041 learned_addr learned my addr 172.21.15.89:0/4016476041 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:34.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.656+0000 7fd304c00700 1 -- 172.21.15.89:0/4016476041 >> v2:172.21.15.113:3300/0 conn(0x7fd3000cec40 msgr2=0x7fd300161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:34.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.656+0000 7fd306000700 1 --2- 172.21.15.89:0/4016476041 >> v2:172.21.15.113:3300/0 conn(0x7fd3000cec40 0x7fd300161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:34.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.656+0000 7fd304c00700 1 --2- 172.21.15.89:0/4016476041 >> v2:172.21.15.113:3300/0 conn(0x7fd3000cec40 0x7fd300161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.656+0000 7fd304c00700 1 -- 172.21.15.89:0/4016476041 >> v1:172.21.15.113:6789/0 conn(0x7fd3001611b0 legacy=0x7fd30015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:34.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.656+0000 7fd304c00700 1 -- 172.21.15.89:0/4016476041 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3000d2a40 con 0x7fd3000d21f0 2024-04-02T07:36:34.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.656+0000 7fd306000700 1 --2- 172.21.15.89:0/4016476041 >> v2:172.21.15.113:3300/0 conn(0x7fd3000cec40 0x7fd300161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:34.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.657+0000 7fd304c00700 1 --2- 172.21.15.89:0/4016476041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3000d21f0 0x7fd3000ce700 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd3000b5cb0 tx=0x7fd2ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b809d205b95291b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:34.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.657+0000 7fd2ffe00700 1 -- 172.21.15.89:0/4016476041 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec004230 con 0x7fd3000d21f0 2024-04-02T07:36:34.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.657+0000 7fd2ffe00700 1 -- 172.21.15.89:0/4016476041 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ec004390 con 0x7fd3000d21f0 2024-04-02T07:36:34.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.657+0000 7fd2ffe00700 1 -- 172.21.15.89:0/4016476041 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec0036a0 con 0x7fd3000d21f0 2024-04-02T07:36:34.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.657+0000 7fd308000700 1 -- 172.21.15.89:0/4016476041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3000d21f0 msgr2=0x7fd3000ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:34.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.657+0000 7fd308000700 1 --2- 172.21.15.89:0/4016476041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3000d21f0 0x7fd3000ce700 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd3000b5cb0 tx=0x7fd2ec009bd0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.658+0000 7fd308000700 1 -- 172.21.15.89:0/4016476041 shutdown_connections 2024-04-02T07:36:34.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.658+0000 7fd308000700 1 --2- 172.21.15.89:0/4016476041 >> v2:172.21.15.113:3300/0 conn(0x7fd3000cec40 0x7fd300161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.658+0000 7fd308000700 1 --2- 172.21.15.89:0/4016476041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3000d21f0 0x7fd3000ce700 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.658+0000 7fd308000700 1 -- 172.21.15.89:0/4016476041 >> 172.21.15.89:0/4016476041 conn(0x7fd30008aff0 msgr2=0x7fd3000d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:34.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.658+0000 7fd308000700 1 -- 172.21.15.89:0/4016476041 shutdown_connections 2024-04-02T07:36:34.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.658+0000 7fd308000700 1 -- 172.21.15.89:0/4016476041 wait complete. 2024-04-02T07:36:34.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.659+0000 7fd308000700 1 Processor -- start 2024-04-02T07:36:34.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.659+0000 7fd308000700 1 -- start start 2024-04-02T07:36:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.660+0000 7fd308000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3000cec40 0x7fd3000cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.660+0000 7fd308000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3000d21f0 0x7fd30015abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.660+0000 7fd308000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3001611b0 0x7fd3000c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.660+0000 7fd308000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000ccab0 con 0x7fd3000cec40 2024-04-02T07:36:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.660+0000 7fd308000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000cced0 con 0x7fd3000d21f0 2024-04-02T07:36:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.660+0000 7fd308000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000cc1a0 con 0x7fd3001611b0 2024-04-02T07:36:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.661+0000 7fd305600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3000cec40 0x7fd3000cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.661+0000 7fd305600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3000cec40 0x7fd3000cd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40790/0 (socket says 172.21.15.89:40790) 2024-04-02T07:36:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.661+0000 7fd305600700 1 -- 172.21.15.89:0/2203350868 learned_addr learned my addr 172.21.15.89:0/2203350868 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.661+0000 7fd304c00700 1 --2- 172.21.15.89:0/2203350868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3000d21f0 0x7fd30015abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.661+0000 7fd306000700 1 --2- 172.21.15.89:0/2203350868 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3001611b0 0x7fd3000c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.661+0000 7fd305600700 1 -- 172.21.15.89:0/2203350868 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3001611b0 msgr2=0x7fd3000c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:34.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.661+0000 7fd305600700 1 --2- 172.21.15.89:0/2203350868 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3001611b0 0x7fd3000c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.661+0000 7fd305600700 1 -- 172.21.15.89:0/2203350868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3000d21f0 msgr2=0x7fd30015abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:34.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.661+0000 7fd305600700 1 --2- 172.21.15.89:0/2203350868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3000d21f0 0x7fd30015abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.661+0000 7fd305600700 1 -- 172.21.15.89:0/2203350868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec009550 con 0x7fd3000cec40 2024-04-02T07:36:34.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.661+0000 7fd305600700 1 --2- 172.21.15.89:0/2203350868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3000cec40 0x7fd3000cd5b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400b4a0 tx=0x7fd2f400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:34.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.661+0000 7fd2fd600700 1 -- 172.21.15.89:0/2203350868 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4011830 con 0x7fd3000cec40 2024-04-02T07:36:34.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.661+0000 7fd2fd600700 1 -- 172.21.15.89:0/2203350868 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f4011990 con 0x7fd3000cec40 2024-04-02T07:36:34.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.662+0000 7fd2fd600700 1 -- 172.21.15.89:0/2203350868 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4011d40 con 0x7fd3000cec40 2024-04-02T07:36:34.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.662+0000 7fd308000700 1 -- 172.21.15.89:0/2203350868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30015dfd0 con 0x7fd3000cec40 2024-04-02T07:36:34.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.662+0000 7fd308000700 1 -- 172.21.15.89:0/2203350868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3000cc3f0 con 0x7fd3000cec40 2024-04-02T07:36:34.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.663+0000 7fd308000700 1 -- 172.21.15.89:0/2203350868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3000c9da0 con 0x7fd3000cec40 2024-04-02T07:36:34.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.664+0000 7fd2fd600700 1 -- 172.21.15.89:0/2203350868 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd2f401a400 con 0x7fd3000cec40 2024-04-02T07:36:34.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.664+0000 7fd2fd600700 1 --2- 172.21.15.89:0/2203350868 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2e006acc0 0x7fd2e006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:34.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.664+0000 7fd2fd600700 1 -- 172.21.15.89:0/2203350868 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2f4088f90 con 0x7fd3000cec40 2024-04-02T07:36:34.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.665+0000 7fd304c00700 1 --2- 172.21.15.89:0/2203350868 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2e006acc0 0x7fd2e006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:34.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.665+0000 7fd304c00700 1 --2- 172.21.15.89:0/2203350868 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2e006acc0 0x7fd2e006d180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009130 tx=0x7fd2ec008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:34.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.669+0000 7fd2fd600700 1 -- 172.21.15.89:0/2203350868 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd2f40bc580 con 0x7fd3000cec40 2024-04-02T07:36:34.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.886+0000 7fd308000700 1 -- 172.21.15.89:0/2203350868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd3000ccd20 con 0x7fd3000cec40 2024-04-02T07:36:34.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.886+0000 7fd2fd600700 1 -- 172.21.15.89:0/2203350868 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd2f4054210 con 0x7fd3000cec40 2024-04-02T07:36:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.889+0000 7fd308000700 1 -- 172.21.15.89:0/2203350868 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2e006acc0 msgr2=0x7fd2e006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.889+0000 7fd308000700 1 --2- 172.21.15.89:0/2203350868 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2e006acc0 0x7fd2e006d180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009130 tx=0x7fd2ec008030 comp rx=0 tx=0).stop 2024-04-02T07:36:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.889+0000 7fd308000700 1 -- 172.21.15.89:0/2203350868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3000cec40 msgr2=0x7fd3000cd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.889+0000 7fd308000700 1 --2- 172.21.15.89:0/2203350868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3000cec40 0x7fd3000cd5b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400b4a0 tx=0x7fd2f400b7b0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.889+0000 7fd308000700 1 -- 172.21.15.89:0/2203350868 shutdown_connections 2024-04-02T07:36:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.889+0000 7fd308000700 1 --2- 172.21.15.89:0/2203350868 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2e006acc0 0x7fd2e006d180 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.889+0000 7fd308000700 1 --2- 172.21.15.89:0/2203350868 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3001611b0 0x7fd3000c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.889+0000 7fd308000700 1 --2- 172.21.15.89:0/2203350868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3000cec40 0x7fd3000cd5b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.889+0000 7fd308000700 1 --2- 172.21.15.89:0/2203350868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3000d21f0 0x7fd30015abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.889+0000 7fd308000700 1 -- 172.21.15.89:0/2203350868 >> 172.21.15.89:0/2203350868 conn(0x7fd30008aff0 msgr2=0x7fd30006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.889+0000 7fd308000700 1 -- 172.21.15.89:0/2203350868 shutdown_connections 2024-04-02T07:36:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:34.889+0000 7fd308000700 1 -- 172.21.15.89:0/2203350868 wait complete. 2024-04-02T07:36:34.893 INFO:teuthology.orchestra.run.smithi089.stderr:listed 0 entries 2024-04-02T07:36:34.909 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T07:36:34.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:34.910 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-04-02T07:36:34.981 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:36:34.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:34.981 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T07:36:35.033 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:36:35.033 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:35.083 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:36:35.083 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:35.129 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:36:35.249 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:35.297 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:36:35.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.486+0000 7f7675a00700 1 Processor -- start 2024-04-02T07:36:35.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.486+0000 7f7675a00700 1 -- start start 2024-04-02T07:36:35.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.487+0000 7f7675a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7670160f10 0x7f76700d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:35.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.487+0000 7f7675a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f76700cf230 0x7f76700d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:35.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.487+0000 7f7675a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76700ceda0 con 0x7f7670161590 2024-04-02T07:36:35.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.487+0000 7f7675a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76700d23d0 con 0x7f7670160f10 2024-04-02T07:36:35.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.487+0000 7f7675a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76701612f0 con 0x7f76700cf230 2024-04-02T07:36:35.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.487+0000 7f766ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7670160f10 0x7f76700d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:35.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.487+0000 7f766f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7670161590 0x7f76700b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46912/0 (socket says 172.21.15.89:46912) 2024-04-02T07:36:35.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.487+0000 7f766f400700 1 -- 172.21.15.89:0/592120950 learned_addr learned my addr 172.21.15.89:0/592120950 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:35.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.487+0000 7f766e000700 1 --2- 172.21.15.89:0/592120950 >> v2:172.21.15.113:3300/0 conn(0x7f76700cf230 0x7f76700d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:35.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.488+0000 7f766ea00700 1 -- 172.21.15.89:0/592120950 >> v2:172.21.15.113:3300/0 conn(0x7f76700cf230 msgr2=0x7f76700d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:35.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.488+0000 7f766ea00700 1 --2- 172.21.15.89:0/592120950 >> v2:172.21.15.113:3300/0 conn(0x7f76700cf230 0x7f76700d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:35.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.488+0000 7f766ea00700 1 -- 172.21.15.89:0/592120950 >> v1:172.21.15.89:6789/0 conn(0x7f7670161590 legacy=0x7f76700b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:35.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.488+0000 7f766ea00700 1 -- 172.21.15.89:0/592120950 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76700ce470 con 0x7f7670160f10 2024-04-02T07:36:35.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.488+0000 7f766d600700 1 -- 172.21.15.89:0/592120950 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176269149 0 0) 0x7f76700ceda0 con 0x7f7670161590 2024-04-02T07:36:35.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.488+0000 7f766ea00700 1 --2- 172.21.15.89:0/592120950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7670160f10 0x7f76700d18b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f767008cdb0 tx=0x7f7660009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aeea28f062f35c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:35.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.488+0000 7f766d600700 1 -- 172.21.15.89:0/592120950 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7660011830 con 0x7f7670160f10 2024-04-02T07:36:35.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.488+0000 7f766d600700 1 -- 172.21.15.89:0/592120950 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7660011990 con 0x7f7670160f10 2024-04-02T07:36:35.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.488+0000 7f766d600700 1 -- 172.21.15.89:0/592120950 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7660011d20 con 0x7f7670160f10 2024-04-02T07:36:35.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.489+0000 7f7675a00700 1 -- 172.21.15.89:0/592120950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7670160f10 msgr2=0x7f76700d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:35.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.489+0000 7f7675a00700 1 --2- 172.21.15.89:0/592120950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7670160f10 0x7f76700d18b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f767008cdb0 tx=0x7f7660009b20 comp rx=0 tx=0).stop 2024-04-02T07:36:35.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.489+0000 7f7675a00700 1 -- 172.21.15.89:0/592120950 shutdown_connections 2024-04-02T07:36:35.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.489+0000 7f7675a00700 1 --2- 172.21.15.89:0/592120950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7670160f10 0x7f76700d18b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:35.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.489+0000 7f7675a00700 1 --2- 172.21.15.89:0/592120950 >> v2:172.21.15.113:3300/0 conn(0x7f76700cf230 0x7f76700d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:35.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.489+0000 7f7675a00700 1 -- 172.21.15.89:0/592120950 >> 172.21.15.89:0/592120950 conn(0x7f767008aff0 msgr2=0x7f76700d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:35.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.489+0000 7f7675a00700 1 -- 172.21.15.89:0/592120950 shutdown_connections 2024-04-02T07:36:35.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.489+0000 7f7675a00700 1 -- 172.21.15.89:0/592120950 wait complete. 2024-04-02T07:36:35.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.491+0000 7f7675a00700 1 Processor -- start 2024-04-02T07:36:35.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.491+0000 7f7675a00700 1 -- start start 2024-04-02T07:36:35.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.492+0000 7f7675a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76700cf230 0x7f76701595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:35.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.492+0000 7f7675a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7670160f10 0x7f76700b7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:35.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.493+0000 7f7675a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7670161590 0x7f76701581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:35.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.493+0000 7f7675a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7670153840 con 0x7f7670160f10 2024-04-02T07:36:35.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.493+0000 7f7675a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76700b80f0 con 0x7f7670161590 2024-04-02T07:36:35.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.493+0000 7f7675a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76700b8390 con 0x7f76700cf230 2024-04-02T07:36:35.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.493+0000 7f766e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7670160f10 0x7f76700b7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:35.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.493+0000 7f766e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7670160f10 0x7f76700b7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40814/0 (socket says 172.21.15.89:40814) 2024-04-02T07:36:35.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.493+0000 7f766e000700 1 -- 172.21.15.89:0/3325949144 learned_addr learned my addr 172.21.15.89:0/3325949144 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:35.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.493+0000 7f766f400700 1 --2- 172.21.15.89:0/3325949144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7670161590 0x7f76701581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:35.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.493+0000 7f766ea00700 1 --2- 172.21.15.89:0/3325949144 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76700cf230 0x7f76701595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:35.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.493+0000 7f766e000700 1 -- 172.21.15.89:0/3325949144 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76700cf230 msgr2=0x7f76701595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:35.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.493+0000 7f766e000700 1 --2- 172.21.15.89:0/3325949144 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76700cf230 0x7f76701595a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:35.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.493+0000 7f766e000700 1 -- 172.21.15.89:0/3325949144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7670161590 msgr2=0x7f76701581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:35.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.493+0000 7f766e000700 1 --2- 172.21.15.89:0/3325949144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7670161590 0x7f76701581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:35.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.493+0000 7f766e000700 1 -- 172.21.15.89:0/3325949144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76600094a0 con 0x7f7670160f10 2024-04-02T07:36:35.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.494+0000 7f766e000700 1 --2- 172.21.15.89:0/3325949144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7670160f10 0x7f76700b7bb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f765c00e870 tx=0x7f765c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:35.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.494+0000 7f7666a00700 1 -- 172.21.15.89:0/3325949144 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c009590 con 0x7f7670160f10 2024-04-02T07:36:35.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.494+0000 7f7666a00700 1 -- 172.21.15.89:0/3325949144 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c013680 con 0x7f7670160f10 2024-04-02T07:36:35.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.494+0000 7f7675a00700 1 -- 172.21.15.89:0/3325949144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76700b8670 con 0x7f7670160f10 2024-04-02T07:36:35.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.494+0000 7f7666a00700 1 -- 172.21.15.89:0/3325949144 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c0139f0 con 0x7f7670160f10 2024-04-02T07:36:35.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.494+0000 7f7675a00700 1 -- 172.21.15.89:0/3325949144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7670153a50 con 0x7f7670160f10 2024-04-02T07:36:35.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.495+0000 7f7675a00700 1 -- 172.21.15.89:0/3325949144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7670067720 con 0x7f7670160f10 2024-04-02T07:36:35.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.496+0000 7f7666a00700 1 -- 172.21.15.89:0/3325949144 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f765c013b50 con 0x7f7670160f10 2024-04-02T07:36:35.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.497+0000 7f7666a00700 1 --2- 172.21.15.89:0/3325949144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f765006ac90 0x7f765006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:35.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.497+0000 7f7666a00700 1 -- 172.21.15.89:0/3325949144 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f765c089220 con 0x7f7670160f10 2024-04-02T07:36:35.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.497+0000 7f766ea00700 1 --2- 172.21.15.89:0/3325949144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f765006ac90 0x7f765006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:35.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.497+0000 7f766ea00700 1 --2- 172.21.15.89:0/3325949144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f765006ac90 0x7f765006d150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f76600090b0 tx=0x7f766001c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:35.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.501+0000 7f7666a00700 1 -- 172.21.15.89:0/3325949144 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f765c054360 con 0x7f7670160f10 2024-04-02T07:36:35.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.703+0000 7f7675a00700 1 -- 172.21.15.89:0/3325949144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f767006c160 con 0x7f7670160f10 2024-04-02T07:36:35.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.705+0000 7f7666a00700 1 -- 172.21.15.89:0/3325949144 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f765c057980 con 0x7f7670160f10 2024-04-02T07:36:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:35.707 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:35:54.238913+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:36:35.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.707+0000 7f7675a00700 1 -- 172.21.15.89:0/3325949144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f765006ac90 msgr2=0x7f765006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.707+0000 7f7675a00700 1 --2- 172.21.15.89:0/3325949144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f765006ac90 0x7f765006d150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f76600090b0 tx=0x7f766001c040 comp rx=0 tx=0).stop 2024-04-02T07:36:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.707+0000 7f7675a00700 1 -- 172.21.15.89:0/3325949144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7670160f10 msgr2=0x7f76700b7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.707+0000 7f7675a00700 1 --2- 172.21.15.89:0/3325949144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7670160f10 0x7f76700b7bb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f765c00e870 tx=0x7f765c00eb80 comp rx=0 tx=0).stop 2024-04-02T07:36:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.708+0000 7f7675a00700 1 -- 172.21.15.89:0/3325949144 shutdown_connections 2024-04-02T07:36:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.708+0000 7f7675a00700 1 --2- 172.21.15.89:0/3325949144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f765006ac90 0x7f765006d150 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.708+0000 7f7675a00700 1 --2- 172.21.15.89:0/3325949144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7670161590 0x7f76701581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.708+0000 7f7675a00700 1 --2- 172.21.15.89:0/3325949144 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76700cf230 0x7f76701595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.708+0000 7f7675a00700 1 --2- 172.21.15.89:0/3325949144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7670160f10 0x7f76700b7bb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.708+0000 7f7675a00700 1 -- 172.21.15.89:0/3325949144 >> 172.21.15.89:0/3325949144 conn(0x7f767008aff0 msgr2=0x7f76701525e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.708+0000 7f7675a00700 1 -- 172.21.15.89:0/3325949144 shutdown_connections 2024-04-02T07:36:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.708+0000 7f7675a00700 1 -- 172.21.15.89:0/3325949144 wait complete. 2024-04-02T07:36:35.728 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:35:51.296561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:35:52.232882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:36:35.728 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:36:35.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.922+0000 7ffb92200700 1 Processor -- start 2024-04-02T07:36:35.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.922+0000 7ffb92200700 1 -- start start 2024-04-02T07:36:35.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.923+0000 7ffb92200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb8c15dfe0 0x7ffb8c0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:35.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.923+0000 7ffb92200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb8c0cb310 con 0x7ffb8c0ce670 2024-04-02T07:36:35.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.923+0000 7ffb92200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb8c0711a0 con 0x7ffb8c15a5c0 2024-04-02T07:36:35.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.923+0000 7ffb92200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c0ceb00 con 0x7ffb8c15dfe0 2024-04-02T07:36:35.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.923+0000 7ffb8be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ffb8c0ce670 0x7ffb8c0cba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46922/0 (socket says 172.21.15.89:46922) 2024-04-02T07:36:35.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.923+0000 7ffb8be00700 1 -- 172.21.15.89:0/2074440892 learned_addr learned my addr 172.21.15.89:0/2074440892 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:35.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.924+0000 7ffb8aa00700 1 --2- 172.21.15.89:0/2074440892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb8c15dfe0 0x7ffb8c0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:35.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.924+0000 7ffb8a000700 1 -- 172.21.15.89:0/2074440892 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509614322 0 0) 0x7ffb8c0cb310 con 0x7ffb8c0ce670 2024-04-02T07:36:35.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.924+0000 7ffb8a000700 1 -- 172.21.15.89:0/2074440892 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb700035b0 con 0x7ffb8c0ce670 2024-04-02T07:36:35.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.924+0000 7ffb8a000700 1 -- 172.21.15.89:0/2074440892 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931772313 0 0) 0x7ffb8c0711a0 con 0x7ffb8c15a5c0 2024-04-02T07:36:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.924+0000 7ffb8a000700 1 -- 172.21.15.89:0/2074440892 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb8c0cb310 con 0x7ffb8c15a5c0 2024-04-02T07:36:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.924+0000 7ffb8aa00700 1 -- 172.21.15.89:0/2074440892 >> v1:172.21.15.113:6789/0 conn(0x7ffb8c15a5c0 legacy=0x7ffb8c0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.924+0000 7ffb8aa00700 1 -- 172.21.15.89:0/2074440892 >> v1:172.21.15.89:6789/0 conn(0x7ffb8c0ce670 legacy=0x7ffb8c0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.924+0000 7ffb8aa00700 1 -- 172.21.15.89:0/2074440892 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8c071480 con 0x7ffb8c15dfe0 2024-04-02T07:36:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.924+0000 7ffb8aa00700 1 --2- 172.21.15.89:0/2074440892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb8c15dfe0 0x7ffb8c0ce130 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009870 tx=0x7ffb78009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a376e87f195ae693 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.925+0000 7ffb8a000700 1 -- 172.21.15.89:0/2074440892 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb78011830 con 0x7ffb8c15dfe0 2024-04-02T07:36:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.925+0000 7ffb8a000700 1 -- 172.21.15.89:0/2074440892 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb78011990 con 0x7ffb8c15dfe0 2024-04-02T07:36:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.925+0000 7ffb8a000700 1 -- 172.21.15.89:0/2074440892 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb78011d20 con 0x7ffb8c15dfe0 2024-04-02T07:36:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.925+0000 7ffb92200700 1 -- 172.21.15.89:0/2074440892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb8c15dfe0 msgr2=0x7ffb8c0ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.925+0000 7ffb92200700 1 --2- 172.21.15.89:0/2074440892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb8c15dfe0 0x7ffb8c0ce130 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009870 tx=0x7ffb78009b80 comp rx=0 tx=0).stop 2024-04-02T07:36:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.925+0000 7ffb92200700 1 -- 172.21.15.89:0/2074440892 shutdown_connections 2024-04-02T07:36:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.925+0000 7ffb92200700 1 --2- 172.21.15.89:0/2074440892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb8c15dfe0 0x7ffb8c0ce130 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.925+0000 7ffb92200700 1 -- 172.21.15.89:0/2074440892 >> 172.21.15.89:0/2074440892 conn(0x7ffb8c0b0d70 msgr2=0x7ffb8c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.925+0000 7ffb92200700 1 -- 172.21.15.89:0/2074440892 shutdown_connections 2024-04-02T07:36:35.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.925+0000 7ffb92200700 1 -- 172.21.15.89:0/2074440892 wait complete. 2024-04-02T07:36:35.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.928+0000 7ffb92200700 1 Processor -- start 2024-04-02T07:36:35.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.928+0000 7ffb92200700 1 -- start start 2024-04-02T07:36:35.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.929+0000 7ffb92200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb8c0ce670 0x7ffb8c150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:35.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.929+0000 7ffb92200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb8c15a5c0 0x7ffb8c1598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:35.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.929+0000 7ffb92200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb8c15dfe0 0x7ffb8c0ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.929+0000 7ffb92200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c159e30 con 0x7ffb8c0ce670 2024-04-02T07:36:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.929+0000 7ffb92200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c1f3890 con 0x7ffb8c15dfe0 2024-04-02T07:36:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.929+0000 7ffb92200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c151650 con 0x7ffb8c15a5c0 2024-04-02T07:36:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.929+0000 7ffb8b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb8c0ce670 0x7ffb8c150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.929+0000 7ffb8b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb8c0ce670 0x7ffb8c150180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40824/0 (socket says 172.21.15.89:40824) 2024-04-02T07:36:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.929+0000 7ffb8b400700 1 -- 172.21.15.89:0/459921268 learned_addr learned my addr 172.21.15.89:0/459921268 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.930+0000 7ffb8aa00700 1 --2- 172.21.15.89:0/459921268 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb8c15a5c0 0x7ffb8c1598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.930+0000 7ffb8be00700 1 --2- 172.21.15.89:0/459921268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb8c15dfe0 0x7ffb8c0ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.930+0000 7ffb8b400700 1 -- 172.21.15.89:0/459921268 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb8c15a5c0 msgr2=0x7ffb8c1598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.930+0000 7ffb8b400700 1 --2- 172.21.15.89:0/459921268 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb8c15a5c0 0x7ffb8c1598f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.930+0000 7ffb8b400700 1 -- 172.21.15.89:0/459921268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb8c15dfe0 msgr2=0x7ffb8c0ad0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.930+0000 7ffb8b400700 1 --2- 172.21.15.89:0/459921268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb8c15dfe0 0x7ffb8c0ad0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.930+0000 7ffb8b400700 1 -- 172.21.15.89:0/459921268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb780094a0 con 0x7ffb8c0ce670 2024-04-02T07:36:35.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.930+0000 7ffb8b400700 1 --2- 172.21.15.89:0/459921268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb8c0ce670 0x7ffb8c150180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00c520 tx=0x7ffb7c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:35.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.930+0000 7ffb83400700 1 -- 172.21.15.89:0/459921268 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c0052d0 con 0x7ffb8c0ce670 2024-04-02T07:36:35.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.930+0000 7ffb83400700 1 -- 172.21.15.89:0/459921268 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7c005430 con 0x7ffb8c0ce670 2024-04-02T07:36:35.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.930+0000 7ffb92200700 1 -- 172.21.15.89:0/459921268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8c151ad0 con 0x7ffb8c0ce670 2024-04-02T07:36:35.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.930+0000 7ffb83400700 1 -- 172.21.15.89:0/459921268 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c0038c0 con 0x7ffb8c0ce670 2024-04-02T07:36:35.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.930+0000 7ffb92200700 1 -- 172.21.15.89:0/459921268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8c152090 con 0x7ffb8c0ce670 2024-04-02T07:36:35.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.931+0000 7ffb92200700 1 -- 172.21.15.89:0/459921268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8c152220 con 0x7ffb8c0ce670 2024-04-02T07:36:35.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.932+0000 7ffb83400700 1 -- 172.21.15.89:0/459921268 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffb7c003a20 con 0x7ffb8c0ce670 2024-04-02T07:36:35.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.933+0000 7ffb83400700 1 --2- 172.21.15.89:0/459921268 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb6c06ac90 0x7ffb6c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:35.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.933+0000 7ffb83400700 1 -- 172.21.15.89:0/459921268 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ffb7c0888c0 con 0x7ffb8c0ce670 2024-04-02T07:36:35.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.933+0000 7ffb8aa00700 1 --2- 172.21.15.89:0/459921268 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb6c06ac90 0x7ffb6c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:35.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.934+0000 7ffb8aa00700 1 --2- 172.21.15.89:0/459921268 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb6c06ac90 0x7ffb6c06d150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffb780097c0 tx=0x7ffb7801c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:35.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:35.937+0000 7ffb83400700 1 -- 172.21.15.89:0/459921268 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffb7c053a80 con 0x7ffb8c0ce670 2024-04-02T07:36:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.136+0000 7ffb92200700 1 -- 172.21.15.89:0/459921268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffb8c067720 con 0x7ffb8c0ce670 2024-04-02T07:36:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.136+0000 7ffb83400700 1 -- 172.21.15.89:0/459921268 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffb7c0170a0 con 0x7ffb8c0ce670 2024-04-02T07:36:36.138 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:36:36.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.139+0000 7ffb92200700 1 -- 172.21.15.89:0/459921268 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb6c06ac90 msgr2=0x7ffb6c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:36.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.139+0000 7ffb92200700 1 --2- 172.21.15.89:0/459921268 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb6c06ac90 0x7ffb6c06d150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffb780097c0 tx=0x7ffb7801c040 comp rx=0 tx=0).stop 2024-04-02T07:36:36.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.139+0000 7ffb92200700 1 -- 172.21.15.89:0/459921268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb8c0ce670 msgr2=0x7ffb8c150180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:36.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.139+0000 7ffb92200700 1 --2- 172.21.15.89:0/459921268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb8c0ce670 0x7ffb8c150180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00c520 tx=0x7ffb7c00c8e0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.139+0000 7ffb92200700 1 -- 172.21.15.89:0/459921268 shutdown_connections 2024-04-02T07:36:36.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.139+0000 7ffb92200700 1 --2- 172.21.15.89:0/459921268 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb6c06ac90 0x7ffb6c06d150 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.139+0000 7ffb92200700 1 --2- 172.21.15.89:0/459921268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb8c15dfe0 0x7ffb8c0ad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.139+0000 7ffb92200700 1 --2- 172.21.15.89:0/459921268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb8c0ce670 0x7ffb8c150180 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.139+0000 7ffb92200700 1 --2- 172.21.15.89:0/459921268 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb8c15a5c0 0x7ffb8c1598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.139+0000 7ffb92200700 1 -- 172.21.15.89:0/459921268 >> 172.21.15.89:0/459921268 conn(0x7ffb8c0b0d70 msgr2=0x7ffb8c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:36.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.139+0000 7ffb92200700 1 -- 172.21.15.89:0/459921268 shutdown_connections 2024-04-02T07:36:36.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.139+0000 7ffb92200700 1 -- 172.21.15.89:0/459921268 wait complete. 2024-04-02T07:36:36.158 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:36:36.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.354+0000 7f1db6200700 1 Processor -- start 2024-04-02T07:36:36.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.354+0000 7f1db6200700 1 -- start start 2024-04-02T07:36:36.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.355+0000 7f1db6200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1db015d9d0 0x7f1db00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:36.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.355+0000 7f1db6200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db0161420 con 0x7f1db00d2540 2024-04-02T07:36:36.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.355+0000 7f1db6200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db0161700 con 0x7f1db00d16a0 2024-04-02T07:36:36.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.355+0000 7f1db6200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db015edc0 con 0x7f1db015d9d0 2024-04-02T07:36:36.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.355+0000 7f1dafe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1db00d2540 0x7f1db00ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46930/0 (socket says 172.21.15.89:46930) 2024-04-02T07:36:36.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.355+0000 7f1dafe00700 1 -- 172.21.15.89:0/703866118 learned_addr learned my addr 172.21.15.89:0/703866118 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:36.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.355+0000 7f1daf400700 1 --2- 172.21.15.89:0/703866118 >> v2:172.21.15.113:3300/0 conn(0x7f1db015d9d0 0x7f1db00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:36.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.356+0000 7f1dae000700 1 -- 172.21.15.89:0/703866118 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837294181 0 0) 0x7f1db0161420 con 0x7f1db00d2540 2024-04-02T07:36:36.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.356+0000 7f1dae000700 1 -- 172.21.15.89:0/703866118 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d940035b0 con 0x7f1db00d2540 2024-04-02T07:36:36.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.356+0000 7f1daf400700 1 -- 172.21.15.89:0/703866118 >> v1:172.21.15.113:6789/0 conn(0x7f1db00d16a0 legacy=0x7f1db00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:36.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.356+0000 7f1daf400700 1 -- 172.21.15.89:0/703866118 >> v1:172.21.15.89:6789/0 conn(0x7f1db00d2540 legacy=0x7f1db00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:36.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.356+0000 7f1daf400700 1 -- 172.21.15.89:0/703866118 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db01619e0 con 0x7f1db015d9d0 2024-04-02T07:36:36.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.356+0000 7f1daf400700 1 --2- 172.21.15.89:0/703866118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1db015d9d0 0x7f1db00d3c30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009840 tx=0x7f1da0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89875019ff1cce21 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:36.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.356+0000 7f1dae000700 1 -- 172.21.15.89:0/703866118 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0011830 con 0x7f1db015d9d0 2024-04-02T07:36:36.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.357+0000 7f1dae000700 1 -- 172.21.15.89:0/703866118 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da0011990 con 0x7f1db015d9d0 2024-04-02T07:36:36.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.357+0000 7f1dae000700 1 -- 172.21.15.89:0/703866118 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0011d00 con 0x7f1db015d9d0 2024-04-02T07:36:36.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.357+0000 7f1db6200700 1 -- 172.21.15.89:0/703866118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1db015d9d0 msgr2=0x7f1db00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:36.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.357+0000 7f1db6200700 1 --2- 172.21.15.89:0/703866118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1db015d9d0 0x7f1db00d3c30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009840 tx=0x7f1da0009b50 comp rx=0 tx=0).stop 2024-04-02T07:36:36.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.357+0000 7f1db6200700 1 -- 172.21.15.89:0/703866118 shutdown_connections 2024-04-02T07:36:36.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.357+0000 7f1db6200700 1 --2- 172.21.15.89:0/703866118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1db015d9d0 0x7f1db00d3c30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.357+0000 7f1db6200700 1 -- 172.21.15.89:0/703866118 >> 172.21.15.89:0/703866118 conn(0x7f1db008aff0 msgr2=0x7f1db00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:36.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.357+0000 7f1db6200700 1 -- 172.21.15.89:0/703866118 shutdown_connections 2024-04-02T07:36:36.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.357+0000 7f1db6200700 1 -- 172.21.15.89:0/703866118 wait complete. 2024-04-02T07:36:36.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.359+0000 7f1db6200700 1 Processor -- start 2024-04-02T07:36:36.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.359+0000 7f1db6200700 1 -- start start 2024-04-02T07:36:36.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1db6200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1db00d16a0 0x7f1db01f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:36.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1db6200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1db00d2540 0x7f1db015bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:36.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1db6200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1db015d9d0 0x7f1db00ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:36.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1db6200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db015c120 con 0x7f1db00d16a0 2024-04-02T07:36:36.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1db6200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db01f7410 con 0x7f1db00d2540 2024-04-02T07:36:36.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1db6200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db01f76f0 con 0x7f1db015d9d0 2024-04-02T07:36:36.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1daf400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1db00d16a0 0x7f1db01f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:36.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1daf400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1db00d16a0 0x7f1db01f6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40826/0 (socket says 172.21.15.89:40826) 2024-04-02T07:36:36.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1daf400700 1 -- 172.21.15.89:0/2829318426 learned_addr learned my addr 172.21.15.89:0/2829318426 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:36.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1daea00700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1db00d2540 0x7f1db015bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:36.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1dafe00700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1db015d9d0 0x7f1db00ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:36.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1daf400700 1 -- 172.21.15.89:0/2829318426 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1db015d9d0 msgr2=0x7f1db00ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:36.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1daf400700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1db015d9d0 0x7f1db00ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1daf400700 1 -- 172.21.15.89:0/2829318426 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1db00d2540 msgr2=0x7f1db015bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:36.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1daf400700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1db00d2540 0x7f1db015bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1daf400700 1 -- 172.21.15.89:0/2829318426 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da00094a0 con 0x7f1db00d16a0 2024-04-02T07:36:36.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.360+0000 7f1dafe00700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1db015d9d0 0x7f1db00ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:36.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.361+0000 7f1daf400700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1db00d16a0 0x7f1db01f6e30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009790 tx=0x7f1da0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:36.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.361+0000 7f1da7400700 1 -- 172.21.15.89:0/2829318426 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da00036a0 con 0x7f1db00d16a0 2024-04-02T07:36:36.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.361+0000 7f1da7400700 1 -- 172.21.15.89:0/2829318426 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da001e070 con 0x7f1db00d16a0 2024-04-02T07:36:36.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.361+0000 7f1db6200700 1 -- 172.21.15.89:0/2829318426 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db01f79d0 con 0x7f1db00d16a0 2024-04-02T07:36:36.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.361+0000 7f1da7400700 1 -- 172.21.15.89:0/2829318426 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0003a00 con 0x7f1db00d16a0 2024-04-02T07:36:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.361+0000 7f1db6200700 1 -- 172.21.15.89:0/2829318426 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db01f7cb0 con 0x7f1db00d16a0 2024-04-02T07:36:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.362+0000 7f1db6200700 1 -- 172.21.15.89:0/2829318426 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db00d1ca0 con 0x7f1db00d16a0 2024-04-02T07:36:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.363+0000 7f1da7400700 1 -- 172.21.15.89:0/2829318426 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1da0003b60 con 0x7f1db00d16a0 2024-04-02T07:36:36.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.364+0000 7f1da7400700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1d9006ac90 0x7f1d9006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:36.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.364+0000 7f1da7400700 1 -- 172.21.15.89:0/2829318426 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1da008a530 con 0x7f1db00d16a0 2024-04-02T07:36:36.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.364+0000 7f1daea00700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1d9006ac90 0x7f1d9006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:36.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.365+0000 7f1daea00700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1d9006ac90 0x7f1d9006d150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1db00d2050 tx=0x7f1d9c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:36.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.369+0000 7f1da7400700 1 -- 172.21.15.89:0/2829318426 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1da00556e0 con 0x7f1db00d16a0 2024-04-02T07:36:36.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.573+0000 7f1db6200700 1 -- 172.21.15.89:0/2829318426 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1db00b8480 con 0x7f1db00d16a0 2024-04-02T07:36:36.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.574+0000 7f1da7400700 1 -- 172.21.15.89:0/2829318426 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1da0058d00 con 0x7f1db00d16a0 2024-04-02T07:36:36.576 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:36:36.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.576+0000 7f1db6200700 1 -- 172.21.15.89:0/2829318426 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1d9006ac90 msgr2=0x7f1d9006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:36.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.576+0000 7f1db6200700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1d9006ac90 0x7f1d9006d150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1db00d2050 tx=0x7f1d9c009040 comp rx=0 tx=0).stop 2024-04-02T07:36:36.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.576+0000 7f1db6200700 1 -- 172.21.15.89:0/2829318426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1db00d16a0 msgr2=0x7f1db01f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:36.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.576+0000 7f1db6200700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1db00d16a0 0x7f1db01f6e30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009790 tx=0x7f1da0004750 comp rx=0 tx=0).stop 2024-04-02T07:36:36.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.577+0000 7f1db6200700 1 -- 172.21.15.89:0/2829318426 shutdown_connections 2024-04-02T07:36:36.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.577+0000 7f1db6200700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1d9006ac90 0x7f1d9006d150 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.577+0000 7f1db6200700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1db015d9d0 0x7f1db00ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.577+0000 7f1db6200700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1db00d16a0 0x7f1db01f6e30 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.577+0000 7f1db6200700 1 --2- 172.21.15.89:0/2829318426 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1db00d2540 0x7f1db015bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.577+0000 7f1db6200700 1 -- 172.21.15.89:0/2829318426 >> 172.21.15.89:0/2829318426 conn(0x7f1db008aff0 msgr2=0x7f1db006c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:36.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.577+0000 7f1db6200700 1 -- 172.21.15.89:0/2829318426 shutdown_connections 2024-04-02T07:36:36.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.577+0000 7f1db6200700 1 -- 172.21.15.89:0/2829318426 wait complete. 2024-04-02T07:36:36.596 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:36:36.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.788+0000 7fb899200700 1 Processor -- start 2024-04-02T07:36:36.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.788+0000 7fb899200700 1 -- start start 2024-04-02T07:36:36.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.789+0000 7fb899200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb89415d9d0 0x7fb8940d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:36.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.789+0000 7fb899200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb8940d16a0 0x7fb8940d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:36.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.789+0000 7fb899200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb894161420 con 0x7fb8940d2370 2024-04-02T07:36:36.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.789+0000 7fb899200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8940d2820 con 0x7fb8940d16a0 2024-04-02T07:36:36.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.789+0000 7fb899200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb894161700 con 0x7fb89415d9d0 2024-04-02T07:36:36.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.789+0000 7fb892800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb8940d2370 0x7fb89415ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46944/0 (socket says 172.21.15.89:46944) 2024-04-02T07:36:36.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.789+0000 7fb892800700 1 -- 172.21.15.89:0/3551506364 learned_addr learned my addr 172.21.15.89:0/3551506364 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:36.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.790+0000 7fb891400700 1 --2- 172.21.15.89:0/3551506364 >> v2:172.21.15.89:3300/0 conn(0x7fb8940d16a0 0x7fb8940d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:36.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.790+0000 7fb891e00700 1 --2- 172.21.15.89:0/3551506364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb89415d9d0 0x7fb8940d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:36.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.790+0000 7fb890a00700 1 -- 172.21.15.89:0/3551506364 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569817847 0 0) 0x7fb894161420 con 0x7fb8940d2370 2024-04-02T07:36:36.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.790+0000 7fb891400700 1 -- 172.21.15.89:0/3551506364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb89415d9d0 msgr2=0x7fb8940d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:36.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.790+0000 7fb891400700 1 --2- 172.21.15.89:0/3551506364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb89415d9d0 0x7fb8940d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.790+0000 7fb891400700 1 -- 172.21.15.89:0/3551506364 >> v1:172.21.15.89:6789/0 conn(0x7fb8940d2370 legacy=0x7fb89415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:36.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.790+0000 7fb891400700 1 -- 172.21.15.89:0/3551506364 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8941619e0 con 0x7fb8940d16a0 2024-04-02T07:36:36.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.790+0000 7fb891400700 1 --2- 172.21.15.89:0/3551506364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8940d16a0 0x7fb8940d1e30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb880009820 tx=0x7fb880009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a70b12e6f3eb1c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:36.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.790+0000 7fb890a00700 1 -- 172.21.15.89:0/3551506364 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb880005e30 con 0x7fb8940d16a0 2024-04-02T07:36:36.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.791+0000 7fb890a00700 1 -- 172.21.15.89:0/3551506364 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb880006030 con 0x7fb8940d16a0 2024-04-02T07:36:36.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.791+0000 7fb890a00700 1 -- 172.21.15.89:0/3551506364 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8800063c0 con 0x7fb8940d16a0 2024-04-02T07:36:36.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.791+0000 7fb899200700 1 -- 172.21.15.89:0/3551506364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8940d16a0 msgr2=0x7fb8940d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:36.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.791+0000 7fb899200700 1 --2- 172.21.15.89:0/3551506364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8940d16a0 0x7fb8940d1e30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb880009820 tx=0x7fb880009b30 comp rx=0 tx=0).stop 2024-04-02T07:36:36.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.791+0000 7fb899200700 1 -- 172.21.15.89:0/3551506364 shutdown_connections 2024-04-02T07:36:36.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.791+0000 7fb899200700 1 --2- 172.21.15.89:0/3551506364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb89415d9d0 0x7fb8940d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.791+0000 7fb899200700 1 --2- 172.21.15.89:0/3551506364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8940d16a0 0x7fb8940d1e30 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.791+0000 7fb899200700 1 -- 172.21.15.89:0/3551506364 >> 172.21.15.89:0/3551506364 conn(0x7fb89408aff0 msgr2=0x7fb8940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:36.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.791+0000 7fb899200700 1 -- 172.21.15.89:0/3551506364 shutdown_connections 2024-04-02T07:36:36.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.791+0000 7fb899200700 1 -- 172.21.15.89:0/3551506364 wait complete. 2024-04-02T07:36:36.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.793+0000 7fb899200700 1 Processor -- start 2024-04-02T07:36:36.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.793+0000 7fb899200700 1 -- start start 2024-04-02T07:36:36.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb899200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8940d16a0 0x7fb89415f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:36.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb899200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8940d2370 0x7fb8941f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:36.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb899200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb89415d9d0 0x7fb8941f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:36.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb899200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940b62c0 con 0x7fb8940d2370 2024-04-02T07:36:36.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb899200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89415f680 con 0x7fb8940d16a0 2024-04-02T07:36:36.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb899200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb894160a10 con 0x7fb89415d9d0 2024-04-02T07:36:36.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb891400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8940d2370 0x7fb8941f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:36.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb891400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8940d2370 0x7fb8941f38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40834/0 (socket says 172.21.15.89:40834) 2024-04-02T07:36:36.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb891400700 1 -- 172.21.15.89:0/292642776 learned_addr learned my addr 172.21.15.89:0/292642776 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:36.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb891e00700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8940d16a0 0x7fb89415f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:36.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb892800700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb89415d9d0 0x7fb8941f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:36.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb891400700 1 -- 172.21.15.89:0/292642776 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb89415d9d0 msgr2=0x7fb8941f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:36.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb891400700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb89415d9d0 0x7fb8941f3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb891400700 1 -- 172.21.15.89:0/292642776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8940d16a0 msgr2=0x7fb89415f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:36.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb891400700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8940d16a0 0x7fb89415f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:36.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb891400700 1 -- 172.21.15.89:0/292642776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb880009430 con 0x7fb8940d2370 2024-04-02T07:36:36.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.794+0000 7fb892800700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb89415d9d0 0x7fb8941f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:36.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.795+0000 7fb891400700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8940d2370 0x7fb8941f38b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb880000c00 tx=0x7fb880004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:36.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.795+0000 7fb88a000700 1 -- 172.21.15.89:0/292642776 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8800063a0 con 0x7fb8940d2370 2024-04-02T07:36:36.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.795+0000 7fb88a000700 1 -- 172.21.15.89:0/292642776 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb880020070 con 0x7fb8940d2370 2024-04-02T07:36:36.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.795+0000 7fb88a000700 1 -- 172.21.15.89:0/292642776 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb880006740 con 0x7fb8940d2370 2024-04-02T07:36:36.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.795+0000 7fb899200700 1 -- 172.21.15.89:0/292642776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8940b6510 con 0x7fb8940d2370 2024-04-02T07:36:36.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.795+0000 7fb899200700 1 -- 172.21.15.89:0/292642776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8940b6ad0 con 0x7fb8940d2370 2024-04-02T07:36:36.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.796+0000 7fb899200700 1 -- 172.21.15.89:0/292642776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8940d1ca0 con 0x7fb8940d2370 2024-04-02T07:36:36.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.797+0000 7fb88a000700 1 -- 172.21.15.89:0/292642776 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb880022030 con 0x7fb8940d2370 2024-04-02T07:36:36.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.798+0000 7fb88a000700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb87406ac90 0x7fb87406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:36.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.798+0000 7fb88a000700 1 -- 172.21.15.89:0/292642776 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb88008a910 con 0x7fb8940d2370 2024-04-02T07:36:36.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.798+0000 7fb891e00700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb87406ac90 0x7fb87406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:36.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.799+0000 7fb891e00700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb87406ac90 0x7fb87406d150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb884009ad0 tx=0x7fb884008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:36.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:36.802+0000 7fb88a000700 1 -- 172.21.15.89:0/292642776 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb880055ac0 con 0x7fb8940d2370 2024-04-02T07:36:37.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.005+0000 7fb899200700 1 -- 172.21.15.89:0/292642776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb894067720 con 0x7fb8940d2370 2024-04-02T07:36:37.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.005+0000 7fb88a000700 1 -- 172.21.15.89:0/292642776 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb8800068a0 con 0x7fb8940d2370 2024-04-02T07:36:37.007 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:36:37.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.008+0000 7fb899200700 1 -- 172.21.15.89:0/292642776 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb87406ac90 msgr2=0x7fb87406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:37.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.008+0000 7fb899200700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb87406ac90 0x7fb87406d150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb884009ad0 tx=0x7fb884008040 comp rx=0 tx=0).stop 2024-04-02T07:36:37.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.008+0000 7fb899200700 1 -- 172.21.15.89:0/292642776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8940d2370 msgr2=0x7fb8941f38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:37.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.008+0000 7fb899200700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8940d2370 0x7fb8941f38b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb880000c00 tx=0x7fb880004750 comp rx=0 tx=0).stop 2024-04-02T07:36:37.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.008+0000 7fb899200700 1 -- 172.21.15.89:0/292642776 shutdown_connections 2024-04-02T07:36:37.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.008+0000 7fb899200700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb87406ac90 0x7fb87406d150 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:37.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.008+0000 7fb899200700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8940d16a0 0x7fb89415f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:37.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.008+0000 7fb899200700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb89415d9d0 0x7fb8941f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:37.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.008+0000 7fb899200700 1 --2- 172.21.15.89:0/292642776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8940d2370 0x7fb8941f38b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:37.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.008+0000 7fb899200700 1 -- 172.21.15.89:0/292642776 >> 172.21.15.89:0/292642776 conn(0x7fb89408aff0 msgr2=0x7fb89415c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:37.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.008+0000 7fb899200700 1 -- 172.21.15.89:0/292642776 shutdown_connections 2024-04-02T07:36:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.008+0000 7fb899200700 1 -- 172.21.15.89:0/292642776 wait complete. 2024-04-02T07:36:37.027 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T07:36:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.219+0000 7f0cd2e00700 1 Processor -- start 2024-04-02T07:36:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.219+0000 7f0cd2e00700 1 -- start start 2024-04-02T07:36:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.220+0000 7f0cd2e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ccc15d9d0 0x7f0ccc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.220+0000 7f0cd2e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc0d16a0 0x7f0ccc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.220+0000 7f0cd2e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ccc161420 con 0x7f0ccc0d2370 2024-04-02T07:36:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.220+0000 7f0cd2e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ccc0d2820 con 0x7f0ccc15d9d0 2024-04-02T07:36:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.220+0000 7f0cd2e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc161700 con 0x7f0ccc0d16a0 2024-04-02T07:36:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.221+0000 7f0ccbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ccc15d9d0 0x7f0ccc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.221+0000 7f0ccbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ccc15d9d0 0x7f0ccc0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40840/0 (socket says 172.21.15.89:40840) 2024-04-02T07:36:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.221+0000 7f0ccbe00700 1 -- 172.21.15.89:0/770142085 learned_addr learned my addr 172.21.15.89:0/770142085 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.221+0000 7f0cd0e00700 1 --1- 172.21.15.89:0/770142085 >> v1:172.21.15.89:6789/0 conn(0x7f0ccc0d2370 0x7f0ccc15ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46952/0 (socket says 172.21.15.89:46952) 2024-04-02T07:36:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.221+0000 7f0ccb400700 1 --2- 172.21.15.89:0/770142085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc0d16a0 0x7f0ccc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.221+0000 7f0ccbe00700 1 -- 172.21.15.89:0/770142085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc0d16a0 msgr2=0x7f0ccc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.221+0000 7f0ccbe00700 1 --2- 172.21.15.89:0/770142085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc0d16a0 0x7f0ccc0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.221+0000 7f0ccbe00700 1 -- 172.21.15.89:0/770142085 >> v1:172.21.15.89:6789/0 conn(0x7f0ccc0d2370 legacy=0x7f0ccc15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.221+0000 7f0ccbe00700 1 -- 172.21.15.89:0/770142085 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc1619e0 con 0x7f0ccc15d9d0 2024-04-02T07:36:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.221+0000 7f0ccb400700 1 --2- 172.21.15.89:0/770142085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc0d16a0 0x7f0ccc0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.221+0000 7f0ccaa00700 1 -- 172.21.15.89:0/770142085 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121392825 0 0) 0x7f0ccc161420 con 0x7f0ccc0d2370 2024-04-02T07:36:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.221+0000 7f0ccbe00700 1 --2- 172.21.15.89:0/770142085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ccc15d9d0 0x7f0ccc0d3c30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc0098f0 tx=0x7f0cbc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e483f2081301cad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:37.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.222+0000 7f0ccaa00700 1 -- 172.21.15.89:0/770142085 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc00b8a0 con 0x7f0ccc15d9d0 2024-04-02T07:36:37.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.222+0000 7f0ccaa00700 1 -- 172.21.15.89:0/770142085 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cbc00ba00 con 0x7f0ccc15d9d0 2024-04-02T07:36:37.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.222+0000 7f0ccaa00700 1 -- 172.21.15.89:0/770142085 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc00bd90 con 0x7f0ccc15d9d0 2024-04-02T07:36:37.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.222+0000 7f0cd2e00700 1 -- 172.21.15.89:0/770142085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ccc15d9d0 msgr2=0x7f0ccc0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:37.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.222+0000 7f0cd2e00700 1 --2- 172.21.15.89:0/770142085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ccc15d9d0 0x7f0ccc0d3c30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc0098f0 tx=0x7f0cbc009c00 comp rx=0 tx=0).stop 2024-04-02T07:36:37.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.222+0000 7f0cd2e00700 1 -- 172.21.15.89:0/770142085 shutdown_connections 2024-04-02T07:36:37.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.222+0000 7f0cd2e00700 1 --2- 172.21.15.89:0/770142085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ccc15d9d0 0x7f0ccc0d3c30 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:37.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.222+0000 7f0cd2e00700 1 --2- 172.21.15.89:0/770142085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc0d16a0 0x7f0ccc0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:37.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.222+0000 7f0cd2e00700 1 -- 172.21.15.89:0/770142085 >> 172.21.15.89:0/770142085 conn(0x7f0ccc08aff0 msgr2=0x7f0ccc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:37.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.222+0000 7f0cd2e00700 1 -- 172.21.15.89:0/770142085 shutdown_connections 2024-04-02T07:36:37.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.222+0000 7f0cd2e00700 1 -- 172.21.15.89:0/770142085 wait complete. 2024-04-02T07:36:37.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.224+0000 7f0cd2e00700 1 Processor -- start 2024-04-02T07:36:37.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.224+0000 7f0cd2e00700 1 -- start start 2024-04-02T07:36:37.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.225+0000 7f0cd2e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ccc0d16a0 0x7f0ccc14e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:37.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.225+0000 7f0cd2e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ccc0d2370 0x7f0ccc0b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:37.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.225+0000 7f0cd2e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc15d9d0 0x7f0ccc0b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:37.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.225+0000 7f0cd2e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc15e4d0 con 0x7f0ccc0d16a0 2024-04-02T07:36:37.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.225+0000 7f0cd2e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc14eeb0 con 0x7f0ccc0d2370 2024-04-02T07:36:37.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.225+0000 7f0cd2e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc15de50 con 0x7f0ccc15d9d0 2024-04-02T07:36:37.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.225+0000 7f0ccbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ccc0d16a0 0x7f0ccc14e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:37.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.225+0000 7f0ccbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ccc0d16a0 0x7f0ccc14e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40842/0 (socket says 172.21.15.89:40842) 2024-04-02T07:36:37.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.225+0000 7f0ccbe00700 1 -- 172.21.15.89:0/1634598034 learned_addr learned my addr 172.21.15.89:0/1634598034 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:37.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.225+0000 7f0cd0e00700 1 --2- 172.21.15.89:0/1634598034 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc15d9d0 0x7f0ccc0b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:37.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.225+0000 7f0ccb400700 1 --2- 172.21.15.89:0/1634598034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ccc0d2370 0x7f0ccc0b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:37.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.226+0000 7f0ccbe00700 1 -- 172.21.15.89:0/1634598034 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc15d9d0 msgr2=0x7f0ccc0b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:37.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.226+0000 7f0ccbe00700 1 --2- 172.21.15.89:0/1634598034 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc15d9d0 0x7f0ccc0b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.226+0000 7f0ccbe00700 1 -- 172.21.15.89:0/1634598034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ccc0d2370 msgr2=0x7f0ccc0b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.226+0000 7f0ccbe00700 1 --2- 172.21.15.89:0/1634598034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ccc0d2370 0x7f0ccc0b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.226+0000 7f0ccbe00700 1 -- 172.21.15.89:0/1634598034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc009550 con 0x7f0ccc0d16a0 2024-04-02T07:36:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.226+0000 7f0ccbe00700 1 --2- 172.21.15.89:0/1634598034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ccc0d16a0 0x7f0ccc14e970 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc014040 tx=0x7f0cbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.226+0000 7f0cc3e00700 1 -- 172.21.15.89:0/1634598034 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc0036a0 con 0x7f0ccc0d16a0 2024-04-02T07:36:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.226+0000 7f0cc3e00700 1 -- 172.21.15.89:0/1634598034 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cbc01e070 con 0x7f0ccc0d16a0 2024-04-02T07:36:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.226+0000 7f0cd2e00700 1 -- 172.21.15.89:0/1634598034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ccc15f8e0 con 0x7f0ccc0d16a0 2024-04-02T07:36:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.226+0000 7f0cc3e00700 1 -- 172.21.15.89:0/1634598034 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc003a20 con 0x7f0ccc0d16a0 2024-04-02T07:36:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.226+0000 7f0cd2e00700 1 -- 172.21.15.89:0/1634598034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ccc15fba0 con 0x7f0ccc0d16a0 2024-04-02T07:36:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.227+0000 7f0cd2e00700 1 -- 172.21.15.89:0/1634598034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ccc0d1ca0 con 0x7f0ccc0d16a0 2024-04-02T07:36:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.228+0000 7f0cc3e00700 1 -- 172.21.15.89:0/1634598034 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0cbc003b80 con 0x7f0ccc0d16a0 2024-04-02T07:36:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.230+0000 7f0cc3e00700 1 --2- 172.21.15.89:0/1634598034 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0cac06ac90 0x7f0cac06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.230+0000 7f0cc3e00700 1 -- 172.21.15.89:0/1634598034 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0cbc020070 con 0x7f0ccc0d16a0 2024-04-02T07:36:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.230+0000 7f0ccb400700 1 --2- 172.21.15.89:0/1634598034 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0cac06ac90 0x7f0cac06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.230+0000 7f0ccb400700 1 --2- 172.21.15.89:0/1634598034 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0cac06ac90 0x7f0cac06d150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc1f6c70 tx=0x7f0cb8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:37.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.234+0000 7f0cc3e00700 1 -- 172.21.15.89:0/1634598034 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0cbc055a90 con 0x7f0ccc0d16a0 2024-04-02T07:36:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.450+0000 7f0cd2e00700 1 -- 172.21.15.89:0/1634598034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0ccc15d0c0 con 0x7f0ccc0d16a0 2024-04-02T07:36:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.451+0000 7f0cc3e00700 1 -- 172.21.15.89:0/1634598034 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0cbc02a080 con 0x7f0ccc0d16a0 2024-04-02T07:36:37.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.453+0000 7f0cd2e00700 1 -- 172.21.15.89:0/1634598034 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0cac06ac90 msgr2=0x7f0cac06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:37.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.453+0000 7f0cd2e00700 1 --2- 172.21.15.89:0/1634598034 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0cac06ac90 0x7f0cac06d150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc1f6c70 tx=0x7f0cb8008040 comp rx=0 tx=0).stop 2024-04-02T07:36:37.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.453+0000 7f0cd2e00700 1 -- 172.21.15.89:0/1634598034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ccc0d16a0 msgr2=0x7f0ccc14e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:37.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.453+0000 7f0cd2e00700 1 --2- 172.21.15.89:0/1634598034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ccc0d16a0 0x7f0ccc14e970 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc014040 tx=0x7f0cbc004750 comp rx=0 tx=0).stop 2024-04-02T07:36:37.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.453+0000 7f0cd2e00700 1 -- 172.21.15.89:0/1634598034 shutdown_connections 2024-04-02T07:36:37.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.453+0000 7f0cd2e00700 1 --2- 172.21.15.89:0/1634598034 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0cac06ac90 0x7f0cac06d150 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:37.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.453+0000 7f0cd2e00700 1 --2- 172.21.15.89:0/1634598034 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc15d9d0 0x7f0ccc0b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:37.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.453+0000 7f0cd2e00700 1 --2- 172.21.15.89:0/1634598034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ccc0d16a0 0x7f0ccc14e970 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:37.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.453+0000 7f0cd2e00700 1 --2- 172.21.15.89:0/1634598034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ccc0d2370 0x7f0ccc0b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:37.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.453+0000 7f0cd2e00700 1 -- 172.21.15.89:0/1634598034 >> 172.21.15.89:0/1634598034 conn(0x7f0ccc08aff0 msgr2=0x7f0ccc06cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:37.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.453+0000 7f0cd2e00700 1 -- 172.21.15.89:0/1634598034 shutdown_connections 2024-04-02T07:36:37.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:37.454+0000 7f0cd2e00700 1 -- 172.21.15.89:0/1634598034 wait complete. 2024-04-02T07:36:37.457 INFO:teuthology.orchestra.run.smithi089.stderr:listed 0 entries 2024-04-02T07:36:37.473 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-02T07:36:37.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:37.473 DEBUG:teuthology.orchestra.run.smithi113:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-04-02T07:36:37.552 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi113.front.sepia.ceph.com 2024-04-02T07:36:37.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:37.552 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-02T07:36:37.741 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:36:37.741 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:37.786 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:36:37.786 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:37.871 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:36:37.871 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:37.955 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:36:38.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.185+0000 7fe8c6400700 1 Processor -- start 2024-04-02T07:36:38.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.185+0000 7fe8c6400700 1 -- start start 2024-04-02T07:36:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.186+0000 7fe8c6400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe8c0160f10 0x7fe8c00d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.186+0000 7fe8c6400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8c00cf230 0x7fe8c00d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.186+0000 7fe8c6400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe8c0161590 0x7fe8c0161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.186+0000 7fe8c6400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8c0161f60 con 0x7fe8c0161590 2024-04-02T07:36:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.186+0000 7fe8c6400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8c00ceda0 con 0x7fe8c0160f10 2024-04-02T07:36:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.186+0000 7fe8c6400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c00d23d0 con 0x7fe8c00cf230 2024-04-02T07:36:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.186+0000 7fe8bfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe8c0161590 0x7fe8c0161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.187+0000 7fe8bfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe8c0161590 0x7fe8c0161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40858/0 (socket says 172.21.15.89:40858) 2024-04-02T07:36:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.187+0000 7fe8bfe00700 1 -- 172.21.15.89:0/358028841 learned_addr learned my addr 172.21.15.89:0/358028841 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.187+0000 7fe8bea00700 1 --2- 172.21.15.89:0/358028841 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8c00cf230 0x7fe8c00d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.187+0000 7fe8bf400700 1 --2- 172.21.15.89:0/358028841 >> v2:172.21.15.113:3300/0 conn(0x7fe8c0160f10 0x7fe8c00d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.187+0000 7fe8bfe00700 1 -- 172.21.15.89:0/358028841 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8c00cf230 msgr2=0x7fe8c00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.187+0000 7fe8bfe00700 1 --2- 172.21.15.89:0/358028841 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8c00cf230 0x7fe8c00d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.187+0000 7fe8bfe00700 1 -- 172.21.15.89:0/358028841 >> v2:172.21.15.113:3300/0 conn(0x7fe8c0160f10 msgr2=0x7fe8c00d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.187+0000 7fe8bfe00700 1 --2- 172.21.15.89:0/358028841 >> v2:172.21.15.113:3300/0 conn(0x7fe8c0160f10 0x7fe8c00d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.187+0000 7fe8bfe00700 1 -- 172.21.15.89:0/358028841 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c01612f0 con 0x7fe8c0161590 2024-04-02T07:36:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.187+0000 7fe8bf400700 1 --2- 172.21.15.89:0/358028841 >> v2:172.21.15.113:3300/0 conn(0x7fe8c0160f10 0x7fe8c00d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.187+0000 7fe8bfe00700 1 --2- 172.21.15.89:0/358028841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8c0161590 0x7fe8c0161a20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe8c00d1080 tx=0x7fe8b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965f5eb245091ce3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.187+0000 7fe8be000700 1 -- 172.21.15.89:0/358028841 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b800e070 con 0x7fe8c0161590 2024-04-02T07:36:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.187+0000 7fe8be000700 1 -- 172.21.15.89:0/358028841 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b800b910 con 0x7fe8c0161590 2024-04-02T07:36:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.187+0000 7fe8be000700 1 -- 172.21.15.89:0/358028841 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b800bcc0 con 0x7fe8c0161590 2024-04-02T07:36:38.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.188+0000 7fe8c6400700 1 -- 172.21.15.89:0/358028841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8c0161590 msgr2=0x7fe8c0161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:38.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.188+0000 7fe8c6400700 1 --2- 172.21.15.89:0/358028841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8c0161590 0x7fe8c0161a20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe8c00d1080 tx=0x7fe8b800b330 comp rx=0 tx=0).stop 2024-04-02T07:36:38.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.188+0000 7fe8c6400700 1 -- 172.21.15.89:0/358028841 shutdown_connections 2024-04-02T07:36:38.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.188+0000 7fe8c6400700 1 --2- 172.21.15.89:0/358028841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8c0161590 0x7fe8c0161a20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.188+0000 7fe8c6400700 1 --2- 172.21.15.89:0/358028841 >> v2:172.21.15.113:3300/0 conn(0x7fe8c0160f10 0x7fe8c00d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.188+0000 7fe8c6400700 1 --2- 172.21.15.89:0/358028841 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8c00cf230 0x7fe8c00d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.188+0000 7fe8c6400700 1 -- 172.21.15.89:0/358028841 >> 172.21.15.89:0/358028841 conn(0x7fe8c008aff0 msgr2=0x7fe8c00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:38.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.188+0000 7fe8c6400700 1 -- 172.21.15.89:0/358028841 shutdown_connections 2024-04-02T07:36:38.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.188+0000 7fe8c6400700 1 -- 172.21.15.89:0/358028841 wait complete. 2024-04-02T07:36:38.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.190+0000 7fe8c6400700 1 Processor -- start 2024-04-02T07:36:38.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.191+0000 7fe8c6400700 1 -- start start 2024-04-02T07:36:38.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8c6400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8c00cf230 0x7fe8c00b6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:38.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8c6400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8c0160f10 0x7fe8c01f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:38.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8c6400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8c0161590 0x7fe8c015f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:38.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8c6400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c00b7670 con 0x7fe8c00cf230 2024-04-02T07:36:38.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8c6400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c00b7b00 con 0x7fe8c0161590 2024-04-02T07:36:38.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8c6400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c00b6a60 con 0x7fe8c0160f10 2024-04-02T07:36:38.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8bf400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8c00cf230 0x7fe8c00b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:38.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8bf400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8c00cf230 0x7fe8c00b6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40868/0 (socket says 172.21.15.89:40868) 2024-04-02T07:36:38.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8bf400700 1 -- 172.21.15.89:0/2058301717 learned_addr learned my addr 172.21.15.89:0/2058301717 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:38.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8bea00700 1 --2- 172.21.15.89:0/2058301717 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8c0160f10 0x7fe8c01f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8bfe00700 1 --2- 172.21.15.89:0/2058301717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8c0161590 0x7fe8c015f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8bf400700 1 -- 172.21.15.89:0/2058301717 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8c0160f10 msgr2=0x7fe8c01f65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8bf400700 1 --2- 172.21.15.89:0/2058301717 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8c0160f10 0x7fe8c01f65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8bf400700 1 -- 172.21.15.89:0/2058301717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8c0161590 msgr2=0x7fe8c015f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8bf400700 1 --2- 172.21.15.89:0/2058301717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8c0161590 0x7fe8c015f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.192+0000 7fe8bf400700 1 -- 172.21.15.89:0/2058301717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b8009b30 con 0x7fe8c00cf230 2024-04-02T07:36:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.193+0000 7fe8bf400700 1 --2- 172.21.15.89:0/2058301717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8c00cf230 0x7fe8c00b6180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000e760 tx=0x7fe8b000ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.193+0000 7fe8b7400700 1 -- 172.21.15.89:0/2058301717 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b000c070 con 0x7fe8c00cf230 2024-04-02T07:36:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.193+0000 7fe8b7400700 1 -- 172.21.15.89:0/2058301717 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b0013610 con 0x7fe8c00cf230 2024-04-02T07:36:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.193+0000 7fe8c6400700 1 -- 172.21.15.89:0/2058301717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c00b6d40 con 0x7fe8c00cf230 2024-04-02T07:36:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.193+0000 7fe8b7400700 1 -- 172.21.15.89:0/2058301717 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b00139c0 con 0x7fe8c00cf230 2024-04-02T07:36:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.193+0000 7fe8c6400700 1 -- 172.21.15.89:0/2058301717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c00b7300 con 0x7fe8c00cf230 2024-04-02T07:36:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.194+0000 7fe8c6400700 1 -- 172.21.15.89:0/2058301717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c0067720 con 0x7fe8c00cf230 2024-04-02T07:36:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.195+0000 7fe8b7400700 1 -- 172.21.15.89:0/2058301717 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe8b0013b20 con 0x7fe8c00cf230 2024-04-02T07:36:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.196+0000 7fe8b7400700 1 --2- 172.21.15.89:0/2058301717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe8a006ac90 0x7fe8a006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.196+0000 7fe8b7400700 1 -- 172.21.15.89:0/2058301717 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe8b0088fb0 con 0x7fe8c00cf230 2024-04-02T07:36:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.196+0000 7fe8bea00700 1 --2- 172.21.15.89:0/2058301717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe8a006ac90 0x7fe8a006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.197+0000 7fe8bea00700 1 --2- 172.21.15.89:0/2058301717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe8a006ac90 0x7fe8a006d150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe8c00b95f0 tx=0x7fe8ac009210 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.200+0000 7fe8b7400700 1 -- 172.21.15.89:0/2058301717 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe8b0054200 con 0x7fe8c00cf230 2024-04-02T07:36:38.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.404+0000 7fe8c6400700 1 -- 172.21.15.89:0/2058301717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8c00d1e90 con 0x7fe8c00cf230 2024-04-02T07:36:38.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.406+0000 7fe8b7400700 1 -- 172.21.15.89:0/2058301717 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7fe8b0057820 con 0x7fe8c00cf230 2024-04-02T07:36:38.408 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:38.408 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:36:36.860889+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:36:38.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.408+0000 7fe8c6400700 1 -- 172.21.15.89:0/2058301717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe8a006ac90 msgr2=0x7fe8a006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:38.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.408+0000 7fe8c6400700 1 --2- 172.21.15.89:0/2058301717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe8a006ac90 0x7fe8a006d150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe8c00b95f0 tx=0x7fe8ac009210 comp rx=0 tx=0).stop 2024-04-02T07:36:38.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.408+0000 7fe8c6400700 1 -- 172.21.15.89:0/2058301717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8c00cf230 msgr2=0x7fe8c00b6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:38.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.408+0000 7fe8c6400700 1 --2- 172.21.15.89:0/2058301717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8c00cf230 0x7fe8c00b6180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000e760 tx=0x7fe8b000ea70 comp rx=0 tx=0).stop 2024-04-02T07:36:38.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.408+0000 7fe8c6400700 1 -- 172.21.15.89:0/2058301717 shutdown_connections 2024-04-02T07:36:38.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.408+0000 7fe8c6400700 1 --2- 172.21.15.89:0/2058301717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe8a006ac90 0x7fe8a006d150 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.408+0000 7fe8c6400700 1 --2- 172.21.15.89:0/2058301717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8c0161590 0x7fe8c015f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.408+0000 7fe8c6400700 1 --2- 172.21.15.89:0/2058301717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8c00cf230 0x7fe8c00b6180 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.408+0000 7fe8c6400700 1 --2- 172.21.15.89:0/2058301717 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe8c0160f10 0x7fe8c01f65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.408+0000 7fe8c6400700 1 -- 172.21.15.89:0/2058301717 >> 172.21.15.89:0/2058301717 conn(0x7fe8c008aff0 msgr2=0x7fe8c00cc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:38.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.408+0000 7fe8c6400700 1 -- 172.21.15.89:0/2058301717 shutdown_connections 2024-04-02T07:36:38.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.408+0000 7fe8c6400700 1 -- 172.21.15.89:0/2058301717 wait complete. 2024-04-02T07:36:38.428 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:35:51.296561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:35:52.232882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:36:38.429 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:36:38.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.619+0000 7fe62c800700 1 Processor -- start 2024-04-02T07:36:38.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.619+0000 7fe62c800700 1 -- start start 2024-04-02T07:36:38.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.620+0000 7fe62c800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe62415d9d0 0x7fe6240d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:38.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.620+0000 7fe62c800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6240d16a0 0x7fe6240d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:38.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.621+0000 7fe62c800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe624161420 con 0x7fe6240d2370 2024-04-02T07:36:38.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.621+0000 7fe62c800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6240d2820 con 0x7fe6240d16a0 2024-04-02T07:36:38.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.621+0000 7fe62c800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe624161700 con 0x7fe62415d9d0 2024-04-02T07:36:38.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.621+0000 7fe629400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6240d16a0 0x7fe6240d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:38.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.621+0000 7fe629400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6240d16a0 0x7fe6240d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40870/0 (socket says 172.21.15.89:40870) 2024-04-02T07:36:38.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.621+0000 7fe629400700 1 -- 172.21.15.89:0/3789924625 learned_addr learned my addr 172.21.15.89:0/3789924625 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:38.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.621+0000 7fe629e00700 1 --2- 172.21.15.89:0/3789924625 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe62415d9d0 0x7fe6240d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:38.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.621+0000 7fe629400700 1 -- 172.21.15.89:0/3789924625 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe62415d9d0 msgr2=0x7fe6240d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:38.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.621+0000 7fe629400700 1 --2- 172.21.15.89:0/3789924625 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe62415d9d0 0x7fe6240d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.621+0000 7fe629400700 1 -- 172.21.15.89:0/3789924625 >> v1:172.21.15.113:6789/0 conn(0x7fe6240d2370 legacy=0x7fe62415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:38.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.621+0000 7fe629400700 1 -- 172.21.15.89:0/3789924625 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6241619e0 con 0x7fe6240d16a0 2024-04-02T07:36:38.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.622+0000 7fe629400700 1 --2- 172.21.15.89:0/3789924625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6240d16a0 0x7fe6240d1e30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe6140098e0 tx=0x7fe614009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16b122c13438e7d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:38.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.622+0000 7fe628a00700 1 -- 172.21.15.89:0/3789924625 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe614011830 con 0x7fe6240d16a0 2024-04-02T07:36:38.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.622+0000 7fe628a00700 1 -- 172.21.15.89:0/3789924625 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe614011990 con 0x7fe6240d16a0 2024-04-02T07:36:38.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.622+0000 7fe628a00700 1 -- 172.21.15.89:0/3789924625 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe614011d40 con 0x7fe6240d16a0 2024-04-02T07:36:38.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.622+0000 7fe62c800700 1 -- 172.21.15.89:0/3789924625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6240d16a0 msgr2=0x7fe6240d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:38.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.622+0000 7fe62c800700 1 --2- 172.21.15.89:0/3789924625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6240d16a0 0x7fe6240d1e30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe6140098e0 tx=0x7fe614009bf0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.622+0000 7fe62c800700 1 -- 172.21.15.89:0/3789924625 shutdown_connections 2024-04-02T07:36:38.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.622+0000 7fe62c800700 1 --2- 172.21.15.89:0/3789924625 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe62415d9d0 0x7fe6240d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.622+0000 7fe62c800700 1 --2- 172.21.15.89:0/3789924625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6240d16a0 0x7fe6240d1e30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.622+0000 7fe62c800700 1 -- 172.21.15.89:0/3789924625 >> 172.21.15.89:0/3789924625 conn(0x7fe62408aff0 msgr2=0x7fe6240d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:38.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.622+0000 7fe62c800700 1 -- 172.21.15.89:0/3789924625 shutdown_connections 2024-04-02T07:36:38.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.623+0000 7fe62c800700 1 -- 172.21.15.89:0/3789924625 wait complete. 2024-04-02T07:36:38.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.624+0000 7fe62c800700 1 Processor -- start 2024-04-02T07:36:38.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.624+0000 7fe62c800700 1 -- start start 2024-04-02T07:36:38.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.625+0000 7fe62c800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6240d16a0 0x7fe624155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:38.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.625+0000 7fe62c800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6240d2370 0x7fe6240cc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:38.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.625+0000 7fe62c800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe62415d9d0 0x7fe624160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:38.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.625+0000 7fe62c800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6240cb1a0 con 0x7fe6240d16a0 2024-04-02T07:36:38.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.625+0000 7fe62c800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6240cb480 con 0x7fe6240d2370 2024-04-02T07:36:38.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.625+0000 7fe62c800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6240cb760 con 0x7fe62415d9d0 2024-04-02T07:36:38.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.625+0000 7fe629e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6240d16a0 0x7fe624155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:38.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.625+0000 7fe629e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6240d16a0 0x7fe624155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40878/0 (socket says 172.21.15.89:40878) 2024-04-02T07:36:38.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.625+0000 7fe629e00700 1 -- 172.21.15.89:0/217324311 learned_addr learned my addr 172.21.15.89:0/217324311 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:38.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.626+0000 7fe629400700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6240d2370 0x7fe6240cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:38.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.626+0000 7fe62a800700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe62415d9d0 0x7fe624160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:38.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.626+0000 7fe629e00700 1 -- 172.21.15.89:0/217324311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe62415d9d0 msgr2=0x7fe624160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:38.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.626+0000 7fe629e00700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe62415d9d0 0x7fe624160410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.626+0000 7fe629e00700 1 -- 172.21.15.89:0/217324311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6240d2370 msgr2=0x7fe6240cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:38.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.626+0000 7fe629e00700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6240d2370 0x7fe6240cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.626+0000 7fe629e00700 1 -- 172.21.15.89:0/217324311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6140094f0 con 0x7fe6240d16a0 2024-04-02T07:36:38.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.626+0000 7fe629400700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6240d2370 0x7fe6240cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:38.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.626+0000 7fe629e00700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6240d16a0 0x7fe624155840 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe61800e870 tx=0x7fe61800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:38.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.626+0000 7fe61e000700 1 -- 172.21.15.89:0/217324311 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe618009590 con 0x7fe6240d16a0 2024-04-02T07:36:38.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.626+0000 7fe61e000700 1 -- 172.21.15.89:0/217324311 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61800ee70 con 0x7fe6240d16a0 2024-04-02T07:36:38.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.626+0000 7fe62c800700 1 -- 172.21.15.89:0/217324311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6240cba40 con 0x7fe6240d16a0 2024-04-02T07:36:38.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.626+0000 7fe61e000700 1 -- 172.21.15.89:0/217324311 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61800ca50 con 0x7fe6240d16a0 2024-04-02T07:36:38.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.627+0000 7fe62c800700 1 -- 172.21.15.89:0/217324311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6240cc780 con 0x7fe6240d16a0 2024-04-02T07:36:38.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.627+0000 7fe62c800700 1 -- 172.21.15.89:0/217324311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6240cc090 con 0x7fe6240d16a0 2024-04-02T07:36:38.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.628+0000 7fe61e000700 1 -- 172.21.15.89:0/217324311 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe61800cbb0 con 0x7fe6240d16a0 2024-04-02T07:36:38.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.629+0000 7fe61e000700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe60806ac90 0x7fe60806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:38.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.629+0000 7fe61e000700 1 -- 172.21.15.89:0/217324311 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe6180893a0 con 0x7fe6240d16a0 2024-04-02T07:36:38.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.630+0000 7fe629400700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe60806ac90 0x7fe60806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:38.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.630+0000 7fe629400700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe60806ac90 0x7fe60806d150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe6240d1ae0 tx=0x7fe614008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:38.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.634+0000 7fe61e000700 1 -- 172.21.15.89:0/217324311 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6180544c0 con 0x7fe6240d16a0 2024-04-02T07:36:38.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.834+0000 7fe62c800700 1 -- 172.21.15.89:0/217324311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe6240d1ca0 con 0x7fe6240d16a0 2024-04-02T07:36:38.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.834+0000 7fe61e000700 1 -- 172.21.15.89:0/217324311 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe61800ce60 con 0x7fe6240d16a0 2024-04-02T07:36:38.836 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:36:38.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.836+0000 7fe62c800700 1 -- 172.21.15.89:0/217324311 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe60806ac90 msgr2=0x7fe60806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:38.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.836+0000 7fe62c800700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe60806ac90 0x7fe60806d150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe6240d1ae0 tx=0x7fe614008030 comp rx=0 tx=0).stop 2024-04-02T07:36:38.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.837+0000 7fe62c800700 1 -- 172.21.15.89:0/217324311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6240d16a0 msgr2=0x7fe624155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:38.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.837+0000 7fe62c800700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6240d16a0 0x7fe624155840 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe61800e870 tx=0x7fe61800ec30 comp rx=0 tx=0).stop 2024-04-02T07:36:38.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.837+0000 7fe62c800700 1 -- 172.21.15.89:0/217324311 shutdown_connections 2024-04-02T07:36:38.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.837+0000 7fe62c800700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe60806ac90 0x7fe60806d150 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.837+0000 7fe62c800700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe62415d9d0 0x7fe624160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.837+0000 7fe62c800700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6240d16a0 0x7fe624155840 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.837+0000 7fe62c800700 1 --2- 172.21.15.89:0/217324311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6240d2370 0x7fe6240cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:38.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.837+0000 7fe62c800700 1 -- 172.21.15.89:0/217324311 >> 172.21.15.89:0/217324311 conn(0x7fe62408aff0 msgr2=0x7fe6240b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:38.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.837+0000 7fe62c800700 1 -- 172.21.15.89:0/217324311 shutdown_connections 2024-04-02T07:36:38.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:38.837+0000 7fe62c800700 1 -- 172.21.15.89:0/217324311 wait complete. 2024-04-02T07:36:38.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:36:39.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.048+0000 7f8a35000700 1 Processor -- start 2024-04-02T07:36:39.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.048+0000 7f8a35000700 1 -- start start 2024-04-02T07:36:39.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.049+0000 7f8a35000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a3015d9d0 0x7f8a300d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.049+0000 7f8a35000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8a300d16a0 0x7f8a300d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.049+0000 7f8a35000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a30161420 con 0x7f8a300d2370 2024-04-02T07:36:39.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.049+0000 7f8a35000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a300d2820 con 0x7f8a300d16a0 2024-04-02T07:36:39.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.049+0000 7f8a35000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a30161700 con 0x7f8a3015d9d0 2024-04-02T07:36:39.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.050+0000 7f8a2d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8a300d16a0 0x7f8a300d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.050+0000 7f8a2e800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8a300d2370 0x7f8a3015ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53060/0 (socket says 172.21.15.89:53060) 2024-04-02T07:36:39.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.050+0000 7f8a2e800700 1 -- 172.21.15.89:0/2012994628 learned_addr learned my addr 172.21.15.89:0/2012994628 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:39.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.050+0000 7f8a2de00700 1 --2- 172.21.15.89:0/2012994628 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a3015d9d0 0x7f8a300d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.050+0000 7f8a2d400700 1 -- 172.21.15.89:0/2012994628 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a3015d9d0 msgr2=0x7f8a300d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:39.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.050+0000 7f8a2d400700 1 --2- 172.21.15.89:0/2012994628 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a3015d9d0 0x7f8a300d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.050+0000 7f8a2d400700 1 -- 172.21.15.89:0/2012994628 >> v1:172.21.15.89:6789/0 conn(0x7f8a300d2370 legacy=0x7f8a3015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.050+0000 7f8a2d400700 1 -- 172.21.15.89:0/2012994628 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a301619e0 con 0x7f8a300d16a0 2024-04-02T07:36:39.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.050+0000 7f8a2ca00700 1 -- 172.21.15.89:0/2012994628 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419389819 0 0) 0x7f8a30161420 con 0x7f8a300d2370 2024-04-02T07:36:39.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.050+0000 7f8a2d400700 1 --2- 172.21.15.89:0/2012994628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a300d16a0 0x7f8a300d1e30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009820 tx=0x7f8a1c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59487b81e62aa90 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:39.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.050+0000 7f8a2ca00700 1 -- 172.21.15.89:0/2012994628 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c005e30 con 0x7f8a300d16a0 2024-04-02T07:36:39.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.051+0000 7f8a2ca00700 1 -- 172.21.15.89:0/2012994628 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c006030 con 0x7f8a300d16a0 2024-04-02T07:36:39.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.051+0000 7f8a2ca00700 1 -- 172.21.15.89:0/2012994628 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c0063e0 con 0x7f8a300d16a0 2024-04-02T07:36:39.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.051+0000 7f8a35000700 1 -- 172.21.15.89:0/2012994628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a300d16a0 msgr2=0x7f8a300d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.051+0000 7f8a35000700 1 --2- 172.21.15.89:0/2012994628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a300d16a0 0x7f8a300d1e30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009820 tx=0x7f8a1c009b30 comp rx=0 tx=0).stop 2024-04-02T07:36:39.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.051+0000 7f8a35000700 1 -- 172.21.15.89:0/2012994628 shutdown_connections 2024-04-02T07:36:39.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.051+0000 7f8a35000700 1 --2- 172.21.15.89:0/2012994628 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a3015d9d0 0x7f8a300d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.051+0000 7f8a35000700 1 --2- 172.21.15.89:0/2012994628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a300d16a0 0x7f8a300d1e30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.051+0000 7f8a35000700 1 -- 172.21.15.89:0/2012994628 >> 172.21.15.89:0/2012994628 conn(0x7f8a3008aff0 msgr2=0x7f8a300d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:39.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.051+0000 7f8a35000700 1 -- 172.21.15.89:0/2012994628 shutdown_connections 2024-04-02T07:36:39.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.051+0000 7f8a35000700 1 -- 172.21.15.89:0/2012994628 wait complete. 2024-04-02T07:36:39.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.053+0000 7f8a35000700 1 Processor -- start 2024-04-02T07:36:39.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.053+0000 7f8a35000700 1 -- start start 2024-04-02T07:36:39.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.054+0000 7f8a35000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a300d16a0 0x7f8a3015f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.054+0000 7f8a35000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a300d2370 0x7f8a301f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.054+0000 7f8a35000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a3015d9d0 0x7f8a301f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.054+0000 7f8a35000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a300b62c0 con 0x7f8a300d2370 2024-04-02T07:36:39.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.054+0000 7f8a35000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3015f680 con 0x7f8a3015d9d0 2024-04-02T07:36:39.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.054+0000 7f8a35000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a30160a10 con 0x7f8a300d16a0 2024-04-02T07:36:39.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.054+0000 7f8a2d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a300d2370 0x7f8a301f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.054+0000 7f8a2d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a300d2370 0x7f8a301f38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48672/0 (socket says 172.21.15.89:48672) 2024-04-02T07:36:39.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.054+0000 7f8a2d400700 1 -- 172.21.15.89:0/333187481 learned_addr learned my addr 172.21.15.89:0/333187481 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:39.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.054+0000 7f8a2de00700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a300d16a0 0x7f8a3015f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.055+0000 7f8a2e800700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a3015d9d0 0x7f8a301f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.055+0000 7f8a2d400700 1 -- 172.21.15.89:0/333187481 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a300d16a0 msgr2=0x7f8a3015f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.055+0000 7f8a2d400700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a300d16a0 0x7f8a3015f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.055+0000 7f8a2d400700 1 -- 172.21.15.89:0/333187481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a3015d9d0 msgr2=0x7f8a301f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.055+0000 7f8a2d400700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a3015d9d0 0x7f8a301f3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.055+0000 7f8a2d400700 1 -- 172.21.15.89:0/333187481 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c009430 con 0x7f8a300d2370 2024-04-02T07:36:39.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.055+0000 7f8a2e800700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a3015d9d0 0x7f8a301f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:39.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.055+0000 7f8a2de00700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a300d16a0 0x7f8a3015f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:39.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.055+0000 7f8a2d400700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a300d2370 0x7f8a301f38b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c013040 tx=0x7f8a1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:39.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.055+0000 7f8a26000700 1 -- 172.21.15.89:0/333187481 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c025430 con 0x7f8a300d2370 2024-04-02T07:36:39.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.055+0000 7f8a26000700 1 -- 172.21.15.89:0/333187481 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c020070 con 0x7f8a300d2370 2024-04-02T07:36:39.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.055+0000 7f8a35000700 1 -- 172.21.15.89:0/333187481 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a300b6510 con 0x7f8a300d2370 2024-04-02T07:36:39.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.055+0000 7f8a26000700 1 -- 172.21.15.89:0/333187481 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c0257d0 con 0x7f8a300d2370 2024-04-02T07:36:39.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.055+0000 7f8a35000700 1 -- 172.21.15.89:0/333187481 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a300b6ad0 con 0x7f8a300d2370 2024-04-02T07:36:39.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.056+0000 7f8a35000700 1 -- 172.21.15.89:0/333187481 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a300d1ca0 con 0x7f8a300d2370 2024-04-02T07:36:39.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.057+0000 7f8a26000700 1 -- 172.21.15.89:0/333187481 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8a1c022030 con 0x7f8a300d2370 2024-04-02T07:36:39.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.059+0000 7f8a26000700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a1006ac90 0x7f8a1006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.059+0000 7f8a26000700 1 -- 172.21.15.89:0/333187481 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8a1c08a940 con 0x7f8a300d2370 2024-04-02T07:36:39.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.059+0000 7f8a2de00700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a1006ac90 0x7f8a1006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.060+0000 7f8a2de00700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a1006ac90 0x7f8a1006d150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8a20007a50 tx=0x7f8a20008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:39.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.064+0000 7f8a26000700 1 -- 172.21.15.89:0/333187481 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8a1c055b90 con 0x7f8a300d2370 2024-04-02T07:36:39.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.266+0000 7f8a35000700 1 -- 172.21.15.89:0/333187481 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8a300b6cf0 con 0x7f8a300d2370 2024-04-02T07:36:39.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.267+0000 7f8a26000700 1 -- 172.21.15.89:0/333187481 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8a1c006c60 con 0x7f8a300d2370 2024-04-02T07:36:39.269 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:36:39.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.269+0000 7f8a35000700 1 -- 172.21.15.89:0/333187481 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a1006ac90 msgr2=0x7f8a1006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.269+0000 7f8a35000700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a1006ac90 0x7f8a1006d150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8a20007a50 tx=0x7f8a20008040 comp rx=0 tx=0).stop 2024-04-02T07:36:39.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.269+0000 7f8a35000700 1 -- 172.21.15.89:0/333187481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a300d2370 msgr2=0x7f8a301f38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.269+0000 7f8a35000700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a300d2370 0x7f8a301f38b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c013040 tx=0x7f8a1c004750 comp rx=0 tx=0).stop 2024-04-02T07:36:39.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.269+0000 7f8a35000700 1 -- 172.21.15.89:0/333187481 shutdown_connections 2024-04-02T07:36:39.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.269+0000 7f8a35000700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a1006ac90 0x7f8a1006d150 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.269+0000 7f8a35000700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a3015d9d0 0x7f8a301f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.269+0000 7f8a35000700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a300d16a0 0x7f8a3015f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.269+0000 7f8a35000700 1 --2- 172.21.15.89:0/333187481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a300d2370 0x7f8a301f38b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.269+0000 7f8a35000700 1 -- 172.21.15.89:0/333187481 >> 172.21.15.89:0/333187481 conn(0x7f8a3008aff0 msgr2=0x7f8a3015c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:39.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.269+0000 7f8a35000700 1 -- 172.21.15.89:0/333187481 shutdown_connections 2024-04-02T07:36:39.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.270+0000 7f8a35000700 1 -- 172.21.15.89:0/333187481 wait complete. 2024-04-02T07:36:39.289 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:36:39.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.481+0000 7ff252800700 1 Processor -- start 2024-04-02T07:36:39.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.481+0000 7ff252800700 1 -- start start 2024-04-02T07:36:39.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.482+0000 7ff252800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff24c0d16a0 0x7ff24c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.482+0000 7ff252800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff24c0d2370 0x7ff24c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.482+0000 7ff252800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c161420 con 0x7ff24c15d9d0 2024-04-02T07:36:39.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.482+0000 7ff252800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff24c0d2820 con 0x7ff24c0d16a0 2024-04-02T07:36:39.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.482+0000 7ff252800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c0d4340 con 0x7ff24c0d2370 2024-04-02T07:36:39.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.482+0000 7ff24aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff24c0d16a0 0x7ff24c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.482+0000 7ff24b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff24c15d9d0 0x7ff24c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53076/0 (socket says 172.21.15.89:53076) 2024-04-02T07:36:39.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.482+0000 7ff24b400700 1 -- 172.21.15.89:0/4245919226 learned_addr learned my addr 172.21.15.89:0/4245919226 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:39.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.483+0000 7ff24be00700 1 --2- 172.21.15.89:0/4245919226 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff24c0d2370 0x7ff24c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.483+0000 7ff24aa00700 1 -- 172.21.15.89:0/4245919226 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff24c0d2370 msgr2=0x7ff24c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:39.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.483+0000 7ff24aa00700 1 --2- 172.21.15.89:0/4245919226 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff24c0d2370 0x7ff24c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.483+0000 7ff24aa00700 1 -- 172.21.15.89:0/4245919226 >> v1:172.21.15.89:6789/0 conn(0x7ff24c15d9d0 legacy=0x7ff24c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.483+0000 7ff24aa00700 1 -- 172.21.15.89:0/4245919226 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c161700 con 0x7ff24c0d16a0 2024-04-02T07:36:39.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.483+0000 7ff24aa00700 1 --2- 172.21.15.89:0/4245919226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24c0d16a0 0x7ff24c0d1e30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff238009830 tx=0x7ff238009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149fbdaeb43db865 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:39.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.483+0000 7ff24a000700 1 -- 172.21.15.89:0/4245919226 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff238004230 con 0x7ff24c0d16a0 2024-04-02T07:36:39.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.483+0000 7ff24a000700 1 -- 172.21.15.89:0/4245919226 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff238004390 con 0x7ff24c0d16a0 2024-04-02T07:36:39.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.483+0000 7ff24a000700 1 -- 172.21.15.89:0/4245919226 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2380036a0 con 0x7ff24c0d16a0 2024-04-02T07:36:39.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.484+0000 7ff252800700 1 -- 172.21.15.89:0/4245919226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24c0d16a0 msgr2=0x7ff24c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.484+0000 7ff252800700 1 --2- 172.21.15.89:0/4245919226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24c0d16a0 0x7ff24c0d1e30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff238009830 tx=0x7ff238009b40 comp rx=0 tx=0).stop 2024-04-02T07:36:39.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.484+0000 7ff252800700 1 -- 172.21.15.89:0/4245919226 shutdown_connections 2024-04-02T07:36:39.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.484+0000 7ff252800700 1 --2- 172.21.15.89:0/4245919226 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff24c0d2370 0x7ff24c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.484+0000 7ff252800700 1 --2- 172.21.15.89:0/4245919226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24c0d16a0 0x7ff24c0d1e30 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.484+0000 7ff252800700 1 -- 172.21.15.89:0/4245919226 >> 172.21.15.89:0/4245919226 conn(0x7ff24c08aff0 msgr2=0x7ff24c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:39.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.484+0000 7ff252800700 1 -- 172.21.15.89:0/4245919226 shutdown_connections 2024-04-02T07:36:39.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.484+0000 7ff252800700 1 -- 172.21.15.89:0/4245919226 wait complete. 2024-04-02T07:36:39.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.486+0000 7ff252800700 1 Processor -- start 2024-04-02T07:36:39.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.486+0000 7ff252800700 1 -- start start 2024-04-02T07:36:39.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff252800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24c0d16a0 0x7ff24c0b8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff252800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff24c0d2370 0x7ff24c0b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff252800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff24c15d9d0 0x7ff24c1e82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff252800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c0cb050 con 0x7ff24c0d16a0 2024-04-02T07:36:39.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff252800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c0b8a10 con 0x7ff24c15d9d0 2024-04-02T07:36:39.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff252800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c0ca720 con 0x7ff24c0d2370 2024-04-02T07:36:39.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff24b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24c0d16a0 0x7ff24c0b8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff24b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24c0d16a0 0x7ff24c0b8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48686/0 (socket says 172.21.15.89:48686) 2024-04-02T07:36:39.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff24b400700 1 -- 172.21.15.89:0/1473454999 learned_addr learned my addr 172.21.15.89:0/1473454999 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:39.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff24aa00700 1 --2- 172.21.15.89:0/1473454999 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff24c0d2370 0x7ff24c0b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff24be00700 1 --2- 172.21.15.89:0/1473454999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff24c15d9d0 0x7ff24c1e82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff24b400700 1 -- 172.21.15.89:0/1473454999 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff24c0d2370 msgr2=0x7ff24c0b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff24b400700 1 --2- 172.21.15.89:0/1473454999 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff24c0d2370 0x7ff24c0b4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff24b400700 1 -- 172.21.15.89:0/1473454999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff24c15d9d0 msgr2=0x7ff24c1e82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff24b400700 1 --2- 172.21.15.89:0/1473454999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff24c15d9d0 0x7ff24c1e82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.487+0000 7ff24b400700 1 -- 172.21.15.89:0/1473454999 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff238009440 con 0x7ff24c0d16a0 2024-04-02T07:36:39.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.488+0000 7ff24b400700 1 --2- 172.21.15.89:0/1473454999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24c0d16a0 0x7ff24c0b8110 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00c780 tx=0x7ff23c00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:39.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.488+0000 7ff243400700 1 -- 172.21.15.89:0/1473454999 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c004c80 con 0x7ff24c0d16a0 2024-04-02T07:36:39.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.488+0000 7ff243400700 1 -- 172.21.15.89:0/1473454999 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23c004de0 con 0x7ff24c0d16a0 2024-04-02T07:36:39.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.488+0000 7ff252800700 1 -- 172.21.15.89:0/1473454999 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24c0caa00 con 0x7ff24c0d16a0 2024-04-02T07:36:39.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.488+0000 7ff243400700 1 -- 172.21.15.89:0/1473454999 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c007310 con 0x7ff24c0d16a0 2024-04-02T07:36:39.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.488+0000 7ff252800700 1 -- 172.21.15.89:0/1473454999 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24c0b8650 con 0x7ff24c0d16a0 2024-04-02T07:36:39.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.489+0000 7ff252800700 1 -- 172.21.15.89:0/1473454999 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff24c0cb330 con 0x7ff24c0d16a0 2024-04-02T07:36:39.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.490+0000 7ff243400700 1 -- 172.21.15.89:0/1473454999 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff23c007470 con 0x7ff24c0d16a0 2024-04-02T07:36:39.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.491+0000 7ff243400700 1 --2- 172.21.15.89:0/1473454999 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff22c06ac90 0x7ff22c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.491+0000 7ff243400700 1 -- 172.21.15.89:0/1473454999 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff23c089120 con 0x7ff24c0d16a0 2024-04-02T07:36:39.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.491+0000 7ff24aa00700 1 --2- 172.21.15.89:0/1473454999 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff22c06ac90 0x7ff22c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.491+0000 7ff24aa00700 1 --2- 172.21.15.89:0/1473454999 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff22c06ac90 0x7ff22c06d150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0d1ae0 tx=0x7ff238008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:39.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.495+0000 7ff243400700 1 -- 172.21.15.89:0/1473454999 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff23c0542d0 con 0x7ff24c0d16a0 2024-04-02T07:36:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.696+0000 7ff252800700 1 -- 172.21.15.89:0/1473454999 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff24c0d1ca0 con 0x7ff24c0d16a0 2024-04-02T07:36:39.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.697+0000 7ff243400700 1 -- 172.21.15.89:0/1473454999 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff23c007720 con 0x7ff24c0d16a0 2024-04-02T07:36:39.699 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:36:39.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.699+0000 7ff252800700 1 -- 172.21.15.89:0/1473454999 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff22c06ac90 msgr2=0x7ff22c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.699+0000 7ff252800700 1 --2- 172.21.15.89:0/1473454999 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff22c06ac90 0x7ff22c06d150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0d1ae0 tx=0x7ff238008030 comp rx=0 tx=0).stop 2024-04-02T07:36:39.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.699+0000 7ff252800700 1 -- 172.21.15.89:0/1473454999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24c0d16a0 msgr2=0x7ff24c0b8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.699+0000 7ff252800700 1 --2- 172.21.15.89:0/1473454999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24c0d16a0 0x7ff24c0b8110 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00c780 tx=0x7ff23c00ca90 comp rx=0 tx=0).stop 2024-04-02T07:36:39.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.699+0000 7ff252800700 1 -- 172.21.15.89:0/1473454999 shutdown_connections 2024-04-02T07:36:39.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.699+0000 7ff252800700 1 --2- 172.21.15.89:0/1473454999 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff22c06ac90 0x7ff22c06d150 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.699+0000 7ff252800700 1 --2- 172.21.15.89:0/1473454999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff24c15d9d0 0x7ff24c1e82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.699+0000 7ff252800700 1 --2- 172.21.15.89:0/1473454999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24c0d16a0 0x7ff24c0b8110 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.699+0000 7ff252800700 1 --2- 172.21.15.89:0/1473454999 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff24c0d2370 0x7ff24c0b4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.699+0000 7ff252800700 1 -- 172.21.15.89:0/1473454999 >> 172.21.15.89:0/1473454999 conn(0x7ff24c08aff0 msgr2=0x7ff24c0b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:39.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.699+0000 7ff252800700 1 -- 172.21.15.89:0/1473454999 shutdown_connections 2024-04-02T07:36:39.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.699+0000 7ff252800700 1 -- 172.21.15.89:0/1473454999 wait complete. 2024-04-02T07:36:39.719 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T07:36:39.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.910+0000 7fb758c00700 1 Processor -- start 2024-04-02T07:36:39.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.911+0000 7fb758c00700 1 -- start start 2024-04-02T07:36:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.912+0000 7fb758c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb75415d9d0 0x7fb7540d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.912+0000 7fb758c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb7540d16a0 0x7fb7540d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.912+0000 7fb758c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb754161420 con 0x7fb7540d2370 2024-04-02T07:36:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.912+0000 7fb758c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7540d27e0 con 0x7fb75415d9d0 2024-04-02T07:36:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.912+0000 7fb758c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb754161700 con 0x7fb7540d16a0 2024-04-02T07:36:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.912+0000 7fb751e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb75415d9d0 0x7fb7540d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.912+0000 7fb752800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb7540d2370 0x7fb75415ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53092/0 (socket says 172.21.15.89:53092) 2024-04-02T07:36:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.912+0000 7fb752800700 1 -- 172.21.15.89:0/2276521046 learned_addr learned my addr 172.21.15.89:0/2276521046 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.912+0000 7fb751400700 1 --2- 172.21.15.89:0/2276521046 >> v2:172.21.15.113:3300/0 conn(0x7fb7540d16a0 0x7fb7540d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.912+0000 7fb751e00700 1 -- 172.21.15.89:0/2276521046 >> v2:172.21.15.113:3300/0 conn(0x7fb7540d16a0 msgr2=0x7fb7540d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.912+0000 7fb751e00700 1 --2- 172.21.15.89:0/2276521046 >> v2:172.21.15.113:3300/0 conn(0x7fb7540d16a0 0x7fb7540d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.912+0000 7fb751e00700 1 -- 172.21.15.89:0/2276521046 >> v1:172.21.15.89:6789/0 conn(0x7fb7540d2370 legacy=0x7fb75415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.913+0000 7fb751e00700 1 -- 172.21.15.89:0/2276521046 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7541619e0 con 0x7fb75415d9d0 2024-04-02T07:36:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.912+0000 7fb750a00700 1 -- 172.21.15.89:0/2276521046 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384228098 0 0) 0x7fb754161420 con 0x7fb7540d2370 2024-04-02T07:36:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.913+0000 7fb751e00700 1 --2- 172.21.15.89:0/2276521046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb75415d9d0 0x7fb7540d3c30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb7540d12f0 tx=0x7fb744009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5046add4df639783 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.913+0000 7fb750a00700 1 -- 172.21.15.89:0/2276521046 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb744011830 con 0x7fb75415d9d0 2024-04-02T07:36:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.913+0000 7fb750a00700 1 -- 172.21.15.89:0/2276521046 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb744011990 con 0x7fb75415d9d0 2024-04-02T07:36:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.913+0000 7fb750a00700 1 -- 172.21.15.89:0/2276521046 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb744011d20 con 0x7fb75415d9d0 2024-04-02T07:36:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.913+0000 7fb758c00700 1 -- 172.21.15.89:0/2276521046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb75415d9d0 msgr2=0x7fb7540d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.913+0000 7fb758c00700 1 --2- 172.21.15.89:0/2276521046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb75415d9d0 0x7fb7540d3c30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb7540d12f0 tx=0x7fb744009b20 comp rx=0 tx=0).stop 2024-04-02T07:36:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.914+0000 7fb758c00700 1 -- 172.21.15.89:0/2276521046 shutdown_connections 2024-04-02T07:36:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.914+0000 7fb758c00700 1 --2- 172.21.15.89:0/2276521046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb75415d9d0 0x7fb7540d3c30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.914+0000 7fb758c00700 1 --2- 172.21.15.89:0/2276521046 >> v2:172.21.15.113:3300/0 conn(0x7fb7540d16a0 0x7fb7540d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.914+0000 7fb758c00700 1 -- 172.21.15.89:0/2276521046 >> 172.21.15.89:0/2276521046 conn(0x7fb75408aff0 msgr2=0x7fb7540d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.914+0000 7fb758c00700 1 -- 172.21.15.89:0/2276521046 shutdown_connections 2024-04-02T07:36:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.914+0000 7fb758c00700 1 -- 172.21.15.89:0/2276521046 wait complete. 2024-04-02T07:36:39.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.916+0000 7fb758c00700 1 Processor -- start 2024-04-02T07:36:39.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.916+0000 7fb758c00700 1 -- start start 2024-04-02T07:36:39.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.917+0000 7fb758c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7540d16a0 0x7fb75415a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.917+0000 7fb758c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7540d2370 0x7fb7540cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.917+0000 7fb758c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb75415d9d0 0x7fb7540a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.917+0000 7fb758c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75415f800 con 0x7fb75415d9d0 2024-04-02T07:36:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.917+0000 7fb758c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7540cdab0 con 0x7fb7540d16a0 2024-04-02T07:36:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.917+0000 7fb758c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7540cd4f0 con 0x7fb7540d2370 2024-04-02T07:36:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb752800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb75415d9d0 0x7fb7540a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb752800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb75415d9d0 0x7fb7540a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48708/0 (socket says 172.21.15.89:48708) 2024-04-02T07:36:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb752800700 1 -- 172.21.15.89:0/3269645501 learned_addr learned my addr 172.21.15.89:0/3269645501 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb751e00700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7540d16a0 0x7fb75415a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb751400700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7540d2370 0x7fb7540cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb752800700 1 -- 172.21.15.89:0/3269645501 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7540d2370 msgr2=0x7fb7540cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb752800700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7540d2370 0x7fb7540cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb752800700 1 -- 172.21.15.89:0/3269645501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7540d16a0 msgr2=0x7fb75415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb752800700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7540d16a0 0x7fb75415a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb752800700 1 -- 172.21.15.89:0/3269645501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7440094a0 con 0x7fb75415d9d0 2024-04-02T07:36:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb751e00700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7540d16a0 0x7fb75415a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:39.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb751400700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7540d2370 0x7fb7540cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:39.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb752800700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb75415d9d0 0x7fb7540a0870 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb74c002230 tx=0x7fb74c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:39.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb74a000700 1 -- 172.21.15.89:0/3269645501 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c00ac20 con 0x7fb75415d9d0 2024-04-02T07:36:39.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb74a000700 1 -- 172.21.15.89:0/3269645501 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74c00ada0 con 0x7fb75415d9d0 2024-04-02T07:36:39.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.918+0000 7fb758c00700 1 -- 172.21.15.89:0/3269645501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7540cd7d0 con 0x7fb75415d9d0 2024-04-02T07:36:39.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.919+0000 7fb74a000700 1 -- 172.21.15.89:0/3269645501 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c007310 con 0x7fb75415d9d0 2024-04-02T07:36:39.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.919+0000 7fb758c00700 1 -- 172.21.15.89:0/3269645501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75415a8f0 con 0x7fb75415d9d0 2024-04-02T07:36:39.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.919+0000 7fb758c00700 1 -- 172.21.15.89:0/3269645501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7540d1ca0 con 0x7fb75415d9d0 2024-04-02T07:36:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.920+0000 7fb74a000700 1 -- 172.21.15.89:0/3269645501 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb74c007470 con 0x7fb75415d9d0 2024-04-02T07:36:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.922+0000 7fb74a000700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb73406ac90 0x7fb73406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:39.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.922+0000 7fb74a000700 1 -- 172.21.15.89:0/3269645501 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb74c089220 con 0x7fb75415d9d0 2024-04-02T07:36:39.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.922+0000 7fb751e00700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb73406ac90 0x7fb73406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:39.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.922+0000 7fb751e00700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb73406ac90 0x7fb73406d150 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb7440090b0 tx=0x7fb744008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:39.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:39.926+0000 7fb74a000700 1 -- 172.21.15.89:0/3269645501 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb74c0543d0 con 0x7fb75415d9d0 2024-04-02T07:36:40.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.142+0000 7fb758c00700 1 -- 172.21.15.89:0/3269645501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb75415f280 con 0x7fb75415d9d0 2024-04-02T07:36:40.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.142+0000 7fb74a000700 1 -- 172.21.15.89:0/3269645501 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb75415f280 con 0x7fb75415d9d0 2024-04-02T07:36:40.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.145+0000 7fb758c00700 1 -- 172.21.15.89:0/3269645501 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb73406ac90 msgr2=0x7fb73406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:40.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.145+0000 7fb758c00700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb73406ac90 0x7fb73406d150 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb7440090b0 tx=0x7fb744008030 comp rx=0 tx=0).stop 2024-04-02T07:36:40.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.145+0000 7fb758c00700 1 -- 172.21.15.89:0/3269645501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb75415d9d0 msgr2=0x7fb7540a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:40.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.145+0000 7fb758c00700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb75415d9d0 0x7fb7540a0870 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb74c002230 tx=0x7fb74c00a060 comp rx=0 tx=0).stop 2024-04-02T07:36:40.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.145+0000 7fb758c00700 1 -- 172.21.15.89:0/3269645501 shutdown_connections 2024-04-02T07:36:40.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.145+0000 7fb758c00700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb73406ac90 0x7fb73406d150 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:40.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.145+0000 7fb758c00700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7540d16a0 0x7fb75415a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:40.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.145+0000 7fb758c00700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb75415d9d0 0x7fb7540a0870 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:40.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.145+0000 7fb758c00700 1 --2- 172.21.15.89:0/3269645501 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7540d2370 0x7fb7540cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:40.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.145+0000 7fb758c00700 1 -- 172.21.15.89:0/3269645501 >> 172.21.15.89:0/3269645501 conn(0x7fb75408aff0 msgr2=0x7fb7540b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:40.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.145+0000 7fb758c00700 1 -- 172.21.15.89:0/3269645501 shutdown_connections 2024-04-02T07:36:40.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.145+0000 7fb758c00700 1 -- 172.21.15.89:0/3269645501 wait complete. 2024-04-02T07:36:40.148 INFO:teuthology.orchestra.run.smithi089.stderr:listed 0 entries 2024-04-02T07:36:40.167 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-04-02T07:36:40.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:40.168 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-04-02T07:36:40.237 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:36:40.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:40.238 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-02T07:36:40.282 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:36:40.283 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:40.333 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:36:40.333 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:40.379 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:36:40.380 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:36:40.466 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:36:40.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.654+0000 7f0c9b000700 1 Processor -- start 2024-04-02T07:36:40.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.655+0000 7f0c9b000700 1 -- start start 2024-04-02T07:36:40.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.656+0000 7f0c9b000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0c940cb3f0 0x7f0c940ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:40.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.656+0000 7f0c9b000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0c940ac600 0x7f0c9406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:40.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.656+0000 7f0c9b000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9415fd50 con 0x7f0c9415b5d0 2024-04-02T07:36:40.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.656+0000 7f0c9b000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c9415b9b0 con 0x7f0c940cb3f0 2024-04-02T07:36:40.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.656+0000 7f0c9b000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c940aca10 con 0x7f0c940ac600 2024-04-02T07:36:40.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.656+0000 7f0c93400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0c940cb3f0 0x7f0c940ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:40.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.656+0000 7f0c93400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0c940cb3f0 0x7f0c940ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48722/0 (socket says 172.21.15.89:48722) 2024-04-02T07:36:40.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.656+0000 7f0c93400700 1 -- 172.21.15.89:0/3048093392 learned_addr learned my addr 172.21.15.89:0/3048093392 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:40.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.656+0000 7f0c99000700 1 --2- 172.21.15.89:0/3048093392 >> v2:172.21.15.113:3300/0 conn(0x7f0c940ac600 0x7f0c9406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:40.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.656+0000 7f0c93400700 1 -- 172.21.15.89:0/3048093392 >> v2:172.21.15.113:3300/0 conn(0x7f0c940ac600 msgr2=0x7f0c9406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:40.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.656+0000 7f0c93400700 1 --2- 172.21.15.89:0/3048093392 >> v2:172.21.15.113:3300/0 conn(0x7f0c940ac600 0x7f0c9406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:40.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.656+0000 7f0c93400700 1 -- 172.21.15.89:0/3048093392 >> v1:172.21.15.113:6789/0 conn(0x7f0c9415b5d0 legacy=0x7f0c940ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:40.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.656+0000 7f0c93400700 1 -- 172.21.15.89:0/3048093392 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9415f140 con 0x7f0c940cb3f0 2024-04-02T07:36:40.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.657+0000 7f0c93400700 1 --2- 172.21.15.89:0/3048093392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c940cb3f0 0x7f0c940ac0c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0c940cfdc0 tx=0x7f0c80009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aefee1c6033855ed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:40.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.657+0000 7f0c92a00700 1 -- 172.21.15.89:0/3048093392 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c80011830 con 0x7f0c940cb3f0 2024-04-02T07:36:40.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.657+0000 7f0c92a00700 1 -- 172.21.15.89:0/3048093392 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c80011990 con 0x7f0c940cb3f0 2024-04-02T07:36:40.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.657+0000 7f0c92a00700 1 -- 172.21.15.89:0/3048093392 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c80011d20 con 0x7f0c940cb3f0 2024-04-02T07:36:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.657+0000 7f0c9b000700 1 -- 172.21.15.89:0/3048093392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c940cb3f0 msgr2=0x7f0c940ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.657+0000 7f0c9b000700 1 --2- 172.21.15.89:0/3048093392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c940cb3f0 0x7f0c940ac0c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0c940cfdc0 tx=0x7f0c80009bd0 comp rx=0 tx=0).stop 2024-04-02T07:36:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.658+0000 7f0c9b000700 1 -- 172.21.15.89:0/3048093392 shutdown_connections 2024-04-02T07:36:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.658+0000 7f0c9b000700 1 --2- 172.21.15.89:0/3048093392 >> v2:172.21.15.113:3300/0 conn(0x7f0c940ac600 0x7f0c9406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.658+0000 7f0c9b000700 1 --2- 172.21.15.89:0/3048093392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c940cb3f0 0x7f0c940ac0c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.658+0000 7f0c9b000700 1 -- 172.21.15.89:0/3048093392 >> 172.21.15.89:0/3048093392 conn(0x7f0c940cf190 msgr2=0x7f0c940cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.658+0000 7f0c9b000700 1 -- 172.21.15.89:0/3048093392 shutdown_connections 2024-04-02T07:36:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.658+0000 7f0c9b000700 1 -- 172.21.15.89:0/3048093392 wait complete. 2024-04-02T07:36:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.659+0000 7f0c9b000700 1 Processor -- start 2024-04-02T07:36:40.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.659+0000 7f0c9b000700 1 -- start start 2024-04-02T07:36:40.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.660+0000 7f0c9b000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c940ac600 0x7f0c941f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:40.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.660+0000 7f0c9b000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c9415b5d0 0x7f0c940c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:40.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.660+0000 7f0c9b000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c940ca3e0 0x7f0c94070a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:40.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.660+0000 7f0c9b000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c940ca890 con 0x7f0c9415b5d0 2024-04-02T07:36:40.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.660+0000 7f0c9b000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c940723b0 con 0x7f0c940ca3e0 2024-04-02T07:36:40.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.660+0000 7f0c9b000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c941f2ff0 con 0x7f0c940ac600 2024-04-02T07:36:40.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.661+0000 7f0c93400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c9415b5d0 0x7f0c940c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:40.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.661+0000 7f0c93400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c9415b5d0 0x7f0c940c9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48726/0 (socket says 172.21.15.89:48726) 2024-04-02T07:36:40.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.661+0000 7f0c93400700 1 -- 172.21.15.89:0/496182911 learned_addr learned my addr 172.21.15.89:0/496182911 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:40.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.661+0000 7f0c93e00700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c940ac600 0x7f0c941f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:40.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.661+0000 7f0c99000700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c940ca3e0 0x7f0c94070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:40.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.661+0000 7f0c93400700 1 -- 172.21.15.89:0/496182911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c940ac600 msgr2=0x7f0c941f2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:40.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.661+0000 7f0c93400700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c940ac600 0x7f0c941f2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:40.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.661+0000 7f0c93400700 1 -- 172.21.15.89:0/496182911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c940ca3e0 msgr2=0x7f0c94070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:40.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.661+0000 7f0c93400700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c940ca3e0 0x7f0c94070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:40.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.661+0000 7f0c93400700 1 -- 172.21.15.89:0/496182911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c80009550 con 0x7f0c9415b5d0 2024-04-02T07:36:40.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.661+0000 7f0c99000700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c940ca3e0 0x7f0c94070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:40.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.661+0000 7f0c93e00700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c940ac600 0x7f0c941f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:40.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.661+0000 7f0c93400700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c9415b5d0 0x7f0c940c9ea0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0c80000c00 tx=0x7f0c80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:40.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.662+0000 7f0c8be00700 1 -- 172.21.15.89:0/496182911 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c80004570 con 0x7f0c9415b5d0 2024-04-02T07:36:40.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.662+0000 7f0c8be00700 1 -- 172.21.15.89:0/496182911 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8001f070 con 0x7f0c9415b5d0 2024-04-02T07:36:40.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.662+0000 7f0c8be00700 1 -- 172.21.15.89:0/496182911 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c800038e0 con 0x7f0c9415b5d0 2024-04-02T07:36:40.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.662+0000 7f0c9b000700 1 -- 172.21.15.89:0/496182911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c940717a0 con 0x7f0c9415b5d0 2024-04-02T07:36:40.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.662+0000 7f0c9b000700 1 -- 172.21.15.89:0/496182911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c94071d60 con 0x7f0c9415b5d0 2024-04-02T07:36:40.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.663+0000 7f0c89600700 1 -- 172.21.15.89:0/496182911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c940cba10 con 0x7f0c9415b5d0 2024-04-02T07:36:40.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.664+0000 7f0c8be00700 1 -- 172.21.15.89:0/496182911 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0c80021030 con 0x7f0c9415b5d0 2024-04-02T07:36:40.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.664+0000 7f0c8be00700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c7406ac90 0x7f0c7406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:40.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.665+0000 7f0c8be00700 1 -- 172.21.15.89:0/496182911 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0c8008a780 con 0x7f0c9415b5d0 2024-04-02T07:36:40.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.665+0000 7f0c93e00700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c7406ac90 0x7f0c7406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:40.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.665+0000 7f0c93e00700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c7406ac90 0x7f0c7406d150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0c84000f60 tx=0x7f0c84009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:40.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.669+0000 7f0c8be00700 1 -- 172.21.15.89:0/496182911 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0c80055af0 con 0x7f0c9415b5d0 2024-04-02T07:36:40.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.875+0000 7f0c89600700 1 -- 172.21.15.89:0/496182911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c94067720 con 0x7f0c9415b5d0 2024-04-02T07:36:40.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.877+0000 7f0c8be00700 1 -- 172.21.15.89:0/496182911 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7f0c80015090 con 0x7f0c9415b5d0 2024-04-02T07:36:40.879 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:40.879 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:36:36.860889+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:36:40.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.879+0000 7f0c89600700 1 -- 172.21.15.89:0/496182911 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c7406ac90 msgr2=0x7f0c7406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:40.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.879+0000 7f0c89600700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c7406ac90 0x7f0c7406d150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0c84000f60 tx=0x7f0c84009040 comp rx=0 tx=0).stop 2024-04-02T07:36:40.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.879+0000 7f0c89600700 1 -- 172.21.15.89:0/496182911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c9415b5d0 msgr2=0x7f0c940c9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:40.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.879+0000 7f0c89600700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c9415b5d0 0x7f0c940c9ea0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0c80000c00 tx=0x7f0c80004750 comp rx=0 tx=0).stop 2024-04-02T07:36:40.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.879+0000 7f0c89600700 1 -- 172.21.15.89:0/496182911 shutdown_connections 2024-04-02T07:36:40.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.879+0000 7f0c89600700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c7406ac90 0x7f0c7406d150 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:40.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.879+0000 7f0c89600700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c940ca3e0 0x7f0c94070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:40.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.879+0000 7f0c89600700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c940ac600 0x7f0c941f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:40.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.879+0000 7f0c89600700 1 --2- 172.21.15.89:0/496182911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c9415b5d0 0x7f0c940c9ea0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:40.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.879+0000 7f0c89600700 1 -- 172.21.15.89:0/496182911 >> 172.21.15.89:0/496182911 conn(0x7f0c940cf190 msgr2=0x7f0c94155f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:40.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.879+0000 7f0c89600700 1 -- 172.21.15.89:0/496182911 shutdown_connections 2024-04-02T07:36:40.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:40.880+0000 7f0c89600700 1 -- 172.21.15.89:0/496182911 wait complete. 2024-04-02T07:36:40.901 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:35:51.296561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:35:52.232882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:36:40.902 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:36:41.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.095+0000 7f45b7a00700 1 Processor -- start 2024-04-02T07:36:41.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.095+0000 7f45b7a00700 1 -- start start 2024-04-02T07:36:41.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.097+0000 7f45b7a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b015d9d0 0x7f45b00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.097+0000 7f45b7a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f45b00d2540 0x7f45b00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.097+0000 7f45b7a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b0161420 con 0x7f45b00d16a0 2024-04-02T07:36:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.097+0000 7f45b7a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b015edc0 con 0x7f45b00d2540 2024-04-02T07:36:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.097+0000 7f45b7a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b0161700 con 0x7f45b015d9d0 2024-04-02T07:36:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.097+0000 7f45b5a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f45b00d2540 0x7f45b00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.097+0000 7f45b5a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f45b00d2540 0x7f45b00ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48732/0 (socket says 172.21.15.89:48732) 2024-04-02T07:36:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.097+0000 7f45b5a00700 1 -- 172.21.15.89:0/583127575 learned_addr learned my addr 172.21.15.89:0/583127575 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.097+0000 7f45b5000700 1 --2- 172.21.15.89:0/583127575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b015d9d0 0x7f45b00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.097+0000 7f45b5a00700 1 -- 172.21.15.89:0/583127575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b015d9d0 msgr2=0x7f45b00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.097+0000 7f45b5a00700 1 --2- 172.21.15.89:0/583127575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b015d9d0 0x7f45b00d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.097+0000 7f45b5a00700 1 -- 172.21.15.89:0/583127575 >> v1:172.21.15.113:6789/0 conn(0x7f45b00d16a0 legacy=0x7f45b00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.097+0000 7f45b5a00700 1 -- 172.21.15.89:0/583127575 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b01619e0 con 0x7f45b00d2540 2024-04-02T07:36:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.098+0000 7f45b5a00700 1 --2- 172.21.15.89:0/583127575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b00d2540 0x7f45b00ce360 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009e80 tx=0x7f45a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c07fea6b1db717 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.098+0000 7f45af400700 1 -- 172.21.15.89:0/583127575 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a800e070 con 0x7f45b00d2540 2024-04-02T07:36:41.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.098+0000 7f45af400700 1 -- 172.21.15.89:0/583127575 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a8003a20 con 0x7f45b00d2540 2024-04-02T07:36:41.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.098+0000 7f45af400700 1 -- 172.21.15.89:0/583127575 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8003dd0 con 0x7f45b00d2540 2024-04-02T07:36:41.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.098+0000 7f45b7a00700 1 -- 172.21.15.89:0/583127575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b00d2540 msgr2=0x7f45b00ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.098+0000 7f45b7a00700 1 --2- 172.21.15.89:0/583127575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b00d2540 0x7f45b00ce360 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009e80 tx=0x7f45a800b330 comp rx=0 tx=0).stop 2024-04-02T07:36:41.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.098+0000 7f45b7a00700 1 -- 172.21.15.89:0/583127575 shutdown_connections 2024-04-02T07:36:41.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.098+0000 7f45b7a00700 1 --2- 172.21.15.89:0/583127575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b015d9d0 0x7f45b00d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.098+0000 7f45b7a00700 1 --2- 172.21.15.89:0/583127575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b00d2540 0x7f45b00ce360 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.098+0000 7f45b7a00700 1 -- 172.21.15.89:0/583127575 >> 172.21.15.89:0/583127575 conn(0x7f45b008aff0 msgr2=0x7f45b00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:41.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.099+0000 7f45b7a00700 1 -- 172.21.15.89:0/583127575 shutdown_connections 2024-04-02T07:36:41.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.099+0000 7f45b7a00700 1 -- 172.21.15.89:0/583127575 wait complete. 2024-04-02T07:36:41.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.100+0000 7f45b7a00700 1 Processor -- start 2024-04-02T07:36:41.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.100+0000 7f45b7a00700 1 -- start start 2024-04-02T07:36:41.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.101+0000 7f45b7a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b00d16a0 0x7f45b01e8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.101+0000 7f45b7a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b00d2540 0x7f45b014f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.101+0000 7f45b7a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b015d9d0 0x7f45b015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.101+0000 7f45b7a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b015cc80 con 0x7f45b00d16a0 2024-04-02T07:36:41.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.101+0000 7f45b7a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b01e9220 con 0x7f45b00d2540 2024-04-02T07:36:41.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.101+0000 7f45b7a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b00b6990 con 0x7f45b015d9d0 2024-04-02T07:36:41.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45b5000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b00d16a0 0x7f45b01e8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45b5000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b00d16a0 0x7f45b01e8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48738/0 (socket says 172.21.15.89:48738) 2024-04-02T07:36:41.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45b5000700 1 -- 172.21.15.89:0/1358562969 learned_addr learned my addr 172.21.15.89:0/1358562969 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:41.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45b5a00700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b015d9d0 0x7f45b015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45afe00700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b00d2540 0x7f45b014f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45b5000700 1 -- 172.21.15.89:0/1358562969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b015d9d0 msgr2=0x7f45b015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45b5000700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b015d9d0 0x7f45b015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45b5000700 1 -- 172.21.15.89:0/1358562969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b00d2540 msgr2=0x7f45b014f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45b5000700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b00d2540 0x7f45b014f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45b5000700 1 -- 172.21.15.89:0/1358562969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a8009b30 con 0x7f45b00d16a0 2024-04-02T07:36:41.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45afe00700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b00d2540 0x7f45b014f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:41.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45b5000700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b00d16a0 0x7f45b01e8ce0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f45a400d800 tx=0x7f45a400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:41.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45acc00700 1 -- 172.21.15.89:0/1358562969 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a40125f0 con 0x7f45b00d16a0 2024-04-02T07:36:41.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45acc00700 1 -- 172.21.15.89:0/1358562969 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a4012750 con 0x7f45b00d16a0 2024-04-02T07:36:41.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45b7a00700 1 -- 172.21.15.89:0/1358562969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b00b6c70 con 0x7f45b00d16a0 2024-04-02T07:36:41.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.102+0000 7f45acc00700 1 -- 172.21.15.89:0/1358562969 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4012b00 con 0x7f45b00d16a0 2024-04-02T07:36:41.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.103+0000 7f45b7a00700 1 -- 172.21.15.89:0/1358562969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b00b7230 con 0x7f45b00d16a0 2024-04-02T07:36:41.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.103+0000 7f45b7a00700 1 -- 172.21.15.89:0/1358562969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b00d1ca0 con 0x7f45b00d16a0 2024-04-02T07:36:41.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.104+0000 7f45acc00700 1 -- 172.21.15.89:0/1358562969 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f45a4012c60 con 0x7f45b00d16a0 2024-04-02T07:36:41.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.105+0000 7f45acc00700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f459006ac90 0x7f459006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.105+0000 7f45acc00700 1 -- 172.21.15.89:0/1358562969 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f45a4089490 con 0x7f45b00d16a0 2024-04-02T07:36:41.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.105+0000 7f45afe00700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f459006ac90 0x7f459006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.106+0000 7f45afe00700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f459006ac90 0x7f459006d150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f459c007fd0 tx=0x7f459c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:41.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.110+0000 7f45acc00700 1 -- 172.21.15.89:0/1358562969 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f45a40545b0 con 0x7f45b00d16a0 2024-04-02T07:36:41.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.316+0000 7f45b7a00700 1 -- 172.21.15.89:0/1358562969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f45b0067720 con 0x7f45b00d16a0 2024-04-02T07:36:41.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.317+0000 7f45acc00700 1 -- 172.21.15.89:0/1358562969 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f45a4057bd0 con 0x7f45b00d16a0 2024-04-02T07:36:41.318 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:36:41.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.319+0000 7f45b7a00700 1 -- 172.21.15.89:0/1358562969 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f459006ac90 msgr2=0x7f459006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.319+0000 7f45b7a00700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f459006ac90 0x7f459006d150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f459c007fd0 tx=0x7f459c009040 comp rx=0 tx=0).stop 2024-04-02T07:36:41.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.319+0000 7f45b7a00700 1 -- 172.21.15.89:0/1358562969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b00d16a0 msgr2=0x7f45b01e8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.319+0000 7f45b7a00700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b00d16a0 0x7f45b01e8ce0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f45a400d800 tx=0x7f45a400db10 comp rx=0 tx=0).stop 2024-04-02T07:36:41.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.319+0000 7f45b7a00700 1 -- 172.21.15.89:0/1358562969 shutdown_connections 2024-04-02T07:36:41.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.319+0000 7f45b7a00700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f459006ac90 0x7f459006d150 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.319+0000 7f45b7a00700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b015d9d0 0x7f45b015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.319+0000 7f45b7a00700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b00d16a0 0x7f45b01e8ce0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.319+0000 7f45b7a00700 1 --2- 172.21.15.89:0/1358562969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b00d2540 0x7f45b014f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.319+0000 7f45b7a00700 1 -- 172.21.15.89:0/1358562969 >> 172.21.15.89:0/1358562969 conn(0x7f45b008aff0 msgr2=0x7f45b00b5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:41.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.319+0000 7f45b7a00700 1 -- 172.21.15.89:0/1358562969 shutdown_connections 2024-04-02T07:36:41.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.319+0000 7f45b7a00700 1 -- 172.21.15.89:0/1358562969 wait complete. 2024-04-02T07:36:41.339 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:36:41.455 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:36:41.460 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:36:41.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.535+0000 7f9604400700 1 Processor -- start 2024-04-02T07:36:41.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.535+0000 7f9604400700 1 -- start start 2024-04-02T07:36:41.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.536+0000 7f9604400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95fc15d9d0 0x7f95fc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.536+0000 7f9604400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f95fc0d2540 0x7f95fc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.536+0000 7f9604400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95fc161420 con 0x7f95fc0d16a0 2024-04-02T07:36:41.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.536+0000 7f9604400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95fc15edc0 con 0x7f95fc0d2540 2024-04-02T07:36:41.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.536+0000 7f9604400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc161700 con 0x7f95fc15d9d0 2024-04-02T07:36:41.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.537+0000 7f9602400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f95fc0d2540 0x7f95fc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.537+0000 7f9601000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f95fc0d16a0 0x7f95fc0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53094/0 (socket says 172.21.15.89:53094) 2024-04-02T07:36:41.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.537+0000 7f9601000700 1 -- 172.21.15.89:0/2505398059 learned_addr learned my addr 172.21.15.89:0/2505398059 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:41.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.537+0000 7f9601a00700 1 --2- 172.21.15.89:0/2505398059 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95fc15d9d0 0x7f95fc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.537+0000 7f95fbe00700 1 -- 172.21.15.89:0/2505398059 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339537053 0 0) 0x7f95fc161420 con 0x7f95fc0d16a0 2024-04-02T07:36:41.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.537+0000 7f95fbe00700 1 -- 172.21.15.89:0/2505398059 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e40035b0 con 0x7f95fc0d16a0 2024-04-02T07:36:41.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.537+0000 7f9602400700 1 -- 172.21.15.89:0/2505398059 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95fc15d9d0 msgr2=0x7f95fc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:41.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.537+0000 7f9602400700 1 --2- 172.21.15.89:0/2505398059 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95fc15d9d0 0x7f95fc0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.537+0000 7f9602400700 1 -- 172.21.15.89:0/2505398059 >> v1:172.21.15.89:6789/0 conn(0x7f95fc0d16a0 legacy=0x7f95fc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.537+0000 7f9602400700 1 -- 172.21.15.89:0/2505398059 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc1619e0 con 0x7f95fc0d2540 2024-04-02T07:36:41.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.538+0000 7f9602400700 1 --2- 172.21.15.89:0/2505398059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0d2540 0x7f95fc0ce360 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009dc0 tx=0x7f95f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7304344866c79f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:41.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.538+0000 7f95fbe00700 1 -- 172.21.15.89:0/2505398059 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f400e070 con 0x7f95fc0d2540 2024-04-02T07:36:41.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.538+0000 7f95fbe00700 1 -- 172.21.15.89:0/2505398059 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f400bae0 con 0x7f95fc0d2540 2024-04-02T07:36:41.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.538+0000 7f95fbe00700 1 -- 172.21.15.89:0/2505398059 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f400be90 con 0x7f95fc0d2540 2024-04-02T07:36:41.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.538+0000 7f9604400700 1 -- 172.21.15.89:0/2505398059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0d2540 msgr2=0x7f95fc0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.538+0000 7f9604400700 1 --2- 172.21.15.89:0/2505398059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0d2540 0x7f95fc0ce360 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009dc0 tx=0x7f95f400b330 comp rx=0 tx=0).stop 2024-04-02T07:36:41.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.538+0000 7f9604400700 1 -- 172.21.15.89:0/2505398059 shutdown_connections 2024-04-02T07:36:41.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.538+0000 7f9604400700 1 --2- 172.21.15.89:0/2505398059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0d2540 0x7f95fc0ce360 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.538+0000 7f9604400700 1 --2- 172.21.15.89:0/2505398059 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95fc15d9d0 0x7f95fc0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.538+0000 7f9604400700 1 -- 172.21.15.89:0/2505398059 >> 172.21.15.89:0/2505398059 conn(0x7f95fc08aff0 msgr2=0x7f95fc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:41.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.538+0000 7f9604400700 1 -- 172.21.15.89:0/2505398059 shutdown_connections 2024-04-02T07:36:41.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.539+0000 7f9604400700 1 -- 172.21.15.89:0/2505398059 wait complete. 2024-04-02T07:36:41.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.540+0000 7f9604400700 1 Processor -- start 2024-04-02T07:36:41.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.540+0000 7f9604400700 1 -- start start 2024-04-02T07:36:41.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.541+0000 7f9604400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95fc0d16a0 0x7f95fc15f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.541+0000 7f9604400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0d2540 0x7f95fc0b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.541+0000 7f9604400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95fc15d9d0 0x7f95fc1585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.541+0000 7f9604400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc0b8f50 con 0x7f95fc0d2540 2024-04-02T07:36:41.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.541+0000 7f9604400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc1f6b20 con 0x7f95fc0d16a0 2024-04-02T07:36:41.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.541+0000 7f9604400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc1f6e00 con 0x7f95fc15d9d0 2024-04-02T07:36:41.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.541+0000 7f9601000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0d2540 0x7f95fc0b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.542+0000 7f9601000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0d2540 0x7f95fc0b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48748/0 (socket says 172.21.15.89:48748) 2024-04-02T07:36:41.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.542+0000 7f9601000700 1 -- 172.21.15.89:0/1397585238 learned_addr learned my addr 172.21.15.89:0/1397585238 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:41.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.542+0000 7f9601a00700 1 --2- 172.21.15.89:0/1397585238 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95fc0d16a0 0x7f95fc15f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.542+0000 7f9602400700 1 --2- 172.21.15.89:0/1397585238 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95fc15d9d0 0x7f95fc1585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.542+0000 7f9601000700 1 -- 172.21.15.89:0/1397585238 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95fc15d9d0 msgr2=0x7f95fc1585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.542+0000 7f9601000700 1 --2- 172.21.15.89:0/1397585238 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95fc15d9d0 0x7f95fc1585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.542+0000 7f9601000700 1 -- 172.21.15.89:0/1397585238 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95fc0d16a0 msgr2=0x7f95fc15f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.542+0000 7f9601000700 1 --2- 172.21.15.89:0/1397585238 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95fc0d16a0 0x7f95fc15f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.542+0000 7f9601000700 1 -- 172.21.15.89:0/1397585238 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f4009a70 con 0x7f95fc0d2540 2024-04-02T07:36:41.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.542+0000 7f9601000700 1 --2- 172.21.15.89:0/1397585238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0d2540 0x7f95fc0b4570 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f95e800c700 tx=0x7f95e800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:41.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.542+0000 7f95f9600700 1 -- 172.21.15.89:0/1397585238 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e80074e0 con 0x7f95fc0d2540 2024-04-02T07:36:41.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.543+0000 7f95f9600700 1 -- 172.21.15.89:0/1397585238 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e8007640 con 0x7f95fc0d2540 2024-04-02T07:36:41.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.543+0000 7f95f9600700 1 -- 172.21.15.89:0/1397585238 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e8005a50 con 0x7f95fc0d2540 2024-04-02T07:36:41.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.543+0000 7f9604400700 1 -- 172.21.15.89:0/1397585238 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95fc1f70e0 con 0x7f95fc0d2540 2024-04-02T07:36:41.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.543+0000 7f9604400700 1 -- 172.21.15.89:0/1397585238 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95fc1f76a0 con 0x7f95fc0d2540 2024-04-02T07:36:41.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.544+0000 7f9604400700 1 -- 172.21.15.89:0/1397585238 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95fc0d1ca0 con 0x7f95fc0d2540 2024-04-02T07:36:41.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.545+0000 7f95f9600700 1 -- 172.21.15.89:0/1397585238 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f95e8005bb0 con 0x7f95fc0d2540 2024-04-02T07:36:41.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.546+0000 7f95f9600700 1 --2- 172.21.15.89:0/1397585238 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f95dc06acc0 0x7f95dc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.546+0000 7f95f9600700 1 -- 172.21.15.89:0/1397585238 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f95e80886d0 con 0x7f95fc0d2540 2024-04-02T07:36:41.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.546+0000 7f9601a00700 1 --2- 172.21.15.89:0/1397585238 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f95dc06acc0 0x7f95dc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.547+0000 7f9601a00700 1 --2- 172.21.15.89:0/1397585238 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f95dc06acc0 0x7f95dc06d180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f95f0009b40 tx=0x7f95f0009210 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:41.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.550+0000 7f95f9600700 1 -- 172.21.15.89:0/1397585238 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f95e8053920 con 0x7f95fc0d2540 2024-04-02T07:36:41.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.755+0000 7f9604400700 1 -- 172.21.15.89:0/1397585238 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f95fc1f7830 con 0x7f95fc0d2540 2024-04-02T07:36:41.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.756+0000 7f95f9600700 1 -- 172.21.15.89:0/1397585238 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f95e8005e60 con 0x7f95fc0d2540 2024-04-02T07:36:41.758 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:36:41.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.758+0000 7f9604400700 1 -- 172.21.15.89:0/1397585238 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f95dc06acc0 msgr2=0x7f95dc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.758+0000 7f9604400700 1 --2- 172.21.15.89:0/1397585238 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f95dc06acc0 0x7f95dc06d180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f95f0009b40 tx=0x7f95f0009210 comp rx=0 tx=0).stop 2024-04-02T07:36:41.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.758+0000 7f9604400700 1 -- 172.21.15.89:0/1397585238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0d2540 msgr2=0x7f95fc0b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.758+0000 7f9604400700 1 --2- 172.21.15.89:0/1397585238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0d2540 0x7f95fc0b4570 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f95e800c700 tx=0x7f95e800ca10 comp rx=0 tx=0).stop 2024-04-02T07:36:41.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.758+0000 7f9604400700 1 -- 172.21.15.89:0/1397585238 shutdown_connections 2024-04-02T07:36:41.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.758+0000 7f9604400700 1 --2- 172.21.15.89:0/1397585238 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f95dc06acc0 0x7f95dc06d180 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.758+0000 7f9604400700 1 --2- 172.21.15.89:0/1397585238 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95fc0d16a0 0x7f95fc15f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.758+0000 7f9604400700 1 --2- 172.21.15.89:0/1397585238 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95fc15d9d0 0x7f95fc1585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.758+0000 7f9604400700 1 --2- 172.21.15.89:0/1397585238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0d2540 0x7f95fc0b4570 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.758+0000 7f9604400700 1 -- 172.21.15.89:0/1397585238 >> 172.21.15.89:0/1397585238 conn(0x7f95fc08aff0 msgr2=0x7f95fc0b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:41.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.758+0000 7f9604400700 1 -- 172.21.15.89:0/1397585238 shutdown_connections 2024-04-02T07:36:41.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.758+0000 7f9604400700 1 -- 172.21.15.89:0/1397585238 wait complete. 2024-04-02T07:36:41.778 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:36:41.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.972+0000 7f9a6c000700 1 Processor -- start 2024-04-02T07:36:41.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.972+0000 7f9a6c000700 1 -- start start 2024-04-02T07:36:41.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.974+0000 7f9a6c000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a640d16a0 0x7f9a640d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.974+0000 7f9a6c000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a64161420 con 0x7f9a640d2370 2024-04-02T07:36:41.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.974+0000 7f9a6c000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a64161700 con 0x7f9a6415d9d0 2024-04-02T07:36:41.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.974+0000 7f9a6c000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640d2820 con 0x7f9a640d16a0 2024-04-02T07:36:41.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.974+0000 7f9a6a000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9a640d2370 0x7f9a640ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53104/0 (socket says 172.21.15.89:53104) 2024-04-02T07:36:41.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.974+0000 7f9a6a000700 1 -- 172.21.15.89:0/2313191699 learned_addr learned my addr 172.21.15.89:0/2313191699 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:41.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.974+0000 7f9a68c00700 1 --2- 172.21.15.89:0/2313191699 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a640d16a0 0x7f9a640d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.974+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706904788 0 0) 0x7f9a64161420 con 0x7f9a640d2370 2024-04-02T07:36:41.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.974+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a4c0035b0 con 0x7f9a640d2370 2024-04-02T07:36:41.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.974+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422245098 0 0) 0x7f9a64161700 con 0x7f9a6415d9d0 2024-04-02T07:36:41.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a64161420 con 0x7f9a6415d9d0 2024-04-02T07:36:41.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1578310068 0 0) 0x7f9a4c0035b0 con 0x7f9a640d2370 2024-04-02T07:36:41.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a640d16a0 msgr2=0x7f9a640d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:41.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a63e00700 1 --2- 172.21.15.89:0/2313191699 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a640d16a0 0x7f9a640d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 >> v1:172.21.15.113:6789/0 conn(0x7f9a6415d9d0 legacy=0x7f9a640d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a641619e0 con 0x7f9a640d2370 2024-04-02T07:36:41.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f9a5c0019f0 con 0x7f9a640d2370 2024-04-02T07:36:41.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 >> v1:172.21.15.89:6789/0 conn(0x7f9a640d2370 legacy=0x7f9a640ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a63e00700 1 --2- 172.21.15.89:0/2313191699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a4c003bc0 0x7f9a4c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a63e00700 1 --2- 172.21.15.89:0/2313191699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a4c004850 0x7f9a4c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a63e00700 1 --2- 172.21.15.89:0/2313191699 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a4c005210 0x7f9a4c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640d2820 con 0x7f9a4c004850 2024-04-02T07:36:41.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c00c000 con 0x7f9a4c003bc0 2024-04-02T07:36:41.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c00c1c0 con 0x7f9a4c005210 2024-04-02T07:36:41.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a68c00700 1 --2- 172.21.15.89:0/2313191699 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a640d16a0 0x7f9a640d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:41.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a68c00700 1 --2- 172.21.15.89:0/2313191699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a4c004850 0x7f9a4c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a6a000700 1 --2- 172.21.15.89:0/2313191699 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a4c005210 0x7f9a4c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a69600700 1 --2- 172.21.15.89:0/2313191699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a4c003bc0 0x7f9a4c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a68c00700 1 -- 172.21.15.89:0/2313191699 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a4c005210 msgr2=0x7f9a4c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:41.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a68c00700 1 --2- 172.21.15.89:0/2313191699 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a4c005210 0x7f9a4c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a68c00700 1 -- 172.21.15.89:0/2313191699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a4c003bc0 msgr2=0x7f9a4c003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:41.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a68c00700 1 --2- 172.21.15.89:0/2313191699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a4c003bc0 0x7f9a4c003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.975+0000 7f9a68c00700 1 -- 172.21.15.89:0/2313191699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c0035b0 con 0x7f9a4c004850 2024-04-02T07:36:41.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.976+0000 7f9a68c00700 1 --2- 172.21.15.89:0/2313191699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a4c004850 0x7f9a4c004cd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9a640cfbc0 tx=0x7f9a4c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52a826b9f09fff server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:41.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.976+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a50004030 con 0x7f9a4c004850 2024-04-02T07:36:41.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.976+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a50004190 con 0x7f9a4c004850 2024-04-02T07:36:41.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.976+0000 7f9a63e00700 1 -- 172.21.15.89:0/2313191699 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a50004540 con 0x7f9a4c004850 2024-04-02T07:36:41.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.976+0000 7f9a6c000700 1 -- 172.21.15.89:0/2313191699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a4c004850 msgr2=0x7f9a4c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.976+0000 7f9a6c000700 1 --2- 172.21.15.89:0/2313191699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a4c004850 0x7f9a4c004cd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9a640cfbc0 tx=0x7f9a4c005880 comp rx=0 tx=0).stop 2024-04-02T07:36:41.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.976+0000 7f9a6a000700 1 -- 172.21.15.89:0/2313191699 reap_dead start 2024-04-02T07:36:41.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.977+0000 7f9a6c000700 1 -- 172.21.15.89:0/2313191699 shutdown_connections 2024-04-02T07:36:41.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.977+0000 7f9a6c000700 1 -- 172.21.15.89:0/2313191699 >> 172.21.15.89:0/2313191699 conn(0x7f9a6408aff0 msgr2=0x7f9a640d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:41.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.977+0000 7f9a6c000700 1 -- 172.21.15.89:0/2313191699 shutdown_connections 2024-04-02T07:36:41.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.977+0000 7f9a6c000700 1 -- 172.21.15.89:0/2313191699 wait complete. 2024-04-02T07:36:41.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.978+0000 7f9a6c000700 1 Processor -- start 2024-04-02T07:36:41.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.979+0000 7f9a6c000700 1 -- start start 2024-04-02T07:36:41.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.979+0000 7f9a6c000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a4c004850 0x7f9a6415fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.980+0000 7f9a6c000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a640b8090 0x7f9a64158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.980+0000 7f9a6c000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a6415e650 0x7f9a6406ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.980+0000 7f9a6c000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6415e230 con 0x7f9a640b8090 2024-04-02T07:36:41.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.980+0000 7f9a6c000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a641603d0 con 0x7f9a6415e650 2024-04-02T07:36:41.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.980+0000 7f9a6c000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a64159030 con 0x7f9a4c004850 2024-04-02T07:36:41.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.980+0000 7f9a68c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a640b8090 0x7f9a64158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.980+0000 7f9a68c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a640b8090 0x7f9a64158af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48778/0 (socket says 172.21.15.89:48778) 2024-04-02T07:36:41.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.980+0000 7f9a68c00700 1 -- 172.21.15.89:0/3720588579 learned_addr learned my addr 172.21.15.89:0/3720588579 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:41.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.980+0000 7f9a68c00700 1 -- 172.21.15.89:0/3720588579 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a4c004850 msgr2=0x7f9a6415fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.980+0000 7f9a68c00700 1 --2- 172.21.15.89:0/3720588579 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a4c004850 0x7f9a6415fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.980+0000 7f9a68c00700 1 -- 172.21.15.89:0/3720588579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a6415e650 msgr2=0x7f9a6406ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:41.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.980+0000 7f9a68c00700 1 --2- 172.21.15.89:0/3720588579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a6415e650 0x7f9a6406ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:41.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.980+0000 7f9a68c00700 1 -- 172.21.15.89:0/3720588579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a640b8850 con 0x7f9a640b8090 2024-04-02T07:36:41.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.980+0000 7f9a68c00700 1 --2- 172.21.15.89:0/3720588579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a640b8090 0x7f9a64158af0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9a500095e0 tx=0x7f9a500047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:41.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.981+0000 7f9a61600700 1 -- 172.21.15.89:0/3720588579 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a50004030 con 0x7f9a640b8090 2024-04-02T07:36:41.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.981+0000 7f9a61600700 1 -- 172.21.15.89:0/3720588579 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a50005c30 con 0x7f9a640b8090 2024-04-02T07:36:41.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.981+0000 7f9a61600700 1 -- 172.21.15.89:0/3720588579 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5000e690 con 0x7f9a640b8090 2024-04-02T07:36:41.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.981+0000 7f9a6c000700 1 -- 172.21.15.89:0/3720588579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a640b8b30 con 0x7f9a640b8090 2024-04-02T07:36:41.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.981+0000 7f9a6c000700 1 -- 172.21.15.89:0/3720588579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6415de50 con 0x7f9a640b8090 2024-04-02T07:36:41.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.982+0000 7f9a6c000700 1 -- 172.21.15.89:0/3720588579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6415e010 con 0x7f9a640b8090 2024-04-02T07:36:41.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.983+0000 7f9a61600700 1 -- 172.21.15.89:0/3720588579 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9a50023070 con 0x7f9a640b8090 2024-04-02T07:36:41.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.984+0000 7f9a61600700 1 --2- 172.21.15.89:0/3720588579 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a4406ac90 0x7f9a4406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:41.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.984+0000 7f9a61600700 1 -- 172.21.15.89:0/3720588579 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9a5008aa30 con 0x7f9a640b8090 2024-04-02T07:36:41.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.984+0000 7f9a69600700 1 --2- 172.21.15.89:0/3720588579 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a4406ac90 0x7f9a4406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:41.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.985+0000 7f9a69600700 1 --2- 172.21.15.89:0/3720588579 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a4406ac90 0x7f9a4406d150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9a58007fd0 tx=0x7f9a58009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:41.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:41.988+0000 7f9a61600700 1 -- 172.21.15.89:0/3720588579 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9a50055be0 con 0x7f9a640b8090 2024-04-02T07:36:42.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.191+0000 7f9a6c000700 1 -- 172.21.15.89:0/3720588579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9a6415e370 con 0x7f9a640b8090 2024-04-02T07:36:42.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.191+0000 7f9a61600700 1 -- 172.21.15.89:0/3720588579 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9a6415e370 con 0x7f9a640b8090 2024-04-02T07:36:42.193 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:36:42.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.193+0000 7f9a6c000700 1 -- 172.21.15.89:0/3720588579 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a4406ac90 msgr2=0x7f9a4406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:42.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.193+0000 7f9a6c000700 1 --2- 172.21.15.89:0/3720588579 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a4406ac90 0x7f9a4406d150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9a58007fd0 tx=0x7f9a58009040 comp rx=0 tx=0).stop 2024-04-02T07:36:42.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.193+0000 7f9a6c000700 1 -- 172.21.15.89:0/3720588579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a640b8090 msgr2=0x7f9a64158af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:42.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.193+0000 7f9a6c000700 1 --2- 172.21.15.89:0/3720588579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a640b8090 0x7f9a64158af0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9a500095e0 tx=0x7f9a500047c0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.194+0000 7f9a6c000700 1 -- 172.21.15.89:0/3720588579 shutdown_connections 2024-04-02T07:36:42.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.194+0000 7f9a6c000700 1 --2- 172.21.15.89:0/3720588579 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a4406ac90 0x7f9a4406d150 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.194+0000 7f9a6c000700 1 --2- 172.21.15.89:0/3720588579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a6415e650 0x7f9a6406ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.194+0000 7f9a6c000700 1 --2- 172.21.15.89:0/3720588579 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a4c004850 0x7f9a6415fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.194+0000 7f9a6c000700 1 --2- 172.21.15.89:0/3720588579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a640b8090 0x7f9a64158af0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.194+0000 7f9a6c000700 1 -- 172.21.15.89:0/3720588579 >> 172.21.15.89:0/3720588579 conn(0x7f9a6408aff0 msgr2=0x7f9a640b53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:42.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.194+0000 7f9a6c000700 1 -- 172.21.15.89:0/3720588579 shutdown_connections 2024-04-02T07:36:42.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.194+0000 7f9a6c000700 1 -- 172.21.15.89:0/3720588579 wait complete. 2024-04-02T07:36:42.215 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T07:36:42.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.406+0000 7f7eb3000700 1 Processor -- start 2024-04-02T07:36:42.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.406+0000 7f7eb3000700 1 -- start start 2024-04-02T07:36:42.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.407+0000 7f7eb3000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7eac1513c0 0x7f7eac06ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:42.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.407+0000 7f7eb3000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7eac15b230 0x7f7eac0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:42.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.407+0000 7f7eb3000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac15b8c0 con 0x7f7eac06b0c0 2024-04-02T07:36:42.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.407+0000 7f7eb3000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eac06b570 con 0x7f7eac1513c0 2024-04-02T07:36:42.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.407+0000 7f7eb3000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac15b6c0 con 0x7f7eac15b230 2024-04-02T07:36:42.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.407+0000 7f7eb1000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7eac15b230 0x7f7eac0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:42.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.407+0000 7f7ea3e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f7eac06b0c0 0x7f7eac15ab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:37046/0 (socket says 172.21.15.89:37046) 2024-04-02T07:36:42.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.407+0000 7f7ea3e00700 1 -- 172.21.15.89:0/1144771382 learned_addr learned my addr 172.21.15.89:0/1144771382 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:42.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.407+0000 7f7eabe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7eac1513c0 0x7f7eac06ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:42.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.408+0000 7f7eab400700 1 -- 172.21.15.89:0/1144771382 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979005521 0 0) 0x7f7eac15b8c0 con 0x7f7eac06b0c0 2024-04-02T07:36:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.408+0000 7f7eb1000700 1 -- 172.21.15.89:0/1144771382 >> v2:172.21.15.113:3300/0 conn(0x7f7eac1513c0 msgr2=0x7f7eac06ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.408+0000 7f7eb1000700 1 --2- 172.21.15.89:0/1144771382 >> v2:172.21.15.113:3300/0 conn(0x7f7eac1513c0 0x7f7eac06ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.408+0000 7f7eb1000700 1 -- 172.21.15.89:0/1144771382 >> v1:172.21.15.113:6789/0 conn(0x7f7eac06b0c0 legacy=0x7f7eac15ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.408+0000 7f7eb1000700 1 -- 172.21.15.89:0/1144771382 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac15ee50 con 0x7f7eac15b230 2024-04-02T07:36:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.408+0000 7f7eb1000700 1 --2- 172.21.15.89:0/1144771382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7eac15b230 0x7f7eac0711b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400bfa0 tx=0x7f7ea400d330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e1c178c942d5ce4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.409+0000 7f7eab400700 1 -- 172.21.15.89:0/1144771382 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4010070 con 0x7f7eac15b230 2024-04-02T07:36:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.409+0000 7f7eab400700 1 -- 172.21.15.89:0/1144771382 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea400b2e0 con 0x7f7eac15b230 2024-04-02T07:36:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.409+0000 7f7eab400700 1 -- 172.21.15.89:0/1144771382 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea400dc40 con 0x7f7eac15b230 2024-04-02T07:36:42.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.409+0000 7f7eb3000700 1 -- 172.21.15.89:0/1144771382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7eac15b230 msgr2=0x7f7eac0711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:42.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.409+0000 7f7eb3000700 1 --2- 172.21.15.89:0/1144771382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7eac15b230 0x7f7eac0711b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400bfa0 tx=0x7f7ea400d330 comp rx=0 tx=0).stop 2024-04-02T07:36:42.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.409+0000 7f7eb3000700 1 -- 172.21.15.89:0/1144771382 shutdown_connections 2024-04-02T07:36:42.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.409+0000 7f7eb3000700 1 --2- 172.21.15.89:0/1144771382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7eac15b230 0x7f7eac0711b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.409+0000 7f7eb3000700 1 --2- 172.21.15.89:0/1144771382 >> v2:172.21.15.113:3300/0 conn(0x7f7eac1513c0 0x7f7eac06ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.409+0000 7f7eb3000700 1 -- 172.21.15.89:0/1144771382 >> 172.21.15.89:0/1144771382 conn(0x7f7eac0b1230 msgr2=0x7f7eac0b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:42.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.409+0000 7f7eb3000700 1 -- 172.21.15.89:0/1144771382 shutdown_connections 2024-04-02T07:36:42.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.409+0000 7f7eb3000700 1 -- 172.21.15.89:0/1144771382 wait complete. 2024-04-02T07:36:42.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.411+0000 7f7eb3000700 1 Processor -- start 2024-04-02T07:36:42.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.411+0000 7f7eb3000700 1 -- start start 2024-04-02T07:36:42.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.412+0000 7f7eb3000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac06b0c0 0x7f7eac06e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:42.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.412+0000 7f7eb3000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7eac1513c0 0x7f7eac098940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:42.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.412+0000 7f7eb3000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7eac15b230 0x7f7eac155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:42.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.412+0000 7f7eb3000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac1517a0 con 0x7f7eac06b0c0 2024-04-02T07:36:42.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.412+0000 7f7eb3000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac06eb40 con 0x7f7eac1513c0 2024-04-02T07:36:42.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.412+0000 7f7eb3000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac151d20 con 0x7f7eac15b230 2024-04-02T07:36:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.412+0000 7f7eabe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac06b0c0 0x7f7eac06e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.412+0000 7f7eabe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac06b0c0 0x7f7eac06e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48788/0 (socket says 172.21.15.89:48788) 2024-04-02T07:36:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.412+0000 7f7eabe00700 1 -- 172.21.15.89:0/2962905191 learned_addr learned my addr 172.21.15.89:0/2962905191 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7ea3e00700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7eac1513c0 0x7f7eac098940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7eb1000700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7eac15b230 0x7f7eac155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7eabe00700 1 -- 172.21.15.89:0/2962905191 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7eac15b230 msgr2=0x7f7eac155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7eabe00700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7eac15b230 0x7f7eac155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7eabe00700 1 -- 172.21.15.89:0/2962905191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7eac1513c0 msgr2=0x7f7eac098940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7eabe00700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7eac1513c0 0x7f7eac098940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7eabe00700 1 -- 172.21.15.89:0/2962905191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea400bc80 con 0x7f7eac06b0c0 2024-04-02T07:36:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7eb1000700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7eac15b230 0x7f7eac155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7ea3e00700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7eac1513c0 0x7f7eac098940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7eabe00700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac06b0c0 0x7f7eac06e600 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00e9b0 tx=0x7f7e9c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7ea8c00700 1 -- 172.21.15.89:0/2962905191 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c009710 con 0x7f7eac06b0c0 2024-04-02T07:36:42.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7ea8c00700 1 -- 172.21.15.89:0/2962905191 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9c00c7f0 con 0x7f7eac06b0c0 2024-04-02T07:36:42.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7eb3000700 1 -- 172.21.15.89:0/2962905191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eac157cf0 con 0x7f7eac06b0c0 2024-04-02T07:36:42.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7ea8c00700 1 -- 172.21.15.89:0/2962905191 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c00cb60 con 0x7f7eac06b0c0 2024-04-02T07:36:42.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.413+0000 7f7eb3000700 1 -- 172.21.15.89:0/2962905191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eac1519b0 con 0x7f7eac06b0c0 2024-04-02T07:36:42.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.414+0000 7f7eb3000700 1 -- 172.21.15.89:0/2962905191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eac152000 con 0x7f7eac06b0c0 2024-04-02T07:36:42.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.415+0000 7f7ea8c00700 1 -- 172.21.15.89:0/2962905191 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7e9c00ccc0 con 0x7f7eac06b0c0 2024-04-02T07:36:42.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.417+0000 7f7ea8c00700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7e8806ac90 0x7f7e8806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:42.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.417+0000 7f7ea8c00700 1 -- 172.21.15.89:0/2962905191 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7e9c089380 con 0x7f7eac06b0c0 2024-04-02T07:36:42.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.417+0000 7f7ea3e00700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7e8806ac90 0x7f7e8806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:42.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.417+0000 7f7ea3e00700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7e8806ac90 0x7f7e8806d150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7e94007fd0 tx=0x7f7e94009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:42.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.421+0000 7f7ea8c00700 1 -- 172.21.15.89:0/2962905191 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7e9c0545d0 con 0x7f7eac06b0c0 2024-04-02T07:36:42.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.639+0000 7f7eb3000700 1 -- 172.21.15.89:0/2962905191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7eac067720 con 0x7f7eac06b0c0 2024-04-02T07:36:42.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.639+0000 7f7ea8c00700 1 -- 172.21.15.89:0/2962905191 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7e9c057bf0 con 0x7f7eac06b0c0 2024-04-02T07:36:42.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.641+0000 7f7eb3000700 1 -- 172.21.15.89:0/2962905191 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7e8806ac90 msgr2=0x7f7e8806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:42.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.641+0000 7f7eb3000700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7e8806ac90 0x7f7e8806d150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7e94007fd0 tx=0x7f7e94009040 comp rx=0 tx=0).stop 2024-04-02T07:36:42.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.641+0000 7f7eb3000700 1 -- 172.21.15.89:0/2962905191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac06b0c0 msgr2=0x7f7eac06e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:42.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.641+0000 7f7eb3000700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac06b0c0 0x7f7eac06e600 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00e9b0 tx=0x7f7e9c00ed70 comp rx=0 tx=0).stop 2024-04-02T07:36:42.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.642+0000 7f7eb3000700 1 -- 172.21.15.89:0/2962905191 shutdown_connections 2024-04-02T07:36:42.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.642+0000 7f7eb3000700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7e8806ac90 0x7f7e8806d150 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.642+0000 7f7eb3000700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7eac15b230 0x7f7eac155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.642+0000 7f7eb3000700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7eac06b0c0 0x7f7eac06e600 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.642+0000 7f7eb3000700 1 --2- 172.21.15.89:0/2962905191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7eac1513c0 0x7f7eac098940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.642+0000 7f7eb3000700 1 -- 172.21.15.89:0/2962905191 >> 172.21.15.89:0/2962905191 conn(0x7f7eac0b1230 msgr2=0x7f7eac0b02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:42.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.642+0000 7f7eb3000700 1 -- 172.21.15.89:0/2962905191 shutdown_connections 2024-04-02T07:36:42.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.642+0000 7f7eb3000700 1 -- 172.21.15.89:0/2962905191 wait complete. 2024-04-02T07:36:42.645 INFO:teuthology.orchestra.run.smithi089.stderr:listed 0 entries 2024-04-02T07:36:42.661 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-04-02T07:36:42.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:42.661 DEBUG:teuthology.orchestra.run.smithi113:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-04-02T07:36:42.740 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi113.front.sepia.ceph.com 2024-04-02T07:36:42.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:36:42.740 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-02T07:36:42.787 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T07:36:42.788 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:36:42.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.979+0000 7fe60cc00700 1 Processor -- start 2024-04-02T07:36:42.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.980+0000 7fe60cc00700 1 -- start start 2024-04-02T07:36:42.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.981+0000 7fe60cc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6080ceec0 0x7fe6081612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:42.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.981+0000 7fe60cc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe608161800 0x7fe6080b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:42.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.981+0000 7fe60cc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6080d30d0 con 0x7fe60815d640 2024-04-02T07:36:42.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.981+0000 7fe60cc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe608161c90 con 0x7fe608161800 2024-04-02T07:36:42.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.981+0000 7fe60cc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080b6840 con 0x7fe6080ceec0 2024-04-02T07:36:42.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.981+0000 7fe606800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe608161800 0x7fe6080b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:42.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.981+0000 7fe5ffe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6080ceec0 0x7fe6081612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:42.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.981+0000 7fe605e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fe60815d640 0x7fe6080d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:37056/0 (socket says 172.21.15.89:37056) 2024-04-02T07:36:42.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.981+0000 7fe605e00700 1 -- 172.21.15.89:0/3110686446 learned_addr learned my addr 172.21.15.89:0/3110686446 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:42.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.982+0000 7fe606800700 1 -- 172.21.15.89:0/3110686446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6080ceec0 msgr2=0x7fe6081612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:42.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.982+0000 7fe606800700 1 --2- 172.21.15.89:0/3110686446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6080ceec0 0x7fe6081612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.982+0000 7fe606800700 1 -- 172.21.15.89:0/3110686446 >> v1:172.21.15.113:6789/0 conn(0x7fe60815d640 legacy=0x7fe6080d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:42.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.982+0000 7fe606800700 1 -- 172.21.15.89:0/3110686446 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6080d2b30 con 0x7fe608161800 2024-04-02T07:36:42.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.982+0000 7fe5ffe00700 1 --2- 172.21.15.89:0/3110686446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6080ceec0 0x7fe6081612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:42.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.982+0000 7fe605400700 1 -- 172.21.15.89:0/3110686446 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224566544 0 0) 0x7fe6080d30d0 con 0x7fe60815d640 2024-04-02T07:36:42.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.982+0000 7fe606800700 1 --2- 172.21.15.89:0/3110686446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe608161800 0x7fe6080b6cd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe600009d90 tx=0x7fe60000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9019f97ef26489c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:42.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.982+0000 7fe605400700 1 -- 172.21.15.89:0/3110686446 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60000e070 con 0x7fe608161800 2024-04-02T07:36:42.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.982+0000 7fe605400700 1 -- 172.21.15.89:0/3110686446 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60000bb20 con 0x7fe608161800 2024-04-02T07:36:42.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.982+0000 7fe605400700 1 -- 172.21.15.89:0/3110686446 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe600012430 con 0x7fe608161800 2024-04-02T07:36:42.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.983+0000 7fe60cc00700 1 -- 172.21.15.89:0/3110686446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe608161800 msgr2=0x7fe6080b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:42.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.983+0000 7fe60cc00700 1 --2- 172.21.15.89:0/3110686446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe608161800 0x7fe6080b6cd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe600009d90 tx=0x7fe60000b330 comp rx=0 tx=0).stop 2024-04-02T07:36:42.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.983+0000 7fe60cc00700 1 -- 172.21.15.89:0/3110686446 shutdown_connections 2024-04-02T07:36:42.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.983+0000 7fe60cc00700 1 --2- 172.21.15.89:0/3110686446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe608161800 0x7fe6080b6cd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.983+0000 7fe60cc00700 1 --2- 172.21.15.89:0/3110686446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6080ceec0 0x7fe6081612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.983+0000 7fe60cc00700 1 -- 172.21.15.89:0/3110686446 >> 172.21.15.89:0/3110686446 conn(0x7fe60808aff0 msgr2=0x7fe6080d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:42.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.983+0000 7fe60cc00700 1 -- 172.21.15.89:0/3110686446 shutdown_connections 2024-04-02T07:36:42.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.983+0000 7fe60cc00700 1 -- 172.21.15.89:0/3110686446 wait complete. 2024-04-02T07:36:42.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.985+0000 7fe60cc00700 1 Processor -- start 2024-04-02T07:36:42.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.985+0000 7fe60cc00700 1 -- start start 2024-04-02T07:36:42.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe60cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6080ceec0 0x7fe60814c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:42.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe60cc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe60815d640 0x7fe6081f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:42.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe60cc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe608161800 0x7fe6081f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:42.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe60cc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6081f4770 con 0x7fe6080ceec0 2024-04-02T07:36:42.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe60cc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6081f3e60 con 0x7fe60815d640 2024-04-02T07:36:42.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe60cc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080c98c0 con 0x7fe608161800 2024-04-02T07:36:42.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe605e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6080ceec0 0x7fe60814c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:42.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe605e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6080ceec0 0x7fe60814c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48802/0 (socket says 172.21.15.89:48802) 2024-04-02T07:36:42.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe605e00700 1 -- 172.21.15.89:0/4173576491 learned_addr learned my addr 172.21.15.89:0/4173576491 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:42.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe606800700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe608161800 0x7fe6081f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:42.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe605e00700 1 -- 172.21.15.89:0/4173576491 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe608161800 msgr2=0x7fe6081f32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:42.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe605e00700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe608161800 0x7fe6081f32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe5ffe00700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe60815d640 0x7fe6081f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:42.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe605e00700 1 -- 172.21.15.89:0/4173576491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe60815d640 msgr2=0x7fe6081f2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:42.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe605e00700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe60815d640 0x7fe6081f2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:42.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe605e00700 1 -- 172.21.15.89:0/4173576491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f00094a0 con 0x7fe6080ceec0 2024-04-02T07:36:42.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe606800700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe608161800 0x7fe6081f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:42.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.986+0000 7fe5ffe00700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe60815d640 0x7fe6081f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:42.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.987+0000 7fe605e00700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6080ceec0 0x7fe60814c910 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80011e0 tx=0x7fe5f800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:42.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.987+0000 7fe5fe000700 1 -- 172.21.15.89:0/4173576491 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8004c80 con 0x7fe6080ceec0 2024-04-02T07:36:42.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.987+0000 7fe5fe000700 1 -- 172.21.15.89:0/4173576491 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f8004e00 con 0x7fe6080ceec0 2024-04-02T07:36:42.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.987+0000 7fe5fe000700 1 -- 172.21.15.89:0/4173576491 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8007310 con 0x7fe6080ceec0 2024-04-02T07:36:42.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.987+0000 7fe60cc00700 1 -- 172.21.15.89:0/4173576491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe600009a70 con 0x7fe6080ceec0 2024-04-02T07:36:42.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.987+0000 7fe60cc00700 1 -- 172.21.15.89:0/4173576491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6081f4400 con 0x7fe6080ceec0 2024-04-02T07:36:42.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.988+0000 7fe60cc00700 1 -- 172.21.15.89:0/4173576491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe608067720 con 0x7fe6080ceec0 2024-04-02T07:36:42.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.989+0000 7fe5fe000700 1 -- 172.21.15.89:0/4173576491 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe5f8007470 con 0x7fe6080ceec0 2024-04-02T07:36:42.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.990+0000 7fe5fe000700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe5e406ac90 0x7fe5e406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:42.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.990+0000 7fe5fe000700 1 -- 172.21.15.89:0/4173576491 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe5f80890e0 con 0x7fe6080ceec0 2024-04-02T07:36:42.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.990+0000 7fe5ffe00700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe5e406ac90 0x7fe5e406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:42.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.990+0000 7fe5ffe00700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe5e406ac90 0x7fe5e406d150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe60815b000 tx=0x7fe5f0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:42.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:42.995+0000 7fe5fe000700 1 -- 172.21.15.89:0/4173576491 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe5f8054290 con 0x7fe6080ceec0 2024-04-02T07:36:43.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.237+0000 7fe60cc00700 1 -- 172.21.15.89:0/4173576491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6080b5c70 con 0x7fe6080ceec0 2024-04-02T07:36:43.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.238+0000 7fe5fe000700 1 -- 172.21.15.89:0/4173576491 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4731 (secure 0 0 0) 0x7fe5f80578b0 con 0x7fe6080ceec0 2024-04-02T07:36:43.240 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:43.240 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2427593953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2427593953},{"type":"v1","addr":"172.21.15.113:6835","nonce":2427593953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/926246816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":926246816},{"type":"v1","addr":"172.21.15.89:6836","nonce":926246816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2701685411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2701685411},{"type":"v1","addr":"172.21.15.89:6837","nonce":2701685411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:35:53.235047+0000","modified":"2024-04-02T07:35:56.245811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6834/3962490902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3962490902},{"type":"v1","addr":"172.21.15.113:6834","nonce":3962490902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-02T07:36:43.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.240+0000 7fe60cc00700 1 -- 172.21.15.89:0/4173576491 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe5e406ac90 msgr2=0x7fe5e406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:43.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.240+0000 7fe60cc00700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe5e406ac90 0x7fe5e406d150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe60815b000 tx=0x7fe5f0008040 comp rx=0 tx=0).stop 2024-04-02T07:36:43.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.240+0000 7fe60cc00700 1 -- 172.21.15.89:0/4173576491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6080ceec0 msgr2=0x7fe60814c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:43.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.240+0000 7fe60cc00700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6080ceec0 0x7fe60814c910 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80011e0 tx=0x7fe5f800cae0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.240+0000 7fe60cc00700 1 -- 172.21.15.89:0/4173576491 shutdown_connections 2024-04-02T07:36:43.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.240+0000 7fe60cc00700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe5e406ac90 0x7fe5e406d150 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.240+0000 7fe60cc00700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe608161800 0x7fe6081f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.240+0000 7fe60cc00700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6080ceec0 0x7fe60814c910 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.240+0000 7fe60cc00700 1 --2- 172.21.15.89:0/4173576491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe60815d640 0x7fe6081f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.240+0000 7fe60cc00700 1 -- 172.21.15.89:0/4173576491 >> 172.21.15.89:0/4173576491 conn(0x7fe60808aff0 msgr2=0x7fe60815b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:43.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.240+0000 7fe60cc00700 1 -- 172.21.15.89:0/4173576491 shutdown_connections 2024-04-02T07:36:43.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.240+0000 7fe60cc00700 1 -- 172.21.15.89:0/4173576491 wait complete. 2024-04-02T07:36:43.243 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2024-04-02T07:36:43.260 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:36:43.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.450+0000 7fcebe200700 1 Processor -- start 2024-04-02T07:36:43.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.450+0000 7fcebe200700 1 -- start start 2024-04-02T07:36:43.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.451+0000 7fcebe200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fceb815d640 0x7fceb80d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:43.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.451+0000 7fcebe200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fceb81619d0 0x7fceb80b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:43.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.451+0000 7fcebe200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb80d30d0 con 0x7fceb80ceec0 2024-04-02T07:36:43.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.452+0000 7fcebe200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb80b6840 con 0x7fceb81619d0 2024-04-02T07:36:43.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.452+0000 7fcebe200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb80d2880 con 0x7fceb815d640 2024-04-02T07:36:43.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.452+0000 7fceb7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fceb81619d0 0x7fceb80b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:43.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.452+0000 7fceb7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fceb81619d0 0x7fceb80b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48808/0 (socket says 172.21.15.89:48808) 2024-04-02T07:36:43.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.452+0000 7fceb7e00700 1 -- 172.21.15.89:0/3712434242 learned_addr learned my addr 172.21.15.89:0/3712434242 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:43.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.452+0000 7fceb7400700 1 --2- 172.21.15.89:0/3712434242 >> v2:172.21.15.113:3300/0 conn(0x7fceb815d640 0x7fceb80d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.452+0000 7fceb7e00700 1 -- 172.21.15.89:0/3712434242 >> v2:172.21.15.113:3300/0 conn(0x7fceb815d640 msgr2=0x7fceb80d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.452+0000 7fceb7e00700 1 --2- 172.21.15.89:0/3712434242 >> v2:172.21.15.113:3300/0 conn(0x7fceb815d640 0x7fceb80d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.452+0000 7fceb7e00700 1 -- 172.21.15.89:0/3712434242 >> v1:172.21.15.113:6789/0 conn(0x7fceb80ceec0 legacy=0x7fceb81612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.452+0000 7fceb7e00700 1 -- 172.21.15.89:0/3712434242 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb80d2b30 con 0x7fceb81619d0 2024-04-02T07:36:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.452+0000 7fceb7400700 1 --2- 172.21.15.89:0/3712434242 >> v2:172.21.15.113:3300/0 conn(0x7fceb815d640 0x7fceb80d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.452+0000 7fceb7e00700 1 --2- 172.21.15.89:0/3712434242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb81619d0 0x7fceb80b6cd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fceb80b5c20 tx=0x7fceb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887fa42f45ebda24 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.453+0000 7fceb6a00700 1 -- 172.21.15.89:0/3712434242 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb000e070 con 0x7fceb81619d0 2024-04-02T07:36:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.453+0000 7fceb6a00700 1 -- 172.21.15.89:0/3712434242 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb0003a20 con 0x7fceb81619d0 2024-04-02T07:36:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.453+0000 7fceb6a00700 1 -- 172.21.15.89:0/3712434242 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb0003dd0 con 0x7fceb81619d0 2024-04-02T07:36:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.453+0000 7fcebe200700 1 -- 172.21.15.89:0/3712434242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb81619d0 msgr2=0x7fceb80b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.453+0000 7fcebe200700 1 --2- 172.21.15.89:0/3712434242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb81619d0 0x7fceb80b6cd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fceb80b5c20 tx=0x7fceb000b330 comp rx=0 tx=0).stop 2024-04-02T07:36:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.453+0000 7fcebe200700 1 -- 172.21.15.89:0/3712434242 shutdown_connections 2024-04-02T07:36:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.453+0000 7fcebe200700 1 --2- 172.21.15.89:0/3712434242 >> v2:172.21.15.113:3300/0 conn(0x7fceb815d640 0x7fceb80d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.453+0000 7fcebe200700 1 --2- 172.21.15.89:0/3712434242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb81619d0 0x7fceb80b6cd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.453+0000 7fcebe200700 1 -- 172.21.15.89:0/3712434242 >> 172.21.15.89:0/3712434242 conn(0x7fceb808aff0 msgr2=0x7fceb80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.453+0000 7fcebe200700 1 -- 172.21.15.89:0/3712434242 shutdown_connections 2024-04-02T07:36:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.453+0000 7fcebe200700 1 -- 172.21.15.89:0/3712434242 wait complete. 2024-04-02T07:36:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.455+0000 7fcebe200700 1 Processor -- start 2024-04-02T07:36:43.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.455+0000 7fcebe200700 1 -- start start 2024-04-02T07:36:43.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.456+0000 7fcebe200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fceb80ceec0 0x7fceb81f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:43.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.456+0000 7fcebe200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb815d640 0x7fceb8155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:43.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.456+0000 7fcebe200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fceb81619d0 0x7fceb81579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:43.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.456+0000 7fcebe200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb8157590 con 0x7fceb815d640 2024-04-02T07:36:43.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.456+0000 7fcebe200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb80c7ca0 con 0x7fceb81619d0 2024-04-02T07:36:43.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.456+0000 7fcebe200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb81f7580 con 0x7fceb80ceec0 2024-04-02T07:36:43.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.456+0000 7fceafe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb815d640 0x7fceb8155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:43.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.456+0000 7fceafe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb815d640 0x7fceb8155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48810/0 (socket says 172.21.15.89:48810) 2024-04-02T07:36:43.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.456+0000 7fceafe00700 1 -- 172.21.15.89:0/521034111 learned_addr learned my addr 172.21.15.89:0/521034111 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:43.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.456+0000 7fceb7e00700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fceb81619d0 0x7fceb81579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:43.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.456+0000 7fceb7400700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fceb80ceec0 0x7fceb81f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:43.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.457+0000 7fceafe00700 1 -- 172.21.15.89:0/521034111 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fceb80ceec0 msgr2=0x7fceb81f7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:43.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.457+0000 7fceafe00700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fceb80ceec0 0x7fceb81f7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.457+0000 7fceafe00700 1 -- 172.21.15.89:0/521034111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fceb81619d0 msgr2=0x7fceb81579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:43.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.457+0000 7fceafe00700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fceb81619d0 0x7fceb81579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.457+0000 7fceafe00700 1 -- 172.21.15.89:0/521034111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb0009b30 con 0x7fceb815d640 2024-04-02T07:36:43.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.457+0000 7fceb7400700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fceb80ceec0 0x7fceb81f7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:43.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.457+0000 7fceafe00700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb815d640 0x7fceb8155c30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcea000b4a0 tx=0x7fcea000b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:43.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.457+0000 7fceaf400700 1 -- 172.21.15.89:0/521034111 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea0011830 con 0x7fceb815d640 2024-04-02T07:36:43.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.457+0000 7fceaf400700 1 -- 172.21.15.89:0/521034111 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea0011990 con 0x7fceb815d640 2024-04-02T07:36:43.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.457+0000 7fceaf400700 1 -- 172.21.15.89:0/521034111 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea0011d40 con 0x7fceb815d640 2024-04-02T07:36:43.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.457+0000 7fcebe200700 1 -- 172.21.15.89:0/521034111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb8156910 con 0x7fceb815d640 2024-04-02T07:36:43.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.457+0000 7fcebe200700 1 -- 172.21.15.89:0/521034111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb8156ed0 con 0x7fceb815d640 2024-04-02T07:36:43.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.458+0000 7fcebe200700 1 -- 172.21.15.89:0/521034111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb81576d0 con 0x7fceb815d640 2024-04-02T07:36:43.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.459+0000 7fceaf400700 1 -- 172.21.15.89:0/521034111 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcea001a400 con 0x7fceb815d640 2024-04-02T07:36:43.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.460+0000 7fceaf400700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce9406ac90 0x7fce9406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:43.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.460+0000 7fceaf400700 1 -- 172.21.15.89:0/521034111 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcea0088fc0 con 0x7fceb815d640 2024-04-02T07:36:43.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.460+0000 7fceb7400700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce9406ac90 0x7fce9406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:43.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.461+0000 7fceb7400700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce9406ac90 0x7fce9406d150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcea800aa20 tx=0x7fcea8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:43.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.464+0000 7fceaf400700 1 -- 172.21.15.89:0/521034111 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fceb81576d0 con 0x7fceb815d640 2024-04-02T07:36:43.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.669+0000 7fcebe200700 1 -- 172.21.15.89:0/521034111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fceb81576d0 con 0x7fceb815d640 2024-04-02T07:36:43.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.671+0000 7fceaf400700 1 -- 172.21.15.89:0/521034111 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7fceb81576d0 con 0x7fceb815d640 2024-04-02T07:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:36:36.860889+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:36:43.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.673+0000 7fcebe200700 1 -- 172.21.15.89:0/521034111 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce9406ac90 msgr2=0x7fce9406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:43.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.673+0000 7fcebe200700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce9406ac90 0x7fce9406d150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcea800aa20 tx=0x7fcea8008040 comp rx=0 tx=0).stop 2024-04-02T07:36:43.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.673+0000 7fcebe200700 1 -- 172.21.15.89:0/521034111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb815d640 msgr2=0x7fceb8155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:43.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.673+0000 7fcebe200700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb815d640 0x7fceb8155c30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcea000b4a0 tx=0x7fcea000b7b0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.673+0000 7fcebe200700 1 -- 172.21.15.89:0/521034111 shutdown_connections 2024-04-02T07:36:43.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.673+0000 7fcebe200700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce9406ac90 0x7fce9406d150 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.673+0000 7fcebe200700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fceb81619d0 0x7fceb81579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.673+0000 7fcebe200700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fceb80ceec0 0x7fceb81f7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.673+0000 7fcebe200700 1 --2- 172.21.15.89:0/521034111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb815d640 0x7fceb8155c30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.673+0000 7fcebe200700 1 -- 172.21.15.89:0/521034111 >> 172.21.15.89:0/521034111 conn(0x7fceb808aff0 msgr2=0x7fceb815beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:43.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.673+0000 7fcebe200700 1 -- 172.21.15.89:0/521034111 shutdown_connections 2024-04-02T07:36:43.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.673+0000 7fcebe200700 1 -- 172.21.15.89:0/521034111 wait complete. 2024-04-02T07:36:43.693 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:35:51.296561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:35:52.232882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:36:43.694 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:36:43.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.886+0000 7f6e83a00700 1 Processor -- start 2024-04-02T07:36:43.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.886+0000 7f6e83a00700 1 -- start start 2024-04-02T07:36:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.887+0000 7f6e83a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6e7c15d9d0 0x7f6e7c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.887+0000 7f6e83a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7c161420 con 0x7f6e7c0d2540 2024-04-02T07:36:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.887+0000 7f6e83a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7c161700 con 0x7f6e7c0d16a0 2024-04-02T07:36:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.887+0000 7f6e83a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7c15edc0 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.887+0000 7f6e81000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6e7c15d9d0 0x7f6e7c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.887+0000 7f6e81a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6e7c0d2540 0x7f6e7c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53108/0 (socket says 172.21.15.89:53108) 2024-04-02T07:36:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.887+0000 7f6e81a00700 1 -- 172.21.15.89:0/1619564969 learned_addr learned my addr 172.21.15.89:0/1619564969 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.888+0000 7f6e81000700 1 -- 172.21.15.89:0/1619564969 >> v1:172.21.15.113:6789/0 conn(0x7f6e7c0d16a0 legacy=0x7f6e7c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.888+0000 7f6e81000700 1 -- 172.21.15.89:0/1619564969 >> v1:172.21.15.89:6789/0 conn(0x7f6e7c0d2540 legacy=0x7f6e7c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.888+0000 7f6e7b400700 1 -- 172.21.15.89:0/1619564969 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676678746 0 0) 0x7f6e7c161420 con 0x7f6e7c0d2540 2024-04-02T07:36:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.888+0000 7f6e81000700 1 -- 172.21.15.89:0/1619564969 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7c1619e0 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.888+0000 7f6e81000700 1 --2- 172.21.15.89:0/1619564969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c15d9d0 0x7f6e7c0d3c30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009840 tx=0x7f6e70009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c59fef19f93398 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.888+0000 7f6e7b400700 1 -- 172.21.15.89:0/1619564969 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70011830 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.888+0000 7f6e7b400700 1 -- 172.21.15.89:0/1619564969 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e70011990 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.888+0000 7f6e7b400700 1 -- 172.21.15.89:0/1619564969 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70011d20 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.889+0000 7f6e83a00700 1 -- 172.21.15.89:0/1619564969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c15d9d0 msgr2=0x7f6e7c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.889+0000 7f6e83a00700 1 --2- 172.21.15.89:0/1619564969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c15d9d0 0x7f6e7c0d3c30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009840 tx=0x7f6e70009b50 comp rx=0 tx=0).stop 2024-04-02T07:36:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.889+0000 7f6e83a00700 1 -- 172.21.15.89:0/1619564969 shutdown_connections 2024-04-02T07:36:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.889+0000 7f6e83a00700 1 --2- 172.21.15.89:0/1619564969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c15d9d0 0x7f6e7c0d3c30 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.889+0000 7f6e83a00700 1 -- 172.21.15.89:0/1619564969 >> 172.21.15.89:0/1619564969 conn(0x7f6e7c08aff0 msgr2=0x7f6e7c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:43.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.889+0000 7f6e83a00700 1 -- 172.21.15.89:0/1619564969 shutdown_connections 2024-04-02T07:36:43.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.889+0000 7f6e83a00700 1 -- 172.21.15.89:0/1619564969 wait complete. 2024-04-02T07:36:43.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.891+0000 7f6e83a00700 1 Processor -- start 2024-04-02T07:36:43.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.891+0000 7f6e83a00700 1 -- start start 2024-04-02T07:36:43.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e83a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c0d16a0 0x7f6e7c0caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:43.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e83a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0d2540 0x7f6e7c0b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:43.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e83a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c15d9d0 0x7f6e7c0b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:43.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e83a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c1e8270 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e83a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0caff0 con 0x7f6e7c0d16a0 2024-04-02T07:36:43.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e83a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c1e7660 con 0x7f6e7c0d2540 2024-04-02T07:36:43.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e81a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c15d9d0 0x7f6e7c0b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:43.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e81a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c15d9d0 0x7f6e7c0b85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48830/0 (socket says 172.21.15.89:48830) 2024-04-02T07:36:43.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e81a00700 1 -- 172.21.15.89:0/2040577699 learned_addr learned my addr 172.21.15.89:0/2040577699 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:43.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e81a00700 1 -- 172.21.15.89:0/2040577699 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0d2540 msgr2=0x7f6e7c0b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:43.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e7be00700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0d2540 0x7f6e7c0b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:43.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e81000700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c0d16a0 0x7f6e7c0caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:43.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e81a00700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0d2540 0x7f6e7c0b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e81a00700 1 -- 172.21.15.89:0/2040577699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c0d16a0 msgr2=0x7f6e7c0caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:43.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e81a00700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c0d16a0 0x7f6e7c0caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:43.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e81a00700 1 -- 172.21.15.89:0/2040577699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e700094a0 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e7be00700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0d2540 0x7f6e7c0b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:43.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e81000700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c0d16a0 0x7f6e7c0caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:43.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.892+0000 7f6e81a00700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c15d9d0 0x7f6e7c0b85a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400cc70 tx=0x7f6e7400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:43.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.893+0000 7f6e78c00700 1 -- 172.21.15.89:0/2040577699 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7400ad80 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.893+0000 7f6e78c00700 1 -- 172.21.15.89:0/2040577699 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7400ce70 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.893+0000 7f6e83a00700 1 -- 172.21.15.89:0/2040577699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e7c1e7940 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.893+0000 7f6e78c00700 1 -- 172.21.15.89:0/2040577699 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e740072f0 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.893+0000 7f6e83a00700 1 -- 172.21.15.89:0/2040577699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7c1e7f00 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.894+0000 7f6e83a00700 1 -- 172.21.15.89:0/2040577699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7c0d1ca0 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.895+0000 7f6e78c00700 1 -- 172.21.15.89:0/2040577699 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6e74007450 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.896+0000 7f6e78c00700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6e5c06ac90 0x7f6e5c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:43.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.896+0000 7f6e78c00700 1 -- 172.21.15.89:0/2040577699 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6e740892f0 con 0x7f6e7c15d9d0 2024-04-02T07:36:43.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.896+0000 7f6e81000700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6e5c06ac90 0x7f6e5c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:43.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.897+0000 7f6e81000700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6e5c06ac90 0x7f6e5c06d150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009b90 tx=0x7f6e7001c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:43.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:43.901+0000 7f6e78c00700 1 -- 172.21.15.89:0/2040577699 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6e74054410 con 0x7f6e7c15d9d0 2024-04-02T07:36:44.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.103+0000 7f6e83a00700 1 -- 172.21.15.89:0/2040577699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6e7c0d3c30 con 0x7f6e7c15d9d0 2024-04-02T07:36:44.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.103+0000 7f6e78c00700 1 -- 172.21.15.89:0/2040577699 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6e74057a30 con 0x7f6e7c15d9d0 2024-04-02T07:36:44.105 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:36:44.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.105+0000 7f6e83a00700 1 -- 172.21.15.89:0/2040577699 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6e5c06ac90 msgr2=0x7f6e5c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.105+0000 7f6e83a00700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6e5c06ac90 0x7f6e5c06d150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009b90 tx=0x7f6e7001c040 comp rx=0 tx=0).stop 2024-04-02T07:36:44.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.105+0000 7f6e83a00700 1 -- 172.21.15.89:0/2040577699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c15d9d0 msgr2=0x7f6e7c0b85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.105+0000 7f6e83a00700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c15d9d0 0x7f6e7c0b85a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400cc70 tx=0x7f6e7400a340 comp rx=0 tx=0).stop 2024-04-02T07:36:44.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.105+0000 7f6e83a00700 1 -- 172.21.15.89:0/2040577699 shutdown_connections 2024-04-02T07:36:44.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.105+0000 7f6e83a00700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6e5c06ac90 0x7f6e5c06d150 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.105+0000 7f6e83a00700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c0d16a0 0x7f6e7c0caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.105+0000 7f6e83a00700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c15d9d0 0x7f6e7c0b85a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.105+0000 7f6e83a00700 1 --2- 172.21.15.89:0/2040577699 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0d2540 0x7f6e7c0b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.105+0000 7f6e83a00700 1 -- 172.21.15.89:0/2040577699 >> 172.21.15.89:0/2040577699 conn(0x7f6e7c08aff0 msgr2=0x7f6e7c0ceca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:44.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.106+0000 7f6e83a00700 1 -- 172.21.15.89:0/2040577699 shutdown_connections 2024-04-02T07:36:44.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.106+0000 7f6e83a00700 1 -- 172.21.15.89:0/2040577699 wait complete. 2024-04-02T07:36:44.126 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:36:44.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.317+0000 7fcdd4600700 1 Processor -- start 2024-04-02T07:36:44.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.318+0000 7fcdd4600700 1 -- start start 2024-04-02T07:36:44.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.319+0000 7fcdd4600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdcc0ce840 0x7fcdcc165370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:44.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.319+0000 7fcdd4600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdcc0cb310 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.319+0000 7fcdd4600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdcc0cca90 con 0x7fcdcc15dfe0 2024-04-02T07:36:44.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.319+0000 7fcdd4600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc0711a0 con 0x7fcdcc0ce840 2024-04-02T07:36:44.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.319+0000 7fcdd1c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcdcc15a5c0 0x7fcdcc0709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53118/0 (socket says 172.21.15.89:53118) 2024-04-02T07:36:44.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.319+0000 7fcdd1c00700 1 -- 172.21.15.89:0/3127946962 learned_addr learned my addr 172.21.15.89:0/3127946962 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:44.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.319+0000 7fcdd2600700 1 --2- 172.21.15.89:0/3127946962 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdcc0ce840 0x7fcdcc165370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:44.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.319+0000 7fcdcbe00700 1 -- 172.21.15.89:0/3127946962 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741771130 0 0) 0x7fcdcc0cb310 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.320+0000 7fcdcbe00700 1 -- 172.21.15.89:0/3127946962 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb40057c0 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.320+0000 7fcdcbe00700 1 -- 172.21.15.89:0/3127946962 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982132694 0 0) 0x7fcdcc0cca90 con 0x7fcdcc15dfe0 2024-04-02T07:36:44.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.320+0000 7fcdcbe00700 1 -- 172.21.15.89:0/3127946962 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdcc0cb310 con 0x7fcdcc15dfe0 2024-04-02T07:36:44.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.320+0000 7fcdd2600700 1 -- 172.21.15.89:0/3127946962 >> v1:172.21.15.113:6789/0 conn(0x7fcdcc15dfe0 legacy=0x7fcdcc0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.320+0000 7fcdd2600700 1 -- 172.21.15.89:0/3127946962 >> v1:172.21.15.89:6789/0 conn(0x7fcdcc15a5c0 legacy=0x7fcdcc0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.320+0000 7fcdd2600700 1 -- 172.21.15.89:0/3127946962 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdcc071480 con 0x7fcdcc0ce840 2024-04-02T07:36:44.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.320+0000 7fcdd2600700 1 --2- 172.21.15.89:0/3127946962 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdcc0ce840 0x7fcdcc165370 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4009dc0 tx=0x7fcdc400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd814fd6733fadac server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:44.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.320+0000 7fcdcbe00700 1 -- 172.21.15.89:0/3127946962 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc400e070 con 0x7fcdcc0ce840 2024-04-02T07:36:44.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.320+0000 7fcdcbe00700 1 -- 172.21.15.89:0/3127946962 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc400bb20 con 0x7fcdcc0ce840 2024-04-02T07:36:44.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.320+0000 7fcdcbe00700 1 -- 172.21.15.89:0/3127946962 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc400be90 con 0x7fcdcc0ce840 2024-04-02T07:36:44.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.321+0000 7fcdd4600700 1 -- 172.21.15.89:0/3127946962 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdcc0ce840 msgr2=0x7fcdcc165370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.321+0000 7fcdd4600700 1 --2- 172.21.15.89:0/3127946962 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdcc0ce840 0x7fcdcc165370 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4009dc0 tx=0x7fcdc400b330 comp rx=0 tx=0).stop 2024-04-02T07:36:44.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.321+0000 7fcdd4600700 1 -- 172.21.15.89:0/3127946962 shutdown_connections 2024-04-02T07:36:44.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.321+0000 7fcdd4600700 1 --2- 172.21.15.89:0/3127946962 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdcc0ce840 0x7fcdcc165370 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.321+0000 7fcdd4600700 1 -- 172.21.15.89:0/3127946962 >> 172.21.15.89:0/3127946962 conn(0x7fcdcc0b0d70 msgr2=0x7fcdcc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:44.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.321+0000 7fcdd4600700 1 -- 172.21.15.89:0/3127946962 shutdown_connections 2024-04-02T07:36:44.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.321+0000 7fcdd4600700 1 -- 172.21.15.89:0/3127946962 wait complete. 2024-04-02T07:36:44.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.323+0000 7fcdd4600700 1 Processor -- start 2024-04-02T07:36:44.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.323+0000 7fcdd4600700 1 -- start start 2024-04-02T07:36:44.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd4600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdcc0ce840 0x7fcdcc158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:44.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd4600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdcc15a5c0 0x7fcdcc0aca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:44.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd4600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdcc15dfe0 0x7fcdcc14ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:44.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd4600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc151650 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd4600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc0acf70 con 0x7fcdcc15dfe0 2024-04-02T07:36:44.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd4600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc0ad240 con 0x7fcdcc0ce840 2024-04-02T07:36:44.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdcc15a5c0 0x7fcdcc0aca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:44.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdcc15a5c0 0x7fcdcc0aca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48836/0 (socket says 172.21.15.89:48836) 2024-04-02T07:36:44.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd1200700 1 -- 172.21.15.89:0/3692226213 learned_addr learned my addr 172.21.15.89:0/3692226213 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:44.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd1c00700 1 --2- 172.21.15.89:0/3692226213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdcc0ce840 0x7fcdcc158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:44.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd2600700 1 --2- 172.21.15.89:0/3692226213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdcc15dfe0 0x7fcdcc14ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:44.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd1200700 1 -- 172.21.15.89:0/3692226213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdcc0ce840 msgr2=0x7fcdcc158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd1200700 1 --2- 172.21.15.89:0/3692226213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdcc0ce840 0x7fcdcc158b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd1200700 1 -- 172.21.15.89:0/3692226213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdcc15dfe0 msgr2=0x7fcdcc14ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd1200700 1 --2- 172.21.15.89:0/3692226213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdcc15dfe0 0x7fcdcc14ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.324+0000 7fcdd1200700 1 -- 172.21.15.89:0/3692226213 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc4009a70 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.325+0000 7fcdd1200700 1 --2- 172.21.15.89:0/3692226213 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdcc15a5c0 0x7fcdcc0aca30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcdb800c500 tx=0x7fcdb800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:44.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.325+0000 7fcdc9600700 1 -- 172.21.15.89:0/3692226213 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb80052d0 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.325+0000 7fcdc9600700 1 -- 172.21.15.89:0/3692226213 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb8005450 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.325+0000 7fcdc9600700 1 -- 172.21.15.89:0/3692226213 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb80038c0 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.325+0000 7fcdd4600700 1 -- 172.21.15.89:0/3692226213 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdcc0ad520 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.325+0000 7fcdd4600700 1 -- 172.21.15.89:0/3692226213 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdcc0adae0 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.326+0000 7fcdd4600700 1 -- 172.21.15.89:0/3692226213 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdcc1517c0 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.327+0000 7fcdc9600700 1 -- 172.21.15.89:0/3692226213 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcdb8003a20 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.328+0000 7fcdc9600700 1 --2- 172.21.15.89:0/3692226213 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcdac06ac90 0x7fcdac06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:44.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.328+0000 7fcdc9600700 1 -- 172.21.15.89:0/3692226213 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcdb80886c0 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.328+0000 7fcdd1c00700 1 --2- 172.21.15.89:0/3692226213 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcdac06ac90 0x7fcdac06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:44.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.329+0000 7fcdd1c00700 1 --2- 172.21.15.89:0/3692226213 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcdac06ac90 0x7fcdac06d150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0007fd0 tx=0x7fcdc0007dc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:44.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.333+0000 7fcdc9600700 1 -- 172.21.15.89:0/3692226213 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcdb8053910 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.532+0000 7fcdd4600700 1 -- 172.21.15.89:0/3692226213 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcdcc067720 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.533+0000 7fcdc9600700 1 -- 172.21.15.89:0/3692226213 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcdb8056f30 con 0x7fcdcc15a5c0 2024-04-02T07:36:44.535 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:36:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.535+0000 7fcdd4600700 1 -- 172.21.15.89:0/3692226213 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcdac06ac90 msgr2=0x7fcdac06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.535+0000 7fcdd4600700 1 --2- 172.21.15.89:0/3692226213 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcdac06ac90 0x7fcdac06d150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0007fd0 tx=0x7fcdc0007dc0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.535+0000 7fcdd4600700 1 -- 172.21.15.89:0/3692226213 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdcc15a5c0 msgr2=0x7fcdcc0aca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.535+0000 7fcdd4600700 1 --2- 172.21.15.89:0/3692226213 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdcc15a5c0 0x7fcdcc0aca30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcdb800c500 tx=0x7fcdb800c810 comp rx=0 tx=0).stop 2024-04-02T07:36:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.535+0000 7fcdd4600700 1 -- 172.21.15.89:0/3692226213 shutdown_connections 2024-04-02T07:36:44.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.535+0000 7fcdd4600700 1 --2- 172.21.15.89:0/3692226213 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcdac06ac90 0x7fcdac06d150 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.535+0000 7fcdd4600700 1 --2- 172.21.15.89:0/3692226213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdcc15dfe0 0x7fcdcc14ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.535+0000 7fcdd4600700 1 --2- 172.21.15.89:0/3692226213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdcc0ce840 0x7fcdcc158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.535+0000 7fcdd4600700 1 --2- 172.21.15.89:0/3692226213 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdcc15a5c0 0x7fcdcc0aca30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.535+0000 7fcdd4600700 1 -- 172.21.15.89:0/3692226213 >> 172.21.15.89:0/3692226213 conn(0x7fcdcc0b0d70 msgr2=0x7fcdcc0b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:44.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.535+0000 7fcdd4600700 1 -- 172.21.15.89:0/3692226213 shutdown_connections 2024-04-02T07:36:44.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.535+0000 7fcdd4600700 1 -- 172.21.15.89:0/3692226213 wait complete. 2024-04-02T07:36:44.555 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:36:44.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.748+0000 7ff355c00700 1 Processor -- start 2024-04-02T07:36:44.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.748+0000 7ff355c00700 1 -- start start 2024-04-02T07:36:44.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.749+0000 7ff355c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff3501611b0 0x7ff35015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:44.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.749+0000 7ff355c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3500ce270 con 0x7ff3500cee10 2024-04-02T07:36:44.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.749+0000 7ff355c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35015b600 con 0x7ff35015f4d0 2024-04-02T07:36:44.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.749+0000 7ff355c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3500d2760 con 0x7ff3501611b0 2024-04-02T07:36:44.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.750+0000 7ff34f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff3500cee10 0x7ff350161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53134/0 (socket says 172.21.15.89:53134) 2024-04-02T07:36:44.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.750+0000 7ff34f400700 1 -- 172.21.15.89:0/1285108606 learned_addr learned my addr 172.21.15.89:0/1285108606 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:44.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.750+0000 7ff34ea00700 1 --2- 172.21.15.89:0/1285108606 >> v2:172.21.15.113:3300/0 conn(0x7ff3501611b0 0x7ff35015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:44.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.750+0000 7ff34d600700 1 -- 172.21.15.89:0/1285108606 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102792111 0 0) 0x7ff3500ce270 con 0x7ff3500cee10 2024-04-02T07:36:44.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.750+0000 7ff34d600700 1 -- 172.21.15.89:0/1285108606 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3340035b0 con 0x7ff3500cee10 2024-04-02T07:36:44.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.750+0000 7ff34ea00700 1 -- 172.21.15.89:0/1285108606 >> v1:172.21.15.113:6789/0 conn(0x7ff35015f4d0 legacy=0x7ff3500ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.750+0000 7ff34ea00700 1 -- 172.21.15.89:0/1285108606 >> v1:172.21.15.89:6789/0 conn(0x7ff3500cee10 legacy=0x7ff350161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.750+0000 7ff34ea00700 1 -- 172.21.15.89:0/1285108606 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3500d2a40 con 0x7ff3501611b0 2024-04-02T07:36:44.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.750+0000 7ff34ea00700 1 --2- 172.21.15.89:0/1285108606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3501611b0 0x7ff35015ef90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff340009840 tx=0x7ff340009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e06df0d02205c478 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:44.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.751+0000 7ff34d600700 1 -- 172.21.15.89:0/1285108606 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340011830 con 0x7ff3501611b0 2024-04-02T07:36:44.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.751+0000 7ff34d600700 1 -- 172.21.15.89:0/1285108606 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff340011990 con 0x7ff3501611b0 2024-04-02T07:36:44.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.751+0000 7ff34d600700 1 -- 172.21.15.89:0/1285108606 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340011d20 con 0x7ff3501611b0 2024-04-02T07:36:44.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.751+0000 7ff355c00700 1 -- 172.21.15.89:0/1285108606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3501611b0 msgr2=0x7ff35015ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.751+0000 7ff355c00700 1 --2- 172.21.15.89:0/1285108606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3501611b0 0x7ff35015ef90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff340009840 tx=0x7ff340009b50 comp rx=0 tx=0).stop 2024-04-02T07:36:44.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.751+0000 7ff355c00700 1 -- 172.21.15.89:0/1285108606 shutdown_connections 2024-04-02T07:36:44.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.751+0000 7ff355c00700 1 --2- 172.21.15.89:0/1285108606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3501611b0 0x7ff35015ef90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.751+0000 7ff355c00700 1 -- 172.21.15.89:0/1285108606 >> 172.21.15.89:0/1285108606 conn(0x7ff35008aff0 msgr2=0x7ff3500d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:44.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.751+0000 7ff355c00700 1 -- 172.21.15.89:0/1285108606 shutdown_connections 2024-04-02T07:36:44.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.751+0000 7ff355c00700 1 -- 172.21.15.89:0/1285108606 wait complete. 2024-04-02T07:36:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.753+0000 7ff355c00700 1 Processor -- start 2024-04-02T07:36:44.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.753+0000 7ff355c00700 1 -- start start 2024-04-02T07:36:44.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.754+0000 7ff355c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3500cee10 0x7ff350153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:44.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.754+0000 7ff355c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff35015f4d0 0x7ff3500c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:44.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.754+0000 7ff355c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3501611b0 0x7ff3501f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:44.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.754+0000 7ff355c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff350153ea0 con 0x7ff35015f4d0 2024-04-02T07:36:44.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.754+0000 7ff355c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3501f3d30 con 0x7ff3500cee10 2024-04-02T07:36:44.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.754+0000 7ff355c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3501f4010 con 0x7ff3501611b0 2024-04-02T07:36:44.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.754+0000 7ff34e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff35015f4d0 0x7ff3500c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:44.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.754+0000 7ff34e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff35015f4d0 0x7ff3500c7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48846/0 (socket says 172.21.15.89:48846) 2024-04-02T07:36:44.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.754+0000 7ff34e000700 1 -- 172.21.15.89:0/3680283402 learned_addr learned my addr 172.21.15.89:0/3680283402 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:44.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.754+0000 7ff34ea00700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3500cee10 0x7ff350153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:44.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.754+0000 7ff34f400700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3501611b0 0x7ff3501f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:44.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.755+0000 7ff34ea00700 1 -- 172.21.15.89:0/3680283402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3501611b0 msgr2=0x7ff3501f3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.755+0000 7ff34ea00700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3501611b0 0x7ff3501f3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.755+0000 7ff34ea00700 1 -- 172.21.15.89:0/3680283402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff35015f4d0 msgr2=0x7ff3500c7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.755+0000 7ff34ea00700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff35015f4d0 0x7ff3500c7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.755+0000 7ff34ea00700 1 -- 172.21.15.89:0/3680283402 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3400094a0 con 0x7ff3500cee10 2024-04-02T07:36:44.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.755+0000 7ff34e000700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff35015f4d0 0x7ff3500c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:44.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.755+0000 7ff34ea00700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3500cee10 0x7ff350153960 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff340016000 tx=0x7ff340004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:44.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.755+0000 7ff346a00700 1 -- 172.21.15.89:0/3680283402 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3400036a0 con 0x7ff3500cee10 2024-04-02T07:36:44.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.755+0000 7ff346a00700 1 -- 172.21.15.89:0/3680283402 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34001f070 con 0x7ff3500cee10 2024-04-02T07:36:44.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.756+0000 7ff355c00700 1 -- 172.21.15.89:0/3680283402 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3500c7580 con 0x7ff3500cee10 2024-04-02T07:36:44.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.756+0000 7ff346a00700 1 -- 172.21.15.89:0/3680283402 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340003a20 con 0x7ff3500cee10 2024-04-02T07:36:44.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.756+0000 7ff355c00700 1 -- 172.21.15.89:0/3680283402 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3500c82c0 con 0x7ff3500cee10 2024-04-02T07:36:44.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.757+0000 7ff355c00700 1 -- 172.21.15.89:0/3680283402 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3500c7aa0 con 0x7ff3500cee10 2024-04-02T07:36:44.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.758+0000 7ff346a00700 1 -- 172.21.15.89:0/3680283402 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff340003b80 con 0x7ff3500cee10 2024-04-02T07:36:44.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.759+0000 7ff346a00700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff33006ac90 0x7ff33006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:44.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.759+0000 7ff346a00700 1 -- 172.21.15.89:0/3680283402 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff340021070 con 0x7ff3500cee10 2024-04-02T07:36:44.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.759+0000 7ff34e000700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff33006ac90 0x7ff33006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:44.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.760+0000 7ff34e000700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff33006ac90 0x7ff33006d150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff33c000f60 tx=0x7ff33c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:44.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.763+0000 7ff346a00700 1 -- 172.21.15.89:0/3680283402 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff3400561e0 con 0x7ff3500cee10 2024-04-02T07:36:44.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.967+0000 7ff355c00700 1 -- 172.21.15.89:0/3680283402 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff3500c7aa0 con 0x7ff3500cee10 2024-04-02T07:36:44.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.968+0000 7ff346a00700 1 -- 172.21.15.89:0/3680283402 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff3500c7aa0 con 0x7ff3500cee10 2024-04-02T07:36:44.970 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:36:44.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.970+0000 7ff355c00700 1 -- 172.21.15.89:0/3680283402 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff33006ac90 msgr2=0x7ff33006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.970+0000 7ff355c00700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff33006ac90 0x7ff33006d150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff33c000f60 tx=0x7ff33c009040 comp rx=0 tx=0).stop 2024-04-02T07:36:44.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.970+0000 7ff355c00700 1 -- 172.21.15.89:0/3680283402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3500cee10 msgr2=0x7ff350153960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:44.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.970+0000 7ff355c00700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3500cee10 0x7ff350153960 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff340016000 tx=0x7ff340004750 comp rx=0 tx=0).stop 2024-04-02T07:36:44.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.970+0000 7ff355c00700 1 -- 172.21.15.89:0/3680283402 shutdown_connections 2024-04-02T07:36:44.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.970+0000 7ff355c00700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff33006ac90 0x7ff33006d150 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.970+0000 7ff355c00700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3500cee10 0x7ff350153960 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.970+0000 7ff355c00700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3501611b0 0x7ff3501f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.970+0000 7ff355c00700 1 --2- 172.21.15.89:0/3680283402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff35015f4d0 0x7ff3500c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:44.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.970+0000 7ff355c00700 1 -- 172.21.15.89:0/3680283402 >> 172.21.15.89:0/3680283402 conn(0x7ff35008aff0 msgr2=0x7ff35006bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:44.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.970+0000 7ff355c00700 1 -- 172.21.15.89:0/3680283402 shutdown_connections 2024-04-02T07:36:44.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:44.970+0000 7ff355c00700 1 -- 172.21.15.89:0/3680283402 wait complete. 2024-04-02T07:36:44.990 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:36:45.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.179+0000 7fccc8c00700 1 Processor -- start 2024-04-02T07:36:45.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.179+0000 7fccc8c00700 1 -- start start 2024-04-02T07:36:45.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.180+0000 7fccc8c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc4160f10 0x7fccc40d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:45.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.180+0000 7fccc8c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fccc40cf230 0x7fccc40d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:45.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.180+0000 7fccc8c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc40ceda0 con 0x7fccc4161590 2024-04-02T07:36:45.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.180+0000 7fccc8c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc40d23d0 con 0x7fccc40cf230 2024-04-02T07:36:45.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.180+0000 7fccc8c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc41612f0 con 0x7fccc4160f10 2024-04-02T07:36:45.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.180+0000 7fccc1e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc4160f10 0x7fccc40d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:45.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.180+0000 7fccc1e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc4160f10 0x7fccc40d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:37900/0 (socket says 172.21.15.89:37900) 2024-04-02T07:36:45.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.180+0000 7fccc1400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fccc40cf230 0x7fccc40d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:45.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.180+0000 7fccc1e00700 1 -- 172.21.15.89:0/1945757008 learned_addr learned my addr 172.21.15.89:0/1945757008 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:45.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.180+0000 7fccc2800700 1 --1- 172.21.15.89:0/1945757008 >> v1:172.21.15.113:6789/0 conn(0x7fccc4161590 0x7fccc40b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:37092/0 (socket says 172.21.15.89:37092) 2024-04-02T07:36:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.181+0000 7fccc1e00700 1 -- 172.21.15.89:0/1945757008 >> v2:172.21.15.113:3300/0 conn(0x7fccc40cf230 msgr2=0x7fccc40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.181+0000 7fccc1e00700 1 --2- 172.21.15.89:0/1945757008 >> v2:172.21.15.113:3300/0 conn(0x7fccc40cf230 0x7fccc40d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.181+0000 7fccc1e00700 1 -- 172.21.15.89:0/1945757008 >> v1:172.21.15.113:6789/0 conn(0x7fccc4161590 legacy=0x7fccc40b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.181+0000 7fccc1e00700 1 -- 172.21.15.89:0/1945757008 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc40ce470 con 0x7fccc4160f10 2024-04-02T07:36:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.181+0000 7fccc0a00700 1 -- 172.21.15.89:0/1945757008 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336805940 0 0) 0x7fccc40ceda0 con 0x7fccc4161590 2024-04-02T07:36:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.181+0000 7fccc1e00700 1 --2- 172.21.15.89:0/1945757008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc4160f10 0x7fccc40d18b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fccb40098f0 tx=0x7fccb4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54e8cc4cabd7914e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.181+0000 7fccc0a00700 1 -- 172.21.15.89:0/1945757008 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4011830 con 0x7fccc4160f10 2024-04-02T07:36:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.182+0000 7fccc0a00700 1 -- 172.21.15.89:0/1945757008 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb4011990 con 0x7fccc4160f10 2024-04-02T07:36:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.182+0000 7fccc0a00700 1 -- 172.21.15.89:0/1945757008 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4011d20 con 0x7fccc4160f10 2024-04-02T07:36:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.182+0000 7fccc8c00700 1 -- 172.21.15.89:0/1945757008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc4160f10 msgr2=0x7fccc40d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:45.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.182+0000 7fccc8c00700 1 --2- 172.21.15.89:0/1945757008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc4160f10 0x7fccc40d18b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fccb40098f0 tx=0x7fccb4009c00 comp rx=0 tx=0).stop 2024-04-02T07:36:45.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.182+0000 7fccc8c00700 1 -- 172.21.15.89:0/1945757008 shutdown_connections 2024-04-02T07:36:45.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.182+0000 7fccc8c00700 1 --2- 172.21.15.89:0/1945757008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc4160f10 0x7fccc40d18b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.182+0000 7fccc8c00700 1 --2- 172.21.15.89:0/1945757008 >> v2:172.21.15.113:3300/0 conn(0x7fccc40cf230 0x7fccc40d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.182+0000 7fccc8c00700 1 -- 172.21.15.89:0/1945757008 >> 172.21.15.89:0/1945757008 conn(0x7fccc408aff0 msgr2=0x7fccc40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:45.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.182+0000 7fccc8c00700 1 -- 172.21.15.89:0/1945757008 shutdown_connections 2024-04-02T07:36:45.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.182+0000 7fccc8c00700 1 -- 172.21.15.89:0/1945757008 wait complete. 2024-04-02T07:36:45.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.184+0000 7fccc8c00700 1 Processor -- start 2024-04-02T07:36:45.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.185+0000 7fccc8c00700 1 -- start start 2024-04-02T07:36:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.186+0000 7fccc8c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc40cf230 0x7fccc4151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.186+0000 7fccc8c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc4160f10 0x7fccc4153ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.186+0000 7fccc8c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc4161590 0x7fccc415fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.186+0000 7fccc8c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc415de10 con 0x7fccc4160f10 2024-04-02T07:36:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.186+0000 7fccc8c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc4153ff0 con 0x7fccc40cf230 2024-04-02T07:36:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.186+0000 7fccc8c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc415d470 con 0x7fccc4161590 2024-04-02T07:36:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.186+0000 7fccc1400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc4160f10 0x7fccc4153ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.186+0000 7fccc1400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc4160f10 0x7fccc4153ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48848/0 (socket says 172.21.15.89:48848) 2024-04-02T07:36:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.186+0000 7fccc1400700 1 -- 172.21.15.89:0/3503459158 learned_addr learned my addr 172.21.15.89:0/3503459158 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.186+0000 7fccc2800700 1 --2- 172.21.15.89:0/3503459158 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc4161590 0x7fccc415fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:45.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.186+0000 7fccc1e00700 1 --2- 172.21.15.89:0/3503459158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc40cf230 0x7fccc4151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:45.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.187+0000 7fccc1400700 1 -- 172.21.15.89:0/3503459158 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc4161590 msgr2=0x7fccc415fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:45.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.187+0000 7fccc1400700 1 --2- 172.21.15.89:0/3503459158 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc4161590 0x7fccc415fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.187+0000 7fccc1400700 1 -- 172.21.15.89:0/3503459158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc40cf230 msgr2=0x7fccc4151ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:45.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.187+0000 7fccc1400700 1 --2- 172.21.15.89:0/3503459158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc40cf230 0x7fccc4151ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.187+0000 7fccc1400700 1 -- 172.21.15.89:0/3503459158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb4009550 con 0x7fccc4160f10 2024-04-02T07:36:45.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.187+0000 7fccc1400700 1 --2- 172.21.15.89:0/3503459158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc4160f10 0x7fccc4153ab0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fccb000e870 tx=0x7fccb000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:45.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.187+0000 7fccba000700 1 -- 172.21.15.89:0/3503459158 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb0009590 con 0x7fccc4160f10 2024-04-02T07:36:45.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.187+0000 7fccba000700 1 -- 172.21.15.89:0/3503459158 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb0013680 con 0x7fccc4160f10 2024-04-02T07:36:45.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.187+0000 7fccba000700 1 -- 172.21.15.89:0/3503459158 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb00139f0 con 0x7fccc4160f10 2024-04-02T07:36:45.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.187+0000 7fccc8c00700 1 -- 172.21.15.89:0/3503459158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc415d750 con 0x7fccc4160f10 2024-04-02T07:36:45.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.187+0000 7fccc8c00700 1 -- 172.21.15.89:0/3503459158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc4159040 con 0x7fccc4160f10 2024-04-02T07:36:45.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.188+0000 7fccc8c00700 1 -- 172.21.15.89:0/3503459158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc415df50 con 0x7fccc4160f10 2024-04-02T07:36:45.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.189+0000 7fccba000700 1 -- 172.21.15.89:0/3503459158 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fccb0013b50 con 0x7fccc4160f10 2024-04-02T07:36:45.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.190+0000 7fccba000700 1 --2- 172.21.15.89:0/3503459158 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcca406ac90 0x7fcca406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:45.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.190+0000 7fccba000700 1 -- 172.21.15.89:0/3503459158 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fccb00890f0 con 0x7fccc4160f10 2024-04-02T07:36:45.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.190+0000 7fccc1e00700 1 --2- 172.21.15.89:0/3503459158 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcca406ac90 0x7fcca406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:45.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.191+0000 7fccc1e00700 1 --2- 172.21.15.89:0/3503459158 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcca406ac90 0x7fcca406d150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fccb4014040 tx=0x7fccb4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:45.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.195+0000 7fccba000700 1 -- 172.21.15.89:0/3503459158 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fccc415df50 con 0x7fccc4160f10 2024-04-02T07:36:45.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.441+0000 7fccc8c00700 1 -- 172.21.15.89:0/3503459158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccc415df50 con 0x7fccc4160f10 2024-04-02T07:36:45.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.442+0000 7fccba000700 1 -- 172.21.15.89:0/3503459158 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4731 (secure 0 0 0) 0x7fccc415df50 con 0x7fccc4160f10 2024-04-02T07:36:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:45.444 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2427593953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2427593953},{"type":"v1","addr":"172.21.15.113:6835","nonce":2427593953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/926246816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":926246816},{"type":"v1","addr":"172.21.15.89:6836","nonce":926246816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2701685411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2701685411},{"type":"v1","addr":"172.21.15.89:6837","nonce":2701685411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:35:53.235047+0000","modified":"2024-04-02T07:35:56.245811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6834/3962490902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3962490902},{"type":"v1","addr":"172.21.15.113:6834","nonce":3962490902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-02T07:36:45.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.444+0000 7fccc8c00700 1 -- 172.21.15.89:0/3503459158 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcca406ac90 msgr2=0x7fcca406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:45.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.444+0000 7fccc8c00700 1 --2- 172.21.15.89:0/3503459158 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcca406ac90 0x7fcca406d150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fccb4014040 tx=0x7fccb4008030 comp rx=0 tx=0).stop 2024-04-02T07:36:45.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.445+0000 7fccc8c00700 1 -- 172.21.15.89:0/3503459158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc4160f10 msgr2=0x7fccc4153ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:45.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.445+0000 7fccc8c00700 1 --2- 172.21.15.89:0/3503459158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc4160f10 0x7fccc4153ab0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fccb000e870 tx=0x7fccb000eb80 comp rx=0 tx=0).stop 2024-04-02T07:36:45.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.445+0000 7fccc8c00700 1 -- 172.21.15.89:0/3503459158 shutdown_connections 2024-04-02T07:36:45.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.445+0000 7fccc8c00700 1 --2- 172.21.15.89:0/3503459158 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcca406ac90 0x7fcca406d150 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.445+0000 7fccc8c00700 1 --2- 172.21.15.89:0/3503459158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc40cf230 0x7fccc4151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.445+0000 7fccc8c00700 1 --2- 172.21.15.89:0/3503459158 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc4161590 0x7fccc415fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.445+0000 7fccc8c00700 1 --2- 172.21.15.89:0/3503459158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc4160f10 0x7fccc4153ab0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.445+0000 7fccc8c00700 1 -- 172.21.15.89:0/3503459158 >> 172.21.15.89:0/3503459158 conn(0x7fccc408aff0 msgr2=0x7fccc406ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:45.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.445+0000 7fccc8c00700 1 -- 172.21.15.89:0/3503459158 shutdown_connections 2024-04-02T07:36:45.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.445+0000 7fccc8c00700 1 -- 172.21.15.89:0/3503459158 wait complete. 2024-04-02T07:36:45.448 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2024-04-02T07:36:45.465 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:36:45.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.652+0000 7fe68be00700 1 Processor -- start 2024-04-02T07:36:45.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.652+0000 7fe68be00700 1 -- start start 2024-04-02T07:36:45.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.653+0000 7fe68be00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe6841619d0 0x7fe6841685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:45.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.654+0000 7fe68be00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6840d30d0 con 0x7fe6840ceec0 2024-04-02T07:36:45.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.654+0000 7fe68be00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6840b6840 con 0x7fe68415d640 2024-04-02T07:36:45.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.654+0000 7fe68be00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6840d2b30 con 0x7fe6841619d0 2024-04-02T07:36:45.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.654+0000 7fe688a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe6840ceec0 0x7fe6841612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53140/0 (socket says 172.21.15.89:53140) 2024-04-02T07:36:45.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.654+0000 7fe688a00700 1 -- 172.21.15.89:0/1637818245 learned_addr learned my addr 172.21.15.89:0/1637818245 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:45.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.654+0000 7fe683e00700 1 -- 172.21.15.89:0/1637818245 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416697070 0 0) 0x7fe6840d30d0 con 0x7fe6840ceec0 2024-04-02T07:36:45.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.654+0000 7fe683e00700 1 -- 172.21.15.89:0/1637818245 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe66c0035b0 con 0x7fe6840ceec0 2024-04-02T07:36:45.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.654+0000 7fe689e00700 1 --2- 172.21.15.89:0/1637818245 >> v2:172.21.15.113:3300/0 conn(0x7fe6841619d0 0x7fe6841685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:45.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.654+0000 7fe683e00700 1 -- 172.21.15.89:0/1637818245 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1190332465 0 0) 0x7fe66c0035b0 con 0x7fe6840ceec0 2024-04-02T07:36:45.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.654+0000 7fe683e00700 1 -- 172.21.15.89:0/1637818245 >> v2:172.21.15.113:3300/0 conn(0x7fe6841619d0 msgr2=0x7fe6841685f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:45.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.654+0000 7fe683e00700 1 --2- 172.21.15.89:0/1637818245 >> v2:172.21.15.113:3300/0 conn(0x7fe6841619d0 0x7fe6841685f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.654+0000 7fe683e00700 1 -- 172.21.15.89:0/1637818245 >> v1:172.21.15.113:6789/0 conn(0x7fe68415d640 legacy=0x7fe6840d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:45.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe683e00700 1 -- 172.21.15.89:0/1637818245 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6840d2df0 con 0x7fe6840ceec0 2024-04-02T07:36:45.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe683e00700 1 -- 172.21.15.89:0/1637818245 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fe670001670 con 0x7fe6840ceec0 2024-04-02T07:36:45.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe683e00700 1 -- 172.21.15.89:0/1637818245 >> v1:172.21.15.89:6789/0 conn(0x7fe6840ceec0 legacy=0x7fe6841612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:45.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe683e00700 1 --2- 172.21.15.89:0/1637818245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe66c003a00 0x7fe66c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:45.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe683e00700 1 --2- 172.21.15.89:0/1637818245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe66c004850 0x7fe66c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:45.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe683e00700 1 --2- 172.21.15.89:0/1637818245 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe66c005210 0x7fe66c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:45.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe683e00700 1 -- 172.21.15.89:0/1637818245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6840d2b30 con 0x7fe66c003a00 2024-04-02T07:36:45.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe683e00700 1 -- 172.21.15.89:0/1637818245 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c00c110 con 0x7fe66c004850 2024-04-02T07:36:45.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe683e00700 1 -- 172.21.15.89:0/1637818245 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c00c2d0 con 0x7fe66c005210 2024-04-02T07:36:45.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe689400700 1 --2- 172.21.15.89:0/1637818245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe66c003a00 0x7fe66c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:45.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe688a00700 1 --2- 172.21.15.89:0/1637818245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe66c004850 0x7fe66c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:45.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe689e00700 1 --2- 172.21.15.89:0/1637818245 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe66c005210 0x7fe66c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:45.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe689400700 1 -- 172.21.15.89:0/1637818245 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe66c005210 msgr2=0x7fe66c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:45.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe689400700 1 --2- 172.21.15.89:0/1637818245 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe66c005210 0x7fe66c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe689400700 1 -- 172.21.15.89:0/1637818245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe66c004850 msgr2=0x7fe66c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:45.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe689400700 1 --2- 172.21.15.89:0/1637818245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe66c004850 0x7fe66c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe689400700 1 -- 172.21.15.89:0/1637818245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c0035b0 con 0x7fe66c003a00 2024-04-02T07:36:45.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.655+0000 7fe689e00700 1 -- 172.21.15.89:0/1637818245 reap_dead start 2024-04-02T07:36:45.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.656+0000 7fe689400700 1 --2- 172.21.15.89:0/1637818245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe66c003a00 0x7fe66c003de0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe6780017c0 tx=0x7fe66c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ce2f366fc439ed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:45.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.656+0000 7fe683e00700 1 -- 172.21.15.89:0/1637818245 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6780052d0 con 0x7fe66c003a00 2024-04-02T07:36:45.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.656+0000 7fe683e00700 1 -- 172.21.15.89:0/1637818245 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe678005430 con 0x7fe66c003a00 2024-04-02T07:36:45.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.656+0000 7fe683e00700 1 -- 172.21.15.89:0/1637818245 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6780038c0 con 0x7fe66c003a00 2024-04-02T07:36:45.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.656+0000 7fe68be00700 1 -- 172.21.15.89:0/1637818245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe66c003a00 msgr2=0x7fe66c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:45.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.656+0000 7fe68be00700 1 --2- 172.21.15.89:0/1637818245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe66c003a00 0x7fe66c003de0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe6780017c0 tx=0x7fe66c005880 comp rx=0 tx=0).stop 2024-04-02T07:36:45.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.656+0000 7fe68be00700 1 -- 172.21.15.89:0/1637818245 shutdown_connections 2024-04-02T07:36:45.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.656+0000 7fe68be00700 1 --2- 172.21.15.89:0/1637818245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe66c003a00 0x7fe66c003de0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.656+0000 7fe68be00700 1 -- 172.21.15.89:0/1637818245 >> 172.21.15.89:0/1637818245 conn(0x7fe68408aff0 msgr2=0x7fe6840d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:45.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.656+0000 7fe68be00700 1 -- 172.21.15.89:0/1637818245 shutdown_connections 2024-04-02T07:36:45.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.657+0000 7fe68be00700 1 -- 172.21.15.89:0/1637818245 wait complete. 2024-04-02T07:36:45.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.658+0000 7fe68be00700 1 Processor -- start 2024-04-02T07:36:45.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.658+0000 7fe68be00700 1 -- start start 2024-04-02T07:36:45.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.659+0000 7fe68be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe66c003a00 0x7fe68415e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:45.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.659+0000 7fe68be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6840c9a20 0x7fe684155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:45.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.659+0000 7fe68be00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe68415fb70 0x7fe6840c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:45.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.659+0000 7fe68be00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6840c9ea0 con 0x7fe6840c9a20 2024-04-02T07:36:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.659+0000 7fe68be00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe684156200 con 0x7fe68415fb70 2024-04-02T07:36:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.659+0000 7fe68be00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68415ead0 con 0x7fe66c003a00 2024-04-02T07:36:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.659+0000 7fe688a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6840c9a20 0x7fe684155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.659+0000 7fe688a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6840c9a20 0x7fe684155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48870/0 (socket says 172.21.15.89:48870) 2024-04-02T07:36:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.659+0000 7fe688a00700 1 -- 172.21.15.89:0/2955042803 learned_addr learned my addr 172.21.15.89:0/2955042803 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.660+0000 7fe689400700 1 --2- 172.21.15.89:0/2955042803 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe66c003a00 0x7fe68415e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.660+0000 7fe689e00700 1 --2- 172.21.15.89:0/2955042803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe68415fb70 0x7fe6840c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.660+0000 7fe688a00700 1 -- 172.21.15.89:0/2955042803 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe66c003a00 msgr2=0x7fe68415e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.660+0000 7fe688a00700 1 --2- 172.21.15.89:0/2955042803 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe66c003a00 0x7fe68415e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.660+0000 7fe688a00700 1 -- 172.21.15.89:0/2955042803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe68415fb70 msgr2=0x7fe6840c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.660+0000 7fe688a00700 1 --2- 172.21.15.89:0/2955042803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe68415fb70 0x7fe6840c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.660+0000 7fe688a00700 1 -- 172.21.15.89:0/2955042803 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68415edb0 con 0x7fe6840c9a20 2024-04-02T07:36:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.660+0000 7fe688a00700 1 --2- 172.21.15.89:0/2955042803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6840c9a20 0x7fe684155cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe6700080e0 tx=0x7fe670005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.660+0000 7fe681600700 1 -- 172.21.15.89:0/2955042803 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670010700 con 0x7fe6840c9a20 2024-04-02T07:36:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.660+0000 7fe681600700 1 -- 172.21.15.89:0/2955042803 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67000e050 con 0x7fe6840c9a20 2024-04-02T07:36:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.660+0000 7fe68be00700 1 -- 172.21.15.89:0/2955042803 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68415f090 con 0x7fe6840c9a20 2024-04-02T07:36:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.660+0000 7fe681600700 1 -- 172.21.15.89:0/2955042803 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670010ac0 con 0x7fe6840c9a20 2024-04-02T07:36:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.661+0000 7fe68be00700 1 -- 172.21.15.89:0/2955042803 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68415f650 con 0x7fe6840c9a20 2024-04-02T07:36:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.662+0000 7fe68be00700 1 -- 172.21.15.89:0/2955042803 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe684067720 con 0x7fe6840c9a20 2024-04-02T07:36:45.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.662+0000 7fe681600700 1 -- 172.21.15.89:0/2955042803 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe670010c20 con 0x7fe6840c9a20 2024-04-02T07:36:45.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.663+0000 7fe681600700 1 --2- 172.21.15.89:0/2955042803 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe66406ac90 0x7fe66406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:45.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.663+0000 7fe681600700 1 -- 172.21.15.89:0/2955042803 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe6700888b0 con 0x7fe6840c9a20 2024-04-02T07:36:45.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.664+0000 7fe689400700 1 --2- 172.21.15.89:0/2955042803 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe66406ac90 0x7fe66406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:45.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.664+0000 7fe689400700 1 --2- 172.21.15.89:0/2955042803 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe66406ac90 0x7fe66406d150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe678015040 tx=0x7fe678009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:45.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.668+0000 7fe681600700 1 -- 172.21.15.89:0/2955042803 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6700539d0 con 0x7fe6840c9a20 2024-04-02T07:36:45.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.869+0000 7fe68be00700 1 -- 172.21.15.89:0/2955042803 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6840b5c70 con 0x7fe6840c9a20 2024-04-02T07:36:45.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.870+0000 7fe681600700 1 -- 172.21.15.89:0/2955042803 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7fe67001a020 con 0x7fe6840c9a20 2024-04-02T07:36:45.872 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:45.872 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:36:36.860889+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:36:45.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.872+0000 7fe68be00700 1 -- 172.21.15.89:0/2955042803 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe66406ac90 msgr2=0x7fe66406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:45.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.872+0000 7fe68be00700 1 --2- 172.21.15.89:0/2955042803 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe66406ac90 0x7fe66406d150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe678015040 tx=0x7fe678009030 comp rx=0 tx=0).stop 2024-04-02T07:36:45.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.872+0000 7fe68be00700 1 -- 172.21.15.89:0/2955042803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6840c9a20 msgr2=0x7fe684155cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:45.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.872+0000 7fe68be00700 1 --2- 172.21.15.89:0/2955042803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6840c9a20 0x7fe684155cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe6700080e0 tx=0x7fe670005f90 comp rx=0 tx=0).stop 2024-04-02T07:36:45.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.872+0000 7fe68be00700 1 -- 172.21.15.89:0/2955042803 shutdown_connections 2024-04-02T07:36:45.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.872+0000 7fe68be00700 1 --2- 172.21.15.89:0/2955042803 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe66406ac90 0x7fe66406d150 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.872+0000 7fe68be00700 1 --2- 172.21.15.89:0/2955042803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe68415fb70 0x7fe6840c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.872+0000 7fe68be00700 1 --2- 172.21.15.89:0/2955042803 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe66c003a00 0x7fe68415e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.872+0000 7fe68be00700 1 --2- 172.21.15.89:0/2955042803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6840c9a20 0x7fe684155cc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:45.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.872+0000 7fe68be00700 1 -- 172.21.15.89:0/2955042803 >> 172.21.15.89:0/2955042803 conn(0x7fe68408aff0 msgr2=0x7fe6840b6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:45.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.872+0000 7fe68be00700 1 -- 172.21.15.89:0/2955042803 shutdown_connections 2024-04-02T07:36:45.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:45.872+0000 7fe68be00700 1 -- 172.21.15.89:0/2955042803 wait complete. 2024-04-02T07:36:45.893 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T07:36:45.893 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:36:46.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.081+0000 7f13cd600700 1 Processor -- start 2024-04-02T07:36:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.082+0000 7f13cd600700 1 -- start start 2024-04-02T07:36:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.083+0000 7f13cd600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13c80d18b0 0x7f13c80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.083+0000 7f13cd600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f13c8161590 0x7f13c8161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.083+0000 7f13cd600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c80ceda0 con 0x7f13c8160f10 2024-04-02T07:36:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.083+0000 7f13cd600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c80d1d30 con 0x7f13c8161590 2024-04-02T07:36:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.083+0000 7f13cd600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c8161f60 con 0x7f13c80d18b0 2024-04-02T07:36:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.083+0000 7f13c7400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f13c8161590 0x7f13c8161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.083+0000 7f13c6000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13c80d18b0 0x7f13c80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.083+0000 7f13c6a00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f13c8160f10 0x7f13c80d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:37122/0 (socket says 172.21.15.89:37122) 2024-04-02T07:36:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.083+0000 7f13c6a00700 1 -- 172.21.15.89:0/4043489660 learned_addr learned my addr 172.21.15.89:0/4043489660 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.084+0000 7f13c6000700 1 -- 172.21.15.89:0/4043489660 >> v2:172.21.15.113:3300/0 conn(0x7f13c8161590 msgr2=0x7f13c8161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.084+0000 7f13c6000700 1 --2- 172.21.15.89:0/4043489660 >> v2:172.21.15.113:3300/0 conn(0x7f13c8161590 0x7f13c8161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.084+0000 7f13c6000700 1 -- 172.21.15.89:0/4043489660 >> v1:172.21.15.113:6789/0 conn(0x7f13c8160f10 legacy=0x7f13c80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.084+0000 7f13c6000700 1 -- 172.21.15.89:0/4043489660 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c81612f0 con 0x7f13c80d18b0 2024-04-02T07:36:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.084+0000 7f13c6000700 1 --2- 172.21.15.89:0/4043489660 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13c80d18b0 0x7f13c80cf230 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009870 tx=0x7f13b4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4662ed22f82aab5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.084+0000 7f13c5600700 1 -- 172.21.15.89:0/4043489660 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4011830 con 0x7f13c80d18b0 2024-04-02T07:36:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.084+0000 7f13c5600700 1 -- 172.21.15.89:0/4043489660 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b4011990 con 0x7f13c80d18b0 2024-04-02T07:36:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.084+0000 7f13c5600700 1 -- 172.21.15.89:0/4043489660 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4011d20 con 0x7f13c80d18b0 2024-04-02T07:36:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.085+0000 7f13cd600700 1 -- 172.21.15.89:0/4043489660 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13c80d18b0 msgr2=0x7f13c80cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.085+0000 7f13cd600700 1 --2- 172.21.15.89:0/4043489660 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13c80d18b0 0x7f13c80cf230 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009870 tx=0x7f13b4009b80 comp rx=0 tx=0).stop 2024-04-02T07:36:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.085+0000 7f13cd600700 1 -- 172.21.15.89:0/4043489660 shutdown_connections 2024-04-02T07:36:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.085+0000 7f13cd600700 1 --2- 172.21.15.89:0/4043489660 >> v2:172.21.15.113:3300/0 conn(0x7f13c8161590 0x7f13c8161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.085+0000 7f13cd600700 1 --2- 172.21.15.89:0/4043489660 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13c80d18b0 0x7f13c80cf230 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.085+0000 7f13cd600700 1 -- 172.21.15.89:0/4043489660 >> 172.21.15.89:0/4043489660 conn(0x7f13c808aff0 msgr2=0x7f13c80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.085+0000 7f13cd600700 1 -- 172.21.15.89:0/4043489660 shutdown_connections 2024-04-02T07:36:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.085+0000 7f13cd600700 1 -- 172.21.15.89:0/4043489660 wait complete. 2024-04-02T07:36:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.086+0000 7f13cd600700 1 Processor -- start 2024-04-02T07:36:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.087+0000 7f13cd600700 1 -- start start 2024-04-02T07:36:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.087+0000 7f13cd600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c80d18b0 0x7f13c814cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.087+0000 7f13cd600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13c8160f10 0x7f13c814e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.087+0000 7f13cd600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13c8161590 0x7f13c814b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.087+0000 7f13cd600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c814e080 con 0x7f13c8161590 2024-04-02T07:36:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.087+0000 7f13cd600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c809d4a0 con 0x7f13c80d18b0 2024-04-02T07:36:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.087+0000 7f13cd600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c814d400 con 0x7f13c8160f10 2024-04-02T07:36:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.088+0000 7f13c7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13c8161590 0x7f13c814b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.088+0000 7f13c7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13c8161590 0x7f13c814b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48880/0 (socket says 172.21.15.89:48880) 2024-04-02T07:36:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.088+0000 7f13c7400700 1 -- 172.21.15.89:0/1142037078 learned_addr learned my addr 172.21.15.89:0/1142037078 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.088+0000 7f13c6000700 1 --2- 172.21.15.89:0/1142037078 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13c8160f10 0x7f13c814e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.088+0000 7f13c6a00700 1 --2- 172.21.15.89:0/1142037078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c80d18b0 0x7f13c814cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.088+0000 7f13c7400700 1 -- 172.21.15.89:0/1142037078 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13c8160f10 msgr2=0x7f13c814e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.088+0000 7f13c7400700 1 --2- 172.21.15.89:0/1142037078 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13c8160f10 0x7f13c814e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.088+0000 7f13c7400700 1 -- 172.21.15.89:0/1142037078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c80d18b0 msgr2=0x7f13c814cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:46.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.088+0000 7f13c7400700 1 --2- 172.21.15.89:0/1142037078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c80d18b0 0x7f13c814cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.088+0000 7f13c7400700 1 -- 172.21.15.89:0/1142037078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b40094a0 con 0x7f13c8161590 2024-04-02T07:36:46.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.088+0000 7f13c7400700 1 --2- 172.21.15.89:0/1142037078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13c8161590 0x7f13c814b870 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f13c000efc0 tx=0x7f13c000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:46.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.089+0000 7f13bea00700 1 -- 172.21.15.89:0/1142037078 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c0009ba0 con 0x7f13c8161590 2024-04-02T07:36:46.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.089+0000 7f13bea00700 1 -- 172.21.15.89:0/1142037078 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c000f040 con 0x7f13c8161590 2024-04-02T07:36:46.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.089+0000 7f13cd600700 1 -- 172.21.15.89:0/1142037078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c814d6e0 con 0x7f13c8161590 2024-04-02T07:36:46.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.089+0000 7f13bea00700 1 -- 172.21.15.89:0/1142037078 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c0014480 con 0x7f13c8161590 2024-04-02T07:36:46.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.089+0000 7f13cd600700 1 -- 172.21.15.89:0/1142037078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c814dca0 con 0x7f13c8161590 2024-04-02T07:36:46.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.090+0000 7f13cd600700 1 -- 172.21.15.89:0/1142037078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c8067720 con 0x7f13c8161590 2024-04-02T07:36:46.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.091+0000 7f13bea00700 1 -- 172.21.15.89:0/1142037078 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f13c00145e0 con 0x7f13c8161590 2024-04-02T07:36:46.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.092+0000 7f13bea00700 1 --2- 172.21.15.89:0/1142037078 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f13a806ac90 0x7f13a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:46.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.092+0000 7f13bea00700 1 -- 172.21.15.89:0/1142037078 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f13c0089b60 con 0x7f13c8161590 2024-04-02T07:36:46.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.092+0000 7f13c6a00700 1 --2- 172.21.15.89:0/1142037078 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f13a806ac90 0x7f13a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:46.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.093+0000 7f13c6a00700 1 --2- 172.21.15.89:0/1142037078 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f13a806ac90 0x7f13a806d150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f13b8005910 tx=0x7f13b8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:46.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.097+0000 7f13bea00700 1 -- 172.21.15.89:0/1142037078 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f13c0054db0 con 0x7f13c8161590 2024-04-02T07:36:46.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.339+0000 7f13cd600700 1 -- 172.21.15.89:0/1142037078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13c80ccb70 con 0x7f13c8161590 2024-04-02T07:36:46.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.339+0000 7f13bea00700 1 -- 172.21.15.89:0/1142037078 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4731 (secure 0 0 0) 0x7f13c0018090 con 0x7f13c8161590 2024-04-02T07:36:46.341 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:46.342 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2427593953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2427593953},{"type":"v1","addr":"172.21.15.113:6835","nonce":2427593953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/926246816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":926246816},{"type":"v1","addr":"172.21.15.89:6836","nonce":926246816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2701685411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2701685411},{"type":"v1","addr":"172.21.15.89:6837","nonce":2701685411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:35:53.235047+0000","modified":"2024-04-02T07:35:56.245811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6834/3962490902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3962490902},{"type":"v1","addr":"172.21.15.113:6834","nonce":3962490902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-02T07:36:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.342+0000 7f13cd600700 1 -- 172.21.15.89:0/1142037078 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f13a806ac90 msgr2=0x7f13a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.342+0000 7f13cd600700 1 --2- 172.21.15.89:0/1142037078 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f13a806ac90 0x7f13a806d150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f13b8005910 tx=0x7f13b8009040 comp rx=0 tx=0).stop 2024-04-02T07:36:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.342+0000 7f13cd600700 1 -- 172.21.15.89:0/1142037078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13c8161590 msgr2=0x7f13c814b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.342+0000 7f13cd600700 1 --2- 172.21.15.89:0/1142037078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13c8161590 0x7f13c814b870 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f13c000efc0 tx=0x7f13c000c3a0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.342+0000 7f13cd600700 1 -- 172.21.15.89:0/1142037078 shutdown_connections 2024-04-02T07:36:46.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.342+0000 7f13cd600700 1 --2- 172.21.15.89:0/1142037078 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f13a806ac90 0x7f13a806d150 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.342+0000 7f13cd600700 1 --2- 172.21.15.89:0/1142037078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c80d18b0 0x7f13c814cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.342+0000 7f13cd600700 1 --2- 172.21.15.89:0/1142037078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13c8161590 0x7f13c814b870 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.342+0000 7f13cd600700 1 --2- 172.21.15.89:0/1142037078 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13c8160f10 0x7f13c814e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.342+0000 7f13cd600700 1 -- 172.21.15.89:0/1142037078 >> 172.21.15.89:0/1142037078 conn(0x7f13c808aff0 msgr2=0x7f13c80ca980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:46.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.342+0000 7f13cd600700 1 -- 172.21.15.89:0/1142037078 shutdown_connections 2024-04-02T07:36:46.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.342+0000 7f13cd600700 1 -- 172.21.15.89:0/1142037078 wait complete. 2024-04-02T07:36:46.345 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2024-04-02T07:36:46.362 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:36:46.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.551+0000 7fcf5e000700 1 Processor -- start 2024-04-02T07:36:46.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.551+0000 7fcf5e000700 1 -- start start 2024-04-02T07:36:46.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.552+0000 7fcf5e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf580d18b0 0x7fcf580cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:46.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.553+0000 7fcf5e000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcf58161590 0x7fcf58161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:46.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.553+0000 7fcf5e000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf580ceda0 con 0x7fcf58160f10 2024-04-02T07:36:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.553+0000 7fcf5e000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf580d1d30 con 0x7fcf58161590 2024-04-02T07:36:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.553+0000 7fcf5e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf58161f60 con 0x7fcf580d18b0 2024-04-02T07:36:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.553+0000 7fcf57e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcf58161590 0x7fcf58161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.553+0000 7fcf57e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcf58161590 0x7fcf58161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48896/0 (socket says 172.21.15.89:48896) 2024-04-02T07:36:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.553+0000 7fcf57e00700 1 -- 172.21.15.89:0/1978772060 learned_addr learned my addr 172.21.15.89:0/1978772060 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.553+0000 7fcf57400700 1 --1- 172.21.15.89:0/1978772060 >> v1:172.21.15.89:6789/0 conn(0x7fcf58160f10 0x7fcf580d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53148/0 (socket says 172.21.15.89:53148) 2024-04-02T07:36:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.553+0000 7fcf57e00700 1 -- 172.21.15.89:0/1978772060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf580d18b0 msgr2=0x7fcf580cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.553+0000 7fcf57e00700 1 --2- 172.21.15.89:0/1978772060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf580d18b0 0x7fcf580cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.553+0000 7fcf57e00700 1 -- 172.21.15.89:0/1978772060 >> v1:172.21.15.89:6789/0 conn(0x7fcf58160f10 legacy=0x7fcf580d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:46.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.553+0000 7fcf57e00700 1 -- 172.21.15.89:0/1978772060 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf581612f0 con 0x7fcf58161590 2024-04-02T07:36:46.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.554+0000 7fcf57e00700 1 --2- 172.21.15.89:0/1978772060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf58161590 0x7fcf58161a20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009e80 tx=0x7fcf5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b76788e101d030 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:46.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.554+0000 7fcf56000700 1 -- 172.21.15.89:0/1978772060 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5000e070 con 0x7fcf58161590 2024-04-02T07:36:46.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.554+0000 7fcf56000700 1 -- 172.21.15.89:0/1978772060 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf5000b980 con 0x7fcf58161590 2024-04-02T07:36:46.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.554+0000 7fcf56000700 1 -- 172.21.15.89:0/1978772060 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5000bd30 con 0x7fcf58161590 2024-04-02T07:36:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.554+0000 7fcf5e000700 1 -- 172.21.15.89:0/1978772060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf58161590 msgr2=0x7fcf58161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.554+0000 7fcf5e000700 1 --2- 172.21.15.89:0/1978772060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf58161590 0x7fcf58161a20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009e80 tx=0x7fcf5000b330 comp rx=0 tx=0).stop 2024-04-02T07:36:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.554+0000 7fcf5e000700 1 -- 172.21.15.89:0/1978772060 shutdown_connections 2024-04-02T07:36:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.554+0000 7fcf5e000700 1 --2- 172.21.15.89:0/1978772060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf58161590 0x7fcf58161a20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.554+0000 7fcf5e000700 1 --2- 172.21.15.89:0/1978772060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf580d18b0 0x7fcf580cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.554+0000 7fcf5e000700 1 -- 172.21.15.89:0/1978772060 >> 172.21.15.89:0/1978772060 conn(0x7fcf5808aff0 msgr2=0x7fcf580d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.554+0000 7fcf5e000700 1 -- 172.21.15.89:0/1978772060 shutdown_connections 2024-04-02T07:36:46.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.554+0000 7fcf5e000700 1 -- 172.21.15.89:0/1978772060 wait complete. 2024-04-02T07:36:46.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.556+0000 7fcf5e000700 1 Processor -- start 2024-04-02T07:36:46.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.556+0000 7fcf5e000700 1 -- start start 2024-04-02T07:36:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.557+0000 7fcf5e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf580d18b0 0x7fcf58155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.557+0000 7fcf5e000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf58160f10 0x7fcf58147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.557+0000 7fcf5e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf58161590 0x7fcf580cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.557+0000 7fcf5e000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf58145bc0 con 0x7fcf580d18b0 2024-04-02T07:36:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.557+0000 7fcf5e000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf58145960 con 0x7fcf58160f10 2024-04-02T07:36:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.557+0000 7fcf5e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf58148190 con 0x7fcf58161590 2024-04-02T07:36:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.557+0000 7fcf57400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf580d18b0 0x7fcf58155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.557+0000 7fcf57400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf580d18b0 0x7fcf58155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48908/0 (socket says 172.21.15.89:48908) 2024-04-02T07:36:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.557+0000 7fcf57400700 1 -- 172.21.15.89:0/887335278 learned_addr learned my addr 172.21.15.89:0/887335278 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.557+0000 7fcf56a00700 1 --2- 172.21.15.89:0/887335278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf58160f10 0x7fcf58147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.558+0000 7fcf57e00700 1 --2- 172.21.15.89:0/887335278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf58161590 0x7fcf580cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.558+0000 7fcf57400700 1 -- 172.21.15.89:0/887335278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf58161590 msgr2=0x7fcf580cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.558+0000 7fcf57400700 1 --2- 172.21.15.89:0/887335278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf58161590 0x7fcf580cae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.558+0000 7fcf57400700 1 -- 172.21.15.89:0/887335278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf58160f10 msgr2=0x7fcf58147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.558+0000 7fcf57400700 1 --2- 172.21.15.89:0/887335278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf58160f10 0x7fcf58147c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.558+0000 7fcf57400700 1 -- 172.21.15.89:0/887335278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf50009b30 con 0x7fcf580d18b0 2024-04-02T07:36:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.558+0000 7fcf57400700 1 --2- 172.21.15.89:0/887335278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf580d18b0 0x7fcf58155c90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcf4800c530 tx=0x7fcf4800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.558+0000 7fcf4f400700 1 -- 172.21.15.89:0/887335278 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4800cea0 con 0x7fcf580d18b0 2024-04-02T07:36:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.558+0000 7fcf4f400700 1 -- 172.21.15.89:0/887335278 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf48005cb0 con 0x7fcf580d18b0 2024-04-02T07:36:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.558+0000 7fcf5e000700 1 -- 172.21.15.89:0/887335278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf581483b0 con 0x7fcf580d18b0 2024-04-02T07:36:46.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.558+0000 7fcf4f400700 1 -- 172.21.15.89:0/887335278 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf480072d0 con 0x7fcf580d18b0 2024-04-02T07:36:46.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.558+0000 7fcf5e000700 1 -- 172.21.15.89:0/887335278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf58148970 con 0x7fcf580d18b0 2024-04-02T07:36:46.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.559+0000 7fcf5e000700 1 -- 172.21.15.89:0/887335278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf58067720 con 0x7fcf580d18b0 2024-04-02T07:36:46.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.560+0000 7fcf4f400700 1 -- 172.21.15.89:0/887335278 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcf48007430 con 0x7fcf580d18b0 2024-04-02T07:36:46.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.561+0000 7fcf4f400700 1 --2- 172.21.15.89:0/887335278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf3806ac90 0x7fcf3806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:46.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.561+0000 7fcf4f400700 1 -- 172.21.15.89:0/887335278 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcf48088520 con 0x7fcf580d18b0 2024-04-02T07:36:46.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.561+0000 7fcf56a00700 1 --2- 172.21.15.89:0/887335278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf3806ac90 0x7fcf3806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:46.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.562+0000 7fcf56a00700 1 --2- 172.21.15.89:0/887335278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf3806ac90 0x7fcf3806d150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcf580a15e0 tx=0x7fcf44009110 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:46.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.565+0000 7fcf4f400700 1 -- 172.21.15.89:0/887335278 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf48053780 con 0x7fcf580d18b0 2024-04-02T07:36:46.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.811+0000 7fcf5e000700 1 -- 172.21.15.89:0/887335278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf58148b30 con 0x7fcf580d18b0 2024-04-02T07:36:46.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.812+0000 7fcf4f400700 1 -- 172.21.15.89:0/887335278 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4731 (secure 0 0 0) 0x7fcf48056da0 con 0x7fcf580d18b0 2024-04-02T07:36:46.814 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:46.814 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2427593953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2427593953},{"type":"v1","addr":"172.21.15.113:6835","nonce":2427593953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/926246816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":926246816},{"type":"v1","addr":"172.21.15.89:6836","nonce":926246816}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2701685411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2701685411},{"type":"v1","addr":"172.21.15.89:6837","nonce":2701685411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:35:53.235047+0000","modified":"2024-04-02T07:35:56.245811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6834/3962490902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3962490902},{"type":"v1","addr":"172.21.15.113:6834","nonce":3962490902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-02T07:36:46.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.814+0000 7fcf5e000700 1 -- 172.21.15.89:0/887335278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf3806ac90 msgr2=0x7fcf3806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:46.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.814+0000 7fcf5e000700 1 --2- 172.21.15.89:0/887335278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf3806ac90 0x7fcf3806d150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcf580a15e0 tx=0x7fcf44009110 comp rx=0 tx=0).stop 2024-04-02T07:36:46.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.814+0000 7fcf5e000700 1 -- 172.21.15.89:0/887335278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf580d18b0 msgr2=0x7fcf58155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:46.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.814+0000 7fcf5e000700 1 --2- 172.21.15.89:0/887335278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf580d18b0 0x7fcf58155c90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcf4800c530 tx=0x7fcf4800c840 comp rx=0 tx=0).stop 2024-04-02T07:36:46.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.814+0000 7fcf5e000700 1 -- 172.21.15.89:0/887335278 shutdown_connections 2024-04-02T07:36:46.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.814+0000 7fcf5e000700 1 --2- 172.21.15.89:0/887335278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf3806ac90 0x7fcf3806d150 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.814+0000 7fcf5e000700 1 --2- 172.21.15.89:0/887335278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf58161590 0x7fcf580cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.814+0000 7fcf5e000700 1 --2- 172.21.15.89:0/887335278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf580d18b0 0x7fcf58155c90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.814+0000 7fcf5e000700 1 --2- 172.21.15.89:0/887335278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf58160f10 0x7fcf58147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:46.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.814+0000 7fcf5e000700 1 -- 172.21.15.89:0/887335278 >> 172.21.15.89:0/887335278 conn(0x7fcf5808aff0 msgr2=0x7fcf580b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:46.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.814+0000 7fcf5e000700 1 -- 172.21.15.89:0/887335278 shutdown_connections 2024-04-02T07:36:46.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:46.814+0000 7fcf5e000700 1 -- 172.21.15.89:0/887335278 wait complete. 2024-04-02T07:36:46.817 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2024-04-02T07:36:46.834 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:36:47.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.025+0000 7f6f1be00700 1 Processor -- start 2024-04-02T07:36:47.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.025+0000 7f6f1be00700 1 -- start start 2024-04-02T07:36:47.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.026+0000 7f6f1be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6f14160f10 0x7f6f140d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.026+0000 7f6f1be00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f140ceda0 con 0x7f6f140cf230 2024-04-02T07:36:47.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.026+0000 7f6f1be00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f141612f0 con 0x7f6f14161590 2024-04-02T07:36:47.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.026+0000 7f6f1be00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f140ce470 con 0x7f6f14160f10 2024-04-02T07:36:47.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.026+0000 7f6f19400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6f14160f10 0x7f6f140d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.026+0000 7f6f19400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6f14160f10 0x7f6f140d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48922/0 (socket says 172.21.15.89:48922) 2024-04-02T07:36:47.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.026+0000 7f6f19400700 1 -- 172.21.15.89:0/803242970 learned_addr learned my addr 172.21.15.89:0/803242970 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:47.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.026+0000 7f6f18a00700 1 --1- 172.21.15.89:0/803242970 >> v1:172.21.15.89:6789/0 conn(0x7f6f140cf230 0x7f6f140d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53156/0 (socket says 172.21.15.89:53156) 2024-04-02T07:36:47.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.027+0000 7f6f19400700 1 -- 172.21.15.89:0/803242970 >> v1:172.21.15.113:6789/0 conn(0x7f6f14161590 legacy=0x7f6f140b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:47.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.027+0000 7f6f19400700 1 -- 172.21.15.89:0/803242970 >> v1:172.21.15.89:6789/0 conn(0x7f6f140cf230 legacy=0x7f6f140d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.027+0000 7f6f19400700 1 -- 172.21.15.89:0/803242970 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f140ce750 con 0x7f6f14160f10 2024-04-02T07:36:47.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.027+0000 7f6f13e00700 1 -- 172.21.15.89:0/803242970 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919138946 0 0) 0x7f6f140ceda0 con 0x7f6f140cf230 2024-04-02T07:36:47.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.027+0000 7f6f19400700 1 --2- 172.21.15.89:0/803242970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f14160f10 0x7f6f140d18b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6f080098f0 tx=0x7f6f08009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79dc80f40fbbe7c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:47.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.027+0000 7f6f13e00700 1 -- 172.21.15.89:0/803242970 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08004230 con 0x7f6f14160f10 2024-04-02T07:36:47.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.027+0000 7f6f13e00700 1 -- 172.21.15.89:0/803242970 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f08004390 con 0x7f6f14160f10 2024-04-02T07:36:47.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.027+0000 7f6f13e00700 1 -- 172.21.15.89:0/803242970 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f080036a0 con 0x7f6f14160f10 2024-04-02T07:36:47.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.028+0000 7f6f1be00700 1 -- 172.21.15.89:0/803242970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f14160f10 msgr2=0x7f6f140d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.028+0000 7f6f1be00700 1 --2- 172.21.15.89:0/803242970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f14160f10 0x7f6f140d18b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6f080098f0 tx=0x7f6f08009c00 comp rx=0 tx=0).stop 2024-04-02T07:36:47.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.028+0000 7f6f1be00700 1 -- 172.21.15.89:0/803242970 shutdown_connections 2024-04-02T07:36:47.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.028+0000 7f6f1be00700 1 --2- 172.21.15.89:0/803242970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f14160f10 0x7f6f140d18b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.028+0000 7f6f1be00700 1 -- 172.21.15.89:0/803242970 >> 172.21.15.89:0/803242970 conn(0x7f6f1408aff0 msgr2=0x7f6f140d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:47.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.028+0000 7f6f1be00700 1 -- 172.21.15.89:0/803242970 shutdown_connections 2024-04-02T07:36:47.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.028+0000 7f6f1be00700 1 -- 172.21.15.89:0/803242970 wait complete. 2024-04-02T07:36:47.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.030+0000 7f6f1be00700 1 Processor -- start 2024-04-02T07:36:47.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.030+0000 7f6f1be00700 1 -- start start 2024-04-02T07:36:47.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f1be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f140cf230 0x7f6f1414b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f1be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f14160f10 0x7f6f1415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f1be00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f14161590 0x7f6f1415f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f1be00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1415e9b0 con 0x7f6f14160f10 2024-04-02T07:36:47.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f1be00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1415de30 con 0x7f6f14161590 2024-04-02T07:36:47.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f1be00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1415e110 con 0x7f6f140cf230 2024-04-02T07:36:47.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f18a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f14160f10 0x7f6f1415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f18a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f14160f10 0x7f6f1415ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48938/0 (socket says 172.21.15.89:48938) 2024-04-02T07:36:47.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f18a00700 1 -- 172.21.15.89:0/1511125035 learned_addr learned my addr 172.21.15.89:0/1511125035 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:47.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f19e00700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f14161590 0x7f6f1415f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f19400700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f140cf230 0x7f6f1414b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f18a00700 1 -- 172.21.15.89:0/1511125035 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f140cf230 msgr2=0x7f6f1414b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f18a00700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f140cf230 0x7f6f1414b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f18a00700 1 -- 172.21.15.89:0/1511125035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f14161590 msgr2=0x7f6f1415f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f18a00700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f14161590 0x7f6f1415f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f18a00700 1 -- 172.21.15.89:0/1511125035 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f08009550 con 0x7f6f14160f10 2024-04-02T07:36:47.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.031+0000 7f6f19e00700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f14161590 0x7f6f1415f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:47.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.032+0000 7f6f18a00700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f14160f10 0x7f6f1415ef90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6f00001eb0 tx=0x7f6f0000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:47.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.032+0000 7f6f11600700 1 -- 172.21.15.89:0/1511125035 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00004c80 con 0x7f6f14160f10 2024-04-02T07:36:47.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.032+0000 7f6f11600700 1 -- 172.21.15.89:0/1511125035 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f00004de0 con 0x7f6f14160f10 2024-04-02T07:36:47.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.032+0000 7f6f1be00700 1 -- 172.21.15.89:0/1511125035 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f1415e3f0 con 0x7f6f14160f10 2024-04-02T07:36:47.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.032+0000 7f6f11600700 1 -- 172.21.15.89:0/1511125035 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00007310 con 0x7f6f14160f10 2024-04-02T07:36:47.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.032+0000 7f6f1be00700 1 -- 172.21.15.89:0/1511125035 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f1414be50 con 0x7f6f14160f10 2024-04-02T07:36:47.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.033+0000 7f6f1be00700 1 -- 172.21.15.89:0/1511125035 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f14067720 con 0x7f6f14160f10 2024-04-02T07:36:47.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.034+0000 7f6f11600700 1 -- 172.21.15.89:0/1511125035 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6f00007470 con 0x7f6f14160f10 2024-04-02T07:36:47.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.035+0000 7f6f11600700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ef406ac90 0x7f6ef406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.035+0000 7f6f11600700 1 -- 172.21.15.89:0/1511125035 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6f000890e0 con 0x7f6f14160f10 2024-04-02T07:36:47.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.035+0000 7f6f19400700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ef406ac90 0x7f6ef406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.035+0000 7f6f19400700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ef406ac90 0x7f6ef406d150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009130 tx=0x7f6f08008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:47.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.039+0000 7f6f11600700 1 -- 172.21.15.89:0/1511125035 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6f00054200 con 0x7f6f14160f10 2024-04-02T07:36:47.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.242+0000 7f6f1be00700 1 -- 172.21.15.89:0/1511125035 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f1408b760 con 0x7f6f14160f10 2024-04-02T07:36:47.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.243+0000 7f6f11600700 1 -- 172.21.15.89:0/1511125035 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7f6f00057820 con 0x7f6f14160f10 2024-04-02T07:36:47.245 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:47.245 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:36:36.860889+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:35:51.296561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T07:35:52.232882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:36:47.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.246+0000 7f6f1be00700 1 -- 172.21.15.89:0/1511125035 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ef406ac90 msgr2=0x7f6ef406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.246+0000 7f6f1be00700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ef406ac90 0x7f6ef406d150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009130 tx=0x7f6f08008030 comp rx=0 tx=0).stop 2024-04-02T07:36:47.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.246+0000 7f6f1be00700 1 -- 172.21.15.89:0/1511125035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f14160f10 msgr2=0x7f6f1415ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.246+0000 7f6f1be00700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f14160f10 0x7f6f1415ef90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6f00001eb0 tx=0x7f6f0000c8b0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.246+0000 7f6f1be00700 1 -- 172.21.15.89:0/1511125035 shutdown_connections 2024-04-02T07:36:47.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.246+0000 7f6f1be00700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ef406ac90 0x7f6ef406d150 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.246+0000 7f6f1be00700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f14161590 0x7f6f1415f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.246+0000 7f6f1be00700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f140cf230 0x7f6f1414b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.246+0000 7f6f1be00700 1 --2- 172.21.15.89:0/1511125035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f14160f10 0x7f6f1415ef90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.246+0000 7f6f1be00700 1 -- 172.21.15.89:0/1511125035 >> 172.21.15.89:0/1511125035 conn(0x7f6f1408aff0 msgr2=0x7f6f1415baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:47.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.246+0000 7f6f1be00700 1 -- 172.21.15.89:0/1511125035 shutdown_connections 2024-04-02T07:36:47.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.246+0000 7f6f1be00700 1 -- 172.21.15.89:0/1511125035 wait complete. 2024-04-02T07:36:47.266 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T07:36:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.458+0000 7fba3e000700 1 Processor -- start 2024-04-02T07:36:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.458+0000 7fba3e000700 1 -- start start 2024-04-02T07:36:47.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.459+0000 7fba3e000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fba380d16a0 0x7fba380d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.459+0000 7fba3e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba380d2370 0x7fba380ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.460+0000 7fba3e000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba38161420 con 0x7fba3815d9d0 2024-04-02T07:36:47.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.460+0000 7fba3e000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba380d2820 con 0x7fba380d16a0 2024-04-02T07:36:47.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.460+0000 7fba3e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba380d4340 con 0x7fba380d2370 2024-04-02T07:36:47.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.460+0000 7fba37e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba380d2370 0x7fba380ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.460+0000 7fba37400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fba3815d9d0 0x7fba380d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:37144/0 (socket says 172.21.15.89:37144) 2024-04-02T07:36:47.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.460+0000 7fba37e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba380d2370 0x7fba380ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:37968/0 (socket says 172.21.15.89:37968) 2024-04-02T07:36:47.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.460+0000 7fba37400700 1 -- 172.21.15.89:0/279818589 learned_addr learned my addr 172.21.15.89:0/279818589 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:47.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.460+0000 7fba2fe00700 1 --2- 172.21.15.89:0/279818589 >> v2:172.21.15.89:3300/0 conn(0x7fba380d16a0 0x7fba380d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.460+0000 7fba37e00700 1 -- 172.21.15.89:0/279818589 >> v2:172.21.15.89:3300/0 conn(0x7fba380d16a0 msgr2=0x7fba380d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:47.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.460+0000 7fba37e00700 1 --2- 172.21.15.89:0/279818589 >> v2:172.21.15.89:3300/0 conn(0x7fba380d16a0 0x7fba380d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.460+0000 7fba37e00700 1 -- 172.21.15.89:0/279818589 >> v1:172.21.15.113:6789/0 conn(0x7fba3815d9d0 legacy=0x7fba380d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.460+0000 7fba37e00700 1 -- 172.21.15.89:0/279818589 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba38161700 con 0x7fba380d2370 2024-04-02T07:36:47.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.460+0000 7fba2fe00700 1 --2- 172.21.15.89:0/279818589 >> v2:172.21.15.89:3300/0 conn(0x7fba380d16a0 0x7fba380d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:47.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.460+0000 7fba36a00700 1 -- 172.21.15.89:0/279818589 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248009472 0 0) 0x7fba38161420 con 0x7fba3815d9d0 2024-04-02T07:36:47.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.461+0000 7fba37e00700 1 --2- 172.21.15.89:0/279818589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba380d2370 0x7fba380ce360 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fba30009d90 tx=0x7fba3000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1104d8886634c411 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:47.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.461+0000 7fba36a00700 1 -- 172.21.15.89:0/279818589 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3000e070 con 0x7fba380d2370 2024-04-02T07:36:47.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.461+0000 7fba36a00700 1 -- 172.21.15.89:0/279818589 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3000bb20 con 0x7fba380d2370 2024-04-02T07:36:47.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.461+0000 7fba36a00700 1 -- 172.21.15.89:0/279818589 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba30012430 con 0x7fba380d2370 2024-04-02T07:36:47.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.462+0000 7fba3e000700 1 -- 172.21.15.89:0/279818589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba380d2370 msgr2=0x7fba380ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.462+0000 7fba3e000700 1 --2- 172.21.15.89:0/279818589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba380d2370 0x7fba380ce360 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fba30009d90 tx=0x7fba3000b330 comp rx=0 tx=0).stop 2024-04-02T07:36:47.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.462+0000 7fba3e000700 1 -- 172.21.15.89:0/279818589 shutdown_connections 2024-04-02T07:36:47.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.462+0000 7fba3e000700 1 --2- 172.21.15.89:0/279818589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba380d2370 0x7fba380ce360 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.462+0000 7fba3e000700 1 --2- 172.21.15.89:0/279818589 >> v2:172.21.15.89:3300/0 conn(0x7fba380d16a0 0x7fba380d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.462+0000 7fba3e000700 1 -- 172.21.15.89:0/279818589 >> 172.21.15.89:0/279818589 conn(0x7fba3808aff0 msgr2=0x7fba380d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:47.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.462+0000 7fba3e000700 1 -- 172.21.15.89:0/279818589 shutdown_connections 2024-04-02T07:36:47.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.462+0000 7fba3e000700 1 -- 172.21.15.89:0/279818589 wait complete. 2024-04-02T07:36:47.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.463+0000 7fba3e000700 1 Processor -- start 2024-04-02T07:36:47.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.464+0000 7fba3e000700 1 -- start start 2024-04-02T07:36:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.464+0000 7fba3e000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba380d16a0 0x7fba3814e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.464+0000 7fba3e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba380d2370 0x7fba380cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.464+0000 7fba3e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3815d9d0 0x7fba3815fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.464+0000 7fba3e000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba380cbdb0 con 0x7fba3815d9d0 2024-04-02T07:36:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.464+0000 7fba3e000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba381f6ac0 con 0x7fba380d16a0 2024-04-02T07:36:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.464+0000 7fba3e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba381f6da0 con 0x7fba380d2370 2024-04-02T07:36:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.465+0000 7fba37e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3815d9d0 0x7fba3815fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.465+0000 7fba37e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3815d9d0 0x7fba3815fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48948/0 (socket says 172.21.15.89:48948) 2024-04-02T07:36:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.465+0000 7fba37e00700 1 -- 172.21.15.89:0/3262092240 learned_addr learned my addr 172.21.15.89:0/3262092240 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.465+0000 7fba37400700 1 --2- 172.21.15.89:0/3262092240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba380d16a0 0x7fba3814e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.465+0000 7fba2fe00700 1 --2- 172.21.15.89:0/3262092240 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba380d2370 0x7fba380cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.465+0000 7fba37e00700 1 -- 172.21.15.89:0/3262092240 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba380d2370 msgr2=0x7fba380cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.465+0000 7fba37e00700 1 --2- 172.21.15.89:0/3262092240 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba380d2370 0x7fba380cb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.465+0000 7fba37e00700 1 -- 172.21.15.89:0/3262092240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba380d16a0 msgr2=0x7fba3814e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.465+0000 7fba37e00700 1 --2- 172.21.15.89:0/3262092240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba380d16a0 0x7fba3814e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.465+0000 7fba37e00700 1 -- 172.21.15.89:0/3262092240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba20009430 con 0x7fba3815d9d0 2024-04-02T07:36:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.465+0000 7fba37e00700 1 --2- 172.21.15.89:0/3262092240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3815d9d0 0x7fba3815fb50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fba30014040 tx=0x7fba30004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.466+0000 7fba2f400700 1 -- 172.21.15.89:0/3262092240 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba30012430 con 0x7fba3815d9d0 2024-04-02T07:36:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.466+0000 7fba2f400700 1 -- 172.21.15.89:0/3262092240 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba30020070 con 0x7fba3815d9d0 2024-04-02T07:36:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.466+0000 7fba3e000700 1 -- 172.21.15.89:0/3262092240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba30009a70 con 0x7fba3815d9d0 2024-04-02T07:36:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.466+0000 7fba2f400700 1 -- 172.21.15.89:0/3262092240 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba30004280 con 0x7fba3815d9d0 2024-04-02T07:36:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.466+0000 7fba3e000700 1 -- 172.21.15.89:0/3262092240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba381f7360 con 0x7fba3815d9d0 2024-04-02T07:36:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.467+0000 7fba3e000700 1 -- 172.21.15.89:0/3262092240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba380d1ca0 con 0x7fba3815d9d0 2024-04-02T07:36:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.468+0000 7fba2f400700 1 -- 172.21.15.89:0/3262092240 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fba300043e0 con 0x7fba3815d9d0 2024-04-02T07:36:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.468+0000 7fba2f400700 1 --2- 172.21.15.89:0/3262092240 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fba1406ac90 0x7fba1406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.468+0000 7fba2f400700 1 -- 172.21.15.89:0/3262092240 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fba30022070 con 0x7fba3815d9d0 2024-04-02T07:36:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.469+0000 7fba37400700 1 --2- 172.21.15.89:0/3262092240 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fba1406ac90 0x7fba1406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.469+0000 7fba37400700 1 --2- 172.21.15.89:0/3262092240 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fba1406ac90 0x7fba1406d150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fba28000f60 tx=0x7fba28007e70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:47.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.473+0000 7fba2f400700 1 -- 172.21.15.89:0/3262092240 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fba30056530 con 0x7fba3815d9d0 2024-04-02T07:36:47.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.718+0000 7fba3e000700 1 -- 172.21.15.89:0/3262092240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fba381f7580 con 0x7fba3815d9d0 2024-04-02T07:36:47.720 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T07:36:47.719+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T07:36:47.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.730+0000 7fba2f400700 1 -- 172.21.15.89:0/3262092240 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7fba30059b50 con 0x7fba3815d9d0 2024-04-02T07:36:47.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.732+0000 7fba3e000700 1 -- 172.21.15.89:0/3262092240 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fba1406ac90 msgr2=0x7fba1406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.732+0000 7fba3e000700 1 --2- 172.21.15.89:0/3262092240 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fba1406ac90 0x7fba1406d150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fba28000f60 tx=0x7fba28007e70 comp rx=0 tx=0).stop 2024-04-02T07:36:47.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.732+0000 7fba3e000700 1 -- 172.21.15.89:0/3262092240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3815d9d0 msgr2=0x7fba3815fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.732+0000 7fba3e000700 1 --2- 172.21.15.89:0/3262092240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3815d9d0 0x7fba3815fb50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fba30014040 tx=0x7fba30004ad0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.732+0000 7fba3e000700 1 -- 172.21.15.89:0/3262092240 shutdown_connections 2024-04-02T07:36:47.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.732+0000 7fba3e000700 1 --2- 172.21.15.89:0/3262092240 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fba1406ac90 0x7fba1406d150 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.732+0000 7fba3e000700 1 --2- 172.21.15.89:0/3262092240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba380d16a0 0x7fba3814e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.732+0000 7fba3e000700 1 --2- 172.21.15.89:0/3262092240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3815d9d0 0x7fba3815fb50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.732+0000 7fba3e000700 1 --2- 172.21.15.89:0/3262092240 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba380d2370 0x7fba380cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.732+0000 7fba3e000700 1 -- 172.21.15.89:0/3262092240 >> 172.21.15.89:0/3262092240 conn(0x7fba3808aff0 msgr2=0x7fba380b5130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:47.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.732+0000 7fba3e000700 1 -- 172.21.15.89:0/3262092240 shutdown_connections 2024-04-02T07:36:47.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.732+0000 7fba3e000700 1 -- 172.21.15.89:0/3262092240 wait complete. 2024-04-02T07:36:47.736 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T07:36:47.753 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T07:36:47.802 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T07:36:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.947+0000 7febba800700 1 Processor -- start 2024-04-02T07:36:47.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.947+0000 7febba800700 1 -- start start 2024-04-02T07:36:47.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.948+0000 7febba800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7febb415dfe0 0x7febb40ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.948+0000 7febba800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb40ce670 0x7febb40cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.948+0000 7febba800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb40cb310 con 0x7febb415a5c0 2024-04-02T07:36:47.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.948+0000 7febba800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb40ceb00 con 0x7febb415dfe0 2024-04-02T07:36:47.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.948+0000 7febba800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb40711a0 con 0x7febb40ce670 2024-04-02T07:36:47.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.948+0000 7febb3400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7febb415a5c0 0x7febb40709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53166/0 (socket says 172.21.15.89:53166) 2024-04-02T07:36:47.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.948+0000 7febb3400700 1 -- 172.21.15.89:0/1123950216 learned_addr learned my addr 172.21.15.89:0/1123950216 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:47.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.948+0000 7febb2a00700 1 --2- 172.21.15.89:0/1123950216 >> v2:172.21.15.89:3300/0 conn(0x7febb415dfe0 0x7febb40ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.949+0000 7febb2000700 1 -- 172.21.15.89:0/1123950216 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622700030 0 0) 0x7febb40cb310 con 0x7febb415a5c0 2024-04-02T07:36:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.949+0000 7febb3e00700 1 --2- 172.21.15.89:0/1123950216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb40ce670 0x7febb40cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.949+0000 7febb2000700 1 -- 172.21.15.89:0/1123950216 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb980035b0 con 0x7febb415a5c0 2024-04-02T07:36:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.949+0000 7febb2a00700 1 -- 172.21.15.89:0/1123950216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb40ce670 msgr2=0x7febb40cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.949+0000 7febb2a00700 1 --2- 172.21.15.89:0/1123950216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb40ce670 0x7febb40cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.949+0000 7febb2a00700 1 -- 172.21.15.89:0/1123950216 >> v1:172.21.15.89:6789/0 conn(0x7febb415a5c0 legacy=0x7febb40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.949+0000 7febb2a00700 1 -- 172.21.15.89:0/1123950216 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb4071480 con 0x7febb415dfe0 2024-04-02T07:36:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.949+0000 7febb3e00700 1 --2- 172.21.15.89:0/1123950216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb40ce670 0x7febb40cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.949+0000 7febb2a00700 1 --2- 172.21.15.89:0/1123950216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb415dfe0 0x7febb40ce130 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7feba0009840 tx=0x7feba0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5191ade5201a50d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.949+0000 7febb2000700 1 -- 172.21.15.89:0/1123950216 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba0011830 con 0x7febb415dfe0 2024-04-02T07:36:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.949+0000 7febb2000700 1 -- 172.21.15.89:0/1123950216 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba0011990 con 0x7febb415dfe0 2024-04-02T07:36:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.949+0000 7febb2000700 1 -- 172.21.15.89:0/1123950216 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba0011d20 con 0x7febb415dfe0 2024-04-02T07:36:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.950+0000 7febba800700 1 -- 172.21.15.89:0/1123950216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb415dfe0 msgr2=0x7febb40ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.950+0000 7febba800700 1 --2- 172.21.15.89:0/1123950216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb415dfe0 0x7febb40ce130 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7feba0009840 tx=0x7feba0009b50 comp rx=0 tx=0).stop 2024-04-02T07:36:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.950+0000 7febba800700 1 -- 172.21.15.89:0/1123950216 shutdown_connections 2024-04-02T07:36:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.950+0000 7febba800700 1 --2- 172.21.15.89:0/1123950216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb40ce670 0x7febb40cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.950+0000 7febba800700 1 --2- 172.21.15.89:0/1123950216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb415dfe0 0x7febb40ce130 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.950+0000 7febba800700 1 -- 172.21.15.89:0/1123950216 >> 172.21.15.89:0/1123950216 conn(0x7febb40b0d70 msgr2=0x7febb40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:47.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.950+0000 7febba800700 1 -- 172.21.15.89:0/1123950216 shutdown_connections 2024-04-02T07:36:47.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.950+0000 7febba800700 1 -- 172.21.15.89:0/1123950216 wait complete. 2024-04-02T07:36:47.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.952+0000 7febba800700 1 Processor -- start 2024-04-02T07:36:47.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.952+0000 7febba800700 1 -- start start 2024-04-02T07:36:47.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febba800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb40ce670 0x7febb40c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febba800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb415a5c0 0x7febb40caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febba800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb415dfe0 0x7febb41493c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febba800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb40c9ff0 con 0x7febb415dfe0 2024-04-02T07:36:47.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febba800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb40c93e0 con 0x7febb415a5c0 2024-04-02T07:36:47.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febba800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb40c96c0 con 0x7febb40ce670 2024-04-02T07:36:47.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febb3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb415dfe0 0x7febb41493c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febb3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb415dfe0 0x7febb41493c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48966/0 (socket says 172.21.15.89:48966) 2024-04-02T07:36:47.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febb3e00700 1 -- 172.21.15.89:0/3394412618 learned_addr learned my addr 172.21.15.89:0/3394412618 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:47.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febb2a00700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb415a5c0 0x7febb40caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febb3400700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb40ce670 0x7febb40c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febb3e00700 1 -- 172.21.15.89:0/3394412618 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb40ce670 msgr2=0x7febb40c8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febb3e00700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb40ce670 0x7febb40c8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febb3e00700 1 -- 172.21.15.89:0/3394412618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb415a5c0 msgr2=0x7febb40caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:47.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febb3e00700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb415a5c0 0x7febb40caa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:47.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febb3e00700 1 -- 172.21.15.89:0/3394412618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba00094a0 con 0x7febb415dfe0 2024-04-02T07:36:47.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.953+0000 7febb3400700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb40ce670 0x7febb40c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:47.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.954+0000 7febb3e00700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb415dfe0 0x7febb41493c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7febac00cef0 tx=0x7febac00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:47.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.954+0000 7febab400700 1 -- 172.21.15.89:0/3394412618 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febac00ae50 con 0x7febb415dfe0 2024-04-02T07:36:47.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.954+0000 7febab400700 1 -- 172.21.15.89:0/3394412618 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febac015440 con 0x7febb415dfe0 2024-04-02T07:36:47.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.954+0000 7febba800700 1 -- 172.21.15.89:0/3394412618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb40c99a0 con 0x7febb415dfe0 2024-04-02T07:36:47.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.954+0000 7febab400700 1 -- 172.21.15.89:0/3394412618 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febac0157f0 con 0x7febb415dfe0 2024-04-02T07:36:47.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.954+0000 7febba800700 1 -- 172.21.15.89:0/3394412618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb40caf90 con 0x7febb415dfe0 2024-04-02T07:36:47.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.955+0000 7febba800700 1 -- 172.21.15.89:0/3394412618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb4070ef0 con 0x7febb415dfe0 2024-04-02T07:36:47.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.956+0000 7febab400700 1 -- 172.21.15.89:0/3394412618 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7febac00d430 con 0x7febb415dfe0 2024-04-02T07:36:47.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.957+0000 7febab400700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb9406ac90 0x7feb9406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:47.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.957+0000 7febab400700 1 -- 172.21.15.89:0/3394412618 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5824+0+0 (secure 0 0 0) 0x7febac089a60 con 0x7febb415dfe0 2024-04-02T07:36:47.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.957+0000 7febb3400700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb9406ac90 0x7feb9406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:47.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.958+0000 7febb3400700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb9406ac90 0x7feb9406d150 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feba40011e0 tx=0x7feba4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:47.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:47.962+0000 7febab400700 1 -- 172.21.15.89:0/3394412618 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7febac054b50 con 0x7febb415dfe0 2024-04-02T07:36:48.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.210+0000 7febba800700 1 -- 172.21.15.89:0/3394412618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7febb4066ff0 con 0x7febb415dfe0 2024-04-02T07:36:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.739+0000 7febab400700 1 -- 172.21.15.89:0/3394412618 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7febac01a0a0 con 0x7febb415dfe0 2024-04-02T07:36:48.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.741+0000 7febba800700 1 -- 172.21.15.89:0/3394412618 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb9406ac90 msgr2=0x7feb9406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:48.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.741+0000 7febba800700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb9406ac90 0x7feb9406d150 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feba40011e0 tx=0x7feba4009040 comp rx=0 tx=0).stop 2024-04-02T07:36:48.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.741+0000 7febba800700 1 -- 172.21.15.89:0/3394412618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb415dfe0 msgr2=0x7febb41493c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:48.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.741+0000 7febba800700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb415dfe0 0x7febb41493c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7febac00cef0 tx=0x7febac00a340 comp rx=0 tx=0).stop 2024-04-02T07:36:48.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.741+0000 7febba800700 1 -- 172.21.15.89:0/3394412618 shutdown_connections 2024-04-02T07:36:48.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.741+0000 7febba800700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb9406ac90 0x7feb9406d150 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:48.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.741+0000 7febba800700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb415dfe0 0x7febb41493c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:48.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.741+0000 7febba800700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb40ce670 0x7febb40c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:48.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.741+0000 7febba800700 1 --2- 172.21.15.89:0/3394412618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb415a5c0 0x7febb40caa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:48.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.741+0000 7febba800700 1 -- 172.21.15.89:0/3394412618 >> 172.21.15.89:0/3394412618 conn(0x7febb40b0d70 msgr2=0x7febb40cdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:48.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.741+0000 7febba800700 1 -- 172.21.15.89:0/3394412618 shutdown_connections 2024-04-02T07:36:48.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.742+0000 7febba800700 1 -- 172.21.15.89:0/3394412618 wait complete. 2024-04-02T07:36:48.762 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T07:36:48.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.957+0000 7f0511000700 1 Processor -- start 2024-04-02T07:36:48.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.958+0000 7f0511000700 1 -- start start 2024-04-02T07:36:48.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.959+0000 7f0511000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f050c15ddf0 0x7f050c0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:48.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.959+0000 7f0511000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f050c0d28e0 0x7f050c15f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:48.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.959+0000 7f0511000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050c0ce8a0 con 0x7f050c15e1d0 2024-04-02T07:36:48.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.959+0000 7f0511000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f050c15ec60 con 0x7f050c0d28e0 2024-04-02T07:36:48.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.959+0000 7f0511000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f050c0cdc90 con 0x7f050c15ddf0 2024-04-02T07:36:48.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.959+0000 7f050a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f050c0d28e0 0x7f050c15f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:48.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.959+0000 7f050a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f050c0d28e0 0x7f050c15f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53724/0 (socket says 172.21.15.89:53724) 2024-04-02T07:36:48.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.959+0000 7f050a800700 1 -- 172.21.15.89:0/4185326596 learned_addr learned my addr 172.21.15.89:0/4185326596 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:48.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.959+0000 7f0509e00700 1 --2- 172.21.15.89:0/4185326596 >> v2:172.21.15.113:3300/0 conn(0x7f050c15ddf0 0x7f050c0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:48.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.959+0000 7f050a800700 1 -- 172.21.15.89:0/4185326596 >> v2:172.21.15.113:3300/0 conn(0x7f050c15ddf0 msgr2=0x7f050c0ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:48.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.960+0000 7f050a800700 1 --2- 172.21.15.89:0/4185326596 >> v2:172.21.15.113:3300/0 conn(0x7f050c15ddf0 0x7f050c0ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:48.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.960+0000 7f050a800700 1 -- 172.21.15.89:0/4185326596 >> v1:172.21.15.113:6789/0 conn(0x7f050c15e1d0 legacy=0x7f050c0d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:48.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.960+0000 7f050a800700 1 -- 172.21.15.89:0/4185326596 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c0cdf70 con 0x7f050c0d28e0 2024-04-02T07:36:48.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.960+0000 7f0509e00700 1 --2- 172.21.15.89:0/4185326596 >> v2:172.21.15.113:3300/0 conn(0x7f050c15ddf0 0x7f050c0ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:48.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.960+0000 7f050a800700 1 --2- 172.21.15.89:0/4185326596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f050c0d28e0 0x7f050c15f0f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f050c06b3e0 tx=0x7f050400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4b3bf501708434 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:48.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.960+0000 7f0509400700 1 -- 172.21.15.89:0/4185326596 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050400e070 con 0x7f050c0d28e0 2024-04-02T07:36:48.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.960+0000 7f0509400700 1 -- 172.21.15.89:0/4185326596 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0504003a20 con 0x7f050c0d28e0 2024-04-02T07:36:48.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.960+0000 7f0509400700 1 -- 172.21.15.89:0/4185326596 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504003dd0 con 0x7f050c0d28e0 2024-04-02T07:36:48.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.960+0000 7f0511000700 1 -- 172.21.15.89:0/4185326596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f050c0d28e0 msgr2=0x7f050c15f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:48.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.960+0000 7f0511000700 1 --2- 172.21.15.89:0/4185326596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f050c0d28e0 0x7f050c15f0f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f050c06b3e0 tx=0x7f050400b330 comp rx=0 tx=0).stop 2024-04-02T07:36:48.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.961+0000 7f0511000700 1 -- 172.21.15.89:0/4185326596 shutdown_connections 2024-04-02T07:36:48.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.961+0000 7f0511000700 1 --2- 172.21.15.89:0/4185326596 >> v2:172.21.15.113:3300/0 conn(0x7f050c15ddf0 0x7f050c0ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:48.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.961+0000 7f0511000700 1 --2- 172.21.15.89:0/4185326596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f050c0d28e0 0x7f050c15f0f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:48.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.961+0000 7f0511000700 1 -- 172.21.15.89:0/4185326596 >> 172.21.15.89:0/4185326596 conn(0x7f050c08aff0 msgr2=0x7f050c0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:48.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.961+0000 7f0511000700 1 -- 172.21.15.89:0/4185326596 shutdown_connections 2024-04-02T07:36:48.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.961+0000 7f0511000700 1 -- 172.21.15.89:0/4185326596 wait complete. 2024-04-02T07:36:48.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.962+0000 7f0511000700 1 Processor -- start 2024-04-02T07:36:48.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.962+0000 7f0511000700 1 -- start start 2024-04-02T07:36:48.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.963+0000 7f0511000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f050c15ddf0 0x7f050c156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:48.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.963+0000 7f0511000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f050c15e1d0 0x7f050c155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:48.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.963+0000 7f0511000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f050c156690 0x7f050c0ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:48.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.963+0000 7f0511000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c160680 con 0x7f050c156690 2024-04-02T07:36:48.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.963+0000 7f0511000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c0cbab0 con 0x7f050c15e1d0 2024-04-02T07:36:48.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.963+0000 7f0511000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050c0cbd00 con 0x7f050c15ddf0 2024-04-02T07:36:48.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.964+0000 7f050a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f050c156690 0x7f050c0ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:48.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.964+0000 7f050a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f050c156690 0x7f050c0ccda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53726/0 (socket says 172.21.15.89:53726) 2024-04-02T07:36:48.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.964+0000 7f050a800700 1 -- 172.21.15.89:0/1913248337 learned_addr learned my addr 172.21.15.89:0/1913248337 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:48.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.964+0000 7f050a800700 1 -- 172.21.15.89:0/1913248337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f050c15ddf0 msgr2=0x7f050c156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:48.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.964+0000 7f050a800700 1 --2- 172.21.15.89:0/1913248337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f050c15ddf0 0x7f050c156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:48.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.964+0000 7f050a800700 1 -- 172.21.15.89:0/1913248337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f050c15e1d0 msgr2=0x7f050c155e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:48.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.964+0000 7f050a800700 1 --2- 172.21.15.89:0/1913248337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f050c15e1d0 0x7f050c155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:48.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.964+0000 7f050a800700 1 -- 172.21.15.89:0/1913248337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0504009b30 con 0x7f050c156690 2024-04-02T07:36:48.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.964+0000 7f050a800700 1 --2- 172.21.15.89:0/1913248337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f050c156690 0x7f050c0ccda0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0504015040 tx=0x7f0504004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:48.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.964+0000 7f0502000700 1 -- 172.21.15.89:0/1913248337 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504004440 con 0x7f050c156690 2024-04-02T07:36:48.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.964+0000 7f0502000700 1 -- 172.21.15.89:0/1913248337 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05040045a0 con 0x7f050c156690 2024-04-02T07:36:48.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.964+0000 7f0502000700 1 -- 172.21.15.89:0/1913248337 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504004950 con 0x7f050c156690 2024-04-02T07:36:48.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.964+0000 7f0511000700 1 -- 172.21.15.89:0/1913248337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050c0cbfe0 con 0x7f050c156690 2024-04-02T07:36:48.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.965+0000 7f0511000700 1 -- 172.21.15.89:0/1913248337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f050c0cc5a0 con 0x7f050c156690 2024-04-02T07:36:48.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.966+0000 7f0511000700 1 -- 172.21.15.89:0/1913248337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f050c15e640 con 0x7f050c156690 2024-04-02T07:36:48.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.966+0000 7f0502000700 1 -- 172.21.15.89:0/1913248337 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0504012430 con 0x7f050c156690 2024-04-02T07:36:48.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.967+0000 7f0502000700 1 --2- 172.21.15.89:0/1913248337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f04e806ac90 0x7f04e806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:48.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.967+0000 7f0502000700 1 -- 172.21.15.89:0/1913248337 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f050408a970 con 0x7f050c156690 2024-04-02T07:36:48.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.968+0000 7f0509e00700 1 --2- 172.21.15.89:0/1913248337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f04e806ac90 0x7f04e806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:48.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.968+0000 7f0509e00700 1 --2- 172.21.15.89:0/1913248337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f04e806ac90 0x7f04e806d150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f050c15ebb0 tx=0x7f04fc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:48.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:48.972+0000 7f0502000700 1 -- 172.21.15.89:0/1913248337 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f050c15e640 con 0x7f050c156690 2024-04-02T07:36:49.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:49.173+0000 7f0511000700 1 -- 172.21.15.89:0/1913248337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f050c15e640 con 0x7f050c156690 2024-04-02T07:36:49.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:49.934+0000 7f0502000700 1 -- 172.21.15.89:0/1913248337 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7f050c15e640 con 0x7f050c156690 2024-04-02T07:36:49.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:49.938+0000 7f0511000700 1 -- 172.21.15.89:0/1913248337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f04e806ac90 msgr2=0x7f04e806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:49.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:49.938+0000 7f0511000700 1 --2- 172.21.15.89:0/1913248337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f04e806ac90 0x7f04e806d150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f050c15ebb0 tx=0x7f04fc008040 comp rx=0 tx=0).stop 2024-04-02T07:36:49.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:49.938+0000 7f0511000700 1 -- 172.21.15.89:0/1913248337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f050c156690 msgr2=0x7f050c0ccda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:49.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:49.938+0000 7f0511000700 1 --2- 172.21.15.89:0/1913248337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f050c156690 0x7f050c0ccda0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0504015040 tx=0x7f0504004ad0 comp rx=0 tx=0).stop 2024-04-02T07:36:49.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:49.938+0000 7f0511000700 1 -- 172.21.15.89:0/1913248337 shutdown_connections 2024-04-02T07:36:49.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:49.938+0000 7f0511000700 1 --2- 172.21.15.89:0/1913248337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f04e806ac90 0x7f04e806d150 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:49.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:49.938+0000 7f0511000700 1 --2- 172.21.15.89:0/1913248337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f050c156690 0x7f050c0ccda0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:49.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:49.938+0000 7f0511000700 1 --2- 172.21.15.89:0/1913248337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f050c15ddf0 0x7f050c156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:49.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:49.939+0000 7f0511000700 1 --2- 172.21.15.89:0/1913248337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f050c15e1d0 0x7f050c155e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:49.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:49.939+0000 7f0511000700 1 -- 172.21.15.89:0/1913248337 >> 172.21.15.89:0/1913248337 conn(0x7f050c08aff0 msgr2=0x7f050c0b8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:49.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:49.939+0000 7f0511000700 1 -- 172.21.15.89:0/1913248337 shutdown_connections 2024-04-02T07:36:49.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:49.939+0000 7f0511000700 1 -- 172.21.15.89:0/1913248337 wait complete. 2024-04-02T07:36:49.942 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T07:36:49.961 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T07:36:50.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.157+0000 7fd08ce00700 1 Processor -- start 2024-04-02T07:36:50.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.157+0000 7fd08ce00700 1 -- start start 2024-04-02T07:36:50.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.158+0000 7fd08ce00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd08815ddf0 0x7fd0880ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:50.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.158+0000 7fd08ce00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd08815e1d0 0x7fd0880d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:50.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.158+0000 7fd08ce00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0880d2b40 con 0x7fd0880d2710 2024-04-02T07:36:50.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.158+0000 7fd08ce00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd08815ec60 con 0x7fd08815ddf0 2024-04-02T07:36:50.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.158+0000 7fd08ce00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0880ce8a0 con 0x7fd08815e1d0 2024-04-02T07:36:50.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.159+0000 7fd085e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd08815ddf0 0x7fd0880ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:50.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.159+0000 7fd085e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd08815ddf0 0x7fd0880ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53738/0 (socket says 172.21.15.89:53738) 2024-04-02T07:36:50.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.159+0000 7fd085e00700 1 -- 172.21.15.89:0/1225008528 learned_addr learned my addr 172.21.15.89:0/1225008528 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:50.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.159+0000 7fd085400700 1 --2- 172.21.15.89:0/1225008528 >> v2:172.21.15.113:3300/0 conn(0x7fd08815e1d0 0x7fd0880d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:50.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.159+0000 7fd085e00700 1 -- 172.21.15.89:0/1225008528 >> v2:172.21.15.113:3300/0 conn(0x7fd08815e1d0 msgr2=0x7fd0880d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:50.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.159+0000 7fd085e00700 1 --2- 172.21.15.89:0/1225008528 >> v2:172.21.15.113:3300/0 conn(0x7fd08815e1d0 0x7fd0880d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:50.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.159+0000 7fd085e00700 1 -- 172.21.15.89:0/1225008528 >> v1:172.21.15.113:6789/0 conn(0x7fd0880d2710 legacy=0x7fd0881612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:50.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.159+0000 7fd085e00700 1 -- 172.21.15.89:0/1225008528 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0880cdc90 con 0x7fd08815ddf0 2024-04-02T07:36:50.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.159+0000 7fd085e00700 1 --2- 172.21.15.89:0/1225008528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815ddf0 0x7fd0880ced30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd0880d0fe0 tx=0x7fd078009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cab250e5d708758 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:50.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.160+0000 7fd084a00700 1 -- 172.21.15.89:0/1225008528 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07800b870 con 0x7fd08815ddf0 2024-04-02T07:36:50.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.160+0000 7fd084a00700 1 -- 172.21.15.89:0/1225008528 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07800b9d0 con 0x7fd08815ddf0 2024-04-02T07:36:50.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.160+0000 7fd084a00700 1 -- 172.21.15.89:0/1225008528 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07800bd60 con 0x7fd08815ddf0 2024-04-02T07:36:50.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.160+0000 7fd08ce00700 1 -- 172.21.15.89:0/1225008528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815ddf0 msgr2=0x7fd0880ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:50.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.160+0000 7fd08ce00700 1 --2- 172.21.15.89:0/1225008528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815ddf0 0x7fd0880ced30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd0880d0fe0 tx=0x7fd078009bd0 comp rx=0 tx=0).stop 2024-04-02T07:36:50.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.160+0000 7fd08ce00700 1 -- 172.21.15.89:0/1225008528 shutdown_connections 2024-04-02T07:36:50.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.160+0000 7fd08ce00700 1 --2- 172.21.15.89:0/1225008528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815ddf0 0x7fd0880ced30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:50.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.160+0000 7fd08ce00700 1 --2- 172.21.15.89:0/1225008528 >> v2:172.21.15.113:3300/0 conn(0x7fd08815e1d0 0x7fd0880d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:50.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.160+0000 7fd08ce00700 1 -- 172.21.15.89:0/1225008528 >> 172.21.15.89:0/1225008528 conn(0x7fd08808aff0 msgr2=0x7fd0880d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:50.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.160+0000 7fd08ce00700 1 -- 172.21.15.89:0/1225008528 shutdown_connections 2024-04-02T07:36:50.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.160+0000 7fd08ce00700 1 -- 172.21.15.89:0/1225008528 wait complete. 2024-04-02T07:36:50.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.162+0000 7fd08ce00700 1 Processor -- start 2024-04-02T07:36:50.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.162+0000 7fd08ce00700 1 -- start start 2024-04-02T07:36:50.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.163+0000 7fd08ce00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0880d2710 0x7fd088156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:50.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.163+0000 7fd08ce00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd08815ddf0 0x7fd088159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:50.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.163+0000 7fd08ce00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815e1d0 0x7fd08815eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:50.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.163+0000 7fd08ce00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd088158550 con 0x7fd08815e1d0 2024-04-02T07:36:50.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.163+0000 7fd08ce00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd088158830 con 0x7fd08815ddf0 2024-04-02T07:36:50.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.163+0000 7fd08ce00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd088158b10 con 0x7fd0880d2710 2024-04-02T07:36:50.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.163+0000 7fd086800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815e1d0 0x7fd08815eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:50.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.163+0000 7fd086800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815e1d0 0x7fd08815eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53748/0 (socket says 172.21.15.89:53748) 2024-04-02T07:36:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.163+0000 7fd086800700 1 -- 172.21.15.89:0/3887303224 learned_addr learned my addr 172.21.15.89:0/3887303224 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.163+0000 7fd085400700 1 --2- 172.21.15.89:0/3887303224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd08815ddf0 0x7fd088159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.163+0000 7fd085e00700 1 --2- 172.21.15.89:0/3887303224 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0880d2710 0x7fd088156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.164+0000 7fd086800700 1 -- 172.21.15.89:0/3887303224 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0880d2710 msgr2=0x7fd088156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.164+0000 7fd086800700 1 --2- 172.21.15.89:0/3887303224 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0880d2710 0x7fd088156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.164+0000 7fd086800700 1 -- 172.21.15.89:0/3887303224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd08815ddf0 msgr2=0x7fd088159f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.164+0000 7fd086800700 1 --2- 172.21.15.89:0/3887303224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd08815ddf0 0x7fd088159f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.164+0000 7fd086800700 1 -- 172.21.15.89:0/3887303224 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd078009550 con 0x7fd08815e1d0 2024-04-02T07:36:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.164+0000 7fd086800700 1 --2- 172.21.15.89:0/3887303224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815e1d0 0x7fd08815eda0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd08000cc50 tx=0x7fd08000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.164+0000 7fd07e000700 1 -- 172.21.15.89:0/3887303224 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd080003c00 con 0x7fd08815e1d0 2024-04-02T07:36:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.164+0000 7fd07e000700 1 -- 172.21.15.89:0/3887303224 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd080003d60 con 0x7fd08815e1d0 2024-04-02T07:36:50.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.164+0000 7fd08ce00700 1 -- 172.21.15.89:0/3887303224 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd088158df0 con 0x7fd08815e1d0 2024-04-02T07:36:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.164+0000 7fd07e000700 1 -- 172.21.15.89:0/3887303224 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08000aab0 con 0x7fd08815e1d0 2024-04-02T07:36:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.164+0000 7fd08ce00700 1 -- 172.21.15.89:0/3887303224 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08815a490 con 0x7fd08815e1d0 2024-04-02T07:36:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.166+0000 7fd073400700 1 -- 172.21.15.89:0/3887303224 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd088159310 con 0x7fd08815e1d0 2024-04-02T07:36:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.167+0000 7fd07e000700 1 -- 172.21.15.89:0/3887303224 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd08000ac10 con 0x7fd08815e1d0 2024-04-02T07:36:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.169+0000 7fd07e000700 1 --2- 172.21.15.89:0/3887303224 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd06806ac90 0x7fd06806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.169+0000 7fd07e000700 1 -- 172.21.15.89:0/3887303224 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5386+0+0 (secure 0 0 0) 0x7fd0800890c0 con 0x7fd08815e1d0 2024-04-02T07:36:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.170+0000 7fd085e00700 1 --2- 172.21.15.89:0/3887303224 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd06806ac90 0x7fd06806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.171+0000 7fd085e00700 1 --2- 172.21.15.89:0/3887303224 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd06806ac90 0x7fd06806d150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd078000c00 tx=0x7fd078008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:50.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.175+0000 7fd07e000700 1 -- 172.21.15.89:0/3887303224 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd088159310 con 0x7fd08815e1d0 2024-04-02T07:36:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.377+0000 7fd073400700 1 -- 172.21.15.89:0/3887303224 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd088159310 con 0x7fd08815e1d0 2024-04-02T07:36:50.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.937+0000 7fd07e000700 1 -- 172.21.15.89:0/3887303224 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd088159310 con 0x7fd08815e1d0 2024-04-02T07:36:50.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.941+0000 7fd073400700 1 -- 172.21.15.89:0/3887303224 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd06806ac90 msgr2=0x7fd06806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:50.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.941+0000 7fd073400700 1 --2- 172.21.15.89:0/3887303224 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd06806ac90 0x7fd06806d150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd078000c00 tx=0x7fd078008030 comp rx=0 tx=0).stop 2024-04-02T07:36:50.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.941+0000 7fd073400700 1 -- 172.21.15.89:0/3887303224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815e1d0 msgr2=0x7fd08815eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:50.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.941+0000 7fd073400700 1 --2- 172.21.15.89:0/3887303224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815e1d0 0x7fd08815eda0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd08000cc50 tx=0x7fd08000cf60 comp rx=0 tx=0).stop 2024-04-02T07:36:50.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.941+0000 7fd073400700 1 -- 172.21.15.89:0/3887303224 shutdown_connections 2024-04-02T07:36:50.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.941+0000 7fd073400700 1 --2- 172.21.15.89:0/3887303224 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd06806ac90 0x7fd06806d150 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:50.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.941+0000 7fd073400700 1 --2- 172.21.15.89:0/3887303224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd08815e1d0 0x7fd08815eda0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:50.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.941+0000 7fd073400700 1 --2- 172.21.15.89:0/3887303224 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0880d2710 0x7fd088156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:50.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.941+0000 7fd073400700 1 --2- 172.21.15.89:0/3887303224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd08815ddf0 0x7fd088159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:50.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.941+0000 7fd073400700 1 -- 172.21.15.89:0/3887303224 >> 172.21.15.89:0/3887303224 conn(0x7fd08808aff0 msgr2=0x7fd0880b7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:50.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.941+0000 7fd073400700 1 -- 172.21.15.89:0/3887303224 shutdown_connections 2024-04-02T07:36:50.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:50.941+0000 7fd073400700 1 -- 172.21.15.89:0/3887303224 wait complete. 2024-04-02T07:36:50.946 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T07:36:50.965 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T07:36:50.965 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T07:36:50.966 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T07:36:50.966 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:36:50.966 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T07:36:50.966 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T07:36:50.966 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T07:36:50.966 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:36:50.967 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T07:36:50.967 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T07:36:50.967 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T07:36:50.967 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:36:50.967 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T07:36:50.967 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T07:36:50.968 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T07:36:50.968 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:36:50.984 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T07:36:50.985 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T07:36:50.986 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T07:36:50.986 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T07:36:50.989 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T07:36:50.989 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T07:36:51.029 INFO:tasks.ceph.mds.c:Started 2024-04-02T07:36:51.029 DEBUG:teuthology.parallel:result is None 2024-04-02T07:36:51.030 INFO:tasks.ceph.mds.d:Started 2024-04-02T07:36:51.030 DEBUG:teuthology.parallel:result is None 2024-04-02T07:36:51.032 INFO:tasks.ceph.mds.b:Started 2024-04-02T07:36:51.033 DEBUG:teuthology.parallel:result is None 2024-04-02T07:36:51.179 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T07:36:51.184 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T07:36:51.184 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T07:36:53.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-04-02T07:36:56.985 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T07:36:56.985 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T07:36:56.990 INFO:tasks.ceph.mds.a:Started 2024-04-02T07:36:56.991 DEBUG:teuthology.parallel:result is None 2024-04-02T07:36:56.991 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T07:36:57.141 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T07:36:57.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.199+0000 7f90bae00700 1 Processor -- start 2024-04-02T07:36:57.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.199+0000 7f90bae00700 1 -- start start 2024-04-02T07:36:57.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.200+0000 7f90bae00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f90b4160e50 0x7f90b40b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:57.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.200+0000 7f90bae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90b40b3f80 0x7f90b40b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:57.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.200+0000 7f90bae00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b40cef00 con 0x7f90b40d3f20 2024-04-02T07:36:57.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.200+0000 7f90bae00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b4161230 con 0x7f90b4160e50 2024-04-02T07:36:57.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.200+0000 7f90bae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b40d2a80 con 0x7f90b40b3f80 2024-04-02T07:36:57.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.200+0000 7f90b3e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f90b4160e50 0x7f90b40b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:57.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.200+0000 7f90b8e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90b40b3f80 0x7f90b40b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:57.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.200+0000 7f90b3e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f90b4160e50 0x7f90b40b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:38190/0 (socket says 172.21.15.89:38190) 2024-04-02T07:36:57.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.200+0000 7f90b3e00700 1 -- 172.21.15.89:0/275915850 learned_addr learned my addr 172.21.15.89:0/275915850 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:57.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.200+0000 7f90b3400700 1 --1- 172.21.15.89:0/275915850 >> v1:172.21.15.113:6789/0 conn(0x7f90b40d3f20 0x7f90b40b2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:37294/0 (socket says 172.21.15.89:37294) 2024-04-02T07:36:57.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.201+0000 7f90b3e00700 1 -- 172.21.15.89:0/275915850 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90b40b3f80 msgr2=0x7f90b40b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:57.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.201+0000 7f90b3e00700 1 --2- 172.21.15.89:0/275915850 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90b40b3f80 0x7f90b40b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:57.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.201+0000 7f90b3e00700 1 -- 172.21.15.89:0/275915850 >> v1:172.21.15.113:6789/0 conn(0x7f90b40d3f20 legacy=0x7f90b40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:57.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.201+0000 7f90b3e00700 1 -- 172.21.15.89:0/275915850 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b40d43b0 con 0x7f90b4160e50 2024-04-02T07:36:57.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.201+0000 7f90b8e00700 1 --2- 172.21.15.89:0/275915850 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90b40b3f80 0x7f90b40b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:57.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.201+0000 7f90b3e00700 1 --2- 172.21.15.89:0/275915850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b4160e50 0x7f90b40b4e90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f90a40098f0 tx=0x7f90a4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56051b21fa5d9ca9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:57.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.201+0000 7f90b2a00700 1 -- 172.21.15.89:0/275915850 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a4011830 con 0x7f90b4160e50 2024-04-02T07:36:57.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.201+0000 7f90b2a00700 1 -- 172.21.15.89:0/275915850 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a4011990 con 0x7f90b4160e50 2024-04-02T07:36:57.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.201+0000 7f90b2a00700 1 -- 172.21.15.89:0/275915850 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a4011d20 con 0x7f90b4160e50 2024-04-02T07:36:57.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.202+0000 7f90bae00700 1 -- 172.21.15.89:0/275915850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b4160e50 msgr2=0x7f90b40b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:57.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.202+0000 7f90bae00700 1 --2- 172.21.15.89:0/275915850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b4160e50 0x7f90b40b4e90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f90a40098f0 tx=0x7f90a4009c00 comp rx=0 tx=0).stop 2024-04-02T07:36:57.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.202+0000 7f90bae00700 1 -- 172.21.15.89:0/275915850 shutdown_connections 2024-04-02T07:36:57.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.202+0000 7f90bae00700 1 --2- 172.21.15.89:0/275915850 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90b40b3f80 0x7f90b40b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:57.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.202+0000 7f90bae00700 1 --2- 172.21.15.89:0/275915850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b4160e50 0x7f90b40b4e90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:57.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.202+0000 7f90bae00700 1 -- 172.21.15.89:0/275915850 >> 172.21.15.89:0/275915850 conn(0x7f90b408aff0 msgr2=0x7f90b40d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:57.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.202+0000 7f90bae00700 1 -- 172.21.15.89:0/275915850 shutdown_connections 2024-04-02T07:36:57.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.202+0000 7f90bae00700 1 -- 172.21.15.89:0/275915850 wait complete. 2024-04-02T07:36:57.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.204+0000 7f90bae00700 1 Processor -- start 2024-04-02T07:36:57.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.204+0000 7f90bae00700 1 -- start start 2024-04-02T07:36:57.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.205+0000 7f90bae00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b40b3f80 0x7f90b4153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:57.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.205+0000 7f90bae00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b4160e50 0x7f90b40c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:57.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.205+0000 7f90bae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90b40c98d0 0x7f90b40b6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:57.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.205+0000 7f90bae00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b4153dc0 con 0x7f90b4160e50 2024-04-02T07:36:57.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.205+0000 7f90bae00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b40ca410 con 0x7f90b40b3f80 2024-04-02T07:36:57.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.205+0000 7f90bae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b40c9e50 con 0x7f90b40c98d0 2024-04-02T07:36:57.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.205+0000 7f90b3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b4160e50 0x7f90b40c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:57.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.205+0000 7f90b3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b4160e50 0x7f90b40c9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53804/0 (socket says 172.21.15.89:53804) 2024-04-02T07:36:57.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.205+0000 7f90b3400700 1 -- 172.21.15.89:0/2489990008 learned_addr learned my addr 172.21.15.89:0/2489990008 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:57.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.205+0000 7f90b3e00700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b40b3f80 0x7f90b4153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:57.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.205+0000 7f90b8e00700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90b40c98d0 0x7f90b40b6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:57.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.206+0000 7f90b3e00700 1 -- 172.21.15.89:0/2489990008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90b40c98d0 msgr2=0x7f90b40b6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:57.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.206+0000 7f90b3e00700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90b40c98d0 0x7f90b40b6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:57.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.206+0000 7f90b3e00700 1 -- 172.21.15.89:0/2489990008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b4160e50 msgr2=0x7f90b40c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:57.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.206+0000 7f90b3e00700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b4160e50 0x7f90b40c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:57.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.206+0000 7f90b3e00700 1 -- 172.21.15.89:0/2489990008 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac009a70 con 0x7f90b40b3f80 2024-04-02T07:36:57.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.206+0000 7f90b3400700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b4160e50 0x7f90b40c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:36:57.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.206+0000 7f90b3e00700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b40b3f80 0x7f90b4153880 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f90a4016000 tx=0x7f90a4004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:57.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.206+0000 7f90abe00700 1 -- 172.21.15.89:0/2489990008 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40044d0 con 0x7f90b40b3f80 2024-04-02T07:36:57.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.206+0000 7f90abe00700 1 -- 172.21.15.89:0/2489990008 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a401e070 con 0x7f90b40b3f80 2024-04-02T07:36:57.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.206+0000 7f90bae00700 1 -- 172.21.15.89:0/2489990008 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a4009550 con 0x7f90b40b3f80 2024-04-02T07:36:57.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.206+0000 7f90abe00700 1 -- 172.21.15.89:0/2489990008 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40038a0 con 0x7f90b40b3f80 2024-04-02T07:36:57.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.206+0000 7f90bae00700 1 -- 172.21.15.89:0/2489990008 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b40b63f0 con 0x7f90b40b3f80 2024-04-02T07:36:57.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.207+0000 7f90bae00700 1 -- 172.21.15.89:0/2489990008 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b4067720 con 0x7f90b40b3f80 2024-04-02T07:36:57.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.208+0000 7f90abe00700 1 -- 172.21.15.89:0/2489990008 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f90a4003a00 con 0x7f90b40b3f80 2024-04-02T07:36:57.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.209+0000 7f90abe00700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f909406ac90 0x7f909406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:57.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.209+0000 7f90abe00700 1 -- 172.21.15.89:0/2489990008 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f90a4089d30 con 0x7f90b40b3f80 2024-04-02T07:36:57.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.209+0000 7f90b3400700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f909406ac90 0x7f909406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:57.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.210+0000 7f90b3400700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f909406ac90 0x7f909406d150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f90b40cd2f0 tx=0x7f90a0007f90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:57.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.215+0000 7f90abe00700 1 -- 172.21.15.89:0/2489990008 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f90a40552a0 con 0x7f90b40b3f80 2024-04-02T07:36:57.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:57.433+0000 7f90bae00700 1 -- 172.21.15.89:0/2489990008 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f90b40cd860 con 0x7f90b40b3f80 2024-04-02T07:36:58.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.164+0000 7f90abe00700 1 -- 172.21.15.89:0/2489990008 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v30) v1 ==== 97+0+0 (secure 0 0 0) 0x7f90a4015080 con 0x7f90b40b3f80 2024-04-02T07:36:58.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.167+0000 7f90bae00700 1 -- 172.21.15.89:0/2489990008 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f909406ac90 msgr2=0x7f909406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:58.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.167+0000 7f90bae00700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f909406ac90 0x7f909406d150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f90b40cd2f0 tx=0x7f90a0007f90 comp rx=0 tx=0).stop 2024-04-02T07:36:58.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.167+0000 7f90bae00700 1 -- 172.21.15.89:0/2489990008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b40b3f80 msgr2=0x7f90b4153880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:58.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.167+0000 7f90bae00700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b40b3f80 0x7f90b4153880 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f90a4016000 tx=0x7f90a4004750 comp rx=0 tx=0).stop 2024-04-02T07:36:58.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.167+0000 7f90bae00700 1 -- 172.21.15.89:0/2489990008 shutdown_connections 2024-04-02T07:36:58.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.167+0000 7f90bae00700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f909406ac90 0x7f909406d150 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.167+0000 7f90bae00700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b40b3f80 0x7f90b4153880 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.167+0000 7f90bae00700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90b40c98d0 0x7f90b40b6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.167+0000 7f90bae00700 1 --2- 172.21.15.89:0/2489990008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b4160e50 0x7f90b40c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.167+0000 7f90bae00700 1 -- 172.21.15.89:0/2489990008 >> 172.21.15.89:0/2489990008 conn(0x7f90b408aff0 msgr2=0x7f90b406cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:58.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.168+0000 7f90bae00700 1 -- 172.21.15.89:0/2489990008 shutdown_connections 2024-04-02T07:36:58.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.168+0000 7f90bae00700 1 -- 172.21.15.89:0/2489990008 wait complete. 2024-04-02T07:36:58.172 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T07:36:58.190 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T07:36:58.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.381+0000 7f5000c00700 1 Processor -- start 2024-04-02T07:36:58.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.381+0000 7f5000c00700 1 -- start start 2024-04-02T07:36:58.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.382+0000 7f5000c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4ffc15d640 0x7f4ffc0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:58.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.382+0000 7f5000c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ffc1619d0 0x7f4ffc0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:58.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.383+0000 7f5000c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ffc0d30d0 con 0x7f4ffc0ceec0 2024-04-02T07:36:58.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.383+0000 7f5000c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ffc0b6840 con 0x7f4ffc15d640 2024-04-02T07:36:58.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.383+0000 7f5000c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc0d2880 con 0x7f4ffc1619d0 2024-04-02T07:36:58.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.383+0000 7f4ff9e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4ffc15d640 0x7f4ffc0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:58.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.383+0000 7f4ff9e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4ffc15d640 0x7f4ffc0d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53810/0 (socket says 172.21.15.89:53810) 2024-04-02T07:36:58.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.383+0000 7f4ff9e00700 1 -- 172.21.15.89:0/3173356531 learned_addr learned my addr 172.21.15.89:0/3173356531 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:58.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.383+0000 7f4ffa800700 1 --2- 172.21.15.89:0/3173356531 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ffc1619d0 0x7f4ffc0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:58.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.383+0000 7f4ff9e00700 1 -- 172.21.15.89:0/3173356531 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ffc1619d0 msgr2=0x7f4ffc0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:58.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.383+0000 7f4ff9e00700 1 --2- 172.21.15.89:0/3173356531 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ffc1619d0 0x7f4ffc0b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.383+0000 7f4ff9e00700 1 -- 172.21.15.89:0/3173356531 >> v1:172.21.15.113:6789/0 conn(0x7f4ffc0ceec0 legacy=0x7f4ffc1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.383+0000 7f4ff9e00700 1 -- 172.21.15.89:0/3173356531 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc0d2b30 con 0x7f4ffc15d640 2024-04-02T07:36:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.383+0000 7f4ffa800700 1 --2- 172.21.15.89:0/3173356531 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ffc1619d0 0x7f4ffc0b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:36:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.384+0000 7f4ff9e00700 1 --2- 172.21.15.89:0/3173356531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc15d640 0x7f4ffc0d2340 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0098f0 tx=0x7f4fec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=967a1baafb76e653 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.384+0000 7f4ff8a00700 1 -- 172.21.15.89:0/3173356531 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec004230 con 0x7f4ffc15d640 2024-04-02T07:36:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.384+0000 7f4ff8a00700 1 -- 172.21.15.89:0/3173356531 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec004390 con 0x7f4ffc15d640 2024-04-02T07:36:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.384+0000 7f4ff8a00700 1 -- 172.21.15.89:0/3173356531 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec0036a0 con 0x7f4ffc15d640 2024-04-02T07:36:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.384+0000 7f5000c00700 1 -- 172.21.15.89:0/3173356531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc15d640 msgr2=0x7f4ffc0d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.384+0000 7f5000c00700 1 --2- 172.21.15.89:0/3173356531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc15d640 0x7f4ffc0d2340 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0098f0 tx=0x7f4fec009c00 comp rx=0 tx=0).stop 2024-04-02T07:36:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.384+0000 7f5000c00700 1 -- 172.21.15.89:0/3173356531 shutdown_connections 2024-04-02T07:36:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.384+0000 7f5000c00700 1 --2- 172.21.15.89:0/3173356531 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ffc1619d0 0x7f4ffc0b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.384+0000 7f5000c00700 1 --2- 172.21.15.89:0/3173356531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc15d640 0x7f4ffc0d2340 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.384+0000 7f5000c00700 1 -- 172.21.15.89:0/3173356531 >> 172.21.15.89:0/3173356531 conn(0x7f4ffc08aff0 msgr2=0x7f4ffc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.384+0000 7f5000c00700 1 -- 172.21.15.89:0/3173356531 shutdown_connections 2024-04-02T07:36:58.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.384+0000 7f5000c00700 1 -- 172.21.15.89:0/3173356531 wait complete. 2024-04-02T07:36:58.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.386+0000 7f5000c00700 1 Processor -- start 2024-04-02T07:36:58.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.386+0000 7f5000c00700 1 -- start start 2024-04-02T07:36:58.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.387+0000 7f5000c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc0ceec0 0x7f4ffc0c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:58.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.387+0000 7f5000c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ffc1619d0 0x7f4ffc09ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:58.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.387+0000 7f5000c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ffc09f4b0 0x7f4ffc1f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:58.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.387+0000 7f5000c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc15f770 con 0x7f4ffc0ceec0 2024-04-02T07:36:58.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.387+0000 7f5000c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc09fa30 con 0x7f4ffc09f4b0 2024-04-02T07:36:58.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.387+0000 7f5000c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc15ee40 con 0x7f4ffc1619d0 2024-04-02T07:36:58.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.387+0000 7f4ff9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc0ceec0 0x7f4ffc0c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:58.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.387+0000 7f4ff9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc0ceec0 0x7f4ffc0c92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53818/0 (socket says 172.21.15.89:53818) 2024-04-02T07:36:58.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.387+0000 7f4ff9e00700 1 -- 172.21.15.89:0/1625066792 learned_addr learned my addr 172.21.15.89:0/1625066792 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:58.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.388+0000 7f4ffa800700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ffc09f4b0 0x7f4ffc1f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:58.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.388+0000 7f4ff9400700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ffc1619d0 0x7f4ffc09ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:58.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.388+0000 7f4ff9e00700 1 -- 172.21.15.89:0/1625066792 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ffc1619d0 msgr2=0x7f4ffc09ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:58.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.388+0000 7f4ff9e00700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ffc1619d0 0x7f4ffc09ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.388+0000 7f4ff9e00700 1 -- 172.21.15.89:0/1625066792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ffc09f4b0 msgr2=0x7f4ffc1f2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:58.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.388+0000 7f4ff9e00700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ffc09f4b0 0x7f4ffc1f2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.388+0000 7f4ff9e00700 1 -- 172.21.15.89:0/1625066792 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec009550 con 0x7f4ffc0ceec0 2024-04-02T07:36:58.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.388+0000 7f4ff9400700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ffc1619d0 0x7f4ffc09ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:58.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.388+0000 7f4ff9e00700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc0ceec0 0x7f4ffc0c92e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009130 tx=0x7f4fec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:58.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.388+0000 7f4ff2000700 1 -- 172.21.15.89:0/1625066792 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec003d80 con 0x7f4ffc0ceec0 2024-04-02T07:36:58.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.388+0000 7f4ff2000700 1 -- 172.21.15.89:0/1625066792 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec01f070 con 0x7f4ffc0ceec0 2024-04-02T07:36:58.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.388+0000 7f4ff2000700 1 -- 172.21.15.89:0/1625066792 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec00b8a0 con 0x7f4ffc0ceec0 2024-04-02T07:36:58.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.388+0000 7f5000c00700 1 -- 172.21.15.89:0/1625066792 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ffc15f120 con 0x7f4ffc0ceec0 2024-04-02T07:36:58.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.389+0000 7f5000c00700 1 -- 172.21.15.89:0/1625066792 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc150330 con 0x7f4ffc0ceec0 2024-04-02T07:36:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.390+0000 7f5000c00700 1 -- 172.21.15.89:0/1625066792 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ffc067720 con 0x7f4ffc0ceec0 2024-04-02T07:36:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.390+0000 7f4ff2000700 1 -- 172.21.15.89:0/1625066792 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4fec021030 con 0x7f4ffc0ceec0 2024-04-02T07:36:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.391+0000 7f4ff2000700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4fdc06ac90 0x7f4fdc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.391+0000 7f4ff2000700 1 -- 172.21.15.89:0/1625066792 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4fec08a550 con 0x7f4ffc0ceec0 2024-04-02T07:36:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.391+0000 7f4ff9400700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4fdc06ac90 0x7f4fdc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.392+0000 7f4ff9400700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4fdc06ac90 0x7f4fdc06d150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc09edd0 tx=0x7f4fe8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:58.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.396+0000 7f4ff2000700 1 -- 172.21.15.89:0/1625066792 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4fec055b90 con 0x7f4ffc0ceec0 2024-04-02T07:36:58.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.685+0000 7f5000c00700 1 -- 172.21.15.89:0/1625066792 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4ffc067720 con 0x7f4ffc0ceec0 2024-04-02T07:36:58.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.686+0000 7f4ff2000700 1 -- 172.21.15.89:0/1625066792 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f4fec0591b0 con 0x7f4ffc0ceec0 2024-04-02T07:36:58.689 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:36:58.689 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:36:58.689 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T07:36:58.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.689 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T07:36:58.689 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T07:36:58.689 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.690 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:36:58.690 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:36:58.690 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:36:58.690 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.690 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.690 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T07:36:58.690 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T07:36:58.690 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:36:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:36:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:36:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T07:36:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T07:36:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:36:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:36:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:36:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T07:36:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T07:36:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:36:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:36:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:36:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T07:36:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T07:36:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:36:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:36:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T07:36:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T07:36:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:36:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:36:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T07:36:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T07:36:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:36:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:36:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T07:36:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T07:36:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:36:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:36:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T07:36:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T07:36:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:36:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:36:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T07:36:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T07:36:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:36:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:36:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T07:36:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T07:36:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:36:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:36:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T07:36:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T07:36:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:36:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:36:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T07:36:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T07:36:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:36:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:36:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:36:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:36:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T07:36:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T07:36:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:36:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:36:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:36:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:36:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.701 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T07:36:58.701 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T07:36:58.701 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.701 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:36:58.701 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:36:58.701 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:36:58.701 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:36:58.701 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.702 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T07:36:58.702 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T07:36:58.702 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.702 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:36:58.702 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:36:58.702 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:36:58.703 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:36:58.703 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.703 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T07:36:58.703 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T07:36:58.703 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.703 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:36:58.703 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:36:58.703 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:36:58.703 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.703 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.704 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.bootstrap-mds", 2024-04-02T07:36:58.704 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQA/tQtmfPAYOBAAk6h3u1DtxcqLqQmHhVdF1g==", 2024-04-02T07:36:58.704 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.704 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile bootstrap-mds" 2024-04-02T07:36:58.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.704 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.bootstrap-mgr", 2024-04-02T07:36:58.704 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQA/tQtmFv0YOBAANuprReYZMR/p9AUA/kpS6w==", 2024-04-02T07:36:58.704 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.704 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-02T07:36:58.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.705 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.bootstrap-osd", 2024-04-02T07:36:58.705 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQA/tQtmCAYZOBAAjN7ttqAXMj68jrpXEH+E1w==", 2024-04-02T07:36:58.705 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.705 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile bootstrap-osd" 2024-04-02T07:36:58.705 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.705 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.bootstrap-rbd", 2024-04-02T07:36:58.705 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQA/tQtmoA4ZOBAAOc/g7dOoNSNaH1oRtUSQ7g==", 2024-04-02T07:36:58.705 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.705 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-02T07:36:58.705 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.706 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-02T07:36:58.706 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQA/tQtmYxcZOBAAB/4HCY0OCuZ5hgORr/KoOg==", 2024-04-02T07:36:58.706 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.706 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-02T07:36:58.706 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.706 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.bootstrap-rgw", 2024-04-02T07:36:58.706 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQA/tQtmKiAZOBAAGdTnyBot1UJiZfmt4Uv6lQ==", 2024-04-02T07:36:58.706 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.706 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-02T07:36:58.707 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.707 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T07:36:58.707 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T07:36:58.707 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.707 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:36:58.707 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:36:58.707 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.707 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:36:58.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:36:58.707 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T07:36:58.707 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T07:36:58.708 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:36:58.708 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:36:58.708 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:36:58.708 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:36:58.708 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.708 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:36:58.708 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T07:36:58.708 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:36:58.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.688+0000 7f5000c00700 1 -- 172.21.15.89:0/1625066792 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4fdc06ac90 msgr2=0x7f4fdc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:58.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.688+0000 7f5000c00700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4fdc06ac90 0x7f4fdc06d150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc09edd0 tx=0x7f4fe8009040 comp rx=0 tx=0).stop 2024-04-02T07:36:58.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.688+0000 7f5000c00700 1 -- 172.21.15.89:0/1625066792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc0ceec0 msgr2=0x7f4ffc0c92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:58.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.688+0000 7f5000c00700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc0ceec0 0x7f4ffc0c92e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009130 tx=0x7f4fec004750 comp rx=0 tx=0).stop 2024-04-02T07:36:58.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.688+0000 7f5000c00700 1 -- 172.21.15.89:0/1625066792 shutdown_connections 2024-04-02T07:36:58.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.688+0000 7f5000c00700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4fdc06ac90 0x7f4fdc06d150 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.688+0000 7f5000c00700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ffc09f4b0 0x7f4ffc1f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.688+0000 7f5000c00700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc0ceec0 0x7f4ffc0c92e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.688+0000 7f5000c00700 1 --2- 172.21.15.89:0/1625066792 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ffc1619d0 0x7f4ffc09ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.688+0000 7f5000c00700 1 -- 172.21.15.89:0/1625066792 >> 172.21.15.89:0/1625066792 conn(0x7f4ffc08aff0 msgr2=0x7f4ffc0b8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:58.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.689+0000 7f5000c00700 1 -- 172.21.15.89:0/1625066792 shutdown_connections 2024-04-02T07:36:58.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.689+0000 7f5000c00700 1 -- 172.21.15.89:0/1625066792 wait complete. 2024-04-02T07:36:58.712 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-02T07:36:58.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.900+0000 7fde17000700 1 Processor -- start 2024-04-02T07:36:58.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.900+0000 7fde17000700 1 -- start start 2024-04-02T07:36:58.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.901+0000 7fde17000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fde1015a5c0 0x7fde100709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:58.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.901+0000 7fde17000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde100cb310 con 0x7fde1015dfe0 2024-04-02T07:36:58.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.901+0000 7fde17000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde10070f00 con 0x7fde100ce840 2024-04-02T07:36:58.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.901+0000 7fde17000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde100711a0 con 0x7fde1015a5c0 2024-04-02T07:36:58.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.902+0000 7fde0f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fde1015dfe0 0x7fde100ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47034/0 (socket says 172.21.15.89:47034) 2024-04-02T07:36:58.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.902+0000 7fde0f400700 1 -- 172.21.15.89:0/3866939575 learned_addr learned my addr 172.21.15.89:0/3866939575 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:58.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.902+0000 7fde0fe00700 1 --2- 172.21.15.89:0/3866939575 >> v2:172.21.15.113:3300/0 conn(0x7fde1015a5c0 0x7fde100709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:58.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.902+0000 7fde0ea00700 1 -- 172.21.15.89:0/3866939575 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141019500 0 0) 0x7fde100cb310 con 0x7fde1015dfe0 2024-04-02T07:36:58.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.902+0000 7fde0ea00700 1 -- 172.21.15.89:0/3866939575 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddf40035b0 con 0x7fde1015dfe0 2024-04-02T07:36:58.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.902+0000 7fde0ea00700 1 -- 172.21.15.89:0/3866939575 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875377841 0 0) 0x7fde10070f00 con 0x7fde100ce840 2024-04-02T07:36:58.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.902+0000 7fde0ea00700 1 -- 172.21.15.89:0/3866939575 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde100cb310 con 0x7fde100ce840 2024-04-02T07:36:58.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.902+0000 7fde0ea00700 1 -- 172.21.15.89:0/3866939575 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 422614374 0 0) 0x7fddf40035b0 con 0x7fde1015dfe0 2024-04-02T07:36:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.902+0000 7fde0fe00700 1 -- 172.21.15.89:0/3866939575 >> v1:172.21.15.113:6789/0 conn(0x7fde100ce840 legacy=0x7fde100cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.902+0000 7fde0fe00700 1 -- 172.21.15.89:0/3866939575 >> v1:172.21.15.89:6789/0 conn(0x7fde1015dfe0 legacy=0x7fde100ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.902+0000 7fde0fe00700 1 -- 172.21.15.89:0/3866939575 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde10071480 con 0x7fde1015a5c0 2024-04-02T07:36:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.903+0000 7fde0fe00700 1 --2- 172.21.15.89:0/3866939575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde1015a5c0 0x7fde100709c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fde00009840 tx=0x7fde00009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7795e29e44823a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.903+0000 7fde0ea00700 1 -- 172.21.15.89:0/3866939575 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc001670 con 0x7fde1015a5c0 2024-04-02T07:36:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.903+0000 7fde0ea00700 1 -- 172.21.15.89:0/3866939575 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde00004030 con 0x7fde1015a5c0 2024-04-02T07:36:58.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.903+0000 7fde0ea00700 1 -- 172.21.15.89:0/3866939575 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde000043c0 con 0x7fde1015a5c0 2024-04-02T07:36:58.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.903+0000 7fde17000700 1 -- 172.21.15.89:0/3866939575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde1015a5c0 msgr2=0x7fde100709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:58.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.903+0000 7fde17000700 1 --2- 172.21.15.89:0/3866939575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde1015a5c0 0x7fde100709c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fde00009840 tx=0x7fde00009b50 comp rx=0 tx=0).stop 2024-04-02T07:36:58.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.903+0000 7fde17000700 1 -- 172.21.15.89:0/3866939575 shutdown_connections 2024-04-02T07:36:58.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.903+0000 7fde17000700 1 --2- 172.21.15.89:0/3866939575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde1015a5c0 0x7fde100709c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.903+0000 7fde17000700 1 -- 172.21.15.89:0/3866939575 >> 172.21.15.89:0/3866939575 conn(0x7fde100b0d70 msgr2=0x7fde100b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.904+0000 7fde17000700 1 -- 172.21.15.89:0/3866939575 shutdown_connections 2024-04-02T07:36:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.904+0000 7fde17000700 1 -- 172.21.15.89:0/3866939575 wait complete. 2024-04-02T07:36:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.905+0000 7fde17000700 1 Processor -- start 2024-04-02T07:36:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.905+0000 7fde17000700 1 -- start start 2024-04-02T07:36:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.906+0000 7fde17000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde100ce840 0x7fde10149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.906+0000 7fde17000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde1015a5c0 0x7fde100afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.906+0000 7fde17000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde1015dfe0 0x7fde100b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.906+0000 7fde17000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1015bdd0 con 0x7fde1015a5c0 2024-04-02T07:36:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.906+0000 7fde17000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1014dd00 con 0x7fde1015dfe0 2024-04-02T07:36:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.906+0000 7fde17000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1014dfe0 con 0x7fde100ce840 2024-04-02T07:36:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.906+0000 7fde0f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde1015a5c0 0x7fde100afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.907+0000 7fde0f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde1015a5c0 0x7fde100afd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60984/0 (socket says 172.21.15.89:60984) 2024-04-02T07:36:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.907+0000 7fde0f400700 1 -- 172.21.15.89:0/292903581 learned_addr learned my addr 172.21.15.89:0/292903581 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:58.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.907+0000 7fde15000700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde1015dfe0 0x7fde100b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:58.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.907+0000 7fde0fe00700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde100ce840 0x7fde10149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:58.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.907+0000 7fde0f400700 1 -- 172.21.15.89:0/292903581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde100ce840 msgr2=0x7fde10149430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:58.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.907+0000 7fde0f400700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde100ce840 0x7fde10149430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.907+0000 7fde0f400700 1 -- 172.21.15.89:0/292903581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde1015dfe0 msgr2=0x7fde100b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:58.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.907+0000 7fde0f400700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde1015dfe0 0x7fde100b0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:58.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.907+0000 7fde0f400700 1 -- 172.21.15.89:0/292903581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde000094a0 con 0x7fde1015a5c0 2024-04-02T07:36:58.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.907+0000 7fde0fe00700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde100ce840 0x7fde10149430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:58.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.907+0000 7fde0f400700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde1015a5c0 0x7fde100afd30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fddfc00c6b0 tx=0x7fddfc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:58.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.907+0000 7fde07e00700 1 -- 172.21.15.89:0/292903581 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc007620 con 0x7fde1015a5c0 2024-04-02T07:36:58.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.907+0000 7fde07e00700 1 -- 172.21.15.89:0/292903581 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc004870 con 0x7fde1015a5c0 2024-04-02T07:36:58.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.907+0000 7fde17000700 1 -- 172.21.15.89:0/292903581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1015c040 con 0x7fde1015a5c0 2024-04-02T07:36:58.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.908+0000 7fde07e00700 1 -- 172.21.15.89:0/292903581 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc004c00 con 0x7fde1015a5c0 2024-04-02T07:36:58.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.908+0000 7fde17000700 1 -- 172.21.15.89:0/292903581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1014e2c0 con 0x7fde1015a5c0 2024-04-02T07:36:58.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.909+0000 7fde17000700 1 -- 172.21.15.89:0/292903581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1006c550 con 0x7fde1015a5c0 2024-04-02T07:36:58.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.909+0000 7fde07e00700 1 -- 172.21.15.89:0/292903581 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fddfc0133c0 con 0x7fde1015a5c0 2024-04-02T07:36:58.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.910+0000 7fde07e00700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fddf006ac90 0x7fddf006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:58.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.910+0000 7fde07e00700 1 -- 172.21.15.89:0/292903581 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fddfc088d80 con 0x7fde1015a5c0 2024-04-02T07:36:58.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.911+0000 7fde0fe00700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fddf006ac90 0x7fddf006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:58.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.911+0000 7fde0fe00700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fddf006ac90 0x7fddf006d150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fde000090b0 tx=0x7fde00008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:58.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:58.915+0000 7fde07e00700 1 -- 172.21.15.89:0/292903581 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fddfc054340 con 0x7fde1015a5c0 2024-04-02T07:36:59.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.197+0000 7fde17000700 1 -- 172.21.15.89:0/292903581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fde10067720 con 0x7fde1015a5c0 2024-04-02T07:36:59.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.201+0000 7fde07e00700 1 -- 172.21.15.89:0/292903581 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fddfc057960 con 0x7fde1015a5c0 2024-04-02T07:36:59.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.204+0000 7fde17000700 1 -- 172.21.15.89:0/292903581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fddf006ac90 msgr2=0x7fddf006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:59.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.204+0000 7fde17000700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fddf006ac90 0x7fddf006d150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fde000090b0 tx=0x7fde00008030 comp rx=0 tx=0).stop 2024-04-02T07:36:59.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.204+0000 7fde17000700 1 -- 172.21.15.89:0/292903581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde1015a5c0 msgr2=0x7fde100afd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:59.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.204+0000 7fde17000700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde1015a5c0 0x7fde100afd30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fddfc00c6b0 tx=0x7fddfc00c9c0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.204+0000 7fde17000700 1 -- 172.21.15.89:0/292903581 shutdown_connections 2024-04-02T07:36:59.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.204+0000 7fde17000700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fddf006ac90 0x7fddf006d150 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.204+0000 7fde17000700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde1015dfe0 0x7fde100b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.204+0000 7fde17000700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde100ce840 0x7fde10149430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.204+0000 7fde17000700 1 --2- 172.21.15.89:0/292903581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde1015a5c0 0x7fde100afd30 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.204+0000 7fde17000700 1 -- 172.21.15.89:0/292903581 >> 172.21.15.89:0/292903581 conn(0x7fde100b0d70 msgr2=0x7fde100b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:59.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.204+0000 7fde17000700 1 -- 172.21.15.89:0/292903581 shutdown_connections 2024-04-02T07:36:59.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.204+0000 7fde17000700 1 -- 172.21.15.89:0/292903581 wait complete. 2024-04-02T07:36:59.225 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-02T07:36:59.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.416+0000 7f7682400700 1 Processor -- start 2024-04-02T07:36:59.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.416+0000 7f7682400700 1 -- start start 2024-04-02T07:36:59.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.417+0000 7f7682400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f767c15d9d0 0x7f767c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.417+0000 7f7682400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f767c0d16a0 0x7f767c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.417+0000 7f7682400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f767c0d2370 0x7f767c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.417+0000 7f7682400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f767c0d2820 con 0x7f767c0d16a0 2024-04-02T07:36:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.417+0000 7f7682400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f767c161420 con 0x7f767c0d2370 2024-04-02T07:36:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.417+0000 7f7682400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767c161700 con 0x7f767c15d9d0 2024-04-02T07:36:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.417+0000 7f7673e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f767c0d16a0 0x7f767c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.417+0000 7f7673e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f767c0d16a0 0x7f767c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60994/0 (socket says 172.21.15.89:60994) 2024-04-02T07:36:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.417+0000 7f7673e00700 1 -- 172.21.15.89:0/2651956636 learned_addr learned my addr 172.21.15.89:0/2651956636 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.417+0000 7f767be00700 1 --2- 172.21.15.89:0/2651956636 >> v2:172.21.15.113:3300/0 conn(0x7f767c0d2370 0x7f767c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.417+0000 7f767b400700 1 --2- 172.21.15.89:0/2651956636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f767c15d9d0 0x7f767c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.418+0000 7f7673e00700 1 -- 172.21.15.89:0/2651956636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f767c15d9d0 msgr2=0x7f767c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.418+0000 7f7673e00700 1 --2- 172.21.15.89:0/2651956636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f767c15d9d0 0x7f767c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.418+0000 7f7673e00700 1 -- 172.21.15.89:0/2651956636 >> v2:172.21.15.113:3300/0 conn(0x7f767c0d2370 msgr2=0x7f767c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.418+0000 7f7673e00700 1 --2- 172.21.15.89:0/2651956636 >> v2:172.21.15.113:3300/0 conn(0x7f767c0d2370 0x7f767c15ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.418+0000 7f7673e00700 1 -- 172.21.15.89:0/2651956636 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c1619e0 con 0x7f767c0d16a0 2024-04-02T07:36:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.418+0000 7f7673e00700 1 --2- 172.21.15.89:0/2651956636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f767c0d16a0 0x7f767c0d1e30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f767c0cdc00 tx=0x7f7664009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4082edf8a5a03c1a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:59.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.418+0000 7f767aa00700 1 -- 172.21.15.89:0/2651956636 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7664005e30 con 0x7f767c0d16a0 2024-04-02T07:36:59.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.418+0000 7f767aa00700 1 -- 172.21.15.89:0/2651956636 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7664004030 con 0x7f767c0d16a0 2024-04-02T07:36:59.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.418+0000 7f767aa00700 1 -- 172.21.15.89:0/2651956636 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76640043e0 con 0x7f767c0d16a0 2024-04-02T07:36:59.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.419+0000 7f7682400700 1 -- 172.21.15.89:0/2651956636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f767c0d16a0 msgr2=0x7f767c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:59.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.419+0000 7f7682400700 1 --2- 172.21.15.89:0/2651956636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f767c0d16a0 0x7f767c0d1e30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f767c0cdc00 tx=0x7f7664009bd0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.419+0000 7f7682400700 1 -- 172.21.15.89:0/2651956636 shutdown_connections 2024-04-02T07:36:59.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.419+0000 7f7682400700 1 --2- 172.21.15.89:0/2651956636 >> v2:172.21.15.113:3300/0 conn(0x7f767c0d2370 0x7f767c15ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.419+0000 7f7682400700 1 --2- 172.21.15.89:0/2651956636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f767c15d9d0 0x7f767c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.419+0000 7f7682400700 1 --2- 172.21.15.89:0/2651956636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f767c0d16a0 0x7f767c0d1e30 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.419+0000 7f7682400700 1 -- 172.21.15.89:0/2651956636 >> 172.21.15.89:0/2651956636 conn(0x7f767c08aff0 msgr2=0x7f767c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:59.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.419+0000 7f7682400700 1 -- 172.21.15.89:0/2651956636 shutdown_connections 2024-04-02T07:36:59.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.419+0000 7f7682400700 1 -- 172.21.15.89:0/2651956636 wait complete. 2024-04-02T07:36:59.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.421+0000 7f7682400700 1 Processor -- start 2024-04-02T07:36:59.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.421+0000 7f7682400700 1 -- start start 2024-04-02T07:36:59.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f7682400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f767c0d16a0 0x7f767c0cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f7682400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f767c0d2370 0x7f767c0b86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f7682400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f767c15d9d0 0x7f767c152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f7682400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c0b8e90 con 0x7f767c0d16a0 2024-04-02T07:36:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f7682400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c0b8c10 con 0x7f767c0d2370 2024-04-02T07:36:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f7682400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767c0b6b60 con 0x7f767c15d9d0 2024-04-02T07:36:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f767b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f767c0d16a0 0x7f767c0cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f767b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f767c0d16a0 0x7f767c0cb920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60996/0 (socket says 172.21.15.89:60996) 2024-04-02T07:36:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f767b400700 1 -- 172.21.15.89:0/793742531 learned_addr learned my addr 172.21.15.89:0/793742531 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f7673e00700 1 --2- 172.21.15.89:0/793742531 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f767c0d2370 0x7f767c0b86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f767be00700 1 --2- 172.21.15.89:0/793742531 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f767c15d9d0 0x7f767c152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f767b400700 1 -- 172.21.15.89:0/793742531 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f767c15d9d0 msgr2=0x7f767c152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f767b400700 1 --2- 172.21.15.89:0/793742531 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f767c15d9d0 0x7f767c152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f767b400700 1 -- 172.21.15.89:0/793742531 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f767c0d2370 msgr2=0x7f767c0b86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f767b400700 1 --2- 172.21.15.89:0/793742531 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f767c0d2370 0x7f767c0b86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.422+0000 7f767b400700 1 -- 172.21.15.89:0/793742531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7664009500 con 0x7f767c0d16a0 2024-04-02T07:36:59.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.423+0000 7f767b400700 1 --2- 172.21.15.89:0/793742531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f767c0d16a0 0x7f767c0cb920 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f766c00d800 tx=0x7f766c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:59.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.423+0000 7f7673400700 1 -- 172.21.15.89:0/793742531 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c0125f0 con 0x7f767c0d16a0 2024-04-02T07:36:59.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.423+0000 7f7673400700 1 -- 172.21.15.89:0/793742531 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766c012750 con 0x7f767c0d16a0 2024-04-02T07:36:59.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.423+0000 7f7673400700 1 -- 172.21.15.89:0/793742531 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c012b00 con 0x7f767c0d16a0 2024-04-02T07:36:59.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.423+0000 7f7682400700 1 -- 172.21.15.89:0/793742531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767c0b6e40 con 0x7f767c0d16a0 2024-04-02T07:36:59.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.423+0000 7f7682400700 1 -- 172.21.15.89:0/793742531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767c0b7400 con 0x7f767c0d16a0 2024-04-02T07:36:59.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.424+0000 7f7682400700 1 -- 172.21.15.89:0/793742531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767c0d1ca0 con 0x7f767c0d16a0 2024-04-02T07:36:59.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.425+0000 7f7673400700 1 -- 172.21.15.89:0/793742531 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f766c012c60 con 0x7f767c0d16a0 2024-04-02T07:36:59.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.426+0000 7f7673400700 1 --2- 172.21.15.89:0/793742531 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f765806ac90 0x7f765806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.426+0000 7f7673400700 1 -- 172.21.15.89:0/793742531 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f766c0892e0 con 0x7f767c0d16a0 2024-04-02T07:36:59.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.426+0000 7f7673e00700 1 --2- 172.21.15.89:0/793742531 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f765806ac90 0x7f765806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.427+0000 7f7673e00700 1 --2- 172.21.15.89:0/793742531 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f765806ac90 0x7f765806d150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f767c0cdc00 tx=0x7f7664008070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:59.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.430+0000 7f7673400700 1 -- 172.21.15.89:0/793742531 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f766c054800 con 0x7f767c0d16a0 2024-04-02T07:36:59.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.715+0000 7f7682400700 1 -- 172.21.15.89:0/793742531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f767c0b75f0 con 0x7f767c0d16a0 2024-04-02T07:36:59.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.719+0000 7f7673400700 1 -- 172.21.15.89:0/793742531 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f766c057e20 con 0x7f767c0d16a0 2024-04-02T07:36:59.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.721+0000 7f7682400700 1 -- 172.21.15.89:0/793742531 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f765806ac90 msgr2=0x7f765806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:59.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.721+0000 7f7682400700 1 --2- 172.21.15.89:0/793742531 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f765806ac90 0x7f765806d150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f767c0cdc00 tx=0x7f7664008070 comp rx=0 tx=0).stop 2024-04-02T07:36:59.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.721+0000 7f7682400700 1 -- 172.21.15.89:0/793742531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f767c0d16a0 msgr2=0x7f767c0cb920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:59.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.721+0000 7f7682400700 1 --2- 172.21.15.89:0/793742531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f767c0d16a0 0x7f767c0cb920 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f766c00d800 tx=0x7f766c00db10 comp rx=0 tx=0).stop 2024-04-02T07:36:59.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.721+0000 7f7682400700 1 -- 172.21.15.89:0/793742531 shutdown_connections 2024-04-02T07:36:59.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.721+0000 7f7682400700 1 --2- 172.21.15.89:0/793742531 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f765806ac90 0x7f765806d150 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.721+0000 7f7682400700 1 --2- 172.21.15.89:0/793742531 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f767c15d9d0 0x7f767c152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.721+0000 7f7682400700 1 --2- 172.21.15.89:0/793742531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f767c0d16a0 0x7f767c0cb920 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.721+0000 7f7682400700 1 --2- 172.21.15.89:0/793742531 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f767c0d2370 0x7f767c0b86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.721+0000 7f7682400700 1 -- 172.21.15.89:0/793742531 >> 172.21.15.89:0/793742531 conn(0x7f767c08aff0 msgr2=0x7f767c06d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:59.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.721+0000 7f7682400700 1 -- 172.21.15.89:0/793742531 shutdown_connections 2024-04-02T07:36:59.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.721+0000 7f7682400700 1 -- 172.21.15.89:0/793742531 wait complete. 2024-04-02T07:36:59.742 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-02T07:36:59.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.929+0000 7f8815000700 1 Processor -- start 2024-04-02T07:36:59.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.930+0000 7f8815000700 1 -- start start 2024-04-02T07:36:59.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.931+0000 7f8815000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88100d16a0 0x7f88100d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.931+0000 7f8815000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8810161420 con 0x7f881015d9d0 2024-04-02T07:36:59.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.931+0000 7f8815000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8810161700 con 0x7f88100d2370 2024-04-02T07:36:59.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.931+0000 7f8815000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88100d2820 con 0x7f88100d16a0 2024-04-02T07:36:59.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.931+0000 7f880de00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f881015d9d0 0x7f88100d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47050/0 (socket says 172.21.15.89:47050) 2024-04-02T07:36:59.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.931+0000 7f880de00700 1 -- 172.21.15.89:0/2345073735 learned_addr learned my addr 172.21.15.89:0/2345073735 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:36:59.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.931+0000 7f880ca00700 1 -- 172.21.15.89:0/2345073735 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143067666 0 0) 0x7f8810161420 con 0x7f881015d9d0 2024-04-02T07:36:59.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.931+0000 7f880ca00700 1 -- 172.21.15.89:0/2345073735 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f40035b0 con 0x7f881015d9d0 2024-04-02T07:36:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.931+0000 7f880d400700 1 --2- 172.21.15.89:0/2345073735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88100d16a0 0x7f88100d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880ca00700 1 -- 172.21.15.89:0/2345073735 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3974445428 0 0) 0x7f87f40035b0 con 0x7f881015d9d0 2024-04-02T07:36:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880ca00700 1 -- 172.21.15.89:0/2345073735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88100d16a0 msgr2=0x7f88100d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880ca00700 1 --2- 172.21.15.89:0/2345073735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88100d16a0 0x7f88100d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880ca00700 1 -- 172.21.15.89:0/2345073735 >> v1:172.21.15.113:6789/0 conn(0x7f88100d2370 legacy=0x7f88100ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880ca00700 1 -- 172.21.15.89:0/2345073735 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88101619e0 con 0x7f881015d9d0 2024-04-02T07:36:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880ca00700 1 -- 172.21.15.89:0/2345073735 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f8800001670 con 0x7f881015d9d0 2024-04-02T07:36:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880ca00700 1 -- 172.21.15.89:0/2345073735 >> v1:172.21.15.89:6789/0 conn(0x7f881015d9d0 legacy=0x7f88100d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880ca00700 1 --2- 172.21.15.89:0/2345073735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87f4003a30 0x7f87f4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880ca00700 1 --2- 172.21.15.89:0/2345073735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f4004850 0x7f87f4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880ca00700 1 --2- 172.21.15.89:0/2345073735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88100d16a0 0x7f87f4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880ca00700 1 -- 172.21.15.89:0/2345073735 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100d2820 con 0x7f87f4004850 2024-04-02T07:36:59.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880ca00700 1 -- 172.21.15.89:0/2345073735 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f400bac0 con 0x7f87f4003a30 2024-04-02T07:36:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880ca00700 1 -- 172.21.15.89:0/2345073735 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f400bc80 con 0x7f88100d16a0 2024-04-02T07:36:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880d400700 1 --2- 172.21.15.89:0/2345073735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f4004850 0x7f87f4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880de00700 1 --2- 172.21.15.89:0/2345073735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87f4003a30 0x7f87f4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.932+0000 7f880e800700 1 --2- 172.21.15.89:0/2345073735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88100d16a0 0x7f87f4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.933+0000 7f880d400700 1 -- 172.21.15.89:0/2345073735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88100d16a0 msgr2=0x7f87f4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.933+0000 7f880d400700 1 --2- 172.21.15.89:0/2345073735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88100d16a0 0x7f87f4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.933+0000 7f880d400700 1 -- 172.21.15.89:0/2345073735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87f4003a30 msgr2=0x7f87f4003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:36:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.933+0000 7f880d400700 1 --2- 172.21.15.89:0/2345073735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87f4003a30 0x7f87f4003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.933+0000 7f880d400700 1 -- 172.21.15.89:0/2345073735 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f40035b0 con 0x7f87f4004850 2024-04-02T07:36:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.933+0000 7f880d400700 1 --2- 172.21.15.89:0/2345073735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f4004850 0x7f87f4004cd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f88100d1ae0 tx=0x7f87f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=911524c741f11888 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.933+0000 7f880ca00700 1 -- 172.21.15.89:0/2345073735 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87fc005a20 con 0x7f87f4004850 2024-04-02T07:36:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.933+0000 7f880ca00700 1 -- 172.21.15.89:0/2345073735 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87fc005b80 con 0x7f87f4004850 2024-04-02T07:36:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.933+0000 7f880ca00700 1 -- 172.21.15.89:0/2345073735 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87fc00e6c0 con 0x7f87f4004850 2024-04-02T07:36:59.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.934+0000 7f8815000700 1 -- 172.21.15.89:0/2345073735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f4004850 msgr2=0x7f87f4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:59.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.934+0000 7f8815000700 1 --2- 172.21.15.89:0/2345073735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f4004850 0x7f87f4004cd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f88100d1ae0 tx=0x7f87f4005880 comp rx=0 tx=0).stop 2024-04-02T07:36:59.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.934+0000 7f880e800700 1 -- 172.21.15.89:0/2345073735 reap_dead start 2024-04-02T07:36:59.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.934+0000 7f8815000700 1 -- 172.21.15.89:0/2345073735 shutdown_connections 2024-04-02T07:36:59.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.934+0000 7f8815000700 1 -- 172.21.15.89:0/2345073735 >> 172.21.15.89:0/2345073735 conn(0x7f881008aff0 msgr2=0x7f88100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:36:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.934+0000 7f8815000700 1 -- 172.21.15.89:0/2345073735 shutdown_connections 2024-04-02T07:36:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.934+0000 7f8815000700 1 -- 172.21.15.89:0/2345073735 wait complete. 2024-04-02T07:36:59.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.936+0000 7f8815000700 1 Processor -- start 2024-04-02T07:36:59.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.936+0000 7f8815000700 1 -- start start 2024-04-02T07:36:59.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f8815000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87f4004850 0x7f88100b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f8815000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88100b46a0 0x7f881015f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f8815000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881015a7f0 0x7f881015ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f8815000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881015b1b0 con 0x7f881015a7f0 2024-04-02T07:36:59.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f8815000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100b5810 con 0x7f88100b46a0 2024-04-02T07:36:59.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f8815000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88100b4450 con 0x7f87f4004850 2024-04-02T07:36:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f880e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881015a7f0 0x7f881015ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f880e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881015a7f0 0x7f881015ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32774/0 (socket says 172.21.15.89:32774) 2024-04-02T07:36:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f880e800700 1 -- 172.21.15.89:0/3804707817 learned_addr learned my addr 172.21.15.89:0/3804707817 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:36:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f880de00700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87f4004850 0x7f88100b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f880d400700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88100b46a0 0x7f881015f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f880e800700 1 -- 172.21.15.89:0/3804707817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87f4004850 msgr2=0x7f88100b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f880e800700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87f4004850 0x7f88100b3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f880e800700 1 -- 172.21.15.89:0/3804707817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88100b46a0 msgr2=0x7f881015f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:36:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f880e800700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88100b46a0 0x7f881015f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:36:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f880e800700 1 -- 172.21.15.89:0/3804707817 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88100b4c00 con 0x7f881015a7f0 2024-04-02T07:36:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.937+0000 7f880de00700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87f4004850 0x7f88100b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:36:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.938+0000 7f880e800700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881015a7f0 0x7f881015ac70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f88100ca580 tx=0x7f880800d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:59.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.938+0000 7f8806000700 1 -- 172.21.15.89:0/3804707817 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808003c10 con 0x7f881015a7f0 2024-04-02T07:36:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.938+0000 7f8806000700 1 -- 172.21.15.89:0/3804707817 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8808003d90 con 0x7f881015a7f0 2024-04-02T07:36:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.938+0000 7f8806000700 1 -- 172.21.15.89:0/3804707817 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880800da10 con 0x7f881015a7f0 2024-04-02T07:36:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.938+0000 7f8815000700 1 -- 172.21.15.89:0/3804707817 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88100b4ee0 con 0x7f881015a7f0 2024-04-02T07:36:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.938+0000 7f8815000700 1 -- 172.21.15.89:0/3804707817 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88100b54a0 con 0x7f881015a7f0 2024-04-02T07:36:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.939+0000 7f8815000700 1 -- 172.21.15.89:0/3804707817 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88100b5660 con 0x7f881015a7f0 2024-04-02T07:36:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.940+0000 7f8806000700 1 -- 172.21.15.89:0/3804707817 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f880800db70 con 0x7f881015a7f0 2024-04-02T07:36:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.941+0000 7f8806000700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f87f006ac90 0x7f87f006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:36:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.941+0000 7f8806000700 1 -- 172.21.15.89:0/3804707817 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8808089280 con 0x7f881015a7f0 2024-04-02T07:36:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.941+0000 7f880de00700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f87f006ac90 0x7f87f006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:36:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.942+0000 7f880de00700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f87f006ac90 0x7f87f006d150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8800000c00 tx=0x7f8800009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:36:59.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:36:59.945+0000 7f8806000700 1 -- 172.21.15.89:0/3804707817 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8808054710 con 0x7f881015a7f0 2024-04-02T07:37:00.133 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-04-02T07:37:00.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.230+0000 7f8815000700 1 -- 172.21.15.89:0/3804707817 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f8810067720 con 0x7f881015a7f0 2024-04-02T07:37:00.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.234+0000 7f8806000700 1 -- 172.21.15.89:0/3804707817 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8808057d30 con 0x7f881015a7f0 2024-04-02T07:37:00.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.236+0000 7f8815000700 1 -- 172.21.15.89:0/3804707817 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f87f006ac90 msgr2=0x7f87f006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:00.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.236+0000 7f8815000700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f87f006ac90 0x7f87f006d150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8800000c00 tx=0x7f8800009040 comp rx=0 tx=0).stop 2024-04-02T07:37:00.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.236+0000 7f8815000700 1 -- 172.21.15.89:0/3804707817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881015a7f0 msgr2=0x7f881015ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:00.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.236+0000 7f8815000700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881015a7f0 0x7f881015ac70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f88100ca580 tx=0x7f880800d150 comp rx=0 tx=0).stop 2024-04-02T07:37:00.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.236+0000 7f8815000700 1 -- 172.21.15.89:0/3804707817 shutdown_connections 2024-04-02T07:37:00.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.236+0000 7f8815000700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f87f006ac90 0x7f87f006d150 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.236+0000 7f8815000700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881015a7f0 0x7f881015ac70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.236+0000 7f8815000700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87f4004850 0x7f88100b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.236+0000 7f8815000700 1 --2- 172.21.15.89:0/3804707817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88100b46a0 0x7f881015f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.236+0000 7f8815000700 1 -- 172.21.15.89:0/3804707817 >> 172.21.15.89:0/3804707817 conn(0x7f881008aff0 msgr2=0x7f88100b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:00.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.236+0000 7f8815000700 1 -- 172.21.15.89:0/3804707817 shutdown_connections 2024-04-02T07:37:00.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.236+0000 7f8815000700 1 -- 172.21.15.89:0/3804707817 wait complete. 2024-04-02T07:37:00.257 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-02T07:37:00.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.446+0000 7f8732600700 1 Processor -- start 2024-04-02T07:37:00.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.447+0000 7f8732600700 1 -- start start 2024-04-02T07:37:00.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f8732600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c15a5c0 0x7f872c0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:00.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f8732600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f872c15dfe0 0x7f872c0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:00.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f8732600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f872c0ce670 0x7f872c0b2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:00.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f8732600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f872c0ceb00 con 0x7f872c0ce670 2024-04-02T07:37:00.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f8732600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f872c0cb310 con 0x7f872c15dfe0 2024-04-02T07:37:00.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f8732600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872c070f00 con 0x7f872c15a5c0 2024-04-02T07:37:00.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f872be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f872c0ce670 0x7f872c0b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:00.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f872be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f872c0ce670 0x7f872c0b2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32778/0 (socket says 172.21.15.89:32778) 2024-04-02T07:37:00.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f872be00700 1 -- 172.21.15.89:0/1909401637 learned_addr learned my addr 172.21.15.89:0/1909401637 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:00.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f872aa00700 1 --2- 172.21.15.89:0/1909401637 >> v2:172.21.15.113:3300/0 conn(0x7f872c15dfe0 0x7f872c0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:00.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f872b400700 1 --2- 172.21.15.89:0/1909401637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c15a5c0 0x7f872c0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:00.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f872be00700 1 -- 172.21.15.89:0/1909401637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c15a5c0 msgr2=0x7f872c0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:00.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f872be00700 1 --2- 172.21.15.89:0/1909401637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c15a5c0 0x7f872c0709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f872be00700 1 -- 172.21.15.89:0/1909401637 >> v2:172.21.15.113:3300/0 conn(0x7f872c15dfe0 msgr2=0x7f872c0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:00.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.448+0000 7f872be00700 1 --2- 172.21.15.89:0/1909401637 >> v2:172.21.15.113:3300/0 conn(0x7f872c15dfe0 0x7f872c0ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.449+0000 7f872be00700 1 -- 172.21.15.89:0/1909401637 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c0711a0 con 0x7f872c0ce670 2024-04-02T07:37:00.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.449+0000 7f872be00700 1 --2- 172.21.15.89:0/1909401637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0ce670 0x7f872c0b2640 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f872c0cf080 tx=0x7f872400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded1a047ada0a1dc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:00.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.449+0000 7f872a000700 1 -- 172.21.15.89:0/1909401637 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872400e070 con 0x7f872c0ce670 2024-04-02T07:37:00.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.449+0000 7f872a000700 1 -- 172.21.15.89:0/1909401637 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872400b9c0 con 0x7f872c0ce670 2024-04-02T07:37:00.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.449+0000 7f872a000700 1 -- 172.21.15.89:0/1909401637 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872400bd70 con 0x7f872c0ce670 2024-04-02T07:37:00.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.450+0000 7f8732600700 1 -- 172.21.15.89:0/1909401637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0ce670 msgr2=0x7f872c0b2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:00.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.450+0000 7f8732600700 1 --2- 172.21.15.89:0/1909401637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0ce670 0x7f872c0b2640 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f872c0cf080 tx=0x7f872400b330 comp rx=0 tx=0).stop 2024-04-02T07:37:00.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.450+0000 7f8732600700 1 -- 172.21.15.89:0/1909401637 shutdown_connections 2024-04-02T07:37:00.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.450+0000 7f8732600700 1 --2- 172.21.15.89:0/1909401637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0ce670 0x7f872c0b2640 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.450+0000 7f8732600700 1 --2- 172.21.15.89:0/1909401637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c15a5c0 0x7f872c0709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.450+0000 7f8732600700 1 --2- 172.21.15.89:0/1909401637 >> v2:172.21.15.113:3300/0 conn(0x7f872c15dfe0 0x7f872c0ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.450+0000 7f8732600700 1 -- 172.21.15.89:0/1909401637 >> 172.21.15.89:0/1909401637 conn(0x7f872c0b0d70 msgr2=0x7f872c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:00.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.450+0000 7f8732600700 1 -- 172.21.15.89:0/1909401637 shutdown_connections 2024-04-02T07:37:00.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.450+0000 7f8732600700 1 -- 172.21.15.89:0/1909401637 wait complete. 2024-04-02T07:37:00.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.451+0000 7f8732600700 1 Processor -- start 2024-04-02T07:37:00.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.452+0000 7f8732600700 1 -- start start 2024-04-02T07:37:00.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.452+0000 7f8732600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0ce670 0x7f872c1ed4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:00.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.452+0000 7f8732600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c15a5c0 0x7f872c153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:00.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f8732600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f872c15dfe0 0x7f872c0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:00.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f8732600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c153df0 con 0x7f872c0ce670 2024-04-02T07:37:00.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f8732600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c1eda20 con 0x7f872c15dfe0 2024-04-02T07:37:00.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f8732600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872c151d10 con 0x7f872c15a5c0 2024-04-02T07:37:00.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f872b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0ce670 0x7f872c1ed4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:00.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f872b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0ce670 0x7f872c1ed4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32794/0 (socket says 172.21.15.89:32794) 2024-04-02T07:37:00.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f872b400700 1 -- 172.21.15.89:0/1843169996 learned_addr learned my addr 172.21.15.89:0/1843169996 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:00.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f872aa00700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c15a5c0 0x7f872c153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:00.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f872be00700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f872c15dfe0 0x7f872c0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:00.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f872b400700 1 -- 172.21.15.89:0/1843169996 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c15a5c0 msgr2=0x7f872c153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:00.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f872b400700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c15a5c0 0x7f872c153330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f872b400700 1 -- 172.21.15.89:0/1843169996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f872c15dfe0 msgr2=0x7f872c0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:00.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f872b400700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f872c15dfe0 0x7f872c0cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f872b400700 1 -- 172.21.15.89:0/1843169996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8724009b30 con 0x7f872c0ce670 2024-04-02T07:37:00.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f872aa00700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c15a5c0 0x7f872c153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:00.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.453+0000 7f872b400700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0ce670 0x7f872c1ed4e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f871c00d800 tx=0x7f871c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:00.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.454+0000 7f8723400700 1 -- 172.21.15.89:0/1843169996 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c0125f0 con 0x7f872c0ce670 2024-04-02T07:37:00.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.454+0000 7f8723400700 1 -- 172.21.15.89:0/1843169996 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871c012750 con 0x7f872c0ce670 2024-04-02T07:37:00.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.454+0000 7f8723400700 1 -- 172.21.15.89:0/1843169996 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c012ae0 con 0x7f872c0ce670 2024-04-02T07:37:00.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.454+0000 7f8732600700 1 -- 172.21.15.89:0/1843169996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872c151ff0 con 0x7f872c0ce670 2024-04-02T07:37:00.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.454+0000 7f8732600700 1 -- 172.21.15.89:0/1843169996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872c1525b0 con 0x7f872c0ce670 2024-04-02T07:37:00.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.455+0000 7f8732600700 1 -- 172.21.15.89:0/1843169996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f872c067720 con 0x7f872c0ce670 2024-04-02T07:37:00.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.456+0000 7f8723400700 1 -- 172.21.15.89:0/1843169996 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f871c012c40 con 0x7f872c0ce670 2024-04-02T07:37:00.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.456+0000 7f8723400700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f870c06ac90 0x7f870c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:00.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.457+0000 7f8723400700 1 -- 172.21.15.89:0/1843169996 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f871c089390 con 0x7f872c0ce670 2024-04-02T07:37:00.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.457+0000 7f872aa00700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f870c06ac90 0x7f870c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:00.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.457+0000 7f872aa00700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f870c06ac90 0x7f870c06d150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f872c0ccb70 tx=0x7f8718008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:00.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.461+0000 7f8723400700 1 -- 172.21.15.89:0/1843169996 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f871c054820 con 0x7f872c0ce670 2024-04-02T07:37:00.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.742+0000 7f8732600700 1 -- 172.21.15.89:0/1843169996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f872c152e30 con 0x7f872c0ce670 2024-04-02T07:37:00.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.746+0000 7f8723400700 1 -- 172.21.15.89:0/1843169996 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f871c057e40 con 0x7f872c0ce670 2024-04-02T07:37:00.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.748+0000 7f8732600700 1 -- 172.21.15.89:0/1843169996 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f870c06ac90 msgr2=0x7f870c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:00.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.748+0000 7f8732600700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f870c06ac90 0x7f870c06d150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f872c0ccb70 tx=0x7f8718008040 comp rx=0 tx=0).stop 2024-04-02T07:37:00.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.748+0000 7f8732600700 1 -- 172.21.15.89:0/1843169996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0ce670 msgr2=0x7f872c1ed4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:00.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.748+0000 7f8732600700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0ce670 0x7f872c1ed4e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f871c00d800 tx=0x7f871c00dbc0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.748+0000 7f8732600700 1 -- 172.21.15.89:0/1843169996 shutdown_connections 2024-04-02T07:37:00.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.748+0000 7f8732600700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f870c06ac90 0x7f870c06d150 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.748+0000 7f8732600700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f872c15dfe0 0x7f872c0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.748+0000 7f8732600700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0ce670 0x7f872c1ed4e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.748+0000 7f8732600700 1 --2- 172.21.15.89:0/1843169996 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c15a5c0 0x7f872c153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.749+0000 7f8732600700 1 -- 172.21.15.89:0/1843169996 >> 172.21.15.89:0/1843169996 conn(0x7f872c0b0d70 msgr2=0x7f872c06d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:00.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.749+0000 7f8732600700 1 -- 172.21.15.89:0/1843169996 shutdown_connections 2024-04-02T07:37:00.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.749+0000 7f8732600700 1 -- 172.21.15.89:0/1843169996 wait complete. 2024-04-02T07:37:00.769 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-02T07:37:00.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.958+0000 7fccdf600700 1 Processor -- start 2024-04-02T07:37:00.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.958+0000 7fccdf600700 1 -- start start 2024-04-02T07:37:00.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.960+0000 7fccdf600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fccd815d9d0 0x7fccd80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:00.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.960+0000 7fccdf600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccd80d16a0 0x7fccd80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:00.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.960+0000 7fccdf600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd8161420 con 0x7fccd80d2370 2024-04-02T07:37:00.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.960+0000 7fccdf600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd80d2820 con 0x7fccd815d9d0 2024-04-02T07:37:00.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.960+0000 7fccdf600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd8161700 con 0x7fccd80d16a0 2024-04-02T07:37:00.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.960+0000 7fccdd600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fccd80d2370 0x7fccd815ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47062/0 (socket says 172.21.15.89:47062) 2024-04-02T07:37:00.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.960+0000 7fccdd600700 1 -- 172.21.15.89:0/2505613085 learned_addr learned my addr 172.21.15.89:0/2505613085 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:00.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.960+0000 7fccdcc00700 1 --2- 172.21.15.89:0/2505613085 >> v2:172.21.15.113:3300/0 conn(0x7fccd815d9d0 0x7fccd80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:00.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.960+0000 7fccd7e00700 1 --2- 172.21.15.89:0/2505613085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccd80d16a0 0x7fccd80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:00.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.960+0000 7fccd7400700 1 -- 172.21.15.89:0/2505613085 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033179130 0 0) 0x7fccd8161420 con 0x7fccd80d2370 2024-04-02T07:37:00.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.960+0000 7fccd7400700 1 -- 172.21.15.89:0/2505613085 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc00035b0 con 0x7fccd80d2370 2024-04-02T07:37:00.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.961+0000 7fccdcc00700 1 -- 172.21.15.89:0/2505613085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccd80d16a0 msgr2=0x7fccd80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:00.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.961+0000 7fccdcc00700 1 --2- 172.21.15.89:0/2505613085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccd80d16a0 0x7fccd80d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.961+0000 7fccdcc00700 1 -- 172.21.15.89:0/2505613085 >> v1:172.21.15.89:6789/0 conn(0x7fccd80d2370 legacy=0x7fccd815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:00.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.961+0000 7fccdcc00700 1 -- 172.21.15.89:0/2505613085 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd81619e0 con 0x7fccd815d9d0 2024-04-02T07:37:00.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.961+0000 7fccd7e00700 1 --2- 172.21.15.89:0/2505613085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccd80d16a0 0x7fccd80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:37:00.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.961+0000 7fccdcc00700 1 --2- 172.21.15.89:0/2505613085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd815d9d0 0x7fccd80d3c30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcccc009840 tx=0x7fcccc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54bfdd0ead99880 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:00.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.961+0000 7fccd7400700 1 -- 172.21.15.89:0/2505613085 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc011830 con 0x7fccd815d9d0 2024-04-02T07:37:00.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.961+0000 7fccd7400700 1 -- 172.21.15.89:0/2505613085 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcccc011990 con 0x7fccd815d9d0 2024-04-02T07:37:00.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.961+0000 7fccd7400700 1 -- 172.21.15.89:0/2505613085 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc011d20 con 0x7fccd815d9d0 2024-04-02T07:37:00.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.962+0000 7fccdf600700 1 -- 172.21.15.89:0/2505613085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd815d9d0 msgr2=0x7fccd80d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:00.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.962+0000 7fccdf600700 1 --2- 172.21.15.89:0/2505613085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd815d9d0 0x7fccd80d3c30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcccc009840 tx=0x7fcccc009b50 comp rx=0 tx=0).stop 2024-04-02T07:37:00.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.962+0000 7fccdf600700 1 -- 172.21.15.89:0/2505613085 shutdown_connections 2024-04-02T07:37:00.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.962+0000 7fccdf600700 1 --2- 172.21.15.89:0/2505613085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd815d9d0 0x7fccd80d3c30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.962+0000 7fccdf600700 1 --2- 172.21.15.89:0/2505613085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccd80d16a0 0x7fccd80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.962+0000 7fccdf600700 1 -- 172.21.15.89:0/2505613085 >> 172.21.15.89:0/2505613085 conn(0x7fccd808aff0 msgr2=0x7fccd80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:00.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.962+0000 7fccdf600700 1 -- 172.21.15.89:0/2505613085 shutdown_connections 2024-04-02T07:37:00.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.962+0000 7fccdf600700 1 -- 172.21.15.89:0/2505613085 wait complete. 2024-04-02T07:37:00.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.963+0000 7fccdf600700 1 Processor -- start 2024-04-02T07:37:00.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.964+0000 7fccdf600700 1 -- start start 2024-04-02T07:37:00.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdf600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd80d16a0 0x7fccd8158390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:00.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdf600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccd80d2370 0x7fccd80b8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:00.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdf600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd815d9d0 0x7fccd80b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:00.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdf600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd81588d0 con 0x7fccd815d9d0 2024-04-02T07:37:00.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdf600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd80b8280 con 0x7fccd80d16a0 2024-04-02T07:37:00.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdf600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd80b7670 con 0x7fccd80d2370 2024-04-02T07:37:00.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdd600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd815d9d0 0x7fccd80b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:00.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdd600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd815d9d0 0x7fccd80b48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32806/0 (socket says 172.21.15.89:32806) 2024-04-02T07:37:00.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdd600700 1 -- 172.21.15.89:0/2358536024 learned_addr learned my addr 172.21.15.89:0/2358536024 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:00.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccd7e00700 1 --2- 172.21.15.89:0/2358536024 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccd80d2370 0x7fccd80b8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:00.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdcc00700 1 --2- 172.21.15.89:0/2358536024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd80d16a0 0x7fccd8158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:00.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdd600700 1 -- 172.21.15.89:0/2358536024 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccd80d2370 msgr2=0x7fccd80b8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:00.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdd600700 1 --2- 172.21.15.89:0/2358536024 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccd80d2370 0x7fccd80b8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdd600700 1 -- 172.21.15.89:0/2358536024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd80d16a0 msgr2=0x7fccd8158390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:00.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdd600700 1 --2- 172.21.15.89:0/2358536024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd80d16a0 0x7fccd8158390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:00.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.965+0000 7fccdd600700 1 -- 172.21.15.89:0/2358536024 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc4009430 con 0x7fccd815d9d0 2024-04-02T07:37:00.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.966+0000 7fccdd600700 1 --2- 172.21.15.89:0/2358536024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd815d9d0 0x7fccd80b48f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fccd000cb40 tx=0x7fccd000ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:00.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.966+0000 7fccd4c00700 1 -- 172.21.15.89:0/2358536024 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd000a9e0 con 0x7fccd815d9d0 2024-04-02T07:37:00.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.966+0000 7fccd4c00700 1 -- 172.21.15.89:0/2358536024 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd000ab40 con 0x7fccd815d9d0 2024-04-02T07:37:00.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.966+0000 7fccdf600700 1 -- 172.21.15.89:0/2358536024 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc0094a0 con 0x7fccd815d9d0 2024-04-02T07:37:00.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.966+0000 7fccd4c00700 1 -- 172.21.15.89:0/2358536024 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd0012400 con 0x7fccd815d9d0 2024-04-02T07:37:00.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.966+0000 7fccdf600700 1 -- 172.21.15.89:0/2358536024 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd80b7c30 con 0x7fccd815d9d0 2024-04-02T07:37:00.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.967+0000 7fccdf600700 1 -- 172.21.15.89:0/2358536024 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd80d1ca0 con 0x7fccd815d9d0 2024-04-02T07:37:00.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.968+0000 7fccd4c00700 1 -- 172.21.15.89:0/2358536024 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fccd001b3b0 con 0x7fccd815d9d0 2024-04-02T07:37:00.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.969+0000 7fccd4c00700 1 --2- 172.21.15.89:0/2358536024 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fccb806ac90 0x7fccb806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:00.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.969+0000 7fccd4c00700 1 -- 172.21.15.89:0/2358536024 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fccd0088b30 con 0x7fccd815d9d0 2024-04-02T07:37:00.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.969+0000 7fccdcc00700 1 --2- 172.21.15.89:0/2358536024 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fccb806ac90 0x7fccb806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:00.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.969+0000 7fccdcc00700 1 --2- 172.21.15.89:0/2358536024 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fccb806ac90 0x7fccb806d150 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcccc000c00 tx=0x7fcccc01c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:00.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:00.973+0000 7fccd4c00700 1 -- 172.21.15.89:0/2358536024 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fccd0054100 con 0x7fccd815d9d0 2024-04-02T07:37:01.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.257+0000 7fccdf600700 1 -- 172.21.15.89:0/2358536024 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fccd80b83f0 con 0x7fccd815d9d0 2024-04-02T07:37:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.261+0000 7fccd4c00700 1 -- 172.21.15.89:0/2358536024 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7fccd00170a0 con 0x7fccd815d9d0 2024-04-02T07:37:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.263+0000 7fccdf600700 1 -- 172.21.15.89:0/2358536024 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fccb806ac90 msgr2=0x7fccb806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.263+0000 7fccdf600700 1 --2- 172.21.15.89:0/2358536024 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fccb806ac90 0x7fccb806d150 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcccc000c00 tx=0x7fcccc01c040 comp rx=0 tx=0).stop 2024-04-02T07:37:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.263+0000 7fccdf600700 1 -- 172.21.15.89:0/2358536024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd815d9d0 msgr2=0x7fccd80b48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.263+0000 7fccdf600700 1 --2- 172.21.15.89:0/2358536024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd815d9d0 0x7fccd80b48f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fccd000cb40 tx=0x7fccd000ce50 comp rx=0 tx=0).stop 2024-04-02T07:37:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.264+0000 7fccdf600700 1 -- 172.21.15.89:0/2358536024 shutdown_connections 2024-04-02T07:37:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.264+0000 7fccdf600700 1 --2- 172.21.15.89:0/2358536024 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fccb806ac90 0x7fccb806d150 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.264+0000 7fccdf600700 1 --2- 172.21.15.89:0/2358536024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd80d16a0 0x7fccd8158390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.264+0000 7fccdf600700 1 --2- 172.21.15.89:0/2358536024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd815d9d0 0x7fccd80b48f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.264+0000 7fccdf600700 1 --2- 172.21.15.89:0/2358536024 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccd80d2370 0x7fccd80b8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.264+0000 7fccdf600700 1 -- 172.21.15.89:0/2358536024 >> 172.21.15.89:0/2358536024 conn(0x7fccd808aff0 msgr2=0x7fccd80ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.264+0000 7fccdf600700 1 -- 172.21.15.89:0/2358536024 shutdown_connections 2024-04-02T07:37:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.264+0000 7fccdf600700 1 -- 172.21.15.89:0/2358536024 wait complete. 2024-04-02T07:37:01.405 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-02T07:37:01.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.595+0000 7f5092800700 1 Processor -- start 2024-04-02T07:37:01.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.595+0000 7f5092800700 1 -- start start 2024-04-02T07:37:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.596+0000 7f5092800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f508c15d9d0 0x7f508c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.596+0000 7f5092800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508c0d16a0 0x7f508c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.596+0000 7f5092800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508c161420 con 0x7f508c0d2370 2024-04-02T07:37:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.596+0000 7f5092800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f508c0d2820 con 0x7f508c15d9d0 2024-04-02T07:37:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.596+0000 7f5092800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f508c161700 con 0x7f508c0d16a0 2024-04-02T07:37:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.596+0000 7f508b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f508c15d9d0 0x7f508c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.596+0000 7f508be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f508c0d2370 0x7f508c15ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47064/0 (socket says 172.21.15.89:47064) 2024-04-02T07:37:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.596+0000 7f508be00700 1 -- 172.21.15.89:0/1548745480 learned_addr learned my addr 172.21.15.89:0/1548745480 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.597+0000 7f508aa00700 1 --2- 172.21.15.89:0/1548745480 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508c0d16a0 0x7f508c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.597+0000 7f508b400700 1 -- 172.21.15.89:0/1548745480 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508c0d16a0 msgr2=0x7f508c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.597+0000 7f508b400700 1 --2- 172.21.15.89:0/1548745480 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508c0d16a0 0x7f508c0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.597+0000 7f508b400700 1 -- 172.21.15.89:0/1548745480 >> v1:172.21.15.89:6789/0 conn(0x7f508c0d2370 legacy=0x7f508c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.597+0000 7f508b400700 1 -- 172.21.15.89:0/1548745480 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c1619e0 con 0x7f508c15d9d0 2024-04-02T07:37:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.597+0000 7f508a000700 1 -- 172.21.15.89:0/1548745480 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876843885 0 0) 0x7f508c161420 con 0x7f508c0d2370 2024-04-02T07:37:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.597+0000 7f508b400700 1 --2- 172.21.15.89:0/1548745480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508c15d9d0 0x7f508c0d3c30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f507c009840 tx=0x7f507c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6192b03ac4968cbd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.597+0000 7f508a000700 1 -- 172.21.15.89:0/1548745480 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c011830 con 0x7f508c15d9d0 2024-04-02T07:37:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.597+0000 7f508a000700 1 -- 172.21.15.89:0/1548745480 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507c011990 con 0x7f508c15d9d0 2024-04-02T07:37:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.597+0000 7f508a000700 1 -- 172.21.15.89:0/1548745480 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c011d20 con 0x7f508c15d9d0 2024-04-02T07:37:01.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.598+0000 7f5092800700 1 -- 172.21.15.89:0/1548745480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508c15d9d0 msgr2=0x7f508c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:01.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.598+0000 7f5092800700 1 --2- 172.21.15.89:0/1548745480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508c15d9d0 0x7f508c0d3c30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f507c009840 tx=0x7f507c009b50 comp rx=0 tx=0).stop 2024-04-02T07:37:01.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.598+0000 7f5092800700 1 -- 172.21.15.89:0/1548745480 shutdown_connections 2024-04-02T07:37:01.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.598+0000 7f5092800700 1 --2- 172.21.15.89:0/1548745480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508c15d9d0 0x7f508c0d3c30 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:01.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.598+0000 7f5092800700 1 --2- 172.21.15.89:0/1548745480 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508c0d16a0 0x7f508c0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:01.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.598+0000 7f5092800700 1 -- 172.21.15.89:0/1548745480 >> 172.21.15.89:0/1548745480 conn(0x7f508c08aff0 msgr2=0x7f508c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:01.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.598+0000 7f5092800700 1 -- 172.21.15.89:0/1548745480 shutdown_connections 2024-04-02T07:37:01.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.598+0000 7f5092800700 1 -- 172.21.15.89:0/1548745480 wait complete. 2024-04-02T07:37:01.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.600+0000 7f5092800700 1 Processor -- start 2024-04-02T07:37:01.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.600+0000 7f5092800700 1 -- start start 2024-04-02T07:37:01.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f5092800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508c0d16a0 0x7f508c158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:01.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f5092800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f508c0d2370 0x7f508c0b7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:01.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f5092800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508c15d9d0 0x7f508c0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:01.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f5092800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c0b75a0 con 0x7f508c15d9d0 2024-04-02T07:37:01.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f5092800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c160390 con 0x7f508c0d2370 2024-04-02T07:37:01.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f5092800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f508c0b7790 con 0x7f508c0d16a0 2024-04-02T07:37:01.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f508be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508c15d9d0 0x7f508c0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:01.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f508be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508c15d9d0 0x7f508c0b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32820/0 (socket says 172.21.15.89:32820) 2024-04-02T07:37:01.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f508be00700 1 -- 172.21.15.89:0/266855196 learned_addr learned my addr 172.21.15.89:0/266855196 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:01.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f508b400700 1 --2- 172.21.15.89:0/266855196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508c0d16a0 0x7f508c158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:01.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f508aa00700 1 --2- 172.21.15.89:0/266855196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f508c0d2370 0x7f508c0b7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:01.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f508be00700 1 -- 172.21.15.89:0/266855196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508c0d16a0 msgr2=0x7f508c158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:01.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f508be00700 1 --2- 172.21.15.89:0/266855196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508c0d16a0 0x7f508c158dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:01.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f508be00700 1 -- 172.21.15.89:0/266855196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f508c0d2370 msgr2=0x7f508c0b7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:01.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f508be00700 1 --2- 172.21.15.89:0/266855196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f508c0d2370 0x7f508c0b7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:01.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.601+0000 7f508be00700 1 -- 172.21.15.89:0/266855196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c0094a0 con 0x7f508c15d9d0 2024-04-02T07:37:01.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.602+0000 7f508be00700 1 --2- 172.21.15.89:0/266855196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508c15d9d0 0x7f508c0b4730 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5084002230 tx=0x7f508400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:01.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.602+0000 7f5083400700 1 -- 172.21.15.89:0/266855196 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508400ac20 con 0x7f508c15d9d0 2024-04-02T07:37:01.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.602+0000 7f5083400700 1 -- 172.21.15.89:0/266855196 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508400ada0 con 0x7f508c15d9d0 2024-04-02T07:37:01.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.602+0000 7f5083400700 1 -- 172.21.15.89:0/266855196 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084007310 con 0x7f508c15d9d0 2024-04-02T07:37:01.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.602+0000 7f5092800700 1 -- 172.21.15.89:0/266855196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508c15de50 con 0x7f508c15d9d0 2024-04-02T07:37:01.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.602+0000 7f5092800700 1 -- 172.21.15.89:0/266855196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508c15e410 con 0x7f508c15d9d0 2024-04-02T07:37:01.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.603+0000 7f5092800700 1 -- 172.21.15.89:0/266855196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508c0d1ca0 con 0x7f508c15d9d0 2024-04-02T07:37:01.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.604+0000 7f5083400700 1 -- 172.21.15.89:0/266855196 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5084007470 con 0x7f508c15d9d0 2024-04-02T07:37:01.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.605+0000 7f5083400700 1 --2- 172.21.15.89:0/266855196 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f506c06ac90 0x7f506c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:01.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.605+0000 7f5083400700 1 -- 172.21.15.89:0/266855196 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5084088b70 con 0x7f508c15d9d0 2024-04-02T07:37:01.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.605+0000 7f508b400700 1 --2- 172.21.15.89:0/266855196 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f506c06ac90 0x7f506c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:01.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.606+0000 7f508b400700 1 --2- 172.21.15.89:0/266855196 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f506c06ac90 0x7f506c06d150 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f507c0090b0 tx=0x7f507c01c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:01.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.610+0000 7f5083400700 1 -- 172.21.15.89:0/266855196 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5084054090 con 0x7f508c15d9d0 2024-04-02T07:37:01.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.893+0000 7f5092800700 1 -- 172.21.15.89:0/266855196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f508c15e620 con 0x7f508c15d9d0 2024-04-02T07:37:01.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.897+0000 7f5083400700 1 -- 172.21.15.89:0/266855196 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5084018020 con 0x7f508c15d9d0 2024-04-02T07:37:01.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.900+0000 7f5092800700 1 -- 172.21.15.89:0/266855196 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f506c06ac90 msgr2=0x7f506c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:01.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.900+0000 7f5092800700 1 --2- 172.21.15.89:0/266855196 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f506c06ac90 0x7f506c06d150 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f507c0090b0 tx=0x7f507c01c040 comp rx=0 tx=0).stop 2024-04-02T07:37:01.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.900+0000 7f5092800700 1 -- 172.21.15.89:0/266855196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508c15d9d0 msgr2=0x7f508c0b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:01.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.900+0000 7f5092800700 1 --2- 172.21.15.89:0/266855196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508c15d9d0 0x7f508c0b4730 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5084002230 tx=0x7f508400a060 comp rx=0 tx=0).stop 2024-04-02T07:37:01.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.900+0000 7f5092800700 1 -- 172.21.15.89:0/266855196 shutdown_connections 2024-04-02T07:37:01.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.900+0000 7f5092800700 1 --2- 172.21.15.89:0/266855196 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f506c06ac90 0x7f506c06d150 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:01.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.900+0000 7f5092800700 1 --2- 172.21.15.89:0/266855196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508c15d9d0 0x7f508c0b4730 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:01.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.900+0000 7f5092800700 1 --2- 172.21.15.89:0/266855196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508c0d16a0 0x7f508c158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:01.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.900+0000 7f5092800700 1 --2- 172.21.15.89:0/266855196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f508c0d2370 0x7f508c0b7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:01.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.900+0000 7f5092800700 1 -- 172.21.15.89:0/266855196 >> 172.21.15.89:0/266855196 conn(0x7f508c08aff0 msgr2=0x7f508c0ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:01.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.900+0000 7f5092800700 1 -- 172.21.15.89:0/266855196 shutdown_connections 2024-04-02T07:37:01.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:01.900+0000 7f5092800700 1 -- 172.21.15.89:0/266855196 wait complete. 2024-04-02T07:37:01.920 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:37:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.107+0000 7fc640000700 1 Processor -- start 2024-04-02T07:37:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.107+0000 7fc640000700 1 -- start start 2024-04-02T07:37:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.108+0000 7fc640000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc638072580 0x7fc6380b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.108+0000 7fc640000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6380b2180 con 0x7fc6380b3af0 2024-04-02T07:37:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.108+0000 7fc640000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc63815b100 con 0x7fc6380b2e50 2024-04-02T07:37:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.108+0000 7fc640000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63815b3e0 con 0x7fc638072580 2024-04-02T07:37:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.109+0000 7fc63d600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc6380b3af0 0x7fc63815a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47076/0 (socket says 172.21.15.89:47076) 2024-04-02T07:37:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.109+0000 7fc63d600700 1 -- 172.21.15.89:0/4176275508 learned_addr learned my addr 172.21.15.89:0/4176275508 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.109+0000 7fc63cc00700 1 --2- 172.21.15.89:0/4176275508 >> v2:172.21.15.113:3300/0 conn(0x7fc638072580 0x7fc6380b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.109+0000 7fc637e00700 1 -- 172.21.15.89:0/4176275508 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431760680 0 0) 0x7fc6380b2180 con 0x7fc6380b3af0 2024-04-02T07:37:02.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.109+0000 7fc637e00700 1 -- 172.21.15.89:0/4176275508 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6200035b0 con 0x7fc6380b3af0 2024-04-02T07:37:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.109+0000 7fc637e00700 1 -- 172.21.15.89:0/4176275508 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1352281340 0 0) 0x7fc6200035b0 con 0x7fc6380b3af0 2024-04-02T07:37:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.109+0000 7fc637e00700 1 -- 172.21.15.89:0/4176275508 >> v2:172.21.15.113:3300/0 conn(0x7fc638072580 msgr2=0x7fc6380b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.109+0000 7fc637e00700 1 --2- 172.21.15.89:0/4176275508 >> v2:172.21.15.113:3300/0 conn(0x7fc638072580 0x7fc6380b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.109+0000 7fc637e00700 1 -- 172.21.15.89:0/4176275508 >> v1:172.21.15.113:6789/0 conn(0x7fc6380b2e50 legacy=0x7fc6380709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.109+0000 7fc637e00700 1 -- 172.21.15.89:0/4176275508 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6380b23f0 con 0x7fc6380b3af0 2024-04-02T07:37:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.109+0000 7fc637e00700 1 -- 172.21.15.89:0/4176275508 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fc62c001670 con 0x7fc6380b3af0 2024-04-02T07:37:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.109+0000 7fc637e00700 1 -- 172.21.15.89:0/4176275508 >> v1:172.21.15.89:6789/0 conn(0x7fc6380b3af0 legacy=0x7fc63815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.109+0000 7fc637e00700 1 --2- 172.21.15.89:0/4176275508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc620003a00 0x7fc620003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.109+0000 7fc637e00700 1 --2- 172.21.15.89:0/4176275508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc620004850 0x7fc620004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc637e00700 1 --2- 172.21.15.89:0/4176275508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc620005210 0x7fc62000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc637e00700 1 -- 172.21.15.89:0/4176275508 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63815b3e0 con 0x7fc620005210 2024-04-02T07:37:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc637e00700 1 -- 172.21.15.89:0/4176275508 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62000c0b0 con 0x7fc620004850 2024-04-02T07:37:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc637e00700 1 -- 172.21.15.89:0/4176275508 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc62000e480 con 0x7fc620003a00 2024-04-02T07:37:02.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc63e000700 1 --2- 172.21.15.89:0/4176275508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc620005210 0x7fc62000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc63cc00700 1 --2- 172.21.15.89:0/4176275508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc620004850 0x7fc620004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc63d600700 1 --2- 172.21.15.89:0/4176275508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc620003a00 0x7fc620003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc63e000700 1 -- 172.21.15.89:0/4176275508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc620003a00 msgr2=0x7fc620003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc63e000700 1 --2- 172.21.15.89:0/4176275508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc620003a00 0x7fc620003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc63e000700 1 -- 172.21.15.89:0/4176275508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc620004850 msgr2=0x7fc620004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc63e000700 1 --2- 172.21.15.89:0/4176275508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc620004850 0x7fc620004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc63e000700 1 -- 172.21.15.89:0/4176275508 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6200035b0 con 0x7fc620005210 2024-04-02T07:37:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc63d600700 1 --2- 172.21.15.89:0/4176275508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc620003a00 0x7fc620003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc63e000700 1 -- 172.21.15.89:0/4176275508 reap_dead start 2024-04-02T07:37:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.110+0000 7fc63e000700 1 --2- 172.21.15.89:0/4176275508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc620005210 0x7fc62000ba10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c190 tx=0x7fc620005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aecdde5de073088 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.111+0000 7fc637e00700 1 -- 172.21.15.89:0/4176275508 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630001db0 con 0x7fc620005210 2024-04-02T07:37:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.111+0000 7fc637e00700 1 -- 172.21.15.89:0/4176275508 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63000ccf0 con 0x7fc620005210 2024-04-02T07:37:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.111+0000 7fc637e00700 1 -- 172.21.15.89:0/4176275508 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630003c40 con 0x7fc620005210 2024-04-02T07:37:02.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.111+0000 7fc640000700 1 -- 172.21.15.89:0/4176275508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc620005210 msgr2=0x7fc62000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.111+0000 7fc640000700 1 --2- 172.21.15.89:0/4176275508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc620005210 0x7fc62000ba10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c190 tx=0x7fc620005880 comp rx=0 tx=0).stop 2024-04-02T07:37:02.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.111+0000 7fc640000700 1 -- 172.21.15.89:0/4176275508 shutdown_connections 2024-04-02T07:37:02.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.111+0000 7fc640000700 1 --2- 172.21.15.89:0/4176275508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc620005210 0x7fc62000ba10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.111+0000 7fc640000700 1 -- 172.21.15.89:0/4176275508 >> 172.21.15.89:0/4176275508 conn(0x7fc6380cf190 msgr2=0x7fc6380cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:02.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.111+0000 7fc640000700 1 -- 172.21.15.89:0/4176275508 shutdown_connections 2024-04-02T07:37:02.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.111+0000 7fc640000700 1 -- 172.21.15.89:0/4176275508 wait complete. 2024-04-02T07:37:02.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.113+0000 7fc640000700 1 Processor -- start 2024-04-02T07:37:02.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.113+0000 7fc640000700 1 -- start start 2024-04-02T07:37:02.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.114+0000 7fc640000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc620005210 0x7fc6380ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.114+0000 7fc640000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc63814c380 0x7fc638147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.114+0000 7fc640000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc638153d30 0x7fc6381541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.114+0000 7fc640000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc638154700 con 0x7fc620005210 2024-04-02T07:37:02.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.114+0000 7fc640000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6380ccab0 con 0x7fc63814c380 2024-04-02T07:37:02.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.114+0000 7fc640000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63814d4a0 con 0x7fc638153d30 2024-04-02T07:37:02.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.114+0000 7fc63d600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc620005210 0x7fc6380ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.114+0000 7fc63d600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc620005210 0x7fc6380ce5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32836/0 (socket says 172.21.15.89:32836) 2024-04-02T07:37:02.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.114+0000 7fc63d600700 1 -- 172.21.15.89:0/3287260822 learned_addr learned my addr 172.21.15.89:0/3287260822 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:02.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc63e000700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc638153d30 0x7fc6381541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc63cc00700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc63814c380 0x7fc638147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc63d600700 1 -- 172.21.15.89:0/3287260822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc638153d30 msgr2=0x7fc6381541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc63d600700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc638153d30 0x7fc6381541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc63d600700 1 -- 172.21.15.89:0/3287260822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc63814c380 msgr2=0x7fc638147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc63d600700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc63814c380 0x7fc638147e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc63d600700 1 -- 172.21.15.89:0/3287260822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc638148900 con 0x7fc620005210 2024-04-02T07:37:02.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc63e000700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc638153d30 0x7fc6381541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:37:02.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc63cc00700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc63814c380 0x7fc638147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:37:02.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc63d600700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc620005210 0x7fc6380ce5b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00ea70 tx=0x7fc62c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:02.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc635600700 1 -- 172.21.15.89:0/3287260822 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c005e70 con 0x7fc620005210 2024-04-02T07:37:02.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc635600700 1 -- 172.21.15.89:0/3287260822 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c00c7e0 con 0x7fc620005210 2024-04-02T07:37:02.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc640000700 1 -- 172.21.15.89:0/3287260822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63814c920 con 0x7fc620005210 2024-04-02T07:37:02.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc635600700 1 -- 172.21.15.89:0/3287260822 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c00cb50 con 0x7fc620005210 2024-04-02T07:37:02.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.115+0000 7fc640000700 1 -- 172.21.15.89:0/3287260822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc63814cee0 con 0x7fc620005210 2024-04-02T07:37:02.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.117+0000 7fc640000700 1 -- 172.21.15.89:0/3287260822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6380ccc20 con 0x7fc620005210 2024-04-02T07:37:02.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.117+0000 7fc635600700 1 -- 172.21.15.89:0/3287260822 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc62c00ccb0 con 0x7fc620005210 2024-04-02T07:37:02.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.118+0000 7fc635600700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc61806ac90 0x7fc61806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.118+0000 7fc635600700 1 -- 172.21.15.89:0/3287260822 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc62c088de0 con 0x7fc620005210 2024-04-02T07:37:02.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.118+0000 7fc63cc00700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc61806ac90 0x7fc61806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.119+0000 7fc63cc00700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc61806ac90 0x7fc61806d150 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc638072a30 tx=0x7fc624008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:02.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.122+0000 7fc635600700 1 -- 172.21.15.89:0/3287260822 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc62c054400 con 0x7fc620005210 2024-04-02T07:37:02.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.327+0000 7fc640000700 1 -- 172.21.15.89:0/3287260822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc638067720 con 0x7fc620005210 2024-04-02T07:37:02.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.328+0000 7fc635600700 1 -- 172.21.15.89:0/3287260822 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12389 (secure 0 0 0) 0x7fc62c057a20 con 0x7fc620005210 2024-04-02T07:37:02.330 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:37:02.331 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":30,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:36:58.156267+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:37:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.330+0000 7fc640000700 1 -- 172.21.15.89:0/3287260822 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc61806ac90 msgr2=0x7fc61806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.330+0000 7fc640000700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc61806ac90 0x7fc61806d150 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc638072a30 tx=0x7fc624008040 comp rx=0 tx=0).stop 2024-04-02T07:37:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.330+0000 7fc640000700 1 -- 172.21.15.89:0/3287260822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc620005210 msgr2=0x7fc6380ce5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.330+0000 7fc640000700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc620005210 0x7fc6380ce5b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00ea70 tx=0x7fc62c00ee30 comp rx=0 tx=0).stop 2024-04-02T07:37:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.330+0000 7fc640000700 1 -- 172.21.15.89:0/3287260822 shutdown_connections 2024-04-02T07:37:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.330+0000 7fc640000700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc61806ac90 0x7fc61806d150 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.330+0000 7fc640000700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc638153d30 0x7fc6381541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.330+0000 7fc640000700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc620005210 0x7fc6380ce5b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.330+0000 7fc640000700 1 --2- 172.21.15.89:0/3287260822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc63814c380 0x7fc638147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.330+0000 7fc640000700 1 -- 172.21.15.89:0/3287260822 >> 172.21.15.89:0/3287260822 conn(0x7fc6380cf190 msgr2=0x7fc6380709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.330+0000 7fc640000700 1 -- 172.21.15.89:0/3287260822 shutdown_connections 2024-04-02T07:37:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.330+0000 7fc640000700 1 -- 172.21.15.89:0/3287260822 wait complete. 2024-04-02T07:37:02.351 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:37:02.351 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:37:02.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.542+0000 7f84cbe00700 1 Processor -- start 2024-04-02T07:37:02.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.542+0000 7f84cbe00700 1 -- start start 2024-04-02T07:37:02.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.543+0000 7f84cbe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c415d9d0 0x7f84c40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.543+0000 7f84cbe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f84c40d2540 0x7f84c40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.543+0000 7f84cbe00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c4161420 con 0x7f84c40d16a0 2024-04-02T07:37:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.543+0000 7f84cbe00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c415edc0 con 0x7f84c40d2540 2024-04-02T07:37:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.543+0000 7f84cbe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c4161700 con 0x7f84c415d9d0 2024-04-02T07:37:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.543+0000 7f84c9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c415d9d0 0x7f84c40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.543+0000 7f84c9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c415d9d0 0x7f84c40d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:35746/0 (socket says 172.21.15.89:35746) 2024-04-02T07:37:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.543+0000 7f84c9400700 1 -- 172.21.15.89:0/2170514387 learned_addr learned my addr 172.21.15.89:0/2170514387 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.543+0000 7f84c9e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f84c40d2540 0x7f84c40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.544+0000 7f84c9400700 1 -- 172.21.15.89:0/2170514387 >> v2:172.21.15.113:3300/0 conn(0x7f84c40d2540 msgr2=0x7f84c40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.544+0000 7f84c9400700 1 --2- 172.21.15.89:0/2170514387 >> v2:172.21.15.113:3300/0 conn(0x7f84c40d2540 0x7f84c40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.544+0000 7f84c9400700 1 -- 172.21.15.89:0/2170514387 >> v1:172.21.15.113:6789/0 conn(0x7f84c40d16a0 legacy=0x7f84c40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.544+0000 7f84c9400700 1 -- 172.21.15.89:0/2170514387 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c41619e0 con 0x7f84c415d9d0 2024-04-02T07:37:02.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.544+0000 7f84c9400700 1 --2- 172.21.15.89:0/2170514387 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c415d9d0 0x7f84c40d3c30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009920 tx=0x7f84b8009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a179bc30f87fea57 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:02.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.544+0000 7f84c3e00700 1 -- 172.21.15.89:0/2170514387 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8011830 con 0x7f84c415d9d0 2024-04-02T07:37:02.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.544+0000 7f84c3e00700 1 -- 172.21.15.89:0/2170514387 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b8011990 con 0x7f84c415d9d0 2024-04-02T07:37:02.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.545+0000 7f84c3e00700 1 -- 172.21.15.89:0/2170514387 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8011d20 con 0x7f84c415d9d0 2024-04-02T07:37:02.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.545+0000 7f84cbe00700 1 -- 172.21.15.89:0/2170514387 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c415d9d0 msgr2=0x7f84c40d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.545+0000 7f84cbe00700 1 --2- 172.21.15.89:0/2170514387 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c415d9d0 0x7f84c40d3c30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009920 tx=0x7f84b8009c30 comp rx=0 tx=0).stop 2024-04-02T07:37:02.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.545+0000 7f84cbe00700 1 -- 172.21.15.89:0/2170514387 shutdown_connections 2024-04-02T07:37:02.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.545+0000 7f84cbe00700 1 --2- 172.21.15.89:0/2170514387 >> v2:172.21.15.113:3300/0 conn(0x7f84c40d2540 0x7f84c40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.545+0000 7f84cbe00700 1 --2- 172.21.15.89:0/2170514387 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c415d9d0 0x7f84c40d3c30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.545+0000 7f84cbe00700 1 -- 172.21.15.89:0/2170514387 >> 172.21.15.89:0/2170514387 conn(0x7f84c408aff0 msgr2=0x7f84c40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:02.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.545+0000 7f84cbe00700 1 -- 172.21.15.89:0/2170514387 shutdown_connections 2024-04-02T07:37:02.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.545+0000 7f84cbe00700 1 -- 172.21.15.89:0/2170514387 wait complete. 2024-04-02T07:37:02.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.547+0000 7f84cbe00700 1 Processor -- start 2024-04-02T07:37:02.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.547+0000 7f84cbe00700 1 -- start start 2024-04-02T07:37:02.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.548+0000 7f84cbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c40d16a0 0x7f84c415a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.548+0000 7f84cbe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c40d2540 0x7f84c4160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.548+0000 7f84cbe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84c415d9d0 0x7f84c4160960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.548+0000 7f84cbe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c415a670 con 0x7f84c40d16a0 2024-04-02T07:37:02.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.548+0000 7f84cbe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40b7810 con 0x7f84c415d9d0 2024-04-02T07:37:02.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.548+0000 7f84cbe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40b7530 con 0x7f84c40d2540 2024-04-02T07:37:02.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.548+0000 7f84c9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c40d16a0 0x7f84c415a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.548+0000 7f84c9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c40d16a0 0x7f84c415a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32838/0 (socket says 172.21.15.89:32838) 2024-04-02T07:37:02.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.548+0000 7f84c9400700 1 -- 172.21.15.89:0/2972139302 learned_addr learned my addr 172.21.15.89:0/2972139302 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:02.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.548+0000 7f84c9e00700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84c415d9d0 0x7f84c4160960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.548+0000 7f84c8a00700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c40d2540 0x7f84c4160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.548+0000 7f84c9400700 1 -- 172.21.15.89:0/2972139302 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c40d2540 msgr2=0x7f84c4160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.548+0000 7f84c9400700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c40d2540 0x7f84c4160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.548+0000 7f84c9400700 1 -- 172.21.15.89:0/2972139302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84c415d9d0 msgr2=0x7f84c4160960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.549+0000 7f84c9400700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84c415d9d0 0x7f84c4160960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.549+0000 7f84c9400700 1 -- 172.21.15.89:0/2972139302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b8009550 con 0x7f84c40d16a0 2024-04-02T07:37:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.549+0000 7f84c9e00700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84c415d9d0 0x7f84c4160960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.549+0000 7f84c9400700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c40d16a0 0x7f84c415a130 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009130 tx=0x7f84b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:02.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.549+0000 7f84c1600700 1 -- 172.21.15.89:0/2972139302 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b80036a0 con 0x7f84c40d16a0 2024-04-02T07:37:02.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.549+0000 7f84c1600700 1 -- 172.21.15.89:0/2972139302 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b801e070 con 0x7f84c40d16a0 2024-04-02T07:37:02.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.549+0000 7f84c1600700 1 -- 172.21.15.89:0/2972139302 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8003a60 con 0x7f84c40d16a0 2024-04-02T07:37:02.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.549+0000 7f84cbe00700 1 -- 172.21.15.89:0/2972139302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c40b52c0 con 0x7f84c40d16a0 2024-04-02T07:37:02.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.549+0000 7f84cbe00700 1 -- 172.21.15.89:0/2972139302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c40b6ee0 con 0x7f84c40d16a0 2024-04-02T07:37:02.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.550+0000 7f84cbe00700 1 -- 172.21.15.89:0/2972139302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c41583e0 con 0x7f84c40d16a0 2024-04-02T07:37:02.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.551+0000 7f84c1600700 1 -- 172.21.15.89:0/2972139302 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f84b8003bc0 con 0x7f84c40d16a0 2024-04-02T07:37:02.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.552+0000 7f84c1600700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84a406ac90 0x7f84a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.552+0000 7f84c1600700 1 -- 172.21.15.89:0/2972139302 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f84b8089ea0 con 0x7f84c40d16a0 2024-04-02T07:37:02.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.552+0000 7f84c8a00700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84a406ac90 0x7f84a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.553+0000 7f84c8a00700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84a406ac90 0x7f84a406d150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f84c4161090 tx=0x7f84b0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:02.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.556+0000 7f84c1600700 1 -- 172.21.15.89:0/2972139302 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f84b8055460 con 0x7f84c40d16a0 2024-04-02T07:37:02.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.758+0000 7f84cbe00700 1 -- 172.21.15.89:0/2972139302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f84c40d1ca0 con 0x7f84c40d16a0 2024-04-02T07:37:02.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.758+0000 7f84c1600700 1 -- 172.21.15.89:0/2972139302 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7f84b80200a0 con 0x7f84c40d16a0 2024-04-02T07:37:02.761 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:37:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.760+0000 7f84cbe00700 1 -- 172.21.15.89:0/2972139302 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84a406ac90 msgr2=0x7f84a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.760+0000 7f84cbe00700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84a406ac90 0x7f84a406d150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f84c4161090 tx=0x7f84b0009040 comp rx=0 tx=0).stop 2024-04-02T07:37:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.760+0000 7f84cbe00700 1 -- 172.21.15.89:0/2972139302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c40d16a0 msgr2=0x7f84c415a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.760+0000 7f84cbe00700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c40d16a0 0x7f84c415a130 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009130 tx=0x7f84b8004750 comp rx=0 tx=0).stop 2024-04-02T07:37:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.761+0000 7f84cbe00700 1 -- 172.21.15.89:0/2972139302 shutdown_connections 2024-04-02T07:37:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.761+0000 7f84cbe00700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84a406ac90 0x7f84a406d150 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.761+0000 7f84cbe00700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84c415d9d0 0x7f84c4160960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.761+0000 7f84cbe00700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c40d16a0 0x7f84c415a130 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.761+0000 7f84cbe00700 1 --2- 172.21.15.89:0/2972139302 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c40d2540 0x7f84c4160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.761+0000 7f84cbe00700 1 -- 172.21.15.89:0/2972139302 >> 172.21.15.89:0/2972139302 conn(0x7f84c408aff0 msgr2=0x7f84c40b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.761+0000 7f84cbe00700 1 -- 172.21.15.89:0/2972139302 shutdown_connections 2024-04-02T07:37:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.761+0000 7f84cbe00700 1 -- 172.21.15.89:0/2972139302 wait complete. 2024-04-02T07:37:02.782 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T07:37:02.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.970+0000 7f16a7600700 1 Processor -- start 2024-04-02T07:37:02.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.970+0000 7f16a7600700 1 -- start start 2024-04-02T07:37:02.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.971+0000 7f16a7600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16a0160f10 0x7f16a00d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.971+0000 7f16a7600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f16a0161590 0x7f16a0161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.971+0000 7f16a7600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a00ceda0 con 0x7f16a00cf230 2024-04-02T07:37:02.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.971+0000 7f16a7600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a0161f80 con 0x7f16a0161590 2024-04-02T07:37:02.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.971+0000 7f16a7600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a01612f0 con 0x7f16a0160f10 2024-04-02T07:37:02.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.971+0000 7f16a5600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f16a0161590 0x7f16a0161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.972+0000 7f16a5600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f16a0161590 0x7f16a0161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32846/0 (socket says 172.21.15.89:32846) 2024-04-02T07:37:02.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.972+0000 7f16a5600700 1 -- 172.21.15.89:0/4155337245 learned_addr learned my addr 172.21.15.89:0/4155337245 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:02.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.972+0000 7f16a4c00700 1 --2- 172.21.15.89:0/4155337245 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16a0160f10 0x7f16a00d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.972+0000 7f16a5600700 1 -- 172.21.15.89:0/4155337245 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16a0160f10 msgr2=0x7f16a00d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.972+0000 7f16a5600700 1 --2- 172.21.15.89:0/4155337245 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16a0160f10 0x7f16a00d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.972+0000 7f16a5600700 1 -- 172.21.15.89:0/4155337245 >> v1:172.21.15.113:6789/0 conn(0x7f16a00cf230 legacy=0x7f16a00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.972+0000 7f16a5600700 1 -- 172.21.15.89:0/4155337245 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a00ce470 con 0x7f16a0161590 2024-04-02T07:37:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.972+0000 7f16a4c00700 1 --2- 172.21.15.89:0/4155337245 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16a0160f10 0x7f16a00d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.972+0000 7f16a5600700 1 --2- 172.21.15.89:0/4155337245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16a0161590 0x7f16a0161a40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1698009e80 tx=0x7f169800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82e852b6b9b815f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.972+0000 7f169f400700 1 -- 172.21.15.89:0/4155337245 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169800e070 con 0x7f16a0161590 2024-04-02T07:37:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.972+0000 7f169f400700 1 -- 172.21.15.89:0/4155337245 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169800b980 con 0x7f16a0161590 2024-04-02T07:37:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.972+0000 7f169f400700 1 -- 172.21.15.89:0/4155337245 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169800bd30 con 0x7f16a0161590 2024-04-02T07:37:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.973+0000 7f16a7600700 1 -- 172.21.15.89:0/4155337245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16a0161590 msgr2=0x7f16a0161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.973+0000 7f16a7600700 1 --2- 172.21.15.89:0/4155337245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16a0161590 0x7f16a0161a40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1698009e80 tx=0x7f169800b330 comp rx=0 tx=0).stop 2024-04-02T07:37:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.973+0000 7f16a7600700 1 -- 172.21.15.89:0/4155337245 shutdown_connections 2024-04-02T07:37:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.973+0000 7f16a7600700 1 --2- 172.21.15.89:0/4155337245 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16a0160f10 0x7f16a00d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.973+0000 7f16a7600700 1 --2- 172.21.15.89:0/4155337245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16a0161590 0x7f16a0161a40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.973+0000 7f16a7600700 1 -- 172.21.15.89:0/4155337245 >> 172.21.15.89:0/4155337245 conn(0x7f16a008aff0 msgr2=0x7f16a00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.973+0000 7f16a7600700 1 -- 172.21.15.89:0/4155337245 shutdown_connections 2024-04-02T07:37:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.973+0000 7f16a7600700 1 -- 172.21.15.89:0/4155337245 wait complete. 2024-04-02T07:37:02.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.975+0000 7f16a7600700 1 Processor -- start 2024-04-02T07:37:02.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.975+0000 7f16a7600700 1 -- start start 2024-04-02T07:37:02.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a7600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16a00cf230 0x7f16a01508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a7600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16a0160f10 0x7f16a015d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a7600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16a0161590 0x7f16a015d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a7600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a009d820 con 0x7f16a0161590 2024-04-02T07:37:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a7600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a015ac70 con 0x7f16a00cf230 2024-04-02T07:37:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a7600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a015dea0 con 0x7f16a0160f10 2024-04-02T07:37:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a5600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16a0161590 0x7f16a015d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a5600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16a0161590 0x7f16a015d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32858/0 (socket says 172.21.15.89:32858) 2024-04-02T07:37:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a5600700 1 -- 172.21.15.89:0/3801009252 learned_addr learned my addr 172.21.15.89:0/3801009252 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a4c00700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16a00cf230 0x7f16a01508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f169fe00700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16a0160f10 0x7f16a015d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a5600700 1 -- 172.21.15.89:0/3801009252 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16a0160f10 msgr2=0x7f16a015d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a5600700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16a0160f10 0x7f16a015d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a5600700 1 -- 172.21.15.89:0/3801009252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16a00cf230 msgr2=0x7f16a01508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a5600700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16a00cf230 0x7f16a01508c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.976+0000 7f16a5600700 1 -- 172.21.15.89:0/3801009252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1698009b30 con 0x7f16a0161590 2024-04-02T07:37:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.977+0000 7f169fe00700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16a0160f10 0x7f16a015d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.977+0000 7f16a5600700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16a0161590 0x7f16a015d870 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f16a008c570 tx=0x7f1698004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.977+0000 7f169cc00700 1 -- 172.21.15.89:0/3801009252 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16980048b0 con 0x7f16a0161590 2024-04-02T07:37:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.977+0000 7f169cc00700 1 -- 172.21.15.89:0/3801009252 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16980099b0 con 0x7f16a0161590 2024-04-02T07:37:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.977+0000 7f16a7600700 1 -- 172.21.15.89:0/3801009252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a015e180 con 0x7f16a0161590 2024-04-02T07:37:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.977+0000 7f169cc00700 1 -- 172.21.15.89:0/3801009252 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169801c680 con 0x7f16a0161590 2024-04-02T07:37:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.977+0000 7f16a7600700 1 -- 172.21.15.89:0/3801009252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a009d440 con 0x7f16a0161590 2024-04-02T07:37:02.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.978+0000 7f16a7600700 1 -- 172.21.15.89:0/3801009252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a009d960 con 0x7f16a0161590 2024-04-02T07:37:02.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.979+0000 7f169cc00700 1 -- 172.21.15.89:0/3801009252 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f169801c7e0 con 0x7f16a0161590 2024-04-02T07:37:02.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.980+0000 7f169cc00700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f168006ac90 0x7f168006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:02.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.980+0000 7f169cc00700 1 -- 172.21.15.89:0/3801009252 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f169808a860 con 0x7f16a0161590 2024-04-02T07:37:02.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.980+0000 7f16a4c00700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f168006ac90 0x7f168006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:02.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.981+0000 7f16a4c00700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f168006ac90 0x7f168006d150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1694006f10 tx=0x7f1694008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:02.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:02.985+0000 7f169cc00700 1 -- 172.21.15.89:0/3801009252 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f16a009d960 con 0x7f16a0161590 2024-04-02T07:37:03.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.190+0000 7f16a7600700 1 -- 172.21.15.89:0/3801009252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f16a009d960 con 0x7f16a0161590 2024-04-02T07:37:03.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.190+0000 7f169cc00700 1 -- 172.21.15.89:0/3801009252 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7f16a009d960 con 0x7f16a0161590 2024-04-02T07:37:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:37:03.193 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:37:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:37:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T07:37:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T07:37:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:37:03.193 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:37:03.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.193+0000 7f16a7600700 1 -- 172.21.15.89:0/3801009252 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f168006ac90 msgr2=0x7f168006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:03.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.193+0000 7f16a7600700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f168006ac90 0x7f168006d150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1694006f10 tx=0x7f1694008040 comp rx=0 tx=0).stop 2024-04-02T07:37:03.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.193+0000 7f16a7600700 1 -- 172.21.15.89:0/3801009252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16a0161590 msgr2=0x7f16a015d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:03.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.193+0000 7f16a7600700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16a0161590 0x7f16a015d870 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f16a008c570 tx=0x7f1698004ad0 comp rx=0 tx=0).stop 2024-04-02T07:37:03.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.193+0000 7f16a7600700 1 -- 172.21.15.89:0/3801009252 shutdown_connections 2024-04-02T07:37:03.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.193+0000 7f16a7600700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f168006ac90 0x7f168006d150 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:03.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.193+0000 7f16a7600700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16a00cf230 0x7f16a01508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:03.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.193+0000 7f16a7600700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16a0161590 0x7f16a015d870 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:03.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.193+0000 7f16a7600700 1 --2- 172.21.15.89:0/3801009252 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16a0160f10 0x7f16a015d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:03.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.193+0000 7f16a7600700 1 -- 172.21.15.89:0/3801009252 >> 172.21.15.89:0/3801009252 conn(0x7f16a008aff0 msgr2=0x7f16a00ca750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:03.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.193+0000 7f16a7600700 1 -- 172.21.15.89:0/3801009252 shutdown_connections 2024-04-02T07:37:03.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.193+0000 7f16a7600700 1 -- 172.21.15.89:0/3801009252 wait complete. 2024-04-02T07:37:03.213 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T07:37:03.214 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T07:37:03.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.407+0000 7ff591c00700 1 Processor -- start 2024-04-02T07:37:03.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.407+0000 7ff591c00700 1 -- start start 2024-04-02T07:37:03.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.408+0000 7ff591c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff58c15ddf0 0x7ff58c0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:03.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.408+0000 7ff591c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff58c0d28e0 0x7ff58c15f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.408+0000 7ff591c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58c0ce8a0 con 0x7ff58c15e1d0 2024-04-02T07:37:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.408+0000 7ff591c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58c15ec60 con 0x7ff58c15ddf0 2024-04-02T07:37:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.409+0000 7ff591c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c0cdc90 con 0x7ff58c0d28e0 2024-04-02T07:37:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.409+0000 7ff583e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff58c15e1d0 0x7ff58c0d21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47080/0 (socket says 172.21.15.89:47080) 2024-04-02T07:37:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.409+0000 7ff583e00700 1 -- 172.21.15.89:0/712550558 learned_addr learned my addr 172.21.15.89:0/712550558 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.409+0000 7ff58b400700 1 --2- 172.21.15.89:0/712550558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff58c0d28e0 0x7ff58c15f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.409+0000 7ff58aa00700 1 --2- 172.21.15.89:0/712550558 >> v2:172.21.15.113:3300/0 conn(0x7ff58c15ddf0 0x7ff58c0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:03.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.409+0000 7ff58a000700 1 -- 172.21.15.89:0/712550558 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525202491 0 0) 0x7ff58c0ce8a0 con 0x7ff58c15e1d0 2024-04-02T07:37:03.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.409+0000 7ff58a000700 1 -- 172.21.15.89:0/712550558 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5700035b0 con 0x7ff58c15e1d0 2024-04-02T07:37:03.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.409+0000 7ff58aa00700 1 -- 172.21.15.89:0/712550558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff58c0d28e0 msgr2=0x7ff58c15f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:03.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.409+0000 7ff58aa00700 1 --2- 172.21.15.89:0/712550558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff58c0d28e0 0x7ff58c15f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:03.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.409+0000 7ff58aa00700 1 -- 172.21.15.89:0/712550558 >> v1:172.21.15.89:6789/0 conn(0x7ff58c15e1d0 legacy=0x7ff58c0d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:03.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.409+0000 7ff58aa00700 1 -- 172.21.15.89:0/712550558 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c0cdf70 con 0x7ff58c15ddf0 2024-04-02T07:37:03.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.409+0000 7ff58b400700 1 --2- 172.21.15.89:0/712550558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff58c0d28e0 0x7ff58c15f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:37:03.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.410+0000 7ff58aa00700 1 --2- 172.21.15.89:0/712550558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff58c15ddf0 0x7ff58c0ced30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009840 tx=0x7ff57c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bda9e0ae41170dae server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:03.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.410+0000 7ff58a000700 1 -- 172.21.15.89:0/712550558 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c011830 con 0x7ff58c15ddf0 2024-04-02T07:37:03.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.410+0000 7ff58a000700 1 -- 172.21.15.89:0/712550558 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c011990 con 0x7ff58c15ddf0 2024-04-02T07:37:03.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.410+0000 7ff58a000700 1 -- 172.21.15.89:0/712550558 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c011d20 con 0x7ff58c15ddf0 2024-04-02T07:37:03.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.411+0000 7ff591c00700 1 -- 172.21.15.89:0/712550558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff58c15ddf0 msgr2=0x7ff58c0ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:03.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.411+0000 7ff591c00700 1 --2- 172.21.15.89:0/712550558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff58c15ddf0 0x7ff58c0ced30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009840 tx=0x7ff57c009b50 comp rx=0 tx=0).stop 2024-04-02T07:37:03.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.411+0000 7ff591c00700 1 -- 172.21.15.89:0/712550558 shutdown_connections 2024-04-02T07:37:03.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.411+0000 7ff591c00700 1 --2- 172.21.15.89:0/712550558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff58c0d28e0 0x7ff58c15f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:03.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.411+0000 7ff591c00700 1 --2- 172.21.15.89:0/712550558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff58c15ddf0 0x7ff58c0ced30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:03.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.411+0000 7ff591c00700 1 -- 172.21.15.89:0/712550558 >> 172.21.15.89:0/712550558 conn(0x7ff58c08aff0 msgr2=0x7ff58c0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:03.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.411+0000 7ff591c00700 1 -- 172.21.15.89:0/712550558 shutdown_connections 2024-04-02T07:37:03.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.411+0000 7ff591c00700 1 -- 172.21.15.89:0/712550558 wait complete. 2024-04-02T07:37:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.412+0000 7ff591c00700 1 Processor -- start 2024-04-02T07:37:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.413+0000 7ff591c00700 1 -- start start 2024-04-02T07:37:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.413+0000 7ff591c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff58c15ddf0 0x7ff58c1612b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.413+0000 7ff591c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff58c15e1d0 0x7ff58c1617f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.413+0000 7ff591c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff58c156590 0x7ff58c0b27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.413+0000 7ff591c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c0b4160 con 0x7ff58c156590 2024-04-02T07:37:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.413+0000 7ff591c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c0c9640 con 0x7ff58c15ddf0 2024-04-02T07:37:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.413+0000 7ff591c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c161070 con 0x7ff58c15e1d0 2024-04-02T07:37:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.414+0000 7ff58b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff58c156590 0x7ff58c0b27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.414+0000 7ff58b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff58c156590 0x7ff58c0b27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32874/0 (socket says 172.21.15.89:32874) 2024-04-02T07:37:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.414+0000 7ff58b400700 1 -- 172.21.15.89:0/2437332800 learned_addr learned my addr 172.21.15.89:0/2437332800 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.414+0000 7ff58aa00700 1 --2- 172.21.15.89:0/2437332800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff58c15ddf0 0x7ff58c1612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.414+0000 7ff583e00700 1 --2- 172.21.15.89:0/2437332800 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff58c15e1d0 0x7ff58c1617f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.414+0000 7ff58b400700 1 -- 172.21.15.89:0/2437332800 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff58c15e1d0 msgr2=0x7ff58c1617f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.414+0000 7ff58b400700 1 --2- 172.21.15.89:0/2437332800 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff58c15e1d0 0x7ff58c1617f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:03.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.414+0000 7ff58b400700 1 -- 172.21.15.89:0/2437332800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff58c15ddf0 msgr2=0x7ff58c1612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:03.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.414+0000 7ff58b400700 1 --2- 172.21.15.89:0/2437332800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff58c15ddf0 0x7ff58c1612b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:03.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.414+0000 7ff58b400700 1 -- 172.21.15.89:0/2437332800 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff584009a70 con 0x7ff58c156590 2024-04-02T07:37:03.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.414+0000 7ff58b400700 1 --2- 172.21.15.89:0/2437332800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff58c156590 0x7ff58c0b27a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff58400efb0 tx=0x7ff58400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:03.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.415+0000 7ff582a00700 1 -- 172.21.15.89:0/2437332800 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58400cc30 con 0x7ff58c156590 2024-04-02T07:37:03.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.415+0000 7ff582a00700 1 -- 172.21.15.89:0/2437332800 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58400f040 con 0x7ff58c156590 2024-04-02T07:37:03.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.415+0000 7ff582a00700 1 -- 172.21.15.89:0/2437332800 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584014480 con 0x7ff58c156590 2024-04-02T07:37:03.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.415+0000 7ff591c00700 1 -- 172.21.15.89:0/2437332800 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57c0094a0 con 0x7ff58c156590 2024-04-02T07:37:03.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.415+0000 7ff591c00700 1 -- 172.21.15.89:0/2437332800 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58c0c8ff0 con 0x7ff58c156590 2024-04-02T07:37:03.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.416+0000 7ff591c00700 1 -- 172.21.15.89:0/2437332800 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58c067720 con 0x7ff58c156590 2024-04-02T07:37:03.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.417+0000 7ff582a00700 1 -- 172.21.15.89:0/2437332800 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff5840145e0 con 0x7ff58c156590 2024-04-02T07:37:03.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.417+0000 7ff582a00700 1 --2- 172.21.15.89:0/2437332800 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff56806ac90 0x7ff56806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:03.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.418+0000 7ff582a00700 1 -- 172.21.15.89:0/2437332800 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff584089c80 con 0x7ff58c156590 2024-04-02T07:37:03.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.418+0000 7ff58aa00700 1 --2- 172.21.15.89:0/2437332800 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff56806ac90 0x7ff56806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:03.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.418+0000 7ff58aa00700 1 --2- 172.21.15.89:0/2437332800 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff56806ac90 0x7ff56806d150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff57c000c00 tx=0x7ff57c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:03.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.422+0000 7ff582a00700 1 -- 172.21.15.89:0/2437332800 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff584055080 con 0x7ff58c156590 2024-04-02T07:37:03.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.625+0000 7ff591c00700 1 -- 172.21.15.89:0/2437332800 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff58c0c9780 con 0x7ff58c156590 2024-04-02T07:37:03.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.994+0000 7ff582a00700 1 -- 172.21.15.89:0/2437332800 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff584018070 con 0x7ff58c156590 2024-04-02T07:37:04.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.997+0000 7ff591c00700 1 -- 172.21.15.89:0/2437332800 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff56806ac90 msgr2=0x7ff56806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:04.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.997+0000 7ff591c00700 1 --2- 172.21.15.89:0/2437332800 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff56806ac90 0x7ff56806d150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff57c000c00 tx=0x7ff57c008030 comp rx=0 tx=0).stop 2024-04-02T07:37:04.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.997+0000 7ff591c00700 1 -- 172.21.15.89:0/2437332800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff58c156590 msgr2=0x7ff58c0b27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:04.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.997+0000 7ff591c00700 1 --2- 172.21.15.89:0/2437332800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff58c156590 0x7ff58c0b27a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff58400efb0 tx=0x7ff58400c3a0 comp rx=0 tx=0).stop 2024-04-02T07:37:04.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.997+0000 7ff591c00700 1 -- 172.21.15.89:0/2437332800 shutdown_connections 2024-04-02T07:37:04.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.997+0000 7ff591c00700 1 --2- 172.21.15.89:0/2437332800 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff56806ac90 0x7ff56806d150 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:04.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.997+0000 7ff591c00700 1 --2- 172.21.15.89:0/2437332800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff58c15ddf0 0x7ff58c1612b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:04.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.997+0000 7ff591c00700 1 --2- 172.21.15.89:0/2437332800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff58c156590 0x7ff58c0b27a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:04.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.997+0000 7ff591c00700 1 --2- 172.21.15.89:0/2437332800 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff58c15e1d0 0x7ff58c1617f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:04.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.997+0000 7ff591c00700 1 -- 172.21.15.89:0/2437332800 >> 172.21.15.89:0/2437332800 conn(0x7ff58c08aff0 msgr2=0x7ff58c0b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:04.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.997+0000 7ff591c00700 1 -- 172.21.15.89:0/2437332800 shutdown_connections 2024-04-02T07:37:04.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:03.997+0000 7ff591c00700 1 -- 172.21.15.89:0/2437332800 wait complete. 2024-04-02T07:37:04.001 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T07:37:04.021 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T07:37:04.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.218+0000 7fd8adc00700 1 Processor -- start 2024-04-02T07:37:04.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.218+0000 7fd8adc00700 1 -- start start 2024-04-02T07:37:04.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.219+0000 7fd8adc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8a815d9d0 0x7fd8a80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:04.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.219+0000 7fd8adc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd8a80d16a0 0x7fd8a80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:04.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.219+0000 7fd8adc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd8a80d2370 0x7fd8a815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:04.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.219+0000 7fd8adc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a80d2820 con 0x7fd8a80d2370 2024-04-02T07:37:04.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.219+0000 7fd8adc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a8161420 con 0x7fd8a80d16a0 2024-04-02T07:37:04.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.219+0000 7fd8adc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a8161700 con 0x7fd8a815d9d0 2024-04-02T07:37:04.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.219+0000 7fd8a7400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd8a80d2370 0x7fd8a815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:04.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.220+0000 7fd8a7400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd8a80d2370 0x7fd8a815ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32886/0 (socket says 172.21.15.89:32886) 2024-04-02T07:37:04.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.220+0000 7fd8a7400700 1 -- 172.21.15.89:0/2848673221 learned_addr learned my addr 172.21.15.89:0/2848673221 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:04.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.220+0000 7fd8a6000700 1 --2- 172.21.15.89:0/2848673221 >> v2:172.21.15.113:3300/0 conn(0x7fd8a80d16a0 0x7fd8a80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:04.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.220+0000 7fd8a6a00700 1 --2- 172.21.15.89:0/2848673221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8a815d9d0 0x7fd8a80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:04.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.220+0000 7fd8a7400700 1 -- 172.21.15.89:0/2848673221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8a815d9d0 msgr2=0x7fd8a80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:04.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.220+0000 7fd8a7400700 1 --2- 172.21.15.89:0/2848673221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8a815d9d0 0x7fd8a80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:04.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.220+0000 7fd8a7400700 1 -- 172.21.15.89:0/2848673221 >> v2:172.21.15.113:3300/0 conn(0x7fd8a80d16a0 msgr2=0x7fd8a80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:04.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.220+0000 7fd8a7400700 1 --2- 172.21.15.89:0/2848673221 >> v2:172.21.15.113:3300/0 conn(0x7fd8a80d16a0 0x7fd8a80d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:04.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.220+0000 7fd8a7400700 1 -- 172.21.15.89:0/2848673221 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a81619e0 con 0x7fd8a80d2370 2024-04-02T07:37:04.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.220+0000 7fd8a6000700 1 --2- 172.21.15.89:0/2848673221 >> v2:172.21.15.113:3300/0 conn(0x7fd8a80d16a0 0x7fd8a80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:04.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.220+0000 7fd8a7400700 1 --2- 172.21.15.89:0/2848673221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8a80d2370 0x7fd8a815ddb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd8a806b5e0 tx=0x7fd8a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de6eb998f50ab50 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:04.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.220+0000 7fd8a5600700 1 -- 172.21.15.89:0/2848673221 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a000e070 con 0x7fd8a80d2370 2024-04-02T07:37:04.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.220+0000 7fd8a5600700 1 -- 172.21.15.89:0/2848673221 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a000b9c0 con 0x7fd8a80d2370 2024-04-02T07:37:04.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.221+0000 7fd8a5600700 1 -- 172.21.15.89:0/2848673221 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a000bd70 con 0x7fd8a80d2370 2024-04-02T07:37:04.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.221+0000 7fd8adc00700 1 -- 172.21.15.89:0/2848673221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8a80d2370 msgr2=0x7fd8a815ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:04.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.221+0000 7fd8adc00700 1 --2- 172.21.15.89:0/2848673221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8a80d2370 0x7fd8a815ddb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd8a806b5e0 tx=0x7fd8a000b330 comp rx=0 tx=0).stop 2024-04-02T07:37:04.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.221+0000 7fd8adc00700 1 -- 172.21.15.89:0/2848673221 shutdown_connections 2024-04-02T07:37:04.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.221+0000 7fd8adc00700 1 --2- 172.21.15.89:0/2848673221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8a80d2370 0x7fd8a815ddb0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:04.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.221+0000 7fd8adc00700 1 --2- 172.21.15.89:0/2848673221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8a815d9d0 0x7fd8a80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:04.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.221+0000 7fd8adc00700 1 --2- 172.21.15.89:0/2848673221 >> v2:172.21.15.113:3300/0 conn(0x7fd8a80d16a0 0x7fd8a80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:04.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.221+0000 7fd8adc00700 1 -- 172.21.15.89:0/2848673221 >> 172.21.15.89:0/2848673221 conn(0x7fd8a808aff0 msgr2=0x7fd8a80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:04.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.221+0000 7fd8adc00700 1 -- 172.21.15.89:0/2848673221 shutdown_connections 2024-04-02T07:37:04.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.221+0000 7fd8adc00700 1 -- 172.21.15.89:0/2848673221 wait complete. 2024-04-02T07:37:04.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.223+0000 7fd8adc00700 1 Processor -- start 2024-04-02T07:37:04.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.223+0000 7fd8adc00700 1 -- start start 2024-04-02T07:37:04.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8adc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8a80d16a0 0x7fd8a8152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:04.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8adc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8a80d2370 0x7fd8a80b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:04.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8adc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8a815d9d0 0x7fd8a80cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:04.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8adc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a80b7240 con 0x7fd8a80d16a0 2024-04-02T07:37:04.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8adc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a80b7520 con 0x7fd8a80d2370 2024-04-02T07:37:04.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8adc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a80b7800 con 0x7fd8a815d9d0 2024-04-02T07:37:04.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8a6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8a80d16a0 0x7fd8a8152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:04.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8a6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8a80d16a0 0x7fd8a8152a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32900/0 (socket says 172.21.15.89:32900) 2024-04-02T07:37:04.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8a6a00700 1 -- 172.21.15.89:0/2104658325 learned_addr learned my addr 172.21.15.89:0/2104658325 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:04.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8a7400700 1 --2- 172.21.15.89:0/2104658325 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8a815d9d0 0x7fd8a80cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:04.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8a6000700 1 --2- 172.21.15.89:0/2104658325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8a80d2370 0x7fd8a80b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:04.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8a6a00700 1 -- 172.21.15.89:0/2104658325 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8a815d9d0 msgr2=0x7fd8a80cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:04.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8a6a00700 1 --2- 172.21.15.89:0/2104658325 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8a815d9d0 0x7fd8a80cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:04.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8a6a00700 1 -- 172.21.15.89:0/2104658325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8a80d2370 msgr2=0x7fd8a80b69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:04.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8a6a00700 1 --2- 172.21.15.89:0/2104658325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8a80d2370 0x7fd8a80b69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:04.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.224+0000 7fd8a6a00700 1 -- 172.21.15.89:0/2104658325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a0009b30 con 0x7fd8a80d16a0 2024-04-02T07:37:04.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.225+0000 7fd8a6a00700 1 --2- 172.21.15.89:0/2104658325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8a80d16a0 0x7fd8a8152a20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd89800d800 tx=0x7fd89800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:04.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.225+0000 7fd89ea00700 1 -- 172.21.15.89:0/2104658325 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8980125f0 con 0x7fd8a80d16a0 2024-04-02T07:37:04.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.225+0000 7fd89ea00700 1 -- 172.21.15.89:0/2104658325 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd898012750 con 0x7fd8a80d16a0 2024-04-02T07:37:04.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.225+0000 7fd8adc00700 1 -- 172.21.15.89:0/2104658325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a80b7ae0 con 0x7fd8a80d16a0 2024-04-02T07:37:04.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.225+0000 7fd89ea00700 1 -- 172.21.15.89:0/2104658325 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898012b00 con 0x7fd8a80d16a0 2024-04-02T07:37:04.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.225+0000 7fd8adc00700 1 -- 172.21.15.89:0/2104658325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a80b6f20 con 0x7fd8a80d16a0 2024-04-02T07:37:04.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.226+0000 7fd8adc00700 1 -- 172.21.15.89:0/2104658325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a80d1ca0 con 0x7fd8a80d16a0 2024-04-02T07:37:04.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.227+0000 7fd89ea00700 1 -- 172.21.15.89:0/2104658325 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd898012c60 con 0x7fd8a80d16a0 2024-04-02T07:37:04.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.228+0000 7fd89ea00700 1 --2- 172.21.15.89:0/2104658325 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd88806ac90 0x7fd88806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:04.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.228+0000 7fd89ea00700 1 -- 172.21.15.89:0/2104658325 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd898089c20 con 0x7fd8a80d16a0 2024-04-02T07:37:04.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.229+0000 7fd8a6000700 1 --2- 172.21.15.89:0/2104658325 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd88806ac90 0x7fd88806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:04.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.229+0000 7fd8a6000700 1 --2- 172.21.15.89:0/2104658325 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd88806ac90 0x7fd88806d150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd894006fd0 tx=0x7fd894008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:04.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.232+0000 7fd89ea00700 1 -- 172.21.15.89:0/2104658325 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd8980550c0 con 0x7fd8a80d16a0 2024-04-02T07:37:04.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:04.432+0000 7fd8adc00700 1 -- 172.21.15.89:0/2104658325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd8a815d580 con 0x7fd8a80d16a0 2024-04-02T07:37:05.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.004+0000 7fd89ea00700 1 -- 172.21.15.89:0/2104658325 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd8980586e0 con 0x7fd8a80d16a0 2024-04-02T07:37:05.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.010+0000 7fd8adc00700 1 -- 172.21.15.89:0/2104658325 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd88806ac90 msgr2=0x7fd88806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:05.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.010+0000 7fd8adc00700 1 --2- 172.21.15.89:0/2104658325 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd88806ac90 0x7fd88806d150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd894006fd0 tx=0x7fd894008040 comp rx=0 tx=0).stop 2024-04-02T07:37:05.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.010+0000 7fd8adc00700 1 -- 172.21.15.89:0/2104658325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8a80d16a0 msgr2=0x7fd8a8152a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:05.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.010+0000 7fd8adc00700 1 --2- 172.21.15.89:0/2104658325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8a80d16a0 0x7fd8a8152a20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd89800d800 tx=0x7fd89800db10 comp rx=0 tx=0).stop 2024-04-02T07:37:05.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.011+0000 7fd8adc00700 1 -- 172.21.15.89:0/2104658325 shutdown_connections 2024-04-02T07:37:05.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.011+0000 7fd8adc00700 1 --2- 172.21.15.89:0/2104658325 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd88806ac90 0x7fd88806d150 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.011+0000 7fd8adc00700 1 --2- 172.21.15.89:0/2104658325 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8a815d9d0 0x7fd8a80cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.011+0000 7fd8adc00700 1 --2- 172.21.15.89:0/2104658325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8a80d16a0 0x7fd8a8152a20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.011+0000 7fd8adc00700 1 --2- 172.21.15.89:0/2104658325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8a80d2370 0x7fd8a80b69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.011+0000 7fd8adc00700 1 -- 172.21.15.89:0/2104658325 >> 172.21.15.89:0/2104658325 conn(0x7fd8a808aff0 msgr2=0x7fd8a80b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:05.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.014+0000 7fd8adc00700 1 -- 172.21.15.89:0/2104658325 shutdown_connections 2024-04-02T07:37:05.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.014+0000 7fd8adc00700 1 -- 172.21.15.89:0/2104658325 wait complete. 2024-04-02T07:37:05.021 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T07:37:05.045 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T07:37:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.237+0000 7f0b2a400700 1 Processor -- start 2024-04-02T07:37:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.238+0000 7f0b2a400700 1 -- start start 2024-04-02T07:37:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b2a400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b24160e50 0x7f0b240b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b2a400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0b240d3f20 0x7f0b240b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b2a400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b240b3f80 0x7f0b240b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b2a400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b24161230 con 0x7f0b240b3f80 2024-04-02T07:37:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b2a400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b240d2a80 con 0x7f0b240d3f20 2024-04-02T07:37:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b2a400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b240d43b0 con 0x7f0b24160e50 2024-04-02T07:37:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b23e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b240b3f80 0x7f0b240b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b23e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b240b3f80 0x7f0b240b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32916/0 (socket says 172.21.15.89:32916) 2024-04-02T07:37:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b23e00700 1 -- 172.21.15.89:0/829946212 learned_addr learned my addr 172.21.15.89:0/829946212 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b23400700 1 --2- 172.21.15.89:0/829946212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b24160e50 0x7f0b240b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b22a00700 1 --2- 172.21.15.89:0/829946212 >> v2:172.21.15.113:3300/0 conn(0x7f0b240d3f20 0x7f0b240b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b23e00700 1 -- 172.21.15.89:0/829946212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b24160e50 msgr2=0x7f0b240b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b23e00700 1 --2- 172.21.15.89:0/829946212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b24160e50 0x7f0b240b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b23e00700 1 -- 172.21.15.89:0/829946212 >> v2:172.21.15.113:3300/0 conn(0x7f0b240d3f20 msgr2=0x7f0b240b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.239+0000 7f0b23e00700 1 --2- 172.21.15.89:0/829946212 >> v2:172.21.15.113:3300/0 conn(0x7f0b240d3f20 0x7f0b240b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.240+0000 7f0b23e00700 1 -- 172.21.15.89:0/829946212 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b240d21e0 con 0x7f0b240b3f80 2024-04-02T07:37:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.240+0000 7f0b23e00700 1 --2- 172.21.15.89:0/829946212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b240b3f80 0x7f0b240b2e90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0b24072e60 tx=0x7f0b1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83b84be3624a4bc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.240+0000 7f0b22000700 1 -- 172.21.15.89:0/829946212 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c00e070 con 0x7f0b240b3f80 2024-04-02T07:37:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.240+0000 7f0b22000700 1 -- 172.21.15.89:0/829946212 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1c00b910 con 0x7f0b240b3f80 2024-04-02T07:37:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.240+0000 7f0b22000700 1 -- 172.21.15.89:0/829946212 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c00bcc0 con 0x7f0b240b3f80 2024-04-02T07:37:05.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.241+0000 7f0b2a400700 1 -- 172.21.15.89:0/829946212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b240b3f80 msgr2=0x7f0b240b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:05.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.241+0000 7f0b2a400700 1 --2- 172.21.15.89:0/829946212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b240b3f80 0x7f0b240b2e90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0b24072e60 tx=0x7f0b1c00b330 comp rx=0 tx=0).stop 2024-04-02T07:37:05.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.241+0000 7f0b2a400700 1 -- 172.21.15.89:0/829946212 shutdown_connections 2024-04-02T07:37:05.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.241+0000 7f0b2a400700 1 --2- 172.21.15.89:0/829946212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b240b3f80 0x7f0b240b2e90 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.241+0000 7f0b2a400700 1 --2- 172.21.15.89:0/829946212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b24160e50 0x7f0b240b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.241+0000 7f0b2a400700 1 --2- 172.21.15.89:0/829946212 >> v2:172.21.15.113:3300/0 conn(0x7f0b240d3f20 0x7f0b240b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.241+0000 7f0b2a400700 1 -- 172.21.15.89:0/829946212 >> 172.21.15.89:0/829946212 conn(0x7f0b2408aff0 msgr2=0x7f0b240d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:05.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.241+0000 7f0b2a400700 1 -- 172.21.15.89:0/829946212 shutdown_connections 2024-04-02T07:37:05.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.241+0000 7f0b2a400700 1 -- 172.21.15.89:0/829946212 wait complete. 2024-04-02T07:37:05.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.242+0000 7f0b2a400700 1 Processor -- start 2024-04-02T07:37:05.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.243+0000 7f0b2a400700 1 -- start start 2024-04-02T07:37:05.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.243+0000 7f0b2a400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b240b3f80 0x7f0b2415aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.243+0000 7f0b2a400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b24160e50 0x7f0b2406cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b2a400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b240b6a60 0x7f0b240b6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b2a400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2406b990 con 0x7f0b240b3f80 2024-04-02T07:37:05.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b2a400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b241f4090 con 0x7f0b240b6a60 2024-04-02T07:37:05.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b2a400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2406bb70 con 0x7f0b24160e50 2024-04-02T07:37:05.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b23400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b240b3f80 0x7f0b2415aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b23400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b240b3f80 0x7f0b2415aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32922/0 (socket says 172.21.15.89:32922) 2024-04-02T07:37:05.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b23400700 1 -- 172.21.15.89:0/2983231503 learned_addr learned my addr 172.21.15.89:0/2983231503 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:05.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b23e00700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b240b6a60 0x7f0b240b6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b22a00700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b24160e50 0x7f0b2406cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b23400700 1 -- 172.21.15.89:0/2983231503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b24160e50 msgr2=0x7f0b2406cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:05.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b23400700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b24160e50 0x7f0b2406cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b23400700 1 -- 172.21.15.89:0/2983231503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b240b6a60 msgr2=0x7f0b240b6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:05.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b23400700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b240b6a60 0x7f0b240b6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b23400700 1 -- 172.21.15.89:0/2983231503 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c009b30 con 0x7f0b240b3f80 2024-04-02T07:37:05.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b22a00700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b24160e50 0x7f0b2406cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:05.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.244+0000 7f0b23400700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b240b3f80 0x7f0b2415aaa0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400e870 tx=0x7f0b1400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:05.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.245+0000 7f0b1b400700 1 -- 172.21.15.89:0/2983231503 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14009590 con 0x7f0b240b3f80 2024-04-02T07:37:05.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.245+0000 7f0b1b400700 1 -- 172.21.15.89:0/2983231503 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b14013680 con 0x7f0b240b3f80 2024-04-02T07:37:05.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.245+0000 7f0b2a400700 1 -- 172.21.15.89:0/2983231503 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2406be50 con 0x7f0b240b3f80 2024-04-02T07:37:05.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.245+0000 7f0b1b400700 1 -- 172.21.15.89:0/2983231503 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b140139f0 con 0x7f0b240b3f80 2024-04-02T07:37:05.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.245+0000 7f0b2a400700 1 -- 172.21.15.89:0/2983231503 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2406c410 con 0x7f0b240b3f80 2024-04-02T07:37:05.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.246+0000 7f0b2a400700 1 -- 172.21.15.89:0/2983231503 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b24067720 con 0x7f0b240b3f80 2024-04-02T07:37:05.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.247+0000 7f0b1b400700 1 -- 172.21.15.89:0/2983231503 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0b14013b50 con 0x7f0b240b3f80 2024-04-02T07:37:05.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.248+0000 7f0b1b400700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b0406ac90 0x7f0b0406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.248+0000 7f0b1b400700 1 -- 172.21.15.89:0/2983231503 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f0b14089990 con 0x7f0b240b3f80 2024-04-02T07:37:05.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.248+0000 7f0b22a00700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b0406ac90 0x7f0b0406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.249+0000 7f0b22a00700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b0406ac90 0x7f0b0406d150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0b240a1420 tx=0x7f0b10008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:05.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.252+0000 7f0b1b400700 1 -- 172.21.15.89:0/2983231503 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b14054bb0 con 0x7f0b240b3f80 2024-04-02T07:37:05.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.498+0000 7f0b2a400700 1 -- 172.21.15.89:0/2983231503 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0b240d2fc0 con 0x7f0b240b3f80 2024-04-02T07:37:05.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.506+0000 7f0b1b400700 1 -- 172.21.15.89:0/2983231503 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0b140170c0 con 0x7f0b240b3f80 2024-04-02T07:37:05.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.514+0000 7f0b2a400700 1 -- 172.21.15.89:0/2983231503 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b0406ac90 msgr2=0x7f0b0406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:05.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.514+0000 7f0b2a400700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b0406ac90 0x7f0b0406d150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0b240a1420 tx=0x7f0b10008040 comp rx=0 tx=0).stop 2024-04-02T07:37:05.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.514+0000 7f0b2a400700 1 -- 172.21.15.89:0/2983231503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b240b3f80 msgr2=0x7f0b2415aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:05.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.514+0000 7f0b2a400700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b240b3f80 0x7f0b2415aaa0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400e870 tx=0x7f0b1400eb80 comp rx=0 tx=0).stop 2024-04-02T07:37:05.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.514+0000 7f0b2a400700 1 -- 172.21.15.89:0/2983231503 shutdown_connections 2024-04-02T07:37:05.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.514+0000 7f0b2a400700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b0406ac90 0x7f0b0406d150 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.514+0000 7f0b2a400700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b240b6a60 0x7f0b240b6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.514+0000 7f0b2a400700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b240b3f80 0x7f0b2415aaa0 secure :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400e870 tx=0x7f0b1400eb80 comp rx=0 tx=0).stop 2024-04-02T07:37:05.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.514+0000 7f0b2a400700 1 --2- 172.21.15.89:0/2983231503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b24160e50 0x7f0b2406cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.514+0000 7f0b2a400700 1 -- 172.21.15.89:0/2983231503 >> 172.21.15.89:0/2983231503 conn(0x7f0b2408aff0 msgr2=0x7f0b240b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:05.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.515+0000 7f0b2a400700 1 -- 172.21.15.89:0/2983231503 shutdown_connections 2024-04-02T07:37:05.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.516+0000 7f0b2a400700 1 -- 172.21.15.89:0/2983231503 wait complete. 2024-04-02T07:37:05.522 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 4 and data pool 5 2024-04-02T07:37:05.548 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:37:05.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.736+0000 7f7c90400700 1 Processor -- start 2024-04-02T07:37:05.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.736+0000 7f7c90400700 1 -- start start 2024-04-02T07:37:05.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.737+0000 7f7c90400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7c880ce590 0x7f7c880ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.737+0000 7f7c90400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c880b6a20 0x7f7c880b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.737+0000 7f7c90400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c880ceef0 con 0x7f7c88160be0 2024-04-02T07:37:05.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.737+0000 7f7c90400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c880b65d0 con 0x7f7c880ce590 2024-04-02T07:37:05.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.737+0000 7f7c90400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880d2a80 con 0x7f7c880b6a20 2024-04-02T07:37:05.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.737+0000 7f7c8da00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7c88160be0 0x7f7c880cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47082/0 (socket says 172.21.15.89:47082) 2024-04-02T07:37:05.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.737+0000 7f7c8da00700 1 -- 172.21.15.89:0/2268980902 learned_addr learned my addr 172.21.15.89:0/2268980902 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:05.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c8d000700 1 --2- 172.21.15.89:0/2268980902 >> v2:172.21.15.113:3300/0 conn(0x7f7c880ce590 0x7f7c880ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c8e400700 1 --2- 172.21.15.89:0/2268980902 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c880b6a20 0x7f7c880b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 -- 172.21.15.89:0/2268980902 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253721995 0 0) 0x7f7c880ceef0 con 0x7f7c88160be0 2024-04-02T07:37:05.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 -- 172.21.15.89:0/2268980902 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c700035b0 con 0x7f7c88160be0 2024-04-02T07:37:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 -- 172.21.15.89:0/2268980902 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 670345830 0 0) 0x7f7c700035b0 con 0x7f7c88160be0 2024-04-02T07:37:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 -- 172.21.15.89:0/2268980902 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c880b6a20 msgr2=0x7f7c880b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 --2- 172.21.15.89:0/2268980902 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c880b6a20 0x7f7c880b6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 -- 172.21.15.89:0/2268980902 >> v2:172.21.15.113:3300/0 conn(0x7f7c880ce590 msgr2=0x7f7c880ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 --2- 172.21.15.89:0/2268980902 >> v2:172.21.15.113:3300/0 conn(0x7f7c880ce590 0x7f7c880ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 -- 172.21.15.89:0/2268980902 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c88160fc0 con 0x7f7c88160be0 2024-04-02T07:37:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 -- 172.21.15.89:0/2268980902 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f7c7c001670 con 0x7f7c88160be0 2024-04-02T07:37:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 -- 172.21.15.89:0/2268980902 >> v1:172.21.15.89:6789/0 conn(0x7f7c88160be0 legacy=0x7f7c880cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 --2- 172.21.15.89:0/2268980902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c70003a00 0x7f7c70003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 --2- 172.21.15.89:0/2268980902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c70004850 0x7f7c70004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 --2- 172.21.15.89:0/2268980902 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c880b6a20 0x7f7c70005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 -- 172.21.15.89:0/2268980902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880b65d0 con 0x7f7c70004850 2024-04-02T07:37:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 -- 172.21.15.89:0/2268980902 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880d2a80 con 0x7f7c70003a00 2024-04-02T07:37:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c87e00700 1 -- 172.21.15.89:0/2268980902 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7000ba10 con 0x7f7c880b6a20 2024-04-02T07:37:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.738+0000 7f7c8d000700 1 --2- 172.21.15.89:0/2268980902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c70004850 0x7f7c70004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.739+0000 7f7c8da00700 1 --2- 172.21.15.89:0/2268980902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c70003a00 0x7f7c70003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.739+0000 7f7c8e400700 1 --2- 172.21.15.89:0/2268980902 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c880b6a20 0x7f7c70005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.739+0000 7f7c8d000700 1 -- 172.21.15.89:0/2268980902 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c880b6a20 msgr2=0x7f7c70005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.739+0000 7f7c8d000700 1 --2- 172.21.15.89:0/2268980902 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c880b6a20 0x7f7c70005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.739+0000 7f7c8d000700 1 -- 172.21.15.89:0/2268980902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c70003a00 msgr2=0x7f7c70003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.739+0000 7f7c8d000700 1 --2- 172.21.15.89:0/2268980902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c70003a00 0x7f7c70003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.739+0000 7f7c8d000700 1 -- 172.21.15.89:0/2268980902 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c700035b0 con 0x7f7c70004850 2024-04-02T07:37:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.739+0000 7f7c8d000700 1 --2- 172.21.15.89:0/2268980902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c70004850 0x7f7c70004cd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7c74009b60 tx=0x7f7c70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a1036a454beb21 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.739+0000 7f7c87e00700 1 -- 172.21.15.89:0/2268980902 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7400e6c0 con 0x7f7c70004850 2024-04-02T07:37:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.739+0000 7f7c87e00700 1 -- 172.21.15.89:0/2268980902 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7400e820 con 0x7f7c70004850 2024-04-02T07:37:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.739+0000 7f7c87e00700 1 -- 172.21.15.89:0/2268980902 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7400eb90 con 0x7f7c70004850 2024-04-02T07:37:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.740+0000 7f7c90400700 1 -- 172.21.15.89:0/2268980902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c70004850 msgr2=0x7f7c70004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:05.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.740+0000 7f7c90400700 1 --2- 172.21.15.89:0/2268980902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c70004850 0x7f7c70004cd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7c74009b60 tx=0x7f7c70005880 comp rx=0 tx=0).stop 2024-04-02T07:37:05.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.740+0000 7f7c8e400700 1 -- 172.21.15.89:0/2268980902 reap_dead start 2024-04-02T07:37:05.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.740+0000 7f7c90400700 1 -- 172.21.15.89:0/2268980902 shutdown_connections 2024-04-02T07:37:05.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.740+0000 7f7c90400700 1 -- 172.21.15.89:0/2268980902 >> 172.21.15.89:0/2268980902 conn(0x7f7c8808aff0 msgr2=0x7f7c880d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:05.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.740+0000 7f7c90400700 1 -- 172.21.15.89:0/2268980902 shutdown_connections 2024-04-02T07:37:05.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.740+0000 7f7c90400700 1 -- 172.21.15.89:0/2268980902 wait complete. 2024-04-02T07:37:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.742+0000 7f7c90400700 1 Processor -- start 2024-04-02T07:37:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.742+0000 7f7c90400700 1 -- start start 2024-04-02T07:37:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.743+0000 7f7c90400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c70004850 0x7f7c880cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.743+0000 7f7c90400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c880c8330 0x7f7c880cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.743+0000 7f7c90400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c880c87b0 0x7f7c880cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.743+0000 7f7c90400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880c7080 con 0x7f7c880c8330 2024-04-02T07:37:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.743+0000 7f7c90400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880c7ea0 con 0x7f7c70004850 2024-04-02T07:37:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.743+0000 7f7c90400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880c7290 con 0x7f7c880c87b0 2024-04-02T07:37:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.743+0000 7f7c8d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c880c8330 0x7f7c880cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.743+0000 7f7c8d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c880c8330 0x7f7c880cb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32932/0 (socket says 172.21.15.89:32932) 2024-04-02T07:37:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.743+0000 7f7c8d000700 1 -- 172.21.15.89:0/4148106489 learned_addr learned my addr 172.21.15.89:0/4148106489 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.743+0000 7f7c8da00700 1 --2- 172.21.15.89:0/4148106489 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c70004850 0x7f7c880cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.743+0000 7f7c8e400700 1 --2- 172.21.15.89:0/4148106489 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c880c87b0 0x7f7c880cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.744+0000 7f7c8d000700 1 -- 172.21.15.89:0/4148106489 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c880c87b0 msgr2=0x7f7c880cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.744+0000 7f7c8d000700 1 --2- 172.21.15.89:0/4148106489 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c880c87b0 0x7f7c880cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.744+0000 7f7c8d000700 1 -- 172.21.15.89:0/4148106489 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c70004850 msgr2=0x7f7c880cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.744+0000 7f7c8d000700 1 --2- 172.21.15.89:0/4148106489 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c70004850 0x7f7c880cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.744+0000 7f7c8d000700 1 -- 172.21.15.89:0/4148106489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c880c7570 con 0x7f7c880c8330 2024-04-02T07:37:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.744+0000 7f7c8d000700 1 --2- 172.21.15.89:0/4148106489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c880c8330 0x7f7c880cb030 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7c740090b0 tx=0x7f7c740047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.744+0000 7f7c85600700 1 -- 172.21.15.89:0/4148106489 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c74021070 con 0x7f7c880c8330 2024-04-02T07:37:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.744+0000 7f7c85600700 1 -- 172.21.15.89:0/4148106489 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c74004430 con 0x7f7c880c8330 2024-04-02T07:37:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.744+0000 7f7c90400700 1 -- 172.21.15.89:0/4148106489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c880c7850 con 0x7f7c880c8330 2024-04-02T07:37:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.744+0000 7f7c85600700 1 -- 172.21.15.89:0/4148106489 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7400e6c0 con 0x7f7c880c8330 2024-04-02T07:37:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.745+0000 7f7c90400700 1 -- 172.21.15.89:0/4148106489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c880cc860 con 0x7f7c880c8330 2024-04-02T07:37:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.746+0000 7f7c90400700 1 -- 172.21.15.89:0/4148106489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c88067720 con 0x7f7c880c8330 2024-04-02T07:37:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.746+0000 7f7c85600700 1 -- 172.21.15.89:0/4148106489 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7c74025070 con 0x7f7c880c8330 2024-04-02T07:37:05.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.748+0000 7f7c85600700 1 --2- 172.21.15.89:0/4148106489 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c6806ac90 0x7f7c6806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:05.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.748+0000 7f7c85600700 1 -- 172.21.15.89:0/4148106489 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c7408a3c0 con 0x7f7c880c8330 2024-04-02T07:37:05.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.748+0000 7f7c8da00700 1 --2- 172.21.15.89:0/4148106489 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c6806ac90 0x7f7c6806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:05.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.749+0000 7f7c8da00700 1 --2- 172.21.15.89:0/4148106489 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c6806ac90 0x7f7c6806d150 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7c8815feb0 tx=0x7f7c7c007980 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:05.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.753+0000 7f7c85600700 1 -- 172.21.15.89:0/4148106489 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7c740556b0 con 0x7f7c880c8330 2024-04-02T07:37:05.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.956+0000 7f7c90400700 1 -- 172.21.15.89:0/4148106489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c880b5cb0 con 0x7f7c880c8330 2024-04-02T07:37:05.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.957+0000 7f7c85600700 1 -- 172.21.15.89:0/4148106489 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f7c74058cd0 con 0x7f7c880c8330 2024-04-02T07:37:05.960 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:37:05.960 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":33,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:37:05.500096+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:37:03.665694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-02T07:37:04.473147+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:37:05.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.960+0000 7f7c90400700 1 -- 172.21.15.89:0/4148106489 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c6806ac90 msgr2=0x7f7c6806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:05.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.960+0000 7f7c90400700 1 --2- 172.21.15.89:0/4148106489 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c6806ac90 0x7f7c6806d150 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7c8815feb0 tx=0x7f7c7c007980 comp rx=0 tx=0).stop 2024-04-02T07:37:05.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.960+0000 7f7c90400700 1 -- 172.21.15.89:0/4148106489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c880c8330 msgr2=0x7f7c880cb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:05.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.960+0000 7f7c90400700 1 --2- 172.21.15.89:0/4148106489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c880c8330 0x7f7c880cb030 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7c740090b0 tx=0x7f7c740047c0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.960+0000 7f7c90400700 1 -- 172.21.15.89:0/4148106489 shutdown_connections 2024-04-02T07:37:05.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.960+0000 7f7c90400700 1 --2- 172.21.15.89:0/4148106489 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c6806ac90 0x7f7c6806d150 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.960+0000 7f7c90400700 1 --2- 172.21.15.89:0/4148106489 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c70004850 0x7f7c880cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.960+0000 7f7c90400700 1 --2- 172.21.15.89:0/4148106489 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c880c87b0 0x7f7c880cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.960+0000 7f7c90400700 1 --2- 172.21.15.89:0/4148106489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c880c8330 0x7f7c880cb030 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:05.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.960+0000 7f7c90400700 1 -- 172.21.15.89:0/4148106489 >> 172.21.15.89:0/4148106489 conn(0x7f7c8808aff0 msgr2=0x7f7c880b8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:05.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.960+0000 7f7c90400700 1 -- 172.21.15.89:0/4148106489 shutdown_connections 2024-04-02T07:37:05.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:05.960+0000 7f7c90400700 1 -- 172.21.15.89:0/4148106489 wait complete. 2024-04-02T07:37:05.981 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:37:06.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.169+0000 7f5fd1000700 1 Processor -- start 2024-04-02T07:37:06.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.170+0000 7f5fd1000700 1 -- start start 2024-04-02T07:37:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.171+0000 7f5fd1000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5fcc0ceec0 0x7f5fcc1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.171+0000 7f5fd1000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fcc161c50 con 0x7f5fcc161800 2024-04-02T07:37:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.171+0000 7f5fd1000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fcc0d30d0 con 0x7f5fcc15d640 2024-04-02T07:37:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.171+0000 7f5fd1000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fcc0b6840 con 0x7f5fcc0ceec0 2024-04-02T07:37:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.171+0000 7f5fc3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5fcc0ceec0 0x7f5fcc1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.171+0000 7f5fc3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5fcc0ceec0 0x7f5fcc1612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32946/0 (socket says 172.21.15.89:32946) 2024-04-02T07:37:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.171+0000 7f5fc3e00700 1 -- 172.21.15.89:0/433030263 learned_addr learned my addr 172.21.15.89:0/433030263 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.171+0000 7f5fca800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5fcc161800 0x7f5fcc0b6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47096/0 (socket says 172.21.15.89:47096) 2024-04-02T07:37:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.171+0000 7f5fc3e00700 1 -- 172.21.15.89:0/433030263 >> v1:172.21.15.113:6789/0 conn(0x7f5fcc15d640 legacy=0x7f5fcc0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.171+0000 7f5fc3e00700 1 -- 172.21.15.89:0/433030263 >> v1:172.21.15.89:6789/0 conn(0x7f5fcc161800 legacy=0x7f5fcc0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:06.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.171+0000 7f5fc3e00700 1 -- 172.21.15.89:0/433030263 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc0d2b30 con 0x7f5fcc0ceec0 2024-04-02T07:37:06.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.172+0000 7f5fc3e00700 1 --2- 172.21.15.89:0/433030263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc0ceec0 0x7f5fcc1612c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40098f0 tx=0x7f5fb4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31a1c6ce21b6789 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:06.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.172+0000 7f5fc9400700 1 -- 172.21.15.89:0/433030263 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4004230 con 0x7f5fcc0ceec0 2024-04-02T07:37:06.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.172+0000 7f5fc9400700 1 -- 172.21.15.89:0/433030263 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb4004390 con 0x7f5fcc0ceec0 2024-04-02T07:37:06.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.172+0000 7f5fc9400700 1 -- 172.21.15.89:0/433030263 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb40036a0 con 0x7f5fcc0ceec0 2024-04-02T07:37:06.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.172+0000 7f5fd1000700 1 -- 172.21.15.89:0/433030263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc0ceec0 msgr2=0x7f5fcc1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:06.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.172+0000 7f5fd1000700 1 --2- 172.21.15.89:0/433030263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc0ceec0 0x7f5fcc1612c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40098f0 tx=0x7f5fb4009c00 comp rx=0 tx=0).stop 2024-04-02T07:37:06.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.173+0000 7f5fd1000700 1 -- 172.21.15.89:0/433030263 shutdown_connections 2024-04-02T07:37:06.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.173+0000 7f5fd1000700 1 --2- 172.21.15.89:0/433030263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc0ceec0 0x7f5fcc1612c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:06.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.173+0000 7f5fd1000700 1 -- 172.21.15.89:0/433030263 >> 172.21.15.89:0/433030263 conn(0x7f5fcc08aff0 msgr2=0x7f5fcc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:06.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.173+0000 7f5fd1000700 1 -- 172.21.15.89:0/433030263 shutdown_connections 2024-04-02T07:37:06.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.173+0000 7f5fd1000700 1 -- 172.21.15.89:0/433030263 wait complete. 2024-04-02T07:37:06.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.174+0000 7f5fd1000700 1 Processor -- start 2024-04-02T07:37:06.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.174+0000 7f5fd1000700 1 -- start start 2024-04-02T07:37:06.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.175+0000 7f5fd1000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fcc0ceec0 0x7f5fcc159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:06.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.175+0000 7f5fd1000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc15d640 0x7f5fcc14f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:06.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.175+0000 7f5fd1000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5fcc161800 0x7f5fcc0cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:06.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.175+0000 7f5fd1000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc0ce4c0 con 0x7f5fcc15d640 2024-04-02T07:37:06.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.175+0000 7f5fd1000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc09fce0 con 0x7f5fcc161800 2024-04-02T07:37:06.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.175+0000 7f5fd1000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc09fa00 con 0x7f5fcc0ceec0 2024-04-02T07:37:06.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.176+0000 7f5fc3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc15d640 0x7f5fcc14f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:06.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.176+0000 7f5fc3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc15d640 0x7f5fcc14f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32958/0 (socket says 172.21.15.89:32958) 2024-04-02T07:37:06.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.176+0000 7f5fc3e00700 1 -- 172.21.15.89:0/2728461496 learned_addr learned my addr 172.21.15.89:0/2728461496 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:06.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.176+0000 7f5fc9e00700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fcc0ceec0 0x7f5fcc159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:06.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.176+0000 7f5fca800700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5fcc161800 0x7f5fcc0cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:06.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.176+0000 7f5fc3e00700 1 -- 172.21.15.89:0/2728461496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fcc0ceec0 msgr2=0x7f5fcc159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:06.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.176+0000 7f5fc3e00700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fcc0ceec0 0x7f5fcc159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:06.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.176+0000 7f5fc3e00700 1 -- 172.21.15.89:0/2728461496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5fcc161800 msgr2=0x7f5fcc0cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:06.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.176+0000 7f5fc3e00700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5fcc161800 0x7f5fcc0cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:06.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.176+0000 7f5fc3e00700 1 -- 172.21.15.89:0/2728461496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb4009550 con 0x7f5fcc15d640 2024-04-02T07:37:06.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.176+0000 7f5fc9e00700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fcc0ceec0 0x7f5fcc159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:37:06.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.176+0000 7f5fca800700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5fcc161800 0x7f5fcc0cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:37:06.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.176+0000 7f5fc3e00700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc15d640 0x7f5fcc14f9b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc08c490 tx=0x7f5fb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:06.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.177+0000 7f5fc2000700 1 -- 172.21.15.89:0/2728461496 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb400b6a0 con 0x7f5fcc15d640 2024-04-02T07:37:06.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.177+0000 7f5fc2000700 1 -- 172.21.15.89:0/2728461496 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb400b820 con 0x7f5fcc15d640 2024-04-02T07:37:06.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.177+0000 7f5fc2000700 1 -- 172.21.15.89:0/2728461496 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb400bbd0 con 0x7f5fcc15d640 2024-04-02T07:37:06.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.177+0000 7f5fd1000700 1 -- 172.21.15.89:0/2728461496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fcc0cde70 con 0x7f5fcc15d640 2024-04-02T07:37:06.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.177+0000 7f5fd1000700 1 -- 172.21.15.89:0/2728461496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fcc09f6b0 con 0x7f5fcc15d640 2024-04-02T07:37:06.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.178+0000 7f5fd1000700 1 -- 172.21.15.89:0/2728461496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fcc0ce670 con 0x7f5fcc15d640 2024-04-02T07:37:06.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.179+0000 7f5fc2000700 1 -- 172.21.15.89:0/2728461496 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5fb400bd30 con 0x7f5fcc15d640 2024-04-02T07:37:06.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.179+0000 7f5fc2000700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5fa806ac90 0x7f5fa806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:06.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.180+0000 7f5fc2000700 1 -- 172.21.15.89:0/2728461496 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5fb408ad00 con 0x7f5fcc15d640 2024-04-02T07:37:06.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.180+0000 7f5fc9e00700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5fa806ac90 0x7f5fa806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:06.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.180+0000 7f5fc9e00700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5fa806ac90 0x7f5fa806d150 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc0017d0 tx=0x7f5fbc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:06.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.184+0000 7f5fc2000700 1 -- 172.21.15.89:0/2728461496 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5fcc0ce670 con 0x7f5fcc15d640 2024-04-02T07:37:06.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.386+0000 7f5fd1000700 1 -- 172.21.15.89:0/2728461496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fcc067720 con 0x7f5fcc15d640 2024-04-02T07:37:06.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.387+0000 7f5fc2000700 1 -- 172.21.15.89:0/2728461496 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f5fb401f090 con 0x7f5fcc15d640 2024-04-02T07:37:06.390 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:37:06.391 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":33,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:37:05.500096+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:37:03.665694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-02T07:37:04.473147+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:37:06.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.390+0000 7f5fd1000700 1 -- 172.21.15.89:0/2728461496 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5fa806ac90 msgr2=0x7f5fa806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:06.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.390+0000 7f5fd1000700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5fa806ac90 0x7f5fa806d150 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc0017d0 tx=0x7f5fbc009040 comp rx=0 tx=0).stop 2024-04-02T07:37:06.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.390+0000 7f5fd1000700 1 -- 172.21.15.89:0/2728461496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc15d640 msgr2=0x7f5fcc14f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:06.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.390+0000 7f5fd1000700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc15d640 0x7f5fcc14f9b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc08c490 tx=0x7f5fb4004750 comp rx=0 tx=0).stop 2024-04-02T07:37:06.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.390+0000 7f5fd1000700 1 -- 172.21.15.89:0/2728461496 shutdown_connections 2024-04-02T07:37:06.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.390+0000 7f5fd1000700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5fa806ac90 0x7f5fa806d150 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:06.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.390+0000 7f5fd1000700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5fcc161800 0x7f5fcc0cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:06.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.390+0000 7f5fd1000700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fcc0ceec0 0x7f5fcc159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:06.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.390+0000 7f5fd1000700 1 --2- 172.21.15.89:0/2728461496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc15d640 0x7f5fcc14f9b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:06.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.390+0000 7f5fd1000700 1 -- 172.21.15.89:0/2728461496 >> 172.21.15.89:0/2728461496 conn(0x7f5fcc08aff0 msgr2=0x7f5fcc0b7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:06.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.390+0000 7f5fd1000700 1 -- 172.21.15.89:0/2728461496 shutdown_connections 2024-04-02T07:37:06.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.390+0000 7f5fd1000700 1 -- 172.21.15.89:0/2728461496 wait complete. 2024-04-02T07:37:06.411 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T07:37:06.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.604+0000 7f2e97600700 1 Processor -- start 2024-04-02T07:37:06.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.604+0000 7f2e97600700 1 -- start start 2024-04-02T07:37:06.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.605+0000 7f2e97600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e901611b0 0x7f2e9015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:06.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.605+0000 7f2e97600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2e9015f4d0 0x7f2e900ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:06.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.605+0000 7f2e97600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2e900d21f0 0x7f2e900cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:06.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.605+0000 7f2e97600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e900cf180 con 0x7f2e900d21f0 2024-04-02T07:37:06.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.605+0000 7f2e97600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e900ce270 con 0x7f2e9015f4d0 2024-04-02T07:37:06.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.605+0000 7f2e97600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9015b600 con 0x7f2e901611b0 2024-04-02T07:37:06.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.606+0000 7f2e95600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2e900d21f0 0x7f2e900cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:06.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.606+0000 7f2e95600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2e900d21f0 0x7f2e900cec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32972/0 (socket says 172.21.15.89:32972) 2024-04-02T07:37:06.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.606+0000 7f2e95600700 1 -- 172.21.15.89:0/404872963 learned_addr learned my addr 172.21.15.89:0/404872963 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:06.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.606+0000 7f2e8fe00700 1 --2- 172.21.15.89:0/404872963 >> v2:172.21.15.113:3300/0 conn(0x7f2e9015f4d0 0x7f2e900ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:06.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.606+0000 7f2e94c00700 1 --2- 172.21.15.89:0/404872963 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e901611b0 0x7f2e9015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:06.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.606+0000 7f2e95600700 1 -- 172.21.15.89:0/404872963 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e901611b0 msgr2=0x7f2e9015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:06.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.606+0000 7f2e95600700 1 --2- 172.21.15.89:0/404872963 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e901611b0 0x7f2e9015ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:06.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.606+0000 7f2e95600700 1 -- 172.21.15.89:0/404872963 >> v2:172.21.15.113:3300/0 conn(0x7f2e9015f4d0 msgr2=0x7f2e900ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:06.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.606+0000 7f2e95600700 1 --2- 172.21.15.89:0/404872963 >> v2:172.21.15.113:3300/0 conn(0x7f2e9015f4d0 0x7f2e900ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:06.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.606+0000 7f2e95600700 1 -- 172.21.15.89:0/404872963 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e900d2760 con 0x7f2e900d21f0 2024-04-02T07:37:06.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.606+0000 7f2e95600700 1 --2- 172.21.15.89:0/404872963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e900d21f0 0x7f2e900cec40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2e9008d540 tx=0x7f2e8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b043f9f0bc8f1303 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:06.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.607+0000 7f2e8f400700 1 -- 172.21.15.89:0/404872963 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8800e070 con 0x7f2e900d21f0 2024-04-02T07:37:06.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.607+0000 7f2e8f400700 1 -- 172.21.15.89:0/404872963 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8800b9c0 con 0x7f2e900d21f0 2024-04-02T07:37:06.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.607+0000 7f2e8f400700 1 -- 172.21.15.89:0/404872963 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8800bd70 con 0x7f2e900d21f0 2024-04-02T07:37:06.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.607+0000 7f2e97600700 1 -- 172.21.15.89:0/404872963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e900d21f0 msgr2=0x7f2e900cec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:06.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.607+0000 7f2e97600700 1 --2- 172.21.15.89:0/404872963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e900d21f0 0x7f2e900cec40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2e9008d540 tx=0x7f2e8800b330 comp rx=0 tx=0).stop 2024-04-02T07:37:06.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.607+0000 7f2e97600700 1 -- 172.21.15.89:0/404872963 shutdown_connections 2024-04-02T07:37:06.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.607+0000 7f2e97600700 1 --2- 172.21.15.89:0/404872963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e900d21f0 0x7f2e900cec40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:06.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.607+0000 7f2e97600700 1 --2- 172.21.15.89:0/404872963 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e901611b0 0x7f2e9015ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:06.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.607+0000 7f2e97600700 1 --2- 172.21.15.89:0/404872963 >> v2:172.21.15.113:3300/0 conn(0x7f2e9015f4d0 0x7f2e900ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:06.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.607+0000 7f2e97600700 1 -- 172.21.15.89:0/404872963 >> 172.21.15.89:0/404872963 conn(0x7f2e9008aff0 msgr2=0x7f2e900d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:06.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.607+0000 7f2e97600700 1 -- 172.21.15.89:0/404872963 shutdown_connections 2024-04-02T07:37:06.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.607+0000 7f2e97600700 1 -- 172.21.15.89:0/404872963 wait complete. 2024-04-02T07:37:06.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.609+0000 7f2e97600700 1 Processor -- start 2024-04-02T07:37:06.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.609+0000 7f2e97600700 1 -- start start 2024-04-02T07:37:06.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.610+0000 7f2e97600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e900d21f0 0x7f2e9015ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:06.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.610+0000 7f2e97600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9015f4d0 0x7f2e900cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:06.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.610+0000 7f2e97600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e901611b0 0x7f2e900cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:06.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.610+0000 7f2e97600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e900cd0d0 con 0x7f2e9015f4d0 2024-04-02T07:37:06.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.610+0000 7f2e97600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e900b91c0 con 0x7f2e901611b0 2024-04-02T07:37:06.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.610+0000 7f2e97600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e900cc4c0 con 0x7f2e900d21f0 2024-04-02T07:37:06.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.610+0000 7f2e8fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9015f4d0 0x7f2e900cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:06.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.610+0000 7f2e8fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9015f4d0 0x7f2e900cb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32986/0 (socket says 172.21.15.89:32986) 2024-04-02T07:37:06.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.610+0000 7f2e8fe00700 1 -- 172.21.15.89:0/3984065070 learned_addr learned my addr 172.21.15.89:0/3984065070 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:06.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.610+0000 7f2e94c00700 1 --2- 172.21.15.89:0/3984065070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e900d21f0 0x7f2e9015ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:06.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.610+0000 7f2e95600700 1 --2- 172.21.15.89:0/3984065070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e901611b0 0x7f2e900cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:06.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.611+0000 7f2e8fe00700 1 -- 172.21.15.89:0/3984065070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e900d21f0 msgr2=0x7f2e9015ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:06.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.611+0000 7f2e8fe00700 1 --2- 172.21.15.89:0/3984065070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e900d21f0 0x7f2e9015ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:06.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.611+0000 7f2e8fe00700 1 -- 172.21.15.89:0/3984065070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e901611b0 msgr2=0x7f2e900cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:06.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.611+0000 7f2e8fe00700 1 --2- 172.21.15.89:0/3984065070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e901611b0 0x7f2e900cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:06.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.611+0000 7f2e8fe00700 1 -- 172.21.15.89:0/3984065070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e88009b30 con 0x7f2e9015f4d0 2024-04-02T07:37:06.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.611+0000 7f2e8fe00700 1 --2- 172.21.15.89:0/3984065070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9015f4d0 0x7f2e900cb900 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c00e7e0 tx=0x7f2e7c00eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:06.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.611+0000 7f2e8cc00700 1 -- 172.21.15.89:0/3984065070 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c00c070 con 0x7f2e9015f4d0 2024-04-02T07:37:06.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.611+0000 7f2e8cc00700 1 -- 172.21.15.89:0/3984065070 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7c013610 con 0x7f2e9015f4d0 2024-04-02T07:37:06.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.611+0000 7f2e8cc00700 1 -- 172.21.15.89:0/3984065070 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c0139a0 con 0x7f2e9015f4d0 2024-04-02T07:37:06.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.611+0000 7f2e97600700 1 -- 172.21.15.89:0/3984065070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e900cc7a0 con 0x7f2e9015f4d0 2024-04-02T07:37:06.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.611+0000 7f2e97600700 1 -- 172.21.15.89:0/3984065070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e900ccd60 con 0x7f2e9015f4d0 2024-04-02T07:37:06.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.612+0000 7f2e97600700 1 -- 172.21.15.89:0/3984065070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e900cd240 con 0x7f2e9015f4d0 2024-04-02T07:37:06.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.613+0000 7f2e8cc00700 1 -- 172.21.15.89:0/3984065070 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2e7c013b00 con 0x7f2e9015f4d0 2024-04-02T07:37:06.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.614+0000 7f2e8cc00700 1 --2- 172.21.15.89:0/3984065070 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2e7006ac90 0x7f2e7006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:06.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.614+0000 7f2e8cc00700 1 -- 172.21.15.89:0/3984065070 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e7c088fa0 con 0x7f2e9015f4d0 2024-04-02T07:37:06.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.614+0000 7f2e94c00700 1 --2- 172.21.15.89:0/3984065070 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2e7006ac90 0x7f2e7006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:06.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.615+0000 7f2e94c00700 1 --2- 172.21.15.89:0/3984065070 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2e7006ac90 0x7f2e7006d150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2e84006b20 tx=0x7f2e84008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:06.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.618+0000 7f2e8cc00700 1 -- 172.21.15.89:0/3984065070 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e7c054190 con 0x7f2e9015f4d0 2024-04-02T07:37:06.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:06.867+0000 7f2e97600700 1 -- 172.21.15.89:0/3984065070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2e900ccf20 con 0x7f2e9015f4d0 2024-04-02T07:37:07.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.511+0000 7f2e8cc00700 1 -- 172.21.15.89:0/3984065070 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2e7c013db0 con 0x7f2e9015f4d0 2024-04-02T07:37:07.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.515+0000 7f2e97600700 1 -- 172.21.15.89:0/3984065070 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2e7006ac90 msgr2=0x7f2e7006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:07.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.515+0000 7f2e97600700 1 --2- 172.21.15.89:0/3984065070 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2e7006ac90 0x7f2e7006d150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2e84006b20 tx=0x7f2e84008040 comp rx=0 tx=0).stop 2024-04-02T07:37:07.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.515+0000 7f2e97600700 1 -- 172.21.15.89:0/3984065070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9015f4d0 msgr2=0x7f2e900cb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:07.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.515+0000 7f2e97600700 1 --2- 172.21.15.89:0/3984065070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9015f4d0 0x7f2e900cb900 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c00e7e0 tx=0x7f2e7c00eaf0 comp rx=0 tx=0).stop 2024-04-02T07:37:07.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.515+0000 7f2e97600700 1 -- 172.21.15.89:0/3984065070 shutdown_connections 2024-04-02T07:37:07.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.515+0000 7f2e97600700 1 --2- 172.21.15.89:0/3984065070 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2e7006ac90 0x7f2e7006d150 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:07.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.515+0000 7f2e97600700 1 --2- 172.21.15.89:0/3984065070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e901611b0 0x7f2e900cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:07.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.515+0000 7f2e97600700 1 --2- 172.21.15.89:0/3984065070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e900d21f0 0x7f2e9015ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:07.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.515+0000 7f2e97600700 1 --2- 172.21.15.89:0/3984065070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9015f4d0 0x7f2e900cb900 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:07.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.515+0000 7f2e97600700 1 -- 172.21.15.89:0/3984065070 >> 172.21.15.89:0/3984065070 conn(0x7f2e9008aff0 msgr2=0x7f2e9015da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:07.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.515+0000 7f2e97600700 1 -- 172.21.15.89:0/3984065070 shutdown_connections 2024-04-02T07:37:07.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.515+0000 7f2e97600700 1 -- 172.21.15.89:0/3984065070 wait complete. 2024-04-02T07:37:07.538 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T07:37:07.539 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T07:37:07.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.733+0000 7f5796200700 1 Processor -- start 2024-04-02T07:37:07.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.733+0000 7f5796200700 1 -- start start 2024-04-02T07:37:07.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.734+0000 7f5796200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57900d16a0 0x7f57900d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:07.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.734+0000 7f5796200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f57900d2370 0x7f57900ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:07.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.734+0000 7f5796200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5790161420 con 0x7f579015d9d0 2024-04-02T07:37:07.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.734+0000 7f5796200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57900d4340 con 0x7f57900d16a0 2024-04-02T07:37:07.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.734+0000 7f5796200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f579015edc0 con 0x7f57900d2370 2024-04-02T07:37:07.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.734+0000 7f578ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57900d16a0 0x7f57900d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:07.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.735+0000 7f578f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f579015d9d0 0x7f57900d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47110/0 (socket says 172.21.15.89:47110) 2024-04-02T07:37:07.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.735+0000 7f578ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57900d16a0 0x7f57900d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32988/0 (socket says 172.21.15.89:32988) 2024-04-02T07:37:07.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.735+0000 7f578f400700 1 -- 172.21.15.89:0/2753586534 learned_addr learned my addr 172.21.15.89:0/2753586534 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:07.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.735+0000 7f578ea00700 1 -- 172.21.15.89:0/2753586534 >> v2:172.21.15.113:3300/0 conn(0x7f57900d2370 msgr2=0x7f57900ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:07.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.735+0000 7f578fe00700 1 --2- 172.21.15.89:0/2753586534 >> v2:172.21.15.113:3300/0 conn(0x7f57900d2370 0x7f57900ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:07.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.735+0000 7f578e000700 1 -- 172.21.15.89:0/2753586534 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685638105 0 0) 0x7f5790161420 con 0x7f579015d9d0 2024-04-02T07:37:07.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.735+0000 7f578ea00700 1 --2- 172.21.15.89:0/2753586534 >> v2:172.21.15.113:3300/0 conn(0x7f57900d2370 0x7f57900ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:07.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.735+0000 7f578ea00700 1 -- 172.21.15.89:0/2753586534 >> v1:172.21.15.89:6789/0 conn(0x7f579015d9d0 legacy=0x7f57900d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:07.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.735+0000 7f578ea00700 1 -- 172.21.15.89:0/2753586534 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5790161700 con 0x7f57900d16a0 2024-04-02T07:37:07.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.735+0000 7f578fe00700 1 --2- 172.21.15.89:0/2753586534 >> v2:172.21.15.113:3300/0 conn(0x7f57900d2370 0x7f57900ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:37:07.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.735+0000 7f578ea00700 1 --2- 172.21.15.89:0/2753586534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57900d16a0 0x7f57900d1e30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f579008ca10 tx=0x7f577c009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6dc70bee96dc118 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:07.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.737+0000 7f578e000700 1 -- 172.21.15.89:0/2753586534 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c005e30 con 0x7f57900d16a0 2024-04-02T07:37:07.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.737+0000 7f578e000700 1 -- 172.21.15.89:0/2753586534 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577c006030 con 0x7f57900d16a0 2024-04-02T07:37:07.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.737+0000 7f578e000700 1 -- 172.21.15.89:0/2753586534 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c0063c0 con 0x7f57900d16a0 2024-04-02T07:37:07.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.738+0000 7f5796200700 1 -- 172.21.15.89:0/2753586534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57900d16a0 msgr2=0x7f57900d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:07.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.738+0000 7f5796200700 1 --2- 172.21.15.89:0/2753586534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57900d16a0 0x7f57900d1e30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f579008ca10 tx=0x7f577c009b00 comp rx=0 tx=0).stop 2024-04-02T07:37:07.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.738+0000 7f5796200700 1 -- 172.21.15.89:0/2753586534 shutdown_connections 2024-04-02T07:37:07.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.738+0000 7f5796200700 1 --2- 172.21.15.89:0/2753586534 >> v2:172.21.15.113:3300/0 conn(0x7f57900d2370 0x7f57900ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:07.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.738+0000 7f5796200700 1 --2- 172.21.15.89:0/2753586534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57900d16a0 0x7f57900d1e30 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:07.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.738+0000 7f5796200700 1 -- 172.21.15.89:0/2753586534 >> 172.21.15.89:0/2753586534 conn(0x7f579008aff0 msgr2=0x7f57900d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:07.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.738+0000 7f5796200700 1 -- 172.21.15.89:0/2753586534 shutdown_connections 2024-04-02T07:37:07.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.738+0000 7f5796200700 1 -- 172.21.15.89:0/2753586534 wait complete. 2024-04-02T07:37:07.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.740+0000 7f5796200700 1 Processor -- start 2024-04-02T07:37:07.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.740+0000 7f5796200700 1 -- start start 2024-04-02T07:37:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.741+0000 7f5796200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57900d16a0 0x7f579015c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.741+0000 7f5796200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57900d2370 0x7f57900cca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.741+0000 7f5796200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f579015d9d0 0x7f57901e7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.741+0000 7f5796200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57900cd6a0 con 0x7f57900d2370 2024-04-02T07:37:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.741+0000 7f5796200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57900cd3c0 con 0x7f57900d16a0 2024-04-02T07:37:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.741+0000 7f5796200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57901e7420 con 0x7f579015d9d0 2024-04-02T07:37:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.741+0000 7f578ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57900d2370 0x7f57900cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.741+0000 7f578ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57900d2370 0x7f57900cca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33004/0 (socket says 172.21.15.89:33004) 2024-04-02T07:37:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.741+0000 7f578ea00700 1 -- 172.21.15.89:0/3658805705 learned_addr learned my addr 172.21.15.89:0/3658805705 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.742+0000 7f578f400700 1 --2- 172.21.15.89:0/3658805705 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57900d16a0 0x7f579015c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.742+0000 7f578fe00700 1 --2- 172.21.15.89:0/3658805705 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f579015d9d0 0x7f57901e7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.742+0000 7f578ea00700 1 -- 172.21.15.89:0/3658805705 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f579015d9d0 msgr2=0x7f57901e7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.742+0000 7f578ea00700 1 --2- 172.21.15.89:0/3658805705 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f579015d9d0 0x7f57901e7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.742+0000 7f578ea00700 1 -- 172.21.15.89:0/3658805705 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57900d16a0 msgr2=0x7f579015c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.742+0000 7f578ea00700 1 --2- 172.21.15.89:0/3658805705 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57900d16a0 0x7f579015c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.742+0000 7f578ea00700 1 -- 172.21.15.89:0/3658805705 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577c009430 con 0x7f57900d2370 2024-04-02T07:37:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.742+0000 7f578ea00700 1 --2- 172.21.15.89:0/3658805705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57900d2370 0x7f57900cca50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f577c000c00 tx=0x7f577c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.742+0000 7f5787400700 1 -- 172.21.15.89:0/3658805705 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c004240 con 0x7f57900d2370 2024-04-02T07:37:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.742+0000 7f5787400700 1 -- 172.21.15.89:0/3658805705 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577c0043c0 con 0x7f57900d2370 2024-04-02T07:37:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.742+0000 7f5787400700 1 -- 172.21.15.89:0/3658805705 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c006940 con 0x7f57900d2370 2024-04-02T07:37:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.742+0000 7f5796200700 1 -- 172.21.15.89:0/3658805705 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57901e7700 con 0x7f57900d2370 2024-04-02T07:37:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.743+0000 7f5796200700 1 -- 172.21.15.89:0/3658805705 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f579015c6d0 con 0x7f57900d2370 2024-04-02T07:37:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.744+0000 7f5796200700 1 -- 172.21.15.89:0/3658805705 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57900d1ca0 con 0x7f57900d2370 2024-04-02T07:37:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.744+0000 7f5787400700 1 -- 172.21.15.89:0/3658805705 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f577c022070 con 0x7f57900d2370 2024-04-02T07:37:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.745+0000 7f5787400700 1 --2- 172.21.15.89:0/3658805705 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f577006ac90 0x7f577006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.745+0000 7f5787400700 1 -- 172.21.15.89:0/3658805705 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f577c08aae0 con 0x7f57900d2370 2024-04-02T07:37:07.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.746+0000 7f578f400700 1 --2- 172.21.15.89:0/3658805705 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f577006ac90 0x7f577006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:07.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.746+0000 7f578f400700 1 --2- 172.21.15.89:0/3658805705 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f577006ac90 0x7f577006d150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5780000f60 tx=0x7f5780007e70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:07.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.750+0000 7f5787400700 1 -- 172.21.15.89:0/3658805705 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f577c055cb0 con 0x7f57900d2370 2024-04-02T07:37:07.974 INFO:tasks.ceph.mds.a.smithi089.stderr:2024-04-02T07:37:07.972+0000 7fe189e00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T07:37:08.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:07.999+0000 7f5796200700 1 -- 172.21.15.89:0/3658805705 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f57901e7280 con 0x7f57900d2370 2024-04-02T07:37:08.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.740+0000 7f5787400700 1 -- 172.21.15.89:0/3658805705 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7f57901e7280 con 0x7f57900d2370 2024-04-02T07:37:08.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.742+0000 7f5796200700 1 -- 172.21.15.89:0/3658805705 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f577006ac90 msgr2=0x7f577006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:08.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.742+0000 7f5796200700 1 --2- 172.21.15.89:0/3658805705 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f577006ac90 0x7f577006d150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5780000f60 tx=0x7f5780007e70 comp rx=0 tx=0).stop 2024-04-02T07:37:08.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.743+0000 7f5796200700 1 -- 172.21.15.89:0/3658805705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57900d2370 msgr2=0x7f57900cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:08.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.743+0000 7f5796200700 1 --2- 172.21.15.89:0/3658805705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57900d2370 0x7f57900cca50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f577c000c00 tx=0x7f577c004750 comp rx=0 tx=0).stop 2024-04-02T07:37:08.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.743+0000 7f5796200700 1 -- 172.21.15.89:0/3658805705 shutdown_connections 2024-04-02T07:37:08.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.743+0000 7f5796200700 1 --2- 172.21.15.89:0/3658805705 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f577006ac90 0x7f577006d150 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:08.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.743+0000 7f5796200700 1 --2- 172.21.15.89:0/3658805705 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57900d16a0 0x7f579015c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:08.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.743+0000 7f5796200700 1 --2- 172.21.15.89:0/3658805705 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f579015d9d0 0x7f57901e7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:08.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.743+0000 7f5796200700 1 --2- 172.21.15.89:0/3658805705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57900d2370 0x7f57900cca50 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:08.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.743+0000 7f5796200700 1 -- 172.21.15.89:0/3658805705 >> 172.21.15.89:0/3658805705 conn(0x7f579008aff0 msgr2=0x7f57900b0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:08.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.743+0000 7f5796200700 1 -- 172.21.15.89:0/3658805705 shutdown_connections 2024-04-02T07:37:08.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.743+0000 7f5796200700 1 -- 172.21.15.89:0/3658805705 wait complete. 2024-04-02T07:37:08.766 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T07:37:08.767 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:37:08.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.958+0000 7fc568800700 1 Processor -- start 2024-04-02T07:37:08.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.958+0000 7fc568800700 1 -- start start 2024-04-02T07:37:08.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.959+0000 7fc568800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc560161ca0 0x7fc5600cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:08.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.959+0000 7fc568800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5600ceda0 con 0x7fc560160f10 2024-04-02T07:37:08.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.959+0000 7fc568800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5601612f0 con 0x7fc5600d18b0 2024-04-02T07:37:08.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.959+0000 7fc568800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5600d1d30 con 0x7fc560161ca0 2024-04-02T07:37:08.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.959+0000 7fc565e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc560160f10 0x7fc5600d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40106/0 (socket says 172.21.15.89:40106) 2024-04-02T07:37:08.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.959+0000 7fc565e00700 1 -- 172.21.15.89:0/135052564 learned_addr learned my addr 172.21.15.89:0/135052564 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:08.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.959+0000 7fc564a00700 1 -- 172.21.15.89:0/135052564 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960231301 0 0) 0x7fc5600ceda0 con 0x7fc560160f10 2024-04-02T07:37:08.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.959+0000 7fc564a00700 1 -- 172.21.15.89:0/135052564 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5480035b0 con 0x7fc560160f10 2024-04-02T07:37:08.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.959+0000 7fc566800700 1 --2- 172.21.15.89:0/135052564 >> v2:172.21.15.113:3300/0 conn(0x7fc560161ca0 0x7fc5600cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:08.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc564a00700 1 -- 172.21.15.89:0/135052564 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1472469839 0 0) 0x7fc5480035b0 con 0x7fc560160f10 2024-04-02T07:37:08.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc564a00700 1 -- 172.21.15.89:0/135052564 >> v2:172.21.15.113:3300/0 conn(0x7fc560161ca0 msgr2=0x7fc5600cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:08.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc564a00700 1 --2- 172.21.15.89:0/135052564 >> v2:172.21.15.113:3300/0 conn(0x7fc560161ca0 0x7fc5600cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:08.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc564a00700 1 -- 172.21.15.89:0/135052564 >> v1:172.21.15.113:6789/0 conn(0x7fc5600d18b0 legacy=0x7fc560161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:08.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc564a00700 1 -- 172.21.15.89:0/135052564 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5600ce470 con 0x7fc560160f10 2024-04-02T07:37:08.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc564a00700 1 -- 172.21.15.89:0/135052564 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fc554001670 con 0x7fc560160f10 2024-04-02T07:37:08.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc564a00700 1 -- 172.21.15.89:0/135052564 >> v1:172.21.15.89:6789/0 conn(0x7fc560160f10 legacy=0x7fc5600d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:08.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc564a00700 1 --2- 172.21.15.89:0/135052564 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548003a00 0x7fc548003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:08.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc564a00700 1 --2- 172.21.15.89:0/135052564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548004850 0x7fc548004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:08.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc564a00700 1 --2- 172.21.15.89:0/135052564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc548005210 0x7fc54800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:08.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc564a00700 1 -- 172.21.15.89:0/135052564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600d1d30 con 0x7fc548004850 2024-04-02T07:37:08.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc564a00700 1 -- 172.21.15.89:0/135052564 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54800c1b0 con 0x7fc548005210 2024-04-02T07:37:08.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc564a00700 1 -- 172.21.15.89:0/135052564 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54800c370 con 0x7fc548003a00 2024-04-02T07:37:08.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc565400700 1 --2- 172.21.15.89:0/135052564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548004850 0x7fc548004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:08.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc566800700 1 --2- 172.21.15.89:0/135052564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc548005210 0x7fc54800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:08.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.960+0000 7fc565e00700 1 --2- 172.21.15.89:0/135052564 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548003a00 0x7fc548003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:08.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.961+0000 7fc565400700 1 -- 172.21.15.89:0/135052564 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548003a00 msgr2=0x7fc548003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:08.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.961+0000 7fc565400700 1 --2- 172.21.15.89:0/135052564 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548003a00 0x7fc548003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:08.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.961+0000 7fc565400700 1 -- 172.21.15.89:0/135052564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc548005210 msgr2=0x7fc54800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:08.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.961+0000 7fc565400700 1 --2- 172.21.15.89:0/135052564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc548005210 0x7fc54800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:08.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.961+0000 7fc565400700 1 -- 172.21.15.89:0/135052564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5480035b0 con 0x7fc548004850 2024-04-02T07:37:08.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.961+0000 7fc566800700 1 -- 172.21.15.89:0/135052564 reap_dead start 2024-04-02T07:37:08.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.961+0000 7fc565400700 1 --2- 172.21.15.89:0/135052564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548004850 0x7fc548004cd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc5500017c0 tx=0x7fc548005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4de3fca32e2fc69c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:08.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.961+0000 7fc564a00700 1 -- 172.21.15.89:0/135052564 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5500052d0 con 0x7fc548004850 2024-04-02T07:37:08.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.961+0000 7fc564a00700 1 -- 172.21.15.89:0/135052564 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc550005450 con 0x7fc548004850 2024-04-02T07:37:08.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.961+0000 7fc564a00700 1 -- 172.21.15.89:0/135052564 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5500038c0 con 0x7fc548004850 2024-04-02T07:37:08.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.962+0000 7fc568800700 1 -- 172.21.15.89:0/135052564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548004850 msgr2=0x7fc548004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:08.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.962+0000 7fc568800700 1 --2- 172.21.15.89:0/135052564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548004850 0x7fc548004cd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc5500017c0 tx=0x7fc548005880 comp rx=0 tx=0).stop 2024-04-02T07:37:08.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.962+0000 7fc568800700 1 -- 172.21.15.89:0/135052564 shutdown_connections 2024-04-02T07:37:08.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.962+0000 7fc568800700 1 --2- 172.21.15.89:0/135052564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548004850 0x7fc548004cd0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:08.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.962+0000 7fc568800700 1 -- 172.21.15.89:0/135052564 >> 172.21.15.89:0/135052564 conn(0x7fc56008aff0 msgr2=0x7fc5600d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:08.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.962+0000 7fc568800700 1 -- 172.21.15.89:0/135052564 shutdown_connections 2024-04-02T07:37:08.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.962+0000 7fc568800700 1 -- 172.21.15.89:0/135052564 wait complete. 2024-04-02T07:37:08.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.964+0000 7fc568800700 1 Processor -- start 2024-04-02T07:37:08.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.964+0000 7fc568800700 1 -- start start 2024-04-02T07:37:08.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.965+0000 7fc568800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548004850 0x7fc5600b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:08.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.965+0000 7fc568800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5601589f0 0x7fc56015f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:08.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc568800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc560158180 0x7fc5601e87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:08.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc568800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56015fe40 con 0x7fc548004850 2024-04-02T07:37:08.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc568800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5601f3510 con 0x7fc560158180 2024-04-02T07:37:08.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc568800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5601590f0 con 0x7fc5601589f0 2024-04-02T07:37:08.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc565e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548004850 0x7fc5600b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:08.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc565e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548004850 0x7fc5600b61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57990/0 (socket says 172.21.15.89:57990) 2024-04-02T07:37:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc565e00700 1 -- 172.21.15.89:0/1242173528 learned_addr learned my addr 172.21.15.89:0/1242173528 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc566800700 1 --2- 172.21.15.89:0/1242173528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc560158180 0x7fc5601e87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc565400700 1 --2- 172.21.15.89:0/1242173528 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5601589f0 0x7fc56015f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc565e00700 1 -- 172.21.15.89:0/1242173528 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5601589f0 msgr2=0x7fc56015f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc565e00700 1 --2- 172.21.15.89:0/1242173528 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5601589f0 0x7fc56015f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc565e00700 1 -- 172.21.15.89:0/1242173528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc560158180 msgr2=0x7fc5601e87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc565e00700 1 --2- 172.21.15.89:0/1242173528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc560158180 0x7fc5601e87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc565e00700 1 -- 172.21.15.89:0/1242173528 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56015f880 con 0x7fc548004850 2024-04-02T07:37:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc565e00700 1 --2- 172.21.15.89:0/1242173528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548004850 0x7fc5600b61d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc554007290 tx=0x7fc554007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.966+0000 7fc55a000700 1 -- 172.21.15.89:0/1242173528 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400f640 con 0x7fc548004850 2024-04-02T07:37:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.967+0000 7fc55a000700 1 -- 172.21.15.89:0/1242173528 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55400f7c0 con 0x7fc548004850 2024-04-02T07:37:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.967+0000 7fc55a000700 1 -- 172.21.15.89:0/1242173528 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400fb70 con 0x7fc548004850 2024-04-02T07:37:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.967+0000 7fc568800700 1 -- 172.21.15.89:0/1242173528 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56015fb60 con 0x7fc548004850 2024-04-02T07:37:08.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.967+0000 7fc568800700 1 -- 172.21.15.89:0/1242173528 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5601e7210 con 0x7fc548004850 2024-04-02T07:37:08.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.968+0000 7fc568800700 1 -- 172.21.15.89:0/1242173528 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5601e6d90 con 0x7fc548004850 2024-04-02T07:37:08.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.969+0000 7fc55a000700 1 -- 172.21.15.89:0/1242173528 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc55400fcd0 con 0x7fc548004850 2024-04-02T07:37:08.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.969+0000 7fc55a000700 1 --2- 172.21.15.89:0/1242173528 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc54406ac90 0x7fc54406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:08.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.970+0000 7fc55a000700 1 -- 172.21.15.89:0/1242173528 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5540892f0 con 0x7fc548004850 2024-04-02T07:37:08.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.970+0000 7fc565400700 1 --2- 172.21.15.89:0/1242173528 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc54406ac90 0x7fc54406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:08.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.970+0000 7fc565400700 1 --2- 172.21.15.89:0/1242173528 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc54406ac90 0x7fc54406d150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc550005040 tx=0x7fc550009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:08.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:08.975+0000 7fc55a000700 1 -- 172.21.15.89:0/1242173528 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc5601e6d90 con 0x7fc548004850 2024-04-02T07:37:09.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.217+0000 7fc568800700 1 -- 172.21.15.89:0/1242173528 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5601e6d90 con 0x7fc548004850 2024-04-02T07:37:09.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.218+0000 7fc55a000700 1 -- 172.21.15.89:0/1242173528 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4737 (secure 0 0 0) 0x7fc5601e6d90 con 0x7fc548004850 2024-04-02T07:37:09.221 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:37:09.221 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5037,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3788023013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3788023013},{"type":"v1","addr":"172.21.15.89:6835","nonce":3788023013}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5040,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4029308904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":4029308904},{"type":"v1","addr":"172.21.15.113:6835","nonce":4029308904}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5043,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2115217020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2115217020},{"type":"v1","addr":"172.21.15.113:6834","nonce":2115217020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:37:05.500422+0000","modified":"2024-04-02T07:37:08.737868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.89:6837/929002087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":929002087},{"type":"v1","addr":"172.21.15.89:6837","nonce":929002087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-02T07:37:09.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.220+0000 7fc568800700 1 -- 172.21.15.89:0/1242173528 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc54406ac90 msgr2=0x7fc54406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:09.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.220+0000 7fc568800700 1 --2- 172.21.15.89:0/1242173528 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc54406ac90 0x7fc54406d150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc550005040 tx=0x7fc550009030 comp rx=0 tx=0).stop 2024-04-02T07:37:09.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.220+0000 7fc568800700 1 -- 172.21.15.89:0/1242173528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548004850 msgr2=0x7fc5600b61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:09.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.220+0000 7fc568800700 1 --2- 172.21.15.89:0/1242173528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548004850 0x7fc5600b61d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc554007290 tx=0x7fc554007650 comp rx=0 tx=0).stop 2024-04-02T07:37:09.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.220+0000 7fc568800700 1 -- 172.21.15.89:0/1242173528 shutdown_connections 2024-04-02T07:37:09.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.220+0000 7fc568800700 1 --2- 172.21.15.89:0/1242173528 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc54406ac90 0x7fc54406d150 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.220+0000 7fc568800700 1 --2- 172.21.15.89:0/1242173528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc560158180 0x7fc5601e87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.220+0000 7fc568800700 1 --2- 172.21.15.89:0/1242173528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548004850 0x7fc5600b61d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.220+0000 7fc568800700 1 --2- 172.21.15.89:0/1242173528 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5601589f0 0x7fc56015f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.221+0000 7fc568800700 1 -- 172.21.15.89:0/1242173528 >> 172.21.15.89:0/1242173528 conn(0x7fc56008aff0 msgr2=0x7fc5600d0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:09.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.221+0000 7fc568800700 1 -- 172.21.15.89:0/1242173528 shutdown_connections 2024-04-02T07:37:09.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.221+0000 7fc568800700 1 -- 172.21.15.89:0/1242173528 wait complete. 2024-04-02T07:37:09.225 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 17 2024-04-02T07:37:09.241 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:37:09.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.429+0000 7f162a800700 1 Processor -- start 2024-04-02T07:37:09.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.429+0000 7f162a800700 1 -- start start 2024-04-02T07:37:09.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.430+0000 7f162a800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1624160f10 0x7f16240d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:09.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.430+0000 7f162a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1624161590 0x7f1624161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:09.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.430+0000 7f162a800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16240ceda0 con 0x7f16240cf230 2024-04-02T07:37:09.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.430+0000 7f162a800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1624161f80 con 0x7f1624160f10 2024-04-02T07:37:09.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.431+0000 7f162a800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16241612f0 con 0x7f1624161590 2024-04-02T07:37:09.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.431+0000 7f161be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f16240cf230 0x7f16240d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40114/0 (socket says 172.21.15.89:40114) 2024-04-02T07:37:09.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.431+0000 7f161be00700 1 -- 172.21.15.89:0/3843238636 learned_addr learned my addr 172.21.15.89:0/3843238636 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:09.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.431+0000 7f1623e00700 1 --2- 172.21.15.89:0/3843238636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1624161590 0x7f1624161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:09.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.431+0000 7f1623400700 1 --2- 172.21.15.89:0/3843238636 >> v2:172.21.15.113:3300/0 conn(0x7f1624160f10 0x7f16240d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:09.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.431+0000 7f1622a00700 1 -- 172.21.15.89:0/3843238636 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774178240 0 0) 0x7f16240ceda0 con 0x7f16240cf230 2024-04-02T07:37:09.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.431+0000 7f1622a00700 1 -- 172.21.15.89:0/3843238636 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16080035b0 con 0x7f16240cf230 2024-04-02T07:37:09.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.431+0000 7f1623e00700 1 -- 172.21.15.89:0/3843238636 >> v2:172.21.15.113:3300/0 conn(0x7f1624160f10 msgr2=0x7f16240d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:09.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.431+0000 7f1623e00700 1 --2- 172.21.15.89:0/3843238636 >> v2:172.21.15.113:3300/0 conn(0x7f1624160f10 0x7f16240d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.431+0000 7f1623e00700 1 -- 172.21.15.89:0/3843238636 >> v1:172.21.15.89:6789/0 conn(0x7f16240cf230 legacy=0x7f16240d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:09.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.431+0000 7f1623e00700 1 -- 172.21.15.89:0/3843238636 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16240ce470 con 0x7f1624161590 2024-04-02T07:37:09.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.432+0000 7f1623e00700 1 --2- 172.21.15.89:0/3843238636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1624161590 0x7f1624161a40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f161c009d90 tx=0x7f161c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb365e88101a4c3f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:09.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.432+0000 7f1622a00700 1 -- 172.21.15.89:0/3843238636 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c00e070 con 0x7f1624161590 2024-04-02T07:37:09.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.432+0000 7f1622a00700 1 -- 172.21.15.89:0/3843238636 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161c00bb20 con 0x7f1624161590 2024-04-02T07:37:09.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.432+0000 7f1622a00700 1 -- 172.21.15.89:0/3843238636 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c012430 con 0x7f1624161590 2024-04-02T07:37:09.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.432+0000 7f162a800700 1 -- 172.21.15.89:0/3843238636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1624161590 msgr2=0x7f1624161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:09.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.432+0000 7f162a800700 1 --2- 172.21.15.89:0/3843238636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1624161590 0x7f1624161a40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f161c009d90 tx=0x7f161c00b330 comp rx=0 tx=0).stop 2024-04-02T07:37:09.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.433+0000 7f162a800700 1 -- 172.21.15.89:0/3843238636 shutdown_connections 2024-04-02T07:37:09.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.433+0000 7f162a800700 1 --2- 172.21.15.89:0/3843238636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1624161590 0x7f1624161a40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.433+0000 7f162a800700 1 --2- 172.21.15.89:0/3843238636 >> v2:172.21.15.113:3300/0 conn(0x7f1624160f10 0x7f16240d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.433+0000 7f162a800700 1 -- 172.21.15.89:0/3843238636 >> 172.21.15.89:0/3843238636 conn(0x7f162408aff0 msgr2=0x7f16240d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:09.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.433+0000 7f162a800700 1 -- 172.21.15.89:0/3843238636 shutdown_connections 2024-04-02T07:37:09.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.433+0000 7f162a800700 1 -- 172.21.15.89:0/3843238636 wait complete. 2024-04-02T07:37:09.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.434+0000 7f162a800700 1 Processor -- start 2024-04-02T07:37:09.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.434+0000 7f162a800700 1 -- start start 2024-04-02T07:37:09.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.435+0000 7f162a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16240cf230 0x7f16241f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:09.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.435+0000 7f162a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1624160f10 0x7f16240b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:09.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.435+0000 7f162a800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1624161590 0x7f162415fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:09.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.435+0000 7f162a800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162409ea00 con 0x7f1624160f10 2024-04-02T07:37:09.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.435+0000 7f162a800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162409ece0 con 0x7f1624161590 2024-04-02T07:37:09.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.435+0000 7f162a800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f162409efc0 con 0x7f16240cf230 2024-04-02T07:37:09.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.436+0000 7f161be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1624160f10 0x7f16240b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:09.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.436+0000 7f161be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1624160f10 0x7f16240b82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57998/0 (socket says 172.21.15.89:57998) 2024-04-02T07:37:09.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.436+0000 7f161be00700 1 -- 172.21.15.89:0/582240384 learned_addr learned my addr 172.21.15.89:0/582240384 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:09.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.436+0000 7f161be00700 1 -- 172.21.15.89:0/582240384 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16240cf230 msgr2=0x7f16241f25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:09.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.436+0000 7f161be00700 1 --2- 172.21.15.89:0/582240384 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16240cf230 0x7f16241f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.436+0000 7f1623e00700 1 --2- 172.21.15.89:0/582240384 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1624161590 0x7f162415fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:09.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.436+0000 7f161be00700 1 -- 172.21.15.89:0/582240384 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1624161590 msgr2=0x7f162415fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:09.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.436+0000 7f161be00700 1 --2- 172.21.15.89:0/582240384 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1624161590 0x7f162415fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.436+0000 7f161be00700 1 -- 172.21.15.89:0/582240384 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161c009a70 con 0x7f1624160f10 2024-04-02T07:37:09.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.436+0000 7f1623e00700 1 --2- 172.21.15.89:0/582240384 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1624161590 0x7f162415fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:37:09.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.436+0000 7f161be00700 1 --2- 172.21.15.89:0/582240384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1624160f10 0x7f16240b82f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f160c005910 tx=0x7f160c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:09.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.436+0000 7f161b400700 1 -- 172.21.15.89:0/582240384 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c005d70 con 0x7f1624160f10 2024-04-02T07:37:09.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.436+0000 7f161b400700 1 -- 172.21.15.89:0/582240384 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160c0070d0 con 0x7f1624160f10 2024-04-02T07:37:09.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.436+0000 7f162a800700 1 -- 172.21.15.89:0/582240384 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162409f2a0 con 0x7f1624160f10 2024-04-02T07:37:09.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.437+0000 7f161b400700 1 -- 172.21.15.89:0/582240384 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c007440 con 0x7f1624160f10 2024-04-02T07:37:09.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.437+0000 7f162a800700 1 -- 172.21.15.89:0/582240384 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16241f2b10 con 0x7f1624160f10 2024-04-02T07:37:09.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.438+0000 7f162a800700 1 -- 172.21.15.89:0/582240384 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f162409f7c0 con 0x7f1624160f10 2024-04-02T07:37:09.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.438+0000 7f161b400700 1 -- 172.21.15.89:0/582240384 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f160c0075a0 con 0x7f1624160f10 2024-04-02T07:37:09.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.439+0000 7f161b400700 1 --2- 172.21.15.89:0/582240384 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f160006ac90 0x7f160006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:09.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.439+0000 7f161b400700 1 -- 172.21.15.89:0/582240384 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f160c0886d0 con 0x7f1624160f10 2024-04-02T07:37:09.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.440+0000 7f1623400700 1 --2- 172.21.15.89:0/582240384 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f160006ac90 0x7f160006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:09.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.440+0000 7f1623400700 1 --2- 172.21.15.89:0/582240384 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f160006ac90 0x7f160006d150 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1614006b20 tx=0x7f1614006ab0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:09.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.445+0000 7f161b400700 1 -- 172.21.15.89:0/582240384 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f162409f7c0 con 0x7f1624160f10 2024-04-02T07:37:09.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.645+0000 7f162a800700 1 -- 172.21.15.89:0/582240384 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f162409f7c0 con 0x7f1624160f10 2024-04-02T07:37:09.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.646+0000 7f161b400700 1 -- 172.21.15.89:0/582240384 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f162409f7c0 con 0x7f1624160f10 2024-04-02T07:37:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:37:09.649 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":34,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:37:06.503107+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:37:03.665694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-02T07:37:04.473147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:37:09.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.649+0000 7f162a800700 1 -- 172.21.15.89:0/582240384 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f160006ac90 msgr2=0x7f160006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:09.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.649+0000 7f162a800700 1 --2- 172.21.15.89:0/582240384 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f160006ac90 0x7f160006d150 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1614006b20 tx=0x7f1614006ab0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.649+0000 7f162a800700 1 -- 172.21.15.89:0/582240384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1624160f10 msgr2=0x7f16240b82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:09.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.649+0000 7f162a800700 1 --2- 172.21.15.89:0/582240384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1624160f10 0x7f16240b82f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f160c005910 tx=0x7f160c00c860 comp rx=0 tx=0).stop 2024-04-02T07:37:09.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.649+0000 7f162a800700 1 -- 172.21.15.89:0/582240384 shutdown_connections 2024-04-02T07:37:09.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.649+0000 7f162a800700 1 --2- 172.21.15.89:0/582240384 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f160006ac90 0x7f160006d150 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.649+0000 7f162a800700 1 --2- 172.21.15.89:0/582240384 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1624161590 0x7f162415fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.649+0000 7f162a800700 1 --2- 172.21.15.89:0/582240384 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16240cf230 0x7f16241f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.649+0000 7f162a800700 1 --2- 172.21.15.89:0/582240384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1624160f10 0x7f16240b82f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.649+0000 7f162a800700 1 -- 172.21.15.89:0/582240384 >> 172.21.15.89:0/582240384 conn(0x7f162408aff0 msgr2=0x7f16240ca650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:09.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.649+0000 7f162a800700 1 -- 172.21.15.89:0/582240384 shutdown_connections 2024-04-02T07:37:09.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.649+0000 7f162a800700 1 -- 172.21.15.89:0/582240384 wait complete. 2024-04-02T07:37:09.669 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T07:37:09.669 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:37:09.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.860+0000 7f590fc00700 1 Processor -- start 2024-04-02T07:37:09.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.860+0000 7f590fc00700 1 -- start start 2024-04-02T07:37:09.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.861+0000 7f590fc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f590815d640 0x7f59080d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:09.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.861+0000 7f590fc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59080d30d0 con 0x7f59081619d0 2024-04-02T07:37:09.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.861+0000 7f590fc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59080b6840 con 0x7f59080ceec0 2024-04-02T07:37:09.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.861+0000 7f590fc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59080d2880 con 0x7f590815d640 2024-04-02T07:37:09.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.861+0000 7f590d200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f590815d640 0x7f59080d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:09.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.861+0000 7f590d200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f590815d640 0x7f59080d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58014/0 (socket says 172.21.15.89:58014) 2024-04-02T07:37:09.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.861+0000 7f590d200700 1 -- 172.21.15.89:0/1328470143 learned_addr learned my addr 172.21.15.89:0/1328470143 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:09.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.862+0000 7f590d200700 1 -- 172.21.15.89:0/1328470143 >> v1:172.21.15.113:6789/0 conn(0x7f59080ceec0 legacy=0x7f59081612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:09.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.862+0000 7f590d200700 1 -- 172.21.15.89:0/1328470143 >> v1:172.21.15.89:6789/0 conn(0x7f59081619d0 legacy=0x7f59080b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:09.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.862+0000 7f590d200700 1 -- 172.21.15.89:0/1328470143 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59080d2b30 con 0x7f590815d640 2024-04-02T07:37:09.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.862+0000 7f590d200700 1 --2- 172.21.15.89:0/1328470143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590815d640 0x7f59080d2340 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0098f0 tx=0x7f58fc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6072f090b289fa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:09.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.862+0000 7f5907400700 1 -- 172.21.15.89:0/1328470143 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc00b8a0 con 0x7f590815d640 2024-04-02T07:37:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.862+0000 7f5907400700 1 -- 172.21.15.89:0/1328470143 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58fc00ba00 con 0x7f590815d640 2024-04-02T07:37:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.862+0000 7f5907400700 1 -- 172.21.15.89:0/1328470143 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc00bd90 con 0x7f590815d640 2024-04-02T07:37:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.863+0000 7f590fc00700 1 -- 172.21.15.89:0/1328470143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590815d640 msgr2=0x7f59080d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.863+0000 7f590fc00700 1 --2- 172.21.15.89:0/1328470143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590815d640 0x7f59080d2340 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0098f0 tx=0x7f58fc009c00 comp rx=0 tx=0).stop 2024-04-02T07:37:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.863+0000 7f590fc00700 1 -- 172.21.15.89:0/1328470143 shutdown_connections 2024-04-02T07:37:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.863+0000 7f590fc00700 1 --2- 172.21.15.89:0/1328470143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590815d640 0x7f59080d2340 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.863+0000 7f590fc00700 1 -- 172.21.15.89:0/1328470143 >> 172.21.15.89:0/1328470143 conn(0x7f590808aff0 msgr2=0x7f59080d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.863+0000 7f590fc00700 1 -- 172.21.15.89:0/1328470143 shutdown_connections 2024-04-02T07:37:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.863+0000 7f590fc00700 1 -- 172.21.15.89:0/1328470143 wait complete. 2024-04-02T07:37:09.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.865+0000 7f590fc00700 1 Processor -- start 2024-04-02T07:37:09.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.865+0000 7f590fc00700 1 -- start start 2024-04-02T07:37:09.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f590fc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59080ceec0 0x7f590815ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:09.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f590fc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590815d640 0x7f59081f2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:09.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f590fc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59081619d0 0x7f590809d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:09.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f590fc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5908160470 con 0x7f590815d640 2024-04-02T07:37:09.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f590fc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59080ca130 con 0x7f59080ceec0 2024-04-02T07:37:09.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f590fc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59080c74d0 con 0x7f59081619d0 2024-04-02T07:37:09.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f5907e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590815d640 0x7f59081f2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:09.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f5907e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590815d640 0x7f59081f2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58024/0 (socket says 172.21.15.89:58024) 2024-04-02T07:37:09.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f5907e00700 1 -- 172.21.15.89:0/2445467650 learned_addr learned my addr 172.21.15.89:0/2445467650 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:09.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f590d200700 1 --2- 172.21.15.89:0/2445467650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59080ceec0 0x7f590815ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:09.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f590dc00700 1 --2- 172.21.15.89:0/2445467650 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59081619d0 0x7f590809d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:09.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f5907e00700 1 -- 172.21.15.89:0/2445467650 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59081619d0 msgr2=0x7f590809d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:09.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f5907e00700 1 --2- 172.21.15.89:0/2445467650 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59081619d0 0x7f590809d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f5907e00700 1 -- 172.21.15.89:0/2445467650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59080ceec0 msgr2=0x7f590815ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:09.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f5907e00700 1 --2- 172.21.15.89:0/2445467650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59080ceec0 0x7f590815ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.866+0000 7f5907e00700 1 -- 172.21.15.89:0/2445467650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc009550 con 0x7f590815d640 2024-04-02T07:37:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.867+0000 7f5907e00700 1 --2- 172.21.15.89:0/2445467650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590815d640 0x7f59081f2730 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f58f400c4e0 tx=0x7f58f400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.867+0000 7f5904c00700 1 -- 172.21.15.89:0/2445467650 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f40052d0 con 0x7f590815d640 2024-04-02T07:37:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.867+0000 7f5904c00700 1 -- 172.21.15.89:0/2445467650 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f4005430 con 0x7f590815d640 2024-04-02T07:37:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.867+0000 7f590fc00700 1 -- 172.21.15.89:0/2445467650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59080c77b0 con 0x7f590815d640 2024-04-02T07:37:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.867+0000 7f5904c00700 1 -- 172.21.15.89:0/2445467650 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f40038c0 con 0x7f590815d640 2024-04-02T07:37:09.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.867+0000 7f590fc00700 1 -- 172.21.15.89:0/2445467650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59080c7d70 con 0x7f590815d640 2024-04-02T07:37:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.868+0000 7f590fc00700 1 -- 172.21.15.89:0/2445467650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f590809f170 con 0x7f590815d640 2024-04-02T07:37:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.869+0000 7f5904c00700 1 -- 172.21.15.89:0/2445467650 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f58f4003a20 con 0x7f590815d640 2024-04-02T07:37:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.870+0000 7f5904c00700 1 --2- 172.21.15.89:0/2445467650 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f58e806ac90 0x7f58e806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.870+0000 7f5904c00700 1 -- 172.21.15.89:0/2445467650 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58f40886e0 con 0x7f590815d640 2024-04-02T07:37:09.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.870+0000 7f590d200700 1 --2- 172.21.15.89:0/2445467650 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f58e806ac90 0x7f58e806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:09.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.871+0000 7f590d200700 1 --2- 172.21.15.89:0/2445467650 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f58e806ac90 0x7f58e806d150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009c40 tx=0x7f58fc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:09.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:09.874+0000 7f5904c00700 1 -- 172.21.15.89:0/2445467650 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f58f4053950 con 0x7f590815d640 2024-04-02T07:37:10.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.072+0000 7f590fc00700 1 -- 172.21.15.89:0/2445467650 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59080b64d0 con 0x7f58e806ac90 2024-04-02T07:37:10.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.083+0000 7f5904c00700 1 -- 172.21.15.89:0/2445467650 <== mgr.4108 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289135 (secure 0 0 0) 0x7f58fc00bbf0 con 0x7f58e806ac90 2024-04-02T07:37:10.089 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:37:10.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.087+0000 7f590fc00700 1 -- 172.21.15.89:0/2445467650 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f58e806ac90 msgr2=0x7f58e806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:10.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.087+0000 7f590fc00700 1 --2- 172.21.15.89:0/2445467650 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f58e806ac90 0x7f58e806d150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009c40 tx=0x7f58fc008030 comp rx=0 tx=0).stop 2024-04-02T07:37:10.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.087+0000 7f590fc00700 1 -- 172.21.15.89:0/2445467650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590815d640 msgr2=0x7f59081f2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:10.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.087+0000 7f590fc00700 1 --2- 172.21.15.89:0/2445467650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590815d640 0x7f59081f2730 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f58f400c4e0 tx=0x7f58f400c7f0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.087+0000 7f590fc00700 1 -- 172.21.15.89:0/2445467650 shutdown_connections 2024-04-02T07:37:10.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.087+0000 7f590fc00700 1 --2- 172.21.15.89:0/2445467650 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f58e806ac90 0x7f58e806d150 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.087+0000 7f590fc00700 1 --2- 172.21.15.89:0/2445467650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59080ceec0 0x7f590815ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.087+0000 7f590fc00700 1 --2- 172.21.15.89:0/2445467650 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59081619d0 0x7f590809d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.087+0000 7f590fc00700 1 --2- 172.21.15.89:0/2445467650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590815d640 0x7f59081f2730 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.087+0000 7f590fc00700 1 -- 172.21.15.89:0/2445467650 >> 172.21.15.89:0/2445467650 conn(0x7f590808aff0 msgr2=0x7f59080b6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:10.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.087+0000 7f590fc00700 1 -- 172.21.15.89:0/2445467650 shutdown_connections 2024-04-02T07:37:10.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.087+0000 7f590fc00700 1 -- 172.21.15.89:0/2445467650 wait complete. 2024-04-02T07:37:10.092 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:37:10.107 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-04-02T07:37:09.197108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220312,"kb_used_data":4096,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754408,"statfs":{"total":773094113280,"available":772868513792,"internally_reserved":0,"allocated":4194304,"data_stored":2330028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.203091"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.532628+0000","last_change":"2024-04-02T07:37:05.526626+0000","last_active":"2024-04-02T07:37:05.532628+0000","last_peered":"2024-04-02T07:37:05.532628+0000","last_clean":"2024-04-02T07:37:05.532628+0000","last_became_active":"2024-04-02T07:37:05.526161+0000","last_became_peered":"2024-04-02T07:37:05.526161+0000","last_unstale":"2024-04-02T07:37:05.532628+0000","last_undegraded":"2024-04-02T07:37:05.532628+0000","last_fullsized":"2024-04-02T07:37:05.532628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:37:25.422881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.526370+0000","last_change":"2024-04-02T07:37:05.520114+0000","last_active":"2024-04-02T07:37:05.526370+0000","last_peered":"2024-04-02T07:37:05.526370+0000","last_clean":"2024-04-02T07:37:05.526370+0000","last_became_active":"2024-04-02T07:37:05.519845+0000","last_became_peered":"2024-04-02T07:37:05.519845+0000","last_unstale":"2024-04-02T07:37:05.526370+0000","last_undegraded":"2024-04-02T07:37:05.526370+0000","last_fullsized":"2024-04-02T07:37:05.526370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:16.240622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.531240+0000","last_change":"2024-04-02T07:37:05.520038+0000","last_active":"2024-04-02T07:37:05.531240+0000","last_peered":"2024-04-02T07:37:05.531240+0000","last_clean":"2024-04-02T07:37:05.531240+0000","last_became_active":"2024-04-02T07:37:05.519763+0000","last_became_peered":"2024-04-02T07:37:05.519763+0000","last_unstale":"2024-04-02T07:37:05.531240+0000","last_undegraded":"2024-04-02T07:37:05.531240+0000","last_fullsized":"2024-04-02T07:37:05.531240+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:37.522877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540866+0000","last_change":"2024-04-02T07:37:05.534692+0000","last_active":"2024-04-02T07:37:05.540866+0000","last_peered":"2024-04-02T07:37:05.540866+0000","last_clean":"2024-04-02T07:37:05.540866+0000","last_became_active":"2024-04-02T07:37:05.526775+0000","last_became_peered":"2024-04-02T07:37:05.526775+0000","last_unstale":"2024-04-02T07:37:05.540866+0000","last_undegraded":"2024-04-02T07:37:05.540866+0000","last_fullsized":"2024-04-02T07:37:05.540866+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:55:17.522766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.534152+0000","last_change":"2024-04-02T07:37:05.531729+0000","last_active":"2024-04-02T07:37:05.534152+0000","last_peered":"2024-04-02T07:37:05.534152+0000","last_clean":"2024-04-02T07:37:05.534152+0000","last_became_active":"2024-04-02T07:37:05.531111+0000","last_became_peered":"2024-04-02T07:37:05.531111+0000","last_unstale":"2024-04-02T07:37:05.534152+0000","last_undegraded":"2024-04-02T07:37:05.534152+0000","last_fullsized":"2024-04-02T07:37:05.534152+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:39.600705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.539623+0000","last_change":"2024-04-02T07:37:05.536711+0000","last_active":"2024-04-02T07:37:05.539623+0000","last_peered":"2024-04-02T07:37:05.539623+0000","last_clean":"2024-04-02T07:37:05.539623+0000","last_became_active":"2024-04-02T07:37:05.536437+0000","last_became_peered":"2024-04-02T07:37:05.536437+0000","last_unstale":"2024-04-02T07:37:05.539623+0000","last_undegraded":"2024-04-02T07:37:05.539623+0000","last_fullsized":"2024-04-02T07:37:05.539623+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:37:31.992660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.529478+0000","last_change":"2024-04-02T07:37:05.520544+0000","last_active":"2024-04-02T07:37:05.529478+0000","last_peered":"2024-04-02T07:37:05.529478+0000","last_clean":"2024-04-02T07:37:05.529478+0000","last_became_active":"2024-04-02T07:37:05.520224+0000","last_became_peered":"2024-04-02T07:37:05.520224+0000","last_unstale":"2024-04-02T07:37:05.529478+0000","last_undegraded":"2024-04-02T07:37:05.529478+0000","last_fullsized":"2024-04-02T07:37:05.529478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:34:05.497678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.522023+0000","last_change":"2024-04-02T07:37:05.517993+0000","last_active":"2024-04-02T07:37:05.522023+0000","last_peered":"2024-04-02T07:37:05.522023+0000","last_clean":"2024-04-02T07:37:05.522023+0000","last_became_active":"2024-04-02T07:37:05.517705+0000","last_became_peered":"2024-04-02T07:37:05.517705+0000","last_unstale":"2024-04-02T07:37:05.522023+0000","last_undegraded":"2024-04-02T07:37:05.522023+0000","last_fullsized":"2024-04-02T07:37:05.522023+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:16.802810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.536719+0000","last_change":"2024-04-02T07:37:05.532414+0000","last_active":"2024-04-02T07:37:05.536719+0000","last_peered":"2024-04-02T07:37:05.536719+0000","last_clean":"2024-04-02T07:37:05.536719+0000","last_became_active":"2024-04-02T07:37:05.532139+0000","last_became_peered":"2024-04-02T07:37:05.532139+0000","last_unstale":"2024-04-02T07:37:05.536719+0000","last_undegraded":"2024-04-02T07:37:05.536719+0000","last_fullsized":"2024-04-02T07:37:05.536719+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:48:35.710658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.532983+0000","last_change":"2024-04-02T07:37:05.520746+0000","last_active":"2024-04-02T07:37:05.532983+0000","last_peered":"2024-04-02T07:37:05.532983+0000","last_clean":"2024-04-02T07:37:05.532983+0000","last_became_active":"2024-04-02T07:37:05.520484+0000","last_became_peered":"2024-04-02T07:37:05.520484+0000","last_unstale":"2024-04-02T07:37:05.532983+0000","last_undegraded":"2024-04-02T07:37:05.532983+0000","last_fullsized":"2024-04-02T07:37:05.532983+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:49:47.968768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.541199+0000","last_change":"2024-04-02T07:37:05.538852+0000","last_active":"2024-04-02T07:37:05.541199+0000","last_peered":"2024-04-02T07:37:05.541199+0000","last_clean":"2024-04-02T07:37:05.541199+0000","last_became_active":"2024-04-02T07:37:05.538257+0000","last_became_peered":"2024-04-02T07:37:05.538257+0000","last_unstale":"2024-04-02T07:37:05.541199+0000","last_undegraded":"2024-04-02T07:37:05.541199+0000","last_fullsized":"2024-04-02T07:37:05.541199+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:52:03.612362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.527656+0000","last_change":"2024-04-02T07:37:05.519945+0000","last_active":"2024-04-02T07:37:05.527656+0000","last_peered":"2024-04-02T07:37:05.527656+0000","last_clean":"2024-04-02T07:37:05.527656+0000","last_became_active":"2024-04-02T07:37:05.519622+0000","last_became_peered":"2024-04-02T07:37:05.519622+0000","last_unstale":"2024-04-02T07:37:05.527656+0000","last_undegraded":"2024-04-02T07:37:05.527656+0000","last_fullsized":"2024-04-02T07:37:05.527656+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:19:37.687845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.529277+0000","last_change":"2024-04-02T07:37:05.523587+0000","last_active":"2024-04-02T07:37:05.529277+0000","last_peered":"2024-04-02T07:37:05.529277+0000","last_clean":"2024-04-02T07:37:05.529277+0000","last_became_active":"2024-04-02T07:37:05.522929+0000","last_became_peered":"2024-04-02T07:37:05.522929+0000","last_unstale":"2024-04-02T07:37:05.529277+0000","last_undegraded":"2024-04-02T07:37:05.529277+0000","last_fullsized":"2024-04-02T07:37:05.529277+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:35:46.006569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540609+0000","last_change":"2024-04-02T07:37:05.529739+0000","last_active":"2024-04-02T07:37:05.540609+0000","last_peered":"2024-04-02T07:37:05.540609+0000","last_clean":"2024-04-02T07:37:05.540609+0000","last_became_active":"2024-04-02T07:37:05.529471+0000","last_became_peered":"2024-04-02T07:37:05.529471+0000","last_unstale":"2024-04-02T07:37:05.540609+0000","last_undegraded":"2024-04-02T07:37:05.540609+0000","last_fullsized":"2024-04-02T07:37:05.540609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:58:10.466063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.538503+0000","last_change":"2024-04-02T07:37:05.530938+0000","last_active":"2024-04-02T07:37:05.538503+0000","last_peered":"2024-04-02T07:37:05.538503+0000","last_clean":"2024-04-02T07:37:05.538503+0000","last_became_active":"2024-04-02T07:37:05.530621+0000","last_became_peered":"2024-04-02T07:37:05.530621+0000","last_unstale":"2024-04-02T07:37:05.538503+0000","last_undegraded":"2024-04-02T07:37:05.538503+0000","last_fullsized":"2024-04-02T07:37:05.538503+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:28.621627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540103+0000","last_change":"2024-04-02T07:37:05.536455+0000","last_active":"2024-04-02T07:37:05.540103+0000","last_peered":"2024-04-02T07:37:05.540103+0000","last_clean":"2024-04-02T07:37:05.540103+0000","last_became_active":"2024-04-02T07:37:05.536183+0000","last_became_peered":"2024-04-02T07:37:05.536183+0000","last_unstale":"2024-04-02T07:37:05.540103+0000","last_undegraded":"2024-04-02T07:37:05.540103+0000","last_fullsized":"2024-04-02T07:37:05.540103+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:16.240622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540962+0000","last_change":"2024-04-02T07:37:05.534556+0000","last_active":"2024-04-02T07:37:05.540962+0000","last_peered":"2024-04-02T07:37:05.540962+0000","last_clean":"2024-04-02T07:37:05.540962+0000","last_became_active":"2024-04-02T07:37:05.534315+0000","last_became_peered":"2024-04-02T07:37:05.534315+0000","last_unstale":"2024-04-02T07:37:05.540962+0000","last_undegraded":"2024-04-02T07:37:05.540962+0000","last_fullsized":"2024-04-02T07:37:05.540962+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:10:22.041218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.539579+0000","last_change":"2024-04-02T07:37:05.535554+0000","last_active":"2024-04-02T07:37:05.539579+0000","last_peered":"2024-04-02T07:37:05.539579+0000","last_clean":"2024-04-02T07:37:05.539579+0000","last_became_active":"2024-04-02T07:37:05.534253+0000","last_became_peered":"2024-04-02T07:37:05.534253+0000","last_unstale":"2024-04-02T07:37:05.539579+0000","last_undegraded":"2024-04-02T07:37:05.539579+0000","last_fullsized":"2024-04-02T07:37:05.539579+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:18.250998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540276+0000","last_change":"2024-04-02T07:37:05.538345+0000","last_active":"2024-04-02T07:37:05.540276+0000","last_peered":"2024-04-02T07:37:05.540276+0000","last_clean":"2024-04-02T07:37:05.540276+0000","last_became_active":"2024-04-02T07:37:05.537983+0000","last_became_peered":"2024-04-02T07:37:05.537983+0000","last_unstale":"2024-04-02T07:37:05.540276+0000","last_undegraded":"2024-04-02T07:37:05.540276+0000","last_fullsized":"2024-04-02T07:37:05.540276+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:47:31.372894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.534512+0000","last_change":"2024-04-02T07:37:05.530867+0000","last_active":"2024-04-02T07:37:05.534512+0000","last_peered":"2024-04-02T07:37:05.534512+0000","last_clean":"2024-04-02T07:37:05.534512+0000","last_became_active":"2024-04-02T07:37:05.526527+0000","last_became_peered":"2024-04-02T07:37:05.526527+0000","last_unstale":"2024-04-02T07:37:05.534512+0000","last_undegraded":"2024-04-02T07:37:05.534512+0000","last_fullsized":"2024-04-02T07:37:05.534512+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:16.802810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.538462+0000","last_change":"2024-04-02T07:37:05.530155+0000","last_active":"2024-04-02T07:37:05.538462+0000","last_peered":"2024-04-02T07:37:05.538462+0000","last_clean":"2024-04-02T07:37:05.538462+0000","last_became_active":"2024-04-02T07:37:05.529885+0000","last_became_peered":"2024-04-02T07:37:05.529885+0000","last_unstale":"2024-04-02T07:37:05.538462+0000","last_undegraded":"2024-04-02T07:37:05.538462+0000","last_fullsized":"2024-04-02T07:37:05.538462+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:49:47.968768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.529980+0000","last_change":"2024-04-02T07:37:05.521517+0000","last_active":"2024-04-02T07:37:05.529980+0000","last_peered":"2024-04-02T07:37:05.529980+0000","last_clean":"2024-04-02T07:37:05.529980+0000","last_became_active":"2024-04-02T07:37:05.521225+0000","last_became_peered":"2024-04-02T07:37:05.521225+0000","last_unstale":"2024-04-02T07:37:05.529980+0000","last_undegraded":"2024-04-02T07:37:05.529980+0000","last_fullsized":"2024-04-02T07:37:05.529980+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:16.240622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.530720+0000","last_change":"2024-04-02T07:37:05.524984+0000","last_active":"2024-04-02T07:37:05.530720+0000","last_peered":"2024-04-02T07:37:05.530720+0000","last_clean":"2024-04-02T07:37:05.530720+0000","last_became_active":"2024-04-02T07:37:05.524594+0000","last_became_peered":"2024-04-02T07:37:05.524594+0000","last_unstale":"2024-04-02T07:37:05.530720+0000","last_undegraded":"2024-04-02T07:37:05.530720+0000","last_fullsized":"2024-04-02T07:37:05.530720+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:37:25.422881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.518751+0000","last_change":"2024-04-02T07:37:05.515902+0000","last_active":"2024-04-02T07:37:05.518751+0000","last_peered":"2024-04-02T07:37:05.518751+0000","last_clean":"2024-04-02T07:37:05.518751+0000","last_became_active":"2024-04-02T07:37:05.515567+0000","last_became_peered":"2024-04-02T07:37:05.515567+0000","last_unstale":"2024-04-02T07:37:05.518751+0000","last_undegraded":"2024-04-02T07:37:05.518751+0000","last_fullsized":"2024-04-02T07:37:05.518751+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:40.461363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540670+0000","last_change":"2024-04-02T07:37:05.531924+0000","last_active":"2024-04-02T07:37:05.540670+0000","last_peered":"2024-04-02T07:37:05.540670+0000","last_clean":"2024-04-02T07:37:05.540670+0000","last_became_active":"2024-04-02T07:37:05.531560+0000","last_became_peered":"2024-04-02T07:37:05.531560+0000","last_unstale":"2024-04-02T07:37:05.540670+0000","last_undegraded":"2024-04-02T07:37:05.540670+0000","last_fullsized":"2024-04-02T07:37:05.540670+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:40.461363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.523826+0000","last_change":"2024-04-02T07:37:05.517114+0000","last_active":"2024-04-02T07:37:05.523826+0000","last_peered":"2024-04-02T07:37:05.523826+0000","last_clean":"2024-04-02T07:37:05.523826+0000","last_became_active":"2024-04-02T07:37:05.516736+0000","last_became_peered":"2024-04-02T07:37:05.516736+0000","last_unstale":"2024-04-02T07:37:05.523826+0000","last_undegraded":"2024-04-02T07:37:05.523826+0000","last_fullsized":"2024-04-02T07:37:05.523826+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:28.621627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.533275+0000","last_change":"2024-04-02T07:37:05.530183+0000","last_active":"2024-04-02T07:37:05.533275+0000","last_peered":"2024-04-02T07:37:05.533275+0000","last_clean":"2024-04-02T07:37:05.533275+0000","last_became_active":"2024-04-02T07:37:05.527801+0000","last_became_peered":"2024-04-02T07:37:05.527801+0000","last_unstale":"2024-04-02T07:37:05.533275+0000","last_undegraded":"2024-04-02T07:37:05.533275+0000","last_fullsized":"2024-04-02T07:37:05.533275+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:16.802810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.520361+0000","last_change":"2024-04-02T07:37:05.517149+0000","last_active":"2024-04-02T07:37:05.520361+0000","last_peered":"2024-04-02T07:37:05.520361+0000","last_clean":"2024-04-02T07:37:05.520361+0000","last_became_active":"2024-04-02T07:37:05.516876+0000","last_became_peered":"2024-04-02T07:37:05.516876+0000","last_unstale":"2024-04-02T07:37:05.520361+0000","last_undegraded":"2024-04-02T07:37:05.520361+0000","last_fullsized":"2024-04-02T07:37:05.520361+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:58:10.466063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.539922+0000","last_change":"2024-04-02T07:37:05.531691+0000","last_active":"2024-04-02T07:37:05.539922+0000","last_peered":"2024-04-02T07:37:05.539922+0000","last_clean":"2024-04-02T07:37:05.539922+0000","last_became_active":"2024-04-02T07:37:05.531273+0000","last_became_peered":"2024-04-02T07:37:05.531273+0000","last_unstale":"2024-04-02T07:37:05.539922+0000","last_undegraded":"2024-04-02T07:37:05.539922+0000","last_fullsized":"2024-04-02T07:37:05.539922+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:26:44.688366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.533108+0000","last_change":"2024-04-02T07:37:05.524509+0000","last_active":"2024-04-02T07:37:05.533108+0000","last_peered":"2024-04-02T07:37:05.533108+0000","last_clean":"2024-04-02T07:37:05.533108+0000","last_became_active":"2024-04-02T07:37:05.524223+0000","last_became_peered":"2024-04-02T07:37:05.524223+0000","last_unstale":"2024-04-02T07:37:05.533108+0000","last_undegraded":"2024-04-02T07:37:05.533108+0000","last_fullsized":"2024-04-02T07:37:05.533108+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:40.461363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540648+0000","last_change":"2024-04-02T07:37:05.535742+0000","last_active":"2024-04-02T07:37:05.540648+0000","last_peered":"2024-04-02T07:37:05.540648+0000","last_clean":"2024-04-02T07:37:05.540648+0000","last_became_active":"2024-04-02T07:37:05.535473+0000","last_became_peered":"2024-04-02T07:37:05.535473+0000","last_unstale":"2024-04-02T07:37:05.540648+0000","last_undegraded":"2024-04-02T07:37:05.540648+0000","last_fullsized":"2024-04-02T07:37:05.540648+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:45:55.869487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540815+0000","last_change":"2024-04-02T07:37:05.534076+0000","last_active":"2024-04-02T07:37:05.540815+0000","last_peered":"2024-04-02T07:37:05.540815+0000","last_clean":"2024-04-02T07:37:05.540815+0000","last_became_active":"2024-04-02T07:37:05.533783+0000","last_became_peered":"2024-04-02T07:37:05.533783+0000","last_unstale":"2024-04-02T07:37:05.540815+0000","last_undegraded":"2024-04-02T07:37:05.540815+0000","last_fullsized":"2024-04-02T07:37:05.540815+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:52:30.610913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.533095+0000","last_change":"2024-04-02T07:37:05.529875+0000","last_active":"2024-04-02T07:37:05.533095+0000","last_peered":"2024-04-02T07:37:05.533095+0000","last_clean":"2024-04-02T07:37:05.533095+0000","last_became_active":"2024-04-02T07:37:05.529654+0000","last_became_peered":"2024-04-02T07:37:05.529654+0000","last_unstale":"2024-04-02T07:37:05.533095+0000","last_undegraded":"2024-04-02T07:37:05.533095+0000","last_fullsized":"2024-04-02T07:37:05.533095+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:45:55.869487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.537203+0000","last_change":"2024-04-02T07:37:05.020065+0000","last_active":"2024-04-02T07:37:05.537203+0000","last_peered":"2024-04-02T07:37:05.537203+0000","last_clean":"2024-04-02T07:37:05.537203+0000","last_became_active":"2024-04-02T07:37:05.019738+0000","last_became_peered":"2024-04-02T07:37:05.019738+0000","last_unstale":"2024-04-02T07:37:05.537203+0000","last_undegraded":"2024-04-02T07:37:05.537203+0000","last_fullsized":"2024-04-02T07:37:05.537203+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:30:41.797989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.532608+0000","last_change":"2024-04-02T07:37:05.526435+0000","last_active":"2024-04-02T07:37:05.532608+0000","last_peered":"2024-04-02T07:37:05.532608+0000","last_clean":"2024-04-02T07:37:05.532608+0000","last_became_active":"2024-04-02T07:37:05.523100+0000","last_became_peered":"2024-04-02T07:37:05.523100+0000","last_unstale":"2024-04-02T07:37:05.532608+0000","last_undegraded":"2024-04-02T07:37:05.532608+0000","last_fullsized":"2024-04-02T07:37:05.532608+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:51:55.529401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.542419+0000","last_change":"2024-04-02T07:37:05.026060+0000","last_active":"2024-04-02T07:37:05.542419+0000","last_peered":"2024-04-02T07:37:05.542419+0000","last_clean":"2024-04-02T07:37:05.542419+0000","last_became_active":"2024-04-02T07:37:05.025805+0000","last_became_peered":"2024-04-02T07:37:05.025805+0000","last_unstale":"2024-04-02T07:37:05.542419+0000","last_undegraded":"2024-04-02T07:37:05.542419+0000","last_fullsized":"2024-04-02T07:37:05.542419+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:48:30.160948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.522043+0000","last_change":"2024-04-02T07:37:05.518097+0000","last_active":"2024-04-02T07:37:05.522043+0000","last_peered":"2024-04-02T07:37:05.522043+0000","last_clean":"2024-04-02T07:37:05.522043+0000","last_became_active":"2024-04-02T07:37:05.517854+0000","last_became_peered":"2024-04-02T07:37:05.517854+0000","last_unstale":"2024-04-02T07:37:05.522043+0000","last_undegraded":"2024-04-02T07:37:05.522043+0000","last_fullsized":"2024-04-02T07:37:05.522043+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:30:42.804445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.535559+0000","last_change":"2024-04-02T07:37:05.023807+0000","last_active":"2024-04-02T07:37:05.535559+0000","last_peered":"2024-04-02T07:37:05.535559+0000","last_clean":"2024-04-02T07:37:05.535559+0000","last_became_active":"2024-04-02T07:37:05.023512+0000","last_became_peered":"2024-04-02T07:37:05.023512+0000","last_unstale":"2024-04-02T07:37:05.535559+0000","last_undegraded":"2024-04-02T07:37:05.535559+0000","last_fullsized":"2024-04-02T07:37:05.535559+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:24:24.154489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.542467+0000","last_change":"2024-04-02T07:37:05.536264+0000","last_active":"2024-04-02T07:37:05.542467+0000","last_peered":"2024-04-02T07:37:05.542467+0000","last_clean":"2024-04-02T07:37:05.542467+0000","last_became_active":"2024-04-02T07:37:05.531691+0000","last_became_peered":"2024-04-02T07:37:05.531691+0000","last_unstale":"2024-04-02T07:37:05.542467+0000","last_undegraded":"2024-04-02T07:37:05.542467+0000","last_fullsized":"2024-04-02T07:37:05.542467+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:02.936396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.542470+0000","last_change":"2024-04-02T07:37:05.028409+0000","last_active":"2024-04-02T07:37:05.542470+0000","last_peered":"2024-04-02T07:37:05.542470+0000","last_clean":"2024-04-02T07:37:05.542470+0000","last_became_active":"2024-04-02T07:37:05.028119+0000","last_became_peered":"2024-04-02T07:37:05.028119+0000","last_unstale":"2024-04-02T07:37:05.542470+0000","last_undegraded":"2024-04-02T07:37:05.542470+0000","last_fullsized":"2024-04-02T07:37:05.542470+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:48:34.704201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540412+0000","last_change":"2024-04-02T07:37:05.535227+0000","last_active":"2024-04-02T07:37:05.540412+0000","last_peered":"2024-04-02T07:37:05.540412+0000","last_clean":"2024-04-02T07:37:05.540412+0000","last_became_active":"2024-04-02T07:37:05.534909+0000","last_became_peered":"2024-04-02T07:37:05.534909+0000","last_unstale":"2024-04-02T07:37:05.540412+0000","last_undegraded":"2024-04-02T07:37:05.540412+0000","last_fullsized":"2024-04-02T07:37:05.540412+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:48:31.167405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.534316+0000","last_change":"2024-04-02T07:37:05.031842+0000","last_active":"2024-04-02T07:37:05.534316+0000","last_peered":"2024-04-02T07:37:05.534316+0000","last_clean":"2024-04-02T07:37:05.534316+0000","last_became_active":"2024-04-02T07:37:05.031576+0000","last_became_peered":"2024-04-02T07:37:05.031576+0000","last_unstale":"2024-04-02T07:37:05.534316+0000","last_undegraded":"2024-04-02T07:37:05.534316+0000","last_fullsized":"2024-04-02T07:37:05.534316+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:24:24.154489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.542691+0000","last_change":"2024-04-02T07:37:05.540654+0000","last_active":"2024-04-02T07:37:05.542691+0000","last_peered":"2024-04-02T07:37:05.542691+0000","last_clean":"2024-04-02T07:37:05.542691+0000","last_became_active":"2024-04-02T07:37:05.538027+0000","last_became_peered":"2024-04-02T07:37:05.538027+0000","last_unstale":"2024-04-02T07:37:05.542691+0000","last_undegraded":"2024-04-02T07:37:05.542691+0000","last_fullsized":"2024-04-02T07:37:05.542691+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:39.600705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.526979+0000","last_change":"2024-04-02T07:37:05.029193+0000","last_active":"2024-04-02T07:37:05.526979+0000","last_peered":"2024-04-02T07:37:05.526979+0000","last_clean":"2024-04-02T07:37:05.526979+0000","last_became_active":"2024-04-02T07:37:05.028953+0000","last_became_peered":"2024-04-02T07:37:05.028953+0000","last_unstale":"2024-04-02T07:37:05.526979+0000","last_undegraded":"2024-04-02T07:37:05.526979+0000","last_fullsized":"2024-04-02T07:37:05.526979+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:35:23.779994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.536375+0000","last_change":"2024-04-02T07:37:05.526476+0000","last_active":"2024-04-02T07:37:05.536375+0000","last_peered":"2024-04-02T07:37:05.536375+0000","last_clean":"2024-04-02T07:37:05.536375+0000","last_became_active":"2024-04-02T07:37:05.526211+0000","last_became_peered":"2024-04-02T07:37:05.526211+0000","last_unstale":"2024-04-02T07:37:05.536375+0000","last_undegraded":"2024-04-02T07:37:05.536375+0000","last_fullsized":"2024-04-02T07:37:05.536375+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:22:53.895269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.518787+0000","last_change":"2024-04-02T07:37:05.023707+0000","last_active":"2024-04-02T07:37:05.518787+0000","last_peered":"2024-04-02T07:37:05.518787+0000","last_clean":"2024-04-02T07:37:05.518787+0000","last_became_active":"2024-04-02T07:37:05.023236+0000","last_became_peered":"2024-04-02T07:37:05.023236+0000","last_unstale":"2024-04-02T07:37:05.518787+0000","last_undegraded":"2024-04-02T07:37:05.518787+0000","last_fullsized":"2024-04-02T07:37:05.518787+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:15:49.804855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.532165+0000","last_change":"2024-04-02T07:37:05.524408+0000","last_active":"2024-04-02T07:37:05.532165+0000","last_peered":"2024-04-02T07:37:05.532165+0000","last_clean":"2024-04-02T07:37:05.532165+0000","last_became_active":"2024-04-02T07:37:05.524050+0000","last_became_peered":"2024-04-02T07:37:05.524050+0000","last_unstale":"2024-04-02T07:37:05.532165+0000","last_undegraded":"2024-04-02T07:37:05.532165+0000","last_fullsized":"2024-04-02T07:37:05.532165+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:16.240622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.505983+0000","last_change":"2024-04-02T07:37:05.023237+0000","last_active":"2024-04-02T07:37:05.505983+0000","last_peered":"2024-04-02T07:37:05.505983+0000","last_clean":"2024-04-02T07:37:05.505983+0000","last_became_active":"2024-04-02T07:37:05.022779+0000","last_became_peered":"2024-04-02T07:37:05.022779+0000","last_unstale":"2024-04-02T07:37:05.505983+0000","last_undegraded":"2024-04-02T07:37:05.505983+0000","last_fullsized":"2024-04-02T07:37:05.505983+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:22:52.888812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.541036+0000","last_change":"2024-04-02T07:37:05.534980+0000","last_active":"2024-04-02T07:37:05.541036+0000","last_peered":"2024-04-02T07:37:05.541036+0000","last_clean":"2024-04-02T07:37:05.541036+0000","last_became_active":"2024-04-02T07:37:05.534725+0000","last_became_peered":"2024-04-02T07:37:05.534725+0000","last_unstale":"2024-04-02T07:37:05.541036+0000","last_undegraded":"2024-04-02T07:37:05.541036+0000","last_fullsized":"2024-04-02T07:37:05.541036+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:52:35.298086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.514241+0000","last_change":"2024-04-02T07:37:05.026942+0000","last_active":"2024-04-02T07:37:05.514241+0000","last_peered":"2024-04-02T07:37:05.514241+0000","last_clean":"2024-04-02T07:37:05.514241+0000","last_became_active":"2024-04-02T07:37:05.026700+0000","last_became_peered":"2024-04-02T07:37:05.026700+0000","last_unstale":"2024-04-02T07:37:05.514241+0000","last_undegraded":"2024-04-02T07:37:05.514241+0000","last_fullsized":"2024-04-02T07:37:05.514241+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:14.637885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.539588+0000","last_change":"2024-04-02T07:37:05.521902+0000","last_active":"2024-04-02T07:37:05.539588+0000","last_peered":"2024-04-02T07:37:05.539588+0000","last_clean":"2024-04-02T07:37:05.539588+0000","last_became_active":"2024-04-02T07:37:05.521581+0000","last_became_peered":"2024-04-02T07:37:05.521581+0000","last_unstale":"2024-04-02T07:37:05.539588+0000","last_undegraded":"2024-04-02T07:37:05.539588+0000","last_fullsized":"2024-04-02T07:37:05.539588+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:30:42.804445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.513027+0000","last_change":"2024-04-02T07:37:05.025628+0000","last_active":"2024-04-02T07:37:05.513027+0000","last_peered":"2024-04-02T07:37:05.513027+0000","last_clean":"2024-04-02T07:37:05.513027+0000","last_became_active":"2024-04-02T07:37:05.019281+0000","last_became_peered":"2024-04-02T07:37:05.019281+0000","last_unstale":"2024-04-02T07:37:05.513027+0000","last_undegraded":"2024-04-02T07:37:05.513027+0000","last_fullsized":"2024-04-02T07:37:05.513027+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:37:30.986204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540845+0000","last_change":"2024-04-02T07:37:05.534430+0000","last_active":"2024-04-02T07:37:05.540845+0000","last_peered":"2024-04-02T07:37:05.540845+0000","last_clean":"2024-04-02T07:37:05.540845+0000","last_became_active":"2024-04-02T07:37:05.530003+0000","last_became_peered":"2024-04-02T07:37:05.530003+0000","last_unstale":"2024-04-02T07:37:05.540845+0000","last_undegraded":"2024-04-02T07:37:05.540845+0000","last_fullsized":"2024-04-02T07:37:05.540845+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:37:08.135605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.539124+0000","last_change":"2024-04-02T07:37:05.015931+0000","last_active":"2024-04-02T07:37:05.539124+0000","last_peered":"2024-04-02T07:37:05.539124+0000","last_clean":"2024-04-02T07:37:05.539124+0000","last_became_active":"2024-04-02T07:37:05.015586+0000","last_became_peered":"2024-04-02T07:37:05.015586+0000","last_unstale":"2024-04-02T07:37:05.539124+0000","last_undegraded":"2024-04-02T07:37:05.539124+0000","last_fullsized":"2024-04-02T07:37:05.539124+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:51:54.522944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.541215+0000","last_change":"2024-04-02T07:37:05.538980+0000","last_active":"2024-04-02T07:37:05.541215+0000","last_peered":"2024-04-02T07:37:05.541215+0000","last_clean":"2024-04-02T07:37:05.541215+0000","last_became_active":"2024-04-02T07:37:05.538575+0000","last_became_peered":"2024-04-02T07:37:05.538575+0000","last_unstale":"2024-04-02T07:37:05.541215+0000","last_undegraded":"2024-04-02T07:37:05.541215+0000","last_fullsized":"2024-04-02T07:37:05.541215+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:55:09.846931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.547957+0000","last_change":"2024-04-02T07:37:05.026802+0000","last_active":"2024-04-02T07:37:05.547957+0000","last_peered":"2024-04-02T07:37:05.547957+0000","last_clean":"2024-04-02T07:37:05.547957+0000","last_became_active":"2024-04-02T07:37:05.025682+0000","last_became_peered":"2024-04-02T07:37:05.025682+0000","last_unstale":"2024-04-02T07:37:05.547957+0000","last_undegraded":"2024-04-02T07:37:05.547957+0000","last_fullsized":"2024-04-02T07:37:05.547957+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:48:30.160948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.528453+0000","last_change":"2024-04-02T07:37:05.523689+0000","last_active":"2024-04-02T07:37:05.528453+0000","last_peered":"2024-04-02T07:37:05.528453+0000","last_clean":"2024-04-02T07:37:05.528453+0000","last_became_active":"2024-04-02T07:37:05.523191+0000","last_became_peered":"2024-04-02T07:37:05.523191+0000","last_unstale":"2024-04-02T07:37:05.528453+0000","last_undegraded":"2024-04-02T07:37:05.528453+0000","last_fullsized":"2024-04-02T07:37:05.528453+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:47:31.372894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.532177+0000","last_change":"2024-04-02T07:37:05.026612+0000","last_active":"2024-04-02T07:37:05.532177+0000","last_peered":"2024-04-02T07:37:05.532177+0000","last_clean":"2024-04-02T07:37:05.532177+0000","last_became_active":"2024-04-02T07:37:05.026394+0000","last_became_peered":"2024-04-02T07:37:05.026394+0000","last_unstale":"2024-04-02T07:37:05.532177+0000","last_undegraded":"2024-04-02T07:37:05.532177+0000","last_fullsized":"2024-04-02T07:37:05.532177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:11:53.146618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.545647+0000","last_change":"2024-04-02T07:37:05.539883+0000","last_active":"2024-04-02T07:37:05.545647+0000","last_peered":"2024-04-02T07:37:05.545647+0000","last_clean":"2024-04-02T07:37:05.545647+0000","last_became_active":"2024-04-02T07:37:05.534733+0000","last_became_peered":"2024-04-02T07:37:05.534733+0000","last_unstale":"2024-04-02T07:37:05.545647+0000","last_undegraded":"2024-04-02T07:37:05.545647+0000","last_fullsized":"2024-04-02T07:37:05.545647+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:32:12.957088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.510286+0000","last_change":"2024-04-02T07:37:05.024394+0000","last_active":"2024-04-02T07:37:05.510286+0000","last_peered":"2024-04-02T07:37:05.510286+0000","last_clean":"2024-04-02T07:37:05.510286+0000","last_became_active":"2024-04-02T07:37:05.024075+0000","last_became_peered":"2024-04-02T07:37:05.024075+0000","last_unstale":"2024-04-02T07:37:05.510286+0000","last_undegraded":"2024-04-02T07:37:05.510286+0000","last_fullsized":"2024-04-02T07:37:05.510286+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:19:36.681389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.518732+0000","last_change":"2024-04-02T07:37:05.514468+0000","last_active":"2024-04-02T07:37:05.518732+0000","last_peered":"2024-04-02T07:37:05.518732+0000","last_clean":"2024-04-02T07:37:05.518732+0000","last_became_active":"2024-04-02T07:37:05.514220+0000","last_became_peered":"2024-04-02T07:37:05.514220+0000","last_unstale":"2024-04-02T07:37:05.518732+0000","last_undegraded":"2024-04-02T07:37:05.518732+0000","last_fullsized":"2024-04-02T07:37:05.518732+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:24:07.674982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.509996+0000","last_change":"2024-04-02T07:37:05.023645+0000","last_active":"2024-04-02T07:37:05.509996+0000","last_peered":"2024-04-02T07:37:05.509996+0000","last_clean":"2024-04-02T07:37:05.509996+0000","last_became_active":"2024-04-02T07:37:05.023325+0000","last_became_peered":"2024-04-02T07:37:05.023325+0000","last_unstale":"2024-04-02T07:37:05.509996+0000","last_undegraded":"2024-04-02T07:37:05.509996+0000","last_fullsized":"2024-04-02T07:37:05.509996+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:01:29.021837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.539792+0000","last_change":"2024-04-02T07:37:05.532872+0000","last_active":"2024-04-02T07:37:05.539792+0000","last_peered":"2024-04-02T07:37:05.539792+0000","last_clean":"2024-04-02T07:37:05.539792+0000","last_became_active":"2024-04-02T07:37:05.531426+0000","last_became_peered":"2024-04-02T07:37:05.531426+0000","last_unstale":"2024-04-02T07:37:05.539792+0000","last_undegraded":"2024-04-02T07:37:05.539792+0000","last_fullsized":"2024-04-02T07:37:05.539792+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:34:07.649125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.538844+0000","last_change":"2024-04-02T07:37:05.021987+0000","last_active":"2024-04-02T07:37:05.538844+0000","last_peered":"2024-04-02T07:37:05.538844+0000","last_clean":"2024-04-02T07:37:05.538844+0000","last_became_active":"2024-04-02T07:37:05.021723+0000","last_became_peered":"2024-04-02T07:37:05.021723+0000","last_unstale":"2024-04-02T07:37:05.538844+0000","last_undegraded":"2024-04-02T07:37:05.538844+0000","last_fullsized":"2024-04-02T07:37:05.538844+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:11.495078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.541252+0000","last_change":"2024-04-02T07:37:05.539477+0000","last_active":"2024-04-02T07:37:05.541252+0000","last_peered":"2024-04-02T07:37:05.541252+0000","last_clean":"2024-04-02T07:37:05.541252+0000","last_became_active":"2024-04-02T07:37:05.537930+0000","last_became_peered":"2024-04-02T07:37:05.537930+0000","last_unstale":"2024-04-02T07:37:05.541252+0000","last_undegraded":"2024-04-02T07:37:05.541252+0000","last_fullsized":"2024-04-02T07:37:05.541252+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:44:34.027889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.541231+0000","last_change":"2024-04-02T07:37:05.026559+0000","last_active":"2024-04-02T07:37:05.541231+0000","last_peered":"2024-04-02T07:37:05.541231+0000","last_clean":"2024-04-02T07:37:05.541231+0000","last_became_active":"2024-04-02T07:37:05.026297+0000","last_became_peered":"2024-04-02T07:37:05.026297+0000","last_unstale":"2024-04-02T07:37:05.541231+0000","last_undegraded":"2024-04-02T07:37:05.541231+0000","last_fullsized":"2024-04-02T07:37:05.541231+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:40:22.235662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.531040+0000","last_change":"2024-04-02T07:37:05.521401+0000","last_active":"2024-04-02T07:37:05.531040+0000","last_peered":"2024-04-02T07:37:05.531040+0000","last_clean":"2024-04-02T07:37:05.531040+0000","last_became_active":"2024-04-02T07:37:05.521155+0000","last_became_peered":"2024-04-02T07:37:05.521155+0000","last_unstale":"2024-04-02T07:37:05.531040+0000","last_undegraded":"2024-04-02T07:37:05.531040+0000","last_fullsized":"2024-04-02T07:37:05.531040+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:01:30.028293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.528745+0000","last_change":"2024-04-02T07:37:05.023509+0000","last_active":"2024-04-02T07:37:05.528745+0000","last_peered":"2024-04-02T07:37:05.528745+0000","last_clean":"2024-04-02T07:37:05.528745+0000","last_became_active":"2024-04-02T07:37:05.021664+0000","last_became_peered":"2024-04-02T07:37:05.021664+0000","last_unstale":"2024-04-02T07:37:05.528745+0000","last_undegraded":"2024-04-02T07:37:05.528745+0000","last_fullsized":"2024-04-02T07:37:05.528745+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:11.495078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.534839+0000","last_change":"2024-04-02T07:37:05.527849+0000","last_active":"2024-04-02T07:37:05.534839+0000","last_peered":"2024-04-02T07:37:05.534839+0000","last_clean":"2024-04-02T07:37:05.534839+0000","last_became_active":"2024-04-02T07:37:05.527496+0000","last_became_peered":"2024-04-02T07:37:05.527496+0000","last_unstale":"2024-04-02T07:37:05.534839+0000","last_undegraded":"2024-04-02T07:37:05.534839+0000","last_fullsized":"2024-04-02T07:37:05.534839+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:58:10.466063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.537993+0000","last_change":"2024-04-02T07:37:05.027202+0000","last_active":"2024-04-02T07:37:05.537993+0000","last_peered":"2024-04-02T07:37:05.537993+0000","last_clean":"2024-04-02T07:37:05.537993+0000","last_became_active":"2024-04-02T07:37:05.026903+0000","last_became_peered":"2024-04-02T07:37:05.026903+0000","last_unstale":"2024-04-02T07:37:05.537993+0000","last_undegraded":"2024-04-02T07:37:05.537993+0000","last_fullsized":"2024-04-02T07:37:05.537993+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:39.454907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.538199+0000","last_change":"2024-04-02T07:37:05.532573+0000","last_active":"2024-04-02T07:37:05.538199+0000","last_peered":"2024-04-02T07:37:05.538199+0000","last_clean":"2024-04-02T07:37:05.538199+0000","last_became_active":"2024-04-02T07:37:05.532298+0000","last_became_peered":"2024-04-02T07:37:05.532298+0000","last_unstale":"2024-04-02T07:37:05.538199+0000","last_undegraded":"2024-04-02T07:37:05.538199+0000","last_fullsized":"2024-04-02T07:37:05.538199+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:52:17.188396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.514949+0000","last_change":"2024-04-02T07:37:05.022359+0000","last_active":"2024-04-02T07:37:05.514949+0000","last_peered":"2024-04-02T07:37:05.514949+0000","last_clean":"2024-04-02T07:37:05.514949+0000","last_became_active":"2024-04-02T07:37:05.022107+0000","last_became_peered":"2024-04-02T07:37:05.022107+0000","last_unstale":"2024-04-02T07:37:05.514949+0000","last_undegraded":"2024-04-02T07:37:05.514949+0000","last_fullsized":"2024-04-02T07:37:05.514949+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:37:30.986204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.532878+0000","last_change":"2024-04-02T07:37:05.528586+0000","last_active":"2024-04-02T07:37:05.532878+0000","last_peered":"2024-04-02T07:37:05.532878+0000","last_clean":"2024-04-02T07:37:05.532878+0000","last_became_active":"2024-04-02T07:37:05.528281+0000","last_became_peered":"2024-04-02T07:37:05.528281+0000","last_unstale":"2024-04-02T07:37:05.532878+0000","last_undegraded":"2024-04-02T07:37:05.532878+0000","last_fullsized":"2024-04-02T07:37:05.532878+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:24:07.674982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.533309+0000","last_change":"2024-04-02T07:37:05.027533+0000","last_active":"2024-04-02T07:37:05.533309+0000","last_peered":"2024-04-02T07:37:05.533309+0000","last_clean":"2024-04-02T07:37:05.533309+0000","last_became_active":"2024-04-02T07:37:05.027198+0000","last_became_peered":"2024-04-02T07:37:05.027198+0000","last_unstale":"2024-04-02T07:37:05.533309+0000","last_undegraded":"2024-04-02T07:37:05.533309+0000","last_fullsized":"2024-04-02T07:37:05.533309+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:39:03.477899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.533446+0000","last_change":"2024-04-02T07:37:05.527861+0000","last_active":"2024-04-02T07:37:05.533446+0000","last_peered":"2024-04-02T07:37:05.533446+0000","last_clean":"2024-04-02T07:37:05.533446+0000","last_became_active":"2024-04-02T07:37:05.527546+0000","last_became_peered":"2024-04-02T07:37:05.527546+0000","last_unstale":"2024-04-02T07:37:05.533446+0000","last_undegraded":"2024-04-02T07:37:05.533446+0000","last_fullsized":"2024-04-02T07:37:05.533446+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:24:07.674982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.506853+0000","last_change":"2024-04-02T07:37:05.025925+0000","last_active":"2024-04-02T07:37:05.506853+0000","last_peered":"2024-04-02T07:37:05.506853+0000","last_clean":"2024-04-02T07:37:05.506853+0000","last_became_active":"2024-04-02T07:37:05.025567+0000","last_became_peered":"2024-04-02T07:37:05.025567+0000","last_unstale":"2024-04-02T07:37:05.506853+0000","last_undegraded":"2024-04-02T07:37:05.506853+0000","last_fullsized":"2024-04-02T07:37:05.506853+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:40:22.235662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.528019+0000","last_change":"2024-04-02T07:37:05.523275+0000","last_active":"2024-04-02T07:37:05.528019+0000","last_peered":"2024-04-02T07:37:05.528019+0000","last_clean":"2024-04-02T07:37:05.528019+0000","last_became_active":"2024-04-02T07:37:05.522194+0000","last_became_peered":"2024-04-02T07:37:05.522194+0000","last_unstale":"2024-04-02T07:37:05.528019+0000","last_undegraded":"2024-04-02T07:37:05.528019+0000","last_fullsized":"2024-04-02T07:37:05.528019+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:18.250998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.508932+0000","last_change":"2024-04-02T07:37:05.023186+0000","last_active":"2024-04-02T07:37:05.508932+0000","last_peered":"2024-04-02T07:37:05.508932+0000","last_clean":"2024-04-02T07:37:05.508932+0000","last_became_active":"2024-04-02T07:37:05.022937+0000","last_became_peered":"2024-04-02T07:37:05.022937+0000","last_unstale":"2024-04-02T07:37:05.508932+0000","last_undegraded":"2024-04-02T07:37:05.508932+0000","last_fullsized":"2024-04-02T07:37:05.508932+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:12:03.425485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.531139+0000","last_change":"2024-04-02T07:37:05.522102+0000","last_active":"2024-04-02T07:37:05.531139+0000","last_peered":"2024-04-02T07:37:05.531139+0000","last_clean":"2024-04-02T07:37:05.531139+0000","last_became_active":"2024-04-02T07:37:05.521891+0000","last_became_peered":"2024-04-02T07:37:05.521891+0000","last_unstale":"2024-04-02T07:37:05.531139+0000","last_undegraded":"2024-04-02T07:37:05.531139+0000","last_fullsized":"2024-04-02T07:37:05.531139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:24:25.160946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.516317+0000","last_change":"2024-04-02T07:37:05.027133+0000","last_active":"2024-04-02T07:37:05.516317+0000","last_peered":"2024-04-02T07:37:05.516317+0000","last_clean":"2024-04-02T07:37:05.516317+0000","last_became_active":"2024-04-02T07:37:05.026860+0000","last_became_peered":"2024-04-02T07:37:05.026860+0000","last_unstale":"2024-04-02T07:37:05.516317+0000","last_undegraded":"2024-04-02T07:37:05.516317+0000","last_fullsized":"2024-04-02T07:37:05.516317+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:15.234166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.539601+0000","last_change":"2024-04-02T07:37:05.536066+0000","last_active":"2024-04-02T07:37:05.539601+0000","last_peered":"2024-04-02T07:37:05.539601+0000","last_clean":"2024-04-02T07:37:05.539601+0000","last_became_active":"2024-04-02T07:37:05.534329+0000","last_became_peered":"2024-04-02T07:37:05.534329+0000","last_unstale":"2024-04-02T07:37:05.539601+0000","last_undegraded":"2024-04-02T07:37:05.539601+0000","last_fullsized":"2024-04-02T07:37:05.539601+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:39:04.484355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.530573+0000","last_change":"2024-04-02T07:37:05.027585+0000","last_active":"2024-04-02T07:37:05.530573+0000","last_peered":"2024-04-02T07:37:05.530573+0000","last_clean":"2024-04-02T07:37:05.530573+0000","last_became_active":"2024-04-02T07:37:05.025889+0000","last_became_peered":"2024-04-02T07:37:05.025889+0000","last_unstale":"2024-04-02T07:37:05.530573+0000","last_undegraded":"2024-04-02T07:37:05.530573+0000","last_fullsized":"2024-04-02T07:37:05.530573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:36.516421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.541206+0000","last_change":"2024-04-02T07:37:05.539090+0000","last_active":"2024-04-02T07:37:05.541206+0000","last_peered":"2024-04-02T07:37:05.541206+0000","last_clean":"2024-04-02T07:37:05.541206+0000","last_became_active":"2024-04-02T07:37:05.537072+0000","last_became_peered":"2024-04-02T07:37:05.537072+0000","last_unstale":"2024-04-02T07:37:05.541206+0000","last_undegraded":"2024-04-02T07:37:05.541206+0000","last_fullsized":"2024-04-02T07:37:05.541206+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:59:52.962129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.513130+0000","last_change":"2024-04-02T07:37:05.022270+0000","last_active":"2024-04-02T07:37:05.513130+0000","last_peered":"2024-04-02T07:37:05.513130+0000","last_clean":"2024-04-02T07:37:05.513130+0000","last_became_active":"2024-04-02T07:37:05.018873+0000","last_became_peered":"2024-04-02T07:37:05.018873+0000","last_unstale":"2024-04-02T07:37:05.513130+0000","last_undegraded":"2024-04-02T07:37:05.513130+0000","last_fullsized":"2024-04-02T07:37:05.513130+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:52:16.181939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540254+0000","last_change":"2024-04-02T07:37:05.538119+0000","last_active":"2024-04-02T07:37:05.540254+0000","last_peered":"2024-04-02T07:37:05.540254+0000","last_clean":"2024-04-02T07:37:05.540254+0000","last_became_active":"2024-04-02T07:37:05.537340+0000","last_became_peered":"2024-04-02T07:37:05.537340+0000","last_unstale":"2024-04-02T07:37:05.540254+0000","last_undegraded":"2024-04-02T07:37:05.540254+0000","last_fullsized":"2024-04-02T07:37:05.540254+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:15.644342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.519223+0000","last_change":"2024-04-02T07:37:05.029711+0000","last_active":"2024-04-02T07:37:05.519223+0000","last_peered":"2024-04-02T07:37:05.519223+0000","last_clean":"2024-04-02T07:37:05.519223+0000","last_became_active":"2024-04-02T07:37:05.029366+0000","last_became_peered":"2024-04-02T07:37:05.029366+0000","last_unstale":"2024-04-02T07:37:05.519223+0000","last_undegraded":"2024-04-02T07:37:05.519223+0000","last_fullsized":"2024-04-02T07:37:05.519223+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:40:22.235662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540471+0000","last_change":"2024-04-02T07:37:05.537247+0000","last_active":"2024-04-02T07:37:05.540471+0000","last_peered":"2024-04-02T07:37:05.540471+0000","last_clean":"2024-04-02T07:37:05.540471+0000","last_became_active":"2024-04-02T07:37:05.537015+0000","last_became_peered":"2024-04-02T07:37:05.537015+0000","last_unstale":"2024-04-02T07:37:05.540471+0000","last_undegraded":"2024-04-02T07:37:05.540471+0000","last_fullsized":"2024-04-02T07:37:05.540471+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:51:55.529401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.518389+0000","last_change":"2024-04-02T07:37:05.024933+0000","last_active":"2024-04-02T07:37:05.518389+0000","last_peered":"2024-04-02T07:37:05.518389+0000","last_clean":"2024-04-02T07:37:05.518389+0000","last_became_active":"2024-04-02T07:37:05.024623+0000","last_became_peered":"2024-04-02T07:37:05.024623+0000","last_unstale":"2024-04-02T07:37:05.518389+0000","last_undegraded":"2024-04-02T07:37:05.518389+0000","last_fullsized":"2024-04-02T07:37:05.518389+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:35:23.779994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540866+0000","last_change":"2024-04-02T07:37:05.534896+0000","last_active":"2024-04-02T07:37:05.540866+0000","last_peered":"2024-04-02T07:37:05.540866+0000","last_clean":"2024-04-02T07:37:05.540866+0000","last_became_active":"2024-04-02T07:37:05.531154+0000","last_became_peered":"2024-04-02T07:37:05.531154+0000","last_unstale":"2024-04-02T07:37:05.540866+0000","last_undegraded":"2024-04-02T07:37:05.540866+0000","last_fullsized":"2024-04-02T07:37:05.540866+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:51:55.529401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.518067+0000","last_change":"2024-04-02T07:37:05.025479+0000","last_active":"2024-04-02T07:37:05.518067+0000","last_peered":"2024-04-02T07:37:05.518067+0000","last_clean":"2024-04-02T07:37:05.518067+0000","last_became_active":"2024-04-02T07:37:05.025204+0000","last_became_peered":"2024-04-02T07:37:05.025204+0000","last_unstale":"2024-04-02T07:37:05.518067+0000","last_undegraded":"2024-04-02T07:37:05.518067+0000","last_fullsized":"2024-04-02T07:37:05.518067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:13:59.421974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.536183+0000","last_change":"2024-04-02T07:37:05.524694+0000","last_active":"2024-04-02T07:37:05.536183+0000","last_peered":"2024-04-02T07:37:05.536183+0000","last_clean":"2024-04-02T07:37:05.536183+0000","last_became_active":"2024-04-02T07:37:05.524428+0000","last_became_peered":"2024-04-02T07:37:05.524428+0000","last_unstale":"2024-04-02T07:37:05.536183+0000","last_undegraded":"2024-04-02T07:37:05.536183+0000","last_fullsized":"2024-04-02T07:37:05.536183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:26:44.688366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.506443+0000","last_change":"2024-04-02T07:37:05.024567+0000","last_active":"2024-04-02T07:37:05.506443+0000","last_peered":"2024-04-02T07:37:05.506443+0000","last_clean":"2024-04-02T07:37:05.506443+0000","last_became_active":"2024-04-02T07:37:05.024250+0000","last_became_peered":"2024-04-02T07:37:05.024250+0000","last_unstale":"2024-04-02T07:37:05.506443+0000","last_undegraded":"2024-04-02T07:37:05.506443+0000","last_fullsized":"2024-04-02T07:37:05.506443+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:13:59.421974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.541098+0000","last_change":"2024-04-02T07:37:05.536764+0000","last_active":"2024-04-02T07:37:05.541098+0000","last_peered":"2024-04-02T07:37:05.541098+0000","last_clean":"2024-04-02T07:37:05.541098+0000","last_became_active":"2024-04-02T07:37:05.536505+0000","last_became_peered":"2024-04-02T07:37:05.536505+0000","last_unstale":"2024-04-02T07:37:05.541098+0000","last_undegraded":"2024-04-02T07:37:05.541098+0000","last_fullsized":"2024-04-02T07:37:05.541098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:43:39.130383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.509905+0000","last_change":"2024-04-02T07:37:05.025098+0000","last_active":"2024-04-02T07:37:05.509905+0000","last_peered":"2024-04-02T07:37:05.509905+0000","last_clean":"2024-04-02T07:37:05.509905+0000","last_became_active":"2024-04-02T07:37:05.024084+0000","last_became_peered":"2024-04-02T07:37:05.024084+0000","last_unstale":"2024-04-02T07:37:05.509905+0000","last_undegraded":"2024-04-02T07:37:05.509905+0000","last_fullsized":"2024-04-02T07:37:05.509905+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:36.516421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540044+0000","last_change":"2024-04-02T07:37:05.535894+0000","last_active":"2024-04-02T07:37:05.540044+0000","last_peered":"2024-04-02T07:37:05.540044+0000","last_clean":"2024-04-02T07:37:05.540044+0000","last_became_active":"2024-04-02T07:37:05.535319+0000","last_became_peered":"2024-04-02T07:37:05.535319+0000","last_unstale":"2024-04-02T07:37:05.540044+0000","last_undegraded":"2024-04-02T07:37:05.540044+0000","last_fullsized":"2024-04-02T07:37:05.540044+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:30:42.804445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.519289+0000","last_change":"2024-04-02T07:37:05.026106+0000","last_active":"2024-04-02T07:37:05.519289+0000","last_peered":"2024-04-02T07:37:05.519289+0000","last_clean":"2024-04-02T07:37:05.519289+0000","last_became_active":"2024-04-02T07:37:05.025886+0000","last_became_peered":"2024-04-02T07:37:05.025886+0000","last_unstale":"2024-04-02T07:37:05.519289+0000","last_undegraded":"2024-04-02T07:37:05.519289+0000","last_fullsized":"2024-04-02T07:37:05.519289+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:58:09.459607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":175,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.021994+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:37:05.021994+0000","last_peered":"2024-04-02T07:37:05.021994+0000","last_clean":"2024-04-02T07:37:05.021994+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:37:05.021994+0000","last_undegraded":"2024-04-02T07:37:05.021994+0000","last_fullsized":"2024-04-02T07:37:05.021994+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":376832,"data_stored":145123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.65500000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.63100000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.61899999999999999}]},{"osd":3,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.58599999999999997}]},{"osd":4,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.66800000000000004}]},{"osd":5,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.59899999999999998}]},{"osd":6,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.51200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":385024,"data_stored":146665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.40400000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.74199999999999999}]},{"osd":2,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.50800000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.53000000000000003}]},{"osd":4,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.57099999999999995}]},{"osd":5,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.82499999999999996}]},{"osd":7,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.63500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":954368,"data_stored":728694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.9079999999999999,"5min":2.9079999999999999,"15min":2.9079999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":2.766,"5min":2.766,"15min":2.766},"last":0.435}]},{"osd":2,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":2.0750000000000002,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":0.45800000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.9430000000000001,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":0.503},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":2.802,"5min":2.802,"15min":2.802},"last":0.36199999999999999}]},{"osd":4,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.65400000000000003}]},{"osd":5,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.63900000000000001}]},{"osd":6,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.012,"5min":2.012,"15min":2.012},"last":0.59399999999999997}]},{"osd":7,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":0.746},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.629}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":372736,"data_stored":144172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.39000000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":2.605,"5min":2.605,"15min":2.605},"last":0.35099999999999998}]},{"osd":3,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":5.4800000000000004,"5min":5.4800000000000004,"15min":5.4800000000000004},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":2.194,"5min":2.194,"15min":2.194},"last":0.61599999999999999}]},{"osd":4,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":6.0899999999999999,"5min":6.0899999999999999,"15min":6.0899999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":5.7169999999999996,"5min":5.7169999999999996,"15min":5.7169999999999996},"last":0.60599999999999998}]},{"osd":5,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":5.7539999999999996,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":5.8929999999999998,"5min":5.8929999999999998,"15min":5.8929999999999998},"last":0.58699999999999997}]},{"osd":6,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":5.931,"5min":5.931,"15min":5.931},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":6.125,"5min":6.125,"15min":6.125},"last":0.68999999999999995}]},{"osd":7,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":6.1580000000000004,"5min":6.1580000000000004,"15min":6.1580000000000004},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":5.7850000000000001,"5min":5.7850000000000001,"15min":5.7850000000000001},"last":0.747}]}]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":376832,"data_stored":145135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.379},{"interface":"front","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.36499999999999999}]},{"osd":1,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.41399999999999998}]},{"osd":3,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.51400000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.94199999999999995}]},{"osd":5,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.91800000000000004}]},{"osd":6,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.96599999999999997}]},{"osd":7,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.81100000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":145877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.476,"5min":2.476,"15min":2.476},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":0.38100000000000001}]},{"osd":1,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":2.2839999999999998,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.2240000000000002,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.49199999999999999}]},{"osd":2,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.9449999999999998,"5min":3.9449999999999998,"15min":3.9449999999999998},"last":0.44}]},{"osd":4,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.68899999999999995}]},{"osd":5,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":0.66600000000000004}]},{"osd":6,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.79900000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.966,"5min":1.966,"15min":1.966},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.71499999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":146687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.77200000000000002}]},{"osd":1,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.63},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.75700000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.71299999999999997}]},{"osd":3,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.59799999999999998}]},{"osd":5,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.65600000000000003}]},{"osd":6,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.80400000000000005}]},{"osd":7,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.74199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":950272,"data_stored":727675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.623}]},{"osd":1,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.64300000000000002}]},{"osd":2,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.73699999999999999}]},{"osd":3,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.89900000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.85199999999999998}]},{"osd":6,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.92400000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.98299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:37:10.116 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:37:10.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.337+0000 7fa2fca00700 1 Processor -- start 2024-04-02T07:37:10.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.337+0000 7fa2fca00700 1 -- start start 2024-04-02T07:37:10.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.338+0000 7fa2fca00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa2f809a910 0x7fa2f806c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:10.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.338+0000 7fa2fca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2f8099a60 0x7fa2f8099e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:10.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.338+0000 7fa2fca00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f80d08d0 con 0x7fa2f815d920 2024-04-02T07:37:10.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.338+0000 7fa2fca00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f806c300 con 0x7fa2f809a910 2024-04-02T07:37:10.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.338+0000 7fa2fca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f809a3d0 con 0x7fa2f8099a60 2024-04-02T07:37:10.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.339+0000 7fa2f5400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa2f809a910 0x7fa2f806c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:10.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.339+0000 7fa2f5400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa2f809a910 0x7fa2f806c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58038/0 (socket says 172.21.15.89:58038) 2024-04-02T07:37:10.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.339+0000 7fa2f5400700 1 -- 172.21.15.89:0/1563665824 learned_addr learned my addr 172.21.15.89:0/1563665824 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:10.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.339+0000 7fa2f5400700 1 -- 172.21.15.89:0/1563665824 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2f8099a60 msgr2=0x7fa2f8099e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:10.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.339+0000 7fa2f5400700 1 --2- 172.21.15.89:0/1563665824 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2f8099a60 0x7fa2f8099e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.339+0000 7fa2f5400700 1 -- 172.21.15.89:0/1563665824 >> v1:172.21.15.113:6789/0 conn(0x7fa2f815d920 legacy=0x7fa2f80cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:10.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.339+0000 7fa2f5400700 1 -- 172.21.15.89:0/1563665824 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f806b9d0 con 0x7fa2f809a910 2024-04-02T07:37:10.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.339+0000 7fa2f5400700 1 --2- 172.21.15.89:0/1563665824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f809a910 0x7fa2f806c790 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40098f0 tx=0x7fa2e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b095b4e3a85126 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:10.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.339+0000 7fa2f4a00700 1 -- 172.21.15.89:0/1563665824 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4004230 con 0x7fa2f809a910 2024-04-02T07:37:10.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.339+0000 7fa2f4a00700 1 -- 172.21.15.89:0/1563665824 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e4004390 con 0x7fa2f809a910 2024-04-02T07:37:10.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.339+0000 7fa2f4a00700 1 -- 172.21.15.89:0/1563665824 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e40036a0 con 0x7fa2f809a910 2024-04-02T07:37:10.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.340+0000 7fa2fca00700 1 -- 172.21.15.89:0/1563665824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f809a910 msgr2=0x7fa2f806c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:10.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.340+0000 7fa2fca00700 1 --2- 172.21.15.89:0/1563665824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f809a910 0x7fa2f806c790 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40098f0 tx=0x7fa2e4009c00 comp rx=0 tx=0).stop 2024-04-02T07:37:10.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.340+0000 7fa2fca00700 1 -- 172.21.15.89:0/1563665824 shutdown_connections 2024-04-02T07:37:10.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.340+0000 7fa2fca00700 1 --2- 172.21.15.89:0/1563665824 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2f8099a60 0x7fa2f8099e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.340+0000 7fa2fca00700 1 --2- 172.21.15.89:0/1563665824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f809a910 0x7fa2f806c790 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.340+0000 7fa2fca00700 1 -- 172.21.15.89:0/1563665824 >> 172.21.15.89:0/1563665824 conn(0x7fa2f80d1350 msgr2=0x7fa2f80d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:10.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.340+0000 7fa2fca00700 1 -- 172.21.15.89:0/1563665824 shutdown_connections 2024-04-02T07:37:10.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.340+0000 7fa2fca00700 1 -- 172.21.15.89:0/1563665824 wait complete. 2024-04-02T07:37:10.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.342+0000 7fa2fca00700 1 Processor -- start 2024-04-02T07:37:10.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.342+0000 7fa2fca00700 1 -- start start 2024-04-02T07:37:10.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2fca00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2f8099a60 0x7fa2f8156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:10.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2fca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2f809a910 0x7fa2f815dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:10.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2fca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f815d920 0x7fa2f80b6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:10.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2fca00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f8156e80 con 0x7fa2f815d920 2024-04-02T07:37:10.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2fca00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f815ecf0 con 0x7fa2f8099a60 2024-04-02T07:37:10.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2fca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f815e730 con 0x7fa2f809a910 2024-04-02T07:37:10.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2f6800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f815d920 0x7fa2f80b6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:10.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2f6800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f815d920 0x7fa2f80b6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58044/0 (socket says 172.21.15.89:58044) 2024-04-02T07:37:10.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2f6800700 1 -- 172.21.15.89:0/510395617 learned_addr learned my addr 172.21.15.89:0/510395617 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:10.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2f5e00700 1 --2- 172.21.15.89:0/510395617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2f8099a60 0x7fa2f8156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:10.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2f5400700 1 --2- 172.21.15.89:0/510395617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2f809a910 0x7fa2f815dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:10.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2f6800700 1 -- 172.21.15.89:0/510395617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2f809a910 msgr2=0x7fa2f815dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:10.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2f6800700 1 --2- 172.21.15.89:0/510395617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2f809a910 0x7fa2f815dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2f6800700 1 -- 172.21.15.89:0/510395617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2f8099a60 msgr2=0x7fa2f8156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:10.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2f6800700 1 --2- 172.21.15.89:0/510395617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2f8099a60 0x7fa2f8156940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.343+0000 7fa2f6800700 1 -- 172.21.15.89:0/510395617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e4009550 con 0x7fa2f815d920 2024-04-02T07:37:10.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.344+0000 7fa2f6800700 1 --2- 172.21.15.89:0/510395617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f815d920 0x7fa2f80b6370 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa2f000b970 tx=0x7fa2f000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:10.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.344+0000 7fa2ee000700 1 -- 172.21.15.89:0/510395617 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f000d860 con 0x7fa2f815d920 2024-04-02T07:37:10.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.344+0000 7fa2ee000700 1 -- 172.21.15.89:0/510395617 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f000d9e0 con 0x7fa2f815d920 2024-04-02T07:37:10.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.344+0000 7fa2fca00700 1 -- 172.21.15.89:0/510395617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f815ea10 con 0x7fa2f815d920 2024-04-02T07:37:10.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.344+0000 7fa2ee000700 1 -- 172.21.15.89:0/510395617 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f000dd90 con 0x7fa2f815d920 2024-04-02T07:37:10.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.344+0000 7fa2fca00700 1 -- 172.21.15.89:0/510395617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f80c87a0 con 0x7fa2f815d920 2024-04-02T07:37:10.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.345+0000 7fa2fca00700 1 -- 172.21.15.89:0/510395617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f80c8990 con 0x7fa2f815d920 2024-04-02T07:37:10.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.346+0000 7fa2ee000700 1 -- 172.21.15.89:0/510395617 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa2f000e430 con 0x7fa2f815d920 2024-04-02T07:37:10.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.348+0000 7fa2ee000700 1 --2- 172.21.15.89:0/510395617 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa2d806ac90 0x7fa2d806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:10.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.348+0000 7fa2ee000700 1 -- 172.21.15.89:0/510395617 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2f0006e40 con 0x7fa2f815d920 2024-04-02T07:37:10.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.348+0000 7fa2f5e00700 1 --2- 172.21.15.89:0/510395617 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa2d806ac90 0x7fa2d806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:10.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.348+0000 7fa2f5e00700 1 --2- 172.21.15.89:0/510395617 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa2d806ac90 0x7fa2d806d150 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8009960 tx=0x7fa2e80090e0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:10.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.351+0000 7fa2ee000700 1 -- 172.21.15.89:0/510395617 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa2f0054600 con 0x7fa2f815d920 2024-04-02T07:37:10.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.546+0000 7fa2fca00700 1 -- 172.21.15.89:0/510395617 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2f806bf90 con 0x7fa2d806ac90 2024-04-02T07:37:10.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.554+0000 7fa2ee000700 1 -- 172.21.15.89:0/510395617 <== mgr.4108 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289135 (secure 0 0 0) 0x7fa2e8009e30 con 0x7fa2d806ac90 2024-04-02T07:37:10.560 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:37:10.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.558+0000 7fa2fca00700 1 -- 172.21.15.89:0/510395617 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa2d806ac90 msgr2=0x7fa2d806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:10.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.558+0000 7fa2fca00700 1 --2- 172.21.15.89:0/510395617 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa2d806ac90 0x7fa2d806d150 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8009960 tx=0x7fa2e80090e0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.558+0000 7fa2fca00700 1 -- 172.21.15.89:0/510395617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f815d920 msgr2=0x7fa2f80b6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:10.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.558+0000 7fa2fca00700 1 --2- 172.21.15.89:0/510395617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f815d920 0x7fa2f80b6370 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa2f000b970 tx=0x7fa2f000bc80 comp rx=0 tx=0).stop 2024-04-02T07:37:10.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.559+0000 7fa2fca00700 1 -- 172.21.15.89:0/510395617 shutdown_connections 2024-04-02T07:37:10.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.559+0000 7fa2fca00700 1 --2- 172.21.15.89:0/510395617 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa2d806ac90 0x7fa2d806d150 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.559+0000 7fa2fca00700 1 --2- 172.21.15.89:0/510395617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2f8099a60 0x7fa2f8156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.559+0000 7fa2fca00700 1 --2- 172.21.15.89:0/510395617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f815d920 0x7fa2f80b6370 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.559+0000 7fa2fca00700 1 --2- 172.21.15.89:0/510395617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2f809a910 0x7fa2f815dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.559+0000 7fa2fca00700 1 -- 172.21.15.89:0/510395617 >> 172.21.15.89:0/510395617 conn(0x7fa2f80d1350 msgr2=0x7fa2f80ccbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:10.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.559+0000 7fa2fca00700 1 -- 172.21.15.89:0/510395617 shutdown_connections 2024-04-02T07:37:10.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.559+0000 7fa2fca00700 1 -- 172.21.15.89:0/510395617 wait complete. 2024-04-02T07:37:10.564 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:37:10.578 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-04-02T07:37:09.197108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220312,"kb_used_data":4096,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754408,"statfs":{"total":773094113280,"available":772868513792,"internally_reserved":0,"allocated":4194304,"data_stored":2330028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.203091"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.532628+0000","last_change":"2024-04-02T07:37:05.526626+0000","last_active":"2024-04-02T07:37:05.532628+0000","last_peered":"2024-04-02T07:37:05.532628+0000","last_clean":"2024-04-02T07:37:05.532628+0000","last_became_active":"2024-04-02T07:37:05.526161+0000","last_became_peered":"2024-04-02T07:37:05.526161+0000","last_unstale":"2024-04-02T07:37:05.532628+0000","last_undegraded":"2024-04-02T07:37:05.532628+0000","last_fullsized":"2024-04-02T07:37:05.532628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:37:25.422881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.526370+0000","last_change":"2024-04-02T07:37:05.520114+0000","last_active":"2024-04-02T07:37:05.526370+0000","last_peered":"2024-04-02T07:37:05.526370+0000","last_clean":"2024-04-02T07:37:05.526370+0000","last_became_active":"2024-04-02T07:37:05.519845+0000","last_became_peered":"2024-04-02T07:37:05.519845+0000","last_unstale":"2024-04-02T07:37:05.526370+0000","last_undegraded":"2024-04-02T07:37:05.526370+0000","last_fullsized":"2024-04-02T07:37:05.526370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:16.240622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.531240+0000","last_change":"2024-04-02T07:37:05.520038+0000","last_active":"2024-04-02T07:37:05.531240+0000","last_peered":"2024-04-02T07:37:05.531240+0000","last_clean":"2024-04-02T07:37:05.531240+0000","last_became_active":"2024-04-02T07:37:05.519763+0000","last_became_peered":"2024-04-02T07:37:05.519763+0000","last_unstale":"2024-04-02T07:37:05.531240+0000","last_undegraded":"2024-04-02T07:37:05.531240+0000","last_fullsized":"2024-04-02T07:37:05.531240+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:37.522877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540866+0000","last_change":"2024-04-02T07:37:05.534692+0000","last_active":"2024-04-02T07:37:05.540866+0000","last_peered":"2024-04-02T07:37:05.540866+0000","last_clean":"2024-04-02T07:37:05.540866+0000","last_became_active":"2024-04-02T07:37:05.526775+0000","last_became_peered":"2024-04-02T07:37:05.526775+0000","last_unstale":"2024-04-02T07:37:05.540866+0000","last_undegraded":"2024-04-02T07:37:05.540866+0000","last_fullsized":"2024-04-02T07:37:05.540866+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:55:17.522766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.534152+0000","last_change":"2024-04-02T07:37:05.531729+0000","last_active":"2024-04-02T07:37:05.534152+0000","last_peered":"2024-04-02T07:37:05.534152+0000","last_clean":"2024-04-02T07:37:05.534152+0000","last_became_active":"2024-04-02T07:37:05.531111+0000","last_became_peered":"2024-04-02T07:37:05.531111+0000","last_unstale":"2024-04-02T07:37:05.534152+0000","last_undegraded":"2024-04-02T07:37:05.534152+0000","last_fullsized":"2024-04-02T07:37:05.534152+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:39.600705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.539623+0000","last_change":"2024-04-02T07:37:05.536711+0000","last_active":"2024-04-02T07:37:05.539623+0000","last_peered":"2024-04-02T07:37:05.539623+0000","last_clean":"2024-04-02T07:37:05.539623+0000","last_became_active":"2024-04-02T07:37:05.536437+0000","last_became_peered":"2024-04-02T07:37:05.536437+0000","last_unstale":"2024-04-02T07:37:05.539623+0000","last_undegraded":"2024-04-02T07:37:05.539623+0000","last_fullsized":"2024-04-02T07:37:05.539623+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:37:31.992660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.529478+0000","last_change":"2024-04-02T07:37:05.520544+0000","last_active":"2024-04-02T07:37:05.529478+0000","last_peered":"2024-04-02T07:37:05.529478+0000","last_clean":"2024-04-02T07:37:05.529478+0000","last_became_active":"2024-04-02T07:37:05.520224+0000","last_became_peered":"2024-04-02T07:37:05.520224+0000","last_unstale":"2024-04-02T07:37:05.529478+0000","last_undegraded":"2024-04-02T07:37:05.529478+0000","last_fullsized":"2024-04-02T07:37:05.529478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:34:05.497678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.522023+0000","last_change":"2024-04-02T07:37:05.517993+0000","last_active":"2024-04-02T07:37:05.522023+0000","last_peered":"2024-04-02T07:37:05.522023+0000","last_clean":"2024-04-02T07:37:05.522023+0000","last_became_active":"2024-04-02T07:37:05.517705+0000","last_became_peered":"2024-04-02T07:37:05.517705+0000","last_unstale":"2024-04-02T07:37:05.522023+0000","last_undegraded":"2024-04-02T07:37:05.522023+0000","last_fullsized":"2024-04-02T07:37:05.522023+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:16.802810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.536719+0000","last_change":"2024-04-02T07:37:05.532414+0000","last_active":"2024-04-02T07:37:05.536719+0000","last_peered":"2024-04-02T07:37:05.536719+0000","last_clean":"2024-04-02T07:37:05.536719+0000","last_became_active":"2024-04-02T07:37:05.532139+0000","last_became_peered":"2024-04-02T07:37:05.532139+0000","last_unstale":"2024-04-02T07:37:05.536719+0000","last_undegraded":"2024-04-02T07:37:05.536719+0000","last_fullsized":"2024-04-02T07:37:05.536719+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:48:35.710658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.532983+0000","last_change":"2024-04-02T07:37:05.520746+0000","last_active":"2024-04-02T07:37:05.532983+0000","last_peered":"2024-04-02T07:37:05.532983+0000","last_clean":"2024-04-02T07:37:05.532983+0000","last_became_active":"2024-04-02T07:37:05.520484+0000","last_became_peered":"2024-04-02T07:37:05.520484+0000","last_unstale":"2024-04-02T07:37:05.532983+0000","last_undegraded":"2024-04-02T07:37:05.532983+0000","last_fullsized":"2024-04-02T07:37:05.532983+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:49:47.968768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.541199+0000","last_change":"2024-04-02T07:37:05.538852+0000","last_active":"2024-04-02T07:37:05.541199+0000","last_peered":"2024-04-02T07:37:05.541199+0000","last_clean":"2024-04-02T07:37:05.541199+0000","last_became_active":"2024-04-02T07:37:05.538257+0000","last_became_peered":"2024-04-02T07:37:05.538257+0000","last_unstale":"2024-04-02T07:37:05.541199+0000","last_undegraded":"2024-04-02T07:37:05.541199+0000","last_fullsized":"2024-04-02T07:37:05.541199+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:52:03.612362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.527656+0000","last_change":"2024-04-02T07:37:05.519945+0000","last_active":"2024-04-02T07:37:05.527656+0000","last_peered":"2024-04-02T07:37:05.527656+0000","last_clean":"2024-04-02T07:37:05.527656+0000","last_became_active":"2024-04-02T07:37:05.519622+0000","last_became_peered":"2024-04-02T07:37:05.519622+0000","last_unstale":"2024-04-02T07:37:05.527656+0000","last_undegraded":"2024-04-02T07:37:05.527656+0000","last_fullsized":"2024-04-02T07:37:05.527656+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:19:37.687845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.529277+0000","last_change":"2024-04-02T07:37:05.523587+0000","last_active":"2024-04-02T07:37:05.529277+0000","last_peered":"2024-04-02T07:37:05.529277+0000","last_clean":"2024-04-02T07:37:05.529277+0000","last_became_active":"2024-04-02T07:37:05.522929+0000","last_became_peered":"2024-04-02T07:37:05.522929+0000","last_unstale":"2024-04-02T07:37:05.529277+0000","last_undegraded":"2024-04-02T07:37:05.529277+0000","last_fullsized":"2024-04-02T07:37:05.529277+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:35:46.006569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540609+0000","last_change":"2024-04-02T07:37:05.529739+0000","last_active":"2024-04-02T07:37:05.540609+0000","last_peered":"2024-04-02T07:37:05.540609+0000","last_clean":"2024-04-02T07:37:05.540609+0000","last_became_active":"2024-04-02T07:37:05.529471+0000","last_became_peered":"2024-04-02T07:37:05.529471+0000","last_unstale":"2024-04-02T07:37:05.540609+0000","last_undegraded":"2024-04-02T07:37:05.540609+0000","last_fullsized":"2024-04-02T07:37:05.540609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:58:10.466063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.538503+0000","last_change":"2024-04-02T07:37:05.530938+0000","last_active":"2024-04-02T07:37:05.538503+0000","last_peered":"2024-04-02T07:37:05.538503+0000","last_clean":"2024-04-02T07:37:05.538503+0000","last_became_active":"2024-04-02T07:37:05.530621+0000","last_became_peered":"2024-04-02T07:37:05.530621+0000","last_unstale":"2024-04-02T07:37:05.538503+0000","last_undegraded":"2024-04-02T07:37:05.538503+0000","last_fullsized":"2024-04-02T07:37:05.538503+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:28.621627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540103+0000","last_change":"2024-04-02T07:37:05.536455+0000","last_active":"2024-04-02T07:37:05.540103+0000","last_peered":"2024-04-02T07:37:05.540103+0000","last_clean":"2024-04-02T07:37:05.540103+0000","last_became_active":"2024-04-02T07:37:05.536183+0000","last_became_peered":"2024-04-02T07:37:05.536183+0000","last_unstale":"2024-04-02T07:37:05.540103+0000","last_undegraded":"2024-04-02T07:37:05.540103+0000","last_fullsized":"2024-04-02T07:37:05.540103+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:16.240622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540962+0000","last_change":"2024-04-02T07:37:05.534556+0000","last_active":"2024-04-02T07:37:05.540962+0000","last_peered":"2024-04-02T07:37:05.540962+0000","last_clean":"2024-04-02T07:37:05.540962+0000","last_became_active":"2024-04-02T07:37:05.534315+0000","last_became_peered":"2024-04-02T07:37:05.534315+0000","last_unstale":"2024-04-02T07:37:05.540962+0000","last_undegraded":"2024-04-02T07:37:05.540962+0000","last_fullsized":"2024-04-02T07:37:05.540962+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:10:22.041218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.539579+0000","last_change":"2024-04-02T07:37:05.535554+0000","last_active":"2024-04-02T07:37:05.539579+0000","last_peered":"2024-04-02T07:37:05.539579+0000","last_clean":"2024-04-02T07:37:05.539579+0000","last_became_active":"2024-04-02T07:37:05.534253+0000","last_became_peered":"2024-04-02T07:37:05.534253+0000","last_unstale":"2024-04-02T07:37:05.539579+0000","last_undegraded":"2024-04-02T07:37:05.539579+0000","last_fullsized":"2024-04-02T07:37:05.539579+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:18.250998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540276+0000","last_change":"2024-04-02T07:37:05.538345+0000","last_active":"2024-04-02T07:37:05.540276+0000","last_peered":"2024-04-02T07:37:05.540276+0000","last_clean":"2024-04-02T07:37:05.540276+0000","last_became_active":"2024-04-02T07:37:05.537983+0000","last_became_peered":"2024-04-02T07:37:05.537983+0000","last_unstale":"2024-04-02T07:37:05.540276+0000","last_undegraded":"2024-04-02T07:37:05.540276+0000","last_fullsized":"2024-04-02T07:37:05.540276+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:47:31.372894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.534512+0000","last_change":"2024-04-02T07:37:05.530867+0000","last_active":"2024-04-02T07:37:05.534512+0000","last_peered":"2024-04-02T07:37:05.534512+0000","last_clean":"2024-04-02T07:37:05.534512+0000","last_became_active":"2024-04-02T07:37:05.526527+0000","last_became_peered":"2024-04-02T07:37:05.526527+0000","last_unstale":"2024-04-02T07:37:05.534512+0000","last_undegraded":"2024-04-02T07:37:05.534512+0000","last_fullsized":"2024-04-02T07:37:05.534512+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:16.802810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.538462+0000","last_change":"2024-04-02T07:37:05.530155+0000","last_active":"2024-04-02T07:37:05.538462+0000","last_peered":"2024-04-02T07:37:05.538462+0000","last_clean":"2024-04-02T07:37:05.538462+0000","last_became_active":"2024-04-02T07:37:05.529885+0000","last_became_peered":"2024-04-02T07:37:05.529885+0000","last_unstale":"2024-04-02T07:37:05.538462+0000","last_undegraded":"2024-04-02T07:37:05.538462+0000","last_fullsized":"2024-04-02T07:37:05.538462+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:49:47.968768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.529980+0000","last_change":"2024-04-02T07:37:05.521517+0000","last_active":"2024-04-02T07:37:05.529980+0000","last_peered":"2024-04-02T07:37:05.529980+0000","last_clean":"2024-04-02T07:37:05.529980+0000","last_became_active":"2024-04-02T07:37:05.521225+0000","last_became_peered":"2024-04-02T07:37:05.521225+0000","last_unstale":"2024-04-02T07:37:05.529980+0000","last_undegraded":"2024-04-02T07:37:05.529980+0000","last_fullsized":"2024-04-02T07:37:05.529980+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:16.240622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.530720+0000","last_change":"2024-04-02T07:37:05.524984+0000","last_active":"2024-04-02T07:37:05.530720+0000","last_peered":"2024-04-02T07:37:05.530720+0000","last_clean":"2024-04-02T07:37:05.530720+0000","last_became_active":"2024-04-02T07:37:05.524594+0000","last_became_peered":"2024-04-02T07:37:05.524594+0000","last_unstale":"2024-04-02T07:37:05.530720+0000","last_undegraded":"2024-04-02T07:37:05.530720+0000","last_fullsized":"2024-04-02T07:37:05.530720+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:37:25.422881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.518751+0000","last_change":"2024-04-02T07:37:05.515902+0000","last_active":"2024-04-02T07:37:05.518751+0000","last_peered":"2024-04-02T07:37:05.518751+0000","last_clean":"2024-04-02T07:37:05.518751+0000","last_became_active":"2024-04-02T07:37:05.515567+0000","last_became_peered":"2024-04-02T07:37:05.515567+0000","last_unstale":"2024-04-02T07:37:05.518751+0000","last_undegraded":"2024-04-02T07:37:05.518751+0000","last_fullsized":"2024-04-02T07:37:05.518751+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:40.461363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540670+0000","last_change":"2024-04-02T07:37:05.531924+0000","last_active":"2024-04-02T07:37:05.540670+0000","last_peered":"2024-04-02T07:37:05.540670+0000","last_clean":"2024-04-02T07:37:05.540670+0000","last_became_active":"2024-04-02T07:37:05.531560+0000","last_became_peered":"2024-04-02T07:37:05.531560+0000","last_unstale":"2024-04-02T07:37:05.540670+0000","last_undegraded":"2024-04-02T07:37:05.540670+0000","last_fullsized":"2024-04-02T07:37:05.540670+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:40.461363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.523826+0000","last_change":"2024-04-02T07:37:05.517114+0000","last_active":"2024-04-02T07:37:05.523826+0000","last_peered":"2024-04-02T07:37:05.523826+0000","last_clean":"2024-04-02T07:37:05.523826+0000","last_became_active":"2024-04-02T07:37:05.516736+0000","last_became_peered":"2024-04-02T07:37:05.516736+0000","last_unstale":"2024-04-02T07:37:05.523826+0000","last_undegraded":"2024-04-02T07:37:05.523826+0000","last_fullsized":"2024-04-02T07:37:05.523826+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:28.621627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.533275+0000","last_change":"2024-04-02T07:37:05.530183+0000","last_active":"2024-04-02T07:37:05.533275+0000","last_peered":"2024-04-02T07:37:05.533275+0000","last_clean":"2024-04-02T07:37:05.533275+0000","last_became_active":"2024-04-02T07:37:05.527801+0000","last_became_peered":"2024-04-02T07:37:05.527801+0000","last_unstale":"2024-04-02T07:37:05.533275+0000","last_undegraded":"2024-04-02T07:37:05.533275+0000","last_fullsized":"2024-04-02T07:37:05.533275+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:16.802810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.520361+0000","last_change":"2024-04-02T07:37:05.517149+0000","last_active":"2024-04-02T07:37:05.520361+0000","last_peered":"2024-04-02T07:37:05.520361+0000","last_clean":"2024-04-02T07:37:05.520361+0000","last_became_active":"2024-04-02T07:37:05.516876+0000","last_became_peered":"2024-04-02T07:37:05.516876+0000","last_unstale":"2024-04-02T07:37:05.520361+0000","last_undegraded":"2024-04-02T07:37:05.520361+0000","last_fullsized":"2024-04-02T07:37:05.520361+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:58:10.466063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.539922+0000","last_change":"2024-04-02T07:37:05.531691+0000","last_active":"2024-04-02T07:37:05.539922+0000","last_peered":"2024-04-02T07:37:05.539922+0000","last_clean":"2024-04-02T07:37:05.539922+0000","last_became_active":"2024-04-02T07:37:05.531273+0000","last_became_peered":"2024-04-02T07:37:05.531273+0000","last_unstale":"2024-04-02T07:37:05.539922+0000","last_undegraded":"2024-04-02T07:37:05.539922+0000","last_fullsized":"2024-04-02T07:37:05.539922+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:26:44.688366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.533108+0000","last_change":"2024-04-02T07:37:05.524509+0000","last_active":"2024-04-02T07:37:05.533108+0000","last_peered":"2024-04-02T07:37:05.533108+0000","last_clean":"2024-04-02T07:37:05.533108+0000","last_became_active":"2024-04-02T07:37:05.524223+0000","last_became_peered":"2024-04-02T07:37:05.524223+0000","last_unstale":"2024-04-02T07:37:05.533108+0000","last_undegraded":"2024-04-02T07:37:05.533108+0000","last_fullsized":"2024-04-02T07:37:05.533108+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:40.461363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540648+0000","last_change":"2024-04-02T07:37:05.535742+0000","last_active":"2024-04-02T07:37:05.540648+0000","last_peered":"2024-04-02T07:37:05.540648+0000","last_clean":"2024-04-02T07:37:05.540648+0000","last_became_active":"2024-04-02T07:37:05.535473+0000","last_became_peered":"2024-04-02T07:37:05.535473+0000","last_unstale":"2024-04-02T07:37:05.540648+0000","last_undegraded":"2024-04-02T07:37:05.540648+0000","last_fullsized":"2024-04-02T07:37:05.540648+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:45:55.869487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540815+0000","last_change":"2024-04-02T07:37:05.534076+0000","last_active":"2024-04-02T07:37:05.540815+0000","last_peered":"2024-04-02T07:37:05.540815+0000","last_clean":"2024-04-02T07:37:05.540815+0000","last_became_active":"2024-04-02T07:37:05.533783+0000","last_became_peered":"2024-04-02T07:37:05.533783+0000","last_unstale":"2024-04-02T07:37:05.540815+0000","last_undegraded":"2024-04-02T07:37:05.540815+0000","last_fullsized":"2024-04-02T07:37:05.540815+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:52:30.610913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.533095+0000","last_change":"2024-04-02T07:37:05.529875+0000","last_active":"2024-04-02T07:37:05.533095+0000","last_peered":"2024-04-02T07:37:05.533095+0000","last_clean":"2024-04-02T07:37:05.533095+0000","last_became_active":"2024-04-02T07:37:05.529654+0000","last_became_peered":"2024-04-02T07:37:05.529654+0000","last_unstale":"2024-04-02T07:37:05.533095+0000","last_undegraded":"2024-04-02T07:37:05.533095+0000","last_fullsized":"2024-04-02T07:37:05.533095+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:45:55.869487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.537203+0000","last_change":"2024-04-02T07:37:05.020065+0000","last_active":"2024-04-02T07:37:05.537203+0000","last_peered":"2024-04-02T07:37:05.537203+0000","last_clean":"2024-04-02T07:37:05.537203+0000","last_became_active":"2024-04-02T07:37:05.019738+0000","last_became_peered":"2024-04-02T07:37:05.019738+0000","last_unstale":"2024-04-02T07:37:05.537203+0000","last_undegraded":"2024-04-02T07:37:05.537203+0000","last_fullsized":"2024-04-02T07:37:05.537203+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:30:41.797989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.532608+0000","last_change":"2024-04-02T07:37:05.526435+0000","last_active":"2024-04-02T07:37:05.532608+0000","last_peered":"2024-04-02T07:37:05.532608+0000","last_clean":"2024-04-02T07:37:05.532608+0000","last_became_active":"2024-04-02T07:37:05.523100+0000","last_became_peered":"2024-04-02T07:37:05.523100+0000","last_unstale":"2024-04-02T07:37:05.532608+0000","last_undegraded":"2024-04-02T07:37:05.532608+0000","last_fullsized":"2024-04-02T07:37:05.532608+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:51:55.529401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.542419+0000","last_change":"2024-04-02T07:37:05.026060+0000","last_active":"2024-04-02T07:37:05.542419+0000","last_peered":"2024-04-02T07:37:05.542419+0000","last_clean":"2024-04-02T07:37:05.542419+0000","last_became_active":"2024-04-02T07:37:05.025805+0000","last_became_peered":"2024-04-02T07:37:05.025805+0000","last_unstale":"2024-04-02T07:37:05.542419+0000","last_undegraded":"2024-04-02T07:37:05.542419+0000","last_fullsized":"2024-04-02T07:37:05.542419+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:48:30.160948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.522043+0000","last_change":"2024-04-02T07:37:05.518097+0000","last_active":"2024-04-02T07:37:05.522043+0000","last_peered":"2024-04-02T07:37:05.522043+0000","last_clean":"2024-04-02T07:37:05.522043+0000","last_became_active":"2024-04-02T07:37:05.517854+0000","last_became_peered":"2024-04-02T07:37:05.517854+0000","last_unstale":"2024-04-02T07:37:05.522043+0000","last_undegraded":"2024-04-02T07:37:05.522043+0000","last_fullsized":"2024-04-02T07:37:05.522043+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:30:42.804445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.535559+0000","last_change":"2024-04-02T07:37:05.023807+0000","last_active":"2024-04-02T07:37:05.535559+0000","last_peered":"2024-04-02T07:37:05.535559+0000","last_clean":"2024-04-02T07:37:05.535559+0000","last_became_active":"2024-04-02T07:37:05.023512+0000","last_became_peered":"2024-04-02T07:37:05.023512+0000","last_unstale":"2024-04-02T07:37:05.535559+0000","last_undegraded":"2024-04-02T07:37:05.535559+0000","last_fullsized":"2024-04-02T07:37:05.535559+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:24:24.154489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.542467+0000","last_change":"2024-04-02T07:37:05.536264+0000","last_active":"2024-04-02T07:37:05.542467+0000","last_peered":"2024-04-02T07:37:05.542467+0000","last_clean":"2024-04-02T07:37:05.542467+0000","last_became_active":"2024-04-02T07:37:05.531691+0000","last_became_peered":"2024-04-02T07:37:05.531691+0000","last_unstale":"2024-04-02T07:37:05.542467+0000","last_undegraded":"2024-04-02T07:37:05.542467+0000","last_fullsized":"2024-04-02T07:37:05.542467+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:02.936396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.542470+0000","last_change":"2024-04-02T07:37:05.028409+0000","last_active":"2024-04-02T07:37:05.542470+0000","last_peered":"2024-04-02T07:37:05.542470+0000","last_clean":"2024-04-02T07:37:05.542470+0000","last_became_active":"2024-04-02T07:37:05.028119+0000","last_became_peered":"2024-04-02T07:37:05.028119+0000","last_unstale":"2024-04-02T07:37:05.542470+0000","last_undegraded":"2024-04-02T07:37:05.542470+0000","last_fullsized":"2024-04-02T07:37:05.542470+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:48:34.704201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540412+0000","last_change":"2024-04-02T07:37:05.535227+0000","last_active":"2024-04-02T07:37:05.540412+0000","last_peered":"2024-04-02T07:37:05.540412+0000","last_clean":"2024-04-02T07:37:05.540412+0000","last_became_active":"2024-04-02T07:37:05.534909+0000","last_became_peered":"2024-04-02T07:37:05.534909+0000","last_unstale":"2024-04-02T07:37:05.540412+0000","last_undegraded":"2024-04-02T07:37:05.540412+0000","last_fullsized":"2024-04-02T07:37:05.540412+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:48:31.167405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.534316+0000","last_change":"2024-04-02T07:37:05.031842+0000","last_active":"2024-04-02T07:37:05.534316+0000","last_peered":"2024-04-02T07:37:05.534316+0000","last_clean":"2024-04-02T07:37:05.534316+0000","last_became_active":"2024-04-02T07:37:05.031576+0000","last_became_peered":"2024-04-02T07:37:05.031576+0000","last_unstale":"2024-04-02T07:37:05.534316+0000","last_undegraded":"2024-04-02T07:37:05.534316+0000","last_fullsized":"2024-04-02T07:37:05.534316+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:24:24.154489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.542691+0000","last_change":"2024-04-02T07:37:05.540654+0000","last_active":"2024-04-02T07:37:05.542691+0000","last_peered":"2024-04-02T07:37:05.542691+0000","last_clean":"2024-04-02T07:37:05.542691+0000","last_became_active":"2024-04-02T07:37:05.538027+0000","last_became_peered":"2024-04-02T07:37:05.538027+0000","last_unstale":"2024-04-02T07:37:05.542691+0000","last_undegraded":"2024-04-02T07:37:05.542691+0000","last_fullsized":"2024-04-02T07:37:05.542691+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:39.600705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.526979+0000","last_change":"2024-04-02T07:37:05.029193+0000","last_active":"2024-04-02T07:37:05.526979+0000","last_peered":"2024-04-02T07:37:05.526979+0000","last_clean":"2024-04-02T07:37:05.526979+0000","last_became_active":"2024-04-02T07:37:05.028953+0000","last_became_peered":"2024-04-02T07:37:05.028953+0000","last_unstale":"2024-04-02T07:37:05.526979+0000","last_undegraded":"2024-04-02T07:37:05.526979+0000","last_fullsized":"2024-04-02T07:37:05.526979+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:35:23.779994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.536375+0000","last_change":"2024-04-02T07:37:05.526476+0000","last_active":"2024-04-02T07:37:05.536375+0000","last_peered":"2024-04-02T07:37:05.536375+0000","last_clean":"2024-04-02T07:37:05.536375+0000","last_became_active":"2024-04-02T07:37:05.526211+0000","last_became_peered":"2024-04-02T07:37:05.526211+0000","last_unstale":"2024-04-02T07:37:05.536375+0000","last_undegraded":"2024-04-02T07:37:05.536375+0000","last_fullsized":"2024-04-02T07:37:05.536375+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:22:53.895269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.518787+0000","last_change":"2024-04-02T07:37:05.023707+0000","last_active":"2024-04-02T07:37:05.518787+0000","last_peered":"2024-04-02T07:37:05.518787+0000","last_clean":"2024-04-02T07:37:05.518787+0000","last_became_active":"2024-04-02T07:37:05.023236+0000","last_became_peered":"2024-04-02T07:37:05.023236+0000","last_unstale":"2024-04-02T07:37:05.518787+0000","last_undegraded":"2024-04-02T07:37:05.518787+0000","last_fullsized":"2024-04-02T07:37:05.518787+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:15:49.804855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.532165+0000","last_change":"2024-04-02T07:37:05.524408+0000","last_active":"2024-04-02T07:37:05.532165+0000","last_peered":"2024-04-02T07:37:05.532165+0000","last_clean":"2024-04-02T07:37:05.532165+0000","last_became_active":"2024-04-02T07:37:05.524050+0000","last_became_peered":"2024-04-02T07:37:05.524050+0000","last_unstale":"2024-04-02T07:37:05.532165+0000","last_undegraded":"2024-04-02T07:37:05.532165+0000","last_fullsized":"2024-04-02T07:37:05.532165+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:16.240622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.505983+0000","last_change":"2024-04-02T07:37:05.023237+0000","last_active":"2024-04-02T07:37:05.505983+0000","last_peered":"2024-04-02T07:37:05.505983+0000","last_clean":"2024-04-02T07:37:05.505983+0000","last_became_active":"2024-04-02T07:37:05.022779+0000","last_became_peered":"2024-04-02T07:37:05.022779+0000","last_unstale":"2024-04-02T07:37:05.505983+0000","last_undegraded":"2024-04-02T07:37:05.505983+0000","last_fullsized":"2024-04-02T07:37:05.505983+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:22:52.888812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.541036+0000","last_change":"2024-04-02T07:37:05.534980+0000","last_active":"2024-04-02T07:37:05.541036+0000","last_peered":"2024-04-02T07:37:05.541036+0000","last_clean":"2024-04-02T07:37:05.541036+0000","last_became_active":"2024-04-02T07:37:05.534725+0000","last_became_peered":"2024-04-02T07:37:05.534725+0000","last_unstale":"2024-04-02T07:37:05.541036+0000","last_undegraded":"2024-04-02T07:37:05.541036+0000","last_fullsized":"2024-04-02T07:37:05.541036+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:52:35.298086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.514241+0000","last_change":"2024-04-02T07:37:05.026942+0000","last_active":"2024-04-02T07:37:05.514241+0000","last_peered":"2024-04-02T07:37:05.514241+0000","last_clean":"2024-04-02T07:37:05.514241+0000","last_became_active":"2024-04-02T07:37:05.026700+0000","last_became_peered":"2024-04-02T07:37:05.026700+0000","last_unstale":"2024-04-02T07:37:05.514241+0000","last_undegraded":"2024-04-02T07:37:05.514241+0000","last_fullsized":"2024-04-02T07:37:05.514241+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:14.637885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.539588+0000","last_change":"2024-04-02T07:37:05.521902+0000","last_active":"2024-04-02T07:37:05.539588+0000","last_peered":"2024-04-02T07:37:05.539588+0000","last_clean":"2024-04-02T07:37:05.539588+0000","last_became_active":"2024-04-02T07:37:05.521581+0000","last_became_peered":"2024-04-02T07:37:05.521581+0000","last_unstale":"2024-04-02T07:37:05.539588+0000","last_undegraded":"2024-04-02T07:37:05.539588+0000","last_fullsized":"2024-04-02T07:37:05.539588+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:30:42.804445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.513027+0000","last_change":"2024-04-02T07:37:05.025628+0000","last_active":"2024-04-02T07:37:05.513027+0000","last_peered":"2024-04-02T07:37:05.513027+0000","last_clean":"2024-04-02T07:37:05.513027+0000","last_became_active":"2024-04-02T07:37:05.019281+0000","last_became_peered":"2024-04-02T07:37:05.019281+0000","last_unstale":"2024-04-02T07:37:05.513027+0000","last_undegraded":"2024-04-02T07:37:05.513027+0000","last_fullsized":"2024-04-02T07:37:05.513027+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:37:30.986204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540845+0000","last_change":"2024-04-02T07:37:05.534430+0000","last_active":"2024-04-02T07:37:05.540845+0000","last_peered":"2024-04-02T07:37:05.540845+0000","last_clean":"2024-04-02T07:37:05.540845+0000","last_became_active":"2024-04-02T07:37:05.530003+0000","last_became_peered":"2024-04-02T07:37:05.530003+0000","last_unstale":"2024-04-02T07:37:05.540845+0000","last_undegraded":"2024-04-02T07:37:05.540845+0000","last_fullsized":"2024-04-02T07:37:05.540845+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:37:08.135605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.539124+0000","last_change":"2024-04-02T07:37:05.015931+0000","last_active":"2024-04-02T07:37:05.539124+0000","last_peered":"2024-04-02T07:37:05.539124+0000","last_clean":"2024-04-02T07:37:05.539124+0000","last_became_active":"2024-04-02T07:37:05.015586+0000","last_became_peered":"2024-04-02T07:37:05.015586+0000","last_unstale":"2024-04-02T07:37:05.539124+0000","last_undegraded":"2024-04-02T07:37:05.539124+0000","last_fullsized":"2024-04-02T07:37:05.539124+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:51:54.522944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.541215+0000","last_change":"2024-04-02T07:37:05.538980+0000","last_active":"2024-04-02T07:37:05.541215+0000","last_peered":"2024-04-02T07:37:05.541215+0000","last_clean":"2024-04-02T07:37:05.541215+0000","last_became_active":"2024-04-02T07:37:05.538575+0000","last_became_peered":"2024-04-02T07:37:05.538575+0000","last_unstale":"2024-04-02T07:37:05.541215+0000","last_undegraded":"2024-04-02T07:37:05.541215+0000","last_fullsized":"2024-04-02T07:37:05.541215+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:55:09.846931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.547957+0000","last_change":"2024-04-02T07:37:05.026802+0000","last_active":"2024-04-02T07:37:05.547957+0000","last_peered":"2024-04-02T07:37:05.547957+0000","last_clean":"2024-04-02T07:37:05.547957+0000","last_became_active":"2024-04-02T07:37:05.025682+0000","last_became_peered":"2024-04-02T07:37:05.025682+0000","last_unstale":"2024-04-02T07:37:05.547957+0000","last_undegraded":"2024-04-02T07:37:05.547957+0000","last_fullsized":"2024-04-02T07:37:05.547957+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:48:30.160948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.528453+0000","last_change":"2024-04-02T07:37:05.523689+0000","last_active":"2024-04-02T07:37:05.528453+0000","last_peered":"2024-04-02T07:37:05.528453+0000","last_clean":"2024-04-02T07:37:05.528453+0000","last_became_active":"2024-04-02T07:37:05.523191+0000","last_became_peered":"2024-04-02T07:37:05.523191+0000","last_unstale":"2024-04-02T07:37:05.528453+0000","last_undegraded":"2024-04-02T07:37:05.528453+0000","last_fullsized":"2024-04-02T07:37:05.528453+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:47:31.372894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.532177+0000","last_change":"2024-04-02T07:37:05.026612+0000","last_active":"2024-04-02T07:37:05.532177+0000","last_peered":"2024-04-02T07:37:05.532177+0000","last_clean":"2024-04-02T07:37:05.532177+0000","last_became_active":"2024-04-02T07:37:05.026394+0000","last_became_peered":"2024-04-02T07:37:05.026394+0000","last_unstale":"2024-04-02T07:37:05.532177+0000","last_undegraded":"2024-04-02T07:37:05.532177+0000","last_fullsized":"2024-04-02T07:37:05.532177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:11:53.146618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.545647+0000","last_change":"2024-04-02T07:37:05.539883+0000","last_active":"2024-04-02T07:37:05.545647+0000","last_peered":"2024-04-02T07:37:05.545647+0000","last_clean":"2024-04-02T07:37:05.545647+0000","last_became_active":"2024-04-02T07:37:05.534733+0000","last_became_peered":"2024-04-02T07:37:05.534733+0000","last_unstale":"2024-04-02T07:37:05.545647+0000","last_undegraded":"2024-04-02T07:37:05.545647+0000","last_fullsized":"2024-04-02T07:37:05.545647+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:32:12.957088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.510286+0000","last_change":"2024-04-02T07:37:05.024394+0000","last_active":"2024-04-02T07:37:05.510286+0000","last_peered":"2024-04-02T07:37:05.510286+0000","last_clean":"2024-04-02T07:37:05.510286+0000","last_became_active":"2024-04-02T07:37:05.024075+0000","last_became_peered":"2024-04-02T07:37:05.024075+0000","last_unstale":"2024-04-02T07:37:05.510286+0000","last_undegraded":"2024-04-02T07:37:05.510286+0000","last_fullsized":"2024-04-02T07:37:05.510286+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:19:36.681389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.518732+0000","last_change":"2024-04-02T07:37:05.514468+0000","last_active":"2024-04-02T07:37:05.518732+0000","last_peered":"2024-04-02T07:37:05.518732+0000","last_clean":"2024-04-02T07:37:05.518732+0000","last_became_active":"2024-04-02T07:37:05.514220+0000","last_became_peered":"2024-04-02T07:37:05.514220+0000","last_unstale":"2024-04-02T07:37:05.518732+0000","last_undegraded":"2024-04-02T07:37:05.518732+0000","last_fullsized":"2024-04-02T07:37:05.518732+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:24:07.674982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.509996+0000","last_change":"2024-04-02T07:37:05.023645+0000","last_active":"2024-04-02T07:37:05.509996+0000","last_peered":"2024-04-02T07:37:05.509996+0000","last_clean":"2024-04-02T07:37:05.509996+0000","last_became_active":"2024-04-02T07:37:05.023325+0000","last_became_peered":"2024-04-02T07:37:05.023325+0000","last_unstale":"2024-04-02T07:37:05.509996+0000","last_undegraded":"2024-04-02T07:37:05.509996+0000","last_fullsized":"2024-04-02T07:37:05.509996+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:01:29.021837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.539792+0000","last_change":"2024-04-02T07:37:05.532872+0000","last_active":"2024-04-02T07:37:05.539792+0000","last_peered":"2024-04-02T07:37:05.539792+0000","last_clean":"2024-04-02T07:37:05.539792+0000","last_became_active":"2024-04-02T07:37:05.531426+0000","last_became_peered":"2024-04-02T07:37:05.531426+0000","last_unstale":"2024-04-02T07:37:05.539792+0000","last_undegraded":"2024-04-02T07:37:05.539792+0000","last_fullsized":"2024-04-02T07:37:05.539792+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:34:07.649125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.538844+0000","last_change":"2024-04-02T07:37:05.021987+0000","last_active":"2024-04-02T07:37:05.538844+0000","last_peered":"2024-04-02T07:37:05.538844+0000","last_clean":"2024-04-02T07:37:05.538844+0000","last_became_active":"2024-04-02T07:37:05.021723+0000","last_became_peered":"2024-04-02T07:37:05.021723+0000","last_unstale":"2024-04-02T07:37:05.538844+0000","last_undegraded":"2024-04-02T07:37:05.538844+0000","last_fullsized":"2024-04-02T07:37:05.538844+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:11.495078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.541252+0000","last_change":"2024-04-02T07:37:05.539477+0000","last_active":"2024-04-02T07:37:05.541252+0000","last_peered":"2024-04-02T07:37:05.541252+0000","last_clean":"2024-04-02T07:37:05.541252+0000","last_became_active":"2024-04-02T07:37:05.537930+0000","last_became_peered":"2024-04-02T07:37:05.537930+0000","last_unstale":"2024-04-02T07:37:05.541252+0000","last_undegraded":"2024-04-02T07:37:05.541252+0000","last_fullsized":"2024-04-02T07:37:05.541252+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:44:34.027889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.541231+0000","last_change":"2024-04-02T07:37:05.026559+0000","last_active":"2024-04-02T07:37:05.541231+0000","last_peered":"2024-04-02T07:37:05.541231+0000","last_clean":"2024-04-02T07:37:05.541231+0000","last_became_active":"2024-04-02T07:37:05.026297+0000","last_became_peered":"2024-04-02T07:37:05.026297+0000","last_unstale":"2024-04-02T07:37:05.541231+0000","last_undegraded":"2024-04-02T07:37:05.541231+0000","last_fullsized":"2024-04-02T07:37:05.541231+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:40:22.235662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.531040+0000","last_change":"2024-04-02T07:37:05.521401+0000","last_active":"2024-04-02T07:37:05.531040+0000","last_peered":"2024-04-02T07:37:05.531040+0000","last_clean":"2024-04-02T07:37:05.531040+0000","last_became_active":"2024-04-02T07:37:05.521155+0000","last_became_peered":"2024-04-02T07:37:05.521155+0000","last_unstale":"2024-04-02T07:37:05.531040+0000","last_undegraded":"2024-04-02T07:37:05.531040+0000","last_fullsized":"2024-04-02T07:37:05.531040+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:01:30.028293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.528745+0000","last_change":"2024-04-02T07:37:05.023509+0000","last_active":"2024-04-02T07:37:05.528745+0000","last_peered":"2024-04-02T07:37:05.528745+0000","last_clean":"2024-04-02T07:37:05.528745+0000","last_became_active":"2024-04-02T07:37:05.021664+0000","last_became_peered":"2024-04-02T07:37:05.021664+0000","last_unstale":"2024-04-02T07:37:05.528745+0000","last_undegraded":"2024-04-02T07:37:05.528745+0000","last_fullsized":"2024-04-02T07:37:05.528745+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:11.495078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.534839+0000","last_change":"2024-04-02T07:37:05.527849+0000","last_active":"2024-04-02T07:37:05.534839+0000","last_peered":"2024-04-02T07:37:05.534839+0000","last_clean":"2024-04-02T07:37:05.534839+0000","last_became_active":"2024-04-02T07:37:05.527496+0000","last_became_peered":"2024-04-02T07:37:05.527496+0000","last_unstale":"2024-04-02T07:37:05.534839+0000","last_undegraded":"2024-04-02T07:37:05.534839+0000","last_fullsized":"2024-04-02T07:37:05.534839+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:58:10.466063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.537993+0000","last_change":"2024-04-02T07:37:05.027202+0000","last_active":"2024-04-02T07:37:05.537993+0000","last_peered":"2024-04-02T07:37:05.537993+0000","last_clean":"2024-04-02T07:37:05.537993+0000","last_became_active":"2024-04-02T07:37:05.026903+0000","last_became_peered":"2024-04-02T07:37:05.026903+0000","last_unstale":"2024-04-02T07:37:05.537993+0000","last_undegraded":"2024-04-02T07:37:05.537993+0000","last_fullsized":"2024-04-02T07:37:05.537993+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:39.454907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.538199+0000","last_change":"2024-04-02T07:37:05.532573+0000","last_active":"2024-04-02T07:37:05.538199+0000","last_peered":"2024-04-02T07:37:05.538199+0000","last_clean":"2024-04-02T07:37:05.538199+0000","last_became_active":"2024-04-02T07:37:05.532298+0000","last_became_peered":"2024-04-02T07:37:05.532298+0000","last_unstale":"2024-04-02T07:37:05.538199+0000","last_undegraded":"2024-04-02T07:37:05.538199+0000","last_fullsized":"2024-04-02T07:37:05.538199+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:52:17.188396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.514949+0000","last_change":"2024-04-02T07:37:05.022359+0000","last_active":"2024-04-02T07:37:05.514949+0000","last_peered":"2024-04-02T07:37:05.514949+0000","last_clean":"2024-04-02T07:37:05.514949+0000","last_became_active":"2024-04-02T07:37:05.022107+0000","last_became_peered":"2024-04-02T07:37:05.022107+0000","last_unstale":"2024-04-02T07:37:05.514949+0000","last_undegraded":"2024-04-02T07:37:05.514949+0000","last_fullsized":"2024-04-02T07:37:05.514949+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:37:30.986204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.532878+0000","last_change":"2024-04-02T07:37:05.528586+0000","last_active":"2024-04-02T07:37:05.532878+0000","last_peered":"2024-04-02T07:37:05.532878+0000","last_clean":"2024-04-02T07:37:05.532878+0000","last_became_active":"2024-04-02T07:37:05.528281+0000","last_became_peered":"2024-04-02T07:37:05.528281+0000","last_unstale":"2024-04-02T07:37:05.532878+0000","last_undegraded":"2024-04-02T07:37:05.532878+0000","last_fullsized":"2024-04-02T07:37:05.532878+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:24:07.674982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.533309+0000","last_change":"2024-04-02T07:37:05.027533+0000","last_active":"2024-04-02T07:37:05.533309+0000","last_peered":"2024-04-02T07:37:05.533309+0000","last_clean":"2024-04-02T07:37:05.533309+0000","last_became_active":"2024-04-02T07:37:05.027198+0000","last_became_peered":"2024-04-02T07:37:05.027198+0000","last_unstale":"2024-04-02T07:37:05.533309+0000","last_undegraded":"2024-04-02T07:37:05.533309+0000","last_fullsized":"2024-04-02T07:37:05.533309+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:39:03.477899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.533446+0000","last_change":"2024-04-02T07:37:05.527861+0000","last_active":"2024-04-02T07:37:05.533446+0000","last_peered":"2024-04-02T07:37:05.533446+0000","last_clean":"2024-04-02T07:37:05.533446+0000","last_became_active":"2024-04-02T07:37:05.527546+0000","last_became_peered":"2024-04-02T07:37:05.527546+0000","last_unstale":"2024-04-02T07:37:05.533446+0000","last_undegraded":"2024-04-02T07:37:05.533446+0000","last_fullsized":"2024-04-02T07:37:05.533446+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:24:07.674982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.506853+0000","last_change":"2024-04-02T07:37:05.025925+0000","last_active":"2024-04-02T07:37:05.506853+0000","last_peered":"2024-04-02T07:37:05.506853+0000","last_clean":"2024-04-02T07:37:05.506853+0000","last_became_active":"2024-04-02T07:37:05.025567+0000","last_became_peered":"2024-04-02T07:37:05.025567+0000","last_unstale":"2024-04-02T07:37:05.506853+0000","last_undegraded":"2024-04-02T07:37:05.506853+0000","last_fullsized":"2024-04-02T07:37:05.506853+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:40:22.235662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.528019+0000","last_change":"2024-04-02T07:37:05.523275+0000","last_active":"2024-04-02T07:37:05.528019+0000","last_peered":"2024-04-02T07:37:05.528019+0000","last_clean":"2024-04-02T07:37:05.528019+0000","last_became_active":"2024-04-02T07:37:05.522194+0000","last_became_peered":"2024-04-02T07:37:05.522194+0000","last_unstale":"2024-04-02T07:37:05.528019+0000","last_undegraded":"2024-04-02T07:37:05.528019+0000","last_fullsized":"2024-04-02T07:37:05.528019+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:18.250998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.508932+0000","last_change":"2024-04-02T07:37:05.023186+0000","last_active":"2024-04-02T07:37:05.508932+0000","last_peered":"2024-04-02T07:37:05.508932+0000","last_clean":"2024-04-02T07:37:05.508932+0000","last_became_active":"2024-04-02T07:37:05.022937+0000","last_became_peered":"2024-04-02T07:37:05.022937+0000","last_unstale":"2024-04-02T07:37:05.508932+0000","last_undegraded":"2024-04-02T07:37:05.508932+0000","last_fullsized":"2024-04-02T07:37:05.508932+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:12:03.425485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.531139+0000","last_change":"2024-04-02T07:37:05.522102+0000","last_active":"2024-04-02T07:37:05.531139+0000","last_peered":"2024-04-02T07:37:05.531139+0000","last_clean":"2024-04-02T07:37:05.531139+0000","last_became_active":"2024-04-02T07:37:05.521891+0000","last_became_peered":"2024-04-02T07:37:05.521891+0000","last_unstale":"2024-04-02T07:37:05.531139+0000","last_undegraded":"2024-04-02T07:37:05.531139+0000","last_fullsized":"2024-04-02T07:37:05.531139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:24:25.160946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.516317+0000","last_change":"2024-04-02T07:37:05.027133+0000","last_active":"2024-04-02T07:37:05.516317+0000","last_peered":"2024-04-02T07:37:05.516317+0000","last_clean":"2024-04-02T07:37:05.516317+0000","last_became_active":"2024-04-02T07:37:05.026860+0000","last_became_peered":"2024-04-02T07:37:05.026860+0000","last_unstale":"2024-04-02T07:37:05.516317+0000","last_undegraded":"2024-04-02T07:37:05.516317+0000","last_fullsized":"2024-04-02T07:37:05.516317+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:15.234166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.539601+0000","last_change":"2024-04-02T07:37:05.536066+0000","last_active":"2024-04-02T07:37:05.539601+0000","last_peered":"2024-04-02T07:37:05.539601+0000","last_clean":"2024-04-02T07:37:05.539601+0000","last_became_active":"2024-04-02T07:37:05.534329+0000","last_became_peered":"2024-04-02T07:37:05.534329+0000","last_unstale":"2024-04-02T07:37:05.539601+0000","last_undegraded":"2024-04-02T07:37:05.539601+0000","last_fullsized":"2024-04-02T07:37:05.539601+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:39:04.484355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.530573+0000","last_change":"2024-04-02T07:37:05.027585+0000","last_active":"2024-04-02T07:37:05.530573+0000","last_peered":"2024-04-02T07:37:05.530573+0000","last_clean":"2024-04-02T07:37:05.530573+0000","last_became_active":"2024-04-02T07:37:05.025889+0000","last_became_peered":"2024-04-02T07:37:05.025889+0000","last_unstale":"2024-04-02T07:37:05.530573+0000","last_undegraded":"2024-04-02T07:37:05.530573+0000","last_fullsized":"2024-04-02T07:37:05.530573+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:36.516421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.541206+0000","last_change":"2024-04-02T07:37:05.539090+0000","last_active":"2024-04-02T07:37:05.541206+0000","last_peered":"2024-04-02T07:37:05.541206+0000","last_clean":"2024-04-02T07:37:05.541206+0000","last_became_active":"2024-04-02T07:37:05.537072+0000","last_became_peered":"2024-04-02T07:37:05.537072+0000","last_unstale":"2024-04-02T07:37:05.541206+0000","last_undegraded":"2024-04-02T07:37:05.541206+0000","last_fullsized":"2024-04-02T07:37:05.541206+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:59:52.962129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.513130+0000","last_change":"2024-04-02T07:37:05.022270+0000","last_active":"2024-04-02T07:37:05.513130+0000","last_peered":"2024-04-02T07:37:05.513130+0000","last_clean":"2024-04-02T07:37:05.513130+0000","last_became_active":"2024-04-02T07:37:05.018873+0000","last_became_peered":"2024-04-02T07:37:05.018873+0000","last_unstale":"2024-04-02T07:37:05.513130+0000","last_undegraded":"2024-04-02T07:37:05.513130+0000","last_fullsized":"2024-04-02T07:37:05.513130+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:52:16.181939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540254+0000","last_change":"2024-04-02T07:37:05.538119+0000","last_active":"2024-04-02T07:37:05.540254+0000","last_peered":"2024-04-02T07:37:05.540254+0000","last_clean":"2024-04-02T07:37:05.540254+0000","last_became_active":"2024-04-02T07:37:05.537340+0000","last_became_peered":"2024-04-02T07:37:05.537340+0000","last_unstale":"2024-04-02T07:37:05.540254+0000","last_undegraded":"2024-04-02T07:37:05.540254+0000","last_fullsized":"2024-04-02T07:37:05.540254+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:15.644342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.519223+0000","last_change":"2024-04-02T07:37:05.029711+0000","last_active":"2024-04-02T07:37:05.519223+0000","last_peered":"2024-04-02T07:37:05.519223+0000","last_clean":"2024-04-02T07:37:05.519223+0000","last_became_active":"2024-04-02T07:37:05.029366+0000","last_became_peered":"2024-04-02T07:37:05.029366+0000","last_unstale":"2024-04-02T07:37:05.519223+0000","last_undegraded":"2024-04-02T07:37:05.519223+0000","last_fullsized":"2024-04-02T07:37:05.519223+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:40:22.235662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540471+0000","last_change":"2024-04-02T07:37:05.537247+0000","last_active":"2024-04-02T07:37:05.540471+0000","last_peered":"2024-04-02T07:37:05.540471+0000","last_clean":"2024-04-02T07:37:05.540471+0000","last_became_active":"2024-04-02T07:37:05.537015+0000","last_became_peered":"2024-04-02T07:37:05.537015+0000","last_unstale":"2024-04-02T07:37:05.540471+0000","last_undegraded":"2024-04-02T07:37:05.540471+0000","last_fullsized":"2024-04-02T07:37:05.540471+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:51:55.529401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.518389+0000","last_change":"2024-04-02T07:37:05.024933+0000","last_active":"2024-04-02T07:37:05.518389+0000","last_peered":"2024-04-02T07:37:05.518389+0000","last_clean":"2024-04-02T07:37:05.518389+0000","last_became_active":"2024-04-02T07:37:05.024623+0000","last_became_peered":"2024-04-02T07:37:05.024623+0000","last_unstale":"2024-04-02T07:37:05.518389+0000","last_undegraded":"2024-04-02T07:37:05.518389+0000","last_fullsized":"2024-04-02T07:37:05.518389+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:35:23.779994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540866+0000","last_change":"2024-04-02T07:37:05.534896+0000","last_active":"2024-04-02T07:37:05.540866+0000","last_peered":"2024-04-02T07:37:05.540866+0000","last_clean":"2024-04-02T07:37:05.540866+0000","last_became_active":"2024-04-02T07:37:05.531154+0000","last_became_peered":"2024-04-02T07:37:05.531154+0000","last_unstale":"2024-04-02T07:37:05.540866+0000","last_undegraded":"2024-04-02T07:37:05.540866+0000","last_fullsized":"2024-04-02T07:37:05.540866+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:51:55.529401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.518067+0000","last_change":"2024-04-02T07:37:05.025479+0000","last_active":"2024-04-02T07:37:05.518067+0000","last_peered":"2024-04-02T07:37:05.518067+0000","last_clean":"2024-04-02T07:37:05.518067+0000","last_became_active":"2024-04-02T07:37:05.025204+0000","last_became_peered":"2024-04-02T07:37:05.025204+0000","last_unstale":"2024-04-02T07:37:05.518067+0000","last_undegraded":"2024-04-02T07:37:05.518067+0000","last_fullsized":"2024-04-02T07:37:05.518067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:13:59.421974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.536183+0000","last_change":"2024-04-02T07:37:05.524694+0000","last_active":"2024-04-02T07:37:05.536183+0000","last_peered":"2024-04-02T07:37:05.536183+0000","last_clean":"2024-04-02T07:37:05.536183+0000","last_became_active":"2024-04-02T07:37:05.524428+0000","last_became_peered":"2024-04-02T07:37:05.524428+0000","last_unstale":"2024-04-02T07:37:05.536183+0000","last_undegraded":"2024-04-02T07:37:05.536183+0000","last_fullsized":"2024-04-02T07:37:05.536183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:26:44.688366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.506443+0000","last_change":"2024-04-02T07:37:05.024567+0000","last_active":"2024-04-02T07:37:05.506443+0000","last_peered":"2024-04-02T07:37:05.506443+0000","last_clean":"2024-04-02T07:37:05.506443+0000","last_became_active":"2024-04-02T07:37:05.024250+0000","last_became_peered":"2024-04-02T07:37:05.024250+0000","last_unstale":"2024-04-02T07:37:05.506443+0000","last_undegraded":"2024-04-02T07:37:05.506443+0000","last_fullsized":"2024-04-02T07:37:05.506443+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:13:59.421974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.541098+0000","last_change":"2024-04-02T07:37:05.536764+0000","last_active":"2024-04-02T07:37:05.541098+0000","last_peered":"2024-04-02T07:37:05.541098+0000","last_clean":"2024-04-02T07:37:05.541098+0000","last_became_active":"2024-04-02T07:37:05.536505+0000","last_became_peered":"2024-04-02T07:37:05.536505+0000","last_unstale":"2024-04-02T07:37:05.541098+0000","last_undegraded":"2024-04-02T07:37:05.541098+0000","last_fullsized":"2024-04-02T07:37:05.541098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:43:39.130383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.509905+0000","last_change":"2024-04-02T07:37:05.025098+0000","last_active":"2024-04-02T07:37:05.509905+0000","last_peered":"2024-04-02T07:37:05.509905+0000","last_clean":"2024-04-02T07:37:05.509905+0000","last_became_active":"2024-04-02T07:37:05.024084+0000","last_became_peered":"2024-04-02T07:37:05.024084+0000","last_unstale":"2024-04-02T07:37:05.509905+0000","last_undegraded":"2024-04-02T07:37:05.509905+0000","last_fullsized":"2024-04-02T07:37:05.509905+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:36.516421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.540044+0000","last_change":"2024-04-02T07:37:05.535894+0000","last_active":"2024-04-02T07:37:05.540044+0000","last_peered":"2024-04-02T07:37:05.540044+0000","last_clean":"2024-04-02T07:37:05.540044+0000","last_became_active":"2024-04-02T07:37:05.535319+0000","last_became_peered":"2024-04-02T07:37:05.535319+0000","last_unstale":"2024-04-02T07:37:05.540044+0000","last_undegraded":"2024-04-02T07:37:05.540044+0000","last_fullsized":"2024-04-02T07:37:05.540044+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:04.994109+0000","last_clean_scrub_stamp":"2024-04-02T07:37:04.994109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:30:42.804445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.519289+0000","last_change":"2024-04-02T07:37:05.026106+0000","last_active":"2024-04-02T07:37:05.519289+0000","last_peered":"2024-04-02T07:37:05.519289+0000","last_clean":"2024-04-02T07:37:05.519289+0000","last_became_active":"2024-04-02T07:37:05.025886+0000","last_became_peered":"2024-04-02T07:37:05.025886+0000","last_unstale":"2024-04-02T07:37:05.519289+0000","last_undegraded":"2024-04-02T07:37:05.519289+0000","last_fullsized":"2024-04-02T07:37:05.519289+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:37:03.987653+0000","last_clean_scrub_stamp":"2024-04-02T07:37:03.987653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:58:09.459607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":175,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-02T07:37:05.021994+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:37:05.021994+0000","last_peered":"2024-04-02T07:37:05.021994+0000","last_clean":"2024-04-02T07:37:05.021994+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:37:05.021994+0000","last_undegraded":"2024-04-02T07:37:05.021994+0000","last_fullsized":"2024-04-02T07:37:05.021994+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":376832,"data_stored":145123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.65500000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.63100000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.61899999999999999}]},{"osd":3,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.58599999999999997}]},{"osd":4,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.66800000000000004}]},{"osd":5,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.59899999999999998}]},{"osd":6,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.51200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":385024,"data_stored":146665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.40400000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.74199999999999999}]},{"osd":2,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.50800000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.53000000000000003}]},{"osd":4,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.57099999999999995}]},{"osd":5,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.82499999999999996}]},{"osd":7,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.63500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":954368,"data_stored":728694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.9079999999999999,"5min":2.9079999999999999,"15min":2.9079999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":2.766,"5min":2.766,"15min":2.766},"last":0.435}]},{"osd":2,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":2.0750000000000002,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":0.45800000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.9430000000000001,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":0.503},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":2.802,"5min":2.802,"15min":2.802},"last":0.36199999999999999}]},{"osd":4,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.65400000000000003}]},{"osd":5,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.63900000000000001}]},{"osd":6,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.012,"5min":2.012,"15min":2.012},"last":0.59399999999999997}]},{"osd":7,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":0.746},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.629}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":372736,"data_stored":144172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.39000000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":2.605,"5min":2.605,"15min":2.605},"last":0.35099999999999998}]},{"osd":3,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":5.4800000000000004,"5min":5.4800000000000004,"15min":5.4800000000000004},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":2.194,"5min":2.194,"15min":2.194},"last":0.61599999999999999}]},{"osd":4,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":6.0899999999999999,"5min":6.0899999999999999,"15min":6.0899999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":5.7169999999999996,"5min":5.7169999999999996,"15min":5.7169999999999996},"last":0.60599999999999998}]},{"osd":5,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":5.7539999999999996,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":5.8929999999999998,"5min":5.8929999999999998,"15min":5.8929999999999998},"last":0.58699999999999997}]},{"osd":6,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":5.931,"5min":5.931,"15min":5.931},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":6.125,"5min":6.125,"15min":6.125},"last":0.68999999999999995}]},{"osd":7,"last update":"Tue Apr 2 07:36:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":6.1580000000000004,"5min":6.1580000000000004,"15min":6.1580000000000004},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":5.7850000000000001,"5min":5.7850000000000001,"15min":5.7850000000000001},"last":0.747}]}]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":376832,"data_stored":145135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.379},{"interface":"front","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.36499999999999999}]},{"osd":1,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.41399999999999998}]},{"osd":3,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.51400000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.94199999999999995}]},{"osd":5,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.91800000000000004}]},{"osd":6,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.96599999999999997}]},{"osd":7,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.81100000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":145877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.476,"5min":2.476,"15min":2.476},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":0.38100000000000001}]},{"osd":1,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":2.2839999999999998,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.2240000000000002,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.49199999999999999}]},{"osd":2,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.9449999999999998,"5min":3.9449999999999998,"15min":3.9449999999999998},"last":0.44}]},{"osd":4,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.68899999999999995}]},{"osd":5,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":0.66600000000000004}]},{"osd":6,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.79900000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.966,"5min":1.966,"15min":1.966},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.71499999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":146687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.77200000000000002}]},{"osd":1,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.63},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.75700000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.71299999999999997}]},{"osd":3,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.59799999999999998}]},{"osd":5,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.65600000000000003}]},{"osd":6,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.80400000000000005}]},{"osd":7,"last update":"Tue Apr 2 07:36:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.74199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":950272,"data_stored":727675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.623}]},{"osd":1,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.64300000000000002}]},{"osd":2,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.73699999999999999}]},{"osd":3,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.89900000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.85199999999999998}]},{"osd":6,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.92400000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:36:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.98299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:37:10.584 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T07:37:10.584 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T07:37:10.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.816+0000 7fa8c8a00700 1 Processor -- start 2024-04-02T07:37:10.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.816+0000 7fa8c8a00700 1 -- start start 2024-04-02T07:37:10.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.817+0000 7fa8c8a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8c40af600 0x7fa8c406a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:10.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.817+0000 7fa8c8a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c40aeb60 con 0x7fa8c40afa60 2024-04-02T07:37:10.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.817+0000 7fa8c8a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c40b5190 con 0x7fa8c4145050 2024-04-02T07:37:10.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.817+0000 7fa8c8a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40aee00 con 0x7fa8c40af600 2024-04-02T07:37:10.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.817+0000 7fa8c2800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa8c40afa60 0x7fa8c40b3200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40120/0 (socket says 172.21.15.89:40120) 2024-04-02T07:37:10.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c2800700 1 -- 172.21.15.89:0/1876753739 learned_addr learned my addr 172.21.15.89:0/1876753739 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:10.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c1400700 1 --2- 172.21.15.89:0/1876753739 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8c40af600 0x7fa8c406a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:10.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 -- 172.21.15.89:0/1876753739 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542803831 0 0) 0x7fa8c40aeb60 con 0x7fa8c40afa60 2024-04-02T07:37:10.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 -- 172.21.15.89:0/1876753739 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a80035b0 con 0x7fa8c40afa60 2024-04-02T07:37:10.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 -- 172.21.15.89:0/1876753739 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3016619508 0 0) 0x7fa8a80035b0 con 0x7fa8c40afa60 2024-04-02T07:37:10.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 -- 172.21.15.89:0/1876753739 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8c40af600 msgr2=0x7fa8c406a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:10.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 --2- 172.21.15.89:0/1876753739 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8c40af600 0x7fa8c406a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 -- 172.21.15.89:0/1876753739 >> v1:172.21.15.113:6789/0 conn(0x7fa8c4145050 legacy=0x7fa8c40d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:10.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 -- 172.21.15.89:0/1876753739 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c40af0e0 con 0x7fa8c40afa60 2024-04-02T07:37:10.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 -- 172.21.15.89:0/1876753739 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fa8bc0019f0 con 0x7fa8c40afa60 2024-04-02T07:37:10.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 -- 172.21.15.89:0/1876753739 >> v1:172.21.15.89:6789/0 conn(0x7fa8c40afa60 legacy=0x7fa8c40b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:10.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 --2- 172.21.15.89:0/1876753739 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8c40af600 0x7fa8a8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:10.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 --2- 172.21.15.89:0/1876753739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8a8003fb0 0x7fa8a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:10.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 --2- 172.21.15.89:0/1876753739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8a8004d90 0x7fa8a8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:10.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 -- 172.21.15.89:0/1876753739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40aee00 con 0x7fa8a8003fb0 2024-04-02T07:37:10.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 -- 172.21.15.89:0/1876753739 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a800bba0 con 0x7fa8a8004d90 2024-04-02T07:37:10.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.818+0000 7fa8c0a00700 1 -- 172.21.15.89:0/1876753739 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a800bd60 con 0x7fa8c40af600 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.819+0000 7fa8c1400700 1 --2- 172.21.15.89:0/1876753739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8a8003fb0 0x7fa8a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.819+0000 7fa8c1e00700 1 --2- 172.21.15.89:0/1876753739 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8c40af600 0x7fa8a8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.819+0000 7fa8c2800700 1 --2- 172.21.15.89:0/1876753739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8a8004d90 0x7fa8a8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.819+0000 7fa8c1400700 1 -- 172.21.15.89:0/1876753739 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8c40af600 msgr2=0x7fa8a8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.819+0000 7fa8c1400700 1 --2- 172.21.15.89:0/1876753739 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8c40af600 0x7fa8a8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.819+0000 7fa8c1400700 1 -- 172.21.15.89:0/1876753739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8a8004d90 msgr2=0x7fa8a8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.819+0000 7fa8c1400700 1 --2- 172.21.15.89:0/1876753739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8a8004d90 0x7fa8a8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.819+0000 7fa8c1400700 1 -- 172.21.15.89:0/1876753739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a80035b0 con 0x7fa8a8003fb0 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.819+0000 7fa8c1400700 1 --2- 172.21.15.89:0/1876753739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8a8003fb0 0x7fa8a8004850 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0009b60 tx=0x7fa8a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5bef2e53f81750d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.819+0000 7fa8c0a00700 1 -- 172.21.15.89:0/1876753739 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b000e6c0 con 0x7fa8a8003fb0 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.819+0000 7fa8c0a00700 1 -- 172.21.15.89:0/1876753739 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b000e820 con 0x7fa8a8003fb0 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.819+0000 7fa8c0a00700 1 -- 172.21.15.89:0/1876753739 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b000eb90 con 0x7fa8a8003fb0 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.820+0000 7fa8c8a00700 1 -- 172.21.15.89:0/1876753739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8a8003fb0 msgr2=0x7fa8a8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.820+0000 7fa8c8a00700 1 --2- 172.21.15.89:0/1876753739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8a8003fb0 0x7fa8a8004850 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0009b60 tx=0x7fa8a8005880 comp rx=0 tx=0).stop 2024-04-02T07:37:10.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.820+0000 7fa8c2800700 1 -- 172.21.15.89:0/1876753739 reap_dead start 2024-04-02T07:37:10.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.820+0000 7fa8c8a00700 1 -- 172.21.15.89:0/1876753739 shutdown_connections 2024-04-02T07:37:10.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.820+0000 7fa8c8a00700 1 -- 172.21.15.89:0/1876753739 >> 172.21.15.89:0/1876753739 conn(0x7fa8c40b5720 msgr2=0x7fa8c40b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:10.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.820+0000 7fa8c8a00700 1 -- 172.21.15.89:0/1876753739 shutdown_connections 2024-04-02T07:37:10.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.820+0000 7fa8c8a00700 1 -- 172.21.15.89:0/1876753739 wait complete. 2024-04-02T07:37:10.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.822+0000 7fa8c8a00700 1 Processor -- start 2024-04-02T07:37:10.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.823+0000 7fa8c8a00700 1 -- start start 2024-04-02T07:37:10.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.824+0000 7fa8c8a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8a8003fb0 0x7fa8c41fa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:10.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.824+0000 7fa8c8a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8c4143330 0x7fa8c41437b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:10.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.824+0000 7fa8c8a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c40b0810 0x7fa8c40b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:10.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.824+0000 7fa8c8a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c4143cf0 con 0x7fa8c40b0810 2024-04-02T07:37:10.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.824+0000 7fa8c8a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c41f7620 con 0x7fa8a8003fb0 2024-04-02T07:37:10.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.824+0000 7fa8c8a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c41f7900 con 0x7fa8c4143330 2024-04-02T07:37:10.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.824+0000 7fa8c2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c40b0810 0x7fa8c40b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.824+0000 7fa8c2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c40b0810 0x7fa8c40b0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58070/0 (socket says 172.21.15.89:58070) 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.824+0000 7fa8c2800700 1 -- 172.21.15.89:0/2274944938 learned_addr learned my addr 172.21.15.89:0/2274944938 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.824+0000 7fa8c1e00700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8a8003fb0 0x7fa8c41fa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.824+0000 7fa8c1400700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8c4143330 0x7fa8c41437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.825+0000 7fa8c2800700 1 -- 172.21.15.89:0/2274944938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8c4143330 msgr2=0x7fa8c41437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.825+0000 7fa8c2800700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8c4143330 0x7fa8c41437b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.825+0000 7fa8c2800700 1 -- 172.21.15.89:0/2274944938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8a8003fb0 msgr2=0x7fa8c41fa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.825+0000 7fa8c2800700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8a8003fb0 0x7fa8c41fa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.825+0000 7fa8c2800700 1 -- 172.21.15.89:0/2274944938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c41f7be0 con 0x7fa8c40b0810 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.825+0000 7fa8c1400700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8c4143330 0x7fa8c41437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.825+0000 7fa8c2800700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c40b0810 0x7fa8c40b0c90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40cff70 tx=0x7fa8bc007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.825+0000 7fa8ba000700 1 -- 172.21.15.89:0/2274944938 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc00ec30 con 0x7fa8c40b0810 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.825+0000 7fa8ba000700 1 -- 172.21.15.89:0/2274944938 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8bc00ed90 con 0x7fa8c40b0810 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.825+0000 7fa8c8a00700 1 -- 172.21.15.89:0/2274944938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c41f7ec0 con 0x7fa8c40b0810 2024-04-02T07:37:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.825+0000 7fa8ba000700 1 -- 172.21.15.89:0/2274944938 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc013660 con 0x7fa8c40b0810 2024-04-02T07:37:10.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.825+0000 7fa8c8a00700 1 -- 172.21.15.89:0/2274944938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c41fa710 con 0x7fa8c40b0810 2024-04-02T07:37:10.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.826+0000 7fa8c8a00700 1 -- 172.21.15.89:0/2274944938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c41f83e0 con 0x7fa8c40b0810 2024-04-02T07:37:10.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.827+0000 7fa8ba000700 1 -- 172.21.15.89:0/2274944938 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa8bc0137c0 con 0x7fa8c40b0810 2024-04-02T07:37:10.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.828+0000 7fa8ba000700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa8a406ac90 0x7fa8a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:10.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.828+0000 7fa8ba000700 1 -- 172.21.15.89:0/2274944938 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8bc0899c0 con 0x7fa8c40b0810 2024-04-02T07:37:10.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.828+0000 7fa8c1e00700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa8a406ac90 0x7fa8a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:10.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.829+0000 7fa8c1e00700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa8a406ac90 0x7fa8a406d150 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4007fd0 tx=0x7fa8b4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:10.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:10.832+0000 7fa8ba000700 1 -- 172.21.15.89:0/2274944938 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa8c41f83e0 con 0x7fa8c40b0810 2024-04-02T07:37:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.116+0000 7fa8c8a00700 1 -- 172.21.15.89:0/2274944938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa8c41f83e0 con 0x7fa8c40b0810 2024-04-02T07:37:11.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.121+0000 7fa8ba000700 1 -- 172.21.15.89:0/2274944938 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+0 (secure 0 0 0) 0x7fa8c41f83e0 con 0x7fa8c40b0810 2024-04-02T07:37:11.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.123+0000 7fa8c8a00700 1 -- 172.21.15.89:0/2274944938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa8a406ac90 msgr2=0x7fa8a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:11.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.123+0000 7fa8c8a00700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa8a406ac90 0x7fa8a406d150 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4007fd0 tx=0x7fa8b4009040 comp rx=0 tx=0).stop 2024-04-02T07:37:11.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.123+0000 7fa8c8a00700 1 -- 172.21.15.89:0/2274944938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c40b0810 msgr2=0x7fa8c40b0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:11.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.123+0000 7fa8c8a00700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c40b0810 0x7fa8c40b0c90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40cff70 tx=0x7fa8bc007c90 comp rx=0 tx=0).stop 2024-04-02T07:37:11.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.123+0000 7fa8c8a00700 1 -- 172.21.15.89:0/2274944938 shutdown_connections 2024-04-02T07:37:11.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.123+0000 7fa8c8a00700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa8a406ac90 0x7fa8a406d150 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:11.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.123+0000 7fa8c8a00700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8a8003fb0 0x7fa8c41fa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:11.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.123+0000 7fa8c8a00700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8c40b0810 0x7fa8c40b0c90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:11.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.123+0000 7fa8c8a00700 1 --2- 172.21.15.89:0/2274944938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8c4143330 0x7fa8c41437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:11.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.123+0000 7fa8c8a00700 1 -- 172.21.15.89:0/2274944938 >> 172.21.15.89:0/2274944938 conn(0x7fa8c40b5720 msgr2=0x7fa8c413c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:11.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.124+0000 7fa8c8a00700 1 -- 172.21.15.89:0/2274944938 shutdown_connections 2024-04-02T07:37:11.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.124+0000 7fa8c8a00700 1 -- 172.21.15.89:0/2274944938 wait complete. 2024-04-02T07:37:11.127 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T07:37:11.143 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:37:11.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.334+0000 7ff6cbe00700 1 Processor -- start 2024-04-02T07:37:11.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.334+0000 7ff6cbe00700 1 -- start start 2024-04-02T07:37:11.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.335+0000 7ff6cbe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6c40d18b0 0x7ff6c40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:11.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.335+0000 7ff6cbe00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c40ceda0 con 0x7ff6c4161590 2024-04-02T07:37:11.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.335+0000 7ff6cbe00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c41612f0 con 0x7ff6c4160f10 2024-04-02T07:37:11.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.335+0000 7ff6cbe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c40d1d30 con 0x7ff6c40d18b0 2024-04-02T07:37:11.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.335+0000 7ff6c9e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff6c4161590 0x7ff6c40b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40130/0 (socket says 172.21.15.89:40130) 2024-04-02T07:37:11.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.335+0000 7ff6c9e00700 1 -- 172.21.15.89:0/3256641616 learned_addr learned my addr 172.21.15.89:0/3256641616 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:11.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.335+0000 7ff6c8a00700 1 --2- 172.21.15.89:0/3256641616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6c40d18b0 0x7ff6c40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:11.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129428667 0 0) 0x7ff6c40ceda0 con 0x7ff6c4161590 2024-04-02T07:37:11.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6ac0035b0 con 0x7ff6c4161590 2024-04-02T07:37:11.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588273195 0 0) 0x7ff6c41612f0 con 0x7ff6c4160f10 2024-04-02T07:37:11.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c40ceda0 con 0x7ff6c4160f10 2024-04-02T07:37:11.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3780363469 0 0) 0x7ff6ac0035b0 con 0x7ff6c4161590 2024-04-02T07:37:11.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6c40d18b0 msgr2=0x7ff6c40cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:11.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 --2- 172.21.15.89:0/3256641616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6c40d18b0 0x7ff6c40cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:11.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 >> v1:172.21.15.113:6789/0 conn(0x7ff6c4160f10 legacy=0x7ff6c40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:11.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c40ce470 con 0x7ff6c4161590 2024-04-02T07:37:11.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7ff6bc0019f0 con 0x7ff6c4161590 2024-04-02T07:37:11.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 >> v1:172.21.15.89:6789/0 conn(0x7ff6c4161590 legacy=0x7ff6c40b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:11.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 --2- 172.21.15.89:0/3256641616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6ac003bc0 0x7ff6ac003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:11.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 --2- 172.21.15.89:0/3256641616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6c40d18b0 0x7ff6ac004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:11.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 --2- 172.21.15.89:0/3256641616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6ac004d90 0x7ff6ac005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:11.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c40d1d30 con 0x7ff6ac003bc0 2024-04-02T07:37:11.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ac00bb10 con 0x7ff6ac004d90 2024-04-02T07:37:11.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ac00bcd0 con 0x7ff6c40d18b0 2024-04-02T07:37:11.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.336+0000 7ff6c9400700 1 --2- 172.21.15.89:0/3256641616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6ac003bc0 0x7ff6ac003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:11.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.337+0000 7ff6c8a00700 1 --2- 172.21.15.89:0/3256641616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6c40d18b0 0x7ff6ac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:11.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.337+0000 7ff6c9e00700 1 --2- 172.21.15.89:0/3256641616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6ac004d90 0x7ff6ac005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:11.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.337+0000 7ff6c9400700 1 -- 172.21.15.89:0/3256641616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6c40d18b0 msgr2=0x7ff6ac004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:11.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.337+0000 7ff6c9400700 1 --2- 172.21.15.89:0/3256641616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6c40d18b0 0x7ff6ac004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:11.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.337+0000 7ff6c9400700 1 -- 172.21.15.89:0/3256641616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6ac004d90 msgr2=0x7ff6ac005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:11.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.337+0000 7ff6c9400700 1 --2- 172.21.15.89:0/3256641616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6ac004d90 0x7ff6ac005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:11.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.337+0000 7ff6c9400700 1 -- 172.21.15.89:0/3256641616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ac0035b0 con 0x7ff6ac003bc0 2024-04-02T07:37:11.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.337+0000 7ff6c9400700 1 --2- 172.21.15.89:0/3256641616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6ac003bc0 0x7ff6ac003fc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8001940 tx=0x7ff6ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a213016e949409 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:11.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.337+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8001430 con 0x7ff6ac003bc0 2024-04-02T07:37:11.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.337+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b8001d70 con 0x7ff6ac003bc0 2024-04-02T07:37:11.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.337+0000 7ff6c3e00700 1 -- 172.21.15.89:0/3256641616 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8005510 con 0x7ff6ac003bc0 2024-04-02T07:37:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.338+0000 7ff6cbe00700 1 -- 172.21.15.89:0/3256641616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6ac003bc0 msgr2=0x7ff6ac003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.338+0000 7ff6cbe00700 1 --2- 172.21.15.89:0/3256641616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6ac003bc0 0x7ff6ac003fc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8001940 tx=0x7ff6ac005880 comp rx=0 tx=0).stop 2024-04-02T07:37:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.338+0000 7ff6c9e00700 1 -- 172.21.15.89:0/3256641616 reap_dead start 2024-04-02T07:37:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.338+0000 7ff6cbe00700 1 -- 172.21.15.89:0/3256641616 shutdown_connections 2024-04-02T07:37:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.338+0000 7ff6cbe00700 1 -- 172.21.15.89:0/3256641616 >> 172.21.15.89:0/3256641616 conn(0x7ff6c408aff0 msgr2=0x7ff6c40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.338+0000 7ff6cbe00700 1 -- 172.21.15.89:0/3256641616 shutdown_connections 2024-04-02T07:37:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.338+0000 7ff6cbe00700 1 -- 172.21.15.89:0/3256641616 wait complete. 2024-04-02T07:37:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.340+0000 7ff6cbe00700 1 Processor -- start 2024-04-02T07:37:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.340+0000 7ff6cbe00700 1 -- start start 2024-04-02T07:37:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6cbe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6ac003bc0 0x7ff6c4156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:11.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6cbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c415e6c0 0x7ff6c415eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:11.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6cbe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6c414b2e0 0x7ff6c414b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:11.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6cbe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c415fda0 con 0x7ff6c415e6c0 2024-04-02T07:37:11.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6cbe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c415f120 con 0x7ff6c414b2e0 2024-04-02T07:37:11.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6cbe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c415f400 con 0x7ff6ac003bc0 2024-04-02T07:37:11.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6c8a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c415e6c0 0x7ff6c415eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:11.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6c8a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c415e6c0 0x7ff6c415eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58094/0 (socket says 172.21.15.89:58094) 2024-04-02T07:37:11.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6c8a00700 1 -- 172.21.15.89:0/471963511 learned_addr learned my addr 172.21.15.89:0/471963511 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:11.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6c9400700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6ac003bc0 0x7ff6c4156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6c9e00700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6c414b2e0 0x7ff6c414b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6c8a00700 1 -- 172.21.15.89:0/471963511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6ac003bc0 msgr2=0x7ff6c4156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6c8a00700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6ac003bc0 0x7ff6c4156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6c8a00700 1 -- 172.21.15.89:0/471963511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6c414b2e0 msgr2=0x7ff6c414b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6c8a00700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6c414b2e0 0x7ff6c414b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6c8a00700 1 -- 172.21.15.89:0/471963511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c415f6e0 con 0x7ff6c415e6c0 2024-04-02T07:37:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.341+0000 7ff6c9e00700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6c414b2e0 0x7ff6c414b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.342+0000 7ff6c8a00700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c415e6c0 0x7ff6c415eb40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0011aa0 tx=0x7ff6b0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.342+0000 7ff6c1600700 1 -- 172.21.15.89:0/471963511 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b000f980 con 0x7ff6c415e6c0 2024-04-02T07:37:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.342+0000 7ff6c1600700 1 -- 172.21.15.89:0/471963511 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b000fae0 con 0x7ff6c415e6c0 2024-04-02T07:37:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.342+0000 7ff6c1600700 1 -- 172.21.15.89:0/471963511 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b000fe90 con 0x7ff6c415e6c0 2024-04-02T07:37:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.342+0000 7ff6cbe00700 1 -- 172.21.15.89:0/471963511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c415f9c0 con 0x7ff6c415e6c0 2024-04-02T07:37:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.342+0000 7ff6cbe00700 1 -- 172.21.15.89:0/471963511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c41568c0 con 0x7ff6c415e6c0 2024-04-02T07:37:11.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.343+0000 7ff6cbe00700 1 -- 172.21.15.89:0/471963511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c4067720 con 0x7ff6c415e6c0 2024-04-02T07:37:11.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.344+0000 7ff6c1600700 1 -- 172.21.15.89:0/471963511 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff6b0014440 con 0x7ff6c415e6c0 2024-04-02T07:37:11.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.345+0000 7ff6c1600700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff6a406ac90 0x7ff6a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:11.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.345+0000 7ff6c1600700 1 -- 172.21.15.89:0/471963511 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6b0089be0 con 0x7ff6c415e6c0 2024-04-02T07:37:11.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.345+0000 7ff6c9400700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff6a406ac90 0x7ff6a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:11.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.346+0000 7ff6c9400700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff6a406ac90 0x7ff6a406d150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8015040 tx=0x7ff6b800a000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:11.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.349+0000 7ff6c1600700 1 -- 172.21.15.89:0/471963511 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff6b0054d50 con 0x7ff6c415e6c0 2024-04-02T07:37:11.579 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:37:11.584 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:37:11.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.597+0000 7ff6cbe00700 1 -- 172.21.15.89:0/471963511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6c408b760 con 0x7ff6c415e6c0 2024-04-02T07:37:11.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.598+0000 7ff6c1600700 1 -- 172.21.15.89:0/471963511 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4737 (secure 0 0 0) 0x7ff6b0022070 con 0x7ff6c415e6c0 2024-04-02T07:37:11.600 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:37:11.600 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5037,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3788023013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3788023013},{"type":"v1","addr":"172.21.15.89:6835","nonce":3788023013}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5040,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4029308904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":4029308904},{"type":"v1","addr":"172.21.15.113:6835","nonce":4029308904}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5043,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2115217020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2115217020},{"type":"v1","addr":"172.21.15.113:6834","nonce":2115217020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:37:05.500422+0000","modified":"2024-04-02T07:37:08.737868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.89:6837/929002087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":929002087},{"type":"v1","addr":"172.21.15.89:6837","nonce":929002087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-02T07:37:11.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.600+0000 7ff6cbe00700 1 -- 172.21.15.89:0/471963511 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff6a406ac90 msgr2=0x7ff6a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:11.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.600+0000 7ff6cbe00700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff6a406ac90 0x7ff6a406d150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8015040 tx=0x7ff6b800a000 comp rx=0 tx=0).stop 2024-04-02T07:37:11.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.600+0000 7ff6cbe00700 1 -- 172.21.15.89:0/471963511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c415e6c0 msgr2=0x7ff6c415eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.600+0000 7ff6cbe00700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c415e6c0 0x7ff6c415eb40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0011aa0 tx=0x7ff6b0011e60 comp rx=0 tx=0).stop 2024-04-02T07:37:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.600+0000 7ff6cbe00700 1 -- 172.21.15.89:0/471963511 shutdown_connections 2024-04-02T07:37:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.600+0000 7ff6cbe00700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff6a406ac90 0x7ff6a406d150 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.600+0000 7ff6cbe00700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6c414b2e0 0x7ff6c414b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.600+0000 7ff6cbe00700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6ac003bc0 0x7ff6c4156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.600+0000 7ff6cbe00700 1 --2- 172.21.15.89:0/471963511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c415e6c0 0x7ff6c415eb40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.600+0000 7ff6cbe00700 1 -- 172.21.15.89:0/471963511 >> 172.21.15.89:0/471963511 conn(0x7ff6c408aff0 msgr2=0x7ff6c40ca650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.601+0000 7ff6cbe00700 1 -- 172.21.15.89:0/471963511 shutdown_connections 2024-04-02T07:37:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:11.601+0000 7ff6cbe00700 1 -- 172.21.15.89:0/471963511 wait complete. 2024-04-02T07:37:11.603 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 17 2024-04-02T07:37:12.026 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T07:37:05.500422+0000', 'modified': '2024-04-02T07:37:08.737868+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5049}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5049': {'gid': 5049, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.89:6837/929002087', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6836', 'nonce': 929002087}, {'type': 'v1', 'addr': '172.21.15.89:6837', 'nonce': 929002087}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T07:37:12.026 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T07:37:12.026 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T07:37:12.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.217+0000 7fa8fd200700 1 Processor -- start 2024-04-02T07:37:12.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.217+0000 7fa8fd200700 1 -- start start 2024-04-02T07:37:12.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.218+0000 7fa8fd200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8f815a5c0 0x7fa8f80709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.218+0000 7fa8fd200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa8f80ce840 0x7fa8f80cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.218+0000 7fa8fd200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f80cb310 con 0x7fa8f815dfe0 2024-04-02T07:37:12.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.218+0000 7fa8fd200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f8070f00 con 0x7fa8f80ce840 2024-04-02T07:37:12.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.218+0000 7fa8fd200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80711a0 con 0x7fa8f815a5c0 2024-04-02T07:37:12.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.218+0000 7fa8f5e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8f815a5c0 0x7fa8f80709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.218+0000 7fa8f5e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8f815a5c0 0x7fa8f80709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:36678/0 (socket says 172.21.15.89:36678) 2024-04-02T07:37:12.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.218+0000 7fa8f5e00700 1 -- 172.21.15.89:0/3220899511 learned_addr learned my addr 172.21.15.89:0/3220899511 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:12.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.218+0000 7fa8f6800700 1 --2- 172.21.15.89:0/3220899511 >> v2:172.21.15.113:3300/0 conn(0x7fa8f80ce840 0x7fa8f80cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.219+0000 7fa8f5e00700 1 -- 172.21.15.89:0/3220899511 >> v2:172.21.15.113:3300/0 conn(0x7fa8f80ce840 msgr2=0x7fa8f80cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:12.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.219+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/3220899511 >> v2:172.21.15.113:3300/0 conn(0x7fa8f80ce840 0x7fa8f80cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.219+0000 7fa8f5e00700 1 -- 172.21.15.89:0/3220899511 >> v1:172.21.15.113:6789/0 conn(0x7fa8f815dfe0 legacy=0x7fa8f80ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.219+0000 7fa8f5e00700 1 -- 172.21.15.89:0/3220899511 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f8071480 con 0x7fa8f815a5c0 2024-04-02T07:37:12.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.219+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/3220899511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8f815a5c0 0x7fa8f80709c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa8e80098f0 tx=0x7fa8e8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5714da8fd7e03d8c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:12.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.219+0000 7fa8f4a00700 1 -- 172.21.15.89:0/3220899511 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8011830 con 0x7fa8f815a5c0 2024-04-02T07:37:12.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.220+0000 7fa8f4a00700 1 -- 172.21.15.89:0/3220899511 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e8011990 con 0x7fa8f815a5c0 2024-04-02T07:37:12.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.220+0000 7fa8f4a00700 1 -- 172.21.15.89:0/3220899511 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8011d20 con 0x7fa8f815a5c0 2024-04-02T07:37:12.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.220+0000 7fa8fd200700 1 -- 172.21.15.89:0/3220899511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8f815a5c0 msgr2=0x7fa8f80709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.220+0000 7fa8fd200700 1 --2- 172.21.15.89:0/3220899511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8f815a5c0 0x7fa8f80709c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa8e80098f0 tx=0x7fa8e8009c00 comp rx=0 tx=0).stop 2024-04-02T07:37:12.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.220+0000 7fa8fd200700 1 -- 172.21.15.89:0/3220899511 shutdown_connections 2024-04-02T07:37:12.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.220+0000 7fa8fd200700 1 --2- 172.21.15.89:0/3220899511 >> v2:172.21.15.113:3300/0 conn(0x7fa8f80ce840 0x7fa8f80cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.220+0000 7fa8fd200700 1 --2- 172.21.15.89:0/3220899511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8f815a5c0 0x7fa8f80709c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.220+0000 7fa8fd200700 1 -- 172.21.15.89:0/3220899511 >> 172.21.15.89:0/3220899511 conn(0x7fa8f80b0d70 msgr2=0x7fa8f80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:12.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.220+0000 7fa8fd200700 1 -- 172.21.15.89:0/3220899511 shutdown_connections 2024-04-02T07:37:12.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.220+0000 7fa8fd200700 1 -- 172.21.15.89:0/3220899511 wait complete. 2024-04-02T07:37:12.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.222+0000 7fa8fd200700 1 Processor -- start 2024-04-02T07:37:12.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.222+0000 7fa8fd200700 1 -- start start 2024-04-02T07:37:12.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.223+0000 7fa8fd200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8f80ce840 0x7fa8f814a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.223+0000 7fa8fd200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8f815a5c0 0x7fa8f814c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.223+0000 7fa8fd200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f815dfe0 0x7fa8f814f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.223+0000 7fa8fd200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f814a0d0 con 0x7fa8f815dfe0 2024-04-02T07:37:12.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.223+0000 7fa8fd200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f814ff20 con 0x7fa8f815a5c0 2024-04-02T07:37:12.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.223+0000 7fa8fd200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f814d070 con 0x7fa8f80ce840 2024-04-02T07:37:12.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.223+0000 7fa8f6800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f815dfe0 0x7fa8f814f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.223+0000 7fa8f6800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f815dfe0 0x7fa8f814f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58096/0 (socket says 172.21.15.89:58096) 2024-04-02T07:37:12.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.223+0000 7fa8f6800700 1 -- 172.21.15.89:0/3125278961 learned_addr learned my addr 172.21.15.89:0/3125278961 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:12.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.223+0000 7fa8f5400700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8f815a5c0 0x7fa8f814c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.223+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8f80ce840 0x7fa8f814a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.224+0000 7fa8f6800700 1 -- 172.21.15.89:0/3125278961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8f80ce840 msgr2=0x7fa8f814a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.224+0000 7fa8f6800700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8f80ce840 0x7fa8f814a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.224+0000 7fa8f6800700 1 -- 172.21.15.89:0/3125278961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8f815a5c0 msgr2=0x7fa8f814c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.224+0000 7fa8f6800700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8f815a5c0 0x7fa8f814c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.224+0000 7fa8f6800700 1 -- 172.21.15.89:0/3125278961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e8009550 con 0x7fa8f815dfe0 2024-04-02T07:37:12.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.224+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8f80ce840 0x7fa8f814a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:12.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.224+0000 7fa8f6800700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f815dfe0 0x7fa8f814f180 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000efc0 tx=0x7fa8f000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:12.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.224+0000 7fa8ee000700 1 -- 172.21.15.89:0/3125278961 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f0009ea0 con 0x7fa8f815dfe0 2024-04-02T07:37:12.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.224+0000 7fa8ee000700 1 -- 172.21.15.89:0/3125278961 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f000f040 con 0x7fa8f815dfe0 2024-04-02T07:37:12.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.224+0000 7fa8fd200700 1 -- 172.21.15.89:0/3125278961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f814c620 con 0x7fa8f815dfe0 2024-04-02T07:37:12.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.224+0000 7fa8ee000700 1 -- 172.21.15.89:0/3125278961 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f0014480 con 0x7fa8f815dfe0 2024-04-02T07:37:12.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.224+0000 7fa8fd200700 1 -- 172.21.15.89:0/3125278961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f814d280 con 0x7fa8f815dfe0 2024-04-02T07:37:12.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.226+0000 7fa8ee000700 1 -- 172.21.15.89:0/3125278961 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa8f00145e0 con 0x7fa8f815dfe0 2024-04-02T07:37:12.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.227+0000 7fa8ee000700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa8d806ac90 0x7fa8d806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.227+0000 7fa8ee000700 1 -- 172.21.15.89:0/3125278961 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8f0089c00 con 0x7fa8f815dfe0 2024-04-02T07:37:12.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.228+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa8d806ac90 0x7fa8d806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.228+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa8d806ac90 0x7fa8d806d150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8009c40 tx=0x7fa8e8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:12.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.229+0000 7fa910d08b80 1 Processor -- start 2024-04-02T07:37:12.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.229+0000 7fa910d08b80 1 -- start start 2024-04-02T07:37:12.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.230+0000 7fa910d08b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d9f830 0x55ec21d9fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.230+0000 7fa910d08b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d72be0 0x55ec21d75a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.230+0000 7fa910d08b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d70b60 0x55ec21d9ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.230+0000 7fa910d08b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec21cea650 con 0x55ec21d72be0 2024-04-02T07:37:12.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.230+0000 7fa910d08b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec21ccbc30 con 0x55ec21d9f830 2024-04-02T07:37:12.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.230+0000 7fa910d08b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ec21cc9430 con 0x55ec21d70b60 2024-04-02T07:37:12.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.231+0000 7fa8f5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d72be0 0x55ec21d75a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.231+0000 7fa8f5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d72be0 0x55ec21d75a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58108/0 (socket says 172.21.15.89:58108) 2024-04-02T07:37:12.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.231+0000 7fa8f5400700 1 -- 172.21.15.89:0/3278193651 learned_addr learned my addr 172.21.15.89:0/3278193651 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:12.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.231+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/3278193651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d9f830 0x55ec21d9fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.231+0000 7fa8f6800700 1 --2- 172.21.15.89:0/3278193651 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d70b60 0x55ec21d9ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.231+0000 7fa8f5400700 1 -- 172.21.15.89:0/3278193651 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d70b60 msgr2=0x55ec21d9ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:12.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.231+0000 7fa8f5400700 1 --2- 172.21.15.89:0/3278193651 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d70b60 0x55ec21d9ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.231+0000 7fa8f5400700 1 -- 172.21.15.89:0/3278193651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d9f830 msgr2=0x55ec21d9fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:12.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.231+0000 7fa8f5400700 1 --2- 172.21.15.89:0/3278193651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d9f830 0x55ec21d9fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.231+0000 7fa8f5400700 1 -- 172.21.15.89:0/3278193651 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec21ab8790 con 0x55ec21d72be0 2024-04-02T07:37:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.231+0000 7fa8f6800700 1 --2- 172.21.15.89:0/3278193651 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d70b60 0x55ec21d9ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.231+0000 7fa8f5400700 1 --2- 172.21.15.89:0/3278193651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d72be0 0x55ec21d75a10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa8e40016a0 tx=0x7fa8e4007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edcbeade962585df server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.232+0000 7fa8e2a00700 1 -- 172.21.15.89:0/3278193651 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4005320 con 0x55ec21d72be0 2024-04-02T07:37:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.232+0000 7fa8e2a00700 1 -- 172.21.15.89:0/3278193651 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e4005480 con 0x55ec21d72be0 2024-04-02T07:37:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.232+0000 7fa8e2a00700 1 -- 172.21.15.89:0/3278193651 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4003880 con 0x55ec21d72be0 2024-04-02T07:37:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.232+0000 7fa910d08b80 1 -- 172.21.15.89:0/3278193651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d72be0 msgr2=0x55ec21d75a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.232+0000 7fa910d08b80 1 --2- 172.21.15.89:0/3278193651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d72be0 0x55ec21d75a10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa8e40016a0 tx=0x7fa8e4007ec0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.232+0000 7fa910d08b80 1 -- 172.21.15.89:0/3278193651 shutdown_connections 2024-04-02T07:37:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.232+0000 7fa910d08b80 1 --2- 172.21.15.89:0/3278193651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d9f830 0x55ec21d9fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.232+0000 7fa910d08b80 1 --2- 172.21.15.89:0/3278193651 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d70b60 0x55ec21d9ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.232+0000 7fa910d08b80 1 --2- 172.21.15.89:0/3278193651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d72be0 0x55ec21d75a10 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.232+0000 7fa910d08b80 1 -- 172.21.15.89:0/3278193651 >> 172.21.15.89:0/3278193651 conn(0x55ec21cd36a0 msgr2=0x55ec21dba640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.232+0000 7fa910d08b80 1 -- 172.21.15.89:0/3278193651 shutdown_connections 2024-04-02T07:37:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.232+0000 7fa910d08b80 1 -- 172.21.15.89:0/3278193651 wait complete. 2024-04-02T07:37:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.233+0000 7fa910d08b80 1 -- start start 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.233+0000 7fa910d08b80 1 Processor -- start 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.234+0000 7fa910d08b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d70b60 0x55ec21d98900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.234+0000 7fa910d08b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d72be0 0x55ec21d99920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.234+0000 7fa910d08b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d9f830 0x55ec21d99e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.234+0000 7fa910d08b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec21cc4d20 con 0x55ec21d70b60 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.234+0000 7fa910d08b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec21c60770 con 0x55ec21d72be0 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.234+0000 7fa910d08b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ec21c836f0 con 0x55ec21d9f830 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.234+0000 7fa910d08b80 10 client.0 _ll_register_callbacks cb 0x55ec21c6a5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.235+0000 7fa8f5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d70b60 0x55ec21d98900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.235+0000 7fa8f5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d70b60 0x55ec21d98900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58114/0 (socket says 172.21.15.89:58114) 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.235+0000 7fa8f5e00700 1 -- 172.21.15.89:0/795334886 learned_addr learned my addr 172.21.15.89:0/795334886 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.235+0000 7fa8f5400700 1 --2- 172.21.15.89:0/795334886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d72be0 0x55ec21d99920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.235+0000 7fa8f6800700 1 --2- 172.21.15.89:0/795334886 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d9f830 0x55ec21d99e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.235+0000 7fa8f5e00700 1 -- 172.21.15.89:0/795334886 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d9f830 msgr2=0x55ec21d99e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.235+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/795334886 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d9f830 0x55ec21d99e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.235+0000 7fa8f5e00700 1 -- 172.21.15.89:0/795334886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d72be0 msgr2=0x55ec21d99920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.235+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/795334886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d72be0 0x55ec21d99920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.235+0000 7fa8f5e00700 1 -- 172.21.15.89:0/795334886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f80cb310 con 0x55ec21d70b60 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.235+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/795334886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d70b60 0x55ec21d98900 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8009f50 tx=0x7fa8e800e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5376c7f9ec225a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.235+0000 7fa8d7400700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa8d7400700 1 -- 172.21.15.89:0/795334886 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e802c040 con 0x55ec21d70b60 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa8d7400700 1 -- 172.21.15.89:0/795334886 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8e8032ae0 con 0x55ec21d70b60 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa910d08b80 1 -- 172.21.15.89:0/795334886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec21d67920 con 0x55ec21d70b60 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa8d7400700 1 -- 172.21.15.89:0/795334886 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8e8032ca0 con 0x55ec21d70b60 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa910d08b80 10 client.5208 fetch_fsmap learned FSMap version 17 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa910d08b80 1 -- 172.21.15.89:0/795334886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55ec21ba50a0 con 0x55ec21d70b60 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa8d7400700 1 -- 172.21.15.89:0/795334886 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ec21d67920 con 0x55ec21d70b60 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa8d7400700 1 -- 172.21.15.89:0/795334886 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 17) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa8e8025660 con 0x55ec21d70b60 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa910d08b80 10 client.5208 fetch_fsmap finished waiting for FSMap version 17 2024-04-02T07:37:12.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa910d08b80 10 client.5208 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T07:37:12.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa910d08b80 20 client.5208 populate_metadata read hostname 'smithi089' 2024-04-02T07:37:12.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa910d08b80 1 --2- 172.21.15.89:0/795334886 >> [v2:172.21.15.89:6836/929002087,v1:172.21.15.89:6837/929002087] conn(0x55ec21da0f70 0x55ec21dcd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa910d08b80 4 client.5208 mds_command: new command op to 5049 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T07:37:12.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa910d08b80 1 -- 172.21.15.89:0/795334886 --> [v2:172.21.15.89:6836/929002087,v1:172.21.15.89:6837/929002087] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec21d446f0 con 0x55ec21da0f70 2024-04-02T07:37:12.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.236+0000 7fa8f5400700 1 --2- 172.21.15.89:0/795334886 >> [v2:172.21.15.89:6836/929002087,v1:172.21.15.89:6837/929002087] conn(0x55ec21da0f70 0x55ec21dcd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.237+0000 7fa8d7400700 10 client.5208 ms_handle_connect on v2:172.21.15.89:6836/929002087 2024-04-02T07:37:12.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.237+0000 7fa8f5400700 1 --2- 172.21.15.89:0/795334886 >> [v2:172.21.15.89:6836/929002087,v1:172.21.15.89:6837/929002087] conn(0x55ec21da0f70 0x55ec21dcd440 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=305576163422193c server_cookie=58d3ec701f414bbc in_seq=0 out_seq=0 2024-04-02T07:37:12.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa8d7400700 1 -- 172.21.15.89:0/795334886 <== mds.0 v2:172.21.15.89:6836/929002087 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ec21d446f0 con 0x55ec21da0f70 2024-04-02T07:37:12.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa8d7400700 10 client.5208 handle_command_reply: tid=0 2024-04-02T07:37:12.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa910d08b80 1 client.5208 shutdown 2024-04-02T07:37:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa910d08b80 1 -- 172.21.15.89:0/795334886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d70b60 msgr2=0x55ec21d98900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa910d08b80 1 --2- 172.21.15.89:0/795334886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d70b60 0x55ec21d98900 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8009f50 tx=0x7fa8e800e000 comp rx=0 tx=0).stop 2024-04-02T07:37:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa910d08b80 1 -- 172.21.15.89:0/795334886 shutdown_connections 2024-04-02T07:37:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa910d08b80 1 --2- 172.21.15.89:0/795334886 >> [v2:172.21.15.89:6836/929002087,v1:172.21.15.89:6837/929002087] conn(0x55ec21da0f70 0x55ec21dcd440 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa910d08b80 1 --2- 172.21.15.89:0/795334886 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d9f830 0x55ec21d99e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa910d08b80 1 --2- 172.21.15.89:0/795334886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d70b60 0x55ec21d98900 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa8d7400700 0 client.5208 ms_handle_reset on v2:172.21.15.89:6836/929002087 2024-04-02T07:37:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa910d08b80 1 --2- 172.21.15.89:0/795334886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d72be0 0x55ec21d99920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa910d08b80 1 -- 172.21.15.89:0/795334886 >> 172.21.15.89:0/795334886 conn(0x55ec21cd36a0 msgr2=0x55ec21d9bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa910d08b80 1 -- 172.21.15.89:0/795334886 shutdown_connections 2024-04-02T07:37:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa910d08b80 1 -- 172.21.15.89:0/795334886 wait complete. 2024-04-02T07:37:12.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.238+0000 7fa910d08b80 20 client.5208 trim_cache size 0 max 16384 2024-04-02T07:37:12.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.256+0000 7fa910d08b80 1 Processor -- start 2024-04-02T07:37:12.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.256+0000 7fa910d08b80 1 -- start start 2024-04-02T07:37:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.257+0000 7fa910d08b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d70b60 0x55ec21d9ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.257+0000 7fa910d08b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d72be0 0x55ec21d9bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.257+0000 7fa910d08b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d9f830 0x55ec21dc85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.257+0000 7fa910d08b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec21ab9390 con 0x55ec21d70b60 2024-04-02T07:37:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.257+0000 7fa910d08b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec21d9e490 con 0x55ec21d72be0 2024-04-02T07:37:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.257+0000 7fa910d08b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ec21dcca90 con 0x55ec21d9f830 2024-04-02T07:37:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.257+0000 7fa8f5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d70b60 0x55ec21d9ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.257+0000 7fa8f5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d70b60 0x55ec21d9ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58118/0 (socket says 172.21.15.89:58118) 2024-04-02T07:37:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.257+0000 7fa8f5e00700 1 -- 172.21.15.89:0/23522276 learned_addr learned my addr 172.21.15.89:0/23522276 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.258+0000 7fa8f5400700 1 --2- 172.21.15.89:0/23522276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d72be0 0x55ec21d9bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.258+0000 7fa8f5e00700 1 -- 172.21.15.89:0/23522276 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d9f830 msgr2=0x55ec21dc85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:12.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.258+0000 7fa8f6800700 1 --2- 172.21.15.89:0/23522276 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d9f830 0x55ec21dc85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.258+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/23522276 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d9f830 0x55ec21dc85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.258+0000 7fa8f5e00700 1 -- 172.21.15.89:0/23522276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d72be0 msgr2=0x55ec21d9bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:12.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.258+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/23522276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d72be0 0x55ec21d9bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.258+0000 7fa8f5e00700 1 -- 172.21.15.89:0/23522276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec21d75950 con 0x55ec21d70b60 2024-04-02T07:37:12.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.258+0000 7fa8f6800700 1 --2- 172.21.15.89:0/23522276 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d9f830 0x55ec21dc85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:37:12.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.258+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/23522276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d70b60 0x55ec21d9ed30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8006e30 tx=0x7fa8e801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bfae6775a7a70 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:12.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.258+0000 7fa8e3400700 1 -- 172.21.15.89:0/23522276 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e802c040 con 0x55ec21d70b60 2024-04-02T07:37:12.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.258+0000 7fa8e3400700 1 -- 172.21.15.89:0/23522276 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e8025b20 con 0x55ec21d70b60 2024-04-02T07:37:12.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.258+0000 7fa8e3400700 1 -- 172.21.15.89:0/23522276 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8025660 con 0x55ec21d70b60 2024-04-02T07:37:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.259+0000 7fa910d08b80 1 -- 172.21.15.89:0/23522276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d70b60 msgr2=0x55ec21d9ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.259+0000 7fa910d08b80 1 --2- 172.21.15.89:0/23522276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d70b60 0x55ec21d9ed30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8006e30 tx=0x7fa8e801e040 comp rx=0 tx=0).stop 2024-04-02T07:37:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.259+0000 7fa910d08b80 1 -- 172.21.15.89:0/23522276 shutdown_connections 2024-04-02T07:37:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.259+0000 7fa910d08b80 1 --2- 172.21.15.89:0/23522276 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d9f830 0x55ec21dc85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.259+0000 7fa910d08b80 1 --2- 172.21.15.89:0/23522276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d70b60 0x55ec21d9ed30 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.259+0000 7fa910d08b80 1 --2- 172.21.15.89:0/23522276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d72be0 0x55ec21d9bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.259+0000 7fa910d08b80 1 -- 172.21.15.89:0/23522276 >> 172.21.15.89:0/23522276 conn(0x55ec21cd36a0 msgr2=0x55ec21d6b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.259+0000 7fa910d08b80 1 -- 172.21.15.89:0/23522276 shutdown_connections 2024-04-02T07:37:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.259+0000 7fa910d08b80 1 -- 172.21.15.89:0/23522276 wait complete. 2024-04-02T07:37:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.259+0000 7fa910d08b80 1 -- start start 2024-04-02T07:37:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.260+0000 7fa910d08b80 1 Processor -- start 2024-04-02T07:37:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa910d08b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d70b60 0x55ec21d736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa910d08b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d72be0 0x55ec21e151e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa910d08b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d9f830 0x55ec21d9c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa910d08b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec21d9a790 con 0x55ec21d72be0 2024-04-02T07:37:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa910d08b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec21d73c30 con 0x55ec21d9f830 2024-04-02T07:37:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa910d08b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ec21d98b90 con 0x55ec21d70b60 2024-04-02T07:37:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa910d08b80 10 client.0 _ll_register_callbacks cb 0x7fa8e8004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:37:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa8f5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d72be0 0x55ec21e151e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa8f5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d72be0 0x55ec21e151e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58128/0 (socket says 172.21.15.89:58128) 2024-04-02T07:37:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa8f5400700 1 -- 172.21.15.89:0/863521580 learned_addr learned my addr 172.21.15.89:0/863521580 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/863521580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d70b60 0x55ec21d736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa8f6800700 1 --2- 172.21.15.89:0/863521580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d9f830 0x55ec21d9c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa8f5400700 1 -- 172.21.15.89:0/863521580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d70b60 msgr2=0x55ec21d736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa8f5400700 1 --2- 172.21.15.89:0/863521580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d70b60 0x55ec21d736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa8f5400700 1 -- 172.21.15.89:0/863521580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d9f830 msgr2=0x55ec21d9c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa8f5400700 1 --2- 172.21.15.89:0/863521580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d9f830 0x55ec21d9c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa8f5400700 1 -- 172.21.15.89:0/863521580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec21ba50a0 con 0x55ec21d72be0 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa8f5400700 1 --2- 172.21.15.89:0/863521580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d72be0 0x55ec21e151e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4001c90 tx=0x7fa8e4000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc732064a2de7ed0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.261+0000 7fa8d7400700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.262+0000 7fa8d7400700 1 -- 172.21.15.89:0/863521580 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e40275c0 con 0x55ec21d72be0 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.262+0000 7fa8d7400700 1 -- 172.21.15.89:0/863521580 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8e40277a0 con 0x55ec21d72be0 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.262+0000 7fa910d08b80 1 -- 172.21.15.89:0/863521580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa8e4005480 con 0x55ec21d72be0 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.262+0000 7fa8d7400700 1 -- 172.21.15.89:0/863521580 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8e4027960 con 0x55ec21d72be0 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.262+0000 7fa910d08b80 10 client.5214 fetch_fsmap learned FSMap version 17 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.262+0000 7fa910d08b80 1 -- 172.21.15.89:0/863521580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55ec21c8b010 con 0x55ec21d72be0 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.262+0000 7fa8d7400700 1 -- 172.21.15.89:0/863521580 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e4005480 con 0x55ec21d72be0 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.262+0000 7fa8d7400700 1 -- 172.21.15.89:0/863521580 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 17) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa8e4012070 con 0x55ec21d72be0 2024-04-02T07:37:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.262+0000 7fa910d08b80 10 client.5214 fetch_fsmap finished waiting for FSMap version 17 2024-04-02T07:37:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.262+0000 7fa910d08b80 10 client.5214 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T07:37:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.262+0000 7fa910d08b80 20 client.5214 populate_metadata read hostname 'smithi089' 2024-04-02T07:37:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.262+0000 7fa910d08b80 1 --2- 172.21.15.89:0/863521580 >> [v2:172.21.15.89:6836/929002087,v1:172.21.15.89:6837/929002087] conn(0x55ec21d6f6b0 0x55ec21d6fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.262+0000 7fa910d08b80 4 client.5214 mds_command: new command op to 5049 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T07:37:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.262+0000 7fa910d08b80 1 -- 172.21.15.89:0/863521580 --> [v2:172.21.15.89:6836/929002087,v1:172.21.15.89:6837/929002087] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ec21afe040 con 0x55ec21d6f6b0 2024-04-02T07:37:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.263+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/863521580 >> [v2:172.21.15.89:6836/929002087,v1:172.21.15.89:6837/929002087] conn(0x55ec21d6f6b0 0x55ec21d6fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.263+0000 7fa8d7400700 10 client.5214 ms_handle_connect on v2:172.21.15.89:6836/929002087 2024-04-02T07:37:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.263+0000 7fa8f5e00700 1 --2- 172.21.15.89:0/863521580 >> [v2:172.21.15.89:6836/929002087,v1:172.21.15.89:6837/929002087] conn(0x55ec21d6f6b0 0x55ec21d6fa90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97f9749214c4cfa2 server_cookie=34e181ba2e92a8f4 in_seq=0 out_seq=0 2024-04-02T07:37:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.263+0000 7fa8d7400700 1 -- 172.21.15.89:0/863521580 <== mds.0 v2:172.21.15.89:6836/929002087 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55ec21afe040 con 0x55ec21d6f6b0 2024-04-02T07:37:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.263+0000 7fa8d7400700 10 client.5214 handle_command_reply: tid=0 2024-04-02T07:37:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.263+0000 7fa910d08b80 1 client.5214 shutdown 2024-04-02T07:37:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.263+0000 7fa910d08b80 1 -- 172.21.15.89:0/863521580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d72be0 msgr2=0x55ec21e151e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.263+0000 7fa910d08b80 1 --2- 172.21.15.89:0/863521580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d72be0 0x55ec21e151e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4001c90 tx=0x7fa8e4000ed0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa910d08b80 1 -- 172.21.15.89:0/863521580 shutdown_connections 2024-04-02T07:37:12.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa910d08b80 1 --2- 172.21.15.89:0/863521580 >> [v2:172.21.15.89:6836/929002087,v1:172.21.15.89:6837/929002087] conn(0x55ec21d6f6b0 0x55ec21d6fa90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa910d08b80 1 --2- 172.21.15.89:0/863521580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ec21d9f830 0x55ec21d9c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa910d08b80 1 --2- 172.21.15.89:0/863521580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ec21d70b60 0x55ec21d736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa8d7400700 0 client.5214 ms_handle_reset on v2:172.21.15.89:6836/929002087 2024-04-02T07:37:12.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa910d08b80 1 --2- 172.21.15.89:0/863521580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec21d72be0 0x55ec21e151e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa910d08b80 1 -- 172.21.15.89:0/863521580 >> 172.21.15.89:0/863521580 conn(0x55ec21cd36a0 msgr2=0x55ec21dc85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:12.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa910d08b80 1 -- 172.21.15.89:0/863521580 shutdown_connections 2024-04-02T07:37:12.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa910d08b80 1 -- 172.21.15.89:0/863521580 wait complete. 2024-04-02T07:37:12.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa910d08b80 20 client.5214 trim_cache size 0 max 16384 2024-04-02T07:37:12.271 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:37:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T07:37:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T07:37:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 5049, 2024-04-02T07:37:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T07:37:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T07:37:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T07:37:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 6.761195238, 2024-04-02T07:37:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 17, 2024-04-02T07:37:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 34, 2024-04-02T07:37:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 34, 2024-04-02T07:37:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 15.118412704000001 2024-04-02T07:37:12.271 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:37:12.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa8fd200700 1 -- 172.21.15.89:0/3125278961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa8d806ac90 msgr2=0x7fa8d806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa8fd200700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa8d806ac90 0x7fa8d806d150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8009c40 tx=0x7fa8e8008030 comp rx=0 tx=0).stop 2024-04-02T07:37:12.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa8fd200700 1 -- 172.21.15.89:0/3125278961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f815dfe0 msgr2=0x7fa8f814f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa8fd200700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f815dfe0 0x7fa8f814f180 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000efc0 tx=0x7fa8f000c3a0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa8fd200700 1 -- 172.21.15.89:0/3125278961 shutdown_connections 2024-04-02T07:37:12.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.264+0000 7fa8fd200700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa8d806ac90 0x7fa8d806d150 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.265+0000 7fa8fd200700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f815dfe0 0x7fa8f814f180 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.265+0000 7fa8fd200700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8f80ce840 0x7fa8f814a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.265+0000 7fa8fd200700 1 --2- 172.21.15.89:0/3125278961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8f815a5c0 0x7fa8f814c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.265+0000 7fa8fd200700 1 -- 172.21.15.89:0/3125278961 >> 172.21.15.89:0/3125278961 conn(0x7fa8f80b0d70 msgr2=0x7fa8f80cd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:12.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.265+0000 7fa8fd200700 1 -- 172.21.15.89:0/3125278961 shutdown_connections 2024-04-02T07:37:12.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.265+0000 7fa8fd200700 1 -- 172.21.15.89:0/3125278961 wait complete. 2024-04-02T07:37:12.284 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T07:37:12.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.472+0000 7febbec00700 1 Processor -- start 2024-04-02T07:37:12.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.472+0000 7febbec00700 1 -- start start 2024-04-02T07:37:12.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.473+0000 7febbec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb80ceec0 0x7febb81612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.473+0000 7febbec00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7febb8161800 0x7febb80b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.473+0000 7febbec00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb80d30d0 con 0x7febb815d640 2024-04-02T07:37:12.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.473+0000 7febbec00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb8161c90 con 0x7febb8161800 2024-04-02T07:37:12.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.473+0000 7febbec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb80b6840 con 0x7febb80ceec0 2024-04-02T07:37:12.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.473+0000 7febbcc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7febb8161800 0x7febb80b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.473+0000 7febbcc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7febb8161800 0x7febb80b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:34292/0 (socket says 172.21.15.89:34292) 2024-04-02T07:37:12.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.473+0000 7febbcc00700 1 -- 172.21.15.89:0/2774158888 learned_addr learned my addr 172.21.15.89:0/2774158888 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:12.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.473+0000 7febb7400700 1 --2- 172.21.15.89:0/2774158888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb80ceec0 0x7febb81612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.474+0000 7febbcc00700 1 -- 172.21.15.89:0/2774158888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb80ceec0 msgr2=0x7febb81612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:12.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.474+0000 7febbcc00700 1 --2- 172.21.15.89:0/2774158888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb80ceec0 0x7febb81612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.474+0000 7febbcc00700 1 -- 172.21.15.89:0/2774158888 >> v1:172.21.15.113:6789/0 conn(0x7febb815d640 legacy=0x7febb80d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.474+0000 7febbcc00700 1 -- 172.21.15.89:0/2774158888 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb80d2b30 con 0x7febb8161800 2024-04-02T07:37:12.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.474+0000 7febb6a00700 1 -- 172.21.15.89:0/2774158888 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263798583 0 0) 0x7febb80d30d0 con 0x7febb815d640 2024-04-02T07:37:12.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.474+0000 7febbcc00700 1 --2- 172.21.15.89:0/2774158888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb8161800 0x7febb80b6cd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7febb0009e50 tx=0x7febb000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6e2e656d97790e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:12.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.474+0000 7febb6a00700 1 -- 172.21.15.89:0/2774158888 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb000e070 con 0x7febb8161800 2024-04-02T07:37:12.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.475+0000 7febb6a00700 1 -- 172.21.15.89:0/2774158888 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb000b910 con 0x7febb8161800 2024-04-02T07:37:12.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.475+0000 7febb6a00700 1 -- 172.21.15.89:0/2774158888 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb000bcc0 con 0x7febb8161800 2024-04-02T07:37:12.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.475+0000 7febbec00700 1 -- 172.21.15.89:0/2774158888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb8161800 msgr2=0x7febb80b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.475+0000 7febbec00700 1 --2- 172.21.15.89:0/2774158888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb8161800 0x7febb80b6cd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7febb0009e50 tx=0x7febb000b330 comp rx=0 tx=0).stop 2024-04-02T07:37:12.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.475+0000 7febbec00700 1 -- 172.21.15.89:0/2774158888 shutdown_connections 2024-04-02T07:37:12.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.475+0000 7febbec00700 1 --2- 172.21.15.89:0/2774158888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb8161800 0x7febb80b6cd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.475+0000 7febbec00700 1 --2- 172.21.15.89:0/2774158888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb80ceec0 0x7febb81612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.475+0000 7febbec00700 1 -- 172.21.15.89:0/2774158888 >> 172.21.15.89:0/2774158888 conn(0x7febb808aff0 msgr2=0x7febb80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:12.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.475+0000 7febbec00700 1 -- 172.21.15.89:0/2774158888 shutdown_connections 2024-04-02T07:37:12.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.475+0000 7febbec00700 1 -- 172.21.15.89:0/2774158888 wait complete. 2024-04-02T07:37:12.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.477+0000 7febbec00700 1 Processor -- start 2024-04-02T07:37:12.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.477+0000 7febbec00700 1 -- start start 2024-04-02T07:37:12.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.478+0000 7febbec00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb80ceec0 0x7febb814c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.478+0000 7febbec00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb815d640 0x7febb81f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.478+0000 7febbec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb8161800 0x7febb81f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.478+0000 7febbec00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb81f4770 con 0x7febb80ceec0 2024-04-02T07:37:12.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.478+0000 7febbec00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb81f3e60 con 0x7febb815d640 2024-04-02T07:37:12.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.478+0000 7febbec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb80c98c0 con 0x7febb8161800 2024-04-02T07:37:12.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.478+0000 7febb7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb80ceec0 0x7febb814c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.478+0000 7febb7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb80ceec0 0x7febb814c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58134/0 (socket says 172.21.15.89:58134) 2024-04-02T07:37:12.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.478+0000 7febb7e00700 1 -- 172.21.15.89:0/259070561 learned_addr learned my addr 172.21.15.89:0/259070561 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:12.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.478+0000 7febb7400700 1 --2- 172.21.15.89:0/259070561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb815d640 0x7febb81f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.478+0000 7febbcc00700 1 --2- 172.21.15.89:0/259070561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb8161800 0x7febb81f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.479+0000 7febb7e00700 1 -- 172.21.15.89:0/259070561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb8161800 msgr2=0x7febb81f32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.479+0000 7febb7e00700 1 --2- 172.21.15.89:0/259070561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb8161800 0x7febb81f32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.479+0000 7febb7e00700 1 -- 172.21.15.89:0/259070561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb815d640 msgr2=0x7febb81f2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.479+0000 7febb7e00700 1 --2- 172.21.15.89:0/259070561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb815d640 0x7febb81f2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.479+0000 7febb7e00700 1 -- 172.21.15.89:0/259070561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb0009b30 con 0x7febb80ceec0 2024-04-02T07:37:12.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.479+0000 7febb7e00700 1 --2- 172.21.15.89:0/259070561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb80ceec0 0x7febb814c910 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7feba80011e0 tx=0x7feba800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:12.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.479+0000 7febafe00700 1 -- 172.21.15.89:0/259070561 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba80054e0 con 0x7febb80ceec0 2024-04-02T07:37:12.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.479+0000 7febafe00700 1 -- 172.21.15.89:0/259070561 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba8003680 con 0x7febb80ceec0 2024-04-02T07:37:12.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.479+0000 7febbec00700 1 -- 172.21.15.89:0/259070561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb81f4120 con 0x7febb80ceec0 2024-04-02T07:37:12.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.479+0000 7febafe00700 1 -- 172.21.15.89:0/259070561 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba8003a30 con 0x7febb80ceec0 2024-04-02T07:37:12.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.479+0000 7febbec00700 1 -- 172.21.15.89:0/259070561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb814ce50 con 0x7febb80ceec0 2024-04-02T07:37:12.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.480+0000 7febbec00700 1 -- 172.21.15.89:0/259070561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb8067720 con 0x7febb80ceec0 2024-04-02T07:37:12.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.481+0000 7febafe00700 1 -- 172.21.15.89:0/259070561 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7feba8003b90 con 0x7febb80ceec0 2024-04-02T07:37:12.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.482+0000 7febafe00700 1 --2- 172.21.15.89:0/259070561 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb9806ac90 0x7feb9806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:12.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.483+0000 7febafe00700 1 -- 172.21.15.89:0/259070561 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feba80886d0 con 0x7febb80ceec0 2024-04-02T07:37:12.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.483+0000 7febb7400700 1 --2- 172.21.15.89:0/259070561 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb9806ac90 0x7feb9806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:12.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.483+0000 7febb7400700 1 --2- 172.21.15.89:0/259070561 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb9806ac90 0x7feb9806d150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7febb815b000 tx=0x7feba4009210 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:12.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.487+0000 7febafe00700 1 -- 172.21.15.89:0/259070561 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feba8053940 con 0x7febb80ceec0 2024-04-02T07:37:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.732+0000 7febbec00700 1 -- 172.21.15.89:0/259070561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7febb80b5c70 con 0x7febb80ceec0 2024-04-02T07:37:12.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.732+0000 7febafe00700 1 -- 172.21.15.89:0/259070561 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7feba8017070 con 0x7febb80ceec0 2024-04-02T07:37:12.734 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T07:37:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.734+0000 7febbec00700 1 -- 172.21.15.89:0/259070561 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb9806ac90 msgr2=0x7feb9806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.734+0000 7febbec00700 1 --2- 172.21.15.89:0/259070561 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb9806ac90 0x7feb9806d150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7febb815b000 tx=0x7feba4009210 comp rx=0 tx=0).stop 2024-04-02T07:37:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.734+0000 7febbec00700 1 -- 172.21.15.89:0/259070561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb80ceec0 msgr2=0x7febb814c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.734+0000 7febbec00700 1 --2- 172.21.15.89:0/259070561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb80ceec0 0x7febb814c910 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7feba80011e0 tx=0x7feba800ca30 comp rx=0 tx=0).stop 2024-04-02T07:37:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.734+0000 7febbec00700 1 -- 172.21.15.89:0/259070561 shutdown_connections 2024-04-02T07:37:12.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.734+0000 7febbec00700 1 --2- 172.21.15.89:0/259070561 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb9806ac90 0x7feb9806d150 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.734+0000 7febbec00700 1 --2- 172.21.15.89:0/259070561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febb8161800 0x7febb81f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.734+0000 7febbec00700 1 --2- 172.21.15.89:0/259070561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febb80ceec0 0x7febb814c910 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.734+0000 7febbec00700 1 --2- 172.21.15.89:0/259070561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb815d640 0x7febb81f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:12.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.734+0000 7febbec00700 1 -- 172.21.15.89:0/259070561 >> 172.21.15.89:0/259070561 conn(0x7febb808aff0 msgr2=0x7febb815b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:12.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.735+0000 7febbec00700 1 -- 172.21.15.89:0/259070561 shutdown_connections 2024-04-02T07:37:12.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:12.735+0000 7febbec00700 1 -- 172.21.15.89:0/259070561 wait complete. 2024-04-02T07:37:12.755 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T07:37:12.755 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T07:37:12.755 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T07:37:12.755 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T07:37:12.755 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T07:37:12.756 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T07:37:12.756 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T07:37:12.756 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T07:37:12.756 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:37:12.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:12.756 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T07:37:12.801 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:37:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:37:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:37:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:37:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42396sec preferred_lft 42396sec 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T07:37:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:37:12.803 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T07:37:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:37:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T07:37:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:37:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T07:37:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:37:12.803 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:37:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T07:37:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T07:37:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:37:12.803 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:37:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T07:37:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T07:37:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:37:12.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:12.804 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T07:37:12.905 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T07:37:12.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:12.908 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T07:37:12.955 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T07:37:12.955 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T07:37:12.955 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T07:37:12.955 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T07:37:12.956 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T07:37:12.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:12.956 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:37:12.956 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:37:12.956 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T07:37:12.956 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T07:37:12.956 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T07:37:12.956 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:37:13.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:13.116 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T07:37:13.160 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T07:37:13.160 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T07:37:13.161 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:37:13.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:13.161 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T07:37:13.269 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:37:13.269 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T07:37:13.314 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:37:13.314 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:37:13.314 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:37:13.397 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T07:37:13.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:13.397 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:37:13.538 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:37:13.539 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:37:13.567 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:37:13.567 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:37:13.567 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T07:37:13.567 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T07:37:13.567 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T07:37:13.573 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T07:37:13.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:13.573 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:37:13.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:13.640 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:37:13.742 INFO:teuthology.orchestra.run.smithi089.stdout:client5220 2024-04-02T07:37:13.742 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:37:13.743 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:37:13.787 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T07:37:14.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.018+0000 7f2709400700 1 Processor -- start 2024-04-02T07:37:14.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.018+0000 7f2709400700 1 -- start start 2024-04-02T07:37:14.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.019+0000 7f2709400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27040d2540 0x7f27041687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:14.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.019+0000 7f2709400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2704161420 con 0x7f270415d9d0 2024-04-02T07:37:14.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.019+0000 7f2709400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2704161700 con 0x7f27040d16a0 2024-04-02T07:37:14.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.019+0000 7f2709400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27040d4340 con 0x7f27040d2540 2024-04-02T07:37:14.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.019+0000 7f2702800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f270415d9d0 0x7f27040d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40142/0 (socket says 172.21.15.89:40142) 2024-04-02T07:37:14.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.019+0000 7f2702800700 1 -- 172.21.15.89:0/1160670267 learned_addr learned my addr 172.21.15.89:0/1160670267 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:14.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2703200700 1 --2- 172.21.15.89:0/1160670267 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27040d2540 0x7f27041687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:14.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266955219 0 0) 0x7f2704161420 con 0x7f270415d9d0 2024-04-02T07:37:14.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e80035b0 con 0x7f270415d9d0 2024-04-02T07:37:14.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376646112 0 0) 0x7f2704161700 con 0x7f27040d16a0 2024-04-02T07:37:14.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2704161420 con 0x7f27040d16a0 2024-04-02T07:37:14.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2797832960 0 0) 0x7f26e80035b0 con 0x7f270415d9d0 2024-04-02T07:37:14.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27040d2540 msgr2=0x7f27041687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:14.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 --2- 172.21.15.89:0/1160670267 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27040d2540 0x7f27041687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 >> v1:172.21.15.113:6789/0 conn(0x7f27040d16a0 legacy=0x7f27040d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27041619e0 con 0x7f270415d9d0 2024-04-02T07:37:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f26f4001670 con 0x7f270415d9d0 2024-04-02T07:37:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 >> v1:172.21.15.89:6789/0 conn(0x7f270415d9d0 legacy=0x7f27040d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 --2- 172.21.15.89:0/1160670267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e8003bc0 0x7f26e8003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 --2- 172.21.15.89:0/1160670267 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27040d2540 0x7f26e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 --2- 172.21.15.89:0/1160670267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26e8004d90 0x7f26e8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e800bac0 con 0x7f26e8003bc0 2024-04-02T07:37:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e800bc30 con 0x7f26e8004d90 2024-04-02T07:37:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e800bdf0 con 0x7f27040d2540 2024-04-02T07:37:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.020+0000 7f2702800700 1 --2- 172.21.15.89:0/1160670267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e8003bc0 0x7f26e8003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.021+0000 7f2703200700 1 --2- 172.21.15.89:0/1160670267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26e8004d90 0x7f26e8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.021+0000 7f2701e00700 1 --2- 172.21.15.89:0/1160670267 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27040d2540 0x7f26e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.021+0000 7f2702800700 1 -- 172.21.15.89:0/1160670267 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27040d2540 msgr2=0x7f26e8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.021+0000 7f2702800700 1 --2- 172.21.15.89:0/1160670267 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27040d2540 0x7f26e8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.021+0000 7f2702800700 1 -- 172.21.15.89:0/1160670267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26e8004d90 msgr2=0x7f26e8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.021+0000 7f2702800700 1 --2- 172.21.15.89:0/1160670267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26e8004d90 0x7f26e8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.021+0000 7f2702800700 1 -- 172.21.15.89:0/1160670267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e80035b0 con 0x7f26e8003bc0 2024-04-02T07:37:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.021+0000 7f2702800700 1 --2- 172.21.15.89:0/1160670267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e8003bc0 0x7f26e8003fc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f26f400c110 tx=0x7f26e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ece8195f41dff3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.021+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f400ccb0 con 0x7f26e8003bc0 2024-04-02T07:37:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.021+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f400ce30 con 0x7f26e8003bc0 2024-04-02T07:37:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.021+0000 7f2701400700 1 -- 172.21.15.89:0/1160670267 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4005ba0 con 0x7f26e8003bc0 2024-04-02T07:37:14.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.022+0000 7f2709400700 1 -- 172.21.15.89:0/1160670267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e8003bc0 msgr2=0x7f26e8003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:14.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.022+0000 7f2709400700 1 --2- 172.21.15.89:0/1160670267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e8003bc0 0x7f26e8003fc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f26f400c110 tx=0x7f26e8005880 comp rx=0 tx=0).stop 2024-04-02T07:37:14.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.022+0000 7f2703200700 1 -- 172.21.15.89:0/1160670267 reap_dead start 2024-04-02T07:37:14.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.022+0000 7f2709400700 1 -- 172.21.15.89:0/1160670267 shutdown_connections 2024-04-02T07:37:14.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.022+0000 7f2709400700 1 -- 172.21.15.89:0/1160670267 >> 172.21.15.89:0/1160670267 conn(0x7f270408aff0 msgr2=0x7f27040d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:14.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.022+0000 7f2709400700 1 -- 172.21.15.89:0/1160670267 shutdown_connections 2024-04-02T07:37:14.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.022+0000 7f2709400700 1 -- 172.21.15.89:0/1160670267 wait complete. 2024-04-02T07:37:14.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.024+0000 7f2709400700 1 Processor -- start 2024-04-02T07:37:14.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.024+0000 7f2709400700 1 -- start start 2024-04-02T07:37:14.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2709400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26e8003bc0 0x7f27041f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:14.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2709400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27040cbd80 0x7f27040b4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:14.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2709400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27040cc200 0x7f27040b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:14.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2709400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27041f4680 con 0x7f27040cbd80 2024-04-02T07:37:14.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2709400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27041f7100 con 0x7f26e8003bc0 2024-04-02T07:37:14.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2709400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27041f73e0 con 0x7f27040cc200 2024-04-02T07:37:14.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2701e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27040cbd80 0x7f27040b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:14.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2701e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27040cbd80 0x7f27040b4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58160/0 (socket says 172.21.15.89:58160) 2024-04-02T07:37:14.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2701e00700 1 -- 172.21.15.89:0/3178824990 learned_addr learned my addr 172.21.15.89:0/3178824990 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:14.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2702800700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26e8003bc0 0x7f27041f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:14.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2703200700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27040cc200 0x7f27040b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:14.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2701e00700 1 -- 172.21.15.89:0/3178824990 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27040cc200 msgr2=0x7f27040b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:14.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2701e00700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27040cc200 0x7f27040b48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2701e00700 1 -- 172.21.15.89:0/3178824990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26e8003bc0 msgr2=0x7f27041f4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:14.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2701e00700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26e8003bc0 0x7f27041f4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2701e00700 1 -- 172.21.15.89:0/3178824990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27041f76c0 con 0x7f27040cbd80 2024-04-02T07:37:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.025+0000 7f2703200700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27040cc200 0x7f27040b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:37:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.026+0000 7f2701e00700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27040cbd80 0x7f27040b4380 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f26f000f7e0 tx=0x7f26f000fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.026+0000 7f26faa00700 1 -- 172.21.15.89:0/3178824990 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f0005320 con 0x7f27040cbd80 2024-04-02T07:37:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.026+0000 7f26faa00700 1 -- 172.21.15.89:0/3178824990 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f00054a0 con 0x7f27040cbd80 2024-04-02T07:37:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.026+0000 7f2709400700 1 -- 172.21.15.89:0/3178824990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27041f79a0 con 0x7f27040cbd80 2024-04-02T07:37:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.026+0000 7f26faa00700 1 -- 172.21.15.89:0/3178824990 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f0006af0 con 0x7f27040cbd80 2024-04-02T07:37:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.026+0000 7f2709400700 1 -- 172.21.15.89:0/3178824990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27040ca5f0 con 0x7f27040cbd80 2024-04-02T07:37:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.027+0000 7f2709400700 1 -- 172.21.15.89:0/3178824990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27040d1ca0 con 0x7f27040cbd80 2024-04-02T07:37:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.028+0000 7f26faa00700 1 -- 172.21.15.89:0/3178824990 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f26f0006c50 con 0x7f27040cbd80 2024-04-02T07:37:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.029+0000 7f26faa00700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26e406ac90 0x7f26e406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.029+0000 7f26faa00700 1 -- 172.21.15.89:0/3178824990 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26f0088dd0 con 0x7f27040cbd80 2024-04-02T07:37:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.029+0000 7f2702800700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26e406ac90 0x7f26e406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.029+0000 7f2702800700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26e406ac90 0x7f26e406d150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f26f4015040 tx=0x7f26f4007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:14.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.033+0000 7f26faa00700 1 -- 172.21.15.89:0/3178824990 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f26f0053f10 con 0x7f27040cbd80 2024-04-02T07:37:14.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.268+0000 7f2709400700 1 -- 172.21.15.89:0/3178824990 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2704161fa0 con 0x7f26e406ac90 2024-04-02T07:37:14.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.270+0000 7f26faa00700 1 -- 172.21.15.89:0/3178824990 <== mgr.4108 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f26f400cab0 con 0x7f26e406ac90 2024-04-02T07:37:14.272 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:37:14.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:37:14.272 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T07:37:14.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:37:14.272 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:37:14.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.272+0000 7f2709400700 1 -- 172.21.15.89:0/3178824990 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26e406ac90 msgr2=0x7f26e406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:14.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.272+0000 7f2709400700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26e406ac90 0x7f26e406d150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f26f4015040 tx=0x7f26f4007000 comp rx=0 tx=0).stop 2024-04-02T07:37:14.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.272+0000 7f2709400700 1 -- 172.21.15.89:0/3178824990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27040cbd80 msgr2=0x7f27040b4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:14.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.272+0000 7f2709400700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27040cbd80 0x7f27040b4380 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f26f000f7e0 tx=0x7f26f000fba0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.272+0000 7f2709400700 1 -- 172.21.15.89:0/3178824990 shutdown_connections 2024-04-02T07:37:14.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.272+0000 7f2709400700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26e406ac90 0x7f26e406d150 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.272+0000 7f2709400700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26e8003bc0 0x7f27041f4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.272+0000 7f2709400700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27040cc200 0x7f27040b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.272+0000 7f2709400700 1 --2- 172.21.15.89:0/3178824990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27040cbd80 0x7f27040b4380 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.272+0000 7f2709400700 1 -- 172.21.15.89:0/3178824990 >> 172.21.15.89:0/3178824990 conn(0x7f270408aff0 msgr2=0x7f270415bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:14.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.272+0000 7f2709400700 1 -- 172.21.15.89:0/3178824990 shutdown_connections 2024-04-02T07:37:14.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.272+0000 7f2709400700 1 -- 172.21.15.89:0/3178824990 wait complete. 2024-04-02T07:37:14.294 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T07:37:14.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.486+0000 7febd0800700 1 Processor -- start 2024-04-02T07:37:14.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.486+0000 7febd0800700 1 -- start start 2024-04-02T07:37:14.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.487+0000 7febd0800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7febc815d9d0 0x7febc80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:14.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.487+0000 7febd0800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febc80d16a0 0x7febc80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:14.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.487+0000 7febd0800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc8161420 con 0x7febc80d2370 2024-04-02T07:37:14.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.487+0000 7febd0800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc80d2820 con 0x7febc815d9d0 2024-04-02T07:37:14.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.487+0000 7febd0800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc8161700 con 0x7febc80d16a0 2024-04-02T07:37:14.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.487+0000 7febcde00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7febc815d9d0 0x7febc80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:14.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.487+0000 7febcde00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7febc815d9d0 0x7febc80d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58174/0 (socket says 172.21.15.89:58174) 2024-04-02T07:37:14.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.487+0000 7febcde00700 1 -- 172.21.15.89:0/1803141892 learned_addr learned my addr 172.21.15.89:0/1803141892 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:14.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.487+0000 7febce800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7febc80d2370 0x7febc815ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40150/0 (socket says 172.21.15.89:40150) 2024-04-02T07:37:14.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.488+0000 7febcd400700 1 --2- 172.21.15.89:0/1803141892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febc80d16a0 0x7febc80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:14.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.488+0000 7febcde00700 1 -- 172.21.15.89:0/1803141892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febc80d16a0 msgr2=0x7febc80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:14.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.488+0000 7febcde00700 1 --2- 172.21.15.89:0/1803141892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febc80d16a0 0x7febc80d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.488+0000 7febcde00700 1 -- 172.21.15.89:0/1803141892 >> v1:172.21.15.89:6789/0 conn(0x7febc80d2370 legacy=0x7febc815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:14.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.488+0000 7febcde00700 1 -- 172.21.15.89:0/1803141892 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc81619e0 con 0x7febc815d9d0 2024-04-02T07:37:14.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.488+0000 7febcde00700 1 --2- 172.21.15.89:0/1803141892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febc815d9d0 0x7febc80d3c30 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7febbc0098f0 tx=0x7febbc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600cc9a4cee321d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:14.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.488+0000 7febcca00700 1 -- 172.21.15.89:0/1803141892 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc00b8a0 con 0x7febc815d9d0 2024-04-02T07:37:14.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.488+0000 7febcca00700 1 -- 172.21.15.89:0/1803141892 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febbc00ba00 con 0x7febc815d9d0 2024-04-02T07:37:14.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.488+0000 7febcca00700 1 -- 172.21.15.89:0/1803141892 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc00bd90 con 0x7febc815d9d0 2024-04-02T07:37:14.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.489+0000 7febd0800700 1 -- 172.21.15.89:0/1803141892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febc815d9d0 msgr2=0x7febc80d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:14.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.489+0000 7febd0800700 1 --2- 172.21.15.89:0/1803141892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febc815d9d0 0x7febc80d3c30 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7febbc0098f0 tx=0x7febbc009c00 comp rx=0 tx=0).stop 2024-04-02T07:37:14.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.489+0000 7febd0800700 1 -- 172.21.15.89:0/1803141892 shutdown_connections 2024-04-02T07:37:14.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.489+0000 7febd0800700 1 --2- 172.21.15.89:0/1803141892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febc815d9d0 0x7febc80d3c30 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.489+0000 7febd0800700 1 --2- 172.21.15.89:0/1803141892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febc80d16a0 0x7febc80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.489+0000 7febd0800700 1 -- 172.21.15.89:0/1803141892 >> 172.21.15.89:0/1803141892 conn(0x7febc808aff0 msgr2=0x7febc80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:14.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.489+0000 7febd0800700 1 -- 172.21.15.89:0/1803141892 shutdown_connections 2024-04-02T07:37:14.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.489+0000 7febd0800700 1 -- 172.21.15.89:0/1803141892 wait complete. 2024-04-02T07:37:14.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.491+0000 7febd0800700 1 Processor -- start 2024-04-02T07:37:14.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.492+0000 7febd0800700 1 -- start start 2024-04-02T07:37:14.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.493+0000 7febd0800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febc80d16a0 0x7febc8155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:14.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.493+0000 7febd0800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febc80d2370 0x7febc80b8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:14.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.493+0000 7febd0800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febc815d9d0 0x7febc8159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:14.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.493+0000 7febd0800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc815e810 con 0x7febc80d16a0 2024-04-02T07:37:14.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.493+0000 7febd0800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc80b8df0 con 0x7febc815d9d0 2024-04-02T07:37:14.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.493+0000 7febd0800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc815dee0 con 0x7febc80d2370 2024-04-02T07:37:14.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.493+0000 7febcde00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febc80d16a0 0x7febc8155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.493+0000 7febcde00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febc80d16a0 0x7febc8155020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58186/0 (socket says 172.21.15.89:58186) 2024-04-02T07:37:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.493+0000 7febcde00700 1 -- 172.21.15.89:0/4153108778 learned_addr learned my addr 172.21.15.89:0/4153108778 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.493+0000 7febcd400700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febc80d2370 0x7febc80b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.493+0000 7febce800700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febc815d9d0 0x7febc8159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.493+0000 7febcde00700 1 -- 172.21.15.89:0/4153108778 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febc80d2370 msgr2=0x7febc80b8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.494+0000 7febcde00700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febc80d2370 0x7febc80b8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.494+0000 7febcde00700 1 -- 172.21.15.89:0/4153108778 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febc815d9d0 msgr2=0x7febc8159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.494+0000 7febcde00700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febc815d9d0 0x7febc8159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.494+0000 7febcde00700 1 -- 172.21.15.89:0/4153108778 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc009550 con 0x7febc80d16a0 2024-04-02T07:37:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.494+0000 7febcd400700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febc80d2370 0x7febc80b8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.494+0000 7febcde00700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febc80d16a0 0x7febc8155020 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7febbc009c40 tx=0x7febbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.494+0000 7febc2000700 1 -- 172.21.15.89:0/4153108778 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc004590 con 0x7febc80d16a0 2024-04-02T07:37:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.494+0000 7febc2000700 1 -- 172.21.15.89:0/4153108778 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febbc01e070 con 0x7febc80d16a0 2024-04-02T07:37:14.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.494+0000 7febc2000700 1 -- 172.21.15.89:0/4153108778 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc0038e0 con 0x7febc80d16a0 2024-04-02T07:37:14.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.494+0000 7febd0800700 1 -- 172.21.15.89:0/4153108778 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc815e1c0 con 0x7febc80d16a0 2024-04-02T07:37:14.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.494+0000 7febd0800700 1 -- 172.21.15.89:0/4153108778 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc8155560 con 0x7febc80d16a0 2024-04-02T07:37:14.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.495+0000 7febd0800700 1 -- 172.21.15.89:0/4153108778 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc80d1ca0 con 0x7febc80d16a0 2024-04-02T07:37:14.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.496+0000 7febc2000700 1 -- 172.21.15.89:0/4153108778 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7febbc003a40 con 0x7febc80d16a0 2024-04-02T07:37:14.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.497+0000 7febc2000700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febac06ac90 0x7febac06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:14.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.497+0000 7febc2000700 1 -- 172.21.15.89:0/4153108778 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7febbc020070 con 0x7febc80d16a0 2024-04-02T07:37:14.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.497+0000 7febcd400700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febac06ac90 0x7febac06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:14.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.498+0000 7febcd400700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febac06ac90 0x7febac06d150 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7febc81f6ce0 tx=0x7febb8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:14.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.502+0000 7febc2000700 1 -- 172.21.15.89:0/4153108778 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7febbc0559c0 con 0x7febc80d16a0 2024-04-02T07:37:14.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:14.759+0000 7febd0800700 1 -- 172.21.15.89:0/4153108778 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7febc8067720 con 0x7febc80d16a0 2024-04-02T07:37:15.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:15.773+0000 7febc2000700 1 -- 172.21.15.89:0/4153108778 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 6) v1 ==== 89154+0+0 (secure 0 0 0) 0x7febbc003cf0 con 0x7febc80d16a0 2024-04-02T07:37:15.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:15.774+0000 7febc2000700 1 -- 172.21.15.89:0/4153108778 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7febbc05ca70 con 0x7febc80d16a0 2024-04-02T07:37:15.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:15.777+0000 7febd0800700 1 -- 172.21.15.89:0/4153108778 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febac06ac90 msgr2=0x7febac06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:15.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:15.777+0000 7febd0800700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febac06ac90 0x7febac06d150 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7febc81f6ce0 tx=0x7febb8008040 comp rx=0 tx=0).stop 2024-04-02T07:37:15.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:15.777+0000 7febd0800700 1 -- 172.21.15.89:0/4153108778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febc80d16a0 msgr2=0x7febc8155020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:15.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:15.777+0000 7febd0800700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febc80d16a0 0x7febc8155020 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7febbc009c40 tx=0x7febbc004750 comp rx=0 tx=0).stop 2024-04-02T07:37:15.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:15.777+0000 7febd0800700 1 -- 172.21.15.89:0/4153108778 shutdown_connections 2024-04-02T07:37:15.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:15.777+0000 7febd0800700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febac06ac90 0x7febac06d150 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:15.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:15.777+0000 7febd0800700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febc815d9d0 0x7febc8159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:15.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:15.777+0000 7febd0800700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febc80d16a0 0x7febc8155020 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:15.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:15.777+0000 7febd0800700 1 --2- 172.21.15.89:0/4153108778 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febc80d2370 0x7febc80b8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:15.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:15.777+0000 7febd0800700 1 -- 172.21.15.89:0/4153108778 >> 172.21.15.89:0/4153108778 conn(0x7febc808aff0 msgr2=0x7febc80b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:15.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:15.777+0000 7febd0800700 1 -- 172.21.15.89:0/4153108778 shutdown_connections 2024-04-02T07:37:15.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:15.777+0000 7febd0800700 1 -- 172.21.15.89:0/4153108778 wait complete. 2024-04-02T07:37:15.801 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T07:37:15.986 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:15.984+0000 7f71c2adf240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:37:16.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.004+0000 7fbf66000700 1 Processor -- start 2024-04-02T07:37:16.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.004+0000 7fbf66000700 1 -- start start 2024-04-02T07:37:16.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.005+0000 7fbf66000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf6015d9d0 0x7fbf600d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:16.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.005+0000 7fbf66000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbf600d16a0 0x7fbf600d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:16.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.006+0000 7fbf66000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf60161420 con 0x7fbf600d2370 2024-04-02T07:37:16.008 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:16.006+0000 7f355b382240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:37:16.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.006+0000 7fbf66000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf600d2820 con 0x7fbf600d16a0 2024-04-02T07:37:16.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.006+0000 7fbf66000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf60161700 con 0x7fbf6015d9d0 2024-04-02T07:37:16.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.006+0000 7fbf5ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbf600d16a0 0x7fbf600d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.006+0000 7fbf5ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbf600d16a0 0x7fbf600d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58216/0 (socket says 172.21.15.89:58216) 2024-04-02T07:37:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.006+0000 7fbf5ea00700 1 -- 172.21.15.89:0/3477009530 learned_addr learned my addr 172.21.15.89:0/3477009530 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.006+0000 7fbf5f400700 1 --2- 172.21.15.89:0/3477009530 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf6015d9d0 0x7fbf600d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.006+0000 7fbf5ea00700 1 -- 172.21.15.89:0/3477009530 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf6015d9d0 msgr2=0x7fbf600d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.006+0000 7fbf5ea00700 1 --2- 172.21.15.89:0/3477009530 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf6015d9d0 0x7fbf600d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.006+0000 7fbf5ea00700 1 -- 172.21.15.89:0/3477009530 >> v1:172.21.15.113:6789/0 conn(0x7fbf600d2370 legacy=0x7fbf6015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.006+0000 7fbf5ea00700 1 -- 172.21.15.89:0/3477009530 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf601619e0 con 0x7fbf600d16a0 2024-04-02T07:37:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.007+0000 7fbf5ea00700 1 --2- 172.21.15.89:0/3477009530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf600d16a0 0x7fbf600d1e30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c006fd0 tx=0x7fbf4c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610109a9123ff199 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.007+0000 7fbf5e000700 1 -- 172.21.15.89:0/3477009530 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c005e30 con 0x7fbf600d16a0 2024-04-02T07:37:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.007+0000 7fbf5e000700 1 -- 172.21.15.89:0/3477009530 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4c004030 con 0x7fbf600d16a0 2024-04-02T07:37:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.007+0000 7fbf5e000700 1 -- 172.21.15.89:0/3477009530 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c0043e0 con 0x7fbf600d16a0 2024-04-02T07:37:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.007+0000 7fbf66000700 1 -- 172.21.15.89:0/3477009530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf600d16a0 msgr2=0x7fbf600d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.007+0000 7fbf66000700 1 --2- 172.21.15.89:0/3477009530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf600d16a0 0x7fbf600d1e30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c006fd0 tx=0x7fbf4c009bd0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.008+0000 7fbf66000700 1 -- 172.21.15.89:0/3477009530 shutdown_connections 2024-04-02T07:37:16.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.008+0000 7fbf66000700 1 --2- 172.21.15.89:0/3477009530 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf6015d9d0 0x7fbf600d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.008+0000 7fbf66000700 1 --2- 172.21.15.89:0/3477009530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf600d16a0 0x7fbf600d1e30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.008+0000 7fbf66000700 1 -- 172.21.15.89:0/3477009530 >> 172.21.15.89:0/3477009530 conn(0x7fbf6008aff0 msgr2=0x7fbf600d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:16.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.008+0000 7fbf66000700 1 -- 172.21.15.89:0/3477009530 shutdown_connections 2024-04-02T07:37:16.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.008+0000 7fbf66000700 1 -- 172.21.15.89:0/3477009530 wait complete. 2024-04-02T07:37:16.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.009+0000 7fbf66000700 1 Processor -- start 2024-04-02T07:37:16.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.010+0000 7fbf66000700 1 -- start start 2024-04-02T07:37:16.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.010+0000 7fbf66000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf600d16a0 0x7fbf601589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.010+0000 7fbf66000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf600d2370 0x7fbf6015c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf66000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf6015d9d0 0x7fbf6014e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf66000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6015ca50 con 0x7fbf6015d9d0 2024-04-02T07:37:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf66000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6014eeb0 con 0x7fbf600d2370 2024-04-02T07:37:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf66000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600b7980 con 0x7fbf600d16a0 2024-04-02T07:37:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf5fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf6015d9d0 0x7fbf6014e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf5f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf600d16a0 0x7fbf601589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf5ea00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf600d2370 0x7fbf6015c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf5f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf600d16a0 0x7fbf601589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:36848/0 (socket says 172.21.15.89:36848) 2024-04-02T07:37:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf5f400700 1 -- 172.21.15.89:0/4035535182 learned_addr learned my addr 172.21.15.89:0/4035535182 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf5fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf6015d9d0 0x7fbf6014e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58226/0 (socket says 172.21.15.89:58226) 2024-04-02T07:37:16.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf5f400700 1 -- 172.21.15.89:0/4035535182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf600d2370 msgr2=0x7fbf6015c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:16.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf5f400700 1 --2- 172.21.15.89:0/4035535182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf600d2370 0x7fbf6015c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf5f400700 1 -- 172.21.15.89:0/4035535182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf6015d9d0 msgr2=0x7fbf6014e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:16.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf5f400700 1 --2- 172.21.15.89:0/4035535182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf6015d9d0 0x7fbf6014e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.011+0000 7fbf5f400700 1 -- 172.21.15.89:0/4035535182 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c009500 con 0x7fbf600d16a0 2024-04-02T07:37:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.012+0000 7fbf5f400700 1 --2- 172.21.15.89:0/4035535182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf600d16a0 0x7fbf601589b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbf5000e870 tx=0x7fbf5000eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.012+0000 7fbf57400700 1 -- 172.21.15.89:0/4035535182 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50009590 con 0x7fbf600d16a0 2024-04-02T07:37:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.012+0000 7fbf57400700 1 -- 172.21.15.89:0/4035535182 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf50013680 con 0x7fbf600d16a0 2024-04-02T07:37:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.012+0000 7fbf66000700 1 -- 172.21.15.89:0/4035535182 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf600b7c60 con 0x7fbf600d16a0 2024-04-02T07:37:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.012+0000 7fbf57400700 1 -- 172.21.15.89:0/4035535182 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf500139f0 con 0x7fbf600d16a0 2024-04-02T07:37:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.012+0000 7fbf66000700 1 -- 172.21.15.89:0/4035535182 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf600b8220 con 0x7fbf600d16a0 2024-04-02T07:37:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.013+0000 7fbf66000700 1 -- 172.21.15.89:0/4035535182 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf600d1ca0 con 0x7fbf600d16a0 2024-04-02T07:37:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.014+0000 7fbf57400700 1 -- 172.21.15.89:0/4035535182 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 6) v1 ==== 89154+0+0 (secure 0 0 0) 0x7fbf50013b50 con 0x7fbf600d16a0 2024-04-02T07:37:16.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.015+0000 7fbf57400700 1 --2- 172.21.15.89:0/4035535182 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbf4006ace0 0x7fbf4006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:16.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.015+0000 7fbf5ea00700 1 -- 172.21.15.89:0/4035535182 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbf4006ace0 msgr2=0x7fbf4006d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:37:16.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.015+0000 7fbf57400700 1 -- 172.21.15.89:0/4035535182 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf500891e0 con 0x7fbf600d16a0 2024-04-02T07:37:16.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.015+0000 7fbf5ea00700 1 --2- 172.21.15.89:0/4035535182 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbf4006ace0 0x7fbf4006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:37:16.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.020+0000 7fbf57400700 1 -- 172.21.15.89:0/4035535182 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf500543b0 con 0x7fbf600d16a0 2024-04-02T07:37:16.173 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:16.170+0000 7f71c2adf240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:37:16.198 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:16.197+0000 7f355b382240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:37:16.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.216+0000 7fbf5ea00700 1 -- 172.21.15.89:0/4035535182 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbf4006ace0 msgr2=0x7fbf4006d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:37:16.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.216+0000 7fbf5ea00700 1 --2- 172.21.15.89:0/4035535182 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbf4006ace0 0x7fbf4006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:37:16.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.269+0000 7fbf66000700 1 -- 172.21.15.89:0/4035535182 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fbf60067720 con 0x7fbf600d16a0 2024-04-02T07:37:16.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.275+0000 7fbf57400700 1 -- 172.21.15.89:0/4035535182 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbf500579d0 con 0x7fbf600d16a0 2024-04-02T07:37:16.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.277+0000 7fbf66000700 1 -- 172.21.15.89:0/4035535182 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbf4006ace0 msgr2=0x7fbf4006d1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:37:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.277+0000 7fbf66000700 1 --2- 172.21.15.89:0/4035535182 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbf4006ace0 0x7fbf4006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.277+0000 7fbf66000700 1 -- 172.21.15.89:0/4035535182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf600d16a0 msgr2=0x7fbf601589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.277+0000 7fbf66000700 1 --2- 172.21.15.89:0/4035535182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf600d16a0 0x7fbf601589b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbf5000e870 tx=0x7fbf5000eb80 comp rx=0 tx=0).stop 2024-04-02T07:37:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.278+0000 7fbf66000700 1 -- 172.21.15.89:0/4035535182 shutdown_connections 2024-04-02T07:37:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.278+0000 7fbf66000700 1 --2- 172.21.15.89:0/4035535182 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbf4006ace0 0x7fbf4006d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.278+0000 7fbf66000700 1 --2- 172.21.15.89:0/4035535182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf6015d9d0 0x7fbf6014e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.278+0000 7fbf66000700 1 --2- 172.21.15.89:0/4035535182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf600d16a0 0x7fbf601589b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.278+0000 7fbf66000700 1 --2- 172.21.15.89:0/4035535182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf600d2370 0x7fbf6015c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.278+0000 7fbf66000700 1 -- 172.21.15.89:0/4035535182 >> 172.21.15.89:0/4035535182 conn(0x7fbf6008aff0 msgr2=0x7fbf6006d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.278+0000 7fbf66000700 1 -- 172.21.15.89:0/4035535182 shutdown_connections 2024-04-02T07:37:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.278+0000 7fbf66000700 1 -- 172.21.15.89:0/4035535182 wait complete. 2024-04-02T07:37:16.298 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T07:37:16.490 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:16.488+0000 7f71c2adf240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:37:16.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.495+0000 7f26f0000700 1 Processor -- start 2024-04-02T07:37:16.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.495+0000 7f26f0000700 1 -- start start 2024-04-02T07:37:16.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.496+0000 7f26f0000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26e8099a60 0x7f26e80d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:16.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.496+0000 7f26f0000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f26e815d990 0x7f26e8099e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:16.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.496+0000 7f26f0000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e80d0970 con 0x7f26e809a750 2024-04-02T07:37:16.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.496+0000 7f26f0000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e806c020 con 0x7f26e815d990 2024-04-02T07:37:16.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.496+0000 7f26f0000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e809a380 con 0x7f26e8099a60 2024-04-02T07:37:16.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.497+0000 7f26ee000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f26e809a750 0x7f26e806c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40162/0 (socket says 172.21.15.89:40162) 2024-04-02T07:37:16.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.497+0000 7f26ee000700 1 -- 172.21.15.89:0/2043631140 learned_addr learned my addr 172.21.15.89:0/2043631140 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:16.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.497+0000 7f26ed600700 1 --2- 172.21.15.89:0/2043631140 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26e8099a60 0x7f26e80d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:16.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.497+0000 7f26ecc00700 1 --2- 172.21.15.89:0/2043631140 >> v2:172.21.15.113:3300/0 conn(0x7f26e815d990 0x7f26e8099e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:16.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.497+0000 7f26e7e00700 1 -- 172.21.15.89:0/2043631140 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673176999 0 0) 0x7f26e80d0970 con 0x7f26e809a750 2024-04-02T07:37:16.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.497+0000 7f26e7e00700 1 -- 172.21.15.89:0/2043631140 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26d00035b0 con 0x7f26e809a750 2024-04-02T07:37:16.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.497+0000 7f26ed600700 1 -- 172.21.15.89:0/2043631140 >> v2:172.21.15.113:3300/0 conn(0x7f26e815d990 msgr2=0x7f26e8099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:16.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.497+0000 7f26ed600700 1 --2- 172.21.15.89:0/2043631140 >> v2:172.21.15.113:3300/0 conn(0x7f26e815d990 0x7f26e8099e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.497+0000 7f26ed600700 1 -- 172.21.15.89:0/2043631140 >> v1:172.21.15.89:6789/0 conn(0x7f26e809a750 legacy=0x7f26e806c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:16.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.497+0000 7f26ed600700 1 -- 172.21.15.89:0/2043631140 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e80c81a0 con 0x7f26e8099a60 2024-04-02T07:37:16.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.498+0000 7f26ed600700 1 --2- 172.21.15.89:0/2043631140 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26e8099a60 0x7f26e80d0060 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009840 tx=0x7f26dc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9854f40d01739a8e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:16.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.498+0000 7f26e7e00700 1 -- 172.21.15.89:0/2043631140 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc011830 con 0x7f26e8099a60 2024-04-02T07:37:16.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.498+0000 7f26e7e00700 1 -- 172.21.15.89:0/2043631140 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc011990 con 0x7f26e8099a60 2024-04-02T07:37:16.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.498+0000 7f26e7e00700 1 -- 172.21.15.89:0/2043631140 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc011d20 con 0x7f26e8099a60 2024-04-02T07:37:16.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.498+0000 7f26f0000700 1 -- 172.21.15.89:0/2043631140 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26e8099a60 msgr2=0x7f26e80d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:16.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.498+0000 7f26f0000700 1 --2- 172.21.15.89:0/2043631140 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26e8099a60 0x7f26e80d0060 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009840 tx=0x7f26dc009b50 comp rx=0 tx=0).stop 2024-04-02T07:37:16.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.499+0000 7f26f0000700 1 -- 172.21.15.89:0/2043631140 shutdown_connections 2024-04-02T07:37:16.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.499+0000 7f26f0000700 1 --2- 172.21.15.89:0/2043631140 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26e8099a60 0x7f26e80d0060 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.499+0000 7f26f0000700 1 --2- 172.21.15.89:0/2043631140 >> v2:172.21.15.113:3300/0 conn(0x7f26e815d990 0x7f26e8099e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.499+0000 7f26f0000700 1 -- 172.21.15.89:0/2043631140 >> 172.21.15.89:0/2043631140 conn(0x7f26e80b2e20 msgr2=0x7f26e80b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:16.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.499+0000 7f26f0000700 1 -- 172.21.15.89:0/2043631140 shutdown_connections 2024-04-02T07:37:16.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.499+0000 7f26f0000700 1 -- 172.21.15.89:0/2043631140 wait complete. 2024-04-02T07:37:16.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.500+0000 7f26f0000700 1 Processor -- start 2024-04-02T07:37:16.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.500+0000 7f26f0000700 1 -- start start 2024-04-02T07:37:16.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.501+0000 7f26f0000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26e8099a60 0x7f26e80ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:16.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.501+0000 7f26f0000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26e809a750 0x7f26e815b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:16.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.501+0000 7f26f0000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e815d990 0x7f26e815b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:16.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.501+0000 7f26f0000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8160090 con 0x7f26e815d990 2024-04-02T07:37:16.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.501+0000 7f26f0000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8160d90 con 0x7f26e809a750 2024-04-02T07:37:16.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.501+0000 7f26f0000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e80c5510 con 0x7f26e8099a60 2024-04-02T07:37:16.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.502+0000 7f26ee000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e815d990 0x7f26e815b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:16.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.502+0000 7f26ed600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26e8099a60 0x7f26e80ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:16.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.502+0000 7f26ecc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26e809a750 0x7f26e815b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:16.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.502+0000 7f26ed600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26e8099a60 0x7f26e80ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:36856/0 (socket says 172.21.15.89:36856) 2024-04-02T07:37:16.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.502+0000 7f26ed600700 1 -- 172.21.15.89:0/3550699338 learned_addr learned my addr 172.21.15.89:0/3550699338 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:16.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.502+0000 7f26ed600700 1 -- 172.21.15.89:0/3550699338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26e809a750 msgr2=0x7f26e815b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:16.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.502+0000 7f26ed600700 1 --2- 172.21.15.89:0/3550699338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26e809a750 0x7f26e815b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.502+0000 7f26ed600700 1 -- 172.21.15.89:0/3550699338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e815d990 msgr2=0x7f26e815b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:16.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.502+0000 7f26ed600700 1 --2- 172.21.15.89:0/3550699338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e815d990 0x7f26e815b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.502+0000 7f26ed600700 1 -- 172.21.15.89:0/3550699338 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc0094a0 con 0x7f26e8099a60 2024-04-02T07:37:16.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.502+0000 7f26ed600700 1 --2- 172.21.15.89:0/3550699338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26e8099a60 0x7f26e80ca840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009fd0 tx=0x7f26dc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:16.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.503+0000 7f26e5600700 1 -- 172.21.15.89:0/3550699338 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc004590 con 0x7f26e8099a60 2024-04-02T07:37:16.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.503+0000 7f26e5600700 1 -- 172.21.15.89:0/3550699338 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc01e070 con 0x7f26e8099a60 2024-04-02T07:37:16.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.503+0000 7f26f0000700 1 -- 172.21.15.89:0/3550699338 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e8160b10 con 0x7f26e8099a60 2024-04-02T07:37:16.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.503+0000 7f26e5600700 1 -- 172.21.15.89:0/3550699338 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc0038e0 con 0x7f26e8099a60 2024-04-02T07:37:16.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.503+0000 7f26f0000700 1 -- 172.21.15.89:0/3550699338 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e815bc30 con 0x7f26e8099a60 2024-04-02T07:37:16.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.504+0000 7f26f0000700 1 -- 172.21.15.89:0/3550699338 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e80c5000 con 0x7f26e8099a60 2024-04-02T07:37:16.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.505+0000 7f26e5600700 1 -- 172.21.15.89:0/3550699338 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 6) v1 ==== 89154+0+0 (secure 0 0 0) 0x7f26dc003a40 con 0x7f26e8099a60 2024-04-02T07:37:16.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.505+0000 7f26e5600700 1 --2- 172.21.15.89:0/3550699338 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26c806ace0 0x7f26c806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:16.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.506+0000 7f26ecc00700 1 -- 172.21.15.89:0/3550699338 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26c806ace0 msgr2=0x7f26c806d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:37:16.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.506+0000 7f26e5600700 1 -- 172.21.15.89:0/3550699338 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26dc020070 con 0x7f26e8099a60 2024-04-02T07:37:16.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.506+0000 7f26ecc00700 1 --2- 172.21.15.89:0/3550699338 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26c806ace0 0x7f26c806d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:37:16.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.510+0000 7f26e5600700 1 -- 172.21.15.89:0/3550699338 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f26e80c5000 con 0x7f26e8099a60 2024-04-02T07:37:16.540 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:16.539+0000 7f355b382240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:37:16.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.707+0000 7f26ecc00700 1 -- 172.21.15.89:0/3550699338 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26c806ace0 msgr2=0x7f26c806d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:37:16.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.707+0000 7f26ecc00700 1 --2- 172.21.15.89:0/3550699338 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26c806ace0 0x7f26c806d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:37:16.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.719+0000 7f26f0000700 1 -- 172.21.15.89:0/3550699338 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f26e80c5000 con 0x7f26e8099a60 2024-04-02T07:37:16.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.725+0000 7f26e5600700 1 -- 172.21.15.89:0/3550699338 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7f26e80c5000 con 0x7f26e8099a60 2024-04-02T07:37:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.728+0000 7f26f0000700 1 -- 172.21.15.89:0/3550699338 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26c806ace0 msgr2=0x7f26c806d1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:37:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.728+0000 7f26f0000700 1 --2- 172.21.15.89:0/3550699338 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26c806ace0 0x7f26c806d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.728+0000 7f26f0000700 1 -- 172.21.15.89:0/3550699338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26e8099a60 msgr2=0x7f26e80ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.728+0000 7f26f0000700 1 --2- 172.21.15.89:0/3550699338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26e8099a60 0x7f26e80ca840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009fd0 tx=0x7f26dc004750 comp rx=0 tx=0).stop 2024-04-02T07:37:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.728+0000 7f26f0000700 1 -- 172.21.15.89:0/3550699338 shutdown_connections 2024-04-02T07:37:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.728+0000 7f26f0000700 1 --2- 172.21.15.89:0/3550699338 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26c806ace0 0x7f26c806d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.728+0000 7f26f0000700 1 --2- 172.21.15.89:0/3550699338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26e815d990 0x7f26e815b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.728+0000 7f26f0000700 1 --2- 172.21.15.89:0/3550699338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26e8099a60 0x7f26e80ca840 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.728+0000 7f26f0000700 1 --2- 172.21.15.89:0/3550699338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26e809a750 0x7f26e815b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.728+0000 7f26f0000700 1 -- 172.21.15.89:0/3550699338 >> 172.21.15.89:0/3550699338 conn(0x7f26e80b2e20 msgr2=0x7f26e80b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.728+0000 7f26f0000700 1 -- 172.21.15.89:0/3550699338 shutdown_connections 2024-04-02T07:37:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:16.728+0000 7f26f0000700 1 -- 172.21.15.89:0/3550699338 wait complete. 2024-04-02T07:37:16.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:16.749 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-02T07:37:16.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:16.802 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2024-04-02T07:37:16.849 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2024-04-02T07:37:17.018 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:17.016+0000 7f71c2adf240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:37:17.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.048+0000 7f15d5e00700 1 Processor -- start 2024-04-02T07:37:17.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.048+0000 7f15d5e00700 1 -- start start 2024-04-02T07:37:17.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.049+0000 7f15d5e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f15d00cd020 0x7f15d00cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:17.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.049+0000 7f15d5e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d00cdae0 con 0x7f15d00cbb30 2024-04-02T07:37:17.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.050+0000 7f15d5e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d00cddc0 con 0x7f15d00b2070 2024-04-02T07:37:17.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.050+0000 7f15d5e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d00ce0a0 con 0x7f15d00cd020 2024-04-02T07:37:17.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.050+0000 7f15cea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f15d00cbb30 0x7f15d00cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40164/0 (socket says 172.21.15.89:40164) 2024-04-02T07:37:17.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.050+0000 7f15cea00700 1 -- 172.21.15.89:0/3445428073 learned_addr learned my addr 172.21.15.89:0/3445428073 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:17.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.050+0000 7f15ce000700 1 --2- 172.21.15.89:0/3445428073 >> v2:172.21.15.113:3300/0 conn(0x7f15d00cd020 0x7f15d00cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:17.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.050+0000 7f15cd600700 1 -- 172.21.15.89:0/3445428073 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146875856 0 0) 0x7f15d00cdae0 con 0x7f15d00cbb30 2024-04-02T07:37:17.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.050+0000 7f15cd600700 1 -- 172.21.15.89:0/3445428073 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15b40035b0 con 0x7f15d00cbb30 2024-04-02T07:37:17.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.050+0000 7f15cd600700 1 -- 172.21.15.89:0/3445428073 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119932498 0 0) 0x7f15d00cddc0 con 0x7f15d00b2070 2024-04-02T07:37:17.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.050+0000 7f15cd600700 1 -- 172.21.15.89:0/3445428073 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15d00cdae0 con 0x7f15d00b2070 2024-04-02T07:37:17.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.050+0000 7f15ce000700 1 -- 172.21.15.89:0/3445428073 >> v1:172.21.15.113:6789/0 conn(0x7f15d00b2070 legacy=0x7f15d006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:17.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.050+0000 7f15ce000700 1 -- 172.21.15.89:0/3445428073 >> v1:172.21.15.89:6789/0 conn(0x7f15d00cbb30 legacy=0x7f15d00cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:17.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.050+0000 7f15ce000700 1 -- 172.21.15.89:0/3445428073 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d006dc20 con 0x7f15d00cd020 2024-04-02T07:37:17.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.051+0000 7f15ce000700 1 --2- 172.21.15.89:0/3445428073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d00cd020 0x7f15d00cd480 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f15bc009840 tx=0x7f15bc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91f5e44b04e8bff1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:17.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.051+0000 7f15cd600700 1 -- 172.21.15.89:0/3445428073 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc011830 con 0x7f15d00cd020 2024-04-02T07:37:17.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.051+0000 7f15cd600700 1 -- 172.21.15.89:0/3445428073 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15bc011990 con 0x7f15d00cd020 2024-04-02T07:37:17.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.051+0000 7f15cd600700 1 -- 172.21.15.89:0/3445428073 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc011d20 con 0x7f15d00cd020 2024-04-02T07:37:17.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.051+0000 7f15d5e00700 1 -- 172.21.15.89:0/3445428073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d00cd020 msgr2=0x7f15d00cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:17.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.051+0000 7f15d5e00700 1 --2- 172.21.15.89:0/3445428073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d00cd020 0x7f15d00cd480 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f15bc009840 tx=0x7f15bc009b50 comp rx=0 tx=0).stop 2024-04-02T07:37:17.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.052+0000 7f15d5e00700 1 -- 172.21.15.89:0/3445428073 shutdown_connections 2024-04-02T07:37:17.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.052+0000 7f15d5e00700 1 --2- 172.21.15.89:0/3445428073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d00cd020 0x7f15d00cd480 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:17.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.052+0000 7f15d5e00700 1 -- 172.21.15.89:0/3445428073 >> 172.21.15.89:0/3445428073 conn(0x7f15d00b0d70 msgr2=0x7f15d00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:17.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.052+0000 7f15d5e00700 1 -- 172.21.15.89:0/3445428073 shutdown_connections 2024-04-02T07:37:17.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.052+0000 7f15d5e00700 1 -- 172.21.15.89:0/3445428073 wait complete. 2024-04-02T07:37:17.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.053+0000 7f15d5e00700 1 Processor -- start 2024-04-02T07:37:17.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.053+0000 7f15d5e00700 1 -- start start 2024-04-02T07:37:17.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.054+0000 7f15d5e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d00b2070 0x7f15d0155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:17.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.054+0000 7f15d5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15d00cbb30 0x7f15d0154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:17.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.054+0000 7f15d5e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00cd020 0x7f15d0099c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:17.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.054+0000 7f15d5e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d01583f0 con 0x7f15d00cbb30 2024-04-02T07:37:17.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.054+0000 7f15d5e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d0157830 con 0x7f15d00b2070 2024-04-02T07:37:17.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.054+0000 7f15d5e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d00991d0 con 0x7f15d00cd020 2024-04-02T07:37:17.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.055+0000 7f15ce000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15d00cbb30 0x7f15d0154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:17.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.055+0000 7f15ce000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15d00cbb30 0x7f15d0154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58240/0 (socket says 172.21.15.89:58240) 2024-04-02T07:37:17.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.055+0000 7f15ce000700 1 -- 172.21.15.89:0/4052986524 learned_addr learned my addr 172.21.15.89:0/4052986524 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:17.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.055+0000 7f15cea00700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d00b2070 0x7f15d0155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:17.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.055+0000 7f15cf400700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00cd020 0x7f15d0099c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:17.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.055+0000 7f15ce000700 1 -- 172.21.15.89:0/4052986524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00cd020 msgr2=0x7f15d0099c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:17.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.055+0000 7f15ce000700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00cd020 0x7f15d0099c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:17.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.055+0000 7f15ce000700 1 -- 172.21.15.89:0/4052986524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d00b2070 msgr2=0x7f15d0155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:17.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.055+0000 7f15ce000700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d00b2070 0x7f15d0155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:17.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.055+0000 7f15ce000700 1 -- 172.21.15.89:0/4052986524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15bc0094a0 con 0x7f15d00cbb30 2024-04-02T07:37:17.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.055+0000 7f15ce000700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15d00cbb30 0x7f15d0154c60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f15bc014040 tx=0x7f15bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:17.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.056+0000 7f15c7400700 1 -- 172.21.15.89:0/4052986524 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc01a6e0 con 0x7f15d00cbb30 2024-04-02T07:37:17.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.056+0000 7f15c7400700 1 -- 172.21.15.89:0/4052986524 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15bc01a840 con 0x7f15d00cbb30 2024-04-02T07:37:17.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.056+0000 7f15c7400700 1 -- 172.21.15.89:0/4052986524 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc01abb0 con 0x7f15d00cbb30 2024-04-02T07:37:17.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.056+0000 7f15d5e00700 1 -- 172.21.15.89:0/4052986524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d0099790 con 0x7f15d00cbb30 2024-04-02T07:37:17.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.056+0000 7f15d5e00700 1 -- 172.21.15.89:0/4052986524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d0157d60 con 0x7f15d00cbb30 2024-04-02T07:37:17.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.057+0000 7f15d5e00700 1 -- 172.21.15.89:0/4052986524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d0099920 con 0x7f15d00cbb30 2024-04-02T07:37:17.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.059+0000 7f15c7400700 1 -- 172.21.15.89:0/4052986524 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 6) v1 ==== 89154+0+0 (secure 0 0 0) 0x7f15bc022070 con 0x7f15d00cbb30 2024-04-02T07:37:17.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.060+0000 7f15c7400700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 0x7f15b006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:17.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.060+0000 7f15c7400700 1 -- 172.21.15.89:0/4052986524 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15bc089df0 con 0x7f15d00cbb30 2024-04-02T07:37:17.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.060+0000 7f15cea00700 1 -- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 msgr2=0x7f15b006d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:37:17.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.060+0000 7f15cea00700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 0x7f15b006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:37:17.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.065+0000 7f15c7400700 1 -- 172.21.15.89:0/4052986524 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15bc055090 con 0x7f15d00cbb30 2024-04-02T07:37:17.096 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:17.095+0000 7f355b382240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:37:17.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.261+0000 7f15cea00700 1 -- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 msgr2=0x7f15b006d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:37:17.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.261+0000 7f15cea00700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 0x7f15b006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:37:17.273 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:17.271+0000 7f71c2adf240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:37:17.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.300+0000 7f15d5e00700 1 -- 172.21.15.89:0/4052986524 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f15d015bdd0 con 0x7f15b006ace0 2024-04-02T07:37:17.356 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:17.355+0000 7f355b382240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:37:17.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.662+0000 7f15cea00700 1 -- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 msgr2=0x7f15b006d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:37:17.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:17.662+0000 7f15cea00700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 0x7f15b006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T07:37:17.724 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:17.722+0000 7f71c2adf240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:37:17.808 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:17.807+0000 7f355b382240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:37:18.427 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:18.425+0000 7f71c2adf240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:37:18.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:18.463+0000 7f15cea00700 1 -- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 msgr2=0x7f15b006d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:37:18.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:18.463+0000 7f15cea00700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 0x7f15b006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T07:37:18.514 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:18.512+0000 7f355b382240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:37:18.517 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:18.515+0000 7f71c2adf240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:37:18.606 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:18.604+0000 7f355b382240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:37:18.675 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:18.673+0000 7f71c2adf240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:37:18.766 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:18.764+0000 7f355b382240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:37:18.886 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:18.884+0000 7f71c2adf240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:37:18.976 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:18.974+0000 7f71c2adf240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:37:18.978 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:18.977+0000 7f355b382240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:37:19.069 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:19.068+0000 7f355b382240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:37:19.074 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:19.072+0000 7f71c2adf240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:37:19.166 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:19.164+0000 7f355b382240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:37:19.170 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:19.168+0000 7f71c2adf240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:37:19.263 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:19.261+0000 7f355b382240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:37:19.268 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:19.266+0000 7f71c2adf240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:37:19.360 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:19.358+0000 7f355b382240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:37:19.478 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:19.476+0000 7f71c2adf240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:37:19.570 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:19.568+0000 7f355b382240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:37:19.729 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:19.727+0000 7f71c2adf240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:37:19.823 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:19.821+0000 7f355b382240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:37:19.918 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:19.916+0000 7f71c2adf240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:37:20.009 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:20.007+0000 7f71c2adf240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:37:20.011 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:20.010+0000 7f355b382240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:37:20.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:20.065+0000 7f15cea00700 1 -- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 msgr2=0x7f15b006d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:37:20.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:20.065+0000 7f15cea00700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 0x7f15b006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T07:37:20.103 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:20.101+0000 7f355b382240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:37:20.730 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:20.728+0000 7f71c2adf240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:37:20.819 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:20.817+0000 7f355b382240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:37:20.936 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:20.933+0000 7f71c2adf240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:37:21.028 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:21.027+0000 7f355b382240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:37:21.141 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:21.139+0000 7f71c2adf240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:37:21.229 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:21.228+0000 7f355b382240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:37:21.611 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:21.609+0000 7f71c2adf240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:37:21.699 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:21.697+0000 7f355b382240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:37:21.814 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:21.812+0000 7f71c2adf240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:37:21.901 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:21.899+0000 7f355b382240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:37:22.249 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:22.246+0000 7f71c2adf240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:37:22.333 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:22.331+0000 7f355b382240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:37:22.356 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:37:22.354+0000 7f71c2adf240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:37:22.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:22.415+0000 7f15c7400700 1 -- 172.21.15.89:0/4052986524 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 7) v1 ==== 89154+0+0 (secure 0 0 0) 0x7f15bc01ad10 con 0x7f15d00cbb30 2024-04-02T07:37:22.445 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:22.443+0000 7f355b382240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:37:22.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:22.454+0000 7f15c7400700 1 -- 172.21.15.89:0/4052986524 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 8) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f15bc055460 con 0x7f15d00cbb30 2024-04-02T07:37:22.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:22.454+0000 7f15c7400700 1 -- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 msgr2=0x7f15b006d1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:37:22.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:22.454+0000 7f15c7400700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 0x7f15b006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:22.635 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:22.633+0000 7f34c1c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:37:22.635 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:22.633+0000 7f34c1c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:37:22.636 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:22.633+0000 7f34c1c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:37:22.636 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:22.633+0000 7f34c1c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:37:22.636 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:22.633+0000 7f34c1c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:37:22.636 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:22.634+0000 7f34c7600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:37:22.636 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:22.634+0000 7f34c7600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:37:22.636 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:22.634+0000 7f34c7600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:37:22.637 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:22.634+0000 7f34c7600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:37:22.637 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:37:22.634+0000 7f34c7600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:37:23.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.460+0000 7f15c7400700 1 -- 172.21.15.89:0/4052986524 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mgrmap(e 9) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f15bc055d30 con 0x7f15d00cbb30 2024-04-02T07:37:23.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.460+0000 7f15c7400700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 0x7f15b006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:23.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.460+0000 7f15c7400700 1 -- 172.21.15.89:0/4052986524 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f15b001d140 con 0x7f15b006ace0 2024-04-02T07:37:23.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.464+0000 7f15cea00700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 0x7f15b006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:23.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.468+0000 7f15cea00700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 0x7f15b006d1a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f15c0007fd0 tx=0x7f15c0007e00 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:23.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.562+0000 7f15c7400700 1 -- 172.21.15.89:0/4052986524 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f15b001d140 con 0x7f15b006ace0 2024-04-02T07:37:23.564 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path /snap_test_dir1/dir0 2024-04-02T07:37:23.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.565+0000 7f15d5e00700 1 -- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 msgr2=0x7f15b006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:23.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.565+0000 7f15d5e00700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 0x7f15b006d1a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f15c0007fd0 tx=0x7f15c0007e00 comp rx=0 tx=0).stop 2024-04-02T07:37:23.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.565+0000 7f15d5e00700 1 -- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15d00cbb30 msgr2=0x7f15d0154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:23.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.565+0000 7f15d5e00700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15d00cbb30 0x7f15d0154c60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f15bc014040 tx=0x7f15bc004750 comp rx=0 tx=0).stop 2024-04-02T07:37:23.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.565+0000 7f15d5e00700 1 -- 172.21.15.89:0/4052986524 shutdown_connections 2024-04-02T07:37:23.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.565+0000 7f15d5e00700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15b006ace0 0x7f15b006d1a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:23.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.565+0000 7f15d5e00700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d00b2070 0x7f15d0155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:23.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.565+0000 7f15d5e00700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00cd020 0x7f15d0099c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:23.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.565+0000 7f15d5e00700 1 --2- 172.21.15.89:0/4052986524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15d00cbb30 0x7f15d0154c60 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:23.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.565+0000 7f15d5e00700 1 -- 172.21.15.89:0/4052986524 >> 172.21.15.89:0/4052986524 conn(0x7f15d00b0d70 msgr2=0x7f15d00b2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:23.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.565+0000 7f15d5e00700 1 -- 172.21.15.89:0/4052986524 shutdown_connections 2024-04-02T07:37:23.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.565+0000 7f15d5e00700 1 -- 172.21.15.89:0/4052986524 wait complete. 2024-04-02T07:37:23.586 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2024-04-02T07:37:23.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:23.586 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2024-04-02T07:37:23.634 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2024-04-02T07:37:23.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.827+0000 7f086d400700 1 Processor -- start 2024-04-02T07:37:23.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.827+0000 7f086d400700 1 -- start start 2024-04-02T07:37:23.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.828+0000 7f086d400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f08680cd020 0x7f08680cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:23.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.828+0000 7f086d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08680b2070 0x7f08680b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:23.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.828+0000 7f086d400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08680cdae0 con 0x7f08680cbb30 2024-04-02T07:37:23.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.828+0000 7f086d400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08680cddc0 con 0x7f08680cd020 2024-04-02T07:37:23.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.828+0000 7f086d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08680ce0a0 con 0x7f08680b2070 2024-04-02T07:37:23.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.828+0000 7f0866800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f08680cbb30 0x7f08680cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56578/0 (socket says 172.21.15.89:56578) 2024-04-02T07:37:23.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.828+0000 7f0866800700 1 -- 172.21.15.89:0/3220715777 learned_addr learned my addr 172.21.15.89:0/3220715777 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:23.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.828+0000 7f0867200700 1 --2- 172.21.15.89:0/3220715777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08680b2070 0x7f08680b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:23.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.828+0000 7f0865e00700 1 --2- 172.21.15.89:0/3220715777 >> v2:172.21.15.113:3300/0 conn(0x7f08680cd020 0x7f08680cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:23.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.829+0000 7f0865400700 1 -- 172.21.15.89:0/3220715777 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422931103 0 0) 0x7f08680cdae0 con 0x7f08680cbb30 2024-04-02T07:37:23.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.829+0000 7f0865400700 1 -- 172.21.15.89:0/3220715777 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f084c0035b0 con 0x7f08680cbb30 2024-04-02T07:37:23.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.829+0000 7f0865e00700 1 -- 172.21.15.89:0/3220715777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08680b2070 msgr2=0x7f08680b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:23.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.829+0000 7f0865e00700 1 --2- 172.21.15.89:0/3220715777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08680b2070 0x7f08680b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:23.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.829+0000 7f0865e00700 1 -- 172.21.15.89:0/3220715777 >> v1:172.21.15.89:6789/0 conn(0x7f08680cbb30 legacy=0x7f08680cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:23.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.829+0000 7f0865e00700 1 -- 172.21.15.89:0/3220715777 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086806dc20 con 0x7f08680cd020 2024-04-02T07:37:23.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.829+0000 7f0865e00700 1 --2- 172.21.15.89:0/3220715777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08680cd020 0x7f08680cd480 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0854009840 tx=0x7f0854009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=941c1735b001bb14 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.829+0000 7f0865400700 1 -- 172.21.15.89:0/3220715777 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0854011830 con 0x7f08680cd020 2024-04-02T07:37:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.829+0000 7f0865400700 1 -- 172.21.15.89:0/3220715777 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0854011990 con 0x7f08680cd020 2024-04-02T07:37:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.829+0000 7f0865400700 1 -- 172.21.15.89:0/3220715777 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0854011d20 con 0x7f08680cd020 2024-04-02T07:37:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.830+0000 7f086d400700 1 -- 172.21.15.89:0/3220715777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08680cd020 msgr2=0x7f08680cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.830+0000 7f086d400700 1 --2- 172.21.15.89:0/3220715777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08680cd020 0x7f08680cd480 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0854009840 tx=0x7f0854009b50 comp rx=0 tx=0).stop 2024-04-02T07:37:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.830+0000 7f086d400700 1 -- 172.21.15.89:0/3220715777 shutdown_connections 2024-04-02T07:37:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.830+0000 7f086d400700 1 --2- 172.21.15.89:0/3220715777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08680b2070 0x7f08680b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.830+0000 7f086d400700 1 --2- 172.21.15.89:0/3220715777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08680cd020 0x7f08680cd480 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.830+0000 7f086d400700 1 -- 172.21.15.89:0/3220715777 >> 172.21.15.89:0/3220715777 conn(0x7f08680b0d70 msgr2=0x7f08680b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.830+0000 7f086d400700 1 -- 172.21.15.89:0/3220715777 shutdown_connections 2024-04-02T07:37:23.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.830+0000 7f086d400700 1 -- 172.21.15.89:0/3220715777 wait complete. 2024-04-02T07:37:23.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.832+0000 7f086d400700 1 Processor -- start 2024-04-02T07:37:23.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.833+0000 7f086d400700 1 -- start start 2024-04-02T07:37:23.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.834+0000 7f086d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08680b2070 0x7f0868098a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:23.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.834+0000 7f086d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08680cbb30 0x7f08680ac600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:23.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.834+0000 7f086d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08680cd020 0x7f08681e48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:23.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.834+0000 7f086d400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0868098f60 con 0x7f08680cd020 2024-04-02T07:37:23.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.834+0000 7f086d400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0868155ff0 con 0x7f08680b2070 2024-04-02T07:37:23.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.834+0000 7f086d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0868156890 con 0x7f08680cbb30 2024-04-02T07:37:23.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.834+0000 7f0867200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08680cd020 0x7f08681e48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:23.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.834+0000 7f0867200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08680cd020 0x7f08681e48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39170/0 (socket says 172.21.15.89:39170) 2024-04-02T07:37:23.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.834+0000 7f0867200700 1 -- 172.21.15.89:0/2864462328 learned_addr learned my addr 172.21.15.89:0/2864462328 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:23.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.834+0000 7f0866800700 1 --2- 172.21.15.89:0/2864462328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08680b2070 0x7f0868098a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:23.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.834+0000 7f0865e00700 1 --2- 172.21.15.89:0/2864462328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08680cbb30 0x7f08680ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:23.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.834+0000 7f0867200700 1 -- 172.21.15.89:0/2864462328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08680cbb30 msgr2=0x7f08680ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:23.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.834+0000 7f0867200700 1 --2- 172.21.15.89:0/2864462328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08680cbb30 0x7f08680ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:23.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.835+0000 7f0867200700 1 -- 172.21.15.89:0/2864462328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08680b2070 msgr2=0x7f0868098a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:23.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.835+0000 7f0867200700 1 --2- 172.21.15.89:0/2864462328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08680b2070 0x7f0868098a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:23.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.835+0000 7f0867200700 1 -- 172.21.15.89:0/2864462328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08540094a0 con 0x7f08680cd020 2024-04-02T07:37:23.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.835+0000 7f0867200700 1 --2- 172.21.15.89:0/2864462328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08680cd020 0x7f08681e48b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f086000efc0 tx=0x7f086000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:23.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.835+0000 7f085ea00700 1 -- 172.21.15.89:0/2864462328 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0860009ea0 con 0x7f08680cd020 2024-04-02T07:37:23.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.835+0000 7f085ea00700 1 -- 172.21.15.89:0/2864462328 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086000f040 con 0x7f08680cd020 2024-04-02T07:37:23.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.835+0000 7f086d400700 1 -- 172.21.15.89:0/2864462328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08680acb40 con 0x7f08680cd020 2024-04-02T07:37:23.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.835+0000 7f085ea00700 1 -- 172.21.15.89:0/2864462328 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086000ced0 con 0x7f08680cd020 2024-04-02T07:37:23.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.835+0000 7f086d400700 1 -- 172.21.15.89:0/2864462328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0868156240 con 0x7f08680cd020 2024-04-02T07:37:23.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.836+0000 7f086d400700 1 -- 172.21.15.89:0/2864462328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0868156a00 con 0x7f08680cd020 2024-04-02T07:37:23.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.837+0000 7f085ea00700 1 -- 172.21.15.89:0/2864462328 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 9) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f0860014480 con 0x7f08680cd020 2024-04-02T07:37:23.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.838+0000 7f085ea00700 1 --2- 172.21.15.89:0/2864462328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f084806ab30 0x7f084806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:23.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.838+0000 7f085ea00700 1 -- 172.21.15.89:0/2864462328 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f0860089880 con 0x7f08680cd020 2024-04-02T07:37:23.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.838+0000 7f0866800700 1 --2- 172.21.15.89:0/2864462328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f084806ab30 0x7f084806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:23.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.839+0000 7f0866800700 1 --2- 172.21.15.89:0/2864462328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f084806ab30 0x7f084806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0858007fd0 tx=0x7f0858007dc0 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:23.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:23.843+0000 7f085ea00700 1 -- 172.21.15.89:0/2864462328 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0860054ca0 con 0x7f08680cd020 2024-04-02T07:37:24.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.070+0000 7f086d400700 1 -- 172.21.15.89:0/2864462328 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f086815bdd0 con 0x7f084806ab30 2024-04-02T07:37:24.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.163+0000 7f085ea00700 1 -- 172.21.15.89:0/2864462328 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f0858006bd0 con 0x7f084806ab30 2024-04-02T07:37:24.165 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path /snap_test_dir1/dir1 2024-04-02T07:37:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.165+0000 7f086d400700 1 -- 172.21.15.89:0/2864462328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f084806ab30 msgr2=0x7f084806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.165+0000 7f086d400700 1 --2- 172.21.15.89:0/2864462328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f084806ab30 0x7f084806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0858007fd0 tx=0x7f0858007dc0 comp rx=0 tx=0).stop 2024-04-02T07:37:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.165+0000 7f086d400700 1 -- 172.21.15.89:0/2864462328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08680cd020 msgr2=0x7f08681e48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.165+0000 7f086d400700 1 --2- 172.21.15.89:0/2864462328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08680cd020 0x7f08681e48b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f086000efc0 tx=0x7f086000c3a0 comp rx=0 tx=0).stop 2024-04-02T07:37:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.166+0000 7f086d400700 1 -- 172.21.15.89:0/2864462328 shutdown_connections 2024-04-02T07:37:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.166+0000 7f086d400700 1 --2- 172.21.15.89:0/2864462328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f084806ab30 0x7f084806cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.166+0000 7f086d400700 1 --2- 172.21.15.89:0/2864462328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08680b2070 0x7f0868098a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.166+0000 7f086d400700 1 --2- 172.21.15.89:0/2864462328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08680cd020 0x7f08681e48b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.166+0000 7f086d400700 1 --2- 172.21.15.89:0/2864462328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08680cbb30 0x7f08680ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.166+0000 7f086d400700 1 -- 172.21.15.89:0/2864462328 >> 172.21.15.89:0/2864462328 conn(0x7f08680b0d70 msgr2=0x7f08680b3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.166+0000 7f086d400700 1 -- 172.21.15.89:0/2864462328 shutdown_connections 2024-04-02T07:37:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.166+0000 7f086d400700 1 -- 172.21.15.89:0/2864462328 wait complete. 2024-04-02T07:37:24.185 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2024-04-02T07:37:24.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:24.185 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2024-04-02T07:37:24.232 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2024-04-02T07:37:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.424+0000 7fe332a00700 1 Processor -- start 2024-04-02T07:37:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.424+0000 7fe332a00700 1 -- start start 2024-04-02T07:37:24.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.425+0000 7fe332a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe32c0d16a0 0x7fe32c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:24.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.425+0000 7fe332a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32c161420 con 0x7fe32c15d9d0 2024-04-02T07:37:24.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.425+0000 7fe332a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32c161700 con 0x7fe32c0d2370 2024-04-02T07:37:24.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.425+0000 7fe332a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe32c0d4340 con 0x7fe32c0d16a0 2024-04-02T07:37:24.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.425+0000 7fe32b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe32c0d16a0 0x7fe32c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:24.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.425+0000 7fe32be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe32c15d9d0 0x7fe32c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56588/0 (socket says 172.21.15.89:56588) 2024-04-02T07:37:24.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.425+0000 7fe32be00700 1 -- 172.21.15.89:0/439417861 learned_addr learned my addr 172.21.15.89:0/439417861 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:24.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.426+0000 7fe32b400700 1 -- 172.21.15.89:0/439417861 >> v1:172.21.15.113:6789/0 conn(0x7fe32c0d2370 legacy=0x7fe32c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:24.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.426+0000 7fe32b400700 1 -- 172.21.15.89:0/439417861 >> v1:172.21.15.89:6789/0 conn(0x7fe32c15d9d0 legacy=0x7fe32c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:24.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.426+0000 7fe32b400700 1 -- 172.21.15.89:0/439417861 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c1619e0 con 0x7fe32c0d16a0 2024-04-02T07:37:24.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.426+0000 7fe32aa00700 1 -- 172.21.15.89:0/439417861 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256186506 0 0) 0x7fe32c161420 con 0x7fe32c15d9d0 2024-04-02T07:37:24.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.426+0000 7fe32b400700 1 --2- 172.21.15.89:0/439417861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe32c0d16a0 0x7fe32c0d1e30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe318009820 tx=0x7fe318009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b2b50f8e5301b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:24.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.426+0000 7fe32aa00700 1 -- 172.21.15.89:0/439417861 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318005e30 con 0x7fe32c0d16a0 2024-04-02T07:37:24.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.426+0000 7fe32aa00700 1 -- 172.21.15.89:0/439417861 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe318004030 con 0x7fe32c0d16a0 2024-04-02T07:37:24.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.426+0000 7fe32aa00700 1 -- 172.21.15.89:0/439417861 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3180043c0 con 0x7fe32c0d16a0 2024-04-02T07:37:24.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.427+0000 7fe332a00700 1 -- 172.21.15.89:0/439417861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe32c0d16a0 msgr2=0x7fe32c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:24.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.427+0000 7fe332a00700 1 --2- 172.21.15.89:0/439417861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe32c0d16a0 0x7fe32c0d1e30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe318009820 tx=0x7fe318009b30 comp rx=0 tx=0).stop 2024-04-02T07:37:24.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.427+0000 7fe332a00700 1 -- 172.21.15.89:0/439417861 shutdown_connections 2024-04-02T07:37:24.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.427+0000 7fe332a00700 1 --2- 172.21.15.89:0/439417861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe32c0d16a0 0x7fe32c0d1e30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:24.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.427+0000 7fe332a00700 1 -- 172.21.15.89:0/439417861 >> 172.21.15.89:0/439417861 conn(0x7fe32c08aff0 msgr2=0x7fe32c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:24.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.427+0000 7fe332a00700 1 -- 172.21.15.89:0/439417861 shutdown_connections 2024-04-02T07:37:24.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.427+0000 7fe332a00700 1 -- 172.21.15.89:0/439417861 wait complete. 2024-04-02T07:37:24.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.429+0000 7fe332a00700 1 Processor -- start 2024-04-02T07:37:24.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.429+0000 7fe332a00700 1 -- start start 2024-04-02T07:37:24.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.430+0000 7fe332a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe32c0d16a0 0x7fe32c1568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:24.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.430+0000 7fe332a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe32c0d2370 0x7fe32c15e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:24.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.430+0000 7fe332a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe32c15d9d0 0x7fe32c156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:24.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.430+0000 7fe332a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c15f280 con 0x7fe32c0d2370 2024-04-02T07:37:24.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.430+0000 7fe332a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c0b7cc0 con 0x7fe32c0d16a0 2024-04-02T07:37:24.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.430+0000 7fe332a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c0b73d0 con 0x7fe32c15d9d0 2024-04-02T07:37:24.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.430+0000 7fe32b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe32c0d2370 0x7fe32c15e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:24.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.430+0000 7fe32b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe32c0d2370 0x7fe32c15e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39194/0 (socket says 172.21.15.89:39194) 2024-04-02T07:37:24.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.430+0000 7fe32b400700 1 -- 172.21.15.89:0/1174377646 learned_addr learned my addr 172.21.15.89:0/1174377646 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:24.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.430+0000 7fe32be00700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe32c0d16a0 0x7fe32c1568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:24.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.430+0000 7fe330a00700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe32c15d9d0 0x7fe32c156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:24.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.431+0000 7fe32b400700 1 -- 172.21.15.89:0/1174377646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe32c15d9d0 msgr2=0x7fe32c156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:24.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.431+0000 7fe32b400700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe32c15d9d0 0x7fe32c156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:24.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.431+0000 7fe32b400700 1 -- 172.21.15.89:0/1174377646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe32c0d16a0 msgr2=0x7fe32c1568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:24.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.431+0000 7fe32b400700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe32c0d16a0 0x7fe32c1568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:24.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.431+0000 7fe32b400700 1 -- 172.21.15.89:0/1174377646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318009430 con 0x7fe32c0d2370 2024-04-02T07:37:24.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.431+0000 7fe330a00700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe32c15d9d0 0x7fe32c156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:24.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.431+0000 7fe32b400700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe32c0d2370 0x7fe32c15e7c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe318000c00 tx=0x7fe318004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:24.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.431+0000 7fe323e00700 1 -- 172.21.15.89:0/1174377646 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318011420 con 0x7fe32c0d2370 2024-04-02T07:37:24.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.431+0000 7fe323e00700 1 -- 172.21.15.89:0/1174377646 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe318011580 con 0x7fe32c0d2370 2024-04-02T07:37:24.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.431+0000 7fe332a00700 1 -- 172.21.15.89:0/1174377646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32c15ed00 con 0x7fe32c0d2370 2024-04-02T07:37:24.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.431+0000 7fe323e00700 1 -- 172.21.15.89:0/1174377646 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318011930 con 0x7fe32c0d2370 2024-04-02T07:37:24.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.431+0000 7fe332a00700 1 -- 172.21.15.89:0/1174377646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32c0b7670 con 0x7fe32c0d2370 2024-04-02T07:37:24.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.432+0000 7fe332a00700 1 -- 172.21.15.89:0/1174377646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32c0d1ca0 con 0x7fe32c0d2370 2024-04-02T07:37:24.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.434+0000 7fe323e00700 1 -- 172.21.15.89:0/1174377646 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 9) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fe318018030 con 0x7fe32c0d2370 2024-04-02T07:37:24.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.434+0000 7fe323e00700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe30c06ab30 0x7fe30c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:24.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.434+0000 7fe323e00700 1 -- 172.21.15.89:0/1174377646 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fe31808a740 con 0x7fe32c0d2370 2024-04-02T07:37:24.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.435+0000 7fe32be00700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe30c06ab30 0x7fe30c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:24.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.435+0000 7fe32be00700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe30c06ab30 0x7fe30c06cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe31c007c40 tx=0x7fe31c007b90 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:24.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.438+0000 7fe323e00700 1 -- 172.21.15.89:0/1174377646 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe318055c20 con 0x7fe32c0d2370 2024-04-02T07:37:24.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.468+0000 7fe323e00700 1 -- 172.21.15.89:0/1174377646 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 10) v1 ==== 89257+0+0 (secure 0 0 0) 0x7fe318055ff0 con 0x7fe32c0d2370 2024-04-02T07:37:24.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.679+0000 7fe332a00700 1 -- 172.21.15.89:0/1174377646 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe32c161fa0 con 0x7fe30c06ab30 2024-04-02T07:37:24.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.759+0000 7fe323e00700 1 -- 172.21.15.89:0/1174377646 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fe31c004bb0 con 0x7fe30c06ab30 2024-04-02T07:37:24.761 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path /snap_test_dir1/dir2 2024-04-02T07:37:24.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.762+0000 7fe332a00700 1 -- 172.21.15.89:0/1174377646 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe30c06ab30 msgr2=0x7fe30c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:24.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.762+0000 7fe332a00700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe30c06ab30 0x7fe30c06cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe31c007c40 tx=0x7fe31c007b90 comp rx=0 tx=0).stop 2024-04-02T07:37:24.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.762+0000 7fe332a00700 1 -- 172.21.15.89:0/1174377646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe32c0d2370 msgr2=0x7fe32c15e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:24.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.762+0000 7fe332a00700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe32c0d2370 0x7fe32c15e7c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe318000c00 tx=0x7fe318004750 comp rx=0 tx=0).stop 2024-04-02T07:37:24.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.762+0000 7fe332a00700 1 -- 172.21.15.89:0/1174377646 shutdown_connections 2024-04-02T07:37:24.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.762+0000 7fe332a00700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe30c06ab30 0x7fe30c06cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:24.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.762+0000 7fe332a00700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe32c0d16a0 0x7fe32c1568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:24.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.762+0000 7fe332a00700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe32c15d9d0 0x7fe32c156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:24.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.762+0000 7fe332a00700 1 --2- 172.21.15.89:0/1174377646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe32c0d2370 0x7fe32c15e7c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:24.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.762+0000 7fe332a00700 1 -- 172.21.15.89:0/1174377646 >> 172.21.15.89:0/1174377646 conn(0x7fe32c08aff0 msgr2=0x7fe32c0b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:24.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.762+0000 7fe332a00700 1 -- 172.21.15.89:0/1174377646 shutdown_connections 2024-04-02T07:37:24.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:24.762+0000 7fe332a00700 1 -- 172.21.15.89:0/1174377646 wait complete. 2024-04-02T07:37:24.783 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2024-04-02T07:37:24.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:24.783 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2024-04-02T07:37:24.829 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2024-04-02T07:37:25.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.023+0000 7f1465600700 1 Processor -- start 2024-04-02T07:37:25.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.023+0000 7f1465600700 1 -- start start 2024-04-02T07:37:25.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.024+0000 7f1465600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f14600d16a0 0x7f14600d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:25.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.024+0000 7f1465600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1460161420 con 0x7f14600d2370 2024-04-02T07:37:25.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.024+0000 7f1465600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1460161700 con 0x7f146015d9d0 2024-04-02T07:37:25.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.024+0000 7f1465600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14600d4340 con 0x7f14600d16a0 2024-04-02T07:37:25.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.024+0000 7f145f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f14600d2370 0x7f14600ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56590/0 (socket says 172.21.15.89:56590) 2024-04-02T07:37:25.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.024+0000 7f145f400700 1 -- 172.21.15.89:0/3723849745 learned_addr learned my addr 172.21.15.89:0/3723849745 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:25.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.025+0000 7f1457e00700 1 --2- 172.21.15.89:0/3723849745 >> v2:172.21.15.113:3300/0 conn(0x7f14600d16a0 0x7f14600d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:25.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.025+0000 7f145e000700 1 -- 172.21.15.89:0/3723849745 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384611973 0 0) 0x7f1460161420 con 0x7f14600d2370 2024-04-02T07:37:25.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.025+0000 7f145e000700 1 -- 172.21.15.89:0/3723849745 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14440035b0 con 0x7f14600d2370 2024-04-02T07:37:25.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.025+0000 7f1457e00700 1 -- 172.21.15.89:0/3723849745 >> v1:172.21.15.113:6789/0 conn(0x7f146015d9d0 legacy=0x7f14600d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:25.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.025+0000 7f1457e00700 1 -- 172.21.15.89:0/3723849745 >> v1:172.21.15.89:6789/0 conn(0x7f14600d2370 legacy=0x7f14600ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:25.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.025+0000 7f1457e00700 1 -- 172.21.15.89:0/3723849745 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14601619e0 con 0x7f14600d16a0 2024-04-02T07:37:25.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.025+0000 7f1457e00700 1 --2- 172.21.15.89:0/3723849745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14600d16a0 0x7f14600d1e30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1448009820 tx=0x7f1448009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cae7a500b0685443 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:25.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.026+0000 7f145e000700 1 -- 172.21.15.89:0/3723849745 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448005e30 con 0x7f14600d16a0 2024-04-02T07:37:25.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.026+0000 7f145e000700 1 -- 172.21.15.89:0/3723849745 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1448006030 con 0x7f14600d16a0 2024-04-02T07:37:25.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.026+0000 7f145e000700 1 -- 172.21.15.89:0/3723849745 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14480063c0 con 0x7f14600d16a0 2024-04-02T07:37:25.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.026+0000 7f1465600700 1 -- 172.21.15.89:0/3723849745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14600d16a0 msgr2=0x7f14600d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:25.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.026+0000 7f1465600700 1 --2- 172.21.15.89:0/3723849745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14600d16a0 0x7f14600d1e30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1448009820 tx=0x7f1448009b30 comp rx=0 tx=0).stop 2024-04-02T07:37:25.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.026+0000 7f1465600700 1 -- 172.21.15.89:0/3723849745 shutdown_connections 2024-04-02T07:37:25.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.026+0000 7f1465600700 1 --2- 172.21.15.89:0/3723849745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14600d16a0 0x7f14600d1e30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.026+0000 7f1465600700 1 -- 172.21.15.89:0/3723849745 >> 172.21.15.89:0/3723849745 conn(0x7f146008aff0 msgr2=0x7f14600d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:25.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.026+0000 7f1465600700 1 -- 172.21.15.89:0/3723849745 shutdown_connections 2024-04-02T07:37:25.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.026+0000 7f1465600700 1 -- 172.21.15.89:0/3723849745 wait complete. 2024-04-02T07:37:25.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.028+0000 7f1465600700 1 Processor -- start 2024-04-02T07:37:25.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.028+0000 7f1465600700 1 -- start start 2024-04-02T07:37:25.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.029+0000 7f1465600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14600d16a0 0x7f14600b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:25.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.029+0000 7f1465600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14600d2370 0x7f14600b8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.029+0000 7f1465600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146015d9d0 0x7f14601607f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.029+0000 7f1465600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14600b76d0 con 0x7f146015d9d0 2024-04-02T07:37:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.029+0000 7f1465600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14600b7110 con 0x7f14600d16a0 2024-04-02T07:37:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.029+0000 7f1465600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14600b73f0 con 0x7f14600d2370 2024-04-02T07:37:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.029+0000 7f145f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146015d9d0 0x7f14601607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.029+0000 7f145f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146015d9d0 0x7f14601607f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39218/0 (socket says 172.21.15.89:39218) 2024-04-02T07:37:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.029+0000 7f145f400700 1 -- 172.21.15.89:0/616182149 learned_addr learned my addr 172.21.15.89:0/616182149 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.029+0000 7f1457e00700 1 --2- 172.21.15.89:0/616182149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14600d2370 0x7f14600b8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.029+0000 7f145ea00700 1 --2- 172.21.15.89:0/616182149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14600d16a0 0x7f14600b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.030+0000 7f145f400700 1 -- 172.21.15.89:0/616182149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14600d2370 msgr2=0x7f14600b8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.030+0000 7f145f400700 1 --2- 172.21.15.89:0/616182149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14600d2370 0x7f14600b8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.030+0000 7f145f400700 1 -- 172.21.15.89:0/616182149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14600d16a0 msgr2=0x7f14600b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.030+0000 7f145f400700 1 --2- 172.21.15.89:0/616182149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14600d16a0 0x7f14600b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.030+0000 7f145f400700 1 -- 172.21.15.89:0/616182149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1448009430 con 0x7f146015d9d0 2024-04-02T07:37:25.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.030+0000 7f145f400700 1 --2- 172.21.15.89:0/616182149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146015d9d0 0x7f14601607f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f145800cb40 tx=0x7f145800ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:25.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.030+0000 7f1456a00700 1 -- 172.21.15.89:0/616182149 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145800a9e0 con 0x7f146015d9d0 2024-04-02T07:37:25.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.030+0000 7f1456a00700 1 -- 172.21.15.89:0/616182149 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145800ab40 con 0x7f146015d9d0 2024-04-02T07:37:25.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.030+0000 7f1465600700 1 -- 172.21.15.89:0/616182149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146015df50 con 0x7f146015d9d0 2024-04-02T07:37:25.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.030+0000 7f1456a00700 1 -- 172.21.15.89:0/616182149 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458012400 con 0x7f146015d9d0 2024-04-02T07:37:25.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.030+0000 7f1465600700 1 -- 172.21.15.89:0/616182149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14600b8980 con 0x7f146015d9d0 2024-04-02T07:37:25.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.031+0000 7f1465600700 1 -- 172.21.15.89:0/616182149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14600d1ca0 con 0x7f146015d9d0 2024-04-02T07:37:25.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.032+0000 7f1456a00700 1 -- 172.21.15.89:0/616182149 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 10) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f145801b3b0 con 0x7f146015d9d0 2024-04-02T07:37:25.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.033+0000 7f1456a00700 1 --2- 172.21.15.89:0/616182149 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f143c06ae00 0x7f143c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:25.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.033+0000 7f1456a00700 1 -- 172.21.15.89:0/616182149 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f14580890b0 con 0x7f146015d9d0 2024-04-02T07:37:25.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.033+0000 7f145ea00700 1 --2- 172.21.15.89:0/616182149 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f143c06ae00 0x7f143c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:25.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.034+0000 7f145ea00700 1 --2- 172.21.15.89:0/616182149 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f143c06ae00 0x7f143c06d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1450007fd0 tx=0x7f1450009040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:25.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.037+0000 7f1456a00700 1 -- 172.21.15.89:0/616182149 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f14580541c0 con 0x7f146015d9d0 2024-04-02T07:37:25.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.264+0000 7f1465600700 1 -- 172.21.15.89:0/616182149 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1460161fa0 con 0x7f143c06ae00 2024-04-02T07:37:25.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.342+0000 7f1456a00700 1 -- 172.21.15.89:0/616182149 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f14500014c0 con 0x7f143c06ae00 2024-04-02T07:37:25.345 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path /snap_test_dir1/dir3 2024-04-02T07:37:25.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.344+0000 7f1465600700 1 -- 172.21.15.89:0/616182149 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f143c06ae00 msgr2=0x7f143c06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:25.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.344+0000 7f1465600700 1 --2- 172.21.15.89:0/616182149 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f143c06ae00 0x7f143c06d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1450007fd0 tx=0x7f1450009040 comp rx=0 tx=0).stop 2024-04-02T07:37:25.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.345+0000 7f1465600700 1 -- 172.21.15.89:0/616182149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146015d9d0 msgr2=0x7f14601607f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:25.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.345+0000 7f1465600700 1 --2- 172.21.15.89:0/616182149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146015d9d0 0x7f14601607f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f145800cb40 tx=0x7f145800ce50 comp rx=0 tx=0).stop 2024-04-02T07:37:25.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.345+0000 7f1465600700 1 -- 172.21.15.89:0/616182149 shutdown_connections 2024-04-02T07:37:25.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.345+0000 7f1465600700 1 --2- 172.21.15.89:0/616182149 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f143c06ae00 0x7f143c06d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.345+0000 7f1465600700 1 --2- 172.21.15.89:0/616182149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14600d16a0 0x7f14600b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.345+0000 7f1465600700 1 --2- 172.21.15.89:0/616182149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146015d9d0 0x7f14601607f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.345+0000 7f1465600700 1 --2- 172.21.15.89:0/616182149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14600d2370 0x7f14600b8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.345+0000 7f1465600700 1 -- 172.21.15.89:0/616182149 >> 172.21.15.89:0/616182149 conn(0x7f146008aff0 msgr2=0x7f14600d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:25.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.345+0000 7f1465600700 1 -- 172.21.15.89:0/616182149 shutdown_connections 2024-04-02T07:37:25.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.345+0000 7f1465600700 1 -- 172.21.15.89:0/616182149 wait complete. 2024-04-02T07:37:25.364 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2024-04-02T07:37:25.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:25.364 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2024-04-02T07:37:25.411 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2024-04-02T07:37:25.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.605+0000 7f1f39200700 1 Processor -- start 2024-04-02T07:37:25.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.605+0000 7f1f39200700 1 -- start start 2024-04-02T07:37:25.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.606+0000 7f1f39200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f341611b0 0x7f1f3415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:25.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.606+0000 7f1f39200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1f3415f4d0 0x7f1f340ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:25.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.606+0000 7f1f39200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f340ce270 con 0x7f1f340d21f0 2024-04-02T07:37:25.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.606+0000 7f1f39200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f3415b600 con 0x7f1f3415f4d0 2024-04-02T07:37:25.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.606+0000 7f1f39200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f340d2760 con 0x7f1f341611b0 2024-04-02T07:37:25.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.606+0000 7f1f31400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1f3415f4d0 0x7f1f340ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:25.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.606+0000 7f1f32800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1f340d21f0 0x7f1f34161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56596/0 (socket says 172.21.15.89:56596) 2024-04-02T07:37:25.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.607+0000 7f1f32800700 1 -- 172.21.15.89:0/3746693196 learned_addr learned my addr 172.21.15.89:0/3746693196 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:25.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.607+0000 7f1f31e00700 1 --2- 172.21.15.89:0/3746693196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f341611b0 0x7f1f3415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:25.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.607+0000 7f1f30a00700 1 -- 172.21.15.89:0/3746693196 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192203520 0 0) 0x7f1f340ce270 con 0x7f1f340d21f0 2024-04-02T07:37:25.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.607+0000 7f1f30a00700 1 -- 172.21.15.89:0/3746693196 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f180035b0 con 0x7f1f340d21f0 2024-04-02T07:37:25.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.607+0000 7f1f31400700 1 -- 172.21.15.89:0/3746693196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f341611b0 msgr2=0x7f1f3415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:25.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.607+0000 7f1f31400700 1 --2- 172.21.15.89:0/3746693196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f341611b0 0x7f1f3415ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.607+0000 7f1f31400700 1 -- 172.21.15.89:0/3746693196 >> v1:172.21.15.89:6789/0 conn(0x7f1f340d21f0 legacy=0x7f1f34161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:25.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.607+0000 7f1f31400700 1 -- 172.21.15.89:0/3746693196 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f340d2a40 con 0x7f1f3415f4d0 2024-04-02T07:37:25.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.607+0000 7f1f31400700 1 --2- 172.21.15.89:0/3746693196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3415f4d0 0x7f1f340ce700 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009840 tx=0x7f1f20009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f804546be87bd7d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:25.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.608+0000 7f1f30a00700 1 -- 172.21.15.89:0/3746693196 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20011830 con 0x7f1f3415f4d0 2024-04-02T07:37:25.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.608+0000 7f1f30a00700 1 -- 172.21.15.89:0/3746693196 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f20011990 con 0x7f1f3415f4d0 2024-04-02T07:37:25.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.608+0000 7f1f30a00700 1 -- 172.21.15.89:0/3746693196 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20011d20 con 0x7f1f3415f4d0 2024-04-02T07:37:25.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.608+0000 7f1f39200700 1 -- 172.21.15.89:0/3746693196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3415f4d0 msgr2=0x7f1f340ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:25.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.608+0000 7f1f39200700 1 --2- 172.21.15.89:0/3746693196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3415f4d0 0x7f1f340ce700 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009840 tx=0x7f1f20009b50 comp rx=0 tx=0).stop 2024-04-02T07:37:25.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.608+0000 7f1f39200700 1 -- 172.21.15.89:0/3746693196 shutdown_connections 2024-04-02T07:37:25.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.608+0000 7f1f39200700 1 --2- 172.21.15.89:0/3746693196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f341611b0 0x7f1f3415ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.608+0000 7f1f39200700 1 --2- 172.21.15.89:0/3746693196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3415f4d0 0x7f1f340ce700 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.608+0000 7f1f39200700 1 -- 172.21.15.89:0/3746693196 >> 172.21.15.89:0/3746693196 conn(0x7f1f3408aff0 msgr2=0x7f1f340d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:25.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.608+0000 7f1f39200700 1 -- 172.21.15.89:0/3746693196 shutdown_connections 2024-04-02T07:37:25.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.608+0000 7f1f39200700 1 -- 172.21.15.89:0/3746693196 wait complete. 2024-04-02T07:37:25.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.611+0000 7f1f39200700 1 Processor -- start 2024-04-02T07:37:25.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.611+0000 7f1f39200700 1 -- start start 2024-04-02T07:37:25.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.612+0000 7f1f39200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f340d21f0 0x7f1f340b7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:25.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.612+0000 7f1f39200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3415f4d0 0x7f1f34158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:25.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.612+0000 7f1f39200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f341611b0 0x7f1f3409e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:25.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.612+0000 7f1f39200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f340b7730 con 0x7f1f3415f4d0 2024-04-02T07:37:25.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.612+0000 7f1f39200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f34159310 con 0x7f1f340d21f0 2024-04-02T07:37:25.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.612+0000 7f1f39200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f340ca7a0 con 0x7f1f341611b0 2024-04-02T07:37:25.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.612+0000 7f1f31400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3415f4d0 0x7f1f34158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:25.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.612+0000 7f1f31400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3415f4d0 0x7f1f34158dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39238/0 (socket says 172.21.15.89:39238) 2024-04-02T07:37:25.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.612+0000 7f1f31400700 1 -- 172.21.15.89:0/321981101 learned_addr learned my addr 172.21.15.89:0/321981101 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:25.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.612+0000 7f1f32800700 1 --2- 172.21.15.89:0/321981101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f341611b0 0x7f1f3409e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:25.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.612+0000 7f1f31e00700 1 --2- 172.21.15.89:0/321981101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f340d21f0 0x7f1f340b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:25.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.613+0000 7f1f31400700 1 -- 172.21.15.89:0/321981101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f341611b0 msgr2=0x7f1f3409e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:25.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.613+0000 7f1f31400700 1 --2- 172.21.15.89:0/321981101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f341611b0 0x7f1f3409e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.613+0000 7f1f31400700 1 -- 172.21.15.89:0/321981101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f340d21f0 msgr2=0x7f1f340b7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:25.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.613+0000 7f1f31400700 1 --2- 172.21.15.89:0/321981101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f340d21f0 0x7f1f340b7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.613+0000 7f1f31400700 1 -- 172.21.15.89:0/321981101 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f200094a0 con 0x7f1f3415f4d0 2024-04-02T07:37:25.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.613+0000 7f1f31400700 1 --2- 172.21.15.89:0/321981101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3415f4d0 0x7f1f34158dd0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1f200090b0 tx=0x7f1f20004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:25.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.613+0000 7f1f2a000700 1 -- 172.21.15.89:0/321981101 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20004590 con 0x7f1f3415f4d0 2024-04-02T07:37:25.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.613+0000 7f1f2a000700 1 -- 172.21.15.89:0/321981101 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2001f070 con 0x7f1f3415f4d0 2024-04-02T07:37:25.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.613+0000 7f1f2a000700 1 -- 172.21.15.89:0/321981101 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f200038e0 con 0x7f1f3415f4d0 2024-04-02T07:37:25.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.613+0000 7f1f39200700 1 -- 172.21.15.89:0/321981101 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f340caa80 con 0x7f1f3415f4d0 2024-04-02T07:37:25.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.613+0000 7f1f39200700 1 -- 172.21.15.89:0/321981101 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f340cb040 con 0x7f1f3415f4d0 2024-04-02T07:37:25.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.614+0000 7f1f39200700 1 -- 172.21.15.89:0/321981101 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f34067720 con 0x7f1f3415f4d0 2024-04-02T07:37:25.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.615+0000 7f1f2a000700 1 -- 172.21.15.89:0/321981101 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 10) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f1f20003a40 con 0x7f1f3415f4d0 2024-04-02T07:37:25.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.616+0000 7f1f2a000700 1 --2- 172.21.15.89:0/321981101 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f1406ae00 0x7f1f1406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:25.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.616+0000 7f1f2a000700 1 -- 172.21.15.89:0/321981101 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f1f20021070 con 0x7f1f3415f4d0 2024-04-02T07:37:25.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.616+0000 7f1f31e00700 1 --2- 172.21.15.89:0/321981101 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f1406ae00 0x7f1f1406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:25.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.617+0000 7f1f31e00700 1 --2- 172.21.15.89:0/321981101 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f1406ae00 0x7f1f1406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009b40 tx=0x7f1f24009210 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:25.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.620+0000 7f1f2a000700 1 -- 172.21.15.89:0/321981101 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1f20055d30 con 0x7f1f3415f4d0 2024-04-02T07:37:25.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.847+0000 7f1f39200700 1 -- 172.21.15.89:0/321981101 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f340cdf00 con 0x7f1f1406ae00 2024-04-02T07:37:25.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.931+0000 7f1f2a000700 1 -- 172.21.15.89:0/321981101 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f1f24004040 con 0x7f1f1406ae00 2024-04-02T07:37:25.933 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path /snap_test_dir1/dir4 2024-04-02T07:37:25.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.934+0000 7f1f39200700 1 -- 172.21.15.89:0/321981101 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f1406ae00 msgr2=0x7f1f1406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:25.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.934+0000 7f1f39200700 1 --2- 172.21.15.89:0/321981101 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f1406ae00 0x7f1f1406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009b40 tx=0x7f1f24009210 comp rx=0 tx=0).stop 2024-04-02T07:37:25.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.934+0000 7f1f39200700 1 -- 172.21.15.89:0/321981101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3415f4d0 msgr2=0x7f1f34158dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:25.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.934+0000 7f1f39200700 1 --2- 172.21.15.89:0/321981101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3415f4d0 0x7f1f34158dd0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1f200090b0 tx=0x7f1f20004750 comp rx=0 tx=0).stop 2024-04-02T07:37:25.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.934+0000 7f1f39200700 1 -- 172.21.15.89:0/321981101 shutdown_connections 2024-04-02T07:37:25.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.934+0000 7f1f39200700 1 --2- 172.21.15.89:0/321981101 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f1406ae00 0x7f1f1406d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.934+0000 7f1f39200700 1 --2- 172.21.15.89:0/321981101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f340d21f0 0x7f1f340b7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.934+0000 7f1f39200700 1 --2- 172.21.15.89:0/321981101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f341611b0 0x7f1f3409e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.934+0000 7f1f39200700 1 --2- 172.21.15.89:0/321981101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f3415f4d0 0x7f1f34158dd0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:25.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.934+0000 7f1f39200700 1 -- 172.21.15.89:0/321981101 >> 172.21.15.89:0/321981101 conn(0x7f1f3408aff0 msgr2=0x7f1f3415da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:25.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.934+0000 7f1f39200700 1 -- 172.21.15.89:0/321981101 shutdown_connections 2024-04-02T07:37:25.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:25.934+0000 7f1f39200700 1 -- 172.21.15.89:0/321981101 wait complete. 2024-04-02T07:37:25.954 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2024-04-02T07:37:25.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:25.954 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2024-04-02T07:37:26.000 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2024-04-02T07:37:26.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.232+0000 7f7261600700 1 Processor -- start 2024-04-02T07:37:26.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.232+0000 7f7261600700 1 -- start start 2024-04-02T07:37:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f7261600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f725c160d70 0x7f725c0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f7261600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f725c15ef70 0x7f725c15e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f7261600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f725c06a790 0x7f725c06ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f7261600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f725c06b180 con 0x7f725c15ef70 2024-04-02T07:37:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f7261600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f725c0b4cd0 con 0x7f725c06a790 2024-04-02T07:37:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f7261600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725c161150 con 0x7f725c160d70 2024-04-02T07:37:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f725a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f725c15ef70 0x7f725c15e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f725a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f725c15ef70 0x7f725c15e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39240/0 (socket says 172.21.15.89:39240) 2024-04-02T07:37:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f725a000700 1 -- 172.21.15.89:0/1343357017 learned_addr learned my addr 172.21.15.89:0/1343357017 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f725b400700 1 --2- 172.21.15.89:0/1343357017 >> v2:172.21.15.113:3300/0 conn(0x7f725c06a790 0x7f725c06ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f725a000700 1 -- 172.21.15.89:0/1343357017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f725c160d70 msgr2=0x7f725c0b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f725aa00700 1 --2- 172.21.15.89:0/1343357017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f725c160d70 0x7f725c0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f725a000700 1 --2- 172.21.15.89:0/1343357017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f725c160d70 0x7f725c0b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f725a000700 1 -- 172.21.15.89:0/1343357017 >> v2:172.21.15.113:3300/0 conn(0x7f725c06a790 msgr2=0x7f725c06ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f725a000700 1 --2- 172.21.15.89:0/1343357017 >> v2:172.21.15.113:3300/0 conn(0x7f725c06a790 0x7f725c06ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f725a000700 1 -- 172.21.15.89:0/1343357017 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725c06dba0 con 0x7f725c15ef70 2024-04-02T07:37:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f725b400700 1 --2- 172.21.15.89:0/1343357017 >> v2:172.21.15.113:3300/0 conn(0x7f725c06a790 0x7f725c06ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:37:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.234+0000 7f725aa00700 1 --2- 172.21.15.89:0/1343357017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f725c160d70 0x7f725c0b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:37:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.235+0000 7f725a000700 1 --2- 172.21.15.89:0/1343357017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c15ef70 0x7f725c15e7f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f725c072840 tx=0x7f7248009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b5b78fcf7625e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.235+0000 7f7259600700 1 -- 172.21.15.89:0/1343357017 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724800b870 con 0x7f725c15ef70 2024-04-02T07:37:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.235+0000 7f7259600700 1 -- 172.21.15.89:0/1343357017 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724800b9d0 con 0x7f725c15ef70 2024-04-02T07:37:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.235+0000 7f7259600700 1 -- 172.21.15.89:0/1343357017 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724800bd60 con 0x7f725c15ef70 2024-04-02T07:37:26.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.235+0000 7f7261600700 1 -- 172.21.15.89:0/1343357017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c15ef70 msgr2=0x7f725c15e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:26.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.235+0000 7f7261600700 1 --2- 172.21.15.89:0/1343357017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c15ef70 0x7f725c15e7f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f725c072840 tx=0x7f7248009bd0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.236+0000 7f7261600700 1 -- 172.21.15.89:0/1343357017 shutdown_connections 2024-04-02T07:37:26.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.236+0000 7f7261600700 1 --2- 172.21.15.89:0/1343357017 >> v2:172.21.15.113:3300/0 conn(0x7f725c06a790 0x7f725c06ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.236+0000 7f7261600700 1 --2- 172.21.15.89:0/1343357017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f725c160d70 0x7f725c0b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.236+0000 7f7261600700 1 --2- 172.21.15.89:0/1343357017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c15ef70 0x7f725c15e7f0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.236+0000 7f7261600700 1 -- 172.21.15.89:0/1343357017 >> 172.21.15.89:0/1343357017 conn(0x7f725c0b5720 msgr2=0x7f725c0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:26.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.236+0000 7f7261600700 1 -- 172.21.15.89:0/1343357017 shutdown_connections 2024-04-02T07:37:26.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.236+0000 7f7261600700 1 -- 172.21.15.89:0/1343357017 wait complete. 2024-04-02T07:37:26.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.237+0000 7f7261600700 1 Processor -- start 2024-04-02T07:37:26.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.238+0000 7f7261600700 1 -- start start 2024-04-02T07:37:26.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.238+0000 7f7261600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c06a790 0x7f725c09c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:26.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.238+0000 7f7261600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f725c15ef70 0x7f725c09de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:26.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.238+0000 7f7261600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f725c160d70 0x7f725c0ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:26.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.239+0000 7f7261600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725c09e370 con 0x7f725c06a790 2024-04-02T07:37:26.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.239+0000 7f7261600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725c0ca660 con 0x7f725c160d70 2024-04-02T07:37:26.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.239+0000 7f7261600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725c09cd90 con 0x7f725c15ef70 2024-04-02T07:37:26.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.239+0000 7f725aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c06a790 0x7f725c09c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:26.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.239+0000 7f725aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c06a790 0x7f725c09c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39250/0 (socket says 172.21.15.89:39250) 2024-04-02T07:37:26.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.239+0000 7f725aa00700 1 -- 172.21.15.89:0/160565685 learned_addr learned my addr 172.21.15.89:0/160565685 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:26.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.239+0000 7f725aa00700 1 -- 172.21.15.89:0/160565685 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f725c15ef70 msgr2=0x7f725c09de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:26.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.239+0000 7f725aa00700 1 --2- 172.21.15.89:0/160565685 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f725c15ef70 0x7f725c09de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.239+0000 7f725aa00700 1 -- 172.21.15.89:0/160565685 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f725c160d70 msgr2=0x7f725c0ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:26.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.239+0000 7f725aa00700 1 --2- 172.21.15.89:0/160565685 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f725c160d70 0x7f725c0ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.239+0000 7f725aa00700 1 -- 172.21.15.89:0/160565685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7248009550 con 0x7f725c06a790 2024-04-02T07:37:26.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.239+0000 7f725aa00700 1 --2- 172.21.15.89:0/160565685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c06a790 0x7f725c09c690 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f724c00c750 tx=0x7f724c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:26.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.240+0000 7f7252a00700 1 -- 172.21.15.89:0/160565685 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c011800 con 0x7f725c06a790 2024-04-02T07:37:26.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.240+0000 7f7252a00700 1 -- 172.21.15.89:0/160565685 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724c011960 con 0x7f725c06a790 2024-04-02T07:37:26.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.240+0000 7f7252a00700 1 -- 172.21.15.89:0/160565685 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c011cd0 con 0x7f725c06a790 2024-04-02T07:37:26.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.240+0000 7f7261600700 1 -- 172.21.15.89:0/160565685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f725c09d070 con 0x7f725c06a790 2024-04-02T07:37:26.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.240+0000 7f7261600700 1 -- 172.21.15.89:0/160565685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725c09d630 con 0x7f725c06a790 2024-04-02T07:37:26.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.241+0000 7f7261600700 1 -- 172.21.15.89:0/160565685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f725c067720 con 0x7f725c06a790 2024-04-02T07:37:26.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.242+0000 7f7252a00700 1 -- 172.21.15.89:0/160565685 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 10) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f724c021350 con 0x7f725c06a790 2024-04-02T07:37:26.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.243+0000 7f7252a00700 1 --2- 172.21.15.89:0/160565685 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f723c06ae00 0x7f723c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:26.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.243+0000 7f7252a00700 1 -- 172.21.15.89:0/160565685 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f724c0896a0 con 0x7f725c06a790 2024-04-02T07:37:26.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.243+0000 7f725a000700 1 --2- 172.21.15.89:0/160565685 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f723c06ae00 0x7f723c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:26.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.243+0000 7f725a000700 1 --2- 172.21.15.89:0/160565685 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f723c06ae00 0x7f723c06d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f725c15f3b0 tx=0x7f7248010000 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:26.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.247+0000 7f7252a00700 1 -- 172.21.15.89:0/160565685 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f724c054560 con 0x7f725c06a790 2024-04-02T07:37:26.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.480+0000 7f7261600700 1 -- 172.21.15.89:0/160565685 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f725c0b51c0 con 0x7f723c06ae00 2024-04-02T07:37:26.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.482+0000 7f7252a00700 1 -- 172.21.15.89:0/160565685 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f724c087ac0 con 0x7f725c06a790 2024-04-02T07:37:26.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.562+0000 7f7252a00700 1 -- 172.21.15.89:0/160565685 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f724801f070 con 0x7f723c06ae00 2024-04-02T07:37:26.564 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path /snap_test_dir1/dir5 2024-04-02T07:37:26.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.565+0000 7f7261600700 1 -- 172.21.15.89:0/160565685 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f723c06ae00 msgr2=0x7f723c06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:26.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.565+0000 7f7261600700 1 --2- 172.21.15.89:0/160565685 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f723c06ae00 0x7f723c06d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f725c15f3b0 tx=0x7f7248010000 comp rx=0 tx=0).stop 2024-04-02T07:37:26.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.565+0000 7f7261600700 1 -- 172.21.15.89:0/160565685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c06a790 msgr2=0x7f725c09c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:26.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.565+0000 7f7261600700 1 --2- 172.21.15.89:0/160565685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c06a790 0x7f725c09c690 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f724c00c750 tx=0x7f724c00cb10 comp rx=0 tx=0).stop 2024-04-02T07:37:26.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.565+0000 7f7261600700 1 -- 172.21.15.89:0/160565685 shutdown_connections 2024-04-02T07:37:26.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.565+0000 7f7261600700 1 --2- 172.21.15.89:0/160565685 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f723c06ae00 0x7f723c06d2c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.565+0000 7f7261600700 1 --2- 172.21.15.89:0/160565685 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f725c160d70 0x7f725c0ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.565+0000 7f7261600700 1 --2- 172.21.15.89:0/160565685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c06a790 0x7f725c09c690 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.565+0000 7f7261600700 1 --2- 172.21.15.89:0/160565685 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f725c15ef70 0x7f725c09de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.565+0000 7f7261600700 1 -- 172.21.15.89:0/160565685 >> 172.21.15.89:0/160565685 conn(0x7f725c0b5720 msgr2=0x7f725c06df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:26.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.565+0000 7f7261600700 1 -- 172.21.15.89:0/160565685 shutdown_connections 2024-04-02T07:37:26.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.565+0000 7f7261600700 1 -- 172.21.15.89:0/160565685 wait complete. 2024-04-02T07:37:26.584 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2024-04-02T07:37:26.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:26.585 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2024-04-02T07:37:26.631 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2024-04-02T07:37:26.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.862+0000 7fb007600700 1 Processor -- start 2024-04-02T07:37:26.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.863+0000 7fb007600700 1 -- start start 2024-04-02T07:37:26.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.864+0000 7fb007600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb0000d16a0 0x7fb0000d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:26.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.864+0000 7fb007600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb000161420 con 0x7fb0000d2370 2024-04-02T07:37:26.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.864+0000 7fb007600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb000161700 con 0x7fb00015d9d0 2024-04-02T07:37:26.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.864+0000 7fb007600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0000d4340 con 0x7fb0000d16a0 2024-04-02T07:37:26.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.864+0000 7fb005600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb0000d2370 0x7fb0000ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56606/0 (socket says 172.21.15.89:56606) 2024-04-02T07:37:26.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.864+0000 7fb005600700 1 -- 172.21.15.89:0/296267041 learned_addr learned my addr 172.21.15.89:0/296267041 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:26.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.864+0000 7fafffe00700 1 --2- 172.21.15.89:0/296267041 >> v2:172.21.15.113:3300/0 conn(0x7fb0000d16a0 0x7fb0000d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:26.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.864+0000 7fafff400700 1 -- 172.21.15.89:0/296267041 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642300355 0 0) 0x7fb000161420 con 0x7fb0000d2370 2024-04-02T07:37:26.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.865+0000 7fafff400700 1 -- 172.21.15.89:0/296267041 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe80035b0 con 0x7fb0000d2370 2024-04-02T07:37:26.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.865+0000 7fafff400700 1 -- 172.21.15.89:0/296267041 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766192596 0 0) 0x7fb000161700 con 0x7fb00015d9d0 2024-04-02T07:37:26.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.865+0000 7fafff400700 1 -- 172.21.15.89:0/296267041 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb000161420 con 0x7fb00015d9d0 2024-04-02T07:37:26.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.865+0000 7fafffe00700 1 -- 172.21.15.89:0/296267041 >> v1:172.21.15.113:6789/0 conn(0x7fb00015d9d0 legacy=0x7fb0000d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:26.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.865+0000 7fafffe00700 1 -- 172.21.15.89:0/296267041 >> v1:172.21.15.89:6789/0 conn(0x7fb0000d2370 legacy=0x7fb0000ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:26.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.865+0000 7fafffe00700 1 -- 172.21.15.89:0/296267041 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0001619e0 con 0x7fb0000d16a0 2024-04-02T07:37:26.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.865+0000 7fafffe00700 1 --2- 172.21.15.89:0/296267041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0000d16a0 0x7fb0000d1e30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fafec009820 tx=0x7fafec009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=886e4aa7e01a32f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:26.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.865+0000 7fafff400700 1 -- 172.21.15.89:0/296267041 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec005e30 con 0x7fb0000d16a0 2024-04-02T07:37:26.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.865+0000 7fafff400700 1 -- 172.21.15.89:0/296267041 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafec006030 con 0x7fb0000d16a0 2024-04-02T07:37:26.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.865+0000 7fafff400700 1 -- 172.21.15.89:0/296267041 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec0063c0 con 0x7fb0000d16a0 2024-04-02T07:37:26.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.866+0000 7fb007600700 1 -- 172.21.15.89:0/296267041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0000d16a0 msgr2=0x7fb0000d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:26.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.866+0000 7fb007600700 1 --2- 172.21.15.89:0/296267041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0000d16a0 0x7fb0000d1e30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fafec009820 tx=0x7fafec009b30 comp rx=0 tx=0).stop 2024-04-02T07:37:26.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.866+0000 7fb007600700 1 -- 172.21.15.89:0/296267041 shutdown_connections 2024-04-02T07:37:26.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.866+0000 7fb007600700 1 --2- 172.21.15.89:0/296267041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0000d16a0 0x7fb0000d1e30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.866+0000 7fb007600700 1 -- 172.21.15.89:0/296267041 >> 172.21.15.89:0/296267041 conn(0x7fb00008aff0 msgr2=0x7fb0000d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:26.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.866+0000 7fb007600700 1 -- 172.21.15.89:0/296267041 shutdown_connections 2024-04-02T07:37:26.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.866+0000 7fb007600700 1 -- 172.21.15.89:0/296267041 wait complete. 2024-04-02T07:37:26.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.868+0000 7fb007600700 1 Processor -- start 2024-04-02T07:37:26.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.868+0000 7fb007600700 1 -- start start 2024-04-02T07:37:26.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb007600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0000d16a0 0x7fb0000cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:26.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb007600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0000d2370 0x7fb0000cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:26.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb007600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb00015d9d0 0x7fb0000b78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:26.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb007600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0000cd580 con 0x7fb0000d16a0 2024-04-02T07:37:26.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb007600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0000cb670 con 0x7fb00015d9d0 2024-04-02T07:37:26.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb007600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0000cb950 con 0x7fb0000d2370 2024-04-02T07:37:26.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb004c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0000d16a0 0x7fb0000cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb004c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0000d16a0 0x7fb0000cb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39262/0 (socket says 172.21.15.89:39262) 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb004c00700 1 -- 172.21.15.89:0/986475563 learned_addr learned my addr 172.21.15.89:0/986475563 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb005600700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb00015d9d0 0x7fb0000b78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fafffe00700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0000d2370 0x7fb0000cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb004c00700 1 -- 172.21.15.89:0/986475563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0000d2370 msgr2=0x7fb0000cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb004c00700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0000d2370 0x7fb0000cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb004c00700 1 -- 172.21.15.89:0/986475563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb00015d9d0 msgr2=0x7fb0000b78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb004c00700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb00015d9d0 0x7fb0000b78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb004c00700 1 -- 172.21.15.89:0/986475563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec009430 con 0x7fb0000d16a0 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.869+0000 7fb005600700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb00015d9d0 0x7fb0000b78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.870+0000 7fb004c00700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0000d16a0 0x7fb0000cb000 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7faff400c500 tx=0x7faff400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.870+0000 7faffcc00700 1 -- 172.21.15.89:0/986475563 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff40052d0 con 0x7fb0000d16a0 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.870+0000 7faffcc00700 1 -- 172.21.15.89:0/986475563 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff4005450 con 0x7fb0000d16a0 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.870+0000 7faffcc00700 1 -- 172.21.15.89:0/986475563 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff40038c0 con 0x7fb0000d16a0 2024-04-02T07:37:26.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.870+0000 7fb007600700 1 -- 172.21.15.89:0/986475563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0000cbc30 con 0x7fb0000d16a0 2024-04-02T07:37:26.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.870+0000 7fb007600700 1 -- 172.21.15.89:0/986475563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0001579d0 con 0x7fb0000d16a0 2024-04-02T07:37:26.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.871+0000 7fb007600700 1 -- 172.21.15.89:0/986475563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0000d1ca0 con 0x7fb0000d16a0 2024-04-02T07:37:26.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.872+0000 7faffcc00700 1 -- 172.21.15.89:0/986475563 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7faff4003a20 con 0x7fb0000d16a0 2024-04-02T07:37:26.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.873+0000 7faffcc00700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fafe006ae90 0x7fafe006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:26.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.873+0000 7faffcc00700 1 -- 172.21.15.89:0/986475563 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7faff4088c60 con 0x7fb0000d16a0 2024-04-02T07:37:26.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.873+0000 7fafffe00700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fafe006ae90 0x7fafe006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:26.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.874+0000 7fafffe00700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fafe006ae90 0x7fafe006d350 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb0000d1ae0 tx=0x7fafec008070 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:26.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:26.877+0000 7faffcc00700 1 -- 172.21.15.89:0/986475563 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faff4053ab0 con 0x7fb0000d16a0 2024-04-02T07:37:27.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.106+0000 7fb007600700 1 -- 172.21.15.89:0/986475563 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb000161fa0 con 0x7fafe006ae90 2024-04-02T07:37:27.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.190+0000 7faffcc00700 1 -- 172.21.15.89:0/986475563 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fafec005c30 con 0x7fafe006ae90 2024-04-02T07:37:27.192 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path /snap_test_dir1/dir6 2024-04-02T07:37:27.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.192+0000 7fb007600700 1 -- 172.21.15.89:0/986475563 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fafe006ae90 msgr2=0x7fafe006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:27.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.192+0000 7fb007600700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fafe006ae90 0x7fafe006d350 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb0000d1ae0 tx=0x7fafec008070 comp rx=0 tx=0).stop 2024-04-02T07:37:27.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.192+0000 7fb007600700 1 -- 172.21.15.89:0/986475563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0000d16a0 msgr2=0x7fb0000cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:27.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.192+0000 7fb007600700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0000d16a0 0x7fb0000cb000 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7faff400c500 tx=0x7faff400c8c0 comp rx=0 tx=0).stop 2024-04-02T07:37:27.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.193+0000 7fb007600700 1 -- 172.21.15.89:0/986475563 shutdown_connections 2024-04-02T07:37:27.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.193+0000 7fb007600700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fafe006ae90 0x7fafe006d350 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:27.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.193+0000 7fb007600700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb00015d9d0 0x7fb0000b78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:27.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.193+0000 7fb007600700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0000d16a0 0x7fb0000cb000 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:27.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.193+0000 7fb007600700 1 --2- 172.21.15.89:0/986475563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0000d2370 0x7fb0000cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:27.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.193+0000 7fb007600700 1 -- 172.21.15.89:0/986475563 >> 172.21.15.89:0/986475563 conn(0x7fb00008aff0 msgr2=0x7fb0000cede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:27.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.193+0000 7fb007600700 1 -- 172.21.15.89:0/986475563 shutdown_connections 2024-04-02T07:37:27.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.193+0000 7fb007600700 1 -- 172.21.15.89:0/986475563 wait complete. 2024-04-02T07:37:27.212 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2024-04-02T07:37:27.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:27.213 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2024-04-02T07:37:27.259 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2024-04-02T07:37:27.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.492+0000 7ffb43600700 1 Processor -- start 2024-04-02T07:37:27.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.492+0000 7ffb43600700 1 -- start start 2024-04-02T07:37:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.493+0000 7ffb43600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb3c15d9d0 0x7ffb3c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.493+0000 7ffb43600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb3c0d16a0 0x7ffb3c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.493+0000 7ffb43600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb3c161420 con 0x7ffb3c0d2370 2024-04-02T07:37:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.493+0000 7ffb43600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb3c0d2820 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.493+0000 7ffb43600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c161700 con 0x7ffb3c0d16a0 2024-04-02T07:37:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.493+0000 7ffb40c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb3c15d9d0 0x7ffb3c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.494+0000 7ffb40c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb3c15d9d0 0x7ffb3c0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39272/0 (socket says 172.21.15.89:39272) 2024-04-02T07:37:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.494+0000 7ffb40c00700 1 -- 172.21.15.89:0/4106714766 learned_addr learned my addr 172.21.15.89:0/4106714766 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.494+0000 7ffb3be00700 1 --2- 172.21.15.89:0/4106714766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb3c0d16a0 0x7ffb3c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.494+0000 7ffb40c00700 1 -- 172.21.15.89:0/4106714766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb3c0d16a0 msgr2=0x7ffb3c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.494+0000 7ffb40c00700 1 --2- 172.21.15.89:0/4106714766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb3c0d16a0 0x7ffb3c0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.494+0000 7ffb40c00700 1 -- 172.21.15.89:0/4106714766 >> v1:172.21.15.113:6789/0 conn(0x7ffb3c0d2370 legacy=0x7ffb3c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.494+0000 7ffb40c00700 1 -- 172.21.15.89:0/4106714766 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3c1619e0 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.494+0000 7ffb40c00700 1 --2- 172.21.15.89:0/4106714766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb3c15d9d0 0x7ffb3c0d3c30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ffb300098f0 tx=0x7ffb30009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3237dd4a9a63cdd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.494+0000 7ffb3b400700 1 -- 172.21.15.89:0/4106714766 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3000b8a0 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.494+0000 7ffb3b400700 1 -- 172.21.15.89:0/4106714766 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3000ba00 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.494+0000 7ffb3b400700 1 -- 172.21.15.89:0/4106714766 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3000bd90 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.495+0000 7ffb43600700 1 -- 172.21.15.89:0/4106714766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb3c15d9d0 msgr2=0x7ffb3c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.495+0000 7ffb43600700 1 --2- 172.21.15.89:0/4106714766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb3c15d9d0 0x7ffb3c0d3c30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ffb300098f0 tx=0x7ffb30009c00 comp rx=0 tx=0).stop 2024-04-02T07:37:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.495+0000 7ffb43600700 1 -- 172.21.15.89:0/4106714766 shutdown_connections 2024-04-02T07:37:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.495+0000 7ffb43600700 1 --2- 172.21.15.89:0/4106714766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb3c15d9d0 0x7ffb3c0d3c30 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.495+0000 7ffb43600700 1 --2- 172.21.15.89:0/4106714766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb3c0d16a0 0x7ffb3c0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.495+0000 7ffb43600700 1 -- 172.21.15.89:0/4106714766 >> 172.21.15.89:0/4106714766 conn(0x7ffb3c08aff0 msgr2=0x7ffb3c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.495+0000 7ffb43600700 1 -- 172.21.15.89:0/4106714766 shutdown_connections 2024-04-02T07:37:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.495+0000 7ffb43600700 1 -- 172.21.15.89:0/4106714766 wait complete. 2024-04-02T07:37:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.497+0000 7ffb43600700 1 Processor -- start 2024-04-02T07:37:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.497+0000 7ffb43600700 1 -- start start 2024-04-02T07:37:27.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.498+0000 7ffb43600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0d16a0 0x7ffb3c155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:27.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.498+0000 7ffb43600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb3c0d2370 0x7ffb3c0b8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:27.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.498+0000 7ffb43600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb3c15d9d0 0x7ffb3c159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:27.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.498+0000 7ffb43600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c15e810 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.498+0000 7ffb43600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c0b8df0 con 0x7ffb3c0d16a0 2024-04-02T07:37:27.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.498+0000 7ffb43600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c15dee0 con 0x7ffb3c0d2370 2024-04-02T07:37:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.498+0000 7ffb41600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb3c15d9d0 0x7ffb3c159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.498+0000 7ffb40c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0d16a0 0x7ffb3c155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.498+0000 7ffb41600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb3c15d9d0 0x7ffb3c159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39284/0 (socket says 172.21.15.89:39284) 2024-04-02T07:37:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.498+0000 7ffb41600700 1 -- 172.21.15.89:0/3415602847 learned_addr learned my addr 172.21.15.89:0/3415602847 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.498+0000 7ffb3be00700 1 --2- 172.21.15.89:0/3415602847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb3c0d2370 0x7ffb3c0b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.499+0000 7ffb41600700 1 -- 172.21.15.89:0/3415602847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb3c0d2370 msgr2=0x7ffb3c0b8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.499+0000 7ffb41600700 1 --2- 172.21.15.89:0/3415602847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb3c0d2370 0x7ffb3c0b8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.499+0000 7ffb41600700 1 -- 172.21.15.89:0/3415602847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0d16a0 msgr2=0x7ffb3c155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.499+0000 7ffb41600700 1 --2- 172.21.15.89:0/3415602847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0d16a0 0x7ffb3c155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.499+0000 7ffb41600700 1 -- 172.21.15.89:0/3415602847 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb30009550 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.499+0000 7ffb41600700 1 --2- 172.21.15.89:0/3415602847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb3c15d9d0 0x7ffb3c159c50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ffb3400cc50 tx=0x7ffb3400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.499+0000 7ffb38c00700 1 -- 172.21.15.89:0/3415602847 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34003c00 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.499+0000 7ffb38c00700 1 -- 172.21.15.89:0/3415602847 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb34003d60 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.499+0000 7ffb43600700 1 -- 172.21.15.89:0/3415602847 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb3c15e1c0 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.499+0000 7ffb38c00700 1 -- 172.21.15.89:0/3415602847 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3400aab0 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.499+0000 7ffb43600700 1 -- 172.21.15.89:0/3415602847 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb3c155560 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.500+0000 7ffb43600700 1 -- 172.21.15.89:0/3415602847 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb3c0d1ca0 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.501+0000 7ffb38c00700 1 -- 172.21.15.89:0/3415602847 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7ffb3400ac10 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.502+0000 7ffb38c00700 1 --2- 172.21.15.89:0/3415602847 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb1c06ae90 0x7ffb1c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:27.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.502+0000 7ffb38c00700 1 -- 172.21.15.89:0/3415602847 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ffb34089720 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.502+0000 7ffb40c00700 1 --2- 172.21.15.89:0/3415602847 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb1c06ae90 0x7ffb1c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:27.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.503+0000 7ffb40c00700 1 --2- 172.21.15.89:0/3415602847 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb1c06ae90 0x7ffb1c06d350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffb30014040 tx=0x7ffb30008030 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:27.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.507+0000 7ffb38c00700 1 -- 172.21.15.89:0/3415602847 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffb34054570 con 0x7ffb3c15d9d0 2024-04-02T07:37:27.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.734+0000 7ffb43600700 1 -- 172.21.15.89:0/3415602847 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb3c15e050 con 0x7ffb1c06ae90 2024-04-02T07:37:27.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.826+0000 7ffb38c00700 1 -- 172.21.15.89:0/3415602847 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7ffb3c15e050 con 0x7ffb1c06ae90 2024-04-02T07:37:27.828 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path /snap_test_dir1/dir7 2024-04-02T07:37:27.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.828+0000 7ffb43600700 1 -- 172.21.15.89:0/3415602847 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb1c06ae90 msgr2=0x7ffb1c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:27.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.828+0000 7ffb43600700 1 --2- 172.21.15.89:0/3415602847 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb1c06ae90 0x7ffb1c06d350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffb30014040 tx=0x7ffb30008030 comp rx=0 tx=0).stop 2024-04-02T07:37:27.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.828+0000 7ffb43600700 1 -- 172.21.15.89:0/3415602847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb3c15d9d0 msgr2=0x7ffb3c159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:27.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.828+0000 7ffb43600700 1 --2- 172.21.15.89:0/3415602847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb3c15d9d0 0x7ffb3c159c50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ffb3400cc50 tx=0x7ffb3400cf60 comp rx=0 tx=0).stop 2024-04-02T07:37:27.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.828+0000 7ffb43600700 1 -- 172.21.15.89:0/3415602847 shutdown_connections 2024-04-02T07:37:27.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.828+0000 7ffb43600700 1 --2- 172.21.15.89:0/3415602847 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb1c06ae90 0x7ffb1c06d350 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:27.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.828+0000 7ffb43600700 1 --2- 172.21.15.89:0/3415602847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0d16a0 0x7ffb3c155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:27.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.828+0000 7ffb43600700 1 --2- 172.21.15.89:0/3415602847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb3c15d9d0 0x7ffb3c159c50 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:27.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.828+0000 7ffb43600700 1 --2- 172.21.15.89:0/3415602847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb3c0d2370 0x7ffb3c0b8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:27.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.828+0000 7ffb43600700 1 -- 172.21.15.89:0/3415602847 >> 172.21.15.89:0/3415602847 conn(0x7ffb3c08aff0 msgr2=0x7ffb3c0b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:27.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.828+0000 7ffb43600700 1 -- 172.21.15.89:0/3415602847 shutdown_connections 2024-04-02T07:37:27.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:27.828+0000 7ffb43600700 1 -- 172.21.15.89:0/3415602847 wait complete. 2024-04-02T07:37:27.848 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2024-04-02T07:37:27.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:27.849 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2024-04-02T07:37:27.895 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2024-04-02T07:37:28.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.126+0000 7fb99cc00700 1 Processor -- start 2024-04-02T07:37:28.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.127+0000 7fb99cc00700 1 -- start start 2024-04-02T07:37:28.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.128+0000 7fb99cc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb9980d16a0 0x7fb9980d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:28.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.128+0000 7fb99cc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9980d2370 0x7fb9980ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:28.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.128+0000 7fb99cc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb998161420 con 0x7fb99815d9d0 2024-04-02T07:37:28.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.128+0000 7fb99cc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9980d2820 con 0x7fb9980d16a0 2024-04-02T07:37:28.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.128+0000 7fb99cc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9980d4340 con 0x7fb9980d2370 2024-04-02T07:37:28.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.128+0000 7fb995400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb9980d16a0 0x7fb9980d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:28.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.128+0000 7fb996800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9980d2370 0x7fb9980ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:28.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.128+0000 7fb995e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb99815d9d0 0x7fb9980d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:49172/0 (socket says 172.21.15.89:49172) 2024-04-02T07:37:28.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.128+0000 7fb995e00700 1 -- 172.21.15.89:0/41987761 learned_addr learned my addr 172.21.15.89:0/41987761 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:28.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.129+0000 7fb994a00700 1 -- 172.21.15.89:0/41987761 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690277122 0 0) 0x7fb998161420 con 0x7fb99815d9d0 2024-04-02T07:37:28.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.129+0000 7fb994a00700 1 -- 172.21.15.89:0/41987761 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb97c0035b0 con 0x7fb99815d9d0 2024-04-02T07:37:28.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.129+0000 7fb995400700 1 -- 172.21.15.89:0/41987761 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9980d2370 msgr2=0x7fb9980ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:28.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.129+0000 7fb995400700 1 --2- 172.21.15.89:0/41987761 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9980d2370 0x7fb9980ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:28.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.129+0000 7fb995400700 1 -- 172.21.15.89:0/41987761 >> v1:172.21.15.113:6789/0 conn(0x7fb99815d9d0 legacy=0x7fb9980d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:28.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.129+0000 7fb995400700 1 -- 172.21.15.89:0/41987761 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb998161700 con 0x7fb9980d16a0 2024-04-02T07:37:28.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.129+0000 7fb996800700 1 --2- 172.21.15.89:0/41987761 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9980d2370 0x7fb9980ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:37:28.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.129+0000 7fb995400700 1 --2- 172.21.15.89:0/41987761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9980d16a0 0x7fb9980d1e30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb984009820 tx=0x7fb984009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4600052a3ea74a8a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:28.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.129+0000 7fb994a00700 1 -- 172.21.15.89:0/41987761 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984005e30 con 0x7fb9980d16a0 2024-04-02T07:37:28.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.129+0000 7fb994a00700 1 -- 172.21.15.89:0/41987761 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb984006030 con 0x7fb9980d16a0 2024-04-02T07:37:28.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.129+0000 7fb994a00700 1 -- 172.21.15.89:0/41987761 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9840063c0 con 0x7fb9980d16a0 2024-04-02T07:37:28.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.130+0000 7fb99cc00700 1 -- 172.21.15.89:0/41987761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9980d16a0 msgr2=0x7fb9980d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:28.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.130+0000 7fb99cc00700 1 --2- 172.21.15.89:0/41987761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9980d16a0 0x7fb9980d1e30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb984009820 tx=0x7fb984009b30 comp rx=0 tx=0).stop 2024-04-02T07:37:28.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.130+0000 7fb99cc00700 1 -- 172.21.15.89:0/41987761 shutdown_connections 2024-04-02T07:37:28.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.130+0000 7fb99cc00700 1 --2- 172.21.15.89:0/41987761 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9980d2370 0x7fb9980ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:28.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.130+0000 7fb99cc00700 1 --2- 172.21.15.89:0/41987761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9980d16a0 0x7fb9980d1e30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:28.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.130+0000 7fb99cc00700 1 -- 172.21.15.89:0/41987761 >> 172.21.15.89:0/41987761 conn(0x7fb99808aff0 msgr2=0x7fb9980d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:28.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.130+0000 7fb99cc00700 1 -- 172.21.15.89:0/41987761 shutdown_connections 2024-04-02T07:37:28.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.130+0000 7fb99cc00700 1 -- 172.21.15.89:0/41987761 wait complete. 2024-04-02T07:37:28.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.132+0000 7fb99cc00700 1 Processor -- start 2024-04-02T07:37:28.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.132+0000 7fb99cc00700 1 -- start start 2024-04-02T07:37:28.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb99cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9980d16a0 0x7fb9980cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:28.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb99cc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9980d2370 0x7fb99814ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:28.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb99cc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb99815d9d0 0x7fb9980cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:28.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb99cc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9980cac40 con 0x7fb9980d16a0 2024-04-02T07:37:28.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb99cc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9980cc600 con 0x7fb99815d9d0 2024-04-02T07:37:28.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb99cc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9980cc8e0 con 0x7fb9980d2370 2024-04-02T07:37:28.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb995e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9980d16a0 0x7fb9980cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:28.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb995e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9980d16a0 0x7fb9980cb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39294/0 (socket says 172.21.15.89:39294) 2024-04-02T07:37:28.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb995e00700 1 -- 172.21.15.89:0/2410764785 learned_addr learned my addr 172.21.15.89:0/2410764785 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:28.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb995400700 1 --2- 172.21.15.89:0/2410764785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9980d2370 0x7fb99814ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:28.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb996800700 1 --2- 172.21.15.89:0/2410764785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb99815d9d0 0x7fb9980cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:28.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb995e00700 1 -- 172.21.15.89:0/2410764785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9980d2370 msgr2=0x7fb99814ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:28.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb995e00700 1 --2- 172.21.15.89:0/2410764785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9980d2370 0x7fb99814ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:28.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb995e00700 1 -- 172.21.15.89:0/2410764785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb99815d9d0 msgr2=0x7fb9980cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:28.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb995e00700 1 --2- 172.21.15.89:0/2410764785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb99815d9d0 0x7fb9980cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:28.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.133+0000 7fb995e00700 1 -- 172.21.15.89:0/2410764785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb990009a70 con 0x7fb9980d16a0 2024-04-02T07:37:28.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.134+0000 7fb995e00700 1 --2- 172.21.15.89:0/2410764785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9980d16a0 0x7fb9980cb0d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb988001eb0 tx=0x7fb98800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:28.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.134+0000 7fb98e000700 1 -- 172.21.15.89:0/2410764785 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb988005d70 con 0x7fb9980d16a0 2024-04-02T07:37:28.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.134+0000 7fb98e000700 1 -- 172.21.15.89:0/2410764785 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9880070d0 con 0x7fb9980d16a0 2024-04-02T07:37:28.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.134+0000 7fb99cc00700 1 -- 172.21.15.89:0/2410764785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb984009430 con 0x7fb9980d16a0 2024-04-02T07:37:28.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.134+0000 7fb98e000700 1 -- 172.21.15.89:0/2410764785 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb988007460 con 0x7fb9980d16a0 2024-04-02T07:37:28.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.134+0000 7fb99cc00700 1 -- 172.21.15.89:0/2410764785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9980ca8d0 con 0x7fb9980d16a0 2024-04-02T07:37:28.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.135+0000 7fb99cc00700 1 -- 172.21.15.89:0/2410764785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9980d1ca0 con 0x7fb9980d16a0 2024-04-02T07:37:28.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.136+0000 7fb98e000700 1 -- 172.21.15.89:0/2410764785 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fb9880075c0 con 0x7fb9980d16a0 2024-04-02T07:37:28.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.138+0000 7fb98e000700 1 --2- 172.21.15.89:0/2410764785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb97806ae90 0x7fb97806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:28.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.138+0000 7fb98e000700 1 -- 172.21.15.89:0/2410764785 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb988088cf0 con 0x7fb9980d16a0 2024-04-02T07:37:28.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.138+0000 7fb995400700 1 --2- 172.21.15.89:0/2410764785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb97806ae90 0x7fb97806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:28.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.139+0000 7fb995400700 1 --2- 172.21.15.89:0/2410764785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb97806ae90 0x7fb97806d350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb9980d1ae0 tx=0x7fb984008070 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:28.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.143+0000 7fb98e000700 1 -- 172.21.15.89:0/2410764785 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb98808d070 con 0x7fb9980d16a0 2024-04-02T07:37:28.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.368+0000 7fb99cc00700 1 -- 172.21.15.89:0/2410764785 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb998161fa0 con 0x7fb97806ae90 2024-04-02T07:37:28.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.450+0000 7fb98e000700 1 -- 172.21.15.89:0/2410764785 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fb984005c30 con 0x7fb97806ae90 2024-04-02T07:37:28.452 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path /snap_test_dir1/dir8 2024-04-02T07:37:28.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.452+0000 7fb99cc00700 1 -- 172.21.15.89:0/2410764785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb97806ae90 msgr2=0x7fb97806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:28.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.452+0000 7fb99cc00700 1 --2- 172.21.15.89:0/2410764785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb97806ae90 0x7fb97806d350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb9980d1ae0 tx=0x7fb984008070 comp rx=0 tx=0).stop 2024-04-02T07:37:28.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.452+0000 7fb99cc00700 1 -- 172.21.15.89:0/2410764785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9980d16a0 msgr2=0x7fb9980cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:28.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.452+0000 7fb99cc00700 1 --2- 172.21.15.89:0/2410764785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9980d16a0 0x7fb9980cb0d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb988001eb0 tx=0x7fb98800c860 comp rx=0 tx=0).stop 2024-04-02T07:37:28.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.452+0000 7fb99cc00700 1 -- 172.21.15.89:0/2410764785 shutdown_connections 2024-04-02T07:37:28.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.452+0000 7fb99cc00700 1 --2- 172.21.15.89:0/2410764785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb97806ae90 0x7fb97806d350 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:28.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.452+0000 7fb99cc00700 1 --2- 172.21.15.89:0/2410764785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb99815d9d0 0x7fb9980cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:28.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.452+0000 7fb99cc00700 1 --2- 172.21.15.89:0/2410764785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9980d16a0 0x7fb9980cb0d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:28.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.452+0000 7fb99cc00700 1 --2- 172.21.15.89:0/2410764785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9980d2370 0x7fb99814ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:28.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.452+0000 7fb99cc00700 1 -- 172.21.15.89:0/2410764785 >> 172.21.15.89:0/2410764785 conn(0x7fb99808aff0 msgr2=0x7fb9980cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:28.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.452+0000 7fb99cc00700 1 -- 172.21.15.89:0/2410764785 shutdown_connections 2024-04-02T07:37:28.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.452+0000 7fb99cc00700 1 -- 172.21.15.89:0/2410764785 wait complete. 2024-04-02T07:37:28.472 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2024-04-02T07:37:28.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:37:28.473 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2024-04-02T07:37:28.519 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2024-04-02T07:37:28.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.711+0000 7fcf0f600700 1 Processor -- start 2024-04-02T07:37:28.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.711+0000 7fcf0f600700 1 -- start start 2024-04-02T07:37:28.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.712+0000 7fcf0f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf080d16a0 0x7fcf080d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:28.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.712+0000 7fcf0f600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcf080d2370 0x7fcf080ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:28.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.712+0000 7fcf0f600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf08161420 con 0x7fcf0815d9d0 2024-04-02T07:37:28.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.712+0000 7fcf0f600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf080d2820 con 0x7fcf080d2370 2024-04-02T07:37:28.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.712+0000 7fcf0f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf080d4340 con 0x7fcf080d16a0 2024-04-02T07:37:28.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.712+0000 7fcf0d600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcf080d2370 0x7fcf080ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:28.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.712+0000 7fcf0cc00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcf0815d9d0 0x7fcf080d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56620/0 (socket says 172.21.15.89:56620) 2024-04-02T07:37:28.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.712+0000 7fcf0cc00700 1 -- 172.21.15.89:0/3822450388 learned_addr learned my addr 172.21.15.89:0/3822450388 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:28.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.713+0000 7fcf07e00700 1 --2- 172.21.15.89:0/3822450388 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf080d16a0 0x7fcf080d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:28.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.713+0000 7fcf07400700 1 -- 172.21.15.89:0/3822450388 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824800423 0 0) 0x7fcf08161420 con 0x7fcf0815d9d0 2024-04-02T07:37:28.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.713+0000 7fcf0d600700 1 -- 172.21.15.89:0/3822450388 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf080d16a0 msgr2=0x7fcf080d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:28.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.713+0000 7fcf0d600700 1 --2- 172.21.15.89:0/3822450388 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf080d16a0 0x7fcf080d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:28.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.713+0000 7fcf0d600700 1 -- 172.21.15.89:0/3822450388 >> v1:172.21.15.89:6789/0 conn(0x7fcf0815d9d0 legacy=0x7fcf080d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:28.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.713+0000 7fcf0d600700 1 -- 172.21.15.89:0/3822450388 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf08161700 con 0x7fcf080d2370 2024-04-02T07:37:28.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.713+0000 7fcf0d600700 1 --2- 172.21.15.89:0/3822450388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf080d2370 0x7fcf080ce360 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009dc0 tx=0x7fcf0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c827625582b78b03 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:28.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.713+0000 7fcf07400700 1 -- 172.21.15.89:0/3822450388 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0000e070 con 0x7fcf080d2370 2024-04-02T07:37:28.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.713+0000 7fcf07400700 1 -- 172.21.15.89:0/3822450388 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0000bae0 con 0x7fcf080d2370 2024-04-02T07:37:28.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.713+0000 7fcf07400700 1 -- 172.21.15.89:0/3822450388 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0000be90 con 0x7fcf080d2370 2024-04-02T07:37:28.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.714+0000 7fcf0f600700 1 -- 172.21.15.89:0/3822450388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf080d2370 msgr2=0x7fcf080ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:28.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.714+0000 7fcf0f600700 1 --2- 172.21.15.89:0/3822450388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf080d2370 0x7fcf080ce360 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009dc0 tx=0x7fcf0000b330 comp rx=0 tx=0).stop 2024-04-02T07:37:28.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.714+0000 7fcf0f600700 1 -- 172.21.15.89:0/3822450388 shutdown_connections 2024-04-02T07:37:28.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.714+0000 7fcf0f600700 1 --2- 172.21.15.89:0/3822450388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf080d2370 0x7fcf080ce360 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:28.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.714+0000 7fcf0f600700 1 --2- 172.21.15.89:0/3822450388 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf080d16a0 0x7fcf080d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:28.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.714+0000 7fcf0f600700 1 -- 172.21.15.89:0/3822450388 >> 172.21.15.89:0/3822450388 conn(0x7fcf0808aff0 msgr2=0x7fcf080d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:28.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.714+0000 7fcf0f600700 1 -- 172.21.15.89:0/3822450388 shutdown_connections 2024-04-02T07:37:28.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.714+0000 7fcf0f600700 1 -- 172.21.15.89:0/3822450388 wait complete. 2024-04-02T07:37:28.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.716+0000 7fcf0f600700 1 Processor -- start 2024-04-02T07:37:28.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.716+0000 7fcf0f600700 1 -- start start 2024-04-02T07:37:28.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.717+0000 7fcf0f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf080d16a0 0x7fcf081f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:28.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.717+0000 7fcf0f600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf080d2370 0x7fcf0815f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:28.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.717+0000 7fcf0f600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf0815d9d0 0x7fcf080b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:28.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.717+0000 7fcf0f600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf08160a80 con 0x7fcf080d2370 2024-04-02T07:37:28.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.717+0000 7fcf0f600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf081f64e0 con 0x7fcf0815d9d0 2024-04-02T07:37:28.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.717+0000 7fcf0f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0815fe00 con 0x7fcf080d16a0 2024-04-02T07:37:28.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.717+0000 7fcf07e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf080d2370 0x7fcf0815f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:28.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.717+0000 7fcf07e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf080d2370 0x7fcf0815f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39316/0 (socket says 172.21.15.89:39316) 2024-04-02T07:37:28.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.717+0000 7fcf07e00700 1 -- 172.21.15.89:0/3720416222 learned_addr learned my addr 172.21.15.89:0/3720416222 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:28.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.717+0000 7fcf07e00700 1 -- 172.21.15.89:0/3720416222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf080d16a0 msgr2=0x7fcf081f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:28.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.717+0000 7fcf07e00700 1 --2- 172.21.15.89:0/3720416222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf080d16a0 0x7fcf081f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:28.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.717+0000 7fcf07e00700 1 -- 172.21.15.89:0/3720416222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf0815d9d0 msgr2=0x7fcf080b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:28.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.717+0000 7fcf07e00700 1 --2- 172.21.15.89:0/3720416222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf0815d9d0 0x7fcf080b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:28.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.717+0000 7fcf07e00700 1 -- 172.21.15.89:0/3720416222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf00009a70 con 0x7fcf080d2370 2024-04-02T07:37:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.718+0000 7fcf07e00700 1 --2- 172.21.15.89:0/3720416222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf080d2370 0x7fcf0815f420 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fcef400e960 tx=0x7fcef400ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.718+0000 7fcf04c00700 1 -- 172.21.15.89:0/3720416222 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef40096f0 con 0x7fcf080d2370 2024-04-02T07:37:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.718+0000 7fcf04c00700 1 -- 172.21.15.89:0/3720416222 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef400c850 con 0x7fcf080d2370 2024-04-02T07:37:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.718+0000 7fcf0f600700 1 -- 172.21.15.89:0/3720416222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf081600e0 con 0x7fcf080d2370 2024-04-02T07:37:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.718+0000 7fcf04c00700 1 -- 172.21.15.89:0/3720416222 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef400cc00 con 0x7fcf080d2370 2024-04-02T07:37:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.718+0000 7fcf0f600700 1 -- 172.21.15.89:0/3720416222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf081606a0 con 0x7fcf080d2370 2024-04-02T07:37:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.719+0000 7fcf0f600700 1 -- 172.21.15.89:0/3720416222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf080d1ca0 con 0x7fcf080d2370 2024-04-02T07:37:28.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.720+0000 7fcf04c00700 1 -- 172.21.15.89:0/3720416222 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fcef4013470 con 0x7fcf080d2370 2024-04-02T07:37:28.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.721+0000 7fcf04c00700 1 --2- 172.21.15.89:0/3720416222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcee806ae90 0x7fcee806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:28.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.721+0000 7fcf04c00700 1 -- 172.21.15.89:0/3720416222 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fcef4089c70 con 0x7fcf080d2370 2024-04-02T07:37:28.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.721+0000 7fcf0cc00700 1 --2- 172.21.15.89:0/3720416222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcee806ae90 0x7fcee806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:28.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.722+0000 7fcf0cc00700 1 --2- 172.21.15.89:0/3720416222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcee806ae90 0x7fcee806d350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcefc001610 tx=0x7fcefc0079f0 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:28.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.725+0000 7fcf04c00700 1 -- 172.21.15.89:0/3720416222 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcef4054ac0 con 0x7fcf080d2370 2024-04-02T07:37:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:28.957+0000 7fcf0f600700 1 -- 172.21.15.89:0/3720416222 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf08161fa0 con 0x7fcee806ae90 2024-04-02T07:37:29.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.040+0000 7fcf04c00700 1 -- 172.21.15.89:0/3720416222 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fcefc004870 con 0x7fcee806ae90 2024-04-02T07:37:29.042 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path /snap_test_dir1/dir9 2024-04-02T07:37:29.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.043+0000 7fcf0f600700 1 -- 172.21.15.89:0/3720416222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcee806ae90 msgr2=0x7fcee806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:29.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.043+0000 7fcf0f600700 1 --2- 172.21.15.89:0/3720416222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcee806ae90 0x7fcee806d350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcefc001610 tx=0x7fcefc0079f0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.043+0000 7fcf0f600700 1 -- 172.21.15.89:0/3720416222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf080d2370 msgr2=0x7fcf0815f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:29.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.043+0000 7fcf0f600700 1 --2- 172.21.15.89:0/3720416222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf080d2370 0x7fcf0815f420 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fcef400e960 tx=0x7fcef400ed20 comp rx=0 tx=0).stop 2024-04-02T07:37:29.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.043+0000 7fcf0f600700 1 -- 172.21.15.89:0/3720416222 shutdown_connections 2024-04-02T07:37:29.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.043+0000 7fcf0f600700 1 --2- 172.21.15.89:0/3720416222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcee806ae90 0x7fcee806d350 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.043+0000 7fcf0f600700 1 --2- 172.21.15.89:0/3720416222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf0815d9d0 0x7fcf080b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.043+0000 7fcf0f600700 1 --2- 172.21.15.89:0/3720416222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcf080d16a0 0x7fcf081f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.043+0000 7fcf0f600700 1 --2- 172.21.15.89:0/3720416222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf080d2370 0x7fcf0815f420 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.043+0000 7fcf0f600700 1 -- 172.21.15.89:0/3720416222 >> 172.21.15.89:0/3720416222 conn(0x7fcf0808aff0 msgr2=0x7fcf0815ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:29.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.043+0000 7fcf0f600700 1 -- 172.21.15.89:0/3720416222 shutdown_connections 2024-04-02T07:37:29.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.043+0000 7fcf0f600700 1 -- 172.21.15.89:0/3720416222 wait complete. 2024-04-02T07:37:29.063 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2024-04-02T07:37:29.064 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2024-04-02T07:37:29.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.255+0000 7fcb27000700 1 Processor -- start 2024-04-02T07:37:29.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.256+0000 7fcb27000700 1 -- start start 2024-04-02T07:37:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.257+0000 7fcb27000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb2015d9d0 0x7fcb200d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.257+0000 7fcb27000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcb200d2540 0x7fcb200ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.257+0000 7fcb27000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb20161420 con 0x7fcb200d16a0 2024-04-02T07:37:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.257+0000 7fcb27000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb2015edc0 con 0x7fcb200d2540 2024-04-02T07:37:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.257+0000 7fcb27000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb20161700 con 0x7fcb2015d9d0 2024-04-02T07:37:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.258+0000 7fcb1f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcb200d16a0 0x7fcb200d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56508/0 (socket says 172.21.15.89:56508) 2024-04-02T07:37:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.258+0000 7fcb1f400700 1 -- 172.21.15.89:0/3502790314 learned_addr learned my addr 172.21.15.89:0/3502790314 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.258+0000 7fcb1fe00700 1 --2- 172.21.15.89:0/3502790314 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb2015d9d0 0x7fcb200d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.258+0000 7fcb25000700 1 --2- 172.21.15.89:0/3502790314 >> v2:172.21.15.113:3300/0 conn(0x7fcb200d2540 0x7fcb200ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.258+0000 7fcb1ea00700 1 -- 172.21.15.89:0/3502790314 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465020724 0 0) 0x7fcb20161420 con 0x7fcb200d16a0 2024-04-02T07:37:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.258+0000 7fcb1ea00700 1 -- 172.21.15.89:0/3502790314 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb040035b0 con 0x7fcb200d16a0 2024-04-02T07:37:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.258+0000 7fcb1ea00700 1 -- 172.21.15.89:0/3502790314 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2549389872 0 0) 0x7fcb040035b0 con 0x7fcb200d16a0 2024-04-02T07:37:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.258+0000 7fcb1ea00700 1 -- 172.21.15.89:0/3502790314 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb2015d9d0 msgr2=0x7fcb200d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.258+0000 7fcb1ea00700 1 --2- 172.21.15.89:0/3502790314 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb2015d9d0 0x7fcb200d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.258+0000 7fcb1ea00700 1 -- 172.21.15.89:0/3502790314 >> v2:172.21.15.113:3300/0 conn(0x7fcb200d2540 msgr2=0x7fcb200ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.258+0000 7fcb1ea00700 1 --2- 172.21.15.89:0/3502790314 >> v2:172.21.15.113:3300/0 conn(0x7fcb200d2540 0x7fcb200ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.258+0000 7fcb1ea00700 1 -- 172.21.15.89:0/3502790314 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb201619e0 con 0x7fcb200d16a0 2024-04-02T07:37:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.258+0000 7fcb1ea00700 1 -- 172.21.15.89:0/3502790314 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fcb0c0017f0 con 0x7fcb200d16a0 2024-04-02T07:37:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.258+0000 7fcb1ea00700 1 -- 172.21.15.89:0/3502790314 >> v1:172.21.15.89:6789/0 conn(0x7fcb200d16a0 legacy=0x7fcb200d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb1ea00700 1 --2- 172.21.15.89:0/3502790314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb04003a00 0x7fcb04003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:29.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb1ea00700 1 --2- 172.21.15.89:0/3502790314 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb2015d9d0 0x7fcb04004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:29.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb1ea00700 1 --2- 172.21.15.89:0/3502790314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04004d90 0x7fcb04005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:29.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb1ea00700 1 -- 172.21.15.89:0/3502790314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb20161700 con 0x7fcb04004d90 2024-04-02T07:37:29.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb1ea00700 1 -- 172.21.15.89:0/3502790314 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0400ba10 con 0x7fcb04003a00 2024-04-02T07:37:29.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb1ea00700 1 -- 172.21.15.89:0/3502790314 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0400bb50 con 0x7fcb2015d9d0 2024-04-02T07:37:29.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb25000700 1 --2- 172.21.15.89:0/3502790314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04004d90 0x7fcb04005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:29.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb1f400700 1 --2- 172.21.15.89:0/3502790314 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb2015d9d0 0x7fcb04004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:29.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb1fe00700 1 --2- 172.21.15.89:0/3502790314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb04003a00 0x7fcb04003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:29.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb25000700 1 -- 172.21.15.89:0/3502790314 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb2015d9d0 msgr2=0x7fcb04004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:29.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb25000700 1 --2- 172.21.15.89:0/3502790314 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb2015d9d0 0x7fcb04004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb25000700 1 -- 172.21.15.89:0/3502790314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb04003a00 msgr2=0x7fcb04003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:29.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb25000700 1 --2- 172.21.15.89:0/3502790314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb04003a00 0x7fcb04003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb25000700 1 -- 172.21.15.89:0/3502790314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb040035b0 con 0x7fcb04004d90 2024-04-02T07:37:29.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.259+0000 7fcb25000700 1 --2- 172.21.15.89:0/3502790314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04004d90 0x7fcb04005210 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fcb2008ca30 tx=0x7fcb04005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b62d217bc13c327 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:29.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.260+0000 7fcb1ea00700 1 -- 172.21.15.89:0/3502790314 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1800ebd0 con 0x7fcb04004d90 2024-04-02T07:37:29.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.260+0000 7fcb1ea00700 1 -- 172.21.15.89:0/3502790314 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1800f040 con 0x7fcb04004d90 2024-04-02T07:37:29.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.260+0000 7fcb1ea00700 1 -- 172.21.15.89:0/3502790314 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb18014480 con 0x7fcb04004d90 2024-04-02T07:37:29.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.260+0000 7fcb27000700 1 -- 172.21.15.89:0/3502790314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04004d90 msgr2=0x7fcb04005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:29.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.260+0000 7fcb27000700 1 --2- 172.21.15.89:0/3502790314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04004d90 0x7fcb04005210 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fcb2008ca30 tx=0x7fcb04005880 comp rx=0 tx=0).stop 2024-04-02T07:37:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.260+0000 7fcb25000700 1 -- 172.21.15.89:0/3502790314 reap_dead start 2024-04-02T07:37:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.260+0000 7fcb27000700 1 -- 172.21.15.89:0/3502790314 shutdown_connections 2024-04-02T07:37:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.260+0000 7fcb27000700 1 -- 172.21.15.89:0/3502790314 >> 172.21.15.89:0/3502790314 conn(0x7fcb2008aff0 msgr2=0x7fcb200d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.260+0000 7fcb27000700 1 -- 172.21.15.89:0/3502790314 shutdown_connections 2024-04-02T07:37:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.260+0000 7fcb27000700 1 -- 172.21.15.89:0/3502790314 wait complete. 2024-04-02T07:37:29.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.263+0000 7fcb27000700 1 Processor -- start 2024-04-02T07:37:29.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.263+0000 7fcb27000700 1 -- start start 2024-04-02T07:37:29.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.264+0000 7fcb27000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04004d90 0x7fcb2015f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:29.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.264+0000 7fcb27000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb200cc920 0x7fcb20150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.264+0000 7fcb27000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb200ccda0 0x7fcb20150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.264+0000 7fcb27000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200b6020 con 0x7fcb04004d90 2024-04-02T07:37:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.264+0000 7fcb27000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2015f730 con 0x7fcb200ccda0 2024-04-02T07:37:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.264+0000 7fcb27000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200cbd70 con 0x7fcb200cc920 2024-04-02T07:37:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.264+0000 7fcb1fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04004d90 0x7fcb2015f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.264+0000 7fcb1fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04004d90 0x7fcb2015f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53750/0 (socket says 172.21.15.89:53750) 2024-04-02T07:37:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.264+0000 7fcb1fe00700 1 -- 172.21.15.89:0/2701490781 learned_addr learned my addr 172.21.15.89:0/2701490781 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.264+0000 7fcb1f400700 1 --2- 172.21.15.89:0/2701490781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb200cc920 0x7fcb20150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.265+0000 7fcb25000700 1 --2- 172.21.15.89:0/2701490781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb200ccda0 0x7fcb20150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.265+0000 7fcb1fe00700 1 -- 172.21.15.89:0/2701490781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb200cc920 msgr2=0x7fcb20150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.265+0000 7fcb1fe00700 1 --2- 172.21.15.89:0/2701490781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb200cc920 0x7fcb20150560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.265+0000 7fcb1fe00700 1 -- 172.21.15.89:0/2701490781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb200ccda0 msgr2=0x7fcb20150aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.265+0000 7fcb1fe00700 1 --2- 172.21.15.89:0/2701490781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb200ccda0 0x7fcb20150aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.265+0000 7fcb1fe00700 1 -- 172.21.15.89:0/2701490781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200b6270 con 0x7fcb04004d90 2024-04-02T07:37:29.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.265+0000 7fcb1fe00700 1 --2- 172.21.15.89:0/2701490781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04004d90 0x7fcb2015f1f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fcb100119d0 tx=0x7fcb10011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:29.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.265+0000 7fcb17e00700 1 -- 172.21.15.89:0/2701490781 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1000f940 con 0x7fcb04004d90 2024-04-02T07:37:29.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.265+0000 7fcb17e00700 1 -- 172.21.15.89:0/2701490781 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1000faa0 con 0x7fcb04004d90 2024-04-02T07:37:29.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.265+0000 7fcb17e00700 1 -- 172.21.15.89:0/2701490781 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1000fe30 con 0x7fcb04004d90 2024-04-02T07:37:29.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.265+0000 7fcb27000700 1 -- 172.21.15.89:0/2701490781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb200b6550 con 0x7fcb04004d90 2024-04-02T07:37:29.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.265+0000 7fcb27000700 1 -- 172.21.15.89:0/2701490781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb200cb000 con 0x7fcb04004d90 2024-04-02T07:37:29.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.266+0000 7fcb27000700 1 -- 172.21.15.89:0/2701490781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb200d1ca0 con 0x7fcb04004d90 2024-04-02T07:37:29.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.267+0000 7fcb17e00700 1 -- 172.21.15.89:0/2701490781 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fcb10014440 con 0x7fcb04004d90 2024-04-02T07:37:29.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.268+0000 7fcb17e00700 1 --2- 172.21.15.89:0/2701490781 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb0006ae90 0x7fcb0006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:29.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.268+0000 7fcb17e00700 1 -- 172.21.15.89:0/2701490781 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fcb10089f50 con 0x7fcb04004d90 2024-04-02T07:37:29.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.268+0000 7fcb1f400700 1 --2- 172.21.15.89:0/2701490781 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb0006ae90 0x7fcb0006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:29.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.269+0000 7fcb1f400700 1 --2- 172.21.15.89:0/2701490781 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb0006ae90 0x7fcb0006d350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcb20160f00 tx=0x7fcb0c009040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:29.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.273+0000 7fcb17e00700 1 -- 172.21.15.89:0/2701490781 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcb10054e70 con 0x7fcb04004d90 2024-04-02T07:37:29.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.507+0000 7fcb27000700 1 -- 172.21.15.89:0/2701490781 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb20161fa0 con 0x7fcb0006ae90 2024-04-02T07:37:29.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.573+0000 7fcb17e00700 1 -- 172.21.15.89:0/2701490781 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fcb0c0106a0 con 0x7fcb0006ae90 2024-04-02T07:37:29.575 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule activated for path /snap_test_dir1/dir0 2024-04-02T07:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.575+0000 7fcb27000700 1 -- 172.21.15.89:0/2701490781 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb0006ae90 msgr2=0x7fcb0006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.575+0000 7fcb27000700 1 --2- 172.21.15.89:0/2701490781 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb0006ae90 0x7fcb0006d350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcb20160f00 tx=0x7fcb0c009040 comp rx=0 tx=0).stop 2024-04-02T07:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.575+0000 7fcb27000700 1 -- 172.21.15.89:0/2701490781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04004d90 msgr2=0x7fcb2015f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.575+0000 7fcb27000700 1 --2- 172.21.15.89:0/2701490781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04004d90 0x7fcb2015f1f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fcb100119d0 tx=0x7fcb10011d90 comp rx=0 tx=0).stop 2024-04-02T07:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.575+0000 7fcb27000700 1 -- 172.21.15.89:0/2701490781 shutdown_connections 2024-04-02T07:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.575+0000 7fcb27000700 1 --2- 172.21.15.89:0/2701490781 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb0006ae90 0x7fcb0006d350 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.575+0000 7fcb27000700 1 --2- 172.21.15.89:0/2701490781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb200ccda0 0x7fcb20150aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.575+0000 7fcb27000700 1 --2- 172.21.15.89:0/2701490781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04004d90 0x7fcb2015f1f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.575+0000 7fcb27000700 1 --2- 172.21.15.89:0/2701490781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb200cc920 0x7fcb20150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.575+0000 7fcb27000700 1 -- 172.21.15.89:0/2701490781 >> 172.21.15.89:0/2701490781 conn(0x7fcb2008aff0 msgr2=0x7fcb200b53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.575+0000 7fcb27000700 1 -- 172.21.15.89:0/2701490781 shutdown_connections 2024-04-02T07:37:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.575+0000 7fcb27000700 1 -- 172.21.15.89:0/2701490781 wait complete. 2024-04-02T07:37:29.595 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2024-04-02T07:37:29.595 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2024-04-02T07:37:29.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.789+0000 7f3812000700 1 Processor -- start 2024-04-02T07:37:29.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.789+0000 7f3812000700 1 -- start start 2024-04-02T07:37:29.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.790+0000 7f3812000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f380c15d9d0 0x7f380c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:29.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.790+0000 7f3812000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c161420 con 0x7f380c0d2540 2024-04-02T07:37:29.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.790+0000 7f3812000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c161700 con 0x7f380c0d16a0 2024-04-02T07:37:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.790+0000 7f3812000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c15edc0 con 0x7f380c15d9d0 2024-04-02T07:37:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.791+0000 7f380be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f380c0d2540 0x7f380c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56510/0 (socket says 172.21.15.89:56510) 2024-04-02T07:37:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.791+0000 7f380be00700 1 -- 172.21.15.89:0/562460801 learned_addr learned my addr 172.21.15.89:0/562460801 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.791+0000 7f380b400700 1 --2- 172.21.15.89:0/562460801 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f380c15d9d0 0x7f380c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.791+0000 7f380a000700 1 -- 172.21.15.89:0/562460801 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960911892 0 0) 0x7f380c161420 con 0x7f380c0d2540 2024-04-02T07:37:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.791+0000 7f380a000700 1 -- 172.21.15.89:0/562460801 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f00035b0 con 0x7f380c0d2540 2024-04-02T07:37:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.791+0000 7f380a000700 1 -- 172.21.15.89:0/562460801 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915958442 0 0) 0x7f380c161700 con 0x7f380c0d16a0 2024-04-02T07:37:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.791+0000 7f380a000700 1 -- 172.21.15.89:0/562460801 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f380c161420 con 0x7f380c0d16a0 2024-04-02T07:37:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.791+0000 7f380b400700 1 -- 172.21.15.89:0/562460801 >> v1:172.21.15.113:6789/0 conn(0x7f380c0d16a0 legacy=0x7f380c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.791+0000 7f380b400700 1 -- 172.21.15.89:0/562460801 >> v1:172.21.15.89:6789/0 conn(0x7f380c0d2540 legacy=0x7f380c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.791+0000 7f380b400700 1 -- 172.21.15.89:0/562460801 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c1619e0 con 0x7f380c15d9d0 2024-04-02T07:37:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.792+0000 7f380b400700 1 --2- 172.21.15.89:0/562460801 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f380c15d9d0 0x7f380c0d3c30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0098a0 tx=0x7f37fc009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7373994186d8901a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.792+0000 7f380a000700 1 -- 172.21.15.89:0/562460801 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc011830 con 0x7f380c15d9d0 2024-04-02T07:37:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.792+0000 7f380a000700 1 -- 172.21.15.89:0/562460801 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37fc011990 con 0x7f380c15d9d0 2024-04-02T07:37:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.792+0000 7f380a000700 1 -- 172.21.15.89:0/562460801 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc011d20 con 0x7f380c15d9d0 2024-04-02T07:37:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.792+0000 7f3812000700 1 -- 172.21.15.89:0/562460801 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f380c15d9d0 msgr2=0x7f380c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.792+0000 7f3812000700 1 --2- 172.21.15.89:0/562460801 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f380c15d9d0 0x7f380c0d3c30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0098a0 tx=0x7f37fc009bb0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.792+0000 7f3812000700 1 -- 172.21.15.89:0/562460801 shutdown_connections 2024-04-02T07:37:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.792+0000 7f3812000700 1 --2- 172.21.15.89:0/562460801 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f380c15d9d0 0x7f380c0d3c30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.793+0000 7f3812000700 1 -- 172.21.15.89:0/562460801 >> 172.21.15.89:0/562460801 conn(0x7f380c08aff0 msgr2=0x7f380c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.793+0000 7f3812000700 1 -- 172.21.15.89:0/562460801 shutdown_connections 2024-04-02T07:37:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.793+0000 7f3812000700 1 -- 172.21.15.89:0/562460801 wait complete. 2024-04-02T07:37:29.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.794+0000 7f3812000700 1 Processor -- start 2024-04-02T07:37:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.794+0000 7f3812000700 1 -- start start 2024-04-02T07:37:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.795+0000 7f3812000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f380c0d16a0 0x7f380c15c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.795+0000 7f3812000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f380c0d2540 0x7f380c0b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.795+0000 7f3812000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f380c15d9d0 0x7f380c14f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.795+0000 7f3812000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c0b7410 con 0x7f380c0d16a0 2024-04-02T07:37:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.795+0000 7f3812000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c0b4610 con 0x7f380c15d9d0 2024-04-02T07:37:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.795+0000 7f3812000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c0b6560 con 0x7f380c0d2540 2024-04-02T07:37:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.795+0000 7f380b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f380c0d16a0 0x7f380c15c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.795+0000 7f380b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f380c0d16a0 0x7f380c15c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53752/0 (socket says 172.21.15.89:53752) 2024-04-02T07:37:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.795+0000 7f380b400700 1 -- 172.21.15.89:0/2112382732 learned_addr learned my addr 172.21.15.89:0/2112382732 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.796+0000 7f380b400700 1 -- 172.21.15.89:0/2112382732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f380c0d2540 msgr2=0x7f380c0b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:29.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.796+0000 7f380b400700 1 --2- 172.21.15.89:0/2112382732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f380c0d2540 0x7f380c0b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.796+0000 7f380b400700 1 -- 172.21.15.89:0/2112382732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f380c15d9d0 msgr2=0x7f380c14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:29.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.796+0000 7f380b400700 1 --2- 172.21.15.89:0/2112382732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f380c15d9d0 0x7f380c14f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:29.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.796+0000 7f380b400700 1 -- 172.21.15.89:0/2112382732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc0094a0 con 0x7f380c0d16a0 2024-04-02T07:37:29.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.796+0000 7f380b400700 1 --2- 172.21.15.89:0/2112382732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f380c0d16a0 0x7f380c15c8a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0090b0 tx=0x7f37fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.796+0000 7f3803400700 1 -- 172.21.15.89:0/2112382732 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc01a9b0 con 0x7f380c0d16a0 2024-04-02T07:37:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.796+0000 7f3803400700 1 -- 172.21.15.89:0/2112382732 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37fc01ab10 con 0x7f380c0d16a0 2024-04-02T07:37:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.796+0000 7f3803400700 1 -- 172.21.15.89:0/2112382732 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc01aea0 con 0x7f380c0d16a0 2024-04-02T07:37:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.796+0000 7f3812000700 1 -- 172.21.15.89:0/2112382732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380c0b6790 con 0x7f380c0d16a0 2024-04-02T07:37:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.797+0000 7f3812000700 1 -- 172.21.15.89:0/2112382732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f380c0b6d50 con 0x7f380c0d16a0 2024-04-02T07:37:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.797+0000 7f3812000700 1 -- 172.21.15.89:0/2112382732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380c0d1ca0 con 0x7f380c0d16a0 2024-04-02T07:37:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.798+0000 7f3803400700 1 -- 172.21.15.89:0/2112382732 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f37fc02d450 con 0x7f380c0d16a0 2024-04-02T07:37:29.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.799+0000 7f3803400700 1 --2- 172.21.15.89:0/2112382732 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f37ec06ae90 0x7f37ec06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:29.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.799+0000 7f3803400700 1 -- 172.21.15.89:0/2112382732 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f37fc08ac00 con 0x7f380c0d16a0 2024-04-02T07:37:29.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.799+0000 7f380aa00700 1 --2- 172.21.15.89:0/2112382732 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f37ec06ae90 0x7f37ec06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:29.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.800+0000 7f380aa00700 1 --2- 172.21.15.89:0/2112382732 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f37ec06ae90 0x7f37ec06d350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f37f8000fb0 tx=0x7f37f80079d0 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:29.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:29.804+0000 7f3803400700 1 -- 172.21.15.89:0/2112382732 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f37fc055c10 con 0x7f380c0d16a0 2024-04-02T07:37:30.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.035+0000 7f3812000700 1 -- 172.21.15.89:0/2112382732 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f380c161fa0 con 0x7f37ec06ae90 2024-04-02T07:37:30.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.098+0000 7f3803400700 1 -- 172.21.15.89:0/2112382732 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f37f80013c0 con 0x7f37ec06ae90 2024-04-02T07:37:30.100 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule activated for path /snap_test_dir1/dir1 2024-04-02T07:37:30.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.100+0000 7f3812000700 1 -- 172.21.15.89:0/2112382732 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f37ec06ae90 msgr2=0x7f37ec06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:30.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.100+0000 7f3812000700 1 --2- 172.21.15.89:0/2112382732 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f37ec06ae90 0x7f37ec06d350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f37f8000fb0 tx=0x7f37f80079d0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.100+0000 7f3812000700 1 -- 172.21.15.89:0/2112382732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f380c0d16a0 msgr2=0x7f380c15c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:30.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.100+0000 7f3812000700 1 --2- 172.21.15.89:0/2112382732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f380c0d16a0 0x7f380c15c8a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0090b0 tx=0x7f37fc004750 comp rx=0 tx=0).stop 2024-04-02T07:37:30.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.100+0000 7f3812000700 1 -- 172.21.15.89:0/2112382732 shutdown_connections 2024-04-02T07:37:30.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.100+0000 7f3812000700 1 --2- 172.21.15.89:0/2112382732 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f37ec06ae90 0x7f37ec06d350 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.100+0000 7f3812000700 1 --2- 172.21.15.89:0/2112382732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f380c15d9d0 0x7f380c14f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.100+0000 7f3812000700 1 --2- 172.21.15.89:0/2112382732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f380c0d16a0 0x7f380c15c8a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.100+0000 7f3812000700 1 --2- 172.21.15.89:0/2112382732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f380c0d2540 0x7f380c0b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.100+0000 7f3812000700 1 -- 172.21.15.89:0/2112382732 >> 172.21.15.89:0/2112382732 conn(0x7f380c08aff0 msgr2=0x7f380c0b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:30.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.100+0000 7f3812000700 1 -- 172.21.15.89:0/2112382732 shutdown_connections 2024-04-02T07:37:30.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.100+0000 7f3812000700 1 -- 172.21.15.89:0/2112382732 wait complete. 2024-04-02T07:37:30.120 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2024-04-02T07:37:30.120 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2024-04-02T07:37:30.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.314+0000 7fea8fa00700 1 Processor -- start 2024-04-02T07:37:30.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.315+0000 7fea8fa00700 1 -- start start 2024-04-02T07:37:30.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.316+0000 7fea8fa00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fea880d16a0 0x7fea880d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:30.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.316+0000 7fea8fa00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea88161420 con 0x7fea8815d9d0 2024-04-02T07:37:30.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.316+0000 7fea8fa00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea88161700 con 0x7fea880d2370 2024-04-02T07:37:30.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.316+0000 7fea8fa00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea880d4340 con 0x7fea880d16a0 2024-04-02T07:37:30.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.316+0000 7fea8d000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fea8815d9d0 0x7fea880d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56514/0 (socket says 172.21.15.89:56514) 2024-04-02T07:37:30.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.316+0000 7fea8d000700 1 -- 172.21.15.89:0/3670983984 learned_addr learned my addr 172.21.15.89:0/3670983984 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:30.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.316+0000 7fea87e00700 1 --2- 172.21.15.89:0/3670983984 >> v2:172.21.15.113:3300/0 conn(0x7fea880d16a0 0x7fea880d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:30.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.316+0000 7fea87400700 1 -- 172.21.15.89:0/3670983984 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741019859 0 0) 0x7fea88161420 con 0x7fea8815d9d0 2024-04-02T07:37:30.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.316+0000 7fea87400700 1 -- 172.21.15.89:0/3670983984 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea700035b0 con 0x7fea8815d9d0 2024-04-02T07:37:30.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87400700 1 -- 172.21.15.89:0/3670983984 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1231349504 0 0) 0x7fea700035b0 con 0x7fea8815d9d0 2024-04-02T07:37:30.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87400700 1 -- 172.21.15.89:0/3670983984 >> v2:172.21.15.113:3300/0 conn(0x7fea880d16a0 msgr2=0x7fea880d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:30.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87400700 1 --2- 172.21.15.89:0/3670983984 >> v2:172.21.15.113:3300/0 conn(0x7fea880d16a0 0x7fea880d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87400700 1 -- 172.21.15.89:0/3670983984 >> v1:172.21.15.113:6789/0 conn(0x7fea880d2370 legacy=0x7fea880ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:30.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87400700 1 -- 172.21.15.89:0/3670983984 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea881619e0 con 0x7fea8815d9d0 2024-04-02T07:37:30.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87400700 1 -- 172.21.15.89:0/3670983984 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fea7c001670 con 0x7fea8815d9d0 2024-04-02T07:37:30.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87400700 1 -- 172.21.15.89:0/3670983984 >> v1:172.21.15.89:6789/0 conn(0x7fea8815d9d0 legacy=0x7fea880d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:30.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87400700 1 --2- 172.21.15.89:0/3670983984 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fea70003a50 0x7fea70003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:30.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87400700 1 --2- 172.21.15.89:0/3670983984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea70004850 0x7fea70004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:30.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87400700 1 --2- 172.21.15.89:0/3670983984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea70005210 0x7fea7000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:30.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87400700 1 -- 172.21.15.89:0/3670983984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7000c0e0 con 0x7fea70004850 2024-04-02T07:37:30.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87400700 1 -- 172.21.15.89:0/3670983984 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7000c250 con 0x7fea70005210 2024-04-02T07:37:30.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87400700 1 -- 172.21.15.89:0/3670983984 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea7000c410 con 0x7fea70003a50 2024-04-02T07:37:30.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87e00700 1 --2- 172.21.15.89:0/3670983984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea70004850 0x7fea70004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:30.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea8d000700 1 --2- 172.21.15.89:0/3670983984 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fea70003a50 0x7fea70003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:30.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea8da00700 1 --2- 172.21.15.89:0/3670983984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea70005210 0x7fea7000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:30.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.317+0000 7fea87e00700 1 -- 172.21.15.89:0/3670983984 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fea70003a50 msgr2=0x7fea70003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:30.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.318+0000 7fea87e00700 1 --2- 172.21.15.89:0/3670983984 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fea70003a50 0x7fea70003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.318+0000 7fea87e00700 1 -- 172.21.15.89:0/3670983984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea70005210 msgr2=0x7fea7000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.318+0000 7fea87e00700 1 --2- 172.21.15.89:0/3670983984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea70005210 0x7fea7000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.318+0000 7fea87e00700 1 -- 172.21.15.89:0/3670983984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea700035b0 con 0x7fea70004850 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.318+0000 7fea8da00700 1 --2- 172.21.15.89:0/3670983984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea70005210 0x7fea7000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.318+0000 7fea8da00700 1 -- 172.21.15.89:0/3670983984 reap_dead start 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.318+0000 7fea87e00700 1 --2- 172.21.15.89:0/3670983984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea70004850 0x7fea70004cd0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fea74009b10 tx=0x7fea70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdfd9627af26afa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.318+0000 7fea87400700 1 -- 172.21.15.89:0/3670983984 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea74005a20 con 0x7fea70004850 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.318+0000 7fea87400700 1 -- 172.21.15.89:0/3670983984 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea74005b80 con 0x7fea70004850 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.318+0000 7fea87400700 1 -- 172.21.15.89:0/3670983984 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7400e6c0 con 0x7fea70004850 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.319+0000 7fea8fa00700 1 -- 172.21.15.89:0/3670983984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea70004850 msgr2=0x7fea70004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.319+0000 7fea8fa00700 1 --2- 172.21.15.89:0/3670983984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea70004850 0x7fea70004cd0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fea74009b10 tx=0x7fea70005880 comp rx=0 tx=0).stop 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.319+0000 7fea8fa00700 1 -- 172.21.15.89:0/3670983984 shutdown_connections 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.319+0000 7fea8fa00700 1 --2- 172.21.15.89:0/3670983984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea70004850 0x7fea70004cd0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.319+0000 7fea8fa00700 1 -- 172.21.15.89:0/3670983984 >> 172.21.15.89:0/3670983984 conn(0x7fea8808aff0 msgr2=0x7fea880d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.319+0000 7fea8fa00700 1 -- 172.21.15.89:0/3670983984 shutdown_connections 2024-04-02T07:37:30.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.319+0000 7fea8fa00700 1 -- 172.21.15.89:0/3670983984 wait complete. 2024-04-02T07:37:30.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.321+0000 7fea8fa00700 1 Processor -- start 2024-04-02T07:37:30.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.321+0000 7fea8fa00700 1 -- start start 2024-04-02T07:37:30.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8fa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea70004850 0x7fea88158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:30.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8fa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fea880b4590 0x7fea880b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:30.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8fa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea880b74d0 0x7fea880b7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:30.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8fa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea880b8d30 con 0x7fea880b74d0 2024-04-02T07:37:30.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8fa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea880b4f50 con 0x7fea70004850 2024-04-02T07:37:30.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8fa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea880b7ea0 con 0x7fea880b4590 2024-04-02T07:37:30.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea880b74d0 0x7fea880b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:30.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea880b74d0 0x7fea880b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53770/0 (socket says 172.21.15.89:53770) 2024-04-02T07:37:30.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8da00700 1 -- 172.21.15.89:0/1026198514 learned_addr learned my addr 172.21.15.89:0/1026198514 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:30.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8d000700 1 --2- 172.21.15.89:0/1026198514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea70004850 0x7fea88158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:30.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea87e00700 1 --2- 172.21.15.89:0/1026198514 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fea880b4590 0x7fea880b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:30.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8da00700 1 -- 172.21.15.89:0/1026198514 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fea880b4590 msgr2=0x7fea880b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:30.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8da00700 1 --2- 172.21.15.89:0/1026198514 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fea880b4590 0x7fea880b4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8da00700 1 -- 172.21.15.89:0/1026198514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea70004850 msgr2=0x7fea88158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:30.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8da00700 1 --2- 172.21.15.89:0/1026198514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea70004850 0x7fea88158b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.322+0000 7fea8da00700 1 -- 172.21.15.89:0/1026198514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea88159050 con 0x7fea880b74d0 2024-04-02T07:37:30.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.323+0000 7fea8da00700 1 --2- 172.21.15.89:0/1026198514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea880b74d0 0x7fea880b7960 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fea8806b5b0 tx=0x7fea8000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:30.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.323+0000 7fea84c00700 1 -- 172.21.15.89:0/1026198514 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80003c10 con 0x7fea880b74d0 2024-04-02T07:37:30.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.323+0000 7fea84c00700 1 -- 172.21.15.89:0/1026198514 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea80003d90 con 0x7fea880b74d0 2024-04-02T07:37:30.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.323+0000 7fea84c00700 1 -- 172.21.15.89:0/1026198514 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80005310 con 0x7fea880b74d0 2024-04-02T07:37:30.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.323+0000 7fea8fa00700 1 -- 172.21.15.89:0/1026198514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea88160e70 con 0x7fea880b74d0 2024-04-02T07:37:30.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.323+0000 7fea8fa00700 1 -- 172.21.15.89:0/1026198514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea880b8390 con 0x7fea880b74d0 2024-04-02T07:37:30.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.324+0000 7fea8fa00700 1 -- 172.21.15.89:0/1026198514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea88067720 con 0x7fea880b74d0 2024-04-02T07:37:30.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.325+0000 7fea84c00700 1 -- 172.21.15.89:0/1026198514 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fea80005470 con 0x7fea880b74d0 2024-04-02T07:37:30.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.326+0000 7fea84c00700 1 --2- 172.21.15.89:0/1026198514 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fea6806ae90 0x7fea6806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:30.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.326+0000 7fea84c00700 1 -- 172.21.15.89:0/1026198514 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fea80089720 con 0x7fea880b74d0 2024-04-02T07:37:30.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.326+0000 7fea8d000700 1 --2- 172.21.15.89:0/1026198514 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fea6806ae90 0x7fea6806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:30.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.326+0000 7fea8d000700 1 --2- 172.21.15.89:0/1026198514 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fea6806ae90 0x7fea6806d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fea7c001050 tx=0x7fea7c009040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:30.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.330+0000 7fea84c00700 1 -- 172.21.15.89:0/1026198514 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fea80054640 con 0x7fea880b74d0 2024-04-02T07:37:30.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.563+0000 7fea8fa00700 1 -- 172.21.15.89:0/1026198514 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fea88161fa0 con 0x7fea6806ae90 2024-04-02T07:37:30.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.628+0000 7fea84c00700 1 -- 172.21.15.89:0/1026198514 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fea7c005e60 con 0x7fea6806ae90 2024-04-02T07:37:30.630 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule activated for path /snap_test_dir1/dir2 2024-04-02T07:37:30.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.630+0000 7fea8fa00700 1 -- 172.21.15.89:0/1026198514 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fea6806ae90 msgr2=0x7fea6806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:30.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.630+0000 7fea8fa00700 1 --2- 172.21.15.89:0/1026198514 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fea6806ae90 0x7fea6806d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fea7c001050 tx=0x7fea7c009040 comp rx=0 tx=0).stop 2024-04-02T07:37:30.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.631+0000 7fea8fa00700 1 -- 172.21.15.89:0/1026198514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea880b74d0 msgr2=0x7fea880b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:30.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.631+0000 7fea8fa00700 1 --2- 172.21.15.89:0/1026198514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea880b74d0 0x7fea880b7960 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fea8806b5b0 tx=0x7fea8000ee00 comp rx=0 tx=0).stop 2024-04-02T07:37:30.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.631+0000 7fea8fa00700 1 -- 172.21.15.89:0/1026198514 shutdown_connections 2024-04-02T07:37:30.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.631+0000 7fea8fa00700 1 --2- 172.21.15.89:0/1026198514 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fea6806ae90 0x7fea6806d350 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.631+0000 7fea8fa00700 1 --2- 172.21.15.89:0/1026198514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea70004850 0x7fea88158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.631+0000 7fea8fa00700 1 --2- 172.21.15.89:0/1026198514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea880b74d0 0x7fea880b7960 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.631+0000 7fea8fa00700 1 --2- 172.21.15.89:0/1026198514 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fea880b4590 0x7fea880b4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.631+0000 7fea8fa00700 1 -- 172.21.15.89:0/1026198514 >> 172.21.15.89:0/1026198514 conn(0x7fea8808aff0 msgr2=0x7fea880b1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:30.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.631+0000 7fea8fa00700 1 -- 172.21.15.89:0/1026198514 shutdown_connections 2024-04-02T07:37:30.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.631+0000 7fea8fa00700 1 -- 172.21.15.89:0/1026198514 wait complete. 2024-04-02T07:37:30.651 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2024-04-02T07:37:30.651 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2024-04-02T07:37:30.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.847+0000 7fb0e6800700 1 Processor -- start 2024-04-02T07:37:30.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.847+0000 7fb0e6800700 1 -- start start 2024-04-02T07:37:30.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.848+0000 7fb0e6800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb0e00cd020 0x7fb0e00cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:30.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.848+0000 7fb0e6800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0e00b2070 0x7fb0e00b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:30.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.848+0000 7fb0e6800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e00cdae0 con 0x7fb0e00cbb30 2024-04-02T07:37:30.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.848+0000 7fb0e6800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e00cddc0 con 0x7fb0e00cd020 2024-04-02T07:37:30.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.848+0000 7fb0e6800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e00ce0a0 con 0x7fb0e00b2070 2024-04-02T07:37:30.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.848+0000 7fb0dea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb0e00cd020 0x7fb0e00cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.848+0000 7fb0df400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb0e00cbb30 0x7fb0e00cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56516/0 (socket says 172.21.15.89:56516) 2024-04-02T07:37:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.849+0000 7fb0df400700 1 -- 172.21.15.89:0/3312903162 learned_addr learned my addr 172.21.15.89:0/3312903162 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.849+0000 7fb0dfe00700 1 --2- 172.21.15.89:0/3312903162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0e00b2070 0x7fb0e00b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.849+0000 7fb0dea00700 1 -- 172.21.15.89:0/3312903162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0e00b2070 msgr2=0x7fb0e00b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.849+0000 7fb0dea00700 1 --2- 172.21.15.89:0/3312903162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0e00b2070 0x7fb0e00b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.849+0000 7fb0dea00700 1 -- 172.21.15.89:0/3312903162 >> v1:172.21.15.89:6789/0 conn(0x7fb0e00cbb30 legacy=0x7fb0e00cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.849+0000 7fb0dea00700 1 -- 172.21.15.89:0/3312903162 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e006dc20 con 0x7fb0e00cd020 2024-04-02T07:37:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.849+0000 7fb0de000700 1 -- 172.21.15.89:0/3312903162 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755246235 0 0) 0x7fb0e00cdae0 con 0x7fb0e00cbb30 2024-04-02T07:37:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.849+0000 7fb0dea00700 1 --2- 172.21.15.89:0/3312903162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e00cd020 0x7fb0e00cd480 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009840 tx=0x7fb0cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e74cfc25438bca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.849+0000 7fb0de000700 1 -- 172.21.15.89:0/3312903162 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc011830 con 0x7fb0e00cd020 2024-04-02T07:37:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.849+0000 7fb0de000700 1 -- 172.21.15.89:0/3312903162 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0cc011990 con 0x7fb0e00cd020 2024-04-02T07:37:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.850+0000 7fb0de000700 1 -- 172.21.15.89:0/3312903162 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc011d20 con 0x7fb0e00cd020 2024-04-02T07:37:30.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.850+0000 7fb0e6800700 1 -- 172.21.15.89:0/3312903162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e00cd020 msgr2=0x7fb0e00cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:30.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.850+0000 7fb0e6800700 1 --2- 172.21.15.89:0/3312903162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e00cd020 0x7fb0e00cd480 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009840 tx=0x7fb0cc009b50 comp rx=0 tx=0).stop 2024-04-02T07:37:30.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.850+0000 7fb0e6800700 1 -- 172.21.15.89:0/3312903162 shutdown_connections 2024-04-02T07:37:30.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.850+0000 7fb0e6800700 1 --2- 172.21.15.89:0/3312903162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0e00b2070 0x7fb0e00b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.850+0000 7fb0e6800700 1 --2- 172.21.15.89:0/3312903162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e00cd020 0x7fb0e00cd480 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.850+0000 7fb0e6800700 1 -- 172.21.15.89:0/3312903162 >> 172.21.15.89:0/3312903162 conn(0x7fb0e00b0d70 msgr2=0x7fb0e00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:30.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.850+0000 7fb0e6800700 1 -- 172.21.15.89:0/3312903162 shutdown_connections 2024-04-02T07:37:30.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.850+0000 7fb0e6800700 1 -- 172.21.15.89:0/3312903162 wait complete. 2024-04-02T07:37:30.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.852+0000 7fb0e6800700 1 Processor -- start 2024-04-02T07:37:30.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.852+0000 7fb0e6800700 1 -- start start 2024-04-02T07:37:30.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.853+0000 7fb0e6800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e00b2070 0x7fb0e014ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:30.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.853+0000 7fb0e6800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0e00cbb30 0x7fb0e00cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:30.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.853+0000 7fb0e6800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00cd020 0x7fb0e015df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:30.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.853+0000 7fb0e6800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e01f4100 con 0x7fb0e00b2070 2024-04-02T07:37:30.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.853+0000 7fb0e6800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e01f4520 con 0x7fb0e00cd020 2024-04-02T07:37:30.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.853+0000 7fb0e6800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e01f3760 con 0x7fb0e00cbb30 2024-04-02T07:37:30.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.853+0000 7fb0df400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e00b2070 0x7fb0e014ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.853+0000 7fb0df400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e00b2070 0x7fb0e014ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53798/0 (socket says 172.21.15.89:53798) 2024-04-02T07:37:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.853+0000 7fb0df400700 1 -- 172.21.15.89:0/3891875628 learned_addr learned my addr 172.21.15.89:0/3891875628 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.853+0000 7fb0dea00700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0e00cbb30 0x7fb0e00cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.854+0000 7fb0dfe00700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00cd020 0x7fb0e015df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.854+0000 7fb0df400700 1 -- 172.21.15.89:0/3891875628 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0e00cbb30 msgr2=0x7fb0e00cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.854+0000 7fb0df400700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0e00cbb30 0x7fb0e00cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.854+0000 7fb0df400700 1 -- 172.21.15.89:0/3891875628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00cd020 msgr2=0x7fb0e015df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.854+0000 7fb0df400700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00cd020 0x7fb0e015df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.854+0000 7fb0df400700 1 -- 172.21.15.89:0/3891875628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc0094a0 con 0x7fb0e00b2070 2024-04-02T07:37:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.854+0000 7fb0dfe00700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00cd020 0x7fb0e015df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.854+0000 7fb0df400700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e00b2070 0x7fb0e014ec90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00011e0 tx=0x7fb0d000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.854+0000 7fb0d7400700 1 -- 172.21.15.89:0/3891875628 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0004c80 con 0x7fb0e00b2070 2024-04-02T07:37:30.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.854+0000 7fb0d7400700 1 -- 172.21.15.89:0/3891875628 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d0004de0 con 0x7fb0e00b2070 2024-04-02T07:37:30.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.854+0000 7fb0d7400700 1 -- 172.21.15.89:0/3891875628 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0007310 con 0x7fb0e00b2070 2024-04-02T07:37:30.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.854+0000 7fb0e6800700 1 -- 172.21.15.89:0/3891875628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e01f3a40 con 0x7fb0e00b2070 2024-04-02T07:37:30.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.854+0000 7fb0e6800700 1 -- 172.21.15.89:0/3891875628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e014f1d0 con 0x7fb0e00b2070 2024-04-02T07:37:30.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.855+0000 7fb0e6800700 1 -- 172.21.15.89:0/3891875628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e01f4240 con 0x7fb0e00b2070 2024-04-02T07:37:30.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.856+0000 7fb0d7400700 1 -- 172.21.15.89:0/3891875628 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fb0d0007470 con 0x7fb0e00b2070 2024-04-02T07:37:30.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.857+0000 7fb0d7400700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb0c006ae90 0x7fb0c006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:30.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.857+0000 7fb0d7400700 1 -- 172.21.15.89:0/3891875628 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb0d00895a0 con 0x7fb0e00b2070 2024-04-02T07:37:30.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.857+0000 7fb0dea00700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb0c006ae90 0x7fb0c006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:30.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.858+0000 7fb0dea00700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb0c006ae90 0x7fb0c006d350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0090b0 tx=0x7fb0cc008030 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:30.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:30.861+0000 7fb0d7400700 1 -- 172.21.15.89:0/3891875628 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb0e01f4240 con 0x7fb0e00b2070 2024-04-02T07:37:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.090+0000 7fb0e6800700 1 -- 172.21.15.89:0/3891875628 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0e015bdd0 con 0x7fb0c006ae90 2024-04-02T07:37:31.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.161+0000 7fb0d7400700 1 -- 172.21.15.89:0/3891875628 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb0cc011c00 con 0x7fb0c006ae90 2024-04-02T07:37:31.164 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule activated for path /snap_test_dir1/dir3 2024-04-02T07:37:31.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.164+0000 7fb0e6800700 1 -- 172.21.15.89:0/3891875628 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb0c006ae90 msgr2=0x7fb0c006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:31.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.164+0000 7fb0e6800700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb0c006ae90 0x7fb0c006d350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0090b0 tx=0x7fb0cc008030 comp rx=0 tx=0).stop 2024-04-02T07:37:31.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.164+0000 7fb0e6800700 1 -- 172.21.15.89:0/3891875628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e00b2070 msgr2=0x7fb0e014ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:31.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.164+0000 7fb0e6800700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e00b2070 0x7fb0e014ec90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00011e0 tx=0x7fb0d000ca30 comp rx=0 tx=0).stop 2024-04-02T07:37:31.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.164+0000 7fb0e6800700 1 -- 172.21.15.89:0/3891875628 shutdown_connections 2024-04-02T07:37:31.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.164+0000 7fb0e6800700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb0c006ae90 0x7fb0c006d350 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.164+0000 7fb0e6800700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00cd020 0x7fb0e015df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.164+0000 7fb0e6800700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e00b2070 0x7fb0e014ec90 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.164+0000 7fb0e6800700 1 --2- 172.21.15.89:0/3891875628 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0e00cbb30 0x7fb0e00cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.164+0000 7fb0e6800700 1 -- 172.21.15.89:0/3891875628 >> 172.21.15.89:0/3891875628 conn(0x7fb0e00b0d70 msgr2=0x7fb0e00cbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:31.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.164+0000 7fb0e6800700 1 -- 172.21.15.89:0/3891875628 shutdown_connections 2024-04-02T07:37:31.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.164+0000 7fb0e6800700 1 -- 172.21.15.89:0/3891875628 wait complete. 2024-04-02T07:37:31.183 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2024-04-02T07:37:31.183 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2024-04-02T07:37:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.375+0000 7fdd29e00700 1 Processor -- start 2024-04-02T07:37:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.376+0000 7fdd29e00700 1 -- start start 2024-04-02T07:37:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.377+0000 7fdd29e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdd240d21f0 0x7fdd240ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.377+0000 7fdd29e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd240cec40 0x7fdd24161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.377+0000 7fdd29e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd240ce270 con 0x7fdd241611b0 2024-04-02T07:37:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.377+0000 7fdd29e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd2415b600 con 0x7fdd240d21f0 2024-04-02T07:37:31.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.377+0000 7fdd29e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd240d2760 con 0x7fdd240cec40 2024-04-02T07:37:31.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.377+0000 7fdd22000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdd240d21f0 0x7fdd240ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:31.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.377+0000 7fdd22a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdd241611b0 0x7fdd2415ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56530/0 (socket says 172.21.15.89:56530) 2024-04-02T07:37:31.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.377+0000 7fdd22a00700 1 -- 172.21.15.89:0/1377893011 learned_addr learned my addr 172.21.15.89:0/1377893011 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:31.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.377+0000 7fdd23400700 1 --2- 172.21.15.89:0/1377893011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd240cec40 0x7fdd24161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:31.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.377+0000 7fdd22000700 1 -- 172.21.15.89:0/1377893011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd240cec40 msgr2=0x7fdd24161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:31.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.377+0000 7fdd22000700 1 --2- 172.21.15.89:0/1377893011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd240cec40 0x7fdd24161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.377+0000 7fdd22000700 1 -- 172.21.15.89:0/1377893011 >> v1:172.21.15.89:6789/0 conn(0x7fdd241611b0 legacy=0x7fdd2415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:31.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.377+0000 7fdd22000700 1 -- 172.21.15.89:0/1377893011 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd240d2a40 con 0x7fdd240d21f0 2024-04-02T07:37:31.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.377+0000 7fdd21600700 1 -- 172.21.15.89:0/1377893011 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727833993 0 0) 0x7fdd240ce270 con 0x7fdd241611b0 2024-04-02T07:37:31.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.378+0000 7fdd22000700 1 --2- 172.21.15.89:0/1377893011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240d21f0 0x7fdd240ce700 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009840 tx=0x7fdd10009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc9bd8f84550e63 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:31.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.378+0000 7fdd21600700 1 -- 172.21.15.89:0/1377893011 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd10011830 con 0x7fdd240d21f0 2024-04-02T07:37:31.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.378+0000 7fdd21600700 1 -- 172.21.15.89:0/1377893011 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd10011990 con 0x7fdd240d21f0 2024-04-02T07:37:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.378+0000 7fdd21600700 1 -- 172.21.15.89:0/1377893011 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd10011d20 con 0x7fdd240d21f0 2024-04-02T07:37:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.378+0000 7fdd29e00700 1 -- 172.21.15.89:0/1377893011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240d21f0 msgr2=0x7fdd240ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.378+0000 7fdd29e00700 1 --2- 172.21.15.89:0/1377893011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240d21f0 0x7fdd240ce700 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009840 tx=0x7fdd10009b50 comp rx=0 tx=0).stop 2024-04-02T07:37:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.379+0000 7fdd29e00700 1 -- 172.21.15.89:0/1377893011 shutdown_connections 2024-04-02T07:37:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.379+0000 7fdd29e00700 1 --2- 172.21.15.89:0/1377893011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd240cec40 0x7fdd24161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.379+0000 7fdd29e00700 1 --2- 172.21.15.89:0/1377893011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd240d21f0 0x7fdd240ce700 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.379+0000 7fdd29e00700 1 -- 172.21.15.89:0/1377893011 >> 172.21.15.89:0/1377893011 conn(0x7fdd2408aff0 msgr2=0x7fdd240d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.379+0000 7fdd29e00700 1 -- 172.21.15.89:0/1377893011 shutdown_connections 2024-04-02T07:37:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.379+0000 7fdd29e00700 1 -- 172.21.15.89:0/1377893011 wait complete. 2024-04-02T07:37:31.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.381+0000 7fdd29e00700 1 Processor -- start 2024-04-02T07:37:31.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.381+0000 7fdd29e00700 1 -- start start 2024-04-02T07:37:31.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.382+0000 7fdd29e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd240cec40 0x7fdd240b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:31.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.382+0000 7fdd29e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd240d21f0 0x7fdd241f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:31.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.382+0000 7fdd29e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd241611b0 0x7fdd24158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.382+0000 7fdd29e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd240cd6a0 con 0x7fdd241611b0 2024-04-02T07:37:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.382+0000 7fdd29e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd240b70a0 con 0x7fdd240cec40 2024-04-02T07:37:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.382+0000 7fdd29e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd240ca600 con 0x7fdd240d21f0 2024-04-02T07:37:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.383+0000 7fdd23400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd241611b0 0x7fdd24158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.383+0000 7fdd23400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd241611b0 0x7fdd24158930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53812/0 (socket says 172.21.15.89:53812) 2024-04-02T07:37:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.383+0000 7fdd23400700 1 -- 172.21.15.89:0/690854049 learned_addr learned my addr 172.21.15.89:0/690854049 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.383+0000 7fdd22a00700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd240cec40 0x7fdd240b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.383+0000 7fdd22000700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd240d21f0 0x7fdd241f34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.383+0000 7fdd23400700 1 -- 172.21.15.89:0/690854049 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd240d21f0 msgr2=0x7fdd241f34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.383+0000 7fdd23400700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd240d21f0 0x7fdd241f34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.383+0000 7fdd23400700 1 -- 172.21.15.89:0/690854049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd240cec40 msgr2=0x7fdd240b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.383+0000 7fdd23400700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd240cec40 0x7fdd240b6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.383+0000 7fdd23400700 1 -- 172.21.15.89:0/690854049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd100094a0 con 0x7fdd241611b0 2024-04-02T07:37:31.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.383+0000 7fdd22a00700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd240cec40 0x7fdd240b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:37:31.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.383+0000 7fdd23400700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd241611b0 0x7fdd24158930 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c007e40 tx=0x7fdd1c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:31.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.384+0000 7fdd1b400700 1 -- 172.21.15.89:0/690854049 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c00eea0 con 0x7fdd241611b0 2024-04-02T07:37:31.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.384+0000 7fdd1b400700 1 -- 172.21.15.89:0/690854049 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1c014420 con 0x7fdd241611b0 2024-04-02T07:37:31.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.384+0000 7fdd29e00700 1 -- 172.21.15.89:0/690854049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd240ca850 con 0x7fdd241611b0 2024-04-02T07:37:31.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.384+0000 7fdd1b400700 1 -- 172.21.15.89:0/690854049 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c0147b0 con 0x7fdd241611b0 2024-04-02T07:37:31.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.384+0000 7fdd29e00700 1 -- 172.21.15.89:0/690854049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd240cae10 con 0x7fdd241611b0 2024-04-02T07:37:31.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.385+0000 7fdd29e00700 1 -- 172.21.15.89:0/690854049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd240d30b0 con 0x7fdd241611b0 2024-04-02T07:37:31.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.386+0000 7fdd1b400700 1 -- 172.21.15.89:0/690854049 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fdd1c014910 con 0x7fdd241611b0 2024-04-02T07:37:31.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.387+0000 7fdd1b400700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd0406ae90 0x7fdd0406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:31.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.387+0000 7fdd1b400700 1 -- 172.21.15.89:0/690854049 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fdd1c089f40 con 0x7fdd241611b0 2024-04-02T07:37:31.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.387+0000 7fdd22a00700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd0406ae90 0x7fdd0406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:31.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.387+0000 7fdd22a00700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd0406ae90 0x7fdd0406d350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdd14007fd0 tx=0x7fdd14009040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:31.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.391+0000 7fdd1b400700 1 -- 172.21.15.89:0/690854049 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdd1c054e30 con 0x7fdd241611b0 2024-04-02T07:37:31.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.619+0000 7fdd29e00700 1 -- 172.21.15.89:0/690854049 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd2415f6a0 con 0x7fdd0406ae90 2024-04-02T07:37:31.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.687+0000 7fdd1b400700 1 -- 172.21.15.89:0/690854049 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fdd14001370 con 0x7fdd0406ae90 2024-04-02T07:37:31.689 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule activated for path /snap_test_dir1/dir4 2024-04-02T07:37:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.690+0000 7fdd29e00700 1 -- 172.21.15.89:0/690854049 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd0406ae90 msgr2=0x7fdd0406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.690+0000 7fdd29e00700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd0406ae90 0x7fdd0406d350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdd14007fd0 tx=0x7fdd14009040 comp rx=0 tx=0).stop 2024-04-02T07:37:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.690+0000 7fdd29e00700 1 -- 172.21.15.89:0/690854049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd241611b0 msgr2=0x7fdd24158930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.690+0000 7fdd29e00700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd241611b0 0x7fdd24158930 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c007e40 tx=0x7fdd1c00e3c0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.690+0000 7fdd29e00700 1 -- 172.21.15.89:0/690854049 shutdown_connections 2024-04-02T07:37:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.690+0000 7fdd29e00700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd0406ae90 0x7fdd0406d350 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.690+0000 7fdd29e00700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd240cec40 0x7fdd240b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.690+0000 7fdd29e00700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd241611b0 0x7fdd24158930 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.690+0000 7fdd29e00700 1 --2- 172.21.15.89:0/690854049 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd240d21f0 0x7fdd241f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.690+0000 7fdd29e00700 1 -- 172.21.15.89:0/690854049 >> 172.21.15.89:0/690854049 conn(0x7fdd2408aff0 msgr2=0x7fdd24159ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.690+0000 7fdd29e00700 1 -- 172.21.15.89:0/690854049 shutdown_connections 2024-04-02T07:37:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.690+0000 7fdd29e00700 1 -- 172.21.15.89:0/690854049 wait complete. 2024-04-02T07:37:31.709 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2024-04-02T07:37:31.709 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2024-04-02T07:37:31.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.901+0000 7effba400700 1 Processor -- start 2024-04-02T07:37:31.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.901+0000 7effba400700 1 -- start start 2024-04-02T07:37:31.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effba400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7effb415a5c0 0x7effb40709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:31.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effba400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb40ce840 0x7effb40cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:31.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effba400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb40cb310 con 0x7effb415dfe0 2024-04-02T07:37:31.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effba400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb4070f00 con 0x7effb415a5c0 2024-04-02T07:37:31.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effba400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb40711a0 con 0x7effb40ce840 2024-04-02T07:37:31.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effb3400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7effb415a5c0 0x7effb40709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:31.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effb3400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7effb415a5c0 0x7effb40709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53814/0 (socket says 172.21.15.89:53814) 2024-04-02T07:37:31.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effb3400700 1 -- 172.21.15.89:0/2278097301 learned_addr learned my addr 172.21.15.89:0/2278097301 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:31.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effb2a00700 1 --1- 172.21.15.89:0/2278097301 >> v1:172.21.15.89:6789/0 conn(0x7effb415dfe0 0x7effb40ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56538/0 (socket says 172.21.15.89:56538) 2024-04-02T07:37:31.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effb3e00700 1 --2- 172.21.15.89:0/2278097301 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb40ce840 0x7effb40cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:31.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effb3400700 1 -- 172.21.15.89:0/2278097301 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb40ce840 msgr2=0x7effb40cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:31.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effb3400700 1 --2- 172.21.15.89:0/2278097301 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb40ce840 0x7effb40cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effb3400700 1 -- 172.21.15.89:0/2278097301 >> v1:172.21.15.89:6789/0 conn(0x7effb415dfe0 legacy=0x7effb40ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:31.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effb3400700 1 -- 172.21.15.89:0/2278097301 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb4071480 con 0x7effb415a5c0 2024-04-02T07:37:31.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effb3e00700 1 --2- 172.21.15.89:0/2278097301 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb40ce840 0x7effb40cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:31.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.903+0000 7effb2000700 1 -- 172.21.15.89:0/2278097301 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469194548 0 0) 0x7effb40cb310 con 0x7effb415dfe0 2024-04-02T07:37:31.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.904+0000 7effb3400700 1 --2- 172.21.15.89:0/2278097301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb415a5c0 0x7effb40709c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7effa40098f0 tx=0x7effa4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b659d73daf1fba29 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:31.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.904+0000 7effb2000700 1 -- 172.21.15.89:0/2278097301 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa400b8a0 con 0x7effb415a5c0 2024-04-02T07:37:31.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.904+0000 7effb2000700 1 -- 172.21.15.89:0/2278097301 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa400ba00 con 0x7effb415a5c0 2024-04-02T07:37:31.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.904+0000 7effb2000700 1 -- 172.21.15.89:0/2278097301 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa400bd90 con 0x7effb415a5c0 2024-04-02T07:37:31.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.904+0000 7effba400700 1 -- 172.21.15.89:0/2278097301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb415a5c0 msgr2=0x7effb40709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:31.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.904+0000 7effba400700 1 --2- 172.21.15.89:0/2278097301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb415a5c0 0x7effb40709c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7effa40098f0 tx=0x7effa4009c00 comp rx=0 tx=0).stop 2024-04-02T07:37:31.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.905+0000 7effba400700 1 -- 172.21.15.89:0/2278097301 shutdown_connections 2024-04-02T07:37:31.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.905+0000 7effba400700 1 --2- 172.21.15.89:0/2278097301 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb40ce840 0x7effb40cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.905+0000 7effba400700 1 --2- 172.21.15.89:0/2278097301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb415a5c0 0x7effb40709c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.905+0000 7effba400700 1 -- 172.21.15.89:0/2278097301 >> 172.21.15.89:0/2278097301 conn(0x7effb40b0d70 msgr2=0x7effb40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:31.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.905+0000 7effba400700 1 -- 172.21.15.89:0/2278097301 shutdown_connections 2024-04-02T07:37:31.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.905+0000 7effba400700 1 -- 172.21.15.89:0/2278097301 wait complete. 2024-04-02T07:37:31.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.907+0000 7effba400700 1 Processor -- start 2024-04-02T07:37:31.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.907+0000 7effba400700 1 -- start start 2024-04-02T07:37:31.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.908+0000 7effba400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb40ce840 0x7effb4166fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:31.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.908+0000 7effba400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb415a5c0 0x7effb415d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:31.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.908+0000 7effba400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb415dfe0 0x7effb415b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:31.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.908+0000 7effba400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb415c310 con 0x7effb415dfe0 2024-04-02T07:37:31.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.908+0000 7effba400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb415dd00 con 0x7effb415a5c0 2024-04-02T07:37:31.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.908+0000 7effba400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb4166b30 con 0x7effb40ce840 2024-04-02T07:37:31.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.908+0000 7effb3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb415dfe0 0x7effb415b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:31.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.909+0000 7effb3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb415dfe0 0x7effb415b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53826/0 (socket says 172.21.15.89:53826) 2024-04-02T07:37:31.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.909+0000 7effb3e00700 1 -- 172.21.15.89:0/1770607436 learned_addr learned my addr 172.21.15.89:0/1770607436 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:31.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.909+0000 7effb3400700 1 --2- 172.21.15.89:0/1770607436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb40ce840 0x7effb4166fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:31.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.909+0000 7effb2a00700 1 --2- 172.21.15.89:0/1770607436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb415a5c0 0x7effb415d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:31.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.909+0000 7effb3e00700 1 -- 172.21.15.89:0/1770607436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb40ce840 msgr2=0x7effb4166fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:31.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.909+0000 7effb3e00700 1 --2- 172.21.15.89:0/1770607436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb40ce840 0x7effb4166fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.909+0000 7effb3e00700 1 -- 172.21.15.89:0/1770607436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb415a5c0 msgr2=0x7effb415d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:31.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.909+0000 7effb3e00700 1 --2- 172.21.15.89:0/1770607436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb415a5c0 0x7effb415d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:31.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.909+0000 7effb3e00700 1 -- 172.21.15.89:0/1770607436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa4009550 con 0x7effb415dfe0 2024-04-02T07:37:31.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.909+0000 7effb3e00700 1 --2- 172.21.15.89:0/1770607436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb415dfe0 0x7effb415b520 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7effac007ec0 tx=0x7effac00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:31.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.909+0000 7effab400700 1 -- 172.21.15.89:0/1770607436 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac00dd80 con 0x7effb415dfe0 2024-04-02T07:37:31.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.909+0000 7effab400700 1 -- 172.21.15.89:0/1770607436 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac00f040 con 0x7effb415dfe0 2024-04-02T07:37:31.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.909+0000 7effab400700 1 -- 172.21.15.89:0/1770607436 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac003c60 con 0x7effb415dfe0 2024-04-02T07:37:31.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.909+0000 7effba400700 1 -- 172.21.15.89:0/1770607436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb4165f20 con 0x7effb415dfe0 2024-04-02T07:37:31.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.910+0000 7effba400700 1 -- 172.21.15.89:0/1770607436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb41664e0 con 0x7effb415dfe0 2024-04-02T07:37:31.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.911+0000 7effba400700 1 -- 172.21.15.89:0/1770607436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb4166ca0 con 0x7effb415dfe0 2024-04-02T07:37:31.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.912+0000 7effab400700 1 -- 172.21.15.89:0/1770607436 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7effac004030 con 0x7effb415dfe0 2024-04-02T07:37:31.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.912+0000 7effab400700 1 --2- 172.21.15.89:0/1770607436 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff9406ae90 0x7eff9406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:31.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.913+0000 7effab400700 1 -- 172.21.15.89:0/1770607436 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7effac0899d0 con 0x7effb415dfe0 2024-04-02T07:37:31.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.913+0000 7effb3400700 1 --2- 172.21.15.89:0/1770607436 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff9406ae90 0x7eff9406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:31.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.913+0000 7effb3400700 1 --2- 172.21.15.89:0/1770607436 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff9406ae90 0x7eff9406d350 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7effa4009130 tx=0x7effa4008030 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:31.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:31.917+0000 7effab400700 1 -- 172.21.15.89:0/1770607436 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7effac054680 con 0x7effb415dfe0 2024-04-02T07:37:32.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.145+0000 7effba400700 1 -- 172.21.15.89:0/1770607436 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7effb40cb560 con 0x7eff9406ae90 2024-04-02T07:37:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.215+0000 7effab400700 1 -- 172.21.15.89:0/1770607436 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7effa400bb70 con 0x7eff9406ae90 2024-04-02T07:37:32.217 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule activated for path /snap_test_dir1/dir5 2024-04-02T07:37:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.217+0000 7effba400700 1 -- 172.21.15.89:0/1770607436 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff9406ae90 msgr2=0x7eff9406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.217+0000 7effba400700 1 --2- 172.21.15.89:0/1770607436 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff9406ae90 0x7eff9406d350 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7effa4009130 tx=0x7effa4008030 comp rx=0 tx=0).stop 2024-04-02T07:37:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.217+0000 7effba400700 1 -- 172.21.15.89:0/1770607436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb415dfe0 msgr2=0x7effb415b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.217+0000 7effba400700 1 --2- 172.21.15.89:0/1770607436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb415dfe0 0x7effb415b520 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7effac007ec0 tx=0x7effac00d3b0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.217+0000 7effba400700 1 -- 172.21.15.89:0/1770607436 shutdown_connections 2024-04-02T07:37:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.217+0000 7effba400700 1 --2- 172.21.15.89:0/1770607436 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff9406ae90 0x7eff9406d350 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.217+0000 7effba400700 1 --2- 172.21.15.89:0/1770607436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb415dfe0 0x7effb415b520 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.217+0000 7effba400700 1 --2- 172.21.15.89:0/1770607436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb40ce840 0x7effb4166fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.217+0000 7effba400700 1 --2- 172.21.15.89:0/1770607436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb415a5c0 0x7effb415d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.217+0000 7effba400700 1 -- 172.21.15.89:0/1770607436 >> 172.21.15.89:0/1770607436 conn(0x7effb40b0d70 msgr2=0x7effb406deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.218+0000 7effba400700 1 -- 172.21.15.89:0/1770607436 shutdown_connections 2024-04-02T07:37:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.218+0000 7effba400700 1 -- 172.21.15.89:0/1770607436 wait complete. 2024-04-02T07:37:32.237 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2024-04-02T07:37:32.238 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2024-04-02T07:37:32.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.430+0000 7f3316c00700 1 Processor -- start 2024-04-02T07:37:32.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.430+0000 7f3316c00700 1 -- start start 2024-04-02T07:37:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.432+0000 7f3316c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ce230 0x7f33100ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.432+0000 7f3316c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33100709c0 con 0x7f331015d800 2024-04-02T07:37:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.432+0000 7f3316c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3310070c80 con 0x7f33100b2070 2024-04-02T07:37:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.432+0000 7f3316c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33100b40f0 con 0x7f33100ce230 2024-04-02T07:37:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.432+0000 7f330fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f331015d800 0x7f33100725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56548/0 (socket says 172.21.15.89:56548) 2024-04-02T07:37:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.432+0000 7f330fe00700 1 -- 172.21.15.89:0/2551281580 learned_addr learned my addr 172.21.15.89:0/2551281580 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.432+0000 7f330f400700 1 --2- 172.21.15.89:0/2551281580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ce230 0x7f33100ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.432+0000 7f330ea00700 1 -- 172.21.15.89:0/2551281580 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511720656 0 0) 0x7f33100709c0 con 0x7f331015d800 2024-04-02T07:37:32.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.432+0000 7f330ea00700 1 -- 172.21.15.89:0/2551281580 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32f40035b0 con 0x7f331015d800 2024-04-02T07:37:32.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330ea00700 1 -- 172.21.15.89:0/2551281580 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2709375308 0 0) 0x7f32f40035b0 con 0x7f331015d800 2024-04-02T07:37:32.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330ea00700 1 -- 172.21.15.89:0/2551281580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ce230 msgr2=0x7f33100ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:32.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330ea00700 1 --2- 172.21.15.89:0/2551281580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ce230 0x7f33100ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330ea00700 1 -- 172.21.15.89:0/2551281580 >> v1:172.21.15.113:6789/0 conn(0x7f33100b2070 legacy=0x7f331015dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:32.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330ea00700 1 -- 172.21.15.89:0/2551281580 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3310070f60 con 0x7f331015d800 2024-04-02T07:37:32.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330ea00700 1 -- 172.21.15.89:0/2551281580 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f3300001670 con 0x7f331015d800 2024-04-02T07:37:32.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330ea00700 1 -- 172.21.15.89:0/2551281580 >> v1:172.21.15.89:6789/0 conn(0x7f331015d800 legacy=0x7f33100725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:32.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330ea00700 1 --2- 172.21.15.89:0/2551281580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32f4003a30 0x7f32f4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:32.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330ea00700 1 --2- 172.21.15.89:0/2551281580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ce230 0x7f32f4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:32.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330ea00700 1 --2- 172.21.15.89:0/2551281580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32f4004d90 0x7f32f4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:32.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330ea00700 1 -- 172.21.15.89:0/2551281580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33100b40f0 con 0x7f32f4003a30 2024-04-02T07:37:32.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330ea00700 1 -- 172.21.15.89:0/2551281580 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f400ccb0 con 0x7f32f4004d90 2024-04-02T07:37:32.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330ea00700 1 -- 172.21.15.89:0/2551281580 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f4005a60 con 0x7f33100ce230 2024-04-02T07:37:32.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330fe00700 1 --2- 172.21.15.89:0/2551281580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32f4003a30 0x7f32f4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:32.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f3314c00700 1 --2- 172.21.15.89:0/2551281580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32f4004d90 0x7f32f4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:32.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330f400700 1 --2- 172.21.15.89:0/2551281580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ce230 0x7f32f4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:32.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330fe00700 1 -- 172.21.15.89:0/2551281580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ce230 msgr2=0x7f32f4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:32.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330fe00700 1 --2- 172.21.15.89:0/2551281580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ce230 0x7f32f4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330fe00700 1 -- 172.21.15.89:0/2551281580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32f4004d90 msgr2=0x7f32f4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:32.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.433+0000 7f330fe00700 1 --2- 172.21.15.89:0/2551281580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32f4004d90 0x7f32f4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.434+0000 7f330fe00700 1 -- 172.21.15.89:0/2551281580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f40035b0 con 0x7f32f4003a30 2024-04-02T07:37:32.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.434+0000 7f330f400700 1 --2- 172.21.15.89:0/2551281580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ce230 0x7f32f4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:32.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.434+0000 7f330fe00700 1 --2- 172.21.15.89:0/2551281580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32f4003a30 0x7f32f4003e50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f330000c160 tx=0x7f32f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=944709c4368ac3f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:32.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.434+0000 7f330ea00700 1 -- 172.21.15.89:0/2551281580 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3300007b70 con 0x7f32f4003a30 2024-04-02T07:37:32.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.434+0000 7f330ea00700 1 -- 172.21.15.89:0/2551281580 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330000cd20 con 0x7f32f4003a30 2024-04-02T07:37:32.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.434+0000 7f330ea00700 1 -- 172.21.15.89:0/2551281580 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3300005ba0 con 0x7f32f4003a30 2024-04-02T07:37:32.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.434+0000 7f3316c00700 1 -- 172.21.15.89:0/2551281580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32f4003a30 msgr2=0x7f32f4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:32.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.434+0000 7f3316c00700 1 --2- 172.21.15.89:0/2551281580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32f4003a30 0x7f32f4003e50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f330000c160 tx=0x7f32f4005880 comp rx=0 tx=0).stop 2024-04-02T07:37:32.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.434+0000 7f3314c00700 1 -- 172.21.15.89:0/2551281580 reap_dead start 2024-04-02T07:37:32.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.435+0000 7f3316c00700 1 -- 172.21.15.89:0/2551281580 shutdown_connections 2024-04-02T07:37:32.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.435+0000 7f3316c00700 1 -- 172.21.15.89:0/2551281580 >> 172.21.15.89:0/2551281580 conn(0x7f33100b0d70 msgr2=0x7f33100b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:32.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.435+0000 7f3316c00700 1 -- 172.21.15.89:0/2551281580 shutdown_connections 2024-04-02T07:37:32.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.435+0000 7f3316c00700 1 -- 172.21.15.89:0/2551281580 wait complete. 2024-04-02T07:37:32.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.436+0000 7f3316c00700 1 Processor -- start 2024-04-02T07:37:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.436+0000 7f3316c00700 1 -- start start 2024-04-02T07:37:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.437+0000 7f3316c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32f4003a30 0x7f33100cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.437+0000 7f3316c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33101e3fe0 0x7f33101f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.437+0000 7f3316c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ac7b0 0x7f33100acc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.437+0000 7f3316c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33100ad170 con 0x7f33101e3fe0 2024-04-02T07:37:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.437+0000 7f3316c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331014f710 con 0x7f32f4003a30 2024-04-02T07:37:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.437+0000 7f3316c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33101e4510 con 0x7f33100ac7b0 2024-04-02T07:37:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.438+0000 7f330f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33101e3fe0 0x7f33101f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.438+0000 7f330f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33101e3fe0 0x7f33101f2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53850/0 (socket says 172.21.15.89:53850) 2024-04-02T07:37:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.438+0000 7f330f400700 1 -- 172.21.15.89:0/523410819 learned_addr learned my addr 172.21.15.89:0/523410819 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.438+0000 7f330fe00700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32f4003a30 0x7f33100cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.438+0000 7f3314c00700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ac7b0 0x7f33100acc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.438+0000 7f330f400700 1 -- 172.21.15.89:0/523410819 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ac7b0 msgr2=0x7f33100acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:32.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.438+0000 7f330f400700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ac7b0 0x7f33100acc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.438+0000 7f330f400700 1 -- 172.21.15.89:0/523410819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32f4003a30 msgr2=0x7f33100cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:32.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.438+0000 7f330f400700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32f4003a30 0x7f33100cbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.438+0000 7f330f400700 1 -- 172.21.15.89:0/523410819 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33101e47f0 con 0x7f33101e3fe0 2024-04-02T07:37:32.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.438+0000 7f3314c00700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ac7b0 0x7f33100acc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:37:32.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.438+0000 7f330f400700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33101e3fe0 0x7f33101f2de0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0118d0 tx=0x7f32fc011be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:32.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.438+0000 7f3307e00700 1 -- 172.21.15.89:0/523410819 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc004220 con 0x7f33101e3fe0 2024-04-02T07:37:32.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.439+0000 7f3307e00700 1 -- 172.21.15.89:0/523410819 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32fc004380 con 0x7f33101e3fe0 2024-04-02T07:37:32.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.439+0000 7f3307e00700 1 -- 172.21.15.89:0/523410819 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc0036a0 con 0x7f33101e3fe0 2024-04-02T07:37:32.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.439+0000 7f3316c00700 1 -- 172.21.15.89:0/523410819 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33101e4ad0 con 0x7f33101e3fe0 2024-04-02T07:37:32.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.439+0000 7f3316c00700 1 -- 172.21.15.89:0/523410819 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33100ac480 con 0x7f33101e3fe0 2024-04-02T07:37:32.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.440+0000 7f3316c00700 1 -- 172.21.15.89:0/523410819 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3310067720 con 0x7f33101e3fe0 2024-04-02T07:37:32.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.440+0000 7f3307e00700 1 -- 172.21.15.89:0/523410819 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f32fc003820 con 0x7f33101e3fe0 2024-04-02T07:37:32.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.441+0000 7f3307e00700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32f006ae90 0x7f32f006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:32.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.441+0000 7f3307e00700 1 -- 172.21.15.89:0/523410819 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f32fc089aa0 con 0x7f33101e3fe0 2024-04-02T07:37:32.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.442+0000 7f330fe00700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32f006ae90 0x7f32f006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:32.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.442+0000 7f330fe00700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32f006ae90 0x7f32f006d350 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3300000c00 tx=0x7f3300007000 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:32.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.446+0000 7f3307e00700 1 -- 172.21.15.89:0/523410819 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f32fc054900 con 0x7f33101e3fe0 2024-04-02T07:37:32.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.679+0000 7f3316c00700 1 -- 172.21.15.89:0/523410819 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f33100cd9e0 con 0x7f32f006ae90 2024-04-02T07:37:32.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.748+0000 7f3307e00700 1 -- 172.21.15.89:0/523410819 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f3300023070 con 0x7f32f006ae90 2024-04-02T07:37:32.750 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule activated for path /snap_test_dir1/dir6 2024-04-02T07:37:32.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.750+0000 7f3316c00700 1 -- 172.21.15.89:0/523410819 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32f006ae90 msgr2=0x7f32f006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:32.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.750+0000 7f3316c00700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32f006ae90 0x7f32f006d350 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3300000c00 tx=0x7f3300007000 comp rx=0 tx=0).stop 2024-04-02T07:37:32.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.750+0000 7f3316c00700 1 -- 172.21.15.89:0/523410819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33101e3fe0 msgr2=0x7f33101f2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:32.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.750+0000 7f3316c00700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33101e3fe0 0x7f33101f2de0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0118d0 tx=0x7f32fc011be0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.750+0000 7f3316c00700 1 -- 172.21.15.89:0/523410819 shutdown_connections 2024-04-02T07:37:32.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.750+0000 7f3316c00700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32f006ae90 0x7f32f006d350 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.750+0000 7f3316c00700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32f4003a30 0x7f33100cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.750+0000 7f3316c00700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33100ac7b0 0x7f33100acc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.750+0000 7f3316c00700 1 --2- 172.21.15.89:0/523410819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33101e3fe0 0x7f33101f2de0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.750+0000 7f3316c00700 1 -- 172.21.15.89:0/523410819 >> 172.21.15.89:0/523410819 conn(0x7f33100b0d70 msgr2=0x7f33100aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:32.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.751+0000 7f3316c00700 1 -- 172.21.15.89:0/523410819 shutdown_connections 2024-04-02T07:37:32.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.751+0000 7f3316c00700 1 -- 172.21.15.89:0/523410819 wait complete. 2024-04-02T07:37:32.771 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2024-04-02T07:37:32.771 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2024-04-02T07:37:32.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.964+0000 7fb2a0c00700 1 Processor -- start 2024-04-02T07:37:32.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.964+0000 7fb2a0c00700 1 -- start start 2024-04-02T07:37:32.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.965+0000 7fb2a0c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb29c0d16a0 0x7fb29c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:32.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.965+0000 7fb2a0c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c161420 con 0x7fb29c15d9d0 2024-04-02T07:37:32.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.965+0000 7fb2a0c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c161700 con 0x7fb29c0d2370 2024-04-02T07:37:32.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.965+0000 7fb2a0c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb29c0d4340 con 0x7fb29c0d16a0 2024-04-02T07:37:32.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.966+0000 7fb299400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb29c0d16a0 0x7fb29c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:32.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.966+0000 7fb299e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb29c15d9d0 0x7fb29c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56558/0 (socket says 172.21.15.89:56558) 2024-04-02T07:37:32.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.966+0000 7fb299e00700 1 -- 172.21.15.89:0/4238499966 learned_addr learned my addr 172.21.15.89:0/4238499966 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:37:32.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.966+0000 7fb299400700 1 -- 172.21.15.89:0/4238499966 >> v1:172.21.15.113:6789/0 conn(0x7fb29c0d2370 legacy=0x7fb29c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:32.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.966+0000 7fb299400700 1 -- 172.21.15.89:0/4238499966 >> v1:172.21.15.89:6789/0 conn(0x7fb29c15d9d0 legacy=0x7fb29c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:32.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.966+0000 7fb299400700 1 -- 172.21.15.89:0/4238499966 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c1619e0 con 0x7fb29c0d16a0 2024-04-02T07:37:32.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.966+0000 7fb298a00700 1 -- 172.21.15.89:0/4238499966 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149191487 0 0) 0x7fb29c161420 con 0x7fb29c15d9d0 2024-04-02T07:37:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.966+0000 7fb299400700 1 --2- 172.21.15.89:0/4238499966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29c0d16a0 0x7fb29c0d1e30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb288009830 tx=0x7fb288009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0a0cfa1f8e0974 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.967+0000 7fb298a00700 1 -- 172.21.15.89:0/4238499966 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb288011830 con 0x7fb29c0d16a0 2024-04-02T07:37:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.967+0000 7fb298a00700 1 -- 172.21.15.89:0/4238499966 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb288011990 con 0x7fb29c0d16a0 2024-04-02T07:37:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.967+0000 7fb298a00700 1 -- 172.21.15.89:0/4238499966 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb288011d40 con 0x7fb29c0d16a0 2024-04-02T07:37:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.967+0000 7fb2a0c00700 1 -- 172.21.15.89:0/4238499966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29c0d16a0 msgr2=0x7fb29c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.967+0000 7fb2a0c00700 1 --2- 172.21.15.89:0/4238499966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29c0d16a0 0x7fb29c0d1e30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb288009830 tx=0x7fb288009b40 comp rx=0 tx=0).stop 2024-04-02T07:37:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.967+0000 7fb2a0c00700 1 -- 172.21.15.89:0/4238499966 shutdown_connections 2024-04-02T07:37:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.967+0000 7fb2a0c00700 1 --2- 172.21.15.89:0/4238499966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29c0d16a0 0x7fb29c0d1e30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.967+0000 7fb2a0c00700 1 -- 172.21.15.89:0/4238499966 >> 172.21.15.89:0/4238499966 conn(0x7fb29c08aff0 msgr2=0x7fb29c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.967+0000 7fb2a0c00700 1 -- 172.21.15.89:0/4238499966 shutdown_connections 2024-04-02T07:37:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.967+0000 7fb2a0c00700 1 -- 172.21.15.89:0/4238499966 wait complete. 2024-04-02T07:37:32.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.969+0000 7fb2a0c00700 1 Processor -- start 2024-04-02T07:37:32.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.969+0000 7fb2a0c00700 1 -- start start 2024-04-02T07:37:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.970+0000 7fb2a0c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb29c0d16a0 0x7fb29c1e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.970+0000 7fb2a0c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb29c0d2370 0x7fb29c1e8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.970+0000 7fb2a0c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29c15d9d0 0x7fb29c0b6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:32.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.970+0000 7fb2a0c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c15fb10 con 0x7fb29c15d9d0 2024-04-02T07:37:32.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.970+0000 7fb2a0c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c1e7b80 con 0x7fb29c0d2370 2024-04-02T07:37:32.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.970+0000 7fb2a0c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c1e7e60 con 0x7fb29c0d16a0 2024-04-02T07:37:32.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.970+0000 7fb29a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29c15d9d0 0x7fb29c0b6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:32.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.970+0000 7fb29a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29c15d9d0 0x7fb29c0b6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53870/0 (socket says 172.21.15.89:53870) 2024-04-02T07:37:32.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.970+0000 7fb29a800700 1 -- 172.21.15.89:0/2884891372 learned_addr learned my addr 172.21.15.89:0/2884891372 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:32.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.971+0000 7fb299e00700 1 --2- 172.21.15.89:0/2884891372 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb29c0d16a0 0x7fb29c1e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:32.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.971+0000 7fb299400700 1 --2- 172.21.15.89:0/2884891372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb29c0d2370 0x7fb29c1e8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:32.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.971+0000 7fb29a800700 1 -- 172.21.15.89:0/2884891372 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb29c0d16a0 msgr2=0x7fb29c1e7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:32.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.971+0000 7fb29a800700 1 --2- 172.21.15.89:0/2884891372 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb29c0d16a0 0x7fb29c1e7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.971+0000 7fb29a800700 1 -- 172.21.15.89:0/2884891372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb29c0d2370 msgr2=0x7fb29c1e8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:32.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.971+0000 7fb29a800700 1 --2- 172.21.15.89:0/2884891372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb29c0d2370 0x7fb29c1e8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:32.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.971+0000 7fb29a800700 1 -- 172.21.15.89:0/2884891372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb288009440 con 0x7fb29c15d9d0 2024-04-02T07:37:32.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.971+0000 7fb29a800700 1 --2- 172.21.15.89:0/2884891372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29c15d9d0 0x7fb29c0b6470 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb29400cc50 tx=0x7fb29400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:32.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.971+0000 7fb292000700 1 -- 172.21.15.89:0/2884891372 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29400ad80 con 0x7fb29c15d9d0 2024-04-02T07:37:32.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.971+0000 7fb292000700 1 -- 172.21.15.89:0/2884891372 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29400ce50 con 0x7fb29c15d9d0 2024-04-02T07:37:32.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.971+0000 7fb292000700 1 -- 172.21.15.89:0/2884891372 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294003c40 con 0x7fb29c15d9d0 2024-04-02T07:37:32.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.971+0000 7fb2a0c00700 1 -- 172.21.15.89:0/2884891372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29c1e8140 con 0x7fb29c15d9d0 2024-04-02T07:37:32.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.971+0000 7fb2a0c00700 1 -- 172.21.15.89:0/2884891372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29c1e77c0 con 0x7fb29c15d9d0 2024-04-02T07:37:32.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.973+0000 7fb2a0c00700 1 -- 172.21.15.89:0/2884891372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29c0d1ca0 con 0x7fb29c15d9d0 2024-04-02T07:37:32.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.973+0000 7fb292000700 1 -- 172.21.15.89:0/2884891372 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fb2940050d0 con 0x7fb29c15d9d0 2024-04-02T07:37:32.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.974+0000 7fb292000700 1 --2- 172.21.15.89:0/2884891372 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb27c06ae90 0x7fb27c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:32.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.974+0000 7fb292000700 1 -- 172.21.15.89:0/2884891372 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb294089840 con 0x7fb29c15d9d0 2024-04-02T07:37:32.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.974+0000 7fb299e00700 1 --2- 172.21.15.89:0/2884891372 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb27c06ae90 0x7fb27c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.975+0000 7fb299e00700 1 --2- 172.21.15.89:0/2884891372 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb27c06ae90 0x7fb27c06d350 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb28c000f60 tx=0x7fb28c007db0 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:32.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:32.979+0000 7fb292000700 1 -- 172.21.15.89:0/2884891372 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb294054730 con 0x7fb29c15d9d0 2024-04-02T07:37:33.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.208+0000 7fb2a0c00700 1 -- 172.21.15.89:0/2884891372 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb29c161fa0 con 0x7fb27c06ae90 2024-04-02T07:37:33.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.272+0000 7fb292000700 1 -- 172.21.15.89:0/2884891372 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb28c004870 con 0x7fb27c06ae90 2024-04-02T07:37:33.274 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule activated for path /snap_test_dir1/dir7 2024-04-02T07:37:33.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.274+0000 7fb2a0c00700 1 -- 172.21.15.89:0/2884891372 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb27c06ae90 msgr2=0x7fb27c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:33.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.274+0000 7fb2a0c00700 1 --2- 172.21.15.89:0/2884891372 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb27c06ae90 0x7fb27c06d350 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb28c000f60 tx=0x7fb28c007db0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.274+0000 7fb2a0c00700 1 -- 172.21.15.89:0/2884891372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29c15d9d0 msgr2=0x7fb29c0b6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:33.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.274+0000 7fb2a0c00700 1 --2- 172.21.15.89:0/2884891372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29c15d9d0 0x7fb29c0b6470 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb29400cc50 tx=0x7fb29400a340 comp rx=0 tx=0).stop 2024-04-02T07:37:33.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.274+0000 7fb2a0c00700 1 -- 172.21.15.89:0/2884891372 shutdown_connections 2024-04-02T07:37:33.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.274+0000 7fb2a0c00700 1 --2- 172.21.15.89:0/2884891372 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb27c06ae90 0x7fb27c06d350 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.274+0000 7fb2a0c00700 1 --2- 172.21.15.89:0/2884891372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb29c15d9d0 0x7fb29c0b6470 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.274+0000 7fb2a0c00700 1 --2- 172.21.15.89:0/2884891372 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb29c0d16a0 0x7fb29c1e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.274+0000 7fb2a0c00700 1 --2- 172.21.15.89:0/2884891372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb29c0d2370 0x7fb29c1e8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.274+0000 7fb2a0c00700 1 -- 172.21.15.89:0/2884891372 >> 172.21.15.89:0/2884891372 conn(0x7fb29c08aff0 msgr2=0x7fb29c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:33.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.274+0000 7fb2a0c00700 1 -- 172.21.15.89:0/2884891372 shutdown_connections 2024-04-02T07:37:33.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.275+0000 7fb2a0c00700 1 -- 172.21.15.89:0/2884891372 wait complete. 2024-04-02T07:37:33.295 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2024-04-02T07:37:33.295 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2024-04-02T07:37:33.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.489+0000 7fe13b400700 1 Processor -- start 2024-04-02T07:37:33.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.490+0000 7fe13b400700 1 -- start start 2024-04-02T07:37:33.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.491+0000 7fe13b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe13415d9d0 0x7fe1340d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:33.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.491+0000 7fe13b400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe1340d16a0 0x7fe1340d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:33.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.491+0000 7fe13b400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe134161420 con 0x7fe1340d2370 2024-04-02T07:37:33.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.491+0000 7fe13b400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1340d27e0 con 0x7fe13415d9d0 2024-04-02T07:37:33.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.491+0000 7fe13b400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe134161700 con 0x7fe1340d16a0 2024-04-02T07:37:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.491+0000 7fe138a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe13415d9d0 0x7fe1340d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.491+0000 7fe138a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe13415d9d0 0x7fe1340d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53886/0 (socket says 172.21.15.89:53886) 2024-04-02T07:37:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.491+0000 7fe139400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe1340d2370 0x7fe13415ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56572/0 (socket says 172.21.15.89:56572) 2024-04-02T07:37:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.491+0000 7fe138a00700 1 -- 172.21.15.89:0/317830775 learned_addr learned my addr 172.21.15.89:0/317830775 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.491+0000 7fe133e00700 1 --2- 172.21.15.89:0/317830775 >> v2:172.21.15.113:3300/0 conn(0x7fe1340d16a0 0x7fe1340d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.491+0000 7fe138a00700 1 -- 172.21.15.89:0/317830775 >> v2:172.21.15.113:3300/0 conn(0x7fe1340d16a0 msgr2=0x7fe1340d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.491+0000 7fe138a00700 1 --2- 172.21.15.89:0/317830775 >> v2:172.21.15.113:3300/0 conn(0x7fe1340d16a0 0x7fe1340d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.491+0000 7fe138a00700 1 -- 172.21.15.89:0/317830775 >> v1:172.21.15.89:6789/0 conn(0x7fe1340d2370 legacy=0x7fe13415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.491+0000 7fe138a00700 1 -- 172.21.15.89:0/317830775 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1341619e0 con 0x7fe13415d9d0 2024-04-02T07:37:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.492+0000 7fe138a00700 1 --2- 172.21.15.89:0/317830775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe13415d9d0 0x7fe1340d3c30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe1340d12f0 tx=0x7fe128009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53e5e1cae32806d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.492+0000 7fe133400700 1 -- 172.21.15.89:0/317830775 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12800b870 con 0x7fe13415d9d0 2024-04-02T07:37:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.492+0000 7fe133400700 1 -- 172.21.15.89:0/317830775 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12800b9d0 con 0x7fe13415d9d0 2024-04-02T07:37:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.492+0000 7fe133400700 1 -- 172.21.15.89:0/317830775 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12800bd60 con 0x7fe13415d9d0 2024-04-02T07:37:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.492+0000 7fe13b400700 1 -- 172.21.15.89:0/317830775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe13415d9d0 msgr2=0x7fe1340d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.492+0000 7fe13b400700 1 --2- 172.21.15.89:0/317830775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe13415d9d0 0x7fe1340d3c30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe1340d12f0 tx=0x7fe128009bd0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.493+0000 7fe13b400700 1 -- 172.21.15.89:0/317830775 shutdown_connections 2024-04-02T07:37:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.493+0000 7fe13b400700 1 --2- 172.21.15.89:0/317830775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe13415d9d0 0x7fe1340d3c30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.493+0000 7fe13b400700 1 --2- 172.21.15.89:0/317830775 >> v2:172.21.15.113:3300/0 conn(0x7fe1340d16a0 0x7fe1340d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.493+0000 7fe13b400700 1 -- 172.21.15.89:0/317830775 >> 172.21.15.89:0/317830775 conn(0x7fe13408aff0 msgr2=0x7fe1340d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.493+0000 7fe13b400700 1 -- 172.21.15.89:0/317830775 shutdown_connections 2024-04-02T07:37:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.493+0000 7fe13b400700 1 -- 172.21.15.89:0/317830775 wait complete. 2024-04-02T07:37:33.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.494+0000 7fe13b400700 1 Processor -- start 2024-04-02T07:37:33.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.494+0000 7fe13b400700 1 -- start start 2024-04-02T07:37:33.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.495+0000 7fe13b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1340d16a0 0x7fe1340b7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.495+0000 7fe13b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1340d2370 0x7fe1341580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.495+0000 7fe13b400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe13415d9d0 0x7fe1340b8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.495+0000 7fe13b400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1340b6fd0 con 0x7fe1340d16a0 2024-04-02T07:37:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.495+0000 7fe13b400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1340b79a0 con 0x7fe13415d9d0 2024-04-02T07:37:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.495+0000 7fe13b400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1340b63c0 con 0x7fe1340d2370 2024-04-02T07:37:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.495+0000 7fe138a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1340d16a0 0x7fe1340b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe138a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1340d16a0 0x7fe1340b7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53892/0 (socket says 172.21.15.89:53892) 2024-04-02T07:37:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe138a00700 1 -- 172.21.15.89:0/3369181950 learned_addr learned my addr 172.21.15.89:0/3369181950 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe133e00700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1340d2370 0x7fe1341580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe139400700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe13415d9d0 0x7fe1340b8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe138a00700 1 -- 172.21.15.89:0/3369181950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1340d2370 msgr2=0x7fe1341580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe138a00700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1340d2370 0x7fe1341580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe138a00700 1 -- 172.21.15.89:0/3369181950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe13415d9d0 msgr2=0x7fe1340b8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe138a00700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe13415d9d0 0x7fe1340b8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe138a00700 1 -- 172.21.15.89:0/3369181950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe128009550 con 0x7fe1340d16a0 2024-04-02T07:37:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe139400700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe13415d9d0 0x7fe1340b8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:37:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe138a00700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1340d16a0 0x7fe1340b7460 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe128000c00 tx=0x7fe128004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe130c00700 1 -- 172.21.15.89:0/3369181950 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe128004520 con 0x7fe1340d16a0 2024-04-02T07:37:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe130c00700 1 -- 172.21.15.89:0/3369181950 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12801e070 con 0x7fe1340d16a0 2024-04-02T07:37:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe13b400700 1 -- 172.21.15.89:0/3369181950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1340b66a0 con 0x7fe1340d16a0 2024-04-02T07:37:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.496+0000 7fe130c00700 1 -- 172.21.15.89:0/3369181950 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1280038a0 con 0x7fe1340d16a0 2024-04-02T07:37:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.497+0000 7fe13b400700 1 -- 172.21.15.89:0/3369181950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1340b6c60 con 0x7fe1340d16a0 2024-04-02T07:37:33.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.498+0000 7fe13b400700 1 -- 172.21.15.89:0/3369181950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1340d1ca0 con 0x7fe1340d16a0 2024-04-02T07:37:33.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.498+0000 7fe130c00700 1 -- 172.21.15.89:0/3369181950 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fe128003a00 con 0x7fe1340d16a0 2024-04-02T07:37:33.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.499+0000 7fe130c00700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe11406ae90 0x7fe11406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:33.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.499+0000 7fe130c00700 1 -- 172.21.15.89:0/3369181950 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fe128020070 con 0x7fe1340d16a0 2024-04-02T07:37:33.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.499+0000 7fe133e00700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe11406ae90 0x7fe11406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:33.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.500+0000 7fe133e00700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe11406ae90 0x7fe11406d350 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe1340cdc00 tx=0x7fe120008040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.503+0000 7fe130c00700 1 -- 172.21.15.89:0/3369181950 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe128055c20 con 0x7fe1340d16a0 2024-04-02T07:37:33.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.733+0000 7fe13b400700 1 -- 172.21.15.89:0/3369181950 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe134161fa0 con 0x7fe11406ae90 2024-04-02T07:37:33.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.797+0000 7fe130c00700 1 -- 172.21.15.89:0/3369181950 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fe1200095e0 con 0x7fe11406ae90 2024-04-02T07:37:33.799 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule activated for path /snap_test_dir1/dir8 2024-04-02T07:37:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.799+0000 7fe13b400700 1 -- 172.21.15.89:0/3369181950 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe11406ae90 msgr2=0x7fe11406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.799+0000 7fe13b400700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe11406ae90 0x7fe11406d350 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe1340cdc00 tx=0x7fe120008040 comp rx=0 tx=0).stop 2024-04-02T07:37:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.799+0000 7fe13b400700 1 -- 172.21.15.89:0/3369181950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1340d16a0 msgr2=0x7fe1340b7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.799+0000 7fe13b400700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1340d16a0 0x7fe1340b7460 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe128000c00 tx=0x7fe128004750 comp rx=0 tx=0).stop 2024-04-02T07:37:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.800+0000 7fe13b400700 1 -- 172.21.15.89:0/3369181950 shutdown_connections 2024-04-02T07:37:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.800+0000 7fe13b400700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe11406ae90 0x7fe11406d350 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.800+0000 7fe13b400700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe13415d9d0 0x7fe1340b8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.800+0000 7fe13b400700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1340d16a0 0x7fe1340b7460 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.800+0000 7fe13b400700 1 --2- 172.21.15.89:0/3369181950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1340d2370 0x7fe1341580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.800+0000 7fe13b400700 1 -- 172.21.15.89:0/3369181950 >> 172.21.15.89:0/3369181950 conn(0x7fe13408aff0 msgr2=0x7fe13415bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.800+0000 7fe13b400700 1 -- 172.21.15.89:0/3369181950 shutdown_connections 2024-04-02T07:37:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:33.800+0000 7fe13b400700 1 -- 172.21.15.89:0/3369181950 wait complete. 2024-04-02T07:37:33.819 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2024-04-02T07:37:33.819 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2024-04-02T07:37:34.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.013+0000 7ffbda200700 1 Processor -- start 2024-04-02T07:37:34.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.013+0000 7ffbda200700 1 -- start start 2024-04-02T07:37:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.014+0000 7ffbda200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd4160e50 0x7ffbd40b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.015+0000 7ffbda200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffbd40d3f20 0x7ffbd40b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:34.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.015+0000 7ffbda200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ffbd40b3f80 0x7ffbd40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:34.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.015+0000 7ffbda200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd4161230 con 0x7ffbd40d3f20 2024-04-02T07:37:34.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.015+0000 7ffbda200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd40d2a80 con 0x7ffbd40b3f80 2024-04-02T07:37:34.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.015+0000 7ffbda200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd40d43b0 con 0x7ffbd4160e50 2024-04-02T07:37:34.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.015+0000 7ffbd2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffbd40d3f20 0x7ffbd40b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:34.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.015+0000 7ffbd2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffbd40d3f20 0x7ffbd40b53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53898/0 (socket says 172.21.15.89:53898) 2024-04-02T07:37:34.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.015+0000 7ffbd2a00700 1 -- 172.21.15.89:0/2309657558 learned_addr learned my addr 172.21.15.89:0/2309657558 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.015+0000 7ffbd2a00700 1 -- 172.21.15.89:0/2309657558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd4160e50 msgr2=0x7ffbd40b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.015+0000 7ffbd2a00700 1 --2- 172.21.15.89:0/2309657558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd4160e50 0x7ffbd40b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.015+0000 7ffbd2a00700 1 -- 172.21.15.89:0/2309657558 >> v2:172.21.15.113:3300/0 conn(0x7ffbd40b3f80 msgr2=0x7ffbd40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:37:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.015+0000 7ffbd2a00700 1 --2- 172.21.15.89:0/2309657558 >> v2:172.21.15.113:3300/0 conn(0x7ffbd40b3f80 0x7ffbd40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.015+0000 7ffbd2a00700 1 -- 172.21.15.89:0/2309657558 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd40d21e0 con 0x7ffbd40d3f20 2024-04-02T07:37:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.015+0000 7ffbd2a00700 1 --2- 172.21.15.89:0/2309657558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd40d3f20 0x7ffbd40b53d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40d3e70 tx=0x7ffbc0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68cf12c3c2772ac3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:34.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.016+0000 7ffbd2000700 1 -- 172.21.15.89:0/2309657558 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc000b870 con 0x7ffbd40d3f20 2024-04-02T07:37:34.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.016+0000 7ffbd2000700 1 -- 172.21.15.89:0/2309657558 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc000b9d0 con 0x7ffbd40d3f20 2024-04-02T07:37:34.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.016+0000 7ffbd2000700 1 -- 172.21.15.89:0/2309657558 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc000bd60 con 0x7ffbd40d3f20 2024-04-02T07:37:34.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.016+0000 7ffbda200700 1 -- 172.21.15.89:0/2309657558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd40d3f20 msgr2=0x7ffbd40b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:34.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.016+0000 7ffbda200700 1 --2- 172.21.15.89:0/2309657558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd40d3f20 0x7ffbd40b53d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40d3e70 tx=0x7ffbc0009bd0 comp rx=0 tx=0).stop 2024-04-02T07:37:34.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.016+0000 7ffbda200700 1 -- 172.21.15.89:0/2309657558 shutdown_connections 2024-04-02T07:37:34.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.016+0000 7ffbda200700 1 --2- 172.21.15.89:0/2309657558 >> v2:172.21.15.113:3300/0 conn(0x7ffbd40b3f80 0x7ffbd40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:34.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.016+0000 7ffbda200700 1 --2- 172.21.15.89:0/2309657558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd4160e50 0x7ffbd40b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:34.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.016+0000 7ffbda200700 1 --2- 172.21.15.89:0/2309657558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd40d3f20 0x7ffbd40b53d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:34.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.016+0000 7ffbda200700 1 -- 172.21.15.89:0/2309657558 >> 172.21.15.89:0/2309657558 conn(0x7ffbd408aff0 msgr2=0x7ffbd40d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:34.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.016+0000 7ffbda200700 1 -- 172.21.15.89:0/2309657558 shutdown_connections 2024-04-02T07:37:34.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.017+0000 7ffbda200700 1 -- 172.21.15.89:0/2309657558 wait complete. 2024-04-02T07:37:34.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.018+0000 7ffbda200700 1 Processor -- start 2024-04-02T07:37:34.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.018+0000 7ffbda200700 1 -- start start 2024-04-02T07:37:34.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.019+0000 7ffbda200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd40b3f80 0x7ffbd40b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:34.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.019+0000 7ffbda200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbd4160e50 0x7ffbd40b77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:34.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.019+0000 7ffbda200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd40b80b0 0x7ffbd40b8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:34.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.019+0000 7ffbda200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd415c080 con 0x7ffbd40b3f80 2024-04-02T07:37:34.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.019+0000 7ffbda200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd409d420 con 0x7ffbd4160e50 2024-04-02T07:37:34.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.019+0000 7ffbda200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd40cb320 con 0x7ffbd40b80b0 2024-04-02T07:37:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.019+0000 7ffbd3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd40b3f80 0x7ffbd40b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.020+0000 7ffbd3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd40b3f80 0x7ffbd40b7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53912/0 (socket says 172.21.15.89:53912) 2024-04-02T07:37:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.020+0000 7ffbd3400700 1 -- 172.21.15.89:0/3686650516 learned_addr learned my addr 172.21.15.89:0/3686650516 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:37:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.020+0000 7ffbd2a00700 1 --2- 172.21.15.89:0/3686650516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbd4160e50 0x7ffbd40b77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.020+0000 7ffbd3e00700 1 --2- 172.21.15.89:0/3686650516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd40b80b0 0x7ffbd40b8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.020+0000 7ffbd3400700 1 -- 172.21.15.89:0/3686650516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd40b80b0 msgr2=0x7ffbd40b8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.020+0000 7ffbd3400700 1 --2- 172.21.15.89:0/3686650516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd40b80b0 0x7ffbd40b8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.020+0000 7ffbd3400700 1 -- 172.21.15.89:0/3686650516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbd4160e50 msgr2=0x7ffbd40b77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.020+0000 7ffbd3400700 1 --2- 172.21.15.89:0/3686650516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbd4160e50 0x7ffbd40b77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.020+0000 7ffbd3400700 1 -- 172.21.15.89:0/3686650516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc0009550 con 0x7ffbd40b3f80 2024-04-02T07:37:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.020+0000 7ffbd3400700 1 --2- 172.21.15.89:0/3686650516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd40b3f80 0x7ffbd40b7260 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ffbc400b4d0 tx=0x7ffbc400b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.020+0000 7ffbcb400700 1 -- 172.21.15.89:0/3686650516 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc400bea0 con 0x7ffbd40b3f80 2024-04-02T07:37:34.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.020+0000 7ffbcb400700 1 -- 172.21.15.89:0/3686650516 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc4011830 con 0x7ffbd40b3f80 2024-04-02T07:37:34.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.020+0000 7ffbda200700 1 -- 172.21.15.89:0/3686650516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd40b7ce0 con 0x7ffbd40b3f80 2024-04-02T07:37:34.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.020+0000 7ffbcb400700 1 -- 172.21.15.89:0/3686650516 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4011be0 con 0x7ffbd40b3f80 2024-04-02T07:37:34.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.021+0000 7ffbda200700 1 -- 172.21.15.89:0/3686650516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd40ca9f0 con 0x7ffbd40b3f80 2024-04-02T07:37:34.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.022+0000 7ffbda200700 1 -- 172.21.15.89:0/3686650516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd4067720 con 0x7ffbd40b3f80 2024-04-02T07:37:34.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.022+0000 7ffbcb400700 1 -- 172.21.15.89:0/3686650516 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 89313+0+0 (secure 0 0 0) 0x7ffbc4011d40 con 0x7ffbd40b3f80 2024-04-02T07:37:34.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.023+0000 7ffbcb400700 1 --2- 172.21.15.89:0/3686650516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffbb406ae90 0x7ffbb406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:37:34.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.023+0000 7ffbcb400700 1 -- 172.21.15.89:0/3686650516 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ffbc40896e0 con 0x7ffbd40b3f80 2024-04-02T07:37:34.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.023+0000 7ffbd2a00700 1 --2- 172.21.15.89:0/3686650516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffbb406ae90 0x7ffbb406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:37:34.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.024+0000 7ffbd2a00700 1 --2- 172.21.15.89:0/3686650516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffbb406ae90 0x7ffbb406d350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0000c00 tx=0x7ffbc0008030 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:37:34.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.027+0000 7ffbcb400700 1 -- 172.21.15.89:0/3686650516 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffbc4054550 con 0x7ffbd40b3f80 2024-04-02T07:37:34.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.259+0000 7ffbda200700 1 -- 172.21.15.89:0/3686650516 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbd40b0100 con 0x7ffbb406ae90 2024-04-02T07:37:34.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.325+0000 7ffbcb400700 1 -- 172.21.15.89:0/3686650516 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ffbc000bc70 con 0x7ffbb406ae90 2024-04-02T07:37:34.327 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule activated for path /snap_test_dir1/dir9 2024-04-02T07:37:34.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.328+0000 7ffbda200700 1 -- 172.21.15.89:0/3686650516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffbb406ae90 msgr2=0x7ffbb406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:34.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.328+0000 7ffbda200700 1 --2- 172.21.15.89:0/3686650516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffbb406ae90 0x7ffbb406d350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0000c00 tx=0x7ffbc0008030 comp rx=0 tx=0).stop 2024-04-02T07:37:34.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.328+0000 7ffbda200700 1 -- 172.21.15.89:0/3686650516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd40b3f80 msgr2=0x7ffbd40b7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:37:34.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.328+0000 7ffbda200700 1 --2- 172.21.15.89:0/3686650516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd40b3f80 0x7ffbd40b7260 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ffbc400b4d0 tx=0x7ffbc400b7e0 comp rx=0 tx=0).stop 2024-04-02T07:37:34.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.328+0000 7ffbda200700 1 -- 172.21.15.89:0/3686650516 shutdown_connections 2024-04-02T07:37:34.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.328+0000 7ffbda200700 1 --2- 172.21.15.89:0/3686650516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffbb406ae90 0x7ffbb406d350 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:34.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.328+0000 7ffbda200700 1 --2- 172.21.15.89:0/3686650516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd40b80b0 0x7ffbd40b8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:34.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.328+0000 7ffbda200700 1 --2- 172.21.15.89:0/3686650516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd40b3f80 0x7ffbd40b7260 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:34.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.328+0000 7ffbda200700 1 --2- 172.21.15.89:0/3686650516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbd4160e50 0x7ffbd40b77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:37:34.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.328+0000 7ffbda200700 1 -- 172.21.15.89:0/3686650516 >> 172.21.15.89:0/3686650516 conn(0x7ffbd408aff0 msgr2=0x7ffbd40b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:37:34.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.328+0000 7ffbda200700 1 -- 172.21.15.89:0/3686650516 shutdown_connections 2024-04-02T07:37:34.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:37:34.328+0000 7ffbda200700 1 -- 172.21.15.89:0/3686650516 wait complete. 2024-04-02T07:37:34.348 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2024-04-02T07:37:41.700 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:37:41.705 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:38:00.112 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:38:00.110+0000 7f34b2c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:38:00.112 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:38:00.110+0000 7f34b2c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:38:00.112 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:38:00.110+0000 7f34b2c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:38:00.113 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:38:00.110+0000 7f34b2c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:38:00.113 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:38:00.110+0000 7f34b2c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:38:11.830 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:38:11.835 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:38:41.945 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:38:41.951 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:39:12.060 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:39:12.065 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:39:42.174 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:39:42.179 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:40:12.293 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:40:12.298 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:40:42.404 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:40:42.409 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:41:12.566 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:41:12.571 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:41:42.671 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:41:42.676 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:42:00.230 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:42:00.228+0000 7f34a7800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:42:00.230 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:42:00.228+0000 7f34a7800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:42:00.230 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:42:00.228+0000 7f34a7800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:42:00.230 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:42:00.228+0000 7f34a7800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:42:00.230 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:42:00.228+0000 7f34a7800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:42:12.776 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:42:12.781 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:42:42.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:42:42.889 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:43:12.997 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:43:13.003 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:43:43.121 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:43:43.126 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:44:13.257 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:44:13.262 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:44:43.666 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:44:43.671 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:45:13.777 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:45:13.782 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:45:43.893 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:45:43.899 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:46:14.001 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:46:14.006 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:46:44.104 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:46:44.110 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:47:14.207 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:47:14.212 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:47:44.328 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:47:44.334 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:48:14.439 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:48:14.444 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:48:20.349 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2024-04-02T07:48:20.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.550+0000 7f21f0400700 1 Processor -- start 2024-04-02T07:48:20.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.550+0000 7f21f0400700 1 -- start start 2024-04-02T07:48:20.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.551+0000 7f21f0400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21e815dfe0 0x7f21e80ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:20.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.551+0000 7f21f0400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e80cb310 con 0x7f21e815a5c0 2024-04-02T07:48:20.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.551+0000 7f21f0400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e80711a0 con 0x7f21e80ce670 2024-04-02T07:48:20.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.551+0000 7f21f0400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e80ceb00 con 0x7f21e815dfe0 2024-04-02T07:48:20.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.551+0000 7f21eda00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f21e815a5c0 0x7f21e80709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60980/0 (socket says 172.21.15.89:60980) 2024-04-02T07:48:20.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.551+0000 7f21eda00700 1 -- 172.21.15.89:0/447616899 learned_addr learned my addr 172.21.15.89:0/447616899 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:20.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21ed000700 1 --2- 172.21.15.89:0/447616899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21e815dfe0 0x7f21e80ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:20.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 -- 172.21.15.89:0/447616899 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963502154 0 0) 0x7f21e80cb310 con 0x7f21e815a5c0 2024-04-02T07:48:20.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 -- 172.21.15.89:0/447616899 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21d00035b0 con 0x7f21e815a5c0 2024-04-02T07:48:20.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 -- 172.21.15.89:0/447616899 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2950049640 0 0) 0x7f21d00035b0 con 0x7f21e815a5c0 2024-04-02T07:48:20.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 -- 172.21.15.89:0/447616899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21e815dfe0 msgr2=0x7f21e80ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:20.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 --2- 172.21.15.89:0/447616899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21e815dfe0 0x7f21e80ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:20.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 -- 172.21.15.89:0/447616899 >> v1:172.21.15.113:6789/0 conn(0x7f21e80ce670 legacy=0x7f21e80cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:20.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 -- 172.21.15.89:0/447616899 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e8071480 con 0x7f21e815a5c0 2024-04-02T07:48:20.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 -- 172.21.15.89:0/447616899 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f21dc001670 con 0x7f21e815a5c0 2024-04-02T07:48:20.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 -- 172.21.15.89:0/447616899 >> v1:172.21.15.89:6789/0 conn(0x7f21e815a5c0 legacy=0x7f21e80709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:20.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 --2- 172.21.15.89:0/447616899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21e815dfe0 0x7f21d0003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:20.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 --2- 172.21.15.89:0/447616899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21d0003fb0 0x7f21d0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:20.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 --2- 172.21.15.89:0/447616899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0004d90 0x7f21d0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:20.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 -- 172.21.15.89:0/447616899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e80ceb00 con 0x7f21d0004d90 2024-04-02T07:48:20.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 -- 172.21.15.89:0/447616899 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d000baf0 con 0x7f21d0003fb0 2024-04-02T07:48:20.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21e7e00700 1 -- 172.21.15.89:0/447616899 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d000dec0 con 0x7f21e815dfe0 2024-04-02T07:48:20.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21ee400700 1 --2- 172.21.15.89:0/447616899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0004d90 0x7f21d0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:20.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21ed000700 1 --2- 172.21.15.89:0/447616899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21d0003fb0 0x7f21d0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:20.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.552+0000 7f21eda00700 1 --2- 172.21.15.89:0/447616899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21e815dfe0 0x7f21d0003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:20.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.553+0000 7f21ee400700 1 -- 172.21.15.89:0/447616899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21e815dfe0 msgr2=0x7f21d0003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:20.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.553+0000 7f21ee400700 1 --2- 172.21.15.89:0/447616899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21e815dfe0 0x7f21d0003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:20.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.553+0000 7f21ee400700 1 -- 172.21.15.89:0/447616899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21d0003fb0 msgr2=0x7f21d0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:20.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.553+0000 7f21ee400700 1 --2- 172.21.15.89:0/447616899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21d0003fb0 0x7f21d0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:20.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.553+0000 7f21ee400700 1 -- 172.21.15.89:0/447616899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d00035b0 con 0x7f21d0004d90 2024-04-02T07:48:20.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.553+0000 7f21ee400700 1 --2- 172.21.15.89:0/447616899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0004d90 0x7f21d0005210 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f21e000c240 tx=0x7f21d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c4060f97c562b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:20.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.553+0000 7f21e7e00700 1 -- 172.21.15.89:0/447616899 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0001df0 con 0x7f21d0004d90 2024-04-02T07:48:20.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.553+0000 7f21e7e00700 1 -- 172.21.15.89:0/447616899 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e000cda0 con 0x7f21d0004d90 2024-04-02T07:48:20.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.553+0000 7f21e7e00700 1 -- 172.21.15.89:0/447616899 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0003c40 con 0x7f21d0004d90 2024-04-02T07:48:20.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.554+0000 7f21f0400700 1 -- 172.21.15.89:0/447616899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0004d90 msgr2=0x7f21d0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:20.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.554+0000 7f21f0400700 1 --2- 172.21.15.89:0/447616899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0004d90 0x7f21d0005210 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f21e000c240 tx=0x7f21d0005880 comp rx=0 tx=0).stop 2024-04-02T07:48:20.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.554+0000 7f21ee400700 1 -- 172.21.15.89:0/447616899 reap_dead start 2024-04-02T07:48:20.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.554+0000 7f21f0400700 1 -- 172.21.15.89:0/447616899 shutdown_connections 2024-04-02T07:48:20.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.554+0000 7f21f0400700 1 -- 172.21.15.89:0/447616899 >> 172.21.15.89:0/447616899 conn(0x7f21e80b0d70 msgr2=0x7f21e80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:20.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.554+0000 7f21f0400700 1 -- 172.21.15.89:0/447616899 shutdown_connections 2024-04-02T07:48:20.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.554+0000 7f21f0400700 1 -- 172.21.15.89:0/447616899 wait complete. 2024-04-02T07:48:20.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.556+0000 7f21f0400700 1 Processor -- start 2024-04-02T07:48:20.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.556+0000 7f21f0400700 1 -- start start 2024-04-02T07:48:20.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21f0400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21d0004d90 0x7f21e80aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:20.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21f0400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21e80b06d0 0x7f21e814ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:20.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21f0400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21e8158c00 0x7f21e81590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:20.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21f0400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e81595f0 con 0x7f21e8158c00 2024-04-02T07:48:20.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21f0400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e814e2f0 con 0x7f21d0004d90 2024-04-02T07:48:20.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21f0400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e80b0b50 con 0x7f21e80b06d0 2024-04-02T07:48:20.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21ee400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21e8158c00 0x7f21e81590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:20.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21ee400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21e8158c00 0x7f21e81590b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41810/0 (socket says 172.21.15.89:41810) 2024-04-02T07:48:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21ee400700 1 -- 172.21.15.89:0/1980024337 learned_addr learned my addr 172.21.15.89:0/1980024337 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21ed000700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21e80b06d0 0x7f21e814ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21eda00700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21d0004d90 0x7f21e80aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21ee400700 1 -- 172.21.15.89:0/1980024337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21e80b06d0 msgr2=0x7f21e814ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21ee400700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21e80b06d0 0x7f21e814ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21ee400700 1 -- 172.21.15.89:0/1980024337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21d0004d90 msgr2=0x7f21e80aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21ee400700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21d0004d90 0x7f21e80aecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21ee400700 1 -- 172.21.15.89:0/1980024337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e81f3030 con 0x7f21e8158c00 2024-04-02T07:48:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21ed000700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21e80b06d0 0x7f21e814ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.557+0000 7f21eda00700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21d0004d90 0x7f21e80aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.558+0000 7f21ee400700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21e8158c00 0x7f21e81590b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f21e0005040 tx=0x7f21e00014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.558+0000 7f21e5600700 1 -- 172.21.15.89:0/1980024337 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0003c10 con 0x7f21e8158c00 2024-04-02T07:48:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.558+0000 7f21e5600700 1 -- 172.21.15.89:0/1980024337 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e0003d90 con 0x7f21e8158c00 2024-04-02T07:48:20.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.558+0000 7f21e5600700 1 -- 172.21.15.89:0/1980024337 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0005310 con 0x7f21e8158c00 2024-04-02T07:48:20.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.558+0000 7f21f0400700 1 -- 172.21.15.89:0/1980024337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e80af350 con 0x7f21e8158c00 2024-04-02T07:48:20.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.558+0000 7f21f0400700 1 -- 172.21.15.89:0/1980024337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e80af910 con 0x7f21e8158c00 2024-04-02T07:48:20.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.559+0000 7f21f0400700 1 -- 172.21.15.89:0/1980024337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e8067720 con 0x7f21e8158c00 2024-04-02T07:48:20.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.560+0000 7f21e5600700 1 -- 172.21.15.89:0/1980024337 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f21e0019070 con 0x7f21e8158c00 2024-04-02T07:48:20.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.561+0000 7f21e5600700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f21c806ae90 0x7f21c806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:20.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.561+0000 7f21e5600700 1 -- 172.21.15.89:0/1980024337 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f21e008a500 con 0x7f21e8158c00 2024-04-02T07:48:20.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.561+0000 7f21eda00700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f21c806ae90 0x7f21c806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:20.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.561+0000 7f21eda00700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f21c806ae90 0x7f21c806d350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0017a0 tx=0x7f21dc009040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:20.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.566+0000 7f21e5600700 1 -- 172.21.15.89:0/1980024337 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f21e00553f0 con 0x7f21e8158c00 2024-04-02T07:48:20.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.796+0000 7f21f0400700 1 -- 172.21.15.89:0/1980024337 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f21e80cb560 con 0x7f21c806ae90 2024-04-02T07:48:20.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.880+0000 7f21e5600700 1 -- 172.21.15.89:0/1980024337 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f21dc007030 con 0x7f21c806ae90 2024-04-02T07:48:20.883 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2024-04-02T07:48:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.883+0000 7f21f0400700 1 -- 172.21.15.89:0/1980024337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f21c806ae90 msgr2=0x7f21c806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.883+0000 7f21f0400700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f21c806ae90 0x7f21c806d350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0017a0 tx=0x7f21dc009040 comp rx=0 tx=0).stop 2024-04-02T07:48:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.883+0000 7f21f0400700 1 -- 172.21.15.89:0/1980024337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21e8158c00 msgr2=0x7f21e81590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.883+0000 7f21f0400700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21e8158c00 0x7f21e81590b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f21e0005040 tx=0x7f21e00014a0 comp rx=0 tx=0).stop 2024-04-02T07:48:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.883+0000 7f21f0400700 1 -- 172.21.15.89:0/1980024337 shutdown_connections 2024-04-02T07:48:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.883+0000 7f21f0400700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f21c806ae90 0x7f21c806d350 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:20.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.883+0000 7f21f0400700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21d0004d90 0x7f21e80aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:20.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.883+0000 7f21f0400700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21e8158c00 0x7f21e81590b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:20.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.883+0000 7f21f0400700 1 --2- 172.21.15.89:0/1980024337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21e80b06d0 0x7f21e814ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:20.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.883+0000 7f21f0400700 1 -- 172.21.15.89:0/1980024337 >> 172.21.15.89:0/1980024337 conn(0x7f21e80b0d70 msgr2=0x7f21e80cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:20.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.883+0000 7f21f0400700 1 -- 172.21.15.89:0/1980024337 shutdown_connections 2024-04-02T07:48:20.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:20.883+0000 7f21f0400700 1 -- 172.21.15.89:0/1980024337 wait complete. 2024-04-02T07:48:20.904 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2024-04-02T07:48:20.904 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2024-04-02T07:48:21.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.096+0000 7f0efa400700 1 Processor -- start 2024-04-02T07:48:21.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.096+0000 7f0efa400700 1 -- start start 2024-04-02T07:48:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.098+0000 7f0efa400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0ef40b4a50 0x7f0ef415b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.098+0000 7f0efa400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ef406eae0 con 0x7f0ef415af30 2024-04-02T07:48:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.098+0000 7f0efa400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ef406f660 con 0x7f0ef406da80 2024-04-02T07:48:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.098+0000 7f0efa400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ef406faf0 con 0x7f0ef40b4a50 2024-04-02T07:48:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.098+0000 7f0ef3400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0ef415af30 0x7f0ef406d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60990/0 (socket says 172.21.15.89:60990) 2024-04-02T07:48:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.098+0000 7f0ef3400700 1 -- 172.21.15.89:0/2928746007 learned_addr learned my addr 172.21.15.89:0/2928746007 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:21.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.098+0000 7f0ef3e00700 1 --2- 172.21.15.89:0/2928746007 >> v2:172.21.15.113:3300/0 conn(0x7f0ef40b4a50 0x7f0ef415b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:21.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.098+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032682705 0 0) 0x7f0ef406eae0 con 0x7f0ef415af30 2024-04-02T07:48:21.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ed80035b0 con 0x7f0ef415af30 2024-04-02T07:48:21.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044591217 0 0) 0x7f0ef406f660 con 0x7f0ef406da80 2024-04-02T07:48:21.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ef406eae0 con 0x7f0ef406da80 2024-04-02T07:48:21.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2940124732 0 0) 0x7f0ed80035b0 con 0x7f0ef415af30 2024-04-02T07:48:21.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 >> v2:172.21.15.113:3300/0 conn(0x7f0ef40b4a50 msgr2=0x7f0ef415b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:21.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 --2- 172.21.15.89:0/2928746007 >> v2:172.21.15.113:3300/0 conn(0x7f0ef40b4a50 0x7f0ef415b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 >> v1:172.21.15.113:6789/0 conn(0x7f0ef406da80 legacy=0x7f0ef40b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef406e150 con 0x7f0ef415af30 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f0ee4001670 con 0x7f0ef415af30 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 >> v1:172.21.15.89:6789/0 conn(0x7f0ef415af30 legacy=0x7f0ef406d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 --2- 172.21.15.89:0/2928746007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ed8003bc0 0x7f0ed8003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 --2- 172.21.15.89:0/2928746007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ed8004850 0x7f0ed8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 --2- 172.21.15.89:0/2928746007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ed8005210 0x7f0ed800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef406faf0 con 0x7f0ed8004850 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed800c160 con 0x7f0ed8003bc0 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed800c320 con 0x7f0ed8005210 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef2a00700 1 --2- 172.21.15.89:0/2928746007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ed8004850 0x7f0ed8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.100+0000 7f0ef3e00700 1 --2- 172.21.15.89:0/2928746007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ed8005210 0x7f0ed800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.099+0000 7f0ef3400700 1 --2- 172.21.15.89:0/2928746007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ed8003bc0 0x7f0ed8003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.100+0000 7f0ef2a00700 1 -- 172.21.15.89:0/2928746007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ed8005210 msgr2=0x7f0ed800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.100+0000 7f0ef2a00700 1 --2- 172.21.15.89:0/2928746007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ed8005210 0x7f0ed800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.100+0000 7f0ef2a00700 1 -- 172.21.15.89:0/2928746007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ed8003bc0 msgr2=0x7f0ed8003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:21.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.100+0000 7f0ef2a00700 1 --2- 172.21.15.89:0/2928746007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ed8003bc0 0x7f0ed8003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.100+0000 7f0ef2a00700 1 -- 172.21.15.89:0/2928746007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed80035b0 con 0x7f0ed8004850 2024-04-02T07:48:21.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.100+0000 7f0ef3400700 1 --2- 172.21.15.89:0/2928746007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ed8003bc0 0x7f0ed8003fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:48:21.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.100+0000 7f0ef3e00700 1 -- 172.21.15.89:0/2928746007 reap_dead start 2024-04-02T07:48:21.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.100+0000 7f0ef2a00700 1 --2- 172.21.15.89:0/2928746007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ed8004850 0x7f0ed8004cd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0001940 tx=0x7f0ed8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70854416b85eedaa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:21.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.100+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0001430 con 0x7f0ed8004850 2024-04-02T07:48:21.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.100+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee0001d70 con 0x7f0ed8004850 2024-04-02T07:48:21.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.100+0000 7f0ef2000700 1 -- 172.21.15.89:0/2928746007 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0005510 con 0x7f0ed8004850 2024-04-02T07:48:21.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.101+0000 7f0efa400700 1 -- 172.21.15.89:0/2928746007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ed8004850 msgr2=0x7f0ed8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:21.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.101+0000 7f0efa400700 1 --2- 172.21.15.89:0/2928746007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ed8004850 0x7f0ed8004cd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0001940 tx=0x7f0ed8005880 comp rx=0 tx=0).stop 2024-04-02T07:48:21.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.101+0000 7f0efa400700 1 -- 172.21.15.89:0/2928746007 shutdown_connections 2024-04-02T07:48:21.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.101+0000 7f0efa400700 1 --2- 172.21.15.89:0/2928746007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ed8004850 0x7f0ed8004cd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.101+0000 7f0efa400700 1 -- 172.21.15.89:0/2928746007 >> 172.21.15.89:0/2928746007 conn(0x7f0ef4071fb0 msgr2=0x7f0ef406a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:21.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.101+0000 7f0efa400700 1 -- 172.21.15.89:0/2928746007 shutdown_connections 2024-04-02T07:48:21.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.101+0000 7f0efa400700 1 -- 172.21.15.89:0/2928746007 wait complete. 2024-04-02T07:48:21.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.103+0000 7f0efa400700 1 Processor -- start 2024-04-02T07:48:21.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.103+0000 7f0efa400700 1 -- start start 2024-04-02T07:48:21.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0efa400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ed8004850 0x7f0ef415bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:21.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0efa400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ef40b0790 0x7f0ef40b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:21.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0efa400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ef414f3f0 0x7f0ef414f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:21.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0efa400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef40afc20 con 0x7f0ed8004850 2024-04-02T07:48:21.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0efa400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef40b1150 con 0x7f0ef414f3f0 2024-04-02T07:48:21.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0efa400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef4153690 con 0x7f0ef40b0790 2024-04-02T07:48:21.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0ef3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ed8004850 0x7f0ef415bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:21.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0ef3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ed8004850 0x7f0ef415bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41834/0 (socket says 172.21.15.89:41834) 2024-04-02T07:48:21.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0ef3400700 1 -- 172.21.15.89:0/2532716825 learned_addr learned my addr 172.21.15.89:0/2532716825 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:21.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0ef3e00700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ef414f3f0 0x7f0ef414f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:21.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0ef2a00700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ef40b0790 0x7f0ef40b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:21.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0ef3400700 1 -- 172.21.15.89:0/2532716825 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ef40b0790 msgr2=0x7f0ef40b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:21.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0ef3400700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ef40b0790 0x7f0ef40b0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0ef3400700 1 -- 172.21.15.89:0/2532716825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ef414f3f0 msgr2=0x7f0ef414f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:21.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0ef3400700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ef414f3f0 0x7f0ef414f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0ef3400700 1 -- 172.21.15.89:0/2532716825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef40b13a0 con 0x7f0ed8004850 2024-04-02T07:48:21.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.104+0000 7f0ef3e00700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ef414f3f0 0x7f0ef414f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:48:21.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.105+0000 7f0ef3400700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ed8004850 0x7f0ef415bd00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4001c20 tx=0x7f0ee400eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:21.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.105+0000 7f0eeb400700 1 -- 172.21.15.89:0/2532716825 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4012590 con 0x7f0ed8004850 2024-04-02T07:48:21.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.105+0000 7f0eeb400700 1 -- 172.21.15.89:0/2532716825 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee40126f0 con 0x7f0ed8004850 2024-04-02T07:48:21.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.105+0000 7f0efa400700 1 -- 172.21.15.89:0/2532716825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef4153930 con 0x7f0ed8004850 2024-04-02T07:48:21.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.105+0000 7f0eeb400700 1 -- 172.21.15.89:0/2532716825 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4012a80 con 0x7f0ed8004850 2024-04-02T07:48:21.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.105+0000 7f0efa400700 1 -- 172.21.15.89:0/2532716825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef40b1680 con 0x7f0ed8004850 2024-04-02T07:48:21.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.106+0000 7f0efa400700 1 -- 172.21.15.89:0/2532716825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef4073df0 con 0x7f0ed8004850 2024-04-02T07:48:21.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.107+0000 7f0eeb400700 1 -- 172.21.15.89:0/2532716825 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f0ee4012be0 con 0x7f0ed8004850 2024-04-02T07:48:21.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.108+0000 7f0eeb400700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0ed406ae90 0x7f0ed406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:21.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.108+0000 7f0eeb400700 1 -- 172.21.15.89:0/2532716825 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f0ee4089520 con 0x7f0ed8004850 2024-04-02T07:48:21.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.108+0000 7f0ef2a00700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0ed406ae90 0x7f0ed406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:21.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.109+0000 7f0ef2a00700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0ed406ae90 0x7f0ed406d350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0005040 tx=0x7f0ee000a000 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:21.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.113+0000 7f0eeb400700 1 -- 172.21.15.89:0/2532716825 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0ee4054260 con 0x7f0ed8004850 2024-04-02T07:48:21.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.339+0000 7f0efa400700 1 -- 172.21.15.89:0/2532716825 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ef406f2f0 con 0x7f0ed406ae90 2024-04-02T07:48:21.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.429+0000 7f0eeb400700 1 -- 172.21.15.89:0/2532716825 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f0ee0005150 con 0x7f0ed406ae90 2024-04-02T07:48:21.433 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2024-04-02T07:48:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.432+0000 7f0efa400700 1 -- 172.21.15.89:0/2532716825 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0ed406ae90 msgr2=0x7f0ed406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.432+0000 7f0efa400700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0ed406ae90 0x7f0ed406d350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0005040 tx=0x7f0ee000a000 comp rx=0 tx=0).stop 2024-04-02T07:48:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.432+0000 7f0efa400700 1 -- 172.21.15.89:0/2532716825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ed8004850 msgr2=0x7f0ef415bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.432+0000 7f0efa400700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ed8004850 0x7f0ef415bd00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4001c20 tx=0x7f0ee400eb10 comp rx=0 tx=0).stop 2024-04-02T07:48:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.432+0000 7f0efa400700 1 -- 172.21.15.89:0/2532716825 shutdown_connections 2024-04-02T07:48:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.432+0000 7f0efa400700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0ed406ae90 0x7f0ed406d350 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.432+0000 7f0efa400700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ef414f3f0 0x7f0ef414f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.432+0000 7f0efa400700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ed8004850 0x7f0ef415bd00 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.432+0000 7f0efa400700 1 --2- 172.21.15.89:0/2532716825 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ef40b0790 0x7f0ef40b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.432+0000 7f0efa400700 1 -- 172.21.15.89:0/2532716825 >> 172.21.15.89:0/2532716825 conn(0x7f0ef4071fb0 msgr2=0x7f0ef40b28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.432+0000 7f0efa400700 1 -- 172.21.15.89:0/2532716825 shutdown_connections 2024-04-02T07:48:21.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.432+0000 7f0efa400700 1 -- 172.21.15.89:0/2532716825 wait complete. 2024-04-02T07:48:21.452 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2024-04-02T07:48:21.453 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2024-04-02T07:48:21.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.642+0000 7fa6c1000700 1 Processor -- start 2024-04-02T07:48:21.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.643+0000 7fa6c1000700 1 -- start start 2024-04-02T07:48:21.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.644+0000 7fa6c1000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6bc1611b0 0x7fa6bc15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:21.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.644+0000 7fa6c1000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa6bc15f4d0 0x7fa6bc0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:21.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.644+0000 7fa6c1000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6bc0ce270 con 0x7fa6bc0d21f0 2024-04-02T07:48:21.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.644+0000 7fa6c1000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6bc15b600 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.644+0000 7fa6c1000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc0d2760 con 0x7fa6bc1611b0 2024-04-02T07:48:21.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.644+0000 7fa6b9400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa6bc15f4d0 0x7fa6bc0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:21.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.644+0000 7fa6b9400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa6bc15f4d0 0x7fa6bc0ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41842/0 (socket says 172.21.15.89:41842) 2024-04-02T07:48:21.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.644+0000 7fa6b9400700 1 -- 172.21.15.89:0/1265221150 learned_addr learned my addr 172.21.15.89:0/1265221150 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:21.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.644+0000 7fa6b9e00700 1 --2- 172.21.15.89:0/1265221150 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6bc1611b0 0x7fa6bc15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:21.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.644+0000 7fa6b9400700 1 -- 172.21.15.89:0/1265221150 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6bc1611b0 msgr2=0x7fa6bc15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:21.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.644+0000 7fa6b9400700 1 --2- 172.21.15.89:0/1265221150 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6bc1611b0 0x7fa6bc15ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.645+0000 7fa6b9400700 1 -- 172.21.15.89:0/1265221150 >> v1:172.21.15.113:6789/0 conn(0x7fa6bc0d21f0 legacy=0x7fa6bc161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:21.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.645+0000 7fa6b9400700 1 -- 172.21.15.89:0/1265221150 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc0d2a40 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.645+0000 7fa6b9400700 1 --2- 172.21.15.89:0/1265221150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6bc15f4d0 0x7fa6bc0ce700 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa6a80098f0 tx=0x7fa6a8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d2a96c6999be3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:21.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.645+0000 7fa6b8a00700 1 -- 172.21.15.89:0/1265221150 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a800b8a0 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.645+0000 7fa6b8a00700 1 -- 172.21.15.89:0/1265221150 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a800ba00 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.645+0000 7fa6b8a00700 1 -- 172.21.15.89:0/1265221150 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a800bd90 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.646+0000 7fa6c1000700 1 -- 172.21.15.89:0/1265221150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6bc15f4d0 msgr2=0x7fa6bc0ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:21.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.646+0000 7fa6c1000700 1 --2- 172.21.15.89:0/1265221150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6bc15f4d0 0x7fa6bc0ce700 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa6a80098f0 tx=0x7fa6a8009c00 comp rx=0 tx=0).stop 2024-04-02T07:48:21.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.646+0000 7fa6c1000700 1 -- 172.21.15.89:0/1265221150 shutdown_connections 2024-04-02T07:48:21.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.646+0000 7fa6c1000700 1 --2- 172.21.15.89:0/1265221150 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6bc1611b0 0x7fa6bc15ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.646+0000 7fa6c1000700 1 --2- 172.21.15.89:0/1265221150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6bc15f4d0 0x7fa6bc0ce700 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.646+0000 7fa6c1000700 1 -- 172.21.15.89:0/1265221150 >> 172.21.15.89:0/1265221150 conn(0x7fa6bc08aff0 msgr2=0x7fa6bc0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:21.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.646+0000 7fa6c1000700 1 -- 172.21.15.89:0/1265221150 shutdown_connections 2024-04-02T07:48:21.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.646+0000 7fa6c1000700 1 -- 172.21.15.89:0/1265221150 wait complete. 2024-04-02T07:48:21.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.647+0000 7fa6c1000700 1 Processor -- start 2024-04-02T07:48:21.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.648+0000 7fa6c1000700 1 -- start start 2024-04-02T07:48:21.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.648+0000 7fa6c1000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6bc0d21f0 0x7fa6bc156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:21.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.648+0000 7fa6c1000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6bc15f4d0 0x7fa6bc06be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:21.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.648+0000 7fa6c1000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6bc1611b0 0x7fa6bc06c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:21.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.648+0000 7fa6c1000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc0ca6a0 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.648+0000 7fa6c1000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc0ca980 con 0x7fa6bc1611b0 2024-04-02T07:48:21.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.649+0000 7fa6c1000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc0cac60 con 0x7fa6bc0d21f0 2024-04-02T07:48:21.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.649+0000 7fa6b9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6bc15f4d0 0x7fa6bc06be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:21.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.649+0000 7fa6b9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6bc15f4d0 0x7fa6bc06be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41848/0 (socket says 172.21.15.89:41848) 2024-04-02T07:48:21.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.649+0000 7fa6b9400700 1 -- 172.21.15.89:0/2539420510 learned_addr learned my addr 172.21.15.89:0/2539420510 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:21.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.649+0000 7fa6b9e00700 1 --2- 172.21.15.89:0/2539420510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6bc0d21f0 0x7fa6bc156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:21.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.649+0000 7fa6ba800700 1 --2- 172.21.15.89:0/2539420510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6bc1611b0 0x7fa6bc06c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:21.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.649+0000 7fa6b9400700 1 -- 172.21.15.89:0/2539420510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6bc0d21f0 msgr2=0x7fa6bc156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:21.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.649+0000 7fa6b9400700 1 --2- 172.21.15.89:0/2539420510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6bc0d21f0 0x7fa6bc156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.649+0000 7fa6b9400700 1 -- 172.21.15.89:0/2539420510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6bc1611b0 msgr2=0x7fa6bc06c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:21.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.649+0000 7fa6b9400700 1 --2- 172.21.15.89:0/2539420510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6bc1611b0 0x7fa6bc06c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.649+0000 7fa6b9400700 1 -- 172.21.15.89:0/2539420510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a8009550 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.649+0000 7fa6b9400700 1 --2- 172.21.15.89:0/2539420510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6bc15f4d0 0x7fa6bc06be80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8000c00 tx=0x7fa6a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:21.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.650+0000 7fa6b2000700 1 -- 172.21.15.89:0/2539420510 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8004300 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.650+0000 7fa6b2000700 1 -- 172.21.15.89:0/2539420510 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a801f070 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.650+0000 7fa6c1000700 1 -- 172.21.15.89:0/2539420510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6bc0caf40 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.650+0000 7fa6b2000700 1 -- 172.21.15.89:0/2539420510 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a80036a0 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.650+0000 7fa6c1000700 1 -- 172.21.15.89:0/2539420510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6bc0b7a70 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.651+0000 7fa6c1000700 1 -- 172.21.15.89:0/2539420510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6bc067720 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.652+0000 7fa6b2000700 1 -- 172.21.15.89:0/2539420510 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fa6a8021030 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.653+0000 7fa6b2000700 1 --2- 172.21.15.89:0/2539420510 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa69c06ae90 0x7fa69c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:21.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.653+0000 7fa6b2000700 1 -- 172.21.15.89:0/2539420510 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fa6a808ab30 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.653+0000 7fa6b9e00700 1 --2- 172.21.15.89:0/2539420510 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa69c06ae90 0x7fa69c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:21.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.654+0000 7fa6b9e00700 1 --2- 172.21.15.89:0/2539420510 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa69c06ae90 0x7fa69c06d350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac007a30 tx=0x7fa6ac008040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:21.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.657+0000 7fa6b2000700 1 -- 172.21.15.89:0/2539420510 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa6a8055920 con 0x7fa6bc15f4d0 2024-04-02T07:48:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.887+0000 7fa6c1000700 1 -- 172.21.15.89:0/2539420510 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6bc0caaf0 con 0x7fa69c06ae90 2024-04-02T07:48:21.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.976+0000 7fa6b2000700 1 -- 172.21.15.89:0/2539420510 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa6bc0caaf0 con 0x7fa69c06ae90 2024-04-02T07:48:21.980 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2024-04-02T07:48:21.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.979+0000 7fa6c1000700 1 -- 172.21.15.89:0/2539420510 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa69c06ae90 msgr2=0x7fa69c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:21.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.979+0000 7fa6c1000700 1 --2- 172.21.15.89:0/2539420510 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa69c06ae90 0x7fa69c06d350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac007a30 tx=0x7fa6ac008040 comp rx=0 tx=0).stop 2024-04-02T07:48:21.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.979+0000 7fa6c1000700 1 -- 172.21.15.89:0/2539420510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6bc15f4d0 msgr2=0x7fa6bc06be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:21.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.979+0000 7fa6c1000700 1 --2- 172.21.15.89:0/2539420510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6bc15f4d0 0x7fa6bc06be80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8000c00 tx=0x7fa6a8004750 comp rx=0 tx=0).stop 2024-04-02T07:48:21.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.979+0000 7fa6c1000700 1 -- 172.21.15.89:0/2539420510 shutdown_connections 2024-04-02T07:48:21.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.979+0000 7fa6c1000700 1 --2- 172.21.15.89:0/2539420510 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa69c06ae90 0x7fa69c06d350 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.979+0000 7fa6c1000700 1 --2- 172.21.15.89:0/2539420510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6bc1611b0 0x7fa6bc06c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.979+0000 7fa6c1000700 1 --2- 172.21.15.89:0/2539420510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6bc0d21f0 0x7fa6bc156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.979+0000 7fa6c1000700 1 --2- 172.21.15.89:0/2539420510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6bc15f4d0 0x7fa6bc06be80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:21.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.979+0000 7fa6c1000700 1 -- 172.21.15.89:0/2539420510 >> 172.21.15.89:0/2539420510 conn(0x7fa6bc08aff0 msgr2=0x7fa6bc0d1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:21.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.979+0000 7fa6c1000700 1 -- 172.21.15.89:0/2539420510 shutdown_connections 2024-04-02T07:48:21.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:21.979+0000 7fa6c1000700 1 -- 172.21.15.89:0/2539420510 wait complete. 2024-04-02T07:48:22.000 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2024-04-02T07:48:22.000 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2024-04-02T07:48:22.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.191+0000 7f1739000700 1 Processor -- start 2024-04-02T07:48:22.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.191+0000 7f1739000700 1 -- start start 2024-04-02T07:48:22.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.193+0000 7f1739000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f173415d9d0 0x7f17340d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:22.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.193+0000 7f1739000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17340d16a0 0x7f17340d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:22.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.193+0000 7f1739000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1734161420 con 0x7f17340d2370 2024-04-02T07:48:22.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.193+0000 7f1739000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17340d2820 con 0x7f173415d9d0 2024-04-02T07:48:22.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.193+0000 7f1739000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1734161700 con 0x7f17340d16a0 2024-04-02T07:48:22.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.193+0000 7f1732800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f17340d2370 0x7f173415ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60998/0 (socket says 172.21.15.89:60998) 2024-04-02T07:48:22.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.193+0000 7f1732800700 1 -- 172.21.15.89:0/337574832 learned_addr learned my addr 172.21.15.89:0/337574832 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:22.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.193+0000 7f1731400700 1 --2- 172.21.15.89:0/337574832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17340d16a0 0x7f17340d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:22.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.193+0000 7f1731e00700 1 --2- 172.21.15.89:0/337574832 >> v2:172.21.15.113:3300/0 conn(0x7f173415d9d0 0x7f17340d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:22.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.193+0000 7f1730a00700 1 -- 172.21.15.89:0/337574832 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550570768 0 0) 0x7f1734161420 con 0x7f17340d2370 2024-04-02T07:48:22.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.193+0000 7f1730a00700 1 -- 172.21.15.89:0/337574832 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17180035b0 con 0x7f17340d2370 2024-04-02T07:48:22.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.194+0000 7f1731400700 1 -- 172.21.15.89:0/337574832 >> v2:172.21.15.113:3300/0 conn(0x7f173415d9d0 msgr2=0x7f17340d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:22.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.194+0000 7f1731400700 1 --2- 172.21.15.89:0/337574832 >> v2:172.21.15.113:3300/0 conn(0x7f173415d9d0 0x7f17340d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.194+0000 7f1731400700 1 -- 172.21.15.89:0/337574832 >> v1:172.21.15.89:6789/0 conn(0x7f17340d2370 legacy=0x7f173415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:22.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.194+0000 7f1731400700 1 -- 172.21.15.89:0/337574832 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17341619e0 con 0x7f17340d16a0 2024-04-02T07:48:22.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.194+0000 7f1731400700 1 --2- 172.21.15.89:0/337574832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17340d16a0 0x7f17340d1e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1720009820 tx=0x7f1720009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ce87ea8f406cfcf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:22.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.194+0000 7f1730a00700 1 -- 172.21.15.89:0/337574832 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720005e30 con 0x7f17340d16a0 2024-04-02T07:48:22.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.194+0000 7f1730a00700 1 -- 172.21.15.89:0/337574832 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1720006030 con 0x7f17340d16a0 2024-04-02T07:48:22.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.194+0000 7f1730a00700 1 -- 172.21.15.89:0/337574832 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17200063c0 con 0x7f17340d16a0 2024-04-02T07:48:22.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.195+0000 7f1739000700 1 -- 172.21.15.89:0/337574832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17340d16a0 msgr2=0x7f17340d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:22.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.195+0000 7f1739000700 1 --2- 172.21.15.89:0/337574832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17340d16a0 0x7f17340d1e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1720009820 tx=0x7f1720009b30 comp rx=0 tx=0).stop 2024-04-02T07:48:22.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.195+0000 7f1739000700 1 -- 172.21.15.89:0/337574832 shutdown_connections 2024-04-02T07:48:22.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.195+0000 7f1739000700 1 --2- 172.21.15.89:0/337574832 >> v2:172.21.15.113:3300/0 conn(0x7f173415d9d0 0x7f17340d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.195+0000 7f1739000700 1 --2- 172.21.15.89:0/337574832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17340d16a0 0x7f17340d1e30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.195+0000 7f1739000700 1 -- 172.21.15.89:0/337574832 >> 172.21.15.89:0/337574832 conn(0x7f173408aff0 msgr2=0x7f17340d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:22.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.195+0000 7f1739000700 1 -- 172.21.15.89:0/337574832 shutdown_connections 2024-04-02T07:48:22.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.195+0000 7f1739000700 1 -- 172.21.15.89:0/337574832 wait complete. 2024-04-02T07:48:22.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.197+0000 7f1739000700 1 Processor -- start 2024-04-02T07:48:22.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.197+0000 7f1739000700 1 -- start start 2024-04-02T07:48:22.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1739000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17340d16a0 0x7f1734151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:22.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1739000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17340d2370 0x7f173415e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:22.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1739000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f173415d9d0 0x7f17341604a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:22.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1739000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173415e400 con 0x7f17340d16a0 2024-04-02T07:48:22.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1739000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17340b80f0 con 0x7f17340d2370 2024-04-02T07:48:22.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1739000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17340b83d0 con 0x7f173415d9d0 2024-04-02T07:48:22.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1731e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17340d16a0 0x7f1734151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:22.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1731e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17340d16a0 0x7f1734151550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41864/0 (socket says 172.21.15.89:41864) 2024-04-02T07:48:22.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1731e00700 1 -- 172.21.15.89:0/4171892640 learned_addr learned my addr 172.21.15.89:0/4171892640 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:22.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1731e00700 1 -- 172.21.15.89:0/4171892640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f173415d9d0 msgr2=0x7f17341604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:22.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1732800700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f173415d9d0 0x7f17341604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:22.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1731400700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17340d2370 0x7f173415e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:22.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1731e00700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f173415d9d0 0x7f17341604a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1731e00700 1 -- 172.21.15.89:0/4171892640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17340d2370 msgr2=0x7f173415e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:22.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1731e00700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17340d2370 0x7f173415e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1731e00700 1 -- 172.21.15.89:0/4171892640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1720009430 con 0x7f17340d16a0 2024-04-02T07:48:22.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1731400700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17340d2370 0x7f173415e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:22.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.198+0000 7f1732800700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f173415d9d0 0x7f17341604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:22.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.199+0000 7f1731e00700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17340d16a0 0x7f1734151550 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f172400e870 tx=0x7f172400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:22.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.199+0000 7f172a000700 1 -- 172.21.15.89:0/4171892640 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724009590 con 0x7f17340d16a0 2024-04-02T07:48:22.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.199+0000 7f172a000700 1 -- 172.21.15.89:0/4171892640 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172400ee70 con 0x7f17340d16a0 2024-04-02T07:48:22.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.199+0000 7f1739000700 1 -- 172.21.15.89:0/4171892640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173415ddb0 con 0x7f17340d16a0 2024-04-02T07:48:22.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.199+0000 7f172a000700 1 -- 172.21.15.89:0/4171892640 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172400ca50 con 0x7f17340d16a0 2024-04-02T07:48:22.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.199+0000 7f1739000700 1 -- 172.21.15.89:0/4171892640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1734151a90 con 0x7f17340d16a0 2024-04-02T07:48:22.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.200+0000 7f1739000700 1 -- 172.21.15.89:0/4171892640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17340d1ca0 con 0x7f17340d16a0 2024-04-02T07:48:22.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.201+0000 7f172a000700 1 -- 172.21.15.89:0/4171892640 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f172400cbb0 con 0x7f17340d16a0 2024-04-02T07:48:22.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.203+0000 7f172a000700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f171406ae90 0x7f171406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:22.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.203+0000 7f172a000700 1 -- 172.21.15.89:0/4171892640 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f17240898d0 con 0x7f17340d16a0 2024-04-02T07:48:22.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.203+0000 7f1731400700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f171406ae90 0x7f171406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:22.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.204+0000 7f1731400700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f171406ae90 0x7f171406d350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f17340d1ae0 tx=0x7f1720008070 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:22.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.207+0000 7f172a000700 1 -- 172.21.15.89:0/4171892640 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1724054610 con 0x7f17340d16a0 2024-04-02T07:48:22.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.440+0000 7f1739000700 1 -- 172.21.15.89:0/4171892640 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f17340b8540 con 0x7f171406ae90 2024-04-02T07:48:22.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.532+0000 7f172a000700 1 -- 172.21.15.89:0/4171892640 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f17340b8540 con 0x7f171406ae90 2024-04-02T07:48:22.536 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2024-04-02T07:48:22.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.535+0000 7f1739000700 1 -- 172.21.15.89:0/4171892640 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f171406ae90 msgr2=0x7f171406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:22.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.535+0000 7f1739000700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f171406ae90 0x7f171406d350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f17340d1ae0 tx=0x7f1720008070 comp rx=0 tx=0).stop 2024-04-02T07:48:22.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.535+0000 7f1739000700 1 -- 172.21.15.89:0/4171892640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17340d16a0 msgr2=0x7f1734151550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:22.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.535+0000 7f1739000700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17340d16a0 0x7f1734151550 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f172400e870 tx=0x7f172400ec30 comp rx=0 tx=0).stop 2024-04-02T07:48:22.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.535+0000 7f1739000700 1 -- 172.21.15.89:0/4171892640 shutdown_connections 2024-04-02T07:48:22.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.535+0000 7f1739000700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f171406ae90 0x7f171406d350 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.535+0000 7f1739000700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f173415d9d0 0x7f17341604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.535+0000 7f1739000700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17340d16a0 0x7f1734151550 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.535+0000 7f1739000700 1 --2- 172.21.15.89:0/4171892640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17340d2370 0x7f173415e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.535+0000 7f1739000700 1 -- 172.21.15.89:0/4171892640 >> 172.21.15.89:0/4171892640 conn(0x7f173408aff0 msgr2=0x7f17340b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:22.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.535+0000 7f1739000700 1 -- 172.21.15.89:0/4171892640 shutdown_connections 2024-04-02T07:48:22.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.535+0000 7f1739000700 1 -- 172.21.15.89:0/4171892640 wait complete. 2024-04-02T07:48:22.556 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2024-04-02T07:48:22.556 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2024-04-02T07:48:22.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.747+0000 7f69cac00700 1 Processor -- start 2024-04-02T07:48:22.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.748+0000 7f69cac00700 1 -- start start 2024-04-02T07:48:22.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69cac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c40cbb30 0x7f69c40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:22.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69cac00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f69c40cd020 0x7f69c40cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:22.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69cac00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f69c40b2070 0x7f69c40b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:22.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69cac00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c40cc450 con 0x7f69c40cd020 2024-04-02T07:48:22.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69cac00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c40cdae0 con 0x7f69c40b2070 2024-04-02T07:48:22.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69cac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c40cddc0 con 0x7f69c40cbb30 2024-04-02T07:48:22.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69c3400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f69c40cd020 0x7f69c40cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:22.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69c3400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f69c40cd020 0x7f69c40cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41866/0 (socket says 172.21.15.89:41866) 2024-04-02T07:48:22.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69c3400700 1 -- 172.21.15.89:0/2926174837 learned_addr learned my addr 172.21.15.89:0/2926174837 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:22.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69c3e00700 1 --2- 172.21.15.89:0/2926174837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c40cbb30 0x7f69c40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:22.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69c8c00700 1 --2- 172.21.15.89:0/2926174837 >> v2:172.21.15.113:3300/0 conn(0x7f69c40b2070 0x7f69c40b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:22.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69c3400700 1 -- 172.21.15.89:0/2926174837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c40cbb30 msgr2=0x7f69c40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:22.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69c3400700 1 --2- 172.21.15.89:0/2926174837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c40cbb30 0x7f69c40cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69c3400700 1 -- 172.21.15.89:0/2926174837 >> v2:172.21.15.113:3300/0 conn(0x7f69c40b2070 msgr2=0x7f69c40b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:22.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69c3400700 1 --2- 172.21.15.89:0/2926174837 >> v2:172.21.15.113:3300/0 conn(0x7f69c40b2070 0x7f69c40b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69c3400700 1 -- 172.21.15.89:0/2926174837 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c40ce0a0 con 0x7f69c40cd020 2024-04-02T07:48:22.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69c8c00700 1 --2- 172.21.15.89:0/2926174837 >> v2:172.21.15.113:3300/0 conn(0x7f69c40b2070 0x7f69c40b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:22.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.749+0000 7f69c3e00700 1 --2- 172.21.15.89:0/2926174837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c40cbb30 0x7f69c40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:22.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.750+0000 7f69c3400700 1 --2- 172.21.15.89:0/2926174837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c40cd020 0x7f69c40cd4a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f69c406c9f0 tx=0x7f69b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e18a274ee428dff server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:22.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.750+0000 7f69c2a00700 1 -- 172.21.15.89:0/2926174837 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b000b870 con 0x7f69c40cd020 2024-04-02T07:48:22.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.750+0000 7f69c2a00700 1 -- 172.21.15.89:0/2926174837 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b000b9d0 con 0x7f69c40cd020 2024-04-02T07:48:22.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.750+0000 7f69c2a00700 1 -- 172.21.15.89:0/2926174837 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b000bd60 con 0x7f69c40cd020 2024-04-02T07:48:22.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.750+0000 7f69cac00700 1 -- 172.21.15.89:0/2926174837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c40cd020 msgr2=0x7f69c40cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:22.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.750+0000 7f69cac00700 1 --2- 172.21.15.89:0/2926174837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c40cd020 0x7f69c40cd4a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f69c406c9f0 tx=0x7f69b0009bd0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.750+0000 7f69cac00700 1 -- 172.21.15.89:0/2926174837 shutdown_connections 2024-04-02T07:48:22.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.750+0000 7f69cac00700 1 --2- 172.21.15.89:0/2926174837 >> v2:172.21.15.113:3300/0 conn(0x7f69c40b2070 0x7f69c40b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.750+0000 7f69cac00700 1 --2- 172.21.15.89:0/2926174837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c40cbb30 0x7f69c40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.750+0000 7f69cac00700 1 --2- 172.21.15.89:0/2926174837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c40cd020 0x7f69c40cd4a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.750+0000 7f69cac00700 1 -- 172.21.15.89:0/2926174837 >> 172.21.15.89:0/2926174837 conn(0x7f69c40b0d70 msgr2=0x7f69c40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:22.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.751+0000 7f69cac00700 1 -- 172.21.15.89:0/2926174837 shutdown_connections 2024-04-02T07:48:22.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.751+0000 7f69cac00700 1 -- 172.21.15.89:0/2926174837 wait complete. 2024-04-02T07:48:22.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.752+0000 7f69cac00700 1 Processor -- start 2024-04-02T07:48:22.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.752+0000 7f69cac00700 1 -- start start 2024-04-02T07:48:22.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.753+0000 7f69cac00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c40b2070 0x7f69c4156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:22.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.753+0000 7f69cac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c40cbb30 0x7f69c415d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:22.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.753+0000 7f69cac00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c40cd020 0x7f69c4158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:22.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.753+0000 7f69cac00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c415dbb0 con 0x7f69c40b2070 2024-04-02T07:48:22.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.753+0000 7f69cac00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c4154630 con 0x7f69c40cd020 2024-04-02T07:48:22.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.753+0000 7f69cac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c415cbd0 con 0x7f69c40cbb30 2024-04-02T07:48:22.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69c3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c40b2070 0x7f69c4156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:22.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69c3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c40b2070 0x7f69c4156480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41878/0 (socket says 172.21.15.89:41878) 2024-04-02T07:48:22.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69c3e00700 1 -- 172.21.15.89:0/426759323 learned_addr learned my addr 172.21.15.89:0/426759323 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:22.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69c3400700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c40cbb30 0x7f69c415d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:22.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69c8c00700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c40cd020 0x7f69c4158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:22.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69c3e00700 1 -- 172.21.15.89:0/426759323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c40cbb30 msgr2=0x7f69c415d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:22.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69c3e00700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c40cbb30 0x7f69c415d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69c3e00700 1 -- 172.21.15.89:0/426759323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c40cd020 msgr2=0x7f69c4158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:22.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69c3e00700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c40cd020 0x7f69c4158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:22.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69c3e00700 1 -- 172.21.15.89:0/426759323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b0009550 con 0x7f69c40b2070 2024-04-02T07:48:22.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69c3400700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c40cbb30 0x7f69c415d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:22.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69c8c00700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c40cd020 0x7f69c4158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:22.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69c3e00700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c40b2070 0x7f69c4156480 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f69b400c7f0 tx=0x7f69b400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:22.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69bbe00700 1 -- 172.21.15.89:0/426759323 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4011800 con 0x7f69c40b2070 2024-04-02T07:48:22.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69bbe00700 1 -- 172.21.15.89:0/426759323 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b4011960 con 0x7f69c40b2070 2024-04-02T07:48:22.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69cac00700 1 -- 172.21.15.89:0/426759323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c415ce90 con 0x7f69c40b2070 2024-04-02T07:48:22.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.754+0000 7f69bbe00700 1 -- 172.21.15.89:0/426759323 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4011d10 con 0x7f69c40b2070 2024-04-02T07:48:22.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.755+0000 7f69cac00700 1 -- 172.21.15.89:0/426759323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c4154b60 con 0x7f69c40b2070 2024-04-02T07:48:22.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.756+0000 7f69cac00700 1 -- 172.21.15.89:0/426759323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c406eb30 con 0x7f69c40b2070 2024-04-02T07:48:22.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.757+0000 7f69bbe00700 1 -- 172.21.15.89:0/426759323 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f69b4018360 con 0x7f69c40b2070 2024-04-02T07:48:22.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.757+0000 7f69bbe00700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69a406ae90 0x7f69a406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:22.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.757+0000 7f69bbe00700 1 -- 172.21.15.89:0/426759323 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f69b4089930 con 0x7f69c40b2070 2024-04-02T07:48:22.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.758+0000 7f69c3400700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69a406ae90 0x7f69a406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:22.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.758+0000 7f69c3400700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69a406ae90 0x7f69a406d350 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f69c40b0140 tx=0x7f69b0008030 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:22.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.762+0000 7f69bbe00700 1 -- 172.21.15.89:0/426759323 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f69b4054560 con 0x7f69c40b2070 2024-04-02T07:48:22.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:22.992+0000 7f69cac00700 1 -- 172.21.15.89:0/426759323 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f69c41594a0 con 0x7f69a406ae90 2024-04-02T07:48:23.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.084+0000 7f69bbe00700 1 -- 172.21.15.89:0/426759323 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f69b000bc40 con 0x7f69a406ae90 2024-04-02T07:48:23.088 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2024-04-02T07:48:23.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.087+0000 7f69cac00700 1 -- 172.21.15.89:0/426759323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69a406ae90 msgr2=0x7f69a406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:23.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.087+0000 7f69cac00700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69a406ae90 0x7f69a406d350 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f69c40b0140 tx=0x7f69b0008030 comp rx=0 tx=0).stop 2024-04-02T07:48:23.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.087+0000 7f69cac00700 1 -- 172.21.15.89:0/426759323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c40b2070 msgr2=0x7f69c4156480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:23.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.087+0000 7f69cac00700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c40b2070 0x7f69c4156480 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f69b400c7f0 tx=0x7f69b400cbb0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.087+0000 7f69cac00700 1 -- 172.21.15.89:0/426759323 shutdown_connections 2024-04-02T07:48:23.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.087+0000 7f69cac00700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69a406ae90 0x7f69a406d350 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.087+0000 7f69cac00700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c40cd020 0x7f69c4158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.087+0000 7f69cac00700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c40b2070 0x7f69c4156480 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.087+0000 7f69cac00700 1 --2- 172.21.15.89:0/426759323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c40cbb30 0x7f69c415d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.087+0000 7f69cac00700 1 -- 172.21.15.89:0/426759323 >> 172.21.15.89:0/426759323 conn(0x7f69c40b0d70 msgr2=0x7f69c406e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:23.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.087+0000 7f69cac00700 1 -- 172.21.15.89:0/426759323 shutdown_connections 2024-04-02T07:48:23.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.087+0000 7f69cac00700 1 -- 172.21.15.89:0/426759323 wait complete. 2024-04-02T07:48:23.108 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2024-04-02T07:48:23.108 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2024-04-02T07:48:23.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.299+0000 7f06d6800700 1 Processor -- start 2024-04-02T07:48:23.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.299+0000 7f06d6800700 1 -- start start 2024-04-02T07:48:23.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.300+0000 7f06d6800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f06d015d9d0 0x7f06d00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:23.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.300+0000 7f06d6800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f06d00d16a0 0x7f06d00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:23.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.301+0000 7f06d6800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d0161420 con 0x7f06d00d2370 2024-04-02T07:48:23.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.301+0000 7f06d6800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d00d27e0 con 0x7f06d015d9d0 2024-04-02T07:48:23.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.301+0000 7f06d6800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d0161700 con 0x7f06d00d16a0 2024-04-02T07:48:23.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.301+0000 7f06cf400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f06d015d9d0 0x7f06d00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:23.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.301+0000 7f06cf400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f06d015d9d0 0x7f06d00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41888/0 (socket says 172.21.15.89:41888) 2024-04-02T07:48:23.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.301+0000 7f06cf400700 1 -- 172.21.15.89:0/2634263736 learned_addr learned my addr 172.21.15.89:0/2634263736 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:23.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.301+0000 7f06cea00700 1 --2- 172.21.15.89:0/2634263736 >> v2:172.21.15.113:3300/0 conn(0x7f06d00d16a0 0x7f06d00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:23.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.301+0000 7f06cf400700 1 -- 172.21.15.89:0/2634263736 >> v2:172.21.15.113:3300/0 conn(0x7f06d00d16a0 msgr2=0x7f06d00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:23.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.301+0000 7f06cf400700 1 --2- 172.21.15.89:0/2634263736 >> v2:172.21.15.113:3300/0 conn(0x7f06d00d16a0 0x7f06d00d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.301+0000 7f06cf400700 1 -- 172.21.15.89:0/2634263736 >> v1:172.21.15.113:6789/0 conn(0x7f06d00d2370 legacy=0x7f06d015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:23.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.301+0000 7f06cf400700 1 -- 172.21.15.89:0/2634263736 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d01619e0 con 0x7f06d015d9d0 2024-04-02T07:48:23.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.301+0000 7f06cea00700 1 --2- 172.21.15.89:0/2634263736 >> v2:172.21.15.113:3300/0 conn(0x7f06d00d16a0 0x7f06d00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:48:23.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.301+0000 7f06cf400700 1 --2- 172.21.15.89:0/2634263736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06d015d9d0 0x7f06d00d3c30 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f06d00d12f0 tx=0x7f06c0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71380072e26cb69f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:23.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.302+0000 7f06ce000700 1 -- 172.21.15.89:0/2634263736 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c000b870 con 0x7f06d015d9d0 2024-04-02T07:48:23.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.302+0000 7f06ce000700 1 -- 172.21.15.89:0/2634263736 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c000b9d0 con 0x7f06d015d9d0 2024-04-02T07:48:23.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.302+0000 7f06ce000700 1 -- 172.21.15.89:0/2634263736 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c000bd60 con 0x7f06d015d9d0 2024-04-02T07:48:23.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.302+0000 7f06d6800700 1 -- 172.21.15.89:0/2634263736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06d015d9d0 msgr2=0x7f06d00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:23.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.302+0000 7f06d6800700 1 --2- 172.21.15.89:0/2634263736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06d015d9d0 0x7f06d00d3c30 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f06d00d12f0 tx=0x7f06c0009bd0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.302+0000 7f06d6800700 1 -- 172.21.15.89:0/2634263736 shutdown_connections 2024-04-02T07:48:23.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.302+0000 7f06d6800700 1 --2- 172.21.15.89:0/2634263736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06d015d9d0 0x7f06d00d3c30 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.302+0000 7f06d6800700 1 --2- 172.21.15.89:0/2634263736 >> v2:172.21.15.113:3300/0 conn(0x7f06d00d16a0 0x7f06d00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.302+0000 7f06d6800700 1 -- 172.21.15.89:0/2634263736 >> 172.21.15.89:0/2634263736 conn(0x7f06d008aff0 msgr2=0x7f06d00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:23.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.302+0000 7f06d6800700 1 -- 172.21.15.89:0/2634263736 shutdown_connections 2024-04-02T07:48:23.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.302+0000 7f06d6800700 1 -- 172.21.15.89:0/2634263736 wait complete. 2024-04-02T07:48:23.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.304+0000 7f06d6800700 1 Processor -- start 2024-04-02T07:48:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.304+0000 7f06d6800700 1 -- start start 2024-04-02T07:48:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.305+0000 7f06d6800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06d00d16a0 0x7f06d00b7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.305+0000 7f06d6800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06d00d2370 0x7f06d01580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.305+0000 7f06d6800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06d015d9d0 0x7f06d00b8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.305+0000 7f06d6800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d00b6fd0 con 0x7f06d00d2370 2024-04-02T07:48:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.305+0000 7f06d6800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d00b79a0 con 0x7f06d00d16a0 2024-04-02T07:48:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.305+0000 7f06d6800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d00b63c0 con 0x7f06d015d9d0 2024-04-02T07:48:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.305+0000 7f06cea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06d00d2370 0x7f06d01580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.305+0000 7f06cea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06d00d2370 0x7f06d01580b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41902/0 (socket says 172.21.15.89:41902) 2024-04-02T07:48:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.305+0000 7f06cea00700 1 -- 172.21.15.89:0/3694006856 learned_addr learned my addr 172.21.15.89:0/3694006856 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.305+0000 7f06cfe00700 1 --2- 172.21.15.89:0/3694006856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06d015d9d0 0x7f06d00b8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.305+0000 7f06cf400700 1 --2- 172.21.15.89:0/3694006856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06d00d16a0 0x7f06d00b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:23.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.306+0000 7f06cea00700 1 -- 172.21.15.89:0/3694006856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06d015d9d0 msgr2=0x7f06d00b8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:23.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.306+0000 7f06cea00700 1 --2- 172.21.15.89:0/3694006856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06d015d9d0 0x7f06d00b8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.306+0000 7f06cea00700 1 -- 172.21.15.89:0/3694006856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06d00d16a0 msgr2=0x7f06d00b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:23.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.306+0000 7f06cea00700 1 --2- 172.21.15.89:0/3694006856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06d00d16a0 0x7f06d00b7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.306+0000 7f06cea00700 1 -- 172.21.15.89:0/3694006856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c0009550 con 0x7f06d00d2370 2024-04-02T07:48:23.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.306+0000 7f06cea00700 1 --2- 172.21.15.89:0/3694006856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06d00d2370 0x7f06d01580b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00e810 tx=0x7f06bc00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:23.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.306+0000 7f06c7400700 1 -- 172.21.15.89:0/3694006856 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc00c070 con 0x7f06d00d2370 2024-04-02T07:48:23.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.306+0000 7f06c7400700 1 -- 172.21.15.89:0/3694006856 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06bc005c20 con 0x7f06d00d2370 2024-04-02T07:48:23.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.306+0000 7f06c7400700 1 -- 172.21.15.89:0/3694006856 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc006030 con 0x7f06d00d2370 2024-04-02T07:48:23.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.306+0000 7f06d6800700 1 -- 172.21.15.89:0/3694006856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d00b66a0 con 0x7f06d00d2370 2024-04-02T07:48:23.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.306+0000 7f06d6800700 1 -- 172.21.15.89:0/3694006856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d00b6c60 con 0x7f06d00d2370 2024-04-02T07:48:23.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.307+0000 7f06d6800700 1 -- 172.21.15.89:0/3694006856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d00d1ca0 con 0x7f06d00d2370 2024-04-02T07:48:23.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.308+0000 7f06c7400700 1 -- 172.21.15.89:0/3694006856 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f06bc006190 con 0x7f06d00d2370 2024-04-02T07:48:23.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.309+0000 7f06c7400700 1 --2- 172.21.15.89:0/3694006856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f06b006ae90 0x7f06b006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:23.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.309+0000 7f06c7400700 1 -- 172.21.15.89:0/3694006856 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f06bc089660 con 0x7f06d00d2370 2024-04-02T07:48:23.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.309+0000 7f06cf400700 1 --2- 172.21.15.89:0/3694006856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f06b006ae90 0x7f06b006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:23.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.310+0000 7f06cf400700 1 --2- 172.21.15.89:0/3694006856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f06b006ae90 0x7f06b006d350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f06c0014040 tx=0x7f06c0008030 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:23.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.314+0000 7f06c7400700 1 -- 172.21.15.89:0/3694006856 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f06bc054290 con 0x7f06d00d2370 2024-04-02T07:48:23.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.545+0000 7f06d6800700 1 -- 172.21.15.89:0/3694006856 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f06d0161fa0 con 0x7f06b006ae90 2024-04-02T07:48:23.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.631+0000 7f06c7400700 1 -- 172.21.15.89:0/3694006856 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f06c000bb40 con 0x7f06b006ae90 2024-04-02T07:48:23.634 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2024-04-02T07:48:23.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.633+0000 7f06d6800700 1 -- 172.21.15.89:0/3694006856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f06b006ae90 msgr2=0x7f06b006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:23.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.633+0000 7f06d6800700 1 --2- 172.21.15.89:0/3694006856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f06b006ae90 0x7f06b006d350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f06c0014040 tx=0x7f06c0008030 comp rx=0 tx=0).stop 2024-04-02T07:48:23.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.633+0000 7f06d6800700 1 -- 172.21.15.89:0/3694006856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06d00d2370 msgr2=0x7f06d01580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:23.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.633+0000 7f06d6800700 1 --2- 172.21.15.89:0/3694006856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06d00d2370 0x7f06d01580b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00e810 tx=0x7f06bc00eb20 comp rx=0 tx=0).stop 2024-04-02T07:48:23.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.633+0000 7f06d6800700 1 -- 172.21.15.89:0/3694006856 shutdown_connections 2024-04-02T07:48:23.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.633+0000 7f06d6800700 1 --2- 172.21.15.89:0/3694006856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f06b006ae90 0x7f06b006d350 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.633+0000 7f06d6800700 1 --2- 172.21.15.89:0/3694006856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06d00d16a0 0x7f06d00b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.633+0000 7f06d6800700 1 --2- 172.21.15.89:0/3694006856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06d015d9d0 0x7f06d00b8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.633+0000 7f06d6800700 1 --2- 172.21.15.89:0/3694006856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06d00d2370 0x7f06d01580b0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.633+0000 7f06d6800700 1 -- 172.21.15.89:0/3694006856 >> 172.21.15.89:0/3694006856 conn(0x7f06d008aff0 msgr2=0x7f06d015bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:23.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.634+0000 7f06d6800700 1 -- 172.21.15.89:0/3694006856 shutdown_connections 2024-04-02T07:48:23.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.634+0000 7f06d6800700 1 -- 172.21.15.89:0/3694006856 wait complete. 2024-04-02T07:48:23.655 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2024-04-02T07:48:23.655 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2024-04-02T07:48:23.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.846+0000 7fb215200700 1 Processor -- start 2024-04-02T07:48:23.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.846+0000 7fb215200700 1 -- start start 2024-04-02T07:48:23.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.847+0000 7fb215200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb21015d9d0 0x7fb2100d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:23.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.847+0000 7fb215200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2100d2540 0x7fb2100ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:23.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.847+0000 7fb215200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb210161420 con 0x7fb2100d16a0 2024-04-02T07:48:23.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.847+0000 7fb215200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21015edc0 con 0x7fb21015d9d0 2024-04-02T07:48:23.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.847+0000 7fb215200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb210161700 con 0x7fb2100d2540 2024-04-02T07:48:23.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.847+0000 7fb20e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2100d2540 0x7fb2100ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:23.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.847+0000 7fb20d400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb2100d16a0 0x7fb2100d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:59402/0 (socket says 172.21.15.89:59402) 2024-04-02T07:48:23.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.847+0000 7fb20d400700 1 -- 172.21.15.89:0/3753888984 learned_addr learned my addr 172.21.15.89:0/3753888984 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:23.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.847+0000 7fb20de00700 1 --2- 172.21.15.89:0/3753888984 >> v2:172.21.15.113:3300/0 conn(0x7fb21015d9d0 0x7fb2100d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:23.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.848+0000 7fb20e800700 1 -- 172.21.15.89:0/3753888984 >> v2:172.21.15.113:3300/0 conn(0x7fb21015d9d0 msgr2=0x7fb2100d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:23.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.848+0000 7fb20e800700 1 --2- 172.21.15.89:0/3753888984 >> v2:172.21.15.113:3300/0 conn(0x7fb21015d9d0 0x7fb2100d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.848+0000 7fb20ca00700 1 -- 172.21.15.89:0/3753888984 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716489452 0 0) 0x7fb210161420 con 0x7fb2100d16a0 2024-04-02T07:48:23.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.848+0000 7fb20e800700 1 -- 172.21.15.89:0/3753888984 >> v1:172.21.15.113:6789/0 conn(0x7fb2100d16a0 legacy=0x7fb2100d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.848+0000 7fb20e800700 1 -- 172.21.15.89:0/3753888984 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2101619e0 con 0x7fb2100d2540 2024-04-02T07:48:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.848+0000 7fb20de00700 1 --2- 172.21.15.89:0/3753888984 >> v2:172.21.15.113:3300/0 conn(0x7fb21015d9d0 0x7fb2100d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:48:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.848+0000 7fb20e800700 1 --2- 172.21.15.89:0/3753888984 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2100d2540 0x7fb2100ce360 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb208009d90 tx=0x7fb20800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1a3d1cf6dbb3c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.848+0000 7fb20ca00700 1 -- 172.21.15.89:0/3753888984 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20800e070 con 0x7fb2100d2540 2024-04-02T07:48:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.848+0000 7fb20ca00700 1 -- 172.21.15.89:0/3753888984 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20800bb20 con 0x7fb2100d2540 2024-04-02T07:48:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.848+0000 7fb20ca00700 1 -- 172.21.15.89:0/3753888984 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208012430 con 0x7fb2100d2540 2024-04-02T07:48:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.849+0000 7fb215200700 1 -- 172.21.15.89:0/3753888984 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2100d2540 msgr2=0x7fb2100ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.849+0000 7fb215200700 1 --2- 172.21.15.89:0/3753888984 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2100d2540 0x7fb2100ce360 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb208009d90 tx=0x7fb20800b330 comp rx=0 tx=0).stop 2024-04-02T07:48:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.849+0000 7fb215200700 1 -- 172.21.15.89:0/3753888984 shutdown_connections 2024-04-02T07:48:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.849+0000 7fb215200700 1 --2- 172.21.15.89:0/3753888984 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2100d2540 0x7fb2100ce360 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.849+0000 7fb215200700 1 --2- 172.21.15.89:0/3753888984 >> v2:172.21.15.113:3300/0 conn(0x7fb21015d9d0 0x7fb2100d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.849+0000 7fb215200700 1 -- 172.21.15.89:0/3753888984 >> 172.21.15.89:0/3753888984 conn(0x7fb21008aff0 msgr2=0x7fb2100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.849+0000 7fb215200700 1 -- 172.21.15.89:0/3753888984 shutdown_connections 2024-04-02T07:48:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.849+0000 7fb215200700 1 -- 172.21.15.89:0/3753888984 wait complete. 2024-04-02T07:48:23.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.851+0000 7fb215200700 1 Processor -- start 2024-04-02T07:48:23.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.851+0000 7fb215200700 1 -- start start 2024-04-02T07:48:23.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.852+0000 7fb215200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2100d16a0 0x7fb210155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.852+0000 7fb215200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2100d2540 0x7fb2100b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.852+0000 7fb215200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb21015d9d0 0x7fb21015ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.852+0000 7fb215200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb210155fb0 con 0x7fb21015d9d0 2024-04-02T07:48:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.852+0000 7fb215200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21015f880 con 0x7fb2100d16a0 2024-04-02T07:48:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.852+0000 7fb215200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21015fb60 con 0x7fb2100d2540 2024-04-02T07:48:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.852+0000 7fb20e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb21015d9d0 0x7fb21015ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.852+0000 7fb20de00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2100d16a0 0x7fb210155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.852+0000 7fb20e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb21015d9d0 0x7fb21015ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41912/0 (socket says 172.21.15.89:41912) 2024-04-02T07:48:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.852+0000 7fb20e800700 1 -- 172.21.15.89:0/370316516 learned_addr learned my addr 172.21.15.89:0/370316516 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.852+0000 7fb20d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2100d2540 0x7fb2100b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.853+0000 7fb20e800700 1 -- 172.21.15.89:0/370316516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2100d2540 msgr2=0x7fb2100b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.853+0000 7fb20e800700 1 --2- 172.21.15.89:0/370316516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2100d2540 0x7fb2100b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.853+0000 7fb20e800700 1 -- 172.21.15.89:0/370316516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2100d16a0 msgr2=0x7fb210155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:23.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.853+0000 7fb20e800700 1 --2- 172.21.15.89:0/370316516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2100d16a0 0x7fb210155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:23.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.853+0000 7fb20e800700 1 -- 172.21.15.89:0/370316516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2000094a0 con 0x7fb21015d9d0 2024-04-02T07:48:23.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.853+0000 7fb20e800700 1 --2- 172.21.15.89:0/370316516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb21015d9d0 0x7fb21015ef10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb208013000 tx=0x7fb208004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:23.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.853+0000 7fb206000700 1 -- 172.21.15.89:0/370316516 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208012430 con 0x7fb21015d9d0 2024-04-02T07:48:23.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.853+0000 7fb206000700 1 -- 172.21.15.89:0/370316516 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb208020070 con 0x7fb21015d9d0 2024-04-02T07:48:23.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.853+0000 7fb215200700 1 -- 172.21.15.89:0/370316516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb208009a70 con 0x7fb21015d9d0 2024-04-02T07:48:23.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.853+0000 7fb206000700 1 -- 172.21.15.89:0/370316516 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208004280 con 0x7fb21015d9d0 2024-04-02T07:48:23.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.853+0000 7fb215200700 1 -- 172.21.15.89:0/370316516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb210160120 con 0x7fb21015d9d0 2024-04-02T07:48:23.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.854+0000 7fb215200700 1 -- 172.21.15.89:0/370316516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2100d1ca0 con 0x7fb21015d9d0 2024-04-02T07:48:23.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.855+0000 7fb206000700 1 -- 172.21.15.89:0/370316516 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fb2080043e0 con 0x7fb21015d9d0 2024-04-02T07:48:23.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.856+0000 7fb206000700 1 --2- 172.21.15.89:0/370316516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb1f006ae90 0x7fb1f006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:23.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.856+0000 7fb206000700 1 -- 172.21.15.89:0/370316516 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fb208022070 con 0x7fb21015d9d0 2024-04-02T07:48:23.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.856+0000 7fb20de00700 1 --2- 172.21.15.89:0/370316516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb1f006ae90 0x7fb1f006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:23.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.857+0000 7fb20de00700 1 --2- 172.21.15.89:0/370316516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb1f006ae90 0x7fb1f006d350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb200009cd0 tx=0x7fb200009210 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:23.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:23.860+0000 7fb206000700 1 -- 172.21.15.89:0/370316516 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb2080561f0 con 0x7fb21015d9d0 2024-04-02T07:48:24.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.093+0000 7fb215200700 1 -- 172.21.15.89:0/370316516 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb210161fa0 con 0x7fb1f006ae90 2024-04-02T07:48:24.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.182+0000 7fb206000700 1 -- 172.21.15.89:0/370316516 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb200009920 con 0x7fb1f006ae90 2024-04-02T07:48:24.185 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2024-04-02T07:48:24.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.184+0000 7fb215200700 1 -- 172.21.15.89:0/370316516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb1f006ae90 msgr2=0x7fb1f006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:24.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.184+0000 7fb215200700 1 --2- 172.21.15.89:0/370316516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb1f006ae90 0x7fb1f006d350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb200009cd0 tx=0x7fb200009210 comp rx=0 tx=0).stop 2024-04-02T07:48:24.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.184+0000 7fb215200700 1 -- 172.21.15.89:0/370316516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb21015d9d0 msgr2=0x7fb21015ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:24.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.184+0000 7fb215200700 1 --2- 172.21.15.89:0/370316516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb21015d9d0 0x7fb21015ef10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb208013000 tx=0x7fb208004ad0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.184+0000 7fb215200700 1 -- 172.21.15.89:0/370316516 shutdown_connections 2024-04-02T07:48:24.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.184+0000 7fb215200700 1 --2- 172.21.15.89:0/370316516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb1f006ae90 0x7fb1f006d350 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.184+0000 7fb215200700 1 --2- 172.21.15.89:0/370316516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2100d16a0 0x7fb210155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.184+0000 7fb215200700 1 --2- 172.21.15.89:0/370316516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb21015d9d0 0x7fb21015ef10 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.184+0000 7fb215200700 1 --2- 172.21.15.89:0/370316516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2100d2540 0x7fb2100b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.184+0000 7fb215200700 1 -- 172.21.15.89:0/370316516 >> 172.21.15.89:0/370316516 conn(0x7fb21008aff0 msgr2=0x7fb2100b1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:24.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.184+0000 7fb215200700 1 -- 172.21.15.89:0/370316516 shutdown_connections 2024-04-02T07:48:24.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.184+0000 7fb215200700 1 -- 172.21.15.89:0/370316516 wait complete. 2024-04-02T07:48:24.205 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2024-04-02T07:48:24.205 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2024-04-02T07:48:24.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.397+0000 7f3d13400700 1 Processor -- start 2024-04-02T07:48:24.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.397+0000 7f3d13400700 1 -- start start 2024-04-02T07:48:24.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.398+0000 7f3d13400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3d0c0d16a0 0x7f3d0c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:24.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.398+0000 7f3d13400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0c161420 con 0x7f3d0c0d2370 2024-04-02T07:48:24.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.398+0000 7f3d13400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0c161700 con 0x7f3d0c15d9d0 2024-04-02T07:48:24.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.398+0000 7f3d13400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0c0d4340 con 0x7f3d0c0d16a0 2024-04-02T07:48:24.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.398+0000 7f3d11400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3d0c0d2370 0x7f3d0c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32774/0 (socket says 172.21.15.89:32774) 2024-04-02T07:48:24.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.398+0000 7f3d11400700 1 -- 172.21.15.89:0/3497929980 learned_addr learned my addr 172.21.15.89:0/3497929980 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:24.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.398+0000 7f3d0be00700 1 --2- 172.21.15.89:0/3497929980 >> v2:172.21.15.113:3300/0 conn(0x7f3d0c0d16a0 0x7f3d0c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:24.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.399+0000 7f3d0b400700 1 -- 172.21.15.89:0/3497929980 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917553701 0 0) 0x7f3d0c161420 con 0x7f3d0c0d2370 2024-04-02T07:48:24.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.399+0000 7f3d0b400700 1 -- 172.21.15.89:0/3497929980 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cf40035b0 con 0x7f3d0c0d2370 2024-04-02T07:48:24.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.399+0000 7f3d0be00700 1 -- 172.21.15.89:0/3497929980 >> v1:172.21.15.113:6789/0 conn(0x7f3d0c15d9d0 legacy=0x7f3d0c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:24.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.399+0000 7f3d0be00700 1 -- 172.21.15.89:0/3497929980 >> v1:172.21.15.89:6789/0 conn(0x7f3d0c0d2370 legacy=0x7f3d0c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:24.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.399+0000 7f3d0be00700 1 -- 172.21.15.89:0/3497929980 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0c1619e0 con 0x7f3d0c0d16a0 2024-04-02T07:48:24.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.399+0000 7f3d0be00700 1 --2- 172.21.15.89:0/3497929980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d0c0d16a0 0x7f3d0c0d1e30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8009820 tx=0x7f3cf8009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18e441a6ff93ca75 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:24.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.399+0000 7f3d0b400700 1 -- 172.21.15.89:0/3497929980 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8005e30 con 0x7f3d0c0d16a0 2024-04-02T07:48:24.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.399+0000 7f3d0b400700 1 -- 172.21.15.89:0/3497929980 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf8006030 con 0x7f3d0c0d16a0 2024-04-02T07:48:24.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.399+0000 7f3d0b400700 1 -- 172.21.15.89:0/3497929980 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf80063c0 con 0x7f3d0c0d16a0 2024-04-02T07:48:24.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.400+0000 7f3d13400700 1 -- 172.21.15.89:0/3497929980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d0c0d16a0 msgr2=0x7f3d0c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:24.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.400+0000 7f3d13400700 1 --2- 172.21.15.89:0/3497929980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d0c0d16a0 0x7f3d0c0d1e30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8009820 tx=0x7f3cf8009b30 comp rx=0 tx=0).stop 2024-04-02T07:48:24.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.400+0000 7f3d13400700 1 -- 172.21.15.89:0/3497929980 shutdown_connections 2024-04-02T07:48:24.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.400+0000 7f3d13400700 1 --2- 172.21.15.89:0/3497929980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d0c0d16a0 0x7f3d0c0d1e30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.400+0000 7f3d13400700 1 -- 172.21.15.89:0/3497929980 >> 172.21.15.89:0/3497929980 conn(0x7f3d0c08aff0 msgr2=0x7f3d0c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:24.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.400+0000 7f3d13400700 1 -- 172.21.15.89:0/3497929980 shutdown_connections 2024-04-02T07:48:24.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.400+0000 7f3d13400700 1 -- 172.21.15.89:0/3497929980 wait complete. 2024-04-02T07:48:24.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.402+0000 7f3d13400700 1 Processor -- start 2024-04-02T07:48:24.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.403+0000 7f3d13400700 1 -- start start 2024-04-02T07:48:24.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d13400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3d0c0d16a0 0x7f3d0c0ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:24.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d13400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d0c0d2370 0x7f3d0c0cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:24.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d13400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c15d9d0 0x7f3d0c15a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:24.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d13400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c15b250 con 0x7f3d0c15d9d0 2024-04-02T07:48:24.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d13400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c0ca1f0 con 0x7f3d0c0d2370 2024-04-02T07:48:24.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d13400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c0b61c0 con 0x7f3d0c0d16a0 2024-04-02T07:48:24.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d11400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c15d9d0 0x7f3d0c15a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:24.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d11400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c15d9d0 0x7f3d0c15a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41916/0 (socket says 172.21.15.89:41916) 2024-04-02T07:48:24.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d11400700 1 -- 172.21.15.89:0/567641942 learned_addr learned my addr 172.21.15.89:0/567641942 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:24.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d0be00700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d0c0d2370 0x7f3d0c0cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:24.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d10a00700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3d0c0d16a0 0x7f3d0c0ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:24.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d11400700 1 -- 172.21.15.89:0/567641942 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3d0c0d16a0 msgr2=0x7f3d0c0ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:24.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d11400700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3d0c0d16a0 0x7f3d0c0ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d11400700 1 -- 172.21.15.89:0/567641942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d0c0d2370 msgr2=0x7f3d0c0cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:24.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.404+0000 7f3d11400700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d0c0d2370 0x7f3d0c0cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.405+0000 7f3d11400700 1 -- 172.21.15.89:0/567641942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf8009430 con 0x7f3d0c15d9d0 2024-04-02T07:48:24.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.405+0000 7f3d10a00700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3d0c0d16a0 0x7f3d0c0ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:48:24.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.405+0000 7f3d11400700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c15d9d0 0x7f3d0c15a3e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400cb40 tx=0x7f3d0400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:24.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.405+0000 7f3d08c00700 1 -- 172.21.15.89:0/567641942 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0400aaa0 con 0x7f3d0c15d9d0 2024-04-02T07:48:24.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.405+0000 7f3d08c00700 1 -- 172.21.15.89:0/567641942 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0400ac00 con 0x7f3d0c15d9d0 2024-04-02T07:48:24.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.405+0000 7f3d13400700 1 -- 172.21.15.89:0/567641942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0c0b64a0 con 0x7f3d0c15d9d0 2024-04-02T07:48:24.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.405+0000 7f3d08c00700 1 -- 172.21.15.89:0/567641942 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d04012400 con 0x7f3d0c15d9d0 2024-04-02T07:48:24.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.405+0000 7f3d13400700 1 -- 172.21.15.89:0/567641942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0c0cd450 con 0x7f3d0c15d9d0 2024-04-02T07:48:24.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.406+0000 7f3d13400700 1 -- 172.21.15.89:0/567641942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0c0d1ca0 con 0x7f3d0c15d9d0 2024-04-02T07:48:24.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.407+0000 7f3d08c00700 1 -- 172.21.15.89:0/567641942 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f3d0401b3b0 con 0x7f3d0c15d9d0 2024-04-02T07:48:24.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.408+0000 7f3d08c00700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3cec06ae90 0x7f3cec06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:24.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.408+0000 7f3d08c00700 1 -- 172.21.15.89:0/567641942 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f3d04089590 con 0x7f3d0c15d9d0 2024-04-02T07:48:24.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.408+0000 7f3d10a00700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3cec06ae90 0x7f3cec06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:24.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.409+0000 7f3d10a00700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3cec06ae90 0x7f3cec06d350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3d00000f60 tx=0x7f3d00009040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:24.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.412+0000 7f3d08c00700 1 -- 172.21.15.89:0/567641942 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3d04054360 con 0x7f3d0c15d9d0 2024-04-02T07:48:24.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.640+0000 7f3d13400700 1 -- 172.21.15.89:0/567641942 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d0c161fa0 con 0x7f3cec06ae90 2024-04-02T07:48:24.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.727+0000 7f3d08c00700 1 -- 172.21.15.89:0/567641942 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f3d000013a0 con 0x7f3cec06ae90 2024-04-02T07:48:24.730 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2024-04-02T07:48:24.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.729+0000 7f3d13400700 1 -- 172.21.15.89:0/567641942 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3cec06ae90 msgr2=0x7f3cec06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:24.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.729+0000 7f3d13400700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3cec06ae90 0x7f3cec06d350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3d00000f60 tx=0x7f3d00009040 comp rx=0 tx=0).stop 2024-04-02T07:48:24.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.729+0000 7f3d13400700 1 -- 172.21.15.89:0/567641942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c15d9d0 msgr2=0x7f3d0c15a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:24.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.729+0000 7f3d13400700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c15d9d0 0x7f3d0c15a3e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400cb40 tx=0x7f3d0400ce50 comp rx=0 tx=0).stop 2024-04-02T07:48:24.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.729+0000 7f3d13400700 1 -- 172.21.15.89:0/567641942 shutdown_connections 2024-04-02T07:48:24.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.729+0000 7f3d13400700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3cec06ae90 0x7f3cec06d350 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.729+0000 7f3d13400700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d0c15d9d0 0x7f3d0c15a3e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.729+0000 7f3d13400700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3d0c0d16a0 0x7f3d0c0ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.729+0000 7f3d13400700 1 --2- 172.21.15.89:0/567641942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d0c0d2370 0x7f3d0c0cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.729+0000 7f3d13400700 1 -- 172.21.15.89:0/567641942 >> 172.21.15.89:0/567641942 conn(0x7f3d0c08aff0 msgr2=0x7f3d0c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:24.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.730+0000 7f3d13400700 1 -- 172.21.15.89:0/567641942 shutdown_connections 2024-04-02T07:48:24.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.730+0000 7f3d13400700 1 -- 172.21.15.89:0/567641942 wait complete. 2024-04-02T07:48:24.750 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2024-04-02T07:48:24.751 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2024-04-02T07:48:24.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.941+0000 7f6654e00700 1 Processor -- start 2024-04-02T07:48:24.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.941+0000 7f6654e00700 1 -- start start 2024-04-02T07:48:24.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.942+0000 7f6654e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f665015d9d0 0x7f66500d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:24.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.943+0000 7f6654e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66500d16a0 0x7f66500d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.943+0000 7f6654e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6650161420 con 0x7f66500d2370 2024-04-02T07:48:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.943+0000 7f6654e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66500d2820 con 0x7f665015d9d0 2024-04-02T07:48:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.943+0000 7f6654e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6650161700 con 0x7f66500d16a0 2024-04-02T07:48:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.943+0000 7f664de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f665015d9d0 0x7f66500d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.943+0000 7f664de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f665015d9d0 0x7f66500d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41926/0 (socket says 172.21.15.89:41926) 2024-04-02T07:48:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.943+0000 7f664de00700 1 -- 172.21.15.89:0/1350533113 learned_addr learned my addr 172.21.15.89:0/1350533113 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.943+0000 7f6647e00700 1 --2- 172.21.15.89:0/1350533113 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66500d16a0 0x7f66500d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.943+0000 7f664de00700 1 -- 172.21.15.89:0/1350533113 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66500d16a0 msgr2=0x7f66500d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.943+0000 7f664de00700 1 --2- 172.21.15.89:0/1350533113 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66500d16a0 0x7f66500d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.943+0000 7f664de00700 1 -- 172.21.15.89:0/1350533113 >> v1:172.21.15.113:6789/0 conn(0x7f66500d2370 legacy=0x7f665015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.943+0000 7f664de00700 1 -- 172.21.15.89:0/1350533113 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66501619e0 con 0x7f665015d9d0 2024-04-02T07:48:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.944+0000 7f664de00700 1 --2- 172.21.15.89:0/1350533113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f665015d9d0 0x7f66500d3c30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f66400098f0 tx=0x7f6640009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7fcc493a7abaac3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:24.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.944+0000 7f664d400700 1 -- 172.21.15.89:0/1350533113 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640011830 con 0x7f665015d9d0 2024-04-02T07:48:24.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.944+0000 7f664d400700 1 -- 172.21.15.89:0/1350533113 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6640011990 con 0x7f665015d9d0 2024-04-02T07:48:24.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.944+0000 7f664d400700 1 -- 172.21.15.89:0/1350533113 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640011d20 con 0x7f665015d9d0 2024-04-02T07:48:24.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.944+0000 7f6654e00700 1 -- 172.21.15.89:0/1350533113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f665015d9d0 msgr2=0x7f66500d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:24.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.944+0000 7f6654e00700 1 --2- 172.21.15.89:0/1350533113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f665015d9d0 0x7f66500d3c30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f66400098f0 tx=0x7f6640009c00 comp rx=0 tx=0).stop 2024-04-02T07:48:24.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.945+0000 7f6654e00700 1 -- 172.21.15.89:0/1350533113 shutdown_connections 2024-04-02T07:48:24.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.945+0000 7f6654e00700 1 --2- 172.21.15.89:0/1350533113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f665015d9d0 0x7f66500d3c30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.945+0000 7f6654e00700 1 --2- 172.21.15.89:0/1350533113 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66500d16a0 0x7f66500d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.945+0000 7f6654e00700 1 -- 172.21.15.89:0/1350533113 >> 172.21.15.89:0/1350533113 conn(0x7f665008aff0 msgr2=0x7f66500d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:24.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.945+0000 7f6654e00700 1 -- 172.21.15.89:0/1350533113 shutdown_connections 2024-04-02T07:48:24.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.945+0000 7f6654e00700 1 -- 172.21.15.89:0/1350533113 wait complete. 2024-04-02T07:48:24.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.946+0000 7f6654e00700 1 Processor -- start 2024-04-02T07:48:24.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.946+0000 7f6654e00700 1 -- start start 2024-04-02T07:48:24.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.947+0000 7f6654e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66500d16a0 0x7f6650155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:24.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.947+0000 7f6654e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66500d2370 0x7f66500b8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:24.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.947+0000 7f6654e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f665015d9d0 0x7f6650159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:24.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.947+0000 7f6654e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665015e810 con 0x7f66500d2370 2024-04-02T07:48:24.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.947+0000 7f6654e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66500b8df0 con 0x7f665015d9d0 2024-04-02T07:48:24.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.947+0000 7f6654e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665015dee0 con 0x7f66500d16a0 2024-04-02T07:48:24.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.948+0000 7f6647e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66500d2370 0x7f66500b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:24.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.948+0000 7f6647e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66500d2370 0x7f66500b8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41930/0 (socket says 172.21.15.89:41930) 2024-04-02T07:48:24.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.948+0000 7f6647e00700 1 -- 172.21.15.89:0/3686392457 learned_addr learned my addr 172.21.15.89:0/3686392457 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:24.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.948+0000 7f664de00700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66500d16a0 0x7f6650155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:24.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.948+0000 7f664e800700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f665015d9d0 0x7f6650159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:24.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.948+0000 7f6647e00700 1 -- 172.21.15.89:0/3686392457 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66500d16a0 msgr2=0x7f6650155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:24.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.948+0000 7f6647e00700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66500d16a0 0x7f6650155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.948+0000 7f6647e00700 1 -- 172.21.15.89:0/3686392457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f665015d9d0 msgr2=0x7f6650159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:24.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.948+0000 7f6647e00700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f665015d9d0 0x7f6650159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:24.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.948+0000 7f6647e00700 1 -- 172.21.15.89:0/3686392457 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6640009550 con 0x7f66500d2370 2024-04-02T07:48:24.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.948+0000 7f664de00700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66500d16a0 0x7f6650155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:48:24.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.948+0000 7f6647e00700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66500d2370 0x7f66500b8480 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f663800e960 tx=0x7f663800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:24.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.949+0000 7f6646000700 1 -- 172.21.15.89:0/3686392457 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66380096f0 con 0x7f66500d2370 2024-04-02T07:48:24.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.949+0000 7f6646000700 1 -- 172.21.15.89:0/3686392457 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6638013680 con 0x7f66500d2370 2024-04-02T07:48:24.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.949+0000 7f6654e00700 1 -- 172.21.15.89:0/3686392457 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665015e1c0 con 0x7f66500d2370 2024-04-02T07:48:24.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.949+0000 7f6646000700 1 -- 172.21.15.89:0/3686392457 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6638013a30 con 0x7f66500d2370 2024-04-02T07:48:24.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.949+0000 7f6654e00700 1 -- 172.21.15.89:0/3686392457 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6650155560 con 0x7f66500d2370 2024-04-02T07:48:24.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.950+0000 7f6654e00700 1 -- 172.21.15.89:0/3686392457 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66500d1ca0 con 0x7f66500d2370 2024-04-02T07:48:24.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.951+0000 7f6646000700 1 -- 172.21.15.89:0/3686392457 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f6638013b90 con 0x7f66500d2370 2024-04-02T07:48:24.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.951+0000 7f6646000700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f662c06ae90 0x7f662c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:24.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.952+0000 7f6646000700 1 -- 172.21.15.89:0/3686392457 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f6638089ba0 con 0x7f66500d2370 2024-04-02T07:48:24.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.952+0000 7f664de00700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f662c06ae90 0x7f662c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:24.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.952+0000 7f664de00700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f662c06ae90 0x7f662c06d350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6640014040 tx=0x7f6640008030 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:24.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:24.956+0000 7f6646000700 1 -- 172.21.15.89:0/3686392457 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6638054970 con 0x7f66500d2370 2024-04-02T07:48:25.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.189+0000 7f6654e00700 1 -- 172.21.15.89:0/3686392457 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f665015e050 con 0x7f662c06ae90 2024-04-02T07:48:25.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.276+0000 7f6646000700 1 -- 172.21.15.89:0/3686392457 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f665015e050 con 0x7f662c06ae90 2024-04-02T07:48:25.280 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2024-04-02T07:48:25.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.279+0000 7f6654e00700 1 -- 172.21.15.89:0/3686392457 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f662c06ae90 msgr2=0x7f662c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:25.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.279+0000 7f6654e00700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f662c06ae90 0x7f662c06d350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6640014040 tx=0x7f6640008030 comp rx=0 tx=0).stop 2024-04-02T07:48:25.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.279+0000 7f6654e00700 1 -- 172.21.15.89:0/3686392457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66500d2370 msgr2=0x7f66500b8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:25.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.279+0000 7f6654e00700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66500d2370 0x7f66500b8480 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f663800e960 tx=0x7f663800ec70 comp rx=0 tx=0).stop 2024-04-02T07:48:25.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.279+0000 7f6654e00700 1 -- 172.21.15.89:0/3686392457 shutdown_connections 2024-04-02T07:48:25.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.279+0000 7f6654e00700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f662c06ae90 0x7f662c06d350 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:25.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.279+0000 7f6654e00700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f665015d9d0 0x7f6650159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:25.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.279+0000 7f6654e00700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66500d16a0 0x7f6650155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:25.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.279+0000 7f6654e00700 1 --2- 172.21.15.89:0/3686392457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66500d2370 0x7f66500b8480 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:25.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.279+0000 7f6654e00700 1 -- 172.21.15.89:0/3686392457 >> 172.21.15.89:0/3686392457 conn(0x7f665008aff0 msgr2=0x7f66500b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:25.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.279+0000 7f6654e00700 1 -- 172.21.15.89:0/3686392457 shutdown_connections 2024-04-02T07:48:25.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.279+0000 7f6654e00700 1 -- 172.21.15.89:0/3686392457 wait complete. 2024-04-02T07:48:25.300 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2024-04-02T07:48:25.300 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2024-04-02T07:48:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.491+0000 7fb9ba600700 1 Processor -- start 2024-04-02T07:48:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.491+0000 7fb9ba600700 1 -- start start 2024-04-02T07:48:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.492+0000 7fb9ba600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb9b415d710 0x7fb9b40d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.492+0000 7fb9ba600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb9b40d2020 0x7fb9b40d0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.492+0000 7fb9ba600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b40cf640 con 0x7fb9b40d1b70 2024-04-02T07:48:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.492+0000 7fb9ba600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b40cf220 con 0x7fb9b40d2020 2024-04-02T07:48:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.492+0000 7fb9ba600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b40ce5a0 con 0x7fb9b415d710 2024-04-02T07:48:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.492+0000 7fb9b3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb9b40d2020 0x7fb9b40d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.492+0000 7fb9abe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb9b40d1b70 0x7fb9b40cff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32776/0 (socket says 172.21.15.89:32776) 2024-04-02T07:48:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.492+0000 7fb9abe00700 1 -- 172.21.15.89:0/2363758035 learned_addr learned my addr 172.21.15.89:0/2363758035 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.492+0000 7fb9b3400700 1 --2- 172.21.15.89:0/2363758035 >> v2:172.21.15.113:3300/0 conn(0x7fb9b415d710 0x7fb9b40d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.493+0000 7fb9b3e00700 1 -- 172.21.15.89:0/2363758035 >> v2:172.21.15.113:3300/0 conn(0x7fb9b415d710 msgr2=0x7fb9b40d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.493+0000 7fb9b3e00700 1 --2- 172.21.15.89:0/2363758035 >> v2:172.21.15.113:3300/0 conn(0x7fb9b415d710 0x7fb9b40d12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.493+0000 7fb9b3e00700 1 -- 172.21.15.89:0/2363758035 >> v1:172.21.15.89:6789/0 conn(0x7fb9b40d1b70 legacy=0x7fb9b40cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.493+0000 7fb9b3e00700 1 -- 172.21.15.89:0/2363758035 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b40ce880 con 0x7fb9b40d2020 2024-04-02T07:48:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.493+0000 7fb9b3400700 1 --2- 172.21.15.89:0/2363758035 >> v2:172.21.15.113:3300/0 conn(0x7fb9b415d710 0x7fb9b40d12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:48:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.493+0000 7fb9b2a00700 1 -- 172.21.15.89:0/2363758035 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109966769 0 0) 0x7fb9b40cf640 con 0x7fb9b40d1b70 2024-04-02T07:48:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.493+0000 7fb9b3e00700 1 --2- 172.21.15.89:0/2363758035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9b40d2020 0x7fb9b40d0620 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb9b408cea0 tx=0x7fb9ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9504a45a76423e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.493+0000 7fb9b2a00700 1 -- 172.21.15.89:0/2363758035 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac00e070 con 0x7fb9b40d2020 2024-04-02T07:48:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.493+0000 7fb9b2a00700 1 -- 172.21.15.89:0/2363758035 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ac00bbd0 con 0x7fb9b40d2020 2024-04-02T07:48:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.493+0000 7fb9b2a00700 1 -- 172.21.15.89:0/2363758035 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac012430 con 0x7fb9b40d2020 2024-04-02T07:48:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.494+0000 7fb9ba600700 1 -- 172.21.15.89:0/2363758035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9b40d2020 msgr2=0x7fb9b40d0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.494+0000 7fb9ba600700 1 --2- 172.21.15.89:0/2363758035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9b40d2020 0x7fb9b40d0620 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb9b408cea0 tx=0x7fb9ac00b330 comp rx=0 tx=0).stop 2024-04-02T07:48:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.494+0000 7fb9ba600700 1 -- 172.21.15.89:0/2363758035 shutdown_connections 2024-04-02T07:48:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.494+0000 7fb9ba600700 1 --2- 172.21.15.89:0/2363758035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9b40d2020 0x7fb9b40d0620 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.494+0000 7fb9ba600700 1 --2- 172.21.15.89:0/2363758035 >> v2:172.21.15.113:3300/0 conn(0x7fb9b415d710 0x7fb9b40d12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.494+0000 7fb9ba600700 1 -- 172.21.15.89:0/2363758035 >> 172.21.15.89:0/2363758035 conn(0x7fb9b408aff0 msgr2=0x7fb9b40d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.494+0000 7fb9ba600700 1 -- 172.21.15.89:0/2363758035 shutdown_connections 2024-04-02T07:48:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.494+0000 7fb9ba600700 1 -- 172.21.15.89:0/2363758035 wait complete. 2024-04-02T07:48:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.495+0000 7fb9ba600700 1 Processor -- start 2024-04-02T07:48:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.496+0000 7fb9ba600700 1 -- start start 2024-04-02T07:48:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.496+0000 7fb9ba600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9b40d1b70 0x7fb9b40c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.496+0000 7fb9ba600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9b40d2020 0x7fb9b40c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.496+0000 7fb9ba600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9b415d710 0x7fb9b40c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.496+0000 7fb9ba600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b41f2f00 con 0x7fb9b415d710 2024-04-02T07:48:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9ba600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b40c91e0 con 0x7fb9b40d1b70 2024-04-02T07:48:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9ba600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b40c8220 con 0x7fb9b40d2020 2024-04-02T07:48:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9b3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9b415d710 0x7fb9b40c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9b3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9b415d710 0x7fb9b40c7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41942/0 (socket says 172.21.15.89:41942) 2024-04-02T07:48:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9b3e00700 1 -- 172.21.15.89:0/201094018 learned_addr learned my addr 172.21.15.89:0/201094018 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9abe00700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9b40d2020 0x7fb9b40c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9b3400700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9b40d1b70 0x7fb9b40c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9b3e00700 1 -- 172.21.15.89:0/201094018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9b40d2020 msgr2=0x7fb9b40c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9b3e00700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9b40d2020 0x7fb9b40c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9b3e00700 1 -- 172.21.15.89:0/201094018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9b40d1b70 msgr2=0x7fb9b40c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9b3e00700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9b40d1b70 0x7fb9b40c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9b3e00700 1 -- 172.21.15.89:0/201094018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ac009a70 con 0x7fb9b415d710 2024-04-02T07:48:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9abe00700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9b40d2020 0x7fb9b40c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9b3400700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9b40d1b70 0x7fb9b40c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.497+0000 7fb9b3e00700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9b415d710 0x7fb9b40c7ce0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb9b408bf10 tx=0x7fb9ac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.498+0000 7fb9ab400700 1 -- 172.21.15.89:0/201094018 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac012430 con 0x7fb9b415d710 2024-04-02T07:48:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.498+0000 7fb9ab400700 1 -- 172.21.15.89:0/201094018 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ac020070 con 0x7fb9b415d710 2024-04-02T07:48:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.498+0000 7fb9ba600700 1 -- 172.21.15.89:0/201094018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b41f3d00 con 0x7fb9b415d710 2024-04-02T07:48:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.498+0000 7fb9ab400700 1 -- 172.21.15.89:0/201094018 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac004280 con 0x7fb9b415d710 2024-04-02T07:48:25.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.498+0000 7fb9ba600700 1 -- 172.21.15.89:0/201094018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b41f33d0 con 0x7fb9b415d710 2024-04-02T07:48:25.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.499+0000 7fb9ba600700 1 -- 172.21.15.89:0/201094018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b4067e30 con 0x7fb9b415d710 2024-04-02T07:48:25.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.500+0000 7fb9ab400700 1 -- 172.21.15.89:0/201094018 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fb9ac0043e0 con 0x7fb9b415d710 2024-04-02T07:48:25.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.501+0000 7fb9ab400700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb99006ae90 0x7fb99006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:25.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.501+0000 7fb9ab400700 1 -- 172.21.15.89:0/201094018 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fb9ac022070 con 0x7fb9b415d710 2024-04-02T07:48:25.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.501+0000 7fb9b3400700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb99006ae90 0x7fb99006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:25.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.501+0000 7fb9b3400700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb99006ae90 0x7fb99006d350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb9b415bfd0 tx=0x7fb9a4008040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:25.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.505+0000 7fb9ab400700 1 -- 172.21.15.89:0/201094018 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb9ac0567c0 con 0x7fb9b415d710 2024-04-02T07:48:25.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.742+0000 7fb9ba600700 1 -- 172.21.15.89:0/201094018 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9b40d2440 con 0x7fb99006ae90 2024-04-02T07:48:25.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.829+0000 7fb9ab400700 1 -- 172.21.15.89:0/201094018 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb9a40091c0 con 0x7fb99006ae90 2024-04-02T07:48:25.831 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2024-04-02T07:48:25.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.831+0000 7fb9ba600700 1 -- 172.21.15.89:0/201094018 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb99006ae90 msgr2=0x7fb99006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:25.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.831+0000 7fb9ba600700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb99006ae90 0x7fb99006d350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb9b415bfd0 tx=0x7fb9a4008040 comp rx=0 tx=0).stop 2024-04-02T07:48:25.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.831+0000 7fb9ba600700 1 -- 172.21.15.89:0/201094018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9b415d710 msgr2=0x7fb9b40c7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:25.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.831+0000 7fb9ba600700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9b415d710 0x7fb9b40c7ce0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb9b408bf10 tx=0x7fb9ac004ad0 comp rx=0 tx=0).stop 2024-04-02T07:48:25.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.831+0000 7fb9ba600700 1 -- 172.21.15.89:0/201094018 shutdown_connections 2024-04-02T07:48:25.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.831+0000 7fb9ba600700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb99006ae90 0x7fb99006d350 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:25.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.831+0000 7fb9ba600700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9b40d1b70 0x7fb9b40c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:25.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.831+0000 7fb9ba600700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9b415d710 0x7fb9b40c7ce0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:25.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.831+0000 7fb9ba600700 1 --2- 172.21.15.89:0/201094018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9b40d2020 0x7fb9b40c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:25.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.831+0000 7fb9ba600700 1 -- 172.21.15.89:0/201094018 >> 172.21.15.89:0/201094018 conn(0x7fb9b408aff0 msgr2=0x7fb9b415b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:25.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.831+0000 7fb9ba600700 1 -- 172.21.15.89:0/201094018 shutdown_connections 2024-04-02T07:48:25.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:25.831+0000 7fb9ba600700 1 -- 172.21.15.89:0/201094018 wait complete. 2024-04-02T07:48:25.851 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2024-04-02T07:48:25.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:25.851 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-04-02T07:48:25.898 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:25.899 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:25.899 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:25.899 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:25.899 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:25.899 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:25.899 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:25.899 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:25.899 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:25.899 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:25.899 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:25.900 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-02-07_38_00_UTC', 'scheduled-2024-04-02-07_39_00_UTC', 'scheduled-2024-04-02-07_40_00_UTC', 'scheduled-2024-04-02-07_41_00_UTC', 'scheduled-2024-04-02-07_42_00_UTC', 'scheduled-2024-04-02-07_43_00_UTC', 'scheduled-2024-04-02-07_44_00_UTC', 'scheduled-2024-04-02-07_45_00_UTC', 'scheduled-2024-04-02-07_46_00_UTC', 'scheduled-2024-04-02-07_47_00_UTC', 'scheduled-2024-04-02-07_48_00_UTC'] 2024-04-02T07:48:25.900 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2024-04-02T07:48:26.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.091+0000 7f8893000700 1 Processor -- start 2024-04-02T07:48:26.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.092+0000 7f8893000700 1 -- start start 2024-04-02T07:48:26.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.093+0000 7f8893000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f888c15d990 0x7f888c0d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:26.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.093+0000 7f8893000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f888c09a750 0x7f888c06c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:26.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.093+0000 7f8893000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888c0d0970 con 0x7f888c099a60 2024-04-02T07:48:26.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.093+0000 7f8893000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888c06c020 con 0x7f888c09a750 2024-04-02T07:48:26.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.093+0000 7f8893000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888c0c81a0 con 0x7f888c15d990 2024-04-02T07:48:26.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.093+0000 7f8891000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f888c09a750 0x7f888c06c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:26.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.093+0000 7f888be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f888c099a60 0x7f888c099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32778/0 (socket says 172.21.15.89:32778) 2024-04-02T07:48:26.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.093+0000 7f888be00700 1 -- 172.21.15.89:0/3069901348 learned_addr learned my addr 172.21.15.89:0/3069901348 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:26.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.093+0000 7f888b400700 1 --2- 172.21.15.89:0/3069901348 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f888c15d990 0x7f888c0d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:26.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.093+0000 7f8891000700 1 -- 172.21.15.89:0/3069901348 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f888c15d990 msgr2=0x7f888c0d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:26.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.093+0000 7f8891000700 1 --2- 172.21.15.89:0/3069901348 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f888c15d990 0x7f888c0d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.094+0000 7f8891000700 1 -- 172.21.15.89:0/3069901348 >> v1:172.21.15.89:6789/0 conn(0x7f888c099a60 legacy=0x7f888c099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:26.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.094+0000 7f8891000700 1 -- 172.21.15.89:0/3069901348 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c0d0690 con 0x7f888c09a750 2024-04-02T07:48:26.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.094+0000 7f888aa00700 1 -- 172.21.15.89:0/3069901348 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040276188 0 0) 0x7f888c0d0970 con 0x7f888c099a60 2024-04-02T07:48:26.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.094+0000 7f8891000700 1 --2- 172.21.15.89:0/3069901348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f888c09a750 0x7f888c06c4b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8884009d90 tx=0x7f888400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8c346c735e1138 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:26.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.094+0000 7f888aa00700 1 -- 172.21.15.89:0/3069901348 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888400e070 con 0x7f888c09a750 2024-04-02T07:48:26.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.094+0000 7f888aa00700 1 -- 172.21.15.89:0/3069901348 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888400bbd0 con 0x7f888c09a750 2024-04-02T07:48:26.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.094+0000 7f888aa00700 1 -- 172.21.15.89:0/3069901348 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884012430 con 0x7f888c09a750 2024-04-02T07:48:26.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.095+0000 7f8893000700 1 -- 172.21.15.89:0/3069901348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f888c09a750 msgr2=0x7f888c06c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:26.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.095+0000 7f8893000700 1 --2- 172.21.15.89:0/3069901348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f888c09a750 0x7f888c06c4b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8884009d90 tx=0x7f888400b330 comp rx=0 tx=0).stop 2024-04-02T07:48:26.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.095+0000 7f8893000700 1 -- 172.21.15.89:0/3069901348 shutdown_connections 2024-04-02T07:48:26.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.095+0000 7f8893000700 1 --2- 172.21.15.89:0/3069901348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f888c09a750 0x7f888c06c4b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.095+0000 7f8893000700 1 --2- 172.21.15.89:0/3069901348 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f888c15d990 0x7f888c0d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.095+0000 7f8893000700 1 -- 172.21.15.89:0/3069901348 >> 172.21.15.89:0/3069901348 conn(0x7f888c0b2e20 msgr2=0x7f888c0b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:26.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.095+0000 7f8893000700 1 -- 172.21.15.89:0/3069901348 shutdown_connections 2024-04-02T07:48:26.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.095+0000 7f8893000700 1 -- 172.21.15.89:0/3069901348 wait complete. 2024-04-02T07:48:26.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.096+0000 7f8893000700 1 Processor -- start 2024-04-02T07:48:26.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.097+0000 7f8893000700 1 -- start start 2024-04-02T07:48:26.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.097+0000 7f8893000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f888c099a60 0x7f888c1567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:26.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f8893000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f888c09a750 0x7f888c06f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:26.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f8893000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f888c15d990 0x7f888c0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:26.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f8893000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c0c6d60 con 0x7f888c15d990 2024-04-02T07:48:26.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f8893000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c0c7040 con 0x7f888c099a60 2024-04-02T07:48:26.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f8893000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888c0c7320 con 0x7f888c09a750 2024-04-02T07:48:26.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f8891000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f888c15d990 0x7f888c0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:26.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f8891000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f888c15d990 0x7f888c0ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41968/0 (socket says 172.21.15.89:41968) 2024-04-02T07:48:26.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f8891000700 1 -- 172.21.15.89:0/3401683490 learned_addr learned my addr 172.21.15.89:0/3401683490 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:26.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f888b400700 1 --2- 172.21.15.89:0/3401683490 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f888c09a750 0x7f888c06f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:26.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f888be00700 1 --2- 172.21.15.89:0/3401683490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f888c099a60 0x7f888c1567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:26.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f8891000700 1 -- 172.21.15.89:0/3401683490 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f888c09a750 msgr2=0x7f888c06f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:26.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f8891000700 1 --2- 172.21.15.89:0/3401683490 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f888c09a750 0x7f888c06f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f8891000700 1 -- 172.21.15.89:0/3401683490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f888c099a60 msgr2=0x7f888c1567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:26.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f8891000700 1 --2- 172.21.15.89:0/3401683490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f888c099a60 0x7f888c1567b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f8891000700 1 -- 172.21.15.89:0/3401683490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8884009a70 con 0x7f888c15d990 2024-04-02T07:48:26.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.098+0000 7f8891000700 1 --2- 172.21.15.89:0/3401683490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f888c15d990 0x7f888c0ccae0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8884014040 tx=0x7f8884004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:26.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.099+0000 7f8883e00700 1 -- 172.21.15.89:0/3401683490 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884012430 con 0x7f888c15d990 2024-04-02T07:48:26.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.099+0000 7f8883e00700 1 -- 172.21.15.89:0/3401683490 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8884020070 con 0x7f888c15d990 2024-04-02T07:48:26.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.099+0000 7f8893000700 1 -- 172.21.15.89:0/3401683490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888c0c7600 con 0x7f888c15d990 2024-04-02T07:48:26.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.099+0000 7f8883e00700 1 -- 172.21.15.89:0/3401683490 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884004280 con 0x7f888c15d990 2024-04-02T07:48:26.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.099+0000 7f8893000700 1 -- 172.21.15.89:0/3401683490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888c156cf0 con 0x7f888c15d990 2024-04-02T07:48:26.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.100+0000 7f8893000700 1 -- 172.21.15.89:0/3401683490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888c0c7b20 con 0x7f888c15d990 2024-04-02T07:48:26.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.101+0000 7f8883e00700 1 -- 172.21.15.89:0/3401683490 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f88840043e0 con 0x7f888c15d990 2024-04-02T07:48:26.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.101+0000 7f8883e00700 1 --2- 172.21.15.89:0/3401683490 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f886c06ae90 0x7f886c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:26.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.102+0000 7f8883e00700 1 -- 172.21.15.89:0/3401683490 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f8884022070 con 0x7f888c15d990 2024-04-02T07:48:26.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.102+0000 7f888be00700 1 --2- 172.21.15.89:0/3401683490 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f886c06ae90 0x7f886c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:26.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.102+0000 7f888be00700 1 --2- 172.21.15.89:0/3401683490 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f886c06ae90 0x7f886c06d350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f887c000f60 tx=0x7f887c007c90 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:26.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.107+0000 7f8883e00700 1 -- 172.21.15.89:0/3401683490 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f888c0c7b20 con 0x7f888c15d990 2024-04-02T07:48:26.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.328+0000 7f8893000700 1 -- 172.21.15.89:0/3401683490 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f888c06bcb0 con 0x7f886c06ae90 2024-04-02T07:48:26.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.340+0000 7f8883e00700 1 -- 172.21.15.89:0/3401683490 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f887c013420 con 0x7f886c06ae90 2024-04-02T07:48:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:26.343 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:23", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:26.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.342+0000 7f8893000700 1 -- 172.21.15.89:0/3401683490 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f886c06ae90 msgr2=0x7f886c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:26.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.342+0000 7f8893000700 1 --2- 172.21.15.89:0/3401683490 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f886c06ae90 0x7f886c06d350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f887c000f60 tx=0x7f887c007c90 comp rx=0 tx=0).stop 2024-04-02T07:48:26.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.342+0000 7f8893000700 1 -- 172.21.15.89:0/3401683490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f888c15d990 msgr2=0x7f888c0ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:26.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.342+0000 7f8893000700 1 --2- 172.21.15.89:0/3401683490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f888c15d990 0x7f888c0ccae0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8884014040 tx=0x7f8884004ad0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.342+0000 7f8893000700 1 -- 172.21.15.89:0/3401683490 shutdown_connections 2024-04-02T07:48:26.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.342+0000 7f8893000700 1 --2- 172.21.15.89:0/3401683490 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f886c06ae90 0x7f886c06d350 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.342+0000 7f8893000700 1 --2- 172.21.15.89:0/3401683490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f888c099a60 0x7f888c1567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.342+0000 7f8893000700 1 --2- 172.21.15.89:0/3401683490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f888c15d990 0x7f888c0ccae0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.342+0000 7f8893000700 1 --2- 172.21.15.89:0/3401683490 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f888c09a750 0x7f888c06f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.342+0000 7f8893000700 1 -- 172.21.15.89:0/3401683490 >> 172.21.15.89:0/3401683490 conn(0x7f888c0b2e20 msgr2=0x7f888c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:26.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.342+0000 7f8893000700 1 -- 172.21.15.89:0/3401683490 shutdown_connections 2024-04-02T07:48:26.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.342+0000 7f8893000700 1 -- 172.21.15.89:0/3401683490 wait complete. 2024-04-02T07:48:26.363 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:23", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:26.363 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2024-04-02T00:00:00', 'created': '2024-04-02T07:37:23', 'first': '2024-04-02T07:38:00', 'last': '2024-04-02T07:48:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-02T07:48:26.363 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-02T07:48:26.363 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-02T07:48:26.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:26.363 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-04-02T07:48:26.409 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:26.409 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:26.409 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:26.409 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:26.409 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:26.409 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:26.409 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:26.409 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:26.409 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:26.410 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:26.410 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:26.411 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-02-07_38_00_UTC', 'scheduled-2024-04-02-07_39_00_UTC', 'scheduled-2024-04-02-07_40_00_UTC', 'scheduled-2024-04-02-07_41_00_UTC', 'scheduled-2024-04-02-07_42_00_UTC', 'scheduled-2024-04-02-07_43_00_UTC', 'scheduled-2024-04-02-07_44_00_UTC', 'scheduled-2024-04-02-07_45_00_UTC', 'scheduled-2024-04-02-07_46_00_UTC', 'scheduled-2024-04-02-07_47_00_UTC', 'scheduled-2024-04-02-07_48_00_UTC'] 2024-04-02T07:48:26.411 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2024-04-02T07:48:26.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.641+0000 7f51f4600700 1 Processor -- start 2024-04-02T07:48:26.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.641+0000 7f51f4600700 1 -- start start 2024-04-02T07:48:26.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.642+0000 7f51f4600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f51ec15d9d0 0x7f51ec0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:26.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.642+0000 7f51f4600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51ec0d16a0 0x7f51ec0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:26.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.642+0000 7f51f4600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51ec161420 con 0x7f51ec0d2370 2024-04-02T07:48:26.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.642+0000 7f51f4600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51ec0d27e0 con 0x7f51ec0d16a0 2024-04-02T07:48:26.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.642+0000 7f51f4600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51ec161700 con 0x7f51ec15d9d0 2024-04-02T07:48:26.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.642+0000 7f51f1200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51ec0d16a0 0x7f51ec0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:26.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.642+0000 7f51f1200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51ec0d16a0 0x7f51ec0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41978/0 (socket says 172.21.15.89:41978) 2024-04-02T07:48:26.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.642+0000 7f51f1200700 1 -- 172.21.15.89:0/3300012295 learned_addr learned my addr 172.21.15.89:0/3300012295 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:26.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.642+0000 7f51f1c00700 1 --2- 172.21.15.89:0/3300012295 >> v2:172.21.15.113:3300/0 conn(0x7f51ec15d9d0 0x7f51ec0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.643+0000 7f51f1200700 1 -- 172.21.15.89:0/3300012295 >> v2:172.21.15.113:3300/0 conn(0x7f51ec15d9d0 msgr2=0x7f51ec0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.643+0000 7f51f1200700 1 --2- 172.21.15.89:0/3300012295 >> v2:172.21.15.113:3300/0 conn(0x7f51ec15d9d0 0x7f51ec0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.643+0000 7f51f1200700 1 -- 172.21.15.89:0/3300012295 >> v1:172.21.15.113:6789/0 conn(0x7f51ec0d2370 legacy=0x7f51ec15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.643+0000 7f51f1200700 1 -- 172.21.15.89:0/3300012295 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ec1619e0 con 0x7f51ec0d16a0 2024-04-02T07:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.643+0000 7f51f1200700 1 --2- 172.21.15.89:0/3300012295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ec0d16a0 0x7f51ec0d1e30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0d0ff0 tx=0x7f51d8009bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f62ec58918bf9b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.643+0000 7f51ebe00700 1 -- 172.21.15.89:0/3300012295 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d8011830 con 0x7f51ec0d16a0 2024-04-02T07:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.643+0000 7f51ebe00700 1 -- 172.21.15.89:0/3300012295 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d8011990 con 0x7f51ec0d16a0 2024-04-02T07:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.643+0000 7f51ebe00700 1 -- 172.21.15.89:0/3300012295 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d8011d40 con 0x7f51ec0d16a0 2024-04-02T07:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.644+0000 7f51f4600700 1 -- 172.21.15.89:0/3300012295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ec0d16a0 msgr2=0x7f51ec0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.644+0000 7f51f4600700 1 --2- 172.21.15.89:0/3300012295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ec0d16a0 0x7f51ec0d1e30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0d0ff0 tx=0x7f51d8009bc0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.644+0000 7f51f4600700 1 -- 172.21.15.89:0/3300012295 shutdown_connections 2024-04-02T07:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.644+0000 7f51f4600700 1 --2- 172.21.15.89:0/3300012295 >> v2:172.21.15.113:3300/0 conn(0x7f51ec15d9d0 0x7f51ec0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.644+0000 7f51f4600700 1 --2- 172.21.15.89:0/3300012295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ec0d16a0 0x7f51ec0d1e30 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.644+0000 7f51f4600700 1 -- 172.21.15.89:0/3300012295 >> 172.21.15.89:0/3300012295 conn(0x7f51ec08aff0 msgr2=0x7f51ec0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.644+0000 7f51f4600700 1 -- 172.21.15.89:0/3300012295 shutdown_connections 2024-04-02T07:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.644+0000 7f51f4600700 1 -- 172.21.15.89:0/3300012295 wait complete. 2024-04-02T07:48:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.646+0000 7f51f4600700 1 Processor -- start 2024-04-02T07:48:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.646+0000 7f51f4600700 1 -- start start 2024-04-02T07:48:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f4600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51ec0d16a0 0x7f51ec0b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f4600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ec0d2370 0x7f51ec0c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f4600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51ec15d9d0 0x7f51ec0b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f4600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0b5820 con 0x7f51ec0d2370 2024-04-02T07:48:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f4600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0b55f0 con 0x7f51ec15d9d0 2024-04-02T07:48:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f4600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec1560a0 con 0x7f51ec0d16a0 2024-04-02T07:48:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ec0d2370 0x7f51ec0c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ec0d2370 0x7f51ec0c9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41982/0 (socket says 172.21.15.89:41982) 2024-04-02T07:48:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f1200700 1 -- 172.21.15.89:0/504746067 learned_addr learned my addr 172.21.15.89:0/504746067 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f1c00700 1 --2- 172.21.15.89:0/504746067 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51ec0d16a0 0x7f51ec0b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f2600700 1 --2- 172.21.15.89:0/504746067 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51ec15d9d0 0x7f51ec0b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f1200700 1 -- 172.21.15.89:0/504746067 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51ec0d16a0 msgr2=0x7f51ec0b50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f1200700 1 --2- 172.21.15.89:0/504746067 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51ec0d16a0 0x7f51ec0b50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f1200700 1 -- 172.21.15.89:0/504746067 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51ec15d9d0 msgr2=0x7f51ec0b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f1200700 1 --2- 172.21.15.89:0/504746067 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51ec15d9d0 0x7f51ec0b4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.647+0000 7f51f1200700 1 -- 172.21.15.89:0/504746067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d80094f0 con 0x7f51ec0d2370 2024-04-02T07:48:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.648+0000 7f51f1200700 1 --2- 172.21.15.89:0/504746067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ec0d2370 0x7f51ec0c9430 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f51ec08c230 tx=0x7f51d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.648+0000 7f51e9600700 1 -- 172.21.15.89:0/504746067 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d801a990 con 0x7f51ec0d2370 2024-04-02T07:48:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.648+0000 7f51e9600700 1 -- 172.21.15.89:0/504746067 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d801f070 con 0x7f51ec0d2370 2024-04-02T07:48:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.648+0000 7f51f4600700 1 -- 172.21.15.89:0/504746067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ec0c9970 con 0x7f51ec0d2370 2024-04-02T07:48:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.648+0000 7f51e9600700 1 -- 172.21.15.89:0/504746067 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d801ad10 con 0x7f51ec0d2370 2024-04-02T07:48:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.648+0000 7f51f4600700 1 -- 172.21.15.89:0/504746067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ec15e230 con 0x7f51ec0d2370 2024-04-02T07:48:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.649+0000 7f51f4600700 1 -- 172.21.15.89:0/504746067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ec0d1ca0 con 0x7f51ec0d2370 2024-04-02T07:48:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.650+0000 7f51e9600700 1 -- 172.21.15.89:0/504746067 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f51d801b430 con 0x7f51ec0d2370 2024-04-02T07:48:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.651+0000 7f51e9600700 1 --2- 172.21.15.89:0/504746067 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f51cc06ae90 0x7f51cc06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.651+0000 7f51e9600700 1 -- 172.21.15.89:0/504746067 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f51d808a9d0 con 0x7f51ec0d2370 2024-04-02T07:48:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.651+0000 7f51f1c00700 1 --2- 172.21.15.89:0/504746067 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f51cc06ae90 0x7f51cc06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.652+0000 7f51f1c00700 1 --2- 172.21.15.89:0/504746067 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f51cc06ae90 0x7f51cc06d350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f51e0009ad0 tx=0x7f51e0008040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:26.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.655+0000 7f51e9600700 1 -- 172.21.15.89:0/504746067 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f51d8055720 con 0x7f51ec0d2370 2024-04-02T07:48:26.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.884+0000 7f51f4600700 1 -- 172.21.15.89:0/504746067 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f51ec161fa0 con 0x7f51cc06ae90 2024-04-02T07:48:26.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.894+0000 7f51e9600700 1 -- 172.21.15.89:0/504746067 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f51e0009720 con 0x7f51cc06ae90 2024-04-02T07:48:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:26.896 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:24", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:26.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.896+0000 7f51f4600700 1 -- 172.21.15.89:0/504746067 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f51cc06ae90 msgr2=0x7f51cc06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:26.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.896+0000 7f51f4600700 1 --2- 172.21.15.89:0/504746067 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f51cc06ae90 0x7f51cc06d350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f51e0009ad0 tx=0x7f51e0008040 comp rx=0 tx=0).stop 2024-04-02T07:48:26.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.896+0000 7f51f4600700 1 -- 172.21.15.89:0/504746067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ec0d2370 msgr2=0x7f51ec0c9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:26.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.896+0000 7f51f4600700 1 --2- 172.21.15.89:0/504746067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ec0d2370 0x7f51ec0c9430 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f51ec08c230 tx=0x7f51d8004750 comp rx=0 tx=0).stop 2024-04-02T07:48:26.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.896+0000 7f51f4600700 1 -- 172.21.15.89:0/504746067 shutdown_connections 2024-04-02T07:48:26.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.896+0000 7f51f4600700 1 --2- 172.21.15.89:0/504746067 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f51cc06ae90 0x7f51cc06d350 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.896+0000 7f51f4600700 1 --2- 172.21.15.89:0/504746067 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51ec15d9d0 0x7f51ec0b4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.896+0000 7f51f4600700 1 --2- 172.21.15.89:0/504746067 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51ec0d16a0 0x7f51ec0b50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.896+0000 7f51f4600700 1 --2- 172.21.15.89:0/504746067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ec0d2370 0x7f51ec0c9430 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:26.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.896+0000 7f51f4600700 1 -- 172.21.15.89:0/504746067 >> 172.21.15.89:0/504746067 conn(0x7f51ec08aff0 msgr2=0x7f51ec15be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:26.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.896+0000 7f51f4600700 1 -- 172.21.15.89:0/504746067 shutdown_connections 2024-04-02T07:48:26.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:26.897+0000 7f51f4600700 1 -- 172.21.15.89:0/504746067 wait complete. 2024-04-02T07:48:26.916 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:24", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:26.917 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2024-04-02T00:00:00', 'created': '2024-04-02T07:37:24', 'first': '2024-04-02T07:38:00', 'last': '2024-04-02T07:48:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-02T07:48:26.917 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-02T07:48:26.917 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-02T07:48:26.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:26.918 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-04-02T07:48:26.962 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:26.963 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:26.963 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:26.963 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:26.963 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:26.963 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:26.963 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:26.963 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:26.963 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:26.963 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:26.963 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:26.964 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-02-07_38_00_UTC', 'scheduled-2024-04-02-07_39_00_UTC', 'scheduled-2024-04-02-07_40_00_UTC', 'scheduled-2024-04-02-07_41_00_UTC', 'scheduled-2024-04-02-07_42_00_UTC', 'scheduled-2024-04-02-07_43_00_UTC', 'scheduled-2024-04-02-07_44_00_UTC', 'scheduled-2024-04-02-07_45_00_UTC', 'scheduled-2024-04-02-07_46_00_UTC', 'scheduled-2024-04-02-07_47_00_UTC', 'scheduled-2024-04-02-07_48_00_UTC'] 2024-04-02T07:48:26.964 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2024-04-02T07:48:27.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.196+0000 7f84c8800700 1 Processor -- start 2024-04-02T07:48:27.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.196+0000 7f84c8800700 1 -- start start 2024-04-02T07:48:27.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.197+0000 7f84c8800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f84c00d16a0 0x7f84c00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:27.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.197+0000 7f84c8800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f84c00d2370 0x7f84c00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:27.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.197+0000 7f84c8800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c0161420 con 0x7f84c015d9d0 2024-04-02T07:48:27.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.197+0000 7f84c8800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c00d4340 con 0x7f84c00d2370 2024-04-02T07:48:27.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.197+0000 7f84c8800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c015edc0 con 0x7f84c00d16a0 2024-04-02T07:48:27.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.198+0000 7f84c6800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f84c00d2370 0x7f84c00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:27.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.198+0000 7f84c6800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f84c00d2370 0x7f84c00ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41984/0 (socket says 172.21.15.89:41984) 2024-04-02T07:48:27.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.198+0000 7f84c6800700 1 -- 172.21.15.89:0/1538882387 learned_addr learned my addr 172.21.15.89:0/1538882387 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:27.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.198+0000 7f84c5400700 1 --2- 172.21.15.89:0/1538882387 >> v2:172.21.15.113:3300/0 conn(0x7f84c00d16a0 0x7f84c00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:27.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.198+0000 7f84c6800700 1 -- 172.21.15.89:0/1538882387 >> v2:172.21.15.113:3300/0 conn(0x7f84c00d16a0 msgr2=0x7f84c00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:27.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.198+0000 7f84c6800700 1 --2- 172.21.15.89:0/1538882387 >> v2:172.21.15.113:3300/0 conn(0x7f84c00d16a0 0x7f84c00d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.198+0000 7f84c6800700 1 -- 172.21.15.89:0/1538882387 >> v1:172.21.15.113:6789/0 conn(0x7f84c015d9d0 legacy=0x7f84c00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:27.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.198+0000 7f84c6800700 1 -- 172.21.15.89:0/1538882387 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c0161700 con 0x7f84c00d2370 2024-04-02T07:48:27.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.198+0000 7f84c5400700 1 --2- 172.21.15.89:0/1538882387 >> v2:172.21.15.113:3300/0 conn(0x7f84c00d16a0 0x7f84c00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:27.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.198+0000 7f84c6800700 1 --2- 172.21.15.89:0/1538882387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c00d2370 0x7f84c00ce360 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f84c00d12f0 tx=0x7f84bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b592799d70958777 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:27.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.199+0000 7f84c4a00700 1 -- 172.21.15.89:0/1538882387 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc00e070 con 0x7f84c00d2370 2024-04-02T07:48:27.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.199+0000 7f84c4a00700 1 -- 172.21.15.89:0/1538882387 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84bc003a20 con 0x7f84c00d2370 2024-04-02T07:48:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.199+0000 7f84c4a00700 1 -- 172.21.15.89:0/1538882387 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc003dd0 con 0x7f84c00d2370 2024-04-02T07:48:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.199+0000 7f84c8800700 1 -- 172.21.15.89:0/1538882387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c00d2370 msgr2=0x7f84c00ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.199+0000 7f84c8800700 1 --2- 172.21.15.89:0/1538882387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c00d2370 0x7f84c00ce360 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f84c00d12f0 tx=0x7f84bc00b330 comp rx=0 tx=0).stop 2024-04-02T07:48:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.199+0000 7f84c8800700 1 -- 172.21.15.89:0/1538882387 shutdown_connections 2024-04-02T07:48:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.199+0000 7f84c8800700 1 --2- 172.21.15.89:0/1538882387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c00d2370 0x7f84c00ce360 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.199+0000 7f84c8800700 1 --2- 172.21.15.89:0/1538882387 >> v2:172.21.15.113:3300/0 conn(0x7f84c00d16a0 0x7f84c00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.199+0000 7f84c8800700 1 -- 172.21.15.89:0/1538882387 >> 172.21.15.89:0/1538882387 conn(0x7f84c008aff0 msgr2=0x7f84c00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.199+0000 7f84c8800700 1 -- 172.21.15.89:0/1538882387 shutdown_connections 2024-04-02T07:48:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.199+0000 7f84c8800700 1 -- 172.21.15.89:0/1538882387 wait complete. 2024-04-02T07:48:27.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.201+0000 7f84c8800700 1 Processor -- start 2024-04-02T07:48:27.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.201+0000 7f84c8800700 1 -- start start 2024-04-02T07:48:27.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.202+0000 7f84c8800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c00d16a0 0x7f84c01e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:27.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.202+0000 7f84c8800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84c00d2370 0x7f84c015c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.202+0000 7f84c8800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c015d9d0 0x7f84c00b8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.202+0000 7f84c8800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c00b6ba0 con 0x7f84c00d16a0 2024-04-02T07:48:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.202+0000 7f84c8800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c00b7ee0 con 0x7f84c00d2370 2024-04-02T07:48:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.202+0000 7f84c8800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c00b6e40 con 0x7f84c015d9d0 2024-04-02T07:48:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.202+0000 7f84c5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c00d16a0 0x7f84c01e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.202+0000 7f84c5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c00d16a0 0x7f84c01e7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42000/0 (socket says 172.21.15.89:42000) 2024-04-02T07:48:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.202+0000 7f84c5e00700 1 -- 172.21.15.89:0/3924808222 learned_addr learned my addr 172.21.15.89:0/3924808222 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.202+0000 7f84c5400700 1 --2- 172.21.15.89:0/3924808222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84c00d2370 0x7f84c015c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.203+0000 7f84c6800700 1 --2- 172.21.15.89:0/3924808222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c015d9d0 0x7f84c00b8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.203+0000 7f84c5e00700 1 -- 172.21.15.89:0/3924808222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c015d9d0 msgr2=0x7f84c00b8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.203+0000 7f84c5e00700 1 --2- 172.21.15.89:0/3924808222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c015d9d0 0x7f84c00b8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.203+0000 7f84c5e00700 1 -- 172.21.15.89:0/3924808222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84c00d2370 msgr2=0x7f84c015c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.203+0000 7f84c5e00700 1 --2- 172.21.15.89:0/3924808222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84c00d2370 0x7f84c015c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.203+0000 7f84c5e00700 1 -- 172.21.15.89:0/3924808222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc009b30 con 0x7f84c00d16a0 2024-04-02T07:48:27.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.203+0000 7f84c5e00700 1 --2- 172.21.15.89:0/3924808222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c00d16a0 0x7f84c01e7c90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f84b400b4a0 tx=0x7f84b400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:27.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.203+0000 7f84ba000700 1 -- 172.21.15.89:0/3924808222 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b4011830 con 0x7f84c00d16a0 2024-04-02T07:48:27.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.203+0000 7f84ba000700 1 -- 172.21.15.89:0/3924808222 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b4011990 con 0x7f84c00d16a0 2024-04-02T07:48:27.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.203+0000 7f84ba000700 1 -- 172.21.15.89:0/3924808222 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b4011d40 con 0x7f84c00d16a0 2024-04-02T07:48:27.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.203+0000 7f84c8800700 1 -- 172.21.15.89:0/3924808222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c00b7120 con 0x7f84c00d16a0 2024-04-02T07:48:27.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.203+0000 7f84c8800700 1 -- 172.21.15.89:0/3924808222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c00b76e0 con 0x7f84c00d16a0 2024-04-02T07:48:27.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.205+0000 7f84c8800700 1 -- 172.21.15.89:0/3924808222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c00d1ca0 con 0x7f84c00d16a0 2024-04-02T07:48:27.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.206+0000 7f84ba000700 1 -- 172.21.15.89:0/3924808222 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f84b401a400 con 0x7f84c00d16a0 2024-04-02T07:48:27.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.207+0000 7f84ba000700 1 --2- 172.21.15.89:0/3924808222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84a406aec0 0x7f84a406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:27.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.207+0000 7f84ba000700 1 -- 172.21.15.89:0/3924808222 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f84b4089480 con 0x7f84c00d16a0 2024-04-02T07:48:27.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.207+0000 7f84c5400700 1 --2- 172.21.15.89:0/3924808222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84a406aec0 0x7f84a406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:27.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.208+0000 7f84c5400700 1 --2- 172.21.15.89:0/3924808222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84a406aec0 0x7f84a406d380 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f84c00a1420 tx=0x7f84b0008040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:27.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.211+0000 7f84ba000700 1 -- 172.21.15.89:0/3924808222 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f84b4054270 con 0x7f84c00d16a0 2024-04-02T07:48:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.437+0000 7f84c8800700 1 -- 172.21.15.89:0/3924808222 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f84c0161fa0 con 0x7f84a406aec0 2024-04-02T07:48:27.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.448+0000 7f84ba000700 1 -- 172.21.15.89:0/3924808222 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f84b000c040 con 0x7f84a406aec0 2024-04-02T07:48:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:27.450 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:24", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:27.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.450+0000 7f84c8800700 1 -- 172.21.15.89:0/3924808222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84a406aec0 msgr2=0x7f84a406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:27.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.450+0000 7f84c8800700 1 --2- 172.21.15.89:0/3924808222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84a406aec0 0x7f84a406d380 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f84c00a1420 tx=0x7f84b0008040 comp rx=0 tx=0).stop 2024-04-02T07:48:27.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.450+0000 7f84c8800700 1 -- 172.21.15.89:0/3924808222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c00d16a0 msgr2=0x7f84c01e7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:27.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.450+0000 7f84c8800700 1 --2- 172.21.15.89:0/3924808222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c00d16a0 0x7f84c01e7c90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f84b400b4a0 tx=0x7f84b400b7b0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.450+0000 7f84c8800700 1 -- 172.21.15.89:0/3924808222 shutdown_connections 2024-04-02T07:48:27.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.450+0000 7f84c8800700 1 --2- 172.21.15.89:0/3924808222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84a406aec0 0x7f84a406d380 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.450+0000 7f84c8800700 1 --2- 172.21.15.89:0/3924808222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84c015d9d0 0x7f84c00b8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.450+0000 7f84c8800700 1 --2- 172.21.15.89:0/3924808222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84c00d16a0 0x7f84c01e7c90 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.450+0000 7f84c8800700 1 --2- 172.21.15.89:0/3924808222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84c00d2370 0x7f84c015c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.450+0000 7f84c8800700 1 -- 172.21.15.89:0/3924808222 >> 172.21.15.89:0/3924808222 conn(0x7f84c008aff0 msgr2=0x7f84c015e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.450+0000 7f84c8800700 1 -- 172.21.15.89:0/3924808222 shutdown_connections 2024-04-02T07:48:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.450+0000 7f84c8800700 1 -- 172.21.15.89:0/3924808222 wait complete. 2024-04-02T07:48:27.470 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:24", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:27.470 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2024-04-02T00:00:00', 'created': '2024-04-02T07:37:24', 'first': '2024-04-02T07:38:00', 'last': '2024-04-02T07:48:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-02T07:48:27.470 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-02T07:48:27.470 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-02T07:48:27.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:27.471 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-04-02T07:48:27.516 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:27.516 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:27.516 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:27.516 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:27.516 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:27.516 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:27.516 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:27.516 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:27.516 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:27.516 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:27.516 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:27.517 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-02-07_38_00_UTC', 'scheduled-2024-04-02-07_39_00_UTC', 'scheduled-2024-04-02-07_40_00_UTC', 'scheduled-2024-04-02-07_41_00_UTC', 'scheduled-2024-04-02-07_42_00_UTC', 'scheduled-2024-04-02-07_43_00_UTC', 'scheduled-2024-04-02-07_44_00_UTC', 'scheduled-2024-04-02-07_45_00_UTC', 'scheduled-2024-04-02-07_46_00_UTC', 'scheduled-2024-04-02-07_47_00_UTC', 'scheduled-2024-04-02-07_48_00_UTC'] 2024-04-02T07:48:27.518 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2024-04-02T07:48:27.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.748+0000 7ffbd7c00700 1 Processor -- start 2024-04-02T07:48:27.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.749+0000 7ffbd7c00700 1 -- start start 2024-04-02T07:48:27.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.750+0000 7ffbd7c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd00d21f0 0x7ffbd00ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:27.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.750+0000 7ffbd7c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd00ce270 con 0x7ffbd01611b0 2024-04-02T07:48:27.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.750+0000 7ffbd7c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd015b600 con 0x7ffbd00cec40 2024-04-02T07:48:27.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.750+0000 7ffbd7c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd00cf0b0 con 0x7ffbd00d21f0 2024-04-02T07:48:27.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.750+0000 7ffbd5200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ffbd01611b0 0x7ffbd015ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32786/0 (socket says 172.21.15.89:32786) 2024-04-02T07:48:27.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.750+0000 7ffbd5200700 1 -- 172.21.15.89:0/1856195350 learned_addr learned my addr 172.21.15.89:0/1856195350 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:27.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.750+0000 7ffbcfe00700 1 --2- 172.21.15.89:0/1856195350 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd00d21f0 0x7ffbd00ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:27.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.750+0000 7ffbcf400700 1 -- 172.21.15.89:0/1856195350 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981071952 0 0) 0x7ffbd00ce270 con 0x7ffbd01611b0 2024-04-02T07:48:27.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.750+0000 7ffbcf400700 1 -- 172.21.15.89:0/1856195350 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb80035b0 con 0x7ffbd01611b0 2024-04-02T07:48:27.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbcf400700 1 -- 172.21.15.89:0/1856195350 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2232670637 0 0) 0x7ffbb80035b0 con 0x7ffbd01611b0 2024-04-02T07:48:27.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbcf400700 1 -- 172.21.15.89:0/1856195350 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd00d21f0 msgr2=0x7ffbd00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:27.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbcf400700 1 --2- 172.21.15.89:0/1856195350 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd00d21f0 0x7ffbd00ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbcf400700 1 -- 172.21.15.89:0/1856195350 >> v1:172.21.15.113:6789/0 conn(0x7ffbd00cec40 legacy=0x7ffbd0161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:27.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbcf400700 1 -- 172.21.15.89:0/1856195350 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd00d2760 con 0x7ffbd01611b0 2024-04-02T07:48:27.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbcf400700 1 -- 172.21.15.89:0/1856195350 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7ffbc4001670 con 0x7ffbd01611b0 2024-04-02T07:48:27.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbcf400700 1 -- 172.21.15.89:0/1856195350 >> v1:172.21.15.89:6789/0 conn(0x7ffbd01611b0 legacy=0x7ffbd015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:27.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbcf400700 1 --2- 172.21.15.89:0/1856195350 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd00d21f0 0x7ffbb8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:27.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbcf400700 1 --2- 172.21.15.89:0/1856195350 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbb8003fb0 0x7ffbb8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:27.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbcf400700 1 --2- 172.21.15.89:0/1856195350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbb8004d90 0x7ffbb8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:27.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbcf400700 1 -- 172.21.15.89:0/1856195350 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd00cf0b0 con 0x7ffbb8004d90 2024-04-02T07:48:27.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbcf400700 1 -- 172.21.15.89:0/1856195350 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb800baf0 con 0x7ffbb8003fb0 2024-04-02T07:48:27.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbcf400700 1 -- 172.21.15.89:0/1856195350 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb800bcb0 con 0x7ffbd00d21f0 2024-04-02T07:48:27.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbd5c00700 1 --2- 172.21.15.89:0/1856195350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbb8004d90 0x7ffbb8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:27.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbcfe00700 1 --2- 172.21.15.89:0/1856195350 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbb8003fb0 0x7ffbb8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:27.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.751+0000 7ffbd5200700 1 --2- 172.21.15.89:0/1856195350 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd00d21f0 0x7ffbb8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:27.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.752+0000 7ffbd5c00700 1 -- 172.21.15.89:0/1856195350 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd00d21f0 msgr2=0x7ffbb8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:27.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.752+0000 7ffbd5c00700 1 --2- 172.21.15.89:0/1856195350 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbd00d21f0 0x7ffbb8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.752+0000 7ffbd5c00700 1 -- 172.21.15.89:0/1856195350 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbb8003fb0 msgr2=0x7ffbb8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:27.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.752+0000 7ffbd5c00700 1 --2- 172.21.15.89:0/1856195350 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbb8003fb0 0x7ffbb8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.752+0000 7ffbd5c00700 1 -- 172.21.15.89:0/1856195350 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb80035b0 con 0x7ffbb8004d90 2024-04-02T07:48:27.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.752+0000 7ffbd5c00700 1 --2- 172.21.15.89:0/1856195350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbb8004d90 0x7ffbb8005210 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800c240 tx=0x7ffbb8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d94d625b6a19ef server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:27.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.752+0000 7ffbcf400700 1 -- 172.21.15.89:0/1856195350 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8001df0 con 0x7ffbb8004d90 2024-04-02T07:48:27.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.752+0000 7ffbcf400700 1 -- 172.21.15.89:0/1856195350 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc800cda0 con 0x7ffbb8004d90 2024-04-02T07:48:27.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.752+0000 7ffbcf400700 1 -- 172.21.15.89:0/1856195350 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8003c40 con 0x7ffbb8004d90 2024-04-02T07:48:27.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.753+0000 7ffbd7c00700 1 -- 172.21.15.89:0/1856195350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbb8004d90 msgr2=0x7ffbb8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:27.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.753+0000 7ffbd7c00700 1 --2- 172.21.15.89:0/1856195350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbb8004d90 0x7ffbb8005210 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800c240 tx=0x7ffbb8005880 comp rx=0 tx=0).stop 2024-04-02T07:48:27.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.753+0000 7ffbd5c00700 1 -- 172.21.15.89:0/1856195350 reap_dead start 2024-04-02T07:48:27.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.753+0000 7ffbd7c00700 1 -- 172.21.15.89:0/1856195350 shutdown_connections 2024-04-02T07:48:27.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.753+0000 7ffbd7c00700 1 -- 172.21.15.89:0/1856195350 >> 172.21.15.89:0/1856195350 conn(0x7ffbd008aff0 msgr2=0x7ffbd00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:27.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.753+0000 7ffbd7c00700 1 -- 172.21.15.89:0/1856195350 shutdown_connections 2024-04-02T07:48:27.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.753+0000 7ffbd7c00700 1 -- 172.21.15.89:0/1856195350 wait complete. 2024-04-02T07:48:27.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.755+0000 7ffbd7c00700 1 Processor -- start 2024-04-02T07:48:27.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.755+0000 7ffbd7c00700 1 -- start start 2024-04-02T07:48:27.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd7c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbb8004d90 0x7ffbd0149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:27.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd7c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbd00b8e00 0x7ffbd00c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:27.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd7c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd00c9a10 0x7ffbd006c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:27.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd7c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd00c8780 con 0x7ffbd00c9a10 2024-04-02T07:48:27.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd7c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd00b9280 con 0x7ffbd00b8e00 2024-04-02T07:48:27.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd7c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd00c8970 con 0x7ffbb8004d90 2024-04-02T07:48:27.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd5c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd00c9a10 0x7ffbd006c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:27.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd5c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd00c9a10 0x7ffbd006c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42024/0 (socket says 172.21.15.89:42024) 2024-04-02T07:48:27.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd5c00700 1 -- 172.21.15.89:0/4112856301 learned_addr learned my addr 172.21.15.89:0/4112856301 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:27.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd5200700 1 --2- 172.21.15.89:0/4112856301 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbb8004d90 0x7ffbd0149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:27.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbcfe00700 1 --2- 172.21.15.89:0/4112856301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbd00b8e00 0x7ffbd00c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:27.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd5c00700 1 -- 172.21.15.89:0/4112856301 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbb8004d90 msgr2=0x7ffbd0149cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:27.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd5c00700 1 --2- 172.21.15.89:0/4112856301 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbb8004d90 0x7ffbd0149cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd5c00700 1 -- 172.21.15.89:0/4112856301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbd00b8e00 msgr2=0x7ffbd00c8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:27.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd5c00700 1 --2- 172.21.15.89:0/4112856301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbd00b8e00 0x7ffbd00c8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:27.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.756+0000 7ffbd5c00700 1 -- 172.21.15.89:0/4112856301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd00c8c50 con 0x7ffbd00c9a10 2024-04-02T07:48:27.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.757+0000 7ffbd5c00700 1 --2- 172.21.15.89:0/4112856301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd00c9a10 0x7ffbd006c340 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8004000 tx=0x7ffbc80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:27.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.757+0000 7ffbccc00700 1 -- 172.21.15.89:0/4112856301 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8003c10 con 0x7ffbd00c9a10 2024-04-02T07:48:27.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.757+0000 7ffbccc00700 1 -- 172.21.15.89:0/4112856301 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc8003d70 con 0x7ffbd00c9a10 2024-04-02T07:48:27.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.757+0000 7ffbccc00700 1 -- 172.21.15.89:0/4112856301 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8005310 con 0x7ffbd00c9a10 2024-04-02T07:48:27.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.757+0000 7ffbd7c00700 1 -- 172.21.15.89:0/4112856301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd00c8f30 con 0x7ffbd00c9a10 2024-04-02T07:48:27.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.757+0000 7ffbd7c00700 1 -- 172.21.15.89:0/4112856301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd00c94f0 con 0x7ffbd00c9a10 2024-04-02T07:48:27.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.758+0000 7ffbd7c00700 1 -- 172.21.15.89:0/4112856301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd0067720 con 0x7ffbd00c9a10 2024-04-02T07:48:27.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.759+0000 7ffbccc00700 1 -- 172.21.15.89:0/4112856301 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7ffbc8019070 con 0x7ffbd00c9a10 2024-04-02T07:48:27.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.760+0000 7ffbccc00700 1 --2- 172.21.15.89:0/4112856301 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffbb006ae90 0x7ffbb006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:27.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.760+0000 7ffbccc00700 1 -- 172.21.15.89:0/4112856301 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7ffbc808a0f0 con 0x7ffbd00c9a10 2024-04-02T07:48:27.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.760+0000 7ffbd5200700 1 --2- 172.21.15.89:0/4112856301 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffbb006ae90 0x7ffbb006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:27.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.761+0000 7ffbd5200700 1 --2- 172.21.15.89:0/4112856301 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffbb006ae90 0x7ffbb006d350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00c6c90 tx=0x7ffbc4001170 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:27.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.764+0000 7ffbccc00700 1 -- 172.21.15.89:0/4112856301 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffbc8055140 con 0x7ffbd00c9a10 2024-04-02T07:48:27.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.988+0000 7ffbd7c00700 1 -- 172.21.15.89:0/4112856301 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffbd00cdf00 con 0x7ffbb006ae90 2024-04-02T07:48:28.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:27.998+0000 7ffbccc00700 1 -- 172.21.15.89:0/4112856301 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7ffbc4007c10 con 0x7ffbb006ae90 2024-04-02T07:48:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:28.000 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:25", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.001+0000 7ffbd7c00700 1 -- 172.21.15.89:0/4112856301 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffbb006ae90 msgr2=0x7ffbb006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.001+0000 7ffbd7c00700 1 --2- 172.21.15.89:0/4112856301 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffbb006ae90 0x7ffbb006d350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00c6c90 tx=0x7ffbc4001170 comp rx=0 tx=0).stop 2024-04-02T07:48:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.001+0000 7ffbd7c00700 1 -- 172.21.15.89:0/4112856301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd00c9a10 msgr2=0x7ffbd006c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.001+0000 7ffbd7c00700 1 --2- 172.21.15.89:0/4112856301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd00c9a10 0x7ffbd006c340 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8004000 tx=0x7ffbc80014a0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.001+0000 7ffbd7c00700 1 -- 172.21.15.89:0/4112856301 shutdown_connections 2024-04-02T07:48:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.001+0000 7ffbd7c00700 1 --2- 172.21.15.89:0/4112856301 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffbb006ae90 0x7ffbb006d350 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.001+0000 7ffbd7c00700 1 --2- 172.21.15.89:0/4112856301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd00c9a10 0x7ffbd006c340 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.001+0000 7ffbd7c00700 1 --2- 172.21.15.89:0/4112856301 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffbb8004d90 0x7ffbd0149cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.001+0000 7ffbd7c00700 1 --2- 172.21.15.89:0/4112856301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbd00b8e00 0x7ffbd00c8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.001+0000 7ffbd7c00700 1 -- 172.21.15.89:0/4112856301 >> 172.21.15.89:0/4112856301 conn(0x7ffbd008aff0 msgr2=0x7ffbd015c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:28.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.001+0000 7ffbd7c00700 1 -- 172.21.15.89:0/4112856301 shutdown_connections 2024-04-02T07:48:28.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.001+0000 7ffbd7c00700 1 -- 172.21.15.89:0/4112856301 wait complete. 2024-04-02T07:48:28.021 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:25", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:28.021 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2024-04-02T00:00:00', 'created': '2024-04-02T07:37:25', 'first': '2024-04-02T07:38:00', 'last': '2024-04-02T07:48:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-02T07:48:28.021 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-02T07:48:28.021 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-02T07:48:28.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:28.021 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-04-02T07:48:28.066 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:28.066 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:28.066 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:28.066 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:28.066 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:28.066 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:28.067 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:28.067 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:28.067 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:28.067 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:28.067 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:28.068 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-02-07_38_00_UTC', 'scheduled-2024-04-02-07_39_00_UTC', 'scheduled-2024-04-02-07_40_00_UTC', 'scheduled-2024-04-02-07_41_00_UTC', 'scheduled-2024-04-02-07_42_00_UTC', 'scheduled-2024-04-02-07_43_00_UTC', 'scheduled-2024-04-02-07_44_00_UTC', 'scheduled-2024-04-02-07_45_00_UTC', 'scheduled-2024-04-02-07_46_00_UTC', 'scheduled-2024-04-02-07_47_00_UTC', 'scheduled-2024-04-02-07_48_00_UTC'] 2024-04-02T07:48:28.068 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2024-04-02T07:48:28.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.298+0000 7f59e9000700 1 Processor -- start 2024-04-02T07:48:28.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.299+0000 7f59e9000700 1 -- start start 2024-04-02T07:48:28.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e9000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f59e415d9d0 0x7f59e40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:28.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e9000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f59e40d16a0 0x7f59e40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:28.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e9000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59e40d2370 0x7f59e415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:28.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e9000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e40d2820 con 0x7f59e415d9d0 2024-04-02T07:48:28.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e9000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e4161420 con 0x7f59e40d16a0 2024-04-02T07:48:28.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e9000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e4161700 con 0x7f59e40d2370 2024-04-02T07:48:28.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f59e415d9d0 0x7f59e40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:28.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f59e415d9d0 0x7f59e40d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42034/0 (socket says 172.21.15.89:42034) 2024-04-02T07:48:28.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e1e00700 1 -- 172.21.15.89:0/3743341821 learned_addr learned my addr 172.21.15.89:0/3743341821 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:28.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e2800700 1 --2- 172.21.15.89:0/3743341821 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59e40d2370 0x7f59e415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:28.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e1400700 1 --2- 172.21.15.89:0/3743341821 >> v2:172.21.15.113:3300/0 conn(0x7f59e40d16a0 0x7f59e40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:28.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e1e00700 1 -- 172.21.15.89:0/3743341821 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59e40d2370 msgr2=0x7f59e415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:28.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e1e00700 1 --2- 172.21.15.89:0/3743341821 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59e40d2370 0x7f59e415ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e1e00700 1 -- 172.21.15.89:0/3743341821 >> v2:172.21.15.113:3300/0 conn(0x7f59e40d16a0 msgr2=0x7f59e40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:28.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e1e00700 1 --2- 172.21.15.89:0/3743341821 >> v2:172.21.15.113:3300/0 conn(0x7f59e40d16a0 0x7f59e40d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.300+0000 7f59e1e00700 1 -- 172.21.15.89:0/3743341821 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e41619e0 con 0x7f59e415d9d0 2024-04-02T07:48:28.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.301+0000 7f59e1e00700 1 --2- 172.21.15.89:0/3743341821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e415d9d0 0x7f59e40d3c30 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f59e40d12f0 tx=0x7f59d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2d295961397e2c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:28.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.301+0000 7f59e0a00700 1 -- 172.21.15.89:0/3743341821 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d400b870 con 0x7f59e415d9d0 2024-04-02T07:48:28.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.301+0000 7f59e0a00700 1 -- 172.21.15.89:0/3743341821 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d400b9d0 con 0x7f59e415d9d0 2024-04-02T07:48:28.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.301+0000 7f59e0a00700 1 -- 172.21.15.89:0/3743341821 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d400bd60 con 0x7f59e415d9d0 2024-04-02T07:48:28.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.301+0000 7f59e9000700 1 -- 172.21.15.89:0/3743341821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e415d9d0 msgr2=0x7f59e40d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:28.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.301+0000 7f59e9000700 1 --2- 172.21.15.89:0/3743341821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e415d9d0 0x7f59e40d3c30 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f59e40d12f0 tx=0x7f59d4009bd0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.302+0000 7f59e9000700 1 -- 172.21.15.89:0/3743341821 shutdown_connections 2024-04-02T07:48:28.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.302+0000 7f59e9000700 1 --2- 172.21.15.89:0/3743341821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e415d9d0 0x7f59e40d3c30 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.302+0000 7f59e9000700 1 --2- 172.21.15.89:0/3743341821 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59e40d2370 0x7f59e415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.302+0000 7f59e9000700 1 --2- 172.21.15.89:0/3743341821 >> v2:172.21.15.113:3300/0 conn(0x7f59e40d16a0 0x7f59e40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.302+0000 7f59e9000700 1 -- 172.21.15.89:0/3743341821 >> 172.21.15.89:0/3743341821 conn(0x7f59e408aff0 msgr2=0x7f59e40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:28.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.302+0000 7f59e9000700 1 -- 172.21.15.89:0/3743341821 shutdown_connections 2024-04-02T07:48:28.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.302+0000 7f59e9000700 1 -- 172.21.15.89:0/3743341821 wait complete. 2024-04-02T07:48:28.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.303+0000 7f59e9000700 1 Processor -- start 2024-04-02T07:48:28.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.303+0000 7f59e9000700 1 -- start start 2024-04-02T07:48:28.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.304+0000 7f59e9000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59e40d16a0 0x7f59e4160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:28.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.304+0000 7f59e9000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59e40d2370 0x7f59e4151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:28.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.304+0000 7f59e9000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e415d9d0 0x7f59e40b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:28.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.304+0000 7f59e9000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e4151e70 con 0x7f59e415d9d0 2024-04-02T07:48:28.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.304+0000 7f59e9000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e40b3ff0 con 0x7f59e40d2370 2024-04-02T07:48:28.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.304+0000 7f59e9000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e40b5000 con 0x7f59e40d16a0 2024-04-02T07:48:28.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.305+0000 7f59e2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e415d9d0 0x7f59e40b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:28.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.305+0000 7f59e2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e415d9d0 0x7f59e40b6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42046/0 (socket says 172.21.15.89:42046) 2024-04-02T07:48:28.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.305+0000 7f59e2800700 1 -- 172.21.15.89:0/2287729688 learned_addr learned my addr 172.21.15.89:0/2287729688 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:28.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.305+0000 7f59e1e00700 1 --2- 172.21.15.89:0/2287729688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59e40d16a0 0x7f59e4160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.305+0000 7f59e1400700 1 --2- 172.21.15.89:0/2287729688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59e40d2370 0x7f59e4151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.305+0000 7f59e2800700 1 -- 172.21.15.89:0/2287729688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59e40d16a0 msgr2=0x7f59e4160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.305+0000 7f59e2800700 1 --2- 172.21.15.89:0/2287729688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59e40d16a0 0x7f59e4160eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.305+0000 7f59e2800700 1 -- 172.21.15.89:0/2287729688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59e40d2370 msgr2=0x7f59e4151930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.305+0000 7f59e2800700 1 --2- 172.21.15.89:0/2287729688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59e40d2370 0x7f59e4151930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.305+0000 7f59e2800700 1 -- 172.21.15.89:0/2287729688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d4009550 con 0x7f59e415d9d0 2024-04-02T07:48:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.305+0000 7f59e2800700 1 --2- 172.21.15.89:0/2287729688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e415d9d0 0x7f59e40b6ff0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f59dc007e40 tx=0x7f59dc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.306+0000 7f59da000700 1 -- 172.21.15.89:0/2287729688 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc00eea0 con 0x7f59e415d9d0 2024-04-02T07:48:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.306+0000 7f59da000700 1 -- 172.21.15.89:0/2287729688 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59dc014420 con 0x7f59e415d9d0 2024-04-02T07:48:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.306+0000 7f59e9000700 1 -- 172.21.15.89:0/2287729688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e40b4240 con 0x7f59e415d9d0 2024-04-02T07:48:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.306+0000 7f59da000700 1 -- 172.21.15.89:0/2287729688 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc0147b0 con 0x7f59e415d9d0 2024-04-02T07:48:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.306+0000 7f59e9000700 1 -- 172.21.15.89:0/2287729688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e40b4800 con 0x7f59e415d9d0 2024-04-02T07:48:28.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.307+0000 7f59e9000700 1 -- 172.21.15.89:0/2287729688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e40d1ca0 con 0x7f59e415d9d0 2024-04-02T07:48:28.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.308+0000 7f59da000700 1 -- 172.21.15.89:0/2287729688 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f59dc014910 con 0x7f59e415d9d0 2024-04-02T07:48:28.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.309+0000 7f59da000700 1 --2- 172.21.15.89:0/2287729688 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f59c406ae90 0x7f59c406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:28.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.309+0000 7f59da000700 1 -- 172.21.15.89:0/2287729688 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f59dc089fd0 con 0x7f59e415d9d0 2024-04-02T07:48:28.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.309+0000 7f59e1e00700 1 --2- 172.21.15.89:0/2287729688 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f59c406ae90 0x7f59c406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:28.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.310+0000 7f59e1e00700 1 --2- 172.21.15.89:0/2287729688 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f59c406ae90 0x7f59c406d350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009c10 tx=0x7f59d4008030 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:28.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.313+0000 7f59da000700 1 -- 172.21.15.89:0/2287729688 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59dc054db0 con 0x7f59e415d9d0 2024-04-02T07:48:28.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.541+0000 7f59e9000700 1 -- 172.21.15.89:0/2287729688 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59e4161fa0 con 0x7f59c406ae90 2024-04-02T07:48:28.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.550+0000 7f59da000700 1 -- 172.21.15.89:0/2287729688 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f59d401f070 con 0x7f59c406ae90 2024-04-02T07:48:28.553 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:28.553 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:25", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:28.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.553+0000 7f59e9000700 1 -- 172.21.15.89:0/2287729688 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f59c406ae90 msgr2=0x7f59c406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:28.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.553+0000 7f59e9000700 1 --2- 172.21.15.89:0/2287729688 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f59c406ae90 0x7f59c406d350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009c10 tx=0x7f59d4008030 comp rx=0 tx=0).stop 2024-04-02T07:48:28.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.553+0000 7f59e9000700 1 -- 172.21.15.89:0/2287729688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e415d9d0 msgr2=0x7f59e40b6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:28.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.553+0000 7f59e9000700 1 --2- 172.21.15.89:0/2287729688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e415d9d0 0x7f59e40b6ff0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f59dc007e40 tx=0x7f59dc00e3c0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.553+0000 7f59e9000700 1 -- 172.21.15.89:0/2287729688 shutdown_connections 2024-04-02T07:48:28.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.553+0000 7f59e9000700 1 --2- 172.21.15.89:0/2287729688 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f59c406ae90 0x7f59c406d350 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.553+0000 7f59e9000700 1 --2- 172.21.15.89:0/2287729688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e415d9d0 0x7f59e40b6ff0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.553+0000 7f59e9000700 1 --2- 172.21.15.89:0/2287729688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59e40d16a0 0x7f59e4160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.553+0000 7f59e9000700 1 --2- 172.21.15.89:0/2287729688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59e40d2370 0x7f59e4151930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.553+0000 7f59e9000700 1 -- 172.21.15.89:0/2287729688 >> 172.21.15.89:0/2287729688 conn(0x7f59e408aff0 msgr2=0x7f59e40c99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:28.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.553+0000 7f59e9000700 1 -- 172.21.15.89:0/2287729688 shutdown_connections 2024-04-02T07:48:28.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.553+0000 7f59e9000700 1 -- 172.21.15.89:0/2287729688 wait complete. 2024-04-02T07:48:28.574 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:25", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:28.574 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2024-04-02T00:00:00', 'created': '2024-04-02T07:37:25', 'first': '2024-04-02T07:38:00', 'last': '2024-04-02T07:48:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-02T07:48:28.574 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-02T07:48:28.574 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-02T07:48:28.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:28.574 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-04-02T07:48:28.619 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:28.619 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:28.620 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:28.620 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:28.620 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:28.620 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:28.620 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:28.620 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:28.620 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:28.620 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:28.620 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:28.621 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-02-07_38_00_UTC', 'scheduled-2024-04-02-07_39_00_UTC', 'scheduled-2024-04-02-07_40_00_UTC', 'scheduled-2024-04-02-07_41_00_UTC', 'scheduled-2024-04-02-07_42_00_UTC', 'scheduled-2024-04-02-07_43_00_UTC', 'scheduled-2024-04-02-07_44_00_UTC', 'scheduled-2024-04-02-07_45_00_UTC', 'scheduled-2024-04-02-07_46_00_UTC', 'scheduled-2024-04-02-07_47_00_UTC', 'scheduled-2024-04-02-07_48_00_UTC'] 2024-04-02T07:48:28.621 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2024-04-02T07:48:28.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.853+0000 7f22a9600700 1 Processor -- start 2024-04-02T07:48:28.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.853+0000 7f22a9600700 1 -- start start 2024-04-02T07:48:28.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.854+0000 7f22a9600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f22a415dad0 0x7f22a40ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:28.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.854+0000 7f22a9600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40d1630 0x7f22a40d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:28.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.854+0000 7f22a9600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a415f820 con 0x7f22a40d1fd0 2024-04-02T07:48:28.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.854+0000 7f22a9600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a40cfcc0 con 0x7f22a415dad0 2024-04-02T07:48:28.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.855+0000 7f22a9600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40cd9f0 con 0x7f22a40d1630 2024-04-02T07:48:28.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.855+0000 7f22a2000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40d1630 0x7f22a40d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:28.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.855+0000 7f22a2000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40d1630 0x7f22a40d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:46412/0 (socket says 172.21.15.89:46412) 2024-04-02T07:48:28.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.855+0000 7f22a2000700 1 -- 172.21.15.89:0/636137322 learned_addr learned my addr 172.21.15.89:0/636137322 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:28.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.855+0000 7f22a2a00700 1 --2- 172.21.15.89:0/636137322 >> v2:172.21.15.113:3300/0 conn(0x7f22a415dad0 0x7f22a40ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:28.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.856+0000 7f22a2000700 1 -- 172.21.15.89:0/636137322 >> v2:172.21.15.113:3300/0 conn(0x7f22a415dad0 msgr2=0x7f22a40ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:28.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.856+0000 7f22a2000700 1 --2- 172.21.15.89:0/636137322 >> v2:172.21.15.113:3300/0 conn(0x7f22a415dad0 0x7f22a40ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.856+0000 7f22a2000700 1 -- 172.21.15.89:0/636137322 >> v1:172.21.15.113:6789/0 conn(0x7f22a40d1fd0 legacy=0x7f22a415deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:28.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.856+0000 7f22a2000700 1 -- 172.21.15.89:0/636137322 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a40d2460 con 0x7f22a40d1630 2024-04-02T07:48:28.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.856+0000 7f22a1600700 1 -- 172.21.15.89:0/636137322 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482923644 0 0) 0x7f22a415f820 con 0x7f22a40d1fd0 2024-04-02T07:48:28.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.856+0000 7f22a2000700 1 --2- 172.21.15.89:0/636137322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40d1630 0x7f22a40d1a90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f22900098f0 tx=0x7f2290009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a18338fe561b2648 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:28.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.856+0000 7f22a1600700 1 -- 172.21.15.89:0/636137322 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290011830 con 0x7f22a40d1630 2024-04-02T07:48:28.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.856+0000 7f22a1600700 1 -- 172.21.15.89:0/636137322 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2290011990 con 0x7f22a40d1630 2024-04-02T07:48:28.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.856+0000 7f22a1600700 1 -- 172.21.15.89:0/636137322 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290011d20 con 0x7f22a40d1630 2024-04-02T07:48:28.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.857+0000 7f22a9600700 1 -- 172.21.15.89:0/636137322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40d1630 msgr2=0x7f22a40d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:28.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.857+0000 7f22a9600700 1 --2- 172.21.15.89:0/636137322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40d1630 0x7f22a40d1a90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f22900098f0 tx=0x7f2290009c00 comp rx=0 tx=0).stop 2024-04-02T07:48:28.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.857+0000 7f22a9600700 1 -- 172.21.15.89:0/636137322 shutdown_connections 2024-04-02T07:48:28.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.857+0000 7f22a9600700 1 --2- 172.21.15.89:0/636137322 >> v2:172.21.15.113:3300/0 conn(0x7f22a415dad0 0x7f22a40ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.857+0000 7f22a9600700 1 --2- 172.21.15.89:0/636137322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40d1630 0x7f22a40d1a90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.857+0000 7f22a9600700 1 -- 172.21.15.89:0/636137322 >> 172.21.15.89:0/636137322 conn(0x7f22a408aff0 msgr2=0x7f22a40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:28.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.857+0000 7f22a9600700 1 -- 172.21.15.89:0/636137322 shutdown_connections 2024-04-02T07:48:28.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.857+0000 7f22a9600700 1 -- 172.21.15.89:0/636137322 wait complete. 2024-04-02T07:48:28.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.859+0000 7f22a9600700 1 Processor -- start 2024-04-02T07:48:28.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.859+0000 7f22a9600700 1 -- start start 2024-04-02T07:48:28.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a9600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40d1630 0x7f22a40cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:28.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a9600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40d1fd0 0x7f22a406cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:28.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a9600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22a415dad0 0x7f22a414e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:28.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a9600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40b6370 con 0x7f22a40d1fd0 2024-04-02T07:48:28.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a9600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40b6580 con 0x7f22a415dad0 2024-04-02T07:48:28.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a9600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40b6860 con 0x7f22a40d1630 2024-04-02T07:48:28.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40d1fd0 0x7f22a406cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:28.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40d1fd0 0x7f22a406cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36172/0 (socket says 172.21.15.89:36172) 2024-04-02T07:48:28.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a2000700 1 -- 172.21.15.89:0/1371995001 learned_addr learned my addr 172.21.15.89:0/1371995001 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:28.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a2a00700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40d1630 0x7f22a40cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:28.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a3400700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22a415dad0 0x7f22a414e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:28.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a2000700 1 -- 172.21.15.89:0/1371995001 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40d1630 msgr2=0x7f22a40cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:28.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a2000700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40d1630 0x7f22a40cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a2000700 1 -- 172.21.15.89:0/1371995001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22a415dad0 msgr2=0x7f22a414e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:28.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a2000700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22a415dad0 0x7f22a414e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:28.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a2000700 1 -- 172.21.15.89:0/1371995001 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2290009550 con 0x7f22a40d1fd0 2024-04-02T07:48:28.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.860+0000 7f22a2a00700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40d1630 0x7f22a40cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:48:28.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.861+0000 7f22a2000700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40d1fd0 0x7f22a406cba0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2290009c40 tx=0x7f2290004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:28.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.861+0000 7f229aa00700 1 -- 172.21.15.89:0/1371995001 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290004480 con 0x7f22a40d1fd0 2024-04-02T07:48:28.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.861+0000 7f229aa00700 1 -- 172.21.15.89:0/1371995001 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229001f070 con 0x7f22a40d1fd0 2024-04-02T07:48:28.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.861+0000 7f22a9600700 1 -- 172.21.15.89:0/1371995001 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a40b6b40 con 0x7f22a40d1fd0 2024-04-02T07:48:28.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.861+0000 7f229aa00700 1 -- 172.21.15.89:0/1371995001 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22900038c0 con 0x7f22a40d1fd0 2024-04-02T07:48:28.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.861+0000 7f22a9600700 1 -- 172.21.15.89:0/1371995001 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a40b7100 con 0x7f22a40d1fd0 2024-04-02T07:48:28.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.862+0000 7f22a9600700 1 -- 172.21.15.89:0/1371995001 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a4067720 con 0x7f22a40d1fd0 2024-04-02T07:48:28.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.863+0000 7f229aa00700 1 -- 172.21.15.89:0/1371995001 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f2290021030 con 0x7f22a40d1fd0 2024-04-02T07:48:28.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.864+0000 7f229aa00700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f228406ae90 0x7f228406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:28.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.864+0000 7f229aa00700 1 -- 172.21.15.89:0/1371995001 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f229008acd0 con 0x7f22a40d1fd0 2024-04-02T07:48:28.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.864+0000 7f22a2a00700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f228406ae90 0x7f228406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:28.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.865+0000 7f22a2a00700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f228406ae90 0x7f228406d350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2294009b90 tx=0x7f2294008040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:28.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:28.868+0000 7f229aa00700 1 -- 172.21.15.89:0/1371995001 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2290055aa0 con 0x7f22a40d1fd0 2024-04-02T07:48:29.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.096+0000 7f22a9600700 1 -- 172.21.15.89:0/1371995001 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f22a40cf9e0 con 0x7f228406ae90 2024-04-02T07:48:29.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.105+0000 7f229aa00700 1 -- 172.21.15.89:0/1371995001 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f22940097e0 con 0x7f228406ae90 2024-04-02T07:48:29.107 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:29.108 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:26", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:29.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.108+0000 7f22a9600700 1 -- 172.21.15.89:0/1371995001 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f228406ae90 msgr2=0x7f228406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:29.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.108+0000 7f22a9600700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f228406ae90 0x7f228406d350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2294009b90 tx=0x7f2294008040 comp rx=0 tx=0).stop 2024-04-02T07:48:29.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.108+0000 7f22a9600700 1 -- 172.21.15.89:0/1371995001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40d1fd0 msgr2=0x7f22a406cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:29.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.108+0000 7f22a9600700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40d1fd0 0x7f22a406cba0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2290009c40 tx=0x7f2290004750 comp rx=0 tx=0).stop 2024-04-02T07:48:29.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.108+0000 7f22a9600700 1 -- 172.21.15.89:0/1371995001 shutdown_connections 2024-04-02T07:48:29.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.108+0000 7f22a9600700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f228406ae90 0x7f228406d350 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.108+0000 7f22a9600700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22a415dad0 0x7f22a414e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.108+0000 7f22a9600700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40d1630 0x7f22a40cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.108+0000 7f22a9600700 1 --2- 172.21.15.89:0/1371995001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a40d1fd0 0x7f22a406cba0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.108+0000 7f22a9600700 1 -- 172.21.15.89:0/1371995001 >> 172.21.15.89:0/1371995001 conn(0x7f22a408aff0 msgr2=0x7f22a415eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:29.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.108+0000 7f22a9600700 1 -- 172.21.15.89:0/1371995001 shutdown_connections 2024-04-02T07:48:29.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.108+0000 7f22a9600700 1 -- 172.21.15.89:0/1371995001 wait complete. 2024-04-02T07:48:29.128 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:26", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:29.129 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2024-04-02T00:00:00', 'created': '2024-04-02T07:37:26', 'first': '2024-04-02T07:38:00', 'last': '2024-04-02T07:48:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-02T07:48:29.129 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-02T07:48:29.129 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-02T07:48:29.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:29.129 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-04-02T07:48:29.175 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:29.175 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:29.175 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:29.175 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:29.175 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:29.175 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:29.175 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:29.175 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:29.175 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:29.175 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:29.175 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:29.176 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-02-07_38_00_UTC', 'scheduled-2024-04-02-07_39_00_UTC', 'scheduled-2024-04-02-07_40_00_UTC', 'scheduled-2024-04-02-07_41_00_UTC', 'scheduled-2024-04-02-07_42_00_UTC', 'scheduled-2024-04-02-07_43_00_UTC', 'scheduled-2024-04-02-07_44_00_UTC', 'scheduled-2024-04-02-07_45_00_UTC', 'scheduled-2024-04-02-07_46_00_UTC', 'scheduled-2024-04-02-07_47_00_UTC', 'scheduled-2024-04-02-07_48_00_UTC'] 2024-04-02T07:48:29.177 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2024-04-02T07:48:29.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.406+0000 7efca6a00700 1 Processor -- start 2024-04-02T07:48:29.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.406+0000 7efca6a00700 1 -- start start 2024-04-02T07:48:29.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.407+0000 7efca6a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca015d9d0 0x7efca00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:29.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.407+0000 7efca6a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca0161420 con 0x7efca00d16a0 2024-04-02T07:48:29.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.407+0000 7efca6a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca0161700 con 0x7efca00d2540 2024-04-02T07:48:29.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.407+0000 7efca6a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca015edc0 con 0x7efca015d9d0 2024-04-02T07:48:29.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.407+0000 7efc97e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efca00d16a0 0x7efca00d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51600/0 (socket says 172.21.15.89:51600) 2024-04-02T07:48:29.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.407+0000 7efc97e00700 1 -- 172.21.15.89:0/4064096392 learned_addr learned my addr 172.21.15.89:0/4064096392 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.407+0000 7efc9fe00700 1 --2- 172.21.15.89:0/4064096392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca015d9d0 0x7efca00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461506252 0 0) 0x7efca0161420 con 0x7efca00d16a0 2024-04-02T07:48:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc840035b0 con 0x7efca00d16a0 2024-04-02T07:48:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000476946 0 0) 0x7efca0161700 con 0x7efca00d2540 2024-04-02T07:48:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca0161420 con 0x7efca00d2540 2024-04-02T07:48:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4259027042 0 0) 0x7efc840035b0 con 0x7efca00d16a0 2024-04-02T07:48:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca015d9d0 msgr2=0x7efca00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 --2- 172.21.15.89:0/4064096392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca015d9d0 0x7efca00d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 >> v1:172.21.15.113:6789/0 conn(0x7efca00d2540 legacy=0x7efca00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca01619e0 con 0x7efca00d16a0 2024-04-02T07:48:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7efc880017f0 con 0x7efca00d16a0 2024-04-02T07:48:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9fe00700 1 --2- 172.21.15.89:0/4064096392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca015d9d0 0x7efca00d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:48:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 >> v1:172.21.15.89:6789/0 conn(0x7efca00d16a0 legacy=0x7efca00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 --2- 172.21.15.89:0/4064096392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efc84003bc0 0x7efc84003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 --2- 172.21.15.89:0/4064096392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc84004850 0x7efc84004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 --2- 172.21.15.89:0/4064096392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc84005210 0x7efc8400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8400c000 con 0x7efc84005210 2024-04-02T07:48:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8400c140 con 0x7efc84004850 2024-04-02T07:48:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.408+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8400c300 con 0x7efc84003bc0 2024-04-02T07:48:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.409+0000 7efca4a00700 1 --2- 172.21.15.89:0/4064096392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc84005210 0x7efc8400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.409+0000 7efc9fe00700 1 --2- 172.21.15.89:0/4064096392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efc84003bc0 0x7efc84003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.409+0000 7efc97e00700 1 --2- 172.21.15.89:0/4064096392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc84004850 0x7efc84004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.409+0000 7efca4a00700 1 -- 172.21.15.89:0/4064096392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efc84003bc0 msgr2=0x7efc84003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.409+0000 7efca4a00700 1 --2- 172.21.15.89:0/4064096392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efc84003bc0 0x7efc84003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.409+0000 7efca4a00700 1 -- 172.21.15.89:0/4064096392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc84004850 msgr2=0x7efc84004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.409+0000 7efca4a00700 1 --2- 172.21.15.89:0/4064096392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc84004850 0x7efc84004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.409+0000 7efca4a00700 1 -- 172.21.15.89:0/4064096392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc840035b0 con 0x7efc84005210 2024-04-02T07:48:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.409+0000 7efca4a00700 1 --2- 172.21.15.89:0/4064096392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc84005210 0x7efc8400ba10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efca008ca30 tx=0x7efc84005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a575713eecf1683 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.409+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98001f30 con 0x7efc84005210 2024-04-02T07:48:29.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.409+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9800ce30 con 0x7efc84005210 2024-04-02T07:48:29.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.409+0000 7efc9f400700 1 -- 172.21.15.89:0/4064096392 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98003c40 con 0x7efc84005210 2024-04-02T07:48:29.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.410+0000 7efca6a00700 1 -- 172.21.15.89:0/4064096392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc84005210 msgr2=0x7efc8400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:29.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.410+0000 7efca6a00700 1 --2- 172.21.15.89:0/4064096392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc84005210 0x7efc8400ba10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efca008ca30 tx=0x7efc84005880 comp rx=0 tx=0).stop 2024-04-02T07:48:29.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.410+0000 7efca4a00700 1 -- 172.21.15.89:0/4064096392 reap_dead start 2024-04-02T07:48:29.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.410+0000 7efca6a00700 1 -- 172.21.15.89:0/4064096392 shutdown_connections 2024-04-02T07:48:29.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.410+0000 7efca6a00700 1 -- 172.21.15.89:0/4064096392 >> 172.21.15.89:0/4064096392 conn(0x7efca008aff0 msgr2=0x7efca00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:29.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.410+0000 7efca6a00700 1 -- 172.21.15.89:0/4064096392 shutdown_connections 2024-04-02T07:48:29.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.410+0000 7efca6a00700 1 -- 172.21.15.89:0/4064096392 wait complete. 2024-04-02T07:48:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.412+0000 7efca6a00700 1 Processor -- start 2024-04-02T07:48:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.412+0000 7efca6a00700 1 -- start start 2024-04-02T07:48:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efca6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc84005210 0x7efca01557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efca6a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca00b4920 0x7efca00b4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efca6a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca00b6fd0 0x7efca00b7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efca6a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca00b52e0 con 0x7efc84005210 2024-04-02T07:48:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efca6a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca00cb380 con 0x7efca00b6fd0 2024-04-02T07:48:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efca6a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca0155cf0 con 0x7efca00b4920 2024-04-02T07:48:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efc9fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc84005210 0x7efca01557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efc9fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc84005210 0x7efca01557b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36186/0 (socket says 172.21.15.89:36186) 2024-04-02T07:48:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efc9fe00700 1 -- 172.21.15.89:0/3525240856 learned_addr learned my addr 172.21.15.89:0/3525240856 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efca4a00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca00b6fd0 0x7efca00b7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efc97e00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca00b4920 0x7efca00b4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efc9fe00700 1 -- 172.21.15.89:0/3525240856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca00b4920 msgr2=0x7efca00b4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efc9fe00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca00b4920 0x7efca00b4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efc9fe00700 1 -- 172.21.15.89:0/3525240856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca00b6fd0 msgr2=0x7efca00b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efc9fe00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca00b6fd0 0x7efca00b7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efc9fe00700 1 -- 172.21.15.89:0/3525240856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca00caa90 con 0x7efc84005210 2024-04-02T07:48:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.413+0000 7efca4a00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca00b6fd0 0x7efca00b7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:48:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.414+0000 7efc9fe00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc84005210 0x7efca01557b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efc90011c00 tx=0x7efc90009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.414+0000 7efc9cc00700 1 -- 172.21.15.89:0/3525240856 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9000fa60 con 0x7efc84005210 2024-04-02T07:48:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.414+0000 7efc9cc00700 1 -- 172.21.15.89:0/3525240856 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9000fbc0 con 0x7efc84005210 2024-04-02T07:48:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.414+0000 7efc9cc00700 1 -- 172.21.15.89:0/3525240856 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc90014440 con 0x7efc84005210 2024-04-02T07:48:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.414+0000 7efca6a00700 1 -- 172.21.15.89:0/3525240856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca00b7d50 con 0x7efc84005210 2024-04-02T07:48:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.414+0000 7efca6a00700 1 -- 172.21.15.89:0/3525240856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca00cad30 con 0x7efc84005210 2024-04-02T07:48:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.415+0000 7efca6a00700 1 -- 172.21.15.89:0/3525240856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca00d1ca0 con 0x7efc84005210 2024-04-02T07:48:29.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.416+0000 7efc9cc00700 1 -- 172.21.15.89:0/3525240856 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7efc900145a0 con 0x7efc84005210 2024-04-02T07:48:29.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.417+0000 7efc9cc00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efc7c06ae90 0x7efc7c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:29.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.417+0000 7efc9cc00700 1 -- 172.21.15.89:0/3525240856 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7efc9008a130 con 0x7efc84005210 2024-04-02T07:48:29.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.417+0000 7efc97e00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efc7c06ae90 0x7efc7c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:29.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.418+0000 7efc97e00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efc7c06ae90 0x7efc7c06d350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc880011d0 tx=0x7efc88009040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:29.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.421+0000 7efc9cc00700 1 -- 172.21.15.89:0/3525240856 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efc90054e70 con 0x7efc84005210 2024-04-02T07:48:29.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.649+0000 7efca6a00700 1 -- 172.21.15.89:0/3525240856 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efca0161fa0 con 0x7efc7c06ae90 2024-04-02T07:48:29.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.659+0000 7efc9cc00700 1 -- 172.21.15.89:0/3525240856 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7efc88001710 con 0x7efc7c06ae90 2024-04-02T07:48:29.661 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:29.661 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:27", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:29.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.661+0000 7efca6a00700 1 -- 172.21.15.89:0/3525240856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efc7c06ae90 msgr2=0x7efc7c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:29.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.661+0000 7efca6a00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efc7c06ae90 0x7efc7c06d350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc880011d0 tx=0x7efc88009040 comp rx=0 tx=0).stop 2024-04-02T07:48:29.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.661+0000 7efca6a00700 1 -- 172.21.15.89:0/3525240856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc84005210 msgr2=0x7efca01557b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:29.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.661+0000 7efca6a00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc84005210 0x7efca01557b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efc90011c00 tx=0x7efc90009ec0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.661+0000 7efca6a00700 1 -- 172.21.15.89:0/3525240856 shutdown_connections 2024-04-02T07:48:29.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.661+0000 7efca6a00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efc7c06ae90 0x7efc7c06d350 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.661+0000 7efca6a00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca00b6fd0 0x7efca00b7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.661+0000 7efca6a00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc84005210 0x7efca01557b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.661+0000 7efca6a00700 1 --2- 172.21.15.89:0/3525240856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca00b4920 0x7efca00b4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.661+0000 7efca6a00700 1 -- 172.21.15.89:0/3525240856 >> 172.21.15.89:0/3525240856 conn(0x7efca008aff0 msgr2=0x7efca015bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:29.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.662+0000 7efca6a00700 1 -- 172.21.15.89:0/3525240856 shutdown_connections 2024-04-02T07:48:29.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.662+0000 7efca6a00700 1 -- 172.21.15.89:0/3525240856 wait complete. 2024-04-02T07:48:29.681 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:27", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:29.682 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2024-04-02T00:00:00', 'created': '2024-04-02T07:37:27', 'first': '2024-04-02T07:38:00', 'last': '2024-04-02T07:48:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-02T07:48:29.682 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-02T07:48:29.682 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-02T07:48:29.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:29.682 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-04-02T07:48:29.730 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:29.730 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:29.730 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:29.731 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:29.731 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:29.731 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:29.731 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:29.731 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:29.731 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:29.731 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:29.731 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:29.732 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-02-07_38_00_UTC', 'scheduled-2024-04-02-07_39_00_UTC', 'scheduled-2024-04-02-07_40_00_UTC', 'scheduled-2024-04-02-07_41_00_UTC', 'scheduled-2024-04-02-07_42_00_UTC', 'scheduled-2024-04-02-07_43_00_UTC', 'scheduled-2024-04-02-07_44_00_UTC', 'scheduled-2024-04-02-07_45_00_UTC', 'scheduled-2024-04-02-07_46_00_UTC', 'scheduled-2024-04-02-07_47_00_UTC', 'scheduled-2024-04-02-07_48_00_UTC'] 2024-04-02T07:48:29.732 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2024-04-02T07:48:29.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.962+0000 7f2be8a00700 1 Processor -- start 2024-04-02T07:48:29.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.963+0000 7f2be8a00700 1 -- start start 2024-04-02T07:48:29.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.964+0000 7f2be8a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2be40cbb30 0x7f2be40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:29.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.964+0000 7f2be8a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2be40cd480 0x7f2be406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:29.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.964+0000 7f2be8a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be40cc450 con 0x7f2be40cd020 2024-04-02T07:48:29.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.964+0000 7f2be8a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be40cd8f0 con 0x7f2be40cbb30 2024-04-02T07:48:29.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.964+0000 7f2be8a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be40cdae0 con 0x7f2be40cd480 2024-04-02T07:48:29.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.964+0000 7f2be1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2be40cbb30 0x7f2be40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:29.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.964+0000 7f2bdbe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2be40cd020 0x7f2be40b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51610/0 (socket says 172.21.15.89:51610) 2024-04-02T07:48:29.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.964+0000 7f2bdbe00700 1 -- 172.21.15.89:0/3996945230 learned_addr learned my addr 172.21.15.89:0/3996945230 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:29.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.964+0000 7f2be2800700 1 --2- 172.21.15.89:0/3996945230 >> v2:172.21.15.113:3300/0 conn(0x7f2be40cd480 0x7f2be406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:29.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.964+0000 7f2be1e00700 1 -- 172.21.15.89:0/3996945230 >> v2:172.21.15.113:3300/0 conn(0x7f2be40cd480 msgr2=0x7f2be406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:29.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.965+0000 7f2be1e00700 1 --2- 172.21.15.89:0/3996945230 >> v2:172.21.15.113:3300/0 conn(0x7f2be40cd480 0x7f2be406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.965+0000 7f2be1e00700 1 -- 172.21.15.89:0/3996945230 >> v1:172.21.15.89:6789/0 conn(0x7f2be40cd020 legacy=0x7f2be40b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:29.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.964+0000 7f2be1400700 1 -- 172.21.15.89:0/3996945230 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168029835 0 0) 0x7f2be40cc450 con 0x7f2be40cd020 2024-04-02T07:48:29.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.965+0000 7f2be1e00700 1 -- 172.21.15.89:0/3996945230 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be40cddc0 con 0x7f2be40cbb30 2024-04-02T07:48:29.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.965+0000 7f2be1e00700 1 --2- 172.21.15.89:0/3996945230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40cbb30 0x7f2be40cbf10 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2be406b6d0 tx=0x7f2bd4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3049f0c7cf22f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:29.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.965+0000 7f2be1400700 1 -- 172.21.15.89:0/3996945230 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4011830 con 0x7f2be40cbb30 2024-04-02T07:48:29.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.965+0000 7f2be1400700 1 -- 172.21.15.89:0/3996945230 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd4011990 con 0x7f2be40cbb30 2024-04-02T07:48:29.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.965+0000 7f2be1400700 1 -- 172.21.15.89:0/3996945230 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4011d20 con 0x7f2be40cbb30 2024-04-02T07:48:29.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.966+0000 7f2be8a00700 1 -- 172.21.15.89:0/3996945230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40cbb30 msgr2=0x7f2be40cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:29.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.966+0000 7f2be8a00700 1 --2- 172.21.15.89:0/3996945230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40cbb30 0x7f2be40cbf10 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2be406b6d0 tx=0x7f2bd4009b20 comp rx=0 tx=0).stop 2024-04-02T07:48:29.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.966+0000 7f2be8a00700 1 -- 172.21.15.89:0/3996945230 shutdown_connections 2024-04-02T07:48:29.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.966+0000 7f2be8a00700 1 --2- 172.21.15.89:0/3996945230 >> v2:172.21.15.113:3300/0 conn(0x7f2be40cd480 0x7f2be406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.966+0000 7f2be8a00700 1 --2- 172.21.15.89:0/3996945230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40cbb30 0x7f2be40cbf10 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.966+0000 7f2be8a00700 1 -- 172.21.15.89:0/3996945230 >> 172.21.15.89:0/3996945230 conn(0x7f2be40b0d70 msgr2=0x7f2be40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:29.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.966+0000 7f2be8a00700 1 -- 172.21.15.89:0/3996945230 shutdown_connections 2024-04-02T07:48:29.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.966+0000 7f2be8a00700 1 -- 172.21.15.89:0/3996945230 wait complete. 2024-04-02T07:48:29.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.967+0000 7f2be8a00700 1 Processor -- start 2024-04-02T07:48:29.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.968+0000 7f2be8a00700 1 -- start start 2024-04-02T07:48:29.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.968+0000 7f2be8a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2be40cbb30 0x7f2be41f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:29.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.968+0000 7f2be8a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40cd020 0x7f2be40ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:29.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.968+0000 7f2be8a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2be40cd480 0x7f2be409c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:29.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.968+0000 7f2be8a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40ac590 con 0x7f2be40cd020 2024-04-02T07:48:29.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.968+0000 7f2be8a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40ac830 con 0x7f2be40cbb30 2024-04-02T07:48:29.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.969+0000 7f2be8a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40acb10 con 0x7f2be40cd480 2024-04-02T07:48:29.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.969+0000 7f2bdbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40cd020 0x7f2be40ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:29.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.969+0000 7f2bdbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40cd020 0x7f2be40ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36210/0 (socket says 172.21.15.89:36210) 2024-04-02T07:48:29.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.969+0000 7f2bdbe00700 1 -- 172.21.15.89:0/2144031885 learned_addr learned my addr 172.21.15.89:0/2144031885 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:29.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.969+0000 7f2be1e00700 1 --2- 172.21.15.89:0/2144031885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2be40cbb30 0x7f2be41f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:29.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.969+0000 7f2bdbe00700 1 -- 172.21.15.89:0/2144031885 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2be40cd480 msgr2=0x7f2be409c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:29.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.969+0000 7f2bdbe00700 1 --2- 172.21.15.89:0/2144031885 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2be40cd480 0x7f2be409c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.969+0000 7f2bdbe00700 1 -- 172.21.15.89:0/2144031885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2be40cbb30 msgr2=0x7f2be41f55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:29.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.969+0000 7f2bdbe00700 1 --2- 172.21.15.89:0/2144031885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2be40cbb30 0x7f2be41f55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:29.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.969+0000 7f2bdbe00700 1 -- 172.21.15.89:0/2144031885 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd40094a0 con 0x7f2be40cd020 2024-04-02T07:48:29.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.969+0000 7f2be1e00700 1 --2- 172.21.15.89:0/2144031885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2be40cbb30 0x7f2be41f55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:29.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.969+0000 7f2bdbe00700 1 --2- 172.21.15.89:0/2144031885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40cd020 0x7f2be40ac050 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc00c4d0 tx=0x7f2bcc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:29.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.970+0000 7f2bda000700 1 -- 172.21.15.89:0/2144031885 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc004c80 con 0x7f2be40cd020 2024-04-02T07:48:29.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.970+0000 7f2bda000700 1 -- 172.21.15.89:0/2144031885 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bcc004de0 con 0x7f2be40cd020 2024-04-02T07:48:29.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.970+0000 7f2be8a00700 1 -- 172.21.15.89:0/2144031885 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be40acdf0 con 0x7f2be40cd020 2024-04-02T07:48:29.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.970+0000 7f2bda000700 1 -- 172.21.15.89:0/2144031885 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc007310 con 0x7f2be40cd020 2024-04-02T07:48:29.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.970+0000 7f2be8a00700 1 -- 172.21.15.89:0/2144031885 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be40ad3b0 con 0x7f2be40cd020 2024-04-02T07:48:29.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.971+0000 7f2be8a00700 1 -- 172.21.15.89:0/2144031885 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be40ad5a0 con 0x7f2be40cd020 2024-04-02T07:48:29.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.972+0000 7f2bda000700 1 -- 172.21.15.89:0/2144031885 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f2bcc007470 con 0x7f2be40cd020 2024-04-02T07:48:29.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.972+0000 7f2bda000700 1 --2- 172.21.15.89:0/2144031885 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2bc006ae90 0x7f2bc006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:29.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.973+0000 7f2bda000700 1 -- 172.21.15.89:0/2144031885 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f2bcc0894b0 con 0x7f2be40cd020 2024-04-02T07:48:29.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.973+0000 7f2be1e00700 1 --2- 172.21.15.89:0/2144031885 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2bc006ae90 0x7f2bc006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:29.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.973+0000 7f2be1e00700 1 --2- 172.21.15.89:0/2144031885 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2bc006ae90 0x7f2bc006d350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2bd40090b0 tx=0x7f2bd4008030 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:29.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:29.977+0000 7f2bda000700 1 -- 172.21.15.89:0/2144031885 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2bcc054290 con 0x7f2be40cd020 2024-04-02T07:48:30.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.202+0000 7f2be8a00700 1 -- 172.21.15.89:0/2144031885 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2be406df00 con 0x7f2bc006ae90 2024-04-02T07:48:30.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.212+0000 7f2bda000700 1 -- 172.21.15.89:0/2144031885 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f2bd4011b80 con 0x7f2bc006ae90 2024-04-02T07:48:30.214 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:30.214 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:27", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:30.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.214+0000 7f2be8a00700 1 -- 172.21.15.89:0/2144031885 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2bc006ae90 msgr2=0x7f2bc006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:30.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.214+0000 7f2be8a00700 1 --2- 172.21.15.89:0/2144031885 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2bc006ae90 0x7f2bc006d350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2bd40090b0 tx=0x7f2bd4008030 comp rx=0 tx=0).stop 2024-04-02T07:48:30.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.214+0000 7f2be8a00700 1 -- 172.21.15.89:0/2144031885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40cd020 msgr2=0x7f2be40ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:30.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.214+0000 7f2be8a00700 1 --2- 172.21.15.89:0/2144031885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40cd020 0x7f2be40ac050 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc00c4d0 tx=0x7f2bcc00c890 comp rx=0 tx=0).stop 2024-04-02T07:48:30.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.214+0000 7f2be8a00700 1 -- 172.21.15.89:0/2144031885 shutdown_connections 2024-04-02T07:48:30.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.214+0000 7f2be8a00700 1 --2- 172.21.15.89:0/2144031885 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2bc006ae90 0x7f2bc006d350 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:30.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.214+0000 7f2be8a00700 1 --2- 172.21.15.89:0/2144031885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2be40cbb30 0x7f2be41f55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:30.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.214+0000 7f2be8a00700 1 --2- 172.21.15.89:0/2144031885 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2be40cd480 0x7f2be409c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:30.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.214+0000 7f2be8a00700 1 --2- 172.21.15.89:0/2144031885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40cd020 0x7f2be40ac050 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:30.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.214+0000 7f2be8a00700 1 -- 172.21.15.89:0/2144031885 >> 172.21.15.89:0/2144031885 conn(0x7f2be40b0d70 msgr2=0x7f2be415bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:30.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.214+0000 7f2be8a00700 1 -- 172.21.15.89:0/2144031885 shutdown_connections 2024-04-02T07:48:30.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.214+0000 7f2be8a00700 1 -- 172.21.15.89:0/2144031885 wait complete. 2024-04-02T07:48:30.234 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:27", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:30.235 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2024-04-02T00:00:00', 'created': '2024-04-02T07:37:27', 'first': '2024-04-02T07:38:00', 'last': '2024-04-02T07:48:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-02T07:48:30.235 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-02T07:48:30.235 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-02T07:48:30.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:30.235 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-04-02T07:48:30.280 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:30.280 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:30.280 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:30.280 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:30.280 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:30.281 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:30.281 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:30.281 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:30.281 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:30.281 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:30.281 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:30.282 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-02-07_38_00_UTC', 'scheduled-2024-04-02-07_39_00_UTC', 'scheduled-2024-04-02-07_40_00_UTC', 'scheduled-2024-04-02-07_41_00_UTC', 'scheduled-2024-04-02-07_42_00_UTC', 'scheduled-2024-04-02-07_43_00_UTC', 'scheduled-2024-04-02-07_44_00_UTC', 'scheduled-2024-04-02-07_45_00_UTC', 'scheduled-2024-04-02-07_46_00_UTC', 'scheduled-2024-04-02-07_47_00_UTC', 'scheduled-2024-04-02-07_48_00_UTC'] 2024-04-02T07:48:30.282 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2024-04-02T07:48:30.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.512+0000 7f2518e00700 1 Processor -- start 2024-04-02T07:48:30.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.513+0000 7f2518e00700 1 -- start start 2024-04-02T07:48:30.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.514+0000 7f2518e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f251415d800 0x7f25140725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:30.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.514+0000 7f2518e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25140709c0 con 0x7f25140ce6a0 2024-04-02T07:48:30.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.514+0000 7f2518e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2514070c80 con 0x7f25140ce230 2024-04-02T07:48:30.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.514+0000 7f2518e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25140b40f0 con 0x7f251415d800 2024-04-02T07:48:30.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.514+0000 7f2512800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f25140ce6a0 0x7f251415dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51622/0 (socket says 172.21.15.89:51622) 2024-04-02T07:48:30.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.514+0000 7f2512800700 1 -- 172.21.15.89:0/3752793455 learned_addr learned my addr 172.21.15.89:0/3752793455 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:30.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.514+0000 7f2511e00700 1 --2- 172.21.15.89:0/3752793455 >> v2:172.21.15.113:3300/0 conn(0x7f251415d800 0x7f25140725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:30.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.514+0000 7f2510a00700 1 -- 172.21.15.89:0/3752793455 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035390621 0 0) 0x7f25140709c0 con 0x7f25140ce6a0 2024-04-02T07:48:30.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.514+0000 7f2510a00700 1 -- 172.21.15.89:0/3752793455 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f80035b0 con 0x7f25140ce6a0 2024-04-02T07:48:30.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.515+0000 7f2511e00700 1 -- 172.21.15.89:0/3752793455 >> v1:172.21.15.113:6789/0 conn(0x7f25140ce230 legacy=0x7f25140b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:30.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.515+0000 7f2511e00700 1 -- 172.21.15.89:0/3752793455 >> v1:172.21.15.89:6789/0 conn(0x7f25140ce6a0 legacy=0x7f251415dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:30.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.515+0000 7f2511e00700 1 -- 172.21.15.89:0/3752793455 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2514070f60 con 0x7f251415d800 2024-04-02T07:48:30.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.515+0000 7f2511e00700 1 --2- 172.21.15.89:0/3752793455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f251415d800 0x7f25140725c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2504009840 tx=0x7f2504009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54d5d35b70f84891 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:30.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.515+0000 7f2510a00700 1 -- 172.21.15.89:0/3752793455 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504011830 con 0x7f251415d800 2024-04-02T07:48:30.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.515+0000 7f2510a00700 1 -- 172.21.15.89:0/3752793455 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2504011990 con 0x7f251415d800 2024-04-02T07:48:30.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.515+0000 7f2510a00700 1 -- 172.21.15.89:0/3752793455 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504011d20 con 0x7f251415d800 2024-04-02T07:48:30.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.516+0000 7f2518e00700 1 -- 172.21.15.89:0/3752793455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f251415d800 msgr2=0x7f25140725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:30.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.516+0000 7f2518e00700 1 --2- 172.21.15.89:0/3752793455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f251415d800 0x7f25140725c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2504009840 tx=0x7f2504009b50 comp rx=0 tx=0).stop 2024-04-02T07:48:30.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.516+0000 7f2518e00700 1 -- 172.21.15.89:0/3752793455 shutdown_connections 2024-04-02T07:48:30.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.516+0000 7f2518e00700 1 --2- 172.21.15.89:0/3752793455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f251415d800 0x7f25140725c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:30.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.516+0000 7f2518e00700 1 -- 172.21.15.89:0/3752793455 >> 172.21.15.89:0/3752793455 conn(0x7f25140b0d70 msgr2=0x7f25140b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:30.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.516+0000 7f2518e00700 1 -- 172.21.15.89:0/3752793455 shutdown_connections 2024-04-02T07:48:30.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.516+0000 7f2518e00700 1 -- 172.21.15.89:0/3752793455 wait complete. 2024-04-02T07:48:30.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.518+0000 7f2518e00700 1 Processor -- start 2024-04-02T07:48:30.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.518+0000 7f2518e00700 1 -- start start 2024-04-02T07:48:30.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.519+0000 7f2518e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25140ce230 0x7f25141579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:30.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2518e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25140ce6a0 0x7f251415a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:30.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2518e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f251415d800 0x7f25140cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:30.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2518e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2514140fb0 con 0x7f25140ce230 2024-04-02T07:48:30.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2518e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25141584b0 con 0x7f25140ce6a0 2024-04-02T07:48:30.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2518e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2514140960 con 0x7f251415d800 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2511e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25140ce230 0x7f25141579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2511e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25140ce230 0x7f25141579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36224/0 (socket says 172.21.15.89:36224) 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2511e00700 1 -- 172.21.15.89:0/3595458085 learned_addr learned my addr 172.21.15.89:0/3595458085 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2511400700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25140ce6a0 0x7f251415a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2511e00700 1 -- 172.21.15.89:0/3595458085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f251415d800 msgr2=0x7f25140cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2512800700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f251415d800 0x7f25140cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2511e00700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f251415d800 0x7f25140cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2511e00700 1 -- 172.21.15.89:0/3595458085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25140ce6a0 msgr2=0x7f251415a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2511e00700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25140ce6a0 0x7f251415a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2511e00700 1 -- 172.21.15.89:0/3595458085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25040094a0 con 0x7f25140ce230 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2511400700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25140ce6a0 0x7f251415a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2512800700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f251415d800 0x7f25140cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.520+0000 7f2511e00700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25140ce230 0x7f25141579d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2504009b90 tx=0x7f2504004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.521+0000 7f250a000700 1 -- 172.21.15.89:0/3595458085 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25040038a0 con 0x7f25140ce230 2024-04-02T07:48:30.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.521+0000 7f250a000700 1 -- 172.21.15.89:0/3595458085 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250401e070 con 0x7f25140ce230 2024-04-02T07:48:30.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.521+0000 7f250a000700 1 -- 172.21.15.89:0/3595458085 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504003c40 con 0x7f25140ce230 2024-04-02T07:48:30.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.521+0000 7f2518e00700 1 -- 172.21.15.89:0/3595458085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2514140c40 con 0x7f25140ce230 2024-04-02T07:48:30.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.521+0000 7f2518e00700 1 -- 172.21.15.89:0/3595458085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2514141440 con 0x7f25140ce230 2024-04-02T07:48:30.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.522+0000 7f2518e00700 1 -- 172.21.15.89:0/3595458085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2514140e00 con 0x7f25140ce230 2024-04-02T07:48:30.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.523+0000 7f250a000700 1 -- 172.21.15.89:0/3595458085 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f250402d450 con 0x7f25140ce230 2024-04-02T07:48:30.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.523+0000 7f250a000700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f24f406ae90 0x7f24f406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:30.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.524+0000 7f250a000700 1 -- 172.21.15.89:0/3595458085 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f250408a8d0 con 0x7f25140ce230 2024-04-02T07:48:30.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.524+0000 7f2511400700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f24f406ae90 0x7f24f406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:30.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.524+0000 7f2511400700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f24f406ae90 0x7f24f406d350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2514144d00 tx=0x7f2500009040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:30.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.528+0000 7f250a000700 1 -- 172.21.15.89:0/3595458085 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2504055610 con 0x7f25140ce230 2024-04-02T07:48:30.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.752+0000 7f2518e00700 1 -- 172.21.15.89:0/3595458085 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f25141407e0 con 0x7f24f406ae90 2024-04-02T07:48:30.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.762+0000 7f250a000700 1 -- 172.21.15.89:0/3595458085 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f2500001470 con 0x7f24f406ae90 2024-04-02T07:48:30.764 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:30.765 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:28", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:30.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.764+0000 7f2518e00700 1 -- 172.21.15.89:0/3595458085 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f24f406ae90 msgr2=0x7f24f406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:30.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.764+0000 7f2518e00700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f24f406ae90 0x7f24f406d350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2514144d00 tx=0x7f2500009040 comp rx=0 tx=0).stop 2024-04-02T07:48:30.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.764+0000 7f2518e00700 1 -- 172.21.15.89:0/3595458085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25140ce230 msgr2=0x7f25141579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:30.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.764+0000 7f2518e00700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25140ce230 0x7f25141579d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2504009b90 tx=0x7f2504004750 comp rx=0 tx=0).stop 2024-04-02T07:48:30.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.764+0000 7f2518e00700 1 -- 172.21.15.89:0/3595458085 shutdown_connections 2024-04-02T07:48:30.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.764+0000 7f2518e00700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f24f406ae90 0x7f24f406d350 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:30.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.764+0000 7f2518e00700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f251415d800 0x7f25140cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:30.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.764+0000 7f2518e00700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25140ce230 0x7f25141579d0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:30.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.764+0000 7f2518e00700 1 --2- 172.21.15.89:0/3595458085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25140ce6a0 0x7f251415a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:30.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.764+0000 7f2518e00700 1 -- 172.21.15.89:0/3595458085 >> 172.21.15.89:0/3595458085 conn(0x7f25140b0d70 msgr2=0x7f25140ac050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:30.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.765+0000 7f2518e00700 1 -- 172.21.15.89:0/3595458085 shutdown_connections 2024-04-02T07:48:30.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:30.765+0000 7f2518e00700 1 -- 172.21.15.89:0/3595458085 wait complete. 2024-04-02T07:48:30.785 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:28", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:30.786 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2024-04-02T00:00:00', 'created': '2024-04-02T07:37:28', 'first': '2024-04-02T07:38:00', 'last': '2024-04-02T07:48:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-02T07:48:30.786 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-02T07:48:30.786 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-02T07:48:30.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:30.786 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-04-02T07:48:30.832 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:30.832 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:30.832 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:30.832 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:30.832 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:30.832 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:30.832 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:30.832 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:30.832 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:30.832 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:30.833 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:30.833 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-02-07_38_00_UTC', 'scheduled-2024-04-02-07_39_00_UTC', 'scheduled-2024-04-02-07_40_00_UTC', 'scheduled-2024-04-02-07_41_00_UTC', 'scheduled-2024-04-02-07_42_00_UTC', 'scheduled-2024-04-02-07_43_00_UTC', 'scheduled-2024-04-02-07_44_00_UTC', 'scheduled-2024-04-02-07_45_00_UTC', 'scheduled-2024-04-02-07_46_00_UTC', 'scheduled-2024-04-02-07_47_00_UTC', 'scheduled-2024-04-02-07_48_00_UTC'] 2024-04-02T07:48:30.834 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2024-04-02T07:48:31.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.064+0000 7fc42c000700 1 Processor -- start 2024-04-02T07:48:31.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.065+0000 7fc42c000700 1 -- start start 2024-04-02T07:48:31.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.066+0000 7fc42c000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc42415d9d0 0x7fc4240d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:31.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.066+0000 7fc42c000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4240d16a0 0x7fc4240d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:31.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.066+0000 7fc42c000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc424161420 con 0x7fc4240d2370 2024-04-02T07:48:31.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.066+0000 7fc42c000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4240d27e0 con 0x7fc4240d16a0 2024-04-02T07:48:31.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.066+0000 7fc42c000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc424161700 con 0x7fc42415d9d0 2024-04-02T07:48:31.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.066+0000 7fc42a000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc4240d2370 0x7fc42415ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51628/0 (socket says 172.21.15.89:51628) 2024-04-02T07:48:31.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.066+0000 7fc428c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4240d16a0 0x7fc4240d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:31.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.066+0000 7fc42a000700 1 -- 172.21.15.89:0/1226215489 learned_addr learned my addr 172.21.15.89:0/1226215489 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:31.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.066+0000 7fc429600700 1 --2- 172.21.15.89:0/1226215489 >> v2:172.21.15.113:3300/0 conn(0x7fc42415d9d0 0x7fc4240d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:31.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.066+0000 7fc423e00700 1 -- 172.21.15.89:0/1226215489 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128904488 0 0) 0x7fc424161420 con 0x7fc4240d2370 2024-04-02T07:48:31.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.066+0000 7fc428c00700 1 -- 172.21.15.89:0/1226215489 >> v2:172.21.15.113:3300/0 conn(0x7fc42415d9d0 msgr2=0x7fc4240d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:31.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.067+0000 7fc428c00700 1 --2- 172.21.15.89:0/1226215489 >> v2:172.21.15.113:3300/0 conn(0x7fc42415d9d0 0x7fc4240d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.067+0000 7fc428c00700 1 -- 172.21.15.89:0/1226215489 >> v1:172.21.15.89:6789/0 conn(0x7fc4240d2370 legacy=0x7fc42415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:31.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.067+0000 7fc428c00700 1 -- 172.21.15.89:0/1226215489 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4241619e0 con 0x7fc4240d16a0 2024-04-02T07:48:31.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.067+0000 7fc428c00700 1 --2- 172.21.15.89:0/1226215489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4240d16a0 0x7fc4240d1e30 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc4240d0ff0 tx=0x7fc410009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f7baedfc43f8b79 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:31.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.067+0000 7fc423e00700 1 -- 172.21.15.89:0/1226215489 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc410005e30 con 0x7fc4240d16a0 2024-04-02T07:48:31.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.067+0000 7fc423e00700 1 -- 172.21.15.89:0/1226215489 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc410006030 con 0x7fc4240d16a0 2024-04-02T07:48:31.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.067+0000 7fc423e00700 1 -- 172.21.15.89:0/1226215489 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4100063c0 con 0x7fc4240d16a0 2024-04-02T07:48:31.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.067+0000 7fc42c000700 1 -- 172.21.15.89:0/1226215489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4240d16a0 msgr2=0x7fc4240d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:31.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.067+0000 7fc42c000700 1 --2- 172.21.15.89:0/1226215489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4240d16a0 0x7fc4240d1e30 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc4240d0ff0 tx=0x7fc410009b00 comp rx=0 tx=0).stop 2024-04-02T07:48:31.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.068+0000 7fc42c000700 1 -- 172.21.15.89:0/1226215489 shutdown_connections 2024-04-02T07:48:31.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.068+0000 7fc42c000700 1 --2- 172.21.15.89:0/1226215489 >> v2:172.21.15.113:3300/0 conn(0x7fc42415d9d0 0x7fc4240d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.068+0000 7fc42c000700 1 --2- 172.21.15.89:0/1226215489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4240d16a0 0x7fc4240d1e30 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.068+0000 7fc42c000700 1 -- 172.21.15.89:0/1226215489 >> 172.21.15.89:0/1226215489 conn(0x7fc42408aff0 msgr2=0x7fc4240d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:31.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.068+0000 7fc42c000700 1 -- 172.21.15.89:0/1226215489 shutdown_connections 2024-04-02T07:48:31.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.068+0000 7fc42c000700 1 -- 172.21.15.89:0/1226215489 wait complete. 2024-04-02T07:48:31.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.069+0000 7fc42c000700 1 Processor -- start 2024-04-02T07:48:31.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.070+0000 7fc42c000700 1 -- start start 2024-04-02T07:48:31.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.071+0000 7fc42c000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4240d16a0 0x7fc4241f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:31.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.071+0000 7fc42c000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4240d2370 0x7fc4240b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:31.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.071+0000 7fc42c000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc42415d9d0 0x7fc4240cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:31.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.071+0000 7fc42c000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240b6a50 con 0x7fc4240d16a0 2024-04-02T07:48:31.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.071+0000 7fc42c000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240b6d30 con 0x7fc4240d2370 2024-04-02T07:48:31.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.071+0000 7fc42c000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240b7010 con 0x7fc42415d9d0 2024-04-02T07:48:31.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.071+0000 7fc429600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4240d16a0 0x7fc4241f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:31.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.071+0000 7fc429600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4240d16a0 0x7fc4241f6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36250/0 (socket says 172.21.15.89:36250) 2024-04-02T07:48:31.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.071+0000 7fc429600700 1 -- 172.21.15.89:0/1816636572 learned_addr learned my addr 172.21.15.89:0/1816636572 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:31.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.071+0000 7fc429600700 1 -- 172.21.15.89:0/1816636572 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc42415d9d0 msgr2=0x7fc4240cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:31.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.071+0000 7fc429600700 1 --2- 172.21.15.89:0/1816636572 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc42415d9d0 0x7fc4240cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.071+0000 7fc429600700 1 -- 172.21.15.89:0/1816636572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4240d2370 msgr2=0x7fc4240b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:31.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.071+0000 7fc429600700 1 --2- 172.21.15.89:0/1816636572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4240d2370 0x7fc4240b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.071+0000 7fc429600700 1 -- 172.21.15.89:0/1816636572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc410009430 con 0x7fc4240d16a0 2024-04-02T07:48:31.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.072+0000 7fc429600700 1 --2- 172.21.15.89:0/1816636572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4240d16a0 0x7fc4241f6d80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc41800d800 tx=0x7fc41800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:31.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.072+0000 7fc421600700 1 -- 172.21.15.89:0/1816636572 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4180125f0 con 0x7fc4240d16a0 2024-04-02T07:48:31.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.072+0000 7fc421600700 1 -- 172.21.15.89:0/1816636572 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc418012750 con 0x7fc4240d16a0 2024-04-02T07:48:31.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.072+0000 7fc42c000700 1 -- 172.21.15.89:0/1816636572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4240b72f0 con 0x7fc4240d16a0 2024-04-02T07:48:31.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.072+0000 7fc421600700 1 -- 172.21.15.89:0/1816636572 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418012b00 con 0x7fc4240d16a0 2024-04-02T07:48:31.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.072+0000 7fc42c000700 1 -- 172.21.15.89:0/1816636572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4240b8a40 con 0x7fc4240d16a0 2024-04-02T07:48:31.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.073+0000 7fc42c000700 1 -- 172.21.15.89:0/1816636572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4240b8bd0 con 0x7fc4240d16a0 2024-04-02T07:48:31.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.074+0000 7fc421600700 1 -- 172.21.15.89:0/1816636572 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fc418012c60 con 0x7fc4240d16a0 2024-04-02T07:48:31.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.075+0000 7fc421600700 1 --2- 172.21.15.89:0/1816636572 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc40406ae90 0x7fc40406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:31.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.075+0000 7fc421600700 1 -- 172.21.15.89:0/1816636572 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7fc418089c70 con 0x7fc4240d16a0 2024-04-02T07:48:31.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.075+0000 7fc428c00700 1 --2- 172.21.15.89:0/1816636572 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc40406ae90 0x7fc40406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:31.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.075+0000 7fc428c00700 1 --2- 172.21.15.89:0/1816636572 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc40406ae90 0x7fc40406d350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc4240d1ae0 tx=0x7fc410010000 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:31.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.079+0000 7fc421600700 1 -- 172.21.15.89:0/1816636572 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc4180549b0 con 0x7fc4240d16a0 2024-04-02T07:48:31.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.305+0000 7fc42c000700 1 -- 172.21.15.89:0/1816636572 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc424161fa0 con 0x7fc40406ae90 2024-04-02T07:48:31.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.316+0000 7fc421600700 1 -- 172.21.15.89:0/1816636572 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fc410020070 con 0x7fc40406ae90 2024-04-02T07:48:31.318 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:31.318 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:28", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:31.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.318+0000 7fc42c000700 1 -- 172.21.15.89:0/1816636572 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc40406ae90 msgr2=0x7fc40406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:31.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.318+0000 7fc42c000700 1 --2- 172.21.15.89:0/1816636572 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc40406ae90 0x7fc40406d350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc4240d1ae0 tx=0x7fc410010000 comp rx=0 tx=0).stop 2024-04-02T07:48:31.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.318+0000 7fc42c000700 1 -- 172.21.15.89:0/1816636572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4240d16a0 msgr2=0x7fc4241f6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:31.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.318+0000 7fc42c000700 1 --2- 172.21.15.89:0/1816636572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4240d16a0 0x7fc4241f6d80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc41800d800 tx=0x7fc41800dbc0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.318+0000 7fc42c000700 1 -- 172.21.15.89:0/1816636572 shutdown_connections 2024-04-02T07:48:31.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.318+0000 7fc42c000700 1 --2- 172.21.15.89:0/1816636572 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc40406ae90 0x7fc40406d350 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.318+0000 7fc42c000700 1 --2- 172.21.15.89:0/1816636572 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc42415d9d0 0x7fc4240cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.318+0000 7fc42c000700 1 --2- 172.21.15.89:0/1816636572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4240d16a0 0x7fc4241f6d80 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.318+0000 7fc42c000700 1 --2- 172.21.15.89:0/1816636572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4240d2370 0x7fc4240b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.318+0000 7fc42c000700 1 -- 172.21.15.89:0/1816636572 >> 172.21.15.89:0/1816636572 conn(0x7fc42408aff0 msgr2=0x7fc4240cebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:31.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.318+0000 7fc42c000700 1 -- 172.21.15.89:0/1816636572 shutdown_connections 2024-04-02T07:48:31.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.318+0000 7fc42c000700 1 -- 172.21.15.89:0/1816636572 wait complete. 2024-04-02T07:48:31.338 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:37:28", "first": "2024-04-02T07:38:00", "last": "2024-04-02T07:48:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-02T07:48:31.338 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2024-04-02T00:00:00', 'created': '2024-04-02T07:37:28', 'first': '2024-04-02T07:38:00', 'last': '2024-04-02T07:48:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-02T07:48:31.339 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-02T07:48:31.339 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-02T07:48:31.339 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2024-04-02T07:48:31.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.530+0000 7f0f5d200700 1 Processor -- start 2024-04-02T07:48:31.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.530+0000 7f0f5d200700 1 -- start start 2024-04-02T07:48:31.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.531+0000 7f0f5d200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f581611b0 0x7f0f5815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:31.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.531+0000 7f0f5d200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0f580cee10 0x7f0f58161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:31.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.531+0000 7f0f5d200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f580ce270 con 0x7f0f5815f4d0 2024-04-02T07:48:31.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.531+0000 7f0f5d200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5815b600 con 0x7f0f581611b0 2024-04-02T07:48:31.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.531+0000 7f0f5d200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f580d2760 con 0x7f0f580cee10 2024-04-02T07:48:31.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.532+0000 7f0f55400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0f5815f4d0 0x7f0f580ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51642/0 (socket says 172.21.15.89:51642) 2024-04-02T07:48:31.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.532+0000 7f0f55400700 1 -- 172.21.15.89:0/3101393757 learned_addr learned my addr 172.21.15.89:0/3101393757 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:31.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.532+0000 7f0f55e00700 1 --2- 172.21.15.89:0/3101393757 >> v2:172.21.15.89:3300/0 conn(0x7f0f581611b0 0x7f0f5815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:31.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.532+0000 7f0f56800700 1 --2- 172.21.15.89:0/3101393757 >> v2:172.21.15.113:3300/0 conn(0x7f0f580cee10 0x7f0f58161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:31.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.532+0000 7f0f55e00700 1 -- 172.21.15.89:0/3101393757 >> v2:172.21.15.113:3300/0 conn(0x7f0f580cee10 msgr2=0x7f0f58161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:31.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.532+0000 7f0f54a00700 1 -- 172.21.15.89:0/3101393757 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517933905 0 0) 0x7f0f580ce270 con 0x7f0f5815f4d0 2024-04-02T07:48:31.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.532+0000 7f0f55e00700 1 --2- 172.21.15.89:0/3101393757 >> v2:172.21.15.113:3300/0 conn(0x7f0f580cee10 0x7f0f58161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.532+0000 7f0f55e00700 1 -- 172.21.15.89:0/3101393757 >> v1:172.21.15.89:6789/0 conn(0x7f0f5815f4d0 legacy=0x7f0f580ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:31.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.532+0000 7f0f55e00700 1 -- 172.21.15.89:0/3101393757 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f580d2a40 con 0x7f0f581611b0 2024-04-02T07:48:31.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.532+0000 7f0f55e00700 1 --2- 172.21.15.89:0/3101393757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f581611b0 0x7f0f5815ef90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0f580d2f60 tx=0x7f0f48009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a08e43f33830f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:31.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.533+0000 7f0f54a00700 1 -- 172.21.15.89:0/3101393757 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f48011830 con 0x7f0f581611b0 2024-04-02T07:48:31.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.533+0000 7f0f54a00700 1 -- 172.21.15.89:0/3101393757 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f48011990 con 0x7f0f581611b0 2024-04-02T07:48:31.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.533+0000 7f0f54a00700 1 -- 172.21.15.89:0/3101393757 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f48011d20 con 0x7f0f581611b0 2024-04-02T07:48:31.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.533+0000 7f0f5d200700 1 -- 172.21.15.89:0/3101393757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f581611b0 msgr2=0x7f0f5815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:31.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.533+0000 7f0f5d200700 1 --2- 172.21.15.89:0/3101393757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f581611b0 0x7f0f5815ef90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0f580d2f60 tx=0x7f0f48009b20 comp rx=0 tx=0).stop 2024-04-02T07:48:31.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.533+0000 7f0f5d200700 1 -- 172.21.15.89:0/3101393757 shutdown_connections 2024-04-02T07:48:31.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.533+0000 7f0f5d200700 1 --2- 172.21.15.89:0/3101393757 >> v2:172.21.15.113:3300/0 conn(0x7f0f580cee10 0x7f0f58161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.533+0000 7f0f5d200700 1 --2- 172.21.15.89:0/3101393757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f581611b0 0x7f0f5815ef90 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.533+0000 7f0f5d200700 1 -- 172.21.15.89:0/3101393757 >> 172.21.15.89:0/3101393757 conn(0x7f0f5808aff0 msgr2=0x7f0f580d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:31.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.533+0000 7f0f5d200700 1 -- 172.21.15.89:0/3101393757 shutdown_connections 2024-04-02T07:48:31.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.533+0000 7f0f5d200700 1 -- 172.21.15.89:0/3101393757 wait complete. 2024-04-02T07:48:31.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.535+0000 7f0f5d200700 1 Processor -- start 2024-04-02T07:48:31.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.535+0000 7f0f5d200700 1 -- start start 2024-04-02T07:48:31.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.536+0000 7f0f5d200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f580cee10 0x7f0f580b7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:31.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.536+0000 7f0f5d200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f5815f4d0 0x7f0f581603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:31.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.536+0000 7f0f5d200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f581611b0 0x7f0f580b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:31.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.536+0000 7f0f5d200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f580b78b0 con 0x7f0f580cee10 2024-04-02T07:48:31.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.536+0000 7f0f5d200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f580b86f0 con 0x7f0f5815f4d0 2024-04-02T07:48:31.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.536+0000 7f0f5d200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f580b7ae0 con 0x7f0f581611b0 2024-04-02T07:48:31.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.536+0000 7f0f55e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f580cee10 0x7f0f580b7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:31.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.536+0000 7f0f55e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f580cee10 0x7f0f580b7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36272/0 (socket says 172.21.15.89:36272) 2024-04-02T07:48:31.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.536+0000 7f0f55e00700 1 -- 172.21.15.89:0/3602553617 learned_addr learned my addr 172.21.15.89:0/3602553617 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:31.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.537+0000 7f0f55400700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f5815f4d0 0x7f0f581603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:31.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.537+0000 7f0f56800700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f581611b0 0x7f0f580b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:31.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.537+0000 7f0f55e00700 1 -- 172.21.15.89:0/3602553617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f581611b0 msgr2=0x7f0f580b6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:31.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.537+0000 7f0f55e00700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f581611b0 0x7f0f580b6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.537+0000 7f0f55e00700 1 -- 172.21.15.89:0/3602553617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f5815f4d0 msgr2=0x7f0f581603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:31.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.537+0000 7f0f55e00700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f5815f4d0 0x7f0f581603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.537+0000 7f0f55e00700 1 -- 172.21.15.89:0/3602553617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f480094a0 con 0x7f0f580cee10 2024-04-02T07:48:31.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.537+0000 7f0f56800700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f581611b0 0x7f0f580b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:48:31.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.537+0000 7f0f55e00700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f580cee10 0x7f0f580b7370 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0f48000c00 tx=0x7f0f48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:31.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.537+0000 7f0f4e000700 1 -- 172.21.15.89:0/3602553617 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f480045d0 con 0x7f0f580cee10 2024-04-02T07:48:31.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.537+0000 7f0f4e000700 1 -- 172.21.15.89:0/3602553617 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4801e070 con 0x7f0f580cee10 2024-04-02T07:48:31.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.537+0000 7f0f4e000700 1 -- 172.21.15.89:0/3602553617 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f480038e0 con 0x7f0f580cee10 2024-04-02T07:48:31.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.538+0000 7f0f5d200700 1 -- 172.21.15.89:0/3602553617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f580b7dc0 con 0x7f0f580cee10 2024-04-02T07:48:31.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.538+0000 7f0f5d200700 1 -- 172.21.15.89:0/3602553617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f580b8380 con 0x7f0f580cee10 2024-04-02T07:48:31.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.539+0000 7f0f5d200700 1 -- 172.21.15.89:0/3602553617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f58067720 con 0x7f0f580cee10 2024-04-02T07:48:31.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.539+0000 7f0f4e000700 1 -- 172.21.15.89:0/3602553617 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f0f48003a40 con 0x7f0f580cee10 2024-04-02T07:48:31.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.540+0000 7f0f4e000700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f3806aec0 0x7f0f3806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:31.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.540+0000 7f0f4e000700 1 -- 172.21.15.89:0/3602553617 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f0f4808a8e0 con 0x7f0f580cee10 2024-04-02T07:48:31.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.541+0000 7f0f55400700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f3806aec0 0x7f0f3806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:31.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.541+0000 7f0f55400700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f3806aec0 0x7f0f3806d380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0f58161990 tx=0x7f0f44007de0 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:31.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.545+0000 7f0f4e000700 1 -- 172.21.15.89:0/3602553617 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0f48055620 con 0x7f0f580cee10 2024-04-02T07:48:31.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.775+0000 7f0f5d200700 1 -- 172.21.15.89:0/3602553617 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f580cdf00 con 0x7f0f3806aec0 2024-04-02T07:48:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.873+0000 7f0f4e000700 1 -- 172.21.15.89:0/3602553617 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f0f44004b10 con 0x7f0f3806aec0 2024-04-02T07:48:31.875 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path /snap_test_dir1/dir0 2024-04-02T07:48:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.875+0000 7f0f5d200700 1 -- 172.21.15.89:0/3602553617 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f3806aec0 msgr2=0x7f0f3806d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.875+0000 7f0f5d200700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f3806aec0 0x7f0f3806d380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0f58161990 tx=0x7f0f44007de0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.875+0000 7f0f5d200700 1 -- 172.21.15.89:0/3602553617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f580cee10 msgr2=0x7f0f580b7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.875+0000 7f0f5d200700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f580cee10 0x7f0f580b7370 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0f48000c00 tx=0x7f0f48004750 comp rx=0 tx=0).stop 2024-04-02T07:48:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.875+0000 7f0f5d200700 1 -- 172.21.15.89:0/3602553617 shutdown_connections 2024-04-02T07:48:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.875+0000 7f0f5d200700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f3806aec0 0x7f0f3806d380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.876+0000 7f0f5d200700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f581611b0 0x7f0f580b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.876+0000 7f0f5d200700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f580cee10 0x7f0f580b7370 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.876+0000 7f0f5d200700 1 --2- 172.21.15.89:0/3602553617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f5815f4d0 0x7f0f581603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.876+0000 7f0f5d200700 1 -- 172.21.15.89:0/3602553617 >> 172.21.15.89:0/3602553617 conn(0x7f0f5808aff0 msgr2=0x7f0f58161a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.876+0000 7f0f5d200700 1 -- 172.21.15.89:0/3602553617 shutdown_connections 2024-04-02T07:48:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:31.876+0000 7f0f5d200700 1 -- 172.21.15.89:0/3602553617 wait complete. 2024-04-02T07:48:31.896 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2024-04-02T07:48:31.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:31.896 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-04-02T07:48:31.943 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:31.943 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:31.943 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:31.943 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:31.943 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:31.943 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:31.943 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:31.943 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:31.943 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:31.943 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:31.943 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:31.944 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:31.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:31.944 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_38_00_UTC) 2024-04-02T07:48:32.036 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:32.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:32.036 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_39_00_UTC) 2024-04-02T07:48:32.088 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:32.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:32.088 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_40_00_UTC) 2024-04-02T07:48:32.140 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:32.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:32.140 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_41_00_UTC) 2024-04-02T07:48:32.193 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:32.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:32.193 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_42_00_UTC) 2024-04-02T07:48:32.245 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:32.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:32.245 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_43_00_UTC) 2024-04-02T07:48:32.299 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:32.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:32.300 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_44_00_UTC) 2024-04-02T07:48:32.352 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:32.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:32.352 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_45_00_UTC) 2024-04-02T07:48:32.404 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:32.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:32.405 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_46_00_UTC) 2024-04-02T07:48:32.456 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:32.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:32.457 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_47_00_UTC) 2024-04-02T07:48:32.508 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:32.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:32.509 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-02-07_48_00_UTC) 2024-04-02T07:48:32.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:32.561 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2024-04-02T07:48:32.609 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2024-04-02T07:48:32.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.802+0000 7f089b800700 1 Processor -- start 2024-04-02T07:48:32.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.802+0000 7f089b800700 1 -- start start 2024-04-02T07:48:32.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.803+0000 7f089b800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f08940d16a0 0x7f08940d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:32.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.803+0000 7f089b800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f08940d2370 0x7f08940ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:32.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.803+0000 7f089b800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0894161420 con 0x7f089415d9d0 2024-04-02T07:48:32.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.803+0000 7f089b800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08940d4340 con 0x7f08940d16a0 2024-04-02T07:48:32.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.803+0000 7f089b800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f089415edc0 con 0x7f08940d2370 2024-04-02T07:48:32.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.803+0000 7f0893e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f08940d16a0 0x7f08940d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.803+0000 7f0893e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f08940d16a0 0x7f08940d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36286/0 (socket says 172.21.15.89:36286) 2024-04-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.803+0000 7f0893e00700 1 -- 172.21.15.89:0/4036669452 learned_addr learned my addr 172.21.15.89:0/4036669452 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.804+0000 7f0899800700 1 --2- 172.21.15.89:0/4036669452 >> v2:172.21.15.113:3300/0 conn(0x7f08940d2370 0x7f08940ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.804+0000 7f0893e00700 1 -- 172.21.15.89:0/4036669452 >> v2:172.21.15.113:3300/0 conn(0x7f08940d2370 msgr2=0x7f08940ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.804+0000 7f0893e00700 1 --2- 172.21.15.89:0/4036669452 >> v2:172.21.15.113:3300/0 conn(0x7f08940d2370 0x7f08940ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.804+0000 7f0893e00700 1 -- 172.21.15.89:0/4036669452 >> v1:172.21.15.113:6789/0 conn(0x7f089415d9d0 legacy=0x7f08940d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.804+0000 7f0893e00700 1 -- 172.21.15.89:0/4036669452 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0894161700 con 0x7f08940d16a0 2024-04-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.804+0000 7f0893e00700 1 --2- 172.21.15.89:0/4036669452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08940d16a0 0x7f08940d1e30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f089408ca10 tx=0x7f0880009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad1d3eee01c5650 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.804+0000 7f0893400700 1 -- 172.21.15.89:0/4036669452 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880005e30 con 0x7f08940d16a0 2024-04-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.804+0000 7f0893400700 1 -- 172.21.15.89:0/4036669452 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0880006030 con 0x7f08940d16a0 2024-04-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.804+0000 7f0893400700 1 -- 172.21.15.89:0/4036669452 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08800063e0 con 0x7f08940d16a0 2024-04-02T07:48:32.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.805+0000 7f089b800700 1 -- 172.21.15.89:0/4036669452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08940d16a0 msgr2=0x7f08940d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:32.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.805+0000 7f089b800700 1 --2- 172.21.15.89:0/4036669452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08940d16a0 0x7f08940d1e30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f089408ca10 tx=0x7f0880009bd0 comp rx=0 tx=0).stop 2024-04-02T07:48:32.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.805+0000 7f089b800700 1 -- 172.21.15.89:0/4036669452 shutdown_connections 2024-04-02T07:48:32.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.805+0000 7f089b800700 1 --2- 172.21.15.89:0/4036669452 >> v2:172.21.15.113:3300/0 conn(0x7f08940d2370 0x7f08940ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:32.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.805+0000 7f089b800700 1 --2- 172.21.15.89:0/4036669452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08940d16a0 0x7f08940d1e30 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:32.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.805+0000 7f089b800700 1 -- 172.21.15.89:0/4036669452 >> 172.21.15.89:0/4036669452 conn(0x7f089408aff0 msgr2=0x7f08940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:32.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.805+0000 7f089b800700 1 -- 172.21.15.89:0/4036669452 shutdown_connections 2024-04-02T07:48:32.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.805+0000 7f089b800700 1 -- 172.21.15.89:0/4036669452 wait complete. 2024-04-02T07:48:32.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.807+0000 7f089b800700 1 Processor -- start 2024-04-02T07:48:32.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.807+0000 7f089b800700 1 -- start start 2024-04-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f089b800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08940d16a0 0x7f089415c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f089b800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08940d2370 0x7f089415bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f089b800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f089415d9d0 0x7f08940b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f089b800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08941e8850 con 0x7f08940d2370 2024-04-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f089b800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08940b7580 con 0x7f08940d16a0 2024-04-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f089b800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08941e8570 con 0x7f089415d9d0 2024-04-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f0893e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08940d2370 0x7f089415bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f0893e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08940d2370 0x7f089415bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36298/0 (socket says 172.21.15.89:36298) 2024-04-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f0893e00700 1 -- 172.21.15.89:0/1206133487 learned_addr learned my addr 172.21.15.89:0/1206133487 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f0898e00700 1 --2- 172.21.15.89:0/1206133487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08940d16a0 0x7f089415c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f0899800700 1 --2- 172.21.15.89:0/1206133487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f089415d9d0 0x7f08940b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f0893e00700 1 -- 172.21.15.89:0/1206133487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f089415d9d0 msgr2=0x7f08940b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f0893e00700 1 --2- 172.21.15.89:0/1206133487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f089415d9d0 0x7f08940b89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f0893e00700 1 -- 172.21.15.89:0/1206133487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08940d16a0 msgr2=0x7f089415c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f0893e00700 1 --2- 172.21.15.89:0/1206133487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08940d16a0 0x7f089415c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.808+0000 7f0893e00700 1 -- 172.21.15.89:0/1206133487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0880009500 con 0x7f08940d2370 2024-04-02T07:48:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.809+0000 7f0893e00700 1 --2- 172.21.15.89:0/1206133487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08940d2370 0x7f089415bf90 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0880006000 tx=0x7f0880004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.809+0000 7f0890c00700 1 -- 172.21.15.89:0/1206133487 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880004250 con 0x7f08940d2370 2024-04-02T07:48:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.809+0000 7f0890c00700 1 -- 172.21.15.89:0/1206133487 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08800043b0 con 0x7f08940d2370 2024-04-02T07:48:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.809+0000 7f0890c00700 1 -- 172.21.15.89:0/1206133487 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08800036a0 con 0x7f08940d2370 2024-04-02T07:48:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.809+0000 7f089b800700 1 -- 172.21.15.89:0/1206133487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08940b7210 con 0x7f08940d2370 2024-04-02T07:48:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.809+0000 7f089b800700 1 -- 172.21.15.89:0/1206133487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08940cac40 con 0x7f08940d2370 2024-04-02T07:48:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.810+0000 7f089b800700 1 -- 172.21.15.89:0/1206133487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08940d1ca0 con 0x7f08940d2370 2024-04-02T07:48:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.811+0000 7f0890c00700 1 -- 172.21.15.89:0/1206133487 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f0880022070 con 0x7f08940d2370 2024-04-02T07:48:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.812+0000 7f0890c00700 1 --2- 172.21.15.89:0/1206133487 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f087406ae90 0x7f087406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.812+0000 7f0890c00700 1 -- 172.21.15.89:0/1206133487 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6083+0+0 (secure 0 0 0) 0x7f088008b0f0 con 0x7f08940d2370 2024-04-02T07:48:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.812+0000 7f0898e00700 1 --2- 172.21.15.89:0/1206133487 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f087406ae90 0x7f087406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.813+0000 7f0898e00700 1 --2- 172.21.15.89:0/1206133487 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f087406ae90 0x7f087406d350 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0888009870 tx=0x7f0888008040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:32.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:32.817+0000 7f0890c00700 1 -- 172.21.15.89:0/1206133487 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0880055ec0 con 0x7f08940d2370 2024-04-02T07:48:33.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:33.048+0000 7f089b800700 1 -- 172.21.15.89:0/1206133487 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f08941e86e0 con 0x7f087406ae90 2024-04-02T07:48:33.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:33.166+0000 7f0890c00700 1 -- 172.21.15.89:0/1206133487 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f08941e86e0 con 0x7f087406ae90 2024-04-02T07:48:33.168 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path /snap_test_dir1/dir1 2024-04-02T07:48:33.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:33.169+0000 7f089b800700 1 -- 172.21.15.89:0/1206133487 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f087406ae90 msgr2=0x7f087406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:33.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:33.169+0000 7f089b800700 1 --2- 172.21.15.89:0/1206133487 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f087406ae90 0x7f087406d350 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0888009870 tx=0x7f0888008040 comp rx=0 tx=0).stop 2024-04-02T07:48:33.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:33.169+0000 7f089b800700 1 -- 172.21.15.89:0/1206133487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08940d2370 msgr2=0x7f089415bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:33.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:33.169+0000 7f089b800700 1 --2- 172.21.15.89:0/1206133487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08940d2370 0x7f089415bf90 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0880006000 tx=0x7f0880004750 comp rx=0 tx=0).stop 2024-04-02T07:48:33.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:33.169+0000 7f089b800700 1 -- 172.21.15.89:0/1206133487 shutdown_connections 2024-04-02T07:48:33.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:33.169+0000 7f089b800700 1 --2- 172.21.15.89:0/1206133487 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f087406ae90 0x7f087406d350 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:33.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:33.169+0000 7f089b800700 1 --2- 172.21.15.89:0/1206133487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08940d16a0 0x7f089415c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:33.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:33.169+0000 7f089b800700 1 --2- 172.21.15.89:0/1206133487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f089415d9d0 0x7f08940b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:33.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:33.169+0000 7f089b800700 1 --2- 172.21.15.89:0/1206133487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08940d2370 0x7f089415bf90 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:33.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:33.169+0000 7f089b800700 1 -- 172.21.15.89:0/1206133487 >> 172.21.15.89:0/1206133487 conn(0x7f089408aff0 msgr2=0x7f08940b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:33.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:33.169+0000 7f089b800700 1 -- 172.21.15.89:0/1206133487 shutdown_connections 2024-04-02T07:48:33.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:33.169+0000 7f089b800700 1 -- 172.21.15.89:0/1206133487 wait complete. 2024-04-02T07:48:33.188 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2024-04-02T07:48:33.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:33.189 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-04-02T07:48:33.234 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:33.234 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:33.234 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:33.234 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:33.234 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:33.234 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:33.234 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:33.234 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:33.235 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:33.235 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:33.235 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:33.236 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:33.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:33.236 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_38_00_UTC) 2024-04-02T07:48:33.332 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:33.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:33.332 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_39_00_UTC) 2024-04-02T07:48:33.387 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:33.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:33.387 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_40_00_UTC) 2024-04-02T07:48:33.438 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:33.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:33.438 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_41_00_UTC) 2024-04-02T07:48:33.489 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:33.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:33.489 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_42_00_UTC) 2024-04-02T07:48:33.540 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:33.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:33.540 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_43_00_UTC) 2024-04-02T07:48:33.593 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:33.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:33.593 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_44_00_UTC) 2024-04-02T07:48:33.644 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:33.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:33.644 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_45_00_UTC) 2024-04-02T07:48:33.697 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:33.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:33.698 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_46_00_UTC) 2024-04-02T07:48:33.748 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:33.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:33.749 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_47_00_UTC) 2024-04-02T07:48:33.800 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:33.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:33.800 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-02-07_48_00_UTC) 2024-04-02T07:48:33.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:33.852 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2024-04-02T07:48:33.899 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2024-04-02T07:48:34.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.104+0000 7f2777400700 1 Processor -- start 2024-04-02T07:48:34.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.105+0000 7f2777400700 1 -- start start 2024-04-02T07:48:34.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.106+0000 7f2777400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f27700cbb30 0x7f27700cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:34.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.106+0000 7f2777400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f27700cd480 0x7f277006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:34.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.106+0000 7f2777400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27700cc450 con 0x7f27700cd020 2024-04-02T07:48:34.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.106+0000 7f2777400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27700cd8f0 con 0x7f27700cd480 2024-04-02T07:48:34.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.106+0000 7f2777400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27700cdae0 con 0x7f27700cbb30 2024-04-02T07:48:34.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.106+0000 7f2775400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f27700cd480 0x7f277006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:34.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.106+0000 7f2775400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f27700cd480 0x7f277006e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36306/0 (socket says 172.21.15.89:36306) 2024-04-02T07:48:34.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.106+0000 7f2775400700 1 -- 172.21.15.89:0/1693565552 learned_addr learned my addr 172.21.15.89:0/1693565552 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:34.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.106+0000 7f2775400700 1 -- 172.21.15.89:0/1693565552 >> v2:172.21.15.113:3300/0 conn(0x7f27700cbb30 msgr2=0x7f27700cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:34.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.106+0000 7f2775400700 1 --2- 172.21.15.89:0/1693565552 >> v2:172.21.15.113:3300/0 conn(0x7f27700cbb30 0x7f27700cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:34.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.107+0000 7f2775400700 1 -- 172.21.15.89:0/1693565552 >> v1:172.21.15.113:6789/0 conn(0x7f27700cd020 legacy=0x7f27700b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:34.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.107+0000 7f2775400700 1 -- 172.21.15.89:0/1693565552 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27700cddc0 con 0x7f27700cd480 2024-04-02T07:48:34.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.107+0000 7f2775400700 1 --2- 172.21.15.89:0/1693565552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27700cd480 0x7f277006e420 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f277006fd00 tx=0x7f276800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b667764cab74618d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:34.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.107+0000 7f276f400700 1 -- 172.21.15.89:0/1693565552 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276800e070 con 0x7f27700cd480 2024-04-02T07:48:34.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.107+0000 7f276f400700 1 -- 172.21.15.89:0/1693565552 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2768003a20 con 0x7f27700cd480 2024-04-02T07:48:34.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.107+0000 7f276f400700 1 -- 172.21.15.89:0/1693565552 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2768003dd0 con 0x7f27700cd480 2024-04-02T07:48:34.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.108+0000 7f2777400700 1 -- 172.21.15.89:0/1693565552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27700cd480 msgr2=0x7f277006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:34.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.108+0000 7f2777400700 1 --2- 172.21.15.89:0/1693565552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27700cd480 0x7f277006e420 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f277006fd00 tx=0x7f276800b330 comp rx=0 tx=0).stop 2024-04-02T07:48:34.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.108+0000 7f2777400700 1 -- 172.21.15.89:0/1693565552 shutdown_connections 2024-04-02T07:48:34.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.108+0000 7f2777400700 1 --2- 172.21.15.89:0/1693565552 >> v2:172.21.15.113:3300/0 conn(0x7f27700cbb30 0x7f27700cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:34.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.108+0000 7f2777400700 1 --2- 172.21.15.89:0/1693565552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27700cd480 0x7f277006e420 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:34.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.108+0000 7f2777400700 1 -- 172.21.15.89:0/1693565552 >> 172.21.15.89:0/1693565552 conn(0x7f27700b0d70 msgr2=0x7f27700b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:34.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.108+0000 7f2777400700 1 -- 172.21.15.89:0/1693565552 shutdown_connections 2024-04-02T07:48:34.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.108+0000 7f2777400700 1 -- 172.21.15.89:0/1693565552 wait complete. 2024-04-02T07:48:34.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.110+0000 7f2777400700 1 Processor -- start 2024-04-02T07:48:34.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.110+0000 7f2777400700 1 -- start start 2024-04-02T07:48:34.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.111+0000 7f2777400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27700cbb30 0x7f277014b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:34.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.111+0000 7f2777400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27700cd020 0x7f27701e4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:34.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.111+0000 7f2777400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27700cd480 0x7f27701e5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:34.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.111+0000 7f2777400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277009b800 con 0x7f27700cbb30 2024-04-02T07:48:34.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.111+0000 7f2777400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277009a1a0 con 0x7f27700cd480 2024-04-02T07:48:34.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.111+0000 7f2777400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f277009a480 con 0x7f27700cd020 2024-04-02T07:48:34.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.112+0000 7f2774a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27700cbb30 0x7f277014b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:34.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.112+0000 7f2774a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27700cbb30 0x7f277014b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36312/0 (socket says 172.21.15.89:36312) 2024-04-02T07:48:34.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.112+0000 7f2774a00700 1 -- 172.21.15.89:0/2269276239 learned_addr learned my addr 172.21.15.89:0/2269276239 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:34.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.112+0000 7f276fe00700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27700cd020 0x7f27701e4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:34.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.112+0000 7f2775400700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27700cd480 0x7f27701e5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:34.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.112+0000 7f2774a00700 1 -- 172.21.15.89:0/2269276239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27700cd020 msgr2=0x7f27701e4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:34.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.112+0000 7f2774a00700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27700cd020 0x7f27701e4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:34.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.112+0000 7f2774a00700 1 -- 172.21.15.89:0/2269276239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27700cd480 msgr2=0x7f27701e5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:34.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.112+0000 7f2774a00700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27700cd480 0x7f27701e5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:34.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.112+0000 7f2774a00700 1 -- 172.21.15.89:0/2269276239 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2768009b30 con 0x7f27700cbb30 2024-04-02T07:48:34.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.112+0000 7f276fe00700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27700cd020 0x7f27701e4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:48:34.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.112+0000 7f2775400700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27700cd480 0x7f27701e5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:48:34.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.112+0000 7f2774a00700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27700cbb30 0x7f277014b450 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f276400b4a0 tx=0x7f276400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:34.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.113+0000 7f276cc00700 1 -- 172.21.15.89:0/2269276239 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2764011830 con 0x7f27700cbb30 2024-04-02T07:48:34.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.113+0000 7f276cc00700 1 -- 172.21.15.89:0/2269276239 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2764011990 con 0x7f27700cbb30 2024-04-02T07:48:34.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.113+0000 7f2777400700 1 -- 172.21.15.89:0/2269276239 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277009a760 con 0x7f27700cbb30 2024-04-02T07:48:34.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.113+0000 7f276cc00700 1 -- 172.21.15.89:0/2269276239 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2764011d40 con 0x7f27700cbb30 2024-04-02T07:48:34.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.113+0000 7f2777400700 1 -- 172.21.15.89:0/2269276239 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27701e3fa0 con 0x7f27700cbb30 2024-04-02T07:48:34.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.114+0000 7f2777400700 1 -- 172.21.15.89:0/2269276239 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2770067720 con 0x7f27700cbb30 2024-04-02T07:48:34.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.115+0000 7f276cc00700 1 -- 172.21.15.89:0/2269276239 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f276401a400 con 0x7f27700cbb30 2024-04-02T07:48:34.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.116+0000 7f276cc00700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f275006ae90 0x7f275006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:34.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.116+0000 7f276cc00700 1 -- 172.21.15.89:0/2269276239 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6395+0+0 (secure 0 0 0) 0x7f2764089900 con 0x7f27700cbb30 2024-04-02T07:48:34.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.116+0000 7f276fe00700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f275006ae90 0x7f275006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:34.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.116+0000 7f276fe00700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f275006ae90 0x7f275006d350 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f277006eea0 tx=0x7f275c008040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:34.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.120+0000 7f276cc00700 1 -- 172.21.15.89:0/2269276239 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f27640545c0 con 0x7f27700cbb30 2024-04-02T07:48:34.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.351+0000 7f2777400700 1 -- 172.21.15.89:0/2269276239 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f277006df00 con 0x7f275006ae90 2024-04-02T07:48:34.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.473+0000 7f276cc00700 1 -- 172.21.15.89:0/2269276239 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f275c009db0 con 0x7f275006ae90 2024-04-02T07:48:34.475 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path /snap_test_dir1/dir2 2024-04-02T07:48:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.475+0000 7f2777400700 1 -- 172.21.15.89:0/2269276239 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f275006ae90 msgr2=0x7f275006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.475+0000 7f2777400700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f275006ae90 0x7f275006d350 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f277006eea0 tx=0x7f275c008040 comp rx=0 tx=0).stop 2024-04-02T07:48:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.475+0000 7f2777400700 1 -- 172.21.15.89:0/2269276239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27700cbb30 msgr2=0x7f277014b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.475+0000 7f2777400700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27700cbb30 0x7f277014b450 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f276400b4a0 tx=0x7f276400b860 comp rx=0 tx=0).stop 2024-04-02T07:48:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.475+0000 7f2777400700 1 -- 172.21.15.89:0/2269276239 shutdown_connections 2024-04-02T07:48:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.475+0000 7f2777400700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f275006ae90 0x7f275006d350 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.475+0000 7f2777400700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27700cd480 0x7f27701e5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.475+0000 7f2777400700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27700cbb30 0x7f277014b450 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.475+0000 7f2777400700 1 --2- 172.21.15.89:0/2269276239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27700cd020 0x7f27701e4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:34.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.475+0000 7f2777400700 1 -- 172.21.15.89:0/2269276239 >> 172.21.15.89:0/2269276239 conn(0x7f27700b0d70 msgr2=0x7f277015aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:34.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.475+0000 7f2777400700 1 -- 172.21.15.89:0/2269276239 shutdown_connections 2024-04-02T07:48:34.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:34.476+0000 7f2777400700 1 -- 172.21.15.89:0/2269276239 wait complete. 2024-04-02T07:48:34.495 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2024-04-02T07:48:34.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:34.495 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-04-02T07:48:34.540 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:34.541 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:34.541 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:34.541 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:34.541 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:34.541 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:34.541 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:34.541 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:34.541 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:34.541 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:34.541 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:34.542 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:34.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:34.542 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_38_00_UTC) 2024-04-02T07:48:34.635 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:34.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:34.636 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_39_00_UTC) 2024-04-02T07:48:34.697 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:34.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:34.697 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_40_00_UTC) 2024-04-02T07:48:34.748 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:34.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:34.749 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_41_00_UTC) 2024-04-02T07:48:34.800 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:34.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:34.800 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_42_00_UTC) 2024-04-02T07:48:34.851 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:34.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:34.851 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_43_00_UTC) 2024-04-02T07:48:34.902 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:34.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:34.902 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_44_00_UTC) 2024-04-02T07:48:34.953 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:34.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:34.953 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_45_00_UTC) 2024-04-02T07:48:35.005 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:35.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:35.005 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_46_00_UTC) 2024-04-02T07:48:35.055 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:35.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:35.056 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_47_00_UTC) 2024-04-02T07:48:35.132 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:35.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:35.133 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-02-07_48_00_UTC) 2024-04-02T07:48:35.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:35.185 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2024-04-02T07:48:35.233 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2024-04-02T07:48:35.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.464+0000 7f7bcaa00700 1 Processor -- start 2024-04-02T07:48:35.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.464+0000 7f7bcaa00700 1 -- start start 2024-04-02T07:48:35.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.465+0000 7f7bcaa00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7bc415dad0 0x7f7bc40ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:35.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.465+0000 7f7bcaa00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc415f820 con 0x7f7bc40d21d0 2024-04-02T07:48:35.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.465+0000 7f7bcaa00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc40cf420 con 0x7f7bc40d1630 2024-04-02T07:48:35.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.465+0000 7f7bcaa00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc40cfcc0 con 0x7f7bc415dad0 2024-04-02T07:48:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.466+0000 7f7bc8a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7bc40d21d0 0x7f7bc415fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51644/0 (socket says 172.21.15.89:51644) 2024-04-02T07:48:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.466+0000 7f7bc8a00700 1 -- 172.21.15.89:0/2361922156 learned_addr learned my addr 172.21.15.89:0/2361922156 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.466+0000 7f7bc3e00700 1 --2- 172.21.15.89:0/2361922156 >> v2:172.21.15.113:3300/0 conn(0x7f7bc415dad0 0x7f7bc40ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.466+0000 7f7bc2a00700 1 -- 172.21.15.89:0/2361922156 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326443336 0 0) 0x7f7bc415f820 con 0x7f7bc40d21d0 2024-04-02T07:48:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.466+0000 7f7bc2a00700 1 -- 172.21.15.89:0/2361922156 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ba80035b0 con 0x7f7bc40d21d0 2024-04-02T07:48:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.466+0000 7f7bc2a00700 1 -- 172.21.15.89:0/2361922156 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220179465 0 0) 0x7f7bc40cf420 con 0x7f7bc40d1630 2024-04-02T07:48:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.466+0000 7f7bc3e00700 1 -- 172.21.15.89:0/2361922156 >> v1:172.21.15.113:6789/0 conn(0x7f7bc40d1630 legacy=0x7f7bc40d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.466+0000 7f7bc3e00700 1 -- 172.21.15.89:0/2361922156 >> v1:172.21.15.89:6789/0 conn(0x7f7bc40d21d0 legacy=0x7f7bc415fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.466+0000 7f7bc3e00700 1 -- 172.21.15.89:0/2361922156 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc40cf700 con 0x7f7bc415dad0 2024-04-02T07:48:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.467+0000 7f7bc3e00700 1 --2- 172.21.15.89:0/2361922156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bc415dad0 0x7f7bc40ced80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009840 tx=0x7f7bb4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1147ad902c892f00 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.467+0000 7f7bc2a00700 1 -- 172.21.15.89:0/2361922156 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb4011830 con 0x7f7bc415dad0 2024-04-02T07:48:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.467+0000 7f7bc2a00700 1 -- 172.21.15.89:0/2361922156 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb4011990 con 0x7f7bc415dad0 2024-04-02T07:48:35.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.467+0000 7f7bc2a00700 1 -- 172.21.15.89:0/2361922156 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb4011d20 con 0x7f7bc415dad0 2024-04-02T07:48:35.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.467+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2361922156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bc415dad0 msgr2=0x7f7bc40ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:35.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.467+0000 7f7bcaa00700 1 --2- 172.21.15.89:0/2361922156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bc415dad0 0x7f7bc40ced80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009840 tx=0x7f7bb4009b50 comp rx=0 tx=0).stop 2024-04-02T07:48:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.467+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2361922156 shutdown_connections 2024-04-02T07:48:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.467+0000 7f7bcaa00700 1 --2- 172.21.15.89:0/2361922156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bc415dad0 0x7f7bc40ced80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.467+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2361922156 >> 172.21.15.89:0/2361922156 conn(0x7f7bc408aff0 msgr2=0x7f7bc40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.467+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2361922156 shutdown_connections 2024-04-02T07:48:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.468+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2361922156 wait complete. 2024-04-02T07:48:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.469+0000 7f7bcaa00700 1 Processor -- start 2024-04-02T07:48:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.469+0000 7f7bcaa00700 1 -- start start 2024-04-02T07:48:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.470+0000 7f7bcaa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7bc40d1630 0x7f7bc4153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.470+0000 7f7bcaa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bc40d21d0 0x7f7bc41f2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.470+0000 7f7bcaa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc415dad0 0x7f7bc4159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.470+0000 7f7bcaa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc4153fc0 con 0x7f7bc415dad0 2024-04-02T07:48:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.470+0000 7f7bcaa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc40b78e0 con 0x7f7bc40d21d0 2024-04-02T07:48:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.470+0000 7f7bcaa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc40b7bc0 con 0x7f7bc40d1630 2024-04-02T07:48:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.470+0000 7f7bc8a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc415dad0 0x7f7bc4159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.470+0000 7f7bc8a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc415dad0 0x7f7bc4159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36322/0 (socket says 172.21.15.89:36322) 2024-04-02T07:48:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.470+0000 7f7bc8a00700 1 -- 172.21.15.89:0/2470326891 learned_addr learned my addr 172.21.15.89:0/2470326891 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.471+0000 7f7bc3e00700 1 --2- 172.21.15.89:0/2470326891 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7bc40d1630 0x7f7bc4153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.471+0000 7f7bc3400700 1 --2- 172.21.15.89:0/2470326891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bc40d21d0 0x7f7bc41f2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.471+0000 7f7bc8a00700 1 -- 172.21.15.89:0/2470326891 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7bc40d1630 msgr2=0x7f7bc4153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.471+0000 7f7bc8a00700 1 --2- 172.21.15.89:0/2470326891 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7bc40d1630 0x7f7bc4153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:35.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.471+0000 7f7bc8a00700 1 -- 172.21.15.89:0/2470326891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bc40d21d0 msgr2=0x7f7bc41f2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:35.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.471+0000 7f7bc8a00700 1 --2- 172.21.15.89:0/2470326891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bc40d21d0 0x7f7bc41f2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:35.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.471+0000 7f7bc8a00700 1 -- 172.21.15.89:0/2470326891 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb40094a0 con 0x7f7bc415dad0 2024-04-02T07:48:35.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.471+0000 7f7bc8a00700 1 --2- 172.21.15.89:0/2470326891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc415dad0 0x7f7bc4159910 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc00cb40 tx=0x7f7bbc00ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:35.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.471+0000 7f7bbbe00700 1 -- 172.21.15.89:0/2470326891 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc00a9e0 con 0x7f7bc415dad0 2024-04-02T07:48:35.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.471+0000 7f7bbbe00700 1 -- 172.21.15.89:0/2470326891 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bbc00ab40 con 0x7f7bc415dad0 2024-04-02T07:48:35.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.471+0000 7f7bbbe00700 1 -- 172.21.15.89:0/2470326891 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc012400 con 0x7f7bc415dad0 2024-04-02T07:48:35.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.471+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2470326891 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc40b7ea0 con 0x7f7bc415dad0 2024-04-02T07:48:35.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.472+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2470326891 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc40b7590 con 0x7f7bc415dad0 2024-04-02T07:48:35.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.472+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2470326891 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc4067720 con 0x7f7bc415dad0 2024-04-02T07:48:35.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.473+0000 7f7bbbe00700 1 -- 172.21.15.89:0/2470326891 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f7bbc01b3b0 con 0x7f7bc415dad0 2024-04-02T07:48:35.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.474+0000 7f7bbbe00700 1 --2- 172.21.15.89:0/2470326891 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7ba406ae90 0x7f7ba406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:35.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.474+0000 7f7bbbe00700 1 -- 172.21.15.89:0/2470326891 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6635+0+0 (secure 0 0 0) 0x7f7bbc089560 con 0x7f7bc415dad0 2024-04-02T07:48:35.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.475+0000 7f7bc3e00700 1 --2- 172.21.15.89:0/2470326891 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7ba406ae90 0x7f7ba406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:35.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.475+0000 7f7bc3e00700 1 --2- 172.21.15.89:0/2470326891 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7ba406ae90 0x7f7ba406d350 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009fd0 tx=0x7f7bb401c040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:35.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.479+0000 7f7bbbe00700 1 -- 172.21.15.89:0/2470326891 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7bbc0541c0 con 0x7f7bc415dad0 2024-04-02T07:48:35.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.706+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2470326891 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bc40b7a50 con 0x7f7ba406ae90 2024-04-02T07:48:35.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.798+0000 7f7bbbe00700 1 -- 172.21.15.89:0/2470326891 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f7bc40b7a50 con 0x7f7ba406ae90 2024-04-02T07:48:35.800 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path /snap_test_dir1/dir3 2024-04-02T07:48:35.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.801+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2470326891 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7ba406ae90 msgr2=0x7f7ba406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:35.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.801+0000 7f7bcaa00700 1 --2- 172.21.15.89:0/2470326891 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7ba406ae90 0x7f7ba406d350 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009fd0 tx=0x7f7bb401c040 comp rx=0 tx=0).stop 2024-04-02T07:48:35.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.801+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2470326891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc415dad0 msgr2=0x7f7bc4159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:35.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.801+0000 7f7bcaa00700 1 --2- 172.21.15.89:0/2470326891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc415dad0 0x7f7bc4159910 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc00cb40 tx=0x7f7bbc00ce50 comp rx=0 tx=0).stop 2024-04-02T07:48:35.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.801+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2470326891 shutdown_connections 2024-04-02T07:48:35.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.801+0000 7f7bcaa00700 1 --2- 172.21.15.89:0/2470326891 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7ba406ae90 0x7f7ba406d350 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:35.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.801+0000 7f7bcaa00700 1 --2- 172.21.15.89:0/2470326891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc415dad0 0x7f7bc4159910 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:35.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.801+0000 7f7bcaa00700 1 --2- 172.21.15.89:0/2470326891 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7bc40d1630 0x7f7bc4153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:35.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.801+0000 7f7bcaa00700 1 --2- 172.21.15.89:0/2470326891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bc40d21d0 0x7f7bc41f2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:35.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.801+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2470326891 >> 172.21.15.89:0/2470326891 conn(0x7f7bc408aff0 msgr2=0x7f7bc41545a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:35.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.801+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2470326891 shutdown_connections 2024-04-02T07:48:35.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:35.801+0000 7f7bcaa00700 1 -- 172.21.15.89:0/2470326891 wait complete. 2024-04-02T07:48:35.821 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2024-04-02T07:48:35.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:35.821 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-04-02T07:48:35.868 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:35.868 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:35.868 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:35.868 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:35.868 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:35.868 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:35.868 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:35.868 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:35.869 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:35.869 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:35.869 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:35.869 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:35.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:35.869 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_38_00_UTC) 2024-04-02T07:48:35.962 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:35.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:35.962 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_39_00_UTC) 2024-04-02T07:48:36.013 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:36.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:36.014 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_40_00_UTC) 2024-04-02T07:48:36.070 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:36.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:36.070 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_41_00_UTC) 2024-04-02T07:48:36.147 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:36.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:36.147 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_42_00_UTC) 2024-04-02T07:48:36.198 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:36.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:36.199 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_43_00_UTC) 2024-04-02T07:48:36.250 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:36.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:36.251 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_44_00_UTC) 2024-04-02T07:48:36.304 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:36.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:36.305 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_45_00_UTC) 2024-04-02T07:48:36.356 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:36.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:36.356 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_46_00_UTC) 2024-04-02T07:48:36.407 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:36.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:36.407 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_47_00_UTC) 2024-04-02T07:48:36.458 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:36.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:36.458 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-02-07_48_00_UTC) 2024-04-02T07:48:36.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:36.536 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2024-04-02T07:48:36.585 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2024-04-02T07:48:36.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.776+0000 7fe76b200700 1 Processor -- start 2024-04-02T07:48:36.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.777+0000 7fe76b200700 1 -- start start 2024-04-02T07:48:36.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe76b200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe7640cd5c0 0x7fe76415e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:36.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe76b200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe76406f310 0x7fe76406f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:36.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe76b200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76415eda0 con 0x7fe76415cfe0 2024-04-02T07:48:36.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe76b200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76406fce0 con 0x7fe7640cd5c0 2024-04-02T07:48:36.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe76b200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe764099dd0 con 0x7fe76406f310 2024-04-02T07:48:36.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe763e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe7640cd5c0 0x7fe76415e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:36.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe769200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe76415cfe0 0x7fe76415d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51654/0 (socket says 172.21.15.89:51654) 2024-04-02T07:48:36.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe769200700 1 -- 172.21.15.89:0/433685122 learned_addr learned my addr 172.21.15.89:0/433685122 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:36.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe763400700 1 --2- 172.21.15.89:0/433685122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe76406f310 0x7fe76406f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:36.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe763e00700 1 -- 172.21.15.89:0/433685122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe76406f310 msgr2=0x7fe76406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:36.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe763e00700 1 --2- 172.21.15.89:0/433685122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe76406f310 0x7fe76406f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe763e00700 1 -- 172.21.15.89:0/433685122 >> v1:172.21.15.89:6789/0 conn(0x7fe76415cfe0 legacy=0x7fe76415d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe763e00700 1 -- 172.21.15.89:0/433685122 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76409a0b0 con 0x7fe7640cd5c0 2024-04-02T07:48:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe763400700 1 --2- 172.21.15.89:0/433685122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe76406f310 0x7fe76406f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.778+0000 7fe762a00700 1 -- 172.21.15.89:0/433685122 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881038264 0 0) 0x7fe75c001490 con 0x7fe76415cfe0 2024-04-02T07:48:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.779+0000 7fe763e00700 1 --2- 172.21.15.89:0/433685122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7640cd5c0 0x7fe76415e860 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe754009840 tx=0x7fe754009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=787dcf29b56223fb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.779+0000 7fe762a00700 1 -- 172.21.15.89:0/433685122 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe754011830 con 0x7fe7640cd5c0 2024-04-02T07:48:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.779+0000 7fe762a00700 1 -- 172.21.15.89:0/433685122 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe754011990 con 0x7fe7640cd5c0 2024-04-02T07:48:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.779+0000 7fe762a00700 1 -- 172.21.15.89:0/433685122 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe754011d20 con 0x7fe7640cd5c0 2024-04-02T07:48:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.779+0000 7fe76b200700 1 -- 172.21.15.89:0/433685122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7640cd5c0 msgr2=0x7fe76415e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.779+0000 7fe76b200700 1 --2- 172.21.15.89:0/433685122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7640cd5c0 0x7fe76415e860 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe754009840 tx=0x7fe754009b50 comp rx=0 tx=0).stop 2024-04-02T07:48:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.780+0000 7fe76b200700 1 -- 172.21.15.89:0/433685122 shutdown_connections 2024-04-02T07:48:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.780+0000 7fe76b200700 1 --2- 172.21.15.89:0/433685122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7640cd5c0 0x7fe76415e860 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.780+0000 7fe76b200700 1 --2- 172.21.15.89:0/433685122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe76406f310 0x7fe76406f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:36.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.780+0000 7fe76b200700 1 -- 172.21.15.89:0/433685122 >> 172.21.15.89:0/433685122 conn(0x7fe7640d1370 msgr2=0x7fe7640d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:36.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.780+0000 7fe76b200700 1 -- 172.21.15.89:0/433685122 shutdown_connections 2024-04-02T07:48:36.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.780+0000 7fe76b200700 1 -- 172.21.15.89:0/433685122 wait complete. 2024-04-02T07:48:36.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.781+0000 7fe76b200700 1 Processor -- start 2024-04-02T07:48:36.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.781+0000 7fe76b200700 1 -- start start 2024-04-02T07:48:36.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.782+0000 7fe76b200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe76406f310 0x7fe7640c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:36.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.782+0000 7fe76b200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7640cd5c0 0x7fe7640c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:36.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.782+0000 7fe76b200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe76415cfe0 0x7fe7640b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:36.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.782+0000 7fe76b200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7640c7d20 con 0x7fe76406f310 2024-04-02T07:48:36.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.782+0000 7fe76b200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7640c54f0 con 0x7fe76415cfe0 2024-04-02T07:48:36.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.782+0000 7fe76b200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76414c290 con 0x7fe7640cd5c0 2024-04-02T07:48:36.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.783+0000 7fe763e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe76406f310 0x7fe7640c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:36.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.783+0000 7fe763e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe76406f310 0x7fe7640c4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36332/0 (socket says 172.21.15.89:36332) 2024-04-02T07:48:36.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.783+0000 7fe763e00700 1 -- 172.21.15.89:0/1635368465 learned_addr learned my addr 172.21.15.89:0/1635368465 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:36.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.783+0000 7fe763e00700 1 -- 172.21.15.89:0/1635368465 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7640cd5c0 msgr2=0x7fe7640c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:36.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.783+0000 7fe763400700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7640cd5c0 0x7fe7640c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:36.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.783+0000 7fe769200700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe76415cfe0 0x7fe7640b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:36.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.783+0000 7fe763e00700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7640cd5c0 0x7fe7640c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:36.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.783+0000 7fe763e00700 1 -- 172.21.15.89:0/1635368465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe76415cfe0 msgr2=0x7fe7640b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:36.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.783+0000 7fe763e00700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe76415cfe0 0x7fe7640b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:36.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.783+0000 7fe763e00700 1 -- 172.21.15.89:0/1635368465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7540094a0 con 0x7fe76406f310 2024-04-02T07:48:36.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.783+0000 7fe769200700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe76415cfe0 0x7fe7640b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:36.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.783+0000 7fe763400700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7640cd5c0 0x7fe7640c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:36.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.783+0000 7fe763e00700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe76406f310 0x7fe7640c4e60 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe7540090b0 tx=0x7fe754004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:36.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.783+0000 7fe75be00700 1 -- 172.21.15.89:0/1635368465 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7540045f0 con 0x7fe76406f310 2024-04-02T07:48:36.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.784+0000 7fe75be00700 1 -- 172.21.15.89:0/1635368465 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75401f070 con 0x7fe76406f310 2024-04-02T07:48:36.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.784+0000 7fe76b200700 1 -- 172.21.15.89:0/1635368465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76414c570 con 0x7fe76406f310 2024-04-02T07:48:36.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.784+0000 7fe75be00700 1 -- 172.21.15.89:0/1635368465 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe754003b00 con 0x7fe76406f310 2024-04-02T07:48:36.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.784+0000 7fe76b200700 1 -- 172.21.15.89:0/1635368465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76414cb30 con 0x7fe76406f310 2024-04-02T07:48:36.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.785+0000 7fe76b200700 1 -- 172.21.15.89:0/1635368465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe764067720 con 0x7fe76406f310 2024-04-02T07:48:36.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.785+0000 7fe75be00700 1 -- 172.21.15.89:0/1635368465 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fe754003c60 con 0x7fe76406f310 2024-04-02T07:48:36.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.786+0000 7fe75be00700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe74406ae90 0x7fe74406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:36.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.786+0000 7fe75be00700 1 -- 172.21.15.89:0/1635368465 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6587+0+0 (secure 0 0 0) 0x7fe754021070 con 0x7fe76406f310 2024-04-02T07:48:36.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.787+0000 7fe763400700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe74406ae90 0x7fe74406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:36.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.787+0000 7fe763400700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe74406ae90 0x7fe74406d350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe7640cc850 tx=0x7fe750008040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:36.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:36.791+0000 7fe75be00700 1 -- 172.21.15.89:0/1635368465 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe7540561a0 con 0x7fe76406f310 2024-04-02T07:48:37.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:37.019+0000 7fe76b200700 1 -- 172.21.15.89:0/1635368465 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe76409a950 con 0x7fe74406ae90 2024-04-02T07:48:37.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:37.143+0000 7fe75be00700 1 -- 172.21.15.89:0/1635368465 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fe75000c040 con 0x7fe74406ae90 2024-04-02T07:48:37.145 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path /snap_test_dir1/dir4 2024-04-02T07:48:37.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:37.145+0000 7fe76b200700 1 -- 172.21.15.89:0/1635368465 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe74406ae90 msgr2=0x7fe74406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:37.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:37.145+0000 7fe76b200700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe74406ae90 0x7fe74406d350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe7640cc850 tx=0x7fe750008040 comp rx=0 tx=0).stop 2024-04-02T07:48:37.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:37.145+0000 7fe76b200700 1 -- 172.21.15.89:0/1635368465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe76406f310 msgr2=0x7fe7640c4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:37.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:37.145+0000 7fe76b200700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe76406f310 0x7fe7640c4e60 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe7540090b0 tx=0x7fe754004750 comp rx=0 tx=0).stop 2024-04-02T07:48:37.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:37.145+0000 7fe76b200700 1 -- 172.21.15.89:0/1635368465 shutdown_connections 2024-04-02T07:48:37.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:37.145+0000 7fe76b200700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe74406ae90 0x7fe74406d350 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:37.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:37.145+0000 7fe76b200700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe76415cfe0 0x7fe7640b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:37.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:37.145+0000 7fe76b200700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe76406f310 0x7fe7640c4e60 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:37.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:37.145+0000 7fe76b200700 1 --2- 172.21.15.89:0/1635368465 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7640cd5c0 0x7fe7640c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:37.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:37.145+0000 7fe76b200700 1 -- 172.21.15.89:0/1635368465 >> 172.21.15.89:0/1635368465 conn(0x7fe7640d1370 msgr2=0x7fe76415d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:37.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:37.145+0000 7fe76b200700 1 -- 172.21.15.89:0/1635368465 shutdown_connections 2024-04-02T07:48:37.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:37.145+0000 7fe76b200700 1 -- 172.21.15.89:0/1635368465 wait complete. 2024-04-02T07:48:37.165 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2024-04-02T07:48:37.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:37.166 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-04-02T07:48:37.211 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:37.211 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:37.211 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:37.211 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:37.212 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:37.212 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:37.212 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:37.212 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:37.212 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:37.212 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:37.212 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:37.213 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:37.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:37.213 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_38_00_UTC) 2024-04-02T07:48:37.307 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:37.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:37.308 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_39_00_UTC) 2024-04-02T07:48:37.358 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:37.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:37.358 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_40_00_UTC) 2024-04-02T07:48:37.409 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:37.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:37.410 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_41_00_UTC) 2024-04-02T07:48:37.461 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:37.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:37.461 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_42_00_UTC) 2024-04-02T07:48:37.512 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:37.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:37.512 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_43_00_UTC) 2024-04-02T07:48:37.562 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:37.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:37.563 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_44_00_UTC) 2024-04-02T07:48:37.620 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:37.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:37.620 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_45_00_UTC) 2024-04-02T07:48:37.671 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:37.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:37.671 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_46_00_UTC) 2024-04-02T07:48:37.723 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:37.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:37.723 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_47_00_UTC) 2024-04-02T07:48:37.775 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:37.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:37.775 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-02-07_48_00_UTC) 2024-04-02T07:48:37.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:37.827 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2024-04-02T07:48:37.874 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2024-04-02T07:48:38.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.065+0000 7f8a2b800700 1 Processor -- start 2024-04-02T07:48:38.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.065+0000 7f8a2b800700 1 -- start start 2024-04-02T07:48:38.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.066+0000 7f8a2b800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8a240cd020 0x7f8a240cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:38.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.066+0000 7f8a2b800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a240b2070 0x7f8a240b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:38.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.066+0000 7f8a2b800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a240cdae0 con 0x7f8a240cbb30 2024-04-02T07:48:38.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.066+0000 7f8a2b800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a240cddc0 con 0x7f8a240cd020 2024-04-02T07:48:38.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.066+0000 7f8a2b800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240ce0a0 con 0x7f8a240b2070 2024-04-02T07:48:38.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.067+0000 7f8a29800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a240b2070 0x7f8a240b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:38.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.067+0000 7f8a29800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a240b2070 0x7f8a240b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:46556/0 (socket says 172.21.15.89:46556) 2024-04-02T07:48:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.067+0000 7f8a29800700 1 -- 172.21.15.89:0/2742060805 learned_addr learned my addr 172.21.15.89:0/2742060805 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.067+0000 7f8a28e00700 1 --1- 172.21.15.89:0/2742060805 >> v1:172.21.15.113:6789/0 conn(0x7f8a240cbb30 0x7f8a240cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:58324/0 (socket says 172.21.15.89:58324) 2024-04-02T07:48:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.067+0000 7f8a23e00700 1 --2- 172.21.15.89:0/2742060805 >> v2:172.21.15.113:3300/0 conn(0x7f8a240cd020 0x7f8a240cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.067+0000 7f8a29800700 1 -- 172.21.15.89:0/2742060805 >> v2:172.21.15.113:3300/0 conn(0x7f8a240cd020 msgr2=0x7f8a240cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.067+0000 7f8a29800700 1 --2- 172.21.15.89:0/2742060805 >> v2:172.21.15.113:3300/0 conn(0x7f8a240cd020 0x7f8a240cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.067+0000 7f8a29800700 1 -- 172.21.15.89:0/2742060805 >> v1:172.21.15.113:6789/0 conn(0x7f8a240cbb30 legacy=0x7f8a240cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.067+0000 7f8a29800700 1 -- 172.21.15.89:0/2742060805 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2406dc20 con 0x7f8a240b2070 2024-04-02T07:48:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.068+0000 7f8a29800700 1 --2- 172.21.15.89:0/2742060805 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a240b2070 0x7f8a240b2500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009e50 tx=0x7f8a1c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a511712ba3f6af95 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.071+0000 7f8a23400700 1 -- 172.21.15.89:0/2742060805 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c00e070 con 0x7f8a240b2070 2024-04-02T07:48:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.071+0000 7f8a23400700 1 -- 172.21.15.89:0/2742060805 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c00b910 con 0x7f8a240b2070 2024-04-02T07:48:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.071+0000 7f8a23400700 1 -- 172.21.15.89:0/2742060805 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c00bcc0 con 0x7f8a240b2070 2024-04-02T07:48:38.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.072+0000 7f8a2b800700 1 -- 172.21.15.89:0/2742060805 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a240b2070 msgr2=0x7f8a240b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.072+0000 7f8a2b800700 1 --2- 172.21.15.89:0/2742060805 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a240b2070 0x7f8a240b2500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009e50 tx=0x7f8a1c00b330 comp rx=0 tx=0).stop 2024-04-02T07:48:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.073+0000 7f8a2b800700 1 -- 172.21.15.89:0/2742060805 shutdown_connections 2024-04-02T07:48:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.073+0000 7f8a2b800700 1 --2- 172.21.15.89:0/2742060805 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a240b2070 0x7f8a240b2500 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.073+0000 7f8a2b800700 1 --2- 172.21.15.89:0/2742060805 >> v2:172.21.15.113:3300/0 conn(0x7f8a240cd020 0x7f8a240cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.073+0000 7f8a2b800700 1 -- 172.21.15.89:0/2742060805 >> 172.21.15.89:0/2742060805 conn(0x7f8a240b0d70 msgr2=0x7f8a240b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.073+0000 7f8a2b800700 1 -- 172.21.15.89:0/2742060805 shutdown_connections 2024-04-02T07:48:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.073+0000 7f8a2b800700 1 -- 172.21.15.89:0/2742060805 wait complete. 2024-04-02T07:48:38.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.076+0000 7f8a2b800700 1 Processor -- start 2024-04-02T07:48:38.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.076+0000 7f8a2b800700 1 -- start start 2024-04-02T07:48:38.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.077+0000 7f8a2b800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a240b2070 0x7f8a241f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:38.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.077+0000 7f8a2b800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a240cbb30 0x7f8a24154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:38.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.077+0000 7f8a2b800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a240cd020 0x7f8a2415a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:38.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.077+0000 7f8a2b800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a24155400 con 0x7f8a240cd020 2024-04-02T07:48:38.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.077+0000 7f8a2b800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a241556a0 con 0x7f8a240cbb30 2024-04-02T07:48:38.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.077+0000 7f8a2b800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a24155980 con 0x7f8a240b2070 2024-04-02T07:48:38.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.078+0000 7f8a29800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a240cd020 0x7f8a2415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:38.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.078+0000 7f8a29800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a240cd020 0x7f8a2415a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36334/0 (socket says 172.21.15.89:36334) 2024-04-02T07:48:38.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.078+0000 7f8a29800700 1 -- 172.21.15.89:0/991971376 learned_addr learned my addr 172.21.15.89:0/991971376 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:38.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.078+0000 7f8a28e00700 1 --2- 172.21.15.89:0/991971376 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a240b2070 0x7f8a241f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:38.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.079+0000 7f8a29800700 1 -- 172.21.15.89:0/991971376 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a240b2070 msgr2=0x7f8a241f7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:38.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.079+0000 7f8a29800700 1 --2- 172.21.15.89:0/991971376 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a240b2070 0x7f8a241f7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:38.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.079+0000 7f8a29800700 1 -- 172.21.15.89:0/991971376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a240cbb30 msgr2=0x7f8a24154ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:38.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.079+0000 7f8a29800700 1 --2- 172.21.15.89:0/991971376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a240cbb30 0x7f8a24154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:38.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.079+0000 7f8a29800700 1 -- 172.21.15.89:0/991971376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c009b30 con 0x7f8a240cd020 2024-04-02T07:48:38.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.080+0000 7f8a29800700 1 --2- 172.21.15.89:0/991971376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a240cd020 0x7f8a2415a670 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8a240ce1e0 tx=0x7f8a1c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:38.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.081+0000 7f8a20c00700 1 -- 172.21.15.89:0/991971376 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c0099b0 con 0x7f8a240cd020 2024-04-02T07:48:38.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.081+0000 7f8a20c00700 1 -- 172.21.15.89:0/991971376 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c004250 con 0x7f8a240cd020 2024-04-02T07:48:38.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.081+0000 7f8a20c00700 1 -- 172.21.15.89:0/991971376 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c0045c0 con 0x7f8a240cd020 2024-04-02T07:48:38.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.082+0000 7f8a2b800700 1 -- 172.21.15.89:0/991971376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a24155c60 con 0x7f8a240cd020 2024-04-02T07:48:38.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.083+0000 7f8a2b800700 1 -- 172.21.15.89:0/991971376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a24156220 con 0x7f8a240cd020 2024-04-02T07:48:38.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.088+0000 7f8a2b800700 1 -- 172.21.15.89:0/991971376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a24067720 con 0x7f8a240cd020 2024-04-02T07:48:38.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.088+0000 7f8a20c00700 1 -- 172.21.15.89:0/991971376 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f8a1c004720 con 0x7f8a240cd020 2024-04-02T07:48:38.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.090+0000 7f8a20c00700 1 --2- 172.21.15.89:0/991971376 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a0406aee0 0x7f8a0406d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:38.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.090+0000 7f8a20c00700 1 -- 172.21.15.89:0/991971376 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f8a1c0593a0 con 0x7f8a240cd020 2024-04-02T07:48:38.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.093+0000 7f8a28e00700 1 --2- 172.21.15.89:0/991971376 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a0406aee0 0x7f8a0406d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:38.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.102+0000 7f8a28e00700 1 --2- 172.21.15.89:0/991971376 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a0406aee0 0x7f8a0406d3a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8a18000f60 tx=0x7f8a18009040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:38.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.105+0000 7f8a20c00700 1 -- 172.21.15.89:0/991971376 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8a1c020080 con 0x7f8a240cd020 2024-04-02T07:48:38.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.362+0000 7f8a2b800700 1 -- 172.21.15.89:0/991971376 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a2415bdd0 con 0x7f8a0406aee0 2024-04-02T07:48:38.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.452+0000 7f8a20c00700 1 -- 172.21.15.89:0/991971376 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f8a180012c0 con 0x7f8a0406aee0 2024-04-02T07:48:38.454 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path /snap_test_dir1/dir5 2024-04-02T07:48:38.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.455+0000 7f8a2b800700 1 -- 172.21.15.89:0/991971376 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a0406aee0 msgr2=0x7f8a0406d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:38.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.455+0000 7f8a2b800700 1 --2- 172.21.15.89:0/991971376 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a0406aee0 0x7f8a0406d3a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8a18000f60 tx=0x7f8a18009040 comp rx=0 tx=0).stop 2024-04-02T07:48:38.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.455+0000 7f8a2b800700 1 -- 172.21.15.89:0/991971376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a240cd020 msgr2=0x7f8a2415a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:38.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.455+0000 7f8a2b800700 1 --2- 172.21.15.89:0/991971376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a240cd020 0x7f8a2415a670 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8a240ce1e0 tx=0x7f8a1c004ad0 comp rx=0 tx=0).stop 2024-04-02T07:48:38.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.455+0000 7f8a2b800700 1 -- 172.21.15.89:0/991971376 shutdown_connections 2024-04-02T07:48:38.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.455+0000 7f8a2b800700 1 --2- 172.21.15.89:0/991971376 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a0406aee0 0x7f8a0406d3a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:38.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.455+0000 7f8a2b800700 1 --2- 172.21.15.89:0/991971376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a240cd020 0x7f8a2415a670 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:38.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.455+0000 7f8a2b800700 1 --2- 172.21.15.89:0/991971376 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a240b2070 0x7f8a241f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:38.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.455+0000 7f8a2b800700 1 --2- 172.21.15.89:0/991971376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a240cbb30 0x7f8a24154ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:38.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.455+0000 7f8a2b800700 1 -- 172.21.15.89:0/991971376 >> 172.21.15.89:0/991971376 conn(0x7f8a240b0d70 msgr2=0x7f8a2415cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:38.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.455+0000 7f8a2b800700 1 -- 172.21.15.89:0/991971376 shutdown_connections 2024-04-02T07:48:38.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:38.455+0000 7f8a2b800700 1 -- 172.21.15.89:0/991971376 wait complete. 2024-04-02T07:48:38.475 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2024-04-02T07:48:38.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:38.475 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-04-02T07:48:38.520 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:38.521 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:38.521 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:38.521 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:38.521 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:38.521 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:38.521 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:38.521 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:38.521 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:38.521 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:38.521 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:38.522 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:38.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:38.523 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_38_00_UTC) 2024-04-02T07:48:38.616 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:38.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:38.616 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_39_00_UTC) 2024-04-02T07:48:38.668 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:38.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:38.668 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_40_00_UTC) 2024-04-02T07:48:38.719 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:38.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:38.720 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_41_00_UTC) 2024-04-02T07:48:38.771 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:38.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:38.772 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_42_00_UTC) 2024-04-02T07:48:38.824 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:38.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:38.824 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_43_00_UTC) 2024-04-02T07:48:38.876 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:38.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:38.876 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_44_00_UTC) 2024-04-02T07:48:38.927 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:38.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:38.927 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_45_00_UTC) 2024-04-02T07:48:38.978 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:38.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:38.978 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_46_00_UTC) 2024-04-02T07:48:39.029 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:39.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:39.029 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_47_00_UTC) 2024-04-02T07:48:39.095 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:39.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:39.095 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-02-07_48_00_UTC) 2024-04-02T07:48:39.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:39.155 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2024-04-02T07:48:39.237 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2024-04-02T07:48:39.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.446+0000 7f2074600700 1 Processor -- start 2024-04-02T07:48:39.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.447+0000 7f2074600700 1 -- start start 2024-04-02T07:48:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.448+0000 7f2074600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c0cd020 0x7f206c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.448+0000 7f2074600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206c0cdae0 con 0x7f206c0cbb30 2024-04-02T07:48:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.448+0000 7f2074600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206c0cddc0 con 0x7f206c0b2070 2024-04-02T07:48:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.448+0000 7f2074600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206c0ce0a0 con 0x7f206c0cd020 2024-04-02T07:48:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.448+0000 7f2071c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f206c0cbb30 0x7f206c0cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32934/0 (socket says 172.21.15.89:32934) 2024-04-02T07:48:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.448+0000 7f2071c00700 1 -- 172.21.15.89:0/380456348 learned_addr learned my addr 172.21.15.89:0/380456348 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.448+0000 7f2071200700 1 --2- 172.21.15.89:0/380456348 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c0cd020 0x7f206c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.449+0000 7f206be00700 1 -- 172.21.15.89:0/380456348 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717614234 0 0) 0x7f206c0cdae0 con 0x7f206c0cbb30 2024-04-02T07:48:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.449+0000 7f206be00700 1 -- 172.21.15.89:0/380456348 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20540035b0 con 0x7f206c0cbb30 2024-04-02T07:48:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.449+0000 7f206be00700 1 -- 172.21.15.89:0/380456348 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439242863 0 0) 0x7f206c0cddc0 con 0x7f206c0b2070 2024-04-02T07:48:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.449+0000 7f206be00700 1 -- 172.21.15.89:0/380456348 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f206c0cdae0 con 0x7f206c0b2070 2024-04-02T07:48:39.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.449+0000 7f2071200700 1 -- 172.21.15.89:0/380456348 >> v1:172.21.15.113:6789/0 conn(0x7f206c0b2070 legacy=0x7f206c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:39.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.449+0000 7f2071200700 1 -- 172.21.15.89:0/380456348 >> v1:172.21.15.89:6789/0 conn(0x7f206c0cbb30 legacy=0x7f206c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:39.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.449+0000 7f2071200700 1 -- 172.21.15.89:0/380456348 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206c06dc20 con 0x7f206c0cd020 2024-04-02T07:48:39.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.449+0000 7f2071200700 1 --2- 172.21.15.89:0/380456348 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c0cd020 0x7f206c0cd480 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2058009870 tx=0x7f2058009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6b70261d7214ce1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:39.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.449+0000 7f206be00700 1 -- 172.21.15.89:0/380456348 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2058011830 con 0x7f206c0cd020 2024-04-02T07:48:39.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.449+0000 7f206be00700 1 -- 172.21.15.89:0/380456348 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2058011990 con 0x7f206c0cd020 2024-04-02T07:48:39.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.449+0000 7f206be00700 1 -- 172.21.15.89:0/380456348 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2058011d20 con 0x7f206c0cd020 2024-04-02T07:48:39.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.450+0000 7f2074600700 1 -- 172.21.15.89:0/380456348 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c0cd020 msgr2=0x7f206c0cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:39.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.450+0000 7f2074600700 1 --2- 172.21.15.89:0/380456348 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c0cd020 0x7f206c0cd480 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2058009870 tx=0x7f2058009b80 comp rx=0 tx=0).stop 2024-04-02T07:48:39.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.450+0000 7f2074600700 1 -- 172.21.15.89:0/380456348 shutdown_connections 2024-04-02T07:48:39.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.450+0000 7f2074600700 1 --2- 172.21.15.89:0/380456348 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c0cd020 0x7f206c0cd480 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:39.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.450+0000 7f2074600700 1 -- 172.21.15.89:0/380456348 >> 172.21.15.89:0/380456348 conn(0x7f206c0b0d70 msgr2=0x7f206c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:39.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.450+0000 7f2074600700 1 -- 172.21.15.89:0/380456348 shutdown_connections 2024-04-02T07:48:39.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.450+0000 7f2074600700 1 -- 172.21.15.89:0/380456348 wait complete. 2024-04-02T07:48:39.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.452+0000 7f2074600700 1 Processor -- start 2024-04-02T07:48:39.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.452+0000 7f2074600700 1 -- start start 2024-04-02T07:48:39.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2074600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c0b2070 0x7f206c14a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2074600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f206c0cbb30 0x7f206c0c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2074600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c0cd020 0x7f206c0717b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2074600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c0caab0 con 0x7f206c0cd020 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2074600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c0caed0 con 0x7f206c0cbb30 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2074600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206c0c9e30 con 0x7f206c0b2070 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2072600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c0cd020 0x7f206c0717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2072600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c0cd020 0x7f206c0717b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51572/0 (socket says 172.21.15.89:51572) 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2072600700 1 -- 172.21.15.89:0/166657313 learned_addr learned my addr 172.21.15.89:0/166657313 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2071c00700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c0b2070 0x7f206c14a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2071200700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f206c0cbb30 0x7f206c0c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2072600700 1 -- 172.21.15.89:0/166657313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c0b2070 msgr2=0x7f206c14a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2072600700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c0b2070 0x7f206c14a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2072600700 1 -- 172.21.15.89:0/166657313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f206c0cbb30 msgr2=0x7f206c0c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2072600700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f206c0cbb30 0x7f206c0c93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.453+0000 7f2072600700 1 -- 172.21.15.89:0/166657313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20580094a0 con 0x7f206c0cd020 2024-04-02T07:48:39.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.454+0000 7f2071200700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f206c0cbb30 0x7f206c0c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:39.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.454+0000 7f2071c00700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c0b2070 0x7f206c14a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:39.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.454+0000 7f2072600700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c0cd020 0x7f206c0717b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f206400cbc0 tx=0x7f206400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:39.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.454+0000 7f2069600700 1 -- 172.21.15.89:0/166657313 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2064003c00 con 0x7f206c0cd020 2024-04-02T07:48:39.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.454+0000 7f2069600700 1 -- 172.21.15.89:0/166657313 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2064003d60 con 0x7f206c0cd020 2024-04-02T07:48:39.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.454+0000 7f2074600700 1 -- 172.21.15.89:0/166657313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206c0ca110 con 0x7f206c0cd020 2024-04-02T07:48:39.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.454+0000 7f2069600700 1 -- 172.21.15.89:0/166657313 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206400abb0 con 0x7f206c0cd020 2024-04-02T07:48:39.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.454+0000 7f2074600700 1 -- 172.21.15.89:0/166657313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f206c0ca6d0 con 0x7f206c0cd020 2024-04-02T07:48:39.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.455+0000 7f2074600700 1 -- 172.21.15.89:0/166657313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f206c067720 con 0x7f206c0cd020 2024-04-02T07:48:39.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.456+0000 7f2069600700 1 -- 172.21.15.89:0/166657313 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f206400ad10 con 0x7f206c0cd020 2024-04-02T07:48:39.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.457+0000 7f2069600700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f204c06ae90 0x7f204c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:39.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.457+0000 7f2069600700 1 -- 172.21.15.89:0/166657313 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6619+0+0 (secure 0 0 0) 0x7f2064089b70 con 0x7f206c0cd020 2024-04-02T07:48:39.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.458+0000 7f2071c00700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f204c06ae90 0x7f204c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:39.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.458+0000 7f2071c00700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f204c06ae90 0x7f204c06d350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f20600011e0 tx=0x7f2060009040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:39.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.462+0000 7f2069600700 1 -- 172.21.15.89:0/166657313 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2064054730 con 0x7f206c0cd020 2024-04-02T07:48:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.688+0000 7f2074600700 1 -- 172.21.15.89:0/166657313 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f206c15bdd0 con 0x7f204c06ae90 2024-04-02T07:48:39.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.781+0000 7f2069600700 1 -- 172.21.15.89:0/166657313 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f20600013b0 con 0x7f204c06ae90 2024-04-02T07:48:39.783 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path /snap_test_dir1/dir6 2024-04-02T07:48:39.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.783+0000 7f2074600700 1 -- 172.21.15.89:0/166657313 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f204c06ae90 msgr2=0x7f204c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.783+0000 7f2074600700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f204c06ae90 0x7f204c06d350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f20600011e0 tx=0x7f2060009040 comp rx=0 tx=0).stop 2024-04-02T07:48:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.783+0000 7f2074600700 1 -- 172.21.15.89:0/166657313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c0cd020 msgr2=0x7f206c0717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.783+0000 7f2074600700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c0cd020 0x7f206c0717b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f206400cbc0 tx=0x7f206400cf80 comp rx=0 tx=0).stop 2024-04-02T07:48:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.783+0000 7f2074600700 1 -- 172.21.15.89:0/166657313 shutdown_connections 2024-04-02T07:48:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.783+0000 7f2074600700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f204c06ae90 0x7f204c06d350 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.783+0000 7f2074600700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c0cd020 0x7f206c0717b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.783+0000 7f2074600700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c0b2070 0x7f206c14a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.783+0000 7f2074600700 1 --2- 172.21.15.89:0/166657313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f206c0cbb30 0x7f206c0c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.783+0000 7f2074600700 1 -- 172.21.15.89:0/166657313 >> 172.21.15.89:0/166657313 conn(0x7f206c0b0d70 msgr2=0x7f206c15e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.783+0000 7f2074600700 1 -- 172.21.15.89:0/166657313 shutdown_connections 2024-04-02T07:48:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:39.783+0000 7f2074600700 1 -- 172.21.15.89:0/166657313 wait complete. 2024-04-02T07:48:39.803 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2024-04-02T07:48:39.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:39.803 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-04-02T07:48:39.849 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:39.849 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:39.849 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:39.849 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:39.849 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:39.849 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:39.849 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:39.849 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:39.850 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:39.850 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:39.850 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:39.851 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:39.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:39.851 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_38_00_UTC) 2024-04-02T07:48:39.904 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:39.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:39.905 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_39_00_UTC) 2024-04-02T07:48:39.956 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:39.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:39.956 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_40_00_UTC) 2024-04-02T07:48:40.009 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:40.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:40.009 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_41_00_UTC) 2024-04-02T07:48:40.061 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:40.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:40.061 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_42_00_UTC) 2024-04-02T07:48:40.137 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:40.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:40.138 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_43_00_UTC) 2024-04-02T07:48:40.191 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:40.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:40.191 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_44_00_UTC) 2024-04-02T07:48:40.280 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:40.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:40.281 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_45_00_UTC) 2024-04-02T07:48:40.342 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:40.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:40.343 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_46_00_UTC) 2024-04-02T07:48:40.394 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:40.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:40.395 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_47_00_UTC) 2024-04-02T07:48:40.446 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:40.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:40.446 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-02-07_48_00_UTC) 2024-04-02T07:48:40.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:40.497 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2024-04-02T07:48:40.544 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2024-04-02T07:48:40.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.736+0000 7fe609000700 1 Processor -- start 2024-04-02T07:48:40.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.736+0000 7fe609000700 1 -- start start 2024-04-02T07:48:40.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.737+0000 7fe609000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6040d3f20 0x7fe6040cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:40.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.737+0000 7fe609000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe604161230 con 0x7fe6040b3f80 2024-04-02T07:48:40.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.737+0000 7fe609000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6040d2a80 con 0x7fe604160e50 2024-04-02T07:48:40.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.737+0000 7fe609000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6040d43b0 con 0x7fe6040d3f20 2024-04-02T07:48:40.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.737+0000 7fe601400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6040d3f20 0x7fe6040cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:40.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.737+0000 7fe602800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe6040b3f80 0x7fe6040ce430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32948/0 (socket says 172.21.15.89:32948) 2024-04-02T07:48:40.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.737+0000 7fe602800700 1 -- 172.21.15.89:0/3567628193 learned_addr learned my addr 172.21.15.89:0/3567628193 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:40.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.738+0000 7fe601400700 1 -- 172.21.15.89:0/3567628193 >> v1:172.21.15.113:6789/0 conn(0x7fe604160e50 legacy=0x7fe6040b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:40.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.738+0000 7fe601400700 1 -- 172.21.15.89:0/3567628193 >> v1:172.21.15.89:6789/0 conn(0x7fe6040b3f80 legacy=0x7fe6040ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:40.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.738+0000 7fe601400700 1 -- 172.21.15.89:0/3567628193 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6040d21e0 con 0x7fe6040d3f20 2024-04-02T07:48:40.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.738+0000 7fe600a00700 1 -- 172.21.15.89:0/3567628193 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423602804 0 0) 0x7fe6040d2a80 con 0x7fe604160e50 2024-04-02T07:48:40.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.738+0000 7fe601400700 1 --2- 172.21.15.89:0/3567628193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6040d3f20 0x7fe6040cdef0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009840 tx=0x7fe5f0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70fd3ab63e401866 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:40.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.738+0000 7fe600a00700 1 -- 172.21.15.89:0/3567628193 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0004230 con 0x7fe6040d3f20 2024-04-02T07:48:40.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.738+0000 7fe600a00700 1 -- 172.21.15.89:0/3567628193 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f0004390 con 0x7fe6040d3f20 2024-04-02T07:48:40.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.738+0000 7fe600a00700 1 -- 172.21.15.89:0/3567628193 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00036a0 con 0x7fe6040d3f20 2024-04-02T07:48:40.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.739+0000 7fe609000700 1 -- 172.21.15.89:0/3567628193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6040d3f20 msgr2=0x7fe6040cdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:40.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.739+0000 7fe609000700 1 --2- 172.21.15.89:0/3567628193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6040d3f20 0x7fe6040cdef0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009840 tx=0x7fe5f0009b50 comp rx=0 tx=0).stop 2024-04-02T07:48:40.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.739+0000 7fe609000700 1 -- 172.21.15.89:0/3567628193 shutdown_connections 2024-04-02T07:48:40.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.739+0000 7fe609000700 1 --2- 172.21.15.89:0/3567628193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6040d3f20 0x7fe6040cdef0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:40.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.739+0000 7fe609000700 1 -- 172.21.15.89:0/3567628193 >> 172.21.15.89:0/3567628193 conn(0x7fe60408aff0 msgr2=0x7fe6040d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:40.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.739+0000 7fe609000700 1 -- 172.21.15.89:0/3567628193 shutdown_connections 2024-04-02T07:48:40.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.739+0000 7fe609000700 1 -- 172.21.15.89:0/3567628193 wait complete. 2024-04-02T07:48:40.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.741+0000 7fe609000700 1 Processor -- start 2024-04-02T07:48:40.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.741+0000 7fe609000700 1 -- start start 2024-04-02T07:48:40.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe609000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6040b3f80 0x7fe60415e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:40.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe609000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe604160e50 0x7fe6040cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe609000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6040cbff0 0x7fe60415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe609000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6040b6540 con 0x7fe6040cbff0 2024-04-02T07:48:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe609000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6040b7de0 con 0x7fe604160e50 2024-04-02T07:48:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe609000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6040b80c0 con 0x7fe6040b3f80 2024-04-02T07:48:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe602800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6040cbff0 0x7fe60415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe602800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6040cbff0 0x7fe60415cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51588/0 (socket says 172.21.15.89:51588) 2024-04-02T07:48:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe601400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe604160e50 0x7fe6040cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe602800700 1 -- 172.21.15.89:0/2526697747 learned_addr learned my addr 172.21.15.89:0/2526697747 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe601e00700 1 --2- 172.21.15.89:0/2526697747 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6040b3f80 0x7fe60415e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe602800700 1 -- 172.21.15.89:0/2526697747 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6040b3f80 msgr2=0x7fe60415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe602800700 1 --2- 172.21.15.89:0/2526697747 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6040b3f80 0x7fe60415e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe602800700 1 -- 172.21.15.89:0/2526697747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe604160e50 msgr2=0x7fe6040cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe602800700 1 --2- 172.21.15.89:0/2526697747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe604160e50 0x7fe6040cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:40.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.742+0000 7fe602800700 1 -- 172.21.15.89:0/2526697747 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f00094a0 con 0x7fe6040cbff0 2024-04-02T07:48:40.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.743+0000 7fe602800700 1 --2- 172.21.15.89:0/2526697747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6040cbff0 0x7fe60415cea0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc00cca0 tx=0x7fe5fc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:40.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.743+0000 7fe5fa000700 1 -- 172.21.15.89:0/2526697747 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc00ab30 con 0x7fe6040cbff0 2024-04-02T07:48:40.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.743+0000 7fe5fa000700 1 -- 172.21.15.89:0/2526697747 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5fc00ac90 con 0x7fe6040cbff0 2024-04-02T07:48:40.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.743+0000 7fe5fa000700 1 -- 172.21.15.89:0/2526697747 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0070d0 con 0x7fe6040cbff0 2024-04-02T07:48:40.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.743+0000 7fe609000700 1 -- 172.21.15.89:0/2526697747 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6040b83a0 con 0x7fe6040cbff0 2024-04-02T07:48:40.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.743+0000 7fe609000700 1 -- 172.21.15.89:0/2526697747 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60415db50 con 0x7fe6040cbff0 2024-04-02T07:48:40.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.744+0000 7fe609000700 1 -- 172.21.15.89:0/2526697747 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe604067720 con 0x7fe6040cbff0 2024-04-02T07:48:40.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.745+0000 7fe5fa000700 1 -- 172.21.15.89:0/2526697747 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fe5fc007250 con 0x7fe6040cbff0 2024-04-02T07:48:40.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.746+0000 7fe5fa000700 1 --2- 172.21.15.89:0/2526697747 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe5e406ae90 0x7fe5e406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:40.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.746+0000 7fe5fa000700 1 -- 172.21.15.89:0/2526697747 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6619+0+0 (secure 0 0 0) 0x7fe5fc089490 con 0x7fe6040cbff0 2024-04-02T07:48:40.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.746+0000 7fe601e00700 1 --2- 172.21.15.89:0/2526697747 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe5e406ae90 0x7fe5e406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:40.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.747+0000 7fe601e00700 1 --2- 172.21.15.89:0/2526697747 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe5e406ae90 0x7fe5e406d350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe60408b820 tx=0x7fe5f4009040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:40.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.750+0000 7fe5fa000700 1 -- 172.21.15.89:0/2526697747 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe5fc054050 con 0x7fe6040cbff0 2024-04-02T07:48:40.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:40.977+0000 7fe609000700 1 -- 172.21.15.89:0/2526697747 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6040b8230 con 0x7fe5e406ae90 2024-04-02T07:48:41.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:41.081+0000 7fe5fa000700 1 -- 172.21.15.89:0/2526697747 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fe6040b8230 con 0x7fe5e406ae90 2024-04-02T07:48:41.083 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path /snap_test_dir1/dir7 2024-04-02T07:48:41.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:41.083+0000 7fe609000700 1 -- 172.21.15.89:0/2526697747 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe5e406ae90 msgr2=0x7fe5e406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:41.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:41.083+0000 7fe609000700 1 --2- 172.21.15.89:0/2526697747 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe5e406ae90 0x7fe5e406d350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe60408b820 tx=0x7fe5f4009040 comp rx=0 tx=0).stop 2024-04-02T07:48:41.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:41.083+0000 7fe609000700 1 -- 172.21.15.89:0/2526697747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6040cbff0 msgr2=0x7fe60415cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:41.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:41.083+0000 7fe609000700 1 --2- 172.21.15.89:0/2526697747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6040cbff0 0x7fe60415cea0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc00cca0 tx=0x7fe5fc00a060 comp rx=0 tx=0).stop 2024-04-02T07:48:41.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:41.084+0000 7fe609000700 1 -- 172.21.15.89:0/2526697747 shutdown_connections 2024-04-02T07:48:41.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:41.084+0000 7fe609000700 1 --2- 172.21.15.89:0/2526697747 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe5e406ae90 0x7fe5e406d350 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:41.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:41.084+0000 7fe609000700 1 --2- 172.21.15.89:0/2526697747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6040cbff0 0x7fe60415cea0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:41.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:41.084+0000 7fe609000700 1 --2- 172.21.15.89:0/2526697747 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6040b3f80 0x7fe60415e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:41.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:41.084+0000 7fe609000700 1 --2- 172.21.15.89:0/2526697747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe604160e50 0x7fe6040cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:41.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:41.084+0000 7fe609000700 1 -- 172.21.15.89:0/2526697747 >> 172.21.15.89:0/2526697747 conn(0x7fe60408aff0 msgr2=0x7fe6040d3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:41.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:41.084+0000 7fe609000700 1 -- 172.21.15.89:0/2526697747 shutdown_connections 2024-04-02T07:48:41.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:41.084+0000 7fe609000700 1 -- 172.21.15.89:0/2526697747 wait complete. 2024-04-02T07:48:41.105 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2024-04-02T07:48:41.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:41.105 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-04-02T07:48:41.150 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:41.150 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:41.151 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:41.151 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:41.151 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:41.151 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:41.151 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:41.151 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:41.151 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:41.151 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:41.151 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:41.152 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:41.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:41.152 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_38_00_UTC) 2024-04-02T07:48:41.246 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:41.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:41.246 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_39_00_UTC) 2024-04-02T07:48:41.298 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:41.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:41.298 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_40_00_UTC) 2024-04-02T07:48:41.349 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:41.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:41.349 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_41_00_UTC) 2024-04-02T07:48:41.404 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:41.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:41.405 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_42_00_UTC) 2024-04-02T07:48:41.462 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:41.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:41.463 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_43_00_UTC) 2024-04-02T07:48:41.514 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:41.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:41.514 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_44_00_UTC) 2024-04-02T07:48:41.574 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:41.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:41.574 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_45_00_UTC) 2024-04-02T07:48:41.625 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:41.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:41.625 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_46_00_UTC) 2024-04-02T07:48:41.677 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:41.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:41.677 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_47_00_UTC) 2024-04-02T07:48:41.733 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:41.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:41.733 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-02-07_48_00_UTC) 2024-04-02T07:48:41.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:41.785 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2024-04-02T07:48:41.838 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2024-04-02T07:48:42.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.032+0000 7f15fe600700 1 Processor -- start 2024-04-02T07:48:42.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.032+0000 7f15fe600700 1 -- start start 2024-04-02T07:48:42.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.033+0000 7f15fe600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15f80d21f0 0x7f15f80ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:42.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.033+0000 7f15fe600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f15f80cec40 0x7f15f8161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:42.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.033+0000 7f15fe600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f80ce270 con 0x7f15f81611b0 2024-04-02T07:48:42.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.033+0000 7f15fe600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15f815b600 con 0x7f15f80cec40 2024-04-02T07:48:42.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.033+0000 7f15fe600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f80cf0b0 con 0x7f15f80d21f0 2024-04-02T07:48:42.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.033+0000 7f15f7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f15f80cec40 0x7f15f8161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:42.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.033+0000 7f15f7400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f15f81611b0 0x7f15f815ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32962/0 (socket says 172.21.15.89:32962) 2024-04-02T07:48:42.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.033+0000 7f15f7400700 1 -- 172.21.15.89:0/1558162879 learned_addr learned my addr 172.21.15.89:0/1558162879 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.033+0000 7f15f6a00700 1 --2- 172.21.15.89:0/1558162879 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15f80d21f0 0x7f15f80ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.034+0000 7f15f6000700 1 -- 172.21.15.89:0/1558162879 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939726577 0 0) 0x7f15f80ce270 con 0x7f15f81611b0 2024-04-02T07:48:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.034+0000 7f15f7e00700 1 -- 172.21.15.89:0/1558162879 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15f80d21f0 msgr2=0x7f15f80ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.034+0000 7f15f7e00700 1 --2- 172.21.15.89:0/1558162879 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15f80d21f0 0x7f15f80ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.034+0000 7f15f7e00700 1 -- 172.21.15.89:0/1558162879 >> v1:172.21.15.89:6789/0 conn(0x7f15f81611b0 legacy=0x7f15f815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.034+0000 7f15f7e00700 1 -- 172.21.15.89:0/1558162879 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f80d2760 con 0x7f15f80cec40 2024-04-02T07:48:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.034+0000 7f15f7e00700 1 --2- 172.21.15.89:0/1558162879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15f80cec40 0x7f15f8161590 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009d90 tx=0x7f15f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693eff7ba8f3c066 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.034+0000 7f15f6000700 1 -- 172.21.15.89:0/1558162879 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f000e070 con 0x7f15f80cec40 2024-04-02T07:48:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.034+0000 7f15f6000700 1 -- 172.21.15.89:0/1558162879 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f000bb20 con 0x7f15f80cec40 2024-04-02T07:48:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.034+0000 7f15f6000700 1 -- 172.21.15.89:0/1558162879 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0012430 con 0x7f15f80cec40 2024-04-02T07:48:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.035+0000 7f15fe600700 1 -- 172.21.15.89:0/1558162879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15f80cec40 msgr2=0x7f15f8161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.035+0000 7f15fe600700 1 --2- 172.21.15.89:0/1558162879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15f80cec40 0x7f15f8161590 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009d90 tx=0x7f15f000b330 comp rx=0 tx=0).stop 2024-04-02T07:48:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.035+0000 7f15fe600700 1 -- 172.21.15.89:0/1558162879 shutdown_connections 2024-04-02T07:48:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.035+0000 7f15fe600700 1 --2- 172.21.15.89:0/1558162879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15f80cec40 0x7f15f8161590 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.035+0000 7f15fe600700 1 --2- 172.21.15.89:0/1558162879 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15f80d21f0 0x7f15f80ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.035+0000 7f15fe600700 1 -- 172.21.15.89:0/1558162879 >> 172.21.15.89:0/1558162879 conn(0x7f15f808aff0 msgr2=0x7f15f80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.035+0000 7f15fe600700 1 -- 172.21.15.89:0/1558162879 shutdown_connections 2024-04-02T07:48:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.035+0000 7f15fe600700 1 -- 172.21.15.89:0/1558162879 wait complete. 2024-04-02T07:48:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.037+0000 7f15fe600700 1 Processor -- start 2024-04-02T07:48:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.037+0000 7f15fe600700 1 -- start start 2024-04-02T07:48:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15fe600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15f80cec40 0x7f15f80cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15fe600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15f80d21f0 0x7f15f80ca860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15fe600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15f81611b0 0x7f15f80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15fe600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f80b8970 con 0x7f15f80d21f0 2024-04-02T07:48:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15fe600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f8148a80 con 0x7f15f80cec40 2024-04-02T07:48:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15fe600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f80cc350 con 0x7f15f81611b0 2024-04-02T07:48:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15f6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15f80d21f0 0x7f15f80ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15f6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15f80d21f0 0x7f15f80ca860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51612/0 (socket says 172.21.15.89:51612) 2024-04-02T07:48:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15f6a00700 1 -- 172.21.15.89:0/3667740860 learned_addr learned my addr 172.21.15.89:0/3667740860 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15f7400700 1 --2- 172.21.15.89:0/3667740860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15f80cec40 0x7f15f80cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15f7e00700 1 --2- 172.21.15.89:0/3667740860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15f81611b0 0x7f15f80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15f6a00700 1 -- 172.21.15.89:0/3667740860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15f81611b0 msgr2=0x7f15f80cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15f6a00700 1 --2- 172.21.15.89:0/3667740860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15f81611b0 0x7f15f80cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15f6a00700 1 -- 172.21.15.89:0/3667740860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15f80cec40 msgr2=0x7f15f80cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15f6a00700 1 --2- 172.21.15.89:0/3667740860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15f80cec40 0x7f15f80cbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.038+0000 7f15f6a00700 1 -- 172.21.15.89:0/3667740860 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f0009a70 con 0x7f15f80d21f0 2024-04-02T07:48:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.039+0000 7f15f6a00700 1 --2- 172.21.15.89:0/3667740860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15f80d21f0 0x7f15f80ca860 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f15e400e9b0 tx=0x7f15e400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.039+0000 7f15ef400700 1 -- 172.21.15.89:0/3667740860 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e4009710 con 0x7f15f80d21f0 2024-04-02T07:48:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.039+0000 7f15ef400700 1 -- 172.21.15.89:0/3667740860 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e400ee50 con 0x7f15f80d21f0 2024-04-02T07:48:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.039+0000 7f15ef400700 1 -- 172.21.15.89:0/3667740860 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e400ca50 con 0x7f15f80d21f0 2024-04-02T07:48:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.039+0000 7f15fe600700 1 -- 172.21.15.89:0/3667740860 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f80b8b60 con 0x7f15f80d21f0 2024-04-02T07:48:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.039+0000 7f15fe600700 1 -- 172.21.15.89:0/3667740860 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f8148c90 con 0x7f15f80d21f0 2024-04-02T07:48:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.040+0000 7f15fe600700 1 -- 172.21.15.89:0/3667740860 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f8067720 con 0x7f15f80d21f0 2024-04-02T07:48:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.041+0000 7f15ef400700 1 -- 172.21.15.89:0/3667740860 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f15e400cbb0 con 0x7f15f80d21f0 2024-04-02T07:48:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.042+0000 7f15ef400700 1 --2- 172.21.15.89:0/3667740860 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15d806ae90 0x7f15d806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.042+0000 7f15ef400700 1 -- 172.21.15.89:0/3667740860 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f15e4089970 con 0x7f15f80d21f0 2024-04-02T07:48:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.042+0000 7f15f7400700 1 --2- 172.21.15.89:0/3667740860 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15d806ae90 0x7f15d806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.043+0000 7f15f7400700 1 --2- 172.21.15.89:0/3667740860 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15d806ae90 0x7f15d806d350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f15e8000f60 tx=0x7f15e8007e70 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:42.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.046+0000 7f15ef400700 1 -- 172.21.15.89:0/3667740860 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15e4054580 con 0x7f15f80d21f0 2024-04-02T07:48:42.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.291+0000 7f15fe600700 1 -- 172.21.15.89:0/3667740860 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f15f80cdf00 con 0x7f15d806ae90 2024-04-02T07:48:42.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.414+0000 7f15ef400700 1 -- 172.21.15.89:0/3667740860 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f15e8004ab0 con 0x7f15d806ae90 2024-04-02T07:48:42.416 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path /snap_test_dir1/dir8 2024-04-02T07:48:42.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.416+0000 7f15fe600700 1 -- 172.21.15.89:0/3667740860 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15d806ae90 msgr2=0x7f15d806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:42.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.416+0000 7f15fe600700 1 --2- 172.21.15.89:0/3667740860 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15d806ae90 0x7f15d806d350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f15e8000f60 tx=0x7f15e8007e70 comp rx=0 tx=0).stop 2024-04-02T07:48:42.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.416+0000 7f15fe600700 1 -- 172.21.15.89:0/3667740860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15f80d21f0 msgr2=0x7f15f80ca860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:42.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.416+0000 7f15fe600700 1 --2- 172.21.15.89:0/3667740860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15f80d21f0 0x7f15f80ca860 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f15e400e9b0 tx=0x7f15e400ecc0 comp rx=0 tx=0).stop 2024-04-02T07:48:42.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.416+0000 7f15fe600700 1 -- 172.21.15.89:0/3667740860 shutdown_connections 2024-04-02T07:48:42.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.416+0000 7f15fe600700 1 --2- 172.21.15.89:0/3667740860 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f15d806ae90 0x7f15d806d350 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:42.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.416+0000 7f15fe600700 1 --2- 172.21.15.89:0/3667740860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15f80cec40 0x7f15f80cbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:42.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.416+0000 7f15fe600700 1 --2- 172.21.15.89:0/3667740860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15f81611b0 0x7f15f80cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:42.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.416+0000 7f15fe600700 1 --2- 172.21.15.89:0/3667740860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15f80d21f0 0x7f15f80ca860 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:42.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.416+0000 7f15fe600700 1 -- 172.21.15.89:0/3667740860 >> 172.21.15.89:0/3667740860 conn(0x7f15f808aff0 msgr2=0x7f15f815a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:42.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.416+0000 7f15fe600700 1 -- 172.21.15.89:0/3667740860 shutdown_connections 2024-04-02T07:48:42.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:42.417+0000 7f15fe600700 1 -- 172.21.15.89:0/3667740860 wait complete. 2024-04-02T07:48:42.436 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2024-04-02T07:48:42.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:42.436 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-04-02T07:48:42.482 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:42.482 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:42.482 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:42.482 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:42.482 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:42.482 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:42.482 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:42.482 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:42.482 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:42.483 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:42.483 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:42.483 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:42.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:42.484 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_38_00_UTC) 2024-04-02T07:48:42.537 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:42.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:42.537 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_39_00_UTC) 2024-04-02T07:48:42.590 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:42.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:42.590 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_40_00_UTC) 2024-04-02T07:48:42.645 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:42.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:42.646 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_41_00_UTC) 2024-04-02T07:48:42.697 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:42.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:42.746 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_42_00_UTC) 2024-04-02T07:48:42.797 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:42.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:42.797 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_43_00_UTC) 2024-04-02T07:48:42.849 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:42.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:42.849 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_44_00_UTC) 2024-04-02T07:48:42.901 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:42.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:42.901 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_45_00_UTC) 2024-04-02T07:48:42.998 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:42.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:42.999 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_46_00_UTC) 2024-04-02T07:48:43.055 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:43.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:43.056 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_47_00_UTC) 2024-04-02T07:48:43.106 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:43.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:43.106 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-02-07_48_00_UTC) 2024-04-02T07:48:43.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:43.158 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2024-04-02T07:48:43.208 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2024-04-02T07:48:43.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.399+0000 7fbf4e800700 1 Processor -- start 2024-04-02T07:48:43.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.400+0000 7fbf4e800700 1 -- start start 2024-04-02T07:48:43.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.401+0000 7fbf4e800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbf480cd5c0 0x7fbf4815e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:43.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.401+0000 7fbf4e800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbf4815cfe0 0x7fbf4815d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:43.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.401+0000 7fbf4e800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf4815eda0 con 0x7fbf4806f310 2024-04-02T07:48:43.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.401+0000 7fbf4e800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf48099dd0 con 0x7fbf480cd5c0 2024-04-02T07:48:43.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.401+0000 7fbf4e800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4809a0b0 con 0x7fbf4815cfe0 2024-04-02T07:48:43.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.401+0000 7fbf46a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbf4806f310 0x7fbf4806f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32970/0 (socket says 172.21.15.89:32970) 2024-04-02T07:48:43.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.401+0000 7fbf47400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbf480cd5c0 0x7fbf4815e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:43.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.401+0000 7fbf46a00700 1 -- 172.21.15.89:0/328304360 learned_addr learned my addr 172.21.15.89:0/328304360 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:43.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.401+0000 7fbf47e00700 1 --2- 172.21.15.89:0/328304360 >> v2:172.21.15.113:3300/0 conn(0x7fbf4815cfe0 0x7fbf4815d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:43.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.401+0000 7fbf46000700 1 -- 172.21.15.89:0/328304360 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101665612 0 0) 0x7fbf34001110 con 0x7fbf4806f310 2024-04-02T07:48:43.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.401+0000 7fbf46000700 1 -- 172.21.15.89:0/328304360 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf2c0035b0 con 0x7fbf4806f310 2024-04-02T07:48:43.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.402+0000 7fbf47400700 1 -- 172.21.15.89:0/328304360 >> v2:172.21.15.113:3300/0 conn(0x7fbf4815cfe0 msgr2=0x7fbf4815d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:43.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.402+0000 7fbf47400700 1 --2- 172.21.15.89:0/328304360 >> v2:172.21.15.113:3300/0 conn(0x7fbf4815cfe0 0x7fbf4815d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:43.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.402+0000 7fbf47400700 1 -- 172.21.15.89:0/328304360 >> v1:172.21.15.89:6789/0 conn(0x7fbf4806f310 legacy=0x7fbf4806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:43.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.402+0000 7fbf47400700 1 -- 172.21.15.89:0/328304360 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4809a390 con 0x7fbf480cd5c0 2024-04-02T07:48:43.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.402+0000 7fbf47400700 1 --2- 172.21.15.89:0/328304360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf480cd5c0 0x7fbf4815e860 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009840 tx=0x7fbf38009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d4cf2f498b9984 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:43.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.402+0000 7fbf46000700 1 -- 172.21.15.89:0/328304360 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38011830 con 0x7fbf480cd5c0 2024-04-02T07:48:43.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.402+0000 7fbf46000700 1 -- 172.21.15.89:0/328304360 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf38011990 con 0x7fbf480cd5c0 2024-04-02T07:48:43.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.402+0000 7fbf46000700 1 -- 172.21.15.89:0/328304360 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38011d20 con 0x7fbf480cd5c0 2024-04-02T07:48:43.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.402+0000 7fbf4e800700 1 -- 172.21.15.89:0/328304360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf480cd5c0 msgr2=0x7fbf4815e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:43.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.402+0000 7fbf4e800700 1 --2- 172.21.15.89:0/328304360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf480cd5c0 0x7fbf4815e860 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009840 tx=0x7fbf38009b50 comp rx=0 tx=0).stop 2024-04-02T07:48:43.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.403+0000 7fbf4e800700 1 -- 172.21.15.89:0/328304360 shutdown_connections 2024-04-02T07:48:43.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.403+0000 7fbf4e800700 1 --2- 172.21.15.89:0/328304360 >> v2:172.21.15.113:3300/0 conn(0x7fbf4815cfe0 0x7fbf4815d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:43.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.403+0000 7fbf4e800700 1 --2- 172.21.15.89:0/328304360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf480cd5c0 0x7fbf4815e860 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:43.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.403+0000 7fbf4e800700 1 -- 172.21.15.89:0/328304360 >> 172.21.15.89:0/328304360 conn(0x7fbf480d1370 msgr2=0x7fbf480d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:43.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.403+0000 7fbf4e800700 1 -- 172.21.15.89:0/328304360 shutdown_connections 2024-04-02T07:48:43.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.403+0000 7fbf4e800700 1 -- 172.21.15.89:0/328304360 wait complete. 2024-04-02T07:48:43.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.404+0000 7fbf4e800700 1 Processor -- start 2024-04-02T07:48:43.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.405+0000 7fbf4e800700 1 -- start start 2024-04-02T07:48:43.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.405+0000 7fbf4e800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf4806f310 0x7fbf481543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:43.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.405+0000 7fbf4e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf480cd5c0 0x7fbf480b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:43.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.406+0000 7fbf4e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf4815cfe0 0x7fbf480cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:43.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.406+0000 7fbf4e800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf480b5190 con 0x7fbf4815cfe0 2024-04-02T07:48:43.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.406+0000 7fbf4e800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf480b55b0 con 0x7fbf4806f310 2024-04-02T07:48:43.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.406+0000 7fbf4e800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf48154930 con 0x7fbf480cd5c0 2024-04-02T07:48:43.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.406+0000 7fbf47e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf4815cfe0 0x7fbf480cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:43.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.406+0000 7fbf47e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf4815cfe0 0x7fbf480cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51632/0 (socket says 172.21.15.89:51632) 2024-04-02T07:48:43.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.406+0000 7fbf47e00700 1 -- 172.21.15.89:0/294977654 learned_addr learned my addr 172.21.15.89:0/294977654 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:43.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.406+0000 7fbf47400700 1 --2- 172.21.15.89:0/294977654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf4806f310 0x7fbf481543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:43.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.406+0000 7fbf46a00700 1 --2- 172.21.15.89:0/294977654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf480cd5c0 0x7fbf480b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:43.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.406+0000 7fbf47e00700 1 -- 172.21.15.89:0/294977654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf480cd5c0 msgr2=0x7fbf480b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:43.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.406+0000 7fbf47e00700 1 --2- 172.21.15.89:0/294977654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf480cd5c0 0x7fbf480b2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:43.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.406+0000 7fbf47e00700 1 -- 172.21.15.89:0/294977654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf4806f310 msgr2=0x7fbf481543f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:43.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.406+0000 7fbf47e00700 1 --2- 172.21.15.89:0/294977654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf4806f310 0x7fbf481543f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:43.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.406+0000 7fbf47e00700 1 -- 172.21.15.89:0/294977654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf380094a0 con 0x7fbf4815cfe0 2024-04-02T07:48:43.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.407+0000 7fbf47e00700 1 --2- 172.21.15.89:0/294977654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf4815cfe0 0x7fbf480cf590 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000efc0 tx=0x7fbf4000c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:43.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.407+0000 7fbf3f400700 1 -- 172.21.15.89:0/294977654 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4000cad0 con 0x7fbf4815cfe0 2024-04-02T07:48:43.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.407+0000 7fbf3f400700 1 -- 172.21.15.89:0/294977654 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4000f040 con 0x7fbf4815cfe0 2024-04-02T07:48:43.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.407+0000 7fbf3f400700 1 -- 172.21.15.89:0/294977654 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4000ce70 con 0x7fbf4815cfe0 2024-04-02T07:48:43.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.407+0000 7fbf4e800700 1 -- 172.21.15.89:0/294977654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf480b4510 con 0x7fbf4815cfe0 2024-04-02T07:48:43.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.407+0000 7fbf4e800700 1 -- 172.21.15.89:0/294977654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf480b4ad0 con 0x7fbf4815cfe0 2024-04-02T07:48:43.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.408+0000 7fbf4e800700 1 -- 172.21.15.89:0/294977654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf48067720 con 0x7fbf4815cfe0 2024-04-02T07:48:43.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.409+0000 7fbf3f400700 1 -- 172.21.15.89:0/294977654 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fbf40014480 con 0x7fbf4815cfe0 2024-04-02T07:48:43.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.410+0000 7fbf3f400700 1 --2- 172.21.15.89:0/294977654 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbf2806ae90 0x7fbf2806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:43.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.410+0000 7fbf3f400700 1 -- 172.21.15.89:0/294977654 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6967+0+0 (secure 0 0 0) 0x7fbf4008a720 con 0x7fbf4815cfe0 2024-04-02T07:48:43.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.410+0000 7fbf47400700 1 --2- 172.21.15.89:0/294977654 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbf2806ae90 0x7fbf2806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:43.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.411+0000 7fbf47400700 1 --2- 172.21.15.89:0/294977654 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbf2806ae90 0x7fbf2806d350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbf380090b0 tx=0x7fbf3801c040 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:43.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.414+0000 7fbf3f400700 1 -- 172.21.15.89:0/294977654 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf40055070 con 0x7fbf4815cfe0 2024-04-02T07:48:43.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.668+0000 7fbf4e800700 1 -- 172.21.15.89:0/294977654 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf48099a60 con 0x7fbf2806ae90 2024-04-02T07:48:43.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.761+0000 7fbf3f400700 1 -- 172.21.15.89:0/294977654 <== mgr.5173 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fbf38011b00 con 0x7fbf2806ae90 2024-04-02T07:48:43.763 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path /snap_test_dir1/dir9 2024-04-02T07:48:43.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.763+0000 7fbf4e800700 1 -- 172.21.15.89:0/294977654 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbf2806ae90 msgr2=0x7fbf2806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:43.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.763+0000 7fbf4e800700 1 --2- 172.21.15.89:0/294977654 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbf2806ae90 0x7fbf2806d350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbf380090b0 tx=0x7fbf3801c040 comp rx=0 tx=0).stop 2024-04-02T07:48:43.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.763+0000 7fbf4e800700 1 -- 172.21.15.89:0/294977654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf4815cfe0 msgr2=0x7fbf480cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:43.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.763+0000 7fbf4e800700 1 --2- 172.21.15.89:0/294977654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf4815cfe0 0x7fbf480cf590 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000efc0 tx=0x7fbf4000c0c0 comp rx=0 tx=0).stop 2024-04-02T07:48:43.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.763+0000 7fbf4e800700 1 -- 172.21.15.89:0/294977654 shutdown_connections 2024-04-02T07:48:43.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.763+0000 7fbf4e800700 1 --2- 172.21.15.89:0/294977654 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbf2806ae90 0x7fbf2806d350 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:43.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.763+0000 7fbf4e800700 1 --2- 172.21.15.89:0/294977654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf4806f310 0x7fbf481543f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:43.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.763+0000 7fbf4e800700 1 --2- 172.21.15.89:0/294977654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf4815cfe0 0x7fbf480cf590 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:43.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.763+0000 7fbf4e800700 1 --2- 172.21.15.89:0/294977654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf480cd5c0 0x7fbf480b2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:43.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.763+0000 7fbf4e800700 1 -- 172.21.15.89:0/294977654 >> 172.21.15.89:0/294977654 conn(0x7fbf480d1370 msgr2=0x7fbf480d1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:43.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.763+0000 7fbf4e800700 1 -- 172.21.15.89:0/294977654 shutdown_connections 2024-04-02T07:48:43.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:43.763+0000 7fbf4e800700 1 -- 172.21.15.89:0/294977654 wait complete. 2024-04-02T07:48:43.783 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2024-04-02T07:48:43.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:43.783 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-04-02T07:48:43.829 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:43.829 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:43.829 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:43.829 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:43.829 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:43.829 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:43.829 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:43.829 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:43.829 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:43.829 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:43.830 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:43.830 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_38_00_UTC 2024-04-02T07:48:43.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:43.831 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_38_00_UTC) 2024-04-02T07:48:43.947 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_39_00_UTC 2024-04-02T07:48:43.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:43.948 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_39_00_UTC) 2024-04-02T07:48:44.001 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_40_00_UTC 2024-04-02T07:48:44.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:44.001 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_40_00_UTC) 2024-04-02T07:48:44.052 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_41_00_UTC 2024-04-02T07:48:44.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:44.053 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_41_00_UTC) 2024-04-02T07:48:44.104 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_42_00_UTC 2024-04-02T07:48:44.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:44.105 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_42_00_UTC) 2024-04-02T07:48:44.156 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_43_00_UTC 2024-04-02T07:48:44.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:44.156 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_43_00_UTC) 2024-04-02T07:48:44.207 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_44_00_UTC 2024-04-02T07:48:44.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:44.208 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_44_00_UTC) 2024-04-02T07:48:44.285 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_45_00_UTC 2024-04-02T07:48:44.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:44.285 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_45_00_UTC) 2024-04-02T07:48:44.337 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_46_00_UTC 2024-04-02T07:48:44.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:44.337 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_46_00_UTC) 2024-04-02T07:48:44.389 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_47_00_UTC 2024-04-02T07:48:44.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:44.389 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_47_00_UTC) 2024-04-02T07:48:44.442 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_48_00_UTC 2024-04-02T07:48:44.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:44.442 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-02-07_48_00_UTC) 2024-04-02T07:48:44.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:44.494 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2024-04-02T07:48:44.541 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T07:48:44.551 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:48:44.589 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:48:44.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.737+0000 7fcb53800700 1 Processor -- start 2024-04-02T07:48:44.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.737+0000 7fcb53800700 1 -- start start 2024-04-02T07:48:44.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.739+0000 7fcb53800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0cbb30 0x7fcb4c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:44.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.739+0000 7fcb53800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb4c0cd020 0x7fcb4c0cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:44.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.739+0000 7fcb53800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4c0cc450 con 0x7fcb4c0b2070 2024-04-02T07:48:44.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.739+0000 7fcb53800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4c0cdae0 con 0x7fcb4c0cd020 2024-04-02T07:48:44.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.739+0000 7fcb53800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c0cddc0 con 0x7fcb4c0cbb30 2024-04-02T07:48:44.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.739+0000 7fcb4be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb4c0cd020 0x7fcb4c0cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:44.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.739+0000 7fcb4be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb4c0cd020 0x7fcb4c0cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51638/0 (socket says 172.21.15.89:51638) 2024-04-02T07:48:44.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.739+0000 7fcb4be00700 1 -- 172.21.15.89:0/4012106160 learned_addr learned my addr 172.21.15.89:0/4012106160 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:44.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.739+0000 7fcb4be00700 1 -- 172.21.15.89:0/4012106160 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0cbb30 msgr2=0x7fcb4c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:44.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.739+0000 7fcb50e00700 1 --2- 172.21.15.89:0/4012106160 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0cbb30 0x7fcb4c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:44.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.739+0000 7fcb4be00700 1 --2- 172.21.15.89:0/4012106160 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0cbb30 0x7fcb4c0cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:44.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.739+0000 7fcb4be00700 1 -- 172.21.15.89:0/4012106160 >> v1:172.21.15.113:6789/0 conn(0x7fcb4c0b2070 legacy=0x7fcb4c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:44.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.739+0000 7fcb4be00700 1 -- 172.21.15.89:0/4012106160 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c0ce0a0 con 0x7fcb4c0cd020 2024-04-02T07:48:44.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.739+0000 7fcb50e00700 1 --2- 172.21.15.89:0/4012106160 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0cbb30 0x7fcb4c0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:44.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.740+0000 7fcb4be00700 1 --2- 172.21.15.89:0/4012106160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0cd020 0x7fcb4c0cd4a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcb380098f0 tx=0x7fcb38009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6159172aba9d1de2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:44.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.740+0000 7fcb4b400700 1 -- 172.21.15.89:0/4012106160 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3800b8a0 con 0x7fcb4c0cd020 2024-04-02T07:48:44.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.740+0000 7fcb4b400700 1 -- 172.21.15.89:0/4012106160 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3800ba00 con 0x7fcb4c0cd020 2024-04-02T07:48:44.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.740+0000 7fcb4b400700 1 -- 172.21.15.89:0/4012106160 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3800bd90 con 0x7fcb4c0cd020 2024-04-02T07:48:44.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.740+0000 7fcb53800700 1 -- 172.21.15.89:0/4012106160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0cd020 msgr2=0x7fcb4c0cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:44.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.740+0000 7fcb53800700 1 --2- 172.21.15.89:0/4012106160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0cd020 0x7fcb4c0cd4a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcb380098f0 tx=0x7fcb38009c00 comp rx=0 tx=0).stop 2024-04-02T07:48:44.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.741+0000 7fcb53800700 1 -- 172.21.15.89:0/4012106160 shutdown_connections 2024-04-02T07:48:44.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.741+0000 7fcb53800700 1 --2- 172.21.15.89:0/4012106160 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0cbb30 0x7fcb4c0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:44.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.741+0000 7fcb53800700 1 --2- 172.21.15.89:0/4012106160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0cd020 0x7fcb4c0cd4a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:44.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.741+0000 7fcb53800700 1 -- 172.21.15.89:0/4012106160 >> 172.21.15.89:0/4012106160 conn(0x7fcb4c0b0d70 msgr2=0x7fcb4c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:44.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.741+0000 7fcb53800700 1 -- 172.21.15.89:0/4012106160 shutdown_connections 2024-04-02T07:48:44.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.741+0000 7fcb53800700 1 -- 172.21.15.89:0/4012106160 wait complete. 2024-04-02T07:48:44.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.742+0000 7fcb53800700 1 Processor -- start 2024-04-02T07:48:44.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.742+0000 7fcb53800700 1 -- start start 2024-04-02T07:48:44.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.743+0000 7fcb53800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0b2070 0x7fcb4c1f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:44.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.743+0000 7fcb53800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0cbb30 0x7fcb4c15de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:44.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.743+0000 7fcb53800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb4c0cd020 0x7fcb4c15e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:44.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.743+0000 7fcb53800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c1561d0 con 0x7fcb4c0b2070 2024-04-02T07:48:44.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.743+0000 7fcb53800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c155650 con 0x7fcb4c0cd020 2024-04-02T07:48:44.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.743+0000 7fcb53800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c155930 con 0x7fcb4c0cbb30 2024-04-02T07:48:44.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.744+0000 7fcb50e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0b2070 0x7fcb4c1f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:44.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.744+0000 7fcb50e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0b2070 0x7fcb4c1f30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51640/0 (socket says 172.21.15.89:51640) 2024-04-02T07:48:44.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.744+0000 7fcb50e00700 1 -- 172.21.15.89:0/2568963581 learned_addr learned my addr 172.21.15.89:0/2568963581 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:44.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.744+0000 7fcb50e00700 1 -- 172.21.15.89:0/2568963581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0cbb30 msgr2=0x7fcb4c15de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:44.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.744+0000 7fcb4be00700 1 --2- 172.21.15.89:0/2568963581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0cbb30 0x7fcb4c15de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:44.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.744+0000 7fcb51800700 1 --2- 172.21.15.89:0/2568963581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb4c0cd020 0x7fcb4c15e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:44.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.744+0000 7fcb50e00700 1 --2- 172.21.15.89:0/2568963581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0cbb30 0x7fcb4c15de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:44.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.744+0000 7fcb50e00700 1 -- 172.21.15.89:0/2568963581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb4c0cd020 msgr2=0x7fcb4c15e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:44.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.744+0000 7fcb50e00700 1 --2- 172.21.15.89:0/2568963581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb4c0cd020 0x7fcb4c15e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:44.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.744+0000 7fcb50e00700 1 -- 172.21.15.89:0/2568963581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb38009550 con 0x7fcb4c0b2070 2024-04-02T07:48:44.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.744+0000 7fcb50e00700 1 --2- 172.21.15.89:0/2568963581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0b2070 0x7fcb4c1f30b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000c7f0 tx=0x7fcb4000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:44.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.744+0000 7fcb48c00700 1 -- 172.21.15.89:0/2568963581 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb40011800 con 0x7fcb4c0b2070 2024-04-02T07:48:44.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.745+0000 7fcb48c00700 1 -- 172.21.15.89:0/2568963581 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb40011960 con 0x7fcb4c0b2070 2024-04-02T07:48:44.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.745+0000 7fcb53800700 1 -- 172.21.15.89:0/2568963581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4c155c10 con 0x7fcb4c0b2070 2024-04-02T07:48:44.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.745+0000 7fcb48c00700 1 -- 172.21.15.89:0/2568963581 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb40011d10 con 0x7fcb4c0b2070 2024-04-02T07:48:44.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.745+0000 7fcb53800700 1 -- 172.21.15.89:0/2568963581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4c1f35f0 con 0x7fcb4c0b2070 2024-04-02T07:48:44.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.746+0000 7fcb53800700 1 -- 172.21.15.89:0/2568963581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4c067720 con 0x7fcb4c0b2070 2024-04-02T07:48:44.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.746+0000 7fcb48c00700 1 -- 172.21.15.89:0/2568963581 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fcb40018360 con 0x7fcb4c0b2070 2024-04-02T07:48:44.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.747+0000 7fcb48c00700 1 --2- 172.21.15.89:0/2568963581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb2c06ae90 0x7fcb2c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:44.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.747+0000 7fcb48c00700 1 -- 172.21.15.89:0/2568963581 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6555+0+0 (secure 0 0 0) 0x7fcb40089b20 con 0x7fcb4c0b2070 2024-04-02T07:48:44.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.748+0000 7fcb4be00700 1 --2- 172.21.15.89:0/2568963581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb2c06ae90 0x7fcb2c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:44.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.748+0000 7fcb4be00700 1 --2- 172.21.15.89:0/2568963581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb2c06ae90 0x7fcb2c06d350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c147f50 tx=0x7fcb38008030 comp rx=0 tx=0).ready entity=mgr.5173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:44.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:44.752+0000 7fcb48c00700 1 -- 172.21.15.89:0/2568963581 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcb40054580 con 0x7fcb4c0b2070 2024-04-02T07:48:45.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.024+0000 7fcb53800700 1 -- 172.21.15.89:0/2568963581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fcb4c15bda0 con 0x7fcb4c0b2070 2024-04-02T07:48:45.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.934+0000 7fcb48c00700 1 -- 172.21.15.89:0/2568963581 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 15) v1 ==== 89308+0+0 (secure 0 0 0) 0x7fcb40054950 con 0x7fcb4c0b2070 2024-04-02T07:48:45.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.936+0000 7fcb48c00700 1 -- 172.21.15.89:0/2568963581 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v15) v1 ==== 93+0+0 (secure 0 0 0) 0x7fcb40070fa0 con 0x7fcb4c0b2070 2024-04-02T07:48:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.939+0000 7fcb53800700 1 -- 172.21.15.89:0/2568963581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb2c06ae90 msgr2=0x7fcb2c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.939+0000 7fcb53800700 1 --2- 172.21.15.89:0/2568963581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb2c06ae90 0x7fcb2c06d350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c147f50 tx=0x7fcb38008030 comp rx=0 tx=0).stop 2024-04-02T07:48:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.939+0000 7fcb53800700 1 -- 172.21.15.89:0/2568963581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0b2070 msgr2=0x7fcb4c1f30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.939+0000 7fcb53800700 1 --2- 172.21.15.89:0/2568963581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0b2070 0x7fcb4c1f30b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000c7f0 tx=0x7fcb4000cbb0 comp rx=0 tx=0).stop 2024-04-02T07:48:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.939+0000 7fcb53800700 1 -- 172.21.15.89:0/2568963581 shutdown_connections 2024-04-02T07:48:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.939+0000 7fcb53800700 1 --2- 172.21.15.89:0/2568963581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb2c06ae90 0x7fcb2c06d350 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.939+0000 7fcb53800700 1 --2- 172.21.15.89:0/2568963581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb4c0cd020 0x7fcb4c15e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.939+0000 7fcb53800700 1 --2- 172.21.15.89:0/2568963581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb4c0b2070 0x7fcb4c1f30b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.939+0000 7fcb53800700 1 --2- 172.21.15.89:0/2568963581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb4c0cbb30 0x7fcb4c15de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:45.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.939+0000 7fcb53800700 1 -- 172.21.15.89:0/2568963581 >> 172.21.15.89:0/2568963581 conn(0x7fcb4c0b0d70 msgr2=0x7fcb4c0afb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:45.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.940+0000 7fcb53800700 1 -- 172.21.15.89:0/2568963581 shutdown_connections 2024-04-02T07:48:45.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:45.940+0000 7fcb53800700 1 -- 172.21.15.89:0/2568963581 wait complete. 2024-04-02T07:48:45.967 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:48:45.967 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:48:46.101 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:48:46.176 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:46.174+0000 7f0d3a67c240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:48:46.188 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:48:46.188 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:48:46.188 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:48:46.189 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:46.186+0000 7f5ffe757240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:48:46.233 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:48:46.234 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:48:46.354 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:48:46.367 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:46.365+0000 7f0d3a67c240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:48:46.373 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:46.370+0000 7f5ffe757240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:48:46.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.604+0000 7f89cd000700 1 Processor -- start 2024-04-02T07:48:46.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.605+0000 7f89cd000700 1 -- start start 2024-04-02T07:48:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.607+0000 7f89cd000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89c8160f10 0x7f89c80d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.607+0000 7f89cd000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f89c8161590 0x7f89c8161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.607+0000 7f89cd000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c80ceda0 con 0x7f89c80cf230 2024-04-02T07:48:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.607+0000 7f89cd000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89c8161f80 con 0x7f89c8161590 2024-04-02T07:48:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.607+0000 7f89cd000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89c81612f0 con 0x7f89c8160f10 2024-04-02T07:48:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.607+0000 7f89c6800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f89c8161590 0x7f89c8161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.607+0000 7f89c5e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89c8160f10 0x7f89c80d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.607+0000 7f89c6800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f89c8161590 0x7f89c8161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51646/0 (socket says 172.21.15.89:51646) 2024-04-02T07:48:46.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.607+0000 7f89c5400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f89c80cf230 0x7f89c80d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:51906/0 (socket says 172.21.15.89:51906) 2024-04-02T07:48:46.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.607+0000 7f89c6800700 1 -- 172.21.15.89:0/2788996554 learned_addr learned my addr 172.21.15.89:0/2788996554 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:46.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.608+0000 7f89c6800700 1 -- 172.21.15.89:0/2788996554 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89c8160f10 msgr2=0x7f89c80d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:46.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.608+0000 7f89c6800700 1 --2- 172.21.15.89:0/2788996554 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89c8160f10 0x7f89c80d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:46.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.608+0000 7f89c6800700 1 -- 172.21.15.89:0/2788996554 >> v1:172.21.15.113:6789/0 conn(0x7f89c80cf230 legacy=0x7f89c80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:46.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.608+0000 7f89c6800700 1 -- 172.21.15.89:0/2788996554 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c80ce470 con 0x7f89c8161590 2024-04-02T07:48:46.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.608+0000 7f89c6800700 1 --2- 172.21.15.89:0/2788996554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8161590 0x7f89c8161a40 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009e80 tx=0x7f89c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c077cd70f0ccb9c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:46.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.609+0000 7f89c4a00700 1 -- 172.21.15.89:0/2788996554 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c000e070 con 0x7f89c8161590 2024-04-02T07:48:46.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.609+0000 7f89c4a00700 1 -- 172.21.15.89:0/2788996554 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c0003a20 con 0x7f89c8161590 2024-04-02T07:48:46.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.609+0000 7f89c4a00700 1 -- 172.21.15.89:0/2788996554 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c0003db0 con 0x7f89c8161590 2024-04-02T07:48:46.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.609+0000 7f89cd000700 1 -- 172.21.15.89:0/2788996554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8161590 msgr2=0x7f89c8161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:46.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.609+0000 7f89cd000700 1 --2- 172.21.15.89:0/2788996554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8161590 0x7f89c8161a40 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009e80 tx=0x7f89c000b330 comp rx=0 tx=0).stop 2024-04-02T07:48:46.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.609+0000 7f89cd000700 1 -- 172.21.15.89:0/2788996554 shutdown_connections 2024-04-02T07:48:46.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.609+0000 7f89cd000700 1 --2- 172.21.15.89:0/2788996554 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89c8160f10 0x7f89c80d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:46.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.609+0000 7f89cd000700 1 --2- 172.21.15.89:0/2788996554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8161590 0x7f89c8161a40 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:46.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.609+0000 7f89cd000700 1 -- 172.21.15.89:0/2788996554 >> 172.21.15.89:0/2788996554 conn(0x7f89c808aff0 msgr2=0x7f89c80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:46.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.610+0000 7f89cd000700 1 -- 172.21.15.89:0/2788996554 shutdown_connections 2024-04-02T07:48:46.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.610+0000 7f89cd000700 1 -- 172.21.15.89:0/2788996554 wait complete. 2024-04-02T07:48:46.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.611+0000 7f89cd000700 1 Processor -- start 2024-04-02T07:48:46.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.612+0000 7f89cd000700 1 -- start start 2024-04-02T07:48:46.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.612+0000 7f89cd000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89c80cf230 0x7f89c81f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:46.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.612+0000 7f89cd000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8160f10 0x7f89c815f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:46.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.612+0000 7f89cd000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89c8161590 0x7f89c815f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:46.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.613+0000 7f89cd000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c8160060 con 0x7f89c8160f10 2024-04-02T07:48:46.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.613+0000 7f89cd000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c80b6300 con 0x7f89c8161590 2024-04-02T07:48:46.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.613+0000 7f89cd000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89c80b65e0 con 0x7f89c80cf230 2024-04-02T07:48:46.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.613+0000 7f89c5e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89c80cf230 0x7f89c81f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:46.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.613+0000 7f89c5e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89c80cf230 0x7f89c81f3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:44118/0 (socket says 172.21.15.89:44118) 2024-04-02T07:48:46.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.613+0000 7f89c5e00700 1 -- 172.21.15.89:0/3166140032 learned_addr learned my addr 172.21.15.89:0/3166140032 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:46.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.613+0000 7f89c5400700 1 --2- 172.21.15.89:0/3166140032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8160f10 0x7f89c815f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:46.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.614+0000 7f89c5e00700 1 -- 172.21.15.89:0/3166140032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89c8161590 msgr2=0x7f89c815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:46.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.614+0000 7f89c5e00700 1 --2- 172.21.15.89:0/3166140032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89c8161590 0x7f89c815f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:46.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.614+0000 7f89c5e00700 1 -- 172.21.15.89:0/3166140032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8160f10 msgr2=0x7f89c815f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:46.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.614+0000 7f89c5e00700 1 --2- 172.21.15.89:0/3166140032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8160f10 0x7f89c815f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:46.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.614+0000 7f89c5e00700 1 -- 172.21.15.89:0/3166140032 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c0009b30 con 0x7f89c80cf230 2024-04-02T07:48:46.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.614+0000 7f89c5e00700 1 --2- 172.21.15.89:0/3166140032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89c80cf230 0x7f89c81f3fc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f89b800e870 tx=0x7f89b800eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:46.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.614+0000 7f89be000700 1 -- 172.21.15.89:0/3166140032 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b8009590 con 0x7f89c80cf230 2024-04-02T07:48:46.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.614+0000 7f89be000700 1 -- 172.21.15.89:0/3166140032 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b8013680 con 0x7f89c80cf230 2024-04-02T07:48:46.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.614+0000 7f89be000700 1 -- 172.21.15.89:0/3166140032 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b80139f0 con 0x7f89c80cf230 2024-04-02T07:48:46.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.614+0000 7f89cd000700 1 -- 172.21.15.89:0/3166140032 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c80b68c0 con 0x7f89c80cf230 2024-04-02T07:48:46.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.615+0000 7f89cd000700 1 -- 172.21.15.89:0/3166140032 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c81f4500 con 0x7f89c80cf230 2024-04-02T07:48:46.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.616+0000 7f89cd000700 1 -- 172.21.15.89:0/3166140032 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c80b6de0 con 0x7f89c80cf230 2024-04-02T07:48:46.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.617+0000 7f89be000700 1 -- 172.21.15.89:0/3166140032 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 15) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f89b8013b50 con 0x7f89c80cf230 2024-04-02T07:48:46.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.618+0000 7f89be000700 1 --2- 172.21.15.89:0/3166140032 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f89a406ae40 0x7f89a406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:46.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.618+0000 7f89c5400700 1 -- 172.21.15.89:0/3166140032 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f89a406ae40 msgr2=0x7f89a406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:46.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.618+0000 7f89be000700 1 -- 172.21.15.89:0/3166140032 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6619+0+0 (secure 0 0 0) 0x7f89b8089860 con 0x7f89c80cf230 2024-04-02T07:48:46.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.618+0000 7f89c5400700 1 --2- 172.21.15.89:0/3166140032 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f89a406ae40 0x7f89a406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:48:46.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.622+0000 7f89be000700 1 -- 172.21.15.89:0/3166140032 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f89b80544a0 con 0x7f89c80cf230 2024-04-02T07:48:46.727 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:46.725+0000 7f5ffe757240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:48:46.749 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:46.747+0000 7f0d3a67c240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:48:46.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.819+0000 7f89c5400700 1 -- 172.21.15.89:0/3166140032 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f89a406ae40 msgr2=0x7f89a406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:46.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.819+0000 7f89c5400700 1 --2- 172.21.15.89:0/3166140032 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f89a406ae40 0x7f89a406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:48:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.850+0000 7f89cd000700 1 -- 172.21.15.89:0/3166140032 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89c80b6de0 con 0x7f89c80cf230 2024-04-02T07:48:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.852+0000 7f89be000700 1 -- 172.21.15.89:0/3166140032 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+17492 (secure 0 0 0) 0x7f89c80b6de0 con 0x7f89c80cf230 2024-04-02T07:48:46.854 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:46.855 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":51,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:48:45.918211+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:37:03.665694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-02T07:37:04.473147+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":51,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.19","mappings":[{"from":4,"to":5}]},{"pgid":"5.1a","mappings":[{"from":4,"to":5}]},{"pgid":"5.1d","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6833/43357":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/2201396539":"2024-04-03T07:37:22.446577+0000","172.21.15.89:6832/43357":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/350807093":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/2645815547":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/399385398":"2024-04-03T07:37:22.446577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":4,"length":3},{"begin":9,"length":3},{"begin":14,"length":3},{"begin":19,"length":4},{"begin":25,"length":3},{"begin":30,"length":4},{"begin":36,"length":3},{"begin":41,"length":4},{"begin":47,"length":3},{"begin":53,"length":3},{"begin":58,"length":3},{"begin":64,"length":3},{"begin":69,"length":4},{"begin":75,"length":3},{"begin":80,"length":4},{"begin":86,"length":3},{"begin":91,"length":4},{"begin":97,"length":3},{"begin":102,"length":3},{"begin":108,"length":3},{"begin":155,"length":66}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":6,"length":1},{"begin":16,"length":1},{"begin":27,"length":1},{"begin":38,"length":1},{"begin":49,"length":1},{"begin":60,"length":1},{"begin":71,"length":2},{"begin":83,"length":1},{"begin":94,"length":1},{"begin":110,"length":1},{"begin":210,"length":11}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:48:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.854+0000 7f89cd000700 1 -- 172.21.15.89:0/3166140032 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f89a406ae40 msgr2=0x7f89a406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:48:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.854+0000 7f89cd000700 1 --2- 172.21.15.89:0/3166140032 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f89a406ae40 0x7f89a406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.854+0000 7f89cd000700 1 -- 172.21.15.89:0/3166140032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89c80cf230 msgr2=0x7f89c81f3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.854+0000 7f89cd000700 1 --2- 172.21.15.89:0/3166140032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89c80cf230 0x7f89c81f3fc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f89b800e870 tx=0x7f89b800eb80 comp rx=0 tx=0).stop 2024-04-02T07:48:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.854+0000 7f89cd000700 1 -- 172.21.15.89:0/3166140032 shutdown_connections 2024-04-02T07:48:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.854+0000 7f89cd000700 1 --2- 172.21.15.89:0/3166140032 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f89a406ae40 0x7f89a406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.854+0000 7f89cd000700 1 --2- 172.21.15.89:0/3166140032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89c8161590 0x7f89c815f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.854+0000 7f89cd000700 1 --2- 172.21.15.89:0/3166140032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89c80cf230 0x7f89c81f3fc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.854+0000 7f89cd000700 1 --2- 172.21.15.89:0/3166140032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8160f10 0x7f89c815f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.854+0000 7f89cd000700 1 -- 172.21.15.89:0/3166140032 >> 172.21.15.89:0/3166140032 conn(0x7f89c808aff0 msgr2=0x7f89c80cc660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.854+0000 7f89cd000700 1 -- 172.21.15.89:0/3166140032 shutdown_connections 2024-04-02T07:48:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:46.854+0000 7f89cd000700 1 -- 172.21.15.89:0/3166140032 wait complete. 2024-04-02T07:48:46.879 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:37:03.665694+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:37:04.473147+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 220, 'snap_epoch': 51, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:48:46.879 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:48:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.120+0000 7fe972600700 1 Processor -- start 2024-04-02T07:48:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.120+0000 7fe972600700 1 -- start start 2024-04-02T07:48:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.121+0000 7fe972600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe96c1611b0 0x7fe96c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.121+0000 7fe972600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe96c15f4d0 0x7fe96c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.121+0000 7fe972600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96c0ce270 con 0x7fe96c0d21f0 2024-04-02T07:48:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.121+0000 7fe972600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe96c15b600 con 0x7fe96c15f4d0 2024-04-02T07:48:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.121+0000 7fe972600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c0d2760 con 0x7fe96c1611b0 2024-04-02T07:48:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.122+0000 7fe96aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe96c15f4d0 0x7fe96c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.122+0000 7fe96be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe96c0d21f0 0x7fe96c161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32984/0 (socket says 172.21.15.89:32984) 2024-04-02T07:48:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.122+0000 7fe96b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe96c1611b0 0x7fe96c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.122+0000 7fe96be00700 1 -- 172.21.15.89:0/2645546401 learned_addr learned my addr 172.21.15.89:0/2645546401 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.122+0000 7fe96b400700 1 -- 172.21.15.89:0/2645546401 >> v2:172.21.15.89:3300/0 conn(0x7fe96c15f4d0 msgr2=0x7fe96c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.122+0000 7fe96b400700 1 --2- 172.21.15.89:0/2645546401 >> v2:172.21.15.89:3300/0 conn(0x7fe96c15f4d0 0x7fe96c0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.122+0000 7fe96b400700 1 -- 172.21.15.89:0/2645546401 >> v1:172.21.15.89:6789/0 conn(0x7fe96c0d21f0 legacy=0x7fe96c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.122+0000 7fe96b400700 1 -- 172.21.15.89:0/2645546401 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96c0d2a40 con 0x7fe96c1611b0 2024-04-02T07:48:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.122+0000 7fe96b400700 1 --2- 172.21.15.89:0/2645546401 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe96c1611b0 0x7fe96c15ef90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe95c009840 tx=0x7fe95c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=697688b9a43c268b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.123+0000 7fe96a000700 1 -- 172.21.15.89:0/2645546401 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c011830 con 0x7fe96c1611b0 2024-04-02T07:48:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.123+0000 7fe96a000700 1 -- 172.21.15.89:0/2645546401 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95c011990 con 0x7fe96c1611b0 2024-04-02T07:48:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.123+0000 7fe96a000700 1 -- 172.21.15.89:0/2645546401 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c011d20 con 0x7fe96c1611b0 2024-04-02T07:48:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.123+0000 7fe972600700 1 -- 172.21.15.89:0/2645546401 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe96c1611b0 msgr2=0x7fe96c15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.123+0000 7fe972600700 1 --2- 172.21.15.89:0/2645546401 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe96c1611b0 0x7fe96c15ef90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe95c009840 tx=0x7fe95c009b50 comp rx=0 tx=0).stop 2024-04-02T07:48:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.123+0000 7fe972600700 1 -- 172.21.15.89:0/2645546401 shutdown_connections 2024-04-02T07:48:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.123+0000 7fe972600700 1 --2- 172.21.15.89:0/2645546401 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe96c1611b0 0x7fe96c15ef90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.123+0000 7fe972600700 1 --2- 172.21.15.89:0/2645546401 >> v2:172.21.15.89:3300/0 conn(0x7fe96c15f4d0 0x7fe96c0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.123+0000 7fe972600700 1 -- 172.21.15.89:0/2645546401 >> 172.21.15.89:0/2645546401 conn(0x7fe96c08aff0 msgr2=0x7fe96c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.124+0000 7fe972600700 1 -- 172.21.15.89:0/2645546401 shutdown_connections 2024-04-02T07:48:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.124+0000 7fe972600700 1 -- 172.21.15.89:0/2645546401 wait complete. 2024-04-02T07:48:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.125+0000 7fe972600700 1 Processor -- start 2024-04-02T07:48:47.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.125+0000 7fe972600700 1 -- start start 2024-04-02T07:48:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.126+0000 7fe972600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe96c0d21f0 0x7fe96c0a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.126+0000 7fe972600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe96c15f4d0 0x7fe96c09eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.126+0000 7fe972600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe96c1611b0 0x7fe96c150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.126+0000 7fe972600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c0a1190 con 0x7fe96c0d21f0 2024-04-02T07:48:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.126+0000 7fe972600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c09e340 con 0x7fe96c1611b0 2024-04-02T07:48:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.126+0000 7fe972600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c09e060 con 0x7fe96c15f4d0 2024-04-02T07:48:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.127+0000 7fe96b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe96c0d21f0 0x7fe96c0a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.127+0000 7fe96aa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe96c15f4d0 0x7fe96c09eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.127+0000 7fe96b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe96c0d21f0 0x7fe96c0a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51684/0 (socket says 172.21.15.89:51684) 2024-04-02T07:48:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.127+0000 7fe96b400700 1 -- 172.21.15.89:0/2953725770 learned_addr learned my addr 172.21.15.89:0/2953725770 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.127+0000 7fe96be00700 1 --2- 172.21.15.89:0/2953725770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe96c1611b0 0x7fe96c150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.127+0000 7fe96b400700 1 -- 172.21.15.89:0/2953725770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe96c15f4d0 msgr2=0x7fe96c09eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.127+0000 7fe96b400700 1 --2- 172.21.15.89:0/2953725770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe96c15f4d0 0x7fe96c09eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.127+0000 7fe96b400700 1 -- 172.21.15.89:0/2953725770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe96c1611b0 msgr2=0x7fe96c150590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.127+0000 7fe96b400700 1 --2- 172.21.15.89:0/2953725770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe96c1611b0 0x7fe96c150590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.127+0000 7fe96b400700 1 -- 172.21.15.89:0/2953725770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c0094a0 con 0x7fe96c0d21f0 2024-04-02T07:48:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.127+0000 7fe96aa00700 1 --2- 172.21.15.89:0/2953725770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe96c15f4d0 0x7fe96c09eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:48:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.127+0000 7fe96b400700 1 --2- 172.21.15.89:0/2953725770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe96c0d21f0 0x7fe96c0a0c50 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe95c009fd0 tx=0x7fe95c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.128+0000 7fe963400700 1 -- 172.21.15.89:0/2953725770 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c0036a0 con 0x7fe96c0d21f0 2024-04-02T07:48:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.128+0000 7fe963400700 1 -- 172.21.15.89:0/2953725770 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95c01e070 con 0x7fe96c0d21f0 2024-04-02T07:48:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.128+0000 7fe963400700 1 -- 172.21.15.89:0/2953725770 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c003a40 con 0x7fe96c0d21f0 2024-04-02T07:48:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.128+0000 7fe972600700 1 -- 172.21.15.89:0/2953725770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96c0cc760 con 0x7fe96c0d21f0 2024-04-02T07:48:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.128+0000 7fe972600700 1 -- 172.21.15.89:0/2953725770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96c0ccd20 con 0x7fe96c0d21f0 2024-04-02T07:48:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.129+0000 7fe972600700 1 -- 172.21.15.89:0/2953725770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe96c067720 con 0x7fe96c0d21f0 2024-04-02T07:48:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.130+0000 7fe963400700 1 -- 172.21.15.89:0/2953725770 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 89308+0+0 (secure 0 0 0) 0x7fe95c003ba0 con 0x7fe96c0d21f0 2024-04-02T07:48:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.131+0000 7fe963400700 1 --2- 172.21.15.89:0/2953725770 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe94c06ae40 0x7fe94c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.131+0000 7fe963400700 1 -- 172.21.15.89:0/2953725770 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6619+0+0 (secure 0 0 0) 0x7fe95c08a920 con 0x7fe96c0d21f0 2024-04-02T07:48:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.131+0000 7fe96aa00700 1 -- 172.21.15.89:0/2953725770 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe94c06ae40 msgr2=0x7fe94c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.131+0000 7fe96aa00700 1 --2- 172.21.15.89:0/2953725770 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe94c06ae40 0x7fe94c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:48:47.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.136+0000 7fe963400700 1 -- 172.21.15.89:0/2953725770 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe95c0554e0 con 0x7fe96c0d21f0 2024-04-02T07:48:47.260 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:47.257+0000 7f5ffe757240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:48:47.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.332+0000 7fe96aa00700 1 -- 172.21.15.89:0/2953725770 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe94c06ae40 msgr2=0x7fe94c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:47.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.332+0000 7fe96aa00700 1 --2- 172.21.15.89:0/2953725770 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe94c06ae40 0x7fe94c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:48:47.348 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:47.345+0000 7f0d3a67c240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:48:47.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.378+0000 7fe972600700 1 -- 172.21.15.89:0/2953725770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe96c0cedf0 con 0x7fe96c0d21f0 2024-04-02T07:48:47.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.379+0000 7fe963400700 1 -- 172.21.15.89:0/2953725770 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe95c015020 con 0x7fe96c0d21f0 2024-04-02T07:48:47.381 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:48:47.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.381+0000 7fe972600700 1 -- 172.21.15.89:0/2953725770 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe94c06ae40 msgr2=0x7fe94c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:48:47.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.381+0000 7fe972600700 1 --2- 172.21.15.89:0/2953725770 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe94c06ae40 0x7fe94c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.381+0000 7fe972600700 1 -- 172.21.15.89:0/2953725770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe96c0d21f0 msgr2=0x7fe96c0a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:47.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.381+0000 7fe972600700 1 --2- 172.21.15.89:0/2953725770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe96c0d21f0 0x7fe96c0a0c50 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe95c009fd0 tx=0x7fe95c004750 comp rx=0 tx=0).stop 2024-04-02T07:48:47.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.381+0000 7fe972600700 1 -- 172.21.15.89:0/2953725770 shutdown_connections 2024-04-02T07:48:47.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.381+0000 7fe972600700 1 --2- 172.21.15.89:0/2953725770 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe94c06ae40 0x7fe94c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.381+0000 7fe972600700 1 --2- 172.21.15.89:0/2953725770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe96c1611b0 0x7fe96c150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.381+0000 7fe972600700 1 --2- 172.21.15.89:0/2953725770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe96c0d21f0 0x7fe96c0a0c50 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.381+0000 7fe972600700 1 --2- 172.21.15.89:0/2953725770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe96c15f4d0 0x7fe96c09eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.381+0000 7fe972600700 1 -- 172.21.15.89:0/2953725770 >> 172.21.15.89:0/2953725770 conn(0x7fe96c08aff0 msgr2=0x7fe96c160510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:47.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.381+0000 7fe972600700 1 -- 172.21.15.89:0/2953725770 shutdown_connections 2024-04-02T07:48:47.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.381+0000 7fe972600700 1 -- 172.21.15.89:0/2953725770 wait complete. 2024-04-02T07:48:47.402 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:48:47.513 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:47.510+0000 7f5ffe757240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:48:47.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.616+0000 7f3930000700 1 Processor -- start 2024-04-02T07:48:47.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.616+0000 7f3930000700 1 -- start start 2024-04-02T07:48:47.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.618+0000 7f3930000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f39280cd020 0x7f39280cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:47.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.618+0000 7f3930000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39280b2070 0x7f39280b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:47.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.618+0000 7f3930000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39280cdae0 con 0x7f39280cbb30 2024-04-02T07:48:47.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.618+0000 7f3930000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39280cddc0 con 0x7f39280cd020 2024-04-02T07:48:47.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.618+0000 7f3930000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39280ce0a0 con 0x7f39280b2070 2024-04-02T07:48:47.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.618+0000 7f392e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39280b2070 0x7f39280b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:47.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.618+0000 7f392cc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f39280cd020 0x7f39280cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:47.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.619+0000 7f392cc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f39280cd020 0x7f39280cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:46074/0 (socket says 172.21.15.89:46074) 2024-04-02T07:48:47.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.619+0000 7f392cc00700 1 -- 172.21.15.89:0/3553787130 learned_addr learned my addr 172.21.15.89:0/3553787130 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:47.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.619+0000 7f392d600700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f39280cbb30 0x7f39280cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:51910/0 (socket says 172.21.15.89:51910) 2024-04-02T07:48:47.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.619+0000 7f3927e00700 1 -- 172.21.15.89:0/3553787130 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131691333 0 0) 0x7f39280cdae0 con 0x7f39280cbb30 2024-04-02T07:48:47.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.619+0000 7f3927e00700 1 -- 172.21.15.89:0/3553787130 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39100035b0 con 0x7f39280cbb30 2024-04-02T07:48:47.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.619+0000 7f392cc00700 1 -- 172.21.15.89:0/3553787130 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39280b2070 msgr2=0x7f39280b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:47.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.619+0000 7f392cc00700 1 --2- 172.21.15.89:0/3553787130 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39280b2070 0x7f39280b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.619+0000 7f392cc00700 1 -- 172.21.15.89:0/3553787130 >> v1:172.21.15.113:6789/0 conn(0x7f39280cbb30 legacy=0x7f39280cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:47.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.619+0000 7f392cc00700 1 -- 172.21.15.89:0/3553787130 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392806dc20 con 0x7f39280cd020 2024-04-02T07:48:47.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.619+0000 7f392e000700 1 --2- 172.21.15.89:0/3553787130 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39280b2070 0x7f39280b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:48:47.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.619+0000 7f392cc00700 1 --2- 172.21.15.89:0/3553787130 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280cd020 0x7f39280cd480 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f39140098f0 tx=0x7f3914009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c03cd4fab29b7ed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:47.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.620+0000 7f3927e00700 1 -- 172.21.15.89:0/3553787130 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914011830 con 0x7f39280cd020 2024-04-02T07:48:47.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.620+0000 7f3927e00700 1 -- 172.21.15.89:0/3553787130 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3914011990 con 0x7f39280cd020 2024-04-02T07:48:47.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.620+0000 7f3927e00700 1 -- 172.21.15.89:0/3553787130 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914011d20 con 0x7f39280cd020 2024-04-02T07:48:47.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.620+0000 7f3930000700 1 -- 172.21.15.89:0/3553787130 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280cd020 msgr2=0x7f39280cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:47.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.620+0000 7f3930000700 1 --2- 172.21.15.89:0/3553787130 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280cd020 0x7f39280cd480 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f39140098f0 tx=0x7f3914009c00 comp rx=0 tx=0).stop 2024-04-02T07:48:47.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.620+0000 7f3930000700 1 -- 172.21.15.89:0/3553787130 shutdown_connections 2024-04-02T07:48:47.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.620+0000 7f3930000700 1 --2- 172.21.15.89:0/3553787130 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39280b2070 0x7f39280b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.620+0000 7f3930000700 1 --2- 172.21.15.89:0/3553787130 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280cd020 0x7f39280cd480 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.620+0000 7f3930000700 1 -- 172.21.15.89:0/3553787130 >> 172.21.15.89:0/3553787130 conn(0x7f39280b0d70 msgr2=0x7f39280b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:47.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.620+0000 7f3930000700 1 -- 172.21.15.89:0/3553787130 shutdown_connections 2024-04-02T07:48:47.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.620+0000 7f3930000700 1 -- 172.21.15.89:0/3553787130 wait complete. 2024-04-02T07:48:47.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.622+0000 7f3930000700 1 Processor -- start 2024-04-02T07:48:47.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.622+0000 7f3930000700 1 -- start start 2024-04-02T07:48:47.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.623+0000 7f3930000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280b2070 0x7f39280ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:47.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.623+0000 7f3930000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39280cbb30 0x7f392815d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:47.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.623+0000 7f3930000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39280cd020 0x7f39280c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:47.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.623+0000 7f3930000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39280c91d0 con 0x7f39280cbb30 2024-04-02T07:48:47.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.623+0000 7f3930000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39280c94b0 con 0x7f39280b2070 2024-04-02T07:48:47.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.623+0000 7f3930000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39280c9790 con 0x7f39280cd020 2024-04-02T07:48:47.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.623+0000 7f392d600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280b2070 0x7f39280ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:47.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.623+0000 7f392cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39280cbb30 0x7f392815d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:47.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.623+0000 7f392d600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280b2070 0x7f39280ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:46086/0 (socket says 172.21.15.89:46086) 2024-04-02T07:48:47.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.623+0000 7f392d600700 1 -- 172.21.15.89:0/2859094995 learned_addr learned my addr 172.21.15.89:0/2859094995 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:47.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.623+0000 7f392e000700 1 --2- 172.21.15.89:0/2859094995 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39280cd020 0x7f39280c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:47.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.624+0000 7f392d600700 1 -- 172.21.15.89:0/2859094995 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39280cd020 msgr2=0x7f39280c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:47.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.624+0000 7f392d600700 1 --2- 172.21.15.89:0/2859094995 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39280cd020 0x7f39280c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.624+0000 7f392d600700 1 -- 172.21.15.89:0/2859094995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39280cbb30 msgr2=0x7f392815d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:47.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.624+0000 7f392d600700 1 --2- 172.21.15.89:0/2859094995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39280cbb30 0x7f392815d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.624+0000 7f392d600700 1 -- 172.21.15.89:0/2859094995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3920009a70 con 0x7f39280b2070 2024-04-02T07:48:47.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.624+0000 7f392cc00700 1 --2- 172.21.15.89:0/2859094995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39280cbb30 0x7f392815d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:48:47.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.624+0000 7f392d600700 1 --2- 172.21.15.89:0/2859094995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280b2070 0x7f39280ca250 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f391c00c700 tx=0x7f391c00ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:47.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.624+0000 7f3925600700 1 -- 172.21.15.89:0/2859094995 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c005cb0 con 0x7f39280b2070 2024-04-02T07:48:47.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.624+0000 7f3925600700 1 -- 172.21.15.89:0/2859094995 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391c005e10 con 0x7f39280b2070 2024-04-02T07:48:47.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.624+0000 7f3925600700 1 -- 172.21.15.89:0/2859094995 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c0072d0 con 0x7f39280b2070 2024-04-02T07:48:47.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.624+0000 7f3930000700 1 -- 172.21.15.89:0/2859094995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3914009550 con 0x7f39280b2070 2024-04-02T07:48:47.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.625+0000 7f3930000700 1 -- 172.21.15.89:0/2859094995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39280c9d50 con 0x7f39280b2070 2024-04-02T07:48:47.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.626+0000 7f3930000700 1 -- 172.21.15.89:0/2859094995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39280ca0c0 con 0x7f39280b2070 2024-04-02T07:48:47.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.626+0000 7f3925600700 1 -- 172.21.15.89:0/2859094995 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 15) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f391c007430 con 0x7f39280b2070 2024-04-02T07:48:47.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.627+0000 7f3925600700 1 --2- 172.21.15.89:0/2859094995 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f390806ae40 0x7f390806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:47.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.627+0000 7f3925600700 1 -- 172.21.15.89:0/2859094995 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6619+0+0 (secure 0 0 0) 0x7f391c088d00 con 0x7f39280b2070 2024-04-02T07:48:47.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.627+0000 7f392cc00700 1 -- 172.21.15.89:0/2859094995 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f390806ae40 msgr2=0x7f390806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:47.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.627+0000 7f392cc00700 1 --2- 172.21.15.89:0/2859094995 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f390806ae40 0x7f390806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:48:47.634 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:47.632+0000 7f0d3a67c240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:48:47.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.632+0000 7f3925600700 1 -- 172.21.15.89:0/2859094995 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f391c053980 con 0x7f39280b2070 2024-04-02T07:48:47.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.828+0000 7f392cc00700 1 -- 172.21.15.89:0/2859094995 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f390806ae40 msgr2=0x7f390806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:47.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.828+0000 7f392cc00700 1 --2- 172.21.15.89:0/2859094995 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f390806ae40 0x7f390806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:48:47.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.842+0000 7f3930000700 1 -- 172.21.15.89:0/2859094995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f392815bda0 con 0x7f39280b2070 2024-04-02T07:48:47.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.843+0000 7f3925600700 1 -- 172.21.15.89:0/2859094995 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v51) v1 ==== 104+0+11 (secure 0 0 0) 0x7f391c0170a0 con 0x7f39280b2070 2024-04-02T07:48:47.845 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:48:47.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.845+0000 7f3930000700 1 -- 172.21.15.89:0/2859094995 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f390806ae40 msgr2=0x7f390806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:48:47.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.845+0000 7f3930000700 1 --2- 172.21.15.89:0/2859094995 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f390806ae40 0x7f390806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.845+0000 7f3930000700 1 -- 172.21.15.89:0/2859094995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280b2070 msgr2=0x7f39280ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:47.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.845+0000 7f3930000700 1 --2- 172.21.15.89:0/2859094995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280b2070 0x7f39280ca250 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f391c00c700 tx=0x7f391c00ca10 comp rx=0 tx=0).stop 2024-04-02T07:48:47.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.846+0000 7f3930000700 1 -- 172.21.15.89:0/2859094995 shutdown_connections 2024-04-02T07:48:47.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.846+0000 7f3930000700 1 --2- 172.21.15.89:0/2859094995 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f390806ae40 0x7f390806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.846+0000 7f3930000700 1 --2- 172.21.15.89:0/2859094995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280b2070 0x7f39280ca250 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.846+0000 7f3930000700 1 --2- 172.21.15.89:0/2859094995 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39280cd020 0x7f39280c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.846+0000 7f3930000700 1 --2- 172.21.15.89:0/2859094995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39280cbb30 0x7f392815d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:47.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.846+0000 7f3930000700 1 -- 172.21.15.89:0/2859094995 >> 172.21.15.89:0/2859094995 conn(0x7f39280b0d70 msgr2=0x7f392815cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:47.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.846+0000 7f3930000700 1 -- 172.21.15.89:0/2859094995 shutdown_connections 2024-04-02T07:48:47.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:47.846+0000 7f3930000700 1 -- 172.21.15.89:0/2859094995 wait complete. 2024-04-02T07:48:47.866 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:48:47.971 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:47.969+0000 7f5ffe757240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:48:48.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.074+0000 7fbe5f000700 1 Processor -- start 2024-04-02T07:48:48.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.074+0000 7fbe5f000700 1 -- start start 2024-04-02T07:48:48.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.075+0000 7fbe5f000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbe5806d370 0x7fbe5815a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:48.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.075+0000 7fbe5f000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe580b3e10 0x7fbe580b4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:48.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.075+0000 7fbe5f000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe580af4f0 con 0x7fbe5815a290 2024-04-02T07:48:48.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.075+0000 7fbe5f000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe580b47a0 con 0x7fbe5806d370 2024-04-02T07:48:48.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.075+0000 7fbe5f000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5806d800 con 0x7fbe580b3e10 2024-04-02T07:48:48.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.075+0000 7fbe57400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbe5806d370 0x7fbe5815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:48.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.075+0000 7fbe57400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbe5806d370 0x7fbe5815a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51700/0 (socket says 172.21.15.89:51700) 2024-04-02T07:48:48.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.075+0000 7fbe57400700 1 -- 172.21.15.89:0/1805573120 learned_addr learned my addr 172.21.15.89:0/1805573120 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:48.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.075+0000 7fbe5d000700 1 --2- 172.21.15.89:0/1805573120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe580b3e10 0x7fbe580b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:48.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.076+0000 7fbe57400700 1 -- 172.21.15.89:0/1805573120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe580b3e10 msgr2=0x7fbe580b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:48.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.076+0000 7fbe57400700 1 --2- 172.21.15.89:0/1805573120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe580b3e10 0x7fbe580b4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.076+0000 7fbe57400700 1 -- 172.21.15.89:0/1805573120 >> v1:172.21.15.113:6789/0 conn(0x7fbe5815a290 legacy=0x7fbe580cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:48.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.076+0000 7fbe57400700 1 -- 172.21.15.89:0/1805573120 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5806dae0 con 0x7fbe5806d370 2024-04-02T07:48:48.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.076+0000 7fbe57400700 1 --2- 172.21.15.89:0/1805573120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe5806d370 0x7fbe5815a670 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fbe440098f0 tx=0x7fbe44009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3270445a0798122 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:48.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.076+0000 7fbe56a00700 1 -- 172.21.15.89:0/1805573120 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe44011830 con 0x7fbe5806d370 2024-04-02T07:48:48.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.076+0000 7fbe56a00700 1 -- 172.21.15.89:0/1805573120 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe44011990 con 0x7fbe5806d370 2024-04-02T07:48:48.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.076+0000 7fbe56a00700 1 -- 172.21.15.89:0/1805573120 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe44011d20 con 0x7fbe5806d370 2024-04-02T07:48:48.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.077+0000 7fbe5f000700 1 -- 172.21.15.89:0/1805573120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe5806d370 msgr2=0x7fbe5815a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:48.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.077+0000 7fbe5f000700 1 --2- 172.21.15.89:0/1805573120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe5806d370 0x7fbe5815a670 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fbe440098f0 tx=0x7fbe44009c00 comp rx=0 tx=0).stop 2024-04-02T07:48:48.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.077+0000 7fbe5f000700 1 -- 172.21.15.89:0/1805573120 shutdown_connections 2024-04-02T07:48:48.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.077+0000 7fbe5f000700 1 --2- 172.21.15.89:0/1805573120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe580b3e10 0x7fbe580b4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.077+0000 7fbe5f000700 1 --2- 172.21.15.89:0/1805573120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe5806d370 0x7fbe5815a670 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.077+0000 7fbe5f000700 1 -- 172.21.15.89:0/1805573120 >> 172.21.15.89:0/1805573120 conn(0x7fbe58071fb0 msgr2=0x7fbe5806a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:48.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.077+0000 7fbe5f000700 1 -- 172.21.15.89:0/1805573120 shutdown_connections 2024-04-02T07:48:48.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.077+0000 7fbe5f000700 1 -- 172.21.15.89:0/1805573120 wait complete. 2024-04-02T07:48:48.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.079+0000 7fbe5f000700 1 Processor -- start 2024-04-02T07:48:48.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.079+0000 7fbe5f000700 1 -- start start 2024-04-02T07:48:48.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe5f000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe580b3e10 0x7fbe58157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:48.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe5f000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe5815a290 0x7fbe581593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:48.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe5f000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe5814f4c0 0x7fbe5815c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:48.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe5f000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe58157f10 con 0x7fbe5815a290 2024-04-02T07:48:48.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe5f000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe581581f0 con 0x7fbe580b3e10 2024-04-02T07:48:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe5f000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe581584d0 con 0x7fbe5814f4c0 2024-04-02T07:48:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe57400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe5815a290 0x7fbe581593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe57400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe5815a290 0x7fbe581593e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51714/0 (socket says 172.21.15.89:51714) 2024-04-02T07:48:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe57400700 1 -- 172.21.15.89:0/1707365763 learned_addr learned my addr 172.21.15.89:0/1707365763 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe5d000700 1 --2- 172.21.15.89:0/1707365763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe5814f4c0 0x7fbe5815c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe57e00700 1 --2- 172.21.15.89:0/1707365763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe580b3e10 0x7fbe58157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe57400700 1 -- 172.21.15.89:0/1707365763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe5814f4c0 msgr2=0x7fbe5815c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe57400700 1 --2- 172.21.15.89:0/1707365763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe5814f4c0 0x7fbe5815c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe57400700 1 -- 172.21.15.89:0/1707365763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe580b3e10 msgr2=0x7fbe58157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe57400700 1 --2- 172.21.15.89:0/1707365763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe580b3e10 0x7fbe58157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.080+0000 7fbe57400700 1 -- 172.21.15.89:0/1707365763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe44009550 con 0x7fbe5815a290 2024-04-02T07:48:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.081+0000 7fbe57400700 1 --2- 172.21.15.89:0/1707365763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe5815a290 0x7fbe581593e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fbe44000c00 tx=0x7fbe44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:48.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.081+0000 7fbe4fe00700 1 -- 172.21.15.89:0/1707365763 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe44004400 con 0x7fbe5815a290 2024-04-02T07:48:48.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.081+0000 7fbe4fe00700 1 -- 172.21.15.89:0/1707365763 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4401f070 con 0x7fbe5815a290 2024-04-02T07:48:48.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.081+0000 7fbe4fe00700 1 -- 172.21.15.89:0/1707365763 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe440038e0 con 0x7fbe5815a290 2024-04-02T07:48:48.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.081+0000 7fbe5f000700 1 -- 172.21.15.89:0/1707365763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe581587b0 con 0x7fbe5815a290 2024-04-02T07:48:48.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.081+0000 7fbe5f000700 1 -- 172.21.15.89:0/1707365763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5814f950 con 0x7fbe5815a290 2024-04-02T07:48:48.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.082+0000 7fbe5f000700 1 -- 172.21.15.89:0/1707365763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe58152600 con 0x7fbe5815a290 2024-04-02T07:48:48.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.083+0000 7fbe4fe00700 1 -- 172.21.15.89:0/1707365763 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 89308+0+0 (secure 0 0 0) 0x7fbe44021030 con 0x7fbe5815a290 2024-04-02T07:48:48.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.084+0000 7fbe4fe00700 1 --2- 172.21.15.89:0/1707365763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbe3806ae40 0x7fbe3806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:48.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.084+0000 7fbe4fe00700 1 -- 172.21.15.89:0/1707365763 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6779+0+0 (secure 0 0 0) 0x7fbe4408ae10 con 0x7fbe5815a290 2024-04-02T07:48:48.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.084+0000 7fbe57e00700 1 -- 172.21.15.89:0/1707365763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbe3806ae40 msgr2=0x7fbe3806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:48.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.085+0000 7fbe57e00700 1 --2- 172.21.15.89:0/1707365763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbe3806ae40 0x7fbe3806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:48:48.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.089+0000 7fbe4fe00700 1 -- 172.21.15.89:0/1707365763 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbe58152600 con 0x7fbe5815a290 2024-04-02T07:48:48.123 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:48.121+0000 7f0d3a67c240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:48:48.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.286+0000 7fbe57e00700 1 -- 172.21.15.89:0/1707365763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbe3806ae40 msgr2=0x7fbe3806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:48.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.286+0000 7fbe57e00700 1 --2- 172.21.15.89:0/1707365763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbe3806ae40 0x7fbe3806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:48:48.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.303+0000 7fbe5f000700 1 -- 172.21.15.89:0/1707365763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbe58152600 con 0x7fbe5815a290 2024-04-02T07:48:48.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.304+0000 7fbe4fe00700 1 -- 172.21.15.89:0/1707365763 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbe58152600 con 0x7fbe5815a290 2024-04-02T07:48:48.306 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:48:48.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.307+0000 7fbe5f000700 1 -- 172.21.15.89:0/1707365763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbe3806ae40 msgr2=0x7fbe3806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:48:48.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.307+0000 7fbe5f000700 1 --2- 172.21.15.89:0/1707365763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbe3806ae40 0x7fbe3806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.307+0000 7fbe5f000700 1 -- 172.21.15.89:0/1707365763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe5815a290 msgr2=0x7fbe581593e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:48.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.307+0000 7fbe5f000700 1 --2- 172.21.15.89:0/1707365763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe5815a290 0x7fbe581593e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fbe44000c00 tx=0x7fbe44004750 comp rx=0 tx=0).stop 2024-04-02T07:48:48.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.307+0000 7fbe5f000700 1 -- 172.21.15.89:0/1707365763 shutdown_connections 2024-04-02T07:48:48.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.307+0000 7fbe5f000700 1 --2- 172.21.15.89:0/1707365763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbe3806ae40 0x7fbe3806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.307+0000 7fbe5f000700 1 --2- 172.21.15.89:0/1707365763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe580b3e10 0x7fbe58157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.307+0000 7fbe5f000700 1 --2- 172.21.15.89:0/1707365763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe5814f4c0 0x7fbe5815c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.307+0000 7fbe5f000700 1 --2- 172.21.15.89:0/1707365763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe5815a290 0x7fbe581593e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.307+0000 7fbe5f000700 1 -- 172.21.15.89:0/1707365763 >> 172.21.15.89:0/1707365763 conn(0x7fbe58071fb0 msgr2=0x7fbe580b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:48.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.307+0000 7fbe5f000700 1 -- 172.21.15.89:0/1707365763 shutdown_connections 2024-04-02T07:48:48.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.307+0000 7fbe5f000700 1 -- 172.21.15.89:0/1707365763 wait complete. 2024-04-02T07:48:48.328 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T07:48:48.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.524+0000 7f629a200700 1 Processor -- start 2024-04-02T07:48:48.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.524+0000 7f629a200700 1 -- start start 2024-04-02T07:48:48.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.525+0000 7f629a200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62940d16a0 0x7f62940d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:48.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.525+0000 7f629a200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f62940d2370 0x7f62940ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:48.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.525+0000 7f629a200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6294161420 con 0x7f629415d9d0 2024-04-02T07:48:48.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.525+0000 7f629a200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62940d2820 con 0x7f62940d2370 2024-04-02T07:48:48.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.525+0000 7f629a200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62940d4340 con 0x7f62940d16a0 2024-04-02T07:48:48.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.525+0000 7f6293e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f62940d2370 0x7f62940ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:48.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.525+0000 7f6293400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f629415d9d0 0x7f62940d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32998/0 (socket says 172.21.15.89:32998) 2024-04-02T07:48:48.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.525+0000 7f6293400700 1 -- 172.21.15.89:0/1807485775 learned_addr learned my addr 172.21.15.89:0/1807485775 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:48.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.525+0000 7f6292a00700 1 --2- 172.21.15.89:0/1807485775 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62940d16a0 0x7f62940d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:48.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.526+0000 7f6293e00700 1 -- 172.21.15.89:0/1807485775 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62940d16a0 msgr2=0x7f62940d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:48.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.526+0000 7f6293e00700 1 --2- 172.21.15.89:0/1807485775 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62940d16a0 0x7f62940d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.526+0000 7f6293e00700 1 -- 172.21.15.89:0/1807485775 >> v1:172.21.15.89:6789/0 conn(0x7f629415d9d0 legacy=0x7f62940d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:48.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.526+0000 7f6293e00700 1 -- 172.21.15.89:0/1807485775 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6294161700 con 0x7f62940d2370 2024-04-02T07:48:48.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.526+0000 7f6292000700 1 -- 172.21.15.89:0/1807485775 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779501019 0 0) 0x7f6294161420 con 0x7f629415d9d0 2024-04-02T07:48:48.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.526+0000 7f6293e00700 1 --2- 172.21.15.89:0/1807485775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62940d2370 0x7f62940ce360 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f628c009dc0 tx=0x7f628c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f55be31af36b7f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:48.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.526+0000 7f6292000700 1 -- 172.21.15.89:0/1807485775 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c00e070 con 0x7f62940d2370 2024-04-02T07:48:48.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.526+0000 7f6292000700 1 -- 172.21.15.89:0/1807485775 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628c00bae0 con 0x7f62940d2370 2024-04-02T07:48:48.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.526+0000 7f6292000700 1 -- 172.21.15.89:0/1807485775 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c00be90 con 0x7f62940d2370 2024-04-02T07:48:48.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.527+0000 7f629a200700 1 -- 172.21.15.89:0/1807485775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62940d2370 msgr2=0x7f62940ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:48.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.527+0000 7f629a200700 1 --2- 172.21.15.89:0/1807485775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62940d2370 0x7f62940ce360 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f628c009dc0 tx=0x7f628c00b330 comp rx=0 tx=0).stop 2024-04-02T07:48:48.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.527+0000 7f629a200700 1 -- 172.21.15.89:0/1807485775 shutdown_connections 2024-04-02T07:48:48.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.527+0000 7f629a200700 1 --2- 172.21.15.89:0/1807485775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62940d2370 0x7f62940ce360 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.527+0000 7f629a200700 1 --2- 172.21.15.89:0/1807485775 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62940d16a0 0x7f62940d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.527+0000 7f629a200700 1 -- 172.21.15.89:0/1807485775 >> 172.21.15.89:0/1807485775 conn(0x7f629408aff0 msgr2=0x7f62940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.527+0000 7f629a200700 1 -- 172.21.15.89:0/1807485775 shutdown_connections 2024-04-02T07:48:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.527+0000 7f629a200700 1 -- 172.21.15.89:0/1807485775 wait complete. 2024-04-02T07:48:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.529+0000 7f629a200700 1 Processor -- start 2024-04-02T07:48:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.529+0000 7f629a200700 1 -- start start 2024-04-02T07:48:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f629a200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62940d16a0 0x7f62940b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f629a200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62940d2370 0x7f6294158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f629a200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f629415d9d0 0x7f6294158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f629a200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62941e8470 con 0x7f629415d9d0 2024-04-02T07:48:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f629a200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6294159390 con 0x7f62940d16a0 2024-04-02T07:48:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f629a200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62941e8190 con 0x7f62940d2370 2024-04-02T07:48:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f6293e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f629415d9d0 0x7f6294158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f6293e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f629415d9d0 0x7f6294158e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51728/0 (socket says 172.21.15.89:51728) 2024-04-02T07:48:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f6293e00700 1 -- 172.21.15.89:0/150658421 learned_addr learned my addr 172.21.15.89:0/150658421 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f6292a00700 1 --2- 172.21.15.89:0/150658421 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62940d2370 0x7f6294158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f6293400700 1 --2- 172.21.15.89:0/150658421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62940d16a0 0x7f62940b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:48.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f6293e00700 1 -- 172.21.15.89:0/150658421 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62940d2370 msgr2=0x7f6294158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:48.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f6293e00700 1 --2- 172.21.15.89:0/150658421 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62940d2370 0x7f6294158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f6293e00700 1 -- 172.21.15.89:0/150658421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62940d16a0 msgr2=0x7f62940b4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:48.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f6293e00700 1 --2- 172.21.15.89:0/150658421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62940d16a0 0x7f62940b4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.530+0000 7f6293e00700 1 -- 172.21.15.89:0/150658421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c009a70 con 0x7f629415d9d0 2024-04-02T07:48:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.531+0000 7f6293e00700 1 --2- 172.21.15.89:0/150658421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f629415d9d0 0x7f6294158e50 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f629408c230 tx=0x7f628c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.531+0000 7f628b400700 1 -- 172.21.15.89:0/150658421 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c004550 con 0x7f629415d9d0 2024-04-02T07:48:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.531+0000 7f628b400700 1 -- 172.21.15.89:0/150658421 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628c020070 con 0x7f629415d9d0 2024-04-02T07:48:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.531+0000 7f629a200700 1 -- 172.21.15.89:0/150658421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629415df50 con 0x7f629415d9d0 2024-04-02T07:48:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.531+0000 7f628b400700 1 -- 172.21.15.89:0/150658421 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c004910 con 0x7f629415d9d0 2024-04-02T07:48:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.531+0000 7f629a200700 1 -- 172.21.15.89:0/150658421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f629415e510 con 0x7f629415d9d0 2024-04-02T07:48:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.532+0000 7f629a200700 1 -- 172.21.15.89:0/150658421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62940d1ca0 con 0x7f629415d9d0 2024-04-02T07:48:48.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.533+0000 7f628b400700 1 -- 172.21.15.89:0/150658421 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f628c012430 con 0x7f629415d9d0 2024-04-02T07:48:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.534+0000 7f628b400700 1 --2- 172.21.15.89:0/150658421 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f627406ae40 0x7f627406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.534+0000 7f6293400700 1 -- 172.21.15.89:0/150658421 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f627406ae40 msgr2=0x7f627406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.535+0000 7f628b400700 1 -- 172.21.15.89:0/150658421 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6779+0+0 (secure 0 0 0) 0x7f628c022070 con 0x7f629415d9d0 2024-04-02T07:48:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.535+0000 7f6293400700 1 --2- 172.21.15.89:0/150658421 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f627406ae40 0x7f627406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:48:48.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.539+0000 7f628b400700 1 -- 172.21.15.89:0/150658421 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f628c056ac0 con 0x7f629415d9d0 2024-04-02T07:48:48.680 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:48.678+0000 7f5ffe757240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:48:48.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.736+0000 7f6293400700 1 -- 172.21.15.89:0/150658421 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f627406ae40 msgr2=0x7f627406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:48.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.736+0000 7f6293400700 1 --2- 172.21.15.89:0/150658421 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f627406ae40 0x7f627406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:48:48.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.764+0000 7f629a200700 1 -- 172.21.15.89:0/150658421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6294067720 con 0x7f629415d9d0 2024-04-02T07:48:48.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.764+0000 7f628b400700 1 -- 172.21.15.89:0/150658421 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v52) v1 ==== 80+0+342 (secure 0 0 0) 0x7f628c05a0e0 con 0x7f629415d9d0 2024-04-02T07:48:48.767 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/43357 2024-04-03T07:37:22.446577+0000 2024-04-02T07:48:48.767 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2201396539 2024-04-03T07:37:22.446577+0000 2024-04-02T07:48:48.767 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/43357 2024-04-03T07:37:22.446577+0000 2024-04-02T07:48:48.767 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/350807093 2024-04-03T07:37:22.446577+0000 2024-04-02T07:48:48.767 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2645815547 2024-04-03T07:37:22.446577+0000 2024-04-02T07:48:48.767 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/399385398 2024-04-03T07:37:22.446577+0000 2024-04-02T07:48:48.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.767+0000 7f629a200700 1 -- 172.21.15.89:0/150658421 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f627406ae40 msgr2=0x7f627406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:48:48.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.767+0000 7f629a200700 1 --2- 172.21.15.89:0/150658421 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f627406ae40 0x7f627406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.767+0000 7f629a200700 1 -- 172.21.15.89:0/150658421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f629415d9d0 msgr2=0x7f6294158e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:48.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.767+0000 7f629a200700 1 --2- 172.21.15.89:0/150658421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f629415d9d0 0x7f6294158e50 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f629408c230 tx=0x7f628c004ad0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.767+0000 7f629a200700 1 -- 172.21.15.89:0/150658421 shutdown_connections 2024-04-02T07:48:48.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.767+0000 7f629a200700 1 --2- 172.21.15.89:0/150658421 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f627406ae40 0x7f627406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.767+0000 7f629a200700 1 --2- 172.21.15.89:0/150658421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62940d16a0 0x7f62940b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.767+0000 7f629a200700 1 --2- 172.21.15.89:0/150658421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f629415d9d0 0x7f6294158e50 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.767+0000 7f629a200700 1 --2- 172.21.15.89:0/150658421 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f62940d2370 0x7f6294158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:48.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.767+0000 7f629a200700 1 -- 172.21.15.89:0/150658421 >> 172.21.15.89:0/150658421 conn(0x7f629408aff0 msgr2=0x7f629415b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:48.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.767+0000 7f629a200700 1 -- 172.21.15.89:0/150658421 shutdown_connections 2024-04-02T07:48:48.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:48.767+0000 7f629a200700 1 -- 172.21.15.89:0/150658421 wait complete. 2024-04-02T07:48:48.771 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:48.768+0000 7f5ffe757240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:48:48.771 INFO:teuthology.orchestra.run.smithi089.stderr:listed 6 entries 2024-04-02T07:48:48.788 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T07:48:48.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:48.788 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T07:48:48.862 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:48:48.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:48:48.862 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T07:48:48.871 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:48.869+0000 7f0d3a67c240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:48:48.907 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:48:48.908 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:48:48.937 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:48.934+0000 7f5ffe757240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:48:48.961 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:48:48.961 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:48:48.962 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:48:48.969 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:48.967+0000 7f0d3a67c240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:48:49.006 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:48:49.006 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:48:49.007 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:48:49.055 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:48:49.055 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:48:49.148 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:49.145+0000 7f0d3a67c240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:48:49.158 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:49.155+0000 7f5ffe757240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:48:49.248 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:49.245+0000 7f5ffe757240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:48:49.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.252+0000 7f2671c00700 1 Processor -- start 2024-04-02T07:48:49.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.252+0000 7f2671c00700 1 -- start start 2024-04-02T07:48:49.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.253+0000 7f2671c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f266c09aa70 0x7f266c0cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:49.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.253+0000 7f2671c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f266c0cefc0 con 0x7f266c099a60 2024-04-02T07:48:49.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.253+0000 7f2671c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f266c0cf230 con 0x7f266c15b560 2024-04-02T07:48:49.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.253+0000 7f2671c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f266c0cf510 con 0x7f266c09aa70 2024-04-02T07:48:49.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.254+0000 7f266b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f266c099a60 0x7f266c0b41a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55660/0 (socket says 172.21.15.89:55660) 2024-04-02T07:48:49.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.254+0000 7f266b400700 1 -- 172.21.15.89:0/743158995 learned_addr learned my addr 172.21.15.89:0/743158995 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:49.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.254+0000 7f266a000700 1 --2- 172.21.15.89:0/743158995 >> v2:172.21.15.113:3300/0 conn(0x7f266c09aa70 0x7f266c0cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:49.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.254+0000 7f2669600700 1 -- 172.21.15.89:0/743158995 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166381650 0 0) 0x7f266c0cf230 con 0x7f266c15b560 2024-04-02T07:48:49.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.254+0000 7f266a000700 1 -- 172.21.15.89:0/743158995 >> v1:172.21.15.113:6789/0 conn(0x7f266c15b560 legacy=0x7f266c0cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:49.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.254+0000 7f266a000700 1 -- 172.21.15.89:0/743158995 >> v1:172.21.15.89:6789/0 conn(0x7f266c099a60 legacy=0x7f266c0b41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:49.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.254+0000 7f266a000700 1 -- 172.21.15.89:0/743158995 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266c0cf7f0 con 0x7f266c09aa70 2024-04-02T07:48:49.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.254+0000 7f266a000700 1 --2- 172.21.15.89:0/743158995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f266c09aa70 0x7f266c0cfd10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2658009840 tx=0x7f2658009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67346ae727dccde1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:49.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.255+0000 7f2669600700 1 -- 172.21.15.89:0/743158995 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2658004230 con 0x7f266c09aa70 2024-04-02T07:48:49.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.255+0000 7f2669600700 1 -- 172.21.15.89:0/743158995 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2658004390 con 0x7f266c09aa70 2024-04-02T07:48:49.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.255+0000 7f2669600700 1 -- 172.21.15.89:0/743158995 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26580036a0 con 0x7f266c09aa70 2024-04-02T07:48:49.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.255+0000 7f2671c00700 1 -- 172.21.15.89:0/743158995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f266c09aa70 msgr2=0x7f266c0cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:49.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.255+0000 7f2671c00700 1 --2- 172.21.15.89:0/743158995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f266c09aa70 0x7f266c0cfd10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2658009840 tx=0x7f2658009b50 comp rx=0 tx=0).stop 2024-04-02T07:48:49.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.255+0000 7f2671c00700 1 -- 172.21.15.89:0/743158995 shutdown_connections 2024-04-02T07:48:49.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.255+0000 7f2671c00700 1 --2- 172.21.15.89:0/743158995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f266c09aa70 0x7f266c0cfd10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.255+0000 7f2671c00700 1 -- 172.21.15.89:0/743158995 >> 172.21.15.89:0/743158995 conn(0x7f266c0d1350 msgr2=0x7f266c0d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:49.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.256+0000 7f2671c00700 1 -- 172.21.15.89:0/743158995 shutdown_connections 2024-04-02T07:48:49.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.256+0000 7f2671c00700 1 -- 172.21.15.89:0/743158995 wait complete. 2024-04-02T07:48:49.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.257+0000 7f2671c00700 1 Processor -- start 2024-04-02T07:48:49.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.257+0000 7f2671c00700 1 -- start start 2024-04-02T07:48:49.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.258+0000 7f2671c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f266c099a60 0x7f266c1f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:49.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.258+0000 7f2671c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f266c09aa70 0x7f266c0c6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:49.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.258+0000 7f2671c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f266c15b560 0x7f266c155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:49.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.258+0000 7f2671c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c1514d0 con 0x7f266c09aa70 2024-04-02T07:48:49.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.258+0000 7f2671c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c0b67e0 con 0x7f266c099a60 2024-04-02T07:48:49.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.258+0000 7f2671c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f266c151740 con 0x7f266c15b560 2024-04-02T07:48:49.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.259+0000 7f266a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f266c09aa70 0x7f266c0c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:49.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.259+0000 7f266a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f266c09aa70 0x7f266c0c6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58300/0 (socket says 172.21.15.89:58300) 2024-04-02T07:48:49.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.259+0000 7f266a000700 1 -- 172.21.15.89:0/747112673 learned_addr learned my addr 172.21.15.89:0/747112673 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:49.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.259+0000 7f266aa00700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f266c099a60 0x7f266c1f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:49.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.259+0000 7f266b400700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f266c15b560 0x7f266c155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:49.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.259+0000 7f266a000700 1 -- 172.21.15.89:0/747112673 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f266c15b560 msgr2=0x7f266c155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:49.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.259+0000 7f266a000700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f266c15b560 0x7f266c155ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.259+0000 7f266a000700 1 -- 172.21.15.89:0/747112673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f266c099a60 msgr2=0x7f266c1f3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:49.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.259+0000 7f266a000700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f266c099a60 0x7f266c1f3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.259+0000 7f266a000700 1 -- 172.21.15.89:0/747112673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26580094a0 con 0x7f266c09aa70 2024-04-02T07:48:49.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.259+0000 7f266aa00700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f266c099a60 0x7f266c1f3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:48:49.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.259+0000 7f266a000700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f266c09aa70 0x7f266c0c6880 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f26580090b0 tx=0x7f2658004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:49.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.260+0000 7f2662a00700 1 -- 172.21.15.89:0/747112673 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2658011630 con 0x7f266c09aa70 2024-04-02T07:48:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.260+0000 7f2662a00700 1 -- 172.21.15.89:0/747112673 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265801f070 con 0x7f266c09aa70 2024-04-02T07:48:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.260+0000 7f2662a00700 1 -- 172.21.15.89:0/747112673 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2658011a10 con 0x7f266c09aa70 2024-04-02T07:48:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.260+0000 7f2671c00700 1 -- 172.21.15.89:0/747112673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266c151a20 con 0x7f266c09aa70 2024-04-02T07:48:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.260+0000 7f2671c00700 1 -- 172.21.15.89:0/747112673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c151fe0 con 0x7f266c09aa70 2024-04-02T07:48:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.261+0000 7f2671c00700 1 -- 172.21.15.89:0/747112673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266c067720 con 0x7f266c09aa70 2024-04-02T07:48:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.262+0000 7f2662a00700 1 -- 172.21.15.89:0/747112673 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f2658011b70 con 0x7f266c09aa70 2024-04-02T07:48:49.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.263+0000 7f2662a00700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f264c06ae40 0x7f264c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:49.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.263+0000 7f2662a00700 1 -- 172.21.15.89:0/747112673 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6475+0+0 (secure 0 0 0) 0x7f2658021070 con 0x7f266c09aa70 2024-04-02T07:48:49.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.263+0000 7f266aa00700 1 -- 172.21.15.89:0/747112673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f264c06ae40 msgr2=0x7f264c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:49.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.263+0000 7f266aa00700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f264c06ae40 0x7f264c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:48:49.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.267+0000 7f2662a00700 1 -- 172.21.15.89:0/747112673 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2658055f90 con 0x7f266c09aa70 2024-04-02T07:48:49.345 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:49.343+0000 7f5ffe757240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:48:49.364 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:49.362+0000 7f0d3a67c240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:48:49.441 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:49.439+0000 7f5ffe757240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:48:49.456 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:49.454+0000 7f0d3a67c240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:48:49.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.464+0000 7f266aa00700 1 -- 172.21.15.89:0/747112673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f264c06ae40 msgr2=0x7f264c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:49.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.464+0000 7f266aa00700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f264c06ae40 0x7f264c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:48:49.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.523+0000 7f2671c00700 1 -- 172.21.15.89:0/747112673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f266c0b26d0 con 0x7f266c09aa70 2024-04-02T07:48:49.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.523+0000 7f2662a00700 1 -- 172.21.15.89:0/747112673 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f26580595b0 con 0x7f266c09aa70 2024-04-02T07:48:49.526 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:49.526 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5037,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3788023013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3788023013},{"type":"v1","addr":"172.21.15.89:6835","nonce":3788023013}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5040,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4029308904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":4029308904},{"type":"v1","addr":"172.21.15.113:6835","nonce":4029308904}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5043,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2115217020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2115217020},{"type":"v1","addr":"172.21.15.113:6834","nonce":2115217020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:37:05.500422+0000","modified":"2024-04-02T07:37:08.737868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.89:6837/929002087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":929002087},{"type":"v1","addr":"172.21.15.89:6837","nonce":929002087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-02T07:48:49.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.526+0000 7f2671c00700 1 -- 172.21.15.89:0/747112673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f264c06ae40 msgr2=0x7f264c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:48:49.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.526+0000 7f2671c00700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f264c06ae40 0x7f264c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.526+0000 7f2671c00700 1 -- 172.21.15.89:0/747112673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f266c09aa70 msgr2=0x7f266c0c6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:49.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.526+0000 7f2671c00700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f266c09aa70 0x7f266c0c6880 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f26580090b0 tx=0x7f2658004750 comp rx=0 tx=0).stop 2024-04-02T07:48:49.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.526+0000 7f2671c00700 1 -- 172.21.15.89:0/747112673 shutdown_connections 2024-04-02T07:48:49.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.526+0000 7f2671c00700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f264c06ae40 0x7f264c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.526+0000 7f2671c00700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f266c099a60 0x7f266c1f3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.526+0000 7f2671c00700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f266c15b560 0x7f266c155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.526+0000 7f2671c00700 1 --2- 172.21.15.89:0/747112673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f266c09aa70 0x7f266c0c6880 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.526+0000 7f2671c00700 1 -- 172.21.15.89:0/747112673 >> 172.21.15.89:0/747112673 conn(0x7f266c0d1350 msgr2=0x7f266c0cc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:49.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.526+0000 7f2671c00700 1 -- 172.21.15.89:0/747112673 shutdown_connections 2024-04-02T07:48:49.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.526+0000 7f2671c00700 1 -- 172.21.15.89:0/747112673 wait complete. 2024-04-02T07:48:49.531 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 17 2024-04-02T07:48:49.543 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:49.540+0000 7f5ffe757240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:48:49.551 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:48:49.566 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:49.564+0000 7f0d3a67c240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:48:49.672 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:49.669+0000 7f0d3a67c240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:48:49.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.747+0000 7f7915000700 1 Processor -- start 2024-04-02T07:48:49.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.747+0000 7f7915000700 1 -- start start 2024-04-02T07:48:49.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.749+0000 7f7915000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7910160be0 0x7f79100cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.749+0000 7f7915000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f79100ce0e0 0x7f79100ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.749+0000 7f7915000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7910160fc0 con 0x7f79100cead0 2024-04-02T07:48:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.749+0000 7f7915000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79100b65d0 con 0x7f79100ce0e0 2024-04-02T07:48:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.749+0000 7f7915000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79100d2a80 con 0x7f7910160be0 2024-04-02T07:48:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.749+0000 7f790d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f79100ce0e0 0x7f79100ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.749+0000 7f790de00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7910160be0 0x7f79100cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.749+0000 7f790e800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f79100cead0 0x7f7910161240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55676/0 (socket says 172.21.15.89:55676) 2024-04-02T07:48:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.749+0000 7f790d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f79100ce0e0 0x7f79100ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58310/0 (socket says 172.21.15.89:58310) 2024-04-02T07:48:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.749+0000 7f790de00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7910160be0 0x7f79100cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:44620/0 (socket says 172.21.15.89:44620) 2024-04-02T07:48:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.750+0000 7f790e800700 1 -- 172.21.15.89:0/2951725721 learned_addr learned my addr 172.21.15.89:0/2951725721 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.750+0000 7f790d400700 1 -- 172.21.15.89:0/2951725721 >> v2:172.21.15.113:3300/0 conn(0x7f7910160be0 msgr2=0x7f79100cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.750+0000 7f790d400700 1 --2- 172.21.15.89:0/2951725721 >> v2:172.21.15.113:3300/0 conn(0x7f7910160be0 0x7f79100cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.750+0000 7f790d400700 1 -- 172.21.15.89:0/2951725721 >> v1:172.21.15.89:6789/0 conn(0x7f79100cead0 legacy=0x7f7910161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.750+0000 7f790d400700 1 -- 172.21.15.89:0/2951725721 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79100d21e0 con 0x7f79100ce0e0 2024-04-02T07:48:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.750+0000 7f790ca00700 1 -- 172.21.15.89:0/2951725721 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855115754 0 0) 0x7f7910160fc0 con 0x7f79100cead0 2024-04-02T07:48:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.750+0000 7f790d400700 1 --2- 172.21.15.89:0/2951725721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79100ce0e0 0x7f79100ce590 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f791008b590 tx=0x7f78fc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea9db1588386720 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.750+0000 7f790ca00700 1 -- 172.21.15.89:0/2951725721 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc011830 con 0x7f79100ce0e0 2024-04-02T07:48:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.750+0000 7f790ca00700 1 -- 172.21.15.89:0/2951725721 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78fc011990 con 0x7f79100ce0e0 2024-04-02T07:48:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.750+0000 7f790ca00700 1 -- 172.21.15.89:0/2951725721 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc011d20 con 0x7f79100ce0e0 2024-04-02T07:48:49.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.751+0000 7f7915000700 1 -- 172.21.15.89:0/2951725721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79100ce0e0 msgr2=0x7f79100ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.751+0000 7f7915000700 1 --2- 172.21.15.89:0/2951725721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79100ce0e0 0x7f79100ce590 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f791008b590 tx=0x7f78fc009b20 comp rx=0 tx=0).stop 2024-04-02T07:48:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.751+0000 7f7915000700 1 -- 172.21.15.89:0/2951725721 shutdown_connections 2024-04-02T07:48:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.751+0000 7f7915000700 1 --2- 172.21.15.89:0/2951725721 >> v2:172.21.15.113:3300/0 conn(0x7f7910160be0 0x7f79100cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.751+0000 7f7915000700 1 --2- 172.21.15.89:0/2951725721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79100ce0e0 0x7f79100ce590 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.751+0000 7f7915000700 1 -- 172.21.15.89:0/2951725721 >> 172.21.15.89:0/2951725721 conn(0x7f791008aff0 msgr2=0x7f79100d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.751+0000 7f7915000700 1 -- 172.21.15.89:0/2951725721 shutdown_connections 2024-04-02T07:48:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.751+0000 7f7915000700 1 -- 172.21.15.89:0/2951725721 wait complete. 2024-04-02T07:48:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.753+0000 7f7915000700 1 Processor -- start 2024-04-02T07:48:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.754+0000 7f7915000700 1 -- start start 2024-04-02T07:48:49.757 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:49.753+0000 7f5ffe757240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:48:49.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.755+0000 7f7915000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79100ce0e0 0x7f79100c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:49.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.755+0000 7f7915000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79100cead0 0x7f79100c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:49.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.755+0000 7f7915000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7910160be0 0x7f79100ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:49.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.755+0000 7f7915000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79101f3cf0 con 0x7f79100cead0 2024-04-02T07:48:49.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.755+0000 7f7915000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79101f6780 con 0x7f7910160be0 2024-04-02T07:48:49.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.755+0000 7f7915000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79101f6a60 con 0x7f79100ce0e0 2024-04-02T07:48:49.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.755+0000 7f790d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79100cead0 0x7f79100c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:49.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.755+0000 7f790d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79100cead0 0x7f79100c89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58322/0 (socket says 172.21.15.89:58322) 2024-04-02T07:48:49.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.755+0000 7f790d400700 1 -- 172.21.15.89:0/768555631 learned_addr learned my addr 172.21.15.89:0/768555631 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:49.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.755+0000 7f790de00700 1 --2- 172.21.15.89:0/768555631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79100ce0e0 0x7f79100c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:49.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.755+0000 7f790e800700 1 --2- 172.21.15.89:0/768555631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7910160be0 0x7f79100ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:49.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.756+0000 7f790d400700 1 -- 172.21.15.89:0/768555631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79100ce0e0 msgr2=0x7f79100c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:49.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.756+0000 7f790d400700 1 --2- 172.21.15.89:0/768555631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79100ce0e0 0x7f79100c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.756+0000 7f790d400700 1 -- 172.21.15.89:0/768555631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7910160be0 msgr2=0x7f79100ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:49.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.756+0000 7f790d400700 1 --2- 172.21.15.89:0/768555631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7910160be0 0x7f79100ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.756+0000 7f790d400700 1 -- 172.21.15.89:0/768555631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc0094a0 con 0x7f79100cead0 2024-04-02T07:48:49.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.756+0000 7f790d400700 1 --2- 172.21.15.89:0/768555631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79100cead0 0x7f79100c89a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009b60 tx=0x7f78fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:49.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.756+0000 7f7906000700 1 -- 172.21.15.89:0/768555631 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc004550 con 0x7f79100cead0 2024-04-02T07:48:49.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.756+0000 7f7906000700 1 -- 172.21.15.89:0/768555631 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78fc01f070 con 0x7f79100cead0 2024-04-02T07:48:49.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.756+0000 7f7906000700 1 -- 172.21.15.89:0/768555631 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc0038e0 con 0x7f79100cead0 2024-04-02T07:48:49.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.756+0000 7f7915000700 1 -- 172.21.15.89:0/768555631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79101f6d40 con 0x7f79100cead0 2024-04-02T07:48:49.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.756+0000 7f7915000700 1 -- 172.21.15.89:0/768555631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79101f3fd0 con 0x7f79100cead0 2024-04-02T07:48:49.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.757+0000 7f7915000700 1 -- 172.21.15.89:0/768555631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7910067720 con 0x7f79100cead0 2024-04-02T07:48:49.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.758+0000 7f7906000700 1 -- 172.21.15.89:0/768555631 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f78fc003a40 con 0x7f79100cead0 2024-04-02T07:48:49.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.759+0000 7f7906000700 1 --2- 172.21.15.89:0/768555631 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f78f006ae40 0x7f78f006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:49.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.759+0000 7f790de00700 1 -- 172.21.15.89:0/768555631 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f78f006ae40 msgr2=0x7f78f006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:49.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.759+0000 7f7906000700 1 -- 172.21.15.89:0/768555631 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6475+0+0 (secure 0 0 0) 0x7f78fc021070 con 0x7f79100cead0 2024-04-02T07:48:49.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.759+0000 7f790de00700 1 --2- 172.21.15.89:0/768555631 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f78f006ae40 0x7f78f006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:48:49.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.763+0000 7f7906000700 1 -- 172.21.15.89:0/768555631 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f78fc055e20 con 0x7f79100cead0 2024-04-02T07:48:49.775 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:49.773+0000 7f0d3a67c240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:48:49.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.960+0000 7f790de00700 1 -- 172.21.15.89:0/768555631 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f78f006ae40 msgr2=0x7f78f006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:49.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.960+0000 7f790de00700 1 --2- 172.21.15.89:0/768555631 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f78f006ae40 0x7f78f006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:48:49.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.976+0000 7f7915000700 1 -- 172.21.15.89:0/768555631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79100b5cb0 con 0x7f79100cead0 2024-04-02T07:48:49.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.978+0000 7f7906000700 1 -- 172.21.15.89:0/768555631 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+17277 (secure 0 0 0) 0x7f78fc015090 con 0x7f79100cead0 2024-04-02T07:48:49.980 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:49.980 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":53,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:48:48.910822+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:37:03.665694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-02T07:37:04.473147+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.19","mappings":[{"from":4,"to":5}]},{"pgid":"5.1a","mappings":[{"from":4,"to":5}]},{"pgid":"5.1d","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6833/43357":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/2201396539":"2024-04-03T07:37:22.446577+0000","172.21.15.89:6832/43357":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/350807093":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/2645815547":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/399385398":"2024-04-03T07:37:22.446577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":105,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:48:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.980+0000 7f7915000700 1 -- 172.21.15.89:0/768555631 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f78f006ae40 msgr2=0x7f78f006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:48:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.980+0000 7f7915000700 1 --2- 172.21.15.89:0/768555631 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f78f006ae40 0x7f78f006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.980+0000 7f7915000700 1 -- 172.21.15.89:0/768555631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79100cead0 msgr2=0x7f79100c89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.980+0000 7f7915000700 1 --2- 172.21.15.89:0/768555631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79100cead0 0x7f79100c89a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009b60 tx=0x7f78fc004750 comp rx=0 tx=0).stop 2024-04-02T07:48:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.980+0000 7f7915000700 1 -- 172.21.15.89:0/768555631 shutdown_connections 2024-04-02T07:48:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.980+0000 7f7915000700 1 --2- 172.21.15.89:0/768555631 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f78f006ae40 0x7f78f006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.980+0000 7f7915000700 1 --2- 172.21.15.89:0/768555631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7910160be0 0x7f79100ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.980+0000 7f7915000700 1 --2- 172.21.15.89:0/768555631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79100ce0e0 0x7f79100c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.980+0000 7f7915000700 1 --2- 172.21.15.89:0/768555631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79100cead0 0x7f79100c89a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.980+0000 7f7915000700 1 -- 172.21.15.89:0/768555631 >> 172.21.15.89:0/768555631 conn(0x7f791008aff0 msgr2=0x7f79100b7410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.980+0000 7f7915000700 1 -- 172.21.15.89:0/768555631 shutdown_connections 2024-04-02T07:48:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:49.980+0000 7f7915000700 1 -- 172.21.15.89:0/768555631 wait complete. 2024-04-02T07:48:50.002 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:49.999+0000 7f0d3a67c240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:48:50.008 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:50.006+0000 7f5ffe757240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:48:50.010 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:37:03.665694+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:37:04.473147+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 221, 'snap_epoch': 53, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:48:50.010 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:48:50.195 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:50.192+0000 7f5ffe757240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:48:50.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.254+0000 7f0b81600700 1 Processor -- start 2024-04-02T07:48:50.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.254+0000 7f0b81600700 1 -- start start 2024-04-02T07:48:50.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.255+0000 7f0b81600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b7c0cbb30 0x7f0b7c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:50.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.255+0000 7f0b81600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0b7c0cd480 0x7f0b7c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:50.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.255+0000 7f0b81600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c0cc450 con 0x7f0b7c0cd020 2024-04-02T07:48:50.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.255+0000 7f0b81600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7c0cd8f0 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.255+0000 7f0b81600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7c0cdae0 con 0x7f0b7c0cd480 2024-04-02T07:48:50.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.255+0000 7f0b7aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b7c0cbb30 0x7f0b7c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:50.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.255+0000 7f0b7aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b7c0cbb30 0x7f0b7c0cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58326/0 (socket says 172.21.15.89:58326) 2024-04-02T07:48:50.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.255+0000 7f0b7aa00700 1 -- 172.21.15.89:0/2774477558 learned_addr learned my addr 172.21.15.89:0/2774477558 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:50.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.255+0000 7f0b7b400700 1 --2- 172.21.15.89:0/2774477558 >> v2:172.21.15.113:3300/0 conn(0x7f0b7c0cd480 0x7f0b7c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:50.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.256+0000 7f0b7aa00700 1 -- 172.21.15.89:0/2774477558 >> v2:172.21.15.113:3300/0 conn(0x7f0b7c0cd480 msgr2=0x7f0b7c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:50.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.256+0000 7f0b7aa00700 1 --2- 172.21.15.89:0/2774477558 >> v2:172.21.15.113:3300/0 conn(0x7f0b7c0cd480 0x7f0b7c06e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.256+0000 7f0b7aa00700 1 -- 172.21.15.89:0/2774477558 >> v1:172.21.15.113:6789/0 conn(0x7f0b7c0cd020 legacy=0x7f0b7c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:50.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.256+0000 7f0b7aa00700 1 -- 172.21.15.89:0/2774477558 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c0cddc0 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.256+0000 7f0b7aa00700 1 --2- 172.21.15.89:0/2774477558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b7c0cbb30 0x7f0b7c0cbf10 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c06b6d0 tx=0x7f0b6c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba201ddc3c934359 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:50.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.256+0000 7f0b79600700 1 -- 172.21.15.89:0/2774477558 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c011830 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.256+0000 7f0b79600700 1 -- 172.21.15.89:0/2774477558 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6c011990 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.256+0000 7f0b79600700 1 -- 172.21.15.89:0/2774477558 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c011d20 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.257+0000 7f0b81600700 1 -- 172.21.15.89:0/2774477558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b7c0cbb30 msgr2=0x7f0b7c0cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:50.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.257+0000 7f0b81600700 1 --2- 172.21.15.89:0/2774477558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b7c0cbb30 0x7f0b7c0cbf10 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c06b6d0 tx=0x7f0b6c009bd0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.257+0000 7f0b81600700 1 -- 172.21.15.89:0/2774477558 shutdown_connections 2024-04-02T07:48:50.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.257+0000 7f0b81600700 1 --2- 172.21.15.89:0/2774477558 >> v2:172.21.15.113:3300/0 conn(0x7f0b7c0cd480 0x7f0b7c06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.257+0000 7f0b81600700 1 --2- 172.21.15.89:0/2774477558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b7c0cbb30 0x7f0b7c0cbf10 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.257+0000 7f0b81600700 1 -- 172.21.15.89:0/2774477558 >> 172.21.15.89:0/2774477558 conn(0x7f0b7c0b0d70 msgr2=0x7f0b7c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:50.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.257+0000 7f0b81600700 1 -- 172.21.15.89:0/2774477558 shutdown_connections 2024-04-02T07:48:50.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.257+0000 7f0b81600700 1 -- 172.21.15.89:0/2774477558 wait complete. 2024-04-02T07:48:50.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.259+0000 7f0b81600700 1 Processor -- start 2024-04-02T07:48:50.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.259+0000 7f0b81600700 1 -- start start 2024-04-02T07:48:50.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b81600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b7c0cbb30 0x7f0b7c1e43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:50.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b81600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b7c0cd020 0x7f0b7c09b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:50.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b81600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b7c0cd480 0x7f0b7c155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:50.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b81600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c09b780 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b81600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c09ba20 con 0x7f0b7c0cd480 2024-04-02T07:48:50.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b81600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c09bd00 con 0x7f0b7c0cd020 2024-04-02T07:48:50.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b7aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b7c0cbb30 0x7f0b7c1e43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:50.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b7aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b7c0cbb30 0x7f0b7c1e43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58334/0 (socket says 172.21.15.89:58334) 2024-04-02T07:48:50.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b7aa00700 1 -- 172.21.15.89:0/3439940362 learned_addr learned my addr 172.21.15.89:0/3439940362 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:50.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b7a000700 1 --2- 172.21.15.89:0/3439940362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b7c0cd020 0x7f0b7c09b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:50.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b7b400700 1 --2- 172.21.15.89:0/3439940362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b7c0cd480 0x7f0b7c155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:50.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b7aa00700 1 -- 172.21.15.89:0/3439940362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b7c0cd020 msgr2=0x7f0b7c09b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:50.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b7aa00700 1 --2- 172.21.15.89:0/3439940362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b7c0cd020 0x7f0b7c09b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b7aa00700 1 -- 172.21.15.89:0/3439940362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b7c0cd480 msgr2=0x7f0b7c155470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:50.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b7aa00700 1 --2- 172.21.15.89:0/3439940362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b7c0cd480 0x7f0b7c155470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.260+0000 7f0b7aa00700 1 -- 172.21.15.89:0/3439940362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c009550 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.261+0000 7f0b7aa00700 1 --2- 172.21.15.89:0/3439940362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b7c0cbb30 0x7f0b7c1e43b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c009c10 tx=0x7f0b6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:50.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.261+0000 7f0b72a00700 1 -- 172.21.15.89:0/3439940362 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c004510 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.261+0000 7f0b72a00700 1 -- 172.21.15.89:0/3439940362 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6c01e070 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.261+0000 7f0b81600700 1 -- 172.21.15.89:0/3439940362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c09bfe0 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.261+0000 7f0b72a00700 1 -- 172.21.15.89:0/3439940362 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c0038e0 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.261+0000 7f0b81600700 1 -- 172.21.15.89:0/3439940362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c09c5a0 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.262+0000 7f0b81600700 1 -- 172.21.15.89:0/3439940362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c067720 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.263+0000 7f0b72a00700 1 -- 172.21.15.89:0/3439940362 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f0b6c003a40 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.265+0000 7f0b72a00700 1 --2- 172.21.15.89:0/3439940362 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b5c06ae40 0x7f0b5c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:50.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.265+0000 7f0b7a000700 1 -- 172.21.15.89:0/3439940362 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b5c06ae40 msgr2=0x7f0b5c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:50.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.265+0000 7f0b72a00700 1 -- 172.21.15.89:0/3439940362 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6475+0+0 (secure 0 0 0) 0x7f0b6c08a740 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.265+0000 7f0b7a000700 1 --2- 172.21.15.89:0/3439940362 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b5c06ae40 0x7f0b5c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:48:50.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.269+0000 7f0b72a00700 1 -- 172.21.15.89:0/3439940362 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b6c055390 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.274 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:50.272+0000 7f0d3a67c240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:48:50.286 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:50.284+0000 7f5ffe757240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:48:50.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.466+0000 7f0b7a000700 1 -- 172.21.15.89:0/3439940362 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b5c06ae40 msgr2=0x7f0b5c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:50.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.466+0000 7f0b7a000700 1 --2- 172.21.15.89:0/3439940362 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b5c06ae40 0x7f0b5c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:48:50.470 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:50.469+0000 7f0d3a67c240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:48:50.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.478+0000 7f0b81600700 1 -- 172.21.15.89:0/3439940362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0b7c0ce2e0 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.478+0000 7f0b72a00700 1 -- 172.21.15.89:0/3439940362 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0b6c0589b0 con 0x7f0b7c0cbb30 2024-04-02T07:48:50.480 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:48:50.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.481+0000 7f0b81600700 1 -- 172.21.15.89:0/3439940362 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b5c06ae40 msgr2=0x7f0b5c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:48:50.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.481+0000 7f0b81600700 1 --2- 172.21.15.89:0/3439940362 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b5c06ae40 0x7f0b5c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.481+0000 7f0b81600700 1 -- 172.21.15.89:0/3439940362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b7c0cbb30 msgr2=0x7f0b7c1e43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:50.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.481+0000 7f0b81600700 1 --2- 172.21.15.89:0/3439940362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b7c0cbb30 0x7f0b7c1e43b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c009c10 tx=0x7f0b6c004750 comp rx=0 tx=0).stop 2024-04-02T07:48:50.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.481+0000 7f0b81600700 1 -- 172.21.15.89:0/3439940362 shutdown_connections 2024-04-02T07:48:50.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.481+0000 7f0b81600700 1 --2- 172.21.15.89:0/3439940362 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b5c06ae40 0x7f0b5c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.481+0000 7f0b81600700 1 --2- 172.21.15.89:0/3439940362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b7c0cd480 0x7f0b7c155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.481+0000 7f0b81600700 1 --2- 172.21.15.89:0/3439940362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b7c0cbb30 0x7f0b7c1e43b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.481+0000 7f0b81600700 1 --2- 172.21.15.89:0/3439940362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b7c0cd020 0x7f0b7c09b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.481+0000 7f0b81600700 1 -- 172.21.15.89:0/3439940362 >> 172.21.15.89:0/3439940362 conn(0x7f0b7c0b0d70 msgr2=0x7f0b7c0b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:50.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.481+0000 7f0b81600700 1 -- 172.21.15.89:0/3439940362 shutdown_connections 2024-04-02T07:48:50.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.481+0000 7f0b81600700 1 -- 172.21.15.89:0/3439940362 wait complete. 2024-04-02T07:48:50.502 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:48:50.577 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:50.574+0000 7f0d3a67c240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:48:50.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.704+0000 7f69ff600700 1 Processor -- start 2024-04-02T07:48:50.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.705+0000 7f69ff600700 1 -- start start 2024-04-02T07:48:50.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.706+0000 7f69ff600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f69f815ef70 0x7f69f80b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:50.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.706+0000 7f69ff600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f8161150 con 0x7f69f806a790 2024-04-02T07:48:50.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.706+0000 7f69ff600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f806dba0 con 0x7f69f8160d70 2024-04-02T07:48:50.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.706+0000 7f69ff600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f80b4cd0 con 0x7f69f815ef70 2024-04-02T07:48:50.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.706+0000 7f69f7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f69f815ef70 0x7f69f80b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:50.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.706+0000 7f69fd600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f69f806a790 0x7f69f806e080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55682/0 (socket says 172.21.15.89:55682) 2024-04-02T07:48:50.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.706+0000 7f69f7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f69f815ef70 0x7f69f80b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58346/0 (socket says 172.21.15.89:58346) 2024-04-02T07:48:50.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.706+0000 7f69fd600700 1 -- 172.21.15.89:0/2467137394 learned_addr learned my addr 172.21.15.89:0/2467137394 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:50.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.707+0000 7f69f7e00700 1 -- 172.21.15.89:0/2467137394 >> v1:172.21.15.113:6789/0 conn(0x7f69f8160d70 legacy=0x7f69f815e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:50.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.707+0000 7f69f7e00700 1 -- 172.21.15.89:0/2467137394 >> v1:172.21.15.89:6789/0 conn(0x7f69f806a790 legacy=0x7f69f806e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:50.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.707+0000 7f69f7e00700 1 -- 172.21.15.89:0/2467137394 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f80b4ee0 con 0x7f69f815ef70 2024-04-02T07:48:50.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.707+0000 7f69f7e00700 1 --2- 172.21.15.89:0/2467137394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f815ef70 0x7f69f80b3200 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009840 tx=0x7f69e4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb84a09b12ba4b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:50.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.707+0000 7f69f7400700 1 -- 172.21.15.89:0/2467137394 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e4011830 con 0x7f69f815ef70 2024-04-02T07:48:50.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.707+0000 7f69f7400700 1 -- 172.21.15.89:0/2467137394 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e4011990 con 0x7f69f815ef70 2024-04-02T07:48:50.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.707+0000 7f69f7400700 1 -- 172.21.15.89:0/2467137394 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e4011d20 con 0x7f69f815ef70 2024-04-02T07:48:50.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.708+0000 7f69ff600700 1 -- 172.21.15.89:0/2467137394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f815ef70 msgr2=0x7f69f80b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:50.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.708+0000 7f69ff600700 1 --2- 172.21.15.89:0/2467137394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f815ef70 0x7f69f80b3200 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009840 tx=0x7f69e4009b50 comp rx=0 tx=0).stop 2024-04-02T07:48:50.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.708+0000 7f69ff600700 1 -- 172.21.15.89:0/2467137394 shutdown_connections 2024-04-02T07:48:50.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.708+0000 7f69ff600700 1 --2- 172.21.15.89:0/2467137394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f815ef70 0x7f69f80b3200 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.708+0000 7f69ff600700 1 -- 172.21.15.89:0/2467137394 >> 172.21.15.89:0/2467137394 conn(0x7f69f80b5720 msgr2=0x7f69f80b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:50.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.708+0000 7f69ff600700 1 -- 172.21.15.89:0/2467137394 shutdown_connections 2024-04-02T07:48:50.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.708+0000 7f69ff600700 1 -- 172.21.15.89:0/2467137394 wait complete. 2024-04-02T07:48:50.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.710+0000 7f69ff600700 1 Processor -- start 2024-04-02T07:48:50.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.710+0000 7f69ff600700 1 -- start start 2024-04-02T07:48:50.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69ff600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69f806a790 0x7f69f81f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:50.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69ff600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f815ef70 0x7f69f8155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:50.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69ff600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69f8160d70 0x7f69f8155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:50.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69ff600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f8153850 con 0x7f69f815ef70 2024-04-02T07:48:50.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69ff600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f8158d10 con 0x7f69f806a790 2024-04-02T07:48:50.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69ff600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f8158ff0 con 0x7f69f8160d70 2024-04-02T07:48:50.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69f7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f815ef70 0x7f69f8155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:50.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69f7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f815ef70 0x7f69f8155a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58354/0 (socket says 172.21.15.89:58354) 2024-04-02T07:48:50.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69f7e00700 1 -- 172.21.15.89:0/1963466048 learned_addr learned my addr 172.21.15.89:0/1963466048 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:50.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69fcc00700 1 --2- 172.21.15.89:0/1963466048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69f806a790 0x7f69f81f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:50.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69fd600700 1 --2- 172.21.15.89:0/1963466048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69f8160d70 0x7f69f8155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:50.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69f7e00700 1 -- 172.21.15.89:0/1963466048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69f8160d70 msgr2=0x7f69f8155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:50.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69f7e00700 1 --2- 172.21.15.89:0/1963466048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69f8160d70 0x7f69f8155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69f7e00700 1 -- 172.21.15.89:0/1963466048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69f806a790 msgr2=0x7f69f81f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:50.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69f7e00700 1 --2- 172.21.15.89:0/1963466048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69f806a790 0x7f69f81f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.711+0000 7f69f7e00700 1 -- 172.21.15.89:0/1963466048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e40094a0 con 0x7f69f815ef70 2024-04-02T07:48:50.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.712+0000 7f69f7e00700 1 --2- 172.21.15.89:0/1963466048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f815ef70 0x7f69f8155a90 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f69e40090b0 tx=0x7f69e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:50.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.712+0000 7f69f4c00700 1 -- 172.21.15.89:0/1963466048 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e40038a0 con 0x7f69f815ef70 2024-04-02T07:48:50.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.712+0000 7f69f4c00700 1 -- 172.21.15.89:0/1963466048 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e401f070 con 0x7f69f815ef70 2024-04-02T07:48:50.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.712+0000 7f69f4c00700 1 -- 172.21.15.89:0/1963466048 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e4003c60 con 0x7f69f815ef70 2024-04-02T07:48:50.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.712+0000 7f69ff600700 1 -- 172.21.15.89:0/1963466048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f81592d0 con 0x7f69f815ef70 2024-04-02T07:48:50.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.712+0000 7f69ff600700 1 -- 172.21.15.89:0/1963466048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f81534c0 con 0x7f69f815ef70 2024-04-02T07:48:50.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.713+0000 7f69ff600700 1 -- 172.21.15.89:0/1963466048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f81536a0 con 0x7f69f815ef70 2024-04-02T07:48:50.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.714+0000 7f69f4c00700 1 -- 172.21.15.89:0/1963466048 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f69e401a430 con 0x7f69f815ef70 2024-04-02T07:48:50.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.715+0000 7f69f4c00700 1 --2- 172.21.15.89:0/1963466048 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69d806ae40 0x7f69d806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:50.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.715+0000 7f69fcc00700 1 -- 172.21.15.89:0/1963466048 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69d806ae40 msgr2=0x7f69d806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:50.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.715+0000 7f69f4c00700 1 -- 172.21.15.89:0/1963466048 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6475+0+0 (secure 0 0 0) 0x7f69e4021070 con 0x7f69f815ef70 2024-04-02T07:48:50.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.716+0000 7f69fcc00700 1 --2- 172.21.15.89:0/1963466048 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69d806ae40 0x7f69d806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:48:50.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.720+0000 7f69f4c00700 1 -- 172.21.15.89:0/1963466048 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f69e4056010 con 0x7f69f815ef70 2024-04-02T07:48:50.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.917+0000 7f69fcc00700 1 -- 172.21.15.89:0/1963466048 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69d806ae40 msgr2=0x7f69d806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:50.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.917+0000 7f69fcc00700 1 --2- 172.21.15.89:0/1963466048 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69d806ae40 0x7f69d806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:48:50.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.928+0000 7f69ff600700 1 -- 172.21.15.89:0/1963466048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f69f81597f0 con 0x7f69f815ef70 2024-04-02T07:48:50.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.928+0000 7f69f4c00700 1 -- 172.21.15.89:0/1963466048 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v53) v1 ==== 104+0+11 (secure 0 0 0) 0x7f69e4059630 con 0x7f69f815ef70 2024-04-02T07:48:50.930 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:48:50.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.931+0000 7f69ff600700 1 -- 172.21.15.89:0/1963466048 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69d806ae40 msgr2=0x7f69d806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:48:50.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.931+0000 7f69ff600700 1 --2- 172.21.15.89:0/1963466048 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69d806ae40 0x7f69d806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.931+0000 7f69ff600700 1 -- 172.21.15.89:0/1963466048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f815ef70 msgr2=0x7f69f8155a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:50.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.931+0000 7f69ff600700 1 --2- 172.21.15.89:0/1963466048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f815ef70 0x7f69f8155a90 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f69e40090b0 tx=0x7f69e4004750 comp rx=0 tx=0).stop 2024-04-02T07:48:50.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.931+0000 7f69ff600700 1 -- 172.21.15.89:0/1963466048 shutdown_connections 2024-04-02T07:48:50.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.931+0000 7f69ff600700 1 --2- 172.21.15.89:0/1963466048 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69d806ae40 0x7f69d806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.931+0000 7f69ff600700 1 --2- 172.21.15.89:0/1963466048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69f806a790 0x7f69f81f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.931+0000 7f69ff600700 1 --2- 172.21.15.89:0/1963466048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69f8160d70 0x7f69f8155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.931+0000 7f69ff600700 1 --2- 172.21.15.89:0/1963466048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f815ef70 0x7f69f8155a90 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:50.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.931+0000 7f69ff600700 1 -- 172.21.15.89:0/1963466048 >> 172.21.15.89:0/1963466048 conn(0x7f69f80b5720 msgr2=0x7f69f80d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:50.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.931+0000 7f69ff600700 1 -- 172.21.15.89:0/1963466048 shutdown_connections 2024-04-02T07:48:50.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:50.931+0000 7f69ff600700 1 -- 172.21.15.89:0/1963466048 wait complete. 2024-04-02T07:48:50.952 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:48:51.008 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:51.006+0000 7f5ffe757240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:48:51.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.148+0000 7eff7a200700 1 Processor -- start 2024-04-02T07:48:51.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.148+0000 7eff7a200700 1 -- start start 2024-04-02T07:48:51.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.149+0000 7eff7a200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7eff740cd5c0 0x7eff7415e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:51.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.150+0000 7eff7a200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7eff7415cfe0 0x7eff7415d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:51.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.150+0000 7eff7a200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7415eda0 con 0x7eff7406f310 2024-04-02T07:48:51.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.150+0000 7eff7a200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff74099dd0 con 0x7eff740cd5c0 2024-04-02T07:48:51.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.150+0000 7eff7a200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff7409a0b0 con 0x7eff7415cfe0 2024-04-02T07:48:51.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.150+0000 7eff73400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7eff740cd5c0 0x7eff7415e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:51.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.150+0000 7eff72a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7eff7406f310 0x7eff7406f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55696/0 (socket says 172.21.15.89:55696) 2024-04-02T07:48:51.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.150+0000 7eff72a00700 1 -- 172.21.15.89:0/1888130172 learned_addr learned my addr 172.21.15.89:0/1888130172 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:51.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.150+0000 7eff73400700 1 --2- 172.21.15.89:0/1888130172 >> v2:172.21.15.89:3300/0 conn(0x7eff740cd5c0 0x7eff7415e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58360/0 (socket says 172.21.15.89:58360) 2024-04-02T07:48:51.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.150+0000 7eff73e00700 1 --2- 172.21.15.89:0/1888130172 >> v2:172.21.15.113:3300/0 conn(0x7eff7415cfe0 0x7eff7415d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:51.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.150+0000 7eff73400700 1 -- 172.21.15.89:0/1888130172 >> v2:172.21.15.113:3300/0 conn(0x7eff7415cfe0 msgr2=0x7eff7415d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:51.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.150+0000 7eff73400700 1 --2- 172.21.15.89:0/1888130172 >> v2:172.21.15.113:3300/0 conn(0x7eff7415cfe0 0x7eff7415d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.150+0000 7eff73400700 1 -- 172.21.15.89:0/1888130172 >> v1:172.21.15.89:6789/0 conn(0x7eff7406f310 legacy=0x7eff7406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:51.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.150+0000 7eff73400700 1 -- 172.21.15.89:0/1888130172 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7409a390 con 0x7eff740cd5c0 2024-04-02T07:48:51.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.151+0000 7eff73400700 1 --2- 172.21.15.89:0/1888130172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff740cd5c0 0x7eff7415e860 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7eff64009840 tx=0x7eff64009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1447f79ab0bbed3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:51.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.151+0000 7eff72000700 1 -- 172.21.15.89:0/1888130172 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64011830 con 0x7eff740cd5c0 2024-04-02T07:48:51.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.151+0000 7eff72000700 1 -- 172.21.15.89:0/1888130172 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff64011990 con 0x7eff740cd5c0 2024-04-02T07:48:51.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.151+0000 7eff72000700 1 -- 172.21.15.89:0/1888130172 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64011d20 con 0x7eff740cd5c0 2024-04-02T07:48:51.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.151+0000 7eff7a200700 1 -- 172.21.15.89:0/1888130172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff740cd5c0 msgr2=0x7eff7415e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:51.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.151+0000 7eff7a200700 1 --2- 172.21.15.89:0/1888130172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff740cd5c0 0x7eff7415e860 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7eff64009840 tx=0x7eff64009b50 comp rx=0 tx=0).stop 2024-04-02T07:48:51.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.152+0000 7eff7a200700 1 -- 172.21.15.89:0/1888130172 shutdown_connections 2024-04-02T07:48:51.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.152+0000 7eff7a200700 1 --2- 172.21.15.89:0/1888130172 >> v2:172.21.15.113:3300/0 conn(0x7eff7415cfe0 0x7eff7415d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.152+0000 7eff7a200700 1 --2- 172.21.15.89:0/1888130172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff740cd5c0 0x7eff7415e860 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.152+0000 7eff7a200700 1 -- 172.21.15.89:0/1888130172 >> 172.21.15.89:0/1888130172 conn(0x7eff740d1370 msgr2=0x7eff740d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:51.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.152+0000 7eff7a200700 1 -- 172.21.15.89:0/1888130172 shutdown_connections 2024-04-02T07:48:51.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.152+0000 7eff7a200700 1 -- 172.21.15.89:0/1888130172 wait complete. 2024-04-02T07:48:51.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.153+0000 7eff7a200700 1 Processor -- start 2024-04-02T07:48:51.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.154+0000 7eff7a200700 1 -- start start 2024-04-02T07:48:51.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.154+0000 7eff7a200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff7406f310 0x7eff7415a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:51.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.154+0000 7eff7a200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff740cd5c0 0x7eff74159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:51.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.154+0000 7eff7a200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff7415cfe0 0x7eff7415c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:51.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.154+0000 7eff7a200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7409b2e0 con 0x7eff7415cfe0 2024-04-02T07:48:51.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.154+0000 7eff7a200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff740c5420 con 0x7eff740cd5c0 2024-04-02T07:48:51.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.154+0000 7eff7a200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff740c5700 con 0x7eff7406f310 2024-04-02T07:48:51.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.155+0000 7eff73e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff7415cfe0 0x7eff7415c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:51.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.155+0000 7eff73e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff7415cfe0 0x7eff7415c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58366/0 (socket says 172.21.15.89:58366) 2024-04-02T07:48:51.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.155+0000 7eff73e00700 1 -- 172.21.15.89:0/916098437 learned_addr learned my addr 172.21.15.89:0/916098437 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:51.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.155+0000 7eff72a00700 1 --2- 172.21.15.89:0/916098437 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff740cd5c0 0x7eff74159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:51.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.155+0000 7eff73400700 1 --2- 172.21.15.89:0/916098437 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff7406f310 0x7eff7415a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:51.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.155+0000 7eff73e00700 1 -- 172.21.15.89:0/916098437 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff7406f310 msgr2=0x7eff7415a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:51.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.155+0000 7eff73e00700 1 --2- 172.21.15.89:0/916098437 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff7406f310 0x7eff7415a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.155+0000 7eff73e00700 1 -- 172.21.15.89:0/916098437 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff740cd5c0 msgr2=0x7eff74159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:51.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.155+0000 7eff73e00700 1 --2- 172.21.15.89:0/916098437 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff740cd5c0 0x7eff74159a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.155+0000 7eff73e00700 1 -- 172.21.15.89:0/916098437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff640094a0 con 0x7eff7415cfe0 2024-04-02T07:48:51.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.155+0000 7eff73e00700 1 --2- 172.21.15.89:0/916098437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff7415cfe0 0x7eff7415c370 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00efc0 tx=0x7eff6c00c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:51.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.156+0000 7eff6b400700 1 -- 172.21.15.89:0/916098437 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c00cad0 con 0x7eff7415cfe0 2024-04-02T07:48:51.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.156+0000 7eff6b400700 1 -- 172.21.15.89:0/916098437 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6c00f040 con 0x7eff7415cfe0 2024-04-02T07:48:51.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.156+0000 7eff7a200700 1 -- 172.21.15.89:0/916098437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff740c59e0 con 0x7eff7415cfe0 2024-04-02T07:48:51.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.156+0000 7eff6b400700 1 -- 172.21.15.89:0/916098437 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c00ce70 con 0x7eff7415cfe0 2024-04-02T07:48:51.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.156+0000 7eff7a200700 1 -- 172.21.15.89:0/916098437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff7415a950 con 0x7eff7415cfe0 2024-04-02T07:48:51.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.157+0000 7eff7a200700 1 -- 172.21.15.89:0/916098437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff74067720 con 0x7eff7415cfe0 2024-04-02T07:48:51.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.158+0000 7eff6b400700 1 -- 172.21.15.89:0/916098437 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 89308+0+0 (secure 0 0 0) 0x7eff6c014480 con 0x7eff7415cfe0 2024-04-02T07:48:51.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.159+0000 7eff6b400700 1 --2- 172.21.15.89:0/916098437 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff5406ae40 0x7eff5406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:51.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.159+0000 7eff6b400700 1 -- 172.21.15.89:0/916098437 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6475+0+0 (secure 0 0 0) 0x7eff6c08a230 con 0x7eff7415cfe0 2024-04-02T07:48:51.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.159+0000 7eff73400700 1 -- 172.21.15.89:0/916098437 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff5406ae40 msgr2=0x7eff5406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:51.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.159+0000 7eff73400700 1 --2- 172.21.15.89:0/916098437 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff5406ae40 0x7eff5406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:48:51.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.167+0000 7eff6b400700 1 -- 172.21.15.89:0/916098437 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff6c054ce0 con 0x7eff7415cfe0 2024-04-02T07:48:51.216 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:51.214+0000 7f5ffe757240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:48:51.349 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:51.347+0000 7f0d3a67c240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:48:51.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.360+0000 7eff73400700 1 -- 172.21.15.89:0/916098437 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff5406ae40 msgr2=0x7eff5406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:51.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.360+0000 7eff73400700 1 --2- 172.21.15.89:0/916098437 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff5406ae40 0x7eff5406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:48:51.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.376+0000 7eff7a200700 1 -- 172.21.15.89:0/916098437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7eff740d0e20 con 0x7eff7415cfe0 2024-04-02T07:48:51.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.377+0000 7eff6b400700 1 -- 172.21.15.89:0/916098437 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v53) v1 ==== 100+0+11 (secure 0 0 0) 0x7eff6c058300 con 0x7eff7415cfe0 2024-04-02T07:48:51.379 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:48:51.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.379+0000 7eff7a200700 1 -- 172.21.15.89:0/916098437 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff5406ae40 msgr2=0x7eff5406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:48:51.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.379+0000 7eff7a200700 1 --2- 172.21.15.89:0/916098437 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff5406ae40 0x7eff5406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.379+0000 7eff7a200700 1 -- 172.21.15.89:0/916098437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff7415cfe0 msgr2=0x7eff7415c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:51.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.379+0000 7eff7a200700 1 --2- 172.21.15.89:0/916098437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff7415cfe0 0x7eff7415c370 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00efc0 tx=0x7eff6c00c0c0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.379+0000 7eff7a200700 1 -- 172.21.15.89:0/916098437 shutdown_connections 2024-04-02T07:48:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.379+0000 7eff7a200700 1 --2- 172.21.15.89:0/916098437 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff5406ae40 0x7eff5406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.379+0000 7eff7a200700 1 --2- 172.21.15.89:0/916098437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff7415cfe0 0x7eff7415c370 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.379+0000 7eff7a200700 1 --2- 172.21.15.89:0/916098437 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff7406f310 0x7eff7415a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.379+0000 7eff7a200700 1 --2- 172.21.15.89:0/916098437 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff740cd5c0 0x7eff74159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.379+0000 7eff7a200700 1 -- 172.21.15.89:0/916098437 >> 172.21.15.89:0/916098437 conn(0x7eff740d1370 msgr2=0x7eff740cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.379+0000 7eff7a200700 1 -- 172.21.15.89:0/916098437 shutdown_connections 2024-04-02T07:48:51.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.379+0000 7eff7a200700 1 -- 172.21.15.89:0/916098437 wait complete. 2024-04-02T07:48:51.400 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:48:51.417 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:51.415+0000 7f5ffe757240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:48:51.575 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:51.573+0000 7f0d3a67c240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:48:51.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.594+0000 7f23aac00700 1 Processor -- start 2024-04-02T07:48:51.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.595+0000 7f23aac00700 1 -- start start 2024-04-02T07:48:51.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.597+0000 7f23aac00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f23a40d05e0 0x7f23a40d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:51.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.597+0000 7f23aac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23a40b6990 0x7f23a40b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:51.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.597+0000 7f23aac00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a40d0f80 con 0x7f23a415da50 2024-04-02T07:48:51.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.597+0000 7f23aac00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a40b95d0 con 0x7f23a40d05e0 2024-04-02T07:48:51.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.597+0000 7f23aac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a415de30 con 0x7f23a40b6990 2024-04-02T07:48:51.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.597+0000 7f23a3400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f23a40d05e0 0x7f23a40d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:51.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.597+0000 7f23a8c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23a40b6990 0x7f23a40b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:51.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.597+0000 7f23a3e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f23a415da50 0x7f23a40ce730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:43264/0 (socket says 172.21.15.89:43264) 2024-04-02T07:48:51.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.597+0000 7f23a3400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f23a40d05e0 0x7f23a40d0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:44668/0 (socket says 172.21.15.89:44668) 2024-04-02T07:48:51.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.597+0000 7f23a8c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23a40b6990 0x7f23a40b6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:54584/0 (socket says 172.21.15.89:54584) 2024-04-02T07:48:51.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.597+0000 7f23a3e00700 1 -- 172.21.15.89:0/2439939066 learned_addr learned my addr 172.21.15.89:0/2439939066 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:51.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.598+0000 7f23a8c00700 1 -- 172.21.15.89:0/2439939066 >> v2:172.21.15.113:3300/0 conn(0x7f23a40d05e0 msgr2=0x7f23a40d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:51.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.598+0000 7f23a8c00700 1 --2- 172.21.15.89:0/2439939066 >> v2:172.21.15.113:3300/0 conn(0x7f23a40d05e0 0x7f23a40d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.598+0000 7f23a2a00700 1 -- 172.21.15.89:0/2439939066 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637822169 0 0) 0x7f23a40d0f80 con 0x7f23a415da50 2024-04-02T07:48:51.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.598+0000 7f23a8c00700 1 -- 172.21.15.89:0/2439939066 >> v1:172.21.15.113:6789/0 conn(0x7f23a415da50 legacy=0x7f23a40ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:51.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.598+0000 7f23a8c00700 1 -- 172.21.15.89:0/2439939066 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a40d24e0 con 0x7f23a40b6990 2024-04-02T07:48:51.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.598+0000 7f23a3400700 1 --2- 172.21.15.89:0/2439939066 >> v2:172.21.15.113:3300/0 conn(0x7f23a40d05e0 0x7f23a40d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:48:51.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.598+0000 7f23a8c00700 1 --2- 172.21.15.89:0/2439939066 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23a40b6990 0x7f23a40b6e40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f239c009d90 tx=0x7f239c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6eb2256228a1ae1e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:51.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.598+0000 7f23a2a00700 1 -- 172.21.15.89:0/2439939066 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c00e070 con 0x7f23a40b6990 2024-04-02T07:48:51.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.598+0000 7f23a2a00700 1 -- 172.21.15.89:0/2439939066 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239c00bb20 con 0x7f23a40b6990 2024-04-02T07:48:51.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.598+0000 7f23a2a00700 1 -- 172.21.15.89:0/2439939066 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c012430 con 0x7f23a40b6990 2024-04-02T07:48:51.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.599+0000 7f23aac00700 1 -- 172.21.15.89:0/2439939066 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23a40b6990 msgr2=0x7f23a40b6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:51.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.599+0000 7f23aac00700 1 --2- 172.21.15.89:0/2439939066 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23a40b6990 0x7f23a40b6e40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f239c009d90 tx=0x7f239c00b330 comp rx=0 tx=0).stop 2024-04-02T07:48:51.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.599+0000 7f23aac00700 1 -- 172.21.15.89:0/2439939066 shutdown_connections 2024-04-02T07:48:51.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.599+0000 7f23aac00700 1 --2- 172.21.15.89:0/2439939066 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23a40b6990 0x7f23a40b6e40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.599+0000 7f23aac00700 1 --2- 172.21.15.89:0/2439939066 >> v2:172.21.15.113:3300/0 conn(0x7f23a40d05e0 0x7f23a40d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.599+0000 7f23aac00700 1 -- 172.21.15.89:0/2439939066 >> 172.21.15.89:0/2439939066 conn(0x7f23a40666a0 msgr2=0x7f23a4066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:51.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.599+0000 7f23aac00700 1 -- 172.21.15.89:0/2439939066 shutdown_connections 2024-04-02T07:48:51.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.599+0000 7f23aac00700 1 -- 172.21.15.89:0/2439939066 wait complete. 2024-04-02T07:48:51.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.601+0000 7f23aac00700 1 Processor -- start 2024-04-02T07:48:51.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.601+0000 7f23aac00700 1 -- start start 2024-04-02T07:48:51.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.602+0000 7f23aac00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a40b6990 0x7f23a40c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:51.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.602+0000 7f23aac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23a40d05e0 0x7f23a40c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:51.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.602+0000 7f23aac00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23a415da50 0x7f23a4151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:51.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.602+0000 7f23aac00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40ca270 con 0x7f23a40b6990 2024-04-02T07:48:51.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.602+0000 7f23aac00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a41f29a0 con 0x7f23a415da50 2024-04-02T07:48:51.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.602+0000 7f23aac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a41f2c80 con 0x7f23a40d05e0 2024-04-02T07:48:51.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.602+0000 7f23a3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a40b6990 0x7f23a40c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:51.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.602+0000 7f23a3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a40b6990 0x7f23a40c80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58370/0 (socket says 172.21.15.89:58370) 2024-04-02T07:48:51.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.602+0000 7f23a3e00700 1 -- 172.21.15.89:0/3395845865 learned_addr learned my addr 172.21.15.89:0/3395845865 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:51.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.602+0000 7f23a8c00700 1 --2- 172.21.15.89:0/3395845865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23a415da50 0x7f23a4151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:51.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.602+0000 7f23a3400700 1 --2- 172.21.15.89:0/3395845865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23a40d05e0 0x7f23a40c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:51.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.602+0000 7f23a3e00700 1 -- 172.21.15.89:0/3395845865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23a40d05e0 msgr2=0x7f23a40c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:51.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.603+0000 7f23a3e00700 1 --2- 172.21.15.89:0/3395845865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23a40d05e0 0x7f23a40c9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.603+0000 7f23a3e00700 1 -- 172.21.15.89:0/3395845865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23a415da50 msgr2=0x7f23a4151620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:51.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.603+0000 7f23a3e00700 1 --2- 172.21.15.89:0/3395845865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23a415da50 0x7f23a4151620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.603+0000 7f23a3e00700 1 -- 172.21.15.89:0/3395845865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23900094a0 con 0x7f23a40b6990 2024-04-02T07:48:51.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.603+0000 7f23a3e00700 1 --2- 172.21.15.89:0/3395845865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a40b6990 0x7f23a40c80e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f239400c550 tx=0x7f239400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:51.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.603+0000 7f239be00700 1 -- 172.21.15.89:0/3395845865 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394004c80 con 0x7f23a40b6990 2024-04-02T07:48:51.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.603+0000 7f239be00700 1 -- 172.21.15.89:0/3395845865 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2394004e00 con 0x7f23a40b6990 2024-04-02T07:48:51.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.603+0000 7f239be00700 1 -- 172.21.15.89:0/3395845865 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394007310 con 0x7f23a40b6990 2024-04-02T07:48:51.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.603+0000 7f23aac00700 1 -- 172.21.15.89:0/3395845865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239c009a70 con 0x7f23a40b6990 2024-04-02T07:48:51.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.603+0000 7f23aac00700 1 -- 172.21.15.89:0/3395845865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a41f3240 con 0x7f23a40b6990 2024-04-02T07:48:51.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.605+0000 7f23aac00700 1 -- 172.21.15.89:0/3395845865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a41f3820 con 0x7f23a40b6990 2024-04-02T07:48:51.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.605+0000 7f239be00700 1 -- 172.21.15.89:0/3395845865 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f2394007470 con 0x7f23a40b6990 2024-04-02T07:48:51.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.606+0000 7f239be00700 1 --2- 172.21.15.89:0/3395845865 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f238406ae40 0x7f238406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:51.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.606+0000 7f239be00700 1 -- 172.21.15.89:0/3395845865 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6475+0+0 (secure 0 0 0) 0x7f23940894b0 con 0x7f23a40b6990 2024-04-02T07:48:51.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.606+0000 7f23a3400700 1 -- 172.21.15.89:0/3395845865 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f238406ae40 msgr2=0x7f238406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:51.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.606+0000 7f23a3400700 1 --2- 172.21.15.89:0/3395845865 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f238406ae40 0x7f238406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:48:51.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.610+0000 7f239be00700 1 -- 172.21.15.89:0/3395845865 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f23940541a0 con 0x7f23a40b6990 2024-04-02T07:48:51.795 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:51.793+0000 7f0d3a67c240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:48:51.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.807+0000 7f23a3400700 1 -- 172.21.15.89:0/3395845865 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f238406ae40 msgr2=0x7f238406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:51.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.807+0000 7f23a3400700 1 --2- 172.21.15.89:0/3395845865 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f238406ae40 0x7f238406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:48:51.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.870+0000 7f23aac00700 1 -- 172.21.15.89:0/3395845865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23a4067720 con 0x7f23a40b6990 2024-04-02T07:48:51.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.871+0000 7f239be00700 1 -- 172.21.15.89:0/3395845865 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f23940577c0 con 0x7f23a40b6990 2024-04-02T07:48:51.873 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:51.873 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5037,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3788023013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3788023013},{"type":"v1","addr":"172.21.15.89:6835","nonce":3788023013}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5040,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4029308904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":4029308904},{"type":"v1","addr":"172.21.15.113:6835","nonce":4029308904}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5043,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2115217020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2115217020},{"type":"v1","addr":"172.21.15.113:6834","nonce":2115217020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:37:05.500422+0000","modified":"2024-04-02T07:37:08.737868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.89:6837/929002087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":929002087},{"type":"v1","addr":"172.21.15.89:6837","nonce":929002087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-02T07:48:51.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.873+0000 7f23aac00700 1 -- 172.21.15.89:0/3395845865 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f238406ae40 msgr2=0x7f238406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:48:51.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.873+0000 7f23aac00700 1 --2- 172.21.15.89:0/3395845865 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f238406ae40 0x7f238406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.873+0000 7f23aac00700 1 -- 172.21.15.89:0/3395845865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a40b6990 msgr2=0x7f23a40c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:51.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.873+0000 7f23aac00700 1 --2- 172.21.15.89:0/3395845865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a40b6990 0x7f23a40c80e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f239400c550 tx=0x7f239400c860 comp rx=0 tx=0).stop 2024-04-02T07:48:51.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.874+0000 7f23aac00700 1 -- 172.21.15.89:0/3395845865 shutdown_connections 2024-04-02T07:48:51.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.874+0000 7f23aac00700 1 --2- 172.21.15.89:0/3395845865 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f238406ae40 0x7f238406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.874+0000 7f23aac00700 1 --2- 172.21.15.89:0/3395845865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23a415da50 0x7f23a4151620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.874+0000 7f23aac00700 1 --2- 172.21.15.89:0/3395845865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a40b6990 0x7f23a40c80e0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.874+0000 7f23aac00700 1 --2- 172.21.15.89:0/3395845865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23a40d05e0 0x7f23a40c9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:51.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.874+0000 7f23aac00700 1 -- 172.21.15.89:0/3395845865 >> 172.21.15.89:0/3395845865 conn(0x7f23a40666a0 msgr2=0x7f23a415cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:51.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.874+0000 7f23aac00700 1 -- 172.21.15.89:0/3395845865 shutdown_connections 2024-04-02T07:48:51.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:51.874+0000 7f23aac00700 1 -- 172.21.15.89:0/3395845865 wait complete. 2024-04-02T07:48:51.877 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 17 2024-04-02T07:48:51.883 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:51.881+0000 7f5ffe757240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:48:51.894 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:48:52.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.085+0000 7f63e2a00700 1 Processor -- start 2024-04-02T07:48:52.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.085+0000 7f63e2a00700 1 -- start start 2024-04-02T07:48:52.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.086+0000 7f63e2a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63dc0d18b0 0x7f63dc0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:52.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.086+0000 7f63e2a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f63dc161590 0x7f63dc161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:52.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.086+0000 7f63e2a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63dc0ceda0 con 0x7f63dc160f10 2024-04-02T07:48:52.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.086+0000 7f63e2a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63dc0d1d30 con 0x7f63dc161590 2024-04-02T07:48:52.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.086+0000 7f63e2a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc161f60 con 0x7f63dc0d18b0 2024-04-02T07:48:52.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.087+0000 7f63db400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63dc0d18b0 0x7f63dc0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:52.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.087+0000 7f63e0a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f63dc161590 0x7f63dc161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:52.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.087+0000 7f63db400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63dc0d18b0 0x7f63dc0cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:54602/0 (socket says 172.21.15.89:54602) 2024-04-02T07:48:52.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.087+0000 7f63db400700 1 -- 172.21.15.89:0/3359934320 learned_addr learned my addr 172.21.15.89:0/3359934320 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:52.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.087+0000 7f63e0a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f63dc161590 0x7f63dc161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:44690/0 (socket says 172.21.15.89:44690) 2024-04-02T07:48:52.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.087+0000 7f63dbe00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f63dc160f10 0x7f63dc0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:43266/0 (socket says 172.21.15.89:43266) 2024-04-02T07:48:52.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.087+0000 7f63db400700 1 -- 172.21.15.89:0/3359934320 >> v2:172.21.15.113:3300/0 conn(0x7f63dc161590 msgr2=0x7f63dc161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:52.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.087+0000 7f63db400700 1 --2- 172.21.15.89:0/3359934320 >> v2:172.21.15.113:3300/0 conn(0x7f63dc161590 0x7f63dc161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.087+0000 7f63db400700 1 -- 172.21.15.89:0/3359934320 >> v1:172.21.15.113:6789/0 conn(0x7f63dc160f10 legacy=0x7f63dc0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:52.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.087+0000 7f63db400700 1 -- 172.21.15.89:0/3359934320 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63dc1612f0 con 0x7f63dc0d18b0 2024-04-02T07:48:52.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.087+0000 7f63e0a00700 1 --2- 172.21.15.89:0/3359934320 >> v2:172.21.15.113:3300/0 conn(0x7f63dc161590 0x7f63dc161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:48:52.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.087+0000 7f63db400700 1 --2- 172.21.15.89:0/3359934320 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63dc0d18b0 0x7f63dc0cf230 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009920 tx=0x7f63c8009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe86d9c0946c8c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:52.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.088+0000 7f63daa00700 1 -- 172.21.15.89:0/3359934320 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8011830 con 0x7f63dc0d18b0 2024-04-02T07:48:52.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.088+0000 7f63daa00700 1 -- 172.21.15.89:0/3359934320 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c8011990 con 0x7f63dc0d18b0 2024-04-02T07:48:52.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.088+0000 7f63daa00700 1 -- 172.21.15.89:0/3359934320 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8011d20 con 0x7f63dc0d18b0 2024-04-02T07:48:52.092 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:52.086+0000 7f5ffe757240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:48:52.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.088+0000 7f63e2a00700 1 -- 172.21.15.89:0/3359934320 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63dc0d18b0 msgr2=0x7f63dc0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:52.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.088+0000 7f63e2a00700 1 --2- 172.21.15.89:0/3359934320 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63dc0d18b0 0x7f63dc0cf230 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009920 tx=0x7f63c8009c30 comp rx=0 tx=0).stop 2024-04-02T07:48:52.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.088+0000 7f63e2a00700 1 -- 172.21.15.89:0/3359934320 shutdown_connections 2024-04-02T07:48:52.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.088+0000 7f63e2a00700 1 --2- 172.21.15.89:0/3359934320 >> v2:172.21.15.113:3300/0 conn(0x7f63dc161590 0x7f63dc161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.088+0000 7f63e2a00700 1 --2- 172.21.15.89:0/3359934320 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63dc0d18b0 0x7f63dc0cf230 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.088+0000 7f63e2a00700 1 -- 172.21.15.89:0/3359934320 >> 172.21.15.89:0/3359934320 conn(0x7f63dc08aff0 msgr2=0x7f63dc0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:52.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.088+0000 7f63e2a00700 1 -- 172.21.15.89:0/3359934320 shutdown_connections 2024-04-02T07:48:52.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.088+0000 7f63e2a00700 1 -- 172.21.15.89:0/3359934320 wait complete. 2024-04-02T07:48:52.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.090+0000 7f63e2a00700 1 Processor -- start 2024-04-02T07:48:52.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.090+0000 7f63e2a00700 1 -- start start 2024-04-02T07:48:52.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.091+0000 7f63e2a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63dc0d18b0 0x7f63dc14ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:52.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.091+0000 7f63e2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63dc160f10 0x7f63dc09f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:52.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.091+0000 7f63e2a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63dc161590 0x7f63dc09fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:52.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.091+0000 7f63e2a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc14f1a0 con 0x7f63dc160f10 2024-04-02T07:48:52.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.091+0000 7f63e2a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc0a0420 con 0x7f63dc0d18b0 2024-04-02T07:48:52.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.091+0000 7f63e2a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc0a0700 con 0x7f63dc161590 2024-04-02T07:48:52.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.091+0000 7f63db400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63dc160f10 0x7f63dc09f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:52.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.091+0000 7f63db400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63dc160f10 0x7f63dc09f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58386/0 (socket says 172.21.15.89:58386) 2024-04-02T07:48:52.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.091+0000 7f63db400700 1 -- 172.21.15.89:0/179680047 learned_addr learned my addr 172.21.15.89:0/179680047 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:52.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.091+0000 7f63dbe00700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63dc0d18b0 0x7f63dc14ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:52.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.091+0000 7f63e0a00700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63dc161590 0x7f63dc09fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:52.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.092+0000 7f63e0a00700 1 -- 172.21.15.89:0/179680047 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63dc0d18b0 msgr2=0x7f63dc14ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:52.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.092+0000 7f63e0a00700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63dc0d18b0 0x7f63dc14ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.092+0000 7f63e0a00700 1 -- 172.21.15.89:0/179680047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63dc160f10 msgr2=0x7f63dc09f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:52.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.092+0000 7f63e0a00700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63dc160f10 0x7f63dc09f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.092+0000 7f63e0a00700 1 -- 172.21.15.89:0/179680047 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d4009a70 con 0x7f63dc161590 2024-04-02T07:48:52.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.092+0000 7f63db400700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63dc160f10 0x7f63dc09f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:48:52.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.092+0000 7f63e0a00700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63dc161590 0x7f63dc09fbe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f63d400efc0 tx=0x7f63d400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:52.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.092+0000 7f63d3e00700 1 -- 172.21.15.89:0/179680047 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d4009e80 con 0x7f63dc161590 2024-04-02T07:48:52.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.092+0000 7f63d3e00700 1 -- 172.21.15.89:0/179680047 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d400f040 con 0x7f63dc161590 2024-04-02T07:48:52.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.092+0000 7f63e2a00700 1 -- 172.21.15.89:0/179680047 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c8009550 con 0x7f63dc161590 2024-04-02T07:48:52.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.092+0000 7f63d3e00700 1 -- 172.21.15.89:0/179680047 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d400ccd0 con 0x7f63dc161590 2024-04-02T07:48:52.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.093+0000 7f63e2a00700 1 -- 172.21.15.89:0/179680047 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63dc0a0cc0 con 0x7f63dc161590 2024-04-02T07:48:52.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.094+0000 7f63e2a00700 1 -- 172.21.15.89:0/179680047 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63dc067720 con 0x7f63dc161590 2024-04-02T07:48:52.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.095+0000 7f63d3e00700 1 -- 172.21.15.89:0/179680047 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 15) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f63d4014480 con 0x7f63dc161590 2024-04-02T07:48:52.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.095+0000 7f63d3e00700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f63bc06ae40 0x7f63bc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:52.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.096+0000 7f63d3e00700 1 -- 172.21.15.89:0/179680047 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6475+0+0 (secure 0 0 0) 0x7f63d4089f60 con 0x7f63dc161590 2024-04-02T07:48:52.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.096+0000 7f63dbe00700 1 -- 172.21.15.89:0/179680047 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f63bc06ae40 msgr2=0x7f63bc06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:52.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.096+0000 7f63dbe00700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f63bc06ae40 0x7f63bc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:48:52.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.100+0000 7f63d3e00700 1 -- 172.21.15.89:0/179680047 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f63d4054aa0 con 0x7f63dc161590 2024-04-02T07:48:52.298 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:52.296+0000 7f0d3a67c240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:48:52.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.297+0000 7f63dbe00700 1 -- 172.21.15.89:0/179680047 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f63bc06ae40 msgr2=0x7f63bc06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:52.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.297+0000 7f63dbe00700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f63bc06ae40 0x7f63bc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:48:52.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.311+0000 7f63e2a00700 1 -- 172.21.15.89:0/179680047 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63dc0a0840 con 0x7f63dc161590 2024-04-02T07:48:52.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.312+0000 7f63d3e00700 1 -- 172.21.15.89:0/179680047 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+17277 (secure 0 0 0) 0x7f63dc0a0840 con 0x7f63dc161590 2024-04-02T07:48:52.314 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:52.315 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":53,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:48:48.910822+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:37:03.665694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-02T07:37:04.473147+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.19","mappings":[{"from":4,"to":5}]},{"pgid":"5.1a","mappings":[{"from":4,"to":5}]},{"pgid":"5.1d","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6833/43357":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/2201396539":"2024-04-03T07:37:22.446577+0000","172.21.15.89:6832/43357":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/350807093":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/2645815547":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/399385398":"2024-04-03T07:37:22.446577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":105,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:48:52.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.316+0000 7f63e2a00700 1 -- 172.21.15.89:0/179680047 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f63bc06ae40 msgr2=0x7f63bc06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:48:52.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.316+0000 7f63e2a00700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f63bc06ae40 0x7f63bc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.316+0000 7f63e2a00700 1 -- 172.21.15.89:0/179680047 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63dc161590 msgr2=0x7f63dc09fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:52.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.316+0000 7f63e2a00700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63dc161590 0x7f63dc09fbe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f63d400efc0 tx=0x7f63d400c3a0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.316+0000 7f63e2a00700 1 -- 172.21.15.89:0/179680047 shutdown_connections 2024-04-02T07:48:52.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.316+0000 7f63e2a00700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f63bc06ae40 0x7f63bc06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.316+0000 7f63e2a00700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63dc0d18b0 0x7f63dc14ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.316+0000 7f63e2a00700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63dc161590 0x7f63dc09fbe0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.316+0000 7f63e2a00700 1 --2- 172.21.15.89:0/179680047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63dc160f10 0x7f63dc09f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.316+0000 7f63e2a00700 1 -- 172.21.15.89:0/179680047 >> 172.21.15.89:0/179680047 conn(0x7f63dc08aff0 msgr2=0x7f63dc0cad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:52.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.316+0000 7f63e2a00700 1 -- 172.21.15.89:0/179680047 shutdown_connections 2024-04-02T07:48:52.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.316+0000 7f63e2a00700 1 -- 172.21.15.89:0/179680047 wait complete. 2024-04-02T07:48:52.345 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T07:48:52.345 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:48:52.522 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:52.520+0000 7f5ffe757240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:48:52.536 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:52.534+0000 7f0d3a67c240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:48:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.549+0000 7fbe1c800700 1 Processor -- start 2024-04-02T07:48:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.549+0000 7fbe1c800700 1 -- start start 2024-04-02T07:48:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.550+0000 7fbe1c800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbe1415d710 0x7fbe140d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.550+0000 7fbe1c800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1406bb40 con 0x7fbe1409aba0 2024-04-02T07:48:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.550+0000 7fbe1c800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe14099b00 con 0x7fbe1406c650 2024-04-02T07:48:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.550+0000 7fbe1c800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe14099de0 con 0x7fbe1415d710 2024-04-02T07:48:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.550+0000 7fbe19400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbe1409aba0 0x7fbe1406bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55708/0 (socket says 172.21.15.89:55708) 2024-04-02T07:48:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.551+0000 7fbe19400700 1 -- 172.21.15.89:0/257941875 learned_addr learned my addr 172.21.15.89:0/257941875 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.551+0000 7fbe19e00700 1 --2- 172.21.15.89:0/257941875 >> v2:172.21.15.113:3300/0 conn(0x7fbe1415d710 0x7fbe140d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.551+0000 7fbe18a00700 1 -- 172.21.15.89:0/257941875 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385863447 0 0) 0x7fbe1406bb40 con 0x7fbe1409aba0 2024-04-02T07:48:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.551+0000 7fbe18a00700 1 -- 172.21.15.89:0/257941875 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc0035b0 con 0x7fbe1409aba0 2024-04-02T07:48:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.551+0000 7fbe19e00700 1 -- 172.21.15.89:0/257941875 >> v1:172.21.15.113:6789/0 conn(0x7fbe1406c650 legacy=0x7fbe140ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.551+0000 7fbe19e00700 1 -- 172.21.15.89:0/257941875 >> v1:172.21.15.89:6789/0 conn(0x7fbe1409aba0 legacy=0x7fbe1406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.551+0000 7fbe19e00700 1 -- 172.21.15.89:0/257941875 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1409a0c0 con 0x7fbe1415d710 2024-04-02T07:48:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.551+0000 7fbe19e00700 1 --2- 172.21.15.89:0/257941875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe1415d710 0x7fbe140d0130 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009840 tx=0x7fbe08009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60b6c97dfdf363df server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.552+0000 7fbe18a00700 1 -- 172.21.15.89:0/257941875 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe08011830 con 0x7fbe1415d710 2024-04-02T07:48:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.552+0000 7fbe18a00700 1 -- 172.21.15.89:0/257941875 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe08011990 con 0x7fbe1415d710 2024-04-02T07:48:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.552+0000 7fbe18a00700 1 -- 172.21.15.89:0/257941875 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe08011d20 con 0x7fbe1415d710 2024-04-02T07:48:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.552+0000 7fbe1c800700 1 -- 172.21.15.89:0/257941875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe1415d710 msgr2=0x7fbe140d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.552+0000 7fbe1c800700 1 --2- 172.21.15.89:0/257941875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe1415d710 0x7fbe140d0130 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009840 tx=0x7fbe08009b50 comp rx=0 tx=0).stop 2024-04-02T07:48:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.552+0000 7fbe1c800700 1 -- 172.21.15.89:0/257941875 shutdown_connections 2024-04-02T07:48:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.552+0000 7fbe1c800700 1 --2- 172.21.15.89:0/257941875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe1415d710 0x7fbe140d0130 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.552+0000 7fbe1c800700 1 -- 172.21.15.89:0/257941875 >> 172.21.15.89:0/257941875 conn(0x7fbe140d13e0 msgr2=0x7fbe140d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.552+0000 7fbe1c800700 1 -- 172.21.15.89:0/257941875 shutdown_connections 2024-04-02T07:48:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.552+0000 7fbe1c800700 1 -- 172.21.15.89:0/257941875 wait complete. 2024-04-02T07:48:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.554+0000 7fbe1c800700 1 Processor -- start 2024-04-02T07:48:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.554+0000 7fbe1c800700 1 -- start start 2024-04-02T07:48:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.555+0000 7fbe1c800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe1406c650 0x7fbe1415ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.555+0000 7fbe1c800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe1409aba0 0x7fbe140c6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.555+0000 7fbe1c800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe1415d710 0x7fbe141f6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.555+0000 7fbe1c800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe140b45c0 con 0x7fbe1409aba0 2024-04-02T07:48:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.555+0000 7fbe1c800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe140b3fd0 con 0x7fbe1415d710 2024-04-02T07:48:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.555+0000 7fbe1c800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1415bf40 con 0x7fbe1406c650 2024-04-02T07:48:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.555+0000 7fbe19400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe1409aba0 0x7fbe140c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.555+0000 7fbe1a800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe1415d710 0x7fbe141f6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.555+0000 7fbe19400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe1409aba0 0x7fbe140c6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58388/0 (socket says 172.21.15.89:58388) 2024-04-02T07:48:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.555+0000 7fbe19400700 1 -- 172.21.15.89:0/1199076204 learned_addr learned my addr 172.21.15.89:0/1199076204 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.555+0000 7fbe19e00700 1 --2- 172.21.15.89:0/1199076204 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe1406c650 0x7fbe1415ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.556+0000 7fbe19400700 1 -- 172.21.15.89:0/1199076204 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe1406c650 msgr2=0x7fbe1415ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.556+0000 7fbe19400700 1 --2- 172.21.15.89:0/1199076204 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe1406c650 0x7fbe1415ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.556+0000 7fbe19400700 1 -- 172.21.15.89:0/1199076204 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe1415d710 msgr2=0x7fbe141f6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.556+0000 7fbe19400700 1 --2- 172.21.15.89:0/1199076204 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe1415d710 0x7fbe141f6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.556+0000 7fbe19400700 1 -- 172.21.15.89:0/1199076204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe080094a0 con 0x7fbe1409aba0 2024-04-02T07:48:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.556+0000 7fbe19400700 1 --2- 172.21.15.89:0/1199076204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe1409aba0 0x7fbe140c6230 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fbe04005910 tx=0x7fbe0400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.556+0000 7fbe0e000700 1 -- 172.21.15.89:0/1199076204 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe04005d70 con 0x7fbe1409aba0 2024-04-02T07:48:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.556+0000 7fbe0e000700 1 -- 172.21.15.89:0/1199076204 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe040070d0 con 0x7fbe1409aba0 2024-04-02T07:48:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.556+0000 7fbe1c800700 1 -- 172.21.15.89:0/1199076204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe140c6770 con 0x7fbe1409aba0 2024-04-02T07:48:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.556+0000 7fbe0e000700 1 -- 172.21.15.89:0/1199076204 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe04007460 con 0x7fbe1409aba0 2024-04-02T07:48:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.557+0000 7fbe1c800700 1 -- 172.21.15.89:0/1199076204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe141f73c0 con 0x7fbe1409aba0 2024-04-02T07:48:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.558+0000 7fbe1c800700 1 -- 172.21.15.89:0/1199076204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe140b4700 con 0x7fbe1409aba0 2024-04-02T07:48:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.558+0000 7fbe0e000700 1 -- 172.21.15.89:0/1199076204 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 89308+0+0 (secure 0 0 0) 0x7fbe040075c0 con 0x7fbe1409aba0 2024-04-02T07:48:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.560+0000 7fbe0e000700 1 --2- 172.21.15.89:0/1199076204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbdf806ae40 0x7fbdf806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.560+0000 7fbe0e000700 1 -- 172.21.15.89:0/1199076204 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6475+0+0 (secure 0 0 0) 0x7fbe04088df0 con 0x7fbe1409aba0 2024-04-02T07:48:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.560+0000 7fbe19e00700 1 -- 172.21.15.89:0/1199076204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbdf806ae40 msgr2=0x7fbdf806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.560+0000 7fbe19e00700 1 --2- 172.21.15.89:0/1199076204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbdf806ae40 0x7fbdf806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:48:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.564+0000 7fbe0e000700 1 -- 172.21.15.89:0/1199076204 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbe140b4700 con 0x7fbe1409aba0 2024-04-02T07:48:52.632 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:48:52.629+0000 7f5ffe757240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:48:52.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.761+0000 7fbe19e00700 1 -- 172.21.15.89:0/1199076204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbdf806ae40 msgr2=0x7fbdf806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:52.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.761+0000 7fbe19e00700 1 --2- 172.21.15.89:0/1199076204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbdf806ae40 0x7fbdf806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:48:52.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.818+0000 7fbe1c800700 1 -- 172.21.15.89:0/1199076204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe140b4700 con 0x7fbe1409aba0 2024-04-02T07:48:52.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.818+0000 7fbe0e000700 1 -- 172.21.15.89:0/1199076204 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4737 (secure 0 0 0) 0x7fbe140b4700 con 0x7fbe1409aba0 2024-04-02T07:48:52.821 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:52.821 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5037,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3788023013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3788023013},{"type":"v1","addr":"172.21.15.89:6835","nonce":3788023013}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5040,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4029308904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":4029308904},{"type":"v1","addr":"172.21.15.113:6835","nonce":4029308904}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5043,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2115217020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2115217020},{"type":"v1","addr":"172.21.15.113:6834","nonce":2115217020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:37:05.500422+0000","modified":"2024-04-02T07:37:08.737868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.89:6837/929002087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":929002087},{"type":"v1","addr":"172.21.15.89:6837","nonce":929002087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-02T07:48:52.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.821+0000 7fbe1c800700 1 -- 172.21.15.89:0/1199076204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbdf806ae40 msgr2=0x7fbdf806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:48:52.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.821+0000 7fbe1c800700 1 --2- 172.21.15.89:0/1199076204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbdf806ae40 0x7fbdf806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.821+0000 7fbe1c800700 1 -- 172.21.15.89:0/1199076204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe1409aba0 msgr2=0x7fbe140c6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:52.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.821+0000 7fbe1c800700 1 --2- 172.21.15.89:0/1199076204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe1409aba0 0x7fbe140c6230 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fbe04005910 tx=0x7fbe0400c9e0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.821+0000 7fbe1c800700 1 -- 172.21.15.89:0/1199076204 shutdown_connections 2024-04-02T07:48:52.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.821+0000 7fbe1c800700 1 --2- 172.21.15.89:0/1199076204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbdf806ae40 0x7fbdf806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.821+0000 7fbe1c800700 1 --2- 172.21.15.89:0/1199076204 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe1415d710 0x7fbe141f6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.821+0000 7fbe1c800700 1 --2- 172.21.15.89:0/1199076204 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe1406c650 0x7fbe1415ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.821+0000 7fbe1c800700 1 --2- 172.21.15.89:0/1199076204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe1409aba0 0x7fbe140c6230 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:52.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.821+0000 7fbe1c800700 1 -- 172.21.15.89:0/1199076204 >> 172.21.15.89:0/1199076204 conn(0x7fbe140d13e0 msgr2=0x7fbe1406ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:52.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.821+0000 7fbe1c800700 1 -- 172.21.15.89:0/1199076204 shutdown_connections 2024-04-02T07:48:52.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:52.822+0000 7fbe1c800700 1 -- 172.21.15.89:0/1199076204 wait complete. 2024-04-02T07:48:52.825 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 17 2024-04-02T07:48:52.842 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:48:52.996 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:52.994+0000 7f0d3a67c240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:48:53.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.037+0000 7fbed2800700 1 Processor -- start 2024-04-02T07:48:53.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.038+0000 7fbed2800700 1 -- start start 2024-04-02T07:48:53.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.040+0000 7fbed2800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbecc072580 0x7fbecc0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.040+0000 7fbed2800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbecc0b2180 con 0x7fbecc0b3af0 2024-04-02T07:48:53.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.040+0000 7fbed2800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbecc15b100 con 0x7fbecc0b2e50 2024-04-02T07:48:53.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.040+0000 7fbed2800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc15b3e0 con 0x7fbecc072580 2024-04-02T07:48:53.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.040+0000 7fbecb400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbecc0b3af0 0x7fbecc15a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55716/0 (socket says 172.21.15.89:55716) 2024-04-02T07:48:53.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.040+0000 7fbecb400700 1 -- 172.21.15.89:0/3249052245 learned_addr learned my addr 172.21.15.89:0/3249052245 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:53.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.040+0000 7fbec3e00700 1 --2- 172.21.15.89:0/3249052245 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbecc072580 0x7fbecc0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:53.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.040+0000 7fbecaa00700 1 -- 172.21.15.89:0/3249052245 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326212573 0 0) 0x7fbecc0b2180 con 0x7fbecc0b3af0 2024-04-02T07:48:53.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.040+0000 7fbecaa00700 1 -- 172.21.15.89:0/3249052245 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeb00035b0 con 0x7fbecc0b3af0 2024-04-02T07:48:53.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.041+0000 7fbecaa00700 1 -- 172.21.15.89:0/3249052245 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594188349 0 0) 0x7fbecc15b100 con 0x7fbecc0b2e50 2024-04-02T07:48:53.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.041+0000 7fbecaa00700 1 -- 172.21.15.89:0/3249052245 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbecc0b2180 con 0x7fbecc0b2e50 2024-04-02T07:48:53.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.041+0000 7fbecaa00700 1 -- 172.21.15.89:0/3249052245 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3406784107 0 0) 0x7fbeb00035b0 con 0x7fbecc0b3af0 2024-04-02T07:48:53.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.041+0000 7fbec3e00700 1 -- 172.21.15.89:0/3249052245 >> v1:172.21.15.113:6789/0 conn(0x7fbecc0b2e50 legacy=0x7fbecc0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.041+0000 7fbec3e00700 1 -- 172.21.15.89:0/3249052245 >> v1:172.21.15.89:6789/0 conn(0x7fbecc0b3af0 legacy=0x7fbecc15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.041+0000 7fbec3e00700 1 -- 172.21.15.89:0/3249052245 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc0b23f0 con 0x7fbecc072580 2024-04-02T07:48:53.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.041+0000 7fbec3e00700 1 --2- 172.21.15.89:0/3249052245 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbecc072580 0x7fbecc0b2910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009870 tx=0x7fbeb4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faa6871fab4c427e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:53.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.041+0000 7fbecaa00700 1 -- 172.21.15.89:0/3249052245 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc001670 con 0x7fbecc072580 2024-04-02T07:48:53.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.041+0000 7fbecaa00700 1 -- 172.21.15.89:0/3249052245 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb4004030 con 0x7fbecc072580 2024-04-02T07:48:53.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.041+0000 7fbecaa00700 1 -- 172.21.15.89:0/3249052245 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40043c0 con 0x7fbecc072580 2024-04-02T07:48:53.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.042+0000 7fbed2800700 1 -- 172.21.15.89:0/3249052245 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbecc072580 msgr2=0x7fbecc0b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.042+0000 7fbed2800700 1 --2- 172.21.15.89:0/3249052245 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbecc072580 0x7fbecc0b2910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009870 tx=0x7fbeb4009b80 comp rx=0 tx=0).stop 2024-04-02T07:48:53.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.042+0000 7fbed2800700 1 -- 172.21.15.89:0/3249052245 shutdown_connections 2024-04-02T07:48:53.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.042+0000 7fbed2800700 1 --2- 172.21.15.89:0/3249052245 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbecc072580 0x7fbecc0b2910 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.042+0000 7fbed2800700 1 -- 172.21.15.89:0/3249052245 >> 172.21.15.89:0/3249052245 conn(0x7fbecc0cf190 msgr2=0x7fbecc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:53.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.042+0000 7fbed2800700 1 -- 172.21.15.89:0/3249052245 shutdown_connections 2024-04-02T07:48:53.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.042+0000 7fbed2800700 1 -- 172.21.15.89:0/3249052245 wait complete. 2024-04-02T07:48:53.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.045+0000 7fbed2800700 1 Processor -- start 2024-04-02T07:48:53.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.045+0000 7fbed2800700 1 -- start start 2024-04-02T07:48:53.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.047+0000 7fbed2800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbecc072580 0x7fbecc14d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.047+0000 7fbed2800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbecc0b2e50 0x7fbecc15b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.047+0000 7fbed2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbecc0b3af0 0x7fbecc15be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.047+0000 7fbed2800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc151fa0 con 0x7fbecc0b3af0 2024-04-02T07:48:53.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.047+0000 7fbed2800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc151670 con 0x7fbecc0b2e50 2024-04-02T07:48:53.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.047+0000 7fbed2800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc151950 con 0x7fbecc072580 2024-04-02T07:48:53.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.047+0000 7fbecb400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbecc072580 0x7fbecc14d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:53.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.047+0000 7fbecbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbecc0b3af0 0x7fbecc15be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:53.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.047+0000 7fbec3e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbecc0b2e50 0x7fbecc15b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:53.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.047+0000 7fbecb400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbecc072580 0x7fbecc14d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:54628/0 (socket says 172.21.15.89:54628) 2024-04-02T07:48:53.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.047+0000 7fbecb400700 1 -- 172.21.15.89:0/3985016298 learned_addr learned my addr 172.21.15.89:0/3985016298 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:53.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.048+0000 7fbecb400700 1 -- 172.21.15.89:0/3985016298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbecc0b2e50 msgr2=0x7fbecc15b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.048+0000 7fbecb400700 1 --2- 172.21.15.89:0/3985016298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbecc0b2e50 0x7fbecc15b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.048+0000 7fbecb400700 1 -- 172.21.15.89:0/3985016298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbecc0b3af0 msgr2=0x7fbecc15be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.048+0000 7fbecb400700 1 --2- 172.21.15.89:0/3985016298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbecc0b3af0 0x7fbecc15be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.048+0000 7fbecb400700 1 -- 172.21.15.89:0/3985016298 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb40094a0 con 0x7fbecc072580 2024-04-02T07:48:53.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.048+0000 7fbec3e00700 1 --2- 172.21.15.89:0/3985016298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbecc0b2e50 0x7fbecc15b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:48:53.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.048+0000 7fbecbe00700 1 --2- 172.21.15.89:0/3985016298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbecc0b3af0 0x7fbecc15be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:48:53.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.048+0000 7fbecb400700 1 --2- 172.21.15.89:0/3985016298 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbecc072580 0x7fbecc14d830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00c700 tx=0x7fbebc00ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:53.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.048+0000 7fbec3400700 1 -- 172.21.15.89:0/3985016298 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc004c80 con 0x7fbecc072580 2024-04-02T07:48:53.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.048+0000 7fbec3400700 1 -- 172.21.15.89:0/3985016298 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbebc004e00 con 0x7fbecc072580 2024-04-02T07:48:53.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.048+0000 7fbed2800700 1 -- 172.21.15.89:0/3985016298 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbecc151c30 con 0x7fbecc072580 2024-04-02T07:48:53.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.048+0000 7fbec3400700 1 -- 172.21.15.89:0/3985016298 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc007310 con 0x7fbecc072580 2024-04-02T07:48:53.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.049+0000 7fbed2800700 1 -- 172.21.15.89:0/3985016298 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbecc14dd70 con 0x7fbecc072580 2024-04-02T07:48:53.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.050+0000 7fbed2800700 1 -- 172.21.15.89:0/3985016298 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbecc067720 con 0x7fbecc072580 2024-04-02T07:48:53.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.051+0000 7fbec3400700 1 -- 172.21.15.89:0/3985016298 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 16) v1 ==== 89308+0+0 (secure 0 0 0) 0x7fbebc007470 con 0x7fbecc072580 2024-04-02T07:48:53.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.051+0000 7fbec3400700 1 --2- 172.21.15.89:0/3985016298 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbea806ae40 0x7fbea806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.051+0000 7fbec3400700 1 -- 172.21.15.89:0/3985016298 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6475+0+0 (secure 0 0 0) 0x7fbebc0896a0 con 0x7fbecc072580 2024-04-02T07:48:53.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.051+0000 7fbec3e00700 1 -- 172.21.15.89:0/3985016298 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbea806ae40 msgr2=0x7fbea806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:48:53.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.052+0000 7fbec3e00700 1 --2- 172.21.15.89:0/3985016298 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbea806ae40 0x7fbea806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:48:53.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.056+0000 7fbec3400700 1 -- 172.21.15.89:0/3985016298 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbebc054260 con 0x7fbecc072580 2024-04-02T07:48:53.115 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:53.113+0000 7f0d3a67c240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:48:53.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.128+0000 7fbec3400700 1 -- 172.21.15.89:0/3985016298 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mgrmap(e 17) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fbebc054630 con 0x7fbecc072580 2024-04-02T07:48:53.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.129+0000 7fbec3400700 1 -- 172.21.15.89:0/3985016298 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbea806ae40 msgr2=0x7fbea806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:48:53.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.129+0000 7fbec3400700 1 --2- 172.21.15.89:0/3985016298 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbea806ae40 0x7fbea806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.195 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:53.193+0000 7f0cace00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:48:53.195 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:53.193+0000 7f0cace00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:48:53.195 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:53.193+0000 7f0cace00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:48:53.195 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:53.193+0000 7f0cace00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:48:53.195 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:53.193+0000 7f0cace00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:48:53.196 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:53.193+0000 7f0ca9200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:48:53.196 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:53.193+0000 7f0ca9200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:48:53.196 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:53.193+0000 7f0ca9200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:48:53.196 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:53.193+0000 7f0ca9200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:48:53.196 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:48:53.193+0000 7f0ca9200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:48:53.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.330+0000 7fbed2800700 1 -- 172.21.15.89:0/3985016298 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbecc15a640 con 0x7fbecc072580 2024-04-02T07:48:53.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.331+0000 7fbec3400700 1 -- 172.21.15.89:0/3985016298 <== mon.2 v2:172.21.15.113:3301/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4737 (secure 0 0 0) 0x7fbebc073f90 con 0x7fbecc072580 2024-04-02T07:48:53.333 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:53.333 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5037,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3788023013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3788023013},{"type":"v1","addr":"172.21.15.89:6835","nonce":3788023013}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5040,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4029308904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":4029308904},{"type":"v1","addr":"172.21.15.113:6835","nonce":4029308904}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5043,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2115217020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2115217020},{"type":"v1","addr":"172.21.15.113:6834","nonce":2115217020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:37:05.500422+0000","modified":"2024-04-02T07:37:08.737868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.89:6837/929002087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":929002087},{"type":"v1","addr":"172.21.15.89:6837","nonce":929002087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-02T07:48:53.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.333+0000 7fbed2800700 1 -- 172.21.15.89:0/3985016298 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbecc072580 msgr2=0x7fbecc14d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.333+0000 7fbed2800700 1 --2- 172.21.15.89:0/3985016298 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbecc072580 0x7fbecc14d830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00c700 tx=0x7fbebc00ca10 comp rx=0 tx=0).stop 2024-04-02T07:48:53.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.333+0000 7fbed2800700 1 -- 172.21.15.89:0/3985016298 shutdown_connections 2024-04-02T07:48:53.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.333+0000 7fbed2800700 1 --2- 172.21.15.89:0/3985016298 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbea806ae40 0x7fbea806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.333+0000 7fbed2800700 1 --2- 172.21.15.89:0/3985016298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbecc0b3af0 0x7fbecc15be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.333+0000 7fbed2800700 1 --2- 172.21.15.89:0/3985016298 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbecc072580 0x7fbecc14d830 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.333+0000 7fbed2800700 1 --2- 172.21.15.89:0/3985016298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbecc0b2e50 0x7fbecc15b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.333+0000 7fbed2800700 1 -- 172.21.15.89:0/3985016298 >> 172.21.15.89:0/3985016298 conn(0x7fbecc0cf190 msgr2=0x7fbecc0aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:53.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.333+0000 7fbed2800700 1 -- 172.21.15.89:0/3985016298 shutdown_connections 2024-04-02T07:48:53.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.333+0000 7fbed2800700 1 -- 172.21.15.89:0/3985016298 wait complete. 2024-04-02T07:48:53.336 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 17 2024-04-02T07:48:53.353 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:48:53.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.541+0000 7f28f4a00700 1 Processor -- start 2024-04-02T07:48:53.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.541+0000 7f28f4a00700 1 -- start start 2024-04-02T07:48:53.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.542+0000 7f28f4a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f28f00d0a40 0x7f28f00b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.542+0000 7f28f4a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f00d0ef0 con 0x7f28f015da50 2024-04-02T07:48:53.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.542+0000 7f28f4a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f00d24e0 con 0x7f28f00d05e0 2024-04-02T07:48:53.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.542+0000 7f28f4a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f00b95d0 con 0x7f28f00d0a40 2024-04-02T07:48:53.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.543+0000 7f28ede00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f28f015da50 0x7f28f00ce730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55726/0 (socket says 172.21.15.89:55726) 2024-04-02T07:48:53.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.543+0000 7f28ede00700 1 -- 172.21.15.89:0/246467895 learned_addr learned my addr 172.21.15.89:0/246467895 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:53.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.543+0000 7f28ee800700 1 --2- 172.21.15.89:0/246467895 >> v2:172.21.15.113:3300/0 conn(0x7f28f00d0a40 0x7f28f00b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:53.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.543+0000 7f28eca00700 1 -- 172.21.15.89:0/246467895 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038095371 0 0) 0x7f28f00d0ef0 con 0x7f28f015da50 2024-04-02T07:48:53.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.543+0000 7f28eca00700 1 -- 172.21.15.89:0/246467895 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28d40035b0 con 0x7f28f015da50 2024-04-02T07:48:53.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.543+0000 7f28eca00700 1 -- 172.21.15.89:0/246467895 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497637872 0 0) 0x7f28f00d24e0 con 0x7f28f00d05e0 2024-04-02T07:48:53.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.543+0000 7f28eca00700 1 -- 172.21.15.89:0/246467895 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28f00d0ef0 con 0x7f28f00d05e0 2024-04-02T07:48:53.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.543+0000 7f28ee800700 1 -- 172.21.15.89:0/246467895 >> v1:172.21.15.113:6789/0 conn(0x7f28f00d05e0 legacy=0x7f28f00b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.543+0000 7f28ee800700 1 -- 172.21.15.89:0/246467895 >> v1:172.21.15.89:6789/0 conn(0x7f28f015da50 legacy=0x7f28f00ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.543+0000 7f28ee800700 1 -- 172.21.15.89:0/246467895 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f00d27c0 con 0x7f28f00d0a40 2024-04-02T07:48:53.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.543+0000 7f28ee800700 1 --2- 172.21.15.89:0/246467895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28f00d0a40 0x7f28f00b70a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009d90 tx=0x7f28e800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d1dfdb14380ded server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:53.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.544+0000 7f28eca00700 1 -- 172.21.15.89:0/246467895 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e800e070 con 0x7f28f00d0a40 2024-04-02T07:48:53.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.544+0000 7f28eca00700 1 -- 172.21.15.89:0/246467895 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e800bb20 con 0x7f28f00d0a40 2024-04-02T07:48:53.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.544+0000 7f28eca00700 1 -- 172.21.15.89:0/246467895 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8012430 con 0x7f28f00d0a40 2024-04-02T07:48:53.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.544+0000 7f28f4a00700 1 -- 172.21.15.89:0/246467895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28f00d0a40 msgr2=0x7f28f00b70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.544+0000 7f28f4a00700 1 --2- 172.21.15.89:0/246467895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28f00d0a40 0x7f28f00b70a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009d90 tx=0x7f28e800b330 comp rx=0 tx=0).stop 2024-04-02T07:48:53.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.544+0000 7f28f4a00700 1 -- 172.21.15.89:0/246467895 shutdown_connections 2024-04-02T07:48:53.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.544+0000 7f28f4a00700 1 --2- 172.21.15.89:0/246467895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28f00d0a40 0x7f28f00b70a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.544+0000 7f28f4a00700 1 -- 172.21.15.89:0/246467895 >> 172.21.15.89:0/246467895 conn(0x7f28f00666a0 msgr2=0x7f28f0066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:53.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.544+0000 7f28f4a00700 1 -- 172.21.15.89:0/246467895 shutdown_connections 2024-04-02T07:48:53.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.544+0000 7f28f4a00700 1 -- 172.21.15.89:0/246467895 wait complete. 2024-04-02T07:48:53.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.546+0000 7f28f4a00700 1 Processor -- start 2024-04-02T07:48:53.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.546+0000 7f28f4a00700 1 -- start start 2024-04-02T07:48:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.547+0000 7f28f4a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28f00d05e0 0x7f28f01f2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.547+0000 7f28f4a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28f00d0a40 0x7f28f015fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.547+0000 7f28f4a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28f015da50 0x7f28f00caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.547+0000 7f28f4a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f0160430 con 0x7f28f015da50 2024-04-02T07:48:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.547+0000 7f28f4a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00ccf70 con 0x7f28f00d05e0 2024-04-02T07:48:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.547+0000 7f28f4a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00ccc90 con 0x7f28f00d0a40 2024-04-02T07:48:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.547+0000 7f28ee800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28f015da50 0x7f28f00caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.547+0000 7f28ee800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28f015da50 0x7f28f00caba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58498/0 (socket says 172.21.15.89:58498) 2024-04-02T07:48:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.547+0000 7f28ee800700 1 -- 172.21.15.89:0/3652861050 learned_addr learned my addr 172.21.15.89:0/3652861050 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:53.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.547+0000 7f28ede00700 1 --2- 172.21.15.89:0/3652861050 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28f00d05e0 0x7f28f01f2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:53.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.547+0000 7f28ed400700 1 --2- 172.21.15.89:0/3652861050 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28f00d0a40 0x7f28f015fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:53.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.548+0000 7f28ee800700 1 -- 172.21.15.89:0/3652861050 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28f00d0a40 msgr2=0x7f28f015fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.548+0000 7f28ee800700 1 --2- 172.21.15.89:0/3652861050 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28f00d0a40 0x7f28f015fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.548+0000 7f28ee800700 1 -- 172.21.15.89:0/3652861050 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28f00d05e0 msgr2=0x7f28f01f2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.548+0000 7f28ee800700 1 --2- 172.21.15.89:0/3652861050 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28f00d05e0 0x7f28f01f2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.548+0000 7f28ee800700 1 -- 172.21.15.89:0/3652861050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e8009a70 con 0x7f28f015da50 2024-04-02T07:48:53.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.548+0000 7f28ee800700 1 --2- 172.21.15.89:0/3652861050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28f015da50 0x7f28f00caba0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f28f00741d0 tx=0x7f28e8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:53.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.548+0000 7f28e6000700 1 -- 172.21.15.89:0/3652861050 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8012430 con 0x7f28f015da50 2024-04-02T07:48:53.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.548+0000 7f28e6000700 1 -- 172.21.15.89:0/3652861050 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e8020070 con 0x7f28f015da50 2024-04-02T07:48:53.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.548+0000 7f28f4a00700 1 -- 172.21.15.89:0/3652861050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f015dfd0 con 0x7f28f015da50 2024-04-02T07:48:53.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.548+0000 7f28e6000700 1 -- 172.21.15.89:0/3652861050 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e801ba70 con 0x7f28f015da50 2024-04-02T07:48:53.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.548+0000 7f28f4a00700 1 -- 172.21.15.89:0/3652861050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f015e590 con 0x7f28f015da50 2024-04-02T07:48:53.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.549+0000 7f28f4a00700 1 -- 172.21.15.89:0/3652861050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f015de30 con 0x7f28f015da50 2024-04-02T07:48:53.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.550+0000 7f28e6000700 1 -- 172.21.15.89:0/3652861050 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f28e801bbd0 con 0x7f28f015da50 2024-04-02T07:48:53.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.551+0000 7f28e6000700 1 -- 172.21.15.89:0/3652861050 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6732+0+0 (secure 0 0 0) 0x7f28e8022070 con 0x7f28f015da50 2024-04-02T07:48:53.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.556+0000 7f28e6000700 1 -- 172.21.15.89:0/3652861050 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f28f015de30 con 0x7f28f015da50 2024-04-02T07:48:53.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.761+0000 7f28f4a00700 1 -- 172.21.15.89:0/3652861050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28f015de30 con 0x7f28f015da50 2024-04-02T07:48:53.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.762+0000 7f28e6000700 1 -- 172.21.15.89:0/3652861050 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+17641 (secure 0 0 0) 0x7f28f015de30 con 0x7f28f015da50 2024-04-02T07:48:53.765 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:48:53.765 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":54,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:48:53.116189+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:37:03.665694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-02T07:37:04.473147+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.19","mappings":[{"from":4,"to":5}]},{"pgid":"5.1a","mappings":[{"from":4,"to":5}]},{"pgid":"5.1d","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/2289792465":"2024-04-03T07:48:53.116114+0000","172.21.15.89:0/1040272582":"2024-04-03T07:48:53.116114+0000","172.21.15.89:0/1510691253":"2024-04-03T07:48:53.116114+0000","172.21.15.89:0/1828720999":"2024-04-03T07:48:53.116114+0000","172.21.15.89:0/4293572429":"2024-04-03T07:48:53.116114+0000","172.21.15.89:0/1862742821":"2024-04-03T07:48:53.116114+0000","172.21.15.89:0/3312436996":"2024-04-03T07:48:53.116114+0000","172.21.15.89:0/399385398":"2024-04-03T07:37:22.446577+0000","172.21.15.89:6832/43357":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/350807093":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/2201396539":"2024-04-03T07:37:22.446577+0000","172.21.15.89:0/2645815547":"2024-04-03T07:37:22.446577+0000","172.21.15.89:6833/43357":"2024-04-03T07:37:22.446577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:48:53.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.765+0000 7f28f4a00700 1 -- 172.21.15.89:0/3652861050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28f015da50 msgr2=0x7f28f00caba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.765+0000 7f28f4a00700 1 --2- 172.21.15.89:0/3652861050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28f015da50 0x7f28f00caba0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f28f00741d0 tx=0x7f28e8004ad0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.765+0000 7f28f4a00700 1 -- 172.21.15.89:0/3652861050 shutdown_connections 2024-04-02T07:48:53.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.765+0000 7f28f4a00700 1 --2- 172.21.15.89:0/3652861050 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28f00d05e0 0x7f28f01f2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.765+0000 7f28f4a00700 1 --2- 172.21.15.89:0/3652861050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28f015da50 0x7f28f00caba0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.765+0000 7f28f4a00700 1 --2- 172.21.15.89:0/3652861050 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28f00d0a40 0x7f28f015fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.765+0000 7f28f4a00700 1 -- 172.21.15.89:0/3652861050 >> 172.21.15.89:0/3652861050 conn(0x7f28f00666a0 msgr2=0x7f28f00b7470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:53.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.765+0000 7f28f4a00700 1 -- 172.21.15.89:0/3652861050 shutdown_connections 2024-04-02T07:48:53.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.765+0000 7f28f4a00700 1 -- 172.21.15.89:0/3652861050 wait complete. 2024-04-02T07:48:53.785 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T07:48:53.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.976+0000 7f63a9000700 1 Processor -- start 2024-04-02T07:48:53.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.976+0000 7f63a9000700 1 -- start start 2024-04-02T07:48:53.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.977+0000 7f63a9000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40cbb30 0x7f63a40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.977+0000 7f63a9000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a40cc450 con 0x7f63a40cd020 2024-04-02T07:48:53.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.977+0000 7f63a9000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a40cdae0 con 0x7f63a40cd4a0 2024-04-02T07:48:53.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.977+0000 7f63a9000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40cddc0 con 0x7f63a40cbb30 2024-04-02T07:48:53.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.978+0000 7f63a1400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f63a40cd020 0x7f63a40b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55728/0 (socket says 172.21.15.89:55728) 2024-04-02T07:48:53.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.978+0000 7f63a1400700 1 -- 172.21.15.89:0/3231689294 learned_addr learned my addr 172.21.15.89:0/3231689294 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:53.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.978+0000 7f63a1e00700 1 --2- 172.21.15.89:0/3231689294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40cbb30 0x7f63a40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:53.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.978+0000 7f63a0a00700 1 -- 172.21.15.89:0/3231689294 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960435487 0 0) 0x7f63a40cc450 con 0x7f63a40cd020 2024-04-02T07:48:53.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.978+0000 7f63a0a00700 1 -- 172.21.15.89:0/3231689294 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63880035b0 con 0x7f63a40cd020 2024-04-02T07:48:53.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.978+0000 7f63a0a00700 1 -- 172.21.15.89:0/3231689294 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1591958934 0 0) 0x7f63880035b0 con 0x7f63a40cd020 2024-04-02T07:48:53.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.978+0000 7f63a0a00700 1 -- 172.21.15.89:0/3231689294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40cbb30 msgr2=0x7f63a40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:53.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.978+0000 7f63a0a00700 1 --2- 172.21.15.89:0/3231689294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40cbb30 0x7f63a40cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.978+0000 7f63a0a00700 1 -- 172.21.15.89:0/3231689294 >> v1:172.21.15.113:6789/0 conn(0x7f63a40cd4a0 legacy=0x7f63a406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.978+0000 7f63a0a00700 1 -- 172.21.15.89:0/3231689294 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a40ce0a0 con 0x7f63a40cd020 2024-04-02T07:48:53.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.978+0000 7f63a0a00700 1 -- 172.21.15.89:0/3231689294 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f6390001670 con 0x7f63a40cd020 2024-04-02T07:48:53.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.978+0000 7f63a0a00700 1 -- 172.21.15.89:0/3231689294 >> v1:172.21.15.89:6789/0 conn(0x7f63a40cd020 legacy=0x7f63a40b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a0a00700 1 --2- 172.21.15.89:0/3231689294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40cbb30 0x7f6388003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a0a00700 1 --2- 172.21.15.89:0/3231689294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6388003fb0 0x7f6388004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a0a00700 1 --2- 172.21.15.89:0/3231689294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6388004d90 0x7f6388005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a0a00700 1 -- 172.21.15.89:0/3231689294 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40cddc0 con 0x7f6388003fb0 2024-04-02T07:48:53.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a0a00700 1 -- 172.21.15.89:0/3231689294 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638800baf0 con 0x7f6388004d90 2024-04-02T07:48:53.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a0a00700 1 -- 172.21.15.89:0/3231689294 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638800bcb0 con 0x7f63a40cbb30 2024-04-02T07:48:53.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a1400700 1 --2- 172.21.15.89:0/3231689294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6388003fb0 0x7f6388004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:53.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a1e00700 1 --2- 172.21.15.89:0/3231689294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40cbb30 0x7f6388003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:53.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a2800700 1 --2- 172.21.15.89:0/3231689294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6388004d90 0x7f6388005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:53.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a1400700 1 -- 172.21.15.89:0/3231689294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40cbb30 msgr2=0x7f6388003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:53.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a1400700 1 --2- 172.21.15.89:0/3231689294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40cbb30 0x7f6388003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a1400700 1 -- 172.21.15.89:0/3231689294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6388004d90 msgr2=0x7f6388005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:53.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a1400700 1 --2- 172.21.15.89:0/3231689294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6388004d90 0x7f6388005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a1400700 1 -- 172.21.15.89:0/3231689294 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63880035b0 con 0x7f6388003fb0 2024-04-02T07:48:53.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a2800700 1 --2- 172.21.15.89:0/3231689294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6388004d90 0x7f6388005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:48:53.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.979+0000 7f63a1400700 1 --2- 172.21.15.89:0/3231689294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6388003fb0 0x7f6388004850 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f639000c240 tx=0x7f6388005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fefbb3d301cacbad server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:53.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.980+0000 7f63a0a00700 1 -- 172.21.15.89:0/3231689294 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6390007c10 con 0x7f6388003fb0 2024-04-02T07:48:53.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.980+0000 7f63a0a00700 1 -- 172.21.15.89:0/3231689294 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639000ce00 con 0x7f6388003fb0 2024-04-02T07:48:53.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.980+0000 7f63a0a00700 1 -- 172.21.15.89:0/3231689294 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6390005ba0 con 0x7f6388003fb0 2024-04-02T07:48:53.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.980+0000 7f63a9000700 1 -- 172.21.15.89:0/3231689294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6388003fb0 msgr2=0x7f6388004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.980+0000 7f63a9000700 1 --2- 172.21.15.89:0/3231689294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6388003fb0 0x7f6388004850 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f639000c240 tx=0x7f6388005880 comp rx=0 tx=0).stop 2024-04-02T07:48:53.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.980+0000 7f63a2800700 1 -- 172.21.15.89:0/3231689294 reap_dead start 2024-04-02T07:48:53.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.980+0000 7f63a9000700 1 -- 172.21.15.89:0/3231689294 shutdown_connections 2024-04-02T07:48:53.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.980+0000 7f63a9000700 1 -- 172.21.15.89:0/3231689294 >> 172.21.15.89:0/3231689294 conn(0x7f63a40b0d70 msgr2=0x7f63a40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:53.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.980+0000 7f63a9000700 1 -- 172.21.15.89:0/3231689294 shutdown_connections 2024-04-02T07:48:53.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.980+0000 7f63a9000700 1 -- 172.21.15.89:0/3231689294 wait complete. 2024-04-02T07:48:53.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.982+0000 7f63a9000700 1 Processor -- start 2024-04-02T07:48:53.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.982+0000 7f63a9000700 1 -- start start 2024-04-02T07:48:53.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.983+0000 7f63a9000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6388003fb0 0x7f63a4155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.983+0000 7f63a9000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40ad0a0 0x7f63a40ad520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.983+0000 7f63a9000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63a415dbd0 0x7f63a415e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:53.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.983+0000 7f63a9000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40c91e0 con 0x7f6388003fb0 2024-04-02T07:48:53.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.983+0000 7f63a9000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40ada60 con 0x7f63a415dbd0 2024-04-02T07:48:53.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.983+0000 7f63a9000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a415c040 con 0x7f63a40ad0a0 2024-04-02T07:48:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.983+0000 7f63a1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6388003fb0 0x7f63a4155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.983+0000 7f63a1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6388003fb0 0x7f63a4155670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58516/0 (socket says 172.21.15.89:58516) 2024-04-02T07:48:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.983+0000 7f63a1e00700 1 -- 172.21.15.89:0/2538898808 learned_addr learned my addr 172.21.15.89:0/2538898808 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.984+0000 7f63a1e00700 1 -- 172.21.15.89:0/2538898808 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40ad0a0 msgr2=0x7f63a40ad520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.984+0000 7f63a1e00700 1 --2- 172.21.15.89:0/2538898808 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40ad0a0 0x7f63a40ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.984+0000 7f63a1e00700 1 -- 172.21.15.89:0/2538898808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63a415dbd0 msgr2=0x7f63a415e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.984+0000 7f63a1e00700 1 --2- 172.21.15.89:0/2538898808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63a415dbd0 0x7f63a415e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.984+0000 7f63a1e00700 1 -- 172.21.15.89:0/2538898808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a415e5c0 con 0x7f6388003fb0 2024-04-02T07:48:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.984+0000 7f63a1e00700 1 --2- 172.21.15.89:0/2538898808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6388003fb0 0x7f63a4155670 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6394011aa0 tx=0x7f6394011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.984+0000 7f639a000700 1 -- 172.21.15.89:0/2538898808 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639400f980 con 0x7f6388003fb0 2024-04-02T07:48:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.984+0000 7f639a000700 1 -- 172.21.15.89:0/2538898808 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639400fae0 con 0x7f6388003fb0 2024-04-02T07:48:53.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.984+0000 7f639a000700 1 -- 172.21.15.89:0/2538898808 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639400fe90 con 0x7f6388003fb0 2024-04-02T07:48:53.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.984+0000 7f63a9000700 1 -- 172.21.15.89:0/2538898808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a4155bb0 con 0x7f6388003fb0 2024-04-02T07:48:53.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.984+0000 7f63a9000700 1 -- 172.21.15.89:0/2538898808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a41f6a30 con 0x7f6388003fb0 2024-04-02T07:48:53.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.985+0000 7f63a9000700 1 -- 172.21.15.89:0/2538898808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a4067720 con 0x7f6388003fb0 2024-04-02T07:48:53.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.986+0000 7f639a000700 1 -- 172.21.15.89:0/2538898808 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f6394014440 con 0x7f6388003fb0 2024-04-02T07:48:53.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.987+0000 7f639a000700 1 -- 172.21.15.89:0/2538898808 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6732+0+0 (secure 0 0 0) 0x7f63940898f0 con 0x7f6388003fb0 2024-04-02T07:48:53.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:53.991+0000 7f639a000700 1 -- 172.21.15.89:0/2538898808 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6394054af0 con 0x7f6388003fb0 2024-04-02T07:48:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.130+0000 7f639a000700 1 -- 172.21.15.89:0/2538898808 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 18) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f63940545a0 con 0x7f6388003fb0 2024-04-02T07:48:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.131+0000 7f639a000700 1 --2- 172.21.15.89:0/2538898808 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6384070760 0x7f6384072b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.133+0000 7f63a1400700 1 --2- 172.21.15.89:0/2538898808 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6384070760 0x7f6384072b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:54.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.137+0000 7f63a1400700 1 --2- 172.21.15.89:0/2538898808 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6384070760 0x7f6384072b50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f63a40cb250 tx=0x7f639000a000 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:54.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.248+0000 7f63a9000700 1 -- 172.21.15.89:0/2538898808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f63a415e5c0 con 0x7f6388003fb0 2024-04-02T07:48:54.251 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T07:48:54.249+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T07:48:54.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.257+0000 7f639a000700 1 -- 172.21.15.89:0/2538898808 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7f63a415e5c0 con 0x7f6388003fb0 2024-04-02T07:48:54.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.260+0000 7f63a9000700 1 -- 172.21.15.89:0/2538898808 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6384070760 msgr2=0x7f6384072b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:54.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.260+0000 7f63a9000700 1 --2- 172.21.15.89:0/2538898808 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6384070760 0x7f6384072b50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f63a40cb250 tx=0x7f639000a000 comp rx=0 tx=0).stop 2024-04-02T07:48:54.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.260+0000 7f63a9000700 1 -- 172.21.15.89:0/2538898808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6388003fb0 msgr2=0x7f63a4155670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:54.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.260+0000 7f63a9000700 1 --2- 172.21.15.89:0/2538898808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6388003fb0 0x7f63a4155670 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6394011aa0 tx=0x7f6394011e60 comp rx=0 tx=0).stop 2024-04-02T07:48:54.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.261+0000 7f63a9000700 1 -- 172.21.15.89:0/2538898808 shutdown_connections 2024-04-02T07:48:54.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.261+0000 7f63a9000700 1 --2- 172.21.15.89:0/2538898808 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6384070760 0x7f6384072b50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:54.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.261+0000 7f63a9000700 1 --2- 172.21.15.89:0/2538898808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63a415dbd0 0x7f63a415e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:54.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.261+0000 7f63a9000700 1 --2- 172.21.15.89:0/2538898808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6388003fb0 0x7f63a4155670 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:54.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.261+0000 7f63a9000700 1 --2- 172.21.15.89:0/2538898808 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40ad0a0 0x7f63a40ad520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:54.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.261+0000 7f63a9000700 1 -- 172.21.15.89:0/2538898808 >> 172.21.15.89:0/2538898808 conn(0x7f63a40b0d70 msgr2=0x7f63a415ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:54.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.261+0000 7f63a9000700 1 -- 172.21.15.89:0/2538898808 shutdown_connections 2024-04-02T07:48:54.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.261+0000 7f63a9000700 1 -- 172.21.15.89:0/2538898808 wait complete. 2024-04-02T07:48:54.266 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T07:48:54.282 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T07:48:54.326 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T07:48:54.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.476+0000 7f47c6200700 1 Processor -- start 2024-04-02T07:48:54.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.476+0000 7f47c6200700 1 -- start start 2024-04-02T07:48:54.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.477+0000 7f47c6200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f47c00d16a0 0x7f47c00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:54.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.477+0000 7f47c6200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f47c00d2370 0x7f47c00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:54.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.477+0000 7f47c6200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c0161420 con 0x7f47c015d9d0 2024-04-02T07:48:54.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.477+0000 7f47c6200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c00d4340 con 0x7f47c00d16a0 2024-04-02T07:48:54.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.477+0000 7f47c6200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c015edc0 con 0x7f47c00d2370 2024-04-02T07:48:54.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.478+0000 7f47bea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f47c00d16a0 0x7f47c00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:54.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.478+0000 7f47bea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f47c00d16a0 0x7f47c00d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58540/0 (socket says 172.21.15.89:58540) 2024-04-02T07:48:54.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.478+0000 7f47bea00700 1 -- 172.21.15.89:0/3734213892 learned_addr learned my addr 172.21.15.89:0/3734213892 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:54.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.478+0000 7f47bfe00700 1 --2- 172.21.15.89:0/3734213892 >> v2:172.21.15.113:3300/0 conn(0x7f47c00d2370 0x7f47c00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:54.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.478+0000 7f47bea00700 1 -- 172.21.15.89:0/3734213892 >> v2:172.21.15.113:3300/0 conn(0x7f47c00d2370 msgr2=0x7f47c00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:54.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.478+0000 7f47bea00700 1 --2- 172.21.15.89:0/3734213892 >> v2:172.21.15.113:3300/0 conn(0x7f47c00d2370 0x7f47c00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:54.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.478+0000 7f47bea00700 1 -- 172.21.15.89:0/3734213892 >> v1:172.21.15.113:6789/0 conn(0x7f47c015d9d0 legacy=0x7f47c00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:54.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.478+0000 7f47bea00700 1 -- 172.21.15.89:0/3734213892 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c0161700 con 0x7f47c00d16a0 2024-04-02T07:48:54.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.478+0000 7f47bea00700 1 --2- 172.21.15.89:0/3734213892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47c00d16a0 0x7f47c00d1e30 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f47c008ca10 tx=0x7f47ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c68517b57f840b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:54.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.478+0000 7f47be000700 1 -- 172.21.15.89:0/3734213892 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac005e30 con 0x7f47c00d16a0 2024-04-02T07:48:54.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.479+0000 7f47be000700 1 -- 172.21.15.89:0/3734213892 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ac004030 con 0x7f47c00d16a0 2024-04-02T07:48:54.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.479+0000 7f47be000700 1 -- 172.21.15.89:0/3734213892 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac0043e0 con 0x7f47c00d16a0 2024-04-02T07:48:54.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.479+0000 7f47c6200700 1 -- 172.21.15.89:0/3734213892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47c00d16a0 msgr2=0x7f47c00d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:54.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.479+0000 7f47c6200700 1 --2- 172.21.15.89:0/3734213892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47c00d16a0 0x7f47c00d1e30 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f47c008ca10 tx=0x7f47ac009bd0 comp rx=0 tx=0).stop 2024-04-02T07:48:54.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.479+0000 7f47c6200700 1 -- 172.21.15.89:0/3734213892 shutdown_connections 2024-04-02T07:48:54.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.479+0000 7f47c6200700 1 --2- 172.21.15.89:0/3734213892 >> v2:172.21.15.113:3300/0 conn(0x7f47c00d2370 0x7f47c00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:54.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.479+0000 7f47c6200700 1 --2- 172.21.15.89:0/3734213892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47c00d16a0 0x7f47c00d1e30 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:54.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.479+0000 7f47c6200700 1 -- 172.21.15.89:0/3734213892 >> 172.21.15.89:0/3734213892 conn(0x7f47c008aff0 msgr2=0x7f47c00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:54.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.479+0000 7f47c6200700 1 -- 172.21.15.89:0/3734213892 shutdown_connections 2024-04-02T07:48:54.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.479+0000 7f47c6200700 1 -- 172.21.15.89:0/3734213892 wait complete. 2024-04-02T07:48:54.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.481+0000 7f47c6200700 1 Processor -- start 2024-04-02T07:48:54.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.481+0000 7f47c6200700 1 -- start start 2024-04-02T07:48:54.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.482+0000 7f47c6200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47c00d16a0 0x7f47c00b8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:54.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.482+0000 7f47c6200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47c00d2370 0x7f47c00cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:54.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.482+0000 7f47c6200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47c015d9d0 0x7f47c0160300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:54.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.482+0000 7f47c6200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c00b0f20 con 0x7f47c015d9d0 2024-04-02T07:48:54.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.482+0000 7f47c6200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c00cbb40 con 0x7f47c00d16a0 2024-04-02T07:48:54.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.482+0000 7f47c6200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c00caf80 con 0x7f47c00d2370 2024-04-02T07:48:54.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.482+0000 7f47bfe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47c015d9d0 0x7f47c0160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:54.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.482+0000 7f47bfe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47c015d9d0 0x7f47c0160300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58550/0 (socket says 172.21.15.89:58550) 2024-04-02T07:48:54.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.482+0000 7f47bfe00700 1 -- 172.21.15.89:0/3743092012 learned_addr learned my addr 172.21.15.89:0/3743092012 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:54.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.482+0000 7f47bf400700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47c00d16a0 0x7f47c00b8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:54.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.482+0000 7f47bea00700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47c00d2370 0x7f47c00cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:54.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.483+0000 7f47bfe00700 1 -- 172.21.15.89:0/3743092012 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47c00d2370 msgr2=0x7f47c00cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:54.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.483+0000 7f47bfe00700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47c00d2370 0x7f47c00cbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:54.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.483+0000 7f47bfe00700 1 -- 172.21.15.89:0/3743092012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47c00d16a0 msgr2=0x7f47c00b8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:54.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.483+0000 7f47bfe00700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47c00d16a0 0x7f47c00b8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:54.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.483+0000 7f47bfe00700 1 -- 172.21.15.89:0/3743092012 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac009500 con 0x7f47c015d9d0 2024-04-02T07:48:54.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.483+0000 7f47bf400700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47c00d16a0 0x7f47c00b8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:48:54.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.483+0000 7f47bfe00700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47c015d9d0 0x7f47c0160300 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f47b8009dd0 tx=0x7f47b800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:54.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.483+0000 7f47b7400700 1 -- 172.21.15.89:0/3743092012 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b800cd40 con 0x7f47c015d9d0 2024-04-02T07:48:54.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.483+0000 7f47b7400700 1 -- 172.21.15.89:0/3743092012 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b800f040 con 0x7f47c015d9d0 2024-04-02T07:48:54.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.483+0000 7f47b7400700 1 -- 172.21.15.89:0/3743092012 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b80146c0 con 0x7f47c015d9d0 2024-04-02T07:48:54.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.483+0000 7f47c6200700 1 -- 172.21.15.89:0/3743092012 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c00cb1a0 con 0x7f47c015d9d0 2024-04-02T07:48:54.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.483+0000 7f47c6200700 1 -- 172.21.15.89:0/3743092012 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c00cb760 con 0x7f47c015d9d0 2024-04-02T07:48:54.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.484+0000 7f47c6200700 1 -- 172.21.15.89:0/3743092012 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c00d1ca0 con 0x7f47c015d9d0 2024-04-02T07:48:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.485+0000 7f47b7400700 1 -- 172.21.15.89:0/3743092012 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f47b8014820 con 0x7f47c015d9d0 2024-04-02T07:48:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.487+0000 7f47b7400700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f47a006aae0 0x7f47a006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.487+0000 7f47b7400700 1 -- 172.21.15.89:0/3743092012 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6818+0+0 (secure 0 0 0) 0x7f47b8089ce0 con 0x7f47c015d9d0 2024-04-02T07:48:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.487+0000 7f47bf400700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f47a006aae0 0x7f47a006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:54.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.487+0000 7f47bf400700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f47a006aae0 0x7f47a006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f47c00b1080 tx=0x7f47b0009040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:54.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.491+0000 7f47b7400700 1 -- 172.21.15.89:0/3743092012 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f47b808e070 con 0x7f47c015d9d0 2024-04-02T07:48:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:54.737+0000 7f47c6200700 1 -- 172.21.15.89:0/3743092012 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f47c0067720 con 0x7f47c015d9d0 2024-04-02T07:48:55.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.263+0000 7f47b7400700 1 -- 172.21.15.89:0/3743092012 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 19) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f47b8055170 con 0x7f47c015d9d0 2024-04-02T07:48:55.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.270+0000 7f47b7400700 1 -- 172.21.15.89:0/3743092012 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f47b8054da0 con 0x7f47c015d9d0 2024-04-02T07:48:55.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.273+0000 7f47c6200700 1 -- 172.21.15.89:0/3743092012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f47a006aae0 msgr2=0x7f47a006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:55.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.273+0000 7f47c6200700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f47a006aae0 0x7f47a006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f47c00b1080 tx=0x7f47b0009040 comp rx=0 tx=0).stop 2024-04-02T07:48:55.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.273+0000 7f47c6200700 1 -- 172.21.15.89:0/3743092012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47c015d9d0 msgr2=0x7f47c0160300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:55.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.273+0000 7f47c6200700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47c015d9d0 0x7f47c0160300 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f47b8009dd0 tx=0x7f47b800c3a0 comp rx=0 tx=0).stop 2024-04-02T07:48:55.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.273+0000 7f47c6200700 1 -- 172.21.15.89:0/3743092012 shutdown_connections 2024-04-02T07:48:55.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.273+0000 7f47c6200700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f47a006aae0 0x7f47a006cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:55.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.273+0000 7f47c6200700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47c00d16a0 0x7f47c00b8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:55.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.273+0000 7f47c6200700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47c015d9d0 0x7f47c0160300 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:55.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.273+0000 7f47c6200700 1 --2- 172.21.15.89:0/3743092012 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47c00d2370 0x7f47c00cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:55.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.273+0000 7f47c6200700 1 -- 172.21.15.89:0/3743092012 >> 172.21.15.89:0/3743092012 conn(0x7f47c008aff0 msgr2=0x7f47c00cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:55.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.273+0000 7f47c6200700 1 -- 172.21.15.89:0/3743092012 shutdown_connections 2024-04-02T07:48:55.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.273+0000 7f47c6200700 1 -- 172.21.15.89:0/3743092012 wait complete. 2024-04-02T07:48:55.294 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T07:48:55.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.528+0000 7fdc2b800700 1 Processor -- start 2024-04-02T07:48:55.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.529+0000 7fdc2b800700 1 -- start start 2024-04-02T07:48:55.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.530+0000 7fdc2b800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdc2415ddf0 0x7fdc240ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:55.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.530+0000 7fdc2b800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdc2415e1d0 0x7fdc240d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:55.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.530+0000 7fdc2b800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc240d2710 0x7fdc241612f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:55.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.530+0000 7fdc2b800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc240d2b80 con 0x7fdc2415e1d0 2024-04-02T07:48:55.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.530+0000 7fdc2b800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2415ec60 con 0x7fdc2415ddf0 2024-04-02T07:48:55.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.530+0000 7fdc2b800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc240ce8a0 con 0x7fdc240d2710 2024-04-02T07:48:55.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.531+0000 7fdc23e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdc2415e1d0 0x7fdc240d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:55.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.531+0000 7fdc23e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdc2415e1d0 0x7fdc240d21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58580/0 (socket says 172.21.15.89:58580) 2024-04-02T07:48:55.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.531+0000 7fdc23e00700 1 -- 172.21.15.89:0/1979170194 learned_addr learned my addr 172.21.15.89:0/1979170194 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:55.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.531+0000 7fdc28e00700 1 --2- 172.21.15.89:0/1979170194 >> v2:172.21.15.113:3300/0 conn(0x7fdc2415ddf0 0x7fdc240ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:55.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.531+0000 7fdc29800700 1 --2- 172.21.15.89:0/1979170194 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc240d2710 0x7fdc241612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:55.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.531+0000 7fdc23e00700 1 -- 172.21.15.89:0/1979170194 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc240d2710 msgr2=0x7fdc241612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:55.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.531+0000 7fdc23e00700 1 --2- 172.21.15.89:0/1979170194 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc240d2710 0x7fdc241612f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:55.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.531+0000 7fdc23e00700 1 -- 172.21.15.89:0/1979170194 >> v2:172.21.15.113:3300/0 conn(0x7fdc2415ddf0 msgr2=0x7fdc240ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:55.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.531+0000 7fdc23e00700 1 --2- 172.21.15.89:0/1979170194 >> v2:172.21.15.113:3300/0 conn(0x7fdc2415ddf0 0x7fdc240ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:55.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.531+0000 7fdc23e00700 1 -- 172.21.15.89:0/1979170194 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc240cdc90 con 0x7fdc2415e1d0 2024-04-02T07:48:55.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.531+0000 7fdc23e00700 1 --2- 172.21.15.89:0/1979170194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc2415e1d0 0x7fdc240d21d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdc2406b3e0 tx=0x7fdc10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbfbe6c23dff593 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:55.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.531+0000 7fdc23400700 1 -- 172.21.15.89:0/1979170194 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1000b870 con 0x7fdc2415e1d0 2024-04-02T07:48:55.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.532+0000 7fdc23400700 1 -- 172.21.15.89:0/1979170194 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1000b9d0 con 0x7fdc2415e1d0 2024-04-02T07:48:55.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.532+0000 7fdc23400700 1 -- 172.21.15.89:0/1979170194 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1000bd60 con 0x7fdc2415e1d0 2024-04-02T07:48:55.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.532+0000 7fdc2b800700 1 -- 172.21.15.89:0/1979170194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc2415e1d0 msgr2=0x7fdc240d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:55.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.532+0000 7fdc2b800700 1 --2- 172.21.15.89:0/1979170194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc2415e1d0 0x7fdc240d21d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdc2406b3e0 tx=0x7fdc10009bd0 comp rx=0 tx=0).stop 2024-04-02T07:48:55.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.532+0000 7fdc2b800700 1 -- 172.21.15.89:0/1979170194 shutdown_connections 2024-04-02T07:48:55.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.532+0000 7fdc2b800700 1 --2- 172.21.15.89:0/1979170194 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc240d2710 0x7fdc241612f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:55.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.532+0000 7fdc2b800700 1 --2- 172.21.15.89:0/1979170194 >> v2:172.21.15.113:3300/0 conn(0x7fdc2415ddf0 0x7fdc240ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:55.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.532+0000 7fdc2b800700 1 --2- 172.21.15.89:0/1979170194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc2415e1d0 0x7fdc240d21d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:55.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.532+0000 7fdc2b800700 1 -- 172.21.15.89:0/1979170194 >> 172.21.15.89:0/1979170194 conn(0x7fdc2408aff0 msgr2=0x7fdc240d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:55.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.532+0000 7fdc2b800700 1 -- 172.21.15.89:0/1979170194 shutdown_connections 2024-04-02T07:48:55.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.532+0000 7fdc2b800700 1 -- 172.21.15.89:0/1979170194 wait complete. 2024-04-02T07:48:55.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.534+0000 7fdc2b800700 1 Processor -- start 2024-04-02T07:48:55.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.534+0000 7fdc2b800700 1 -- start start 2024-04-02T07:48:55.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.535+0000 7fdc2b800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc240d2710 0x7fdc240c8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:55.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.535+0000 7fdc2b800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc2415ddf0 0x7fdc2406c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:55.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.535+0000 7fdc2b800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc2415e1d0 0x7fdc2406ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:55.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.535+0000 7fdc2b800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc240c85b0 con 0x7fdc2415ddf0 2024-04-02T07:48:55.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.535+0000 7fdc2b800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc240c9360 con 0x7fdc240d2710 2024-04-02T07:48:55.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.535+0000 7fdc2b800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc240c87e0 con 0x7fdc2415e1d0 2024-04-02T07:48:55.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.535+0000 7fdc23e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc2415ddf0 0x7fdc2406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:55.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.535+0000 7fdc23e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc2415ddf0 0x7fdc2406c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58592/0 (socket says 172.21.15.89:58592) 2024-04-02T07:48:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.535+0000 7fdc23e00700 1 -- 172.21.15.89:0/3302189712 learned_addr learned my addr 172.21.15.89:0/3302189712 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.535+0000 7fdc28e00700 1 --2- 172.21.15.89:0/3302189712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc240d2710 0x7fdc240c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.536+0000 7fdc29800700 1 --2- 172.21.15.89:0/3302189712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc2415e1d0 0x7fdc2406ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.536+0000 7fdc23e00700 1 -- 172.21.15.89:0/3302189712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc2415e1d0 msgr2=0x7fdc2406ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.536+0000 7fdc23e00700 1 --2- 172.21.15.89:0/3302189712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc2415e1d0 0x7fdc2406ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.536+0000 7fdc23e00700 1 -- 172.21.15.89:0/3302189712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc240d2710 msgr2=0x7fdc240c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.536+0000 7fdc23e00700 1 --2- 172.21.15.89:0/3302189712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc240d2710 0x7fdc240c8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.536+0000 7fdc23e00700 1 -- 172.21.15.89:0/3302189712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc10009550 con 0x7fdc2415ddf0 2024-04-02T07:48:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.536+0000 7fdc23e00700 1 --2- 172.21.15.89:0/3302189712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc2415ddf0 0x7fdc2406c910 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fdc10009c10 tx=0x7fdc10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.536+0000 7fdc20c00700 1 -- 172.21.15.89:0/3302189712 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc10004540 con 0x7fdc2415ddf0 2024-04-02T07:48:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.536+0000 7fdc20c00700 1 -- 172.21.15.89:0/3302189712 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc100036a0 con 0x7fdc2415ddf0 2024-04-02T07:48:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.536+0000 7fdc2b800700 1 -- 172.21.15.89:0/3302189712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc240c8ac0 con 0x7fdc2415ddf0 2024-04-02T07:48:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.536+0000 7fdc20c00700 1 -- 172.21.15.89:0/3302189712 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc10003a50 con 0x7fdc2415ddf0 2024-04-02T07:48:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.536+0000 7fdc2b800700 1 -- 172.21.15.89:0/3302189712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc240c9080 con 0x7fdc2415ddf0 2024-04-02T07:48:55.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.537+0000 7fdc2b800700 1 -- 172.21.15.89:0/3302189712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2415edf0 con 0x7fdc2415ddf0 2024-04-02T07:48:55.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.538+0000 7fdc20c00700 1 -- 172.21.15.89:0/3302189712 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fdc10021070 con 0x7fdc2415ddf0 2024-04-02T07:48:55.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.539+0000 7fdc20c00700 1 --2- 172.21.15.89:0/3302189712 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdc0406ad20 0x7fdc0406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:55.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.539+0000 7fdc20c00700 1 -- 172.21.15.89:0/3302189712 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6818+0+0 (secure 0 0 0) 0x7fdc1008ae30 con 0x7fdc2415ddf0 2024-04-02T07:48:55.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.539+0000 7fdc28e00700 1 --2- 172.21.15.89:0/3302189712 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdc0406ad20 0x7fdc0406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:55.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.540+0000 7fdc28e00700 1 --2- 172.21.15.89:0/3302189712 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdc0406ad20 0x7fdc0406d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdc18006b20 tx=0x7fdc18008040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:55.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.544+0000 7fdc20c00700 1 -- 172.21.15.89:0/3302189712 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdc10055c70 con 0x7fdc2415ddf0 2024-04-02T07:48:55.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:55.744+0000 7fdc2b800700 1 -- 172.21.15.89:0/3302189712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdc24067720 con 0x7fdc2415ddf0 2024-04-02T07:48:56.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.272+0000 7fdc20c00700 1 -- 172.21.15.89:0/3302189712 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v56) v1 ==== 181+0+0 (secure 0 0 0) 0x7fdc100150d0 con 0x7fdc2415ddf0 2024-04-02T07:48:56.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.276+0000 7fdc2b800700 1 -- 172.21.15.89:0/3302189712 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdc0406ad20 msgr2=0x7fdc0406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:56.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.276+0000 7fdc2b800700 1 --2- 172.21.15.89:0/3302189712 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdc0406ad20 0x7fdc0406d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdc18006b20 tx=0x7fdc18008040 comp rx=0 tx=0).stop 2024-04-02T07:48:56.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.276+0000 7fdc2b800700 1 -- 172.21.15.89:0/3302189712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc2415ddf0 msgr2=0x7fdc2406c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:56.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.276+0000 7fdc2b800700 1 --2- 172.21.15.89:0/3302189712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc2415ddf0 0x7fdc2406c910 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fdc10009c10 tx=0x7fdc10004750 comp rx=0 tx=0).stop 2024-04-02T07:48:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.276+0000 7fdc2b800700 1 -- 172.21.15.89:0/3302189712 shutdown_connections 2024-04-02T07:48:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.276+0000 7fdc2b800700 1 --2- 172.21.15.89:0/3302189712 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdc0406ad20 0x7fdc0406d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.277+0000 7fdc2b800700 1 --2- 172.21.15.89:0/3302189712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc240d2710 0x7fdc240c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.277+0000 7fdc2b800700 1 --2- 172.21.15.89:0/3302189712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc2415e1d0 0x7fdc2406ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.277+0000 7fdc2b800700 1 --2- 172.21.15.89:0/3302189712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc2415ddf0 0x7fdc2406c910 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.277+0000 7fdc2b800700 1 -- 172.21.15.89:0/3302189712 >> 172.21.15.89:0/3302189712 conn(0x7fdc2408aff0 msgr2=0x7fdc240d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.277+0000 7fdc2b800700 1 -- 172.21.15.89:0/3302189712 shutdown_connections 2024-04-02T07:48:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.277+0000 7fdc2b800700 1 -- 172.21.15.89:0/3302189712 wait complete. 2024-04-02T07:48:56.282 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T07:48:56.301 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T07:48:56.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.501+0000 7f1eddc00700 1 Processor -- start 2024-04-02T07:48:56.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.501+0000 7f1eddc00700 1 -- start start 2024-04-02T07:48:56.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.502+0000 7f1eddc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ed80d28e0 0x7f1ed8168680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:56.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.502+0000 7f1eddc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ed80ce8a0 con 0x7f1ed815ddf0 2024-04-02T07:48:56.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.502+0000 7f1eddc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ed80cdc90 con 0x7f1ed80ced30 2024-04-02T07:48:56.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.502+0000 7f1eddc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed815ec60 con 0x7f1ed80d28e0 2024-04-02T07:48:56.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.502+0000 7f1ed6a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1ed815ddf0 0x7f1ed815e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55752/0 (socket says 172.21.15.89:55752) 2024-04-02T07:48:56.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.502+0000 7f1ed6a00700 1 -- 172.21.15.89:0/4059852872 learned_addr learned my addr 172.21.15.89:0/4059852872 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:56.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.503+0000 7f1ed7400700 1 --2- 172.21.15.89:0/4059852872 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ed80d28e0 0x7f1ed8168680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:56.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.503+0000 7f1ed5600700 1 -- 172.21.15.89:0/4059852872 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692900912 0 0) 0x7f1ed80ce8a0 con 0x7f1ed815ddf0 2024-04-02T07:48:56.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.503+0000 7f1ed5600700 1 -- 172.21.15.89:0/4059852872 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ebc0035b0 con 0x7f1ed815ddf0 2024-04-02T07:48:56.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.503+0000 7f1ed5600700 1 -- 172.21.15.89:0/4059852872 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607970277 0 0) 0x7f1ed80cdc90 con 0x7f1ed80ced30 2024-04-02T07:48:56.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.503+0000 7f1ed5600700 1 -- 172.21.15.89:0/4059852872 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ed80ce8a0 con 0x7f1ed80ced30 2024-04-02T07:48:56.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.503+0000 7f1ed7400700 1 -- 172.21.15.89:0/4059852872 >> v1:172.21.15.113:6789/0 conn(0x7f1ed80ced30 legacy=0x7f1ed80d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:56.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.503+0000 7f1ed7400700 1 -- 172.21.15.89:0/4059852872 >> v1:172.21.15.89:6789/0 conn(0x7f1ed815ddf0 legacy=0x7f1ed815e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:56.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.503+0000 7f1ed5600700 1 -- 172.21.15.89:0/4059852872 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 917653416 0 0) 0x7f1ebc0035b0 con 0x7f1ed815ddf0 2024-04-02T07:48:56.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.503+0000 7f1ed7400700 1 -- 172.21.15.89:0/4059852872 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed80cdf70 con 0x7f1ed80d28e0 2024-04-02T07:48:56.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.504+0000 7f1ed7400700 1 --2- 172.21.15.89:0/4059852872 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ed80d28e0 0x7f1ed8168680 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009dc0 tx=0x7f1ed000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fbd18f690700f05 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:56.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.504+0000 7f1ed5600700 1 -- 172.21.15.89:0/4059852872 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed000e070 con 0x7f1ed80d28e0 2024-04-02T07:48:56.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.504+0000 7f1ed5600700 1 -- 172.21.15.89:0/4059852872 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed0003a20 con 0x7f1ed80d28e0 2024-04-02T07:48:56.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.504+0000 7f1ed5600700 1 -- 172.21.15.89:0/4059852872 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8001670 con 0x7f1ed80d28e0 2024-04-02T07:48:56.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.504+0000 7f1eddc00700 1 -- 172.21.15.89:0/4059852872 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ed80d28e0 msgr2=0x7f1ed8168680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:56.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.504+0000 7f1eddc00700 1 --2- 172.21.15.89:0/4059852872 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ed80d28e0 0x7f1ed8168680 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009dc0 tx=0x7f1ed000b330 comp rx=0 tx=0).stop 2024-04-02T07:48:56.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.504+0000 7f1eddc00700 1 -- 172.21.15.89:0/4059852872 shutdown_connections 2024-04-02T07:48:56.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.504+0000 7f1eddc00700 1 --2- 172.21.15.89:0/4059852872 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ed80d28e0 0x7f1ed8168680 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:56.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.504+0000 7f1eddc00700 1 -- 172.21.15.89:0/4059852872 >> 172.21.15.89:0/4059852872 conn(0x7f1ed808aff0 msgr2=0x7f1ed80d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:56.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.505+0000 7f1eddc00700 1 -- 172.21.15.89:0/4059852872 shutdown_connections 2024-04-02T07:48:56.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.505+0000 7f1eddc00700 1 -- 172.21.15.89:0/4059852872 wait complete. 2024-04-02T07:48:56.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.506+0000 7f1eddc00700 1 Processor -- start 2024-04-02T07:48:56.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.506+0000 7f1eddc00700 1 -- start start 2024-04-02T07:48:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.507+0000 7f1eddc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ed80ced30 0x7f1ed80b5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.507+0000 7f1eddc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ed815ddf0 0x7f1ed80b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.507+0000 7f1eddc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ed80ca5e0 0x7f1ed80b8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.507+0000 7f1eddc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed80c7d80 con 0x7f1ed815ddf0 2024-04-02T07:48:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.507+0000 7f1eddc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed80b4fe0 con 0x7f1ed80ca5e0 2024-04-02T07:48:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.507+0000 7f1eddc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed80cab80 con 0x7f1ed80ced30 2024-04-02T07:48:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.507+0000 7f1ed6000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ed815ddf0 0x7f1ed80b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.508+0000 7f1ed6a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ed80ced30 0x7f1ed80b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.508+0000 7f1ed6000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ed815ddf0 0x7f1ed80b7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58604/0 (socket says 172.21.15.89:58604) 2024-04-02T07:48:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.508+0000 7f1ed6000700 1 -- 172.21.15.89:0/2362945952 learned_addr learned my addr 172.21.15.89:0/2362945952 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.508+0000 7f1ed7400700 1 --2- 172.21.15.89:0/2362945952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ed80ca5e0 0x7f1ed80b8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.508+0000 7f1ed6000700 1 -- 172.21.15.89:0/2362945952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ed80ced30 msgr2=0x7f1ed80b5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:56.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.508+0000 7f1ed6000700 1 --2- 172.21.15.89:0/2362945952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ed80ced30 0x7f1ed80b5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:56.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.508+0000 7f1ed6000700 1 -- 172.21.15.89:0/2362945952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ed80ca5e0 msgr2=0x7f1ed80b8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:56.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.508+0000 7f1ed6000700 1 --2- 172.21.15.89:0/2362945952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ed80ca5e0 0x7f1ed80b8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:56.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.508+0000 7f1ed6000700 1 -- 172.21.15.89:0/2362945952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed0009a70 con 0x7f1ed815ddf0 2024-04-02T07:48:56.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.508+0000 7f1ed6000700 1 --2- 172.21.15.89:0/2362945952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ed815ddf0 0x7f1ed80b7c30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400c520 tx=0x7f1ec400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:56.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.508+0000 7f1ecea00700 1 -- 172.21.15.89:0/2362945952 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec40052d0 con 0x7f1ed815ddf0 2024-04-02T07:48:56.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.508+0000 7f1ecea00700 1 -- 172.21.15.89:0/2362945952 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec4005430 con 0x7f1ed815ddf0 2024-04-02T07:48:56.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.508+0000 7f1ecea00700 1 -- 172.21.15.89:0/2362945952 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec40038c0 con 0x7f1ed815ddf0 2024-04-02T07:48:56.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.508+0000 7f1eddc00700 1 -- 172.21.15.89:0/2362945952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed80cae60 con 0x7f1ed815ddf0 2024-04-02T07:48:56.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.509+0000 7f1eddc00700 1 -- 172.21.15.89:0/2362945952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed80cb420 con 0x7f1ed815ddf0 2024-04-02T07:48:56.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.510+0000 7f1eddc00700 1 -- 172.21.15.89:0/2362945952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed80c7ef0 con 0x7f1ed815ddf0 2024-04-02T07:48:56.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.510+0000 7f1ecea00700 1 -- 172.21.15.89:0/2362945952 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f1ec4003a20 con 0x7f1ed815ddf0 2024-04-02T07:48:56.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.511+0000 7f1ecea00700 1 --2- 172.21.15.89:0/2362945952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1eb806ad20 0x7f1eb806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:56.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.511+0000 7f1ecea00700 1 -- 172.21.15.89:0/2362945952 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6380+0+0 (secure 0 0 0) 0x7f1ec40889e0 con 0x7f1ed815ddf0 2024-04-02T07:48:56.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.511+0000 7f1ed6a00700 1 --2- 172.21.15.89:0/2362945952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1eb806ad20 0x7f1eb806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:56.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.512+0000 7f1ed6a00700 1 --2- 172.21.15.89:0/2362945952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1eb806ad20 0x7f1eb806d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8000f60 tx=0x7f1ec8001690 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:56.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.515+0000 7f1ecea00700 1 -- 172.21.15.89:0/2362945952 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ec4053940 con 0x7f1ed815ddf0 2024-04-02T07:48:56.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:56.716+0000 7f1eddc00700 1 -- 172.21.15.89:0/2362945952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1ed8067720 con 0x7f1ed815ddf0 2024-04-02T07:48:57.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.286+0000 7f1ecea00700 1 -- 172.21.15.89:0/2362945952 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1ec4053d10 con 0x7f1ed815ddf0 2024-04-02T07:48:57.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.288+0000 7f1ecea00700 1 -- 172.21.15.89:0/2362945952 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v57) v1 ==== 169+0+0 (secure 0 0 0) 0x7f1ec40850c0 con 0x7f1ed815ddf0 2024-04-02T07:48:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.294+0000 7f1eddc00700 1 -- 172.21.15.89:0/2362945952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1eb806ad20 msgr2=0x7f1eb806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.294+0000 7f1eddc00700 1 --2- 172.21.15.89:0/2362945952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1eb806ad20 0x7f1eb806d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8000f60 tx=0x7f1ec8001690 comp rx=0 tx=0).stop 2024-04-02T07:48:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.294+0000 7f1eddc00700 1 -- 172.21.15.89:0/2362945952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ed815ddf0 msgr2=0x7f1ed80b7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.294+0000 7f1eddc00700 1 --2- 172.21.15.89:0/2362945952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ed815ddf0 0x7f1ed80b7c30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400c520 tx=0x7f1ec400c830 comp rx=0 tx=0).stop 2024-04-02T07:48:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.294+0000 7f1eddc00700 1 -- 172.21.15.89:0/2362945952 shutdown_connections 2024-04-02T07:48:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.294+0000 7f1eddc00700 1 --2- 172.21.15.89:0/2362945952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1eb806ad20 0x7f1eb806d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.294+0000 7f1eddc00700 1 --2- 172.21.15.89:0/2362945952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ed80ca5e0 0x7f1ed80b8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.294+0000 7f1eddc00700 1 --2- 172.21.15.89:0/2362945952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ed80ced30 0x7f1ed80b5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.294+0000 7f1eddc00700 1 --2- 172.21.15.89:0/2362945952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ed815ddf0 0x7f1ed80b7c30 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.294+0000 7f1eddc00700 1 -- 172.21.15.89:0/2362945952 >> 172.21.15.89:0/2362945952 conn(0x7f1ed808aff0 msgr2=0x7f1ed806cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.294+0000 7f1eddc00700 1 -- 172.21.15.89:0/2362945952 shutdown_connections 2024-04-02T07:48:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.294+0000 7f1eddc00700 1 -- 172.21.15.89:0/2362945952 wait complete. 2024-04-02T07:48:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T07:48:57.317 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T07:48:57.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.509+0000 7fbbb5200700 1 Processor -- start 2024-04-02T07:48:57.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.510+0000 7fbbb5200700 1 -- start start 2024-04-02T07:48:57.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.511+0000 7fbbb5200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbbb00cd020 0x7fbbb00cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:57.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.511+0000 7fbbb5200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbbb00b2070 0x7fbbb00b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.511+0000 7fbbb5200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb00cdae0 con 0x7fbbb00cbb30 2024-04-02T07:48:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.511+0000 7fbbb5200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbb00cddc0 con 0x7fbbb00b2070 2024-04-02T07:48:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.511+0000 7fbbb5200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbb00ce0a0 con 0x7fbbb00cd020 2024-04-02T07:48:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.511+0000 7fbbae800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbbb00b2070 0x7fbbb00b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.511+0000 7fbbae800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbbb00b2070 0x7fbbb00b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58614/0 (socket says 172.21.15.89:58614) 2024-04-02T07:48:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.511+0000 7fbbae800700 1 -- 172.21.15.89:0/2425330051 learned_addr learned my addr 172.21.15.89:0/2425330051 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.511+0000 7fbbad400700 1 --2- 172.21.15.89:0/2425330051 >> v2:172.21.15.113:3300/0 conn(0x7fbbb00cd020 0x7fbbb00cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.511+0000 7fbbae800700 1 -- 172.21.15.89:0/2425330051 >> v2:172.21.15.113:3300/0 conn(0x7fbbb00cd020 msgr2=0x7fbbb00cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.511+0000 7fbbae800700 1 --2- 172.21.15.89:0/2425330051 >> v2:172.21.15.113:3300/0 conn(0x7fbbb00cd020 0x7fbbb00cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.511+0000 7fbbae800700 1 -- 172.21.15.89:0/2425330051 >> v1:172.21.15.113:6789/0 conn(0x7fbbb00cbb30 legacy=0x7fbbb00cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.511+0000 7fbbae800700 1 -- 172.21.15.89:0/2425330051 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb006dc20 con 0x7fbbb00b2070 2024-04-02T07:48:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.512+0000 7fbbae800700 1 --2- 172.21.15.89:0/2425330051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbb00b2070 0x7fbbb00b2500 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00b1e30 tx=0x7fbba800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629f7d9a1f61c9ec server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:57.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.512+0000 7fbbaca00700 1 -- 172.21.15.89:0/2425330051 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba800e070 con 0x7fbbb00b2070 2024-04-02T07:48:57.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.512+0000 7fbbaca00700 1 -- 172.21.15.89:0/2425330051 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba800b910 con 0x7fbbb00b2070 2024-04-02T07:48:57.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.512+0000 7fbbaca00700 1 -- 172.21.15.89:0/2425330051 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba800bcc0 con 0x7fbbb00b2070 2024-04-02T07:48:57.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.512+0000 7fbbb5200700 1 -- 172.21.15.89:0/2425330051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbb00b2070 msgr2=0x7fbbb00b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:57.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.512+0000 7fbbb5200700 1 --2- 172.21.15.89:0/2425330051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbb00b2070 0x7fbbb00b2500 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00b1e30 tx=0x7fbba800b330 comp rx=0 tx=0).stop 2024-04-02T07:48:57.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.513+0000 7fbbb5200700 1 -- 172.21.15.89:0/2425330051 shutdown_connections 2024-04-02T07:48:57.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.513+0000 7fbbb5200700 1 --2- 172.21.15.89:0/2425330051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbb00b2070 0x7fbbb00b2500 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.513+0000 7fbbb5200700 1 --2- 172.21.15.89:0/2425330051 >> v2:172.21.15.113:3300/0 conn(0x7fbbb00cd020 0x7fbbb00cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.513+0000 7fbbb5200700 1 -- 172.21.15.89:0/2425330051 >> 172.21.15.89:0/2425330051 conn(0x7fbbb00b0d70 msgr2=0x7fbbb00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:57.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.513+0000 7fbbb5200700 1 -- 172.21.15.89:0/2425330051 shutdown_connections 2024-04-02T07:48:57.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.513+0000 7fbbb5200700 1 -- 172.21.15.89:0/2425330051 wait complete. 2024-04-02T07:48:57.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.514+0000 7fbbb5200700 1 Processor -- start 2024-04-02T07:48:57.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.515+0000 7fbbb5200700 1 -- start start 2024-04-02T07:48:57.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.515+0000 7fbbb5200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbb00b2070 0x7fbbb0072690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.515+0000 7fbbb5200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbb00cbb30 0x7fbbb00c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbb5200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbbb00cd020 0x7fbbb015a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbb5200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb00c8c20 con 0x7fbbb00b2070 2024-04-02T07:48:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbb5200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb00c8f00 con 0x7fbbb00cbb30 2024-04-02T07:48:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbb5200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb00c91e0 con 0x7fbbb00cd020 2024-04-02T07:48:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbade00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbb00b2070 0x7fbbb0072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbade00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbb00b2070 0x7fbbb0072690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58618/0 (socket says 172.21.15.89:58618) 2024-04-02T07:48:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbade00700 1 -- 172.21.15.89:0/651745889 learned_addr learned my addr 172.21.15.89:0/651745889 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbad400700 1 --2- 172.21.15.89:0/651745889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbb00cbb30 0x7fbbb00c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbae800700 1 --2- 172.21.15.89:0/651745889 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbbb00cd020 0x7fbbb015a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbade00700 1 -- 172.21.15.89:0/651745889 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbbb00cd020 msgr2=0x7fbbb015a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbade00700 1 --2- 172.21.15.89:0/651745889 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbbb00cd020 0x7fbbb015a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbade00700 1 -- 172.21.15.89:0/651745889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbb00cbb30 msgr2=0x7fbbb00c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:57.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbade00700 1 --2- 172.21.15.89:0/651745889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbb00cbb30 0x7fbbb00c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbade00700 1 -- 172.21.15.89:0/651745889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba8009b30 con 0x7fbbb00b2070 2024-04-02T07:48:57.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.516+0000 7fbbade00700 1 --2- 172.21.15.89:0/651745889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbb00b2070 0x7fbbb0072690 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fbba000c4e0 tx=0x7fbba000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:57.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.517+0000 7fbba6000700 1 -- 172.21.15.89:0/651745889 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba00052d0 con 0x7fbbb00b2070 2024-04-02T07:48:57.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.517+0000 7fbba6000700 1 -- 172.21.15.89:0/651745889 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba0005430 con 0x7fbbb00b2070 2024-04-02T07:48:57.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.517+0000 7fbba6000700 1 -- 172.21.15.89:0/651745889 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba00038c0 con 0x7fbbb00b2070 2024-04-02T07:48:57.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.517+0000 7fbbb5200700 1 -- 172.21.15.89:0/651745889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb00c94c0 con 0x7fbbb00b2070 2024-04-02T07:48:57.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.517+0000 7fbbb5200700 1 -- 172.21.15.89:0/651745889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb00c8930 con 0x7fbbb00b2070 2024-04-02T07:48:57.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.518+0000 7fbbb5200700 1 -- 172.21.15.89:0/651745889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb00cd400 con 0x7fbbb00b2070 2024-04-02T07:48:57.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.519+0000 7fbba6000700 1 -- 172.21.15.89:0/651745889 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbba0003a20 con 0x7fbbb00b2070 2024-04-02T07:48:57.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.520+0000 7fbba6000700 1 --2- 172.21.15.89:0/651745889 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbb9006adb0 0x7fbb9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:57.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.520+0000 7fbba6000700 1 -- 172.21.15.89:0/651745889 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5850+0+0 (secure 0 0 0) 0x7fbba0088960 con 0x7fbbb00b2070 2024-04-02T07:48:57.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.520+0000 7fbbad400700 1 --2- 172.21.15.89:0/651745889 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbb9006adb0 0x7fbb9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:57.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.520+0000 7fbbad400700 1 --2- 172.21.15.89:0/651745889 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbb9006adb0 0x7fbb9006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00ae370 tx=0x7fbb9c009210 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:57.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.524+0000 7fbba6000700 1 -- 172.21.15.89:0/651745889 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbba0053900 con 0x7fbbb00b2070 2024-04-02T07:48:57.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.729+0000 7fbbb5200700 1 -- 172.21.15.89:0/651745889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fbbb0067720 con 0x7fbbb00b2070 2024-04-02T07:48:57.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.733+0000 7fbba6000700 1 -- 172.21.15.89:0/651745889 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v13) v1 ==== 115+0+0 (secure 0 0 0) 0x7fbba00170b0 con 0x7fbbb00b2070 2024-04-02T07:48:57.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.736+0000 7fbbb5200700 1 -- 172.21.15.89:0/651745889 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbb9006adb0 msgr2=0x7fbb9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:57.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.736+0000 7fbbb5200700 1 --2- 172.21.15.89:0/651745889 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbb9006adb0 0x7fbb9006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00ae370 tx=0x7fbb9c009210 comp rx=0 tx=0).stop 2024-04-02T07:48:57.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.736+0000 7fbbb5200700 1 -- 172.21.15.89:0/651745889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbb00b2070 msgr2=0x7fbbb0072690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:57.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.736+0000 7fbbb5200700 1 --2- 172.21.15.89:0/651745889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbb00b2070 0x7fbbb0072690 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fbba000c4e0 tx=0x7fbba000c7f0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.736+0000 7fbbb5200700 1 -- 172.21.15.89:0/651745889 shutdown_connections 2024-04-02T07:48:57.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.736+0000 7fbbb5200700 1 --2- 172.21.15.89:0/651745889 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbb9006adb0 0x7fbb9006d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.736+0000 7fbbb5200700 1 --2- 172.21.15.89:0/651745889 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbbb00cd020 0x7fbbb015a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.736+0000 7fbbb5200700 1 --2- 172.21.15.89:0/651745889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbb00b2070 0x7fbbb0072690 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.736+0000 7fbbb5200700 1 --2- 172.21.15.89:0/651745889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbb00cbb30 0x7fbbb00c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.736+0000 7fbbb5200700 1 -- 172.21.15.89:0/651745889 >> 172.21.15.89:0/651745889 conn(0x7fbbb00b0d70 msgr2=0x7fbbb00ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:57.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.736+0000 7fbbb5200700 1 -- 172.21.15.89:0/651745889 shutdown_connections 2024-04-02T07:48:57.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.736+0000 7fbbb5200700 1 -- 172.21.15.89:0/651745889 wait complete. 2024-04-02T07:48:57.757 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T07:48:57.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.947+0000 7f0ba5a00700 1 Processor -- start 2024-04-02T07:48:57.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.947+0000 7f0ba5a00700 1 -- start start 2024-04-02T07:48:57.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.948+0000 7f0ba5a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ba00cbb30 0x7f0ba00cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:57.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.948+0000 7f0ba5a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0ba00cd480 0x7f0ba006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:57.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.948+0000 7f0ba5a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba00cc450 con 0x7f0ba00cd020 2024-04-02T07:48:57.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.948+0000 7f0ba5a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba00cd8f0 con 0x7f0ba00cbb30 2024-04-02T07:48:57.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.948+0000 7f0ba5a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba00cdae0 con 0x7f0ba00cd480 2024-04-02T07:48:57.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.949+0000 7f0b9ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ba00cbb30 0x7f0ba00cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:57.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.949+0000 7f0b9e000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0ba00cd020 0x7f0ba00b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55768/0 (socket says 172.21.15.89:55768) 2024-04-02T07:48:57.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.949+0000 7f0b9e000700 1 -- 172.21.15.89:0/1868505198 learned_addr learned my addr 172.21.15.89:0/1868505198 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:57.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.949+0000 7f0b9ea00700 1 -- 172.21.15.89:0/1868505198 >> v2:172.21.15.113:3300/0 conn(0x7f0ba00cd480 msgr2=0x7f0ba006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:57.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.949+0000 7f0b9ea00700 1 --2- 172.21.15.89:0/1868505198 >> v2:172.21.15.113:3300/0 conn(0x7f0ba00cd480 0x7f0ba006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.949+0000 7f0b9ea00700 1 -- 172.21.15.89:0/1868505198 >> v1:172.21.15.89:6789/0 conn(0x7f0ba00cd020 legacy=0x7f0ba00b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:57.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.949+0000 7f0b9ea00700 1 -- 172.21.15.89:0/1868505198 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba00cddc0 con 0x7f0ba00cbb30 2024-04-02T07:48:57.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.949+0000 7f0b9ea00700 1 --2- 172.21.15.89:0/1868505198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba00cbb30 0x7f0ba00cbf10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0ba006b6d0 tx=0x7f0b90009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76cfd67eb2d8bfd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:57.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.950+0000 7f0b9d600700 1 -- 172.21.15.89:0/1868505198 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b90011830 con 0x7f0ba00cbb30 2024-04-02T07:48:57.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.950+0000 7f0b9d600700 1 -- 172.21.15.89:0/1868505198 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b90011990 con 0x7f0ba00cbb30 2024-04-02T07:48:57.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.950+0000 7f0b9d600700 1 -- 172.21.15.89:0/1868505198 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b90011d20 con 0x7f0ba00cbb30 2024-04-02T07:48:57.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.950+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1868505198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba00cbb30 msgr2=0x7f0ba00cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:57.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.950+0000 7f0ba5a00700 1 --2- 172.21.15.89:0/1868505198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba00cbb30 0x7f0ba00cbf10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0ba006b6d0 tx=0x7f0b90009b20 comp rx=0 tx=0).stop 2024-04-02T07:48:57.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.950+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1868505198 shutdown_connections 2024-04-02T07:48:57.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.950+0000 7f0ba5a00700 1 --2- 172.21.15.89:0/1868505198 >> v2:172.21.15.113:3300/0 conn(0x7f0ba00cd480 0x7f0ba006e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.950+0000 7f0ba5a00700 1 --2- 172.21.15.89:0/1868505198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba00cbb30 0x7f0ba00cbf10 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.950+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1868505198 >> 172.21.15.89:0/1868505198 conn(0x7f0ba00b0d70 msgr2=0x7f0ba00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:57.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.950+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1868505198 shutdown_connections 2024-04-02T07:48:57.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.950+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1868505198 wait complete. 2024-04-02T07:48:57.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.952+0000 7f0ba5a00700 1 Processor -- start 2024-04-02T07:48:57.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.952+0000 7f0ba5a00700 1 -- start start 2024-04-02T07:48:57.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.953+0000 7f0ba5a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00cbb30 0x7f0ba00cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:57.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.953+0000 7f0ba5a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba00cd020 0x7f0ba009b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.953+0000 7f0ba5a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ba00cd480 0x7f0ba009b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.953+0000 7f0ba5a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba009d140 con 0x7f0ba00cd020 2024-04-02T07:48:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.953+0000 7f0ba5a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba009c0a0 con 0x7f0ba00cbb30 2024-04-02T07:48:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.953+0000 7f0ba5a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba009c380 con 0x7f0ba00cd480 2024-04-02T07:48:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.953+0000 7f0b9e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba00cd020 0x7f0ba009b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.953+0000 7f0b9e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba00cd020 0x7f0ba009b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58636/0 (socket says 172.21.15.89:58636) 2024-04-02T07:48:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.953+0000 7f0b9e000700 1 -- 172.21.15.89:0/1865405448 learned_addr learned my addr 172.21.15.89:0/1865405448 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:57.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.953+0000 7f0b9ea00700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00cbb30 0x7f0ba00cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:57.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.954+0000 7f0b9f400700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ba00cd480 0x7f0ba009b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:57.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.954+0000 7f0b9e000700 1 -- 172.21.15.89:0/1865405448 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ba00cd480 msgr2=0x7f0ba009b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:57.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.954+0000 7f0b9e000700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ba00cd480 0x7f0ba009b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.954+0000 7f0b9e000700 1 -- 172.21.15.89:0/1865405448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00cbb30 msgr2=0x7f0ba00cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:57.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.954+0000 7f0b9e000700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00cbb30 0x7f0ba00cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:57.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.954+0000 7f0b9e000700 1 -- 172.21.15.89:0/1865405448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b900094a0 con 0x7f0ba00cd020 2024-04-02T07:48:57.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.954+0000 7f0b9f400700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ba00cd480 0x7f0ba009b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:57.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.954+0000 7f0b9e000700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba00cd020 0x7f0ba009b300 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00c480 tx=0x7f0b8c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:57.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.954+0000 7f0b96a00700 1 -- 172.21.15.89:0/1865405448 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c005d70 con 0x7f0ba00cd020 2024-04-02T07:48:57.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.954+0000 7f0b96a00700 1 -- 172.21.15.89:0/1865405448 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c0070d0 con 0x7f0ba00cd020 2024-04-02T07:48:57.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.954+0000 7f0b96a00700 1 -- 172.21.15.89:0/1865405448 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c007440 con 0x7f0ba00cd020 2024-04-02T07:48:57.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.954+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1865405448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba009c660 con 0x7f0ba00cd020 2024-04-02T07:48:57.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.954+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1865405448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba009cc20 con 0x7f0ba00cd020 2024-04-02T07:48:57.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.955+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1865405448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba00b40b0 con 0x7f0ba00cd020 2024-04-02T07:48:57.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.956+0000 7f0b96a00700 1 -- 172.21.15.89:0/1865405448 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0b8c0075a0 con 0x7f0ba00cd020 2024-04-02T07:48:57.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.957+0000 7f0b96a00700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b8006adb0 0x7f0b8006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:57.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.957+0000 7f0b96a00700 1 -- 172.21.15.89:0/1865405448 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5850+0+0 (secure 0 0 0) 0x7f0b8c088970 con 0x7f0ba00cd020 2024-04-02T07:48:57.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.957+0000 7f0b9ea00700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b8006adb0 0x7f0b8006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:57.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.958+0000 7f0b9ea00700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b8006adb0 0x7f0b8006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0b900090b0 tx=0x7f0b9001c040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:57.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:57.961+0000 7f0b96a00700 1 -- 172.21.15.89:0/1865405448 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b8c053a50 con 0x7f0ba00cd020 2024-04-02T07:48:58.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.162+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1865405448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f0ba0067720 con 0x7f0ba00cd020 2024-04-02T07:48:58.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.166+0000 7f0b96a00700 1 -- 172.21.15.89:0/1865405448 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v14) v1 ==== 120+0+0 (secure 0 0 0) 0x7f0b8c057070 con 0x7f0ba00cd020 2024-04-02T07:48:58.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.169+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1865405448 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b8006adb0 msgr2=0x7f0b8006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:58.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.169+0000 7f0ba5a00700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b8006adb0 0x7f0b8006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0b900090b0 tx=0x7f0b9001c040 comp rx=0 tx=0).stop 2024-04-02T07:48:58.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.169+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1865405448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba00cd020 msgr2=0x7f0ba009b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:58.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.169+0000 7f0ba5a00700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba00cd020 0x7f0ba009b300 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00c480 tx=0x7f0b8c00c840 comp rx=0 tx=0).stop 2024-04-02T07:48:58.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.169+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1865405448 shutdown_connections 2024-04-02T07:48:58.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.169+0000 7f0ba5a00700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0b8006adb0 0x7f0b8006d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.169+0000 7f0ba5a00700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00cbb30 0x7f0ba00cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.169+0000 7f0ba5a00700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ba00cd480 0x7f0ba009b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.169+0000 7f0ba5a00700 1 --2- 172.21.15.89:0/1865405448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ba00cd020 0x7f0ba009b300 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.169+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1865405448 >> 172.21.15.89:0/1865405448 conn(0x7f0ba00b0d70 msgr2=0x7f0ba015af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:58.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.169+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1865405448 shutdown_connections 2024-04-02T07:48:58.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.169+0000 7f0ba5a00700 1 -- 172.21.15.89:0/1865405448 wait complete. 2024-04-02T07:48:58.191 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T07:48:58.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.382+0000 7f1ced600700 1 Processor -- start 2024-04-02T07:48:58.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.382+0000 7f1ced600700 1 -- start start 2024-04-02T07:48:58.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.383+0000 7f1ced600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce815d9d0 0x7f1ce80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:58.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.383+0000 7f1ced600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce8161420 con 0x7f1ce80d2540 2024-04-02T07:48:58.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.383+0000 7f1ced600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce8161700 con 0x7f1ce80d16a0 2024-04-02T07:48:58.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.383+0000 7f1ced600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce815edc0 con 0x7f1ce815d9d0 2024-04-02T07:48:58.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.383+0000 7f1ce7400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1ce80d2540 0x7f1ce80ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55774/0 (socket says 172.21.15.89:55774) 2024-04-02T07:48:58.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.383+0000 7f1ce7400700 1 -- 172.21.15.89:0/1406340969 learned_addr learned my addr 172.21.15.89:0/1406340969 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:48:58.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.383+0000 7f1ce6a00700 1 --2- 172.21.15.89:0/1406340969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce815d9d0 0x7f1ce80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758288470 0 0) 0x7f1ce8161420 con 0x7f1ce80d2540 2024-04-02T07:48:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ccc0035b0 con 0x7f1ce80d2540 2024-04-02T07:48:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541695902 0 0) 0x7f1ce8161700 con 0x7f1ce80d16a0 2024-04-02T07:48:58.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce8161420 con 0x7f1ce80d16a0 2024-04-02T07:48:58.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2011651968 0 0) 0x7f1ccc0035b0 con 0x7f1ce80d2540 2024-04-02T07:48:58.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce815d9d0 msgr2=0x7f1ce80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:58.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 --2- 172.21.15.89:0/1406340969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce815d9d0 0x7f1ce80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 >> v1:172.21.15.113:6789/0 conn(0x7f1ce80d16a0 legacy=0x7f1ce80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:58.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce81619e0 con 0x7f1ce80d2540 2024-04-02T07:48:58.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f1ce00019f0 con 0x7f1ce80d2540 2024-04-02T07:48:58.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 >> v1:172.21.15.89:6789/0 conn(0x7f1ce80d2540 legacy=0x7f1ce80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:58.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 --2- 172.21.15.89:0/1406340969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ccc003bc0 0x7f1ccc003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:58.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 --2- 172.21.15.89:0/1406340969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ccc004850 0x7f1ccc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:58.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 --2- 172.21.15.89:0/1406340969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce815d9d0 0x7f1ccc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:58.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc00bac0 con 0x7f1ccc003bc0 2024-04-02T07:48:58.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc00bc00 con 0x7f1ccc004850 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc00bdc0 con 0x7f1ce815d9d0 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.384+0000 7f1ce6a00700 1 --2- 172.21.15.89:0/1406340969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ccc003bc0 0x7f1ccc003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.385+0000 7f1ce7400700 1 --2- 172.21.15.89:0/1406340969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce815d9d0 0x7f1ccc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.385+0000 7f1ce6000700 1 --2- 172.21.15.89:0/1406340969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ccc004850 0x7f1ccc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.385+0000 7f1ce6a00700 1 -- 172.21.15.89:0/1406340969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce815d9d0 msgr2=0x7f1ccc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.385+0000 7f1ce6a00700 1 --2- 172.21.15.89:0/1406340969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce815d9d0 0x7f1ccc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.385+0000 7f1ce6a00700 1 -- 172.21.15.89:0/1406340969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ccc004850 msgr2=0x7f1ccc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.385+0000 7f1ce6a00700 1 --2- 172.21.15.89:0/1406340969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ccc004850 0x7f1ccc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.385+0000 7f1ce6a00700 1 -- 172.21.15.89:0/1406340969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc0035b0 con 0x7f1ccc003bc0 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.385+0000 7f1ce7400700 1 --2- 172.21.15.89:0/1406340969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce815d9d0 0x7f1ccc005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.385+0000 7f1ce6a00700 1 --2- 172.21.15.89:0/1406340969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ccc003bc0 0x7f1ccc003fc0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8009b60 tx=0x7f1ccc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0e01c41af7e709 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.385+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd800e6c0 con 0x7f1ccc003bc0 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.385+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd800e820 con 0x7f1ccc003bc0 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.385+0000 7f1ce5600700 1 -- 172.21.15.89:0/1406340969 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd800eb90 con 0x7f1ccc003bc0 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.386+0000 7f1ced600700 1 -- 172.21.15.89:0/1406340969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ccc003bc0 msgr2=0x7f1ccc003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.386+0000 7f1ced600700 1 --2- 172.21.15.89:0/1406340969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ccc003bc0 0x7f1ccc003fc0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8009b60 tx=0x7f1ccc005880 comp rx=0 tx=0).stop 2024-04-02T07:48:58.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.386+0000 7f1ce7400700 1 -- 172.21.15.89:0/1406340969 reap_dead start 2024-04-02T07:48:58.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.386+0000 7f1ced600700 1 -- 172.21.15.89:0/1406340969 shutdown_connections 2024-04-02T07:48:58.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.386+0000 7f1ced600700 1 -- 172.21.15.89:0/1406340969 >> 172.21.15.89:0/1406340969 conn(0x7f1ce808aff0 msgr2=0x7f1ce80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:58.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.386+0000 7f1ced600700 1 -- 172.21.15.89:0/1406340969 shutdown_connections 2024-04-02T07:48:58.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.386+0000 7f1ced600700 1 -- 172.21.15.89:0/1406340969 wait complete. 2024-04-02T07:48:58.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.388+0000 7f1ced600700 1 Processor -- start 2024-04-02T07:48:58.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.388+0000 7f1ced600700 1 -- start start 2024-04-02T07:48:58.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ced600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ccc003bc0 0x7f1ce815aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:58.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ced600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce80b89e0 0x7f1ce8160320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:58.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ced600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ce815e200 0x7f1ce815e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:58.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ced600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce815ebc0 con 0x7f1ccc003bc0 2024-04-02T07:48:58.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ced600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce80b1dc0 con 0x7f1ce815e200 2024-04-02T07:48:58.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ced600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce815b430 con 0x7f1ce80b89e0 2024-04-02T07:48:58.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ce6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ccc003bc0 0x7f1ce815aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:58.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ce6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ccc003bc0 0x7f1ce815aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58654/0 (socket says 172.21.15.89:58654) 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ce6a00700 1 -- 172.21.15.89:0/519078830 learned_addr learned my addr 172.21.15.89:0/519078830 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ce7400700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ce815e200 0x7f1ce815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ce6a00700 1 -- 172.21.15.89:0/519078830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce80b89e0 msgr2=0x7f1ce8160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ce6000700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce80b89e0 0x7f1ce8160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ce6a00700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce80b89e0 0x7f1ce8160320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ce6a00700 1 -- 172.21.15.89:0/519078830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ce815e200 msgr2=0x7f1ce815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ce6a00700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ce815e200 0x7f1ce815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ce6a00700 1 -- 172.21.15.89:0/519078830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce8160b90 con 0x7f1ccc003bc0 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ce7400700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ce815e200 0x7f1ce815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.389+0000 7f1ce6000700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce80b89e0 0x7f1ce8160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.390+0000 7f1ce6a00700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ccc003bc0 0x7f1ce815aef0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc005830 tx=0x7f1cd80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.390+0000 7f1cdea00700 1 -- 172.21.15.89:0/519078830 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd800e6c0 con 0x7f1ccc003bc0 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.390+0000 7f1cdea00700 1 -- 172.21.15.89:0/519078830 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd8003ab0 con 0x7f1ccc003bc0 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.390+0000 7f1cdea00700 1 -- 172.21.15.89:0/519078830 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8003e60 con 0x7f1ccc003bc0 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.390+0000 7f1ced600700 1 -- 172.21.15.89:0/519078830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce80b1490 con 0x7f1ccc003bc0 2024-04-02T07:48:58.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.390+0000 7f1ced600700 1 -- 172.21.15.89:0/519078830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce80b1a50 con 0x7f1ccc003bc0 2024-04-02T07:48:58.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.391+0000 7f1ced600700 1 -- 172.21.15.89:0/519078830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce80b20a0 con 0x7f1ccc003bc0 2024-04-02T07:48:58.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.392+0000 7f1cdea00700 1 -- 172.21.15.89:0/519078830 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1cd8013470 con 0x7f1ccc003bc0 2024-04-02T07:48:58.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.393+0000 7f1cdea00700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1cc806adb0 0x7f1cc806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:58.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.393+0000 7f1cdea00700 1 -- 172.21.15.89:0/519078830 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5850+0+0 (secure 0 0 0) 0x7f1cd808b090 con 0x7f1ccc003bc0 2024-04-02T07:48:58.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.393+0000 7f1ce6000700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1cc806adb0 0x7f1cc806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:58.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.393+0000 7f1ce6000700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1cc806adb0 0x7f1cc806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4007bc0 tx=0x7f1cd4009040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:58.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.397+0000 7f1cdea00700 1 -- 172.21.15.89:0/519078830 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1cd80561e0 con 0x7f1ccc003bc0 2024-04-02T07:48:58.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.598+0000 7f1ced600700 1 -- 172.21.15.89:0/519078830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1ce80d1ca0 con 0x7f1ccc003bc0 2024-04-02T07:48:58.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.602+0000 7f1cdea00700 1 -- 172.21.15.89:0/519078830 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1cd8059800 con 0x7f1ccc003bc0 2024-04-02T07:48:58.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.604+0000 7f1ced600700 1 -- 172.21.15.89:0/519078830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1cc806adb0 msgr2=0x7f1cc806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:58.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.604+0000 7f1ced600700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1cc806adb0 0x7f1cc806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4007bc0 tx=0x7f1cd4009040 comp rx=0 tx=0).stop 2024-04-02T07:48:58.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.604+0000 7f1ced600700 1 -- 172.21.15.89:0/519078830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ccc003bc0 msgr2=0x7f1ce815aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:58.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.604+0000 7f1ced600700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ccc003bc0 0x7f1ce815aef0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc005830 tx=0x7f1cd80047c0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.604+0000 7f1ced600700 1 -- 172.21.15.89:0/519078830 shutdown_connections 2024-04-02T07:48:58.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.604+0000 7f1ced600700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1cc806adb0 0x7f1cc806d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.604+0000 7f1ced600700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ce815e200 0x7f1ce815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.604+0000 7f1ced600700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ccc003bc0 0x7f1ce815aef0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.604+0000 7f1ced600700 1 --2- 172.21.15.89:0/519078830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ce80b89e0 0x7f1ce8160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.604+0000 7f1ced600700 1 -- 172.21.15.89:0/519078830 >> 172.21.15.89:0/519078830 conn(0x7f1ce808aff0 msgr2=0x7f1ce806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:58.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.605+0000 7f1ced600700 1 -- 172.21.15.89:0/519078830 shutdown_connections 2024-04-02T07:48:58.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.605+0000 7f1ced600700 1 -- 172.21.15.89:0/519078830 wait complete. 2024-04-02T07:48:58.626 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-04-02T07:48:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.813+0000 7f17c9600700 1 Processor -- start 2024-04-02T07:48:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.814+0000 7f17c9600700 1 -- start start 2024-04-02T07:48:58.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.815+0000 7f17c9600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f17c4160f10 0x7f17c40d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:58.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.815+0000 7f17c9600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f17c40cf230 0x7f17c40d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:58.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.815+0000 7f17c9600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c40ceda0 con 0x7f17c4161590 2024-04-02T07:48:58.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.815+0000 7f17c9600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c40d23d0 con 0x7f17c4160f10 2024-04-02T07:48:58.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.815+0000 7f17c9600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c41612f0 con 0x7f17c40cf230 2024-04-02T07:48:58.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.815+0000 7f17c2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f17c4160f10 0x7f17c40d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:58.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.815+0000 7f17c2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f17c4160f10 0x7f17c40d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36020/0 (socket says 172.21.15.89:36020) 2024-04-02T07:48:58.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.815+0000 7f17c2a00700 1 -- 172.21.15.89:0/667817975 learned_addr learned my addr 172.21.15.89:0/667817975 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:58.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.815+0000 7f17c2000700 1 --2- 172.21.15.89:0/667817975 >> v2:172.21.15.113:3300/0 conn(0x7f17c40cf230 0x7f17c40d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:58.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.815+0000 7f17c2a00700 1 -- 172.21.15.89:0/667817975 >> v2:172.21.15.113:3300/0 conn(0x7f17c40cf230 msgr2=0x7f17c40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.815+0000 7f17c2a00700 1 --2- 172.21.15.89:0/667817975 >> v2:172.21.15.113:3300/0 conn(0x7f17c40cf230 0x7f17c40d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.815+0000 7f17c2a00700 1 -- 172.21.15.89:0/667817975 >> v1:172.21.15.89:6789/0 conn(0x7f17c4161590 legacy=0x7f17c40b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.815+0000 7f17c2a00700 1 -- 172.21.15.89:0/667817975 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c40ce470 con 0x7f17c4160f10 2024-04-02T07:48:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.816+0000 7f17c2a00700 1 --2- 172.21.15.89:0/667817975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c4160f10 0x7f17c40d18b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f17c408cdb0 tx=0x7f17b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b20239c9183952e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.816+0000 7f17c1600700 1 -- 172.21.15.89:0/667817975 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b400b870 con 0x7f17c4160f10 2024-04-02T07:48:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.816+0000 7f17c1600700 1 -- 172.21.15.89:0/667817975 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b400b9d0 con 0x7f17c4160f10 2024-04-02T07:48:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.816+0000 7f17c1600700 1 -- 172.21.15.89:0/667817975 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b400bd60 con 0x7f17c4160f10 2024-04-02T07:48:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.816+0000 7f17c9600700 1 -- 172.21.15.89:0/667817975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c4160f10 msgr2=0x7f17c40d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.816+0000 7f17c9600700 1 --2- 172.21.15.89:0/667817975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c4160f10 0x7f17c40d18b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f17c408cdb0 tx=0x7f17b4009bd0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.817+0000 7f17c9600700 1 -- 172.21.15.89:0/667817975 shutdown_connections 2024-04-02T07:48:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.817+0000 7f17c9600700 1 --2- 172.21.15.89:0/667817975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c4160f10 0x7f17c40d18b0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.817+0000 7f17c9600700 1 --2- 172.21.15.89:0/667817975 >> v2:172.21.15.113:3300/0 conn(0x7f17c40cf230 0x7f17c40d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.817+0000 7f17c9600700 1 -- 172.21.15.89:0/667817975 >> 172.21.15.89:0/667817975 conn(0x7f17c408aff0 msgr2=0x7f17c40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.817+0000 7f17c9600700 1 -- 172.21.15.89:0/667817975 shutdown_connections 2024-04-02T07:48:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.817+0000 7f17c9600700 1 -- 172.21.15.89:0/667817975 wait complete. 2024-04-02T07:48:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.818+0000 7f17c9600700 1 Processor -- start 2024-04-02T07:48:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.818+0000 7f17c9600700 1 -- start start 2024-04-02T07:48:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.819+0000 7f17c9600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c40cf230 0x7f17c41525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.819+0000 7f17c9600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17c4160f10 0x7f17c4158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.819+0000 7f17c9600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c4161590 0x7f17c4155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.819+0000 7f17c9600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c41f74f0 con 0x7f17c40cf230 2024-04-02T07:48:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.819+0000 7f17c9600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c4157f00 con 0x7f17c4160f10 2024-04-02T07:48:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.819+0000 7f17c9600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c41581e0 con 0x7f17c4161590 2024-04-02T07:48:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.819+0000 7f17c2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c40cf230 0x7f17c41525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.820+0000 7f17c2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c40cf230 0x7f17c41525c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36030/0 (socket says 172.21.15.89:36030) 2024-04-02T07:48:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.820+0000 7f17c2a00700 1 -- 172.21.15.89:0/3508725683 learned_addr learned my addr 172.21.15.89:0/3508725683 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.820+0000 7f17c2000700 1 --2- 172.21.15.89:0/3508725683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17c4160f10 0x7f17c4158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.820+0000 7f17c3400700 1 --2- 172.21.15.89:0/3508725683 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c4161590 0x7f17c4155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.820+0000 7f17c2a00700 1 -- 172.21.15.89:0/3508725683 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c4161590 msgr2=0x7f17c4155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.820+0000 7f17c2a00700 1 --2- 172.21.15.89:0/3508725683 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c4161590 0x7f17c4155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.820+0000 7f17c2a00700 1 -- 172.21.15.89:0/3508725683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17c4160f10 msgr2=0x7f17c4158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.820+0000 7f17c2a00700 1 --2- 172.21.15.89:0/3508725683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17c4160f10 0x7f17c4158fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.820+0000 7f17c2a00700 1 -- 172.21.15.89:0/3508725683 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b4009550 con 0x7f17c40cf230 2024-04-02T07:48:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.820+0000 7f17c2a00700 1 --2- 172.21.15.89:0/3508725683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c40cf230 0x7f17c41525c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f17b4000c00 tx=0x7f17b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.820+0000 7f17baa00700 1 -- 172.21.15.89:0/3508725683 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b4004310 con 0x7f17c40cf230 2024-04-02T07:48:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.821+0000 7f17baa00700 1 -- 172.21.15.89:0/3508725683 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b401e070 con 0x7f17c40cf230 2024-04-02T07:48:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.821+0000 7f17c9600700 1 -- 172.21.15.89:0/3508725683 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c41584c0 con 0x7f17c40cf230 2024-04-02T07:48:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.821+0000 7f17baa00700 1 -- 172.21.15.89:0/3508725683 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b40036a0 con 0x7f17c40cf230 2024-04-02T07:48:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.821+0000 7f17c9600700 1 -- 172.21.15.89:0/3508725683 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c4158a80 con 0x7f17c40cf230 2024-04-02T07:48:58.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.822+0000 7f17c9600700 1 -- 172.21.15.89:0/3508725683 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c4158c70 con 0x7f17c40cf230 2024-04-02T07:48:58.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.823+0000 7f17baa00700 1 -- 172.21.15.89:0/3508725683 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f17b4020030 con 0x7f17c40cf230 2024-04-02T07:48:58.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.824+0000 7f17baa00700 1 --2- 172.21.15.89:0/3508725683 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f17a406adb0 0x7f17a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:58.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.824+0000 7f17baa00700 1 -- 172.21.15.89:0/3508725683 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5850+0+0 (secure 0 0 0) 0x7f17b408a510 con 0x7f17c40cf230 2024-04-02T07:48:58.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.824+0000 7f17c2000700 1 --2- 172.21.15.89:0/3508725683 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f17a406adb0 0x7f17a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:58.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.825+0000 7f17c2000700 1 --2- 172.21.15.89:0/3508725683 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f17a406adb0 0x7f17a406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f17b0006b20 tx=0x7f17b0008040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:58.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:58.828+0000 7f17baa00700 1 -- 172.21.15.89:0/3508725683 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f17b40555e0 con 0x7f17c40cf230 2024-04-02T07:48:59.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.111+0000 7f17c9600700 1 -- 172.21.15.89:0/3508725683 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f17c4067720 con 0x7f17c40cf230 2024-04-02T07:48:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.283+0000 7f17baa00700 1 -- 172.21.15.89:0/3508725683 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v474) v1 ==== 158+0+0 (secure 0 0 0) 0x7f17b4021e60 con 0x7f17c40cf230 2024-04-02T07:48:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.285+0000 7f17c9600700 1 -- 172.21.15.89:0/3508725683 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f17a406adb0 msgr2=0x7f17a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.285+0000 7f17c9600700 1 --2- 172.21.15.89:0/3508725683 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f17a406adb0 0x7f17a406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f17b0006b20 tx=0x7f17b0008040 comp rx=0 tx=0).stop 2024-04-02T07:48:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.285+0000 7f17c9600700 1 -- 172.21.15.89:0/3508725683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c40cf230 msgr2=0x7f17c41525c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.285+0000 7f17c9600700 1 --2- 172.21.15.89:0/3508725683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c40cf230 0x7f17c41525c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f17b4000c00 tx=0x7f17b4004750 comp rx=0 tx=0).stop 2024-04-02T07:48:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.286+0000 7f17c9600700 1 -- 172.21.15.89:0/3508725683 shutdown_connections 2024-04-02T07:48:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.286+0000 7f17c9600700 1 --2- 172.21.15.89:0/3508725683 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f17a406adb0 0x7f17a406d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.286+0000 7f17c9600700 1 --2- 172.21.15.89:0/3508725683 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c4161590 0x7f17c4155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.286+0000 7f17c9600700 1 --2- 172.21.15.89:0/3508725683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c40cf230 0x7f17c41525c0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.286+0000 7f17c9600700 1 --2- 172.21.15.89:0/3508725683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17c4160f10 0x7f17c4158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.286+0000 7f17c9600700 1 -- 172.21.15.89:0/3508725683 >> 172.21.15.89:0/3508725683 conn(0x7f17c408aff0 msgr2=0x7f17c40cad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.286+0000 7f17c9600700 1 -- 172.21.15.89:0/3508725683 shutdown_connections 2024-04-02T07:48:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.286+0000 7f17c9600700 1 -- 172.21.15.89:0/3508725683 wait complete. 2024-04-02T07:48:59.307 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2024-04-02T07:48:59.307 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2024-04-02T07:48:59.307 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-02T07:48:59.307 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-04-02T07:48:59.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.495+0000 7f4130e00700 1 Processor -- start 2024-04-02T07:48:59.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.495+0000 7f4130e00700 1 -- start start 2024-04-02T07:48:59.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.496+0000 7f4130e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f412c0ceec0 0x7f412c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:59.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.496+0000 7f4130e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f412c161800 0x7f412c0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:59.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.496+0000 7f4130e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412c161c50 con 0x7f412c15d640 2024-04-02T07:48:59.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.496+0000 7f4130e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412c0d30d0 con 0x7f412c0ceec0 2024-04-02T07:48:59.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.496+0000 7f4130e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412c0b6840 con 0x7f412c161800 2024-04-02T07:48:59.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.496+0000 7f4129400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f412c0ceec0 0x7f412c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:59.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.496+0000 7f4129400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f412c0ceec0 0x7f412c1612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36040/0 (socket says 172.21.15.89:36040) 2024-04-02T07:48:59.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.496+0000 7f4129400700 1 -- 172.21.15.89:0/1721053319 learned_addr learned my addr 172.21.15.89:0/1721053319 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:59.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.497+0000 7f412a800700 1 --2- 172.21.15.89:0/1721053319 >> v2:172.21.15.113:3300/0 conn(0x7f412c161800 0x7f412c0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:59.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.497+0000 7f4129400700 1 -- 172.21.15.89:0/1721053319 >> v2:172.21.15.113:3300/0 conn(0x7f412c161800 msgr2=0x7f412c0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:48:59.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.497+0000 7f4129400700 1 --2- 172.21.15.89:0/1721053319 >> v2:172.21.15.113:3300/0 conn(0x7f412c161800 0x7f412c0b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:59.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.497+0000 7f4129400700 1 -- 172.21.15.89:0/1721053319 >> v1:172.21.15.89:6789/0 conn(0x7f412c15d640 legacy=0x7f412c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:59.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.497+0000 7f4129400700 1 -- 172.21.15.89:0/1721053319 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c0d2b30 con 0x7f412c0ceec0 2024-04-02T07:48:59.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.497+0000 7f4129400700 1 --2- 172.21.15.89:0/1721053319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0ceec0 0x7f412c1612c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f412c162360 tx=0x7f4118009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a4e8336e7d7c198 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:59.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.497+0000 7f4128a00700 1 -- 172.21.15.89:0/1721053319 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411800b870 con 0x7f412c0ceec0 2024-04-02T07:48:59.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.497+0000 7f4128a00700 1 -- 172.21.15.89:0/1721053319 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411800b9d0 con 0x7f412c0ceec0 2024-04-02T07:48:59.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.497+0000 7f4128a00700 1 -- 172.21.15.89:0/1721053319 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411800bd60 con 0x7f412c0ceec0 2024-04-02T07:48:59.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.498+0000 7f4130e00700 1 -- 172.21.15.89:0/1721053319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0ceec0 msgr2=0x7f412c1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:59.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.498+0000 7f4130e00700 1 --2- 172.21.15.89:0/1721053319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0ceec0 0x7f412c1612c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f412c162360 tx=0x7f4118009bd0 comp rx=0 tx=0).stop 2024-04-02T07:48:59.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.498+0000 7f4130e00700 1 -- 172.21.15.89:0/1721053319 shutdown_connections 2024-04-02T07:48:59.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.498+0000 7f4130e00700 1 --2- 172.21.15.89:0/1721053319 >> v2:172.21.15.113:3300/0 conn(0x7f412c161800 0x7f412c0b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:59.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.498+0000 7f4130e00700 1 --2- 172.21.15.89:0/1721053319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0ceec0 0x7f412c1612c0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:59.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.498+0000 7f4130e00700 1 -- 172.21.15.89:0/1721053319 >> 172.21.15.89:0/1721053319 conn(0x7f412c08aff0 msgr2=0x7f412c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:48:59.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.498+0000 7f4130e00700 1 -- 172.21.15.89:0/1721053319 shutdown_connections 2024-04-02T07:48:59.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.498+0000 7f4130e00700 1 -- 172.21.15.89:0/1721053319 wait complete. 2024-04-02T07:48:59.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.500+0000 7f4130e00700 1 Processor -- start 2024-04-02T07:48:59.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.500+0000 7f4130e00700 1 -- start start 2024-04-02T07:48:59.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4130e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f412c0ceec0 0x7f412c14d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:59.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4130e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c15d640 0x7f412c09fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:59.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4130e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f412c161800 0x7f412c0cd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:59.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4130e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c14b580 con 0x7f412c15d640 2024-04-02T07:48:59.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4130e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c14d950 con 0x7f412c0ceec0 2024-04-02T07:48:59.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4130e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412c0a05c0 con 0x7f412c161800 2024-04-02T07:48:59.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4129400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c15d640 0x7f412c09fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:59.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4129400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c15d640 0x7f412c09fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36056/0 (socket says 172.21.15.89:36056) 2024-04-02T07:48:59.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4129400700 1 -- 172.21.15.89:0/657490261 learned_addr learned my addr 172.21.15.89:0/657490261 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:48:59.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f412a800700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f412c161800 0x7f412c0cd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:59.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4129e00700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f412c0ceec0 0x7f412c14d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:59.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4129400700 1 -- 172.21.15.89:0/657490261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f412c161800 msgr2=0x7f412c0cd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:59.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4129400700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f412c161800 0x7f412c0cd4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:59.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4129400700 1 -- 172.21.15.89:0/657490261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f412c0ceec0 msgr2=0x7f412c14d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:48:59.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4129400700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f412c0ceec0 0x7f412c14d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:48:59.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4129400700 1 -- 172.21.15.89:0/657490261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4118009550 con 0x7f412c15d640 2024-04-02T07:48:59.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4129e00700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f412c0ceec0 0x7f412c14d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:48:59.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.501+0000 7f4129400700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c15d640 0x7f412c09fc50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4118014040 tx=0x7f4118004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:59.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.502+0000 7f4122000700 1 -- 172.21.15.89:0/657490261 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118004520 con 0x7f412c15d640 2024-04-02T07:48:59.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.502+0000 7f4122000700 1 -- 172.21.15.89:0/657490261 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411801e070 con 0x7f412c15d640 2024-04-02T07:48:59.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.502+0000 7f4130e00700 1 -- 172.21.15.89:0/657490261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f412c0a08a0 con 0x7f412c15d640 2024-04-02T07:48:59.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.502+0000 7f4122000700 1 -- 172.21.15.89:0/657490261 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41180038a0 con 0x7f412c15d640 2024-04-02T07:48:59.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.502+0000 7f4130e00700 1 -- 172.21.15.89:0/657490261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412c0a0e60 con 0x7f412c15d640 2024-04-02T07:48:59.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.503+0000 7f4130e00700 1 -- 172.21.15.89:0/657490261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412c067720 con 0x7f412c15d640 2024-04-02T07:48:59.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.504+0000 7f4122000700 1 -- 172.21.15.89:0/657490261 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4118003a00 con 0x7f412c15d640 2024-04-02T07:48:59.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.505+0000 7f4122000700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f410c06adb0 0x7f410c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:48:59.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.505+0000 7f4122000700 1 -- 172.21.15.89:0/657490261 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5850+0+0 (secure 0 0 0) 0x7f4118020070 con 0x7f412c15d640 2024-04-02T07:48:59.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.505+0000 7f4129e00700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f410c06adb0 0x7f410c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:48:59.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.506+0000 7f4129e00700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f410c06adb0 0x7f410c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f412c1607b0 tx=0x7f411c009040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:48:59.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.509+0000 7f4122000700 1 -- 172.21.15.89:0/657490261 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4118055aa0 con 0x7f412c15d640 2024-04-02T07:48:59.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:48:59.791+0000 7f4130e00700 1 -- 172.21.15.89:0/657490261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f412c0b5c70 con 0x7f412c15d640 2024-04-02T07:49:00.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.287+0000 7f4122000700 1 -- 172.21.15.89:0/657490261 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v475) v1 ==== 157+0+0 (secure 0 0 0) 0x7f411802a0e0 con 0x7f412c15d640 2024-04-02T07:49:00.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.290+0000 7f4130e00700 1 -- 172.21.15.89:0/657490261 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f410c06adb0 msgr2=0x7f410c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:00.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.290+0000 7f4130e00700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f410c06adb0 0x7f410c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f412c1607b0 tx=0x7f411c009040 comp rx=0 tx=0).stop 2024-04-02T07:49:00.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.290+0000 7f4130e00700 1 -- 172.21.15.89:0/657490261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c15d640 msgr2=0x7f412c09fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:00.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.290+0000 7f4130e00700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c15d640 0x7f412c09fc50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4118014040 tx=0x7f4118004750 comp rx=0 tx=0).stop 2024-04-02T07:49:00.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.290+0000 7f4130e00700 1 -- 172.21.15.89:0/657490261 shutdown_connections 2024-04-02T07:49:00.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.290+0000 7f4130e00700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f410c06adb0 0x7f410c06d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:00.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.290+0000 7f4130e00700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f412c0ceec0 0x7f412c14d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:00.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.290+0000 7f4130e00700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f412c161800 0x7f412c0cd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:00.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.290+0000 7f4130e00700 1 --2- 172.21.15.89:0/657490261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c15d640 0x7f412c09fc50 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:00.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.290+0000 7f4130e00700 1 -- 172.21.15.89:0/657490261 >> 172.21.15.89:0/657490261 conn(0x7f412c08aff0 msgr2=0x7f412c0b74d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:00.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.290+0000 7f4130e00700 1 -- 172.21.15.89:0/657490261 shutdown_connections 2024-04-02T07:49:00.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.291+0000 7f4130e00700 1 -- 172.21.15.89:0/657490261 wait complete. 2024-04-02T07:49:00.314 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T07:49:00.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.506+0000 7f4d54200700 1 Processor -- start 2024-04-02T07:49:00.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.506+0000 7f4d54200700 1 -- start start 2024-04-02T07:49:00.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.507+0000 7f4d54200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d4c15d9d0 0x7f4d4c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:00.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.507+0000 7f4d54200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c161420 con 0x7f4d4c0d2540 2024-04-02T07:49:00.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.508+0000 7f4d54200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c161700 con 0x7f4d4c0d16a0 2024-04-02T07:49:00.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.508+0000 7f4d54200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c15edc0 con 0x7f4d4c15d9d0 2024-04-02T07:49:00.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.508+0000 7f4d52200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4d4c0d2540 0x7f4d4c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44504/0 (socket says 172.21.15.89:44504) 2024-04-02T07:49:00.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.508+0000 7f4d52200700 1 -- 172.21.15.89:0/1313028581 learned_addr learned my addr 172.21.15.89:0/1313028581 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:00.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.508+0000 7f4d51800700 1 --2- 172.21.15.89:0/1313028581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d4c15d9d0 0x7f4d4c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:00.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.508+0000 7f4d4be00700 1 -- 172.21.15.89:0/1313028581 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780229913 0 0) 0x7f4d4c161420 con 0x7f4d4c0d2540 2024-04-02T07:49:00.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.508+0000 7f4d4be00700 1 -- 172.21.15.89:0/1313028581 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d340035b0 con 0x7f4d4c0d2540 2024-04-02T07:49:00.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.508+0000 7f4d51800700 1 -- 172.21.15.89:0/1313028581 >> v1:172.21.15.113:6789/0 conn(0x7f4d4c0d16a0 legacy=0x7f4d4c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:00.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.508+0000 7f4d51800700 1 -- 172.21.15.89:0/1313028581 >> v1:172.21.15.89:6789/0 conn(0x7f4d4c0d2540 legacy=0x7f4d4c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:00.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.508+0000 7f4d4be00700 1 -- 172.21.15.89:0/1313028581 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014678121 0 0) 0x7f4d4c161700 con 0x7f4d4c0d16a0 2024-04-02T07:49:00.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.508+0000 7f4d51800700 1 -- 172.21.15.89:0/1313028581 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4c1619e0 con 0x7f4d4c15d9d0 2024-04-02T07:49:00.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.509+0000 7f4d51800700 1 --2- 172.21.15.89:0/1313028581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d4c15d9d0 0x7f4d4c0d3c30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4d400098a0 tx=0x7f4d40009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8107f7905ffe8f85 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:00.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.509+0000 7f4d4be00700 1 -- 172.21.15.89:0/1313028581 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d40011830 con 0x7f4d4c15d9d0 2024-04-02T07:49:00.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.509+0000 7f4d4be00700 1 -- 172.21.15.89:0/1313028581 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d40011990 con 0x7f4d4c15d9d0 2024-04-02T07:49:00.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.509+0000 7f4d4be00700 1 -- 172.21.15.89:0/1313028581 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d40011d00 con 0x7f4d4c15d9d0 2024-04-02T07:49:00.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.510+0000 7f4d54200700 1 -- 172.21.15.89:0/1313028581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d4c15d9d0 msgr2=0x7f4d4c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:00.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.510+0000 7f4d54200700 1 --2- 172.21.15.89:0/1313028581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d4c15d9d0 0x7f4d4c0d3c30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4d400098a0 tx=0x7f4d40009bb0 comp rx=0 tx=0).stop 2024-04-02T07:49:00.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.510+0000 7f4d54200700 1 -- 172.21.15.89:0/1313028581 shutdown_connections 2024-04-02T07:49:00.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.510+0000 7f4d54200700 1 --2- 172.21.15.89:0/1313028581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d4c15d9d0 0x7f4d4c0d3c30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:00.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.510+0000 7f4d54200700 1 -- 172.21.15.89:0/1313028581 >> 172.21.15.89:0/1313028581 conn(0x7f4d4c08aff0 msgr2=0x7f4d4c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:00.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.510+0000 7f4d54200700 1 -- 172.21.15.89:0/1313028581 shutdown_connections 2024-04-02T07:49:00.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.510+0000 7f4d54200700 1 -- 172.21.15.89:0/1313028581 wait complete. 2024-04-02T07:49:00.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.511+0000 7f4d54200700 1 Processor -- start 2024-04-02T07:49:00.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.511+0000 7f4d54200700 1 -- start start 2024-04-02T07:49:00.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.512+0000 7f4d54200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d4c0d16a0 0x7f4d4c1589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:00.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.512+0000 7f4d54200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d4c0d2540 0x7f4d4c06cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:00.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.512+0000 7f4d54200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d4c15d9d0 0x7f4d4c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:00.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.512+0000 7f4d54200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c0ca700 con 0x7f4d4c0d2540 2024-04-02T07:49:00.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.512+0000 7f4d54200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c1f7940 con 0x7f4d4c0d16a0 2024-04-02T07:49:00.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.512+0000 7f4d54200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c1f6d30 con 0x7f4d4c15d9d0 2024-04-02T07:49:00.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.513+0000 7f4d50e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d4c0d2540 0x7f4d4c06cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:00.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.513+0000 7f4d50e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d4c0d2540 0x7f4d4c06cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36068/0 (socket says 172.21.15.89:36068) 2024-04-02T07:49:00.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.513+0000 7f4d50e00700 1 -- 172.21.15.89:0/3192409768 learned_addr learned my addr 172.21.15.89:0/3192409768 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:00.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.513+0000 7f4d51800700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d4c0d16a0 0x7f4d4c1589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:00.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.513+0000 7f4d52200700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d4c15d9d0 0x7f4d4c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:00.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.513+0000 7f4d50e00700 1 -- 172.21.15.89:0/3192409768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d4c15d9d0 msgr2=0x7f4d4c06d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:00.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.513+0000 7f4d50e00700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d4c15d9d0 0x7f4d4c06d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:00.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.513+0000 7f4d50e00700 1 -- 172.21.15.89:0/3192409768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d4c0d16a0 msgr2=0x7f4d4c1589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:00.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.513+0000 7f4d50e00700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d4c0d16a0 0x7f4d4c1589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:00.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.513+0000 7f4d50e00700 1 -- 172.21.15.89:0/3192409768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d400094a0 con 0x7f4d4c0d2540 2024-04-02T07:49:00.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.513+0000 7f4d51800700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d4c0d16a0 0x7f4d4c1589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:49:00.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.513+0000 7f4d50e00700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d4c0d2540 0x7f4d4c06cd20 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800c4d0 tx=0x7f4d3800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:00.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.514+0000 7f4d49600700 1 -- 172.21.15.89:0/3192409768 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d380054e0 con 0x7f4d4c0d2540 2024-04-02T07:49:00.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.514+0000 7f4d49600700 1 -- 172.21.15.89:0/3192409768 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d380036d0 con 0x7f4d4c0d2540 2024-04-02T07:49:00.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.514+0000 7f4d54200700 1 -- 172.21.15.89:0/3192409768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4c1f7010 con 0x7f4d4c0d2540 2024-04-02T07:49:00.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.514+0000 7f4d49600700 1 -- 172.21.15.89:0/3192409768 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d38003a80 con 0x7f4d4c0d2540 2024-04-02T07:49:00.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.514+0000 7f4d54200700 1 -- 172.21.15.89:0/3192409768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4c1f75d0 con 0x7f4d4c0d2540 2024-04-02T07:49:00.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.515+0000 7f4d54200700 1 -- 172.21.15.89:0/3192409768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4c0d1ca0 con 0x7f4d4c0d2540 2024-04-02T07:49:00.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.516+0000 7f4d49600700 1 -- 172.21.15.89:0/3192409768 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4d38003be0 con 0x7f4d4c0d2540 2024-04-02T07:49:00.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.517+0000 7f4d49600700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4d2c06adb0 0x7f4d2c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:00.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.517+0000 7f4d49600700 1 -- 172.21.15.89:0/3192409768 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5850+0+0 (secure 0 0 0) 0x7f4d38088ad0 con 0x7f4d4c0d2540 2024-04-02T07:49:00.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.517+0000 7f4d51800700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4d2c06adb0 0x7f4d2c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:00.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.517+0000 7f4d51800700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4d2c06adb0 0x7f4d2c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4d400090b0 tx=0x7f4d4001c040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:00.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.521+0000 7f4d49600700 1 -- 172.21.15.89:0/3192409768 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4d38053a80 con 0x7f4d4c0d2540 2024-04-02T07:49:00.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.722+0000 7f4d54200700 1 -- 172.21.15.89:0/3192409768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4d4c1f7a80 con 0x7f4d4c0d2540 2024-04-02T07:49:00.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.727+0000 7f4d49600700 1 -- 172.21.15.89:0/3192409768 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4d4c1f7a80 con 0x7f4d4c0d2540 2024-04-02T07:49:00.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.730+0000 7f4d54200700 1 -- 172.21.15.89:0/3192409768 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4d2c06adb0 msgr2=0x7f4d2c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:00.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.730+0000 7f4d54200700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4d2c06adb0 0x7f4d2c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4d400090b0 tx=0x7f4d4001c040 comp rx=0 tx=0).stop 2024-04-02T07:49:00.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.730+0000 7f4d54200700 1 -- 172.21.15.89:0/3192409768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d4c0d2540 msgr2=0x7f4d4c06cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:00.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.730+0000 7f4d54200700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d4c0d2540 0x7f4d4c06cd20 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800c4d0 tx=0x7f4d3800c890 comp rx=0 tx=0).stop 2024-04-02T07:49:00.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.730+0000 7f4d54200700 1 -- 172.21.15.89:0/3192409768 shutdown_connections 2024-04-02T07:49:00.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.730+0000 7f4d54200700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4d2c06adb0 0x7f4d2c06d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:00.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.730+0000 7f4d54200700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d4c0d16a0 0x7f4d4c1589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:00.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.730+0000 7f4d54200700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d4c15d9d0 0x7f4d4c06d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:00.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.730+0000 7f4d54200700 1 --2- 172.21.15.89:0/3192409768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d4c0d2540 0x7f4d4c06cd20 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:00.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.730+0000 7f4d54200700 1 -- 172.21.15.89:0/3192409768 >> 172.21.15.89:0/3192409768 conn(0x7f4d4c08aff0 msgr2=0x7f4d4c15fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:00.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.730+0000 7f4d54200700 1 -- 172.21.15.89:0/3192409768 shutdown_connections 2024-04-02T07:49:00.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:00.730+0000 7f4d54200700 1 -- 172.21.15.89:0/3192409768 wait complete. 2024-04-02T07:49:00.751 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:49:00.751 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:49:00.840 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:49:00.927 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:49:00.927 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:49:00.927 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:49:00.971 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:49:00.971 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:49:00.971 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:49:01.016 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:49:01.016 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:49:01.016 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:49:01.061 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:49:01.061 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:49:01.061 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:49:01.105 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:49:01.105 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T07:49:01.106 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:49:01.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.294+0000 7f9c3ba00700 1 Processor -- start 2024-04-02T07:49:01.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.294+0000 7f9c3ba00700 1 -- start start 2024-04-02T07:49:01.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.295+0000 7f9c3ba00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9c340ce590 0x7f9c340ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:01.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.295+0000 7f9c3ba00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c340b6a20 0x7f9c340b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:01.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.295+0000 7f9c3ba00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c340ceef0 con 0x7f9c34160be0 2024-04-02T07:49:01.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.295+0000 7f9c3ba00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c340b65d0 con 0x7f9c340ce590 2024-04-02T07:49:01.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.295+0000 7f9c3ba00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c340d2a80 con 0x7f9c340b6a20 2024-04-02T07:49:01.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.295+0000 7f9c33e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9c340ce590 0x7f9c340ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:01.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.295+0000 7f9c33e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9c340ce590 0x7f9c340ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36078/0 (socket says 172.21.15.89:36078) 2024-04-02T07:49:01.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.295+0000 7f9c33e00700 1 -- 172.21.15.89:0/763757830 learned_addr learned my addr 172.21.15.89:0/763757830 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:01.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.295+0000 7f9c39a00700 1 --2- 172.21.15.89:0/763757830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c340b6a20 0x7f9c340b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:01.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.296+0000 7f9c33e00700 1 -- 172.21.15.89:0/763757830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c340b6a20 msgr2=0x7f9c340b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:01.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.296+0000 7f9c33e00700 1 --2- 172.21.15.89:0/763757830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c340b6a20 0x7f9c340b6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:01.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.296+0000 7f9c33e00700 1 -- 172.21.15.89:0/763757830 >> v1:172.21.15.113:6789/0 conn(0x7f9c34160be0 legacy=0x7f9c340cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:01.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.296+0000 7f9c33e00700 1 -- 172.21.15.89:0/763757830 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c34160fc0 con 0x7f9c340ce590 2024-04-02T07:49:01.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.296+0000 7f9c33e00700 1 --2- 172.21.15.89:0/763757830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c340ce590 0x7f9c340ce9b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9c200098f0 tx=0x7f9c20009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b493cc4bd446b87 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:01.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.296+0000 7f9c33400700 1 -- 172.21.15.89:0/763757830 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20011830 con 0x7f9c340ce590 2024-04-02T07:49:01.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.296+0000 7f9c33400700 1 -- 172.21.15.89:0/763757830 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c20011990 con 0x7f9c340ce590 2024-04-02T07:49:01.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.296+0000 7f9c33400700 1 -- 172.21.15.89:0/763757830 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20011d20 con 0x7f9c340ce590 2024-04-02T07:49:01.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.297+0000 7f9c3ba00700 1 -- 172.21.15.89:0/763757830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c340ce590 msgr2=0x7f9c340ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:01.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.297+0000 7f9c3ba00700 1 --2- 172.21.15.89:0/763757830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c340ce590 0x7f9c340ce9b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9c200098f0 tx=0x7f9c20009c00 comp rx=0 tx=0).stop 2024-04-02T07:49:01.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.297+0000 7f9c3ba00700 1 -- 172.21.15.89:0/763757830 shutdown_connections 2024-04-02T07:49:01.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.297+0000 7f9c3ba00700 1 --2- 172.21.15.89:0/763757830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c340b6a20 0x7f9c340b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:01.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.297+0000 7f9c3ba00700 1 --2- 172.21.15.89:0/763757830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c340ce590 0x7f9c340ce9b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:01.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.297+0000 7f9c3ba00700 1 -- 172.21.15.89:0/763757830 >> 172.21.15.89:0/763757830 conn(0x7f9c3408aff0 msgr2=0x7f9c340d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:01.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.297+0000 7f9c3ba00700 1 -- 172.21.15.89:0/763757830 shutdown_connections 2024-04-02T07:49:01.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.297+0000 7f9c3ba00700 1 -- 172.21.15.89:0/763757830 wait complete. 2024-04-02T07:49:01.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.299+0000 7f9c3ba00700 1 Processor -- start 2024-04-02T07:49:01.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.299+0000 7f9c3ba00700 1 -- start start 2024-04-02T07:49:01.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c3ba00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c340b6a20 0x7f9c340c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:01.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c3ba00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340ce590 0x7f9c341f28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:01.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c3ba00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c34160be0 0x7f9c340c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:01.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c3ba00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c341f2e30 con 0x7f9c340b6a20 2024-04-02T07:49:01.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c3ba00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c341553b0 con 0x7f9c340ce590 2024-04-02T07:49:01.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c3ba00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c340ca620 con 0x7f9c34160be0 2024-04-02T07:49:01.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c39000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c340b6a20 0x7f9c340c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:01.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c39000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c340b6a20 0x7f9c340c7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36088/0 (socket says 172.21.15.89:36088) 2024-04-02T07:49:01.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c39000700 1 -- 172.21.15.89:0/844519918 learned_addr learned my addr 172.21.15.89:0/844519918 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:01.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c33e00700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340ce590 0x7f9c341f28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:01.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c39a00700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c34160be0 0x7f9c340c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:01.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c39000700 1 -- 172.21.15.89:0/844519918 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c34160be0 msgr2=0x7f9c340c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:01.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c39000700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c34160be0 0x7f9c340c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:01.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c39000700 1 -- 172.21.15.89:0/844519918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340ce590 msgr2=0x7f9c341f28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:01.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c39000700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340ce590 0x7f9c341f28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:01.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c39000700 1 -- 172.21.15.89:0/844519918 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c20009550 con 0x7f9c340b6a20 2024-04-02T07:49:01.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.300+0000 7f9c33e00700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340ce590 0x7f9c341f28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:49:01.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.301+0000 7f9c39000700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c340b6a20 0x7f9c340c7a90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800c480 tx=0x7f9c2800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:01.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.301+0000 7f9c30c00700 1 -- 172.21.15.89:0/844519918 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c280052d0 con 0x7f9c340b6a20 2024-04-02T07:49:01.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.301+0000 7f9c30c00700 1 -- 172.21.15.89:0/844519918 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c28005430 con 0x7f9c340b6a20 2024-04-02T07:49:01.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.301+0000 7f9c30c00700 1 -- 172.21.15.89:0/844519918 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c280038c0 con 0x7f9c340b6a20 2024-04-02T07:49:01.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.301+0000 7f9c3ba00700 1 -- 172.21.15.89:0/844519918 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c340cabe0 con 0x7f9c340b6a20 2024-04-02T07:49:01.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.301+0000 7f9c3ba00700 1 -- 172.21.15.89:0/844519918 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c340c8470 con 0x7f9c340b6a20 2024-04-02T07:49:01.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.302+0000 7f9c3ba00700 1 -- 172.21.15.89:0/844519918 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c340c8600 con 0x7f9c340b6a20 2024-04-02T07:49:01.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.303+0000 7f9c30c00700 1 -- 172.21.15.89:0/844519918 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9c28003a20 con 0x7f9c340b6a20 2024-04-02T07:49:01.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.304+0000 7f9c30c00700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c1406adb0 0x7f9c1406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:01.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.304+0000 7f9c30c00700 1 -- 172.21.15.89:0/844519918 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5850+0+0 (secure 0 0 0) 0x7f9c28088930 con 0x7f9c340b6a20 2024-04-02T07:49:01.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.304+0000 7f9c33e00700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c1406adb0 0x7f9c1406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:01.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.304+0000 7f9c33e00700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c1406adb0 0x7f9c1406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009c40 tx=0x7f9c20008030 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:01.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.308+0000 7f9c30c00700 1 -- 172.21.15.89:0/844519918 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9c28053960 con 0x7f9c340b6a20 2024-04-02T07:49:01.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.551+0000 7f9c3ba00700 1 -- 172.21.15.89:0/844519918 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c34067720 con 0x7f9c340b6a20 2024-04-02T07:49:01.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.552+0000 7f9c30c00700 1 -- 172.21.15.89:0/844519918 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f9c28017090 con 0x7f9c340b6a20 2024-04-02T07:49:01.555 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:49:01.555 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5037,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3788023013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3788023013},{"type":"v1","addr":"172.21.15.89:6835","nonce":3788023013}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5040,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4029308904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":4029308904},{"type":"v1","addr":"172.21.15.113:6835","nonce":4029308904}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5043,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2115217020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2115217020},{"type":"v1","addr":"172.21.15.113:6834","nonce":2115217020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5808,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3045742538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3045742538},{"type":"v1","addr":"172.21.15.89:6837","nonce":3045742538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[]} 2024-04-02T07:49:01.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.554+0000 7f9c3ba00700 1 -- 172.21.15.89:0/844519918 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c1406adb0 msgr2=0x7f9c1406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:01.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.554+0000 7f9c3ba00700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c1406adb0 0x7f9c1406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009c40 tx=0x7f9c20008030 comp rx=0 tx=0).stop 2024-04-02T07:49:01.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.554+0000 7f9c3ba00700 1 -- 172.21.15.89:0/844519918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c340b6a20 msgr2=0x7f9c340c7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:01.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.554+0000 7f9c3ba00700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c340b6a20 0x7f9c340c7a90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800c480 tx=0x7f9c2800c840 comp rx=0 tx=0).stop 2024-04-02T07:49:01.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.554+0000 7f9c3ba00700 1 -- 172.21.15.89:0/844519918 shutdown_connections 2024-04-02T07:49:01.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.554+0000 7f9c3ba00700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c1406adb0 0x7f9c1406d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:01.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.554+0000 7f9c3ba00700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c34160be0 0x7f9c340c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:01.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.554+0000 7f9c3ba00700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c340b6a20 0x7f9c340c7a90 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:01.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.554+0000 7f9c3ba00700 1 --2- 172.21.15.89:0/844519918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340ce590 0x7f9c341f28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:01.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.554+0000 7f9c3ba00700 1 -- 172.21.15.89:0/844519918 >> 172.21.15.89:0/844519918 conn(0x7f9c3408aff0 msgr2=0x7f9c340b8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:01.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.555+0000 7f9c3ba00700 1 -- 172.21.15.89:0/844519918 shutdown_connections 2024-04-02T07:49:01.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.555+0000 7f9c3ba00700 1 -- 172.21.15.89:0/844519918 wait complete. 2024-04-02T07:49:01.559 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 19 2024-04-02T07:49:01.576 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T07:49:01.576 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T07:49:01.576 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T07:49:01.576 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:49:01.576 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T07:49:01.577 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T07:49:01.577 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T07:49:01.577 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:49:01.577 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T07:49:01.577 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T07:49:01.577 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T07:49:01.578 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:49:01.578 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T07:49:01.578 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T07:49:01.578 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T07:49:01.578 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:49:01.592 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T07:49:01.592 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T07:49:01.593 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T07:49:01.593 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T07:49:01.594 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T07:49:01.595 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T07:49:01.595 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T07:49:01.596 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T07:49:01.637 INFO:tasks.ceph.mds.a:Started 2024-04-02T07:49:01.637 DEBUG:teuthology.parallel:result is None 2024-04-02T07:49:01.639 INFO:tasks.ceph.mds.c:Started 2024-04-02T07:49:01.639 DEBUG:teuthology.parallel:result is None 2024-04-02T07:49:01.640 INFO:tasks.ceph.mds.b:Started 2024-04-02T07:49:01.640 DEBUG:teuthology.parallel:result is None 2024-04-02T07:49:01.642 INFO:tasks.ceph.mds.d:Started 2024-04-02T07:49:01.642 DEBUG:teuthology.parallel:result is None 2024-04-02T07:49:01.642 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T07:49:01.783 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T07:49:01.783 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T07:49:01.790 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T07:49:01.790 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T07:49:01.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.850+0000 7fb03b400700 1 Processor -- start 2024-04-02T07:49:01.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.850+0000 7fb03b400700 1 -- start start 2024-04-02T07:49:01.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.852+0000 7fb03b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb034160cc0 0x7fb034168730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:01.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.852+0000 7fb03b400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03415f820 con 0x7fb03415dad0 2024-04-02T07:49:01.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.852+0000 7fb03b400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0340d2450 con 0x7fb0340ced80 2024-04-02T07:49:01.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.852+0000 7fb03b400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0340cfcc0 con 0x7fb034160cc0 2024-04-02T07:49:01.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.852+0000 7fb038a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb03415dad0 0x7fb0340d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44534/0 (socket says 172.21.15.89:44534) 2024-04-02T07:49:01.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.852+0000 7fb038a00700 1 -- 172.21.15.89:0/2088747718 learned_addr learned my addr 172.21.15.89:0/2088747718 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:01.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.852+0000 7fb039400700 1 --2- 172.21.15.89:0/2088747718 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb034160cc0 0x7fb034168730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:01.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.852+0000 7fb033400700 1 -- 172.21.15.89:0/2088747718 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242500553 0 0) 0x7fb03415f820 con 0x7fb03415dad0 2024-04-02T07:49:01.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.853+0000 7fb033400700 1 -- 172.21.15.89:0/2088747718 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb01c0035b0 con 0x7fb03415dad0 2024-04-02T07:49:01.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.853+0000 7fb033400700 1 -- 172.21.15.89:0/2088747718 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551320391 0 0) 0x7fb0340d2450 con 0x7fb0340ced80 2024-04-02T07:49:01.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.853+0000 7fb033400700 1 -- 172.21.15.89:0/2088747718 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb03415f820 con 0x7fb0340ced80 2024-04-02T07:49:01.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.853+0000 7fb039400700 1 -- 172.21.15.89:0/2088747718 >> v1:172.21.15.113:6789/0 conn(0x7fb0340ced80 legacy=0x7fb0340d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:01.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.853+0000 7fb033400700 1 -- 172.21.15.89:0/2088747718 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4250565032 0 0) 0x7fb01c0035b0 con 0x7fb03415dad0 2024-04-02T07:49:01.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.853+0000 7fb039400700 1 -- 172.21.15.89:0/2088747718 >> v1:172.21.15.89:6789/0 conn(0x7fb03415dad0 legacy=0x7fb0340d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:01.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.853+0000 7fb039400700 1 -- 172.21.15.89:0/2088747718 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0340cf420 con 0x7fb034160cc0 2024-04-02T07:49:01.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.853+0000 7fb039400700 1 --2- 172.21.15.89:0/2088747718 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb034160cc0 0x7fb034168730 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb02c009dc0 tx=0x7fb02c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b643466d1c848ef server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:01.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.853+0000 7fb033400700 1 -- 172.21.15.89:0/2088747718 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c00e070 con 0x7fb034160cc0 2024-04-02T07:49:01.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.853+0000 7fb033400700 1 -- 172.21.15.89:0/2088747718 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02c003a20 con 0x7fb034160cc0 2024-04-02T07:49:01.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.853+0000 7fb033400700 1 -- 172.21.15.89:0/2088747718 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb028001670 con 0x7fb034160cc0 2024-04-02T07:49:01.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.854+0000 7fb03b400700 1 -- 172.21.15.89:0/2088747718 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb034160cc0 msgr2=0x7fb034168730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:01.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.854+0000 7fb03b400700 1 --2- 172.21.15.89:0/2088747718 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb034160cc0 0x7fb034168730 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb02c009dc0 tx=0x7fb02c00b330 comp rx=0 tx=0).stop 2024-04-02T07:49:01.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.854+0000 7fb03b400700 1 -- 172.21.15.89:0/2088747718 shutdown_connections 2024-04-02T07:49:01.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.854+0000 7fb03b400700 1 --2- 172.21.15.89:0/2088747718 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb034160cc0 0x7fb034168730 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:01.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.854+0000 7fb03b400700 1 -- 172.21.15.89:0/2088747718 >> 172.21.15.89:0/2088747718 conn(0x7fb03408aff0 msgr2=0x7fb0340d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:01.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.854+0000 7fb03b400700 1 -- 172.21.15.89:0/2088747718 shutdown_connections 2024-04-02T07:49:01.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.854+0000 7fb03b400700 1 -- 172.21.15.89:0/2088747718 wait complete. 2024-04-02T07:49:01.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.856+0000 7fb03b400700 1 Processor -- start 2024-04-02T07:49:01.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.857+0000 7fb03b400700 1 -- start start 2024-04-02T07:49:01.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb03b400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0340ced80 0x7fb0341f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:01.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb03b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb03415dad0 0x7fb0340d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:01.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb03b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb034160cc0 0x7fb0340d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:01.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb03b400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb034143370 con 0x7fb034160cc0 2024-04-02T07:49:01.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb03b400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0341435c0 con 0x7fb0340ced80 2024-04-02T07:49:01.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb03b400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0341438a0 con 0x7fb03415dad0 2024-04-02T07:49:01.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb039400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb034160cc0 0x7fb0340d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:01.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb039400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb034160cc0 0x7fb0340d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36124/0 (socket says 172.21.15.89:36124) 2024-04-02T07:49:01.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb039400700 1 -- 172.21.15.89:0/2382526961 learned_addr learned my addr 172.21.15.89:0/2382526961 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb033e00700 1 --2- 172.21.15.89:0/2382526961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb03415dad0 0x7fb0340d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb038a00700 1 --2- 172.21.15.89:0/2382526961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0340ced80 0x7fb0341f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb039400700 1 -- 172.21.15.89:0/2382526961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb03415dad0 msgr2=0x7fb0340d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb039400700 1 --2- 172.21.15.89:0/2382526961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb03415dad0 0x7fb0340d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb039400700 1 -- 172.21.15.89:0/2382526961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0340ced80 msgr2=0x7fb0341f2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb039400700 1 --2- 172.21.15.89:0/2382526961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0340ced80 0x7fb0341f2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.858+0000 7fb039400700 1 -- 172.21.15.89:0/2382526961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c009a70 con 0x7fb034160cc0 2024-04-02T07:49:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.859+0000 7fb039400700 1 --2- 172.21.15.89:0/2382526961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb034160cc0 0x7fb0340d1df0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fb02c009d90 tx=0x7fb02c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.859+0000 7fb030c00700 1 -- 172.21.15.89:0/2382526961 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c0042c0 con 0x7fb034160cc0 2024-04-02T07:49:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.859+0000 7fb030c00700 1 -- 172.21.15.89:0/2382526961 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02c004440 con 0x7fb034160cc0 2024-04-02T07:49:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.859+0000 7fb03b400700 1 -- 172.21.15.89:0/2382526961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb034143b80 con 0x7fb034160cc0 2024-04-02T07:49:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.859+0000 7fb030c00700 1 -- 172.21.15.89:0/2382526961 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c0047f0 con 0x7fb034160cc0 2024-04-02T07:49:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.859+0000 7fb03b400700 1 -- 172.21.15.89:0/2382526961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0341f2ac0 con 0x7fb034160cc0 2024-04-02T07:49:01.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.860+0000 7fb03b400700 1 -- 172.21.15.89:0/2382526961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0341441b0 con 0x7fb034160cc0 2024-04-02T07:49:01.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.861+0000 7fb030c00700 1 -- 172.21.15.89:0/2382526961 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb02c01c460 con 0x7fb034160cc0 2024-04-02T07:49:01.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.862+0000 7fb030c00700 1 --2- 172.21.15.89:0/2382526961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb01406adb0 0x7fb01406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:01.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.862+0000 7fb030c00700 1 -- 172.21.15.89:0/2382526961 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5850+0+0 (secure 0 0 0) 0x7fb02c08acc0 con 0x7fb034160cc0 2024-04-02T07:49:01.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.862+0000 7fb038a00700 1 --2- 172.21.15.89:0/2382526961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb01406adb0 0x7fb01406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:01.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.863+0000 7fb038a00700 1 --2- 172.21.15.89:0/2382526961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb01406adb0 0x7fb01406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb028001380 tx=0x7fb028001770 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:01.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:01.867+0000 7fb030c00700 1 -- 172.21.15.89:0/2382526961 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb02c091080 con 0x7fb034160cc0 2024-04-02T07:49:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:02.080+0000 7fb03b400700 1 -- 172.21.15.89:0/2382526961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb034067720 con 0x7fb034160cc0 2024-04-02T07:49:02.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:02.820+0000 7fb030c00700 1 -- 172.21.15.89:0/2382526961 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v62) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb02c020080 con 0x7fb034160cc0 2024-04-02T07:49:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:02.823+0000 7fb03b400700 1 -- 172.21.15.89:0/2382526961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb01406adb0 msgr2=0x7fb01406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:02.823+0000 7fb03b400700 1 --2- 172.21.15.89:0/2382526961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb01406adb0 0x7fb01406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb028001380 tx=0x7fb028001770 comp rx=0 tx=0).stop 2024-04-02T07:49:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:02.823+0000 7fb03b400700 1 -- 172.21.15.89:0/2382526961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb034160cc0 msgr2=0x7fb0340d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:02.823+0000 7fb03b400700 1 --2- 172.21.15.89:0/2382526961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb034160cc0 0x7fb0340d1df0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fb02c009d90 tx=0x7fb02c004ad0 comp rx=0 tx=0).stop 2024-04-02T07:49:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:02.824+0000 7fb03b400700 1 -- 172.21.15.89:0/2382526961 shutdown_connections 2024-04-02T07:49:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:02.824+0000 7fb03b400700 1 --2- 172.21.15.89:0/2382526961 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb01406adb0 0x7fb01406d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:02.824+0000 7fb03b400700 1 --2- 172.21.15.89:0/2382526961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0340ced80 0x7fb0341f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:02.824+0000 7fb03b400700 1 --2- 172.21.15.89:0/2382526961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb034160cc0 0x7fb0340d1df0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:02.824+0000 7fb03b400700 1 --2- 172.21.15.89:0/2382526961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb03415dad0 0x7fb0340d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:02.824+0000 7fb03b400700 1 -- 172.21.15.89:0/2382526961 >> 172.21.15.89:0/2382526961 conn(0x7fb03408aff0 msgr2=0x7fb03415ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:02.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:02.824+0000 7fb03b400700 1 -- 172.21.15.89:0/2382526961 shutdown_connections 2024-04-02T07:49:02.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:02.824+0000 7fb03b400700 1 -- 172.21.15.89:0/2382526961 wait complete. 2024-04-02T07:49:02.828 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T07:49:02.849 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T07:49:03.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.041+0000 7fdee1400700 1 Processor -- start 2024-04-02T07:49:03.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.042+0000 7fdee1400700 1 -- start start 2024-04-02T07:49:03.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.043+0000 7fdee1400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdedc15da50 0x7fdedc0ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.043+0000 7fdee1400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdedc0d0cf0 0x7fdedc0b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.043+0000 7fdee1400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdedc0d24e0 con 0x7fdedc0cec70 2024-04-02T07:49:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.043+0000 7fdee1400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdedc0b95d0 con 0x7fdedc15da50 2024-04-02T07:49:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.043+0000 7fdee1400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdedc15de30 con 0x7fdedc0d0cf0 2024-04-02T07:49:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.043+0000 7fded9e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdedc0cec70 0x7fdedc0d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44540/0 (socket says 172.21.15.89:44540) 2024-04-02T07:49:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.043+0000 7fded9e00700 1 -- 172.21.15.89:0/2190160602 learned_addr learned my addr 172.21.15.89:0/2190160602 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.043+0000 7fdeda800700 1 --2- 172.21.15.89:0/2190160602 >> v2:172.21.15.113:3300/0 conn(0x7fdedc15da50 0x7fdedc0ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.043+0000 7fdedb200700 1 --2- 172.21.15.89:0/2190160602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdedc0d0cf0 0x7fdedc0b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.043+0000 7fded9400700 1 -- 172.21.15.89:0/2190160602 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016053097 0 0) 0x7fdedc0d24e0 con 0x7fdedc0cec70 2024-04-02T07:49:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.044+0000 7fded9400700 1 -- 172.21.15.89:0/2190160602 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdec00035b0 con 0x7fdedc0cec70 2024-04-02T07:49:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.044+0000 7fdedb200700 1 -- 172.21.15.89:0/2190160602 >> v2:172.21.15.113:3300/0 conn(0x7fdedc15da50 msgr2=0x7fdedc0ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.044+0000 7fdedb200700 1 --2- 172.21.15.89:0/2190160602 >> v2:172.21.15.113:3300/0 conn(0x7fdedc15da50 0x7fdedc0ce730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.044+0000 7fdedb200700 1 -- 172.21.15.89:0/2190160602 >> v1:172.21.15.89:6789/0 conn(0x7fdedc0cec70 legacy=0x7fdedc0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.044+0000 7fdedb200700 1 -- 172.21.15.89:0/2190160602 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdedc0d27c0 con 0x7fdedc0d0cf0 2024-04-02T07:49:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.044+0000 7fdeda800700 1 --2- 172.21.15.89:0/2190160602 >> v2:172.21.15.113:3300/0 conn(0x7fdedc15da50 0x7fdedc0ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:49:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.044+0000 7fdedb200700 1 --2- 172.21.15.89:0/2190160602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdedc0d0cf0 0x7fdedc0b6990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fded4009d90 tx=0x7fded400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35bd9c878fed28e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.044+0000 7fded9400700 1 -- 172.21.15.89:0/2190160602 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded400e070 con 0x7fdedc0d0cf0 2024-04-02T07:49:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.044+0000 7fded9400700 1 -- 172.21.15.89:0/2190160602 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded400bb20 con 0x7fdedc0d0cf0 2024-04-02T07:49:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.044+0000 7fded9400700 1 -- 172.21.15.89:0/2190160602 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded400be90 con 0x7fdedc0d0cf0 2024-04-02T07:49:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.045+0000 7fdee1400700 1 -- 172.21.15.89:0/2190160602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdedc0d0cf0 msgr2=0x7fdedc0b6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.045+0000 7fdee1400700 1 --2- 172.21.15.89:0/2190160602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdedc0d0cf0 0x7fdedc0b6990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fded4009d90 tx=0x7fded400b330 comp rx=0 tx=0).stop 2024-04-02T07:49:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.045+0000 7fdee1400700 1 -- 172.21.15.89:0/2190160602 shutdown_connections 2024-04-02T07:49:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.045+0000 7fdee1400700 1 --2- 172.21.15.89:0/2190160602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdedc0d0cf0 0x7fdedc0b6990 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.045+0000 7fdee1400700 1 --2- 172.21.15.89:0/2190160602 >> v2:172.21.15.113:3300/0 conn(0x7fdedc15da50 0x7fdedc0ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.045+0000 7fdee1400700 1 -- 172.21.15.89:0/2190160602 >> 172.21.15.89:0/2190160602 conn(0x7fdedc0666a0 msgr2=0x7fdedc066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.045+0000 7fdee1400700 1 -- 172.21.15.89:0/2190160602 shutdown_connections 2024-04-02T07:49:03.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.045+0000 7fdee1400700 1 -- 172.21.15.89:0/2190160602 wait complete. 2024-04-02T07:49:03.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.047+0000 7fdee1400700 1 Processor -- start 2024-04-02T07:49:03.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.047+0000 7fdee1400700 1 -- start start 2024-04-02T07:49:03.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.048+0000 7fdee1400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdedc0cec70 0x7fdedc1f3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:03.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.048+0000 7fdee1400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdedc0d0cf0 0x7fdedc154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:03.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.048+0000 7fdee1400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdedc15da50 0x7fdedc1603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:03.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.048+0000 7fdee1400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdedc1f3970 con 0x7fdedc0cec70 2024-04-02T07:49:03.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.048+0000 7fdee1400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdedc1611e0 con 0x7fdedc15da50 2024-04-02T07:49:03.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.048+0000 7fdee1400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdedc15e6c0 con 0x7fdedc0d0cf0 2024-04-02T07:49:03.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.048+0000 7fdeda800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdedc0cec70 0x7fdedc1f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:03.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.048+0000 7fdeda800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdedc0cec70 0x7fdedc1f3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36128/0 (socket says 172.21.15.89:36128) 2024-04-02T07:49:03.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.048+0000 7fdeda800700 1 -- 172.21.15.89:0/2901205291 learned_addr learned my addr 172.21.15.89:0/2901205291 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:03.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.048+0000 7fdeda800700 1 -- 172.21.15.89:0/2901205291 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdedc0d0cf0 msgr2=0x7fdedc154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:03.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.048+0000 7fdeda800700 1 --2- 172.21.15.89:0/2901205291 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdedc0d0cf0 0x7fdedc154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.048+0000 7fdeda800700 1 -- 172.21.15.89:0/2901205291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdedc15da50 msgr2=0x7fdedc1603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:03.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.048+0000 7fdeda800700 1 --2- 172.21.15.89:0/2901205291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdedc15da50 0x7fdedc1603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.048+0000 7fdeda800700 1 -- 172.21.15.89:0/2901205291 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdecc0094a0 con 0x7fdedc0cec70 2024-04-02T07:49:03.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.049+0000 7fdeda800700 1 --2- 172.21.15.89:0/2901205291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdedc0cec70 0x7fdedc1f3430 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fdecc00ebe0 tx=0x7fdecc009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:03.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.049+0000 7fded2a00700 1 -- 172.21.15.89:0/2901205291 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdecc00c8b0 con 0x7fdedc0cec70 2024-04-02T07:49:03.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.049+0000 7fded2a00700 1 -- 172.21.15.89:0/2901205291 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdecc00ca10 con 0x7fdedc0cec70 2024-04-02T07:49:03.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.049+0000 7fded2a00700 1 -- 172.21.15.89:0/2901205291 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdecc00cda0 con 0x7fdedc0cec70 2024-04-02T07:49:03.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.049+0000 7fdee1400700 1 -- 172.21.15.89:0/2901205291 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded4009a70 con 0x7fdedc0cec70 2024-04-02T07:49:03.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.049+0000 7fdee1400700 1 -- 172.21.15.89:0/2901205291 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdedc15ec80 con 0x7fdedc0cec70 2024-04-02T07:49:03.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.050+0000 7fdee1400700 1 -- 172.21.15.89:0/2901205291 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdedc067720 con 0x7fdedc0cec70 2024-04-02T07:49:03.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.051+0000 7fded2a00700 1 -- 172.21.15.89:0/2901205291 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdecc013470 con 0x7fdedc0cec70 2024-04-02T07:49:03.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.052+0000 7fded2a00700 1 --2- 172.21.15.89:0/2901205291 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdebc06adb0 0x7fdebc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:03.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.052+0000 7fded2a00700 1 -- 172.21.15.89:0/2901205291 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fdecc089490 con 0x7fdedc0cec70 2024-04-02T07:49:03.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.052+0000 7fded9e00700 1 --2- 172.21.15.89:0/2901205291 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdebc06adb0 0x7fdebc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:03.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.052+0000 7fded9e00700 1 --2- 172.21.15.89:0/2901205291 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdebc06adb0 0x7fdebc06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdec8007fd0 tx=0x7fdec8007980 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:03.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.056+0000 7fded2a00700 1 -- 172.21.15.89:0/2901205291 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdecc0546b0 con 0x7fdedc0cec70 2024-04-02T07:49:03.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.337+0000 7fdee1400700 1 -- 172.21.15.89:0/2901205291 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fdedc15cbe0 con 0x7fdedc0cec70 2024-04-02T07:49:03.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.338+0000 7fded2a00700 1 -- 172.21.15.89:0/2901205291 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fdecc057cd0 con 0x7fdedc0cec70 2024-04-02T07:49:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:49:03.348 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:49:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T07:49:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T07:49:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T07:49:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:49:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:49:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:49:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T07:49:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T07:49:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:49:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:49:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:49:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T07:49:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T07:49:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:49:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:49:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:49:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T07:49:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T07:49:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:49:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:49:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:49:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T07:49:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T07:49:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:49:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:49:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T07:49:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T07:49:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:49:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:49:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T07:49:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T07:49:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:49:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:49:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T07:49:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T07:49:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:49:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:49:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T07:49:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T07:49:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:49:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:49:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T07:49:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T07:49:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:49:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:49:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T07:49:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T07:49:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:49:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:49:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T07:49:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T07:49:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:49:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:49:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T07:49:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T07:49:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:49:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2024-04-02T07:49:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T07:49:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T07:49:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T07:49:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:49:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:49:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:49:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:49:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T07:49:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T07:49:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:49:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:49:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:49:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:49:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T07:49:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T07:49:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:49:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:49:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:49:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:49:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T07:49:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T07:49:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:49:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:49:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:49:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T07:49:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T07:49:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:49:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:49:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:49:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T07:49:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T07:49:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:49:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:49:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:49:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:49:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T07:49:03.373 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:49:03.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.340+0000 7fdee1400700 1 -- 172.21.15.89:0/2901205291 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdebc06adb0 msgr2=0x7fdebc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:03.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.340+0000 7fdee1400700 1 --2- 172.21.15.89:0/2901205291 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdebc06adb0 0x7fdebc06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdec8007fd0 tx=0x7fdec8007980 comp rx=0 tx=0).stop 2024-04-02T07:49:03.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.340+0000 7fdee1400700 1 -- 172.21.15.89:0/2901205291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdedc0cec70 msgr2=0x7fdedc1f3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:03.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.340+0000 7fdee1400700 1 --2- 172.21.15.89:0/2901205291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdedc0cec70 0x7fdedc1f3430 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fdecc00ebe0 tx=0x7fdecc009e20 comp rx=0 tx=0).stop 2024-04-02T07:49:03.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.340+0000 7fdee1400700 1 -- 172.21.15.89:0/2901205291 shutdown_connections 2024-04-02T07:49:03.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.340+0000 7fdee1400700 1 --2- 172.21.15.89:0/2901205291 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdebc06adb0 0x7fdebc06d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.340+0000 7fdee1400700 1 --2- 172.21.15.89:0/2901205291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdedc15da50 0x7fdedc1603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.340+0000 7fdee1400700 1 --2- 172.21.15.89:0/2901205291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdedc0cec70 0x7fdedc1f3430 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.340+0000 7fdee1400700 1 --2- 172.21.15.89:0/2901205291 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdedc0d0cf0 0x7fdedc154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.340+0000 7fdee1400700 1 -- 172.21.15.89:0/2901205291 >> 172.21.15.89:0/2901205291 conn(0x7fdedc0666a0 msgr2=0x7fdedc15ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:03.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.341+0000 7fdee1400700 1 -- 172.21.15.89:0/2901205291 shutdown_connections 2024-04-02T07:49:03.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.341+0000 7fdee1400700 1 -- 172.21.15.89:0/2901205291 wait complete. 2024-04-02T07:49:03.378 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:49:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.576+0000 7fce4d000700 1 Processor -- start 2024-04-02T07:49:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.577+0000 7fce4d000700 1 -- start start 2024-04-02T07:49:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.578+0000 7fce4d000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fce4815b5d0 0x7fce480ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.578+0000 7fce4d000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce480ac7d0 0x7fce4806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.578+0000 7fce4d000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4815fd50 con 0x7fce480cb3f0 2024-04-02T07:49:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.578+0000 7fce4d000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce4815b9b0 con 0x7fce4815b5d0 2024-04-02T07:49:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.578+0000 7fce4d000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce4815f140 con 0x7fce480ac7d0 2024-04-02T07:49:03.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.578+0000 7fce45e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fce4815b5d0 0x7fce480ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:03.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.578+0000 7fce45400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fce480cb3f0 0x7fce480ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44550/0 (socket says 172.21.15.89:44550) 2024-04-02T07:49:03.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.578+0000 7fce45400700 1 -- 172.21.15.89:0/1997007035 learned_addr learned my addr 172.21.15.89:0/1997007035 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:03.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.578+0000 7fce46800700 1 --2- 172.21.15.89:0/1997007035 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce480ac7d0 0x7fce4806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:03.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.578+0000 7fce45e00700 1 -- 172.21.15.89:0/1997007035 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce480ac7d0 msgr2=0x7fce4806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:03.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.578+0000 7fce45e00700 1 --2- 172.21.15.89:0/1997007035 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce480ac7d0 0x7fce4806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.578+0000 7fce45e00700 1 -- 172.21.15.89:0/1997007035 >> v1:172.21.15.89:6789/0 conn(0x7fce480cb3f0 legacy=0x7fce480ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:03.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.578+0000 7fce45e00700 1 -- 172.21.15.89:0/1997007035 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4815f420 con 0x7fce4815b5d0 2024-04-02T07:49:03.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.578+0000 7fce44a00700 1 -- 172.21.15.89:0/1997007035 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675654882 0 0) 0x7fce4815fd50 con 0x7fce480cb3f0 2024-04-02T07:49:03.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.579+0000 7fce45e00700 1 --2- 172.21.15.89:0/1997007035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce4815b5d0 0x7fce480ae3f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fce38009840 tx=0x7fce38009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d2afd668e2f729 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:03.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.579+0000 7fce44a00700 1 -- 172.21.15.89:0/1997007035 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38011830 con 0x7fce4815b5d0 2024-04-02T07:49:03.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.579+0000 7fce44a00700 1 -- 172.21.15.89:0/1997007035 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce38011990 con 0x7fce4815b5d0 2024-04-02T07:49:03.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.579+0000 7fce44a00700 1 -- 172.21.15.89:0/1997007035 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38011d20 con 0x7fce4815b5d0 2024-04-02T07:49:03.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.579+0000 7fce4d000700 1 -- 172.21.15.89:0/1997007035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce4815b5d0 msgr2=0x7fce480ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:03.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.579+0000 7fce4d000700 1 --2- 172.21.15.89:0/1997007035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce4815b5d0 0x7fce480ae3f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fce38009840 tx=0x7fce38009b50 comp rx=0 tx=0).stop 2024-04-02T07:49:03.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.580+0000 7fce4d000700 1 -- 172.21.15.89:0/1997007035 shutdown_connections 2024-04-02T07:49:03.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.580+0000 7fce4d000700 1 --2- 172.21.15.89:0/1997007035 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce480ac7d0 0x7fce4806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.580+0000 7fce4d000700 1 --2- 172.21.15.89:0/1997007035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce4815b5d0 0x7fce480ae3f0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.580+0000 7fce4d000700 1 -- 172.21.15.89:0/1997007035 >> 172.21.15.89:0/1997007035 conn(0x7fce480cf190 msgr2=0x7fce480cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:03.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.580+0000 7fce4d000700 1 -- 172.21.15.89:0/1997007035 shutdown_connections 2024-04-02T07:49:03.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.580+0000 7fce4d000700 1 -- 172.21.15.89:0/1997007035 wait complete. 2024-04-02T07:49:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.582+0000 7fce4d000700 1 Processor -- start 2024-04-02T07:49:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.582+0000 7fce4d000700 1 -- start start 2024-04-02T07:49:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.583+0000 7fce4d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce480ac7d0 0x7fce481f18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.583+0000 7fce4d000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce4815b5d0 0x7fce4814f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.583+0000 7fce4d000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce480b4250 0x7fce48157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.583+0000 7fce4d000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce481571f0 con 0x7fce480ac7d0 2024-04-02T07:49:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.583+0000 7fce4d000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce481f1df0 con 0x7fce480b4250 2024-04-02T07:49:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.583+0000 7fce4d000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce481f1420 con 0x7fce4815b5d0 2024-04-02T07:49:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.584+0000 7fce45e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce480ac7d0 0x7fce481f18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.584+0000 7fce45e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce480ac7d0 0x7fce481f18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36146/0 (socket says 172.21.15.89:36146) 2024-04-02T07:49:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.584+0000 7fce45e00700 1 -- 172.21.15.89:0/1134834495 learned_addr learned my addr 172.21.15.89:0/1134834495 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.584+0000 7fce46800700 1 --2- 172.21.15.89:0/1134834495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce480b4250 0x7fce48157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.584+0000 7fce45e00700 1 -- 172.21.15.89:0/1134834495 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce4815b5d0 msgr2=0x7fce4814f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.584+0000 7fce45e00700 1 --2- 172.21.15.89:0/1134834495 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce4815b5d0 0x7fce4814f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.584+0000 7fce45e00700 1 -- 172.21.15.89:0/1134834495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce480b4250 msgr2=0x7fce48157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:03.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.584+0000 7fce45e00700 1 --2- 172.21.15.89:0/1134834495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce480b4250 0x7fce48157b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.584+0000 7fce45e00700 1 -- 172.21.15.89:0/1134834495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce380094a0 con 0x7fce480ac7d0 2024-04-02T07:49:03.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.584+0000 7fce46800700 1 --2- 172.21.15.89:0/1134834495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce480b4250 0x7fce48157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:49:03.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.584+0000 7fce45e00700 1 --2- 172.21.15.89:0/1134834495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce480ac7d0 0x7fce481f18b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fce38009fd0 tx=0x7fce38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:03.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.584+0000 7fce3e000700 1 -- 172.21.15.89:0/1134834495 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce380036a0 con 0x7fce480ac7d0 2024-04-02T07:49:03.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.585+0000 7fce3e000700 1 -- 172.21.15.89:0/1134834495 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3801e070 con 0x7fce480ac7d0 2024-04-02T07:49:03.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.585+0000 7fce3e000700 1 -- 172.21.15.89:0/1134834495 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38003a60 con 0x7fce480ac7d0 2024-04-02T07:49:03.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.585+0000 7fce4d000700 1 -- 172.21.15.89:0/1134834495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce481f0df0 con 0x7fce480ac7d0 2024-04-02T07:49:03.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.585+0000 7fce4d000700 1 -- 172.21.15.89:0/1134834495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce481576b0 con 0x7fce480ac7d0 2024-04-02T07:49:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.586+0000 7fce4d000700 1 -- 172.21.15.89:0/1134834495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce481f1700 con 0x7fce480ac7d0 2024-04-02T07:49:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.586+0000 7fce3e000700 1 -- 172.21.15.89:0/1134834495 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fce38003bc0 con 0x7fce480ac7d0 2024-04-02T07:49:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.587+0000 7fce3e000700 1 --2- 172.21.15.89:0/1134834495 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce2806adb0 0x7fce2806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.587+0000 7fce3e000700 1 -- 172.21.15.89:0/1134834495 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fce3808a2c0 con 0x7fce480ac7d0 2024-04-02T07:49:03.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.587+0000 7fce45400700 1 --2- 172.21.15.89:0/1134834495 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce2806adb0 0x7fce2806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:03.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.588+0000 7fce45400700 1 --2- 172.21.15.89:0/1134834495 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce2806adb0 0x7fce2806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fce340011e0 tx=0x7fce340079b0 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.592+0000 7fce3e000700 1 -- 172.21.15.89:0/1134834495 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fce38055570 con 0x7fce480ac7d0 2024-04-02T07:49:03.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.794+0000 7fce4d000700 1 -- 172.21.15.89:0/1134834495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce48067720 con 0x7fce480ac7d0 2024-04-02T07:49:03.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.795+0000 7fce3e000700 1 -- 172.21.15.89:0/1134834495 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+12916 (secure 0 0 0) 0x7fce38058b90 con 0x7fce480ac7d0 2024-04-02T07:49:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:49:03.798 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":62,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:49:02.817145+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:49:03.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.797+0000 7fce4d000700 1 -- 172.21.15.89:0/1134834495 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce2806adb0 msgr2=0x7fce2806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:03.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.797+0000 7fce4d000700 1 --2- 172.21.15.89:0/1134834495 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce2806adb0 0x7fce2806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fce340011e0 tx=0x7fce340079b0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.797+0000 7fce4d000700 1 -- 172.21.15.89:0/1134834495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce480ac7d0 msgr2=0x7fce481f18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:03.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.797+0000 7fce4d000700 1 --2- 172.21.15.89:0/1134834495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce480ac7d0 0x7fce481f18b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fce38009fd0 tx=0x7fce38004750 comp rx=0 tx=0).stop 2024-04-02T07:49:03.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.797+0000 7fce4d000700 1 -- 172.21.15.89:0/1134834495 shutdown_connections 2024-04-02T07:49:03.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.797+0000 7fce4d000700 1 --2- 172.21.15.89:0/1134834495 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce2806adb0 0x7fce2806d270 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fce340011e0 tx=0x7fce340079b0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.797+0000 7fce4d000700 1 --2- 172.21.15.89:0/1134834495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce480b4250 0x7fce48157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.797+0000 7fce4d000700 1 --2- 172.21.15.89:0/1134834495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce480ac7d0 0x7fce481f18b0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.797+0000 7fce4d000700 1 --2- 172.21.15.89:0/1134834495 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fce4815b5d0 0x7fce4814f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:03.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.797+0000 7fce4d000700 1 -- 172.21.15.89:0/1134834495 >> 172.21.15.89:0/1134834495 conn(0x7fce480cf190 msgr2=0x7fce480b0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:03.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.797+0000 7fce4d000700 1 -- 172.21.15.89:0/1134834495 shutdown_connections 2024-04-02T07:49:03.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:03.797+0000 7fce4d000700 1 -- 172.21.15.89:0/1134834495 wait complete. 2024-04-02T07:49:03.819 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:49:03.820 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:49:04.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.013+0000 7f5475200700 1 Processor -- start 2024-04-02T07:49:04.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.014+0000 7f5475200700 1 -- start start 2024-04-02T07:49:04.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.015+0000 7f5475200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54700d16a0 0x7f54700d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.015+0000 7f5475200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f54700d2370 0x7f54700ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.015+0000 7f5475200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5470161420 con 0x7f547015d9d0 2024-04-02T07:49:04.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.015+0000 7f5475200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54700d2820 con 0x7f54700d2370 2024-04-02T07:49:04.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.015+0000 7f5475200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54700d4340 con 0x7f54700d16a0 2024-04-02T07:49:04.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.015+0000 7f546e800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f54700d2370 0x7f54700ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.015+0000 7f546e800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f54700d2370 0x7f54700ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36160/0 (socket says 172.21.15.89:36160) 2024-04-02T07:49:04.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.015+0000 7f546e800700 1 -- 172.21.15.89:0/3348925937 learned_addr learned my addr 172.21.15.89:0/3348925937 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:04.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.015+0000 7f5467e00700 1 --2- 172.21.15.89:0/3348925937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54700d16a0 0x7f54700d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.015+0000 7f546e800700 1 -- 172.21.15.89:0/3348925937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54700d16a0 msgr2=0x7f54700d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:04.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.015+0000 7f546e800700 1 --2- 172.21.15.89:0/3348925937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54700d16a0 0x7f54700d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.015+0000 7f546e800700 1 -- 172.21.15.89:0/3348925937 >> v1:172.21.15.113:6789/0 conn(0x7f547015d9d0 legacy=0x7f54700d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.015+0000 7f546e800700 1 -- 172.21.15.89:0/3348925937 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5470161700 con 0x7f54700d2370 2024-04-02T07:49:04.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.016+0000 7f546e800700 1 --2- 172.21.15.89:0/3348925937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54700d2370 0x7f54700ce360 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f5468009e80 tx=0x7f546800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef3a129bde585f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:04.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.016+0000 7f546d400700 1 -- 172.21.15.89:0/3348925937 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546800e070 con 0x7f54700d2370 2024-04-02T07:49:04.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.016+0000 7f546d400700 1 -- 172.21.15.89:0/3348925937 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546800b980 con 0x7f54700d2370 2024-04-02T07:49:04.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.016+0000 7f546d400700 1 -- 172.21.15.89:0/3348925937 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546800bd30 con 0x7f54700d2370 2024-04-02T07:49:04.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.016+0000 7f5475200700 1 -- 172.21.15.89:0/3348925937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54700d2370 msgr2=0x7f54700ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.016+0000 7f5475200700 1 --2- 172.21.15.89:0/3348925937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54700d2370 0x7f54700ce360 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f5468009e80 tx=0x7f546800b330 comp rx=0 tx=0).stop 2024-04-02T07:49:04.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.016+0000 7f5475200700 1 -- 172.21.15.89:0/3348925937 shutdown_connections 2024-04-02T07:49:04.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.016+0000 7f5475200700 1 --2- 172.21.15.89:0/3348925937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54700d2370 0x7f54700ce360 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.016+0000 7f5475200700 1 --2- 172.21.15.89:0/3348925937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54700d16a0 0x7f54700d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.016+0000 7f5475200700 1 -- 172.21.15.89:0/3348925937 >> 172.21.15.89:0/3348925937 conn(0x7f547008aff0 msgr2=0x7f54700d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:04.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.017+0000 7f5475200700 1 -- 172.21.15.89:0/3348925937 shutdown_connections 2024-04-02T07:49:04.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.017+0000 7f5475200700 1 -- 172.21.15.89:0/3348925937 wait complete. 2024-04-02T07:49:04.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.019+0000 7f5475200700 1 Processor -- start 2024-04-02T07:49:04.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.019+0000 7f5475200700 1 -- start start 2024-04-02T07:49:04.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.020+0000 7f5475200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54700d16a0 0x7f54700cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.020+0000 7f5475200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54700d2370 0x7f5470159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.020+0000 7f5475200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f547015d9d0 0x7f54700cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.020+0000 7f5475200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54700b6020 con 0x7f54700d16a0 2024-04-02T07:49:04.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.020+0000 7f5475200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547015a440 con 0x7f54700d2370 2024-04-02T07:49:04.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.020+0000 7f5475200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54700cb2e0 con 0x7f547015d9d0 2024-04-02T07:49:04.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.020+0000 7f546de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54700d16a0 0x7f54700cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.020+0000 7f546de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54700d16a0 0x7f54700cbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36176/0 (socket says 172.21.15.89:36176) 2024-04-02T07:49:04.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.020+0000 7f546de00700 1 -- 172.21.15.89:0/3624771553 learned_addr learned my addr 172.21.15.89:0/3624771553 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:04.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.021+0000 7f5467e00700 1 --2- 172.21.15.89:0/3624771553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54700d2370 0x7f5470159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.021+0000 7f546e800700 1 --2- 172.21.15.89:0/3624771553 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f547015d9d0 0x7f54700cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.021+0000 7f546de00700 1 -- 172.21.15.89:0/3624771553 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f547015d9d0 msgr2=0x7f54700cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.021+0000 7f546de00700 1 --2- 172.21.15.89:0/3624771553 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f547015d9d0 0x7f54700cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.021+0000 7f546de00700 1 -- 172.21.15.89:0/3624771553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54700d2370 msgr2=0x7f5470159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.021+0000 7f546de00700 1 --2- 172.21.15.89:0/3624771553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54700d2370 0x7f5470159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.021+0000 7f546de00700 1 -- 172.21.15.89:0/3624771553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5468009b30 con 0x7f54700d16a0 2024-04-02T07:49:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.021+0000 7f546de00700 1 --2- 172.21.15.89:0/3624771553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54700d16a0 0x7f54700cbdb0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f546000c450 tx=0x7f546000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.021+0000 7f5466000700 1 -- 172.21.15.89:0/3624771553 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54600052d0 con 0x7f54700d16a0 2024-04-02T07:49:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.021+0000 7f5466000700 1 -- 172.21.15.89:0/3624771553 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5460005430 con 0x7f54700d16a0 2024-04-02T07:49:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.021+0000 7f5475200700 1 -- 172.21.15.89:0/3624771553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54700cb5c0 con 0x7f54700d16a0 2024-04-02T07:49:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.021+0000 7f5466000700 1 -- 172.21.15.89:0/3624771553 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54600038c0 con 0x7f54700d16a0 2024-04-02T07:49:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.021+0000 7f5475200700 1 -- 172.21.15.89:0/3624771553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547015f490 con 0x7f54700d16a0 2024-04-02T07:49:04.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.022+0000 7f5475200700 1 -- 172.21.15.89:0/3624771553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54700d1ca0 con 0x7f54700d16a0 2024-04-02T07:49:04.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.023+0000 7f5466000700 1 -- 172.21.15.89:0/3624771553 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5460003a20 con 0x7f54700d16a0 2024-04-02T07:49:04.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.024+0000 7f5466000700 1 --2- 172.21.15.89:0/3624771553 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f544c06ade0 0x7f544c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.024+0000 7f5466000700 1 -- 172.21.15.89:0/3624771553 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f54600886f0 con 0x7f54700d16a0 2024-04-02T07:49:04.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.024+0000 7f5467e00700 1 --2- 172.21.15.89:0/3624771553 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f544c06ade0 0x7f544c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.025+0000 7f5467e00700 1 --2- 172.21.15.89:0/3624771553 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f544c06ade0 0x7f544c06d2a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f54700b8300 tx=0x7f5458008040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:04.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.029+0000 7f5466000700 1 -- 172.21.15.89:0/3624771553 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5460053910 con 0x7f54700d16a0 2024-04-02T07:49:04.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.230+0000 7f5475200700 1 -- 172.21.15.89:0/3624771553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5470067720 con 0x7f54700d16a0 2024-04-02T07:49:04.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.230+0000 7f5466000700 1 -- 172.21.15.89:0/3624771553 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5460056f30 con 0x7f54700d16a0 2024-04-02T07:49:04.233 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:49:04.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.232+0000 7f5475200700 1 -- 172.21.15.89:0/3624771553 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f544c06ade0 msgr2=0x7f544c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.232+0000 7f5475200700 1 --2- 172.21.15.89:0/3624771553 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f544c06ade0 0x7f544c06d2a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f54700b8300 tx=0x7f5458008040 comp rx=0 tx=0).stop 2024-04-02T07:49:04.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.232+0000 7f5475200700 1 -- 172.21.15.89:0/3624771553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54700d16a0 msgr2=0x7f54700cbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.232+0000 7f5475200700 1 --2- 172.21.15.89:0/3624771553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54700d16a0 0x7f54700cbdb0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f546000c450 tx=0x7f546000c810 comp rx=0 tx=0).stop 2024-04-02T07:49:04.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.232+0000 7f5475200700 1 -- 172.21.15.89:0/3624771553 shutdown_connections 2024-04-02T07:49:04.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.232+0000 7f5475200700 1 --2- 172.21.15.89:0/3624771553 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f544c06ade0 0x7f544c06d2a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.232+0000 7f5475200700 1 --2- 172.21.15.89:0/3624771553 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f547015d9d0 0x7f54700cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.232+0000 7f5475200700 1 --2- 172.21.15.89:0/3624771553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54700d16a0 0x7f54700cbdb0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.232+0000 7f5475200700 1 --2- 172.21.15.89:0/3624771553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54700d2370 0x7f5470159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.232+0000 7f5475200700 1 -- 172.21.15.89:0/3624771553 >> 172.21.15.89:0/3624771553 conn(0x7f547008aff0 msgr2=0x7f54700b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:04.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.232+0000 7f5475200700 1 -- 172.21.15.89:0/3624771553 shutdown_connections 2024-04-02T07:49:04.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.232+0000 7f5475200700 1 -- 172.21.15.89:0/3624771553 wait complete. 2024-04-02T07:49:04.254 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T07:49:04.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.442+0000 7fa67a000700 1 Processor -- start 2024-04-02T07:49:04.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.442+0000 7fa67a000700 1 -- start start 2024-04-02T07:49:04.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.443+0000 7fa67a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa6740d18b0 0x7fa6740cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.443+0000 7fa67a000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa674161590 0x7fa674161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.443+0000 7fa67a000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6740ceda0 con 0x7fa674160f10 2024-04-02T07:49:04.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.443+0000 7fa67a000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6740d1d30 con 0x7fa6740d18b0 2024-04-02T07:49:04.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.443+0000 7fa67a000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa674161f60 con 0x7fa674161590 2024-04-02T07:49:04.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.443+0000 7fa673400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa674160f10 0x7fa6740d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44564/0 (socket says 172.21.15.89:44564) 2024-04-02T07:49:04.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.444+0000 7fa673400700 1 -- 172.21.15.89:0/1405448175 learned_addr learned my addr 172.21.15.89:0/1405448175 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:04.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.444+0000 7fa672a00700 1 --2- 172.21.15.89:0/1405448175 >> v2:172.21.15.89:3300/0 conn(0x7fa6740d18b0 0x7fa6740cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.444+0000 7fa673e00700 1 --2- 172.21.15.89:0/1405448175 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa674161590 0x7fa674161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.444+0000 7fa672a00700 1 -- 172.21.15.89:0/1405448175 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa674161590 msgr2=0x7fa674161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:04.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.444+0000 7fa672a00700 1 --2- 172.21.15.89:0/1405448175 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa674161590 0x7fa674161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.444+0000 7fa672a00700 1 -- 172.21.15.89:0/1405448175 >> v1:172.21.15.89:6789/0 conn(0x7fa674160f10 legacy=0x7fa6740d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.444+0000 7fa672000700 1 -- 172.21.15.89:0/1405448175 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639736509 0 0) 0x7fa6740ceda0 con 0x7fa674160f10 2024-04-02T07:49:04.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.444+0000 7fa672a00700 1 -- 172.21.15.89:0/1405448175 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6741612f0 con 0x7fa6740d18b0 2024-04-02T07:49:04.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.444+0000 7fa672a00700 1 --2- 172.21.15.89:0/1405448175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6740d18b0 0x7fa6740cf230 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa660009870 tx=0x7fa660009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b2bcb3863bc5ea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:04.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.444+0000 7fa672000700 1 -- 172.21.15.89:0/1405448175 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa660011830 con 0x7fa6740d18b0 2024-04-02T07:49:04.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.444+0000 7fa672000700 1 -- 172.21.15.89:0/1405448175 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa660011990 con 0x7fa6740d18b0 2024-04-02T07:49:04.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.445+0000 7fa672000700 1 -- 172.21.15.89:0/1405448175 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa660011d20 con 0x7fa6740d18b0 2024-04-02T07:49:04.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.445+0000 7fa67a000700 1 -- 172.21.15.89:0/1405448175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6740d18b0 msgr2=0x7fa6740cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.445+0000 7fa67a000700 1 --2- 172.21.15.89:0/1405448175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6740d18b0 0x7fa6740cf230 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa660009870 tx=0x7fa660009b80 comp rx=0 tx=0).stop 2024-04-02T07:49:04.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.445+0000 7fa67a000700 1 -- 172.21.15.89:0/1405448175 shutdown_connections 2024-04-02T07:49:04.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.445+0000 7fa67a000700 1 --2- 172.21.15.89:0/1405448175 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa674161590 0x7fa674161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.445+0000 7fa67a000700 1 --2- 172.21.15.89:0/1405448175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6740d18b0 0x7fa6740cf230 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.445+0000 7fa67a000700 1 -- 172.21.15.89:0/1405448175 >> 172.21.15.89:0/1405448175 conn(0x7fa67408aff0 msgr2=0x7fa6740d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:04.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.445+0000 7fa67a000700 1 -- 172.21.15.89:0/1405448175 shutdown_connections 2024-04-02T07:49:04.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.445+0000 7fa67a000700 1 -- 172.21.15.89:0/1405448175 wait complete. 2024-04-02T07:49:04.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.447+0000 7fa67a000700 1 Processor -- start 2024-04-02T07:49:04.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.447+0000 7fa67a000700 1 -- start start 2024-04-02T07:49:04.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa67a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6740d18b0 0x7fa674158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa67a000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa674160f10 0x7fa6741590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa67a000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa674161590 0x7fa6740b7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa67a000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6740c9c00 con 0x7fa6740d18b0 2024-04-02T07:49:04.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa67a000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6740b86f0 con 0x7fa674161590 2024-04-02T07:49:04.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa67a000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6740b89d0 con 0x7fa674160f10 2024-04-02T07:49:04.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa673400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6740d18b0 0x7fa674158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa673400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6740d18b0 0x7fa674158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36196/0 (socket says 172.21.15.89:36196) 2024-04-02T07:49:04.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa673400700 1 -- 172.21.15.89:0/133108834 learned_addr learned my addr 172.21.15.89:0/133108834 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:04.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa672a00700 1 --2- 172.21.15.89:0/133108834 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa674160f10 0x7fa6741590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa673e00700 1 --2- 172.21.15.89:0/133108834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa674161590 0x7fa6740b7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa673400700 1 -- 172.21.15.89:0/133108834 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa674160f10 msgr2=0x7fa6741590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa673400700 1 --2- 172.21.15.89:0/133108834 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa674160f10 0x7fa6741590c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa673400700 1 -- 172.21.15.89:0/133108834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa674161590 msgr2=0x7fa6740b7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.448+0000 7fa673400700 1 --2- 172.21.15.89:0/133108834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa674161590 0x7fa6740b7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.449+0000 7fa673400700 1 -- 172.21.15.89:0/133108834 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6600094a0 con 0x7fa6740d18b0 2024-04-02T07:49:04.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.449+0000 7fa673400700 1 --2- 172.21.15.89:0/133108834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6740d18b0 0x7fa674158b80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa66400c550 tx=0x7fa66400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:04.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.449+0000 7fa66b400700 1 -- 172.21.15.89:0/133108834 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa664004c80 con 0x7fa6740d18b0 2024-04-02T07:49:04.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.449+0000 7fa66b400700 1 -- 172.21.15.89:0/133108834 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa664004e00 con 0x7fa6740d18b0 2024-04-02T07:49:04.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.449+0000 7fa67a000700 1 -- 172.21.15.89:0/133108834 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa674145880 con 0x7fa6740d18b0 2024-04-02T07:49:04.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.449+0000 7fa66b400700 1 -- 172.21.15.89:0/133108834 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa664007310 con 0x7fa6740d18b0 2024-04-02T07:49:04.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.449+0000 7fa67a000700 1 -- 172.21.15.89:0/133108834 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa674155a60 con 0x7fa6740d18b0 2024-04-02T07:49:04.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.450+0000 7fa67a000700 1 -- 172.21.15.89:0/133108834 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa674145da0 con 0x7fa6740d18b0 2024-04-02T07:49:04.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.451+0000 7fa66b400700 1 -- 172.21.15.89:0/133108834 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa664007470 con 0x7fa6740d18b0 2024-04-02T07:49:04.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.452+0000 7fa66b400700 1 --2- 172.21.15.89:0/133108834 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa65406adb0 0x7fa65406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.452+0000 7fa66b400700 1 -- 172.21.15.89:0/133108834 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fa664088e20 con 0x7fa6740d18b0 2024-04-02T07:49:04.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.452+0000 7fa672a00700 1 --2- 172.21.15.89:0/133108834 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa65406adb0 0x7fa65406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.453+0000 7fa672a00700 1 --2- 172.21.15.89:0/133108834 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa65406adb0 0x7fa65406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa6600090b0 tx=0x7fa66001c040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:04.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.456+0000 7fa66b400700 1 -- 172.21.15.89:0/133108834 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa674145da0 con 0x7fa6740d18b0 2024-04-02T07:49:04.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.663+0000 7fa67a000700 1 -- 172.21.15.89:0/133108834 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa674145da0 con 0x7fa6740d18b0 2024-04-02T07:49:04.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.663+0000 7fa66b400700 1 -- 172.21.15.89:0/133108834 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v62) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa674145da0 con 0x7fa6740d18b0 2024-04-02T07:49:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:49:04.667 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:49:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T07:49:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T07:49:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:04.667 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:49:04.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.666+0000 7fa67a000700 1 -- 172.21.15.89:0/133108834 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa65406adb0 msgr2=0x7fa65406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.666+0000 7fa67a000700 1 --2- 172.21.15.89:0/133108834 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa65406adb0 0x7fa65406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa6600090b0 tx=0x7fa66001c040 comp rx=0 tx=0).stop 2024-04-02T07:49:04.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.666+0000 7fa67a000700 1 -- 172.21.15.89:0/133108834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6740d18b0 msgr2=0x7fa674158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.666+0000 7fa67a000700 1 --2- 172.21.15.89:0/133108834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6740d18b0 0x7fa674158b80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa66400c550 tx=0x7fa66400c860 comp rx=0 tx=0).stop 2024-04-02T07:49:04.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.666+0000 7fa67a000700 1 -- 172.21.15.89:0/133108834 shutdown_connections 2024-04-02T07:49:04.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.666+0000 7fa67a000700 1 --2- 172.21.15.89:0/133108834 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa65406adb0 0x7fa65406d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.666+0000 7fa67a000700 1 --2- 172.21.15.89:0/133108834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa674161590 0x7fa6740b7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.666+0000 7fa67a000700 1 --2- 172.21.15.89:0/133108834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6740d18b0 0x7fa674158b80 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.666+0000 7fa67a000700 1 --2- 172.21.15.89:0/133108834 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa674160f10 0x7fa6741590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.666+0000 7fa67a000700 1 -- 172.21.15.89:0/133108834 >> 172.21.15.89:0/133108834 conn(0x7fa67408aff0 msgr2=0x7fa6740cc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:04.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.666+0000 7fa67a000700 1 -- 172.21.15.89:0/133108834 shutdown_connections 2024-04-02T07:49:04.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.666+0000 7fa67a000700 1 -- 172.21.15.89:0/133108834 wait complete. 2024-04-02T07:49:04.687 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T07:49:04.687 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T07:49:04.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.881+0000 7f07ad600700 1 Processor -- start 2024-04-02T07:49:04.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.881+0000 7f07ad600700 1 -- start start 2024-04-02T07:49:04.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.882+0000 7f07ad600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f07a80ced30 0x7f07a80d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.882+0000 7f07ad600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07a80d2710 0x7f07a815f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.882+0000 7f07ad600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a80ce8a0 con 0x7f07a815ddf0 2024-04-02T07:49:04.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.882+0000 7f07ad600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a815ec60 con 0x7f07a80ced30 2024-04-02T07:49:04.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.882+0000 7f07ad600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80cdc90 con 0x7f07a80d2710 2024-04-02T07:49:04.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.882+0000 7f07a6000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f07a80ced30 0x7f07a80d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.883+0000 7f07a6000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f07a80ced30 0x7f07a80d21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36198/0 (socket says 172.21.15.89:36198) 2024-04-02T07:49:04.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.883+0000 7f07a6000700 1 -- 172.21.15.89:0/3195375473 learned_addr learned my addr 172.21.15.89:0/3195375473 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:04.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.883+0000 7f07a7400700 1 --2- 172.21.15.89:0/3195375473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07a80d2710 0x7f07a815f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.883+0000 7f07a6000700 1 -- 172.21.15.89:0/3195375473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07a80d2710 msgr2=0x7f07a815f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:04.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.883+0000 7f07a6000700 1 --2- 172.21.15.89:0/3195375473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07a80d2710 0x7f07a815f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.883+0000 7f07a6000700 1 -- 172.21.15.89:0/3195375473 >> v1:172.21.15.113:6789/0 conn(0x7f07a815ddf0 legacy=0x7f07a815e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.883+0000 7f07a6000700 1 -- 172.21.15.89:0/3195375473 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a80cdf70 con 0x7f07a80ced30 2024-04-02T07:49:04.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.883+0000 7f07a7400700 1 --2- 172.21.15.89:0/3195375473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07a80d2710 0x7f07a815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:49:04.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.883+0000 7f07a6000700 1 --2- 172.21.15.89:0/3195375473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07a80ced30 0x7f07a80d21d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f07940098f0 tx=0x7f0794009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64b0cea994de2906 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:04.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.883+0000 7f07a5600700 1 -- 172.21.15.89:0/3195375473 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079400b8a0 con 0x7f07a80ced30 2024-04-02T07:49:04.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.884+0000 7f07a5600700 1 -- 172.21.15.89:0/3195375473 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079400ba00 con 0x7f07a80ced30 2024-04-02T07:49:04.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.884+0000 7f07a5600700 1 -- 172.21.15.89:0/3195375473 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079400bd90 con 0x7f07a80ced30 2024-04-02T07:49:04.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.884+0000 7f07ad600700 1 -- 172.21.15.89:0/3195375473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07a80ced30 msgr2=0x7f07a80d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.884+0000 7f07ad600700 1 --2- 172.21.15.89:0/3195375473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07a80ced30 0x7f07a80d21d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f07940098f0 tx=0x7f0794009c00 comp rx=0 tx=0).stop 2024-04-02T07:49:04.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.884+0000 7f07ad600700 1 -- 172.21.15.89:0/3195375473 shutdown_connections 2024-04-02T07:49:04.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.884+0000 7f07ad600700 1 --2- 172.21.15.89:0/3195375473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07a80d2710 0x7f07a815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.884+0000 7f07ad600700 1 --2- 172.21.15.89:0/3195375473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07a80ced30 0x7f07a80d21d0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.884+0000 7f07ad600700 1 -- 172.21.15.89:0/3195375473 >> 172.21.15.89:0/3195375473 conn(0x7f07a808aff0 msgr2=0x7f07a80d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:04.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.884+0000 7f07ad600700 1 -- 172.21.15.89:0/3195375473 shutdown_connections 2024-04-02T07:49:04.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.884+0000 7f07ad600700 1 -- 172.21.15.89:0/3195375473 wait complete. 2024-04-02T07:49:04.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.886+0000 7f07ad600700 1 Processor -- start 2024-04-02T07:49:04.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.886+0000 7f07ad600700 1 -- start start 2024-04-02T07:49:04.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.887+0000 7f07ad600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07a80ced30 0x7f07a80b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.887+0000 7f07ad600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07a80d2710 0x7f07a8156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.887+0000 7f07ad600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07a815ddf0 0x7f07a80b8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.887+0000 7f07ad600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80c9040 con 0x7f07a80ced30 2024-04-02T07:49:04.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.887+0000 7f07ad600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80b3230 con 0x7f07a80d2710 2024-04-02T07:49:04.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.887+0000 7f07ad600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80b2900 con 0x7f07a815ddf0 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.887+0000 7f07a6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07a80ced30 0x7f07a80b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.887+0000 7f07a6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07a80ced30 0x7f07a80b36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36206/0 (socket says 172.21.15.89:36206) 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.887+0000 7f07a6a00700 1 -- 172.21.15.89:0/3314545952 learned_addr learned my addr 172.21.15.89:0/3314545952 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.887+0000 7f07a6000700 1 --2- 172.21.15.89:0/3314545952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07a80d2710 0x7f07a8156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.887+0000 7f07a7400700 1 --2- 172.21.15.89:0/3314545952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07a815ddf0 0x7f07a80b8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.888+0000 7f07a6a00700 1 -- 172.21.15.89:0/3314545952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07a815ddf0 msgr2=0x7f07a80b8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.888+0000 7f07a6a00700 1 --2- 172.21.15.89:0/3314545952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07a815ddf0 0x7f07a80b8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.888+0000 7f07a6a00700 1 -- 172.21.15.89:0/3314545952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07a80d2710 msgr2=0x7f07a8156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.888+0000 7f07a6a00700 1 --2- 172.21.15.89:0/3314545952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07a80d2710 0x7f07a8156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.888+0000 7f07a6a00700 1 -- 172.21.15.89:0/3314545952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0794009550 con 0x7f07a80ced30 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.888+0000 7f07a6a00700 1 --2- 172.21.15.89:0/3314545952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07a80ced30 0x7f07a80b36c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f079800c450 tx=0x7f079800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.888+0000 7f079ea00700 1 -- 172.21.15.89:0/3314545952 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07980052d0 con 0x7f07a80ced30 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.888+0000 7f079ea00700 1 -- 172.21.15.89:0/3314545952 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0798005430 con 0x7f07a80ced30 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.888+0000 7f079ea00700 1 -- 172.21.15.89:0/3314545952 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07980038c0 con 0x7f07a80ced30 2024-04-02T07:49:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.888+0000 7f07ad600700 1 -- 172.21.15.89:0/3314545952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a80b2be0 con 0x7f07a80ced30 2024-04-02T07:49:04.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.888+0000 7f07ad600700 1 -- 172.21.15.89:0/3314545952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a8157040 con 0x7f07a80ced30 2024-04-02T07:49:04.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.889+0000 7f07ad600700 1 -- 172.21.15.89:0/3314545952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a80b3510 con 0x7f07a80ced30 2024-04-02T07:49:04.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.890+0000 7f079ea00700 1 -- 172.21.15.89:0/3314545952 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0798003a20 con 0x7f07a80ced30 2024-04-02T07:49:04.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.891+0000 7f079ea00700 1 --2- 172.21.15.89:0/3314545952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f078806adb0 0x7f078806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:04.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.891+0000 7f079ea00700 1 -- 172.21.15.89:0/3314545952 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f07980885d0 con 0x7f07a80ced30 2024-04-02T07:49:04.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.891+0000 7f07a6000700 1 --2- 172.21.15.89:0/3314545952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f078806adb0 0x7f078806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:04.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.892+0000 7f07a6000700 1 --2- 172.21.15.89:0/3314545952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f078806adb0 0x7f078806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0794009c40 tx=0x7f0794008030 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:04.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:04.895+0000 7f079ea00700 1 -- 172.21.15.89:0/3314545952 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0798053880 con 0x7f07a80ced30 2024-04-02T07:49:05.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.099+0000 7f07ad600700 1 -- 172.21.15.89:0/3314545952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f07a8067720 con 0x7f07a80ced30 2024-04-02T07:49:05.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.743+0000 7f079ea00700 1 -- 172.21.15.89:0/3314545952 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v63) v1 ==== 138+0+0 (secure 0 0 0) 0x7f079800ce60 con 0x7f07a80ced30 2024-04-02T07:49:05.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.747+0000 7f07ad600700 1 -- 172.21.15.89:0/3314545952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f078806adb0 msgr2=0x7f078806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:05.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.747+0000 7f07ad600700 1 --2- 172.21.15.89:0/3314545952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f078806adb0 0x7f078806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0794009c40 tx=0x7f0794008030 comp rx=0 tx=0).stop 2024-04-02T07:49:05.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.747+0000 7f07ad600700 1 -- 172.21.15.89:0/3314545952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07a80ced30 msgr2=0x7f07a80b36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:05.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.747+0000 7f07ad600700 1 --2- 172.21.15.89:0/3314545952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07a80ced30 0x7f07a80b36c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f079800c450 tx=0x7f079800c760 comp rx=0 tx=0).stop 2024-04-02T07:49:05.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.749+0000 7f07ad600700 1 -- 172.21.15.89:0/3314545952 shutdown_connections 2024-04-02T07:49:05.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.749+0000 7f07ad600700 1 --2- 172.21.15.89:0/3314545952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f078806adb0 0x7f078806d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:05.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.749+0000 7f07ad600700 1 --2- 172.21.15.89:0/3314545952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f07a815ddf0 0x7f07a80b8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:05.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.749+0000 7f07ad600700 1 --2- 172.21.15.89:0/3314545952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07a80ced30 0x7f07a80b36c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:05.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.749+0000 7f07ad600700 1 --2- 172.21.15.89:0/3314545952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07a80d2710 0x7f07a8156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:05.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.749+0000 7f07ad600700 1 -- 172.21.15.89:0/3314545952 >> 172.21.15.89:0/3314545952 conn(0x7f07a808aff0 msgr2=0x7f07a80b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:05.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.751+0000 7f07ad600700 1 -- 172.21.15.89:0/3314545952 shutdown_connections 2024-04-02T07:49:05.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.752+0000 7f07ad600700 1 -- 172.21.15.89:0/3314545952 wait complete. 2024-04-02T07:49:05.757 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T07:49:05.775 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T07:49:05.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.973+0000 7f599bc00700 1 Processor -- start 2024-04-02T07:49:05.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.973+0000 7f599bc00700 1 -- start start 2024-04-02T07:49:05.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.974+0000 7f599bc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f59941611b0 0x7f599415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:05.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.974+0000 7f599bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f599415f4d0 0x7f59940ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:05.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.974+0000 7f599bc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59940ce270 con 0x7f59940d21f0 2024-04-02T07:49:05.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.974+0000 7f599bc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f599415b600 con 0x7f59941611b0 2024-04-02T07:49:05.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.974+0000 7f599bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59940d2760 con 0x7f599415f4d0 2024-04-02T07:49:05.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.975+0000 7f5999200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f59941611b0 0x7f599415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:05.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.975+0000 7f5999200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f59941611b0 0x7f599415ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36218/0 (socket says 172.21.15.89:36218) 2024-04-02T07:49:05.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.975+0000 7f5999200700 1 -- 172.21.15.89:0/610276928 learned_addr learned my addr 172.21.15.89:0/610276928 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:05.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.975+0000 7f5993e00700 1 --2- 172.21.15.89:0/610276928 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f599415f4d0 0x7f59940ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:05.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.975+0000 7f5999200700 1 -- 172.21.15.89:0/610276928 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f599415f4d0 msgr2=0x7f59940ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:05.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.975+0000 7f5999200700 1 --2- 172.21.15.89:0/610276928 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f599415f4d0 0x7f59940ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:05.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.975+0000 7f5999200700 1 -- 172.21.15.89:0/610276928 >> v1:172.21.15.113:6789/0 conn(0x7f59940d21f0 legacy=0x7f5994161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:05.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.975+0000 7f5999200700 1 -- 172.21.15.89:0/610276928 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59940d2a40 con 0x7f59941611b0 2024-04-02T07:49:05.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.976+0000 7f5999200700 1 --2- 172.21.15.89:0/610276928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59941611b0 0x7f599415ef90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f59880098f0 tx=0x7f5988009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc79672a9ae5a1f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:05.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.976+0000 7f5993400700 1 -- 172.21.15.89:0/610276928 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598800b8a0 con 0x7f59941611b0 2024-04-02T07:49:05.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.976+0000 7f5993400700 1 -- 172.21.15.89:0/610276928 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598800ba00 con 0x7f59941611b0 2024-04-02T07:49:05.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.976+0000 7f5993400700 1 -- 172.21.15.89:0/610276928 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598800bd90 con 0x7f59941611b0 2024-04-02T07:49:05.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.976+0000 7f599bc00700 1 -- 172.21.15.89:0/610276928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59941611b0 msgr2=0x7f599415ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:05.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.976+0000 7f599bc00700 1 --2- 172.21.15.89:0/610276928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59941611b0 0x7f599415ef90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f59880098f0 tx=0x7f5988009c00 comp rx=0 tx=0).stop 2024-04-02T07:49:05.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.976+0000 7f599bc00700 1 -- 172.21.15.89:0/610276928 shutdown_connections 2024-04-02T07:49:05.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.976+0000 7f599bc00700 1 --2- 172.21.15.89:0/610276928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59941611b0 0x7f599415ef90 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:05.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.976+0000 7f599bc00700 1 --2- 172.21.15.89:0/610276928 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f599415f4d0 0x7f59940ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:05.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.976+0000 7f599bc00700 1 -- 172.21.15.89:0/610276928 >> 172.21.15.89:0/610276928 conn(0x7f599408aff0 msgr2=0x7f59940d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:05.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.976+0000 7f599bc00700 1 -- 172.21.15.89:0/610276928 shutdown_connections 2024-04-02T07:49:05.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.976+0000 7f599bc00700 1 -- 172.21.15.89:0/610276928 wait complete. 2024-04-02T07:49:05.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.978+0000 7f599bc00700 1 Processor -- start 2024-04-02T07:49:05.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.978+0000 7f599bc00700 1 -- start start 2024-04-02T07:49:05.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.979+0000 7f599bc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59940d21f0 0x7f59940ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:05.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.979+0000 7f599bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f599415f4d0 0x7f5994153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:05.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.979+0000 7f599bc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59941611b0 0x7f59940b8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:05.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.979+0000 7f599bc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5994155210 con 0x7f59940d21f0 2024-04-02T07:49:05.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.979+0000 7f599bc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5994153f80 con 0x7f59941611b0 2024-04-02T07:49:05.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.979+0000 7f599bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5994154590 con 0x7f599415f4d0 2024-04-02T07:49:05.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.979+0000 7f5999200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59940d21f0 0x7f59940ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:05.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.979+0000 7f5999200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59940d21f0 0x7f59940ca770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36220/0 (socket says 172.21.15.89:36220) 2024-04-02T07:49:05.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.979+0000 7f5999200700 1 -- 172.21.15.89:0/116028059 learned_addr learned my addr 172.21.15.89:0/116028059 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:05.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.979+0000 7f5999c00700 1 --2- 172.21.15.89:0/116028059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59941611b0 0x7f59940b8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:05.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.979+0000 7f5993e00700 1 --2- 172.21.15.89:0/116028059 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f599415f4d0 0x7f5994153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:05.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.980+0000 7f5999200700 1 -- 172.21.15.89:0/116028059 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f599415f4d0 msgr2=0x7f5994153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:05.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.980+0000 7f5999200700 1 --2- 172.21.15.89:0/116028059 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f599415f4d0 0x7f5994153a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:05.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.980+0000 7f5999200700 1 -- 172.21.15.89:0/116028059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59941611b0 msgr2=0x7f59940b8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:05.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.980+0000 7f5999200700 1 --2- 172.21.15.89:0/116028059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59941611b0 0x7f59940b8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:05.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.980+0000 7f5999200700 1 -- 172.21.15.89:0/116028059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5988009550 con 0x7f59940d21f0 2024-04-02T07:49:05.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.980+0000 7f5999200700 1 --2- 172.21.15.89:0/116028059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59940d21f0 0x7f59940ca770 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5988009c40 tx=0x7f5988004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:05.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.980+0000 7f5990c00700 1 -- 172.21.15.89:0/116028059 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988004590 con 0x7f59940d21f0 2024-04-02T07:49:05.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.980+0000 7f5990c00700 1 -- 172.21.15.89:0/116028059 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598801e070 con 0x7f59940d21f0 2024-04-02T07:49:05.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.980+0000 7f5990c00700 1 -- 172.21.15.89:0/116028059 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59880038e0 con 0x7f59940d21f0 2024-04-02T07:49:05.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.980+0000 7f599bc00700 1 -- 172.21.15.89:0/116028059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5994154870 con 0x7f59940d21f0 2024-04-02T07:49:05.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.981+0000 7f599bc00700 1 -- 172.21.15.89:0/116028059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5994154e30 con 0x7f59940d21f0 2024-04-02T07:49:05.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.982+0000 7f599bc00700 1 -- 172.21.15.89:0/116028059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5994155350 con 0x7f59940d21f0 2024-04-02T07:49:05.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.984+0000 7f5990c00700 1 -- 172.21.15.89:0/116028059 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5988003a40 con 0x7f59940d21f0 2024-04-02T07:49:05.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.985+0000 7f5990c00700 1 --2- 172.21.15.89:0/116028059 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f597406adb0 0x7f597406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:05.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.985+0000 7f5990c00700 1 -- 172.21.15.89:0/116028059 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f5988020070 con 0x7f59940d21f0 2024-04-02T07:49:05.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.985+0000 7f5993e00700 1 --2- 172.21.15.89:0/116028059 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f597406adb0 0x7f597406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:05.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.986+0000 7f5993e00700 1 --2- 172.21.15.89:0/116028059 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f597406adb0 0x7f597406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f59940c9d40 tx=0x7f5980008040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:05.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:05.989+0000 7f5990c00700 1 -- 172.21.15.89:0/116028059 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5988055d10 con 0x7f59940d21f0 2024-04-02T07:49:06.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.191+0000 7f599bc00700 1 -- 172.21.15.89:0/116028059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5994067720 con 0x7f59940d21f0 2024-04-02T07:49:06.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.751+0000 7f5990c00700 1 -- 172.21.15.89:0/116028059 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v64) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5988059330 con 0x7f59940d21f0 2024-04-02T07:49:06.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.757+0000 7f599bc00700 1 -- 172.21.15.89:0/116028059 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f597406adb0 msgr2=0x7f597406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:06.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.757+0000 7f599bc00700 1 --2- 172.21.15.89:0/116028059 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f597406adb0 0x7f597406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f59940c9d40 tx=0x7f5980008040 comp rx=0 tx=0).stop 2024-04-02T07:49:06.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.757+0000 7f599bc00700 1 -- 172.21.15.89:0/116028059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59940d21f0 msgr2=0x7f59940ca770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:06.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.757+0000 7f599bc00700 1 --2- 172.21.15.89:0/116028059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59940d21f0 0x7f59940ca770 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5988009c40 tx=0x7f5988004750 comp rx=0 tx=0).stop 2024-04-02T07:49:06.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.758+0000 7f599bc00700 1 -- 172.21.15.89:0/116028059 shutdown_connections 2024-04-02T07:49:06.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.758+0000 7f599bc00700 1 --2- 172.21.15.89:0/116028059 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f597406adb0 0x7f597406d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:06.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.758+0000 7f599bc00700 1 --2- 172.21.15.89:0/116028059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59941611b0 0x7f59940b8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:06.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.758+0000 7f599bc00700 1 --2- 172.21.15.89:0/116028059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59940d21f0 0x7f59940ca770 secure :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5988009c40 tx=0x7f5988004750 comp rx=0 tx=0).stop 2024-04-02T07:49:06.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.758+0000 7f599bc00700 1 --2- 172.21.15.89:0/116028059 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f599415f4d0 0x7f5994153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:06.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.758+0000 7f599bc00700 1 -- 172.21.15.89:0/116028059 >> 172.21.15.89:0/116028059 conn(0x7f599408aff0 msgr2=0x7f5994161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:06.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.759+0000 7f599bc00700 1 -- 172.21.15.89:0/116028059 shutdown_connections 2024-04-02T07:49:06.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.759+0000 7f599bc00700 1 -- 172.21.15.89:0/116028059 wait complete. 2024-04-02T07:49:06.764 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T07:49:06.791 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T07:49:06.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.982+0000 7f5db9400700 1 Processor -- start 2024-04-02T07:49:06.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.982+0000 7f5db9400700 1 -- start start 2024-04-02T07:49:06.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.983+0000 7f5db9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db40d16a0 0x7f5db40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:06.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.983+0000 7f5db9400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db4161420 con 0x7f5db415d9d0 2024-04-02T07:49:06.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.983+0000 7f5db9400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db4161700 con 0x7f5db40d2370 2024-04-02T07:49:06.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.983+0000 7f5db9400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db40d2820 con 0x7f5db40d16a0 2024-04-02T07:49:06.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.983+0000 7f5db2800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5db415d9d0 0x7f5db40d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44570/0 (socket says 172.21.15.89:44570) 2024-04-02T07:49:06.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.983+0000 7f5db2800700 1 -- 172.21.15.89:0/370964729 learned_addr learned my addr 172.21.15.89:0/370964729 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:06.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.983+0000 7f5dabe00700 1 --2- 172.21.15.89:0/370964729 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db40d16a0 0x7f5db40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:06.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.984+0000 7f5db1e00700 1 -- 172.21.15.89:0/370964729 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623069639 0 0) 0x7f5db4161420 con 0x7f5db415d9d0 2024-04-02T07:49:06.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.984+0000 7f5db1e00700 1 -- 172.21.15.89:0/370964729 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d980035b0 con 0x7f5db415d9d0 2024-04-02T07:49:06.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.984+0000 7f5db1e00700 1 -- 172.21.15.89:0/370964729 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530645573 0 0) 0x7f5db4161700 con 0x7f5db40d2370 2024-04-02T07:49:06.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.984+0000 7f5db1e00700 1 -- 172.21.15.89:0/370964729 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5db4161420 con 0x7f5db40d2370 2024-04-02T07:49:06.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.984+0000 7f5dabe00700 1 -- 172.21.15.89:0/370964729 >> v1:172.21.15.113:6789/0 conn(0x7f5db40d2370 legacy=0x7f5db40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:06.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.984+0000 7f5dabe00700 1 -- 172.21.15.89:0/370964729 >> v1:172.21.15.89:6789/0 conn(0x7f5db415d9d0 legacy=0x7f5db40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:06.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.984+0000 7f5dabe00700 1 -- 172.21.15.89:0/370964729 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db41619e0 con 0x7f5db40d16a0 2024-04-02T07:49:06.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.984+0000 7f5dabe00700 1 --2- 172.21.15.89:0/370964729 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db40d16a0 0x7f5db40d1e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009850 tx=0x7f5d9c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57a8aa3525dbf47b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:06.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.985+0000 7f5db1e00700 1 -- 172.21.15.89:0/370964729 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c005e30 con 0x7f5db40d16a0 2024-04-02T07:49:06.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.985+0000 7f5db1e00700 1 -- 172.21.15.89:0/370964729 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9c006030 con 0x7f5db40d16a0 2024-04-02T07:49:06.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.985+0000 7f5db1e00700 1 -- 172.21.15.89:0/370964729 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c0063e0 con 0x7f5db40d16a0 2024-04-02T07:49:06.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.985+0000 7f5db9400700 1 -- 172.21.15.89:0/370964729 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db40d16a0 msgr2=0x7f5db40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:06.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.985+0000 7f5db9400700 1 --2- 172.21.15.89:0/370964729 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db40d16a0 0x7f5db40d1e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009850 tx=0x7f5d9c009b60 comp rx=0 tx=0).stop 2024-04-02T07:49:06.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.985+0000 7f5db9400700 1 -- 172.21.15.89:0/370964729 shutdown_connections 2024-04-02T07:49:06.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.985+0000 7f5db9400700 1 --2- 172.21.15.89:0/370964729 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db40d16a0 0x7f5db40d1e30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:06.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.985+0000 7f5db9400700 1 -- 172.21.15.89:0/370964729 >> 172.21.15.89:0/370964729 conn(0x7f5db408aff0 msgr2=0x7f5db40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:06.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.985+0000 7f5db9400700 1 -- 172.21.15.89:0/370964729 shutdown_connections 2024-04-02T07:49:06.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.985+0000 7f5db9400700 1 -- 172.21.15.89:0/370964729 wait complete. 2024-04-02T07:49:06.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.987+0000 7f5db9400700 1 Processor -- start 2024-04-02T07:49:06.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.987+0000 7f5db9400700 1 -- start start 2024-04-02T07:49:06.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.988+0000 7f5db9400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db40d16a0 0x7f5db41e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:06.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.988+0000 7f5db9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db40d2370 0x7f5db41601c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:06.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.988+0000 7f5db9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5db415d9d0 0x7f5db40b6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:06.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.988+0000 7f5db9400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db415ebb0 con 0x7f5db415d9d0 2024-04-02T07:49:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.988+0000 7f5db9400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db41e81d0 con 0x7f5db40d16a0 2024-04-02T07:49:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.988+0000 7f5db9400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db415df30 con 0x7f5db40d2370 2024-04-02T07:49:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.988+0000 7f5db3200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5db415d9d0 0x7f5db40b6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.988+0000 7f5db3200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5db415d9d0 0x7f5db40b6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36226/0 (socket says 172.21.15.89:36226) 2024-04-02T07:49:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.988+0000 7f5db3200700 1 -- 172.21.15.89:0/97509192 learned_addr learned my addr 172.21.15.89:0/97509192 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.988+0000 7f5dabe00700 1 --2- 172.21.15.89:0/97509192 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db40d2370 0x7f5db41601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.988+0000 7f5db2800700 1 --2- 172.21.15.89:0/97509192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db40d16a0 0x7f5db41e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.989+0000 7f5db3200700 1 -- 172.21.15.89:0/97509192 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db40d2370 msgr2=0x7f5db41601c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.989+0000 7f5db3200700 1 --2- 172.21.15.89:0/97509192 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db40d2370 0x7f5db41601c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.989+0000 7f5db3200700 1 -- 172.21.15.89:0/97509192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db40d16a0 msgr2=0x7f5db41e7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.989+0000 7f5db3200700 1 --2- 172.21.15.89:0/97509192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db40d16a0 0x7f5db41e7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.989+0000 7f5db3200700 1 -- 172.21.15.89:0/97509192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9c009430 con 0x7f5db415d9d0 2024-04-02T07:49:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.989+0000 7f5db3200700 1 --2- 172.21.15.89:0/97509192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5db415d9d0 0x7f5db40b6c10 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5dac00cc20 tx=0x7f5dac00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.989+0000 7f5daaa00700 1 -- 172.21.15.89:0/97509192 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac003c00 con 0x7f5db415d9d0 2024-04-02T07:49:06.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.989+0000 7f5daaa00700 1 -- 172.21.15.89:0/97509192 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dac003d60 con 0x7f5db415d9d0 2024-04-02T07:49:06.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.989+0000 7f5daaa00700 1 -- 172.21.15.89:0/97509192 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac00aae0 con 0x7f5db415d9d0 2024-04-02T07:49:06.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.989+0000 7f5db9400700 1 -- 172.21.15.89:0/97509192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db415e210 con 0x7f5db415d9d0 2024-04-02T07:49:06.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.989+0000 7f5db9400700 1 -- 172.21.15.89:0/97509192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db415e7d0 con 0x7f5db415d9d0 2024-04-02T07:49:06.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.990+0000 7f5db9400700 1 -- 172.21.15.89:0/97509192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db40d1ca0 con 0x7f5db415d9d0 2024-04-02T07:49:06.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.992+0000 7f5daaa00700 1 -- 172.21.15.89:0/97509192 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5dac00ac40 con 0x7f5db415d9d0 2024-04-02T07:49:06.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.992+0000 7f5daaa00700 1 --2- 172.21.15.89:0/97509192 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5d9006adb0 0x7f5d9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.993+0000 7f5daaa00700 1 -- 172.21.15.89:0/97509192 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f5dac08a100 con 0x7f5db415d9d0 2024-04-02T07:49:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.993+0000 7f5db2800700 1 --2- 172.21.15.89:0/97509192 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5d9006adb0 0x7f5d9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.993+0000 7f5db2800700 1 --2- 172.21.15.89:0/97509192 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5d9006adb0 0x7f5d9006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5da4007fd0 tx=0x7f5da4007dc0 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:07.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:06.997+0000 7f5daaa00700 1 -- 172.21.15.89:0/97509192 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5dac0550b0 con 0x7f5db415d9d0 2024-04-02T07:49:07.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.246+0000 7f5db9400700 1 -- 172.21.15.89:0/97509192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f5db415ecf0 con 0x7f5db415d9d0 2024-04-02T07:49:07.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.255+0000 7f5daaa00700 1 -- 172.21.15.89:0/97509192 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7f5db415ecf0 con 0x7f5db415d9d0 2024-04-02T07:49:07.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.259+0000 7f5db9400700 1 -- 172.21.15.89:0/97509192 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5d9006adb0 msgr2=0x7f5d9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.259+0000 7f5db9400700 1 --2- 172.21.15.89:0/97509192 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5d9006adb0 0x7f5d9006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5da4007fd0 tx=0x7f5da4007dc0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.259+0000 7f5db9400700 1 -- 172.21.15.89:0/97509192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5db415d9d0 msgr2=0x7f5db40b6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.259+0000 7f5db9400700 1 --2- 172.21.15.89:0/97509192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5db415d9d0 0x7f5db40b6c10 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5dac00cc20 tx=0x7f5dac00cf30 comp rx=0 tx=0).stop 2024-04-02T07:49:07.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.260+0000 7f5db9400700 1 -- 172.21.15.89:0/97509192 shutdown_connections 2024-04-02T07:49:07.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.260+0000 7f5db9400700 1 --2- 172.21.15.89:0/97509192 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5d9006adb0 0x7f5d9006d270 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5da4007fd0 tx=0x7f5da4007dc0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.260+0000 7f5db9400700 1 --2- 172.21.15.89:0/97509192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db40d16a0 0x7f5db41e7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.260+0000 7f5db9400700 1 --2- 172.21.15.89:0/97509192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5db415d9d0 0x7f5db40b6c10 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.260+0000 7f5db9400700 1 --2- 172.21.15.89:0/97509192 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db40d2370 0x7f5db41601c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.260+0000 7f5db9400700 1 -- 172.21.15.89:0/97509192 >> 172.21.15.89:0/97509192 conn(0x7f5db408aff0 msgr2=0x7f5db40b30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:07.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.265+0000 7f5db9400700 1 -- 172.21.15.89:0/97509192 shutdown_connections 2024-04-02T07:49:07.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.265+0000 7f5db9400700 1 -- 172.21.15.89:0/97509192 wait complete. 2024-04-02T07:49:07.272 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 6 and data pool 7 2024-04-02T07:49:07.302 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:49:07.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.489+0000 7f5222c00700 1 Processor -- start 2024-04-02T07:49:07.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.489+0000 7f5222c00700 1 -- start start 2024-04-02T07:49:07.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.490+0000 7f5222c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f521c15b5d0 0x7f521c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:07.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.490+0000 7f5222c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f521c0ac7d0 0x7f521c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:07.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.490+0000 7f5222c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c15fd50 con 0x7f521c0cb3f0 2024-04-02T07:49:07.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.490+0000 7f5222c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521c15b9b0 con 0x7f521c0ac7d0 2024-04-02T07:49:07.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.490+0000 7f5222c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521c15f140 con 0x7f521c15b5d0 2024-04-02T07:49:07.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.490+0000 7f5220c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f521c0ac7d0 0x7f521c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:07.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.490+0000 7f5220c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f521c0ac7d0 0x7f521c06c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36238/0 (socket says 172.21.15.89:36238) 2024-04-02T07:49:07.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.490+0000 7f5220c00700 1 -- 172.21.15.89:0/1273580870 learned_addr learned my addr 172.21.15.89:0/1273580870 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:07.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.491+0000 7f521be00700 1 --2- 172.21.15.89:0/1273580870 >> v2:172.21.15.113:3300/0 conn(0x7f521c15b5d0 0x7f521c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:07.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.491+0000 7f5220c00700 1 -- 172.21.15.89:0/1273580870 >> v2:172.21.15.113:3300/0 conn(0x7f521c15b5d0 msgr2=0x7f521c0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:07.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.491+0000 7f5220c00700 1 --2- 172.21.15.89:0/1273580870 >> v2:172.21.15.113:3300/0 conn(0x7f521c15b5d0 0x7f521c0ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.491+0000 7f5220c00700 1 -- 172.21.15.89:0/1273580870 >> v1:172.21.15.113:6789/0 conn(0x7f521c0cb3f0 legacy=0x7f521c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:07.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.491+0000 7f5220c00700 1 -- 172.21.15.89:0/1273580870 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c15f420 con 0x7f521c0ac7d0 2024-04-02T07:49:07.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.491+0000 7f521be00700 1 --2- 172.21.15.89:0/1273580870 >> v2:172.21.15.113:3300/0 conn(0x7f521c15b5d0 0x7f521c0ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:49:07.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.491+0000 7f5220c00700 1 --2- 172.21.15.89:0/1273580870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f521c0ac7d0 0x7f521c06c9d0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f521c06b8b0 tx=0x7f521400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d2e0071f988e3b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:07.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.491+0000 7f521b400700 1 -- 172.21.15.89:0/1273580870 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521400e070 con 0x7f521c0ac7d0 2024-04-02T07:49:07.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.491+0000 7f521b400700 1 -- 172.21.15.89:0/1273580870 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521400b9c0 con 0x7f521c0ac7d0 2024-04-02T07:49:07.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.491+0000 7f521b400700 1 -- 172.21.15.89:0/1273580870 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521400bd70 con 0x7f521c0ac7d0 2024-04-02T07:49:07.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.492+0000 7f5222c00700 1 -- 172.21.15.89:0/1273580870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f521c0ac7d0 msgr2=0x7f521c06c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:07.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.492+0000 7f5222c00700 1 --2- 172.21.15.89:0/1273580870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f521c0ac7d0 0x7f521c06c9d0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f521c06b8b0 tx=0x7f521400b330 comp rx=0 tx=0).stop 2024-04-02T07:49:07.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.492+0000 7f5222c00700 1 -- 172.21.15.89:0/1273580870 shutdown_connections 2024-04-02T07:49:07.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.492+0000 7f5222c00700 1 --2- 172.21.15.89:0/1273580870 >> v2:172.21.15.113:3300/0 conn(0x7f521c15b5d0 0x7f521c0ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.492+0000 7f5222c00700 1 --2- 172.21.15.89:0/1273580870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f521c0ac7d0 0x7f521c06c9d0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.492+0000 7f5222c00700 1 -- 172.21.15.89:0/1273580870 >> 172.21.15.89:0/1273580870 conn(0x7f521c0cf190 msgr2=0x7f521c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:07.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.492+0000 7f5222c00700 1 -- 172.21.15.89:0/1273580870 shutdown_connections 2024-04-02T07:49:07.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.492+0000 7f5222c00700 1 -- 172.21.15.89:0/1273580870 wait complete. 2024-04-02T07:49:07.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.494+0000 7f5222c00700 1 Processor -- start 2024-04-02T07:49:07.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.495+0000 7f5222c00700 1 -- start start 2024-04-02T07:49:07.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5222c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0ac7d0 0x7f521c1564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:07.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5222c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f521c15b5d0 0x7f521c156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:07.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5222c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f521c0cbba0 0x7f521c071010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:07.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5222c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0cccb0 con 0x7f521c15b5d0 2024-04-02T07:49:07.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5222c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0cd0d0 con 0x7f521c0ac7d0 2024-04-02T07:49:07.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5222c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0cc030 con 0x7f521c0cbba0 2024-04-02T07:49:07.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5213e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f521c15b5d0 0x7f521c156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:07.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5213e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f521c15b5d0 0x7f521c156b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36252/0 (socket says 172.21.15.89:36252) 2024-04-02T07:49:07.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5213e00700 1 -- 172.21.15.89:0/705541744 learned_addr learned my addr 172.21.15.89:0/705541744 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:07.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f521be00700 1 --2- 172.21.15.89:0/705541744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0ac7d0 0x7f521c1564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:07.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5220c00700 1 --2- 172.21.15.89:0/705541744 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f521c0cbba0 0x7f521c071010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:07.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5213e00700 1 -- 172.21.15.89:0/705541744 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f521c0cbba0 msgr2=0x7f521c071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:07.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5213e00700 1 --2- 172.21.15.89:0/705541744 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f521c0cbba0 0x7f521c071010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5213e00700 1 -- 172.21.15.89:0/705541744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0ac7d0 msgr2=0x7f521c1564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:07.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5213e00700 1 --2- 172.21.15.89:0/705541744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0ac7d0 0x7f521c1564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.496+0000 7f5213e00700 1 -- 172.21.15.89:0/705541744 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5214009b30 con 0x7f521c15b5d0 2024-04-02T07:49:07.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.497+0000 7f5213e00700 1 --2- 172.21.15.89:0/705541744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f521c15b5d0 0x7f521c156b80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f520400c4e0 tx=0x7f520400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:07.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.497+0000 7f5218c00700 1 -- 172.21.15.89:0/705541744 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52040052d0 con 0x7f521c15b5d0 2024-04-02T07:49:07.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.497+0000 7f5218c00700 1 -- 172.21.15.89:0/705541744 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5204005430 con 0x7f521c15b5d0 2024-04-02T07:49:07.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.497+0000 7f5218c00700 1 -- 172.21.15.89:0/705541744 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52040038c0 con 0x7f521c15b5d0 2024-04-02T07:49:07.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.497+0000 7f5222c00700 1 -- 172.21.15.89:0/705541744 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521c0cc310 con 0x7f521c15b5d0 2024-04-02T07:49:07.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.497+0000 7f5222c00700 1 -- 172.21.15.89:0/705541744 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521c0cc8d0 con 0x7f521c15b5d0 2024-04-02T07:49:07.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.498+0000 7f5222c00700 1 -- 172.21.15.89:0/705541744 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521c0ccdf0 con 0x7f521c15b5d0 2024-04-02T07:49:07.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.499+0000 7f5218c00700 1 -- 172.21.15.89:0/705541744 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5204003a20 con 0x7f521c15b5d0 2024-04-02T07:49:07.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.500+0000 7f5218c00700 1 --2- 172.21.15.89:0/705541744 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f51f806adb0 0x7f51f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:07.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.500+0000 7f5218c00700 1 -- 172.21.15.89:0/705541744 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5204088990 con 0x7f521c15b5d0 2024-04-02T07:49:07.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.500+0000 7f521be00700 1 --2- 172.21.15.89:0/705541744 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f51f806adb0 0x7f51f806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:07.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.501+0000 7f521be00700 1 --2- 172.21.15.89:0/705541744 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f51f806adb0 0x7f51f806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f520c00c940 tx=0x7f520c008040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.504+0000 7f5218c00700 1 -- 172.21.15.89:0/705541744 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f521c0ccdf0 con 0x7f521c15b5d0 2024-04-02T07:49:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.705+0000 7f5222c00700 1 -- 172.21.15.89:0/705541744 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f521c0cca90 con 0x7f521c15b5d0 2024-04-02T07:49:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.706+0000 7f5218c00700 1 -- 172.21.15.89:0/705541744 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16581 (secure 0 0 0) 0x7f5204017090 con 0x7f521c15b5d0 2024-04-02T07:49:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:49:07.709 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":65,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:49:07.248605+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:49:05.141462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-02T07:49:06.232906+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:49:07.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.708+0000 7f5222c00700 1 -- 172.21.15.89:0/705541744 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f51f806adb0 msgr2=0x7f51f806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:07.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.708+0000 7f5222c00700 1 --2- 172.21.15.89:0/705541744 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f51f806adb0 0x7f51f806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f520c00c940 tx=0x7f520c008040 comp rx=0 tx=0).stop 2024-04-02T07:49:07.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.708+0000 7f5222c00700 1 -- 172.21.15.89:0/705541744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f521c15b5d0 msgr2=0x7f521c156b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:07.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.708+0000 7f5222c00700 1 --2- 172.21.15.89:0/705541744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f521c15b5d0 0x7f521c156b80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f520400c4e0 tx=0x7f520400c7f0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.708+0000 7f5222c00700 1 -- 172.21.15.89:0/705541744 shutdown_connections 2024-04-02T07:49:07.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.708+0000 7f5222c00700 1 --2- 172.21.15.89:0/705541744 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f51f806adb0 0x7f51f806d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.708+0000 7f5222c00700 1 --2- 172.21.15.89:0/705541744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0ac7d0 0x7f521c1564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.708+0000 7f5222c00700 1 --2- 172.21.15.89:0/705541744 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f521c0cbba0 0x7f521c071010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.708+0000 7f5222c00700 1 --2- 172.21.15.89:0/705541744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f521c15b5d0 0x7f521c156b80 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.708+0000 7f5222c00700 1 -- 172.21.15.89:0/705541744 >> 172.21.15.89:0/705541744 conn(0x7f521c0cf190 msgr2=0x7f521c15e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.709+0000 7f5222c00700 1 -- 172.21.15.89:0/705541744 shutdown_connections 2024-04-02T07:49:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.709+0000 7f5222c00700 1 -- 172.21.15.89:0/705541744 wait complete. 2024-04-02T07:49:07.730 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:49:07.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.919+0000 7fc484a00700 1 Processor -- start 2024-04-02T07:49:07.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.919+0000 7fc484a00700 1 -- start start 2024-04-02T07:49:07.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.920+0000 7fc484a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4800ac7d0 0x7fc480166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.920+0000 7fc484a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48015fd50 con 0x7fc4800cb3f0 2024-04-02T07:49:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.920+0000 7fc484a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48015f140 con 0x7fc48015b5d0 2024-04-02T07:49:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.920+0000 7fc484a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48015b9b0 con 0x7fc4800ac7d0 2024-04-02T07:49:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.920+0000 7fc477e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc4800cb3f0 0x7fc4800ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44580/0 (socket says 172.21.15.89:44580) 2024-04-02T07:49:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.920+0000 7fc477e00700 1 -- 172.21.15.89:0/1325669094 learned_addr learned my addr 172.21.15.89:0/1325669094 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.920+0000 7fc47e800700 1 --2- 172.21.15.89:0/1325669094 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4800ac7d0 0x7fc480166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.921+0000 7fc47d400700 1 -- 172.21.15.89:0/1325669094 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147640239 0 0) 0x7fc48015fd50 con 0x7fc4800cb3f0 2024-04-02T07:49:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.921+0000 7fc47d400700 1 -- 172.21.15.89:0/1325669094 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4640035b0 con 0x7fc4800cb3f0 2024-04-02T07:49:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.921+0000 7fc47d400700 1 -- 172.21.15.89:0/1325669094 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126771728 0 0) 0x7fc48015f140 con 0x7fc48015b5d0 2024-04-02T07:49:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.921+0000 7fc47d400700 1 -- 172.21.15.89:0/1325669094 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc48015fd50 con 0x7fc48015b5d0 2024-04-02T07:49:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.921+0000 7fc47e800700 1 -- 172.21.15.89:0/1325669094 >> v1:172.21.15.113:6789/0 conn(0x7fc48015b5d0 legacy=0x7fc4800ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.921+0000 7fc47d400700 1 -- 172.21.15.89:0/1325669094 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2739618430 0 0) 0x7fc4640035b0 con 0x7fc4800cb3f0 2024-04-02T07:49:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.921+0000 7fc47e800700 1 -- 172.21.15.89:0/1325669094 >> v1:172.21.15.89:6789/0 conn(0x7fc4800cb3f0 legacy=0x7fc4800ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.921+0000 7fc47e800700 1 -- 172.21.15.89:0/1325669094 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48015f420 con 0x7fc4800ac7d0 2024-04-02T07:49:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.921+0000 7fc47e800700 1 --2- 172.21.15.89:0/1325669094 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4800ac7d0 0x7fc480166430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc478009dc0 tx=0x7fc47800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14f3b260c2b98a2f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.922+0000 7fc47d400700 1 -- 172.21.15.89:0/1325669094 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47800e070 con 0x7fc4800ac7d0 2024-04-02T07:49:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.922+0000 7fc47d400700 1 -- 172.21.15.89:0/1325669094 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc478003a20 con 0x7fc4800ac7d0 2024-04-02T07:49:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.922+0000 7fc47d400700 1 -- 172.21.15.89:0/1325669094 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc468001670 con 0x7fc4800ac7d0 2024-04-02T07:49:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.922+0000 7fc484a00700 1 -- 172.21.15.89:0/1325669094 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4800ac7d0 msgr2=0x7fc480166430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.922+0000 7fc484a00700 1 --2- 172.21.15.89:0/1325669094 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4800ac7d0 0x7fc480166430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc478009dc0 tx=0x7fc47800b330 comp rx=0 tx=0).stop 2024-04-02T07:49:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.922+0000 7fc484a00700 1 -- 172.21.15.89:0/1325669094 shutdown_connections 2024-04-02T07:49:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.922+0000 7fc484a00700 1 --2- 172.21.15.89:0/1325669094 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4800ac7d0 0x7fc480166430 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.922+0000 7fc484a00700 1 -- 172.21.15.89:0/1325669094 >> 172.21.15.89:0/1325669094 conn(0x7fc4800cf190 msgr2=0x7fc4800cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.922+0000 7fc484a00700 1 -- 172.21.15.89:0/1325669094 shutdown_connections 2024-04-02T07:49:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.922+0000 7fc484a00700 1 -- 172.21.15.89:0/1325669094 wait complete. 2024-04-02T07:49:07.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.924+0000 7fc484a00700 1 Processor -- start 2024-04-02T07:49:07.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.924+0000 7fc484a00700 1 -- start start 2024-04-02T07:49:07.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.925+0000 7fc484a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4800ac7d0 0x7fc480159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:07.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.925+0000 7fc484a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48015b5d0 0x7fc4800ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:07.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.925+0000 7fc484a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4800cd510 0x7fc480144ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:07.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.925+0000 7fc484a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4800cea40 con 0x7fc4800cd510 2024-04-02T07:49:07.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.925+0000 7fc484a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4800ce7a0 con 0x7fc4800ac7d0 2024-04-02T07:49:07.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.925+0000 7fc484a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4800cc750 con 0x7fc48015b5d0 2024-04-02T07:49:07.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.925+0000 7fc47e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4800cd510 0x7fc480144ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:07.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.925+0000 7fc47e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4800cd510 0x7fc480144ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36264/0 (socket says 172.21.15.89:36264) 2024-04-02T07:49:07.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.925+0000 7fc47e800700 1 -- 172.21.15.89:0/681596881 learned_addr learned my addr 172.21.15.89:0/681596881 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:07.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.925+0000 7fc477e00700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48015b5d0 0x7fc4800ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:07.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.925+0000 7fc47de00700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4800ac7d0 0x7fc480159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:07.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.926+0000 7fc47e800700 1 -- 172.21.15.89:0/681596881 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48015b5d0 msgr2=0x7fc4800ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:07.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.926+0000 7fc47e800700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48015b5d0 0x7fc4800ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.926+0000 7fc47e800700 1 -- 172.21.15.89:0/681596881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4800ac7d0 msgr2=0x7fc480159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:07.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.926+0000 7fc47e800700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4800ac7d0 0x7fc480159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:07.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.926+0000 7fc47e800700 1 -- 172.21.15.89:0/681596881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc478009a70 con 0x7fc4800cd510 2024-04-02T07:49:07.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.926+0000 7fc47de00700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4800ac7d0 0x7fc480159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:49:07.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.926+0000 7fc47e800700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4800cd510 0x7fc480144ad0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc478009d90 tx=0x7fc478004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:07.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.926+0000 7fc476000700 1 -- 172.21.15.89:0/681596881 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc478004540 con 0x7fc4800cd510 2024-04-02T07:49:07.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.926+0000 7fc476000700 1 -- 172.21.15.89:0/681596881 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4780046a0 con 0x7fc4800cd510 2024-04-02T07:49:07.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.926+0000 7fc484a00700 1 -- 172.21.15.89:0/681596881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4800cca30 con 0x7fc4800cd510 2024-04-02T07:49:07.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.926+0000 7fc476000700 1 -- 172.21.15.89:0/681596881 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47801d470 con 0x7fc4800cd510 2024-04-02T07:49:07.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.926+0000 7fc484a00700 1 -- 172.21.15.89:0/681596881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4800ccff0 con 0x7fc4800cd510 2024-04-02T07:49:07.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.927+0000 7fc484a00700 1 -- 172.21.15.89:0/681596881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc48015f9a0 con 0x7fc4800cd510 2024-04-02T07:49:07.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.928+0000 7fc476000700 1 -- 172.21.15.89:0/681596881 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc47801d5d0 con 0x7fc4800cd510 2024-04-02T07:49:07.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.929+0000 7fc476000700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc45c06adb0 0x7fc45c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:07.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.929+0000 7fc476000700 1 -- 172.21.15.89:0/681596881 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc47808b450 con 0x7fc4800cd510 2024-04-02T07:49:07.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.929+0000 7fc47de00700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc45c06adb0 0x7fc45c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:07.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.930+0000 7fc47de00700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc45c06adb0 0x7fc45c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc4700011e0 tx=0x7fc470009040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:07.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:07.934+0000 7fc476000700 1 -- 172.21.15.89:0/681596881 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc478056220 con 0x7fc4800cd510 2024-04-02T07:49:08.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.135+0000 7fc484a00700 1 -- 172.21.15.89:0/681596881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc480067720 con 0x7fc4800cd510 2024-04-02T07:49:08.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.136+0000 7fc476000700 1 -- 172.21.15.89:0/681596881 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16581 (secure 0 0 0) 0x7fc478059840 con 0x7fc4800cd510 2024-04-02T07:49:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:49:08.140 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":65,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:49:07.248605+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:49:05.141462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-02T07:49:06.232906+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:49:08.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.138+0000 7fc484a00700 1 -- 172.21.15.89:0/681596881 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc45c06adb0 msgr2=0x7fc45c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:08.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.138+0000 7fc484a00700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc45c06adb0 0x7fc45c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc4700011e0 tx=0x7fc470009040 comp rx=0 tx=0).stop 2024-04-02T07:49:08.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.138+0000 7fc484a00700 1 -- 172.21.15.89:0/681596881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4800cd510 msgr2=0x7fc480144ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:08.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.138+0000 7fc484a00700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4800cd510 0x7fc480144ad0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc478009d90 tx=0x7fc478004ad0 comp rx=0 tx=0).stop 2024-04-02T07:49:08.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.138+0000 7fc484a00700 1 -- 172.21.15.89:0/681596881 shutdown_connections 2024-04-02T07:49:08.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.139+0000 7fc484a00700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc45c06adb0 0x7fc45c06d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:08.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.139+0000 7fc484a00700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4800ac7d0 0x7fc480159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:08.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.139+0000 7fc484a00700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4800cd510 0x7fc480144ad0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:08.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.139+0000 7fc484a00700 1 --2- 172.21.15.89:0/681596881 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48015b5d0 0x7fc4800ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:08.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.139+0000 7fc484a00700 1 -- 172.21.15.89:0/681596881 >> 172.21.15.89:0/681596881 conn(0x7fc4800cf190 msgr2=0x7fc4800aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:08.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.139+0000 7fc484a00700 1 -- 172.21.15.89:0/681596881 shutdown_connections 2024-04-02T07:49:08.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.139+0000 7fc484a00700 1 -- 172.21.15.89:0/681596881 wait complete. 2024-04-02T07:49:08.160 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T07:49:08.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.352+0000 7f73c9a00700 1 Processor -- start 2024-04-02T07:49:08.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.352+0000 7f73c9a00700 1 -- start start 2024-04-02T07:49:08.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.353+0000 7f73c9a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73c415a5c0 0x7f73c40709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:08.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.353+0000 7f73c9a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c40cb310 con 0x7f73c40ce840 2024-04-02T07:49:08.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.353+0000 7f73c9a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c4070f00 con 0x7f73c415dfe0 2024-04-02T07:49:08.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.353+0000 7f73c9a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c40711a0 con 0x7f73c415a5c0 2024-04-02T07:49:08.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.354+0000 7f73c3400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f73c40ce840 0x7f73c40cba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44590/0 (socket says 172.21.15.89:44590) 2024-04-02T07:49:08.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.354+0000 7f73c3400700 1 -- 172.21.15.89:0/337546000 learned_addr learned my addr 172.21.15.89:0/337546000 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:08.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.354+0000 7f73c2a00700 1 --2- 172.21.15.89:0/337546000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73c415a5c0 0x7f73c40709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:08.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.354+0000 7f73c1600700 1 -- 172.21.15.89:0/337546000 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871916998 0 0) 0x7f73c40cb310 con 0x7f73c40ce840 2024-04-02T07:49:08.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.354+0000 7f73c1600700 1 -- 172.21.15.89:0/337546000 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a80035b0 con 0x7f73c40ce840 2024-04-02T07:49:08.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.354+0000 7f73c1600700 1 -- 172.21.15.89:0/337546000 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615147551 0 0) 0x7f73c4070f00 con 0x7f73c415dfe0 2024-04-02T07:49:08.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.354+0000 7f73c1600700 1 -- 172.21.15.89:0/337546000 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73c40cb310 con 0x7f73c415dfe0 2024-04-02T07:49:08.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.354+0000 7f73c2a00700 1 -- 172.21.15.89:0/337546000 >> v1:172.21.15.113:6789/0 conn(0x7f73c415dfe0 legacy=0x7f73c40ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:08.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.354+0000 7f73c2a00700 1 -- 172.21.15.89:0/337546000 >> v1:172.21.15.89:6789/0 conn(0x7f73c40ce840 legacy=0x7f73c40cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:08.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.354+0000 7f73c1600700 1 -- 172.21.15.89:0/337546000 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1500864833 0 0) 0x7f73a80035b0 con 0x7f73c40ce840 2024-04-02T07:49:08.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.354+0000 7f73c2a00700 1 -- 172.21.15.89:0/337546000 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c4071480 con 0x7f73c415a5c0 2024-04-02T07:49:08.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.355+0000 7f73c2a00700 1 --2- 172.21.15.89:0/337546000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73c415a5c0 0x7f73c40709c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f73b4009870 tx=0x7f73b4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d72e930c88b95453 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:08.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.355+0000 7f73c1600700 1 -- 172.21.15.89:0/337546000 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc0019f0 con 0x7f73c415a5c0 2024-04-02T07:49:08.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.355+0000 7f73c1600700 1 -- 172.21.15.89:0/337546000 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b4004030 con 0x7f73c415a5c0 2024-04-02T07:49:08.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.355+0000 7f73c1600700 1 -- 172.21.15.89:0/337546000 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b40043a0 con 0x7f73c415a5c0 2024-04-02T07:49:08.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.355+0000 7f73c9a00700 1 -- 172.21.15.89:0/337546000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73c415a5c0 msgr2=0x7f73c40709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:08.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.355+0000 7f73c9a00700 1 --2- 172.21.15.89:0/337546000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73c415a5c0 0x7f73c40709c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f73b4009870 tx=0x7f73b4009b80 comp rx=0 tx=0).stop 2024-04-02T07:49:08.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.355+0000 7f73c9a00700 1 -- 172.21.15.89:0/337546000 shutdown_connections 2024-04-02T07:49:08.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.355+0000 7f73c9a00700 1 --2- 172.21.15.89:0/337546000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73c415a5c0 0x7f73c40709c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:08.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.356+0000 7f73c9a00700 1 -- 172.21.15.89:0/337546000 >> 172.21.15.89:0/337546000 conn(0x7f73c40b0d70 msgr2=0x7f73c40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:08.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.356+0000 7f73c9a00700 1 -- 172.21.15.89:0/337546000 shutdown_connections 2024-04-02T07:49:08.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.356+0000 7f73c9a00700 1 -- 172.21.15.89:0/337546000 wait complete. 2024-04-02T07:49:08.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.358+0000 7f73c9a00700 1 Processor -- start 2024-04-02T07:49:08.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.358+0000 7f73c9a00700 1 -- start start 2024-04-02T07:49:08.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.359+0000 7f73c9a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73c40ce840 0x7f73c414fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:08.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.359+0000 7f73c9a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c415a5c0 0x7f73c4150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:08.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.359+0000 7f73c9a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73c415dfe0 0x7f73c415bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:08.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.359+0000 7f73c9a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c415aa30 con 0x7f73c415a5c0 2024-04-02T07:49:08.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.359+0000 7f73c9a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c415ac80 con 0x7f73c40ce840 2024-04-02T07:49:08.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.359+0000 7f73c9a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c415af60 con 0x7f73c415dfe0 2024-04-02T07:49:08.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.359+0000 7f73c2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c415a5c0 0x7f73c4150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:08.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.360+0000 7f73c2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c415a5c0 0x7f73c4150df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36268/0 (socket says 172.21.15.89:36268) 2024-04-02T07:49:08.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.360+0000 7f73c2000700 1 -- 172.21.15.89:0/1233587716 learned_addr learned my addr 172.21.15.89:0/1233587716 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:08.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.360+0000 7f73c2a00700 1 --2- 172.21.15.89:0/1233587716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73c40ce840 0x7f73c414fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:08.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.360+0000 7f73c3400700 1 --2- 172.21.15.89:0/1233587716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73c415dfe0 0x7f73c415bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:08.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.360+0000 7f73c2000700 1 -- 172.21.15.89:0/1233587716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73c415dfe0 msgr2=0x7f73c415bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:08.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.360+0000 7f73c2000700 1 --2- 172.21.15.89:0/1233587716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73c415dfe0 0x7f73c415bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:08.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.360+0000 7f73c2000700 1 -- 172.21.15.89:0/1233587716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73c40ce840 msgr2=0x7f73c414fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:08.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.360+0000 7f73c2000700 1 --2- 172.21.15.89:0/1233587716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73c40ce840 0x7f73c414fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:08.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.360+0000 7f73c2000700 1 -- 172.21.15.89:0/1233587716 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b40094a0 con 0x7f73c415a5c0 2024-04-02T07:49:08.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.360+0000 7f73c2000700 1 --2- 172.21.15.89:0/1233587716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c415a5c0 0x7f73c4150df0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f73b000c500 tx=0x7f73b000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:08.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.360+0000 7f73baa00700 1 -- 172.21.15.89:0/1233587716 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b00052d0 con 0x7f73c415a5c0 2024-04-02T07:49:08.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.360+0000 7f73baa00700 1 -- 172.21.15.89:0/1233587716 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b0005450 con 0x7f73c415a5c0 2024-04-02T07:49:08.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.360+0000 7f73baa00700 1 -- 172.21.15.89:0/1233587716 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b00038c0 con 0x7f73c415a5c0 2024-04-02T07:49:08.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.360+0000 7f73c9a00700 1 -- 172.21.15.89:0/1233587716 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c415b240 con 0x7f73c415a5c0 2024-04-02T07:49:08.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.361+0000 7f73c9a00700 1 -- 172.21.15.89:0/1233587716 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c415b800 con 0x7f73c415a5c0 2024-04-02T07:49:08.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.362+0000 7f73c9a00700 1 -- 172.21.15.89:0/1233587716 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c4067720 con 0x7f73c415a5c0 2024-04-02T07:49:08.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.362+0000 7f73baa00700 1 -- 172.21.15.89:0/1233587716 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f73b0003a20 con 0x7f73c415a5c0 2024-04-02T07:49:08.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.364+0000 7f73baa00700 1 --2- 172.21.15.89:0/1233587716 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f73a406adb0 0x7f73a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:08.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.364+0000 7f73baa00700 1 -- 172.21.15.89:0/1233587716 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f73b0088be0 con 0x7f73c415a5c0 2024-04-02T07:49:08.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.364+0000 7f73c2a00700 1 --2- 172.21.15.89:0/1233587716 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f73a406adb0 0x7f73a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:08.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.365+0000 7f73c2a00700 1 --2- 172.21.15.89:0/1233587716 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f73a406adb0 0x7f73a406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f73b40090b0 tx=0x7f73b4008030 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:08.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.368+0000 7f73baa00700 1 -- 172.21.15.89:0/1233587716 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f73b0053a30 con 0x7f73c415a5c0 2024-04-02T07:49:08.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:08.611+0000 7f73c9a00700 1 -- 172.21.15.89:0/1233587716 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f73c40b1ad0 con 0x7f73c415a5c0 2024-04-02T07:49:09.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.255+0000 7f73baa00700 1 -- 172.21.15.89:0/1233587716 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v23) v1 ==== 118+0+0 (secure 0 0 0) 0x7f73b0057050 con 0x7f73c415a5c0 2024-04-02T07:49:09.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.257+0000 7f73c9a00700 1 -- 172.21.15.89:0/1233587716 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f73a406adb0 msgr2=0x7f73a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:09.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.257+0000 7f73c9a00700 1 --2- 172.21.15.89:0/1233587716 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f73a406adb0 0x7f73a406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f73b40090b0 tx=0x7f73b4008030 comp rx=0 tx=0).stop 2024-04-02T07:49:09.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.257+0000 7f73c9a00700 1 -- 172.21.15.89:0/1233587716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c415a5c0 msgr2=0x7f73c4150df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:09.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.257+0000 7f73c9a00700 1 --2- 172.21.15.89:0/1233587716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c415a5c0 0x7f73c4150df0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f73b000c500 tx=0x7f73b000c8c0 comp rx=0 tx=0).stop 2024-04-02T07:49:09.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.258+0000 7f73c9a00700 1 -- 172.21.15.89:0/1233587716 shutdown_connections 2024-04-02T07:49:09.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.258+0000 7f73c9a00700 1 --2- 172.21.15.89:0/1233587716 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f73a406adb0 0x7f73a406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:09.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.258+0000 7f73c9a00700 1 --2- 172.21.15.89:0/1233587716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73c40ce840 0x7f73c414fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:09.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.258+0000 7f73c9a00700 1 --2- 172.21.15.89:0/1233587716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73c415dfe0 0x7f73c415bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:09.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.258+0000 7f73c9a00700 1 --2- 172.21.15.89:0/1233587716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c415a5c0 0x7f73c4150df0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:09.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.258+0000 7f73c9a00700 1 -- 172.21.15.89:0/1233587716 >> 172.21.15.89:0/1233587716 conn(0x7f73c40b0d70 msgr2=0x7f73c406cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:09.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.258+0000 7f73c9a00700 1 -- 172.21.15.89:0/1233587716 shutdown_connections 2024-04-02T07:49:09.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.258+0000 7f73c9a00700 1 -- 172.21.15.89:0/1233587716 wait complete. 2024-04-02T07:49:09.280 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T07:49:09.280 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T07:49:09.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.471+0000 7f5ca9000700 1 Processor -- start 2024-04-02T07:49:09.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.471+0000 7f5ca9000700 1 -- start start 2024-04-02T07:49:09.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.472+0000 7f5ca9000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ca415a5c0 0x7f5ca40709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:09.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.472+0000 7f5ca9000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca40cb310 con 0x7f5ca415dfe0 2024-04-02T07:49:09.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.472+0000 7f5ca9000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca4070f00 con 0x7f5ca40ce840 2024-04-02T07:49:09.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.472+0000 7f5ca9000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40711a0 con 0x7f5ca415a5c0 2024-04-02T07:49:09.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.473+0000 7f5ca1400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5ca415dfe0 0x7f5ca40ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51460/0 (socket says 172.21.15.89:51460) 2024-04-02T07:49:09.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.473+0000 7f5ca1400700 1 -- 172.21.15.89:0/2927540129 learned_addr learned my addr 172.21.15.89:0/2927540129 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:09.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.473+0000 7f5ca1e00700 1 --2- 172.21.15.89:0/2927540129 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ca415a5c0 0x7f5ca40709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:09.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.473+0000 7f5ca0a00700 1 -- 172.21.15.89:0/2927540129 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706126615 0 0) 0x7f5ca40cb310 con 0x7f5ca415dfe0 2024-04-02T07:49:09.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.473+0000 7f5ca0a00700 1 -- 172.21.15.89:0/2927540129 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c880035b0 con 0x7f5ca415dfe0 2024-04-02T07:49:09.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.473+0000 7f5ca0a00700 1 -- 172.21.15.89:0/2927540129 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921113129 0 0) 0x7f5ca4070f00 con 0x7f5ca40ce840 2024-04-02T07:49:09.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.473+0000 7f5ca1e00700 1 -- 172.21.15.89:0/2927540129 >> v1:172.21.15.113:6789/0 conn(0x7f5ca40ce840 legacy=0x7f5ca40cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:09.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.473+0000 7f5ca1e00700 1 -- 172.21.15.89:0/2927540129 >> v1:172.21.15.89:6789/0 conn(0x7f5ca415dfe0 legacy=0x7f5ca40ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:09.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.473+0000 7f5ca1e00700 1 -- 172.21.15.89:0/2927540129 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca4071480 con 0x7f5ca415a5c0 2024-04-02T07:49:09.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.474+0000 7f5ca1e00700 1 --2- 172.21.15.89:0/2927540129 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ca415a5c0 0x7f5ca40709c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5c94009870 tx=0x7f5c94009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37141b3e35c1975d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:09.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.474+0000 7f5ca0a00700 1 -- 172.21.15.89:0/2927540129 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c94011830 con 0x7f5ca415a5c0 2024-04-02T07:49:09.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.474+0000 7f5ca0a00700 1 -- 172.21.15.89:0/2927540129 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c94011990 con 0x7f5ca415a5c0 2024-04-02T07:49:09.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.474+0000 7f5ca0a00700 1 -- 172.21.15.89:0/2927540129 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c94011d20 con 0x7f5ca415a5c0 2024-04-02T07:49:09.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.474+0000 7f5ca9000700 1 -- 172.21.15.89:0/2927540129 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ca415a5c0 msgr2=0x7f5ca40709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:09.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.474+0000 7f5ca9000700 1 --2- 172.21.15.89:0/2927540129 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ca415a5c0 0x7f5ca40709c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5c94009870 tx=0x7f5c94009b80 comp rx=0 tx=0).stop 2024-04-02T07:49:09.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.474+0000 7f5ca9000700 1 -- 172.21.15.89:0/2927540129 shutdown_connections 2024-04-02T07:49:09.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.474+0000 7f5ca9000700 1 --2- 172.21.15.89:0/2927540129 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ca415a5c0 0x7f5ca40709c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:09.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.474+0000 7f5ca9000700 1 -- 172.21.15.89:0/2927540129 >> 172.21.15.89:0/2927540129 conn(0x7f5ca40b0d70 msgr2=0x7f5ca40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:09.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.474+0000 7f5ca9000700 1 -- 172.21.15.89:0/2927540129 shutdown_connections 2024-04-02T07:49:09.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.474+0000 7f5ca9000700 1 -- 172.21.15.89:0/2927540129 wait complete. 2024-04-02T07:49:09.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.477+0000 7f5ca9000700 1 Processor -- start 2024-04-02T07:49:09.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.477+0000 7f5ca9000700 1 -- start start 2024-04-02T07:49:09.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.478+0000 7f5ca9000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca40ce840 0x7f5ca41e9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:09.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.478+0000 7f5ca9000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ca415a5c0 0x7f5ca40cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:09.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.478+0000 7f5ca9000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ca415dfe0 0x7f5ca40cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:09.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.478+0000 7f5ca9000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca415aa00 con 0x7f5ca415dfe0 2024-04-02T07:49:09.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.478+0000 7f5ca9000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca415ace0 con 0x7f5ca40ce840 2024-04-02T07:49:09.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.478+0000 7f5ca9000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca415afc0 con 0x7f5ca415a5c0 2024-04-02T07:49:09.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.478+0000 7f5ca2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ca415dfe0 0x7f5ca40cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:09.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.478+0000 7f5ca2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ca415dfe0 0x7f5ca40cc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53360/0 (socket says 172.21.15.89:53360) 2024-04-02T07:49:09.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.478+0000 7f5ca2800700 1 -- 172.21.15.89:0/2410973209 learned_addr learned my addr 172.21.15.89:0/2410973209 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:09.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.478+0000 7f5ca1400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ca415a5c0 0x7f5ca40cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:09.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.479+0000 7f5ca1e00700 1 --2- 172.21.15.89:0/2410973209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca40ce840 0x7f5ca41e9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:09.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.479+0000 7f5ca2800700 1 -- 172.21.15.89:0/2410973209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ca415a5c0 msgr2=0x7f5ca40cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:09.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.479+0000 7f5ca2800700 1 --2- 172.21.15.89:0/2410973209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ca415a5c0 0x7f5ca40cbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:09.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.479+0000 7f5ca2800700 1 -- 172.21.15.89:0/2410973209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca40ce840 msgr2=0x7f5ca41e9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:09.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.479+0000 7f5ca2800700 1 --2- 172.21.15.89:0/2410973209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca40ce840 0x7f5ca41e9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:09.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.479+0000 7f5ca2800700 1 -- 172.21.15.89:0/2410973209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c940094a0 con 0x7f5ca415dfe0 2024-04-02T07:49:09.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.479+0000 7f5ca2800700 1 --2- 172.21.15.89:0/2410973209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ca415dfe0 0x7f5ca40cc2b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c001560 tx=0x7f5c9c00cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:09.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.479+0000 7f5c9a000700 1 -- 172.21.15.89:0/2410973209 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c003e30 con 0x7f5ca415dfe0 2024-04-02T07:49:09.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.479+0000 7f5c9a000700 1 -- 172.21.15.89:0/2410973209 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9c005070 con 0x7f5ca415dfe0 2024-04-02T07:49:09.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.479+0000 7f5c9a000700 1 -- 172.21.15.89:0/2410973209 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c005420 con 0x7f5ca415dfe0 2024-04-02T07:49:09.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.479+0000 7f5ca9000700 1 -- 172.21.15.89:0/2410973209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca415b2a0 con 0x7f5ca415dfe0 2024-04-02T07:49:09.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.479+0000 7f5ca9000700 1 -- 172.21.15.89:0/2410973209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca415b860 con 0x7f5ca415dfe0 2024-04-02T07:49:09.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.480+0000 7f5ca9000700 1 -- 172.21.15.89:0/2410973209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca4067720 con 0x7f5ca415dfe0 2024-04-02T07:49:09.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.481+0000 7f5c9a000700 1 -- 172.21.15.89:0/2410973209 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5c9c005580 con 0x7f5ca415dfe0 2024-04-02T07:49:09.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.482+0000 7f5c9a000700 1 --2- 172.21.15.89:0/2410973209 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5c8406adb0 0x7f5c8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:09.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.482+0000 7f5c9a000700 1 -- 172.21.15.89:0/2410973209 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5c9c088c00 con 0x7f5ca415dfe0 2024-04-02T07:49:09.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.483+0000 7f5ca1e00700 1 --2- 172.21.15.89:0/2410973209 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5c8406adb0 0x7f5c8406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:09.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.483+0000 7f5ca1e00700 1 --2- 172.21.15.89:0/2410973209 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5c8406adb0 0x7f5c8406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5c940097c0 tx=0x7f5c9401c040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:09.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.487+0000 7f5c9a000700 1 -- 172.21.15.89:0/2410973209 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c9c08f080 con 0x7f5ca415dfe0 2024-04-02T07:49:09.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:09.735+0000 7f5ca9000700 1 -- 172.21.15.89:0/2410973209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5ca40ae530 con 0x7f5ca415dfe0 2024-04-02T07:49:10.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.256+0000 7f5c9a000700 1 -- 172.21.15.89:0/2410973209 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5c9c0170c0 con 0x7f5ca415dfe0 2024-04-02T07:49:10.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.259+0000 7f5ca9000700 1 -- 172.21.15.89:0/2410973209 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5c8406adb0 msgr2=0x7f5c8406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:10.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.259+0000 7f5ca9000700 1 --2- 172.21.15.89:0/2410973209 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5c8406adb0 0x7f5c8406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5c940097c0 tx=0x7f5c9401c040 comp rx=0 tx=0).stop 2024-04-02T07:49:10.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.259+0000 7f5ca9000700 1 -- 172.21.15.89:0/2410973209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ca415dfe0 msgr2=0x7f5ca40cc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:10.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.259+0000 7f5ca9000700 1 --2- 172.21.15.89:0/2410973209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ca415dfe0 0x7f5ca40cc2b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c001560 tx=0x7f5c9c00cb50 comp rx=0 tx=0).stop 2024-04-02T07:49:10.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.259+0000 7f5ca9000700 1 -- 172.21.15.89:0/2410973209 shutdown_connections 2024-04-02T07:49:10.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.259+0000 7f5ca9000700 1 --2- 172.21.15.89:0/2410973209 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5c8406adb0 0x7f5c8406d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.259+0000 7f5ca9000700 1 --2- 172.21.15.89:0/2410973209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca40ce840 0x7f5ca41e9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.259+0000 7f5ca9000700 1 --2- 172.21.15.89:0/2410973209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ca415dfe0 0x7f5ca40cc2b0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.259+0000 7f5ca9000700 1 --2- 172.21.15.89:0/2410973209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ca415a5c0 0x7f5ca40cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.259+0000 7f5ca9000700 1 -- 172.21.15.89:0/2410973209 >> 172.21.15.89:0/2410973209 conn(0x7f5ca40b0d70 msgr2=0x7f5ca40cd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:10.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.259+0000 7f5ca9000700 1 -- 172.21.15.89:0/2410973209 shutdown_connections 2024-04-02T07:49:10.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.259+0000 7f5ca9000700 1 -- 172.21.15.89:0/2410973209 wait complete. 2024-04-02T07:49:10.282 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T07:49:10.282 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:49:10.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.470+0000 7f6183e00700 1 Processor -- start 2024-04-02T07:49:10.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.470+0000 7f6183e00700 1 -- start start 2024-04-02T07:49:10.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.471+0000 7f6183e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f617c15d640 0x7f617c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:10.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6183e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f617c0ceec0 0x7f617c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:10.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6183e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f617c161800 0x7f617c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:10.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6183e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f617c161c90 con 0x7f617c15d640 2024-04-02T07:49:10.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6183e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f617c0d30d0 con 0x7f617c161800 2024-04-02T07:49:10.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6183e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617c0b6840 con 0x7f617c0ceec0 2024-04-02T07:49:10.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6181400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f617c15d640 0x7f617c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:10.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6181400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f617c15d640 0x7f617c0d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53362/0 (socket says 172.21.15.89:53362) 2024-04-02T07:49:10.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6181400700 1 -- 172.21.15.89:0/3952045921 learned_addr learned my addr 172.21.15.89:0/3952045921 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:10.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6180a00700 1 --2- 172.21.15.89:0/3952045921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f617c0ceec0 0x7f617c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:10.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6181e00700 1 --2- 172.21.15.89:0/3952045921 >> v2:172.21.15.113:3300/0 conn(0x7f617c161800 0x7f617c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:10.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6181400700 1 -- 172.21.15.89:0/3952045921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f617c0ceec0 msgr2=0x7f617c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:10.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6181400700 1 --2- 172.21.15.89:0/3952045921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f617c0ceec0 0x7f617c1612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6181400700 1 -- 172.21.15.89:0/3952045921 >> v2:172.21.15.113:3300/0 conn(0x7f617c161800 msgr2=0x7f617c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:10.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6181400700 1 --2- 172.21.15.89:0/3952045921 >> v2:172.21.15.113:3300/0 conn(0x7f617c161800 0x7f617c0cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.472+0000 7f6181400700 1 -- 172.21.15.89:0/3952045921 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c0d2880 con 0x7f617c15d640 2024-04-02T07:49:10.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.473+0000 7f6181400700 1 --2- 172.21.15.89:0/3952045921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f617c15d640 0x7f617c0d2340 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f617c0c7050 tx=0x7f6170009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3598d500b48fb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:10.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.473+0000 7f617be00700 1 -- 172.21.15.89:0/3952045921 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617000b870 con 0x7f617c15d640 2024-04-02T07:49:10.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.473+0000 7f617be00700 1 -- 172.21.15.89:0/3952045921 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f617000b9d0 con 0x7f617c15d640 2024-04-02T07:49:10.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.473+0000 7f617be00700 1 -- 172.21.15.89:0/3952045921 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617000bd60 con 0x7f617c15d640 2024-04-02T07:49:10.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.473+0000 7f6183e00700 1 -- 172.21.15.89:0/3952045921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f617c15d640 msgr2=0x7f617c0d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:10.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.473+0000 7f6183e00700 1 --2- 172.21.15.89:0/3952045921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f617c15d640 0x7f617c0d2340 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f617c0c7050 tx=0x7f6170009bd0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.473+0000 7f6183e00700 1 -- 172.21.15.89:0/3952045921 shutdown_connections 2024-04-02T07:49:10.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.473+0000 7f6183e00700 1 --2- 172.21.15.89:0/3952045921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f617c15d640 0x7f617c0d2340 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.473+0000 7f6183e00700 1 --2- 172.21.15.89:0/3952045921 >> v2:172.21.15.113:3300/0 conn(0x7f617c161800 0x7f617c0cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.473+0000 7f6183e00700 1 --2- 172.21.15.89:0/3952045921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f617c0ceec0 0x7f617c1612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.473+0000 7f6183e00700 1 -- 172.21.15.89:0/3952045921 >> 172.21.15.89:0/3952045921 conn(0x7f617c08aff0 msgr2=0x7f617c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:10.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.473+0000 7f6183e00700 1 -- 172.21.15.89:0/3952045921 shutdown_connections 2024-04-02T07:49:10.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.474+0000 7f6183e00700 1 -- 172.21.15.89:0/3952045921 wait complete. 2024-04-02T07:49:10.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.475+0000 7f6183e00700 1 Processor -- start 2024-04-02T07:49:10.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.475+0000 7f6183e00700 1 -- start start 2024-04-02T07:49:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.476+0000 7f6183e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f617c0ceec0 0x7f617c1552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.476+0000 7f6183e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f617c15d640 0x7f617c1f63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.476+0000 7f6183e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f617c161800 0x7f617c1f6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.476+0000 7f6183e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c146f00 con 0x7f617c15d640 2024-04-02T07:49:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.476+0000 7f6183e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c153cb0 con 0x7f617c0ceec0 2024-04-02T07:49:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.476+0000 7f6183e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617c1f3810 con 0x7f617c161800 2024-04-02T07:49:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.476+0000 7f6180a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f617c15d640 0x7f617c1f63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.476+0000 7f6180a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f617c15d640 0x7f617c1f63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53376/0 (socket says 172.21.15.89:53376) 2024-04-02T07:49:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.476+0000 7f6180a00700 1 -- 172.21.15.89:0/381252609 learned_addr learned my addr 172.21.15.89:0/381252609 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.476+0000 7f6181400700 1 --2- 172.21.15.89:0/381252609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f617c0ceec0 0x7f617c1552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.477+0000 7f6181e00700 1 --2- 172.21.15.89:0/381252609 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f617c161800 0x7f617c1f6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.477+0000 7f6180a00700 1 -- 172.21.15.89:0/381252609 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f617c161800 msgr2=0x7f617c1f6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.477+0000 7f6180a00700 1 --2- 172.21.15.89:0/381252609 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f617c161800 0x7f617c1f6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.477+0000 7f6180a00700 1 -- 172.21.15.89:0/381252609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f617c0ceec0 msgr2=0x7f617c1552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.477+0000 7f6180a00700 1 --2- 172.21.15.89:0/381252609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f617c0ceec0 0x7f617c1552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.477+0000 7f6180a00700 1 -- 172.21.15.89:0/381252609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6170009550 con 0x7f617c15d640 2024-04-02T07:49:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.477+0000 7f6180a00700 1 --2- 172.21.15.89:0/381252609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f617c15d640 0x7f617c1f63e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f616800d800 tx=0x7f616800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.477+0000 7f6179600700 1 -- 172.21.15.89:0/381252609 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61680125f0 con 0x7f617c15d640 2024-04-02T07:49:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.477+0000 7f6179600700 1 -- 172.21.15.89:0/381252609 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6168012750 con 0x7f617c15d640 2024-04-02T07:49:10.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.477+0000 7f6179600700 1 -- 172.21.15.89:0/381252609 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168012b00 con 0x7f617c15d640 2024-04-02T07:49:10.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.477+0000 7f6183e00700 1 -- 172.21.15.89:0/381252609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617c1f3af0 con 0x7f617c15d640 2024-04-02T07:49:10.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.477+0000 7f6183e00700 1 -- 172.21.15.89:0/381252609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617c15ef90 con 0x7f617c15d640 2024-04-02T07:49:10.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.478+0000 7f6183e00700 1 -- 172.21.15.89:0/381252609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617c1f4010 con 0x7f617c15d640 2024-04-02T07:49:10.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.479+0000 7f6179600700 1 -- 172.21.15.89:0/381252609 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6168012c60 con 0x7f617c15d640 2024-04-02T07:49:10.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.480+0000 7f6179600700 1 --2- 172.21.15.89:0/381252609 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f615c06ade0 0x7f615c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:10.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.480+0000 7f6179600700 1 -- 172.21.15.89:0/381252609 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6168089520 con 0x7f617c15d640 2024-04-02T07:49:10.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.480+0000 7f6181400700 1 --2- 172.21.15.89:0/381252609 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f615c06ade0 0x7f615c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:10.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.481+0000 7f6181400700 1 --2- 172.21.15.89:0/381252609 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f615c06ade0 0x7f615c06d2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6170000c00 tx=0x7f6170008030 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:10.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.485+0000 7f6179600700 1 -- 172.21.15.89:0/381252609 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f617c1f4010 con 0x7f617c15d640 2024-04-02T07:49:10.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.727+0000 7f6183e00700 1 -- 172.21.15.89:0/381252609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f617c15edf0 con 0x7f617c15d640 2024-04-02T07:49:10.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.727+0000 7f6179600700 1 -- 172.21.15.89:0/381252609 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f617c15edf0 con 0x7f617c15d640 2024-04-02T07:49:10.731 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:49:10.731 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5710,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3098065397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3098065397},{"type":"v1","addr":"172.21.15.89:6836","nonce":3098065397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5713,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1408411829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1408411829},{"type":"v1","addr":"172.21.15.89:6837","nonce":1408411829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5716,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3266955664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3266955664},{"type":"v1","addr":"172.21.15.113:6834","nonce":3266955664}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:49:07.249043+0000","modified":"2024-04-02T07:49:10.254959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5719":{"gid":5719,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6835/427074923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":427074923},{"type":"v1","addr":"172.21.15.113:6835","nonce":427074923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-02T07:49:10.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.730+0000 7f6183e00700 1 -- 172.21.15.89:0/381252609 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f615c06ade0 msgr2=0x7f615c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:10.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.730+0000 7f6183e00700 1 --2- 172.21.15.89:0/381252609 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f615c06ade0 0x7f615c06d2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6170000c00 tx=0x7f6170008030 comp rx=0 tx=0).stop 2024-04-02T07:49:10.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.730+0000 7f6183e00700 1 -- 172.21.15.89:0/381252609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f617c15d640 msgr2=0x7f617c1f63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:10.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.730+0000 7f6183e00700 1 --2- 172.21.15.89:0/381252609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f617c15d640 0x7f617c1f63e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f616800d800 tx=0x7f616800db10 comp rx=0 tx=0).stop 2024-04-02T07:49:10.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.730+0000 7f6183e00700 1 -- 172.21.15.89:0/381252609 shutdown_connections 2024-04-02T07:49:10.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.730+0000 7f6183e00700 1 --2- 172.21.15.89:0/381252609 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f615c06ade0 0x7f615c06d2a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.730+0000 7f6183e00700 1 --2- 172.21.15.89:0/381252609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f617c0ceec0 0x7f617c1552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.730+0000 7f6183e00700 1 --2- 172.21.15.89:0/381252609 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f617c161800 0x7f617c1f6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.730+0000 7f6183e00700 1 --2- 172.21.15.89:0/381252609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f617c15d640 0x7f617c1f63e0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.730+0000 7f6183e00700 1 -- 172.21.15.89:0/381252609 >> 172.21.15.89:0/381252609 conn(0x7f617c08aff0 msgr2=0x7f617c0b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:10.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.730+0000 7f6183e00700 1 -- 172.21.15.89:0/381252609 shutdown_connections 2024-04-02T07:49:10.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.730+0000 7f6183e00700 1 -- 172.21.15.89:0/381252609 wait complete. 2024-04-02T07:49:10.735 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 24 2024-04-02T07:49:10.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:49:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.940+0000 7f99b8400700 1 Processor -- start 2024-04-02T07:49:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.941+0000 7f99b8400700 1 -- start start 2024-04-02T07:49:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.942+0000 7f99b8400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f99b0160f10 0x7f99b00d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.942+0000 7f99b8400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f99b00cf230 0x7f99b00d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.942+0000 7f99b8400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b00ceda0 con 0x7f99b0161590 2024-04-02T07:49:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.942+0000 7f99b8400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b00d23d0 con 0x7f99b00cf230 2024-04-02T07:49:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.942+0000 7f99b8400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b01612f0 con 0x7f99b0160f10 2024-04-02T07:49:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.942+0000 7f99b5000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f99b00cf230 0x7f99b00d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.942+0000 7f99b5000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f99b00cf230 0x7f99b00d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53388/0 (socket says 172.21.15.89:53388) 2024-04-02T07:49:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.942+0000 7f99b5000700 1 -- 172.21.15.89:0/3017291992 learned_addr learned my addr 172.21.15.89:0/3017291992 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.942+0000 7f99b6400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f99b0161590 0x7f99b00b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51470/0 (socket says 172.21.15.89:51470) 2024-04-02T07:49:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.942+0000 7f99b5a00700 1 --2- 172.21.15.89:0/3017291992 >> v2:172.21.15.113:3300/0 conn(0x7f99b0160f10 0x7f99b00d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.943+0000 7f99b5000700 1 -- 172.21.15.89:0/3017291992 >> v2:172.21.15.113:3300/0 conn(0x7f99b0160f10 msgr2=0x7f99b00d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.943+0000 7f99b5000700 1 --2- 172.21.15.89:0/3017291992 >> v2:172.21.15.113:3300/0 conn(0x7f99b0160f10 0x7f99b00d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.943+0000 7f99b5000700 1 -- 172.21.15.89:0/3017291992 >> v1:172.21.15.89:6789/0 conn(0x7f99b0161590 legacy=0x7f99b00b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.943+0000 7f99b5000700 1 -- 172.21.15.89:0/3017291992 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b00ce470 con 0x7f99b00cf230 2024-04-02T07:49:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.943+0000 7f99b5a00700 1 --2- 172.21.15.89:0/3017291992 >> v2:172.21.15.113:3300/0 conn(0x7f99b0160f10 0x7f99b00d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:49:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.943+0000 7f99afe00700 1 -- 172.21.15.89:0/3017291992 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177012067 0 0) 0x7f99b00ceda0 con 0x7f99b0161590 2024-04-02T07:49:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.943+0000 7f99b5000700 1 --2- 172.21.15.89:0/3017291992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b00cf230 0x7f99b00d1e90 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f99b00d1080 tx=0x7f999c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e94a276a625228 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.943+0000 7f99afe00700 1 -- 172.21.15.89:0/3017291992 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c00b870 con 0x7f99b00cf230 2024-04-02T07:49:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.943+0000 7f99afe00700 1 -- 172.21.15.89:0/3017291992 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999c00b9d0 con 0x7f99b00cf230 2024-04-02T07:49:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.943+0000 7f99afe00700 1 -- 172.21.15.89:0/3017291992 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c00bd60 con 0x7f99b00cf230 2024-04-02T07:49:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.944+0000 7f99b8400700 1 -- 172.21.15.89:0/3017291992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b00cf230 msgr2=0x7f99b00d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.944+0000 7f99b8400700 1 --2- 172.21.15.89:0/3017291992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b00cf230 0x7f99b00d1e90 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f99b00d1080 tx=0x7f999c009bd0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.944+0000 7f99b8400700 1 -- 172.21.15.89:0/3017291992 shutdown_connections 2024-04-02T07:49:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.944+0000 7f99b8400700 1 --2- 172.21.15.89:0/3017291992 >> v2:172.21.15.113:3300/0 conn(0x7f99b0160f10 0x7f99b00d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.944+0000 7f99b8400700 1 --2- 172.21.15.89:0/3017291992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b00cf230 0x7f99b00d1e90 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.944+0000 7f99b8400700 1 -- 172.21.15.89:0/3017291992 >> 172.21.15.89:0/3017291992 conn(0x7f99b008aff0 msgr2=0x7f99b00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.944+0000 7f99b8400700 1 -- 172.21.15.89:0/3017291992 shutdown_connections 2024-04-02T07:49:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.944+0000 7f99b8400700 1 -- 172.21.15.89:0/3017291992 wait complete. 2024-04-02T07:49:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.945+0000 7f99b8400700 1 Processor -- start 2024-04-02T07:49:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.946+0000 7f99b8400700 1 -- start start 2024-04-02T07:49:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.946+0000 7f99b8400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99b00cf230 0x7f99b01543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.946+0000 7f99b8400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99b0160f10 0x7f99b01e7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.946+0000 7f99b8400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b0161590 0x7f99b01522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.946+0000 7f99b8400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b01f32f0 con 0x7f99b0161590 2024-04-02T07:49:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.946+0000 7f99b8400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b0154920 con 0x7f99b00cf230 2024-04-02T07:49:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.946+0000 7f99b8400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b01533d0 con 0x7f99b0160f10 2024-04-02T07:49:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.947+0000 7f99b6400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b0161590 0x7f99b01522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.947+0000 7f99b6400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b0161590 0x7f99b01522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53402/0 (socket says 172.21.15.89:53402) 2024-04-02T07:49:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.947+0000 7f99b6400700 1 -- 172.21.15.89:0/511147687 learned_addr learned my addr 172.21.15.89:0/511147687 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.947+0000 7f99b6400700 1 -- 172.21.15.89:0/511147687 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99b0160f10 msgr2=0x7f99b01e7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.947+0000 7f99b6400700 1 --2- 172.21.15.89:0/511147687 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99b0160f10 0x7f99b01e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.947+0000 7f99b6400700 1 -- 172.21.15.89:0/511147687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99b00cf230 msgr2=0x7f99b01543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.947+0000 7f99b6400700 1 --2- 172.21.15.89:0/511147687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99b00cf230 0x7f99b01543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.947+0000 7f99b6400700 1 -- 172.21.15.89:0/511147687 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c009550 con 0x7f99b0161590 2024-04-02T07:49:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.947+0000 7f99b6400700 1 --2- 172.21.15.89:0/511147687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b0161590 0x7f99b01522b0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f99a800cba0 tx=0x7f99a800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.947+0000 7f99ad600700 1 -- 172.21.15.89:0/511147687 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a800ab80 con 0x7f99b0161590 2024-04-02T07:49:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.947+0000 7f99ad600700 1 -- 172.21.15.89:0/511147687 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a800ad00 con 0x7f99b0161590 2024-04-02T07:49:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.947+0000 7f99ad600700 1 -- 172.21.15.89:0/511147687 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8007310 con 0x7f99b0161590 2024-04-02T07:49:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.948+0000 7f99b8400700 1 -- 172.21.15.89:0/511147687 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b0153620 con 0x7f99b0161590 2024-04-02T07:49:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.948+0000 7f99b8400700 1 -- 172.21.15.89:0/511147687 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b0153ec0 con 0x7f99b0161590 2024-04-02T07:49:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.949+0000 7f99b8400700 1 -- 172.21.15.89:0/511147687 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b0153a40 con 0x7f99b0161590 2024-04-02T07:49:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.950+0000 7f99ad600700 1 -- 172.21.15.89:0/511147687 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f99a8007470 con 0x7f99b0161590 2024-04-02T07:49:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.950+0000 7f99ad600700 1 --2- 172.21.15.89:0/511147687 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f999006adb0 0x7f999006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.950+0000 7f99ad600700 1 -- 172.21.15.89:0/511147687 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f99a80890e0 con 0x7f99b0161590 2024-04-02T07:49:10.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.951+0000 7f99b5a00700 1 --2- 172.21.15.89:0/511147687 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f999006adb0 0x7f999006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:10.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.951+0000 7f99b5a00700 1 --2- 172.21.15.89:0/511147687 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f999006adb0 0x7f999006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f99a40098f0 tx=0x7f99a4008040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:10.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:10.956+0000 7f99ad600700 1 -- 172.21.15.89:0/511147687 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f99b0153a40 con 0x7f99b0161590 2024-04-02T07:49:11.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.160+0000 7f99b8400700 1 -- 172.21.15.89:0/511147687 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99b0153a40 con 0x7f99b0161590 2024-04-02T07:49:11.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.161+0000 7f99ad600700 1 -- 172.21.15.89:0/511147687 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16568 (secure 0 0 0) 0x7f99b0153a40 con 0x7f99b0161590 2024-04-02T07:49:11.164 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:49:11.164 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":66,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:49:08.251202+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:49:05.141462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-02T07:49:06.232906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:49:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.164+0000 7f99b8400700 1 -- 172.21.15.89:0/511147687 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f999006adb0 msgr2=0x7f999006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.164+0000 7f99b8400700 1 --2- 172.21.15.89:0/511147687 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f999006adb0 0x7f999006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f99a40098f0 tx=0x7f99a4008040 comp rx=0 tx=0).stop 2024-04-02T07:49:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.164+0000 7f99b8400700 1 -- 172.21.15.89:0/511147687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b0161590 msgr2=0x7f99b01522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.164+0000 7f99b8400700 1 --2- 172.21.15.89:0/511147687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b0161590 0x7f99b01522b0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f99a800cba0 tx=0x7f99a800cf60 comp rx=0 tx=0).stop 2024-04-02T07:49:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.164+0000 7f99b8400700 1 -- 172.21.15.89:0/511147687 shutdown_connections 2024-04-02T07:49:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.164+0000 7f99b8400700 1 --2- 172.21.15.89:0/511147687 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f999006adb0 0x7f999006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.164+0000 7f99b8400700 1 --2- 172.21.15.89:0/511147687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99b00cf230 0x7f99b01543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.164+0000 7f99b8400700 1 --2- 172.21.15.89:0/511147687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99b0161590 0x7f99b01522b0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.164+0000 7f99b8400700 1 --2- 172.21.15.89:0/511147687 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99b0160f10 0x7f99b01e7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.164+0000 7f99b8400700 1 -- 172.21.15.89:0/511147687 >> 172.21.15.89:0/511147687 conn(0x7f99b008aff0 msgr2=0x7f99b006cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.164+0000 7f99b8400700 1 -- 172.21.15.89:0/511147687 shutdown_connections 2024-04-02T07:49:11.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.165+0000 7f99b8400700 1 -- 172.21.15.89:0/511147687 wait complete. 2024-04-02T07:49:11.185 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T07:49:11.185 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:49:11.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.375+0000 7fe118600700 1 Processor -- start 2024-04-02T07:49:11.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.376+0000 7fe118600700 1 -- start start 2024-04-02T07:49:11.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.377+0000 7fe118600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe1100d18b0 0x7fe1100cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.377+0000 7fe118600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1100ceda0 con 0x7fe110160f10 2024-04-02T07:49:11.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.377+0000 7fe118600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1101612f0 con 0x7fe110161590 2024-04-02T07:49:11.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.377+0000 7fe118600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1100d1d30 con 0x7fe1100d18b0 2024-04-02T07:49:11.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.377+0000 7fe115c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe110160f10 0x7fe1100d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51474/0 (socket says 172.21.15.89:51474) 2024-04-02T07:49:11.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.377+0000 7fe115c00700 1 -- 172.21.15.89:0/2494713758 learned_addr learned my addr 172.21.15.89:0/2494713758 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:11.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.377+0000 7fe115200700 1 --2- 172.21.15.89:0/2494713758 >> v2:172.21.15.113:3300/0 conn(0x7fe1100d18b0 0x7fe1100cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.377+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441017716 0 0) 0x7fe1100ceda0 con 0x7fe110160f10 2024-04-02T07:49:11.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.377+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f80035b0 con 0x7fe110160f10 2024-04-02T07:49:11.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433627811 0 0) 0x7fe1101612f0 con 0x7fe110161590 2024-04-02T07:49:11.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1100ceda0 con 0x7fe110161590 2024-04-02T07:49:11.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 324807811 0 0) 0x7fe0f80035b0 con 0x7fe110160f10 2024-04-02T07:49:11.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 >> v2:172.21.15.113:3300/0 conn(0x7fe1100d18b0 msgr2=0x7fe1100cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:11.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 --2- 172.21.15.89:0/2494713758 >> v2:172.21.15.113:3300/0 conn(0x7fe1100d18b0 0x7fe1100cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 >> v1:172.21.15.113:6789/0 conn(0x7fe110161590 legacy=0x7fe1100b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:11.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1100ce470 con 0x7fe110160f10 2024-04-02T07:49:11.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fe104001670 con 0x7fe110160f10 2024-04-02T07:49:11.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 >> v1:172.21.15.89:6789/0 conn(0x7fe110160f10 legacy=0x7fe1100d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:11.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 --2- 172.21.15.89:0/2494713758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0f8003be0 0x7fe0f8003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 --2- 172.21.15.89:0/2494713758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0f8004850 0x7fe0f8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 --2- 172.21.15.89:0/2494713758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0f8005260 0x7fe0f800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1100d1d30 con 0x7fe0f8004850 2024-04-02T07:49:11.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f800c190 con 0x7fe0f8005260 2024-04-02T07:49:11.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f800c350 con 0x7fe0f8003be0 2024-04-02T07:49:11.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe115200700 1 --2- 172.21.15.89:0/2494713758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0f8004850 0x7fe0f8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe115c00700 1 --2- 172.21.15.89:0/2494713758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0f8003be0 0x7fe0f8003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.378+0000 7fe116600700 1 --2- 172.21.15.89:0/2494713758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0f8005260 0x7fe0f800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.379+0000 7fe115200700 1 -- 172.21.15.89:0/2494713758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0f8003be0 msgr2=0x7fe0f8003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.379+0000 7fe115200700 1 --2- 172.21.15.89:0/2494713758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0f8003be0 0x7fe0f8003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.379+0000 7fe115200700 1 -- 172.21.15.89:0/2494713758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0f8005260 msgr2=0x7fe0f800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.379+0000 7fe115200700 1 --2- 172.21.15.89:0/2494713758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0f8005260 0x7fe0f800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.379+0000 7fe115200700 1 -- 172.21.15.89:0/2494713758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f80035b0 con 0x7fe0f8004850 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.379+0000 7fe116600700 1 -- 172.21.15.89:0/2494713758 reap_dead start 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.379+0000 7fe115200700 1 --2- 172.21.15.89:0/2494713758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0f8004850 0x7fe0f8004cd0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009b60 tx=0x7fe0f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dbfb9a6556d9673 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.379+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc00e6c0 con 0x7fe0f8004850 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.379+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc00e820 con 0x7fe0f8004850 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.379+0000 7fe10fe00700 1 -- 172.21.15.89:0/2494713758 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc00eb90 con 0x7fe0f8004850 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.379+0000 7fe118600700 1 -- 172.21.15.89:0/2494713758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0f8004850 msgr2=0x7fe0f8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.379+0000 7fe118600700 1 --2- 172.21.15.89:0/2494713758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0f8004850 0x7fe0f8004cd0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009b60 tx=0x7fe0f8005880 comp rx=0 tx=0).stop 2024-04-02T07:49:11.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.380+0000 7fe118600700 1 -- 172.21.15.89:0/2494713758 shutdown_connections 2024-04-02T07:49:11.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.380+0000 7fe118600700 1 --2- 172.21.15.89:0/2494713758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0f8004850 0x7fe0f8004cd0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.380+0000 7fe118600700 1 -- 172.21.15.89:0/2494713758 >> 172.21.15.89:0/2494713758 conn(0x7fe11008aff0 msgr2=0x7fe1100d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:11.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.380+0000 7fe118600700 1 -- 172.21.15.89:0/2494713758 shutdown_connections 2024-04-02T07:49:11.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.380+0000 7fe118600700 1 -- 172.21.15.89:0/2494713758 wait complete. 2024-04-02T07:49:11.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.381+0000 7fe118600700 1 Processor -- start 2024-04-02T07:49:11.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.382+0000 7fe118600700 1 -- start start 2024-04-02T07:49:11.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.382+0000 7fe118600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0f8004850 0x7fe1101559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.382+0000 7fe118600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe110159460 0x7fe1101586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.382+0000 7fe118600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe110158c00 0x7fe1100b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.382+0000 7fe118600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe110157e20 con 0x7fe110158c00 2024-04-02T07:49:11.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.382+0000 7fe118600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1101590b0 con 0x7fe110159460 2024-04-02T07:49:11.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.382+0000 7fe118600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe110157200 con 0x7fe0f8004850 2024-04-02T07:49:11.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.383+0000 7fe116600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe110158c00 0x7fe1100b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.383+0000 7fe116600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe110158c00 0x7fe1100b7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53416/0 (socket says 172.21.15.89:53416) 2024-04-02T07:49:11.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.383+0000 7fe116600700 1 -- 172.21.15.89:0/782414674 learned_addr learned my addr 172.21.15.89:0/782414674 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:11.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.383+0000 7fe115c00700 1 --2- 172.21.15.89:0/782414674 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0f8004850 0x7fe1101559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.383+0000 7fe115200700 1 --2- 172.21.15.89:0/782414674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe110159460 0x7fe1101586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.383+0000 7fe116600700 1 -- 172.21.15.89:0/782414674 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0f8004850 msgr2=0x7fe1101559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:11.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.383+0000 7fe116600700 1 --2- 172.21.15.89:0/782414674 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0f8004850 0x7fe1101559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.383+0000 7fe116600700 1 -- 172.21.15.89:0/782414674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe110159460 msgr2=0x7fe1101586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.383+0000 7fe116600700 1 --2- 172.21.15.89:0/782414674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe110159460 0x7fe1101586c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.383+0000 7fe116600700 1 -- 172.21.15.89:0/782414674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe110157480 con 0x7fe110158c00 2024-04-02T07:49:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.383+0000 7fe116600700 1 --2- 172.21.15.89:0/782414674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe110158c00 0x7fe1100b7230 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe11008d550 tx=0x7fe10800c0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.383+0000 7fe10d600700 1 -- 172.21.15.89:0/782414674 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe108003c10 con 0x7fe110158c00 2024-04-02T07:49:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.384+0000 7fe10d600700 1 -- 172.21.15.89:0/782414674 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe108003d70 con 0x7fe110158c00 2024-04-02T07:49:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.384+0000 7fe10d600700 1 -- 172.21.15.89:0/782414674 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10800c9f0 con 0x7fe110158c00 2024-04-02T07:49:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.384+0000 7fe118600700 1 -- 172.21.15.89:0/782414674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe110157760 con 0x7fe110158c00 2024-04-02T07:49:11.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.384+0000 7fe118600700 1 -- 172.21.15.89:0/782414674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe110155ef0 con 0x7fe110158c00 2024-04-02T07:49:11.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.385+0000 7fe118600700 1 -- 172.21.15.89:0/782414674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe110157f60 con 0x7fe110158c00 2024-04-02T07:49:11.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.385+0000 7fe10d600700 1 -- 172.21.15.89:0/782414674 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe10800cb50 con 0x7fe110158c00 2024-04-02T07:49:11.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.386+0000 7fe10d600700 1 --2- 172.21.15.89:0/782414674 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe0f006adb0 0x7fe0f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.386+0000 7fe10d600700 1 -- 172.21.15.89:0/782414674 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe108089810 con 0x7fe110158c00 2024-04-02T07:49:11.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.387+0000 7fe115c00700 1 --2- 172.21.15.89:0/782414674 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe0f006adb0 0x7fe0f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.387+0000 7fe115c00700 1 --2- 172.21.15.89:0/782414674 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe0f006adb0 0x7fe0f006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe104001eb0 tx=0x7fe104009040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:11.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.391+0000 7fe10d600700 1 -- 172.21.15.89:0/782414674 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe110157f60 con 0x7fe110158c00 2024-04-02T07:49:11.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.589+0000 7fe118600700 1 -- 172.21.15.89:0/782414674 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe1100ce120 con 0x7fe0f006adb0 2024-04-02T07:49:11.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.599+0000 7fe10d600700 1 -- 172.21.15.89:0/782414674 <== mgr.5661 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289304 (secure 0 0 0) 0x7fe104005de0 con 0x7fe0f006adb0 2024-04-02T07:49:11.606 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:49:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.604+0000 7fe118600700 1 -- 172.21.15.89:0/782414674 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe0f006adb0 msgr2=0x7fe0f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.604+0000 7fe118600700 1 --2- 172.21.15.89:0/782414674 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe0f006adb0 0x7fe0f006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe104001eb0 tx=0x7fe104009040 comp rx=0 tx=0).stop 2024-04-02T07:49:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.604+0000 7fe118600700 1 -- 172.21.15.89:0/782414674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe110158c00 msgr2=0x7fe1100b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.604+0000 7fe118600700 1 --2- 172.21.15.89:0/782414674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe110158c00 0x7fe1100b7230 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe11008d550 tx=0x7fe10800c0e0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.604+0000 7fe118600700 1 -- 172.21.15.89:0/782414674 shutdown_connections 2024-04-02T07:49:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.604+0000 7fe118600700 1 --2- 172.21.15.89:0/782414674 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe0f006adb0 0x7fe0f006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.604+0000 7fe118600700 1 --2- 172.21.15.89:0/782414674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe110158c00 0x7fe1100b7230 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.604+0000 7fe118600700 1 --2- 172.21.15.89:0/782414674 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0f8004850 0x7fe1101559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.604+0000 7fe118600700 1 --2- 172.21.15.89:0/782414674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe110159460 0x7fe1101586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.604+0000 7fe118600700 1 -- 172.21.15.89:0/782414674 >> 172.21.15.89:0/782414674 conn(0x7fe11008aff0 msgr2=0x7fe11015bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.604+0000 7fe118600700 1 -- 172.21.15.89:0/782414674 shutdown_connections 2024-04-02T07:49:11.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.604+0000 7fe118600700 1 -- 172.21.15.89:0/782414674 wait complete. 2024-04-02T07:49:11.610 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:49:11.624 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T07:49:11.141591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268552,"kb_used_data":7312,"kb_used_omap":12,"kb_used_meta":261107,"kb_avail":754706168,"statfs":{"total":773094113280,"available":772819116032,"internally_reserved":0,"allocated":7487488,"data_stored":4134966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":267374136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.398324"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721452+0000","last_change":"2024-04-02T07:49:07.276132+0000","last_active":"2024-04-02T07:49:08.721452+0000","last_peered":"2024-04-02T07:49:08.721452+0000","last_clean":"2024-04-02T07:49:08.721452+0000","last_became_active":"2024-04-02T07:49:07.275562+0000","last_became_peered":"2024-04-02T07:49:07.275562+0000","last_unstale":"2024-04-02T07:49:08.721452+0000","last_undegraded":"2024-04-02T07:49:08.721452+0000","last_fullsized":"2024-04-02T07:49:08.721452+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:48.734491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255598+0000","last_change":"2024-04-02T07:49:07.271520+0000","last_active":"2024-04-02T07:49:08.255598+0000","last_peered":"2024-04-02T07:49:08.255598+0000","last_clean":"2024-04-02T07:49:08.255598+0000","last_became_active":"2024-04-02T07:49:07.271249+0000","last_became_peered":"2024-04-02T07:49:07.271249+0000","last_unstale":"2024-04-02T07:49:08.255598+0000","last_undegraded":"2024-04-02T07:49:08.255598+0000","last_fullsized":"2024-04-02T07:49:08.255598+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:39:26.559000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720960+0000","last_change":"2024-04-02T07:49:07.283831+0000","last_active":"2024-04-02T07:49:08.720960+0000","last_peered":"2024-04-02T07:49:08.720960+0000","last_clean":"2024-04-02T07:49:08.720960+0000","last_became_active":"2024-04-02T07:49:07.282360+0000","last_became_peered":"2024-04-02T07:49:07.282360+0000","last_unstale":"2024-04-02T07:49:08.720960+0000","last_undegraded":"2024-04-02T07:49:08.720960+0000","last_fullsized":"2024-04-02T07:49:08.720960+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:33.794321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721558+0000","last_change":"2024-04-02T07:49:07.292520+0000","last_active":"2024-04-02T07:49:08.721558+0000","last_peered":"2024-04-02T07:49:08.721558+0000","last_clean":"2024-04-02T07:49:08.721558+0000","last_became_active":"2024-04-02T07:49:07.292201+0000","last_became_peered":"2024-04-02T07:49:07.292201+0000","last_unstale":"2024-04-02T07:49:08.721558+0000","last_undegraded":"2024-04-02T07:49:08.721558+0000","last_fullsized":"2024-04-02T07:49:08.721558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:21:40.105740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721518+0000","last_change":"2024-04-02T07:49:07.278990+0000","last_active":"2024-04-02T07:49:08.721518+0000","last_peered":"2024-04-02T07:49:08.721518+0000","last_clean":"2024-04-02T07:49:08.721518+0000","last_became_active":"2024-04-02T07:49:07.278696+0000","last_became_peered":"2024-04-02T07:49:07.278696+0000","last_unstale":"2024-04-02T07:49:08.721518+0000","last_undegraded":"2024-04-02T07:49:08.721518+0000","last_fullsized":"2024-04-02T07:49:08.721518+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:26.176308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720556+0000","last_change":"2024-04-02T07:49:07.277421+0000","last_active":"2024-04-02T07:49:08.720556+0000","last_peered":"2024-04-02T07:49:08.720556+0000","last_clean":"2024-04-02T07:49:08.720556+0000","last_became_active":"2024-04-02T07:49:07.277133+0000","last_became_peered":"2024-04-02T07:49:07.277133+0000","last_unstale":"2024-04-02T07:49:08.720556+0000","last_undegraded":"2024-04-02T07:49:08.720556+0000","last_fullsized":"2024-04-02T07:49:08.720556+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:41.434877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720335+0000","last_change":"2024-04-02T07:49:07.286456+0000","last_active":"2024-04-02T07:49:08.720335+0000","last_peered":"2024-04-02T07:49:08.720335+0000","last_clean":"2024-04-02T07:49:08.720335+0000","last_became_active":"2024-04-02T07:49:07.286109+0000","last_became_peered":"2024-04-02T07:49:07.286109+0000","last_unstale":"2024-04-02T07:49:08.720335+0000","last_undegraded":"2024-04-02T07:49:08.720335+0000","last_fullsized":"2024-04-02T07:49:08.720335+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:38:04.979730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720907+0000","last_change":"2024-04-02T07:49:07.276664+0000","last_active":"2024-04-02T07:49:08.720907+0000","last_peered":"2024-04-02T07:49:08.720907+0000","last_clean":"2024-04-02T07:49:08.720907+0000","last_became_active":"2024-04-02T07:49:07.276203+0000","last_became_peered":"2024-04-02T07:49:07.276203+0000","last_unstale":"2024-04-02T07:49:08.720907+0000","last_undegraded":"2024-04-02T07:49:08.720907+0000","last_fullsized":"2024-04-02T07:49:08.720907+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:48.734491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720850+0000","last_change":"2024-04-02T07:49:07.276619+0000","last_active":"2024-04-02T07:49:08.720850+0000","last_peered":"2024-04-02T07:49:08.720850+0000","last_clean":"2024-04-02T07:49:08.720850+0000","last_became_active":"2024-04-02T07:49:07.276274+0000","last_became_peered":"2024-04-02T07:49:07.276274+0000","last_unstale":"2024-04-02T07:49:08.720850+0000","last_undegraded":"2024-04-02T07:49:08.720850+0000","last_fullsized":"2024-04-02T07:49:08.720850+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:59:22.168507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.261297+0000","last_change":"2024-04-02T07:49:07.279836+0000","last_active":"2024-04-02T07:49:08.261297+0000","last_peered":"2024-04-02T07:49:08.261297+0000","last_clean":"2024-04-02T07:49:08.261297+0000","last_became_active":"2024-04-02T07:49:07.277770+0000","last_became_peered":"2024-04-02T07:49:07.277770+0000","last_unstale":"2024-04-02T07:49:08.261297+0000","last_undegraded":"2024-04-02T07:49:08.261297+0000","last_fullsized":"2024-04-02T07:49:08.261297+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:21:40.105740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721866+0000","last_change":"2024-04-02T07:49:07.279840+0000","last_active":"2024-04-02T07:49:08.721866+0000","last_peered":"2024-04-02T07:49:08.721866+0000","last_clean":"2024-04-02T07:49:08.721866+0000","last_became_active":"2024-04-02T07:49:07.279579+0000","last_became_peered":"2024-04-02T07:49:07.279579+0000","last_unstale":"2024-04-02T07:49:08.721866+0000","last_undegraded":"2024-04-02T07:49:08.721866+0000","last_fullsized":"2024-04-02T07:49:08.721866+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:46:51.306140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255255+0000","last_change":"2024-04-02T07:49:07.263112+0000","last_active":"2024-04-02T07:49:08.255255+0000","last_peered":"2024-04-02T07:49:08.255255+0000","last_clean":"2024-04-02T07:49:08.255255+0000","last_became_active":"2024-04-02T07:49:07.262831+0000","last_became_peered":"2024-04-02T07:49:07.262831+0000","last_unstale":"2024-04-02T07:49:08.255255+0000","last_undegraded":"2024-04-02T07:49:08.255255+0000","last_fullsized":"2024-04-02T07:49:08.255255+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:59:22.168507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721098+0000","last_change":"2024-04-02T07:49:07.289697+0000","last_active":"2024-04-02T07:49:08.721098+0000","last_peered":"2024-04-02T07:49:08.721098+0000","last_clean":"2024-04-02T07:49:08.721098+0000","last_became_active":"2024-04-02T07:49:07.289397+0000","last_became_peered":"2024-04-02T07:49:07.289397+0000","last_unstale":"2024-04-02T07:49:08.721098+0000","last_undegraded":"2024-04-02T07:49:08.721098+0000","last_fullsized":"2024-04-02T07:49:08.721098+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:39.433909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721472+0000","last_change":"2024-04-02T07:49:07.285828+0000","last_active":"2024-04-02T07:49:08.721472+0000","last_peered":"2024-04-02T07:49:08.721472+0000","last_clean":"2024-04-02T07:49:08.721472+0000","last_became_active":"2024-04-02T07:49:07.285523+0000","last_became_peered":"2024-04-02T07:49:07.285523+0000","last_unstale":"2024-04-02T07:49:08.721472+0000","last_undegraded":"2024-04-02T07:49:08.721472+0000","last_fullsized":"2024-04-02T07:49:08.721472+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:09:33.011128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720980+0000","last_change":"2024-04-02T07:49:07.285077+0000","last_active":"2024-04-02T07:49:08.720980+0000","last_peered":"2024-04-02T07:49:08.720980+0000","last_clean":"2024-04-02T07:49:08.720980+0000","last_became_active":"2024-04-02T07:49:07.284708+0000","last_became_peered":"2024-04-02T07:49:07.284708+0000","last_unstale":"2024-04-02T07:49:08.720980+0000","last_undegraded":"2024-04-02T07:49:08.720980+0000","last_fullsized":"2024-04-02T07:49:08.720980+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:54.506312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720404+0000","last_change":"2024-04-02T07:49:07.278701+0000","last_active":"2024-04-02T07:49:08.720404+0000","last_peered":"2024-04-02T07:49:08.720404+0000","last_clean":"2024-04-02T07:49:08.720404+0000","last_became_active":"2024-04-02T07:49:07.278403+0000","last_became_peered":"2024-04-02T07:49:07.278403+0000","last_unstale":"2024-04-02T07:49:08.720404+0000","last_undegraded":"2024-04-02T07:49:08.720404+0000","last_fullsized":"2024-04-02T07:49:08.720404+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:30:00.443550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255706+0000","last_change":"2024-04-02T07:49:07.269083+0000","last_active":"2024-04-02T07:49:08.255706+0000","last_peered":"2024-04-02T07:49:08.255706+0000","last_clean":"2024-04-02T07:49:08.255706+0000","last_became_active":"2024-04-02T07:49:07.268289+0000","last_became_peered":"2024-04-02T07:49:07.268289+0000","last_unstale":"2024-04-02T07:49:08.255706+0000","last_undegraded":"2024-04-02T07:49:08.255706+0000","last_fullsized":"2024-04-02T07:49:08.255706+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:41.434877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720906+0000","last_change":"2024-04-02T07:49:07.287962+0000","last_active":"2024-04-02T07:49:08.720906+0000","last_peered":"2024-04-02T07:49:08.720906+0000","last_clean":"2024-04-02T07:49:08.720906+0000","last_became_active":"2024-04-02T07:49:07.287732+0000","last_became_peered":"2024-04-02T07:49:07.287732+0000","last_unstale":"2024-04-02T07:49:08.720906+0000","last_undegraded":"2024-04-02T07:49:08.720906+0000","last_fullsized":"2024-04-02T07:49:08.720906+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:39:26.559000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721203+0000","last_change":"2024-04-02T07:49:07.277544+0000","last_active":"2024-04-02T07:49:08.721203+0000","last_peered":"2024-04-02T07:49:08.721203+0000","last_clean":"2024-04-02T07:49:08.721203+0000","last_became_active":"2024-04-02T07:49:07.277286+0000","last_became_peered":"2024-04-02T07:49:07.277286+0000","last_unstale":"2024-04-02T07:49:08.721203+0000","last_undegraded":"2024-04-02T07:49:08.721203+0000","last_fullsized":"2024-04-02T07:49:08.721203+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:06.730666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.261187+0000","last_change":"2024-04-02T07:49:07.268066+0000","last_active":"2024-04-02T07:49:08.261187+0000","last_peered":"2024-04-02T07:49:08.261187+0000","last_clean":"2024-04-02T07:49:08.261187+0000","last_became_active":"2024-04-02T07:49:07.267634+0000","last_became_peered":"2024-04-02T07:49:07.267634+0000","last_unstale":"2024-04-02T07:49:08.261187+0000","last_undegraded":"2024-04-02T07:49:08.261187+0000","last_fullsized":"2024-04-02T07:49:08.261187+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:14.047532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721605+0000","last_change":"2024-04-02T07:49:07.291933+0000","last_active":"2024-04-02T07:49:08.721605+0000","last_peered":"2024-04-02T07:49:08.721605+0000","last_clean":"2024-04-02T07:49:08.721605+0000","last_became_active":"2024-04-02T07:49:07.291513+0000","last_became_peered":"2024-04-02T07:49:07.291513+0000","last_unstale":"2024-04-02T07:49:08.721605+0000","last_undegraded":"2024-04-02T07:49:08.721605+0000","last_fullsized":"2024-04-02T07:49:08.721605+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:07:49.813861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260240+0000","last_change":"2024-04-02T07:49:07.267328+0000","last_active":"2024-04-02T07:49:08.260240+0000","last_peered":"2024-04-02T07:49:08.260240+0000","last_clean":"2024-04-02T07:49:08.260240+0000","last_became_active":"2024-04-02T07:49:07.267021+0000","last_became_peered":"2024-04-02T07:49:07.267021+0000","last_unstale":"2024-04-02T07:49:08.260240+0000","last_undegraded":"2024-04-02T07:49:08.260240+0000","last_fullsized":"2024-04-02T07:49:08.260240+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:43.126776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720491+0000","last_change":"2024-04-02T07:49:07.270666+0000","last_active":"2024-04-02T07:49:08.720491+0000","last_peered":"2024-04-02T07:49:08.720491+0000","last_clean":"2024-04-02T07:49:08.720491+0000","last_became_active":"2024-04-02T07:49:07.268707+0000","last_became_peered":"2024-04-02T07:49:07.268707+0000","last_unstale":"2024-04-02T07:49:08.720491+0000","last_undegraded":"2024-04-02T07:49:08.720491+0000","last_fullsized":"2024-04-02T07:49:08.720491+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:37:02.063955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255051+0000","last_change":"2024-04-02T07:49:07.285534+0000","last_active":"2024-04-02T07:49:08.255051+0000","last_peered":"2024-04-02T07:49:08.255051+0000","last_clean":"2024-04-02T07:49:08.255051+0000","last_became_active":"2024-04-02T07:49:07.285267+0000","last_became_peered":"2024-04-02T07:49:07.285267+0000","last_unstale":"2024-04-02T07:49:08.255051+0000","last_undegraded":"2024-04-02T07:49:08.255051+0000","last_fullsized":"2024-04-02T07:49:08.255051+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:55:05.885676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721540+0000","last_change":"2024-04-02T07:49:07.266772+0000","last_active":"2024-04-02T07:49:08.721540+0000","last_peered":"2024-04-02T07:49:08.721540+0000","last_clean":"2024-04-02T07:49:08.721540+0000","last_became_active":"2024-04-02T07:49:07.266510+0000","last_became_peered":"2024-04-02T07:49:07.266510+0000","last_unstale":"2024-04-02T07:49:08.721540+0000","last_undegraded":"2024-04-02T07:49:08.721540+0000","last_fullsized":"2024-04-02T07:49:08.721540+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:54.506312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721650+0000","last_change":"2024-04-02T07:49:07.292024+0000","last_active":"2024-04-02T07:49:08.721650+0000","last_peered":"2024-04-02T07:49:08.721650+0000","last_clean":"2024-04-02T07:49:08.721650+0000","last_became_active":"2024-04-02T07:49:07.291417+0000","last_became_peered":"2024-04-02T07:49:07.291417+0000","last_unstale":"2024-04-02T07:49:08.721650+0000","last_undegraded":"2024-04-02T07:49:08.721650+0000","last_fullsized":"2024-04-02T07:49:08.721650+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:14.047532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721158+0000","last_change":"2024-04-02T07:49:07.267529+0000","last_active":"2024-04-02T07:49:08.721158+0000","last_peered":"2024-04-02T07:49:08.721158+0000","last_clean":"2024-04-02T07:49:08.721158+0000","last_became_active":"2024-04-02T07:49:07.267239+0000","last_became_peered":"2024-04-02T07:49:07.267239+0000","last_unstale":"2024-04-02T07:49:08.721158+0000","last_undegraded":"2024-04-02T07:49:08.721158+0000","last_fullsized":"2024-04-02T07:49:08.721158+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:06:37.929045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720830+0000","last_change":"2024-04-02T07:49:07.264297+0000","last_active":"2024-04-02T07:49:08.720830+0000","last_peered":"2024-04-02T07:49:08.720830+0000","last_clean":"2024-04-02T07:49:08.720830+0000","last_became_active":"2024-04-02T07:49:07.264002+0000","last_became_peered":"2024-04-02T07:49:07.264002+0000","last_unstale":"2024-04-02T07:49:08.720830+0000","last_undegraded":"2024-04-02T07:49:08.720830+0000","last_fullsized":"2024-04-02T07:49:08.720830+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:57:08.130262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721631+0000","last_change":"2024-04-02T07:49:07.267326+0000","last_active":"2024-04-02T07:49:08.721631+0000","last_peered":"2024-04-02T07:49:08.721631+0000","last_clean":"2024-04-02T07:49:08.721631+0000","last_became_active":"2024-04-02T07:49:07.267026+0000","last_became_peered":"2024-04-02T07:49:07.267026+0000","last_unstale":"2024-04-02T07:49:08.721631+0000","last_undegraded":"2024-04-02T07:49:08.721631+0000","last_fullsized":"2024-04-02T07:49:08.721631+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:15.424782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255104+0000","last_change":"2024-04-02T07:49:07.264874+0000","last_active":"2024-04-02T07:49:08.255104+0000","last_peered":"2024-04-02T07:49:08.255104+0000","last_clean":"2024-04-02T07:49:08.255104+0000","last_became_active":"2024-04-02T07:49:07.263663+0000","last_became_peered":"2024-04-02T07:49:07.263663+0000","last_unstale":"2024-04-02T07:49:08.255104+0000","last_undegraded":"2024-04-02T07:49:08.255104+0000","last_fullsized":"2024-04-02T07:49:08.255104+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:42:43.149354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720826+0000","last_change":"2024-04-02T07:49:07.280144+0000","last_active":"2024-04-02T07:49:08.720826+0000","last_peered":"2024-04-02T07:49:08.720826+0000","last_clean":"2024-04-02T07:49:08.720826+0000","last_became_active":"2024-04-02T07:49:07.279063+0000","last_became_peered":"2024-04-02T07:49:07.279063+0000","last_unstale":"2024-04-02T07:49:08.720826+0000","last_undegraded":"2024-04-02T07:49:08.720826+0000","last_fullsized":"2024-04-02T07:49:08.720826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:30.069211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720907+0000","last_change":"2024-04-02T07:49:07.288423+0000","last_active":"2024-04-02T07:49:08.720907+0000","last_peered":"2024-04-02T07:49:08.720907+0000","last_clean":"2024-04-02T07:49:08.720907+0000","last_became_active":"2024-04-02T07:49:07.283287+0000","last_became_peered":"2024-04-02T07:49:07.283287+0000","last_unstale":"2024-04-02T07:49:08.720907+0000","last_undegraded":"2024-04-02T07:49:08.720907+0000","last_fullsized":"2024-04-02T07:49:08.720907+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:21.336897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260861+0000","last_change":"2024-04-02T07:49:07.271024+0000","last_active":"2024-04-02T07:49:08.260861+0000","last_peered":"2024-04-02T07:49:08.260861+0000","last_clean":"2024-04-02T07:49:08.260861+0000","last_became_active":"2024-04-02T07:49:07.270736+0000","last_became_peered":"2024-04-02T07:49:07.270736+0000","last_unstale":"2024-04-02T07:49:08.260861+0000","last_undegraded":"2024-04-02T07:49:08.260861+0000","last_fullsized":"2024-04-02T07:49:08.260861+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:08:24.551656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"65'4","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720850+0000","last_change":"2024-04-02T07:49:06.775388+0000","last_active":"2024-04-02T07:49:08.720850+0000","last_peered":"2024-04-02T07:49:08.720850+0000","last_clean":"2024-04-02T07:49:08.720850+0000","last_became_active":"2024-04-02T07:49:06.775096+0000","last_became_peered":"2024-04-02T07:49:06.775096+0000","last_unstale":"2024-04-02T07:49:08.720850+0000","last_undegraded":"2024-04-02T07:49:08.720850+0000","last_fullsized":"2024-04-02T07:49:08.720850+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:47:15.729728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721544+0000","last_change":"2024-04-02T07:49:07.279068+0000","last_active":"2024-04-02T07:49:08.721544+0000","last_peered":"2024-04-02T07:49:08.721544+0000","last_clean":"2024-04-02T07:49:08.721544+0000","last_became_active":"2024-04-02T07:49:07.278777+0000","last_became_peered":"2024-04-02T07:49:07.278777+0000","last_unstale":"2024-04-02T07:49:08.721544+0000","last_undegraded":"2024-04-02T07:49:08.721544+0000","last_fullsized":"2024-04-02T07:49:08.721544+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:21.336897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"65'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721502+0000","last_change":"2024-04-02T07:49:06.772305+0000","last_active":"2024-04-02T07:49:08.721502+0000","last_peered":"2024-04-02T07:49:08.721502+0000","last_clean":"2024-04-02T07:49:08.721502+0000","last_became_active":"2024-04-02T07:49:06.772023+0000","last_became_peered":"2024-04-02T07:49:06.772023+0000","last_unstale":"2024-04-02T07:49:08.721502+0000","last_undegraded":"2024-04-02T07:49:08.721502+0000","last_fullsized":"2024-04-02T07:49:08.721502+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:05.723370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255279+0000","last_change":"2024-04-02T07:49:07.269722+0000","last_active":"2024-04-02T07:49:08.255279+0000","last_peered":"2024-04-02T07:49:08.255279+0000","last_clean":"2024-04-02T07:49:08.255279+0000","last_became_active":"2024-04-02T07:49:07.269464+0000","last_became_peered":"2024-04-02T07:49:07.269464+0000","last_unstale":"2024-04-02T07:49:08.255279+0000","last_undegraded":"2024-04-02T07:49:08.255279+0000","last_fullsized":"2024-04-02T07:49:08.255279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:43.126776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"65'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720765+0000","last_change":"2024-04-02T07:49:06.770671+0000","last_active":"2024-04-02T07:49:08.720765+0000","last_peered":"2024-04-02T07:49:08.720765+0000","last_clean":"2024-04-02T07:49:08.720765+0000","last_became_active":"2024-04-02T07:49:06.770402+0000","last_became_peered":"2024-04-02T07:49:06.770402+0000","last_unstale":"2024-04-02T07:49:08.720765+0000","last_undegraded":"2024-04-02T07:49:08.720765+0000","last_fullsized":"2024-04-02T07:49:08.720765+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:46:46.772743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721885+0000","last_change":"2024-04-02T07:49:07.285066+0000","last_active":"2024-04-02T07:49:08.721885+0000","last_peered":"2024-04-02T07:49:08.721885+0000","last_clean":"2024-04-02T07:49:08.721885+0000","last_became_active":"2024-04-02T07:49:07.284773+0000","last_became_peered":"2024-04-02T07:49:07.284773+0000","last_unstale":"2024-04-02T07:49:08.721885+0000","last_undegraded":"2024-04-02T07:49:08.721885+0000","last_fullsized":"2024-04-02T07:49:08.721885+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:37.672940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720911+0000","last_change":"2024-04-02T07:49:06.775799+0000","last_active":"2024-04-02T07:49:08.720911+0000","last_peered":"2024-04-02T07:49:08.720911+0000","last_clean":"2024-04-02T07:49:08.720911+0000","last_became_active":"2024-04-02T07:49:06.775519+0000","last_became_peered":"2024-04-02T07:49:06.775519+0000","last_unstale":"2024-04-02T07:49:08.720911+0000","last_undegraded":"2024-04-02T07:49:08.720911+0000","last_fullsized":"2024-04-02T07:49:08.720911+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:25.169012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720618+0000","last_change":"2024-04-02T07:49:07.267601+0000","last_active":"2024-04-02T07:49:08.720618+0000","last_peered":"2024-04-02T07:49:08.720618+0000","last_clean":"2024-04-02T07:49:08.720618+0000","last_became_active":"2024-04-02T07:49:07.267348+0000","last_became_peered":"2024-04-02T07:49:07.267348+0000","last_unstale":"2024-04-02T07:49:08.720618+0000","last_undegraded":"2024-04-02T07:49:08.720618+0000","last_fullsized":"2024-04-02T07:49:08.720618+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:27:21.879339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"65'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721159+0000","last_change":"2024-04-02T07:49:06.771720+0000","last_active":"2024-04-02T07:49:08.721159+0000","last_peered":"2024-04-02T07:49:08.721159+0000","last_clean":"2024-04-02T07:49:08.721159+0000","last_became_active":"2024-04-02T07:49:06.771156+0000","last_became_peered":"2024-04-02T07:49:06.771156+0000","last_unstale":"2024-04-02T07:49:08.721159+0000","last_undegraded":"2024-04-02T07:49:08.721159+0000","last_fullsized":"2024-04-02T07:49:08.721159+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:12.571159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260800+0000","last_change":"2024-04-02T07:49:07.292076+0000","last_active":"2024-04-02T07:49:08.260800+0000","last_peered":"2024-04-02T07:49:08.260800+0000","last_clean":"2024-04-02T07:49:08.260800+0000","last_became_active":"2024-04-02T07:49:07.290712+0000","last_became_peered":"2024-04-02T07:49:07.290712+0000","last_unstale":"2024-04-02T07:49:08.260800+0000","last_undegraded":"2024-04-02T07:49:08.260800+0000","last_fullsized":"2024-04-02T07:49:08.260800+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:40:57.913800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"65'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720725+0000","last_change":"2024-04-02T07:49:06.777869+0000","last_active":"2024-04-02T07:49:08.720725+0000","last_peered":"2024-04-02T07:49:08.720725+0000","last_clean":"2024-04-02T07:49:08.720725+0000","last_became_active":"2024-04-02T07:49:06.777599+0000","last_became_peered":"2024-04-02T07:49:06.777599+0000","last_unstale":"2024-04-02T07:49:08.720725+0000","last_undegraded":"2024-04-02T07:49:08.720725+0000","last_fullsized":"2024-04-02T07:49:08.720725+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:53.499016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260318+0000","last_change":"2024-04-02T07:49:07.273326+0000","last_active":"2024-04-02T07:49:08.260318+0000","last_peered":"2024-04-02T07:49:08.260318+0000","last_clean":"2024-04-02T07:49:08.260318+0000","last_became_active":"2024-04-02T07:49:07.272995+0000","last_became_peered":"2024-04-02T07:49:07.272995+0000","last_unstale":"2024-04-02T07:49:08.260318+0000","last_undegraded":"2024-04-02T07:49:08.260318+0000","last_fullsized":"2024-04-02T07:49:08.260318+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:34:00.481993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721088+0000","last_change":"2024-04-02T07:49:06.769138+0000","last_active":"2024-04-02T07:49:08.721088+0000","last_peered":"2024-04-02T07:49:08.721088+0000","last_clean":"2024-04-02T07:49:08.721088+0000","last_became_active":"2024-04-02T07:49:06.768833+0000","last_became_peered":"2024-04-02T07:49:06.768833+0000","last_unstale":"2024-04-02T07:49:08.721088+0000","last_undegraded":"2024-04-02T07:49:08.721088+0000","last_fullsized":"2024-04-02T07:49:08.721088+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:59:21.161210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720582+0000","last_change":"2024-04-02T07:49:07.273900+0000","last_active":"2024-04-02T07:49:08.720582+0000","last_peered":"2024-04-02T07:49:08.720582+0000","last_clean":"2024-04-02T07:49:08.720582+0000","last_became_active":"2024-04-02T07:49:07.273645+0000","last_became_peered":"2024-04-02T07:49:07.273645+0000","last_unstale":"2024-04-02T07:49:08.720582+0000","last_undegraded":"2024-04-02T07:49:08.720582+0000","last_fullsized":"2024-04-02T07:49:08.720582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:34.296961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260767+0000","last_change":"2024-04-02T07:49:06.773089+0000","last_active":"2024-04-02T07:49:08.260767+0000","last_peered":"2024-04-02T07:49:08.260767+0000","last_clean":"2024-04-02T07:49:08.260767+0000","last_became_active":"2024-04-02T07:49:06.772754+0000","last_became_peered":"2024-04-02T07:49:06.772754+0000","last_unstale":"2024-04-02T07:49:08.260767+0000","last_undegraded":"2024-04-02T07:49:08.260767+0000","last_fullsized":"2024-04-02T07:49:08.260767+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:31.991732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721009+0000","last_change":"2024-04-02T07:49:07.276475+0000","last_active":"2024-04-02T07:49:08.721009+0000","last_peered":"2024-04-02T07:49:08.721009+0000","last_clean":"2024-04-02T07:49:08.721009+0000","last_became_active":"2024-04-02T07:49:07.275708+0000","last_became_peered":"2024-04-02T07:49:07.275708+0000","last_unstale":"2024-04-02T07:49:08.721009+0000","last_undegraded":"2024-04-02T07:49:08.721009+0000","last_fullsized":"2024-04-02T07:49:08.721009+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:30.069211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720987+0000","last_change":"2024-04-02T07:49:06.773667+0000","last_active":"2024-04-02T07:49:08.720987+0000","last_peered":"2024-04-02T07:49:08.720987+0000","last_clean":"2024-04-02T07:49:08.720987+0000","last_became_active":"2024-04-02T07:49:06.773195+0000","last_became_peered":"2024-04-02T07:49:06.773195+0000","last_unstale":"2024-04-02T07:49:08.720987+0000","last_undegraded":"2024-04-02T07:49:08.720987+0000","last_fullsized":"2024-04-02T07:49:08.720987+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:12.571159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720901+0000","last_change":"2024-04-02T07:49:07.274350+0000","last_active":"2024-04-02T07:49:08.720901+0000","last_peered":"2024-04-02T07:49:08.720901+0000","last_clean":"2024-04-02T07:49:08.720901+0000","last_became_active":"2024-04-02T07:49:07.274095+0000","last_became_peered":"2024-04-02T07:49:07.274095+0000","last_unstale":"2024-04-02T07:49:08.720901+0000","last_undegraded":"2024-04-02T07:49:08.720901+0000","last_fullsized":"2024-04-02T07:49:08.720901+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:59:22.168507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.261288+0000","last_change":"2024-04-02T07:49:06.772056+0000","last_active":"2024-04-02T07:49:08.261288+0000","last_peered":"2024-04-02T07:49:08.261288+0000","last_clean":"2024-04-02T07:49:08.261288+0000","last_became_active":"2024-04-02T07:49:06.769275+0000","last_became_peered":"2024-04-02T07:49:06.769275+0000","last_unstale":"2024-04-02T07:49:08.261288+0000","last_undegraded":"2024-04-02T07:49:08.261288+0000","last_fullsized":"2024-04-02T07:49:08.261288+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:36:38.777298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721571+0000","last_change":"2024-04-02T07:49:07.292660+0000","last_active":"2024-04-02T07:49:08.721571+0000","last_peered":"2024-04-02T07:49:08.721571+0000","last_clean":"2024-04-02T07:49:08.721571+0000","last_became_active":"2024-04-02T07:49:07.292445+0000","last_became_peered":"2024-04-02T07:49:07.292445+0000","last_unstale":"2024-04-02T07:49:08.721571+0000","last_undegraded":"2024-04-02T07:49:08.721571+0000","last_fullsized":"2024-04-02T07:49:08.721571+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:23:30.112352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"65'3","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255375+0000","last_change":"2024-04-02T07:49:06.770296+0000","last_active":"2024-04-02T07:49:08.255375+0000","last_peered":"2024-04-02T07:49:08.255375+0000","last_clean":"2024-04-02T07:49:08.255375+0000","last_became_active":"2024-04-02T07:49:06.767492+0000","last_became_peered":"2024-04-02T07:49:06.767492+0000","last_unstale":"2024-04-02T07:49:08.255375+0000","last_undegraded":"2024-04-02T07:49:08.255375+0000","last_fullsized":"2024-04-02T07:49:08.255375+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:59.964498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721245+0000","last_change":"2024-04-02T07:49:07.287153+0000","last_active":"2024-04-02T07:49:08.721245+0000","last_peered":"2024-04-02T07:49:08.721245+0000","last_clean":"2024-04-02T07:49:08.721245+0000","last_became_active":"2024-04-02T07:49:07.285894+0000","last_became_peered":"2024-04-02T07:49:07.285894+0000","last_unstale":"2024-04-02T07:49:08.721245+0000","last_undegraded":"2024-04-02T07:49:08.721245+0000","last_fullsized":"2024-04-02T07:49:08.721245+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:08.988202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"65'6","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260930+0000","last_change":"2024-04-02T07:49:06.774244+0000","last_active":"2024-04-02T07:49:08.260930+0000","last_peered":"2024-04-02T07:49:08.260930+0000","last_clean":"2024-04-02T07:49:08.260930+0000","last_became_active":"2024-04-02T07:49:06.773643+0000","last_became_peered":"2024-04-02T07:49:06.773643+0000","last_unstale":"2024-04-02T07:49:08.260930+0000","last_undegraded":"2024-04-02T07:49:08.260930+0000","last_fullsized":"2024-04-02T07:49:08.260930+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:18:56.421172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720695+0000","last_change":"2024-04-02T07:49:07.287960+0000","last_active":"2024-04-02T07:49:08.720695+0000","last_peered":"2024-04-02T07:49:08.720695+0000","last_clean":"2024-04-02T07:49:08.720695+0000","last_became_active":"2024-04-02T07:49:07.286839+0000","last_became_peered":"2024-04-02T07:49:07.286839+0000","last_unstale":"2024-04-02T07:49:08.720695+0000","last_undegraded":"2024-04-02T07:49:08.720695+0000","last_fullsized":"2024-04-02T07:49:08.720695+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:39.433909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"65'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260497+0000","last_change":"2024-04-02T07:49:06.766693+0000","last_active":"2024-04-02T07:49:08.260497+0000","last_peered":"2024-04-02T07:49:08.260497+0000","last_clean":"2024-04-02T07:49:08.260497+0000","last_became_active":"2024-04-02T07:49:06.766422+0000","last_became_peered":"2024-04-02T07:49:06.766422+0000","last_unstale":"2024-04-02T07:49:08.260497+0000","last_undegraded":"2024-04-02T07:49:08.260497+0000","last_fullsized":"2024-04-02T07:49:08.260497+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:14:44.817850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260294+0000","last_change":"2024-04-02T07:49:07.266844+0000","last_active":"2024-04-02T07:49:08.260294+0000","last_peered":"2024-04-02T07:49:08.260294+0000","last_clean":"2024-04-02T07:49:08.260294+0000","last_became_active":"2024-04-02T07:49:07.266526+0000","last_became_peered":"2024-04-02T07:49:07.266526+0000","last_unstale":"2024-04-02T07:49:08.260294+0000","last_undegraded":"2024-04-02T07:49:08.260294+0000","last_fullsized":"2024-04-02T07:49:08.260294+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:01:57.567763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721154+0000","last_change":"2024-04-02T07:49:06.763592+0000","last_active":"2024-04-02T07:49:08.721154+0000","last_peered":"2024-04-02T07:49:08.721154+0000","last_clean":"2024-04-02T07:49:08.721154+0000","last_became_active":"2024-04-02T07:49:06.763337+0000","last_became_peered":"2024-04-02T07:49:06.763337+0000","last_unstale":"2024-04-02T07:49:08.721154+0000","last_undegraded":"2024-04-02T07:49:08.721154+0000","last_fullsized":"2024-04-02T07:49:08.721154+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:39:25.551704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.259960+0000","last_change":"2024-04-02T07:49:07.266589+0000","last_active":"2024-04-02T07:49:08.259960+0000","last_peered":"2024-04-02T07:49:08.259960+0000","last_clean":"2024-04-02T07:49:08.259960+0000","last_became_active":"2024-04-02T07:49:07.266103+0000","last_became_peered":"2024-04-02T07:49:07.266103+0000","last_unstale":"2024-04-02T07:49:08.259960+0000","last_undegraded":"2024-04-02T07:49:08.259960+0000","last_fullsized":"2024-04-02T07:49:08.259960+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:30.369003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720914+0000","last_change":"2024-04-02T07:49:06.774807+0000","last_active":"2024-04-02T07:49:08.720914+0000","last_peered":"2024-04-02T07:49:08.720914+0000","last_clean":"2024-04-02T07:49:08.720914+0000","last_became_active":"2024-04-02T07:49:06.774537+0000","last_became_peered":"2024-04-02T07:49:06.774537+0000","last_unstale":"2024-04-02T07:49:08.720914+0000","last_undegraded":"2024-04-02T07:49:08.720914+0000","last_fullsized":"2024-04-02T07:49:08.720914+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:35.405121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255054+0000","last_change":"2024-04-02T07:49:07.268689+0000","last_active":"2024-04-02T07:49:08.255054+0000","last_peered":"2024-04-02T07:49:08.255054+0000","last_clean":"2024-04-02T07:49:08.255054+0000","last_became_active":"2024-04-02T07:49:07.268408+0000","last_became_peered":"2024-04-02T07:49:07.268408+0000","last_unstale":"2024-04-02T07:49:08.255054+0000","last_undegraded":"2024-04-02T07:49:08.255054+0000","last_fullsized":"2024-04-02T07:49:08.255054+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:30.069211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"65'8","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260881+0000","last_change":"2024-04-02T07:49:06.772071+0000","last_active":"2024-04-02T07:49:08.260881+0000","last_peered":"2024-04-02T07:49:08.260881+0000","last_clean":"2024-04-02T07:49:08.260881+0000","last_became_active":"2024-04-02T07:49:06.765463+0000","last_became_peered":"2024-04-02T07:49:06.765463+0000","last_unstale":"2024-04-02T07:49:08.260881+0000","last_undegraded":"2024-04-02T07:49:08.260881+0000","last_fullsized":"2024-04-02T07:49:08.260881+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:24:42.052205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260953+0000","last_change":"2024-04-02T07:49:07.267320+0000","last_active":"2024-04-02T07:49:08.260953+0000","last_peered":"2024-04-02T07:49:08.260953+0000","last_clean":"2024-04-02T07:49:08.260953+0000","last_became_active":"2024-04-02T07:49:07.266855+0000","last_became_peered":"2024-04-02T07:49:07.266855+0000","last_unstale":"2024-04-02T07:49:08.260953+0000","last_undegraded":"2024-04-02T07:49:08.260953+0000","last_fullsized":"2024-04-02T07:49:08.260953+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:07.809075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"65'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721817+0000","last_change":"2024-04-02T07:49:06.776827+0000","last_active":"2024-04-02T07:49:08.721817+0000","last_peered":"2024-04-02T07:49:08.721817+0000","last_clean":"2024-04-02T07:49:08.721817+0000","last_became_active":"2024-04-02T07:49:06.776535+0000","last_became_peered":"2024-04-02T07:49:06.776535+0000","last_unstale":"2024-04-02T07:49:08.721817+0000","last_undegraded":"2024-04-02T07:49:08.721817+0000","last_fullsized":"2024-04-02T07:49:08.721817+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:06.801779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260600+0000","last_change":"2024-04-02T07:49:07.271937+0000","last_active":"2024-04-02T07:49:08.260600+0000","last_peered":"2024-04-02T07:49:08.260600+0000","last_clean":"2024-04-02T07:49:08.260600+0000","last_became_active":"2024-04-02T07:49:07.269243+0000","last_became_peered":"2024-04-02T07:49:07.269243+0000","last_unstale":"2024-04-02T07:49:08.260600+0000","last_undegraded":"2024-04-02T07:49:08.260600+0000","last_fullsized":"2024-04-02T07:49:08.260600+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:10:05.251356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"65'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721092+0000","last_change":"2024-04-02T07:49:06.772431+0000","last_active":"2024-04-02T07:49:08.721092+0000","last_peered":"2024-04-02T07:49:08.721092+0000","last_clean":"2024-04-02T07:49:08.721092+0000","last_became_active":"2024-04-02T07:49:06.771981+0000","last_became_peered":"2024-04-02T07:49:06.771981+0000","last_unstale":"2024-04-02T07:49:08.721092+0000","last_undegraded":"2024-04-02T07:49:08.721092+0000","last_fullsized":"2024-04-02T07:49:08.721092+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:35.405121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255352+0000","last_change":"2024-04-02T07:49:07.285539+0000","last_active":"2024-04-02T07:49:08.255352+0000","last_peered":"2024-04-02T07:49:08.255352+0000","last_clean":"2024-04-02T07:49:08.255352+0000","last_became_active":"2024-04-02T07:49:07.285282+0000","last_became_peered":"2024-04-02T07:49:07.285282+0000","last_unstale":"2024-04-02T07:49:08.255352+0000","last_undegraded":"2024-04-02T07:49:08.255352+0000","last_fullsized":"2024-04-02T07:49:08.255352+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:39.433909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"65'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721384+0000","last_change":"2024-04-02T07:49:06.777265+0000","last_active":"2024-04-02T07:49:08.721384+0000","last_peered":"2024-04-02T07:49:08.721384+0000","last_clean":"2024-04-02T07:49:08.721384+0000","last_became_active":"2024-04-02T07:49:06.776996+0000","last_became_peered":"2024-04-02T07:49:06.776996+0000","last_unstale":"2024-04-02T07:49:08.721384+0000","last_undegraded":"2024-04-02T07:49:08.721384+0000","last_fullsized":"2024-04-02T07:49:08.721384+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:40:17.483842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721391+0000","last_change":"2024-04-02T07:49:07.282868+0000","last_active":"2024-04-02T07:49:08.721391+0000","last_peered":"2024-04-02T07:49:08.721391+0000","last_clean":"2024-04-02T07:49:08.721391+0000","last_became_active":"2024-04-02T07:49:07.282469+0000","last_became_peered":"2024-04-02T07:49:07.282469+0000","last_unstale":"2024-04-02T07:49:08.721391+0000","last_undegraded":"2024-04-02T07:49:08.721391+0000","last_fullsized":"2024-04-02T07:49:08.721391+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:55:05.885676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720590+0000","last_change":"2024-04-02T07:49:06.774712+0000","last_active":"2024-04-02T07:49:08.720590+0000","last_peered":"2024-04-02T07:49:08.720590+0000","last_clean":"2024-04-02T07:49:08.720590+0000","last_became_active":"2024-04-02T07:49:06.774321+0000","last_became_peered":"2024-04-02T07:49:06.774321+0000","last_unstale":"2024-04-02T07:49:08.720590+0000","last_undegraded":"2024-04-02T07:49:08.720590+0000","last_fullsized":"2024-04-02T07:49:08.720590+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:40:17.483842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260343+0000","last_change":"2024-04-02T07:49:07.284161+0000","last_active":"2024-04-02T07:49:08.260343+0000","last_peered":"2024-04-02T07:49:08.260343+0000","last_clean":"2024-04-02T07:49:08.260343+0000","last_became_active":"2024-04-02T07:49:07.283662+0000","last_became_peered":"2024-04-02T07:49:07.283662+0000","last_unstale":"2024-04-02T07:49:08.260343+0000","last_undegraded":"2024-04-02T07:49:08.260343+0000","last_fullsized":"2024-04-02T07:49:08.260343+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:37.672940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"65'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260906+0000","last_change":"2024-04-02T07:49:06.770365+0000","last_active":"2024-04-02T07:49:08.260906+0000","last_peered":"2024-04-02T07:49:08.260906+0000","last_clean":"2024-04-02T07:49:08.260906+0000","last_became_active":"2024-04-02T07:49:06.769151+0000","last_became_peered":"2024-04-02T07:49:06.769151+0000","last_unstale":"2024-04-02T07:49:08.260906+0000","last_undegraded":"2024-04-02T07:49:08.260906+0000","last_fullsized":"2024-04-02T07:49:08.260906+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:15.218952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721150+0000","last_change":"2024-04-02T07:49:07.276632+0000","last_active":"2024-04-02T07:49:08.721150+0000","last_peered":"2024-04-02T07:49:08.721150+0000","last_clean":"2024-04-02T07:49:08.721150+0000","last_became_active":"2024-04-02T07:49:07.275010+0000","last_became_peered":"2024-04-02T07:49:07.275010+0000","last_unstale":"2024-04-02T07:49:08.721150+0000","last_undegraded":"2024-04-02T07:49:08.721150+0000","last_fullsized":"2024-04-02T07:49:08.721150+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:39:26.559000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721235+0000","last_change":"2024-04-02T07:49:06.769474+0000","last_active":"2024-04-02T07:49:08.721235+0000","last_peered":"2024-04-02T07:49:08.721235+0000","last_clean":"2024-04-02T07:49:08.721235+0000","last_became_active":"2024-04-02T07:49:06.769236+0000","last_became_peered":"2024-04-02T07:49:06.769236+0000","last_unstale":"2024-04-02T07:49:08.721235+0000","last_undegraded":"2024-04-02T07:49:08.721235+0000","last_fullsized":"2024-04-02T07:49:08.721235+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:40.427580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721018+0000","last_change":"2024-04-02T07:49:07.288867+0000","last_active":"2024-04-02T07:49:08.721018+0000","last_peered":"2024-04-02T07:49:08.721018+0000","last_clean":"2024-04-02T07:49:08.721018+0000","last_became_active":"2024-04-02T07:49:07.288122+0000","last_became_peered":"2024-04-02T07:49:07.288122+0000","last_unstale":"2024-04-02T07:49:08.721018+0000","last_undegraded":"2024-04-02T07:49:08.721018+0000","last_fullsized":"2024-04-02T07:49:08.721018+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:26.176308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260146+0000","last_change":"2024-04-02T07:49:06.774146+0000","last_active":"2024-04-02T07:49:08.260146+0000","last_peered":"2024-04-02T07:49:08.260146+0000","last_clean":"2024-04-02T07:49:08.260146+0000","last_became_active":"2024-04-02T07:49:06.770861+0000","last_became_peered":"2024-04-02T07:49:06.770861+0000","last_unstale":"2024-04-02T07:49:08.260146+0000","last_undegraded":"2024-04-02T07:49:08.260146+0000","last_fullsized":"2024-04-02T07:49:08.260146+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:14.417485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721279+0000","last_change":"2024-04-02T07:49:07.285416+0000","last_active":"2024-04-02T07:49:08.721279+0000","last_peered":"2024-04-02T07:49:08.721279+0000","last_clean":"2024-04-02T07:49:08.721279+0000","last_became_active":"2024-04-02T07:49:07.285136+0000","last_became_peered":"2024-04-02T07:49:07.285136+0000","last_unstale":"2024-04-02T07:49:08.721279+0000","last_undegraded":"2024-04-02T07:49:08.721279+0000","last_fullsized":"2024-04-02T07:49:08.721279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:34:00.481993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720320+0000","last_change":"2024-04-02T07:49:06.768690+0000","last_active":"2024-04-02T07:49:08.720320+0000","last_peered":"2024-04-02T07:49:08.720320+0000","last_clean":"2024-04-02T07:49:08.720320+0000","last_became_active":"2024-04-02T07:49:06.768410+0000","last_became_peered":"2024-04-02T07:49:06.768410+0000","last_unstale":"2024-04-02T07:49:08.720320+0000","last_undegraded":"2024-04-02T07:49:08.720320+0000","last_fullsized":"2024-04-02T07:49:08.720320+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:20.329600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720649+0000","last_change":"2024-04-02T07:49:07.279081+0000","last_active":"2024-04-02T07:49:08.720649+0000","last_peered":"2024-04-02T07:49:08.720649+0000","last_clean":"2024-04-02T07:49:08.720649+0000","last_became_active":"2024-04-02T07:49:07.278803+0000","last_became_peered":"2024-04-02T07:49:07.278803+0000","last_unstale":"2024-04-02T07:49:08.720649+0000","last_undegraded":"2024-04-02T07:49:08.720649+0000","last_fullsized":"2024-04-02T07:49:08.720649+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:06:38.738379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721814+0000","last_change":"2024-04-02T07:49:06.774899+0000","last_active":"2024-04-02T07:49:08.721814+0000","last_peered":"2024-04-02T07:49:08.721814+0000","last_clean":"2024-04-02T07:49:08.721814+0000","last_became_active":"2024-04-02T07:49:06.773640+0000","last_became_peered":"2024-04-02T07:49:06.773640+0000","last_unstale":"2024-04-02T07:49:08.721814+0000","last_undegraded":"2024-04-02T07:49:08.721814+0000","last_fullsized":"2024-04-02T07:49:08.721814+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:42.119480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260800+0000","last_change":"2024-04-02T07:49:07.267917+0000","last_active":"2024-04-02T07:49:08.260800+0000","last_peered":"2024-04-02T07:49:08.260800+0000","last_clean":"2024-04-02T07:49:08.260800+0000","last_became_active":"2024-04-02T07:49:07.267308+0000","last_became_peered":"2024-04-02T07:49:07.267308+0000","last_unstale":"2024-04-02T07:49:08.260800+0000","last_undegraded":"2024-04-02T07:49:08.260800+0000","last_fullsized":"2024-04-02T07:49:08.260800+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:07:49.813861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721307+0000","last_change":"2024-04-02T07:49:06.775967+0000","last_active":"2024-04-02T07:49:08.721307+0000","last_peered":"2024-04-02T07:49:08.721307+0000","last_clean":"2024-04-02T07:49:08.721307+0000","last_became_active":"2024-04-02T07:49:06.775406+0000","last_became_peered":"2024-04-02T07:49:06.775406+0000","last_unstale":"2024-04-02T07:49:08.721307+0000","last_undegraded":"2024-04-02T07:49:08.721307+0000","last_fullsized":"2024-04-02T07:49:08.721307+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:38.426612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721606+0000","last_change":"2024-04-02T07:49:07.286788+0000","last_active":"2024-04-02T07:49:08.721606+0000","last_peered":"2024-04-02T07:49:08.721606+0000","last_clean":"2024-04-02T07:49:08.721606+0000","last_became_active":"2024-04-02T07:49:07.285696+0000","last_became_peered":"2024-04-02T07:49:07.285696+0000","last_unstale":"2024-04-02T07:49:08.721606+0000","last_undegraded":"2024-04-02T07:49:08.721606+0000","last_fullsized":"2024-04-02T07:49:08.721606+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:05.238772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260630+0000","last_change":"2024-04-02T07:49:06.770811+0000","last_active":"2024-04-02T07:49:08.260630+0000","last_peered":"2024-04-02T07:49:08.260630+0000","last_clean":"2024-04-02T07:49:08.260630+0000","last_became_active":"2024-04-02T07:49:06.770515+0000","last_became_peered":"2024-04-02T07:49:06.770515+0000","last_unstale":"2024-04-02T07:49:08.260630+0000","last_undegraded":"2024-04-02T07:49:08.260630+0000","last_fullsized":"2024-04-02T07:49:08.260630+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:41:59.449223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260983+0000","last_change":"2024-04-02T07:49:07.284033+0000","last_active":"2024-04-02T07:49:08.260983+0000","last_peered":"2024-04-02T07:49:08.260983+0000","last_clean":"2024-04-02T07:49:08.260983+0000","last_became_active":"2024-04-02T07:49:07.283785+0000","last_became_peered":"2024-04-02T07:49:07.283785+0000","last_unstale":"2024-04-02T07:49:08.260983+0000","last_undegraded":"2024-04-02T07:49:08.260983+0000","last_fullsized":"2024-04-02T07:49:08.260983+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:46:51.306140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721646+0000","last_change":"2024-04-02T07:49:06.774465+0000","last_active":"2024-04-02T07:49:08.721646+0000","last_peered":"2024-04-02T07:49:08.721646+0000","last_clean":"2024-04-02T07:49:08.721646+0000","last_became_active":"2024-04-02T07:49:06.774182+0000","last_became_peered":"2024-04-02T07:49:06.774182+0000","last_unstale":"2024-04-02T07:49:08.721646+0000","last_undegraded":"2024-04-02T07:49:08.721646+0000","last_fullsized":"2024-04-02T07:49:08.721646+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:42.119480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721683+0000","last_change":"2024-04-02T07:49:07.288465+0000","last_active":"2024-04-02T07:49:08.721683+0000","last_peered":"2024-04-02T07:49:08.721683+0000","last_clean":"2024-04-02T07:49:08.721683+0000","last_became_active":"2024-04-02T07:49:07.288113+0000","last_became_peered":"2024-04-02T07:49:07.288113+0000","last_unstale":"2024-04-02T07:49:08.721683+0000","last_undegraded":"2024-04-02T07:49:08.721683+0000","last_fullsized":"2024-04-02T07:49:08.721683+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:40:57.913800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720809+0000","last_change":"2024-04-02T07:49:06.773102+0000","last_active":"2024-04-02T07:49:08.720809+0000","last_peered":"2024-04-02T07:49:08.720809+0000","last_clean":"2024-04-02T07:49:08.720809+0000","last_became_active":"2024-04-02T07:49:06.771951+0000","last_became_peered":"2024-04-02T07:49:06.771951+0000","last_unstale":"2024-04-02T07:49:08.720809+0000","last_undegraded":"2024-04-02T07:49:08.720809+0000","last_fullsized":"2024-04-02T07:49:08.720809+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:36:38.777298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720270+0000","last_change":"2024-04-02T07:49:07.288118+0000","last_active":"2024-04-02T07:49:08.720270+0000","last_peered":"2024-04-02T07:49:08.720270+0000","last_clean":"2024-04-02T07:49:08.720270+0000","last_became_active":"2024-04-02T07:49:07.287705+0000","last_became_peered":"2024-04-02T07:49:07.287705+0000","last_unstale":"2024-04-02T07:49:08.720270+0000","last_undegraded":"2024-04-02T07:49:08.720270+0000","last_fullsized":"2024-04-02T07:49:08.720270+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:41.434877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720992+0000","last_change":"2024-04-02T07:49:06.773781+0000","last_active":"2024-04-02T07:49:08.720992+0000","last_peered":"2024-04-02T07:49:08.720992+0000","last_clean":"2024-04-02T07:49:08.720992+0000","last_became_active":"2024-04-02T07:49:06.773043+0000","last_became_peered":"2024-04-02T07:49:06.773043+0000","last_unstale":"2024-04-02T07:49:08.720992+0000","last_undegraded":"2024-04-02T07:49:08.720992+0000","last_fullsized":"2024-04-02T07:49:08.720992+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:59.964498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255203+0000","last_change":"2024-04-02T07:49:07.269078+0000","last_active":"2024-04-02T07:49:08.255203+0000","last_peered":"2024-04-02T07:49:08.255203+0000","last_clean":"2024-04-02T07:49:08.255203+0000","last_became_active":"2024-04-02T07:49:07.267893+0000","last_became_peered":"2024-04-02T07:49:07.267893+0000","last_unstale":"2024-04-02T07:49:08.255203+0000","last_undegraded":"2024-04-02T07:49:08.255203+0000","last_fullsized":"2024-04-02T07:49:08.255203+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:33.794321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260032+0000","last_change":"2024-04-02T07:49:06.765671+0000","last_active":"2024-04-02T07:49:08.260032+0000","last_peered":"2024-04-02T07:49:08.260032+0000","last_clean":"2024-04-02T07:49:08.260032+0000","last_became_active":"2024-04-02T07:49:06.765377+0000","last_became_peered":"2024-04-02T07:49:06.765377+0000","last_unstale":"2024-04-02T07:49:08.260032+0000","last_undegraded":"2024-04-02T07:49:08.260032+0000","last_fullsized":"2024-04-02T07:49:08.260032+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:19:10.575751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721771+0000","last_change":"2024-04-02T07:49:07.283554+0000","last_active":"2024-04-02T07:49:08.721771+0000","last_peered":"2024-04-02T07:49:08.721771+0000","last_clean":"2024-04-02T07:49:08.721771+0000","last_became_active":"2024-04-02T07:49:07.282688+0000","last_became_peered":"2024-04-02T07:49:07.282688+0000","last_unstale":"2024-04-02T07:49:08.721771+0000","last_undegraded":"2024-04-02T07:49:08.721771+0000","last_fullsized":"2024-04-02T07:49:08.721771+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:30:00.443550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"65'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720759+0000","last_change":"2024-04-02T07:49:06.772504+0000","last_active":"2024-04-02T07:49:08.720759+0000","last_peered":"2024-04-02T07:49:08.720759+0000","last_clean":"2024-04-02T07:49:08.720759+0000","last_became_active":"2024-04-02T07:49:06.771939+0000","last_became_peered":"2024-04-02T07:49:06.771939+0000","last_unstale":"2024-04-02T07:49:08.720759+0000","last_undegraded":"2024-04-02T07:49:08.720759+0000","last_fullsized":"2024-04-02T07:49:08.720759+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:14.417485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"55'139","reported_seq":288,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T07:49:07.262003+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:49:07.262003+0000","last_peered":"2024-04-02T07:49:07.262003+0000","last_clean":"2024-04-02T07:49:07.262003+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:49:07.262003+0000","last_undegraded":"2024-04-02T07:49:07.262003+0000","last_fullsized":"2024-04-02T07:49:07.262003+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575011,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31896,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339944,"statfs":{"total":96636764160,"available":96604102656,"internally_reserved":0,"allocated":790528,"data_stored":372301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.66500000000000004,"15min":0.66400000000000003},"min":{"1min":0.40000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.79800000000000004,"5min":0.90900000000000003,"15min":0.93200000000000005},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.67800000000000005,"15min":0.66500000000000004},"min":{"1min":0.45900000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90100000000000002,"5min":1.0109999999999999,"15min":1.024},"last":0.72399999999999998}]},{"osd":1,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67800000000000005,"15min":0.67100000000000004},"min":{"1min":0.46400000000000002,"5min":0.44600000000000001,"15min":0.39000000000000001},"max":{"1min":0.81899999999999995,"5min":0.89900000000000002,"15min":0.96499999999999997},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.625,"5min":0.67200000000000004,"15min":0.66200000000000003},"min":{"1min":0.379,"5min":0.35599999999999998,"15min":0.34699999999999998},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.84099999999999997}]},{"osd":2,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.71299999999999997,"15min":0.69699999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":1.127},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.76900000000000002,"15min":0.75},"min":{"1min":0.501,"5min":0.501,"15min":0.434},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":1.048}]},{"osd":3,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.69999999999999996,"15min":0.69399999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90700000000000003,"5min":1.026,"15min":1.274},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.72599999999999998,"15min":0.71699999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.89500000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.30199999999999999},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67900000000000005,"15min":0.68799999999999994},"min":{"1min":0.42199999999999999,"5min":0.40000000000000002,"15min":0.38100000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.4890000000000001},"last":1.0069999999999999}]},{"osd":5,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65000000000000002,"15min":0.64700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.68500000000000005,"15min":0.68899999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":1.026}]},{"osd":6,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.67200000000000004,"15min":0.65900000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.65500000000000003,"15min":0.65600000000000003},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39600000000000002},"max":{"1min":0.93799999999999994,"5min":1.0509999999999999,"15min":1.329},"last":0.93799999999999994}]}]},{"osd":6,"up_from":13,"seq":55834575009,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":782336,"data_stored":366235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.70199999999999996,"15min":0.68300000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.32400000000000001},"max":{"1min":0.90300000000000002,"5min":0.90600000000000003,"15min":0.98099999999999998},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.71699999999999997,"15min":0.69299999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.375},"max":{"1min":0.84899999999999998,"5min":0.95299999999999996,"15min":1.0249999999999999},"last":0.68799999999999994}]},{"osd":1,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.69399999999999995,"15min":0.68600000000000005},"min":{"1min":0.46500000000000002,"5min":0.41799999999999998,"15min":0.41199999999999998},"max":{"1min":0.80900000000000005,"5min":0.91600000000000004,"15min":1.085},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.70199999999999996,"15min":0.69799999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86599999999999999,"5min":0.93300000000000005,"15min":1.0009999999999999},"last":0.67700000000000005}]},{"osd":2,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.72599999999999998,"15min":0.69499999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.38200000000000001},"max":{"1min":0.84899999999999998,"5min":0.91200000000000003,"15min":1},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.51400000000000001,"5min":0.499,"15min":0.45900000000000002},"max":{"1min":0.85299999999999998,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.70299999999999996}]},{"osd":3,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.73099999999999998,"15min":0.72799999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.47099999999999997},"max":{"1min":0.89600000000000002,"5min":0.95399999999999996,"15min":0.98199999999999998},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.746,"15min":0.73199999999999998},"min":{"1min":0.52400000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.85899999999999999,"5min":0.995,"15min":1.0309999999999999},"last":0.64000000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.60899999999999999,"15min":0.60199999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.92500000000000004,"5min":0.92700000000000005,"15min":0.94599999999999995},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64700000000000002,"15min":0.63300000000000001},"min":{"1min":0.42799999999999999,"5min":0.39600000000000002,"15min":0.35499999999999998},"max":{"1min":0.89700000000000002,"5min":0.94099999999999995,"15min":1.0049999999999999},"last":0.75700000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63400000000000001,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.379,"15min":0.35799999999999998},"max":{"1min":0.82599999999999996,"5min":0.88200000000000001,"15min":1.1599999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68300000000000005,"15min":0.68400000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.40300000000000002},"max":{"1min":0.90400000000000003,"5min":0.97899999999999998,"15min":1.095},"last":0.60899999999999999}]},{"osd":7,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.69899999999999995,"15min":0.68500000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.69599999999999995,"15min":0.69999999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.42099999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.4399999999999999},"last":0.83599999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36556,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335284,"statfs":{"total":96636764160,"available":96599330816,"internally_reserved":0,"allocated":1372160,"data_stored":956057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.45900000000000002,"15min":0.46200000000000002},"min":{"1min":0.32400000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.73199999999999998,"5min":0.73299999999999998,"15min":2.9079999999999999},"last":0.315},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.34799999999999998,"5min":0.34100000000000003,"15min":0.313},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":2.766},"last":0.496}]},{"osd":2,"last update":"Tue Apr 2 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.38300000000000001,"5min":0.32200000000000001,"15min":0.30299999999999999},"max":{"1min":0.65100000000000002,"5min":0.71599999999999997,"15min":2.0750000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58499999999999996,"15min":0.57699999999999996},"min":{"1min":0.433,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":1.7410000000000001},"last":0.46700000000000003}]},{"osd":3,"last update":"Tue Apr 2 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.35999999999999999},"max":{"1min":0.71899999999999997,"5min":0.879,"15min":2.9430000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55600000000000005,"15min":0.56100000000000005},"min":{"1min":0.41999999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77500000000000002,"5min":0.85499999999999998,"15min":2.802},"last":0.42499999999999999}]},{"osd":4,"last update":"Tue Apr 2 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78900000000000003,"15min":0.77000000000000002},"min":{"1min":0.60599999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.93799999999999994,"5min":1.0860000000000001,"15min":1.7969999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78500000000000003,"15min":0.76600000000000001},"min":{"1min":0.61299999999999999,"5min":0.47799999999999998,"15min":0.46500000000000002},"max":{"1min":1.0089999999999999,"5min":1.135,"15min":2.0419999999999998},"last":0.77800000000000002}]},{"osd":5,"last update":"Tue Apr 2 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.79400000000000004,"15min":0.77000000000000002},"min":{"1min":0.61099999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.117,"5min":1.117,"15min":1.9690000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.77300000000000002},"min":{"1min":0.70299999999999996,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.96499999999999997,"5min":1.0920000000000001,"15min":1.3009999999999999},"last":0.748}]},{"osd":6,"last update":"Tue Apr 2 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.81999999999999995,"15min":0.79300000000000004},"min":{"1min":0.66200000000000003,"5min":0.52600000000000002,"15min":0.47799999999999998},"max":{"1min":1.0900000000000001,"5min":1.1919999999999999,"15min":1.768},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.81599999999999995,"15min":0.78700000000000003},"min":{"1min":0.61599999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.048,"5min":1.0820000000000001,"15min":2.012},"last":0.73099999999999998}]},{"osd":7,"last update":"Tue Apr 2 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81999999999999995,"15min":0.80300000000000005},"min":{"1min":0.62,"5min":0.56799999999999995,"15min":0.502},"max":{"1min":1.552,"5min":1.552,"15min":1.706},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.85099999999999998,"15min":0.81499999999999995},"min":{"1min":0.63900000000000001,"5min":0.55200000000000005,"15min":0.502},"max":{"1min":1.147,"5min":1.1519999999999999,"15min":1.8240000000000001},"last":0.66900000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575010,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":790528,"data_stored":371920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.434,"15min":0.45900000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":2.1640000000000001},"last":0.442},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.497,"15min":0.51800000000000002},"min":{"1min":0.32800000000000001,"5min":0.30499999999999999,"15min":0.26200000000000001},"max":{"1min":0.71299999999999997,"5min":0.79100000000000004,"15min":2.6469999999999998},"last":0.42599999999999999}]},{"osd":2,"last update":"Tue Apr 2 07:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.495,"15min":0.51300000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33000000000000002},"max":{"1min":0.68200000000000005,"5min":0.76300000000000001,"15min":2.125},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.54500000000000004,"15min":0.56100000000000005},"min":{"1min":0.40400000000000003,"5min":0.35199999999999998,"15min":0.33300000000000002},"max":{"1min":0.745,"5min":0.77800000000000002,"15min":2.605},"last":0.48599999999999999}]},{"osd":3,"last update":"Tue Apr 2 07:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.60099999999999998},"min":{"1min":0.377,"5min":0.34899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73999999999999999,"5min":0.79400000000000004,"15min":5.4800000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61899999999999999,"15min":0.63},"min":{"1min":0.44,"5min":0.44,"15min":0.40100000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":2.194},"last":0.55500000000000005}]},{"osd":4,"last update":"Tue Apr 2 07:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.80400000000000005,"15min":0.85999999999999999},"min":{"1min":0.60299999999999998,"5min":0.44800000000000001,"15min":0.35599999999999998},"max":{"1min":1.0169999999999999,"5min":1.181,"15min":6.0899999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.84899999999999998},"min":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.433},"max":{"1min":0.98799999999999999,"5min":1.331,"15min":5.7169999999999996},"last":0.96099999999999997}]},{"osd":5,"last update":"Tue Apr 2 07:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.81200000000000006,"15min":0.85899999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.42099999999999999},"max":{"1min":1.1779999999999999,"5min":1.282,"15min":5.7539999999999996},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81799999999999995,"15min":0.86199999999999999},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.51800000000000002},"max":{"1min":1.0349999999999999,"5min":1.2330000000000001,"15min":5.8929999999999998},"last":1.0169999999999999}]},{"osd":6,"last update":"Tue Apr 2 07:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76400000000000001,"15min":0.82099999999999995},"min":{"1min":0.61899999999999999,"5min":0.52900000000000003,"15min":0.46800000000000003},"max":{"1min":0.93100000000000005,"5min":1.1200000000000001,"15min":5.931},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78000000000000003,"15min":0.83799999999999997},"min":{"1min":0.53900000000000003,"5min":0.52000000000000002,"15min":0.376},"max":{"1min":0.97399999999999998,"5min":1.1339999999999999,"15min":6.125},"last":0.92200000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.81899999999999995,"15min":0.88},"min":{"1min":0.61899999999999999,"5min":0.54700000000000004,"15min":0.40799999999999997},"max":{"1min":1.0189999999999999,"5min":1.2170000000000001,"15min":6.1580000000000004},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.83999999999999997,"15min":0.89000000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.47699999999999998},"max":{"1min":1.1000000000000001,"5min":1.212,"15min":5.7850000000000001},"last":0.85099999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":790528,"data_stored":372357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.438,"15min":0.442},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.25700000000000001},"max":{"1min":0.55200000000000005,"5min":0.69099999999999995,"15min":0.79000000000000004},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.26200000000000001},"max":{"1min":0.63500000000000001,"5min":0.752,"15min":0.84299999999999997},"last":0.46600000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.49399999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30299999999999999},"max":{"1min":0.65600000000000003,"5min":0.72399999999999998,"15min":0.93999999999999995},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.52400000000000002,"15min":0.51800000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":0.57399999999999995,"5min":0.80300000000000005,"15min":0.90000000000000002},"last":0.41999999999999998}]},{"osd":3,"last update":"Tue Apr 2 07:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.27800000000000002},"max":{"1min":0.67900000000000005,"5min":0.79800000000000004,"15min":1.4339999999999999},"last":0.316},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.60299999999999998,"15min":0.61099999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.39000000000000001},"max":{"1min":0.72499999999999998,"5min":0.878,"15min":1.3740000000000001},"last":0.437}]},{"osd":4,"last update":"Tue Apr 2 07:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.78200000000000003,"15min":0.76400000000000001},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.47199999999999998},"max":{"1min":1.0049999999999999,"5min":1.1519999999999999,"15min":1.157},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.77400000000000002,"15min":0.76400000000000001},"min":{"1min":0.52300000000000002,"5min":0.51600000000000001,"15min":0.48499999999999999},"max":{"1min":0.92100000000000004,"5min":1.1479999999999999,"15min":1.179},"last":0.75600000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.77200000000000002,"15min":0.754},"min":{"1min":0.57199999999999995,"5min":0.55800000000000005,"15min":0.42199999999999999},"max":{"1min":0.90700000000000003,"5min":1.2,"15min":1.2},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.78100000000000003,"15min":0.77600000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.50900000000000001},"max":{"1min":0.91100000000000003,"5min":1.1080000000000001,"15min":1.1899999999999999},"last":0.74199999999999999}]},{"osd":6,"last update":"Tue Apr 2 07:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.80000000000000004,"15min":0.77800000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.442},"max":{"1min":0.97099999999999997,"5min":1.175,"15min":1.175},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.79000000000000004,"15min":0.78800000000000003},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.52100000000000002},"max":{"1min":0.89300000000000002,"5min":1.075,"15min":1.2130000000000001},"last":0.78800000000000003}]},{"osd":7,"last update":"Tue Apr 2 07:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.80300000000000005,"15min":0.78900000000000003},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.5},"max":{"1min":0.94399999999999995,"5min":1.1539999999999999,"15min":1.1779999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.81399999999999995,"15min":0.79400000000000004},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.52900000000000003},"max":{"1min":0.98899999999999999,"5min":1.1339999999999999,"15min":1.145},"last":0.91300000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575009,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35984,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335856,"statfs":{"total":96636764160,"available":96599916544,"internally_reserved":0,"allocated":786432,"data_stored":367183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.42099999999999999,"15min":0.45100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26000000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":2.476},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.45900000000000002,"15min":0.503},"min":{"1min":0.29699999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.51700000000000002,"5min":0.74399999999999999,"15min":2.04},"last":0.46100000000000002}]},{"osd":1,"last update":"Tue Apr 2 07:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.48699999999999999,"15min":0.50600000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":2.2839999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55500000000000005,"15min":0.56599999999999995},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.34200000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":2.2240000000000002},"last":0.52500000000000002}]},{"osd":2,"last update":"Tue Apr 2 07:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59999999999999998,"15min":0.61699999999999999},"min":{"1min":0.41499999999999998,"5min":0.39700000000000002,"15min":0.376},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":1.7310000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59799999999999998,"15min":0.64600000000000002},"min":{"1min":0.438,"5min":0.39700000000000002,"15min":0.35699999999999998},"max":{"1min":1.244,"5min":1.244,"15min":3.9449999999999998},"last":0.48399999999999999}]},{"osd":4,"last update":"Tue Apr 2 07:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.72399999999999998,"15min":0.76200000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.38400000000000001},"max":{"1min":0.93700000000000006,"5min":1.032,"15min":2.0030000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.749,"15min":0.76300000000000001},"min":{"1min":0.51600000000000001,"5min":0.503,"15min":0.47399999999999998},"max":{"1min":0.88800000000000001,"5min":1.02,"15min":1.5249999999999999},"last":0.63}]},{"osd":5,"last update":"Tue Apr 2 07:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.73499999999999999,"15min":0.751},"min":{"1min":0.48599999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.90100000000000002,"5min":1.0449999999999999,"15min":1.2909999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.74099999999999999,"15min":0.77700000000000002},"min":{"1min":0.48899999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85199999999999998,"5min":1.0449999999999999,"15min":1.9159999999999999},"last":0.68400000000000005}]},{"osd":6,"last update":"Tue Apr 2 07:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.746,"15min":0.76900000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.89700000000000002,"5min":1.0620000000000001,"15min":1.548},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.76500000000000001,"15min":0.77900000000000003},"min":{"1min":0.55100000000000005,"5min":0.54000000000000004,"15min":0.53500000000000003},"max":{"1min":0.92300000000000004,"5min":1.0329999999999999,"15min":1.4379999999999999},"last":0.65700000000000003}]},{"osd":7,"last update":"Tue Apr 2 07:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.76900000000000002,"15min":0.79900000000000004},"min":{"1min":0.55300000000000005,"5min":0.54600000000000004,"15min":0.439},"max":{"1min":0.97999999999999998,"5min":1.0369999999999999,"15min":1.966},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.77500000000000002,"15min":0.79400000000000004},"min":{"1min":0.499,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.181,"5min":1.181,"15min":1.514},"last":0.73899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35988,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335852,"statfs":{"total":96636764160,"available":96599912448,"internally_reserved":0,"allocated":794624,"data_stored":367275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.67600000000000005,"15min":0.65000000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.32100000000000001},"max":{"1min":0.872,"5min":0.94299999999999995,"15min":1.0189999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68200000000000005,"15min":0.66600000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.35299999999999998},"max":{"1min":0.81799999999999995,"5min":0.90100000000000002,"15min":1.7849999999999999},"last":0.58099999999999996}]},{"osd":1,"last update":"Tue Apr 2 07:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.73399999999999999,"15min":0.71499999999999997},"min":{"1min":0.54200000000000004,"5min":0.501,"15min":0.42699999999999999},"max":{"1min":0.90900000000000003,"5min":1.0349999999999999,"15min":1.036},"last":0.438},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.73699999999999999,"15min":0.72799999999999998},"min":{"1min":0.47299999999999998,"5min":0.46700000000000003,"15min":0.45500000000000002},"max":{"1min":0.82299999999999995,"5min":0.97899999999999998,"15min":0.99399999999999999},"last":0.67500000000000004}]},{"osd":2,"last update":"Tue Apr 2 07:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.69699999999999995,"15min":0.68500000000000005},"min":{"1min":0.46400000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.82999999999999996,"5min":1.0069999999999999,"15min":1.03},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.71399999999999997,"15min":0.71599999999999997},"min":{"1min":0.54600000000000004,"5min":0.50700000000000001,"15min":0.41899999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.6459999999999999},"last":0.61899999999999999}]},{"osd":3,"last update":"Tue Apr 2 07:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.72499999999999998,"15min":0.71699999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":0.95399999999999996,"5min":0.97699999999999998,"15min":1.002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.73399999999999999,"15min":0.71799999999999997},"min":{"1min":0.497,"5min":0.48799999999999999,"15min":0.41499999999999998},"max":{"1min":0.92500000000000004,"5min":0.96399999999999997,"15min":0.96999999999999997},"last":0.73699999999999999}]},{"osd":5,"last update":"Tue Apr 2 07:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56999999999999995,"15min":0.58999999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.76900000000000002,"5min":0.78500000000000003,"15min":1.034},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58599999999999997,"15min":0.60299999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.33700000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":1.0129999999999999},"last":0.68899999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59799999999999998,"15min":0.625},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.86099999999999999,"5min":0.93400000000000005,"15min":1.2589999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63900000000000001,"15min":0.66400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":1.0609999999999999},"last":0.80900000000000005}]},{"osd":7,"last update":"Tue Apr 2 07:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47699999999999998,"5min":0.432,"15min":0.38300000000000001},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":1.0680000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.74099999999999999,"15min":0.747},"min":{"1min":0.52800000000000002,"5min":0.434,"15min":0.434},"max":{"1min":0.999,"5min":1.008,"15min":1.502},"last":0.64700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575012,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1380352,"data_stored":961638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:49:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.68700000000000006,"15min":0.66600000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.85999999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.69899999999999995,"15min":0.68100000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.39500000000000002}]},{"osd":1,"last update":"Tue Apr 2 07:49:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69499999999999995,"15min":0.68400000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40200000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69799999999999995,"15min":0.68700000000000006},"min":{"1min":0.41499999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.41499999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:49:07 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.71399999999999997,"15min":0.70599999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.70899999999999996,"15min":0.70599999999999996},"min":{"1min":0.47399999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.59399999999999997}]},{"osd":3,"last update":"Tue Apr 2 07:49:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.73699999999999999,"15min":0.74299999999999999},"min":{"1min":0.45900000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.749,"5min":0.77400000000000002,"15min":0.75900000000000001},"min":{"1min":0.54600000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.63700000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:49:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.60799999999999998,"15min":0.61099999999999999},"min":{"1min":0.38400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63900000000000001,"15min":0.65600000000000003},"min":{"1min":0.42499999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":0.61099999999999999}]},{"osd":6,"last update":"Tue Apr 2 07:49:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.44700000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.83399999999999996,"5min":0.93400000000000005,"15min":1.107},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67600000000000005,"15min":0.67700000000000005},"min":{"1min":0.44900000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.622}]},{"osd":7,"last update":"Tue Apr 2 07:49:07 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.495,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.55400000000000005,"5min":0.434,"15min":0.434},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.70199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:49:11.629 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:49:11.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.853+0000 7feae0a00700 1 Processor -- start 2024-04-02T07:49:11.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.853+0000 7feae0a00700 1 -- start start 2024-04-02T07:49:11.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.854+0000 7feae0a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feadc0d05e0 0x7feadc0d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.854+0000 7feae0a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feadc0d0f80 con 0x7feadc15da50 2024-04-02T07:49:11.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.854+0000 7feae0a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feadc0d24e0 con 0x7feadc0b6990 2024-04-02T07:49:11.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.854+0000 7feae0a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feadc0b95d0 con 0x7feadc0d05e0 2024-04-02T07:49:11.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.855+0000 7fead9e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7feadc15da50 0x7feadc0ce730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51482/0 (socket says 172.21.15.89:51482) 2024-04-02T07:49:11.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.855+0000 7fead9e00700 1 -- 172.21.15.89:0/2755994212 learned_addr learned my addr 172.21.15.89:0/2755994212 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:11.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.855+0000 7fead3e00700 1 --2- 172.21.15.89:0/2755994212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feadc0d05e0 0x7feadc0d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.855+0000 7fead9400700 1 -- 172.21.15.89:0/2755994212 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324493290 0 0) 0x7feadc0d0f80 con 0x7feadc15da50 2024-04-02T07:49:11.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.855+0000 7fead9400700 1 -- 172.21.15.89:0/2755994212 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac00035b0 con 0x7feadc15da50 2024-04-02T07:49:11.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.855+0000 7fead9400700 1 -- 172.21.15.89:0/2755994212 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1299246854 0 0) 0x7feac00035b0 con 0x7feadc15da50 2024-04-02T07:49:11.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.855+0000 7fead9400700 1 -- 172.21.15.89:0/2755994212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feadc0d05e0 msgr2=0x7feadc0d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:11.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.855+0000 7fead9400700 1 --2- 172.21.15.89:0/2755994212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feadc0d05e0 0x7feadc0d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.855+0000 7fead9400700 1 -- 172.21.15.89:0/2755994212 >> v1:172.21.15.113:6789/0 conn(0x7feadc0b6990 legacy=0x7feadc0b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:11.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.855+0000 7fead9400700 1 -- 172.21.15.89:0/2755994212 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feadc0d27c0 con 0x7feadc15da50 2024-04-02T07:49:11.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.855+0000 7fead9400700 1 -- 172.21.15.89:0/2755994212 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7feacc001670 con 0x7feadc15da50 2024-04-02T07:49:11.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.855+0000 7fead9400700 1 -- 172.21.15.89:0/2755994212 >> v1:172.21.15.89:6789/0 conn(0x7feadc15da50 legacy=0x7feadc0ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:11.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.856+0000 7fead9400700 1 --2- 172.21.15.89:0/2755994212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feac0003a30 0x7feac0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.856+0000 7fead9400700 1 --2- 172.21.15.89:0/2755994212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feac0004850 0x7feac0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.856+0000 7fead9400700 1 --2- 172.21.15.89:0/2755994212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feadc0d05e0 0x7feac0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.856+0000 7fead9400700 1 -- 172.21.15.89:0/2755994212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac000bac0 con 0x7feac0003a30 2024-04-02T07:49:11.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.856+0000 7fead9400700 1 -- 172.21.15.89:0/2755994212 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac000bc30 con 0x7feac0004850 2024-04-02T07:49:11.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.856+0000 7fead9400700 1 -- 172.21.15.89:0/2755994212 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac000bdf0 con 0x7feadc0d05e0 2024-04-02T07:49:11.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.856+0000 7fead9e00700 1 --2- 172.21.15.89:0/2755994212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feac0003a30 0x7feac0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.856+0000 7fead3e00700 1 --2- 172.21.15.89:0/2755994212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feac0004850 0x7feac0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.856+0000 7feada800700 1 --2- 172.21.15.89:0/2755994212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feadc0d05e0 0x7feac0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.856+0000 7fead9e00700 1 -- 172.21.15.89:0/2755994212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feadc0d05e0 msgr2=0x7feac0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:11.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.856+0000 7fead9e00700 1 --2- 172.21.15.89:0/2755994212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feadc0d05e0 0x7feac0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.856+0000 7fead9e00700 1 -- 172.21.15.89:0/2755994212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feac0004850 msgr2=0x7feac0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:11.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.856+0000 7fead9e00700 1 --2- 172.21.15.89:0/2755994212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feac0004850 0x7feac0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.856+0000 7fead9e00700 1 -- 172.21.15.89:0/2755994212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac00035b0 con 0x7feac0003a30 2024-04-02T07:49:11.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.857+0000 7fead9e00700 1 --2- 172.21.15.89:0/2755994212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feac0003a30 0x7feac0003e50 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7feacc001eb0 tx=0x7feac0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d88a2bd3cc6195e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:11.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.857+0000 7fead9400700 1 -- 172.21.15.89:0/2755994212 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc007b70 con 0x7feac0003a30 2024-04-02T07:49:11.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.857+0000 7fead9400700 1 -- 172.21.15.89:0/2755994212 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feacc00ccf0 con 0x7feac0003a30 2024-04-02T07:49:11.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.857+0000 7fead9400700 1 -- 172.21.15.89:0/2755994212 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc005ba0 con 0x7feac0003a30 2024-04-02T07:49:11.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.857+0000 7feae0a00700 1 -- 172.21.15.89:0/2755994212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feac0003a30 msgr2=0x7feac0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:11.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.857+0000 7feae0a00700 1 --2- 172.21.15.89:0/2755994212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feac0003a30 0x7feac0003e50 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7feacc001eb0 tx=0x7feac0005880 comp rx=0 tx=0).stop 2024-04-02T07:49:11.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.857+0000 7feada800700 1 -- 172.21.15.89:0/2755994212 reap_dead start 2024-04-02T07:49:11.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.857+0000 7feae0a00700 1 -- 172.21.15.89:0/2755994212 shutdown_connections 2024-04-02T07:49:11.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.857+0000 7feae0a00700 1 -- 172.21.15.89:0/2755994212 >> 172.21.15.89:0/2755994212 conn(0x7feadc0666a0 msgr2=0x7feadc066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:11.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.857+0000 7feae0a00700 1 -- 172.21.15.89:0/2755994212 shutdown_connections 2024-04-02T07:49:11.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.857+0000 7feae0a00700 1 -- 172.21.15.89:0/2755994212 wait complete. 2024-04-02T07:49:11.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.859+0000 7feae0a00700 1 Processor -- start 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.859+0000 7feae0a00700 1 -- start start 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.860+0000 7feae0a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feac0003a30 0x7feadc1f6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.860+0000 7feae0a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feadc0c9020 0x7feadc147c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.860+0000 7feae0a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feadc0c94a0 0x7feadc155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.860+0000 7feae0a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feadc1f70d0 con 0x7feadc0c9020 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.860+0000 7feae0a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feadc1f3b10 con 0x7feadc0c94a0 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.860+0000 7feae0a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feadc1f3270 con 0x7feac0003a30 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.860+0000 7fead3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feadc0c9020 0x7feadc147c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.860+0000 7fead3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feadc0c9020 0x7feadc147c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53448/0 (socket says 172.21.15.89:53448) 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.860+0000 7fead3e00700 1 -- 172.21.15.89:0/4182078804 learned_addr learned my addr 172.21.15.89:0/4182078804 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.860+0000 7fead9e00700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feac0003a30 0x7feadc1f6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.861+0000 7feada800700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feadc0c94a0 0x7feadc155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.861+0000 7fead3e00700 1 -- 172.21.15.89:0/4182078804 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feac0003a30 msgr2=0x7feadc1f6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.861+0000 7fead3e00700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feac0003a30 0x7feadc1f6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.861+0000 7fead3e00700 1 -- 172.21.15.89:0/4182078804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feadc0c94a0 msgr2=0x7feadc155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:11.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.861+0000 7fead3e00700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feadc0c94a0 0x7feadc155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:11.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.861+0000 7fead3e00700 1 -- 172.21.15.89:0/4182078804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feadc1f3550 con 0x7feadc0c9020 2024-04-02T07:49:11.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.861+0000 7fead9e00700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feac0003a30 0x7feadc1f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:49:11.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.861+0000 7feada800700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feadc0c94a0 0x7feadc155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:49:11.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.861+0000 7fead3e00700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feadc0c9020 0x7feadc147c80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7feac40119c0 tx=0x7feac4011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:11.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.861+0000 7fead2000700 1 -- 172.21.15.89:0/4182078804 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac400f940 con 0x7feadc0c9020 2024-04-02T07:49:11.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.861+0000 7fead2000700 1 -- 172.21.15.89:0/4182078804 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac400faa0 con 0x7feadc0c9020 2024-04-02T07:49:11.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.861+0000 7fead2000700 1 -- 172.21.15.89:0/4182078804 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac400fe50 con 0x7feadc0c9020 2024-04-02T07:49:11.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.861+0000 7feae0a00700 1 -- 172.21.15.89:0/4182078804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feadc1f3830 con 0x7feadc0c9020 2024-04-02T07:49:11.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.861+0000 7feae0a00700 1 -- 172.21.15.89:0/4182078804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feadc1481c0 con 0x7feadc0c9020 2024-04-02T07:49:11.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.862+0000 7feae0a00700 1 -- 172.21.15.89:0/4182078804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feadc067720 con 0x7feadc0c9020 2024-04-02T07:49:11.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.863+0000 7fead2000700 1 -- 172.21.15.89:0/4182078804 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7feac4014440 con 0x7feadc0c9020 2024-04-02T07:49:11.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.865+0000 7fead2000700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feab806adb0 0x7feab806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:11.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.865+0000 7fead2000700 1 -- 172.21.15.89:0/4182078804 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7feac4089cf0 con 0x7feadc0c9020 2024-04-02T07:49:11.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.865+0000 7fead9e00700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feab806adb0 0x7feab806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:11.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.865+0000 7fead9e00700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feab806adb0 0x7feab806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feacc000c00 tx=0x7feacc007000 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:11.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:11.869+0000 7fead2000700 1 -- 172.21.15.89:0/4182078804 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feac4054c90 con 0x7feadc0c9020 2024-04-02T07:49:12.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.067+0000 7feae0a00700 1 -- 172.21.15.89:0/4182078804 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feadc0d3060 con 0x7feab806adb0 2024-04-02T07:49:12.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.075+0000 7fead2000700 1 -- 172.21.15.89:0/4182078804 <== mgr.5661 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289304 (secure 0 0 0) 0x7feacc023070 con 0x7feab806adb0 2024-04-02T07:49:12.081 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:49:12.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.080+0000 7feae0a00700 1 -- 172.21.15.89:0/4182078804 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feab806adb0 msgr2=0x7feab806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:12.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.080+0000 7feae0a00700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feab806adb0 0x7feab806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feacc000c00 tx=0x7feacc007000 comp rx=0 tx=0).stop 2024-04-02T07:49:12.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.080+0000 7feae0a00700 1 -- 172.21.15.89:0/4182078804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feadc0c9020 msgr2=0x7feadc147c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:12.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.080+0000 7feae0a00700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feadc0c9020 0x7feadc147c80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7feac40119c0 tx=0x7feac4011d80 comp rx=0 tx=0).stop 2024-04-02T07:49:12.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.080+0000 7feae0a00700 1 -- 172.21.15.89:0/4182078804 shutdown_connections 2024-04-02T07:49:12.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.080+0000 7feae0a00700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feab806adb0 0x7feab806d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.080+0000 7feae0a00700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feadc0c94a0 0x7feadc155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.080+0000 7feae0a00700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feac0003a30 0x7feadc1f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.080+0000 7feae0a00700 1 --2- 172.21.15.89:0/4182078804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feadc0c9020 0x7feadc147c80 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.080+0000 7feae0a00700 1 -- 172.21.15.89:0/4182078804 >> 172.21.15.89:0/4182078804 conn(0x7feadc0666a0 msgr2=0x7feadc0c7370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:12.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.080+0000 7feae0a00700 1 -- 172.21.15.89:0/4182078804 shutdown_connections 2024-04-02T07:49:12.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.080+0000 7feae0a00700 1 -- 172.21.15.89:0/4182078804 wait complete. 2024-04-02T07:49:12.085 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:49:12.099 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T07:49:11.141591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268552,"kb_used_data":7312,"kb_used_omap":12,"kb_used_meta":261107,"kb_avail":754706168,"statfs":{"total":773094113280,"available":772819116032,"internally_reserved":0,"allocated":7487488,"data_stored":4134966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":267374136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.398324"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721452+0000","last_change":"2024-04-02T07:49:07.276132+0000","last_active":"2024-04-02T07:49:08.721452+0000","last_peered":"2024-04-02T07:49:08.721452+0000","last_clean":"2024-04-02T07:49:08.721452+0000","last_became_active":"2024-04-02T07:49:07.275562+0000","last_became_peered":"2024-04-02T07:49:07.275562+0000","last_unstale":"2024-04-02T07:49:08.721452+0000","last_undegraded":"2024-04-02T07:49:08.721452+0000","last_fullsized":"2024-04-02T07:49:08.721452+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:48.734491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255598+0000","last_change":"2024-04-02T07:49:07.271520+0000","last_active":"2024-04-02T07:49:08.255598+0000","last_peered":"2024-04-02T07:49:08.255598+0000","last_clean":"2024-04-02T07:49:08.255598+0000","last_became_active":"2024-04-02T07:49:07.271249+0000","last_became_peered":"2024-04-02T07:49:07.271249+0000","last_unstale":"2024-04-02T07:49:08.255598+0000","last_undegraded":"2024-04-02T07:49:08.255598+0000","last_fullsized":"2024-04-02T07:49:08.255598+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:39:26.559000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720960+0000","last_change":"2024-04-02T07:49:07.283831+0000","last_active":"2024-04-02T07:49:08.720960+0000","last_peered":"2024-04-02T07:49:08.720960+0000","last_clean":"2024-04-02T07:49:08.720960+0000","last_became_active":"2024-04-02T07:49:07.282360+0000","last_became_peered":"2024-04-02T07:49:07.282360+0000","last_unstale":"2024-04-02T07:49:08.720960+0000","last_undegraded":"2024-04-02T07:49:08.720960+0000","last_fullsized":"2024-04-02T07:49:08.720960+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:33.794321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721558+0000","last_change":"2024-04-02T07:49:07.292520+0000","last_active":"2024-04-02T07:49:08.721558+0000","last_peered":"2024-04-02T07:49:08.721558+0000","last_clean":"2024-04-02T07:49:08.721558+0000","last_became_active":"2024-04-02T07:49:07.292201+0000","last_became_peered":"2024-04-02T07:49:07.292201+0000","last_unstale":"2024-04-02T07:49:08.721558+0000","last_undegraded":"2024-04-02T07:49:08.721558+0000","last_fullsized":"2024-04-02T07:49:08.721558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:21:40.105740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721518+0000","last_change":"2024-04-02T07:49:07.278990+0000","last_active":"2024-04-02T07:49:08.721518+0000","last_peered":"2024-04-02T07:49:08.721518+0000","last_clean":"2024-04-02T07:49:08.721518+0000","last_became_active":"2024-04-02T07:49:07.278696+0000","last_became_peered":"2024-04-02T07:49:07.278696+0000","last_unstale":"2024-04-02T07:49:08.721518+0000","last_undegraded":"2024-04-02T07:49:08.721518+0000","last_fullsized":"2024-04-02T07:49:08.721518+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:26.176308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720556+0000","last_change":"2024-04-02T07:49:07.277421+0000","last_active":"2024-04-02T07:49:08.720556+0000","last_peered":"2024-04-02T07:49:08.720556+0000","last_clean":"2024-04-02T07:49:08.720556+0000","last_became_active":"2024-04-02T07:49:07.277133+0000","last_became_peered":"2024-04-02T07:49:07.277133+0000","last_unstale":"2024-04-02T07:49:08.720556+0000","last_undegraded":"2024-04-02T07:49:08.720556+0000","last_fullsized":"2024-04-02T07:49:08.720556+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:41.434877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720335+0000","last_change":"2024-04-02T07:49:07.286456+0000","last_active":"2024-04-02T07:49:08.720335+0000","last_peered":"2024-04-02T07:49:08.720335+0000","last_clean":"2024-04-02T07:49:08.720335+0000","last_became_active":"2024-04-02T07:49:07.286109+0000","last_became_peered":"2024-04-02T07:49:07.286109+0000","last_unstale":"2024-04-02T07:49:08.720335+0000","last_undegraded":"2024-04-02T07:49:08.720335+0000","last_fullsized":"2024-04-02T07:49:08.720335+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:38:04.979730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720907+0000","last_change":"2024-04-02T07:49:07.276664+0000","last_active":"2024-04-02T07:49:08.720907+0000","last_peered":"2024-04-02T07:49:08.720907+0000","last_clean":"2024-04-02T07:49:08.720907+0000","last_became_active":"2024-04-02T07:49:07.276203+0000","last_became_peered":"2024-04-02T07:49:07.276203+0000","last_unstale":"2024-04-02T07:49:08.720907+0000","last_undegraded":"2024-04-02T07:49:08.720907+0000","last_fullsized":"2024-04-02T07:49:08.720907+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:48.734491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720850+0000","last_change":"2024-04-02T07:49:07.276619+0000","last_active":"2024-04-02T07:49:08.720850+0000","last_peered":"2024-04-02T07:49:08.720850+0000","last_clean":"2024-04-02T07:49:08.720850+0000","last_became_active":"2024-04-02T07:49:07.276274+0000","last_became_peered":"2024-04-02T07:49:07.276274+0000","last_unstale":"2024-04-02T07:49:08.720850+0000","last_undegraded":"2024-04-02T07:49:08.720850+0000","last_fullsized":"2024-04-02T07:49:08.720850+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:59:22.168507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.261297+0000","last_change":"2024-04-02T07:49:07.279836+0000","last_active":"2024-04-02T07:49:08.261297+0000","last_peered":"2024-04-02T07:49:08.261297+0000","last_clean":"2024-04-02T07:49:08.261297+0000","last_became_active":"2024-04-02T07:49:07.277770+0000","last_became_peered":"2024-04-02T07:49:07.277770+0000","last_unstale":"2024-04-02T07:49:08.261297+0000","last_undegraded":"2024-04-02T07:49:08.261297+0000","last_fullsized":"2024-04-02T07:49:08.261297+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:21:40.105740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721866+0000","last_change":"2024-04-02T07:49:07.279840+0000","last_active":"2024-04-02T07:49:08.721866+0000","last_peered":"2024-04-02T07:49:08.721866+0000","last_clean":"2024-04-02T07:49:08.721866+0000","last_became_active":"2024-04-02T07:49:07.279579+0000","last_became_peered":"2024-04-02T07:49:07.279579+0000","last_unstale":"2024-04-02T07:49:08.721866+0000","last_undegraded":"2024-04-02T07:49:08.721866+0000","last_fullsized":"2024-04-02T07:49:08.721866+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:46:51.306140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255255+0000","last_change":"2024-04-02T07:49:07.263112+0000","last_active":"2024-04-02T07:49:08.255255+0000","last_peered":"2024-04-02T07:49:08.255255+0000","last_clean":"2024-04-02T07:49:08.255255+0000","last_became_active":"2024-04-02T07:49:07.262831+0000","last_became_peered":"2024-04-02T07:49:07.262831+0000","last_unstale":"2024-04-02T07:49:08.255255+0000","last_undegraded":"2024-04-02T07:49:08.255255+0000","last_fullsized":"2024-04-02T07:49:08.255255+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:59:22.168507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721098+0000","last_change":"2024-04-02T07:49:07.289697+0000","last_active":"2024-04-02T07:49:08.721098+0000","last_peered":"2024-04-02T07:49:08.721098+0000","last_clean":"2024-04-02T07:49:08.721098+0000","last_became_active":"2024-04-02T07:49:07.289397+0000","last_became_peered":"2024-04-02T07:49:07.289397+0000","last_unstale":"2024-04-02T07:49:08.721098+0000","last_undegraded":"2024-04-02T07:49:08.721098+0000","last_fullsized":"2024-04-02T07:49:08.721098+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:39.433909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721472+0000","last_change":"2024-04-02T07:49:07.285828+0000","last_active":"2024-04-02T07:49:08.721472+0000","last_peered":"2024-04-02T07:49:08.721472+0000","last_clean":"2024-04-02T07:49:08.721472+0000","last_became_active":"2024-04-02T07:49:07.285523+0000","last_became_peered":"2024-04-02T07:49:07.285523+0000","last_unstale":"2024-04-02T07:49:08.721472+0000","last_undegraded":"2024-04-02T07:49:08.721472+0000","last_fullsized":"2024-04-02T07:49:08.721472+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:09:33.011128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720980+0000","last_change":"2024-04-02T07:49:07.285077+0000","last_active":"2024-04-02T07:49:08.720980+0000","last_peered":"2024-04-02T07:49:08.720980+0000","last_clean":"2024-04-02T07:49:08.720980+0000","last_became_active":"2024-04-02T07:49:07.284708+0000","last_became_peered":"2024-04-02T07:49:07.284708+0000","last_unstale":"2024-04-02T07:49:08.720980+0000","last_undegraded":"2024-04-02T07:49:08.720980+0000","last_fullsized":"2024-04-02T07:49:08.720980+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:54.506312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720404+0000","last_change":"2024-04-02T07:49:07.278701+0000","last_active":"2024-04-02T07:49:08.720404+0000","last_peered":"2024-04-02T07:49:08.720404+0000","last_clean":"2024-04-02T07:49:08.720404+0000","last_became_active":"2024-04-02T07:49:07.278403+0000","last_became_peered":"2024-04-02T07:49:07.278403+0000","last_unstale":"2024-04-02T07:49:08.720404+0000","last_undegraded":"2024-04-02T07:49:08.720404+0000","last_fullsized":"2024-04-02T07:49:08.720404+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:30:00.443550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255706+0000","last_change":"2024-04-02T07:49:07.269083+0000","last_active":"2024-04-02T07:49:08.255706+0000","last_peered":"2024-04-02T07:49:08.255706+0000","last_clean":"2024-04-02T07:49:08.255706+0000","last_became_active":"2024-04-02T07:49:07.268289+0000","last_became_peered":"2024-04-02T07:49:07.268289+0000","last_unstale":"2024-04-02T07:49:08.255706+0000","last_undegraded":"2024-04-02T07:49:08.255706+0000","last_fullsized":"2024-04-02T07:49:08.255706+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:41.434877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720906+0000","last_change":"2024-04-02T07:49:07.287962+0000","last_active":"2024-04-02T07:49:08.720906+0000","last_peered":"2024-04-02T07:49:08.720906+0000","last_clean":"2024-04-02T07:49:08.720906+0000","last_became_active":"2024-04-02T07:49:07.287732+0000","last_became_peered":"2024-04-02T07:49:07.287732+0000","last_unstale":"2024-04-02T07:49:08.720906+0000","last_undegraded":"2024-04-02T07:49:08.720906+0000","last_fullsized":"2024-04-02T07:49:08.720906+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:39:26.559000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721203+0000","last_change":"2024-04-02T07:49:07.277544+0000","last_active":"2024-04-02T07:49:08.721203+0000","last_peered":"2024-04-02T07:49:08.721203+0000","last_clean":"2024-04-02T07:49:08.721203+0000","last_became_active":"2024-04-02T07:49:07.277286+0000","last_became_peered":"2024-04-02T07:49:07.277286+0000","last_unstale":"2024-04-02T07:49:08.721203+0000","last_undegraded":"2024-04-02T07:49:08.721203+0000","last_fullsized":"2024-04-02T07:49:08.721203+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:06.730666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.261187+0000","last_change":"2024-04-02T07:49:07.268066+0000","last_active":"2024-04-02T07:49:08.261187+0000","last_peered":"2024-04-02T07:49:08.261187+0000","last_clean":"2024-04-02T07:49:08.261187+0000","last_became_active":"2024-04-02T07:49:07.267634+0000","last_became_peered":"2024-04-02T07:49:07.267634+0000","last_unstale":"2024-04-02T07:49:08.261187+0000","last_undegraded":"2024-04-02T07:49:08.261187+0000","last_fullsized":"2024-04-02T07:49:08.261187+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:14.047532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721605+0000","last_change":"2024-04-02T07:49:07.291933+0000","last_active":"2024-04-02T07:49:08.721605+0000","last_peered":"2024-04-02T07:49:08.721605+0000","last_clean":"2024-04-02T07:49:08.721605+0000","last_became_active":"2024-04-02T07:49:07.291513+0000","last_became_peered":"2024-04-02T07:49:07.291513+0000","last_unstale":"2024-04-02T07:49:08.721605+0000","last_undegraded":"2024-04-02T07:49:08.721605+0000","last_fullsized":"2024-04-02T07:49:08.721605+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:07:49.813861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260240+0000","last_change":"2024-04-02T07:49:07.267328+0000","last_active":"2024-04-02T07:49:08.260240+0000","last_peered":"2024-04-02T07:49:08.260240+0000","last_clean":"2024-04-02T07:49:08.260240+0000","last_became_active":"2024-04-02T07:49:07.267021+0000","last_became_peered":"2024-04-02T07:49:07.267021+0000","last_unstale":"2024-04-02T07:49:08.260240+0000","last_undegraded":"2024-04-02T07:49:08.260240+0000","last_fullsized":"2024-04-02T07:49:08.260240+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:43.126776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720491+0000","last_change":"2024-04-02T07:49:07.270666+0000","last_active":"2024-04-02T07:49:08.720491+0000","last_peered":"2024-04-02T07:49:08.720491+0000","last_clean":"2024-04-02T07:49:08.720491+0000","last_became_active":"2024-04-02T07:49:07.268707+0000","last_became_peered":"2024-04-02T07:49:07.268707+0000","last_unstale":"2024-04-02T07:49:08.720491+0000","last_undegraded":"2024-04-02T07:49:08.720491+0000","last_fullsized":"2024-04-02T07:49:08.720491+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:37:02.063955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255051+0000","last_change":"2024-04-02T07:49:07.285534+0000","last_active":"2024-04-02T07:49:08.255051+0000","last_peered":"2024-04-02T07:49:08.255051+0000","last_clean":"2024-04-02T07:49:08.255051+0000","last_became_active":"2024-04-02T07:49:07.285267+0000","last_became_peered":"2024-04-02T07:49:07.285267+0000","last_unstale":"2024-04-02T07:49:08.255051+0000","last_undegraded":"2024-04-02T07:49:08.255051+0000","last_fullsized":"2024-04-02T07:49:08.255051+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:55:05.885676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721540+0000","last_change":"2024-04-02T07:49:07.266772+0000","last_active":"2024-04-02T07:49:08.721540+0000","last_peered":"2024-04-02T07:49:08.721540+0000","last_clean":"2024-04-02T07:49:08.721540+0000","last_became_active":"2024-04-02T07:49:07.266510+0000","last_became_peered":"2024-04-02T07:49:07.266510+0000","last_unstale":"2024-04-02T07:49:08.721540+0000","last_undegraded":"2024-04-02T07:49:08.721540+0000","last_fullsized":"2024-04-02T07:49:08.721540+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:54.506312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721650+0000","last_change":"2024-04-02T07:49:07.292024+0000","last_active":"2024-04-02T07:49:08.721650+0000","last_peered":"2024-04-02T07:49:08.721650+0000","last_clean":"2024-04-02T07:49:08.721650+0000","last_became_active":"2024-04-02T07:49:07.291417+0000","last_became_peered":"2024-04-02T07:49:07.291417+0000","last_unstale":"2024-04-02T07:49:08.721650+0000","last_undegraded":"2024-04-02T07:49:08.721650+0000","last_fullsized":"2024-04-02T07:49:08.721650+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:14.047532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721158+0000","last_change":"2024-04-02T07:49:07.267529+0000","last_active":"2024-04-02T07:49:08.721158+0000","last_peered":"2024-04-02T07:49:08.721158+0000","last_clean":"2024-04-02T07:49:08.721158+0000","last_became_active":"2024-04-02T07:49:07.267239+0000","last_became_peered":"2024-04-02T07:49:07.267239+0000","last_unstale":"2024-04-02T07:49:08.721158+0000","last_undegraded":"2024-04-02T07:49:08.721158+0000","last_fullsized":"2024-04-02T07:49:08.721158+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:06:37.929045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720830+0000","last_change":"2024-04-02T07:49:07.264297+0000","last_active":"2024-04-02T07:49:08.720830+0000","last_peered":"2024-04-02T07:49:08.720830+0000","last_clean":"2024-04-02T07:49:08.720830+0000","last_became_active":"2024-04-02T07:49:07.264002+0000","last_became_peered":"2024-04-02T07:49:07.264002+0000","last_unstale":"2024-04-02T07:49:08.720830+0000","last_undegraded":"2024-04-02T07:49:08.720830+0000","last_fullsized":"2024-04-02T07:49:08.720830+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:57:08.130262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721631+0000","last_change":"2024-04-02T07:49:07.267326+0000","last_active":"2024-04-02T07:49:08.721631+0000","last_peered":"2024-04-02T07:49:08.721631+0000","last_clean":"2024-04-02T07:49:08.721631+0000","last_became_active":"2024-04-02T07:49:07.267026+0000","last_became_peered":"2024-04-02T07:49:07.267026+0000","last_unstale":"2024-04-02T07:49:08.721631+0000","last_undegraded":"2024-04-02T07:49:08.721631+0000","last_fullsized":"2024-04-02T07:49:08.721631+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:15.424782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255104+0000","last_change":"2024-04-02T07:49:07.264874+0000","last_active":"2024-04-02T07:49:08.255104+0000","last_peered":"2024-04-02T07:49:08.255104+0000","last_clean":"2024-04-02T07:49:08.255104+0000","last_became_active":"2024-04-02T07:49:07.263663+0000","last_became_peered":"2024-04-02T07:49:07.263663+0000","last_unstale":"2024-04-02T07:49:08.255104+0000","last_undegraded":"2024-04-02T07:49:08.255104+0000","last_fullsized":"2024-04-02T07:49:08.255104+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:42:43.149354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720826+0000","last_change":"2024-04-02T07:49:07.280144+0000","last_active":"2024-04-02T07:49:08.720826+0000","last_peered":"2024-04-02T07:49:08.720826+0000","last_clean":"2024-04-02T07:49:08.720826+0000","last_became_active":"2024-04-02T07:49:07.279063+0000","last_became_peered":"2024-04-02T07:49:07.279063+0000","last_unstale":"2024-04-02T07:49:08.720826+0000","last_undegraded":"2024-04-02T07:49:08.720826+0000","last_fullsized":"2024-04-02T07:49:08.720826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:30.069211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720907+0000","last_change":"2024-04-02T07:49:07.288423+0000","last_active":"2024-04-02T07:49:08.720907+0000","last_peered":"2024-04-02T07:49:08.720907+0000","last_clean":"2024-04-02T07:49:08.720907+0000","last_became_active":"2024-04-02T07:49:07.283287+0000","last_became_peered":"2024-04-02T07:49:07.283287+0000","last_unstale":"2024-04-02T07:49:08.720907+0000","last_undegraded":"2024-04-02T07:49:08.720907+0000","last_fullsized":"2024-04-02T07:49:08.720907+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:21.336897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260861+0000","last_change":"2024-04-02T07:49:07.271024+0000","last_active":"2024-04-02T07:49:08.260861+0000","last_peered":"2024-04-02T07:49:08.260861+0000","last_clean":"2024-04-02T07:49:08.260861+0000","last_became_active":"2024-04-02T07:49:07.270736+0000","last_became_peered":"2024-04-02T07:49:07.270736+0000","last_unstale":"2024-04-02T07:49:08.260861+0000","last_undegraded":"2024-04-02T07:49:08.260861+0000","last_fullsized":"2024-04-02T07:49:08.260861+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:08:24.551656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"65'4","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720850+0000","last_change":"2024-04-02T07:49:06.775388+0000","last_active":"2024-04-02T07:49:08.720850+0000","last_peered":"2024-04-02T07:49:08.720850+0000","last_clean":"2024-04-02T07:49:08.720850+0000","last_became_active":"2024-04-02T07:49:06.775096+0000","last_became_peered":"2024-04-02T07:49:06.775096+0000","last_unstale":"2024-04-02T07:49:08.720850+0000","last_undegraded":"2024-04-02T07:49:08.720850+0000","last_fullsized":"2024-04-02T07:49:08.720850+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:47:15.729728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721544+0000","last_change":"2024-04-02T07:49:07.279068+0000","last_active":"2024-04-02T07:49:08.721544+0000","last_peered":"2024-04-02T07:49:08.721544+0000","last_clean":"2024-04-02T07:49:08.721544+0000","last_became_active":"2024-04-02T07:49:07.278777+0000","last_became_peered":"2024-04-02T07:49:07.278777+0000","last_unstale":"2024-04-02T07:49:08.721544+0000","last_undegraded":"2024-04-02T07:49:08.721544+0000","last_fullsized":"2024-04-02T07:49:08.721544+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:21.336897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"65'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721502+0000","last_change":"2024-04-02T07:49:06.772305+0000","last_active":"2024-04-02T07:49:08.721502+0000","last_peered":"2024-04-02T07:49:08.721502+0000","last_clean":"2024-04-02T07:49:08.721502+0000","last_became_active":"2024-04-02T07:49:06.772023+0000","last_became_peered":"2024-04-02T07:49:06.772023+0000","last_unstale":"2024-04-02T07:49:08.721502+0000","last_undegraded":"2024-04-02T07:49:08.721502+0000","last_fullsized":"2024-04-02T07:49:08.721502+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:05.723370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255279+0000","last_change":"2024-04-02T07:49:07.269722+0000","last_active":"2024-04-02T07:49:08.255279+0000","last_peered":"2024-04-02T07:49:08.255279+0000","last_clean":"2024-04-02T07:49:08.255279+0000","last_became_active":"2024-04-02T07:49:07.269464+0000","last_became_peered":"2024-04-02T07:49:07.269464+0000","last_unstale":"2024-04-02T07:49:08.255279+0000","last_undegraded":"2024-04-02T07:49:08.255279+0000","last_fullsized":"2024-04-02T07:49:08.255279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:43.126776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"65'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720765+0000","last_change":"2024-04-02T07:49:06.770671+0000","last_active":"2024-04-02T07:49:08.720765+0000","last_peered":"2024-04-02T07:49:08.720765+0000","last_clean":"2024-04-02T07:49:08.720765+0000","last_became_active":"2024-04-02T07:49:06.770402+0000","last_became_peered":"2024-04-02T07:49:06.770402+0000","last_unstale":"2024-04-02T07:49:08.720765+0000","last_undegraded":"2024-04-02T07:49:08.720765+0000","last_fullsized":"2024-04-02T07:49:08.720765+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:46:46.772743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721885+0000","last_change":"2024-04-02T07:49:07.285066+0000","last_active":"2024-04-02T07:49:08.721885+0000","last_peered":"2024-04-02T07:49:08.721885+0000","last_clean":"2024-04-02T07:49:08.721885+0000","last_became_active":"2024-04-02T07:49:07.284773+0000","last_became_peered":"2024-04-02T07:49:07.284773+0000","last_unstale":"2024-04-02T07:49:08.721885+0000","last_undegraded":"2024-04-02T07:49:08.721885+0000","last_fullsized":"2024-04-02T07:49:08.721885+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:37.672940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720911+0000","last_change":"2024-04-02T07:49:06.775799+0000","last_active":"2024-04-02T07:49:08.720911+0000","last_peered":"2024-04-02T07:49:08.720911+0000","last_clean":"2024-04-02T07:49:08.720911+0000","last_became_active":"2024-04-02T07:49:06.775519+0000","last_became_peered":"2024-04-02T07:49:06.775519+0000","last_unstale":"2024-04-02T07:49:08.720911+0000","last_undegraded":"2024-04-02T07:49:08.720911+0000","last_fullsized":"2024-04-02T07:49:08.720911+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:25.169012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720618+0000","last_change":"2024-04-02T07:49:07.267601+0000","last_active":"2024-04-02T07:49:08.720618+0000","last_peered":"2024-04-02T07:49:08.720618+0000","last_clean":"2024-04-02T07:49:08.720618+0000","last_became_active":"2024-04-02T07:49:07.267348+0000","last_became_peered":"2024-04-02T07:49:07.267348+0000","last_unstale":"2024-04-02T07:49:08.720618+0000","last_undegraded":"2024-04-02T07:49:08.720618+0000","last_fullsized":"2024-04-02T07:49:08.720618+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:27:21.879339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"65'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721159+0000","last_change":"2024-04-02T07:49:06.771720+0000","last_active":"2024-04-02T07:49:08.721159+0000","last_peered":"2024-04-02T07:49:08.721159+0000","last_clean":"2024-04-02T07:49:08.721159+0000","last_became_active":"2024-04-02T07:49:06.771156+0000","last_became_peered":"2024-04-02T07:49:06.771156+0000","last_unstale":"2024-04-02T07:49:08.721159+0000","last_undegraded":"2024-04-02T07:49:08.721159+0000","last_fullsized":"2024-04-02T07:49:08.721159+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:12.571159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260800+0000","last_change":"2024-04-02T07:49:07.292076+0000","last_active":"2024-04-02T07:49:08.260800+0000","last_peered":"2024-04-02T07:49:08.260800+0000","last_clean":"2024-04-02T07:49:08.260800+0000","last_became_active":"2024-04-02T07:49:07.290712+0000","last_became_peered":"2024-04-02T07:49:07.290712+0000","last_unstale":"2024-04-02T07:49:08.260800+0000","last_undegraded":"2024-04-02T07:49:08.260800+0000","last_fullsized":"2024-04-02T07:49:08.260800+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:40:57.913800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"65'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720725+0000","last_change":"2024-04-02T07:49:06.777869+0000","last_active":"2024-04-02T07:49:08.720725+0000","last_peered":"2024-04-02T07:49:08.720725+0000","last_clean":"2024-04-02T07:49:08.720725+0000","last_became_active":"2024-04-02T07:49:06.777599+0000","last_became_peered":"2024-04-02T07:49:06.777599+0000","last_unstale":"2024-04-02T07:49:08.720725+0000","last_undegraded":"2024-04-02T07:49:08.720725+0000","last_fullsized":"2024-04-02T07:49:08.720725+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:53.499016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260318+0000","last_change":"2024-04-02T07:49:07.273326+0000","last_active":"2024-04-02T07:49:08.260318+0000","last_peered":"2024-04-02T07:49:08.260318+0000","last_clean":"2024-04-02T07:49:08.260318+0000","last_became_active":"2024-04-02T07:49:07.272995+0000","last_became_peered":"2024-04-02T07:49:07.272995+0000","last_unstale":"2024-04-02T07:49:08.260318+0000","last_undegraded":"2024-04-02T07:49:08.260318+0000","last_fullsized":"2024-04-02T07:49:08.260318+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:34:00.481993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721088+0000","last_change":"2024-04-02T07:49:06.769138+0000","last_active":"2024-04-02T07:49:08.721088+0000","last_peered":"2024-04-02T07:49:08.721088+0000","last_clean":"2024-04-02T07:49:08.721088+0000","last_became_active":"2024-04-02T07:49:06.768833+0000","last_became_peered":"2024-04-02T07:49:06.768833+0000","last_unstale":"2024-04-02T07:49:08.721088+0000","last_undegraded":"2024-04-02T07:49:08.721088+0000","last_fullsized":"2024-04-02T07:49:08.721088+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:59:21.161210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720582+0000","last_change":"2024-04-02T07:49:07.273900+0000","last_active":"2024-04-02T07:49:08.720582+0000","last_peered":"2024-04-02T07:49:08.720582+0000","last_clean":"2024-04-02T07:49:08.720582+0000","last_became_active":"2024-04-02T07:49:07.273645+0000","last_became_peered":"2024-04-02T07:49:07.273645+0000","last_unstale":"2024-04-02T07:49:08.720582+0000","last_undegraded":"2024-04-02T07:49:08.720582+0000","last_fullsized":"2024-04-02T07:49:08.720582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:34.296961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260767+0000","last_change":"2024-04-02T07:49:06.773089+0000","last_active":"2024-04-02T07:49:08.260767+0000","last_peered":"2024-04-02T07:49:08.260767+0000","last_clean":"2024-04-02T07:49:08.260767+0000","last_became_active":"2024-04-02T07:49:06.772754+0000","last_became_peered":"2024-04-02T07:49:06.772754+0000","last_unstale":"2024-04-02T07:49:08.260767+0000","last_undegraded":"2024-04-02T07:49:08.260767+0000","last_fullsized":"2024-04-02T07:49:08.260767+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:31.991732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721009+0000","last_change":"2024-04-02T07:49:07.276475+0000","last_active":"2024-04-02T07:49:08.721009+0000","last_peered":"2024-04-02T07:49:08.721009+0000","last_clean":"2024-04-02T07:49:08.721009+0000","last_became_active":"2024-04-02T07:49:07.275708+0000","last_became_peered":"2024-04-02T07:49:07.275708+0000","last_unstale":"2024-04-02T07:49:08.721009+0000","last_undegraded":"2024-04-02T07:49:08.721009+0000","last_fullsized":"2024-04-02T07:49:08.721009+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:30.069211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720987+0000","last_change":"2024-04-02T07:49:06.773667+0000","last_active":"2024-04-02T07:49:08.720987+0000","last_peered":"2024-04-02T07:49:08.720987+0000","last_clean":"2024-04-02T07:49:08.720987+0000","last_became_active":"2024-04-02T07:49:06.773195+0000","last_became_peered":"2024-04-02T07:49:06.773195+0000","last_unstale":"2024-04-02T07:49:08.720987+0000","last_undegraded":"2024-04-02T07:49:08.720987+0000","last_fullsized":"2024-04-02T07:49:08.720987+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:12.571159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720901+0000","last_change":"2024-04-02T07:49:07.274350+0000","last_active":"2024-04-02T07:49:08.720901+0000","last_peered":"2024-04-02T07:49:08.720901+0000","last_clean":"2024-04-02T07:49:08.720901+0000","last_became_active":"2024-04-02T07:49:07.274095+0000","last_became_peered":"2024-04-02T07:49:07.274095+0000","last_unstale":"2024-04-02T07:49:08.720901+0000","last_undegraded":"2024-04-02T07:49:08.720901+0000","last_fullsized":"2024-04-02T07:49:08.720901+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:59:22.168507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.261288+0000","last_change":"2024-04-02T07:49:06.772056+0000","last_active":"2024-04-02T07:49:08.261288+0000","last_peered":"2024-04-02T07:49:08.261288+0000","last_clean":"2024-04-02T07:49:08.261288+0000","last_became_active":"2024-04-02T07:49:06.769275+0000","last_became_peered":"2024-04-02T07:49:06.769275+0000","last_unstale":"2024-04-02T07:49:08.261288+0000","last_undegraded":"2024-04-02T07:49:08.261288+0000","last_fullsized":"2024-04-02T07:49:08.261288+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:36:38.777298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721571+0000","last_change":"2024-04-02T07:49:07.292660+0000","last_active":"2024-04-02T07:49:08.721571+0000","last_peered":"2024-04-02T07:49:08.721571+0000","last_clean":"2024-04-02T07:49:08.721571+0000","last_became_active":"2024-04-02T07:49:07.292445+0000","last_became_peered":"2024-04-02T07:49:07.292445+0000","last_unstale":"2024-04-02T07:49:08.721571+0000","last_undegraded":"2024-04-02T07:49:08.721571+0000","last_fullsized":"2024-04-02T07:49:08.721571+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:23:30.112352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"65'3","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255375+0000","last_change":"2024-04-02T07:49:06.770296+0000","last_active":"2024-04-02T07:49:08.255375+0000","last_peered":"2024-04-02T07:49:08.255375+0000","last_clean":"2024-04-02T07:49:08.255375+0000","last_became_active":"2024-04-02T07:49:06.767492+0000","last_became_peered":"2024-04-02T07:49:06.767492+0000","last_unstale":"2024-04-02T07:49:08.255375+0000","last_undegraded":"2024-04-02T07:49:08.255375+0000","last_fullsized":"2024-04-02T07:49:08.255375+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:59.964498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721245+0000","last_change":"2024-04-02T07:49:07.287153+0000","last_active":"2024-04-02T07:49:08.721245+0000","last_peered":"2024-04-02T07:49:08.721245+0000","last_clean":"2024-04-02T07:49:08.721245+0000","last_became_active":"2024-04-02T07:49:07.285894+0000","last_became_peered":"2024-04-02T07:49:07.285894+0000","last_unstale":"2024-04-02T07:49:08.721245+0000","last_undegraded":"2024-04-02T07:49:08.721245+0000","last_fullsized":"2024-04-02T07:49:08.721245+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:08.988202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"65'6","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260930+0000","last_change":"2024-04-02T07:49:06.774244+0000","last_active":"2024-04-02T07:49:08.260930+0000","last_peered":"2024-04-02T07:49:08.260930+0000","last_clean":"2024-04-02T07:49:08.260930+0000","last_became_active":"2024-04-02T07:49:06.773643+0000","last_became_peered":"2024-04-02T07:49:06.773643+0000","last_unstale":"2024-04-02T07:49:08.260930+0000","last_undegraded":"2024-04-02T07:49:08.260930+0000","last_fullsized":"2024-04-02T07:49:08.260930+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:18:56.421172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720695+0000","last_change":"2024-04-02T07:49:07.287960+0000","last_active":"2024-04-02T07:49:08.720695+0000","last_peered":"2024-04-02T07:49:08.720695+0000","last_clean":"2024-04-02T07:49:08.720695+0000","last_became_active":"2024-04-02T07:49:07.286839+0000","last_became_peered":"2024-04-02T07:49:07.286839+0000","last_unstale":"2024-04-02T07:49:08.720695+0000","last_undegraded":"2024-04-02T07:49:08.720695+0000","last_fullsized":"2024-04-02T07:49:08.720695+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:39.433909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"65'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260497+0000","last_change":"2024-04-02T07:49:06.766693+0000","last_active":"2024-04-02T07:49:08.260497+0000","last_peered":"2024-04-02T07:49:08.260497+0000","last_clean":"2024-04-02T07:49:08.260497+0000","last_became_active":"2024-04-02T07:49:06.766422+0000","last_became_peered":"2024-04-02T07:49:06.766422+0000","last_unstale":"2024-04-02T07:49:08.260497+0000","last_undegraded":"2024-04-02T07:49:08.260497+0000","last_fullsized":"2024-04-02T07:49:08.260497+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:14:44.817850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260294+0000","last_change":"2024-04-02T07:49:07.266844+0000","last_active":"2024-04-02T07:49:08.260294+0000","last_peered":"2024-04-02T07:49:08.260294+0000","last_clean":"2024-04-02T07:49:08.260294+0000","last_became_active":"2024-04-02T07:49:07.266526+0000","last_became_peered":"2024-04-02T07:49:07.266526+0000","last_unstale":"2024-04-02T07:49:08.260294+0000","last_undegraded":"2024-04-02T07:49:08.260294+0000","last_fullsized":"2024-04-02T07:49:08.260294+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:01:57.567763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721154+0000","last_change":"2024-04-02T07:49:06.763592+0000","last_active":"2024-04-02T07:49:08.721154+0000","last_peered":"2024-04-02T07:49:08.721154+0000","last_clean":"2024-04-02T07:49:08.721154+0000","last_became_active":"2024-04-02T07:49:06.763337+0000","last_became_peered":"2024-04-02T07:49:06.763337+0000","last_unstale":"2024-04-02T07:49:08.721154+0000","last_undegraded":"2024-04-02T07:49:08.721154+0000","last_fullsized":"2024-04-02T07:49:08.721154+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:39:25.551704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.259960+0000","last_change":"2024-04-02T07:49:07.266589+0000","last_active":"2024-04-02T07:49:08.259960+0000","last_peered":"2024-04-02T07:49:08.259960+0000","last_clean":"2024-04-02T07:49:08.259960+0000","last_became_active":"2024-04-02T07:49:07.266103+0000","last_became_peered":"2024-04-02T07:49:07.266103+0000","last_unstale":"2024-04-02T07:49:08.259960+0000","last_undegraded":"2024-04-02T07:49:08.259960+0000","last_fullsized":"2024-04-02T07:49:08.259960+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:30.369003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720914+0000","last_change":"2024-04-02T07:49:06.774807+0000","last_active":"2024-04-02T07:49:08.720914+0000","last_peered":"2024-04-02T07:49:08.720914+0000","last_clean":"2024-04-02T07:49:08.720914+0000","last_became_active":"2024-04-02T07:49:06.774537+0000","last_became_peered":"2024-04-02T07:49:06.774537+0000","last_unstale":"2024-04-02T07:49:08.720914+0000","last_undegraded":"2024-04-02T07:49:08.720914+0000","last_fullsized":"2024-04-02T07:49:08.720914+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:35.405121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255054+0000","last_change":"2024-04-02T07:49:07.268689+0000","last_active":"2024-04-02T07:49:08.255054+0000","last_peered":"2024-04-02T07:49:08.255054+0000","last_clean":"2024-04-02T07:49:08.255054+0000","last_became_active":"2024-04-02T07:49:07.268408+0000","last_became_peered":"2024-04-02T07:49:07.268408+0000","last_unstale":"2024-04-02T07:49:08.255054+0000","last_undegraded":"2024-04-02T07:49:08.255054+0000","last_fullsized":"2024-04-02T07:49:08.255054+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:30.069211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"65'8","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260881+0000","last_change":"2024-04-02T07:49:06.772071+0000","last_active":"2024-04-02T07:49:08.260881+0000","last_peered":"2024-04-02T07:49:08.260881+0000","last_clean":"2024-04-02T07:49:08.260881+0000","last_became_active":"2024-04-02T07:49:06.765463+0000","last_became_peered":"2024-04-02T07:49:06.765463+0000","last_unstale":"2024-04-02T07:49:08.260881+0000","last_undegraded":"2024-04-02T07:49:08.260881+0000","last_fullsized":"2024-04-02T07:49:08.260881+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:24:42.052205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260953+0000","last_change":"2024-04-02T07:49:07.267320+0000","last_active":"2024-04-02T07:49:08.260953+0000","last_peered":"2024-04-02T07:49:08.260953+0000","last_clean":"2024-04-02T07:49:08.260953+0000","last_became_active":"2024-04-02T07:49:07.266855+0000","last_became_peered":"2024-04-02T07:49:07.266855+0000","last_unstale":"2024-04-02T07:49:08.260953+0000","last_undegraded":"2024-04-02T07:49:08.260953+0000","last_fullsized":"2024-04-02T07:49:08.260953+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:07.809075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"65'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721817+0000","last_change":"2024-04-02T07:49:06.776827+0000","last_active":"2024-04-02T07:49:08.721817+0000","last_peered":"2024-04-02T07:49:08.721817+0000","last_clean":"2024-04-02T07:49:08.721817+0000","last_became_active":"2024-04-02T07:49:06.776535+0000","last_became_peered":"2024-04-02T07:49:06.776535+0000","last_unstale":"2024-04-02T07:49:08.721817+0000","last_undegraded":"2024-04-02T07:49:08.721817+0000","last_fullsized":"2024-04-02T07:49:08.721817+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:06.801779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260600+0000","last_change":"2024-04-02T07:49:07.271937+0000","last_active":"2024-04-02T07:49:08.260600+0000","last_peered":"2024-04-02T07:49:08.260600+0000","last_clean":"2024-04-02T07:49:08.260600+0000","last_became_active":"2024-04-02T07:49:07.269243+0000","last_became_peered":"2024-04-02T07:49:07.269243+0000","last_unstale":"2024-04-02T07:49:08.260600+0000","last_undegraded":"2024-04-02T07:49:08.260600+0000","last_fullsized":"2024-04-02T07:49:08.260600+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:10:05.251356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"65'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721092+0000","last_change":"2024-04-02T07:49:06.772431+0000","last_active":"2024-04-02T07:49:08.721092+0000","last_peered":"2024-04-02T07:49:08.721092+0000","last_clean":"2024-04-02T07:49:08.721092+0000","last_became_active":"2024-04-02T07:49:06.771981+0000","last_became_peered":"2024-04-02T07:49:06.771981+0000","last_unstale":"2024-04-02T07:49:08.721092+0000","last_undegraded":"2024-04-02T07:49:08.721092+0000","last_fullsized":"2024-04-02T07:49:08.721092+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:35.405121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255352+0000","last_change":"2024-04-02T07:49:07.285539+0000","last_active":"2024-04-02T07:49:08.255352+0000","last_peered":"2024-04-02T07:49:08.255352+0000","last_clean":"2024-04-02T07:49:08.255352+0000","last_became_active":"2024-04-02T07:49:07.285282+0000","last_became_peered":"2024-04-02T07:49:07.285282+0000","last_unstale":"2024-04-02T07:49:08.255352+0000","last_undegraded":"2024-04-02T07:49:08.255352+0000","last_fullsized":"2024-04-02T07:49:08.255352+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:39.433909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"65'1","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721384+0000","last_change":"2024-04-02T07:49:06.777265+0000","last_active":"2024-04-02T07:49:08.721384+0000","last_peered":"2024-04-02T07:49:08.721384+0000","last_clean":"2024-04-02T07:49:08.721384+0000","last_became_active":"2024-04-02T07:49:06.776996+0000","last_became_peered":"2024-04-02T07:49:06.776996+0000","last_unstale":"2024-04-02T07:49:08.721384+0000","last_undegraded":"2024-04-02T07:49:08.721384+0000","last_fullsized":"2024-04-02T07:49:08.721384+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:40:17.483842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721391+0000","last_change":"2024-04-02T07:49:07.282868+0000","last_active":"2024-04-02T07:49:08.721391+0000","last_peered":"2024-04-02T07:49:08.721391+0000","last_clean":"2024-04-02T07:49:08.721391+0000","last_became_active":"2024-04-02T07:49:07.282469+0000","last_became_peered":"2024-04-02T07:49:07.282469+0000","last_unstale":"2024-04-02T07:49:08.721391+0000","last_undegraded":"2024-04-02T07:49:08.721391+0000","last_fullsized":"2024-04-02T07:49:08.721391+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:55:05.885676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720590+0000","last_change":"2024-04-02T07:49:06.774712+0000","last_active":"2024-04-02T07:49:08.720590+0000","last_peered":"2024-04-02T07:49:08.720590+0000","last_clean":"2024-04-02T07:49:08.720590+0000","last_became_active":"2024-04-02T07:49:06.774321+0000","last_became_peered":"2024-04-02T07:49:06.774321+0000","last_unstale":"2024-04-02T07:49:08.720590+0000","last_undegraded":"2024-04-02T07:49:08.720590+0000","last_fullsized":"2024-04-02T07:49:08.720590+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:40:17.483842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260343+0000","last_change":"2024-04-02T07:49:07.284161+0000","last_active":"2024-04-02T07:49:08.260343+0000","last_peered":"2024-04-02T07:49:08.260343+0000","last_clean":"2024-04-02T07:49:08.260343+0000","last_became_active":"2024-04-02T07:49:07.283662+0000","last_became_peered":"2024-04-02T07:49:07.283662+0000","last_unstale":"2024-04-02T07:49:08.260343+0000","last_undegraded":"2024-04-02T07:49:08.260343+0000","last_fullsized":"2024-04-02T07:49:08.260343+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:37.672940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"65'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260906+0000","last_change":"2024-04-02T07:49:06.770365+0000","last_active":"2024-04-02T07:49:08.260906+0000","last_peered":"2024-04-02T07:49:08.260906+0000","last_clean":"2024-04-02T07:49:08.260906+0000","last_became_active":"2024-04-02T07:49:06.769151+0000","last_became_peered":"2024-04-02T07:49:06.769151+0000","last_unstale":"2024-04-02T07:49:08.260906+0000","last_undegraded":"2024-04-02T07:49:08.260906+0000","last_fullsized":"2024-04-02T07:49:08.260906+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:15.218952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721150+0000","last_change":"2024-04-02T07:49:07.276632+0000","last_active":"2024-04-02T07:49:08.721150+0000","last_peered":"2024-04-02T07:49:08.721150+0000","last_clean":"2024-04-02T07:49:08.721150+0000","last_became_active":"2024-04-02T07:49:07.275010+0000","last_became_peered":"2024-04-02T07:49:07.275010+0000","last_unstale":"2024-04-02T07:49:08.721150+0000","last_undegraded":"2024-04-02T07:49:08.721150+0000","last_fullsized":"2024-04-02T07:49:08.721150+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:39:26.559000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721235+0000","last_change":"2024-04-02T07:49:06.769474+0000","last_active":"2024-04-02T07:49:08.721235+0000","last_peered":"2024-04-02T07:49:08.721235+0000","last_clean":"2024-04-02T07:49:08.721235+0000","last_became_active":"2024-04-02T07:49:06.769236+0000","last_became_peered":"2024-04-02T07:49:06.769236+0000","last_unstale":"2024-04-02T07:49:08.721235+0000","last_undegraded":"2024-04-02T07:49:08.721235+0000","last_fullsized":"2024-04-02T07:49:08.721235+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:40.427580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721018+0000","last_change":"2024-04-02T07:49:07.288867+0000","last_active":"2024-04-02T07:49:08.721018+0000","last_peered":"2024-04-02T07:49:08.721018+0000","last_clean":"2024-04-02T07:49:08.721018+0000","last_became_active":"2024-04-02T07:49:07.288122+0000","last_became_peered":"2024-04-02T07:49:07.288122+0000","last_unstale":"2024-04-02T07:49:08.721018+0000","last_undegraded":"2024-04-02T07:49:08.721018+0000","last_fullsized":"2024-04-02T07:49:08.721018+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:26.176308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260146+0000","last_change":"2024-04-02T07:49:06.774146+0000","last_active":"2024-04-02T07:49:08.260146+0000","last_peered":"2024-04-02T07:49:08.260146+0000","last_clean":"2024-04-02T07:49:08.260146+0000","last_became_active":"2024-04-02T07:49:06.770861+0000","last_became_peered":"2024-04-02T07:49:06.770861+0000","last_unstale":"2024-04-02T07:49:08.260146+0000","last_undegraded":"2024-04-02T07:49:08.260146+0000","last_fullsized":"2024-04-02T07:49:08.260146+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:14.417485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721279+0000","last_change":"2024-04-02T07:49:07.285416+0000","last_active":"2024-04-02T07:49:08.721279+0000","last_peered":"2024-04-02T07:49:08.721279+0000","last_clean":"2024-04-02T07:49:08.721279+0000","last_became_active":"2024-04-02T07:49:07.285136+0000","last_became_peered":"2024-04-02T07:49:07.285136+0000","last_unstale":"2024-04-02T07:49:08.721279+0000","last_undegraded":"2024-04-02T07:49:08.721279+0000","last_fullsized":"2024-04-02T07:49:08.721279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:34:00.481993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720320+0000","last_change":"2024-04-02T07:49:06.768690+0000","last_active":"2024-04-02T07:49:08.720320+0000","last_peered":"2024-04-02T07:49:08.720320+0000","last_clean":"2024-04-02T07:49:08.720320+0000","last_became_active":"2024-04-02T07:49:06.768410+0000","last_became_peered":"2024-04-02T07:49:06.768410+0000","last_unstale":"2024-04-02T07:49:08.720320+0000","last_undegraded":"2024-04-02T07:49:08.720320+0000","last_fullsized":"2024-04-02T07:49:08.720320+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:20.329600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720649+0000","last_change":"2024-04-02T07:49:07.279081+0000","last_active":"2024-04-02T07:49:08.720649+0000","last_peered":"2024-04-02T07:49:08.720649+0000","last_clean":"2024-04-02T07:49:08.720649+0000","last_became_active":"2024-04-02T07:49:07.278803+0000","last_became_peered":"2024-04-02T07:49:07.278803+0000","last_unstale":"2024-04-02T07:49:08.720649+0000","last_undegraded":"2024-04-02T07:49:08.720649+0000","last_fullsized":"2024-04-02T07:49:08.720649+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:06:38.738379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721814+0000","last_change":"2024-04-02T07:49:06.774899+0000","last_active":"2024-04-02T07:49:08.721814+0000","last_peered":"2024-04-02T07:49:08.721814+0000","last_clean":"2024-04-02T07:49:08.721814+0000","last_became_active":"2024-04-02T07:49:06.773640+0000","last_became_peered":"2024-04-02T07:49:06.773640+0000","last_unstale":"2024-04-02T07:49:08.721814+0000","last_undegraded":"2024-04-02T07:49:08.721814+0000","last_fullsized":"2024-04-02T07:49:08.721814+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:42.119480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260800+0000","last_change":"2024-04-02T07:49:07.267917+0000","last_active":"2024-04-02T07:49:08.260800+0000","last_peered":"2024-04-02T07:49:08.260800+0000","last_clean":"2024-04-02T07:49:08.260800+0000","last_became_active":"2024-04-02T07:49:07.267308+0000","last_became_peered":"2024-04-02T07:49:07.267308+0000","last_unstale":"2024-04-02T07:49:08.260800+0000","last_undegraded":"2024-04-02T07:49:08.260800+0000","last_fullsized":"2024-04-02T07:49:08.260800+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:07:49.813861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721307+0000","last_change":"2024-04-02T07:49:06.775967+0000","last_active":"2024-04-02T07:49:08.721307+0000","last_peered":"2024-04-02T07:49:08.721307+0000","last_clean":"2024-04-02T07:49:08.721307+0000","last_became_active":"2024-04-02T07:49:06.775406+0000","last_became_peered":"2024-04-02T07:49:06.775406+0000","last_unstale":"2024-04-02T07:49:08.721307+0000","last_undegraded":"2024-04-02T07:49:08.721307+0000","last_fullsized":"2024-04-02T07:49:08.721307+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:38.426612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721606+0000","last_change":"2024-04-02T07:49:07.286788+0000","last_active":"2024-04-02T07:49:08.721606+0000","last_peered":"2024-04-02T07:49:08.721606+0000","last_clean":"2024-04-02T07:49:08.721606+0000","last_became_active":"2024-04-02T07:49:07.285696+0000","last_became_peered":"2024-04-02T07:49:07.285696+0000","last_unstale":"2024-04-02T07:49:08.721606+0000","last_undegraded":"2024-04-02T07:49:08.721606+0000","last_fullsized":"2024-04-02T07:49:08.721606+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:05.238772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260630+0000","last_change":"2024-04-02T07:49:06.770811+0000","last_active":"2024-04-02T07:49:08.260630+0000","last_peered":"2024-04-02T07:49:08.260630+0000","last_clean":"2024-04-02T07:49:08.260630+0000","last_became_active":"2024-04-02T07:49:06.770515+0000","last_became_peered":"2024-04-02T07:49:06.770515+0000","last_unstale":"2024-04-02T07:49:08.260630+0000","last_undegraded":"2024-04-02T07:49:08.260630+0000","last_fullsized":"2024-04-02T07:49:08.260630+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:41:59.449223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260983+0000","last_change":"2024-04-02T07:49:07.284033+0000","last_active":"2024-04-02T07:49:08.260983+0000","last_peered":"2024-04-02T07:49:08.260983+0000","last_clean":"2024-04-02T07:49:08.260983+0000","last_became_active":"2024-04-02T07:49:07.283785+0000","last_became_peered":"2024-04-02T07:49:07.283785+0000","last_unstale":"2024-04-02T07:49:08.260983+0000","last_undegraded":"2024-04-02T07:49:08.260983+0000","last_fullsized":"2024-04-02T07:49:08.260983+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:46:51.306140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721646+0000","last_change":"2024-04-02T07:49:06.774465+0000","last_active":"2024-04-02T07:49:08.721646+0000","last_peered":"2024-04-02T07:49:08.721646+0000","last_clean":"2024-04-02T07:49:08.721646+0000","last_became_active":"2024-04-02T07:49:06.774182+0000","last_became_peered":"2024-04-02T07:49:06.774182+0000","last_unstale":"2024-04-02T07:49:08.721646+0000","last_undegraded":"2024-04-02T07:49:08.721646+0000","last_fullsized":"2024-04-02T07:49:08.721646+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:42.119480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721683+0000","last_change":"2024-04-02T07:49:07.288465+0000","last_active":"2024-04-02T07:49:08.721683+0000","last_peered":"2024-04-02T07:49:08.721683+0000","last_clean":"2024-04-02T07:49:08.721683+0000","last_became_active":"2024-04-02T07:49:07.288113+0000","last_became_peered":"2024-04-02T07:49:07.288113+0000","last_unstale":"2024-04-02T07:49:08.721683+0000","last_undegraded":"2024-04-02T07:49:08.721683+0000","last_fullsized":"2024-04-02T07:49:08.721683+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:40:57.913800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720809+0000","last_change":"2024-04-02T07:49:06.773102+0000","last_active":"2024-04-02T07:49:08.720809+0000","last_peered":"2024-04-02T07:49:08.720809+0000","last_clean":"2024-04-02T07:49:08.720809+0000","last_became_active":"2024-04-02T07:49:06.771951+0000","last_became_peered":"2024-04-02T07:49:06.771951+0000","last_unstale":"2024-04-02T07:49:08.720809+0000","last_undegraded":"2024-04-02T07:49:08.720809+0000","last_fullsized":"2024-04-02T07:49:08.720809+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:36:38.777298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720270+0000","last_change":"2024-04-02T07:49:07.288118+0000","last_active":"2024-04-02T07:49:08.720270+0000","last_peered":"2024-04-02T07:49:08.720270+0000","last_clean":"2024-04-02T07:49:08.720270+0000","last_became_active":"2024-04-02T07:49:07.287705+0000","last_became_peered":"2024-04-02T07:49:07.287705+0000","last_unstale":"2024-04-02T07:49:08.720270+0000","last_undegraded":"2024-04-02T07:49:08.720270+0000","last_fullsized":"2024-04-02T07:49:08.720270+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:41.434877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720992+0000","last_change":"2024-04-02T07:49:06.773781+0000","last_active":"2024-04-02T07:49:08.720992+0000","last_peered":"2024-04-02T07:49:08.720992+0000","last_clean":"2024-04-02T07:49:08.720992+0000","last_became_active":"2024-04-02T07:49:06.773043+0000","last_became_peered":"2024-04-02T07:49:06.773043+0000","last_unstale":"2024-04-02T07:49:08.720992+0000","last_undegraded":"2024-04-02T07:49:08.720992+0000","last_fullsized":"2024-04-02T07:49:08.720992+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:59.964498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.255203+0000","last_change":"2024-04-02T07:49:07.269078+0000","last_active":"2024-04-02T07:49:08.255203+0000","last_peered":"2024-04-02T07:49:08.255203+0000","last_clean":"2024-04-02T07:49:08.255203+0000","last_became_active":"2024-04-02T07:49:07.267893+0000","last_became_peered":"2024-04-02T07:49:07.267893+0000","last_unstale":"2024-04-02T07:49:08.255203+0000","last_undegraded":"2024-04-02T07:49:08.255203+0000","last_fullsized":"2024-04-02T07:49:08.255203+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:33.794321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.260032+0000","last_change":"2024-04-02T07:49:06.765671+0000","last_active":"2024-04-02T07:49:08.260032+0000","last_peered":"2024-04-02T07:49:08.260032+0000","last_clean":"2024-04-02T07:49:08.260032+0000","last_became_active":"2024-04-02T07:49:06.765377+0000","last_became_peered":"2024-04-02T07:49:06.765377+0000","last_unstale":"2024-04-02T07:49:08.260032+0000","last_undegraded":"2024-04-02T07:49:08.260032+0000","last_fullsized":"2024-04-02T07:49:08.260032+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:19:10.575751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.721771+0000","last_change":"2024-04-02T07:49:07.283554+0000","last_active":"2024-04-02T07:49:08.721771+0000","last_peered":"2024-04-02T07:49:08.721771+0000","last_clean":"2024-04-02T07:49:08.721771+0000","last_became_active":"2024-04-02T07:49:07.282688+0000","last_became_peered":"2024-04-02T07:49:07.282688+0000","last_unstale":"2024-04-02T07:49:08.721771+0000","last_undegraded":"2024-04-02T07:49:08.721771+0000","last_fullsized":"2024-04-02T07:49:08.721771+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:06.743616+0000","last_clean_scrub_stamp":"2024-04-02T07:49:06.743616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:30:00.443550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"65'2","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-02T07:49:08.720759+0000","last_change":"2024-04-02T07:49:06.772504+0000","last_active":"2024-04-02T07:49:08.720759+0000","last_peered":"2024-04-02T07:49:08.720759+0000","last_clean":"2024-04-02T07:49:08.720759+0000","last_became_active":"2024-04-02T07:49:06.771939+0000","last_became_peered":"2024-04-02T07:49:06.771939+0000","last_unstale":"2024-04-02T07:49:08.720759+0000","last_undegraded":"2024-04-02T07:49:08.720759+0000","last_fullsized":"2024-04-02T07:49:08.720759+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:49:05.736319+0000","last_clean_scrub_stamp":"2024-04-02T07:49:05.736319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:14.417485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"55'139","reported_seq":288,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-02T07:49:07.262003+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:49:07.262003+0000","last_peered":"2024-04-02T07:49:07.262003+0000","last_clean":"2024-04-02T07:49:07.262003+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:49:07.262003+0000","last_undegraded":"2024-04-02T07:49:07.262003+0000","last_fullsized":"2024-04-02T07:49:07.262003+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575011,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31896,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339944,"statfs":{"total":96636764160,"available":96604102656,"internally_reserved":0,"allocated":790528,"data_stored":372301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.66500000000000004,"15min":0.66400000000000003},"min":{"1min":0.40000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.79800000000000004,"5min":0.90900000000000003,"15min":0.93200000000000005},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.67800000000000005,"15min":0.66500000000000004},"min":{"1min":0.45900000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90100000000000002,"5min":1.0109999999999999,"15min":1.024},"last":0.72399999999999998}]},{"osd":1,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67800000000000005,"15min":0.67100000000000004},"min":{"1min":0.46400000000000002,"5min":0.44600000000000001,"15min":0.39000000000000001},"max":{"1min":0.81899999999999995,"5min":0.89900000000000002,"15min":0.96499999999999997},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.625,"5min":0.67200000000000004,"15min":0.66200000000000003},"min":{"1min":0.379,"5min":0.35599999999999998,"15min":0.34699999999999998},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.84099999999999997}]},{"osd":2,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.71299999999999997,"15min":0.69699999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":1.127},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.76900000000000002,"15min":0.75},"min":{"1min":0.501,"5min":0.501,"15min":0.434},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":1.048}]},{"osd":3,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.69999999999999996,"15min":0.69399999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90700000000000003,"5min":1.026,"15min":1.274},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.72599999999999998,"15min":0.71699999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.89500000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.30199999999999999},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67900000000000005,"15min":0.68799999999999994},"min":{"1min":0.42199999999999999,"5min":0.40000000000000002,"15min":0.38100000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.4890000000000001},"last":1.0069999999999999}]},{"osd":5,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65000000000000002,"15min":0.64700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.68500000000000005,"15min":0.68899999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":1.026}]},{"osd":6,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.67200000000000004,"15min":0.65900000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.65500000000000003,"15min":0.65600000000000003},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39600000000000002},"max":{"1min":0.93799999999999994,"5min":1.0509999999999999,"15min":1.329},"last":0.93799999999999994}]}]},{"osd":6,"up_from":13,"seq":55834575009,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":782336,"data_stored":366235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.70199999999999996,"15min":0.68300000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.32400000000000001},"max":{"1min":0.90300000000000002,"5min":0.90600000000000003,"15min":0.98099999999999998},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.71699999999999997,"15min":0.69299999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.375},"max":{"1min":0.84899999999999998,"5min":0.95299999999999996,"15min":1.0249999999999999},"last":0.68799999999999994}]},{"osd":1,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.69399999999999995,"15min":0.68600000000000005},"min":{"1min":0.46500000000000002,"5min":0.41799999999999998,"15min":0.41199999999999998},"max":{"1min":0.80900000000000005,"5min":0.91600000000000004,"15min":1.085},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.70199999999999996,"15min":0.69799999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86599999999999999,"5min":0.93300000000000005,"15min":1.0009999999999999},"last":0.67700000000000005}]},{"osd":2,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.72599999999999998,"15min":0.69499999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.38200000000000001},"max":{"1min":0.84899999999999998,"5min":0.91200000000000003,"15min":1},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.51400000000000001,"5min":0.499,"15min":0.45900000000000002},"max":{"1min":0.85299999999999998,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.70299999999999996}]},{"osd":3,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.73099999999999998,"15min":0.72799999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.47099999999999997},"max":{"1min":0.89600000000000002,"5min":0.95399999999999996,"15min":0.98199999999999998},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.746,"15min":0.73199999999999998},"min":{"1min":0.52400000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.85899999999999999,"5min":0.995,"15min":1.0309999999999999},"last":0.64000000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.60899999999999999,"15min":0.60199999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.92500000000000004,"5min":0.92700000000000005,"15min":0.94599999999999995},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64700000000000002,"15min":0.63300000000000001},"min":{"1min":0.42799999999999999,"5min":0.39600000000000002,"15min":0.35499999999999998},"max":{"1min":0.89700000000000002,"5min":0.94099999999999995,"15min":1.0049999999999999},"last":0.75700000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63400000000000001,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.379,"15min":0.35799999999999998},"max":{"1min":0.82599999999999996,"5min":0.88200000000000001,"15min":1.1599999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68300000000000005,"15min":0.68400000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.40300000000000002},"max":{"1min":0.90400000000000003,"5min":0.97899999999999998,"15min":1.095},"last":0.60899999999999999}]},{"osd":7,"last update":"Tue Apr 2 07:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.69899999999999995,"15min":0.68500000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.69599999999999995,"15min":0.69999999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.42099999999999999},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.4399999999999999},"last":0.83599999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36556,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335284,"statfs":{"total":96636764160,"available":96599330816,"internally_reserved":0,"allocated":1372160,"data_stored":956057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.45900000000000002,"15min":0.46200000000000002},"min":{"1min":0.32400000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.73199999999999998,"5min":0.73299999999999998,"15min":2.9079999999999999},"last":0.315},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.34799999999999998,"5min":0.34100000000000003,"15min":0.313},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":2.766},"last":0.496}]},{"osd":2,"last update":"Tue Apr 2 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.38300000000000001,"5min":0.32200000000000001,"15min":0.30299999999999999},"max":{"1min":0.65100000000000002,"5min":0.71599999999999997,"15min":2.0750000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58499999999999996,"15min":0.57699999999999996},"min":{"1min":0.433,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":1.7410000000000001},"last":0.46700000000000003}]},{"osd":3,"last update":"Tue Apr 2 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.35999999999999999},"max":{"1min":0.71899999999999997,"5min":0.879,"15min":2.9430000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55600000000000005,"15min":0.56100000000000005},"min":{"1min":0.41999999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77500000000000002,"5min":0.85499999999999998,"15min":2.802},"last":0.42499999999999999}]},{"osd":4,"last update":"Tue Apr 2 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78900000000000003,"15min":0.77000000000000002},"min":{"1min":0.60599999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.93799999999999994,"5min":1.0860000000000001,"15min":1.7969999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78500000000000003,"15min":0.76600000000000001},"min":{"1min":0.61299999999999999,"5min":0.47799999999999998,"15min":0.46500000000000002},"max":{"1min":1.0089999999999999,"5min":1.135,"15min":2.0419999999999998},"last":0.77800000000000002}]},{"osd":5,"last update":"Tue Apr 2 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.79400000000000004,"15min":0.77000000000000002},"min":{"1min":0.61099999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.117,"5min":1.117,"15min":1.9690000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.77300000000000002},"min":{"1min":0.70299999999999996,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.96499999999999997,"5min":1.0920000000000001,"15min":1.3009999999999999},"last":0.748}]},{"osd":6,"last update":"Tue Apr 2 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.81999999999999995,"15min":0.79300000000000004},"min":{"1min":0.66200000000000003,"5min":0.52600000000000002,"15min":0.47799999999999998},"max":{"1min":1.0900000000000001,"5min":1.1919999999999999,"15min":1.768},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.81599999999999995,"15min":0.78700000000000003},"min":{"1min":0.61599999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.048,"5min":1.0820000000000001,"15min":2.012},"last":0.73099999999999998}]},{"osd":7,"last update":"Tue Apr 2 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81999999999999995,"15min":0.80300000000000005},"min":{"1min":0.62,"5min":0.56799999999999995,"15min":0.502},"max":{"1min":1.552,"5min":1.552,"15min":1.706},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.85099999999999998,"15min":0.81499999999999995},"min":{"1min":0.63900000000000001,"5min":0.55200000000000005,"15min":0.502},"max":{"1min":1.147,"5min":1.1519999999999999,"15min":1.8240000000000001},"last":0.66900000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575010,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":790528,"data_stored":371920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.434,"15min":0.45900000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":2.1640000000000001},"last":0.442},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.497,"15min":0.51800000000000002},"min":{"1min":0.32800000000000001,"5min":0.30499999999999999,"15min":0.26200000000000001},"max":{"1min":0.71299999999999997,"5min":0.79100000000000004,"15min":2.6469999999999998},"last":0.42599999999999999}]},{"osd":2,"last update":"Tue Apr 2 07:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.495,"15min":0.51300000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33000000000000002},"max":{"1min":0.68200000000000005,"5min":0.76300000000000001,"15min":2.125},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.54500000000000004,"15min":0.56100000000000005},"min":{"1min":0.40400000000000003,"5min":0.35199999999999998,"15min":0.33300000000000002},"max":{"1min":0.745,"5min":0.77800000000000002,"15min":2.605},"last":0.48599999999999999}]},{"osd":3,"last update":"Tue Apr 2 07:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.60099999999999998},"min":{"1min":0.377,"5min":0.34899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73999999999999999,"5min":0.79400000000000004,"15min":5.4800000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61899999999999999,"15min":0.63},"min":{"1min":0.44,"5min":0.44,"15min":0.40100000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":2.194},"last":0.55500000000000005}]},{"osd":4,"last update":"Tue Apr 2 07:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.80400000000000005,"15min":0.85999999999999999},"min":{"1min":0.60299999999999998,"5min":0.44800000000000001,"15min":0.35599999999999998},"max":{"1min":1.0169999999999999,"5min":1.181,"15min":6.0899999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.84899999999999998},"min":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.433},"max":{"1min":0.98799999999999999,"5min":1.331,"15min":5.7169999999999996},"last":0.96099999999999997}]},{"osd":5,"last update":"Tue Apr 2 07:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.81200000000000006,"15min":0.85899999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.42099999999999999},"max":{"1min":1.1779999999999999,"5min":1.282,"15min":5.7539999999999996},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81799999999999995,"15min":0.86199999999999999},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.51800000000000002},"max":{"1min":1.0349999999999999,"5min":1.2330000000000001,"15min":5.8929999999999998},"last":1.0169999999999999}]},{"osd":6,"last update":"Tue Apr 2 07:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76400000000000001,"15min":0.82099999999999995},"min":{"1min":0.61899999999999999,"5min":0.52900000000000003,"15min":0.46800000000000003},"max":{"1min":0.93100000000000005,"5min":1.1200000000000001,"15min":5.931},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78000000000000003,"15min":0.83799999999999997},"min":{"1min":0.53900000000000003,"5min":0.52000000000000002,"15min":0.376},"max":{"1min":0.97399999999999998,"5min":1.1339999999999999,"15min":6.125},"last":0.92200000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.81899999999999995,"15min":0.88},"min":{"1min":0.61899999999999999,"5min":0.54700000000000004,"15min":0.40799999999999997},"max":{"1min":1.0189999999999999,"5min":1.2170000000000001,"15min":6.1580000000000004},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.83999999999999997,"15min":0.89000000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.47699999999999998},"max":{"1min":1.1000000000000001,"5min":1.212,"15min":5.7850000000000001},"last":0.85099999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":790528,"data_stored":372357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.438,"15min":0.442},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.25700000000000001},"max":{"1min":0.55200000000000005,"5min":0.69099999999999995,"15min":0.79000000000000004},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.26200000000000001},"max":{"1min":0.63500000000000001,"5min":0.752,"15min":0.84299999999999997},"last":0.46600000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.49399999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30299999999999999},"max":{"1min":0.65600000000000003,"5min":0.72399999999999998,"15min":0.93999999999999995},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.52400000000000002,"15min":0.51800000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":0.57399999999999995,"5min":0.80300000000000005,"15min":0.90000000000000002},"last":0.41999999999999998}]},{"osd":3,"last update":"Tue Apr 2 07:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.27800000000000002},"max":{"1min":0.67900000000000005,"5min":0.79800000000000004,"15min":1.4339999999999999},"last":0.316},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.60299999999999998,"15min":0.61099999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.39000000000000001},"max":{"1min":0.72499999999999998,"5min":0.878,"15min":1.3740000000000001},"last":0.437}]},{"osd":4,"last update":"Tue Apr 2 07:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.78200000000000003,"15min":0.76400000000000001},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.47199999999999998},"max":{"1min":1.0049999999999999,"5min":1.1519999999999999,"15min":1.157},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.77400000000000002,"15min":0.76400000000000001},"min":{"1min":0.52300000000000002,"5min":0.51600000000000001,"15min":0.48499999999999999},"max":{"1min":0.92100000000000004,"5min":1.1479999999999999,"15min":1.179},"last":0.75600000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.77200000000000002,"15min":0.754},"min":{"1min":0.57199999999999995,"5min":0.55800000000000005,"15min":0.42199999999999999},"max":{"1min":0.90700000000000003,"5min":1.2,"15min":1.2},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.78100000000000003,"15min":0.77600000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.50900000000000001},"max":{"1min":0.91100000000000003,"5min":1.1080000000000001,"15min":1.1899999999999999},"last":0.74199999999999999}]},{"osd":6,"last update":"Tue Apr 2 07:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.80000000000000004,"15min":0.77800000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.442},"max":{"1min":0.97099999999999997,"5min":1.175,"15min":1.175},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.79000000000000004,"15min":0.78800000000000003},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.52100000000000002},"max":{"1min":0.89300000000000002,"5min":1.075,"15min":1.2130000000000001},"last":0.78800000000000003}]},{"osd":7,"last update":"Tue Apr 2 07:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.80300000000000005,"15min":0.78900000000000003},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.5},"max":{"1min":0.94399999999999995,"5min":1.1539999999999999,"15min":1.1779999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.81399999999999995,"15min":0.79400000000000004},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.52900000000000003},"max":{"1min":0.98899999999999999,"5min":1.1339999999999999,"15min":1.145},"last":0.91300000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575009,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35984,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335856,"statfs":{"total":96636764160,"available":96599916544,"internally_reserved":0,"allocated":786432,"data_stored":367183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.42099999999999999,"15min":0.45100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26000000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":2.476},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.45900000000000002,"15min":0.503},"min":{"1min":0.29699999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.51700000000000002,"5min":0.74399999999999999,"15min":2.04},"last":0.46100000000000002}]},{"osd":1,"last update":"Tue Apr 2 07:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.48699999999999999,"15min":0.50600000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":2.2839999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55500000000000005,"15min":0.56599999999999995},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.34200000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":2.2240000000000002},"last":0.52500000000000002}]},{"osd":2,"last update":"Tue Apr 2 07:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59999999999999998,"15min":0.61699999999999999},"min":{"1min":0.41499999999999998,"5min":0.39700000000000002,"15min":0.376},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":1.7310000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59799999999999998,"15min":0.64600000000000002},"min":{"1min":0.438,"5min":0.39700000000000002,"15min":0.35699999999999998},"max":{"1min":1.244,"5min":1.244,"15min":3.9449999999999998},"last":0.48399999999999999}]},{"osd":4,"last update":"Tue Apr 2 07:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.72399999999999998,"15min":0.76200000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.38400000000000001},"max":{"1min":0.93700000000000006,"5min":1.032,"15min":2.0030000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.749,"15min":0.76300000000000001},"min":{"1min":0.51600000000000001,"5min":0.503,"15min":0.47399999999999998},"max":{"1min":0.88800000000000001,"5min":1.02,"15min":1.5249999999999999},"last":0.63}]},{"osd":5,"last update":"Tue Apr 2 07:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.73499999999999999,"15min":0.751},"min":{"1min":0.48599999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.90100000000000002,"5min":1.0449999999999999,"15min":1.2909999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.74099999999999999,"15min":0.77700000000000002},"min":{"1min":0.48899999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85199999999999998,"5min":1.0449999999999999,"15min":1.9159999999999999},"last":0.68400000000000005}]},{"osd":6,"last update":"Tue Apr 2 07:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.746,"15min":0.76900000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.89700000000000002,"5min":1.0620000000000001,"15min":1.548},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.76500000000000001,"15min":0.77900000000000003},"min":{"1min":0.55100000000000005,"5min":0.54000000000000004,"15min":0.53500000000000003},"max":{"1min":0.92300000000000004,"5min":1.0329999999999999,"15min":1.4379999999999999},"last":0.65700000000000003}]},{"osd":7,"last update":"Tue Apr 2 07:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.76900000000000002,"15min":0.79900000000000004},"min":{"1min":0.55300000000000005,"5min":0.54600000000000004,"15min":0.439},"max":{"1min":0.97999999999999998,"5min":1.0369999999999999,"15min":1.966},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.77500000000000002,"15min":0.79400000000000004},"min":{"1min":0.499,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.181,"5min":1.181,"15min":1.514},"last":0.73899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35988,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335852,"statfs":{"total":96636764160,"available":96599912448,"internally_reserved":0,"allocated":794624,"data_stored":367275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.67600000000000005,"15min":0.65000000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.32100000000000001},"max":{"1min":0.872,"5min":0.94299999999999995,"15min":1.0189999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68200000000000005,"15min":0.66600000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.35299999999999998},"max":{"1min":0.81799999999999995,"5min":0.90100000000000002,"15min":1.7849999999999999},"last":0.58099999999999996}]},{"osd":1,"last update":"Tue Apr 2 07:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.73399999999999999,"15min":0.71499999999999997},"min":{"1min":0.54200000000000004,"5min":0.501,"15min":0.42699999999999999},"max":{"1min":0.90900000000000003,"5min":1.0349999999999999,"15min":1.036},"last":0.438},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.73699999999999999,"15min":0.72799999999999998},"min":{"1min":0.47299999999999998,"5min":0.46700000000000003,"15min":0.45500000000000002},"max":{"1min":0.82299999999999995,"5min":0.97899999999999998,"15min":0.99399999999999999},"last":0.67500000000000004}]},{"osd":2,"last update":"Tue Apr 2 07:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.69699999999999995,"15min":0.68500000000000005},"min":{"1min":0.46400000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.82999999999999996,"5min":1.0069999999999999,"15min":1.03},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.71399999999999997,"15min":0.71599999999999997},"min":{"1min":0.54600000000000004,"5min":0.50700000000000001,"15min":0.41899999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.6459999999999999},"last":0.61899999999999999}]},{"osd":3,"last update":"Tue Apr 2 07:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.72499999999999998,"15min":0.71699999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":0.95399999999999996,"5min":0.97699999999999998,"15min":1.002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.73399999999999999,"15min":0.71799999999999997},"min":{"1min":0.497,"5min":0.48799999999999999,"15min":0.41499999999999998},"max":{"1min":0.92500000000000004,"5min":0.96399999999999997,"15min":0.96999999999999997},"last":0.73699999999999999}]},{"osd":5,"last update":"Tue Apr 2 07:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56999999999999995,"15min":0.58999999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.76900000000000002,"5min":0.78500000000000003,"15min":1.034},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58599999999999997,"15min":0.60299999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.33700000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":1.0129999999999999},"last":0.68899999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59799999999999998,"15min":0.625},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.86099999999999999,"5min":0.93400000000000005,"15min":1.2589999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63900000000000001,"15min":0.66400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":1.0609999999999999},"last":0.80900000000000005}]},{"osd":7,"last update":"Tue Apr 2 07:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47699999999999998,"5min":0.432,"15min":0.38300000000000001},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":1.0680000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.74099999999999999,"15min":0.747},"min":{"1min":0.52800000000000002,"5min":0.434,"15min":0.434},"max":{"1min":0.999,"5min":1.008,"15min":1.502},"last":0.64700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575012,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1380352,"data_stored":961638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:49:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.68700000000000006,"15min":0.66600000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.85999999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.69899999999999995,"15min":0.68100000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.39500000000000002}]},{"osd":1,"last update":"Tue Apr 2 07:49:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69499999999999995,"15min":0.68400000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40200000000000002},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69799999999999995,"15min":0.68700000000000006},"min":{"1min":0.41499999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.41499999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:49:07 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.71399999999999997,"15min":0.70599999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.70899999999999996,"15min":0.70599999999999996},"min":{"1min":0.47399999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.59399999999999997}]},{"osd":3,"last update":"Tue Apr 2 07:49:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.73699999999999999,"15min":0.74299999999999999},"min":{"1min":0.45900000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.749,"5min":0.77400000000000002,"15min":0.75900000000000001},"min":{"1min":0.54600000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.63700000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:49:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.60799999999999998,"15min":0.61099999999999999},"min":{"1min":0.38400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63900000000000001,"15min":0.65600000000000003},"min":{"1min":0.42499999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":0.61099999999999999}]},{"osd":6,"last update":"Tue Apr 2 07:49:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.44700000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.83399999999999996,"5min":0.93400000000000005,"15min":1.107},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67600000000000005,"15min":0.67700000000000005},"min":{"1min":0.44900000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.622}]},{"osd":7,"last update":"Tue Apr 2 07:49:07 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.495,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.55400000000000005,"5min":0.434,"15min":0.434},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.70199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:49:12.104 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T07:49:12.104 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T07:49:12.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.336+0000 7f0e48600700 1 Processor -- start 2024-04-02T07:49:12.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.337+0000 7f0e48600700 1 -- start start 2024-04-02T07:49:12.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.338+0000 7f0e48600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0e4015d9d0 0x7f0e400d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.338+0000 7f0e48600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e400d16a0 0x7f0e400d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.338+0000 7f0e48600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e40161420 con 0x7f0e400d2370 2024-04-02T07:49:12.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.338+0000 7f0e48600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e400d2820 con 0x7f0e4015d9d0 2024-04-02T07:49:12.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.338+0000 7f0e48600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e40161700 con 0x7f0e400d16a0 2024-04-02T07:49:12.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.338+0000 7f0e46600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0e400d2370 0x7f0e4015ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51488/0 (socket says 172.21.15.89:51488) 2024-04-02T07:49:12.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.338+0000 7f0e46600700 1 -- 172.21.15.89:0/1976634562 learned_addr learned my addr 172.21.15.89:0/1976634562 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:12.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.338+0000 7f0e45200700 1 --2- 172.21.15.89:0/1976634562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e400d16a0 0x7f0e400d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.338+0000 7f0e45c00700 1 --2- 172.21.15.89:0/1976634562 >> v2:172.21.15.113:3300/0 conn(0x7f0e4015d9d0 0x7f0e400d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.338+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1976634562 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355441643 0 0) 0x7f0e40161420 con 0x7f0e400d2370 2024-04-02T07:49:12.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.338+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1976634562 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e280035b0 con 0x7f0e400d2370 2024-04-02T07:49:12.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1976634562 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2885968370 0 0) 0x7f0e280035b0 con 0x7f0e400d2370 2024-04-02T07:49:12.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1976634562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e400d16a0 msgr2=0x7f0e400d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:12.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e3fe00700 1 --2- 172.21.15.89:0/1976634562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e400d16a0 0x7f0e400d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1976634562 >> v2:172.21.15.113:3300/0 conn(0x7f0e4015d9d0 msgr2=0x7f0e400d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:12.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e3fe00700 1 --2- 172.21.15.89:0/1976634562 >> v2:172.21.15.113:3300/0 conn(0x7f0e4015d9d0 0x7f0e400d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1976634562 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e401619e0 con 0x7f0e400d2370 2024-04-02T07:49:12.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1976634562 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f0e380019f0 con 0x7f0e400d2370 2024-04-02T07:49:12.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1976634562 >> v1:172.21.15.89:6789/0 conn(0x7f0e400d2370 legacy=0x7f0e4015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:12.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e3fe00700 1 --2- 172.21.15.89:0/1976634562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e28003a00 0x7f0e28003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e3fe00700 1 --2- 172.21.15.89:0/1976634562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e400d16a0 0x7f0e28004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e3fe00700 1 --2- 172.21.15.89:0/1976634562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e28004d90 0x7f0e28005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1976634562 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e400d2820 con 0x7f0e28004d90 2024-04-02T07:49:12.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1976634562 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e40161700 con 0x7f0e28003a00 2024-04-02T07:49:12.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1976634562 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2800ba10 con 0x7f0e400d16a0 2024-04-02T07:49:12.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e46600700 1 --2- 172.21.15.89:0/1976634562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e28004d90 0x7f0e28005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e45c00700 1 --2- 172.21.15.89:0/1976634562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e28003a00 0x7f0e28003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.339+0000 7f0e45200700 1 --2- 172.21.15.89:0/1976634562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e400d16a0 0x7f0e28004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.340+0000 7f0e46600700 1 -- 172.21.15.89:0/1976634562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e400d16a0 msgr2=0x7f0e28004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.340+0000 7f0e46600700 1 --2- 172.21.15.89:0/1976634562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e400d16a0 0x7f0e28004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.340+0000 7f0e46600700 1 -- 172.21.15.89:0/1976634562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e28003a00 msgr2=0x7f0e28003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.340+0000 7f0e46600700 1 --2- 172.21.15.89:0/1976634562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e28003a00 0x7f0e28003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.340+0000 7f0e46600700 1 -- 172.21.15.89:0/1976634562 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e280035b0 con 0x7f0e28004d90 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.340+0000 7f0e46600700 1 --2- 172.21.15.89:0/1976634562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e28004d90 0x7f0e28005210 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0e3800a860 tx=0x7f0e28005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809825ea876444bc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.340+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1976634562 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e38012600 con 0x7f0e28004d90 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.340+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1976634562 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e38012780 con 0x7f0e28004d90 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.340+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1976634562 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e38012b30 con 0x7f0e28004d90 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.340+0000 7f0e48600700 1 -- 172.21.15.89:0/1976634562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e28004d90 msgr2=0x7f0e28005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.340+0000 7f0e48600700 1 --2- 172.21.15.89:0/1976634562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e28004d90 0x7f0e28005210 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0e3800a860 tx=0x7f0e28005880 comp rx=0 tx=0).stop 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.341+0000 7f0e46600700 1 -- 172.21.15.89:0/1976634562 reap_dead start 2024-04-02T07:49:12.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.341+0000 7f0e48600700 1 -- 172.21.15.89:0/1976634562 shutdown_connections 2024-04-02T07:49:12.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.341+0000 7f0e48600700 1 -- 172.21.15.89:0/1976634562 >> 172.21.15.89:0/1976634562 conn(0x7f0e4008aff0 msgr2=0x7f0e400d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:12.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.341+0000 7f0e48600700 1 -- 172.21.15.89:0/1976634562 shutdown_connections 2024-04-02T07:49:12.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.341+0000 7f0e48600700 1 -- 172.21.15.89:0/1976634562 wait complete. 2024-04-02T07:49:12.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.343+0000 7f0e48600700 1 Processor -- start 2024-04-02T07:49:12.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.343+0000 7f0e48600700 1 -- start start 2024-04-02T07:49:12.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.344+0000 7f0e48600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e28004d90 0x7f0e40152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.344+0000 7f0e48600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e4015a200 0x7f0e4015a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.344+0000 7f0e48600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e400cade0 0x7f0e400cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.344+0000 7f0e48600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4015f850 con 0x7f0e28004d90 2024-04-02T07:49:12.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.344+0000 7f0e48600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4015abc0 con 0x7f0e400cade0 2024-04-02T07:49:12.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.345+0000 7f0e48600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e400cbad0 con 0x7f0e4015a200 2024-04-02T07:49:12.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.345+0000 7f0e45c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e28004d90 0x7f0e40152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.345+0000 7f0e45c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e28004d90 0x7f0e40152680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53460/0 (socket says 172.21.15.89:53460) 2024-04-02T07:49:12.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.345+0000 7f0e45c00700 1 -- 172.21.15.89:0/3228470818 learned_addr learned my addr 172.21.15.89:0/3228470818 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:12.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.345+0000 7f0e46600700 1 --2- 172.21.15.89:0/3228470818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e400cade0 0x7f0e400cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.345+0000 7f0e45200700 1 --2- 172.21.15.89:0/3228470818 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e4015a200 0x7f0e4015a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.345+0000 7f0e45c00700 1 -- 172.21.15.89:0/3228470818 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e4015a200 msgr2=0x7f0e4015a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:12.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.345+0000 7f0e45c00700 1 --2- 172.21.15.89:0/3228470818 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e4015a200 0x7f0e4015a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.345+0000 7f0e45c00700 1 -- 172.21.15.89:0/3228470818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e400cade0 msgr2=0x7f0e400cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:12.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.345+0000 7f0e45c00700 1 --2- 172.21.15.89:0/3228470818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e400cade0 0x7f0e400cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.345+0000 7f0e45c00700 1 -- 172.21.15.89:0/3228470818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4015f190 con 0x7f0e28004d90 2024-04-02T07:49:12.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.345+0000 7f0e45c00700 1 --2- 172.21.15.89:0/3228470818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e28004d90 0x7f0e40152680 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0e34011aa0 tx=0x7f0e34011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:12.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.345+0000 7f0e3d600700 1 -- 172.21.15.89:0/3228470818 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3400f9e0 con 0x7f0e28004d90 2024-04-02T07:49:12.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.346+0000 7f0e3d600700 1 -- 172.21.15.89:0/3228470818 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3400fb40 con 0x7f0e28004d90 2024-04-02T07:49:12.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.346+0000 7f0e48600700 1 -- 172.21.15.89:0/3228470818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4015f470 con 0x7f0e28004d90 2024-04-02T07:49:12.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.346+0000 7f0e3d600700 1 -- 172.21.15.89:0/3228470818 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e34014440 con 0x7f0e28004d90 2024-04-02T07:49:12.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.346+0000 7f0e48600700 1 -- 172.21.15.89:0/3228470818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e400cb7a0 con 0x7f0e28004d90 2024-04-02T07:49:12.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.347+0000 7f0e48600700 1 -- 172.21.15.89:0/3228470818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e4015fac0 con 0x7f0e28004d90 2024-04-02T07:49:12.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.347+0000 7f0e3d600700 1 -- 172.21.15.89:0/3228470818 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0e340145a0 con 0x7f0e28004d90 2024-04-02T07:49:12.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.348+0000 7f0e3d600700 1 --2- 172.21.15.89:0/3228470818 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e2006adb0 0x7f0e2006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.348+0000 7f0e3d600700 1 -- 172.21.15.89:0/3228470818 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0e34089dc0 con 0x7f0e28004d90 2024-04-02T07:49:12.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.349+0000 7f0e45200700 1 --2- 172.21.15.89:0/3228470818 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e2006adb0 0x7f0e2006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.349+0000 7f0e45200700 1 --2- 172.21.15.89:0/3228470818 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e2006adb0 0x7f0e2006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0e400caba0 tx=0x7f0e2c008040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:12.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.352+0000 7f0e3d600700 1 -- 172.21.15.89:0/3228470818 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0e34054d20 con 0x7f0e28004d90 2024-04-02T07:49:12.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.639+0000 7f0e48600700 1 -- 172.21.15.89:0/3228470818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0e400d1ca0 con 0x7f0e28004d90 2024-04-02T07:49:12.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.645+0000 7f0e3d600700 1 -- 172.21.15.89:0/3228470818 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0e34058340 con 0x7f0e28004d90 2024-04-02T07:49:12.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.648+0000 7f0e48600700 1 -- 172.21.15.89:0/3228470818 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e2006adb0 msgr2=0x7f0e2006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:12.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.648+0000 7f0e48600700 1 --2- 172.21.15.89:0/3228470818 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e2006adb0 0x7f0e2006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0e400caba0 tx=0x7f0e2c008040 comp rx=0 tx=0).stop 2024-04-02T07:49:12.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.648+0000 7f0e48600700 1 -- 172.21.15.89:0/3228470818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e28004d90 msgr2=0x7f0e40152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:12.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.648+0000 7f0e48600700 1 --2- 172.21.15.89:0/3228470818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e28004d90 0x7f0e40152680 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0e34011aa0 tx=0x7f0e34011e60 comp rx=0 tx=0).stop 2024-04-02T07:49:12.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.648+0000 7f0e48600700 1 -- 172.21.15.89:0/3228470818 shutdown_connections 2024-04-02T07:49:12.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.648+0000 7f0e48600700 1 --2- 172.21.15.89:0/3228470818 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e2006adb0 0x7f0e2006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.648+0000 7f0e48600700 1 --2- 172.21.15.89:0/3228470818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e400cade0 0x7f0e400cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.648+0000 7f0e48600700 1 --2- 172.21.15.89:0/3228470818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e28004d90 0x7f0e40152680 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.648+0000 7f0e48600700 1 --2- 172.21.15.89:0/3228470818 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e4015a200 0x7f0e4015a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.648+0000 7f0e48600700 1 -- 172.21.15.89:0/3228470818 >> 172.21.15.89:0/3228470818 conn(0x7f0e4008aff0 msgr2=0x7f0e400b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:12.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.648+0000 7f0e48600700 1 -- 172.21.15.89:0/3228470818 shutdown_connections 2024-04-02T07:49:12.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.648+0000 7f0e48600700 1 -- 172.21.15.89:0/3228470818 wait complete. 2024-04-02T07:49:12.651 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T07:49:12.668 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:49:12.744 INFO:tasks.ceph.mds.d.smithi113.stderr:2024-04-02T07:49:12.742+0000 7f3be2c00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T07:49:12.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.857+0000 7fc2c7a00700 1 Processor -- start 2024-04-02T07:49:12.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.857+0000 7fc2c7a00700 1 -- start start 2024-04-02T07:49:12.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.858+0000 7fc2c7a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc2c00ac580 0x7fc2c00b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.858+0000 7fc2c7a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c00b3890 con 0x7fc2c00b3000 2024-04-02T07:49:12.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.858+0000 7fc2c7a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c0159f50 con 0x7fc2c015d670 2024-04-02T07:49:12.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.858+0000 7fc2c7a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c00b35b0 con 0x7fc2c00ac580 2024-04-02T07:49:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.858+0000 7fc2c5a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc2c00b3000 0x7fc2c006dee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51500/0 (socket says 172.21.15.89:51500) 2024-04-02T07:49:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.858+0000 7fc2c5a00700 1 -- 172.21.15.89:0/2199297478 learned_addr learned my addr 172.21.15.89:0/2199297478 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bfe00700 1 --2- 172.21.15.89:0/2199297478 >> v2:172.21.15.113:3300/0 conn(0x7fc2c00ac580 0x7fc2c00b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130058346 0 0) 0x7fc2c00b3890 con 0x7fc2c00b3000 2024-04-02T07:49:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a80035b0 con 0x7fc2c00b3000 2024-04-02T07:49:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916307049 0 0) 0x7fc2c0159f50 con 0x7fc2c015d670 2024-04-02T07:49:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c00b3890 con 0x7fc2c015d670 2024-04-02T07:49:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4042718233 0 0) 0x7fc2a80035b0 con 0x7fc2c00b3000 2024-04-02T07:49:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 >> v2:172.21.15.113:3300/0 conn(0x7fc2c00ac580 msgr2=0x7fc2c00b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 --2- 172.21.15.89:0/2199297478 >> v2:172.21.15.113:3300/0 conn(0x7fc2c00ac580 0x7fc2c00b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 >> v1:172.21.15.113:6789/0 conn(0x7fc2c015d670 legacy=0x7fc2c015da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c006d400 con 0x7fc2c00b3000 2024-04-02T07:49:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fc2b80019f0 con 0x7fc2c00b3000 2024-04-02T07:49:12.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 >> v1:172.21.15.89:6789/0 conn(0x7fc2c00b3000 legacy=0x7fc2c006dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 --2- 172.21.15.89:0/2199297478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a8003b90 0x7fc2a8003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 --2- 172.21.15.89:0/2199297478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2a8004850 0x7fc2a8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 --2- 172.21.15.89:0/2199297478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a8005210 0x7fc2a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c00b35b0 con 0x7fc2a8005210 2024-04-02T07:49:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a800c090 con 0x7fc2a8004850 2024-04-02T07:49:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a800c250 con 0x7fc2a8003b90 2024-04-02T07:49:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.859+0000 7fc2c5a00700 1 --2- 172.21.15.89:0/2199297478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a8005210 0x7fc2a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.860+0000 7fc2c5000700 1 --2- 172.21.15.89:0/2199297478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a8003b90 0x7fc2a8003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.860+0000 7fc2bfe00700 1 --2- 172.21.15.89:0/2199297478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2a8004850 0x7fc2a8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.860+0000 7fc2c5a00700 1 -- 172.21.15.89:0/2199297478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a8003b90 msgr2=0x7fc2a8003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.860+0000 7fc2c5a00700 1 --2- 172.21.15.89:0/2199297478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a8003b90 0x7fc2a8003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.860+0000 7fc2c5a00700 1 -- 172.21.15.89:0/2199297478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2a8004850 msgr2=0x7fc2a8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.860+0000 7fc2c5a00700 1 --2- 172.21.15.89:0/2199297478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2a8004850 0x7fc2a8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.860+0000 7fc2c5a00700 1 -- 172.21.15.89:0/2199297478 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a80035b0 con 0x7fc2a8005210 2024-04-02T07:49:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.860+0000 7fc2bfe00700 1 --2- 172.21.15.89:0/2199297478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2a8004850 0x7fc2a8004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:49:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.860+0000 7fc2c5a00700 1 -- 172.21.15.89:0/2199297478 reap_dead start 2024-04-02T07:49:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.860+0000 7fc2c5a00700 1 --2- 172.21.15.89:0/2199297478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a8005210 0x7fc2a800ba10 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800a7a0 tx=0x7fc2a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65665c9db0fd0e48 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.860+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8012600 con 0x7fc2a8005210 2024-04-02T07:49:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.860+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b8012780 con 0x7fc2a8005210 2024-04-02T07:49:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.860+0000 7fc2bf400700 1 -- 172.21.15.89:0/2199297478 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8012b30 con 0x7fc2a8005210 2024-04-02T07:49:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.861+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2199297478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a8005210 msgr2=0x7fc2a800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.861+0000 7fc2c7a00700 1 --2- 172.21.15.89:0/2199297478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a8005210 0x7fc2a800ba10 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800a7a0 tx=0x7fc2a8005880 comp rx=0 tx=0).stop 2024-04-02T07:49:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.861+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2199297478 shutdown_connections 2024-04-02T07:49:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.861+0000 7fc2c7a00700 1 --2- 172.21.15.89:0/2199297478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a8005210 0x7fc2a800ba10 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.861+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2199297478 >> 172.21.15.89:0/2199297478 conn(0x7fc2c00cf190 msgr2=0x7fc2c00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.861+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2199297478 shutdown_connections 2024-04-02T07:49:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.861+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2199297478 wait complete. 2024-04-02T07:49:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.863+0000 7fc2c7a00700 1 Processor -- start 2024-04-02T07:49:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.863+0000 7fc2c7a00700 1 -- start start 2024-04-02T07:49:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c7a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a8005210 0x7fc2c01f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c7a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2c00ccd00 0x7fc2c00cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c7a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c01f66d0 0x7fc2c01f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c7a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c00cbc60 con 0x7fc2c01f66d0 2024-04-02T07:49:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c7a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c00cbf40 con 0x7fc2c00ccd00 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c7a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c00cc220 con 0x7fc2a8005210 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c5a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c01f66d0 0x7fc2c01f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c5a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c01f66d0 0x7fc2c01f6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53466/0 (socket says 172.21.15.89:53466) 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c5a00700 1 -- 172.21.15.89:0/2004207989 learned_addr learned my addr 172.21.15.89:0/2004207989 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c5000700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a8005210 0x7fc2c01f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2bfe00700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2c00ccd00 0x7fc2c00cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c5a00700 1 -- 172.21.15.89:0/2004207989 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a8005210 msgr2=0x7fc2c01f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c5a00700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a8005210 0x7fc2c01f4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c5a00700 1 -- 172.21.15.89:0/2004207989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2c00ccd00 msgr2=0x7fc2c00cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c5a00700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2c00ccd00 0x7fc2c00cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2c5a00700 1 -- 172.21.15.89:0/2004207989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c00cc500 con 0x7fc2c01f66d0 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.864+0000 7fc2bfe00700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2c00ccd00 0x7fc2c00cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.865+0000 7fc2c5a00700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c01f66d0 0x7fc2c01f6b80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc2c015d5d0 tx=0x7fc2b8001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.865+0000 7fc2bcc00700 1 -- 172.21.15.89:0/2004207989 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b801e630 con 0x7fc2c01f66d0 2024-04-02T07:49:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.865+0000 7fc2bcc00700 1 -- 172.21.15.89:0/2004207989 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b801e790 con 0x7fc2c01f66d0 2024-04-02T07:49:12.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.865+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2004207989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c00cc7e0 con 0x7fc2c01f66d0 2024-04-02T07:49:12.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.865+0000 7fc2bcc00700 1 -- 172.21.15.89:0/2004207989 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b801eb00 con 0x7fc2c01f66d0 2024-04-02T07:49:12.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.865+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2004207989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c00cd6c0 con 0x7fc2c01f66d0 2024-04-02T07:49:12.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.866+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2004207989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c00cc9d0 con 0x7fc2c01f66d0 2024-04-02T07:49:12.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.867+0000 7fc2bcc00700 1 -- 172.21.15.89:0/2004207989 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc2b801ec60 con 0x7fc2c01f66d0 2024-04-02T07:49:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.868+0000 7fc2bcc00700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc2a006ade0 0x7fc2a006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.868+0000 7fc2bcc00700 1 -- 172.21.15.89:0/2004207989 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc2b808b600 con 0x7fc2c01f66d0 2024-04-02T07:49:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.868+0000 7fc2c5000700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc2a006ade0 0x7fc2a006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.869+0000 7fc2c5000700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc2a006ade0 0x7fc2a006d2a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4007fd0 tx=0x7fc2b4009040 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:12.872+0000 7fc2bcc00700 1 -- 172.21.15.89:0/2004207989 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc2b8056720 con 0x7fc2c01f66d0 2024-04-02T07:49:13.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.118+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2004207989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2c0067720 con 0x7fc2c01f66d0 2024-04-02T07:49:13.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.118+0000 7fc2bcc00700 1 -- 172.21.15.89:0/2004207989 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4737 (secure 0 0 0) 0x7fc2b8017090 con 0x7fc2c01f66d0 2024-04-02T07:49:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:49:13.121 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5710,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3098065397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3098065397},{"type":"v1","addr":"172.21.15.89:6836","nonce":3098065397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5713,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1408411829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1408411829},{"type":"v1","addr":"172.21.15.89:6837","nonce":1408411829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5716,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3266955664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3266955664},{"type":"v1","addr":"172.21.15.113:6834","nonce":3266955664}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:49:07.249043+0000","modified":"2024-04-02T07:49:10.254959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5719":{"gid":5719,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6835/427074923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":427074923},{"type":"v1","addr":"172.21.15.113:6835","nonce":427074923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-02T07:49:13.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.121+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2004207989 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc2a006ade0 msgr2=0x7fc2a006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:13.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.121+0000 7fc2c7a00700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc2a006ade0 0x7fc2a006d2a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4007fd0 tx=0x7fc2b4009040 comp rx=0 tx=0).stop 2024-04-02T07:49:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.121+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2004207989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c01f66d0 msgr2=0x7fc2c01f6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.121+0000 7fc2c7a00700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c01f66d0 0x7fc2c01f6b80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc2c015d5d0 tx=0x7fc2b8001d00 comp rx=0 tx=0).stop 2024-04-02T07:49:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.121+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2004207989 shutdown_connections 2024-04-02T07:49:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.121+0000 7fc2c7a00700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc2a006ade0 0x7fc2a006d2a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.121+0000 7fc2c7a00700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c01f66d0 0x7fc2c01f6b80 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.121+0000 7fc2c7a00700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a8005210 0x7fc2c01f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.121+0000 7fc2c7a00700 1 --2- 172.21.15.89:0/2004207989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2c00ccd00 0x7fc2c00cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.121+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2004207989 >> 172.21.15.89:0/2004207989 conn(0x7fc2c00cf190 msgr2=0x7fc2c01592a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.121+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2004207989 shutdown_connections 2024-04-02T07:49:13.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.121+0000 7fc2c7a00700 1 -- 172.21.15.89:0/2004207989 wait complete. 2024-04-02T07:49:13.125 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 24 2024-04-02T07:49:13.543 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T07:49:07.249043+0000', 'modified': '2024-04-02T07:49:10.254959+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5719': {'gid': 5719, 'name': 'd', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.113:6835/427074923', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6833', 'nonce': 427074923}, {'type': 'v1', 'addr': '172.21.15.113:6835', 'nonce': 427074923}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T07:49:13.544 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T07:49:13.544 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-02T07:49:13.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.737+0000 7f8730a00700 1 Processor -- start 2024-04-02T07:49:13.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.737+0000 7f8730a00700 1 -- start start 2024-04-02T07:49:13.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.738+0000 7f8730a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f872c0d3eb0 0x7f872c0b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.738+0000 7f8730a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c0cf1b0 0x7f872c0cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.739+0000 7f8730a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872c0ced20 con 0x7f872c15d8a0 2024-04-02T07:49:13.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.739+0000 7f8730a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f872c0d4340 con 0x7f872c0d3eb0 2024-04-02T07:49:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.739+0000 7f8730a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872c0ce6d0 con 0x7f872c0cf1b0 2024-04-02T07:49:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.739+0000 7f8729400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f872c0d3eb0 0x7f872c0b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.739+0000 7f8729400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f872c0d3eb0 0x7f872c0b2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53468/0 (socket says 172.21.15.89:53468) 2024-04-02T07:49:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.739+0000 7f8729400700 1 -- 172.21.15.89:0/3566503254 learned_addr learned my addr 172.21.15.89:0/3566503254 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.739+0000 7f872a800700 1 --2- 172.21.15.89:0/3566503254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c0cf1b0 0x7f872c0cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.739+0000 7f8729400700 1 -- 172.21.15.89:0/3566503254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c0cf1b0 msgr2=0x7f872c0cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.739+0000 7f8729400700 1 --2- 172.21.15.89:0/3566503254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c0cf1b0 0x7f872c0cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.739+0000 7f8729400700 1 -- 172.21.15.89:0/3566503254 >> v1:172.21.15.113:6789/0 conn(0x7f872c15d8a0 legacy=0x7f872c06b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.739+0000 7f8729400700 1 -- 172.21.15.89:0/3566503254 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c0ce9b0 con 0x7f872c0d3eb0 2024-04-02T07:49:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.739+0000 7f8729400700 1 --2- 172.21.15.89:0/3566503254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0d3eb0 0x7f872c0b2990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8718006fd0 tx=0x7f8718009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3463c382349cca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:13.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.740+0000 7f8728a00700 1 -- 172.21.15.89:0/3566503254 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8718005e30 con 0x7f872c0d3eb0 2024-04-02T07:49:13.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.740+0000 7f8728a00700 1 -- 172.21.15.89:0/3566503254 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8718006030 con 0x7f872c0d3eb0 2024-04-02T07:49:13.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.740+0000 7f8728a00700 1 -- 172.21.15.89:0/3566503254 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87180063c0 con 0x7f872c0d3eb0 2024-04-02T07:49:13.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.740+0000 7f8730a00700 1 -- 172.21.15.89:0/3566503254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0d3eb0 msgr2=0x7f872c0b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:13.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.740+0000 7f8730a00700 1 --2- 172.21.15.89:0/3566503254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0d3eb0 0x7f872c0b2990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8718006fd0 tx=0x7f8718009bd0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.740+0000 7f8730a00700 1 -- 172.21.15.89:0/3566503254 shutdown_connections 2024-04-02T07:49:13.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.740+0000 7f8730a00700 1 --2- 172.21.15.89:0/3566503254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c0cf1b0 0x7f872c0cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.740+0000 7f8730a00700 1 --2- 172.21.15.89:0/3566503254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c0d3eb0 0x7f872c0b2990 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.740+0000 7f8730a00700 1 -- 172.21.15.89:0/3566503254 >> 172.21.15.89:0/3566503254 conn(0x7f872c08aff0 msgr2=0x7f872c0d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:13.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.740+0000 7f8730a00700 1 -- 172.21.15.89:0/3566503254 shutdown_connections 2024-04-02T07:49:13.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.740+0000 7f8730a00700 1 -- 172.21.15.89:0/3566503254 wait complete. 2024-04-02T07:49:13.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.743+0000 7f8730a00700 1 Processor -- start 2024-04-02T07:49:13.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.743+0000 7f8730a00700 1 -- start start 2024-04-02T07:49:13.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.744+0000 7f8730a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f872c0cf1b0 0x7f872c150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.744+0000 7f8730a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c15d8a0 0x7f872c0c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.744+0000 7f8730a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c159e70 0x7f872c0c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.744+0000 7f8730a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c0c9860 con 0x7f872c15d8a0 2024-04-02T07:49:13.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.744+0000 7f8730a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c0c7190 con 0x7f872c0cf1b0 2024-04-02T07:49:13.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.744+0000 7f8730a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872c0c7470 con 0x7f872c159e70 2024-04-02T07:49:13.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.744+0000 7f8729400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c15d8a0 0x7f872c0c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.744+0000 7f8729400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c15d8a0 0x7f872c0c85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53478/0 (socket says 172.21.15.89:53478) 2024-04-02T07:49:13.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.744+0000 7f8729400700 1 -- 172.21.15.89:0/2417468317 learned_addr learned my addr 172.21.15.89:0/2417468317 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:13.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.745+0000 7f872a800700 1 --2- 172.21.15.89:0/2417468317 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c159e70 0x7f872c0c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.745+0000 7f8729e00700 1 --2- 172.21.15.89:0/2417468317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f872c0cf1b0 0x7f872c150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.745+0000 7f8729400700 1 -- 172.21.15.89:0/2417468317 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c159e70 msgr2=0x7f872c0c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:13.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.745+0000 7f8729400700 1 --2- 172.21.15.89:0/2417468317 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c159e70 0x7f872c0c7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.745+0000 7f8729400700 1 -- 172.21.15.89:0/2417468317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f872c0cf1b0 msgr2=0x7f872c150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:13.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.745+0000 7f8729400700 1 --2- 172.21.15.89:0/2417468317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f872c0cf1b0 0x7f872c150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.745+0000 7f8729400700 1 -- 172.21.15.89:0/2417468317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8718009500 con 0x7f872c15d8a0 2024-04-02T07:49:13.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.745+0000 7f8729400700 1 --2- 172.21.15.89:0/2417468317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c15d8a0 0x7f872c0c85e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8718000c00 tx=0x7f8718004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:13.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.745+0000 7f8722000700 1 -- 172.21.15.89:0/2417468317 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87180067d0 con 0x7f872c15d8a0 2024-04-02T07:49:13.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.745+0000 7f8722000700 1 -- 172.21.15.89:0/2417468317 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8718020070 con 0x7f872c15d8a0 2024-04-02T07:49:13.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.745+0000 7f8722000700 1 -- 172.21.15.89:0/2417468317 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8718006950 con 0x7f872c15d8a0 2024-04-02T07:49:13.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.745+0000 7f8730a00700 1 -- 172.21.15.89:0/2417468317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872c0c9ab0 con 0x7f872c15d8a0 2024-04-02T07:49:13.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.746+0000 7f8730a00700 1 -- 172.21.15.89:0/2417468317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872c0c8b20 con 0x7f872c15d8a0 2024-04-02T07:49:13.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.747+0000 7f8722000700 1 -- 172.21.15.89:0/2417468317 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8718022030 con 0x7f872c15d8a0 2024-04-02T07:49:13.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.748+0000 7f8722000700 1 --2- 172.21.15.89:0/2417468317 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f870c06adb0 0x7f870c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.748+0000 7f8722000700 1 -- 172.21.15.89:0/2417468317 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f871808ab20 con 0x7f872c15d8a0 2024-04-02T07:49:13.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.749+0000 7f8729e00700 1 --2- 172.21.15.89:0/2417468317 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f870c06adb0 0x7f870c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.749+0000 7f8729e00700 1 --2- 172.21.15.89:0/2417468317 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f870c06adb0 0x7f870c06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f871c009960 tx=0x7f871c0090e0 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:13.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.750+0000 7f87445c8b80 1 Processor -- start 2024-04-02T07:49:13.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.751+0000 7f87445c8b80 1 -- start start 2024-04-02T07:49:13.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.751+0000 7f87445c8b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296eb830 0x55da296ebc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.751+0000 7f87445c8b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bebe0 0x55da296c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.751+0000 7f87445c8b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296bcb60 0x55da296e8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.751+0000 7f87445c8b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da29636650 con 0x55da296eb830 2024-04-02T07:49:13.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.751+0000 7f87445c8b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da29617c30 con 0x55da296bebe0 2024-04-02T07:49:13.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.751+0000 7f87445c8b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da29615430 con 0x55da296bcb60 2024-04-02T07:49:13.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.752+0000 7f8729e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296eb830 0x55da296ebc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.752+0000 7f8729e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296eb830 0x55da296ebc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53480/0 (socket says 172.21.15.89:53480) 2024-04-02T07:49:13.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.752+0000 7f8729e00700 1 -- 172.21.15.89:0/2448306146 learned_addr learned my addr 172.21.15.89:0/2448306146 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:13.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.752+0000 7f8729400700 1 --2- 172.21.15.89:0/2448306146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bebe0 0x55da296c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.752+0000 7f872a800700 1 --2- 172.21.15.89:0/2448306146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296bcb60 0x55da296e8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.752+0000 7f8729e00700 1 -- 172.21.15.89:0/2448306146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296bcb60 msgr2=0x55da296e8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:13.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.752+0000 7f8729e00700 1 --2- 172.21.15.89:0/2448306146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296bcb60 0x55da296e8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.752+0000 7f8729e00700 1 -- 172.21.15.89:0/2448306146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bebe0 msgr2=0x55da296c1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:13.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.752+0000 7f8729e00700 1 --2- 172.21.15.89:0/2448306146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bebe0 0x55da296c1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.752+0000 7f8729e00700 1 -- 172.21.15.89:0/2448306146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da29404790 con 0x55da296eb830 2024-04-02T07:49:13.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.752+0000 7f872a800700 1 --2- 172.21.15.89:0/2448306146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296bcb60 0x55da296e8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:49:13.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.752+0000 7f8729400700 1 --2- 172.21.15.89:0/2448306146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bebe0 0x55da296c1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:49:13.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.752+0000 7f8729e00700 1 --2- 172.21.15.89:0/2448306146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296eb830 0x55da296ebc10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f871c00e9e0 tx=0x7f871c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f31bcc077f87963 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:13.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.753+0000 7f8716600700 1 -- 172.21.15.89:0/2448306146 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c023070 con 0x55da296eb830 2024-04-02T07:49:13.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.753+0000 7f8716600700 1 -- 172.21.15.89:0/2448306146 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871c009e30 con 0x55da296eb830 2024-04-02T07:49:13.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.753+0000 7f8716600700 1 -- 172.21.15.89:0/2448306146 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c004020 con 0x55da296eb830 2024-04-02T07:49:13.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.753+0000 7f87445c8b80 1 -- 172.21.15.89:0/2448306146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296eb830 msgr2=0x55da296ebc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:13.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.753+0000 7f87445c8b80 1 --2- 172.21.15.89:0/2448306146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296eb830 0x55da296ebc10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f871c00e9e0 tx=0x7f871c011000 comp rx=0 tx=0).stop 2024-04-02T07:49:13.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.753+0000 7f87445c8b80 1 -- 172.21.15.89:0/2448306146 shutdown_connections 2024-04-02T07:49:13.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.753+0000 7f87445c8b80 1 --2- 172.21.15.89:0/2448306146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296bcb60 0x55da296e8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.753+0000 7f87445c8b80 1 --2- 172.21.15.89:0/2448306146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296eb830 0x55da296ebc10 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.753+0000 7f87445c8b80 1 --2- 172.21.15.89:0/2448306146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bebe0 0x55da296c1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.753+0000 7f87445c8b80 1 -- 172.21.15.89:0/2448306146 >> 172.21.15.89:0/2448306146 conn(0x55da2961f6a0 msgr2=0x55da29706640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:13.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.753+0000 7f87445c8b80 1 -- 172.21.15.89:0/2448306146 shutdown_connections 2024-04-02T07:49:13.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.753+0000 7f87445c8b80 1 -- 172.21.15.89:0/2448306146 wait complete. 2024-04-02T07:49:13.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.754+0000 7f87445c8b80 1 -- start start 2024-04-02T07:49:13.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.754+0000 7f87445c8b80 1 Processor -- start 2024-04-02T07:49:13.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.755+0000 7f87445c8b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bcb60 0x55da296eec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.755+0000 7f87445c8b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29714fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.755+0000 7f87445c8b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296eb830 0x55da29715520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.755+0000 7f87445c8b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da29610d20 con 0x55da296bebe0 2024-04-02T07:49:13.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.755+0000 7f87445c8b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da295ac770 con 0x55da296bcb60 2024-04-02T07:49:13.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.755+0000 7f87445c8b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da295cf6f0 con 0x55da296eb830 2024-04-02T07:49:13.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.755+0000 7f87445c8b80 10 client.0 _ll_register_callbacks cb 0x55da295b65e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:49:13.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f8729400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29714fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f8729400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29714fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53492/0 (socket says 172.21.15.89:53492) 2024-04-02T07:49:13.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f8729400700 1 -- 172.21.15.89:0/1555544469 learned_addr learned my addr 172.21.15.89:0/1555544469 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:13.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f872a800700 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296eb830 0x55da29715520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f8729400700 1 -- 172.21.15.89:0/1555544469 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296eb830 msgr2=0x55da29715520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:13.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f8729400700 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296eb830 0x55da29715520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f8729e00700 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bcb60 0x55da296eec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f8729400700 1 -- 172.21.15.89:0/1555544469 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bcb60 msgr2=0x55da296eec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:13.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f8729400700 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bcb60 0x55da296eec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f8729400700 1 -- 172.21.15.89:0/1555544469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c00e6c0 con 0x55da296bebe0 2024-04-02T07:49:13.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f872a800700 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296eb830 0x55da29715520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:49:13.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f8729e00700 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bcb60 0x55da296eec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:49:13.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f8729400700 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29714fe0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8718059640 tx=0x7f8718059760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d904cf083158a10e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:13.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f870aa00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:49:13.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f870aa00700 1 -- 172.21.15.89:0/1555544469 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87180193b0 con 0x55da296bebe0 2024-04-02T07:49:13.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f870aa00700 1 -- 172.21.15.89:0/1555544469 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8718019510 con 0x55da296bebe0 2024-04-02T07:49:13.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.756+0000 7f87445c8b80 1 -- 172.21.15.89:0/1555544469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55da296b3920 con 0x55da296bebe0 2024-04-02T07:49:13.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.757+0000 7f870aa00700 1 -- 172.21.15.89:0/1555544469 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f871805d0d0 con 0x55da296bebe0 2024-04-02T07:49:13.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.757+0000 7f87445c8b80 10 client.6036 fetch_fsmap learned FSMap version 24 2024-04-02T07:49:13.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.757+0000 7f87445c8b80 1 -- 172.21.15.89:0/1555544469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x55da294f10a0 con 0x55da296bebe0 2024-04-02T07:49:13.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.757+0000 7f870aa00700 1 -- 172.21.15.89:0/1555544469 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55da296b3920 con 0x55da296bebe0 2024-04-02T07:49:13.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.757+0000 7f870aa00700 1 -- 172.21.15.89:0/1555544469 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 24) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f8718059910 con 0x55da296bebe0 2024-04-02T07:49:13.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.757+0000 7f87445c8b80 10 client.6036 fetch_fsmap finished waiting for FSMap version 24 2024-04-02T07:49:13.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.757+0000 7f87445c8b80 10 client.6036 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-02T07:49:13.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.757+0000 7f87445c8b80 20 client.6036 populate_metadata read hostname 'smithi089' 2024-04-02T07:49:13.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.757+0000 7f87445c8b80 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.113:6833/427074923,v1:172.21.15.113:6835/427074923] conn(0x55da296e9850 0x55da29723280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.757+0000 7f87445c8b80 4 client.6036 mds_command: new command op to 5719 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T07:49:13.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.757+0000 7f87445c8b80 1 -- 172.21.15.89:0/1555544469 --> [v2:172.21.15.113:6833/427074923,v1:172.21.15.113:6835/427074923] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55da296906f0 con 0x55da296e9850 2024-04-02T07:49:13.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.757+0000 7f8729e00700 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.113:6833/427074923,v1:172.21.15.113:6835/427074923] conn(0x55da296e9850 0x55da29723280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.758+0000 7f870aa00700 10 client.6036 ms_handle_connect on v2:172.21.15.113:6833/427074923 2024-04-02T07:49:13.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.758+0000 7f8729e00700 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.113:6833/427074923,v1:172.21.15.113:6835/427074923] conn(0x55da296e9850 0x55da29723280 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82f31614be93314a server_cookie=6169613a9538b693 in_seq=0 out_seq=0 2024-04-02T07:49:13.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.759+0000 7f870aa00700 1 -- 172.21.15.89:0/1555544469 <== mds.0 v2:172.21.15.113:6833/427074923 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55da296906f0 con 0x55da296e9850 2024-04-02T07:49:13.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.759+0000 7f870aa00700 10 client.6036 handle_command_reply: tid=0 2024-04-02T07:49:13.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.759+0000 7f87445c8b80 1 client.6036 shutdown 2024-04-02T07:49:13.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.759+0000 7f87445c8b80 1 -- 172.21.15.89:0/1555544469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 msgr2=0x55da29714fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:13.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.759+0000 7f87445c8b80 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29714fe0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8718059640 tx=0x7f8718059760 comp rx=0 tx=0).stop 2024-04-02T07:49:13.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.759+0000 7f87445c8b80 1 -- 172.21.15.89:0/1555544469 shutdown_connections 2024-04-02T07:49:13.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.759+0000 7f87445c8b80 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.113:6833/427074923,v1:172.21.15.113:6835/427074923] conn(0x55da296e9850 0x55da29723280 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.759+0000 7f87445c8b80 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bcb60 0x55da296eec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.759+0000 7f87445c8b80 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296eb830 0x55da29715520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.759+0000 7f87445c8b80 1 --2- 172.21.15.89:0/1555544469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29714fe0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.759+0000 7f870aa00700 0 client.6036 ms_handle_reset on v2:172.21.15.113:6833/427074923 2024-04-02T07:49:13.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.759+0000 7f87445c8b80 1 -- 172.21.15.89:0/1555544469 >> 172.21.15.89:0/1555544469 conn(0x55da2961f6a0 msgr2=0x55da296c0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:13.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.759+0000 7f87445c8b80 1 -- 172.21.15.89:0/1555544469 shutdown_connections 2024-04-02T07:49:13.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.759+0000 7f87445c8b80 1 -- 172.21.15.89:0/1555544469 wait complete. 2024-04-02T07:49:13.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.760+0000 7f87445c8b80 20 client.6036 trim_cache size 0 max 16384 2024-04-02T07:49:13.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.778+0000 7f87445c8b80 1 Processor -- start 2024-04-02T07:49:13.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.778+0000 7f87445c8b80 1 -- start start 2024-04-02T07:49:13.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.779+0000 7f87445c8b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296bcb60 0x55da296c17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.779+0000 7f87445c8b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29703500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.779+0000 7f87445c8b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296eb830 0x55da29703a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.779+0000 7f87445c8b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da29405390 con 0x55da296bebe0 2024-04-02T07:49:13.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.779+0000 7f87445c8b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da297272a0 con 0x55da296eb830 2024-04-02T07:49:13.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.779+0000 7f87445c8b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da296c29c0 con 0x55da296bcb60 2024-04-02T07:49:13.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.780+0000 7f8729400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29703500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.780+0000 7f8729400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29703500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53504/0 (socket says 172.21.15.89:53504) 2024-04-02T07:49:13.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.780+0000 7f8729400700 1 -- 172.21.15.89:0/3050781661 learned_addr learned my addr 172.21.15.89:0/3050781661 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:13.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.780+0000 7f8729400700 1 -- 172.21.15.89:0/3050781661 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296bcb60 msgr2=0x55da296c17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:13.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.780+0000 7f8729400700 1 --2- 172.21.15.89:0/3050781661 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296bcb60 0x55da296c17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.780+0000 7f8729400700 1 -- 172.21.15.89:0/3050781661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296eb830 msgr2=0x55da29703a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:13.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.780+0000 7f8729400700 1 --2- 172.21.15.89:0/3050781661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296eb830 0x55da29703a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.780+0000 7f8729400700 1 -- 172.21.15.89:0/3050781661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da296bc710 con 0x55da296bebe0 2024-04-02T07:49:13.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.780+0000 7f8729400700 1 --2- 172.21.15.89:0/3050781661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29703500 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8718006000 tx=0x7f871805b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e8edfd3b1ef4f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:13.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.780+0000 7f8717000700 1 -- 172.21.15.89:0/3050781661 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8718059910 con 0x55da296bebe0 2024-04-02T07:49:13.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.780+0000 7f8717000700 1 -- 172.21.15.89:0/3050781661 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8718059a70 con 0x55da296bebe0 2024-04-02T07:49:13.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.780+0000 7f8717000700 1 -- 172.21.15.89:0/3050781661 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87180195b0 con 0x55da296bebe0 2024-04-02T07:49:13.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.781+0000 7f87445c8b80 1 -- 172.21.15.89:0/3050781661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 msgr2=0x55da29703500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:13.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.781+0000 7f87445c8b80 1 --2- 172.21.15.89:0/3050781661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29703500 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8718006000 tx=0x7f871805b000 comp rx=0 tx=0).stop 2024-04-02T07:49:13.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.781+0000 7f87445c8b80 1 -- 172.21.15.89:0/3050781661 shutdown_connections 2024-04-02T07:49:13.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.781+0000 7f87445c8b80 1 --2- 172.21.15.89:0/3050781661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296eb830 0x55da29703a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.781+0000 7f87445c8b80 1 --2- 172.21.15.89:0/3050781661 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296bcb60 0x55da296c17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.781+0000 7f87445c8b80 1 --2- 172.21.15.89:0/3050781661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29703500 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.781+0000 7f87445c8b80 1 -- 172.21.15.89:0/3050781661 >> 172.21.15.89:0/3050781661 conn(0x55da2961f6a0 msgr2=0x55da296c0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:13.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.781+0000 7f87445c8b80 1 -- 172.21.15.89:0/3050781661 shutdown_connections 2024-04-02T07:49:13.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.781+0000 7f87445c8b80 1 -- 172.21.15.89:0/3050781661 wait complete. 2024-04-02T07:49:13.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.782+0000 7f87445c8b80 1 -- start start 2024-04-02T07:49:13.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.782+0000 7f87445c8b80 1 Processor -- start 2024-04-02T07:49:13.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.783+0000 7f87445c8b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bcb60 0x55da2976ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.783+0000 7f87445c8b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29703500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.783+0000 7f87445c8b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296eb830 0x55da29703a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.783+0000 7f87445c8b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da29701ae0 con 0x55da296bebe0 2024-04-02T07:49:13.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.783+0000 7f87445c8b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da296edeb0 con 0x55da296bcb60 2024-04-02T07:49:13.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.783+0000 7f87445c8b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da296ee100 con 0x55da296eb830 2024-04-02T07:49:13.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.783+0000 7f87445c8b80 10 client.0 _ll_register_callbacks cb 0x55da295b65e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:49:13.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.783+0000 7f8729400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29703500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f8729400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29703500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53520/0 (socket says 172.21.15.89:53520) 2024-04-02T07:49:13.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f8729400700 1 -- 172.21.15.89:0/3629823478 learned_addr learned my addr 172.21.15.89:0/3629823478 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:13.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f872a800700 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296eb830 0x55da29703a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f8729e00700 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bcb60 0x55da2976ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f8729400700 1 -- 172.21.15.89:0/3629823478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296eb830 msgr2=0x55da29703a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:13.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f8729400700 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296eb830 0x55da29703a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f8729400700 1 -- 172.21.15.89:0/3629823478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bcb60 msgr2=0x55da2976ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:13.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f8729400700 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bcb60 0x55da2976ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f8729400700 1 -- 172.21.15.89:0/3629823478 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c00e6c0 con 0x55da296bebe0 2024-04-02T07:49:13.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f872a800700 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296eb830 0x55da29703a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:49:13.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f8729e00700 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bcb60 0x55da2976ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:49:13.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f8729400700 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29703500 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f871800b330 tx=0x7f871800b2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e563072afb4ed0dc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:13.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f870aa00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:49:13.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f870aa00700 1 -- 172.21.15.89:0/3629823478 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871805d320 con 0x55da296bebe0 2024-04-02T07:49:13.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f870aa00700 1 -- 172.21.15.89:0/3629823478 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f87180193b0 con 0x55da296bebe0 2024-04-02T07:49:13.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f87445c8b80 1 -- 172.21.15.89:0/3629823478 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f871c009e30 con 0x55da296bebe0 2024-04-02T07:49:13.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f870aa00700 1 -- 172.21.15.89:0/3629823478 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8718019570 con 0x55da296bebe0 2024-04-02T07:49:13.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f87445c8b80 10 client.6042 fetch_fsmap learned FSMap version 24 2024-04-02T07:49:13.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.784+0000 7f87445c8b80 1 -- 172.21.15.89:0/3629823478 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x55da295d7010 con 0x55da296bebe0 2024-04-02T07:49:13.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.785+0000 7f870aa00700 1 -- 172.21.15.89:0/3629823478 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871c009e30 con 0x55da296bebe0 2024-04-02T07:49:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.785+0000 7f870aa00700 1 -- 172.21.15.89:0/3629823478 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 24) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f8718058ba0 con 0x55da296bebe0 2024-04-02T07:49:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.785+0000 7f87445c8b80 10 client.6042 fetch_fsmap finished waiting for FSMap version 24 2024-04-02T07:49:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.785+0000 7f87445c8b80 10 client.6042 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-02T07:49:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.785+0000 7f87445c8b80 20 client.6042 populate_metadata read hostname 'smithi089' 2024-04-02T07:49:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.785+0000 7f87445c8b80 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.113:6833/427074923,v1:172.21.15.113:6835/427074923] conn(0x55da2976d810 0x55da2976dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.785+0000 7f87445c8b80 4 client.6042 mds_command: new command op to 5719 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T07:49:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.785+0000 7f87445c8b80 1 -- 172.21.15.89:0/3629823478 --> [v2:172.21.15.113:6833/427074923,v1:172.21.15.113:6835/427074923] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55da2944a040 con 0x55da2976d810 2024-04-02T07:49:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.785+0000 7f8729e00700 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.113:6833/427074923,v1:172.21.15.113:6835/427074923] conn(0x55da2976d810 0x55da2976dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f8729e00700 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.113:6833/427074923,v1:172.21.15.113:6835/427074923] conn(0x55da2976d810 0x55da2976dbf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81001344d3e3ef2a server_cookie=d098cd9afd652643 in_seq=0 out_seq=0 2024-04-02T07:49:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f870aa00700 10 client.6042 ms_handle_connect on v2:172.21.15.113:6833/427074923 2024-04-02T07:49:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f870aa00700 1 -- 172.21.15.89:0/3629823478 <== mds.0 v2:172.21.15.113:6833/427074923 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55da2944a040 con 0x55da2976d810 2024-04-02T07:49:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f870aa00700 10 client.6042 handle_command_reply: tid=0 2024-04-02T07:49:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f87445c8b80 1 client.6042 shutdown 2024-04-02T07:49:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f87445c8b80 1 -- 172.21.15.89:0/3629823478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 msgr2=0x55da29703500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f87445c8b80 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29703500 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f871800b330 tx=0x7f871800b2a0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f87445c8b80 1 -- 172.21.15.89:0/3629823478 shutdown_connections 2024-04-02T07:49:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f87445c8b80 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.113:6833/427074923,v1:172.21.15.113:6835/427074923] conn(0x55da2976d810 0x55da2976dbf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f87445c8b80 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da296bcb60 0x55da2976ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f87445c8b80 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55da296eb830 0x55da29703a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f87445c8b80 1 --2- 172.21.15.89:0/3629823478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da296bebe0 0x55da29703500 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f87445c8b80 1 -- 172.21.15.89:0/3629823478 >> 172.21.15.89:0/3629823478 conn(0x55da2961f6a0 msgr2=0x55da29727530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f870aa00700 0 client.6042 ms_handle_reset on v2:172.21.15.113:6833/427074923 2024-04-02T07:49:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f87445c8b80 1 -- 172.21.15.89:0/3629823478 shutdown_connections 2024-04-02T07:49:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.786+0000 7f87445c8b80 1 -- 172.21.15.89:0/3629823478 wait complete. 2024-04-02T07:49:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.787+0000 7f87445c8b80 20 client.6042 trim_cache size 0 max 16384 2024-04-02T07:49:13.794 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:49:13.794 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T07:49:13.794 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T07:49:13.794 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 5719, 2024-04-02T07:49:13.794 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T07:49:13.794 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T07:49:13.794 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T07:49:13.794 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 6.5338943450000002, 2024-04-02T07:49:13.794 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 24, 2024-04-02T07:49:13.794 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 66, 2024-04-02T07:49:13.794 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 66, 2024-04-02T07:49:13.794 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 11.994233502 2024-04-02T07:49:13.795 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:49:13.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.787+0000 7f8730a00700 1 -- 172.21.15.89:0/2417468317 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f870c06adb0 msgr2=0x7f870c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:13.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.787+0000 7f8730a00700 1 --2- 172.21.15.89:0/2417468317 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f870c06adb0 0x7f870c06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f871c009960 tx=0x7f871c0090e0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.787+0000 7f8730a00700 1 -- 172.21.15.89:0/2417468317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c15d8a0 msgr2=0x7f872c0c85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:13.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.787+0000 7f8730a00700 1 --2- 172.21.15.89:0/2417468317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c15d8a0 0x7f872c0c85e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8718000c00 tx=0x7f8718004750 comp rx=0 tx=0).stop 2024-04-02T07:49:13.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.787+0000 7f8730a00700 1 -- 172.21.15.89:0/2417468317 shutdown_connections 2024-04-02T07:49:13.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.787+0000 7f8730a00700 1 --2- 172.21.15.89:0/2417468317 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f870c06adb0 0x7f870c06d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.787+0000 7f8730a00700 1 --2- 172.21.15.89:0/2417468317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f872c0cf1b0 0x7f872c150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.787+0000 7f8730a00700 1 --2- 172.21.15.89:0/2417468317 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f872c159e70 0x7f872c0c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.787+0000 7f8730a00700 1 --2- 172.21.15.89:0/2417468317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f872c15d8a0 0x7f872c0c85e0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:13.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.787+0000 7f8730a00700 1 -- 172.21.15.89:0/2417468317 >> 172.21.15.89:0/2417468317 conn(0x7f872c08aff0 msgr2=0x7f872c15df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:13.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.787+0000 7f8730a00700 1 -- 172.21.15.89:0/2417468317 shutdown_connections 2024-04-02T07:49:13.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.787+0000 7f8730a00700 1 -- 172.21.15.89:0/2417468317 wait complete. 2024-04-02T07:49:13.808 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T07:49:14.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.998+0000 7f8d21a00700 1 Processor -- start 2024-04-02T07:49:14.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.998+0000 7f8d21a00700 1 -- start start 2024-04-02T07:49:14.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.999+0000 7f8d21a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8d1c160f10 0x7f8d1c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:14.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.999+0000 7f8d21a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8d1c0cf230 0x7f8d1c0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:14.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.999+0000 7f8d21a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1c0ceda0 con 0x7f8d1c161590 2024-04-02T07:49:14.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.999+0000 7f8d21a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d1c0d23d0 con 0x7f8d1c0cf230 2024-04-02T07:49:14.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.999+0000 7f8d21a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d1c1612f0 con 0x7f8d1c160f10 2024-04-02T07:49:14.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.999+0000 7f8d1a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8d1c0cf230 0x7f8d1c0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:14.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.999+0000 7f8d1a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8d1c0cf230 0x7f8d1c0d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53534/0 (socket says 172.21.15.89:53534) 2024-04-02T07:49:14.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.999+0000 7f8d1a000700 1 -- 172.21.15.89:0/2782824200 learned_addr learned my addr 172.21.15.89:0/2782824200 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:14.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:13.999+0000 7f8d1aa00700 1 --2- 172.21.15.89:0/2782824200 >> v2:172.21.15.113:3300/0 conn(0x7f8d1c160f10 0x7f8d1c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:14.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.000+0000 7f8d1a000700 1 -- 172.21.15.89:0/2782824200 >> v2:172.21.15.113:3300/0 conn(0x7f8d1c160f10 msgr2=0x7f8d1c0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:14.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.000+0000 7f8d1a000700 1 --2- 172.21.15.89:0/2782824200 >> v2:172.21.15.113:3300/0 conn(0x7f8d1c160f10 0x7f8d1c0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:14.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.000+0000 7f8d1a000700 1 -- 172.21.15.89:0/2782824200 >> v1:172.21.15.113:6789/0 conn(0x7f8d1c161590 legacy=0x7f8d1c0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:14.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.000+0000 7f8d1a000700 1 -- 172.21.15.89:0/2782824200 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c0ce470 con 0x7f8d1c0cf230 2024-04-02T07:49:14.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.000+0000 7f8d1a000700 1 --2- 172.21.15.89:0/2782824200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d1c0cf230 0x7f8d1c0d1e90 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c0d1080 tx=0x7f8d08009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a29452cae7b9ec server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:14.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.000+0000 7f8d19600700 1 -- 172.21.15.89:0/2782824200 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08011830 con 0x7f8d1c0cf230 2024-04-02T07:49:14.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.000+0000 7f8d19600700 1 -- 172.21.15.89:0/2782824200 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d08011990 con 0x7f8d1c0cf230 2024-04-02T07:49:14.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.000+0000 7f8d19600700 1 -- 172.21.15.89:0/2782824200 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08011d20 con 0x7f8d1c0cf230 2024-04-02T07:49:14.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.001+0000 7f8d21a00700 1 -- 172.21.15.89:0/2782824200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d1c0cf230 msgr2=0x7f8d1c0d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:14.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.001+0000 7f8d21a00700 1 --2- 172.21.15.89:0/2782824200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d1c0cf230 0x7f8d1c0d1e90 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c0d1080 tx=0x7f8d08009bd0 comp rx=0 tx=0).stop 2024-04-02T07:49:14.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.001+0000 7f8d21a00700 1 -- 172.21.15.89:0/2782824200 shutdown_connections 2024-04-02T07:49:14.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.001+0000 7f8d21a00700 1 --2- 172.21.15.89:0/2782824200 >> v2:172.21.15.113:3300/0 conn(0x7f8d1c160f10 0x7f8d1c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:14.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.001+0000 7f8d21a00700 1 --2- 172.21.15.89:0/2782824200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d1c0cf230 0x7f8d1c0d1e90 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:14.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.001+0000 7f8d21a00700 1 -- 172.21.15.89:0/2782824200 >> 172.21.15.89:0/2782824200 conn(0x7f8d1c08aff0 msgr2=0x7f8d1c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:14.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.001+0000 7f8d21a00700 1 -- 172.21.15.89:0/2782824200 shutdown_connections 2024-04-02T07:49:14.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.001+0000 7f8d21a00700 1 -- 172.21.15.89:0/2782824200 wait complete. 2024-04-02T07:49:14.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.002+0000 7f8d21a00700 1 Processor -- start 2024-04-02T07:49:14.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.003+0000 7f8d21a00700 1 -- start start 2024-04-02T07:49:14.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.003+0000 7f8d21a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d1c0cf230 0x7f8d1c15e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:14.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.003+0000 7f8d21a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d1c160f10 0x7f8d1c1f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:14.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.003+0000 7f8d21a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d1c161590 0x7f8d1c0b8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:14.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.003+0000 7f8d21a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c0b6350 con 0x7f8d1c160f10 2024-04-02T07:49:14.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.003+0000 7f8d21a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c15e9c0 con 0x7f8d1c0cf230 2024-04-02T07:49:14.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.003+0000 7f8d21a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c15eca0 con 0x7f8d1c161590 2024-04-02T07:49:14.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.004+0000 7f8d1a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d1c160f10 0x7f8d1c1f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:14.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.004+0000 7f8d1a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d1c160f10 0x7f8d1c1f2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53544/0 (socket says 172.21.15.89:53544) 2024-04-02T07:49:14.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.004+0000 7f8d1a000700 1 -- 172.21.15.89:0/4265588901 learned_addr learned my addr 172.21.15.89:0/4265588901 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:14.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.004+0000 7f8d1aa00700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d1c0cf230 0x7f8d1c15e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:14.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.004+0000 7f8d1b400700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d1c161590 0x7f8d1c0b8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:14.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.004+0000 7f8d1a000700 1 -- 172.21.15.89:0/4265588901 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d1c161590 msgr2=0x7f8d1c0b8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:14.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.004+0000 7f8d1a000700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d1c161590 0x7f8d1c0b8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:14.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.004+0000 7f8d1a000700 1 -- 172.21.15.89:0/4265588901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d1c0cf230 msgr2=0x7f8d1c15e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:14.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.004+0000 7f8d1a000700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d1c0cf230 0x7f8d1c15e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:14.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.004+0000 7f8d1a000700 1 -- 172.21.15.89:0/4265588901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d08009550 con 0x7f8d1c160f10 2024-04-02T07:49:14.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.004+0000 7f8d1b400700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d1c161590 0x7f8d1c0b8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:49:14.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.004+0000 7f8d1a000700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d1c160f10 0x7f8d1c1f2c10 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009130 tx=0x7f8d08004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:14.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.005+0000 7f8d12a00700 1 -- 172.21.15.89:0/4265588901 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08004590 con 0x7f8d1c160f10 2024-04-02T07:49:14.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.005+0000 7f8d12a00700 1 -- 172.21.15.89:0/4265588901 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0801e070 con 0x7f8d1c160f10 2024-04-02T07:49:14.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.005+0000 7f8d21a00700 1 -- 172.21.15.89:0/4265588901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d1c15ef80 con 0x7f8d1c160f10 2024-04-02T07:49:14.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.005+0000 7f8d12a00700 1 -- 172.21.15.89:0/4265588901 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d080038a0 con 0x7f8d1c160f10 2024-04-02T07:49:14.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.005+0000 7f8d21a00700 1 -- 172.21.15.89:0/4265588901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d1c1f3150 con 0x7f8d1c160f10 2024-04-02T07:49:14.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.006+0000 7f8d21a00700 1 -- 172.21.15.89:0/4265588901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d1c0b64c0 con 0x7f8d1c160f10 2024-04-02T07:49:14.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.007+0000 7f8d12a00700 1 -- 172.21.15.89:0/4265588901 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8d08003a00 con 0x7f8d1c160f10 2024-04-02T07:49:14.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.008+0000 7f8d12a00700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8cfc06adb0 0x7f8cfc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:14.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.008+0000 7f8d12a00700 1 -- 172.21.15.89:0/4265588901 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8d0808a4f0 con 0x7f8d1c160f10 2024-04-02T07:49:14.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.008+0000 7f8d1aa00700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8cfc06adb0 0x7f8cfc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:14.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.008+0000 7f8d1aa00700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8cfc06adb0 0x7f8cfc06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c009b40 tx=0x7f8d0c009210 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:14.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.012+0000 7f8d12a00700 1 -- 172.21.15.89:0/4265588901 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8d1c0b64c0 con 0x7f8d1c160f10 2024-04-02T07:49:14.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.258+0000 7f8d21a00700 1 -- 172.21.15.89:0/4265588901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8d1c0b6490 con 0x7f8d1c160f10 2024-04-02T07:49:14.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.258+0000 7f8d12a00700 1 -- 172.21.15.89:0/4265588901 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8d08020060 con 0x7f8d1c160f10 2024-04-02T07:49:14.260 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T07:49:14.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.260+0000 7f8d21a00700 1 -- 172.21.15.89:0/4265588901 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8cfc06adb0 msgr2=0x7f8cfc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:14.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.260+0000 7f8d21a00700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8cfc06adb0 0x7f8cfc06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c009b40 tx=0x7f8d0c009210 comp rx=0 tx=0).stop 2024-04-02T07:49:14.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.260+0000 7f8d21a00700 1 -- 172.21.15.89:0/4265588901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d1c160f10 msgr2=0x7f8d1c1f2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:14.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.260+0000 7f8d21a00700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d1c160f10 0x7f8d1c1f2c10 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009130 tx=0x7f8d08004750 comp rx=0 tx=0).stop 2024-04-02T07:49:14.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.260+0000 7f8d21a00700 1 -- 172.21.15.89:0/4265588901 shutdown_connections 2024-04-02T07:49:14.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.260+0000 7f8d21a00700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8cfc06adb0 0x7f8cfc06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:14.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.260+0000 7f8d21a00700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d1c0cf230 0x7f8d1c15e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:14.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.260+0000 7f8d21a00700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d1c161590 0x7f8d1c0b8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:14.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.260+0000 7f8d21a00700 1 --2- 172.21.15.89:0/4265588901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d1c160f10 0x7f8d1c1f2c10 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:14.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.260+0000 7f8d21a00700 1 -- 172.21.15.89:0/4265588901 >> 172.21.15.89:0/4265588901 conn(0x7f8d1c08aff0 msgr2=0x7f8d1c0caf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:14.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.261+0000 7f8d21a00700 1 -- 172.21.15.89:0/4265588901 shutdown_connections 2024-04-02T07:49:14.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:14.261+0000 7f8d21a00700 1 -- 172.21.15.89:0/4265588901 wait complete. 2024-04-02T07:49:14.280 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T07:49:14.280 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T07:49:14.281 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T07:49:14.281 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T07:49:14.281 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T07:49:14.281 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T07:49:14.281 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T07:49:14.281 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T07:49:14.281 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:49:14.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:14.281 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T07:49:14.326 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:49:14.327 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:49:14.327 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:49:14.327 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:49:14.327 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T07:49:14.327 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:49:14.327 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:49:14.327 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:49:14.327 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T07:49:14.327 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:49:14.327 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:49:14.327 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T07:49:14.327 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 41674sec preferred_lft 41674sec 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:49:14.328 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T07:49:14.329 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T07:49:14.329 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:49:14.329 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:49:14.329 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T07:49:14.329 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T07:49:14.329 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:49:14.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:14.330 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T07:49:14.434 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T07:49:14.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:14.437 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T07:49:14.484 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T07:49:14.484 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T07:49:14.484 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T07:49:14.485 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T07:49:14.485 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T07:49:14.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:14.486 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:49:14.486 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:49:14.486 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T07:49:14.486 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T07:49:14.486 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T07:49:14.486 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:49:14.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:14.645 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T07:49:14.690 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T07:49:14.690 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T07:49:14.691 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:49:14.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:14.691 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T07:49:14.709 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:49:14.737 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:49:14.801 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:49:14.801 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T07:49:14.910 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:49:14.911 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:49:14.911 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:49:14.995 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T07:49:14.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:14.995 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:49:15.134 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:49:15.135 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:49:15.163 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:49:15.163 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:49:15.163 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T07:49:15.163 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T07:49:15.163 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T07:49:15.167 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T07:49:15.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:15.168 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:49:15.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:15.236 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:49:15.342 INFO:teuthology.orchestra.run.smithi089.stdout:client5830 2024-04-02T07:49:15.343 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:49:15.343 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:49:15.386 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T07:49:15.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.618+0000 7f25e7a00700 1 Processor -- start 2024-04-02T07:49:15.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.618+0000 7f25e7a00700 1 -- start start 2024-04-02T07:49:15.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.619+0000 7f25e7a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f25e00ce230 0x7f25e00ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:15.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.619+0000 7f25e7a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25e00b2070 0x7f25e00b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:15.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.619+0000 7f25e7a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e00709c0 con 0x7f25e015d800 2024-04-02T07:49:15.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.619+0000 7f25e7a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25e00b40f0 con 0x7f25e00ce230 2024-04-02T07:49:15.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.619+0000 7f25e7a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e0070c80 con 0x7f25e00b2070 2024-04-02T07:49:15.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.619+0000 7f25dfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f25e00ce230 0x7f25e00ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:15.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.619+0000 7f25e5000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f25e015d800 0x7f25e00725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51506/0 (socket says 172.21.15.89:51506) 2024-04-02T07:49:15.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.619+0000 7f25e5000700 1 -- 172.21.15.89:0/2417796105 learned_addr learned my addr 172.21.15.89:0/2417796105 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:15.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.620+0000 7f25dfe00700 1 -- 172.21.15.89:0/2417796105 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25e00b2070 msgr2=0x7f25e00b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:15.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.620+0000 7f25dfe00700 1 --2- 172.21.15.89:0/2417796105 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25e00b2070 0x7f25e00b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:15.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.620+0000 7f25dfe00700 1 -- 172.21.15.89:0/2417796105 >> v1:172.21.15.89:6789/0 conn(0x7f25e015d800 legacy=0x7f25e00725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:15.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.620+0000 7f25dfe00700 1 -- 172.21.15.89:0/2417796105 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e0070f60 con 0x7f25e00ce230 2024-04-02T07:49:15.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.620+0000 7f25df400700 1 -- 172.21.15.89:0/2417796105 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307972135 0 0) 0x7f25e00709c0 con 0x7f25e015d800 2024-04-02T07:49:15.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.620+0000 7f25dfe00700 1 --2- 172.21.15.89:0/2417796105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25e00ce230 0x7f25e00ce6a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009840 tx=0x7f25cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37fdb77d0469e00 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.620+0000 7f25df400700 1 -- 172.21.15.89:0/2417796105 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc011830 con 0x7f25e00ce230 2024-04-02T07:49:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.620+0000 7f25df400700 1 -- 172.21.15.89:0/2417796105 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25cc011990 con 0x7f25e00ce230 2024-04-02T07:49:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.620+0000 7f25df400700 1 -- 172.21.15.89:0/2417796105 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc011d20 con 0x7f25e00ce230 2024-04-02T07:49:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.621+0000 7f25e7a00700 1 -- 172.21.15.89:0/2417796105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25e00ce230 msgr2=0x7f25e00ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.621+0000 7f25e7a00700 1 --2- 172.21.15.89:0/2417796105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25e00ce230 0x7f25e00ce6a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009840 tx=0x7f25cc009b50 comp rx=0 tx=0).stop 2024-04-02T07:49:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.621+0000 7f25e7a00700 1 -- 172.21.15.89:0/2417796105 shutdown_connections 2024-04-02T07:49:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.621+0000 7f25e7a00700 1 --2- 172.21.15.89:0/2417796105 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25e00b2070 0x7f25e00b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.621+0000 7f25e7a00700 1 --2- 172.21.15.89:0/2417796105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25e00ce230 0x7f25e00ce6a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.621+0000 7f25e7a00700 1 -- 172.21.15.89:0/2417796105 >> 172.21.15.89:0/2417796105 conn(0x7f25e00b0d70 msgr2=0x7f25e00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.621+0000 7f25e7a00700 1 -- 172.21.15.89:0/2417796105 shutdown_connections 2024-04-02T07:49:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.621+0000 7f25e7a00700 1 -- 172.21.15.89:0/2417796105 wait complete. 2024-04-02T07:49:15.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.623+0000 7f25e7a00700 1 Processor -- start 2024-04-02T07:49:15.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.623+0000 7f25e7a00700 1 -- start start 2024-04-02T07:49:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e7a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25e00b2070 0x7f25e014f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e7a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25e00ce230 0x7f25e01519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e7a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e015d800 0x7f25e0155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e7a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e014fb70 con 0x7f25e00b2070 2024-04-02T07:49:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e7a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e015b340 con 0x7f25e015d800 2024-04-02T07:49:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e7a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e015b620 con 0x7f25e00ce230 2024-04-02T07:49:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e5000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25e00b2070 0x7f25e014f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e5000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25e00b2070 0x7f25e014f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53572/0 (socket says 172.21.15.89:53572) 2024-04-02T07:49:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e5000700 1 -- 172.21.15.89:0/904559953 learned_addr learned my addr 172.21.15.89:0/904559953 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25dfe00700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25e00ce230 0x7f25e01519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e5000700 1 -- 172.21.15.89:0/904559953 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25e00ce230 msgr2=0x7f25e01519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e5000700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25e00ce230 0x7f25e01519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e5000700 1 -- 172.21.15.89:0/904559953 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e015d800 msgr2=0x7f25e0155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e5a00700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e015d800 0x7f25e0155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e5000700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e015d800 0x7f25e0155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e5000700 1 -- 172.21.15.89:0/904559953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc0094a0 con 0x7f25e00b2070 2024-04-02T07:49:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e5a00700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e015d800 0x7f25e0155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:49:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.624+0000 7f25e5000700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25e00b2070 0x7f25e014f630 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f25d400c550 tx=0x7f25d400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.625+0000 7f25dcc00700 1 -- 172.21.15.89:0/904559953 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d4004c80 con 0x7f25e00b2070 2024-04-02T07:49:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.625+0000 7f25dcc00700 1 -- 172.21.15.89:0/904559953 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d4004e00 con 0x7f25e00b2070 2024-04-02T07:49:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.625+0000 7f25dcc00700 1 -- 172.21.15.89:0/904559953 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d4007310 con 0x7f25e00b2070 2024-04-02T07:49:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.625+0000 7f25e7a00700 1 -- 172.21.15.89:0/904559953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e0152450 con 0x7f25e00b2070 2024-04-02T07:49:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.625+0000 7f25e7a00700 1 -- 172.21.15.89:0/904559953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e0152a10 con 0x7f25e00b2070 2024-04-02T07:49:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.626+0000 7f25e7a00700 1 -- 172.21.15.89:0/904559953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e0152c00 con 0x7f25e00b2070 2024-04-02T07:49:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.627+0000 7f25dcc00700 1 -- 172.21.15.89:0/904559953 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f25d4007470 con 0x7f25e00b2070 2024-04-02T07:49:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.627+0000 7f25dcc00700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f25c006adb0 0x7f25c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:15.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.628+0000 7f25dcc00700 1 -- 172.21.15.89:0/904559953 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f25d4089270 con 0x7f25e00b2070 2024-04-02T07:49:15.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.628+0000 7f25dfe00700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f25c006adb0 0x7f25c006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:15.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.628+0000 7f25dfe00700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f25c006adb0 0x7f25c006d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f25cc0090b0 tx=0x7f25cc008030 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:15.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.632+0000 7f25dcc00700 1 -- 172.21.15.89:0/904559953 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f25d4054270 con 0x7f25e00b2070 2024-04-02T07:49:15.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.862+0000 7f25e7a00700 1 -- 172.21.15.89:0/904559953 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f25e00cd9e0 con 0x7f25c006adb0 2024-04-02T07:49:15.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.864+0000 7f25dcc00700 1 -- 172.21.15.89:0/904559953 <== mgr.5661 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f25cc011c00 con 0x7f25c006adb0 2024-04-02T07:49:15.866 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:49:15.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:49:15.866 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T07:49:15.866 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:49:15.866 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:49:15.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.866+0000 7f25e7a00700 1 -- 172.21.15.89:0/904559953 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f25c006adb0 msgr2=0x7f25c006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:15.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.866+0000 7f25e7a00700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f25c006adb0 0x7f25c006d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f25cc0090b0 tx=0x7f25cc008030 comp rx=0 tx=0).stop 2024-04-02T07:49:15.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.866+0000 7f25e7a00700 1 -- 172.21.15.89:0/904559953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25e00b2070 msgr2=0x7f25e014f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:15.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.866+0000 7f25e7a00700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25e00b2070 0x7f25e014f630 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f25d400c550 tx=0x7f25d400c910 comp rx=0 tx=0).stop 2024-04-02T07:49:15.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.866+0000 7f25e7a00700 1 -- 172.21.15.89:0/904559953 shutdown_connections 2024-04-02T07:49:15.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.866+0000 7f25e7a00700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f25c006adb0 0x7f25c006d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:15.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.866+0000 7f25e7a00700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e015d800 0x7f25e0155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:15.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.866+0000 7f25e7a00700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25e00b2070 0x7f25e014f630 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:15.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.866+0000 7f25e7a00700 1 --2- 172.21.15.89:0/904559953 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25e00ce230 0x7f25e01519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:15.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.866+0000 7f25e7a00700 1 -- 172.21.15.89:0/904559953 >> 172.21.15.89:0/904559953 conn(0x7f25e00b0d70 msgr2=0x7f25e0153630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:15.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.866+0000 7f25e7a00700 1 -- 172.21.15.89:0/904559953 shutdown_connections 2024-04-02T07:49:15.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:15.866+0000 7f25e7a00700 1 -- 172.21.15.89:0/904559953 wait complete. 2024-04-02T07:49:15.886 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T07:49:16.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.078+0000 7f1215c00700 1 Processor -- start 2024-04-02T07:49:16.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.079+0000 7f1215c00700 1 -- start start 2024-04-02T07:49:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.080+0000 7f1215c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f121015d9d0 0x7f12100d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.080+0000 7f1215c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1210161420 con 0x7f12100d2540 2024-04-02T07:49:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.080+0000 7f1215c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1210161700 con 0x7f12100d16a0 2024-04-02T07:49:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.080+0000 7f1215c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f121015edc0 con 0x7f121015d9d0 2024-04-02T07:49:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.080+0000 7f120f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f12100d2540 0x7f12100ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51518/0 (socket says 172.21.15.89:51518) 2024-04-02T07:49:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.080+0000 7f120f400700 1 -- 172.21.15.89:0/2693481149 learned_addr learned my addr 172.21.15.89:0/2693481149 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:16.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.080+0000 7f120ea00700 1 --2- 172.21.15.89:0/2693481149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f121015d9d0 0x7f12100d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:16.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.080+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811497408 0 0) 0x7f1210161420 con 0x7f12100d2540 2024-04-02T07:49:16.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f40035b0 con 0x7f12100d2540 2024-04-02T07:49:16.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545735974 0 0) 0x7f1210161700 con 0x7f12100d16a0 2024-04-02T07:49:16.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1210161420 con 0x7f12100d16a0 2024-04-02T07:49:16.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2628848397 0 0) 0x7f11f40035b0 con 0x7f12100d2540 2024-04-02T07:49:16.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f121015d9d0 msgr2=0x7f12100d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:16.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 --2- 172.21.15.89:0/2693481149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f121015d9d0 0x7f12100d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:16.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 >> v1:172.21.15.113:6789/0 conn(0x7f12100d16a0 legacy=0x7f12100d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:16.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12101619e0 con 0x7f12100d2540 2024-04-02T07:49:16.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f12080019f0 con 0x7f12100d2540 2024-04-02T07:49:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120ea00700 1 --2- 172.21.15.89:0/2693481149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f121015d9d0 0x7f12100d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:49:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 >> v1:172.21.15.89:6789/0 conn(0x7f12100d2540 legacy=0x7f12100ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 --2- 172.21.15.89:0/2693481149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f121015d9d0 0x7f11f4003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 --2- 172.21.15.89:0/2693481149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11f4004100 0x7f11f4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 --2- 172.21.15.89:0/2693481149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f4004d90 0x7f11f4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f400bac0 con 0x7f11f4004d90 2024-04-02T07:49:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f400bc00 con 0x7f11f4004100 2024-04-02T07:49:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f400bdc0 con 0x7f121015d9d0 2024-04-02T07:49:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120f400700 1 --2- 172.21.15.89:0/2693481149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f4004d90 0x7f11f4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120ea00700 1 --2- 172.21.15.89:0/2693481149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f121015d9d0 0x7f11f4003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.081+0000 7f120e000700 1 --2- 172.21.15.89:0/2693481149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11f4004100 0x7f11f4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.082+0000 7f120f400700 1 -- 172.21.15.89:0/2693481149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f121015d9d0 msgr2=0x7f11f4003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.082+0000 7f120f400700 1 --2- 172.21.15.89:0/2693481149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f121015d9d0 0x7f11f4003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.082+0000 7f120f400700 1 -- 172.21.15.89:0/2693481149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11f4004100 msgr2=0x7f11f4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.082+0000 7f120f400700 1 --2- 172.21.15.89:0/2693481149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11f4004100 0x7f11f4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.082+0000 7f120f400700 1 -- 172.21.15.89:0/2693481149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12000094a0 con 0x7f11f4004d90 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.082+0000 7f120f400700 1 --2- 172.21.15.89:0/2693481149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f4004d90 0x7f11f4005210 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f121008c320 tx=0x7f11f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9fc92e702327e55 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.082+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12080072d0 con 0x7f11f4004d90 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.082+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1208007450 con 0x7f11f4004d90 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.082+0000 7f120d600700 1 -- 172.21.15.89:0/2693481149 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1208007800 con 0x7f11f4004d90 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.083+0000 7f1215c00700 1 -- 172.21.15.89:0/2693481149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f4004d90 msgr2=0x7f11f4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.083+0000 7f1215c00700 1 --2- 172.21.15.89:0/2693481149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f4004d90 0x7f11f4005210 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f121008c320 tx=0x7f11f4005880 comp rx=0 tx=0).stop 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.083+0000 7f120f400700 1 -- 172.21.15.89:0/2693481149 reap_dead start 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.083+0000 7f1215c00700 1 -- 172.21.15.89:0/2693481149 shutdown_connections 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.083+0000 7f1215c00700 1 -- 172.21.15.89:0/2693481149 >> 172.21.15.89:0/2693481149 conn(0x7f121008aff0 msgr2=0x7f12100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.083+0000 7f1215c00700 1 -- 172.21.15.89:0/2693481149 shutdown_connections 2024-04-02T07:49:16.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.083+0000 7f1215c00700 1 -- 172.21.15.89:0/2693481149 wait complete. 2024-04-02T07:49:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.084+0000 7f1215c00700 1 Processor -- start 2024-04-02T07:49:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.085+0000 7f1215c00700 1 -- start start 2024-04-02T07:49:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.085+0000 7f1215c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11f4004d90 0x7f12101e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.085+0000 7f1215c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12101e77c0 0x7f12101e7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.085+0000 7f1215c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12101f6a90 0x7f12101f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.085+0000 7f1215c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12100cb3a0 con 0x7f12101f6a90 2024-04-02T07:49:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.086+0000 7f1215c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12100ca720 con 0x7f11f4004d90 2024-04-02T07:49:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.086+0000 7f1215c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12100caa00 con 0x7f12101e77c0 2024-04-02T07:49:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.086+0000 7f120f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12101f6a90 0x7f12101f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.086+0000 7f120f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12101f6a90 0x7f12101f6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53586/0 (socket says 172.21.15.89:53586) 2024-04-02T07:49:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.086+0000 7f120f400700 1 -- 172.21.15.89:0/176766827 learned_addr learned my addr 172.21.15.89:0/176766827 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:16.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.086+0000 7f120f400700 1 -- 172.21.15.89:0/176766827 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12101e77c0 msgr2=0x7f12101e7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.086+0000 7f120f400700 1 --2- 172.21.15.89:0/176766827 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12101e77c0 0x7f12101e7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.086+0000 7f120f400700 1 -- 172.21.15.89:0/176766827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11f4004d90 msgr2=0x7f12101e7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.086+0000 7f120f400700 1 --2- 172.21.15.89:0/176766827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11f4004d90 0x7f12101e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.086+0000 7f120f400700 1 -- 172.21.15.89:0/176766827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12100cace0 con 0x7f12101f6a90 2024-04-02T07:49:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.086+0000 7f120f400700 1 --2- 172.21.15.89:0/176766827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12101f6a90 0x7f12101f6f10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1208000f80 tx=0x7f1208001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.086+0000 7f1206a00700 1 -- 172.21.15.89:0/176766827 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12080079d0 con 0x7f12101f6a90 2024-04-02T07:49:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.087+0000 7f1206a00700 1 -- 172.21.15.89:0/176766827 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1208007b50 con 0x7f12101f6a90 2024-04-02T07:49:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.087+0000 7f1215c00700 1 -- 172.21.15.89:0/176766827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12100cafc0 con 0x7f12101f6a90 2024-04-02T07:49:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.087+0000 7f1206a00700 1 -- 172.21.15.89:0/176766827 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1208012640 con 0x7f12101f6a90 2024-04-02T07:49:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.087+0000 7f1215c00700 1 -- 172.21.15.89:0/176766827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12101f7450 con 0x7f12101f6a90 2024-04-02T07:49:16.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.088+0000 7f1215c00700 1 -- 172.21.15.89:0/176766827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12100d1ca0 con 0x7f12101f6a90 2024-04-02T07:49:16.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.089+0000 7f1206a00700 1 -- 172.21.15.89:0/176766827 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f120801a070 con 0x7f12101f6a90 2024-04-02T07:49:16.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.090+0000 7f1206a00700 1 --2- 172.21.15.89:0/176766827 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f11f006adb0 0x7f11f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:16.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.090+0000 7f1206a00700 1 -- 172.21.15.89:0/176766827 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f120808b970 con 0x7f12101f6a90 2024-04-02T07:49:16.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.090+0000 7f120ea00700 1 --2- 172.21.15.89:0/176766827 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f11f006adb0 0x7f11f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:16.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.090+0000 7f120ea00700 1 --2- 172.21.15.89:0/176766827 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f11f006adb0 0x7f11f006d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1200009ce0 tx=0x7f1200009800 comp rx=0 tx=0).ready entity=mgr.5661 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:16.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.093+0000 7f1206a00700 1 -- 172.21.15.89:0/176766827 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1208056a50 con 0x7f12101f6a90 2024-04-02T07:49:16.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.354+0000 7f1215c00700 1 -- 172.21.15.89:0/176766827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f12100cb4e0 con 0x7f12101f6a90 2024-04-02T07:49:16.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.937+0000 7f1206a00700 1 -- 172.21.15.89:0/176766827 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 21) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f1208056e20 con 0x7f12101f6a90 2024-04-02T07:49:16.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.939+0000 7f1206a00700 1 -- 172.21.15.89:0/176766827 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v21) v1 ==== 92+0+0 (secure 0 0 0) 0x7f120806f6a0 con 0x7f12101f6a90 2024-04-02T07:49:16.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.942+0000 7f1215c00700 1 -- 172.21.15.89:0/176766827 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f11f006adb0 msgr2=0x7f11f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:16.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.942+0000 7f1215c00700 1 --2- 172.21.15.89:0/176766827 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f11f006adb0 0x7f11f006d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1200009ce0 tx=0x7f1200009800 comp rx=0 tx=0).stop 2024-04-02T07:49:16.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.942+0000 7f1215c00700 1 -- 172.21.15.89:0/176766827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12101f6a90 msgr2=0x7f12101f6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:16.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.942+0000 7f1215c00700 1 --2- 172.21.15.89:0/176766827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12101f6a90 0x7f12101f6f10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1208000f80 tx=0x7f1208001d00 comp rx=0 tx=0).stop 2024-04-02T07:49:16.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.942+0000 7f1215c00700 1 -- 172.21.15.89:0/176766827 shutdown_connections 2024-04-02T07:49:16.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.942+0000 7f1215c00700 1 --2- 172.21.15.89:0/176766827 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f11f006adb0 0x7f11f006d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:16.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.942+0000 7f1215c00700 1 --2- 172.21.15.89:0/176766827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11f4004d90 0x7f12101e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:16.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.942+0000 7f1215c00700 1 --2- 172.21.15.89:0/176766827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12101f6a90 0x7f12101f6f10 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:16.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.942+0000 7f1215c00700 1 --2- 172.21.15.89:0/176766827 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12101e77c0 0x7f12101e7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:16.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.942+0000 7f1215c00700 1 -- 172.21.15.89:0/176766827 >> 172.21.15.89:0/176766827 conn(0x7f121008aff0 msgr2=0x7f12100b0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:16.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.942+0000 7f1215c00700 1 -- 172.21.15.89:0/176766827 shutdown_connections 2024-04-02T07:49:16.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:16.942+0000 7f1215c00700 1 -- 172.21.15.89:0/176766827 wait complete. 2024-04-02T07:49:16.964 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T07:49:17.154 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:17.152+0000 7fe005d55240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:49:17.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.176+0000 7f994a000700 1 Processor -- start 2024-04-02T07:49:17.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.177+0000 7f994a000700 1 -- start start 2024-04-02T07:49:17.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.178+0000 7f994a000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f99440cd020 0x7f99440cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:17.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.178+0000 7f994a000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99440b2070 0x7f99440b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:17.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.178+0000 7f994a000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99440cdae0 con 0x7f99440cbb30 2024-04-02T07:49:17.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.178+0000 7f994a000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99440cddc0 con 0x7f99440cd020 2024-04-02T07:49:17.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.178+0000 7f994a000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99440ce0a0 con 0x7f99440b2070 2024-04-02T07:49:17.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.178+0000 7f9943e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99440b2070 0x7f99440b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:17.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.178+0000 7f9943400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f99440cbb30 0x7f99440cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:58122/0 (socket says 172.21.15.89:58122) 2024-04-02T07:49:17.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.178+0000 7f9943400700 1 -- 172.21.15.89:0/942238916 learned_addr learned my addr 172.21.15.89:0/942238916 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:17.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.178+0000 7f9942a00700 1 --2- 172.21.15.89:0/942238916 >> v2:172.21.15.113:3300/0 conn(0x7f99440cd020 0x7f99440cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:17.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.179+0000 7f9943e00700 1 -- 172.21.15.89:0/942238916 >> v2:172.21.15.113:3300/0 conn(0x7f99440cd020 msgr2=0x7f99440cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:17.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.179+0000 7f9943e00700 1 --2- 172.21.15.89:0/942238916 >> v2:172.21.15.113:3300/0 conn(0x7f99440cd020 0x7f99440cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.179+0000 7f9943e00700 1 -- 172.21.15.89:0/942238916 >> v1:172.21.15.113:6789/0 conn(0x7f99440cbb30 legacy=0x7f99440cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:17.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.179+0000 7f9943e00700 1 -- 172.21.15.89:0/942238916 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994406dc20 con 0x7f99440b2070 2024-04-02T07:49:17.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.179+0000 7f9942a00700 1 --2- 172.21.15.89:0/942238916 >> v2:172.21.15.113:3300/0 conn(0x7f99440cd020 0x7f99440cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:49:17.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.179+0000 7f9943e00700 1 --2- 172.21.15.89:0/942238916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99440b2070 0x7f99440b2500 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f993c009d90 tx=0x7f993c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26d5a7a08e8a4107 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:17.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.179+0000 7f9942000700 1 -- 172.21.15.89:0/942238916 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c00e070 con 0x7f99440b2070 2024-04-02T07:49:17.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.179+0000 7f9942000700 1 -- 172.21.15.89:0/942238916 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c00bb20 con 0x7f99440b2070 2024-04-02T07:49:17.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.179+0000 7f9942000700 1 -- 172.21.15.89:0/942238916 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c012430 con 0x7f99440b2070 2024-04-02T07:49:17.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.180+0000 7f994a000700 1 -- 172.21.15.89:0/942238916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99440b2070 msgr2=0x7f99440b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:17.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.180+0000 7f994a000700 1 --2- 172.21.15.89:0/942238916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99440b2070 0x7f99440b2500 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f993c009d90 tx=0x7f993c00b330 comp rx=0 tx=0).stop 2024-04-02T07:49:17.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.180+0000 7f994a000700 1 -- 172.21.15.89:0/942238916 shutdown_connections 2024-04-02T07:49:17.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.180+0000 7f994a000700 1 --2- 172.21.15.89:0/942238916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99440b2070 0x7f99440b2500 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.180+0000 7f994a000700 1 --2- 172.21.15.89:0/942238916 >> v2:172.21.15.113:3300/0 conn(0x7f99440cd020 0x7f99440cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.180+0000 7f994a000700 1 -- 172.21.15.89:0/942238916 >> 172.21.15.89:0/942238916 conn(0x7f99440b0d70 msgr2=0x7f99440b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:17.183 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:17.180+0000 7fcc84054240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:49:17.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.180+0000 7f994a000700 1 -- 172.21.15.89:0/942238916 shutdown_connections 2024-04-02T07:49:17.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.180+0000 7f994a000700 1 -- 172.21.15.89:0/942238916 wait complete. 2024-04-02T07:49:17.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.182+0000 7f994a000700 1 Processor -- start 2024-04-02T07:49:17.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.182+0000 7f994a000700 1 -- start start 2024-04-02T07:49:17.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.183+0000 7f994a000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440b2070 0x7f994414b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:17.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.183+0000 7f994a000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99440cbb30 0x7f9944157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:17.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.183+0000 7f994a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99440cd020 0x7f9944071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:17.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.183+0000 7f994a000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99441570f0 con 0x7f99440cd020 2024-04-02T07:49:17.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.183+0000 7f994a000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9944156570 con 0x7f99440b2070 2024-04-02T07:49:17.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.183+0000 7f994a000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9944156850 con 0x7f99440cbb30 2024-04-02T07:49:17.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.183+0000 7f9943400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440b2070 0x7f994414b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:17.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.183+0000 7f9943400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440b2070 0x7f994414b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:41640/0 (socket says 172.21.15.89:41640) 2024-04-02T07:49:17.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.183+0000 7f9943400700 1 -- 172.21.15.89:0/2427524837 learned_addr learned my addr 172.21.15.89:0/2427524837 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:17.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.183+0000 7f9942a00700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99440cbb30 0x7f9944157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:17.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.183+0000 7f9943e00700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99440cd020 0x7f9944071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:17.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.184+0000 7f9943400700 1 -- 172.21.15.89:0/2427524837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99440cbb30 msgr2=0x7f9944157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:17.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.184+0000 7f9943400700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99440cbb30 0x7f9944157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.184+0000 7f9943400700 1 -- 172.21.15.89:0/2427524837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99440cd020 msgr2=0x7f9944071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:17.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.184+0000 7f9943400700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99440cd020 0x7f9944071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.184+0000 7f9943400700 1 -- 172.21.15.89:0/2427524837 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99300094a0 con 0x7f99440b2070 2024-04-02T07:49:17.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.184+0000 7f9942a00700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99440cbb30 0x7f9944157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:49:17.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.184+0000 7f9943400700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440b2070 0x7f994414b510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f993400c520 tx=0x7f993400c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:17.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.184+0000 7f993b400700 1 -- 172.21.15.89:0/2427524837 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9934005d70 con 0x7f99440b2070 2024-04-02T07:49:17.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.184+0000 7f993b400700 1 -- 172.21.15.89:0/2427524837 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99340070d0 con 0x7f99440b2070 2024-04-02T07:49:17.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.184+0000 7f993b400700 1 -- 172.21.15.89:0/2427524837 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9934007440 con 0x7f99440b2070 2024-04-02T07:49:17.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.184+0000 7f994a000700 1 -- 172.21.15.89:0/2427524837 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f993c009a70 con 0x7f99440b2070 2024-04-02T07:49:17.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.184+0000 7f994a000700 1 -- 172.21.15.89:0/2427524837 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9944156e10 con 0x7f99440b2070 2024-04-02T07:49:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.185+0000 7f994a000700 1 -- 172.21.15.89:0/2427524837 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9944067720 con 0x7f99440b2070 2024-04-02T07:49:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.186+0000 7f993b400700 1 -- 172.21.15.89:0/2427524837 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 21) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f99340075a0 con 0x7f99440b2070 2024-04-02T07:49:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.187+0000 7f993b400700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f992406ae30 0x7f992406d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.187+0000 7f993b400700 1 -- 172.21.15.89:0/2427524837 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f99340889d0 con 0x7f99440b2070 2024-04-02T07:49:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.187+0000 7f9942a00700 1 -- 172.21.15.89:0/2427524837 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f992406ae30 msgr2=0x7f992406d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:49:17.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.187+0000 7f9942a00700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f992406ae30 0x7f992406d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:49:17.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.198+0000 7f993b400700 1 -- 172.21.15.89:0/2427524837 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9934053950 con 0x7f99440b2070 2024-04-02T07:49:17.344 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:17.342+0000 7fe005d55240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:49:17.370 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:17.368+0000 7fcc84054240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:49:17.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.388+0000 7f9942a00700 1 -- 172.21.15.89:0/2427524837 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f992406ae30 msgr2=0x7f992406d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:49:17.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.388+0000 7f9942a00700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f992406ae30 0x7f992406d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:49:17.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.442+0000 7f994a000700 1 -- 172.21.15.89:0/2427524837 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f9944158bc0 con 0x7f99440b2070 2024-04-02T07:49:17.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.449+0000 7f993b400700 1 -- 172.21.15.89:0/2427524837 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f99340170d0 con 0x7f99440b2070 2024-04-02T07:49:17.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.451+0000 7f994a000700 1 -- 172.21.15.89:0/2427524837 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f992406ae30 msgr2=0x7f992406d2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:49:17.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.451+0000 7f994a000700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f992406ae30 0x7f992406d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.451+0000 7f994a000700 1 -- 172.21.15.89:0/2427524837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440b2070 msgr2=0x7f994414b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:17.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.451+0000 7f994a000700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440b2070 0x7f994414b510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f993400c520 tx=0x7f993400c830 comp rx=0 tx=0).stop 2024-04-02T07:49:17.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.451+0000 7f994a000700 1 -- 172.21.15.89:0/2427524837 shutdown_connections 2024-04-02T07:49:17.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.451+0000 7f994a000700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f992406ae30 0x7f992406d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.451+0000 7f994a000700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440b2070 0x7f994414b510 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.451+0000 7f994a000700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99440cd020 0x7f9944071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.451+0000 7f994a000700 1 --2- 172.21.15.89:0/2427524837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99440cbb30 0x7f9944157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.451+0000 7f994a000700 1 -- 172.21.15.89:0/2427524837 >> 172.21.15.89:0/2427524837 conn(0x7f99440b0d70 msgr2=0x7f99440ae580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:17.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.451+0000 7f994a000700 1 -- 172.21.15.89:0/2427524837 shutdown_connections 2024-04-02T07:49:17.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.451+0000 7f994a000700 1 -- 172.21.15.89:0/2427524837 wait complete. 2024-04-02T07:49:17.473 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T07:49:17.661 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:17.659+0000 7fe005d55240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:49:17.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.673+0000 7f0439000700 1 Processor -- start 2024-04-02T07:49:17.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.673+0000 7f0439000700 1 -- start start 2024-04-02T07:49:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.674+0000 7f0439000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f043415a5c0 0x7f04340709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.674+0000 7f0439000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f04340ce840 0x7f04340cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.674+0000 7f0439000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04340cb310 con 0x7f043415dfe0 2024-04-02T07:49:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.674+0000 7f0439000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0434070f00 con 0x7f04340ce840 2024-04-02T07:49:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.674+0000 7f0439000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04340711a0 con 0x7f043415a5c0 2024-04-02T07:49:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.675+0000 7f0431400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f043415dfe0 0x7f04340ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51530/0 (socket says 172.21.15.89:51530) 2024-04-02T07:49:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.675+0000 7f0431400700 1 -- 172.21.15.89:0/304716985 learned_addr learned my addr 172.21.15.89:0/304716985 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:49:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.675+0000 7f0432800700 1 --2- 172.21.15.89:0/304716985 >> v2:172.21.15.89:3300/0 conn(0x7f04340ce840 0x7f04340cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.675+0000 7f0431e00700 1 --2- 172.21.15.89:0/304716985 >> v2:172.21.15.113:3300/0 conn(0x7f043415a5c0 0x7f04340709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.675+0000 7f0430a00700 1 -- 172.21.15.89:0/304716985 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034711643 0 0) 0x7f04340cb310 con 0x7f043415dfe0 2024-04-02T07:49:17.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.675+0000 7f0430a00700 1 -- 172.21.15.89:0/304716985 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04180057c0 con 0x7f043415dfe0 2024-04-02T07:49:17.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.675+0000 7f0432800700 1 -- 172.21.15.89:0/304716985 >> v2:172.21.15.113:3300/0 conn(0x7f043415a5c0 msgr2=0x7f04340709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:17.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.675+0000 7f0432800700 1 --2- 172.21.15.89:0/304716985 >> v2:172.21.15.113:3300/0 conn(0x7f043415a5c0 0x7f04340709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.675+0000 7f0432800700 1 -- 172.21.15.89:0/304716985 >> v1:172.21.15.89:6789/0 conn(0x7f043415dfe0 legacy=0x7f04340ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:17.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.675+0000 7f0432800700 1 -- 172.21.15.89:0/304716985 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0434071480 con 0x7f04340ce840 2024-04-02T07:49:17.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.676+0000 7f0432800700 1 --2- 172.21.15.89:0/304716985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04340ce840 0x7f04340cba80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f0434072e70 tx=0x7f042c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeee8da3a37362fb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:17.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.676+0000 7f0430a00700 1 -- 172.21.15.89:0/304716985 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c00e070 con 0x7f04340ce840 2024-04-02T07:49:17.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.676+0000 7f0430a00700 1 -- 172.21.15.89:0/304716985 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042c00bb20 con 0x7f04340ce840 2024-04-02T07:49:17.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.676+0000 7f0430a00700 1 -- 172.21.15.89:0/304716985 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c00be90 con 0x7f04340ce840 2024-04-02T07:49:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.676+0000 7f0439000700 1 -- 172.21.15.89:0/304716985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04340ce840 msgr2=0x7f04340cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.676+0000 7f0439000700 1 --2- 172.21.15.89:0/304716985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04340ce840 0x7f04340cba80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f0434072e70 tx=0x7f042c00b330 comp rx=0 tx=0).stop 2024-04-02T07:49:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.677+0000 7f0439000700 1 -- 172.21.15.89:0/304716985 shutdown_connections 2024-04-02T07:49:17.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.677+0000 7f0439000700 1 --2- 172.21.15.89:0/304716985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04340ce840 0x7f04340cba80 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.677+0000 7f0439000700 1 --2- 172.21.15.89:0/304716985 >> v2:172.21.15.113:3300/0 conn(0x7f043415a5c0 0x7f04340709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.677+0000 7f0439000700 1 -- 172.21.15.89:0/304716985 >> 172.21.15.89:0/304716985 conn(0x7f04340b0d70 msgr2=0x7f04340b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.677+0000 7f0439000700 1 -- 172.21.15.89:0/304716985 shutdown_connections 2024-04-02T07:49:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.677+0000 7f0439000700 1 -- 172.21.15.89:0/304716985 wait complete. 2024-04-02T07:49:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.678+0000 7f0439000700 1 Processor -- start 2024-04-02T07:49:17.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.679+0000 7f0439000700 1 -- start start 2024-04-02T07:49:17.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.679+0000 7f0439000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04340ce840 0x7f0434157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:17.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.679+0000 7f0439000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f043415a5c0 0x7f043414bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:17.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.679+0000 7f0439000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f043415dfe0 0x7f04341403a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:17.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.680+0000 7f0439000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04341408e0 con 0x7f043415a5c0 2024-04-02T07:49:17.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.680+0000 7f0439000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04340b0370 con 0x7f043415dfe0 2024-04-02T07:49:17.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.680+0000 7f0439000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04340b0b80 con 0x7f04340ce840 2024-04-02T07:49:17.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.680+0000 7f0431400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f043415a5c0 0x7f043414bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:17.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.680+0000 7f0431400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f043415a5c0 0x7f043414bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53620/0 (socket says 172.21.15.89:53620) 2024-04-02T07:49:17.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.680+0000 7f0431400700 1 -- 172.21.15.89:0/2256467581 learned_addr learned my addr 172.21.15.89:0/2256467581 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:17.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.680+0000 7f0431e00700 1 --2- 172.21.15.89:0/2256467581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04340ce840 0x7f0434157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:17.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.680+0000 7f0432800700 1 --2- 172.21.15.89:0/2256467581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f043415dfe0 0x7f04341403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:17.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.680+0000 7f0431400700 1 -- 172.21.15.89:0/2256467581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04340ce840 msgr2=0x7f0434157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:17.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.680+0000 7f0431400700 1 --2- 172.21.15.89:0/2256467581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04340ce840 0x7f0434157bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.680+0000 7f0431400700 1 -- 172.21.15.89:0/2256467581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f043415dfe0 msgr2=0x7f04341403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:17.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.680+0000 7f0431400700 1 --2- 172.21.15.89:0/2256467581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f043415dfe0 0x7f04341403a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.680+0000 7f0431400700 1 -- 172.21.15.89:0/2256467581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042c009a70 con 0x7f043415a5c0 2024-04-02T07:49:17.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.681+0000 7f0431400700 1 --2- 172.21.15.89:0/2256467581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f043415a5c0 0x7f043414bfc0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f042000c420 tx=0x7f042000c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:17.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.681+0000 7f042a000700 1 -- 172.21.15.89:0/2256467581 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420005d70 con 0x7f043415a5c0 2024-04-02T07:49:17.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.681+0000 7f042a000700 1 -- 172.21.15.89:0/2256467581 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042000ce30 con 0x7f043415a5c0 2024-04-02T07:49:17.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.681+0000 7f0439000700 1 -- 172.21.15.89:0/2256467581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0434155170 con 0x7f043415a5c0 2024-04-02T07:49:17.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.681+0000 7f042a000700 1 -- 172.21.15.89:0/2256467581 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04200072d0 con 0x7f043415a5c0 2024-04-02T07:49:17.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.681+0000 7f0439000700 1 -- 172.21.15.89:0/2256467581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04340b08a0 con 0x7f043415a5c0 2024-04-02T07:49:17.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.682+0000 7f0439000700 1 -- 172.21.15.89:0/2256467581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0434067720 con 0x7f043415a5c0 2024-04-02T07:49:17.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.684+0000 7f042a000700 1 -- 172.21.15.89:0/2256467581 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 21) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f0420007430 con 0x7f043415a5c0 2024-04-02T07:49:17.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.685+0000 7f042a000700 1 --2- 172.21.15.89:0/2256467581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f041406ae00 0x7f041406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:17.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.685+0000 7f042a000700 1 -- 172.21.15.89:0/2256467581 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f04200887d0 con 0x7f043415a5c0 2024-04-02T07:49:17.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.685+0000 7f0431e00700 1 -- 172.21.15.89:0/2256467581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f041406ae00 msgr2=0x7f041406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:49:17.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.685+0000 7f0431e00700 1 --2- 172.21.15.89:0/2256467581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f041406ae00 0x7f041406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:49:17.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.689+0000 7f042a000700 1 -- 172.21.15.89:0/2256467581 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0420053880 con 0x7f043415a5c0 2024-04-02T07:49:17.712 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:17.710+0000 7fcc84054240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:49:17.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.886+0000 7f0431e00700 1 -- 172.21.15.89:0/2256467581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f041406ae00 msgr2=0x7f041406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:49:17.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.886+0000 7f0431e00700 1 --2- 172.21.15.89:0/2256467581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f041406ae00 0x7f041406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:49:17.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.899+0000 7f0439000700 1 -- 172.21.15.89:0/2256467581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f0434155300 con 0x7f043415a5c0 2024-04-02T07:49:17.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.910+0000 7f042a000700 1 -- 172.21.15.89:0/2256467581 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7f04200170c0 con 0x7f043415a5c0 2024-04-02T07:49:17.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.912+0000 7f0439000700 1 -- 172.21.15.89:0/2256467581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f041406ae00 msgr2=0x7f041406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:49:17.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.912+0000 7f0439000700 1 --2- 172.21.15.89:0/2256467581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f041406ae00 0x7f041406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.913+0000 7f0439000700 1 -- 172.21.15.89:0/2256467581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f043415a5c0 msgr2=0x7f043414bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:17.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.913+0000 7f0439000700 1 --2- 172.21.15.89:0/2256467581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f043415a5c0 0x7f043414bfc0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f042000c420 tx=0x7f042000c730 comp rx=0 tx=0).stop 2024-04-02T07:49:17.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.913+0000 7f0439000700 1 -- 172.21.15.89:0/2256467581 shutdown_connections 2024-04-02T07:49:17.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.913+0000 7f0439000700 1 --2- 172.21.15.89:0/2256467581 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f041406ae00 0x7f041406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.913+0000 7f0439000700 1 --2- 172.21.15.89:0/2256467581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f043415dfe0 0x7f04341403a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.913+0000 7f0439000700 1 --2- 172.21.15.89:0/2256467581 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04340ce840 0x7f0434157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.913+0000 7f0439000700 1 --2- 172.21.15.89:0/2256467581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f043415a5c0 0x7f043414bfc0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:17.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.913+0000 7f0439000700 1 -- 172.21.15.89:0/2256467581 >> 172.21.15.89:0/2256467581 conn(0x7f04340b0d70 msgr2=0x7f043415e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:17.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.913+0000 7f0439000700 1 -- 172.21.15.89:0/2256467581 shutdown_connections 2024-04-02T07:49:17.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:17.913+0000 7f0439000700 1 -- 172.21.15.89:0/2256467581 wait complete. 2024-04-02T07:49:17.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:17.935 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-02T07:49:17.985 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-04-02T07:49:18.193 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:18.191+0000 7fe005d55240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:49:18.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.195+0000 7f217f600700 1 Processor -- start 2024-04-02T07:49:18.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.196+0000 7f217f600700 1 -- start start 2024-04-02T07:49:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.197+0000 7f217f600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f21780cbb30 0x7f21780cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.197+0000 7f217f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21780cd020 0x7f21780cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.197+0000 7f217f600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f21780b2070 0x7f21780b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.197+0000 7f217f600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21780cc450 con 0x7f21780cbb30 2024-04-02T07:49:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.197+0000 7f217f600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21780cdae0 con 0x7f21780b2070 2024-04-02T07:49:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.197+0000 7f217f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21780cddc0 con 0x7f21780cd020 2024-04-02T07:49:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.197+0000 7f217cc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f21780cbb30 0x7f21780cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.197+0000 7f217cc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f21780cbb30 0x7f21780cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53630/0 (socket says 172.21.15.89:53630) 2024-04-02T07:49:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.197+0000 7f217cc00700 1 -- 172.21.15.89:0/1548380327 learned_addr learned my addr 172.21.15.89:0/1548380327 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.197+0000 7f217d600700 1 --2- 172.21.15.89:0/1548380327 >> v2:172.21.15.113:3300/0 conn(0x7f21780b2070 0x7f21780b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.197+0000 7f2177e00700 1 --2- 172.21.15.89:0/1548380327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21780cd020 0x7f21780cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.198+0000 7f217cc00700 1 -- 172.21.15.89:0/1548380327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21780cd020 msgr2=0x7f21780cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:18.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.198+0000 7f217cc00700 1 --2- 172.21.15.89:0/1548380327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21780cd020 0x7f21780cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:18.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.198+0000 7f217cc00700 1 -- 172.21.15.89:0/1548380327 >> v2:172.21.15.113:3300/0 conn(0x7f21780b2070 msgr2=0x7f21780b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:18.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.198+0000 7f217cc00700 1 --2- 172.21.15.89:0/1548380327 >> v2:172.21.15.113:3300/0 conn(0x7f21780b2070 0x7f21780b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:18.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.198+0000 7f217cc00700 1 -- 172.21.15.89:0/1548380327 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21780ce0a0 con 0x7f21780cbb30 2024-04-02T07:49:18.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.198+0000 7f217cc00700 1 --2- 172.21.15.89:0/1548380327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21780cbb30 0x7f21780cbf10 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f21780b0d00 tx=0x7f216c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91db5fe2eb7d41c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:18.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.198+0000 7f2177400700 1 -- 172.21.15.89:0/1548380327 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c011830 con 0x7f21780cbb30 2024-04-02T07:49:18.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.198+0000 7f2177400700 1 -- 172.21.15.89:0/1548380327 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216c011990 con 0x7f21780cbb30 2024-04-02T07:49:18.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.198+0000 7f2177400700 1 -- 172.21.15.89:0/1548380327 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c011d20 con 0x7f21780cbb30 2024-04-02T07:49:18.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.199+0000 7f217f600700 1 -- 172.21.15.89:0/1548380327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21780cbb30 msgr2=0x7f21780cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:18.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.199+0000 7f217f600700 1 --2- 172.21.15.89:0/1548380327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21780cbb30 0x7f21780cbf10 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f21780b0d00 tx=0x7f216c009bd0 comp rx=0 tx=0).stop 2024-04-02T07:49:18.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.199+0000 7f217f600700 1 -- 172.21.15.89:0/1548380327 shutdown_connections 2024-04-02T07:49:18.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.199+0000 7f217f600700 1 --2- 172.21.15.89:0/1548380327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21780cbb30 0x7f21780cbf10 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:18.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.199+0000 7f217f600700 1 --2- 172.21.15.89:0/1548380327 >> v2:172.21.15.113:3300/0 conn(0x7f21780b2070 0x7f21780b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:18.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.199+0000 7f217f600700 1 --2- 172.21.15.89:0/1548380327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21780cd020 0x7f21780cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:18.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.199+0000 7f217f600700 1 -- 172.21.15.89:0/1548380327 >> 172.21.15.89:0/1548380327 conn(0x7f21780b0d70 msgr2=0x7f21780b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:18.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.199+0000 7f217f600700 1 -- 172.21.15.89:0/1548380327 shutdown_connections 2024-04-02T07:49:18.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.199+0000 7f217f600700 1 -- 172.21.15.89:0/1548380327 wait complete. 2024-04-02T07:49:18.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.201+0000 7f217f600700 1 Processor -- start 2024-04-02T07:49:18.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.201+0000 7f217f600700 1 -- start start 2024-04-02T07:49:18.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f217f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21780b2070 0x7f2178148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:18.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f217f600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21780cbb30 0x7f21780ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:18.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f217f600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21780cd020 0x7f2178071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:18.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f217f600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21780c9930 con 0x7f21780cbb30 2024-04-02T07:49:18.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f217f600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2178148a70 con 0x7f21780cd020 2024-04-02T07:49:18.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f217f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21780aecd0 con 0x7f21780b2070 2024-04-02T07:49:18.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f2177e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21780cbb30 0x7f21780ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:18.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f2177e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21780cbb30 0x7f21780ae280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53638/0 (socket says 172.21.15.89:53638) 2024-04-02T07:49:18.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f2177e00700 1 -- 172.21.15.89:0/183092944 learned_addr learned my addr 172.21.15.89:0/183092944 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:18.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f217cc00700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21780b2070 0x7f2178148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:18.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f217d600700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21780cd020 0x7f2178071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:18.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f2177e00700 1 -- 172.21.15.89:0/183092944 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21780b2070 msgr2=0x7f2178148530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:18.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f2177e00700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21780b2070 0x7f2178148530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:18.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f2177e00700 1 -- 172.21.15.89:0/183092944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21780cd020 msgr2=0x7f2178071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:18.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f2177e00700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21780cd020 0x7f2178071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:18.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.202+0000 7f2177e00700 1 -- 172.21.15.89:0/183092944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c009550 con 0x7f21780cbb30 2024-04-02T07:49:18.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.203+0000 7f2177e00700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21780cbb30 0x7f21780ae280 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f216400e9b0 tx=0x7f216400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:18.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.203+0000 7f2174c00700 1 -- 172.21.15.89:0/183092944 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2164009710 con 0x7f21780cbb30 2024-04-02T07:49:18.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.203+0000 7f2174c00700 1 -- 172.21.15.89:0/183092944 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216400ee50 con 0x7f21780cbb30 2024-04-02T07:49:18.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.203+0000 7f2174c00700 1 -- 172.21.15.89:0/183092944 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216400ca50 con 0x7f21780cbb30 2024-04-02T07:49:18.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.203+0000 7f217f600700 1 -- 172.21.15.89:0/183092944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21780aefb0 con 0x7f21780cbb30 2024-04-02T07:49:18.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.203+0000 7f217f600700 1 -- 172.21.15.89:0/183092944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21780af570 con 0x7f21780cbb30 2024-04-02T07:49:18.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.204+0000 7f217f600700 1 -- 172.21.15.89:0/183092944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21780c9c60 con 0x7f21780cbb30 2024-04-02T07:49:18.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.205+0000 7f2174c00700 1 -- 172.21.15.89:0/183092944 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 21) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f216400cbb0 con 0x7f21780cbb30 2024-04-02T07:49:18.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.206+0000 7f2174c00700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 0x7f215806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:18.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.206+0000 7f2174c00700 1 -- 172.21.15.89:0/183092944 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2164089640 con 0x7f21780cbb30 2024-04-02T07:49:18.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.206+0000 7f217cc00700 1 -- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 msgr2=0x7f215806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:49:18.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.206+0000 7f217cc00700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 0x7f215806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:49:18.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.211+0000 7f2174c00700 1 -- 172.21.15.89:0/183092944 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2164054510 con 0x7f21780cbb30 2024-04-02T07:49:18.281 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:18.279+0000 7fcc84054240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:49:18.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.407+0000 7f217cc00700 1 -- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 msgr2=0x7f215806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:49:18.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.407+0000 7f217cc00700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 0x7f215806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:49:18.448 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:18.446+0000 7fe005d55240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:49:18.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.449+0000 7f217f600700 1 -- 172.21.15.89:0/183092944 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f21781594a0 con 0x7f215806ae00 2024-04-02T07:49:18.543 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:18.540+0000 7fcc84054240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:49:18.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.808+0000 7f217cc00700 1 -- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 msgr2=0x7f215806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:49:18.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:18.808+0000 7f217cc00700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 0x7f215806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T07:49:18.901 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:18.899+0000 7fe005d55240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:49:18.998 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:18.995+0000 7fcc84054240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:49:19.603 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:19.602+0000 7fe005d55240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:49:19.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:19.609+0000 7f217cc00700 1 -- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 msgr2=0x7f215806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:49:19.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:19.609+0000 7f217cc00700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 0x7f215806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T07:49:19.693 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:19.691+0000 7fe005d55240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:49:19.701 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:19.699+0000 7fcc84054240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:49:19.792 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:19.790+0000 7fcc84054240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:49:19.851 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:19.850+0000 7fe005d55240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:49:19.949 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:19.947+0000 7fcc84054240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:49:20.062 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:20.060+0000 7fe005d55240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:49:20.152 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:20.151+0000 7fe005d55240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:49:20.161 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:20.159+0000 7fcc84054240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:49:20.248 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:20.246+0000 7fe005d55240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:49:20.252 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:20.250+0000 7fcc84054240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:49:20.345 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:20.343+0000 7fe005d55240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:49:20.349 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:20.347+0000 7fcc84054240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:49:20.441 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:20.439+0000 7fe005d55240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:49:20.447 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:20.445+0000 7fcc84054240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:49:20.544 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:20.542+0000 7fcc84054240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:49:20.650 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:20.648+0000 7fe005d55240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:49:20.754 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:20.752+0000 7fcc84054240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:49:20.901 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:20.899+0000 7fe005d55240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:49:21.009 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:21.007+0000 7fcc84054240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:49:21.091 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:21.089+0000 7fe005d55240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:49:21.182 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:21.180+0000 7fe005d55240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:49:21.200 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:21.198+0000 7fcc84054240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:49:21.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:21.210+0000 7f217cc00700 1 -- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 msgr2=0x7f215806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:49:21.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:21.210+0000 7f217cc00700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 0x7f215806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T07:49:21.291 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:21.289+0000 7fcc84054240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:49:21.900 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:21.898+0000 7fe005d55240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:49:22.016 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:22.014+0000 7fcc84054240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:49:22.108 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:22.107+0000 7fe005d55240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:49:22.227 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:22.225+0000 7fcc84054240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:49:22.310 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:22.308+0000 7fe005d55240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:49:22.432 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:22.430+0000 7fcc84054240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:49:22.777 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:22.775+0000 7fe005d55240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:49:22.906 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:22.904+0000 7fcc84054240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:49:22.979 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:22.977+0000 7fe005d55240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:49:23.112 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:23.110+0000 7fcc84054240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:49:23.415 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:23.412+0000 7fe005d55240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:49:23.523 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:49:23.520+0000 7fe005d55240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:49:23.554 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:23.552+0000 7fcc84054240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:49:23.664 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:23.662+0000 7fcc84054240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:49:23.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:23.672+0000 7f2174c00700 1 -- 172.21.15.89:0/183092944 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 22) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f21640548e0 con 0x7f21780cbb30 2024-04-02T07:49:23.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:23.672+0000 7f2174c00700 1 -- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 msgr2=0x7f215806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:49:23.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:23.672+0000 7f2174c00700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 0x7f215806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:23.849 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:23.847+0000 7fcbec800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:49:23.849 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:23.847+0000 7fcbec800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:49:23.850 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:23.847+0000 7fcbec800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:49:23.850 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:23.847+0000 7fcbec800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:49:23.850 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:23.847+0000 7fcbec800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:49:23.851 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:23.848+0000 7fcbf0400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:49:23.851 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:23.848+0000 7fcbf0400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:49:23.851 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:23.848+0000 7fcbf0400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:49:23.851 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:23.848+0000 7fcbf0400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:49:23.851 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:49:23.848+0000 7fcbf0400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:49:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.677+0000 7f2174c00700 1 -- 172.21.15.89:0/183092944 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 23) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f216406fde0 con 0x7f21780cbb30 2024-04-02T07:49:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.677+0000 7f2174c00700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 0x7f215806d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.677+0000 7f2174c00700 1 -- 172.21.15.89:0/183092944 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2158035920 con 0x7f215806ae00 2024-04-02T07:49:24.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.679+0000 7f217cc00700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 0x7f215806d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:24.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.682+0000 7f217cc00700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 0x7f215806d240 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f216c014040 tx=0x7f216c008030 comp rx=0 tx=0).ready entity=mgr.6075 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:24.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.752+0000 7f2174c00700 1 -- 172.21.15.89:0/183092944 <== mgr.6075 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2158035920 con 0x7f215806ae00 2024-04-02T07:49:24.754 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path snap_test_dir1 2024-04-02T07:49:24.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.755+0000 7f217f600700 1 -- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 msgr2=0x7f215806d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:24.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.755+0000 7f217f600700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 0x7f215806d240 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f216c014040 tx=0x7f216c008030 comp rx=0 tx=0).stop 2024-04-02T07:49:24.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.755+0000 7f217f600700 1 -- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21780cbb30 msgr2=0x7f21780ae280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:24.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.755+0000 7f217f600700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21780cbb30 0x7f21780ae280 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f216400e9b0 tx=0x7f216400ecc0 comp rx=0 tx=0).stop 2024-04-02T07:49:24.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.755+0000 7f217f600700 1 -- 172.21.15.89:0/183092944 shutdown_connections 2024-04-02T07:49:24.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.755+0000 7f217f600700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f215806ae00 0x7f215806d240 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:24.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.755+0000 7f217f600700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21780cd020 0x7f2178071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:24.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.755+0000 7f217f600700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21780b2070 0x7f2178148530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:24.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.755+0000 7f217f600700 1 --2- 172.21.15.89:0/183092944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21780cbb30 0x7f21780ae280 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:24.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.755+0000 7f217f600700 1 -- 172.21.15.89:0/183092944 >> 172.21.15.89:0/183092944 conn(0x7f21780b0d70 msgr2=0x7f217806e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:24.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.755+0000 7f217f600700 1 -- 172.21.15.89:0/183092944 shutdown_connections 2024-04-02T07:49:24.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.755+0000 7f217f600700 1 -- 172.21.15.89:0/183092944 wait complete. 2024-04-02T07:49:24.775 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-02T07:49:24.775 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2024-04-02T07:49:24.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.967+0000 7fc73ec00700 1 Processor -- start 2024-04-02T07:49:24.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.967+0000 7fc73ec00700 1 -- start start 2024-04-02T07:49:24.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc73ec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc738160e50 0x7fc7380b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:24.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc73ec00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc7380d3f20 0x7fc7380b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:24.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc73ec00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc7380b3f80 0x7fc7380b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:24.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc73ec00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc738161230 con 0x7fc7380d3f20 2024-04-02T07:49:24.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc73ec00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7380d2a80 con 0x7fc7380b3f80 2024-04-02T07:49:24.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc73ec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7380d43b0 con 0x7fc738160e50 2024-04-02T07:49:24.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc72fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc7380d3f20 0x7fc7380b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:24.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc72fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc7380d3f20 0x7fc7380b53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37698/0 (socket says 172.21.15.89:37698) 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc72fe00700 1 -- 172.21.15.89:0/703263498 learned_addr learned my addr 172.21.15.89:0/703263498 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc737e00700 1 --2- 172.21.15.89:0/703263498 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc738160e50 0x7fc7380b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc73cc00700 1 --2- 172.21.15.89:0/703263498 >> v2:172.21.15.113:3300/0 conn(0x7fc7380b3f80 0x7fc7380b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc72fe00700 1 -- 172.21.15.89:0/703263498 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc738160e50 msgr2=0x7fc7380b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc72fe00700 1 --2- 172.21.15.89:0/703263498 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc738160e50 0x7fc7380b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc72fe00700 1 -- 172.21.15.89:0/703263498 >> v2:172.21.15.113:3300/0 conn(0x7fc7380b3f80 msgr2=0x7fc7380b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc72fe00700 1 --2- 172.21.15.89:0/703263498 >> v2:172.21.15.113:3300/0 conn(0x7fc7380b3f80 0x7fc7380b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc72fe00700 1 -- 172.21.15.89:0/703263498 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7380d21e0 con 0x7fc7380d3f20 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.969+0000 7fc737e00700 1 --2- 172.21.15.89:0/703263498 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc738160e50 0x7fc7380b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.970+0000 7fc72fe00700 1 --2- 172.21.15.89:0/703263498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7380d3f20 0x7fc7380b53d0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc7380d3e70 tx=0x7fc720009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7708a053d28b16a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.970+0000 7fc737400700 1 -- 172.21.15.89:0/703263498 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72000b870 con 0x7fc7380d3f20 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.970+0000 7fc737400700 1 -- 172.21.15.89:0/703263498 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72000b9d0 con 0x7fc7380d3f20 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.970+0000 7fc737400700 1 -- 172.21.15.89:0/703263498 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72000bd60 con 0x7fc7380d3f20 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.970+0000 7fc73ec00700 1 -- 172.21.15.89:0/703263498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7380d3f20 msgr2=0x7fc7380b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:24.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.970+0000 7fc73ec00700 1 --2- 172.21.15.89:0/703263498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7380d3f20 0x7fc7380b53d0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc7380d3e70 tx=0x7fc720009bd0 comp rx=0 tx=0).stop 2024-04-02T07:49:24.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.970+0000 7fc73ec00700 1 -- 172.21.15.89:0/703263498 shutdown_connections 2024-04-02T07:49:24.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.970+0000 7fc73ec00700 1 --2- 172.21.15.89:0/703263498 >> v2:172.21.15.113:3300/0 conn(0x7fc7380b3f80 0x7fc7380b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:24.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.970+0000 7fc73ec00700 1 --2- 172.21.15.89:0/703263498 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc738160e50 0x7fc7380b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:24.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.970+0000 7fc73ec00700 1 --2- 172.21.15.89:0/703263498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7380d3f20 0x7fc7380b53d0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:24.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.970+0000 7fc73ec00700 1 -- 172.21.15.89:0/703263498 >> 172.21.15.89:0/703263498 conn(0x7fc73808aff0 msgr2=0x7fc7380d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:24.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.970+0000 7fc73ec00700 1 -- 172.21.15.89:0/703263498 shutdown_connections 2024-04-02T07:49:24.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.971+0000 7fc73ec00700 1 -- 172.21.15.89:0/703263498 wait complete. 2024-04-02T07:49:24.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.972+0000 7fc73ec00700 1 Processor -- start 2024-04-02T07:49:24.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.972+0000 7fc73ec00700 1 -- start start 2024-04-02T07:49:24.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.973+0000 7fc73ec00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7380b3f80 0x7fc7381f7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:24.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.973+0000 7fc73ec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc738160e50 0x7fc7380cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:24.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.973+0000 7fc73ec00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7381f32e0 0x7fc7381f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:24.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.973+0000 7fc73ec00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7381f7890 con 0x7fc7380b3f80 2024-04-02T07:49:24.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.973+0000 7fc73ec00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7381f6900 con 0x7fc7381f32e0 2024-04-02T07:49:24.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.973+0000 7fc73ec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7381f3e60 con 0x7fc738160e50 2024-04-02T07:49:24.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.973+0000 7fc737e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7380b3f80 0x7fc7381f7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:24.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.973+0000 7fc737e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7380b3f80 0x7fc7381f7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37708/0 (socket says 172.21.15.89:37708) 2024-04-02T07:49:24.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.973+0000 7fc737e00700 1 -- 172.21.15.89:0/3613436651 learned_addr learned my addr 172.21.15.89:0/3613436651 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:24.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.973+0000 7fc72fe00700 1 --2- 172.21.15.89:0/3613436651 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc738160e50 0x7fc7380cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:24.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.974+0000 7fc73cc00700 1 --2- 172.21.15.89:0/3613436651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7381f32e0 0x7fc7381f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:24.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.974+0000 7fc737e00700 1 -- 172.21.15.89:0/3613436651 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc738160e50 msgr2=0x7fc7380cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:24.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.974+0000 7fc737e00700 1 --2- 172.21.15.89:0/3613436651 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc738160e50 0x7fc7380cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:24.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.974+0000 7fc737e00700 1 -- 172.21.15.89:0/3613436651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7381f32e0 msgr2=0x7fc7381f3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:24.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.974+0000 7fc737e00700 1 --2- 172.21.15.89:0/3613436651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7381f32e0 0x7fc7381f3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:24.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.974+0000 7fc737e00700 1 -- 172.21.15.89:0/3613436651 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc720009550 con 0x7fc7380b3f80 2024-04-02T07:49:24.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.974+0000 7fc737e00700 1 --2- 172.21.15.89:0/3613436651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7380b3f80 0x7fc7381f7350 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc72800e810 tx=0x7fc72800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:24.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.974+0000 7fc734c00700 1 -- 172.21.15.89:0/3613436651 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72800c020 con 0x7fc7380b3f80 2024-04-02T07:49:24.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.974+0000 7fc734c00700 1 -- 172.21.15.89:0/3613436651 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc728004250 con 0x7fc7380b3f80 2024-04-02T07:49:24.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.974+0000 7fc734c00700 1 -- 172.21.15.89:0/3613436651 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7280036a0 con 0x7fc7380b3f80 2024-04-02T07:49:24.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.974+0000 7fc73ec00700 1 -- 172.21.15.89:0/3613436651 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7381f4140 con 0x7fc7380b3f80 2024-04-02T07:49:24.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.974+0000 7fc73ec00700 1 -- 172.21.15.89:0/3613436651 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7381f6e30 con 0x7fc7380b3f80 2024-04-02T07:49:24.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.975+0000 7fc73ec00700 1 -- 172.21.15.89:0/3613436651 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7381f7050 con 0x7fc7380b3f80 2024-04-02T07:49:24.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.976+0000 7fc734c00700 1 -- 172.21.15.89:0/3613436651 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fc728003800 con 0x7fc7380b3f80 2024-04-02T07:49:24.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.977+0000 7fc734c00700 1 --2- 172.21.15.89:0/3613436651 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc71406ab30 0x7fc71406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:24.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.977+0000 7fc734c00700 1 -- 172.21.15.89:0/3613436651 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fc728088d00 con 0x7fc7380b3f80 2024-04-02T07:49:24.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.977+0000 7fc72fe00700 1 --2- 172.21.15.89:0/3613436651 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc71406ab30 0x7fc71406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:24.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.978+0000 7fc72fe00700 1 --2- 172.21.15.89:0/3613436651 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc71406ab30 0x7fc71406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc720000c00 tx=0x7fc720008030 comp rx=0 tx=0).ready entity=mgr.6075 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:24.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:24.982+0000 7fc734c00700 1 -- 172.21.15.89:0/3613436651 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc728053e90 con 0x7fc7380b3f80 2024-04-02T07:49:25.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.209+0000 7fc73ec00700 1 -- 172.21.15.89:0/3613436651 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7380b0100 con 0x7fc71406ab30 2024-04-02T07:49:25.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.277+0000 7fc734c00700 1 -- 172.21.15.89:0/3613436651 <== mgr.6075 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc72000bb40 con 0x7fc71406ab30 2024-04-02T07:49:25.280 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path snap_test_dir1 2024-04-02T07:49:25.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.280+0000 7fc73ec00700 1 -- 172.21.15.89:0/3613436651 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc71406ab30 msgr2=0x7fc71406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:25.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.280+0000 7fc73ec00700 1 --2- 172.21.15.89:0/3613436651 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc71406ab30 0x7fc71406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc720000c00 tx=0x7fc720008030 comp rx=0 tx=0).stop 2024-04-02T07:49:25.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.280+0000 7fc73ec00700 1 -- 172.21.15.89:0/3613436651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7380b3f80 msgr2=0x7fc7381f7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:25.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.280+0000 7fc73ec00700 1 --2- 172.21.15.89:0/3613436651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7380b3f80 0x7fc7381f7350 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc72800e810 tx=0x7fc72800eb20 comp rx=0 tx=0).stop 2024-04-02T07:49:25.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.280+0000 7fc73ec00700 1 -- 172.21.15.89:0/3613436651 shutdown_connections 2024-04-02T07:49:25.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.280+0000 7fc73ec00700 1 --2- 172.21.15.89:0/3613436651 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc71406ab30 0x7fc71406cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:25.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.280+0000 7fc73ec00700 1 --2- 172.21.15.89:0/3613436651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7381f32e0 0x7fc7381f3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:25.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.280+0000 7fc73ec00700 1 --2- 172.21.15.89:0/3613436651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7380b3f80 0x7fc7381f7350 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:25.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.280+0000 7fc73ec00700 1 --2- 172.21.15.89:0/3613436651 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc738160e50 0x7fc7380cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:25.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.280+0000 7fc73ec00700 1 -- 172.21.15.89:0/3613436651 >> 172.21.15.89:0/3613436651 conn(0x7fc73808aff0 msgr2=0x7fc7380b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:25.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.280+0000 7fc73ec00700 1 -- 172.21.15.89:0/3613436651 shutdown_connections 2024-04-02T07:49:25.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.280+0000 7fc73ec00700 1 -- 172.21.15.89:0/3613436651 wait complete. 2024-04-02T07:49:25.299 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-02T07:49:25.300 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-02-07_50_00 in ~35s... 2024-04-02T07:49:25.300 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-02-07_51_00 in ~95s... 2024-04-02T07:49:25.300 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2024-04-02T07:49:25.300 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-04-02T07:49:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.491+0000 7f15a8400700 1 Processor -- start 2024-04-02T07:49:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.491+0000 7f15a8400700 1 -- start start 2024-04-02T07:49:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.492+0000 7f15a8400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f15a015d9d0 0x7f15a00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.492+0000 7f15a8400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15a00d16a0 0x7f15a00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.492+0000 7f15a8400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a0161420 con 0x7f15a00d2370 2024-04-02T07:49:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.492+0000 7f15a8400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a00d2820 con 0x7f15a015d9d0 2024-04-02T07:49:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.492+0000 7f15a8400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a0161700 con 0x7f15a00d16a0 2024-04-02T07:49:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.492+0000 7f15a5a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f15a015d9d0 0x7f15a00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.492+0000 7f15a5a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f15a015d9d0 0x7f15a00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37720/0 (socket says 172.21.15.89:37720) 2024-04-02T07:49:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.492+0000 7f15a5a00700 1 -- 172.21.15.89:0/1558104912 learned_addr learned my addr 172.21.15.89:0/1558104912 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.493+0000 7f15a5000700 1 --2- 172.21.15.89:0/1558104912 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15a00d16a0 0x7f15a00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.493+0000 7f15a5a00700 1 -- 172.21.15.89:0/1558104912 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15a00d16a0 msgr2=0x7f15a00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:49:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.493+0000 7f15a5a00700 1 --2- 172.21.15.89:0/1558104912 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15a00d16a0 0x7f15a00d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.493+0000 7f15a5a00700 1 -- 172.21.15.89:0/1558104912 >> v1:172.21.15.113:6789/0 conn(0x7f15a00d2370 legacy=0x7f15a015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.493+0000 7f15a5a00700 1 -- 172.21.15.89:0/1558104912 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a01619e0 con 0x7f15a015d9d0 2024-04-02T07:49:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.493+0000 7f15a5000700 1 --2- 172.21.15.89:0/1558104912 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15a00d16a0 0x7f15a00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:49:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.493+0000 7f15a5a00700 1 --2- 172.21.15.89:0/1558104912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a015d9d0 0x7f15a00d3c30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f15940098f0 tx=0x7f1594009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf98a5e1ff056b0c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.493+0000 7f159fe00700 1 -- 172.21.15.89:0/1558104912 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159400b8a0 con 0x7f15a015d9d0 2024-04-02T07:49:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.494+0000 7f159fe00700 1 -- 172.21.15.89:0/1558104912 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159400ba00 con 0x7f15a015d9d0 2024-04-02T07:49:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.494+0000 7f159fe00700 1 -- 172.21.15.89:0/1558104912 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159400bd90 con 0x7f15a015d9d0 2024-04-02T07:49:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.494+0000 7f15a8400700 1 -- 172.21.15.89:0/1558104912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a015d9d0 msgr2=0x7f15a00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.494+0000 7f15a8400700 1 --2- 172.21.15.89:0/1558104912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a015d9d0 0x7f15a00d3c30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f15940098f0 tx=0x7f1594009c00 comp rx=0 tx=0).stop 2024-04-02T07:49:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.494+0000 7f15a8400700 1 -- 172.21.15.89:0/1558104912 shutdown_connections 2024-04-02T07:49:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.494+0000 7f15a8400700 1 --2- 172.21.15.89:0/1558104912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a015d9d0 0x7f15a00d3c30 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.494+0000 7f15a8400700 1 --2- 172.21.15.89:0/1558104912 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15a00d16a0 0x7f15a00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.494+0000 7f15a8400700 1 -- 172.21.15.89:0/1558104912 >> 172.21.15.89:0/1558104912 conn(0x7f15a008aff0 msgr2=0x7f15a00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.494+0000 7f15a8400700 1 -- 172.21.15.89:0/1558104912 shutdown_connections 2024-04-02T07:49:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.494+0000 7f15a8400700 1 -- 172.21.15.89:0/1558104912 wait complete. 2024-04-02T07:49:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.496+0000 7f15a8400700 1 Processor -- start 2024-04-02T07:49:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.496+0000 7f15a8400700 1 -- start start 2024-04-02T07:49:25.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.497+0000 7f15a8400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a00d16a0 0x7f15a0155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.497+0000 7f15a8400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15a00d2370 0x7f15a00b8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.497+0000 7f15a8400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a015d9d0 0x7f15a0159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.497+0000 7f15a8400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a015e810 con 0x7f15a00d16a0 2024-04-02T07:49:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.497+0000 7f15a8400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a00b8df0 con 0x7f15a015d9d0 2024-04-02T07:49:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.497+0000 7f15a8400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a015dee0 con 0x7f15a00d2370 2024-04-02T07:49:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.497+0000 7f15a5a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a00d16a0 0x7f15a0155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.497+0000 7f15a5a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a00d16a0 0x7f15a0155020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37728/0 (socket says 172.21.15.89:37728) 2024-04-02T07:49:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.497+0000 7f15a5a00700 1 -- 172.21.15.89:0/3723232738 learned_addr learned my addr 172.21.15.89:0/3723232738 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:49:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.497+0000 7f15a6400700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a015d9d0 0x7f15a0159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.498+0000 7f15a5000700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15a00d2370 0x7f15a00b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.498+0000 7f15a5a00700 1 -- 172.21.15.89:0/3723232738 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15a00d2370 msgr2=0x7f15a00b8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.498+0000 7f15a5a00700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15a00d2370 0x7f15a00b8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.498+0000 7f15a5a00700 1 -- 172.21.15.89:0/3723232738 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a015d9d0 msgr2=0x7f15a0159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.498+0000 7f15a5a00700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a015d9d0 0x7f15a0159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.498+0000 7f15a5a00700 1 -- 172.21.15.89:0/3723232738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1594009550 con 0x7f15a00d16a0 2024-04-02T07:49:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.498+0000 7f15a6400700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a015d9d0 0x7f15a0159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:49:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.498+0000 7f15a5a00700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a00d16a0 0x7f15a0155020 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1594009c40 tx=0x7f1594004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.498+0000 7f159d600700 1 -- 172.21.15.89:0/3723232738 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15940036a0 con 0x7f15a00d16a0 2024-04-02T07:49:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.498+0000 7f159d600700 1 -- 172.21.15.89:0/3723232738 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159401e070 con 0x7f15a00d16a0 2024-04-02T07:49:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.498+0000 7f15a8400700 1 -- 172.21.15.89:0/3723232738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a015e1c0 con 0x7f15a00d16a0 2024-04-02T07:49:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.498+0000 7f159d600700 1 -- 172.21.15.89:0/3723232738 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594003a40 con 0x7f15a00d16a0 2024-04-02T07:49:25.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.498+0000 7f15a8400700 1 -- 172.21.15.89:0/3723232738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a0155560 con 0x7f15a00d16a0 2024-04-02T07:49:25.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.499+0000 7f15a8400700 1 -- 172.21.15.89:0/3723232738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a00d1ca0 con 0x7f15a00d16a0 2024-04-02T07:49:25.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.500+0000 7f159d600700 1 -- 172.21.15.89:0/3723232738 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f1594003ba0 con 0x7f15a00d16a0 2024-04-02T07:49:25.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.501+0000 7f159d600700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f158006ab30 0x7f158006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:49:25.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.501+0000 7f159d600700 1 -- 172.21.15.89:0/3723232738 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f1594020070 con 0x7f15a00d16a0 2024-04-02T07:49:25.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.501+0000 7f15a5000700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f158006ab30 0x7f158006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:49:25.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.502+0000 7f15a5000700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f158006ab30 0x7f158006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f15a01f6ce0 tx=0x7f158c008040 comp rx=0 tx=0).ready entity=mgr.6075 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:49:25.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.505+0000 7f159d600700 1 -- 172.21.15.89:0/3723232738 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1594055a40 con 0x7f15a00d16a0 2024-04-02T07:49:25.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.686+0000 7f159d600700 1 -- 172.21.15.89:0/3723232738 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 24) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f1594055e10 con 0x7f15a00d16a0 2024-04-02T07:49:25.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.743+0000 7f15a8400700 1 -- 172.21.15.89:0/3723232738 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f15a0161fa0 con 0x7f158006ab30 2024-04-02T07:49:25.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.754+0000 7f159d600700 1 -- 172.21.15.89:0/3723232738 <== mgr.6075 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7f158c00c5b0 con 0x7f158006ab30 2024-04-02T07:49:25.757 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:49:25.757 INFO:teuthology.orchestra.run.smithi089.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-04-02T07:49:25.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.757+0000 7f15a8400700 1 -- 172.21.15.89:0/3723232738 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f158006ab30 msgr2=0x7f158006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:25.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.757+0000 7f15a8400700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f158006ab30 0x7f158006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f15a01f6ce0 tx=0x7f158c008040 comp rx=0 tx=0).stop 2024-04-02T07:49:25.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.757+0000 7f15a8400700 1 -- 172.21.15.89:0/3723232738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a00d16a0 msgr2=0x7f15a0155020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:49:25.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.757+0000 7f15a8400700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a00d16a0 0x7f15a0155020 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1594009c40 tx=0x7f1594004750 comp rx=0 tx=0).stop 2024-04-02T07:49:25.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.757+0000 7f15a8400700 1 -- 172.21.15.89:0/3723232738 shutdown_connections 2024-04-02T07:49:25.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.757+0000 7f15a8400700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f158006ab30 0x7f158006cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:25.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.757+0000 7f15a8400700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a015d9d0 0x7f15a0159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:25.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.757+0000 7f15a8400700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a00d16a0 0x7f15a0155020 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:25.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.757+0000 7f15a8400700 1 --2- 172.21.15.89:0/3723232738 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15a00d2370 0x7f15a00b8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:49:25.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.757+0000 7f15a8400700 1 -- 172.21.15.89:0/3723232738 >> 172.21.15.89:0/3723232738 conn(0x7f15a008aff0 msgr2=0x7f15a00b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:49:25.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.757+0000 7f15a8400700 1 -- 172.21.15.89:0/3723232738 shutdown_connections 2024-04-02T07:49:25.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:49:25.757+0000 7f15a8400700 1 -- 172.21.15.89:0/3723232738 wait complete. 2024-04-02T07:49:25.777 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-04-02T07:49:25.777 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2024-04-02T07:49:25.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:25.777 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:25.824 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:25.824 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:25.824 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:26.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:26.825 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:26.877 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:26.877 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:26.877 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:27.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:27.878 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:27.931 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:27.931 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:27.931 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:28.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:28.932 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:28.984 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:28.984 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:28.984 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:29.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:29.986 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:30.037 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:30.037 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:30.037 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:31.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:31.039 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:31.090 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:31.090 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:31.090 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:32.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:32.092 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:32.143 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:32.143 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:32.143 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:33.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:33.145 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:33.197 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:33.197 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:33.197 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:34.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:34.198 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:34.250 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:34.250 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:34.250 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:35.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:35.252 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:35.304 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:35.305 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:35.305 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:36.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:36.305 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:36.359 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:36.359 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:36.359 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:37.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:37.360 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:37.412 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:37.412 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:37.412 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:38.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:38.414 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:38.466 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:38.466 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:38.466 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:39.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:39.468 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:39.519 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:39.520 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:39.520 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:40.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:40.521 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:40.573 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:40.573 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:40.573 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:41.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:41.574 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:41.626 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:41.626 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:41.626 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:42.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:42.627 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:42.679 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:42.679 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:42.679 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:43.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:43.679 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:43.731 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:43.731 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:43.731 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:44.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:44.732 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:44.784 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:44.784 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:44.784 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:44.926 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:49:44.930 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:49:45.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:45.785 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:45.837 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:45.837 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:45.837 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:46.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:46.838 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:46.889 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:46.889 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:46.889 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:47.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:47.890 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:47.942 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:47.942 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:47.942 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:48.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:48.943 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:48.996 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:48.996 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:48.996 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:49.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:49.997 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:50.049 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:50.049 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:50.049 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:51.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:51.050 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:51.103 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:51.103 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:51.103 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:52.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:52.105 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:52.156 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:52.157 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:52.157 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:53.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:53.158 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:53.210 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:53.210 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:53.210 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:54.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:54.211 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:54.263 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:54.263 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:54.263 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:55.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:55.264 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:55.317 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:55.317 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:55.317 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:56.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:56.318 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:56.370 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:56.370 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:56.370 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:57.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:57.371 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:57.423 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:57.424 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:57.424 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:58.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:58.425 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:58.477 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:58.477 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:58.477 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:49:59.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:49:59.478 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:49:59.530 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:49:59.530 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:49:59.530 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:00.293 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:50:00.290+0000 7fcbdb000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:50:00.293 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:50:00.290+0000 7fcbdb000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:50:00.293 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:50:00.290+0000 7fcbdb000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:50:00.293 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:50:00.290+0000 7fcbdb000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:50:00.293 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:50:00.291+0000 7fcbdb000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:50:00.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:00.532 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:00.580 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:00.581 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:00.581 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:00.581 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:00.582 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-04-02-07_50_00_UTC'] 2024-04-02T07:50:00.582 DEBUG:tasks.cephfs.test_snap_schedules:exec=1712044165.300076, now = 1712044200.5821128, timo = 35 2024-04-02T07:50:01.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:01.582 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:01.633 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:01.634 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:01.634 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:01.634 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:02.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:02.636 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:02.687 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:02.688 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:02.688 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:02.688 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:03.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:03.689 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:03.740 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:03.741 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:03.741 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:03.742 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:04.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:04.743 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:04.794 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:04.795 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:04.795 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:04.795 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:05.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:05.797 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:05.847 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:05.848 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:05.848 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:05.848 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:06.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:06.849 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:06.900 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:06.901 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:06.902 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:06.902 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:07.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:07.903 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:07.955 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:07.956 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:07.956 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:07.956 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:08.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:08.957 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:09.008 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:09.009 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:09.009 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:09.009 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:10.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:10.011 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:10.064 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:10.065 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:10.065 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:10.065 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:11.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:11.067 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:11.117 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:11.118 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:11.118 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:11.118 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:12.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:12.120 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:12.170 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:12.172 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:12.172 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:12.172 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:13.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:13.174 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:13.224 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:13.225 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:13.225 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:13.225 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:14.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:14.226 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:14.278 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:14.279 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:14.279 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:14.279 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:15.087 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:50:15.093 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:50:15.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:15.281 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:15.327 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:15.329 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:15.329 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:15.329 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:16.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:16.330 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:16.381 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:16.382 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:16.382 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:16.382 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:17.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:17.383 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:17.434 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:17.435 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:17.436 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:17.436 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:18.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:18.437 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:18.488 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:18.489 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:18.489 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:18.489 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:19.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:19.490 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:19.541 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:19.542 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:19.542 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:19.542 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:20.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:20.543 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:20.594 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:20.595 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:20.595 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:20.595 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:21.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:21.596 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:21.647 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:21.648 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:21.648 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:21.649 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:22.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:22.650 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:22.700 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:22.701 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:22.701 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:22.701 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:23.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:23.702 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:23.752 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:23.754 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:23.754 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:23.754 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:24.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:24.755 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:24.806 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:24.807 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:24.808 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:24.808 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:25.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:25.809 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:25.860 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:25.861 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:25.861 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:25.861 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:26.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:26.862 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:26.913 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:26.914 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:26.914 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:26.914 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:27.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:27.915 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:27.966 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:27.967 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:27.967 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:27.967 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:28.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:28.968 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:29.019 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:29.020 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:29.020 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:29.020 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:30.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:30.021 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:30.072 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:30.073 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:30.073 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:30.073 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:31.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:31.074 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:31.125 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:31.126 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:31.126 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:31.126 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:32.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:32.128 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:32.179 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:32.179 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:32.179 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:33.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:33.180 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:33.233 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:33.234 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:33.235 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:33.235 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:34.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:34.236 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:34.286 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:34.287 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:34.288 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:34.288 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:35.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:35.289 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:35.340 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:35.340 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:35.341 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:35.341 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:36.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:36.342 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:36.393 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:36.394 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:36.394 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:36.394 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:37.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:37.396 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:37.446 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:37.447 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:37.447 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:37.447 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:38.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:38.448 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:38.499 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:38.500 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:38.500 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:38.500 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:39.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:39.501 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:39.552 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:39.553 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:39.553 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:39.554 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:40.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:40.554 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:40.605 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:40.606 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:40.606 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:40.606 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:41.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:41.607 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:41.658 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:41.659 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:41.659 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:41.659 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:42.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:42.660 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:42.710 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:42.712 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:42.712 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:42.712 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:43.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:43.713 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:43.764 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:43.765 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:43.765 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:43.766 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:44.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:44.767 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:44.817 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:44.818 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:44.819 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:44.819 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:45.475 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:50:45.480 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:50:45.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:45.820 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:45.869 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:45.870 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:45.870 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:45.871 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:46.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:46.872 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:46.922 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:46.923 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:46.923 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:46.923 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:47.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:47.924 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:47.975 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:47.976 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:47.976 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:47.976 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:48.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:48.977 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:49.027 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:49.028 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:49.028 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:49.028 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:50.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:50.030 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:50.080 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:50.081 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:50.081 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:50.081 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:51.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:51.083 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:51.134 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:51.135 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:51.135 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:51.135 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:52.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:52.136 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:52.187 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:52.188 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:52.188 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:52.188 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:53.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:53.190 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:53.241 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:53.242 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:53.242 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:53.242 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:54.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:54.243 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:54.294 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:54.295 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:54.295 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:54.295 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:55.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:55.296 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:55.346 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:55.347 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:55.348 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:55.348 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:56.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:56.349 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:56.399 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:56.400 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:56.400 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:56.400 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:57.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:57.401 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:57.453 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:57.454 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:57.454 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:57.454 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:58.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:58.455 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:58.506 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:58.507 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:58.507 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:58.507 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:50:59.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:50:59.508 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:50:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:50:59.560 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:50:59.560 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:50:59.560 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:51:00.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:00.562 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:51:00.611 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:51:00.611 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_51_00_UTC 2024-04-02T07:51:00.612 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_51_00_UTC', 'scheduled-2024-04-02-07_50_00_UTC'} 2024-04-02T07:51:00.612 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-02-07_51_00_UTC'} 2024-04-02T07:51:00.612 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:51:00.612 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-04-02-07_51_00_UTC'] 2024-04-02T07:51:00.612 DEBUG:tasks.cephfs.test_snap_schedules:exec=1712044165.300076, now = 1712044260.6125197, timo = 95 2024-04-02T07:51:01.613 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-02T07:51:01.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.815+0000 7f2579600700 1 Processor -- start 2024-04-02T07:51:01.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.815+0000 7f2579600700 1 -- start start 2024-04-02T07:51:01.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.816+0000 7f2579600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f257409a750 0x7f25740d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:01.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.816+0000 7f2579600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25740d0970 con 0x7f257415d990 2024-04-02T07:51:01.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.816+0000 7f2579600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257406c020 con 0x7f2574099a60 2024-04-02T07:51:01.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.816+0000 7f2579600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25740c81a0 con 0x7f257409a750 2024-04-02T07:51:01.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.816+0000 7f2572000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f257415d990 0x7f257406c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40048/0 (socket says 172.21.15.89:40048) 2024-04-02T07:51:01.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.816+0000 7f2572000700 1 -- 172.21.15.89:0/2974201758 learned_addr learned my addr 172.21.15.89:0/2974201758 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:01.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2573400700 1 --2- 172.21.15.89:0/2974201758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f257409a750 0x7f25740d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:01.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 -- 172.21.15.89:0/2974201758 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683713081 0 0) 0x7f25740d0970 con 0x7f257415d990 2024-04-02T07:51:01.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 -- 172.21.15.89:0/2974201758 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25580035b0 con 0x7f257415d990 2024-04-02T07:51:01.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 -- 172.21.15.89:0/2974201758 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3070981490 0 0) 0x7f25580035b0 con 0x7f257415d990 2024-04-02T07:51:01.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 -- 172.21.15.89:0/2974201758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f257409a750 msgr2=0x7f25740d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:01.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 --2- 172.21.15.89:0/2974201758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f257409a750 0x7f25740d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 -- 172.21.15.89:0/2974201758 >> v1:172.21.15.113:6789/0 conn(0x7f2574099a60 legacy=0x7f2574099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 -- 172.21.15.89:0/2974201758 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25740d0690 con 0x7f257415d990 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 -- 172.21.15.89:0/2974201758 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f2560001670 con 0x7f257415d990 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 -- 172.21.15.89:0/2974201758 >> v1:172.21.15.89:6789/0 conn(0x7f257415d990 legacy=0x7f257406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 --2- 172.21.15.89:0/2974201758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558003a30 0x7f2558003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 --2- 172.21.15.89:0/2974201758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f257409a750 0x7f2558004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 --2- 172.21.15.89:0/2974201758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2558004d90 0x7f25580051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 -- 172.21.15.89:0/2974201758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25740c81a0 con 0x7f2558003a30 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 -- 172.21.15.89:0/2974201758 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255800bc20 con 0x7f2558004d90 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.817+0000 7f2571600700 1 -- 172.21.15.89:0/2974201758 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f255800bde0 con 0x7f257409a750 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.818+0000 7f2572a00700 1 --2- 172.21.15.89:0/2974201758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558003a30 0x7f2558003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.818+0000 7f2572000700 1 --2- 172.21.15.89:0/2974201758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f257409a750 0x7f2558004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.818+0000 7f2573400700 1 --2- 172.21.15.89:0/2974201758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2558004d90 0x7f25580051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.818+0000 7f2572a00700 1 -- 172.21.15.89:0/2974201758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f257409a750 msgr2=0x7f2558004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:01.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.818+0000 7f2572a00700 1 --2- 172.21.15.89:0/2974201758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f257409a750 0x7f2558004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.818+0000 7f2572a00700 1 -- 172.21.15.89:0/2974201758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2558004d90 msgr2=0x7f25580051f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.818+0000 7f2572a00700 1 --2- 172.21.15.89:0/2974201758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2558004d90 0x7f25580051f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.818+0000 7f2572a00700 1 -- 172.21.15.89:0/2974201758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25580035b0 con 0x7f2558003a30 2024-04-02T07:51:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.818+0000 7f2573400700 1 --2- 172.21.15.89:0/2974201758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2558004d90 0x7f25580051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.818+0000 7f2572a00700 1 --2- 172.21.15.89:0/2974201758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558003a30 0x7f2558003e50 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2564001800 tx=0x7f2558005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc9bcbb63e844b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.818+0000 7f2571600700 1 -- 172.21.15.89:0/2974201758 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25640012f0 con 0x7f2558003a30 2024-04-02T07:51:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.818+0000 7f2571600700 1 -- 172.21.15.89:0/2974201758 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25640052f0 con 0x7f2558003a30 2024-04-02T07:51:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.819+0000 7f2571600700 1 -- 172.21.15.89:0/2974201758 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2564006860 con 0x7f2558003a30 2024-04-02T07:51:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.819+0000 7f2579600700 1 -- 172.21.15.89:0/2974201758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558003a30 msgr2=0x7f2558003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.819+0000 7f2579600700 1 --2- 172.21.15.89:0/2974201758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558003a30 0x7f2558003e50 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2564001800 tx=0x7f2558005880 comp rx=0 tx=0).stop 2024-04-02T07:51:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.819+0000 7f2573400700 1 -- 172.21.15.89:0/2974201758 reap_dead start 2024-04-02T07:51:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.819+0000 7f2579600700 1 -- 172.21.15.89:0/2974201758 shutdown_connections 2024-04-02T07:51:01.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.819+0000 7f2579600700 1 -- 172.21.15.89:0/2974201758 >> 172.21.15.89:0/2974201758 conn(0x7f25740b2e20 msgr2=0x7f25740b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:01.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.819+0000 7f2579600700 1 -- 172.21.15.89:0/2974201758 shutdown_connections 2024-04-02T07:51:01.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.819+0000 7f2579600700 1 -- 172.21.15.89:0/2974201758 wait complete. 2024-04-02T07:51:01.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.821+0000 7f2579600700 1 Processor -- start 2024-04-02T07:51:01.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.821+0000 7f2579600700 1 -- start start 2024-04-02T07:51:01.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.822+0000 7f2579600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558003a30 0x7f25740cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:01.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.822+0000 7f2579600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25741608e0 0x7f25741f3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:01.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.822+0000 7f2579600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25741f3690 0x7f25741f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:01.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.822+0000 7f2579600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25740cd310 con 0x7f2558003a30 2024-04-02T07:51:01.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.822+0000 7f2579600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25741f4060 con 0x7f25741f3690 2024-04-02T07:51:01.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.822+0000 7f2579600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2574160d60 con 0x7f25741608e0 2024-04-02T07:51:01.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.822+0000 7f2572a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558003a30 0x7f25740cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:01.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.822+0000 7f2572a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558003a30 0x7f25740cc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54810/0 (socket says 172.21.15.89:54810) 2024-04-02T07:51:01.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.822+0000 7f2572a00700 1 -- 172.21.15.89:0/724533041 learned_addr learned my addr 172.21.15.89:0/724533041 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:01.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.822+0000 7f2572000700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25741608e0 0x7f25741f3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:01.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.822+0000 7f2573400700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25741f3690 0x7f25741f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:01.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.823+0000 7f2572a00700 1 -- 172.21.15.89:0/724533041 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25741608e0 msgr2=0x7f25741f3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:01.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.823+0000 7f2572a00700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25741608e0 0x7f25741f3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.823+0000 7f2572a00700 1 -- 172.21.15.89:0/724533041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25741f3690 msgr2=0x7f25741f3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.823+0000 7f2572a00700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25741f3690 0x7f25741f3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.823+0000 7f2572a00700 1 -- 172.21.15.89:0/724533041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25740c9a10 con 0x7f2558003a30 2024-04-02T07:51:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.823+0000 7f2573400700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25741f3690 0x7f25741f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.823+0000 7f2572a00700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558003a30 0x7f25740cc370 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f255800bbf0 tx=0x7f2564001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.823+0000 7f256aa00700 1 -- 172.21.15.89:0/724533041 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2564006b70 con 0x7f2558003a30 2024-04-02T07:51:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.823+0000 7f256aa00700 1 -- 172.21.15.89:0/724533041 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2564006cd0 con 0x7f2558003a30 2024-04-02T07:51:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.823+0000 7f256aa00700 1 -- 172.21.15.89:0/724533041 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2564007260 con 0x7f2558003a30 2024-04-02T07:51:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.823+0000 7f2579600700 1 -- 172.21.15.89:0/724533041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25740cca70 con 0x7f2558003a30 2024-04-02T07:51:01.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.823+0000 7f2579600700 1 -- 172.21.15.89:0/724533041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25740cd030 con 0x7f2558003a30 2024-04-02T07:51:01.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.824+0000 7f2579600700 1 -- 172.21.15.89:0/724533041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25740d1110 con 0x7f2558003a30 2024-04-02T07:51:01.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.825+0000 7f256aa00700 1 -- 172.21.15.89:0/724533041 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 27) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f25640073c0 con 0x7f2558003a30 2024-04-02T07:51:01.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.826+0000 7f256aa00700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f255406ae00 0x7f255406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:01.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.827+0000 7f256aa00700 1 -- 172.21.15.89:0/724533041 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f256408a8d0 con 0x7f2558003a30 2024-04-02T07:51:01.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.827+0000 7f2572000700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f255406ae00 0x7f255406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:01.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.827+0000 7f2572000700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f255406ae00 0x7f255406d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2560000c00 tx=0x7f2560000f80 comp rx=0 tx=0).ready entity=mgr.6075 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:01.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:01.831+0000 7f256aa00700 1 -- 172.21.15.89:0/724533041 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2564055700 con 0x7f2558003a30 2024-04-02T07:51:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.059+0000 7f2579600700 1 -- 172.21.15.89:0/724533041 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f25740c8620 con 0x7f255406ae00 2024-04-02T07:51:02.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.132+0000 7f256aa00700 1 -- 172.21.15.89:0/724533041 <== mgr.6075 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f25740c8620 con 0x7f255406ae00 2024-04-02T07:51:02.134 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path snap_test_dir1 2024-04-02T07:51:02.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.134+0000 7f2579600700 1 -- 172.21.15.89:0/724533041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f255406ae00 msgr2=0x7f255406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:02.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.134+0000 7f2579600700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f255406ae00 0x7f255406d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2560000c00 tx=0x7f2560000f80 comp rx=0 tx=0).stop 2024-04-02T07:51:02.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.134+0000 7f2579600700 1 -- 172.21.15.89:0/724533041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558003a30 msgr2=0x7f25740cc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:02.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.134+0000 7f2579600700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558003a30 0x7f25740cc370 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f255800bbf0 tx=0x7f2564001120 comp rx=0 tx=0).stop 2024-04-02T07:51:02.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.134+0000 7f2579600700 1 -- 172.21.15.89:0/724533041 shutdown_connections 2024-04-02T07:51:02.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.134+0000 7f2579600700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f255406ae00 0x7f255406d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:02.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.134+0000 7f2579600700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25741f3690 0x7f25741f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:02.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.134+0000 7f2579600700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2558003a30 0x7f25740cc370 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:02.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.134+0000 7f2579600700 1 --2- 172.21.15.89:0/724533041 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25741608e0 0x7f25741f3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:02.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.134+0000 7f2579600700 1 -- 172.21.15.89:0/724533041 >> 172.21.15.89:0/724533041 conn(0x7f25740b2e20 msgr2=0x7f257406c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:02.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.134+0000 7f2579600700 1 -- 172.21.15.89:0/724533041 shutdown_connections 2024-04-02T07:51:02.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.134+0000 7f2579600700 1 -- 172.21.15.89:0/724533041 wait complete. 2024-04-02T07:51:02.154 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-02T07:51:02.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:02.154 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:51:02.198 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:51:02.199 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_51_00_UTC 2024-04-02T07:51:02.199 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-07_50_00_UTC 2024-04-02T07:51:02.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:02.200 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-07_50_00_UTC) 2024-04-02T07:51:02.290 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-07_51_00_UTC 2024-04-02T07:51:02.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:02.290 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-07_51_00_UTC) 2024-04-02T07:51:02.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:02.340 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-02T07:51:02.388 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T07:51:02.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.578+0000 7f290f400700 1 Processor -- start 2024-04-02T07:51:02.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.578+0000 7f290f400700 1 -- start start 2024-04-02T07:51:02.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.579+0000 7f290f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29081611b0 0x7f290815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:02.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.579+0000 7f290f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f290815f4d0 0x7f29080ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:02.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.579+0000 7f290f400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29080ce270 con 0x7f29080d21f0 2024-04-02T07:51:02.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.579+0000 7f290f400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f290815b600 con 0x7f290815f4d0 2024-04-02T07:51:02.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.580+0000 7f290f400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29080d2760 con 0x7f29081611b0 2024-04-02T07:51:02.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.580+0000 7f2907e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f290815f4d0 0x7f29080ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:02.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.580+0000 7f2907e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f290815f4d0 0x7f29080ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54820/0 (socket says 172.21.15.89:54820) 2024-04-02T07:51:02.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.580+0000 7f2907e00700 1 -- 172.21.15.89:0/3588529007 learned_addr learned my addr 172.21.15.89:0/3588529007 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:02.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.580+0000 7f290ca00700 1 --2- 172.21.15.89:0/3588529007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29081611b0 0x7f290815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:02.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.580+0000 7f2907e00700 1 -- 172.21.15.89:0/3588529007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29081611b0 msgr2=0x7f290815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:02.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.580+0000 7f2907e00700 1 --2- 172.21.15.89:0/3588529007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29081611b0 0x7f290815ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:02.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.580+0000 7f2907e00700 1 -- 172.21.15.89:0/3588529007 >> v1:172.21.15.113:6789/0 conn(0x7f29080d21f0 legacy=0x7f2908161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:02.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.580+0000 7f2907e00700 1 -- 172.21.15.89:0/3588529007 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29080d2a40 con 0x7f290815f4d0 2024-04-02T07:51:02.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.580+0000 7f2907e00700 1 --2- 172.21.15.89:0/3588529007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f290815f4d0 0x7f29080ce700 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f28f40098f0 tx=0x7f28f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b866c679654855f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:02.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.581+0000 7f2907400700 1 -- 172.21.15.89:0/3588529007 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f400b8a0 con 0x7f290815f4d0 2024-04-02T07:51:02.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.581+0000 7f2907400700 1 -- 172.21.15.89:0/3588529007 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f400ba00 con 0x7f290815f4d0 2024-04-02T07:51:02.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.581+0000 7f2907400700 1 -- 172.21.15.89:0/3588529007 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f400bd90 con 0x7f290815f4d0 2024-04-02T07:51:02.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.581+0000 7f290f400700 1 -- 172.21.15.89:0/3588529007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f290815f4d0 msgr2=0x7f29080ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:02.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.581+0000 7f290f400700 1 --2- 172.21.15.89:0/3588529007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f290815f4d0 0x7f29080ce700 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f28f40098f0 tx=0x7f28f4009c00 comp rx=0 tx=0).stop 2024-04-02T07:51:02.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.581+0000 7f290f400700 1 -- 172.21.15.89:0/3588529007 shutdown_connections 2024-04-02T07:51:02.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.581+0000 7f290f400700 1 --2- 172.21.15.89:0/3588529007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29081611b0 0x7f290815ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.581+0000 7f290f400700 1 --2- 172.21.15.89:0/3588529007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f290815f4d0 0x7f29080ce700 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.581+0000 7f290f400700 1 -- 172.21.15.89:0/3588529007 >> 172.21.15.89:0/3588529007 conn(0x7f290808aff0 msgr2=0x7f29080d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.581+0000 7f290f400700 1 -- 172.21.15.89:0/3588529007 shutdown_connections 2024-04-02T07:51:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.581+0000 7f290f400700 1 -- 172.21.15.89:0/3588529007 wait complete. 2024-04-02T07:51:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.583+0000 7f290f400700 1 Processor -- start 2024-04-02T07:51:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.583+0000 7f290f400700 1 -- start start 2024-04-02T07:51:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.584+0000 7f290f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29080d21f0 0x7f2908156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.584+0000 7f290f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f290815f4d0 0x7f290806be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.584+0000 7f290f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29081611b0 0x7f290806c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.584+0000 7f290f400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29080ca6a0 con 0x7f290815f4d0 2024-04-02T07:51:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.584+0000 7f290f400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29080ca980 con 0x7f29080d21f0 2024-04-02T07:51:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.584+0000 7f290f400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29080cac60 con 0x7f29081611b0 2024-04-02T07:51:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.584+0000 7f2907e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f290815f4d0 0x7f290806be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.584+0000 7f2907e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f290815f4d0 0x7f290806be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54822/0 (socket says 172.21.15.89:54822) 2024-04-02T07:51:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.584+0000 7f2907e00700 1 -- 172.21.15.89:0/491342337 learned_addr learned my addr 172.21.15.89:0/491342337 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.584+0000 7f290ca00700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29080d21f0 0x7f2908156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.585+0000 7f290d400700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29081611b0 0x7f290806c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.585+0000 7f2907e00700 1 -- 172.21.15.89:0/491342337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29081611b0 msgr2=0x7f290806c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.585+0000 7f2907e00700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29081611b0 0x7f290806c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.585+0000 7f2907e00700 1 -- 172.21.15.89:0/491342337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29080d21f0 msgr2=0x7f2908156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.585+0000 7f2907e00700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29080d21f0 0x7f2908156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.585+0000 7f2907e00700 1 -- 172.21.15.89:0/491342337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f4009550 con 0x7f290815f4d0 2024-04-02T07:51:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.585+0000 7f290ca00700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29080d21f0 0x7f2908156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.585+0000 7f2907e00700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f290815f4d0 0x7f290806be80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f28f4000c00 tx=0x7f28f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.585+0000 7f2904c00700 1 -- 172.21.15.89:0/491342337 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4004480 con 0x7f290815f4d0 2024-04-02T07:51:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.585+0000 7f2904c00700 1 -- 172.21.15.89:0/491342337 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f401f070 con 0x7f290815f4d0 2024-04-02T07:51:02.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.585+0000 7f290f400700 1 -- 172.21.15.89:0/491342337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29080caf40 con 0x7f290815f4d0 2024-04-02T07:51:02.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.585+0000 7f2904c00700 1 -- 172.21.15.89:0/491342337 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f40038c0 con 0x7f290815f4d0 2024-04-02T07:51:02.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.585+0000 7f290f400700 1 -- 172.21.15.89:0/491342337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29080b7a70 con 0x7f290815f4d0 2024-04-02T07:51:02.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.586+0000 7f290f400700 1 -- 172.21.15.89:0/491342337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2908067720 con 0x7f290815f4d0 2024-04-02T07:51:02.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.587+0000 7f2904c00700 1 -- 172.21.15.89:0/491342337 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 27) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f28f4021030 con 0x7f290815f4d0 2024-04-02T07:51:02.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.588+0000 7f2904c00700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f28e806ae00 0x7f28e806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:02.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.588+0000 7f2904c00700 1 -- 172.21.15.89:0/491342337 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f28f408ad90 con 0x7f290815f4d0 2024-04-02T07:51:02.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.588+0000 7f290ca00700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f28e806ae00 0x7f28e806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:02.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.589+0000 7f290ca00700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f28e806ae00 0x7f28e806d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00ba70 tx=0x7f28fc008040 comp rx=0 tx=0).ready entity=mgr.6075 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:02.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.593+0000 7f2904c00700 1 -- 172.21.15.89:0/491342337 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f28f4055ca0 con 0x7f290815f4d0 2024-04-02T07:51:02.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:02.854+0000 7f290f400700 1 -- 172.21.15.89:0/491342337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f29080ca7e0 con 0x7f290815f4d0 2024-04-02T07:51:03.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.007+0000 7f2904c00700 1 -- 172.21.15.89:0/491342337 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f28f4090020 con 0x7f290815f4d0 2024-04-02T07:51:03.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.008+0000 7f2904c00700 1 -- 172.21.15.89:0/491342337 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v28) v1 ==== 93+0+0 (secure 0 0 0) 0x7f28f406de90 con 0x7f290815f4d0 2024-04-02T07:51:03.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.011+0000 7f290f400700 1 -- 172.21.15.89:0/491342337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f28e806ae00 msgr2=0x7f28e806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:03.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.011+0000 7f290f400700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f28e806ae00 0x7f28e806d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00ba70 tx=0x7f28fc008040 comp rx=0 tx=0).stop 2024-04-02T07:51:03.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.011+0000 7f290f400700 1 -- 172.21.15.89:0/491342337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f290815f4d0 msgr2=0x7f290806be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:03.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.011+0000 7f290f400700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f290815f4d0 0x7f290806be80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f28f4000c00 tx=0x7f28f4004750 comp rx=0 tx=0).stop 2024-04-02T07:51:03.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.011+0000 7f290f400700 1 -- 172.21.15.89:0/491342337 shutdown_connections 2024-04-02T07:51:03.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.011+0000 7f290f400700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f28e806ae00 0x7f28e806d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:03.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.011+0000 7f290f400700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29080d21f0 0x7f2908156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:03.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.011+0000 7f290f400700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29081611b0 0x7f290806c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:03.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.011+0000 7f290f400700 1 --2- 172.21.15.89:0/491342337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f290815f4d0 0x7f290806be80 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:03.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.011+0000 7f290f400700 1 -- 172.21.15.89:0/491342337 >> 172.21.15.89:0/491342337 conn(0x7f290808aff0 msgr2=0x7f29080d1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:03.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.011+0000 7f290f400700 1 -- 172.21.15.89:0/491342337 shutdown_connections 2024-04-02T07:51:03.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.011+0000 7f290f400700 1 -- 172.21.15.89:0/491342337 wait complete. 2024-04-02T07:51:03.032 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:51:03.032 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:51:03.117 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:51:03.208 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:51:03.208 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:51:03.208 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:03.229 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:03.227+0000 7ff1aa269240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:51:03.233 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:03.232+0000 7fde23f68240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:51:03.253 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:51:03.253 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:03.339 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:51:03.415 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:03.412+0000 7ff1aa269240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:51:03.423 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:03.421+0000 7fde23f68240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:51:03.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.593+0000 7f9da4e00700 1 Processor -- start 2024-04-02T07:51:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.593+0000 7f9da4e00700 1 -- start start 2024-04-02T07:51:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.594+0000 7f9da4e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9da00b3af0 0x7f9da015a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.595+0000 7f9da4e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9da0072580 0x7f9da00b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.595+0000 7f9da4e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da00b2180 con 0x7f9da00b2e50 2024-04-02T07:51:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.595+0000 7f9da4e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da015b100 con 0x7f9da00b3af0 2024-04-02T07:51:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.595+0000 7f9da4e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da015b3e0 con 0x7f9da0072580 2024-04-02T07:51:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.595+0000 7f9d9de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9da00b3af0 0x7f9da015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.595+0000 7f9d9e800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9da00b2e50 0x7f9da00709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40066/0 (socket says 172.21.15.89:40066) 2024-04-02T07:51:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.595+0000 7f9d9e800700 1 -- 172.21.15.89:0/3545671469 learned_addr learned my addr 172.21.15.89:0/3545671469 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.595+0000 7f9d9d400700 1 --2- 172.21.15.89:0/3545671469 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9da0072580 0x7f9da00b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.595+0000 7f9d9de00700 1 -- 172.21.15.89:0/3545671469 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9da0072580 msgr2=0x7f9da00b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.595+0000 7f9d9de00700 1 --2- 172.21.15.89:0/3545671469 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9da0072580 0x7f9da00b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.595+0000 7f9d9de00700 1 -- 172.21.15.89:0/3545671469 >> v1:172.21.15.89:6789/0 conn(0x7f9da00b2e50 legacy=0x7f9da00709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.595+0000 7f9d9ca00700 1 -- 172.21.15.89:0/3545671469 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679198870 0 0) 0x7f9da00b2180 con 0x7f9da00b2e50 2024-04-02T07:51:03.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.596+0000 7f9d9de00700 1 -- 172.21.15.89:0/3545671469 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da00b23f0 con 0x7f9da00b3af0 2024-04-02T07:51:03.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.596+0000 7f9d9de00700 1 --2- 172.21.15.89:0/3545671469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9da00b3af0 0x7f9da015a860 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9d90009840 tx=0x7f9d90009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c1dc6f1dd72f1d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:03.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.596+0000 7f9d9ca00700 1 -- 172.21.15.89:0/3545671469 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d90011830 con 0x7f9da00b3af0 2024-04-02T07:51:03.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.596+0000 7f9d9ca00700 1 -- 172.21.15.89:0/3545671469 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d90011990 con 0x7f9da00b3af0 2024-04-02T07:51:03.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.596+0000 7f9d9ca00700 1 -- 172.21.15.89:0/3545671469 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d90011d20 con 0x7f9da00b3af0 2024-04-02T07:51:03.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.597+0000 7f9da4e00700 1 -- 172.21.15.89:0/3545671469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9da00b3af0 msgr2=0x7f9da015a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:03.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.597+0000 7f9da4e00700 1 --2- 172.21.15.89:0/3545671469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9da00b3af0 0x7f9da015a860 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9d90009840 tx=0x7f9d90009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:03.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.597+0000 7f9da4e00700 1 -- 172.21.15.89:0/3545671469 shutdown_connections 2024-04-02T07:51:03.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.597+0000 7f9da4e00700 1 --2- 172.21.15.89:0/3545671469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9da00b3af0 0x7f9da015a860 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:03.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.597+0000 7f9da4e00700 1 --2- 172.21.15.89:0/3545671469 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9da0072580 0x7f9da00b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:03.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.597+0000 7f9da4e00700 1 -- 172.21.15.89:0/3545671469 >> 172.21.15.89:0/3545671469 conn(0x7f9da00cf190 msgr2=0x7f9da00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:03.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.597+0000 7f9da4e00700 1 -- 172.21.15.89:0/3545671469 shutdown_connections 2024-04-02T07:51:03.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.597+0000 7f9da4e00700 1 -- 172.21.15.89:0/3545671469 wait complete. 2024-04-02T07:51:03.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.599+0000 7f9da4e00700 1 Processor -- start 2024-04-02T07:51:03.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.599+0000 7f9da4e00700 1 -- start start 2024-04-02T07:51:03.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9da4e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9da0072580 0x7f9da01574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:03.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9da4e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9da00b2e50 0x7f9da014f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:03.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9da4e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9da00b3af0 0x7f9da014fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9da4e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da0150020 con 0x7f9da00b3af0 2024-04-02T07:51:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9da4e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da01553e0 con 0x7f9da00b2e50 2024-04-02T07:51:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9da4e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da01556c0 con 0x7f9da0072580 2024-04-02T07:51:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9d9e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9da00b3af0 0x7f9da014fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9d9de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9da0072580 0x7f9da01574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9d9e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9da00b3af0 0x7f9da014fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54872/0 (socket says 172.21.15.89:54872) 2024-04-02T07:51:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9d9e800700 1 -- 172.21.15.89:0/1969394878 learned_addr learned my addr 172.21.15.89:0/1969394878 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9d9de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9da0072580 0x7f9da01574e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:51230/0 (socket says 172.21.15.89:51230) 2024-04-02T07:51:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9d9d400700 1 --2- 172.21.15.89:0/1969394878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9da00b2e50 0x7f9da014f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9d9de00700 1 -- 172.21.15.89:0/1969394878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9da00b2e50 msgr2=0x7f9da014f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9d9de00700 1 --2- 172.21.15.89:0/1969394878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9da00b2e50 0x7f9da014f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9d9de00700 1 -- 172.21.15.89:0/1969394878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9da00b3af0 msgr2=0x7f9da014fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:03.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9d9de00700 1 --2- 172.21.15.89:0/1969394878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9da00b3af0 0x7f9da014fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:03.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.600+0000 7f9d9de00700 1 -- 172.21.15.89:0/1969394878 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d900094a0 con 0x7f9da0072580 2024-04-02T07:51:03.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.601+0000 7f9d9de00700 1 --2- 172.21.15.89:0/1969394878 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9da0072580 0x7f9da01574e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9d90016000 tx=0x7f9d90004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:03.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.601+0000 7f9d96000700 1 -- 172.21.15.89:0/1969394878 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d900036a0 con 0x7f9da0072580 2024-04-02T07:51:03.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.601+0000 7f9d96000700 1 -- 172.21.15.89:0/1969394878 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9001f070 con 0x7f9da0072580 2024-04-02T07:51:03.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.601+0000 7f9da4e00700 1 -- 172.21.15.89:0/1969394878 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da01559a0 con 0x7f9da0072580 2024-04-02T07:51:03.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.601+0000 7f9d96000700 1 -- 172.21.15.89:0/1969394878 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d90003a20 con 0x7f9da0072580 2024-04-02T07:51:03.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.601+0000 7f9da4e00700 1 -- 172.21.15.89:0/1969394878 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da0155c80 con 0x7f9da0072580 2024-04-02T07:51:03.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.602+0000 7f9da4e00700 1 -- 172.21.15.89:0/1969394878 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da01589e0 con 0x7f9da0072580 2024-04-02T07:51:03.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.603+0000 7f9d96000700 1 -- 172.21.15.89:0/1969394878 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f9d90003b80 con 0x7f9da0072580 2024-04-02T07:51:03.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.604+0000 7f9d96000700 1 --2- 172.21.15.89:0/1969394878 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d8006adb0 0x7f9d8006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:03.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.604+0000 7f9d96000700 1 -- 172.21.15.89:0/1969394878 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f9d90021070 con 0x7f9da0072580 2024-04-02T07:51:03.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.604+0000 7f9d9d400700 1 -- 172.21.15.89:0/1969394878 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d8006adb0 msgr2=0x7f9d8006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:03.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.604+0000 7f9d9d400700 1 --2- 172.21.15.89:0/1969394878 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d8006adb0 0x7f9d8006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:03.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.609+0000 7f9d96000700 1 -- 172.21.15.89:0/1969394878 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9da01589e0 con 0x7f9da0072580 2024-04-02T07:51:03.730 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:03.728+0000 7ff1aa269240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:51:03.757 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:03.755+0000 7fde23f68240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:51:03.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.805+0000 7f9d9d400700 1 -- 172.21.15.89:0/1969394878 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d8006adb0 msgr2=0x7f9d8006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:03.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.805+0000 7f9d9d400700 1 --2- 172.21.15.89:0/1969394878 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d8006adb0 0x7f9d8006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:03.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.820+0000 7f9da4e00700 1 -- 172.21.15.89:0/1969394878 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9da01589e0 con 0x7f9da0072580 2024-04-02T07:51:03.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.822+0000 7f9d96000700 1 -- 172.21.15.89:0/1969394878 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+17349 (secure 0 0 0) 0x7f9da01589e0 con 0x7f9da0072580 2024-04-02T07:51:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:03.824 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":69,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:50:25.037631+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:49:05.141462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-02T07:49:06.232906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.9","mappings":[{"from":5,"to":0}]},{"pgid":"7.14","mappings":[{"from":5,"to":7}]},{"pgid":"7.15","mappings":[{"from":5,"to":7}]},{"pgid":"7.30","mappings":[{"from":5,"to":7}]},{"pgid":"7.35","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/379543030":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/3951589296":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/3713107589":"2024-04-03T07:49:23.665126+0000","172.21.15.89:6833/43357":"2024-04-03T07:49:23.665126+0000","172.21.15.89:6832/43357":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/552943692":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/2589801503":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/3028439559":"2024-04-03T07:49:23.665126+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:51:03.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.824+0000 7f9da4e00700 1 -- 172.21.15.89:0/1969394878 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d8006adb0 msgr2=0x7f9d8006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:03.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.824+0000 7f9da4e00700 1 --2- 172.21.15.89:0/1969394878 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d8006adb0 0x7f9d8006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:03.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.824+0000 7f9da4e00700 1 -- 172.21.15.89:0/1969394878 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9da0072580 msgr2=0x7f9da01574e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:03.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.824+0000 7f9da4e00700 1 --2- 172.21.15.89:0/1969394878 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9da0072580 0x7f9da01574e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9d90016000 tx=0x7f9d90004750 comp rx=0 tx=0).stop 2024-04-02T07:51:03.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.824+0000 7f9da4e00700 1 -- 172.21.15.89:0/1969394878 shutdown_connections 2024-04-02T07:51:03.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.824+0000 7f9da4e00700 1 --2- 172.21.15.89:0/1969394878 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d8006adb0 0x7f9d8006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:03.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.824+0000 7f9da4e00700 1 --2- 172.21.15.89:0/1969394878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9da00b3af0 0x7f9da014fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:03.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.824+0000 7f9da4e00700 1 --2- 172.21.15.89:0/1969394878 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9da0072580 0x7f9da01574e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:03.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.824+0000 7f9da4e00700 1 --2- 172.21.15.89:0/1969394878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9da00b2e50 0x7f9da014f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:03.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.824+0000 7f9da4e00700 1 -- 172.21.15.89:0/1969394878 >> 172.21.15.89:0/1969394878 conn(0x7f9da00cf190 msgr2=0x7f9da00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:03.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.824+0000 7f9da4e00700 1 -- 172.21.15.89:0/1969394878 shutdown_connections 2024-04-02T07:51:03.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:03.824+0000 7f9da4e00700 1 -- 172.21.15.89:0/1969394878 wait complete. 2024-04-02T07:51:03.845 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:49:05.141462+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:49:06.232906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:51:03.845 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:51:04.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.083+0000 7f99a5200700 1 Processor -- start 2024-04-02T07:51:04.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.083+0000 7f99a5200700 1 -- start start 2024-04-02T07:51:04.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.085+0000 7f99a5200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f99a015d9d0 0x7f99a00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:04.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.085+0000 7f99a5200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a0161420 con 0x7f99a00d2540 2024-04-02T07:51:04.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.085+0000 7f99a5200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a0161700 con 0x7f99a00d16a0 2024-04-02T07:51:04.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.085+0000 7f99a5200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a015edc0 con 0x7f99a015d9d0 2024-04-02T07:51:04.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.085+0000 7f999de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f99a015d9d0 0x7f99a00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:04.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.085+0000 7f999e800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f99a00d2540 0x7f99a00ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40078/0 (socket says 172.21.15.89:40078) 2024-04-02T07:51:04.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.085+0000 7f999de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f99a015d9d0 0x7f99a00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54878/0 (socket says 172.21.15.89:54878) 2024-04-02T07:51:04.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.085+0000 7f999e800700 1 -- 172.21.15.89:0/533871391 learned_addr learned my addr 172.21.15.89:0/533871391 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:04.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.086+0000 7f999de00700 1 -- 172.21.15.89:0/533871391 >> v1:172.21.15.113:6789/0 conn(0x7f99a00d16a0 legacy=0x7f99a00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:04.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.086+0000 7f999de00700 1 -- 172.21.15.89:0/533871391 >> v1:172.21.15.89:6789/0 conn(0x7f99a00d2540 legacy=0x7f99a00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:04.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.086+0000 7f999ca00700 1 -- 172.21.15.89:0/533871391 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233954292 0 0) 0x7f99a0161700 con 0x7f99a00d16a0 2024-04-02T07:51:04.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.086+0000 7f999de00700 1 -- 172.21.15.89:0/533871391 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a01619e0 con 0x7f99a015d9d0 2024-04-02T07:51:04.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.086+0000 7f999de00700 1 --2- 172.21.15.89:0/533871391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99a015d9d0 0x7f99a00d3c30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9990009840 tx=0x7f9990009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd18051d7392622e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:04.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.086+0000 7f999ca00700 1 -- 172.21.15.89:0/533871391 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9990004230 con 0x7f99a015d9d0 2024-04-02T07:51:04.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.086+0000 7f999ca00700 1 -- 172.21.15.89:0/533871391 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9990004390 con 0x7f99a015d9d0 2024-04-02T07:51:04.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.086+0000 7f999ca00700 1 -- 172.21.15.89:0/533871391 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99900036a0 con 0x7f99a015d9d0 2024-04-02T07:51:04.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.087+0000 7f99a5200700 1 -- 172.21.15.89:0/533871391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99a015d9d0 msgr2=0x7f99a00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:04.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.087+0000 7f99a5200700 1 --2- 172.21.15.89:0/533871391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99a015d9d0 0x7f99a00d3c30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9990009840 tx=0x7f9990009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:04.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.087+0000 7f99a5200700 1 -- 172.21.15.89:0/533871391 shutdown_connections 2024-04-02T07:51:04.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.087+0000 7f99a5200700 1 --2- 172.21.15.89:0/533871391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99a015d9d0 0x7f99a00d3c30 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.087+0000 7f99a5200700 1 -- 172.21.15.89:0/533871391 >> 172.21.15.89:0/533871391 conn(0x7f99a008aff0 msgr2=0x7f99a00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:04.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.087+0000 7f99a5200700 1 -- 172.21.15.89:0/533871391 shutdown_connections 2024-04-02T07:51:04.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.087+0000 7f99a5200700 1 -- 172.21.15.89:0/533871391 wait complete. 2024-04-02T07:51:04.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.089+0000 7f99a5200700 1 Processor -- start 2024-04-02T07:51:04.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.089+0000 7f99a5200700 1 -- start start 2024-04-02T07:51:04.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.090+0000 7f99a5200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99a00d16a0 0x7f99a00b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:04.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.090+0000 7f99a5200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99a00d2540 0x7f99a014f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:04.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.090+0000 7f99a5200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99a015d9d0 0x7f99a01e8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:04.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.090+0000 7f99a5200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a00b6900 con 0x7f99a00d2540 2024-04-02T07:51:04.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.090+0000 7f99a5200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a00b6be0 con 0x7f99a00d16a0 2024-04-02T07:51:04.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.090+0000 7f99a5200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a00b6ec0 con 0x7f99a015d9d0 2024-04-02T07:51:04.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.090+0000 7f999d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99a00d2540 0x7f99a014f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:04.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.090+0000 7f999de00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99a00d16a0 0x7f99a00b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:04.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.090+0000 7f999d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99a00d2540 0x7f99a014f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54892/0 (socket says 172.21.15.89:54892) 2024-04-02T07:51:04.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.090+0000 7f999d400700 1 -- 172.21.15.89:0/588598040 learned_addr learned my addr 172.21.15.89:0/588598040 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:04.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.091+0000 7f999e800700 1 --2- 172.21.15.89:0/588598040 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99a015d9d0 0x7f99a01e8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:04.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.091+0000 7f999d400700 1 -- 172.21.15.89:0/588598040 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99a015d9d0 msgr2=0x7f99a01e8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:04.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.091+0000 7f999d400700 1 --2- 172.21.15.89:0/588598040 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99a015d9d0 0x7f99a01e8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.091+0000 7f999d400700 1 -- 172.21.15.89:0/588598040 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99a00d16a0 msgr2=0x7f99a00b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:04.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.091+0000 7f999d400700 1 --2- 172.21.15.89:0/588598040 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99a00d16a0 0x7f99a00b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.091+0000 7f999d400700 1 -- 172.21.15.89:0/588598040 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99900094a0 con 0x7f99a00d2540 2024-04-02T07:51:04.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.091+0000 7f999de00700 1 --2- 172.21.15.89:0/588598040 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99a00d16a0 0x7f99a00b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:04.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.091+0000 7f999d400700 1 --2- 172.21.15.89:0/588598040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99a00d2540 0x7f99a014f850 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f998c00c4a0 tx=0x7f998c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:04.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.091+0000 7f9996000700 1 -- 172.21.15.89:0/588598040 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c0054e0 con 0x7f99a00d2540 2024-04-02T07:51:04.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.091+0000 7f9996000700 1 -- 172.21.15.89:0/588598040 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998c0036d0 con 0x7f99a00d2540 2024-04-02T07:51:04.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.092+0000 7f99a5200700 1 -- 172.21.15.89:0/588598040 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a00b71a0 con 0x7f99a00d2540 2024-04-02T07:51:04.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.092+0000 7f9996000700 1 -- 172.21.15.89:0/588598040 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c003a80 con 0x7f99a00d2540 2024-04-02T07:51:04.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.092+0000 7f99a5200700 1 -- 172.21.15.89:0/588598040 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a00b6560 con 0x7f99a00d2540 2024-04-02T07:51:04.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.093+0000 7f99a5200700 1 -- 172.21.15.89:0/588598040 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a00d1ca0 con 0x7f99a00d2540 2024-04-02T07:51:04.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.094+0000 7f9996000700 1 -- 172.21.15.89:0/588598040 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f998c003be0 con 0x7f99a00d2540 2024-04-02T07:51:04.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.096+0000 7f9996000700 1 --2- 172.21.15.89:0/588598040 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f998006ade0 0x7f998006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:04.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.096+0000 7f999de00700 1 -- 172.21.15.89:0/588598040 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f998006ade0 msgr2=0x7f998006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:04.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.096+0000 7f9996000700 1 -- 172.21.15.89:0/588598040 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f998c088b10 con 0x7f99a00d2540 2024-04-02T07:51:04.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.096+0000 7f999de00700 1 --2- 172.21.15.89:0/588598040 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f998006ade0 0x7f998006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:04.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.101+0000 7f9996000700 1 -- 172.21.15.89:0/588598040 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f998c053890 con 0x7f99a00d2540 2024-04-02T07:51:04.254 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:04.252+0000 7ff1aa269240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:51:04.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.297+0000 7f999de00700 1 -- 172.21.15.89:0/588598040 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f998006ade0 msgr2=0x7f998006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:04.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.297+0000 7f999de00700 1 --2- 172.21.15.89:0/588598040 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f998006ade0 0x7f998006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:04.332 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:04.331+0000 7fde23f68240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:51:04.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.342+0000 7f99a5200700 1 -- 172.21.15.89:0/588598040 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f99a00b76c0 con 0x7f99a00d2540 2024-04-02T07:51:04.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.343+0000 7f9996000700 1 -- 172.21.15.89:0/588598040 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f99a00b76c0 con 0x7f99a00d2540 2024-04-02T07:51:04.345 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:51:04.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.345+0000 7f99a5200700 1 -- 172.21.15.89:0/588598040 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f998006ade0 msgr2=0x7f998006d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:04.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.345+0000 7f99a5200700 1 --2- 172.21.15.89:0/588598040 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f998006ade0 0x7f998006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.345+0000 7f99a5200700 1 -- 172.21.15.89:0/588598040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99a00d2540 msgr2=0x7f99a014f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:04.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.345+0000 7f99a5200700 1 --2- 172.21.15.89:0/588598040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99a00d2540 0x7f99a014f850 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f998c00c4a0 tx=0x7f998c00c7b0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.345+0000 7f99a5200700 1 -- 172.21.15.89:0/588598040 shutdown_connections 2024-04-02T07:51:04.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.345+0000 7f99a5200700 1 --2- 172.21.15.89:0/588598040 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f998006ade0 0x7f998006d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.345+0000 7f99a5200700 1 --2- 172.21.15.89:0/588598040 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99a00d16a0 0x7f99a00b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.345+0000 7f99a5200700 1 --2- 172.21.15.89:0/588598040 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99a015d9d0 0x7f99a01e8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.345+0000 7f99a5200700 1 --2- 172.21.15.89:0/588598040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99a00d2540 0x7f99a014f850 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.345+0000 7f99a5200700 1 -- 172.21.15.89:0/588598040 >> 172.21.15.89:0/588598040 conn(0x7f99a008aff0 msgr2=0x7f99a00b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:04.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.346+0000 7f99a5200700 1 -- 172.21.15.89:0/588598040 shutdown_connections 2024-04-02T07:51:04.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.346+0000 7f99a5200700 1 -- 172.21.15.89:0/588598040 wait complete. 2024-04-02T07:51:04.366 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:51:04.508 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:04.506+0000 7ff1aa269240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:51:04.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.577+0000 7f3f5f400700 1 Processor -- start 2024-04-02T07:51:04.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.577+0000 7f3f5f400700 1 -- start start 2024-04-02T07:51:04.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.578+0000 7f3f5f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3f5815a5c0 0x7f3f580709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:04.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.578+0000 7f3f5f400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3f580ce840 0x7f3f580cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:04.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.578+0000 7f3f5f400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f580cb310 con 0x7f3f5815dfe0 2024-04-02T07:51:04.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.578+0000 7f3f5f400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f58070f00 con 0x7f3f5815a5c0 2024-04-02T07:51:04.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.578+0000 7f3f5f400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f580711a0 con 0x7f3f580ce840 2024-04-02T07:51:04.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.578+0000 7f3f5ca00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3f5815a5c0 0x7f3f580709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:04.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.578+0000 7f3f5ca00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3f5815a5c0 0x7f3f580709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54900/0 (socket says 172.21.15.89:54900) 2024-04-02T07:51:04.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.578+0000 7f3f5ca00700 1 -- 172.21.15.89:0/2268739791 learned_addr learned my addr 172.21.15.89:0/2268739791 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:04.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.579+0000 7f3f5d400700 1 --2- 172.21.15.89:0/2268739791 >> v2:172.21.15.113:3300/0 conn(0x7f3f580ce840 0x7f3f580cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:04.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.579+0000 7f3f5ca00700 1 -- 172.21.15.89:0/2268739791 >> v2:172.21.15.113:3300/0 conn(0x7f3f580ce840 msgr2=0x7f3f580cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:04.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.579+0000 7f3f5ca00700 1 --2- 172.21.15.89:0/2268739791 >> v2:172.21.15.113:3300/0 conn(0x7f3f580ce840 0x7f3f580cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.579+0000 7f3f5ca00700 1 -- 172.21.15.89:0/2268739791 >> v1:172.21.15.113:6789/0 conn(0x7f3f5815dfe0 legacy=0x7f3f580ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:04.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.579+0000 7f3f57400700 1 -- 172.21.15.89:0/2268739791 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444368542 0 0) 0x7f3f580cb310 con 0x7f3f5815dfe0 2024-04-02T07:51:04.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.579+0000 7f3f5ca00700 1 -- 172.21.15.89:0/2268739791 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f58071480 con 0x7f3f5815a5c0 2024-04-02T07:51:04.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.579+0000 7f3f5ca00700 1 --2- 172.21.15.89:0/2268739791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5815a5c0 0x7f3f580709c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f3f5806b630 tx=0x7f3f4c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0a51d350db0594c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:04.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.579+0000 7f3f57400700 1 -- 172.21.15.89:0/2268739791 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c011830 con 0x7f3f5815a5c0 2024-04-02T07:51:04.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.580+0000 7f3f57400700 1 -- 172.21.15.89:0/2268739791 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4c011990 con 0x7f3f5815a5c0 2024-04-02T07:51:04.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.580+0000 7f3f57400700 1 -- 172.21.15.89:0/2268739791 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c011d20 con 0x7f3f5815a5c0 2024-04-02T07:51:04.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.580+0000 7f3f5f400700 1 -- 172.21.15.89:0/2268739791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5815a5c0 msgr2=0x7f3f580709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:04.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.580+0000 7f3f5f400700 1 --2- 172.21.15.89:0/2268739791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5815a5c0 0x7f3f580709c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f3f5806b630 tx=0x7f3f4c009bd0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.581+0000 7f3f5f400700 1 -- 172.21.15.89:0/2268739791 shutdown_connections 2024-04-02T07:51:04.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.581+0000 7f3f5f400700 1 --2- 172.21.15.89:0/2268739791 >> v2:172.21.15.113:3300/0 conn(0x7f3f580ce840 0x7f3f580cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.581+0000 7f3f5f400700 1 --2- 172.21.15.89:0/2268739791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5815a5c0 0x7f3f580709c0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.581+0000 7f3f5f400700 1 -- 172.21.15.89:0/2268739791 >> 172.21.15.89:0/2268739791 conn(0x7f3f580b0d70 msgr2=0x7f3f580b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:04.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.581+0000 7f3f5f400700 1 -- 172.21.15.89:0/2268739791 shutdown_connections 2024-04-02T07:51:04.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.581+0000 7f3f5f400700 1 -- 172.21.15.89:0/2268739791 wait complete. 2024-04-02T07:51:04.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.583+0000 7f3f5f400700 1 Processor -- start 2024-04-02T07:51:04.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.584+0000 7f3f5f400700 1 -- start start 2024-04-02T07:51:04.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.585+0000 7f3f5f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f580ce840 0x7f3f5815ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:04.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.585+0000 7f3f5f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5815a5c0 0x7f3f580cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:04.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.585+0000 7f3f5f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5815dfe0 0x7f3f580cd560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:04.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.585+0000 7f3f5f400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5815b5d0 con 0x7f3f5815a5c0 2024-04-02T07:51:04.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.585+0000 7f3f5f400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f581f6890 con 0x7f3f580ce840 2024-04-02T07:51:04.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.585+0000 7f3f5f400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f581f6b70 con 0x7f3f5815dfe0 2024-04-02T07:51:04.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.586+0000 7f3f57e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5815a5c0 0x7f3f580cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:04.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.586+0000 7f3f57e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5815a5c0 0x7f3f580cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54910/0 (socket says 172.21.15.89:54910) 2024-04-02T07:51:04.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.586+0000 7f3f5ca00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f580ce840 0x7f3f5815ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:04.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.586+0000 7f3f57e00700 1 -- 172.21.15.89:0/524064829 learned_addr learned my addr 172.21.15.89:0/524064829 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:04.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.586+0000 7f3f5d400700 1 --2- 172.21.15.89:0/524064829 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5815dfe0 0x7f3f580cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:04.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.586+0000 7f3f57e00700 1 -- 172.21.15.89:0/524064829 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5815dfe0 msgr2=0x7f3f580cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:04.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.586+0000 7f3f57e00700 1 --2- 172.21.15.89:0/524064829 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5815dfe0 0x7f3f580cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.586+0000 7f3f57e00700 1 -- 172.21.15.89:0/524064829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f580ce840 msgr2=0x7f3f5815ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:04.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.586+0000 7f3f57e00700 1 --2- 172.21.15.89:0/524064829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f580ce840 0x7f3f5815ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.586+0000 7f3f57e00700 1 -- 172.21.15.89:0/524064829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c009550 con 0x7f3f5815a5c0 2024-04-02T07:51:04.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.586+0000 7f3f57e00700 1 --2- 172.21.15.89:0/524064829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5815a5c0 0x7f3f580cd020 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3f4400c550 tx=0x7f3f4400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:04.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.586+0000 7f3f54c00700 1 -- 172.21.15.89:0/524064829 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f440054e0 con 0x7f3f5815a5c0 2024-04-02T07:51:04.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.587+0000 7f3f54c00700 1 -- 172.21.15.89:0/524064829 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f44003680 con 0x7f3f5815a5c0 2024-04-02T07:51:04.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.587+0000 7f3f5f400700 1 -- 172.21.15.89:0/524064829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f581f6e50 con 0x7f3f5815a5c0 2024-04-02T07:51:04.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.587+0000 7f3f54c00700 1 -- 172.21.15.89:0/524064829 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44003a30 con 0x7f3f5815a5c0 2024-04-02T07:51:04.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.587+0000 7f3f5f400700 1 -- 172.21.15.89:0/524064829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f581f7650 con 0x7f3f5815a5c0 2024-04-02T07:51:04.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.588+0000 7f3f5f400700 1 -- 172.21.15.89:0/524064829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f581f7370 con 0x7f3f5815a5c0 2024-04-02T07:51:04.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.589+0000 7f3f54c00700 1 -- 172.21.15.89:0/524064829 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f3f44003b90 con 0x7f3f5815a5c0 2024-04-02T07:51:04.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.590+0000 7f3f54c00700 1 --2- 172.21.15.89:0/524064829 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f3806adb0 0x7f3f3806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:04.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.591+0000 7f3f5ca00700 1 -- 172.21.15.89:0/524064829 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f3806adb0 msgr2=0x7f3f3806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:04.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.591+0000 7f3f54c00700 1 -- 172.21.15.89:0/524064829 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f3f44088b40 con 0x7f3f5815a5c0 2024-04-02T07:51:04.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.591+0000 7f3f5ca00700 1 --2- 172.21.15.89:0/524064829 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f3806adb0 0x7f3f3806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:04.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.594+0000 7f3f54c00700 1 -- 172.21.15.89:0/524064829 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3f581f7370 con 0x7f3f5815a5c0 2024-04-02T07:51:04.615 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:04.614+0000 7fde23f68240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:51:04.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.792+0000 7f3f5ca00700 1 -- 172.21.15.89:0/524064829 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f3806adb0 msgr2=0x7f3f3806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:04.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.792+0000 7f3f5ca00700 1 --2- 172.21.15.89:0/524064829 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f3806adb0 0x7f3f3806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:04.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.803+0000 7f3f5f400700 1 -- 172.21.15.89:0/524064829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3f581f7370 con 0x7f3f5815a5c0 2024-04-02T07:51:04.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.804+0000 7f3f54c00700 1 -- 172.21.15.89:0/524064829 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3f581f7370 con 0x7f3f5815a5c0 2024-04-02T07:51:04.806 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:51:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.806+0000 7f3f5f400700 1 -- 172.21.15.89:0/524064829 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f3806adb0 msgr2=0x7f3f3806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.806+0000 7f3f5f400700 1 --2- 172.21.15.89:0/524064829 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f3806adb0 0x7f3f3806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.806+0000 7f3f5f400700 1 -- 172.21.15.89:0/524064829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5815a5c0 msgr2=0x7f3f580cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.806+0000 7f3f5f400700 1 --2- 172.21.15.89:0/524064829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5815a5c0 0x7f3f580cd020 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3f4400c550 tx=0x7f3f4400c860 comp rx=0 tx=0).stop 2024-04-02T07:51:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.807+0000 7f3f5f400700 1 -- 172.21.15.89:0/524064829 shutdown_connections 2024-04-02T07:51:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.807+0000 7f3f5f400700 1 --2- 172.21.15.89:0/524064829 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f3806adb0 0x7f3f3806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.807+0000 7f3f5f400700 1 --2- 172.21.15.89:0/524064829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f580ce840 0x7f3f5815ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.807+0000 7f3f5f400700 1 --2- 172.21.15.89:0/524064829 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5815dfe0 0x7f3f580cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.807+0000 7f3f5f400700 1 --2- 172.21.15.89:0/524064829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5815a5c0 0x7f3f580cd020 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.807+0000 7f3f5f400700 1 -- 172.21.15.89:0/524064829 >> 172.21.15.89:0/524064829 conn(0x7f3f580b0d70 msgr2=0x7f3f5815e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.807+0000 7f3f5f400700 1 -- 172.21.15.89:0/524064829 shutdown_connections 2024-04-02T07:51:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:04.807+0000 7f3f5f400700 1 -- 172.21.15.89:0/524064829 wait complete. 2024-04-02T07:51:04.827 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:51:04.963 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:04.961+0000 7ff1aa269240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:51:05.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.026+0000 7f21be600700 1 Processor -- start 2024-04-02T07:51:05.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.026+0000 7f21be600700 1 -- start start 2024-04-02T07:51:05.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.027+0000 7f21be600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f21b80af600 0x7f21b806a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:05.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.027+0000 7f21be600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f21b80afa90 0x7f21b806acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:05.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.027+0000 7f21be600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b80aeb60 con 0x7f21b8145050 2024-04-02T07:51:05.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.027+0000 7f21be600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b80b5190 con 0x7f21b80af600 2024-04-02T07:51:05.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.027+0000 7f21be600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b80aee00 con 0x7f21b80afa90 2024-04-02T07:51:05.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.028+0000 7f21b6a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f21b80af600 0x7f21b806a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:05.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.028+0000 7f21b6a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f21b80af600 0x7f21b806a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54926/0 (socket says 172.21.15.89:54926) 2024-04-02T07:51:05.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.028+0000 7f21b6a00700 1 -- 172.21.15.89:0/1477927953 learned_addr learned my addr 172.21.15.89:0/1477927953 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:05.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.028+0000 7f21b7e00700 1 --2- 172.21.15.89:0/1477927953 >> v2:172.21.15.113:3300/0 conn(0x7f21b80afa90 0x7f21b806acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:05.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.028+0000 7f21b6a00700 1 -- 172.21.15.89:0/1477927953 >> v2:172.21.15.113:3300/0 conn(0x7f21b80afa90 msgr2=0x7f21b806acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:05.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.028+0000 7f21b6a00700 1 --2- 172.21.15.89:0/1477927953 >> v2:172.21.15.113:3300/0 conn(0x7f21b80afa90 0x7f21b806acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.028+0000 7f21b6a00700 1 -- 172.21.15.89:0/1477927953 >> v1:172.21.15.113:6789/0 conn(0x7f21b8145050 legacy=0x7f21b80d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:05.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.028+0000 7f21b6a00700 1 -- 172.21.15.89:0/1477927953 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b80af0e0 con 0x7f21b80af600 2024-04-02T07:51:05.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.028+0000 7f21b6a00700 1 --2- 172.21.15.89:0/1477927953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21b80af600 0x7f21b806a790 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f21b8072840 tx=0x7f21a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d11e8085125291 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:05.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.029+0000 7f21b6000700 1 -- 172.21.15.89:0/1477927953 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a4011830 con 0x7f21b80af600 2024-04-02T07:51:05.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.029+0000 7f21b6000700 1 -- 172.21.15.89:0/1477927953 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a4011990 con 0x7f21b80af600 2024-04-02T07:51:05.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.029+0000 7f21b6000700 1 -- 172.21.15.89:0/1477927953 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a4011d20 con 0x7f21b80af600 2024-04-02T07:51:05.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.029+0000 7f21be600700 1 -- 172.21.15.89:0/1477927953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21b80af600 msgr2=0x7f21b806a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:05.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.029+0000 7f21be600700 1 --2- 172.21.15.89:0/1477927953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21b80af600 0x7f21b806a790 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f21b8072840 tx=0x7f21a4009bd0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.030+0000 7f21be600700 1 -- 172.21.15.89:0/1477927953 shutdown_connections 2024-04-02T07:51:05.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.030+0000 7f21be600700 1 --2- 172.21.15.89:0/1477927953 >> v2:172.21.15.113:3300/0 conn(0x7f21b80afa90 0x7f21b806acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.030+0000 7f21be600700 1 --2- 172.21.15.89:0/1477927953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21b80af600 0x7f21b806a790 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.030+0000 7f21be600700 1 -- 172.21.15.89:0/1477927953 >> 172.21.15.89:0/1477927953 conn(0x7f21b80b5720 msgr2=0x7f21b80b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:05.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.030+0000 7f21be600700 1 -- 172.21.15.89:0/1477927953 shutdown_connections 2024-04-02T07:51:05.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.030+0000 7f21be600700 1 -- 172.21.15.89:0/1477927953 wait complete. 2024-04-02T07:51:05.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.031+0000 7f21be600700 1 Processor -- start 2024-04-02T07:51:05.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.032+0000 7f21be600700 1 -- start start 2024-04-02T07:51:05.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21be600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21b80af600 0x7f21b813e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:05.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21be600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21b80afa90 0x7f21b80b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:05.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21be600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b8145050 0x7f21b813edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:05.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21be600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80cfb90 con 0x7f21b80afa90 2024-04-02T07:51:05.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21be600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80b0fd0 con 0x7f21b8145050 2024-04-02T07:51:05.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21be600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80b12b0 con 0x7f21b80af600 2024-04-02T07:51:05.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21b6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21b80afa90 0x7f21b80b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:05.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21b7e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b8145050 0x7f21b813edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:05.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21b6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21b80afa90 0x7f21b80b2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54942/0 (socket says 172.21.15.89:54942) 2024-04-02T07:51:05.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21b6a00700 1 -- 172.21.15.89:0/1136311809 learned_addr learned my addr 172.21.15.89:0/1136311809 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:05.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21b7400700 1 --2- 172.21.15.89:0/1136311809 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21b80af600 0x7f21b813e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:05.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21b6a00700 1 -- 172.21.15.89:0/1136311809 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21b80af600 msgr2=0x7f21b813e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:05.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21b6a00700 1 --2- 172.21.15.89:0/1136311809 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21b80af600 0x7f21b813e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21b6a00700 1 -- 172.21.15.89:0/1136311809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b8145050 msgr2=0x7f21b813edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:05.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21b6a00700 1 --2- 172.21.15.89:0/1136311809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b8145050 0x7f21b813edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21b6a00700 1 -- 172.21.15.89:0/1136311809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a4009550 con 0x7f21b80afa90 2024-04-02T07:51:05.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.033+0000 7f21b7e00700 1 --2- 172.21.15.89:0/1136311809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b8145050 0x7f21b813edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:05.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.034+0000 7f21b6a00700 1 --2- 172.21.15.89:0/1136311809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21b80afa90 0x7f21b80b2070 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f21a4000c00 tx=0x7f21a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:05.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.034+0000 7f21af400700 1 -- 172.21.15.89:0/1136311809 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a4004590 con 0x7f21b80afa90 2024-04-02T07:51:05.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.034+0000 7f21af400700 1 -- 172.21.15.89:0/1136311809 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a40036a0 con 0x7f21b80afa90 2024-04-02T07:51:05.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.034+0000 7f21be600700 1 -- 172.21.15.89:0/1136311809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b80b1590 con 0x7f21b80afa90 2024-04-02T07:51:05.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.034+0000 7f21af400700 1 -- 172.21.15.89:0/1136311809 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a4003a50 con 0x7f21b80afa90 2024-04-02T07:51:05.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.034+0000 7f21be600700 1 -- 172.21.15.89:0/1136311809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b80b1b50 con 0x7f21b80afa90 2024-04-02T07:51:05.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.036+0000 7f21be600700 1 -- 172.21.15.89:0/1136311809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b8067720 con 0x7f21b80afa90 2024-04-02T07:51:05.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.036+0000 7f21af400700 1 -- 172.21.15.89:0/1136311809 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f21a4021070 con 0x7f21b80afa90 2024-04-02T07:51:05.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.038+0000 7f21af400700 1 --2- 172.21.15.89:0/1136311809 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f219806adb0 0x7f219806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:05.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.038+0000 7f21b7400700 1 -- 172.21.15.89:0/1136311809 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f219806adb0 msgr2=0x7f219806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:05.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.038+0000 7f21af400700 1 -- 172.21.15.89:0/1136311809 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f21a408ad70 con 0x7f21b80afa90 2024-04-02T07:51:05.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.038+0000 7f21b7400700 1 --2- 172.21.15.89:0/1136311809 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f219806adb0 0x7f219806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:05.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.042+0000 7f21af400700 1 -- 172.21.15.89:0/1136311809 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f21a4055b80 con 0x7f21b80afa90 2024-04-02T07:51:05.095 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:05.093+0000 7fde23f68240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:51:05.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.239+0000 7f21b7400700 1 -- 172.21.15.89:0/1136311809 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f219806adb0 msgr2=0x7f219806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:05.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.239+0000 7f21b7400700 1 --2- 172.21.15.89:0/1136311809 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f219806adb0 0x7f219806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:05.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.253+0000 7f21be600700 1 -- 172.21.15.89:0/1136311809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f21b815c780 con 0x7f21b80afa90 2024-04-02T07:51:05.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.253+0000 7f21af400700 1 -- 172.21.15.89:0/1136311809 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f21a4015020 con 0x7f21b80afa90 2024-04-02T07:51:05.255 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:51:05.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.257+0000 7f21be600700 1 -- 172.21.15.89:0/1136311809 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f219806adb0 msgr2=0x7f219806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:05.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.257+0000 7f21be600700 1 --2- 172.21.15.89:0/1136311809 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f219806adb0 0x7f219806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.257+0000 7f21be600700 1 -- 172.21.15.89:0/1136311809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21b80afa90 msgr2=0x7f21b80b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:05.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.257+0000 7f21be600700 1 --2- 172.21.15.89:0/1136311809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21b80afa90 0x7f21b80b2070 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f21a4000c00 tx=0x7f21a4004750 comp rx=0 tx=0).stop 2024-04-02T07:51:05.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.257+0000 7f21be600700 1 -- 172.21.15.89:0/1136311809 shutdown_connections 2024-04-02T07:51:05.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.257+0000 7f21be600700 1 --2- 172.21.15.89:0/1136311809 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f219806adb0 0x7f219806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.257+0000 7f21be600700 1 --2- 172.21.15.89:0/1136311809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b8145050 0x7f21b813edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.257+0000 7f21be600700 1 --2- 172.21.15.89:0/1136311809 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21b80af600 0x7f21b813e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.257+0000 7f21be600700 1 --2- 172.21.15.89:0/1136311809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21b80afa90 0x7f21b80b2070 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.257+0000 7f21be600700 1 -- 172.21.15.89:0/1136311809 >> 172.21.15.89:0/1136311809 conn(0x7f21b80b5720 msgr2=0x7f21b80d0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:05.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.257+0000 7f21be600700 1 -- 172.21.15.89:0/1136311809 shutdown_connections 2024-04-02T07:51:05.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.257+0000 7f21be600700 1 -- 172.21.15.89:0/1136311809 wait complete. 2024-04-02T07:51:05.285 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T07:51:05.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.478+0000 7feab2c00700 1 Processor -- start 2024-04-02T07:51:05.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.478+0000 7feab2c00700 1 -- start start 2024-04-02T07:51:05.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.479+0000 7feab2c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7feaac15d9d0 0x7feaac0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:05.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.480+0000 7feab2c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7feaac0d2540 0x7feaac0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.480+0000 7feab2c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaac161420 con 0x7feaac0d16a0 2024-04-02T07:51:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.480+0000 7feab2c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaac15edc0 con 0x7feaac0d2540 2024-04-02T07:51:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.480+0000 7feab2c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaac161700 con 0x7feaac15d9d0 2024-04-02T07:51:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.480+0000 7feab0c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7feaac0d2540 0x7feaac0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.480+0000 7feab0c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7feaac0d2540 0x7feaac0ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54952/0 (socket says 172.21.15.89:54952) 2024-04-02T07:51:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.480+0000 7feab0c00700 1 -- 172.21.15.89:0/2324068216 learned_addr learned my addr 172.21.15.89:0/2324068216 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.480+0000 7feaab400700 1 --1- 172.21.15.89:0/2324068216 >> v1:172.21.15.89:6789/0 conn(0x7feaac0d16a0 0x7feaac0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40094/0 (socket says 172.21.15.89:40094) 2024-04-02T07:51:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.480+0000 7feaabe00700 1 --2- 172.21.15.89:0/2324068216 >> v2:172.21.15.113:3300/0 conn(0x7feaac15d9d0 0x7feaac0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.480+0000 7feab0c00700 1 -- 172.21.15.89:0/2324068216 >> v2:172.21.15.113:3300/0 conn(0x7feaac15d9d0 msgr2=0x7feaac0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.480+0000 7feab0c00700 1 --2- 172.21.15.89:0/2324068216 >> v2:172.21.15.113:3300/0 conn(0x7feaac15d9d0 0x7feaac0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.480+0000 7feab0c00700 1 -- 172.21.15.89:0/2324068216 >> v1:172.21.15.89:6789/0 conn(0x7feaac0d16a0 legacy=0x7feaac0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.480+0000 7feab0c00700 1 -- 172.21.15.89:0/2324068216 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac1619e0 con 0x7feaac0d2540 2024-04-02T07:51:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.480+0000 7feaaaa00700 1 -- 172.21.15.89:0/2324068216 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889239877 0 0) 0x7feaac161420 con 0x7feaac0d16a0 2024-04-02T07:51:05.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.481+0000 7feab0c00700 1 --2- 172.21.15.89:0/2324068216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaac0d2540 0x7feaac0ce360 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7feaac0d0ff0 tx=0x7feaa400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=127be2a23cc8ff16 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:05.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.481+0000 7feaaaa00700 1 -- 172.21.15.89:0/2324068216 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa400e070 con 0x7feaac0d2540 2024-04-02T07:51:05.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.481+0000 7feaaaa00700 1 -- 172.21.15.89:0/2324068216 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa400b910 con 0x7feaac0d2540 2024-04-02T07:51:05.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.481+0000 7feaaaa00700 1 -- 172.21.15.89:0/2324068216 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa400bcc0 con 0x7feaac0d2540 2024-04-02T07:51:05.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.481+0000 7feab2c00700 1 -- 172.21.15.89:0/2324068216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaac0d2540 msgr2=0x7feaac0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.481+0000 7feab2c00700 1 --2- 172.21.15.89:0/2324068216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaac0d2540 0x7feaac0ce360 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7feaac0d0ff0 tx=0x7feaa400b330 comp rx=0 tx=0).stop 2024-04-02T07:51:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.482+0000 7feab2c00700 1 -- 172.21.15.89:0/2324068216 shutdown_connections 2024-04-02T07:51:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.482+0000 7feab2c00700 1 --2- 172.21.15.89:0/2324068216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaac0d2540 0x7feaac0ce360 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.482+0000 7feab2c00700 1 --2- 172.21.15.89:0/2324068216 >> v2:172.21.15.113:3300/0 conn(0x7feaac15d9d0 0x7feaac0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.482+0000 7feab2c00700 1 -- 172.21.15.89:0/2324068216 >> 172.21.15.89:0/2324068216 conn(0x7feaac08aff0 msgr2=0x7feaac0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.482+0000 7feab2c00700 1 -- 172.21.15.89:0/2324068216 shutdown_connections 2024-04-02T07:51:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.482+0000 7feab2c00700 1 -- 172.21.15.89:0/2324068216 wait complete. 2024-04-02T07:51:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.484+0000 7feab2c00700 1 Processor -- start 2024-04-02T07:51:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.484+0000 7feab2c00700 1 -- start start 2024-04-02T07:51:05.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.485+0000 7feab2c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feaac0d16a0 0x7feaac1f7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:05.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.485+0000 7feab2c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feaac0d2540 0x7feaac0b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.485+0000 7feab2c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaac15d9d0 0x7feaac158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.485+0000 7feab2c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaac1f3210 con 0x7feaac15d9d0 2024-04-02T07:51:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.485+0000 7feab2c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaac0b0f00 con 0x7feaac0d16a0 2024-04-02T07:51:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.485+0000 7feab2c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaac0b6d50 con 0x7feaac0d2540 2024-04-02T07:51:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.486+0000 7feab0c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaac15d9d0 0x7feaac158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.486+0000 7feaabe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feaac0d16a0 0x7feaac1f7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.486+0000 7feaabe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feaac0d16a0 0x7feaac1f7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:53046/0 (socket says 172.21.15.89:53046) 2024-04-02T07:51:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.486+0000 7feab0c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaac15d9d0 0x7feaac158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54966/0 (socket says 172.21.15.89:54966) 2024-04-02T07:51:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.486+0000 7feaabe00700 1 -- 172.21.15.89:0/1760584932 learned_addr learned my addr 172.21.15.89:0/1760584932 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.486+0000 7feaab400700 1 --2- 172.21.15.89:0/1760584932 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feaac0d2540 0x7feaac0b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.486+0000 7feaabe00700 1 -- 172.21.15.89:0/1760584932 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feaac0d2540 msgr2=0x7feaac0b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.486+0000 7feaabe00700 1 --2- 172.21.15.89:0/1760584932 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feaac0d2540 0x7feaac0b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.486+0000 7feaabe00700 1 -- 172.21.15.89:0/1760584932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaac15d9d0 msgr2=0x7feaac158aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.486+0000 7feaabe00700 1 --2- 172.21.15.89:0/1760584932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaac15d9d0 0x7feaac158aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.486+0000 7feaabe00700 1 -- 172.21.15.89:0/1760584932 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa4009b30 con 0x7feaac0d16a0 2024-04-02T07:51:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.486+0000 7feab0c00700 1 --2- 172.21.15.89:0/1760584932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaac15d9d0 0x7feaac158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.487+0000 7feaabe00700 1 --2- 172.21.15.89:0/1760584932 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feaac0d16a0 0x7feaac1f7240 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00e870 tx=0x7fea9c00eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.487+0000 7feaa3e00700 1 -- 172.21.15.89:0/1760584932 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9c009590 con 0x7feaac0d16a0 2024-04-02T07:51:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.487+0000 7feaa3e00700 1 -- 172.21.15.89:0/1760584932 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea9c013680 con 0x7feaac0d16a0 2024-04-02T07:51:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.487+0000 7feab2c00700 1 -- 172.21.15.89:0/1760584932 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaac158fe0 con 0x7feaac0d16a0 2024-04-02T07:51:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.487+0000 7feaa3e00700 1 -- 172.21.15.89:0/1760584932 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9c0139f0 con 0x7feaac0d16a0 2024-04-02T07:51:05.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.487+0000 7feab2c00700 1 -- 172.21.15.89:0/1760584932 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaac0b7290 con 0x7feaac0d16a0 2024-04-02T07:51:05.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.488+0000 7feab2c00700 1 -- 172.21.15.89:0/1760584932 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaac0d1ca0 con 0x7feaac0d16a0 2024-04-02T07:51:05.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.489+0000 7feaa3e00700 1 -- 172.21.15.89:0/1760584932 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fea9c013b50 con 0x7feaac0d16a0 2024-04-02T07:51:05.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.490+0000 7feaa3e00700 1 --2- 172.21.15.89:0/1760584932 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fea8c06adb0 0x7fea8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:05.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.490+0000 7feaa3e00700 1 -- 172.21.15.89:0/1760584932 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fea9c089ac0 con 0x7feaac0d16a0 2024-04-02T07:51:05.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.490+0000 7feaab400700 1 -- 172.21.15.89:0/1760584932 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fea8c06adb0 msgr2=0x7fea8c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:05.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.490+0000 7feaab400700 1 --2- 172.21.15.89:0/1760584932 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fea8c06adb0 0x7fea8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:05.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.497+0000 7feaa3e00700 1 -- 172.21.15.89:0/1760584932 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fea9c0547a0 con 0x7feaac0d16a0 2024-04-02T07:51:05.672 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:05.670+0000 7ff1aa269240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:51:05.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.691+0000 7feaab400700 1 -- 172.21.15.89:0/1760584932 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fea8c06adb0 msgr2=0x7fea8c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:05.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.691+0000 7feaab400700 1 --2- 172.21.15.89:0/1760584932 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fea8c06adb0 0x7fea8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:05.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.724+0000 7feab2c00700 1 -- 172.21.15.89:0/1760584932 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7feaac067720 con 0x7feaac0d16a0 2024-04-02T07:51:05.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.724+0000 7feaa3e00700 1 -- 172.21.15.89:0/1760584932 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v69) v1 ==== 80+0+458 (secure 0 0 0) 0x7fea9c0211e0 con 0x7feaac0d16a0 2024-04-02T07:51:05.726 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/379543030 2024-04-03T07:49:23.665126+0000 2024-04-02T07:51:05.726 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3951589296 2024-04-03T07:49:23.665126+0000 2024-04-02T07:51:05.726 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3713107589 2024-04-03T07:49:23.665126+0000 2024-04-02T07:51:05.726 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/43357 2024-04-03T07:49:23.665126+0000 2024-04-02T07:51:05.727 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/43357 2024-04-03T07:49:23.665126+0000 2024-04-02T07:51:05.727 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/552943692 2024-04-03T07:49:23.665126+0000 2024-04-02T07:51:05.727 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2589801503 2024-04-03T07:49:23.665126+0000 2024-04-02T07:51:05.727 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3028439559 2024-04-03T07:49:23.665126+0000 2024-04-02T07:51:05.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.727+0000 7feab2c00700 1 -- 172.21.15.89:0/1760584932 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fea8c06adb0 msgr2=0x7fea8c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:05.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.727+0000 7feab2c00700 1 --2- 172.21.15.89:0/1760584932 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fea8c06adb0 0x7fea8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.727+0000 7feab2c00700 1 -- 172.21.15.89:0/1760584932 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feaac0d16a0 msgr2=0x7feaac1f7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:05.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.727+0000 7feab2c00700 1 --2- 172.21.15.89:0/1760584932 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feaac0d16a0 0x7feaac1f7240 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00e870 tx=0x7fea9c00eb80 comp rx=0 tx=0).stop 2024-04-02T07:51:05.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.727+0000 7feab2c00700 1 -- 172.21.15.89:0/1760584932 shutdown_connections 2024-04-02T07:51:05.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.727+0000 7feab2c00700 1 --2- 172.21.15.89:0/1760584932 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fea8c06adb0 0x7fea8c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.727+0000 7feab2c00700 1 --2- 172.21.15.89:0/1760584932 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feaac0d16a0 0x7feaac1f7240 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.727+0000 7feab2c00700 1 --2- 172.21.15.89:0/1760584932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feaac15d9d0 0x7feaac158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.727+0000 7feab2c00700 1 --2- 172.21.15.89:0/1760584932 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feaac0d2540 0x7feaac0b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:05.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.727+0000 7feab2c00700 1 -- 172.21.15.89:0/1760584932 >> 172.21.15.89:0/1760584932 conn(0x7feaac08aff0 msgr2=0x7feaac06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:05.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.727+0000 7feab2c00700 1 -- 172.21.15.89:0/1760584932 shutdown_connections 2024-04-02T07:51:05.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:05.727+0000 7feab2c00700 1 -- 172.21.15.89:0/1760584932 wait complete. 2024-04-02T07:51:05.730 INFO:teuthology.orchestra.run.smithi089.stderr:listed 8 entries 2024-04-02T07:51:05.747 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T07:51:05.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:05.747 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T07:51:05.763 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:05.761+0000 7ff1aa269240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:51:05.825 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:51:05.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:05.825 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T07:51:05.855 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:05.853+0000 7fde23f68240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:51:05.871 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:51:05.871 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:05.915 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:51:05.915 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:51:05.915 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:05.926 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:05.925+0000 7ff1aa269240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:51:05.954 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:05.950+0000 7fde23f68240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:51:05.960 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:51:05.960 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:51:05.960 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:06.005 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:51:06.006 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:51:06.122 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:06.120+0000 7fde23f68240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:51:06.144 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:06.142+0000 7ff1aa269240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:51:06.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.201+0000 7fad86a00700 1 Processor -- start 2024-04-02T07:51:06.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.201+0000 7fad86a00700 1 -- start start 2024-04-02T07:51:06.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.203+0000 7fad86a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad8015d710 0x7fad800d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:06.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.203+0000 7fad86a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8006bb40 con 0x7fad8006c650 2024-04-02T07:51:06.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.203+0000 7fad86a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad80099b00 con 0x7fad8009aba0 2024-04-02T07:51:06.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.203+0000 7fad86a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad80099de0 con 0x7fad8015d710 2024-04-02T07:51:06.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.204+0000 7fad84a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fad8006c650 0x7fad800ccbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40102/0 (socket says 172.21.15.89:40102) 2024-04-02T07:51:06.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.204+0000 7fad84a00700 1 -- 172.21.15.89:0/1186498761 learned_addr learned my addr 172.21.15.89:0/1186498761 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:06.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.204+0000 7fad7fe00700 1 --2- 172.21.15.89:0/1186498761 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad8015d710 0x7fad800d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.204+0000 7fad7ea00700 1 -- 172.21.15.89:0/1186498761 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060279749 0 0) 0x7fad8006bb40 con 0x7fad8006c650 2024-04-02T07:51:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.204+0000 7fad7ea00700 1 -- 172.21.15.89:0/1186498761 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad640035b0 con 0x7fad8006c650 2024-04-02T07:51:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.204+0000 7fad7fe00700 1 -- 172.21.15.89:0/1186498761 >> v1:172.21.15.113:6789/0 conn(0x7fad8009aba0 legacy=0x7fad8006bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.204+0000 7fad7ea00700 1 -- 172.21.15.89:0/1186498761 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334150087 0 0) 0x7fad80099b00 con 0x7fad8009aba0 2024-04-02T07:51:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.204+0000 7fad7fe00700 1 -- 172.21.15.89:0/1186498761 >> v1:172.21.15.89:6789/0 conn(0x7fad8006c650 legacy=0x7fad800ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.204+0000 7fad7fe00700 1 -- 172.21.15.89:0/1186498761 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8009a0c0 con 0x7fad8015d710 2024-04-02T07:51:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.205+0000 7fad7fe00700 1 --2- 172.21.15.89:0/1186498761 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad8015d710 0x7fad800d0130 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fad70009870 tx=0x7fad70009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8205eaac9cce6c47 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.205+0000 7fad7ea00700 1 -- 172.21.15.89:0/1186498761 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad70011830 con 0x7fad8015d710 2024-04-02T07:51:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.205+0000 7fad7ea00700 1 -- 172.21.15.89:0/1186498761 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad70011990 con 0x7fad8015d710 2024-04-02T07:51:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.205+0000 7fad7ea00700 1 -- 172.21.15.89:0/1186498761 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad70011d00 con 0x7fad8015d710 2024-04-02T07:51:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.206+0000 7fad86a00700 1 -- 172.21.15.89:0/1186498761 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad8015d710 msgr2=0x7fad800d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.206+0000 7fad86a00700 1 --2- 172.21.15.89:0/1186498761 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad8015d710 0x7fad800d0130 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fad70009870 tx=0x7fad70009b80 comp rx=0 tx=0).stop 2024-04-02T07:51:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.206+0000 7fad86a00700 1 -- 172.21.15.89:0/1186498761 shutdown_connections 2024-04-02T07:51:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.206+0000 7fad86a00700 1 --2- 172.21.15.89:0/1186498761 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad8015d710 0x7fad800d0130 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.206+0000 7fad86a00700 1 -- 172.21.15.89:0/1186498761 >> 172.21.15.89:0/1186498761 conn(0x7fad800d13e0 msgr2=0x7fad800d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:06.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.206+0000 7fad86a00700 1 -- 172.21.15.89:0/1186498761 shutdown_connections 2024-04-02T07:51:06.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.206+0000 7fad86a00700 1 -- 172.21.15.89:0/1186498761 wait complete. 2024-04-02T07:51:06.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.208+0000 7fad86a00700 1 Processor -- start 2024-04-02T07:51:06.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.208+0000 7fad86a00700 1 -- start start 2024-04-02T07:51:06.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.209+0000 7fad86a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad8006c650 0x7fad800cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:06.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.209+0000 7fad86a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad8009aba0 0x7fad800c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:06.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.209+0000 7fad86a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad8015d710 0x7fad800c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:06.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.209+0000 7fad86a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad80148310 con 0x7fad8006c650 2024-04-02T07:51:06.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.209+0000 7fad86a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad801f3100 con 0x7fad8009aba0 2024-04-02T07:51:06.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.209+0000 7fad86a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad800cb790 con 0x7fad8015d710 2024-04-02T07:51:06.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.209+0000 7fad7fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad8006c650 0x7fad800cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:06.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.210+0000 7fad7fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad8006c650 0x7fad800cb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54982/0 (socket says 172.21.15.89:54982) 2024-04-02T07:51:06.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.210+0000 7fad7f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad8009aba0 0x7fad800c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:06.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.210+0000 7fad7fe00700 1 -- 172.21.15.89:0/2840254920 learned_addr learned my addr 172.21.15.89:0/2840254920 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:06.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.210+0000 7fad84a00700 1 --2- 172.21.15.89:0/2840254920 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad8015d710 0x7fad800c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:06.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.210+0000 7fad7fe00700 1 -- 172.21.15.89:0/2840254920 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad8015d710 msgr2=0x7fad800c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:06.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.210+0000 7fad7fe00700 1 --2- 172.21.15.89:0/2840254920 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad8015d710 0x7fad800c5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.210+0000 7fad7fe00700 1 -- 172.21.15.89:0/2840254920 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad8009aba0 msgr2=0x7fad800c72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:06.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.210+0000 7fad7fe00700 1 --2- 172.21.15.89:0/2840254920 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad8009aba0 0x7fad800c72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.210+0000 7fad7fe00700 1 -- 172.21.15.89:0/2840254920 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad700094a0 con 0x7fad8006c650 2024-04-02T07:51:06.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.210+0000 7fad7fe00700 1 --2- 172.21.15.89:0/2840254920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad8006c650 0x7fad800cb0a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fad700097c0 tx=0x7fad70004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:06.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.211+0000 7fad77e00700 1 -- 172.21.15.89:0/2840254920 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad70004550 con 0x7fad8006c650 2024-04-02T07:51:06.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.211+0000 7fad77e00700 1 -- 172.21.15.89:0/2840254920 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7001f070 con 0x7fad8006c650 2024-04-02T07:51:06.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.211+0000 7fad77e00700 1 -- 172.21.15.89:0/2840254920 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad700038c0 con 0x7fad8006c650 2024-04-02T07:51:06.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.211+0000 7fad86a00700 1 -- 172.21.15.89:0/2840254920 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad801479e0 con 0x7fad8006c650 2024-04-02T07:51:06.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.211+0000 7fad86a00700 1 -- 172.21.15.89:0/2840254920 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad80147fa0 con 0x7fad8006c650 2024-04-02T07:51:06.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.212+0000 7fad86a00700 1 -- 172.21.15.89:0/2840254920 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad80148130 con 0x7fad8006c650 2024-04-02T07:51:06.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.213+0000 7fad77e00700 1 -- 172.21.15.89:0/2840254920 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fad70003a20 con 0x7fad8006c650 2024-04-02T07:51:06.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.213+0000 7fad77e00700 1 --2- 172.21.15.89:0/2840254920 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fad6006adb0 0x7fad6006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:06.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.214+0000 7fad77e00700 1 -- 172.21.15.89:0/2840254920 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fad70021070 con 0x7fad8006c650 2024-04-02T07:51:06.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.214+0000 7fad7f400700 1 -- 172.21.15.89:0/2840254920 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fad6006adb0 msgr2=0x7fad6006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:06.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.214+0000 7fad7f400700 1 --2- 172.21.15.89:0/2840254920 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fad6006adb0 0x7fad6006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:06.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.218+0000 7fad77e00700 1 -- 172.21.15.89:0/2840254920 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fad70055d90 con 0x7fad8006c650 2024-04-02T07:51:06.235 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:06.233+0000 7ff1aa269240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:51:06.333 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:06.330+0000 7ff1aa269240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:51:06.345 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:06.343+0000 7fde23f68240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:51:06.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.415+0000 7fad7f400700 1 -- 172.21.15.89:0/2840254920 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fad6006adb0 msgr2=0x7fad6006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:06.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.415+0000 7fad7f400700 1 --2- 172.21.15.89:0/2840254920 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fad6006adb0 0x7fad6006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:06.430 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:06.427+0000 7ff1aa269240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:51:06.440 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:06.438+0000 7fde23f68240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:51:06.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.472+0000 7fad86a00700 1 -- 172.21.15.89:0/2840254920 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad80067720 con 0x7fad8006c650 2024-04-02T07:51:06.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.472+0000 7fad77e00700 1 -- 172.21.15.89:0/2840254920 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4737 (secure 0 0 0) 0x7fad7002b020 con 0x7fad8006c650 2024-04-02T07:51:06.475 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:06.476 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5710,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3098065397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3098065397},{"type":"v1","addr":"172.21.15.89:6836","nonce":3098065397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5713,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1408411829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1408411829},{"type":"v1","addr":"172.21.15.89:6837","nonce":1408411829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5716,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3266955664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3266955664},{"type":"v1","addr":"172.21.15.113:6834","nonce":3266955664}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:49:07.249043+0000","modified":"2024-04-02T07:49:10.254959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5719":{"gid":5719,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6835/427074923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":427074923},{"type":"v1","addr":"172.21.15.113:6835","nonce":427074923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-02T07:51:06.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.475+0000 7fad86a00700 1 -- 172.21.15.89:0/2840254920 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fad6006adb0 msgr2=0x7fad6006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:06.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.475+0000 7fad86a00700 1 --2- 172.21.15.89:0/2840254920 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fad6006adb0 0x7fad6006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.475+0000 7fad86a00700 1 -- 172.21.15.89:0/2840254920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad8006c650 msgr2=0x7fad800cb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:06.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.475+0000 7fad86a00700 1 --2- 172.21.15.89:0/2840254920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad8006c650 0x7fad800cb0a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fad700097c0 tx=0x7fad70004750 comp rx=0 tx=0).stop 2024-04-02T07:51:06.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.475+0000 7fad86a00700 1 -- 172.21.15.89:0/2840254920 shutdown_connections 2024-04-02T07:51:06.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.475+0000 7fad86a00700 1 --2- 172.21.15.89:0/2840254920 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fad6006adb0 0x7fad6006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.475+0000 7fad86a00700 1 --2- 172.21.15.89:0/2840254920 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad8015d710 0x7fad800c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.475+0000 7fad86a00700 1 --2- 172.21.15.89:0/2840254920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad8006c650 0x7fad800cb0a0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.475+0000 7fad86a00700 1 --2- 172.21.15.89:0/2840254920 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad8009aba0 0x7fad800c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.475+0000 7fad86a00700 1 -- 172.21.15.89:0/2840254920 >> 172.21.15.89:0/2840254920 conn(0x7fad800d13e0 msgr2=0x7fad800b6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:06.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.475+0000 7fad86a00700 1 -- 172.21.15.89:0/2840254920 shutdown_connections 2024-04-02T07:51:06.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.475+0000 7fad86a00700 1 -- 172.21.15.89:0/2840254920 wait complete. 2024-04-02T07:51:06.479 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 24 2024-04-02T07:51:06.496 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:51:06.525 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:06.523+0000 7ff1aa269240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:51:06.544 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:06.542+0000 7fde23f68240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:51:06.648 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:06.646+0000 7fde23f68240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:51:06.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.689+0000 7f7328c00700 1 Processor -- start 2024-04-02T07:51:06.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.690+0000 7f7328c00700 1 -- start start 2024-04-02T07:51:06.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.691+0000 7f7328c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f732415b660 0x7f73240d1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:06.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.691+0000 7f7328c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f73240cfb70 0x7f732409a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:06.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.691+0000 7f7328c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73240cce30 con 0x7f73240c4f30 2024-04-02T07:51:06.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.691+0000 7f7328c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7324099a60 con 0x7f732415b660 2024-04-02T07:51:06.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.691+0000 7f7328c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73240cd110 con 0x7f73240cfb70 2024-04-02T07:51:06.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.691+0000 7f7321e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f732415b660 0x7f73240d1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:06.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.691+0000 7f7321e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f732415b660 0x7f73240d1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54990/0 (socket says 172.21.15.89:54990) 2024-04-02T07:51:06.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.691+0000 7f7321e00700 1 -- 172.21.15.89:0/573241611 learned_addr learned my addr 172.21.15.89:0/573241611 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:06.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.691+0000 7f7322800700 1 --2- 172.21.15.89:0/573241611 >> v2:172.21.15.113:3300/0 conn(0x7f73240cfb70 0x7f732409a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:06.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.691+0000 7f7321e00700 1 -- 172.21.15.89:0/573241611 >> v2:172.21.15.113:3300/0 conn(0x7f73240cfb70 msgr2=0x7f732409a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:06.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.691+0000 7f7321e00700 1 --2- 172.21.15.89:0/573241611 >> v2:172.21.15.113:3300/0 conn(0x7f73240cfb70 0x7f732409a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.691+0000 7f7321e00700 1 -- 172.21.15.89:0/573241611 >> v1:172.21.15.113:6789/0 conn(0x7f73240c4f30 legacy=0x7f73240cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:06.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.691+0000 7f7321e00700 1 -- 172.21.15.89:0/573241611 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7324099c70 con 0x7f732415b660 2024-04-02T07:51:06.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.692+0000 7f7321e00700 1 --2- 172.21.15.89:0/573241611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f732415b660 0x7f73240d1ed0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f73240c4b60 tx=0x7f7314009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09cd04f4bf0ae6d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:06.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.692+0000 7f7321400700 1 -- 172.21.15.89:0/573241611 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731400b870 con 0x7f732415b660 2024-04-02T07:51:06.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.692+0000 7f7321400700 1 -- 172.21.15.89:0/573241611 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731400b9d0 con 0x7f732415b660 2024-04-02T07:51:06.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.692+0000 7f7321400700 1 -- 172.21.15.89:0/573241611 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731400bd60 con 0x7f732415b660 2024-04-02T07:51:06.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.692+0000 7f7328c00700 1 -- 172.21.15.89:0/573241611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f732415b660 msgr2=0x7f73240d1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:06.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.692+0000 7f7328c00700 1 --2- 172.21.15.89:0/573241611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f732415b660 0x7f73240d1ed0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f73240c4b60 tx=0x7f7314009bd0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.692+0000 7f7328c00700 1 -- 172.21.15.89:0/573241611 shutdown_connections 2024-04-02T07:51:06.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.692+0000 7f7328c00700 1 --2- 172.21.15.89:0/573241611 >> v2:172.21.15.113:3300/0 conn(0x7f73240cfb70 0x7f732409a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.692+0000 7f7328c00700 1 --2- 172.21.15.89:0/573241611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f732415b660 0x7f73240d1ed0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.692+0000 7f7328c00700 1 -- 172.21.15.89:0/573241611 >> 172.21.15.89:0/573241611 conn(0x7f73240d13e0 msgr2=0x7f73240d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:06.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.693+0000 7f7328c00700 1 -- 172.21.15.89:0/573241611 shutdown_connections 2024-04-02T07:51:06.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.693+0000 7f7328c00700 1 -- 172.21.15.89:0/573241611 wait complete. 2024-04-02T07:51:06.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.695+0000 7f7328c00700 1 Processor -- start 2024-04-02T07:51:06.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.695+0000 7f7328c00700 1 -- start start 2024-04-02T07:51:06.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.697+0000 7f7328c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73240c4f30 0x7f73240b6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:06.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.697+0000 7f7328c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73240cfb70 0x7f732415d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:06.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.697+0000 7f7328c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f732415b660 0x7f732415d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:06.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.697+0000 7f7328c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73240b5a30 con 0x7f732415b660 2024-04-02T07:51:06.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.697+0000 7f7328c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73240b5ff0 con 0x7f73240c4f30 2024-04-02T07:51:06.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.697+0000 7f7328c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732415ce20 con 0x7f73240cfb70 2024-04-02T07:51:06.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.697+0000 7f7321e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73240c4f30 0x7f73240b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:06.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.697+0000 7f7322800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f732415b660 0x7f732415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:06.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.697+0000 7f7321e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73240c4f30 0x7f73240b6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:53070/0 (socket says 172.21.15.89:53070) 2024-04-02T07:51:06.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.697+0000 7f7321e00700 1 -- 172.21.15.89:0/3429059785 learned_addr learned my addr 172.21.15.89:0/3429059785 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:06.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.697+0000 7f731be00700 1 --2- 172.21.15.89:0/3429059785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73240cfb70 0x7f732415d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:06.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.698+0000 7f7321e00700 1 -- 172.21.15.89:0/3429059785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73240cfb70 msgr2=0x7f732415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:06.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.698+0000 7f7321e00700 1 --2- 172.21.15.89:0/3429059785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73240cfb70 0x7f732415d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.698+0000 7f7321e00700 1 -- 172.21.15.89:0/3429059785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f732415b660 msgr2=0x7f732415d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:06.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.698+0000 7f7321e00700 1 --2- 172.21.15.89:0/3429059785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f732415b660 0x7f732415d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.698+0000 7f7321e00700 1 -- 172.21.15.89:0/3429059785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7314009550 con 0x7f73240c4f30 2024-04-02T07:51:06.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.698+0000 7f7322800700 1 --2- 172.21.15.89:0/3429059785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f732415b660 0x7f732415d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:06.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.698+0000 7f7321e00700 1 --2- 172.21.15.89:0/3429059785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73240c4f30 0x7f73240b6480 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7314000c00 tx=0x7f7314004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:06.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.698+0000 7f731a000700 1 -- 172.21.15.89:0/3429059785 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314004310 con 0x7f73240c4f30 2024-04-02T07:51:06.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.698+0000 7f731a000700 1 -- 172.21.15.89:0/3429059785 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731401e070 con 0x7f73240c4f30 2024-04-02T07:51:06.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.698+0000 7f731a000700 1 -- 172.21.15.89:0/3429059785 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73140036a0 con 0x7f73240c4f30 2024-04-02T07:51:06.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.698+0000 7f7328c00700 1 -- 172.21.15.89:0/3429059785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73240b5c80 con 0x7f73240c4f30 2024-04-02T07:51:06.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.699+0000 7f7328c00700 1 -- 172.21.15.89:0/3429059785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73240b2ae0 con 0x7f73240c4f30 2024-04-02T07:51:06.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.700+0000 7f7328c00700 1 -- 172.21.15.89:0/3429059785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732406e470 con 0x7f73240c4f30 2024-04-02T07:51:06.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.700+0000 7f731a000700 1 -- 172.21.15.89:0/3429059785 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f7314020030 con 0x7f73240c4f30 2024-04-02T07:51:06.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.701+0000 7f731a000700 1 --2- 172.21.15.89:0/3429059785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f730006ade0 0x7f730006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:06.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.701+0000 7f731a000700 1 -- 172.21.15.89:0/3429059785 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f731408a750 con 0x7f73240c4f30 2024-04-02T07:51:06.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.701+0000 7f731be00700 1 -- 172.21.15.89:0/3429059785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f730006ade0 msgr2=0x7f730006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:06.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.701+0000 7f731be00700 1 --2- 172.21.15.89:0/3429059785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f730006ade0 0x7f730006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:06.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.706+0000 7f731a000700 1 -- 172.21.15.89:0/3429059785 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7314055440 con 0x7f73240c4f30 2024-04-02T07:51:06.736 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:06.734+0000 7ff1aa269240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:51:06.753 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:06.752+0000 7fde23f68240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:51:06.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.902+0000 7f731be00700 1 -- 172.21.15.89:0/3429059785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f730006ade0 msgr2=0x7f730006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:06.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.902+0000 7f731be00700 1 --2- 172.21.15.89:0/3429059785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f730006ade0 0x7f730006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:06.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.919+0000 7f7328c00700 1 -- 172.21.15.89:0/3429059785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73240b62d0 con 0x7f73240c4f30 2024-04-02T07:51:06.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.920+0000 7f731a000700 1 -- 172.21.15.89:0/3429059785 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+17349 (secure 0 0 0) 0x7f7314058a60 con 0x7f73240c4f30 2024-04-02T07:51:06.922 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:06.923 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":69,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:50:25.037631+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:49:05.141462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-02T07:49:06.232906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.9","mappings":[{"from":5,"to":0}]},{"pgid":"7.14","mappings":[{"from":5,"to":7}]},{"pgid":"7.15","mappings":[{"from":5,"to":7}]},{"pgid":"7.30","mappings":[{"from":5,"to":7}]},{"pgid":"7.35","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/379543030":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/3951589296":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/3713107589":"2024-04-03T07:49:23.665126+0000","172.21.15.89:6833/43357":"2024-04-03T07:49:23.665126+0000","172.21.15.89:6832/43357":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/552943692":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/2589801503":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/3028439559":"2024-04-03T07:49:23.665126+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:51:06.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.923+0000 7f7328c00700 1 -- 172.21.15.89:0/3429059785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f730006ade0 msgr2=0x7f730006d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:06.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.923+0000 7f7328c00700 1 --2- 172.21.15.89:0/3429059785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f730006ade0 0x7f730006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.923+0000 7f7328c00700 1 -- 172.21.15.89:0/3429059785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73240c4f30 msgr2=0x7f73240b6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:06.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.923+0000 7f7328c00700 1 --2- 172.21.15.89:0/3429059785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73240c4f30 0x7f73240b6480 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7314000c00 tx=0x7f7314004750 comp rx=0 tx=0).stop 2024-04-02T07:51:06.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.923+0000 7f7328c00700 1 -- 172.21.15.89:0/3429059785 shutdown_connections 2024-04-02T07:51:06.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.923+0000 7f7328c00700 1 --2- 172.21.15.89:0/3429059785 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f730006ade0 0x7f730006d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.923+0000 7f7328c00700 1 --2- 172.21.15.89:0/3429059785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73240c4f30 0x7f73240b6480 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.923+0000 7f7328c00700 1 --2- 172.21.15.89:0/3429059785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f732415b660 0x7f732415d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.923+0000 7f7328c00700 1 --2- 172.21.15.89:0/3429059785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73240cfb70 0x7f732415d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:06.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.923+0000 7f7328c00700 1 -- 172.21.15.89:0/3429059785 >> 172.21.15.89:0/3429059785 conn(0x7f73240d13e0 msgr2=0x7f732406fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:06.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.923+0000 7f7328c00700 1 -- 172.21.15.89:0/3429059785 shutdown_connections 2024-04-02T07:51:06.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:06.923+0000 7f7328c00700 1 -- 172.21.15.89:0/3429059785 wait complete. 2024-04-02T07:51:06.944 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:49:05.141462+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:49:06.232906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:51:06.944 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:51:06.977 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:06.975+0000 7fde23f68240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:51:06.990 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:06.988+0000 7ff1aa269240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:51:07.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.147+0000 7f2151600700 1 Processor -- start 2024-04-02T07:51:07.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.147+0000 7f2151600700 1 -- start start 2024-04-02T07:51:07.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f2151600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f214c0cd020 0x7f214c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:07.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f2151600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f214c0b2070 0x7f214c0b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:07.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f2151600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214c0cdae0 con 0x7f214c0cbb30 2024-04-02T07:51:07.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f2151600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f214c0cddc0 con 0x7f214c0b2070 2024-04-02T07:51:07.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f2151600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214c0ce0a0 con 0x7f214c0cd020 2024-04-02T07:51:07.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f214b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f214c0b2070 0x7f214c0b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:07.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f214b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f214c0b2070 0x7f214c0b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54994/0 (socket says 172.21.15.89:54994) 2024-04-02T07:51:07.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f214b400700 1 -- 172.21.15.89:0/1163584186 learned_addr learned my addr 172.21.15.89:0/1163584186 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:07.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f214aa00700 1 --1- 172.21.15.89:0/1163584186 >> v1:172.21.15.89:6789/0 conn(0x7f214c0cbb30 0x7f214c0cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40116/0 (socket says 172.21.15.89:40116) 2024-04-02T07:51:07.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f2143e00700 1 --2- 172.21.15.89:0/1163584186 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f214c0cd020 0x7f214c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:07.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f214b400700 1 -- 172.21.15.89:0/1163584186 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f214c0cd020 msgr2=0x7f214c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:07.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f214b400700 1 --2- 172.21.15.89:0/1163584186 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f214c0cd020 0x7f214c0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f214b400700 1 -- 172.21.15.89:0/1163584186 >> v1:172.21.15.89:6789/0 conn(0x7f214c0cbb30 legacy=0x7f214c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:07.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f214b400700 1 -- 172.21.15.89:0/1163584186 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214c06dc20 con 0x7f214c0b2070 2024-04-02T07:51:07.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.149+0000 7f214a000700 1 -- 172.21.15.89:0/1163584186 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802331194 0 0) 0x7f214c0cdae0 con 0x7f214c0cbb30 2024-04-02T07:51:07.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.150+0000 7f214b400700 1 --2- 172.21.15.89:0/1163584186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c0b2070 0x7f214c0b2500 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2144009e50 tx=0x7f214400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e2b3c8b38a3b78 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:07.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.150+0000 7f214a000700 1 -- 172.21.15.89:0/1163584186 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214400e070 con 0x7f214c0b2070 2024-04-02T07:51:07.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.150+0000 7f214a000700 1 -- 172.21.15.89:0/1163584186 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214400b910 con 0x7f214c0b2070 2024-04-02T07:51:07.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.150+0000 7f214a000700 1 -- 172.21.15.89:0/1163584186 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214400bcc0 con 0x7f214c0b2070 2024-04-02T07:51:07.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.151+0000 7f2151600700 1 -- 172.21.15.89:0/1163584186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c0b2070 msgr2=0x7f214c0b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:07.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.151+0000 7f2151600700 1 --2- 172.21.15.89:0/1163584186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c0b2070 0x7f214c0b2500 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2144009e50 tx=0x7f214400b330 comp rx=0 tx=0).stop 2024-04-02T07:51:07.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.151+0000 7f2151600700 1 -- 172.21.15.89:0/1163584186 shutdown_connections 2024-04-02T07:51:07.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.151+0000 7f2151600700 1 --2- 172.21.15.89:0/1163584186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c0b2070 0x7f214c0b2500 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.151+0000 7f2151600700 1 --2- 172.21.15.89:0/1163584186 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f214c0cd020 0x7f214c0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.151+0000 7f2151600700 1 -- 172.21.15.89:0/1163584186 >> 172.21.15.89:0/1163584186 conn(0x7f214c0b0d70 msgr2=0x7f214c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:07.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.151+0000 7f2151600700 1 -- 172.21.15.89:0/1163584186 shutdown_connections 2024-04-02T07:51:07.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.151+0000 7f2151600700 1 -- 172.21.15.89:0/1163584186 wait complete. 2024-04-02T07:51:07.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.153+0000 7f2151600700 1 Processor -- start 2024-04-02T07:51:07.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.153+0000 7f2151600700 1 -- start start 2024-04-02T07:51:07.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f2151600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f214c0b2070 0x7f214c0acce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:07.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f2151600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c0cbb30 0x7f214c1f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:07.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f2151600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f214c0cd020 0x7f214c1f3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:07.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f2151600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c0709c0 con 0x7f214c0cbb30 2024-04-02T07:51:07.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f2151600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c0ad670 con 0x7f214c0b2070 2024-04-02T07:51:07.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f2151600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214c0ad950 con 0x7f214c0cd020 2024-04-02T07:51:07.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f2143e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c0cbb30 0x7f214c1f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:07.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f2143e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c0cbb30 0x7f214c1f2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55006/0 (socket says 172.21.15.89:55006) 2024-04-02T07:51:07.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f2143e00700 1 -- 172.21.15.89:0/2925606297 learned_addr learned my addr 172.21.15.89:0/2925606297 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:07.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f214b400700 1 --2- 172.21.15.89:0/2925606297 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f214c0cd020 0x7f214c1f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:07.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f214aa00700 1 --2- 172.21.15.89:0/2925606297 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f214c0b2070 0x7f214c0acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:07.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f2143e00700 1 -- 172.21.15.89:0/2925606297 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f214c0cd020 msgr2=0x7f214c1f3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:07.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f2143e00700 1 --2- 172.21.15.89:0/2925606297 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f214c0cd020 0x7f214c1f3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f2143e00700 1 -- 172.21.15.89:0/2925606297 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f214c0b2070 msgr2=0x7f214c0acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:07.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f2143e00700 1 --2- 172.21.15.89:0/2925606297 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f214c0b2070 0x7f214c0acce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.154+0000 7f2143e00700 1 -- 172.21.15.89:0/2925606297 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2144009b30 con 0x7f214c0cbb30 2024-04-02T07:51:07.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.155+0000 7f2143e00700 1 --2- 172.21.15.89:0/2925606297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c0cbb30 0x7f214c1f2ec0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f213400e870 tx=0x7f213400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:07.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.155+0000 7f2142a00700 1 -- 172.21.15.89:0/2925606297 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134009590 con 0x7f214c0cbb30 2024-04-02T07:51:07.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.155+0000 7f2142a00700 1 -- 172.21.15.89:0/2925606297 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2134013680 con 0x7f214c0cbb30 2024-04-02T07:51:07.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.155+0000 7f2151600700 1 -- 172.21.15.89:0/2925606297 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214c0adc30 con 0x7f214c0cbb30 2024-04-02T07:51:07.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.155+0000 7f2142a00700 1 -- 172.21.15.89:0/2925606297 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21340139f0 con 0x7f214c0cbb30 2024-04-02T07:51:07.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.155+0000 7f2151600700 1 -- 172.21.15.89:0/2925606297 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214c155940 con 0x7f214c0cbb30 2024-04-02T07:51:07.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.156+0000 7f2151600700 1 -- 172.21.15.89:0/2925606297 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f214c15bd50 con 0x7f214c0cbb30 2024-04-02T07:51:07.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.157+0000 7f2142a00700 1 -- 172.21.15.89:0/2925606297 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f2134013b50 con 0x7f214c0cbb30 2024-04-02T07:51:07.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.158+0000 7f2142a00700 1 --2- 172.21.15.89:0/2925606297 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f212806adb0 0x7f212806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:07.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.158+0000 7f2142a00700 1 -- 172.21.15.89:0/2925606297 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6602+0+0 (secure 0 0 0) 0x7f2134089690 con 0x7f214c0cbb30 2024-04-02T07:51:07.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.158+0000 7f214aa00700 1 -- 172.21.15.89:0/2925606297 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f212806adb0 msgr2=0x7f212806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:07.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.158+0000 7f214aa00700 1 --2- 172.21.15.89:0/2925606297 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f212806adb0 0x7f212806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:07.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.162+0000 7f2142a00700 1 -- 172.21.15.89:0/2925606297 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2134054470 con 0x7f214c0cbb30 2024-04-02T07:51:07.189 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:07.187+0000 7ff1aa269240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:51:07.247 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:07.246+0000 7fde23f68240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:51:07.281 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:07.279+0000 7ff1aa269240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:51:07.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.359+0000 7f214aa00700 1 -- 172.21.15.89:0/2925606297 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f212806adb0 msgr2=0x7f212806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:07.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.359+0000 7f214aa00700 1 --2- 172.21.15.89:0/2925606297 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f212806adb0 0x7f212806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:07.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.383+0000 7f2151600700 1 -- 172.21.15.89:0/2925606297 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f214c067720 con 0x7f214c0cbb30 2024-04-02T07:51:07.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.384+0000 7f2142a00700 1 -- 172.21.15.89:0/2925606297 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2134057a90 con 0x7f214c0cbb30 2024-04-02T07:51:07.386 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:51:07.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.386+0000 7f2151600700 1 -- 172.21.15.89:0/2925606297 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f212806adb0 msgr2=0x7f212806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:07.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.386+0000 7f2151600700 1 --2- 172.21.15.89:0/2925606297 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f212806adb0 0x7f212806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.386+0000 7f2151600700 1 -- 172.21.15.89:0/2925606297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c0cbb30 msgr2=0x7f214c1f2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:07.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.386+0000 7f2151600700 1 --2- 172.21.15.89:0/2925606297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c0cbb30 0x7f214c1f2ec0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f213400e870 tx=0x7f213400eb80 comp rx=0 tx=0).stop 2024-04-02T07:51:07.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.386+0000 7f2151600700 1 -- 172.21.15.89:0/2925606297 shutdown_connections 2024-04-02T07:51:07.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.386+0000 7f2151600700 1 --2- 172.21.15.89:0/2925606297 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f212806adb0 0x7f212806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.386+0000 7f2151600700 1 --2- 172.21.15.89:0/2925606297 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f214c0b2070 0x7f214c0acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.386+0000 7f2151600700 1 --2- 172.21.15.89:0/2925606297 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f214c0cd020 0x7f214c1f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.386+0000 7f2151600700 1 --2- 172.21.15.89:0/2925606297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f214c0cbb30 0x7f214c1f2ec0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.386+0000 7f2151600700 1 -- 172.21.15.89:0/2925606297 >> 172.21.15.89:0/2925606297 conn(0x7f214c0b0d70 msgr2=0x7f214c0b0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:07.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.386+0000 7f2151600700 1 -- 172.21.15.89:0/2925606297 shutdown_connections 2024-04-02T07:51:07.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.387+0000 7f2151600700 1 -- 172.21.15.89:0/2925606297 wait complete. 2024-04-02T07:51:07.407 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:51:07.458 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:07.457+0000 7fde23f68240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:51:07.560 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:07.558+0000 7fde23f68240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:51:07.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.645+0000 7f9b61a00700 1 Processor -- start 2024-04-02T07:51:07.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.646+0000 7f9b61a00700 1 -- start start 2024-04-02T07:51:07.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.647+0000 7f9b61a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9b5c1611b0 0x7f9b5c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:07.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.647+0000 7f9b61a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b5c15f4d0 0x7f9b5c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:07.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.647+0000 7f9b61a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b5c0ce270 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.647+0000 7f9b61a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b5c15b600 con 0x7f9b5c1611b0 2024-04-02T07:51:07.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.647+0000 7f9b61a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0d2760 con 0x7f9b5c15f4d0 2024-04-02T07:51:07.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.647+0000 7f9b5a000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b5c15f4d0 0x7f9b5c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:07.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.647+0000 7f9b5aa00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9b5c1611b0 0x7f9b5c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:07.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.647+0000 7f9b5b400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f9b5c0d21f0 0x7f9b5c161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:52156/0 (socket says 172.21.15.89:52156) 2024-04-02T07:51:07.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.647+0000 7f9b5b400700 1 -- 172.21.15.89:0/1329853034 learned_addr learned my addr 172.21.15.89:0/1329853034 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:07.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.648+0000 7f9b59600700 1 -- 172.21.15.89:0/1329853034 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929033224 0 0) 0x7f9b5c0ce270 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.648+0000 7f9b59600700 1 -- 172.21.15.89:0/1329853034 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b400035b0 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.648+0000 7f9b5aa00700 1 -- 172.21.15.89:0/1329853034 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b5c15f4d0 msgr2=0x7f9b5c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:07.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.648+0000 7f9b5aa00700 1 --2- 172.21.15.89:0/1329853034 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b5c15f4d0 0x7f9b5c0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.648+0000 7f9b5aa00700 1 -- 172.21.15.89:0/1329853034 >> v1:172.21.15.113:6789/0 conn(0x7f9b5c0d21f0 legacy=0x7f9b5c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:07.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.648+0000 7f9b5aa00700 1 -- 172.21.15.89:0/1329853034 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5c0d2a40 con 0x7f9b5c1611b0 2024-04-02T07:51:07.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.648+0000 7f9b5a000700 1 --2- 172.21.15.89:0/1329853034 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b5c15f4d0 0x7f9b5c0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:07.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.648+0000 7f9b5aa00700 1 --2- 172.21.15.89:0/1329853034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b5c1611b0 0x7f9b5c15ef90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c009840 tx=0x7f9b4c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c534919b67b7c293 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:07.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.648+0000 7f9b59600700 1 -- 172.21.15.89:0/1329853034 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c011830 con 0x7f9b5c1611b0 2024-04-02T07:51:07.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.648+0000 7f9b59600700 1 -- 172.21.15.89:0/1329853034 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4c011990 con 0x7f9b5c1611b0 2024-04-02T07:51:07.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.648+0000 7f9b59600700 1 -- 172.21.15.89:0/1329853034 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c011d20 con 0x7f9b5c1611b0 2024-04-02T07:51:07.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.649+0000 7f9b61a00700 1 -- 172.21.15.89:0/1329853034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b5c1611b0 msgr2=0x7f9b5c15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:07.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.649+0000 7f9b61a00700 1 --2- 172.21.15.89:0/1329853034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b5c1611b0 0x7f9b5c15ef90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c009840 tx=0x7f9b4c009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:07.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.649+0000 7f9b61a00700 1 -- 172.21.15.89:0/1329853034 shutdown_connections 2024-04-02T07:51:07.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.649+0000 7f9b61a00700 1 --2- 172.21.15.89:0/1329853034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b5c1611b0 0x7f9b5c15ef90 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.649+0000 7f9b61a00700 1 --2- 172.21.15.89:0/1329853034 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b5c15f4d0 0x7f9b5c0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.649+0000 7f9b61a00700 1 -- 172.21.15.89:0/1329853034 >> 172.21.15.89:0/1329853034 conn(0x7f9b5c08aff0 msgr2=0x7f9b5c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:07.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.649+0000 7f9b61a00700 1 -- 172.21.15.89:0/1329853034 shutdown_connections 2024-04-02T07:51:07.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.649+0000 7f9b61a00700 1 -- 172.21.15.89:0/1329853034 wait complete. 2024-04-02T07:51:07.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.651+0000 7f9b61a00700 1 Processor -- start 2024-04-02T07:51:07.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.652+0000 7f9b61a00700 1 -- start start 2024-04-02T07:51:07.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b61a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b5c0d21f0 0x7f9b5c156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:07.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b61a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b5c15f4d0 0x7f9b5c0b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:07.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b61a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b5c1611b0 0x7f9b5c15c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:07.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b61a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c156b80 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b61a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0cbbc0 con 0x7f9b5c15f4d0 2024-04-02T07:51:07.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b61a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0b7420 con 0x7f9b5c1611b0 2024-04-02T07:51:07.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b5aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b5c0d21f0 0x7f9b5c156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:07.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b5aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b5c0d21f0 0x7f9b5c156640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55018/0 (socket says 172.21.15.89:55018) 2024-04-02T07:51:07.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b5aa00700 1 -- 172.21.15.89:0/2015145454 learned_addr learned my addr 172.21.15.89:0/2015145454 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:07.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b5a000700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b5c15f4d0 0x7f9b5c0b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:07.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b5b400700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b5c1611b0 0x7f9b5c15c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:07.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b5aa00700 1 -- 172.21.15.89:0/2015145454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b5c1611b0 msgr2=0x7f9b5c15c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:07.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b5aa00700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b5c1611b0 0x7f9b5c15c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b5aa00700 1 -- 172.21.15.89:0/2015145454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b5c15f4d0 msgr2=0x7f9b5c0b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:07.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b5aa00700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b5c15f4d0 0x7f9b5c0b6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b5aa00700 1 -- 172.21.15.89:0/2015145454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b480094a0 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.653+0000 7f9b5a000700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b5c15f4d0 0x7f9b5c0b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:07.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.654+0000 7f9b5aa00700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b5c0d21f0 0x7f9b5c156640 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c009fd0 tx=0x7f9b4c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:07.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.654+0000 7f9b52a00700 1 -- 172.21.15.89:0/2015145454 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c0036a0 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.654+0000 7f9b52a00700 1 -- 172.21.15.89:0/2015145454 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4c01e070 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.654+0000 7f9b61a00700 1 -- 172.21.15.89:0/2015145454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b4c0094a0 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.654+0000 7f9b52a00700 1 -- 172.21.15.89:0/2015145454 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c003a20 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.654+0000 7f9b61a00700 1 -- 172.21.15.89:0/2015145454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b5c0cbe60 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.655+0000 7f9b61a00700 1 -- 172.21.15.89:0/2015145454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5c067720 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.656+0000 7f9b52a00700 1 -- 172.21.15.89:0/2015145454 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f9b4c003b80 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.657+0000 7f9b52a00700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9b3c06adb0 0x7f9b3c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:07.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.658+0000 7f9b5a000700 1 -- 172.21.15.89:0/2015145454 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9b3c06adb0 msgr2=0x7f9b3c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:07.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.658+0000 7f9b52a00700 1 -- 172.21.15.89:0/2015145454 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6602+0+0 (secure 0 0 0) 0x7f9b4c08a7f0 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.658+0000 7f9b5a000700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9b3c06adb0 0x7f9b3c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:07.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.661+0000 7f9b52a00700 1 -- 172.21.15.89:0/2015145454 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9b4c055530 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.859+0000 7f9b5a000700 1 -- 172.21.15.89:0/2015145454 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9b3c06adb0 msgr2=0x7f9b3c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:07.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.859+0000 7f9b5a000700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9b3c06adb0 0x7f9b3c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:07.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.872+0000 7f9b61a00700 1 -- 172.21.15.89:0/2015145454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9b5c0b5d00 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.873+0000 7f9b52a00700 1 -- 172.21.15.89:0/2015145454 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9b4c058b50 con 0x7f9b5c0d21f0 2024-04-02T07:51:07.875 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:51:07.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.876+0000 7f9b61a00700 1 -- 172.21.15.89:0/2015145454 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9b3c06adb0 msgr2=0x7f9b3c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.876+0000 7f9b61a00700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9b3c06adb0 0x7f9b3c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.876+0000 7f9b61a00700 1 -- 172.21.15.89:0/2015145454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b5c0d21f0 msgr2=0x7f9b5c156640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.876+0000 7f9b61a00700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b5c0d21f0 0x7f9b5c156640 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c009fd0 tx=0x7f9b4c004750 comp rx=0 tx=0).stop 2024-04-02T07:51:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.876+0000 7f9b61a00700 1 -- 172.21.15.89:0/2015145454 shutdown_connections 2024-04-02T07:51:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.876+0000 7f9b61a00700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9b3c06adb0 0x7f9b3c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.876+0000 7f9b61a00700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b5c1611b0 0x7f9b5c15c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.876+0000 7f9b61a00700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b5c0d21f0 0x7f9b5c156640 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.876+0000 7f9b61a00700 1 --2- 172.21.15.89:0/2015145454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b5c15f4d0 0x7f9b5c0b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.876+0000 7f9b61a00700 1 -- 172.21.15.89:0/2015145454 >> 172.21.15.89:0/2015145454 conn(0x7f9b5c08aff0 msgr2=0x7f9b5c15fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.876+0000 7f9b61a00700 1 -- 172.21.15.89:0/2015145454 shutdown_connections 2024-04-02T07:51:07.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:07.876+0000 7f9b61a00700 1 -- 172.21.15.89:0/2015145454 wait complete. 2024-04-02T07:51:07.895 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:51:08.014 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:08.012+0000 7ff1aa269240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:51:08.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.134+0000 7fefc0400700 1 Processor -- start 2024-04-02T07:51:08.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.134+0000 7fefc0400700 1 -- start start 2024-04-02T07:51:08.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.135+0000 7fefc0400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefb815dad0 0x7fefb80ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:08.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.135+0000 7fefc0400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fefb80d21a0 0x7fefb815fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:08.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.136+0000 7fefc0400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb815f820 con 0x7fefb80d1630 2024-04-02T07:51:08.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.136+0000 7fefc0400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb80cfcc0 con 0x7fefb80d21a0 2024-04-02T07:51:08.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.136+0000 7fefc0400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80cd9f0 con 0x7fefb815dad0 2024-04-02T07:51:08.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.136+0000 7fefbd000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fefb80d1630 0x7fefb80d1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40122/0 (socket says 172.21.15.89:40122) 2024-04-02T07:51:08.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.136+0000 7fefbd000700 1 -- 172.21.15.89:0/2638945471 learned_addr learned my addr 172.21.15.89:0/2638945471 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:08.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.136+0000 7fefbda00700 1 --2- 172.21.15.89:0/2638945471 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefb815dad0 0x7fefb80ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:08.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.136+0000 7fefbe400700 1 --2- 172.21.15.89:0/2638945471 >> v2:172.21.15.113:3300/0 conn(0x7fefb80d21a0 0x7fefb815fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:08.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.136+0000 7fefb7e00700 1 -- 172.21.15.89:0/2638945471 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330612666 0 0) 0x7fefb815f820 con 0x7fefb80d1630 2024-04-02T07:51:08.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.136+0000 7fefb7e00700 1 -- 172.21.15.89:0/2638945471 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa00035b0 con 0x7fefb80d1630 2024-04-02T07:51:08.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.136+0000 7fefbda00700 1 -- 172.21.15.89:0/2638945471 >> v2:172.21.15.113:3300/0 conn(0x7fefb80d21a0 msgr2=0x7fefb815fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:08.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.136+0000 7fefbda00700 1 --2- 172.21.15.89:0/2638945471 >> v2:172.21.15.113:3300/0 conn(0x7fefb80d21a0 0x7fefb815fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.136+0000 7fefbda00700 1 -- 172.21.15.89:0/2638945471 >> v1:172.21.15.89:6789/0 conn(0x7fefb80d1630 legacy=0x7fefb80d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:08.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.136+0000 7fefbda00700 1 -- 172.21.15.89:0/2638945471 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb80cf420 con 0x7fefb815dad0 2024-04-02T07:51:08.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.136+0000 7fefbe400700 1 --2- 172.21.15.89:0/2638945471 >> v2:172.21.15.113:3300/0 conn(0x7fefb80d21a0 0x7fefb815fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:08.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.137+0000 7fefbda00700 1 --2- 172.21.15.89:0/2638945471 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefb815dad0 0x7fefb80ced80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fefac009840 tx=0x7fefac009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd3c9672d483b941 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:08.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.137+0000 7fefb7e00700 1 -- 172.21.15.89:0/2638945471 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac011830 con 0x7fefb815dad0 2024-04-02T07:51:08.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.137+0000 7fefb7e00700 1 -- 172.21.15.89:0/2638945471 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefac011990 con 0x7fefb815dad0 2024-04-02T07:51:08.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.137+0000 7fefb7e00700 1 -- 172.21.15.89:0/2638945471 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac011d20 con 0x7fefb815dad0 2024-04-02T07:51:08.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.138+0000 7fefc0400700 1 -- 172.21.15.89:0/2638945471 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefb815dad0 msgr2=0x7fefb80ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:08.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.138+0000 7fefc0400700 1 --2- 172.21.15.89:0/2638945471 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefb815dad0 0x7fefb80ced80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fefac009840 tx=0x7fefac009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:08.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.138+0000 7fefc0400700 1 -- 172.21.15.89:0/2638945471 shutdown_connections 2024-04-02T07:51:08.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.138+0000 7fefc0400700 1 --2- 172.21.15.89:0/2638945471 >> v2:172.21.15.113:3300/0 conn(0x7fefb80d21a0 0x7fefb815fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.138+0000 7fefc0400700 1 --2- 172.21.15.89:0/2638945471 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefb815dad0 0x7fefb80ced80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.138+0000 7fefc0400700 1 -- 172.21.15.89:0/2638945471 >> 172.21.15.89:0/2638945471 conn(0x7fefb808aff0 msgr2=0x7fefb80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:08.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.138+0000 7fefc0400700 1 -- 172.21.15.89:0/2638945471 shutdown_connections 2024-04-02T07:51:08.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.138+0000 7fefc0400700 1 -- 172.21.15.89:0/2638945471 wait complete. 2024-04-02T07:51:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.140+0000 7fefc0400700 1 Processor -- start 2024-04-02T07:51:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.140+0000 7fefc0400700 1 -- start start 2024-04-02T07:51:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.140+0000 7fefc0400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefb80d1630 0x7fefb8151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefc0400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefb80d21a0 0x7fefb80c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefc0400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefb815dad0 0x7fefb80cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefc0400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80c7740 con 0x7fefb80d21a0 2024-04-02T07:51:08.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefc0400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80b8210 con 0x7fefb815dad0 2024-04-02T07:51:08.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefc0400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80b84f0 con 0x7fefb80d1630 2024-04-02T07:51:08.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefbd000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefb80d21a0 0x7fefb80c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:08.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefbda00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefb80d1630 0x7fefb8151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:08.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefbda00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefb80d1630 0x7fefb8151aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:51342/0 (socket says 172.21.15.89:51342) 2024-04-02T07:51:08.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefbda00700 1 -- 172.21.15.89:0/1151048836 learned_addr learned my addr 172.21.15.89:0/1151048836 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:08.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefbd000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefb80d21a0 0x7fefb80c8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55026/0 (socket says 172.21.15.89:55026) 2024-04-02T07:51:08.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefbe400700 1 --2- 172.21.15.89:0/1151048836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefb815dad0 0x7fefb80cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:08.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefbda00700 1 -- 172.21.15.89:0/1151048836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefb815dad0 msgr2=0x7fefb80cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:08.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefbda00700 1 --2- 172.21.15.89:0/1151048836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefb815dad0 0x7fefb80cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefbda00700 1 -- 172.21.15.89:0/1151048836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefb80d21a0 msgr2=0x7fefb80c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:08.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefbda00700 1 --2- 172.21.15.89:0/1151048836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefb80d21a0 0x7fefb80c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefbda00700 1 -- 172.21.15.89:0/1151048836 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb0009a70 con 0x7fefb80d1630 2024-04-02T07:51:08.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.141+0000 7fefbd000700 1 --2- 172.21.15.89:0/1151048836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefb80d21a0 0x7fefb80c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:08.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.142+0000 7fefbda00700 1 --2- 172.21.15.89:0/1151048836 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefb80d1630 0x7fefb8151aa0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fefac014040 tx=0x7fefac004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:08.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.142+0000 7fefb5600700 1 -- 172.21.15.89:0/1151048836 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac004550 con 0x7fefb80d1630 2024-04-02T07:51:08.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.142+0000 7fefb5600700 1 -- 172.21.15.89:0/1151048836 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefac01f070 con 0x7fefb80d1630 2024-04-02T07:51:08.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.142+0000 7fefb5600700 1 -- 172.21.15.89:0/1151048836 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac0038c0 con 0x7fefb80d1630 2024-04-02T07:51:08.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.142+0000 7fefc0400700 1 -- 172.21.15.89:0/1151048836 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefac0094a0 con 0x7fefb80d1630 2024-04-02T07:51:08.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.142+0000 7fefc0400700 1 -- 172.21.15.89:0/1151048836 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb80c73d0 con 0x7fefb80d1630 2024-04-02T07:51:08.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.143+0000 7fefc0400700 1 -- 172.21.15.89:0/1151048836 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb80d1420 con 0x7fefb80d1630 2024-04-02T07:51:08.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.144+0000 7fefb5600700 1 -- 172.21.15.89:0/1151048836 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fefac003a20 con 0x7fefb80d1630 2024-04-02T07:51:08.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.145+0000 7fefb5600700 1 --2- 172.21.15.89:0/1151048836 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fef9806adb0 0x7fef9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:08.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.145+0000 7fefb5600700 1 -- 172.21.15.89:0/1151048836 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6602+0+0 (secure 0 0 0) 0x7fefac021070 con 0x7fefb80d1630 2024-04-02T07:51:08.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.145+0000 7fefbd000700 1 -- 172.21.15.89:0/1151048836 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fef9806adb0 msgr2=0x7fef9806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:08.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.145+0000 7fefbd000700 1 --2- 172.21.15.89:0/1151048836 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fef9806adb0 0x7fef9806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:08.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.149+0000 7fefb5600700 1 -- 172.21.15.89:0/1151048836 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fefac090020 con 0x7fefb80d1630 2024-04-02T07:51:08.227 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:08.225+0000 7ff1aa269240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:51:08.332 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:08.330+0000 7fde23f68240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:51:08.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.346+0000 7fefbd000700 1 -- 172.21.15.89:0/1151048836 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fef9806adb0 msgr2=0x7fef9806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:08.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.346+0000 7fefbd000700 1 --2- 172.21.15.89:0/1151048836 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fef9806adb0 0x7fef9806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:08.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.360+0000 7fefc0400700 1 -- 172.21.15.89:0/1151048836 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fefb8067720 con 0x7fefb80d1630 2024-04-02T07:51:08.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.360+0000 7fefb5600700 1 -- 172.21.15.89:0/1151048836 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7fefac02b0a0 con 0x7fefb80d1630 2024-04-02T07:51:08.362 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:51:08.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.364+0000 7fefc0400700 1 -- 172.21.15.89:0/1151048836 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fef9806adb0 msgr2=0x7fef9806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:08.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.364+0000 7fefc0400700 1 --2- 172.21.15.89:0/1151048836 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fef9806adb0 0x7fef9806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.364+0000 7fefc0400700 1 -- 172.21.15.89:0/1151048836 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefb80d1630 msgr2=0x7fefb8151aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:08.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.364+0000 7fefc0400700 1 --2- 172.21.15.89:0/1151048836 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefb80d1630 0x7fefb8151aa0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fefac014040 tx=0x7fefac004750 comp rx=0 tx=0).stop 2024-04-02T07:51:08.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.364+0000 7fefc0400700 1 -- 172.21.15.89:0/1151048836 shutdown_connections 2024-04-02T07:51:08.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.364+0000 7fefc0400700 1 --2- 172.21.15.89:0/1151048836 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fef9806adb0 0x7fef9806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.364+0000 7fefc0400700 1 --2- 172.21.15.89:0/1151048836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefb815dad0 0x7fefb80cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.364+0000 7fefc0400700 1 --2- 172.21.15.89:0/1151048836 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefb80d1630 0x7fefb8151aa0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.364+0000 7fefc0400700 1 --2- 172.21.15.89:0/1151048836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefb80d21a0 0x7fefb80c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.364+0000 7fefc0400700 1 -- 172.21.15.89:0/1151048836 >> 172.21.15.89:0/1151048836 conn(0x7fefb808aff0 msgr2=0x7fefb80cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:08.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.365+0000 7fefc0400700 1 -- 172.21.15.89:0/1151048836 shutdown_connections 2024-04-02T07:51:08.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.365+0000 7fefc0400700 1 -- 172.21.15.89:0/1151048836 wait complete. 2024-04-02T07:51:08.392 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:51:08.433 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:08.430+0000 7ff1aa269240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:51:08.568 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:08.566+0000 7fde23f68240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:51:08.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.590+0000 7f985b000700 1 Processor -- start 2024-04-02T07:51:08.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.590+0000 7f985b000700 1 -- start start 2024-04-02T07:51:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.592+0000 7f985b000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98540b3af0 0x7f985415a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.592+0000 7f985b000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9854072580 0x7f98540b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.592+0000 7f985b000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98540b2180 con 0x7f98540b2e50 2024-04-02T07:51:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.592+0000 7f985b000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f985415b100 con 0x7f9854072580 2024-04-02T07:51:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.592+0000 7f985b000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985415b3e0 con 0x7f98540b3af0 2024-04-02T07:51:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.592+0000 7f9853e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98540b3af0 0x7f985415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.592+0000 7f9853400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9854072580 0x7f98540b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:08.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.593+0000 7f9853e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98540b3af0 0x7f985415a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:51358/0 (socket says 172.21.15.89:51358) 2024-04-02T07:51:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.593+0000 7f9853e00700 1 -- 172.21.15.89:0/1815963520 learned_addr learned my addr 172.21.15.89:0/1815963520 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.593+0000 7f9853e00700 1 -- 172.21.15.89:0/1815963520 >> v2:172.21.15.113:3300/0 conn(0x7f9854072580 msgr2=0x7f98540b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.593+0000 7f9853e00700 1 --2- 172.21.15.89:0/1815963520 >> v2:172.21.15.113:3300/0 conn(0x7f9854072580 0x7f98540b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.593+0000 7f9853e00700 1 -- 172.21.15.89:0/1815963520 >> v1:172.21.15.113:6789/0 conn(0x7f98540b2e50 legacy=0x7f98540709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.593+0000 7f9853e00700 1 -- 172.21.15.89:0/1815963520 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98540b23f0 con 0x7f98540b3af0 2024-04-02T07:51:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.593+0000 7f9853e00700 1 --2- 172.21.15.89:0/1815963520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98540b3af0 0x7f985415a860 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f984400bb90 tx=0x7f984400bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4f357f8345f12ab server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.593+0000 7f9852a00700 1 -- 172.21.15.89:0/1815963520 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984400dd30 con 0x7f98540b3af0 2024-04-02T07:51:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.594+0000 7f9852a00700 1 -- 172.21.15.89:0/1815963520 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984400de90 con 0x7f98540b3af0 2024-04-02T07:51:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.594+0000 7f9852a00700 1 -- 172.21.15.89:0/1815963520 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844010690 con 0x7f98540b3af0 2024-04-02T07:51:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.594+0000 7f985b000700 1 -- 172.21.15.89:0/1815963520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98540b3af0 msgr2=0x7f985415a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.594+0000 7f985b000700 1 --2- 172.21.15.89:0/1815963520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98540b3af0 0x7f985415a860 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f984400bb90 tx=0x7f984400bea0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.594+0000 7f985b000700 1 -- 172.21.15.89:0/1815963520 shutdown_connections 2024-04-02T07:51:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.594+0000 7f985b000700 1 --2- 172.21.15.89:0/1815963520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98540b3af0 0x7f985415a860 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.594+0000 7f985b000700 1 --2- 172.21.15.89:0/1815963520 >> v2:172.21.15.113:3300/0 conn(0x7f9854072580 0x7f98540b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.594+0000 7f985b000700 1 -- 172.21.15.89:0/1815963520 >> 172.21.15.89:0/1815963520 conn(0x7f98540cf190 msgr2=0x7f98540cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.594+0000 7f985b000700 1 -- 172.21.15.89:0/1815963520 shutdown_connections 2024-04-02T07:51:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.594+0000 7f985b000700 1 -- 172.21.15.89:0/1815963520 wait complete. 2024-04-02T07:51:08.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.596+0000 7f985b000700 1 Processor -- start 2024-04-02T07:51:08.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.596+0000 7f985b000700 1 -- start start 2024-04-02T07:51:08.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.597+0000 7f985b000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9854072580 0x7f985414e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:08.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.597+0000 7f985b000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98540b2e50 0x7f9854152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:08.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.597+0000 7f985b000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98540b3af0 0x7f9854152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:08.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.597+0000 7f985b000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98541f2fd0 con 0x7f9854072580 2024-04-02T07:51:08.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.597+0000 7f985b000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985414ec40 con 0x7f98540b3af0 2024-04-02T07:51:08.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.597+0000 7f985b000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98541f31e0 con 0x7f98540b2e50 2024-04-02T07:51:08.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.597+0000 7f9853e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9854072580 0x7f985414e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:08.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.597+0000 7f9853400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98540b2e50 0x7f9854152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:08.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.597+0000 7f9853e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9854072580 0x7f985414e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55032/0 (socket says 172.21.15.89:55032) 2024-04-02T07:51:08.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.597+0000 7f9853e00700 1 -- 172.21.15.89:0/1816771290 learned_addr learned my addr 172.21.15.89:0/1816771290 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:08.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.597+0000 7f9859000700 1 --2- 172.21.15.89:0/1816771290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98540b3af0 0x7f9854152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.598+0000 7f9853e00700 1 -- 172.21.15.89:0/1816771290 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98540b2e50 msgr2=0x7f9854152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.598+0000 7f9853e00700 1 --2- 172.21.15.89:0/1816771290 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98540b2e50 0x7f9854152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.598+0000 7f9853e00700 1 -- 172.21.15.89:0/1816771290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98540b3af0 msgr2=0x7f9854152bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.598+0000 7f9853e00700 1 --2- 172.21.15.89:0/1816771290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98540b3af0 0x7f9854152bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.598+0000 7f9853e00700 1 -- 172.21.15.89:0/1816771290 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984400b7f0 con 0x7f9854072580 2024-04-02T07:51:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.598+0000 7f9853400700 1 --2- 172.21.15.89:0/1816771290 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98540b2e50 0x7f9854152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.598+0000 7f9853e00700 1 --2- 172.21.15.89:0/1816771290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9854072580 0x7f985414e700 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9844000c00 tx=0x7f9844004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.598+0000 7f984be00700 1 -- 172.21.15.89:0/1816771290 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844003b70 con 0x7f9854072580 2024-04-02T07:51:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.598+0000 7f985b000700 1 -- 172.21.15.89:0/1816771290 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98541f34c0 con 0x7f9854072580 2024-04-02T07:51:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.598+0000 7f984be00700 1 -- 172.21.15.89:0/1816771290 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9844003cd0 con 0x7f9854072580 2024-04-02T07:51:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.598+0000 7f984be00700 1 -- 172.21.15.89:0/1816771290 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98440106b0 con 0x7f9854072580 2024-04-02T07:51:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.599+0000 7f985b000700 1 -- 172.21.15.89:0/1816771290 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98541f3a80 con 0x7f9854072580 2024-04-02T07:51:08.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.600+0000 7f985b000700 1 -- 172.21.15.89:0/1816771290 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9854067720 con 0x7f9854072580 2024-04-02T07:51:08.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.601+0000 7f984be00700 1 -- 172.21.15.89:0/1816771290 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f9844019400 con 0x7f9854072580 2024-04-02T07:51:08.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.602+0000 7f984be00700 1 --2- 172.21.15.89:0/1816771290 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f983406adb0 0x7f983406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:08.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.602+0000 7f984be00700 1 -- 172.21.15.89:0/1816771290 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6602+0+0 (secure 0 0 0) 0x7f984408b380 con 0x7f9854072580 2024-04-02T07:51:08.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.602+0000 7f9853400700 1 -- 172.21.15.89:0/1816771290 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f983406adb0 msgr2=0x7f983406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:08.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.602+0000 7f9853400700 1 --2- 172.21.15.89:0/1816771290 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f983406adb0 0x7f983406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:08.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.606+0000 7f984be00700 1 -- 172.21.15.89:0/1816771290 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9844056050 con 0x7f9854072580 2024-04-02T07:51:08.787 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:08.785+0000 7fde23f68240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:51:08.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.802+0000 7f9853400700 1 -- 172.21.15.89:0/1816771290 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f983406adb0 msgr2=0x7f983406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:08.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.802+0000 7f9853400700 1 --2- 172.21.15.89:0/1816771290 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f983406adb0 0x7f983406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:08.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.859+0000 7f985b000700 1 -- 172.21.15.89:0/1816771290 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98540cef70 con 0x7f9854072580 2024-04-02T07:51:08.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.860+0000 7f984be00700 1 -- 172.21.15.89:0/1816771290 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f9844059670 con 0x7f9854072580 2024-04-02T07:51:08.863 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:08.863 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5710,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3098065397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3098065397},{"type":"v1","addr":"172.21.15.89:6836","nonce":3098065397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5713,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1408411829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1408411829},{"type":"v1","addr":"172.21.15.89:6837","nonce":1408411829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5716,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3266955664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3266955664},{"type":"v1","addr":"172.21.15.113:6834","nonce":3266955664}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:49:07.249043+0000","modified":"2024-04-02T07:49:10.254959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5719":{"gid":5719,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6835/427074923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":427074923},{"type":"v1","addr":"172.21.15.113:6835","nonce":427074923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-02T07:51:08.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.863+0000 7f985b000700 1 -- 172.21.15.89:0/1816771290 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f983406adb0 msgr2=0x7f983406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:08.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.863+0000 7f985b000700 1 --2- 172.21.15.89:0/1816771290 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f983406adb0 0x7f983406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.863+0000 7f985b000700 1 -- 172.21.15.89:0/1816771290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9854072580 msgr2=0x7f985414e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.863+0000 7f985b000700 1 --2- 172.21.15.89:0/1816771290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9854072580 0x7f985414e700 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9844000c00 tx=0x7f9844004750 comp rx=0 tx=0).stop 2024-04-02T07:51:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.863+0000 7f985b000700 1 -- 172.21.15.89:0/1816771290 shutdown_connections 2024-04-02T07:51:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.863+0000 7f985b000700 1 --2- 172.21.15.89:0/1816771290 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f983406adb0 0x7f983406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.863+0000 7f985b000700 1 --2- 172.21.15.89:0/1816771290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98540b3af0 0x7f9854152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.863+0000 7f985b000700 1 --2- 172.21.15.89:0/1816771290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9854072580 0x7f985414e700 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.863+0000 7f985b000700 1 --2- 172.21.15.89:0/1816771290 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98540b2e50 0x7f9854152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.863+0000 7f985b000700 1 -- 172.21.15.89:0/1816771290 >> 172.21.15.89:0/1816771290 conn(0x7f98540cf190 msgr2=0x7f98540709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.863+0000 7f985b000700 1 -- 172.21.15.89:0/1816771290 shutdown_connections 2024-04-02T07:51:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:08.863+0000 7f985b000700 1 -- 172.21.15.89:0/1816771290 wait complete. 2024-04-02T07:51:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 24 2024-04-02T07:51:08.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:51:08.903 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:08.901+0000 7ff1aa269240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:51:09.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.075+0000 7f2d45200700 1 Processor -- start 2024-04-02T07:51:09.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.075+0000 7f2d45200700 1 -- start start 2024-04-02T07:51:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.076+0000 7f2d45200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d400b3af0 0x7f2d4015a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.077+0000 7f2d45200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2d40072580 0x7f2d400b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.077+0000 7f2d45200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d400b2180 con 0x7f2d400b2e50 2024-04-02T07:51:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.077+0000 7f2d45200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d4015b100 con 0x7f2d40072580 2024-04-02T07:51:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.077+0000 7f2d45200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d4015b3e0 con 0x7f2d400b3af0 2024-04-02T07:51:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.077+0000 7f2d3d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2d40072580 0x7f2d400b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.077+0000 7f2d3d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2d40072580 0x7f2d400b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34236/0 (socket says 172.21.15.89:34236) 2024-04-02T07:51:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.077+0000 7f2d3d400700 1 -- 172.21.15.89:0/3049833881 learned_addr learned my addr 172.21.15.89:0/3049833881 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.077+0000 7f2d3e800700 1 --1- 172.21.15.89:0/3049833881 >> v1:172.21.15.89:6789/0 conn(0x7f2d400b2e50 0x7f2d400709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32998/0 (socket says 172.21.15.89:32998) 2024-04-02T07:51:09.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.077+0000 7f2d3de00700 1 --2- 172.21.15.89:0/3049833881 >> v2:172.21.15.113:3300/0 conn(0x7f2d400b3af0 0x7f2d4015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:09.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.077+0000 7f2d3d400700 1 -- 172.21.15.89:0/3049833881 >> v2:172.21.15.113:3300/0 conn(0x7f2d400b3af0 msgr2=0x7f2d4015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:09.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.077+0000 7f2d3d400700 1 --2- 172.21.15.89:0/3049833881 >> v2:172.21.15.113:3300/0 conn(0x7f2d400b3af0 0x7f2d4015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.077+0000 7f2d3d400700 1 -- 172.21.15.89:0/3049833881 >> v1:172.21.15.89:6789/0 conn(0x7f2d400b2e50 legacy=0x7f2d400709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:09.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.077+0000 7f2d3d400700 1 -- 172.21.15.89:0/3049833881 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d400b23f0 con 0x7f2d40072580 2024-04-02T07:51:09.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.077+0000 7f2d3ca00700 1 -- 172.21.15.89:0/3049833881 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633723482 0 0) 0x7f2d400b2180 con 0x7f2d400b2e50 2024-04-02T07:51:09.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.078+0000 7f2d3d400700 1 --2- 172.21.15.89:0/3049833881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d40072580 0x7f2d400b2910 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f2d4006b860 tx=0x7f2d2c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f9a2c2e276425e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:09.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.078+0000 7f2d3ca00700 1 -- 172.21.15.89:0/3049833881 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c00b870 con 0x7f2d40072580 2024-04-02T07:51:09.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.078+0000 7f2d3ca00700 1 -- 172.21.15.89:0/3049833881 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2c00b9d0 con 0x7f2d40072580 2024-04-02T07:51:09.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.078+0000 7f2d3ca00700 1 -- 172.21.15.89:0/3049833881 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c00bd60 con 0x7f2d40072580 2024-04-02T07:51:09.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.078+0000 7f2d45200700 1 -- 172.21.15.89:0/3049833881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d40072580 msgr2=0x7f2d400b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:09.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.078+0000 7f2d45200700 1 --2- 172.21.15.89:0/3049833881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d40072580 0x7f2d400b2910 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f2d4006b860 tx=0x7f2d2c009bd0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.079+0000 7f2d45200700 1 -- 172.21.15.89:0/3049833881 shutdown_connections 2024-04-02T07:51:09.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.079+0000 7f2d45200700 1 --2- 172.21.15.89:0/3049833881 >> v2:172.21.15.113:3300/0 conn(0x7f2d400b3af0 0x7f2d4015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.079+0000 7f2d45200700 1 --2- 172.21.15.89:0/3049833881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d40072580 0x7f2d400b2910 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.079+0000 7f2d45200700 1 -- 172.21.15.89:0/3049833881 >> 172.21.15.89:0/3049833881 conn(0x7f2d400cf190 msgr2=0x7f2d400cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:09.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.079+0000 7f2d45200700 1 -- 172.21.15.89:0/3049833881 shutdown_connections 2024-04-02T07:51:09.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.079+0000 7f2d45200700 1 -- 172.21.15.89:0/3049833881 wait complete. 2024-04-02T07:51:09.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.080+0000 7f2d45200700 1 Processor -- start 2024-04-02T07:51:09.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.080+0000 7f2d45200700 1 -- start start 2024-04-02T07:51:09.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.081+0000 7f2d45200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d40072580 0x7f2d40151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:09.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.081+0000 7f2d45200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d400b2e50 0x7f2d4015d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:09.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.081+0000 7f2d45200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d400b3af0 0x7f2d4014efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:09.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.081+0000 7f2d45200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4015c4f0 con 0x7f2d400b3af0 2024-04-02T07:51:09.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.081+0000 7f2d45200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4015d840 con 0x7f2d40072580 2024-04-02T07:51:09.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.081+0000 7f2d45200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4015b8e0 con 0x7f2d400b2e50 2024-04-02T07:51:09.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.082+0000 7f2d3e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d400b3af0 0x7f2d4014efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:09.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.082+0000 7f2d3e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d400b3af0 0x7f2d4014efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34248/0 (socket says 172.21.15.89:34248) 2024-04-02T07:51:09.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.082+0000 7f2d3e800700 1 -- 172.21.15.89:0/487628605 learned_addr learned my addr 172.21.15.89:0/487628605 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:09.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.082+0000 7f2d3de00700 1 --2- 172.21.15.89:0/487628605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d40072580 0x7f2d40151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:09.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.082+0000 7f2d3d400700 1 --2- 172.21.15.89:0/487628605 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d400b2e50 0x7f2d4015d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:09.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.082+0000 7f2d3e800700 1 -- 172.21.15.89:0/487628605 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d400b2e50 msgr2=0x7f2d4015d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:09.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.082+0000 7f2d3e800700 1 --2- 172.21.15.89:0/487628605 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d400b2e50 0x7f2d4015d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.082+0000 7f2d3e800700 1 -- 172.21.15.89:0/487628605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d40072580 msgr2=0x7f2d40151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:09.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.082+0000 7f2d3e800700 1 --2- 172.21.15.89:0/487628605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d40072580 0x7f2d40151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.082+0000 7f2d3e800700 1 -- 172.21.15.89:0/487628605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c009550 con 0x7f2d400b3af0 2024-04-02T07:51:09.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.082+0000 7f2d3e800700 1 --2- 172.21.15.89:0/487628605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d400b3af0 0x7f2d4014efb0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f2d3800caf0 tx=0x7f2d3800ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:09.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.082+0000 7f2d36000700 1 -- 172.21.15.89:0/487628605 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3800a5d0 con 0x7f2d400b3af0 2024-04-02T07:51:09.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.083+0000 7f2d36000700 1 -- 172.21.15.89:0/487628605 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3800a730 con 0x7f2d400b3af0 2024-04-02T07:51:09.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.083+0000 7f2d36000700 1 -- 172.21.15.89:0/487628605 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3800aac0 con 0x7f2d400b3af0 2024-04-02T07:51:09.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.083+0000 7f2d45200700 1 -- 172.21.15.89:0/487628605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d4015bbc0 con 0x7f2d400b3af0 2024-04-02T07:51:09.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.083+0000 7f2d45200700 1 -- 172.21.15.89:0/487628605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d4015c180 con 0x7f2d400b3af0 2024-04-02T07:51:09.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.084+0000 7f2d45200700 1 -- 172.21.15.89:0/487628605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d4015c660 con 0x7f2d400b3af0 2024-04-02T07:51:09.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.085+0000 7f2d36000700 1 -- 172.21.15.89:0/487628605 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f2d3800ac20 con 0x7f2d400b3af0 2024-04-02T07:51:09.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.085+0000 7f2d36000700 1 --2- 172.21.15.89:0/487628605 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2d2006adb0 0x7f2d2006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:09.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.086+0000 7f2d36000700 1 -- 172.21.15.89:0/487628605 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6602+0+0 (secure 0 0 0) 0x7f2d38089630 con 0x7f2d400b3af0 2024-04-02T07:51:09.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.086+0000 7f2d3de00700 1 -- 172.21.15.89:0/487628605 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2d2006adb0 msgr2=0x7f2d2006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:09.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.086+0000 7f2d3de00700 1 --2- 172.21.15.89:0/487628605 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2d2006adb0 0x7f2d2006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:09.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.090+0000 7f2d36000700 1 -- 172.21.15.89:0/487628605 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2d38054420 con 0x7f2d400b3af0 2024-04-02T07:51:09.106 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:09.104+0000 7ff1aa269240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:51:09.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.287+0000 7f2d3de00700 1 -- 172.21.15.89:0/487628605 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2d2006adb0 msgr2=0x7f2d2006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:09.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.287+0000 7f2d3de00700 1 --2- 172.21.15.89:0/487628605 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2d2006adb0 0x7f2d2006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:09.293 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:09.291+0000 7fde23f68240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:51:09.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.301+0000 7f2d45200700 1 -- 172.21.15.89:0/487628605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d40067720 con 0x7f2d400b3af0 2024-04-02T07:51:09.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.303+0000 7f2d36000700 1 -- 172.21.15.89:0/487628605 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+17458 (secure 0 0 0) 0x7f2d38057a40 con 0x7f2d400b3af0 2024-04-02T07:51:09.306 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:09.306 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":70,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:51:07.016234+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:49:05.141462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-02T07:49:06.232906+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.9","mappings":[{"from":5,"to":0}]},{"pgid":"7.14","mappings":[{"from":5,"to":7}]},{"pgid":"7.15","mappings":[{"from":5,"to":7}]},{"pgid":"7.30","mappings":[{"from":5,"to":7}]},{"pgid":"7.35","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/379543030":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/3951589296":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/3713107589":"2024-04-03T07:49:23.665126+0000","172.21.15.89:6833/43357":"2024-04-03T07:49:23.665126+0000","172.21.15.89:6832/43357":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/552943692":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/2589801503":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/3028439559":"2024-04-03T07:49:23.665126+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:51:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.305+0000 7f2d45200700 1 -- 172.21.15.89:0/487628605 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2d2006adb0 msgr2=0x7f2d2006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.305+0000 7f2d45200700 1 --2- 172.21.15.89:0/487628605 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2d2006adb0 0x7f2d2006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.305+0000 7f2d45200700 1 -- 172.21.15.89:0/487628605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d400b3af0 msgr2=0x7f2d4014efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.305+0000 7f2d45200700 1 --2- 172.21.15.89:0/487628605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d400b3af0 0x7f2d4014efb0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f2d3800caf0 tx=0x7f2d3800ce00 comp rx=0 tx=0).stop 2024-04-02T07:51:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.305+0000 7f2d45200700 1 -- 172.21.15.89:0/487628605 shutdown_connections 2024-04-02T07:51:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.305+0000 7f2d45200700 1 --2- 172.21.15.89:0/487628605 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2d2006adb0 0x7f2d2006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.305+0000 7f2d45200700 1 --2- 172.21.15.89:0/487628605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d40072580 0x7f2d40151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.305+0000 7f2d45200700 1 --2- 172.21.15.89:0/487628605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d400b3af0 0x7f2d4014efb0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.305+0000 7f2d45200700 1 --2- 172.21.15.89:0/487628605 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d400b2e50 0x7f2d4015d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.305+0000 7f2d45200700 1 -- 172.21.15.89:0/487628605 >> 172.21.15.89:0/487628605 conn(0x7f2d400cf190 msgr2=0x7f2d400cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:09.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.305+0000 7f2d45200700 1 -- 172.21.15.89:0/487628605 shutdown_connections 2024-04-02T07:51:09.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.305+0000 7f2d45200700 1 -- 172.21.15.89:0/487628605 wait complete. 2024-04-02T07:51:09.326 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T07:51:09.326 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:51:09.516 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:09.514+0000 7fde23f68240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:51:09.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.517+0000 7f39c0400700 1 Processor -- start 2024-04-02T07:51:09.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.517+0000 7f39c0400700 1 -- start start 2024-04-02T07:51:09.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.518+0000 7f39c0400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f39b80d18b0 0x7f39b80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:09.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.518+0000 7f39c0400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b80ceda0 con 0x7f39b8161590 2024-04-02T07:51:09.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.518+0000 7f39c0400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b81612f0 con 0x7f39b8160f10 2024-04-02T07:51:09.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.518+0000 7f39c0400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b80d1d30 con 0x7f39b80d18b0 2024-04-02T07:51:09.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.519+0000 7f39bd000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f39b80d18b0 0x7f39b80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:09.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.519+0000 7f39bd000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f39b80d18b0 0x7f39b80cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34262/0 (socket says 172.21.15.89:34262) 2024-04-02T07:51:09.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.519+0000 7f39be400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f39b8161590 0x7f39b80b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33004/0 (socket says 172.21.15.89:33004) 2024-04-02T07:51:09.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.519+0000 7f39bd000700 1 -- 172.21.15.89:0/2727153430 learned_addr learned my addr 172.21.15.89:0/2727153430 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:09.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.519+0000 7f39bd000700 1 -- 172.21.15.89:0/2727153430 >> v1:172.21.15.113:6789/0 conn(0x7f39b8160f10 legacy=0x7f39b80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:09.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.519+0000 7f39bd000700 1 -- 172.21.15.89:0/2727153430 >> v1:172.21.15.89:6789/0 conn(0x7f39b8161590 legacy=0x7f39b80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:09.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.519+0000 7f39bd000700 1 -- 172.21.15.89:0/2727153430 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80ce470 con 0x7f39b80d18b0 2024-04-02T07:51:09.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.519+0000 7f39bd000700 1 --2- 172.21.15.89:0/2727153430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80d18b0 0x7f39b80cf230 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f39a40098f0 tx=0x7f39a4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff2e628e7ffe20d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:09.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.520+0000 7f39b7e00700 1 -- 172.21.15.89:0/2727153430 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a400b8a0 con 0x7f39b80d18b0 2024-04-02T07:51:09.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.520+0000 7f39b7e00700 1 -- 172.21.15.89:0/2727153430 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a400ba00 con 0x7f39b80d18b0 2024-04-02T07:51:09.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.520+0000 7f39b7e00700 1 -- 172.21.15.89:0/2727153430 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a400bd90 con 0x7f39b80d18b0 2024-04-02T07:51:09.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.520+0000 7f39c0400700 1 -- 172.21.15.89:0/2727153430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80d18b0 msgr2=0x7f39b80cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:09.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.520+0000 7f39c0400700 1 --2- 172.21.15.89:0/2727153430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80d18b0 0x7f39b80cf230 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f39a40098f0 tx=0x7f39a4009c00 comp rx=0 tx=0).stop 2024-04-02T07:51:09.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.520+0000 7f39c0400700 1 -- 172.21.15.89:0/2727153430 shutdown_connections 2024-04-02T07:51:09.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.520+0000 7f39c0400700 1 --2- 172.21.15.89:0/2727153430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80d18b0 0x7f39b80cf230 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.520+0000 7f39c0400700 1 -- 172.21.15.89:0/2727153430 >> 172.21.15.89:0/2727153430 conn(0x7f39b808aff0 msgr2=0x7f39b80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:09.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.520+0000 7f39c0400700 1 -- 172.21.15.89:0/2727153430 shutdown_connections 2024-04-02T07:51:09.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.520+0000 7f39c0400700 1 -- 172.21.15.89:0/2727153430 wait complete. 2024-04-02T07:51:09.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.522+0000 7f39c0400700 1 Processor -- start 2024-04-02T07:51:09.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.522+0000 7f39c0400700 1 -- start start 2024-04-02T07:51:09.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.523+0000 7f39c0400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80d18b0 0x7f39b80b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:09.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.523+0000 7f39c0400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39b8160f10 0x7f39b81f67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:09.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.523+0000 7f39c0400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39b8161590 0x7f39b81596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:09.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.523+0000 7f39c0400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80b86c0 con 0x7f39b80d18b0 2024-04-02T07:51:09.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.523+0000 7f39c0400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80b9090 con 0x7f39b8161590 2024-04-02T07:51:09.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.523+0000 7f39c0400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80b7ab0 con 0x7f39b8160f10 2024-04-02T07:51:09.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.523+0000 7f39bda00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80d18b0 0x7f39b80b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:09.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.523+0000 7f39bda00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80d18b0 0x7f39b80b8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34266/0 (socket says 172.21.15.89:34266) 2024-04-02T07:51:09.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.523+0000 7f39bda00700 1 -- 172.21.15.89:0/4045051804 learned_addr learned my addr 172.21.15.89:0/4045051804 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:09.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.523+0000 7f39be400700 1 --2- 172.21.15.89:0/4045051804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39b8161590 0x7f39b81596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:09.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.524+0000 7f39bd000700 1 --2- 172.21.15.89:0/4045051804 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39b8160f10 0x7f39b81f67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:09.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.524+0000 7f39bda00700 1 -- 172.21.15.89:0/4045051804 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39b8160f10 msgr2=0x7f39b81f67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:09.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.524+0000 7f39bda00700 1 --2- 172.21.15.89:0/4045051804 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39b8160f10 0x7f39b81f67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.524+0000 7f39bda00700 1 -- 172.21.15.89:0/4045051804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39b8161590 msgr2=0x7f39b81596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:09.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.524+0000 7f39bda00700 1 --2- 172.21.15.89:0/4045051804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39b8161590 0x7f39b81596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.524+0000 7f39bda00700 1 -- 172.21.15.89:0/4045051804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a4009550 con 0x7f39b80d18b0 2024-04-02T07:51:09.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.524+0000 7f39bda00700 1 --2- 172.21.15.89:0/4045051804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80d18b0 0x7f39b80b8b50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00c450 tx=0x7f39ac00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:09.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.524+0000 7f39b5600700 1 -- 172.21.15.89:0/4045051804 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac0052d0 con 0x7f39b80d18b0 2024-04-02T07:51:09.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.524+0000 7f39b5600700 1 -- 172.21.15.89:0/4045051804 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ac005430 con 0x7f39b80d18b0 2024-04-02T07:51:09.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.524+0000 7f39c0400700 1 -- 172.21.15.89:0/4045051804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b80b7d90 con 0x7f39b80d18b0 2024-04-02T07:51:09.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.524+0000 7f39b5600700 1 -- 172.21.15.89:0/4045051804 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac0038c0 con 0x7f39b80d18b0 2024-04-02T07:51:09.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.524+0000 7f39c0400700 1 -- 172.21.15.89:0/4045051804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b80b8350 con 0x7f39b80d18b0 2024-04-02T07:51:09.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.525+0000 7f39c0400700 1 -- 172.21.15.89:0/4045051804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b80b89a0 con 0x7f39b80d18b0 2024-04-02T07:51:09.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.526+0000 7f39b5600700 1 -- 172.21.15.89:0/4045051804 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f39ac003a20 con 0x7f39b80d18b0 2024-04-02T07:51:09.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.527+0000 7f39b5600700 1 --2- 172.21.15.89:0/4045051804 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 0x7f399806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:09.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.527+0000 7f39bd000700 1 -- 172.21.15.89:0/4045051804 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 msgr2=0x7f399806d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:09.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.527+0000 7f39b5600700 1 -- 172.21.15.89:0/4045051804 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6602+0+0 (secure 0 0 0) 0x7f39ac088bd0 con 0x7f39b80d18b0 2024-04-02T07:51:09.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.527+0000 7f39bd000700 1 --2- 172.21.15.89:0/4045051804 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 0x7f399806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:09.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.531+0000 7f39b5600700 1 -- 172.21.15.89:0/4045051804 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f39ac053890 con 0x7f39b80d18b0 2024-04-02T07:51:09.541 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:09.539+0000 7ff1aa269240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:51:09.649 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:09.647+0000 7ff1aa269240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:51:09.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.728+0000 7f39bd000700 1 -- 172.21.15.89:0/4045051804 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 msgr2=0x7f399806d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:09.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.728+0000 7f39bd000700 1 --2- 172.21.15.89:0/4045051804 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 0x7f399806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:09.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.787+0000 7f39c0400700 1 -- 172.21.15.89:0/4045051804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39b80cf740 con 0x7f39b80d18b0 2024-04-02T07:51:09.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.787+0000 7f39b5600700 1 -- 172.21.15.89:0/4045051804 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f39ac017020 con 0x7f39b80d18b0 2024-04-02T07:51:09.790 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:09.790 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5710,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3098065397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3098065397},{"type":"v1","addr":"172.21.15.89:6836","nonce":3098065397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5713,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1408411829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1408411829},{"type":"v1","addr":"172.21.15.89:6837","nonce":1408411829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5716,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3266955664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3266955664},{"type":"v1","addr":"172.21.15.113:6834","nonce":3266955664}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:49:07.249043+0000","modified":"2024-04-02T07:49:10.254959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5719":{"gid":5719,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6835/427074923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":427074923},{"type":"v1","addr":"172.21.15.113:6835","nonce":427074923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-02T07:51:09.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.790+0000 7f39c0400700 1 -- 172.21.15.89:0/4045051804 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 msgr2=0x7f399806d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:09.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.790+0000 7f39c0400700 1 --2- 172.21.15.89:0/4045051804 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 0x7f399806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.790+0000 7f39c0400700 1 -- 172.21.15.89:0/4045051804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80d18b0 msgr2=0x7f39b80b8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:09.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.790+0000 7f39c0400700 1 --2- 172.21.15.89:0/4045051804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80d18b0 0x7f39b80b8b50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00c450 tx=0x7f39ac00c760 comp rx=0 tx=0).stop 2024-04-02T07:51:09.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.790+0000 7f39c0400700 1 -- 172.21.15.89:0/4045051804 shutdown_connections 2024-04-02T07:51:09.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.790+0000 7f39c0400700 1 --2- 172.21.15.89:0/4045051804 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 0x7f399806d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.790+0000 7f39c0400700 1 --2- 172.21.15.89:0/4045051804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39b8161590 0x7f39b81596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.790+0000 7f39c0400700 1 --2- 172.21.15.89:0/4045051804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80d18b0 0x7f39b80b8b50 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.790+0000 7f39c0400700 1 --2- 172.21.15.89:0/4045051804 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39b8160f10 0x7f39b81f67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:09.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.790+0000 7f39c0400700 1 -- 172.21.15.89:0/4045051804 >> 172.21.15.89:0/4045051804 conn(0x7f39b808aff0 msgr2=0x7f39b8160230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:09.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.790+0000 7f39c0400700 1 -- 172.21.15.89:0/4045051804 shutdown_connections 2024-04-02T07:51:09.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:09.790+0000 7f39c0400700 1 -- 172.21.15.89:0/4045051804 wait complete. 2024-04-02T07:51:09.794 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 24 2024-04-02T07:51:09.813 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:51:09.979 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:09.977+0000 7fde23f68240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:51:10.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.008+0000 7f518a800700 1 Processor -- start 2024-04-02T07:51:10.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.009+0000 7f518a800700 1 -- start start 2024-04-02T07:51:10.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.010+0000 7f518a800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f518415b5d0 0x7f51840ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.010+0000 7f518a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51840cb3f0 0x7f51840ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.010+0000 7f518a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51840ae930 0x7f51840ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.010+0000 7f518a800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f518415b9b0 con 0x7f51840cb3f0 2024-04-02T07:51:10.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.010+0000 7f518a800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f518415fd50 con 0x7f518415b5d0 2024-04-02T07:51:10.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.010+0000 7f518a800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518415f140 con 0x7f51840ae930 2024-04-02T07:51:10.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.010+0000 7f5182a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51840cb3f0 0x7f51840ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.010+0000 7f5182a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51840cb3f0 0x7f51840ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34268/0 (socket says 172.21.15.89:34268) 2024-04-02T07:51:10.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.010+0000 7f5182a00700 1 -- 172.21.15.89:0/2267919051 learned_addr learned my addr 172.21.15.89:0/2267919051 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:10.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.010+0000 7f5183400700 1 --2- 172.21.15.89:0/2267919051 >> v2:172.21.15.113:3300/0 conn(0x7f518415b5d0 0x7f51840ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.010+0000 7f5183e00700 1 --2- 172.21.15.89:0/2267919051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51840ae930 0x7f51840ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.011+0000 7f5183400700 1 -- 172.21.15.89:0/2267919051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51840ae930 msgr2=0x7f51840ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.011+0000 7f5183400700 1 --2- 172.21.15.89:0/2267919051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51840ae930 0x7f51840ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.011+0000 7f5183400700 1 -- 172.21.15.89:0/2267919051 >> v2:172.21.15.89:3300/0 conn(0x7f51840cb3f0 msgr2=0x7f51840ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.011+0000 7f5183400700 1 --2- 172.21.15.89:0/2267919051 >> v2:172.21.15.89:3300/0 conn(0x7f51840cb3f0 0x7f51840ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.011+0000 7f5183400700 1 -- 172.21.15.89:0/2267919051 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518415f420 con 0x7f518415b5d0 2024-04-02T07:51:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.011+0000 7f5182a00700 1 --2- 172.21.15.89:0/2267919051 >> v2:172.21.15.89:3300/0 conn(0x7f51840cb3f0 0x7f51840ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.011+0000 7f5183400700 1 --2- 172.21.15.89:0/2267919051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f518415b5d0 0x7f51840ae3f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f518406b8b0 tx=0x7f5174009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7385c41558036d75 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.011+0000 7f5182000700 1 -- 172.21.15.89:0/2267919051 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5174011830 con 0x7f518415b5d0 2024-04-02T07:51:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.011+0000 7f5182000700 1 -- 172.21.15.89:0/2267919051 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5174011990 con 0x7f518415b5d0 2024-04-02T07:51:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.011+0000 7f5182000700 1 -- 172.21.15.89:0/2267919051 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5174011d20 con 0x7f518415b5d0 2024-04-02T07:51:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.012+0000 7f518a800700 1 -- 172.21.15.89:0/2267919051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f518415b5d0 msgr2=0x7f51840ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.012+0000 7f518a800700 1 --2- 172.21.15.89:0/2267919051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f518415b5d0 0x7f51840ae3f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f518406b8b0 tx=0x7f5174009b20 comp rx=0 tx=0).stop 2024-04-02T07:51:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.012+0000 7f518a800700 1 -- 172.21.15.89:0/2267919051 shutdown_connections 2024-04-02T07:51:10.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.012+0000 7f518a800700 1 --2- 172.21.15.89:0/2267919051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51840ae930 0x7f51840ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.012+0000 7f518a800700 1 --2- 172.21.15.89:0/2267919051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f518415b5d0 0x7f51840ae3f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.012+0000 7f518a800700 1 --2- 172.21.15.89:0/2267919051 >> v2:172.21.15.89:3300/0 conn(0x7f51840cb3f0 0x7f51840ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.012+0000 7f518a800700 1 -- 172.21.15.89:0/2267919051 >> 172.21.15.89:0/2267919051 conn(0x7f51840cf190 msgr2=0x7f51840cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.012+0000 7f518a800700 1 -- 172.21.15.89:0/2267919051 shutdown_connections 2024-04-02T07:51:10.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.012+0000 7f518a800700 1 -- 172.21.15.89:0/2267919051 wait complete. 2024-04-02T07:51:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.014+0000 7f518a800700 1 Processor -- start 2024-04-02T07:51:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.014+0000 7f518a800700 1 -- start start 2024-04-02T07:51:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f518a800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51840ae930 0x7f5184152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f518a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f518415b5d0 0x7f51840b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f518a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51840b3460 0x7f51840c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f518a800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51840b3910 con 0x7f51840b3460 2024-04-02T07:51:10.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f518a800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5184148180 con 0x7f51840ae930 2024-04-02T07:51:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f518a800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5184152840 con 0x7f518415b5d0 2024-04-02T07:51:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f5183e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51840b3460 0x7f51840c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f5183e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51840b3460 0x7f51840c9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34280/0 (socket says 172.21.15.89:34280) 2024-04-02T07:51:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f5183e00700 1 -- 172.21.15.89:0/1475441044 learned_addr learned my addr 172.21.15.89:0/1475441044 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f5183400700 1 --2- 172.21.15.89:0/1475441044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51840ae930 0x7f5184152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f5182a00700 1 --2- 172.21.15.89:0/1475441044 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f518415b5d0 0x7f51840b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f5183e00700 1 -- 172.21.15.89:0/1475441044 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f518415b5d0 msgr2=0x7f51840b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f5183e00700 1 --2- 172.21.15.89:0/1475441044 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f518415b5d0 0x7f51840b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f5183e00700 1 -- 172.21.15.89:0/1475441044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51840ae930 msgr2=0x7f5184152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f5183e00700 1 --2- 172.21.15.89:0/1475441044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51840ae930 0x7f5184152300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.015+0000 7f5183e00700 1 -- 172.21.15.89:0/1475441044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5170009550 con 0x7f51840b3460 2024-04-02T07:51:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.016+0000 7f5183e00700 1 --2- 172.21.15.89:0/1475441044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51840b3460 0x7f51840c9380 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f517c00ef30 tx=0x7f517c00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:10.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.016+0000 7f517b400700 1 -- 172.21.15.89:0/1475441044 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c010040 con 0x7f51840b3460 2024-04-02T07:51:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.016+0000 7f517b400700 1 -- 172.21.15.89:0/1475441044 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517c00bca0 con 0x7f51840b3460 2024-04-02T07:51:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.016+0000 7f518a800700 1 -- 172.21.15.89:0/1475441044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51740094a0 con 0x7f51840b3460 2024-04-02T07:51:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.016+0000 7f517b400700 1 -- 172.21.15.89:0/1475441044 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c009ea0 con 0x7f51840b3460 2024-04-02T07:51:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.016+0000 7f518a800700 1 -- 172.21.15.89:0/1475441044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51840ca090 con 0x7f51840b3460 2024-04-02T07:51:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.017+0000 7f518a800700 1 -- 172.21.15.89:0/1475441044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51840cba10 con 0x7f51840b3460 2024-04-02T07:51:10.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.018+0000 7f517b400700 1 -- 172.21.15.89:0/1475441044 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 28) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f517c014640 con 0x7f51840b3460 2024-04-02T07:51:10.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.019+0000 7f517b400700 1 --2- 172.21.15.89:0/1475441044 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f516406adb0 0x7f516406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.019+0000 7f517b400700 1 -- 172.21.15.89:0/1475441044 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6602+0+0 (secure 0 0 0) 0x7f517c089fe0 con 0x7f51840b3460 2024-04-02T07:51:10.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.019+0000 7f5183400700 1 -- 172.21.15.89:0/1475441044 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f516406adb0 msgr2=0x7f516406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:10.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.019+0000 7f5183400700 1 --2- 172.21.15.89:0/1475441044 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f516406adb0 0x7f516406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:10.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.023+0000 7f517b400700 1 -- 172.21.15.89:0/1475441044 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f517c054c10 con 0x7f51840b3460 2024-04-02T07:51:10.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.042+0000 7f517b400700 1 -- 172.21.15.89:0/1475441044 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 29) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f517c054fe0 con 0x7f51840b3460 2024-04-02T07:51:10.102 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:10.099+0000 7fde23f68240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:51:10.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.108+0000 7f517b400700 1 -- 172.21.15.89:0/1475441044 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 30) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f517c060830 con 0x7f51840b3460 2024-04-02T07:51:10.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.109+0000 7f517b400700 1 -- 172.21.15.89:0/1475441044 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f516406adb0 msgr2=0x7f516406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:10.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.109+0000 7f517b400700 1 --2- 172.21.15.89:0/1475441044 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f516406adb0 0x7f516406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.182 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:10.179+0000 7fdd91600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:10.182 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:10.179+0000 7fdd91600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:10.183 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:10.179+0000 7fdd97000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:10.183 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:10.179+0000 7fdd97000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:10.183 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:10.179+0000 7fdd97000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:10.183 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:10.179+0000 7fdd97000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:10.183 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:10.179+0000 7fdd97000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:10.183 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:10.179+0000 7fdd91600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:10.183 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:10.179+0000 7fdd91600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:10.183 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:10.179+0000 7fdd91600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.298+0000 7f518a800700 1 -- 172.21.15.89:0/1475441044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51840adb70 con 0x7f51840b3460 2024-04-02T07:51:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.298+0000 7f517b400700 1 -- 172.21.15.89:0/1475441044 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f517c081e60 con 0x7f51840b3460 2024-04-02T07:51:10.301 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:10.301 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5710,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3098065397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3098065397},{"type":"v1","addr":"172.21.15.89:6836","nonce":3098065397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5713,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1408411829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1408411829},{"type":"v1","addr":"172.21.15.89:6837","nonce":1408411829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5716,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3266955664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3266955664},{"type":"v1","addr":"172.21.15.113:6834","nonce":3266955664}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:49:07.249043+0000","modified":"2024-04-02T07:49:10.254959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5719":{"gid":5719,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6835/427074923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":427074923},{"type":"v1","addr":"172.21.15.113:6835","nonce":427074923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-02T07:51:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.300+0000 7f518a800700 1 -- 172.21.15.89:0/1475441044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51840b3460 msgr2=0x7f51840c9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.300+0000 7f518a800700 1 --2- 172.21.15.89:0/1475441044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51840b3460 0x7f51840c9380 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f517c00ef30 tx=0x7f517c00b390 comp rx=0 tx=0).stop 2024-04-02T07:51:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.300+0000 7f518a800700 1 -- 172.21.15.89:0/1475441044 shutdown_connections 2024-04-02T07:51:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.300+0000 7f518a800700 1 --2- 172.21.15.89:0/1475441044 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f516406adb0 0x7f516406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.300+0000 7f518a800700 1 --2- 172.21.15.89:0/1475441044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51840ae930 0x7f5184152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.300+0000 7f518a800700 1 --2- 172.21.15.89:0/1475441044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51840b3460 0x7f51840c9380 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.300+0000 7f518a800700 1 --2- 172.21.15.89:0/1475441044 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f518415b5d0 0x7f51840b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.300+0000 7f518a800700 1 -- 172.21.15.89:0/1475441044 >> 172.21.15.89:0/1475441044 conn(0x7f51840cf190 msgr2=0x7f518415cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.301+0000 7f518a800700 1 -- 172.21.15.89:0/1475441044 shutdown_connections 2024-04-02T07:51:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.301+0000 7f518a800700 1 -- 172.21.15.89:0/1475441044 wait complete. 2024-04-02T07:51:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 24 2024-04-02T07:51:10.322 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:51:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.510+0000 7f7757400700 1 Processor -- start 2024-04-02T07:51:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.510+0000 7f7757400700 1 -- start start 2024-04-02T07:51:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.511+0000 7f7757400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f775006a790 0x7f775006ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.511+0000 7f7757400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f77500afbe0 0x7f775006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.511+0000 7f7757400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775006f850 con 0x7f77500cebd0 2024-04-02T07:51:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.511+0000 7f7757400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77500cf6f0 con 0x7f77500afbe0 2024-04-02T07:51:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.511+0000 7f7757400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f775006b0b0 con 0x7f775006a790 2024-04-02T07:51:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.512+0000 7f774fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f77500afbe0 0x7f775006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.512+0000 7f774fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f77500afbe0 0x7f775006dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34394/0 (socket says 172.21.15.89:34394) 2024-04-02T07:51:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.512+0000 7f774fe00700 1 -- 172.21.15.89:0/1399883075 learned_addr learned my addr 172.21.15.89:0/1399883075 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.512+0000 7f7754a00700 1 --2- 172.21.15.89:0/1399883075 >> v2:172.21.15.113:3300/0 conn(0x7f775006a790 0x7f775006ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.512+0000 7f774fe00700 1 -- 172.21.15.89:0/1399883075 >> v2:172.21.15.113:3300/0 conn(0x7f775006a790 msgr2=0x7f775006ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.512+0000 7f774fe00700 1 --2- 172.21.15.89:0/1399883075 >> v2:172.21.15.113:3300/0 conn(0x7f775006a790 0x7f775006ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.512+0000 7f774fe00700 1 -- 172.21.15.89:0/1399883075 >> v1:172.21.15.113:6789/0 conn(0x7f77500cebd0 legacy=0x7f77500cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.512+0000 7f774fe00700 1 -- 172.21.15.89:0/1399883075 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775006fad0 con 0x7f77500afbe0 2024-04-02T07:51:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.512+0000 7f774fe00700 1 --2- 172.21.15.89:0/1399883075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77500afbe0 0x7f775006dce0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f775006d9a0 tx=0x7f773c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dffb9a2ffea4981d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.513+0000 7f774f400700 1 -- 172.21.15.89:0/1399883075 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c004230 con 0x7f77500afbe0 2024-04-02T07:51:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.513+0000 7f774f400700 1 -- 172.21.15.89:0/1399883075 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773c004390 con 0x7f77500afbe0 2024-04-02T07:51:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.513+0000 7f774f400700 1 -- 172.21.15.89:0/1399883075 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c0036a0 con 0x7f77500afbe0 2024-04-02T07:51:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.513+0000 7f7757400700 1 -- 172.21.15.89:0/1399883075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77500afbe0 msgr2=0x7f775006dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.513+0000 7f7757400700 1 --2- 172.21.15.89:0/1399883075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77500afbe0 0x7f775006dce0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f775006d9a0 tx=0x7f773c009bd0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.513+0000 7f7757400700 1 -- 172.21.15.89:0/1399883075 shutdown_connections 2024-04-02T07:51:10.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.513+0000 7f7757400700 1 --2- 172.21.15.89:0/1399883075 >> v2:172.21.15.113:3300/0 conn(0x7f775006a790 0x7f775006ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.513+0000 7f7757400700 1 --2- 172.21.15.89:0/1399883075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77500afbe0 0x7f775006dce0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.513+0000 7f7757400700 1 -- 172.21.15.89:0/1399883075 >> 172.21.15.89:0/1399883075 conn(0x7f77500b5ab0 msgr2=0x7f77500b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:10.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.513+0000 7f7757400700 1 -- 172.21.15.89:0/1399883075 shutdown_connections 2024-04-02T07:51:10.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.513+0000 7f7757400700 1 -- 172.21.15.89:0/1399883075 wait complete. 2024-04-02T07:51:10.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.515+0000 7f7757400700 1 Processor -- start 2024-04-02T07:51:10.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.515+0000 7f7757400700 1 -- start start 2024-04-02T07:51:10.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.516+0000 7f7757400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f775006a790 0x7f7750149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.516+0000 7f7757400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77500afbe0 0x7f77500cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.516+0000 7f7757400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77500cebd0 0x7f77500cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.516+0000 7f7757400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775009e540 con 0x7f77500afbe0 2024-04-02T07:51:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.516+0000 7f7757400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77500b4070 con 0x7f77500cebd0 2024-04-02T07:51:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.516+0000 7f7757400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77500b4350 con 0x7f775006a790 2024-04-02T07:51:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.516+0000 7f774fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77500afbe0 0x7f77500cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.516+0000 7f774fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77500afbe0 0x7f77500cbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34410/0 (socket says 172.21.15.89:34410) 2024-04-02T07:51:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.516+0000 7f774fe00700 1 -- 172.21.15.89:0/2978810931 learned_addr learned my addr 172.21.15.89:0/2978810931 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.517+0000 7f7754a00700 1 --2- 172.21.15.89:0/2978810931 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f775006a790 0x7f7750149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.517+0000 7f7755400700 1 --2- 172.21.15.89:0/2978810931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77500cebd0 0x7f77500cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.517+0000 7f774fe00700 1 -- 172.21.15.89:0/2978810931 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f775006a790 msgr2=0x7f7750149340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.517+0000 7f774fe00700 1 --2- 172.21.15.89:0/2978810931 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f775006a790 0x7f7750149340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.517+0000 7f774fe00700 1 -- 172.21.15.89:0/2978810931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77500cebd0 msgr2=0x7f77500cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.517+0000 7f774fe00700 1 --2- 172.21.15.89:0/2978810931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77500cebd0 0x7f77500cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.517+0000 7f774fe00700 1 -- 172.21.15.89:0/2978810931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c009550 con 0x7f77500afbe0 2024-04-02T07:51:10.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.517+0000 7f774fe00700 1 --2- 172.21.15.89:0/2978810931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77500afbe0 0x7f77500cbd60 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f773c009130 tx=0x7f773c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:10.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.517+0000 7f774cc00700 1 -- 172.21.15.89:0/2978810931 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c003b90 con 0x7f77500afbe0 2024-04-02T07:51:10.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.517+0000 7f774cc00700 1 -- 172.21.15.89:0/2978810931 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773c01f070 con 0x7f77500afbe0 2024-04-02T07:51:10.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.517+0000 7f7757400700 1 -- 172.21.15.89:0/2978810931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77500b4630 con 0x7f77500afbe0 2024-04-02T07:51:10.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.517+0000 7f774cc00700 1 -- 172.21.15.89:0/2978810931 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c011630 con 0x7f77500afbe0 2024-04-02T07:51:10.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.518+0000 7f7757400700 1 -- 172.21.15.89:0/2978810931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7750149880 con 0x7f77500afbe0 2024-04-02T07:51:10.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.519+0000 7f7757400700 1 -- 172.21.15.89:0/2978810931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7750149ad0 con 0x7f77500afbe0 2024-04-02T07:51:10.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.519+0000 7f774cc00700 1 -- 172.21.15.89:0/2978810931 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 30) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f773c021030 con 0x7f77500afbe0 2024-04-02T07:51:10.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.521+0000 7f774cc00700 1 -- 172.21.15.89:0/2978810931 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6832+0+0 (secure 0 0 0) 0x7f773c08a670 con 0x7f77500afbe0 2024-04-02T07:51:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.526+0000 7f774cc00700 1 -- 172.21.15.89:0/2978810931 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7750149ad0 con 0x7f77500afbe0 2024-04-02T07:51:10.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.725+0000 7f7757400700 1 -- 172.21.15.89:0/2978810931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7750149ad0 con 0x7f77500afbe0 2024-04-02T07:51:10.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.726+0000 7f774cc00700 1 -- 172.21.15.89:0/2978810931 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+17787 (secure 0 0 0) 0x7f7750149ad0 con 0x7f77500afbe0 2024-04-02T07:51:10.729 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:10.729 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":71,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:51:10.103182+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:49:05.141462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-02T07:49:06.232906+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.9","mappings":[{"from":5,"to":0}]},{"pgid":"7.14","mappings":[{"from":5,"to":7}]},{"pgid":"7.15","mappings":[{"from":5,"to":7}]},{"pgid":"7.30","mappings":[{"from":5,"to":7}]},{"pgid":"7.35","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/2076776472":"2024-04-03T07:51:10.103145+0000","172.21.15.89:0/1227485517":"2024-04-03T07:51:10.103145+0000","172.21.15.89:0/3841523471":"2024-04-03T07:51:10.103145+0000","172.21.15.89:0/379543030":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/3951589296":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/3031662055":"2024-04-03T07:51:10.103145+0000","172.21.15.89:0/3713107589":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/2218077449":"2024-04-03T07:51:10.103145+0000","172.21.15.89:6833/43357":"2024-04-03T07:49:23.665126+0000","172.21.15.89:6832/43357":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/552943692":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/2589801503":"2024-04-03T07:49:23.665126+0000","172.21.15.89:0/2939778340":"2024-04-03T07:51:10.103145+0000","172.21.15.89:0/3028439559":"2024-04-03T07:49:23.665126+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:51:10.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.728+0000 7f7757400700 1 -- 172.21.15.89:0/2978810931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77500afbe0 msgr2=0x7f77500cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:10.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.728+0000 7f7757400700 1 --2- 172.21.15.89:0/2978810931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77500afbe0 0x7f77500cbd60 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f773c009130 tx=0x7f773c004750 comp rx=0 tx=0).stop 2024-04-02T07:51:10.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.729+0000 7f7757400700 1 -- 172.21.15.89:0/2978810931 shutdown_connections 2024-04-02T07:51:10.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.729+0000 7f7757400700 1 --2- 172.21.15.89:0/2978810931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77500cebd0 0x7f77500cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.729+0000 7f7757400700 1 --2- 172.21.15.89:0/2978810931 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f775006a790 0x7f7750149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.729+0000 7f7757400700 1 --2- 172.21.15.89:0/2978810931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77500afbe0 0x7f77500cbd60 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.729+0000 7f7757400700 1 -- 172.21.15.89:0/2978810931 >> 172.21.15.89:0/2978810931 conn(0x7f77500b5ab0 msgr2=0x7f77500adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:10.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.729+0000 7f7757400700 1 -- 172.21.15.89:0/2978810931 shutdown_connections 2024-04-02T07:51:10.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.729+0000 7f7757400700 1 -- 172.21.15.89:0/2978810931 wait complete. 2024-04-02T07:51:10.749 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T07:51:10.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.938+0000 7f4a2c400700 1 Processor -- start 2024-04-02T07:51:10.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.938+0000 7f4a2c400700 1 -- start start 2024-04-02T07:51:10.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.939+0000 7f4a2c400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4a240d16a0 0x7f4a240d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.940+0000 7f4a2c400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a24161420 con 0x7f4a240d2370 2024-04-02T07:51:10.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.940+0000 7f4a2c400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a24161700 con 0x7f4a2415d9d0 2024-04-02T07:51:10.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.940+0000 7f4a2c400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a240d4340 con 0x7f4a240d16a0 2024-04-02T07:51:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.940+0000 7f4a29000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4a240d16a0 0x7f4a240d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.940+0000 7f4a2a400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4a240d2370 0x7f4a240ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33014/0 (socket says 172.21.15.89:33014) 2024-04-02T07:51:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.940+0000 7f4a2a400700 1 -- 172.21.15.89:0/722629906 learned_addr learned my addr 172.21.15.89:0/722629906 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.940+0000 7f4a29000700 1 -- 172.21.15.89:0/722629906 >> v1:172.21.15.113:6789/0 conn(0x7f4a2415d9d0 legacy=0x7f4a240d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.940+0000 7f4a29000700 1 -- 172.21.15.89:0/722629906 >> v1:172.21.15.89:6789/0 conn(0x7f4a240d2370 legacy=0x7f4a240ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.940+0000 7f4a29000700 1 -- 172.21.15.89:0/722629906 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a241619e0 con 0x7f4a240d16a0 2024-04-02T07:51:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.940+0000 7f4a29000700 1 --2- 172.21.15.89:0/722629906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a240d16a0 0x7f4a240d1e30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009820 tx=0x7f4a10009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a7dc2c14318520 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.941+0000 7f4a23e00700 1 -- 172.21.15.89:0/722629906 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a10004230 con 0x7f4a240d16a0 2024-04-02T07:51:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.941+0000 7f4a23e00700 1 -- 172.21.15.89:0/722629906 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a10004390 con 0x7f4a240d16a0 2024-04-02T07:51:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.941+0000 7f4a23e00700 1 -- 172.21.15.89:0/722629906 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a100036a0 con 0x7f4a240d16a0 2024-04-02T07:51:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.941+0000 7f4a2c400700 1 -- 172.21.15.89:0/722629906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a240d16a0 msgr2=0x7f4a240d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.941+0000 7f4a2c400700 1 --2- 172.21.15.89:0/722629906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a240d16a0 0x7f4a240d1e30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009820 tx=0x7f4a10009b30 comp rx=0 tx=0).stop 2024-04-02T07:51:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.941+0000 7f4a2c400700 1 -- 172.21.15.89:0/722629906 shutdown_connections 2024-04-02T07:51:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.941+0000 7f4a2c400700 1 --2- 172.21.15.89:0/722629906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a240d16a0 0x7f4a240d1e30 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.941+0000 7f4a2c400700 1 -- 172.21.15.89:0/722629906 >> 172.21.15.89:0/722629906 conn(0x7f4a2408aff0 msgr2=0x7f4a240d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.941+0000 7f4a2c400700 1 -- 172.21.15.89:0/722629906 shutdown_connections 2024-04-02T07:51:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.941+0000 7f4a2c400700 1 -- 172.21.15.89:0/722629906 wait complete. 2024-04-02T07:51:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.943+0000 7f4a2c400700 1 Processor -- start 2024-04-02T07:51:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.943+0000 7f4a2c400700 1 -- start start 2024-04-02T07:51:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.944+0000 7f4a2c400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a240d16a0 0x7f4a24155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.944+0000 7f4a2c400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a240d2370 0x7f4a240b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.944+0000 7f4a2c400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a2415d9d0 0x7f4a240b4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.944+0000 7f4a2c400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2415ea30 con 0x7f4a240d2370 2024-04-02T07:51:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.944+0000 7f4a2c400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a240cc5d0 con 0x7f4a2415d9d0 2024-04-02T07:51:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.944+0000 7f4a2c400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a240cc8b0 con 0x7f4a240d16a0 2024-04-02T07:51:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.944+0000 7f4a29000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a240d2370 0x7f4a240b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.944+0000 7f4a29000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a240d2370 0x7f4a240b42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34438/0 (socket says 172.21.15.89:34438) 2024-04-02T07:51:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.944+0000 7f4a29000700 1 -- 172.21.15.89:0/944218359 learned_addr learned my addr 172.21.15.89:0/944218359 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.944+0000 7f4a29a00700 1 --2- 172.21.15.89:0/944218359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a240d16a0 0x7f4a24155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.944+0000 7f4a2a400700 1 --2- 172.21.15.89:0/944218359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a2415d9d0 0x7f4a240b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.945+0000 7f4a29000700 1 -- 172.21.15.89:0/944218359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a240d16a0 msgr2=0x7f4a24155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.945+0000 7f4a29000700 1 --2- 172.21.15.89:0/944218359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a240d16a0 0x7f4a24155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.945+0000 7f4a29000700 1 -- 172.21.15.89:0/944218359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a2415d9d0 msgr2=0x7f4a240b4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.945+0000 7f4a29000700 1 --2- 172.21.15.89:0/944218359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a2415d9d0 0x7f4a240b4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.945+0000 7f4a29000700 1 -- 172.21.15.89:0/944218359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a10009430 con 0x7f4a240d2370 2024-04-02T07:51:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.945+0000 7f4a29000700 1 --2- 172.21.15.89:0/944218359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a240d2370 0x7f4a240b42c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4a10004000 tx=0x7f4a10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.945+0000 7f4a21600700 1 -- 172.21.15.89:0/944218359 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a10011830 con 0x7f4a240d2370 2024-04-02T07:51:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.945+0000 7f4a21600700 1 -- 172.21.15.89:0/944218359 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a10011990 con 0x7f4a240d2370 2024-04-02T07:51:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.945+0000 7f4a2c400700 1 -- 172.21.15.89:0/944218359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a240ccb90 con 0x7f4a240d2370 2024-04-02T07:51:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.945+0000 7f4a21600700 1 -- 172.21.15.89:0/944218359 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a10011d20 con 0x7f4a240d2370 2024-04-02T07:51:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.945+0000 7f4a2c400700 1 -- 172.21.15.89:0/944218359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a240cd150 con 0x7f4a240d2370 2024-04-02T07:51:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.946+0000 7f4a2c400700 1 -- 172.21.15.89:0/944218359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a240d1ca0 con 0x7f4a240d2370 2024-04-02T07:51:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.947+0000 7f4a21600700 1 -- 172.21.15.89:0/944218359 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 30) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f4a10022070 con 0x7f4a240d2370 2024-04-02T07:51:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.949+0000 7f4a21600700 1 -- 172.21.15.89:0/944218359 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6832+0+0 (secure 0 0 0) 0x7f4a1008af40 con 0x7f4a240d2370 2024-04-02T07:51:10.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:10.954+0000 7f4a21600700 1 -- 172.21.15.89:0/944218359 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4a10056040 con 0x7f4a240d2370 2024-04-02T07:51:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.116+0000 7f4a21600700 1 -- 172.21.15.89:0/944218359 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 31) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f4a10055af0 con 0x7f4a240d2370 2024-04-02T07:51:11.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.116+0000 7f4a21600700 1 --2- 172.21.15.89:0/944218359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a04036150 0x7f4a040728e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:11.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.117+0000 7f4a29a00700 1 --2- 172.21.15.89:0/944218359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a04036150 0x7f4a040728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:11.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.122+0000 7f4a29a00700 1 --2- 172.21.15.89:0/944218359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a04036150 0x7f4a040728e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4a241f6980 tx=0x7f4a180090e0 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:11.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.208+0000 7f4a2c400700 1 -- 172.21.15.89:0/944218359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4a240cd2e0 con 0x7f4a240d2370 2024-04-02T07:51:11.211 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T07:51:11.209+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T07:51:11.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.221+0000 7f4a21600700 1 -- 172.21.15.89:0/944218359 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4a1006f780 con 0x7f4a240d2370 2024-04-02T07:51:11.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.223+0000 7f4a2c400700 1 -- 172.21.15.89:0/944218359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a04036150 msgr2=0x7f4a040728e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:11.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.223+0000 7f4a2c400700 1 --2- 172.21.15.89:0/944218359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a04036150 0x7f4a040728e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4a241f6980 tx=0x7f4a180090e0 comp rx=0 tx=0).stop 2024-04-02T07:51:11.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.223+0000 7f4a2c400700 1 -- 172.21.15.89:0/944218359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a240d2370 msgr2=0x7f4a240b42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:11.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.223+0000 7f4a2c400700 1 --2- 172.21.15.89:0/944218359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a240d2370 0x7f4a240b42c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4a10004000 tx=0x7f4a10004750 comp rx=0 tx=0).stop 2024-04-02T07:51:11.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.223+0000 7f4a2c400700 1 -- 172.21.15.89:0/944218359 shutdown_connections 2024-04-02T07:51:11.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.223+0000 7f4a2c400700 1 --2- 172.21.15.89:0/944218359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a04036150 0x7f4a040728e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:11.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.223+0000 7f4a2c400700 1 --2- 172.21.15.89:0/944218359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a2415d9d0 0x7f4a240b4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:11.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.223+0000 7f4a2c400700 1 --2- 172.21.15.89:0/944218359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a240d16a0 0x7f4a24155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:11.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.223+0000 7f4a2c400700 1 --2- 172.21.15.89:0/944218359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a240d2370 0x7f4a240b42c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:11.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.223+0000 7f4a2c400700 1 -- 172.21.15.89:0/944218359 >> 172.21.15.89:0/944218359 conn(0x7f4a2408aff0 msgr2=0x7f4a2406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:11.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.223+0000 7f4a2c400700 1 -- 172.21.15.89:0/944218359 shutdown_connections 2024-04-02T07:51:11.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.224+0000 7f4a2c400700 1 -- 172.21.15.89:0/944218359 wait complete. 2024-04-02T07:51:11.227 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T07:51:11.246 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T07:51:11.293 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T07:51:11.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.437+0000 7fa4fbe00700 1 Processor -- start 2024-04-02T07:51:11.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.437+0000 7fa4fbe00700 1 -- start start 2024-04-02T07:51:11.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.439+0000 7fa4fbe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa4f40d16a0 0x7fa4f40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:11.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.439+0000 7fa4fbe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4f40d2370 0x7fa4f40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:11.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.439+0000 7fa4fbe00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f4161420 con 0x7fa4f415d9d0 2024-04-02T07:51:11.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.439+0000 7fa4fbe00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f40d2820 con 0x7fa4f40d16a0 2024-04-02T07:51:11.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.439+0000 7fa4fbe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f40d4340 con 0x7fa4f40d2370 2024-04-02T07:51:11.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.439+0000 7fa4f8a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa4f40d16a0 0x7fa4f40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:11.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.439+0000 7fa4f9e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4f40d2370 0x7fa4f40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:11.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.439+0000 7fa4f9e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4f40d2370 0x7fa4f40ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:47642/0 (socket says 172.21.15.89:47642) 2024-04-02T07:51:11.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.439+0000 7fa4f9e00700 1 -- 172.21.15.89:0/2045694865 learned_addr learned my addr 172.21.15.89:0/2045694865 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:11.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.439+0000 7fa4f9e00700 1 -- 172.21.15.89:0/2045694865 >> v2:172.21.15.113:3300/0 conn(0x7fa4f40d16a0 msgr2=0x7fa4f40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:11.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.439+0000 7fa4f9e00700 1 --2- 172.21.15.89:0/2045694865 >> v2:172.21.15.113:3300/0 conn(0x7fa4f40d16a0 0x7fa4f40d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:11.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.439+0000 7fa4f9e00700 1 -- 172.21.15.89:0/2045694865 >> v1:172.21.15.113:6789/0 conn(0x7fa4f415d9d0 legacy=0x7fa4f40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:11.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.439+0000 7fa4f9e00700 1 -- 172.21.15.89:0/2045694865 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f4161700 con 0x7fa4f40d2370 2024-04-02T07:51:11.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.440+0000 7fa4f9e00700 1 --2- 172.21.15.89:0/2045694865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4f40d2370 0x7fa4f40ce360 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec009e50 tx=0x7fa4ec00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea5633eb59804715 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:11.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.440+0000 7fa4f3e00700 1 -- 172.21.15.89:0/2045694865 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec00e070 con 0x7fa4f40d2370 2024-04-02T07:51:11.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.440+0000 7fa4f3e00700 1 -- 172.21.15.89:0/2045694865 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ec00b910 con 0x7fa4f40d2370 2024-04-02T07:51:11.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.440+0000 7fa4f3e00700 1 -- 172.21.15.89:0/2045694865 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec00bcc0 con 0x7fa4f40d2370 2024-04-02T07:51:11.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.440+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2045694865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4f40d2370 msgr2=0x7fa4f40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:11.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.441+0000 7fa4fbe00700 1 --2- 172.21.15.89:0/2045694865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4f40d2370 0x7fa4f40ce360 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec009e50 tx=0x7fa4ec00b330 comp rx=0 tx=0).stop 2024-04-02T07:51:11.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.441+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2045694865 shutdown_connections 2024-04-02T07:51:11.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.441+0000 7fa4fbe00700 1 --2- 172.21.15.89:0/2045694865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4f40d2370 0x7fa4f40ce360 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:11.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.441+0000 7fa4fbe00700 1 --2- 172.21.15.89:0/2045694865 >> v2:172.21.15.113:3300/0 conn(0x7fa4f40d16a0 0x7fa4f40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:11.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.441+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2045694865 >> 172.21.15.89:0/2045694865 conn(0x7fa4f408aff0 msgr2=0x7fa4f40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:11.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.441+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2045694865 shutdown_connections 2024-04-02T07:51:11.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.441+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2045694865 wait complete. 2024-04-02T07:51:11.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.442+0000 7fa4fbe00700 1 Processor -- start 2024-04-02T07:51:11.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.443+0000 7fa4fbe00700 1 -- start start 2024-04-02T07:51:11.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.443+0000 7fa4fbe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4f40d16a0 0x7fa4f4160220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:11.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.443+0000 7fa4fbe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4f40d2370 0x7fa4f4158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:11.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.443+0000 7fa4fbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f415d9d0 0x7fa4f40cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:11.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4fbe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f4159370 con 0x7fa4f415d9d0 2024-04-02T07:51:11.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4fbe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f415f180 con 0x7fa4f40d2370 2024-04-02T07:51:11.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4fbe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f415f460 con 0x7fa4f40d16a0 2024-04-02T07:51:11.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4f9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f415d9d0 0x7fa4f40cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:11.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4f9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f415d9d0 0x7fa4f40cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34448/0 (socket says 172.21.15.89:34448) 2024-04-02T07:51:11.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4f8a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4f40d2370 0x7fa4f4158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:11.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4f9e00700 1 -- 172.21.15.89:0/2249313175 learned_addr learned my addr 172.21.15.89:0/2249313175 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:11.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4f9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4f40d16a0 0x7fa4f4160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:11.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4f9e00700 1 -- 172.21.15.89:0/2249313175 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4f40d16a0 msgr2=0x7fa4f4160220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:11.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4f9e00700 1 --2- 172.21.15.89:0/2249313175 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4f40d16a0 0x7fa4f4160220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:11.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4f9e00700 1 -- 172.21.15.89:0/2249313175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4f40d2370 msgr2=0x7fa4f4158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:11.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4f9e00700 1 --2- 172.21.15.89:0/2249313175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4f40d2370 0x7fa4f4158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:11.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4f9e00700 1 -- 172.21.15.89:0/2249313175 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec009b30 con 0x7fa4f415d9d0 2024-04-02T07:51:11.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4f8a00700 1 --2- 172.21.15.89:0/2249313175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4f40d2370 0x7fa4f4158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:11.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.444+0000 7fa4f9e00700 1 --2- 172.21.15.89:0/2249313175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f415d9d0 0x7fa4f40cae50 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fa4f40ca580 tx=0x7fa4ec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:11.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.445+0000 7fa4f1600700 1 -- 172.21.15.89:0/2249313175 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec004800 con 0x7fa4f415d9d0 2024-04-02T07:51:11.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.445+0000 7fa4f1600700 1 -- 172.21.15.89:0/2249313175 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ec004960 con 0x7fa4f415d9d0 2024-04-02T07:51:11.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.445+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2249313175 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f415f740 con 0x7fa4f415d9d0 2024-04-02T07:51:11.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.445+0000 7fa4f1600700 1 -- 172.21.15.89:0/2249313175 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec01c680 con 0x7fa4f415d9d0 2024-04-02T07:51:11.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.445+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2249313175 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f415fd00 con 0x7fa4f415d9d0 2024-04-02T07:51:11.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.446+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2249313175 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f40d1ca0 con 0x7fa4f415d9d0 2024-04-02T07:51:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.447+0000 7fa4f1600700 1 -- 172.21.15.89:0/2249313175 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 31) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fa4ec01c7e0 con 0x7fa4f415d9d0 2024-04-02T07:51:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.448+0000 7fa4f1600700 1 --2- 172.21.15.89:0/2249313175 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa4d406aae0 0x7fa4d406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.448+0000 7fa4f1600700 1 -- 172.21.15.89:0/2249313175 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6918+0+0 (secure 0 0 0) 0x7fa4ec08aff0 con 0x7fa4f415d9d0 2024-04-02T07:51:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.448+0000 7fa4f9400700 1 --2- 172.21.15.89:0/2249313175 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa4d406aae0 0x7fa4d406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:11.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.448+0000 7fa4f9400700 1 --2- 172.21.15.89:0/2249313175 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa4d406aae0 0x7fa4d406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa4f41f6670 tx=0x7fa4e8009040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:11.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.453+0000 7fa4f1600700 1 -- 172.21.15.89:0/2249313175 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa4ec056050 con 0x7fa4f415d9d0 2024-04-02T07:51:11.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:11.694+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2249313175 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa4f4067720 con 0x7fa4f415d9d0 2024-04-02T07:51:12.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.226+0000 7fa4f1600700 1 -- 172.21.15.89:0/2249313175 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 32) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fa4ec056420 con 0x7fa4f415d9d0 2024-04-02T07:51:12.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.231+0000 7fa4f1600700 1 -- 172.21.15.89:0/2249313175 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa4ec06f930 con 0x7fa4f415d9d0 2024-04-02T07:51:12.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.234+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2249313175 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa4d406aae0 msgr2=0x7fa4d406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:12.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.234+0000 7fa4fbe00700 1 --2- 172.21.15.89:0/2249313175 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa4d406aae0 0x7fa4d406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa4f41f6670 tx=0x7fa4e8009040 comp rx=0 tx=0).stop 2024-04-02T07:51:12.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.234+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2249313175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f415d9d0 msgr2=0x7fa4f40cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:12.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.234+0000 7fa4fbe00700 1 --2- 172.21.15.89:0/2249313175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f415d9d0 0x7fa4f40cae50 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fa4f40ca580 tx=0x7fa4ec004ad0 comp rx=0 tx=0).stop 2024-04-02T07:51:12.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.234+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2249313175 shutdown_connections 2024-04-02T07:51:12.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.234+0000 7fa4fbe00700 1 --2- 172.21.15.89:0/2249313175 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa4d406aae0 0x7fa4d406cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:12.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.234+0000 7fa4fbe00700 1 --2- 172.21.15.89:0/2249313175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4f415d9d0 0x7fa4f40cae50 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:12.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.234+0000 7fa4fbe00700 1 --2- 172.21.15.89:0/2249313175 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4f40d16a0 0x7fa4f4160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:12.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.234+0000 7fa4fbe00700 1 --2- 172.21.15.89:0/2249313175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4f40d2370 0x7fa4f4158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:12.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.234+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2249313175 >> 172.21.15.89:0/2249313175 conn(0x7fa4f408aff0 msgr2=0x7fa4f415bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:12.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.234+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2249313175 shutdown_connections 2024-04-02T07:51:12.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.234+0000 7fa4fbe00700 1 -- 172.21.15.89:0/2249313175 wait complete. 2024-04-02T07:51:12.255 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T07:51:12.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.447+0000 7ff86fc00700 1 Processor -- start 2024-04-02T07:51:12.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.447+0000 7ff86fc00700 1 -- start start 2024-04-02T07:51:12.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.448+0000 7ff86fc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff8680d28e0 0x7ff868168680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:12.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.448+0000 7ff86fc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8680ce8a0 con 0x7ff86815ddf0 2024-04-02T07:51:12.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.448+0000 7ff86fc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8680cdc90 con 0x7ff8680ced30 2024-04-02T07:51:12.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.448+0000 7ff86fc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86815ec60 con 0x7ff8680d28e0 2024-04-02T07:51:12.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.448+0000 7ff86dc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff8680d28e0 0x7ff868168680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:12.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.448+0000 7ff86dc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff8680d28e0 0x7ff868168680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34494/0 (socket says 172.21.15.89:34494) 2024-04-02T07:51:12.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.448+0000 7ff86dc00700 1 -- 172.21.15.89:0/61749403 learned_addr learned my addr 172.21.15.89:0/61749403 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:12.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.449+0000 7ff86dc00700 1 -- 172.21.15.89:0/61749403 >> v1:172.21.15.113:6789/0 conn(0x7ff8680ced30 legacy=0x7ff8680d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:12.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.449+0000 7ff86dc00700 1 -- 172.21.15.89:0/61749403 >> v1:172.21.15.89:6789/0 conn(0x7ff86815ddf0 legacy=0x7ff86815e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:12.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.449+0000 7ff867400700 1 -- 172.21.15.89:0/61749403 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237365131 0 0) 0x7ff8680ce8a0 con 0x7ff86815ddf0 2024-04-02T07:51:12.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.449+0000 7ff86dc00700 1 -- 172.21.15.89:0/61749403 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8680cdf70 con 0x7ff8680d28e0 2024-04-02T07:51:12.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.449+0000 7ff86dc00700 1 --2- 172.21.15.89:0/61749403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8680d28e0 0x7ff868168680 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff860009e50 tx=0x7ff86000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39528a25ce8ea27 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:12.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.449+0000 7ff867400700 1 -- 172.21.15.89:0/61749403 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86000e070 con 0x7ff8680d28e0 2024-04-02T07:51:12.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.449+0000 7ff867400700 1 -- 172.21.15.89:0/61749403 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff860003a20 con 0x7ff8680d28e0 2024-04-02T07:51:12.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.449+0000 7ff867400700 1 -- 172.21.15.89:0/61749403 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff860003dd0 con 0x7ff8680d28e0 2024-04-02T07:51:12.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.450+0000 7ff86fc00700 1 -- 172.21.15.89:0/61749403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8680d28e0 msgr2=0x7ff868168680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:12.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.450+0000 7ff86fc00700 1 --2- 172.21.15.89:0/61749403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8680d28e0 0x7ff868168680 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff860009e50 tx=0x7ff86000b330 comp rx=0 tx=0).stop 2024-04-02T07:51:12.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.450+0000 7ff86fc00700 1 -- 172.21.15.89:0/61749403 shutdown_connections 2024-04-02T07:51:12.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.450+0000 7ff86fc00700 1 --2- 172.21.15.89:0/61749403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8680d28e0 0x7ff868168680 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:12.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.450+0000 7ff86fc00700 1 -- 172.21.15.89:0/61749403 >> 172.21.15.89:0/61749403 conn(0x7ff86808aff0 msgr2=0x7ff8680d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:12.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.450+0000 7ff86fc00700 1 -- 172.21.15.89:0/61749403 shutdown_connections 2024-04-02T07:51:12.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.450+0000 7ff86fc00700 1 -- 172.21.15.89:0/61749403 wait complete. 2024-04-02T07:51:12.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.452+0000 7ff86fc00700 1 Processor -- start 2024-04-02T07:51:12.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.452+0000 7ff86fc00700 1 -- start start 2024-04-02T07:51:12.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff86fc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8680ced30 0x7ff8680b5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:12.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff86fc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff86815ddf0 0x7ff868149610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff86fc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff868149b50 0x7ff8680c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff86fc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86814a000 con 0x7ff86815ddf0 2024-04-02T07:51:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff86fc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8680b5790 con 0x7ff868149b50 2024-04-02T07:51:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff86fc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8680b3440 con 0x7ff8680ced30 2024-04-02T07:51:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff867e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff86815ddf0 0x7ff868149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff867e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff86815ddf0 0x7ff868149610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34504/0 (socket says 172.21.15.89:34504) 2024-04-02T07:51:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff867e00700 1 -- 172.21.15.89:0/3828160701 learned_addr learned my addr 172.21.15.89:0/3828160701 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff867e00700 1 -- 172.21.15.89:0/3828160701 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8680ced30 msgr2=0x7ff8680b5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff86dc00700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff868149b50 0x7ff8680c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff86d200700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8680ced30 0x7ff8680b5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff867e00700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8680ced30 0x7ff8680b5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff867e00700 1 -- 172.21.15.89:0/3828160701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff868149b50 msgr2=0x7ff8680c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff867e00700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff868149b50 0x7ff8680c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:12.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff867e00700 1 -- 172.21.15.89:0/3828160701 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff860009b30 con 0x7ff86815ddf0 2024-04-02T07:51:12.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff86dc00700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff868149b50 0x7ff8680c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:12.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.453+0000 7ff86d200700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8680ced30 0x7ff8680b5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:12.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.454+0000 7ff867e00700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff86815ddf0 0x7ff868149610 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff85400c500 tx=0x7ff85400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:12.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.454+0000 7ff864c00700 1 -- 172.21.15.89:0/3828160701 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854011790 con 0x7ff86815ddf0 2024-04-02T07:51:12.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.454+0000 7ff864c00700 1 -- 172.21.15.89:0/3828160701 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff854011930 con 0x7ff86815ddf0 2024-04-02T07:51:12.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.454+0000 7ff864c00700 1 -- 172.21.15.89:0/3828160701 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854011ce0 con 0x7ff86815ddf0 2024-04-02T07:51:12.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.454+0000 7ff86fc00700 1 -- 172.21.15.89:0/3828160701 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8680b2e30 con 0x7ff86815ddf0 2024-04-02T07:51:12.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.454+0000 7ff86fc00700 1 -- 172.21.15.89:0/3828160701 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8680b30d0 con 0x7ff86815ddf0 2024-04-02T07:51:12.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.455+0000 7ff86fc00700 1 -- 172.21.15.89:0/3828160701 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff868067720 con 0x7ff86815ddf0 2024-04-02T07:51:12.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.456+0000 7ff864c00700 1 -- 172.21.15.89:0/3828160701 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 32) v1 ==== 89183+0+0 (secure 0 0 0) 0x7ff85401c460 con 0x7ff86815ddf0 2024-04-02T07:51:12.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.457+0000 7ff864c00700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff84806ad20 0x7ff84806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:12.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.457+0000 7ff864c00700 1 -- 172.21.15.89:0/3828160701 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6918+0+0 (secure 0 0 0) 0x7ff854089610 con 0x7ff86815ddf0 2024-04-02T07:51:12.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.457+0000 7ff86d200700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff84806ad20 0x7ff84806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:12.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.458+0000 7ff86d200700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff84806ad20 0x7ff84806d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff85c000f60 tx=0x7ff85c009040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:12.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.461+0000 7ff864c00700 1 -- 172.21.15.89:0/3828160701 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff854054340 con 0x7ff86815ddf0 2024-04-02T07:51:12.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.660+0000 7ff86fc00700 1 -- 172.21.15.89:0/3828160701 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff8680b3580 con 0x7ff86815ddf0 2024-04-02T07:51:12.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.950+0000 7ff864c00700 1 -- 172.21.15.89:0/3828160701 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v73) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff854003990 con 0x7ff86815ddf0 2024-04-02T07:51:12.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.954+0000 7ff86fc00700 1 -- 172.21.15.89:0/3828160701 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff84806ad20 msgr2=0x7ff84806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:12.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.954+0000 7ff86fc00700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff84806ad20 0x7ff84806d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff85c000f60 tx=0x7ff85c009040 comp rx=0 tx=0).stop 2024-04-02T07:51:12.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.954+0000 7ff86fc00700 1 -- 172.21.15.89:0/3828160701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff86815ddf0 msgr2=0x7ff868149610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:12.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.954+0000 7ff86fc00700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff86815ddf0 0x7ff868149610 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff85400c500 tx=0x7ff85400c8c0 comp rx=0 tx=0).stop 2024-04-02T07:51:12.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.954+0000 7ff86fc00700 1 -- 172.21.15.89:0/3828160701 shutdown_connections 2024-04-02T07:51:12.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.954+0000 7ff86fc00700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff84806ad20 0x7ff84806d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:12.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.954+0000 7ff86fc00700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff868149b50 0x7ff8680c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:12.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.954+0000 7ff86fc00700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8680ced30 0x7ff8680b5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:12.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.954+0000 7ff86fc00700 1 --2- 172.21.15.89:0/3828160701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff86815ddf0 0x7ff868149610 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:12.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.954+0000 7ff86fc00700 1 -- 172.21.15.89:0/3828160701 >> 172.21.15.89:0/3828160701 conn(0x7ff86808aff0 msgr2=0x7ff86806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:12.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.954+0000 7ff86fc00700 1 -- 172.21.15.89:0/3828160701 shutdown_connections 2024-04-02T07:51:12.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:12.954+0000 7ff86fc00700 1 -- 172.21.15.89:0/3828160701 wait complete. 2024-04-02T07:51:12.958 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T07:51:12.979 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T07:51:13.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.173+0000 7fdf3e000700 1 Processor -- start 2024-04-02T07:51:13.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.173+0000 7fdf3e000700 1 -- start start 2024-04-02T07:51:13.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.174+0000 7fdf3e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf380ced30 0x7fdf380d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:13.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.174+0000 7fdf3e000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdf380d2710 0x7fdf3815f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:13.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.174+0000 7fdf3e000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf380ce8a0 con 0x7fdf3815ddf0 2024-04-02T07:51:13.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.174+0000 7fdf3e000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf3815ec60 con 0x7fdf380d2710 2024-04-02T07:51:13.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.174+0000 7fdf3e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380cdc90 con 0x7fdf380ced30 2024-04-02T07:51:13.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.175+0000 7fdf37400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdf3815ddf0 0x7fdf3815e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33028/0 (socket says 172.21.15.89:33028) 2024-04-02T07:51:13.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.175+0000 7fdf37400700 1 -- 172.21.15.89:0/3111827952 learned_addr learned my addr 172.21.15.89:0/3111827952 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:13.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.175+0000 7fdf36a00700 1 --2- 172.21.15.89:0/3111827952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf380ced30 0x7fdf380d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:13.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.175+0000 7fdf37e00700 1 --2- 172.21.15.89:0/3111827952 >> v2:172.21.15.113:3300/0 conn(0x7fdf380d2710 0x7fdf3815f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:13.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.175+0000 7fdf36000700 1 -- 172.21.15.89:0/3111827952 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347158323 0 0) 0x7fdf380ce8a0 con 0x7fdf3815ddf0 2024-04-02T07:51:13.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.175+0000 7fdf36000700 1 -- 172.21.15.89:0/3111827952 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf1c0035b0 con 0x7fdf3815ddf0 2024-04-02T07:51:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.175+0000 7fdf36000700 1 -- 172.21.15.89:0/3111827952 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 473974193 0 0) 0x7fdf1c0035b0 con 0x7fdf3815ddf0 2024-04-02T07:51:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.175+0000 7fdf36000700 1 -- 172.21.15.89:0/3111827952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf380ced30 msgr2=0x7fdf380d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.175+0000 7fdf36000700 1 --2- 172.21.15.89:0/3111827952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf380ced30 0x7fdf380d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.175+0000 7fdf36000700 1 -- 172.21.15.89:0/3111827952 >> v2:172.21.15.113:3300/0 conn(0x7fdf380d2710 msgr2=0x7fdf3815f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.175+0000 7fdf36000700 1 --2- 172.21.15.89:0/3111827952 >> v2:172.21.15.113:3300/0 conn(0x7fdf380d2710 0x7fdf3815f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.175+0000 7fdf36000700 1 -- 172.21.15.89:0/3111827952 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf380cdf70 con 0x7fdf3815ddf0 2024-04-02T07:51:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.175+0000 7fdf36000700 1 -- 172.21.15.89:0/3111827952 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fdf28001670 con 0x7fdf3815ddf0 2024-04-02T07:51:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf36000700 1 -- 172.21.15.89:0/3111827952 >> v1:172.21.15.89:6789/0 conn(0x7fdf3815ddf0 legacy=0x7fdf3815e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf36000700 1 --2- 172.21.15.89:0/3111827952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf380ced30 0x7fdf1c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf36000700 1 --2- 172.21.15.89:0/3111827952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf1c003fd0 0x7fdf1c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf36000700 1 --2- 172.21.15.89:0/3111827952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf1c004d90 0x7fdf1c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf36000700 1 -- 172.21.15.89:0/3111827952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3815ec60 con 0x7fdf1c003fd0 2024-04-02T07:51:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf36000700 1 -- 172.21.15.89:0/3111827952 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380cdc90 con 0x7fdf1c004d90 2024-04-02T07:51:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf36000700 1 -- 172.21.15.89:0/3111827952 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c00bac0 con 0x7fdf380ced30 2024-04-02T07:51:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf36a00700 1 --2- 172.21.15.89:0/3111827952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf1c003fd0 0x7fdf1c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf37e00700 1 --2- 172.21.15.89:0/3111827952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf1c004d90 0x7fdf1c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf37400700 1 --2- 172.21.15.89:0/3111827952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf380ced30 0x7fdf1c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf36a00700 1 -- 172.21.15.89:0/3111827952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf380ced30 msgr2=0x7fdf1c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf36a00700 1 --2- 172.21.15.89:0/3111827952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf380ced30 0x7fdf1c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf36a00700 1 -- 172.21.15.89:0/3111827952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf1c004d90 msgr2=0x7fdf1c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf36a00700 1 --2- 172.21.15.89:0/3111827952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf1c004d90 0x7fdf1c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf36a00700 1 -- 172.21.15.89:0/3111827952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c0035b0 con 0x7fdf1c003fd0 2024-04-02T07:51:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.176+0000 7fdf36a00700 1 --2- 172.21.15.89:0/3111827952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf1c003fd0 0x7fdf1c004850 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fdf24009b60 tx=0x7fdf1c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593f7c2748e37c10 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.177+0000 7fdf36000700 1 -- 172.21.15.89:0/3111827952 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2400e6c0 con 0x7fdf1c003fd0 2024-04-02T07:51:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.177+0000 7fdf36000700 1 -- 172.21.15.89:0/3111827952 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2400e820 con 0x7fdf1c003fd0 2024-04-02T07:51:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.177+0000 7fdf36000700 1 -- 172.21.15.89:0/3111827952 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2400eb90 con 0x7fdf1c003fd0 2024-04-02T07:51:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.177+0000 7fdf3e000700 1 -- 172.21.15.89:0/3111827952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf1c003fd0 msgr2=0x7fdf1c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:13.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.177+0000 7fdf3e000700 1 --2- 172.21.15.89:0/3111827952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf1c003fd0 0x7fdf1c004850 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fdf24009b60 tx=0x7fdf1c005880 comp rx=0 tx=0).stop 2024-04-02T07:51:13.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.177+0000 7fdf37e00700 1 -- 172.21.15.89:0/3111827952 reap_dead start 2024-04-02T07:51:13.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.177+0000 7fdf3e000700 1 -- 172.21.15.89:0/3111827952 shutdown_connections 2024-04-02T07:51:13.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.177+0000 7fdf3e000700 1 -- 172.21.15.89:0/3111827952 >> 172.21.15.89:0/3111827952 conn(0x7fdf3808aff0 msgr2=0x7fdf380d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:13.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.177+0000 7fdf3e000700 1 -- 172.21.15.89:0/3111827952 shutdown_connections 2024-04-02T07:51:13.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.177+0000 7fdf3e000700 1 -- 172.21.15.89:0/3111827952 wait complete. 2024-04-02T07:51:13.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.179+0000 7fdf3e000700 1 Processor -- start 2024-04-02T07:51:13.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.179+0000 7fdf3e000700 1 -- start start 2024-04-02T07:51:13.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.180+0000 7fdf3e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf1c003fd0 0x7fdf38149770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.180+0000 7fdf3e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf3815edd0 0x7fdf3815f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.180+0000 7fdf3e000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf381e9300 0x7fdf381e9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.180+0000 7fdf3e000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf381e9cc0 con 0x7fdf3815edd0 2024-04-02T07:51:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.180+0000 7fdf3e000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3815fe50 con 0x7fdf381e9300 2024-04-02T07:51:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.180+0000 7fdf3e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3815f890 con 0x7fdf1c003fd0 2024-04-02T07:51:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.180+0000 7fdf36a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf3815edd0 0x7fdf3815f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.180+0000 7fdf36a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf3815edd0 0x7fdf3815f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34530/0 (socket says 172.21.15.89:34530) 2024-04-02T07:51:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.180+0000 7fdf36a00700 1 -- 172.21.15.89:0/470742438 learned_addr learned my addr 172.21.15.89:0/470742438 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.181+0000 7fdf37400700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf1c003fd0 0x7fdf38149770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.181+0000 7fdf37e00700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf381e9300 0x7fdf381e9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.181+0000 7fdf36a00700 1 -- 172.21.15.89:0/470742438 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf1c003fd0 msgr2=0x7fdf38149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.181+0000 7fdf36a00700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf1c003fd0 0x7fdf38149770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.181+0000 7fdf36a00700 1 -- 172.21.15.89:0/470742438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf381e9300 msgr2=0x7fdf381e9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.181+0000 7fdf36a00700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf381e9300 0x7fdf381e9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.181+0000 7fdf36a00700 1 -- 172.21.15.89:0/470742438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3815fb70 con 0x7fdf3815edd0 2024-04-02T07:51:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.181+0000 7fdf37400700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf1c003fd0 0x7fdf38149770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.181+0000 7fdf36a00700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf3815edd0 0x7fdf3815f250 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00ba10 tx=0x7fdf240047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.181+0000 7fdf2f400700 1 -- 172.21.15.89:0/470742438 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2400e6c0 con 0x7fdf3815edd0 2024-04-02T07:51:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.181+0000 7fdf2f400700 1 -- 172.21.15.89:0/470742438 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf24003ab0 con 0x7fdf3815edd0 2024-04-02T07:51:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.181+0000 7fdf2f400700 1 -- 172.21.15.89:0/470742438 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24003e40 con 0x7fdf3815edd0 2024-04-02T07:51:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.181+0000 7fdf3e000700 1 -- 172.21.15.89:0/470742438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf380c7150 con 0x7fdf3815edd0 2024-04-02T07:51:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.182+0000 7fdf3e000700 1 -- 172.21.15.89:0/470742438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf38156d10 con 0x7fdf3815edd0 2024-04-02T07:51:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.182+0000 7fdf3e000700 1 -- 172.21.15.89:0/470742438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf38067720 con 0x7fdf3815edd0 2024-04-02T07:51:13.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.183+0000 7fdf2f400700 1 -- 172.21.15.89:0/470742438 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 32) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fdf24013470 con 0x7fdf3815edd0 2024-04-02T07:51:13.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.184+0000 7fdf2f400700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdf1806ad20 0x7fdf1806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:13.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.184+0000 7fdf2f400700 1 -- 172.21.15.89:0/470742438 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6480+0+0 (secure 0 0 0) 0x7fdf2408b0b0 con 0x7fdf3815edd0 2024-04-02T07:51:13.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.184+0000 7fdf37400700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdf1806ad20 0x7fdf1806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:13.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.185+0000 7fdf37400700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdf1806ad20 0x7fdf1806d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdf28001050 tx=0x7fdf28009040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:13.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.189+0000 7fdf2f400700 1 -- 172.21.15.89:0/470742438 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdf24055ff0 con 0x7fdf3815edd0 2024-04-02T07:51:13.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:13.398+0000 7fdf3e000700 1 -- 172.21.15.89:0/470742438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdf381e9cc0 con 0x7fdf3815edd0 2024-04-02T07:51:14.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.140+0000 7fdf2f400700 1 -- 172.21.15.89:0/470742438 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdf240563c0 con 0x7fdf3815edd0 2024-04-02T07:51:14.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.141+0000 7fdf2f400700 1 -- 172.21.15.89:0/470742438 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v74) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdf24055ff0 con 0x7fdf3815edd0 2024-04-02T07:51:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.145+0000 7fdf3e000700 1 -- 172.21.15.89:0/470742438 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdf1806ad20 msgr2=0x7fdf1806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.145+0000 7fdf3e000700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdf1806ad20 0x7fdf1806d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdf28001050 tx=0x7fdf28009040 comp rx=0 tx=0).stop 2024-04-02T07:51:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.145+0000 7fdf3e000700 1 -- 172.21.15.89:0/470742438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf3815edd0 msgr2=0x7fdf3815f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.145+0000 7fdf3e000700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf3815edd0 0x7fdf3815f250 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00ba10 tx=0x7fdf240047c0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.145+0000 7fdf3e000700 1 -- 172.21.15.89:0/470742438 shutdown_connections 2024-04-02T07:51:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.145+0000 7fdf3e000700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdf1806ad20 0x7fdf1806d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.145+0000 7fdf3e000700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf381e9300 0x7fdf381e9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.145+0000 7fdf3e000700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf1c003fd0 0x7fdf38149770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.145+0000 7fdf3e000700 1 --2- 172.21.15.89:0/470742438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf3815edd0 0x7fdf3815f250 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.145+0000 7fdf3e000700 1 -- 172.21.15.89:0/470742438 >> 172.21.15.89:0/470742438 conn(0x7fdf3808aff0 msgr2=0x7fdf380b8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.145+0000 7fdf3e000700 1 -- 172.21.15.89:0/470742438 shutdown_connections 2024-04-02T07:51:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.145+0000 7fdf3e000700 1 -- 172.21.15.89:0/470742438 wait complete. 2024-04-02T07:51:14.149 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T07:51:14.167 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T07:51:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.357+0000 7f6e13e00700 1 Processor -- start 2024-04-02T07:51:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.357+0000 7f6e13e00700 1 -- start start 2024-04-02T07:51:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.358+0000 7f6e13e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6e0c15d9d0 0x7f6e0c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.358+0000 7f6e13e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0c161420 con 0x7f6e0c0d2540 2024-04-02T07:51:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.358+0000 7f6e13e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0c161700 con 0x7f6e0c0d16a0 2024-04-02T07:51:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.358+0000 7f6e13e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e0c15edc0 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.358+0000 7f6e11400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6e0c15d9d0 0x7f6e0c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.358+0000 7f6e11e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6e0c0d2540 0x7f6e0c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33038/0 (socket says 172.21.15.89:33038) 2024-04-02T07:51:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.358+0000 7f6e11e00700 1 -- 172.21.15.89:0/2452711376 learned_addr learned my addr 172.21.15.89:0/2452711376 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:14.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.359+0000 7f6e11400700 1 -- 172.21.15.89:0/2452711376 >> v1:172.21.15.113:6789/0 conn(0x7f6e0c0d16a0 legacy=0x7f6e0c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:14.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.359+0000 7f6e11400700 1 -- 172.21.15.89:0/2452711376 >> v1:172.21.15.89:6789/0 conn(0x7f6e0c0d2540 legacy=0x7f6e0c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.359+0000 7f6e11400700 1 -- 172.21.15.89:0/2452711376 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c1619e0 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.359+0000 7f6e0be00700 1 -- 172.21.15.89:0/2452711376 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860819901 0 0) 0x7f6e0c161420 con 0x7f6e0c0d2540 2024-04-02T07:51:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.359+0000 7f6e11400700 1 --2- 172.21.15.89:0/2452711376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e0c15d9d0 0x7f6e0c0d3c30 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009840 tx=0x7f6e00009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c77cc69e84f621 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.359+0000 7f6e0be00700 1 -- 172.21.15.89:0/2452711376 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00004230 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.359+0000 7f6e0be00700 1 -- 172.21.15.89:0/2452711376 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e00004390 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.359+0000 7f6e0be00700 1 -- 172.21.15.89:0/2452711376 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e000036a0 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.360+0000 7f6e13e00700 1 -- 172.21.15.89:0/2452711376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e0c15d9d0 msgr2=0x7f6e0c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.360+0000 7f6e13e00700 1 --2- 172.21.15.89:0/2452711376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e0c15d9d0 0x7f6e0c0d3c30 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009840 tx=0x7f6e00009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.360+0000 7f6e13e00700 1 -- 172.21.15.89:0/2452711376 shutdown_connections 2024-04-02T07:51:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.360+0000 7f6e13e00700 1 --2- 172.21.15.89:0/2452711376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e0c15d9d0 0x7f6e0c0d3c30 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.360+0000 7f6e13e00700 1 -- 172.21.15.89:0/2452711376 >> 172.21.15.89:0/2452711376 conn(0x7f6e0c08aff0 msgr2=0x7f6e0c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:14.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.360+0000 7f6e13e00700 1 -- 172.21.15.89:0/2452711376 shutdown_connections 2024-04-02T07:51:14.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.360+0000 7f6e13e00700 1 -- 172.21.15.89:0/2452711376 wait complete. 2024-04-02T07:51:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.361+0000 7f6e13e00700 1 Processor -- start 2024-04-02T07:51:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.362+0000 7f6e13e00700 1 -- start start 2024-04-02T07:51:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.362+0000 7f6e13e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e0c0d16a0 0x7f6e0c14f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.362+0000 7f6e13e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e0c0d2540 0x7f6e0c0cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.362+0000 7f6e13e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e0c15d9d0 0x7f6e0c15f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e13e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c14fea0 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e13e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c1e75f0 con 0x7f6e0c0d16a0 2024-04-02T07:51:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e13e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c1e78d0 con 0x7f6e0c0d2540 2024-04-02T07:51:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e11e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e0c15d9d0 0x7f6e0c15f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e11e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e0c15d9d0 0x7f6e0c15f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34548/0 (socket says 172.21.15.89:34548) 2024-04-02T07:51:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e11e00700 1 -- 172.21.15.89:0/2153266060 learned_addr learned my addr 172.21.15.89:0/2153266060 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e11400700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e0c0d16a0 0x7f6e0c14f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e10a00700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e0c0d2540 0x7f6e0c0cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e11e00700 1 -- 172.21.15.89:0/2153266060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e0c0d2540 msgr2=0x7f6e0c0cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e11e00700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e0c0d2540 0x7f6e0c0cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e11e00700 1 -- 172.21.15.89:0/2153266060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e0c0d16a0 msgr2=0x7f6e0c14f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:14.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e11e00700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e0c0d16a0 0x7f6e0c14f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e11e00700 1 -- 172.21.15.89:0/2153266060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e000094a0 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e10a00700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e0c0d2540 0x7f6e0c0cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:14.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.363+0000 7f6e11e00700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e0c15d9d0 0x7f6e0c15f620 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400cc10 tx=0x7f6e0400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:14.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.364+0000 7f6e09600700 1 -- 172.21.15.89:0/2153266060 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0400ab10 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.364+0000 7f6e09600700 1 -- 172.21.15.89:0/2153266060 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0400ac70 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.364+0000 7f6e09600700 1 -- 172.21.15.89:0/2153266060 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e040070d0 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.364+0000 7f6e13e00700 1 -- 172.21.15.89:0/2153266060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0c1e7bb0 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.364+0000 7f6e13e00700 1 -- 172.21.15.89:0/2153266060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0c1e8170 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.365+0000 7f6e13e00700 1 -- 172.21.15.89:0/2153266060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0c0d1ca0 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.366+0000 7f6e09600700 1 -- 172.21.15.89:0/2153266060 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6e04007230 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.367+0000 7f6e09600700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6dec06adb0 0x7f6dec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:14.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.367+0000 7f6e09600700 1 -- 172.21.15.89:0/2153266060 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f6e04089150 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.367+0000 7f6e11400700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6dec06adb0 0x7f6dec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:14.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.367+0000 7f6e11400700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6dec06adb0 0x7f6dec06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6e000090b0 tx=0x7f6e00008030 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.371+0000 7f6e09600700 1 -- 172.21.15.89:0/2153266060 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6e040540c0 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.571+0000 7f6e13e00700 1 -- 172.21.15.89:0/2153266060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f6e0c1e8390 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.576+0000 7f6e09600700 1 -- 172.21.15.89:0/2153266060 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6e040180b0 con 0x7f6e0c15d9d0 2024-04-02T07:51:14.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.578+0000 7f6e13e00700 1 -- 172.21.15.89:0/2153266060 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6dec06adb0 msgr2=0x7f6dec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.579+0000 7f6e13e00700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6dec06adb0 0x7f6dec06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6e000090b0 tx=0x7f6e00008030 comp rx=0 tx=0).stop 2024-04-02T07:51:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.579+0000 7f6e13e00700 1 -- 172.21.15.89:0/2153266060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e0c15d9d0 msgr2=0x7f6e0c15f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.579+0000 7f6e13e00700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e0c15d9d0 0x7f6e0c15f620 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400cc10 tx=0x7f6e0400cf20 comp rx=0 tx=0).stop 2024-04-02T07:51:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.579+0000 7f6e13e00700 1 -- 172.21.15.89:0/2153266060 shutdown_connections 2024-04-02T07:51:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.579+0000 7f6e13e00700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6dec06adb0 0x7f6dec06d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.579+0000 7f6e13e00700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e0c0d16a0 0x7f6e0c14f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.579+0000 7f6e13e00700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e0c15d9d0 0x7f6e0c15f620 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.579+0000 7f6e13e00700 1 --2- 172.21.15.89:0/2153266060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e0c0d2540 0x7f6e0c0cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.579+0000 7f6e13e00700 1 -- 172.21.15.89:0/2153266060 >> 172.21.15.89:0/2153266060 conn(0x7f6e0c08aff0 msgr2=0x7f6e0c0b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.579+0000 7f6e13e00700 1 -- 172.21.15.89:0/2153266060 shutdown_connections 2024-04-02T07:51:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.579+0000 7f6e13e00700 1 -- 172.21.15.89:0/2153266060 wait complete. 2024-04-02T07:51:14.600 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T07:51:14.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.791+0000 7fc5df800700 1 Processor -- start 2024-04-02T07:51:14.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.791+0000 7fc5df800700 1 -- start start 2024-04-02T07:51:14.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.792+0000 7fc5df800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc5d815d9d0 0x7fc5d80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:14.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.792+0000 7fc5df800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc5d80d2540 0x7fc5d80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:14.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.792+0000 7fc5df800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d8161420 con 0x7fc5d80d16a0 2024-04-02T07:51:14.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.792+0000 7fc5df800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d815edc0 con 0x7fc5d815d9d0 2024-04-02T07:51:14.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.792+0000 7fc5df800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d8161700 con 0x7fc5d80d2540 2024-04-02T07:51:14.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.792+0000 7fc5dce00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc5d815d9d0 0x7fc5d80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:14.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.792+0000 7fc5d7e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc5d80d16a0 0x7fc5d80d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33054/0 (socket says 172.21.15.89:33054) 2024-04-02T07:51:14.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.792+0000 7fc5d7e00700 1 -- 172.21.15.89:0/772872645 learned_addr learned my addr 172.21.15.89:0/772872645 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:14.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.793+0000 7fc5dd800700 1 --2- 172.21.15.89:0/772872645 >> v2:172.21.15.113:3300/0 conn(0x7fc5d80d2540 0x7fc5d80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:14.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.793+0000 7fc5dce00700 1 -- 172.21.15.89:0/772872645 >> v2:172.21.15.113:3300/0 conn(0x7fc5d80d2540 msgr2=0x7fc5d80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:14.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.793+0000 7fc5dce00700 1 --2- 172.21.15.89:0/772872645 >> v2:172.21.15.113:3300/0 conn(0x7fc5d80d2540 0x7fc5d80ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.793+0000 7fc5dce00700 1 -- 172.21.15.89:0/772872645 >> v1:172.21.15.89:6789/0 conn(0x7fc5d80d16a0 legacy=0x7fc5d80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:14.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.793+0000 7fc5dce00700 1 -- 172.21.15.89:0/772872645 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d81619e0 con 0x7fc5d815d9d0 2024-04-02T07:51:14.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.793+0000 7fc5d7400700 1 -- 172.21.15.89:0/772872645 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310888579 0 0) 0x7fc5d8161420 con 0x7fc5d80d16a0 2024-04-02T07:51:14.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.793+0000 7fc5dce00700 1 --2- 172.21.15.89:0/772872645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5d815d9d0 0x7fc5d80d3c30 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc5d808ca10 tx=0x7fc5cc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e698b6d8cdc5b1e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:14.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.793+0000 7fc5d7400700 1 -- 172.21.15.89:0/772872645 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc011830 con 0x7fc5d815d9d0 2024-04-02T07:51:14.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.793+0000 7fc5d7400700 1 -- 172.21.15.89:0/772872645 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5cc011990 con 0x7fc5d815d9d0 2024-04-02T07:51:14.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.794+0000 7fc5d7400700 1 -- 172.21.15.89:0/772872645 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc011d20 con 0x7fc5d815d9d0 2024-04-02T07:51:14.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.794+0000 7fc5df800700 1 -- 172.21.15.89:0/772872645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5d815d9d0 msgr2=0x7fc5d80d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:14.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.794+0000 7fc5df800700 1 --2- 172.21.15.89:0/772872645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5d815d9d0 0x7fc5d80d3c30 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc5d808ca10 tx=0x7fc5cc009b20 comp rx=0 tx=0).stop 2024-04-02T07:51:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.794+0000 7fc5df800700 1 -- 172.21.15.89:0/772872645 shutdown_connections 2024-04-02T07:51:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.794+0000 7fc5df800700 1 --2- 172.21.15.89:0/772872645 >> v2:172.21.15.113:3300/0 conn(0x7fc5d80d2540 0x7fc5d80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.794+0000 7fc5df800700 1 --2- 172.21.15.89:0/772872645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5d815d9d0 0x7fc5d80d3c30 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.794+0000 7fc5df800700 1 -- 172.21.15.89:0/772872645 >> 172.21.15.89:0/772872645 conn(0x7fc5d808aff0 msgr2=0x7fc5d80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.794+0000 7fc5df800700 1 -- 172.21.15.89:0/772872645 shutdown_connections 2024-04-02T07:51:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.794+0000 7fc5df800700 1 -- 172.21.15.89:0/772872645 wait complete. 2024-04-02T07:51:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.796+0000 7fc5df800700 1 Processor -- start 2024-04-02T07:51:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.796+0000 7fc5df800700 1 -- start start 2024-04-02T07:51:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.797+0000 7fc5df800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5d80d16a0 0x7fc5d814bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.797+0000 7fc5df800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5d80d2540 0x7fc5d80cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.797+0000 7fc5df800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5d815d9d0 0x7fc5d80cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.797+0000 7fc5df800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d80cb1e0 con 0x7fc5d80d2540 2024-04-02T07:51:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.797+0000 7fc5df800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d80ca5f0 con 0x7fc5d815d9d0 2024-04-02T07:51:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.797+0000 7fc5df800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d814c180 con 0x7fc5d80d16a0 2024-04-02T07:51:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.797+0000 7fc5d7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5d80d2540 0x7fc5d80cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.797+0000 7fc5d7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5d80d2540 0x7fc5d80cba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34564/0 (socket says 172.21.15.89:34564) 2024-04-02T07:51:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.797+0000 7fc5d7e00700 1 -- 172.21.15.89:0/751156440 learned_addr learned my addr 172.21.15.89:0/751156440 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.797+0000 7fc5dd800700 1 --2- 172.21.15.89:0/751156440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5d815d9d0 0x7fc5d80cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.797+0000 7fc5dce00700 1 --2- 172.21.15.89:0/751156440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5d80d16a0 0x7fc5d814bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.798+0000 7fc5d7e00700 1 -- 172.21.15.89:0/751156440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5d80d16a0 msgr2=0x7fc5d814bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.798+0000 7fc5d7e00700 1 --2- 172.21.15.89:0/751156440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5d80d16a0 0x7fc5d814bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.798+0000 7fc5d7e00700 1 -- 172.21.15.89:0/751156440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5d815d9d0 msgr2=0x7fc5d80cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.798+0000 7fc5d7e00700 1 --2- 172.21.15.89:0/751156440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5d815d9d0 0x7fc5d80cbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.798+0000 7fc5d7e00700 1 -- 172.21.15.89:0/751156440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc0094a0 con 0x7fc5d80d2540 2024-04-02T07:51:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.798+0000 7fc5d7e00700 1 --2- 172.21.15.89:0/751156440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5d80d2540 0x7fc5d80cba60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400c6d0 tx=0x7fc5c400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.798+0000 7fc5d4c00700 1 -- 172.21.15.89:0/751156440 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c40074e0 con 0x7fc5d80d2540 2024-04-02T07:51:14.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.798+0000 7fc5d4c00700 1 -- 172.21.15.89:0/751156440 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c4007660 con 0x7fc5d80d2540 2024-04-02T07:51:14.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.798+0000 7fc5d4c00700 1 -- 172.21.15.89:0/751156440 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4005a70 con 0x7fc5d80d2540 2024-04-02T07:51:14.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.798+0000 7fc5df800700 1 -- 172.21.15.89:0/751156440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d80ca840 con 0x7fc5d80d2540 2024-04-02T07:51:14.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.798+0000 7fc5df800700 1 -- 172.21.15.89:0/751156440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d80cae00 con 0x7fc5d80d2540 2024-04-02T07:51:14.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.799+0000 7fc5df800700 1 -- 172.21.15.89:0/751156440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d80d1ca0 con 0x7fc5d80d2540 2024-04-02T07:51:14.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.800+0000 7fc5d4c00700 1 -- 172.21.15.89:0/751156440 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc5c4005bd0 con 0x7fc5d80d2540 2024-04-02T07:51:14.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.801+0000 7fc5d4c00700 1 --2- 172.21.15.89:0/751156440 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc5b806adb0 0x7fc5b806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:14.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.802+0000 7fc5d4c00700 1 -- 172.21.15.89:0/751156440 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5893+0+0 (secure 0 0 0) 0x7fc5c40889c0 con 0x7fc5d80d2540 2024-04-02T07:51:14.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.802+0000 7fc5dce00700 1 --2- 172.21.15.89:0/751156440 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc5b806adb0 0x7fc5b806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.802+0000 7fc5dce00700 1 --2- 172.21.15.89:0/751156440 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc5b806adb0 0x7fc5b806d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc0090b0 tx=0x7fc5cc01c040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:14.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:14.806+0000 7fc5d4c00700 1 -- 172.21.15.89:0/751156440 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc5c40539c0 con 0x7fc5d80d2540 2024-04-02T07:51:15.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.008+0000 7fc5df800700 1 -- 172.21.15.89:0/751156440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fc5d80cb320 con 0x7fc5d80d2540 2024-04-02T07:51:15.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.012+0000 7fc5d4c00700 1 -- 172.21.15.89:0/751156440 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v20) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc5d80cb320 con 0x7fc5d80d2540 2024-04-02T07:51:15.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.015+0000 7fc5df800700 1 -- 172.21.15.89:0/751156440 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc5b806adb0 msgr2=0x7fc5b806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:15.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.015+0000 7fc5df800700 1 --2- 172.21.15.89:0/751156440 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc5b806adb0 0x7fc5b806d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc0090b0 tx=0x7fc5cc01c040 comp rx=0 tx=0).stop 2024-04-02T07:51:15.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.015+0000 7fc5df800700 1 -- 172.21.15.89:0/751156440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5d80d2540 msgr2=0x7fc5d80cba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:15.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.015+0000 7fc5df800700 1 --2- 172.21.15.89:0/751156440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5d80d2540 0x7fc5d80cba60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400c6d0 tx=0x7fc5c400c9e0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.015+0000 7fc5df800700 1 -- 172.21.15.89:0/751156440 shutdown_connections 2024-04-02T07:51:15.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.015+0000 7fc5df800700 1 --2- 172.21.15.89:0/751156440 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc5b806adb0 0x7fc5b806d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.015+0000 7fc5df800700 1 --2- 172.21.15.89:0/751156440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5d815d9d0 0x7fc5d80cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.015+0000 7fc5df800700 1 --2- 172.21.15.89:0/751156440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5d80d16a0 0x7fc5d814bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.015+0000 7fc5df800700 1 --2- 172.21.15.89:0/751156440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5d80d2540 0x7fc5d80cba60 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.015+0000 7fc5df800700 1 -- 172.21.15.89:0/751156440 >> 172.21.15.89:0/751156440 conn(0x7fc5d808aff0 msgr2=0x7fc5d815ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:15.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.015+0000 7fc5df800700 1 -- 172.21.15.89:0/751156440 shutdown_connections 2024-04-02T07:51:15.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.015+0000 7fc5df800700 1 -- 172.21.15.89:0/751156440 wait complete. 2024-04-02T07:51:15.037 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T07:51:15.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.238+0000 7f2f79a00700 1 Processor -- start 2024-04-02T07:51:15.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.238+0000 7f2f79a00700 1 -- start start 2024-04-02T07:51:15.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.239+0000 7f2f79a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2f7415d9d0 0x7f2f740d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:15.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.239+0000 7f2f79a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2f740d2540 0x7f2f740ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:15.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.239+0000 7f2f79a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f74161420 con 0x7f2f740d16a0 2024-04-02T07:51:15.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.239+0000 7f2f79a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7415edc0 con 0x7f2f7415d9d0 2024-04-02T07:51:15.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.239+0000 7f2f79a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f74161700 con 0x7f2f740d2540 2024-04-02T07:51:15.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.239+0000 7f2f72a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2f7415d9d0 0x7f2f740d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:15.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.239+0000 7f2f72000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2f740d16a0 0x7f2f740d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33068/0 (socket says 172.21.15.89:33068) 2024-04-02T07:51:15.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.239+0000 7f2f72000700 1 -- 172.21.15.89:0/3355455466 learned_addr learned my addr 172.21.15.89:0/3355455466 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:15.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.239+0000 7f2f73400700 1 --2- 172.21.15.89:0/3355455466 >> v2:172.21.15.113:3300/0 conn(0x7f2f740d2540 0x7f2f740ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.240+0000 7f2f72a00700 1 -- 172.21.15.89:0/3355455466 >> v2:172.21.15.113:3300/0 conn(0x7f2f740d2540 msgr2=0x7f2f740ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.240+0000 7f2f72a00700 1 --2- 172.21.15.89:0/3355455466 >> v2:172.21.15.113:3300/0 conn(0x7f2f740d2540 0x7f2f740ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.240+0000 7f2f72a00700 1 -- 172.21.15.89:0/3355455466 >> v1:172.21.15.89:6789/0 conn(0x7f2f740d16a0 legacy=0x7f2f740d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.240+0000 7f2f72a00700 1 -- 172.21.15.89:0/3355455466 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f741619e0 con 0x7f2f7415d9d0 2024-04-02T07:51:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.240+0000 7f2f73400700 1 --2- 172.21.15.89:0/3355455466 >> v2:172.21.15.113:3300/0 conn(0x7f2f740d2540 0x7f2f740ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.240+0000 7f2f71600700 1 -- 172.21.15.89:0/3355455466 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857970761 0 0) 0x7f2f74161420 con 0x7f2f740d16a0 2024-04-02T07:51:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.240+0000 7f2f72a00700 1 --2- 172.21.15.89:0/3355455466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f7415d9d0 0x7f2f740d3c30 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2f7408ca10 tx=0x7f2f64009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65941d0727aca976 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.240+0000 7f2f71600700 1 -- 172.21.15.89:0/3355455466 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64011830 con 0x7f2f7415d9d0 2024-04-02T07:51:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.240+0000 7f2f71600700 1 -- 172.21.15.89:0/3355455466 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f64011990 con 0x7f2f7415d9d0 2024-04-02T07:51:15.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.240+0000 7f2f71600700 1 -- 172.21.15.89:0/3355455466 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64011d20 con 0x7f2f7415d9d0 2024-04-02T07:51:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.241+0000 7f2f79a00700 1 -- 172.21.15.89:0/3355455466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f7415d9d0 msgr2=0x7f2f740d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.241+0000 7f2f79a00700 1 --2- 172.21.15.89:0/3355455466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f7415d9d0 0x7f2f740d3c30 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2f7408ca10 tx=0x7f2f64009b20 comp rx=0 tx=0).stop 2024-04-02T07:51:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.241+0000 7f2f79a00700 1 -- 172.21.15.89:0/3355455466 shutdown_connections 2024-04-02T07:51:15.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.241+0000 7f2f79a00700 1 --2- 172.21.15.89:0/3355455466 >> v2:172.21.15.113:3300/0 conn(0x7f2f740d2540 0x7f2f740ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.241+0000 7f2f79a00700 1 --2- 172.21.15.89:0/3355455466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f7415d9d0 0x7f2f740d3c30 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.241+0000 7f2f79a00700 1 -- 172.21.15.89:0/3355455466 >> 172.21.15.89:0/3355455466 conn(0x7f2f7408aff0 msgr2=0x7f2f740d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.241+0000 7f2f79a00700 1 -- 172.21.15.89:0/3355455466 shutdown_connections 2024-04-02T07:51:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.241+0000 7f2f79a00700 1 -- 172.21.15.89:0/3355455466 wait complete. 2024-04-02T07:51:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.243+0000 7f2f79a00700 1 Processor -- start 2024-04-02T07:51:15.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.243+0000 7f2f79a00700 1 -- start start 2024-04-02T07:51:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f79a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f740d16a0 0x7f2f7415c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f79a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f740d2540 0x7f2f740cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f79a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f7415d9d0 0x7f2f740b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:15.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f79a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7415ce60 con 0x7f2f7415d9d0 2024-04-02T07:51:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f79a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f740ca700 con 0x7f2f740d16a0 2024-04-02T07:51:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f79a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f740ca9e0 con 0x7f2f740d2540 2024-04-02T07:51:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f73400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f7415d9d0 0x7f2f740b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f73400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f7415d9d0 0x7f2f740b06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34574/0 (socket says 172.21.15.89:34574) 2024-04-02T07:51:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f73400700 1 -- 172.21.15.89:0/3052952056 learned_addr learned my addr 172.21.15.89:0/3052952056 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f72000700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f740d2540 0x7f2f740cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f72a00700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f740d16a0 0x7f2f7415c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f73400700 1 -- 172.21.15.89:0/3052952056 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f740d2540 msgr2=0x7f2f740cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f73400700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f740d2540 0x7f2f740cbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f73400700 1 -- 172.21.15.89:0/3052952056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f740d16a0 msgr2=0x7f2f7415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f73400700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f740d16a0 0x7f2f7415c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f73400700 1 -- 172.21.15.89:0/3052952056 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f640094a0 con 0x7f2f7415d9d0 2024-04-02T07:51:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f72000700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f740d2540 0x7f2f740cbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:15.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.244+0000 7f2f72a00700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f740d16a0 0x7f2f7415c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:15.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.245+0000 7f2f73400700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f7415d9d0 0x7f2f740b06f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c007ec0 tx=0x7f2f6c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:15.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.245+0000 7f2f6aa00700 1 -- 172.21.15.89:0/3052952056 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c00de30 con 0x7f2f7415d9d0 2024-04-02T07:51:15.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.245+0000 7f2f6aa00700 1 -- 172.21.15.89:0/3052952056 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6c00f040 con 0x7f2f7415d9d0 2024-04-02T07:51:15.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.245+0000 7f2f6aa00700 1 -- 172.21.15.89:0/3052952056 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c003c60 con 0x7f2f7415d9d0 2024-04-02T07:51:15.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.245+0000 7f2f79a00700 1 -- 172.21.15.89:0/3052952056 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f740cacc0 con 0x7f2f7415d9d0 2024-04-02T07:51:15.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.245+0000 7f2f79a00700 1 -- 172.21.15.89:0/3052952056 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f740cc5d0 con 0x7f2f7415d9d0 2024-04-02T07:51:15.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.246+0000 7f2f79a00700 1 -- 172.21.15.89:0/3052952056 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f740d1ca0 con 0x7f2f7415d9d0 2024-04-02T07:51:15.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.247+0000 7f2f6aa00700 1 -- 172.21.15.89:0/3052952056 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2f6c004030 con 0x7f2f7415d9d0 2024-04-02T07:51:15.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.248+0000 7f2f6aa00700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f5406adb0 0x7f2f5406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:15.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.248+0000 7f2f6aa00700 1 -- 172.21.15.89:0/3052952056 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f2f6c089840 con 0x7f2f7415d9d0 2024-04-02T07:51:15.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.248+0000 7f2f72a00700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f5406adb0 0x7f2f5406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:15.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.249+0000 7f2f72a00700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f5406adb0 0x7f2f5406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2f740b7ef0 tx=0x7f2f64008030 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:15.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.252+0000 7f2f6aa00700 1 -- 172.21.15.89:0/3052952056 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2f6c054740 con 0x7f2f7415d9d0 2024-04-02T07:51:15.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.459+0000 7f2f79a00700 1 -- 172.21.15.89:0/3052952056 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2f74067720 con 0x7f2f7415d9d0 2024-04-02T07:51:15.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.463+0000 7f2f6aa00700 1 -- 172.21.15.89:0/3052952056 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2f6c057d60 con 0x7f2f7415d9d0 2024-04-02T07:51:15.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.465+0000 7f2f79a00700 1 -- 172.21.15.89:0/3052952056 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f5406adb0 msgr2=0x7f2f5406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:15.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.465+0000 7f2f79a00700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f5406adb0 0x7f2f5406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2f740b7ef0 tx=0x7f2f64008030 comp rx=0 tx=0).stop 2024-04-02T07:51:15.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.465+0000 7f2f79a00700 1 -- 172.21.15.89:0/3052952056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f7415d9d0 msgr2=0x7f2f740b06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:15.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.465+0000 7f2f79a00700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f7415d9d0 0x7f2f740b06f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c007ec0 tx=0x7f2f6c00d3b0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.465+0000 7f2f79a00700 1 -- 172.21.15.89:0/3052952056 shutdown_connections 2024-04-02T07:51:15.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.465+0000 7f2f79a00700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f5406adb0 0x7f2f5406d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.465+0000 7f2f79a00700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f740d16a0 0x7f2f7415c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.465+0000 7f2f79a00700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f7415d9d0 0x7f2f740b06f0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.465+0000 7f2f79a00700 1 --2- 172.21.15.89:0/3052952056 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f740d2540 0x7f2f740cbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.465+0000 7f2f79a00700 1 -- 172.21.15.89:0/3052952056 >> 172.21.15.89:0/3052952056 conn(0x7f2f7408aff0 msgr2=0x7f2f740b16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:15.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.465+0000 7f2f79a00700 1 -- 172.21.15.89:0/3052952056 shutdown_connections 2024-04-02T07:51:15.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.465+0000 7f2f79a00700 1 -- 172.21.15.89:0/3052952056 wait complete. 2024-04-02T07:51:15.486 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-04-02T07:51:15.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.674+0000 7fcbc2200700 1 Processor -- start 2024-04-02T07:51:15.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.674+0000 7fcbc2200700 1 -- start start 2024-04-02T07:51:15.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.675+0000 7fcbc2200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcbbc1619d0 0x7fcbbc1685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:15.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.675+0000 7fcbc2200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc0d30d0 con 0x7fcbbc15d640 2024-04-02T07:51:15.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.675+0000 7fcbc2200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc0b6840 con 0x7fcbbc0ceec0 2024-04-02T07:51:15.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.675+0000 7fcbc2200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbbc0d2b30 con 0x7fcbbc1619d0 2024-04-02T07:51:15.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.675+0000 7fcbbbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcbbc1619d0 0x7fcbbc1685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:15.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.675+0000 7fcbbb400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcbbc15d640 0x7fcbbc0d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33084/0 (socket says 172.21.15.89:33084) 2024-04-02T07:51:15.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.676+0000 7fcbbb400700 1 -- 172.21.15.89:0/2302888165 learned_addr learned my addr 172.21.15.89:0/2302888165 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:15.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.676+0000 7fcbbbe00700 1 -- 172.21.15.89:0/2302888165 >> v1:172.21.15.113:6789/0 conn(0x7fcbbc0ceec0 legacy=0x7fcbbc1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:15.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.676+0000 7fcbbbe00700 1 -- 172.21.15.89:0/2302888165 >> v1:172.21.15.89:6789/0 conn(0x7fcbbc15d640 legacy=0x7fcbbc0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:15.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.676+0000 7fcbbbe00700 1 -- 172.21.15.89:0/2302888165 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc0d2df0 con 0x7fcbbc1619d0 2024-04-02T07:51:15.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.676+0000 7fcbbbe00700 1 --2- 172.21.15.89:0/2302888165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbbc1619d0 0x7fcbbc1685f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009d90 tx=0x7fcbb400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf7e4091959db26 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:15.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.676+0000 7fcbba000700 1 -- 172.21.15.89:0/2302888165 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb400e070 con 0x7fcbbc1619d0 2024-04-02T07:51:15.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.676+0000 7fcbba000700 1 -- 172.21.15.89:0/2302888165 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb400bbd0 con 0x7fcbbc1619d0 2024-04-02T07:51:15.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.676+0000 7fcbba000700 1 -- 172.21.15.89:0/2302888165 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4012430 con 0x7fcbbc1619d0 2024-04-02T07:51:15.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.677+0000 7fcbc2200700 1 -- 172.21.15.89:0/2302888165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbbc1619d0 msgr2=0x7fcbbc1685f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:15.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.677+0000 7fcbc2200700 1 --2- 172.21.15.89:0/2302888165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbbc1619d0 0x7fcbbc1685f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009d90 tx=0x7fcbb400b330 comp rx=0 tx=0).stop 2024-04-02T07:51:15.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.677+0000 7fcbc2200700 1 -- 172.21.15.89:0/2302888165 shutdown_connections 2024-04-02T07:51:15.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.677+0000 7fcbc2200700 1 --2- 172.21.15.89:0/2302888165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbbc1619d0 0x7fcbbc1685f0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.677+0000 7fcbc2200700 1 -- 172.21.15.89:0/2302888165 >> 172.21.15.89:0/2302888165 conn(0x7fcbbc08aff0 msgr2=0x7fcbbc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:15.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.677+0000 7fcbc2200700 1 -- 172.21.15.89:0/2302888165 shutdown_connections 2024-04-02T07:51:15.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.677+0000 7fcbc2200700 1 -- 172.21.15.89:0/2302888165 wait complete. 2024-04-02T07:51:15.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.679+0000 7fcbc2200700 1 Processor -- start 2024-04-02T07:51:15.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.679+0000 7fcbc2200700 1 -- start start 2024-04-02T07:51:15.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbc2200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbbc0ceec0 0x7fcbbc15bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:15.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbc2200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbbc15d640 0x7fcbbc09d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:15.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbc2200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbbc1619d0 0x7fcbbc09f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:15.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbc2200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0a0240 con 0x7fcbbc1619d0 2024-04-02T07:51:15.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbc2200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0a0520 con 0x7fcbbc15d640 2024-04-02T07:51:15.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbc2200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0a0800 con 0x7fcbbc0ceec0 2024-04-02T07:51:15.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbbbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbbc1619d0 0x7fcbbc09f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:15.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbbbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbbc1619d0 0x7fcbbc09f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34588/0 (socket says 172.21.15.89:34588) 2024-04-02T07:51:15.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbbbe00700 1 -- 172.21.15.89:0/3446102629 learned_addr learned my addr 172.21.15.89:0/3446102629 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:15.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbbb400700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbbc0ceec0 0x7fcbbc15bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:15.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbbaa00700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbbc15d640 0x7fcbbc09d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:15.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbbbe00700 1 -- 172.21.15.89:0/3446102629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbbc0ceec0 msgr2=0x7fcbbc15bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:15.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbbbe00700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbbc0ceec0 0x7fcbbc15bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbbbe00700 1 -- 172.21.15.89:0/3446102629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbbc15d640 msgr2=0x7fcbbc09d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:15.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbbbe00700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbbc15d640 0x7fcbbc09d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:15.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbbbe00700 1 -- 172.21.15.89:0/3446102629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb4009a70 con 0x7fcbbc1619d0 2024-04-02T07:51:15.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbbaa00700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbbc15d640 0x7fcbbc09d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:15.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.680+0000 7fcbbb400700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbbc0ceec0 0x7fcbbc15bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:15.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.681+0000 7fcbbbe00700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbbc1619d0 0x7fcbbc09f9d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4014040 tx=0x7fcbb4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:15.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.681+0000 7fcbb3400700 1 -- 172.21.15.89:0/3446102629 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb40098f0 con 0x7fcbbc1619d0 2024-04-02T07:51:15.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.681+0000 7fcbb3400700 1 -- 172.21.15.89:0/3446102629 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb4020070 con 0x7fcbbc1619d0 2024-04-02T07:51:15.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.681+0000 7fcbb3400700 1 -- 172.21.15.89:0/3446102629 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4004280 con 0x7fcbbc1619d0 2024-04-02T07:51:15.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.681+0000 7fcbc2200700 1 -- 172.21.15.89:0/3446102629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbbc0a0ae0 con 0x7fcbbc1619d0 2024-04-02T07:51:15.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.681+0000 7fcbc2200700 1 -- 172.21.15.89:0/3446102629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbbc09ff10 con 0x7fcbbc1619d0 2024-04-02T07:51:15.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.682+0000 7fcbc2200700 1 -- 172.21.15.89:0/3446102629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbbc0a1000 con 0x7fcbbc1619d0 2024-04-02T07:51:15.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.683+0000 7fcbb3400700 1 -- 172.21.15.89:0/3446102629 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcbb40043e0 con 0x7fcbbc1619d0 2024-04-02T07:51:15.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.684+0000 7fcbb3400700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb9c06adb0 0x7fcb9c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:15.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.684+0000 7fcbb3400700 1 -- 172.21.15.89:0/3446102629 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5893+0+0 (secure 0 0 0) 0x7fcbb4022070 con 0x7fcbbc1619d0 2024-04-02T07:51:15.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.684+0000 7fcbbb400700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb9c06adb0 0x7fcb9c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:15.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.685+0000 7fcbbb400700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb9c06adb0 0x7fcb9c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcbac005910 tx=0x7fcbac009040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:15.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.688+0000 7fcbb3400700 1 -- 172.21.15.89:0/3446102629 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcbbc0a1000 con 0x7fcbbc1619d0 2024-04-02T07:51:15.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:51:15.888 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:51:15.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:15.973+0000 7fcbc2200700 1 -- 172.21.15.89:0/3446102629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7fcbbc0a1000 con 0x7fcbbc1619d0 2024-04-02T07:51:16.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.138+0000 7fcbb3400700 1 -- 172.21.15.89:0/3446102629 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v556) v1 ==== 154+0+0 (secure 0 0 0) 0x7fcbbc0a1000 con 0x7fcbbc1619d0 2024-04-02T07:51:16.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.140+0000 7fcbc2200700 1 -- 172.21.15.89:0/3446102629 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb9c06adb0 msgr2=0x7fcb9c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:16.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.140+0000 7fcbc2200700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb9c06adb0 0x7fcb9c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcbac005910 tx=0x7fcbac009040 comp rx=0 tx=0).stop 2024-04-02T07:51:16.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.140+0000 7fcbc2200700 1 -- 172.21.15.89:0/3446102629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbbc1619d0 msgr2=0x7fcbbc09f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:16.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.140+0000 7fcbc2200700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbbc1619d0 0x7fcbbc09f9d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4014040 tx=0x7fcbb4004ad0 comp rx=0 tx=0).stop 2024-04-02T07:51:16.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.140+0000 7fcbc2200700 1 -- 172.21.15.89:0/3446102629 shutdown_connections 2024-04-02T07:51:16.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.140+0000 7fcbc2200700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb9c06adb0 0x7fcb9c06d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:16.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.140+0000 7fcbc2200700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbbc1619d0 0x7fcbbc09f9d0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:16.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.140+0000 7fcbc2200700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbbc0ceec0 0x7fcbbc15bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:16.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.140+0000 7fcbc2200700 1 --2- 172.21.15.89:0/3446102629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbbc15d640 0x7fcbbc09d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:16.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.140+0000 7fcbc2200700 1 -- 172.21.15.89:0/3446102629 >> 172.21.15.89:0/3446102629 conn(0x7fcbbc08aff0 msgr2=0x7fcbbc15ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:16.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.141+0000 7fcbc2200700 1 -- 172.21.15.89:0/3446102629 shutdown_connections 2024-04-02T07:51:16.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.141+0000 7fcbc2200700 1 -- 172.21.15.89:0/3446102629 wait complete. 2024-04-02T07:51:16.162 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2024-04-02T07:51:16.162 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2024-04-02T07:51:16.162 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-02T07:51:16.163 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-04-02T07:51:16.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.351+0000 7f5686000700 1 Processor -- start 2024-04-02T07:51:16.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.351+0000 7f5686000700 1 -- start start 2024-04-02T07:51:16.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.352+0000 7f5686000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5680160f10 0x7f56800d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:16.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.353+0000 7f5686000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56800ceda0 con 0x7f56800cf230 2024-04-02T07:51:16.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.353+0000 7f5686000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56801612f0 con 0x7f5680161590 2024-04-02T07:51:16.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.353+0000 7f5686000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56800ce470 con 0x7f5680160f10 2024-04-02T07:51:16.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.353+0000 7f567ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f56800cf230 0x7f56800d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33098/0 (socket says 172.21.15.89:33098) 2024-04-02T07:51:16.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.353+0000 7f567ea00700 1 -- 172.21.15.89:0/3251831125 learned_addr learned my addr 172.21.15.89:0/3251831125 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:16.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.353+0000 7f567f400700 1 --2- 172.21.15.89:0/3251831125 >> v2:172.21.15.113:3300/0 conn(0x7f5680160f10 0x7f56800d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:16.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.353+0000 7f567e000700 1 -- 172.21.15.89:0/3251831125 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763935682 0 0) 0x7f56800ceda0 con 0x7f56800cf230 2024-04-02T07:51:16.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.353+0000 7f567e000700 1 -- 172.21.15.89:0/3251831125 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56640035b0 con 0x7f56800cf230 2024-04-02T07:51:16.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.354+0000 7f567e000700 1 -- 172.21.15.89:0/3251831125 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823719718 0 0) 0x7f56801612f0 con 0x7f5680161590 2024-04-02T07:51:16.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.354+0000 7f567e000700 1 -- 172.21.15.89:0/3251831125 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56800ceda0 con 0x7f5680161590 2024-04-02T07:51:16.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.354+0000 7f567f400700 1 -- 172.21.15.89:0/3251831125 >> v1:172.21.15.113:6789/0 conn(0x7f5680161590 legacy=0x7f56800b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:16.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.354+0000 7f567e000700 1 -- 172.21.15.89:0/3251831125 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 501859396 0 0) 0x7f56640035b0 con 0x7f56800cf230 2024-04-02T07:51:16.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.354+0000 7f567f400700 1 -- 172.21.15.89:0/3251831125 >> v1:172.21.15.89:6789/0 conn(0x7f56800cf230 legacy=0x7f56800d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:16.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.354+0000 7f567f400700 1 -- 172.21.15.89:0/3251831125 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56800ce750 con 0x7f5680160f10 2024-04-02T07:51:16.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.354+0000 7f567f400700 1 --2- 172.21.15.89:0/3251831125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5680160f10 0x7f56800d18b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5670009840 tx=0x7f5670009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cce16bf29369eff server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:16.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.354+0000 7f567e000700 1 -- 172.21.15.89:0/3251831125 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c001670 con 0x7f5680160f10 2024-04-02T07:51:16.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.354+0000 7f567e000700 1 -- 172.21.15.89:0/3251831125 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5670004030 con 0x7f5680160f10 2024-04-02T07:51:16.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.354+0000 7f567e000700 1 -- 172.21.15.89:0/3251831125 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56700043c0 con 0x7f5680160f10 2024-04-02T07:51:16.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.355+0000 7f5686000700 1 -- 172.21.15.89:0/3251831125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5680160f10 msgr2=0x7f56800d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:16.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.355+0000 7f5686000700 1 --2- 172.21.15.89:0/3251831125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5680160f10 0x7f56800d18b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5670009840 tx=0x7f5670009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:16.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.355+0000 7f5686000700 1 -- 172.21.15.89:0/3251831125 shutdown_connections 2024-04-02T07:51:16.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.355+0000 7f5686000700 1 --2- 172.21.15.89:0/3251831125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5680160f10 0x7f56800d18b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:16.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.355+0000 7f5686000700 1 -- 172.21.15.89:0/3251831125 >> 172.21.15.89:0/3251831125 conn(0x7f568008aff0 msgr2=0x7f56800d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:16.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.355+0000 7f5686000700 1 -- 172.21.15.89:0/3251831125 shutdown_connections 2024-04-02T07:51:16.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.355+0000 7f5686000700 1 -- 172.21.15.89:0/3251831125 wait complete. 2024-04-02T07:51:16.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.356+0000 7f5686000700 1 Processor -- start 2024-04-02T07:51:16.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.357+0000 7f5686000700 1 -- start start 2024-04-02T07:51:16.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.357+0000 7f5686000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56800cf230 0x7f5680156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:16.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.358+0000 7f5686000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5680160f10 0x7f5680154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:16.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.358+0000 7f5686000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5680161590 0x7f56801f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:16.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.358+0000 7f5686000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56800b6920 con 0x7f5680160f10 2024-04-02T07:51:16.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.358+0000 7f5686000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56801f7470 con 0x7f56800cf230 2024-04-02T07:51:16.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.358+0000 7f5686000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56801f71d0 con 0x7f5680161590 2024-04-02T07:51:16.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.358+0000 7f567ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5680160f10 0x7f5680154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:16.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.358+0000 7f567ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5680160f10 0x7f5680154d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34590/0 (socket says 172.21.15.89:34590) 2024-04-02T07:51:16.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.358+0000 7f567ea00700 1 -- 172.21.15.89:0/1423022326 learned_addr learned my addr 172.21.15.89:0/1423022326 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.358+0000 7f567ea00700 1 -- 172.21.15.89:0/1423022326 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5680161590 msgr2=0x7f56801f6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.358+0000 7f567ea00700 1 --2- 172.21.15.89:0/1423022326 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5680161590 0x7f56801f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.358+0000 7f567ea00700 1 -- 172.21.15.89:0/1423022326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56800cf230 msgr2=0x7f5680156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.358+0000 7f567ea00700 1 --2- 172.21.15.89:0/1423022326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56800cf230 0x7f5680156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.358+0000 7f567ea00700 1 -- 172.21.15.89:0/1423022326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56700094a0 con 0x7f5680160f10 2024-04-02T07:51:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.358+0000 7f567ea00700 1 --2- 172.21.15.89:0/1423022326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5680160f10 0x7f5680154d50 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f566c00c7a0 tx=0x7f566c00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.359+0000 7f5677400700 1 -- 172.21.15.89:0/1423022326 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c007560 con 0x7f5680160f10 2024-04-02T07:51:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.359+0000 7f5677400700 1 -- 172.21.15.89:0/1423022326 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566c0076c0 con 0x7f5680160f10 2024-04-02T07:51:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.359+0000 7f5677400700 1 -- 172.21.15.89:0/1423022326 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c004a70 con 0x7f5680160f10 2024-04-02T07:51:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.359+0000 7f5686000700 1 -- 172.21.15.89:0/1423022326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56800b6b10 con 0x7f5680160f10 2024-04-02T07:51:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.359+0000 7f5686000700 1 -- 172.21.15.89:0/1423022326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56800b6db0 con 0x7f5680160f10 2024-04-02T07:51:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.360+0000 7f5686000700 1 -- 172.21.15.89:0/1423022326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5680067720 con 0x7f5680160f10 2024-04-02T07:51:16.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.361+0000 7f5677400700 1 -- 172.21.15.89:0/1423022326 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f566c004bd0 con 0x7f5680160f10 2024-04-02T07:51:16.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.362+0000 7f5677400700 1 --2- 172.21.15.89:0/1423022326 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f566006adb0 0x7f566006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:16.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.362+0000 7f5677400700 1 -- 172.21.15.89:0/1423022326 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f566c088640 con 0x7f5680160f10 2024-04-02T07:51:16.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.362+0000 7f567f400700 1 --2- 172.21.15.89:0/1423022326 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f566006adb0 0x7f566006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:16.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.362+0000 7f567f400700 1 --2- 172.21.15.89:0/1423022326 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f566006adb0 0x7f566006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f56700090b0 tx=0x7f5670010000 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:16.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.366+0000 7f5677400700 1 -- 172.21.15.89:0/1423022326 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f566c04d580 con 0x7f5680160f10 2024-04-02T07:51:16.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:16.649+0000 7f5686000700 1 -- 172.21.15.89:0/1423022326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f568008b760 con 0x7f5680160f10 2024-04-02T07:51:17.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.139+0000 7f5677400700 1 -- 172.21.15.89:0/1423022326 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v557) v1 ==== 159+0+0 (secure 0 0 0) 0x7f566c0536f0 con 0x7f5680160f10 2024-04-02T07:51:17.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.142+0000 7f5686000700 1 -- 172.21.15.89:0/1423022326 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f566006adb0 msgr2=0x7f566006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:17.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.142+0000 7f5686000700 1 --2- 172.21.15.89:0/1423022326 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f566006adb0 0x7f566006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f56700090b0 tx=0x7f5670010000 comp rx=0 tx=0).stop 2024-04-02T07:51:17.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.142+0000 7f5686000700 1 -- 172.21.15.89:0/1423022326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5680160f10 msgr2=0x7f5680154d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:17.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.142+0000 7f5686000700 1 --2- 172.21.15.89:0/1423022326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5680160f10 0x7f5680154d50 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f566c00c7a0 tx=0x7f566c00cb60 comp rx=0 tx=0).stop 2024-04-02T07:51:17.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.142+0000 7f5686000700 1 -- 172.21.15.89:0/1423022326 shutdown_connections 2024-04-02T07:51:17.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.142+0000 7f5686000700 1 --2- 172.21.15.89:0/1423022326 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f566006adb0 0x7f566006d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:17.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.142+0000 7f5686000700 1 --2- 172.21.15.89:0/1423022326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56800cf230 0x7f5680156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:17.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.142+0000 7f5686000700 1 --2- 172.21.15.89:0/1423022326 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5680161590 0x7f56801f6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:17.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.142+0000 7f5686000700 1 --2- 172.21.15.89:0/1423022326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5680160f10 0x7f5680154d50 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:17.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.142+0000 7f5686000700 1 -- 172.21.15.89:0/1423022326 >> 172.21.15.89:0/1423022326 conn(0x7f568008aff0 msgr2=0x7f56800d0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:17.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.142+0000 7f5686000700 1 -- 172.21.15.89:0/1423022326 shutdown_connections 2024-04-02T07:51:17.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.142+0000 7f5686000700 1 -- 172.21.15.89:0/1423022326 wait complete. 2024-04-02T07:51:17.164 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T07:51:17.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.356+0000 7fc09e600700 1 Processor -- start 2024-04-02T07:51:17.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.356+0000 7fc09e600700 1 -- start start 2024-04-02T07:51:17.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.358+0000 7fc09e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc09815d9d0 0x7fc0980d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:17.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.358+0000 7fc09e600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc0980d2540 0x7fc0980ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:17.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.358+0000 7fc09e600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc098161420 con 0x7fc0980d16a0 2024-04-02T07:51:17.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.358+0000 7fc09e600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09815edc0 con 0x7fc0980d2540 2024-04-02T07:51:17.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.358+0000 7fc09e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc098161700 con 0x7fc09815d9d0 2024-04-02T07:51:17.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.358+0000 7fc097400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc09815d9d0 0x7fc0980d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:17.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.358+0000 7fc097e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc0980d2540 0x7fc0980ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:17.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.358+0000 7fc097400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc09815d9d0 0x7fc0980d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:47764/0 (socket says 172.21.15.89:47764) 2024-04-02T07:51:17.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.358+0000 7fc097400700 1 -- 172.21.15.89:0/1670069424 learned_addr learned my addr 172.21.15.89:0/1670069424 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:17.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.359+0000 7fc097400700 1 -- 172.21.15.89:0/1670069424 >> v2:172.21.15.113:3300/0 conn(0x7fc0980d2540 msgr2=0x7fc0980ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:17.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.359+0000 7fc097400700 1 --2- 172.21.15.89:0/1670069424 >> v2:172.21.15.113:3300/0 conn(0x7fc0980d2540 0x7fc0980ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:17.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.359+0000 7fc097400700 1 -- 172.21.15.89:0/1670069424 >> v1:172.21.15.113:6789/0 conn(0x7fc0980d16a0 legacy=0x7fc0980d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:17.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.359+0000 7fc097400700 1 -- 172.21.15.89:0/1670069424 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0981619e0 con 0x7fc09815d9d0 2024-04-02T07:51:17.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.359+0000 7fc097400700 1 --2- 172.21.15.89:0/1670069424 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc09815d9d0 0x7fc0980d3c30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc088009920 tx=0x7fc088009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdec0e58b19680bb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:17.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.359+0000 7fc096000700 1 -- 172.21.15.89:0/1670069424 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088011830 con 0x7fc09815d9d0 2024-04-02T07:51:17.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.359+0000 7fc096000700 1 -- 172.21.15.89:0/1670069424 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc088011990 con 0x7fc09815d9d0 2024-04-02T07:51:17.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.359+0000 7fc096000700 1 -- 172.21.15.89:0/1670069424 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088011d20 con 0x7fc09815d9d0 2024-04-02T07:51:17.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.360+0000 7fc09e600700 1 -- 172.21.15.89:0/1670069424 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc09815d9d0 msgr2=0x7fc0980d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:17.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.360+0000 7fc09e600700 1 --2- 172.21.15.89:0/1670069424 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc09815d9d0 0x7fc0980d3c30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc088009920 tx=0x7fc088009c30 comp rx=0 tx=0).stop 2024-04-02T07:51:17.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.360+0000 7fc09e600700 1 -- 172.21.15.89:0/1670069424 shutdown_connections 2024-04-02T07:51:17.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.360+0000 7fc09e600700 1 --2- 172.21.15.89:0/1670069424 >> v2:172.21.15.113:3300/0 conn(0x7fc0980d2540 0x7fc0980ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:17.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.360+0000 7fc09e600700 1 --2- 172.21.15.89:0/1670069424 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc09815d9d0 0x7fc0980d3c30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:17.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.360+0000 7fc09e600700 1 -- 172.21.15.89:0/1670069424 >> 172.21.15.89:0/1670069424 conn(0x7fc09808aff0 msgr2=0x7fc0980d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:17.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.360+0000 7fc09e600700 1 -- 172.21.15.89:0/1670069424 shutdown_connections 2024-04-02T07:51:17.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.360+0000 7fc09e600700 1 -- 172.21.15.89:0/1670069424 wait complete. 2024-04-02T07:51:17.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.362+0000 7fc09e600700 1 Processor -- start 2024-04-02T07:51:17.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.362+0000 7fc09e600700 1 -- start start 2024-04-02T07:51:17.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc09e600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0980d16a0 0x7fc0980cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:17.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc09e600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0980d2540 0x7fc0981f7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:17.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc09e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc09815d9d0 0x7fc0981f8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:17.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc09e600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0980cd570 con 0x7fc0980d2540 2024-04-02T07:51:17.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc09e600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0981f3f40 con 0x7fc0980d16a0 2024-04-02T07:51:17.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc09e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0980cd290 con 0x7fc09815d9d0 2024-04-02T07:51:17.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc096a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0980d2540 0x7fc0981f7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:17.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc096a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0980d2540 0x7fc0981f7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34602/0 (socket says 172.21.15.89:34602) 2024-04-02T07:51:17.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc096a00700 1 -- 172.21.15.89:0/3502801487 learned_addr learned my addr 172.21.15.89:0/3502801487 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:17.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc097400700 1 --2- 172.21.15.89:0/3502801487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0980d16a0 0x7fc0980cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:17.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc097e00700 1 --2- 172.21.15.89:0/3502801487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc09815d9d0 0x7fc0981f8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:17.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc096a00700 1 -- 172.21.15.89:0/3502801487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc09815d9d0 msgr2=0x7fc0981f8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:17.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc096a00700 1 --2- 172.21.15.89:0/3502801487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc09815d9d0 0x7fc0981f8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:17.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc096a00700 1 -- 172.21.15.89:0/3502801487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0980d16a0 msgr2=0x7fc0980cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:17.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc096a00700 1 --2- 172.21.15.89:0/3502801487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0980d16a0 0x7fc0980cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:17.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.363+0000 7fc096a00700 1 -- 172.21.15.89:0/3502801487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc088009550 con 0x7fc0980d2540 2024-04-02T07:51:17.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.364+0000 7fc096a00700 1 --2- 172.21.15.89:0/3502801487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0980d2540 0x7fc0981f7c30 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fc08400c450 tx=0x7fc08400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:17.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.364+0000 7fc08f400700 1 -- 172.21.15.89:0/3502801487 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0840052d0 con 0x7fc0980d2540 2024-04-02T07:51:17.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.364+0000 7fc08f400700 1 -- 172.21.15.89:0/3502801487 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc084005450 con 0x7fc0980d2540 2024-04-02T07:51:17.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.364+0000 7fc09e600700 1 -- 172.21.15.89:0/3502801487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0981f41b0 con 0x7fc0980d2540 2024-04-02T07:51:17.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.364+0000 7fc08f400700 1 -- 172.21.15.89:0/3502801487 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc084006af0 con 0x7fc0980d2540 2024-04-02T07:51:17.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.364+0000 7fc09e600700 1 -- 172.21.15.89:0/3502801487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0981f4770 con 0x7fc0980d2540 2024-04-02T07:51:17.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.365+0000 7fc09e600700 1 -- 172.21.15.89:0/3502801487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0980d1ca0 con 0x7fc0980d2540 2024-04-02T07:51:17.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.366+0000 7fc08f400700 1 -- 172.21.15.89:0/3502801487 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc0840036d0 con 0x7fc0980d2540 2024-04-02T07:51:17.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.367+0000 7fc08f400700 1 --2- 172.21.15.89:0/3502801487 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc07806adb0 0x7fc07806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:17.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.367+0000 7fc08f400700 1 -- 172.21.15.89:0/3502801487 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5893+0+0 (secure 0 0 0) 0x7fc084088780 con 0x7fc0980d2540 2024-04-02T07:51:17.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.367+0000 7fc097400700 1 --2- 172.21.15.89:0/3502801487 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc07806adb0 0x7fc07806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:17.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.368+0000 7fc097400700 1 --2- 172.21.15.89:0/3502801487 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc07806adb0 0x7fc07806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc088000c00 tx=0x7fc088008030 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:17.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.371+0000 7fc08f400700 1 -- 172.21.15.89:0/3502801487 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc0840536f0 con 0x7fc0980d2540 2024-04-02T07:51:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.576+0000 7fc09e600700 1 -- 172.21.15.89:0/3502801487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc0980681e0 con 0x7fc0980d2540 2024-04-02T07:51:17.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.580+0000 7fc08f400700 1 -- 172.21.15.89:0/3502801487 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc08400ce60 con 0x7fc0980d2540 2024-04-02T07:51:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.583+0000 7fc09e600700 1 -- 172.21.15.89:0/3502801487 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc07806adb0 msgr2=0x7fc07806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.583+0000 7fc09e600700 1 --2- 172.21.15.89:0/3502801487 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc07806adb0 0x7fc07806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc088000c00 tx=0x7fc088008030 comp rx=0 tx=0).stop 2024-04-02T07:51:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.583+0000 7fc09e600700 1 -- 172.21.15.89:0/3502801487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0980d2540 msgr2=0x7fc0981f7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.583+0000 7fc09e600700 1 --2- 172.21.15.89:0/3502801487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0980d2540 0x7fc0981f7c30 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fc08400c450 tx=0x7fc08400c760 comp rx=0 tx=0).stop 2024-04-02T07:51:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.583+0000 7fc09e600700 1 -- 172.21.15.89:0/3502801487 shutdown_connections 2024-04-02T07:51:17.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.583+0000 7fc09e600700 1 --2- 172.21.15.89:0/3502801487 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc07806adb0 0x7fc07806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:17.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.583+0000 7fc09e600700 1 --2- 172.21.15.89:0/3502801487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0980d16a0 0x7fc0980cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:17.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.583+0000 7fc09e600700 1 --2- 172.21.15.89:0/3502801487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc09815d9d0 0x7fc0981f8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:17.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.583+0000 7fc09e600700 1 --2- 172.21.15.89:0/3502801487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0980d2540 0x7fc0981f7c30 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:17.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.583+0000 7fc09e600700 1 -- 172.21.15.89:0/3502801487 >> 172.21.15.89:0/3502801487 conn(0x7fc09808aff0 msgr2=0x7fc0980b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:17.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.583+0000 7fc09e600700 1 -- 172.21.15.89:0/3502801487 shutdown_connections 2024-04-02T07:51:17.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:17.583+0000 7fc09e600700 1 -- 172.21.15.89:0/3502801487 wait complete. 2024-04-02T07:51:17.604 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:51:17.604 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:51:17.696 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:51:17.787 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:51:17.787 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:51:17.787 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:17.831 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:51:17.832 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:51:17.832 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:17.878 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:51:17.878 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:51:17.879 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:17.922 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:51:17.923 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:51:17.923 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:17.968 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:51:17.969 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T07:51:17.969 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:51:18.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.160+0000 7fdf55800700 1 Processor -- start 2024-04-02T07:51:18.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.160+0000 7fdf55800700 1 -- start start 2024-04-02T07:51:18.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.161+0000 7fdf55800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdf500ce590 0x7fdf500ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:18.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.161+0000 7fdf55800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf500b6a20 0x7fdf500b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:18.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.161+0000 7fdf55800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf500ceef0 con 0x7fdf50160be0 2024-04-02T07:51:18.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.161+0000 7fdf55800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf500b65d0 con 0x7fdf500ce590 2024-04-02T07:51:18.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.161+0000 7fdf55800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf500d2a80 con 0x7fdf500b6a20 2024-04-02T07:51:18.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.162+0000 7fdf4e000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdf500ce590 0x7fdf500ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:18.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.162+0000 7fdf4ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdf50160be0 0x7fdf500cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33110/0 (socket says 172.21.15.89:33110) 2024-04-02T07:51:18.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.162+0000 7fdf4ea00700 1 -- 172.21.15.89:0/2782788109 learned_addr learned my addr 172.21.15.89:0/2782788109 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:18.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.162+0000 7fdf4f400700 1 --2- 172.21.15.89:0/2782788109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf500b6a20 0x7fdf500b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:18.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.162+0000 7fdf4e000700 1 -- 172.21.15.89:0/2782788109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf500b6a20 msgr2=0x7fdf500b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:18.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.162+0000 7fdf4e000700 1 --2- 172.21.15.89:0/2782788109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf500b6a20 0x7fdf500b6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:18.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.162+0000 7fdf4e000700 1 -- 172.21.15.89:0/2782788109 >> v1:172.21.15.89:6789/0 conn(0x7fdf50160be0 legacy=0x7fdf500cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:18.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.162+0000 7fdf4e000700 1 -- 172.21.15.89:0/2782788109 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf50160fc0 con 0x7fdf500ce590 2024-04-02T07:51:18.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.162+0000 7fdf4f400700 1 --2- 172.21.15.89:0/2782788109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf500b6a20 0x7fdf500b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:18.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.162+0000 7fdf4e000700 1 --2- 172.21.15.89:0/2782788109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf500ce590 0x7fdf500ce9b0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009840 tx=0x7fdf3c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500d1639d629737a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:18.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.162+0000 7fdf4d600700 1 -- 172.21.15.89:0/2782788109 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c011830 con 0x7fdf500ce590 2024-04-02T07:51:18.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.163+0000 7fdf4d600700 1 -- 172.21.15.89:0/2782788109 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c011990 con 0x7fdf500ce590 2024-04-02T07:51:18.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.163+0000 7fdf4d600700 1 -- 172.21.15.89:0/2782788109 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c011d20 con 0x7fdf500ce590 2024-04-02T07:51:18.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.163+0000 7fdf55800700 1 -- 172.21.15.89:0/2782788109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf500ce590 msgr2=0x7fdf500ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:18.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.163+0000 7fdf55800700 1 --2- 172.21.15.89:0/2782788109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf500ce590 0x7fdf500ce9b0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009840 tx=0x7fdf3c009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:18.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.163+0000 7fdf55800700 1 -- 172.21.15.89:0/2782788109 shutdown_connections 2024-04-02T07:51:18.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.163+0000 7fdf55800700 1 --2- 172.21.15.89:0/2782788109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf500b6a20 0x7fdf500b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:18.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.163+0000 7fdf55800700 1 --2- 172.21.15.89:0/2782788109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf500ce590 0x7fdf500ce9b0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:18.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.163+0000 7fdf55800700 1 -- 172.21.15.89:0/2782788109 >> 172.21.15.89:0/2782788109 conn(0x7fdf5008aff0 msgr2=0x7fdf500d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:18.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.163+0000 7fdf55800700 1 -- 172.21.15.89:0/2782788109 shutdown_connections 2024-04-02T07:51:18.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.163+0000 7fdf55800700 1 -- 172.21.15.89:0/2782788109 wait complete. 2024-04-02T07:51:18.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.165+0000 7fdf55800700 1 Processor -- start 2024-04-02T07:51:18.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.165+0000 7fdf55800700 1 -- start start 2024-04-02T07:51:18.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.166+0000 7fdf55800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf500b6a20 0x7fdf501f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:18.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.166+0000 7fdf55800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf500ce590 0x7fdf500cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:18.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.166+0000 7fdf55800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf50160be0 0x7fdf500cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:18.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.166+0000 7fdf55800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf500c8000 con 0x7fdf500b6a20 2024-04-02T07:51:18.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.166+0000 7fdf55800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf500c82e0 con 0x7fdf50160be0 2024-04-02T07:51:18.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.166+0000 7fdf55800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf500c85c0 con 0x7fdf500ce590 2024-04-02T07:51:18.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.166+0000 7fdf4ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf500b6a20 0x7fdf501f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:18.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.166+0000 7fdf4ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf500b6a20 0x7fdf501f2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34626/0 (socket says 172.21.15.89:34626) 2024-04-02T07:51:18.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.166+0000 7fdf4ea00700 1 -- 172.21.15.89:0/2556024373 learned_addr learned my addr 172.21.15.89:0/2556024373 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:18.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.166+0000 7fdf4e000700 1 --2- 172.21.15.89:0/2556024373 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf500ce590 0x7fdf500cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:18.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.166+0000 7fdf4f400700 1 --2- 172.21.15.89:0/2556024373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf50160be0 0x7fdf500cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:18.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.167+0000 7fdf4ea00700 1 -- 172.21.15.89:0/2556024373 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf500ce590 msgr2=0x7fdf500cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:18.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.167+0000 7fdf4ea00700 1 --2- 172.21.15.89:0/2556024373 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf500ce590 0x7fdf500cc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:18.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.167+0000 7fdf4ea00700 1 -- 172.21.15.89:0/2556024373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf50160be0 msgr2=0x7fdf500cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:18.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.167+0000 7fdf4ea00700 1 --2- 172.21.15.89:0/2556024373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf50160be0 0x7fdf500cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:18.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.167+0000 7fdf4ea00700 1 -- 172.21.15.89:0/2556024373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c0094a0 con 0x7fdf500b6a20 2024-04-02T07:51:18.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.167+0000 7fdf4ea00700 1 --2- 172.21.15.89:0/2556024373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf500b6a20 0x7fdf501f2760 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fdf4000c480 tx=0x7fdf4000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:18.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.167+0000 7fdf46a00700 1 -- 172.21.15.89:0/2556024373 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf40005d70 con 0x7fdf500b6a20 2024-04-02T07:51:18.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.167+0000 7fdf46a00700 1 -- 172.21.15.89:0/2556024373 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4000ce90 con 0x7fdf500b6a20 2024-04-02T07:51:18.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.167+0000 7fdf55800700 1 -- 172.21.15.89:0/2556024373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf500c88a0 con 0x7fdf500b6a20 2024-04-02T07:51:18.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.167+0000 7fdf46a00700 1 -- 172.21.15.89:0/2556024373 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf400072d0 con 0x7fdf500b6a20 2024-04-02T07:51:18.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.167+0000 7fdf55800700 1 -- 172.21.15.89:0/2556024373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf500c7a90 con 0x7fdf500b6a20 2024-04-02T07:51:18.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.168+0000 7fdf55800700 1 -- 172.21.15.89:0/2556024373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf501f2e80 con 0x7fdf500b6a20 2024-04-02T07:51:18.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.169+0000 7fdf46a00700 1 -- 172.21.15.89:0/2556024373 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdf40007430 con 0x7fdf500b6a20 2024-04-02T07:51:18.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.170+0000 7fdf46a00700 1 --2- 172.21.15.89:0/2556024373 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdf3006adb0 0x7fdf3006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:18.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.170+0000 7fdf46a00700 1 -- 172.21.15.89:0/2556024373 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5893+0+0 (secure 0 0 0) 0x7fdf400886e0 con 0x7fdf500b6a20 2024-04-02T07:51:18.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.170+0000 7fdf4e000700 1 --2- 172.21.15.89:0/2556024373 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdf3006adb0 0x7fdf3006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:18.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.171+0000 7fdf4e000700 1 --2- 172.21.15.89:0/2556024373 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdf3006adb0 0x7fdf3006d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0090b0 tx=0x7fdf3c008030 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:18.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.175+0000 7fdf46a00700 1 -- 172.21.15.89:0/2556024373 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdf40053830 con 0x7fdf500b6a20 2024-04-02T07:51:18.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.418+0000 7fdf55800700 1 -- 172.21.15.89:0/2556024373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf50067720 con 0x7fdf500b6a20 2024-04-02T07:51:18.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.418+0000 7fdf46a00700 1 -- 172.21.15.89:0/2556024373 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+3535 (secure 0 0 0) 0x7fdf40017de0 con 0x7fdf500b6a20 2024-04-02T07:51:18.421 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:18.421 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":26,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5710,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3098065397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3098065397},{"type":"v1","addr":"172.21.15.89:6836","nonce":3098065397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5713,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1408411829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1408411829},{"type":"v1","addr":"172.21.15.89:6837","nonce":1408411829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5716,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3266955664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3266955664},{"type":"v1","addr":"172.21.15.113:6834","nonce":3266955664}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6073,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2455323633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2455323633},{"type":"v1","addr":"172.21.15.113:6835","nonce":2455323633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[]} 2024-04-02T07:51:18.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.421+0000 7fdf55800700 1 -- 172.21.15.89:0/2556024373 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdf3006adb0 msgr2=0x7fdf3006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:18.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.421+0000 7fdf55800700 1 --2- 172.21.15.89:0/2556024373 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdf3006adb0 0x7fdf3006d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0090b0 tx=0x7fdf3c008030 comp rx=0 tx=0).stop 2024-04-02T07:51:18.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.421+0000 7fdf55800700 1 -- 172.21.15.89:0/2556024373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf500b6a20 msgr2=0x7fdf501f2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:18.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.421+0000 7fdf55800700 1 --2- 172.21.15.89:0/2556024373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf500b6a20 0x7fdf501f2760 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fdf4000c480 tx=0x7fdf4000c790 comp rx=0 tx=0).stop 2024-04-02T07:51:18.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.421+0000 7fdf55800700 1 -- 172.21.15.89:0/2556024373 shutdown_connections 2024-04-02T07:51:18.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.421+0000 7fdf55800700 1 --2- 172.21.15.89:0/2556024373 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdf3006adb0 0x7fdf3006d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:18.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.421+0000 7fdf55800700 1 --2- 172.21.15.89:0/2556024373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf50160be0 0x7fdf500cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:18.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.421+0000 7fdf55800700 1 --2- 172.21.15.89:0/2556024373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf500b6a20 0x7fdf501f2760 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:18.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.421+0000 7fdf55800700 1 --2- 172.21.15.89:0/2556024373 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf500ce590 0x7fdf500cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:18.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.421+0000 7fdf55800700 1 -- 172.21.15.89:0/2556024373 >> 172.21.15.89:0/2556024373 conn(0x7fdf5008aff0 msgr2=0x7fdf500b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:18.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.421+0000 7fdf55800700 1 -- 172.21.15.89:0/2556024373 shutdown_connections 2024-04-02T07:51:18.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:18.421+0000 7fdf55800700 1 -- 172.21.15.89:0/2556024373 wait complete. 2024-04-02T07:51:18.425 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 26 2024-04-02T07:51:18.441 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T07:51:18.442 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T07:51:18.442 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T07:51:18.442 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:51:18.442 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T07:51:18.442 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T07:51:18.442 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T07:51:18.443 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:51:18.443 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T07:51:18.443 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T07:51:18.443 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T07:51:18.443 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:51:18.443 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T07:51:18.443 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T07:51:18.443 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T07:51:18.444 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:51:18.458 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T07:51:18.458 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T07:51:18.460 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T07:51:18.460 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T07:51:18.461 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T07:51:18.461 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T07:51:18.502 INFO:tasks.ceph.mds.c:Started 2024-04-02T07:51:18.503 DEBUG:teuthology.parallel:result is None 2024-04-02T07:51:18.504 INFO:tasks.ceph.mds.b:Started 2024-04-02T07:51:18.504 DEBUG:teuthology.parallel:result is None 2024-04-02T07:51:18.506 INFO:tasks.ceph.mds.d:Started 2024-04-02T07:51:18.506 DEBUG:teuthology.parallel:result is None 2024-04-02T07:51:18.639 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T07:51:18.653 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T07:51:18.653 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T07:51:18.654 INFO:tasks.ceph.mds.b.smithi113.stderr:2024-04-02T07:51:18.653+0000 7f29f2c00700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-04-02T07:51:19.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-04-02T07:51:24.458 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T07:51:24.458 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T07:51:24.463 INFO:tasks.ceph.mds.a:Started 2024-04-02T07:51:24.463 DEBUG:teuthology.parallel:result is None 2024-04-02T07:51:24.464 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T07:51:24.615 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T07:51:24.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.670+0000 7fad02200700 1 Processor -- start 2024-04-02T07:51:24.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.670+0000 7fad02200700 1 -- start start 2024-04-02T07:51:24.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.671+0000 7fad02200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7facfc0cbb30 0x7facfc0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:24.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.671+0000 7fad02200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facfc0cd020 0x7facfc0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:24.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.671+0000 7fad02200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7facfc0b2070 0x7facfc0b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:24.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.671+0000 7fad02200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facfc0cc450 con 0x7facfc0b2070 2024-04-02T07:51:24.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.671+0000 7fad02200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facfc0cdae0 con 0x7facfc0cbb30 2024-04-02T07:51:24.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.671+0000 7fad02200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facfc0cddc0 con 0x7facfc0cd020 2024-04-02T07:51:24.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.671+0000 7facfbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7facfc0b2070 0x7facfc0b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:24.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.671+0000 7facfbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7facfc0b2070 0x7facfc0b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54068/0 (socket says 172.21.15.89:54068) 2024-04-02T07:51:24.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.671+0000 7facfbe00700 1 -- 172.21.15.89:0/2719781649 learned_addr learned my addr 172.21.15.89:0/2719781649 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:24.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.672+0000 7facfb400700 1 --2- 172.21.15.89:0/2719781649 >> v2:172.21.15.113:3300/0 conn(0x7facfc0cbb30 0x7facfc0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:24.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.672+0000 7facfaa00700 1 --2- 172.21.15.89:0/2719781649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facfc0cd020 0x7facfc0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:24.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.672+0000 7facfbe00700 1 -- 172.21.15.89:0/2719781649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facfc0cd020 msgr2=0x7facfc0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:24.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.672+0000 7facfbe00700 1 --2- 172.21.15.89:0/2719781649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facfc0cd020 0x7facfc0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:24.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.672+0000 7facfbe00700 1 -- 172.21.15.89:0/2719781649 >> v2:172.21.15.113:3300/0 conn(0x7facfc0cbb30 msgr2=0x7facfc0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:24.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.672+0000 7facfbe00700 1 --2- 172.21.15.89:0/2719781649 >> v2:172.21.15.113:3300/0 conn(0x7facfc0cbb30 0x7facfc0cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:24.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.672+0000 7facfbe00700 1 -- 172.21.15.89:0/2719781649 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facfc0ce0a0 con 0x7facfc0b2070 2024-04-02T07:51:24.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.672+0000 7facfb400700 1 --2- 172.21.15.89:0/2719781649 >> v2:172.21.15.113:3300/0 conn(0x7facfc0cbb30 0x7facfc0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:24.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.672+0000 7facfaa00700 1 --2- 172.21.15.89:0/2719781649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facfc0cd020 0x7facfc0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:24.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.672+0000 7facfbe00700 1 --2- 172.21.15.89:0/2719781649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc0b2070 0x7facfc0b2500 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7facfc06fd00 tx=0x7facf400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a6cfe31e99987a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:24.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.672+0000 7facfa000700 1 -- 172.21.15.89:0/2719781649 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf400e070 con 0x7facfc0b2070 2024-04-02T07:51:24.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.672+0000 7facfa000700 1 -- 172.21.15.89:0/2719781649 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf400b9c0 con 0x7facfc0b2070 2024-04-02T07:51:24.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.672+0000 7facfa000700 1 -- 172.21.15.89:0/2719781649 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf400bd70 con 0x7facfc0b2070 2024-04-02T07:51:24.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.673+0000 7fad02200700 1 -- 172.21.15.89:0/2719781649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc0b2070 msgr2=0x7facfc0b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:24.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.673+0000 7fad02200700 1 --2- 172.21.15.89:0/2719781649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc0b2070 0x7facfc0b2500 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7facfc06fd00 tx=0x7facf400b330 comp rx=0 tx=0).stop 2024-04-02T07:51:24.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.673+0000 7fad02200700 1 -- 172.21.15.89:0/2719781649 shutdown_connections 2024-04-02T07:51:24.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.673+0000 7fad02200700 1 --2- 172.21.15.89:0/2719781649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc0b2070 0x7facfc0b2500 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:24.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.673+0000 7fad02200700 1 --2- 172.21.15.89:0/2719781649 >> v2:172.21.15.113:3300/0 conn(0x7facfc0cbb30 0x7facfc0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:24.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.673+0000 7fad02200700 1 --2- 172.21.15.89:0/2719781649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facfc0cd020 0x7facfc0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:24.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.673+0000 7fad02200700 1 -- 172.21.15.89:0/2719781649 >> 172.21.15.89:0/2719781649 conn(0x7facfc0b0d70 msgr2=0x7facfc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:24.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.673+0000 7fad02200700 1 -- 172.21.15.89:0/2719781649 shutdown_connections 2024-04-02T07:51:24.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.673+0000 7fad02200700 1 -- 172.21.15.89:0/2719781649 wait complete. 2024-04-02T07:51:24.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.675+0000 7fad02200700 1 Processor -- start 2024-04-02T07:51:24.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.675+0000 7fad02200700 1 -- start start 2024-04-02T07:51:24.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.676+0000 7fad02200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facfc0b2070 0x7facfc157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.676+0000 7fad02200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc0cbb30 0x7facfc06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.676+0000 7fad02200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facfc0cd020 0x7facfc06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.676+0000 7fad02200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facfc1f5460 con 0x7facfc0cbb30 2024-04-02T07:51:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.676+0000 7fad02200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facfc1f47e0 con 0x7facfc0b2070 2024-04-02T07:51:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.676+0000 7fad02200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facfc1f4ac0 con 0x7facfc0cd020 2024-04-02T07:51:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.676+0000 7facfaa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc0cbb30 0x7facfc06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.676+0000 7facfaa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc0cbb30 0x7facfc06e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54084/0 (socket says 172.21.15.89:54084) 2024-04-02T07:51:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.676+0000 7facfaa00700 1 -- 172.21.15.89:0/4087835028 learned_addr learned my addr 172.21.15.89:0/4087835028 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.676+0000 7facfaa00700 1 -- 172.21.15.89:0/4087835028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facfc0cd020 msgr2=0x7facfc06e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.676+0000 7facfaa00700 1 --2- 172.21.15.89:0/4087835028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facfc0cd020 0x7facfc06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.676+0000 7facfaa00700 1 -- 172.21.15.89:0/4087835028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facfc0b2070 msgr2=0x7facfc157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.676+0000 7facfaa00700 1 --2- 172.21.15.89:0/4087835028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facfc0b2070 0x7facfc157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:24.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.676+0000 7facfaa00700 1 -- 172.21.15.89:0/4087835028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf4009b30 con 0x7facfc0cbb30 2024-04-02T07:51:24.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.677+0000 7facfaa00700 1 --2- 172.21.15.89:0/4087835028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc0cbb30 0x7facfc06e320 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7face800e810 tx=0x7face800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:24.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.677+0000 7facf3400700 1 -- 172.21.15.89:0/4087835028 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face8013610 con 0x7facfc0cbb30 2024-04-02T07:51:24.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.677+0000 7facf3400700 1 -- 172.21.15.89:0/4087835028 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face8013770 con 0x7facfc0cbb30 2024-04-02T07:51:24.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.677+0000 7fad02200700 1 -- 172.21.15.89:0/4087835028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facfc1f4da0 con 0x7facfc0cbb30 2024-04-02T07:51:24.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.677+0000 7facf3400700 1 -- 172.21.15.89:0/4087835028 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face8013b20 con 0x7facfc0cbb30 2024-04-02T07:51:24.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.677+0000 7fad02200700 1 -- 172.21.15.89:0/4087835028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facfc1578a0 con 0x7facfc0cbb30 2024-04-02T07:51:24.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.678+0000 7fad02200700 1 -- 172.21.15.89:0/4087835028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facfc067720 con 0x7facfc0cbb30 2024-04-02T07:51:24.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.679+0000 7facf3400700 1 -- 172.21.15.89:0/4087835028 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7face8013c80 con 0x7facfc0cbb30 2024-04-02T07:51:24.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.680+0000 7facf3400700 1 --2- 172.21.15.89:0/4087835028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7facdc06adb0 0x7facdc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:24.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.680+0000 7facf3400700 1 -- 172.21.15.89:0/4087835028 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5893+0+0 (secure 0 0 0) 0x7face80892e0 con 0x7facfc0cbb30 2024-04-02T07:51:24.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.680+0000 7facfb400700 1 --2- 172.21.15.89:0/4087835028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7facdc06adb0 0x7facdc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:24.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.681+0000 7facfb400700 1 --2- 172.21.15.89:0/4087835028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7facdc06adb0 0x7facdc06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7facec006fd0 tx=0x7facec008040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:24.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.685+0000 7facf3400700 1 -- 172.21.15.89:0/4087835028 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7face80541d0 con 0x7facfc0cbb30 2024-04-02T07:51:24.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:24.900+0000 7fad02200700 1 -- 172.21.15.89:0/4087835028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7facfc15bda0 con 0x7facfc0cbb30 2024-04-02T07:51:25.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.641+0000 7facf3400700 1 -- 172.21.15.89:0/4087835028 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v79) v1 ==== 97+0+0 (secure 0 0 0) 0x7face80577f0 con 0x7facfc0cbb30 2024-04-02T07:51:25.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.644+0000 7fad02200700 1 -- 172.21.15.89:0/4087835028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7facdc06adb0 msgr2=0x7facdc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:25.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.644+0000 7fad02200700 1 --2- 172.21.15.89:0/4087835028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7facdc06adb0 0x7facdc06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7facec006fd0 tx=0x7facec008040 comp rx=0 tx=0).stop 2024-04-02T07:51:25.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.644+0000 7fad02200700 1 -- 172.21.15.89:0/4087835028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc0cbb30 msgr2=0x7facfc06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:25.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.644+0000 7fad02200700 1 --2- 172.21.15.89:0/4087835028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc0cbb30 0x7facfc06e320 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7face800e810 tx=0x7face800ebd0 comp rx=0 tx=0).stop 2024-04-02T07:51:25.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.644+0000 7fad02200700 1 -- 172.21.15.89:0/4087835028 shutdown_connections 2024-04-02T07:51:25.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.644+0000 7fad02200700 1 --2- 172.21.15.89:0/4087835028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7facdc06adb0 0x7facdc06d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:25.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.644+0000 7fad02200700 1 --2- 172.21.15.89:0/4087835028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facfc0b2070 0x7facfc157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:25.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.644+0000 7fad02200700 1 --2- 172.21.15.89:0/4087835028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facfc0cd020 0x7facfc06e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:25.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.644+0000 7fad02200700 1 --2- 172.21.15.89:0/4087835028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc0cbb30 0x7facfc06e320 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:25.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.644+0000 7fad02200700 1 -- 172.21.15.89:0/4087835028 >> 172.21.15.89:0/4087835028 conn(0x7facfc0b0d70 msgr2=0x7facfc15e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:25.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.645+0000 7fad02200700 1 -- 172.21.15.89:0/4087835028 shutdown_connections 2024-04-02T07:51:25.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.645+0000 7fad02200700 1 -- 172.21.15.89:0/4087835028 wait complete. 2024-04-02T07:51:25.647 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T07:51:25.668 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T07:51:25.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.860+0000 7ff806000700 1 Processor -- start 2024-04-02T07:51:25.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.860+0000 7ff806000700 1 -- start start 2024-04-02T07:51:25.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.861+0000 7ff806000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff800072580 0x7ff8000b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.861+0000 7ff806000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8000b2e50 0x7ff8000709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.862+0000 7ff806000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8000b2180 con 0x7ff8000b3af0 2024-04-02T07:51:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.862+0000 7ff806000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff80015b100 con 0x7ff800072580 2024-04-02T07:51:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.862+0000 7ff806000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80015b3e0 con 0x7ff8000b2e50 2024-04-02T07:51:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.862+0000 7ff7ff400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff8000b3af0 0x7ff80015a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36512/0 (socket says 172.21.15.89:36512) 2024-04-02T07:51:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.862+0000 7ff7ff400700 1 -- 172.21.15.89:0/3987523791 learned_addr learned my addr 172.21.15.89:0/3987523791 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.862+0000 7ff7fea00700 1 --2- 172.21.15.89:0/3987523791 >> v2:172.21.15.113:3300/0 conn(0x7ff800072580 0x7ff8000b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.862+0000 7ff7ffe00700 1 --2- 172.21.15.89:0/3987523791 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8000b2e50 0x7ff8000709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.862+0000 7ff7fe000700 1 -- 172.21.15.89:0/3987523791 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434315130 0 0) 0x7ff8000b2180 con 0x7ff8000b3af0 2024-04-02T07:51:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.862+0000 7ff7fe000700 1 -- 172.21.15.89:0/3987523791 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e40035b0 con 0x7ff8000b3af0 2024-04-02T07:51:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.862+0000 7ff7fe000700 1 -- 172.21.15.89:0/3987523791 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3992009770 0 0) 0x7ff7e40035b0 con 0x7ff8000b3af0 2024-04-02T07:51:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7fe000700 1 -- 172.21.15.89:0/3987523791 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8000b2e50 msgr2=0x7ff8000709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7fe000700 1 --2- 172.21.15.89:0/3987523791 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8000b2e50 0x7ff8000709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7fe000700 1 -- 172.21.15.89:0/3987523791 >> v2:172.21.15.113:3300/0 conn(0x7ff800072580 msgr2=0x7ff8000b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7fe000700 1 --2- 172.21.15.89:0/3987523791 >> v2:172.21.15.113:3300/0 conn(0x7ff800072580 0x7ff8000b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7fe000700 1 -- 172.21.15.89:0/3987523791 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8000b23f0 con 0x7ff8000b3af0 2024-04-02T07:51:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7fe000700 1 -- 172.21.15.89:0/3987523791 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7ff7f0001670 con 0x7ff8000b3af0 2024-04-02T07:51:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7fe000700 1 -- 172.21.15.89:0/3987523791 >> v1:172.21.15.89:6789/0 conn(0x7ff8000b3af0 legacy=0x7ff80015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7fe000700 1 --2- 172.21.15.89:0/3987523791 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8000b2e50 0x7ff7e4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7fe000700 1 --2- 172.21.15.89:0/3987523791 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7e4003f80 0x7ff7e4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7fe000700 1 --2- 172.21.15.89:0/3987523791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7e4004d90 0x7ff7e4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7fe000700 1 -- 172.21.15.89:0/3987523791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80015b100 con 0x7ff7e4004d90 2024-04-02T07:51:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7fe000700 1 -- 172.21.15.89:0/3987523791 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80015b3e0 con 0x7ff7e4003f80 2024-04-02T07:51:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7fe000700 1 -- 172.21.15.89:0/3987523791 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e400ba10 con 0x7ff8000b2e50 2024-04-02T07:51:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7ffe00700 1 --2- 172.21.15.89:0/3987523791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7e4004d90 0x7ff7e4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7ff400700 1 --2- 172.21.15.89:0/3987523791 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8000b2e50 0x7ff7e4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7fea00700 1 --2- 172.21.15.89:0/3987523791 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7e4003f80 0x7ff7e4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7ffe00700 1 -- 172.21.15.89:0/3987523791 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8000b2e50 msgr2=0x7ff7e4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7ffe00700 1 --2- 172.21.15.89:0/3987523791 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8000b2e50 0x7ff7e4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7ffe00700 1 -- 172.21.15.89:0/3987523791 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7e4003f80 msgr2=0x7ff7e4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7ffe00700 1 --2- 172.21.15.89:0/3987523791 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7e4003f80 0x7ff7e4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.863+0000 7ff7ffe00700 1 -- 172.21.15.89:0/3987523791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e40035b0 con 0x7ff7e4004d90 2024-04-02T07:51:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.864+0000 7ff7ffe00700 1 --2- 172.21.15.89:0/3987523791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7e4004d90 0x7ff7e4005210 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8010480 tx=0x7ff7e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39be6c43a272d630 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.864+0000 7ff7fe000700 1 -- 172.21.15.89:0/3987523791 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8010ec0 con 0x7ff7e4004d90 2024-04-02T07:51:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.864+0000 7ff7fe000700 1 -- 172.21.15.89:0/3987523791 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f8012430 con 0x7ff7e4004d90 2024-04-02T07:51:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.864+0000 7ff7fe000700 1 -- 172.21.15.89:0/3987523791 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f80127a0 con 0x7ff7e4004d90 2024-04-02T07:51:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.864+0000 7ff806000700 1 -- 172.21.15.89:0/3987523791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7e4004d90 msgr2=0x7ff7e4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.864+0000 7ff806000700 1 --2- 172.21.15.89:0/3987523791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7e4004d90 0x7ff7e4005210 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8010480 tx=0x7ff7e4005880 comp rx=0 tx=0).stop 2024-04-02T07:51:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.864+0000 7ff7ffe00700 1 -- 172.21.15.89:0/3987523791 reap_dead start 2024-04-02T07:51:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.864+0000 7ff806000700 1 -- 172.21.15.89:0/3987523791 shutdown_connections 2024-04-02T07:51:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.864+0000 7ff806000700 1 -- 172.21.15.89:0/3987523791 >> 172.21.15.89:0/3987523791 conn(0x7ff8000cf190 msgr2=0x7ff8000cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.865+0000 7ff806000700 1 -- 172.21.15.89:0/3987523791 shutdown_connections 2024-04-02T07:51:25.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.865+0000 7ff806000700 1 -- 172.21.15.89:0/3987523791 wait complete. 2024-04-02T07:51:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.866+0000 7ff806000700 1 Processor -- start 2024-04-02T07:51:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.866+0000 7ff806000700 1 -- start start 2024-04-02T07:51:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.867+0000 7ff806000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7e4004d90 0x7ff800150c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.867+0000 7ff806000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8000c9370 0x7ff8001486c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.867+0000 7ff806000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8001511e0 0x7ff8000aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.867+0000 7ff806000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff800151cf0 con 0x7ff7e4004d90 2024-04-02T07:51:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.867+0000 7ff806000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80015c050 con 0x7ff8001511e0 2024-04-02T07:51:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.867+0000 7ff806000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff800148c00 con 0x7ff8000c9370 2024-04-02T07:51:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7ff400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7e4004d90 0x7ff800150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7ff400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7e4004d90 0x7ff800150c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54106/0 (socket says 172.21.15.89:54106) 2024-04-02T07:51:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7ff400700 1 -- 172.21.15.89:0/1796357288 learned_addr learned my addr 172.21.15.89:0/1796357288 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7fea00700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8000c9370 0x7ff8001486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7ffe00700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8001511e0 0x7ff8000aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7ff400700 1 -- 172.21.15.89:0/1796357288 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8000c9370 msgr2=0x7ff8001486c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7ff400700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8000c9370 0x7ff8001486c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7ff400700 1 -- 172.21.15.89:0/1796357288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8001511e0 msgr2=0x7ff8000aec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7ff400700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8001511e0 0x7ff8000aec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7ff400700 1 -- 172.21.15.89:0/1796357288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff800151a10 con 0x7ff7e4004d90 2024-04-02T07:51:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7ffe00700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8001511e0 0x7ff8000aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7ff400700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7e4004d90 0x7ff800150c80 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff7f00017a0 tx=0x7ff7f00076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7f7400700 1 -- 172.21.15.89:0/1796357288 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0007820 con 0x7ff7e4004d90 2024-04-02T07:51:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7f7400700 1 -- 172.21.15.89:0/1796357288 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f000f8d0 con 0x7ff7e4004d90 2024-04-02T07:51:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff806000700 1 -- 172.21.15.89:0/1796357288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80015ba00 con 0x7ff7e4004d90 2024-04-02T07:51:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.868+0000 7ff7f7400700 1 -- 172.21.15.89:0/1796357288 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f000fc80 con 0x7ff7e4004d90 2024-04-02T07:51:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.869+0000 7ff806000700 1 -- 172.21.15.89:0/1796357288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8000c97f0 con 0x7ff7e4004d90 2024-04-02T07:51:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.870+0000 7ff806000700 1 -- 172.21.15.89:0/1796357288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff800067720 con 0x7ff7e4004d90 2024-04-02T07:51:25.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.870+0000 7ff7f7400700 1 -- 172.21.15.89:0/1796357288 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff7f001e3b0 con 0x7ff7e4004d90 2024-04-02T07:51:25.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.871+0000 7ff7f7400700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff7e006adb0 0x7ff7e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:25.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.871+0000 7ff7f7400700 1 -- 172.21.15.89:0/1796357288 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5205+0+0 (secure 0 0 0) 0x7ff7f0088e50 con 0x7ff7e4004d90 2024-04-02T07:51:25.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.872+0000 7ff7fea00700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff7e006adb0 0x7ff7e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:25.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.872+0000 7ff7fea00700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff7e006adb0 0x7ff7e006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff800072a30 tx=0x7ff7ec009210 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:25.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:25.876+0000 7ff7f7400700 1 -- 172.21.15.89:0/1796357288 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff7f0054100 con 0x7ff7e4004d90 2024-04-02T07:51:25.995 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-04-02T07:51:26.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.160+0000 7ff806000700 1 -- 172.21.15.89:0/1796357288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff8000cef70 con 0x7ff7e4004d90 2024-04-02T07:51:26.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.160+0000 7ff7f7400700 1 -- 172.21.15.89:0/1796357288 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff7f001a090 con 0x7ff7e4004d90 2024-04-02T07:51:26.162 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:26.162 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:51:26.162 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T07:51:26.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.162 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T07:51:26.162 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T07:51:26.162 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:51:26.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:51:26.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T07:51:26.164 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T07:51:26.164 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:51:26.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:51:26.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:51:26.164 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.164 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.164 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T07:51:26.164 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T07:51:26.164 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:51:26.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:51:26.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:51:26.165 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.165 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.165 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T07:51:26.165 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T07:51:26.165 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:51:26.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:51:26.165 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.165 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.166 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T07:51:26.166 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T07:51:26.166 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:51:26.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:51:26.166 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.166 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.166 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T07:51:26.166 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T07:51:26.166 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:51:26.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T07:51:26.167 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.169 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T07:51:26.169 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T07:51:26.169 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:51:26.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:51:26.169 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.169 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.169 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T07:51:26.169 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T07:51:26.169 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:51:26.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:51:26.170 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.170 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.170 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T07:51:26.170 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T07:51:26.170 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:51:26.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2024-04-02T07:51:26.170 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T07:51:26.170 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:51:26.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:51:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:51:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:51:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T07:51:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T07:51:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:51:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:51:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:51:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:51:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:51:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:51:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:51:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T07:51:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T07:51:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:51:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:51:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:51:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:51:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T07:51:26.175 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:51:26.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.162+0000 7ff806000700 1 -- 172.21.15.89:0/1796357288 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff7e006adb0 msgr2=0x7ff7e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:26.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.162+0000 7ff806000700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff7e006adb0 0x7ff7e006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff800072a30 tx=0x7ff7ec009210 comp rx=0 tx=0).stop 2024-04-02T07:51:26.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.162+0000 7ff806000700 1 -- 172.21.15.89:0/1796357288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7e4004d90 msgr2=0x7ff800150c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:26.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.162+0000 7ff806000700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7e4004d90 0x7ff800150c80 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff7f00017a0 tx=0x7ff7f00076a0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.163+0000 7ff806000700 1 -- 172.21.15.89:0/1796357288 shutdown_connections 2024-04-02T07:51:26.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.163+0000 7ff806000700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff7e006adb0 0x7ff7e006d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.163+0000 7ff806000700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8001511e0 0x7ff8000aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.163+0000 7ff806000700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7e4004d90 0x7ff800150c80 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.163+0000 7ff806000700 1 --2- 172.21.15.89:0/1796357288 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8000c9370 0x7ff8001486c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.163+0000 7ff806000700 1 -- 172.21.15.89:0/1796357288 >> 172.21.15.89:0/1796357288 conn(0x7ff8000cf190 msgr2=0x7ff8000cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:26.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.163+0000 7ff806000700 1 -- 172.21.15.89:0/1796357288 shutdown_connections 2024-04-02T07:51:26.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.163+0000 7ff806000700 1 -- 172.21.15.89:0/1796357288 wait complete. 2024-04-02T07:51:26.184 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:51:26.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.373+0000 7f95aee00700 1 Processor -- start 2024-04-02T07:51:26.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.373+0000 7f95aee00700 1 -- start start 2024-04-02T07:51:26.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.374+0000 7f95aee00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f95a8160f10 0x7f95a80d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:26.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.374+0000 7f95aee00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f95a80cf230 0x7f95a80d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:26.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.374+0000 7f95aee00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95a8161590 0x7f95a8161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:26.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.374+0000 7f95aee00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a8161f60 con 0x7f95a80cf230 2024-04-02T07:51:26.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.374+0000 7f95aee00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a80ceda0 con 0x7f95a8160f10 2024-04-02T07:51:26.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.374+0000 7f95aee00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a80d23d0 con 0x7f95a8161590 2024-04-02T07:51:26.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.375+0000 7f959fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f95a80cf230 0x7f95a80d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.375+0000 7f959fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f95a80cf230 0x7f95a80d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54116/0 (socket says 172.21.15.89:54116) 2024-04-02T07:51:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.375+0000 7f959fe00700 1 -- 172.21.15.89:0/364969257 learned_addr learned my addr 172.21.15.89:0/364969257 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.375+0000 7f95ace00700 1 --2- 172.21.15.89:0/364969257 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95a8161590 0x7f95a8161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.375+0000 7f95a7e00700 1 --2- 172.21.15.89:0/364969257 >> v2:172.21.15.113:3300/0 conn(0x7f95a8160f10 0x7f95a80d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.375+0000 7f959fe00700 1 -- 172.21.15.89:0/364969257 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95a8161590 msgr2=0x7f95a8161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.375+0000 7f959fe00700 1 --2- 172.21.15.89:0/364969257 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95a8161590 0x7f95a8161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.375+0000 7f959fe00700 1 -- 172.21.15.89:0/364969257 >> v2:172.21.15.113:3300/0 conn(0x7f95a8160f10 msgr2=0x7f95a80d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.375+0000 7f959fe00700 1 --2- 172.21.15.89:0/364969257 >> v2:172.21.15.113:3300/0 conn(0x7f95a8160f10 0x7f95a80d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.375+0000 7f959fe00700 1 -- 172.21.15.89:0/364969257 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a81612f0 con 0x7f95a80cf230 2024-04-02T07:51:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.375+0000 7f959fe00700 1 --2- 172.21.15.89:0/364969257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a80cf230 0x7f95a80d1e90 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f95a80d1080 tx=0x7f9590009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b903f2960049ac54 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.376+0000 7f95a7400700 1 -- 172.21.15.89:0/364969257 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959000b870 con 0x7f95a80cf230 2024-04-02T07:51:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.376+0000 7f95a7400700 1 -- 172.21.15.89:0/364969257 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959000b9d0 con 0x7f95a80cf230 2024-04-02T07:51:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.376+0000 7f95a7400700 1 -- 172.21.15.89:0/364969257 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959000bd60 con 0x7f95a80cf230 2024-04-02T07:51:26.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.376+0000 7f95aee00700 1 -- 172.21.15.89:0/364969257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a80cf230 msgr2=0x7f95a80d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:26.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.376+0000 7f95aee00700 1 --2- 172.21.15.89:0/364969257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a80cf230 0x7f95a80d1e90 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f95a80d1080 tx=0x7f9590009bd0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.376+0000 7f95aee00700 1 -- 172.21.15.89:0/364969257 shutdown_connections 2024-04-02T07:51:26.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.376+0000 7f95aee00700 1 --2- 172.21.15.89:0/364969257 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95a8161590 0x7f95a8161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.376+0000 7f95aee00700 1 --2- 172.21.15.89:0/364969257 >> v2:172.21.15.113:3300/0 conn(0x7f95a8160f10 0x7f95a80d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.376+0000 7f95aee00700 1 --2- 172.21.15.89:0/364969257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a80cf230 0x7f95a80d1e90 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.376+0000 7f95aee00700 1 -- 172.21.15.89:0/364969257 >> 172.21.15.89:0/364969257 conn(0x7f95a808aff0 msgr2=0x7f95a80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:26.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.376+0000 7f95aee00700 1 -- 172.21.15.89:0/364969257 shutdown_connections 2024-04-02T07:51:26.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.376+0000 7f95aee00700 1 -- 172.21.15.89:0/364969257 wait complete. 2024-04-02T07:51:26.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.378+0000 7f95aee00700 1 Processor -- start 2024-04-02T07:51:26.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.378+0000 7f95aee00700 1 -- start start 2024-04-02T07:51:26.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.379+0000 7f95aee00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95a80cf230 0x7f95a814ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.379+0000 7f95aee00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95a8160f10 0x7f95a815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.379+0000 7f95aee00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a8161590 0x7f95a815f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.379+0000 7f95aee00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a81568d0 con 0x7f95a8161590 2024-04-02T07:51:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.379+0000 7f95aee00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a815fd00 con 0x7f95a8160f10 2024-04-02T07:51:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.379+0000 7f95aee00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a815ffe0 con 0x7f95a80cf230 2024-04-02T07:51:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.379+0000 7f95ace00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a8161590 0x7f95a815f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.379+0000 7f95ace00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a8161590 0x7f95a815f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54132/0 (socket says 172.21.15.89:54132) 2024-04-02T07:51:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.379+0000 7f95ace00700 1 -- 172.21.15.89:0/2423275990 learned_addr learned my addr 172.21.15.89:0/2423275990 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.380+0000 7f959fe00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95a8160f10 0x7f95a815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.380+0000 7f95a7e00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95a80cf230 0x7f95a814ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.380+0000 7f95ace00700 1 -- 172.21.15.89:0/2423275990 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95a80cf230 msgr2=0x7f95a814ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.380+0000 7f95ace00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95a80cf230 0x7f95a814ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.380+0000 7f95ace00700 1 -- 172.21.15.89:0/2423275990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95a8160f10 msgr2=0x7f95a815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.380+0000 7f95ace00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95a8160f10 0x7f95a815ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.380+0000 7f95ace00700 1 -- 172.21.15.89:0/2423275990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9590009550 con 0x7f95a8161590 2024-04-02T07:51:26.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.380+0000 7f95a7e00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95a80cf230 0x7f95a814ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:26.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.380+0000 7f95ace00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a8161590 0x7f95a815f4d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f95a0007e60 tx=0x7f95a000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:26.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.380+0000 7f95a4c00700 1 -- 172.21.15.89:0/2423275990 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a000ede0 con 0x7f95a8161590 2024-04-02T07:51:26.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.380+0000 7f95a4c00700 1 -- 172.21.15.89:0/2423275990 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a0014420 con 0x7f95a8161590 2024-04-02T07:51:26.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.380+0000 7f95aee00700 1 -- 172.21.15.89:0/2423275990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a8156280 con 0x7f95a8161590 2024-04-02T07:51:26.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.380+0000 7f95a4c00700 1 -- 172.21.15.89:0/2423275990 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a00147d0 con 0x7f95a8161590 2024-04-02T07:51:26.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.381+0000 7f95aee00700 1 -- 172.21.15.89:0/2423275990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a815fa10 con 0x7f95a8161590 2024-04-02T07:51:26.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.381+0000 7f95aee00700 1 -- 172.21.15.89:0/2423275990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a8156a40 con 0x7f95a8161590 2024-04-02T07:51:26.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.382+0000 7f95a4c00700 1 -- 172.21.15.89:0/2423275990 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f95a0014930 con 0x7f95a8161590 2024-04-02T07:51:26.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.383+0000 7f95a4c00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f958406ade0 0x7f958406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:26.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.383+0000 7f95a4c00700 1 -- 172.21.15.89:0/2423275990 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f95a0089ad0 con 0x7f95a8161590 2024-04-02T07:51:26.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.383+0000 7f95a7e00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f958406ade0 0x7f958406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:26.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.384+0000 7f95a7e00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f958406ade0 0x7f958406d2a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9598009820 tx=0x7f9598008040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:26.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.388+0000 7f95a4c00700 1 -- 172.21.15.89:0/2423275990 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f95a0054c60 con 0x7f95a8161590 2024-04-02T07:51:26.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.589+0000 7f95aee00700 1 -- 172.21.15.89:0/2423275990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95a8067720 con 0x7f95a8161590 2024-04-02T07:51:26.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.590+0000 7f95a4c00700 1 -- 172.21.15.89:0/2423275990 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+12916 (secure 0 0 0) 0x7f95a0058280 con 0x7f95a8161590 2024-04-02T07:51:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:26.592 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":79,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:51:25.633793+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:51:26.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.592+0000 7f95aee00700 1 -- 172.21.15.89:0/2423275990 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f958406ade0 msgr2=0x7f958406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:26.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.592+0000 7f95aee00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f958406ade0 0x7f958406d2a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9598009820 tx=0x7f9598008040 comp rx=0 tx=0).stop 2024-04-02T07:51:26.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.593+0000 7f95aee00700 1 -- 172.21.15.89:0/2423275990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a8161590 msgr2=0x7f95a815f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:26.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.593+0000 7f95aee00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a8161590 0x7f95a815f4d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f95a0007e60 tx=0x7f95a000e3c0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.593+0000 7f95aee00700 1 -- 172.21.15.89:0/2423275990 shutdown_connections 2024-04-02T07:51:26.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.593+0000 7f95aee00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f958406ade0 0x7f958406d2a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.593+0000 7f95aee00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a8161590 0x7f95a815f4d0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.593+0000 7f95aee00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f95a80cf230 0x7f95a814ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.593+0000 7f95aee00700 1 --2- 172.21.15.89:0/2423275990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95a8160f10 0x7f95a815ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.593+0000 7f95aee00700 1 -- 172.21.15.89:0/2423275990 >> 172.21.15.89:0/2423275990 conn(0x7f95a808aff0 msgr2=0x7f95a80cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:26.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.593+0000 7f95aee00700 1 -- 172.21.15.89:0/2423275990 shutdown_connections 2024-04-02T07:51:26.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.593+0000 7f95aee00700 1 -- 172.21.15.89:0/2423275990 wait complete. 2024-04-02T07:51:26.612 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:51:26.613 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:51:26.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.806+0000 7f2127200700 1 Processor -- start 2024-04-02T07:51:26.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.806+0000 7f2127200700 1 -- start start 2024-04-02T07:51:26.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.807+0000 7f2127200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f212015d9d0 0x7f21200d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:26.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.807+0000 7f2127200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f21200d2540 0x7f21200ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:26.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.807+0000 7f2127200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2120161420 con 0x7f21200d16a0 2024-04-02T07:51:26.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.808+0000 7f2127200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f212015edc0 con 0x7f212015d9d0 2024-04-02T07:51:26.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.808+0000 7f2127200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2120161700 con 0x7f21200d2540 2024-04-02T07:51:26.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.808+0000 7f211fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f212015d9d0 0x7f21200d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:26.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.808+0000 7f211fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f212015d9d0 0x7f21200d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54138/0 (socket says 172.21.15.89:54138) 2024-04-02T07:51:26.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.808+0000 7f211fe00700 1 -- 172.21.15.89:0/657729788 learned_addr learned my addr 172.21.15.89:0/657729788 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:26.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.808+0000 7f2117e00700 1 --1- 172.21.15.89:0/657729788 >> v1:172.21.15.89:6789/0 conn(0x7f21200d16a0 0x7f21200d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36526/0 (socket says 172.21.15.89:36526) 2024-04-02T07:51:26.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.808+0000 7f2125200700 1 --2- 172.21.15.89:0/657729788 >> v2:172.21.15.113:3300/0 conn(0x7f21200d2540 0x7f21200ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.808+0000 7f211fe00700 1 -- 172.21.15.89:0/657729788 >> v2:172.21.15.113:3300/0 conn(0x7f21200d2540 msgr2=0x7f21200ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.808+0000 7f211fe00700 1 --2- 172.21.15.89:0/657729788 >> v2:172.21.15.113:3300/0 conn(0x7f21200d2540 0x7f21200ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.808+0000 7f211fe00700 1 -- 172.21.15.89:0/657729788 >> v1:172.21.15.89:6789/0 conn(0x7f21200d16a0 legacy=0x7f21200d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.808+0000 7f211fe00700 1 -- 172.21.15.89:0/657729788 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21201619e0 con 0x7f212015d9d0 2024-04-02T07:51:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.808+0000 7f211f400700 1 -- 172.21.15.89:0/657729788 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027662633 0 0) 0x7f2120161420 con 0x7f21200d16a0 2024-04-02T07:51:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.809+0000 7f211fe00700 1 --2- 172.21.15.89:0/657729788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f212015d9d0 0x7f21200d3c30 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f212008ca10 tx=0x7f2110009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee287a358e754e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.809+0000 7f211f400700 1 -- 172.21.15.89:0/657729788 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211000b870 con 0x7f212015d9d0 2024-04-02T07:51:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.809+0000 7f211f400700 1 -- 172.21.15.89:0/657729788 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211000b9d0 con 0x7f212015d9d0 2024-04-02T07:51:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.809+0000 7f211f400700 1 -- 172.21.15.89:0/657729788 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211000bd60 con 0x7f212015d9d0 2024-04-02T07:51:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.809+0000 7f2127200700 1 -- 172.21.15.89:0/657729788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f212015d9d0 msgr2=0x7f21200d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.809+0000 7f2127200700 1 --2- 172.21.15.89:0/657729788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f212015d9d0 0x7f21200d3c30 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f212008ca10 tx=0x7f2110009bd0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.809+0000 7f2127200700 1 -- 172.21.15.89:0/657729788 shutdown_connections 2024-04-02T07:51:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.809+0000 7f2127200700 1 --2- 172.21.15.89:0/657729788 >> v2:172.21.15.113:3300/0 conn(0x7f21200d2540 0x7f21200ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.809+0000 7f2127200700 1 --2- 172.21.15.89:0/657729788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f212015d9d0 0x7f21200d3c30 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.809+0000 7f2127200700 1 -- 172.21.15.89:0/657729788 >> 172.21.15.89:0/657729788 conn(0x7f212008aff0 msgr2=0x7f21200d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.809+0000 7f2127200700 1 -- 172.21.15.89:0/657729788 shutdown_connections 2024-04-02T07:51:26.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.809+0000 7f2127200700 1 -- 172.21.15.89:0/657729788 wait complete. 2024-04-02T07:51:26.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.811+0000 7f2127200700 1 Processor -- start 2024-04-02T07:51:26.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.811+0000 7f2127200700 1 -- start start 2024-04-02T07:51:26.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.812+0000 7f2127200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21200d16a0 0x7f21200b7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:26.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.812+0000 7f2127200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21200d2540 0x7f212015c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:26.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.812+0000 7f2127200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f212015d9d0 0x7f21201e8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:26.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.812+0000 7f2127200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21200b6a30 con 0x7f21200d2540 2024-04-02T07:51:26.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.812+0000 7f2127200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21200b6cd0 con 0x7f212015d9d0 2024-04-02T07:51:26.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.812+0000 7f2127200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21200b6fb0 con 0x7f21200d16a0 2024-04-02T07:51:26.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.812+0000 7f2117e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21200d2540 0x7f212015c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:26.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.812+0000 7f2117e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21200d2540 0x7f212015c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54148/0 (socket says 172.21.15.89:54148) 2024-04-02T07:51:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.812+0000 7f2117e00700 1 -- 172.21.15.89:0/4122842857 learned_addr learned my addr 172.21.15.89:0/4122842857 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.813+0000 7f2125200700 1 --2- 172.21.15.89:0/4122842857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f212015d9d0 0x7f21201e8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.813+0000 7f211fe00700 1 --2- 172.21.15.89:0/4122842857 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21200d16a0 0x7f21200b7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.813+0000 7f2117e00700 1 -- 172.21.15.89:0/4122842857 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21200d16a0 msgr2=0x7f21200b7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.813+0000 7f2117e00700 1 --2- 172.21.15.89:0/4122842857 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21200d16a0 0x7f21200b7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.813+0000 7f2117e00700 1 -- 172.21.15.89:0/4122842857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f212015d9d0 msgr2=0x7f21201e8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.813+0000 7f2117e00700 1 --2- 172.21.15.89:0/4122842857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f212015d9d0 0x7f21201e8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.813+0000 7f2117e00700 1 -- 172.21.15.89:0/4122842857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2110009550 con 0x7f21200d2540 2024-04-02T07:51:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.813+0000 7f2117e00700 1 --2- 172.21.15.89:0/4122842857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21200d2540 0x7f212015c000 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f210800c500 tx=0x7f210800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.813+0000 7f211cc00700 1 -- 172.21.15.89:0/4122842857 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21080074e0 con 0x7f21200d2540 2024-04-02T07:51:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.813+0000 7f211cc00700 1 -- 172.21.15.89:0/4122842857 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2108007640 con 0x7f21200d2540 2024-04-02T07:51:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.813+0000 7f211cc00700 1 -- 172.21.15.89:0/4122842857 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2108005a50 con 0x7f21200d2540 2024-04-02T07:51:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.813+0000 7f2127200700 1 -- 172.21.15.89:0/4122842857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21200b7290 con 0x7f21200d2540 2024-04-02T07:51:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.814+0000 7f2127200700 1 -- 172.21.15.89:0/4122842857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21200b7850 con 0x7f21200d2540 2024-04-02T07:51:26.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.815+0000 7f2127200700 1 -- 172.21.15.89:0/4122842857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21200d1ca0 con 0x7f21200d2540 2024-04-02T07:51:26.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.815+0000 7f211cc00700 1 -- 172.21.15.89:0/4122842857 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2108005bb0 con 0x7f21200d2540 2024-04-02T07:51:26.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.816+0000 7f211cc00700 1 --2- 172.21.15.89:0/4122842857 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f20fc06adb0 0x7f20fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:26.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.816+0000 7f211cc00700 1 -- 172.21.15.89:0/4122842857 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f21080887c0 con 0x7f21200d2540 2024-04-02T07:51:26.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.817+0000 7f211fe00700 1 --2- 172.21.15.89:0/4122842857 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f20fc06adb0 0x7f20fc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:26.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.817+0000 7f211fe00700 1 --2- 172.21.15.89:0/4122842857 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f20fc06adb0 0x7f20fc06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2110000c00 tx=0x7f2110008030 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:26.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:26.821+0000 7f211cc00700 1 -- 172.21.15.89:0/4122842857 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f21080539e0 con 0x7f21200d2540 2024-04-02T07:51:27.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.021+0000 7f2127200700 1 -- 172.21.15.89:0/4122842857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f21200b7a40 con 0x7f21200d2540 2024-04-02T07:51:27.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.022+0000 7f211cc00700 1 -- 172.21.15.89:0/4122842857 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f21080170a0 con 0x7f21200d2540 2024-04-02T07:51:27.024 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:51:27.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.024+0000 7f2127200700 1 -- 172.21.15.89:0/4122842857 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f20fc06adb0 msgr2=0x7f20fc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:27.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.024+0000 7f2127200700 1 --2- 172.21.15.89:0/4122842857 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f20fc06adb0 0x7f20fc06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2110000c00 tx=0x7f2110008030 comp rx=0 tx=0).stop 2024-04-02T07:51:27.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.024+0000 7f2127200700 1 -- 172.21.15.89:0/4122842857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21200d2540 msgr2=0x7f212015c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:27.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.024+0000 7f2127200700 1 --2- 172.21.15.89:0/4122842857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21200d2540 0x7f212015c000 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f210800c500 tx=0x7f210800c810 comp rx=0 tx=0).stop 2024-04-02T07:51:27.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.024+0000 7f2127200700 1 -- 172.21.15.89:0/4122842857 shutdown_connections 2024-04-02T07:51:27.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.024+0000 7f2127200700 1 --2- 172.21.15.89:0/4122842857 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f20fc06adb0 0x7f20fc06d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.024+0000 7f2127200700 1 --2- 172.21.15.89:0/4122842857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f212015d9d0 0x7f21201e8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.024+0000 7f2127200700 1 --2- 172.21.15.89:0/4122842857 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f21200d16a0 0x7f21200b7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.024+0000 7f2127200700 1 --2- 172.21.15.89:0/4122842857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21200d2540 0x7f212015c000 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.024+0000 7f2127200700 1 -- 172.21.15.89:0/4122842857 >> 172.21.15.89:0/4122842857 conn(0x7f212008aff0 msgr2=0x7f21200d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:27.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.024+0000 7f2127200700 1 -- 172.21.15.89:0/4122842857 shutdown_connections 2024-04-02T07:51:27.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.024+0000 7f2127200700 1 -- 172.21.15.89:0/4122842857 wait complete. 2024-04-02T07:51:27.044 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T07:51:27.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.235+0000 7f5536200700 1 Processor -- start 2024-04-02T07:51:27.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.235+0000 7f5536200700 1 -- start start 2024-04-02T07:51:27.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.236+0000 7f5536200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f553015b7e0 0x7f5530099b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:27.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.236+0000 7f5536200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55300b39d0 con 0x7f553015f200 2024-04-02T07:51:27.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.236+0000 7f5536200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55300b3410 con 0x7f55300b2780 2024-04-02T07:51:27.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.236+0000 7f5536200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55300b36f0 con 0x7f553015b7e0 2024-04-02T07:51:27.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.236+0000 7f552fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f553015f200 0x7f55300ce1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36532/0 (socket says 172.21.15.89:36532) 2024-04-02T07:51:27.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.236+0000 7f552fe00700 1 -- 172.21.15.89:0/1642210020 learned_addr learned my addr 172.21.15.89:0/1642210020 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:27.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 -- 172.21.15.89:0/1642210020 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104927819 0 0) 0x7f55300b39d0 con 0x7f553015f200 2024-04-02T07:51:27.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 -- 172.21.15.89:0/1642210020 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55140035b0 con 0x7f553015f200 2024-04-02T07:51:27.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552f400700 1 --2- 172.21.15.89:0/1642210020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f553015b7e0 0x7f5530099b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:27.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 -- 172.21.15.89:0/1642210020 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1274790631 0 0) 0x7f55140035b0 con 0x7f553015f200 2024-04-02T07:51:27.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 -- 172.21.15.89:0/1642210020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f553015b7e0 msgr2=0x7f5530099b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:27.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 --2- 172.21.15.89:0/1642210020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f553015b7e0 0x7f5530099b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 -- 172.21.15.89:0/1642210020 >> v1:172.21.15.113:6789/0 conn(0x7f55300b2780 legacy=0x7f553006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:27.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 -- 172.21.15.89:0/1642210020 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553006a8c0 con 0x7f553015f200 2024-04-02T07:51:27.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 -- 172.21.15.89:0/1642210020 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f55280019f0 con 0x7f553015f200 2024-04-02T07:51:27.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 -- 172.21.15.89:0/1642210020 >> v1:172.21.15.89:6789/0 conn(0x7f553015f200 legacy=0x7f55300ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:27.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 --2- 172.21.15.89:0/1642210020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f553015b7e0 0x7f5514003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:27.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 --2- 172.21.15.89:0/1642210020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5514003f70 0x7f5514004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:27.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 --2- 172.21.15.89:0/1642210020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5514004d90 0x7f5514005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:27.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 -- 172.21.15.89:0/1642210020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55300b36f0 con 0x7f5514003f70 2024-04-02T07:51:27.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 -- 172.21.15.89:0/1642210020 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551400bb20 con 0x7f5514004d90 2024-04-02T07:51:27.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552e000700 1 -- 172.21.15.89:0/1642210020 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f551400bce0 con 0x7f553015b7e0 2024-04-02T07:51:27.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552ea00700 1 --2- 172.21.15.89:0/1642210020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5514003f70 0x7f5514004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:27.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.237+0000 7f552f400700 1 --2- 172.21.15.89:0/1642210020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f553015b7e0 0x7f5514003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:27.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.238+0000 7f552fe00700 1 --2- 172.21.15.89:0/1642210020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5514004d90 0x7f5514005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:27.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.238+0000 7f552ea00700 1 -- 172.21.15.89:0/1642210020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f553015b7e0 msgr2=0x7f5514003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:27.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.238+0000 7f552ea00700 1 --2- 172.21.15.89:0/1642210020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f553015b7e0 0x7f5514003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.238+0000 7f552ea00700 1 -- 172.21.15.89:0/1642210020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5514004d90 msgr2=0x7f5514005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:27.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.238+0000 7f552ea00700 1 --2- 172.21.15.89:0/1642210020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5514004d90 0x7f5514005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.238+0000 7f552ea00700 1 -- 172.21.15.89:0/1642210020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55140035b0 con 0x7f5514003f70 2024-04-02T07:51:27.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.238+0000 7f552ea00700 1 --2- 172.21.15.89:0/1642210020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5514003f70 0x7f5514004850 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f551c0018d0 tx=0x7f5514005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87c2bcc7e51acc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:27.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.238+0000 7f552e000700 1 -- 172.21.15.89:0/1642210020 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c00c690 con 0x7f5514003f70 2024-04-02T07:51:27.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.238+0000 7f552e000700 1 -- 172.21.15.89:0/1642210020 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c00c7f0 con 0x7f5514003f70 2024-04-02T07:51:27.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.238+0000 7f552e000700 1 -- 172.21.15.89:0/1642210020 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c00cba0 con 0x7f5514003f70 2024-04-02T07:51:27.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.239+0000 7f5536200700 1 -- 172.21.15.89:0/1642210020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5514003f70 msgr2=0x7f5514004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:27.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.239+0000 7f5536200700 1 --2- 172.21.15.89:0/1642210020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5514003f70 0x7f5514004850 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f551c0018d0 tx=0x7f5514005880 comp rx=0 tx=0).stop 2024-04-02T07:51:27.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.239+0000 7f552fe00700 1 -- 172.21.15.89:0/1642210020 reap_dead start 2024-04-02T07:51:27.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.239+0000 7f5536200700 1 -- 172.21.15.89:0/1642210020 shutdown_connections 2024-04-02T07:51:27.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.239+0000 7f5536200700 1 -- 172.21.15.89:0/1642210020 >> 172.21.15.89:0/1642210020 conn(0x7f55300b5ab0 msgr2=0x7f55300b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:27.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.239+0000 7f5536200700 1 -- 172.21.15.89:0/1642210020 shutdown_connections 2024-04-02T07:51:27.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.239+0000 7f5536200700 1 -- 172.21.15.89:0/1642210020 wait complete. 2024-04-02T07:51:27.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.241+0000 7f5536200700 1 Processor -- start 2024-04-02T07:51:27.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.241+0000 7f5536200700 1 -- start start 2024-04-02T07:51:27.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f5536200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5514003f70 0x7f5530141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:27.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f5536200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f553014e430 0x7f5530159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:27.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f5536200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f553014e8b0 0x7f553015a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:27.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f5536200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553014c5f0 con 0x7f553014e430 2024-04-02T07:51:27.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f5536200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55301f58c0 con 0x7f553014e8b0 2024-04-02T07:51:27.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f5536200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55301f55e0 con 0x7f5514003f70 2024-04-02T07:51:27.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f552ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f553014e430 0x7f5530159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:27.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f552ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f553014e430 0x7f5530159d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54166/0 (socket says 172.21.15.89:54166) 2024-04-02T07:51:27.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f552ea00700 1 -- 172.21.15.89:0/1589954011 learned_addr learned my addr 172.21.15.89:0/1589954011 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:27.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f552f400700 1 --2- 172.21.15.89:0/1589954011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5514003f70 0x7f5530141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:27.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f552fe00700 1 --2- 172.21.15.89:0/1589954011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f553014e8b0 0x7f553015a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:27.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f552ea00700 1 -- 172.21.15.89:0/1589954011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5514003f70 msgr2=0x7f5530141b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:27.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f552ea00700 1 --2- 172.21.15.89:0/1589954011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5514003f70 0x7f5530141b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f552ea00700 1 -- 172.21.15.89:0/1589954011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f553014e8b0 msgr2=0x7f553015a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:27.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f552ea00700 1 --2- 172.21.15.89:0/1589954011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f553014e8b0 0x7f553015a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.242+0000 7f552ea00700 1 -- 172.21.15.89:0/1589954011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55301504c0 con 0x7f553014e430 2024-04-02T07:51:27.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.243+0000 7f552ea00700 1 --2- 172.21.15.89:0/1589954011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f553014e430 0x7f5530159d00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f551400baf0 tx=0x7f551c001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:27.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.243+0000 7f5527400700 1 -- 172.21.15.89:0/1589954011 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c000ed0 con 0x7f553014e430 2024-04-02T07:51:27.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.243+0000 7f5527400700 1 -- 172.21.15.89:0/1589954011 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c021070 con 0x7f553014e430 2024-04-02T07:51:27.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.243+0000 7f5536200700 1 -- 172.21.15.89:0/1589954011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55301507a0 con 0x7f553014e430 2024-04-02T07:51:27.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.243+0000 7f5527400700 1 -- 172.21.15.89:0/1589954011 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c00cba0 con 0x7f553014e430 2024-04-02T07:51:27.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.243+0000 7f5536200700 1 -- 172.21.15.89:0/1589954011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55301420d0 con 0x7f553014e430 2024-04-02T07:51:27.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.244+0000 7f5536200700 1 -- 172.21.15.89:0/1589954011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5530067720 con 0x7f553014e430 2024-04-02T07:51:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.245+0000 7f5527400700 1 -- 172.21.15.89:0/1589954011 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f551c023030 con 0x7f553014e430 2024-04-02T07:51:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.246+0000 7f5527400700 1 --2- 172.21.15.89:0/1589954011 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f551006adb0 0x7f551006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.246+0000 7f5527400700 1 -- 172.21.15.89:0/1589954011 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f551c08a500 con 0x7f553014e430 2024-04-02T07:51:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.247+0000 7f552f400700 1 --2- 172.21.15.89:0/1589954011 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f551006adb0 0x7f551006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:27.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.247+0000 7f552f400700 1 --2- 172.21.15.89:0/1589954011 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f551006adb0 0x7f551006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5520009ba0 tx=0x7f5520008040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:27.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.251+0000 7f5527400700 1 -- 172.21.15.89:0/1589954011 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f551c0558d0 con 0x7f553014e430 2024-04-02T07:51:27.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.455+0000 7f5536200700 1 -- 172.21.15.89:0/1589954011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5530072750 con 0x7f553014e430 2024-04-02T07:51:27.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.455+0000 7f5527400700 1 -- 172.21.15.89:0/1589954011 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+65 (secure 0 0 0) 0x7f551c058ef0 con 0x7f553014e430 2024-04-02T07:51:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:27.457 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:51:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T07:51:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T07:51:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:27.457 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:51:27.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.458+0000 7f5536200700 1 -- 172.21.15.89:0/1589954011 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f551006adb0 msgr2=0x7f551006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:27.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.458+0000 7f5536200700 1 --2- 172.21.15.89:0/1589954011 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f551006adb0 0x7f551006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5520009ba0 tx=0x7f5520008040 comp rx=0 tx=0).stop 2024-04-02T07:51:27.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.458+0000 7f5536200700 1 -- 172.21.15.89:0/1589954011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f553014e430 msgr2=0x7f5530159d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:27.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.458+0000 7f5536200700 1 --2- 172.21.15.89:0/1589954011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f553014e430 0x7f5530159d00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f551400baf0 tx=0x7f551c001230 comp rx=0 tx=0).stop 2024-04-02T07:51:27.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.458+0000 7f5536200700 1 -- 172.21.15.89:0/1589954011 shutdown_connections 2024-04-02T07:51:27.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.458+0000 7f5536200700 1 --2- 172.21.15.89:0/1589954011 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f551006adb0 0x7f551006d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.458+0000 7f5536200700 1 --2- 172.21.15.89:0/1589954011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f553014e8b0 0x7f553015a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.458+0000 7f5536200700 1 --2- 172.21.15.89:0/1589954011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5514003f70 0x7f5530141b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.458+0000 7f5536200700 1 --2- 172.21.15.89:0/1589954011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f553014e430 0x7f5530159d00 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.458+0000 7f5536200700 1 -- 172.21.15.89:0/1589954011 >> 172.21.15.89:0/1589954011 conn(0x7f55300b5ab0 msgr2=0x7f5530158060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:27.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.458+0000 7f5536200700 1 -- 172.21.15.89:0/1589954011 shutdown_connections 2024-04-02T07:51:27.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.458+0000 7f5536200700 1 -- 172.21.15.89:0/1589954011 wait complete. 2024-04-02T07:51:27.477 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T07:51:27.477 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T07:51:27.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.670+0000 7f6ce3000700 1 Processor -- start 2024-04-02T07:51:27.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.670+0000 7f6ce3000700 1 -- start start 2024-04-02T07:51:27.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.671+0000 7f6ce3000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6cdc15ddf0 0x7f6cdc0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:27.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.671+0000 7f6ce3000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6cdc15e1d0 0x7f6cdc0d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:27.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.671+0000 7f6ce3000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc0d2b40 con 0x7f6cdc0d2710 2024-04-02T07:51:27.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.671+0000 7f6ce3000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cdc15ec60 con 0x7f6cdc15ddf0 2024-04-02T07:51:27.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.671+0000 7f6ce3000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cdc0ce8a0 con 0x7f6cdc15e1d0 2024-04-02T07:51:27.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.671+0000 7f6cdbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6cdc15ddf0 0x7f6cdc0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:27.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.671+0000 7f6ce1000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6cdc0d2710 0x7f6cdc1612f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36536/0 (socket says 172.21.15.89:36536) 2024-04-02T07:51:27.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.671+0000 7f6cdbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6cdc15ddf0 0x7f6cdc0ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54182/0 (socket says 172.21.15.89:54182) 2024-04-02T07:51:27.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.671+0000 7f6ce1000700 1 -- 172.21.15.89:0/1603360972 learned_addr learned my addr 172.21.15.89:0/1603360972 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:27.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.672+0000 7f6cdb400700 1 --2- 172.21.15.89:0/1603360972 >> v2:172.21.15.113:3300/0 conn(0x7f6cdc15e1d0 0x7f6cdc0d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:27.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.672+0000 7f6cdbe00700 1 -- 172.21.15.89:0/1603360972 >> v2:172.21.15.113:3300/0 conn(0x7f6cdc15e1d0 msgr2=0x7f6cdc0d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:27.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.672+0000 7f6cdbe00700 1 --2- 172.21.15.89:0/1603360972 >> v2:172.21.15.113:3300/0 conn(0x7f6cdc15e1d0 0x7f6cdc0d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.672+0000 7f6cdbe00700 1 -- 172.21.15.89:0/1603360972 >> v1:172.21.15.89:6789/0 conn(0x7f6cdc0d2710 legacy=0x7f6cdc1612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:27.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.672+0000 7f6cdbe00700 1 -- 172.21.15.89:0/1603360972 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc0cdc90 con 0x7f6cdc15ddf0 2024-04-02T07:51:27.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.672+0000 7f6cdb400700 1 --2- 172.21.15.89:0/1603360972 >> v2:172.21.15.113:3300/0 conn(0x7f6cdc15e1d0 0x7f6cdc0d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:27.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.672+0000 7f6cdaa00700 1 -- 172.21.15.89:0/1603360972 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705432667 0 0) 0x7f6cdc0d2b40 con 0x7f6cdc0d2710 2024-04-02T07:51:27.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.672+0000 7f6cdbe00700 1 --2- 172.21.15.89:0/1603360972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cdc15ddf0 0x7f6cdc0ced30 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc0d0fe0 tx=0x7f6ccc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f670a75c62388529 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:27.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.672+0000 7f6cdaa00700 1 -- 172.21.15.89:0/1603360972 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc011830 con 0x7f6cdc15ddf0 2024-04-02T07:51:27.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.672+0000 7f6cdaa00700 1 -- 172.21.15.89:0/1603360972 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc011990 con 0x7f6cdc15ddf0 2024-04-02T07:51:27.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.672+0000 7f6cdaa00700 1 -- 172.21.15.89:0/1603360972 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc011d20 con 0x7f6cdc15ddf0 2024-04-02T07:51:27.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.673+0000 7f6ce3000700 1 -- 172.21.15.89:0/1603360972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cdc15ddf0 msgr2=0x7f6cdc0ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:27.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.673+0000 7f6ce3000700 1 --2- 172.21.15.89:0/1603360972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cdc15ddf0 0x7f6cdc0ced30 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc0d0fe0 tx=0x7f6ccc009b20 comp rx=0 tx=0).stop 2024-04-02T07:51:27.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.673+0000 7f6ce3000700 1 -- 172.21.15.89:0/1603360972 shutdown_connections 2024-04-02T07:51:27.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.673+0000 7f6ce3000700 1 --2- 172.21.15.89:0/1603360972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cdc15ddf0 0x7f6cdc0ced30 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.673+0000 7f6ce3000700 1 --2- 172.21.15.89:0/1603360972 >> v2:172.21.15.113:3300/0 conn(0x7f6cdc15e1d0 0x7f6cdc0d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.673+0000 7f6ce3000700 1 -- 172.21.15.89:0/1603360972 >> 172.21.15.89:0/1603360972 conn(0x7f6cdc08aff0 msgr2=0x7f6cdc0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:27.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.673+0000 7f6ce3000700 1 -- 172.21.15.89:0/1603360972 shutdown_connections 2024-04-02T07:51:27.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.673+0000 7f6ce3000700 1 -- 172.21.15.89:0/1603360972 wait complete. 2024-04-02T07:51:27.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.675+0000 7f6ce3000700 1 Processor -- start 2024-04-02T07:51:27.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.675+0000 7f6ce3000700 1 -- start start 2024-04-02T07:51:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6ce3000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cdc0d2710 0x7f6cdc0b27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6ce3000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cdc15ddf0 0x7f6cdc160950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6ce3000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6cdc15e1d0 0x7f6cdc15eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6ce3000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0b3c70 con 0x7f6cdc0d2710 2024-04-02T07:51:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6ce3000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0b2ff0 con 0x7f6cdc15ddf0 2024-04-02T07:51:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6ce3000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0b32d0 con 0x7f6cdc15e1d0 2024-04-02T07:51:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6cdbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cdc0d2710 0x7f6cdc0b27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6cdbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cdc0d2710 0x7f6cdc0b27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54198/0 (socket says 172.21.15.89:54198) 2024-04-02T07:51:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6cdbe00700 1 -- 172.21.15.89:0/1879813122 learned_addr learned my addr 172.21.15.89:0/1879813122 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6ce1000700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6cdc15e1d0 0x7f6cdc15eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6cdbe00700 1 -- 172.21.15.89:0/1879813122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6cdc15e1d0 msgr2=0x7f6cdc15eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6cdb400700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cdc15ddf0 0x7f6cdc160950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6cdbe00700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6cdc15e1d0 0x7f6cdc15eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6cdbe00700 1 -- 172.21.15.89:0/1879813122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cdc15ddf0 msgr2=0x7f6cdc160950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6cdbe00700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cdc15ddf0 0x7f6cdc160950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6cdbe00700 1 -- 172.21.15.89:0/1879813122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ccc0094a0 con 0x7f6cdc0d2710 2024-04-02T07:51:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6ce1000700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6cdc15e1d0 0x7f6cdc15eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:27.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.676+0000 7f6cdb400700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cdc15ddf0 0x7f6cdc160950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.677+0000 7f6cdbe00700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cdc0d2710 0x7f6cdc0b27a0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0090b0 tx=0x7f6ccc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.677+0000 7f6cd3e00700 1 -- 172.21.15.89:0/1879813122 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc0038a0 con 0x7f6cdc0d2710 2024-04-02T07:51:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.677+0000 7f6cd3e00700 1 -- 172.21.15.89:0/1879813122 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc01f070 con 0x7f6cdc0d2710 2024-04-02T07:51:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.677+0000 7f6cd3e00700 1 -- 172.21.15.89:0/1879813122 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc003c20 con 0x7f6cdc0d2710 2024-04-02T07:51:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.677+0000 7f6ce3000700 1 -- 172.21.15.89:0/1879813122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc0b35b0 con 0x7f6cdc0d2710 2024-04-02T07:51:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.677+0000 7f6ce3000700 1 -- 172.21.15.89:0/1879813122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc0b2ce0 con 0x7f6cdc0d2710 2024-04-02T07:51:27.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.678+0000 7f6ce3000700 1 -- 172.21.15.89:0/1879813122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cdc067720 con 0x7f6cdc0d2710 2024-04-02T07:51:27.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.679+0000 7f6cd3e00700 1 -- 172.21.15.89:0/1879813122 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6ccc01a430 con 0x7f6cdc0d2710 2024-04-02T07:51:27.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.680+0000 7f6cd3e00700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6cbc06adb0 0x7f6cbc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:27.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.680+0000 7f6cd3e00700 1 -- 172.21.15.89:0/1879813122 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f6ccc021070 con 0x7f6cdc0d2710 2024-04-02T07:51:27.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.680+0000 7f6cdb400700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6cbc06adb0 0x7f6cbc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:27.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.681+0000 7f6cdb400700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6cbc06adb0 0x7f6cbc06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8009820 tx=0x7f6cc8008040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:27.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.684+0000 7f6cd3e00700 1 -- 172.21.15.89:0/1879813122 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ccc055fa0 con 0x7f6cdc0d2710 2024-04-02T07:51:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:27.885+0000 7f6ce3000700 1 -- 172.21.15.89:0/1879813122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f6cdc0d3050 con 0x7f6cdc0d2710 2024-04-02T07:51:28.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.185+0000 7f6cd3e00700 1 -- 172.21.15.89:0/1879813122 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6ccc02b020 con 0x7f6cdc0d2710 2024-04-02T07:51:28.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.189+0000 7f6ce3000700 1 -- 172.21.15.89:0/1879813122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6cbc06adb0 msgr2=0x7f6cbc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:28.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.189+0000 7f6ce3000700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6cbc06adb0 0x7f6cbc06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8009820 tx=0x7f6cc8008040 comp rx=0 tx=0).stop 2024-04-02T07:51:28.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.190+0000 7f6ce3000700 1 -- 172.21.15.89:0/1879813122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cdc0d2710 msgr2=0x7f6cdc0b27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:28.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.190+0000 7f6ce3000700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cdc0d2710 0x7f6cdc0b27a0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0090b0 tx=0x7f6ccc004750 comp rx=0 tx=0).stop 2024-04-02T07:51:28.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.190+0000 7f6ce3000700 1 -- 172.21.15.89:0/1879813122 shutdown_connections 2024-04-02T07:51:28.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.190+0000 7f6ce3000700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6cbc06adb0 0x7f6cbc06d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:28.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.190+0000 7f6ce3000700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6cdc15e1d0 0x7f6cdc15eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:28.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.190+0000 7f6ce3000700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cdc0d2710 0x7f6cdc0b27a0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:28.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.190+0000 7f6ce3000700 1 --2- 172.21.15.89:0/1879813122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cdc15ddf0 0x7f6cdc160950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:28.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.190+0000 7f6ce3000700 1 -- 172.21.15.89:0/1879813122 >> 172.21.15.89:0/1879813122 conn(0x7f6cdc08aff0 msgr2=0x7f6cdc0d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:28.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.190+0000 7f6ce3000700 1 -- 172.21.15.89:0/1879813122 shutdown_connections 2024-04-02T07:51:28.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.190+0000 7f6ce3000700 1 -- 172.21.15.89:0/1879813122 wait complete. 2024-04-02T07:51:28.194 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T07:51:28.211 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T07:51:28.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.408+0000 7efedc800700 1 Processor -- start 2024-04-02T07:51:28.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.408+0000 7efedc800700 1 -- start start 2024-04-02T07:51:28.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.409+0000 7efedc800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efed40d16a0 0x7efed40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:28.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.409+0000 7efedc800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efed40d2370 0x7efed40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:28.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.409+0000 7efedc800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed4161420 con 0x7efed415d9d0 2024-04-02T07:51:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.409+0000 7efedc800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed40d4340 con 0x7efed40d2370 2024-04-02T07:51:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.409+0000 7efedc800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed415edc0 con 0x7efed40d16a0 2024-04-02T07:51:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.409+0000 7efeda800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efed40d2370 0x7efed40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.409+0000 7efed9e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efed415d9d0 0x7efed40d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36552/0 (socket says 172.21.15.89:36552) 2024-04-02T07:51:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.409+0000 7efed9e00700 1 -- 172.21.15.89:0/1975609419 learned_addr learned my addr 172.21.15.89:0/1975609419 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.410+0000 7efed9400700 1 --2- 172.21.15.89:0/1975609419 >> v2:172.21.15.113:3300/0 conn(0x7efed40d16a0 0x7efed40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.410+0000 7efeda800700 1 -- 172.21.15.89:0/1975609419 >> v2:172.21.15.113:3300/0 conn(0x7efed40d16a0 msgr2=0x7efed40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.410+0000 7efeda800700 1 --2- 172.21.15.89:0/1975609419 >> v2:172.21.15.113:3300/0 conn(0x7efed40d16a0 0x7efed40d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.410+0000 7efeda800700 1 -- 172.21.15.89:0/1975609419 >> v1:172.21.15.89:6789/0 conn(0x7efed415d9d0 legacy=0x7efed40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.410+0000 7efeda800700 1 -- 172.21.15.89:0/1975609419 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed4161700 con 0x7efed40d2370 2024-04-02T07:51:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.410+0000 7efed8a00700 1 -- 172.21.15.89:0/1975609419 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339835670 0 0) 0x7efed4161420 con 0x7efed415d9d0 2024-04-02T07:51:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.410+0000 7efeda800700 1 --2- 172.21.15.89:0/1975609419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed40d2370 0x7efed40ce360 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7efed40d12f0 tx=0x7efed000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2879c11100fe2ebb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:28.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.410+0000 7efed8a00700 1 -- 172.21.15.89:0/1975609419 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed000e070 con 0x7efed40d2370 2024-04-02T07:51:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.410+0000 7efed8a00700 1 -- 172.21.15.89:0/1975609419 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed000bbd0 con 0x7efed40d2370 2024-04-02T07:51:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.410+0000 7efed8a00700 1 -- 172.21.15.89:0/1975609419 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed0012430 con 0x7efed40d2370 2024-04-02T07:51:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.411+0000 7efedc800700 1 -- 172.21.15.89:0/1975609419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed40d2370 msgr2=0x7efed40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.411+0000 7efedc800700 1 --2- 172.21.15.89:0/1975609419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed40d2370 0x7efed40ce360 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7efed40d12f0 tx=0x7efed000b330 comp rx=0 tx=0).stop 2024-04-02T07:51:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.411+0000 7efedc800700 1 -- 172.21.15.89:0/1975609419 shutdown_connections 2024-04-02T07:51:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.411+0000 7efedc800700 1 --2- 172.21.15.89:0/1975609419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed40d2370 0x7efed40ce360 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.411+0000 7efedc800700 1 --2- 172.21.15.89:0/1975609419 >> v2:172.21.15.113:3300/0 conn(0x7efed40d16a0 0x7efed40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.411+0000 7efedc800700 1 -- 172.21.15.89:0/1975609419 >> 172.21.15.89:0/1975609419 conn(0x7efed408aff0 msgr2=0x7efed40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.411+0000 7efedc800700 1 -- 172.21.15.89:0/1975609419 shutdown_connections 2024-04-02T07:51:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.411+0000 7efedc800700 1 -- 172.21.15.89:0/1975609419 wait complete. 2024-04-02T07:51:28.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.413+0000 7efedc800700 1 Processor -- start 2024-04-02T07:51:28.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.413+0000 7efedc800700 1 -- start start 2024-04-02T07:51:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efedc800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efed40d16a0 0x7efed41529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efedc800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed40d2370 0x7efed406b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efedc800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efed415d9d0 0x7efed406be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efedc800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed40ccdf0 con 0x7efed40d2370 2024-04-02T07:51:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efedc800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed40ca5f0 con 0x7efed415d9d0 2024-04-02T07:51:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efedc800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed4152ef0 con 0x7efed40d16a0 2024-04-02T07:51:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efed9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed40d2370 0x7efed406b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efed9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed40d2370 0x7efed406b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54220/0 (socket says 172.21.15.89:54220) 2024-04-02T07:51:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efed9400700 1 -- 172.21.15.89:0/613751136 learned_addr learned my addr 172.21.15.89:0/613751136 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efed9400700 1 -- 172.21.15.89:0/613751136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efed40d16a0 msgr2=0x7efed41529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efed9400700 1 --2- 172.21.15.89:0/613751136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efed40d16a0 0x7efed41529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efeda800700 1 --2- 172.21.15.89:0/613751136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efed415d9d0 0x7efed406be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:28.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efed9400700 1 -- 172.21.15.89:0/613751136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efed415d9d0 msgr2=0x7efed406be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:28.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efed9400700 1 --2- 172.21.15.89:0/613751136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efed415d9d0 0x7efed406be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:28.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.414+0000 7efed9400700 1 -- 172.21.15.89:0/613751136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed0009a70 con 0x7efed40d2370 2024-04-02T07:51:28.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.415+0000 7efeda800700 1 --2- 172.21.15.89:0/613751136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efed415d9d0 0x7efed406be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:28.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.415+0000 7efed9400700 1 --2- 172.21.15.89:0/613751136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed40d2370 0x7efed406b900 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7efec400e840 tx=0x7efec400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:28.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.415+0000 7efece000700 1 -- 172.21.15.89:0/613751136 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec4013610 con 0x7efed40d2370 2024-04-02T07:51:28.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.415+0000 7efece000700 1 -- 172.21.15.89:0/613751136 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec4013770 con 0x7efed40d2370 2024-04-02T07:51:28.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.415+0000 7efedc800700 1 -- 172.21.15.89:0/613751136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed40ca840 con 0x7efed40d2370 2024-04-02T07:51:28.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.415+0000 7efece000700 1 -- 172.21.15.89:0/613751136 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec4013b20 con 0x7efed40d2370 2024-04-02T07:51:28.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.415+0000 7efedc800700 1 -- 172.21.15.89:0/613751136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed40cae00 con 0x7efed40d2370 2024-04-02T07:51:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.416+0000 7efedc800700 1 -- 172.21.15.89:0/613751136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed40cb450 con 0x7efed40d2370 2024-04-02T07:51:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.417+0000 7efece000700 1 -- 172.21.15.89:0/613751136 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7efec4013c80 con 0x7efed40d2370 2024-04-02T07:51:28.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.418+0000 7efece000700 1 --2- 172.21.15.89:0/613751136 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efeb806adb0 0x7efeb806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:28.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.419+0000 7efece000700 1 -- 172.21.15.89:0/613751136 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5575+0+0 (secure 0 0 0) 0x7efec40896c0 con 0x7efed40d2370 2024-04-02T07:51:28.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.419+0000 7efed9e00700 1 --2- 172.21.15.89:0/613751136 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efeb806adb0 0x7efeb806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:28.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.419+0000 7efed9e00700 1 --2- 172.21.15.89:0/613751136 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efeb806adb0 0x7efeb806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efec8001610 tx=0x7efec80079f0 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:28.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.423+0000 7efece000700 1 -- 172.21.15.89:0/613751136 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efec40546e0 con 0x7efed40d2370 2024-04-02T07:51:28.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:28.621+0000 7efedc800700 1 -- 172.21.15.89:0/613751136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7efed40d1ca0 con 0x7efed40d2370 2024-04-02T07:51:29.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.187+0000 7efece000700 1 -- 172.21.15.89:0/613751136 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v81) v1 ==== 170+0+0 (secure 0 0 0) 0x7efec4057d00 con 0x7efed40d2370 2024-04-02T07:51:29.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.193+0000 7efedc800700 1 -- 172.21.15.89:0/613751136 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efeb806adb0 msgr2=0x7efeb806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:29.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.193+0000 7efedc800700 1 --2- 172.21.15.89:0/613751136 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efeb806adb0 0x7efeb806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efec8001610 tx=0x7efec80079f0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.193+0000 7efedc800700 1 -- 172.21.15.89:0/613751136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed40d2370 msgr2=0x7efed406b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:29.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.193+0000 7efedc800700 1 --2- 172.21.15.89:0/613751136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed40d2370 0x7efed406b900 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7efec400e840 tx=0x7efec400ec00 comp rx=0 tx=0).stop 2024-04-02T07:51:29.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.196+0000 7efedc800700 1 -- 172.21.15.89:0/613751136 shutdown_connections 2024-04-02T07:51:29.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.196+0000 7efedc800700 1 --2- 172.21.15.89:0/613751136 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efeb806adb0 0x7efeb806d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.196+0000 7efedc800700 1 --2- 172.21.15.89:0/613751136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efed415d9d0 0x7efed406be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.196+0000 7efedc800700 1 --2- 172.21.15.89:0/613751136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efed40d16a0 0x7efed41529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.196+0000 7efedc800700 1 --2- 172.21.15.89:0/613751136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efed40d2370 0x7efed406b900 secure :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7efec400e840 tx=0x7efec400ec00 comp rx=0 tx=0).stop 2024-04-02T07:51:29.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.196+0000 7efedc800700 1 -- 172.21.15.89:0/613751136 >> 172.21.15.89:0/613751136 conn(0x7efed408aff0 msgr2=0x7efed40b04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:29.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.197+0000 7efedc800700 1 -- 172.21.15.89:0/613751136 shutdown_connections 2024-04-02T07:51:29.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.198+0000 7efedc800700 1 -- 172.21.15.89:0/613751136 wait complete. 2024-04-02T07:51:29.203 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T07:51:29.228 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T07:51:29.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.421+0000 7f04b9a00700 1 Processor -- start 2024-04-02T07:51:29.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.421+0000 7f04b9a00700 1 -- start start 2024-04-02T07:51:29.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.423+0000 7f04b9a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f04b40cbb30 0x7f04b40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:29.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.423+0000 7f04b9a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04b40cd020 0x7f04b40cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:29.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.423+0000 7f04b9a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b40cc450 con 0x7f04b40b2070 2024-04-02T07:51:29.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.423+0000 7f04b9a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b40cdae0 con 0x7f04b40cbb30 2024-04-02T07:51:29.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.423+0000 7f04b9a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b40cddc0 con 0x7f04b40cd020 2024-04-02T07:51:29.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.423+0000 7f04b3400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f04b40b2070 0x7f04b406e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58728/0 (socket says 172.21.15.89:58728) 2024-04-02T07:51:29.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.423+0000 7f04b3400700 1 -- 172.21.15.89:0/512927440 learned_addr learned my addr 172.21.15.89:0/512927440 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:29.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.423+0000 7f04b2a00700 1 --2- 172.21.15.89:0/512927440 >> v2:172.21.15.113:3300/0 conn(0x7f04b40cbb30 0x7f04b40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:29.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.423+0000 7f04b2000700 1 --2- 172.21.15.89:0/512927440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04b40cd020 0x7f04b40cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:29.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.423+0000 7f04b1600700 1 -- 172.21.15.89:0/512927440 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253695754 0 0) 0x7f04b40cc450 con 0x7f04b40b2070 2024-04-02T07:51:29.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.424+0000 7f04b1600700 1 -- 172.21.15.89:0/512927440 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04980035b0 con 0x7f04b40b2070 2024-04-02T07:51:29.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.424+0000 7f04b1600700 1 -- 172.21.15.89:0/512927440 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 775178092 0 0) 0x7f04980035b0 con 0x7f04b40b2070 2024-04-02T07:51:29.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.424+0000 7f04b2000700 1 -- 172.21.15.89:0/512927440 >> v2:172.21.15.113:3300/0 conn(0x7f04b40cbb30 msgr2=0x7f04b40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:29.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.424+0000 7f04b2000700 1 --2- 172.21.15.89:0/512927440 >> v2:172.21.15.113:3300/0 conn(0x7f04b40cbb30 0x7f04b40cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.424+0000 7f04b2000700 1 -- 172.21.15.89:0/512927440 >> v1:172.21.15.89:6789/0 conn(0x7f04b40b2070 legacy=0x7f04b406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:29.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.424+0000 7f04b2000700 1 -- 172.21.15.89:0/512927440 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b40ce0a0 con 0x7f04b40cd020 2024-04-02T07:51:29.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.424+0000 7f04b2a00700 1 --2- 172.21.15.89:0/512927440 >> v2:172.21.15.113:3300/0 conn(0x7f04b40cbb30 0x7f04b40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.424+0000 7f04b2000700 1 --2- 172.21.15.89:0/512927440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04b40cd020 0x7f04b40cd480 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009840 tx=0x7f04a0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e190a10236364a16 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.424+0000 7f04b1600700 1 -- 172.21.15.89:0/512927440 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac0019f0 con 0x7f04b40cd020 2024-04-02T07:51:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.425+0000 7f04b1600700 1 -- 172.21.15.89:0/512927440 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a0004030 con 0x7f04b40cd020 2024-04-02T07:51:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.425+0000 7f04b1600700 1 -- 172.21.15.89:0/512927440 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a00043c0 con 0x7f04b40cd020 2024-04-02T07:51:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.425+0000 7f04b9a00700 1 -- 172.21.15.89:0/512927440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04b40cd020 msgr2=0x7f04b40cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.425+0000 7f04b9a00700 1 --2- 172.21.15.89:0/512927440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04b40cd020 0x7f04b40cd480 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009840 tx=0x7f04a0009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.425+0000 7f04b9a00700 1 -- 172.21.15.89:0/512927440 shutdown_connections 2024-04-02T07:51:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.425+0000 7f04b9a00700 1 --2- 172.21.15.89:0/512927440 >> v2:172.21.15.113:3300/0 conn(0x7f04b40cbb30 0x7f04b40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.425+0000 7f04b9a00700 1 --2- 172.21.15.89:0/512927440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04b40cd020 0x7f04b40cd480 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.425+0000 7f04b9a00700 1 -- 172.21.15.89:0/512927440 >> 172.21.15.89:0/512927440 conn(0x7f04b40b0d70 msgr2=0x7f04b40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.425+0000 7f04b9a00700 1 -- 172.21.15.89:0/512927440 shutdown_connections 2024-04-02T07:51:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.425+0000 7f04b9a00700 1 -- 172.21.15.89:0/512927440 wait complete. 2024-04-02T07:51:29.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.427+0000 7f04b9a00700 1 Processor -- start 2024-04-02T07:51:29.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.428+0000 7f04b9a00700 1 -- start start 2024-04-02T07:51:29.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b9a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04b40b2070 0x7f04b409bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:29.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b9a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04b40cbb30 0x7f04b4157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:29.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b9a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b40cd020 0x7f04b40adfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:29.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b9a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b409b930 con 0x7f04b40cbb30 2024-04-02T07:51:29.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b9a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b409c300 con 0x7f04b40cd020 2024-04-02T07:51:29.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b9a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b409ad20 con 0x7f04b40b2070 2024-04-02T07:51:29.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04b40cbb30 0x7f04b4157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:29.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04b40cbb30 0x7f04b4157c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44534/0 (socket says 172.21.15.89:44534) 2024-04-02T07:51:29.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b2000700 1 -- 172.21.15.89:0/3583997625 learned_addr learned my addr 172.21.15.89:0/3583997625 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:29.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b2a00700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04b40b2070 0x7f04b409bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:29.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b3400700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b40cd020 0x7f04b40adfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:29.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b2000700 1 -- 172.21.15.89:0/3583997625 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04b40b2070 msgr2=0x7f04b409bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:29.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b2000700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04b40b2070 0x7f04b409bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b2000700 1 -- 172.21.15.89:0/3583997625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b40cd020 msgr2=0x7f04b40adfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:29.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b2000700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b40cd020 0x7f04b40adfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.429+0000 7f04b2000700 1 -- 172.21.15.89:0/3583997625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a40094a0 con 0x7f04b40cbb30 2024-04-02T07:51:29.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.430+0000 7f04b3400700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b40cd020 0x7f04b40adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:29.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.430+0000 7f04b2000700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04b40cbb30 0x7f04b4157c70 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f04a000a010 tx=0x7f04a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:29.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.430+0000 7f04aaa00700 1 -- 172.21.15.89:0/3583997625 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a0003e50 con 0x7f04b40cbb30 2024-04-02T07:51:29.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.430+0000 7f04aaa00700 1 -- 172.21.15.89:0/3583997625 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a001f070 con 0x7f04b40cbb30 2024-04-02T07:51:29.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.430+0000 7f04b9a00700 1 -- 172.21.15.89:0/3583997625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a00094a0 con 0x7f04b40cbb30 2024-04-02T07:51:29.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.430+0000 7f04aaa00700 1 -- 172.21.15.89:0/3583997625 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a0011870 con 0x7f04b40cbb30 2024-04-02T07:51:29.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.430+0000 7f04b9a00700 1 -- 172.21.15.89:0/3583997625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b409b2e0 con 0x7f04b40cbb30 2024-04-02T07:51:29.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.431+0000 7f04b9a00700 1 -- 172.21.15.89:0/3583997625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b4067720 con 0x7f04b40cbb30 2024-04-02T07:51:29.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.433+0000 7f04aaa00700 1 -- 172.21.15.89:0/3583997625 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f04a00119d0 con 0x7f04b40cbb30 2024-04-02T07:51:29.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.433+0000 7f04aaa00700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f049406adb0 0x7f049406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:29.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.434+0000 7f04aaa00700 1 -- 172.21.15.89:0/3583997625 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f04a0021070 con 0x7f04b40cbb30 2024-04-02T07:51:29.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.434+0000 7f04b2a00700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f049406adb0 0x7f049406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:29.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.434+0000 7f04b2a00700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f049406adb0 0x7f049406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009c60 tx=0x7f04a4008040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:29.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.437+0000 7f04aaa00700 1 -- 172.21.15.89:0/3583997625 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f04a0056160 con 0x7f04b40cbb30 2024-04-02T07:51:29.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.682+0000 7f04b9a00700 1 -- 172.21.15.89:0/3583997625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f04b415bda0 con 0x7f04b40cbb30 2024-04-02T07:51:29.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.691+0000 7f04aaa00700 1 -- 172.21.15.89:0/3583997625 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v29) v1 ==== 172+0+0 (secure 0 0 0) 0x7f04a0059780 con 0x7f04b40cbb30 2024-04-02T07:51:29.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.697+0000 7f04b9a00700 1 -- 172.21.15.89:0/3583997625 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f049406adb0 msgr2=0x7f049406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:29.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.697+0000 7f04b9a00700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f049406adb0 0x7f049406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009c60 tx=0x7f04a4008040 comp rx=0 tx=0).stop 2024-04-02T07:51:29.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.697+0000 7f04b9a00700 1 -- 172.21.15.89:0/3583997625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04b40cbb30 msgr2=0x7f04b4157c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:29.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.697+0000 7f04b9a00700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04b40cbb30 0x7f04b4157c70 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f04a000a010 tx=0x7f04a0004750 comp rx=0 tx=0).stop 2024-04-02T07:51:29.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.700+0000 7f04b9a00700 1 -- 172.21.15.89:0/3583997625 shutdown_connections 2024-04-02T07:51:29.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.700+0000 7f04b9a00700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f049406adb0 0x7f049406d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.700+0000 7f04b9a00700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b40cd020 0x7f04b40adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.700+0000 7f04b9a00700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04b40b2070 0x7f04b409bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.700+0000 7f04b9a00700 1 --2- 172.21.15.89:0/3583997625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04b40cbb30 0x7f04b4157c70 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.700+0000 7f04b9a00700 1 -- 172.21.15.89:0/3583997625 >> 172.21.15.89:0/3583997625 conn(0x7f04b40b0d70 msgr2=0x7f04b40afc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:29.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.701+0000 7f04b9a00700 1 -- 172.21.15.89:0/3583997625 shutdown_connections 2024-04-02T07:51:29.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.701+0000 7f04b9a00700 1 -- 172.21.15.89:0/3583997625 wait complete. 2024-04-02T07:51:29.713 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 8 and data pool 9 2024-04-02T07:51:29.733 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:51:29.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.921+0000 7fb7cd600700 1 Processor -- start 2024-04-02T07:51:29.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.921+0000 7fb7cd600700 1 -- start start 2024-04-02T07:51:29.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.922+0000 7fb7cd600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8160f10 0x7fb7c80d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:29.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.922+0000 7fb7cd600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7c80cf230 0x7fb7c80d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:29.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.923+0000 7fb7cd600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c80ceda0 con 0x7fb7c8161590 2024-04-02T07:51:29.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.923+0000 7fb7cd600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c80d23d0 con 0x7fb7c80cf230 2024-04-02T07:51:29.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.923+0000 7fb7cd600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c81612f0 con 0x7fb7c8160f10 2024-04-02T07:51:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.923+0000 7fb7c6000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7c80cf230 0x7fb7c80d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.923+0000 7fb7c6000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7c80cf230 0x7fb7c80d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44540/0 (socket says 172.21.15.89:44540) 2024-04-02T07:51:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.923+0000 7fb7c6000700 1 -- 172.21.15.89:0/2095892353 learned_addr learned my addr 172.21.15.89:0/2095892353 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.923+0000 7fb7c6a00700 1 --2- 172.21.15.89:0/2095892353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8160f10 0x7fb7c80d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.923+0000 7fb7c6000700 1 -- 172.21.15.89:0/2095892353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8160f10 msgr2=0x7fb7c80d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.923+0000 7fb7c6000700 1 --2- 172.21.15.89:0/2095892353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8160f10 0x7fb7c80d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.923+0000 7fb7c6000700 1 -- 172.21.15.89:0/2095892353 >> v1:172.21.15.89:6789/0 conn(0x7fb7c8161590 legacy=0x7fb7c80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.923+0000 7fb7c6000700 1 -- 172.21.15.89:0/2095892353 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c80ce470 con 0x7fb7c80cf230 2024-04-02T07:51:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.923+0000 7fb7c6a00700 1 --2- 172.21.15.89:0/2095892353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8160f10 0x7fb7c80d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.923+0000 7fb7c5600700 1 -- 172.21.15.89:0/2095892353 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286333170 0 0) 0x7fb7c80ceda0 con 0x7fb7c8161590 2024-04-02T07:51:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.924+0000 7fb7c6000700 1 --2- 172.21.15.89:0/2095892353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c80cf230 0x7fb7c80d1e90 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb7b40098f0 tx=0x7fb7b4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa42aed856c1ce45 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.924+0000 7fb7c5600700 1 -- 172.21.15.89:0/2095892353 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b400b8a0 con 0x7fb7c80cf230 2024-04-02T07:51:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.924+0000 7fb7c5600700 1 -- 172.21.15.89:0/2095892353 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b400ba00 con 0x7fb7c80cf230 2024-04-02T07:51:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.924+0000 7fb7c5600700 1 -- 172.21.15.89:0/2095892353 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b400bd90 con 0x7fb7c80cf230 2024-04-02T07:51:29.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.924+0000 7fb7cd600700 1 -- 172.21.15.89:0/2095892353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c80cf230 msgr2=0x7fb7c80d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:29.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.924+0000 7fb7cd600700 1 --2- 172.21.15.89:0/2095892353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c80cf230 0x7fb7c80d1e90 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb7b40098f0 tx=0x7fb7b4009c00 comp rx=0 tx=0).stop 2024-04-02T07:51:29.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.924+0000 7fb7cd600700 1 -- 172.21.15.89:0/2095892353 shutdown_connections 2024-04-02T07:51:29.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.924+0000 7fb7cd600700 1 --2- 172.21.15.89:0/2095892353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8160f10 0x7fb7c80d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.924+0000 7fb7cd600700 1 --2- 172.21.15.89:0/2095892353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c80cf230 0x7fb7c80d1e90 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.924+0000 7fb7cd600700 1 -- 172.21.15.89:0/2095892353 >> 172.21.15.89:0/2095892353 conn(0x7fb7c808aff0 msgr2=0x7fb7c80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:29.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.924+0000 7fb7cd600700 1 -- 172.21.15.89:0/2095892353 shutdown_connections 2024-04-02T07:51:29.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.924+0000 7fb7cd600700 1 -- 172.21.15.89:0/2095892353 wait complete. 2024-04-02T07:51:29.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.926+0000 7fb7cd600700 1 Processor -- start 2024-04-02T07:51:29.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.926+0000 7fb7cd600700 1 -- start start 2024-04-02T07:51:29.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.927+0000 7fb7cd600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80cf230 0x7fb7c80c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:29.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.927+0000 7fb7cd600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8160f10 0x7fb7c80b7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:29.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.927+0000 7fb7cd600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8161590 0x7fb7c80b7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:29.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.927+0000 7fb7cd600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80b8dc0 con 0x7fb7c8160f10 2024-04-02T07:51:29.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.927+0000 7fb7cd600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80b7e90 con 0x7fb7c80cf230 2024-04-02T07:51:29.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.927+0000 7fb7cd600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80b8140 con 0x7fb7c8161590 2024-04-02T07:51:29.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.927+0000 7fb7c6000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8160f10 0x7fb7c80b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:29.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.927+0000 7fb7c6000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8160f10 0x7fb7c80b7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44556/0 (socket says 172.21.15.89:44556) 2024-04-02T07:51:29.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.927+0000 7fb7c6000700 1 -- 172.21.15.89:0/302879772 learned_addr learned my addr 172.21.15.89:0/302879772 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:29.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.927+0000 7fb7c7400700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8161590 0x7fb7c80b7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:29.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.928+0000 7fb7c6a00700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80cf230 0x7fb7c80c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:29.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.928+0000 7fb7c6000700 1 -- 172.21.15.89:0/302879772 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8161590 msgr2=0x7fb7c80b7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.928+0000 7fb7c6000700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8161590 0x7fb7c80b7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.928+0000 7fb7c6000700 1 -- 172.21.15.89:0/302879772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80cf230 msgr2=0x7fb7c80c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.928+0000 7fb7c6000700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80cf230 0x7fb7c80c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.928+0000 7fb7c6000700 1 -- 172.21.15.89:0/302879772 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b4009550 con 0x7fb7c8160f10 2024-04-02T07:51:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.928+0000 7fb7c7400700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8161590 0x7fb7c80b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.928+0000 7fb7c6a00700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80cf230 0x7fb7c80c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.928+0000 7fb7c6000700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8160f10 0x7fb7c80b7410 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4014040 tx=0x7fb7b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.928+0000 7fb7bea00700 1 -- 172.21.15.89:0/302879772 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40036a0 con 0x7fb7c8160f10 2024-04-02T07:51:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.928+0000 7fb7bea00700 1 -- 172.21.15.89:0/302879772 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b401e070 con 0x7fb7c8160f10 2024-04-02T07:51:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.928+0000 7fb7cd600700 1 -- 172.21.15.89:0/302879772 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c80b8420 con 0x7fb7c8160f10 2024-04-02T07:51:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.928+0000 7fb7bea00700 1 -- 172.21.15.89:0/302879772 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4003a20 con 0x7fb7c8160f10 2024-04-02T07:51:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.928+0000 7fb7cd600700 1 -- 172.21.15.89:0/302879772 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c80b89e0 con 0x7fb7c8160f10 2024-04-02T07:51:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.929+0000 7fb7cd600700 1 -- 172.21.15.89:0/302879772 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c8156740 con 0x7fb7c8160f10 2024-04-02T07:51:29.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.930+0000 7fb7bea00700 1 -- 172.21.15.89:0/302879772 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb7b4003b80 con 0x7fb7c8160f10 2024-04-02T07:51:29.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.931+0000 7fb7bea00700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb7a806adb0 0x7fb7a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:29.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.931+0000 7fb7bea00700 1 -- 172.21.15.89:0/302879772 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb7b4020070 con 0x7fb7c8160f10 2024-04-02T07:51:29.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.931+0000 7fb7c6a00700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb7a806adb0 0x7fb7a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:29.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.932+0000 7fb7c6a00700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb7a806adb0 0x7fb7a806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80099b0 tx=0x7fb7b8008040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:29.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:29.936+0000 7fb7bea00700 1 -- 172.21.15.89:0/302879772 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb7b4055ac0 con 0x7fb7c8160f10 2024-04-02T07:51:30.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.139+0000 7fb7cd600700 1 -- 172.21.15.89:0/302879772 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7c8067720 con 0x7fb7c8160f10 2024-04-02T07:51:30.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.141+0000 7fb7bea00700 1 -- 172.21.15.89:0/302879772 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16637 (secure 0 0 0) 0x7fb7b402a090 con 0x7fb7c8160f10 2024-04-02T07:51:30.143 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:30.143 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":82,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:51:29.683771+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:51:27.926367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-02T07:51:28.662600+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:51:30.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.143+0000 7fb7cd600700 1 -- 172.21.15.89:0/302879772 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb7a806adb0 msgr2=0x7fb7a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:30.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.143+0000 7fb7cd600700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb7a806adb0 0x7fb7a806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80099b0 tx=0x7fb7b8008040 comp rx=0 tx=0).stop 2024-04-02T07:51:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.143+0000 7fb7cd600700 1 -- 172.21.15.89:0/302879772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8160f10 msgr2=0x7fb7c80b7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.143+0000 7fb7cd600700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8160f10 0x7fb7c80b7410 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4014040 tx=0x7fb7b4004750 comp rx=0 tx=0).stop 2024-04-02T07:51:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.144+0000 7fb7cd600700 1 -- 172.21.15.89:0/302879772 shutdown_connections 2024-04-02T07:51:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.144+0000 7fb7cd600700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb7a806adb0 0x7fb7a806d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.144+0000 7fb7cd600700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80cf230 0x7fb7c80c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.144+0000 7fb7cd600700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8161590 0x7fb7c80b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.144+0000 7fb7cd600700 1 --2- 172.21.15.89:0/302879772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8160f10 0x7fb7c80b7410 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.144+0000 7fb7cd600700 1 -- 172.21.15.89:0/302879772 >> 172.21.15.89:0/302879772 conn(0x7fb7c808aff0 msgr2=0x7fb7c806c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.144+0000 7fb7cd600700 1 -- 172.21.15.89:0/302879772 shutdown_connections 2024-04-02T07:51:30.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.144+0000 7fb7cd600700 1 -- 172.21.15.89:0/302879772 wait complete. 2024-04-02T07:51:30.163 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:51:30.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.353+0000 7fc556600700 1 Processor -- start 2024-04-02T07:51:30.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.354+0000 7fc556600700 1 -- start start 2024-04-02T07:51:30.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc556600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc55015d640 0x7fc5500d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:30.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc556600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc5500ceec0 0x7fc5501612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc556600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5500d30d0 con 0x7fc550161800 2024-04-02T07:51:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc556600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5500b6840 con 0x7fc5500ceec0 2024-04-02T07:51:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc556600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc550161c20 con 0x7fc55015d640 2024-04-02T07:51:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc54ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc5500ceec0 0x7fc5501612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc54fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc550161800 0x7fc5500cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58736/0 (socket says 172.21.15.89:58736) 2024-04-02T07:51:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc54fe00700 1 -- 172.21.15.89:0/1034571454 learned_addr learned my addr 172.21.15.89:0/1034571454 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc54f400700 1 --2- 172.21.15.89:0/1034571454 >> v2:172.21.15.113:3300/0 conn(0x7fc55015d640 0x7fc5500d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc54ea00700 1 -- 172.21.15.89:0/1034571454 >> v2:172.21.15.113:3300/0 conn(0x7fc55015d640 msgr2=0x7fc5500d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc54ea00700 1 --2- 172.21.15.89:0/1034571454 >> v2:172.21.15.113:3300/0 conn(0x7fc55015d640 0x7fc5500d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc54ea00700 1 -- 172.21.15.89:0/1034571454 >> v1:172.21.15.89:6789/0 conn(0x7fc550161800 legacy=0x7fc5500cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc54ea00700 1 -- 172.21.15.89:0/1034571454 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5500d2880 con 0x7fc5500ceec0 2024-04-02T07:51:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc54e000700 1 -- 172.21.15.89:0/1034571454 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594166481 0 0) 0x7fc5500d30d0 con 0x7fc550161800 2024-04-02T07:51:30.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.355+0000 7fc54f400700 1 --2- 172.21.15.89:0/1034571454 >> v2:172.21.15.113:3300/0 conn(0x7fc55015d640 0x7fc5500d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:30.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.356+0000 7fc54ea00700 1 --2- 172.21.15.89:0/1034571454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5500ceec0 0x7fc5501612c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fc5500b5c20 tx=0x7fc53c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7258600e46eac77 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:30.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.356+0000 7fc54e000700 1 -- 172.21.15.89:0/1034571454 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c011830 con 0x7fc5500ceec0 2024-04-02T07:51:30.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.356+0000 7fc54e000700 1 -- 172.21.15.89:0/1034571454 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc53c011990 con 0x7fc5500ceec0 2024-04-02T07:51:30.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.356+0000 7fc54e000700 1 -- 172.21.15.89:0/1034571454 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c011d20 con 0x7fc5500ceec0 2024-04-02T07:51:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.356+0000 7fc556600700 1 -- 172.21.15.89:0/1034571454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5500ceec0 msgr2=0x7fc5501612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.356+0000 7fc556600700 1 --2- 172.21.15.89:0/1034571454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5500ceec0 0x7fc5501612c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fc5500b5c20 tx=0x7fc53c009b20 comp rx=0 tx=0).stop 2024-04-02T07:51:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.356+0000 7fc556600700 1 -- 172.21.15.89:0/1034571454 shutdown_connections 2024-04-02T07:51:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.357+0000 7fc556600700 1 --2- 172.21.15.89:0/1034571454 >> v2:172.21.15.113:3300/0 conn(0x7fc55015d640 0x7fc5500d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.357+0000 7fc556600700 1 --2- 172.21.15.89:0/1034571454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5500ceec0 0x7fc5501612c0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.357+0000 7fc556600700 1 -- 172.21.15.89:0/1034571454 >> 172.21.15.89:0/1034571454 conn(0x7fc55008aff0 msgr2=0x7fc5500d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.357+0000 7fc556600700 1 -- 172.21.15.89:0/1034571454 shutdown_connections 2024-04-02T07:51:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.357+0000 7fc556600700 1 -- 172.21.15.89:0/1034571454 wait complete. 2024-04-02T07:51:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.358+0000 7fc556600700 1 Processor -- start 2024-04-02T07:51:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.358+0000 7fc556600700 1 -- start start 2024-04-02T07:51:30.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.359+0000 7fc556600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5500ceec0 0x7fc55014eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:30.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.359+0000 7fc556600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc55015d640 0x7fc55015f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:30.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.359+0000 7fc556600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc550161800 0x7fc55015f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:30.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.359+0000 7fc556600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5501f3190 con 0x7fc5500ceec0 2024-04-02T07:51:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.359+0000 7fc556600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5500a1290 con 0x7fc55015d640 2024-04-02T07:51:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.359+0000 7fc556600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55014f070 con 0x7fc550161800 2024-04-02T07:51:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.360+0000 7fc54f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5500ceec0 0x7fc55014eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.360+0000 7fc54f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5500ceec0 0x7fc55014eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44584/0 (socket says 172.21.15.89:44584) 2024-04-02T07:51:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.360+0000 7fc54f400700 1 -- 172.21.15.89:0/643862983 learned_addr learned my addr 172.21.15.89:0/643862983 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.360+0000 7fc54f400700 1 -- 172.21.15.89:0/643862983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc550161800 msgr2=0x7fc55015f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.360+0000 7fc54f400700 1 --2- 172.21.15.89:0/643862983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc550161800 0x7fc55015f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.360+0000 7fc54f400700 1 -- 172.21.15.89:0/643862983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc55015d640 msgr2=0x7fc55015f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.360+0000 7fc54f400700 1 --2- 172.21.15.89:0/643862983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc55015d640 0x7fc55015f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.360+0000 7fc54f400700 1 -- 172.21.15.89:0/643862983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c0094a0 con 0x7fc5500ceec0 2024-04-02T07:51:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.360+0000 7fc54f400700 1 --2- 172.21.15.89:0/643862983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5500ceec0 0x7fc55014eb30 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fc54000c750 tx=0x7fc54000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.360+0000 7fc547400700 1 -- 172.21.15.89:0/643862983 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc540011800 con 0x7fc5500ceec0 2024-04-02T07:51:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.360+0000 7fc547400700 1 -- 172.21.15.89:0/643862983 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc540011960 con 0x7fc5500ceec0 2024-04-02T07:51:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.361+0000 7fc547400700 1 -- 172.21.15.89:0/643862983 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc540011cd0 con 0x7fc5500ceec0 2024-04-02T07:51:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.361+0000 7fc556600700 1 -- 172.21.15.89:0/643862983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5500a0cd0 con 0x7fc5500ceec0 2024-04-02T07:51:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.361+0000 7fc556600700 1 -- 172.21.15.89:0/643862983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5500c86d0 con 0x7fc5500ceec0 2024-04-02T07:51:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.362+0000 7fc556600700 1 -- 172.21.15.89:0/643862983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc550067f90 con 0x7fc5500ceec0 2024-04-02T07:51:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.362+0000 7fc547400700 1 -- 172.21.15.89:0/643862983 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc540021350 con 0x7fc5500ceec0 2024-04-02T07:51:30.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.363+0000 7fc547400700 1 --2- 172.21.15.89:0/643862983 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc53006adb0 0x7fc53006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:30.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.363+0000 7fc547400700 1 -- 172.21.15.89:0/643862983 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc540089720 con 0x7fc5500ceec0 2024-04-02T07:51:30.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.364+0000 7fc54ea00700 1 --2- 172.21.15.89:0/643862983 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc53006adb0 0x7fc53006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:30.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.364+0000 7fc54ea00700 1 --2- 172.21.15.89:0/643862983 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc53006adb0 0x7fc53006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5500cea00 tx=0x7fc53c010000 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:30.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.368+0000 7fc547400700 1 -- 172.21.15.89:0/643862983 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc540054570 con 0x7fc5500ceec0 2024-04-02T07:51:30.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.567+0000 7fc556600700 1 -- 172.21.15.89:0/643862983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5500c6e40 con 0x7fc5500ceec0 2024-04-02T07:51:30.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.569+0000 7fc547400700 1 -- 172.21.15.89:0/643862983 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16637 (secure 0 0 0) 0x7fc540057b90 con 0x7fc5500ceec0 2024-04-02T07:51:30.570 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:30.570 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":82,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:51:29.683771+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:51:27.926367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-02T07:51:28.662600+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:51:30.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.571+0000 7fc556600700 1 -- 172.21.15.89:0/643862983 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc53006adb0 msgr2=0x7fc53006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:30.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.571+0000 7fc556600700 1 --2- 172.21.15.89:0/643862983 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc53006adb0 0x7fc53006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5500cea00 tx=0x7fc53c010000 comp rx=0 tx=0).stop 2024-04-02T07:51:30.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.571+0000 7fc556600700 1 -- 172.21.15.89:0/643862983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5500ceec0 msgr2=0x7fc55014eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:30.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.571+0000 7fc556600700 1 --2- 172.21.15.89:0/643862983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5500ceec0 0x7fc55014eb30 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fc54000c750 tx=0x7fc54000cb10 comp rx=0 tx=0).stop 2024-04-02T07:51:30.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.571+0000 7fc556600700 1 -- 172.21.15.89:0/643862983 shutdown_connections 2024-04-02T07:51:30.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.571+0000 7fc556600700 1 --2- 172.21.15.89:0/643862983 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc53006adb0 0x7fc53006d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.571+0000 7fc556600700 1 --2- 172.21.15.89:0/643862983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc550161800 0x7fc55015f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.571+0000 7fc556600700 1 --2- 172.21.15.89:0/643862983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5500ceec0 0x7fc55014eb30 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.571+0000 7fc556600700 1 --2- 172.21.15.89:0/643862983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc55015d640 0x7fc55015f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.571+0000 7fc556600700 1 -- 172.21.15.89:0/643862983 >> 172.21.15.89:0/643862983 conn(0x7fc55008aff0 msgr2=0x7fc55006b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:30.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.571+0000 7fc556600700 1 -- 172.21.15.89:0/643862983 shutdown_connections 2024-04-02T07:51:30.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.571+0000 7fc556600700 1 -- 172.21.15.89:0/643862983 wait complete. 2024-04-02T07:51:30.591 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T07:51:30.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.787+0000 7f5967200700 1 Processor -- start 2024-04-02T07:51:30.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.787+0000 7f5967200700 1 -- start start 2024-04-02T07:51:30.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.788+0000 7f5967200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f596015d9d0 0x7f59600d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:30.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.788+0000 7f5967200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f59600d2540 0x7f59600ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:30.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.788+0000 7f5967200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5960161420 con 0x7f59600d16a0 2024-04-02T07:51:30.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.788+0000 7f5967200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596015edc0 con 0x7f596015d9d0 2024-04-02T07:51:30.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.788+0000 7f5967200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5960161700 con 0x7f59600d2540 2024-04-02T07:51:30.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.788+0000 7f595fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f596015d9d0 0x7f59600d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:30.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.788+0000 7f595fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f596015d9d0 0x7f59600d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44592/0 (socket says 172.21.15.89:44592) 2024-04-02T07:51:30.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.788+0000 7f595f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f59600d16a0 0x7f59600d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58750/0 (socket says 172.21.15.89:58750) 2024-04-02T07:51:30.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.788+0000 7f595fe00700 1 -- 172.21.15.89:0/661995239 learned_addr learned my addr 172.21.15.89:0/661995239 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:30.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.789+0000 7f595fe00700 1 -- 172.21.15.89:0/661995239 >> v2:172.21.15.113:3300/0 conn(0x7f59600d2540 msgr2=0x7f59600ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:30.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.789+0000 7f595fe00700 1 --2- 172.21.15.89:0/661995239 >> v2:172.21.15.113:3300/0 conn(0x7f59600d2540 0x7f59600ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.789+0000 7f595fe00700 1 -- 172.21.15.89:0/661995239 >> v1:172.21.15.89:6789/0 conn(0x7f59600d16a0 legacy=0x7f59600d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:30.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.789+0000 7f595fe00700 1 -- 172.21.15.89:0/661995239 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59601619e0 con 0x7f596015d9d0 2024-04-02T07:51:30.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.789+0000 7f595fe00700 1 --2- 172.21.15.89:0/661995239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f596015d9d0 0x7f59600d3c30 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f596008ca10 tx=0x7f5950009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609659585b2f86c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:30.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.789+0000 7f595ea00700 1 -- 172.21.15.89:0/661995239 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595000b870 con 0x7f596015d9d0 2024-04-02T07:51:30.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.789+0000 7f595ea00700 1 -- 172.21.15.89:0/661995239 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595000b9d0 con 0x7f596015d9d0 2024-04-02T07:51:30.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.789+0000 7f595ea00700 1 -- 172.21.15.89:0/661995239 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595000bd60 con 0x7f596015d9d0 2024-04-02T07:51:30.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.790+0000 7f5967200700 1 -- 172.21.15.89:0/661995239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f596015d9d0 msgr2=0x7f59600d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:30.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.790+0000 7f5967200700 1 --2- 172.21.15.89:0/661995239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f596015d9d0 0x7f59600d3c30 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f596008ca10 tx=0x7f5950009bd0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.790+0000 7f5967200700 1 -- 172.21.15.89:0/661995239 shutdown_connections 2024-04-02T07:51:30.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.790+0000 7f5967200700 1 --2- 172.21.15.89:0/661995239 >> v2:172.21.15.113:3300/0 conn(0x7f59600d2540 0x7f59600ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.790+0000 7f5967200700 1 --2- 172.21.15.89:0/661995239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f596015d9d0 0x7f59600d3c30 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.790+0000 7f5967200700 1 -- 172.21.15.89:0/661995239 >> 172.21.15.89:0/661995239 conn(0x7f596008aff0 msgr2=0x7f59600d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:30.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.790+0000 7f5967200700 1 -- 172.21.15.89:0/661995239 shutdown_connections 2024-04-02T07:51:30.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.790+0000 7f5967200700 1 -- 172.21.15.89:0/661995239 wait complete. 2024-04-02T07:51:30.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.791+0000 7f5967200700 1 Processor -- start 2024-04-02T07:51:30.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.792+0000 7f5967200700 1 -- start start 2024-04-02T07:51:30.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.792+0000 7f5967200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59600d16a0 0x7f59600b8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:30.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.792+0000 7f5967200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59600d2540 0x7f59600b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:30.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.792+0000 7f5967200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f596015d9d0 0x7f59600cd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:30.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.792+0000 7f5967200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59600cb350 con 0x7f596015d9d0 2024-04-02T07:51:30.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.792+0000 7f5967200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59600b6560 con 0x7f59600d2540 2024-04-02T07:51:30.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.792+0000 7f5967200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59600cda60 con 0x7f59600d16a0 2024-04-02T07:51:30.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.793+0000 7f5965200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f596015d9d0 0x7f59600cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:30.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.793+0000 7f5965200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f596015d9d0 0x7f59600cd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44606/0 (socket says 172.21.15.89:44606) 2024-04-02T07:51:30.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.793+0000 7f5965200700 1 -- 172.21.15.89:0/2219003099 learned_addr learned my addr 172.21.15.89:0/2219003099 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:30.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.793+0000 7f595f400700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59600d2540 0x7f59600b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:30.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.793+0000 7f595fe00700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59600d16a0 0x7f59600b8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:30.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.793+0000 7f5965200700 1 -- 172.21.15.89:0/2219003099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59600d16a0 msgr2=0x7f59600b8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:30.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.793+0000 7f5965200700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59600d16a0 0x7f59600b8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.793+0000 7f5965200700 1 -- 172.21.15.89:0/2219003099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59600d2540 msgr2=0x7f59600b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:30.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.793+0000 7f5965200700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59600d2540 0x7f59600b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:30.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.793+0000 7f5965200700 1 -- 172.21.15.89:0/2219003099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5950009550 con 0x7f596015d9d0 2024-04-02T07:51:30.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.793+0000 7f595f400700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59600d2540 0x7f59600b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:30.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.793+0000 7f5965200700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f596015d9d0 0x7f59600cd520 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f595800b9e0 tx=0x7f595800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:30.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.794+0000 7f5957e00700 1 -- 172.21.15.89:0/2219003099 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595800c9b0 con 0x7f596015d9d0 2024-04-02T07:51:30.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.794+0000 7f5957e00700 1 -- 172.21.15.89:0/2219003099 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595800cb30 con 0x7f596015d9d0 2024-04-02T07:51:30.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.794+0000 7f5967200700 1 -- 172.21.15.89:0/2219003099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59600b7340 con 0x7f596015d9d0 2024-04-02T07:51:30.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.794+0000 7f5957e00700 1 -- 172.21.15.89:0/2219003099 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958014440 con 0x7f596015d9d0 2024-04-02T07:51:30.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.794+0000 7f5967200700 1 -- 172.21.15.89:0/2219003099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59600b6aa0 con 0x7f596015d9d0 2024-04-02T07:51:30.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.795+0000 7f5967200700 1 -- 172.21.15.89:0/2219003099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59600d1ca0 con 0x7f596015d9d0 2024-04-02T07:51:30.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.796+0000 7f5957e00700 1 -- 172.21.15.89:0/2219003099 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f595801d3b0 con 0x7f596015d9d0 2024-04-02T07:51:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.796+0000 7f5957e00700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f594006adb0 0x7f594006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.796+0000 7f5957e00700 1 -- 172.21.15.89:0/2219003099 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5958006e40 con 0x7f596015d9d0 2024-04-02T07:51:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.797+0000 7f595fe00700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f594006adb0 0x7f594006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.797+0000 7f595fe00700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f594006adb0 0x7f594006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5950014040 tx=0x7f5950008030 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:30.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:30.801+0000 7f5957e00700 1 -- 172.21.15.89:0/2219003099 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5958054260 con 0x7f596015d9d0 2024-04-02T07:51:31.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.047+0000 7f5967200700 1 -- 172.21.15.89:0/2219003099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5960067720 con 0x7f596015d9d0 2024-04-02T07:51:31.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.707+0000 7f5957e00700 1 -- 172.21.15.89:0/2219003099 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5958057880 con 0x7f596015d9d0 2024-04-02T07:51:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.710+0000 7f5967200700 1 -- 172.21.15.89:0/2219003099 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f594006adb0 msgr2=0x7f594006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.710+0000 7f5967200700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f594006adb0 0x7f594006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5950014040 tx=0x7f5950008030 comp rx=0 tx=0).stop 2024-04-02T07:51:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.710+0000 7f5967200700 1 -- 172.21.15.89:0/2219003099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f596015d9d0 msgr2=0x7f59600cd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.710+0000 7f5967200700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f596015d9d0 0x7f59600cd520 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f595800b9e0 tx=0x7f595800bda0 comp rx=0 tx=0).stop 2024-04-02T07:51:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.710+0000 7f5967200700 1 -- 172.21.15.89:0/2219003099 shutdown_connections 2024-04-02T07:51:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.710+0000 7f5967200700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f594006adb0 0x7f594006d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.710+0000 7f5967200700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f596015d9d0 0x7f59600cd520 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.710+0000 7f5967200700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59600d16a0 0x7f59600b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.710+0000 7f5967200700 1 --2- 172.21.15.89:0/2219003099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59600d2540 0x7f59600b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.710+0000 7f5967200700 1 -- 172.21.15.89:0/2219003099 >> 172.21.15.89:0/2219003099 conn(0x7f596008aff0 msgr2=0x7f59600d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:31.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.710+0000 7f5967200700 1 -- 172.21.15.89:0/2219003099 shutdown_connections 2024-04-02T07:51:31.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.710+0000 7f5967200700 1 -- 172.21.15.89:0/2219003099 wait complete. 2024-04-02T07:51:31.731 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T07:51:31.731 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T07:51:31.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.923+0000 7fa78ca00700 1 Processor -- start 2024-04-02T07:51:32.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.924+0000 7fa78ca00700 1 -- start start 2024-04-02T07:51:32.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.925+0000 7fa78ca00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7880d21f0 0x7fa7880ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:32.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.925+0000 7fa78ca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7880cec40 0x7fa788161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:32.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.925+0000 7fa78ca00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7880ce270 con 0x7fa7881611b0 2024-04-02T07:51:32.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.925+0000 7fa78ca00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa78815b600 con 0x7fa7880d21f0 2024-04-02T07:51:32.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.925+0000 7fa78ca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7880d2760 con 0x7fa7880cec40 2024-04-02T07:51:32.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.925+0000 7fa785400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7880d21f0 0x7fa7880ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:32.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.925+0000 7fa785400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7880d21f0 0x7fa7880ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44614/0 (socket says 172.21.15.89:44614) 2024-04-02T07:51:32.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.925+0000 7fa785400700 1 -- 172.21.15.89:0/2617213570 learned_addr learned my addr 172.21.15.89:0/2617213570 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:32.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.925+0000 7fa786800700 1 --2- 172.21.15.89:0/2617213570 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7880cec40 0x7fa788161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:32.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.925+0000 7fa785400700 1 -- 172.21.15.89:0/2617213570 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7880cec40 msgr2=0x7fa788161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:32.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.925+0000 7fa785400700 1 --2- 172.21.15.89:0/2617213570 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7880cec40 0x7fa788161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.925+0000 7fa785400700 1 -- 172.21.15.89:0/2617213570 >> v1:172.21.15.113:6789/0 conn(0x7fa7881611b0 legacy=0x7fa78815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:32.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.925+0000 7fa785400700 1 -- 172.21.15.89:0/2617213570 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7880d2a40 con 0x7fa7880d21f0 2024-04-02T07:51:32.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.926+0000 7fa785400700 1 --2- 172.21.15.89:0/2617213570 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7880d21f0 0x7fa7880ce700 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa7740098f0 tx=0x7fa774009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0441627cb1b1ee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:32.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.926+0000 7fa784a00700 1 -- 172.21.15.89:0/2617213570 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77400b8a0 con 0x7fa7880d21f0 2024-04-02T07:51:32.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.926+0000 7fa784a00700 1 -- 172.21.15.89:0/2617213570 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77400ba00 con 0x7fa7880d21f0 2024-04-02T07:51:32.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.926+0000 7fa784a00700 1 -- 172.21.15.89:0/2617213570 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77400bd90 con 0x7fa7880d21f0 2024-04-02T07:51:32.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.926+0000 7fa78ca00700 1 -- 172.21.15.89:0/2617213570 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7880d21f0 msgr2=0x7fa7880ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:32.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.926+0000 7fa78ca00700 1 --2- 172.21.15.89:0/2617213570 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7880d21f0 0x7fa7880ce700 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa7740098f0 tx=0x7fa774009c00 comp rx=0 tx=0).stop 2024-04-02T07:51:32.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.927+0000 7fa78ca00700 1 -- 172.21.15.89:0/2617213570 shutdown_connections 2024-04-02T07:51:32.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.927+0000 7fa78ca00700 1 --2- 172.21.15.89:0/2617213570 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7880cec40 0x7fa788161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.927+0000 7fa78ca00700 1 --2- 172.21.15.89:0/2617213570 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7880d21f0 0x7fa7880ce700 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.927+0000 7fa78ca00700 1 -- 172.21.15.89:0/2617213570 >> 172.21.15.89:0/2617213570 conn(0x7fa78808aff0 msgr2=0x7fa7880d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:32.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.927+0000 7fa78ca00700 1 -- 172.21.15.89:0/2617213570 shutdown_connections 2024-04-02T07:51:32.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.927+0000 7fa78ca00700 1 -- 172.21.15.89:0/2617213570 wait complete. 2024-04-02T07:51:32.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.928+0000 7fa78ca00700 1 Processor -- start 2024-04-02T07:51:32.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.928+0000 7fa78ca00700 1 -- start start 2024-04-02T07:51:32.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.929+0000 7fa78ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7880cec40 0x7fa788159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:32.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.929+0000 7fa78ca00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7880d21f0 0x7fa7881e9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:32.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.929+0000 7fa78ca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7881611b0 0x7fa7880b7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:32.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.929+0000 7fa78ca00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78815a530 con 0x7fa7880cec40 2024-04-02T07:51:32.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.929+0000 7fa78ca00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7881e7aa0 con 0x7fa7880d21f0 2024-04-02T07:51:32.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.929+0000 7fa78ca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7881e7d40 con 0x7fa7881611b0 2024-04-02T07:51:32.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa785e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7880cec40 0x7fa788159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:32.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa785e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7880cec40 0x7fa788159ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44626/0 (socket says 172.21.15.89:44626) 2024-04-02T07:51:32.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa785e00700 1 -- 172.21.15.89:0/551924616 learned_addr learned my addr 172.21.15.89:0/551924616 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:32.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa785400700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7880d21f0 0x7fa7881e9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:32.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa786800700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7881611b0 0x7fa7880b7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:32.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa785e00700 1 -- 172.21.15.89:0/551924616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7881611b0 msgr2=0x7fa7880b7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:32.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa785e00700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7881611b0 0x7fa7880b7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa785e00700 1 -- 172.21.15.89:0/551924616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7880d21f0 msgr2=0x7fa7881e9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:32.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa785e00700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7880d21f0 0x7fa7881e9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa785e00700 1 -- 172.21.15.89:0/551924616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa774009550 con 0x7fa7880cec40 2024-04-02T07:51:32.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa786800700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7881611b0 0x7fa7880b7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:32.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa785e00700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7880cec40 0x7fa788159ff0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fa77800c450 tx=0x7fa77800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:32.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa77e000700 1 -- 172.21.15.89:0/551924616 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7780052d0 con 0x7fa7880cec40 2024-04-02T07:51:32.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa77e000700 1 -- 172.21.15.89:0/551924616 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa778005430 con 0x7fa7880cec40 2024-04-02T07:51:32.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.930+0000 7fa78ca00700 1 -- 172.21.15.89:0/551924616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7881e8020 con 0x7fa7880cec40 2024-04-02T07:51:32.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.931+0000 7fa77e000700 1 -- 172.21.15.89:0/551924616 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7780038c0 con 0x7fa7880cec40 2024-04-02T07:51:32.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.931+0000 7fa78ca00700 1 -- 172.21.15.89:0/551924616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7881e85e0 con 0x7fa7880cec40 2024-04-02T07:51:32.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.932+0000 7fa78ca00700 1 -- 172.21.15.89:0/551924616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa788067720 con 0x7fa7880cec40 2024-04-02T07:51:32.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.932+0000 7fa77e000700 1 -- 172.21.15.89:0/551924616 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa778003a20 con 0x7fa7880cec40 2024-04-02T07:51:32.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.933+0000 7fa77e000700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa76806adb0 0x7fa76806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:32.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.934+0000 7fa77e000700 1 -- 172.21.15.89:0/551924616 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa778088a40 con 0x7fa7880cec40 2024-04-02T07:51:32.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.934+0000 7fa785400700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa76806adb0 0x7fa76806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:32.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.934+0000 7fa785400700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa76806adb0 0x7fa76806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa774009840 tx=0x7fa774008030 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:32.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:31.938+0000 7fa77e000700 1 -- 172.21.15.89:0/551924616 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa778053910 con 0x7fa7880cec40 2024-04-02T07:51:32.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.185+0000 7fa78ca00700 1 -- 172.21.15.89:0/551924616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa78815d600 con 0x7fa7880cec40 2024-04-02T07:51:32.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.713+0000 7fa77e000700 1 -- 172.21.15.89:0/551924616 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v33) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa778056f30 con 0x7fa7880cec40 2024-04-02T07:51:32.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.716+0000 7fa78ca00700 1 -- 172.21.15.89:0/551924616 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa76806adb0 msgr2=0x7fa76806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:32.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.716+0000 7fa78ca00700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa76806adb0 0x7fa76806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa774009840 tx=0x7fa774008030 comp rx=0 tx=0).stop 2024-04-02T07:51:32.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.716+0000 7fa78ca00700 1 -- 172.21.15.89:0/551924616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7880cec40 msgr2=0x7fa788159ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:32.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.716+0000 7fa78ca00700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7880cec40 0x7fa788159ff0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fa77800c450 tx=0x7fa77800c810 comp rx=0 tx=0).stop 2024-04-02T07:51:32.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.716+0000 7fa78ca00700 1 -- 172.21.15.89:0/551924616 shutdown_connections 2024-04-02T07:51:32.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.716+0000 7fa78ca00700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa76806adb0 0x7fa76806d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.716+0000 7fa78ca00700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7881611b0 0x7fa7880b7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.716+0000 7fa78ca00700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7880cec40 0x7fa788159ff0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.716+0000 7fa78ca00700 1 --2- 172.21.15.89:0/551924616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7880d21f0 0x7fa7881e9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.716+0000 7fa78ca00700 1 -- 172.21.15.89:0/551924616 >> 172.21.15.89:0/551924616 conn(0x7fa78808aff0 msgr2=0x7fa78815f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:32.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.716+0000 7fa78ca00700 1 -- 172.21.15.89:0/551924616 shutdown_connections 2024-04-02T07:51:32.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.716+0000 7fa78ca00700 1 -- 172.21.15.89:0/551924616 wait complete. 2024-04-02T07:51:32.736 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T07:51:32.736 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:51:32.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.925+0000 7f349f000700 1 Processor -- start 2024-04-02T07:51:32.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.926+0000 7f349f000700 1 -- start start 2024-04-02T07:51:32.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.927+0000 7f349f000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f349815da50 0x7f34980ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:32.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.927+0000 7f349f000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f34980d0cf0 0x7f34980b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:32.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.927+0000 7f349f000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34980d24e0 con 0x7f34980cec70 2024-04-02T07:51:32.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.927+0000 7f349f000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34980b95d0 con 0x7f349815da50 2024-04-02T07:51:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.927+0000 7f349f000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f349815de30 con 0x7f34980d0cf0 2024-04-02T07:51:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.927+0000 7f3497e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f349815da50 0x7f34980ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.927+0000 7f3497e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f349815da50 0x7f34980ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44634/0 (socket says 172.21.15.89:44634) 2024-04-02T07:51:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.927+0000 7f3497e00700 1 -- 172.21.15.89:0/3620865596 learned_addr learned my addr 172.21.15.89:0/3620865596 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.927+0000 7f349d000700 1 --2- 172.21.15.89:0/3620865596 >> v2:172.21.15.113:3300/0 conn(0x7f34980d0cf0 0x7f34980b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.927+0000 7f3497e00700 1 -- 172.21.15.89:0/3620865596 >> v2:172.21.15.113:3300/0 conn(0x7f34980d0cf0 msgr2=0x7f34980b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.927+0000 7f3497e00700 1 --2- 172.21.15.89:0/3620865596 >> v2:172.21.15.113:3300/0 conn(0x7f34980d0cf0 0x7f34980b6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.927+0000 7f3497e00700 1 -- 172.21.15.89:0/3620865596 >> v1:172.21.15.113:6789/0 conn(0x7f34980cec70 legacy=0x7f34980d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.928+0000 7f3497e00700 1 -- 172.21.15.89:0/3620865596 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34980d27c0 con 0x7f349815da50 2024-04-02T07:51:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.928+0000 7f349d000700 1 --2- 172.21.15.89:0/3620865596 >> v2:172.21.15.113:3300/0 conn(0x7f34980d0cf0 0x7f34980b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.928+0000 7f3497e00700 1 --2- 172.21.15.89:0/3620865596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f349815da50 0x7f34980ce730 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f34980d1660 tx=0x7f3488009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a5422712a2e261 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.928+0000 7f3496a00700 1 -- 172.21.15.89:0/3620865596 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3488004230 con 0x7f349815da50 2024-04-02T07:51:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.928+0000 7f3496a00700 1 -- 172.21.15.89:0/3620865596 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3488004390 con 0x7f349815da50 2024-04-02T07:51:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.928+0000 7f3496a00700 1 -- 172.21.15.89:0/3620865596 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34880036a0 con 0x7f349815da50 2024-04-02T07:51:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.928+0000 7f349f000700 1 -- 172.21.15.89:0/3620865596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f349815da50 msgr2=0x7f34980ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.928+0000 7f349f000700 1 --2- 172.21.15.89:0/3620865596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f349815da50 0x7f34980ce730 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f34980d1660 tx=0x7f3488009bd0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.929+0000 7f349f000700 1 -- 172.21.15.89:0/3620865596 shutdown_connections 2024-04-02T07:51:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.929+0000 7f349f000700 1 --2- 172.21.15.89:0/3620865596 >> v2:172.21.15.113:3300/0 conn(0x7f34980d0cf0 0x7f34980b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.929+0000 7f349f000700 1 --2- 172.21.15.89:0/3620865596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f349815da50 0x7f34980ce730 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.929+0000 7f349f000700 1 -- 172.21.15.89:0/3620865596 >> 172.21.15.89:0/3620865596 conn(0x7f34980666a0 msgr2=0x7f3498066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.929+0000 7f349f000700 1 -- 172.21.15.89:0/3620865596 shutdown_connections 2024-04-02T07:51:32.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.929+0000 7f349f000700 1 -- 172.21.15.89:0/3620865596 wait complete. 2024-04-02T07:51:32.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.930+0000 7f349f000700 1 Processor -- start 2024-04-02T07:51:32.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.930+0000 7f349f000700 1 -- start start 2024-04-02T07:51:32.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.931+0000 7f349f000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34980cec70 0x7f3498157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:32.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.931+0000 7f349f000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34980d0cf0 0x7f349815efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:32.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.931+0000 7f349f000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f349815da50 0x7f349815f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:32.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.931+0000 7f349f000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34980cc3c0 con 0x7f34980d0cf0 2024-04-02T07:51:32.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.931+0000 7f349f000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349815fa60 con 0x7f34980cec70 2024-04-02T07:51:32.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.931+0000 7f349f000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34981538b0 con 0x7f349815da50 2024-04-02T07:51:32.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f3497400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34980d0cf0 0x7f349815efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:32.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f3497400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34980d0cf0 0x7f349815efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44640/0 (socket says 172.21.15.89:44640) 2024-04-02T07:51:32.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f3497400700 1 -- 172.21.15.89:0/3676527460 learned_addr learned my addr 172.21.15.89:0/3676527460 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:32.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f3497e00700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34980cec70 0x7f3498157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:32.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f349d000700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f349815da50 0x7f349815f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:32.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f3497400700 1 -- 172.21.15.89:0/3676527460 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f349815da50 msgr2=0x7f349815f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:32.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f3497400700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f349815da50 0x7f349815f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f3497400700 1 -- 172.21.15.89:0/3676527460 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34980cec70 msgr2=0x7f3498157150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:32.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f3497400700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34980cec70 0x7f3498157150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:32.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f3497400700 1 -- 172.21.15.89:0/3676527460 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3488009550 con 0x7f34980d0cf0 2024-04-02T07:51:32.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f349d000700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f349815da50 0x7f349815f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:32.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f3497400700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34980d0cf0 0x7f349815efe0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f348400c4a0 tx=0x7f348400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:32.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f348fe00700 1 -- 172.21.15.89:0/3676527460 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34840054e0 con 0x7f34980d0cf0 2024-04-02T07:51:32.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f348fe00700 1 -- 172.21.15.89:0/3676527460 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3484003680 con 0x7f34980d0cf0 2024-04-02T07:51:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f349f000700 1 -- 172.21.15.89:0/3676527460 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34980cba20 con 0x7f34980d0cf0 2024-04-02T07:51:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.932+0000 7f348fe00700 1 -- 172.21.15.89:0/3676527460 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3484003a10 con 0x7f34980d0cf0 2024-04-02T07:51:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.933+0000 7f349f000700 1 -- 172.21.15.89:0/3676527460 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34980cbfe0 con 0x7f34980d0cf0 2024-04-02T07:51:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.933+0000 7f349f000700 1 -- 172.21.15.89:0/3676527460 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34980cc500 con 0x7f34980d0cf0 2024-04-02T07:51:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.934+0000 7f348fe00700 1 -- 172.21.15.89:0/3676527460 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3484003b70 con 0x7f34980d0cf0 2024-04-02T07:51:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.935+0000 7f348fe00700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f347806adb0 0x7f347806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.935+0000 7f348fe00700 1 -- 172.21.15.89:0/3676527460 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3484088930 con 0x7f34980d0cf0 2024-04-02T07:51:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.935+0000 7f3497e00700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f347806adb0 0x7f347806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.936+0000 7f3497e00700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f347806adb0 0x7f347806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3488009130 tx=0x7f3488008030 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:32.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:32.940+0000 7f348fe00700 1 -- 172.21.15.89:0/3676527460 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f34980cc500 con 0x7f34980d0cf0 2024-04-02T07:51:33.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.186+0000 7f349f000700 1 -- 172.21.15.89:0/3676527460 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34980cc500 con 0x7f34980d0cf0 2024-04-02T07:51:33.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.187+0000 7f348fe00700 1 -- 172.21.15.89:0/3676527460 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f34980cc500 con 0x7f34980d0cf0 2024-04-02T07:51:33.188 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:33.189 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2025634273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2025634273},{"type":"v1","addr":"172.21.15.113:6834","nonce":2025634273}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2247445394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2247445394},{"type":"v1","addr":"172.21.15.113:6835","nonce":2247445394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1785147820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1785147820},{"type":"v1","addr":"172.21.15.89:6835","nonce":1785147820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:51:29.684122+0000","modified":"2024-04-02T07:51:32.705923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6432":{"gid":6432,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/1479795102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1479795102},{"type":"v1","addr":"172.21.15.89:6837","nonce":1479795102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-02T07:51:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.189+0000 7f349f000700 1 -- 172.21.15.89:0/3676527460 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f347806adb0 msgr2=0x7f347806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.189+0000 7f349f000700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f347806adb0 0x7f347806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3488009130 tx=0x7f3488008030 comp rx=0 tx=0).stop 2024-04-02T07:51:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.189+0000 7f349f000700 1 -- 172.21.15.89:0/3676527460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34980d0cf0 msgr2=0x7f349815efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.189+0000 7f349f000700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34980d0cf0 0x7f349815efe0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f348400c4a0 tx=0x7f348400c7b0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.189+0000 7f349f000700 1 -- 172.21.15.89:0/3676527460 shutdown_connections 2024-04-02T07:51:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.189+0000 7f349f000700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f347806adb0 0x7f347806d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.189+0000 7f349f000700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34980cec70 0x7f3498157150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.189+0000 7f349f000700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f349815da50 0x7f349815f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.189+0000 7f349f000700 1 --2- 172.21.15.89:0/3676527460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34980d0cf0 0x7f349815efe0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.189+0000 7f349f000700 1 -- 172.21.15.89:0/3676527460 >> 172.21.15.89:0/3676527460 conn(0x7f34980666a0 msgr2=0x7f34980b8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.189+0000 7f349f000700 1 -- 172.21.15.89:0/3676527460 shutdown_connections 2024-04-02T07:51:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.189+0000 7f349f000700 1 -- 172.21.15.89:0/3676527460 wait complete. 2024-04-02T07:51:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 33 2024-04-02T07:51:33.209 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:51:33.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.397+0000 7fb7cd400700 1 Processor -- start 2024-04-02T07:51:33.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.397+0000 7fb7cd400700 1 -- start start 2024-04-02T07:51:33.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.398+0000 7fb7cd400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb7c80d18b0 0x7fb7c80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.399+0000 7fb7cd400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7c8161590 0x7fb7c8161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.399+0000 7fb7cd400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c80ceda0 con 0x7fb7c8160f10 2024-04-02T07:51:33.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.399+0000 7fb7cd400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c80d1d30 con 0x7fb7c8161590 2024-04-02T07:51:33.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.399+0000 7fb7cd400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c8161f60 con 0x7fb7c80d18b0 2024-04-02T07:51:33.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.399+0000 7fb7c7200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7c8161590 0x7fb7c8161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.399+0000 7fb7c6800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb7c8160f10 0x7fb7c80d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58762/0 (socket says 172.21.15.89:58762) 2024-04-02T07:51:33.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.399+0000 7fb7c6800700 1 -- 172.21.15.89:0/849936672 learned_addr learned my addr 172.21.15.89:0/849936672 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:33.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.399+0000 7fb7c5e00700 1 --2- 172.21.15.89:0/849936672 >> v2:172.21.15.113:3300/0 conn(0x7fb7c80d18b0 0x7fb7c80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.399+0000 7fb7c7200700 1 -- 172.21.15.89:0/849936672 >> v2:172.21.15.113:3300/0 conn(0x7fb7c80d18b0 msgr2=0x7fb7c80cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:33.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.399+0000 7fb7c7200700 1 --2- 172.21.15.89:0/849936672 >> v2:172.21.15.113:3300/0 conn(0x7fb7c80d18b0 0x7fb7c80cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.399+0000 7fb7c7200700 1 -- 172.21.15.89:0/849936672 >> v1:172.21.15.89:6789/0 conn(0x7fb7c8160f10 legacy=0x7fb7c80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:33.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.399+0000 7fb7c7200700 1 -- 172.21.15.89:0/849936672 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c81612f0 con 0x7fb7c8161590 2024-04-02T07:51:33.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.400+0000 7fb7c7200700 1 --2- 172.21.15.89:0/849936672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8161590 0x7fb7c8161a20 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb7c808ca30 tx=0x7fb7c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbce35336f0812 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:33.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.400+0000 7fb7c5400700 1 -- 172.21.15.89:0/849936672 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c000e070 con 0x7fb7c8161590 2024-04-02T07:51:33.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.400+0000 7fb7c5400700 1 -- 172.21.15.89:0/849936672 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c000bbd0 con 0x7fb7c8161590 2024-04-02T07:51:33.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.400+0000 7fb7c5400700 1 -- 172.21.15.89:0/849936672 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0012430 con 0x7fb7c8161590 2024-04-02T07:51:33.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.400+0000 7fb7cd400700 1 -- 172.21.15.89:0/849936672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8161590 msgr2=0x7fb7c8161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:33.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.400+0000 7fb7cd400700 1 --2- 172.21.15.89:0/849936672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8161590 0x7fb7c8161a20 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb7c808ca30 tx=0x7fb7c000b330 comp rx=0 tx=0).stop 2024-04-02T07:51:33.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.400+0000 7fb7cd400700 1 -- 172.21.15.89:0/849936672 shutdown_connections 2024-04-02T07:51:33.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.400+0000 7fb7cd400700 1 --2- 172.21.15.89:0/849936672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8161590 0x7fb7c8161a20 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.400+0000 7fb7cd400700 1 --2- 172.21.15.89:0/849936672 >> v2:172.21.15.113:3300/0 conn(0x7fb7c80d18b0 0x7fb7c80cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.400+0000 7fb7cd400700 1 -- 172.21.15.89:0/849936672 >> 172.21.15.89:0/849936672 conn(0x7fb7c808aff0 msgr2=0x7fb7c80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:33.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.400+0000 7fb7cd400700 1 -- 172.21.15.89:0/849936672 shutdown_connections 2024-04-02T07:51:33.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.400+0000 7fb7cd400700 1 -- 172.21.15.89:0/849936672 wait complete. 2024-04-02T07:51:33.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.402+0000 7fb7cd400700 1 Processor -- start 2024-04-02T07:51:33.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.402+0000 7fb7cd400700 1 -- start start 2024-04-02T07:51:33.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.403+0000 7fb7cd400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d18b0 0x7fb7c8155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.403+0000 7fb7cd400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8160f10 0x7fb7c815cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.403+0000 7fb7cd400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8161590 0x7fb7c80b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.403+0000 7fb7cd400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c81571c0 con 0x7fb7c8161590 2024-04-02T07:51:33.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.403+0000 7fb7cd400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c81565b0 con 0x7fb7c80d18b0 2024-04-02T07:51:33.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.403+0000 7fb7cd400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c8156890 con 0x7fb7c8160f10 2024-04-02T07:51:33.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.403+0000 7fb7c7200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8161590 0x7fb7c80b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.404+0000 7fb7c7200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8161590 0x7fb7c80b82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44656/0 (socket says 172.21.15.89:44656) 2024-04-02T07:51:33.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.404+0000 7fb7c7200700 1 -- 172.21.15.89:0/673149726 learned_addr learned my addr 172.21.15.89:0/673149726 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:33.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.404+0000 7fb7c6800700 1 --2- 172.21.15.89:0/673149726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d18b0 0x7fb7c8155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.404+0000 7fb7c5e00700 1 --2- 172.21.15.89:0/673149726 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8160f10 0x7fb7c815cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.404+0000 7fb7c7200700 1 -- 172.21.15.89:0/673149726 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8160f10 msgr2=0x7fb7c815cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:33.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.404+0000 7fb7c7200700 1 --2- 172.21.15.89:0/673149726 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8160f10 0x7fb7c815cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.404+0000 7fb7c7200700 1 -- 172.21.15.89:0/673149726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d18b0 msgr2=0x7fb7c8155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:33.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.404+0000 7fb7c7200700 1 --2- 172.21.15.89:0/673149726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d18b0 0x7fb7c8155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.404+0000 7fb7c7200700 1 -- 172.21.15.89:0/673149726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c0009a70 con 0x7fb7c8161590 2024-04-02T07:51:33.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.404+0000 7fb7c7200700 1 --2- 172.21.15.89:0/673149726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8161590 0x7fb7c80b82f0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000b370 tx=0x7fb7c0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:33.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.404+0000 7fb7bea00700 1 -- 172.21.15.89:0/673149726 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0012430 con 0x7fb7c8161590 2024-04-02T07:51:33.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.404+0000 7fb7bea00700 1 -- 172.21.15.89:0/673149726 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c0020070 con 0x7fb7c8161590 2024-04-02T07:51:33.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.404+0000 7fb7bea00700 1 -- 172.21.15.89:0/673149726 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c00098f0 con 0x7fb7c8161590 2024-04-02T07:51:33.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.404+0000 7fb7cd400700 1 -- 172.21.15.89:0/673149726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c8156b70 con 0x7fb7c8161590 2024-04-02T07:51:33.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.405+0000 7fb7cd400700 1 -- 172.21.15.89:0/673149726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c80cb490 con 0x7fb7c8161590 2024-04-02T07:51:33.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.406+0000 7fb7cd400700 1 -- 172.21.15.89:0/673149726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c8067720 con 0x7fb7c8161590 2024-04-02T07:51:33.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.407+0000 7fb7bea00700 1 -- 172.21.15.89:0/673149726 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb7c0004040 con 0x7fb7c8161590 2024-04-02T07:51:33.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.408+0000 7fb7bea00700 1 --2- 172.21.15.89:0/673149726 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb7a806adb0 0x7fb7a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.408+0000 7fb7bea00700 1 -- 172.21.15.89:0/673149726 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb7c0022070 con 0x7fb7c8161590 2024-04-02T07:51:33.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.408+0000 7fb7c6800700 1 --2- 172.21.15.89:0/673149726 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb7a806adb0 0x7fb7a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.409+0000 7fb7c6800700 1 --2- 172.21.15.89:0/673149726 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb7a806adb0 0x7fb7a806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8005910 tx=0x7fb7b8009040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:33.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.413+0000 7fb7bea00700 1 -- 172.21.15.89:0/673149726 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb7c0056240 con 0x7fb7c8161590 2024-04-02T07:51:33.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.614+0000 7fb7cd400700 1 -- 172.21.15.89:0/673149726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7c8157300 con 0x7fb7c8161590 2024-04-02T07:51:33.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.615+0000 7fb7bea00700 1 -- 172.21.15.89:0/673149726 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16624 (secure 0 0 0) 0x7fb7c0059860 con 0x7fb7c8161590 2024-04-02T07:51:33.617 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:33.617 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":83,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:51:30.686759+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:51:27.926367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-02T07:51:28.662600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:51:33.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.618+0000 7fb7cd400700 1 -- 172.21.15.89:0/673149726 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb7a806adb0 msgr2=0x7fb7a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:33.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.618+0000 7fb7cd400700 1 --2- 172.21.15.89:0/673149726 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb7a806adb0 0x7fb7a806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8005910 tx=0x7fb7b8009040 comp rx=0 tx=0).stop 2024-04-02T07:51:33.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.618+0000 7fb7cd400700 1 -- 172.21.15.89:0/673149726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8161590 msgr2=0x7fb7c80b82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:33.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.618+0000 7fb7cd400700 1 --2- 172.21.15.89:0/673149726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8161590 0x7fb7c80b82f0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000b370 tx=0x7fb7c0004ad0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.618+0000 7fb7cd400700 1 -- 172.21.15.89:0/673149726 shutdown_connections 2024-04-02T07:51:33.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.618+0000 7fb7cd400700 1 --2- 172.21.15.89:0/673149726 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb7a806adb0 0x7fb7a806d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.618+0000 7fb7cd400700 1 --2- 172.21.15.89:0/673149726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7c80d18b0 0x7fb7c8155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.618+0000 7fb7cd400700 1 --2- 172.21.15.89:0/673149726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c8161590 0x7fb7c80b82f0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.618+0000 7fb7cd400700 1 --2- 172.21.15.89:0/673149726 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7c8160f10 0x7fb7c815cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.618+0000 7fb7cd400700 1 -- 172.21.15.89:0/673149726 >> 172.21.15.89:0/673149726 conn(0x7fb7c808aff0 msgr2=0x7fb7c80ccd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:33.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.618+0000 7fb7cd400700 1 -- 172.21.15.89:0/673149726 shutdown_connections 2024-04-02T07:51:33.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.618+0000 7fb7cd400700 1 -- 172.21.15.89:0/673149726 wait complete. 2024-04-02T07:51:33.638 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T07:51:33.638 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:51:33.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.827+0000 7f40c9e00700 1 Processor -- start 2024-04-02T07:51:33.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.828+0000 7f40c9e00700 1 -- start start 2024-04-02T07:51:33.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.829+0000 7f40c9e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f40c40d18b0 0x7f40c40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.829+0000 7f40c9e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40c4161590 0x7f40c4161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.829+0000 7f40c9e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c40ceda0 con 0x7f40c4160f10 2024-04-02T07:51:33.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.829+0000 7f40c9e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c40d1d30 con 0x7f40c40d18b0 2024-04-02T07:51:33.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.829+0000 7f40c9e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c4161f60 con 0x7f40c4161590 2024-04-02T07:51:33.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.829+0000 7f40c2a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f40c4160f10 0x7f40c40d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58772/0 (socket says 172.21.15.89:58772) 2024-04-02T07:51:33.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.829+0000 7f40c2a00700 1 -- 172.21.15.89:0/3444491635 learned_addr learned my addr 172.21.15.89:0/3444491635 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:33.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.829+0000 7f40c1600700 1 -- 172.21.15.89:0/3444491635 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851944770 0 0) 0x7f40c40ceda0 con 0x7f40c4160f10 2024-04-02T07:51:33.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.829+0000 7f40c1600700 1 -- 172.21.15.89:0/3444491635 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40a80035b0 con 0x7f40c4160f10 2024-04-02T07:51:33.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.829+0000 7f40c2000700 1 --2- 172.21.15.89:0/3444491635 >> v2:172.21.15.113:3300/0 conn(0x7f40c40d18b0 0x7f40c40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.829+0000 7f40c3400700 1 --2- 172.21.15.89:0/3444491635 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40c4161590 0x7f40c4161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c1600700 1 -- 172.21.15.89:0/3444491635 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3820242192 0 0) 0x7f40a80035b0 con 0x7f40c4160f10 2024-04-02T07:51:33.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c1600700 1 -- 172.21.15.89:0/3444491635 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40c4161590 msgr2=0x7f40c4161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:33.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c1600700 1 --2- 172.21.15.89:0/3444491635 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40c4161590 0x7f40c4161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c1600700 1 -- 172.21.15.89:0/3444491635 >> v2:172.21.15.113:3300/0 conn(0x7f40c40d18b0 msgr2=0x7f40c40cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:33.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c1600700 1 --2- 172.21.15.89:0/3444491635 >> v2:172.21.15.113:3300/0 conn(0x7f40c40d18b0 0x7f40c40cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c1600700 1 -- 172.21.15.89:0/3444491635 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c41612f0 con 0x7f40c4160f10 2024-04-02T07:51:33.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c1600700 1 -- 172.21.15.89:0/3444491635 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f40b4001670 con 0x7f40c4160f10 2024-04-02T07:51:33.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c1600700 1 -- 172.21.15.89:0/3444491635 >> v1:172.21.15.89:6789/0 conn(0x7f40c4160f10 legacy=0x7f40c40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:33.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c1600700 1 --2- 172.21.15.89:0/3444491635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a8003a30 0x7f40a8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c1600700 1 --2- 172.21.15.89:0/3444491635 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40c4161590 0x7f40a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c1600700 1 --2- 172.21.15.89:0/3444491635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a8004d90 0x7f40a8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c1600700 1 -- 172.21.15.89:0/3444491635 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c40d1d30 con 0x7f40a8004d90 2024-04-02T07:51:33.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c1600700 1 -- 172.21.15.89:0/3444491635 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c4161f60 con 0x7f40a8003a30 2024-04-02T07:51:33.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c1600700 1 -- 172.21.15.89:0/3444491635 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a800bbc0 con 0x7f40c4161590 2024-04-02T07:51:33.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c3400700 1 --2- 172.21.15.89:0/3444491635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a8004d90 0x7f40a8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c2a00700 1 --2- 172.21.15.89:0/3444491635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a8003a30 0x7f40a8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c2000700 1 --2- 172.21.15.89:0/3444491635 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40c4161590 0x7f40a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c3400700 1 -- 172.21.15.89:0/3444491635 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40c4161590 msgr2=0x7f40a8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:33.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c3400700 1 --2- 172.21.15.89:0/3444491635 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40c4161590 0x7f40a8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c3400700 1 -- 172.21.15.89:0/3444491635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a8003a30 msgr2=0x7f40a8003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:33.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c3400700 1 --2- 172.21.15.89:0/3444491635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a8003a30 0x7f40a8003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.830+0000 7f40c3400700 1 -- 172.21.15.89:0/3444491635 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a80035b0 con 0x7f40a8004d90 2024-04-02T07:51:33.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.831+0000 7f40c3400700 1 --2- 172.21.15.89:0/3444491635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a8004d90 0x7f40a8005210 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f40bc007360 tx=0x7f40a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5adf1c19efe0d153 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:33.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.831+0000 7f40c1600700 1 -- 172.21.15.89:0/3444491635 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc007e80 con 0x7f40a8004d90 2024-04-02T07:51:33.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.831+0000 7f40c1600700 1 -- 172.21.15.89:0/3444491635 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40bc014400 con 0x7f40a8004d90 2024-04-02T07:51:33.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.831+0000 7f40c1600700 1 -- 172.21.15.89:0/3444491635 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc014790 con 0x7f40a8004d90 2024-04-02T07:51:33.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.831+0000 7f40c9e00700 1 -- 172.21.15.89:0/3444491635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a8004d90 msgr2=0x7f40a8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:33.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.831+0000 7f40c9e00700 1 --2- 172.21.15.89:0/3444491635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a8004d90 0x7f40a8005210 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f40bc007360 tx=0x7f40a8005880 comp rx=0 tx=0).stop 2024-04-02T07:51:33.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.831+0000 7f40c3400700 1 -- 172.21.15.89:0/3444491635 reap_dead start 2024-04-02T07:51:33.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.831+0000 7f40c9e00700 1 -- 172.21.15.89:0/3444491635 shutdown_connections 2024-04-02T07:51:33.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.831+0000 7f40c9e00700 1 -- 172.21.15.89:0/3444491635 >> 172.21.15.89:0/3444491635 conn(0x7f40c408aff0 msgr2=0x7f40c40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:33.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.832+0000 7f40c9e00700 1 -- 172.21.15.89:0/3444491635 shutdown_connections 2024-04-02T07:51:33.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.832+0000 7f40c9e00700 1 -- 172.21.15.89:0/3444491635 wait complete. 2024-04-02T07:51:33.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.833+0000 7f40c9e00700 1 Processor -- start 2024-04-02T07:51:33.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.833+0000 7f40c9e00700 1 -- start start 2024-04-02T07:51:33.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.834+0000 7f40c9e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a8004d90 0x7f40c40b8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.834+0000 7f40c9e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40c415f170 0x7f40c40b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.834+0000 7f40c9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40c40cc4c0 0x7f40c414e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.834+0000 7f40c9e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c40b85f0 con 0x7f40c40cc4c0 2024-04-02T07:51:33.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.834+0000 7f40c9e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c40b79e0 con 0x7f40a8004d90 2024-04-02T07:51:33.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.834+0000 7f40c9e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c40b7cc0 con 0x7f40c415f170 2024-04-02T07:51:33.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.835+0000 7f40c3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40c40cc4c0 0x7f40c414e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.835+0000 7f40c3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40c40cc4c0 0x7f40c414e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44672/0 (socket says 172.21.15.89:44672) 2024-04-02T07:51:33.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.835+0000 7f40c3400700 1 -- 172.21.15.89:0/3131133027 learned_addr learned my addr 172.21.15.89:0/3131133027 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:33.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.835+0000 7f40c2a00700 1 --2- 172.21.15.89:0/3131133027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a8004d90 0x7f40c40b8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.835+0000 7f40c2000700 1 --2- 172.21.15.89:0/3131133027 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40c415f170 0x7f40c40b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.835+0000 7f40c3400700 1 -- 172.21.15.89:0/3131133027 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40c415f170 msgr2=0x7f40c40b70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:33.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.835+0000 7f40c3400700 1 --2- 172.21.15.89:0/3131133027 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40c415f170 0x7f40c40b70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.835+0000 7f40c3400700 1 -- 172.21.15.89:0/3131133027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a8004d90 msgr2=0x7f40c40b8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:33.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.835+0000 7f40c3400700 1 --2- 172.21.15.89:0/3131133027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a8004d90 0x7f40c40b8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:33.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.835+0000 7f40c3400700 1 -- 172.21.15.89:0/3131133027 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c40b7fa0 con 0x7f40c40cc4c0 2024-04-02T07:51:33.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.835+0000 7f40c3400700 1 --2- 172.21.15.89:0/3131133027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40c40cc4c0 0x7f40c414e5c0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f40a800bb10 tx=0x7f40bc004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:33.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.836+0000 7f40bb400700 1 -- 172.21.15.89:0/3131133027 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc007000 con 0x7f40c40cc4c0 2024-04-02T07:51:33.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.836+0000 7f40bb400700 1 -- 172.21.15.89:0/3131133027 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40bc003a20 con 0x7f40c40cc4c0 2024-04-02T07:51:33.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.836+0000 7f40c9e00700 1 -- 172.21.15.89:0/3131133027 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c40b8280 con 0x7f40c40cc4c0 2024-04-02T07:51:33.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.836+0000 7f40bb400700 1 -- 172.21.15.89:0/3131133027 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc003e10 con 0x7f40c40cc4c0 2024-04-02T07:51:33.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.836+0000 7f40c9e00700 1 -- 172.21.15.89:0/3131133027 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c40cc940 con 0x7f40c40cc4c0 2024-04-02T07:51:33.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.837+0000 7f40c9e00700 1 -- 172.21.15.89:0/3131133027 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c40b87a0 con 0x7f40c40cc4c0 2024-04-02T07:51:33.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.838+0000 7f40bb400700 1 -- 172.21.15.89:0/3131133027 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f40bc01b070 con 0x7f40c40cc4c0 2024-04-02T07:51:33.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.838+0000 7f40bb400700 1 --2- 172.21.15.89:0/3131133027 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40a406adb0 0x7f40a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:33.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.839+0000 7f40bb400700 1 -- 172.21.15.89:0/3131133027 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f40bc08a810 con 0x7f40c40cc4c0 2024-04-02T07:51:33.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.839+0000 7f40c2a00700 1 --2- 172.21.15.89:0/3131133027 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40a406adb0 0x7f40a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:33.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.839+0000 7f40c2a00700 1 --2- 172.21.15.89:0/3131133027 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40a406adb0 0x7f40a406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f40b4001050 tx=0x7f40b4000f80 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:33.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:33.843+0000 7f40bb400700 1 -- 172.21.15.89:0/3131133027 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f40c40b87a0 con 0x7f40c40cc4c0 2024-04-02T07:51:34.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.039+0000 7f40c9e00700 1 -- 172.21.15.89:0/3131133027 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40c40ce750 con 0x7f40a406adb0 2024-04-02T07:51:34.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.050+0000 7f40bb400700 1 -- 172.21.15.89:0/3131133027 <== mgr.6180 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289094 (secure 0 0 0) 0x7f40b40070c0 con 0x7f40a406adb0 2024-04-02T07:51:34.056 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:34.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.054+0000 7f40c9e00700 1 -- 172.21.15.89:0/3131133027 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40a406adb0 msgr2=0x7f40a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:34.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.054+0000 7f40c9e00700 1 --2- 172.21.15.89:0/3131133027 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40a406adb0 0x7f40a406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f40b4001050 tx=0x7f40b4000f80 comp rx=0 tx=0).stop 2024-04-02T07:51:34.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.054+0000 7f40c9e00700 1 -- 172.21.15.89:0/3131133027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40c40cc4c0 msgr2=0x7f40c414e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:34.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.054+0000 7f40c9e00700 1 --2- 172.21.15.89:0/3131133027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40c40cc4c0 0x7f40c414e5c0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f40a800bb10 tx=0x7f40bc004b40 comp rx=0 tx=0).stop 2024-04-02T07:51:34.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.054+0000 7f40c9e00700 1 -- 172.21.15.89:0/3131133027 shutdown_connections 2024-04-02T07:51:34.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.054+0000 7f40c9e00700 1 --2- 172.21.15.89:0/3131133027 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40a406adb0 0x7f40a406d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.054+0000 7f40c9e00700 1 --2- 172.21.15.89:0/3131133027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a8004d90 0x7f40c40b8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.054+0000 7f40c9e00700 1 --2- 172.21.15.89:0/3131133027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40c40cc4c0 0x7f40c414e5c0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.054+0000 7f40c9e00700 1 --2- 172.21.15.89:0/3131133027 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40c415f170 0x7f40c40b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.054+0000 7f40c9e00700 1 -- 172.21.15.89:0/3131133027 >> 172.21.15.89:0/3131133027 conn(0x7f40c408aff0 msgr2=0x7f40c40cab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:34.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.054+0000 7f40c9e00700 1 -- 172.21.15.89:0/3131133027 shutdown_connections 2024-04-02T07:51:34.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.054+0000 7f40c9e00700 1 -- 172.21.15.89:0/3131133027 wait complete. 2024-04-02T07:51:34.060 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:51:34.075 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-02T07:51:32.127096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":270228,"kb_used_data":8960,"kb_used_omap":12,"kb_used_meta":261107,"kb_avail":754704492,"statfs":{"total":773094113280,"available":772817399808,"internally_reserved":0,"allocated":9175040,"data_stored":5012830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":267374136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000700"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.718181+0000","last_change":"2024-04-02T07:51:29.708984+0000","last_active":"2024-04-02T07:51:29.718181+0000","last_peered":"2024-04-02T07:51:29.718181+0000","last_clean":"2024-04-02T07:51:29.718181+0000","last_became_active":"2024-04-02T07:51:29.708051+0000","last_became_peered":"2024-04-02T07:51:29.708051+0000","last_unstale":"2024-04-02T07:51:29.718181+0000","last_undegraded":"2024-04-02T07:51:29.718181+0000","last_fullsized":"2024-04-02T07:51:29.718181+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:42:13.876241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702347+0000","last_change":"2024-04-02T07:51:29.718862+0000","last_active":"2024-04-02T07:51:30.702347+0000","last_peered":"2024-04-02T07:51:30.702347+0000","last_clean":"2024-04-02T07:51:30.702347+0000","last_became_active":"2024-04-02T07:51:29.718570+0000","last_became_peered":"2024-04-02T07:51:29.718570+0000","last_unstale":"2024-04-02T07:51:30.702347+0000","last_undegraded":"2024-04-02T07:51:30.702347+0000","last_fullsized":"2024-04-02T07:51:30.702347+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:51.717980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103382+0000","last_change":"2024-04-02T07:51:29.710138+0000","last_active":"2024-04-02T07:51:31.103382+0000","last_peered":"2024-04-02T07:51:31.103382+0000","last_clean":"2024-04-02T07:51:31.103382+0000","last_became_active":"2024-04-02T07:51:29.709891+0000","last_became_peered":"2024-04-02T07:51:29.709891+0000","last_unstale":"2024-04-02T07:51:31.103382+0000","last_undegraded":"2024-04-02T07:51:31.103382+0000","last_fullsized":"2024-04-02T07:51:31.103382+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:42:13.876241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696788+0000","last_change":"2024-04-02T07:51:29.719115+0000","last_active":"2024-04-02T07:51:30.696788+0000","last_peered":"2024-04-02T07:51:30.696788+0000","last_clean":"2024-04-02T07:51:30.696788+0000","last_became_active":"2024-04-02T07:51:29.716813+0000","last_became_peered":"2024-04-02T07:51:29.716813+0000","last_unstale":"2024-04-02T07:51:30.696788+0000","last_undegraded":"2024-04-02T07:51:30.696788+0000","last_fullsized":"2024-04-02T07:51:30.696788+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:04:04.846110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.724585+0000","last_change":"2024-04-02T07:51:29.722981+0000","last_active":"2024-04-02T07:51:29.724585+0000","last_peered":"2024-04-02T07:51:29.724585+0000","last_clean":"2024-04-02T07:51:29.724585+0000","last_became_active":"2024-04-02T07:51:29.722635+0000","last_became_peered":"2024-04-02T07:51:29.722635+0000","last_unstale":"2024-04-02T07:51:29.724585+0000","last_undegraded":"2024-04-02T07:51:29.724585+0000","last_fullsized":"2024-04-02T07:51:29.724585+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:51.717980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.717838+0000","last_change":"2024-04-02T07:51:29.709285+0000","last_active":"2024-04-02T07:51:29.717838+0000","last_peered":"2024-04-02T07:51:29.717838+0000","last_clean":"2024-04-02T07:51:29.717838+0000","last_became_active":"2024-04-02T07:51:29.707754+0000","last_became_peered":"2024-04-02T07:51:29.707754+0000","last_unstale":"2024-04-02T07:51:29.717838+0000","last_undegraded":"2024-04-02T07:51:29.717838+0000","last_fullsized":"2024-04-02T07:51:29.717838+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:21:07.376954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696369+0000","last_change":"2024-04-02T07:51:29.707857+0000","last_active":"2024-04-02T07:51:30.696369+0000","last_peered":"2024-04-02T07:51:30.696369+0000","last_clean":"2024-04-02T07:51:30.696369+0000","last_became_active":"2024-04-02T07:51:29.705640+0000","last_became_peered":"2024-04-02T07:51:29.705640+0000","last_unstale":"2024-04-02T07:51:30.696369+0000","last_undegraded":"2024-04-02T07:51:30.696369+0000","last_fullsized":"2024-04-02T07:51:30.696369+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:42:30.084208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.709565+0000","last_change":"2024-04-02T07:51:29.707298+0000","last_active":"2024-04-02T07:51:29.709565+0000","last_peered":"2024-04-02T07:51:29.709565+0000","last_clean":"2024-04-02T07:51:29.709565+0000","last_became_active":"2024-04-02T07:51:29.706557+0000","last_became_peered":"2024-04-02T07:51:29.706557+0000","last_unstale":"2024-04-02T07:51:29.709565+0000","last_undegraded":"2024-04-02T07:51:29.709565+0000","last_fullsized":"2024-04-02T07:51:29.709565+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:30:23.324413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.722316+0000","last_change":"2024-04-02T07:51:29.720164+0000","last_active":"2024-04-02T07:51:29.722316+0000","last_peered":"2024-04-02T07:51:29.722316+0000","last_clean":"2024-04-02T07:51:29.722316+0000","last_became_active":"2024-04-02T07:51:29.719928+0000","last_became_peered":"2024-04-02T07:51:29.719928+0000","last_unstale":"2024-04-02T07:51:29.722316+0000","last_undegraded":"2024-04-02T07:51:29.722316+0000","last_fullsized":"2024-04-02T07:51:29.722316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:21:07.376954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.717325+0000","last_change":"2024-04-02T07:51:29.713404+0000","last_active":"2024-04-02T07:51:29.717325+0000","last_peered":"2024-04-02T07:51:29.717325+0000","last_clean":"2024-04-02T07:51:29.717325+0000","last_became_active":"2024-04-02T07:51:29.713127+0000","last_became_peered":"2024-04-02T07:51:29.713127+0000","last_unstale":"2024-04-02T07:51:29.717325+0000","last_undegraded":"2024-04-02T07:51:29.717325+0000","last_fullsized":"2024-04-02T07:51:29.717325+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:18:41.743679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.701468+0000","last_change":"2024-04-02T07:51:29.712512+0000","last_active":"2024-04-02T07:51:30.701468+0000","last_peered":"2024-04-02T07:51:30.701468+0000","last_clean":"2024-04-02T07:51:30.701468+0000","last_became_active":"2024-04-02T07:51:29.712164+0000","last_became_peered":"2024-04-02T07:51:29.712164+0000","last_unstale":"2024-04-02T07:51:30.701468+0000","last_undegraded":"2024-04-02T07:51:30.701468+0000","last_fullsized":"2024-04-02T07:51:30.701468+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:06:44.825237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.718839+0000","last_change":"2024-04-02T07:51:29.712495+0000","last_active":"2024-04-02T07:51:29.718839+0000","last_peered":"2024-04-02T07:51:29.718839+0000","last_clean":"2024-04-02T07:51:29.718839+0000","last_became_active":"2024-04-02T07:51:29.712276+0000","last_became_peered":"2024-04-02T07:51:29.712276+0000","last_unstale":"2024-04-02T07:51:29.718839+0000","last_undegraded":"2024-04-02T07:51:29.718839+0000","last_fullsized":"2024-04-02T07:51:29.718839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:31:53.301807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.711505+0000","last_change":"2024-04-02T07:51:29.701268+0000","last_active":"2024-04-02T07:51:29.711505+0000","last_peered":"2024-04-02T07:51:29.711505+0000","last_clean":"2024-04-02T07:51:29.711505+0000","last_became_active":"2024-04-02T07:51:29.699100+0000","last_became_peered":"2024-04-02T07:51:29.699100+0000","last_unstale":"2024-04-02T07:51:29.711505+0000","last_undegraded":"2024-04-02T07:51:29.711505+0000","last_fullsized":"2024-04-02T07:51:29.711505+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:26.509824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103157+0000","last_change":"2024-04-02T07:51:29.707165+0000","last_active":"2024-04-02T07:51:31.103157+0000","last_peered":"2024-04-02T07:51:31.103157+0000","last_clean":"2024-04-02T07:51:31.103157+0000","last_became_active":"2024-04-02T07:51:29.706118+0000","last_became_peered":"2024-04-02T07:51:29.706118+0000","last_unstale":"2024-04-02T07:51:31.103157+0000","last_undegraded":"2024-04-02T07:51:31.103157+0000","last_fullsized":"2024-04-02T07:51:31.103157+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:59:03.414703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.707189+0000","last_change":"2024-04-02T07:51:29.700256+0000","last_active":"2024-04-02T07:51:29.707189+0000","last_peered":"2024-04-02T07:51:29.707189+0000","last_clean":"2024-04-02T07:51:29.707189+0000","last_became_active":"2024-04-02T07:51:29.700016+0000","last_became_peered":"2024-04-02T07:51:29.700016+0000","last_unstale":"2024-04-02T07:51:29.707189+0000","last_undegraded":"2024-04-02T07:51:29.707189+0000","last_fullsized":"2024-04-02T07:51:29.707189+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:06:00.157486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.719564+0000","last_change":"2024-04-02T07:51:29.718399+0000","last_active":"2024-04-02T07:51:29.719564+0000","last_peered":"2024-04-02T07:51:29.719564+0000","last_clean":"2024-04-02T07:51:29.719564+0000","last_became_active":"2024-04-02T07:51:29.716526+0000","last_became_peered":"2024-04-02T07:51:29.716526+0000","last_unstale":"2024-04-02T07:51:29.719564+0000","last_undegraded":"2024-04-02T07:51:29.719564+0000","last_fullsized":"2024-04-02T07:51:29.719564+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:59:28.338083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.718209+0000","last_change":"2024-04-02T07:51:29.714366+0000","last_active":"2024-04-02T07:51:29.718209+0000","last_peered":"2024-04-02T07:51:29.718209+0000","last_clean":"2024-04-02T07:51:29.718209+0000","last_became_active":"2024-04-02T07:51:29.713548+0000","last_became_peered":"2024-04-02T07:51:29.713548+0000","last_unstale":"2024-04-02T07:51:29.718209+0000","last_undegraded":"2024-04-02T07:51:29.718209+0000","last_fullsized":"2024-04-02T07:51:29.718209+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:37:43.467101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103352+0000","last_change":"2024-04-02T07:51:29.718663+0000","last_active":"2024-04-02T07:51:31.103352+0000","last_peered":"2024-04-02T07:51:31.103352+0000","last_clean":"2024-04-02T07:51:31.103352+0000","last_became_active":"2024-04-02T07:51:29.718412+0000","last_became_peered":"2024-04-02T07:51:29.718412+0000","last_unstale":"2024-04-02T07:51:31.103352+0000","last_undegraded":"2024-04-02T07:51:31.103352+0000","last_fullsized":"2024-04-02T07:51:31.103352+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:51.717980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702439+0000","last_change":"2024-04-02T07:51:29.715618+0000","last_active":"2024-04-02T07:51:30.702439+0000","last_peered":"2024-04-02T07:51:30.702439+0000","last_clean":"2024-04-02T07:51:30.702439+0000","last_became_active":"2024-04-02T07:51:29.703080+0000","last_became_peered":"2024-04-02T07:51:29.703080+0000","last_unstale":"2024-04-02T07:51:30.702439+0000","last_undegraded":"2024-04-02T07:51:30.702439+0000","last_fullsized":"2024-04-02T07:51:30.702439+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:42:13.876241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.701488+0000","last_change":"2024-04-02T07:51:29.715915+0000","last_active":"2024-04-02T07:51:30.701488+0000","last_peered":"2024-04-02T07:51:30.701488+0000","last_clean":"2024-04-02T07:51:30.701488+0000","last_became_active":"2024-04-02T07:51:29.715679+0000","last_became_peered":"2024-04-02T07:51:29.715679+0000","last_unstale":"2024-04-02T07:51:30.701488+0000","last_undegraded":"2024-04-02T07:51:30.701488+0000","last_fullsized":"2024-04-02T07:51:30.701488+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:59:01.142628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.711539+0000","last_change":"2024-04-02T07:51:29.704509+0000","last_active":"2024-04-02T07:51:29.711539+0000","last_peered":"2024-04-02T07:51:29.711539+0000","last_clean":"2024-04-02T07:51:29.711539+0000","last_became_active":"2024-04-02T07:51:29.702810+0000","last_became_peered":"2024-04-02T07:51:29.702810+0000","last_unstale":"2024-04-02T07:51:29.711539+0000","last_undegraded":"2024-04-02T07:51:29.711539+0000","last_fullsized":"2024-04-02T07:51:29.711539+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:17:49.862029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.701426+0000","last_change":"2024-04-02T07:51:29.718728+0000","last_active":"2024-04-02T07:51:30.701426+0000","last_peered":"2024-04-02T07:51:30.701426+0000","last_clean":"2024-04-02T07:51:30.701426+0000","last_became_active":"2024-04-02T07:51:29.718439+0000","last_became_peered":"2024-04-02T07:51:29.718439+0000","last_unstale":"2024-04-02T07:51:30.701426+0000","last_undegraded":"2024-04-02T07:51:30.701426+0000","last_fullsized":"2024-04-02T07:51:30.701426+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:04:57.953364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696603+0000","last_change":"2024-04-02T07:51:29.708035+0000","last_active":"2024-04-02T07:51:30.696603+0000","last_peered":"2024-04-02T07:51:30.696603+0000","last_clean":"2024-04-02T07:51:30.696603+0000","last_became_active":"2024-04-02T07:51:29.705946+0000","last_became_peered":"2024-04-02T07:51:29.705946+0000","last_unstale":"2024-04-02T07:51:30.696603+0000","last_undegraded":"2024-04-02T07:51:30.696603+0000","last_fullsized":"2024-04-02T07:51:30.696603+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:38:36.206166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.722372+0000","last_change":"2024-04-02T07:51:29.721011+0000","last_active":"2024-04-02T07:51:29.722372+0000","last_peered":"2024-04-02T07:51:29.722372+0000","last_clean":"2024-04-02T07:51:29.722372+0000","last_became_active":"2024-04-02T07:51:29.720430+0000","last_became_peered":"2024-04-02T07:51:29.720430+0000","last_unstale":"2024-04-02T07:51:29.722372+0000","last_undegraded":"2024-04-02T07:51:29.722372+0000","last_fullsized":"2024-04-02T07:51:29.722372+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:58:01.264306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702621+0000","last_change":"2024-04-02T07:51:29.719460+0000","last_active":"2024-04-02T07:51:30.702621+0000","last_peered":"2024-04-02T07:51:30.702621+0000","last_clean":"2024-04-02T07:51:30.702621+0000","last_became_active":"2024-04-02T07:51:29.719134+0000","last_became_peered":"2024-04-02T07:51:29.719134+0000","last_unstale":"2024-04-02T07:51:30.702621+0000","last_undegraded":"2024-04-02T07:51:30.702621+0000","last_fullsized":"2024-04-02T07:51:30.702621+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:18:41.743679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.717061+0000","last_change":"2024-04-02T07:51:29.713185+0000","last_active":"2024-04-02T07:51:29.717061+0000","last_peered":"2024-04-02T07:51:29.717061+0000","last_clean":"2024-04-02T07:51:29.717061+0000","last_became_active":"2024-04-02T07:51:29.712960+0000","last_became_peered":"2024-04-02T07:51:29.712960+0000","last_unstale":"2024-04-02T07:51:29.717061+0000","last_undegraded":"2024-04-02T07:51:29.717061+0000","last_fullsized":"2024-04-02T07:51:29.717061+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:02:41.176141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702523+0000","last_change":"2024-04-02T07:51:29.716218+0000","last_active":"2024-04-02T07:51:30.702523+0000","last_peered":"2024-04-02T07:51:30.702523+0000","last_clean":"2024-04-02T07:51:30.702523+0000","last_became_active":"2024-04-02T07:51:29.715890+0000","last_became_peered":"2024-04-02T07:51:29.715890+0000","last_unstale":"2024-04-02T07:51:30.702523+0000","last_undegraded":"2024-04-02T07:51:30.702523+0000","last_fullsized":"2024-04-02T07:51:30.702523+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:59:28.338083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.715422+0000","last_change":"2024-04-02T07:51:29.709886+0000","last_active":"2024-04-02T07:51:29.715422+0000","last_peered":"2024-04-02T07:51:29.715422+0000","last_clean":"2024-04-02T07:51:29.715422+0000","last_became_active":"2024-04-02T07:51:29.708495+0000","last_became_peered":"2024-04-02T07:51:29.708495+0000","last_unstale":"2024-04-02T07:51:29.715422+0000","last_undegraded":"2024-04-02T07:51:29.715422+0000","last_fullsized":"2024-04-02T07:51:29.715422+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:38:36.206166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.709668+0000","last_change":"2024-04-02T07:51:29.706033+0000","last_active":"2024-04-02T07:51:29.709668+0000","last_peered":"2024-04-02T07:51:29.709668+0000","last_clean":"2024-04-02T07:51:29.709668+0000","last_became_active":"2024-04-02T07:51:29.705760+0000","last_became_peered":"2024-04-02T07:51:29.705760+0000","last_unstale":"2024-04-02T07:51:29.709668+0000","last_undegraded":"2024-04-02T07:51:29.709668+0000","last_fullsized":"2024-04-02T07:51:29.709668+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:42:30.084208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696548+0000","last_change":"2024-04-02T07:51:29.716230+0000","last_active":"2024-04-02T07:51:30.696548+0000","last_peered":"2024-04-02T07:51:30.696548+0000","last_clean":"2024-04-02T07:51:30.696548+0000","last_became_active":"2024-04-02T07:51:29.715641+0000","last_became_peered":"2024-04-02T07:51:29.715641+0000","last_unstale":"2024-04-02T07:51:30.696548+0000","last_undegraded":"2024-04-02T07:51:30.696548+0000","last_fullsized":"2024-04-02T07:51:30.696548+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:17.086170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702318+0000","last_change":"2024-04-02T07:51:29.712337+0000","last_active":"2024-04-02T07:51:30.702318+0000","last_peered":"2024-04-02T07:51:30.702318+0000","last_clean":"2024-04-02T07:51:30.702318+0000","last_became_active":"2024-04-02T07:51:29.711742+0000","last_became_peered":"2024-04-02T07:51:29.711742+0000","last_unstale":"2024-04-02T07:51:30.702318+0000","last_undegraded":"2024-04-02T07:51:30.702318+0000","last_fullsized":"2024-04-02T07:51:30.702318+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:08:52.637936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.701278+0000","last_change":"2024-04-02T07:51:29.718809+0000","last_active":"2024-04-02T07:51:30.701278+0000","last_peered":"2024-04-02T07:51:30.701278+0000","last_clean":"2024-04-02T07:51:30.701278+0000","last_became_active":"2024-04-02T07:51:29.718537+0000","last_became_peered":"2024-04-02T07:51:29.718537+0000","last_unstale":"2024-04-02T07:51:30.701278+0000","last_undegraded":"2024-04-02T07:51:30.701278+0000","last_fullsized":"2024-04-02T07:51:30.701278+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:56:25.462955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.717238+0000","last_change":"2024-04-02T07:51:29.709083+0000","last_active":"2024-04-02T07:51:29.717238+0000","last_peered":"2024-04-02T07:51:29.717238+0000","last_clean":"2024-04-02T07:51:29.717238+0000","last_became_active":"2024-04-02T07:51:29.708394+0000","last_became_peered":"2024-04-02T07:51:29.708394+0000","last_unstale":"2024-04-02T07:51:29.717238+0000","last_undegraded":"2024-04-02T07:51:29.717238+0000","last_fullsized":"2024-04-02T07:51:29.717238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:08.732683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1f","version":"82'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.701874+0000","last_change":"2024-04-02T07:51:29.198522+0000","last_active":"2024-04-02T07:51:30.701874+0000","last_peered":"2024-04-02T07:51:30.701874+0000","last_clean":"2024-04-02T07:51:30.701874+0000","last_became_active":"2024-04-02T07:51:29.196839+0000","last_became_peered":"2024-04-02T07:51:29.196839+0000","last_unstale":"2024-04-02T07:51:30.701874+0000","last_undegraded":"2024-04-02T07:51:30.701874+0000","last_fullsized":"2024-04-02T07:51:30.701874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:50.709598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.709071+0000","last_change":"2024-04-02T07:51:29.706632+0000","last_active":"2024-04-02T07:51:29.709071+0000","last_peered":"2024-04-02T07:51:29.709071+0000","last_clean":"2024-04-02T07:51:29.709071+0000","last_became_active":"2024-04-02T07:51:29.706343+0000","last_became_peered":"2024-04-02T07:51:29.706343+0000","last_unstale":"2024-04-02T07:51:29.709071+0000","last_undegraded":"2024-04-02T07:51:29.709071+0000","last_fullsized":"2024-04-02T07:51:29.709071+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:58:01.264306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"82'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702545+0000","last_change":"2024-04-02T07:51:29.210507+0000","last_active":"2024-04-02T07:51:30.702545+0000","last_peered":"2024-04-02T07:51:30.702545+0000","last_clean":"2024-04-02T07:51:30.702545+0000","last_became_active":"2024-04-02T07:51:29.210204+0000","last_became_peered":"2024-04-02T07:51:29.210204+0000","last_unstale":"2024-04-02T07:51:30.702545+0000","last_undegraded":"2024-04-02T07:51:30.702545+0000","last_fullsized":"2024-04-02T07:51:30.702545+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:10:14.951279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.716857+0000","last_change":"2024-04-02T07:51:29.714388+0000","last_active":"2024-04-02T07:51:29.716857+0000","last_peered":"2024-04-02T07:51:29.716857+0000","last_clean":"2024-04-02T07:51:29.716857+0000","last_became_active":"2024-04-02T07:51:29.714082+0000","last_became_peered":"2024-04-02T07:51:29.714082+0000","last_unstale":"2024-04-02T07:51:29.716857+0000","last_undegraded":"2024-04-02T07:51:29.716857+0000","last_fullsized":"2024-04-02T07:51:29.716857+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:04:04.846110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"82'2","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.723953+0000","last_change":"2024-04-02T07:51:29.208521+0000","last_active":"2024-04-02T07:51:29.723953+0000","last_peered":"2024-04-02T07:51:29.723953+0000","last_clean":"2024-04-02T07:51:29.723953+0000","last_became_active":"2024-04-02T07:51:29.208245+0000","last_became_peered":"2024-04-02T07:51:29.208245+0000","last_unstale":"2024-04-02T07:51:29.723953+0000","last_undegraded":"2024-04-02T07:51:29.723953+0000","last_fullsized":"2024-04-02T07:51:29.723953+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:03.852318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.709047+0000","last_change":"2024-04-02T07:51:29.697968+0000","last_active":"2024-04-02T07:51:29.709047+0000","last_peered":"2024-04-02T07:51:29.709047+0000","last_clean":"2024-04-02T07:51:29.709047+0000","last_became_active":"2024-04-02T07:51:29.697191+0000","last_became_peered":"2024-04-02T07:51:29.697191+0000","last_unstale":"2024-04-02T07:51:29.709047+0000","last_undegraded":"2024-04-02T07:51:29.709047+0000","last_fullsized":"2024-04-02T07:51:29.709047+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:31:53.301807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"82'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696942+0000","last_change":"2024-04-02T07:51:29.211917+0000","last_active":"2024-04-02T07:51:30.696942+0000","last_peered":"2024-04-02T07:51:30.696942+0000","last_clean":"2024-04-02T07:51:30.696942+0000","last_became_active":"2024-04-02T07:51:29.211624+0000","last_became_peered":"2024-04-02T07:51:29.211624+0000","last_unstale":"2024-04-02T07:51:30.696942+0000","last_undegraded":"2024-04-02T07:51:30.696942+0000","last_fullsized":"2024-04-02T07:51:30.696942+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:41:46.481303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.715337+0000","last_change":"2024-04-02T07:51:29.709126+0000","last_active":"2024-04-02T07:51:29.715337+0000","last_peered":"2024-04-02T07:51:29.715337+0000","last_clean":"2024-04-02T07:51:29.715337+0000","last_became_active":"2024-04-02T07:51:29.708861+0000","last_became_peered":"2024-04-02T07:51:29.708861+0000","last_unstale":"2024-04-02T07:51:29.715337+0000","last_undegraded":"2024-04-02T07:51:29.715337+0000","last_fullsized":"2024-04-02T07:51:29.715337+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:26:51.319846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1b","version":"82'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103304+0000","last_change":"2024-04-02T07:51:29.202610+0000","last_active":"2024-04-02T07:51:31.103304+0000","last_peered":"2024-04-02T07:51:31.103304+0000","last_clean":"2024-04-02T07:51:31.103304+0000","last_became_active":"2024-04-02T07:51:29.202339+0000","last_became_peered":"2024-04-02T07:51:29.202339+0000","last_unstale":"2024-04-02T07:51:31.103304+0000","last_undegraded":"2024-04-02T07:51:31.103304+0000","last_fullsized":"2024-04-02T07:51:31.103304+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:53:15.044497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.701629+0000","last_change":"2024-04-02T07:51:29.720513+0000","last_active":"2024-04-02T07:51:30.701629+0000","last_peered":"2024-04-02T07:51:30.701629+0000","last_clean":"2024-04-02T07:51:30.701629+0000","last_became_active":"2024-04-02T07:51:29.717979+0000","last_became_peered":"2024-04-02T07:51:29.717979+0000","last_unstale":"2024-04-02T07:51:30.701629+0000","last_undegraded":"2024-04-02T07:51:30.701629+0000","last_fullsized":"2024-04-02T07:51:30.701629+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:28.398387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"82'1","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.709224+0000","last_change":"2024-04-02T07:51:29.211974+0000","last_active":"2024-04-02T07:51:29.709224+0000","last_peered":"2024-04-02T07:51:29.709224+0000","last_clean":"2024-04-02T07:51:29.709224+0000","last_became_active":"2024-04-02T07:51:29.211714+0000","last_became_peered":"2024-04-02T07:51:29.211714+0000","last_unstale":"2024-04-02T07:51:29.709224+0000","last_undegraded":"2024-04-02T07:51:29.709224+0000","last_fullsized":"2024-04-02T07:51:29.709224+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:59:24.112828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103132+0000","last_change":"2024-04-02T07:51:29.715224+0000","last_active":"2024-04-02T07:51:31.103132+0000","last_peered":"2024-04-02T07:51:31.103132+0000","last_clean":"2024-04-02T07:51:31.103132+0000","last_became_active":"2024-04-02T07:51:29.714932+0000","last_became_peered":"2024-04-02T07:51:29.714932+0000","last_unstale":"2024-04-02T07:51:31.103132+0000","last_undegraded":"2024-04-02T07:51:31.103132+0000","last_fullsized":"2024-04-02T07:51:31.103132+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:37:43.467101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.703214+0000","last_change":"2024-04-02T07:51:29.197304+0000","last_active":"2024-04-02T07:51:29.703214+0000","last_peered":"2024-04-02T07:51:29.703214+0000","last_clean":"2024-04-02T07:51:29.703214+0000","last_became_active":"2024-04-02T07:51:29.196275+0000","last_became_peered":"2024-04-02T07:51:29.196275+0000","last_unstale":"2024-04-02T07:51:29.703214+0000","last_undegraded":"2024-04-02T07:51:29.703214+0000","last_fullsized":"2024-04-02T07:51:29.703214+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:53:15.044497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.724652+0000","last_change":"2024-04-02T07:51:29.723487+0000","last_active":"2024-04-02T07:51:29.724652+0000","last_peered":"2024-04-02T07:51:29.724652+0000","last_clean":"2024-04-02T07:51:29.724652+0000","last_became_active":"2024-04-02T07:51:29.723277+0000","last_became_peered":"2024-04-02T07:51:29.723277+0000","last_unstale":"2024-04-02T07:51:29.724652+0000","last_undegraded":"2024-04-02T07:51:29.724652+0000","last_fullsized":"2024-04-02T07:51:29.724652+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:08:52.637936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.689040+0000","last_change":"2024-04-02T07:51:29.202118+0000","last_active":"2024-04-02T07:51:29.689040+0000","last_peered":"2024-04-02T07:51:29.689040+0000","last_clean":"2024-04-02T07:51:29.689040+0000","last_became_active":"2024-04-02T07:51:29.200536+0000","last_became_peered":"2024-04-02T07:51:29.200536+0000","last_unstale":"2024-04-02T07:51:29.689040+0000","last_undegraded":"2024-04-02T07:51:29.689040+0000","last_fullsized":"2024-04-02T07:51:29.689040+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:24:11.272656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.710912+0000","last_change":"2024-04-02T07:51:29.703723+0000","last_active":"2024-04-02T07:51:29.710912+0000","last_peered":"2024-04-02T07:51:29.710912+0000","last_clean":"2024-04-02T07:51:29.710912+0000","last_became_active":"2024-04-02T07:51:29.703033+0000","last_became_peered":"2024-04-02T07:51:29.703033+0000","last_unstale":"2024-04-02T07:51:29.710912+0000","last_undegraded":"2024-04-02T07:51:29.710912+0000","last_fullsized":"2024-04-02T07:51:29.710912+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:06:15.474439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.702623+0000","last_change":"2024-04-02T07:51:29.205089+0000","last_active":"2024-04-02T07:51:29.702623+0000","last_peered":"2024-04-02T07:51:29.702623+0000","last_clean":"2024-04-02T07:51:29.702623+0000","last_became_active":"2024-04-02T07:51:29.204756+0000","last_became_peered":"2024-04-02T07:51:29.204756+0000","last_unstale":"2024-04-02T07:51:29.702623+0000","last_undegraded":"2024-04-02T07:51:29.702623+0000","last_fullsized":"2024-04-02T07:51:29.702623+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:32:45.599873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103484+0000","last_change":"2024-04-02T07:51:29.707289+0000","last_active":"2024-04-02T07:51:31.103484+0000","last_peered":"2024-04-02T07:51:31.103484+0000","last_clean":"2024-04-02T07:51:31.103484+0000","last_became_active":"2024-04-02T07:51:29.706957+0000","last_became_peered":"2024-04-02T07:51:29.706957+0000","last_unstale":"2024-04-02T07:51:31.103484+0000","last_undegraded":"2024-04-02T07:51:31.103484+0000","last_fullsized":"2024-04-02T07:51:31.103484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:57:02.013161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.700376+0000","last_change":"2024-04-02T07:51:29.208251+0000","last_active":"2024-04-02T07:51:29.700376+0000","last_peered":"2024-04-02T07:51:29.700376+0000","last_clean":"2024-04-02T07:51:29.700376+0000","last_became_active":"2024-04-02T07:51:29.207979+0000","last_became_peered":"2024-04-02T07:51:29.207979+0000","last_unstale":"2024-04-02T07:51:29.700376+0000","last_undegraded":"2024-04-02T07:51:29.700376+0000","last_fullsized":"2024-04-02T07:51:29.700376+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:37:14.168678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.716875+0000","last_change":"2024-04-02T07:51:29.715158+0000","last_active":"2024-04-02T07:51:29.716875+0000","last_peered":"2024-04-02T07:51:29.716875+0000","last_clean":"2024-04-02T07:51:29.716875+0000","last_became_active":"2024-04-02T07:51:29.714147+0000","last_became_peered":"2024-04-02T07:51:29.714147+0000","last_unstale":"2024-04-02T07:51:29.716875+0000","last_undegraded":"2024-04-02T07:51:29.716875+0000","last_fullsized":"2024-04-02T07:51:29.716875+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:17.086170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"82'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696943+0000","last_change":"2024-04-02T07:51:29.209225+0000","last_active":"2024-04-02T07:51:30.696943+0000","last_peered":"2024-04-02T07:51:30.696943+0000","last_clean":"2024-04-02T07:51:30.696943+0000","last_became_active":"2024-04-02T07:51:29.208698+0000","last_became_peered":"2024-04-02T07:51:29.208698+0000","last_unstale":"2024-04-02T07:51:30.696943+0000","last_undegraded":"2024-04-02T07:51:30.696943+0000","last_fullsized":"2024-04-02T07:51:30.696943+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:19.741531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696871+0000","last_change":"2024-04-02T07:51:29.709404+0000","last_active":"2024-04-02T07:51:30.696871+0000","last_peered":"2024-04-02T07:51:30.696871+0000","last_clean":"2024-04-02T07:51:30.696871+0000","last_became_active":"2024-04-02T07:51:29.705558+0000","last_became_peered":"2024-04-02T07:51:29.705558+0000","last_unstale":"2024-04-02T07:51:30.696871+0000","last_undegraded":"2024-04-02T07:51:30.696871+0000","last_fullsized":"2024-04-02T07:51:30.696871+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:24:50.492280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"82'6","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.719460+0000","last_change":"2024-04-02T07:51:29.211399+0000","last_active":"2024-04-02T07:51:29.719460+0000","last_peered":"2024-04-02T07:51:29.719460+0000","last_clean":"2024-04-02T07:51:29.719460+0000","last_became_active":"2024-04-02T07:51:29.211120+0000","last_became_peered":"2024-04-02T07:51:29.211120+0000","last_unstale":"2024-04-02T07:51:29.719460+0000","last_undegraded":"2024-04-02T07:51:29.719460+0000","last_fullsized":"2024-04-02T07:51:29.719460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:56:24.454573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.717631+0000","last_change":"2024-04-02T07:51:29.711525+0000","last_active":"2024-04-02T07:51:29.717631+0000","last_peered":"2024-04-02T07:51:29.717631+0000","last_clean":"2024-04-02T07:51:29.717631+0000","last_became_active":"2024-04-02T07:51:29.711242+0000","last_became_peered":"2024-04-02T07:51:29.711242+0000","last_unstale":"2024-04-02T07:51:29.717631+0000","last_undegraded":"2024-04-02T07:51:29.717631+0000","last_fullsized":"2024-04-02T07:51:29.717631+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:02:53.831136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"82'2","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.722128+0000","last_change":"2024-04-02T07:51:29.207330+0000","last_active":"2024-04-02T07:51:29.722128+0000","last_peered":"2024-04-02T07:51:29.722128+0000","last_clean":"2024-04-02T07:51:29.722128+0000","last_became_active":"2024-04-02T07:51:29.207056+0000","last_became_peered":"2024-04-02T07:51:29.207056+0000","last_unstale":"2024-04-02T07:51:29.722128+0000","last_undegraded":"2024-04-02T07:51:29.722128+0000","last_fullsized":"2024-04-02T07:51:29.722128+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:06:43.816854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103269+0000","last_change":"2024-04-02T07:51:29.703907+0000","last_active":"2024-04-02T07:51:31.103269+0000","last_peered":"2024-04-02T07:51:31.103269+0000","last_clean":"2024-04-02T07:51:31.103269+0000","last_became_active":"2024-04-02T07:51:29.703653+0000","last_became_peered":"2024-04-02T07:51:29.703653+0000","last_unstale":"2024-04-02T07:51:31.103269+0000","last_undegraded":"2024-04-02T07:51:31.103269+0000","last_fullsized":"2024-04-02T07:51:31.103269+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:08.732683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696597+0000","last_change":"2024-04-02T07:51:29.206142+0000","last_active":"2024-04-02T07:51:30.696597+0000","last_peered":"2024-04-02T07:51:30.696597+0000","last_clean":"2024-04-02T07:51:30.696597+0000","last_became_active":"2024-04-02T07:51:29.205364+0000","last_became_peered":"2024-04-02T07:51:29.205364+0000","last_unstale":"2024-04-02T07:51:30.696597+0000","last_undegraded":"2024-04-02T07:51:30.696597+0000","last_fullsized":"2024-04-02T07:51:30.696597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:59.149104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.715019+0000","last_change":"2024-04-02T07:51:29.710447+0000","last_active":"2024-04-02T07:51:29.715019+0000","last_peered":"2024-04-02T07:51:29.715019+0000","last_clean":"2024-04-02T07:51:29.715019+0000","last_became_active":"2024-04-02T07:51:29.709373+0000","last_became_peered":"2024-04-02T07:51:29.709373+0000","last_unstale":"2024-04-02T07:51:29.715019+0000","last_undegraded":"2024-04-02T07:51:29.715019+0000","last_fullsized":"2024-04-02T07:51:29.715019+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:31:29.187096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696695+0000","last_change":"2024-04-02T07:51:29.203986+0000","last_active":"2024-04-02T07:51:30.696695+0000","last_peered":"2024-04-02T07:51:30.696695+0000","last_clean":"2024-04-02T07:51:30.696695+0000","last_became_active":"2024-04-02T07:51:29.203312+0000","last_became_peered":"2024-04-02T07:51:29.203312+0000","last_unstale":"2024-04-02T07:51:30.696695+0000","last_undegraded":"2024-04-02T07:51:30.696695+0000","last_fullsized":"2024-04-02T07:51:30.696695+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:02:40.167758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696508+0000","last_change":"2024-04-02T07:51:29.718710+0000","last_active":"2024-04-02T07:51:30.696508+0000","last_peered":"2024-04-02T07:51:30.696508+0000","last_clean":"2024-04-02T07:51:30.696508+0000","last_became_active":"2024-04-02T07:51:29.718415+0000","last_became_peered":"2024-04-02T07:51:29.718415+0000","last_unstale":"2024-04-02T07:51:30.696508+0000","last_undegraded":"2024-04-02T07:51:30.696508+0000","last_fullsized":"2024-04-02T07:51:30.696508+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:36:52.283458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"82'8","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696424+0000","last_change":"2024-04-02T07:51:29.205842+0000","last_active":"2024-04-02T07:51:30.696424+0000","last_peered":"2024-04-02T07:51:30.696424+0000","last_clean":"2024-04-02T07:51:30.696424+0000","last_became_active":"2024-04-02T07:51:29.205501+0000","last_became_peered":"2024-04-02T07:51:29.205501+0000","last_unstale":"2024-04-02T07:51:30.696424+0000","last_undegraded":"2024-04-02T07:51:30.696424+0000","last_fullsized":"2024-04-02T07:51:30.696424+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:25.501441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103460+0000","last_change":"2024-04-02T07:51:29.705712+0000","last_active":"2024-04-02T07:51:31.103460+0000","last_peered":"2024-04-02T07:51:31.103460+0000","last_clean":"2024-04-02T07:51:31.103460+0000","last_became_active":"2024-04-02T07:51:29.704265+0000","last_became_peered":"2024-04-02T07:51:29.704265+0000","last_unstale":"2024-04-02T07:51:31.103460+0000","last_undegraded":"2024-04-02T07:51:31.103460+0000","last_fullsized":"2024-04-02T07:51:31.103460+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:08:52.637936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"82'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702751+0000","last_change":"2024-04-02T07:51:29.211451+0000","last_active":"2024-04-02T07:51:30.702751+0000","last_peered":"2024-04-02T07:51:30.702751+0000","last_clean":"2024-04-02T07:51:30.702751+0000","last_became_active":"2024-04-02T07:51:29.211207+0000","last_became_peered":"2024-04-02T07:51:29.211207+0000","last_unstale":"2024-04-02T07:51:30.702751+0000","last_undegraded":"2024-04-02T07:51:30.702751+0000","last_fullsized":"2024-04-02T07:51:30.702751+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:32:45.599873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103514+0000","last_change":"2024-04-02T07:51:29.709572+0000","last_active":"2024-04-02T07:51:31.103514+0000","last_peered":"2024-04-02T07:51:31.103514+0000","last_clean":"2024-04-02T07:51:31.103514+0000","last_became_active":"2024-04-02T07:51:29.707369+0000","last_became_peered":"2024-04-02T07:51:29.707369+0000","last_unstale":"2024-04-02T07:51:31.103514+0000","last_undegraded":"2024-04-02T07:51:31.103514+0000","last_fullsized":"2024-04-02T07:51:31.103514+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:29:54.419105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"82'2","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.721380+0000","last_change":"2024-04-02T07:51:29.209179+0000","last_active":"2024-04-02T07:51:29.721380+0000","last_peered":"2024-04-02T07:51:29.721380+0000","last_clean":"2024-04-02T07:51:29.721380+0000","last_became_active":"2024-04-02T07:51:29.207991+0000","last_became_peered":"2024-04-02T07:51:29.207991+0000","last_unstale":"2024-04-02T07:51:29.721380+0000","last_undegraded":"2024-04-02T07:51:29.721380+0000","last_fullsized":"2024-04-02T07:51:29.721380+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:04:56.944982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702144+0000","last_change":"2024-04-02T07:51:29.711748+0000","last_active":"2024-04-02T07:51:30.702144+0000","last_peered":"2024-04-02T07:51:30.702144+0000","last_clean":"2024-04-02T07:51:30.702144+0000","last_became_active":"2024-04-02T07:51:29.711452+0000","last_became_peered":"2024-04-02T07:51:29.711452+0000","last_unstale":"2024-04-02T07:51:30.702144+0000","last_undegraded":"2024-04-02T07:51:30.702144+0000","last_fullsized":"2024-04-02T07:51:30.702144+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:58:01.264306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"82'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103458+0000","last_change":"2024-04-02T07:51:29.206737+0000","last_active":"2024-04-02T07:51:31.103458+0000","last_peered":"2024-04-02T07:51:31.103458+0000","last_clean":"2024-04-02T07:51:31.103458+0000","last_became_active":"2024-04-02T07:51:29.206484+0000","last_became_peered":"2024-04-02T07:51:29.206484+0000","last_unstale":"2024-04-02T07:51:31.103458+0000","last_undegraded":"2024-04-02T07:51:31.103458+0000","last_fullsized":"2024-04-02T07:51:31.103458+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:32:45.599873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696861+0000","last_change":"2024-04-02T07:51:29.717733+0000","last_active":"2024-04-02T07:51:30.696861+0000","last_peered":"2024-04-02T07:51:30.696861+0000","last_clean":"2024-04-02T07:51:30.696861+0000","last_became_active":"2024-04-02T07:51:29.717450+0000","last_became_peered":"2024-04-02T07:51:29.717450+0000","last_unstale":"2024-04-02T07:51:30.696861+0000","last_undegraded":"2024-04-02T07:51:30.696861+0000","last_fullsized":"2024-04-02T07:51:30.696861+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:26:51.319846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.696788+0000","last_change":"2024-04-02T07:51:29.205966+0000","last_active":"2024-04-02T07:51:29.696788+0000","last_peered":"2024-04-02T07:51:29.696788+0000","last_clean":"2024-04-02T07:51:29.696788+0000","last_became_active":"2024-04-02T07:51:29.205052+0000","last_became_peered":"2024-04-02T07:51:29.205052+0000","last_unstale":"2024-04-02T07:51:29.696788+0000","last_undegraded":"2024-04-02T07:51:29.696788+0000","last_fullsized":"2024-04-02T07:51:29.696788+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:27.390005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.722338+0000","last_change":"2024-04-02T07:51:29.719782+0000","last_active":"2024-04-02T07:51:29.722338+0000","last_peered":"2024-04-02T07:51:29.722338+0000","last_clean":"2024-04-02T07:51:29.722338+0000","last_became_active":"2024-04-02T07:51:29.719509+0000","last_became_peered":"2024-04-02T07:51:29.719509+0000","last_unstale":"2024-04-02T07:51:29.722338+0000","last_undegraded":"2024-04-02T07:51:29.722338+0000","last_fullsized":"2024-04-02T07:51:29.722338+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:59:28.338083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"82'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.701720+0000","last_change":"2024-04-02T07:51:29.204446+0000","last_active":"2024-04-02T07:51:30.701720+0000","last_peered":"2024-04-02T07:51:30.701720+0000","last_clean":"2024-04-02T07:51:30.701720+0000","last_became_active":"2024-04-02T07:51:29.203629+0000","last_became_peered":"2024-04-02T07:51:29.203629+0000","last_unstale":"2024-04-02T07:51:30.701720+0000","last_undegraded":"2024-04-02T07:51:30.701720+0000","last_fullsized":"2024-04-02T07:51:30.701720+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:07.724301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.714764+0000","last_change":"2024-04-02T07:51:29.707646+0000","last_active":"2024-04-02T07:51:29.714764+0000","last_peered":"2024-04-02T07:51:29.714764+0000","last_clean":"2024-04-02T07:51:29.714764+0000","last_became_active":"2024-04-02T07:51:29.706390+0000","last_became_peered":"2024-04-02T07:51:29.706390+0000","last_unstale":"2024-04-02T07:51:29.714764+0000","last_undegraded":"2024-04-02T07:51:29.714764+0000","last_fullsized":"2024-04-02T07:51:29.714764+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:40:37.534515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.695743+0000","last_change":"2024-04-02T07:51:29.208807+0000","last_active":"2024-04-02T07:51:29.695743+0000","last_peered":"2024-04-02T07:51:29.695743+0000","last_clean":"2024-04-02T07:51:29.695743+0000","last_became_active":"2024-04-02T07:51:29.208501+0000","last_became_peered":"2024-04-02T07:51:29.208501+0000","last_unstale":"2024-04-02T07:51:29.695743+0000","last_undegraded":"2024-04-02T07:51:29.695743+0000","last_fullsized":"2024-04-02T07:51:29.695743+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:56:24.454573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.706743+0000","last_change":"2024-04-02T07:51:29.704031+0000","last_active":"2024-04-02T07:51:29.706743+0000","last_peered":"2024-04-02T07:51:29.706743+0000","last_clean":"2024-04-02T07:51:29.706743+0000","last_became_active":"2024-04-02T07:51:29.703743+0000","last_became_peered":"2024-04-02T07:51:29.703743+0000","last_unstale":"2024-04-02T07:51:29.706743+0000","last_undegraded":"2024-04-02T07:51:29.706743+0000","last_fullsized":"2024-04-02T07:51:29.706743+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:36:52.283458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.698123+0000","last_change":"2024-04-02T07:51:29.209527+0000","last_active":"2024-04-02T07:51:29.698123+0000","last_peered":"2024-04-02T07:51:29.698123+0000","last_clean":"2024-04-02T07:51:29.698123+0000","last_became_active":"2024-04-02T07:51:29.207961+0000","last_became_peered":"2024-04-02T07:51:29.207961+0000","last_unstale":"2024-04-02T07:51:29.698123+0000","last_undegraded":"2024-04-02T07:51:29.698123+0000","last_fullsized":"2024-04-02T07:51:29.698123+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:10:14.951279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103489+0000","last_change":"2024-04-02T07:51:29.717488+0000","last_active":"2024-04-02T07:51:31.103489+0000","last_peered":"2024-04-02T07:51:31.103489+0000","last_clean":"2024-04-02T07:51:31.103489+0000","last_became_active":"2024-04-02T07:51:29.717204+0000","last_became_peered":"2024-04-02T07:51:29.717204+0000","last_unstale":"2024-04-02T07:51:31.103489+0000","last_undegraded":"2024-04-02T07:51:31.103489+0000","last_fullsized":"2024-04-02T07:51:31.103489+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:59:28.338083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"82'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702742+0000","last_change":"2024-04-02T07:51:29.210933+0000","last_active":"2024-04-02T07:51:30.702742+0000","last_peered":"2024-04-02T07:51:30.702742+0000","last_clean":"2024-04-02T07:51:30.702742+0000","last_became_active":"2024-04-02T07:51:29.210685+0000","last_became_peered":"2024-04-02T07:51:29.210685+0000","last_unstale":"2024-04-02T07:51:30.702742+0000","last_undegraded":"2024-04-02T07:51:30.702742+0000","last_fullsized":"2024-04-02T07:51:30.702742+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:10.744141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103549+0000","last_change":"2024-04-02T07:51:29.706642+0000","last_active":"2024-04-02T07:51:31.103549+0000","last_peered":"2024-04-02T07:51:31.103549+0000","last_clean":"2024-04-02T07:51:31.103549+0000","last_became_active":"2024-04-02T07:51:29.705512+0000","last_became_peered":"2024-04-02T07:51:29.705512+0000","last_unstale":"2024-04-02T07:51:31.103549+0000","last_undegraded":"2024-04-02T07:51:31.103549+0000","last_fullsized":"2024-04-02T07:51:31.103549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:18:41.743679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.693035+0000","last_change":"2024-04-02T07:51:29.212027+0000","last_active":"2024-04-02T07:51:29.693035+0000","last_peered":"2024-04-02T07:51:29.693035+0000","last_clean":"2024-04-02T07:51:29.693035+0000","last_became_active":"2024-04-02T07:51:29.211739+0000","last_became_peered":"2024-04-02T07:51:29.211739+0000","last_unstale":"2024-04-02T07:51:29.693035+0000","last_undegraded":"2024-04-02T07:51:29.693035+0000","last_fullsized":"2024-04-02T07:51:29.693035+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:45:44.289973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702411+0000","last_change":"2024-04-02T07:51:29.718952+0000","last_active":"2024-04-02T07:51:30.702411+0000","last_peered":"2024-04-02T07:51:30.702411+0000","last_clean":"2024-04-02T07:51:30.702411+0000","last_became_active":"2024-04-02T07:51:29.714973+0000","last_became_peered":"2024-04-02T07:51:29.714973+0000","last_unstale":"2024-04-02T07:51:30.702411+0000","last_undegraded":"2024-04-02T07:51:30.702411+0000","last_fullsized":"2024-04-02T07:51:30.702411+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:08.732683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.697582+0000","last_change":"2024-04-02T07:51:29.201725+0000","last_active":"2024-04-02T07:51:29.697582+0000","last_peered":"2024-04-02T07:51:29.697582+0000","last_clean":"2024-04-02T07:51:29.697582+0000","last_became_active":"2024-04-02T07:51:29.201438+0000","last_became_peered":"2024-04-02T07:51:29.201438+0000","last_unstale":"2024-04-02T07:51:29.697582+0000","last_undegraded":"2024-04-02T07:51:29.697582+0000","last_fullsized":"2024-04-02T07:51:29.697582+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:03.852318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.707292+0000","last_change":"2024-04-02T07:51:29.704125+0000","last_active":"2024-04-02T07:51:29.707292+0000","last_peered":"2024-04-02T07:51:29.707292+0000","last_clean":"2024-04-02T07:51:29.707292+0000","last_became_active":"2024-04-02T07:51:29.703842+0000","last_became_peered":"2024-04-02T07:51:29.703842+0000","last_unstale":"2024-04-02T07:51:29.707292+0000","last_undegraded":"2024-04-02T07:51:29.707292+0000","last_fullsized":"2024-04-02T07:51:29.707292+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:24:50.492280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103273+0000","last_change":"2024-04-02T07:51:29.205685+0000","last_active":"2024-04-02T07:51:31.103273+0000","last_peered":"2024-04-02T07:51:31.103273+0000","last_clean":"2024-04-02T07:51:31.103273+0000","last_became_active":"2024-04-02T07:51:29.204332+0000","last_became_peered":"2024-04-02T07:51:29.204332+0000","last_unstale":"2024-04-02T07:51:31.103273+0000","last_undegraded":"2024-04-02T07:51:31.103273+0000","last_fullsized":"2024-04-02T07:51:31.103273+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:10.744141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.717861+0000","last_change":"2024-04-02T07:51:29.712341+0000","last_active":"2024-04-02T07:51:29.717861+0000","last_peered":"2024-04-02T07:51:29.717861+0000","last_clean":"2024-04-02T07:51:29.717861+0000","last_became_active":"2024-04-02T07:51:29.710914+0000","last_became_peered":"2024-04-02T07:51:29.710914+0000","last_unstale":"2024-04-02T07:51:29.717861+0000","last_undegraded":"2024-04-02T07:51:29.717861+0000","last_fullsized":"2024-04-02T07:51:29.717861+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:02:53.831136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702244+0000","last_change":"2024-04-02T07:51:29.210518+0000","last_active":"2024-04-02T07:51:30.702244+0000","last_peered":"2024-04-02T07:51:30.702244+0000","last_clean":"2024-04-02T07:51:30.702244+0000","last_became_active":"2024-04-02T07:51:29.210225+0000","last_became_peered":"2024-04-02T07:51:29.210225+0000","last_unstale":"2024-04-02T07:51:30.702244+0000","last_undegraded":"2024-04-02T07:51:30.702244+0000","last_fullsized":"2024-04-02T07:51:30.702244+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:53:15.044497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.711523+0000","last_change":"2024-04-02T07:51:29.702103+0000","last_active":"2024-04-02T07:51:29.711523+0000","last_peered":"2024-04-02T07:51:29.711523+0000","last_clean":"2024-04-02T07:51:29.711523+0000","last_became_active":"2024-04-02T07:51:29.701851+0000","last_became_peered":"2024-04-02T07:51:29.701851+0000","last_unstale":"2024-04-02T07:51:29.711523+0000","last_undegraded":"2024-04-02T07:51:29.711523+0000","last_fullsized":"2024-04-02T07:51:29.711523+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:20.749913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103176+0000","last_change":"2024-04-02T07:51:29.208725+0000","last_active":"2024-04-02T07:51:31.103176+0000","last_peered":"2024-04-02T07:51:31.103176+0000","last_clean":"2024-04-02T07:51:31.103176+0000","last_became_active":"2024-04-02T07:51:29.208140+0000","last_became_peered":"2024-04-02T07:51:29.208140+0000","last_unstale":"2024-04-02T07:51:31.103176+0000","last_undegraded":"2024-04-02T07:51:31.103176+0000","last_fullsized":"2024-04-02T07:51:31.103176+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:10:14.951279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.714024+0000","last_change":"2024-04-02T07:51:29.707370+0000","last_active":"2024-04-02T07:51:29.714024+0000","last_peered":"2024-04-02T07:51:29.714024+0000","last_clean":"2024-04-02T07:51:29.714024+0000","last_became_active":"2024-04-02T07:51:29.706978+0000","last_became_peered":"2024-04-02T07:51:29.706978+0000","last_unstale":"2024-04-02T07:51:29.714024+0000","last_undegraded":"2024-04-02T07:51:29.714024+0000","last_fullsized":"2024-04-02T07:51:29.714024+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:02:41.176141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.692278+0000","last_change":"2024-04-02T07:51:29.204430+0000","last_active":"2024-04-02T07:51:29.692278+0000","last_peered":"2024-04-02T07:51:29.692278+0000","last_clean":"2024-04-02T07:51:29.692278+0000","last_became_active":"2024-04-02T07:51:29.204162+0000","last_became_peered":"2024-04-02T07:51:29.204162+0000","last_unstale":"2024-04-02T07:51:29.692278+0000","last_undegraded":"2024-04-02T07:51:29.692278+0000","last_fullsized":"2024-04-02T07:51:29.692278+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:17:59.216495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702451+0000","last_change":"2024-04-02T07:51:29.710211+0000","last_active":"2024-04-02T07:51:30.702451+0000","last_peered":"2024-04-02T07:51:30.702451+0000","last_clean":"2024-04-02T07:51:30.702451+0000","last_became_active":"2024-04-02T07:51:29.709902+0000","last_became_peered":"2024-04-02T07:51:29.709902+0000","last_unstale":"2024-04-02T07:51:30.702451+0000","last_undegraded":"2024-04-02T07:51:30.702451+0000","last_fullsized":"2024-04-02T07:51:30.702451+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:21:07.376954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.696398+0000","last_change":"2024-04-02T07:51:29.206364+0000","last_active":"2024-04-02T07:51:29.696398+0000","last_peered":"2024-04-02T07:51:29.696398+0000","last_clean":"2024-04-02T07:51:29.696398+0000","last_became_active":"2024-04-02T07:51:29.206071+0000","last_became_peered":"2024-04-02T07:51:29.206071+0000","last_unstale":"2024-04-02T07:51:29.696398+0000","last_undegraded":"2024-04-02T07:51:29.696398+0000","last_fullsized":"2024-04-02T07:51:29.696398+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:59:00.134245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.707423+0000","last_change":"2024-04-02T07:51:29.703939+0000","last_active":"2024-04-02T07:51:29.707423+0000","last_peered":"2024-04-02T07:51:29.707423+0000","last_clean":"2024-04-02T07:51:29.707423+0000","last_became_active":"2024-04-02T07:51:29.703313+0000","last_became_peered":"2024-04-02T07:51:29.703313+0000","last_unstale":"2024-04-02T07:51:29.707423+0000","last_undegraded":"2024-04-02T07:51:29.707423+0000","last_fullsized":"2024-04-02T07:51:29.707423+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:19:28.826550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.689168+0000","last_change":"2024-04-02T07:51:29.211829+0000","last_active":"2024-04-02T07:51:29.689168+0000","last_peered":"2024-04-02T07:51:29.689168+0000","last_clean":"2024-04-02T07:51:29.689168+0000","last_became_active":"2024-04-02T07:51:29.211549+0000","last_became_peered":"2024-04-02T07:51:29.211549+0000","last_unstale":"2024-04-02T07:51:29.689168+0000","last_undegraded":"2024-04-02T07:51:29.689168+0000","last_fullsized":"2024-04-02T07:51:29.689168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:04:56.944982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"72'161","reported_seq":381,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.190340+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:51:29.190340+0000","last_peered":"2024-04-02T07:51:29.190340+0000","last_clean":"2024-04-02T07:51:29.190340+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:51:29.190340+0000","last_undegraded":"2024-04-02T07:51:29.190340+0000","last_fullsized":"2024-04-02T07:51:29.190340+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575037,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":999424,"data_stored":478768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66800000000000004,"15min":0.67800000000000005},"min":{"1min":0.39400000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.67800000000000005},"min":{"1min":0.47399999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.95699999999999996,"5min":1.0109999999999999,"15min":1.024},"last":0.68300000000000005}]},{"osd":1,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.67300000000000004,"15min":0.68000000000000005},"min":{"1min":0.499,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.89100000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66500000000000004,"15min":0.67500000000000004},"min":{"1min":0.42399999999999999,"5min":0.35599999999999998,"15min":0.34699999999999998},"max":{"1min":0.88300000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.71699999999999997,"15min":0.71099999999999997},"min":{"1min":0.50700000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.94199999999999995,"5min":1.127,"15min":1.127},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.753,"15min":0.76000000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.434},"max":{"1min":0.93799999999999994,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.64200000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.69599999999999995,"15min":0.70399999999999996},"min":{"1min":0.44,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.97799999999999998,"5min":1.026,"15min":1.274},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.52600000000000002,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.90600000000000003,"5min":1.196,"15min":1.196},"last":0.58899999999999997}]},{"osd":4,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.376,"5min":0.32000000000000001,"15min":0.30199999999999999},"max":{"1min":0.84099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68200000000000005,"15min":0.68700000000000006},"min":{"1min":0.51800000000000002,"5min":0.40000000000000002,"15min":0.38100000000000001},"max":{"1min":0.84599999999999997,"5min":1.0669999999999999,"15min":1.4890000000000001},"last":0.76100000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.46100000000000002,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":0.86299999999999999,"5min":1.167,"15min":1.167},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68500000000000005,"15min":0.68700000000000006},"min":{"1min":0.51100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.69799999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67600000000000005,"15min":0.65800000000000003},"min":{"1min":0.505,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.877,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66000000000000003,"15min":0.65300000000000002},"min":{"1min":0.47999999999999998,"5min":0.42299999999999999,"15min":0.39600000000000002},"max":{"1min":0.89300000000000002,"5min":1.0509999999999999,"15min":1.329},"last":0.66500000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":995328,"data_stored":478698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.51300000000000001,"5min":0.38700000000000001,"15min":0.32400000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.98099999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.749,"5min":0.71599999999999997,"15min":0.70899999999999996},"min":{"1min":0.55200000000000005,"5min":0.38300000000000001,"15min":0.375},"max":{"1min":0.89400000000000002,"5min":0.96899999999999997,"15min":1.0249999999999999},"last":0.56799999999999995}]},{"osd":1,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69399999999999995,"15min":0.69599999999999995},"min":{"1min":0.53900000000000003,"5min":0.46500000000000002,"15min":0.41199999999999998},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":1.085},"last":0.745},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.56999999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.84399999999999997}]},{"osd":2,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73099999999999998,"15min":0.71499999999999997},"min":{"1min":0.48999999999999999,"5min":0.45500000000000002,"15min":0.38200000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71799999999999997,"15min":0.70699999999999996},"min":{"1min":0.53300000000000003,"5min":0.51400000000000001,"15min":0.45900000000000002},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96599999999999997},"last":0.82499999999999996}]},{"osd":3,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.73699999999999999,"15min":0.73899999999999999},"min":{"1min":0.55400000000000005,"5min":0.5,"15min":0.47099999999999997},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.748,"15min":0.73999999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.45300000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.0309999999999999},"last":0.80600000000000005}]},{"osd":4,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61399999999999999,"15min":0.60499999999999998},"min":{"1min":0.39300000000000002,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.64000000000000001,"15min":0.629},"min":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.35499999999999998},"max":{"1min":0.93200000000000005,"5min":0.99299999999999999,"15min":1.0049999999999999},"last":0.86699999999999999}]},{"osd":5,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.629,"15min":0.621},"min":{"1min":0.41599999999999998,"5min":0.379,"15min":0.35799999999999998},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":1.1599999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.67200000000000004,"15min":0.68600000000000005},"min":{"1min":0.504,"5min":0.42099999999999999,"15min":0.40300000000000002},"max":{"1min":0.93899999999999995,"5min":1.01,"15min":1.095},"last":0.64100000000000001}]},{"osd":7,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.68899999999999995,"15min":0.68600000000000005},"min":{"1min":0.48099999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.96199999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.45200000000000001,"5min":0.44,"15min":0.42099999999999999},"max":{"1min":0.92500000000000004,"5min":1.0149999999999999,"15min":1.4399999999999999},"last":0.60799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575036,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36768,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335072,"statfs":{"total":96636764160,"available":96599113728,"internally_reserved":0,"allocated":1589248,"data_stored":1069626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.432,"15min":0.44},"min":{"1min":0.29499999999999998,"5min":0.27500000000000002,"15min":0.25700000000000001},"max":{"1min":0.58899999999999997,"5min":0.77600000000000002,"15min":2.9079999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.50700000000000001,"15min":0.504},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.70699999999999996,"5min":0.86499999999999999,"15min":2.766},"last":0.46700000000000003}]},{"osd":2,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.627,"5min":0.71599999999999997,"15min":2.0750000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.443,"5min":0.36699999999999999,"15min":0.35699999999999998},"max":{"1min":0.71499999999999997,"5min":0.85399999999999998,"15min":1.7410000000000001},"last":0.51200000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.41799999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72399999999999998,"5min":0.81799999999999995,"15min":2.9430000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68000000000000005,"5min":0.82699999999999996,"15min":2.802},"last":0.52000000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.78300000000000003,"15min":0.77400000000000002},"min":{"1min":0.67400000000000004,"5min":0.46700000000000003,"15min":0.46300000000000002},"max":{"1min":1.0609999999999999,"5min":1.861,"15min":1.861},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.77700000000000002,"15min":0.76800000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46500000000000002},"max":{"1min":1.232,"5min":1.232,"15min":2.0419999999999998},"last":0.64100000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.77900000000000003,"15min":0.77100000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.41999999999999998},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.9690000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79400000000000004,"15min":0.78300000000000003},"min":{"1min":0.65600000000000003,"5min":0.51000000000000001,"15min":0.48799999999999999},"max":{"1min":1.147,"5min":1.54,"15min":1.54},"last":0.71699999999999997}]},{"osd":6,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.79900000000000004,"15min":0.79500000000000004},"min":{"1min":0.53200000000000003,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.097,"5min":1.577,"15min":1.768},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.80400000000000005,"15min":0.79100000000000004},"min":{"1min":0.61399999999999999,"5min":0.51400000000000001,"15min":0.38100000000000001},"max":{"1min":1.351,"5min":1.351,"15min":2.012},"last":0.77800000000000002}]},{"osd":7,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.80900000000000005,"15min":0.80300000000000005},"min":{"1min":0.628,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.298,"5min":1.552,"15min":1.706},"last":0.62},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.64300000000000002,"5min":0.53200000000000003,"15min":0.502},"max":{"1min":1.194,"5min":1.2290000000000001,"15min":1.8240000000000001},"last":0.871}]}]},{"osd":0,"up_from":13,"seq":55834575036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":999424,"data_stored":479221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.441,"15min":0.44400000000000001},"min":{"1min":0.29699999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71199999999999997,"5min":0.78900000000000003,"15min":2.1640000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.495,"15min":0.496},"min":{"1min":0.34899999999999998,"5min":0.307,"15min":0.26200000000000001},"max":{"1min":0.70599999999999996,"5min":0.72199999999999998,"15min":2.6469999999999998},"last":0.39800000000000002}]},{"osd":2,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48999999999999999,"15min":0.495},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63100000000000001,"5min":0.749,"15min":2.125},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":2.605},"last":0.52300000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52900000000000003,"15min":0.54800000000000004},"min":{"1min":0.33600000000000002,"5min":0.32700000000000001,"15min":0.28899999999999998},"max":{"1min":0.69799999999999995,"5min":0.79400000000000004,"15min":5.4800000000000004},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.40100000000000002},"max":{"1min":0.83199999999999996,"5min":0.93400000000000005,"15min":2.194},"last":0.65400000000000003}]},{"osd":4,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.77700000000000002,"15min":0.80100000000000005},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.35599999999999998},"max":{"1min":0.97999999999999998,"5min":1.0640000000000001,"15min":6.0899999999999999},"last":0.623},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78400000000000003,"15min":0.79600000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.433},"max":{"1min":1.048,"5min":1.095,"15min":5.7169999999999996},"last":0.70499999999999996}]},{"osd":5,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.81100000000000005,"15min":0.81200000000000006},"min":{"1min":0.55800000000000005,"5min":0.51000000000000001,"15min":0.42099999999999999},"max":{"1min":0.98999999999999999,"5min":2.1869999999999998,"15min":5.7539999999999996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.81799999999999995},"min":{"1min":0.63100000000000001,"5min":0.54100000000000004,"15min":0.51800000000000002},"max":{"1min":0.96299999999999997,"5min":2.6230000000000002,"15min":5.8929999999999998},"last":0.72099999999999997}]},{"osd":6,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.75700000000000001,"15min":0.77100000000000002},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.46800000000000003},"max":{"1min":0.96599999999999997,"5min":1.044,"15min":5.931},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.755,"15min":0.77700000000000002},"min":{"1min":0.64700000000000002,"5min":0.50700000000000001,"15min":0.376},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":6.125},"last":0.68899999999999995}]},{"osd":7,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.83799999999999997},"min":{"1min":0.70299999999999996,"5min":0.51600000000000001,"15min":0.40799999999999997},"max":{"1min":1.0900000000000001,"5min":2.3959999999999999,"15min":6.1580000000000004},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.81399999999999995,"15min":0.83199999999999996},"min":{"1min":0.54100000000000004,"5min":0.48299999999999998,"15min":0.47699999999999998},"max":{"1min":1.0249999999999999,"5min":2.145,"15min":5.7850000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575035,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":1011712,"data_stored":480400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.441,"15min":0.44700000000000001},"min":{"1min":0.27400000000000002,"5min":0.26400000000000001,"15min":0.25700000000000001},"max":{"1min":0.71099999999999997,"5min":0.88,"15min":0.88},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46300000000000002,"15min":0.47399999999999998},"min":{"1min":0.33800000000000002,"5min":0.27900000000000003,"15min":0.26200000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.84299999999999997},"last":0.41299999999999998}]},{"osd":1,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.49399999999999999},"min":{"1min":0.32700000000000001,"5min":0.30599999999999999,"15min":0.30299999999999999},"max":{"1min":0.81899999999999995,"5min":0.85699999999999998,"15min":0.93999999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51800000000000002,"15min":0.52300000000000002},"min":{"1min":0.40799999999999997,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.90000000000000002},"last":0.45900000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.52600000000000002,"15min":0.53800000000000003},"min":{"1min":0.38500000000000001,"5min":0.316,"15min":0.27800000000000002},"max":{"1min":0.79000000000000004,"5min":0.79800000000000004,"15min":1.4339999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.60499999999999998,"15min":0.61199999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39000000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":1.3740000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.76600000000000001,"15min":0.76900000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.47199999999999998},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.157},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.755,"15min":0.77000000000000002},"min":{"1min":0.56399999999999995,"5min":0.51600000000000001,"15min":0.48499999999999999},"max":{"1min":0.999,"5min":1.075,"15min":1.179},"last":0.76100000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.754,"15min":0.75800000000000001},"min":{"1min":0.57799999999999996,"5min":0.54300000000000004,"15min":0.42199999999999999},"max":{"1min":1.004,"5min":1.0229999999999999,"15min":1.2},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.76700000000000002,"15min":0.78000000000000003},"min":{"1min":0.59599999999999997,"5min":0.53800000000000003,"15min":0.50900000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1899999999999999},"last":0.73199999999999998}]},{"osd":6,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.78700000000000003,"15min":0.78600000000000003},"min":{"1min":0.60399999999999998,"5min":0.53600000000000003,"15min":0.442},"max":{"1min":1.0149999999999999,"5min":1.0880000000000001,"15min":1.175},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.76300000000000001,"15min":0.78700000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.018,"5min":1.0620000000000001,"15min":1.2130000000000001},"last":0.79300000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78200000000000003,"15min":0.79500000000000004},"min":{"1min":0.57299999999999995,"5min":0.55900000000000005,"15min":0.5},"max":{"1min":1.0469999999999999,"5min":1.054,"15min":1.1779999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.79400000000000004},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.52900000000000003},"max":{"1min":1.022,"5min":1.0920000000000001,"15min":1.145},"last":0.82099999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36192,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335648,"statfs":{"total":96636764160,"available":96599703552,"internally_reserved":0,"allocated":995328,"data_stored":478766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.42699999999999999,"15min":0.438},"min":{"1min":0.32000000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.748,"5min":0.748,"15min":2.476},"last":0.314},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44700000000000001,"15min":0.47899999999999998},"min":{"1min":0.34499999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.61099999999999999,"5min":0.70799999999999996,"15min":2.04},"last":0.373}]},{"osd":1,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47999999999999998,"15min":0.497},"min":{"1min":0.32700000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.61599999999999999,"5min":1.2809999999999999,"15min":2.2839999999999998},"last":0.438},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55700000000000005,"15min":0.56399999999999995},"min":{"1min":0.45100000000000001,"5min":0.35799999999999998,"15min":0.34200000000000003},"max":{"1min":0.77500000000000002,"5min":0.86299999999999999,"15min":2.2240000000000002},"last":0.49199999999999999}]},{"osd":2,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.60299999999999998,"15min":0.61299999999999999},"min":{"1min":0.49299999999999999,"5min":0.38700000000000001,"15min":0.376},"max":{"1min":0.84099999999999997,"5min":0.92100000000000004,"15min":1.7310000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58899999999999997,"15min":0.62},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.35699999999999998},"max":{"1min":0.87,"5min":1.244,"15min":3.9449999999999998},"last":0.61899999999999999}]},{"osd":4,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.72199999999999998,"15min":0.74299999999999999},"min":{"1min":0.499,"5min":0.436,"15min":0.38400000000000001},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":2.0030000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.73999999999999999,"15min":0.75900000000000001},"min":{"1min":0.52500000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89800000000000002,"5min":1.02,"15min":1.5249999999999999},"last":0.96499999999999997}]},{"osd":5,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.73499999999999999,"15min":0.754},"min":{"1min":0.51700000000000002,"5min":0.41699999999999998,"15min":0.40899999999999997},"max":{"1min":0.91800000000000004,"5min":1.079,"15min":1.2909999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.72699999999999998,"15min":0.75800000000000001},"min":{"1min":0.55100000000000005,"5min":0.48899999999999999,"15min":0.40200000000000002},"max":{"1min":0.95799999999999996,"5min":1.0449999999999999,"15min":1.9159999999999999},"last":0.94199999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.73499999999999999,"15min":0.76400000000000001},"min":{"1min":0.56599999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.95099999999999996,"5min":1.0620000000000001,"15min":1.548},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76200000000000001,"15min":0.77900000000000003},"min":{"1min":0.58499999999999996,"5min":0.54800000000000004,"15min":0.53500000000000003},"max":{"1min":1.0209999999999999,"5min":1.1140000000000001,"15min":1.4379999999999999},"last":0.77900000000000003}]},{"osd":7,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.76100000000000001,"15min":0.78200000000000003},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.439},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.966},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.76200000000000001,"15min":0.78900000000000003},"min":{"1min":0.57199999999999995,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.96899999999999997,"5min":1.181,"15min":1.514},"last":1.0089999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36196,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335644,"statfs":{"total":96636764160,"available":96599699456,"internally_reserved":0,"allocated":999424,"data_stored":479292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67700000000000005,"15min":0.66300000000000003},"min":{"1min":0.436,"5min":0.38100000000000001,"15min":0.32100000000000001},"max":{"1min":0.879,"5min":0.94299999999999995,"15min":1.0189999999999999},"last":0.498},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68300000000000005,"15min":0.67400000000000004},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.35299999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":1.7849999999999999},"last":0.69099999999999995}]},{"osd":1,"last update":"Tue Apr 2 07:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.73199999999999998,"15min":0.72399999999999998},"min":{"1min":0.53400000000000003,"5min":0.438,"15min":0.42699999999999999},"max":{"1min":0.90900000000000003,"5min":0.97299999999999998,"15min":1.036},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72699999999999998,"15min":0.73399999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.95099999999999996,"5min":0.97899999999999998,"15min":0.99399999999999999},"last":0.64700000000000002}]},{"osd":2,"last update":"Tue Apr 2 07:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.68600000000000005,"15min":0.68799999999999994},"min":{"1min":0.51900000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91500000000000004,"5min":0.93500000000000005,"15min":1.03},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.71599999999999997,"15min":0.71499999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41899999999999998},"max":{"1min":0.91600000000000004,"5min":0.97199999999999998,"15min":1.6459999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Tue Apr 2 07:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.72999999999999998,"15min":0.71999999999999997},"min":{"1min":0.53700000000000003,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":0.90900000000000003,"5min":0.95699999999999996,"15min":1.002},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.62,"5min":0.48799999999999999,"15min":0.41499999999999998},"max":{"1min":0.82999999999999996,"5min":0.96399999999999997,"15min":0.96999999999999997},"last":0.58399999999999996}]},{"osd":5,"last update":"Tue Apr 2 07:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56299999999999994,"15min":0.57699999999999996},"min":{"1min":0.38600000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63200000000000001,"5min":0.84799999999999998,"15min":1.034},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.58999999999999997},"min":{"1min":0.41399999999999998,"5min":0.38800000000000001,"15min":0.33700000000000002},"max":{"1min":0.875,"5min":0.89700000000000002,"15min":1.0129999999999999},"last":0.51800000000000002}]},{"osd":6,"last update":"Tue Apr 2 07:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59799999999999998,"15min":0.60799999999999998},"min":{"1min":0.48599999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.83099999999999996,"5min":0.89200000000000002,"15min":1.2589999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.64900000000000002,"15min":0.65300000000000002},"min":{"1min":0.498,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.78200000000000003,"5min":0.93100000000000005,"15min":1.0609999999999999},"last":0.68000000000000005}]},{"osd":7,"last update":"Tue Apr 2 07:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68700000000000006,"15min":0.67200000000000004},"min":{"1min":0.50700000000000001,"5min":0.432,"15min":0.38300000000000001},"max":{"1min":0.92900000000000005,"5min":0.95199999999999996,"15min":1.0680000000000001},"last":0.63},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.745,"15min":0.73399999999999999},"min":{"1min":0.58999999999999997,"5min":0.47799999999999998,"15min":0.434},"max":{"1min":0.88200000000000001,"5min":0.999,"15min":1.502},"last":0.74199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575038,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32672,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":1585152,"data_stored":1068059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67800000000000005,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.80900000000000005,"5min":0.95299999999999996,"15min":0.99199999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.878,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.71899999999999997}]},{"osd":1,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67600000000000005,"15min":0.69399999999999995},"min":{"1min":0.41999999999999998,"5min":0.40300000000000002,"15min":0.40200000000000002},"max":{"1min":1.0620000000000001,"5min":1.131,"15min":1.131},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.45000000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83999999999999997,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.69299999999999995}]},{"osd":2,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70699999999999996,"15min":0.71299999999999997},"min":{"1min":0.47399999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.878,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.69399999999999995,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.47099999999999997,"15min":0.372},"max":{"1min":1.111,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.72899999999999998,"15min":0.74299999999999999},"min":{"1min":0.54800000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92300000000000004,"5min":1.341,"15min":1.341},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77000000000000002,"15min":0.76600000000000001},"min":{"1min":0.60699999999999998,"5min":0.54600000000000004,"15min":0.44900000000000001},"max":{"1min":1.1839999999999999,"5min":1.484,"15min":1.484},"last":0.504}]},{"osd":4,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59299999999999997,"15min":0.59999999999999998},"min":{"1min":0.39500000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.35599999999999998},"max":{"1min":0.96199999999999997,"5min":1.504,"15min":1.504},"last":0.628}]},{"osd":6,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.64900000000000002,"15min":0.64300000000000002},"min":{"1min":0.45600000000000002,"5min":0.44700000000000001,"15min":0.39900000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.69199999999999995,"15min":0.67700000000000005},"min":{"1min":0.45500000000000002,"5min":0.44900000000000001,"15min":0.41199999999999998},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":0.65100000000000002}]},{"osd":7,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.71399999999999997,"15min":0.70299999999999996},"min":{"1min":0.502,"5min":0.47599999999999998,"15min":0.38900000000000001},"max":{"1min":1.0129999999999999,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.754,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.53600000000000003,"5min":0.46300000000000002,"15min":0.434},"max":{"1min":1.149,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.72999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:51:34.080 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:51:34.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.306+0000 7f6bb4e00700 1 Processor -- start 2024-04-02T07:51:34.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.306+0000 7f6bb4e00700 1 -- start start 2024-04-02T07:51:34.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.307+0000 7f6bb4e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb0160f10 0x7f6bb00d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:34.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.307+0000 7f6bb4e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6bb00cf230 0x7f6bb00d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:34.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.307+0000 7f6bb4e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00ceda0 con 0x7f6bb0161590 2024-04-02T07:51:34.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.307+0000 7f6bb4e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb00d23d0 con 0x7f6bb00cf230 2024-04-02T07:51:34.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.307+0000 7f6bb4e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb01612f0 con 0x7f6bb0160f10 2024-04-02T07:51:34.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.308+0000 7f6bad400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6bb00cf230 0x7f6bb00d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:34.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.308+0000 7f6bad400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6bb00cf230 0x7f6bb00d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44678/0 (socket says 172.21.15.89:44678) 2024-04-02T07:51:34.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.308+0000 7f6bad400700 1 -- 172.21.15.89:0/3010255273 learned_addr learned my addr 172.21.15.89:0/3010255273 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:34.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.308+0000 7f6bade00700 1 --2- 172.21.15.89:0/3010255273 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb0160f10 0x7f6bb00d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:34.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.308+0000 7f6bad400700 1 -- 172.21.15.89:0/3010255273 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb0160f10 msgr2=0x7f6bb00d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:34.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.308+0000 7f6bad400700 1 --2- 172.21.15.89:0/3010255273 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb0160f10 0x7f6bb00d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.308+0000 7f6bad400700 1 -- 172.21.15.89:0/3010255273 >> v1:172.21.15.113:6789/0 conn(0x7f6bb0161590 legacy=0x7f6bb00b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:34.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.308+0000 7f6bad400700 1 -- 172.21.15.89:0/3010255273 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb00ce470 con 0x7f6bb00cf230 2024-04-02T07:51:34.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.308+0000 7f6bad400700 1 --2- 172.21.15.89:0/3010255273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb00cf230 0x7f6bb00d1e90 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c0098f0 tx=0x7f6b9c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3dd26a808b7d9d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:34.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.309+0000 7f6baca00700 1 -- 172.21.15.89:0/3010255273 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c004230 con 0x7f6bb00cf230 2024-04-02T07:51:34.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.309+0000 7f6baca00700 1 -- 172.21.15.89:0/3010255273 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9c004390 con 0x7f6bb00cf230 2024-04-02T07:51:34.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.309+0000 7f6baca00700 1 -- 172.21.15.89:0/3010255273 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c0036a0 con 0x7f6bb00cf230 2024-04-02T07:51:34.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.309+0000 7f6bb4e00700 1 -- 172.21.15.89:0/3010255273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb00cf230 msgr2=0x7f6bb00d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:34.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.309+0000 7f6bb4e00700 1 --2- 172.21.15.89:0/3010255273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb00cf230 0x7f6bb00d1e90 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c0098f0 tx=0x7f6b9c009c00 comp rx=0 tx=0).stop 2024-04-02T07:51:34.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.309+0000 7f6bb4e00700 1 -- 172.21.15.89:0/3010255273 shutdown_connections 2024-04-02T07:51:34.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.309+0000 7f6bb4e00700 1 --2- 172.21.15.89:0/3010255273 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb0160f10 0x7f6bb00d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.309+0000 7f6bb4e00700 1 --2- 172.21.15.89:0/3010255273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb00cf230 0x7f6bb00d1e90 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.309+0000 7f6bb4e00700 1 -- 172.21.15.89:0/3010255273 >> 172.21.15.89:0/3010255273 conn(0x7f6bb008aff0 msgr2=0x7f6bb00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:34.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.309+0000 7f6bb4e00700 1 -- 172.21.15.89:0/3010255273 shutdown_connections 2024-04-02T07:51:34.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.309+0000 7f6bb4e00700 1 -- 172.21.15.89:0/3010255273 wait complete. 2024-04-02T07:51:34.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.311+0000 7f6bb4e00700 1 Processor -- start 2024-04-02T07:51:34.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.311+0000 7f6bb4e00700 1 -- start start 2024-04-02T07:51:34.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.312+0000 7f6bb4e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bb00cf230 0x7f6bb01590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:34.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.312+0000 7f6bb4e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb0160f10 0x7f6bb0151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:34.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.312+0000 7f6bb4e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb0161590 0x7f6bb0153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:34.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.312+0000 7f6bb4e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb01568e0 con 0x7f6bb0161590 2024-04-02T07:51:34.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.312+0000 7f6bb4e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb0156bc0 con 0x7f6bb00cf230 2024-04-02T07:51:34.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.312+0000 7f6bb4e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb0156ea0 con 0x7f6bb0160f10 2024-04-02T07:51:34.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.312+0000 7f6bae800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb0161590 0x7f6bb0153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:34.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.312+0000 7f6bae800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb0161590 0x7f6bb0153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44682/0 (socket says 172.21.15.89:44682) 2024-04-02T07:51:34.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.312+0000 7f6bae800700 1 -- 172.21.15.89:0/4151627196 learned_addr learned my addr 172.21.15.89:0/4151627196 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:34.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.312+0000 7f6bade00700 1 --2- 172.21.15.89:0/4151627196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bb00cf230 0x7f6bb01590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:34.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.312+0000 7f6bad400700 1 --2- 172.21.15.89:0/4151627196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb0160f10 0x7f6bb0151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:34.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.313+0000 7f6bae800700 1 -- 172.21.15.89:0/4151627196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb0160f10 msgr2=0x7f6bb0151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:34.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.313+0000 7f6bae800700 1 --2- 172.21.15.89:0/4151627196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb0160f10 0x7f6bb0151d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.313+0000 7f6bae800700 1 -- 172.21.15.89:0/4151627196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bb00cf230 msgr2=0x7f6bb01590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:34.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.313+0000 7f6bae800700 1 --2- 172.21.15.89:0/4151627196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bb00cf230 0x7f6bb01590f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.313+0000 7f6bae800700 1 -- 172.21.15.89:0/4151627196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c009550 con 0x7f6bb0161590 2024-04-02T07:51:34.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.313+0000 7f6bae800700 1 --2- 172.21.15.89:0/4151627196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb0161590 0x7f6bb0153700 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800b970 tx=0x7f6ba800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:34.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.313+0000 7f6ba6000700 1 -- 172.21.15.89:0/4151627196 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba800d860 con 0x7f6bb0161590 2024-04-02T07:51:34.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.313+0000 7f6ba6000700 1 -- 172.21.15.89:0/4151627196 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba800d9c0 con 0x7f6bb0161590 2024-04-02T07:51:34.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.313+0000 7f6ba6000700 1 -- 172.21.15.89:0/4151627196 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba800dd70 con 0x7f6bb0161590 2024-04-02T07:51:34.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.313+0000 7f6bb4e00700 1 -- 172.21.15.89:0/4151627196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb0157180 con 0x7f6bb0161590 2024-04-02T07:51:34.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.313+0000 7f6bb4e00700 1 -- 172.21.15.89:0/4151627196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb0159630 con 0x7f6bb0161590 2024-04-02T07:51:34.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.314+0000 7f6bb4e00700 1 -- 172.21.15.89:0/4151627196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb01576a0 con 0x7f6bb0161590 2024-04-02T07:51:34.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.315+0000 7f6ba6000700 1 -- 172.21.15.89:0/4151627196 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6ba800e430 con 0x7f6bb0161590 2024-04-02T07:51:34.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.316+0000 7f6ba6000700 1 --2- 172.21.15.89:0/4151627196 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b9006adb0 0x7f6b9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:34.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.316+0000 7f6ba6000700 1 -- 172.21.15.89:0/4151627196 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6ba8089810 con 0x7f6bb0161590 2024-04-02T07:51:34.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.316+0000 7f6bade00700 1 --2- 172.21.15.89:0/4151627196 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b9006adb0 0x7f6b9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:34.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.317+0000 7f6bade00700 1 --2- 172.21.15.89:0/4151627196 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b9006adb0 0x7f6b9006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6bb01f69c0 tx=0x7f6ba0008040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:34.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.320+0000 7f6ba6000700 1 -- 172.21.15.89:0/4151627196 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6bb01576a0 con 0x7f6bb0161590 2024-04-02T07:51:34.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.517+0000 7f6bb4e00700 1 -- 172.21.15.89:0/4151627196 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6bb0156d30 con 0x7f6b9006adb0 2024-04-02T07:51:34.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.525+0000 7f6ba6000700 1 -- 172.21.15.89:0/4151627196 <== mgr.6180 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289094 (secure 0 0 0) 0x7f6bb0156d30 con 0x7f6b9006adb0 2024-04-02T07:51:34.530 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:34.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.530+0000 7f6bb4e00700 1 -- 172.21.15.89:0/4151627196 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b9006adb0 msgr2=0x7f6b9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:34.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.530+0000 7f6bb4e00700 1 --2- 172.21.15.89:0/4151627196 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b9006adb0 0x7f6b9006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6bb01f69c0 tx=0x7f6ba0008040 comp rx=0 tx=0).stop 2024-04-02T07:51:34.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.530+0000 7f6bb4e00700 1 -- 172.21.15.89:0/4151627196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb0161590 msgr2=0x7f6bb0153700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:34.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.530+0000 7f6bb4e00700 1 --2- 172.21.15.89:0/4151627196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb0161590 0x7f6bb0153700 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800b970 tx=0x7f6ba800bc80 comp rx=0 tx=0).stop 2024-04-02T07:51:34.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.530+0000 7f6bb4e00700 1 -- 172.21.15.89:0/4151627196 shutdown_connections 2024-04-02T07:51:34.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.530+0000 7f6bb4e00700 1 --2- 172.21.15.89:0/4151627196 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b9006adb0 0x7f6b9006d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.530+0000 7f6bb4e00700 1 --2- 172.21.15.89:0/4151627196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bb00cf230 0x7f6bb01590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.530+0000 7f6bb4e00700 1 --2- 172.21.15.89:0/4151627196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb0161590 0x7f6bb0153700 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.530+0000 7f6bb4e00700 1 --2- 172.21.15.89:0/4151627196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb0160f10 0x7f6bb0151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.530+0000 7f6bb4e00700 1 -- 172.21.15.89:0/4151627196 >> 172.21.15.89:0/4151627196 conn(0x7f6bb008aff0 msgr2=0x7f6bb00b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:34.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.530+0000 7f6bb4e00700 1 -- 172.21.15.89:0/4151627196 shutdown_connections 2024-04-02T07:51:34.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.530+0000 7f6bb4e00700 1 -- 172.21.15.89:0/4151627196 wait complete. 2024-04-02T07:51:34.534 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:51:34.550 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-02T07:51:34.127535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":270228,"kb_used_data":8960,"kb_used_omap":12,"kb_used_meta":261107,"kb_avail":754704492,"statfs":{"total":773094113280,"available":772817399808,"internally_reserved":0,"allocated":9175040,"data_stored":5012830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":267374136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.943651"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.718181+0000","last_change":"2024-04-02T07:51:29.708984+0000","last_active":"2024-04-02T07:51:29.718181+0000","last_peered":"2024-04-02T07:51:29.718181+0000","last_clean":"2024-04-02T07:51:29.718181+0000","last_became_active":"2024-04-02T07:51:29.708051+0000","last_became_peered":"2024-04-02T07:51:29.708051+0000","last_unstale":"2024-04-02T07:51:29.718181+0000","last_undegraded":"2024-04-02T07:51:29.718181+0000","last_fullsized":"2024-04-02T07:51:29.718181+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:42:13.876241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702347+0000","last_change":"2024-04-02T07:51:29.718862+0000","last_active":"2024-04-02T07:51:30.702347+0000","last_peered":"2024-04-02T07:51:30.702347+0000","last_clean":"2024-04-02T07:51:30.702347+0000","last_became_active":"2024-04-02T07:51:29.718570+0000","last_became_peered":"2024-04-02T07:51:29.718570+0000","last_unstale":"2024-04-02T07:51:30.702347+0000","last_undegraded":"2024-04-02T07:51:30.702347+0000","last_fullsized":"2024-04-02T07:51:30.702347+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:51.717980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103382+0000","last_change":"2024-04-02T07:51:29.710138+0000","last_active":"2024-04-02T07:51:31.103382+0000","last_peered":"2024-04-02T07:51:31.103382+0000","last_clean":"2024-04-02T07:51:31.103382+0000","last_became_active":"2024-04-02T07:51:29.709891+0000","last_became_peered":"2024-04-02T07:51:29.709891+0000","last_unstale":"2024-04-02T07:51:31.103382+0000","last_undegraded":"2024-04-02T07:51:31.103382+0000","last_fullsized":"2024-04-02T07:51:31.103382+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:42:13.876241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696788+0000","last_change":"2024-04-02T07:51:29.719115+0000","last_active":"2024-04-02T07:51:30.696788+0000","last_peered":"2024-04-02T07:51:30.696788+0000","last_clean":"2024-04-02T07:51:30.696788+0000","last_became_active":"2024-04-02T07:51:29.716813+0000","last_became_peered":"2024-04-02T07:51:29.716813+0000","last_unstale":"2024-04-02T07:51:30.696788+0000","last_undegraded":"2024-04-02T07:51:30.696788+0000","last_fullsized":"2024-04-02T07:51:30.696788+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:04:04.846110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.724585+0000","last_change":"2024-04-02T07:51:29.722981+0000","last_active":"2024-04-02T07:51:29.724585+0000","last_peered":"2024-04-02T07:51:29.724585+0000","last_clean":"2024-04-02T07:51:29.724585+0000","last_became_active":"2024-04-02T07:51:29.722635+0000","last_became_peered":"2024-04-02T07:51:29.722635+0000","last_unstale":"2024-04-02T07:51:29.724585+0000","last_undegraded":"2024-04-02T07:51:29.724585+0000","last_fullsized":"2024-04-02T07:51:29.724585+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:51.717980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.717838+0000","last_change":"2024-04-02T07:51:29.709285+0000","last_active":"2024-04-02T07:51:29.717838+0000","last_peered":"2024-04-02T07:51:29.717838+0000","last_clean":"2024-04-02T07:51:29.717838+0000","last_became_active":"2024-04-02T07:51:29.707754+0000","last_became_peered":"2024-04-02T07:51:29.707754+0000","last_unstale":"2024-04-02T07:51:29.717838+0000","last_undegraded":"2024-04-02T07:51:29.717838+0000","last_fullsized":"2024-04-02T07:51:29.717838+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:21:07.376954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696369+0000","last_change":"2024-04-02T07:51:29.707857+0000","last_active":"2024-04-02T07:51:30.696369+0000","last_peered":"2024-04-02T07:51:30.696369+0000","last_clean":"2024-04-02T07:51:30.696369+0000","last_became_active":"2024-04-02T07:51:29.705640+0000","last_became_peered":"2024-04-02T07:51:29.705640+0000","last_unstale":"2024-04-02T07:51:30.696369+0000","last_undegraded":"2024-04-02T07:51:30.696369+0000","last_fullsized":"2024-04-02T07:51:30.696369+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:42:30.084208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.709565+0000","last_change":"2024-04-02T07:51:29.707298+0000","last_active":"2024-04-02T07:51:29.709565+0000","last_peered":"2024-04-02T07:51:29.709565+0000","last_clean":"2024-04-02T07:51:29.709565+0000","last_became_active":"2024-04-02T07:51:29.706557+0000","last_became_peered":"2024-04-02T07:51:29.706557+0000","last_unstale":"2024-04-02T07:51:29.709565+0000","last_undegraded":"2024-04-02T07:51:29.709565+0000","last_fullsized":"2024-04-02T07:51:29.709565+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:30:23.324413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.722316+0000","last_change":"2024-04-02T07:51:29.720164+0000","last_active":"2024-04-02T07:51:29.722316+0000","last_peered":"2024-04-02T07:51:29.722316+0000","last_clean":"2024-04-02T07:51:29.722316+0000","last_became_active":"2024-04-02T07:51:29.719928+0000","last_became_peered":"2024-04-02T07:51:29.719928+0000","last_unstale":"2024-04-02T07:51:29.722316+0000","last_undegraded":"2024-04-02T07:51:29.722316+0000","last_fullsized":"2024-04-02T07:51:29.722316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:21:07.376954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.717325+0000","last_change":"2024-04-02T07:51:29.713404+0000","last_active":"2024-04-02T07:51:29.717325+0000","last_peered":"2024-04-02T07:51:29.717325+0000","last_clean":"2024-04-02T07:51:29.717325+0000","last_became_active":"2024-04-02T07:51:29.713127+0000","last_became_peered":"2024-04-02T07:51:29.713127+0000","last_unstale":"2024-04-02T07:51:29.717325+0000","last_undegraded":"2024-04-02T07:51:29.717325+0000","last_fullsized":"2024-04-02T07:51:29.717325+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:18:41.743679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.701468+0000","last_change":"2024-04-02T07:51:29.712512+0000","last_active":"2024-04-02T07:51:30.701468+0000","last_peered":"2024-04-02T07:51:30.701468+0000","last_clean":"2024-04-02T07:51:30.701468+0000","last_became_active":"2024-04-02T07:51:29.712164+0000","last_became_peered":"2024-04-02T07:51:29.712164+0000","last_unstale":"2024-04-02T07:51:30.701468+0000","last_undegraded":"2024-04-02T07:51:30.701468+0000","last_fullsized":"2024-04-02T07:51:30.701468+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:06:44.825237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.718839+0000","last_change":"2024-04-02T07:51:29.712495+0000","last_active":"2024-04-02T07:51:29.718839+0000","last_peered":"2024-04-02T07:51:29.718839+0000","last_clean":"2024-04-02T07:51:29.718839+0000","last_became_active":"2024-04-02T07:51:29.712276+0000","last_became_peered":"2024-04-02T07:51:29.712276+0000","last_unstale":"2024-04-02T07:51:29.718839+0000","last_undegraded":"2024-04-02T07:51:29.718839+0000","last_fullsized":"2024-04-02T07:51:29.718839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:31:53.301807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.711505+0000","last_change":"2024-04-02T07:51:29.701268+0000","last_active":"2024-04-02T07:51:29.711505+0000","last_peered":"2024-04-02T07:51:29.711505+0000","last_clean":"2024-04-02T07:51:29.711505+0000","last_became_active":"2024-04-02T07:51:29.699100+0000","last_became_peered":"2024-04-02T07:51:29.699100+0000","last_unstale":"2024-04-02T07:51:29.711505+0000","last_undegraded":"2024-04-02T07:51:29.711505+0000","last_fullsized":"2024-04-02T07:51:29.711505+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:26.509824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103157+0000","last_change":"2024-04-02T07:51:29.707165+0000","last_active":"2024-04-02T07:51:31.103157+0000","last_peered":"2024-04-02T07:51:31.103157+0000","last_clean":"2024-04-02T07:51:31.103157+0000","last_became_active":"2024-04-02T07:51:29.706118+0000","last_became_peered":"2024-04-02T07:51:29.706118+0000","last_unstale":"2024-04-02T07:51:31.103157+0000","last_undegraded":"2024-04-02T07:51:31.103157+0000","last_fullsized":"2024-04-02T07:51:31.103157+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:59:03.414703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.707189+0000","last_change":"2024-04-02T07:51:29.700256+0000","last_active":"2024-04-02T07:51:29.707189+0000","last_peered":"2024-04-02T07:51:29.707189+0000","last_clean":"2024-04-02T07:51:29.707189+0000","last_became_active":"2024-04-02T07:51:29.700016+0000","last_became_peered":"2024-04-02T07:51:29.700016+0000","last_unstale":"2024-04-02T07:51:29.707189+0000","last_undegraded":"2024-04-02T07:51:29.707189+0000","last_fullsized":"2024-04-02T07:51:29.707189+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:06:00.157486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.719564+0000","last_change":"2024-04-02T07:51:29.718399+0000","last_active":"2024-04-02T07:51:29.719564+0000","last_peered":"2024-04-02T07:51:29.719564+0000","last_clean":"2024-04-02T07:51:29.719564+0000","last_became_active":"2024-04-02T07:51:29.716526+0000","last_became_peered":"2024-04-02T07:51:29.716526+0000","last_unstale":"2024-04-02T07:51:29.719564+0000","last_undegraded":"2024-04-02T07:51:29.719564+0000","last_fullsized":"2024-04-02T07:51:29.719564+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:59:28.338083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.718209+0000","last_change":"2024-04-02T07:51:29.714366+0000","last_active":"2024-04-02T07:51:29.718209+0000","last_peered":"2024-04-02T07:51:29.718209+0000","last_clean":"2024-04-02T07:51:29.718209+0000","last_became_active":"2024-04-02T07:51:29.713548+0000","last_became_peered":"2024-04-02T07:51:29.713548+0000","last_unstale":"2024-04-02T07:51:29.718209+0000","last_undegraded":"2024-04-02T07:51:29.718209+0000","last_fullsized":"2024-04-02T07:51:29.718209+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:37:43.467101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103352+0000","last_change":"2024-04-02T07:51:29.718663+0000","last_active":"2024-04-02T07:51:31.103352+0000","last_peered":"2024-04-02T07:51:31.103352+0000","last_clean":"2024-04-02T07:51:31.103352+0000","last_became_active":"2024-04-02T07:51:29.718412+0000","last_became_peered":"2024-04-02T07:51:29.718412+0000","last_unstale":"2024-04-02T07:51:31.103352+0000","last_undegraded":"2024-04-02T07:51:31.103352+0000","last_fullsized":"2024-04-02T07:51:31.103352+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:51.717980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702439+0000","last_change":"2024-04-02T07:51:29.715618+0000","last_active":"2024-04-02T07:51:30.702439+0000","last_peered":"2024-04-02T07:51:30.702439+0000","last_clean":"2024-04-02T07:51:30.702439+0000","last_became_active":"2024-04-02T07:51:29.703080+0000","last_became_peered":"2024-04-02T07:51:29.703080+0000","last_unstale":"2024-04-02T07:51:30.702439+0000","last_undegraded":"2024-04-02T07:51:30.702439+0000","last_fullsized":"2024-04-02T07:51:30.702439+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:42:13.876241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.701488+0000","last_change":"2024-04-02T07:51:29.715915+0000","last_active":"2024-04-02T07:51:30.701488+0000","last_peered":"2024-04-02T07:51:30.701488+0000","last_clean":"2024-04-02T07:51:30.701488+0000","last_became_active":"2024-04-02T07:51:29.715679+0000","last_became_peered":"2024-04-02T07:51:29.715679+0000","last_unstale":"2024-04-02T07:51:30.701488+0000","last_undegraded":"2024-04-02T07:51:30.701488+0000","last_fullsized":"2024-04-02T07:51:30.701488+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:59:01.142628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.711539+0000","last_change":"2024-04-02T07:51:29.704509+0000","last_active":"2024-04-02T07:51:29.711539+0000","last_peered":"2024-04-02T07:51:29.711539+0000","last_clean":"2024-04-02T07:51:29.711539+0000","last_became_active":"2024-04-02T07:51:29.702810+0000","last_became_peered":"2024-04-02T07:51:29.702810+0000","last_unstale":"2024-04-02T07:51:29.711539+0000","last_undegraded":"2024-04-02T07:51:29.711539+0000","last_fullsized":"2024-04-02T07:51:29.711539+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:17:49.862029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.701426+0000","last_change":"2024-04-02T07:51:29.718728+0000","last_active":"2024-04-02T07:51:30.701426+0000","last_peered":"2024-04-02T07:51:30.701426+0000","last_clean":"2024-04-02T07:51:30.701426+0000","last_became_active":"2024-04-02T07:51:29.718439+0000","last_became_peered":"2024-04-02T07:51:29.718439+0000","last_unstale":"2024-04-02T07:51:30.701426+0000","last_undegraded":"2024-04-02T07:51:30.701426+0000","last_fullsized":"2024-04-02T07:51:30.701426+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:04:57.953364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696603+0000","last_change":"2024-04-02T07:51:29.708035+0000","last_active":"2024-04-02T07:51:30.696603+0000","last_peered":"2024-04-02T07:51:30.696603+0000","last_clean":"2024-04-02T07:51:30.696603+0000","last_became_active":"2024-04-02T07:51:29.705946+0000","last_became_peered":"2024-04-02T07:51:29.705946+0000","last_unstale":"2024-04-02T07:51:30.696603+0000","last_undegraded":"2024-04-02T07:51:30.696603+0000","last_fullsized":"2024-04-02T07:51:30.696603+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:38:36.206166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.722372+0000","last_change":"2024-04-02T07:51:29.721011+0000","last_active":"2024-04-02T07:51:29.722372+0000","last_peered":"2024-04-02T07:51:29.722372+0000","last_clean":"2024-04-02T07:51:29.722372+0000","last_became_active":"2024-04-02T07:51:29.720430+0000","last_became_peered":"2024-04-02T07:51:29.720430+0000","last_unstale":"2024-04-02T07:51:29.722372+0000","last_undegraded":"2024-04-02T07:51:29.722372+0000","last_fullsized":"2024-04-02T07:51:29.722372+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:58:01.264306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702621+0000","last_change":"2024-04-02T07:51:29.719460+0000","last_active":"2024-04-02T07:51:30.702621+0000","last_peered":"2024-04-02T07:51:30.702621+0000","last_clean":"2024-04-02T07:51:30.702621+0000","last_became_active":"2024-04-02T07:51:29.719134+0000","last_became_peered":"2024-04-02T07:51:29.719134+0000","last_unstale":"2024-04-02T07:51:30.702621+0000","last_undegraded":"2024-04-02T07:51:30.702621+0000","last_fullsized":"2024-04-02T07:51:30.702621+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:18:41.743679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.717061+0000","last_change":"2024-04-02T07:51:29.713185+0000","last_active":"2024-04-02T07:51:29.717061+0000","last_peered":"2024-04-02T07:51:29.717061+0000","last_clean":"2024-04-02T07:51:29.717061+0000","last_became_active":"2024-04-02T07:51:29.712960+0000","last_became_peered":"2024-04-02T07:51:29.712960+0000","last_unstale":"2024-04-02T07:51:29.717061+0000","last_undegraded":"2024-04-02T07:51:29.717061+0000","last_fullsized":"2024-04-02T07:51:29.717061+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:02:41.176141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702523+0000","last_change":"2024-04-02T07:51:29.716218+0000","last_active":"2024-04-02T07:51:30.702523+0000","last_peered":"2024-04-02T07:51:30.702523+0000","last_clean":"2024-04-02T07:51:30.702523+0000","last_became_active":"2024-04-02T07:51:29.715890+0000","last_became_peered":"2024-04-02T07:51:29.715890+0000","last_unstale":"2024-04-02T07:51:30.702523+0000","last_undegraded":"2024-04-02T07:51:30.702523+0000","last_fullsized":"2024-04-02T07:51:30.702523+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:59:28.338083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.715422+0000","last_change":"2024-04-02T07:51:29.709886+0000","last_active":"2024-04-02T07:51:29.715422+0000","last_peered":"2024-04-02T07:51:29.715422+0000","last_clean":"2024-04-02T07:51:29.715422+0000","last_became_active":"2024-04-02T07:51:29.708495+0000","last_became_peered":"2024-04-02T07:51:29.708495+0000","last_unstale":"2024-04-02T07:51:29.715422+0000","last_undegraded":"2024-04-02T07:51:29.715422+0000","last_fullsized":"2024-04-02T07:51:29.715422+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:38:36.206166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.709668+0000","last_change":"2024-04-02T07:51:29.706033+0000","last_active":"2024-04-02T07:51:29.709668+0000","last_peered":"2024-04-02T07:51:29.709668+0000","last_clean":"2024-04-02T07:51:29.709668+0000","last_became_active":"2024-04-02T07:51:29.705760+0000","last_became_peered":"2024-04-02T07:51:29.705760+0000","last_unstale":"2024-04-02T07:51:29.709668+0000","last_undegraded":"2024-04-02T07:51:29.709668+0000","last_fullsized":"2024-04-02T07:51:29.709668+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:42:30.084208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696548+0000","last_change":"2024-04-02T07:51:29.716230+0000","last_active":"2024-04-02T07:51:30.696548+0000","last_peered":"2024-04-02T07:51:30.696548+0000","last_clean":"2024-04-02T07:51:30.696548+0000","last_became_active":"2024-04-02T07:51:29.715641+0000","last_became_peered":"2024-04-02T07:51:29.715641+0000","last_unstale":"2024-04-02T07:51:30.696548+0000","last_undegraded":"2024-04-02T07:51:30.696548+0000","last_fullsized":"2024-04-02T07:51:30.696548+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:17.086170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702318+0000","last_change":"2024-04-02T07:51:29.712337+0000","last_active":"2024-04-02T07:51:30.702318+0000","last_peered":"2024-04-02T07:51:30.702318+0000","last_clean":"2024-04-02T07:51:30.702318+0000","last_became_active":"2024-04-02T07:51:29.711742+0000","last_became_peered":"2024-04-02T07:51:29.711742+0000","last_unstale":"2024-04-02T07:51:30.702318+0000","last_undegraded":"2024-04-02T07:51:30.702318+0000","last_fullsized":"2024-04-02T07:51:30.702318+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:08:52.637936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.701278+0000","last_change":"2024-04-02T07:51:29.718809+0000","last_active":"2024-04-02T07:51:30.701278+0000","last_peered":"2024-04-02T07:51:30.701278+0000","last_clean":"2024-04-02T07:51:30.701278+0000","last_became_active":"2024-04-02T07:51:29.718537+0000","last_became_peered":"2024-04-02T07:51:29.718537+0000","last_unstale":"2024-04-02T07:51:30.701278+0000","last_undegraded":"2024-04-02T07:51:30.701278+0000","last_fullsized":"2024-04-02T07:51:30.701278+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:56:25.462955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.717238+0000","last_change":"2024-04-02T07:51:29.709083+0000","last_active":"2024-04-02T07:51:29.717238+0000","last_peered":"2024-04-02T07:51:29.717238+0000","last_clean":"2024-04-02T07:51:29.717238+0000","last_became_active":"2024-04-02T07:51:29.708394+0000","last_became_peered":"2024-04-02T07:51:29.708394+0000","last_unstale":"2024-04-02T07:51:29.717238+0000","last_undegraded":"2024-04-02T07:51:29.717238+0000","last_fullsized":"2024-04-02T07:51:29.717238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:08.732683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1f","version":"82'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.701874+0000","last_change":"2024-04-02T07:51:29.198522+0000","last_active":"2024-04-02T07:51:30.701874+0000","last_peered":"2024-04-02T07:51:30.701874+0000","last_clean":"2024-04-02T07:51:30.701874+0000","last_became_active":"2024-04-02T07:51:29.196839+0000","last_became_peered":"2024-04-02T07:51:29.196839+0000","last_unstale":"2024-04-02T07:51:30.701874+0000","last_undegraded":"2024-04-02T07:51:30.701874+0000","last_fullsized":"2024-04-02T07:51:30.701874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:50.709598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.709071+0000","last_change":"2024-04-02T07:51:29.706632+0000","last_active":"2024-04-02T07:51:29.709071+0000","last_peered":"2024-04-02T07:51:29.709071+0000","last_clean":"2024-04-02T07:51:29.709071+0000","last_became_active":"2024-04-02T07:51:29.706343+0000","last_became_peered":"2024-04-02T07:51:29.706343+0000","last_unstale":"2024-04-02T07:51:29.709071+0000","last_undegraded":"2024-04-02T07:51:29.709071+0000","last_fullsized":"2024-04-02T07:51:29.709071+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:58:01.264306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"82'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702545+0000","last_change":"2024-04-02T07:51:29.210507+0000","last_active":"2024-04-02T07:51:30.702545+0000","last_peered":"2024-04-02T07:51:30.702545+0000","last_clean":"2024-04-02T07:51:30.702545+0000","last_became_active":"2024-04-02T07:51:29.210204+0000","last_became_peered":"2024-04-02T07:51:29.210204+0000","last_unstale":"2024-04-02T07:51:30.702545+0000","last_undegraded":"2024-04-02T07:51:30.702545+0000","last_fullsized":"2024-04-02T07:51:30.702545+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:10:14.951279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.716857+0000","last_change":"2024-04-02T07:51:29.714388+0000","last_active":"2024-04-02T07:51:29.716857+0000","last_peered":"2024-04-02T07:51:29.716857+0000","last_clean":"2024-04-02T07:51:29.716857+0000","last_became_active":"2024-04-02T07:51:29.714082+0000","last_became_peered":"2024-04-02T07:51:29.714082+0000","last_unstale":"2024-04-02T07:51:29.716857+0000","last_undegraded":"2024-04-02T07:51:29.716857+0000","last_fullsized":"2024-04-02T07:51:29.716857+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:04:04.846110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"82'2","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.723953+0000","last_change":"2024-04-02T07:51:29.208521+0000","last_active":"2024-04-02T07:51:29.723953+0000","last_peered":"2024-04-02T07:51:29.723953+0000","last_clean":"2024-04-02T07:51:29.723953+0000","last_became_active":"2024-04-02T07:51:29.208245+0000","last_became_peered":"2024-04-02T07:51:29.208245+0000","last_unstale":"2024-04-02T07:51:29.723953+0000","last_undegraded":"2024-04-02T07:51:29.723953+0000","last_fullsized":"2024-04-02T07:51:29.723953+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:03.852318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.709047+0000","last_change":"2024-04-02T07:51:29.697968+0000","last_active":"2024-04-02T07:51:29.709047+0000","last_peered":"2024-04-02T07:51:29.709047+0000","last_clean":"2024-04-02T07:51:29.709047+0000","last_became_active":"2024-04-02T07:51:29.697191+0000","last_became_peered":"2024-04-02T07:51:29.697191+0000","last_unstale":"2024-04-02T07:51:29.709047+0000","last_undegraded":"2024-04-02T07:51:29.709047+0000","last_fullsized":"2024-04-02T07:51:29.709047+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:31:53.301807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"82'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696942+0000","last_change":"2024-04-02T07:51:29.211917+0000","last_active":"2024-04-02T07:51:30.696942+0000","last_peered":"2024-04-02T07:51:30.696942+0000","last_clean":"2024-04-02T07:51:30.696942+0000","last_became_active":"2024-04-02T07:51:29.211624+0000","last_became_peered":"2024-04-02T07:51:29.211624+0000","last_unstale":"2024-04-02T07:51:30.696942+0000","last_undegraded":"2024-04-02T07:51:30.696942+0000","last_fullsized":"2024-04-02T07:51:30.696942+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:41:46.481303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.715337+0000","last_change":"2024-04-02T07:51:29.709126+0000","last_active":"2024-04-02T07:51:29.715337+0000","last_peered":"2024-04-02T07:51:29.715337+0000","last_clean":"2024-04-02T07:51:29.715337+0000","last_became_active":"2024-04-02T07:51:29.708861+0000","last_became_peered":"2024-04-02T07:51:29.708861+0000","last_unstale":"2024-04-02T07:51:29.715337+0000","last_undegraded":"2024-04-02T07:51:29.715337+0000","last_fullsized":"2024-04-02T07:51:29.715337+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:26:51.319846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1b","version":"82'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103304+0000","last_change":"2024-04-02T07:51:29.202610+0000","last_active":"2024-04-02T07:51:31.103304+0000","last_peered":"2024-04-02T07:51:31.103304+0000","last_clean":"2024-04-02T07:51:31.103304+0000","last_became_active":"2024-04-02T07:51:29.202339+0000","last_became_peered":"2024-04-02T07:51:29.202339+0000","last_unstale":"2024-04-02T07:51:31.103304+0000","last_undegraded":"2024-04-02T07:51:31.103304+0000","last_fullsized":"2024-04-02T07:51:31.103304+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:53:15.044497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.701629+0000","last_change":"2024-04-02T07:51:29.720513+0000","last_active":"2024-04-02T07:51:30.701629+0000","last_peered":"2024-04-02T07:51:30.701629+0000","last_clean":"2024-04-02T07:51:30.701629+0000","last_became_active":"2024-04-02T07:51:29.717979+0000","last_became_peered":"2024-04-02T07:51:29.717979+0000","last_unstale":"2024-04-02T07:51:30.701629+0000","last_undegraded":"2024-04-02T07:51:30.701629+0000","last_fullsized":"2024-04-02T07:51:30.701629+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:28.398387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"82'1","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.709224+0000","last_change":"2024-04-02T07:51:29.211974+0000","last_active":"2024-04-02T07:51:29.709224+0000","last_peered":"2024-04-02T07:51:29.709224+0000","last_clean":"2024-04-02T07:51:29.709224+0000","last_became_active":"2024-04-02T07:51:29.211714+0000","last_became_peered":"2024-04-02T07:51:29.211714+0000","last_unstale":"2024-04-02T07:51:29.709224+0000","last_undegraded":"2024-04-02T07:51:29.709224+0000","last_fullsized":"2024-04-02T07:51:29.709224+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:59:24.112828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103132+0000","last_change":"2024-04-02T07:51:29.715224+0000","last_active":"2024-04-02T07:51:31.103132+0000","last_peered":"2024-04-02T07:51:31.103132+0000","last_clean":"2024-04-02T07:51:31.103132+0000","last_became_active":"2024-04-02T07:51:29.714932+0000","last_became_peered":"2024-04-02T07:51:29.714932+0000","last_unstale":"2024-04-02T07:51:31.103132+0000","last_undegraded":"2024-04-02T07:51:31.103132+0000","last_fullsized":"2024-04-02T07:51:31.103132+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:37:43.467101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.703214+0000","last_change":"2024-04-02T07:51:29.197304+0000","last_active":"2024-04-02T07:51:29.703214+0000","last_peered":"2024-04-02T07:51:29.703214+0000","last_clean":"2024-04-02T07:51:29.703214+0000","last_became_active":"2024-04-02T07:51:29.196275+0000","last_became_peered":"2024-04-02T07:51:29.196275+0000","last_unstale":"2024-04-02T07:51:29.703214+0000","last_undegraded":"2024-04-02T07:51:29.703214+0000","last_fullsized":"2024-04-02T07:51:29.703214+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:53:15.044497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.724652+0000","last_change":"2024-04-02T07:51:29.723487+0000","last_active":"2024-04-02T07:51:29.724652+0000","last_peered":"2024-04-02T07:51:29.724652+0000","last_clean":"2024-04-02T07:51:29.724652+0000","last_became_active":"2024-04-02T07:51:29.723277+0000","last_became_peered":"2024-04-02T07:51:29.723277+0000","last_unstale":"2024-04-02T07:51:29.724652+0000","last_undegraded":"2024-04-02T07:51:29.724652+0000","last_fullsized":"2024-04-02T07:51:29.724652+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:08:52.637936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.689040+0000","last_change":"2024-04-02T07:51:29.202118+0000","last_active":"2024-04-02T07:51:29.689040+0000","last_peered":"2024-04-02T07:51:29.689040+0000","last_clean":"2024-04-02T07:51:29.689040+0000","last_became_active":"2024-04-02T07:51:29.200536+0000","last_became_peered":"2024-04-02T07:51:29.200536+0000","last_unstale":"2024-04-02T07:51:29.689040+0000","last_undegraded":"2024-04-02T07:51:29.689040+0000","last_fullsized":"2024-04-02T07:51:29.689040+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:24:11.272656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.710912+0000","last_change":"2024-04-02T07:51:29.703723+0000","last_active":"2024-04-02T07:51:29.710912+0000","last_peered":"2024-04-02T07:51:29.710912+0000","last_clean":"2024-04-02T07:51:29.710912+0000","last_became_active":"2024-04-02T07:51:29.703033+0000","last_became_peered":"2024-04-02T07:51:29.703033+0000","last_unstale":"2024-04-02T07:51:29.710912+0000","last_undegraded":"2024-04-02T07:51:29.710912+0000","last_fullsized":"2024-04-02T07:51:29.710912+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:06:15.474439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.702623+0000","last_change":"2024-04-02T07:51:29.205089+0000","last_active":"2024-04-02T07:51:29.702623+0000","last_peered":"2024-04-02T07:51:29.702623+0000","last_clean":"2024-04-02T07:51:29.702623+0000","last_became_active":"2024-04-02T07:51:29.204756+0000","last_became_peered":"2024-04-02T07:51:29.204756+0000","last_unstale":"2024-04-02T07:51:29.702623+0000","last_undegraded":"2024-04-02T07:51:29.702623+0000","last_fullsized":"2024-04-02T07:51:29.702623+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:32:45.599873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103484+0000","last_change":"2024-04-02T07:51:29.707289+0000","last_active":"2024-04-02T07:51:31.103484+0000","last_peered":"2024-04-02T07:51:31.103484+0000","last_clean":"2024-04-02T07:51:31.103484+0000","last_became_active":"2024-04-02T07:51:29.706957+0000","last_became_peered":"2024-04-02T07:51:29.706957+0000","last_unstale":"2024-04-02T07:51:31.103484+0000","last_undegraded":"2024-04-02T07:51:31.103484+0000","last_fullsized":"2024-04-02T07:51:31.103484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:57:02.013161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.700376+0000","last_change":"2024-04-02T07:51:29.208251+0000","last_active":"2024-04-02T07:51:29.700376+0000","last_peered":"2024-04-02T07:51:29.700376+0000","last_clean":"2024-04-02T07:51:29.700376+0000","last_became_active":"2024-04-02T07:51:29.207979+0000","last_became_peered":"2024-04-02T07:51:29.207979+0000","last_unstale":"2024-04-02T07:51:29.700376+0000","last_undegraded":"2024-04-02T07:51:29.700376+0000","last_fullsized":"2024-04-02T07:51:29.700376+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:37:14.168678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.716875+0000","last_change":"2024-04-02T07:51:29.715158+0000","last_active":"2024-04-02T07:51:29.716875+0000","last_peered":"2024-04-02T07:51:29.716875+0000","last_clean":"2024-04-02T07:51:29.716875+0000","last_became_active":"2024-04-02T07:51:29.714147+0000","last_became_peered":"2024-04-02T07:51:29.714147+0000","last_unstale":"2024-04-02T07:51:29.716875+0000","last_undegraded":"2024-04-02T07:51:29.716875+0000","last_fullsized":"2024-04-02T07:51:29.716875+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:17.086170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"82'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696943+0000","last_change":"2024-04-02T07:51:29.209225+0000","last_active":"2024-04-02T07:51:30.696943+0000","last_peered":"2024-04-02T07:51:30.696943+0000","last_clean":"2024-04-02T07:51:30.696943+0000","last_became_active":"2024-04-02T07:51:29.208698+0000","last_became_peered":"2024-04-02T07:51:29.208698+0000","last_unstale":"2024-04-02T07:51:30.696943+0000","last_undegraded":"2024-04-02T07:51:30.696943+0000","last_fullsized":"2024-04-02T07:51:30.696943+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:19.741531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696871+0000","last_change":"2024-04-02T07:51:29.709404+0000","last_active":"2024-04-02T07:51:30.696871+0000","last_peered":"2024-04-02T07:51:30.696871+0000","last_clean":"2024-04-02T07:51:30.696871+0000","last_became_active":"2024-04-02T07:51:29.705558+0000","last_became_peered":"2024-04-02T07:51:29.705558+0000","last_unstale":"2024-04-02T07:51:30.696871+0000","last_undegraded":"2024-04-02T07:51:30.696871+0000","last_fullsized":"2024-04-02T07:51:30.696871+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:24:50.492280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"82'6","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.719460+0000","last_change":"2024-04-02T07:51:29.211399+0000","last_active":"2024-04-02T07:51:29.719460+0000","last_peered":"2024-04-02T07:51:29.719460+0000","last_clean":"2024-04-02T07:51:29.719460+0000","last_became_active":"2024-04-02T07:51:29.211120+0000","last_became_peered":"2024-04-02T07:51:29.211120+0000","last_unstale":"2024-04-02T07:51:29.719460+0000","last_undegraded":"2024-04-02T07:51:29.719460+0000","last_fullsized":"2024-04-02T07:51:29.719460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:56:24.454573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.717631+0000","last_change":"2024-04-02T07:51:29.711525+0000","last_active":"2024-04-02T07:51:29.717631+0000","last_peered":"2024-04-02T07:51:29.717631+0000","last_clean":"2024-04-02T07:51:29.717631+0000","last_became_active":"2024-04-02T07:51:29.711242+0000","last_became_peered":"2024-04-02T07:51:29.711242+0000","last_unstale":"2024-04-02T07:51:29.717631+0000","last_undegraded":"2024-04-02T07:51:29.717631+0000","last_fullsized":"2024-04-02T07:51:29.717631+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:02:53.831136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"82'2","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.722128+0000","last_change":"2024-04-02T07:51:29.207330+0000","last_active":"2024-04-02T07:51:29.722128+0000","last_peered":"2024-04-02T07:51:29.722128+0000","last_clean":"2024-04-02T07:51:29.722128+0000","last_became_active":"2024-04-02T07:51:29.207056+0000","last_became_peered":"2024-04-02T07:51:29.207056+0000","last_unstale":"2024-04-02T07:51:29.722128+0000","last_undegraded":"2024-04-02T07:51:29.722128+0000","last_fullsized":"2024-04-02T07:51:29.722128+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:06:43.816854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103269+0000","last_change":"2024-04-02T07:51:29.703907+0000","last_active":"2024-04-02T07:51:31.103269+0000","last_peered":"2024-04-02T07:51:31.103269+0000","last_clean":"2024-04-02T07:51:31.103269+0000","last_became_active":"2024-04-02T07:51:29.703653+0000","last_became_peered":"2024-04-02T07:51:29.703653+0000","last_unstale":"2024-04-02T07:51:31.103269+0000","last_undegraded":"2024-04-02T07:51:31.103269+0000","last_fullsized":"2024-04-02T07:51:31.103269+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:08.732683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696597+0000","last_change":"2024-04-02T07:51:29.206142+0000","last_active":"2024-04-02T07:51:30.696597+0000","last_peered":"2024-04-02T07:51:30.696597+0000","last_clean":"2024-04-02T07:51:30.696597+0000","last_became_active":"2024-04-02T07:51:29.205364+0000","last_became_peered":"2024-04-02T07:51:29.205364+0000","last_unstale":"2024-04-02T07:51:30.696597+0000","last_undegraded":"2024-04-02T07:51:30.696597+0000","last_fullsized":"2024-04-02T07:51:30.696597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:59.149104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.715019+0000","last_change":"2024-04-02T07:51:29.710447+0000","last_active":"2024-04-02T07:51:29.715019+0000","last_peered":"2024-04-02T07:51:29.715019+0000","last_clean":"2024-04-02T07:51:29.715019+0000","last_became_active":"2024-04-02T07:51:29.709373+0000","last_became_peered":"2024-04-02T07:51:29.709373+0000","last_unstale":"2024-04-02T07:51:29.715019+0000","last_undegraded":"2024-04-02T07:51:29.715019+0000","last_fullsized":"2024-04-02T07:51:29.715019+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:31:29.187096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696695+0000","last_change":"2024-04-02T07:51:29.203986+0000","last_active":"2024-04-02T07:51:30.696695+0000","last_peered":"2024-04-02T07:51:30.696695+0000","last_clean":"2024-04-02T07:51:30.696695+0000","last_became_active":"2024-04-02T07:51:29.203312+0000","last_became_peered":"2024-04-02T07:51:29.203312+0000","last_unstale":"2024-04-02T07:51:30.696695+0000","last_undegraded":"2024-04-02T07:51:30.696695+0000","last_fullsized":"2024-04-02T07:51:30.696695+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:02:40.167758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696508+0000","last_change":"2024-04-02T07:51:29.718710+0000","last_active":"2024-04-02T07:51:30.696508+0000","last_peered":"2024-04-02T07:51:30.696508+0000","last_clean":"2024-04-02T07:51:30.696508+0000","last_became_active":"2024-04-02T07:51:29.718415+0000","last_became_peered":"2024-04-02T07:51:29.718415+0000","last_unstale":"2024-04-02T07:51:30.696508+0000","last_undegraded":"2024-04-02T07:51:30.696508+0000","last_fullsized":"2024-04-02T07:51:30.696508+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:36:52.283458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"82'8","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696424+0000","last_change":"2024-04-02T07:51:29.205842+0000","last_active":"2024-04-02T07:51:30.696424+0000","last_peered":"2024-04-02T07:51:30.696424+0000","last_clean":"2024-04-02T07:51:30.696424+0000","last_became_active":"2024-04-02T07:51:29.205501+0000","last_became_peered":"2024-04-02T07:51:29.205501+0000","last_unstale":"2024-04-02T07:51:30.696424+0000","last_undegraded":"2024-04-02T07:51:30.696424+0000","last_fullsized":"2024-04-02T07:51:30.696424+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:25.501441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103460+0000","last_change":"2024-04-02T07:51:29.705712+0000","last_active":"2024-04-02T07:51:31.103460+0000","last_peered":"2024-04-02T07:51:31.103460+0000","last_clean":"2024-04-02T07:51:31.103460+0000","last_became_active":"2024-04-02T07:51:29.704265+0000","last_became_peered":"2024-04-02T07:51:29.704265+0000","last_unstale":"2024-04-02T07:51:31.103460+0000","last_undegraded":"2024-04-02T07:51:31.103460+0000","last_fullsized":"2024-04-02T07:51:31.103460+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:08:52.637936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"82'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702751+0000","last_change":"2024-04-02T07:51:29.211451+0000","last_active":"2024-04-02T07:51:30.702751+0000","last_peered":"2024-04-02T07:51:30.702751+0000","last_clean":"2024-04-02T07:51:30.702751+0000","last_became_active":"2024-04-02T07:51:29.211207+0000","last_became_peered":"2024-04-02T07:51:29.211207+0000","last_unstale":"2024-04-02T07:51:30.702751+0000","last_undegraded":"2024-04-02T07:51:30.702751+0000","last_fullsized":"2024-04-02T07:51:30.702751+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:32:45.599873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103514+0000","last_change":"2024-04-02T07:51:29.709572+0000","last_active":"2024-04-02T07:51:31.103514+0000","last_peered":"2024-04-02T07:51:31.103514+0000","last_clean":"2024-04-02T07:51:31.103514+0000","last_became_active":"2024-04-02T07:51:29.707369+0000","last_became_peered":"2024-04-02T07:51:29.707369+0000","last_unstale":"2024-04-02T07:51:31.103514+0000","last_undegraded":"2024-04-02T07:51:31.103514+0000","last_fullsized":"2024-04-02T07:51:31.103514+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:29:54.419105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"82'2","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.721380+0000","last_change":"2024-04-02T07:51:29.209179+0000","last_active":"2024-04-02T07:51:29.721380+0000","last_peered":"2024-04-02T07:51:29.721380+0000","last_clean":"2024-04-02T07:51:29.721380+0000","last_became_active":"2024-04-02T07:51:29.207991+0000","last_became_peered":"2024-04-02T07:51:29.207991+0000","last_unstale":"2024-04-02T07:51:29.721380+0000","last_undegraded":"2024-04-02T07:51:29.721380+0000","last_fullsized":"2024-04-02T07:51:29.721380+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:04:56.944982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702144+0000","last_change":"2024-04-02T07:51:29.711748+0000","last_active":"2024-04-02T07:51:30.702144+0000","last_peered":"2024-04-02T07:51:30.702144+0000","last_clean":"2024-04-02T07:51:30.702144+0000","last_became_active":"2024-04-02T07:51:29.711452+0000","last_became_peered":"2024-04-02T07:51:29.711452+0000","last_unstale":"2024-04-02T07:51:30.702144+0000","last_undegraded":"2024-04-02T07:51:30.702144+0000","last_fullsized":"2024-04-02T07:51:30.702144+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:58:01.264306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"82'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103458+0000","last_change":"2024-04-02T07:51:29.206737+0000","last_active":"2024-04-02T07:51:31.103458+0000","last_peered":"2024-04-02T07:51:31.103458+0000","last_clean":"2024-04-02T07:51:31.103458+0000","last_became_active":"2024-04-02T07:51:29.206484+0000","last_became_peered":"2024-04-02T07:51:29.206484+0000","last_unstale":"2024-04-02T07:51:31.103458+0000","last_undegraded":"2024-04-02T07:51:31.103458+0000","last_fullsized":"2024-04-02T07:51:31.103458+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:32:45.599873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.696861+0000","last_change":"2024-04-02T07:51:29.717733+0000","last_active":"2024-04-02T07:51:30.696861+0000","last_peered":"2024-04-02T07:51:30.696861+0000","last_clean":"2024-04-02T07:51:30.696861+0000","last_became_active":"2024-04-02T07:51:29.717450+0000","last_became_peered":"2024-04-02T07:51:29.717450+0000","last_unstale":"2024-04-02T07:51:30.696861+0000","last_undegraded":"2024-04-02T07:51:30.696861+0000","last_fullsized":"2024-04-02T07:51:30.696861+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:26:51.319846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.696788+0000","last_change":"2024-04-02T07:51:29.205966+0000","last_active":"2024-04-02T07:51:29.696788+0000","last_peered":"2024-04-02T07:51:29.696788+0000","last_clean":"2024-04-02T07:51:29.696788+0000","last_became_active":"2024-04-02T07:51:29.205052+0000","last_became_peered":"2024-04-02T07:51:29.205052+0000","last_unstale":"2024-04-02T07:51:29.696788+0000","last_undegraded":"2024-04-02T07:51:29.696788+0000","last_fullsized":"2024-04-02T07:51:29.696788+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:27.390005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.722338+0000","last_change":"2024-04-02T07:51:29.719782+0000","last_active":"2024-04-02T07:51:29.722338+0000","last_peered":"2024-04-02T07:51:29.722338+0000","last_clean":"2024-04-02T07:51:29.722338+0000","last_became_active":"2024-04-02T07:51:29.719509+0000","last_became_peered":"2024-04-02T07:51:29.719509+0000","last_unstale":"2024-04-02T07:51:29.722338+0000","last_undegraded":"2024-04-02T07:51:29.722338+0000","last_fullsized":"2024-04-02T07:51:29.722338+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:59:28.338083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"82'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.701720+0000","last_change":"2024-04-02T07:51:29.204446+0000","last_active":"2024-04-02T07:51:30.701720+0000","last_peered":"2024-04-02T07:51:30.701720+0000","last_clean":"2024-04-02T07:51:30.701720+0000","last_became_active":"2024-04-02T07:51:29.203629+0000","last_became_peered":"2024-04-02T07:51:29.203629+0000","last_unstale":"2024-04-02T07:51:30.701720+0000","last_undegraded":"2024-04-02T07:51:30.701720+0000","last_fullsized":"2024-04-02T07:51:30.701720+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:07.724301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.714764+0000","last_change":"2024-04-02T07:51:29.707646+0000","last_active":"2024-04-02T07:51:29.714764+0000","last_peered":"2024-04-02T07:51:29.714764+0000","last_clean":"2024-04-02T07:51:29.714764+0000","last_became_active":"2024-04-02T07:51:29.706390+0000","last_became_peered":"2024-04-02T07:51:29.706390+0000","last_unstale":"2024-04-02T07:51:29.714764+0000","last_undegraded":"2024-04-02T07:51:29.714764+0000","last_fullsized":"2024-04-02T07:51:29.714764+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:40:37.534515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.695743+0000","last_change":"2024-04-02T07:51:29.208807+0000","last_active":"2024-04-02T07:51:29.695743+0000","last_peered":"2024-04-02T07:51:29.695743+0000","last_clean":"2024-04-02T07:51:29.695743+0000","last_became_active":"2024-04-02T07:51:29.208501+0000","last_became_peered":"2024-04-02T07:51:29.208501+0000","last_unstale":"2024-04-02T07:51:29.695743+0000","last_undegraded":"2024-04-02T07:51:29.695743+0000","last_fullsized":"2024-04-02T07:51:29.695743+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:56:24.454573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.706743+0000","last_change":"2024-04-02T07:51:29.704031+0000","last_active":"2024-04-02T07:51:29.706743+0000","last_peered":"2024-04-02T07:51:29.706743+0000","last_clean":"2024-04-02T07:51:29.706743+0000","last_became_active":"2024-04-02T07:51:29.703743+0000","last_became_peered":"2024-04-02T07:51:29.703743+0000","last_unstale":"2024-04-02T07:51:29.706743+0000","last_undegraded":"2024-04-02T07:51:29.706743+0000","last_fullsized":"2024-04-02T07:51:29.706743+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:36:52.283458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.698123+0000","last_change":"2024-04-02T07:51:29.209527+0000","last_active":"2024-04-02T07:51:29.698123+0000","last_peered":"2024-04-02T07:51:29.698123+0000","last_clean":"2024-04-02T07:51:29.698123+0000","last_became_active":"2024-04-02T07:51:29.207961+0000","last_became_peered":"2024-04-02T07:51:29.207961+0000","last_unstale":"2024-04-02T07:51:29.698123+0000","last_undegraded":"2024-04-02T07:51:29.698123+0000","last_fullsized":"2024-04-02T07:51:29.698123+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:10:14.951279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103489+0000","last_change":"2024-04-02T07:51:29.717488+0000","last_active":"2024-04-02T07:51:31.103489+0000","last_peered":"2024-04-02T07:51:31.103489+0000","last_clean":"2024-04-02T07:51:31.103489+0000","last_became_active":"2024-04-02T07:51:29.717204+0000","last_became_peered":"2024-04-02T07:51:29.717204+0000","last_unstale":"2024-04-02T07:51:31.103489+0000","last_undegraded":"2024-04-02T07:51:31.103489+0000","last_fullsized":"2024-04-02T07:51:31.103489+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:59:28.338083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"82'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702742+0000","last_change":"2024-04-02T07:51:29.210933+0000","last_active":"2024-04-02T07:51:30.702742+0000","last_peered":"2024-04-02T07:51:30.702742+0000","last_clean":"2024-04-02T07:51:30.702742+0000","last_became_active":"2024-04-02T07:51:29.210685+0000","last_became_peered":"2024-04-02T07:51:29.210685+0000","last_unstale":"2024-04-02T07:51:30.702742+0000","last_undegraded":"2024-04-02T07:51:30.702742+0000","last_fullsized":"2024-04-02T07:51:30.702742+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:10.744141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103549+0000","last_change":"2024-04-02T07:51:29.706642+0000","last_active":"2024-04-02T07:51:31.103549+0000","last_peered":"2024-04-02T07:51:31.103549+0000","last_clean":"2024-04-02T07:51:31.103549+0000","last_became_active":"2024-04-02T07:51:29.705512+0000","last_became_peered":"2024-04-02T07:51:29.705512+0000","last_unstale":"2024-04-02T07:51:31.103549+0000","last_undegraded":"2024-04-02T07:51:31.103549+0000","last_fullsized":"2024-04-02T07:51:31.103549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:18:41.743679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.693035+0000","last_change":"2024-04-02T07:51:29.212027+0000","last_active":"2024-04-02T07:51:29.693035+0000","last_peered":"2024-04-02T07:51:29.693035+0000","last_clean":"2024-04-02T07:51:29.693035+0000","last_became_active":"2024-04-02T07:51:29.211739+0000","last_became_peered":"2024-04-02T07:51:29.211739+0000","last_unstale":"2024-04-02T07:51:29.693035+0000","last_undegraded":"2024-04-02T07:51:29.693035+0000","last_fullsized":"2024-04-02T07:51:29.693035+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:45:44.289973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702411+0000","last_change":"2024-04-02T07:51:29.718952+0000","last_active":"2024-04-02T07:51:30.702411+0000","last_peered":"2024-04-02T07:51:30.702411+0000","last_clean":"2024-04-02T07:51:30.702411+0000","last_became_active":"2024-04-02T07:51:29.714973+0000","last_became_peered":"2024-04-02T07:51:29.714973+0000","last_unstale":"2024-04-02T07:51:30.702411+0000","last_undegraded":"2024-04-02T07:51:30.702411+0000","last_fullsized":"2024-04-02T07:51:30.702411+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:08.732683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.697582+0000","last_change":"2024-04-02T07:51:29.201725+0000","last_active":"2024-04-02T07:51:29.697582+0000","last_peered":"2024-04-02T07:51:29.697582+0000","last_clean":"2024-04-02T07:51:29.697582+0000","last_became_active":"2024-04-02T07:51:29.201438+0000","last_became_peered":"2024-04-02T07:51:29.201438+0000","last_unstale":"2024-04-02T07:51:29.697582+0000","last_undegraded":"2024-04-02T07:51:29.697582+0000","last_fullsized":"2024-04-02T07:51:29.697582+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:03.852318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.707292+0000","last_change":"2024-04-02T07:51:29.704125+0000","last_active":"2024-04-02T07:51:29.707292+0000","last_peered":"2024-04-02T07:51:29.707292+0000","last_clean":"2024-04-02T07:51:29.707292+0000","last_became_active":"2024-04-02T07:51:29.703842+0000","last_became_peered":"2024-04-02T07:51:29.703842+0000","last_unstale":"2024-04-02T07:51:29.707292+0000","last_undegraded":"2024-04-02T07:51:29.707292+0000","last_fullsized":"2024-04-02T07:51:29.707292+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:24:50.492280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103273+0000","last_change":"2024-04-02T07:51:29.205685+0000","last_active":"2024-04-02T07:51:31.103273+0000","last_peered":"2024-04-02T07:51:31.103273+0000","last_clean":"2024-04-02T07:51:31.103273+0000","last_became_active":"2024-04-02T07:51:29.204332+0000","last_became_peered":"2024-04-02T07:51:29.204332+0000","last_unstale":"2024-04-02T07:51:31.103273+0000","last_undegraded":"2024-04-02T07:51:31.103273+0000","last_fullsized":"2024-04-02T07:51:31.103273+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:10.744141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.717861+0000","last_change":"2024-04-02T07:51:29.712341+0000","last_active":"2024-04-02T07:51:29.717861+0000","last_peered":"2024-04-02T07:51:29.717861+0000","last_clean":"2024-04-02T07:51:29.717861+0000","last_became_active":"2024-04-02T07:51:29.710914+0000","last_became_peered":"2024-04-02T07:51:29.710914+0000","last_unstale":"2024-04-02T07:51:29.717861+0000","last_undegraded":"2024-04-02T07:51:29.717861+0000","last_fullsized":"2024-04-02T07:51:29.717861+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:02:53.831136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702244+0000","last_change":"2024-04-02T07:51:29.210518+0000","last_active":"2024-04-02T07:51:30.702244+0000","last_peered":"2024-04-02T07:51:30.702244+0000","last_clean":"2024-04-02T07:51:30.702244+0000","last_became_active":"2024-04-02T07:51:29.210225+0000","last_became_peered":"2024-04-02T07:51:29.210225+0000","last_unstale":"2024-04-02T07:51:30.702244+0000","last_undegraded":"2024-04-02T07:51:30.702244+0000","last_fullsized":"2024-04-02T07:51:30.702244+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:53:15.044497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.711523+0000","last_change":"2024-04-02T07:51:29.702103+0000","last_active":"2024-04-02T07:51:29.711523+0000","last_peered":"2024-04-02T07:51:29.711523+0000","last_clean":"2024-04-02T07:51:29.711523+0000","last_became_active":"2024-04-02T07:51:29.701851+0000","last_became_peered":"2024-04-02T07:51:29.701851+0000","last_unstale":"2024-04-02T07:51:29.711523+0000","last_undegraded":"2024-04-02T07:51:29.711523+0000","last_fullsized":"2024-04-02T07:51:29.711523+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:20.749913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:31.103176+0000","last_change":"2024-04-02T07:51:29.208725+0000","last_active":"2024-04-02T07:51:31.103176+0000","last_peered":"2024-04-02T07:51:31.103176+0000","last_clean":"2024-04-02T07:51:31.103176+0000","last_became_active":"2024-04-02T07:51:29.208140+0000","last_became_peered":"2024-04-02T07:51:29.208140+0000","last_unstale":"2024-04-02T07:51:31.103176+0000","last_undegraded":"2024-04-02T07:51:31.103176+0000","last_fullsized":"2024-04-02T07:51:31.103176+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:10:14.951279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.714024+0000","last_change":"2024-04-02T07:51:29.707370+0000","last_active":"2024-04-02T07:51:29.714024+0000","last_peered":"2024-04-02T07:51:29.714024+0000","last_clean":"2024-04-02T07:51:29.714024+0000","last_became_active":"2024-04-02T07:51:29.706978+0000","last_became_peered":"2024-04-02T07:51:29.706978+0000","last_unstale":"2024-04-02T07:51:29.714024+0000","last_undegraded":"2024-04-02T07:51:29.714024+0000","last_fullsized":"2024-04-02T07:51:29.714024+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:02:41.176141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.692278+0000","last_change":"2024-04-02T07:51:29.204430+0000","last_active":"2024-04-02T07:51:29.692278+0000","last_peered":"2024-04-02T07:51:29.692278+0000","last_clean":"2024-04-02T07:51:29.692278+0000","last_became_active":"2024-04-02T07:51:29.204162+0000","last_became_peered":"2024-04-02T07:51:29.204162+0000","last_unstale":"2024-04-02T07:51:29.692278+0000","last_undegraded":"2024-04-02T07:51:29.692278+0000","last_fullsized":"2024-04-02T07:51:29.692278+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:17:59.216495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-02T07:51:30.702451+0000","last_change":"2024-04-02T07:51:29.710211+0000","last_active":"2024-04-02T07:51:30.702451+0000","last_peered":"2024-04-02T07:51:30.702451+0000","last_clean":"2024-04-02T07:51:30.702451+0000","last_became_active":"2024-04-02T07:51:29.709902+0000","last_became_peered":"2024-04-02T07:51:29.709902+0000","last_unstale":"2024-04-02T07:51:30.702451+0000","last_undegraded":"2024-04-02T07:51:30.702451+0000","last_fullsized":"2024-04-02T07:51:30.702451+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:21:07.376954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.696398+0000","last_change":"2024-04-02T07:51:29.206364+0000","last_active":"2024-04-02T07:51:29.696398+0000","last_peered":"2024-04-02T07:51:29.696398+0000","last_clean":"2024-04-02T07:51:29.696398+0000","last_became_active":"2024-04-02T07:51:29.206071+0000","last_became_peered":"2024-04-02T07:51:29.206071+0000","last_unstale":"2024-04-02T07:51:29.696398+0000","last_undegraded":"2024-04-02T07:51:29.696398+0000","last_fullsized":"2024-04-02T07:51:29.696398+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:59:00.134245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.707423+0000","last_change":"2024-04-02T07:51:29.703939+0000","last_active":"2024-04-02T07:51:29.707423+0000","last_peered":"2024-04-02T07:51:29.707423+0000","last_clean":"2024-04-02T07:51:29.707423+0000","last_became_active":"2024-04-02T07:51:29.703313+0000","last_became_peered":"2024-04-02T07:51:29.703313+0000","last_unstale":"2024-04-02T07:51:29.707423+0000","last_undegraded":"2024-04-02T07:51:29.707423+0000","last_fullsized":"2024-04-02T07:51:29.707423+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:29.181007+0000","last_clean_scrub_stamp":"2024-04-02T07:51:29.181007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:19:28.826550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.689168+0000","last_change":"2024-04-02T07:51:29.211829+0000","last_active":"2024-04-02T07:51:29.689168+0000","last_peered":"2024-04-02T07:51:29.689168+0000","last_clean":"2024-04-02T07:51:29.689168+0000","last_became_active":"2024-04-02T07:51:29.211549+0000","last_became_peered":"2024-04-02T07:51:29.211549+0000","last_unstale":"2024-04-02T07:51:29.689168+0000","last_undegraded":"2024-04-02T07:51:29.689168+0000","last_fullsized":"2024-04-02T07:51:29.689168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:51:28.172625+0000","last_clean_scrub_stamp":"2024-04-02T07:51:28.172625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:04:56.944982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"72'161","reported_seq":381,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-02T07:51:29.190340+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:51:29.190340+0000","last_peered":"2024-04-02T07:51:29.190340+0000","last_clean":"2024-04-02T07:51:29.190340+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:51:29.190340+0000","last_undegraded":"2024-04-02T07:51:29.190340+0000","last_fullsized":"2024-04-02T07:51:29.190340+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575037,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":999424,"data_stored":478768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66800000000000004,"15min":0.67800000000000005},"min":{"1min":0.39400000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.67800000000000005},"min":{"1min":0.47399999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.95699999999999996,"5min":1.0109999999999999,"15min":1.024},"last":0.68300000000000005}]},{"osd":1,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.67300000000000004,"15min":0.68000000000000005},"min":{"1min":0.499,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.89100000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66500000000000004,"15min":0.67500000000000004},"min":{"1min":0.42399999999999999,"5min":0.35599999999999998,"15min":0.34699999999999998},"max":{"1min":0.88300000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.71699999999999997,"15min":0.71099999999999997},"min":{"1min":0.50700000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.94199999999999995,"5min":1.127,"15min":1.127},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.753,"15min":0.76000000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.434},"max":{"1min":0.93799999999999994,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.64200000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.69599999999999995,"15min":0.70399999999999996},"min":{"1min":0.44,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.97799999999999998,"5min":1.026,"15min":1.274},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.52600000000000002,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.90600000000000003,"5min":1.196,"15min":1.196},"last":0.58899999999999997}]},{"osd":4,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.376,"5min":0.32000000000000001,"15min":0.30199999999999999},"max":{"1min":0.84099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68200000000000005,"15min":0.68700000000000006},"min":{"1min":0.51800000000000002,"5min":0.40000000000000002,"15min":0.38100000000000001},"max":{"1min":0.84599999999999997,"5min":1.0669999999999999,"15min":1.4890000000000001},"last":0.76100000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.46100000000000002,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":0.86299999999999999,"5min":1.167,"15min":1.167},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68500000000000005,"15min":0.68700000000000006},"min":{"1min":0.51100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.69799999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67600000000000005,"15min":0.65800000000000003},"min":{"1min":0.505,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.877,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66000000000000003,"15min":0.65300000000000002},"min":{"1min":0.47999999999999998,"5min":0.42299999999999999,"15min":0.39600000000000002},"max":{"1min":0.89300000000000002,"5min":1.0509999999999999,"15min":1.329},"last":0.66500000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":995328,"data_stored":478698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.51300000000000001,"5min":0.38700000000000001,"15min":0.32400000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.98099999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.749,"5min":0.71599999999999997,"15min":0.70899999999999996},"min":{"1min":0.55200000000000005,"5min":0.38300000000000001,"15min":0.375},"max":{"1min":0.89400000000000002,"5min":0.96899999999999997,"15min":1.0249999999999999},"last":0.56799999999999995}]},{"osd":1,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69399999999999995,"15min":0.69599999999999995},"min":{"1min":0.53900000000000003,"5min":0.46500000000000002,"15min":0.41199999999999998},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":1.085},"last":0.745},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.56999999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.84399999999999997}]},{"osd":2,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73099999999999998,"15min":0.71499999999999997},"min":{"1min":0.48999999999999999,"5min":0.45500000000000002,"15min":0.38200000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71799999999999997,"15min":0.70699999999999996},"min":{"1min":0.53300000000000003,"5min":0.51400000000000001,"15min":0.45900000000000002},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96599999999999997},"last":0.82499999999999996}]},{"osd":3,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.73699999999999999,"15min":0.73899999999999999},"min":{"1min":0.55400000000000005,"5min":0.5,"15min":0.47099999999999997},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.748,"15min":0.73999999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.45300000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.0309999999999999},"last":0.80600000000000005}]},{"osd":4,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61399999999999999,"15min":0.60499999999999998},"min":{"1min":0.39300000000000002,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.64000000000000001,"15min":0.629},"min":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.35499999999999998},"max":{"1min":0.93200000000000005,"5min":0.99299999999999999,"15min":1.0049999999999999},"last":0.86699999999999999}]},{"osd":5,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.629,"15min":0.621},"min":{"1min":0.41599999999999998,"5min":0.379,"15min":0.35799999999999998},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":1.1599999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.67200000000000004,"15min":0.68600000000000005},"min":{"1min":0.504,"5min":0.42099999999999999,"15min":0.40300000000000002},"max":{"1min":0.93899999999999995,"5min":1.01,"15min":1.095},"last":0.64100000000000001}]},{"osd":7,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.68899999999999995,"15min":0.68600000000000005},"min":{"1min":0.48099999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.96199999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.45200000000000001,"5min":0.44,"15min":0.42099999999999999},"max":{"1min":0.92500000000000004,"5min":1.0149999999999999,"15min":1.4399999999999999},"last":0.60799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575036,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36768,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335072,"statfs":{"total":96636764160,"available":96599113728,"internally_reserved":0,"allocated":1589248,"data_stored":1069626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.432,"15min":0.44},"min":{"1min":0.29499999999999998,"5min":0.27500000000000002,"15min":0.25700000000000001},"max":{"1min":0.58899999999999997,"5min":0.77600000000000002,"15min":2.9079999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.50700000000000001,"15min":0.504},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.70699999999999996,"5min":0.86499999999999999,"15min":2.766},"last":0.46700000000000003}]},{"osd":2,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.627,"5min":0.71599999999999997,"15min":2.0750000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.443,"5min":0.36699999999999999,"15min":0.35699999999999998},"max":{"1min":0.71499999999999997,"5min":0.85399999999999998,"15min":1.7410000000000001},"last":0.51200000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.41799999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72399999999999998,"5min":0.81799999999999995,"15min":2.9430000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68000000000000005,"5min":0.82699999999999996,"15min":2.802},"last":0.52000000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.78300000000000003,"15min":0.77400000000000002},"min":{"1min":0.67400000000000004,"5min":0.46700000000000003,"15min":0.46300000000000002},"max":{"1min":1.0609999999999999,"5min":1.861,"15min":1.861},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.77700000000000002,"15min":0.76800000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46500000000000002},"max":{"1min":1.232,"5min":1.232,"15min":2.0419999999999998},"last":0.64100000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.77900000000000003,"15min":0.77100000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.41999999999999998},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.9690000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79400000000000004,"15min":0.78300000000000003},"min":{"1min":0.65600000000000003,"5min":0.51000000000000001,"15min":0.48799999999999999},"max":{"1min":1.147,"5min":1.54,"15min":1.54},"last":0.71699999999999997}]},{"osd":6,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.79900000000000004,"15min":0.79500000000000004},"min":{"1min":0.53200000000000003,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.097,"5min":1.577,"15min":1.768},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.80400000000000005,"15min":0.79100000000000004},"min":{"1min":0.61399999999999999,"5min":0.51400000000000001,"15min":0.38100000000000001},"max":{"1min":1.351,"5min":1.351,"15min":2.012},"last":0.77800000000000002}]},{"osd":7,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.80900000000000005,"15min":0.80300000000000005},"min":{"1min":0.628,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.298,"5min":1.552,"15min":1.706},"last":0.62},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.64300000000000002,"5min":0.53200000000000003,"15min":0.502},"max":{"1min":1.194,"5min":1.2290000000000001,"15min":1.8240000000000001},"last":0.871}]}]},{"osd":0,"up_from":13,"seq":55834575036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":999424,"data_stored":479221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.441,"15min":0.44400000000000001},"min":{"1min":0.29699999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71199999999999997,"5min":0.78900000000000003,"15min":2.1640000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.495,"15min":0.496},"min":{"1min":0.34899999999999998,"5min":0.307,"15min":0.26200000000000001},"max":{"1min":0.70599999999999996,"5min":0.72199999999999998,"15min":2.6469999999999998},"last":0.39800000000000002}]},{"osd":2,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48999999999999999,"15min":0.495},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63100000000000001,"5min":0.749,"15min":2.125},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":2.605},"last":0.52300000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52900000000000003,"15min":0.54800000000000004},"min":{"1min":0.33600000000000002,"5min":0.32700000000000001,"15min":0.28899999999999998},"max":{"1min":0.69799999999999995,"5min":0.79400000000000004,"15min":5.4800000000000004},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.40100000000000002},"max":{"1min":0.83199999999999996,"5min":0.93400000000000005,"15min":2.194},"last":0.65400000000000003}]},{"osd":4,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.77700000000000002,"15min":0.80100000000000005},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.35599999999999998},"max":{"1min":0.97999999999999998,"5min":1.0640000000000001,"15min":6.0899999999999999},"last":0.623},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78400000000000003,"15min":0.79600000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.433},"max":{"1min":1.048,"5min":1.095,"15min":5.7169999999999996},"last":0.70499999999999996}]},{"osd":5,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.81100000000000005,"15min":0.81200000000000006},"min":{"1min":0.55800000000000005,"5min":0.51000000000000001,"15min":0.42099999999999999},"max":{"1min":0.98999999999999999,"5min":2.1869999999999998,"15min":5.7539999999999996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.81799999999999995},"min":{"1min":0.63100000000000001,"5min":0.54100000000000004,"15min":0.51800000000000002},"max":{"1min":0.96299999999999997,"5min":2.6230000000000002,"15min":5.8929999999999998},"last":0.72099999999999997}]},{"osd":6,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.75700000000000001,"15min":0.77100000000000002},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.46800000000000003},"max":{"1min":0.96599999999999997,"5min":1.044,"15min":5.931},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.755,"15min":0.77700000000000002},"min":{"1min":0.64700000000000002,"5min":0.50700000000000001,"15min":0.376},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":6.125},"last":0.68899999999999995}]},{"osd":7,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.83799999999999997},"min":{"1min":0.70299999999999996,"5min":0.51600000000000001,"15min":0.40799999999999997},"max":{"1min":1.0900000000000001,"5min":2.3959999999999999,"15min":6.1580000000000004},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.81399999999999995,"15min":0.83199999999999996},"min":{"1min":0.54100000000000004,"5min":0.48299999999999998,"15min":0.47699999999999998},"max":{"1min":1.0249999999999999,"5min":2.145,"15min":5.7850000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575035,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":1011712,"data_stored":480400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.441,"15min":0.44700000000000001},"min":{"1min":0.27400000000000002,"5min":0.26400000000000001,"15min":0.25700000000000001},"max":{"1min":0.71099999999999997,"5min":0.88,"15min":0.88},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46300000000000002,"15min":0.47399999999999998},"min":{"1min":0.33800000000000002,"5min":0.27900000000000003,"15min":0.26200000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.84299999999999997},"last":0.41299999999999998}]},{"osd":1,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.49399999999999999},"min":{"1min":0.32700000000000001,"5min":0.30599999999999999,"15min":0.30299999999999999},"max":{"1min":0.81899999999999995,"5min":0.85699999999999998,"15min":0.93999999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51800000000000002,"15min":0.52300000000000002},"min":{"1min":0.40799999999999997,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.90000000000000002},"last":0.45900000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.52600000000000002,"15min":0.53800000000000003},"min":{"1min":0.38500000000000001,"5min":0.316,"15min":0.27800000000000002},"max":{"1min":0.79000000000000004,"5min":0.79800000000000004,"15min":1.4339999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.60499999999999998,"15min":0.61199999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39000000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":1.3740000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.76600000000000001,"15min":0.76900000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.47199999999999998},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.157},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.755,"15min":0.77000000000000002},"min":{"1min":0.56399999999999995,"5min":0.51600000000000001,"15min":0.48499999999999999},"max":{"1min":0.999,"5min":1.075,"15min":1.179},"last":0.76100000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.754,"15min":0.75800000000000001},"min":{"1min":0.57799999999999996,"5min":0.54300000000000004,"15min":0.42199999999999999},"max":{"1min":1.004,"5min":1.0229999999999999,"15min":1.2},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.76700000000000002,"15min":0.78000000000000003},"min":{"1min":0.59599999999999997,"5min":0.53800000000000003,"15min":0.50900000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1899999999999999},"last":0.73199999999999998}]},{"osd":6,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.78700000000000003,"15min":0.78600000000000003},"min":{"1min":0.60399999999999998,"5min":0.53600000000000003,"15min":0.442},"max":{"1min":1.0149999999999999,"5min":1.0880000000000001,"15min":1.175},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.76300000000000001,"15min":0.78700000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.018,"5min":1.0620000000000001,"15min":1.2130000000000001},"last":0.79300000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78200000000000003,"15min":0.79500000000000004},"min":{"1min":0.57299999999999995,"5min":0.55900000000000005,"15min":0.5},"max":{"1min":1.0469999999999999,"5min":1.054,"15min":1.1779999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.79400000000000004},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.52900000000000003},"max":{"1min":1.022,"5min":1.0920000000000001,"15min":1.145},"last":0.82099999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36192,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335648,"statfs":{"total":96636764160,"available":96599703552,"internally_reserved":0,"allocated":995328,"data_stored":478766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.42699999999999999,"15min":0.438},"min":{"1min":0.32000000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.748,"5min":0.748,"15min":2.476},"last":0.314},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44700000000000001,"15min":0.47899999999999998},"min":{"1min":0.34499999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.61099999999999999,"5min":0.70799999999999996,"15min":2.04},"last":0.373}]},{"osd":1,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47999999999999998,"15min":0.497},"min":{"1min":0.32700000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.61599999999999999,"5min":1.2809999999999999,"15min":2.2839999999999998},"last":0.438},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55700000000000005,"15min":0.56399999999999995},"min":{"1min":0.45100000000000001,"5min":0.35799999999999998,"15min":0.34200000000000003},"max":{"1min":0.77500000000000002,"5min":0.86299999999999999,"15min":2.2240000000000002},"last":0.49199999999999999}]},{"osd":2,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.60299999999999998,"15min":0.61299999999999999},"min":{"1min":0.49299999999999999,"5min":0.38700000000000001,"15min":0.376},"max":{"1min":0.84099999999999997,"5min":0.92100000000000004,"15min":1.7310000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58899999999999997,"15min":0.62},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.35699999999999998},"max":{"1min":0.87,"5min":1.244,"15min":3.9449999999999998},"last":0.61899999999999999}]},{"osd":4,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.72199999999999998,"15min":0.74299999999999999},"min":{"1min":0.499,"5min":0.436,"15min":0.38400000000000001},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":2.0030000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.73999999999999999,"15min":0.75900000000000001},"min":{"1min":0.52500000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89800000000000002,"5min":1.02,"15min":1.5249999999999999},"last":0.96499999999999997}]},{"osd":5,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.73499999999999999,"15min":0.754},"min":{"1min":0.51700000000000002,"5min":0.41699999999999998,"15min":0.40899999999999997},"max":{"1min":0.91800000000000004,"5min":1.079,"15min":1.2909999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.72699999999999998,"15min":0.75800000000000001},"min":{"1min":0.55100000000000005,"5min":0.48899999999999999,"15min":0.40200000000000002},"max":{"1min":0.95799999999999996,"5min":1.0449999999999999,"15min":1.9159999999999999},"last":0.94199999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.73499999999999999,"15min":0.76400000000000001},"min":{"1min":0.56599999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.95099999999999996,"5min":1.0620000000000001,"15min":1.548},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76200000000000001,"15min":0.77900000000000003},"min":{"1min":0.58499999999999996,"5min":0.54800000000000004,"15min":0.53500000000000003},"max":{"1min":1.0209999999999999,"5min":1.1140000000000001,"15min":1.4379999999999999},"last":0.77900000000000003}]},{"osd":7,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.76100000000000001,"15min":0.78200000000000003},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.439},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.966},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.76200000000000001,"15min":0.78900000000000003},"min":{"1min":0.57199999999999995,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.96899999999999997,"5min":1.181,"15min":1.514},"last":1.0089999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36196,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335644,"statfs":{"total":96636764160,"available":96599699456,"internally_reserved":0,"allocated":999424,"data_stored":479292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67700000000000005,"15min":0.66300000000000003},"min":{"1min":0.436,"5min":0.38100000000000001,"15min":0.32100000000000001},"max":{"1min":0.879,"5min":0.94299999999999995,"15min":1.0189999999999999},"last":0.498},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68300000000000005,"15min":0.67400000000000004},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.35299999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":1.7849999999999999},"last":0.69099999999999995}]},{"osd":1,"last update":"Tue Apr 2 07:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.73199999999999998,"15min":0.72399999999999998},"min":{"1min":0.53400000000000003,"5min":0.438,"15min":0.42699999999999999},"max":{"1min":0.90900000000000003,"5min":0.97299999999999998,"15min":1.036},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72699999999999998,"15min":0.73399999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.95099999999999996,"5min":0.97899999999999998,"15min":0.99399999999999999},"last":0.64700000000000002}]},{"osd":2,"last update":"Tue Apr 2 07:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.68600000000000005,"15min":0.68799999999999994},"min":{"1min":0.51900000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91500000000000004,"5min":0.93500000000000005,"15min":1.03},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.71599999999999997,"15min":0.71499999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.41899999999999998},"max":{"1min":0.91600000000000004,"5min":0.97199999999999998,"15min":1.6459999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Tue Apr 2 07:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.72999999999999998,"15min":0.71999999999999997},"min":{"1min":0.53700000000000003,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":0.90900000000000003,"5min":0.95699999999999996,"15min":1.002},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.62,"5min":0.48799999999999999,"15min":0.41499999999999998},"max":{"1min":0.82999999999999996,"5min":0.96399999999999997,"15min":0.96999999999999997},"last":0.58399999999999996}]},{"osd":5,"last update":"Tue Apr 2 07:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56299999999999994,"15min":0.57699999999999996},"min":{"1min":0.38600000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63200000000000001,"5min":0.84799999999999998,"15min":1.034},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.58999999999999997},"min":{"1min":0.41399999999999998,"5min":0.38800000000000001,"15min":0.33700000000000002},"max":{"1min":0.875,"5min":0.89700000000000002,"15min":1.0129999999999999},"last":0.51800000000000002}]},{"osd":6,"last update":"Tue Apr 2 07:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59799999999999998,"15min":0.60799999999999998},"min":{"1min":0.48599999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.83099999999999996,"5min":0.89200000000000002,"15min":1.2589999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.64900000000000002,"15min":0.65300000000000002},"min":{"1min":0.498,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.78200000000000003,"5min":0.93100000000000005,"15min":1.0609999999999999},"last":0.68000000000000005}]},{"osd":7,"last update":"Tue Apr 2 07:51:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68700000000000006,"15min":0.67200000000000004},"min":{"1min":0.50700000000000001,"5min":0.432,"15min":0.38300000000000001},"max":{"1min":0.92900000000000005,"5min":0.95199999999999996,"15min":1.0680000000000001},"last":0.63},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.745,"15min":0.73399999999999999},"min":{"1min":0.58999999999999997,"5min":0.47799999999999998,"15min":0.434},"max":{"1min":0.88200000000000001,"5min":0.999,"15min":1.502},"last":0.74199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575038,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32672,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":1585152,"data_stored":1068059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67800000000000005,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.80900000000000005,"5min":0.95299999999999996,"15min":0.99199999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.878,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.71899999999999997}]},{"osd":1,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67600000000000005,"15min":0.69399999999999995},"min":{"1min":0.41999999999999998,"5min":0.40300000000000002,"15min":0.40200000000000002},"max":{"1min":1.0620000000000001,"5min":1.131,"15min":1.131},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.45000000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83999999999999997,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.69299999999999995}]},{"osd":2,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70699999999999996,"15min":0.71299999999999997},"min":{"1min":0.47399999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.878,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.69399999999999995,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.47099999999999997,"15min":0.372},"max":{"1min":1.111,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.72899999999999998,"15min":0.74299999999999999},"min":{"1min":0.54800000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92300000000000004,"5min":1.341,"15min":1.341},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77000000000000002,"15min":0.76600000000000001},"min":{"1min":0.60699999999999998,"5min":0.54600000000000004,"15min":0.44900000000000001},"max":{"1min":1.1839999999999999,"5min":1.484,"15min":1.484},"last":0.504}]},{"osd":4,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59299999999999997,"15min":0.59999999999999998},"min":{"1min":0.39500000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.35599999999999998},"max":{"1min":0.96199999999999997,"5min":1.504,"15min":1.504},"last":0.628}]},{"osd":6,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.64900000000000002,"15min":0.64300000000000002},"min":{"1min":0.45600000000000002,"5min":0.44700000000000001,"15min":0.39900000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.69199999999999995,"15min":0.67700000000000005},"min":{"1min":0.45500000000000002,"5min":0.44900000000000001,"15min":0.41199999999999998},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":0.65100000000000002}]},{"osd":7,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.71399999999999997,"15min":0.70299999999999996},"min":{"1min":0.502,"5min":0.47599999999999998,"15min":0.38900000000000001},"max":{"1min":1.0129999999999999,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.754,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.53600000000000003,"5min":0.46300000000000002,"15min":0.434},"max":{"1min":1.149,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.72999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:51:34.557 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T07:51:34.557 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T07:51:34.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.789+0000 7f3783600700 1 Processor -- start 2024-04-02T07:51:34.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.789+0000 7f3783600700 1 -- start start 2024-04-02T07:51:34.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.790+0000 7f3783600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f377c1611b0 0x7f377c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:34.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.790+0000 7f3783600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f377c0cee10 0x7f377c161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:34.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.790+0000 7f3783600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377c0ce270 con 0x7f377c15f4d0 2024-04-02T07:51:34.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.790+0000 7f3783600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f377c15b600 con 0x7f377c1611b0 2024-04-02T07:51:34.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.790+0000 7f3783600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0d2760 con 0x7f377c0cee10 2024-04-02T07:51:34.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.791+0000 7f3780c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f377c1611b0 0x7f377c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:34.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.791+0000 7f3780c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f377c1611b0 0x7f377c15ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44690/0 (socket says 172.21.15.89:44690) 2024-04-02T07:51:34.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.791+0000 7f3780c00700 1 -- 172.21.15.89:0/820746460 learned_addr learned my addr 172.21.15.89:0/820746460 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:34.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.791+0000 7f3780c00700 1 -- 172.21.15.89:0/820746460 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f377c0cee10 msgr2=0x7f377c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:34.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.791+0000 7f3780c00700 1 --2- 172.21.15.89:0/820746460 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f377c0cee10 0x7f377c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.791+0000 7f3780c00700 1 -- 172.21.15.89:0/820746460 >> v1:172.21.15.113:6789/0 conn(0x7f377c15f4d0 legacy=0x7f377c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:34.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.791+0000 7f3780c00700 1 -- 172.21.15.89:0/820746460 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c0d2a40 con 0x7f377c1611b0 2024-04-02T07:51:34.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.791+0000 7f3780c00700 1 --2- 172.21.15.89:0/820746460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f377c1611b0 0x7f377c15ef90 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f37700098f0 tx=0x7f3770009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c4b88cc14e8a39 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:34.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.791+0000 7f377b400700 1 -- 172.21.15.89:0/820746460 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3770004230 con 0x7f377c1611b0 2024-04-02T07:51:34.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.792+0000 7f377b400700 1 -- 172.21.15.89:0/820746460 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3770004390 con 0x7f377c1611b0 2024-04-02T07:51:34.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.792+0000 7f377b400700 1 -- 172.21.15.89:0/820746460 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37700036a0 con 0x7f377c1611b0 2024-04-02T07:51:34.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.792+0000 7f3783600700 1 -- 172.21.15.89:0/820746460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f377c1611b0 msgr2=0x7f377c15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:34.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.792+0000 7f3783600700 1 --2- 172.21.15.89:0/820746460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f377c1611b0 0x7f377c15ef90 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f37700098f0 tx=0x7f3770009c00 comp rx=0 tx=0).stop 2024-04-02T07:51:34.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.792+0000 7f3783600700 1 -- 172.21.15.89:0/820746460 shutdown_connections 2024-04-02T07:51:34.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.792+0000 7f3783600700 1 --2- 172.21.15.89:0/820746460 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f377c0cee10 0x7f377c161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.792+0000 7f3783600700 1 --2- 172.21.15.89:0/820746460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f377c1611b0 0x7f377c15ef90 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.792+0000 7f3783600700 1 -- 172.21.15.89:0/820746460 >> 172.21.15.89:0/820746460 conn(0x7f377c08aff0 msgr2=0x7f377c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:34.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.792+0000 7f3783600700 1 -- 172.21.15.89:0/820746460 shutdown_connections 2024-04-02T07:51:34.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.792+0000 7f3783600700 1 -- 172.21.15.89:0/820746460 wait complete. 2024-04-02T07:51:34.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.794+0000 7f3783600700 1 Processor -- start 2024-04-02T07:51:34.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.795+0000 7f3783600700 1 -- start start 2024-04-02T07:51:34.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.796+0000 7f3783600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f377c0cee10 0x7f377c1f3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:34.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.796+0000 7f3783600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f377c15f4d0 0x7f377c06c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:34.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.796+0000 7f3783600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f377c1611b0 0x7f377c0b80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:34.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.796+0000 7f3783600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0b6f90 con 0x7f377c15f4d0 2024-04-02T07:51:34.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.796+0000 7f3783600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0b6150 con 0x7f377c1611b0 2024-04-02T07:51:34.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.796+0000 7f3783600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377c1f37c0 con 0x7f377c0cee10 2024-04-02T07:51:34.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.796+0000 7f377be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f377c15f4d0 0x7f377c06c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:34.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.796+0000 7f377be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f377c15f4d0 0x7f377c06c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44700/0 (socket says 172.21.15.89:44700) 2024-04-02T07:51:34.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.796+0000 7f377be00700 1 -- 172.21.15.89:0/676698550 learned_addr learned my addr 172.21.15.89:0/676698550 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:34.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.796+0000 7f3780c00700 1 --2- 172.21.15.89:0/676698550 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f377c0cee10 0x7f377c1f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:34.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.796+0000 7f3781600700 1 --2- 172.21.15.89:0/676698550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f377c1611b0 0x7f377c0b80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:34.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.797+0000 7f377be00700 1 -- 172.21.15.89:0/676698550 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f377c0cee10 msgr2=0x7f377c1f3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:34.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.797+0000 7f377be00700 1 --2- 172.21.15.89:0/676698550 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f377c0cee10 0x7f377c1f3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.797+0000 7f377be00700 1 -- 172.21.15.89:0/676698550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f377c1611b0 msgr2=0x7f377c0b80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:34.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.797+0000 7f377be00700 1 --2- 172.21.15.89:0/676698550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f377c1611b0 0x7f377c0b80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:34.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.797+0000 7f377be00700 1 -- 172.21.15.89:0/676698550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3770009550 con 0x7f377c15f4d0 2024-04-02T07:51:34.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.797+0000 7f377be00700 1 --2- 172.21.15.89:0/676698550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f377c15f4d0 0x7f377c06c340 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f376800c4a0 tx=0x7f376800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:34.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.797+0000 7f3778c00700 1 -- 172.21.15.89:0/676698550 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37680054e0 con 0x7f377c15f4d0 2024-04-02T07:51:34.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.797+0000 7f3778c00700 1 -- 172.21.15.89:0/676698550 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3768003680 con 0x7f377c15f4d0 2024-04-02T07:51:34.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.797+0000 7f3783600700 1 -- 172.21.15.89:0/676698550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377c0b6410 con 0x7f377c15f4d0 2024-04-02T07:51:34.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.797+0000 7f3778c00700 1 -- 172.21.15.89:0/676698550 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3768003a10 con 0x7f377c15f4d0 2024-04-02T07:51:34.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.797+0000 7f3783600700 1 -- 172.21.15.89:0/676698550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377c0b69d0 con 0x7f377c15f4d0 2024-04-02T07:51:34.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.798+0000 7f3783600700 1 -- 172.21.15.89:0/676698550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f377c1f68e0 con 0x7f377c15f4d0 2024-04-02T07:51:34.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.799+0000 7f3778c00700 1 -- 172.21.15.89:0/676698550 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3768003b70 con 0x7f377c15f4d0 2024-04-02T07:51:34.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.800+0000 7f3778c00700 1 --2- 172.21.15.89:0/676698550 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f375c06adb0 0x7f375c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:34.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.800+0000 7f3778c00700 1 -- 172.21.15.89:0/676698550 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3768088830 con 0x7f377c15f4d0 2024-04-02T07:51:34.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.800+0000 7f3780c00700 1 --2- 172.21.15.89:0/676698550 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f375c06adb0 0x7f375c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:34.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.801+0000 7f3780c00700 1 --2- 172.21.15.89:0/676698550 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f375c06adb0 0x7f375c06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3770009130 tx=0x7f3770008030 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:34.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:34.804+0000 7f3778c00700 1 -- 172.21.15.89:0/676698550 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f377c1f68e0 con 0x7f377c15f4d0 2024-04-02T07:51:35.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.091+0000 7f3783600700 1 -- 172.21.15.89:0/676698550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f377c1f68e0 con 0x7f377c15f4d0 2024-04-02T07:51:35.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.095+0000 7f3778c00700 1 -- 172.21.15.89:0/676698550 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+0 (secure 0 0 0) 0x7f377c1f68e0 con 0x7f377c15f4d0 2024-04-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.098+0000 7f3783600700 1 -- 172.21.15.89:0/676698550 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f375c06adb0 msgr2=0x7f375c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.098+0000 7f3783600700 1 --2- 172.21.15.89:0/676698550 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f375c06adb0 0x7f375c06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3770009130 tx=0x7f3770008030 comp rx=0 tx=0).stop 2024-04-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.098+0000 7f3783600700 1 -- 172.21.15.89:0/676698550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f377c15f4d0 msgr2=0x7f377c06c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.098+0000 7f3783600700 1 --2- 172.21.15.89:0/676698550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f377c15f4d0 0x7f377c06c340 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f376800c4a0 tx=0x7f376800c7b0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.098+0000 7f3783600700 1 -- 172.21.15.89:0/676698550 shutdown_connections 2024-04-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.098+0000 7f3783600700 1 --2- 172.21.15.89:0/676698550 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f375c06adb0 0x7f375c06d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.098+0000 7f3783600700 1 --2- 172.21.15.89:0/676698550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f377c1611b0 0x7f377c0b80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.098+0000 7f3783600700 1 --2- 172.21.15.89:0/676698550 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f377c0cee10 0x7f377c1f3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.098+0000 7f3783600700 1 --2- 172.21.15.89:0/676698550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f377c15f4d0 0x7f377c06c340 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.098+0000 7f3783600700 1 -- 172.21.15.89:0/676698550 >> 172.21.15.89:0/676698550 conn(0x7f377c08aff0 msgr2=0x7f377c15f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.098+0000 7f3783600700 1 -- 172.21.15.89:0/676698550 shutdown_connections 2024-04-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.098+0000 7f3783600700 1 -- 172.21.15.89:0/676698550 wait complete. 2024-04-02T07:51:35.101 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T07:51:35.118 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:51:35.176 INFO:tasks.ceph.mds.a.smithi089.stderr:2024-04-02T07:51:35.175+0000 7ff2c8600700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T07:51:35.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.306+0000 7f75a9200700 1 Processor -- start 2024-04-02T07:51:35.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.307+0000 7f75a9200700 1 -- start start 2024-04-02T07:51:35.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.308+0000 7f75a9200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f75a40b3af0 0x7f75a415a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:35.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.308+0000 7f75a9200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75a40709c0 0x7f75a4070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:35.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.308+0000 7f75a9200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a40b2180 con 0x7f75a4072580 2024-04-02T07:51:35.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.308+0000 7f75a9200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a415b100 con 0x7f75a40b3af0 2024-04-02T07:51:35.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.308+0000 7f75a9200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a415b3e0 con 0x7f75a40709c0 2024-04-02T07:51:35.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.308+0000 7f75a1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f75a40b3af0 0x7f75a415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:35.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.308+0000 7f75a1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f75a40b3af0 0x7f75a415a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44706/0 (socket says 172.21.15.89:44706) 2024-04-02T07:51:35.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.308+0000 7f75a1e00700 1 -- 172.21.15.89:0/1251214737 learned_addr learned my addr 172.21.15.89:0/1251214737 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:35.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.308+0000 7f75a1400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f75a4072580 0x7f75a40b2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58780/0 (socket says 172.21.15.89:58780) 2024-04-02T07:51:35.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.308+0000 7f75a1e00700 1 -- 172.21.15.89:0/1251214737 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75a40709c0 msgr2=0x7f75a4070e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:35.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.308+0000 7f75a1e00700 1 --2- 172.21.15.89:0/1251214737 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75a40709c0 0x7f75a4070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.308+0000 7f75a1e00700 1 -- 172.21.15.89:0/1251214737 >> v1:172.21.15.89:6789/0 conn(0x7f75a4072580 legacy=0x7f75a40b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:35.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.309+0000 7f75a1e00700 1 -- 172.21.15.89:0/1251214737 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a40b23f0 con 0x7f75a40b3af0 2024-04-02T07:51:35.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.309+0000 7f75a0a00700 1 -- 172.21.15.89:0/1251214737 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291304797 0 0) 0x7f75a40b2180 con 0x7f75a4072580 2024-04-02T07:51:35.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.309+0000 7f75a1e00700 1 --2- 172.21.15.89:0/1251214737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a40b3af0 0x7f75a415a860 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f759400bb90 tx=0x7f759400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713fcc3c61f63ec server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:35.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.309+0000 7f75a0a00700 1 -- 172.21.15.89:0/1251214737 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759400dd50 con 0x7f75a40b3af0 2024-04-02T07:51:35.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.309+0000 7f75a0a00700 1 -- 172.21.15.89:0/1251214737 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7594010470 con 0x7f75a40b3af0 2024-04-02T07:51:35.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.309+0000 7f75a0a00700 1 -- 172.21.15.89:0/1251214737 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7594010800 con 0x7f75a40b3af0 2024-04-02T07:51:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.309+0000 7f75a9200700 1 -- 172.21.15.89:0/1251214737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a40b3af0 msgr2=0x7f75a415a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.309+0000 7f75a9200700 1 --2- 172.21.15.89:0/1251214737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a40b3af0 0x7f75a415a860 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f759400bb90 tx=0x7f759400bea0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.310+0000 7f75a9200700 1 -- 172.21.15.89:0/1251214737 shutdown_connections 2024-04-02T07:51:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.310+0000 7f75a9200700 1 --2- 172.21.15.89:0/1251214737 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75a40709c0 0x7f75a4070e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.310+0000 7f75a9200700 1 --2- 172.21.15.89:0/1251214737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a40b3af0 0x7f75a415a860 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.310+0000 7f75a9200700 1 -- 172.21.15.89:0/1251214737 >> 172.21.15.89:0/1251214737 conn(0x7f75a40cf190 msgr2=0x7f75a40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.310+0000 7f75a9200700 1 -- 172.21.15.89:0/1251214737 shutdown_connections 2024-04-02T07:51:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.310+0000 7f75a9200700 1 -- 172.21.15.89:0/1251214737 wait complete. 2024-04-02T07:51:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.312+0000 7f75a9200700 1 Processor -- start 2024-04-02T07:51:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.312+0000 7f75a9200700 1 -- start start 2024-04-02T07:51:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.313+0000 7f75a9200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75a40709c0 0x7f75a40c9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:35.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.313+0000 7f75a9200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75a4072580 0x7f75a415c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:35.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.313+0000 7f75a9200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a40b3af0 0x7f75a414e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:35.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.313+0000 7f75a9200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a415ceb0 con 0x7f75a40b3af0 2024-04-02T07:51:35.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.313+0000 7f75a9200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a40c9de0 con 0x7f75a4072580 2024-04-02T07:51:35.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.313+0000 7f75a9200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a40ca0c0 con 0x7f75a40709c0 2024-04-02T07:51:35.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.314+0000 7f75a2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a40b3af0 0x7f75a414e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:35.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.314+0000 7f75a2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a40b3af0 0x7f75a414e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44716/0 (socket says 172.21.15.89:44716) 2024-04-02T07:51:35.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.314+0000 7f75a2800700 1 -- 172.21.15.89:0/3003051766 learned_addr learned my addr 172.21.15.89:0/3003051766 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:35.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.314+0000 7f75a1e00700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75a40709c0 0x7f75a40c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:35.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.314+0000 7f75a1400700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75a4072580 0x7f75a415c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:35.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.314+0000 7f75a2800700 1 -- 172.21.15.89:0/3003051766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75a40709c0 msgr2=0x7f75a40c9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:35.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.314+0000 7f75a2800700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75a40709c0 0x7f75a40c9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.314+0000 7f75a2800700 1 -- 172.21.15.89:0/3003051766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75a4072580 msgr2=0x7f75a415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:35.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.314+0000 7f75a2800700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75a4072580 0x7f75a415c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.314+0000 7f75a2800700 1 -- 172.21.15.89:0/3003051766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759400b7f0 con 0x7f75a40b3af0 2024-04-02T07:51:35.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.314+0000 7f75a1400700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75a4072580 0x7f75a415c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:35.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.314+0000 7f75a2800700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a40b3af0 0x7f75a414e5e0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f759c00b970 tx=0x7f759c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.315+0000 7f759a000700 1 -- 172.21.15.89:0/3003051766 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c00c930 con 0x7f75a40b3af0 2024-04-02T07:51:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.315+0000 7f759a000700 1 -- 172.21.15.89:0/3003051766 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759c00cab0 con 0x7f75a40b3af0 2024-04-02T07:51:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.315+0000 7f759a000700 1 -- 172.21.15.89:0/3003051766 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c00ce60 con 0x7f75a40b3af0 2024-04-02T07:51:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.315+0000 7f75a9200700 1 -- 172.21.15.89:0/3003051766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a40ca3a0 con 0x7f75a40b3af0 2024-04-02T07:51:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.315+0000 7f75a9200700 1 -- 172.21.15.89:0/3003051766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a40ca960 con 0x7f75a40b3af0 2024-04-02T07:51:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.316+0000 7f75a9200700 1 -- 172.21.15.89:0/3003051766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a415a610 con 0x7f75a40b3af0 2024-04-02T07:51:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.317+0000 7f759a000700 1 -- 172.21.15.89:0/3003051766 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f759c01d3b0 con 0x7f75a40b3af0 2024-04-02T07:51:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.318+0000 7f759a000700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f758406adb0 0x7f758406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.318+0000 7f759a000700 1 -- 172.21.15.89:0/3003051766 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f759c089310 con 0x7f75a40b3af0 2024-04-02T07:51:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.318+0000 7f75a1e00700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f758406adb0 0x7f758406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:35.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.318+0000 7f75a1e00700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f758406adb0 0x7f758406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7594000c00 tx=0x7f759401c040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:35.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.322+0000 7f759a000700 1 -- 172.21.15.89:0/3003051766 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f759c054280 con 0x7f75a40b3af0 2024-04-02T07:51:35.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.567+0000 7f75a9200700 1 -- 172.21.15.89:0/3003051766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75a4067720 con 0x7f75a40b3af0 2024-04-02T07:51:35.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.568+0000 7f759a000700 1 -- 172.21.15.89:0/3003051766 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f759c0578a0 con 0x7f75a40b3af0 2024-04-02T07:51:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:35.570 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2025634273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2025634273},{"type":"v1","addr":"172.21.15.113:6834","nonce":2025634273}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2247445394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2247445394},{"type":"v1","addr":"172.21.15.113:6835","nonce":2247445394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1785147820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1785147820},{"type":"v1","addr":"172.21.15.89:6835","nonce":1785147820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:51:29.684122+0000","modified":"2024-04-02T07:51:32.705923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6432":{"gid":6432,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/1479795102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1479795102},{"type":"v1","addr":"172.21.15.89:6837","nonce":1479795102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-02T07:51:35.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.570+0000 7f75a9200700 1 -- 172.21.15.89:0/3003051766 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f758406adb0 msgr2=0x7f758406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:35.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.570+0000 7f75a9200700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f758406adb0 0x7f758406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7594000c00 tx=0x7f759401c040 comp rx=0 tx=0).stop 2024-04-02T07:51:35.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.570+0000 7f75a9200700 1 -- 172.21.15.89:0/3003051766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a40b3af0 msgr2=0x7f75a414e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:35.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.570+0000 7f75a9200700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a40b3af0 0x7f75a414e5e0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f759c00b970 tx=0x7f759c00bc80 comp rx=0 tx=0).stop 2024-04-02T07:51:35.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.570+0000 7f75a9200700 1 -- 172.21.15.89:0/3003051766 shutdown_connections 2024-04-02T07:51:35.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.570+0000 7f75a9200700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f758406adb0 0x7f758406d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.570+0000 7f75a9200700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a40b3af0 0x7f75a414e5e0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.570+0000 7f75a9200700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75a40709c0 0x7f75a40c9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.570+0000 7f75a9200700 1 --2- 172.21.15.89:0/3003051766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75a4072580 0x7f75a415c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:35.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.570+0000 7f75a9200700 1 -- 172.21.15.89:0/3003051766 >> 172.21.15.89:0/3003051766 conn(0x7f75a40cf190 msgr2=0x7f75a40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:35.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.570+0000 7f75a9200700 1 -- 172.21.15.89:0/3003051766 shutdown_connections 2024-04-02T07:51:35.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:35.570+0000 7f75a9200700 1 -- 172.21.15.89:0/3003051766 wait complete. 2024-04-02T07:51:35.573 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 33 2024-04-02T07:51:35.995 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T07:51:29.684122+0000', 'modified': '2024-04-02T07:51:32.705923+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6432}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6432': {'gid': 6432, 'name': 'a', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6837/1479795102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6836', 'nonce': 1479795102}, {'type': 'v1', 'addr': '172.21.15.89:6837', 'nonce': 1479795102}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T07:51:35.996 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T07:51:35.996 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T07:51:36.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.188+0000 7f3ebde00700 1 Processor -- start 2024-04-02T07:51:36.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.188+0000 7f3ebde00700 1 -- start start 2024-04-02T07:51:36.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.189+0000 7f3ebde00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb80d2540 0x7f3eb81687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.189+0000 7f3ebde00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb8161420 con 0x7f3eb815d9d0 2024-04-02T07:51:36.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.189+0000 7f3ebde00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb8161700 con 0x7f3eb80d16a0 2024-04-02T07:51:36.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.189+0000 7f3ebde00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80d4340 con 0x7f3eb80d2540 2024-04-02T07:51:36.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.190+0000 7f3eb6a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3eb815d9d0 0x7f3eb80d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58788/0 (socket says 172.21.15.89:58788) 2024-04-02T07:51:36.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.190+0000 7f3eb6a00700 1 -- 172.21.15.89:0/17361132 learned_addr learned my addr 172.21.15.89:0/17361132 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.190+0000 7f3eb5600700 1 -- 172.21.15.89:0/17361132 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759949991 0 0) 0x7f3eb8161420 con 0x7f3eb815d9d0 2024-04-02T07:51:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.190+0000 7f3eb7400700 1 --2- 172.21.15.89:0/17361132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb80d2540 0x7f3eb81687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.190+0000 7f3eb5600700 1 -- 172.21.15.89:0/17361132 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e9c0035b0 con 0x7f3eb815d9d0 2024-04-02T07:51:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.190+0000 7f3eb5600700 1 -- 172.21.15.89:0/17361132 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 122103111 0 0) 0x7f3e9c0035b0 con 0x7f3eb815d9d0 2024-04-02T07:51:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.190+0000 7f3eb5600700 1 -- 172.21.15.89:0/17361132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb80d2540 msgr2=0x7f3eb81687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.190+0000 7f3eb5600700 1 --2- 172.21.15.89:0/17361132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb80d2540 0x7f3eb81687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.190+0000 7f3eb5600700 1 -- 172.21.15.89:0/17361132 >> v1:172.21.15.113:6789/0 conn(0x7f3eb80d16a0 legacy=0x7f3eb80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.190+0000 7f3eb5600700 1 -- 172.21.15.89:0/17361132 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb81619e0 con 0x7f3eb815d9d0 2024-04-02T07:51:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.190+0000 7f3eb5600700 1 -- 172.21.15.89:0/17361132 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f3ea8001670 con 0x7f3eb815d9d0 2024-04-02T07:51:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.190+0000 7f3eb5600700 1 -- 172.21.15.89:0/17361132 >> v1:172.21.15.89:6789/0 conn(0x7f3eb815d9d0 legacy=0x7f3eb80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.191+0000 7f3eb5600700 1 --2- 172.21.15.89:0/17361132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c003a30 0x7f3e9c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.191+0000 7f3eb5600700 1 --2- 172.21.15.89:0/17361132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb80d2540 0x7f3e9c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.191+0000 7f3eb5600700 1 --2- 172.21.15.89:0/17361132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e9c004d90 0x7f3e9c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.191+0000 7f3eb5600700 1 -- 172.21.15.89:0/17361132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c00baf0 con 0x7f3e9c003a30 2024-04-02T07:51:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.191+0000 7f3eb5600700 1 -- 172.21.15.89:0/17361132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c00bc60 con 0x7f3e9c004d90 2024-04-02T07:51:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.191+0000 7f3eb5600700 1 -- 172.21.15.89:0/17361132 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c00be20 con 0x7f3eb80d2540 2024-04-02T07:51:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.191+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/17361132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c003a30 0x7f3e9c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.191+0000 7f3eb7400700 1 --2- 172.21.15.89:0/17361132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e9c004d90 0x7f3e9c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.191+0000 7f3eb6000700 1 --2- 172.21.15.89:0/17361132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb80d2540 0x7f3e9c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.191+0000 7f3eb6a00700 1 -- 172.21.15.89:0/17361132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb80d2540 msgr2=0x7f3e9c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.191+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/17361132 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb80d2540 0x7f3e9c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.191+0000 7f3eb6a00700 1 -- 172.21.15.89:0/17361132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e9c004d90 msgr2=0x7f3e9c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.191+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/17361132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e9c004d90 0x7f3e9c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.191+0000 7f3eb6a00700 1 -- 172.21.15.89:0/17361132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9c0035b0 con 0x7f3e9c003a30 2024-04-02T07:51:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.192+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/17361132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c003a30 0x7f3e9c003e50 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800c160 tx=0x7f3e9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee86ef447f74fe1b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.192+0000 7f3eb5600700 1 -- 172.21.15.89:0/17361132 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8007b30 con 0x7f3e9c003a30 2024-04-02T07:51:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.192+0000 7f3eb5600700 1 -- 172.21.15.89:0/17361132 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea800cce0 con 0x7f3e9c003a30 2024-04-02T07:51:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.192+0000 7f3eb5600700 1 -- 172.21.15.89:0/17361132 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8005ba0 con 0x7f3e9c003a30 2024-04-02T07:51:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.192+0000 7f3ebde00700 1 -- 172.21.15.89:0/17361132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c003a30 msgr2=0x7f3e9c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.192+0000 7f3ebde00700 1 --2- 172.21.15.89:0/17361132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c003a30 0x7f3e9c003e50 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800c160 tx=0x7f3e9c005880 comp rx=0 tx=0).stop 2024-04-02T07:51:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.192+0000 7f3eb7400700 1 -- 172.21.15.89:0/17361132 reap_dead start 2024-04-02T07:51:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.192+0000 7f3ebde00700 1 -- 172.21.15.89:0/17361132 shutdown_connections 2024-04-02T07:51:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.192+0000 7f3ebde00700 1 -- 172.21.15.89:0/17361132 >> 172.21.15.89:0/17361132 conn(0x7f3eb808aff0 msgr2=0x7f3eb80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.193+0000 7f3ebde00700 1 -- 172.21.15.89:0/17361132 shutdown_connections 2024-04-02T07:51:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.193+0000 7f3ebde00700 1 -- 172.21.15.89:0/17361132 wait complete. 2024-04-02T07:51:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.194+0000 7f3ebde00700 1 Processor -- start 2024-04-02T07:51:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.194+0000 7f3ebde00700 1 -- start start 2024-04-02T07:51:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.195+0000 7f3ebde00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e9c003a30 0x7f3eb81f34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.195+0000 7f3ebde00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb8152330 0x7f3eb815f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.195+0000 7f3ebde00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb815fcd0 0x7f3eb80b4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.195+0000 7f3ebde00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80b6930 con 0x7f3eb8152330 2024-04-02T07:51:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.195+0000 7f3ebde00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80b6020 con 0x7f3eb815fcd0 2024-04-02T07:51:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.195+0000 7f3ebde00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb81601e0 con 0x7f3e9c003a30 2024-04-02T07:51:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.195+0000 7f3eb6000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb8152330 0x7f3eb815f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.195+0000 7f3eb6000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb8152330 0x7f3eb815f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44730/0 (socket says 172.21.15.89:44730) 2024-04-02T07:51:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.195+0000 7f3eb6000700 1 -- 172.21.15.89:0/4210150061 learned_addr learned my addr 172.21.15.89:0/4210150061 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.196+0000 7f3eb7400700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb815fcd0 0x7f3eb80b4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.196+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e9c003a30 0x7f3eb81f34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.196+0000 7f3eb6000700 1 -- 172.21.15.89:0/4210150061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e9c003a30 msgr2=0x7f3eb81f34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.196+0000 7f3eb6000700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e9c003a30 0x7f3eb81f34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.196+0000 7f3eb6000700 1 -- 172.21.15.89:0/4210150061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb815fcd0 msgr2=0x7f3eb80b4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.196+0000 7f3eb6000700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb815fcd0 0x7f3eb80b4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.196+0000 7f3eb6000700 1 -- 172.21.15.89:0/4210150061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb80b6270 con 0x7f3eb8152330 2024-04-02T07:51:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.196+0000 7f3eb7400700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb815fcd0 0x7f3eb80b4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.196+0000 7f3eb6000700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb8152330 0x7f3eb815f790 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400f820 tx=0x7f3ea400fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.196+0000 7f3eaf400700 1 -- 172.21.15.89:0/4210150061 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40052e0 con 0x7f3eb8152330 2024-04-02T07:51:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.196+0000 7f3eaf400700 1 -- 172.21.15.89:0/4210150061 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea4005440 con 0x7f3eb8152330 2024-04-02T07:51:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.196+0000 7f3eaf400700 1 -- 172.21.15.89:0/4210150061 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40055b0 con 0x7f3eb8152330 2024-04-02T07:51:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.196+0000 7f3ebde00700 1 -- 172.21.15.89:0/4210150061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb80b6550 con 0x7f3eb8152330 2024-04-02T07:51:36.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.196+0000 7f3ebde00700 1 -- 172.21.15.89:0/4210150061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb81527b0 con 0x7f3eb8152330 2024-04-02T07:51:36.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.199+0000 7f3eaf400700 1 -- 172.21.15.89:0/4210150061 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3ea4003910 con 0x7f3eb8152330 2024-04-02T07:51:36.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.200+0000 7f3eaf400700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e9806adb0 0x7f3e9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.200+0000 7f3eaf400700 1 -- 172.21.15.89:0/4210150061 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3ea4089520 con 0x7f3eb8152330 2024-04-02T07:51:36.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.200+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e9806adb0 0x7f3e9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.201+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e9806adb0 0x7f3e9806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8015040 tx=0x7f3ea8007000 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:36.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.202+0000 7f3ed187db80 1 Processor -- start 2024-04-02T07:51:36.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.202+0000 7f3ed187db80 1 -- start start 2024-04-02T07:51:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3ed187db80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 0x557457275c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3ed187db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457248be0 0x55745724ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3ed187db80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457246b60 0x557457272a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3ed187db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574571c0650 con 0x557457248be0 2024-04-02T07:51:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3ed187db80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574571a1c30 con 0x557457275830 2024-04-02T07:51:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3ed187db80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55745719f430 con 0x557457246b60 2024-04-02T07:51:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3eb6000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457248be0 0x55745724ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3eb6000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457248be0 0x55745724ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44734/0 (socket says 172.21.15.89:44734) 2024-04-02T07:51:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3eb6000700 1 -- 172.21.15.89:0/2969881745 learned_addr learned my addr 172.21.15.89:0/2969881745 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3eb7400700 1 --2- 172.21.15.89:0/2969881745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457246b60 0x557457272a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/2969881745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 0x557457275c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3eb6000700 1 -- 172.21.15.89:0/2969881745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457246b60 msgr2=0x557457272a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3eb6000700 1 --2- 172.21.15.89:0/2969881745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457246b60 0x557457272a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3eb6000700 1 -- 172.21.15.89:0/2969881745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 msgr2=0x557457275c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3eb6000700 1 --2- 172.21.15.89:0/2969881745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 0x557457275c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3eb6000700 1 -- 172.21.15.89:0/2969881745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557456f8e790 con 0x557457248be0 2024-04-02T07:51:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3eb7400700 1 --2- 172.21.15.89:0/2969881745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457246b60 0x557457272a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.203+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/2969881745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 0x557457275c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.204+0000 7f3eb6000700 1 --2- 172.21.15.89:0/2969881745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457248be0 0x55745724ba10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40050a0 tx=0x7f3ea4057a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c21b76be2b4460 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.204+0000 7f3ea3400700 1 -- 172.21.15.89:0/2969881745 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea405bef0 con 0x557457248be0 2024-04-02T07:51:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.204+0000 7f3ea3400700 1 -- 172.21.15.89:0/2969881745 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea40604b0 con 0x557457248be0 2024-04-02T07:51:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.204+0000 7f3ea3400700 1 -- 172.21.15.89:0/2969881745 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea405e070 con 0x557457248be0 2024-04-02T07:51:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.204+0000 7f3ed187db80 1 -- 172.21.15.89:0/2969881745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457248be0 msgr2=0x55745724ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.204+0000 7f3ed187db80 1 --2- 172.21.15.89:0/2969881745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457248be0 0x55745724ba10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40050a0 tx=0x7f3ea4057a20 comp rx=0 tx=0).stop 2024-04-02T07:51:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.204+0000 7f3ed187db80 1 -- 172.21.15.89:0/2969881745 shutdown_connections 2024-04-02T07:51:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.204+0000 7f3ed187db80 1 --2- 172.21.15.89:0/2969881745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 0x557457275c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.204+0000 7f3ed187db80 1 --2- 172.21.15.89:0/2969881745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457246b60 0x557457272a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.204+0000 7f3ed187db80 1 --2- 172.21.15.89:0/2969881745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457248be0 0x55745724ba10 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.204+0000 7f3ed187db80 1 -- 172.21.15.89:0/2969881745 >> 172.21.15.89:0/2969881745 conn(0x5574571a96a0 msgr2=0x557457290640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.205+0000 7f3ed187db80 1 -- 172.21.15.89:0/2969881745 shutdown_connections 2024-04-02T07:51:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.205+0000 7f3ed187db80 1 -- 172.21.15.89:0/2969881745 wait complete. 2024-04-02T07:51:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.205+0000 7f3ed187db80 1 -- start start 2024-04-02T07:51:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.205+0000 7f3ed187db80 1 Processor -- start 2024-04-02T07:51:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.206+0000 7f3ed187db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457246b60 0x557457295de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.206+0000 7f3ed187db80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 0x557457277e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.206+0000 7f3ed187db80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 0x557457272f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.206+0000 7f3ed187db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55745719ad20 con 0x557457246b60 2024-04-02T07:51:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.206+0000 7f3ed187db80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x557457136770 con 0x557457275830 2024-04-02T07:51:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.206+0000 7f3ed187db80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x5574571596f0 con 0x557457248be0 2024-04-02T07:51:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.207+0000 7f3ed187db80 10 client.0 _ll_register_callbacks cb 0x5574571405e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:51:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.207+0000 7f3eb6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457246b60 0x557457295de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.207+0000 7f3eb6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457246b60 0x557457295de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44748/0 (socket says 172.21.15.89:44748) 2024-04-02T07:51:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.207+0000 7f3eb6a00700 1 -- 172.21.15.89:0/2858892509 learned_addr learned my addr 172.21.15.89:0/2858892509 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.207+0000 7f3eb7400700 1 --2- 172.21.15.89:0/2858892509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 0x557457272f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.207+0000 7f3eb6000700 1 --2- 172.21.15.89:0/2858892509 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 0x557457277e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.207+0000 7f3eb6a00700 1 -- 172.21.15.89:0/2858892509 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 msgr2=0x557457277e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.207+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/2858892509 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 0x557457277e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.207+0000 7f3eb6a00700 1 -- 172.21.15.89:0/2858892509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 msgr2=0x557457272f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.207+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/2858892509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 0x557457272f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.207+0000 7f3eb6a00700 1 -- 172.21.15.89:0/2858892509 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb81527b0 con 0x557457246b60 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.207+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/2858892509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457246b60 0x557457295de0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8021040 tx=0x7f3ea8009050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2926402cf7e64184 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.207+0000 7f3e97e00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.207+0000 7f3e97e00700 1 -- 172.21.15.89:0/2858892509 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80072d0 con 0x557457246b60 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.208+0000 7f3e97e00700 1 -- 172.21.15.89:0/2858892509 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3ea8007430 con 0x557457246b60 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.208+0000 7f3ed187db80 1 -- 172.21.15.89:0/2858892509 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55745723d920 con 0x557457246b60 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.208+0000 7f3e97e00700 1 -- 172.21.15.89:0/2858892509 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ea800cb40 con 0x557457246b60 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.208+0000 7f3ed187db80 10 client.6576 fetch_fsmap learned FSMap version 33 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.208+0000 7f3ed187db80 1 -- 172.21.15.89:0/2858892509 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x55745707b0a0 con 0x557457246b60 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.208+0000 7f3e97e00700 1 -- 172.21.15.89:0/2858892509 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55745723d920 con 0x557457246b60 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.208+0000 7f3e97e00700 1 -- 172.21.15.89:0/2858892509 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3ea8030020 con 0x557457246b60 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.208+0000 7f3ed187db80 10 client.6576 fetch_fsmap finished waiting for FSMap version 33 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.208+0000 7f3ed187db80 10 client.6576 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T07:51:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.208+0000 7f3ed187db80 20 client.6576 populate_metadata read hostname 'smithi089' 2024-04-02T07:51:36.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.208+0000 7f3ed187db80 1 --2- 172.21.15.89:0/2858892509 >> [v2:172.21.15.89:6836/1479795102,v1:172.21.15.89:6837/1479795102] conn(0x5574572734e0 0x557457292550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.208+0000 7f3ed187db80 4 client.6576 mds_command: new command op to 6432 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T07:51:36.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.208+0000 7f3ed187db80 1 -- 172.21.15.89:0/2858892509 --> [v2:172.21.15.89:6836/1479795102,v1:172.21.15.89:6837/1479795102] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55745721a6f0 con 0x5574572734e0 2024-04-02T07:51:36.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.208+0000 7f3eb6000700 1 --2- 172.21.15.89:0/2858892509 >> [v2:172.21.15.89:6836/1479795102,v1:172.21.15.89:6837/1479795102] conn(0x5574572734e0 0x557457292550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.209+0000 7f3e97e00700 10 client.6576 ms_handle_connect on v2:172.21.15.89:6836/1479795102 2024-04-02T07:51:36.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.209+0000 7f3eb6000700 1 --2- 172.21.15.89:0/2858892509 >> [v2:172.21.15.89:6836/1479795102,v1:172.21.15.89:6837/1479795102] conn(0x5574572734e0 0x557457292550 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc612f132f31648f server_cookie=df329f78ec407969 in_seq=0 out_seq=0 2024-04-02T07:51:36.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3e97e00700 1 -- 172.21.15.89:0/2858892509 <== mds.0 v2:172.21.15.89:6836/1479795102 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55745721a6f0 con 0x5574572734e0 2024-04-02T07:51:36.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3e97e00700 10 client.6576 handle_command_reply: tid=0 2024-04-02T07:51:36.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3ed187db80 1 client.6576 shutdown 2024-04-02T07:51:36.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3ed187db80 1 -- 172.21.15.89:0/2858892509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457246b60 msgr2=0x557457295de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3ed187db80 1 --2- 172.21.15.89:0/2858892509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457246b60 0x557457295de0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8021040 tx=0x7f3ea8009050 comp rx=0 tx=0).stop 2024-04-02T07:51:36.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3ed187db80 1 -- 172.21.15.89:0/2858892509 shutdown_connections 2024-04-02T07:51:36.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3ed187db80 1 --2- 172.21.15.89:0/2858892509 >> [v2:172.21.15.89:6836/1479795102,v1:172.21.15.89:6837/1479795102] conn(0x5574572734e0 0x557457292550 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3ed187db80 1 --2- 172.21.15.89:0/2858892509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 0x557457272f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3ed187db80 1 --2- 172.21.15.89:0/2858892509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457246b60 0x557457295de0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3e97e00700 0 client.6576 ms_handle_reset on v2:172.21.15.89:6836/1479795102 2024-04-02T07:51:36.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3ed187db80 1 --2- 172.21.15.89:0/2858892509 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 0x557457277e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3ed187db80 1 -- 172.21.15.89:0/2858892509 >> 172.21.15.89:0/2858892509 conn(0x5574571a96a0 msgr2=0x55745724ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:36.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3ed187db80 1 -- 172.21.15.89:0/2858892509 shutdown_connections 2024-04-02T07:51:36.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3ed187db80 1 -- 172.21.15.89:0/2858892509 wait complete. 2024-04-02T07:51:36.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.210+0000 7f3ed187db80 20 client.6576 trim_cache size 0 max 16384 2024-04-02T07:51:36.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.228+0000 7f3ed187db80 1 Processor -- start 2024-04-02T07:51:36.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.228+0000 7f3ed187db80 1 -- start start 2024-04-02T07:51:36.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.229+0000 7f3ed187db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457246b60 0x55745728d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.229+0000 7f3ed187db80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 0x557457292cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.229+0000 7f3ed187db80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 0x557457274020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.229+0000 7f3ed187db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557456f8f390 con 0x557457246b60 2024-04-02T07:51:36.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.229+0000 7f3ed187db80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x557457291f10 con 0x557457275830 2024-04-02T07:51:36.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.229+0000 7f3ed187db80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x557457294aa0 con 0x557457248be0 2024-04-02T07:51:36.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.229+0000 7f3eb6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457246b60 0x55745728d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.229+0000 7f3eb6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457246b60 0x55745728d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44764/0 (socket says 172.21.15.89:44764) 2024-04-02T07:51:36.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.229+0000 7f3eb6a00700 1 -- 172.21.15.89:0/1729663689 learned_addr learned my addr 172.21.15.89:0/1729663689 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:36.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.229+0000 7f3eb6000700 1 --2- 172.21.15.89:0/1729663689 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 0x557457292cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.229+0000 7f3eb7400700 1 --2- 172.21.15.89:0/1729663689 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 0x557457274020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.230+0000 7f3eb6a00700 1 -- 172.21.15.89:0/1729663689 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 msgr2=0x557457292cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:36.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.230+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/1729663689 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 0x557457292cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.230+0000 7f3eb6a00700 1 -- 172.21.15.89:0/1729663689 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 msgr2=0x557457274020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:36.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.230+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/1729663689 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 0x557457274020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.230+0000 7f3eb6a00700 1 -- 172.21.15.89:0/1729663689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557457161010 con 0x557457246b60 2024-04-02T07:51:36.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.230+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/1729663689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457246b60 0x55745728d520 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800c910 tx=0x7f3ea802b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254aa066f7a7bc53 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:36.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.230+0000 7f3ea3e00700 1 -- 172.21.15.89:0/1729663689 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8007070 con 0x557457246b60 2024-04-02T07:51:36.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.230+0000 7f3ea3e00700 1 -- 172.21.15.89:0/1729663689 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea80071d0 con 0x557457246b60 2024-04-02T07:51:36.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.230+0000 7f3ea3e00700 1 -- 172.21.15.89:0/1729663689 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea802a600 con 0x557457246b60 2024-04-02T07:51:36.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.231+0000 7f3ed187db80 1 -- 172.21.15.89:0/1729663689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457246b60 msgr2=0x55745728d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.231+0000 7f3ed187db80 1 --2- 172.21.15.89:0/1729663689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457246b60 0x55745728d520 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800c910 tx=0x7f3ea802b000 comp rx=0 tx=0).stop 2024-04-02T07:51:36.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.231+0000 7f3ed187db80 1 -- 172.21.15.89:0/1729663689 shutdown_connections 2024-04-02T07:51:36.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.231+0000 7f3ed187db80 1 --2- 172.21.15.89:0/1729663689 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457275830 0x557457274020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.231+0000 7f3ed187db80 1 --2- 172.21.15.89:0/1729663689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457246b60 0x55745728d520 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.231+0000 7f3ed187db80 1 --2- 172.21.15.89:0/1729663689 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 0x557457292cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.231+0000 7f3ed187db80 1 -- 172.21.15.89:0/1729663689 >> 172.21.15.89:0/1729663689 conn(0x5574571a96a0 msgr2=0x557457292550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:36.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.231+0000 7f3ed187db80 1 -- 172.21.15.89:0/1729663689 shutdown_connections 2024-04-02T07:51:36.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.231+0000 7f3ed187db80 1 -- 172.21.15.89:0/1729663689 wait complete. 2024-04-02T07:51:36.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.231+0000 7f3ed187db80 1 -- start start 2024-04-02T07:51:36.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.232+0000 7f3ed187db80 1 Processor -- start 2024-04-02T07:51:36.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.232+0000 7f3ed187db80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457246b60 0x5574572a8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.232+0000 7f3ed187db80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 0x557457294d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.232+0000 7f3ed187db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457275830 0x5574572ed820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.232+0000 7f3ed187db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55745726f570 con 0x557457275830 2024-04-02T07:51:36.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.232+0000 7f3ed187db80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574572faac0 con 0x557457246b60 2024-04-02T07:51:36.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3ed187db80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x5574572a9260 con 0x557457248be0 2024-04-02T07:51:36.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3ed187db80 10 client.0 _ll_register_callbacks cb 0x7f3ea8015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:51:36.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3eb7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457275830 0x5574572ed820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3eb7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457275830 0x5574572ed820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44774/0 (socket says 172.21.15.89:44774) 2024-04-02T07:51:36.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3eb7400700 1 -- 172.21.15.89:0/3231284520 learned_addr learned my addr 172.21.15.89:0/3231284520 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:36.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/3231284520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457246b60 0x5574572a8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3eb6000700 1 --2- 172.21.15.89:0/3231284520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 0x557457294d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3eb7400700 1 -- 172.21.15.89:0/3231284520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 msgr2=0x557457294d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:36.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3eb7400700 1 --2- 172.21.15.89:0/3231284520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 0x557457294d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3eb7400700 1 -- 172.21.15.89:0/3231284520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457246b60 msgr2=0x5574572a8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:36.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3eb7400700 1 --2- 172.21.15.89:0/3231284520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457246b60 0x5574572a8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3eb7400700 1 -- 172.21.15.89:0/3231284520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55745707b0a0 con 0x557457275830 2024-04-02T07:51:36.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3eb6000700 1 --2- 172.21.15.89:0/3231284520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 0x557457294d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:36.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3eb7400700 1 --2- 172.21.15.89:0/3231284520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457275830 0x5574572ed820 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3eb808c320 tx=0x7f3eb0018d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ade175dca34b8c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:36.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.233+0000 7f3e97e00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:51:36.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.234+0000 7f3e97e00700 1 -- 172.21.15.89:0/3231284520 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb000e040 con 0x557457275830 2024-04-02T07:51:36.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.234+0000 7f3e97e00700 1 -- 172.21.15.89:0/3231284520 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3eb001f070 con 0x557457275830 2024-04-02T07:51:36.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.234+0000 7f3ed187db80 1 -- 172.21.15.89:0/3231284520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3eb815e440 con 0x557457275830 2024-04-02T07:51:36.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.234+0000 7f3e97e00700 1 -- 172.21.15.89:0/3231284520 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3eb0016b90 con 0x557457275830 2024-04-02T07:51:36.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.234+0000 7f3ed187db80 10 client.6582 fetch_fsmap learned FSMap version 33 2024-04-02T07:51:36.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.234+0000 7f3ed187db80 1 -- 172.21.15.89:0/3231284520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x557456fef1d0 con 0x557457275830 2024-04-02T07:51:36.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.234+0000 7f3e97e00700 1 -- 172.21.15.89:0/3231284520 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb815e440 con 0x557457275830 2024-04-02T07:51:36.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.234+0000 7f3e97e00700 1 -- 172.21.15.89:0/3231284520 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3eb0016cf0 con 0x557457275830 2024-04-02T07:51:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.234+0000 7f3ed187db80 10 client.6582 fetch_fsmap finished waiting for FSMap version 33 2024-04-02T07:51:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.234+0000 7f3ed187db80 10 client.6582 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T07:51:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.234+0000 7f3ed187db80 20 client.6582 populate_metadata read hostname 'smithi089' 2024-04-02T07:51:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.234+0000 7f3ed187db80 1 --2- 172.21.15.89:0/3231284520 >> [v2:172.21.15.89:6836/1479795102,v1:172.21.15.89:6837/1479795102] conn(0x5574572f7a00 0x5574572f7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.234+0000 7f3ed187db80 4 client.6582 mds_command: new command op to 6432 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T07:51:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.234+0000 7f3ed187db80 1 -- 172.21.15.89:0/3231284520 --> [v2:172.21.15.89:6836/1479795102,v1:172.21.15.89:6837/1479795102] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f3ea40604b0 con 0x5574572f7a00 2024-04-02T07:51:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.235+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/3231284520 >> [v2:172.21.15.89:6836/1479795102,v1:172.21.15.89:6837/1479795102] conn(0x5574572f7a00 0x5574572f7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.235+0000 7f3e97e00700 10 client.6582 ms_handle_connect on v2:172.21.15.89:6836/1479795102 2024-04-02T07:51:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.235+0000 7f3eb6a00700 1 --2- 172.21.15.89:0/3231284520 >> [v2:172.21.15.89:6836/1479795102,v1:172.21.15.89:6837/1479795102] conn(0x5574572f7a00 0x5574572f7de0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d980338cf9737d59 server_cookie=e0ca192d655106b8 in_seq=0 out_seq=0 2024-04-02T07:51:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.235+0000 7f3e97e00700 1 -- 172.21.15.89:0/3231284520 <== mds.0 v2:172.21.15.89:6836/1479795102 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x7f3ea40604b0 con 0x5574572f7a00 2024-04-02T07:51:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.235+0000 7f3e97e00700 10 client.6582 handle_command_reply: tid=0 2024-04-02T07:51:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.235+0000 7f3ed187db80 1 client.6582 shutdown 2024-04-02T07:51:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.235+0000 7f3ed187db80 1 -- 172.21.15.89:0/3231284520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457275830 msgr2=0x5574572ed820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.235+0000 7f3ed187db80 1 --2- 172.21.15.89:0/3231284520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457275830 0x5574572ed820 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3eb808c320 tx=0x7f3eb0018d00 comp rx=0 tx=0).stop 2024-04-02T07:51:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.236+0000 7f3ed187db80 1 -- 172.21.15.89:0/3231284520 shutdown_connections 2024-04-02T07:51:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.236+0000 7f3ed187db80 1 --2- 172.21.15.89:0/3231284520 >> [v2:172.21.15.89:6836/1479795102,v1:172.21.15.89:6837/1479795102] conn(0x5574572f7a00 0x5574572f7de0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.236+0000 7f3ed187db80 1 --2- 172.21.15.89:0/3231284520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557457246b60 0x5574572a8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.236+0000 7f3ed187db80 1 --2- 172.21.15.89:0/3231284520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557457275830 0x5574572ed820 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.236+0000 7f3e97e00700 0 client.6582 ms_handle_reset on v2:172.21.15.89:6836/1479795102 2024-04-02T07:51:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.236+0000 7f3ed187db80 1 --2- 172.21.15.89:0/3231284520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557457248be0 0x557457294d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.236+0000 7f3ed187db80 1 -- 172.21.15.89:0/3231284520 >> 172.21.15.89:0/3231284520 conn(0x5574571a96a0 msgr2=0x557457272300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.236+0000 7f3ed187db80 1 -- 172.21.15.89:0/3231284520 shutdown_connections 2024-04-02T07:51:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.236+0000 7f3ed187db80 1 -- 172.21.15.89:0/3231284520 wait complete. 2024-04-02T07:51:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.236+0000 7f3ed187db80 20 client.6582 trim_cache size 0 max 16384 2024-04-02T07:51:36.242 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:51:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T07:51:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T07:51:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 6432, 2024-04-02T07:51:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T07:51:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T07:51:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T07:51:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 6.5492091510000003, 2024-04-02T07:51:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 33, 2024-04-02T07:51:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 83, 2024-04-02T07:51:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 83, 2024-04-02T07:51:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 11.618439467 2024-04-02T07:51:36.244 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:51:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.236+0000 7f3ebde00700 1 -- 172.21.15.89:0/4210150061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e9806adb0 msgr2=0x7f3e9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.236+0000 7f3ebde00700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e9806adb0 0x7f3e9806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8015040 tx=0x7f3ea8007000 comp rx=0 tx=0).stop 2024-04-02T07:51:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.236+0000 7f3ebde00700 1 -- 172.21.15.89:0/4210150061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb8152330 msgr2=0x7f3eb815f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.236+0000 7f3ebde00700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb8152330 0x7f3eb815f790 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400f820 tx=0x7f3ea400fbe0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.237+0000 7f3ebde00700 1 -- 172.21.15.89:0/4210150061 shutdown_connections 2024-04-02T07:51:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.237+0000 7f3ebde00700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e9806adb0 0x7f3e9806d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.237+0000 7f3ebde00700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb815fcd0 0x7f3eb80b4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.237+0000 7f3ebde00700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e9c003a30 0x7f3eb81f34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.237+0000 7f3ebde00700 1 --2- 172.21.15.89:0/4210150061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb8152330 0x7f3eb815f790 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.237+0000 7f3ebde00700 1 -- 172.21.15.89:0/4210150061 >> 172.21.15.89:0/4210150061 conn(0x7f3eb808aff0 msgr2=0x7f3eb80b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.237+0000 7f3ebde00700 1 -- 172.21.15.89:0/4210150061 shutdown_connections 2024-04-02T07:51:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.237+0000 7f3ebde00700 1 -- 172.21.15.89:0/4210150061 wait complete. 2024-04-02T07:51:36.256 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T07:51:36.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.446+0000 7fe79b600700 1 Processor -- start 2024-04-02T07:51:36.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.447+0000 7fe79b600700 1 -- start start 2024-04-02T07:51:36.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.448+0000 7fe79b600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe79415b5d0 0x7fe7940ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.448+0000 7fe79b600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79415fd50 con 0x7fe7940ac7d0 2024-04-02T07:51:36.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.448+0000 7fe79b600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79415f140 con 0x7fe7940cb3f0 2024-04-02T07:51:36.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.448+0000 7fe79b600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe79415b9b0 con 0x7fe79415b5d0 2024-04-02T07:51:36.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.448+0000 7fe798c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe79415b5d0 0x7fe7940ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.448+0000 7fe798c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe79415b5d0 0x7fe7940ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44782/0 (socket says 172.21.15.89:44782) 2024-04-02T07:51:36.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.448+0000 7fe798c00700 1 -- 172.21.15.89:0/4151751781 learned_addr learned my addr 172.21.15.89:0/4151751781 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:36.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.448+0000 7fe798c00700 1 -- 172.21.15.89:0/4151751781 >> v1:172.21.15.113:6789/0 conn(0x7fe7940cb3f0 legacy=0x7fe7940ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:36.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.448+0000 7fe798c00700 1 -- 172.21.15.89:0/4151751781 >> v1:172.21.15.89:6789/0 conn(0x7fe7940ac7d0 legacy=0x7fe79406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.448+0000 7fe798c00700 1 -- 172.21.15.89:0/4151751781 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79415f420 con 0x7fe79415b5d0 2024-04-02T07:51:36.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.449+0000 7fe798c00700 1 --2- 172.21.15.89:0/4151751781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe79415b5d0 0x7fe7940ae3f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe7880098f0 tx=0x7fe788009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d2654598a58747 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:36.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.449+0000 7fe793400700 1 -- 172.21.15.89:0/4151751781 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe788004230 con 0x7fe79415b5d0 2024-04-02T07:51:36.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.449+0000 7fe793400700 1 -- 172.21.15.89:0/4151751781 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe788004390 con 0x7fe79415b5d0 2024-04-02T07:51:36.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.449+0000 7fe793400700 1 -- 172.21.15.89:0/4151751781 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7880036a0 con 0x7fe79415b5d0 2024-04-02T07:51:36.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.449+0000 7fe79b600700 1 -- 172.21.15.89:0/4151751781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe79415b5d0 msgr2=0x7fe7940ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.449+0000 7fe79b600700 1 --2- 172.21.15.89:0/4151751781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe79415b5d0 0x7fe7940ae3f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe7880098f0 tx=0x7fe788009c00 comp rx=0 tx=0).stop 2024-04-02T07:51:36.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.450+0000 7fe79b600700 1 -- 172.21.15.89:0/4151751781 shutdown_connections 2024-04-02T07:51:36.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.450+0000 7fe79b600700 1 --2- 172.21.15.89:0/4151751781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe79415b5d0 0x7fe7940ae3f0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.450+0000 7fe79b600700 1 -- 172.21.15.89:0/4151751781 >> 172.21.15.89:0/4151751781 conn(0x7fe7940cf190 msgr2=0x7fe7940cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:36.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.450+0000 7fe79b600700 1 -- 172.21.15.89:0/4151751781 shutdown_connections 2024-04-02T07:51:36.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.450+0000 7fe79b600700 1 -- 172.21.15.89:0/4151751781 wait complete. 2024-04-02T07:51:36.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.451+0000 7fe79b600700 1 Processor -- start 2024-04-02T07:51:36.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.451+0000 7fe79b600700 1 -- start start 2024-04-02T07:51:36.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.452+0000 7fe79b600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7940ac7d0 0x7fe7940ce420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.452+0000 7fe79b600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe79415b5d0 0x7fe7940ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.452+0000 7fe79b600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7940b41b0 0x7fe7940cd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.452+0000 7fe79b600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe794072940 con 0x7fe7940ac7d0 2024-04-02T07:51:36.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.452+0000 7fe79b600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940c9110 con 0x7fe7940b41b0 2024-04-02T07:51:36.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.452+0000 7fe79b600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940c8500 con 0x7fe79415b5d0 2024-04-02T07:51:36.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.453+0000 7fe798c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7940ac7d0 0x7fe7940ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.453+0000 7fe798c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7940ac7d0 0x7fe7940ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44784/0 (socket says 172.21.15.89:44784) 2024-04-02T07:51:36.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.453+0000 7fe798c00700 1 -- 172.21.15.89:0/1385336088 learned_addr learned my addr 172.21.15.89:0/1385336088 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:36.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.453+0000 7fe799600700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7940b41b0 0x7fe7940cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.453+0000 7fe793e00700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe79415b5d0 0x7fe7940ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.453+0000 7fe798c00700 1 -- 172.21.15.89:0/1385336088 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe79415b5d0 msgr2=0x7fe7940ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.453+0000 7fe798c00700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe79415b5d0 0x7fe7940ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.453+0000 7fe798c00700 1 -- 172.21.15.89:0/1385336088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7940b41b0 msgr2=0x7fe7940cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.453+0000 7fe798c00700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7940b41b0 0x7fe7940cd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.453+0000 7fe798c00700 1 -- 172.21.15.89:0/1385336088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe788009550 con 0x7fe7940ac7d0 2024-04-02T07:51:36.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.453+0000 7fe799600700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7940b41b0 0x7fe7940cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:36.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.453+0000 7fe793e00700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe79415b5d0 0x7fe7940ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:36.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.453+0000 7fe798c00700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7940ac7d0 0x7fe7940ce420 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe78800b0e0 tx=0x7fe788004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:36.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.454+0000 7fe790c00700 1 -- 172.21.15.89:0/1385336088 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78800b660 con 0x7fe7940ac7d0 2024-04-02T07:51:36.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.454+0000 7fe790c00700 1 -- 172.21.15.89:0/1385336088 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78800b7e0 con 0x7fe7940ac7d0 2024-04-02T07:51:36.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.454+0000 7fe79b600700 1 -- 172.21.15.89:0/1385336088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7940c87e0 con 0x7fe7940ac7d0 2024-04-02T07:51:36.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.454+0000 7fe790c00700 1 -- 172.21.15.89:0/1385336088 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78800bb90 con 0x7fe7940ac7d0 2024-04-02T07:51:36.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.454+0000 7fe79b600700 1 -- 172.21.15.89:0/1385336088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7940c8da0 con 0x7fe7940ac7d0 2024-04-02T07:51:36.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.455+0000 7fe79b600700 1 -- 172.21.15.89:0/1385336088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7940c8f60 con 0x7fe7940ac7d0 2024-04-02T07:51:36.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.456+0000 7fe790c00700 1 -- 172.21.15.89:0/1385336088 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe78800bcf0 con 0x7fe7940ac7d0 2024-04-02T07:51:36.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.456+0000 7fe790c00700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe77406adb0 0x7fe77406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:36.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.457+0000 7fe790c00700 1 -- 172.21.15.89:0/1385336088 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe78808afa0 con 0x7fe7940ac7d0 2024-04-02T07:51:36.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.457+0000 7fe793e00700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe77406adb0 0x7fe77406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:36.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.457+0000 7fe793e00700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe77406adb0 0x7fe77406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe794153710 tx=0x7fe780009040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:36.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.461+0000 7fe790c00700 1 -- 172.21.15.89:0/1385336088 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe7880be580 con 0x7fe7940ac7d0 2024-04-02T07:51:36.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.704+0000 7fe79b600700 1 -- 172.21.15.89:0/1385336088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe7940cb9f0 con 0x7fe7940ac7d0 2024-04-02T07:51:36.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.704+0000 7fe790c00700 1 -- 172.21.15.89:0/1385336088 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe788055fa0 con 0x7fe7940ac7d0 2024-04-02T07:51:36.706 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T07:51:36.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.707+0000 7fe79b600700 1 -- 172.21.15.89:0/1385336088 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe77406adb0 msgr2=0x7fe77406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.707+0000 7fe79b600700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe77406adb0 0x7fe77406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe794153710 tx=0x7fe780009040 comp rx=0 tx=0).stop 2024-04-02T07:51:36.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.707+0000 7fe79b600700 1 -- 172.21.15.89:0/1385336088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7940ac7d0 msgr2=0x7fe7940ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:36.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.707+0000 7fe79b600700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7940ac7d0 0x7fe7940ce420 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe78800b0e0 tx=0x7fe788004750 comp rx=0 tx=0).stop 2024-04-02T07:51:36.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.707+0000 7fe79b600700 1 -- 172.21.15.89:0/1385336088 shutdown_connections 2024-04-02T07:51:36.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.707+0000 7fe79b600700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe77406adb0 0x7fe77406d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.707+0000 7fe79b600700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7940b41b0 0x7fe7940cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.707+0000 7fe79b600700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7940ac7d0 0x7fe7940ce420 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.707+0000 7fe79b600700 1 --2- 172.21.15.89:0/1385336088 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe79415b5d0 0x7fe7940ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:36.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.707+0000 7fe79b600700 1 -- 172.21.15.89:0/1385336088 >> 172.21.15.89:0/1385336088 conn(0x7fe7940cf190 msgr2=0x7fe79406da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:36.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.707+0000 7fe79b600700 1 -- 172.21.15.89:0/1385336088 shutdown_connections 2024-04-02T07:51:36.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:36.707+0000 7fe79b600700 1 -- 172.21.15.89:0/1385336088 wait complete. 2024-04-02T07:51:36.727 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T07:51:36.727 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T07:51:36.727 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T07:51:36.727 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T07:51:36.728 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T07:51:36.728 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T07:51:36.728 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T07:51:36.728 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T07:51:36.728 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:51:36.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:36.728 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T07:51:36.773 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:51:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:51:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:51:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:51:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T07:51:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:51:36.774 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:51:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:51:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T07:51:36.774 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:51:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:51:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T07:51:36.774 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:51:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:51:36.774 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T07:51:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:51:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T07:51:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 41532sec preferred_lft 41532sec 2024-04-02T07:51:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T07:51:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:51:36.775 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T07:51:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:51:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T07:51:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:51:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T07:51:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:51:36.775 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:51:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T07:51:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T07:51:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:51:36.775 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:51:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T07:51:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T07:51:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:51:36.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:36.776 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T07:51:36.877 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T07:51:36.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:36.880 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T07:51:36.928 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T07:51:36.928 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T07:51:36.928 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T07:51:36.928 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T07:51:36.928 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T07:51:36.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:36.929 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:51:36.929 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:51:36.929 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T07:51:36.929 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T07:51:36.929 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T07:51:36.929 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:51:37.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:37.089 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T07:51:37.143 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T07:51:37.143 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T07:51:37.144 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:51:37.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:37.144 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T07:51:37.260 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:51:37.261 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T07:51:37.310 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:51:37.310 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:51:37.310 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:51:37.397 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T07:51:37.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:37.397 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:51:37.541 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:51:37.542 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:51:37.568 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:51:37.568 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:51:37.569 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T07:51:37.569 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T07:51:37.569 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T07:51:37.573 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T07:51:37.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:37.573 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:51:37.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:37.641 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:51:37.745 INFO:teuthology.orchestra.run.smithi089.stdout:client5744 2024-04-02T07:51:37.745 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:51:37.745 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:37.792 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T07:51:38.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.024+0000 7f55d7600700 1 Processor -- start 2024-04-02T07:51:38.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.025+0000 7f55d7600700 1 -- start start 2024-04-02T07:51:38.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.026+0000 7f55d7600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f55d00d16a0 0x7f55d00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:38.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.026+0000 7f55d7600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d0161420 con 0x7f55d00d2370 2024-04-02T07:51:38.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.026+0000 7f55d7600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d0161700 con 0x7f55d015d9d0 2024-04-02T07:51:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.026+0000 7f55d7600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d00d4340 con 0x7f55d00d16a0 2024-04-02T07:51:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.026+0000 7f55cfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f55d00d16a0 0x7f55d00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.026+0000 7f55d5600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f55d00d2370 0x7f55d00ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58796/0 (socket says 172.21.15.89:58796) 2024-04-02T07:51:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.026+0000 7f55cfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f55d00d16a0 0x7f55d00d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44792/0 (socket says 172.21.15.89:44792) 2024-04-02T07:51:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.026+0000 7f55d5600700 1 -- 172.21.15.89:0/2427270885 learned_addr learned my addr 172.21.15.89:0/2427270885 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.027+0000 7f55cfe00700 1 -- 172.21.15.89:0/2427270885 >> v1:172.21.15.113:6789/0 conn(0x7f55d015d9d0 legacy=0x7f55d00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.027+0000 7f55cfe00700 1 -- 172.21.15.89:0/2427270885 >> v1:172.21.15.89:6789/0 conn(0x7f55d00d2370 legacy=0x7f55d00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.027+0000 7f55cf400700 1 -- 172.21.15.89:0/2427270885 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520939535 0 0) 0x7f55d0161420 con 0x7f55d00d2370 2024-04-02T07:51:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.027+0000 7f55cfe00700 1 -- 172.21.15.89:0/2427270885 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d01619e0 con 0x7f55d00d16a0 2024-04-02T07:51:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.027+0000 7f55cfe00700 1 --2- 172.21.15.89:0/2427270885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55d00d16a0 0x7f55d00d1e30 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f55bc009830 tx=0x7f55bc009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7127eadc22f25cf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.027+0000 7f55cf400700 1 -- 172.21.15.89:0/2427270885 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc011830 con 0x7f55d00d16a0 2024-04-02T07:51:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.027+0000 7f55cf400700 1 -- 172.21.15.89:0/2427270885 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55bc011990 con 0x7f55d00d16a0 2024-04-02T07:51:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.027+0000 7f55cf400700 1 -- 172.21.15.89:0/2427270885 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc011d40 con 0x7f55d00d16a0 2024-04-02T07:51:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.028+0000 7f55d7600700 1 -- 172.21.15.89:0/2427270885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55d00d16a0 msgr2=0x7f55d00d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.028+0000 7f55d7600700 1 --2- 172.21.15.89:0/2427270885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55d00d16a0 0x7f55d00d1e30 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f55bc009830 tx=0x7f55bc009b40 comp rx=0 tx=0).stop 2024-04-02T07:51:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.028+0000 7f55d7600700 1 -- 172.21.15.89:0/2427270885 shutdown_connections 2024-04-02T07:51:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.028+0000 7f55d7600700 1 --2- 172.21.15.89:0/2427270885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55d00d16a0 0x7f55d00d1e30 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.028+0000 7f55d7600700 1 -- 172.21.15.89:0/2427270885 >> 172.21.15.89:0/2427270885 conn(0x7f55d008aff0 msgr2=0x7f55d00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.028+0000 7f55d7600700 1 -- 172.21.15.89:0/2427270885 shutdown_connections 2024-04-02T07:51:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.028+0000 7f55d7600700 1 -- 172.21.15.89:0/2427270885 wait complete. 2024-04-02T07:51:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.029+0000 7f55d7600700 1 Processor -- start 2024-04-02T07:51:38.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.030+0000 7f55d7600700 1 -- start start 2024-04-02T07:51:38.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.030+0000 7f55d7600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55d00d16a0 0x7f55d014e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:38.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.030+0000 7f55d7600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55d00d2370 0x7f55d00cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:38.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.030+0000 7f55d7600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d015d9d0 0x7f55d01606d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:38.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.030+0000 7f55d7600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d014ef20 con 0x7f55d00d16a0 2024-04-02T07:51:38.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.030+0000 7f55d7600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d00cc2f0 con 0x7f55d00d2370 2024-04-02T07:51:38.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.030+0000 7f55d7600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d00cba70 con 0x7f55d015d9d0 2024-04-02T07:51:38.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.031+0000 7f55d4c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55d00d16a0 0x7f55d014e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:38.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.031+0000 7f55d4c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55d00d16a0 0x7f55d014e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44804/0 (socket says 172.21.15.89:44804) 2024-04-02T07:51:38.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.031+0000 7f55d4c00700 1 -- 172.21.15.89:0/1126932255 learned_addr learned my addr 172.21.15.89:0/1126932255 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:38.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.031+0000 7f55cfe00700 1 --2- 172.21.15.89:0/1126932255 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55d00d2370 0x7f55d00cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:38.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.031+0000 7f55d5600700 1 --2- 172.21.15.89:0/1126932255 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d015d9d0 0x7f55d01606d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:38.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.031+0000 7f55d4c00700 1 -- 172.21.15.89:0/1126932255 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d015d9d0 msgr2=0x7f55d01606d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:38.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.031+0000 7f55d4c00700 1 --2- 172.21.15.89:0/1126932255 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d015d9d0 0x7f55d01606d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.031+0000 7f55d4c00700 1 -- 172.21.15.89:0/1126932255 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55d00d2370 msgr2=0x7f55d00cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.031+0000 7f55d4c00700 1 --2- 172.21.15.89:0/1126932255 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55d00d2370 0x7f55d00cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.031+0000 7f55d4c00700 1 -- 172.21.15.89:0/1126932255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55bc009440 con 0x7f55d00d16a0 2024-04-02T07:51:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.031+0000 7f55d4c00700 1 --2- 172.21.15.89:0/1126932255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55d00d16a0 0x7f55d014e9e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f55c400c450 tx=0x7f55c400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.032+0000 7f55ccc00700 1 -- 172.21.15.89:0/1126932255 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c40052d0 con 0x7f55d00d16a0 2024-04-02T07:51:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.032+0000 7f55ccc00700 1 -- 172.21.15.89:0/1126932255 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c4005430 con 0x7f55d00d16a0 2024-04-02T07:51:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.032+0000 7f55d7600700 1 -- 172.21.15.89:0/1126932255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d01f39d0 con 0x7f55d00d16a0 2024-04-02T07:51:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.032+0000 7f55ccc00700 1 -- 172.21.15.89:0/1126932255 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c40038c0 con 0x7f55d00d16a0 2024-04-02T07:51:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.032+0000 7f55d7600700 1 -- 172.21.15.89:0/1126932255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d01f3f90 con 0x7f55d00d16a0 2024-04-02T07:51:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.033+0000 7f55d7600700 1 -- 172.21.15.89:0/1126932255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d00d1ca0 con 0x7f55d00d16a0 2024-04-02T07:51:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.034+0000 7f55ccc00700 1 -- 172.21.15.89:0/1126932255 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f55c4003a20 con 0x7f55d00d16a0 2024-04-02T07:51:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.034+0000 7f55ccc00700 1 --2- 172.21.15.89:0/1126932255 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f55b006adb0 0x7f55b006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.035+0000 7f55ccc00700 1 -- 172.21.15.89:0/1126932255 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f55c40889d0 con 0x7f55d00d16a0 2024-04-02T07:51:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.035+0000 7f55cfe00700 1 --2- 172.21.15.89:0/1126932255 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f55b006adb0 0x7f55b006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:38.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.035+0000 7f55cfe00700 1 --2- 172.21.15.89:0/1126932255 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f55b006adb0 0x7f55b006d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f55d00d1ae0 tx=0x7f55bc01c040 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:38.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.040+0000 7f55ccc00700 1 -- 172.21.15.89:0/1126932255 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f55c4053940 con 0x7f55d00d16a0 2024-04-02T07:51:38.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.271+0000 7f55d7600700 1 -- 172.21.15.89:0/1126932255 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f55d0161fa0 con 0x7f55b006adb0 2024-04-02T07:51:38.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.273+0000 7f55ccc00700 1 -- 172.21.15.89:0/1126932255 <== mgr.6180 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f55bc011630 con 0x7f55b006adb0 2024-04-02T07:51:38.274 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:51:38.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:51:38.274 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T07:51:38.275 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:51:38.275 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:51:38.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.275+0000 7f55d7600700 1 -- 172.21.15.89:0/1126932255 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f55b006adb0 msgr2=0x7f55b006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:38.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.275+0000 7f55d7600700 1 --2- 172.21.15.89:0/1126932255 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f55b006adb0 0x7f55b006d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f55d00d1ae0 tx=0x7f55bc01c040 comp rx=0 tx=0).stop 2024-04-02T07:51:38.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.275+0000 7f55d7600700 1 -- 172.21.15.89:0/1126932255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55d00d16a0 msgr2=0x7f55d014e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:38.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.275+0000 7f55d7600700 1 --2- 172.21.15.89:0/1126932255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55d00d16a0 0x7f55d014e9e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f55c400c450 tx=0x7f55c400c760 comp rx=0 tx=0).stop 2024-04-02T07:51:38.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.275+0000 7f55d7600700 1 -- 172.21.15.89:0/1126932255 shutdown_connections 2024-04-02T07:51:38.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.275+0000 7f55d7600700 1 --2- 172.21.15.89:0/1126932255 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f55b006adb0 0x7f55b006d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.275+0000 7f55d7600700 1 --2- 172.21.15.89:0/1126932255 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d015d9d0 0x7f55d01606d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.275+0000 7f55d7600700 1 --2- 172.21.15.89:0/1126932255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55d00d16a0 0x7f55d014e9e0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.275+0000 7f55d7600700 1 --2- 172.21.15.89:0/1126932255 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55d00d2370 0x7f55d00cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.275+0000 7f55d7600700 1 -- 172.21.15.89:0/1126932255 >> 172.21.15.89:0/1126932255 conn(0x7f55d008aff0 msgr2=0x7f55d00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:38.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.276+0000 7f55d7600700 1 -- 172.21.15.89:0/1126932255 shutdown_connections 2024-04-02T07:51:38.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.276+0000 7f55d7600700 1 -- 172.21.15.89:0/1126932255 wait complete. 2024-04-02T07:51:38.295 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T07:51:38.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.486+0000 7f4be7000700 1 Processor -- start 2024-04-02T07:51:38.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.486+0000 7f4be7000700 1 -- start start 2024-04-02T07:51:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.487+0000 7f4be7000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4be015d9d0 0x7f4be00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4be7000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4be00d2540 0x7f4be00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4be7000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be0161420 con 0x7f4be00d16a0 2024-04-02T07:51:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4be7000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be015edc0 con 0x7f4be015d9d0 2024-04-02T07:51:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4be7000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be0161700 con 0x7f4be00d2540 2024-04-02T07:51:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4bdfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4be015d9d0 0x7f4be00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4bdfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4be015d9d0 0x7f4be00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44818/0 (socket says 172.21.15.89:44818) 2024-04-02T07:51:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4bdfe00700 1 -- 172.21.15.89:0/3941232703 learned_addr learned my addr 172.21.15.89:0/3941232703 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4bdf400700 1 --1- 172.21.15.89:0/3941232703 >> v1:172.21.15.89:6789/0 conn(0x7f4be00d16a0 0x7f4be00d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58802/0 (socket says 172.21.15.89:58802) 2024-04-02T07:51:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4be5000700 1 --2- 172.21.15.89:0/3941232703 >> v2:172.21.15.113:3300/0 conn(0x7f4be00d2540 0x7f4be00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4bdfe00700 1 -- 172.21.15.89:0/3941232703 >> v2:172.21.15.113:3300/0 conn(0x7f4be00d2540 msgr2=0x7f4be00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4bdfe00700 1 --2- 172.21.15.89:0/3941232703 >> v2:172.21.15.113:3300/0 conn(0x7f4be00d2540 0x7f4be00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4bdfe00700 1 -- 172.21.15.89:0/3941232703 >> v1:172.21.15.89:6789/0 conn(0x7f4be00d16a0 legacy=0x7f4be00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4bdfe00700 1 -- 172.21.15.89:0/3941232703 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be01619e0 con 0x7f4be015d9d0 2024-04-02T07:51:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4be5000700 1 --2- 172.21.15.89:0/3941232703 >> v2:172.21.15.113:3300/0 conn(0x7f4be00d2540 0x7f4be00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.488+0000 7f4bdea00700 1 -- 172.21.15.89:0/3941232703 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2980611541 0 0) 0x7f4be0161420 con 0x7f4be00d16a0 2024-04-02T07:51:38.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.489+0000 7f4bdfe00700 1 --2- 172.21.15.89:0/3941232703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4be015d9d0 0x7f4be00d3c30 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4be008ca10 tx=0x7f4bd0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e7285d567c9944 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:38.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.489+0000 7f4bdea00700 1 -- 172.21.15.89:0/3941232703 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd000b870 con 0x7f4be015d9d0 2024-04-02T07:51:38.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.489+0000 7f4bdea00700 1 -- 172.21.15.89:0/3941232703 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd000b9d0 con 0x7f4be015d9d0 2024-04-02T07:51:38.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.489+0000 7f4bdea00700 1 -- 172.21.15.89:0/3941232703 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd000bd60 con 0x7f4be015d9d0 2024-04-02T07:51:38.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.489+0000 7f4be7000700 1 -- 172.21.15.89:0/3941232703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4be015d9d0 msgr2=0x7f4be00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:38.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.489+0000 7f4be7000700 1 --2- 172.21.15.89:0/3941232703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4be015d9d0 0x7f4be00d3c30 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4be008ca10 tx=0x7f4bd0009bd0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.489+0000 7f4be7000700 1 -- 172.21.15.89:0/3941232703 shutdown_connections 2024-04-02T07:51:38.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.489+0000 7f4be7000700 1 --2- 172.21.15.89:0/3941232703 >> v2:172.21.15.113:3300/0 conn(0x7f4be00d2540 0x7f4be00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.489+0000 7f4be7000700 1 --2- 172.21.15.89:0/3941232703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4be015d9d0 0x7f4be00d3c30 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.489+0000 7f4be7000700 1 -- 172.21.15.89:0/3941232703 >> 172.21.15.89:0/3941232703 conn(0x7f4be008aff0 msgr2=0x7f4be00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:38.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.490+0000 7f4be7000700 1 -- 172.21.15.89:0/3941232703 shutdown_connections 2024-04-02T07:51:38.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.490+0000 7f4be7000700 1 -- 172.21.15.89:0/3941232703 wait complete. 2024-04-02T07:51:38.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.491+0000 7f4be7000700 1 Processor -- start 2024-04-02T07:51:38.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.491+0000 7f4be7000700 1 -- start start 2024-04-02T07:51:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.492+0000 7f4be7000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4be00d16a0 0x7f4be015cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.492+0000 7f4be7000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4be00d2540 0x7f4be00ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.492+0000 7f4be7000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4be015d9d0 0x7f4be00b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.492+0000 7f4be7000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be015d070 con 0x7f4be015d9d0 2024-04-02T07:51:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.492+0000 7f4be7000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00caef0 con 0x7f4be00d2540 2024-04-02T07:51:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.492+0000 7f4be7000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00cb1d0 con 0x7f4be00d16a0 2024-04-02T07:51:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.493+0000 7f4be5000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4be015d9d0 0x7f4be00b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.493+0000 7f4be5000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4be015d9d0 0x7f4be00b6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44828/0 (socket says 172.21.15.89:44828) 2024-04-02T07:51:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.493+0000 7f4be5000700 1 -- 172.21.15.89:0/229092375 learned_addr learned my addr 172.21.15.89:0/229092375 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.493+0000 7f4bdfe00700 1 --2- 172.21.15.89:0/229092375 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4be00d16a0 0x7f4be015cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.493+0000 7f4bdf400700 1 --2- 172.21.15.89:0/229092375 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4be00d2540 0x7f4be00ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.493+0000 7f4be5000700 1 -- 172.21.15.89:0/229092375 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4be00d16a0 msgr2=0x7f4be015cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.493+0000 7f4be5000700 1 --2- 172.21.15.89:0/229092375 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4be00d16a0 0x7f4be015cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.493+0000 7f4be5000700 1 -- 172.21.15.89:0/229092375 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4be00d2540 msgr2=0x7f4be00ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.493+0000 7f4be5000700 1 --2- 172.21.15.89:0/229092375 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4be00d2540 0x7f4be00ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.493+0000 7f4be5000700 1 -- 172.21.15.89:0/229092375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd0009550 con 0x7f4be015d9d0 2024-04-02T07:51:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.493+0000 7f4be5000700 1 --2- 172.21.15.89:0/229092375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4be015d9d0 0x7f4be00b6020 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800ce80 tx=0x7f4bd800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.493+0000 7f4bd7e00700 1 -- 172.21.15.89:0/229092375 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd800ae20 con 0x7f4be015d9d0 2024-04-02T07:51:38.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.494+0000 7f4bd7e00700 1 -- 172.21.15.89:0/229092375 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd8015440 con 0x7f4be015d9d0 2024-04-02T07:51:38.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.494+0000 7f4bd7e00700 1 -- 172.21.15.89:0/229092375 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd80157f0 con 0x7f4be015d9d0 2024-04-02T07:51:38.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.494+0000 7f4be7000700 1 -- 172.21.15.89:0/229092375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be00b8080 con 0x7f4be015d9d0 2024-04-02T07:51:38.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.494+0000 7f4be7000700 1 -- 172.21.15.89:0/229092375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be00cb4b0 con 0x7f4be015d9d0 2024-04-02T07:51:38.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.495+0000 7f4be7000700 1 -- 172.21.15.89:0/229092375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be00d1ca0 con 0x7f4be015d9d0 2024-04-02T07:51:38.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.496+0000 7f4bd7e00700 1 -- 172.21.15.89:0/229092375 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 33) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4bd800d430 con 0x7f4be015d9d0 2024-04-02T07:51:38.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.496+0000 7f4bd7e00700 1 --2- 172.21.15.89:0/229092375 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4bc006adb0 0x7f4bc006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:38.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.497+0000 7f4bd7e00700 1 -- 172.21.15.89:0/229092375 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4bd8089b60 con 0x7f4be015d9d0 2024-04-02T07:51:38.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.497+0000 7f4bdfe00700 1 --2- 172.21.15.89:0/229092375 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4bc006adb0 0x7f4bc006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:38.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.497+0000 7f4bdfe00700 1 --2- 172.21.15.89:0/229092375 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4bc006adb0 0x7f4bc006d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0014040 tx=0x7f4bd0008030 comp rx=0 tx=0).ready entity=mgr.6180 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:38.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.501+0000 7f4bd7e00700 1 -- 172.21.15.89:0/229092375 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4bd8054b60 con 0x7f4be015d9d0 2024-04-02T07:51:38.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.760+0000 7f4be7000700 1 -- 172.21.15.89:0/229092375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f4be00b85a0 con 0x7f4be015d9d0 2024-04-02T07:51:38.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.969+0000 7f4bd7e00700 1 -- 172.21.15.89:0/229092375 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 34) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f4bd8054f30 con 0x7f4be015d9d0 2024-04-02T07:51:38.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.971+0000 7f4bd7e00700 1 -- 172.21.15.89:0/229092375 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v34) v1 ==== 92+0+0 (secure 0 0 0) 0x7f4bd8072260 con 0x7f4be015d9d0 2024-04-02T07:51:38.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.974+0000 7f4be7000700 1 -- 172.21.15.89:0/229092375 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4bc006adb0 msgr2=0x7f4bc006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.974+0000 7f4be7000700 1 --2- 172.21.15.89:0/229092375 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4bc006adb0 0x7f4bc006d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0014040 tx=0x7f4bd0008030 comp rx=0 tx=0).stop 2024-04-02T07:51:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.974+0000 7f4be7000700 1 -- 172.21.15.89:0/229092375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4be015d9d0 msgr2=0x7f4be00b6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.974+0000 7f4be7000700 1 --2- 172.21.15.89:0/229092375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4be015d9d0 0x7f4be00b6020 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800ce80 tx=0x7f4bd800cf60 comp rx=0 tx=0).stop 2024-04-02T07:51:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.974+0000 7f4be7000700 1 -- 172.21.15.89:0/229092375 shutdown_connections 2024-04-02T07:51:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.974+0000 7f4be7000700 1 --2- 172.21.15.89:0/229092375 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4bc006adb0 0x7f4bc006d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.974+0000 7f4be7000700 1 --2- 172.21.15.89:0/229092375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4be015d9d0 0x7f4be00b6020 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.974+0000 7f4be7000700 1 --2- 172.21.15.89:0/229092375 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4be00d16a0 0x7f4be015cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.974+0000 7f4be7000700 1 --2- 172.21.15.89:0/229092375 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4be00d2540 0x7f4be00ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.974+0000 7f4be7000700 1 -- 172.21.15.89:0/229092375 >> 172.21.15.89:0/229092375 conn(0x7f4be008aff0 msgr2=0x7f4be00b16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.974+0000 7f4be7000700 1 -- 172.21.15.89:0/229092375 shutdown_connections 2024-04-02T07:51:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:38.974+0000 7f4be7000700 1 -- 172.21.15.89:0/229092375 wait complete. 2024-04-02T07:51:38.994 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T07:51:39.184 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:39.182+0000 7f49d7392240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:51:39.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.195+0000 7fbc50c00700 1 Processor -- start 2024-04-02T07:51:39.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.196+0000 7fbc50c00700 1 -- start start 2024-04-02T07:51:39.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.197+0000 7fbc50c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc4c160e50 0x7fbc4c0b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:39.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.197+0000 7fbc50c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbc4c0b3f80 0x7fbc4c0b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:39.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.197+0000 7fbc50c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c0cef00 con 0x7fbc4c0d3f20 2024-04-02T07:51:39.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.197+0000 7fbc50c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4c161230 con 0x7fbc4c0b3f80 2024-04-02T07:51:39.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.197+0000 7fbc50c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c0d2a80 con 0x7fbc4c160e50 2024-04-02T07:51:39.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.197+0000 7fbc49400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbc4c0d3f20 0x7fbc4c0b2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55850/0 (socket says 172.21.15.89:55850) 2024-04-02T07:51:39.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.197+0000 7fbc49400700 1 -- 172.21.15.89:0/1970692277 learned_addr learned my addr 172.21.15.89:0/1970692277 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:39.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.197+0000 7fbc49e00700 1 --2- 172.21.15.89:0/1970692277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc4c160e50 0x7fbc4c0b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:39.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.197+0000 7fbc4a800700 1 --2- 172.21.15.89:0/1970692277 >> v2:172.21.15.89:3300/0 conn(0x7fbc4c0b3f80 0x7fbc4c0b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:39.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.198+0000 7fbc49e00700 1 -- 172.21.15.89:0/1970692277 >> v2:172.21.15.89:3300/0 conn(0x7fbc4c0b3f80 msgr2=0x7fbc4c0b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:39.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.198+0000 7fbc48a00700 1 -- 172.21.15.89:0/1970692277 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643659221 0 0) 0x7fbc4c0cef00 con 0x7fbc4c0d3f20 2024-04-02T07:51:39.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.198+0000 7fbc49e00700 1 --2- 172.21.15.89:0/1970692277 >> v2:172.21.15.89:3300/0 conn(0x7fbc4c0b3f80 0x7fbc4c0b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.198+0000 7fbc49e00700 1 -- 172.21.15.89:0/1970692277 >> v1:172.21.15.89:6789/0 conn(0x7fbc4c0d3f20 legacy=0x7fbc4c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:39.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.198+0000 7fbc49e00700 1 -- 172.21.15.89:0/1970692277 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c0d43b0 con 0x7fbc4c160e50 2024-04-02T07:51:39.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.198+0000 7fbc49e00700 1 --2- 172.21.15.89:0/1970692277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc4c160e50 0x7fbc4c0b4e90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c009840 tx=0x7fbc3c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd2cdca843e52382 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:39.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.198+0000 7fbc48a00700 1 -- 172.21.15.89:0/1970692277 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c011830 con 0x7fbc4c160e50 2024-04-02T07:51:39.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.198+0000 7fbc48a00700 1 -- 172.21.15.89:0/1970692277 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3c011990 con 0x7fbc4c160e50 2024-04-02T07:51:39.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.198+0000 7fbc48a00700 1 -- 172.21.15.89:0/1970692277 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c011d20 con 0x7fbc4c160e50 2024-04-02T07:51:39.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.199+0000 7fbc50c00700 1 -- 172.21.15.89:0/1970692277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc4c160e50 msgr2=0x7fbc4c0b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:39.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.199+0000 7fbc50c00700 1 --2- 172.21.15.89:0/1970692277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc4c160e50 0x7fbc4c0b4e90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c009840 tx=0x7fbc3c009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:39.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.199+0000 7fbc50c00700 1 -- 172.21.15.89:0/1970692277 shutdown_connections 2024-04-02T07:51:39.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.199+0000 7fbc50c00700 1 --2- 172.21.15.89:0/1970692277 >> v2:172.21.15.89:3300/0 conn(0x7fbc4c0b3f80 0x7fbc4c0b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.199+0000 7fbc50c00700 1 --2- 172.21.15.89:0/1970692277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc4c160e50 0x7fbc4c0b4e90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.199+0000 7fbc50c00700 1 -- 172.21.15.89:0/1970692277 >> 172.21.15.89:0/1970692277 conn(0x7fbc4c08aff0 msgr2=0x7fbc4c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:39.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.199+0000 7fbc50c00700 1 -- 172.21.15.89:0/1970692277 shutdown_connections 2024-04-02T07:51:39.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.199+0000 7fbc50c00700 1 -- 172.21.15.89:0/1970692277 wait complete. 2024-04-02T07:51:39.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.201+0000 7fbc50c00700 1 Processor -- start 2024-04-02T07:51:39.203 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:39.201+0000 7fad0dc4e240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:51:39.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.202+0000 7fbc50c00700 1 -- start start 2024-04-02T07:51:39.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc50c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc4c0b3f80 0x7fbc4c0b6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:39.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc50c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc4c160e50 0x7fbc4c1f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:39.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc50c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc4c0b81b0 0x7fbc4c1f5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:39.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc50c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c1f4230 con 0x7fbc4c0b3f80 2024-04-02T07:51:39.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc50c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c0cb650 con 0x7fbc4c160e50 2024-04-02T07:51:39.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc50c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c0b6cd0 con 0x7fbc4c0b81b0 2024-04-02T07:51:39.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc49e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc4c0b3f80 0x7fbc4c0b6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:39.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc49e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc4c0b3f80 0x7fbc4c0b6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59532/0 (socket says 172.21.15.89:59532) 2024-04-02T07:51:39.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc49e00700 1 -- 172.21.15.89:0/2907253590 learned_addr learned my addr 172.21.15.89:0/2907253590 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:39.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc4a800700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc4c0b81b0 0x7fbc4c1f5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:39.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc49400700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc4c160e50 0x7fbc4c1f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:39.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc49e00700 1 -- 172.21.15.89:0/2907253590 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc4c0b81b0 msgr2=0x7fbc4c1f5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:39.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc49e00700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc4c0b81b0 0x7fbc4c1f5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc49e00700 1 -- 172.21.15.89:0/2907253590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc4c160e50 msgr2=0x7fbc4c1f3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:39.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc49e00700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc4c160e50 0x7fbc4c1f3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc49e00700 1 -- 172.21.15.89:0/2907253590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3c0094a0 con 0x7fbc4c0b3f80 2024-04-02T07:51:39.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.203+0000 7fbc49400700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc4c160e50 0x7fbc4c1f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:39.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.204+0000 7fbc49e00700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc4c0b3f80 0x7fbc4c0b6640 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0090b0 tx=0x7fbc3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:39.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.204+0000 7fbc42000700 1 -- 172.21.15.89:0/2907253590 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c0036a0 con 0x7fbc4c0b3f80 2024-04-02T07:51:39.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.204+0000 7fbc42000700 1 -- 172.21.15.89:0/2907253590 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3c01f070 con 0x7fbc4c0b3f80 2024-04-02T07:51:39.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.204+0000 7fbc50c00700 1 -- 172.21.15.89:0/2907253590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4c0b6fb0 con 0x7fbc4c0b3f80 2024-04-02T07:51:39.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.204+0000 7fbc42000700 1 -- 172.21.15.89:0/2907253590 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c003a40 con 0x7fbc4c0b3f80 2024-04-02T07:51:39.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.204+0000 7fbc50c00700 1 -- 172.21.15.89:0/2907253590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4c0b7570 con 0x7fbc4c0b3f80 2024-04-02T07:51:39.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.205+0000 7fbc50c00700 1 -- 172.21.15.89:0/2907253590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4c0b7760 con 0x7fbc4c0b3f80 2024-04-02T07:51:39.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.206+0000 7fbc42000700 1 -- 172.21.15.89:0/2907253590 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 34) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fbc3c003ba0 con 0x7fbc4c0b3f80 2024-04-02T07:51:39.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.207+0000 7fbc42000700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbc2c06ae00 0x7fbc2c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:39.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.207+0000 7fbc42000700 1 -- 172.21.15.89:0/2907253590 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fbc3c021070 con 0x7fbc4c0b3f80 2024-04-02T07:51:39.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.207+0000 7fbc49400700 1 -- 172.21.15.89:0/2907253590 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbc2c06ae00 msgr2=0x7fbc2c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:39.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.207+0000 7fbc49400700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbc2c06ae00 0x7fbc2c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:39.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.211+0000 7fbc42000700 1 -- 172.21.15.89:0/2907253590 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbc3c055de0 con 0x7fbc4c0b3f80 2024-04-02T07:51:39.368 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:39.367+0000 7f49d7392240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:51:39.395 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:39.393+0000 7fad0dc4e240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:51:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.408+0000 7fbc49400700 1 -- 172.21.15.89:0/2907253590 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbc2c06ae00 msgr2=0x7fbc2c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.408+0000 7fbc49400700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbc2c06ae00 0x7fbc2c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:39.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.461+0000 7fbc50c00700 1 -- 172.21.15.89:0/2907253590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fbc4c067720 con 0x7fbc4c0b3f80 2024-04-02T07:51:39.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.464+0000 7fbc42000700 1 -- 172.21.15.89:0/2907253590 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbc3c015020 con 0x7fbc4c0b3f80 2024-04-02T07:51:39.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.467+0000 7fbc50c00700 1 -- 172.21.15.89:0/2907253590 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbc2c06ae00 msgr2=0x7fbc2c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:39.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.467+0000 7fbc50c00700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbc2c06ae00 0x7fbc2c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.467+0000 7fbc50c00700 1 -- 172.21.15.89:0/2907253590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc4c0b3f80 msgr2=0x7fbc4c0b6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:39.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.467+0000 7fbc50c00700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc4c0b3f80 0x7fbc4c0b6640 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0090b0 tx=0x7fbc3c004750 comp rx=0 tx=0).stop 2024-04-02T07:51:39.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.467+0000 7fbc50c00700 1 -- 172.21.15.89:0/2907253590 shutdown_connections 2024-04-02T07:51:39.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.467+0000 7fbc50c00700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbc2c06ae00 0x7fbc2c06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.467+0000 7fbc50c00700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc4c0b81b0 0x7fbc4c1f5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.467+0000 7fbc50c00700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc4c0b3f80 0x7fbc4c0b6640 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.467+0000 7fbc50c00700 1 --2- 172.21.15.89:0/2907253590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc4c160e50 0x7fbc4c1f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.467+0000 7fbc50c00700 1 -- 172.21.15.89:0/2907253590 >> 172.21.15.89:0/2907253590 conn(0x7fbc4c08aff0 msgr2=0x7fbc4c0b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:39.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.467+0000 7fbc50c00700 1 -- 172.21.15.89:0/2907253590 shutdown_connections 2024-04-02T07:51:39.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.467+0000 7fbc50c00700 1 -- 172.21.15.89:0/2907253590 wait complete. 2024-04-02T07:51:39.490 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T07:51:39.683 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:39.683+0000 7f49d7392240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:51:39.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.688+0000 7fb57be00700 1 Processor -- start 2024-04-02T07:51:39.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.688+0000 7fb57be00700 1 -- start start 2024-04-02T07:51:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.689+0000 7fb57be00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb5740cd020 0x7fb5740cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.689+0000 7fb57be00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5740cdae0 con 0x7fb5740b2070 2024-04-02T07:51:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.689+0000 7fb57be00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5740cddc0 con 0x7fb5740cbb30 2024-04-02T07:51:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.689+0000 7fb57be00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5740ce0a0 con 0x7fb5740cd020 2024-04-02T07:51:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.689+0000 7fb579e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb5740b2070 0x7fb57406e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55864/0 (socket says 172.21.15.89:55864) 2024-04-02T07:51:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.690+0000 7fb579e00700 1 -- 172.21.15.89:0/2122881494 learned_addr learned my addr 172.21.15.89:0/2122881494 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.690+0000 7fb578a00700 1 --2- 172.21.15.89:0/2122881494 >> v2:172.21.15.113:3300/0 conn(0x7fb5740cd020 0x7fb5740cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.690+0000 7fb573e00700 1 -- 172.21.15.89:0/2122881494 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962163 0 0) 0x7fb5740cdae0 con 0x7fb5740b2070 2024-04-02T07:51:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.690+0000 7fb573e00700 1 -- 172.21.15.89:0/2122881494 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb55c0035b0 con 0x7fb5740b2070 2024-04-02T07:51:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.690+0000 7fb578a00700 1 -- 172.21.15.89:0/2122881494 >> v1:172.21.15.113:6789/0 conn(0x7fb5740cbb30 legacy=0x7fb5740cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.690+0000 7fb578a00700 1 -- 172.21.15.89:0/2122881494 >> v1:172.21.15.89:6789/0 conn(0x7fb5740b2070 legacy=0x7fb57406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:39.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.690+0000 7fb578a00700 1 -- 172.21.15.89:0/2122881494 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57406dc20 con 0x7fb5740cd020 2024-04-02T07:51:39.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.690+0000 7fb578a00700 1 --2- 172.21.15.89:0/2122881494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5740cd020 0x7fb5740cd480 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb560009840 tx=0x7fb560009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=360803c0383d51de server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:39.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.691+0000 7fb573e00700 1 -- 172.21.15.89:0/2122881494 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560011830 con 0x7fb5740cd020 2024-04-02T07:51:39.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.691+0000 7fb573e00700 1 -- 172.21.15.89:0/2122881494 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb560011990 con 0x7fb5740cd020 2024-04-02T07:51:39.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.691+0000 7fb573e00700 1 -- 172.21.15.89:0/2122881494 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560011d20 con 0x7fb5740cd020 2024-04-02T07:51:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.691+0000 7fb57be00700 1 -- 172.21.15.89:0/2122881494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5740cd020 msgr2=0x7fb5740cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.691+0000 7fb57be00700 1 --2- 172.21.15.89:0/2122881494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5740cd020 0x7fb5740cd480 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb560009840 tx=0x7fb560009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.691+0000 7fb57be00700 1 -- 172.21.15.89:0/2122881494 shutdown_connections 2024-04-02T07:51:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.691+0000 7fb57be00700 1 --2- 172.21.15.89:0/2122881494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5740cd020 0x7fb5740cd480 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.691+0000 7fb57be00700 1 -- 172.21.15.89:0/2122881494 >> 172.21.15.89:0/2122881494 conn(0x7fb5740b0d70 msgr2=0x7fb5740b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.691+0000 7fb57be00700 1 -- 172.21.15.89:0/2122881494 shutdown_connections 2024-04-02T07:51:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.692+0000 7fb57be00700 1 -- 172.21.15.89:0/2122881494 wait complete. 2024-04-02T07:51:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.693+0000 7fb57be00700 1 Processor -- start 2024-04-02T07:51:39.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.693+0000 7fb57be00700 1 -- start start 2024-04-02T07:51:39.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.694+0000 7fb57be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5740b2070 0x7fb5741546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:39.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.694+0000 7fb57be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5740cbb30 0x7fb5740ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:39.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.694+0000 7fb57be00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5740cd020 0x7fb574072010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:39.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.694+0000 7fb57be00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740c8ec0 con 0x7fb5740cbb30 2024-04-02T07:51:39.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.694+0000 7fb57be00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740ae000 con 0x7fb5740cd020 2024-04-02T07:51:39.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.694+0000 7fb57be00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740c8520 con 0x7fb5740b2070 2024-04-02T07:51:39.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb578a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5740cbb30 0x7fb5740ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:39.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb578a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5740cbb30 0x7fb5740ad750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59536/0 (socket says 172.21.15.89:59536) 2024-04-02T07:51:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb578a00700 1 -- 172.21.15.89:0/1502573434 learned_addr learned my addr 172.21.15.89:0/1502573434 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb579400700 1 --2- 172.21.15.89:0/1502573434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5740b2070 0x7fb5741546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb579e00700 1 --2- 172.21.15.89:0/1502573434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5740cd020 0x7fb574072010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb578a00700 1 -- 172.21.15.89:0/1502573434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5740b2070 msgr2=0x7fb5741546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb578a00700 1 --2- 172.21.15.89:0/1502573434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5740b2070 0x7fb5741546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb578a00700 1 -- 172.21.15.89:0/1502573434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5740cd020 msgr2=0x7fb574072010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb578a00700 1 --2- 172.21.15.89:0/1502573434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5740cd020 0x7fb574072010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb578a00700 1 -- 172.21.15.89:0/1502573434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5600094a0 con 0x7fb5740cbb30 2024-04-02T07:51:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb578a00700 1 --2- 172.21.15.89:0/1502573434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5740cbb30 0x7fb5740ad750 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb560009b90 tx=0x7fb560004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb571600700 1 -- 172.21.15.89:0/1502573434 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5600036a0 con 0x7fb5740cbb30 2024-04-02T07:51:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb571600700 1 -- 172.21.15.89:0/1502573434 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56001e070 con 0x7fb5740cbb30 2024-04-02T07:51:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb571600700 1 -- 172.21.15.89:0/1502573434 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560003a60 con 0x7fb5740cbb30 2024-04-02T07:51:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.695+0000 7fb57be00700 1 -- 172.21.15.89:0/1502573434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5740c8800 con 0x7fb5740cbb30 2024-04-02T07:51:39.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.696+0000 7fb57be00700 1 -- 172.21.15.89:0/1502573434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5740adc90 con 0x7fb5740cbb30 2024-04-02T07:51:39.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.697+0000 7fb57be00700 1 -- 172.21.15.89:0/1502573434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5740c9000 con 0x7fb5740cbb30 2024-04-02T07:51:39.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.698+0000 7fb571600700 1 -- 172.21.15.89:0/1502573434 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 34) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fb560003bc0 con 0x7fb5740cbb30 2024-04-02T07:51:39.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.698+0000 7fb571600700 1 --2- 172.21.15.89:0/1502573434 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb55406ae00 0x7fb55406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:39.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.699+0000 7fb571600700 1 -- 172.21.15.89:0/1502573434 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb56008a4b0 con 0x7fb5740cbb30 2024-04-02T07:51:39.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.699+0000 7fb579400700 1 -- 172.21.15.89:0/1502573434 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb55406ae00 msgr2=0x7fb55406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:39.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.699+0000 7fb579400700 1 --2- 172.21.15.89:0/1502573434 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb55406ae00 0x7fb55406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:39.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.703+0000 7fb571600700 1 -- 172.21.15.89:0/1502573434 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb5740c9000 con 0x7fb5740cbb30 2024-04-02T07:51:39.735 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:39.734+0000 7fad0dc4e240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:51:39.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.900+0000 7fb579400700 1 -- 172.21.15.89:0/1502573434 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb55406ae00 msgr2=0x7fb55406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:39.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.900+0000 7fb579400700 1 --2- 172.21.15.89:0/1502573434 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb55406ae00 0x7fb55406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.916+0000 7fb57be00700 1 -- 172.21.15.89:0/1502573434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fb5740c9000 con 0x7fb5740cbb30 2024-04-02T07:51:39.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.921+0000 7fb571600700 1 -- 172.21.15.89:0/1502573434 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb5740c9000 con 0x7fb5740cbb30 2024-04-02T07:51:39.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.923+0000 7fb57be00700 1 -- 172.21.15.89:0/1502573434 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb55406ae00 msgr2=0x7fb55406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:39.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.923+0000 7fb57be00700 1 --2- 172.21.15.89:0/1502573434 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb55406ae00 0x7fb55406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.923+0000 7fb57be00700 1 -- 172.21.15.89:0/1502573434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5740cbb30 msgr2=0x7fb5740ad750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:39.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.923+0000 7fb57be00700 1 --2- 172.21.15.89:0/1502573434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5740cbb30 0x7fb5740ad750 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb560009b90 tx=0x7fb560004750 comp rx=0 tx=0).stop 2024-04-02T07:51:39.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.923+0000 7fb57be00700 1 -- 172.21.15.89:0/1502573434 shutdown_connections 2024-04-02T07:51:39.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.923+0000 7fb57be00700 1 --2- 172.21.15.89:0/1502573434 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb55406ae00 0x7fb55406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.923+0000 7fb57be00700 1 --2- 172.21.15.89:0/1502573434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5740cd020 0x7fb574072010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.923+0000 7fb57be00700 1 --2- 172.21.15.89:0/1502573434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5740b2070 0x7fb5741546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.923+0000 7fb57be00700 1 --2- 172.21.15.89:0/1502573434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5740cbb30 0x7fb5740ad750 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:39.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.923+0000 7fb57be00700 1 -- 172.21.15.89:0/1502573434 >> 172.21.15.89:0/1502573434 conn(0x7fb5740b0d70 msgr2=0x7fb5740cbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:39.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.923+0000 7fb57be00700 1 -- 172.21.15.89:0/1502573434 shutdown_connections 2024-04-02T07:51:39.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:39.923+0000 7fb57be00700 1 -- 172.21.15.89:0/1502573434 wait complete. 2024-04-02T07:51:39.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:39.943 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-02T07:51:39.998 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-04-02T07:51:40.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.199+0000 7f9d51400700 1 Processor -- start 2024-04-02T07:51:40.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.200+0000 7f9d51400700 1 -- start start 2024-04-02T07:51:40.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.201+0000 7f9d51400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9d4c15f1e0 0x7f9d4c15f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:40.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.201+0000 7f9d51400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4c0cef40 0x7f9d4c15bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:40.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.201+0000 7f9d51400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4c0ccdd0 con 0x7f9d4c0cf3f0 2024-04-02T07:51:40.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.201+0000 7f9d51400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4c0cd260 con 0x7f9d4c15f1e0 2024-04-02T07:51:40.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.201+0000 7f9d51400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c1600c0 con 0x7f9d4c0cef40 2024-04-02T07:51:40.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.201+0000 7f9d4a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9d4c15f1e0 0x7f9d4c15f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:40.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.201+0000 7f9d4b200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9d4c0cf3f0 0x7f9d4c15c010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55874/0 (socket says 172.21.15.89:55874) 2024-04-02T07:51:40.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.201+0000 7f9d4b200700 1 -- 172.21.15.89:0/3015822953 learned_addr learned my addr 172.21.15.89:0/3015822953 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:40.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.201+0000 7f9d49e00700 1 --2- 172.21.15.89:0/3015822953 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4c0cef40 0x7f9d4c15bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:40.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.202+0000 7f9d4a800700 1 -- 172.21.15.89:0/3015822953 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4c0cef40 msgr2=0x7f9d4c15bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:40.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.202+0000 7f9d4a800700 1 --2- 172.21.15.89:0/3015822953 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4c0cef40 0x7f9d4c15bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:40.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.202+0000 7f9d4a800700 1 -- 172.21.15.89:0/3015822953 >> v1:172.21.15.89:6789/0 conn(0x7f9d4c0cf3f0 legacy=0x7f9d4c15c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:40.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.202+0000 7f9d4a800700 1 -- 172.21.15.89:0/3015822953 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c15fb00 con 0x7f9d4c15f1e0 2024-04-02T07:51:40.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.202+0000 7f9d4a800700 1 --2- 172.21.15.89:0/3015822953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d4c15f1e0 0x7f9d4c15f5c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009840 tx=0x7f9d3c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c6006f4d752c72c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:40.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.202+0000 7f9d49400700 1 -- 172.21.15.89:0/3015822953 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c011830 con 0x7f9d4c15f1e0 2024-04-02T07:51:40.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.202+0000 7f9d49400700 1 -- 172.21.15.89:0/3015822953 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3c011990 con 0x7f9d4c15f1e0 2024-04-02T07:51:40.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.202+0000 7f9d49400700 1 -- 172.21.15.89:0/3015822953 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c011d20 con 0x7f9d4c15f1e0 2024-04-02T07:51:40.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.203+0000 7f9d51400700 1 -- 172.21.15.89:0/3015822953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d4c15f1e0 msgr2=0x7f9d4c15f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:40.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.203+0000 7f9d51400700 1 --2- 172.21.15.89:0/3015822953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d4c15f1e0 0x7f9d4c15f5c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009840 tx=0x7f9d3c009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:40.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.203+0000 7f9d51400700 1 -- 172.21.15.89:0/3015822953 shutdown_connections 2024-04-02T07:51:40.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.203+0000 7f9d51400700 1 --2- 172.21.15.89:0/3015822953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d4c15f1e0 0x7f9d4c15f5c0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:40.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.203+0000 7f9d51400700 1 --2- 172.21.15.89:0/3015822953 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4c0cef40 0x7f9d4c15bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:40.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.203+0000 7f9d51400700 1 -- 172.21.15.89:0/3015822953 >> 172.21.15.89:0/3015822953 conn(0x7f9d4c0d1370 msgr2=0x7f9d4c0d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:40.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.203+0000 7f9d51400700 1 -- 172.21.15.89:0/3015822953 shutdown_connections 2024-04-02T07:51:40.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.203+0000 7f9d51400700 1 -- 172.21.15.89:0/3015822953 wait complete. 2024-04-02T07:51:40.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.205+0000 7f9d51400700 1 Processor -- start 2024-04-02T07:51:40.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.205+0000 7f9d51400700 1 -- start start 2024-04-02T07:51:40.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d51400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d4c0cef40 0x7f9d4c15e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:40.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d51400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d4c0cf3f0 0x7f9d4c0b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:40.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d51400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4c15f1e0 0x7f9d4c0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:40.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d51400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c0b4700 con 0x7f9d4c0cf3f0 2024-04-02T07:51:40.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d51400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c0b5e70 con 0x7f9d4c0cef40 2024-04-02T07:51:40.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d51400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c0b6150 con 0x7f9d4c15f1e0 2024-04-02T07:51:40.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d49e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d4c0cf3f0 0x7f9d4c0b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:40.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d49e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d4c0cf3f0 0x7f9d4c0b5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59558/0 (socket says 172.21.15.89:59558) 2024-04-02T07:51:40.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d49e00700 1 -- 172.21.15.89:0/4289419988 learned_addr learned my addr 172.21.15.89:0/4289419988 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:40.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d4a800700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d4c0cef40 0x7f9d4c15e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:40.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d4b200700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4c15f1e0 0x7f9d4c0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:40.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d49e00700 1 -- 172.21.15.89:0/4289419988 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4c15f1e0 msgr2=0x7f9d4c0b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:40.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d49e00700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4c15f1e0 0x7f9d4c0b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:40.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d49e00700 1 -- 172.21.15.89:0/4289419988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d4c0cef40 msgr2=0x7f9d4c15e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:40.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d49e00700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d4c0cef40 0x7f9d4c15e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:40.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.206+0000 7f9d49e00700 1 -- 172.21.15.89:0/4289419988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c0094a0 con 0x7f9d4c0cf3f0 2024-04-02T07:51:40.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.207+0000 7f9d49e00700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d4c0cf3f0 0x7f9d4c0b5030 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800e9b0 tx=0x7f9d3800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:40.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.207+0000 7f9d42a00700 1 -- 172.21.15.89:0/4289419988 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d38009710 con 0x7f9d4c0cf3f0 2024-04-02T07:51:40.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.207+0000 7f9d42a00700 1 -- 172.21.15.89:0/4289419988 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3800c850 con 0x7f9d4c0cf3f0 2024-04-02T07:51:40.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.207+0000 7f9d42a00700 1 -- 172.21.15.89:0/4289419988 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3800cbc0 con 0x7f9d4c0cf3f0 2024-04-02T07:51:40.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.207+0000 7f9d51400700 1 -- 172.21.15.89:0/4289419988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4c0b6430 con 0x7f9d4c0cf3f0 2024-04-02T07:51:40.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.207+0000 7f9d51400700 1 -- 172.21.15.89:0/4289419988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4c0b6710 con 0x7f9d4c0cf3f0 2024-04-02T07:51:40.210 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:40.209+0000 7f49d7392240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:51:40.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.208+0000 7f9d51400700 1 -- 172.21.15.89:0/4289419988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d4c067720 con 0x7f9d4c0cf3f0 2024-04-02T07:51:40.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.210+0000 7f9d42a00700 1 -- 172.21.15.89:0/4289419988 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 34) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f9d3800cd20 con 0x7f9d4c0cf3f0 2024-04-02T07:51:40.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.211+0000 7f9d42a00700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 0x7f9d2c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:40.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.211+0000 7f9d42a00700 1 -- 172.21.15.89:0/4289419988 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9d380896b0 con 0x7f9d4c0cf3f0 2024-04-02T07:51:40.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.211+0000 7f9d4a800700 1 -- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 msgr2=0x7f9d2c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:40.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.211+0000 7f9d4a800700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 0x7f9d2c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:40.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.215+0000 7f9d42a00700 1 -- 172.21.15.89:0/4289419988 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9d38054620 con 0x7f9d4c0cf3f0 2024-04-02T07:51:40.306 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:40.304+0000 7fad0dc4e240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:51:40.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.412+0000 7f9d4a800700 1 -- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 msgr2=0x7f9d2c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:40.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.412+0000 7f9d4a800700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 0x7f9d2c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:40.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.452+0000 7f9d51400700 1 -- 172.21.15.89:0/4289419988 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d4c0cc780 con 0x7f9d2c06ae00 2024-04-02T07:51:40.461 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:40.460+0000 7f49d7392240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:51:40.566 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:40.565+0000 7fad0dc4e240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:51:40.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.813+0000 7f9d4a800700 1 -- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 msgr2=0x7f9d2c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:40.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:40.813+0000 7f9d4a800700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 0x7f9d2c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T07:51:40.910 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:40.909+0000 7f49d7392240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:51:41.024 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:41.022+0000 7fad0dc4e240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:51:41.608 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:41.607+0000 7f49d7392240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:51:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:41.614+0000 7f9d4a800700 1 -- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 msgr2=0x7f9d2c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:41.614+0000 7f9d4a800700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 0x7f9d2c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T07:51:41.696 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:41.695+0000 7f49d7392240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:51:41.732 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:41.730+0000 7fad0dc4e240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:51:41.824 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:41.822+0000 7fad0dc4e240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:51:41.851 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:41.850+0000 7f49d7392240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:51:41.982 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:41.981+0000 7fad0dc4e240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:51:42.059 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:42.058+0000 7f49d7392240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:51:42.149 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:42.148+0000 7f49d7392240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:51:42.196 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:42.194+0000 7fad0dc4e240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:51:42.246 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:42.245+0000 7f49d7392240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:51:42.289 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:42.287+0000 7fad0dc4e240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:51:42.341 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:42.340+0000 7f49d7392240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:51:42.388 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:42.386+0000 7fad0dc4e240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:51:42.437 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:42.436+0000 7f49d7392240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:51:42.487 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:42.484+0000 7fad0dc4e240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:51:42.585 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:42.583+0000 7fad0dc4e240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:51:42.643 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:42.642+0000 7f49d7392240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:51:42.797 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:42.796+0000 7fad0dc4e240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:51:42.894 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:42.894+0000 7f49d7392240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:51:43.054 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:43.052+0000 7fad0dc4e240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:51:43.084 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:43.083+0000 7f49d7392240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:51:43.175 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:43.174+0000 7f49d7392240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:51:43.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:43.216+0000 7f9d4a800700 1 -- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 msgr2=0x7f9d2c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:43.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:43.216+0000 7f9d4a800700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 0x7f9d2c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T07:51:43.246 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:43.244+0000 7fad0dc4e240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:51:43.338 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:43.336+0000 7fad0dc4e240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:51:43.899 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:43.898+0000 7f49d7392240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:51:44.064 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:44.063+0000 7fad0dc4e240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:51:44.109 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:44.108+0000 7f49d7392240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:51:44.278 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:44.277+0000 7fad0dc4e240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:51:44.314 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:44.313+0000 7f49d7392240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:51:44.486 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:44.485+0000 7fad0dc4e240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:51:44.787 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:44.786+0000 7f49d7392240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:51:44.963 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:44.962+0000 7fad0dc4e240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:51:44.990 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:44.988+0000 7f49d7392240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:51:45.170 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:45.169+0000 7fad0dc4e240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:51:45.424 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:45.423+0000 7f49d7392240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:51:45.533 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:45.531+0000 7f49d7392240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:51:45.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:45.591+0000 7f9d42a00700 1 -- 172.21.15.89:0/4289419988 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 35) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f9d38087a70 con 0x7f9d4c0cf3f0 2024-04-02T07:51:45.614 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:45.612+0000 7fad0dc4e240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:51:45.729 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:45.728+0000 7fad0dc4e240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:51:45.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:45.738+0000 7f9d42a00700 1 -- 172.21.15.89:0/4289419988 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 36) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f9d3806fef0 con 0x7f9d4c0cf3f0 2024-04-02T07:51:45.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:45.738+0000 7f9d42a00700 1 -- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 msgr2=0x7f9d2c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:45.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:45.738+0000 7f9d42a00700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 0x7f9d2c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:45.920 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:45.918+0000 7fac76400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:45.920 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:45.918+0000 7fac76400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:45.920 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:45.918+0000 7fac76400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:45.920 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:45.918+0000 7fac76400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:45.920 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:45.918+0000 7fac76400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:45.921 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:45.919+0000 7fac7a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:45.921 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:45.919+0000 7fac7a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:45.921 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:45.919+0000 7fac7a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:45.921 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:45.919+0000 7fac7a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:45.921 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:45.919+0000 7fac7a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:46.039 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:51:46.043 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:51:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:46.743+0000 7f9d42a00700 1 -- 172.21.15.89:0/4289419988 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mgrmap(e 37) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f9d380552c0 con 0x7f9d4c0cf3f0 2024-04-02T07:51:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:46.744+0000 7f9d42a00700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 0x7f9d2c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:46.744+0000 7f9d42a00700 1 -- 172.21.15.89:0/4289419988 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d2c035100 con 0x7f9d2c06ae00 2024-04-02T07:51:46.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:46.746+0000 7f9d4a800700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 0x7f9d2c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:46.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:46.750+0000 7f9d4a800700 1 --2- 172.21.15.89:0/4289419988 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d2c06ae00 0x7f9d2c06d2c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009fd0 tx=0x7f9d3c01c040 comp rx=0 tx=0).ready entity=mgr.6621 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:46.761 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:46.760+0000 7fac82c00700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-04-02T07:51:46.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:46.760+0000 7f9d42a00700 1 -- 172.21.15.89:0/4289419988 <== mgr.6621 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f9d3c0117d0 con 0x7f9d2c06ae00 2024-04-02T07:51:46.763 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-04-02T07:51:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-02T07:51:46.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:46.772 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-02T07:51:46.819 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T07:51:47.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.050+0000 7f75df600700 1 Processor -- start 2024-04-02T07:51:47.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.050+0000 7f75df600700 1 -- start start 2024-04-02T07:51:47.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.051+0000 7f75df600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75d815a5c0 0x7f75d80709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:47.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.051+0000 7f75df600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f75d815dfe0 0x7f75d80ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:47.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.051+0000 7f75df600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d80cb310 con 0x7f75d80ce670 2024-04-02T07:51:47.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.051+0000 7f75df600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d80ceb00 con 0x7f75d815dfe0 2024-04-02T07:51:47.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.051+0000 7f75df600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d8070f00 con 0x7f75d815a5c0 2024-04-02T07:51:47.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.052+0000 7f75d7e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f75d815dfe0 0x7f75d80ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:47.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.052+0000 7f75dcc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75d815a5c0 0x7f75d80709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:47.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.052+0000 7f75dcc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75d815a5c0 0x7f75d80709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:44246/0 (socket says 172.21.15.89:44246) 2024-04-02T07:51:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.052+0000 7f75dd600700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f75d80ce670 0x7f75d80b2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:52444/0 (socket says 172.21.15.89:52444) 2024-04-02T07:51:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.052+0000 7f75d7e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f75d815dfe0 0x7f75d80ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:54758/0 (socket says 172.21.15.89:54758) 2024-04-02T07:51:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.052+0000 7f75dcc00700 1 -- 172.21.15.89:0/2882275097 learned_addr learned my addr 172.21.15.89:0/2882275097 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.052+0000 7f75dcc00700 1 -- 172.21.15.89:0/2882275097 >> v2:172.21.15.113:3300/0 conn(0x7f75d815dfe0 msgr2=0x7f75d80ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.052+0000 7f75dcc00700 1 --2- 172.21.15.89:0/2882275097 >> v2:172.21.15.113:3300/0 conn(0x7f75d815dfe0 0x7f75d80ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.052+0000 7f75dcc00700 1 -- 172.21.15.89:0/2882275097 >> v1:172.21.15.113:6789/0 conn(0x7f75d80ce670 legacy=0x7f75d80b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.052+0000 7f75dcc00700 1 -- 172.21.15.89:0/2882275097 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d80cca90 con 0x7f75d815a5c0 2024-04-02T07:51:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.052+0000 7f75d7e00700 1 --2- 172.21.15.89:0/2882275097 >> v2:172.21.15.113:3300/0 conn(0x7f75d815dfe0 0x7f75d80ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:47.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.052+0000 7f75dcc00700 1 --2- 172.21.15.89:0/2882275097 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75d815a5c0 0x7f75d80709c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0098f0 tx=0x7f75cc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d8dd0e9fce87e62 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:47.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.053+0000 7f75d7400700 1 -- 172.21.15.89:0/2882275097 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc011830 con 0x7f75d815a5c0 2024-04-02T07:51:47.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.053+0000 7f75d7400700 1 -- 172.21.15.89:0/2882275097 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75cc011990 con 0x7f75d815a5c0 2024-04-02T07:51:47.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.053+0000 7f75d7400700 1 -- 172.21.15.89:0/2882275097 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc011d20 con 0x7f75d815a5c0 2024-04-02T07:51:47.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.053+0000 7f75df600700 1 -- 172.21.15.89:0/2882275097 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75d815a5c0 msgr2=0x7f75d80709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:47.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.053+0000 7f75df600700 1 --2- 172.21.15.89:0/2882275097 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75d815a5c0 0x7f75d80709c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0098f0 tx=0x7f75cc009c00 comp rx=0 tx=0).stop 2024-04-02T07:51:47.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.053+0000 7f75df600700 1 -- 172.21.15.89:0/2882275097 shutdown_connections 2024-04-02T07:51:47.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.053+0000 7f75df600700 1 --2- 172.21.15.89:0/2882275097 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75d815a5c0 0x7f75d80709c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:47.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.053+0000 7f75df600700 1 --2- 172.21.15.89:0/2882275097 >> v2:172.21.15.113:3300/0 conn(0x7f75d815dfe0 0x7f75d80ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:47.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.053+0000 7f75df600700 1 -- 172.21.15.89:0/2882275097 >> 172.21.15.89:0/2882275097 conn(0x7f75d80b0d70 msgr2=0x7f75d80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:47.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.053+0000 7f75df600700 1 -- 172.21.15.89:0/2882275097 shutdown_connections 2024-04-02T07:51:47.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.053+0000 7f75df600700 1 -- 172.21.15.89:0/2882275097 wait complete. 2024-04-02T07:51:47.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.055+0000 7f75df600700 1 Processor -- start 2024-04-02T07:51:47.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.055+0000 7f75df600700 1 -- start start 2024-04-02T07:51:47.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.056+0000 7f75df600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75d80ce670 0x7f75d8098900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:47.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.056+0000 7f75df600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75d815a5c0 0x7f75d815c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.056+0000 7f75df600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75d815dfe0 0x7f75d815cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.056+0000 7f75df600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d8156760 con 0x7f75d815dfe0 2024-04-02T07:51:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.056+0000 7f75df600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d815b840 con 0x7f75d815a5c0 2024-04-02T07:51:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.056+0000 7f75df600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d815bb20 con 0x7f75d80ce670 2024-04-02T07:51:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.057+0000 7f75dcc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75d80ce670 0x7f75d8098900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.057+0000 7f75d7e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75d815a5c0 0x7f75d815c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.057+0000 7f75dcc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75d80ce670 0x7f75d8098900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:44254/0 (socket says 172.21.15.89:44254) 2024-04-02T07:51:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.057+0000 7f75dcc00700 1 -- 172.21.15.89:0/4272044659 learned_addr learned my addr 172.21.15.89:0/4272044659 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.057+0000 7f75dd600700 1 --2- 172.21.15.89:0/4272044659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75d815dfe0 0x7f75d815cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.057+0000 7f75dcc00700 1 -- 172.21.15.89:0/4272044659 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75d815a5c0 msgr2=0x7f75d815c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.057+0000 7f75dcc00700 1 --2- 172.21.15.89:0/4272044659 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75d815a5c0 0x7f75d815c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.057+0000 7f75dcc00700 1 -- 172.21.15.89:0/4272044659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75d815dfe0 msgr2=0x7f75d815cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.057+0000 7f75dcc00700 1 --2- 172.21.15.89:0/4272044659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75d815dfe0 0x7f75d815cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.057+0000 7f75dcc00700 1 -- 172.21.15.89:0/4272044659 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c40094a0 con 0x7f75d80ce670 2024-04-02T07:51:47.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.057+0000 7f75dcc00700 1 --2- 172.21.15.89:0/4272044659 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75d80ce670 0x7f75d8098900 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f75cc000c00 tx=0x7f75cc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:47.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.058+0000 7f75d4c00700 1 -- 172.21.15.89:0/4272044659 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc004480 con 0x7f75d80ce670 2024-04-02T07:51:47.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.058+0000 7f75d4c00700 1 -- 172.21.15.89:0/4272044659 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75cc01f070 con 0x7f75d80ce670 2024-04-02T07:51:47.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.058+0000 7f75d4c00700 1 -- 172.21.15.89:0/4272044659 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc0038c0 con 0x7f75d80ce670 2024-04-02T07:51:47.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.058+0000 7f75df600700 1 -- 172.21.15.89:0/4272044659 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc009550 con 0x7f75d80ce670 2024-04-02T07:51:47.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.058+0000 7f75df600700 1 -- 172.21.15.89:0/4272044659 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d815c0e0 con 0x7f75d80ce670 2024-04-02T07:51:47.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.059+0000 7f75df600700 1 -- 172.21.15.89:0/4272044659 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d8067720 con 0x7f75d80ce670 2024-04-02T07:51:47.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.060+0000 7f75d4c00700 1 -- 172.21.15.89:0/4272044659 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 37) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f75cc021030 con 0x7f75d80ce670 2024-04-02T07:51:47.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.060+0000 7f75d4c00700 1 --2- 172.21.15.89:0/4272044659 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f75b806ab30 0x7f75b806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:47.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.061+0000 7f75d4c00700 1 -- 172.21.15.89:0/4272044659 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f75cc08a7d0 con 0x7f75d80ce670 2024-04-02T07:51:47.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.061+0000 7f75d7e00700 1 --2- 172.21.15.89:0/4272044659 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f75b806ab30 0x7f75b806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:47.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.061+0000 7f75d7e00700 1 --2- 172.21.15.89:0/4272044659 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f75b806ab30 0x7f75b806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f75d815df40 tx=0x7f75c4009210 comp rx=0 tx=0).ready entity=mgr.6621 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:47.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.066+0000 7f75d4c00700 1 -- 172.21.15.89:0/4272044659 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f75cc055b10 con 0x7f75d80ce670 2024-04-02T07:51:47.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.326+0000 7f75df600700 1 -- 172.21.15.89:0/4272044659 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f75d815c270 con 0x7f75d80ce670 2024-04-02T07:51:47.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.753+0000 7f75d4c00700 1 -- 172.21.15.89:0/4272044659 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f75cc08cce0 con 0x7f75d80ce670 2024-04-02T07:51:47.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.754+0000 7f75d4c00700 1 -- 172.21.15.89:0/4272044659 <== mon.2 v2:172.21.15.113:3301/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v38) v1 ==== 93+0+0 (secure 0 0 0) 0x7f75cc06d610 con 0x7f75d80ce670 2024-04-02T07:51:47.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.757+0000 7f75df600700 1 -- 172.21.15.89:0/4272044659 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f75b806ab30 msgr2=0x7f75b806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:47.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.757+0000 7f75df600700 1 --2- 172.21.15.89:0/4272044659 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f75b806ab30 0x7f75b806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f75d815df40 tx=0x7f75c4009210 comp rx=0 tx=0).stop 2024-04-02T07:51:47.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.757+0000 7f75df600700 1 -- 172.21.15.89:0/4272044659 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75d80ce670 msgr2=0x7f75d8098900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:47.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.757+0000 7f75df600700 1 --2- 172.21.15.89:0/4272044659 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75d80ce670 0x7f75d8098900 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f75cc000c00 tx=0x7f75cc004750 comp rx=0 tx=0).stop 2024-04-02T07:51:47.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.757+0000 7f75df600700 1 -- 172.21.15.89:0/4272044659 shutdown_connections 2024-04-02T07:51:47.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.757+0000 7f75df600700 1 --2- 172.21.15.89:0/4272044659 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f75b806ab30 0x7f75b806cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:47.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.757+0000 7f75df600700 1 --2- 172.21.15.89:0/4272044659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75d815dfe0 0x7f75d815cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:47.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.757+0000 7f75df600700 1 --2- 172.21.15.89:0/4272044659 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f75d80ce670 0x7f75d8098900 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:47.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.757+0000 7f75df600700 1 --2- 172.21.15.89:0/4272044659 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75d815a5c0 0x7f75d815c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:47.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.757+0000 7f75df600700 1 -- 172.21.15.89:0/4272044659 >> 172.21.15.89:0/4272044659 conn(0x7f75d80b0d70 msgr2=0x7f75d80aef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:47.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.758+0000 7f75df600700 1 -- 172.21.15.89:0/4272044659 shutdown_connections 2024-04-02T07:51:47.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:47.758+0000 7f75df600700 1 -- 172.21.15.89:0/4272044659 wait complete. 2024-04-02T07:51:47.779 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:51:47.779 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:51:47.876 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:51:47.961 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:51:47.961 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:51:47.961 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:47.974 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:47.972+0000 7f99d5098240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:51:47.976 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:47.975+0000 7fe5e17b3240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:51:48.005 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:51:48.005 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:48.091 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:51:48.160 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:48.158+0000 7f99d5098240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:51:48.165 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:48.164+0000 7fe5e17b3240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:51:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.341+0000 7f7f4fc00700 1 Processor -- start 2024-04-02T07:51:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.342+0000 7f7f4fc00700 1 -- start start 2024-04-02T07:51:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.343+0000 7f7f4fc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7f4815d670 0x7f7f480ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.344+0000 7f7f4fc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f480b3890 con 0x7f7f4815da50 2024-04-02T07:51:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.344+0000 7f7f4fc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f480acac0 con 0x7f7f4806dee0 2024-04-02T07:51:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.344+0000 7f7f4fc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f48159f50 con 0x7f7f4815d670 2024-04-02T07:51:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.344+0000 7f7f4d200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7f4815d670 0x7f7f480ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.344+0000 7f7f47e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7f4815da50 0x7f7f480b2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55888/0 (socket says 172.21.15.89:55888) 2024-04-02T07:51:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.344+0000 7f7f47e00700 1 -- 172.21.15.89:0/4204744120 learned_addr learned my addr 172.21.15.89:0/4204744120 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.344+0000 7f7f4d200700 1 -- 172.21.15.89:0/4204744120 >> v1:172.21.15.113:6789/0 conn(0x7f7f4806dee0 legacy=0x7f7f4806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.344+0000 7f7f4d200700 1 -- 172.21.15.89:0/4204744120 >> v1:172.21.15.89:6789/0 conn(0x7f7f4815da50 legacy=0x7f7f480b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.345+0000 7f7f4d200700 1 -- 172.21.15.89:0/4204744120 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f480b35b0 con 0x7f7f4815d670 2024-04-02T07:51:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.344+0000 7f7f47400700 1 -- 172.21.15.89:0/4204744120 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317986678 0 0) 0x7f7f480b3890 con 0x7f7f4815da50 2024-04-02T07:51:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.345+0000 7f7f4d200700 1 --2- 172.21.15.89:0/4204744120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4815d670 0x7f7f480ac580 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c009840 tx=0x7f7f3c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0faba6875e12507 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.345+0000 7f7f47400700 1 -- 172.21.15.89:0/4204744120 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c011830 con 0x7f7f4815d670 2024-04-02T07:51:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.345+0000 7f7f47400700 1 -- 172.21.15.89:0/4204744120 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3c011990 con 0x7f7f4815d670 2024-04-02T07:51:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.345+0000 7f7f47400700 1 -- 172.21.15.89:0/4204744120 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c011d20 con 0x7f7f4815d670 2024-04-02T07:51:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.346+0000 7f7f4fc00700 1 -- 172.21.15.89:0/4204744120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4815d670 msgr2=0x7f7f480ac580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.346+0000 7f7f4fc00700 1 --2- 172.21.15.89:0/4204744120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4815d670 0x7f7f480ac580 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c009840 tx=0x7f7f3c009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.346+0000 7f7f4fc00700 1 -- 172.21.15.89:0/4204744120 shutdown_connections 2024-04-02T07:51:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.346+0000 7f7f4fc00700 1 --2- 172.21.15.89:0/4204744120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4815d670 0x7f7f480ac580 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.346+0000 7f7f4fc00700 1 -- 172.21.15.89:0/4204744120 >> 172.21.15.89:0/4204744120 conn(0x7f7f480cf190 msgr2=0x7f7f480cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.346+0000 7f7f4fc00700 1 -- 172.21.15.89:0/4204744120 shutdown_connections 2024-04-02T07:51:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.346+0000 7f7f4fc00700 1 -- 172.21.15.89:0/4204744120 wait complete. 2024-04-02T07:51:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.348+0000 7f7f4fc00700 1 Processor -- start 2024-04-02T07:51:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.348+0000 7f7f4fc00700 1 -- start start 2024-04-02T07:51:48.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4fc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f4806dee0 0x7f7f48098b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:48.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4fc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f4815d670 0x7f7f48150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:48.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4fc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4815da50 0x7f7f480b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:48.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4fc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f48151440 con 0x7f7f4815da50 2024-04-02T07:51:48.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4fc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f480cc5c0 con 0x7f7f4806dee0 2024-04-02T07:51:48.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4fc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f48152710 con 0x7f7f4815d670 2024-04-02T07:51:48.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4dc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4815da50 0x7f7f480b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:48.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4dc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4815da50 0x7f7f480b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59722/0 (socket says 172.21.15.89:59722) 2024-04-02T07:51:48.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4dc00700 1 -- 172.21.15.89:0/490942493 learned_addr learned my addr 172.21.15.89:0/490942493 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:48.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4d200700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f4806dee0 0x7f7f48098b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:48.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f47e00700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f4815d670 0x7f7f48150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:48.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4dc00700 1 -- 172.21.15.89:0/490942493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f4815d670 msgr2=0x7f7f48150f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:48.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4dc00700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f4815d670 0x7f7f48150f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:48.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4dc00700 1 -- 172.21.15.89:0/490942493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f4806dee0 msgr2=0x7f7f48098b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:48.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4dc00700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f4806dee0 0x7f7f48098b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:48.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f4dc00700 1 -- 172.21.15.89:0/490942493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3c0094a0 con 0x7f7f4815da50 2024-04-02T07:51:48.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.349+0000 7f7f47e00700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f4815d670 0x7f7f48150f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:48.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.350+0000 7f7f4dc00700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4815da50 0x7f7f480b2470 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000cc50 tx=0x7f7f4000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:48.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.350+0000 7f7f44c00700 1 -- 172.21.15.89:0/490942493 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f40003c00 con 0x7f7f4815da50 2024-04-02T07:51:48.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.350+0000 7f7f44c00700 1 -- 172.21.15.89:0/490942493 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f40003d60 con 0x7f7f4815da50 2024-04-02T07:51:48.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.350+0000 7f7f4fc00700 1 -- 172.21.15.89:0/490942493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f48151670 con 0x7f7f4815da50 2024-04-02T07:51:48.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.350+0000 7f7f44c00700 1 -- 172.21.15.89:0/490942493 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4000ab60 con 0x7f7f4815da50 2024-04-02T07:51:48.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.350+0000 7f7f4fc00700 1 -- 172.21.15.89:0/490942493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f48151c30 con 0x7f7f4815da50 2024-04-02T07:51:48.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.351+0000 7f7f4fc00700 1 -- 172.21.15.89:0/490942493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f480ae9a0 con 0x7f7f4815da50 2024-04-02T07:51:48.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.352+0000 7f7f44c00700 1 -- 172.21.15.89:0/490942493 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f7f4000acc0 con 0x7f7f4815da50 2024-04-02T07:51:48.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.353+0000 7f7f44c00700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7f2806adb0 0x7f7f2806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:48.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.353+0000 7f7f4d200700 1 -- 172.21.15.89:0/490942493 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7f2806adb0 msgr2=0x7f7f2806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:48.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.353+0000 7f7f44c00700 1 -- 172.21.15.89:0/490942493 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f7f400897b0 con 0x7f7f4815da50 2024-04-02T07:51:48.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.353+0000 7f7f4d200700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7f2806adb0 0x7f7f2806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:48.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.358+0000 7f7f44c00700 1 -- 172.21.15.89:0/490942493 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7f400545b0 con 0x7f7f4815da50 2024-04-02T07:51:48.476 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:48.475+0000 7f99d5098240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:51:48.498 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:48.496+0000 7fe5e17b3240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:51:48.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.554+0000 7f7f4d200700 1 -- 172.21.15.89:0/490942493 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7f2806adb0 msgr2=0x7f7f2806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:48.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.554+0000 7f7f4d200700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7f2806adb0 0x7f7f2806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:48.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.566+0000 7f7f4fc00700 1 -- 172.21.15.89:0/490942493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f48067720 con 0x7f7f4815da50 2024-04-02T07:51:48.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.567+0000 7f7f44c00700 1 -- 172.21.15.89:0/490942493 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+17111 (secure 0 0 0) 0x7f7f40057bd0 con 0x7f7f4815da50 2024-04-02T07:51:48.569 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:48.569 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":84,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:51:45.730961+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:51:27.926367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-02T07:51:28.662600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/822661295":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/907732608":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/3083880072":"2024-04-03T07:51:45.730929+0000","172.21.15.89:6833/43357":"2024-04-03T07:51:45.730929+0000","172.21.15.89:6832/43357":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/1211892288":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/23281819":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/3435265311":"2024-04-03T07:51:45.730929+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:51:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.570+0000 7f7f4fc00700 1 -- 172.21.15.89:0/490942493 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7f2806adb0 msgr2=0x7f7f2806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.570+0000 7f7f4fc00700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7f2806adb0 0x7f7f2806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.570+0000 7f7f4fc00700 1 -- 172.21.15.89:0/490942493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4815da50 msgr2=0x7f7f480b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.570+0000 7f7f4fc00700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4815da50 0x7f7f480b2470 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000cc50 tx=0x7f7f4000cf60 comp rx=0 tx=0).stop 2024-04-02T07:51:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.570+0000 7f7f4fc00700 1 -- 172.21.15.89:0/490942493 shutdown_connections 2024-04-02T07:51:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.570+0000 7f7f4fc00700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7f2806adb0 0x7f7f2806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.570+0000 7f7f4fc00700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f4806dee0 0x7f7f48098b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.570+0000 7f7f4fc00700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4815da50 0x7f7f480b2470 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.570+0000 7f7f4fc00700 1 --2- 172.21.15.89:0/490942493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f4815d670 0x7f7f48150f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.570+0000 7f7f4fc00700 1 -- 172.21.15.89:0/490942493 >> 172.21.15.89:0/490942493 conn(0x7f7f480cf190 msgr2=0x7f7f4806c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.570+0000 7f7f4fc00700 1 -- 172.21.15.89:0/490942493 shutdown_connections 2024-04-02T07:51:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.570+0000 7f7f4fc00700 1 -- 172.21.15.89:0/490942493 wait complete. 2024-04-02T07:51:48.590 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:51:27.926367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:51:28.662600+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:51:48.591 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:51:48.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.828+0000 7f61ff400700 1 Processor -- start 2024-04-02T07:51:48.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.829+0000 7f61ff400700 1 -- start start 2024-04-02T07:51:48.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.830+0000 7f61ff400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61f80ce230 0x7f61f80ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:48.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.830+0000 7f61ff400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f80709c0 con 0x7f61f80b2070 2024-04-02T07:51:48.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.830+0000 7f61ff400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f8070c80 con 0x7f61f815d800 2024-04-02T07:51:48.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.830+0000 7f61ff400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f80b40f0 con 0x7f61f80ce230 2024-04-02T07:51:48.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.831+0000 7f61fd400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f61f80b2070 0x7f61f815dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45972/0 (socket says 172.21.15.89:45972) 2024-04-02T07:51:48.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.831+0000 7f61fd400700 1 -- 172.21.15.89:0/2397487654 learned_addr learned my addr 172.21.15.89:0/2397487654 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:48.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.831+0000 7f61f7e00700 1 --2- 172.21.15.89:0/2397487654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61f80ce230 0x7f61f80ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:48.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.831+0000 7f61f7400700 1 -- 172.21.15.89:0/2397487654 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688093926 0 0) 0x7f61f80709c0 con 0x7f61f80b2070 2024-04-02T07:51:48.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.831+0000 7f61f7400700 1 -- 172.21.15.89:0/2397487654 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e00035b0 con 0x7f61f80b2070 2024-04-02T07:51:48.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.831+0000 7f61f7400700 1 -- 172.21.15.89:0/2397487654 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552505685 0 0) 0x7f61f8070c80 con 0x7f61f815d800 2024-04-02T07:51:48.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.831+0000 7f61f7400700 1 -- 172.21.15.89:0/2397487654 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f80709c0 con 0x7f61f815d800 2024-04-02T07:51:48.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.831+0000 7f61f7e00700 1 -- 172.21.15.89:0/2397487654 >> v1:172.21.15.113:6789/0 conn(0x7f61f815d800 legacy=0x7f61f80725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:48.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.831+0000 7f61f7e00700 1 -- 172.21.15.89:0/2397487654 >> v1:172.21.15.89:6789/0 conn(0x7f61f80b2070 legacy=0x7f61f815dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:48.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.831+0000 7f61f7e00700 1 -- 172.21.15.89:0/2397487654 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8070f60 con 0x7f61f80ce230 2024-04-02T07:51:48.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.831+0000 7f61f7e00700 1 --2- 172.21.15.89:0/2397487654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61f80ce230 0x7f61f80ce670 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f61e400aa10 tx=0x7f61e400ad20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5519e812db6dd1d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:48.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.832+0000 7f61f7400700 1 -- 172.21.15.89:0/2397487654 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e400cb70 con 0x7f61f80ce230 2024-04-02T07:51:48.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.832+0000 7f61f7400700 1 -- 172.21.15.89:0/2397487654 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e400ccd0 con 0x7f61f80ce230 2024-04-02T07:51:48.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.832+0000 7f61f7400700 1 -- 172.21.15.89:0/2397487654 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e4018600 con 0x7f61f80ce230 2024-04-02T07:51:48.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.833+0000 7f61ff400700 1 -- 172.21.15.89:0/2397487654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61f80ce230 msgr2=0x7f61f80ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:48.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.833+0000 7f61ff400700 1 --2- 172.21.15.89:0/2397487654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61f80ce230 0x7f61f80ce670 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f61e400aa10 tx=0x7f61e400ad20 comp rx=0 tx=0).stop 2024-04-02T07:51:48.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.833+0000 7f61ff400700 1 -- 172.21.15.89:0/2397487654 shutdown_connections 2024-04-02T07:51:48.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.833+0000 7f61ff400700 1 --2- 172.21.15.89:0/2397487654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61f80ce230 0x7f61f80ce670 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:48.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.833+0000 7f61ff400700 1 -- 172.21.15.89:0/2397487654 >> 172.21.15.89:0/2397487654 conn(0x7f61f80b0d70 msgr2=0x7f61f80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:48.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.833+0000 7f61ff400700 1 -- 172.21.15.89:0/2397487654 shutdown_connections 2024-04-02T07:51:48.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.833+0000 7f61ff400700 1 -- 172.21.15.89:0/2397487654 wait complete. 2024-04-02T07:51:48.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.835+0000 7f61ff400700 1 Processor -- start 2024-04-02T07:51:48.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.836+0000 7f61ff400700 1 -- start start 2024-04-02T07:51:48.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.837+0000 7f61ff400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61f80b2070 0x7f61f80c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:48.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.837+0000 7f61ff400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f80ce230 0x7f61f815b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:48.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.837+0000 7f61ff400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61f815d800 0x7f61f814c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:48.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.837+0000 7f61ff400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f815c780 con 0x7f61f80ce230 2024-04-02T07:51:48.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.837+0000 7f61ff400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f80c93e0 con 0x7f61f815d800 2024-04-02T07:51:48.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.837+0000 7f61ff400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f815b880 con 0x7f61f80b2070 2024-04-02T07:51:48.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.837+0000 7f61f7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f80ce230 0x7f61f815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:48.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.837+0000 7f61fca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61f80b2070 0x7f61f80c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:48.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.837+0000 7f61f7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f80ce230 0x7f61f815b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43066/0 (socket says 172.21.15.89:43066) 2024-04-02T07:51:48.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.837+0000 7f61f7e00700 1 -- 172.21.15.89:0/3705810337 learned_addr learned my addr 172.21.15.89:0/3705810337 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:48.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.837+0000 7f61fd400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61f815d800 0x7f61f814c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:48.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.838+0000 7f61fca00700 1 -- 172.21.15.89:0/3705810337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61f815d800 msgr2=0x7f61f814c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:48.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.838+0000 7f61fca00700 1 --2- 172.21.15.89:0/3705810337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61f815d800 0x7f61f814c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:48.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.838+0000 7f61fca00700 1 -- 172.21.15.89:0/3705810337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f80ce230 msgr2=0x7f61f815b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:48.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.838+0000 7f61fca00700 1 --2- 172.21.15.89:0/3705810337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f80ce230 0x7f61f815b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:48.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.838+0000 7f61fca00700 1 -- 172.21.15.89:0/3705810337 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e400a640 con 0x7f61f80b2070 2024-04-02T07:51:48.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.838+0000 7f61fd400700 1 --2- 172.21.15.89:0/3705810337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61f815d800 0x7f61f814c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:48.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.838+0000 7f61fca00700 1 --2- 172.21.15.89:0/3705810337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61f80b2070 0x7f61f80c8ea0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00c450 tx=0x7f61ec00c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:48.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.838+0000 7f61f4c00700 1 -- 172.21.15.89:0/3705810337 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ec0052d0 con 0x7f61f80b2070 2024-04-02T07:51:48.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.838+0000 7f61f4c00700 1 -- 172.21.15.89:0/3705810337 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ec005430 con 0x7f61f80b2070 2024-04-02T07:51:48.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.838+0000 7f61f4c00700 1 -- 172.21.15.89:0/3705810337 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ec0038c0 con 0x7f61f80b2070 2024-04-02T07:51:48.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.838+0000 7f61ff400700 1 -- 172.21.15.89:0/3705810337 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f815bb00 con 0x7f61f80b2070 2024-04-02T07:51:48.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.838+0000 7f61ff400700 1 -- 172.21.15.89:0/3705810337 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f815c0c0 con 0x7f61f80b2070 2024-04-02T07:51:48.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.839+0000 7f61ff400700 1 -- 172.21.15.89:0/3705810337 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f815c8c0 con 0x7f61f80b2070 2024-04-02T07:51:48.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.840+0000 7f61f4c00700 1 -- 172.21.15.89:0/3705810337 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f61ec003a20 con 0x7f61f80b2070 2024-04-02T07:51:48.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.841+0000 7f61f4c00700 1 --2- 172.21.15.89:0/3705810337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f61d806adb0 0x7f61d806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:48.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.841+0000 7f61f4c00700 1 -- 172.21.15.89:0/3705810337 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f61ec088ab0 con 0x7f61f80b2070 2024-04-02T07:51:48.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.841+0000 7f61f7e00700 1 -- 172.21.15.89:0/3705810337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f61d806adb0 msgr2=0x7f61d806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:48.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.841+0000 7f61f7e00700 1 --2- 172.21.15.89:0/3705810337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f61d806adb0 0x7f61d806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:48.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:48.847+0000 7f61f4c00700 1 -- 172.21.15.89:0/3705810337 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f61ec053970 con 0x7f61f80b2070 2024-04-02T07:51:49.004 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:49.002+0000 7f99d5098240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:51:49.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.042+0000 7f61f7e00700 1 -- 172.21.15.89:0/3705810337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f61d806adb0 msgr2=0x7f61d806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:49.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.042+0000 7f61f7e00700 1 --2- 172.21.15.89:0/3705810337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f61d806adb0 0x7f61d806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:49.049 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:49.048+0000 7fe5e17b3240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:51:49.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.054+0000 7f61ff400700 1 -- 172.21.15.89:0/3705810337 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f61f8067720 con 0x7f61f80b2070 2024-04-02T07:51:49.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.055+0000 7f61f4c00700 1 -- 172.21.15.89:0/3705810337 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f61ec0170a0 con 0x7f61f80b2070 2024-04-02T07:51:49.057 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:51:49.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.058+0000 7f61ff400700 1 -- 172.21.15.89:0/3705810337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f61d806adb0 msgr2=0x7f61d806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:49.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.058+0000 7f61ff400700 1 --2- 172.21.15.89:0/3705810337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f61d806adb0 0x7f61d806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.058+0000 7f61ff400700 1 -- 172.21.15.89:0/3705810337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61f80b2070 msgr2=0x7f61f80c8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:49.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.058+0000 7f61ff400700 1 --2- 172.21.15.89:0/3705810337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61f80b2070 0x7f61f80c8ea0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00c450 tx=0x7f61ec00c760 comp rx=0 tx=0).stop 2024-04-02T07:51:49.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.058+0000 7f61ff400700 1 -- 172.21.15.89:0/3705810337 shutdown_connections 2024-04-02T07:51:49.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.058+0000 7f61ff400700 1 --2- 172.21.15.89:0/3705810337 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f61d806adb0 0x7f61d806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.058+0000 7f61ff400700 1 --2- 172.21.15.89:0/3705810337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61f815d800 0x7f61f814c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.058+0000 7f61ff400700 1 --2- 172.21.15.89:0/3705810337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61f80b2070 0x7f61f80c8ea0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.058+0000 7f61ff400700 1 --2- 172.21.15.89:0/3705810337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61f80ce230 0x7f61f815b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.058+0000 7f61ff400700 1 -- 172.21.15.89:0/3705810337 >> 172.21.15.89:0/3705810337 conn(0x7f61f80b0d70 msgr2=0x7f61f80af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:49.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.058+0000 7f61ff400700 1 -- 172.21.15.89:0/3705810337 shutdown_connections 2024-04-02T07:51:49.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.058+0000 7f61ff400700 1 -- 172.21.15.89:0/3705810337 wait complete. 2024-04-02T07:51:49.079 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:51:49.257 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:49.255+0000 7f99d5098240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:51:49.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.284+0000 7f23c8600700 1 Processor -- start 2024-04-02T07:51:49.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.284+0000 7f23c8600700 1 -- start start 2024-04-02T07:51:49.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.285+0000 7f23c8600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23c015dad0 0x7f23c00ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:49.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.285+0000 7f23c8600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23c00d1630 0x7f23c00d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:49.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.285+0000 7f23c8600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f23c00d1fd0 0x7f23c015deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:49.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.285+0000 7f23c8600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c00cfcc0 con 0x7f23c00d1630 2024-04-02T07:51:49.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.285+0000 7f23c8600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c00cd9f0 con 0x7f23c00d1fd0 2024-04-02T07:51:49.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.285+0000 7f23c8600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c015f820 con 0x7f23c015dad0 2024-04-02T07:51:49.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.285+0000 7f23c5200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23c00d1630 0x7f23c00d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:49.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.286+0000 7f23c5200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23c00d1630 0x7f23c00d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43074/0 (socket says 172.21.15.89:43074) 2024-04-02T07:51:49.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.286+0000 7f23c5200700 1 -- 172.21.15.89:0/2133706328 learned_addr learned my addr 172.21.15.89:0/2133706328 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:49.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.286+0000 7f23c5c00700 1 --2- 172.21.15.89:0/2133706328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23c015dad0 0x7f23c00ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:49.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.286+0000 7f23c6600700 1 --2- 172.21.15.89:0/2133706328 >> v2:172.21.15.113:3300/0 conn(0x7f23c00d1fd0 0x7f23c015deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:49.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.286+0000 7f23c5200700 1 -- 172.21.15.89:0/2133706328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23c015dad0 msgr2=0x7f23c00ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:49.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.286+0000 7f23c5200700 1 --2- 172.21.15.89:0/2133706328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23c015dad0 0x7f23c00ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.286+0000 7f23c5200700 1 -- 172.21.15.89:0/2133706328 >> v2:172.21.15.113:3300/0 conn(0x7f23c00d1fd0 msgr2=0x7f23c015deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:49.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.286+0000 7f23c5200700 1 --2- 172.21.15.89:0/2133706328 >> v2:172.21.15.113:3300/0 conn(0x7f23c00d1fd0 0x7f23c015deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.286+0000 7f23c5200700 1 -- 172.21.15.89:0/2133706328 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c00d2460 con 0x7f23c00d1630 2024-04-02T07:51:49.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.286+0000 7f23c5200700 1 --2- 172.21.15.89:0/2133706328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c00d1630 0x7f23c00d1a90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f23c00b95a0 tx=0x7f23ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a45c75dcd687be server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:49.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.286+0000 7f23bfe00700 1 -- 172.21.15.89:0/2133706328 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac011830 con 0x7f23c00d1630 2024-04-02T07:51:49.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.286+0000 7f23bfe00700 1 -- 172.21.15.89:0/2133706328 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac011990 con 0x7f23c00d1630 2024-04-02T07:51:49.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.286+0000 7f23bfe00700 1 -- 172.21.15.89:0/2133706328 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac011d20 con 0x7f23c00d1630 2024-04-02T07:51:49.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.287+0000 7f23c8600700 1 -- 172.21.15.89:0/2133706328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c00d1630 msgr2=0x7f23c00d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:49.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.287+0000 7f23c8600700 1 --2- 172.21.15.89:0/2133706328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c00d1630 0x7f23c00d1a90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f23c00b95a0 tx=0x7f23ac009bd0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.288+0000 7f23c8600700 1 -- 172.21.15.89:0/2133706328 shutdown_connections 2024-04-02T07:51:49.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.288+0000 7f23c8600700 1 --2- 172.21.15.89:0/2133706328 >> v2:172.21.15.113:3300/0 conn(0x7f23c00d1fd0 0x7f23c015deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.288+0000 7f23c8600700 1 --2- 172.21.15.89:0/2133706328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23c015dad0 0x7f23c00ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.288+0000 7f23c8600700 1 --2- 172.21.15.89:0/2133706328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c00d1630 0x7f23c00d1a90 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.288+0000 7f23c8600700 1 -- 172.21.15.89:0/2133706328 >> 172.21.15.89:0/2133706328 conn(0x7f23c008aff0 msgr2=0x7f23c00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:49.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.288+0000 7f23c8600700 1 -- 172.21.15.89:0/2133706328 shutdown_connections 2024-04-02T07:51:49.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.288+0000 7f23c8600700 1 -- 172.21.15.89:0/2133706328 wait complete. 2024-04-02T07:51:49.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.291+0000 7f23c8600700 1 Processor -- start 2024-04-02T07:51:49.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.291+0000 7f23c8600700 1 -- start start 2024-04-02T07:51:49.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c8600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c00d1630 0x7f23c01f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:49.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c8600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23c00d1fd0 0x7f23c00c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:49.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c8600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23c015dad0 0x7f23c00c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:49.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c8600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c00b87b0 con 0x7f23c00d1630 2024-04-02T07:51:49.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c8600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c00b7b60 con 0x7f23c015dad0 2024-04-02T07:51:49.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c8600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c00b6f50 con 0x7f23c00d1fd0 2024-04-02T07:51:49.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c5c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c00d1630 0x7f23c01f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:49.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c5200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23c00d1fd0 0x7f23c00c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:49.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c5200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23c00d1fd0 0x7f23c00c8ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:46914/0 (socket says 172.21.15.89:46914) 2024-04-02T07:51:49.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c5200700 1 -- 172.21.15.89:0/2917748436 learned_addr learned my addr 172.21.15.89:0/2917748436 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:49.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c6600700 1 --2- 172.21.15.89:0/2917748436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23c015dad0 0x7f23c00c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:49.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c5200700 1 -- 172.21.15.89:0/2917748436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23c015dad0 msgr2=0x7f23c00c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:49.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c5200700 1 --2- 172.21.15.89:0/2917748436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23c015dad0 0x7f23c00c9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c5200700 1 -- 172.21.15.89:0/2917748436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c00d1630 msgr2=0x7f23c01f5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:49.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c5200700 1 --2- 172.21.15.89:0/2917748436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c00d1630 0x7f23c01f5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c5200700 1 -- 172.21.15.89:0/2917748436 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac009550 con 0x7f23c00d1fd0 2024-04-02T07:51:49.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.293+0000 7f23c6600700 1 --2- 172.21.15.89:0/2917748436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23c015dad0 0x7f23c00c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:49.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.294+0000 7f23c5200700 1 --2- 172.21.15.89:0/2917748436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23c00d1fd0 0x7f23c00c8ad0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009c10 tx=0x7f23ac004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:49.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.294+0000 7f23bd600700 1 -- 172.21.15.89:0/2917748436 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac004240 con 0x7f23c00d1fd0 2024-04-02T07:51:49.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.294+0000 7f23bd600700 1 -- 172.21.15.89:0/2917748436 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac01f070 con 0x7f23c00d1fd0 2024-04-02T07:51:49.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.294+0000 7f23c8600700 1 -- 172.21.15.89:0/2917748436 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c00b7230 con 0x7f23c00d1fd0 2024-04-02T07:51:49.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.294+0000 7f23bd600700 1 -- 172.21.15.89:0/2917748436 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac0045e0 con 0x7f23c00d1fd0 2024-04-02T07:51:49.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.294+0000 7f23c8600700 1 -- 172.21.15.89:0/2917748436 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c00b77f0 con 0x7f23c00d1fd0 2024-04-02T07:51:49.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.295+0000 7f23c8600700 1 -- 172.21.15.89:0/2917748436 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c00b7d10 con 0x7f23c00d1fd0 2024-04-02T07:51:49.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.296+0000 7f23bd600700 1 -- 172.21.15.89:0/2917748436 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f23ac021030 con 0x7f23c00d1fd0 2024-04-02T07:51:49.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.298+0000 7f23bd600700 1 --2- 172.21.15.89:0/2917748436 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f23a006adb0 0x7f23a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:49.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.298+0000 7f23c5c00700 1 -- 172.21.15.89:0/2917748436 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f23a006adb0 msgr2=0x7f23a006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:49.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.298+0000 7f23bd600700 1 -- 172.21.15.89:0/2917748436 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f23ac08a940 con 0x7f23c00d1fd0 2024-04-02T07:51:49.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.298+0000 7f23c5c00700 1 --2- 172.21.15.89:0/2917748436 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f23a006adb0 0x7f23a006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:49.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.301+0000 7f23bd600700 1 -- 172.21.15.89:0/2917748436 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f23c00b7d10 con 0x7f23c00d1fd0 2024-04-02T07:51:49.324 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:49.323+0000 7fe5e17b3240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:51:49.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.499+0000 7f23c5c00700 1 -- 172.21.15.89:0/2917748436 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f23a006adb0 msgr2=0x7f23a006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:49.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.499+0000 7f23c5c00700 1 --2- 172.21.15.89:0/2917748436 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f23a006adb0 0x7f23a006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:49.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.511+0000 7f23c8600700 1 -- 172.21.15.89:0/2917748436 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f23c00b7d10 con 0x7f23c00d1fd0 2024-04-02T07:51:49.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.511+0000 7f23bd600700 1 -- 172.21.15.89:0/2917748436 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v84) v1 ==== 104+0+11 (secure 0 0 0) 0x7f23ac015020 con 0x7f23c00d1fd0 2024-04-02T07:51:49.513 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:51:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.514+0000 7f23c8600700 1 -- 172.21.15.89:0/2917748436 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f23a006adb0 msgr2=0x7f23a006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.514+0000 7f23c8600700 1 --2- 172.21.15.89:0/2917748436 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f23a006adb0 0x7f23a006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.514+0000 7f23c8600700 1 -- 172.21.15.89:0/2917748436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23c00d1fd0 msgr2=0x7f23c00c8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.514+0000 7f23c8600700 1 --2- 172.21.15.89:0/2917748436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23c00d1fd0 0x7f23c00c8ad0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009c10 tx=0x7f23ac004750 comp rx=0 tx=0).stop 2024-04-02T07:51:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.514+0000 7f23c8600700 1 -- 172.21.15.89:0/2917748436 shutdown_connections 2024-04-02T07:51:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.514+0000 7f23c8600700 1 --2- 172.21.15.89:0/2917748436 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f23a006adb0 0x7f23a006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.514+0000 7f23c8600700 1 --2- 172.21.15.89:0/2917748436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23c015dad0 0x7f23c00c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.514+0000 7f23c8600700 1 --2- 172.21.15.89:0/2917748436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23c00d1630 0x7f23c01f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.514+0000 7f23c8600700 1 --2- 172.21.15.89:0/2917748436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23c00d1fd0 0x7f23c00c8ad0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.514+0000 7f23c8600700 1 -- 172.21.15.89:0/2917748436 >> 172.21.15.89:0/2917748436 conn(0x7f23c008aff0 msgr2=0x7f23c015aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.514+0000 7f23c8600700 1 -- 172.21.15.89:0/2917748436 shutdown_connections 2024-04-02T07:51:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.514+0000 7f23c8600700 1 -- 172.21.15.89:0/2917748436 wait complete. 2024-04-02T07:51:49.536 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:51:49.715 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:49.713+0000 7f99d5098240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:51:49.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.743+0000 7effb9200700 1 Processor -- start 2024-04-02T07:51:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.743+0000 7effb9200700 1 -- start start 2024-04-02T07:51:49.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.744+0000 7effb9200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7effb4160cc0 0x7effb4168730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:49.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.744+0000 7effb9200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb415f820 con 0x7effb415dad0 2024-04-02T07:51:49.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.744+0000 7effb9200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb40d2450 con 0x7effb40ced80 2024-04-02T07:51:49.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.744+0000 7effb9200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb40cfcc0 con 0x7effb4160cc0 2024-04-02T07:51:49.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.744+0000 7effb1e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7effb415dad0 0x7effb40d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45986/0 (socket says 172.21.15.89:45986) 2024-04-02T07:51:49.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.744+0000 7effb1e00700 1 -- 172.21.15.89:0/2267175552 learned_addr learned my addr 172.21.15.89:0/2267175552 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:49.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.745+0000 7effb2800700 1 --2- 172.21.15.89:0/2267175552 >> v2:172.21.15.113:3300/0 conn(0x7effb4160cc0 0x7effb4168730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:49.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.745+0000 7effb0a00700 1 -- 172.21.15.89:0/2267175552 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341429702 0 0) 0x7effb415f820 con 0x7effb415dad0 2024-04-02T07:51:49.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.745+0000 7effb0a00700 1 -- 172.21.15.89:0/2267175552 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff980035b0 con 0x7effb415dad0 2024-04-02T07:51:49.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.745+0000 7effb0a00700 1 -- 172.21.15.89:0/2267175552 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414842406 0 0) 0x7effb40d2450 con 0x7effb40ced80 2024-04-02T07:51:49.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.745+0000 7effb0a00700 1 -- 172.21.15.89:0/2267175552 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb415f820 con 0x7effb40ced80 2024-04-02T07:51:49.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.745+0000 7effb2800700 1 -- 172.21.15.89:0/2267175552 >> v1:172.21.15.113:6789/0 conn(0x7effb40ced80 legacy=0x7effb40d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.745+0000 7effb2800700 1 -- 172.21.15.89:0/2267175552 >> v1:172.21.15.89:6789/0 conn(0x7effb415dad0 legacy=0x7effb40d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.745+0000 7effb2800700 1 -- 172.21.15.89:0/2267175552 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb40cf420 con 0x7effb4160cc0 2024-04-02T07:51:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.745+0000 7effb2800700 1 --2- 172.21.15.89:0/2267175552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb4160cc0 0x7effb4168730 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7effac009d90 tx=0x7effac00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32bba674b06c7283 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.746+0000 7effb0a00700 1 -- 172.21.15.89:0/2267175552 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac00e070 con 0x7effb4160cc0 2024-04-02T07:51:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.746+0000 7effb0a00700 1 -- 172.21.15.89:0/2267175552 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac00bb20 con 0x7effb4160cc0 2024-04-02T07:51:49.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.746+0000 7effb0a00700 1 -- 172.21.15.89:0/2267175552 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac012430 con 0x7effb4160cc0 2024-04-02T07:51:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.746+0000 7effb9200700 1 -- 172.21.15.89:0/2267175552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb4160cc0 msgr2=0x7effb4168730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.746+0000 7effb9200700 1 --2- 172.21.15.89:0/2267175552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb4160cc0 0x7effb4168730 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7effac009d90 tx=0x7effac00b330 comp rx=0 tx=0).stop 2024-04-02T07:51:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.746+0000 7effb9200700 1 -- 172.21.15.89:0/2267175552 shutdown_connections 2024-04-02T07:51:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.746+0000 7effb9200700 1 --2- 172.21.15.89:0/2267175552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb4160cc0 0x7effb4168730 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.746+0000 7effb9200700 1 -- 172.21.15.89:0/2267175552 >> 172.21.15.89:0/2267175552 conn(0x7effb408aff0 msgr2=0x7effb40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.747+0000 7effb9200700 1 -- 172.21.15.89:0/2267175552 shutdown_connections 2024-04-02T07:51:49.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.747+0000 7effb9200700 1 -- 172.21.15.89:0/2267175552 wait complete. 2024-04-02T07:51:49.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.748+0000 7effb9200700 1 Processor -- start 2024-04-02T07:51:49.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.748+0000 7effb9200700 1 -- start start 2024-04-02T07:51:49.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.749+0000 7effb9200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb40ced80 0x7effb41f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:49.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.749+0000 7effb9200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb415dad0 0x7effb40b6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:49.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.749+0000 7effb9200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb4160cc0 0x7effb40c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:49.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.749+0000 7effb9200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb40b7700 con 0x7effb40ced80 2024-04-02T07:51:49.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.749+0000 7effb9200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb40b79e0 con 0x7effb4160cc0 2024-04-02T07:51:49.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.749+0000 7effb9200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb40b7cc0 con 0x7effb415dad0 2024-04-02T07:51:49.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.750+0000 7effb1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb40ced80 0x7effb41f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:49.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.750+0000 7effb1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb40ced80 0x7effb41f28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43086/0 (socket says 172.21.15.89:43086) 2024-04-02T07:51:49.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.750+0000 7effb1e00700 1 -- 172.21.15.89:0/2522963322 learned_addr learned my addr 172.21.15.89:0/2522963322 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:49.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.750+0000 7effb2800700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb4160cc0 0x7effb40c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:49.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.750+0000 7effb1400700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb415dad0 0x7effb40b6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.750+0000 7effb1e00700 1 -- 172.21.15.89:0/2522963322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb415dad0 msgr2=0x7effb40b6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.750+0000 7effb1e00700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb415dad0 0x7effb40b6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.750+0000 7effb1e00700 1 -- 172.21.15.89:0/2522963322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb4160cc0 msgr2=0x7effb40c70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.750+0000 7effb1e00700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb4160cc0 0x7effb40c70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.750+0000 7effb1e00700 1 -- 172.21.15.89:0/2522963322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac009a70 con 0x7effb40ced80 2024-04-02T07:51:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.750+0000 7effb1400700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb415dad0 0x7effb40b6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.750+0000 7effb1e00700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb40ced80 0x7effb41f28d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7effa400c700 tx=0x7effa400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:49.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.751+0000 7effaa000700 1 -- 172.21.15.89:0/2522963322 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa4005d70 con 0x7effb40ced80 2024-04-02T07:51:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.751+0000 7effaa000700 1 -- 172.21.15.89:0/2522963322 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa40070d0 con 0x7effb40ced80 2024-04-02T07:51:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.751+0000 7effaa000700 1 -- 172.21.15.89:0/2522963322 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa4007460 con 0x7effb40ced80 2024-04-02T07:51:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.751+0000 7effb9200700 1 -- 172.21.15.89:0/2522963322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb40b7fa0 con 0x7effb40ced80 2024-04-02T07:51:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.751+0000 7effb9200700 1 -- 172.21.15.89:0/2522963322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb40c94f0 con 0x7effb40ced80 2024-04-02T07:51:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.752+0000 7effb9200700 1 -- 172.21.15.89:0/2522963322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb40b84c0 con 0x7effb40ced80 2024-04-02T07:51:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.753+0000 7effaa000700 1 -- 172.21.15.89:0/2522963322 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7effa40075c0 con 0x7effb40ced80 2024-04-02T07:51:49.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.754+0000 7effaa000700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff9406adb0 0x7eff9406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.754+0000 7effaa000700 1 -- 172.21.15.89:0/2522963322 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7effa4088c70 con 0x7effb40ced80 2024-04-02T07:51:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.754+0000 7effb1400700 1 -- 172.21.15.89:0/2522963322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff9406adb0 msgr2=0x7eff9406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.754+0000 7effb1400700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff9406adb0 0x7eff9406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:49.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.760+0000 7effaa000700 1 -- 172.21.15.89:0/2522963322 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7effa4053a80 con 0x7effb40ced80 2024-04-02T07:51:49.810 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:49.808+0000 7fe5e17b3240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:51:49.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.955+0000 7effb1400700 1 -- 172.21.15.89:0/2522963322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff9406adb0 msgr2=0x7eff9406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:49.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.955+0000 7effb1400700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff9406adb0 0x7eff9406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.969+0000 7effb9200700 1 -- 172.21.15.89:0/2522963322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7effb4067720 con 0x7effb40ced80 2024-04-02T07:51:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.969+0000 7effaa000700 1 -- 172.21.15.89:0/2522963322 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v84) v1 ==== 100+0+11 (secure 0 0 0) 0x7effa40170a0 con 0x7effb40ced80 2024-04-02T07:51:49.971 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:51:49.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.972+0000 7effb9200700 1 -- 172.21.15.89:0/2522963322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff9406adb0 msgr2=0x7eff9406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:49.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.972+0000 7effb9200700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff9406adb0 0x7eff9406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.972+0000 7effb9200700 1 -- 172.21.15.89:0/2522963322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb40ced80 msgr2=0x7effb41f28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:49.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.972+0000 7effb9200700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb40ced80 0x7effb41f28d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7effa400c700 tx=0x7effa400ca10 comp rx=0 tx=0).stop 2024-04-02T07:51:49.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.972+0000 7effb9200700 1 -- 172.21.15.89:0/2522963322 shutdown_connections 2024-04-02T07:51:49.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.972+0000 7effb9200700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff9406adb0 0x7eff9406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.972+0000 7effb9200700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb4160cc0 0x7effb40c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.972+0000 7effb9200700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effb40ced80 0x7effb41f28d0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.972+0000 7effb9200700 1 --2- 172.21.15.89:0/2522963322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effb415dad0 0x7effb40b6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:49.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.972+0000 7effb9200700 1 -- 172.21.15.89:0/2522963322 >> 172.21.15.89:0/2522963322 conn(0x7effb408aff0 msgr2=0x7effb406ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:49.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.972+0000 7effb9200700 1 -- 172.21.15.89:0/2522963322 shutdown_connections 2024-04-02T07:51:49.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:49.972+0000 7effb9200700 1 -- 172.21.15.89:0/2522963322 wait complete. 2024-04-02T07:51:49.992 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T07:51:50.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.187+0000 7fa34da00700 1 Processor -- start 2024-04-02T07:51:50.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.187+0000 7fa34da00700 1 -- start start 2024-04-02T07:51:50.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.188+0000 7fa34da00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa34815b740 0x7fa3480cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:50.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.188+0000 7fa34da00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa3480cf440 0x7fa3480cf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:50.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.188+0000 7fa34da00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34809adf0 con 0x7fa3480c57b0 2024-04-02T07:51:50.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.188+0000 7fa34da00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34815bb20 con 0x7fa3480cf440 2024-04-02T07:51:50.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.188+0000 7fa34da00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34809a830 con 0x7fa34815b740 2024-04-02T07:51:50.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.188+0000 7fa347400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa3480c57b0 0x7fa3480c5c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45988/0 (socket says 172.21.15.89:45988) 2024-04-02T07:51:50.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.188+0000 7fa347400700 1 -- 172.21.15.89:0/2468569224 learned_addr learned my addr 172.21.15.89:0/2468569224 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:50.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.188+0000 7fa346000700 1 --2- 172.21.15.89:0/2468569224 >> v2:172.21.15.113:3300/0 conn(0x7fa3480cf440 0x7fa3480cf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:50.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.188+0000 7fa346a00700 1 --2- 172.21.15.89:0/2468569224 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa34815b740 0x7fa3480cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:50.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.189+0000 7fa345600700 1 -- 172.21.15.89:0/2468569224 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895552916 0 0) 0x7fa34809adf0 con 0x7fa3480c57b0 2024-04-02T07:51:50.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.189+0000 7fa345600700 1 -- 172.21.15.89:0/2468569224 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa32c0035b0 con 0x7fa3480c57b0 2024-04-02T07:51:50.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.189+0000 7fa346000700 1 -- 172.21.15.89:0/2468569224 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa34815b740 msgr2=0x7fa3480cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:50.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.189+0000 7fa346000700 1 --2- 172.21.15.89:0/2468569224 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa34815b740 0x7fa3480cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.189+0000 7fa346000700 1 -- 172.21.15.89:0/2468569224 >> v1:172.21.15.89:6789/0 conn(0x7fa3480c57b0 legacy=0x7fa3480c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:50.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.189+0000 7fa346000700 1 -- 172.21.15.89:0/2468569224 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34809ab10 con 0x7fa3480cf440 2024-04-02T07:51:50.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.189+0000 7fa346000700 1 --2- 172.21.15.89:0/2468569224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3480cf440 0x7fa3480cf8c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa334009840 tx=0x7fa334009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb7e459ed0234a79 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:50.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.190+0000 7fa345600700 1 -- 172.21.15.89:0/2468569224 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334011830 con 0x7fa3480cf440 2024-04-02T07:51:50.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.190+0000 7fa345600700 1 -- 172.21.15.89:0/2468569224 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa334011990 con 0x7fa3480cf440 2024-04-02T07:51:50.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.190+0000 7fa345600700 1 -- 172.21.15.89:0/2468569224 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334011d20 con 0x7fa3480cf440 2024-04-02T07:51:50.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.190+0000 7fa34da00700 1 -- 172.21.15.89:0/2468569224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3480cf440 msgr2=0x7fa3480cf8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:50.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.190+0000 7fa34da00700 1 --2- 172.21.15.89:0/2468569224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3480cf440 0x7fa3480cf8c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa334009840 tx=0x7fa334009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:50.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.190+0000 7fa34da00700 1 -- 172.21.15.89:0/2468569224 shutdown_connections 2024-04-02T07:51:50.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.190+0000 7fa34da00700 1 --2- 172.21.15.89:0/2468569224 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa34815b740 0x7fa3480cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.190+0000 7fa34da00700 1 --2- 172.21.15.89:0/2468569224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3480cf440 0x7fa3480cf8c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.190+0000 7fa34da00700 1 -- 172.21.15.89:0/2468569224 >> 172.21.15.89:0/2468569224 conn(0x7fa3480b2e20 msgr2=0x7fa3480b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:50.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.190+0000 7fa34da00700 1 -- 172.21.15.89:0/2468569224 shutdown_connections 2024-04-02T07:51:50.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.190+0000 7fa34da00700 1 -- 172.21.15.89:0/2468569224 wait complete. 2024-04-02T07:51:50.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.192+0000 7fa34da00700 1 Processor -- start 2024-04-02T07:51:50.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.192+0000 7fa34da00700 1 -- start start 2024-04-02T07:51:50.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.193+0000 7fa34da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3480c57b0 0x7fa3481593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:50.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.193+0000 7fa34da00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3480cf440 0x7fa348158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:50.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.193+0000 7fa34da00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa34815b740 0x7fa3480c9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:50.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.193+0000 7fa34da00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3480c95d0 con 0x7fa3480c57b0 2024-04-02T07:51:50.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.193+0000 7fa34da00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3480c9850 con 0x7fa3480cf440 2024-04-02T07:51:50.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.193+0000 7fa34da00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3480c9b30 con 0x7fa34815b740 2024-04-02T07:51:50.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.193+0000 7fa346a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3480c57b0 0x7fa3481593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:50.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.193+0000 7fa346a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3480c57b0 0x7fa3481593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43100/0 (socket says 172.21.15.89:43100) 2024-04-02T07:51:50.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.193+0000 7fa346a00700 1 -- 172.21.15.89:0/3948882830 learned_addr learned my addr 172.21.15.89:0/3948882830 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:50.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.193+0000 7fa346000700 1 --2- 172.21.15.89:0/3948882830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3480cf440 0x7fa348158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:50.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.193+0000 7fa347400700 1 --2- 172.21.15.89:0/3948882830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa34815b740 0x7fa3480c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:50.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.194+0000 7fa346a00700 1 -- 172.21.15.89:0/3948882830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa34815b740 msgr2=0x7fa3480c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:50.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.194+0000 7fa346a00700 1 --2- 172.21.15.89:0/3948882830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa34815b740 0x7fa3480c9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.194+0000 7fa346a00700 1 -- 172.21.15.89:0/3948882830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3480cf440 msgr2=0x7fa348158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:50.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.194+0000 7fa346a00700 1 --2- 172.21.15.89:0/3948882830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3480cf440 0x7fa348158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.194+0000 7fa346a00700 1 -- 172.21.15.89:0/3948882830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3340094a0 con 0x7fa3480c57b0 2024-04-02T07:51:50.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.194+0000 7fa346a00700 1 --2- 172.21.15.89:0/3948882830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3480c57b0 0x7fa3481593f0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fa33800e9b0 tx=0x7fa33800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:50.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.194+0000 7fa33ea00700 1 -- 172.21.15.89:0/3948882830 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338009710 con 0x7fa3480c57b0 2024-04-02T07:51:50.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.194+0000 7fa33ea00700 1 -- 172.21.15.89:0/3948882830 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33800ee50 con 0x7fa3480c57b0 2024-04-02T07:51:50.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.194+0000 7fa33ea00700 1 -- 172.21.15.89:0/3948882830 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33800ca50 con 0x7fa3480c57b0 2024-04-02T07:51:50.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.194+0000 7fa34da00700 1 -- 172.21.15.89:0/3948882830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3480c9e10 con 0x7fa3480c57b0 2024-04-02T07:51:50.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.194+0000 7fa34da00700 1 -- 172.21.15.89:0/3948882830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3480ca3d0 con 0x7fa3480c57b0 2024-04-02T07:51:50.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.196+0000 7fa34da00700 1 -- 172.21.15.89:0/3948882830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3480ca740 con 0x7fa3480c57b0 2024-04-02T07:51:50.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.197+0000 7fa33ea00700 1 -- 172.21.15.89:0/3948882830 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fa33800cbb0 con 0x7fa3480c57b0 2024-04-02T07:51:50.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.197+0000 7fa33ea00700 1 --2- 172.21.15.89:0/3948882830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa32806adb0 0x7fa32806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:50.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.198+0000 7fa33ea00700 1 -- 172.21.15.89:0/3948882830 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fa3380897b0 con 0x7fa3480c57b0 2024-04-02T07:51:50.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.198+0000 7fa346000700 1 -- 172.21.15.89:0/3948882830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa32806adb0 msgr2=0x7fa32806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:50.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.198+0000 7fa346000700 1 --2- 172.21.15.89:0/3948882830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa32806adb0 0x7fa32806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:50.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.202+0000 7fa33ea00700 1 -- 172.21.15.89:0/3948882830 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa3380545b0 con 0x7fa3480c57b0 2024-04-02T07:51:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.399+0000 7fa346000700 1 -- 172.21.15.89:0/3948882830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa32806adb0 msgr2=0x7fa32806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.399+0000 7fa346000700 1 --2- 172.21.15.89:0/3948882830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa32806adb0 0x7fa32806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:50.424 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:50.423+0000 7f99d5098240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:51:50.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.427+0000 7fa34da00700 1 -- 172.21.15.89:0/3948882830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa348070840 con 0x7fa3480c57b0 2024-04-02T07:51:50.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.428+0000 7fa33ea00700 1 -- 172.21.15.89:0/3948882830 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v84) v1 ==== 80+0+456 (secure 0 0 0) 0x7fa33800ce60 con 0x7fa3480c57b0 2024-04-02T07:51:50.429 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/822661295 2024-04-03T07:51:45.730929+0000 2024-04-02T07:51:50.429 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/907732608 2024-04-03T07:51:45.730929+0000 2024-04-02T07:51:50.429 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3083880072 2024-04-03T07:51:45.730929+0000 2024-04-02T07:51:50.430 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/43357 2024-04-03T07:51:45.730929+0000 2024-04-02T07:51:50.430 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/43357 2024-04-03T07:51:45.730929+0000 2024-04-02T07:51:50.430 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1211892288 2024-04-03T07:51:45.730929+0000 2024-04-02T07:51:50.430 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/23281819 2024-04-03T07:51:45.730929+0000 2024-04-02T07:51:50.430 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3435265311 2024-04-03T07:51:45.730929+0000 2024-04-02T07:51:50.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.430+0000 7fa34da00700 1 -- 172.21.15.89:0/3948882830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa32806adb0 msgr2=0x7fa32806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:50.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.430+0000 7fa34da00700 1 --2- 172.21.15.89:0/3948882830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa32806adb0 0x7fa32806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.430+0000 7fa34da00700 1 -- 172.21.15.89:0/3948882830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3480c57b0 msgr2=0x7fa3481593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:50.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.430+0000 7fa34da00700 1 --2- 172.21.15.89:0/3948882830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3480c57b0 0x7fa3481593f0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fa33800e9b0 tx=0x7fa33800ecc0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.430+0000 7fa34da00700 1 -- 172.21.15.89:0/3948882830 shutdown_connections 2024-04-02T07:51:50.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.430+0000 7fa34da00700 1 --2- 172.21.15.89:0/3948882830 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa32806adb0 0x7fa32806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.430+0000 7fa34da00700 1 --2- 172.21.15.89:0/3948882830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa34815b740 0x7fa3480c9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.430+0000 7fa34da00700 1 --2- 172.21.15.89:0/3948882830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3480c57b0 0x7fa3481593f0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.430+0000 7fa34da00700 1 --2- 172.21.15.89:0/3948882830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3480cf440 0x7fa348158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.430+0000 7fa34da00700 1 -- 172.21.15.89:0/3948882830 >> 172.21.15.89:0/3948882830 conn(0x7fa3480b2e20 msgr2=0x7fa3480cd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:50.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.430+0000 7fa34da00700 1 -- 172.21.15.89:0/3948882830 shutdown_connections 2024-04-02T07:51:50.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.430+0000 7fa34da00700 1 -- 172.21.15.89:0/3948882830 wait complete. 2024-04-02T07:51:50.433 INFO:teuthology.orchestra.run.smithi089.stderr:listed 8 entries 2024-04-02T07:51:50.450 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T07:51:50.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:50.450 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T07:51:50.518 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:50.515+0000 7f99d5098240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:51:50.529 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:51:50.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:51:50.530 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T07:51:50.565 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:50.563+0000 7fe5e17b3240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:51:50.575 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:51:50.575 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:50.620 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:51:50.620 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:51:50.620 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:50.665 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:51:50.665 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:51:50.665 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:51:50.665 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:50.663+0000 7fe5e17b3240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:51:50.680 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:50.678+0000 7f99d5098240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:51:50.711 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:51:50.712 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:51:50.838 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:50.837+0000 7fe5e17b3240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:51:50.898 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:50.897+0000 7f99d5098240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:51:50.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.906+0000 7f26a6400700 1 Processor -- start 2024-04-02T07:51:50.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.906+0000 7f26a6400700 1 -- start start 2024-04-02T07:51:50.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.908+0000 7f26a6400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f26a00ceec0 0x7f26a01612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:50.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.908+0000 7f26a6400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26a0161800 0x7f26a00b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:50.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.908+0000 7f26a6400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a00d30d0 con 0x7f26a015d640 2024-04-02T07:51:50.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.908+0000 7f26a6400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a0161c90 con 0x7f26a00ceec0 2024-04-02T07:51:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.908+0000 7f26a6400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a00b6840 con 0x7f26a0161800 2024-04-02T07:51:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.908+0000 7f269ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f26a00ceec0 0x7f26a01612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.908+0000 7f269f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f26a015d640 0x7f26a00d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45994/0 (socket says 172.21.15.89:45994) 2024-04-02T07:51:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.908+0000 7f269f400700 1 -- 172.21.15.89:0/342770107 learned_addr learned my addr 172.21.15.89:0/342770107 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.908+0000 7f269fe00700 1 --2- 172.21.15.89:0/342770107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26a0161800 0x7f26a00b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.908+0000 7f269ea00700 1 -- 172.21.15.89:0/342770107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26a0161800 msgr2=0x7f26a00b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.908+0000 7f269ea00700 1 --2- 172.21.15.89:0/342770107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26a0161800 0x7f26a00b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.908+0000 7f269ea00700 1 -- 172.21.15.89:0/342770107 >> v1:172.21.15.89:6789/0 conn(0x7f26a015d640 legacy=0x7f26a00d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.908+0000 7f269ea00700 1 -- 172.21.15.89:0/342770107 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a00d2b30 con 0x7f26a00ceec0 2024-04-02T07:51:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.908+0000 7f269e000700 1 -- 172.21.15.89:0/342770107 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936753712 0 0) 0x7f26a00d30d0 con 0x7f26a015d640 2024-04-02T07:51:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.909+0000 7f269ea00700 1 --2- 172.21.15.89:0/342770107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a00ceec0 0x7f26a01612c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f268c009840 tx=0x7f268c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f90c28ed39e7bf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.909+0000 7f269e000700 1 -- 172.21.15.89:0/342770107 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c011830 con 0x7f26a00ceec0 2024-04-02T07:51:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.909+0000 7f269e000700 1 -- 172.21.15.89:0/342770107 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268c011990 con 0x7f26a00ceec0 2024-04-02T07:51:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.909+0000 7f269e000700 1 -- 172.21.15.89:0/342770107 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c011d20 con 0x7f26a00ceec0 2024-04-02T07:51:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.909+0000 7f26a6400700 1 -- 172.21.15.89:0/342770107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a00ceec0 msgr2=0x7f26a01612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.909+0000 7f26a6400700 1 --2- 172.21.15.89:0/342770107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a00ceec0 0x7f26a01612c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f268c009840 tx=0x7f268c009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.910+0000 7f26a6400700 1 -- 172.21.15.89:0/342770107 shutdown_connections 2024-04-02T07:51:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.910+0000 7f26a6400700 1 --2- 172.21.15.89:0/342770107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26a0161800 0x7f26a00b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.910+0000 7f26a6400700 1 --2- 172.21.15.89:0/342770107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a00ceec0 0x7f26a01612c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.910+0000 7f26a6400700 1 -- 172.21.15.89:0/342770107 >> 172.21.15.89:0/342770107 conn(0x7f26a008aff0 msgr2=0x7f26a00d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:50.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.910+0000 7f26a6400700 1 -- 172.21.15.89:0/342770107 shutdown_connections 2024-04-02T07:51:50.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.910+0000 7f26a6400700 1 -- 172.21.15.89:0/342770107 wait complete. 2024-04-02T07:51:50.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.912+0000 7f26a6400700 1 Processor -- start 2024-04-02T07:51:50.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.913+0000 7f26a6400700 1 -- start start 2024-04-02T07:51:50.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.914+0000 7f26a6400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a00ceec0 0x7f26a01f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:50.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.914+0000 7f26a6400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26a015d640 0x7f26a00a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:50.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.914+0000 7f26a6400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26a0161800 0x7f26a009d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:50.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.914+0000 7f26a6400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a01f6b90 con 0x7f26a00ceec0 2024-04-02T07:51:50.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.914+0000 7f26a6400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a01f6e70 con 0x7f26a015d640 2024-04-02T07:51:50.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.914+0000 7f26a6400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a01f7150 con 0x7f26a0161800 2024-04-02T07:51:50.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.914+0000 7f269f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a00ceec0 0x7f26a01f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:50.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.914+0000 7f269f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a00ceec0 0x7f26a01f65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43120/0 (socket says 172.21.15.89:43120) 2024-04-02T07:51:50.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.914+0000 7f269f400700 1 -- 172.21.15.89:0/2921725389 learned_addr learned my addr 172.21.15.89:0/2921725389 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:50.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.915+0000 7f269ea00700 1 --2- 172.21.15.89:0/2921725389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26a015d640 0x7f26a00a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:50.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.915+0000 7f269fe00700 1 --2- 172.21.15.89:0/2921725389 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26a0161800 0x7f26a009d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:50.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.915+0000 7f269f400700 1 -- 172.21.15.89:0/2921725389 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26a0161800 msgr2=0x7f26a009d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:50.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.915+0000 7f269f400700 1 --2- 172.21.15.89:0/2921725389 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26a0161800 0x7f26a009d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.915+0000 7f269f400700 1 -- 172.21.15.89:0/2921725389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26a015d640 msgr2=0x7f26a00a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:50.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.915+0000 7f269f400700 1 --2- 172.21.15.89:0/2921725389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26a015d640 0x7f26a00a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:50.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.915+0000 7f269f400700 1 -- 172.21.15.89:0/2921725389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c0094a0 con 0x7f26a00ceec0 2024-04-02T07:51:50.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.915+0000 7f269f400700 1 --2- 172.21.15.89:0/2921725389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a00ceec0 0x7f26a01f65b0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f26900011e0 tx=0x7f269000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:50.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.915+0000 7f2697400700 1 -- 172.21.15.89:0/2921725389 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690004c80 con 0x7f26a00ceec0 2024-04-02T07:51:50.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.915+0000 7f2697400700 1 -- 172.21.15.89:0/2921725389 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2690004e00 con 0x7f26a00ceec0 2024-04-02T07:51:50.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.915+0000 7f26a6400700 1 -- 172.21.15.89:0/2921725389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a00c7120 con 0x7f26a00ceec0 2024-04-02T07:51:50.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.915+0000 7f2697400700 1 -- 172.21.15.89:0/2921725389 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690007310 con 0x7f26a00ceec0 2024-04-02T07:51:50.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.916+0000 7f26a6400700 1 -- 172.21.15.89:0/2921725389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a00ce6f0 con 0x7f26a00ceec0 2024-04-02T07:51:50.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.917+0000 7f26a6400700 1 -- 172.21.15.89:0/2921725389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a00c7360 con 0x7f26a00ceec0 2024-04-02T07:51:50.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.918+0000 7f2697400700 1 -- 172.21.15.89:0/2921725389 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f2690007470 con 0x7f26a00ceec0 2024-04-02T07:51:50.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.919+0000 7f2697400700 1 --2- 172.21.15.89:0/2921725389 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f268006adb0 0x7f268006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:50.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.919+0000 7f2697400700 1 -- 172.21.15.89:0/2921725389 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f2690089430 con 0x7f26a00ceec0 2024-04-02T07:51:50.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.919+0000 7f269ea00700 1 -- 172.21.15.89:0/2921725389 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f268006adb0 msgr2=0x7f268006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:50.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.919+0000 7f269ea00700 1 --2- 172.21.15.89:0/2921725389 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f268006adb0 0x7f268006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:50.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:50.923+0000 7f2697400700 1 -- 172.21.15.89:0/2921725389 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f26a00c7360 con 0x7f26a00ceec0 2024-04-02T07:51:50.990 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:50.989+0000 7f99d5098240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:51:51.061 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:51.059+0000 7fe5e17b3240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:51:51.088 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:51.086+0000 7f99d5098240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:51:51.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.120+0000 7f269ea00700 1 -- 172.21.15.89:0/2921725389 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f268006adb0 msgr2=0x7f268006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:51.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.120+0000 7f269ea00700 1 --2- 172.21.15.89:0/2921725389 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f268006adb0 0x7f268006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:51.157 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:51.156+0000 7fe5e17b3240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:51:51.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.179+0000 7f26a6400700 1 -- 172.21.15.89:0/2921725389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26a00d3030 con 0x7f26a00ceec0 2024-04-02T07:51:51.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.180+0000 7f2697400700 1 -- 172.21.15.89:0/2921725389 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f26900541a0 con 0x7f26a00ceec0 2024-04-02T07:51:51.182 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:51.182 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2025634273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2025634273},{"type":"v1","addr":"172.21.15.113:6834","nonce":2025634273}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2247445394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2247445394},{"type":"v1","addr":"172.21.15.113:6835","nonce":2247445394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1785147820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1785147820},{"type":"v1","addr":"172.21.15.89:6835","nonce":1785147820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:51:29.684122+0000","modified":"2024-04-02T07:51:32.705923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6432":{"gid":6432,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/1479795102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1479795102},{"type":"v1","addr":"172.21.15.89:6837","nonce":1479795102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-02T07:51:51.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.182+0000 7f26a6400700 1 -- 172.21.15.89:0/2921725389 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f268006adb0 msgr2=0x7f268006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:51.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.182+0000 7f26a6400700 1 --2- 172.21.15.89:0/2921725389 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f268006adb0 0x7f268006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.182+0000 7f26a6400700 1 -- 172.21.15.89:0/2921725389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a00ceec0 msgr2=0x7f26a01f65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:51.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.182+0000 7f26a6400700 1 --2- 172.21.15.89:0/2921725389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a00ceec0 0x7f26a01f65b0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f26900011e0 tx=0x7f269000ca30 comp rx=0 tx=0).stop 2024-04-02T07:51:51.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.182+0000 7f26a6400700 1 -- 172.21.15.89:0/2921725389 shutdown_connections 2024-04-02T07:51:51.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.182+0000 7f26a6400700 1 --2- 172.21.15.89:0/2921725389 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f268006adb0 0x7f268006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.182+0000 7f26a6400700 1 --2- 172.21.15.89:0/2921725389 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26a0161800 0x7f26a009d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.182+0000 7f26a6400700 1 --2- 172.21.15.89:0/2921725389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26a00ceec0 0x7f26a01f65b0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.182+0000 7f26a6400700 1 --2- 172.21.15.89:0/2921725389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26a015d640 0x7f26a00a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.183+0000 7f26a6400700 1 -- 172.21.15.89:0/2921725389 >> 172.21.15.89:0/2921725389 conn(0x7f26a008aff0 msgr2=0x7f26a00d1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:51.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.183+0000 7f26a6400700 1 -- 172.21.15.89:0/2921725389 shutdown_connections 2024-04-02T07:51:51.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.183+0000 7f26a6400700 1 -- 172.21.15.89:0/2921725389 wait complete. 2024-04-02T07:51:51.185 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:51.184+0000 7f99d5098240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:51:51.186 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 33 2024-04-02T07:51:51.203 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:51:51.263 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:51.262+0000 7fe5e17b3240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:51:51.283 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:51.281+0000 7f99d5098240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:51:51.367 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:51.365+0000 7fe5e17b3240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:51:51.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.397+0000 7f40d9600700 1 Processor -- start 2024-04-02T07:51:51.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.398+0000 7f40d9600700 1 -- start start 2024-04-02T07:51:51.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.399+0000 7f40d9600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f40d415b5d0 0x7f40d40ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:51.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.399+0000 7f40d9600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f40d40ac7d0 0x7f40d406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:51.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.399+0000 7f40d9600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d415fd50 con 0x7f40d40cb3f0 2024-04-02T07:51:51.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.399+0000 7f40d9600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40d415b9b0 con 0x7f40d40ac7d0 2024-04-02T07:51:51.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.399+0000 7f40d9600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40d415f140 con 0x7f40d415b5d0 2024-04-02T07:51:51.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.399+0000 7f40d3400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f40d40ac7d0 0x7f40d406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:51.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.399+0000 7f40d2a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f40d415b5d0 0x7f40d40ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:51.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.399+0000 7f40d2a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f40d415b5d0 0x7f40d40ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:49274/0 (socket says 172.21.15.89:49274) 2024-04-02T07:51:51.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.399+0000 7f40d2a00700 1 -- 172.21.15.89:0/2401764617 learned_addr learned my addr 172.21.15.89:0/2401764617 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:51.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.399+0000 7f40d3400700 1 -- 172.21.15.89:0/2401764617 >> v2:172.21.15.113:3300/0 conn(0x7f40d415b5d0 msgr2=0x7f40d40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:51.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.399+0000 7f40d3400700 1 --2- 172.21.15.89:0/2401764617 >> v2:172.21.15.113:3300/0 conn(0x7f40d415b5d0 0x7f40d40ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.399+0000 7f40d3400700 1 -- 172.21.15.89:0/2401764617 >> v1:172.21.15.113:6789/0 conn(0x7f40d40cb3f0 legacy=0x7f40d40ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:51.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.400+0000 7f40d3400700 1 -- 172.21.15.89:0/2401764617 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d415f420 con 0x7f40d40ac7d0 2024-04-02T07:51:51.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.400+0000 7f40d3400700 1 --2- 172.21.15.89:0/2401764617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d40ac7d0 0x7f40d406c9d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f40d406b8b0 tx=0x7f40cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d2ca85c2922848 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:51.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.400+0000 7f40d1600700 1 -- 172.21.15.89:0/2401764617 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc00e070 con 0x7f40d40ac7d0 2024-04-02T07:51:51.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.400+0000 7f40d1600700 1 -- 172.21.15.89:0/2401764617 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40cc00bb20 con 0x7f40d40ac7d0 2024-04-02T07:51:51.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.400+0000 7f40d1600700 1 -- 172.21.15.89:0/2401764617 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc012430 con 0x7f40d40ac7d0 2024-04-02T07:51:51.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.401+0000 7f40d9600700 1 -- 172.21.15.89:0/2401764617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d40ac7d0 msgr2=0x7f40d406c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:51.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.401+0000 7f40d9600700 1 --2- 172.21.15.89:0/2401764617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d40ac7d0 0x7f40d406c9d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f40d406b8b0 tx=0x7f40cc00b330 comp rx=0 tx=0).stop 2024-04-02T07:51:51.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.401+0000 7f40d9600700 1 -- 172.21.15.89:0/2401764617 shutdown_connections 2024-04-02T07:51:51.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.401+0000 7f40d9600700 1 --2- 172.21.15.89:0/2401764617 >> v2:172.21.15.113:3300/0 conn(0x7f40d415b5d0 0x7f40d40ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.401+0000 7f40d9600700 1 --2- 172.21.15.89:0/2401764617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d40ac7d0 0x7f40d406c9d0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.401+0000 7f40d9600700 1 -- 172.21.15.89:0/2401764617 >> 172.21.15.89:0/2401764617 conn(0x7f40d40cf190 msgr2=0x7f40d40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:51.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.401+0000 7f40d9600700 1 -- 172.21.15.89:0/2401764617 shutdown_connections 2024-04-02T07:51:51.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.401+0000 7f40d9600700 1 -- 172.21.15.89:0/2401764617 wait complete. 2024-04-02T07:51:51.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.402+0000 7f40d9600700 1 Processor -- start 2024-04-02T07:51:51.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.403+0000 7f40d9600700 1 -- start start 2024-04-02T07:51:51.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.403+0000 7f40d9600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40d40ac7d0 0x7f40d41564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:51.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.403+0000 7f40d9600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40d415b5d0 0x7f40d4156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:51.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d9600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d40cbba0 0x7f40d4071010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:51.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d9600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d40cccb0 con 0x7f40d40cbba0 2024-04-02T07:51:51.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d9600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d40cd0d0 con 0x7f40d415b5d0 2024-04-02T07:51:51.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d9600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d40cc030 con 0x7f40d40ac7d0 2024-04-02T07:51:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d40cbba0 0x7f40d4071010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d40cbba0 0x7f40d4071010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43142/0 (socket says 172.21.15.89:43142) 2024-04-02T07:51:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d3400700 1 -- 172.21.15.89:0/1539280324 learned_addr learned my addr 172.21.15.89:0/1539280324 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d2000700 1 --2- 172.21.15.89:0/1539280324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40d415b5d0 0x7f40d4156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d2a00700 1 --2- 172.21.15.89:0/1539280324 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40d40ac7d0 0x7f40d41564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d3400700 1 -- 172.21.15.89:0/1539280324 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40d40ac7d0 msgr2=0x7f40d41564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d3400700 1 --2- 172.21.15.89:0/1539280324 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40d40ac7d0 0x7f40d41564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d3400700 1 -- 172.21.15.89:0/1539280324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40d415b5d0 msgr2=0x7f40d4156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:51.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d3400700 1 --2- 172.21.15.89:0/1539280324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40d415b5d0 0x7f40d4156b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d3400700 1 -- 172.21.15.89:0/1539280324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc009a70 con 0x7f40d40cbba0 2024-04-02T07:51:51.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.404+0000 7f40d3400700 1 --2- 172.21.15.89:0/1539280324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d40cbba0 0x7f40d4071010 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f40cc014040 tx=0x7f40cc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:51.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.405+0000 7f40caa00700 1 -- 172.21.15.89:0/1539280324 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc012430 con 0x7f40d40cbba0 2024-04-02T07:51:51.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.405+0000 7f40caa00700 1 -- 172.21.15.89:0/1539280324 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40cc020070 con 0x7f40d40cbba0 2024-04-02T07:51:51.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.405+0000 7f40caa00700 1 -- 172.21.15.89:0/1539280324 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc0098f0 con 0x7f40d40cbba0 2024-04-02T07:51:51.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.405+0000 7f40d9600700 1 -- 172.21.15.89:0/1539280324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d40cc310 con 0x7f40d40cbba0 2024-04-02T07:51:51.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.405+0000 7f40d9600700 1 -- 172.21.15.89:0/1539280324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d40cc8d0 con 0x7f40d40cbba0 2024-04-02T07:51:51.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.406+0000 7f40d9600700 1 -- 172.21.15.89:0/1539280324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d40ccdf0 con 0x7f40d40cbba0 2024-04-02T07:51:51.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.407+0000 7f40caa00700 1 -- 172.21.15.89:0/1539280324 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f40cc004040 con 0x7f40d40cbba0 2024-04-02T07:51:51.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.408+0000 7f40caa00700 1 --2- 172.21.15.89:0/1539280324 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40b406adb0 0x7f40b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:51.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.408+0000 7f40caa00700 1 -- 172.21.15.89:0/1539280324 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f40cc022070 con 0x7f40d40cbba0 2024-04-02T07:51:51.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.408+0000 7f40d2a00700 1 -- 172.21.15.89:0/1539280324 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40b406adb0 msgr2=0x7f40b406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:51.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.408+0000 7f40d2a00700 1 --2- 172.21.15.89:0/1539280324 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40b406adb0 0x7f40b406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:51.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.412+0000 7f40caa00700 1 -- 172.21.15.89:0/1539280324 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f40cc056180 con 0x7f40d40cbba0 2024-04-02T07:51:51.471 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:51.470+0000 7fe5e17b3240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:51:51.494 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:51.491+0000 7f99d5098240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:51:51.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.609+0000 7f40d2a00700 1 -- 172.21.15.89:0/1539280324 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40b406adb0 msgr2=0x7f40b406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:51.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.609+0000 7f40d2a00700 1 --2- 172.21.15.89:0/1539280324 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40b406adb0 0x7f40b406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:51.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.626+0000 7f40d9600700 1 -- 172.21.15.89:0/1539280324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40d4067720 con 0x7f40d40cbba0 2024-04-02T07:51:51.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.627+0000 7f40caa00700 1 -- 172.21.15.89:0/1539280324 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+17111 (secure 0 0 0) 0x7f40cc0597a0 con 0x7f40d40cbba0 2024-04-02T07:51:51.629 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:51.630 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":84,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:51:45.730961+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:51:27.926367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-02T07:51:28.662600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/822661295":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/907732608":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/3083880072":"2024-04-03T07:51:45.730929+0000","172.21.15.89:6833/43357":"2024-04-03T07:51:45.730929+0000","172.21.15.89:6832/43357":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/1211892288":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/23281819":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/3435265311":"2024-04-03T07:51:45.730929+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:51:51.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.630+0000 7f40d9600700 1 -- 172.21.15.89:0/1539280324 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40b406adb0 msgr2=0x7f40b406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:51.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.630+0000 7f40d9600700 1 --2- 172.21.15.89:0/1539280324 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40b406adb0 0x7f40b406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.630+0000 7f40d9600700 1 -- 172.21.15.89:0/1539280324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d40cbba0 msgr2=0x7f40d4071010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:51.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.630+0000 7f40d9600700 1 --2- 172.21.15.89:0/1539280324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d40cbba0 0x7f40d4071010 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f40cc014040 tx=0x7f40cc004ad0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.630+0000 7f40d9600700 1 -- 172.21.15.89:0/1539280324 shutdown_connections 2024-04-02T07:51:51.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.630+0000 7f40d9600700 1 --2- 172.21.15.89:0/1539280324 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f40b406adb0 0x7f40b406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.630+0000 7f40d9600700 1 --2- 172.21.15.89:0/1539280324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40d40cbba0 0x7f40d4071010 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.630+0000 7f40d9600700 1 --2- 172.21.15.89:0/1539280324 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40d40ac7d0 0x7f40d41564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.630+0000 7f40d9600700 1 --2- 172.21.15.89:0/1539280324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40d415b5d0 0x7f40d4156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.630+0000 7f40d9600700 1 -- 172.21.15.89:0/1539280324 >> 172.21.15.89:0/1539280324 conn(0x7f40d40cf190 msgr2=0x7f40d415e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:51.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.630+0000 7f40d9600700 1 -- 172.21.15.89:0/1539280324 shutdown_connections 2024-04-02T07:51:51.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.630+0000 7f40d9600700 1 -- 172.21.15.89:0/1539280324 wait complete. 2024-04-02T07:51:51.650 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:51:27.926367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:51:28.662600+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:51:51.651 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:51:51.695 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:51.693+0000 7fe5e17b3240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:51:51.746 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:51.744+0000 7f99d5098240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:51:51.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.854+0000 7f272b400700 1 Processor -- start 2024-04-02T07:51:51.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.854+0000 7f272b400700 1 -- start start 2024-04-02T07:51:51.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.856+0000 7f272b400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f27240d16a0 0x7f27240d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:51.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.856+0000 7f272b400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2724161420 con 0x7f272415d9d0 2024-04-02T07:51:51.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.856+0000 7f272b400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2724161700 con 0x7f27240d2370 2024-04-02T07:51:51.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.856+0000 7f272b400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27240d4340 con 0x7f27240d16a0 2024-04-02T07:51:51.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.857+0000 7f2728a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f272415d9d0 0x7f27240d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45998/0 (socket says 172.21.15.89:45998) 2024-04-02T07:51:51.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.857+0000 7f2728a00700 1 -- 172.21.15.89:0/3963052188 learned_addr learned my addr 172.21.15.89:0/3963052188 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:51.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.857+0000 7f2723e00700 1 --2- 172.21.15.89:0/3963052188 >> v2:172.21.15.113:3300/0 conn(0x7f27240d16a0 0x7f27240d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:51.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.857+0000 7f2723400700 1 -- 172.21.15.89:0/3963052188 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168850876 0 0) 0x7f2724161420 con 0x7f272415d9d0 2024-04-02T07:51:51.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.857+0000 7f2723400700 1 -- 172.21.15.89:0/3963052188 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f270c0035b0 con 0x7f272415d9d0 2024-04-02T07:51:51.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.857+0000 7f2723e00700 1 -- 172.21.15.89:0/3963052188 >> v1:172.21.15.113:6789/0 conn(0x7f27240d2370 legacy=0x7f27240ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:51.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.857+0000 7f2723400700 1 -- 172.21.15.89:0/3963052188 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469879741 0 0) 0x7f2724161700 con 0x7f27240d2370 2024-04-02T07:51:51.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.857+0000 7f2723e00700 1 -- 172.21.15.89:0/3963052188 >> v1:172.21.15.89:6789/0 conn(0x7f272415d9d0 legacy=0x7f27240d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:51.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.857+0000 7f2723e00700 1 -- 172.21.15.89:0/3963052188 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27241619e0 con 0x7f27240d16a0 2024-04-02T07:51:51.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.858+0000 7f2723e00700 1 --2- 172.21.15.89:0/3963052188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240d16a0 0x7f27240d1e30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2710009820 tx=0x7f2710009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a4bbbefbb123b21 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:51.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.858+0000 7f2723400700 1 -- 172.21.15.89:0/3963052188 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710005e30 con 0x7f27240d16a0 2024-04-02T07:51:51.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.858+0000 7f2723400700 1 -- 172.21.15.89:0/3963052188 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2710006030 con 0x7f27240d16a0 2024-04-02T07:51:51.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.858+0000 7f2723400700 1 -- 172.21.15.89:0/3963052188 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27100063a0 con 0x7f27240d16a0 2024-04-02T07:51:51.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.859+0000 7f272b400700 1 -- 172.21.15.89:0/3963052188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240d16a0 msgr2=0x7f27240d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:51.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.859+0000 7f272b400700 1 --2- 172.21.15.89:0/3963052188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240d16a0 0x7f27240d1e30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2710009820 tx=0x7f2710009b30 comp rx=0 tx=0).stop 2024-04-02T07:51:51.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.859+0000 7f272b400700 1 -- 172.21.15.89:0/3963052188 shutdown_connections 2024-04-02T07:51:51.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.859+0000 7f272b400700 1 --2- 172.21.15.89:0/3963052188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240d16a0 0x7f27240d1e30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.859+0000 7f272b400700 1 -- 172.21.15.89:0/3963052188 >> 172.21.15.89:0/3963052188 conn(0x7f272408aff0 msgr2=0x7f27240d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:51.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.859+0000 7f272b400700 1 -- 172.21.15.89:0/3963052188 shutdown_connections 2024-04-02T07:51:51.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.859+0000 7f272b400700 1 -- 172.21.15.89:0/3963052188 wait complete. 2024-04-02T07:51:51.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.862+0000 7f272b400700 1 Processor -- start 2024-04-02T07:51:51.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.862+0000 7f272b400700 1 -- start start 2024-04-02T07:51:51.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.864+0000 7f272b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27240d16a0 0x7f272415bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:51.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.864+0000 7f272b400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240d2370 0x7f272415c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:51.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.864+0000 7f272b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f272415d9d0 0x7f27241f6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:51.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.864+0000 7f272b400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27241f7a50 con 0x7f27240d16a0 2024-04-02T07:51:51.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.864+0000 7f272b400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27241f7770 con 0x7f27240d2370 2024-04-02T07:51:51.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.864+0000 7f272b400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f272415f5a0 con 0x7f272415d9d0 2024-04-02T07:51:51.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.864+0000 7f2728a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27240d16a0 0x7f272415bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:51.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.865+0000 7f2723e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240d2370 0x7f272415c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:51.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.865+0000 7f2728a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27240d16a0 0x7f272415bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43150/0 (socket says 172.21.15.89:43150) 2024-04-02T07:51:51.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.865+0000 7f2728a00700 1 -- 172.21.15.89:0/263400537 learned_addr learned my addr 172.21.15.89:0/263400537 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:51.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.865+0000 7f2729400700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f272415d9d0 0x7f27241f6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:51.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.865+0000 7f2723e00700 1 -- 172.21.15.89:0/263400537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f272415d9d0 msgr2=0x7f27241f6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:51.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.865+0000 7f2723e00700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f272415d9d0 0x7f27241f6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.865+0000 7f2723e00700 1 -- 172.21.15.89:0/263400537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27240d16a0 msgr2=0x7f272415bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:51.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.865+0000 7f2723e00700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27240d16a0 0x7f272415bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:51.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.865+0000 7f2723e00700 1 -- 172.21.15.89:0/263400537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2710009430 con 0x7f27240d2370 2024-04-02T07:51:51.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.865+0000 7f2729400700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f272415d9d0 0x7f27241f6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:51.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.865+0000 7f2728a00700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27240d16a0 0x7f272415bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:51.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.865+0000 7f2723e00700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240d2370 0x7f272415c5b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2710015000 tx=0x7f2710004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:51.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.866+0000 7f2720c00700 1 -- 172.21.15.89:0/263400537 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710006910 con 0x7f27240d2370 2024-04-02T07:51:51.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.866+0000 7f2720c00700 1 -- 172.21.15.89:0/263400537 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2710020070 con 0x7f27240d2370 2024-04-02T07:51:51.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.866+0000 7f272b400700 1 -- 172.21.15.89:0/263400537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272415f880 con 0x7f27240d2370 2024-04-02T07:51:51.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.866+0000 7f2720c00700 1 -- 172.21.15.89:0/263400537 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710004280 con 0x7f27240d2370 2024-04-02T07:51:51.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.866+0000 7f272b400700 1 -- 172.21.15.89:0/263400537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f272415c310 con 0x7f27240d2370 2024-04-02T07:51:51.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.867+0000 7f272b400700 1 -- 172.21.15.89:0/263400537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27240d1ca0 con 0x7f27240d2370 2024-04-02T07:51:51.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.868+0000 7f2720c00700 1 -- 172.21.15.89:0/263400537 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f2710022030 con 0x7f27240d2370 2024-04-02T07:51:51.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.869+0000 7f2720c00700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f270406adb0 0x7f270406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:51.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.869+0000 7f2728a00700 1 -- 172.21.15.89:0/263400537 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f270406adb0 msgr2=0x7f270406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:51.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.869+0000 7f2720c00700 1 -- 172.21.15.89:0/263400537 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f271008ac70 con 0x7f27240d2370 2024-04-02T07:51:51.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.870+0000 7f2728a00700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f270406adb0 0x7f270406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:51.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:51.879+0000 7f2720c00700 1 -- 172.21.15.89:0/263400537 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2710055b10 con 0x7f27240d2370 2024-04-02T07:51:51.942 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:51.940+0000 7f99d5098240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:51:51.966 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:51.963+0000 7fe5e17b3240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:51:52.034 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:52.032+0000 7f99d5098240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:51:52.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.071+0000 7f2728a00700 1 -- 172.21.15.89:0/263400537 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f270406adb0 msgr2=0x7f270406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:52.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.071+0000 7f2728a00700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f270406adb0 0x7f270406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:52.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.090+0000 7f272b400700 1 -- 172.21.15.89:0/263400537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f272415f400 con 0x7f27240d2370 2024-04-02T07:51:52.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.090+0000 7f2720c00700 1 -- 172.21.15.89:0/263400537 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f272415f400 con 0x7f27240d2370 2024-04-02T07:51:52.092 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:51:52.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.093+0000 7f272b400700 1 -- 172.21.15.89:0/263400537 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f270406adb0 msgr2=0x7f270406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:52.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.093+0000 7f272b400700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f270406adb0 0x7f270406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.093+0000 7f272b400700 1 -- 172.21.15.89:0/263400537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240d2370 msgr2=0x7f272415c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:52.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.093+0000 7f272b400700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240d2370 0x7f272415c5b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2710015000 tx=0x7f2710004750 comp rx=0 tx=0).stop 2024-04-02T07:51:52.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.093+0000 7f272b400700 1 -- 172.21.15.89:0/263400537 shutdown_connections 2024-04-02T07:51:52.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.093+0000 7f272b400700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f270406adb0 0x7f270406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.093+0000 7f272b400700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f272415d9d0 0x7f27241f6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.093+0000 7f272b400700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27240d16a0 0x7f272415bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.093+0000 7f272b400700 1 --2- 172.21.15.89:0/263400537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27240d2370 0x7f272415c5b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.093+0000 7f272b400700 1 -- 172.21.15.89:0/263400537 >> 172.21.15.89:0/263400537 conn(0x7f272408aff0 msgr2=0x7f27241e9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:52.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.093+0000 7f272b400700 1 -- 172.21.15.89:0/263400537 shutdown_connections 2024-04-02T07:51:52.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.093+0000 7f272b400700 1 -- 172.21.15.89:0/263400537 wait complete. 2024-04-02T07:51:52.115 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:51:52.165 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:52.163+0000 7fe5e17b3240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:51:52.263 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:52.262+0000 7fe5e17b3240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:51:52.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.352+0000 7fd07d400700 1 Processor -- start 2024-04-02T07:51:52.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.352+0000 7fd07d400700 1 -- start start 2024-04-02T07:51:52.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.353+0000 7fd07d400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd0780d16a0 0x7fd0780d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:52.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.353+0000 7fd07d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0780d2370 0x7fd0780ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:52.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.353+0000 7fd07d400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd078161420 con 0x7fd07815d9d0 2024-04-02T07:51:52.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.353+0000 7fd07d400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0780d4340 con 0x7fd0780d2370 2024-04-02T07:51:52.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.354+0000 7fd07d400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07815edc0 con 0x7fd0780d16a0 2024-04-02T07:51:52.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.354+0000 7fd077200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0780d2370 0x7fd0780ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:52.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.354+0000 7fd06fe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd0780d16a0 0x7fd0780d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:52.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.354+0000 7fd077200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0780d2370 0x7fd0780ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43158/0 (socket says 172.21.15.89:43158) 2024-04-02T07:51:52.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.354+0000 7fd077200700 1 -- 172.21.15.89:0/1693597585 learned_addr learned my addr 172.21.15.89:0/1693597585 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:52.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.354+0000 7fd077200700 1 -- 172.21.15.89:0/1693597585 >> v2:172.21.15.113:3300/0 conn(0x7fd0780d16a0 msgr2=0x7fd0780d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:52.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.354+0000 7fd077200700 1 --2- 172.21.15.89:0/1693597585 >> v2:172.21.15.113:3300/0 conn(0x7fd0780d16a0 0x7fd0780d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.354+0000 7fd077200700 1 -- 172.21.15.89:0/1693597585 >> v1:172.21.15.113:6789/0 conn(0x7fd07815d9d0 legacy=0x7fd0780d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:52.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.354+0000 7fd077200700 1 -- 172.21.15.89:0/1693597585 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd078161700 con 0x7fd0780d2370 2024-04-02T07:51:52.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.354+0000 7fd075e00700 1 -- 172.21.15.89:0/1693597585 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234059682 0 0) 0x7fd078161420 con 0x7fd07815d9d0 2024-04-02T07:51:52.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.355+0000 7fd077200700 1 --2- 172.21.15.89:0/1693597585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0780d2370 0x7fd0780ce360 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fd0780d12f0 tx=0x7fd07000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3787dee894837b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:52.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.355+0000 7fd075e00700 1 -- 172.21.15.89:0/1693597585 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07000e070 con 0x7fd0780d2370 2024-04-02T07:51:52.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.355+0000 7fd075e00700 1 -- 172.21.15.89:0/1693597585 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07000b910 con 0x7fd0780d2370 2024-04-02T07:51:52.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.355+0000 7fd075e00700 1 -- 172.21.15.89:0/1693597585 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07000bcc0 con 0x7fd0780d2370 2024-04-02T07:51:52.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.355+0000 7fd07d400700 1 -- 172.21.15.89:0/1693597585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0780d2370 msgr2=0x7fd0780ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:52.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.355+0000 7fd07d400700 1 --2- 172.21.15.89:0/1693597585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0780d2370 0x7fd0780ce360 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fd0780d12f0 tx=0x7fd07000b330 comp rx=0 tx=0).stop 2024-04-02T07:51:52.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.356+0000 7fd07d400700 1 -- 172.21.15.89:0/1693597585 shutdown_connections 2024-04-02T07:51:52.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.356+0000 7fd07d400700 1 --2- 172.21.15.89:0/1693597585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0780d2370 0x7fd0780ce360 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.356+0000 7fd07d400700 1 --2- 172.21.15.89:0/1693597585 >> v2:172.21.15.113:3300/0 conn(0x7fd0780d16a0 0x7fd0780d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.356+0000 7fd07d400700 1 -- 172.21.15.89:0/1693597585 >> 172.21.15.89:0/1693597585 conn(0x7fd07808aff0 msgr2=0x7fd0780d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:52.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.356+0000 7fd07d400700 1 -- 172.21.15.89:0/1693597585 shutdown_connections 2024-04-02T07:51:52.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.356+0000 7fd07d400700 1 -- 172.21.15.89:0/1693597585 wait complete. 2024-04-02T07:51:52.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.358+0000 7fd07d400700 1 Processor -- start 2024-04-02T07:51:52.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.358+0000 7fd07d400700 1 -- start start 2024-04-02T07:51:52.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.359+0000 7fd07d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0780d16a0 0x7fd07815c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:52.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.359+0000 7fd07d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780d2370 0x7fd0780cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:52.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.359+0000 7fd07d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd07815d9d0 0x7fd07815bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:52.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.359+0000 7fd07d400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0780b6390 con 0x7fd07815d9d0 2024-04-02T07:51:52.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.359+0000 7fd07d400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0780b6670 con 0x7fd0780d16a0 2024-04-02T07:51:52.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.359+0000 7fd07d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0780b6950 con 0x7fd0780d2370 2024-04-02T07:51:52.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.360+0000 7fd077200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd07815d9d0 0x7fd07815bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:52.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.360+0000 7fd06fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780d2370 0x7fd0780cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:52.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.360+0000 7fd077200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd07815d9d0 0x7fd07815bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43160/0 (socket says 172.21.15.89:43160) 2024-04-02T07:51:52.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.360+0000 7fd076800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0780d16a0 0x7fd07815c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:52.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.360+0000 7fd077200700 1 -- 172.21.15.89:0/1322504626 learned_addr learned my addr 172.21.15.89:0/1322504626 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:52.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.360+0000 7fd077200700 1 -- 172.21.15.89:0/1322504626 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780d2370 msgr2=0x7fd0780cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:52.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.360+0000 7fd077200700 1 --2- 172.21.15.89:0/1322504626 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780d2370 0x7fd0780cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.360+0000 7fd077200700 1 -- 172.21.15.89:0/1322504626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0780d16a0 msgr2=0x7fd07815c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:52.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.360+0000 7fd077200700 1 --2- 172.21.15.89:0/1322504626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0780d16a0 0x7fd07815c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.360+0000 7fd077200700 1 -- 172.21.15.89:0/1322504626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd070009b30 con 0x7fd07815d9d0 2024-04-02T07:51:52.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.360+0000 7fd077200700 1 --2- 172.21.15.89:0/1322504626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd07815d9d0 0x7fd07815bb20 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fd07808c230 tx=0x7fd070004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:52.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.361+0000 7fd06ea00700 1 -- 172.21.15.89:0/1322504626 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070004840 con 0x7fd07815d9d0 2024-04-02T07:51:52.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.361+0000 7fd06ea00700 1 -- 172.21.15.89:0/1322504626 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0700099b0 con 0x7fd07815d9d0 2024-04-02T07:51:52.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.361+0000 7fd07d400700 1 -- 172.21.15.89:0/1322504626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0780b6c30 con 0x7fd07815d9d0 2024-04-02T07:51:52.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.361+0000 7fd06ea00700 1 -- 172.21.15.89:0/1322504626 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07001c680 con 0x7fd07815d9d0 2024-04-02T07:51:52.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.361+0000 7fd07d400700 1 -- 172.21.15.89:0/1322504626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0780cb230 con 0x7fd07815d9d0 2024-04-02T07:51:52.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.362+0000 7fd07d400700 1 -- 172.21.15.89:0/1322504626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0780d1ca0 con 0x7fd07815d9d0 2024-04-02T07:51:52.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.363+0000 7fd06ea00700 1 -- 172.21.15.89:0/1322504626 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fd07001c7e0 con 0x7fd07815d9d0 2024-04-02T07:51:52.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.364+0000 7fd06ea00700 1 --2- 172.21.15.89:0/1322504626 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd05406adb0 0x7fd05406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:52.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.364+0000 7fd06ea00700 1 -- 172.21.15.89:0/1322504626 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd07008b140 con 0x7fd07815d9d0 2024-04-02T07:51:52.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.364+0000 7fd076800700 1 -- 172.21.15.89:0/1322504626 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd05406adb0 msgr2=0x7fd05406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:52.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.364+0000 7fd076800700 1 --2- 172.21.15.89:0/1322504626 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd05406adb0 0x7fd05406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:52.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.368+0000 7fd06ea00700 1 -- 172.21.15.89:0/1322504626 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd070055e30 con 0x7fd07815d9d0 2024-04-02T07:51:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.565+0000 7fd076800700 1 -- 172.21.15.89:0/1322504626 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd05406adb0 msgr2=0x7fd05406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.565+0000 7fd076800700 1 --2- 172.21.15.89:0/1322504626 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd05406adb0 0x7fd05406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:52.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.575+0000 7fd07d400700 1 -- 172.21.15.89:0/1322504626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd078067720 con 0x7fd07815d9d0 2024-04-02T07:51:52.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.576+0000 7fd06ea00700 1 -- 172.21.15.89:0/1322504626 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v84) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd070059450 con 0x7fd07815d9d0 2024-04-02T07:51:52.578 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:51:52.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.578+0000 7fd07d400700 1 -- 172.21.15.89:0/1322504626 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd05406adb0 msgr2=0x7fd05406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:52.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.578+0000 7fd07d400700 1 --2- 172.21.15.89:0/1322504626 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd05406adb0 0x7fd05406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.578+0000 7fd07d400700 1 -- 172.21.15.89:0/1322504626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd07815d9d0 msgr2=0x7fd07815bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:52.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.578+0000 7fd07d400700 1 --2- 172.21.15.89:0/1322504626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd07815d9d0 0x7fd07815bb20 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fd07808c230 tx=0x7fd070004ad0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.578+0000 7fd07d400700 1 -- 172.21.15.89:0/1322504626 shutdown_connections 2024-04-02T07:51:52.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.578+0000 7fd07d400700 1 --2- 172.21.15.89:0/1322504626 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd05406adb0 0x7fd05406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.578+0000 7fd07d400700 1 --2- 172.21.15.89:0/1322504626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0780d16a0 0x7fd07815c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.578+0000 7fd07d400700 1 --2- 172.21.15.89:0/1322504626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd07815d9d0 0x7fd07815bb20 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.578+0000 7fd07d400700 1 --2- 172.21.15.89:0/1322504626 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780d2370 0x7fd0780cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.578+0000 7fd07d400700 1 -- 172.21.15.89:0/1322504626 >> 172.21.15.89:0/1322504626 conn(0x7fd07808aff0 msgr2=0x7fd0780b1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:52.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.579+0000 7fd07d400700 1 -- 172.21.15.89:0/1322504626 shutdown_connections 2024-04-02T07:51:52.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.579+0000 7fd07d400700 1 -- 172.21.15.89:0/1322504626 wait complete. 2024-04-02T07:51:52.598 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:51:52.760 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:52.758+0000 7f99d5098240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:51:52.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.835+0000 7fc717a00700 1 Processor -- start 2024-04-02T07:51:52.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.835+0000 7fc717a00700 1 -- start start 2024-04-02T07:51:52.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.837+0000 7fc717a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc7100cbb30 0x7fc7100cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:52.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.837+0000 7fc717a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc7100cd4a0 0x7fc71006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:52.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.837+0000 7fc717a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7100cc450 con 0x7fc7100cd020 2024-04-02T07:51:52.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.837+0000 7fc717a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7100cdae0 con 0x7fc7100cd4a0 2024-04-02T07:51:52.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.837+0000 7fc717a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7100cddc0 con 0x7fc7100cbb30 2024-04-02T07:51:52.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.837+0000 7fc715000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc7100cbb30 0x7fc7100cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:52.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.837+0000 7fc715a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc7100cd4a0 0x7fc71006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:52.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.837+0000 7fc715000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc7100cbb30 0x7fc7100cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:47004/0 (socket says 172.21.15.89:47004) 2024-04-02T07:51:52.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.837+0000 7fc715000700 1 -- 172.21.15.89:0/1965451053 learned_addr learned my addr 172.21.15.89:0/1965451053 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:52.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.837+0000 7fc715000700 1 -- 172.21.15.89:0/1965451053 >> v2:172.21.15.113:3300/0 conn(0x7fc7100cd4a0 msgr2=0x7fc71006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:52.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.837+0000 7fc715000700 1 --2- 172.21.15.89:0/1965451053 >> v2:172.21.15.113:3300/0 conn(0x7fc7100cd4a0 0x7fc71006e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.837+0000 7fc715000700 1 -- 172.21.15.89:0/1965451053 >> v1:172.21.15.113:6789/0 conn(0x7fc7100cd020 legacy=0x7fc7100b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:52.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.837+0000 7fc715000700 1 -- 172.21.15.89:0/1965451053 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7100ce0a0 con 0x7fc7100cbb30 2024-04-02T07:51:52.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.838+0000 7fc715000700 1 --2- 172.21.15.89:0/1965451053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc7100cbb30 0x7fc7100cbf10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc7040098f0 tx=0x7fc704009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f20fd6c2937d6627 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:52.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.838+0000 7fc70f400700 1 -- 172.21.15.89:0/1965451053 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704011830 con 0x7fc7100cbb30 2024-04-02T07:51:52.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.838+0000 7fc70f400700 1 -- 172.21.15.89:0/1965451053 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc704011990 con 0x7fc7100cbb30 2024-04-02T07:51:52.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.838+0000 7fc70f400700 1 -- 172.21.15.89:0/1965451053 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704011d20 con 0x7fc7100cbb30 2024-04-02T07:51:52.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.839+0000 7fc717a00700 1 -- 172.21.15.89:0/1965451053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc7100cbb30 msgr2=0x7fc7100cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:52.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.839+0000 7fc717a00700 1 --2- 172.21.15.89:0/1965451053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc7100cbb30 0x7fc7100cbf10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc7040098f0 tx=0x7fc704009c00 comp rx=0 tx=0).stop 2024-04-02T07:51:52.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.839+0000 7fc717a00700 1 -- 172.21.15.89:0/1965451053 shutdown_connections 2024-04-02T07:51:52.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.839+0000 7fc717a00700 1 --2- 172.21.15.89:0/1965451053 >> v2:172.21.15.113:3300/0 conn(0x7fc7100cd4a0 0x7fc71006e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.839+0000 7fc717a00700 1 --2- 172.21.15.89:0/1965451053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc7100cbb30 0x7fc7100cbf10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.839+0000 7fc717a00700 1 -- 172.21.15.89:0/1965451053 >> 172.21.15.89:0/1965451053 conn(0x7fc7100b0d70 msgr2=0x7fc7100b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:52.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.839+0000 7fc717a00700 1 -- 172.21.15.89:0/1965451053 shutdown_connections 2024-04-02T07:51:52.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.839+0000 7fc717a00700 1 -- 172.21.15.89:0/1965451053 wait complete. 2024-04-02T07:51:52.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.840+0000 7fc717a00700 1 Processor -- start 2024-04-02T07:51:52.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.841+0000 7fc717a00700 1 -- start start 2024-04-02T07:51:52.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.841+0000 7fc717a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7100cbb30 0x7fc7100b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:52.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.841+0000 7fc717a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7100cd020 0x7fc7100ad410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:52.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.841+0000 7fc717a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc7100cd4a0 0x7fc7100c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:52.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.841+0000 7fc717a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7100c8dd0 con 0x7fc7100cbb30 2024-04-02T07:51:52.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.841+0000 7fc717a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7100b1ed0 con 0x7fc7100cd020 2024-04-02T07:51:52.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.841+0000 7fc717a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7100b21b0 con 0x7fc7100cd4a0 2024-04-02T07:51:52.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.842+0000 7fc715000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7100cbb30 0x7fc7100b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:52.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.842+0000 7fc715000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7100cbb30 0x7fc7100b29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43172/0 (socket says 172.21.15.89:43172) 2024-04-02T07:51:52.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.842+0000 7fc715000700 1 -- 172.21.15.89:0/3397658045 learned_addr learned my addr 172.21.15.89:0/3397658045 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:52.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.842+0000 7fc715a00700 1 --2- 172.21.15.89:0/3397658045 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc7100cd4a0 0x7fc7100c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:52.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.842+0000 7fc70fe00700 1 --2- 172.21.15.89:0/3397658045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7100cd020 0x7fc7100ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:52.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.842+0000 7fc715000700 1 -- 172.21.15.89:0/3397658045 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc7100cd4a0 msgr2=0x7fc7100c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:52.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.842+0000 7fc715000700 1 --2- 172.21.15.89:0/3397658045 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc7100cd4a0 0x7fc7100c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.842+0000 7fc715000700 1 -- 172.21.15.89:0/3397658045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7100cd020 msgr2=0x7fc7100ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:52.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.842+0000 7fc715000700 1 --2- 172.21.15.89:0/3397658045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7100cd020 0x7fc7100ad410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:52.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.842+0000 7fc715000700 1 -- 172.21.15.89:0/3397658045 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc704009550 con 0x7fc7100cbb30 2024-04-02T07:51:52.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.842+0000 7fc715000700 1 --2- 172.21.15.89:0/3397658045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7100cbb30 0x7fc7100b29b0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fc704009c40 tx=0x7fc704004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:52.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.843+0000 7fc70cc00700 1 -- 172.21.15.89:0/3397658045 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704004510 con 0x7fc7100cbb30 2024-04-02T07:51:52.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.843+0000 7fc70cc00700 1 -- 172.21.15.89:0/3397658045 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70401e070 con 0x7fc7100cbb30 2024-04-02T07:51:52.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.843+0000 7fc717a00700 1 -- 172.21.15.89:0/3397658045 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7100b2490 con 0x7fc7100cbb30 2024-04-02T07:51:52.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.843+0000 7fc70cc00700 1 -- 172.21.15.89:0/3397658045 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7040038e0 con 0x7fc7100cbb30 2024-04-02T07:51:52.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.843+0000 7fc717a00700 1 -- 172.21.15.89:0/3397658045 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7100c90b0 con 0x7fc7100cbb30 2024-04-02T07:51:52.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.844+0000 7fc717a00700 1 -- 172.21.15.89:0/3397658045 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc710067720 con 0x7fc7100cbb30 2024-04-02T07:51:52.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.846+0000 7fc70cc00700 1 -- 172.21.15.89:0/3397658045 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fc704003a40 con 0x7fc7100cbb30 2024-04-02T07:51:52.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.847+0000 7fc70cc00700 1 --2- 172.21.15.89:0/3397658045 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc6f006adb0 0x7fc6f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:52.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.847+0000 7fc70cc00700 1 -- 172.21.15.89:0/3397658045 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fc70408a6d0 con 0x7fc7100cbb30 2024-04-02T07:51:52.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.847+0000 7fc70fe00700 1 -- 172.21.15.89:0/3397658045 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc6f006adb0 msgr2=0x7fc6f006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:52.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.847+0000 7fc70fe00700 1 --2- 172.21.15.89:0/3397658045 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc6f006adb0 0x7fc6f006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:52.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:52.851+0000 7fc70cc00700 1 -- 172.21.15.89:0/3397658045 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc704055450 con 0x7fc7100cbb30 2024-04-02T07:51:52.970 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:52.969+0000 7f99d5098240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:51:53.028 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:53.026+0000 7fe5e17b3240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:51:53.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.047+0000 7fc70fe00700 1 -- 172.21.15.89:0/3397658045 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc6f006adb0 msgr2=0x7fc6f006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:53.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.047+0000 7fc70fe00700 1 --2- 172.21.15.89:0/3397658045 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc6f006adb0 0x7fc6f006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:53.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.060+0000 7fc717a00700 1 -- 172.21.15.89:0/3397658045 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc71015bda0 con 0x7fc7100cbb30 2024-04-02T07:51:53.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.060+0000 7fc70cc00700 1 -- 172.21.15.89:0/3397658045 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v84) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc70405de60 con 0x7fc7100cbb30 2024-04-02T07:51:53.063 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:51:53.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.063+0000 7fc717a00700 1 -- 172.21.15.89:0/3397658045 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc6f006adb0 msgr2=0x7fc6f006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:53.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.063+0000 7fc717a00700 1 --2- 172.21.15.89:0/3397658045 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc6f006adb0 0x7fc6f006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.063+0000 7fc717a00700 1 -- 172.21.15.89:0/3397658045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7100cbb30 msgr2=0x7fc7100b29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.063+0000 7fc717a00700 1 --2- 172.21.15.89:0/3397658045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7100cbb30 0x7fc7100b29b0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fc704009c40 tx=0x7fc704004750 comp rx=0 tx=0).stop 2024-04-02T07:51:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.063+0000 7fc717a00700 1 -- 172.21.15.89:0/3397658045 shutdown_connections 2024-04-02T07:51:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.063+0000 7fc717a00700 1 --2- 172.21.15.89:0/3397658045 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc6f006adb0 0x7fc6f006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.063+0000 7fc717a00700 1 --2- 172.21.15.89:0/3397658045 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc7100cd4a0 0x7fc7100c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.063+0000 7fc717a00700 1 --2- 172.21.15.89:0/3397658045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7100cbb30 0x7fc7100b29b0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.063+0000 7fc717a00700 1 --2- 172.21.15.89:0/3397658045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7100cd020 0x7fc7100ad410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.063+0000 7fc717a00700 1 -- 172.21.15.89:0/3397658045 >> 172.21.15.89:0/3397658045 conn(0x7fc7100b0d70 msgr2=0x7fc7101584b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.063+0000 7fc717a00700 1 -- 172.21.15.89:0/3397658045 shutdown_connections 2024-04-02T07:51:53.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.063+0000 7fc717a00700 1 -- 172.21.15.89:0/3397658045 wait complete. 2024-04-02T07:51:53.084 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:51:53.175 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:53.173+0000 7f99d5098240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:51:53.256 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:53.252+0000 7fe5e17b3240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:51:53.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.278+0000 7f52cac00700 1 Processor -- start 2024-04-02T07:51:53.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.278+0000 7f52cac00700 1 -- start start 2024-04-02T07:51:53.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.279+0000 7f52cac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52c40cb3f0 0x7f52c40ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:53.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.279+0000 7f52cac00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f52c40ac600 0x7f52c406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:53.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.279+0000 7f52cac00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c415fd50 con 0x7f52c415b5d0 2024-04-02T07:51:53.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.279+0000 7f52cac00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c415b9b0 con 0x7f52c40ac600 2024-04-02T07:51:53.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.279+0000 7f52cac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40aca70 con 0x7f52c40cb3f0 2024-04-02T07:51:53.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.280+0000 7f52c3e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f52c415b5d0 0x7f52c40ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:44658/0 (socket says 172.21.15.89:44658) 2024-04-02T07:51:53.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.280+0000 7f52c3e00700 1 -- 172.21.15.89:0/121060093 learned_addr learned my addr 172.21.15.89:0/121060093 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:53.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.280+0000 7f52c3400700 1 --2- 172.21.15.89:0/121060093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52c40cb3f0 0x7f52c40ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:53.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.280+0000 7f52c8c00700 1 --2- 172.21.15.89:0/121060093 >> v2:172.21.15.113:3300/0 conn(0x7f52c40ac600 0x7f52c406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:53.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.280+0000 7f52c2a00700 1 -- 172.21.15.89:0/121060093 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571379076 0 0) 0x7f52c415fd50 con 0x7f52c415b5d0 2024-04-02T07:51:53.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.280+0000 7f52c2a00700 1 -- 172.21.15.89:0/121060093 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a80035b0 con 0x7f52c415b5d0 2024-04-02T07:51:53.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.280+0000 7f52c3400700 1 -- 172.21.15.89:0/121060093 >> v2:172.21.15.113:3300/0 conn(0x7f52c40ac600 msgr2=0x7f52c406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:53.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.280+0000 7f52c3400700 1 --2- 172.21.15.89:0/121060093 >> v2:172.21.15.113:3300/0 conn(0x7f52c40ac600 0x7f52c406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.280+0000 7f52c3400700 1 -- 172.21.15.89:0/121060093 >> v1:172.21.15.113:6789/0 conn(0x7f52c415b5d0 legacy=0x7f52c40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:53.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.280+0000 7f52c3400700 1 -- 172.21.15.89:0/121060093 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c415f140 con 0x7f52c40cb3f0 2024-04-02T07:51:53.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.280+0000 7f52c8c00700 1 --2- 172.21.15.89:0/121060093 >> v2:172.21.15.113:3300/0 conn(0x7f52c40ac600 0x7f52c406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:53.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.281+0000 7f52c3400700 1 --2- 172.21.15.89:0/121060093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52c40cb3f0 0x7f52c40ac0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f52b0009840 tx=0x7f52b0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77fc444545dd0fe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:53.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.281+0000 7f52c2a00700 1 -- 172.21.15.89:0/121060093 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0011830 con 0x7f52c40cb3f0 2024-04-02T07:51:53.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.281+0000 7f52c2a00700 1 -- 172.21.15.89:0/121060093 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b0011990 con 0x7f52c40cb3f0 2024-04-02T07:51:53.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.281+0000 7f52c2a00700 1 -- 172.21.15.89:0/121060093 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0011d00 con 0x7f52c40cb3f0 2024-04-02T07:51:53.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.281+0000 7f52cac00700 1 -- 172.21.15.89:0/121060093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52c40cb3f0 msgr2=0x7f52c40ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:53.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.281+0000 7f52cac00700 1 --2- 172.21.15.89:0/121060093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52c40cb3f0 0x7f52c40ac0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f52b0009840 tx=0x7f52b0009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:53.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.282+0000 7f52cac00700 1 -- 172.21.15.89:0/121060093 shutdown_connections 2024-04-02T07:51:53.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.282+0000 7f52cac00700 1 --2- 172.21.15.89:0/121060093 >> v2:172.21.15.113:3300/0 conn(0x7f52c40ac600 0x7f52c406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.282+0000 7f52cac00700 1 --2- 172.21.15.89:0/121060093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52c40cb3f0 0x7f52c40ac0c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.282+0000 7f52cac00700 1 -- 172.21.15.89:0/121060093 >> 172.21.15.89:0/121060093 conn(0x7f52c40cf190 msgr2=0x7f52c40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:53.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.282+0000 7f52cac00700 1 -- 172.21.15.89:0/121060093 shutdown_connections 2024-04-02T07:51:53.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.282+0000 7f52cac00700 1 -- 172.21.15.89:0/121060093 wait complete. 2024-04-02T07:51:53.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.283+0000 7f52cac00700 1 Processor -- start 2024-04-02T07:51:53.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.284+0000 7f52cac00700 1 -- start start 2024-04-02T07:51:53.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.284+0000 7f52cac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52c40ac600 0x7f52c41530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:53.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.284+0000 7f52cac00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52c415b5d0 0x7f52c41553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:53.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.284+0000 7f52cac00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40b0690 0x7f52c41558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:53.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.285+0000 7f52cac00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40cab60 con 0x7f52c40b0690 2024-04-02T07:51:53.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.285+0000 7f52cac00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c4155e20 con 0x7f52c415b5d0 2024-04-02T07:51:53.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.285+0000 7f52cac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c41f7030 con 0x7f52c40ac600 2024-04-02T07:51:53.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.285+0000 7f52c8c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40b0690 0x7f52c41558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:53.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.285+0000 7f52c3400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52c415b5d0 0x7f52c41553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:53.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.285+0000 7f52c3e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52c40ac600 0x7f52c41530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:53.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.285+0000 7f52c3400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52c415b5d0 0x7f52c41553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:49350/0 (socket says 172.21.15.89:49350) 2024-04-02T07:51:53.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.285+0000 7f52c3400700 1 -- 172.21.15.89:0/1762268827 learned_addr learned my addr 172.21.15.89:0/1762268827 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:53.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.285+0000 7f52c8c00700 1 -- 172.21.15.89:0/1762268827 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52c40ac600 msgr2=0x7f52c41530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:53.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.285+0000 7f52c8c00700 1 --2- 172.21.15.89:0/1762268827 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52c40ac600 0x7f52c41530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.285+0000 7f52c8c00700 1 -- 172.21.15.89:0/1762268827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52c415b5d0 msgr2=0x7f52c41553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:53.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.285+0000 7f52c8c00700 1 --2- 172.21.15.89:0/1762268827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52c415b5d0 0x7f52c41553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.285+0000 7f52c8c00700 1 -- 172.21.15.89:0/1762268827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc009a70 con 0x7f52c40b0690 2024-04-02T07:51:53.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.285+0000 7f52c3400700 1 --2- 172.21.15.89:0/1762268827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52c415b5d0 0x7f52c41553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:53.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.286+0000 7f52c8c00700 1 --2- 172.21.15.89:0/1762268827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40b0690 0x7f52c41558e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009db0 tx=0x7f52bc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:53.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.286+0000 7f52bbe00700 1 -- 172.21.15.89:0/1762268827 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc009c20 con 0x7f52c40b0690 2024-04-02T07:51:53.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.286+0000 7f52bbe00700 1 -- 172.21.15.89:0/1762268827 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52bc00f040 con 0x7f52c40b0690 2024-04-02T07:51:53.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.286+0000 7f52cac00700 1 -- 172.21.15.89:0/1762268827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b00094a0 con 0x7f52c40b0690 2024-04-02T07:51:53.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.286+0000 7f52bbe00700 1 -- 172.21.15.89:0/1762268827 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc00ccb0 con 0x7f52c40b0690 2024-04-02T07:51:53.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.286+0000 7f52cac00700 1 -- 172.21.15.89:0/1762268827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c40cf810 con 0x7f52c40b0690 2024-04-02T07:51:53.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.287+0000 7f52cac00700 1 -- 172.21.15.89:0/1762268827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c4067720 con 0x7f52c40b0690 2024-04-02T07:51:53.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.288+0000 7f52bbe00700 1 -- 172.21.15.89:0/1762268827 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f52bc014480 con 0x7f52c40b0690 2024-04-02T07:51:53.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.289+0000 7f52bbe00700 1 --2- 172.21.15.89:0/1762268827 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f52a406adb0 0x7f52a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:53.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.289+0000 7f52bbe00700 1 -- 172.21.15.89:0/1762268827 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f52bc089e90 con 0x7f52c40b0690 2024-04-02T07:51:53.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.289+0000 7f52c3e00700 1 -- 172.21.15.89:0/1762268827 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f52a406adb0 msgr2=0x7f52a406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:53.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.289+0000 7f52c3e00700 1 --2- 172.21.15.89:0/1762268827 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f52a406adb0 0x7f52a406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.298+0000 7f52bbe00700 1 -- 172.21.15.89:0/1762268827 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f52bc054a70 con 0x7f52c40b0690 2024-04-02T07:51:53.476 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:53.474+0000 7fe5e17b3240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:51:53.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.490+0000 7f52c3e00700 1 -- 172.21.15.89:0/1762268827 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f52a406adb0 msgr2=0x7f52a406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:53.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.490+0000 7f52c3e00700 1 --2- 172.21.15.89:0/1762268827 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f52a406adb0 0x7f52a406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:53.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.552+0000 7f52cac00700 1 -- 172.21.15.89:0/1762268827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52c40b41c0 con 0x7f52c40b0690 2024-04-02T07:51:53.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.553+0000 7f52bbe00700 1 -- 172.21.15.89:0/1762268827 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f52bc058090 con 0x7f52c40b0690 2024-04-02T07:51:53.556 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:53.556 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2025634273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2025634273},{"type":"v1","addr":"172.21.15.113:6834","nonce":2025634273}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2247445394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2247445394},{"type":"v1","addr":"172.21.15.113:6835","nonce":2247445394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1785147820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1785147820},{"type":"v1","addr":"172.21.15.89:6835","nonce":1785147820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:51:29.684122+0000","modified":"2024-04-02T07:51:32.705923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6432":{"gid":6432,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/1479795102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1479795102},{"type":"v1","addr":"172.21.15.89:6837","nonce":1479795102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-02T07:51:53.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.556+0000 7f52cac00700 1 -- 172.21.15.89:0/1762268827 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f52a406adb0 msgr2=0x7f52a406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:53.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.556+0000 7f52cac00700 1 --2- 172.21.15.89:0/1762268827 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f52a406adb0 0x7f52a406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.556+0000 7f52cac00700 1 -- 172.21.15.89:0/1762268827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40b0690 msgr2=0x7f52c41558e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.556+0000 7f52cac00700 1 --2- 172.21.15.89:0/1762268827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40b0690 0x7f52c41558e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009db0 tx=0x7f52bc00c3a0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.556+0000 7f52cac00700 1 -- 172.21.15.89:0/1762268827 shutdown_connections 2024-04-02T07:51:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.556+0000 7f52cac00700 1 --2- 172.21.15.89:0/1762268827 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f52a406adb0 0x7f52a406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.556+0000 7f52cac00700 1 --2- 172.21.15.89:0/1762268827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c40b0690 0x7f52c41558e0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.556+0000 7f52cac00700 1 --2- 172.21.15.89:0/1762268827 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f52c40ac600 0x7f52c41530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.556+0000 7f52cac00700 1 --2- 172.21.15.89:0/1762268827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52c415b5d0 0x7f52c41553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.556+0000 7f52cac00700 1 -- 172.21.15.89:0/1762268827 >> 172.21.15.89:0/1762268827 conn(0x7f52c40cf190 msgr2=0x7f52c40afad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.556+0000 7f52cac00700 1 -- 172.21.15.89:0/1762268827 shutdown_connections 2024-04-02T07:51:53.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.556+0000 7f52cac00700 1 -- 172.21.15.89:0/1762268827 wait complete. 2024-04-02T07:51:53.560 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 33 2024-04-02T07:51:53.576 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:51:53.648 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:53.646+0000 7f99d5098240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:51:53.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.771+0000 7f1f52800700 1 Processor -- start 2024-04-02T07:51:53.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.771+0000 7f1f52800700 1 -- start start 2024-04-02T07:51:53.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.772+0000 7f1f52800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1f4c09aba0 0x7f1f4c06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:53.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.772+0000 7f1f52800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1f4c06c480 0x7f1f4c0ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:53.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.772+0000 7f1f52800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c06c8d0 con 0x7f1f4c15d710 2024-04-02T07:51:53.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.772+0000 7f1f52800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4c06bb40 con 0x7f1f4c09aba0 2024-04-02T07:51:53.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.772+0000 7f1f52800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4c099b00 con 0x7f1f4c06c480 2024-04-02T07:51:53.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.773+0000 7f1f4aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1f4c09aba0 0x7f1f4c06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:53.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.773+0000 7f1f4b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1f4c15d710 0x7f1f4c0d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46000/0 (socket says 172.21.15.89:46000) 2024-04-02T07:51:53.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.773+0000 7f1f4b400700 1 -- 172.21.15.89:0/1424634418 learned_addr learned my addr 172.21.15.89:0/1424634418 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:53.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.773+0000 7f1f4be00700 1 --2- 172.21.15.89:0/1424634418 >> v2:172.21.15.113:3300/0 conn(0x7f1f4c06c480 0x7f1f4c0ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:53.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.773+0000 7f1f4aa00700 1 -- 172.21.15.89:0/1424634418 >> v2:172.21.15.113:3300/0 conn(0x7f1f4c06c480 msgr2=0x7f1f4c0ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:53.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.773+0000 7f1f4aa00700 1 --2- 172.21.15.89:0/1424634418 >> v2:172.21.15.113:3300/0 conn(0x7f1f4c06c480 0x7f1f4c0ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.773+0000 7f1f4aa00700 1 -- 172.21.15.89:0/1424634418 >> v1:172.21.15.89:6789/0 conn(0x7f1f4c15d710 legacy=0x7f1f4c0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:53.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.773+0000 7f1f4aa00700 1 -- 172.21.15.89:0/1424634418 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c099de0 con 0x7f1f4c09aba0 2024-04-02T07:51:53.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.773+0000 7f1f4be00700 1 --2- 172.21.15.89:0/1424634418 >> v2:172.21.15.113:3300/0 conn(0x7f1f4c06c480 0x7f1f4c0ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:53.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.773+0000 7f1f4a000700 1 -- 172.21.15.89:0/1424634418 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126004011 0 0) 0x7f1f4c06c8d0 con 0x7f1f4c15d710 2024-04-02T07:51:53.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.774+0000 7f1f4aa00700 1 --2- 172.21.15.89:0/1424634418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f4c09aba0 0x7f1f4c06bf40 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c0722c0 tx=0x7f1f38009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118a6286626ba206 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:53.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.774+0000 7f1f4a000700 1 -- 172.21.15.89:0/1424634418 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f38011830 con 0x7f1f4c09aba0 2024-04-02T07:51:53.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.774+0000 7f1f4a000700 1 -- 172.21.15.89:0/1424634418 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f38011990 con 0x7f1f4c09aba0 2024-04-02T07:51:53.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.774+0000 7f1f4a000700 1 -- 172.21.15.89:0/1424634418 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f38011d20 con 0x7f1f4c09aba0 2024-04-02T07:51:53.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.774+0000 7f1f52800700 1 -- 172.21.15.89:0/1424634418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f4c09aba0 msgr2=0x7f1f4c06bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:53.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.774+0000 7f1f52800700 1 --2- 172.21.15.89:0/1424634418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f4c09aba0 0x7f1f4c06bf40 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c0722c0 tx=0x7f1f38009b20 comp rx=0 tx=0).stop 2024-04-02T07:51:53.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.774+0000 7f1f52800700 1 -- 172.21.15.89:0/1424634418 shutdown_connections 2024-04-02T07:51:53.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.774+0000 7f1f52800700 1 --2- 172.21.15.89:0/1424634418 >> v2:172.21.15.113:3300/0 conn(0x7f1f4c06c480 0x7f1f4c0ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.774+0000 7f1f52800700 1 --2- 172.21.15.89:0/1424634418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f4c09aba0 0x7f1f4c06bf40 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.774+0000 7f1f52800700 1 -- 172.21.15.89:0/1424634418 >> 172.21.15.89:0/1424634418 conn(0x7f1f4c0d13e0 msgr2=0x7f1f4c0d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:53.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.775+0000 7f1f52800700 1 -- 172.21.15.89:0/1424634418 shutdown_connections 2024-04-02T07:51:53.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.775+0000 7f1f52800700 1 -- 172.21.15.89:0/1424634418 wait complete. 2024-04-02T07:51:53.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.776+0000 7f1f52800700 1 Processor -- start 2024-04-02T07:51:53.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.776+0000 7f1f52800700 1 -- start start 2024-04-02T07:51:53.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.777+0000 7f1f52800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f4c06c480 0x7f1f4c1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:53.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.777+0000 7f1f52800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f4c09aba0 0x7f1f4c06f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:53.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.777+0000 7f1f52800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f4c15d710 0x7f1f4c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:53.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.777+0000 7f1f52800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c1f2e80 con 0x7f1f4c06c480 2024-04-02T07:51:53.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.777+0000 7f1f52800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c15db90 con 0x7f1f4c15d710 2024-04-02T07:51:53.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.777+0000 7f1f52800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c15de70 con 0x7f1f4c09aba0 2024-04-02T07:51:53.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.777+0000 7f1f4b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f4c06c480 0x7f1f4c1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:53.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.778+0000 7f1f4b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f4c06c480 0x7f1f4c1551b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43198/0 (socket says 172.21.15.89:43198) 2024-04-02T07:51:53.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.778+0000 7f1f4b400700 1 -- 172.21.15.89:0/1861804815 learned_addr learned my addr 172.21.15.89:0/1861804815 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:53.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.778+0000 7f1f4aa00700 1 --2- 172.21.15.89:0/1861804815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f4c09aba0 0x7f1f4c06f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:53.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.778+0000 7f1f4be00700 1 --2- 172.21.15.89:0/1861804815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f4c15d710 0x7f1f4c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:53.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.778+0000 7f1f4b400700 1 -- 172.21.15.89:0/1861804815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f4c09aba0 msgr2=0x7f1f4c06f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:53.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.778+0000 7f1f4b400700 1 --2- 172.21.15.89:0/1861804815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f4c09aba0 0x7f1f4c06f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.778+0000 7f1f4b400700 1 -- 172.21.15.89:0/1861804815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f4c15d710 msgr2=0x7f1f4c06fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:53.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.778+0000 7f1f4b400700 1 --2- 172.21.15.89:0/1861804815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f4c15d710 0x7f1f4c06fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:53.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.778+0000 7f1f4b400700 1 -- 172.21.15.89:0/1861804815 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f380094a0 con 0x7f1f4c06c480 2024-04-02T07:51:53.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.778+0000 7f1f4b400700 1 --2- 172.21.15.89:0/1861804815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f4c06c480 0x7f1f4c1551b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c550 tx=0x7f1f3c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:53.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.778+0000 7f1f43400700 1 -- 172.21.15.89:0/1861804815 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c004c80 con 0x7f1f4c06c480 2024-04-02T07:51:53.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.779+0000 7f1f43400700 1 -- 172.21.15.89:0/1861804815 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3c004e00 con 0x7f1f4c06c480 2024-04-02T07:51:53.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.779+0000 7f1f52800700 1 -- 172.21.15.89:0/1861804815 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4c15e150 con 0x7f1f4c06c480 2024-04-02T07:51:53.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.779+0000 7f1f43400700 1 -- 172.21.15.89:0/1861804815 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c007310 con 0x7f1f4c06c480 2024-04-02T07:51:53.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.779+0000 7f1f52800700 1 -- 172.21.15.89:0/1861804815 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4c155b50 con 0x7f1f4c06c480 2024-04-02T07:51:53.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.780+0000 7f1f52800700 1 -- 172.21.15.89:0/1861804815 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f4c067720 con 0x7f1f4c06c480 2024-04-02T07:51:53.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.781+0000 7f1f43400700 1 -- 172.21.15.89:0/1861804815 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f1f3c007470 con 0x7f1f4c06c480 2024-04-02T07:51:53.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.782+0000 7f1f43400700 1 --2- 172.21.15.89:0/1861804815 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f2c06adb0 0x7f1f2c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:53.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.782+0000 7f1f4aa00700 1 -- 172.21.15.89:0/1861804815 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f2c06adb0 msgr2=0x7f1f2c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:53.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.782+0000 7f1f43400700 1 -- 172.21.15.89:0/1861804815 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f1f3c0892f0 con 0x7f1f4c06c480 2024-04-02T07:51:53.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.782+0000 7f1f4aa00700 1 --2- 172.21.15.89:0/1861804815 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f2c06adb0 0x7f1f2c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:53.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.787+0000 7f1f43400700 1 -- 172.21.15.89:0/1861804815 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1f3c054100 con 0x7f1f4c06c480 2024-04-02T07:51:53.858 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:53.855+0000 7f99d5098240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:51:53.979 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:53.977+0000 7fe5e17b3240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:51:53.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.983+0000 7f1f4aa00700 1 -- 172.21.15.89:0/1861804815 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f2c06adb0 msgr2=0x7f1f2c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:53.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:53.983+0000 7f1f4aa00700 1 --2- 172.21.15.89:0/1861804815 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f2c06adb0 0x7f1f2c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:54.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.000+0000 7f1f52800700 1 -- 172.21.15.89:0/1861804815 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f4c0d0f20 con 0x7f1f4c06c480 2024-04-02T07:51:54.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.001+0000 7f1f43400700 1 -- 172.21.15.89:0/1861804815 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+17111 (secure 0 0 0) 0x7f1f3c057720 con 0x7f1f4c06c480 2024-04-02T07:51:54.004 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:54.004 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":84,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:51:45.730961+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:51:27.926367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-02T07:51:28.662600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/822661295":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/907732608":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/3083880072":"2024-04-03T07:51:45.730929+0000","172.21.15.89:6833/43357":"2024-04-03T07:51:45.730929+0000","172.21.15.89:6832/43357":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/1211892288":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/23281819":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/3435265311":"2024-04-03T07:51:45.730929+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:51:54.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.004+0000 7f1f52800700 1 -- 172.21.15.89:0/1861804815 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f2c06adb0 msgr2=0x7f1f2c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.004+0000 7f1f52800700 1 --2- 172.21.15.89:0/1861804815 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f2c06adb0 0x7f1f2c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.004+0000 7f1f52800700 1 -- 172.21.15.89:0/1861804815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f4c06c480 msgr2=0x7f1f4c1551b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.004+0000 7f1f52800700 1 --2- 172.21.15.89:0/1861804815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f4c06c480 0x7f1f4c1551b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c550 tx=0x7f1f3c00c860 comp rx=0 tx=0).stop 2024-04-02T07:51:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.004+0000 7f1f52800700 1 -- 172.21.15.89:0/1861804815 shutdown_connections 2024-04-02T07:51:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.004+0000 7f1f52800700 1 --2- 172.21.15.89:0/1861804815 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1f2c06adb0 0x7f1f2c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.004+0000 7f1f52800700 1 --2- 172.21.15.89:0/1861804815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f4c15d710 0x7f1f4c06fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.004+0000 7f1f52800700 1 --2- 172.21.15.89:0/1861804815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f4c06c480 0x7f1f4c1551b0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.004+0000 7f1f52800700 1 --2- 172.21.15.89:0/1861804815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f4c09aba0 0x7f1f4c06f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.004+0000 7f1f52800700 1 -- 172.21.15.89:0/1861804815 >> 172.21.15.89:0/1861804815 conn(0x7f1f4c0d13e0 msgr2=0x7f1f4c0af460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.004+0000 7f1f52800700 1 -- 172.21.15.89:0/1861804815 shutdown_connections 2024-04-02T07:51:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.004+0000 7f1f52800700 1 -- 172.21.15.89:0/1861804815 wait complete. 2024-04-02T07:51:54.026 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T07:51:54.027 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:51:54.201 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:54.199+0000 7fe5e17b3240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:51:54.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.225+0000 7fec21e00700 1 Processor -- start 2024-04-02T07:51:54.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.226+0000 7fec21e00700 1 -- start start 2024-04-02T07:51:54.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.227+0000 7fec21e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fec1c09aba0 0x7fec1c06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:54.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.227+0000 7fec21e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec1c06c480 0x7fec1c0ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:54.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.227+0000 7fec21e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec1c06bb40 con 0x7fec1c15d710 2024-04-02T07:51:54.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.227+0000 7fec21e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec1c06c910 con 0x7fec1c09aba0 2024-04-02T07:51:54.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.227+0000 7fec21e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c099b00 con 0x7fec1c06c480 2024-04-02T07:51:54.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.227+0000 7fec1b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec1c06c480 0x7fec1c0ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:54.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.227+0000 7fec1a000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fec1c09aba0 0x7fec1c06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:54.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.227+0000 7fec1aa00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fec1c15d710 0x7fec1c0d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:44674/0 (socket says 172.21.15.89:44674) 2024-04-02T07:51:54.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.227+0000 7fec1aa00700 1 -- 172.21.15.89:0/1765760024 learned_addr learned my addr 172.21.15.89:0/1765760024 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:54.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.228+0000 7fec1b400700 1 -- 172.21.15.89:0/1765760024 >> v2:172.21.15.113:3300/0 conn(0x7fec1c09aba0 msgr2=0x7fec1c06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:54.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.228+0000 7fec1b400700 1 --2- 172.21.15.89:0/1765760024 >> v2:172.21.15.113:3300/0 conn(0x7fec1c09aba0 0x7fec1c06bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.228+0000 7fec1b400700 1 -- 172.21.15.89:0/1765760024 >> v1:172.21.15.113:6789/0 conn(0x7fec1c15d710 legacy=0x7fec1c0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:54.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.228+0000 7fec1b400700 1 -- 172.21.15.89:0/1765760024 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c099de0 con 0x7fec1c06c480 2024-04-02T07:51:54.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.228+0000 7fec19600700 1 -- 172.21.15.89:0/1765760024 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360054995 0 0) 0x7fec1c06bb40 con 0x7fec1c15d710 2024-04-02T07:51:54.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.228+0000 7fec1b400700 1 --2- 172.21.15.89:0/1765760024 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec1c06c480 0x7fec1c0ccbe0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fec14009d90 tx=0x7fec1400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=371028c7af8add98 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:54.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.228+0000 7fec19600700 1 -- 172.21.15.89:0/1765760024 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1400e070 con 0x7fec1c06c480 2024-04-02T07:51:54.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.228+0000 7fec19600700 1 -- 172.21.15.89:0/1765760024 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1400bb20 con 0x7fec1c06c480 2024-04-02T07:51:54.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.228+0000 7fec19600700 1 -- 172.21.15.89:0/1765760024 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14012430 con 0x7fec1c06c480 2024-04-02T07:51:54.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.229+0000 7fec21e00700 1 -- 172.21.15.89:0/1765760024 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec1c06c480 msgr2=0x7fec1c0ccbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:54.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.229+0000 7fec21e00700 1 --2- 172.21.15.89:0/1765760024 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec1c06c480 0x7fec1c0ccbe0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fec14009d90 tx=0x7fec1400b330 comp rx=0 tx=0).stop 2024-04-02T07:51:54.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.229+0000 7fec21e00700 1 -- 172.21.15.89:0/1765760024 shutdown_connections 2024-04-02T07:51:54.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.229+0000 7fec21e00700 1 --2- 172.21.15.89:0/1765760024 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec1c06c480 0x7fec1c0ccbe0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.229+0000 7fec21e00700 1 --2- 172.21.15.89:0/1765760024 >> v2:172.21.15.113:3300/0 conn(0x7fec1c09aba0 0x7fec1c06bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.229+0000 7fec21e00700 1 -- 172.21.15.89:0/1765760024 >> 172.21.15.89:0/1765760024 conn(0x7fec1c0d13e0 msgr2=0x7fec1c0d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:54.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.229+0000 7fec21e00700 1 -- 172.21.15.89:0/1765760024 shutdown_connections 2024-04-02T07:51:54.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.229+0000 7fec21e00700 1 -- 172.21.15.89:0/1765760024 wait complete. 2024-04-02T07:51:54.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.231+0000 7fec21e00700 1 Processor -- start 2024-04-02T07:51:54.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.231+0000 7fec21e00700 1 -- start start 2024-04-02T07:51:54.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.232+0000 7fec21e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec1c06c480 0x7fec1c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:54.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.232+0000 7fec21e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec1c09aba0 0x7fec1c14b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:54.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.232+0000 7fec21e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec1c15d710 0x7fec1c1484a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:54.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.232+0000 7fec21e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c0c78e0 con 0x7fec1c06c480 2024-04-02T07:51:54.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.232+0000 7fec21e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c14bdd0 con 0x7fec1c15d710 2024-04-02T07:51:54.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.232+0000 7fec21e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c0c6300 con 0x7fec1c09aba0 2024-04-02T07:51:54.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.232+0000 7fec1aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec1c06c480 0x7fec1c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:54.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.232+0000 7fec1aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec1c06c480 0x7fec1c0c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43210/0 (socket says 172.21.15.89:43210) 2024-04-02T07:51:54.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.232+0000 7fec1a000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec1c09aba0 0x7fec1c14b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:54.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.232+0000 7fec1aa00700 1 -- 172.21.15.89:0/2738374725 learned_addr learned my addr 172.21.15.89:0/2738374725 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:54.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.232+0000 7fec1b400700 1 --2- 172.21.15.89:0/2738374725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec1c15d710 0x7fec1c1484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:54.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.233+0000 7fec1aa00700 1 -- 172.21.15.89:0/2738374725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec1c09aba0 msgr2=0x7fec1c14b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:54.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.233+0000 7fec1aa00700 1 --2- 172.21.15.89:0/2738374725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec1c09aba0 0x7fec1c14b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.233+0000 7fec1aa00700 1 -- 172.21.15.89:0/2738374725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec1c15d710 msgr2=0x7fec1c1484a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:54.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.233+0000 7fec1aa00700 1 --2- 172.21.15.89:0/2738374725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec1c15d710 0x7fec1c1484a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.233+0000 7fec1aa00700 1 -- 172.21.15.89:0/2738374725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec14009a70 con 0x7fec1c06c480 2024-04-02T07:51:54.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.233+0000 7fec1aa00700 1 --2- 172.21.15.89:0/2738374725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec1c06c480 0x7fec1c0c73a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00c550 tx=0x7fec0c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:54.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.233+0000 7fec13400700 1 -- 172.21.15.89:0/2738374725 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c004c80 con 0x7fec1c06c480 2024-04-02T07:51:54.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.233+0000 7fec21e00700 1 -- 172.21.15.89:0/2738374725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1c0c65e0 con 0x7fec1c06c480 2024-04-02T07:51:54.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.233+0000 7fec13400700 1 -- 172.21.15.89:0/2738374725 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c004e00 con 0x7fec1c06c480 2024-04-02T07:51:54.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.233+0000 7fec13400700 1 -- 172.21.15.89:0/2738374725 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c007310 con 0x7fec1c06c480 2024-04-02T07:51:54.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.233+0000 7fec21e00700 1 -- 172.21.15.89:0/2738374725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1c0c6ba0 con 0x7fec1c06c480 2024-04-02T07:51:54.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.235+0000 7fec21e00700 1 -- 172.21.15.89:0/2738374725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1c067720 con 0x7fec1c06c480 2024-04-02T07:51:54.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.236+0000 7fec13400700 1 -- 172.21.15.89:0/2738374725 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fec0c007470 con 0x7fec1c06c480 2024-04-02T07:51:54.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.237+0000 7fec13400700 1 --2- 172.21.15.89:0/2738374725 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febfc06adb0 0x7febfc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:54.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.237+0000 7fec13400700 1 -- 172.21.15.89:0/2738374725 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fec0c0893f0 con 0x7fec1c06c480 2024-04-02T07:51:54.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.237+0000 7fec1a000700 1 -- 172.21.15.89:0/2738374725 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febfc06adb0 msgr2=0x7febfc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:54.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.237+0000 7fec1a000700 1 --2- 172.21.15.89:0/2738374725 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febfc06adb0 0x7febfc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:54.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.242+0000 7fec13400700 1 -- 172.21.15.89:0/2738374725 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fec0c0540e0 con 0x7fec1c06c480 2024-04-02T07:51:54.295 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:54.293+0000 7f99d5098240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:51:54.405 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:51:54.403+0000 7f99d5098240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:51:54.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.438+0000 7fec1a000700 1 -- 172.21.15.89:0/2738374725 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febfc06adb0 msgr2=0x7febfc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:54.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.438+0000 7fec1a000700 1 --2- 172.21.15.89:0/2738374725 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febfc06adb0 0x7febfc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:51:54.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.498+0000 7fec21e00700 1 -- 172.21.15.89:0/2738374725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec1c0d0f20 con 0x7fec1c06c480 2024-04-02T07:51:54.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.498+0000 7fec13400700 1 -- 172.21.15.89:0/2738374725 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fec0c057700 con 0x7fec1c06c480 2024-04-02T07:51:54.503 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:54.503 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2025634273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2025634273},{"type":"v1","addr":"172.21.15.113:6834","nonce":2025634273}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2247445394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2247445394},{"type":"v1","addr":"172.21.15.113:6835","nonce":2247445394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1785147820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1785147820},{"type":"v1","addr":"172.21.15.89:6835","nonce":1785147820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:51:29.684122+0000","modified":"2024-04-02T07:51:32.705923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6432":{"gid":6432,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/1479795102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1479795102},{"type":"v1","addr":"172.21.15.89:6837","nonce":1479795102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-02T07:51:54.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.501+0000 7fec21e00700 1 -- 172.21.15.89:0/2738374725 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febfc06adb0 msgr2=0x7febfc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:54.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.501+0000 7fec21e00700 1 --2- 172.21.15.89:0/2738374725 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febfc06adb0 0x7febfc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.501+0000 7fec21e00700 1 -- 172.21.15.89:0/2738374725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec1c06c480 msgr2=0x7fec1c0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:54.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.501+0000 7fec21e00700 1 --2- 172.21.15.89:0/2738374725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec1c06c480 0x7fec1c0c73a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00c550 tx=0x7fec0c00c860 comp rx=0 tx=0).stop 2024-04-02T07:51:54.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.501+0000 7fec21e00700 1 -- 172.21.15.89:0/2738374725 shutdown_connections 2024-04-02T07:51:54.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.501+0000 7fec21e00700 1 --2- 172.21.15.89:0/2738374725 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febfc06adb0 0x7febfc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.501+0000 7fec21e00700 1 --2- 172.21.15.89:0/2738374725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec1c15d710 0x7fec1c1484a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.501+0000 7fec21e00700 1 --2- 172.21.15.89:0/2738374725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec1c06c480 0x7fec1c0c73a0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.501+0000 7fec21e00700 1 --2- 172.21.15.89:0/2738374725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec1c09aba0 0x7fec1c14b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.501+0000 7fec21e00700 1 -- 172.21.15.89:0/2738374725 >> 172.21.15.89:0/2738374725 conn(0x7fec1c0d13e0 msgr2=0x7fec1c0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:54.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.501+0000 7fec21e00700 1 -- 172.21.15.89:0/2738374725 shutdown_connections 2024-04-02T07:51:54.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.501+0000 7fec21e00700 1 -- 172.21.15.89:0/2738374725 wait complete. 2024-04-02T07:51:54.505 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 33 2024-04-02T07:51:54.522 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:51:54.665 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:54.663+0000 7fe5e17b3240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:51:54.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.716+0000 7fd360000700 1 Processor -- start 2024-04-02T07:51:54.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.716+0000 7fd360000700 1 -- start start 2024-04-02T07:51:54.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.717+0000 7fd360000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd35815d880 0x7fd3580d3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:54.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.717+0000 7fd360000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3580d1a90 0x7fd3580b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:54.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.717+0000 7fd360000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3580b97e0 con 0x7fd3580d05c0 2024-04-02T07:51:54.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.717+0000 7fd360000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd35815dc60 con 0x7fd35815d880 2024-04-02T07:51:54.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.718+0000 7fd360000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3580d1ef0 con 0x7fd3580d1a90 2024-04-02T07:51:54.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.718+0000 7fd35cc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3580d1a90 0x7fd3580b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:54.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.718+0000 7fd35d600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd35815d880 0x7fd3580d3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:54.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.718+0000 7fd35d600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd35815d880 0x7fd3580d3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:49396/0 (socket says 172.21.15.89:49396) 2024-04-02T07:51:54.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.718+0000 7fd35d600700 1 -- 172.21.15.89:0/1160321639 learned_addr learned my addr 172.21.15.89:0/1160321639 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:54.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.718+0000 7fd35d600700 1 -- 172.21.15.89:0/1160321639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3580d1a90 msgr2=0x7fd3580b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:54.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.718+0000 7fd35d600700 1 --2- 172.21.15.89:0/1160321639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3580d1a90 0x7fd3580b92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.718+0000 7fd35d600700 1 -- 172.21.15.89:0/1160321639 >> v1:172.21.15.113:6789/0 conn(0x7fd3580d05c0 legacy=0x7fd3580b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:54.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.719+0000 7fd35d600700 1 -- 172.21.15.89:0/1160321639 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3580d2190 con 0x7fd35815d880 2024-04-02T07:51:54.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.719+0000 7fd35cc00700 1 --2- 172.21.15.89:0/1160321639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3580d1a90 0x7fd3580b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:54.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.719+0000 7fd357e00700 1 -- 172.21.15.89:0/1160321639 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963944479 0 0) 0x7fd3580b97e0 con 0x7fd3580d05c0 2024-04-02T07:51:54.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.719+0000 7fd35d600700 1 --2- 172.21.15.89:0/1160321639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd35815d880 0x7fd3580d3230 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd34c0098f0 tx=0x7fd34c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c95ecf21b9704fc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:54.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.719+0000 7fd357e00700 1 -- 172.21.15.89:0/1160321639 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c011830 con 0x7fd35815d880 2024-04-02T07:51:54.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.719+0000 7fd357e00700 1 -- 172.21.15.89:0/1160321639 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34c011990 con 0x7fd35815d880 2024-04-02T07:51:54.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.719+0000 7fd357e00700 1 -- 172.21.15.89:0/1160321639 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c011d20 con 0x7fd35815d880 2024-04-02T07:51:54.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.720+0000 7fd360000700 1 -- 172.21.15.89:0/1160321639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd35815d880 msgr2=0x7fd3580d3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:54.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.720+0000 7fd360000700 1 --2- 172.21.15.89:0/1160321639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd35815d880 0x7fd3580d3230 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd34c0098f0 tx=0x7fd34c009c00 comp rx=0 tx=0).stop 2024-04-02T07:51:54.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.720+0000 7fd360000700 1 -- 172.21.15.89:0/1160321639 shutdown_connections 2024-04-02T07:51:54.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.720+0000 7fd360000700 1 --2- 172.21.15.89:0/1160321639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd35815d880 0x7fd3580d3230 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.720+0000 7fd360000700 1 --2- 172.21.15.89:0/1160321639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3580d1a90 0x7fd3580b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.720+0000 7fd360000700 1 -- 172.21.15.89:0/1160321639 >> 172.21.15.89:0/1160321639 conn(0x7fd3580666a0 msgr2=0x7fd358066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:54.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.720+0000 7fd360000700 1 -- 172.21.15.89:0/1160321639 shutdown_connections 2024-04-02T07:51:54.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.720+0000 7fd360000700 1 -- 172.21.15.89:0/1160321639 wait complete. 2024-04-02T07:51:54.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.722+0000 7fd360000700 1 Processor -- start 2024-04-02T07:51:54.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.722+0000 7fd360000700 1 -- start start 2024-04-02T07:51:54.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd360000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3580d05c0 0x7fd35815fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:54.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd360000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3580d1a90 0x7fd35814cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:54.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd360000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd35815d880 0x7fd35814d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:54.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd360000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3580b5930 con 0x7fd3580d05c0 2024-04-02T07:51:54.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd360000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3580b3c50 con 0x7fd35815d880 2024-04-02T07:51:54.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd360000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35815f3c0 con 0x7fd3580d1a90 2024-04-02T07:51:54.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd35d600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3580d05c0 0x7fd35815fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:54.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd35d600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3580d05c0 0x7fd35815fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43212/0 (socket says 172.21.15.89:43212) 2024-04-02T07:51:54.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd35d600700 1 -- 172.21.15.89:0/3171827602 learned_addr learned my addr 172.21.15.89:0/3171827602 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:54.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd35cc00700 1 --2- 172.21.15.89:0/3171827602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3580d1a90 0x7fd35814cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:54.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd35e000700 1 --2- 172.21.15.89:0/3171827602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd35815d880 0x7fd35814d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:54.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd35d600700 1 -- 172.21.15.89:0/3171827602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3580d1a90 msgr2=0x7fd35814cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:54.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd35d600700 1 --2- 172.21.15.89:0/3171827602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3580d1a90 0x7fd35814cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd35d600700 1 -- 172.21.15.89:0/3171827602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd35815d880 msgr2=0x7fd35814d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:54.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd35d600700 1 --2- 172.21.15.89:0/3171827602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd35815d880 0x7fd35814d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.723+0000 7fd35d600700 1 -- 172.21.15.89:0/3171827602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3440094a0 con 0x7fd3580d05c0 2024-04-02T07:51:54.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.724+0000 7fd35d600700 1 --2- 172.21.15.89:0/3171827602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3580d05c0 0x7fd35815fc50 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd34c014040 tx=0x7fd34c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:54.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.724+0000 7fd355600700 1 -- 172.21.15.89:0/3171827602 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c0036a0 con 0x7fd3580d05c0 2024-04-02T07:51:54.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.724+0000 7fd355600700 1 -- 172.21.15.89:0/3171827602 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34c01e070 con 0x7fd3580d05c0 2024-04-02T07:51:54.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.724+0000 7fd355600700 1 -- 172.21.15.89:0/3171827602 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c003a20 con 0x7fd3580d05c0 2024-04-02T07:51:54.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.724+0000 7fd360000700 1 -- 172.21.15.89:0/3171827602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34c009550 con 0x7fd3580d05c0 2024-04-02T07:51:54.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.724+0000 7fd360000700 1 -- 172.21.15.89:0/3171827602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3580b3600 con 0x7fd3580d05c0 2024-04-02T07:51:54.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.725+0000 7fd360000700 1 -- 172.21.15.89:0/3171827602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd358067720 con 0x7fd3580d05c0 2024-04-02T07:51:54.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.727+0000 7fd355600700 1 -- 172.21.15.89:0/3171827602 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 38) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fd34c003b80 con 0x7fd3580d05c0 2024-04-02T07:51:54.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.727+0000 7fd355600700 1 --2- 172.21.15.89:0/3171827602 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd33806adb0 0x7fd33806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:54.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.728+0000 7fd355600700 1 -- 172.21.15.89:0/3171827602 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd34c08a800 con 0x7fd3580d05c0 2024-04-02T07:51:54.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.728+0000 7fd35cc00700 1 -- 172.21.15.89:0/3171827602 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd33806adb0 msgr2=0x7fd33806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:51:54.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.728+0000 7fd35cc00700 1 --2- 172.21.15.89:0/3171827602 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd33806adb0 0x7fd33806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:51:54.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.732+0000 7fd355600700 1 -- 172.21.15.89:0/3171827602 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd34c0554f0 con 0x7fd3580d05c0 2024-04-02T07:51:54.784 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:54.782+0000 7fe5e17b3240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:51:54.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.792+0000 7fd355600700 1 -- 172.21.15.89:0/3171827602 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 39) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fd34c0558c0 con 0x7fd3580d05c0 2024-04-02T07:51:54.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.792+0000 7fd355600700 1 -- 172.21.15.89:0/3171827602 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd33806adb0 msgr2=0x7fd33806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:51:54.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:54.792+0000 7fd355600700 1 --2- 172.21.15.89:0/3171827602 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd33806adb0 0x7fd33806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:54.870 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:54.867+0000 7fe554a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:54.870 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:54.867+0000 7fe554a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:54.870 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:54.867+0000 7fe554a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:54.870 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:54.867+0000 7fe554a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:54.870 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:54.867+0000 7fe554a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:54.871 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:54.869+0000 7fe550e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:54.872 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:54.869+0000 7fe550e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:54.872 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:54.869+0000 7fe550e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:54.872 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:54.869+0000 7fe550e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:54.872 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:51:54.869+0000 7fe550e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:51:55.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.009+0000 7fd360000700 1 -- 172.21.15.89:0/3171827602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd358159420 con 0x7fd3580d05c0 2024-04-02T07:51:55.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.009+0000 7fd355600700 1 -- 172.21.15.89:0/3171827602 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fd34c05c510 con 0x7fd3580d05c0 2024-04-02T07:51:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:55.012 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2025634273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2025634273},{"type":"v1","addr":"172.21.15.113:6834","nonce":2025634273}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2247445394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2247445394},{"type":"v1","addr":"172.21.15.113:6835","nonce":2247445394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1785147820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1785147820},{"type":"v1","addr":"172.21.15.89:6835","nonce":1785147820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:51:29.684122+0000","modified":"2024-04-02T07:51:32.705923+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6432":{"gid":6432,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/1479795102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1479795102},{"type":"v1","addr":"172.21.15.89:6837","nonce":1479795102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-02T07:51:55.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.012+0000 7fd360000700 1 -- 172.21.15.89:0/3171827602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3580d05c0 msgr2=0x7fd35815fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:55.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.012+0000 7fd360000700 1 --2- 172.21.15.89:0/3171827602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3580d05c0 0x7fd35815fc50 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd34c014040 tx=0x7fd34c004750 comp rx=0 tx=0).stop 2024-04-02T07:51:55.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.012+0000 7fd360000700 1 -- 172.21.15.89:0/3171827602 shutdown_connections 2024-04-02T07:51:55.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.012+0000 7fd360000700 1 --2- 172.21.15.89:0/3171827602 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd33806adb0 0x7fd33806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.012+0000 7fd360000700 1 --2- 172.21.15.89:0/3171827602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd35815d880 0x7fd35814d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.012+0000 7fd360000700 1 --2- 172.21.15.89:0/3171827602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3580d05c0 0x7fd35815fc50 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.012+0000 7fd360000700 1 --2- 172.21.15.89:0/3171827602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd3580d1a90 0x7fd35814cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.012+0000 7fd360000700 1 -- 172.21.15.89:0/3171827602 >> 172.21.15.89:0/3171827602 conn(0x7fd3580666a0 msgr2=0x7fd3580b8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:55.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.012+0000 7fd360000700 1 -- 172.21.15.89:0/3171827602 shutdown_connections 2024-04-02T07:51:55.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.012+0000 7fd360000700 1 -- 172.21.15.89:0/3171827602 wait complete. 2024-04-02T07:51:55.015 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 33 2024-04-02T07:51:55.032 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:51:55.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.221+0000 7f2d59400700 1 Processor -- start 2024-04-02T07:51:55.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.222+0000 7f2d59400700 1 -- start start 2024-04-02T07:51:55.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.223+0000 7f2d59400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2d540d18b0 0x7f2d540cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:55.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.223+0000 7f2d59400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d54161590 0x7f2d54161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:55.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.223+0000 7f2d59400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d540ceda0 con 0x7f2d54160f10 2024-04-02T07:51:55.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.223+0000 7f2d59400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d540d1d30 con 0x7f2d540d18b0 2024-04-02T07:51:55.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.223+0000 7f2d59400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d54161f60 con 0x7f2d54161590 2024-04-02T07:51:55.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.223+0000 7f2d51e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2d540d18b0 0x7f2d540cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:55.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.223+0000 7f2d52800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2d54160f10 0x7f2d540d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46012/0 (socket says 172.21.15.89:46012) 2024-04-02T07:51:55.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.223+0000 7f2d52800700 1 -- 172.21.15.89:0/4240620667 learned_addr learned my addr 172.21.15.89:0/4240620667 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:55.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.223+0000 7f2d53200700 1 --2- 172.21.15.89:0/4240620667 >> v2:172.21.15.113:3300/0 conn(0x7f2d54161590 0x7f2d54161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:55.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.223+0000 7f2d51400700 1 -- 172.21.15.89:0/4240620667 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567247374 0 0) 0x7f2d540ceda0 con 0x7f2d54160f10 2024-04-02T07:51:55.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.223+0000 7f2d51400700 1 -- 172.21.15.89:0/4240620667 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d380035b0 con 0x7f2d54160f10 2024-04-02T07:51:55.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.223+0000 7f2d51e00700 1 -- 172.21.15.89:0/4240620667 >> v2:172.21.15.113:3300/0 conn(0x7f2d54161590 msgr2=0x7f2d54161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:55.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.224+0000 7f2d51e00700 1 --2- 172.21.15.89:0/4240620667 >> v2:172.21.15.113:3300/0 conn(0x7f2d54161590 0x7f2d54161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.224+0000 7f2d51e00700 1 -- 172.21.15.89:0/4240620667 >> v1:172.21.15.89:6789/0 conn(0x7f2d54160f10 legacy=0x7f2d540d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:55.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.224+0000 7f2d51e00700 1 -- 172.21.15.89:0/4240620667 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d541612f0 con 0x7f2d540d18b0 2024-04-02T07:51:55.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.224+0000 7f2d51e00700 1 --2- 172.21.15.89:0/4240620667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d540d18b0 0x7f2d540cf230 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f2d540d1030 tx=0x7f2d40009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36137e419b3ebe33 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:55.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.224+0000 7f2d51400700 1 -- 172.21.15.89:0/4240620667 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d40011830 con 0x7f2d540d18b0 2024-04-02T07:51:55.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.224+0000 7f2d51400700 1 -- 172.21.15.89:0/4240620667 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d40011990 con 0x7f2d540d18b0 2024-04-02T07:51:55.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.224+0000 7f2d51400700 1 -- 172.21.15.89:0/4240620667 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d40011d20 con 0x7f2d540d18b0 2024-04-02T07:51:55.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.224+0000 7f2d59400700 1 -- 172.21.15.89:0/4240620667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d540d18b0 msgr2=0x7f2d540cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:55.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.224+0000 7f2d59400700 1 --2- 172.21.15.89:0/4240620667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d540d18b0 0x7f2d540cf230 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f2d540d1030 tx=0x7f2d40009b20 comp rx=0 tx=0).stop 2024-04-02T07:51:55.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.225+0000 7f2d59400700 1 -- 172.21.15.89:0/4240620667 shutdown_connections 2024-04-02T07:51:55.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.225+0000 7f2d59400700 1 --2- 172.21.15.89:0/4240620667 >> v2:172.21.15.113:3300/0 conn(0x7f2d54161590 0x7f2d54161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.225+0000 7f2d59400700 1 --2- 172.21.15.89:0/4240620667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d540d18b0 0x7f2d540cf230 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.225+0000 7f2d59400700 1 -- 172.21.15.89:0/4240620667 >> 172.21.15.89:0/4240620667 conn(0x7f2d5408aff0 msgr2=0x7f2d540d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:55.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.225+0000 7f2d59400700 1 -- 172.21.15.89:0/4240620667 shutdown_connections 2024-04-02T07:51:55.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.225+0000 7f2d59400700 1 -- 172.21.15.89:0/4240620667 wait complete. 2024-04-02T07:51:55.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.226+0000 7f2d59400700 1 Processor -- start 2024-04-02T07:51:55.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.227+0000 7f2d59400700 1 -- start start 2024-04-02T07:51:55.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.227+0000 7f2d59400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d540d18b0 0x7f2d5415fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:55.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.227+0000 7f2d59400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d54160f10 0x7f2d54156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:55.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.227+0000 7f2d59400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d54161590 0x7f2d54156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:55.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.228+0000 7f2d59400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d54159b30 con 0x7f2d540d18b0 2024-04-02T07:51:55.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.228+0000 7f2d59400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d54159dd0 con 0x7f2d54160f10 2024-04-02T07:51:55.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.228+0000 7f2d59400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5415a0b0 con 0x7f2d54161590 2024-04-02T07:51:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.228+0000 7f2d52800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d540d18b0 0x7f2d5415fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.228+0000 7f2d52800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d540d18b0 0x7f2d5415fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43294/0 (socket says 172.21.15.89:43294) 2024-04-02T07:51:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.228+0000 7f2d52800700 1 -- 172.21.15.89:0/3305989759 learned_addr learned my addr 172.21.15.89:0/3305989759 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.228+0000 7f2d52800700 1 -- 172.21.15.89:0/3305989759 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d54161590 msgr2=0x7f2d54156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.228+0000 7f2d52800700 1 --2- 172.21.15.89:0/3305989759 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d54161590 0x7f2d54156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.228+0000 7f2d52800700 1 -- 172.21.15.89:0/3305989759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d54160f10 msgr2=0x7f2d54156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.228+0000 7f2d51e00700 1 --2- 172.21.15.89:0/3305989759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d54160f10 0x7f2d54156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.228+0000 7f2d52800700 1 --2- 172.21.15.89:0/3305989759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d54160f10 0x7f2d54156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.228+0000 7f2d52800700 1 -- 172.21.15.89:0/3305989759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d400094a0 con 0x7f2d540d18b0 2024-04-02T07:51:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.228+0000 7f2d51e00700 1 --2- 172.21.15.89:0/3305989759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d54160f10 0x7f2d54156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:51:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.228+0000 7f2d52800700 1 --2- 172.21.15.89:0/3305989759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d540d18b0 0x7f2d5415fff0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f2d44001eb0 tx=0x7f2d4400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.229+0000 7f2d4aa00700 1 -- 172.21.15.89:0/3305989759 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d44005d70 con 0x7f2d540d18b0 2024-04-02T07:51:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.229+0000 7f2d4aa00700 1 -- 172.21.15.89:0/3305989759 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d440070d0 con 0x7f2d540d18b0 2024-04-02T07:51:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.229+0000 7f2d59400700 1 -- 172.21.15.89:0/3305989759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5415a390 con 0x7f2d540d18b0 2024-04-02T07:51:55.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.229+0000 7f2d4aa00700 1 -- 172.21.15.89:0/3305989759 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d44007460 con 0x7f2d540d18b0 2024-04-02T07:51:55.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.229+0000 7f2d59400700 1 -- 172.21.15.89:0/3305989759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5415d400 con 0x7f2d540d18b0 2024-04-02T07:51:55.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.230+0000 7f2d59400700 1 -- 172.21.15.89:0/3305989759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d54067720 con 0x7f2d540d18b0 2024-04-02T07:51:55.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.231+0000 7f2d4aa00700 1 -- 172.21.15.89:0/3305989759 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 39) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f2d440075c0 con 0x7f2d540d18b0 2024-04-02T07:51:55.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.232+0000 7f2d4aa00700 1 -- 172.21.15.89:0/3305989759 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f2d440886f0 con 0x7f2d540d18b0 2024-04-02T07:51:55.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.236+0000 7f2d4aa00700 1 -- 172.21.15.89:0/3305989759 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2d440538d0 con 0x7f2d540d18b0 2024-04-02T07:51:55.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.437+0000 7f2d59400700 1 -- 172.21.15.89:0/3305989759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d540ce1b0 con 0x7f2d540d18b0 2024-04-02T07:51:55.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.439+0000 7f2d4aa00700 1 -- 172.21.15.89:0/3305989759 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+17482 (secure 0 0 0) 0x7f2d44017090 con 0x7f2d540d18b0 2024-04-02T07:51:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:51:55.441 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":85,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:51:54.785699+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:51:27.926367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-02T07:51:28.662600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/3327519964":"2024-04-03T07:51:54.785636+0000","172.21.15.89:0/2449365692":"2024-04-03T07:51:54.785636+0000","172.21.15.89:0/2535782241":"2024-04-03T07:51:54.785636+0000","172.21.15.89:0/3681777721":"2024-04-03T07:51:54.785636+0000","172.21.15.89:0/822661295":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/907732608":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/3083880072":"2024-04-03T07:51:45.730929+0000","172.21.15.89:6833/43357":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/952215678":"2024-04-03T07:51:54.785636+0000","172.21.15.89:6832/43357":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/1211892288":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/3540824059":"2024-04-03T07:51:54.785636+0000","172.21.15.89:0/23281819":"2024-04-03T07:51:45.730929+0000","172.21.15.89:0/3435265311":"2024-04-03T07:51:45.730929+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:51:55.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.441+0000 7f2d59400700 1 -- 172.21.15.89:0/3305989759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d540d18b0 msgr2=0x7f2d5415fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:55.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.441+0000 7f2d59400700 1 --2- 172.21.15.89:0/3305989759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d540d18b0 0x7f2d5415fff0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f2d44001eb0 tx=0x7f2d4400c7e0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.441+0000 7f2d59400700 1 -- 172.21.15.89:0/3305989759 shutdown_connections 2024-04-02T07:51:55.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.441+0000 7f2d59400700 1 --2- 172.21.15.89:0/3305989759 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d54161590 0x7f2d54156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.441+0000 7f2d59400700 1 --2- 172.21.15.89:0/3305989759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d540d18b0 0x7f2d5415fff0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.441+0000 7f2d59400700 1 --2- 172.21.15.89:0/3305989759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d54160f10 0x7f2d54156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.441+0000 7f2d59400700 1 -- 172.21.15.89:0/3305989759 >> 172.21.15.89:0/3305989759 conn(0x7f2d5408aff0 msgr2=0x7f2d540cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:55.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.441+0000 7f2d59400700 1 -- 172.21.15.89:0/3305989759 shutdown_connections 2024-04-02T07:51:55.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.441+0000 7f2d59400700 1 -- 172.21.15.89:0/3305989759 wait complete. 2024-04-02T07:51:55.461 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T07:51:55.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.650+0000 7fa78b600700 1 Processor -- start 2024-04-02T07:51:55.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.650+0000 7fa78b600700 1 -- start start 2024-04-02T07:51:55.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.651+0000 7fa78b600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7840cd5c0 0x7fa78415e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:55.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.651+0000 7fa78b600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa78415cfe0 0x7fa78415d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:55.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.652+0000 7fa78b600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78415eda0 con 0x7fa78406f310 2024-04-02T07:51:55.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.652+0000 7fa78b600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa784099dd0 con 0x7fa7840cd5c0 2024-04-02T07:51:55.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.652+0000 7fa78b600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78409a0b0 con 0x7fa78415cfe0 2024-04-02T07:51:55.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.652+0000 7fa788c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa7840cd5c0 0x7fa78415e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:55.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.652+0000 7fa783e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa78406f310 0x7fa78406f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46014/0 (socket says 172.21.15.89:46014) 2024-04-02T07:51:55.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.652+0000 7fa783e00700 1 -- 172.21.15.89:0/766201921 learned_addr learned my addr 172.21.15.89:0/766201921 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:55.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.652+0000 7fa789600700 1 --2- 172.21.15.89:0/766201921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa78415cfe0 0x7fa78415d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:55.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.652+0000 7fa788c00700 1 -- 172.21.15.89:0/766201921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa78415cfe0 msgr2=0x7fa78415d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:55.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.652+0000 7fa788c00700 1 --2- 172.21.15.89:0/766201921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa78415cfe0 0x7fa78415d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.652+0000 7fa788c00700 1 -- 172.21.15.89:0/766201921 >> v1:172.21.15.89:6789/0 conn(0x7fa78406f310 legacy=0x7fa78406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:55.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.652+0000 7fa788c00700 1 -- 172.21.15.89:0/766201921 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78409a390 con 0x7fa7840cd5c0 2024-04-02T07:51:55.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.652+0000 7fa789600700 1 --2- 172.21.15.89:0/766201921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa78415cfe0 0x7fa78415d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:55.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.652+0000 7fa783400700 1 -- 172.21.15.89:0/766201921 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334769969 0 0) 0x7fa770001110 con 0x7fa78406f310 2024-04-02T07:51:55.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.653+0000 7fa788c00700 1 --2- 172.21.15.89:0/766201921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7840cd5c0 0x7fa78415e860 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fa778009870 tx=0x7fa778009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5253e8933c46102f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:55.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.653+0000 7fa783400700 1 -- 172.21.15.89:0/766201921 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778011830 con 0x7fa7840cd5c0 2024-04-02T07:51:55.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.653+0000 7fa783400700 1 -- 172.21.15.89:0/766201921 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa778011990 con 0x7fa7840cd5c0 2024-04-02T07:51:55.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.653+0000 7fa783400700 1 -- 172.21.15.89:0/766201921 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778011d20 con 0x7fa7840cd5c0 2024-04-02T07:51:55.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.653+0000 7fa78b600700 1 -- 172.21.15.89:0/766201921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7840cd5c0 msgr2=0x7fa78415e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:55.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.653+0000 7fa78b600700 1 --2- 172.21.15.89:0/766201921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7840cd5c0 0x7fa78415e860 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fa778009870 tx=0x7fa778009b80 comp rx=0 tx=0).stop 2024-04-02T07:51:55.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.653+0000 7fa78b600700 1 -- 172.21.15.89:0/766201921 shutdown_connections 2024-04-02T07:51:55.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.653+0000 7fa78b600700 1 --2- 172.21.15.89:0/766201921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa78415cfe0 0x7fa78415d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.653+0000 7fa78b600700 1 --2- 172.21.15.89:0/766201921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7840cd5c0 0x7fa78415e860 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.653+0000 7fa78b600700 1 -- 172.21.15.89:0/766201921 >> 172.21.15.89:0/766201921 conn(0x7fa7840d1370 msgr2=0x7fa7840d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:55.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.653+0000 7fa78b600700 1 -- 172.21.15.89:0/766201921 shutdown_connections 2024-04-02T07:51:55.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.653+0000 7fa78b600700 1 -- 172.21.15.89:0/766201921 wait complete. 2024-04-02T07:51:55.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.655+0000 7fa78b600700 1 Processor -- start 2024-04-02T07:51:55.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.655+0000 7fa78b600700 1 -- start start 2024-04-02T07:51:55.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.656+0000 7fa78b600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa78406f310 0x7fa7840b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:55.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.656+0000 7fa78b600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7840cd5c0 0x7fa7841f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:55.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.656+0000 7fa78b600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa78415cfe0 0x7fa7841f73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:55.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.656+0000 7fa78b600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7840b17b0 con 0x7fa78415cfe0 2024-04-02T07:51:55.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.656+0000 7fa78b600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7840b1a90 con 0x7fa78406f310 2024-04-02T07:51:55.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.656+0000 7fa78b600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7840b1d70 con 0x7fa7840cd5c0 2024-04-02T07:51:55.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.656+0000 7fa789600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa78415cfe0 0x7fa7841f73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:55.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.656+0000 7fa789600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa78415cfe0 0x7fa7841f73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43306/0 (socket says 172.21.15.89:43306) 2024-04-02T07:51:55.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.656+0000 7fa789600700 1 -- 172.21.15.89:0/1472170167 learned_addr learned my addr 172.21.15.89:0/1472170167 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:55.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.657+0000 7fa783e00700 1 --2- 172.21.15.89:0/1472170167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7840cd5c0 0x7fa7841f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:55.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.657+0000 7fa788c00700 1 --2- 172.21.15.89:0/1472170167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa78406f310 0x7fa7840b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:55.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.657+0000 7fa789600700 1 -- 172.21.15.89:0/1472170167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7840cd5c0 msgr2=0x7fa7841f6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:55.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.657+0000 7fa789600700 1 --2- 172.21.15.89:0/1472170167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7840cd5c0 0x7fa7841f6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.657+0000 7fa789600700 1 -- 172.21.15.89:0/1472170167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa78406f310 msgr2=0x7fa7840b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:55.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.657+0000 7fa789600700 1 --2- 172.21.15.89:0/1472170167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa78406f310 0x7fa7840b0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.657+0000 7fa789600700 1 -- 172.21.15.89:0/1472170167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7780094a0 con 0x7fa78415cfe0 2024-04-02T07:51:55.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.657+0000 7fa789600700 1 --2- 172.21.15.89:0/1472170167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa78415cfe0 0x7fa7841f73c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fa77c007e70 tx=0x7fa77c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:55.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.657+0000 7fa780c00700 1 -- 172.21.15.89:0/1472170167 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77c00dd20 con 0x7fa78415cfe0 2024-04-02T07:51:55.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.657+0000 7fa780c00700 1 -- 172.21.15.89:0/1472170167 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77c00f040 con 0x7fa78415cfe0 2024-04-02T07:51:55.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.658+0000 7fa78b600700 1 -- 172.21.15.89:0/1472170167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7840b4260 con 0x7fa78415cfe0 2024-04-02T07:51:55.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.658+0000 7fa780c00700 1 -- 172.21.15.89:0/1472170167 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77c00dea0 con 0x7fa78415cfe0 2024-04-02T07:51:55.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.658+0000 7fa78b600700 1 -- 172.21.15.89:0/1472170167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7840b6a70 con 0x7fa78415cfe0 2024-04-02T07:51:55.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.659+0000 7fa78b600700 1 -- 172.21.15.89:0/1472170167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7840b4780 con 0x7fa78415cfe0 2024-04-02T07:51:55.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.660+0000 7fa780c00700 1 -- 172.21.15.89:0/1472170167 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 39) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fa77c003c60 con 0x7fa78415cfe0 2024-04-02T07:51:55.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.660+0000 7fa780c00700 1 -- 172.21.15.89:0/1472170167 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fa77c0891f0 con 0x7fa78415cfe0 2024-04-02T07:51:55.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.665+0000 7fa780c00700 1 -- 172.21.15.89:0/1472170167 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa7840b4780 con 0x7fa78415cfe0 2024-04-02T07:51:55.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.797+0000 7fa780c00700 1 -- 172.21.15.89:0/1472170167 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 40) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fa77c053e20 con 0x7fa78415cfe0 2024-04-02T07:51:55.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.797+0000 7fa780c00700 1 --2- 172.21.15.89:0/1472170167 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa764036100 0x7fa764072620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:55.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.798+0000 7fa788c00700 1 --2- 172.21.15.89:0/1472170167 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa764036100 0x7fa764072620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:55.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.802+0000 7fa788c00700 1 --2- 172.21.15.89:0/1472170167 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa764036100 0x7fa764072620 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa778009790 tx=0x7fa778008030 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:55.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.918+0000 7fa78b600700 1 -- 172.21.15.89:0/1472170167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa7840b4780 con 0x7fa78415cfe0 2024-04-02T07:51:55.921 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T07:51:55.919+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T07:51:55.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.928+0000 7fa780c00700 1 -- 172.21.15.89:0/1472170167 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa7840b4780 con 0x7fa78415cfe0 2024-04-02T07:51:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.931+0000 7fa78b600700 1 -- 172.21.15.89:0/1472170167 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa764036100 msgr2=0x7fa764072620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.931+0000 7fa78b600700 1 --2- 172.21.15.89:0/1472170167 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa764036100 0x7fa764072620 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa778009790 tx=0x7fa778008030 comp rx=0 tx=0).stop 2024-04-02T07:51:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.931+0000 7fa78b600700 1 -- 172.21.15.89:0/1472170167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa78415cfe0 msgr2=0x7fa7841f73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.931+0000 7fa78b600700 1 --2- 172.21.15.89:0/1472170167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa78415cfe0 0x7fa7841f73c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fa77c007e70 tx=0x7fa77c00d3b0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.931+0000 7fa78b600700 1 -- 172.21.15.89:0/1472170167 shutdown_connections 2024-04-02T07:51:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.931+0000 7fa78b600700 1 --2- 172.21.15.89:0/1472170167 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa764036100 0x7fa764072620 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.931+0000 7fa78b600700 1 --2- 172.21.15.89:0/1472170167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa78406f310 0x7fa7840b0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.931+0000 7fa78b600700 1 --2- 172.21.15.89:0/1472170167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa78415cfe0 0x7fa7841f73c0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.931+0000 7fa78b600700 1 --2- 172.21.15.89:0/1472170167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7840cd5c0 0x7fa7841f6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.931+0000 7fa78b600700 1 -- 172.21.15.89:0/1472170167 >> 172.21.15.89:0/1472170167 conn(0x7fa7840d1370 msgr2=0x7fa78415c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.931+0000 7fa78b600700 1 -- 172.21.15.89:0/1472170167 shutdown_connections 2024-04-02T07:51:55.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:55.931+0000 7fa78b600700 1 -- 172.21.15.89:0/1472170167 wait complete. 2024-04-02T07:51:55.935 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T07:51:55.941 INFO:tasks.ceph.mds.a.smithi089.stderr: -4938> 2024-04-02T07:51:35.175+0000 7ff2c8600700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T07:51:55.954 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T07:51:55.992 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T07:51:56.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.146+0000 7f3fef600700 1 Processor -- start 2024-04-02T07:51:56.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.146+0000 7f3fef600700 1 -- start start 2024-04-02T07:51:56.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.147+0000 7f3fef600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fe8160cc0 0x7f3fe8168730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:56.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.147+0000 7f3fef600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe815f820 con 0x7f3fe80ced80 2024-04-02T07:51:56.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.147+0000 7f3fef600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe80d2450 con 0x7f3fe815dad0 2024-04-02T07:51:56.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.147+0000 7f3fef600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe80cfcc0 con 0x7f3fe8160cc0 2024-04-02T07:51:56.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.147+0000 7f3fe7e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3fe80ced80 0x7f3fe80d1d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46028/0 (socket says 172.21.15.89:46028) 2024-04-02T07:51:56.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.147+0000 7f3fe7e00700 1 -- 172.21.15.89:0/795359511 learned_addr learned my addr 172.21.15.89:0/795359511 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:56.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fed600700 1 --2- 172.21.15.89:0/795359511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fe8160cc0 0x7f3fe8168730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:56.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408124967 0 0) 0x7f3fe815f820 con 0x7f3fe80ced80 2024-04-02T07:51:56.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd00035b0 con 0x7f3fe80ced80 2024-04-02T07:51:56.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590134843 0 0) 0x7f3fe80d2450 con 0x7f3fe815dad0 2024-04-02T07:51:56.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe815f820 con 0x7f3fe815dad0 2024-04-02T07:51:56.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1082510544 0 0) 0x7f3fd00035b0 con 0x7f3fe80ced80 2024-04-02T07:51:56.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fe8160cc0 msgr2=0x7f3fe8168730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:56.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 --2- 172.21.15.89:0/795359511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fe8160cc0 0x7f3fe8168730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:56.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 >> v1:172.21.15.113:6789/0 conn(0x7f3fe815dad0 legacy=0x7f3fe80d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:56.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe80cf420 con 0x7f3fe80ced80 2024-04-02T07:51:56.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f3fd4001670 con 0x7f3fe80ced80 2024-04-02T07:51:56.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 >> v1:172.21.15.89:6789/0 conn(0x7f3fe80ced80 legacy=0x7f3fe80d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:56.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 --2- 172.21.15.89:0/795359511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fe8160cc0 0x7f3fd0003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:56.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 --2- 172.21.15.89:0/795359511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd0004120 0x7f3fd0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:56.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 --2- 172.21.15.89:0/795359511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fd0004d90 0x7f3fd0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:56.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe80cfcc0 con 0x7f3fd0004120 2024-04-02T07:51:56.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd000baf0 con 0x7f3fd0004d90 2024-04-02T07:51:56.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.148+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd000bcb0 con 0x7f3fe8160cc0 2024-04-02T07:51:56.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.149+0000 7f3fe7e00700 1 --2- 172.21.15.89:0/795359511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd0004120 0x7f3fd0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:56.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.149+0000 7f3fecc00700 1 --2- 172.21.15.89:0/795359511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fe8160cc0 0x7f3fd0003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:56.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.149+0000 7f3fed600700 1 --2- 172.21.15.89:0/795359511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fd0004d90 0x7f3fd0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:56.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.149+0000 7f3fe7e00700 1 -- 172.21.15.89:0/795359511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fe8160cc0 msgr2=0x7f3fd0003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:56.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.149+0000 7f3fe7e00700 1 --2- 172.21.15.89:0/795359511 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fe8160cc0 0x7f3fd0003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:56.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.149+0000 7f3fe7e00700 1 -- 172.21.15.89:0/795359511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fd0004d90 msgr2=0x7f3fd0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:56.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.149+0000 7f3fe7e00700 1 --2- 172.21.15.89:0/795359511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fd0004d90 0x7f3fd0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:56.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.149+0000 7f3fe7e00700 1 -- 172.21.15.89:0/795359511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd00035b0 con 0x7f3fd0004120 2024-04-02T07:51:56.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.149+0000 7f3fe7e00700 1 --2- 172.21.15.89:0/795359511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd0004120 0x7f3fd0004850 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400c210 tx=0x7f3fd0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef012b98afe2dd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:56.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.149+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd400cd50 con 0x7f3fd0004120 2024-04-02T07:51:56.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.150+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd4005960 con 0x7f3fd0004120 2024-04-02T07:51:56.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.150+0000 7f3fe7400700 1 -- 172.21.15.89:0/795359511 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4005d10 con 0x7f3fd0004120 2024-04-02T07:51:56.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.150+0000 7f3fef600700 1 -- 172.21.15.89:0/795359511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd0004120 msgr2=0x7f3fd0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:56.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.150+0000 7f3fef600700 1 --2- 172.21.15.89:0/795359511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd0004120 0x7f3fd0004850 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400c210 tx=0x7f3fd0005880 comp rx=0 tx=0).stop 2024-04-02T07:51:56.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.150+0000 7f3fed600700 1 -- 172.21.15.89:0/795359511 reap_dead start 2024-04-02T07:51:56.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.150+0000 7f3fef600700 1 -- 172.21.15.89:0/795359511 shutdown_connections 2024-04-02T07:51:56.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.150+0000 7f3fef600700 1 -- 172.21.15.89:0/795359511 >> 172.21.15.89:0/795359511 conn(0x7f3fe808aff0 msgr2=0x7f3fe80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:56.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.150+0000 7f3fef600700 1 -- 172.21.15.89:0/795359511 shutdown_connections 2024-04-02T07:51:56.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.150+0000 7f3fef600700 1 -- 172.21.15.89:0/795359511 wait complete. 2024-04-02T07:51:56.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.152+0000 7f3fef600700 1 Processor -- start 2024-04-02T07:51:56.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.152+0000 7f3fef600700 1 -- start start 2024-04-02T07:51:56.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.153+0000 7f3fef600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fd0004120 0x7f3fe80c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:56.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.153+0000 7f3fef600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe80c7920 0x7f3fe80c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:56.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.153+0000 7f3fef600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe8150480 0x7f3fe81508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:56.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.153+0000 7f3fef600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe80cc050 con 0x7f3fe8150480 2024-04-02T07:51:56.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.153+0000 7f3fef600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe80cceb0 con 0x7f3fe80c7920 2024-04-02T07:51:56.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.153+0000 7f3fef600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe80c82e0 con 0x7f3fd0004120 2024-04-02T07:51:56.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.153+0000 7f3fed600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe8150480 0x7f3fe81508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:56.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.153+0000 7f3fecc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fd0004120 0x7f3fe80c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:56.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.153+0000 7f3fecc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fd0004120 0x7f3fe80c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:47264/0 (socket says 172.21.15.89:47264) 2024-04-02T07:51:56.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.153+0000 7f3fecc00700 1 -- 172.21.15.89:0/78183741 learned_addr learned my addr 172.21.15.89:0/78183741 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:56.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.153+0000 7f3fe7e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe80c7920 0x7f3fe80c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:56.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.154+0000 7f3fecc00700 1 -- 172.21.15.89:0/78183741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe80c7920 msgr2=0x7f3fe80c7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:56.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.154+0000 7f3fecc00700 1 --2- 172.21.15.89:0/78183741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe80c7920 0x7f3fe80c7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:56.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.154+0000 7f3fecc00700 1 -- 172.21.15.89:0/78183741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe8150480 msgr2=0x7f3fe81508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:56.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.154+0000 7f3fecc00700 1 --2- 172.21.15.89:0/78183741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe8150480 0x7f3fe81508c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:56.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.154+0000 7f3fecc00700 1 -- 172.21.15.89:0/78183741 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe80cc2a0 con 0x7f3fd0004120 2024-04-02T07:51:56.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.154+0000 7f3fed600700 1 --2- 172.21.15.89:0/78183741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe8150480 0x7f3fe81508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:56.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.154+0000 7f3fecc00700 1 --2- 172.21.15.89:0/78183741 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fd0004120 0x7f3fe80c9ab0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc00f840 tx=0x7f3fdc00fb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:56.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.154+0000 7f3fe4c00700 1 -- 172.21.15.89:0/78183741 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc0052e0 con 0x7f3fd0004120 2024-04-02T07:51:56.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.154+0000 7f3fe4c00700 1 -- 172.21.15.89:0/78183741 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fdc005440 con 0x7f3fd0004120 2024-04-02T07:51:56.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.154+0000 7f3fef600700 1 -- 172.21.15.89:0/78183741 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe80cc580 con 0x7f3fd0004120 2024-04-02T07:51:56.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.154+0000 7f3fe4c00700 1 -- 172.21.15.89:0/78183741 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc0038c0 con 0x7f3fd0004120 2024-04-02T07:51:56.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.155+0000 7f3fef600700 1 -- 172.21.15.89:0/78183741 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe80ccb40 con 0x7f3fd0004120 2024-04-02T07:51:56.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.155+0000 7f3fef600700 1 -- 172.21.15.89:0/78183741 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe8067720 con 0x7f3fd0004120 2024-04-02T07:51:56.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.157+0000 7f3fe4c00700 1 -- 172.21.15.89:0/78183741 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 40) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f3fdc003a20 con 0x7f3fd0004120 2024-04-02T07:51:56.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.157+0000 7f3fe4c00700 1 --2- 172.21.15.89:0/78183741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3fc806aae0 0x7f3fc806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:56.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.158+0000 7f3fe4c00700 1 -- 172.21.15.89:0/78183741 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f3fdc089630 con 0x7f3fd0004120 2024-04-02T07:51:56.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.158+0000 7f3fe7e00700 1 --2- 172.21.15.89:0/78183741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3fc806aae0 0x7f3fc806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:56.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.158+0000 7f3fe7e00700 1 --2- 172.21.15.89:0/78183741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3fc806aae0 0x7f3fc806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4000c00 tx=0x7f3fd4007f90 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:56.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.161+0000 7f3fe4c00700 1 -- 172.21.15.89:0/78183741 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3fdc0546f0 con 0x7f3fd0004120 2024-04-02T07:51:56.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.403+0000 7f3fef600700 1 -- 172.21.15.89:0/78183741 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3fe806c2b0 con 0x7f3fd0004120 2024-04-02T07:51:56.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.974+0000 7f3fe4c00700 1 -- 172.21.15.89:0/78183741 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mgrmap(e 41) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f3fdc003cd0 con 0x7f3fd0004120 2024-04-02T07:51:56.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.980+0000 7f3fe4c00700 1 -- 172.21.15.89:0/78183741 <== mon.2 v2:172.21.15.113:3301/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3fdc040050 con 0x7f3fd0004120 2024-04-02T07:51:56.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.983+0000 7f3fef600700 1 -- 172.21.15.89:0/78183741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3fc806aae0 msgr2=0x7f3fc806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:56.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.983+0000 7f3fef600700 1 --2- 172.21.15.89:0/78183741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3fc806aae0 0x7f3fc806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4000c00 tx=0x7f3fd4007f90 comp rx=0 tx=0).stop 2024-04-02T07:51:56.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.983+0000 7f3fef600700 1 -- 172.21.15.89:0/78183741 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fd0004120 msgr2=0x7f3fe80c9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:56.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.983+0000 7f3fef600700 1 --2- 172.21.15.89:0/78183741 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fd0004120 0x7f3fe80c9ab0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc00f840 tx=0x7f3fdc00fb50 comp rx=0 tx=0).stop 2024-04-02T07:51:56.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.983+0000 7f3fef600700 1 -- 172.21.15.89:0/78183741 shutdown_connections 2024-04-02T07:51:56.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.983+0000 7f3fef600700 1 --2- 172.21.15.89:0/78183741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3fc806aae0 0x7f3fc806cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:56.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.983+0000 7f3fef600700 1 --2- 172.21.15.89:0/78183741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe8150480 0x7f3fe81508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:56.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.983+0000 7f3fef600700 1 --2- 172.21.15.89:0/78183741 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fd0004120 0x7f3fe80c9ab0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:56.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.983+0000 7f3fef600700 1 --2- 172.21.15.89:0/78183741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe80c7920 0x7f3fe80c7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:56.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.983+0000 7f3fef600700 1 -- 172.21.15.89:0/78183741 >> 172.21.15.89:0/78183741 conn(0x7f3fe808aff0 msgr2=0x7f3fe815ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:56.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.983+0000 7f3fef600700 1 -- 172.21.15.89:0/78183741 shutdown_connections 2024-04-02T07:51:56.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:56.983+0000 7f3fef600700 1 -- 172.21.15.89:0/78183741 wait complete. 2024-04-02T07:51:57.004 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T07:51:57.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.236+0000 7f44b9e00700 1 Processor -- start 2024-04-02T07:51:57.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.236+0000 7f44b9e00700 1 -- start start 2024-04-02T07:51:57.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.237+0000 7f44b9e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f44b41580c0 0x7f44b406e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:57.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.237+0000 7f44b9e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b4070980 con 0x7f44b406e7f0 2024-04-02T07:51:57.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.237+0000 7f44b9e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b415b760 con 0x7f44b415b330 2024-04-02T07:51:57.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.237+0000 7f44b9e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b4071570 con 0x7f44b41580c0 2024-04-02T07:51:57.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.238+0000 7f44abe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f44b406e7f0 0x7f44b415ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46030/0 (socket says 172.21.15.89:46030) 2024-04-02T07:51:57.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.238+0000 7f44abe00700 1 -- 172.21.15.89:0/11249136 learned_addr learned my addr 172.21.15.89:0/11249136 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:57.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.238+0000 7f44b2a00700 1 --2- 172.21.15.89:0/11249136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f44b41580c0 0x7f44b406e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:57.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.238+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578445793 0 0) 0x7f44b4070980 con 0x7f44b406e7f0 2024-04-02T07:51:57.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.238+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44980035b0 con 0x7f44b406e7f0 2024-04-02T07:51:57.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.238+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426801434 0 0) 0x7f44b415b760 con 0x7f44b415b330 2024-04-02T07:51:57.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.238+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b4070980 con 0x7f44b415b330 2024-04-02T07:51:57.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.238+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3551238737 0 0) 0x7f44980035b0 con 0x7f44b406e7f0 2024-04-02T07:51:57.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.238+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f44b41580c0 msgr2=0x7f44b406e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:57.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.238+0000 7f44b2000700 1 --2- 172.21.15.89:0/11249136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f44b41580c0 0x7f44b406e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:57.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.238+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 >> v1:172.21.15.113:6789/0 conn(0x7f44b415b330 legacy=0x7f44b40b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:57.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.238+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b4070bd0 con 0x7f44b406e7f0 2024-04-02T07:51:57.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.238+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f449c001670 con 0x7f44b406e7f0 2024-04-02T07:51:57.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 >> v1:172.21.15.89:6789/0 conn(0x7f44b406e7f0 legacy=0x7f44b415ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:57.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b2000700 1 --2- 172.21.15.89:0/11249136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4498003bc0 0x7f4498003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:57.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b2000700 1 --2- 172.21.15.89:0/11249136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4498004850 0x7f4498004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:57.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b2000700 1 --2- 172.21.15.89:0/11249136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4498005210 0x7f449800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:57.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b4071570 con 0x7f4498003bc0 2024-04-02T07:51:57.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449800c0e0 con 0x7f4498004850 2024-04-02T07:51:57.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449800c2a0 con 0x7f4498005210 2024-04-02T07:51:57.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b2a00700 1 --2- 172.21.15.89:0/11249136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f44b41580c0 0x7f44b406e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:57.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44abe00700 1 --2- 172.21.15.89:0/11249136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4498004850 0x7f4498004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:57.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b2a00700 1 --2- 172.21.15.89:0/11249136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4498003bc0 0x7f4498003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:57.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b3400700 1 --2- 172.21.15.89:0/11249136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4498005210 0x7f449800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:57.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b2a00700 1 -- 172.21.15.89:0/11249136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4498005210 msgr2=0x7f449800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:57.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b2a00700 1 --2- 172.21.15.89:0/11249136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4498005210 0x7f449800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:57.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b2a00700 1 -- 172.21.15.89:0/11249136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4498004850 msgr2=0x7f4498004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:57.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b2a00700 1 --2- 172.21.15.89:0/11249136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4498004850 0x7f4498004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:57.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.239+0000 7f44b2a00700 1 -- 172.21.15.89:0/11249136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44980035b0 con 0x7f4498003bc0 2024-04-02T07:51:57.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.240+0000 7f44b2a00700 1 --2- 172.21.15.89:0/11249136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4498003bc0 0x7f4498003fc0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f44b4070390 tx=0x7f4498005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48071aaef4b69411 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:57.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.240+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a4004030 con 0x7f4498003bc0 2024-04-02T07:51:57.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.240+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a4004190 con 0x7f4498003bc0 2024-04-02T07:51:57.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.240+0000 7f44b2000700 1 -- 172.21.15.89:0/11249136 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a4004540 con 0x7f4498003bc0 2024-04-02T07:51:57.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.240+0000 7f44b9e00700 1 -- 172.21.15.89:0/11249136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4498003bc0 msgr2=0x7f4498003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:57.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.240+0000 7f44b9e00700 1 --2- 172.21.15.89:0/11249136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4498003bc0 0x7f4498003fc0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f44b4070390 tx=0x7f4498005880 comp rx=0 tx=0).stop 2024-04-02T07:51:57.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.240+0000 7f44b3400700 1 -- 172.21.15.89:0/11249136 reap_dead start 2024-04-02T07:51:57.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.240+0000 7f44b9e00700 1 -- 172.21.15.89:0/11249136 shutdown_connections 2024-04-02T07:51:57.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.240+0000 7f44b9e00700 1 -- 172.21.15.89:0/11249136 >> 172.21.15.89:0/11249136 conn(0x7f44b40cf210 msgr2=0x7f44b40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:57.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.241+0000 7f44b9e00700 1 -- 172.21.15.89:0/11249136 shutdown_connections 2024-04-02T07:51:57.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.241+0000 7f44b9e00700 1 -- 172.21.15.89:0/11249136 wait complete. 2024-04-02T07:51:57.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.242+0000 7f44b9e00700 1 Processor -- start 2024-04-02T07:51:57.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.242+0000 7f44b9e00700 1 -- start start 2024-04-02T07:51:57.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.243+0000 7f44b9e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4498003bc0 0x7f44b414b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:57.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.243+0000 7f44b9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44b415e4c0 0x7f44b415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:57.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.243+0000 7f44b9e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44b4156900 0x7f44b4156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:57.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.243+0000 7f44b9e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b4071990 con 0x7f44b415e4c0 2024-04-02T07:51:57.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.243+0000 7f44b9e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b41572c0 con 0x7f44b4156900 2024-04-02T07:51:57.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.243+0000 7f44b9e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b415e030 con 0x7f4498003bc0 2024-04-02T07:51:57.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44abe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44b415e4c0 0x7f44b415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:57.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44abe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44b415e4c0 0x7f44b415a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43384/0 (socket says 172.21.15.89:43384) 2024-04-02T07:51:57.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44abe00700 1 -- 172.21.15.89:0/1680022432 learned_addr learned my addr 172.21.15.89:0/1680022432 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:57.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44b2a00700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4498003bc0 0x7f44b414b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:57.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44b3400700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44b4156900 0x7f44b4156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:57.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44abe00700 1 -- 172.21.15.89:0/1680022432 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4498003bc0 msgr2=0x7f44b414b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:57.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44abe00700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4498003bc0 0x7f44b414b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:57.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44abe00700 1 -- 172.21.15.89:0/1680022432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44b4156900 msgr2=0x7f44b4156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:57.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44abe00700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44b4156900 0x7f44b4156d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:57.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44abe00700 1 -- 172.21.15.89:0/1680022432 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b414bb40 con 0x7f44b415e4c0 2024-04-02T07:51:57.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44b3400700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44b4156900 0x7f44b4156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:57.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44abe00700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44b415e4c0 0x7f44b415a2e0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f449c001d30 tx=0x7f449c007320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:57.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44ab400700 1 -- 172.21.15.89:0/1680022432 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c011850 con 0x7f44b415e4c0 2024-04-02T07:51:57.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44ab400700 1 -- 172.21.15.89:0/1680022432 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449c0119b0 con 0x7f44b415e4c0 2024-04-02T07:51:57.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44b9e00700 1 -- 172.21.15.89:0/1680022432 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b415a820 con 0x7f44b415e4c0 2024-04-02T07:51:57.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.244+0000 7f44ab400700 1 -- 172.21.15.89:0/1680022432 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c011d60 con 0x7f44b415e4c0 2024-04-02T07:51:57.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.245+0000 7f44b9e00700 1 -- 172.21.15.89:0/1680022432 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b415d700 con 0x7f44b415e4c0 2024-04-02T07:51:57.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.246+0000 7f44b9e00700 1 -- 172.21.15.89:0/1680022432 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b4067720 con 0x7f44b415e4c0 2024-04-02T07:51:57.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.246+0000 7f44ab400700 1 -- 172.21.15.89:0/1680022432 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 41) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f449c0103d0 con 0x7f44b415e4c0 2024-04-02T07:51:57.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.247+0000 7f44ab400700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f449006ad20 0x7f449006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:57.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.247+0000 7f44ab400700 1 -- 172.21.15.89:0/1680022432 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f449c08a0c0 con 0x7f44b415e4c0 2024-04-02T07:51:57.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.248+0000 7f44b2a00700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f449006ad20 0x7f449006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:57.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.248+0000 7f44b2a00700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f449006ad20 0x7f449006d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f44a4016040 tx=0x7f44a401f040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:57.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.252+0000 7f44ab400700 1 -- 172.21.15.89:0/1680022432 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f449c058720 con 0x7f44b415e4c0 2024-04-02T07:51:57.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:57.451+0000 7f44b9e00700 1 -- 172.21.15.89:0/1680022432 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f44b415ec50 con 0x7f44b415e4c0 2024-04-02T07:51:58.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.092+0000 7f44ab400700 1 -- 172.21.15.89:0/1680022432 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v87) v1 ==== 181+0+0 (secure 0 0 0) 0x7f449c01f0d0 con 0x7f44b415e4c0 2024-04-02T07:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.096+0000 7f44b9e00700 1 -- 172.21.15.89:0/1680022432 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f449006ad20 msgr2=0x7f449006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.096+0000 7f44b9e00700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f449006ad20 0x7f449006d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f44a4016040 tx=0x7f44a401f040 comp rx=0 tx=0).stop 2024-04-02T07:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.096+0000 7f44b9e00700 1 -- 172.21.15.89:0/1680022432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44b415e4c0 msgr2=0x7f44b415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.096+0000 7f44b9e00700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44b415e4c0 0x7f44b415a2e0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f449c001d30 tx=0x7f449c007320 comp rx=0 tx=0).stop 2024-04-02T07:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.096+0000 7f44b9e00700 1 -- 172.21.15.89:0/1680022432 shutdown_connections 2024-04-02T07:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.096+0000 7f44b9e00700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f449006ad20 0x7f449006d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.096+0000 7f44b9e00700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44b4156900 0x7f44b4156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.096+0000 7f44b9e00700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4498003bc0 0x7f44b414b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.096+0000 7f44b9e00700 1 --2- 172.21.15.89:0/1680022432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44b415e4c0 0x7f44b415a2e0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.096+0000 7f44b9e00700 1 -- 172.21.15.89:0/1680022432 >> 172.21.15.89:0/1680022432 conn(0x7f44b40cf210 msgr2=0x7f44b40ad0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.096+0000 7f44b9e00700 1 -- 172.21.15.89:0/1680022432 shutdown_connections 2024-04-02T07:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.096+0000 7f44b9e00700 1 -- 172.21.15.89:0/1680022432 wait complete. 2024-04-02T07:51:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T07:51:58.119 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T07:51:58.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.313+0000 7f4e2f400700 1 Processor -- start 2024-04-02T07:51:58.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.313+0000 7f4e2f400700 1 -- start start 2024-04-02T07:51:58.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.314+0000 7f4e2f400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4e2815ddf0 0x7f4e280ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:58.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.314+0000 7f4e2f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4e280d28e0 0x7f4e2815f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:58.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.314+0000 7f4e2f400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e280ce8a0 con 0x7f4e2815e1d0 2024-04-02T07:51:58.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.314+0000 7f4e2f400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2815ec60 con 0x7f4e2815ddf0 2024-04-02T07:51:58.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.314+0000 7f4e2f400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280cdc90 con 0x7f4e280d28e0 2024-04-02T07:51:58.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.315+0000 7f4e2ca00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4e2815ddf0 0x7f4e280ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:58.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.315+0000 7f4e2d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4e280d28e0 0x7f4e2815f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:58.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.315+0000 7f4e27e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f4e2815e1d0 0x7f4e280d21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:44706/0 (socket says 172.21.15.89:44706) 2024-04-02T07:51:58.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.315+0000 7f4e2d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4e280d28e0 0x7f4e2815f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:47322/0 (socket says 172.21.15.89:47322) 2024-04-02T07:51:58.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.315+0000 7f4e27e00700 1 -- 172.21.15.89:0/542182708 learned_addr learned my addr 172.21.15.89:0/542182708 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:58.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.316+0000 7f4e2ca00700 1 -- 172.21.15.89:0/542182708 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4e280d28e0 msgr2=0x7f4e2815f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:58.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.316+0000 7f4e2ca00700 1 --2- 172.21.15.89:0/542182708 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4e280d28e0 0x7f4e2815f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:58.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.316+0000 7f4e2ca00700 1 -- 172.21.15.89:0/542182708 >> v1:172.21.15.113:6789/0 conn(0x7f4e2815e1d0 legacy=0x7f4e280d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:58.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.316+0000 7f4e2ca00700 1 -- 172.21.15.89:0/542182708 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e280cdf70 con 0x7f4e2815ddf0 2024-04-02T07:51:58.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.316+0000 7f4e2d400700 1 --2- 172.21.15.89:0/542182708 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4e280d28e0 0x7f4e2815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:51:58.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.316+0000 7f4e2ca00700 1 --2- 172.21.15.89:0/542182708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e2815ddf0 0x7f4e280ced30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009840 tx=0x7f4e1c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e15f6bbef9df40b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:58.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.316+0000 7f4e27400700 1 -- 172.21.15.89:0/542182708 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c011830 con 0x7f4e2815ddf0 2024-04-02T07:51:58.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.316+0000 7f4e27400700 1 -- 172.21.15.89:0/542182708 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e1c011990 con 0x7f4e2815ddf0 2024-04-02T07:51:58.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.316+0000 7f4e27400700 1 -- 172.21.15.89:0/542182708 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c011d20 con 0x7f4e2815ddf0 2024-04-02T07:51:58.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.317+0000 7f4e2f400700 1 -- 172.21.15.89:0/542182708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e2815ddf0 msgr2=0x7f4e280ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:58.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.317+0000 7f4e2f400700 1 --2- 172.21.15.89:0/542182708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e2815ddf0 0x7f4e280ced30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009840 tx=0x7f4e1c009b50 comp rx=0 tx=0).stop 2024-04-02T07:51:58.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.317+0000 7f4e2f400700 1 -- 172.21.15.89:0/542182708 shutdown_connections 2024-04-02T07:51:58.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.317+0000 7f4e2f400700 1 --2- 172.21.15.89:0/542182708 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4e280d28e0 0x7f4e2815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:58.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.317+0000 7f4e2f400700 1 --2- 172.21.15.89:0/542182708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e2815ddf0 0x7f4e280ced30 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:58.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.317+0000 7f4e2f400700 1 -- 172.21.15.89:0/542182708 >> 172.21.15.89:0/542182708 conn(0x7f4e2808aff0 msgr2=0x7f4e280d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:58.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.317+0000 7f4e2f400700 1 -- 172.21.15.89:0/542182708 shutdown_connections 2024-04-02T07:51:58.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.317+0000 7f4e2f400700 1 -- 172.21.15.89:0/542182708 wait complete. 2024-04-02T07:51:58.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.319+0000 7f4e2f400700 1 Processor -- start 2024-04-02T07:51:58.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.319+0000 7f4e2f400700 1 -- start start 2024-04-02T07:51:58.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.320+0000 7f4e2f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4e2815ddf0 0x7f4e280b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:58.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e2f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e2815e1d0 0x7f4e280cb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:58.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e2f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e280c9950 0x7f4e281f6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:58.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e2f400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280b2c40 con 0x7f4e280c9950 2024-04-02T07:51:58.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e2f400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e281f7680 con 0x7f4e2815e1d0 2024-04-02T07:51:58.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e2f400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e281f70c0 con 0x7f4e2815ddf0 2024-04-02T07:51:58.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e2d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e280c9950 0x7f4e281f6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:58.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e2d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e280c9950 0x7f4e281f6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43394/0 (socket says 172.21.15.89:43394) 2024-04-02T07:51:58.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e2d400700 1 -- 172.21.15.89:0/2648456555 learned_addr learned my addr 172.21.15.89:0/2648456555 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:58.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e2ca00700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4e2815ddf0 0x7f4e280b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:58.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e27e00700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e2815e1d0 0x7f4e280cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:58.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e2d400700 1 -- 172.21.15.89:0/2648456555 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4e2815ddf0 msgr2=0x7f4e280b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:58.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e2d400700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4e2815ddf0 0x7f4e280b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:58.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e2d400700 1 -- 172.21.15.89:0/2648456555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e2815e1d0 msgr2=0x7f4e280cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:58.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e2d400700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e2815e1d0 0x7f4e280cb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:58.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e2d400700 1 -- 172.21.15.89:0/2648456555 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e20009a70 con 0x7f4e280c9950 2024-04-02T07:51:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.321+0000 7f4e27e00700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e2815e1d0 0x7f4e280cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:51:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.322+0000 7f4e2d400700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e280c9950 0x7f4e281f6750 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000efb0 tx=0x7f4e2000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.322+0000 7f4e24c00700 1 -- 172.21.15.89:0/2648456555 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2000cd40 con 0x7f4e280c9950 2024-04-02T07:51:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.322+0000 7f4e24c00700 1 -- 172.21.15.89:0/2648456555 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2000f040 con 0x7f4e280c9950 2024-04-02T07:51:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.322+0000 7f4e24c00700 1 -- 172.21.15.89:0/2648456555 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2000cea0 con 0x7f4e280c9950 2024-04-02T07:51:58.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.322+0000 7f4e2f400700 1 -- 172.21.15.89:0/2648456555 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e1c0094a0 con 0x7f4e280c9950 2024-04-02T07:51:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.322+0000 7f4e2f400700 1 -- 172.21.15.89:0/2648456555 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e280b28d0 con 0x7f4e280c9950 2024-04-02T07:51:58.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.323+0000 7f4e2f400700 1 -- 172.21.15.89:0/2648456555 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e28067720 con 0x7f4e280c9950 2024-04-02T07:51:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.324+0000 7f4e24c00700 1 -- 172.21.15.89:0/2648456555 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 41) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f4e200146a0 con 0x7f4e280c9950 2024-04-02T07:51:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.325+0000 7f4e24c00700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4e0806ad20 0x7f4e0806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:58.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.325+0000 7f4e24c00700 1 -- 172.21.15.89:0/2648456555 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6307+0+0 (secure 0 0 0) 0x7f4e2008a500 con 0x7f4e280c9950 2024-04-02T07:51:58.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.325+0000 7f4e2ca00700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4e0806ad20 0x7f4e0806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:58.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.326+0000 7f4e2ca00700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4e0806ad20 0x7f4e0806d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c000c00 tx=0x7f4e1c008030 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:58.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.330+0000 7f4e24c00700 1 -- 172.21.15.89:0/2648456555 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4e20055290 con 0x7f4e280c9950 2024-04-02T07:51:58.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:58.531+0000 7f4e2f400700 1 -- 172.21.15.89:0/2648456555 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4e280b2d80 con 0x7f4e280c9950 2024-04-02T07:51:59.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.104+0000 7f4e24c00700 1 -- 172.21.15.89:0/2648456555 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4e20055660 con 0x7f4e280c9950 2024-04-02T07:51:59.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.105+0000 7f4e24c00700 1 -- 172.21.15.89:0/2648456555 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4e200518d0 con 0x7f4e280c9950 2024-04-02T07:51:59.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.110+0000 7f4e2f400700 1 -- 172.21.15.89:0/2648456555 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4e0806ad20 msgr2=0x7f4e0806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.110+0000 7f4e2f400700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4e0806ad20 0x7f4e0806d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c000c00 tx=0x7f4e1c008030 comp rx=0 tx=0).stop 2024-04-02T07:51:59.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.110+0000 7f4e2f400700 1 -- 172.21.15.89:0/2648456555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e280c9950 msgr2=0x7f4e281f6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.110+0000 7f4e2f400700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e280c9950 0x7f4e281f6750 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000efb0 tx=0x7f4e2000c3a0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.112+0000 7f4e2f400700 1 -- 172.21.15.89:0/2648456555 shutdown_connections 2024-04-02T07:51:59.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.112+0000 7f4e2f400700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4e0806ad20 0x7f4e0806d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.112+0000 7f4e2f400700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e280c9950 0x7f4e281f6750 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.112+0000 7f4e2f400700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4e2815ddf0 0x7f4e280b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.112+0000 7f4e2f400700 1 --2- 172.21.15.89:0/2648456555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e2815e1d0 0x7f4e280cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.112+0000 7f4e2f400700 1 -- 172.21.15.89:0/2648456555 >> 172.21.15.89:0/2648456555 conn(0x7f4e2808aff0 msgr2=0x7f4e280b18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:59.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.112+0000 7f4e2f400700 1 -- 172.21.15.89:0/2648456555 shutdown_connections 2024-04-02T07:51:59.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.113+0000 7f4e2f400700 1 -- 172.21.15.89:0/2648456555 wait complete. 2024-04-02T07:51:59.118 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T07:51:59.138 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T07:51:59.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.329+0000 7f40b6c00700 1 Processor -- start 2024-04-02T07:51:59.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.330+0000 7f40b6c00700 1 -- start start 2024-04-02T07:51:59.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.331+0000 7f40b6c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f40b015d800 0x7f40b00725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:59.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.331+0000 7f40b6c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b00709c0 con 0x7f40b00ce6a0 2024-04-02T07:51:59.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.332+0000 7f40b6c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b0070c80 con 0x7f40b00ce230 2024-04-02T07:51:59.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.332+0000 7f40b6c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b00b40f0 con 0x7f40b015d800 2024-04-02T07:51:59.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.332+0000 7f40afe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f40b015d800 0x7f40b00725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:59.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.332+0000 7f40b4c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f40b00ce6a0 0x7f40b015dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48910/0 (socket says 172.21.15.89:48910) 2024-04-02T07:51:59.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.332+0000 7f40b4c00700 1 -- 172.21.15.89:0/2560586916 learned_addr learned my addr 172.21.15.89:0/2560586916 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:51:59.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.332+0000 7f40afe00700 1 -- 172.21.15.89:0/2560586916 >> v1:172.21.15.113:6789/0 conn(0x7f40b00ce230 legacy=0x7f40b00b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.332+0000 7f40afe00700 1 -- 172.21.15.89:0/2560586916 >> v1:172.21.15.89:6789/0 conn(0x7f40b00ce6a0 legacy=0x7f40b015dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.332+0000 7f40afe00700 1 -- 172.21.15.89:0/2560586916 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b0070f60 con 0x7f40b015d800 2024-04-02T07:51:59.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.332+0000 7f40aea00700 1 -- 172.21.15.89:0/2560586916 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882631608 0 0) 0x7f40b00709c0 con 0x7f40b00ce6a0 2024-04-02T07:51:59.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.332+0000 7f40afe00700 1 --2- 172.21.15.89:0/2560586916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40b015d800 0x7f40b00725c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f40a000bae0 tx=0x7f40a000bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7ea664d5c4354a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:59.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.333+0000 7f40aea00700 1 -- 172.21.15.89:0/2560586916 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a000dd30 con 0x7f40b015d800 2024-04-02T07:51:59.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.333+0000 7f40aea00700 1 -- 172.21.15.89:0/2560586916 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a000de90 con 0x7f40b015d800 2024-04-02T07:51:59.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.333+0000 7f40aea00700 1 -- 172.21.15.89:0/2560586916 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a0010690 con 0x7f40b015d800 2024-04-02T07:51:59.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.333+0000 7f40b6c00700 1 -- 172.21.15.89:0/2560586916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40b015d800 msgr2=0x7f40b00725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.333+0000 7f40b6c00700 1 --2- 172.21.15.89:0/2560586916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40b015d800 0x7f40b00725c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f40a000bae0 tx=0x7f40a000bdf0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.333+0000 7f40b6c00700 1 -- 172.21.15.89:0/2560586916 shutdown_connections 2024-04-02T07:51:59.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.333+0000 7f40b6c00700 1 --2- 172.21.15.89:0/2560586916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40b015d800 0x7f40b00725c0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.333+0000 7f40b6c00700 1 -- 172.21.15.89:0/2560586916 >> 172.21.15.89:0/2560586916 conn(0x7f40b00b0d70 msgr2=0x7f40b00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:59.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.333+0000 7f40b6c00700 1 -- 172.21.15.89:0/2560586916 shutdown_connections 2024-04-02T07:51:59.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.333+0000 7f40b6c00700 1 -- 172.21.15.89:0/2560586916 wait complete. 2024-04-02T07:51:59.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.335+0000 7f40b6c00700 1 Processor -- start 2024-04-02T07:51:59.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.335+0000 7f40b6c00700 1 -- start start 2024-04-02T07:51:59.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.336+0000 7f40b6c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40b00ce230 0x7f40b01579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:59.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.336+0000 7f40b6c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40b00ce6a0 0x7f40b0158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:59.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.336+0000 7f40b6c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40b015d800 0x7f40b01eee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:59.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.336+0000 7f40b6c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b0159380 con 0x7f40b00ce230 2024-04-02T07:51:59.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.336+0000 7f40b6c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b0157f10 con 0x7f40b015d800 2024-04-02T07:51:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.336+0000 7f40b6c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b00ac180 con 0x7f40b00ce6a0 2024-04-02T07:51:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.336+0000 7f40afe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40b00ce230 0x7f40b01579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.336+0000 7f40afe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40b00ce230 0x7f40b01579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52996/0 (socket says 172.21.15.89:52996) 2024-04-02T07:51:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.336+0000 7f40afe00700 1 -- 172.21.15.89:0/136137225 learned_addr learned my addr 172.21.15.89:0/136137225 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.336+0000 7f40b4c00700 1 --2- 172.21.15.89:0/136137225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40b015d800 0x7f40b01eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.336+0000 7f40af400700 1 --2- 172.21.15.89:0/136137225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40b00ce6a0 0x7f40b0158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.337+0000 7f40afe00700 1 -- 172.21.15.89:0/136137225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40b00ce6a0 msgr2=0x7f40b0158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.337+0000 7f40afe00700 1 --2- 172.21.15.89:0/136137225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40b00ce6a0 0x7f40b0158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.337+0000 7f40afe00700 1 -- 172.21.15.89:0/136137225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40b015d800 msgr2=0x7f40b01eee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.337+0000 7f40afe00700 1 --2- 172.21.15.89:0/136137225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40b015d800 0x7f40b01eee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.337+0000 7f40afe00700 1 -- 172.21.15.89:0/136137225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a000b740 con 0x7f40b00ce230 2024-04-02T07:51:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.337+0000 7f40afe00700 1 --2- 172.21.15.89:0/136137225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40b00ce230 0x7f40b01579d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f40a000bf90 tx=0x7f40a00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.337+0000 7f40a7e00700 1 -- 172.21.15.89:0/136137225 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a001e040 con 0x7f40b00ce230 2024-04-02T07:51:59.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.337+0000 7f40a7e00700 1 -- 172.21.15.89:0/136137225 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a0019600 con 0x7f40b00ce230 2024-04-02T07:51:59.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.337+0000 7f40a7e00700 1 -- 172.21.15.89:0/136137225 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a0019970 con 0x7f40b00ce230 2024-04-02T07:51:59.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.337+0000 7f40b6c00700 1 -- 172.21.15.89:0/136137225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b00ac460 con 0x7f40b00ce230 2024-04-02T07:51:59.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.337+0000 7f40b6c00700 1 -- 172.21.15.89:0/136137225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b00aca20 con 0x7f40b00ce230 2024-04-02T07:51:59.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.339+0000 7f40b6c00700 1 -- 172.21.15.89:0/136137225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b0067720 con 0x7f40b00ce230 2024-04-02T07:51:59.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.339+0000 7f40a7e00700 1 -- 172.21.15.89:0/136137225 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f40a0019ce0 con 0x7f40b00ce230 2024-04-02T07:51:59.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.340+0000 7f40a7e00700 1 --2- 172.21.15.89:0/136137225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f409006adb0 0x7f409006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:59.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.340+0000 7f40a7e00700 1 -- 172.21.15.89:0/136137225 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f40a008d620 con 0x7f40b00ce230 2024-04-02T07:51:59.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.341+0000 7f40af400700 1 --2- 172.21.15.89:0/136137225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f409006adb0 0x7f409006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:59.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.341+0000 7f40af400700 1 --2- 172.21.15.89:0/136137225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f409006adb0 0x7f409006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f40b01f2360 tx=0x7f409c009040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:59.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.345+0000 7f40a7e00700 1 -- 172.21.15.89:0/136137225 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f40a00585b0 con 0x7f40b00ce230 2024-04-02T07:51:59.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.547+0000 7f40b6c00700 1 -- 172.21.15.89:0/136137225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f40b0072580 con 0x7f40b00ce230 2024-04-02T07:51:59.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.551+0000 7f40a7e00700 1 -- 172.21.15.89:0/136137225 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v25) v1 ==== 115+0+0 (secure 0 0 0) 0x7f40a005bbd0 con 0x7f40b00ce230 2024-04-02T07:51:59.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.554+0000 7f40b6c00700 1 -- 172.21.15.89:0/136137225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f409006adb0 msgr2=0x7f409006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.554+0000 7f40b6c00700 1 --2- 172.21.15.89:0/136137225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f409006adb0 0x7f409006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f40b01f2360 tx=0x7f409c009040 comp rx=0 tx=0).stop 2024-04-02T07:51:59.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.554+0000 7f40b6c00700 1 -- 172.21.15.89:0/136137225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40b00ce230 msgr2=0x7f40b01579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.554+0000 7f40b6c00700 1 --2- 172.21.15.89:0/136137225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40b00ce230 0x7f40b01579d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f40a000bf90 tx=0x7f40a00047c0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.554+0000 7f40b6c00700 1 -- 172.21.15.89:0/136137225 shutdown_connections 2024-04-02T07:51:59.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.554+0000 7f40b6c00700 1 --2- 172.21.15.89:0/136137225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f409006adb0 0x7f409006d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.554+0000 7f40b6c00700 1 --2- 172.21.15.89:0/136137225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40b015d800 0x7f40b01eee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.554+0000 7f40b6c00700 1 --2- 172.21.15.89:0/136137225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40b00ce230 0x7f40b01579d0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.554+0000 7f40b6c00700 1 --2- 172.21.15.89:0/136137225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40b00ce6a0 0x7f40b0158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.554+0000 7f40b6c00700 1 -- 172.21.15.89:0/136137225 >> 172.21.15.89:0/136137225 conn(0x7f40b00b0d70 msgr2=0x7f40b00af760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:59.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.554+0000 7f40b6c00700 1 -- 172.21.15.89:0/136137225 shutdown_connections 2024-04-02T07:51:59.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.554+0000 7f40b6c00700 1 -- 172.21.15.89:0/136137225 wait complete. 2024-04-02T07:51:59.574 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T07:51:59.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.765+0000 7fb6c2e00700 1 Processor -- start 2024-04-02T07:51:59.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.765+0000 7fb6c2e00700 1 -- start start 2024-04-02T07:51:59.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6c2e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb6bc15d9d0 0x7fb6bc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6c2e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb6bc0d2540 0x7fb6bc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6c2e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6bc161420 con 0x7fb6bc0d16a0 2024-04-02T07:51:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6c2e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6bc15edc0 con 0x7fb6bc15d9d0 2024-04-02T07:51:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6c2e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6bc161700 con 0x7fb6bc0d2540 2024-04-02T07:51:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6bbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb6bc15d9d0 0x7fb6bc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6bbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb6bc15d9d0 0x7fb6bc0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53004/0 (socket says 172.21.15.89:53004) 2024-04-02T07:51:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6bb400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb6bc0d16a0 0x7fb6bc0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48926/0 (socket says 172.21.15.89:48926) 2024-04-02T07:51:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6bbe00700 1 -- 172.21.15.89:0/1246649527 learned_addr learned my addr 172.21.15.89:0/1246649527 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6c0e00700 1 --2- 172.21.15.89:0/1246649527 >> v2:172.21.15.113:3300/0 conn(0x7fb6bc0d2540 0x7fb6bc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6bbe00700 1 -- 172.21.15.89:0/1246649527 >> v2:172.21.15.113:3300/0 conn(0x7fb6bc0d2540 msgr2=0x7fb6bc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:51:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6bbe00700 1 --2- 172.21.15.89:0/1246649527 >> v2:172.21.15.113:3300/0 conn(0x7fb6bc0d2540 0x7fb6bc0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6bbe00700 1 -- 172.21.15.89:0/1246649527 >> v1:172.21.15.89:6789/0 conn(0x7fb6bc0d16a0 legacy=0x7fb6bc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6bbe00700 1 -- 172.21.15.89:0/1246649527 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc1619e0 con 0x7fb6bc15d9d0 2024-04-02T07:51:59.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.767+0000 7fb6baa00700 1 -- 172.21.15.89:0/1246649527 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340375371 0 0) 0x7fb6bc161420 con 0x7fb6bc0d16a0 2024-04-02T07:51:59.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.768+0000 7fb6bbe00700 1 --2- 172.21.15.89:0/1246649527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6bc15d9d0 0x7fb6bc0d3c30 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc08ca10 tx=0x7fb6ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532bbb766b932faa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:59.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.768+0000 7fb6baa00700 1 -- 172.21.15.89:0/1246649527 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac00b870 con 0x7fb6bc15d9d0 2024-04-02T07:51:59.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.768+0000 7fb6baa00700 1 -- 172.21.15.89:0/1246649527 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ac00b9d0 con 0x7fb6bc15d9d0 2024-04-02T07:51:59.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.768+0000 7fb6baa00700 1 -- 172.21.15.89:0/1246649527 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac00bd60 con 0x7fb6bc15d9d0 2024-04-02T07:51:59.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.768+0000 7fb6c2e00700 1 -- 172.21.15.89:0/1246649527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6bc15d9d0 msgr2=0x7fb6bc0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.768+0000 7fb6c2e00700 1 --2- 172.21.15.89:0/1246649527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6bc15d9d0 0x7fb6bc0d3c30 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc08ca10 tx=0x7fb6ac009bd0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.768+0000 7fb6c2e00700 1 -- 172.21.15.89:0/1246649527 shutdown_connections 2024-04-02T07:51:59.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.768+0000 7fb6c2e00700 1 --2- 172.21.15.89:0/1246649527 >> v2:172.21.15.113:3300/0 conn(0x7fb6bc0d2540 0x7fb6bc0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.768+0000 7fb6c2e00700 1 --2- 172.21.15.89:0/1246649527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6bc15d9d0 0x7fb6bc0d3c30 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.768+0000 7fb6c2e00700 1 -- 172.21.15.89:0/1246649527 >> 172.21.15.89:0/1246649527 conn(0x7fb6bc08aff0 msgr2=0x7fb6bc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:59.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.769+0000 7fb6c2e00700 1 -- 172.21.15.89:0/1246649527 shutdown_connections 2024-04-02T07:51:59.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.769+0000 7fb6c2e00700 1 -- 172.21.15.89:0/1246649527 wait complete. 2024-04-02T07:51:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.770+0000 7fb6c2e00700 1 Processor -- start 2024-04-02T07:51:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.770+0000 7fb6c2e00700 1 -- start start 2024-04-02T07:51:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.771+0000 7fb6c2e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6bc0d16a0 0x7fb6bc14bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.771+0000 7fb6c2e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6bc0d2540 0x7fb6bc0cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.771+0000 7fb6c2e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6bc15d9d0 0x7fb6bc0cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.771+0000 7fb6c2e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc0cb1e0 con 0x7fb6bc0d2540 2024-04-02T07:51:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.771+0000 7fb6c2e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc0ca5f0 con 0x7fb6bc0d16a0 2024-04-02T07:51:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.771+0000 7fb6c2e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc14c180 con 0x7fb6bc15d9d0 2024-04-02T07:51:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.771+0000 7fb6bb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6bc0d2540 0x7fb6bc0cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.771+0000 7fb6bb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6bc0d2540 0x7fb6bc0cba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53008/0 (socket says 172.21.15.89:53008) 2024-04-02T07:51:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.771+0000 7fb6bb400700 1 -- 172.21.15.89:0/4101982 learned_addr learned my addr 172.21.15.89:0/4101982 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:51:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.772+0000 7fb6bbe00700 1 --2- 172.21.15.89:0/4101982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6bc0d16a0 0x7fb6bc14bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.772+0000 7fb6c0e00700 1 --2- 172.21.15.89:0/4101982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6bc15d9d0 0x7fb6bc0cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.772+0000 7fb6bb400700 1 -- 172.21.15.89:0/4101982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6bc15d9d0 msgr2=0x7fb6bc0cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.772+0000 7fb6bb400700 1 --2- 172.21.15.89:0/4101982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6bc15d9d0 0x7fb6bc0cbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.772+0000 7fb6bb400700 1 -- 172.21.15.89:0/4101982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6bc0d16a0 msgr2=0x7fb6bc14bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.772+0000 7fb6bb400700 1 --2- 172.21.15.89:0/4101982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6bc0d16a0 0x7fb6bc14bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.772+0000 7fb6bb400700 1 -- 172.21.15.89:0/4101982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac009550 con 0x7fb6bc0d2540 2024-04-02T07:51:59.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.772+0000 7fb6bb400700 1 --2- 172.21.15.89:0/4101982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6bc0d2540 0x7fb6bc0cba60 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800c520 tx=0x7fb6a800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:59.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.772+0000 7fb6b3e00700 1 -- 172.21.15.89:0/4101982 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a80074e0 con 0x7fb6bc0d2540 2024-04-02T07:51:59.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.772+0000 7fb6b3e00700 1 -- 172.21.15.89:0/4101982 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a8007660 con 0x7fb6bc0d2540 2024-04-02T07:51:59.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.772+0000 7fb6c2e00700 1 -- 172.21.15.89:0/4101982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6bc0ca840 con 0x7fb6bc0d2540 2024-04-02T07:51:59.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.772+0000 7fb6b3e00700 1 -- 172.21.15.89:0/4101982 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8005a70 con 0x7fb6bc0d2540 2024-04-02T07:51:59.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.772+0000 7fb6c2e00700 1 -- 172.21.15.89:0/4101982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6bc0cae00 con 0x7fb6bc0d2540 2024-04-02T07:51:59.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.773+0000 7fb6c2e00700 1 -- 172.21.15.89:0/4101982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6bc0d1ca0 con 0x7fb6bc0d2540 2024-04-02T07:51:59.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.774+0000 7fb6b3e00700 1 -- 172.21.15.89:0/4101982 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb6a8005bd0 con 0x7fb6bc0d2540 2024-04-02T07:51:59.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.775+0000 7fb6b3e00700 1 --2- 172.21.15.89:0/4101982 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb69c06adb0 0x7fb69c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:51:59.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.775+0000 7fb6b3e00700 1 -- 172.21.15.89:0/4101982 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5893+0+0 (secure 0 0 0) 0x7fb6a8088a60 con 0x7fb6bc0d2540 2024-04-02T07:51:59.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.775+0000 7fb6bbe00700 1 --2- 172.21.15.89:0/4101982 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb69c06adb0 0x7fb69c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:51:59.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.776+0000 7fb6bbe00700 1 --2- 172.21.15.89:0/4101982 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb69c06adb0 0x7fb69c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac014040 tx=0x7fb6ac008030 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:51:59.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.780+0000 7fb6b3e00700 1 -- 172.21.15.89:0/4101982 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb6a8053a60 con 0x7fb6bc0d2540 2024-04-02T07:51:59.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.982+0000 7fb6c2e00700 1 -- 172.21.15.89:0/4101982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fb6bc0cb320 con 0x7fb6bc0d2540 2024-04-02T07:51:59.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.986+0000 7fb6b3e00700 1 -- 172.21.15.89:0/4101982 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v26) v1 ==== 120+0+0 (secure 0 0 0) 0x7fb6bc0cb320 con 0x7fb6bc0d2540 2024-04-02T07:51:59.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.989+0000 7fb6c2e00700 1 -- 172.21.15.89:0/4101982 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb69c06adb0 msgr2=0x7fb69c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.989+0000 7fb6c2e00700 1 --2- 172.21.15.89:0/4101982 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb69c06adb0 0x7fb69c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac014040 tx=0x7fb6ac008030 comp rx=0 tx=0).stop 2024-04-02T07:51:59.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.989+0000 7fb6c2e00700 1 -- 172.21.15.89:0/4101982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6bc0d2540 msgr2=0x7fb6bc0cba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:51:59.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.989+0000 7fb6c2e00700 1 --2- 172.21.15.89:0/4101982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6bc0d2540 0x7fb6bc0cba60 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800c520 tx=0x7fb6a800c830 comp rx=0 tx=0).stop 2024-04-02T07:51:59.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.989+0000 7fb6c2e00700 1 -- 172.21.15.89:0/4101982 shutdown_connections 2024-04-02T07:51:59.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.989+0000 7fb6c2e00700 1 --2- 172.21.15.89:0/4101982 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb69c06adb0 0x7fb69c06d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.989+0000 7fb6c2e00700 1 --2- 172.21.15.89:0/4101982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6bc0d16a0 0x7fb6bc14bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.989+0000 7fb6c2e00700 1 --2- 172.21.15.89:0/4101982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6bc15d9d0 0x7fb6bc0cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.989+0000 7fb6c2e00700 1 --2- 172.21.15.89:0/4101982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6bc0d2540 0x7fb6bc0cba60 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:51:59.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.989+0000 7fb6c2e00700 1 -- 172.21.15.89:0/4101982 >> 172.21.15.89:0/4101982 conn(0x7fb6bc08aff0 msgr2=0x7fb6bc15ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:51:59.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.989+0000 7fb6c2e00700 1 -- 172.21.15.89:0/4101982 shutdown_connections 2024-04-02T07:51:59.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:51:59.989+0000 7fb6c2e00700 1 -- 172.21.15.89:0/4101982 wait complete. 2024-04-02T07:52:00.009 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T07:52:00.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.202+0000 7fedc8200700 1 Processor -- start 2024-04-02T07:52:00.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.202+0000 7fedc8200700 1 -- start start 2024-04-02T07:52:00.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.204+0000 7fedc8200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedc015d9d0 0x7fedc00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:00.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.204+0000 7fedc8200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fedc00d2540 0x7fedc00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:00.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.204+0000 7fedc8200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc0161420 con 0x7fedc00d16a0 2024-04-02T07:52:00.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.204+0000 7fedc8200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedc015edc0 con 0x7fedc00d2540 2024-04-02T07:52:00.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.204+0000 7fedc8200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc0161700 con 0x7fedc015d9d0 2024-04-02T07:52:00.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.204+0000 7fedc6200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fedc00d2540 0x7fedc00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:00.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.204+0000 7fedc4e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fedc00d16a0 0x7fedc00d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48930/0 (socket says 172.21.15.89:48930) 2024-04-02T07:52:00.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.204+0000 7fedc4e00700 1 -- 172.21.15.89:0/3093030233 learned_addr learned my addr 172.21.15.89:0/3093030233 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:00.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.204+0000 7fedc5800700 1 --2- 172.21.15.89:0/3093030233 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedc015d9d0 0x7fedc00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:00.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.204+0000 7fedbfe00700 1 -- 172.21.15.89:0/3093030233 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354141116 0 0) 0x7fedc0161420 con 0x7fedc00d16a0 2024-04-02T07:52:00.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.205+0000 7fedbfe00700 1 -- 172.21.15.89:0/3093030233 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda80035b0 con 0x7fedc00d16a0 2024-04-02T07:52:00.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.205+0000 7fedc6200700 1 -- 172.21.15.89:0/3093030233 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedc015d9d0 msgr2=0x7fedc00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:00.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.205+0000 7fedc6200700 1 --2- 172.21.15.89:0/3093030233 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedc015d9d0 0x7fedc00d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:00.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.205+0000 7fedc6200700 1 -- 172.21.15.89:0/3093030233 >> v1:172.21.15.89:6789/0 conn(0x7fedc00d16a0 legacy=0x7fedc00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:00.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.205+0000 7fedc6200700 1 -- 172.21.15.89:0/3093030233 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc01619e0 con 0x7fedc00d2540 2024-04-02T07:52:00.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.205+0000 7fedc6200700 1 --2- 172.21.15.89:0/3093030233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedc00d2540 0x7fedc00ce360 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fedb8009dc0 tx=0x7fedb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f7fa012a9e2404 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:00.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.205+0000 7fedbfe00700 1 -- 172.21.15.89:0/3093030233 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb800e070 con 0x7fedc00d2540 2024-04-02T07:52:00.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.205+0000 7fedbfe00700 1 -- 172.21.15.89:0/3093030233 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb800bae0 con 0x7fedc00d2540 2024-04-02T07:52:00.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.205+0000 7fedbfe00700 1 -- 172.21.15.89:0/3093030233 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb800be90 con 0x7fedc00d2540 2024-04-02T07:52:00.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.206+0000 7fedc8200700 1 -- 172.21.15.89:0/3093030233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedc00d2540 msgr2=0x7fedc00ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:00.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.206+0000 7fedc8200700 1 --2- 172.21.15.89:0/3093030233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedc00d2540 0x7fedc00ce360 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fedb8009dc0 tx=0x7fedb800b330 comp rx=0 tx=0).stop 2024-04-02T07:52:00.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.206+0000 7fedc8200700 1 -- 172.21.15.89:0/3093030233 shutdown_connections 2024-04-02T07:52:00.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.206+0000 7fedc8200700 1 --2- 172.21.15.89:0/3093030233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedc00d2540 0x7fedc00ce360 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:00.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.206+0000 7fedc8200700 1 --2- 172.21.15.89:0/3093030233 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedc015d9d0 0x7fedc00d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:00.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.206+0000 7fedc8200700 1 -- 172.21.15.89:0/3093030233 >> 172.21.15.89:0/3093030233 conn(0x7fedc008aff0 msgr2=0x7fedc00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:00.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.206+0000 7fedc8200700 1 -- 172.21.15.89:0/3093030233 shutdown_connections 2024-04-02T07:52:00.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.206+0000 7fedc8200700 1 -- 172.21.15.89:0/3093030233 wait complete. 2024-04-02T07:52:00.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.208+0000 7fedc8200700 1 Processor -- start 2024-04-02T07:52:00.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.208+0000 7fedc8200700 1 -- start start 2024-04-02T07:52:00.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.209+0000 7fedc8200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedc00d16a0 0x7fedc0151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:00.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.209+0000 7fedc8200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedc00d2540 0x7fedc015f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:00.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.209+0000 7fedc8200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedc015d9d0 0x7fedc015fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:00.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.209+0000 7fedc8200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc00b74c0 con 0x7fedc00d16a0 2024-04-02T07:52:00.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.209+0000 7fedc8200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc00b00d0 con 0x7fedc015d9d0 2024-04-02T07:52:00.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.209+0000 7fedc8200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc00b03b0 con 0x7fedc00d2540 2024-04-02T07:52:00.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.209+0000 7fedc5800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedc00d16a0 0x7fedc0151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:00.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.209+0000 7fedc5800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedc00d16a0 0x7fedc0151d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53016/0 (socket says 172.21.15.89:53016) 2024-04-02T07:52:00.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.209+0000 7fedc5800700 1 -- 172.21.15.89:0/2026048356 learned_addr learned my addr 172.21.15.89:0/2026048356 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:00.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.209+0000 7fedc4e00700 1 --2- 172.21.15.89:0/2026048356 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedc00d2540 0x7fedc015f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:00.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.209+0000 7fedc6200700 1 --2- 172.21.15.89:0/2026048356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedc015d9d0 0x7fedc015fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:00.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.210+0000 7fedc5800700 1 -- 172.21.15.89:0/2026048356 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedc00d2540 msgr2=0x7fedc015f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:00.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.210+0000 7fedc5800700 1 --2- 172.21.15.89:0/2026048356 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedc00d2540 0x7fedc015f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:00.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.210+0000 7fedc5800700 1 -- 172.21.15.89:0/2026048356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedc015d9d0 msgr2=0x7fedc015fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:00.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.210+0000 7fedc5800700 1 --2- 172.21.15.89:0/2026048356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedc015d9d0 0x7fedc015fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:00.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.210+0000 7fedc5800700 1 -- 172.21.15.89:0/2026048356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb8009a70 con 0x7fedc00d16a0 2024-04-02T07:52:00.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.210+0000 7fedc5800700 1 --2- 172.21.15.89:0/2026048356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedc00d16a0 0x7fedc0151d30 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fedb400e870 tx=0x7fedb400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:00.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.210+0000 7fedbd600700 1 -- 172.21.15.89:0/2026048356 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb4009590 con 0x7fedc00d16a0 2024-04-02T07:52:00.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.210+0000 7fedbd600700 1 -- 172.21.15.89:0/2026048356 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb4013680 con 0x7fedc00d16a0 2024-04-02T07:52:00.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.210+0000 7fedbd600700 1 -- 172.21.15.89:0/2026048356 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb40139f0 con 0x7fedc00d16a0 2024-04-02T07:52:00.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.210+0000 7fedc8200700 1 -- 172.21.15.89:0/2026048356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc00b0690 con 0x7fedc00d16a0 2024-04-02T07:52:00.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.210+0000 7fedc8200700 1 -- 172.21.15.89:0/2026048356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc0152270 con 0x7fedc00d16a0 2024-04-02T07:52:00.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.211+0000 7fedc8200700 1 -- 172.21.15.89:0/2026048356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc00b0ce0 con 0x7fedc00d16a0 2024-04-02T07:52:00.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.212+0000 7fedbd600700 1 -- 172.21.15.89:0/2026048356 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fedb4013b50 con 0x7fedc00d16a0 2024-04-02T07:52:00.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.213+0000 7fedbd600700 1 --2- 172.21.15.89:0/2026048356 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feda006adb0 0x7feda006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:00.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.214+0000 7fedbd600700 1 -- 172.21.15.89:0/2026048356 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5893+0+0 (secure 0 0 0) 0x7fedb40893c0 con 0x7fedc00d16a0 2024-04-02T07:52:00.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.214+0000 7fedc4e00700 1 --2- 172.21.15.89:0/2026048356 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feda006adb0 0x7feda006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:00.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.214+0000 7fedc4e00700 1 --2- 172.21.15.89:0/2026048356 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feda006adb0 0x7feda006d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fedc00b9410 tx=0x7fedac007c70 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:00.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.218+0000 7fedbd600700 1 -- 172.21.15.89:0/2026048356 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fedb40543c0 con 0x7fedc00d16a0 2024-04-02T07:52:00.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.420+0000 7fedc8200700 1 -- 172.21.15.89:0/2026048356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fedc00d1ca0 con 0x7fedc00d16a0 2024-04-02T07:52:00.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.426+0000 7fedbd600700 1 -- 172.21.15.89:0/2026048356 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7fedb40579e0 con 0x7fedc00d16a0 2024-04-02T07:52:00.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.429+0000 7fedc8200700 1 -- 172.21.15.89:0/2026048356 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feda006adb0 msgr2=0x7feda006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.429+0000 7fedc8200700 1 --2- 172.21.15.89:0/2026048356 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feda006adb0 0x7feda006d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fedc00b9410 tx=0x7fedac007c70 comp rx=0 tx=0).stop 2024-04-02T07:52:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.429+0000 7fedc8200700 1 -- 172.21.15.89:0/2026048356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedc00d16a0 msgr2=0x7fedc0151d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.429+0000 7fedc8200700 1 --2- 172.21.15.89:0/2026048356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedc00d16a0 0x7fedc0151d30 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fedb400e870 tx=0x7fedb400eb80 comp rx=0 tx=0).stop 2024-04-02T07:52:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.429+0000 7fedc8200700 1 -- 172.21.15.89:0/2026048356 shutdown_connections 2024-04-02T07:52:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.429+0000 7fedc8200700 1 --2- 172.21.15.89:0/2026048356 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feda006adb0 0x7feda006d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.429+0000 7fedc8200700 1 --2- 172.21.15.89:0/2026048356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedc015d9d0 0x7fedc015fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.429+0000 7fedc8200700 1 --2- 172.21.15.89:0/2026048356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedc00d16a0 0x7fedc0151d30 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.429+0000 7fedc8200700 1 --2- 172.21.15.89:0/2026048356 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedc00d2540 0x7fedc015f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.429+0000 7fedc8200700 1 -- 172.21.15.89:0/2026048356 >> 172.21.15.89:0/2026048356 conn(0x7fedc008aff0 msgr2=0x7fedc00b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.429+0000 7fedc8200700 1 -- 172.21.15.89:0/2026048356 shutdown_connections 2024-04-02T07:52:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.429+0000 7fedc8200700 1 -- 172.21.15.89:0/2026048356 wait complete. 2024-04-02T07:52:00.449 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-04-02T07:52:00.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.637+0000 7f64efa00700 1 Processor -- start 2024-04-02T07:52:00.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.638+0000 7f64efa00700 1 -- start start 2024-04-02T07:52:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.639+0000 7f64efa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f64e8159b40 0x7f64e8072810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.639+0000 7f64efa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64e80b2f30 0x7f64e80b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.639+0000 7f64efa00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e8159f20 con 0x7f64e806d2e0 2024-04-02T07:52:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.639+0000 7f64efa00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e80b38f0 con 0x7f64e8159b40 2024-04-02T07:52:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.639+0000 7f64efa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e8072d50 con 0x7f64e80b2f30 2024-04-02T07:52:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.640+0000 7f64ed000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f64e8159b40 0x7f64e8072810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.640+0000 7f64eda00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f64e806d2e0 0x7f64e80ac5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48946/0 (socket says 172.21.15.89:48946) 2024-04-02T07:52:00.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.640+0000 7f64eda00700 1 -- 172.21.15.89:0/1354012902 learned_addr learned my addr 172.21.15.89:0/1354012902 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:00.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.640+0000 7f64e7e00700 1 --2- 172.21.15.89:0/1354012902 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64e80b2f30 0x7f64e80b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.640+0000 7f64ed000700 1 -- 172.21.15.89:0/1354012902 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64e80b2f30 msgr2=0x7f64e80b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.640+0000 7f64ed000700 1 --2- 172.21.15.89:0/1354012902 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64e80b2f30 0x7f64e80b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.640+0000 7f64ed000700 1 -- 172.21.15.89:0/1354012902 >> v1:172.21.15.89:6789/0 conn(0x7f64e806d2e0 legacy=0x7f64e80ac5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.640+0000 7f64ed000700 1 -- 172.21.15.89:0/1354012902 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e8071770 con 0x7f64e8159b40 2024-04-02T07:52:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.640+0000 7f64e7400700 1 -- 172.21.15.89:0/1354012902 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424851057 0 0) 0x7f64e8159f20 con 0x7f64e806d2e0 2024-04-02T07:52:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.640+0000 7f64ed000700 1 --2- 172.21.15.89:0/1354012902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64e8159b40 0x7f64e8072810 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f64dc009840 tx=0x7f64dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e3390311dba9ed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.641+0000 7f64e7400700 1 -- 172.21.15.89:0/1354012902 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc011830 con 0x7f64e8159b40 2024-04-02T07:52:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.641+0000 7f64e7400700 1 -- 172.21.15.89:0/1354012902 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc011990 con 0x7f64e8159b40 2024-04-02T07:52:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.641+0000 7f64e7400700 1 -- 172.21.15.89:0/1354012902 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc011d20 con 0x7f64e8159b40 2024-04-02T07:52:00.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.641+0000 7f64efa00700 1 -- 172.21.15.89:0/1354012902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64e8159b40 msgr2=0x7f64e8072810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:00.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.641+0000 7f64efa00700 1 --2- 172.21.15.89:0/1354012902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64e8159b40 0x7f64e8072810 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f64dc009840 tx=0x7f64dc009b50 comp rx=0 tx=0).stop 2024-04-02T07:52:00.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.641+0000 7f64efa00700 1 -- 172.21.15.89:0/1354012902 shutdown_connections 2024-04-02T07:52:00.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.641+0000 7f64efa00700 1 --2- 172.21.15.89:0/1354012902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64e8159b40 0x7f64e8072810 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:00.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.641+0000 7f64efa00700 1 --2- 172.21.15.89:0/1354012902 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64e80b2f30 0x7f64e80b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:00.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.641+0000 7f64efa00700 1 -- 172.21.15.89:0/1354012902 >> 172.21.15.89:0/1354012902 conn(0x7f64e80b0ef0 msgr2=0x7f64e80b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:00.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.641+0000 7f64efa00700 1 -- 172.21.15.89:0/1354012902 shutdown_connections 2024-04-02T07:52:00.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.641+0000 7f64efa00700 1 -- 172.21.15.89:0/1354012902 wait complete. 2024-04-02T07:52:00.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.643+0000 7f64efa00700 1 Processor -- start 2024-04-02T07:52:00.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.643+0000 7f64efa00700 1 -- start start 2024-04-02T07:52:00.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.644+0000 7f64efa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64e806d2e0 0x7f64e8159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.644+0000 7f64efa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64e80b2f30 0x7f64e8155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.644+0000 7f64efa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64e8159b40 0x7f64e8148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.644+0000 7f64efa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e8155a20 con 0x7f64e80b2f30 2024-04-02T07:52:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.644+0000 7f64efa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e815bce0 con 0x7f64e8159b40 2024-04-02T07:52:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.644+0000 7f64efa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e815bfc0 con 0x7f64e806d2e0 2024-04-02T07:52:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.644+0000 7f64e7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64e80b2f30 0x7f64e8155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.644+0000 7f64e7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64e80b2f30 0x7f64e8155e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53034/0 (socket says 172.21.15.89:53034) 2024-04-02T07:52:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.644+0000 7f64e7e00700 1 -- 172.21.15.89:0/3339633502 learned_addr learned my addr 172.21.15.89:0/3339633502 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.644+0000 7f64ed000700 1 --2- 172.21.15.89:0/3339633502 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64e806d2e0 0x7f64e8159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.645+0000 7f64eda00700 1 --2- 172.21.15.89:0/3339633502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64e8159b40 0x7f64e8148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.645+0000 7f64e7e00700 1 -- 172.21.15.89:0/3339633502 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64e806d2e0 msgr2=0x7f64e8159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:00.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.645+0000 7f64e7e00700 1 --2- 172.21.15.89:0/3339633502 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64e806d2e0 0x7f64e8159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:00.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.645+0000 7f64e7e00700 1 -- 172.21.15.89:0/3339633502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64e8159b40 msgr2=0x7f64e8148440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:00.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.645+0000 7f64e7e00700 1 --2- 172.21.15.89:0/3339633502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64e8159b40 0x7f64e8148440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:00.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.645+0000 7f64e7e00700 1 -- 172.21.15.89:0/3339633502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64dc0094a0 con 0x7f64e80b2f30 2024-04-02T07:52:00.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.645+0000 7f64e7e00700 1 --2- 172.21.15.89:0/3339633502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64e80b2f30 0x7f64e8155e60 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f64d400e810 tx=0x7f64d400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:00.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.645+0000 7f64e4c00700 1 -- 172.21.15.89:0/3339633502 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d4013610 con 0x7f64e80b2f30 2024-04-02T07:52:00.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.645+0000 7f64e4c00700 1 -- 172.21.15.89:0/3339633502 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d4013770 con 0x7f64e80b2f30 2024-04-02T07:52:00.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.645+0000 7f64efa00700 1 -- 172.21.15.89:0/3339633502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e815c2a0 con 0x7f64e80b2f30 2024-04-02T07:52:00.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.645+0000 7f64e4c00700 1 -- 172.21.15.89:0/3339633502 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d4013b20 con 0x7f64e80b2f30 2024-04-02T07:52:00.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.645+0000 7f64efa00700 1 -- 172.21.15.89:0/3339633502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e8148980 con 0x7f64e80b2f30 2024-04-02T07:52:00.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.646+0000 7f64efa00700 1 -- 172.21.15.89:0/3339633502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e8067720 con 0x7f64e80b2f30 2024-04-02T07:52:00.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.647+0000 7f64e4c00700 1 -- 172.21.15.89:0/3339633502 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f64d4013c80 con 0x7f64e80b2f30 2024-04-02T07:52:00.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.648+0000 7f64e4c00700 1 --2- 172.21.15.89:0/3339633502 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f64c806adb0 0x7f64c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:00.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.648+0000 7f64e4c00700 1 -- 172.21.15.89:0/3339633502 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f64d40892f0 con 0x7f64e80b2f30 2024-04-02T07:52:00.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.648+0000 7f64ed000700 1 --2- 172.21.15.89:0/3339633502 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f64c806adb0 0x7f64c806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:00.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.649+0000 7f64ed000700 1 --2- 172.21.15.89:0/3339633502 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f64c806adb0 0x7f64c806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0090b0 tx=0x7f64dc01c040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:00.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.652+0000 7f64e4c00700 1 -- 172.21.15.89:0/3339633502 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f64d4054160 con 0x7f64e80b2f30 2024-04-02T07:52:00.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:00.936+0000 7f64efa00700 1 -- 172.21.15.89:0/3339633502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f64e806e120 con 0x7f64e80b2f30 2024-04-02T07:52:01.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.102+0000 7f64e4c00700 1 -- 172.21.15.89:0/3339633502 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v592) v1 ==== 156+0+0 (secure 0 0 0) 0x7f64d4057780 con 0x7f64e80b2f30 2024-04-02T07:52:01.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.105+0000 7f64efa00700 1 -- 172.21.15.89:0/3339633502 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f64c806adb0 msgr2=0x7f64c806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:01.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.105+0000 7f64efa00700 1 --2- 172.21.15.89:0/3339633502 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f64c806adb0 0x7f64c806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0090b0 tx=0x7f64dc01c040 comp rx=0 tx=0).stop 2024-04-02T07:52:01.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.105+0000 7f64efa00700 1 -- 172.21.15.89:0/3339633502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64e80b2f30 msgr2=0x7f64e8155e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:01.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.105+0000 7f64efa00700 1 --2- 172.21.15.89:0/3339633502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64e80b2f30 0x7f64e8155e60 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f64d400e810 tx=0x7f64d400eb20 comp rx=0 tx=0).stop 2024-04-02T07:52:01.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.105+0000 7f64efa00700 1 -- 172.21.15.89:0/3339633502 shutdown_connections 2024-04-02T07:52:01.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.105+0000 7f64efa00700 1 --2- 172.21.15.89:0/3339633502 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f64c806adb0 0x7f64c806d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:01.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.105+0000 7f64efa00700 1 --2- 172.21.15.89:0/3339633502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64e8159b40 0x7f64e8148440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:01.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.105+0000 7f64efa00700 1 --2- 172.21.15.89:0/3339633502 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64e806d2e0 0x7f64e8159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:01.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.105+0000 7f64efa00700 1 --2- 172.21.15.89:0/3339633502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64e80b2f30 0x7f64e8155e60 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:01.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.105+0000 7f64efa00700 1 -- 172.21.15.89:0/3339633502 >> 172.21.15.89:0/3339633502 conn(0x7f64e80b0ef0 msgr2=0x7f64e80ccef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:01.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.105+0000 7f64efa00700 1 -- 172.21.15.89:0/3339633502 shutdown_connections 2024-04-02T07:52:01.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.105+0000 7f64efa00700 1 -- 172.21.15.89:0/3339633502 wait complete. 2024-04-02T07:52:01.125 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2024-04-02T07:52:01.126 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2024-04-02T07:52:01.126 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-02T07:52:01.126 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-04-02T07:52:01.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.314+0000 7f2c6f600700 1 Processor -- start 2024-04-02T07:52:01.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.315+0000 7f2c6f600700 1 -- start start 2024-04-02T07:52:01.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c6f600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2c6815d640 0x7f2c680d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:01.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c6f600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2c680ceec0 0x7f2c681612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:01.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c6f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2c68161800 0x7f2c680cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:01.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c6f600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c68161c70 con 0x7f2c680ceec0 2024-04-02T07:52:01.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c6f600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c680d30d0 con 0x7f2c6815d640 2024-04-02T07:52:01.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c6f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c680b6840 con 0x7f2c68161800 2024-04-02T07:52:01.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c67e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2c680ceec0 0x7f2c681612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:01.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c67e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2c680ceec0 0x7f2c681612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53040/0 (socket says 172.21.15.89:53040) 2024-04-02T07:52:01.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c67e00700 1 -- 172.21.15.89:0/1598756497 learned_addr learned my addr 172.21.15.89:0/1598756497 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:01.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c6cc00700 1 --2- 172.21.15.89:0/1598756497 >> v2:172.21.15.113:3300/0 conn(0x7f2c6815d640 0x7f2c680d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:01.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c6d600700 1 --2- 172.21.15.89:0/1598756497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2c68161800 0x7f2c680cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:01.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c67e00700 1 -- 172.21.15.89:0/1598756497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2c68161800 msgr2=0x7f2c680cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:01.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c67e00700 1 --2- 172.21.15.89:0/1598756497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2c68161800 0x7f2c680cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:01.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c67e00700 1 -- 172.21.15.89:0/1598756497 >> v2:172.21.15.113:3300/0 conn(0x7f2c6815d640 msgr2=0x7f2c680d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:01.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c67e00700 1 --2- 172.21.15.89:0/1598756497 >> v2:172.21.15.113:3300/0 conn(0x7f2c6815d640 0x7f2c680d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:01.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c67e00700 1 -- 172.21.15.89:0/1598756497 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c680d2880 con 0x7f2c680ceec0 2024-04-02T07:52:01.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.316+0000 7f2c6cc00700 1 --2- 172.21.15.89:0/1598756497 >> v2:172.21.15.113:3300/0 conn(0x7f2c6815d640 0x7f2c680d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:01.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.317+0000 7f2c67e00700 1 --2- 172.21.15.89:0/1598756497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c680ceec0 0x7f2c681612c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f2c680b5c20 tx=0x7f2c54009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7cd1158713d421 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:01.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.317+0000 7f2c67400700 1 -- 172.21.15.89:0/1598756497 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5400b870 con 0x7f2c680ceec0 2024-04-02T07:52:01.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.317+0000 7f2c67400700 1 -- 172.21.15.89:0/1598756497 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5400b9d0 con 0x7f2c680ceec0 2024-04-02T07:52:01.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.317+0000 7f2c67400700 1 -- 172.21.15.89:0/1598756497 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5400bd60 con 0x7f2c680ceec0 2024-04-02T07:52:01.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.317+0000 7f2c6f600700 1 -- 172.21.15.89:0/1598756497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c680ceec0 msgr2=0x7f2c681612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:01.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.317+0000 7f2c6f600700 1 --2- 172.21.15.89:0/1598756497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c680ceec0 0x7f2c681612c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f2c680b5c20 tx=0x7f2c54009bd0 comp rx=0 tx=0).stop 2024-04-02T07:52:01.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.318+0000 7f2c6f600700 1 -- 172.21.15.89:0/1598756497 shutdown_connections 2024-04-02T07:52:01.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.318+0000 7f2c6f600700 1 --2- 172.21.15.89:0/1598756497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2c68161800 0x7f2c680cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:01.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.318+0000 7f2c6f600700 1 --2- 172.21.15.89:0/1598756497 >> v2:172.21.15.113:3300/0 conn(0x7f2c6815d640 0x7f2c680d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:01.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.318+0000 7f2c6f600700 1 --2- 172.21.15.89:0/1598756497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c680ceec0 0x7f2c681612c0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:01.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.318+0000 7f2c6f600700 1 -- 172.21.15.89:0/1598756497 >> 172.21.15.89:0/1598756497 conn(0x7f2c6808aff0 msgr2=0x7f2c680d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:01.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.318+0000 7f2c6f600700 1 -- 172.21.15.89:0/1598756497 shutdown_connections 2024-04-02T07:52:01.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.318+0000 7f2c6f600700 1 -- 172.21.15.89:0/1598756497 wait complete. 2024-04-02T07:52:01.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.319+0000 7f2c6f600700 1 Processor -- start 2024-04-02T07:52:01.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.319+0000 7f2c6f600700 1 -- start start 2024-04-02T07:52:01.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.320+0000 7f2c6f600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c680ceec0 0x7f2c680ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:01.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.320+0000 7f2c6f600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c6815d640 0x7f2c680a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:01.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.320+0000 7f2c6f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2c68161800 0x7f2c680cd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:01.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.320+0000 7f2c6f600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c680b4ca0 con 0x7f2c6815d640 2024-04-02T07:52:01.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.320+0000 7f2c6f600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c680cbb00 con 0x7f2c680ceec0 2024-04-02T07:52:01.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.320+0000 7f2c6f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c680cbde0 con 0x7f2c68161800 2024-04-02T07:52:01.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.321+0000 7f2c67e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c6815d640 0x7f2c680a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:01.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.321+0000 7f2c67e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c6815d640 0x7f2c680a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53048/0 (socket says 172.21.15.89:53048) 2024-04-02T07:52:01.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.321+0000 7f2c67e00700 1 -- 172.21.15.89:0/1295244671 learned_addr learned my addr 172.21.15.89:0/1295244671 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:01.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.321+0000 7f2c6cc00700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c680ceec0 0x7f2c680ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:01.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.321+0000 7f2c6d600700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2c68161800 0x7f2c680cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:01.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.321+0000 7f2c67e00700 1 -- 172.21.15.89:0/1295244671 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2c68161800 msgr2=0x7f2c680cd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:01.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.321+0000 7f2c67e00700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2c68161800 0x7f2c680cd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:01.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.321+0000 7f2c67e00700 1 -- 172.21.15.89:0/1295244671 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c680ceec0 msgr2=0x7f2c680ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:01.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.321+0000 7f2c67e00700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c680ceec0 0x7f2c680ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:01.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.321+0000 7f2c67e00700 1 -- 172.21.15.89:0/1295244671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c54009550 con 0x7f2c6815d640 2024-04-02T07:52:01.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.321+0000 7f2c6d600700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2c68161800 0x7f2c680cd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:01.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.321+0000 7f2c6cc00700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c680ceec0 0x7f2c680ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:01.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.321+0000 7f2c67e00700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c6815d640 0x7f2c680a0b90 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f2c54000c00 tx=0x7f2c54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:01.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.321+0000 7f2c64c00700 1 -- 172.21.15.89:0/1295244671 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c54004520 con 0x7f2c6815d640 2024-04-02T07:52:01.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.322+0000 7f2c64c00700 1 -- 172.21.15.89:0/1295244671 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5401e070 con 0x7f2c6815d640 2024-04-02T07:52:01.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.322+0000 7f2c64c00700 1 -- 172.21.15.89:0/1295244671 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c540038c0 con 0x7f2c6815d640 2024-04-02T07:52:01.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.322+0000 7f2c6f600700 1 -- 172.21.15.89:0/1295244671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c680cc0c0 con 0x7f2c6815d640 2024-04-02T07:52:01.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.322+0000 7f2c6f600700 1 -- 172.21.15.89:0/1295244671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c680cc680 con 0x7f2c6815d640 2024-04-02T07:52:01.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.323+0000 7f2c6f600700 1 -- 172.21.15.89:0/1295244671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c68067720 con 0x7f2c6815d640 2024-04-02T07:52:01.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.324+0000 7f2c64c00700 1 -- 172.21.15.89:0/1295244671 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2c54003a20 con 0x7f2c6815d640 2024-04-02T07:52:01.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.324+0000 7f2c64c00700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2c4806adb0 0x7f2c4806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:01.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.325+0000 7f2c64c00700 1 -- 172.21.15.89:0/1295244671 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f2c54020070 con 0x7f2c6815d640 2024-04-02T07:52:01.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.325+0000 7f2c6cc00700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2c4806adb0 0x7f2c4806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:01.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.325+0000 7f2c6cc00700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2c4806adb0 0x7f2c4806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0099f0 tx=0x7f2c5c008040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:01.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.329+0000 7f2c64c00700 1 -- 172.21.15.89:0/1295244671 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2c54055b00 con 0x7f2c6815d640 2024-04-02T07:52:01.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:01.609+0000 7f2c6f600700 1 -- 172.21.15.89:0/1295244671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f2c680b5c70 con 0x7f2c6815d640 2024-04-02T07:52:02.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.104+0000 7f2c64c00700 1 -- 172.21.15.89:0/1295244671 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v593) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2c5402a0f0 con 0x7f2c6815d640 2024-04-02T07:52:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.107+0000 7f2c6f600700 1 -- 172.21.15.89:0/1295244671 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2c4806adb0 msgr2=0x7f2c4806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.107+0000 7f2c6f600700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2c4806adb0 0x7f2c4806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0099f0 tx=0x7f2c5c008040 comp rx=0 tx=0).stop 2024-04-02T07:52:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.107+0000 7f2c6f600700 1 -- 172.21.15.89:0/1295244671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c6815d640 msgr2=0x7f2c680a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.107+0000 7f2c6f600700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c6815d640 0x7f2c680a0b90 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f2c54000c00 tx=0x7f2c54004750 comp rx=0 tx=0).stop 2024-04-02T07:52:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.107+0000 7f2c6f600700 1 -- 172.21.15.89:0/1295244671 shutdown_connections 2024-04-02T07:52:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.107+0000 7f2c6f600700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2c4806adb0 0x7f2c4806d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.107+0000 7f2c6f600700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c680ceec0 0x7f2c680ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.107+0000 7f2c6f600700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2c68161800 0x7f2c680cd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.107+0000 7f2c6f600700 1 --2- 172.21.15.89:0/1295244671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c6815d640 0x7f2c680a0b90 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:02.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.107+0000 7f2c6f600700 1 -- 172.21.15.89:0/1295244671 >> 172.21.15.89:0/1295244671 conn(0x7f2c6808aff0 msgr2=0x7f2c680b7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:02.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.107+0000 7f2c6f600700 1 -- 172.21.15.89:0/1295244671 shutdown_connections 2024-04-02T07:52:02.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.108+0000 7f2c6f600700 1 -- 172.21.15.89:0/1295244671 wait complete. 2024-04-02T07:52:02.130 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T07:52:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.324+0000 7f2f8ea00700 1 Processor -- start 2024-04-02T07:52:02.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.324+0000 7f2f8ea00700 1 -- start start 2024-04-02T07:52:02.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.325+0000 7f2f8ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2f8815a5c0 0x7f2f880709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:02.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.325+0000 7f2f8ea00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2f8815dfe0 0x7f2f880ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:02.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.325+0000 7f2f8ea00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f880cb310 con 0x7f2f880ce670 2024-04-02T07:52:02.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.325+0000 7f2f8ea00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f880ceae0 con 0x7f2f8815a5c0 2024-04-02T07:52:02.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.325+0000 7f2f8ea00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f88070f00 con 0x7f2f8815dfe0 2024-04-02T07:52:02.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.326+0000 7f2f87e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2f8815a5c0 0x7f2f880709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:02.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.326+0000 7f2f8ca00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2f880ce670 0x7f2f880b2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48950/0 (socket says 172.21.15.89:48950) 2024-04-02T07:52:02.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.326+0000 7f2f87e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2f8815a5c0 0x7f2f880709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53058/0 (socket says 172.21.15.89:53058) 2024-04-02T07:52:02.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.326+0000 7f2f8ca00700 1 -- 172.21.15.89:0/1858892443 learned_addr learned my addr 172.21.15.89:0/1858892443 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:02.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.326+0000 7f2f87400700 1 --2- 172.21.15.89:0/1858892443 >> v2:172.21.15.113:3300/0 conn(0x7f2f8815dfe0 0x7f2f880ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:02.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.326+0000 7f2f87e00700 1 -- 172.21.15.89:0/1858892443 >> v2:172.21.15.113:3300/0 conn(0x7f2f8815dfe0 msgr2=0x7f2f880ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:02.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.326+0000 7f2f87e00700 1 --2- 172.21.15.89:0/1858892443 >> v2:172.21.15.113:3300/0 conn(0x7f2f8815dfe0 0x7f2f880ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:02.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.326+0000 7f2f87e00700 1 -- 172.21.15.89:0/1858892443 >> v1:172.21.15.89:6789/0 conn(0x7f2f880ce670 legacy=0x7f2f880b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:02.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.326+0000 7f2f87e00700 1 -- 172.21.15.89:0/1858892443 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f880cca90 con 0x7f2f8815a5c0 2024-04-02T07:52:02.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.326+0000 7f2f86a00700 1 -- 172.21.15.89:0/1858892443 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204673266 0 0) 0x7f2f880cb310 con 0x7f2f880ce670 2024-04-02T07:52:02.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.326+0000 7f2f87e00700 1 --2- 172.21.15.89:0/1858892443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f8815a5c0 0x7f2f880709c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f2f880d0220 tx=0x7f2f7800bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7b9cb39aed330b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.327+0000 7f2f86a00700 1 -- 172.21.15.89:0/1858892443 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7800dc70 con 0x7f2f8815a5c0 2024-04-02T07:52:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.327+0000 7f2f86a00700 1 -- 172.21.15.89:0/1858892443 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f7800ddd0 con 0x7f2f8815a5c0 2024-04-02T07:52:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.327+0000 7f2f86a00700 1 -- 172.21.15.89:0/1858892443 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f78010690 con 0x7f2f8815a5c0 2024-04-02T07:52:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.327+0000 7f2f8ea00700 1 -- 172.21.15.89:0/1858892443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f8815a5c0 msgr2=0x7f2f880709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.327+0000 7f2f8ea00700 1 --2- 172.21.15.89:0/1858892443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f8815a5c0 0x7f2f880709c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f2f880d0220 tx=0x7f2f7800bdc0 comp rx=0 tx=0).stop 2024-04-02T07:52:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.327+0000 7f2f8ea00700 1 -- 172.21.15.89:0/1858892443 shutdown_connections 2024-04-02T07:52:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.327+0000 7f2f8ea00700 1 --2- 172.21.15.89:0/1858892443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f8815a5c0 0x7f2f880709c0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.327+0000 7f2f8ea00700 1 --2- 172.21.15.89:0/1858892443 >> v2:172.21.15.113:3300/0 conn(0x7f2f8815dfe0 0x7f2f880ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.327+0000 7f2f8ea00700 1 -- 172.21.15.89:0/1858892443 >> 172.21.15.89:0/1858892443 conn(0x7f2f880b0d70 msgr2=0x7f2f880b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.327+0000 7f2f8ea00700 1 -- 172.21.15.89:0/1858892443 shutdown_connections 2024-04-02T07:52:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.327+0000 7f2f8ea00700 1 -- 172.21.15.89:0/1858892443 wait complete. 2024-04-02T07:52:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.329+0000 7f2f8ea00700 1 Processor -- start 2024-04-02T07:52:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.329+0000 7f2f8ea00700 1 -- start start 2024-04-02T07:52:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.330+0000 7f2f8ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f880ce670 0x7f2f8814e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.330+0000 7f2f8ea00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f8815a5c0 0x7f2f880c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.330+0000 7f2f8ea00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f8815dfe0 0x7f2f880aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.330+0000 7f2f8ea00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8815c260 con 0x7f2f880ce670 2024-04-02T07:52:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.330+0000 7f2f8ea00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8814ea00 con 0x7f2f8815dfe0 2024-04-02T07:52:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.330+0000 7f2f8ea00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8815b5e0 con 0x7f2f8815a5c0 2024-04-02T07:52:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.330+0000 7f2f87e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f880ce670 0x7f2f8814e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.330+0000 7f2f87e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f880ce670 0x7f2f8814e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53068/0 (socket says 172.21.15.89:53068) 2024-04-02T07:52:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.330+0000 7f2f87e00700 1 -- 172.21.15.89:0/4262427778 learned_addr learned my addr 172.21.15.89:0/4262427778 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f87400700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f8815a5c0 0x7f2f880c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f8ca00700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f8815dfe0 0x7f2f880aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f87e00700 1 -- 172.21.15.89:0/4262427778 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f8815a5c0 msgr2=0x7f2f880c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f87e00700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f8815a5c0 0x7f2f880c96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f87e00700 1 -- 172.21.15.89:0/4262427778 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f8815dfe0 msgr2=0x7f2f880aece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f87e00700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f8815dfe0 0x7f2f880aece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f87e00700 1 -- 172.21.15.89:0/4262427778 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7800b740 con 0x7f2f880ce670 2024-04-02T07:52:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f87400700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f8815a5c0 0x7f2f880c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f8ca00700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f8815dfe0 0x7f2f880aece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:02.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f87e00700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f880ce670 0x7f2f8814e0d0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f2f7800b2c0 tx=0x7f2f78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:02.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f7fe00700 1 -- 172.21.15.89:0/4262427778 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f78003cf0 con 0x7f2f880ce670 2024-04-02T07:52:02.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f7fe00700 1 -- 172.21.15.89:0/4262427778 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f78003e50 con 0x7f2f880ce670 2024-04-02T07:52:02.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f7fe00700 1 -- 172.21.15.89:0/4262427778 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f78010670 con 0x7f2f880ce670 2024-04-02T07:52:02.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f8ea00700 1 -- 172.21.15.89:0/4262427778 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8815b8c0 con 0x7f2f880ce670 2024-04-02T07:52:02.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.331+0000 7f2f8ea00700 1 -- 172.21.15.89:0/4262427778 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8815be80 con 0x7f2f880ce670 2024-04-02T07:52:02.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.332+0000 7f2f8ea00700 1 -- 172.21.15.89:0/4262427778 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8815c580 con 0x7f2f880ce670 2024-04-02T07:52:02.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.333+0000 7f2f7fe00700 1 -- 172.21.15.89:0/4262427778 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2f78019400 con 0x7f2f880ce670 2024-04-02T07:52:02.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.335+0000 7f2f7fe00700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f6806adb0 0x7f2f6806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:02.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.335+0000 7f2f7fe00700 1 -- 172.21.15.89:0/4262427778 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f2f7808b1a0 con 0x7f2f880ce670 2024-04-02T07:52:02.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.335+0000 7f2f87400700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f6806adb0 0x7f2f6806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:02.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.335+0000 7f2f87400700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f6806adb0 0x7f2f6806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2f88150150 tx=0x7f2f74008040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:02.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.339+0000 7f2f7fe00700 1 -- 172.21.15.89:0/4262427778 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2f8815c580 con 0x7f2f880ce670 2024-04-02T07:52:02.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.543+0000 7f2f8ea00700 1 -- 172.21.15.89:0/4262427778 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2f8815c580 con 0x7f2f880ce670 2024-04-02T07:52:02.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.547+0000 7f2f7fe00700 1 -- 172.21.15.89:0/4262427778 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2f8815c580 con 0x7f2f880ce670 2024-04-02T07:52:02.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.550+0000 7f2f8ea00700 1 -- 172.21.15.89:0/4262427778 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f6806adb0 msgr2=0x7f2f6806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.550+0000 7f2f8ea00700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f6806adb0 0x7f2f6806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2f88150150 tx=0x7f2f74008040 comp rx=0 tx=0).stop 2024-04-02T07:52:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.550+0000 7f2f8ea00700 1 -- 172.21.15.89:0/4262427778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f880ce670 msgr2=0x7f2f8814e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.550+0000 7f2f8ea00700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f880ce670 0x7f2f8814e0d0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f2f7800b2c0 tx=0x7f2f78004750 comp rx=0 tx=0).stop 2024-04-02T07:52:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.550+0000 7f2f8ea00700 1 -- 172.21.15.89:0/4262427778 shutdown_connections 2024-04-02T07:52:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.550+0000 7f2f8ea00700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2f6806adb0 0x7f2f6806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.550+0000 7f2f8ea00700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f8815dfe0 0x7f2f880aece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.550+0000 7f2f8ea00700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f880ce670 0x7f2f8814e0d0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.550+0000 7f2f8ea00700 1 --2- 172.21.15.89:0/4262427778 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f8815a5c0 0x7f2f880c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.550+0000 7f2f8ea00700 1 -- 172.21.15.89:0/4262427778 >> 172.21.15.89:0/4262427778 conn(0x7f2f880b0d70 msgr2=0x7f2f881598d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.550+0000 7f2f8ea00700 1 -- 172.21.15.89:0/4262427778 shutdown_connections 2024-04-02T07:52:02.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:02.550+0000 7f2f8ea00700 1 -- 172.21.15.89:0/4262427778 wait complete. 2024-04-02T07:52:02.570 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:52:02.571 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:52:02.656 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:52:02.745 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:52:02.745 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:52:02.746 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:02.790 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:52:02.790 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:52:02.790 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:02.836 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:52:02.837 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:52:02.837 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:02.882 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:52:02.950 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:52:02.950 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:02.995 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:52:02.995 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T07:52:02.996 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:52:03.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.187+0000 7f850fc00700 1 Processor -- start 2024-04-02T07:52:03.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.187+0000 7f850fc00700 1 -- start start 2024-04-02T07:52:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.188+0000 7f850fc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8508160c90 0x7f85080b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.188+0000 7f850fc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85080718e0 0x7f85080b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.188+0000 7f850fc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85080ac4d0 con 0x7f85080709c0 2024-04-02T07:52:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.188+0000 7f850fc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8508071e50 con 0x7f8508160c90 2024-04-02T07:52:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.188+0000 7f850fc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8508072130 con 0x7f85080718e0 2024-04-02T07:52:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.189+0000 7f850d200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8508160c90 0x7f85080b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.189+0000 7f8507e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85080718e0 0x7f85080b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.189+0000 7f850d200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8508160c90 0x7f85080b20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:41206/0 (socket says 172.21.15.89:41206) 2024-04-02T07:52:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.189+0000 7f850d200700 1 -- 172.21.15.89:0/835432251 learned_addr learned my addr 172.21.15.89:0/835432251 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.189+0000 7f850d200700 1 -- 172.21.15.89:0/835432251 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85080718e0 msgr2=0x7f85080b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.189+0000 7f850d200700 1 --2- 172.21.15.89:0/835432251 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85080718e0 0x7f85080b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.189+0000 7f850d200700 1 -- 172.21.15.89:0/835432251 >> v1:172.21.15.113:6789/0 conn(0x7f85080709c0 legacy=0x7f850806cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.189+0000 7f850d200700 1 -- 172.21.15.89:0/835432251 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8508072410 con 0x7f8508160c90 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.189+0000 7f8507400700 1 -- 172.21.15.89:0/835432251 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714683012 0 0) 0x7f85080ac4d0 con 0x7f85080709c0 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.189+0000 7f850d200700 1 --2- 172.21.15.89:0/835432251 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8508160c90 0x7f85080b20e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f84fc0098f0 tx=0x7f84fc009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b6892182efd84d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.190+0000 7f8507400700 1 -- 172.21.15.89:0/835432251 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc011830 con 0x7f8508160c90 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.190+0000 7f8507400700 1 -- 172.21.15.89:0/835432251 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc011990 con 0x7f8508160c90 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.190+0000 7f8507400700 1 -- 172.21.15.89:0/835432251 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc011d20 con 0x7f8508160c90 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.190+0000 7f850fc00700 1 -- 172.21.15.89:0/835432251 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8508160c90 msgr2=0x7f85080b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.190+0000 7f850fc00700 1 --2- 172.21.15.89:0/835432251 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8508160c90 0x7f85080b20e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f84fc0098f0 tx=0x7f84fc009c00 comp rx=0 tx=0).stop 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.190+0000 7f850fc00700 1 -- 172.21.15.89:0/835432251 shutdown_connections 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.190+0000 7f850fc00700 1 --2- 172.21.15.89:0/835432251 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8508160c90 0x7f85080b20e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.190+0000 7f850fc00700 1 --2- 172.21.15.89:0/835432251 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85080718e0 0x7f85080b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.190+0000 7f850fc00700 1 -- 172.21.15.89:0/835432251 >> 172.21.15.89:0/835432251 conn(0x7f85080cf190 msgr2=0x7f85080cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.190+0000 7f850fc00700 1 -- 172.21.15.89:0/835432251 shutdown_connections 2024-04-02T07:52:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.190+0000 7f850fc00700 1 -- 172.21.15.89:0/835432251 wait complete. 2024-04-02T07:52:03.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.192+0000 7f850fc00700 1 Processor -- start 2024-04-02T07:52:03.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.192+0000 7f850fc00700 1 -- start start 2024-04-02T07:52:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.193+0000 7f850fc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85080709c0 0x7f85080b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.193+0000 7f850fc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85080718e0 0x7f85080cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.193+0000 7f850fc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8508160c90 0x7f85080c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.193+0000 7f850fc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85080b0830 con 0x7f85080709c0 2024-04-02T07:52:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.193+0000 7f850fc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85080ccd10 con 0x7f85080718e0 2024-04-02T07:52:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.193+0000 7f850fc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85080ccf60 con 0x7f8508160c90 2024-04-02T07:52:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.193+0000 7f850d200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85080709c0 0x7f85080b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.193+0000 7f850d200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85080709c0 0x7f85080b02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53074/0 (socket says 172.21.15.89:53074) 2024-04-02T07:52:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.193+0000 7f850d200700 1 -- 172.21.15.89:0/3983361065 learned_addr learned my addr 172.21.15.89:0/3983361065 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.193+0000 7f8507e00700 1 --2- 172.21.15.89:0/3983361065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85080718e0 0x7f85080cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.193+0000 7f850dc00700 1 --2- 172.21.15.89:0/3983361065 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8508160c90 0x7f85080c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.194+0000 7f850d200700 1 -- 172.21.15.89:0/3983361065 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8508160c90 msgr2=0x7f85080c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.194+0000 7f850d200700 1 --2- 172.21.15.89:0/3983361065 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8508160c90 0x7f85080c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.194+0000 7f850d200700 1 -- 172.21.15.89:0/3983361065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85080718e0 msgr2=0x7f85080cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.194+0000 7f850d200700 1 --2- 172.21.15.89:0/3983361065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85080718e0 0x7f85080cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.194+0000 7f850d200700 1 -- 172.21.15.89:0/3983361065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc009550 con 0x7f85080709c0 2024-04-02T07:52:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.194+0000 7f850d200700 1 --2- 172.21.15.89:0/3983361065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85080709c0 0x7f85080b02f0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f84fc009c40 tx=0x7f84fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.194+0000 7f8504c00700 1 -- 172.21.15.89:0/3983361065 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc004510 con 0x7f85080709c0 2024-04-02T07:52:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.194+0000 7f8504c00700 1 -- 172.21.15.89:0/3983361065 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc01e070 con 0x7f85080709c0 2024-04-02T07:52:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.194+0000 7f8504c00700 1 -- 172.21.15.89:0/3983361065 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc0038c0 con 0x7f85080709c0 2024-04-02T07:52:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.194+0000 7f850fc00700 1 -- 172.21.15.89:0/3983361065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85080cd240 con 0x7f85080709c0 2024-04-02T07:52:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.194+0000 7f850fc00700 1 -- 172.21.15.89:0/3983361065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85080cd800 con 0x7f85080709c0 2024-04-02T07:52:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.195+0000 7f850fc00700 1 -- 172.21.15.89:0/3983361065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8508067720 con 0x7f85080709c0 2024-04-02T07:52:03.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.196+0000 7f8504c00700 1 -- 172.21.15.89:0/3983361065 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f84fc003a20 con 0x7f85080709c0 2024-04-02T07:52:03.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.197+0000 7f8504c00700 1 --2- 172.21.15.89:0/3983361065 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84e806adb0 0x7f84e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:03.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.197+0000 7f8504c00700 1 -- 172.21.15.89:0/3983361065 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f84fc08a440 con 0x7f85080709c0 2024-04-02T07:52:03.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.197+0000 7f8507e00700 1 --2- 172.21.15.89:0/3983361065 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84e806adb0 0x7f84e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:03.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.198+0000 7f8507e00700 1 --2- 172.21.15.89:0/3983361065 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84e806adb0 0x7f84e806d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8508153070 tx=0x7f84f4009210 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:03.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.202+0000 7f8504c00700 1 -- 172.21.15.89:0/3983361065 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f84fc0553c0 con 0x7f85080709c0 2024-04-02T07:52:03.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.445+0000 7f850fc00700 1 -- 172.21.15.89:0/3983361065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85080b39a0 con 0x7f85080709c0 2024-04-02T07:52:03.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.445+0000 7f8504c00700 1 -- 172.21.15.89:0/3983361065 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f84fc020de0 con 0x7f85080709c0 2024-04-02T07:52:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:03.449 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5660,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2025634273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2025634273},{"type":"v1","addr":"172.21.15.113:6834","nonce":2025634273}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2247445394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2247445394},{"type":"v1","addr":"172.21.15.113:6835","nonce":2247445394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6414,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/1785147820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1785147820},{"type":"v1","addr":"172.21.15.89:6835","nonce":1785147820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3616563415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3616563415},{"type":"v1","addr":"172.21.15.89:6837","nonce":3616563415}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[]} 2024-04-02T07:52:03.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.448+0000 7f850fc00700 1 -- 172.21.15.89:0/3983361065 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84e806adb0 msgr2=0x7f84e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:03.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.448+0000 7f850fc00700 1 --2- 172.21.15.89:0/3983361065 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84e806adb0 0x7f84e806d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8508153070 tx=0x7f84f4009210 comp rx=0 tx=0).stop 2024-04-02T07:52:03.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.448+0000 7f850fc00700 1 -- 172.21.15.89:0/3983361065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85080709c0 msgr2=0x7f85080b02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:03.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.448+0000 7f850fc00700 1 --2- 172.21.15.89:0/3983361065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85080709c0 0x7f85080b02f0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f84fc009c40 tx=0x7f84fc004750 comp rx=0 tx=0).stop 2024-04-02T07:52:03.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.448+0000 7f850fc00700 1 -- 172.21.15.89:0/3983361065 shutdown_connections 2024-04-02T07:52:03.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.448+0000 7f850fc00700 1 --2- 172.21.15.89:0/3983361065 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84e806adb0 0x7f84e806d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:03.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.448+0000 7f850fc00700 1 --2- 172.21.15.89:0/3983361065 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8508160c90 0x7f85080c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:03.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.448+0000 7f850fc00700 1 --2- 172.21.15.89:0/3983361065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85080709c0 0x7f85080b02f0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:03.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.448+0000 7f850fc00700 1 --2- 172.21.15.89:0/3983361065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85080718e0 0x7f85080cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:03.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.448+0000 7f850fc00700 1 -- 172.21.15.89:0/3983361065 >> 172.21.15.89:0/3983361065 conn(0x7f85080cf190 msgr2=0x7f85080ce6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:03.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.448+0000 7f850fc00700 1 -- 172.21.15.89:0/3983361065 shutdown_connections 2024-04-02T07:52:03.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.448+0000 7f850fc00700 1 -- 172.21.15.89:0/3983361065 wait complete. 2024-04-02T07:52:03.453 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 35 2024-04-02T07:52:03.469 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T07:52:03.469 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T07:52:03.470 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T07:52:03.470 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:52:03.470 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T07:52:03.470 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T07:52:03.471 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T07:52:03.471 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:52:03.471 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T07:52:03.471 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T07:52:03.472 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T07:52:03.472 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:52:03.472 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T07:52:03.472 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T07:52:03.473 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T07:52:03.473 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:52:03.485 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T07:52:03.485 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T07:52:03.487 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T07:52:03.487 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T07:52:03.489 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T07:52:03.489 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T07:52:03.491 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T07:52:03.491 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T07:52:03.529 INFO:tasks.ceph.mds.a:Started 2024-04-02T07:52:03.530 DEBUG:teuthology.parallel:result is None 2024-04-02T07:52:03.532 INFO:tasks.ceph.mds.c:Started 2024-04-02T07:52:03.532 DEBUG:teuthology.parallel:result is None 2024-04-02T07:52:03.533 INFO:tasks.ceph.mds.b:Started 2024-04-02T07:52:03.534 DEBUG:teuthology.parallel:result is None 2024-04-02T07:52:03.536 INFO:tasks.ceph.mds.d:Started 2024-04-02T07:52:03.536 DEBUG:teuthology.parallel:result is None 2024-04-02T07:52:03.536 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T07:52:03.675 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T07:52:03.676 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T07:52:03.682 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T07:52:03.683 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T07:52:03.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.739+0000 7f5bd8600700 1 Processor -- start 2024-04-02T07:52:03.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.739+0000 7f5bd8600700 1 -- start start 2024-04-02T07:52:03.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.740+0000 7f5bd8600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd00b3f80 0x7f5bd00ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.740+0000 7f5bd8600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd0161230 con 0x7f5bd00d3f20 2024-04-02T07:52:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.740+0000 7f5bd8600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd00d2a80 con 0x7f5bd0160e50 2024-04-02T07:52:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.740+0000 7f5bd8600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd00d43b0 con 0x7f5bd00b3f80 2024-04-02T07:52:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.740+0000 7f5bd5200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5bd00d3f20 0x7f5bd00cdef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48982/0 (socket says 172.21.15.89:48982) 2024-04-02T07:52:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.740+0000 7f5bd5200700 1 -- 172.21.15.89:0/658429366 learned_addr learned my addr 172.21.15.89:0/658429366 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.740+0000 7f5bd6600700 1 --2- 172.21.15.89:0/658429366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd00b3f80 0x7f5bd00ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.740+0000 7f5bcfe00700 1 -- 172.21.15.89:0/658429366 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343094092 0 0) 0x7f5bd0161230 con 0x7f5bd00d3f20 2024-04-02T07:52:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.741+0000 7f5bcfe00700 1 -- 172.21.15.89:0/658429366 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb80035b0 con 0x7f5bd00d3f20 2024-04-02T07:52:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.741+0000 7f5bcfe00700 1 -- 172.21.15.89:0/658429366 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579796685 0 0) 0x7f5bd00d2a80 con 0x7f5bd0160e50 2024-04-02T07:52:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.741+0000 7f5bcfe00700 1 -- 172.21.15.89:0/658429366 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd0161230 con 0x7f5bd0160e50 2024-04-02T07:52:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.741+0000 7f5bd6600700 1 -- 172.21.15.89:0/658429366 >> v1:172.21.15.113:6789/0 conn(0x7f5bd0160e50 legacy=0x7f5bd00b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.741+0000 7f5bd6600700 1 -- 172.21.15.89:0/658429366 >> v1:172.21.15.89:6789/0 conn(0x7f5bd00d3f20 legacy=0x7f5bd00cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.741+0000 7f5bd6600700 1 -- 172.21.15.89:0/658429366 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd00d21e0 con 0x7f5bd00b3f80 2024-04-02T07:52:03.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.741+0000 7f5bd6600700 1 --2- 172.21.15.89:0/658429366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd00b3f80 0x7f5bd00ce600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009dc0 tx=0x7f5bc800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99acdeb961f3b74c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:03.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.741+0000 7f5bcfe00700 1 -- 172.21.15.89:0/658429366 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc800e070 con 0x7f5bd00b3f80 2024-04-02T07:52:03.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.741+0000 7f5bcfe00700 1 -- 172.21.15.89:0/658429366 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc800bb20 con 0x7f5bd00b3f80 2024-04-02T07:52:03.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.741+0000 7f5bcfe00700 1 -- 172.21.15.89:0/658429366 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8012430 con 0x7f5bd00b3f80 2024-04-02T07:52:03.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.742+0000 7f5bd8600700 1 -- 172.21.15.89:0/658429366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd00b3f80 msgr2=0x7f5bd00ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.742+0000 7f5bd8600700 1 --2- 172.21.15.89:0/658429366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd00b3f80 0x7f5bd00ce600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009dc0 tx=0x7f5bc800b330 comp rx=0 tx=0).stop 2024-04-02T07:52:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.742+0000 7f5bd8600700 1 -- 172.21.15.89:0/658429366 shutdown_connections 2024-04-02T07:52:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.742+0000 7f5bd8600700 1 --2- 172.21.15.89:0/658429366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd00b3f80 0x7f5bd00ce600 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.742+0000 7f5bd8600700 1 -- 172.21.15.89:0/658429366 >> 172.21.15.89:0/658429366 conn(0x7f5bd008aff0 msgr2=0x7f5bd00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.742+0000 7f5bd8600700 1 -- 172.21.15.89:0/658429366 shutdown_connections 2024-04-02T07:52:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.742+0000 7f5bd8600700 1 -- 172.21.15.89:0/658429366 wait complete. 2024-04-02T07:52:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.744+0000 7f5bd8600700 1 Processor -- start 2024-04-02T07:52:03.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.744+0000 7f5bd8600700 1 -- start start 2024-04-02T07:52:03.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd8600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd00b3f80 0x7f5bd015acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:03.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd8600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bd0160e50 0x7f5bd00b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:03.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd8600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd00b6690 0x7f5bd00b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:03.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd8600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd015b210 con 0x7f5bd0160e50 2024-04-02T07:52:03.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd8600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd00b7140 con 0x7f5bd00b3f80 2024-04-02T07:52:03.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd8600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd00b6b80 con 0x7f5bd00b6690 2024-04-02T07:52:03.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd5200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bd0160e50 0x7f5bd00b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:03.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd5200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bd0160e50 0x7f5bd00b6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53096/0 (socket says 172.21.15.89:53096) 2024-04-02T07:52:03.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd5200700 1 -- 172.21.15.89:0/3631459010 learned_addr learned my addr 172.21.15.89:0/3631459010 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:03.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd5c00700 1 --2- 172.21.15.89:0/3631459010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd00b3f80 0x7f5bd015acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:03.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd6600700 1 --2- 172.21.15.89:0/3631459010 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd00b6690 0x7f5bd00b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:03.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd5200700 1 -- 172.21.15.89:0/3631459010 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd00b6690 msgr2=0x7f5bd00b0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:03.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd5200700 1 --2- 172.21.15.89:0/3631459010 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd00b6690 0x7f5bd00b0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:03.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd5200700 1 -- 172.21.15.89:0/3631459010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd00b3f80 msgr2=0x7f5bd015acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:03.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.745+0000 7f5bd5200700 1 --2- 172.21.15.89:0/3631459010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd00b3f80 0x7f5bd015acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:03.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.746+0000 7f5bd5200700 1 -- 172.21.15.89:0/3631459010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc8009a70 con 0x7f5bd0160e50 2024-04-02T07:52:03.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.746+0000 7f5bd5200700 1 --2- 172.21.15.89:0/3631459010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bd0160e50 0x7f5bd00b6150 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc00c700 tx=0x7f5bbc00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:03.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.746+0000 7f5bcd600700 1 -- 172.21.15.89:0/3631459010 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc005d70 con 0x7f5bd0160e50 2024-04-02T07:52:03.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.746+0000 7f5bcd600700 1 -- 172.21.15.89:0/3631459010 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bbc0070d0 con 0x7f5bd0160e50 2024-04-02T07:52:03.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.746+0000 7f5bd8600700 1 -- 172.21.15.89:0/3631459010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd00b6e60 con 0x7f5bd0160e50 2024-04-02T07:52:03.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.746+0000 7f5bcd600700 1 -- 172.21.15.89:0/3631459010 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc007440 con 0x7f5bd0160e50 2024-04-02T07:52:03.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.746+0000 7f5bd8600700 1 -- 172.21.15.89:0/3631459010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd00b7b90 con 0x7f5bd0160e50 2024-04-02T07:52:03.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.747+0000 7f5bd8600700 1 -- 172.21.15.89:0/3631459010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd0067720 con 0x7f5bd0160e50 2024-04-02T07:52:03.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.748+0000 7f5bcd600700 1 -- 172.21.15.89:0/3631459010 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5bbc0075a0 con 0x7f5bd0160e50 2024-04-02T07:52:03.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.749+0000 7f5bcd600700 1 --2- 172.21.15.89:0/3631459010 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5bb006adb0 0x7f5bb006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:03.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.749+0000 7f5bcd600700 1 -- 172.21.15.89:0/3631459010 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f5bbc088830 con 0x7f5bd0160e50 2024-04-02T07:52:03.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.749+0000 7f5bd5c00700 1 --2- 172.21.15.89:0/3631459010 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5bb006adb0 0x7f5bb006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:03.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.750+0000 7f5bd5c00700 1 --2- 172.21.15.89:0/3631459010 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5bb006adb0 0x7f5bb006d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40011e0 tx=0x7f5bc4009040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:03.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.753+0000 7f5bcd600700 1 -- 172.21.15.89:0/3631459010 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5bbc0538e0 con 0x7f5bd0160e50 2024-04-02T07:52:03.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:03.971+0000 7f5bd8600700 1 -- 172.21.15.89:0/3631459010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5bd00d2fc0 con 0x7f5bd0160e50 2024-04-02T07:52:04.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.710+0000 7f5bcd600700 1 -- 172.21.15.89:0/3631459010 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v93) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5bbc017080 con 0x7f5bd0160e50 2024-04-02T07:52:04.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.714+0000 7f5bd8600700 1 -- 172.21.15.89:0/3631459010 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5bb006adb0 msgr2=0x7f5bb006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.714+0000 7f5bd8600700 1 --2- 172.21.15.89:0/3631459010 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5bb006adb0 0x7f5bb006d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40011e0 tx=0x7f5bc4009040 comp rx=0 tx=0).stop 2024-04-02T07:52:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.714+0000 7f5bd8600700 1 -- 172.21.15.89:0/3631459010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bd0160e50 msgr2=0x7f5bd00b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.714+0000 7f5bd8600700 1 --2- 172.21.15.89:0/3631459010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bd0160e50 0x7f5bd00b6150 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc00c700 tx=0x7f5bbc00ca10 comp rx=0 tx=0).stop 2024-04-02T07:52:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.714+0000 7f5bd8600700 1 -- 172.21.15.89:0/3631459010 shutdown_connections 2024-04-02T07:52:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.714+0000 7f5bd8600700 1 --2- 172.21.15.89:0/3631459010 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5bb006adb0 0x7f5bb006d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.714+0000 7f5bd8600700 1 --2- 172.21.15.89:0/3631459010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd00b3f80 0x7f5bd015acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.714+0000 7f5bd8600700 1 --2- 172.21.15.89:0/3631459010 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd00b6690 0x7f5bd00b0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.714+0000 7f5bd8600700 1 --2- 172.21.15.89:0/3631459010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bd0160e50 0x7f5bd00b6150 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.714+0000 7f5bd8600700 1 -- 172.21.15.89:0/3631459010 >> 172.21.15.89:0/3631459010 conn(0x7f5bd008aff0 msgr2=0x7f5bd00b03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.714+0000 7f5bd8600700 1 -- 172.21.15.89:0/3631459010 shutdown_connections 2024-04-02T07:52:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.714+0000 7f5bd8600700 1 -- 172.21.15.89:0/3631459010 wait complete. 2024-04-02T07:52:04.718 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T07:52:04.738 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T07:52:04.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.931+0000 7f66fca00700 1 Processor -- start 2024-04-02T07:52:04.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.932+0000 7f66fca00700 1 -- start start 2024-04-02T07:52:04.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.933+0000 7f66fca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f80d18b0 0x7f66f80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:04.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.933+0000 7f66fca00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f66f8161590 0x7f66f8161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:04.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.933+0000 7f66fca00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f80ceda0 con 0x7f66f8160f10 2024-04-02T07:52:04.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.933+0000 7f66fca00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66f80d1d30 con 0x7f66f8161590 2024-04-02T07:52:04.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.933+0000 7f66fca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f8161f60 con 0x7f66f80d18b0 2024-04-02T07:52:04.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.933+0000 7f66f5e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f66f8160f10 0x7f66f80d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48990/0 (socket says 172.21.15.89:48990) 2024-04-02T07:52:04.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.933+0000 7f66f5e00700 1 -- 172.21.15.89:0/1572112299 learned_addr learned my addr 172.21.15.89:0/1572112299 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:04.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.933+0000 7f66f6800700 1 --2- 172.21.15.89:0/1572112299 >> v2:172.21.15.89:3300/0 conn(0x7f66f8161590 0x7f66f8161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:04.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.933+0000 7f66f5400700 1 --2- 172.21.15.89:0/1572112299 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f80d18b0 0x7f66f80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:04.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.933+0000 7f66f4a00700 1 -- 172.21.15.89:0/1572112299 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231955453 0 0) 0x7f66f80ceda0 con 0x7f66f8160f10 2024-04-02T07:52:04.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.933+0000 7f66f4a00700 1 -- 172.21.15.89:0/1572112299 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66dc0035b0 con 0x7f66f8160f10 2024-04-02T07:52:04.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.934+0000 7f66f6800700 1 -- 172.21.15.89:0/1572112299 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f80d18b0 msgr2=0x7f66f80cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:04.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.934+0000 7f66f6800700 1 --2- 172.21.15.89:0/1572112299 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f80d18b0 0x7f66f80cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:04.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.934+0000 7f66f6800700 1 -- 172.21.15.89:0/1572112299 >> v1:172.21.15.89:6789/0 conn(0x7f66f8160f10 legacy=0x7f66f80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:04.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.934+0000 7f66f6800700 1 -- 172.21.15.89:0/1572112299 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f81612f0 con 0x7f66f8161590 2024-04-02T07:52:04.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.934+0000 7f66f6800700 1 --2- 172.21.15.89:0/1572112299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f8161590 0x7f66f8161a20 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f66f0009dc0 tx=0x7f66f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a347dfbafbc324 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:04.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.934+0000 7f66f4a00700 1 -- 172.21.15.89:0/1572112299 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f000e070 con 0x7f66f8161590 2024-04-02T07:52:04.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.934+0000 7f66f4a00700 1 -- 172.21.15.89:0/1572112299 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f000bae0 con 0x7f66f8161590 2024-04-02T07:52:04.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.934+0000 7f66f4a00700 1 -- 172.21.15.89:0/1572112299 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f000be90 con 0x7f66f8161590 2024-04-02T07:52:04.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.935+0000 7f66fca00700 1 -- 172.21.15.89:0/1572112299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f8161590 msgr2=0x7f66f8161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:04.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.935+0000 7f66fca00700 1 --2- 172.21.15.89:0/1572112299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f8161590 0x7f66f8161a20 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f66f0009dc0 tx=0x7f66f000b330 comp rx=0 tx=0).stop 2024-04-02T07:52:04.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.935+0000 7f66fca00700 1 -- 172.21.15.89:0/1572112299 shutdown_connections 2024-04-02T07:52:04.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.935+0000 7f66fca00700 1 --2- 172.21.15.89:0/1572112299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f8161590 0x7f66f8161a20 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:04.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.935+0000 7f66fca00700 1 --2- 172.21.15.89:0/1572112299 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f80d18b0 0x7f66f80cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:04.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.935+0000 7f66fca00700 1 -- 172.21.15.89:0/1572112299 >> 172.21.15.89:0/1572112299 conn(0x7f66f808aff0 msgr2=0x7f66f80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:04.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.935+0000 7f66fca00700 1 -- 172.21.15.89:0/1572112299 shutdown_connections 2024-04-02T07:52:04.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.935+0000 7f66fca00700 1 -- 172.21.15.89:0/1572112299 wait complete. 2024-04-02T07:52:04.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.936+0000 7f66fca00700 1 Processor -- start 2024-04-02T07:52:04.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.937+0000 7f66fca00700 1 -- start start 2024-04-02T07:52:04.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.937+0000 7f66fca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f80d18b0 0x7f66f8158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:04.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.937+0000 7f66fca00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f8160f10 0x7f66f80b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:04.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.937+0000 7f66fca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f8161590 0x7f66f806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:04.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.937+0000 7f66fca00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f8158fb0 con 0x7f66f80d18b0 2024-04-02T07:52:04.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.937+0000 7f66fca00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f80b7170 con 0x7f66f8160f10 2024-04-02T07:52:04.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.937+0000 7f66fca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f81f3e80 con 0x7f66f8161590 2024-04-02T07:52:04.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66f5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f80d18b0 0x7f66f8158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:04.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66f5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f80d18b0 0x7f66f8158a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53126/0 (socket says 172.21.15.89:53126) 2024-04-02T07:52:04.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66f5e00700 1 -- 172.21.15.89:0/1525902028 learned_addr learned my addr 172.21.15.89:0/1525902028 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:04.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66f6800700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f8161590 0x7f66f806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:04.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66f5e00700 1 -- 172.21.15.89:0/1525902028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f8161590 msgr2=0x7f66f806c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:04.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66f5400700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f8160f10 0x7f66f80b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:04.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66f5e00700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f8161590 0x7f66f806c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:04.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66f5e00700 1 -- 172.21.15.89:0/1525902028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f8160f10 msgr2=0x7f66f80b7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:04.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66f5e00700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f8160f10 0x7f66f80b7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:04.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66f5e00700 1 -- 172.21.15.89:0/1525902028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f0009a70 con 0x7f66f80d18b0 2024-04-02T07:52:04.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66f6800700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f8161590 0x7f66f806c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:04.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66f5400700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f8160f10 0x7f66f80b7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:04.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66f5e00700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f80d18b0 0x7f66f8158a70 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f66e8001eb0 tx=0x7f66e800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:04.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66ee000700 1 -- 172.21.15.89:0/1525902028 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e8005d70 con 0x7f66f80d18b0 2024-04-02T07:52:04.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66ee000700 1 -- 172.21.15.89:0/1525902028 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e80070d0 con 0x7f66f80d18b0 2024-04-02T07:52:04.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66ee000700 1 -- 172.21.15.89:0/1525902028 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e8007460 con 0x7f66f80d18b0 2024-04-02T07:52:04.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.938+0000 7f66fca00700 1 -- 172.21.15.89:0/1525902028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f81f4160 con 0x7f66f80d18b0 2024-04-02T07:52:04.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.939+0000 7f66fca00700 1 -- 172.21.15.89:0/1525902028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f80b7710 con 0x7f66f80d18b0 2024-04-02T07:52:04.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.940+0000 7f66fca00700 1 -- 172.21.15.89:0/1525902028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f80cf710 con 0x7f66f80d18b0 2024-04-02T07:52:04.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.941+0000 7f66ee000700 1 -- 172.21.15.89:0/1525902028 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f66e80075c0 con 0x7f66f80d18b0 2024-04-02T07:52:04.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.941+0000 7f66ee000700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f66d806adb0 0x7f66d806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:04.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.941+0000 7f66ee000700 1 -- 172.21.15.89:0/1525902028 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f66e8088890 con 0x7f66f80d18b0 2024-04-02T07:52:04.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.942+0000 7f66f5400700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f66d806adb0 0x7f66d806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:04.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.942+0000 7f66f5400700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f66d806adb0 0x7f66d806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f66f80a15e0 tx=0x7f66e4008040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:04.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:04.946+0000 7f66ee000700 1 -- 172.21.15.89:0/1525902028 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f66e8053ab0 con 0x7f66f80d18b0 2024-04-02T07:52:05.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.221+0000 7f66fca00700 1 -- 172.21.15.89:0/1525902028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f66f8067720 con 0x7f66f80d18b0 2024-04-02T07:52:05.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.221+0000 7f66ee000700 1 -- 172.21.15.89:0/1525902028 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f66e8017090 con 0x7f66f80d18b0 2024-04-02T07:52:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:05.224 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:52:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T07:52:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T07:52:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T07:52:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:52:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:52:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T07:52:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T07:52:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:52:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:52:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T07:52:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T07:52:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:52:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:52:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T07:52:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T07:52:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:52:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:52:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T07:52:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T07:52:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T07:52:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T07:52:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T07:52:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T07:52:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T07:52:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T07:52:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T07:52:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T07:52:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T07:52:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T07:52:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T07:52:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T07:52:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2024-04-02T07:52:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T07:52:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T07:52:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T07:52:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:52:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:52:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:52:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T07:52:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T07:52:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:52:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:52:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:52:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T07:52:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T07:52:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:52:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:52:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:52:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T07:52:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T07:52:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:52:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:52:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:52:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T07:52:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T07:52:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:52:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:52:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T07:52:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T07:52:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:52:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:52:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T07:52:05.237 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:52:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.223+0000 7f66fca00700 1 -- 172.21.15.89:0/1525902028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f66d806adb0 msgr2=0x7f66d806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.223+0000 7f66fca00700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f66d806adb0 0x7f66d806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f66f80a15e0 tx=0x7f66e4008040 comp rx=0 tx=0).stop 2024-04-02T07:52:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.223+0000 7f66fca00700 1 -- 172.21.15.89:0/1525902028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f80d18b0 msgr2=0x7f66f8158a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.223+0000 7f66fca00700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f80d18b0 0x7f66f8158a70 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f66e8001eb0 tx=0x7f66e800c860 comp rx=0 tx=0).stop 2024-04-02T07:52:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.224+0000 7f66fca00700 1 -- 172.21.15.89:0/1525902028 shutdown_connections 2024-04-02T07:52:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.224+0000 7f66fca00700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f66d806adb0 0x7f66d806d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.224+0000 7f66fca00700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f8161590 0x7f66f806c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.224+0000 7f66fca00700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f80d18b0 0x7f66f8158a70 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.224+0000 7f66fca00700 1 --2- 172.21.15.89:0/1525902028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f8160f10 0x7f66f80b7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.224+0000 7f66fca00700 1 -- 172.21.15.89:0/1525902028 >> 172.21.15.89:0/1525902028 conn(0x7f66f808aff0 msgr2=0x7f66f80cab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.224+0000 7f66fca00700 1 -- 172.21.15.89:0/1525902028 shutdown_connections 2024-04-02T07:52:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.224+0000 7f66fca00700 1 -- 172.21.15.89:0/1525902028 wait complete. 2024-04-02T07:52:05.245 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:52:05.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.433+0000 7f5076000700 1 Processor -- start 2024-04-02T07:52:05.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.434+0000 7f5076000700 1 -- start start 2024-04-02T07:52:05.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.435+0000 7f5076000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f50700ceec0 0x7f50701612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:05.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.435+0000 7f5076000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5070161c50 con 0x7f507015d640 2024-04-02T07:52:05.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.435+0000 7f5076000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50700d30d0 con 0x7f5070161800 2024-04-02T07:52:05.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.435+0000 7f5076000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50700b6840 con 0x7f50700ceec0 2024-04-02T07:52:05.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.435+0000 7f506f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f507015d640 0x7f50700d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48996/0 (socket says 172.21.15.89:48996) 2024-04-02T07:52:05.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.435+0000 7f506f400700 1 -- 172.21.15.89:0/528327619 learned_addr learned my addr 172.21.15.89:0/528327619 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:05.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.435+0000 7f506ea00700 1 --2- 172.21.15.89:0/528327619 >> v2:172.21.15.113:3300/0 conn(0x7f50700ceec0 0x7f50701612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:05.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.435+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 386327110 0 0) 0x7f5060001110 con 0x7f507015d640 2024-04-02T07:52:05.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.435+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50540035b0 con 0x7f507015d640 2024-04-02T07:52:05.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235306361 0 0) 0x7f50700d30d0 con 0x7f5070161800 2024-04-02T07:52:05.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5060001110 con 0x7f5070161800 2024-04-02T07:52:05.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3712979903 0 0) 0x7f50540035b0 con 0x7f507015d640 2024-04-02T07:52:05.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 >> v2:172.21.15.113:3300/0 conn(0x7f50700ceec0 msgr2=0x7f50701612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 --2- 172.21.15.89:0/528327619 >> v2:172.21.15.113:3300/0 conn(0x7f50700ceec0 0x7f50701612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 >> v1:172.21.15.113:6789/0 conn(0x7f5070161800 legacy=0x7f50700b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50700d2b30 con 0x7f507015d640 2024-04-02T07:52:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f5060001800 con 0x7f507015d640 2024-04-02T07:52:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 >> v1:172.21.15.89:6789/0 conn(0x7f507015d640 legacy=0x7f50700d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 --2- 172.21.15.89:0/528327619 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5054003b90 0x7f5054003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:05.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 --2- 172.21.15.89:0/528327619 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5054004850 0x7f5054004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:05.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 --2- 172.21.15.89:0/528327619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5054005210 0x7f505400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:05.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50700b6840 con 0x7f5054005210 2024-04-02T07:52:05.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505400c140 con 0x7f5054003b90 2024-04-02T07:52:05.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f505400c300 con 0x7f5054004850 2024-04-02T07:52:05.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506fe00700 1 --2- 172.21.15.89:0/528327619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5054005210 0x7f505400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:05.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506ea00700 1 --2- 172.21.15.89:0/528327619 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5054004850 0x7f5054004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:05.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506f400700 1 --2- 172.21.15.89:0/528327619 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5054003b90 0x7f5054003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:05.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506fe00700 1 -- 172.21.15.89:0/528327619 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5054004850 msgr2=0x7f5054004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:05.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506fe00700 1 --2- 172.21.15.89:0/528327619 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5054004850 0x7f5054004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506fe00700 1 -- 172.21.15.89:0/528327619 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5054003b90 msgr2=0x7f5054003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:05.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506fe00700 1 --2- 172.21.15.89:0/528327619 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5054003b90 0x7f5054003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.436+0000 7f506fe00700 1 -- 172.21.15.89:0/528327619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50540035b0 con 0x7f5054005210 2024-04-02T07:52:05.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.437+0000 7f506fe00700 1 -- 172.21.15.89:0/528327619 reap_dead start 2024-04-02T07:52:05.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.437+0000 7f506fe00700 1 --2- 172.21.15.89:0/528327619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5054005210 0x7f505400ba10 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f506800c4c0 tx=0x7f5054005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a402fd41f078b081 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:05.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.437+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068003a00 con 0x7f5054005210 2024-04-02T07:52:05.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.437+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5068003b80 con 0x7f5054005210 2024-04-02T07:52:05.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.437+0000 7f506e000700 1 -- 172.21.15.89:0/528327619 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50680050d0 con 0x7f5054005210 2024-04-02T07:52:05.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.437+0000 7f5076000700 1 -- 172.21.15.89:0/528327619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5054005210 msgr2=0x7f505400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:05.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.437+0000 7f5076000700 1 --2- 172.21.15.89:0/528327619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5054005210 0x7f505400ba10 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f506800c4c0 tx=0x7f5054005880 comp rx=0 tx=0).stop 2024-04-02T07:52:05.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.437+0000 7f5076000700 1 -- 172.21.15.89:0/528327619 shutdown_connections 2024-04-02T07:52:05.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.437+0000 7f5076000700 1 --2- 172.21.15.89:0/528327619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5054005210 0x7f505400ba10 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.437+0000 7f5076000700 1 -- 172.21.15.89:0/528327619 >> 172.21.15.89:0/528327619 conn(0x7f507008aff0 msgr2=0x7f50700d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:05.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.438+0000 7f5076000700 1 -- 172.21.15.89:0/528327619 shutdown_connections 2024-04-02T07:52:05.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.438+0000 7f5076000700 1 -- 172.21.15.89:0/528327619 wait complete. 2024-04-02T07:52:05.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.439+0000 7f5076000700 1 Processor -- start 2024-04-02T07:52:05.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.440+0000 7f5076000700 1 -- start start 2024-04-02T07:52:05.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.440+0000 7f5076000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5054005210 0x7f50700c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:05.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.440+0000 7f5076000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5070158710 0x7f50700c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:05.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.440+0000 7f5076000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50701f31a0 0x7f50701f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:05.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.440+0000 7f5076000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50700c7ea0 con 0x7f5070158710 2024-04-02T07:52:05.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.440+0000 7f5076000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5070157730 con 0x7f50701f31a0 2024-04-02T07:52:05.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.440+0000 7f5076000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50700c99e0 con 0x7f5054005210 2024-04-02T07:52:05.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.441+0000 7f506ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5070158710 0x7f50700c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:05.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.441+0000 7f506ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5070158710 0x7f50700c7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53158/0 (socket says 172.21.15.89:53158) 2024-04-02T07:52:05.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.441+0000 7f506ea00700 1 -- 172.21.15.89:0/50303603 learned_addr learned my addr 172.21.15.89:0/50303603 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:05.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.441+0000 7f506fe00700 1 --2- 172.21.15.89:0/50303603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50701f31a0 0x7f50701f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:05.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.441+0000 7f506f400700 1 --2- 172.21.15.89:0/50303603 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5054005210 0x7f50700c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.441+0000 7f506ea00700 1 -- 172.21.15.89:0/50303603 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5054005210 msgr2=0x7f50700c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.441+0000 7f506ea00700 1 --2- 172.21.15.89:0/50303603 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5054005210 0x7f50700c7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.441+0000 7f506ea00700 1 -- 172.21.15.89:0/50303603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50701f31a0 msgr2=0x7f50701f3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.441+0000 7f506ea00700 1 --2- 172.21.15.89:0/50303603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50701f31a0 0x7f50701f3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.441+0000 7f506ea00700 1 -- 172.21.15.89:0/50303603 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5070156e00 con 0x7f5070158710 2024-04-02T07:52:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.441+0000 7f506ea00700 1 --2- 172.21.15.89:0/50303603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5070158710 0x7f50700c7580 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f505c0119d0 tx=0x7f505c011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.442+0000 7f5067400700 1 -- 172.21.15.89:0/50303603 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c00f940 con 0x7f5070158710 2024-04-02T07:52:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.442+0000 7f5067400700 1 -- 172.21.15.89:0/50303603 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505c00faa0 con 0x7f5070158710 2024-04-02T07:52:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.442+0000 7f5067400700 1 -- 172.21.15.89:0/50303603 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c00fe30 con 0x7f5070158710 2024-04-02T07:52:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.442+0000 7f5076000700 1 -- 172.21.15.89:0/50303603 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50701570e0 con 0x7f5070158710 2024-04-02T07:52:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.442+0000 7f5076000700 1 -- 172.21.15.89:0/50303603 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5070157bc0 con 0x7f5070158710 2024-04-02T07:52:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.443+0000 7f5076000700 1 -- 172.21.15.89:0/50303603 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50701578c0 con 0x7f5070158710 2024-04-02T07:52:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.444+0000 7f5067400700 1 -- 172.21.15.89:0/50303603 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f505c014440 con 0x7f5070158710 2024-04-02T07:52:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.445+0000 7f5067400700 1 --2- 172.21.15.89:0/50303603 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f505006adb0 0x7f505006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.445+0000 7f5067400700 1 -- 172.21.15.89:0/50303603 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f505c089a80 con 0x7f5070158710 2024-04-02T07:52:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.445+0000 7f506f400700 1 --2- 172.21.15.89:0/50303603 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f505006adb0 0x7f505006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.445+0000 7f506f400700 1 --2- 172.21.15.89:0/50303603 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f505006adb0 0x7f505006d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f50600017a0 tx=0x7f5060009040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:05.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.449+0000 7f5067400700 1 -- 172.21.15.89:0/50303603 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f505c054d50 con 0x7f5070158710 2024-04-02T07:52:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.654+0000 7f5076000700 1 -- 172.21.15.89:0/50303603 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5070067720 con 0x7f5070158710 2024-04-02T07:52:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.655+0000 7f5067400700 1 -- 172.21.15.89:0/50303603 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+12916 (secure 0 0 0) 0x7f505c058370 con 0x7f5070158710 2024-04-02T07:52:05.661 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:05.661 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":93,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:52:04.707254+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:52:05.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.657+0000 7f5076000700 1 -- 172.21.15.89:0/50303603 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f505006adb0 msgr2=0x7f505006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:05.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.657+0000 7f5076000700 1 --2- 172.21.15.89:0/50303603 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f505006adb0 0x7f505006d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f50600017a0 tx=0x7f5060009040 comp rx=0 tx=0).stop 2024-04-02T07:52:05.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.657+0000 7f5076000700 1 -- 172.21.15.89:0/50303603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5070158710 msgr2=0x7f50700c7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:05.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.657+0000 7f5076000700 1 --2- 172.21.15.89:0/50303603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5070158710 0x7f50700c7580 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f505c0119d0 tx=0x7f505c011d90 comp rx=0 tx=0).stop 2024-04-02T07:52:05.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.657+0000 7f5076000700 1 -- 172.21.15.89:0/50303603 shutdown_connections 2024-04-02T07:52:05.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.657+0000 7f5076000700 1 --2- 172.21.15.89:0/50303603 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f505006adb0 0x7f505006d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.657+0000 7f5076000700 1 --2- 172.21.15.89:0/50303603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50701f31a0 0x7f50701f3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.657+0000 7f5076000700 1 --2- 172.21.15.89:0/50303603 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5054005210 0x7f50700c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.657+0000 7f5076000700 1 --2- 172.21.15.89:0/50303603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5070158710 0x7f50700c7580 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.657+0000 7f5076000700 1 -- 172.21.15.89:0/50303603 >> 172.21.15.89:0/50303603 conn(0x7f507008aff0 msgr2=0x7f507006d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:05.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.658+0000 7f5076000700 1 -- 172.21.15.89:0/50303603 shutdown_connections 2024-04-02T07:52:05.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.658+0000 7f5076000700 1 -- 172.21.15.89:0/50303603 wait complete. 2024-04-02T07:52:05.679 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:52:05.680 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:52:05.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.874+0000 7f0fa0c00700 1 Processor -- start 2024-04-02T07:52:05.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.874+0000 7f0fa0c00700 1 -- start start 2024-04-02T07:52:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.875+0000 7f0fa0c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f9c0d3f20 0x7f0f9c0cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.875+0000 7f0fa0c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f9c0b3f80 0x7f0f9c0ce430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.875+0000 7f0fa0c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c161230 con 0x7f0f9c160e50 2024-04-02T07:52:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.875+0000 7f0fa0c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9c0d2a80 con 0x7f0f9c0b3f80 2024-04-02T07:52:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.875+0000 7f0fa0c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c0d43b0 con 0x7f0f9c0d3f20 2024-04-02T07:52:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.875+0000 7f0f9a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f9c0b3f80 0x7f0f9c0ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.875+0000 7f0f9a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f9c0b3f80 0x7f0f9c0ce430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53172/0 (socket says 172.21.15.89:53172) 2024-04-02T07:52:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.875+0000 7f0f9a800700 1 -- 172.21.15.89:0/2134091767 learned_addr learned my addr 172.21.15.89:0/2134091767 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.876+0000 7f0f9a800700 1 -- 172.21.15.89:0/2134091767 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f9c0d3f20 msgr2=0x7f0f9c0cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.876+0000 7f0f99400700 1 --2- 172.21.15.89:0/2134091767 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f9c0d3f20 0x7f0f9c0cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.876+0000 7f0f9a800700 1 --2- 172.21.15.89:0/2134091767 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f9c0d3f20 0x7f0f9c0cdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.876+0000 7f0f9a800700 1 -- 172.21.15.89:0/2134091767 >> v1:172.21.15.113:6789/0 conn(0x7f0f9c160e50 legacy=0x7f0f9c0b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.876+0000 7f0f9a800700 1 -- 172.21.15.89:0/2134091767 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c0d21e0 con 0x7f0f9c0b3f80 2024-04-02T07:52:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.876+0000 7f0f99400700 1 --2- 172.21.15.89:0/2134091767 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f9c0d3f20 0x7f0f9c0cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.876+0000 7f0f9a800700 1 --2- 172.21.15.89:0/2134091767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c0b3f80 0x7f0f9c0ce430 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0f94009e50 tx=0x7f0f9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e23ccaf081759cd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.876+0000 7f0f98a00700 1 -- 172.21.15.89:0/2134091767 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9400e070 con 0x7f0f9c0b3f80 2024-04-02T07:52:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.876+0000 7f0f98a00700 1 -- 172.21.15.89:0/2134091767 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f94003a20 con 0x7f0f9c0b3f80 2024-04-02T07:52:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.876+0000 7f0f98a00700 1 -- 172.21.15.89:0/2134091767 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f94003dd0 con 0x7f0f9c0b3f80 2024-04-02T07:52:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.877+0000 7f0fa0c00700 1 -- 172.21.15.89:0/2134091767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c0b3f80 msgr2=0x7f0f9c0ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.877+0000 7f0fa0c00700 1 --2- 172.21.15.89:0/2134091767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c0b3f80 0x7f0f9c0ce430 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0f94009e50 tx=0x7f0f9400b330 comp rx=0 tx=0).stop 2024-04-02T07:52:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.877+0000 7f0fa0c00700 1 -- 172.21.15.89:0/2134091767 shutdown_connections 2024-04-02T07:52:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.877+0000 7f0fa0c00700 1 --2- 172.21.15.89:0/2134091767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c0b3f80 0x7f0f9c0ce430 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.877+0000 7f0fa0c00700 1 --2- 172.21.15.89:0/2134091767 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f9c0d3f20 0x7f0f9c0cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.877+0000 7f0fa0c00700 1 -- 172.21.15.89:0/2134091767 >> 172.21.15.89:0/2134091767 conn(0x7f0f9c08aff0 msgr2=0x7f0f9c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.877+0000 7f0fa0c00700 1 -- 172.21.15.89:0/2134091767 shutdown_connections 2024-04-02T07:52:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.877+0000 7f0fa0c00700 1 -- 172.21.15.89:0/2134091767 wait complete. 2024-04-02T07:52:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.878+0000 7f0fa0c00700 1 Processor -- start 2024-04-02T07:52:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.879+0000 7f0fa0c00700 1 -- start start 2024-04-02T07:52:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.879+0000 7f0fa0c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f9c0b3f80 0x7f0f9c0c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.879+0000 7f0fa0c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f9c160e50 0x7f0f9c0b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0fa0c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c15bbe0 0x7f0f9c06bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0fa0c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c0c91f0 con 0x7f0f9c15bbe0 2024-04-02T07:52:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0fa0c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c0b9150 con 0x7f0f9c160e50 2024-04-02T07:52:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0fa0c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c0c8570 con 0x7f0f9c0b3f80 2024-04-02T07:52:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0f9a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c15bbe0 0x7f0f9c06bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0f9a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c15bbe0 0x7f0f9c06bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53182/0 (socket says 172.21.15.89:53182) 2024-04-02T07:52:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0f9a800700 1 -- 172.21.15.89:0/4146762609 learned_addr learned my addr 172.21.15.89:0/4146762609 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0f99400700 1 --2- 172.21.15.89:0/4146762609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f9c160e50 0x7f0f9c0b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0f99e00700 1 --2- 172.21.15.89:0/4146762609 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f9c0b3f80 0x7f0f9c0c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0f9a800700 1 -- 172.21.15.89:0/4146762609 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f9c0b3f80 msgr2=0x7f0f9c0c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0f9a800700 1 --2- 172.21.15.89:0/4146762609 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f9c0b3f80 0x7f0f9c0c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0f9a800700 1 -- 172.21.15.89:0/4146762609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f9c160e50 msgr2=0x7f0f9c0b8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0f9a800700 1 --2- 172.21.15.89:0/4146762609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f9c160e50 0x7f0f9c0b8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0f9a800700 1 -- 172.21.15.89:0/4146762609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f94009b30 con 0x7f0f9c15bbe0 2024-04-02T07:52:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.880+0000 7f0f9a800700 1 --2- 172.21.15.89:0/4146762609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c15bbe0 0x7f0f9c06bf90 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f0f94004000 tx=0x7f0f94004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.881+0000 7f0f92000700 1 -- 172.21.15.89:0/4146762609 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f94004440 con 0x7f0f9c15bbe0 2024-04-02T07:52:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.881+0000 7f0f92000700 1 -- 172.21.15.89:0/4146762609 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f940045a0 con 0x7f0f9c15bbe0 2024-04-02T07:52:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.881+0000 7f0f92000700 1 -- 172.21.15.89:0/4146762609 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f94004950 con 0x7f0f9c15bbe0 2024-04-02T07:52:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.881+0000 7f0fa0c00700 1 -- 172.21.15.89:0/4146762609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c0c8850 con 0x7f0f9c15bbe0 2024-04-02T07:52:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.881+0000 7f0fa0c00700 1 -- 172.21.15.89:0/4146762609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c0c8e10 con 0x7f0f9c15bbe0 2024-04-02T07:52:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.882+0000 7f0fa0c00700 1 -- 172.21.15.89:0/4146762609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c0c9330 con 0x7f0f9c15bbe0 2024-04-02T07:52:05.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.883+0000 7f0f92000700 1 -- 172.21.15.89:0/4146762609 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0f94012430 con 0x7f0f9c15bbe0 2024-04-02T07:52:05.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.883+0000 7f0f92000700 1 --2- 172.21.15.89:0/4146762609 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f7c06adb0 0x7f0f7c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:05.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.884+0000 7f0f92000700 1 -- 172.21.15.89:0/4146762609 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f0f9408a8d0 con 0x7f0f9c15bbe0 2024-04-02T07:52:05.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.884+0000 7f0f99e00700 1 --2- 172.21.15.89:0/4146762609 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f7c06adb0 0x7f0f7c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:05.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.884+0000 7f0f99e00700 1 --2- 172.21.15.89:0/4146762609 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f7c06adb0 0x7f0f7c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c009870 tx=0x7f0f8c008040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:05.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:05.888+0000 7f0f92000700 1 -- 172.21.15.89:0/4146762609 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0f9c0c9330 con 0x7f0f9c15bbe0 2024-04-02T07:52:06.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.087+0000 7f0fa0c00700 1 -- 172.21.15.89:0/4146762609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0f9c0c9330 con 0x7f0f9c15bbe0 2024-04-02T07:52:06.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.088+0000 7f0f92000700 1 -- 172.21.15.89:0/4146762609 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0f9c0c9330 con 0x7f0f9c15bbe0 2024-04-02T07:52:06.090 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:52:06.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.090+0000 7f0fa0c00700 1 -- 172.21.15.89:0/4146762609 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f7c06adb0 msgr2=0x7f0f7c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:06.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.090+0000 7f0fa0c00700 1 --2- 172.21.15.89:0/4146762609 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f7c06adb0 0x7f0f7c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c009870 tx=0x7f0f8c008040 comp rx=0 tx=0).stop 2024-04-02T07:52:06.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.090+0000 7f0fa0c00700 1 -- 172.21.15.89:0/4146762609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c15bbe0 msgr2=0x7f0f9c06bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:06.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.090+0000 7f0fa0c00700 1 --2- 172.21.15.89:0/4146762609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c15bbe0 0x7f0f9c06bf90 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f0f94004000 tx=0x7f0f94004ad0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.090+0000 7f0fa0c00700 1 -- 172.21.15.89:0/4146762609 shutdown_connections 2024-04-02T07:52:06.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.090+0000 7f0fa0c00700 1 --2- 172.21.15.89:0/4146762609 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f7c06adb0 0x7f0f7c06d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.090+0000 7f0fa0c00700 1 --2- 172.21.15.89:0/4146762609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c15bbe0 0x7f0f9c06bf90 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.090+0000 7f0fa0c00700 1 --2- 172.21.15.89:0/4146762609 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f9c0b3f80 0x7f0f9c0c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.090+0000 7f0fa0c00700 1 --2- 172.21.15.89:0/4146762609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f9c160e50 0x7f0f9c0b8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.090+0000 7f0fa0c00700 1 -- 172.21.15.89:0/4146762609 >> 172.21.15.89:0/4146762609 conn(0x7f0f9c08aff0 msgr2=0x7f0f9c0ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:06.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.090+0000 7f0fa0c00700 1 -- 172.21.15.89:0/4146762609 shutdown_connections 2024-04-02T07:52:06.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.090+0000 7f0fa0c00700 1 -- 172.21.15.89:0/4146762609 wait complete. 2024-04-02T07:52:06.111 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T07:52:06.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.301+0000 7f4793400700 1 Processor -- start 2024-04-02T07:52:06.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.301+0000 7f4793400700 1 -- start start 2024-04-02T07:52:06.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.302+0000 7f4793400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f478c15d670 0x7f478c0ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:06.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.302+0000 7f4793400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f478c06dee0 0x7f478c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:06.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.302+0000 7f4793400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478c0b3890 con 0x7f478c15da50 2024-04-02T07:52:06.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.302+0000 7f4793400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f478c0acac0 con 0x7f478c06dee0 2024-04-02T07:52:06.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.302+0000 7f4793400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f478c159f50 con 0x7f478c15d670 2024-04-02T07:52:06.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.302+0000 7f478be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f478c15da50 0x7f478c0b2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48998/0 (socket says 172.21.15.89:48998) 2024-04-02T07:52:06.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.302+0000 7f478be00700 1 -- 172.21.15.89:0/2449860656 learned_addr learned my addr 172.21.15.89:0/2449860656 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:06.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.302+0000 7f4791400700 1 --2- 172.21.15.89:0/2449860656 >> v2:172.21.15.89:3300/0 conn(0x7f478c06dee0 0x7f478c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:06.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.303+0000 7f4790a00700 1 --2- 172.21.15.89:0/2449860656 >> v2:172.21.15.113:3300/0 conn(0x7f478c15d670 0x7f478c0ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:06.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.303+0000 7f478b400700 1 -- 172.21.15.89:0/2449860656 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781762773 0 0) 0x7f478c0b3890 con 0x7f478c15da50 2024-04-02T07:52:06.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.303+0000 7f478b400700 1 -- 172.21.15.89:0/2449860656 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47740057c0 con 0x7f478c15da50 2024-04-02T07:52:06.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.303+0000 7f4791400700 1 -- 172.21.15.89:0/2449860656 >> v2:172.21.15.113:3300/0 conn(0x7f478c15d670 msgr2=0x7f478c0ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:06.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.303+0000 7f4791400700 1 --2- 172.21.15.89:0/2449860656 >> v2:172.21.15.113:3300/0 conn(0x7f478c15d670 0x7f478c0ac580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.303+0000 7f4791400700 1 -- 172.21.15.89:0/2449860656 >> v1:172.21.15.89:6789/0 conn(0x7f478c15da50 legacy=0x7f478c0b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:06.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.303+0000 7f4791400700 1 -- 172.21.15.89:0/2449860656 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478c0b35b0 con 0x7f478c06dee0 2024-04-02T07:52:06.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.303+0000 7f4790a00700 1 --2- 172.21.15.89:0/2449860656 >> v2:172.21.15.113:3300/0 conn(0x7f478c15d670 0x7f478c0ac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:06.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.303+0000 7f4791400700 1 --2- 172.21.15.89:0/2449860656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f478c06dee0 0x7f478c06e350 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f478c070090 tx=0x7f478400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5c4d358232195a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:06.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.303+0000 7f478b400700 1 -- 172.21.15.89:0/2449860656 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478400e070 con 0x7f478c06dee0 2024-04-02T07:52:06.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.304+0000 7f478b400700 1 -- 172.21.15.89:0/2449860656 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478400bb20 con 0x7f478c06dee0 2024-04-02T07:52:06.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.304+0000 7f478b400700 1 -- 172.21.15.89:0/2449860656 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4784012430 con 0x7f478c06dee0 2024-04-02T07:52:06.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.304+0000 7f4793400700 1 -- 172.21.15.89:0/2449860656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f478c06dee0 msgr2=0x7f478c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:06.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.304+0000 7f4793400700 1 --2- 172.21.15.89:0/2449860656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f478c06dee0 0x7f478c06e350 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f478c070090 tx=0x7f478400b330 comp rx=0 tx=0).stop 2024-04-02T07:52:06.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.304+0000 7f4793400700 1 -- 172.21.15.89:0/2449860656 shutdown_connections 2024-04-02T07:52:06.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.304+0000 7f4793400700 1 --2- 172.21.15.89:0/2449860656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f478c06dee0 0x7f478c06e350 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.304+0000 7f4793400700 1 --2- 172.21.15.89:0/2449860656 >> v2:172.21.15.113:3300/0 conn(0x7f478c15d670 0x7f478c0ac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.304+0000 7f4793400700 1 -- 172.21.15.89:0/2449860656 >> 172.21.15.89:0/2449860656 conn(0x7f478c0cf190 msgr2=0x7f478c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:06.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.304+0000 7f4793400700 1 -- 172.21.15.89:0/2449860656 shutdown_connections 2024-04-02T07:52:06.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.304+0000 7f4793400700 1 -- 172.21.15.89:0/2449860656 wait complete. 2024-04-02T07:52:06.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.306+0000 7f4793400700 1 Processor -- start 2024-04-02T07:52:06.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.306+0000 7f4793400700 1 -- start start 2024-04-02T07:52:06.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.307+0000 7f4793400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c06dee0 0x7f478c098ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:06.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.307+0000 7f4793400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f478c15d670 0x7f478c156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:06.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.307+0000 7f4793400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f478c15da50 0x7f478c156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:06.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.307+0000 7f4793400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c0b2c60 con 0x7f478c15d670 2024-04-02T07:52:06.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.307+0000 7f4793400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c06ebb0 con 0x7f478c06dee0 2024-04-02T07:52:06.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.307+0000 7f4793400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478c151450 con 0x7f478c15da50 2024-04-02T07:52:06.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.307+0000 7f478be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f478c15d670 0x7f478c156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:06.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.307+0000 7f478be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f478c15d670 0x7f478c156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53198/0 (socket says 172.21.15.89:53198) 2024-04-02T07:52:06.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.307+0000 7f478be00700 1 -- 172.21.15.89:0/2095376321 learned_addr learned my addr 172.21.15.89:0/2095376321 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:06.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f4791400700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f478c15da50 0x7f478c156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:06.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f4790a00700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c06dee0 0x7f478c098ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:06.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f478be00700 1 -- 172.21.15.89:0/2095376321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f478c15da50 msgr2=0x7f478c156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:06.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f478be00700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f478c15da50 0x7f478c156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f478be00700 1 -- 172.21.15.89:0/2095376321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c06dee0 msgr2=0x7f478c098ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:06.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f478be00700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c06dee0 0x7f478c098ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f478be00700 1 -- 172.21.15.89:0/2095376321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4784009a70 con 0x7f478c15d670 2024-04-02T07:52:06.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f4791400700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f478c15da50 0x7f478c156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:06.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f4790a00700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c06dee0 0x7f478c098ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:06.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f478be00700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f478c15d670 0x7f478c156960 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f4778001eb0 tx=0x7f477800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:06.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f4788c00700 1 -- 172.21.15.89:0/2095376321 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778005d70 con 0x7f478c15d670 2024-04-02T07:52:06.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f4788c00700 1 -- 172.21.15.89:0/2095376321 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47780070d0 con 0x7f478c15d670 2024-04-02T07:52:06.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f4788c00700 1 -- 172.21.15.89:0/2095376321 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778007460 con 0x7f478c15d670 2024-04-02T07:52:06.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f4793400700 1 -- 172.21.15.89:0/2095376321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478c0b2e70 con 0x7f478c15d670 2024-04-02T07:52:06.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.308+0000 7f4793400700 1 -- 172.21.15.89:0/2095376321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478c15be40 con 0x7f478c15d670 2024-04-02T07:52:06.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.309+0000 7f4793400700 1 -- 172.21.15.89:0/2095376321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478c15bc70 con 0x7f478c15d670 2024-04-02T07:52:06.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.310+0000 7f4788c00700 1 -- 172.21.15.89:0/2095376321 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f47780075c0 con 0x7f478c15d670 2024-04-02T07:52:06.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.311+0000 7f4788c00700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f476c06adb0 0x7f476c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:06.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.311+0000 7f4788c00700 1 -- 172.21.15.89:0/2095376321 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f4778088930 con 0x7f478c15d670 2024-04-02T07:52:06.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.311+0000 7f4790a00700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f476c06adb0 0x7f476c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:06.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.312+0000 7f4790a00700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f476c06adb0 0x7f476c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4780009a00 tx=0x7f4780008040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:06.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.315+0000 7f4788c00700 1 -- 172.21.15.89:0/2095376321 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4778053ad0 con 0x7f478c15d670 2024-04-02T07:52:06.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.518+0000 7f4793400700 1 -- 172.21.15.89:0/2095376321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f478c067720 con 0x7f478c15d670 2024-04-02T07:52:06.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.518+0000 7f4788c00700 1 -- 172.21.15.89:0/2095376321 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v93) v1 ==== 84+0+65 (secure 0 0 0) 0x7f47780570f0 con 0x7f478c15d670 2024-04-02T07:52:06.521 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:06.521 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:52:06.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:06.521 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T07:52:06.521 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T07:52:06.521 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:06.521 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:52:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.521+0000 7f4793400700 1 -- 172.21.15.89:0/2095376321 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f476c06adb0 msgr2=0x7f476c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.521+0000 7f4793400700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f476c06adb0 0x7f476c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4780009a00 tx=0x7f4780008040 comp rx=0 tx=0).stop 2024-04-02T07:52:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.521+0000 7f4793400700 1 -- 172.21.15.89:0/2095376321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f478c15d670 msgr2=0x7f478c156960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.521+0000 7f4793400700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f478c15d670 0x7f478c156960 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f4778001eb0 tx=0x7f477800c860 comp rx=0 tx=0).stop 2024-04-02T07:52:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.521+0000 7f4793400700 1 -- 172.21.15.89:0/2095376321 shutdown_connections 2024-04-02T07:52:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.521+0000 7f4793400700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f476c06adb0 0x7f476c06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.521+0000 7f4793400700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c06dee0 0x7f478c098ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.521+0000 7f4793400700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f478c15da50 0x7f478c156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.521+0000 7f4793400700 1 --2- 172.21.15.89:0/2095376321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f478c15d670 0x7f478c156960 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.521+0000 7f4793400700 1 -- 172.21.15.89:0/2095376321 >> 172.21.15.89:0/2095376321 conn(0x7f478c0cf190 msgr2=0x7f478c06e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:06.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.521+0000 7f4793400700 1 -- 172.21.15.89:0/2095376321 shutdown_connections 2024-04-02T07:52:06.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.521+0000 7f4793400700 1 -- 172.21.15.89:0/2095376321 wait complete. 2024-04-02T07:52:06.541 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T07:52:06.541 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T07:52:06.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.734+0000 7f58d9000700 1 Processor -- start 2024-04-02T07:52:06.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.734+0000 7f58d9000700 1 -- start start 2024-04-02T07:52:06.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.734+0000 7f58d9000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58d415d930 0x7f58d40d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:06.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.734+0000 7f58d9000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f58d4162030 0x7f58d40d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:06.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.734+0000 7f58d9000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f58d40d20d0 0x7f58d40cfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:06.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.734+0000 7f58d9000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d415ec40 con 0x7f58d40d20d0 2024-04-02T07:52:06.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.734+0000 7f58d9000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d40d2560 con 0x7f58d4162030 2024-04-02T07:52:06.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.734+0000 7f58d9000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d415dd10 con 0x7f58d415d930 2024-04-02T07:52:06.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.734+0000 7f58d2800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f58d40d20d0 0x7f58d40cfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:06.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.734+0000 7f58d2800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f58d40d20d0 0x7f58d40cfc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53200/0 (socket says 172.21.15.89:53200) 2024-04-02T07:52:06.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.734+0000 7f58d2800700 1 -- 172.21.15.89:0/663144211 learned_addr learned my addr 172.21.15.89:0/663144211 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:06.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.735+0000 7f58d2800700 1 -- 172.21.15.89:0/663144211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58d415d930 msgr2=0x7f58d40d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:06.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.735+0000 7f58d2800700 1 --2- 172.21.15.89:0/663144211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58d415d930 0x7f58d40d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.735+0000 7f58d2800700 1 -- 172.21.15.89:0/663144211 >> v2:172.21.15.113:3300/0 conn(0x7f58d4162030 msgr2=0x7f58d40d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:06.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.735+0000 7f58d1400700 1 --2- 172.21.15.89:0/663144211 >> v2:172.21.15.113:3300/0 conn(0x7f58d4162030 0x7f58d40d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:06.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.735+0000 7f58d2800700 1 --2- 172.21.15.89:0/663144211 >> v2:172.21.15.113:3300/0 conn(0x7f58d4162030 0x7f58d40d1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.735+0000 7f58d2800700 1 -- 172.21.15.89:0/663144211 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d415dfc0 con 0x7f58d40d20d0 2024-04-02T07:52:06.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.735+0000 7f58d1400700 1 --2- 172.21.15.89:0/663144211 >> v2:172.21.15.113:3300/0 conn(0x7f58d4162030 0x7f58d40d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:06.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.735+0000 7f58d2800700 1 --2- 172.21.15.89:0/663144211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d40d20d0 0x7f58d40cfc00 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f58d408cca0 tx=0x7f58cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea658be737f56857 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:06.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.735+0000 7f58d0a00700 1 -- 172.21.15.89:0/663144211 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc00e070 con 0x7f58d40d20d0 2024-04-02T07:52:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.735+0000 7f58d0a00700 1 -- 172.21.15.89:0/663144211 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58cc00b9c0 con 0x7f58d40d20d0 2024-04-02T07:52:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.735+0000 7f58d0a00700 1 -- 172.21.15.89:0/663144211 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc00bd70 con 0x7f58d40d20d0 2024-04-02T07:52:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.736+0000 7f58d9000700 1 -- 172.21.15.89:0/663144211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d40d20d0 msgr2=0x7f58d40cfc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.736+0000 7f58d9000700 1 --2- 172.21.15.89:0/663144211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d40d20d0 0x7f58d40cfc00 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f58d408cca0 tx=0x7f58cc00b330 comp rx=0 tx=0).stop 2024-04-02T07:52:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.736+0000 7f58d9000700 1 -- 172.21.15.89:0/663144211 shutdown_connections 2024-04-02T07:52:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.736+0000 7f58d9000700 1 --2- 172.21.15.89:0/663144211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d40d20d0 0x7f58d40cfc00 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.736+0000 7f58d9000700 1 --2- 172.21.15.89:0/663144211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58d415d930 0x7f58d40d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.736+0000 7f58d9000700 1 --2- 172.21.15.89:0/663144211 >> v2:172.21.15.113:3300/0 conn(0x7f58d4162030 0x7f58d40d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.736+0000 7f58d9000700 1 -- 172.21.15.89:0/663144211 >> 172.21.15.89:0/663144211 conn(0x7f58d408aff0 msgr2=0x7f58d40d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.736+0000 7f58d9000700 1 -- 172.21.15.89:0/663144211 shutdown_connections 2024-04-02T07:52:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.736+0000 7f58d9000700 1 -- 172.21.15.89:0/663144211 wait complete. 2024-04-02T07:52:06.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.738+0000 7f58d9000700 1 Processor -- start 2024-04-02T07:52:06.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.738+0000 7f58d9000700 1 -- start start 2024-04-02T07:52:06.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d9000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d40d20d0 0x7f58d4159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:06.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d9000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58d415d930 0x7f58d40d0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:06.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d9000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58d4162030 0x7f58d40b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:06.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d9000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d4159f80 con 0x7f58d40d20d0 2024-04-02T07:52:06.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d9000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d40ca680 con 0x7f58d4162030 2024-04-02T07:52:06.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d9000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d415a4d0 con 0x7f58d415d930 2024-04-02T07:52:06.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d40d20d0 0x7f58d4159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:06.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d40d20d0 0x7f58d4159a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53216/0 (socket says 172.21.15.89:53216) 2024-04-02T07:52:06.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d1e00700 1 -- 172.21.15.89:0/840862060 learned_addr learned my addr 172.21.15.89:0/840862060 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:06.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d1400700 1 --2- 172.21.15.89:0/840862060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58d415d930 0x7f58d40d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:06.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d2800700 1 --2- 172.21.15.89:0/840862060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58d4162030 0x7f58d40b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:06.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d1e00700 1 -- 172.21.15.89:0/840862060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58d415d930 msgr2=0x7f58d40d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:06.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d1e00700 1 --2- 172.21.15.89:0/840862060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58d415d930 0x7f58d40d0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d1e00700 1 -- 172.21.15.89:0/840862060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58d4162030 msgr2=0x7f58d40b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:06.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d1e00700 1 --2- 172.21.15.89:0/840862060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58d4162030 0x7f58d40b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:06.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.739+0000 7f58d1e00700 1 -- 172.21.15.89:0/840862060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58cc009b30 con 0x7f58d40d20d0 2024-04-02T07:52:06.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.740+0000 7f58d1e00700 1 --2- 172.21.15.89:0/840862060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d40d20d0 0x7f58d4159a40 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f58c400b530 tx=0x7f58c400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:06.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.740+0000 7f58ca000700 1 -- 172.21.15.89:0/840862060 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c4011830 con 0x7f58d40d20d0 2024-04-02T07:52:06.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.740+0000 7f58ca000700 1 -- 172.21.15.89:0/840862060 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c40119b0 con 0x7f58d40d20d0 2024-04-02T07:52:06.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.740+0000 7f58ca000700 1 -- 172.21.15.89:0/840862060 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c4011d60 con 0x7f58d40d20d0 2024-04-02T07:52:06.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.740+0000 7f58d9000700 1 -- 172.21.15.89:0/840862060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d40ca920 con 0x7f58d40d20d0 2024-04-02T07:52:06.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.740+0000 7f58d9000700 1 -- 172.21.15.89:0/840862060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d40caee0 con 0x7f58d40d20d0 2024-04-02T07:52:06.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.741+0000 7f58d9000700 1 -- 172.21.15.89:0/840862060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d4067720 con 0x7f58d40d20d0 2024-04-02T07:52:06.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.742+0000 7f58ca000700 1 -- 172.21.15.89:0/840862060 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f58c401a400 con 0x7f58d40d20d0 2024-04-02T07:52:06.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.743+0000 7f58ca000700 1 --2- 172.21.15.89:0/840862060 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f58b406adb0 0x7f58b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:06.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.743+0000 7f58ca000700 1 -- 172.21.15.89:0/840862060 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f58c4089470 con 0x7f58d40d20d0 2024-04-02T07:52:06.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.743+0000 7f58d1400700 1 --2- 172.21.15.89:0/840862060 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f58b406adb0 0x7f58b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:06.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.744+0000 7f58d1400700 1 --2- 172.21.15.89:0/840862060 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f58b406adb0 0x7f58b406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f58c0006fd0 tx=0x7f58c0008040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:06.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.748+0000 7f58ca000700 1 -- 172.21.15.89:0/840862060 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f58c40546a0 con 0x7f58d40d20d0 2024-04-02T07:52:06.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:06.949+0000 7f58d9000700 1 -- 172.21.15.89:0/840862060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f58d40b9530 con 0x7f58d40d20d0 2024-04-02T07:52:07.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.570+0000 7f58ca000700 1 -- 172.21.15.89:0/840862060 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v94) v1 ==== 138+0+0 (secure 0 0 0) 0x7f58c40150b0 con 0x7f58d40d20d0 2024-04-02T07:52:07.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.575+0000 7f58d9000700 1 -- 172.21.15.89:0/840862060 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f58b406adb0 msgr2=0x7f58b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:07.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.575+0000 7f58d9000700 1 --2- 172.21.15.89:0/840862060 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f58b406adb0 0x7f58b406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f58c0006fd0 tx=0x7f58c0008040 comp rx=0 tx=0).stop 2024-04-02T07:52:07.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.575+0000 7f58d9000700 1 -- 172.21.15.89:0/840862060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d40d20d0 msgr2=0x7f58d4159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:07.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.575+0000 7f58d9000700 1 --2- 172.21.15.89:0/840862060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d40d20d0 0x7f58d4159a40 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f58c400b530 tx=0x7f58c400b8f0 comp rx=0 tx=0).stop 2024-04-02T07:52:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.575+0000 7f58d9000700 1 -- 172.21.15.89:0/840862060 shutdown_connections 2024-04-02T07:52:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.575+0000 7f58d9000700 1 --2- 172.21.15.89:0/840862060 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f58b406adb0 0x7f58b406d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.575+0000 7f58d9000700 1 --2- 172.21.15.89:0/840862060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58d4162030 0x7f58d40b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.575+0000 7f58d9000700 1 --2- 172.21.15.89:0/840862060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58d40d20d0 0x7f58d4159a40 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.575+0000 7f58d9000700 1 --2- 172.21.15.89:0/840862060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58d415d930 0x7f58d40d0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.575+0000 7f58d9000700 1 -- 172.21.15.89:0/840862060 >> 172.21.15.89:0/840862060 conn(0x7f58d408aff0 msgr2=0x7f58d40b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.575+0000 7f58d9000700 1 -- 172.21.15.89:0/840862060 shutdown_connections 2024-04-02T07:52:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.575+0000 7f58d9000700 1 -- 172.21.15.89:0/840862060 wait complete. 2024-04-02T07:52:07.579 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T07:52:07.598 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T07:52:07.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.797+0000 7f4aa2000700 1 Processor -- start 2024-04-02T07:52:07.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.798+0000 7f4aa2000700 1 -- start start 2024-04-02T07:52:07.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.799+0000 7f4aa2000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4a9c0d16a0 0x7f4a9c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:07.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.799+0000 7f4aa2000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9c161420 con 0x7f4a9c0d2370 2024-04-02T07:52:07.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.799+0000 7f4aa2000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9c161700 con 0x7f4a9c15d9d0 2024-04-02T07:52:07.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.799+0000 7f4aa2000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a9c0d4340 con 0x7f4a9c0d16a0 2024-04-02T07:52:07.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.799+0000 7f4a9be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4a9c0d2370 0x7f4a9c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49006/0 (socket says 172.21.15.89:49006) 2024-04-02T07:52:07.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.799+0000 7f4a9be00700 1 -- 172.21.15.89:0/2470650257 learned_addr learned my addr 172.21.15.89:0/2470650257 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:07.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.799+0000 7f4a9aa00700 1 --2- 172.21.15.89:0/2470650257 >> v2:172.21.15.113:3300/0 conn(0x7f4a9c0d16a0 0x7f4a9c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:07.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.800+0000 7f4a9a000700 1 -- 172.21.15.89:0/2470650257 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828681576 0 0) 0x7f4a9c161420 con 0x7f4a9c0d2370 2024-04-02T07:52:07.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.800+0000 7f4a9a000700 1 -- 172.21.15.89:0/2470650257 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a800035b0 con 0x7f4a9c0d2370 2024-04-02T07:52:07.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.800+0000 7f4a9a000700 1 -- 172.21.15.89:0/2470650257 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904299485 0 0) 0x7f4a9c161700 con 0x7f4a9c15d9d0 2024-04-02T07:52:07.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.800+0000 7f4a9a000700 1 -- 172.21.15.89:0/2470650257 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a9c161420 con 0x7f4a9c15d9d0 2024-04-02T07:52:07.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.800+0000 7f4a9aa00700 1 -- 172.21.15.89:0/2470650257 >> v1:172.21.15.113:6789/0 conn(0x7f4a9c15d9d0 legacy=0x7f4a9c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:07.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.800+0000 7f4a9aa00700 1 -- 172.21.15.89:0/2470650257 >> v1:172.21.15.89:6789/0 conn(0x7f4a9c0d2370 legacy=0x7f4a9c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:07.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.800+0000 7f4a9aa00700 1 -- 172.21.15.89:0/2470650257 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c1619e0 con 0x7f4a9c0d16a0 2024-04-02T07:52:07.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.800+0000 7f4a9aa00700 1 --2- 172.21.15.89:0/2470650257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a9c0d16a0 0x7f4a9c0d1e30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4a88009820 tx=0x7f4a88009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=664333d4f2f4abb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:07.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.800+0000 7f4a9a000700 1 -- 172.21.15.89:0/2470650257 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88005e30 con 0x7f4a9c0d16a0 2024-04-02T07:52:07.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.800+0000 7f4a9a000700 1 -- 172.21.15.89:0/2470650257 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a88006030 con 0x7f4a9c0d16a0 2024-04-02T07:52:07.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.800+0000 7f4a9a000700 1 -- 172.21.15.89:0/2470650257 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a880063a0 con 0x7f4a9c0d16a0 2024-04-02T07:52:07.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.801+0000 7f4aa2000700 1 -- 172.21.15.89:0/2470650257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a9c0d16a0 msgr2=0x7f4a9c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:07.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.801+0000 7f4aa2000700 1 --2- 172.21.15.89:0/2470650257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a9c0d16a0 0x7f4a9c0d1e30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4a88009820 tx=0x7f4a88009b30 comp rx=0 tx=0).stop 2024-04-02T07:52:07.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.801+0000 7f4aa2000700 1 -- 172.21.15.89:0/2470650257 shutdown_connections 2024-04-02T07:52:07.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.801+0000 7f4aa2000700 1 --2- 172.21.15.89:0/2470650257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a9c0d16a0 0x7f4a9c0d1e30 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:07.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.801+0000 7f4aa2000700 1 -- 172.21.15.89:0/2470650257 >> 172.21.15.89:0/2470650257 conn(0x7f4a9c08aff0 msgr2=0x7f4a9c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:07.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.801+0000 7f4aa2000700 1 -- 172.21.15.89:0/2470650257 shutdown_connections 2024-04-02T07:52:07.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.801+0000 7f4aa2000700 1 -- 172.21.15.89:0/2470650257 wait complete. 2024-04-02T07:52:07.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.803+0000 7f4aa2000700 1 Processor -- start 2024-04-02T07:52:07.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.803+0000 7f4aa2000700 1 -- start start 2024-04-02T07:52:07.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4aa2000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a9c0d16a0 0x7f4a9c14b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:07.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4aa2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a9c0d2370 0x7f4a9c15f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:07.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4aa2000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a9c15d9d0 0x7f4a9c1e9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:07.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4aa2000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c15eca0 con 0x7f4a9c0d2370 2024-04-02T07:52:07.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4aa2000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c15ddb0 con 0x7f4a9c15d9d0 2024-04-02T07:52:07.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4aa2000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c15e090 con 0x7f4a9c0d16a0 2024-04-02T07:52:07.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4a9aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a9c0d2370 0x7f4a9c15f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:07.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4a9aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a9c0d2370 0x7f4a9c15f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53226/0 (socket says 172.21.15.89:53226) 2024-04-02T07:52:07.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4a9aa00700 1 -- 172.21.15.89:0/2539376634 learned_addr learned my addr 172.21.15.89:0/2539376634 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:07.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4a9be00700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a9c15d9d0 0x7f4a9c1e9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:07.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4a9b400700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a9c0d16a0 0x7f4a9c14b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:07.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4a9aa00700 1 -- 172.21.15.89:0/2539376634 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a9c0d16a0 msgr2=0x7f4a9c14b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:07.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4a9aa00700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a9c0d16a0 0x7f4a9c14b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:07.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4a9aa00700 1 -- 172.21.15.89:0/2539376634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a9c15d9d0 msgr2=0x7f4a9c1e9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:07.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.804+0000 7f4a9aa00700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a9c15d9d0 0x7f4a9c1e9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:07.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.805+0000 7f4a9aa00700 1 -- 172.21.15.89:0/2539376634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a88009430 con 0x7f4a9c0d2370 2024-04-02T07:52:07.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.805+0000 7f4a9b400700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a9c0d16a0 0x7f4a9c14b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:07.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.805+0000 7f4a9aa00700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a9c0d2370 0x7f4a9c15f130 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f4a88013040 tx=0x7f4a88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:07.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.805+0000 7f4a93400700 1 -- 172.21.15.89:0/2539376634 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88004240 con 0x7f4a9c0d2370 2024-04-02T07:52:07.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.805+0000 7f4a93400700 1 -- 172.21.15.89:0/2539376634 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a880043a0 con 0x7f4a9c0d2370 2024-04-02T07:52:07.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.805+0000 7f4aa2000700 1 -- 172.21.15.89:0/2539376634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9c15e370 con 0x7f4a9c0d2370 2024-04-02T07:52:07.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.805+0000 7f4a93400700 1 -- 172.21.15.89:0/2539376634 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88006920 con 0x7f4a9c0d2370 2024-04-02T07:52:07.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.805+0000 7f4aa2000700 1 -- 172.21.15.89:0/2539376634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9c15e930 con 0x7f4a9c0d2370 2024-04-02T07:52:07.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.806+0000 7f4aa2000700 1 -- 172.21.15.89:0/2539376634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9c0d1ca0 con 0x7f4a9c0d2370 2024-04-02T07:52:07.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.808+0000 7f4a93400700 1 -- 172.21.15.89:0/2539376634 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4a88022070 con 0x7f4a9c0d2370 2024-04-02T07:52:07.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.808+0000 7f4a93400700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a7c06adb0 0x7f4a7c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:07.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.808+0000 7f4a93400700 1 -- 172.21.15.89:0/2539376634 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f4a8808b190 con 0x7f4a9c0d2370 2024-04-02T07:52:07.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.809+0000 7f4a9b400700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a7c06adb0 0x7f4a7c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:07.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.809+0000 7f4a9b400700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a7c06adb0 0x7f4a7c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0011e0 tx=0x7f4a8c009040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:07.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:07.812+0000 7f4a93400700 1 -- 172.21.15.89:0/2539376634 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4a880562d0 con 0x7f4a9c0d2370 2024-04-02T07:52:08.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.013+0000 7f4aa2000700 1 -- 172.21.15.89:0/2539376634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4a9c15eac0 con 0x7f4a9c0d2370 2024-04-02T07:52:08.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.571+0000 7f4a93400700 1 -- 172.21.15.89:0/2539376634 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v95) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4a880598f0 con 0x7f4a9c0d2370 2024-04-02T07:52:08.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.576+0000 7f4aa2000700 1 -- 172.21.15.89:0/2539376634 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a7c06adb0 msgr2=0x7f4a7c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:08.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.576+0000 7f4aa2000700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a7c06adb0 0x7f4a7c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0011e0 tx=0x7f4a8c009040 comp rx=0 tx=0).stop 2024-04-02T07:52:08.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.576+0000 7f4aa2000700 1 -- 172.21.15.89:0/2539376634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a9c0d2370 msgr2=0x7f4a9c15f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:08.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.576+0000 7f4aa2000700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a9c0d2370 0x7f4a9c15f130 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f4a88013040 tx=0x7f4a88004750 comp rx=0 tx=0).stop 2024-04-02T07:52:08.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.577+0000 7f4aa2000700 1 -- 172.21.15.89:0/2539376634 shutdown_connections 2024-04-02T07:52:08.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.577+0000 7f4aa2000700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a7c06adb0 0x7f4a7c06d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:08.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.577+0000 7f4aa2000700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a9c15d9d0 0x7f4a9c1e9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:08.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.577+0000 7f4aa2000700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a9c0d16a0 0x7f4a9c14b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:08.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.577+0000 7f4aa2000700 1 --2- 172.21.15.89:0/2539376634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a9c0d2370 0x7f4a9c15f130 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:08.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.577+0000 7f4aa2000700 1 -- 172.21.15.89:0/2539376634 >> 172.21.15.89:0/2539376634 conn(0x7f4a9c08aff0 msgr2=0x7f4a9c0b5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:08.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.578+0000 7f4aa2000700 1 -- 172.21.15.89:0/2539376634 shutdown_connections 2024-04-02T07:52:08.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.578+0000 7f4aa2000700 1 -- 172.21.15.89:0/2539376634 wait complete. 2024-04-02T07:52:08.587 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T07:52:08.611 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T07:52:08.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.804+0000 7f9eee400700 1 Processor -- start 2024-04-02T07:52:08.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.805+0000 7f9eee400700 1 -- start start 2024-04-02T07:52:08.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.806+0000 7f9eee400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ee815d9d0 0x7f9ee80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:08.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.806+0000 7f9eee400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ee80d16a0 0x7f9ee80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:08.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.806+0000 7f9eee400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee8161420 con 0x7f9ee80d2370 2024-04-02T07:52:08.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.806+0000 7f9eee400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee80d2820 con 0x7f9ee80d16a0 2024-04-02T07:52:08.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.806+0000 7f9eee400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee8161700 con 0x7f9ee815d9d0 2024-04-02T07:52:08.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.806+0000 7f9ee6a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ee80d16a0 0x7f9ee80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:08.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.806+0000 7f9ee6a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ee80d16a0 0x7f9ee80d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41898/0 (socket says 172.21.15.89:41898) 2024-04-02T07:52:08.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.806+0000 7f9ee6a00700 1 -- 172.21.15.89:0/2224551950 learned_addr learned my addr 172.21.15.89:0/2224551950 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:08.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.806+0000 7f9ee7400700 1 --2- 172.21.15.89:0/2224551950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ee815d9d0 0x7f9ee80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:08.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.806+0000 7f9ee6a00700 1 -- 172.21.15.89:0/2224551950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ee815d9d0 msgr2=0x7f9ee80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:08.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.806+0000 7f9ee6a00700 1 --2- 172.21.15.89:0/2224551950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ee815d9d0 0x7f9ee80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:08.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.806+0000 7f9ee6a00700 1 -- 172.21.15.89:0/2224551950 >> v1:172.21.15.89:6789/0 conn(0x7f9ee80d2370 legacy=0x7f9ee815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:08.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.806+0000 7f9ee6a00700 1 -- 172.21.15.89:0/2224551950 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee81619e0 con 0x7f9ee80d16a0 2024-04-02T07:52:08.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.807+0000 7f9ee6a00700 1 --2- 172.21.15.89:0/2224551950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee80d16a0 0x7f9ee80d1e30 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4006fd0 tx=0x7f9ed4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e0c8d9d7c20899 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:08.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.807+0000 7f9ee6000700 1 -- 172.21.15.89:0/2224551950 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4005e30 con 0x7f9ee80d16a0 2024-04-02T07:52:08.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.807+0000 7f9ee6000700 1 -- 172.21.15.89:0/2224551950 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed4004030 con 0x7f9ee80d16a0 2024-04-02T07:52:08.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.807+0000 7f9ee6000700 1 -- 172.21.15.89:0/2224551950 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed40043e0 con 0x7f9ee80d16a0 2024-04-02T07:52:08.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.808+0000 7f9eee400700 1 -- 172.21.15.89:0/2224551950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee80d16a0 msgr2=0x7f9ee80d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:08.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.808+0000 7f9eee400700 1 --2- 172.21.15.89:0/2224551950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee80d16a0 0x7f9ee80d1e30 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4006fd0 tx=0x7f9ed4009bd0 comp rx=0 tx=0).stop 2024-04-02T07:52:08.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.808+0000 7f9eee400700 1 -- 172.21.15.89:0/2224551950 shutdown_connections 2024-04-02T07:52:08.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.808+0000 7f9eee400700 1 --2- 172.21.15.89:0/2224551950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ee815d9d0 0x7f9ee80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:08.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.808+0000 7f9eee400700 1 --2- 172.21.15.89:0/2224551950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee80d16a0 0x7f9ee80d1e30 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:08.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.808+0000 7f9eee400700 1 -- 172.21.15.89:0/2224551950 >> 172.21.15.89:0/2224551950 conn(0x7f9ee808aff0 msgr2=0x7f9ee80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:08.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.808+0000 7f9eee400700 1 -- 172.21.15.89:0/2224551950 shutdown_connections 2024-04-02T07:52:08.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.808+0000 7f9eee400700 1 -- 172.21.15.89:0/2224551950 wait complete. 2024-04-02T07:52:08.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.809+0000 7f9eee400700 1 Processor -- start 2024-04-02T07:52:08.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.810+0000 7f9eee400700 1 -- start start 2024-04-02T07:52:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.810+0000 7f9eee400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee80d16a0 0x7f9ee8151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9eee400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ee80d2370 0x7f9ee80b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9eee400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ee815d9d0 0x7f9ee80b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9eee400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee8153830 con 0x7f9ee80d16a0 2024-04-02T07:52:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9eee400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee8153b10 con 0x7f9ee815d9d0 2024-04-02T07:52:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9eee400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee8153df0 con 0x7f9ee80d2370 2024-04-02T07:52:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9ee7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee80d16a0 0x7f9ee8151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9ee7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee80d16a0 0x7f9ee8151dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41914/0 (socket says 172.21.15.89:41914) 2024-04-02T07:52:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9ee7400700 1 -- 172.21.15.89:0/1353134157 learned_addr learned my addr 172.21.15.89:0/1353134157 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9ee6a00700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ee80d2370 0x7f9ee80b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9ee7e00700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ee815d9d0 0x7f9ee80b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9ee7400700 1 -- 172.21.15.89:0/1353134157 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ee80d2370 msgr2=0x7f9ee80b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9ee7400700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ee80d2370 0x7f9ee80b7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9ee7400700 1 -- 172.21.15.89:0/1353134157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ee815d9d0 msgr2=0x7f9ee80b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9ee7400700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ee815d9d0 0x7f9ee80b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9ee7400700 1 -- 172.21.15.89:0/1353134157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed4009500 con 0x7f9ee80d16a0 2024-04-02T07:52:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9ee7e00700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ee815d9d0 0x7f9ee80b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.811+0000 7f9ee7400700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee80d16a0 0x7f9ee8151dc0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800d800 tx=0x7f9ed800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.812+0000 7f9edf400700 1 -- 172.21.15.89:0/1353134157 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed80125f0 con 0x7f9ee80d16a0 2024-04-02T07:52:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.812+0000 7f9edf400700 1 -- 172.21.15.89:0/1353134157 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed8012750 con 0x7f9ee80d16a0 2024-04-02T07:52:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.812+0000 7f9edf400700 1 -- 172.21.15.89:0/1353134157 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8012b00 con 0x7f9ee80d16a0 2024-04-02T07:52:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.812+0000 7f9eee400700 1 -- 172.21.15.89:0/1353134157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee81540d0 con 0x7f9ee80d16a0 2024-04-02T07:52:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.812+0000 7f9eee400700 1 -- 172.21.15.89:0/1353134157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee80b84a0 con 0x7f9ee80d16a0 2024-04-02T07:52:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.813+0000 7f9eee400700 1 -- 172.21.15.89:0/1353134157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee80d1ca0 con 0x7f9ee80d16a0 2024-04-02T07:52:08.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.814+0000 7f9edf400700 1 -- 172.21.15.89:0/1353134157 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9ed8012c60 con 0x7f9ee80d16a0 2024-04-02T07:52:08.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.815+0000 7f9edf400700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9ec806adb0 0x7f9ec806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:08.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.815+0000 7f9edf400700 1 -- 172.21.15.89:0/1353134157 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f9ed80898a0 con 0x7f9ee80d16a0 2024-04-02T07:52:08.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.815+0000 7f9ee6a00700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9ec806adb0 0x7f9ec806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:08.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.816+0000 7f9ee6a00700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9ec806adb0 0x7f9ec806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80d1ae0 tx=0x7f9ed4008070 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:08.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:08.820+0000 7f9edf400700 1 -- 172.21.15.89:0/1353134157 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9ed8054860 con 0x7f9ee80d16a0 2024-04-02T07:52:09.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.065+0000 7f9eee400700 1 -- 172.21.15.89:0/1353134157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9ee8067720 con 0x7f9ee80d16a0 2024-04-02T07:52:09.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.072+0000 7f9edf400700 1 -- 172.21.15.89:0/1353134157 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v37) v1 ==== 174+0+0 (secure 0 0 0) 0x7f9ed8057e80 con 0x7f9ee80d16a0 2024-04-02T07:52:09.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.077+0000 7f9eee400700 1 -- 172.21.15.89:0/1353134157 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9ec806adb0 msgr2=0x7f9ec806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:09.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.077+0000 7f9eee400700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9ec806adb0 0x7f9ec806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80d1ae0 tx=0x7f9ed4008070 comp rx=0 tx=0).stop 2024-04-02T07:52:09.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.077+0000 7f9eee400700 1 -- 172.21.15.89:0/1353134157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee80d16a0 msgr2=0x7f9ee8151dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.077+0000 7f9eee400700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee80d16a0 0x7f9ee8151dc0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800d800 tx=0x7f9ed800db10 comp rx=0 tx=0).stop 2024-04-02T07:52:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.077+0000 7f9eee400700 1 -- 172.21.15.89:0/1353134157 shutdown_connections 2024-04-02T07:52:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.077+0000 7f9eee400700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9ec806adb0 0x7f9ec806d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.077+0000 7f9eee400700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ee815d9d0 0x7f9ee80b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.077+0000 7f9eee400700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee80d16a0 0x7f9ee8151dc0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.077+0000 7f9eee400700 1 --2- 172.21.15.89:0/1353134157 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ee80d2370 0x7f9ee80b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.077+0000 7f9eee400700 1 -- 172.21.15.89:0/1353134157 >> 172.21.15.89:0/1353134157 conn(0x7f9ee808aff0 msgr2=0x7f9ee80b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.078+0000 7f9eee400700 1 -- 172.21.15.89:0/1353134157 shutdown_connections 2024-04-02T07:52:09.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.078+0000 7f9eee400700 1 -- 172.21.15.89:0/1353134157 wait complete. 2024-04-02T07:52:09.085 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 10 and data pool 11 2024-04-02T07:52:09.115 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:52:09.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.304+0000 7f5f57c00700 1 Processor -- start 2024-04-02T07:52:09.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.304+0000 7f5f57c00700 1 -- start start 2024-04-02T07:52:09.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.305+0000 7f5f57c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5f500ce590 0x7f5f500ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:09.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.305+0000 7f5f57c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5f500b6a20 0x7f5f500b6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.305+0000 7f5f57c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f500ceef0 con 0x7f5f50160be0 2024-04-02T07:52:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.305+0000 7f5f57c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f500b65d0 con 0x7f5f500b6a20 2024-04-02T07:52:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.305+0000 7f5f57c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f500d2a80 con 0x7f5f500ce590 2024-04-02T07:52:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.305+0000 7f5f55c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5f500b6a20 0x7f5f500b6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.305+0000 7f5f55200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5f50160be0 0x7f5f500cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60344/0 (socket says 172.21.15.89:60344) 2024-04-02T07:52:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.305+0000 7f5f55c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5f500b6a20 0x7f5f500b6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41920/0 (socket says 172.21.15.89:41920) 2024-04-02T07:52:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.305+0000 7f5f55200700 1 -- 172.21.15.89:0/3978230656 learned_addr learned my addr 172.21.15.89:0/3978230656 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.305+0000 7f5f4fe00700 1 --2- 172.21.15.89:0/3978230656 >> v2:172.21.15.113:3300/0 conn(0x7f5f500ce590 0x7f5f500ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.306+0000 7f5f55c00700 1 -- 172.21.15.89:0/3978230656 >> v2:172.21.15.113:3300/0 conn(0x7f5f500ce590 msgr2=0x7f5f500ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.306+0000 7f5f55c00700 1 --2- 172.21.15.89:0/3978230656 >> v2:172.21.15.113:3300/0 conn(0x7f5f500ce590 0x7f5f500ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.306+0000 7f5f55c00700 1 -- 172.21.15.89:0/3978230656 >> v1:172.21.15.89:6789/0 conn(0x7f5f50160be0 legacy=0x7f5f500cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.306+0000 7f5f55c00700 1 -- 172.21.15.89:0/3978230656 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f50160fc0 con 0x7f5f500b6a20 2024-04-02T07:52:09.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.306+0000 7f5f4f400700 1 -- 172.21.15.89:0/3978230656 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594249243 0 0) 0x7f5f500ceef0 con 0x7f5f50160be0 2024-04-02T07:52:09.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.306+0000 7f5f55c00700 1 --2- 172.21.15.89:0/3978230656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f500b6a20 0x7f5f500b6e90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f5f500b9600 tx=0x7f5f4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32572bb9a893470e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:09.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.306+0000 7f5f4f400700 1 -- 172.21.15.89:0/3978230656 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4800e070 con 0x7f5f500b6a20 2024-04-02T07:52:09.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.306+0000 7f5f4f400700 1 -- 172.21.15.89:0/3978230656 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4800bb20 con 0x7f5f500b6a20 2024-04-02T07:52:09.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.306+0000 7f5f4f400700 1 -- 172.21.15.89:0/3978230656 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f48012430 con 0x7f5f500b6a20 2024-04-02T07:52:09.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.307+0000 7f5f57c00700 1 -- 172.21.15.89:0/3978230656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f500b6a20 msgr2=0x7f5f500b6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:09.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.307+0000 7f5f57c00700 1 --2- 172.21.15.89:0/3978230656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f500b6a20 0x7f5f500b6e90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f5f500b9600 tx=0x7f5f4800b330 comp rx=0 tx=0).stop 2024-04-02T07:52:09.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.307+0000 7f5f57c00700 1 -- 172.21.15.89:0/3978230656 shutdown_connections 2024-04-02T07:52:09.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.307+0000 7f5f57c00700 1 --2- 172.21.15.89:0/3978230656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f500b6a20 0x7f5f500b6e90 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.307+0000 7f5f57c00700 1 --2- 172.21.15.89:0/3978230656 >> v2:172.21.15.113:3300/0 conn(0x7f5f500ce590 0x7f5f500ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.307+0000 7f5f57c00700 1 -- 172.21.15.89:0/3978230656 >> 172.21.15.89:0/3978230656 conn(0x7f5f5008aff0 msgr2=0x7f5f500d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:09.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.307+0000 7f5f57c00700 1 -- 172.21.15.89:0/3978230656 shutdown_connections 2024-04-02T07:52:09.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.307+0000 7f5f57c00700 1 -- 172.21.15.89:0/3978230656 wait complete. 2024-04-02T07:52:09.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.309+0000 7f5f57c00700 1 Processor -- start 2024-04-02T07:52:09.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.310+0000 7f5f57c00700 1 -- start start 2024-04-02T07:52:09.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f57c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f500b6a20 0x7f5f50154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:09.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f57c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5f500ce590 0x7f5f5006caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:09.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f57c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f50160be0 0x7f5f5006d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:09.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f57c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f501f7610 con 0x7f5f500b6a20 2024-04-02T07:52:09.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f57c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f501f6990 con 0x7f5f50160be0 2024-04-02T07:52:09.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f57c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f501f6c70 con 0x7f5f500ce590 2024-04-02T07:52:09.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f55200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f500b6a20 0x7f5f50154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:09.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f55200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f500b6a20 0x7f5f50154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41934/0 (socket says 172.21.15.89:41934) 2024-04-02T07:52:09.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f55200700 1 -- 172.21.15.89:0/3372874566 learned_addr learned my addr 172.21.15.89:0/3372874566 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:09.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f4fe00700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5f500ce590 0x7f5f5006caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:09.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f55c00700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f50160be0 0x7f5f5006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:09.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f55200700 1 -- 172.21.15.89:0/3372874566 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5f500ce590 msgr2=0x7f5f5006caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:09.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f55200700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5f500ce590 0x7f5f5006caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f55200700 1 -- 172.21.15.89:0/3372874566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f50160be0 msgr2=0x7f5f5006d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:09.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f55200700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f50160be0 0x7f5f5006d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f55200700 1 -- 172.21.15.89:0/3372874566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f48009a70 con 0x7f5f500b6a20 2024-04-02T07:52:09.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.311+0000 7f5f55c00700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f50160be0 0x7f5f5006d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:09.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.312+0000 7f5f55200700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f500b6a20 0x7f5f50154a70 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400c550 tx=0x7f5f4400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:09.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.312+0000 7f5f4cc00700 1 -- 172.21.15.89:0/3372874566 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44004c80 con 0x7f5f500b6a20 2024-04-02T07:52:09.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.312+0000 7f5f4cc00700 1 -- 172.21.15.89:0/3372874566 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f44004e00 con 0x7f5f500b6a20 2024-04-02T07:52:09.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.312+0000 7f5f57c00700 1 -- 172.21.15.89:0/3372874566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f501f6f50 con 0x7f5f500b6a20 2024-04-02T07:52:09.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.312+0000 7f5f4cc00700 1 -- 172.21.15.89:0/3372874566 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44007310 con 0x7f5f500b6a20 2024-04-02T07:52:09.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.312+0000 7f5f57c00700 1 -- 172.21.15.89:0/3372874566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f50154fb0 con 0x7f5f500b6a20 2024-04-02T07:52:09.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.313+0000 7f5f57c00700 1 -- 172.21.15.89:0/3372874566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f50067720 con 0x7f5f500b6a20 2024-04-02T07:52:09.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.314+0000 7f5f4cc00700 1 -- 172.21.15.89:0/3372874566 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5f44007470 con 0x7f5f500b6a20 2024-04-02T07:52:09.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.315+0000 7f5f4cc00700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5f3006adb0 0x7f5f3006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:09.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.315+0000 7f5f4cc00700 1 -- 172.21.15.89:0/3372874566 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5f440892c0 con 0x7f5f500b6a20 2024-04-02T07:52:09.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.315+0000 7f5f4fe00700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5f3006adb0 0x7f5f3006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:09.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.316+0000 7f5f4fe00700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5f3006adb0 0x7f5f3006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c009b70 tx=0x7f5f3c008040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:09.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.320+0000 7f5f4cc00700 1 -- 172.21.15.89:0/3372874566 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5f440542c0 con 0x7f5f500b6a20 2024-04-02T07:52:09.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.521+0000 7f5f57c00700 1 -- 172.21.15.89:0/3372874566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f500b5cb0 con 0x7f5f500b6a20 2024-04-02T07:52:09.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.522+0000 7f5f4cc00700 1 -- 172.21.15.89:0/3372874566 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16580 (secure 0 0 0) 0x7f5f440578e0 con 0x7f5f500b6a20 2024-04-02T07:52:09.524 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:09.524 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":96,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:52:09.066109+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:52:06.991300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-02T07:52:08.054066+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:52:09.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.524+0000 7f5f57c00700 1 -- 172.21.15.89:0/3372874566 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5f3006adb0 msgr2=0x7f5f3006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:09.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.524+0000 7f5f57c00700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5f3006adb0 0x7f5f3006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c009b70 tx=0x7f5f3c008040 comp rx=0 tx=0).stop 2024-04-02T07:52:09.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.524+0000 7f5f57c00700 1 -- 172.21.15.89:0/3372874566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f500b6a20 msgr2=0x7f5f50154a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:09.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.524+0000 7f5f57c00700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f500b6a20 0x7f5f50154a70 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400c550 tx=0x7f5f4400c910 comp rx=0 tx=0).stop 2024-04-02T07:52:09.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.524+0000 7f5f57c00700 1 -- 172.21.15.89:0/3372874566 shutdown_connections 2024-04-02T07:52:09.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.524+0000 7f5f57c00700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5f3006adb0 0x7f5f3006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.524+0000 7f5f57c00700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f50160be0 0x7f5f5006d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.524+0000 7f5f57c00700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f500b6a20 0x7f5f50154a70 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.524+0000 7f5f57c00700 1 --2- 172.21.15.89:0/3372874566 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5f500ce590 0x7f5f5006caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.524+0000 7f5f57c00700 1 -- 172.21.15.89:0/3372874566 >> 172.21.15.89:0/3372874566 conn(0x7f5f5008aff0 msgr2=0x7f5f500b73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:09.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.524+0000 7f5f57c00700 1 -- 172.21.15.89:0/3372874566 shutdown_connections 2024-04-02T07:52:09.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.524+0000 7f5f57c00700 1 -- 172.21.15.89:0/3372874566 wait complete. 2024-04-02T07:52:09.544 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:52:09.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.735+0000 7fe25f600700 1 Processor -- start 2024-04-02T07:52:09.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.736+0000 7fe25f600700 1 -- start start 2024-04-02T07:52:09.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.737+0000 7fe25f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2580ce590 0x7fe2580ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:09.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.737+0000 7fe25f600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe2580b6a20 0x7fe2580b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:09.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.737+0000 7fe25f600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2580ceef0 con 0x7fe258160be0 2024-04-02T07:52:09.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.737+0000 7fe25f600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2580b65d0 con 0x7fe2580b6a20 2024-04-02T07:52:09.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.737+0000 7fe25f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2580d2a80 con 0x7fe2580ce590 2024-04-02T07:52:09.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.737+0000 7fe25cc00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe258160be0 0x7fe2580cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60358/0 (socket says 172.21.15.89:60358) 2024-04-02T07:52:09.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.737+0000 7fe25cc00700 1 -- 172.21.15.89:0/1127853145 learned_addr learned my addr 172.21.15.89:0/1127853145 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:09.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.737+0000 7fe25d600700 1 --2- 172.21.15.89:0/1127853145 >> v2:172.21.15.113:3300/0 conn(0x7fe2580b6a20 0x7fe2580b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:09.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.737+0000 7fe257e00700 1 --2- 172.21.15.89:0/1127853145 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2580ce590 0x7fe2580ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:09.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.737+0000 7fe257400700 1 -- 172.21.15.89:0/1127853145 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339179469 0 0) 0x7fe2580ceef0 con 0x7fe258160be0 2024-04-02T07:52:09.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.737+0000 7fe257400700 1 -- 172.21.15.89:0/1127853145 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2400035b0 con 0x7fe258160be0 2024-04-02T07:52:09.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.738+0000 7fe257400700 1 -- 172.21.15.89:0/1127853145 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3474830862 0 0) 0x7fe2400035b0 con 0x7fe258160be0 2024-04-02T07:52:09.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.738+0000 7fe25d600700 1 -- 172.21.15.89:0/1127853145 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2580ce590 msgr2=0x7fe2580ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:09.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.738+0000 7fe25d600700 1 --2- 172.21.15.89:0/1127853145 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2580ce590 0x7fe2580ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.738+0000 7fe25d600700 1 -- 172.21.15.89:0/1127853145 >> v1:172.21.15.89:6789/0 conn(0x7fe258160be0 legacy=0x7fe2580cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:09.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.738+0000 7fe25d600700 1 -- 172.21.15.89:0/1127853145 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe258160fc0 con 0x7fe2580b6a20 2024-04-02T07:52:09.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.738+0000 7fe25d600700 1 --2- 172.21.15.89:0/1127853145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2580b6a20 0x7fe2580b6eb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe250009d90 tx=0x7fe25000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf545e8c9ac89f45 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:09.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.738+0000 7fe257400700 1 -- 172.21.15.89:0/1127853145 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25000e070 con 0x7fe2580b6a20 2024-04-02T07:52:09.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.738+0000 7fe257400700 1 -- 172.21.15.89:0/1127853145 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe250003a20 con 0x7fe2580b6a20 2024-04-02T07:52:09.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.738+0000 7fe257400700 1 -- 172.21.15.89:0/1127853145 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c001670 con 0x7fe2580b6a20 2024-04-02T07:52:09.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.739+0000 7fe25f600700 1 -- 172.21.15.89:0/1127853145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2580b6a20 msgr2=0x7fe2580b6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:09.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.739+0000 7fe25f600700 1 --2- 172.21.15.89:0/1127853145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2580b6a20 0x7fe2580b6eb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe250009d90 tx=0x7fe25000b330 comp rx=0 tx=0).stop 2024-04-02T07:52:09.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.739+0000 7fe25f600700 1 -- 172.21.15.89:0/1127853145 shutdown_connections 2024-04-02T07:52:09.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.739+0000 7fe25f600700 1 --2- 172.21.15.89:0/1127853145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2580b6a20 0x7fe2580b6eb0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.739+0000 7fe25f600700 1 --2- 172.21.15.89:0/1127853145 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2580ce590 0x7fe2580ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.739+0000 7fe25f600700 1 -- 172.21.15.89:0/1127853145 >> 172.21.15.89:0/1127853145 conn(0x7fe25808aff0 msgr2=0x7fe2580d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:09.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.739+0000 7fe25f600700 1 -- 172.21.15.89:0/1127853145 shutdown_connections 2024-04-02T07:52:09.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.739+0000 7fe25f600700 1 -- 172.21.15.89:0/1127853145 wait complete. 2024-04-02T07:52:09.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.741+0000 7fe25f600700 1 Processor -- start 2024-04-02T07:52:09.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.741+0000 7fe25f600700 1 -- start start 2024-04-02T07:52:09.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25f600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2580b6a20 0x7fe2580c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:09.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2580ce590 0x7fe25814b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:09.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25f600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe258160be0 0x7fe2581f2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:09.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25f600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2580cc630 con 0x7fe2580b6a20 2024-04-02T07:52:09.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25f600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2580caea0 con 0x7fe258160be0 2024-04-02T07:52:09.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2580cc300 con 0x7fe2580ce590 2024-04-02T07:52:09.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2580b6a20 0x7fe2580c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:09.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2580b6a20 0x7fe2580c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41950/0 (socket says 172.21.15.89:41950) 2024-04-02T07:52:09.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25cc00700 1 -- 172.21.15.89:0/988601211 learned_addr learned my addr 172.21.15.89:0/988601211 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:09.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe257e00700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2580ce590 0x7fe25814b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:09.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25d600700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe258160be0 0x7fe2581f2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:09.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25cc00700 1 -- 172.21.15.89:0/988601211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2580ce590 msgr2=0x7fe25814b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:09.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25cc00700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2580ce590 0x7fe25814b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25cc00700 1 -- 172.21.15.89:0/988601211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe258160be0 msgr2=0x7fe2581f2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:09.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25cc00700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe258160be0 0x7fe2581f2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25cc00700 1 -- 172.21.15.89:0/988601211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe250009a70 con 0x7fe2580b6a20 2024-04-02T07:52:09.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe25d600700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe258160be0 0x7fe2581f2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:09.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.742+0000 7fe257e00700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2580ce590 0x7fe25814b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:09.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.743+0000 7fe25cc00700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2580b6a20 0x7fe2580c74a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00c7b0 tx=0x7fe24c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:09.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.743+0000 7fe254c00700 1 -- 172.21.15.89:0/988601211 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c004c80 con 0x7fe2580b6a20 2024-04-02T07:52:09.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.743+0000 7fe254c00700 1 -- 172.21.15.89:0/988601211 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24c004de0 con 0x7fe2580b6a20 2024-04-02T07:52:09.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.743+0000 7fe254c00700 1 -- 172.21.15.89:0/988601211 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c0072d0 con 0x7fe2580b6a20 2024-04-02T07:52:09.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.743+0000 7fe25f600700 1 -- 172.21.15.89:0/988601211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2580ca620 con 0x7fe2580b6a20 2024-04-02T07:52:09.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.743+0000 7fe25f600700 1 -- 172.21.15.89:0/988601211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2580cabc0 con 0x7fe2580b6a20 2024-04-02T07:52:09.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.744+0000 7fe25f600700 1 -- 172.21.15.89:0/988601211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe258067720 con 0x7fe2580b6a20 2024-04-02T07:52:09.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.745+0000 7fe254c00700 1 -- 172.21.15.89:0/988601211 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe24c007430 con 0x7fe2580b6a20 2024-04-02T07:52:09.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.746+0000 7fe254c00700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe23806adb0 0x7fe23806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:09.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.746+0000 7fe254c00700 1 -- 172.21.15.89:0/988601211 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe24c0888b0 con 0x7fe2580b6a20 2024-04-02T07:52:09.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.746+0000 7fe257e00700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe23806adb0 0x7fe23806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:09.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.747+0000 7fe257e00700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe23806adb0 0x7fe23806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe2581f6570 tx=0x7fe244008040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:09.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.750+0000 7fe254c00700 1 -- 172.21.15.89:0/988601211 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe24c053810 con 0x7fe2580b6a20 2024-04-02T07:52:09.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.952+0000 7fe25f600700 1 -- 172.21.15.89:0/988601211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2580b5cb0 con 0x7fe2580b6a20 2024-04-02T07:52:09.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.953+0000 7fe254c00700 1 -- 172.21.15.89:0/988601211 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16580 (secure 0 0 0) 0x7fe24c018020 con 0x7fe2580b6a20 2024-04-02T07:52:09.955 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:09.955 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":96,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:52:09.066109+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:52:06.991300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-02T07:52:08.054066+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:52:09.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.955+0000 7fe25f600700 1 -- 172.21.15.89:0/988601211 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe23806adb0 msgr2=0x7fe23806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:09.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.955+0000 7fe25f600700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe23806adb0 0x7fe23806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe2581f6570 tx=0x7fe244008040 comp rx=0 tx=0).stop 2024-04-02T07:52:09.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.955+0000 7fe25f600700 1 -- 172.21.15.89:0/988601211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2580b6a20 msgr2=0x7fe2580c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:09.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.955+0000 7fe25f600700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2580b6a20 0x7fe2580c74a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00c7b0 tx=0x7fe24c00cb70 comp rx=0 tx=0).stop 2024-04-02T07:52:09.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.955+0000 7fe25f600700 1 -- 172.21.15.89:0/988601211 shutdown_connections 2024-04-02T07:52:09.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.955+0000 7fe25f600700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe23806adb0 0x7fe23806d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.955+0000 7fe25f600700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe258160be0 0x7fe2581f2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.955+0000 7fe25f600700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2580b6a20 0x7fe2580c74a0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.955+0000 7fe25f600700 1 --2- 172.21.15.89:0/988601211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2580ce590 0x7fe25814b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:09.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.955+0000 7fe25f600700 1 -- 172.21.15.89:0/988601211 >> 172.21.15.89:0/988601211 conn(0x7fe25808aff0 msgr2=0x7fe2580b8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:09.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.955+0000 7fe25f600700 1 -- 172.21.15.89:0/988601211 shutdown_connections 2024-04-02T07:52:09.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:09.955+0000 7fe25f600700 1 -- 172.21.15.89:0/988601211 wait complete. 2024-04-02T07:52:09.976 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T07:52:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.173+0000 7f23d9400700 1 Processor -- start 2024-04-02T07:52:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.173+0000 7f23d9400700 1 -- start start 2024-04-02T07:52:10.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.174+0000 7f23d9400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f23d415d710 0x7f23d40d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:10.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.175+0000 7f23d9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d40d2020 0x7f23d40d0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:10.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.175+0000 7f23d9400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d40cf640 con 0x7f23d40d1b70 2024-04-02T07:52:10.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.175+0000 7f23d9400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d40cf220 con 0x7f23d415d710 2024-04-02T07:52:10.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.175+0000 7f23d9400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d40ce5a0 con 0x7f23d40d2020 2024-04-02T07:52:10.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.175+0000 7f23d1e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f23d40d1b70 0x7f23d40cff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60368/0 (socket says 172.21.15.89:60368) 2024-04-02T07:52:10.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.175+0000 7f23d1e00700 1 -- 172.21.15.89:0/2460217905 learned_addr learned my addr 172.21.15.89:0/2460217905 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:10.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.175+0000 7f23d2800700 1 --2- 172.21.15.89:0/2460217905 >> v2:172.21.15.113:3300/0 conn(0x7f23d415d710 0x7f23d40d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:10.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.175+0000 7f23d3200700 1 --2- 172.21.15.89:0/2460217905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d40d2020 0x7f23d40d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:10.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.175+0000 7f23d1400700 1 -- 172.21.15.89:0/2460217905 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253897465 0 0) 0x7f23d40cf640 con 0x7f23d40d1b70 2024-04-02T07:52:10.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.175+0000 7f23d1400700 1 -- 172.21.15.89:0/2460217905 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b80035b0 con 0x7f23d40d1b70 2024-04-02T07:52:10.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.176+0000 7f23d2800700 1 -- 172.21.15.89:0/2460217905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d40d2020 msgr2=0x7f23d40d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:10.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.176+0000 7f23d2800700 1 --2- 172.21.15.89:0/2460217905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d40d2020 0x7f23d40d0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:10.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.176+0000 7f23d2800700 1 -- 172.21.15.89:0/2460217905 >> v1:172.21.15.89:6789/0 conn(0x7f23d40d1b70 legacy=0x7f23d40cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:10.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.176+0000 7f23d2800700 1 -- 172.21.15.89:0/2460217905 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d40ce880 con 0x7f23d415d710 2024-04-02T07:52:10.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.176+0000 7f23d3200700 1 --2- 172.21.15.89:0/2460217905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d40d2020 0x7f23d40d0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.176+0000 7f23d2800700 1 --2- 172.21.15.89:0/2460217905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d415d710 0x7f23d40d12f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f23c4009840 tx=0x7f23c4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad7cf56f17b97e3b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.176+0000 7f23d1400700 1 -- 172.21.15.89:0/2460217905 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c4011830 con 0x7f23d415d710 2024-04-02T07:52:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.176+0000 7f23d1400700 1 -- 172.21.15.89:0/2460217905 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c4011990 con 0x7f23d415d710 2024-04-02T07:52:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.176+0000 7f23d1400700 1 -- 172.21.15.89:0/2460217905 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c4011d20 con 0x7f23d415d710 2024-04-02T07:52:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.177+0000 7f23d9400700 1 -- 172.21.15.89:0/2460217905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d415d710 msgr2=0x7f23d40d12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.177+0000 7f23d9400700 1 --2- 172.21.15.89:0/2460217905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d415d710 0x7f23d40d12f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f23c4009840 tx=0x7f23c4009b50 comp rx=0 tx=0).stop 2024-04-02T07:52:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.177+0000 7f23d9400700 1 -- 172.21.15.89:0/2460217905 shutdown_connections 2024-04-02T07:52:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.177+0000 7f23d9400700 1 --2- 172.21.15.89:0/2460217905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d40d2020 0x7f23d40d0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.177+0000 7f23d9400700 1 --2- 172.21.15.89:0/2460217905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d415d710 0x7f23d40d12f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.177+0000 7f23d9400700 1 -- 172.21.15.89:0/2460217905 >> 172.21.15.89:0/2460217905 conn(0x7f23d408aff0 msgr2=0x7f23d40d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.177+0000 7f23d9400700 1 -- 172.21.15.89:0/2460217905 shutdown_connections 2024-04-02T07:52:10.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.177+0000 7f23d9400700 1 -- 172.21.15.89:0/2460217905 wait complete. 2024-04-02T07:52:10.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.179+0000 7f23d9400700 1 Processor -- start 2024-04-02T07:52:10.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.180+0000 7f23d9400700 1 -- start start 2024-04-02T07:52:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d40d1b70 0x7f23d415dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d40d2020 0x7f23d40b5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:10.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d9400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d415d710 0x7f23d40b5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d9400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d40ccf40 con 0x7f23d40d1b70 2024-04-02T07:52:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d9400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d40cc330 con 0x7f23d415d710 2024-04-02T07:52:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d9400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d40cc610 con 0x7f23d40d2020 2024-04-02T07:52:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d40d1b70 0x7f23d415dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d40d1b70 0x7f23d415dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41960/0 (socket says 172.21.15.89:41960) 2024-04-02T07:52:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d2800700 1 -- 172.21.15.89:0/1757425396 learned_addr learned my addr 172.21.15.89:0/1757425396 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d3200700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d415d710 0x7f23d40b5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d1e00700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d40d2020 0x7f23d40b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d2800700 1 -- 172.21.15.89:0/1757425396 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d40d2020 msgr2=0x7f23d40b5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d2800700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d40d2020 0x7f23d40b5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d2800700 1 -- 172.21.15.89:0/1757425396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d415d710 msgr2=0x7f23d40b5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:10.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d2800700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d415d710 0x7f23d40b5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:10.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d2800700 1 -- 172.21.15.89:0/1757425396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc009a70 con 0x7f23d40d1b70 2024-04-02T07:52:10.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.181+0000 7f23d3200700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d415d710 0x7f23d40b5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:10.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.182+0000 7f23d2800700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d40d1b70 0x7f23d415dd60 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f23c4009fd0 tx=0x7f23c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:10.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.182+0000 7f23caa00700 1 -- 172.21.15.89:0/1757425396 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c40036a0 con 0x7f23d40d1b70 2024-04-02T07:52:10.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.182+0000 7f23caa00700 1 -- 172.21.15.89:0/1757425396 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c401e070 con 0x7f23d40d1b70 2024-04-02T07:52:10.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.182+0000 7f23caa00700 1 -- 172.21.15.89:0/1757425396 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c4003a60 con 0x7f23d40d1b70 2024-04-02T07:52:10.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.182+0000 7f23d9400700 1 -- 172.21.15.89:0/1757425396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c40094a0 con 0x7f23d40d1b70 2024-04-02T07:52:10.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.182+0000 7f23d9400700 1 -- 172.21.15.89:0/1757425396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d40ccbd0 con 0x7f23d40d1b70 2024-04-02T07:52:10.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.183+0000 7f23d9400700 1 -- 172.21.15.89:0/1757425396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d408b5a0 con 0x7f23d40d1b70 2024-04-02T07:52:10.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.184+0000 7f23caa00700 1 -- 172.21.15.89:0/1757425396 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f23c4003bc0 con 0x7f23d40d1b70 2024-04-02T07:52:10.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.185+0000 7f23caa00700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f23b406adb0 0x7f23b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:10.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.185+0000 7f23caa00700 1 -- 172.21.15.89:0/1757425396 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f23c408a510 con 0x7f23d40d1b70 2024-04-02T07:52:10.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.185+0000 7f23d1e00700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f23b406adb0 0x7f23b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:10.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.185+0000 7f23d1e00700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f23b406adb0 0x7f23b406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f23c0007fd0 tx=0x7f23c0009040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:10.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.189+0000 7f23caa00700 1 -- 172.21.15.89:0/1757425396 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f23c4055470 con 0x7f23d40d1b70 2024-04-02T07:52:10.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:10.437+0000 7f23d9400700 1 -- 172.21.15.89:0/1757425396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f23d4067e30 con 0x7f23d40d1b70 2024-04-02T07:52:11.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.081+0000 7f23caa00700 1 -- 172.21.15.89:0/1757425396 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v40) v1 ==== 118+0+0 (secure 0 0 0) 0x7f23c4058a90 con 0x7f23d40d1b70 2024-04-02T07:52:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.084+0000 7f23d9400700 1 -- 172.21.15.89:0/1757425396 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f23b406adb0 msgr2=0x7f23b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.084+0000 7f23d9400700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f23b406adb0 0x7f23b406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f23c0007fd0 tx=0x7f23c0009040 comp rx=0 tx=0).stop 2024-04-02T07:52:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.084+0000 7f23d9400700 1 -- 172.21.15.89:0/1757425396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d40d1b70 msgr2=0x7f23d415dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.084+0000 7f23d9400700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d40d1b70 0x7f23d415dd60 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f23c4009fd0 tx=0x7f23c4004750 comp rx=0 tx=0).stop 2024-04-02T07:52:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.084+0000 7f23d9400700 1 -- 172.21.15.89:0/1757425396 shutdown_connections 2024-04-02T07:52:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.084+0000 7f23d9400700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f23b406adb0 0x7f23b406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.084+0000 7f23d9400700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d415d710 0x7f23d40b5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.084+0000 7f23d9400700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d40d1b70 0x7f23d415dd60 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.084+0000 7f23d9400700 1 --2- 172.21.15.89:0/1757425396 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d40d2020 0x7f23d40b5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.084+0000 7f23d9400700 1 -- 172.21.15.89:0/1757425396 >> 172.21.15.89:0/1757425396 conn(0x7f23d408aff0 msgr2=0x7f23d40b15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.084+0000 7f23d9400700 1 -- 172.21.15.89:0/1757425396 shutdown_connections 2024-04-02T07:52:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.084+0000 7f23d9400700 1 -- 172.21.15.89:0/1757425396 wait complete. 2024-04-02T07:52:11.104 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T07:52:11.105 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T07:52:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.299+0000 7f0c5de00700 1 Processor -- start 2024-04-02T07:52:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.299+0000 7f0c5de00700 1 -- start start 2024-04-02T07:52:11.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.300+0000 7f0c5de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0c580cbb30 0x7f0c580cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:11.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.300+0000 7f0c5de00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c580cc450 con 0x7f0c580cd020 2024-04-02T07:52:11.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.300+0000 7f0c5de00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c580cdae0 con 0x7f0c580cd480 2024-04-02T07:52:11.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.300+0000 7f0c5de00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c580cd8f0 con 0x7f0c580cbb30 2024-04-02T07:52:11.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.300+0000 7f0c56a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0c580cbb30 0x7f0c580cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:11.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.300+0000 7f0c56a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0c580cbb30 0x7f0c580cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41976/0 (socket says 172.21.15.89:41976) 2024-04-02T07:52:11.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.300+0000 7f0c56a00700 1 -- 172.21.15.89:0/923788470 learned_addr learned my addr 172.21.15.89:0/923788470 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:11.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.300+0000 7f0c56000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0c580cd020 0x7f0c580b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60372/0 (socket says 172.21.15.89:60372) 2024-04-02T07:52:11.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.301+0000 7f0c56a00700 1 -- 172.21.15.89:0/923788470 >> v1:172.21.15.113:6789/0 conn(0x7f0c580cd480 legacy=0x7f0c5806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:11.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.301+0000 7f0c56a00700 1 -- 172.21.15.89:0/923788470 >> v1:172.21.15.89:6789/0 conn(0x7f0c580cd020 legacy=0x7f0c580b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:11.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.301+0000 7f0c56a00700 1 -- 172.21.15.89:0/923788470 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c580cddc0 con 0x7f0c580cbb30 2024-04-02T07:52:11.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.301+0000 7f0c55600700 1 -- 172.21.15.89:0/923788470 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786561545 0 0) 0x7f0c580cc450 con 0x7f0c580cd020 2024-04-02T07:52:11.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.301+0000 7f0c56a00700 1 --2- 172.21.15.89:0/923788470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c580cbb30 0x7f0c580cbf10 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f0c480098f0 tx=0x7f0c48009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e6e120d1f25126 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:11.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.301+0000 7f0c55600700 1 -- 172.21.15.89:0/923788470 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48004230 con 0x7f0c580cbb30 2024-04-02T07:52:11.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.301+0000 7f0c55600700 1 -- 172.21.15.89:0/923788470 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c48004390 con 0x7f0c580cbb30 2024-04-02T07:52:11.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.301+0000 7f0c55600700 1 -- 172.21.15.89:0/923788470 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c480036a0 con 0x7f0c580cbb30 2024-04-02T07:52:11.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.302+0000 7f0c5de00700 1 -- 172.21.15.89:0/923788470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c580cbb30 msgr2=0x7f0c580cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:11.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.302+0000 7f0c5de00700 1 --2- 172.21.15.89:0/923788470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c580cbb30 0x7f0c580cbf10 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f0c480098f0 tx=0x7f0c48009c00 comp rx=0 tx=0).stop 2024-04-02T07:52:11.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.302+0000 7f0c5de00700 1 -- 172.21.15.89:0/923788470 shutdown_connections 2024-04-02T07:52:11.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.302+0000 7f0c5de00700 1 --2- 172.21.15.89:0/923788470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c580cbb30 0x7f0c580cbf10 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:11.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.302+0000 7f0c5de00700 1 -- 172.21.15.89:0/923788470 >> 172.21.15.89:0/923788470 conn(0x7f0c580b0d70 msgr2=0x7f0c580b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:11.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.302+0000 7f0c5de00700 1 -- 172.21.15.89:0/923788470 shutdown_connections 2024-04-02T07:52:11.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.302+0000 7f0c5de00700 1 -- 172.21.15.89:0/923788470 wait complete. 2024-04-02T07:52:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.303+0000 7f0c5de00700 1 Processor -- start 2024-04-02T07:52:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.304+0000 7f0c5de00700 1 -- start start 2024-04-02T07:52:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.304+0000 7f0c5de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c580cbb30 0x7f0c58157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.304+0000 7f0c5de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c580cd020 0x7f0c580af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c5de00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c580cd480 0x7f0c580b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c5de00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580aef30 con 0x7f0c580cbb30 2024-04-02T07:52:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c5de00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c58158430 con 0x7f0c580cd480 2024-04-02T07:52:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c5de00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580ae600 con 0x7f0c580cd020 2024-04-02T07:52:11.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c56a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c580cbb30 0x7f0c58157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:11.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c56a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c580cbb30 0x7f0c58157b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41978/0 (socket says 172.21.15.89:41978) 2024-04-02T07:52:11.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c56a00700 1 -- 172.21.15.89:0/4205532259 learned_addr learned my addr 172.21.15.89:0/4205532259 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:11.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c57400700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c580cd480 0x7f0c580b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:11.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c56a00700 1 -- 172.21.15.89:0/4205532259 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c580cd020 msgr2=0x7f0c580af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:11.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c56000700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c580cd020 0x7f0c580af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:11.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c56a00700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c580cd020 0x7f0c580af3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:11.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c56a00700 1 -- 172.21.15.89:0/4205532259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c580cd480 msgr2=0x7f0c580b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:11.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c56a00700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c580cd480 0x7f0c580b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:11.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c56a00700 1 -- 172.21.15.89:0/4205532259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c48009550 con 0x7f0c580cbb30 2024-04-02T07:52:11.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c56000700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c580cd020 0x7f0c580af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:11.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c57400700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c580cd480 0x7f0c580b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:11.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.305+0000 7f0c56a00700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c580cbb30 0x7f0c58157b20 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f0c5806c130 tx=0x7f0c48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:11.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.306+0000 7f0c4f400700 1 -- 172.21.15.89:0/4205532259 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4800b660 con 0x7f0c580cbb30 2024-04-02T07:52:11.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.306+0000 7f0c4f400700 1 -- 172.21.15.89:0/4205532259 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4800b7e0 con 0x7f0c580cbb30 2024-04-02T07:52:11.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.306+0000 7f0c5de00700 1 -- 172.21.15.89:0/4205532259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c580ae8e0 con 0x7f0c580cbb30 2024-04-02T07:52:11.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.306+0000 7f0c4f400700 1 -- 172.21.15.89:0/4205532259 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4800bb90 con 0x7f0c580cbb30 2024-04-02T07:52:11.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.306+0000 7f0c5de00700 1 -- 172.21.15.89:0/4205532259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c58158060 con 0x7f0c580cbb30 2024-04-02T07:52:11.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.307+0000 7f0c5de00700 1 -- 172.21.15.89:0/4205532259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c580af210 con 0x7f0c580cbb30 2024-04-02T07:52:11.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.308+0000 7f0c4f400700 1 -- 172.21.15.89:0/4205532259 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0c4800bcf0 con 0x7f0c580cbb30 2024-04-02T07:52:11.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.309+0000 7f0c4f400700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c3806ade0 0x7f0c3806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:11.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.309+0000 7f0c4f400700 1 -- 172.21.15.89:0/4205532259 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0c4808af30 con 0x7f0c580cbb30 2024-04-02T07:52:11.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.309+0000 7f0c56000700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c3806ade0 0x7f0c3806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:11.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.309+0000 7f0c56000700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c3806ade0 0x7f0c3806d2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c44007fd0 tx=0x7f0c44009040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:11.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.313+0000 7f0c4f400700 1 -- 172.21.15.89:0/4205532259 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0c48056050 con 0x7f0c580cbb30 2024-04-02T07:52:11.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:11.556+0000 7f0c5de00700 1 -- 172.21.15.89:0/4205532259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f0c58067720 con 0x7f0c580cbb30 2024-04-02T07:52:12.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.083+0000 7f0c4f400700 1 -- 172.21.15.89:0/4205532259 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v41) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0c48059670 con 0x7f0c580cbb30 2024-04-02T07:52:12.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.085+0000 7f0c5de00700 1 -- 172.21.15.89:0/4205532259 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c3806ade0 msgr2=0x7f0c3806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:12.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.085+0000 7f0c5de00700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c3806ade0 0x7f0c3806d2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c44007fd0 tx=0x7f0c44009040 comp rx=0 tx=0).stop 2024-04-02T07:52:12.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.085+0000 7f0c5de00700 1 -- 172.21.15.89:0/4205532259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c580cbb30 msgr2=0x7f0c58157b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:12.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.085+0000 7f0c5de00700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c580cbb30 0x7f0c58157b20 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f0c5806c130 tx=0x7f0c48004750 comp rx=0 tx=0).stop 2024-04-02T07:52:12.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.086+0000 7f0c5de00700 1 -- 172.21.15.89:0/4205532259 shutdown_connections 2024-04-02T07:52:12.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.086+0000 7f0c5de00700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c3806ade0 0x7f0c3806d2a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.086+0000 7f0c5de00700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c580cd480 0x7f0c580b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.086+0000 7f0c5de00700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c580cbb30 0x7f0c58157b20 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.086+0000 7f0c5de00700 1 --2- 172.21.15.89:0/4205532259 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c580cd020 0x7f0c580af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.086+0000 7f0c5de00700 1 -- 172.21.15.89:0/4205532259 >> 172.21.15.89:0/4205532259 conn(0x7f0c580b0d70 msgr2=0x7f0c580ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:12.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.086+0000 7f0c5de00700 1 -- 172.21.15.89:0/4205532259 shutdown_connections 2024-04-02T07:52:12.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.086+0000 7f0c5de00700 1 -- 172.21.15.89:0/4205532259 wait complete. 2024-04-02T07:52:12.106 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T07:52:12.106 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:52:12.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.297+0000 7fc214000700 1 Processor -- start 2024-04-02T07:52:12.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.297+0000 7fc214000700 1 -- start start 2024-04-02T07:52:12.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.298+0000 7fc214000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc20c15b5d0 0x7fc20c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:12.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.298+0000 7fc214000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc20c0cb3f0 0x7fc20c0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:12.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.298+0000 7fc214000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc20c0ae930 0x7fc20c0ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:12.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.298+0000 7fc214000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc20c15b9b0 con 0x7fc20c0ae930 2024-04-02T07:52:12.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.298+0000 7fc214000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc20c15fd50 con 0x7fc20c0cb3f0 2024-04-02T07:52:12.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.298+0000 7fc214000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c15f140 con 0x7fc20c15b5d0 2024-04-02T07:52:12.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.299+0000 7fc212000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc20c0ae930 0x7fc20c0ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:12.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.299+0000 7fc212000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc20c0ae930 0x7fc20c0ac600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41984/0 (socket says 172.21.15.89:41984) 2024-04-02T07:52:12.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.299+0000 7fc212000700 1 -- 172.21.15.89:0/3333186696 learned_addr learned my addr 172.21.15.89:0/3333186696 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:12.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.299+0000 7fc210c00700 1 --2- 172.21.15.89:0/3333186696 >> v2:172.21.15.113:3300/0 conn(0x7fc20c0cb3f0 0x7fc20c0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:12.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.299+0000 7fc211600700 1 --2- 172.21.15.89:0/3333186696 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc20c15b5d0 0x7fc20c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:12.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.299+0000 7fc212000700 1 -- 172.21.15.89:0/3333186696 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc20c15b5d0 msgr2=0x7fc20c0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:12.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.299+0000 7fc212000700 1 --2- 172.21.15.89:0/3333186696 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc20c15b5d0 0x7fc20c0ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.299+0000 7fc212000700 1 -- 172.21.15.89:0/3333186696 >> v2:172.21.15.113:3300/0 conn(0x7fc20c0cb3f0 msgr2=0x7fc20c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:12.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.299+0000 7fc212000700 1 --2- 172.21.15.89:0/3333186696 >> v2:172.21.15.113:3300/0 conn(0x7fc20c0cb3f0 0x7fc20c0ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.299+0000 7fc212000700 1 -- 172.21.15.89:0/3333186696 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20c15f420 con 0x7fc20c0ae930 2024-04-02T07:52:12.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.299+0000 7fc210c00700 1 --2- 172.21.15.89:0/3333186696 >> v2:172.21.15.113:3300/0 conn(0x7fc20c0cb3f0 0x7fc20c0ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:12.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.299+0000 7fc212000700 1 --2- 172.21.15.89:0/3333186696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20c0ae930 0x7fc20c0ac600 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0700f0 tx=0x7fc20400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ec258fcb7f5d18 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:12.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.299+0000 7fc20be00700 1 -- 172.21.15.89:0/3333186696 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20400e070 con 0x7fc20c0ae930 2024-04-02T07:52:12.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.299+0000 7fc20be00700 1 -- 172.21.15.89:0/3333186696 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20400b9c0 con 0x7fc20c0ae930 2024-04-02T07:52:12.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.300+0000 7fc20be00700 1 -- 172.21.15.89:0/3333186696 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20400bd70 con 0x7fc20c0ae930 2024-04-02T07:52:12.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.300+0000 7fc214000700 1 -- 172.21.15.89:0/3333186696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20c0ae930 msgr2=0x7fc20c0ac600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:12.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.300+0000 7fc214000700 1 --2- 172.21.15.89:0/3333186696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20c0ae930 0x7fc20c0ac600 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0700f0 tx=0x7fc20400b330 comp rx=0 tx=0).stop 2024-04-02T07:52:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.300+0000 7fc214000700 1 -- 172.21.15.89:0/3333186696 shutdown_connections 2024-04-02T07:52:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.300+0000 7fc214000700 1 --2- 172.21.15.89:0/3333186696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20c0ae930 0x7fc20c0ac600 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.300+0000 7fc214000700 1 --2- 172.21.15.89:0/3333186696 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc20c15b5d0 0x7fc20c0ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.300+0000 7fc214000700 1 --2- 172.21.15.89:0/3333186696 >> v2:172.21.15.113:3300/0 conn(0x7fc20c0cb3f0 0x7fc20c0ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.300+0000 7fc214000700 1 -- 172.21.15.89:0/3333186696 >> 172.21.15.89:0/3333186696 conn(0x7fc20c0cf190 msgr2=0x7fc20c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.300+0000 7fc214000700 1 -- 172.21.15.89:0/3333186696 shutdown_connections 2024-04-02T07:52:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.300+0000 7fc214000700 1 -- 172.21.15.89:0/3333186696 wait complete. 2024-04-02T07:52:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.302+0000 7fc214000700 1 Processor -- start 2024-04-02T07:52:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.302+0000 7fc214000700 1 -- start start 2024-04-02T07:52:12.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.303+0000 7fc214000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc20c0ae930 0x7fc20c14eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:12.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.303+0000 7fc214000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc20c15b5d0 0x7fc20c1f6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:12.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.303+0000 7fc214000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20c1f6950 0x7fc20c158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:12.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.303+0000 7fc214000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c1f7400 con 0x7fc20c1f6950 2024-04-02T07:52:12.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.303+0000 7fc214000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c1f6e40 con 0x7fc20c15b5d0 2024-04-02T07:52:12.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.303+0000 7fc214000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c1f7120 con 0x7fc20c0ae930 2024-04-02T07:52:12.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.303+0000 7fc212000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20c1f6950 0x7fc20c158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.303+0000 7fc212000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20c1f6950 0x7fc20c158010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41992/0 (socket says 172.21.15.89:41992) 2024-04-02T07:52:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.303+0000 7fc212000700 1 -- 172.21.15.89:0/71293106 learned_addr learned my addr 172.21.15.89:0/71293106 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.303+0000 7fc210c00700 1 --2- 172.21.15.89:0/71293106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc20c15b5d0 0x7fc20c1f6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.303+0000 7fc211600700 1 --2- 172.21.15.89:0/71293106 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc20c0ae930 0x7fc20c14eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.304+0000 7fc212000700 1 -- 172.21.15.89:0/71293106 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc20c0ae930 msgr2=0x7fc20c14eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.304+0000 7fc212000700 1 --2- 172.21.15.89:0/71293106 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc20c0ae930 0x7fc20c14eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.304+0000 7fc212000700 1 -- 172.21.15.89:0/71293106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc20c15b5d0 msgr2=0x7fc20c1f6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.304+0000 7fc212000700 1 --2- 172.21.15.89:0/71293106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc20c15b5d0 0x7fc20c1f6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.304+0000 7fc212000700 1 -- 172.21.15.89:0/71293106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc204009b30 con 0x7fc20c1f6950 2024-04-02T07:52:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.304+0000 7fc212000700 1 --2- 172.21.15.89:0/71293106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20c1f6950 0x7fc20c158010 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fc204012000 tx=0x7fc204004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.304+0000 7fc209600700 1 -- 172.21.15.89:0/71293106 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2040099b0 con 0x7fc20c1f6950 2024-04-02T07:52:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.304+0000 7fc209600700 1 -- 172.21.15.89:0/71293106 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc204004250 con 0x7fc20c1f6950 2024-04-02T07:52:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.304+0000 7fc214000700 1 -- 172.21.15.89:0/71293106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc20c0cbd40 con 0x7fc20c1f6950 2024-04-02T07:52:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.304+0000 7fc209600700 1 -- 172.21.15.89:0/71293106 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc204004600 con 0x7fc20c1f6950 2024-04-02T07:52:12.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.304+0000 7fc214000700 1 -- 172.21.15.89:0/71293106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20c0b3790 con 0x7fc20c1f6950 2024-04-02T07:52:12.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.305+0000 7fc214000700 1 -- 172.21.15.89:0/71293106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc20c0cc260 con 0x7fc20c1f6950 2024-04-02T07:52:12.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.306+0000 7fc209600700 1 -- 172.21.15.89:0/71293106 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc204004760 con 0x7fc20c1f6950 2024-04-02T07:52:12.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.307+0000 7fc209600700 1 --2- 172.21.15.89:0/71293106 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1ec06adb0 0x7fc1ec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:12.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.307+0000 7fc209600700 1 -- 172.21.15.89:0/71293106 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc20408aac0 con 0x7fc20c1f6950 2024-04-02T07:52:12.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.307+0000 7fc211600700 1 --2- 172.21.15.89:0/71293106 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1ec06adb0 0x7fc1ec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:12.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.308+0000 7fc211600700 1 --2- 172.21.15.89:0/71293106 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1ec06adb0 0x7fc1ec06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc200006b20 tx=0x7fc200008040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:12.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.311+0000 7fc209600700 1 -- 172.21.15.89:0/71293106 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc20c0cc260 con 0x7fc20c1f6950 2024-04-02T07:52:12.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.559+0000 7fc214000700 1 -- 172.21.15.89:0/71293106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc20c0cc260 con 0x7fc20c1f6950 2024-04-02T07:52:12.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.560+0000 7fc209600700 1 -- 172.21.15.89:0/71293106 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fc20c0cc260 con 0x7fc20c1f6950 2024-04-02T07:52:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:12.562 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6556,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3307197948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3307197948},{"type":"v1","addr":"172.21.15.113:6835","nonce":3307197948}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1797655686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1797655686},{"type":"v1","addr":"172.21.15.113:6834","nonce":1797655686}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4203087300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4203087300},{"type":"v1","addr":"172.21.15.89:6836","nonce":4203087300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:52:09.066449+0000","modified":"2024-04-02T07:52:12.080422+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3228984565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3228984565},{"type":"v1","addr":"172.21.15.89:6837","nonce":3228984565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-02T07:52:12.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.563+0000 7fc214000700 1 -- 172.21.15.89:0/71293106 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1ec06adb0 msgr2=0x7fc1ec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:12.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.563+0000 7fc214000700 1 --2- 172.21.15.89:0/71293106 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1ec06adb0 0x7fc1ec06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc200006b20 tx=0x7fc200008040 comp rx=0 tx=0).stop 2024-04-02T07:52:12.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.563+0000 7fc214000700 1 -- 172.21.15.89:0/71293106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20c1f6950 msgr2=0x7fc20c158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:12.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.563+0000 7fc214000700 1 --2- 172.21.15.89:0/71293106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20c1f6950 0x7fc20c158010 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fc204012000 tx=0x7fc204004ad0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.563+0000 7fc214000700 1 -- 172.21.15.89:0/71293106 shutdown_connections 2024-04-02T07:52:12.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.563+0000 7fc214000700 1 --2- 172.21.15.89:0/71293106 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1ec06adb0 0x7fc1ec06d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.563+0000 7fc214000700 1 --2- 172.21.15.89:0/71293106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20c1f6950 0x7fc20c158010 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.563+0000 7fc214000700 1 --2- 172.21.15.89:0/71293106 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc20c0ae930 0x7fc20c14eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.563+0000 7fc214000700 1 --2- 172.21.15.89:0/71293106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc20c15b5d0 0x7fc20c1f6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.563+0000 7fc214000700 1 -- 172.21.15.89:0/71293106 >> 172.21.15.89:0/71293106 conn(0x7fc20c0cf190 msgr2=0x7fc20c15e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:12.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.563+0000 7fc214000700 1 -- 172.21.15.89:0/71293106 shutdown_connections 2024-04-02T07:52:12.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.563+0000 7fc214000700 1 -- 172.21.15.89:0/71293106 wait complete. 2024-04-02T07:52:12.565 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 41 2024-04-02T07:52:12.582 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:52:12.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.773+0000 7f3870400700 1 Processor -- start 2024-04-02T07:52:12.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.773+0000 7f3870400700 1 -- start start 2024-04-02T07:52:12.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.774+0000 7f3870400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3868160f10 0x7f38680d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:12.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.774+0000 7f3870400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3868161590 0x7f3868161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:12.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.774+0000 7f3870400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38680ceda0 con 0x7f38680cf230 2024-04-02T07:52:12.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.774+0000 7f3870400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3868161f80 con 0x7f3868160f10 2024-04-02T07:52:12.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.774+0000 7f3870400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38681612f0 con 0x7f3868161590 2024-04-02T07:52:12.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.775+0000 7f386da00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3868160f10 0x7f38680d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:12.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.775+0000 7f386d000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f38680cf230 0x7f38680d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60374/0 (socket says 172.21.15.89:60374) 2024-04-02T07:52:12.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.775+0000 7f386d000700 1 -- 172.21.15.89:0/2523803147 learned_addr learned my addr 172.21.15.89:0/2523803147 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:12.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.775+0000 7f386da00700 1 --2- 172.21.15.89:0/2523803147 >> v2:172.21.15.89:3300/0 conn(0x7f3868160f10 0x7f38680d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42000/0 (socket says 172.21.15.89:42000) 2024-04-02T07:52:12.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.775+0000 7f386e400700 1 --2- 172.21.15.89:0/2523803147 >> v2:172.21.15.113:3300/0 conn(0x7f3868161590 0x7f3868161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:12.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.775+0000 7f386da00700 1 -- 172.21.15.89:0/2523803147 >> v2:172.21.15.113:3300/0 conn(0x7f3868161590 msgr2=0x7f3868161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:12.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.775+0000 7f386da00700 1 --2- 172.21.15.89:0/2523803147 >> v2:172.21.15.113:3300/0 conn(0x7f3868161590 0x7f3868161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.775+0000 7f386da00700 1 -- 172.21.15.89:0/2523803147 >> v1:172.21.15.89:6789/0 conn(0x7f38680cf230 legacy=0x7f38680d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:12.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.775+0000 7f386da00700 1 -- 172.21.15.89:0/2523803147 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38680ce470 con 0x7f3868160f10 2024-04-02T07:52:12.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.775+0000 7f3867e00700 1 -- 172.21.15.89:0/2523803147 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852419104 0 0) 0x7f38680ceda0 con 0x7f38680cf230 2024-04-02T07:52:12.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.775+0000 7f386da00700 1 --2- 172.21.15.89:0/2523803147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3868160f10 0x7f38680d18b0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f38680d1030 tx=0x7f385c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea52d7aa207e74d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:12.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.776+0000 7f3867e00700 1 -- 172.21.15.89:0/2523803147 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c011830 con 0x7f3868160f10 2024-04-02T07:52:12.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.776+0000 7f3867e00700 1 -- 172.21.15.89:0/2523803147 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385c011990 con 0x7f3868160f10 2024-04-02T07:52:12.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.776+0000 7f3867e00700 1 -- 172.21.15.89:0/2523803147 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c011d20 con 0x7f3868160f10 2024-04-02T07:52:12.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.776+0000 7f3870400700 1 -- 172.21.15.89:0/2523803147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3868160f10 msgr2=0x7f38680d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:12.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.776+0000 7f3870400700 1 --2- 172.21.15.89:0/2523803147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3868160f10 0x7f38680d18b0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f38680d1030 tx=0x7f385c009b20 comp rx=0 tx=0).stop 2024-04-02T07:52:12.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.776+0000 7f3870400700 1 -- 172.21.15.89:0/2523803147 shutdown_connections 2024-04-02T07:52:12.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.776+0000 7f3870400700 1 --2- 172.21.15.89:0/2523803147 >> v2:172.21.15.113:3300/0 conn(0x7f3868161590 0x7f3868161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.776+0000 7f3870400700 1 --2- 172.21.15.89:0/2523803147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3868160f10 0x7f38680d18b0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.776+0000 7f3870400700 1 -- 172.21.15.89:0/2523803147 >> 172.21.15.89:0/2523803147 conn(0x7f386808aff0 msgr2=0x7f38680d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:12.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.776+0000 7f3870400700 1 -- 172.21.15.89:0/2523803147 shutdown_connections 2024-04-02T07:52:12.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.776+0000 7f3870400700 1 -- 172.21.15.89:0/2523803147 wait complete. 2024-04-02T07:52:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.778+0000 7f3870400700 1 Processor -- start 2024-04-02T07:52:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.778+0000 7f3870400700 1 -- start start 2024-04-02T07:52:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.779+0000 7f3870400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38680cf230 0x7f386815f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.779+0000 7f3870400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3868160f10 0x7f38680b8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.779+0000 7f3870400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3868161590 0x7f38680a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.779+0000 7f3870400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386815fb90 con 0x7f3868161590 2024-04-02T07:52:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.779+0000 7f3870400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386815fe70 con 0x7f38680cf230 2024-04-02T07:52:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.779+0000 7f3870400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3868160150 con 0x7f3868160f10 2024-04-02T07:52:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.779+0000 7f386e400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3868161590 0x7f38680a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.779+0000 7f386e400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3868161590 0x7f38680a0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42002/0 (socket says 172.21.15.89:42002) 2024-04-02T07:52:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.779+0000 7f386e400700 1 -- 172.21.15.89:0/3820282873 learned_addr learned my addr 172.21.15.89:0/3820282873 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.779+0000 7f386da00700 1 --2- 172.21.15.89:0/3820282873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38680cf230 0x7f386815f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.779+0000 7f386d000700 1 --2- 172.21.15.89:0/3820282873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3868160f10 0x7f38680b8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.780+0000 7f386e400700 1 -- 172.21.15.89:0/3820282873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3868160f10 msgr2=0x7f38680b8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.780+0000 7f386e400700 1 --2- 172.21.15.89:0/3820282873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3868160f10 0x7f38680b8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.780+0000 7f386e400700 1 -- 172.21.15.89:0/3820282873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38680cf230 msgr2=0x7f386815f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.780+0000 7f386e400700 1 --2- 172.21.15.89:0/3820282873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38680cf230 0x7f386815f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.780+0000 7f386e400700 1 -- 172.21.15.89:0/3820282873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385c0094a0 con 0x7f3868161590 2024-04-02T07:52:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.780+0000 7f386e400700 1 --2- 172.21.15.89:0/3820282873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3868161590 0x7f38680a0a70 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f3860007e60 tx=0x7f386000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.780+0000 7f3865600700 1 -- 172.21.15.89:0/3820282873 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386000ed30 con 0x7f3868161590 2024-04-02T07:52:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.780+0000 7f3865600700 1 -- 172.21.15.89:0/3820282873 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3860014420 con 0x7f3868161590 2024-04-02T07:52:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.780+0000 7f3870400700 1 -- 172.21.15.89:0/3820282873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f386809d440 con 0x7f3868161590 2024-04-02T07:52:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.780+0000 7f3865600700 1 -- 172.21.15.89:0/3820282873 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38600147d0 con 0x7f3868161590 2024-04-02T07:52:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.780+0000 7f3870400700 1 -- 172.21.15.89:0/3820282873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386815f7f0 con 0x7f3868161590 2024-04-02T07:52:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.781+0000 7f3870400700 1 -- 172.21.15.89:0/3820282873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3868067720 con 0x7f3868161590 2024-04-02T07:52:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.782+0000 7f3865600700 1 -- 172.21.15.89:0/3820282873 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3860014930 con 0x7f3868161590 2024-04-02T07:52:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.783+0000 7f3865600700 1 --2- 172.21.15.89:0/3820282873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f384806ade0 0x7f384806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.783+0000 7f3865600700 1 -- 172.21.15.89:0/3820282873 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3860089bb0 con 0x7f3868161590 2024-04-02T07:52:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.783+0000 7f386da00700 1 --2- 172.21.15.89:0/3820282873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f384806ade0 0x7f384806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.784+0000 7f386da00700 1 --2- 172.21.15.89:0/3820282873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f384806ade0 0x7f384806d2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f385c000c00 tx=0x7f385c01c040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.787+0000 7f3865600700 1 -- 172.21.15.89:0/3820282873 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3860054bd0 con 0x7f3868161590 2024-04-02T07:52:12.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.994+0000 7f3870400700 1 -- 172.21.15.89:0/3820282873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f386815c7c0 con 0x7f3868161590 2024-04-02T07:52:12.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.995+0000 7f3865600700 1 -- 172.21.15.89:0/3820282873 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16567 (secure 0 0 0) 0x7f38600581f0 con 0x7f3868161590 2024-04-02T07:52:12.997 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:12.997 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":97,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:52:10.069297+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:52:06.991300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-02T07:52:08.054066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:52:12.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.997+0000 7f3870400700 1 -- 172.21.15.89:0/3820282873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f384806ade0 msgr2=0x7f384806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:12.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.997+0000 7f3870400700 1 --2- 172.21.15.89:0/3820282873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f384806ade0 0x7f384806d2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f385c000c00 tx=0x7f385c01c040 comp rx=0 tx=0).stop 2024-04-02T07:52:12.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.997+0000 7f3870400700 1 -- 172.21.15.89:0/3820282873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3868161590 msgr2=0x7f38680a0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:12.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.997+0000 7f3870400700 1 --2- 172.21.15.89:0/3820282873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3868161590 0x7f38680a0a70 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f3860007e60 tx=0x7f386000e3c0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.997+0000 7f3870400700 1 -- 172.21.15.89:0/3820282873 shutdown_connections 2024-04-02T07:52:12.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.997+0000 7f3870400700 1 --2- 172.21.15.89:0/3820282873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f384806ade0 0x7f384806d2a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.997+0000 7f3870400700 1 --2- 172.21.15.89:0/3820282873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38680cf230 0x7f386815f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:12.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.997+0000 7f3870400700 1 --2- 172.21.15.89:0/3820282873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3868161590 0x7f38680a0a70 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.997+0000 7f3870400700 1 --2- 172.21.15.89:0/3820282873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3868160f10 0x7f38680b8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.997+0000 7f3870400700 1 -- 172.21.15.89:0/3820282873 >> 172.21.15.89:0/3820282873 conn(0x7f386808aff0 msgr2=0x7f38680cc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:13.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.998+0000 7f3870400700 1 -- 172.21.15.89:0/3820282873 shutdown_connections 2024-04-02T07:52:13.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:12.998+0000 7f3870400700 1 -- 172.21.15.89:0/3820282873 wait complete. 2024-04-02T07:52:13.018 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T07:52:13.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:52:13.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.209+0000 7fc48e400700 1 Processor -- start 2024-04-02T07:52:13.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.209+0000 7fc48e400700 1 -- start start 2024-04-02T07:52:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.210+0000 7fc48e400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48815d640 0x7fc4880d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.210+0000 7fc48e400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4880d30d0 con 0x7fc4881619d0 2024-04-02T07:52:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.210+0000 7fc48e400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4880b6840 con 0x7fc4880ceec0 2024-04-02T07:52:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.210+0000 7fc48e400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4880d2880 con 0x7fc48815d640 2024-04-02T07:52:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.211+0000 7fc487e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc4881619d0 0x7fc4880b6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60378/0 (socket says 172.21.15.89:60378) 2024-04-02T07:52:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.211+0000 7fc487e00700 1 -- 172.21.15.89:0/2932910527 learned_addr learned my addr 172.21.15.89:0/2932910527 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.211+0000 7fc487400700 1 --2- 172.21.15.89:0/2932910527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48815d640 0x7fc4880d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.211+0000 7fc486000700 1 -- 172.21.15.89:0/2932910527 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565059194 0 0) 0x7fc4880d30d0 con 0x7fc4881619d0 2024-04-02T07:52:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.211+0000 7fc486000700 1 -- 172.21.15.89:0/2932910527 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc46c0035b0 con 0x7fc4881619d0 2024-04-02T07:52:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.211+0000 7fc486000700 1 -- 172.21.15.89:0/2932910527 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650831531 0 0) 0x7fc4880b6840 con 0x7fc4880ceec0 2024-04-02T07:52:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.211+0000 7fc487400700 1 -- 172.21.15.89:0/2932910527 >> v1:172.21.15.113:6789/0 conn(0x7fc4880ceec0 legacy=0x7fc4881612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.211+0000 7fc487400700 1 -- 172.21.15.89:0/2932910527 >> v1:172.21.15.89:6789/0 conn(0x7fc4881619d0 legacy=0x7fc4880b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.211+0000 7fc487400700 1 -- 172.21.15.89:0/2932910527 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4880d2b30 con 0x7fc48815d640 2024-04-02T07:52:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.211+0000 7fc487400700 1 --2- 172.21.15.89:0/2932910527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48815d640 0x7fc4880d2340 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc478009870 tx=0x7fc478009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d0a43ce45abbba3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.212+0000 7fc486000700 1 -- 172.21.15.89:0/2932910527 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc478011830 con 0x7fc48815d640 2024-04-02T07:52:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.212+0000 7fc486000700 1 -- 172.21.15.89:0/2932910527 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc478011990 con 0x7fc48815d640 2024-04-02T07:52:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.212+0000 7fc486000700 1 -- 172.21.15.89:0/2932910527 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc478011d20 con 0x7fc48815d640 2024-04-02T07:52:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.212+0000 7fc48e400700 1 -- 172.21.15.89:0/2932910527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48815d640 msgr2=0x7fc4880d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.212+0000 7fc48e400700 1 --2- 172.21.15.89:0/2932910527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48815d640 0x7fc4880d2340 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc478009870 tx=0x7fc478009b80 comp rx=0 tx=0).stop 2024-04-02T07:52:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.212+0000 7fc48e400700 1 -- 172.21.15.89:0/2932910527 shutdown_connections 2024-04-02T07:52:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.212+0000 7fc48e400700 1 --2- 172.21.15.89:0/2932910527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48815d640 0x7fc4880d2340 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.212+0000 7fc48e400700 1 -- 172.21.15.89:0/2932910527 >> 172.21.15.89:0/2932910527 conn(0x7fc48808aff0 msgr2=0x7fc4880d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.212+0000 7fc48e400700 1 -- 172.21.15.89:0/2932910527 shutdown_connections 2024-04-02T07:52:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.212+0000 7fc48e400700 1 -- 172.21.15.89:0/2932910527 wait complete. 2024-04-02T07:52:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.214+0000 7fc48e400700 1 Processor -- start 2024-04-02T07:52:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.214+0000 7fc48e400700 1 -- start start 2024-04-02T07:52:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.215+0000 7fc48e400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4880ceec0 0x7fc4880b5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.215+0000 7fc48e400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc48815d640 0x7fc4880cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.215+0000 7fc48e400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4881619d0 0x7fc4880cd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.215+0000 7fc48e400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4880b5890 con 0x7fc4880ceec0 2024-04-02T07:52:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.215+0000 7fc48e400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48815bb70 con 0x7fc48815d640 2024-04-02T07:52:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.215+0000 7fc48e400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48814bde0 con 0x7fc4881619d0 2024-04-02T07:52:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.215+0000 7fc487400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4880ceec0 0x7fc4880b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.215+0000 7fc487400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4880ceec0 0x7fc4880b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42014/0 (socket says 172.21.15.89:42014) 2024-04-02T07:52:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.215+0000 7fc487400700 1 -- 172.21.15.89:0/2602383093 learned_addr learned my addr 172.21.15.89:0/2602383093 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.215+0000 7fc486a00700 1 --2- 172.21.15.89:0/2602383093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc48815d640 0x7fc4880cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.216+0000 7fc487e00700 1 --2- 172.21.15.89:0/2602383093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4881619d0 0x7fc4880cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.216+0000 7fc487400700 1 -- 172.21.15.89:0/2602383093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4881619d0 msgr2=0x7fc4880cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.216+0000 7fc487400700 1 --2- 172.21.15.89:0/2602383093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4881619d0 0x7fc4880cd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.216+0000 7fc487400700 1 -- 172.21.15.89:0/2602383093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc48815d640 msgr2=0x7fc4880cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:13.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.216+0000 7fc487400700 1 --2- 172.21.15.89:0/2602383093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc48815d640 0x7fc4880cd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.216+0000 7fc487400700 1 -- 172.21.15.89:0/2602383093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4780094a0 con 0x7fc4880ceec0 2024-04-02T07:52:13.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.216+0000 7fc487400700 1 --2- 172.21.15.89:0/2602383093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4880ceec0 0x7fc4880b5350 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fc478009bc0 tx=0x7fc478004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:13.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.216+0000 7fc47f400700 1 -- 172.21.15.89:0/2602383093 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47801a860 con 0x7fc4880ceec0 2024-04-02T07:52:13.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.216+0000 7fc47f400700 1 -- 172.21.15.89:0/2602383093 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47801a9c0 con 0x7fc4880ceec0 2024-04-02T07:52:13.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.216+0000 7fc47f400700 1 -- 172.21.15.89:0/2602383093 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47801ad50 con 0x7fc4880ceec0 2024-04-02T07:52:13.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.216+0000 7fc48e400700 1 -- 172.21.15.89:0/2602383093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4880cde10 con 0x7fc4880ceec0 2024-04-02T07:52:13.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.216+0000 7fc48e400700 1 -- 172.21.15.89:0/2602383093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc48814b790 con 0x7fc4880ceec0 2024-04-02T07:52:13.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.217+0000 7fc47cc00700 1 -- 172.21.15.89:0/2602383093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45c0052f0 con 0x7fc4880ceec0 2024-04-02T07:52:13.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.218+0000 7fc47f400700 1 -- 172.21.15.89:0/2602383093 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc478022070 con 0x7fc4880ceec0 2024-04-02T07:52:13.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.220+0000 7fc47f400700 1 --2- 172.21.15.89:0/2602383093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc46806adb0 0x7fc46806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:13.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.220+0000 7fc47f400700 1 -- 172.21.15.89:0/2602383093 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc47808a7a0 con 0x7fc4880ceec0 2024-04-02T07:52:13.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.220+0000 7fc486a00700 1 --2- 172.21.15.89:0/2602383093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc46806adb0 0x7fc46806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:13.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.220+0000 7fc486a00700 1 --2- 172.21.15.89:0/2602383093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc46806adb0 0x7fc46806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc48815c1d0 tx=0x7fc474009040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:13.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.224+0000 7fc47f400700 1 -- 172.21.15.89:0/2602383093 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc478055820 con 0x7fc4880ceec0 2024-04-02T07:52:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.421+0000 7fc47cc00700 1 -- 172.21.15.89:0/2602383093 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc45c000bc0 con 0x7fc46806adb0 2024-04-02T07:52:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.432+0000 7fc47f400700 1 -- 172.21.15.89:0/2602383093 <== mgr.6696 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289256 (secure 0 0 0) 0x7fc45c000bc0 con 0x7fc46806adb0 2024-04-02T07:52:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.436+0000 7fc47cc00700 1 -- 172.21.15.89:0/2602383093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc46806adb0 msgr2=0x7fc46806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.436+0000 7fc47cc00700 1 --2- 172.21.15.89:0/2602383093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc46806adb0 0x7fc46806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc48815c1d0 tx=0x7fc474009040 comp rx=0 tx=0).stop 2024-04-02T07:52:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.436+0000 7fc47cc00700 1 -- 172.21.15.89:0/2602383093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4880ceec0 msgr2=0x7fc4880b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.436+0000 7fc47cc00700 1 --2- 172.21.15.89:0/2602383093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4880ceec0 0x7fc4880b5350 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fc478009bc0 tx=0x7fc478004750 comp rx=0 tx=0).stop 2024-04-02T07:52:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.436+0000 7fc47cc00700 1 -- 172.21.15.89:0/2602383093 shutdown_connections 2024-04-02T07:52:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.436+0000 7fc47cc00700 1 --2- 172.21.15.89:0/2602383093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc46806adb0 0x7fc46806d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.436+0000 7fc47cc00700 1 --2- 172.21.15.89:0/2602383093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4881619d0 0x7fc4880cd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.436+0000 7fc47cc00700 1 --2- 172.21.15.89:0/2602383093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4880ceec0 0x7fc4880b5350 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.436+0000 7fc47cc00700 1 --2- 172.21.15.89:0/2602383093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc48815d640 0x7fc4880cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.436+0000 7fc47cc00700 1 -- 172.21.15.89:0/2602383093 >> 172.21.15.89:0/2602383093 conn(0x7fc48808aff0 msgr2=0x7fc48806c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.436+0000 7fc47cc00700 1 -- 172.21.15.89:0/2602383093 shutdown_connections 2024-04-02T07:52:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.436+0000 7fc47cc00700 1 -- 172.21.15.89:0/2602383093 wait complete. 2024-04-02T07:52:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:52:13.456 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T07:52:12.808643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":303,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":169,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":302024,"kb_used_data":10428,"kb_used_omap":12,"kb_used_meta":291443,"kb_avail":754672696,"statfs":{"total":773094113280,"available":772784840704,"internally_reserved":0,"allocated":10678272,"data_stored":5778350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":298438200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.242421"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174682+0000","last_change":"2024-04-02T07:52:09.100629+0000","last_active":"2024-04-02T07:52:10.174682+0000","last_peered":"2024-04-02T07:52:10.174682+0000","last_clean":"2024-04-02T07:52:10.174682+0000","last_became_active":"2024-04-02T07:52:09.100358+0000","last_became_peered":"2024-04-02T07:52:09.100358+0000","last_unstale":"2024-04-02T07:52:10.174682+0000","last_undegraded":"2024-04-02T07:52:10.174682+0000","last_fullsized":"2024-04-02T07:52:10.174682+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:35.764053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081216+0000","last_change":"2024-04-02T07:52:09.104514+0000","last_active":"2024-04-02T07:52:10.081216+0000","last_peered":"2024-04-02T07:52:10.081216+0000","last_clean":"2024-04-02T07:52:10.081216+0000","last_became_active":"2024-04-02T07:52:09.104272+0000","last_became_peered":"2024-04-02T07:52:09.104272+0000","last_unstale":"2024-04-02T07:52:10.081216+0000","last_undegraded":"2024-04-02T07:52:10.081216+0000","last_fullsized":"2024-04-02T07:52:10.081216+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:33:50.420553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371303+0000","last_change":"2024-04-02T07:52:09.099462+0000","last_active":"2024-04-02T07:52:10.371303+0000","last_peered":"2024-04-02T07:52:10.371303+0000","last_clean":"2024-04-02T07:52:10.371303+0000","last_became_active":"2024-04-02T07:52:09.098893+0000","last_became_peered":"2024-04-02T07:52:09.098893+0000","last_unstale":"2024-04-02T07:52:10.371303+0000","last_undegraded":"2024-04-02T07:52:10.371303+0000","last_fullsized":"2024-04-02T07:52:10.371303+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:31:29.266098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371390+0000","last_change":"2024-04-02T07:52:09.099355+0000","last_active":"2024-04-02T07:52:10.371390+0000","last_peered":"2024-04-02T07:52:10.371390+0000","last_clean":"2024-04-02T07:52:10.371390+0000","last_became_active":"2024-04-02T07:52:09.099010+0000","last_became_peered":"2024-04-02T07:52:09.099010+0000","last_unstale":"2024-04-02T07:52:10.371390+0000","last_undegraded":"2024-04-02T07:52:10.371390+0000","last_fullsized":"2024-04-02T07:52:10.371390+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:29:10.607669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371534+0000","last_change":"2024-04-02T07:52:09.085483+0000","last_active":"2024-04-02T07:52:10.371534+0000","last_peered":"2024-04-02T07:52:10.371534+0000","last_clean":"2024-04-02T07:52:10.371534+0000","last_became_active":"2024-04-02T07:52:09.085203+0000","last_became_peered":"2024-04-02T07:52:09.085203+0000","last_unstale":"2024-04-02T07:52:10.371534+0000","last_undegraded":"2024-04-02T07:52:10.371534+0000","last_fullsized":"2024-04-02T07:52:10.371534+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:17.505215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208121+0000","last_change":"2024-04-02T07:52:09.086817+0000","last_active":"2024-04-02T07:52:10.208121+0000","last_peered":"2024-04-02T07:52:10.208121+0000","last_clean":"2024-04-02T07:52:10.208121+0000","last_became_active":"2024-04-02T07:52:09.085142+0000","last_became_peered":"2024-04-02T07:52:09.085142+0000","last_unstale":"2024-04-02T07:52:10.208121+0000","last_undegraded":"2024-04-02T07:52:10.208121+0000","last_fullsized":"2024-04-02T07:52:10.208121+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:31:30.741833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085387+0000","last_change":"2024-04-02T07:52:09.101035+0000","last_active":"2024-04-02T07:52:10.085387+0000","last_peered":"2024-04-02T07:52:10.085387+0000","last_clean":"2024-04-02T07:52:10.085387+0000","last_became_active":"2024-04-02T07:52:09.100703+0000","last_became_peered":"2024-04-02T07:52:09.100703+0000","last_unstale":"2024-04-02T07:52:10.085387+0000","last_undegraded":"2024-04-02T07:52:10.085387+0000","last_fullsized":"2024-04-02T07:52:10.085387+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:10.755054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174200+0000","last_change":"2024-04-02T07:52:09.084193+0000","last_active":"2024-04-02T07:52:10.174200+0000","last_peered":"2024-04-02T07:52:10.174200+0000","last_clean":"2024-04-02T07:52:10.174200+0000","last_became_active":"2024-04-02T07:52:09.083959+0000","last_became_peered":"2024-04-02T07:52:09.083959+0000","last_unstale":"2024-04-02T07:52:10.174200+0000","last_undegraded":"2024-04-02T07:52:10.174200+0000","last_fullsized":"2024-04-02T07:52:10.174200+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:20:51.784990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371920+0000","last_change":"2024-04-02T07:52:09.089606+0000","last_active":"2024-04-02T07:52:10.371920+0000","last_peered":"2024-04-02T07:52:10.371920+0000","last_clean":"2024-04-02T07:52:10.371920+0000","last_became_active":"2024-04-02T07:52:09.089322+0000","last_became_peered":"2024-04-02T07:52:09.089322+0000","last_unstale":"2024-04-02T07:52:10.371920+0000","last_undegraded":"2024-04-02T07:52:10.371920+0000","last_fullsized":"2024-04-02T07:52:10.371920+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:15.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085757+0000","last_change":"2024-04-02T07:52:09.099115+0000","last_active":"2024-04-02T07:52:10.085757+0000","last_peered":"2024-04-02T07:52:10.085757+0000","last_clean":"2024-04-02T07:52:10.085757+0000","last_became_active":"2024-04-02T07:52:09.098804+0000","last_became_peered":"2024-04-02T07:52:09.098804+0000","last_unstale":"2024-04-02T07:52:10.085757+0000","last_undegraded":"2024-04-02T07:52:10.085757+0000","last_fullsized":"2024-04-02T07:52:10.085757+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:29:10.607669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174702+0000","last_change":"2024-04-02T07:52:09.103096+0000","last_active":"2024-04-02T07:52:10.174702+0000","last_peered":"2024-04-02T07:52:10.174702+0000","last_clean":"2024-04-02T07:52:10.174702+0000","last_became_active":"2024-04-02T07:52:09.102796+0000","last_became_peered":"2024-04-02T07:52:09.102796+0000","last_unstale":"2024-04-02T07:52:10.174702+0000","last_undegraded":"2024-04-02T07:52:10.174702+0000","last_fullsized":"2024-04-02T07:52:10.174702+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:58:29.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085926+0000","last_change":"2024-04-02T07:52:09.091339+0000","last_active":"2024-04-02T07:52:10.085926+0000","last_peered":"2024-04-02T07:52:10.085926+0000","last_clean":"2024-04-02T07:52:10.085926+0000","last_became_active":"2024-04-02T07:52:09.090479+0000","last_became_peered":"2024-04-02T07:52:09.090479+0000","last_unstale":"2024-04-02T07:52:10.085926+0000","last_undegraded":"2024-04-02T07:52:10.085926+0000","last_fullsized":"2024-04-02T07:52:10.085926+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:41:56.288382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371443+0000","last_change":"2024-04-02T07:52:09.099793+0000","last_active":"2024-04-02T07:52:10.371443+0000","last_peered":"2024-04-02T07:52:10.371443+0000","last_clean":"2024-04-02T07:52:10.371443+0000","last_became_active":"2024-04-02T07:52:09.099144+0000","last_became_peered":"2024-04-02T07:52:09.099144+0000","last_unstale":"2024-04-02T07:52:10.371443+0000","last_undegraded":"2024-04-02T07:52:10.371443+0000","last_fullsized":"2024-04-02T07:52:10.371443+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:37:15.846519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085933+0000","last_change":"2024-04-02T07:52:09.095802+0000","last_active":"2024-04-02T07:52:10.085933+0000","last_peered":"2024-04-02T07:52:10.085933+0000","last_clean":"2024-04-02T07:52:10.085933+0000","last_became_active":"2024-04-02T07:52:09.095527+0000","last_became_peered":"2024-04-02T07:52:09.095527+0000","last_unstale":"2024-04-02T07:52:10.085933+0000","last_undegraded":"2024-04-02T07:52:10.085933+0000","last_fullsized":"2024-04-02T07:52:10.085933+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:05.554358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085705+0000","last_change":"2024-04-02T07:52:09.081789+0000","last_active":"2024-04-02T07:52:10.085705+0000","last_peered":"2024-04-02T07:52:10.085705+0000","last_clean":"2024-04-02T07:52:10.085705+0000","last_became_active":"2024-04-02T07:52:09.081507+0000","last_became_peered":"2024-04-02T07:52:09.081507+0000","last_unstale":"2024-04-02T07:52:10.085705+0000","last_undegraded":"2024-04-02T07:52:10.085705+0000","last_fullsized":"2024-04-02T07:52:10.085705+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:58:55.222923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174198+0000","last_change":"2024-04-02T07:52:09.097717+0000","last_active":"2024-04-02T07:52:10.174198+0000","last_peered":"2024-04-02T07:52:10.174198+0000","last_clean":"2024-04-02T07:52:10.174198+0000","last_became_active":"2024-04-02T07:52:09.088955+0000","last_became_peered":"2024-04-02T07:52:09.088955+0000","last_unstale":"2024-04-02T07:52:10.174198+0000","last_undegraded":"2024-04-02T07:52:10.174198+0000","last_fullsized":"2024-04-02T07:52:10.174198+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:54:54.390550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371874+0000","last_change":"2024-04-02T07:52:09.088181+0000","last_active":"2024-04-02T07:52:10.371874+0000","last_peered":"2024-04-02T07:52:10.371874+0000","last_clean":"2024-04-02T07:52:10.371874+0000","last_became_active":"2024-04-02T07:52:09.087890+0000","last_became_peered":"2024-04-02T07:52:09.087890+0000","last_unstale":"2024-04-02T07:52:10.371874+0000","last_undegraded":"2024-04-02T07:52:10.371874+0000","last_fullsized":"2024-04-02T07:52:10.371874+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:20.566311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.372004+0000","last_change":"2024-04-02T07:52:09.104157+0000","last_active":"2024-04-02T07:52:10.372004+0000","last_peered":"2024-04-02T07:52:10.372004+0000","last_clean":"2024-04-02T07:52:10.372004+0000","last_became_active":"2024-04-02T07:52:09.103892+0000","last_became_peered":"2024-04-02T07:52:09.103892+0000","last_unstale":"2024-04-02T07:52:10.372004+0000","last_undegraded":"2024-04-02T07:52:10.372004+0000","last_fullsized":"2024-04-02T07:52:10.372004+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:32.841698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174704+0000","last_change":"2024-04-02T07:52:09.098739+0000","last_active":"2024-04-02T07:52:10.174704+0000","last_peered":"2024-04-02T07:52:10.174704+0000","last_clean":"2024-04-02T07:52:10.174704+0000","last_became_active":"2024-04-02T07:52:09.098482+0000","last_became_peered":"2024-04-02T07:52:09.098482+0000","last_unstale":"2024-04-02T07:52:10.174704+0000","last_undegraded":"2024-04-02T07:52:10.174704+0000","last_fullsized":"2024-04-02T07:52:10.174704+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:20.566311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371689+0000","last_change":"2024-04-02T07:52:09.085517+0000","last_active":"2024-04-02T07:52:10.371689+0000","last_peered":"2024-04-02T07:52:10.371689+0000","last_clean":"2024-04-02T07:52:10.371689+0000","last_became_active":"2024-04-02T07:52:09.084507+0000","last_became_peered":"2024-04-02T07:52:09.084507+0000","last_unstale":"2024-04-02T07:52:10.371689+0000","last_undegraded":"2024-04-02T07:52:10.371689+0000","last_fullsized":"2024-04-02T07:52:10.371689+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:21:34.806245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081018+0000","last_change":"2024-04-02T07:52:09.090136+0000","last_active":"2024-04-02T07:52:10.081018+0000","last_peered":"2024-04-02T07:52:10.081018+0000","last_clean":"2024-04-02T07:52:10.081018+0000","last_became_active":"2024-04-02T07:52:09.089666+0000","last_became_peered":"2024-04-02T07:52:09.089666+0000","last_unstale":"2024-04-02T07:52:10.081018+0000","last_undegraded":"2024-04-02T07:52:10.081018+0000","last_fullsized":"2024-04-02T07:52:10.081018+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:40.830548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208019+0000","last_change":"2024-04-02T07:52:09.080447+0000","last_active":"2024-04-02T07:52:10.208019+0000","last_peered":"2024-04-02T07:52:10.208019+0000","last_clean":"2024-04-02T07:52:10.208019+0000","last_became_active":"2024-04-02T07:52:09.080193+0000","last_became_peered":"2024-04-02T07:52:09.080193+0000","last_unstale":"2024-04-02T07:52:10.208019+0000","last_undegraded":"2024-04-02T07:52:10.208019+0000","last_fullsized":"2024-04-02T07:52:10.208019+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:00.296862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085378+0000","last_change":"2024-04-02T07:52:09.097706+0000","last_active":"2024-04-02T07:52:10.085378+0000","last_peered":"2024-04-02T07:52:10.085378+0000","last_clean":"2024-04-02T07:52:10.085378+0000","last_became_active":"2024-04-02T07:52:09.097376+0000","last_became_peered":"2024-04-02T07:52:09.097376+0000","last_unstale":"2024-04-02T07:52:10.085378+0000","last_undegraded":"2024-04-02T07:52:10.085378+0000","last_fullsized":"2024-04-02T07:52:10.085378+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:23:52.181861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174319+0000","last_change":"2024-04-02T07:52:09.083281+0000","last_active":"2024-04-02T07:52:10.174319+0000","last_peered":"2024-04-02T07:52:10.174319+0000","last_clean":"2024-04-02T07:52:10.174319+0000","last_became_active":"2024-04-02T07:52:09.083001+0000","last_became_peered":"2024-04-02T07:52:09.083001+0000","last_unstale":"2024-04-02T07:52:10.174319+0000","last_undegraded":"2024-04-02T07:52:10.174319+0000","last_fullsized":"2024-04-02T07:52:10.174319+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:21:45.009281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085623+0000","last_change":"2024-04-02T07:52:09.096280+0000","last_active":"2024-04-02T07:52:10.085623+0000","last_peered":"2024-04-02T07:52:10.085623+0000","last_clean":"2024-04-02T07:52:10.085623+0000","last_became_active":"2024-04-02T07:52:09.095975+0000","last_became_peered":"2024-04-02T07:52:09.095975+0000","last_unstale":"2024-04-02T07:52:10.085623+0000","last_undegraded":"2024-04-02T07:52:10.085623+0000","last_fullsized":"2024-04-02T07:52:10.085623+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:35:41.761205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174484+0000","last_change":"2024-04-02T07:52:09.107913+0000","last_active":"2024-04-02T07:52:10.174484+0000","last_peered":"2024-04-02T07:52:10.174484+0000","last_clean":"2024-04-02T07:52:10.174484+0000","last_became_active":"2024-04-02T07:52:09.107639+0000","last_became_peered":"2024-04-02T07:52:09.107639+0000","last_unstale":"2024-04-02T07:52:10.174484+0000","last_undegraded":"2024-04-02T07:52:10.174484+0000","last_fullsized":"2024-04-02T07:52:10.174484+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:39:13.805896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371738+0000","last_change":"2024-04-02T07:52:09.096147+0000","last_active":"2024-04-02T07:52:10.371738+0000","last_peered":"2024-04-02T07:52:10.371738+0000","last_clean":"2024-04-02T07:52:10.371738+0000","last_became_active":"2024-04-02T07:52:09.093917+0000","last_became_peered":"2024-04-02T07:52:09.093917+0000","last_unstale":"2024-04-02T07:52:10.371738+0000","last_undegraded":"2024-04-02T07:52:10.371738+0000","last_fullsized":"2024-04-02T07:52:10.371738+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:08:37.815656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.080801+0000","last_change":"2024-04-02T07:52:09.097264+0000","last_active":"2024-04-02T07:52:10.080801+0000","last_peered":"2024-04-02T07:52:10.080801+0000","last_clean":"2024-04-02T07:52:10.080801+0000","last_became_active":"2024-04-02T07:52:09.096914+0000","last_became_peered":"2024-04-02T07:52:09.096914+0000","last_unstale":"2024-04-02T07:52:10.080801+0000","last_undegraded":"2024-04-02T07:52:10.080801+0000","last_fullsized":"2024-04-02T07:52:10.080801+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:48:43.137803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371550+0000","last_change":"2024-04-02T07:52:09.104361+0000","last_active":"2024-04-02T07:52:10.371550+0000","last_peered":"2024-04-02T07:52:10.371550+0000","last_clean":"2024-04-02T07:52:10.371550+0000","last_became_active":"2024-04-02T07:52:09.104057+0000","last_became_peered":"2024-04-02T07:52:09.104057+0000","last_unstale":"2024-04-02T07:52:10.371550+0000","last_undegraded":"2024-04-02T07:52:10.371550+0000","last_fullsized":"2024-04-02T07:52:10.371550+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:13.811440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174413+0000","last_change":"2024-04-02T07:52:09.100074+0000","last_active":"2024-04-02T07:52:10.174413+0000","last_peered":"2024-04-02T07:52:10.174413+0000","last_clean":"2024-04-02T07:52:10.174413+0000","last_became_active":"2024-04-02T07:52:09.099847+0000","last_became_peered":"2024-04-02T07:52:09.099847+0000","last_unstale":"2024-04-02T07:52:10.174413+0000","last_undegraded":"2024-04-02T07:52:10.174413+0000","last_fullsized":"2024-04-02T07:52:10.174413+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:43.287818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081062+0000","last_change":"2024-04-02T07:52:09.094190+0000","last_active":"2024-04-02T07:52:10.081062+0000","last_peered":"2024-04-02T07:52:10.081062+0000","last_clean":"2024-04-02T07:52:10.081062+0000","last_became_active":"2024-04-02T07:52:09.093702+0000","last_became_peered":"2024-04-02T07:52:09.093702+0000","last_unstale":"2024-04-02T07:52:10.081062+0000","last_undegraded":"2024-04-02T07:52:10.081062+0000","last_fullsized":"2024-04-02T07:52:10.081062+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:11:41.093903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.080811+0000","last_change":"2024-04-02T07:52:09.096817+0000","last_active":"2024-04-02T07:52:10.080811+0000","last_peered":"2024-04-02T07:52:10.080811+0000","last_clean":"2024-04-02T07:52:10.080811+0000","last_became_active":"2024-04-02T07:52:09.094257+0000","last_became_peered":"2024-04-02T07:52:09.094257+0000","last_unstale":"2024-04-02T07:52:10.080811+0000","last_undegraded":"2024-04-02T07:52:10.080811+0000","last_fullsized":"2024-04-02T07:52:10.080811+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:06:33.391572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174542+0000","last_change":"2024-04-02T07:52:09.099100+0000","last_active":"2024-04-02T07:52:10.174542+0000","last_peered":"2024-04-02T07:52:10.174542+0000","last_clean":"2024-04-02T07:52:10.174542+0000","last_became_active":"2024-04-02T07:52:09.098341+0000","last_became_peered":"2024-04-02T07:52:09.098341+0000","last_unstale":"2024-04-02T07:52:10.174542+0000","last_undegraded":"2024-04-02T07:52:10.174542+0000","last_fullsized":"2024-04-02T07:52:10.174542+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:08:37.815656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"96'4","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208004+0000","last_change":"2024-04-02T07:52:08.581911+0000","last_active":"2024-04-02T07:52:10.208004+0000","last_peered":"2024-04-02T07:52:10.208004+0000","last_clean":"2024-04-02T07:52:10.208004+0000","last_became_active":"2024-04-02T07:52:08.580945+0000","last_became_peered":"2024-04-02T07:52:08.580945+0000","last_unstale":"2024-04-02T07:52:10.208004+0000","last_undegraded":"2024-04-02T07:52:10.208004+0000","last_fullsized":"2024-04-02T07:52:10.208004+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:45:13.517646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.372012+0000","last_change":"2024-04-02T07:52:09.095472+0000","last_active":"2024-04-02T07:52:10.372012+0000","last_peered":"2024-04-02T07:52:10.372012+0000","last_clean":"2024-04-02T07:52:10.372012+0000","last_became_active":"2024-04-02T07:52:09.093337+0000","last_became_peered":"2024-04-02T07:52:09.093337+0000","last_unstale":"2024-04-02T07:52:10.372012+0000","last_undegraded":"2024-04-02T07:52:10.372012+0000","last_fullsized":"2024-04-02T07:52:10.372012+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:45:14.526108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1e","version":"96'1","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.372061+0000","last_change":"2024-04-02T07:52:08.590194+0000","last_active":"2024-04-02T07:52:10.372061+0000","last_peered":"2024-04-02T07:52:10.372061+0000","last_clean":"2024-04-02T07:52:10.372061+0000","last_became_active":"2024-04-02T07:52:08.588845+0000","last_became_peered":"2024-04-02T07:52:08.588845+0000","last_unstale":"2024-04-02T07:52:10.372061+0000","last_undegraded":"2024-04-02T07:52:10.372061+0000","last_fullsized":"2024-04-02T07:52:10.372061+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:32:56.123575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.207985+0000","last_change":"2024-04-02T07:52:09.107829+0000","last_active":"2024-04-02T07:52:10.207985+0000","last_peered":"2024-04-02T07:52:10.207985+0000","last_clean":"2024-04-02T07:52:10.207985+0000","last_became_active":"2024-04-02T07:52:09.107559+0000","last_became_peered":"2024-04-02T07:52:09.107559+0000","last_unstale":"2024-04-02T07:52:10.207985+0000","last_undegraded":"2024-04-02T07:52:10.207985+0000","last_fullsized":"2024-04-02T07:52:10.207985+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:10.755054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"96'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174757+0000","last_change":"2024-04-02T07:52:08.590593+0000","last_active":"2024-04-02T07:52:10.174757+0000","last_peered":"2024-04-02T07:52:10.174757+0000","last_clean":"2024-04-02T07:52:10.174757+0000","last_became_active":"2024-04-02T07:52:08.590348+0000","last_became_peered":"2024-04-02T07:52:08.590348+0000","last_unstale":"2024-04-02T07:52:10.174757+0000","last_undegraded":"2024-04-02T07:52:10.174757+0000","last_fullsized":"2024-04-02T07:52:10.174757+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:13:26.420771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085586+0000","last_change":"2024-04-02T07:52:09.102786+0000","last_active":"2024-04-02T07:52:10.085586+0000","last_peered":"2024-04-02T07:52:10.085586+0000","last_clean":"2024-04-02T07:52:10.085586+0000","last_became_active":"2024-04-02T07:52:09.102509+0000","last_became_peered":"2024-04-02T07:52:09.102509+0000","last_unstale":"2024-04-02T07:52:10.085586+0000","last_undegraded":"2024-04-02T07:52:10.085586+0000","last_fullsized":"2024-04-02T07:52:10.085586+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:20.302770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1c","version":"96'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208055+0000","last_change":"2024-04-02T07:52:08.589183+0000","last_active":"2024-04-02T07:52:10.208055+0000","last_peered":"2024-04-02T07:52:10.208055+0000","last_clean":"2024-04-02T07:52:10.208055+0000","last_became_active":"2024-04-02T07:52:08.588879+0000","last_became_peered":"2024-04-02T07:52:08.588879+0000","last_unstale":"2024-04-02T07:52:10.208055+0000","last_undegraded":"2024-04-02T07:52:10.208055+0000","last_fullsized":"2024-04-02T07:52:10.208055+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:32:37.852523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174318+0000","last_change":"2024-04-02T07:52:09.090582+0000","last_active":"2024-04-02T07:52:10.174318+0000","last_peered":"2024-04-02T07:52:10.174318+0000","last_clean":"2024-04-02T07:52:10.174318+0000","last_became_active":"2024-04-02T07:52:09.090330+0000","last_became_peered":"2024-04-02T07:52:09.090330+0000","last_unstale":"2024-04-02T07:52:10.174318+0000","last_undegraded":"2024-04-02T07:52:10.174318+0000","last_fullsized":"2024-04-02T07:52:10.174318+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:33.887916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"96'1","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.086029+0000","last_change":"2024-04-02T07:52:08.579847+0000","last_active":"2024-04-02T07:52:10.086029+0000","last_peered":"2024-04-02T07:52:10.086029+0000","last_clean":"2024-04-02T07:52:10.086029+0000","last_became_active":"2024-04-02T07:52:08.579586+0000","last_became_peered":"2024-04-02T07:52:08.579586+0000","last_unstale":"2024-04-02T07:52:10.086029+0000","last_undegraded":"2024-04-02T07:52:10.086029+0000","last_fullsized":"2024-04-02T07:52:10.086029+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:32:11.605975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208073+0000","last_change":"2024-04-02T07:52:09.100358+0000","last_active":"2024-04-02T07:52:10.208073+0000","last_peered":"2024-04-02T07:52:10.208073+0000","last_clean":"2024-04-02T07:52:10.208073+0000","last_became_active":"2024-04-02T07:52:09.099806+0000","last_became_peered":"2024-04-02T07:52:09.099806+0000","last_unstale":"2024-04-02T07:52:10.208073+0000","last_undegraded":"2024-04-02T07:52:10.208073+0000","last_fullsized":"2024-04-02T07:52:10.208073+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:35:41.761205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1a","version":"96'1","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085713+0000","last_change":"2024-04-02T07:52:08.591780+0000","last_active":"2024-04-02T07:52:10.085713+0000","last_peered":"2024-04-02T07:52:10.085713+0000","last_clean":"2024-04-02T07:52:10.085713+0000","last_became_active":"2024-04-02T07:52:08.588742+0000","last_became_peered":"2024-04-02T07:52:08.588742+0000","last_unstale":"2024-04-02T07:52:10.085713+0000","last_undegraded":"2024-04-02T07:52:10.085713+0000","last_fullsized":"2024-04-02T07:52:10.085713+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:21.289613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.370700+0000","last_change":"2024-04-02T07:52:09.097716+0000","last_active":"2024-04-02T07:52:10.370700+0000","last_peered":"2024-04-02T07:52:10.370700+0000","last_clean":"2024-04-02T07:52:10.370700+0000","last_became_active":"2024-04-02T07:52:09.097466+0000","last_became_peered":"2024-04-02T07:52:09.097466+0000","last_unstale":"2024-04-02T07:52:10.370700+0000","last_undegraded":"2024-04-02T07:52:10.370700+0000","last_fullsized":"2024-04-02T07:52:10.370700+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:45:14.526108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371087+0000","last_change":"2024-04-02T07:52:08.595222+0000","last_active":"2024-04-02T07:52:10.371087+0000","last_peered":"2024-04-02T07:52:10.371087+0000","last_clean":"2024-04-02T07:52:10.371087+0000","last_became_active":"2024-04-02T07:52:08.594888+0000","last_became_peered":"2024-04-02T07:52:08.594888+0000","last_unstale":"2024-04-02T07:52:10.371087+0000","last_undegraded":"2024-04-02T07:52:10.371087+0000","last_fullsized":"2024-04-02T07:52:10.371087+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:47:14.750072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085515+0000","last_change":"2024-04-02T07:52:09.104975+0000","last_active":"2024-04-02T07:52:10.085515+0000","last_peered":"2024-04-02T07:52:10.085515+0000","last_clean":"2024-04-02T07:52:10.085515+0000","last_became_active":"2024-04-02T07:52:09.104691+0000","last_became_peered":"2024-04-02T07:52:09.104691+0000","last_unstale":"2024-04-02T07:52:10.085515+0000","last_undegraded":"2024-04-02T07:52:10.085515+0000","last_fullsized":"2024-04-02T07:52:10.085515+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:10:42.388029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085449+0000","last_change":"2024-04-02T07:52:08.579443+0000","last_active":"2024-04-02T07:52:10.085449+0000","last_peered":"2024-04-02T07:52:10.085449+0000","last_clean":"2024-04-02T07:52:10.085449+0000","last_became_active":"2024-04-02T07:52:08.576729+0000","last_became_peered":"2024-04-02T07:52:08.576729+0000","last_unstale":"2024-04-02T07:52:10.085449+0000","last_undegraded":"2024-04-02T07:52:10.085449+0000","last_fullsized":"2024-04-02T07:52:10.085449+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:29:09.599207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085851+0000","last_change":"2024-04-02T07:52:09.103166+0000","last_active":"2024-04-02T07:52:10.085851+0000","last_peered":"2024-04-02T07:52:10.085851+0000","last_clean":"2024-04-02T07:52:10.085851+0000","last_became_active":"2024-04-02T07:52:09.101914+0000","last_became_peered":"2024-04-02T07:52:09.101914+0000","last_unstale":"2024-04-02T07:52:10.085851+0000","last_undegraded":"2024-04-02T07:52:10.085851+0000","last_fullsized":"2024-04-02T07:52:10.085851+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:31:30.741833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174662+0000","last_change":"2024-04-02T07:52:08.587968+0000","last_active":"2024-04-02T07:52:10.174662+0000","last_peered":"2024-04-02T07:52:10.174662+0000","last_clean":"2024-04-02T07:52:10.174662+0000","last_became_active":"2024-04-02T07:52:08.587729+0000","last_became_peered":"2024-04-02T07:52:08.587729+0000","last_unstale":"2024-04-02T07:52:10.174662+0000","last_undegraded":"2024-04-02T07:52:10.174662+0000","last_fullsized":"2024-04-02T07:52:10.174662+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:32:56.123575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081104+0000","last_change":"2024-04-02T07:52:09.095465+0000","last_active":"2024-04-02T07:52:10.081104+0000","last_peered":"2024-04-02T07:52:10.081104+0000","last_clean":"2024-04-02T07:52:10.081104+0000","last_became_active":"2024-04-02T07:52:09.095217+0000","last_became_peered":"2024-04-02T07:52:09.095217+0000","last_unstale":"2024-04-02T07:52:10.081104+0000","last_undegraded":"2024-04-02T07:52:10.081104+0000","last_fullsized":"2024-04-02T07:52:10.081104+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:30:44.307779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081213+0000","last_change":"2024-04-02T07:52:08.587695+0000","last_active":"2024-04-02T07:52:10.081213+0000","last_peered":"2024-04-02T07:52:10.081213+0000","last_clean":"2024-04-02T07:52:10.081213+0000","last_became_active":"2024-04-02T07:52:08.587413+0000","last_became_peered":"2024-04-02T07:52:08.587413+0000","last_unstale":"2024-04-02T07:52:10.081213+0000","last_undegraded":"2024-04-02T07:52:10.081213+0000","last_fullsized":"2024-04-02T07:52:10.081213+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:08:46.492307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174470+0000","last_change":"2024-04-02T07:52:09.108705+0000","last_active":"2024-04-02T07:52:10.174470+0000","last_peered":"2024-04-02T07:52:10.174470+0000","last_clean":"2024-04-02T07:52:10.174470+0000","last_became_active":"2024-04-02T07:52:09.107625+0000","last_became_peered":"2024-04-02T07:52:09.107625+0000","last_unstale":"2024-04-02T07:52:10.174470+0000","last_undegraded":"2024-04-02T07:52:10.174470+0000","last_fullsized":"2024-04-02T07:52:10.174470+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:17.505215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"96'3","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085827+0000","last_change":"2024-04-02T07:52:08.587926+0000","last_active":"2024-04-02T07:52:10.085827+0000","last_peered":"2024-04-02T07:52:10.085827+0000","last_clean":"2024-04-02T07:52:10.085827+0000","last_became_active":"2024-04-02T07:52:08.586936+0000","last_became_peered":"2024-04-02T07:52:08.586936+0000","last_unstale":"2024-04-02T07:52:10.085827+0000","last_undegraded":"2024-04-02T07:52:10.085827+0000","last_fullsized":"2024-04-02T07:52:10.085827+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:23:51.173399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371903+0000","last_change":"2024-04-02T07:52:09.096032+0000","last_active":"2024-04-02T07:52:10.371903+0000","last_peered":"2024-04-02T07:52:10.371903+0000","last_clean":"2024-04-02T07:52:10.371903+0000","last_became_active":"2024-04-02T07:52:09.095334+0000","last_became_peered":"2024-04-02T07:52:09.095334+0000","last_unstale":"2024-04-02T07:52:10.371903+0000","last_undegraded":"2024-04-02T07:52:10.371903+0000","last_fullsized":"2024-04-02T07:52:10.371903+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:43.287818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"96'6","reported_seq":17,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371621+0000","last_change":"2024-04-02T07:52:08.588906+0000","last_active":"2024-04-02T07:52:10.371621+0000","last_peered":"2024-04-02T07:52:10.371621+0000","last_clean":"2024-04-02T07:52:10.371621+0000","last_became_active":"2024-04-02T07:52:08.588557+0000","last_became_peered":"2024-04-02T07:52:08.588557+0000","last_unstale":"2024-04-02T07:52:10.371621+0000","last_undegraded":"2024-04-02T07:52:10.371621+0000","last_fullsized":"2024-04-02T07:52:10.371621+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:21.289613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208028+0000","last_change":"2024-04-02T07:52:09.083316+0000","last_active":"2024-04-02T07:52:10.208028+0000","last_peered":"2024-04-02T07:52:10.208028+0000","last_clean":"2024-04-02T07:52:10.208028+0000","last_became_active":"2024-04-02T07:52:09.083063+0000","last_became_peered":"2024-04-02T07:52:09.083063+0000","last_unstale":"2024-04-02T07:52:10.208028+0000","last_undegraded":"2024-04-02T07:52:10.208028+0000","last_fullsized":"2024-04-02T07:52:10.208028+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:20.302770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"96'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371681+0000","last_change":"2024-04-02T07:52:08.592021+0000","last_active":"2024-04-02T07:52:10.371681+0000","last_peered":"2024-04-02T07:52:10.371681+0000","last_clean":"2024-04-02T07:52:10.371681+0000","last_became_active":"2024-04-02T07:52:08.591399+0000","last_became_peered":"2024-04-02T07:52:08.591399+0000","last_unstale":"2024-04-02T07:52:10.371681+0000","last_undegraded":"2024-04-02T07:52:10.371681+0000","last_fullsized":"2024-04-02T07:52:10.371681+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:51.201737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085473+0000","last_change":"2024-04-02T07:52:09.108800+0000","last_active":"2024-04-02T07:52:10.085473+0000","last_peered":"2024-04-02T07:52:10.085473+0000","last_clean":"2024-04-02T07:52:10.085473+0000","last_became_active":"2024-04-02T07:52:09.108548+0000","last_became_peered":"2024-04-02T07:52:09.108548+0000","last_unstale":"2024-04-02T07:52:10.085473+0000","last_undegraded":"2024-04-02T07:52:10.085473+0000","last_fullsized":"2024-04-02T07:52:10.085473+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:31:29.266098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085478+0000","last_change":"2024-04-02T07:52:08.592667+0000","last_active":"2024-04-02T07:52:10.085478+0000","last_peered":"2024-04-02T07:52:10.085478+0000","last_clean":"2024-04-02T07:52:10.085478+0000","last_became_active":"2024-04-02T07:52:08.592373+0000","last_became_peered":"2024-04-02T07:52:08.592373+0000","last_unstale":"2024-04-02T07:52:10.085478+0000","last_undegraded":"2024-04-02T07:52:10.085478+0000","last_fullsized":"2024-04-02T07:52:10.085478+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:27.543483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.370821+0000","last_change":"2024-04-02T07:52:09.081544+0000","last_active":"2024-04-02T07:52:10.370821+0000","last_peered":"2024-04-02T07:52:10.370821+0000","last_clean":"2024-04-02T07:52:10.370821+0000","last_became_active":"2024-04-02T07:52:09.081258+0000","last_became_peered":"2024-04-02T07:52:09.081258+0000","last_unstale":"2024-04-02T07:52:10.370821+0000","last_undegraded":"2024-04-02T07:52:10.370821+0000","last_fullsized":"2024-04-02T07:52:10.370821+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:58:29.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085478+0000","last_change":"2024-04-02T07:52:08.597717+0000","last_active":"2024-04-02T07:52:10.085478+0000","last_peered":"2024-04-02T07:52:10.085478+0000","last_clean":"2024-04-02T07:52:10.085478+0000","last_became_active":"2024-04-02T07:52:08.597509+0000","last_became_peered":"2024-04-02T07:52:08.597509+0000","last_unstale":"2024-04-02T07:52:10.085478+0000","last_undegraded":"2024-04-02T07:52:10.085478+0000","last_fullsized":"2024-04-02T07:52:10.085478+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:08:35.958483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371066+0000","last_change":"2024-04-02T07:52:09.091579+0000","last_active":"2024-04-02T07:52:10.371066+0000","last_peered":"2024-04-02T07:52:10.371066+0000","last_clean":"2024-04-02T07:52:10.371066+0000","last_became_active":"2024-04-02T07:52:09.091320+0000","last_became_peered":"2024-04-02T07:52:09.091320+0000","last_unstale":"2024-04-02T07:52:10.371066+0000","last_undegraded":"2024-04-02T07:52:10.371066+0000","last_fullsized":"2024-04-02T07:52:10.371066+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:52:35.122693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.10","version":"96'8","reported_seq":19,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085255+0000","last_change":"2024-04-02T07:52:08.589916+0000","last_active":"2024-04-02T07:52:10.085255+0000","last_peered":"2024-04-02T07:52:10.085255+0000","last_clean":"2024-04-02T07:52:10.085255+0000","last_became_active":"2024-04-02T07:52:08.589115+0000","last_became_peered":"2024-04-02T07:52:08.589115+0000","last_unstale":"2024-04-02T07:52:10.085255+0000","last_undegraded":"2024-04-02T07:52:10.085255+0000","last_fullsized":"2024-04-02T07:52:10.085255+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:19:15.104550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081125+0000","last_change":"2024-04-02T07:52:09.082446+0000","last_active":"2024-04-02T07:52:10.081125+0000","last_peered":"2024-04-02T07:52:10.081125+0000","last_clean":"2024-04-02T07:52:10.081125+0000","last_became_active":"2024-04-02T07:52:09.082203+0000","last_became_peered":"2024-04-02T07:52:09.082203+0000","last_unstale":"2024-04-02T07:52:10.081125+0000","last_undegraded":"2024-04-02T07:52:10.081125+0000","last_fullsized":"2024-04-02T07:52:10.081125+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:56:28.091780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"96'1","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174755+0000","last_change":"2024-04-02T07:52:08.593143+0000","last_active":"2024-04-02T07:52:10.174755+0000","last_peered":"2024-04-02T07:52:10.174755+0000","last_clean":"2024-04-02T07:52:10.174755+0000","last_became_active":"2024-04-02T07:52:08.592516+0000","last_became_peered":"2024-04-02T07:52:08.592516+0000","last_unstale":"2024-04-02T07:52:10.174755+0000","last_undegraded":"2024-04-02T07:52:10.174755+0000","last_fullsized":"2024-04-02T07:52:10.174755+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:10:00.180401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081176+0000","last_change":"2024-04-02T07:52:09.087842+0000","last_active":"2024-04-02T07:52:10.081176+0000","last_peered":"2024-04-02T07:52:10.081176+0000","last_clean":"2024-04-02T07:52:10.081176+0000","last_became_active":"2024-04-02T07:52:09.087582+0000","last_became_peered":"2024-04-02T07:52:09.087582+0000","last_unstale":"2024-04-02T07:52:10.081176+0000","last_undegraded":"2024-04-02T07:52:10.081176+0000","last_fullsized":"2024-04-02T07:52:10.081176+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:50:40.257112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"96'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081263+0000","last_change":"2024-04-02T07:52:08.595233+0000","last_active":"2024-04-02T07:52:10.081263+0000","last_peered":"2024-04-02T07:52:10.081263+0000","last_clean":"2024-04-02T07:52:10.081263+0000","last_became_active":"2024-04-02T07:52:08.594931+0000","last_became_peered":"2024-04-02T07:52:08.594931+0000","last_unstale":"2024-04-02T07:52:10.081263+0000","last_undegraded":"2024-04-02T07:52:10.081263+0000","last_fullsized":"2024-04-02T07:52:10.081263+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:41:55.279920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371499+0000","last_change":"2024-04-02T07:52:09.086089+0000","last_active":"2024-04-02T07:52:10.371499+0000","last_peered":"2024-04-02T07:52:10.371499+0000","last_clean":"2024-04-02T07:52:10.371499+0000","last_became_active":"2024-04-02T07:52:09.085661+0000","last_became_peered":"2024-04-02T07:52:09.085661+0000","last_unstale":"2024-04-02T07:52:10.371499+0000","last_undegraded":"2024-04-02T07:52:10.371499+0000","last_fullsized":"2024-04-02T07:52:10.371499+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:32:38.860985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"96'1","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.080980+0000","last_change":"2024-04-02T07:52:08.594106+0000","last_active":"2024-04-02T07:52:10.080980+0000","last_peered":"2024-04-02T07:52:10.080980+0000","last_clean":"2024-04-02T07:52:10.080980+0000","last_became_active":"2024-04-02T07:52:08.593089+0000","last_became_peered":"2024-04-02T07:52:08.593089+0000","last_unstale":"2024-04-02T07:52:10.080980+0000","last_undegraded":"2024-04-02T07:52:10.080980+0000","last_fullsized":"2024-04-02T07:52:10.080980+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:31:06.509089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371415+0000","last_change":"2024-04-02T07:52:09.104377+0000","last_active":"2024-04-02T07:52:10.371415+0000","last_peered":"2024-04-02T07:52:10.371415+0000","last_clean":"2024-04-02T07:52:10.371415+0000","last_became_active":"2024-04-02T07:52:09.103930+0000","last_became_peered":"2024-04-02T07:52:09.103930+0000","last_unstale":"2024-04-02T07:52:10.371415+0000","last_undegraded":"2024-04-02T07:52:10.371415+0000","last_fullsized":"2024-04-02T07:52:10.371415+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:53:48.468830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208083+0000","last_change":"2024-04-02T07:52:08.583436+0000","last_active":"2024-04-02T07:52:10.208083+0000","last_peered":"2024-04-02T07:52:10.208083+0000","last_clean":"2024-04-02T07:52:10.208083+0000","last_became_active":"2024-04-02T07:52:08.582022+0000","last_became_peered":"2024-04-02T07:52:08.582022+0000","last_unstale":"2024-04-02T07:52:10.208083+0000","last_undegraded":"2024-04-02T07:52:10.208083+0000","last_fullsized":"2024-04-02T07:52:10.208083+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:12.802978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371583+0000","last_change":"2024-04-02T07:52:09.097397+0000","last_active":"2024-04-02T07:52:10.371583+0000","last_peered":"2024-04-02T07:52:10.371583+0000","last_clean":"2024-04-02T07:52:10.371583+0000","last_became_active":"2024-04-02T07:52:09.097093+0000","last_became_peered":"2024-04-02T07:52:09.097093+0000","last_unstale":"2024-04-02T07:52:10.371583+0000","last_undegraded":"2024-04-02T07:52:10.371583+0000","last_fullsized":"2024-04-02T07:52:10.371583+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:29:10.607669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208149+0000","last_change":"2024-04-02T07:52:08.580244+0000","last_active":"2024-04-02T07:52:10.208149+0000","last_peered":"2024-04-02T07:52:10.208149+0000","last_clean":"2024-04-02T07:52:10.208149+0000","last_became_active":"2024-04-02T07:52:08.579997+0000","last_became_peered":"2024-04-02T07:52:08.579997+0000","last_unstale":"2024-04-02T07:52:10.208149+0000","last_undegraded":"2024-04-02T07:52:10.208149+0000","last_fullsized":"2024-04-02T07:52:10.208149+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:53:47.460368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.370806+0000","last_change":"2024-04-02T07:52:09.091632+0000","last_active":"2024-04-02T07:52:10.370806+0000","last_peered":"2024-04-02T07:52:10.370806+0000","last_clean":"2024-04-02T07:52:10.370806+0000","last_became_active":"2024-04-02T07:52:09.089418+0000","last_became_peered":"2024-04-02T07:52:09.089418+0000","last_unstale":"2024-04-02T07:52:10.370806+0000","last_undegraded":"2024-04-02T07:52:10.370806+0000","last_fullsized":"2024-04-02T07:52:10.370806+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:15.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.080865+0000","last_change":"2024-04-02T07:52:08.590287+0000","last_active":"2024-04-02T07:52:10.080865+0000","last_peered":"2024-04-02T07:52:10.080865+0000","last_clean":"2024-04-02T07:52:10.080865+0000","last_became_active":"2024-04-02T07:52:08.589813+0000","last_became_peered":"2024-04-02T07:52:08.589813+0000","last_unstale":"2024-04-02T07:52:10.080865+0000","last_undegraded":"2024-04-02T07:52:10.080865+0000","last_fullsized":"2024-04-02T07:52:10.080865+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:36.132695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085664+0000","last_change":"2024-04-02T07:52:09.088271+0000","last_active":"2024-04-02T07:52:10.085664+0000","last_peered":"2024-04-02T07:52:10.085664+0000","last_clean":"2024-04-02T07:52:10.085664+0000","last_became_active":"2024-04-02T07:52:09.087944+0000","last_became_peered":"2024-04-02T07:52:09.087944+0000","last_unstale":"2024-04-02T07:52:10.085664+0000","last_undegraded":"2024-04-02T07:52:10.085664+0000","last_fullsized":"2024-04-02T07:52:10.085664+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:37.141157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.370778+0000","last_change":"2024-04-02T07:52:08.590519+0000","last_active":"2024-04-02T07:52:10.370778+0000","last_peered":"2024-04-02T07:52:10.370778+0000","last_clean":"2024-04-02T07:52:10.370778+0000","last_became_active":"2024-04-02T07:52:08.589950+0000","last_became_peered":"2024-04-02T07:52:08.589950+0000","last_unstale":"2024-04-02T07:52:10.370778+0000","last_undegraded":"2024-04-02T07:52:10.370778+0000","last_fullsized":"2024-04-02T07:52:10.370778+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:49:34.574435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.372061+0000","last_change":"2024-04-02T07:52:09.095415+0000","last_active":"2024-04-02T07:52:10.372061+0000","last_peered":"2024-04-02T07:52:10.372061+0000","last_clean":"2024-04-02T07:52:10.372061+0000","last_became_active":"2024-04-02T07:52:09.094034+0000","last_became_peered":"2024-04-02T07:52:09.094034+0000","last_unstale":"2024-04-02T07:52:10.372061+0000","last_undegraded":"2024-04-02T07:52:10.372061+0000","last_fullsized":"2024-04-02T07:52:10.372061+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:50:09.238288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"96'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174779+0000","last_change":"2024-04-02T07:52:08.593299+0000","last_active":"2024-04-02T07:52:10.174779+0000","last_peered":"2024-04-02T07:52:10.174779+0000","last_clean":"2024-04-02T07:52:10.174779+0000","last_became_active":"2024-04-02T07:52:08.592890+0000","last_became_peered":"2024-04-02T07:52:08.592890+0000","last_unstale":"2024-04-02T07:52:10.174779+0000","last_undegraded":"2024-04-02T07:52:10.174779+0000","last_fullsized":"2024-04-02T07:52:10.174779+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:49:34.574435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085651+0000","last_change":"2024-04-02T07:52:09.095722+0000","last_active":"2024-04-02T07:52:10.085651+0000","last_peered":"2024-04-02T07:52:10.085651+0000","last_clean":"2024-04-02T07:52:10.085651+0000","last_became_active":"2024-04-02T07:52:09.094015+0000","last_became_peered":"2024-04-02T07:52:09.094015+0000","last_unstale":"2024-04-02T07:52:10.085651+0000","last_undegraded":"2024-04-02T07:52:10.085651+0000","last_fullsized":"2024-04-02T07:52:10.085651+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:37:15.846519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174591+0000","last_change":"2024-04-02T07:52:08.596215+0000","last_active":"2024-04-02T07:52:10.174591+0000","last_peered":"2024-04-02T07:52:10.174591+0000","last_clean":"2024-04-02T07:52:10.174591+0000","last_became_active":"2024-04-02T07:52:08.595949+0000","last_became_peered":"2024-04-02T07:52:08.595949+0000","last_unstale":"2024-04-02T07:52:10.174591+0000","last_undegraded":"2024-04-02T07:52:10.174591+0000","last_fullsized":"2024-04-02T07:52:10.174591+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:39.822086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.080943+0000","last_change":"2024-04-02T07:52:09.084594+0000","last_active":"2024-04-02T07:52:10.080943+0000","last_peered":"2024-04-02T07:52:10.080943+0000","last_clean":"2024-04-02T07:52:10.080943+0000","last_became_active":"2024-04-02T07:52:09.083527+0000","last_became_peered":"2024-04-02T07:52:09.083527+0000","last_unstale":"2024-04-02T07:52:10.080943+0000","last_undegraded":"2024-04-02T07:52:10.080943+0000","last_fullsized":"2024-04-02T07:52:10.080943+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:17:07.389471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085995+0000","last_change":"2024-04-02T07:52:08.588758+0000","last_active":"2024-04-02T07:52:10.085995+0000","last_peered":"2024-04-02T07:52:10.085995+0000","last_clean":"2024-04-02T07:52:10.085995+0000","last_became_active":"2024-04-02T07:52:08.588484+0000","last_became_peered":"2024-04-02T07:52:08.588484+0000","last_unstale":"2024-04-02T07:52:10.085995+0000","last_undegraded":"2024-04-02T07:52:10.085995+0000","last_fullsized":"2024-04-02T07:52:10.085995+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:31:28.257636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085631+0000","last_change":"2024-04-02T07:52:09.096665+0000","last_active":"2024-04-02T07:52:10.085631+0000","last_peered":"2024-04-02T07:52:10.085631+0000","last_clean":"2024-04-02T07:52:10.085631+0000","last_became_active":"2024-04-02T07:52:09.095310+0000","last_became_peered":"2024-04-02T07:52:09.095310+0000","last_unstale":"2024-04-02T07:52:10.085631+0000","last_undegraded":"2024-04-02T07:52:10.085631+0000","last_fullsized":"2024-04-02T07:52:10.085631+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:32:12.614437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085781+0000","last_change":"2024-04-02T07:52:08.584788+0000","last_active":"2024-04-02T07:52:10.085781+0000","last_peered":"2024-04-02T07:52:10.085781+0000","last_clean":"2024-04-02T07:52:10.085781+0000","last_became_active":"2024-04-02T07:52:08.584513+0000","last_became_peered":"2024-04-02T07:52:08.584513+0000","last_unstale":"2024-04-02T07:52:10.085781+0000","last_undegraded":"2024-04-02T07:52:10.085781+0000","last_fullsized":"2024-04-02T07:52:10.085781+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:14.420847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371219+0000","last_change":"2024-04-02T07:52:09.089155+0000","last_active":"2024-04-02T07:52:10.371219+0000","last_peered":"2024-04-02T07:52:10.371219+0000","last_clean":"2024-04-02T07:52:10.371219+0000","last_became_active":"2024-04-02T07:52:09.088869+0000","last_became_peered":"2024-04-02T07:52:09.088869+0000","last_unstale":"2024-04-02T07:52:10.371219+0000","last_undegraded":"2024-04-02T07:52:10.371219+0000","last_fullsized":"2024-04-02T07:52:10.371219+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:35:04.350838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085980+0000","last_change":"2024-04-02T07:52:08.579841+0000","last_active":"2024-04-02T07:52:10.085980+0000","last_peered":"2024-04-02T07:52:10.085980+0000","last_clean":"2024-04-02T07:52:10.085980+0000","last_became_active":"2024-04-02T07:52:08.579164+0000","last_became_peered":"2024-04-02T07:52:08.579164+0000","last_unstale":"2024-04-02T07:52:10.085980+0000","last_undegraded":"2024-04-02T07:52:10.085980+0000","last_fullsized":"2024-04-02T07:52:10.085980+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:37:14.838057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174523+0000","last_change":"2024-04-02T07:52:09.097930+0000","last_active":"2024-04-02T07:52:10.174523+0000","last_peered":"2024-04-02T07:52:10.174523+0000","last_clean":"2024-04-02T07:52:10.174523+0000","last_became_active":"2024-04-02T07:52:09.097615+0000","last_became_peered":"2024-04-02T07:52:09.097615+0000","last_unstale":"2024-04-02T07:52:10.174523+0000","last_undegraded":"2024-04-02T07:52:10.174523+0000","last_fullsized":"2024-04-02T07:52:10.174523+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:21:34.806245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371399+0000","last_change":"2024-04-02T07:52:08.592398+0000","last_active":"2024-04-02T07:52:10.371399+0000","last_peered":"2024-04-02T07:52:10.371399+0000","last_clean":"2024-04-02T07:52:10.371399+0000","last_became_active":"2024-04-02T07:52:08.592042+0000","last_became_peered":"2024-04-02T07:52:08.592042+0000","last_unstale":"2024-04-02T07:52:10.371399+0000","last_undegraded":"2024-04-02T07:52:10.371399+0000","last_fullsized":"2024-04-02T07:52:10.371399+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:10:00.180401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.370815+0000","last_change":"2024-04-02T07:52:09.093427+0000","last_active":"2024-04-02T07:52:10.370815+0000","last_peered":"2024-04-02T07:52:10.370815+0000","last_clean":"2024-04-02T07:52:10.370815+0000","last_became_active":"2024-04-02T07:52:09.093155+0000","last_became_peered":"2024-04-02T07:52:09.093155+0000","last_unstale":"2024-04-02T07:52:10.370815+0000","last_undegraded":"2024-04-02T07:52:10.370815+0000","last_fullsized":"2024-04-02T07:52:10.370815+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:52:35.122693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.080952+0000","last_change":"2024-04-02T07:52:08.587748+0000","last_active":"2024-04-02T07:52:10.080952+0000","last_peered":"2024-04-02T07:52:10.080952+0000","last_clean":"2024-04-02T07:52:10.080952+0000","last_became_active":"2024-04-02T07:52:08.587461+0000","last_became_peered":"2024-04-02T07:52:08.587461+0000","last_unstale":"2024-04-02T07:52:10.080952+0000","last_undegraded":"2024-04-02T07:52:10.080952+0000","last_fullsized":"2024-04-02T07:52:10.080952+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:27:59.288400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.080777+0000","last_change":"2024-04-02T07:52:09.107720+0000","last_active":"2024-04-02T07:52:10.080777+0000","last_peered":"2024-04-02T07:52:10.080777+0000","last_clean":"2024-04-02T07:52:10.080777+0000","last_became_active":"2024-04-02T07:52:09.106387+0000","last_became_peered":"2024-04-02T07:52:09.106387+0000","last_unstale":"2024-04-02T07:52:10.080777+0000","last_undegraded":"2024-04-02T07:52:10.080777+0000","last_fullsized":"2024-04-02T07:52:10.080777+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:28:11.709852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208235+0000","last_change":"2024-04-02T07:52:08.585985+0000","last_active":"2024-04-02T07:52:10.208235+0000","last_peered":"2024-04-02T07:52:10.208235+0000","last_clean":"2024-04-02T07:52:10.208235+0000","last_became_active":"2024-04-02T07:52:08.585720+0000","last_became_peered":"2024-04-02T07:52:08.585720+0000","last_unstale":"2024-04-02T07:52:10.208235+0000","last_undegraded":"2024-04-02T07:52:10.208235+0000","last_fullsized":"2024-04-02T07:52:10.208235+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:52:34.114231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371316+0000","last_change":"2024-04-02T07:52:09.084990+0000","last_active":"2024-04-02T07:52:10.371316+0000","last_peered":"2024-04-02T07:52:10.371316+0000","last_clean":"2024-04-02T07:52:10.371316+0000","last_became_active":"2024-04-02T07:52:09.084740+0000","last_became_peered":"2024-04-02T07:52:09.084740+0000","last_unstale":"2024-04-02T07:52:10.371316+0000","last_undegraded":"2024-04-02T07:52:10.371316+0000","last_fullsized":"2024-04-02T07:52:10.371316+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:33.887916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"96'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085452+0000","last_change":"2024-04-02T07:52:08.595290+0000","last_active":"2024-04-02T07:52:10.085452+0000","last_peered":"2024-04-02T07:52:10.085452+0000","last_clean":"2024-04-02T07:52:10.085452+0000","last_became_active":"2024-04-02T07:52:08.593652+0000","last_became_peered":"2024-04-02T07:52:08.593652+0000","last_unstale":"2024-04-02T07:52:10.085452+0000","last_undegraded":"2024-04-02T07:52:10.085452+0000","last_fullsized":"2024-04-02T07:52:10.085452+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:46:35.968508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"86'172","reported_seq":433,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-02T07:52:09.085835+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:52:09.085835+0000","last_peered":"2024-04-02T07:52:09.085835+0000","last_clean":"2024-04-02T07:52:09.085835+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:52:09.085835+0000","last_undegraded":"2024-04-02T07:52:09.085835+0000","last_fullsized":"2024-04-02T07:52:09.085835+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575043,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1183744,"data_stored":571775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66800000000000004,"15min":0.67800000000000005},"min":{"1min":0.39400000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.67800000000000005},"min":{"1min":0.47399999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.95699999999999996,"5min":1.0109999999999999,"15min":1.024},"last":0.437}]},{"osd":1,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.67300000000000004,"15min":0.68000000000000005},"min":{"1min":0.499,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.89100000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66500000000000004,"15min":0.67500000000000004},"min":{"1min":0.42399999999999999,"5min":0.35599999999999998,"15min":0.34699999999999998},"max":{"1min":0.88300000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.62}]},{"osd":2,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.71699999999999997,"15min":0.71099999999999997},"min":{"1min":0.50700000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.94199999999999995,"5min":1.127,"15min":1.127},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.748,"5min":0.753,"15min":0.76000000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.434},"max":{"1min":0.93799999999999994,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.50800000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.69599999999999995,"15min":0.70399999999999996},"min":{"1min":0.44,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.97799999999999998,"5min":1.026,"15min":1.274},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.52600000000000002,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.90600000000000003,"5min":1.196,"15min":1.196},"last":0.59299999999999997}]},{"osd":4,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.376,"5min":0.32000000000000001,"15min":0.30199999999999999},"max":{"1min":0.84099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68200000000000005,"15min":0.68700000000000006},"min":{"1min":0.51800000000000002,"5min":0.40000000000000002,"15min":0.38100000000000001},"max":{"1min":0.84599999999999997,"5min":1.0669999999999999,"15min":1.4890000000000001},"last":0.52200000000000002}]},{"osd":5,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.46100000000000002,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":0.86299999999999999,"5min":1.167,"15min":1.167},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68500000000000005,"15min":0.68700000000000006},"min":{"1min":0.51100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.58099999999999996}]},{"osd":6,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67600000000000005,"15min":0.65800000000000003},"min":{"1min":0.505,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.877,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66000000000000003,"15min":0.65300000000000002},"min":{"1min":0.47999999999999998,"5min":0.42299999999999999,"15min":0.39600000000000002},"max":{"1min":0.89300000000000002,"5min":1.0509999999999999,"15min":1.329},"last":0.60899999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575041,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36396,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335444,"statfs":{"total":96636764160,"available":96599494656,"internally_reserved":0,"allocated":1200128,"data_stored":578457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.51300000000000001,"5min":0.38700000000000001,"15min":0.32400000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.98099999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.749,"5min":0.71599999999999997,"15min":0.70899999999999996},"min":{"1min":0.55200000000000005,"5min":0.38300000000000001,"15min":0.375},"max":{"1min":0.89400000000000002,"5min":0.96899999999999997,"15min":1.0249999999999999},"last":0.71299999999999997}]},{"osd":1,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69399999999999995,"15min":0.69599999999999995},"min":{"1min":0.53900000000000003,"5min":0.46500000000000002,"15min":0.41199999999999998},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":1.085},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.56999999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.752}]},{"osd":2,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73099999999999998,"15min":0.71499999999999997},"min":{"1min":0.48999999999999999,"5min":0.45500000000000002,"15min":0.38200000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71799999999999997,"15min":0.70699999999999996},"min":{"1min":0.53300000000000003,"5min":0.51400000000000001,"15min":0.45900000000000002},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96599999999999997},"last":0.53500000000000003}]},{"osd":3,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.73699999999999999,"15min":0.73899999999999999},"min":{"1min":0.55400000000000005,"5min":0.5,"15min":0.47099999999999997},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.748,"15min":0.73999999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.45300000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.0309999999999999},"last":0.68100000000000005}]},{"osd":4,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61399999999999999,"15min":0.60499999999999998},"min":{"1min":0.39300000000000002,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.64000000000000001,"15min":0.629},"min":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.35499999999999998},"max":{"1min":0.93200000000000005,"5min":0.99299999999999999,"15min":1.0049999999999999},"last":0.63200000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.629,"15min":0.621},"min":{"1min":0.41599999999999998,"5min":0.379,"15min":0.35799999999999998},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":1.1599999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.67200000000000004,"15min":0.68600000000000005},"min":{"1min":0.504,"5min":0.42099999999999999,"15min":0.40300000000000002},"max":{"1min":0.93899999999999995,"5min":1.01,"15min":1.095},"last":0.79300000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.68899999999999995,"15min":0.68600000000000005},"min":{"1min":0.48099999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.96199999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.45200000000000001,"5min":0.44,"15min":0.42099999999999999},"max":{"1min":0.92500000000000004,"5min":1.0149999999999999,"15min":1.4399999999999999},"last":0.61499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575042,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55004,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316836,"statfs":{"total":96636764160,"available":96580440064,"internally_reserved":0,"allocated":1777664,"data_stored":1167227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.432,"15min":0.44},"min":{"1min":0.29499999999999998,"5min":0.27500000000000002,"15min":0.25700000000000001},"max":{"1min":0.58899999999999997,"5min":0.77600000000000002,"15min":2.9079999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.50700000000000001,"15min":0.504},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.70699999999999996,"5min":0.86499999999999999,"15min":2.766},"last":0.27500000000000002}]},{"osd":2,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.627,"5min":0.71599999999999997,"15min":2.0750000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.443,"5min":0.36699999999999999,"15min":0.35699999999999998},"max":{"1min":0.71499999999999997,"5min":0.85399999999999998,"15min":1.7410000000000001},"last":0.45800000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.41799999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72399999999999998,"5min":0.81799999999999995,"15min":2.9430000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68000000000000005,"5min":0.82699999999999996,"15min":2.802},"last":0.35599999999999998}]},{"osd":4,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.78300000000000003,"15min":0.77400000000000002},"min":{"1min":0.67400000000000004,"5min":0.46700000000000003,"15min":0.46300000000000002},"max":{"1min":1.0609999999999999,"5min":1.861,"15min":1.861},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.77700000000000002,"15min":0.76800000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46500000000000002},"max":{"1min":1.232,"5min":1.232,"15min":2.0419999999999998},"last":0.69499999999999995}]},{"osd":5,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.77900000000000003,"15min":0.77100000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.41999999999999998},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.9690000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79400000000000004,"15min":0.78300000000000003},"min":{"1min":0.65600000000000003,"5min":0.51000000000000001,"15min":0.48799999999999999},"max":{"1min":1.147,"5min":1.54,"15min":1.54},"last":0.627}]},{"osd":6,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.79900000000000004,"15min":0.79500000000000004},"min":{"1min":0.53200000000000003,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.097,"5min":1.577,"15min":1.768},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.80400000000000005,"15min":0.79100000000000004},"min":{"1min":0.61399999999999999,"5min":0.51400000000000001,"15min":0.38100000000000001},"max":{"1min":1.351,"5min":1.351,"15min":2.012},"last":0.60899999999999999}]},{"osd":7,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.80900000000000005,"15min":0.80300000000000005},"min":{"1min":0.628,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.298,"5min":1.552,"15min":1.706},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.64300000000000002,"5min":0.53200000000000003,"15min":0.502},"max":{"1min":1.194,"5min":1.2290000000000001,"15min":1.8240000000000001},"last":0.64100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575042,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1183744,"data_stored":575910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.441,"15min":0.44400000000000001},"min":{"1min":0.29699999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71199999999999997,"5min":0.78900000000000003,"15min":2.1640000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.495,"15min":0.496},"min":{"1min":0.34899999999999998,"5min":0.307,"15min":0.26200000000000001},"max":{"1min":0.70599999999999996,"5min":0.72199999999999998,"15min":2.6469999999999998},"last":0.47299999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48999999999999999,"15min":0.495},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63100000000000001,"5min":0.749,"15min":2.125},"last":0.496},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":2.605},"last":0.51600000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52900000000000003,"15min":0.54800000000000004},"min":{"1min":0.33600000000000002,"5min":0.32700000000000001,"15min":0.28899999999999998},"max":{"1min":0.69799999999999995,"5min":0.79400000000000004,"15min":5.4800000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.40100000000000002},"max":{"1min":0.83199999999999996,"5min":0.93400000000000005,"15min":2.194},"last":0.52600000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.77700000000000002,"15min":0.80100000000000005},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.35599999999999998},"max":{"1min":0.97999999999999998,"5min":1.0640000000000001,"15min":6.0899999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78400000000000003,"15min":0.79600000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.433},"max":{"1min":1.048,"5min":1.095,"15min":5.7169999999999996},"last":0.84799999999999998}]},{"osd":5,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.81100000000000005,"15min":0.81200000000000006},"min":{"1min":0.55800000000000005,"5min":0.51000000000000001,"15min":0.42099999999999999},"max":{"1min":0.98999999999999999,"5min":2.1869999999999998,"15min":5.7539999999999996},"last":0.876},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.81799999999999995},"min":{"1min":0.63100000000000001,"5min":0.54100000000000004,"15min":0.51800000000000002},"max":{"1min":0.96299999999999997,"5min":2.6230000000000002,"15min":5.8929999999999998},"last":0.81599999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.75700000000000001,"15min":0.77100000000000002},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.46800000000000003},"max":{"1min":0.96599999999999997,"5min":1.044,"15min":5.931},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.755,"15min":0.77700000000000002},"min":{"1min":0.64700000000000002,"5min":0.50700000000000001,"15min":0.376},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":6.125},"last":0.76200000000000001}]},{"osd":7,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.83799999999999997},"min":{"1min":0.70299999999999996,"5min":0.51600000000000001,"15min":0.40799999999999997},"max":{"1min":1.0900000000000001,"5min":2.3959999999999999,"15min":6.1580000000000004},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.81399999999999995,"15min":0.83199999999999996},"min":{"1min":0.54100000000000004,"5min":0.48299999999999998,"15min":0.47699999999999998},"max":{"1min":1.0249999999999999,"5min":2.145,"15min":5.7850000000000001},"last":0.68700000000000006}]}]},{"osd":2,"up_from":13,"seq":55834575041,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36368,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335472,"statfs":{"total":96636764160,"available":96599523328,"internally_reserved":0,"allocated":1175552,"data_stored":570793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.44,"15min":0.44800000000000001},"min":{"1min":0.26700000000000002,"5min":0.26400000000000001,"15min":0.25700000000000001},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46300000000000002,"15min":0.47699999999999998},"min":{"1min":0.36199999999999999,"5min":0.27900000000000003,"15min":0.26200000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.52100000000000002}]},{"osd":1,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.49199999999999999,"15min":0.49099999999999999},"min":{"1min":0.33000000000000002,"5min":0.30599999999999999,"15min":0.30299999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.52400000000000002},"min":{"1min":0.39700000000000002,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.47899999999999998}]},{"osd":3,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.35999999999999999,"5min":0.316,"15min":0.27800000000000002},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59599999999999997,"15min":0.60999999999999999},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39000000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.54500000000000004}]},{"osd":4,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76300000000000001,"15min":0.77000000000000002},"min":{"1min":0.5,"5min":0.5,"15min":0.47199999999999998},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.748,"15min":0.77200000000000002},"min":{"1min":0.56499999999999995,"5min":0.51600000000000001,"15min":0.48499999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.76400000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.752,"15min":0.76100000000000001},"min":{"1min":0.56499999999999995,"5min":0.54300000000000004,"15min":0.42199999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.55600000000000005,"5min":0.53800000000000003,"15min":0.50900000000000001},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.81799999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.79400000000000004,"15min":0.79200000000000004},"min":{"1min":0.55000000000000004,"5min":0.53600000000000003,"15min":0.442},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.878},{"interface":"front","average":{"1min":0.752,"5min":0.75600000000000001,"15min":0.78700000000000003},"min":{"1min":0.53100000000000003,"5min":0.502,"15min":0.502},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.745}]},{"osd":7,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.77200000000000002,"15min":0.79500000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.5},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.77100000000000002,"15min":0.79300000000000004},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.70799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575041,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36380,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335460,"statfs":{"total":96636764160,"available":96599511040,"internally_reserved":0,"allocated":1196032,"data_stored":576985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.42399999999999999,"15min":0.42999999999999999},"min":{"1min":0.26500000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.72599999999999998,"5min":0.748,"15min":0.79600000000000004},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.435,"15min":0.46400000000000002},"min":{"1min":0.31,"5min":0.29699999999999999,"15min":0.28699999999999998},"max":{"1min":0.57099999999999995,"5min":0.70799999999999996,"15min":0.81899999999999995},"last":0.38800000000000001}]},{"osd":1,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.47899999999999998,"15min":0.49199999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61899999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.439,"5min":0.35799999999999998,"15min":0.34200000000000003},"max":{"1min":0.77500000000000002,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.624}]},{"osd":2,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59099999999999997,"15min":0.60699999999999998},"min":{"1min":0.439,"5min":0.38700000000000001,"15min":0.376},"max":{"1min":0.78400000000000003,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57699999999999996,"15min":0.60399999999999998},"min":{"1min":0.39500000000000002,"5min":0.38100000000000001,"15min":0.35899999999999999},"max":{"1min":0.70699999999999996,"5min":1.244,"15min":1.244},"last":0.64800000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.70899999999999996,"15min":0.73199999999999998},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.38400000000000001},"max":{"1min":0.83499999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73199999999999998,"15min":0.75700000000000001},"min":{"1min":0.52300000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":1.1759999999999999},"last":0.89000000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.72399999999999998,"15min":0.754},"min":{"1min":0.51300000000000001,"5min":0.41699999999999998,"15min":0.40899999999999997},"max":{"1min":0.97299999999999998,"5min":1.079,"15min":1.151},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72099999999999997,"15min":0.749},"min":{"1min":0.49199999999999999,"5min":0.48899999999999999,"15min":0.40200000000000002},"max":{"1min":0.94199999999999995,"5min":0.97599999999999998,"15min":1.1439999999999999},"last":0.79800000000000004}]},{"osd":6,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72199999999999998,"15min":0.76000000000000001},"min":{"1min":0.52800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.86899999999999999,"5min":0.98999999999999999,"15min":1.0620000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.748,"15min":0.77400000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.90400000000000003,"5min":1.1140000000000001,"15min":1.1899999999999999},"last":0.878}]},{"osd":7,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.73799999999999999,"15min":0.77000000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.439},"max":{"1min":0.91100000000000003,"5min":1.0409999999999999,"15min":1.0680000000000001},"last":0.751},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.75,"15min":0.78500000000000003},"min":{"1min":0.60299999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0089999999999999,"5min":1.181,"15min":1.181},"last":0.77000000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575041,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36376,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335464,"statfs":{"total":96636764160,"available":96599515136,"internally_reserved":0,"allocated":1183744,"data_stored":570905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.42499999999999999,"5min":0.38100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86599999999999999,"5min":0.91900000000000004,"15min":1.0189999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.66800000000000004,"15min":0.67500000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.879,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.50600000000000001}]},{"osd":1,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.71599999999999997,"15min":0.72799999999999998},"min":{"1min":0.47599999999999998,"5min":0.438,"15min":0.433},"max":{"1min":0.90300000000000002,"5min":0.97299999999999998,"15min":1.036},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.73799999999999999},"min":{"1min":0.58099999999999996,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.91900000000000004,"5min":0.97899999999999998,"15min":0.99399999999999999},"last":0.70999999999999996}]},{"osd":2,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.67400000000000004,"15min":0.69099999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.47299999999999998,"5min":0.47199999999999998,"15min":0.41899999999999998},"max":{"1min":0.874,"5min":0.97199999999999998,"15min":1.0620000000000001},"last":0.81299999999999994}]},{"osd":3,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72099999999999997},"min":{"1min":0.51500000000000001,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":0.95299999999999996,"5min":0.95699999999999996,"15min":1.002},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.71299999999999997,"15min":0.72999999999999998},"min":{"1min":0.49299999999999999,"5min":0.48799999999999999,"15min":0.41499999999999998},"max":{"1min":0.92100000000000004,"5min":0.93999999999999995,"15min":0.96999999999999997},"last":0.63200000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56200000000000006,"15min":0.57399999999999995},"min":{"1min":0.45800000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.746,"5min":0.84799999999999998,"15min":0.93999999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.621,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.434,"5min":0.38800000000000001,"15min":0.33700000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.56899999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.61199999999999999,"15min":0.60399999999999998},"min":{"1min":0.45200000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.51600000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.038},"last":0.85099999999999998}]},{"osd":7,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68000000000000005,"15min":0.66800000000000004},"min":{"1min":0.55700000000000005,"5min":0.432,"15min":0.38300000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.74099999999999999,"15min":0.72999999999999998},"min":{"1min":0.58999999999999997,"5min":0.47799999999999998,"15min":0.434},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.93300000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575044,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36948,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334892,"statfs":{"total":96636764160,"available":96598929408,"internally_reserved":0,"allocated":1777664,"data_stored":1166298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67800000000000005,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.80900000000000005,"5min":0.95299999999999996,"15min":0.99199999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.878,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.92700000000000005}]},{"osd":1,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67600000000000005,"15min":0.69399999999999995},"min":{"1min":0.41999999999999998,"5min":0.40300000000000002,"15min":0.40200000000000002},"max":{"1min":1.0620000000000001,"5min":1.131,"15min":1.131},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.45000000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83999999999999997,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.623}]},{"osd":2,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70699999999999996,"15min":0.71299999999999997},"min":{"1min":0.47399999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.878,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.69399999999999995,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.47099999999999997,"15min":0.372},"max":{"1min":1.111,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.82599999999999996}]},{"osd":3,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.72899999999999998,"15min":0.74299999999999999},"min":{"1min":0.54800000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92300000000000004,"5min":1.341,"15min":1.341},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77000000000000002,"15min":0.76600000000000001},"min":{"1min":0.60699999999999998,"5min":0.54600000000000004,"15min":0.44900000000000001},"max":{"1min":1.1839999999999999,"5min":1.484,"15min":1.484},"last":0.88100000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59299999999999997,"15min":0.59999999999999998},"min":{"1min":0.39500000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.35599999999999998},"max":{"1min":0.96199999999999997,"5min":1.504,"15min":1.504},"last":0.85299999999999998}]},{"osd":6,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.64900000000000002,"15min":0.64300000000000002},"min":{"1min":0.45600000000000002,"5min":0.44700000000000001,"15min":0.39900000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.69199999999999995,"15min":0.67700000000000005},"min":{"1min":0.45500000000000002,"5min":0.44900000000000001,"15min":0.41199999999999998},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":0.90300000000000002}]},{"osd":7,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.71399999999999997,"15min":0.70299999999999996},"min":{"1min":0.502,"5min":0.47599999999999998,"15min":0.38900000000000001},"max":{"1min":1.0129999999999999,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.754,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.53600000000000003,"5min":0.46300000000000002,"15min":0.434},"max":{"1min":1.149,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.98099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:52:13.464 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:52:13.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.694+0000 7f6fa9600700 1 Processor -- start 2024-04-02T07:52:13.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.694+0000 7f6fa9600700 1 -- start start 2024-04-02T07:52:13.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.695+0000 7f6fa9600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6fa4161ca0 0x7f6fa40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:13.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.695+0000 7f6fa9600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fa40ceda0 con 0x7f6fa40d18b0 2024-04-02T07:52:13.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.695+0000 7f6fa9600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fa41612f0 con 0x7f6fa4160f10 2024-04-02T07:52:13.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.695+0000 7f6fa9600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fa40d1d30 con 0x7f6fa4161ca0 2024-04-02T07:52:13.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.696+0000 7f6fa2000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6fa40d18b0 0x7f6fa4161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60386/0 (socket says 172.21.15.89:60386) 2024-04-02T07:52:13.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.696+0000 7f6fa2000700 1 -- 172.21.15.89:0/699156183 learned_addr learned my addr 172.21.15.89:0/699156183 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:13.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.696+0000 7f6fa1600700 1 -- 172.21.15.89:0/699156183 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403691040 0 0) 0x7f6fa40ceda0 con 0x7f6fa40d18b0 2024-04-02T07:52:13.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.696+0000 7f6fa1600700 1 -- 172.21.15.89:0/699156183 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f880035b0 con 0x7f6fa40d18b0 2024-04-02T07:52:13.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.696+0000 7f6fa3400700 1 --2- 172.21.15.89:0/699156183 >> v2:172.21.15.113:3300/0 conn(0x7f6fa4161ca0 0x7f6fa40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:13.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.696+0000 7f6fa1600700 1 -- 172.21.15.89:0/699156183 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 480869530 0 0) 0x7f6f880035b0 con 0x7f6fa40d18b0 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.696+0000 7f6fa1600700 1 -- 172.21.15.89:0/699156183 >> v2:172.21.15.113:3300/0 conn(0x7f6fa4161ca0 msgr2=0x7f6fa40cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.696+0000 7f6fa1600700 1 --2- 172.21.15.89:0/699156183 >> v2:172.21.15.113:3300/0 conn(0x7f6fa4161ca0 0x7f6fa40cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.696+0000 7f6fa1600700 1 -- 172.21.15.89:0/699156183 >> v1:172.21.15.113:6789/0 conn(0x7f6fa4160f10 legacy=0x7f6fa40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.696+0000 7f6fa1600700 1 -- 172.21.15.89:0/699156183 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa40ce470 con 0x7f6fa40d18b0 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.696+0000 7f6fa1600700 1 -- 172.21.15.89:0/699156183 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f6f90001670 con 0x7f6fa40d18b0 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.696+0000 7f6fa1600700 1 -- 172.21.15.89:0/699156183 >> v1:172.21.15.89:6789/0 conn(0x7f6fa40d18b0 legacy=0x7f6fa4161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa1600700 1 --2- 172.21.15.89:0/699156183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f88003a00 0x7f6f88003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa1600700 1 --2- 172.21.15.89:0/699156183 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f88004850 0x7f6f88004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa1600700 1 --2- 172.21.15.89:0/699156183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f88005210 0x7f6f8800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa1600700 1 -- 172.21.15.89:0/699156183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa40d1d30 con 0x7f6f88005210 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa1600700 1 -- 172.21.15.89:0/699156183 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8800c1b0 con 0x7f6f88003a00 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa1600700 1 -- 172.21.15.89:0/699156183 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8800c370 con 0x7f6f88004850 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa3400700 1 --2- 172.21.15.89:0/699156183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f88005210 0x7f6f8800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa2a00700 1 --2- 172.21.15.89:0/699156183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f88003a00 0x7f6f88003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa2000700 1 --2- 172.21.15.89:0/699156183 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f88004850 0x7f6f88004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:13.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa3400700 1 -- 172.21.15.89:0/699156183 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f88004850 msgr2=0x7f6f88004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:13.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa3400700 1 --2- 172.21.15.89:0/699156183 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f88004850 0x7f6f88004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa3400700 1 -- 172.21.15.89:0/699156183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f88003a00 msgr2=0x7f6f88003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:13.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa3400700 1 --2- 172.21.15.89:0/699156183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f88003a00 0x7f6f88003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa3400700 1 -- 172.21.15.89:0/699156183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f880035b0 con 0x7f6f88005210 2024-04-02T07:52:13.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.697+0000 7f6fa3400700 1 -- 172.21.15.89:0/699156183 reap_dead start 2024-04-02T07:52:13.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.698+0000 7f6fa3400700 1 --2- 172.21.15.89:0/699156183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f88005210 0x7f6f8800ba10 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009fc0 tx=0x7f6f88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f02ac9d651aae3fe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:13.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.698+0000 7f6fa1600700 1 -- 172.21.15.89:0/699156183 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c003a20 con 0x7f6f88005210 2024-04-02T07:52:13.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.698+0000 7f6fa1600700 1 -- 172.21.15.89:0/699156183 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9c00f040 con 0x7f6f88005210 2024-04-02T07:52:13.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.698+0000 7f6fa1600700 1 -- 172.21.15.89:0/699156183 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c003da0 con 0x7f6f88005210 2024-04-02T07:52:13.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.698+0000 7f6fa9600700 1 -- 172.21.15.89:0/699156183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f88005210 msgr2=0x7f6f8800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:13.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.698+0000 7f6fa9600700 1 --2- 172.21.15.89:0/699156183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f88005210 0x7f6f8800ba10 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009fc0 tx=0x7f6f88005880 comp rx=0 tx=0).stop 2024-04-02T07:52:13.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.698+0000 7f6fa9600700 1 -- 172.21.15.89:0/699156183 shutdown_connections 2024-04-02T07:52:13.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.698+0000 7f6fa9600700 1 --2- 172.21.15.89:0/699156183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f88005210 0x7f6f8800ba10 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.698+0000 7f6fa9600700 1 -- 172.21.15.89:0/699156183 >> 172.21.15.89:0/699156183 conn(0x7f6fa408aff0 msgr2=0x7f6fa40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:13.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.698+0000 7f6fa9600700 1 -- 172.21.15.89:0/699156183 shutdown_connections 2024-04-02T07:52:13.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.698+0000 7f6fa9600700 1 -- 172.21.15.89:0/699156183 wait complete. 2024-04-02T07:52:13.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.700+0000 7f6fa9600700 1 Processor -- start 2024-04-02T07:52:13.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.700+0000 7f6fa9600700 1 -- start start 2024-04-02T07:52:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.701+0000 7f6fa9600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f88005210 0x7f6fa40b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.701+0000 7f6fa9600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fa414b9a0 0x7f6fa40b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.701+0000 7f6fa9600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fa40b6690 0x7f6fa40b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.701+0000 7f6fa9600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa415f900 con 0x7f6f88005210 2024-04-02T07:52:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.701+0000 7f6fa9600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa41e6f50 con 0x7f6fa40b6690 2024-04-02T07:52:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.701+0000 7f6fa9600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa4158e50 con 0x7f6fa414b9a0 2024-04-02T07:52:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.701+0000 7f6fa2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f88005210 0x7f6fa40b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6fa2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f88005210 0x7f6fa40b8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42026/0 (socket says 172.21.15.89:42026) 2024-04-02T07:52:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6fa2a00700 1 -- 172.21.15.89:0/3884528746 learned_addr learned my addr 172.21.15.89:0/3884528746 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6fa2000700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fa414b9a0 0x7f6fa40b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6fa3400700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fa40b6690 0x7f6fa40b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6fa2a00700 1 -- 172.21.15.89:0/3884528746 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fa414b9a0 msgr2=0x7f6fa40b6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6fa2a00700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fa414b9a0 0x7f6fa40b6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6fa2a00700 1 -- 172.21.15.89:0/3884528746 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fa40b6690 msgr2=0x7f6fa40b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6fa2a00700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fa40b6690 0x7f6fa40b6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6fa2a00700 1 -- 172.21.15.89:0/3884528746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa40b8f80 con 0x7f6f88005210 2024-04-02T07:52:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6fa2000700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fa414b9a0 0x7f6fa40b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6fa3400700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fa40b6690 0x7f6fa40b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6fa2a00700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f88005210 0x7f6fa40b8a40 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f6f9400f850 tx=0x7f6f9400fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6f9aa00700 1 -- 172.21.15.89:0/3884528746 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f940050d0 con 0x7f6f88005210 2024-04-02T07:52:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6f9aa00700 1 -- 172.21.15.89:0/3884528746 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f94005230 con 0x7f6f88005210 2024-04-02T07:52:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6f9aa00700 1 -- 172.21.15.89:0/3884528746 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f94003680 con 0x7f6f88005210 2024-04-02T07:52:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.702+0000 7f6fa9600700 1 -- 172.21.15.89:0/3884528746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa415f060 con 0x7f6f88005210 2024-04-02T07:52:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.703+0000 7f6fa9600700 1 -- 172.21.15.89:0/3884528746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa415f620 con 0x7f6f88005210 2024-04-02T07:52:13.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.704+0000 7f6fa9600700 1 -- 172.21.15.89:0/3884528746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa40d2e40 con 0x7f6f88005210 2024-04-02T07:52:13.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.704+0000 7f6f9aa00700 1 -- 172.21.15.89:0/3884528746 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6f940037e0 con 0x7f6f88005210 2024-04-02T07:52:13.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.705+0000 7f6f9aa00700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6f8406adb0 0x7f6f8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:13.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.705+0000 7f6f9aa00700 1 -- 172.21.15.89:0/3884528746 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6f940892a0 con 0x7f6f88005210 2024-04-02T07:52:13.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.706+0000 7f6fa2000700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6f8406adb0 0x7f6f8406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:13.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.706+0000 7f6fa2000700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6f8406adb0 0x7f6f8406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6f90001050 tx=0x7f6f90009040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:13.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.710+0000 7f6f9aa00700 1 -- 172.21.15.89:0/3884528746 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6f940542a0 con 0x7f6f88005210 2024-04-02T07:52:13.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.904+0000 7f6fa9600700 1 -- 172.21.15.89:0/3884528746 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6fa40ce120 con 0x7f6f8406adb0 2024-04-02T07:52:13.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.912+0000 7f6f9aa00700 1 -- 172.21.15.89:0/3884528746 <== mgr.6696 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289256 (secure 0 0 0) 0x7f6f90001550 con 0x7f6f8406adb0 2024-04-02T07:52:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.916+0000 7f6fa9600700 1 -- 172.21.15.89:0/3884528746 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6f8406adb0 msgr2=0x7f6f8406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.916+0000 7f6fa9600700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6f8406adb0 0x7f6f8406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6f90001050 tx=0x7f6f90009040 comp rx=0 tx=0).stop 2024-04-02T07:52:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.916+0000 7f6fa9600700 1 -- 172.21.15.89:0/3884528746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f88005210 msgr2=0x7f6fa40b8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.916+0000 7f6fa9600700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f88005210 0x7f6fa40b8a40 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f6f9400f850 tx=0x7f6f9400fc10 comp rx=0 tx=0).stop 2024-04-02T07:52:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.917+0000 7f6fa9600700 1 -- 172.21.15.89:0/3884528746 shutdown_connections 2024-04-02T07:52:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.917+0000 7f6fa9600700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6f8406adb0 0x7f6f8406d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.917+0000 7f6fa9600700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fa40b6690 0x7f6fa40b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.917+0000 7f6fa9600700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f88005210 0x7f6fa40b8a40 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.917+0000 7f6fa9600700 1 --2- 172.21.15.89:0/3884528746 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fa414b9a0 0x7f6fa40b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.917+0000 7f6fa9600700 1 -- 172.21.15.89:0/3884528746 >> 172.21.15.89:0/3884528746 conn(0x7f6fa408aff0 msgr2=0x7f6fa40ca650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.917+0000 7f6fa9600700 1 -- 172.21.15.89:0/3884528746 shutdown_connections 2024-04-02T07:52:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:13.917+0000 7f6fa9600700 1 -- 172.21.15.89:0/3884528746 wait complete. 2024-04-02T07:52:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:52:13.936 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T07:52:12.808643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":303,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":169,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":302024,"kb_used_data":10428,"kb_used_omap":12,"kb_used_meta":291443,"kb_avail":754672696,"statfs":{"total":773094113280,"available":772784840704,"internally_reserved":0,"allocated":10678272,"data_stored":5778350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":298438200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.242421"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174682+0000","last_change":"2024-04-02T07:52:09.100629+0000","last_active":"2024-04-02T07:52:10.174682+0000","last_peered":"2024-04-02T07:52:10.174682+0000","last_clean":"2024-04-02T07:52:10.174682+0000","last_became_active":"2024-04-02T07:52:09.100358+0000","last_became_peered":"2024-04-02T07:52:09.100358+0000","last_unstale":"2024-04-02T07:52:10.174682+0000","last_undegraded":"2024-04-02T07:52:10.174682+0000","last_fullsized":"2024-04-02T07:52:10.174682+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:35.764053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081216+0000","last_change":"2024-04-02T07:52:09.104514+0000","last_active":"2024-04-02T07:52:10.081216+0000","last_peered":"2024-04-02T07:52:10.081216+0000","last_clean":"2024-04-02T07:52:10.081216+0000","last_became_active":"2024-04-02T07:52:09.104272+0000","last_became_peered":"2024-04-02T07:52:09.104272+0000","last_unstale":"2024-04-02T07:52:10.081216+0000","last_undegraded":"2024-04-02T07:52:10.081216+0000","last_fullsized":"2024-04-02T07:52:10.081216+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:33:50.420553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371303+0000","last_change":"2024-04-02T07:52:09.099462+0000","last_active":"2024-04-02T07:52:10.371303+0000","last_peered":"2024-04-02T07:52:10.371303+0000","last_clean":"2024-04-02T07:52:10.371303+0000","last_became_active":"2024-04-02T07:52:09.098893+0000","last_became_peered":"2024-04-02T07:52:09.098893+0000","last_unstale":"2024-04-02T07:52:10.371303+0000","last_undegraded":"2024-04-02T07:52:10.371303+0000","last_fullsized":"2024-04-02T07:52:10.371303+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:31:29.266098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371390+0000","last_change":"2024-04-02T07:52:09.099355+0000","last_active":"2024-04-02T07:52:10.371390+0000","last_peered":"2024-04-02T07:52:10.371390+0000","last_clean":"2024-04-02T07:52:10.371390+0000","last_became_active":"2024-04-02T07:52:09.099010+0000","last_became_peered":"2024-04-02T07:52:09.099010+0000","last_unstale":"2024-04-02T07:52:10.371390+0000","last_undegraded":"2024-04-02T07:52:10.371390+0000","last_fullsized":"2024-04-02T07:52:10.371390+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:29:10.607669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371534+0000","last_change":"2024-04-02T07:52:09.085483+0000","last_active":"2024-04-02T07:52:10.371534+0000","last_peered":"2024-04-02T07:52:10.371534+0000","last_clean":"2024-04-02T07:52:10.371534+0000","last_became_active":"2024-04-02T07:52:09.085203+0000","last_became_peered":"2024-04-02T07:52:09.085203+0000","last_unstale":"2024-04-02T07:52:10.371534+0000","last_undegraded":"2024-04-02T07:52:10.371534+0000","last_fullsized":"2024-04-02T07:52:10.371534+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:17.505215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208121+0000","last_change":"2024-04-02T07:52:09.086817+0000","last_active":"2024-04-02T07:52:10.208121+0000","last_peered":"2024-04-02T07:52:10.208121+0000","last_clean":"2024-04-02T07:52:10.208121+0000","last_became_active":"2024-04-02T07:52:09.085142+0000","last_became_peered":"2024-04-02T07:52:09.085142+0000","last_unstale":"2024-04-02T07:52:10.208121+0000","last_undegraded":"2024-04-02T07:52:10.208121+0000","last_fullsized":"2024-04-02T07:52:10.208121+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:31:30.741833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085387+0000","last_change":"2024-04-02T07:52:09.101035+0000","last_active":"2024-04-02T07:52:10.085387+0000","last_peered":"2024-04-02T07:52:10.085387+0000","last_clean":"2024-04-02T07:52:10.085387+0000","last_became_active":"2024-04-02T07:52:09.100703+0000","last_became_peered":"2024-04-02T07:52:09.100703+0000","last_unstale":"2024-04-02T07:52:10.085387+0000","last_undegraded":"2024-04-02T07:52:10.085387+0000","last_fullsized":"2024-04-02T07:52:10.085387+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:10.755054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174200+0000","last_change":"2024-04-02T07:52:09.084193+0000","last_active":"2024-04-02T07:52:10.174200+0000","last_peered":"2024-04-02T07:52:10.174200+0000","last_clean":"2024-04-02T07:52:10.174200+0000","last_became_active":"2024-04-02T07:52:09.083959+0000","last_became_peered":"2024-04-02T07:52:09.083959+0000","last_unstale":"2024-04-02T07:52:10.174200+0000","last_undegraded":"2024-04-02T07:52:10.174200+0000","last_fullsized":"2024-04-02T07:52:10.174200+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:20:51.784990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371920+0000","last_change":"2024-04-02T07:52:09.089606+0000","last_active":"2024-04-02T07:52:10.371920+0000","last_peered":"2024-04-02T07:52:10.371920+0000","last_clean":"2024-04-02T07:52:10.371920+0000","last_became_active":"2024-04-02T07:52:09.089322+0000","last_became_peered":"2024-04-02T07:52:09.089322+0000","last_unstale":"2024-04-02T07:52:10.371920+0000","last_undegraded":"2024-04-02T07:52:10.371920+0000","last_fullsized":"2024-04-02T07:52:10.371920+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:15.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085757+0000","last_change":"2024-04-02T07:52:09.099115+0000","last_active":"2024-04-02T07:52:10.085757+0000","last_peered":"2024-04-02T07:52:10.085757+0000","last_clean":"2024-04-02T07:52:10.085757+0000","last_became_active":"2024-04-02T07:52:09.098804+0000","last_became_peered":"2024-04-02T07:52:09.098804+0000","last_unstale":"2024-04-02T07:52:10.085757+0000","last_undegraded":"2024-04-02T07:52:10.085757+0000","last_fullsized":"2024-04-02T07:52:10.085757+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:29:10.607669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174702+0000","last_change":"2024-04-02T07:52:09.103096+0000","last_active":"2024-04-02T07:52:10.174702+0000","last_peered":"2024-04-02T07:52:10.174702+0000","last_clean":"2024-04-02T07:52:10.174702+0000","last_became_active":"2024-04-02T07:52:09.102796+0000","last_became_peered":"2024-04-02T07:52:09.102796+0000","last_unstale":"2024-04-02T07:52:10.174702+0000","last_undegraded":"2024-04-02T07:52:10.174702+0000","last_fullsized":"2024-04-02T07:52:10.174702+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:58:29.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085926+0000","last_change":"2024-04-02T07:52:09.091339+0000","last_active":"2024-04-02T07:52:10.085926+0000","last_peered":"2024-04-02T07:52:10.085926+0000","last_clean":"2024-04-02T07:52:10.085926+0000","last_became_active":"2024-04-02T07:52:09.090479+0000","last_became_peered":"2024-04-02T07:52:09.090479+0000","last_unstale":"2024-04-02T07:52:10.085926+0000","last_undegraded":"2024-04-02T07:52:10.085926+0000","last_fullsized":"2024-04-02T07:52:10.085926+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:41:56.288382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371443+0000","last_change":"2024-04-02T07:52:09.099793+0000","last_active":"2024-04-02T07:52:10.371443+0000","last_peered":"2024-04-02T07:52:10.371443+0000","last_clean":"2024-04-02T07:52:10.371443+0000","last_became_active":"2024-04-02T07:52:09.099144+0000","last_became_peered":"2024-04-02T07:52:09.099144+0000","last_unstale":"2024-04-02T07:52:10.371443+0000","last_undegraded":"2024-04-02T07:52:10.371443+0000","last_fullsized":"2024-04-02T07:52:10.371443+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:37:15.846519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085933+0000","last_change":"2024-04-02T07:52:09.095802+0000","last_active":"2024-04-02T07:52:10.085933+0000","last_peered":"2024-04-02T07:52:10.085933+0000","last_clean":"2024-04-02T07:52:10.085933+0000","last_became_active":"2024-04-02T07:52:09.095527+0000","last_became_peered":"2024-04-02T07:52:09.095527+0000","last_unstale":"2024-04-02T07:52:10.085933+0000","last_undegraded":"2024-04-02T07:52:10.085933+0000","last_fullsized":"2024-04-02T07:52:10.085933+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:05:05.554358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085705+0000","last_change":"2024-04-02T07:52:09.081789+0000","last_active":"2024-04-02T07:52:10.085705+0000","last_peered":"2024-04-02T07:52:10.085705+0000","last_clean":"2024-04-02T07:52:10.085705+0000","last_became_active":"2024-04-02T07:52:09.081507+0000","last_became_peered":"2024-04-02T07:52:09.081507+0000","last_unstale":"2024-04-02T07:52:10.085705+0000","last_undegraded":"2024-04-02T07:52:10.085705+0000","last_fullsized":"2024-04-02T07:52:10.085705+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:58:55.222923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174198+0000","last_change":"2024-04-02T07:52:09.097717+0000","last_active":"2024-04-02T07:52:10.174198+0000","last_peered":"2024-04-02T07:52:10.174198+0000","last_clean":"2024-04-02T07:52:10.174198+0000","last_became_active":"2024-04-02T07:52:09.088955+0000","last_became_peered":"2024-04-02T07:52:09.088955+0000","last_unstale":"2024-04-02T07:52:10.174198+0000","last_undegraded":"2024-04-02T07:52:10.174198+0000","last_fullsized":"2024-04-02T07:52:10.174198+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:54:54.390550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371874+0000","last_change":"2024-04-02T07:52:09.088181+0000","last_active":"2024-04-02T07:52:10.371874+0000","last_peered":"2024-04-02T07:52:10.371874+0000","last_clean":"2024-04-02T07:52:10.371874+0000","last_became_active":"2024-04-02T07:52:09.087890+0000","last_became_peered":"2024-04-02T07:52:09.087890+0000","last_unstale":"2024-04-02T07:52:10.371874+0000","last_undegraded":"2024-04-02T07:52:10.371874+0000","last_fullsized":"2024-04-02T07:52:10.371874+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:20.566311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.372004+0000","last_change":"2024-04-02T07:52:09.104157+0000","last_active":"2024-04-02T07:52:10.372004+0000","last_peered":"2024-04-02T07:52:10.372004+0000","last_clean":"2024-04-02T07:52:10.372004+0000","last_became_active":"2024-04-02T07:52:09.103892+0000","last_became_peered":"2024-04-02T07:52:09.103892+0000","last_unstale":"2024-04-02T07:52:10.372004+0000","last_undegraded":"2024-04-02T07:52:10.372004+0000","last_fullsized":"2024-04-02T07:52:10.372004+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:32.841698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174704+0000","last_change":"2024-04-02T07:52:09.098739+0000","last_active":"2024-04-02T07:52:10.174704+0000","last_peered":"2024-04-02T07:52:10.174704+0000","last_clean":"2024-04-02T07:52:10.174704+0000","last_became_active":"2024-04-02T07:52:09.098482+0000","last_became_peered":"2024-04-02T07:52:09.098482+0000","last_unstale":"2024-04-02T07:52:10.174704+0000","last_undegraded":"2024-04-02T07:52:10.174704+0000","last_fullsized":"2024-04-02T07:52:10.174704+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:20.566311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371689+0000","last_change":"2024-04-02T07:52:09.085517+0000","last_active":"2024-04-02T07:52:10.371689+0000","last_peered":"2024-04-02T07:52:10.371689+0000","last_clean":"2024-04-02T07:52:10.371689+0000","last_became_active":"2024-04-02T07:52:09.084507+0000","last_became_peered":"2024-04-02T07:52:09.084507+0000","last_unstale":"2024-04-02T07:52:10.371689+0000","last_undegraded":"2024-04-02T07:52:10.371689+0000","last_fullsized":"2024-04-02T07:52:10.371689+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:21:34.806245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081018+0000","last_change":"2024-04-02T07:52:09.090136+0000","last_active":"2024-04-02T07:52:10.081018+0000","last_peered":"2024-04-02T07:52:10.081018+0000","last_clean":"2024-04-02T07:52:10.081018+0000","last_became_active":"2024-04-02T07:52:09.089666+0000","last_became_peered":"2024-04-02T07:52:09.089666+0000","last_unstale":"2024-04-02T07:52:10.081018+0000","last_undegraded":"2024-04-02T07:52:10.081018+0000","last_fullsized":"2024-04-02T07:52:10.081018+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:40.830548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208019+0000","last_change":"2024-04-02T07:52:09.080447+0000","last_active":"2024-04-02T07:52:10.208019+0000","last_peered":"2024-04-02T07:52:10.208019+0000","last_clean":"2024-04-02T07:52:10.208019+0000","last_became_active":"2024-04-02T07:52:09.080193+0000","last_became_peered":"2024-04-02T07:52:09.080193+0000","last_unstale":"2024-04-02T07:52:10.208019+0000","last_undegraded":"2024-04-02T07:52:10.208019+0000","last_fullsized":"2024-04-02T07:52:10.208019+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:28:00.296862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085378+0000","last_change":"2024-04-02T07:52:09.097706+0000","last_active":"2024-04-02T07:52:10.085378+0000","last_peered":"2024-04-02T07:52:10.085378+0000","last_clean":"2024-04-02T07:52:10.085378+0000","last_became_active":"2024-04-02T07:52:09.097376+0000","last_became_peered":"2024-04-02T07:52:09.097376+0000","last_unstale":"2024-04-02T07:52:10.085378+0000","last_undegraded":"2024-04-02T07:52:10.085378+0000","last_fullsized":"2024-04-02T07:52:10.085378+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:23:52.181861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174319+0000","last_change":"2024-04-02T07:52:09.083281+0000","last_active":"2024-04-02T07:52:10.174319+0000","last_peered":"2024-04-02T07:52:10.174319+0000","last_clean":"2024-04-02T07:52:10.174319+0000","last_became_active":"2024-04-02T07:52:09.083001+0000","last_became_peered":"2024-04-02T07:52:09.083001+0000","last_unstale":"2024-04-02T07:52:10.174319+0000","last_undegraded":"2024-04-02T07:52:10.174319+0000","last_fullsized":"2024-04-02T07:52:10.174319+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:21:45.009281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085623+0000","last_change":"2024-04-02T07:52:09.096280+0000","last_active":"2024-04-02T07:52:10.085623+0000","last_peered":"2024-04-02T07:52:10.085623+0000","last_clean":"2024-04-02T07:52:10.085623+0000","last_became_active":"2024-04-02T07:52:09.095975+0000","last_became_peered":"2024-04-02T07:52:09.095975+0000","last_unstale":"2024-04-02T07:52:10.085623+0000","last_undegraded":"2024-04-02T07:52:10.085623+0000","last_fullsized":"2024-04-02T07:52:10.085623+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:35:41.761205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174484+0000","last_change":"2024-04-02T07:52:09.107913+0000","last_active":"2024-04-02T07:52:10.174484+0000","last_peered":"2024-04-02T07:52:10.174484+0000","last_clean":"2024-04-02T07:52:10.174484+0000","last_became_active":"2024-04-02T07:52:09.107639+0000","last_became_peered":"2024-04-02T07:52:09.107639+0000","last_unstale":"2024-04-02T07:52:10.174484+0000","last_undegraded":"2024-04-02T07:52:10.174484+0000","last_fullsized":"2024-04-02T07:52:10.174484+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:39:13.805896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371738+0000","last_change":"2024-04-02T07:52:09.096147+0000","last_active":"2024-04-02T07:52:10.371738+0000","last_peered":"2024-04-02T07:52:10.371738+0000","last_clean":"2024-04-02T07:52:10.371738+0000","last_became_active":"2024-04-02T07:52:09.093917+0000","last_became_peered":"2024-04-02T07:52:09.093917+0000","last_unstale":"2024-04-02T07:52:10.371738+0000","last_undegraded":"2024-04-02T07:52:10.371738+0000","last_fullsized":"2024-04-02T07:52:10.371738+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:08:37.815656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.080801+0000","last_change":"2024-04-02T07:52:09.097264+0000","last_active":"2024-04-02T07:52:10.080801+0000","last_peered":"2024-04-02T07:52:10.080801+0000","last_clean":"2024-04-02T07:52:10.080801+0000","last_became_active":"2024-04-02T07:52:09.096914+0000","last_became_peered":"2024-04-02T07:52:09.096914+0000","last_unstale":"2024-04-02T07:52:10.080801+0000","last_undegraded":"2024-04-02T07:52:10.080801+0000","last_fullsized":"2024-04-02T07:52:10.080801+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:48:43.137803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371550+0000","last_change":"2024-04-02T07:52:09.104361+0000","last_active":"2024-04-02T07:52:10.371550+0000","last_peered":"2024-04-02T07:52:10.371550+0000","last_clean":"2024-04-02T07:52:10.371550+0000","last_became_active":"2024-04-02T07:52:09.104057+0000","last_became_peered":"2024-04-02T07:52:09.104057+0000","last_unstale":"2024-04-02T07:52:10.371550+0000","last_undegraded":"2024-04-02T07:52:10.371550+0000","last_fullsized":"2024-04-02T07:52:10.371550+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:13.811440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174413+0000","last_change":"2024-04-02T07:52:09.100074+0000","last_active":"2024-04-02T07:52:10.174413+0000","last_peered":"2024-04-02T07:52:10.174413+0000","last_clean":"2024-04-02T07:52:10.174413+0000","last_became_active":"2024-04-02T07:52:09.099847+0000","last_became_peered":"2024-04-02T07:52:09.099847+0000","last_unstale":"2024-04-02T07:52:10.174413+0000","last_undegraded":"2024-04-02T07:52:10.174413+0000","last_fullsized":"2024-04-02T07:52:10.174413+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:43.287818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081062+0000","last_change":"2024-04-02T07:52:09.094190+0000","last_active":"2024-04-02T07:52:10.081062+0000","last_peered":"2024-04-02T07:52:10.081062+0000","last_clean":"2024-04-02T07:52:10.081062+0000","last_became_active":"2024-04-02T07:52:09.093702+0000","last_became_peered":"2024-04-02T07:52:09.093702+0000","last_unstale":"2024-04-02T07:52:10.081062+0000","last_undegraded":"2024-04-02T07:52:10.081062+0000","last_fullsized":"2024-04-02T07:52:10.081062+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:11:41.093903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.080811+0000","last_change":"2024-04-02T07:52:09.096817+0000","last_active":"2024-04-02T07:52:10.080811+0000","last_peered":"2024-04-02T07:52:10.080811+0000","last_clean":"2024-04-02T07:52:10.080811+0000","last_became_active":"2024-04-02T07:52:09.094257+0000","last_became_peered":"2024-04-02T07:52:09.094257+0000","last_unstale":"2024-04-02T07:52:10.080811+0000","last_undegraded":"2024-04-02T07:52:10.080811+0000","last_fullsized":"2024-04-02T07:52:10.080811+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:06:33.391572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174542+0000","last_change":"2024-04-02T07:52:09.099100+0000","last_active":"2024-04-02T07:52:10.174542+0000","last_peered":"2024-04-02T07:52:10.174542+0000","last_clean":"2024-04-02T07:52:10.174542+0000","last_became_active":"2024-04-02T07:52:09.098341+0000","last_became_peered":"2024-04-02T07:52:09.098341+0000","last_unstale":"2024-04-02T07:52:10.174542+0000","last_undegraded":"2024-04-02T07:52:10.174542+0000","last_fullsized":"2024-04-02T07:52:10.174542+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:08:37.815656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"96'4","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208004+0000","last_change":"2024-04-02T07:52:08.581911+0000","last_active":"2024-04-02T07:52:10.208004+0000","last_peered":"2024-04-02T07:52:10.208004+0000","last_clean":"2024-04-02T07:52:10.208004+0000","last_became_active":"2024-04-02T07:52:08.580945+0000","last_became_peered":"2024-04-02T07:52:08.580945+0000","last_unstale":"2024-04-02T07:52:10.208004+0000","last_undegraded":"2024-04-02T07:52:10.208004+0000","last_fullsized":"2024-04-02T07:52:10.208004+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:45:13.517646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.372012+0000","last_change":"2024-04-02T07:52:09.095472+0000","last_active":"2024-04-02T07:52:10.372012+0000","last_peered":"2024-04-02T07:52:10.372012+0000","last_clean":"2024-04-02T07:52:10.372012+0000","last_became_active":"2024-04-02T07:52:09.093337+0000","last_became_peered":"2024-04-02T07:52:09.093337+0000","last_unstale":"2024-04-02T07:52:10.372012+0000","last_undegraded":"2024-04-02T07:52:10.372012+0000","last_fullsized":"2024-04-02T07:52:10.372012+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:45:14.526108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1e","version":"96'1","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.372061+0000","last_change":"2024-04-02T07:52:08.590194+0000","last_active":"2024-04-02T07:52:10.372061+0000","last_peered":"2024-04-02T07:52:10.372061+0000","last_clean":"2024-04-02T07:52:10.372061+0000","last_became_active":"2024-04-02T07:52:08.588845+0000","last_became_peered":"2024-04-02T07:52:08.588845+0000","last_unstale":"2024-04-02T07:52:10.372061+0000","last_undegraded":"2024-04-02T07:52:10.372061+0000","last_fullsized":"2024-04-02T07:52:10.372061+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:32:56.123575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.207985+0000","last_change":"2024-04-02T07:52:09.107829+0000","last_active":"2024-04-02T07:52:10.207985+0000","last_peered":"2024-04-02T07:52:10.207985+0000","last_clean":"2024-04-02T07:52:10.207985+0000","last_became_active":"2024-04-02T07:52:09.107559+0000","last_became_peered":"2024-04-02T07:52:09.107559+0000","last_unstale":"2024-04-02T07:52:10.207985+0000","last_undegraded":"2024-04-02T07:52:10.207985+0000","last_fullsized":"2024-04-02T07:52:10.207985+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:10.755054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"96'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174757+0000","last_change":"2024-04-02T07:52:08.590593+0000","last_active":"2024-04-02T07:52:10.174757+0000","last_peered":"2024-04-02T07:52:10.174757+0000","last_clean":"2024-04-02T07:52:10.174757+0000","last_became_active":"2024-04-02T07:52:08.590348+0000","last_became_peered":"2024-04-02T07:52:08.590348+0000","last_unstale":"2024-04-02T07:52:10.174757+0000","last_undegraded":"2024-04-02T07:52:10.174757+0000","last_fullsized":"2024-04-02T07:52:10.174757+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:13:26.420771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085586+0000","last_change":"2024-04-02T07:52:09.102786+0000","last_active":"2024-04-02T07:52:10.085586+0000","last_peered":"2024-04-02T07:52:10.085586+0000","last_clean":"2024-04-02T07:52:10.085586+0000","last_became_active":"2024-04-02T07:52:09.102509+0000","last_became_peered":"2024-04-02T07:52:09.102509+0000","last_unstale":"2024-04-02T07:52:10.085586+0000","last_undegraded":"2024-04-02T07:52:10.085586+0000","last_fullsized":"2024-04-02T07:52:10.085586+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:20.302770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1c","version":"96'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208055+0000","last_change":"2024-04-02T07:52:08.589183+0000","last_active":"2024-04-02T07:52:10.208055+0000","last_peered":"2024-04-02T07:52:10.208055+0000","last_clean":"2024-04-02T07:52:10.208055+0000","last_became_active":"2024-04-02T07:52:08.588879+0000","last_became_peered":"2024-04-02T07:52:08.588879+0000","last_unstale":"2024-04-02T07:52:10.208055+0000","last_undegraded":"2024-04-02T07:52:10.208055+0000","last_fullsized":"2024-04-02T07:52:10.208055+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:32:37.852523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174318+0000","last_change":"2024-04-02T07:52:09.090582+0000","last_active":"2024-04-02T07:52:10.174318+0000","last_peered":"2024-04-02T07:52:10.174318+0000","last_clean":"2024-04-02T07:52:10.174318+0000","last_became_active":"2024-04-02T07:52:09.090330+0000","last_became_peered":"2024-04-02T07:52:09.090330+0000","last_unstale":"2024-04-02T07:52:10.174318+0000","last_undegraded":"2024-04-02T07:52:10.174318+0000","last_fullsized":"2024-04-02T07:52:10.174318+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:33.887916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"96'1","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.086029+0000","last_change":"2024-04-02T07:52:08.579847+0000","last_active":"2024-04-02T07:52:10.086029+0000","last_peered":"2024-04-02T07:52:10.086029+0000","last_clean":"2024-04-02T07:52:10.086029+0000","last_became_active":"2024-04-02T07:52:08.579586+0000","last_became_peered":"2024-04-02T07:52:08.579586+0000","last_unstale":"2024-04-02T07:52:10.086029+0000","last_undegraded":"2024-04-02T07:52:10.086029+0000","last_fullsized":"2024-04-02T07:52:10.086029+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:32:11.605975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208073+0000","last_change":"2024-04-02T07:52:09.100358+0000","last_active":"2024-04-02T07:52:10.208073+0000","last_peered":"2024-04-02T07:52:10.208073+0000","last_clean":"2024-04-02T07:52:10.208073+0000","last_became_active":"2024-04-02T07:52:09.099806+0000","last_became_peered":"2024-04-02T07:52:09.099806+0000","last_unstale":"2024-04-02T07:52:10.208073+0000","last_undegraded":"2024-04-02T07:52:10.208073+0000","last_fullsized":"2024-04-02T07:52:10.208073+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:35:41.761205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1a","version":"96'1","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085713+0000","last_change":"2024-04-02T07:52:08.591780+0000","last_active":"2024-04-02T07:52:10.085713+0000","last_peered":"2024-04-02T07:52:10.085713+0000","last_clean":"2024-04-02T07:52:10.085713+0000","last_became_active":"2024-04-02T07:52:08.588742+0000","last_became_peered":"2024-04-02T07:52:08.588742+0000","last_unstale":"2024-04-02T07:52:10.085713+0000","last_undegraded":"2024-04-02T07:52:10.085713+0000","last_fullsized":"2024-04-02T07:52:10.085713+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:21.289613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.370700+0000","last_change":"2024-04-02T07:52:09.097716+0000","last_active":"2024-04-02T07:52:10.370700+0000","last_peered":"2024-04-02T07:52:10.370700+0000","last_clean":"2024-04-02T07:52:10.370700+0000","last_became_active":"2024-04-02T07:52:09.097466+0000","last_became_peered":"2024-04-02T07:52:09.097466+0000","last_unstale":"2024-04-02T07:52:10.370700+0000","last_undegraded":"2024-04-02T07:52:10.370700+0000","last_fullsized":"2024-04-02T07:52:10.370700+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:45:14.526108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371087+0000","last_change":"2024-04-02T07:52:08.595222+0000","last_active":"2024-04-02T07:52:10.371087+0000","last_peered":"2024-04-02T07:52:10.371087+0000","last_clean":"2024-04-02T07:52:10.371087+0000","last_became_active":"2024-04-02T07:52:08.594888+0000","last_became_peered":"2024-04-02T07:52:08.594888+0000","last_unstale":"2024-04-02T07:52:10.371087+0000","last_undegraded":"2024-04-02T07:52:10.371087+0000","last_fullsized":"2024-04-02T07:52:10.371087+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:47:14.750072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085515+0000","last_change":"2024-04-02T07:52:09.104975+0000","last_active":"2024-04-02T07:52:10.085515+0000","last_peered":"2024-04-02T07:52:10.085515+0000","last_clean":"2024-04-02T07:52:10.085515+0000","last_became_active":"2024-04-02T07:52:09.104691+0000","last_became_peered":"2024-04-02T07:52:09.104691+0000","last_unstale":"2024-04-02T07:52:10.085515+0000","last_undegraded":"2024-04-02T07:52:10.085515+0000","last_fullsized":"2024-04-02T07:52:10.085515+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:10:42.388029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085449+0000","last_change":"2024-04-02T07:52:08.579443+0000","last_active":"2024-04-02T07:52:10.085449+0000","last_peered":"2024-04-02T07:52:10.085449+0000","last_clean":"2024-04-02T07:52:10.085449+0000","last_became_active":"2024-04-02T07:52:08.576729+0000","last_became_peered":"2024-04-02T07:52:08.576729+0000","last_unstale":"2024-04-02T07:52:10.085449+0000","last_undegraded":"2024-04-02T07:52:10.085449+0000","last_fullsized":"2024-04-02T07:52:10.085449+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:29:09.599207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085851+0000","last_change":"2024-04-02T07:52:09.103166+0000","last_active":"2024-04-02T07:52:10.085851+0000","last_peered":"2024-04-02T07:52:10.085851+0000","last_clean":"2024-04-02T07:52:10.085851+0000","last_became_active":"2024-04-02T07:52:09.101914+0000","last_became_peered":"2024-04-02T07:52:09.101914+0000","last_unstale":"2024-04-02T07:52:10.085851+0000","last_undegraded":"2024-04-02T07:52:10.085851+0000","last_fullsized":"2024-04-02T07:52:10.085851+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:31:30.741833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174662+0000","last_change":"2024-04-02T07:52:08.587968+0000","last_active":"2024-04-02T07:52:10.174662+0000","last_peered":"2024-04-02T07:52:10.174662+0000","last_clean":"2024-04-02T07:52:10.174662+0000","last_became_active":"2024-04-02T07:52:08.587729+0000","last_became_peered":"2024-04-02T07:52:08.587729+0000","last_unstale":"2024-04-02T07:52:10.174662+0000","last_undegraded":"2024-04-02T07:52:10.174662+0000","last_fullsized":"2024-04-02T07:52:10.174662+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:32:56.123575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081104+0000","last_change":"2024-04-02T07:52:09.095465+0000","last_active":"2024-04-02T07:52:10.081104+0000","last_peered":"2024-04-02T07:52:10.081104+0000","last_clean":"2024-04-02T07:52:10.081104+0000","last_became_active":"2024-04-02T07:52:09.095217+0000","last_became_peered":"2024-04-02T07:52:09.095217+0000","last_unstale":"2024-04-02T07:52:10.081104+0000","last_undegraded":"2024-04-02T07:52:10.081104+0000","last_fullsized":"2024-04-02T07:52:10.081104+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:30:44.307779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081213+0000","last_change":"2024-04-02T07:52:08.587695+0000","last_active":"2024-04-02T07:52:10.081213+0000","last_peered":"2024-04-02T07:52:10.081213+0000","last_clean":"2024-04-02T07:52:10.081213+0000","last_became_active":"2024-04-02T07:52:08.587413+0000","last_became_peered":"2024-04-02T07:52:08.587413+0000","last_unstale":"2024-04-02T07:52:10.081213+0000","last_undegraded":"2024-04-02T07:52:10.081213+0000","last_fullsized":"2024-04-02T07:52:10.081213+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:08:46.492307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174470+0000","last_change":"2024-04-02T07:52:09.108705+0000","last_active":"2024-04-02T07:52:10.174470+0000","last_peered":"2024-04-02T07:52:10.174470+0000","last_clean":"2024-04-02T07:52:10.174470+0000","last_became_active":"2024-04-02T07:52:09.107625+0000","last_became_peered":"2024-04-02T07:52:09.107625+0000","last_unstale":"2024-04-02T07:52:10.174470+0000","last_undegraded":"2024-04-02T07:52:10.174470+0000","last_fullsized":"2024-04-02T07:52:10.174470+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:17.505215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"96'3","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085827+0000","last_change":"2024-04-02T07:52:08.587926+0000","last_active":"2024-04-02T07:52:10.085827+0000","last_peered":"2024-04-02T07:52:10.085827+0000","last_clean":"2024-04-02T07:52:10.085827+0000","last_became_active":"2024-04-02T07:52:08.586936+0000","last_became_peered":"2024-04-02T07:52:08.586936+0000","last_unstale":"2024-04-02T07:52:10.085827+0000","last_undegraded":"2024-04-02T07:52:10.085827+0000","last_fullsized":"2024-04-02T07:52:10.085827+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:23:51.173399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371903+0000","last_change":"2024-04-02T07:52:09.096032+0000","last_active":"2024-04-02T07:52:10.371903+0000","last_peered":"2024-04-02T07:52:10.371903+0000","last_clean":"2024-04-02T07:52:10.371903+0000","last_became_active":"2024-04-02T07:52:09.095334+0000","last_became_peered":"2024-04-02T07:52:09.095334+0000","last_unstale":"2024-04-02T07:52:10.371903+0000","last_undegraded":"2024-04-02T07:52:10.371903+0000","last_fullsized":"2024-04-02T07:52:10.371903+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:43.287818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"96'6","reported_seq":17,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371621+0000","last_change":"2024-04-02T07:52:08.588906+0000","last_active":"2024-04-02T07:52:10.371621+0000","last_peered":"2024-04-02T07:52:10.371621+0000","last_clean":"2024-04-02T07:52:10.371621+0000","last_became_active":"2024-04-02T07:52:08.588557+0000","last_became_peered":"2024-04-02T07:52:08.588557+0000","last_unstale":"2024-04-02T07:52:10.371621+0000","last_undegraded":"2024-04-02T07:52:10.371621+0000","last_fullsized":"2024-04-02T07:52:10.371621+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:21.289613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208028+0000","last_change":"2024-04-02T07:52:09.083316+0000","last_active":"2024-04-02T07:52:10.208028+0000","last_peered":"2024-04-02T07:52:10.208028+0000","last_clean":"2024-04-02T07:52:10.208028+0000","last_became_active":"2024-04-02T07:52:09.083063+0000","last_became_peered":"2024-04-02T07:52:09.083063+0000","last_unstale":"2024-04-02T07:52:10.208028+0000","last_undegraded":"2024-04-02T07:52:10.208028+0000","last_fullsized":"2024-04-02T07:52:10.208028+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:12:20.302770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"96'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371681+0000","last_change":"2024-04-02T07:52:08.592021+0000","last_active":"2024-04-02T07:52:10.371681+0000","last_peered":"2024-04-02T07:52:10.371681+0000","last_clean":"2024-04-02T07:52:10.371681+0000","last_became_active":"2024-04-02T07:52:08.591399+0000","last_became_peered":"2024-04-02T07:52:08.591399+0000","last_unstale":"2024-04-02T07:52:10.371681+0000","last_undegraded":"2024-04-02T07:52:10.371681+0000","last_fullsized":"2024-04-02T07:52:10.371681+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:51.201737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085473+0000","last_change":"2024-04-02T07:52:09.108800+0000","last_active":"2024-04-02T07:52:10.085473+0000","last_peered":"2024-04-02T07:52:10.085473+0000","last_clean":"2024-04-02T07:52:10.085473+0000","last_became_active":"2024-04-02T07:52:09.108548+0000","last_became_peered":"2024-04-02T07:52:09.108548+0000","last_unstale":"2024-04-02T07:52:10.085473+0000","last_undegraded":"2024-04-02T07:52:10.085473+0000","last_fullsized":"2024-04-02T07:52:10.085473+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:31:29.266098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085478+0000","last_change":"2024-04-02T07:52:08.592667+0000","last_active":"2024-04-02T07:52:10.085478+0000","last_peered":"2024-04-02T07:52:10.085478+0000","last_clean":"2024-04-02T07:52:10.085478+0000","last_became_active":"2024-04-02T07:52:08.592373+0000","last_became_peered":"2024-04-02T07:52:08.592373+0000","last_unstale":"2024-04-02T07:52:10.085478+0000","last_undegraded":"2024-04-02T07:52:10.085478+0000","last_fullsized":"2024-04-02T07:52:10.085478+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:27.543483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.370821+0000","last_change":"2024-04-02T07:52:09.081544+0000","last_active":"2024-04-02T07:52:10.370821+0000","last_peered":"2024-04-02T07:52:10.370821+0000","last_clean":"2024-04-02T07:52:10.370821+0000","last_became_active":"2024-04-02T07:52:09.081258+0000","last_became_peered":"2024-04-02T07:52:09.081258+0000","last_unstale":"2024-04-02T07:52:10.370821+0000","last_undegraded":"2024-04-02T07:52:10.370821+0000","last_fullsized":"2024-04-02T07:52:10.370821+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:58:29.946952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085478+0000","last_change":"2024-04-02T07:52:08.597717+0000","last_active":"2024-04-02T07:52:10.085478+0000","last_peered":"2024-04-02T07:52:10.085478+0000","last_clean":"2024-04-02T07:52:10.085478+0000","last_became_active":"2024-04-02T07:52:08.597509+0000","last_became_peered":"2024-04-02T07:52:08.597509+0000","last_unstale":"2024-04-02T07:52:10.085478+0000","last_undegraded":"2024-04-02T07:52:10.085478+0000","last_fullsized":"2024-04-02T07:52:10.085478+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:08:35.958483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371066+0000","last_change":"2024-04-02T07:52:09.091579+0000","last_active":"2024-04-02T07:52:10.371066+0000","last_peered":"2024-04-02T07:52:10.371066+0000","last_clean":"2024-04-02T07:52:10.371066+0000","last_became_active":"2024-04-02T07:52:09.091320+0000","last_became_peered":"2024-04-02T07:52:09.091320+0000","last_unstale":"2024-04-02T07:52:10.371066+0000","last_undegraded":"2024-04-02T07:52:10.371066+0000","last_fullsized":"2024-04-02T07:52:10.371066+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:52:35.122693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.10","version":"96'8","reported_seq":19,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085255+0000","last_change":"2024-04-02T07:52:08.589916+0000","last_active":"2024-04-02T07:52:10.085255+0000","last_peered":"2024-04-02T07:52:10.085255+0000","last_clean":"2024-04-02T07:52:10.085255+0000","last_became_active":"2024-04-02T07:52:08.589115+0000","last_became_peered":"2024-04-02T07:52:08.589115+0000","last_unstale":"2024-04-02T07:52:10.085255+0000","last_undegraded":"2024-04-02T07:52:10.085255+0000","last_fullsized":"2024-04-02T07:52:10.085255+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:19:15.104550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081125+0000","last_change":"2024-04-02T07:52:09.082446+0000","last_active":"2024-04-02T07:52:10.081125+0000","last_peered":"2024-04-02T07:52:10.081125+0000","last_clean":"2024-04-02T07:52:10.081125+0000","last_became_active":"2024-04-02T07:52:09.082203+0000","last_became_peered":"2024-04-02T07:52:09.082203+0000","last_unstale":"2024-04-02T07:52:10.081125+0000","last_undegraded":"2024-04-02T07:52:10.081125+0000","last_fullsized":"2024-04-02T07:52:10.081125+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:56:28.091780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"96'1","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174755+0000","last_change":"2024-04-02T07:52:08.593143+0000","last_active":"2024-04-02T07:52:10.174755+0000","last_peered":"2024-04-02T07:52:10.174755+0000","last_clean":"2024-04-02T07:52:10.174755+0000","last_became_active":"2024-04-02T07:52:08.592516+0000","last_became_peered":"2024-04-02T07:52:08.592516+0000","last_unstale":"2024-04-02T07:52:10.174755+0000","last_undegraded":"2024-04-02T07:52:10.174755+0000","last_fullsized":"2024-04-02T07:52:10.174755+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:10:00.180401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081176+0000","last_change":"2024-04-02T07:52:09.087842+0000","last_active":"2024-04-02T07:52:10.081176+0000","last_peered":"2024-04-02T07:52:10.081176+0000","last_clean":"2024-04-02T07:52:10.081176+0000","last_became_active":"2024-04-02T07:52:09.087582+0000","last_became_peered":"2024-04-02T07:52:09.087582+0000","last_unstale":"2024-04-02T07:52:10.081176+0000","last_undegraded":"2024-04-02T07:52:10.081176+0000","last_fullsized":"2024-04-02T07:52:10.081176+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:50:40.257112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"96'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.081263+0000","last_change":"2024-04-02T07:52:08.595233+0000","last_active":"2024-04-02T07:52:10.081263+0000","last_peered":"2024-04-02T07:52:10.081263+0000","last_clean":"2024-04-02T07:52:10.081263+0000","last_became_active":"2024-04-02T07:52:08.594931+0000","last_became_peered":"2024-04-02T07:52:08.594931+0000","last_unstale":"2024-04-02T07:52:10.081263+0000","last_undegraded":"2024-04-02T07:52:10.081263+0000","last_fullsized":"2024-04-02T07:52:10.081263+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:41:55.279920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371499+0000","last_change":"2024-04-02T07:52:09.086089+0000","last_active":"2024-04-02T07:52:10.371499+0000","last_peered":"2024-04-02T07:52:10.371499+0000","last_clean":"2024-04-02T07:52:10.371499+0000","last_became_active":"2024-04-02T07:52:09.085661+0000","last_became_peered":"2024-04-02T07:52:09.085661+0000","last_unstale":"2024-04-02T07:52:10.371499+0000","last_undegraded":"2024-04-02T07:52:10.371499+0000","last_fullsized":"2024-04-02T07:52:10.371499+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:32:38.860985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"96'1","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.080980+0000","last_change":"2024-04-02T07:52:08.594106+0000","last_active":"2024-04-02T07:52:10.080980+0000","last_peered":"2024-04-02T07:52:10.080980+0000","last_clean":"2024-04-02T07:52:10.080980+0000","last_became_active":"2024-04-02T07:52:08.593089+0000","last_became_peered":"2024-04-02T07:52:08.593089+0000","last_unstale":"2024-04-02T07:52:10.080980+0000","last_undegraded":"2024-04-02T07:52:10.080980+0000","last_fullsized":"2024-04-02T07:52:10.080980+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:31:06.509089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371415+0000","last_change":"2024-04-02T07:52:09.104377+0000","last_active":"2024-04-02T07:52:10.371415+0000","last_peered":"2024-04-02T07:52:10.371415+0000","last_clean":"2024-04-02T07:52:10.371415+0000","last_became_active":"2024-04-02T07:52:09.103930+0000","last_became_peered":"2024-04-02T07:52:09.103930+0000","last_unstale":"2024-04-02T07:52:10.371415+0000","last_undegraded":"2024-04-02T07:52:10.371415+0000","last_fullsized":"2024-04-02T07:52:10.371415+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:53:48.468830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208083+0000","last_change":"2024-04-02T07:52:08.583436+0000","last_active":"2024-04-02T07:52:10.208083+0000","last_peered":"2024-04-02T07:52:10.208083+0000","last_clean":"2024-04-02T07:52:10.208083+0000","last_became_active":"2024-04-02T07:52:08.582022+0000","last_became_peered":"2024-04-02T07:52:08.582022+0000","last_unstale":"2024-04-02T07:52:10.208083+0000","last_undegraded":"2024-04-02T07:52:10.208083+0000","last_fullsized":"2024-04-02T07:52:10.208083+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:19:12.802978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371583+0000","last_change":"2024-04-02T07:52:09.097397+0000","last_active":"2024-04-02T07:52:10.371583+0000","last_peered":"2024-04-02T07:52:10.371583+0000","last_clean":"2024-04-02T07:52:10.371583+0000","last_became_active":"2024-04-02T07:52:09.097093+0000","last_became_peered":"2024-04-02T07:52:09.097093+0000","last_unstale":"2024-04-02T07:52:10.371583+0000","last_undegraded":"2024-04-02T07:52:10.371583+0000","last_fullsized":"2024-04-02T07:52:10.371583+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:29:10.607669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208149+0000","last_change":"2024-04-02T07:52:08.580244+0000","last_active":"2024-04-02T07:52:10.208149+0000","last_peered":"2024-04-02T07:52:10.208149+0000","last_clean":"2024-04-02T07:52:10.208149+0000","last_became_active":"2024-04-02T07:52:08.579997+0000","last_became_peered":"2024-04-02T07:52:08.579997+0000","last_unstale":"2024-04-02T07:52:10.208149+0000","last_undegraded":"2024-04-02T07:52:10.208149+0000","last_fullsized":"2024-04-02T07:52:10.208149+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:53:47.460368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.370806+0000","last_change":"2024-04-02T07:52:09.091632+0000","last_active":"2024-04-02T07:52:10.370806+0000","last_peered":"2024-04-02T07:52:10.370806+0000","last_clean":"2024-04-02T07:52:10.370806+0000","last_became_active":"2024-04-02T07:52:09.089418+0000","last_became_peered":"2024-04-02T07:52:09.089418+0000","last_unstale":"2024-04-02T07:52:10.370806+0000","last_undegraded":"2024-04-02T07:52:10.370806+0000","last_fullsized":"2024-04-02T07:52:10.370806+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:15.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.080865+0000","last_change":"2024-04-02T07:52:08.590287+0000","last_active":"2024-04-02T07:52:10.080865+0000","last_peered":"2024-04-02T07:52:10.080865+0000","last_clean":"2024-04-02T07:52:10.080865+0000","last_became_active":"2024-04-02T07:52:08.589813+0000","last_became_peered":"2024-04-02T07:52:08.589813+0000","last_unstale":"2024-04-02T07:52:10.080865+0000","last_undegraded":"2024-04-02T07:52:10.080865+0000","last_fullsized":"2024-04-02T07:52:10.080865+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:36.132695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085664+0000","last_change":"2024-04-02T07:52:09.088271+0000","last_active":"2024-04-02T07:52:10.085664+0000","last_peered":"2024-04-02T07:52:10.085664+0000","last_clean":"2024-04-02T07:52:10.085664+0000","last_became_active":"2024-04-02T07:52:09.087944+0000","last_became_peered":"2024-04-02T07:52:09.087944+0000","last_unstale":"2024-04-02T07:52:10.085664+0000","last_undegraded":"2024-04-02T07:52:10.085664+0000","last_fullsized":"2024-04-02T07:52:10.085664+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:37.141157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.370778+0000","last_change":"2024-04-02T07:52:08.590519+0000","last_active":"2024-04-02T07:52:10.370778+0000","last_peered":"2024-04-02T07:52:10.370778+0000","last_clean":"2024-04-02T07:52:10.370778+0000","last_became_active":"2024-04-02T07:52:08.589950+0000","last_became_peered":"2024-04-02T07:52:08.589950+0000","last_unstale":"2024-04-02T07:52:10.370778+0000","last_undegraded":"2024-04-02T07:52:10.370778+0000","last_fullsized":"2024-04-02T07:52:10.370778+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:49:34.574435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.372061+0000","last_change":"2024-04-02T07:52:09.095415+0000","last_active":"2024-04-02T07:52:10.372061+0000","last_peered":"2024-04-02T07:52:10.372061+0000","last_clean":"2024-04-02T07:52:10.372061+0000","last_became_active":"2024-04-02T07:52:09.094034+0000","last_became_peered":"2024-04-02T07:52:09.094034+0000","last_unstale":"2024-04-02T07:52:10.372061+0000","last_undegraded":"2024-04-02T07:52:10.372061+0000","last_fullsized":"2024-04-02T07:52:10.372061+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:50:09.238288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"96'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174779+0000","last_change":"2024-04-02T07:52:08.593299+0000","last_active":"2024-04-02T07:52:10.174779+0000","last_peered":"2024-04-02T07:52:10.174779+0000","last_clean":"2024-04-02T07:52:10.174779+0000","last_became_active":"2024-04-02T07:52:08.592890+0000","last_became_peered":"2024-04-02T07:52:08.592890+0000","last_unstale":"2024-04-02T07:52:10.174779+0000","last_undegraded":"2024-04-02T07:52:10.174779+0000","last_fullsized":"2024-04-02T07:52:10.174779+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:49:34.574435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085651+0000","last_change":"2024-04-02T07:52:09.095722+0000","last_active":"2024-04-02T07:52:10.085651+0000","last_peered":"2024-04-02T07:52:10.085651+0000","last_clean":"2024-04-02T07:52:10.085651+0000","last_became_active":"2024-04-02T07:52:09.094015+0000","last_became_peered":"2024-04-02T07:52:09.094015+0000","last_unstale":"2024-04-02T07:52:10.085651+0000","last_undegraded":"2024-04-02T07:52:10.085651+0000","last_fullsized":"2024-04-02T07:52:10.085651+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:37:15.846519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174591+0000","last_change":"2024-04-02T07:52:08.596215+0000","last_active":"2024-04-02T07:52:10.174591+0000","last_peered":"2024-04-02T07:52:10.174591+0000","last_clean":"2024-04-02T07:52:10.174591+0000","last_became_active":"2024-04-02T07:52:08.595949+0000","last_became_peered":"2024-04-02T07:52:08.595949+0000","last_unstale":"2024-04-02T07:52:10.174591+0000","last_undegraded":"2024-04-02T07:52:10.174591+0000","last_fullsized":"2024-04-02T07:52:10.174591+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:39.822086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.080943+0000","last_change":"2024-04-02T07:52:09.084594+0000","last_active":"2024-04-02T07:52:10.080943+0000","last_peered":"2024-04-02T07:52:10.080943+0000","last_clean":"2024-04-02T07:52:10.080943+0000","last_became_active":"2024-04-02T07:52:09.083527+0000","last_became_peered":"2024-04-02T07:52:09.083527+0000","last_unstale":"2024-04-02T07:52:10.080943+0000","last_undegraded":"2024-04-02T07:52:10.080943+0000","last_fullsized":"2024-04-02T07:52:10.080943+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:17:07.389471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085995+0000","last_change":"2024-04-02T07:52:08.588758+0000","last_active":"2024-04-02T07:52:10.085995+0000","last_peered":"2024-04-02T07:52:10.085995+0000","last_clean":"2024-04-02T07:52:10.085995+0000","last_became_active":"2024-04-02T07:52:08.588484+0000","last_became_peered":"2024-04-02T07:52:08.588484+0000","last_unstale":"2024-04-02T07:52:10.085995+0000","last_undegraded":"2024-04-02T07:52:10.085995+0000","last_fullsized":"2024-04-02T07:52:10.085995+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:31:28.257636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085631+0000","last_change":"2024-04-02T07:52:09.096665+0000","last_active":"2024-04-02T07:52:10.085631+0000","last_peered":"2024-04-02T07:52:10.085631+0000","last_clean":"2024-04-02T07:52:10.085631+0000","last_became_active":"2024-04-02T07:52:09.095310+0000","last_became_peered":"2024-04-02T07:52:09.095310+0000","last_unstale":"2024-04-02T07:52:10.085631+0000","last_undegraded":"2024-04-02T07:52:10.085631+0000","last_fullsized":"2024-04-02T07:52:10.085631+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:32:12.614437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085781+0000","last_change":"2024-04-02T07:52:08.584788+0000","last_active":"2024-04-02T07:52:10.085781+0000","last_peered":"2024-04-02T07:52:10.085781+0000","last_clean":"2024-04-02T07:52:10.085781+0000","last_became_active":"2024-04-02T07:52:08.584513+0000","last_became_peered":"2024-04-02T07:52:08.584513+0000","last_unstale":"2024-04-02T07:52:10.085781+0000","last_undegraded":"2024-04-02T07:52:10.085781+0000","last_fullsized":"2024-04-02T07:52:10.085781+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:14.420847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371219+0000","last_change":"2024-04-02T07:52:09.089155+0000","last_active":"2024-04-02T07:52:10.371219+0000","last_peered":"2024-04-02T07:52:10.371219+0000","last_clean":"2024-04-02T07:52:10.371219+0000","last_became_active":"2024-04-02T07:52:09.088869+0000","last_became_peered":"2024-04-02T07:52:09.088869+0000","last_unstale":"2024-04-02T07:52:10.371219+0000","last_undegraded":"2024-04-02T07:52:10.371219+0000","last_fullsized":"2024-04-02T07:52:10.371219+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:35:04.350838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085980+0000","last_change":"2024-04-02T07:52:08.579841+0000","last_active":"2024-04-02T07:52:10.085980+0000","last_peered":"2024-04-02T07:52:10.085980+0000","last_clean":"2024-04-02T07:52:10.085980+0000","last_became_active":"2024-04-02T07:52:08.579164+0000","last_became_peered":"2024-04-02T07:52:08.579164+0000","last_unstale":"2024-04-02T07:52:10.085980+0000","last_undegraded":"2024-04-02T07:52:10.085980+0000","last_fullsized":"2024-04-02T07:52:10.085980+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:37:14.838057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.174523+0000","last_change":"2024-04-02T07:52:09.097930+0000","last_active":"2024-04-02T07:52:10.174523+0000","last_peered":"2024-04-02T07:52:10.174523+0000","last_clean":"2024-04-02T07:52:10.174523+0000","last_became_active":"2024-04-02T07:52:09.097615+0000","last_became_peered":"2024-04-02T07:52:09.097615+0000","last_unstale":"2024-04-02T07:52:10.174523+0000","last_undegraded":"2024-04-02T07:52:10.174523+0000","last_fullsized":"2024-04-02T07:52:10.174523+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:21:34.806245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371399+0000","last_change":"2024-04-02T07:52:08.592398+0000","last_active":"2024-04-02T07:52:10.371399+0000","last_peered":"2024-04-02T07:52:10.371399+0000","last_clean":"2024-04-02T07:52:10.371399+0000","last_became_active":"2024-04-02T07:52:08.592042+0000","last_became_peered":"2024-04-02T07:52:08.592042+0000","last_unstale":"2024-04-02T07:52:10.371399+0000","last_undegraded":"2024-04-02T07:52:10.371399+0000","last_fullsized":"2024-04-02T07:52:10.371399+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:10:00.180401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.370815+0000","last_change":"2024-04-02T07:52:09.093427+0000","last_active":"2024-04-02T07:52:10.370815+0000","last_peered":"2024-04-02T07:52:10.370815+0000","last_clean":"2024-04-02T07:52:10.370815+0000","last_became_active":"2024-04-02T07:52:09.093155+0000","last_became_peered":"2024-04-02T07:52:09.093155+0000","last_unstale":"2024-04-02T07:52:10.370815+0000","last_undegraded":"2024-04-02T07:52:10.370815+0000","last_fullsized":"2024-04-02T07:52:10.370815+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:52:35.122693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.080952+0000","last_change":"2024-04-02T07:52:08.587748+0000","last_active":"2024-04-02T07:52:10.080952+0000","last_peered":"2024-04-02T07:52:10.080952+0000","last_clean":"2024-04-02T07:52:10.080952+0000","last_became_active":"2024-04-02T07:52:08.587461+0000","last_became_peered":"2024-04-02T07:52:08.587461+0000","last_unstale":"2024-04-02T07:52:10.080952+0000","last_undegraded":"2024-04-02T07:52:10.080952+0000","last_fullsized":"2024-04-02T07:52:10.080952+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:27:59.288400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.080777+0000","last_change":"2024-04-02T07:52:09.107720+0000","last_active":"2024-04-02T07:52:10.080777+0000","last_peered":"2024-04-02T07:52:10.080777+0000","last_clean":"2024-04-02T07:52:10.080777+0000","last_became_active":"2024-04-02T07:52:09.106387+0000","last_became_peered":"2024-04-02T07:52:09.106387+0000","last_unstale":"2024-04-02T07:52:10.080777+0000","last_undegraded":"2024-04-02T07:52:10.080777+0000","last_fullsized":"2024-04-02T07:52:10.080777+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:28:11.709852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.208235+0000","last_change":"2024-04-02T07:52:08.585985+0000","last_active":"2024-04-02T07:52:10.208235+0000","last_peered":"2024-04-02T07:52:10.208235+0000","last_clean":"2024-04-02T07:52:10.208235+0000","last_became_active":"2024-04-02T07:52:08.585720+0000","last_became_peered":"2024-04-02T07:52:08.585720+0000","last_unstale":"2024-04-02T07:52:10.208235+0000","last_undegraded":"2024-04-02T07:52:10.208235+0000","last_fullsized":"2024-04-02T07:52:10.208235+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:52:34.114231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.371316+0000","last_change":"2024-04-02T07:52:09.084990+0000","last_active":"2024-04-02T07:52:10.371316+0000","last_peered":"2024-04-02T07:52:10.371316+0000","last_clean":"2024-04-02T07:52:10.371316+0000","last_became_active":"2024-04-02T07:52:09.084740+0000","last_became_peered":"2024-04-02T07:52:09.084740+0000","last_unstale":"2024-04-02T07:52:10.371316+0000","last_undegraded":"2024-04-02T07:52:10.371316+0000","last_fullsized":"2024-04-02T07:52:10.371316+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:08.563769+0000","last_clean_scrub_stamp":"2024-04-02T07:52:08.563769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:33.887916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"96'2","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-02T07:52:10.085452+0000","last_change":"2024-04-02T07:52:08.595290+0000","last_active":"2024-04-02T07:52:10.085452+0000","last_peered":"2024-04-02T07:52:10.085452+0000","last_clean":"2024-04-02T07:52:10.085452+0000","last_became_active":"2024-04-02T07:52:08.593652+0000","last_became_peered":"2024-04-02T07:52:08.593652+0000","last_unstale":"2024-04-02T07:52:10.085452+0000","last_undegraded":"2024-04-02T07:52:10.085452+0000","last_fullsized":"2024-04-02T07:52:10.085452+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:07.555307+0000","last_clean_scrub_stamp":"2024-04-02T07:52:07.555307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:46:35.968508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"86'172","reported_seq":433,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-02T07:52:09.085835+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:52:09.085835+0000","last_peered":"2024-04-02T07:52:09.085835+0000","last_clean":"2024-04-02T07:52:09.085835+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:52:09.085835+0000","last_undegraded":"2024-04-02T07:52:09.085835+0000","last_fullsized":"2024-04-02T07:52:09.085835+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575043,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1183744,"data_stored":571775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66800000000000004,"15min":0.67800000000000005},"min":{"1min":0.39400000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.67800000000000005},"min":{"1min":0.47399999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.95699999999999996,"5min":1.0109999999999999,"15min":1.024},"last":0.437}]},{"osd":1,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.67300000000000004,"15min":0.68000000000000005},"min":{"1min":0.499,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.89100000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66500000000000004,"15min":0.67500000000000004},"min":{"1min":0.42399999999999999,"5min":0.35599999999999998,"15min":0.34699999999999998},"max":{"1min":0.88300000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.62}]},{"osd":2,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.71699999999999997,"15min":0.71099999999999997},"min":{"1min":0.50700000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.94199999999999995,"5min":1.127,"15min":1.127},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.748,"5min":0.753,"15min":0.76000000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.434},"max":{"1min":0.93799999999999994,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.50800000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.69599999999999995,"15min":0.70399999999999996},"min":{"1min":0.44,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.97799999999999998,"5min":1.026,"15min":1.274},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.52600000000000002,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.90600000000000003,"5min":1.196,"15min":1.196},"last":0.59299999999999997}]},{"osd":4,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.376,"5min":0.32000000000000001,"15min":0.30199999999999999},"max":{"1min":0.84099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68200000000000005,"15min":0.68700000000000006},"min":{"1min":0.51800000000000002,"5min":0.40000000000000002,"15min":0.38100000000000001},"max":{"1min":0.84599999999999997,"5min":1.0669999999999999,"15min":1.4890000000000001},"last":0.52200000000000002}]},{"osd":5,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.46100000000000002,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":0.86299999999999999,"5min":1.167,"15min":1.167},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68500000000000005,"15min":0.68700000000000006},"min":{"1min":0.51100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.58099999999999996}]},{"osd":6,"last update":"Tue Apr 2 07:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67600000000000005,"15min":0.65800000000000003},"min":{"1min":0.505,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.877,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66000000000000003,"15min":0.65300000000000002},"min":{"1min":0.47999999999999998,"5min":0.42299999999999999,"15min":0.39600000000000002},"max":{"1min":0.89300000000000002,"5min":1.0509999999999999,"15min":1.329},"last":0.60899999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575041,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36396,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335444,"statfs":{"total":96636764160,"available":96599494656,"internally_reserved":0,"allocated":1200128,"data_stored":578457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.51300000000000001,"5min":0.38700000000000001,"15min":0.32400000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.98099999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.749,"5min":0.71599999999999997,"15min":0.70899999999999996},"min":{"1min":0.55200000000000005,"5min":0.38300000000000001,"15min":0.375},"max":{"1min":0.89400000000000002,"5min":0.96899999999999997,"15min":1.0249999999999999},"last":0.71299999999999997}]},{"osd":1,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69399999999999995,"15min":0.69599999999999995},"min":{"1min":0.53900000000000003,"5min":0.46500000000000002,"15min":0.41199999999999998},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":1.085},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.56999999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.752}]},{"osd":2,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73099999999999998,"15min":0.71499999999999997},"min":{"1min":0.48999999999999999,"5min":0.45500000000000002,"15min":0.38200000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71799999999999997,"15min":0.70699999999999996},"min":{"1min":0.53300000000000003,"5min":0.51400000000000001,"15min":0.45900000000000002},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96599999999999997},"last":0.53500000000000003}]},{"osd":3,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.73699999999999999,"15min":0.73899999999999999},"min":{"1min":0.55400000000000005,"5min":0.5,"15min":0.47099999999999997},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.748,"15min":0.73999999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.45300000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.0309999999999999},"last":0.68100000000000005}]},{"osd":4,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61399999999999999,"15min":0.60499999999999998},"min":{"1min":0.39300000000000002,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.64000000000000001,"15min":0.629},"min":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.35499999999999998},"max":{"1min":0.93200000000000005,"5min":0.99299999999999999,"15min":1.0049999999999999},"last":0.63200000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.629,"15min":0.621},"min":{"1min":0.41599999999999998,"5min":0.379,"15min":0.35799999999999998},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":1.1599999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.67200000000000004,"15min":0.68600000000000005},"min":{"1min":0.504,"5min":0.42099999999999999,"15min":0.40300000000000002},"max":{"1min":0.93899999999999995,"5min":1.01,"15min":1.095},"last":0.79300000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:51:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.68899999999999995,"15min":0.68600000000000005},"min":{"1min":0.48099999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.96199999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.45200000000000001,"5min":0.44,"15min":0.42099999999999999},"max":{"1min":0.92500000000000004,"5min":1.0149999999999999,"15min":1.4399999999999999},"last":0.61499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575042,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55004,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316836,"statfs":{"total":96636764160,"available":96580440064,"internally_reserved":0,"allocated":1777664,"data_stored":1167227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.432,"15min":0.44},"min":{"1min":0.29499999999999998,"5min":0.27500000000000002,"15min":0.25700000000000001},"max":{"1min":0.58899999999999997,"5min":0.77600000000000002,"15min":2.9079999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.50700000000000001,"15min":0.504},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.70699999999999996,"5min":0.86499999999999999,"15min":2.766},"last":0.27500000000000002}]},{"osd":2,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.627,"5min":0.71599999999999997,"15min":2.0750000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.443,"5min":0.36699999999999999,"15min":0.35699999999999998},"max":{"1min":0.71499999999999997,"5min":0.85399999999999998,"15min":1.7410000000000001},"last":0.45800000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.41799999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72399999999999998,"5min":0.81799999999999995,"15min":2.9430000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68000000000000005,"5min":0.82699999999999996,"15min":2.802},"last":0.35599999999999998}]},{"osd":4,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.78300000000000003,"15min":0.77400000000000002},"min":{"1min":0.67400000000000004,"5min":0.46700000000000003,"15min":0.46300000000000002},"max":{"1min":1.0609999999999999,"5min":1.861,"15min":1.861},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.77700000000000002,"15min":0.76800000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46500000000000002},"max":{"1min":1.232,"5min":1.232,"15min":2.0419999999999998},"last":0.69499999999999995}]},{"osd":5,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.77900000000000003,"15min":0.77100000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.41999999999999998},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.9690000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79400000000000004,"15min":0.78300000000000003},"min":{"1min":0.65600000000000003,"5min":0.51000000000000001,"15min":0.48799999999999999},"max":{"1min":1.147,"5min":1.54,"15min":1.54},"last":0.627}]},{"osd":6,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.79900000000000004,"15min":0.79500000000000004},"min":{"1min":0.53200000000000003,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.097,"5min":1.577,"15min":1.768},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.80400000000000005,"15min":0.79100000000000004},"min":{"1min":0.61399999999999999,"5min":0.51400000000000001,"15min":0.38100000000000001},"max":{"1min":1.351,"5min":1.351,"15min":2.012},"last":0.60899999999999999}]},{"osd":7,"last update":"Tue Apr 2 07:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.80900000000000005,"15min":0.80300000000000005},"min":{"1min":0.628,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.298,"5min":1.552,"15min":1.706},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.64300000000000002,"5min":0.53200000000000003,"15min":0.502},"max":{"1min":1.194,"5min":1.2290000000000001,"15min":1.8240000000000001},"last":0.64100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575042,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1183744,"data_stored":575910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.441,"15min":0.44400000000000001},"min":{"1min":0.29699999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71199999999999997,"5min":0.78900000000000003,"15min":2.1640000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.495,"15min":0.496},"min":{"1min":0.34899999999999998,"5min":0.307,"15min":0.26200000000000001},"max":{"1min":0.70599999999999996,"5min":0.72199999999999998,"15min":2.6469999999999998},"last":0.47299999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48999999999999999,"15min":0.495},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63100000000000001,"5min":0.749,"15min":2.125},"last":0.496},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":2.605},"last":0.51600000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52900000000000003,"15min":0.54800000000000004},"min":{"1min":0.33600000000000002,"5min":0.32700000000000001,"15min":0.28899999999999998},"max":{"1min":0.69799999999999995,"5min":0.79400000000000004,"15min":5.4800000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.40100000000000002},"max":{"1min":0.83199999999999996,"5min":0.93400000000000005,"15min":2.194},"last":0.52600000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.77700000000000002,"15min":0.80100000000000005},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.35599999999999998},"max":{"1min":0.97999999999999998,"5min":1.0640000000000001,"15min":6.0899999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78400000000000003,"15min":0.79600000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.433},"max":{"1min":1.048,"5min":1.095,"15min":5.7169999999999996},"last":0.84799999999999998}]},{"osd":5,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.81100000000000005,"15min":0.81200000000000006},"min":{"1min":0.55800000000000005,"5min":0.51000000000000001,"15min":0.42099999999999999},"max":{"1min":0.98999999999999999,"5min":2.1869999999999998,"15min":5.7539999999999996},"last":0.876},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.81799999999999995},"min":{"1min":0.63100000000000001,"5min":0.54100000000000004,"15min":0.51800000000000002},"max":{"1min":0.96299999999999997,"5min":2.6230000000000002,"15min":5.8929999999999998},"last":0.81599999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.75700000000000001,"15min":0.77100000000000002},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.46800000000000003},"max":{"1min":0.96599999999999997,"5min":1.044,"15min":5.931},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.755,"15min":0.77700000000000002},"min":{"1min":0.64700000000000002,"5min":0.50700000000000001,"15min":0.376},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":6.125},"last":0.76200000000000001}]},{"osd":7,"last update":"Tue Apr 2 07:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.83799999999999997},"min":{"1min":0.70299999999999996,"5min":0.51600000000000001,"15min":0.40799999999999997},"max":{"1min":1.0900000000000001,"5min":2.3959999999999999,"15min":6.1580000000000004},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.81399999999999995,"15min":0.83199999999999996},"min":{"1min":0.54100000000000004,"5min":0.48299999999999998,"15min":0.47699999999999998},"max":{"1min":1.0249999999999999,"5min":2.145,"15min":5.7850000000000001},"last":0.68700000000000006}]}]},{"osd":2,"up_from":13,"seq":55834575041,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36368,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335472,"statfs":{"total":96636764160,"available":96599523328,"internally_reserved":0,"allocated":1175552,"data_stored":570793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.44,"15min":0.44800000000000001},"min":{"1min":0.26700000000000002,"5min":0.26400000000000001,"15min":0.25700000000000001},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46300000000000002,"15min":0.47699999999999998},"min":{"1min":0.36199999999999999,"5min":0.27900000000000003,"15min":0.26200000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.52100000000000002}]},{"osd":1,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.49199999999999999,"15min":0.49099999999999999},"min":{"1min":0.33000000000000002,"5min":0.30599999999999999,"15min":0.30299999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.52400000000000002},"min":{"1min":0.39700000000000002,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.47899999999999998}]},{"osd":3,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.35999999999999999,"5min":0.316,"15min":0.27800000000000002},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59599999999999997,"15min":0.60999999999999999},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39000000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.54500000000000004}]},{"osd":4,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76300000000000001,"15min":0.77000000000000002},"min":{"1min":0.5,"5min":0.5,"15min":0.47199999999999998},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.748,"15min":0.77200000000000002},"min":{"1min":0.56499999999999995,"5min":0.51600000000000001,"15min":0.48499999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.76400000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.752,"15min":0.76100000000000001},"min":{"1min":0.56499999999999995,"5min":0.54300000000000004,"15min":0.42199999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.55600000000000005,"5min":0.53800000000000003,"15min":0.50900000000000001},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.81799999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.79400000000000004,"15min":0.79200000000000004},"min":{"1min":0.55000000000000004,"5min":0.53600000000000003,"15min":0.442},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.878},{"interface":"front","average":{"1min":0.752,"5min":0.75600000000000001,"15min":0.78700000000000003},"min":{"1min":0.53100000000000003,"5min":0.502,"15min":0.502},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.745}]},{"osd":7,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.77200000000000002,"15min":0.79500000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.5},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.77100000000000002,"15min":0.79300000000000004},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.70799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575041,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36380,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335460,"statfs":{"total":96636764160,"available":96599511040,"internally_reserved":0,"allocated":1196032,"data_stored":576985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.42399999999999999,"15min":0.42999999999999999},"min":{"1min":0.26500000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.72599999999999998,"5min":0.748,"15min":0.79600000000000004},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.435,"15min":0.46400000000000002},"min":{"1min":0.31,"5min":0.29699999999999999,"15min":0.28699999999999998},"max":{"1min":0.57099999999999995,"5min":0.70799999999999996,"15min":0.81899999999999995},"last":0.38800000000000001}]},{"osd":1,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.47899999999999998,"15min":0.49199999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61899999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.439,"5min":0.35799999999999998,"15min":0.34200000000000003},"max":{"1min":0.77500000000000002,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.624}]},{"osd":2,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59099999999999997,"15min":0.60699999999999998},"min":{"1min":0.439,"5min":0.38700000000000001,"15min":0.376},"max":{"1min":0.78400000000000003,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57699999999999996,"15min":0.60399999999999998},"min":{"1min":0.39500000000000002,"5min":0.38100000000000001,"15min":0.35899999999999999},"max":{"1min":0.70699999999999996,"5min":1.244,"15min":1.244},"last":0.64800000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.70899999999999996,"15min":0.73199999999999998},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.38400000000000001},"max":{"1min":0.83499999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73199999999999998,"15min":0.75700000000000001},"min":{"1min":0.52300000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":1.1759999999999999},"last":0.89000000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.72399999999999998,"15min":0.754},"min":{"1min":0.51300000000000001,"5min":0.41699999999999998,"15min":0.40899999999999997},"max":{"1min":0.97299999999999998,"5min":1.079,"15min":1.151},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72099999999999997,"15min":0.749},"min":{"1min":0.49199999999999999,"5min":0.48899999999999999,"15min":0.40200000000000002},"max":{"1min":0.94199999999999995,"5min":0.97599999999999998,"15min":1.1439999999999999},"last":0.79800000000000004}]},{"osd":6,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72199999999999998,"15min":0.76000000000000001},"min":{"1min":0.52800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.86899999999999999,"5min":0.98999999999999999,"15min":1.0620000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.748,"15min":0.77400000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.90400000000000003,"5min":1.1140000000000001,"15min":1.1899999999999999},"last":0.878}]},{"osd":7,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.73799999999999999,"15min":0.77000000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.439},"max":{"1min":0.91100000000000003,"5min":1.0409999999999999,"15min":1.0680000000000001},"last":0.751},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.75,"15min":0.78500000000000003},"min":{"1min":0.60299999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0089999999999999,"5min":1.181,"15min":1.181},"last":0.77000000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575041,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36376,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335464,"statfs":{"total":96636764160,"available":96599515136,"internally_reserved":0,"allocated":1183744,"data_stored":570905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.42499999999999999,"5min":0.38100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86599999999999999,"5min":0.91900000000000004,"15min":1.0189999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.66800000000000004,"15min":0.67500000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.879,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.50600000000000001}]},{"osd":1,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.71599999999999997,"15min":0.72799999999999998},"min":{"1min":0.47599999999999998,"5min":0.438,"15min":0.433},"max":{"1min":0.90300000000000002,"5min":0.97299999999999998,"15min":1.036},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.73799999999999999},"min":{"1min":0.58099999999999996,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.91900000000000004,"5min":0.97899999999999998,"15min":0.99399999999999999},"last":0.70999999999999996}]},{"osd":2,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.67400000000000004,"15min":0.69099999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.47299999999999998,"5min":0.47199999999999998,"15min":0.41899999999999998},"max":{"1min":0.874,"5min":0.97199999999999998,"15min":1.0620000000000001},"last":0.81299999999999994}]},{"osd":3,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72099999999999997},"min":{"1min":0.51500000000000001,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":0.95299999999999996,"5min":0.95699999999999996,"15min":1.002},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.71299999999999997,"15min":0.72999999999999998},"min":{"1min":0.49299999999999999,"5min":0.48799999999999999,"15min":0.41499999999999998},"max":{"1min":0.92100000000000004,"5min":0.93999999999999995,"15min":0.96999999999999997},"last":0.63200000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56200000000000006,"15min":0.57399999999999995},"min":{"1min":0.45800000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.746,"5min":0.84799999999999998,"15min":0.93999999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.621,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.434,"5min":0.38800000000000001,"15min":0.33700000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.56899999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.61199999999999999,"15min":0.60399999999999998},"min":{"1min":0.45200000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.51600000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.038},"last":0.85099999999999998}]},{"osd":7,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68000000000000005,"15min":0.66800000000000004},"min":{"1min":0.55700000000000005,"5min":0.432,"15min":0.38300000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.74099999999999999,"15min":0.72999999999999998},"min":{"1min":0.58999999999999997,"5min":0.47799999999999998,"15min":0.434},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.93300000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575044,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36948,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334892,"statfs":{"total":96636764160,"available":96598929408,"internally_reserved":0,"allocated":1777664,"data_stored":1166298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67800000000000005,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.80900000000000005,"5min":0.95299999999999996,"15min":0.99199999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.878,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.92700000000000005}]},{"osd":1,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67600000000000005,"15min":0.69399999999999995},"min":{"1min":0.41999999999999998,"5min":0.40300000000000002,"15min":0.40200000000000002},"max":{"1min":1.0620000000000001,"5min":1.131,"15min":1.131},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.45000000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83999999999999997,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.623}]},{"osd":2,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70699999999999996,"15min":0.71299999999999997},"min":{"1min":0.47399999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.878,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.69399999999999995,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.47099999999999997,"15min":0.372},"max":{"1min":1.111,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.82599999999999996}]},{"osd":3,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.72899999999999998,"15min":0.74299999999999999},"min":{"1min":0.54800000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92300000000000004,"5min":1.341,"15min":1.341},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77000000000000002,"15min":0.76600000000000001},"min":{"1min":0.60699999999999998,"5min":0.54600000000000004,"15min":0.44900000000000001},"max":{"1min":1.1839999999999999,"5min":1.484,"15min":1.484},"last":0.88100000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59299999999999997,"15min":0.59999999999999998},"min":{"1min":0.39500000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78700000000000003,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.35599999999999998},"max":{"1min":0.96199999999999997,"5min":1.504,"15min":1.504},"last":0.85299999999999998}]},{"osd":6,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.64900000000000002,"15min":0.64300000000000002},"min":{"1min":0.45600000000000002,"5min":0.44700000000000001,"15min":0.39900000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.69199999999999995,"15min":0.67700000000000005},"min":{"1min":0.45500000000000002,"5min":0.44900000000000001,"15min":0.41199999999999998},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":0.90300000000000002}]},{"osd":7,"last update":"Tue Apr 2 07:51:10 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.71399999999999997,"15min":0.70299999999999996},"min":{"1min":0.502,"5min":0.47599999999999998,"15min":0.38900000000000001},"max":{"1min":1.0129999999999999,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.754,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.53600000000000003,"5min":0.46300000000000002,"15min":0.434},"max":{"1min":1.149,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.98099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:52:13.941 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T07:52:13.941 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T07:52:14.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.173+0000 7febaa600700 1 Processor -- start 2024-04-02T07:52:14.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.174+0000 7febaa600700 1 -- start start 2024-04-02T07:52:14.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.175+0000 7febaa600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7feba415ef70 0x7feba40b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:14.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.175+0000 7febaa600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba4161150 con 0x7feba406a790 2024-04-02T07:52:14.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.175+0000 7febaa600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba406dba0 con 0x7feba4160d70 2024-04-02T07:52:14.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.175+0000 7febaa600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba40b4cd0 con 0x7feba415ef70 2024-04-02T07:52:14.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.175+0000 7feba3e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7feba406a790 0x7feba406e080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60392/0 (socket says 172.21.15.89:60392) 2024-04-02T07:52:14.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.175+0000 7feba3e00700 1 -- 172.21.15.89:0/2186825940 learned_addr learned my addr 172.21.15.89:0/2186825940 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:14.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.175+0000 7feba2a00700 1 --2- 172.21.15.89:0/2186825940 >> v2:172.21.15.113:3300/0 conn(0x7feba415ef70 0x7feba40b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:14.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.176+0000 7feba2000700 1 -- 172.21.15.89:0/2186825940 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714238319 0 0) 0x7feba4161150 con 0x7feba406a790 2024-04-02T07:52:14.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.176+0000 7feba2000700 1 -- 172.21.15.89:0/2186825940 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb880035b0 con 0x7feba406a790 2024-04-02T07:52:14.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.176+0000 7feba2a00700 1 -- 172.21.15.89:0/2186825940 >> v1:172.21.15.113:6789/0 conn(0x7feba4160d70 legacy=0x7feba415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:14.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.176+0000 7feba2000700 1 -- 172.21.15.89:0/2186825940 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629806263 0 0) 0x7feba406dba0 con 0x7feba4160d70 2024-04-02T07:52:14.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.176+0000 7feba2a00700 1 -- 172.21.15.89:0/2186825940 >> v1:172.21.15.89:6789/0 conn(0x7feba406a790 legacy=0x7feba406e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:14.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.176+0000 7feba2a00700 1 -- 172.21.15.89:0/2186825940 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba40b4ee0 con 0x7feba415ef70 2024-04-02T07:52:14.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.176+0000 7feba2a00700 1 --2- 172.21.15.89:0/2186825940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba415ef70 0x7feba40b3200 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7feb90009840 tx=0x7feb90009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61f3b09083cca7fd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:14.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.177+0000 7feba2000700 1 -- 172.21.15.89:0/2186825940 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb90011830 con 0x7feba415ef70 2024-04-02T07:52:14.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.177+0000 7feba2000700 1 -- 172.21.15.89:0/2186825940 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb90011990 con 0x7feba415ef70 2024-04-02T07:52:14.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.177+0000 7feba2000700 1 -- 172.21.15.89:0/2186825940 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb90011d20 con 0x7feba415ef70 2024-04-02T07:52:14.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.177+0000 7febaa600700 1 -- 172.21.15.89:0/2186825940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba415ef70 msgr2=0x7feba40b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:14.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.177+0000 7febaa600700 1 --2- 172.21.15.89:0/2186825940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba415ef70 0x7feba40b3200 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7feb90009840 tx=0x7feb90009b50 comp rx=0 tx=0).stop 2024-04-02T07:52:14.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.177+0000 7febaa600700 1 -- 172.21.15.89:0/2186825940 shutdown_connections 2024-04-02T07:52:14.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.177+0000 7febaa600700 1 --2- 172.21.15.89:0/2186825940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba415ef70 0x7feba40b3200 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:14.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.177+0000 7febaa600700 1 -- 172.21.15.89:0/2186825940 >> 172.21.15.89:0/2186825940 conn(0x7feba40b5720 msgr2=0x7feba40b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:14.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.177+0000 7febaa600700 1 -- 172.21.15.89:0/2186825940 shutdown_connections 2024-04-02T07:52:14.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.177+0000 7febaa600700 1 -- 172.21.15.89:0/2186825940 wait complete. 2024-04-02T07:52:14.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.179+0000 7febaa600700 1 Processor -- start 2024-04-02T07:52:14.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.179+0000 7febaa600700 1 -- start start 2024-04-02T07:52:14.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.180+0000 7febaa600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feba406a790 0x7feba4149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:14.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.180+0000 7febaa600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba415ef70 0x7feba40b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:14.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.180+0000 7febaa600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba4160d70 0x7feba415d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:14.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.180+0000 7febaa600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba41f2cc0 con 0x7feba415ef70 2024-04-02T07:52:14.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.180+0000 7febaa600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba414b950 con 0x7feba4160d70 2024-04-02T07:52:14.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.180+0000 7febaa600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba414bc30 con 0x7feba406a790 2024-04-02T07:52:14.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.180+0000 7feba2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba415ef70 0x7feba40b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:14.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.180+0000 7feba2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba415ef70 0x7feba40b1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42040/0 (socket says 172.21.15.89:42040) 2024-04-02T07:52:14.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.180+0000 7feba2a00700 1 -- 172.21.15.89:0/3578776901 learned_addr learned my addr 172.21.15.89:0/3578776901 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:14.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.181+0000 7feba3400700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feba406a790 0x7feba4149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:14.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.181+0000 7feba3e00700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba4160d70 0x7feba415d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:14.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.181+0000 7feba2a00700 1 -- 172.21.15.89:0/3578776901 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feba406a790 msgr2=0x7feba4149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:14.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.181+0000 7feba2a00700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feba406a790 0x7feba4149e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:14.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.181+0000 7feba2a00700 1 -- 172.21.15.89:0/3578776901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba4160d70 msgr2=0x7feba415d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:14.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.181+0000 7feba2a00700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba4160d70 0x7feba415d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:14.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.181+0000 7feba2a00700 1 -- 172.21.15.89:0/3578776901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb900094a0 con 0x7feba415ef70 2024-04-02T07:52:14.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.181+0000 7feba3e00700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba4160d70 0x7feba415d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:14.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.181+0000 7feba2a00700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba415ef70 0x7feba40b1d40 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7feb900090b0 tx=0x7feb90004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:14.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.181+0000 7feb9b400700 1 -- 172.21.15.89:0/3578776901 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9001a860 con 0x7feba415ef70 2024-04-02T07:52:14.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.181+0000 7feb9b400700 1 -- 172.21.15.89:0/3578776901 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9001a9c0 con 0x7feba415ef70 2024-04-02T07:52:14.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.181+0000 7febaa600700 1 -- 172.21.15.89:0/3578776901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba414bf10 con 0x7feba415ef70 2024-04-02T07:52:14.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.181+0000 7feb9b400700 1 -- 172.21.15.89:0/3578776901 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9001ad30 con 0x7feba415ef70 2024-04-02T07:52:14.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.182+0000 7febaa600700 1 -- 172.21.15.89:0/3578776901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba414c2a0 con 0x7feba415ef70 2024-04-02T07:52:14.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.183+0000 7febaa600700 1 -- 172.21.15.89:0/3578776901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba414c710 con 0x7feba415ef70 2024-04-02T07:52:14.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.184+0000 7feb9b400700 1 -- 172.21.15.89:0/3578776901 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7feb90022070 con 0x7feba415ef70 2024-04-02T07:52:14.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.184+0000 7feb9b400700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb8406adb0 0x7feb8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:14.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.185+0000 7feb9b400700 1 -- 172.21.15.89:0/3578776901 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7feb9008a6c0 con 0x7feba415ef70 2024-04-02T07:52:14.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.185+0000 7feba3400700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb8406adb0 0x7feb8406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:14.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.185+0000 7feba3400700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb8406adb0 0x7feb8406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feb94000f60 tx=0x7feb94009040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:14.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.189+0000 7feb9b400700 1 -- 172.21.15.89:0/3578776901 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feba414c710 con 0x7feba415ef70 2024-04-02T07:52:14.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.472+0000 7febaa600700 1 -- 172.21.15.89:0/3578776901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7feba414c710 con 0x7feba415ef70 2024-04-02T07:52:14.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.476+0000 7feb9b400700 1 -- 172.21.15.89:0/3578776901 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+0 (secure 0 0 0) 0x7feba414c710 con 0x7feba415ef70 2024-04-02T07:52:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.479+0000 7febaa600700 1 -- 172.21.15.89:0/3578776901 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb8406adb0 msgr2=0x7feb8406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.479+0000 7febaa600700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb8406adb0 0x7feb8406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feb94000f60 tx=0x7feb94009040 comp rx=0 tx=0).stop 2024-04-02T07:52:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.479+0000 7febaa600700 1 -- 172.21.15.89:0/3578776901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba415ef70 msgr2=0x7feba40b1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.479+0000 7febaa600700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba415ef70 0x7feba40b1d40 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7feb900090b0 tx=0x7feb90004750 comp rx=0 tx=0).stop 2024-04-02T07:52:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.479+0000 7febaa600700 1 -- 172.21.15.89:0/3578776901 shutdown_connections 2024-04-02T07:52:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.479+0000 7febaa600700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb8406adb0 0x7feb8406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.479+0000 7febaa600700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba4160d70 0x7feba415d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.479+0000 7febaa600700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feba406a790 0x7feba4149e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.479+0000 7febaa600700 1 --2- 172.21.15.89:0/3578776901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba415ef70 0x7feba40b1d40 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.479+0000 7febaa600700 1 -- 172.21.15.89:0/3578776901 >> 172.21.15.89:0/3578776901 conn(0x7feba40b5720 msgr2=0x7feba406e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.479+0000 7febaa600700 1 -- 172.21.15.89:0/3578776901 shutdown_connections 2024-04-02T07:52:14.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.480+0000 7febaa600700 1 -- 172.21.15.89:0/3578776901 wait complete. 2024-04-02T07:52:14.482 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T07:52:14.499 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:52:14.561 INFO:tasks.ceph.mds.a.smithi089.stderr:2024-04-02T07:52:14.559+0000 7fd7eee00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T07:52:14.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.691+0000 7f0d0b000700 1 Processor -- start 2024-04-02T07:52:14.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.691+0000 7f0d0b000700 1 -- start start 2024-04-02T07:52:14.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.692+0000 7f0d0b000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0d040cf080 0x7f0d040b3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:14.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.692+0000 7f0d0b000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d04140ac0 con 0x7f0d04161f70 2024-04-02T07:52:14.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.692+0000 7f0d0b000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0415ec10 con 0x7f0d0415dbe0 2024-04-02T07:52:14.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.692+0000 7f0d0b000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d040b36d0 con 0x7f0d040cf080 2024-04-02T07:52:14.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.693+0000 7f0d09000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0d04161f70 0x7f0d04070d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60402/0 (socket says 172.21.15.89:60402) 2024-04-02T07:52:14.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.693+0000 7f0d09000700 1 -- 172.21.15.89:0/1216777037 learned_addr learned my addr 172.21.15.89:0/1216777037 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:14.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.693+0000 7f0d03e00700 1 --2- 172.21.15.89:0/1216777037 >> v2:172.21.15.113:3300/0 conn(0x7f0d040cf080 0x7f0d040b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:14.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.693+0000 7f0d02a00700 1 -- 172.21.15.89:0/1216777037 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687910088 0 0) 0x7f0d04140ac0 con 0x7f0d04161f70 2024-04-02T07:52:14.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.693+0000 7f0d02a00700 1 -- 172.21.15.89:0/1216777037 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce80035b0 con 0x7f0d04161f70 2024-04-02T07:52:14.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.693+0000 7f0d03e00700 1 -- 172.21.15.89:0/1216777037 >> v1:172.21.15.113:6789/0 conn(0x7f0d0415dbe0 legacy=0x7f0d04161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:14.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.693+0000 7f0d03e00700 1 -- 172.21.15.89:0/1216777037 >> v1:172.21.15.89:6789/0 conn(0x7f0d04161f70 legacy=0x7f0d04070d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:14.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.693+0000 7f0d03e00700 1 -- 172.21.15.89:0/1216777037 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0415e370 con 0x7f0d040cf080 2024-04-02T07:52:14.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.694+0000 7f0d03e00700 1 --2- 172.21.15.89:0/1216777037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d040cf080 0x7f0d040b3190 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4009840 tx=0x7f0cf4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29847a6b4af0d1d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:14.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.694+0000 7f0d02a00700 1 -- 172.21.15.89:0/1216777037 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4011830 con 0x7f0d040cf080 2024-04-02T07:52:14.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.694+0000 7f0d02a00700 1 -- 172.21.15.89:0/1216777037 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf4011990 con 0x7f0d040cf080 2024-04-02T07:52:14.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.694+0000 7f0d02a00700 1 -- 172.21.15.89:0/1216777037 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4011d00 con 0x7f0d040cf080 2024-04-02T07:52:14.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.694+0000 7f0d0b000700 1 -- 172.21.15.89:0/1216777037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d040cf080 msgr2=0x7f0d040b3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:14.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.694+0000 7f0d0b000700 1 --2- 172.21.15.89:0/1216777037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d040cf080 0x7f0d040b3190 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4009840 tx=0x7f0cf4009b50 comp rx=0 tx=0).stop 2024-04-02T07:52:14.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.694+0000 7f0d0b000700 1 -- 172.21.15.89:0/1216777037 shutdown_connections 2024-04-02T07:52:14.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.694+0000 7f0d0b000700 1 --2- 172.21.15.89:0/1216777037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d040cf080 0x7f0d040b3190 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:14.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.694+0000 7f0d0b000700 1 -- 172.21.15.89:0/1216777037 >> 172.21.15.89:0/1216777037 conn(0x7f0d0406a790 msgr2=0x7f0d0406ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:14.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.695+0000 7f0d0b000700 1 -- 172.21.15.89:0/1216777037 shutdown_connections 2024-04-02T07:52:14.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.695+0000 7f0d0b000700 1 -- 172.21.15.89:0/1216777037 wait complete. 2024-04-02T07:52:14.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.696+0000 7f0d0b000700 1 Processor -- start 2024-04-02T07:52:14.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.696+0000 7f0d0b000700 1 -- start start 2024-04-02T07:52:14.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.697+0000 7f0d0b000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d040cf080 0x7f0d041f5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:14.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.697+0000 7f0d0b000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d0415dbe0 0x7f0d040b2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:14.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.697+0000 7f0d0b000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04161f70 0x7f0d040b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:14.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.697+0000 7f0d0b000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d041f70e0 con 0x7f0d04161f70 2024-04-02T07:52:14.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.697+0000 7f0d0b000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d041f6460 con 0x7f0d040cf080 2024-04-02T07:52:14.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.697+0000 7f0d0b000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d041f6740 con 0x7f0d0415dbe0 2024-04-02T07:52:14.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0d09000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04161f70 0x7f0d040b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:14.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0d09000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04161f70 0x7f0d040b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42050/0 (socket says 172.21.15.89:42050) 2024-04-02T07:52:14.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0d09000700 1 -- 172.21.15.89:0/2703754242 learned_addr learned my addr 172.21.15.89:0/2703754242 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:14.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0d03e00700 1 --2- 172.21.15.89:0/2703754242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d040cf080 0x7f0d041f5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:14.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0d03400700 1 --2- 172.21.15.89:0/2703754242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d0415dbe0 0x7f0d040b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:14.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0d09000700 1 -- 172.21.15.89:0/2703754242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d0415dbe0 msgr2=0x7f0d040b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:14.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0d09000700 1 --2- 172.21.15.89:0/2703754242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d0415dbe0 0x7f0d040b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:14.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0d09000700 1 -- 172.21.15.89:0/2703754242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d040cf080 msgr2=0x7f0d041f5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:14.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0d09000700 1 --2- 172.21.15.89:0/2703754242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d040cf080 0x7f0d041f5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:14.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0d09000700 1 -- 172.21.15.89:0/2703754242 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf40094a0 con 0x7f0d04161f70 2024-04-02T07:52:14.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0d09000700 1 --2- 172.21.15.89:0/2703754242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04161f70 0x7f0d040b2a40 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00cb40 tx=0x7f0cfc00ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:14.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0cfbe00700 1 -- 172.21.15.89:0/2703754242 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc00aaa0 con 0x7f0d04161f70 2024-04-02T07:52:14.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0cfbe00700 1 -- 172.21.15.89:0/2703754242 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc00ac00 con 0x7f0d04161f70 2024-04-02T07:52:14.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0cfbe00700 1 -- 172.21.15.89:0/2703754242 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc012400 con 0x7f0d04161f70 2024-04-02T07:52:14.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.698+0000 7f0d0b000700 1 -- 172.21.15.89:0/2703754242 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d041f6a20 con 0x7f0d04161f70 2024-04-02T07:52:14.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.699+0000 7f0d0b000700 1 -- 172.21.15.89:0/2703754242 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d041f60d0 con 0x7f0d04161f70 2024-04-02T07:52:14.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.699+0000 7f0d0b000700 1 -- 172.21.15.89:0/2703754242 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d040b34c0 con 0x7f0d04161f70 2024-04-02T07:52:14.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.700+0000 7f0cfbe00700 1 -- 172.21.15.89:0/2703754242 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0cfc01b3b0 con 0x7f0d04161f70 2024-04-02T07:52:14.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.702+0000 7f0cfbe00700 1 --2- 172.21.15.89:0/2703754242 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0ce406adb0 0x7f0ce406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:14.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.702+0000 7f0cfbe00700 1 -- 172.21.15.89:0/2703754242 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0cfc089360 con 0x7f0d04161f70 2024-04-02T07:52:14.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.702+0000 7f0d03e00700 1 --2- 172.21.15.89:0/2703754242 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0ce406adb0 0x7f0ce406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:14.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.703+0000 7f0d03e00700 1 --2- 172.21.15.89:0/2703754242 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0ce406adb0 0x7f0ce406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4000c00 tx=0x7f0cf401c040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:14.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.706+0000 7f0cfbe00700 1 -- 172.21.15.89:0/2703754242 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0cfc0542c0 con 0x7f0d04161f70 2024-04-02T07:52:14.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.954+0000 7f0d0b000700 1 -- 172.21.15.89:0/2703754242 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d04067720 con 0x7f0d04161f70 2024-04-02T07:52:14.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.954+0000 7f0cfbe00700 1 -- 172.21.15.89:0/2703754242 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f0cfc0578e0 con 0x7f0d04161f70 2024-04-02T07:52:14.956 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:14.956 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6556,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3307197948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3307197948},{"type":"v1","addr":"172.21.15.113:6835","nonce":3307197948}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1797655686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1797655686},{"type":"v1","addr":"172.21.15.113:6834","nonce":1797655686}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4203087300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4203087300},{"type":"v1","addr":"172.21.15.89:6836","nonce":4203087300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:52:09.066449+0000","modified":"2024-04-02T07:52:12.080422+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3228984565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3228984565},{"type":"v1","addr":"172.21.15.89:6837","nonce":3228984565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-02T07:52:14.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.957+0000 7f0d0b000700 1 -- 172.21.15.89:0/2703754242 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0ce406adb0 msgr2=0x7f0ce406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:14.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.957+0000 7f0d0b000700 1 --2- 172.21.15.89:0/2703754242 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0ce406adb0 0x7f0ce406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4000c00 tx=0x7f0cf401c040 comp rx=0 tx=0).stop 2024-04-02T07:52:14.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.957+0000 7f0d0b000700 1 -- 172.21.15.89:0/2703754242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04161f70 msgr2=0x7f0d040b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:14.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.957+0000 7f0d0b000700 1 --2- 172.21.15.89:0/2703754242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04161f70 0x7f0d040b2a40 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00cb40 tx=0x7f0cfc00ce50 comp rx=0 tx=0).stop 2024-04-02T07:52:14.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.957+0000 7f0d0b000700 1 -- 172.21.15.89:0/2703754242 shutdown_connections 2024-04-02T07:52:14.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.957+0000 7f0d0b000700 1 --2- 172.21.15.89:0/2703754242 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0ce406adb0 0x7f0ce406d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:14.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.957+0000 7f0d0b000700 1 --2- 172.21.15.89:0/2703754242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d040cf080 0x7f0d041f5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:14.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.957+0000 7f0d0b000700 1 --2- 172.21.15.89:0/2703754242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d04161f70 0x7f0d040b2a40 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:14.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.957+0000 7f0d0b000700 1 --2- 172.21.15.89:0/2703754242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d0415dbe0 0x7f0d040b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:14.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.957+0000 7f0d0b000700 1 -- 172.21.15.89:0/2703754242 >> 172.21.15.89:0/2703754242 conn(0x7f0d0406a790 msgr2=0x7f0d0413fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:14.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.957+0000 7f0d0b000700 1 -- 172.21.15.89:0/2703754242 shutdown_connections 2024-04-02T07:52:14.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:14.957+0000 7f0d0b000700 1 -- 172.21.15.89:0/2703754242 wait complete. 2024-04-02T07:52:14.959 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 41 2024-04-02T07:52:15.379 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T07:52:09.066449+0000', 'modified': '2024-04-02T07:52:12.080422+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6918}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6918': {'gid': 6918, 'name': 'a', 'rank': 0, 'incarnation': 37, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6837/3228984565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6835', 'nonce': 3228984565}, {'type': 'v1', 'addr': '172.21.15.89:6837', 'nonce': 3228984565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T07:52:15.379 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T07:52:15.379 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T07:52:15.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.569+0000 7f39a6600700 1 Processor -- start 2024-04-02T07:52:15.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.569+0000 7f39a6600700 1 -- start start 2024-04-02T07:52:15.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.570+0000 7f39a6600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f39a015d9d0 0x7f39a00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.570+0000 7f39a6600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a0161420 con 0x7f39a00d16a0 2024-04-02T07:52:15.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.570+0000 7f39a6600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a0161700 con 0x7f39a00d2540 2024-04-02T07:52:15.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.570+0000 7f39a6600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a015edc0 con 0x7f39a015d9d0 2024-04-02T07:52:15.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.570+0000 7f399ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f39a00d16a0 0x7f39a00d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60406/0 (socket says 172.21.15.89:60406) 2024-04-02T07:52:15.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.570+0000 7f399ea00700 1 -- 172.21.15.89:0/412990425 learned_addr learned my addr 172.21.15.89:0/412990425 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:15.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.571+0000 7f399f400700 1 --2- 172.21.15.89:0/412990425 >> v2:172.21.15.113:3300/0 conn(0x7f39a015d9d0 0x7f39a00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.571+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739439188 0 0) 0x7f39a0161420 con 0x7f39a00d16a0 2024-04-02T07:52:15.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.571+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39840035b0 con 0x7f39a00d16a0 2024-04-02T07:52:15.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.571+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838013004 0 0) 0x7f39a0161700 con 0x7f39a00d2540 2024-04-02T07:52:15.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.571+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a0161420 con 0x7f39a00d2540 2024-04-02T07:52:15.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.571+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1554018571 0 0) 0x7f39840035b0 con 0x7f39a00d16a0 2024-04-02T07:52:15.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.571+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 >> v2:172.21.15.113:3300/0 conn(0x7f39a015d9d0 msgr2=0x7f39a00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.571+0000 7f399e000700 1 --2- 172.21.15.89:0/412990425 >> v2:172.21.15.113:3300/0 conn(0x7f39a015d9d0 0x7f39a00d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.571+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 >> v1:172.21.15.113:6789/0 conn(0x7f39a00d2540 legacy=0x7f39a00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.571+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a01619e0 con 0x7f39a00d16a0 2024-04-02T07:52:15.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.571+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f398c0017f0 con 0x7f39a00d16a0 2024-04-02T07:52:15.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.571+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 >> v1:172.21.15.89:6789/0 conn(0x7f39a00d16a0 legacy=0x7f39a00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399e000700 1 --2- 172.21.15.89:0/412990425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3984003b90 0x7f3984003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399e000700 1 --2- 172.21.15.89:0/412990425 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3984004850 0x7f3984004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399e000700 1 --2- 172.21.15.89:0/412990425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3984005210 0x7f398400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398400c110 con 0x7f3984003b90 2024-04-02T07:52:15.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398400c280 con 0x7f3984005210 2024-04-02T07:52:15.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f398400c440 con 0x7f3984004850 2024-04-02T07:52:15.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399f400700 1 --2- 172.21.15.89:0/412990425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3984003b90 0x7f3984003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399ea00700 1 --2- 172.21.15.89:0/412990425 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3984004850 0x7f3984004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399fe00700 1 --2- 172.21.15.89:0/412990425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3984005210 0x7f398400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399f400700 1 -- 172.21.15.89:0/412990425 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3984004850 msgr2=0x7f3984004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399f400700 1 --2- 172.21.15.89:0/412990425 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3984004850 0x7f3984004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399f400700 1 -- 172.21.15.89:0/412990425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3984005210 msgr2=0x7f398400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399f400700 1 --2- 172.21.15.89:0/412990425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3984005210 0x7f398400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399f400700 1 -- 172.21.15.89:0/412990425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39840035b0 con 0x7f3984003b90 2024-04-02T07:52:15.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399fe00700 1 --2- 172.21.15.89:0/412990425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3984005210 0x7f398400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:15.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399fe00700 1 -- 172.21.15.89:0/412990425 reap_dead start 2024-04-02T07:52:15.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.572+0000 7f399f400700 1 --2- 172.21.15.89:0/412990425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3984003b90 0x7f3984003f90 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3990009b60 tx=0x7f3984005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456fa6cc34a2ef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:15.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.573+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399000e6c0 con 0x7f3984003b90 2024-04-02T07:52:15.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.573+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399000e820 con 0x7f3984003b90 2024-04-02T07:52:15.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.573+0000 7f399e000700 1 -- 172.21.15.89:0/412990425 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399000eb90 con 0x7f3984003b90 2024-04-02T07:52:15.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.573+0000 7f39a6600700 1 -- 172.21.15.89:0/412990425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3984003b90 msgr2=0x7f3984003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.573+0000 7f39a6600700 1 --2- 172.21.15.89:0/412990425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3984003b90 0x7f3984003f90 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3990009b60 tx=0x7f3984005880 comp rx=0 tx=0).stop 2024-04-02T07:52:15.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.573+0000 7f39a6600700 1 -- 172.21.15.89:0/412990425 shutdown_connections 2024-04-02T07:52:15.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.573+0000 7f39a6600700 1 --2- 172.21.15.89:0/412990425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3984003b90 0x7f3984003f90 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.573+0000 7f39a6600700 1 -- 172.21.15.89:0/412990425 >> 172.21.15.89:0/412990425 conn(0x7f39a008aff0 msgr2=0x7f39a00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:15.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.573+0000 7f39a6600700 1 -- 172.21.15.89:0/412990425 shutdown_connections 2024-04-02T07:52:15.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.573+0000 7f39a6600700 1 -- 172.21.15.89:0/412990425 wait complete. 2024-04-02T07:52:15.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.575+0000 7f39a6600700 1 Processor -- start 2024-04-02T07:52:15.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.575+0000 7f39a6600700 1 -- start start 2024-04-02T07:52:15.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.576+0000 7f39a6600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3984003b90 0x7f39a01f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.576+0000 7f39a6600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39a015ee30 0x7f39a00cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.576+0000 7f39a6600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39a0155360 0x7f39a00b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.576+0000 7f39a6600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a00cc560 con 0x7f39a015ee30 2024-04-02T07:52:15.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.576+0000 7f39a6600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a015f2b0 con 0x7f39a0155360 2024-04-02T07:52:15.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.576+0000 7f39a6600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a015f560 con 0x7f3984003b90 2024-04-02T07:52:15.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.576+0000 7f399ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39a015ee30 0x7f39a00cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.576+0000 7f399ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39a015ee30 0x7f39a00cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42060/0 (socket says 172.21.15.89:42060) 2024-04-02T07:52:15.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.576+0000 7f399ea00700 1 -- 172.21.15.89:0/2888667895 learned_addr learned my addr 172.21.15.89:0/2888667895 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:15.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.576+0000 7f399f400700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3984003b90 0x7f39a01f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.576+0000 7f399fe00700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39a0155360 0x7f39a00b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.577+0000 7f399ea00700 1 -- 172.21.15.89:0/2888667895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3984003b90 msgr2=0x7f39a01f7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.577+0000 7f399ea00700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3984003b90 0x7f39a01f7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.577+0000 7f399ea00700 1 -- 172.21.15.89:0/2888667895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39a0155360 msgr2=0x7f39a00b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.577+0000 7f399ea00700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39a0155360 0x7f39a00b4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.577+0000 7f399ea00700 1 -- 172.21.15.89:0/2888667895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a015f840 con 0x7f39a015ee30 2024-04-02T07:52:15.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.577+0000 7f399fe00700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39a0155360 0x7f39a00b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:15.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.577+0000 7f399ea00700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39a015ee30 0x7f39a00cc020 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f398c001eb0 tx=0x7f398c00e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:15.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.577+0000 7f3997400700 1 -- 172.21.15.89:0/2888667895 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c0073a0 con 0x7f39a015ee30 2024-04-02T07:52:15.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.577+0000 7f3997400700 1 -- 172.21.15.89:0/2888667895 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c007520 con 0x7f39a015ee30 2024-04-02T07:52:15.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.577+0000 7f3997400700 1 -- 172.21.15.89:0/2888667895 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c005a70 con 0x7f39a015ee30 2024-04-02T07:52:15.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.577+0000 7f39a6600700 1 -- 172.21.15.89:0/2888667895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a01f7090 con 0x7f39a015ee30 2024-04-02T07:52:15.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.577+0000 7f39a6600700 1 -- 172.21.15.89:0/2888667895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a01f7650 con 0x7f39a015ee30 2024-04-02T07:52:15.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.579+0000 7f3997400700 1 -- 172.21.15.89:0/2888667895 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f398c005bd0 con 0x7f39a015ee30 2024-04-02T07:52:15.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.581+0000 7f3997400700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f398006adb0 0x7f398006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.581+0000 7f3997400700 1 -- 172.21.15.89:0/2888667895 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f398c088e70 con 0x7f39a015ee30 2024-04-02T07:52:15.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.581+0000 7f399f400700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f398006adb0 0x7f398006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.581+0000 7f399f400700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f398006adb0 0x7f398006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f39900090b0 tx=0x7f399001e040 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:15.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.583+0000 7f39ba107b80 1 Processor -- start 2024-04-02T07:52:15.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.583+0000 7f39ba107b80 1 -- start start 2024-04-02T07:52:15.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.584+0000 7f39ba107b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x55629746ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.584+0000 7f39ba107b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 0x556297440a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.584+0000 7f39ba107b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x556297467a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.584+0000 7f39ba107b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562973b5650 con 0x55629746a830 2024-04-02T07:52:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.584+0000 7f39ba107b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x556297396c30 con 0x55629743bb60 2024-04-02T07:52:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.584+0000 7f39ba107b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x556297394430 con 0x55629743dbe0 2024-04-02T07:52:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.584+0000 7f399f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x55629746ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.584+0000 7f399f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x55629746ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42072/0 (socket says 172.21.15.89:42072) 2024-04-02T07:52:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.584+0000 7f399f400700 1 -- 172.21.15.89:0/1805755310 learned_addr learned my addr 172.21.15.89:0/1805755310 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.585+0000 7f399ea00700 1 --2- 172.21.15.89:0/1805755310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 0x556297440a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.585+0000 7f399fe00700 1 --2- 172.21.15.89:0/1805755310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x556297467a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.585+0000 7f399f400700 1 -- 172.21.15.89:0/1805755310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 msgr2=0x556297440a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.585+0000 7f399f400700 1 --2- 172.21.15.89:0/1805755310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 0x556297440a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.585+0000 7f399f400700 1 -- 172.21.15.89:0/1805755310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 msgr2=0x556297467a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.585+0000 7f399f400700 1 --2- 172.21.15.89:0/1805755310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x556297467a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.585+0000 7f399f400700 1 -- 172.21.15.89:0/1805755310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556297183790 con 0x55629746a830 2024-04-02T07:52:15.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.585+0000 7f399f400700 1 --2- 172.21.15.89:0/1805755310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x55629746ac10 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f398400c250 tx=0x7f3990013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31273c80494df45d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:15.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.585+0000 7f398be00700 1 -- 172.21.15.89:0/1805755310 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3990021040 con 0x55629746a830 2024-04-02T07:52:15.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.585+0000 7f398be00700 1 -- 172.21.15.89:0/1805755310 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399002d040 con 0x55629746a830 2024-04-02T07:52:15.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.585+0000 7f398be00700 1 -- 172.21.15.89:0/1805755310 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3990004470 con 0x55629746a830 2024-04-02T07:52:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.586+0000 7f39ba107b80 1 -- 172.21.15.89:0/1805755310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 msgr2=0x55629746ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.586+0000 7f39ba107b80 1 --2- 172.21.15.89:0/1805755310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x55629746ac10 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f398400c250 tx=0x7f3990013000 comp rx=0 tx=0).stop 2024-04-02T07:52:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.586+0000 7f39ba107b80 1 -- 172.21.15.89:0/1805755310 shutdown_connections 2024-04-02T07:52:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.586+0000 7f39ba107b80 1 --2- 172.21.15.89:0/1805755310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x556297467a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.586+0000 7f39ba107b80 1 --2- 172.21.15.89:0/1805755310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x55629746ac10 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.586+0000 7f39ba107b80 1 --2- 172.21.15.89:0/1805755310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 0x556297440a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.586+0000 7f39ba107b80 1 -- 172.21.15.89:0/1805755310 >> 172.21.15.89:0/1805755310 conn(0x55629739e6a0 msgr2=0x556297485640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.586+0000 7f39ba107b80 1 -- 172.21.15.89:0/1805755310 shutdown_connections 2024-04-02T07:52:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.586+0000 7f39ba107b80 1 -- 172.21.15.89:0/1805755310 wait complete. 2024-04-02T07:52:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.587+0000 7f39ba107b80 1 -- start start 2024-04-02T07:52:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.587+0000 7f39ba107b80 1 Processor -- start 2024-04-02T07:52:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.588+0000 7f39ba107b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x556297466b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.588+0000 7f39ba107b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629743dbe0 0x556297497cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.588+0000 7f39ba107b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629746a830 0x556297494690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.588+0000 7f39ba107b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55629738fd20 con 0x55629743dbe0 2024-04-02T07:52:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.588+0000 7f39ba107b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55629732b770 con 0x55629743bb60 2024-04-02T07:52:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.588+0000 7f39ba107b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55629734e6f0 con 0x55629746a830 2024-04-02T07:52:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.588+0000 7f39ba107b80 10 client.0 _ll_register_callbacks cb 0x5562973355e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:52:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.588+0000 7f399ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629743dbe0 0x556297497cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.588+0000 7f399ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629743dbe0 0x556297497cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42082/0 (socket says 172.21.15.89:42082) 2024-04-02T07:52:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.588+0000 7f399ea00700 1 -- 172.21.15.89:0/133867229 learned_addr learned my addr 172.21.15.89:0/133867229 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.588+0000 7f399f400700 1 --2- 172.21.15.89:0/133867229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x556297466b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.589+0000 7f399fe00700 1 --2- 172.21.15.89:0/133867229 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629746a830 0x556297494690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.589+0000 7f399ea00700 1 -- 172.21.15.89:0/133867229 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629746a830 msgr2=0x556297494690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.589+0000 7f399ea00700 1 --2- 172.21.15.89:0/133867229 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629746a830 0x556297494690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.589+0000 7f399ea00700 1 -- 172.21.15.89:0/133867229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 msgr2=0x556297466b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.589+0000 7f399ea00700 1 --2- 172.21.15.89:0/133867229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x556297466b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.589+0000 7f399ea00700 1 -- 172.21.15.89:0/133867229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39840035b0 con 0x55629743dbe0 2024-04-02T07:52:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.589+0000 7f399ea00700 1 --2- 172.21.15.89:0/133867229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629743dbe0 0x556297497cb0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f398c00ecb0 tx=0x7f398c058200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c375e50dd460b31 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.589+0000 7f3988c00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:52:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.589+0000 7f3988c00700 1 -- 172.21.15.89:0/133867229 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c05b8c0 con 0x55629743dbe0 2024-04-02T07:52:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.589+0000 7f3988c00700 1 -- 172.21.15.89:0/133867229 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f398c05fe80 con 0x55629743dbe0 2024-04-02T07:52:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.589+0000 7f39ba107b80 1 -- 172.21.15.89:0/133867229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556297432920 con 0x55629743dbe0 2024-04-02T07:52:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.589+0000 7f3988c00700 1 -- 172.21.15.89:0/133867229 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f398c056f80 con 0x55629743dbe0 2024-04-02T07:52:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.589+0000 7f39ba107b80 10 client.7062 fetch_fsmap learned FSMap version 41 2024-04-02T07:52:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.589+0000 7f39ba107b80 1 -- 172.21.15.89:0/133867229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) v3 -- 0x5562972700a0 con 0x55629743dbe0 2024-04-02T07:52:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.590+0000 7f3988c00700 1 -- 172.21.15.89:0/133867229 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c073020 con 0x55629743dbe0 2024-04-02T07:52:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.590+0000 7f3988c00700 1 -- 172.21.15.89:0/133867229 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 41) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f398c05f9f0 con 0x55629743dbe0 2024-04-02T07:52:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.590+0000 7f39ba107b80 10 client.7062 fetch_fsmap finished waiting for FSMap version 41 2024-04-02T07:52:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.590+0000 7f39ba107b80 10 client.7062 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T07:52:15.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.590+0000 7f39ba107b80 20 client.7062 populate_metadata read hostname 'smithi089' 2024-04-02T07:52:15.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.590+0000 7f39ba107b80 1 --2- 172.21.15.89:0/133867229 >> [v2:172.21.15.89:6835/3228984565,v1:172.21.15.89:6837/3228984565] conn(0x55629746d920 0x55629746dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.590+0000 7f39ba107b80 4 client.7062 mds_command: new command op to 6918 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T07:52:15.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.590+0000 7f39ba107b80 1 -- 172.21.15.89:0/133867229 --> [v2:172.21.15.89:6835/3228984565,v1:172.21.15.89:6837/3228984565] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55629740f6f0 con 0x55629746d920 2024-04-02T07:52:15.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.590+0000 7f399f400700 1 --2- 172.21.15.89:0/133867229 >> [v2:172.21.15.89:6835/3228984565,v1:172.21.15.89:6837/3228984565] conn(0x55629746d920 0x55629746dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.590+0000 7f399f400700 1 --2- 172.21.15.89:0/133867229 >> [v2:172.21.15.89:6835/3228984565,v1:172.21.15.89:6837/3228984565] conn(0x55629746d920 0x55629746dd00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ff9c33fad1da9f5 server_cookie=c0340ab8308a96d4 in_seq=0 out_seq=0 2024-04-02T07:52:15.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.590+0000 7f3988c00700 10 client.7062 ms_handle_connect on v2:172.21.15.89:6835/3228984565 2024-04-02T07:52:15.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.591+0000 7f3988c00700 1 -- 172.21.15.89:0/133867229 <== mds.0 v2:172.21.15.89:6835/3228984565 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55629740f6f0 con 0x55629746d920 2024-04-02T07:52:15.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.591+0000 7f3988c00700 10 client.7062 handle_command_reply: tid=0 2024-04-02T07:52:15.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.591+0000 7f39ba107b80 1 client.7062 shutdown 2024-04-02T07:52:15.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.592+0000 7f39ba107b80 1 -- 172.21.15.89:0/133867229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629743dbe0 msgr2=0x556297497cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.592+0000 7f39ba107b80 1 --2- 172.21.15.89:0/133867229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629743dbe0 0x556297497cb0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f398c00ecb0 tx=0x7f398c058200 comp rx=0 tx=0).stop 2024-04-02T07:52:15.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.592+0000 7f39ba107b80 1 -- 172.21.15.89:0/133867229 shutdown_connections 2024-04-02T07:52:15.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.592+0000 7f39ba107b80 1 --2- 172.21.15.89:0/133867229 >> [v2:172.21.15.89:6835/3228984565,v1:172.21.15.89:6837/3228984565] conn(0x55629746d920 0x55629746dd00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.592+0000 7f39ba107b80 1 --2- 172.21.15.89:0/133867229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x556297466b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.592+0000 7f39ba107b80 1 --2- 172.21.15.89:0/133867229 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629746a830 0x556297494690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.592+0000 7f39ba107b80 1 --2- 172.21.15.89:0/133867229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629743dbe0 0x556297497cb0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.592+0000 7f3988c00700 0 client.7062 ms_handle_reset on v2:172.21.15.89:6835/3228984565 2024-04-02T07:52:15.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.592+0000 7f39ba107b80 1 -- 172.21.15.89:0/133867229 >> 172.21.15.89:0/133867229 conn(0x55629739e6a0 msgr2=0x556297488a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:15.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.592+0000 7f39ba107b80 1 -- 172.21.15.89:0/133867229 shutdown_connections 2024-04-02T07:52:15.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.592+0000 7f39ba107b80 1 -- 172.21.15.89:0/133867229 wait complete. 2024-04-02T07:52:15.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.592+0000 7f39ba107b80 20 client.7062 trim_cache size 0 max 16384 2024-04-02T07:52:15.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.612+0000 7f39ba107b80 1 Processor -- start 2024-04-02T07:52:15.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.612+0000 7f39ba107b80 1 -- start start 2024-04-02T07:52:15.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.613+0000 7f39ba107b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x5562974396a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.613+0000 7f39ba107b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 0x55629743f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.613+0000 7f39ba107b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x55629743feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.613+0000 7f39ba107b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556297184390 con 0x55629746a830 2024-04-02T07:52:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.613+0000 7f39ba107b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x556297436a20 con 0x55629743bb60 2024-04-02T07:52:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.613+0000 7f39ba107b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55629743c850 con 0x55629743dbe0 2024-04-02T07:52:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.613+0000 7f399fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x55629743feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.613+0000 7f399fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x55629743feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42084/0 (socket says 172.21.15.89:42084) 2024-04-02T07:52:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.613+0000 7f399fe00700 1 -- 172.21.15.89:0/357060751 learned_addr learned my addr 172.21.15.89:0/357060751 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.613+0000 7f399ea00700 1 --2- 172.21.15.89:0/357060751 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 0x55629743f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.613+0000 7f399f400700 1 --2- 172.21.15.89:0/357060751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x5562974396a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.614+0000 7f399fe00700 1 -- 172.21.15.89:0/357060751 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 msgr2=0x55629743f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.614+0000 7f399fe00700 1 --2- 172.21.15.89:0/357060751 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 0x55629743f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.614+0000 7f399fe00700 1 -- 172.21.15.89:0/357060751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 msgr2=0x5562974396a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.614+0000 7f399fe00700 1 --2- 172.21.15.89:0/357060751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x5562974396a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.614+0000 7f399fe00700 1 -- 172.21.15.89:0/357060751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556297356010 con 0x55629746a830 2024-04-02T07:52:15.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.614+0000 7f399f400700 1 --2- 172.21.15.89:0/357060751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x5562974396a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:15.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.614+0000 7f399fe00700 1 --2- 172.21.15.89:0/357060751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x55629743feb0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f39a00ca5a0 tx=0x7f3998004e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b85bc432406e49 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:15.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.614+0000 7f3994c00700 1 -- 172.21.15.89:0/357060751 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3998005410 con 0x55629746a830 2024-04-02T07:52:15.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.614+0000 7f3994c00700 1 -- 172.21.15.89:0/357060751 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3998005570 con 0x55629746a830 2024-04-02T07:52:15.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.614+0000 7f3994c00700 1 -- 172.21.15.89:0/357060751 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39980056e0 con 0x55629746a830 2024-04-02T07:52:15.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.615+0000 7f39ba107b80 1 -- 172.21.15.89:0/357060751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 msgr2=0x55629743feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.615+0000 7f39ba107b80 1 --2- 172.21.15.89:0/357060751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x55629743feb0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f39a00ca5a0 tx=0x7f3998004e10 comp rx=0 tx=0).stop 2024-04-02T07:52:15.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.615+0000 7f39ba107b80 1 -- 172.21.15.89:0/357060751 shutdown_connections 2024-04-02T07:52:15.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.615+0000 7f39ba107b80 1 --2- 172.21.15.89:0/357060751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x5562974396a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.615+0000 7f39ba107b80 1 --2- 172.21.15.89:0/357060751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x55629743feb0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.615+0000 7f39ba107b80 1 --2- 172.21.15.89:0/357060751 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 0x55629743f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.615+0000 7f39ba107b80 1 -- 172.21.15.89:0/357060751 >> 172.21.15.89:0/357060751 conn(0x55629739e6a0 msgr2=0x556297466000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:15.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.615+0000 7f39ba107b80 1 -- 172.21.15.89:0/357060751 shutdown_connections 2024-04-02T07:52:15.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.615+0000 7f39ba107b80 1 -- 172.21.15.89:0/357060751 wait complete. 2024-04-02T07:52:15.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.616+0000 7f39ba107b80 1 -- start start 2024-04-02T07:52:15.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.616+0000 7f39ba107b80 1 Processor -- start 2024-04-02T07:52:15.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.617+0000 7f39ba107b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x556297489430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.617+0000 7f39ba107b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 0x556297439680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.617+0000 7f39ba107b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x5562974819b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.617+0000 7f39ba107b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562974677e0 con 0x55629746a830 2024-04-02T07:52:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.617+0000 7f39ba107b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55629743eb20 con 0x55629743bb60 2024-04-02T07:52:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.617+0000 7f39ba107b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55629746c930 con 0x55629743dbe0 2024-04-02T07:52:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.617+0000 7f39ba107b80 10 client.0 _ll_register_callbacks cb 0x5562973355e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:52:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.617+0000 7f399fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x5562974819b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.617+0000 7f399fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x5562974819b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42096/0 (socket says 172.21.15.89:42096) 2024-04-02T07:52:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.617+0000 7f399fe00700 1 -- 172.21.15.89:0/2628693985 learned_addr learned my addr 172.21.15.89:0/2628693985 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.617+0000 7f399f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x556297489430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.617+0000 7f399ea00700 1 --2- 172.21.15.89:0/2628693985 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 0x556297439680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.617+0000 7f399fe00700 1 -- 172.21.15.89:0/2628693985 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 msgr2=0x556297439680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.618+0000 7f399fe00700 1 --2- 172.21.15.89:0/2628693985 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 0x556297439680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.618+0000 7f399fe00700 1 -- 172.21.15.89:0/2628693985 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 msgr2=0x556297489430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.618+0000 7f399fe00700 1 --2- 172.21.15.89:0/2628693985 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x556297489430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.618+0000 7f399fe00700 1 -- 172.21.15.89:0/2628693985 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f399800f040 con 0x55629746a830 2024-04-02T07:52:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.618+0000 7f399fe00700 1 --2- 172.21.15.89:0/2628693985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x5562974819b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f3998009010 tx=0x7f3998001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dfd0d28e8e28bbc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.618+0000 7f3988c00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:52:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.618+0000 7f3988c00700 1 -- 172.21.15.89:0/2628693985 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3998001750 con 0x55629746a830 2024-04-02T07:52:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.618+0000 7f3988c00700 1 -- 172.21.15.89:0/2628693985 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3998022a10 con 0x55629746a830 2024-04-02T07:52:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.618+0000 7f39ba107b80 1 -- 172.21.15.89:0/2628693985 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39a00b03a0 con 0x55629746a830 2024-04-02T07:52:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.618+0000 7f3988c00700 1 -- 172.21.15.89:0/2628693985 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3998022bd0 con 0x55629746a830 2024-04-02T07:52:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.618+0000 7f39ba107b80 10 client.7068 fetch_fsmap learned FSMap version 41 2024-04-02T07:52:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.618+0000 7f39ba107b80 1 -- 172.21.15.89:0/2628693985 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) v3 -- 0x5562971e41d0 con 0x55629746a830 2024-04-02T07:52:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.618+0000 7f3988c00700 1 -- 172.21.15.89:0/2628693985 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a00b03a0 con 0x55629746a830 2024-04-02T07:52:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.619+0000 7f3988c00700 1 -- 172.21.15.89:0/2628693985 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 41) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3998017430 con 0x55629746a830 2024-04-02T07:52:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.619+0000 7f39ba107b80 10 client.7068 fetch_fsmap finished waiting for FSMap version 41 2024-04-02T07:52:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.619+0000 7f39ba107b80 10 client.7068 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T07:52:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.619+0000 7f39ba107b80 20 client.7068 populate_metadata read hostname 'smithi089' 2024-04-02T07:52:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.619+0000 7f39ba107b80 1 --2- 172.21.15.89:0/2628693985 >> [v2:172.21.15.89:6835/3228984565,v1:172.21.15.89:6837/3228984565] conn(0x5562974821d0 0x55629746b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.619+0000 7f39ba107b80 4 client.7068 mds_command: new command op to 6918 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T07:52:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.619+0000 7f39ba107b80 1 -- 172.21.15.89:0/2628693985 --> [v2:172.21.15.89:6835/3228984565,v1:172.21.15.89:6837/3228984565] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f399002d040 con 0x5562974821d0 2024-04-02T07:52:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.619+0000 7f399f400700 1 --2- 172.21.15.89:0/2628693985 >> [v2:172.21.15.89:6835/3228984565,v1:172.21.15.89:6837/3228984565] conn(0x5562974821d0 0x55629746b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.619+0000 7f3988c00700 10 client.7068 ms_handle_connect on v2:172.21.15.89:6835/3228984565 2024-04-02T07:52:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.619+0000 7f399f400700 1 --2- 172.21.15.89:0/2628693985 >> [v2:172.21.15.89:6835/3228984565,v1:172.21.15.89:6837/3228984565] conn(0x5562974821d0 0x55629746b760 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f77615f8f56559f server_cookie=81a9ec32eefe6ea4 in_seq=0 out_seq=0 2024-04-02T07:52:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f3988c00700 1 -- 172.21.15.89:0/2628693985 <== mds.0 v2:172.21.15.89:6835/3228984565 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x7f399002d040 con 0x5562974821d0 2024-04-02T07:52:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f3988c00700 10 client.7068 handle_command_reply: tid=0 2024-04-02T07:52:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f39ba107b80 1 client.7068 shutdown 2024-04-02T07:52:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f39ba107b80 1 -- 172.21.15.89:0/2628693985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 msgr2=0x5562974819b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f39ba107b80 1 --2- 172.21.15.89:0/2628693985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x5562974819b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f3998009010 tx=0x7f3998001650 comp rx=0 tx=0).stop 2024-04-02T07:52:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f39ba107b80 1 -- 172.21.15.89:0/2628693985 shutdown_connections 2024-04-02T07:52:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f39ba107b80 1 --2- 172.21.15.89:0/2628693985 >> [v2:172.21.15.89:6835/3228984565,v1:172.21.15.89:6837/3228984565] conn(0x5562974821d0 0x55629746b760 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f39ba107b80 1 --2- 172.21.15.89:0/2628693985 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55629743bb60 0x556297489430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f39ba107b80 1 --2- 172.21.15.89:0/2628693985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55629746a830 0x5562974819b0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f3988c00700 0 client.7068 ms_handle_reset on v2:172.21.15.89:6835/3228984565 2024-04-02T07:52:15.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f39ba107b80 1 --2- 172.21.15.89:0/2628693985 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55629743dbe0 0x556297439680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f39ba107b80 1 -- 172.21.15.89:0/2628693985 >> 172.21.15.89:0/2628693985 conn(0x55629739e6a0 msgr2=0x556297440840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:15.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f39ba107b80 1 -- 172.21.15.89:0/2628693985 shutdown_connections 2024-04-02T07:52:15.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f39ba107b80 1 -- 172.21.15.89:0/2628693985 wait complete. 2024-04-02T07:52:15.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.620+0000 7f39ba107b80 20 client.7068 trim_cache size 0 max 16384 2024-04-02T07:52:15.630 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:52:15.631 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T07:52:15.631 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T07:52:15.631 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 6918, 2024-04-02T07:52:15.631 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T07:52:15.631 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T07:52:15.631 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T07:52:15.631 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 6.5510435129999998, 2024-04-02T07:52:15.631 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 41, 2024-04-02T07:52:15.631 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 97, 2024-04-02T07:52:15.631 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 97, 2024-04-02T07:52:15.631 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 11.940089413000001 2024-04-02T07:52:15.631 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:52:15.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.621+0000 7f39a6600700 1 -- 172.21.15.89:0/2888667895 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f398006adb0 msgr2=0x7f398006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.621+0000 7f39a6600700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f398006adb0 0x7f398006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f39900090b0 tx=0x7f399001e040 comp rx=0 tx=0).stop 2024-04-02T07:52:15.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.621+0000 7f39a6600700 1 -- 172.21.15.89:0/2888667895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39a015ee30 msgr2=0x7f39a00cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.621+0000 7f39a6600700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39a015ee30 0x7f39a00cc020 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f398c001eb0 tx=0x7f398c00e8b0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.621+0000 7f39a6600700 1 -- 172.21.15.89:0/2888667895 shutdown_connections 2024-04-02T07:52:15.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.621+0000 7f39a6600700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f398006adb0 0x7f398006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.621+0000 7f39a6600700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39a0155360 0x7f39a00b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.621+0000 7f39a6600700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3984003b90 0x7f39a01f7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.621+0000 7f39a6600700 1 --2- 172.21.15.89:0/2888667895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39a015ee30 0x7f39a00cc020 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.621+0000 7f39a6600700 1 -- 172.21.15.89:0/2888667895 >> 172.21.15.89:0/2888667895 conn(0x7f39a008aff0 msgr2=0x7f39a00c99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:15.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.621+0000 7f39a6600700 1 -- 172.21.15.89:0/2888667895 shutdown_connections 2024-04-02T07:52:15.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.621+0000 7f39a6600700 1 -- 172.21.15.89:0/2888667895 wait complete. 2024-04-02T07:52:15.639 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T07:52:15.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.829+0000 7ff52ae00700 1 Processor -- start 2024-04-02T07:52:15.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.830+0000 7ff52ae00700 1 -- start start 2024-04-02T07:52:15.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.831+0000 7ff52ae00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff52415b5d0 0x7ff5240ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.831+0000 7ff52ae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5240ac7d0 0x7ff52406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.831+0000 7ff52ae00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff52415fd50 con 0x7ff5240cb3f0 2024-04-02T07:52:15.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.831+0000 7ff52ae00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff52415b9b0 con 0x7ff52415b5d0 2024-04-02T07:52:15.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.831+0000 7ff52ae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52415f140 con 0x7ff5240ac7d0 2024-04-02T07:52:15.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.831+0000 7ff51be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff5240cb3f0 0x7ff5240ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60422/0 (socket says 172.21.15.89:60422) 2024-04-02T07:52:15.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.831+0000 7ff51be00700 1 -- 172.21.15.89:0/3979469710 learned_addr learned my addr 172.21.15.89:0/3979469710 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:15.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.831+0000 7ff523400700 1 -- 172.21.15.89:0/3979469710 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733421392 0 0) 0x7ff52415fd50 con 0x7ff5240cb3f0 2024-04-02T07:52:15.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.831+0000 7ff528e00700 1 --2- 172.21.15.89:0/3979469710 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5240ac7d0 0x7ff52406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.831+0000 7ff523400700 1 -- 172.21.15.89:0/3979469710 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5080035b0 con 0x7ff5240cb3f0 2024-04-02T07:52:15.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.831+0000 7ff523e00700 1 --2- 172.21.15.89:0/3979469710 >> v2:172.21.15.113:3300/0 conn(0x7ff52415b5d0 0x7ff5240ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523400700 1 -- 172.21.15.89:0/3979469710 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2820613294 0 0) 0x7ff5080035b0 con 0x7ff5240cb3f0 2024-04-02T07:52:15.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523400700 1 -- 172.21.15.89:0/3979469710 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5240ac7d0 msgr2=0x7ff52406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523400700 1 --2- 172.21.15.89:0/3979469710 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5240ac7d0 0x7ff52406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523400700 1 -- 172.21.15.89:0/3979469710 >> v2:172.21.15.113:3300/0 conn(0x7ff52415b5d0 msgr2=0x7ff5240ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523400700 1 --2- 172.21.15.89:0/3979469710 >> v2:172.21.15.113:3300/0 conn(0x7ff52415b5d0 0x7ff5240ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523400700 1 -- 172.21.15.89:0/3979469710 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52415f420 con 0x7ff5240cb3f0 2024-04-02T07:52:15.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523400700 1 -- 172.21.15.89:0/3979469710 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7ff50c001670 con 0x7ff5240cb3f0 2024-04-02T07:52:15.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523400700 1 -- 172.21.15.89:0/3979469710 >> v1:172.21.15.89:6789/0 conn(0x7ff5240cb3f0 legacy=0x7ff5240ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523400700 1 --2- 172.21.15.89:0/3979469710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff508003a00 0x7ff508003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523400700 1 --2- 172.21.15.89:0/3979469710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff508004850 0x7ff508004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523400700 1 --2- 172.21.15.89:0/3979469710 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5240ac7d0 0x7ff508005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523400700 1 -- 172.21.15.89:0/3979469710 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52415b9b0 con 0x7ff508003a00 2024-04-02T07:52:15.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523400700 1 -- 172.21.15.89:0/3979469710 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52415f140 con 0x7ff508004850 2024-04-02T07:52:15.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523400700 1 -- 172.21.15.89:0/3979469710 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50800ba40 con 0x7ff5240ac7d0 2024-04-02T07:52:15.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff523e00700 1 --2- 172.21.15.89:0/3979469710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff508003a00 0x7ff508003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff51be00700 1 --2- 172.21.15.89:0/3979469710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff508004850 0x7ff508004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.832+0000 7ff528e00700 1 --2- 172.21.15.89:0/3979469710 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5240ac7d0 0x7ff508005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.833+0000 7ff523e00700 1 -- 172.21.15.89:0/3979469710 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5240ac7d0 msgr2=0x7ff508005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.833+0000 7ff523e00700 1 --2- 172.21.15.89:0/3979469710 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5240ac7d0 0x7ff508005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.833+0000 7ff523e00700 1 -- 172.21.15.89:0/3979469710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff508004850 msgr2=0x7ff508004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:15.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.833+0000 7ff523e00700 1 --2- 172.21.15.89:0/3979469710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff508004850 0x7ff508004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.833+0000 7ff523e00700 1 -- 172.21.15.89:0/3979469710 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5080035b0 con 0x7ff508003a00 2024-04-02T07:52:15.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.833+0000 7ff523e00700 1 --2- 172.21.15.89:0/3979469710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff508003a00 0x7ff508003e20 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7ff514009b60 tx=0x7ff508005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca81cb2c747fe093 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:15.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.833+0000 7ff523400700 1 -- 172.21.15.89:0/3979469710 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51400e660 con 0x7ff508003a00 2024-04-02T07:52:15.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.833+0000 7ff523400700 1 -- 172.21.15.89:0/3979469710 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51400e7c0 con 0x7ff508003a00 2024-04-02T07:52:15.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.833+0000 7ff523400700 1 -- 172.21.15.89:0/3979469710 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51400eb30 con 0x7ff508003a00 2024-04-02T07:52:15.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.834+0000 7ff52ae00700 1 -- 172.21.15.89:0/3979469710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff508003a00 msgr2=0x7ff508003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.834+0000 7ff52ae00700 1 --2- 172.21.15.89:0/3979469710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff508003a00 0x7ff508003e20 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7ff514009b60 tx=0x7ff508005880 comp rx=0 tx=0).stop 2024-04-02T07:52:15.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.834+0000 7ff528e00700 1 -- 172.21.15.89:0/3979469710 reap_dead start 2024-04-02T07:52:15.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.834+0000 7ff52ae00700 1 -- 172.21.15.89:0/3979469710 shutdown_connections 2024-04-02T07:52:15.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.834+0000 7ff52ae00700 1 -- 172.21.15.89:0/3979469710 >> 172.21.15.89:0/3979469710 conn(0x7ff5240cf190 msgr2=0x7ff5240cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:15.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.834+0000 7ff52ae00700 1 -- 172.21.15.89:0/3979469710 shutdown_connections 2024-04-02T07:52:15.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.834+0000 7ff52ae00700 1 -- 172.21.15.89:0/3979469710 wait complete. 2024-04-02T07:52:15.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.836+0000 7ff52ae00700 1 Processor -- start 2024-04-02T07:52:15.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.836+0000 7ff52ae00700 1 -- start start 2024-04-02T07:52:15.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.837+0000 7ff52ae00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff508003a00 0x7ff52415e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.837+0000 7ff52ae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5240c88b0 0x7ff5241f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.837+0000 7ff52ae00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5240c8de0 0x7ff5240c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.837+0000 7ff52ae00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240caa10 con 0x7ff508003a00 2024-04-02T07:52:15.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.837+0000 7ff52ae00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240c97a0 con 0x7ff5240c8de0 2024-04-02T07:52:15.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.837+0000 7ff52ae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5240af230 con 0x7ff5240c88b0 2024-04-02T07:52:15.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.837+0000 7ff523e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff508003a00 0x7ff52415e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.837+0000 7ff523e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff508003a00 0x7ff52415e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42124/0 (socket says 172.21.15.89:42124) 2024-04-02T07:52:15.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.837+0000 7ff523e00700 1 -- 172.21.15.89:0/2315560536 learned_addr learned my addr 172.21.15.89:0/2315560536 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:15.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.837+0000 7ff523e00700 1 -- 172.21.15.89:0/2315560536 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5240c88b0 msgr2=0x7ff5241f37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.837+0000 7ff523e00700 1 --2- 172.21.15.89:0/2315560536 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5240c88b0 0x7ff5241f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.837+0000 7ff523e00700 1 -- 172.21.15.89:0/2315560536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5240c8de0 msgr2=0x7ff5240c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:15.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.837+0000 7ff523e00700 1 --2- 172.21.15.89:0/2315560536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5240c8de0 0x7ff5240c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:15.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.837+0000 7ff523e00700 1 -- 172.21.15.89:0/2315560536 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5241f3d00 con 0x7ff508003a00 2024-04-02T07:52:15.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.838+0000 7ff523e00700 1 --2- 172.21.15.89:0/2315560536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff508003a00 0x7ff52415e7e0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff508004770 tx=0x7ff5140047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:15.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.838+0000 7ff520c00700 1 -- 172.21.15.89:0/2315560536 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51400e660 con 0x7ff508003a00 2024-04-02T07:52:15.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.838+0000 7ff520c00700 1 -- 172.21.15.89:0/2315560536 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff514003ab0 con 0x7ff508003a00 2024-04-02T07:52:15.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.838+0000 7ff52ae00700 1 -- 172.21.15.89:0/2315560536 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5240aef50 con 0x7ff508003a00 2024-04-02T07:52:15.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.838+0000 7ff520c00700 1 -- 172.21.15.89:0/2315560536 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514003e60 con 0x7ff508003a00 2024-04-02T07:52:15.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.838+0000 7ff52ae00700 1 -- 172.21.15.89:0/2315560536 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5241f3fc0 con 0x7ff508003a00 2024-04-02T07:52:15.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.839+0000 7ff52ae00700 1 -- 172.21.15.89:0/2315560536 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5241f4330 con 0x7ff508003a00 2024-04-02T07:52:15.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.840+0000 7ff520c00700 1 -- 172.21.15.89:0/2315560536 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff514013470 con 0x7ff508003a00 2024-04-02T07:52:15.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.841+0000 7ff520c00700 1 --2- 172.21.15.89:0/2315560536 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff50006adb0 0x7ff50006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:15.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.841+0000 7ff520c00700 1 -- 172.21.15.89:0/2315560536 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff51408add0 con 0x7ff508003a00 2024-04-02T07:52:15.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.841+0000 7ff51be00700 1 --2- 172.21.15.89:0/2315560536 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff50006adb0 0x7ff50006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:15.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.842+0000 7ff51be00700 1 --2- 172.21.15.89:0/2315560536 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff50006adb0 0x7ff50006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff50c001050 tx=0x7ff50c001110 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:15.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:15.846+0000 7ff520c00700 1 -- 172.21.15.89:0/2315560536 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff514055dd0 con 0x7ff508003a00 2024-04-02T07:52:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:16.088+0000 7ff52ae00700 1 -- 172.21.15.89:0/2315560536 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff524067720 con 0x7ff508003a00 2024-04-02T07:52:16.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:16.089+0000 7ff520c00700 1 -- 172.21.15.89:0/2315560536 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v41) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff5140593f0 con 0x7ff508003a00 2024-04-02T07:52:16.090 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T07:52:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:16.091+0000 7ff52ae00700 1 -- 172.21.15.89:0/2315560536 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff50006adb0 msgr2=0x7ff50006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:16.091+0000 7ff52ae00700 1 --2- 172.21.15.89:0/2315560536 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff50006adb0 0x7ff50006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff50c001050 tx=0x7ff50c001110 comp rx=0 tx=0).stop 2024-04-02T07:52:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:16.091+0000 7ff52ae00700 1 -- 172.21.15.89:0/2315560536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff508003a00 msgr2=0x7ff52415e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:16.091+0000 7ff52ae00700 1 --2- 172.21.15.89:0/2315560536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff508003a00 0x7ff52415e7e0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff508004770 tx=0x7ff5140047c0 comp rx=0 tx=0).stop 2024-04-02T07:52:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:16.091+0000 7ff52ae00700 1 -- 172.21.15.89:0/2315560536 shutdown_connections 2024-04-02T07:52:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:16.091+0000 7ff52ae00700 1 --2- 172.21.15.89:0/2315560536 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff50006adb0 0x7ff50006d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:16.091+0000 7ff52ae00700 1 --2- 172.21.15.89:0/2315560536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5240c8de0 0x7ff5240c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:16.091+0000 7ff52ae00700 1 --2- 172.21.15.89:0/2315560536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff508003a00 0x7ff52415e7e0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:16.091+0000 7ff52ae00700 1 --2- 172.21.15.89:0/2315560536 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff5240c88b0 0x7ff5241f37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:16.091+0000 7ff52ae00700 1 -- 172.21.15.89:0/2315560536 >> 172.21.15.89:0/2315560536 conn(0x7ff5240cf190 msgr2=0x7ff52414c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:16.091+0000 7ff52ae00700 1 -- 172.21.15.89:0/2315560536 shutdown_connections 2024-04-02T07:52:16.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:16.091+0000 7ff52ae00700 1 -- 172.21.15.89:0/2315560536 wait complete. 2024-04-02T07:52:16.111 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T07:52:16.111 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T07:52:16.111 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T07:52:16.111 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T07:52:16.111 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T07:52:16.111 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T07:52:16.111 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T07:52:16.112 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T07:52:16.112 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:52:16.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:16.112 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T07:52:16.158 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:52:16.158 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:52:16.158 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:52:16.158 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:52:16.158 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T07:52:16.158 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:52:16.158 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:52:16.159 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:52:16.159 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T07:52:16.159 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:52:16.159 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:52:16.159 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T07:52:16.159 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:52:16.159 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:52:16.159 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T07:52:16.159 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:52:16.159 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T07:52:16.159 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 41493sec preferred_lft 41493sec 2024-04-02T07:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T07:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T07:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T07:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T07:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T07:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T07:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T07:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T07:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:52:16.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:16.162 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T07:52:16.262 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T07:52:16.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:16.265 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T07:52:16.312 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T07:52:16.312 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T07:52:16.312 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T07:52:16.312 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T07:52:16.312 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T07:52:16.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:16.313 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:52:16.313 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:52:16.313 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T07:52:16.314 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T07:52:16.314 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T07:52:16.314 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:52:16.416 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:52:16.420 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:52:16.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:16.476 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T07:52:16.520 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T07:52:16.521 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T07:52:16.521 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:52:16.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:16.522 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T07:52:16.632 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:52:16.632 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T07:52:16.676 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:52:16.677 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:52:16.677 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:52:16.761 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T07:52:16.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:16.762 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:52:16.902 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:52:16.902 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:52:16.930 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:52:16.930 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:52:16.930 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T07:52:16.930 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T07:52:16.930 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T07:52:16.936 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T07:52:16.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:16.936 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:52:17.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:17.002 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:52:17.101 INFO:teuthology.orchestra.run.smithi089.stdout:client6682 2024-04-02T07:52:17.101 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:52:17.101 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:17.146 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T07:52:17.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.377+0000 7fb8f1800700 1 Processor -- start 2024-04-02T07:52:17.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.377+0000 7fb8f1800700 1 -- start start 2024-04-02T07:52:17.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.379+0000 7fb8f1800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb8ec0d21f0 0x7fb8ec0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:17.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.379+0000 7fb8f1800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ec0ce270 con 0x7fb8ec1611b0 2024-04-02T07:52:17.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.379+0000 7fb8f1800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ec15b600 con 0x7fb8ec0cec40 2024-04-02T07:52:17.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.379+0000 7fb8f1800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ec0d2760 con 0x7fb8ec0d21f0 2024-04-02T07:52:17.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.379+0000 7fb8eaa00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb8ec1611b0 0x7fb8ec15ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60438/0 (socket says 172.21.15.89:60438) 2024-04-02T07:52:17.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.379+0000 7fb8eaa00700 1 -- 172.21.15.89:0/282480456 learned_addr learned my addr 172.21.15.89:0/282480456 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:17.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.379+0000 7fb8ea000700 1 --2- 172.21.15.89:0/282480456 >> v2:172.21.15.89:3300/0 conn(0x7fb8ec0d21f0 0x7fb8ec0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:17.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.379+0000 7fb8e9600700 1 -- 172.21.15.89:0/282480456 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156325670 0 0) 0x7fb8ec0ce270 con 0x7fb8ec1611b0 2024-04-02T07:52:17.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.379+0000 7fb8e9600700 1 -- 172.21.15.89:0/282480456 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d00035b0 con 0x7fb8ec1611b0 2024-04-02T07:52:17.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.379+0000 7fb8ea000700 1 -- 172.21.15.89:0/282480456 >> v1:172.21.15.113:6789/0 conn(0x7fb8ec0cec40 legacy=0x7fb8ec161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:17.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.379+0000 7fb8ea000700 1 -- 172.21.15.89:0/282480456 >> v1:172.21.15.89:6789/0 conn(0x7fb8ec1611b0 legacy=0x7fb8ec15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:17.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.379+0000 7fb8ea000700 1 -- 172.21.15.89:0/282480456 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ec0d2a40 con 0x7fb8ec0d21f0 2024-04-02T07:52:17.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.380+0000 7fb8ea000700 1 --2- 172.21.15.89:0/282480456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec0d21f0 0x7fb8ec0ce700 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8009840 tx=0x7fb8d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8980e45cb41fb0d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:17.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.380+0000 7fb8e9600700 1 -- 172.21.15.89:0/282480456 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8004230 con 0x7fb8ec0d21f0 2024-04-02T07:52:17.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.380+0000 7fb8e9600700 1 -- 172.21.15.89:0/282480456 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d8004390 con 0x7fb8ec0d21f0 2024-04-02T07:52:17.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.380+0000 7fb8e9600700 1 -- 172.21.15.89:0/282480456 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d80036a0 con 0x7fb8ec0d21f0 2024-04-02T07:52:17.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.380+0000 7fb8f1800700 1 -- 172.21.15.89:0/282480456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec0d21f0 msgr2=0x7fb8ec0ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:17.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.380+0000 7fb8f1800700 1 --2- 172.21.15.89:0/282480456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec0d21f0 0x7fb8ec0ce700 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8009840 tx=0x7fb8d8009b50 comp rx=0 tx=0).stop 2024-04-02T07:52:17.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.380+0000 7fb8f1800700 1 -- 172.21.15.89:0/282480456 shutdown_connections 2024-04-02T07:52:17.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.380+0000 7fb8f1800700 1 --2- 172.21.15.89:0/282480456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec0d21f0 0x7fb8ec0ce700 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:17.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.380+0000 7fb8f1800700 1 -- 172.21.15.89:0/282480456 >> 172.21.15.89:0/282480456 conn(0x7fb8ec08aff0 msgr2=0x7fb8ec0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:17.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.381+0000 7fb8f1800700 1 -- 172.21.15.89:0/282480456 shutdown_connections 2024-04-02T07:52:17.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.381+0000 7fb8f1800700 1 -- 172.21.15.89:0/282480456 wait complete. 2024-04-02T07:52:17.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.382+0000 7fb8f1800700 1 Processor -- start 2024-04-02T07:52:17.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.382+0000 7fb8f1800700 1 -- start start 2024-04-02T07:52:17.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.383+0000 7fb8f1800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8ec0cec40 0x7fb8ec15a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:17.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.383+0000 7fb8f1800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8ec0d21f0 0x7fb8ec0cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:17.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.383+0000 7fb8f1800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec1611b0 0x7fb8ec0ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:17.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.383+0000 7fb8f1800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec0b56e0 con 0x7fb8ec1611b0 2024-04-02T07:52:17.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.383+0000 7fb8f1800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec1f71f0 con 0x7fb8ec0cec40 2024-04-02T07:52:17.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.383+0000 7fb8f1800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec0b84b0 con 0x7fb8ec0d21f0 2024-04-02T07:52:17.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.384+0000 7fb8eb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec1611b0 0x7fb8ec0ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:17.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.384+0000 7fb8eb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec1611b0 0x7fb8ec0ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42148/0 (socket says 172.21.15.89:42148) 2024-04-02T07:52:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.384+0000 7fb8eb400700 1 -- 172.21.15.89:0/363422977 learned_addr learned my addr 172.21.15.89:0/363422977 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.384+0000 7fb8ea000700 1 --2- 172.21.15.89:0/363422977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8ec0d21f0 0x7fb8ec0cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.384+0000 7fb8eaa00700 1 --2- 172.21.15.89:0/363422977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8ec0cec40 0x7fb8ec15a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.384+0000 7fb8eb400700 1 -- 172.21.15.89:0/363422977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8ec0d21f0 msgr2=0x7fb8ec0cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.384+0000 7fb8eb400700 1 --2- 172.21.15.89:0/363422977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8ec0d21f0 0x7fb8ec0cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.384+0000 7fb8eb400700 1 -- 172.21.15.89:0/363422977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8ec0cec40 msgr2=0x7fb8ec15a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.384+0000 7fb8eb400700 1 --2- 172.21.15.89:0/363422977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8ec0cec40 0x7fb8ec15a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.384+0000 7fb8eb400700 1 -- 172.21.15.89:0/363422977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d80094a0 con 0x7fb8ec1611b0 2024-04-02T07:52:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.384+0000 7fb8eb400700 1 --2- 172.21.15.89:0/363422977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec1611b0 0x7fb8ec0ccd10 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fb8e400b970 tx=0x7fb8e400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.385+0000 7fb8e2a00700 1 -- 172.21.15.89:0/363422977 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e400d860 con 0x7fb8ec1611b0 2024-04-02T07:52:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.385+0000 7fb8e2a00700 1 -- 172.21.15.89:0/363422977 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e400d9c0 con 0x7fb8ec1611b0 2024-04-02T07:52:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.385+0000 7fb8e2a00700 1 -- 172.21.15.89:0/363422977 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e400dd70 con 0x7fb8ec1611b0 2024-04-02T07:52:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.385+0000 7fb8f1800700 1 -- 172.21.15.89:0/363422977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ec1f68c0 con 0x7fb8ec1611b0 2024-04-02T07:52:17.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.385+0000 7fb8f1800700 1 -- 172.21.15.89:0/363422977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ec1f6e80 con 0x7fb8ec1611b0 2024-04-02T07:52:17.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.386+0000 7fb8f1800700 1 -- 172.21.15.89:0/363422977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ec1f7380 con 0x7fb8ec1611b0 2024-04-02T07:52:17.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.387+0000 7fb8e2a00700 1 -- 172.21.15.89:0/363422977 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb8e400e430 con 0x7fb8ec1611b0 2024-04-02T07:52:17.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.387+0000 7fb8e2a00700 1 --2- 172.21.15.89:0/363422977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb8cc06adb0 0x7fb8cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:17.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.388+0000 7fb8e2a00700 1 -- 172.21.15.89:0/363422977 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb8e40898a0 con 0x7fb8ec1611b0 2024-04-02T07:52:17.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.388+0000 7fb8eaa00700 1 --2- 172.21.15.89:0/363422977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb8cc06adb0 0x7fb8cc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:17.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.388+0000 7fb8eaa00700 1 --2- 172.21.15.89:0/363422977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb8cc06adb0 0x7fb8cc06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc007fd0 tx=0x7fb8dc007dc0 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:17.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.392+0000 7fb8e2a00700 1 -- 172.21.15.89:0/363422977 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb8e4054660 con 0x7fb8ec1611b0 2024-04-02T07:52:17.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.621+0000 7fb8f1800700 1 -- 172.21.15.89:0/363422977 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8ec15f6a0 con 0x7fb8cc06adb0 2024-04-02T07:52:17.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.623+0000 7fb8e2a00700 1 -- 172.21.15.89:0/363422977 <== mgr.6696 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb8dc001370 con 0x7fb8cc06adb0 2024-04-02T07:52:17.624 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:52:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T07:52:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:17.625 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:52:17.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.625+0000 7fb8f1800700 1 -- 172.21.15.89:0/363422977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb8cc06adb0 msgr2=0x7fb8cc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:17.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.625+0000 7fb8f1800700 1 --2- 172.21.15.89:0/363422977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb8cc06adb0 0x7fb8cc06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc007fd0 tx=0x7fb8dc007dc0 comp rx=0 tx=0).stop 2024-04-02T07:52:17.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.625+0000 7fb8f1800700 1 -- 172.21.15.89:0/363422977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec1611b0 msgr2=0x7fb8ec0ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:17.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.625+0000 7fb8f1800700 1 --2- 172.21.15.89:0/363422977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec1611b0 0x7fb8ec0ccd10 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fb8e400b970 tx=0x7fb8e400bc80 comp rx=0 tx=0).stop 2024-04-02T07:52:17.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.625+0000 7fb8f1800700 1 -- 172.21.15.89:0/363422977 shutdown_connections 2024-04-02T07:52:17.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.625+0000 7fb8f1800700 1 --2- 172.21.15.89:0/363422977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb8cc06adb0 0x7fb8cc06d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:17.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.625+0000 7fb8f1800700 1 --2- 172.21.15.89:0/363422977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8ec0cec40 0x7fb8ec15a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:17.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.625+0000 7fb8f1800700 1 --2- 172.21.15.89:0/363422977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec1611b0 0x7fb8ec0ccd10 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:17.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.625+0000 7fb8f1800700 1 --2- 172.21.15.89:0/363422977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8ec0d21f0 0x7fb8ec0cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:17.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.625+0000 7fb8f1800700 1 -- 172.21.15.89:0/363422977 >> 172.21.15.89:0/363422977 conn(0x7fb8ec08aff0 msgr2=0x7fb8ec06bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:17.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.626+0000 7fb8f1800700 1 -- 172.21.15.89:0/363422977 shutdown_connections 2024-04-02T07:52:17.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.626+0000 7fb8f1800700 1 -- 172.21.15.89:0/363422977 wait complete. 2024-04-02T07:52:17.645 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T07:52:17.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.836+0000 7f26f8400700 1 Processor -- start 2024-04-02T07:52:17.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.837+0000 7f26f8400700 1 -- start start 2024-04-02T07:52:17.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.838+0000 7f26f8400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f00cbb30 0x7f26f00cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:17.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.838+0000 7f26f8400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f26f00cd020 0x7f26f00cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:17.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.838+0000 7f26f8400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f00cc450 con 0x7f26f00b2070 2024-04-02T07:52:17.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.838+0000 7f26f8400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f00cdae0 con 0x7f26f00cd020 2024-04-02T07:52:17.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.838+0000 7f26f8400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00cddc0 con 0x7f26f00cbb30 2024-04-02T07:52:17.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.838+0000 7f26f6400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f26f00b2070 0x7f26f006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60450/0 (socket says 172.21.15.89:60450) 2024-04-02T07:52:17.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.838+0000 7f26f6400700 1 -- 172.21.15.89:0/4203969603 learned_addr learned my addr 172.21.15.89:0/4203969603 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:17.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.838+0000 7f26f5000700 1 --2- 172.21.15.89:0/4203969603 >> v2:172.21.15.113:3300/0 conn(0x7f26f00cd020 0x7f26f00cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.838+0000 7f26f5a00700 1 --2- 172.21.15.89:0/4203969603 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f00cbb30 0x7f26f00cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.838+0000 7f26efe00700 1 -- 172.21.15.89:0/4203969603 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363533203 0 0) 0x7f26f00cc450 con 0x7f26f00b2070 2024-04-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 -- 172.21.15.89:0/4203969603 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26d80035b0 con 0x7f26f00b2070 2024-04-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 -- 172.21.15.89:0/4203969603 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4150869329 0 0) 0x7f26d80035b0 con 0x7f26f00b2070 2024-04-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 -- 172.21.15.89:0/4203969603 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f00cbb30 msgr2=0x7f26f00cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 --2- 172.21.15.89:0/4203969603 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f00cbb30 0x7f26f00cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 -- 172.21.15.89:0/4203969603 >> v2:172.21.15.113:3300/0 conn(0x7f26f00cd020 msgr2=0x7f26f00cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 --2- 172.21.15.89:0/4203969603 >> v2:172.21.15.113:3300/0 conn(0x7f26f00cd020 0x7f26f00cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 -- 172.21.15.89:0/4203969603 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f00ce0a0 con 0x7f26f00b2070 2024-04-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 -- 172.21.15.89:0/4203969603 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f26e80019f0 con 0x7f26f00b2070 2024-04-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26f5000700 1 --2- 172.21.15.89:0/4203969603 >> v2:172.21.15.113:3300/0 conn(0x7f26f00cd020 0x7f26f00cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 -- 172.21.15.89:0/4203969603 >> v1:172.21.15.89:6789/0 conn(0x7f26f00b2070 legacy=0x7f26f006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 --2- 172.21.15.89:0/4203969603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26d8003a00 0x7f26d8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 --2- 172.21.15.89:0/4203969603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26d8004850 0x7f26d8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 --2- 172.21.15.89:0/4203969603 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f00cbb30 0x7f26d8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 -- 172.21.15.89:0/4203969603 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00cdae0 con 0x7f26d8003a00 2024-04-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 -- 172.21.15.89:0/4203969603 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00cddc0 con 0x7f26d8004850 2024-04-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26efe00700 1 -- 172.21.15.89:0/4203969603 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d800ba10 con 0x7f26f00cbb30 2024-04-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26f5a00700 1 --2- 172.21.15.89:0/4203969603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26d8003a00 0x7f26d8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.839+0000 7f26f5000700 1 --2- 172.21.15.89:0/4203969603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26d8004850 0x7f26d8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.840+0000 7f26f6400700 1 --2- 172.21.15.89:0/4203969603 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f00cbb30 0x7f26d8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.840+0000 7f26f5a00700 1 -- 172.21.15.89:0/4203969603 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f00cbb30 msgr2=0x7f26d8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.840+0000 7f26f5a00700 1 --2- 172.21.15.89:0/4203969603 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f00cbb30 0x7f26d8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.840+0000 7f26f5a00700 1 -- 172.21.15.89:0/4203969603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26d8004850 msgr2=0x7f26d8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.840+0000 7f26f5a00700 1 --2- 172.21.15.89:0/4203969603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26d8004850 0x7f26d8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.840+0000 7f26f5a00700 1 -- 172.21.15.89:0/4203969603 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d80035b0 con 0x7f26d8003a00 2024-04-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.840+0000 7f26f5a00700 1 --2- 172.21.15.89:0/4203969603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26d8003a00 0x7f26d8003e20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009b60 tx=0x7f26d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5bbb6a2f6d1ffb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.840+0000 7f26efe00700 1 -- 172.21.15.89:0/4203969603 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e400e6c0 con 0x7f26d8003a00 2024-04-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.840+0000 7f26efe00700 1 -- 172.21.15.89:0/4203969603 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e400e820 con 0x7f26d8003a00 2024-04-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.840+0000 7f26efe00700 1 -- 172.21.15.89:0/4203969603 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e400eb90 con 0x7f26d8003a00 2024-04-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.841+0000 7f26f8400700 1 -- 172.21.15.89:0/4203969603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26d8003a00 msgr2=0x7f26d8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.841+0000 7f26f8400700 1 --2- 172.21.15.89:0/4203969603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26d8003a00 0x7f26d8003e20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009b60 tx=0x7f26d8005880 comp rx=0 tx=0).stop 2024-04-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.841+0000 7f26f6400700 1 -- 172.21.15.89:0/4203969603 reap_dead start 2024-04-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.841+0000 7f26f8400700 1 -- 172.21.15.89:0/4203969603 shutdown_connections 2024-04-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.841+0000 7f26f8400700 1 -- 172.21.15.89:0/4203969603 >> 172.21.15.89:0/4203969603 conn(0x7f26f00b0d70 msgr2=0x7f26f00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.841+0000 7f26f8400700 1 -- 172.21.15.89:0/4203969603 shutdown_connections 2024-04-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.841+0000 7f26f8400700 1 -- 172.21.15.89:0/4203969603 wait complete. 2024-04-02T07:52:17.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.843+0000 7f26f8400700 1 Processor -- start 2024-04-02T07:52:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.843+0000 7f26f8400700 1 -- start start 2024-04-02T07:52:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f8400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26d8003a00 0x7f26f014f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f8400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f00c83f0 0x7f26f00c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f8400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26f00ac3d0 0x7f26f00ac860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f8400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00c8db0 con 0x7f26f00ac3d0 2024-04-02T07:52:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f8400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00709c0 con 0x7f26d8003a00 2024-04-02T07:52:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f8400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00ad0a0 con 0x7f26f00c83f0 2024-04-02T07:52:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f6400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26f00ac3d0 0x7f26f00ac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f6400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26f00ac3d0 0x7f26f00ac860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42178/0 (socket says 172.21.15.89:42178) 2024-04-02T07:52:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f6400700 1 -- 172.21.15.89:0/3949885997 learned_addr learned my addr 172.21.15.89:0/3949885997 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f5a00700 1 --2- 172.21.15.89:0/3949885997 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26d8003a00 0x7f26f014f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:17.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f6400700 1 -- 172.21.15.89:0/3949885997 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f00c83f0 msgr2=0x7f26f00c8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f6400700 1 --2- 172.21.15.89:0/3949885997 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f00c83f0 0x7f26f00c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f6400700 1 -- 172.21.15.89:0/3949885997 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26d8003a00 msgr2=0x7f26f014f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f6400700 1 --2- 172.21.15.89:0/3949885997 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26d8003a00 0x7f26f014f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f6400700 1 -- 172.21.15.89:0/3949885997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f00ad380 con 0x7f26f00ac3d0 2024-04-02T07:52:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.844+0000 7f26f5a00700 1 --2- 172.21.15.89:0/3949885997 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26d8003a00 0x7f26f014f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.845+0000 7f26f6400700 1 --2- 172.21.15.89:0/3949885997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26f00ac3d0 0x7f26f00ac860 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f26e8000f80 tx=0x7f26e800e4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.845+0000 7f26ed600700 1 -- 172.21.15.89:0/3949885997 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e800ee00 con 0x7f26f00ac3d0 2024-04-02T07:52:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.845+0000 7f26ed600700 1 -- 172.21.15.89:0/3949885997 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e8013420 con 0x7f26f00ac3d0 2024-04-02T07:52:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.845+0000 7f26ed600700 1 -- 172.21.15.89:0/3949885997 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e80137d0 con 0x7f26f00ac3d0 2024-04-02T07:52:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.845+0000 7f26f8400700 1 -- 172.21.15.89:0/3949885997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f00ad660 con 0x7f26f00ac3d0 2024-04-02T07:52:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.845+0000 7f26f8400700 1 -- 172.21.15.89:0/3949885997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f00acda0 con 0x7f26f00ac3d0 2024-04-02T07:52:17.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.846+0000 7f26f8400700 1 -- 172.21.15.89:0/3949885997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f0070b50 con 0x7f26f00ac3d0 2024-04-02T07:52:17.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.847+0000 7f26ed600700 1 -- 172.21.15.89:0/3949885997 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 42) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f26e8013930 con 0x7f26f00ac3d0 2024-04-02T07:52:17.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.848+0000 7f26ed600700 1 --2- 172.21.15.89:0/3949885997 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26d006adb0 0x7f26d006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:17.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.848+0000 7f26ed600700 1 -- 172.21.15.89:0/3949885997 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f26e8089c80 con 0x7f26f00ac3d0 2024-04-02T07:52:17.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.848+0000 7f26f5a00700 1 --2- 172.21.15.89:0/3949885997 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26d006adb0 0x7f26d006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:17.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.849+0000 7f26f5a00700 1 --2- 172.21.15.89:0/3949885997 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26d006adb0 0x7f26d006d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f26f00cb090 tx=0x7f26e4008030 comp rx=0 tx=0).ready entity=mgr.6696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:17.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:17.852+0000 7f26ed600700 1 -- 172.21.15.89:0/3949885997 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f26e8054c80 con 0x7f26f00ac3d0 2024-04-02T07:52:18.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.112+0000 7f26f8400700 1 -- 172.21.15.89:0/3949885997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f26f0070b50 con 0x7f26f00ac3d0 2024-04-02T07:52:18.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.975+0000 7f26ed600700 1 -- 172.21.15.89:0/3949885997 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 43) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f26e8011dc0 con 0x7f26f00ac3d0 2024-04-02T07:52:18.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.976+0000 7f26ed600700 1 -- 172.21.15.89:0/3949885997 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v43) v1 ==== 92+0+0 (secure 0 0 0) 0x7f26e8058710 con 0x7f26f00ac3d0 2024-04-02T07:52:18.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.978+0000 7f26f8400700 1 -- 172.21.15.89:0/3949885997 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26d006adb0 msgr2=0x7f26d006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:18.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.978+0000 7f26f8400700 1 --2- 172.21.15.89:0/3949885997 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26d006adb0 0x7f26d006d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f26f00cb090 tx=0x7f26e4008030 comp rx=0 tx=0).stop 2024-04-02T07:52:18.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.978+0000 7f26f8400700 1 -- 172.21.15.89:0/3949885997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26f00ac3d0 msgr2=0x7f26f00ac860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:18.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.978+0000 7f26f8400700 1 --2- 172.21.15.89:0/3949885997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26f00ac3d0 0x7f26f00ac860 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f26e8000f80 tx=0x7f26e800e4f0 comp rx=0 tx=0).stop 2024-04-02T07:52:18.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.978+0000 7f26f8400700 1 -- 172.21.15.89:0/3949885997 shutdown_connections 2024-04-02T07:52:18.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.978+0000 7f26f8400700 1 --2- 172.21.15.89:0/3949885997 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f26d006adb0 0x7f26d006d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:18.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.978+0000 7f26f8400700 1 --2- 172.21.15.89:0/3949885997 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26d8003a00 0x7f26f014f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:18.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.978+0000 7f26f8400700 1 --2- 172.21.15.89:0/3949885997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26f00ac3d0 0x7f26f00ac860 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:18.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.978+0000 7f26f8400700 1 --2- 172.21.15.89:0/3949885997 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f00c83f0 0x7f26f00c8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:18.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.978+0000 7f26f8400700 1 -- 172.21.15.89:0/3949885997 >> 172.21.15.89:0/3949885997 conn(0x7f26f00b0d70 msgr2=0x7f26f006e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:18.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.979+0000 7f26f8400700 1 -- 172.21.15.89:0/3949885997 shutdown_connections 2024-04-02T07:52:18.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:18.979+0000 7f26f8400700 1 -- 172.21.15.89:0/3949885997 wait complete. 2024-04-02T07:52:19.002 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T07:52:19.192 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:19.190+0000 7fb57fdc7240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:52:19.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.203+0000 7f8574a00700 1 Processor -- start 2024-04-02T07:52:19.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.203+0000 7f8574a00700 1 -- start start 2024-04-02T07:52:19.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.204+0000 7f8574a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85700d16a0 0x7f85700d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:19.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.205+0000 7f8574a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f85700d2370 0x7f85700ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:19.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.205+0000 7f8574a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8570161420 con 0x7f857015d9d0 2024-04-02T07:52:19.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.205+0000 7f8574a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85700d2820 con 0x7f85700d2370 2024-04-02T07:52:19.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.205+0000 7f8574a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85700d4340 con 0x7f85700d16a0 2024-04-02T07:52:19.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.205+0000 7f856d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85700d16a0 0x7f85700d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:19.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.205+0000 7f856e800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f85700d2370 0x7f85700ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:19.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.205+0000 7f856de00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f857015d9d0 0x7f85700d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:34078/0 (socket says 172.21.15.89:34078) 2024-04-02T07:52:19.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.205+0000 7f856de00700 1 -- 172.21.15.89:0/2423801265 learned_addr learned my addr 172.21.15.89:0/2423801265 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:19.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.205+0000 7f856e800700 1 -- 172.21.15.89:0/2423801265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85700d16a0 msgr2=0x7f85700d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:19.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.205+0000 7f856e800700 1 --2- 172.21.15.89:0/2423801265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85700d16a0 0x7f85700d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.205+0000 7f856e800700 1 -- 172.21.15.89:0/2423801265 >> v1:172.21.15.113:6789/0 conn(0x7f857015d9d0 legacy=0x7f85700d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:19.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.205+0000 7f856e800700 1 -- 172.21.15.89:0/2423801265 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8570161700 con 0x7f85700d2370 2024-04-02T07:52:19.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.205+0000 7f856d400700 1 --2- 172.21.15.89:0/2423801265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85700d16a0 0x7f85700d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:19.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.205+0000 7f856ca00700 1 -- 172.21.15.89:0/2423801265 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252693176 0 0) 0x7f8570161420 con 0x7f857015d9d0 2024-04-02T07:52:19.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.206+0000 7f856e800700 1 --2- 172.21.15.89:0/2423801265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85700d2370 0x7f85700ce360 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8568009dc0 tx=0x7f856800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2716aa07e58fe9b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:19.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.206+0000 7f856ca00700 1 -- 172.21.15.89:0/2423801265 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856800e070 con 0x7f85700d2370 2024-04-02T07:52:19.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.206+0000 7f856ca00700 1 -- 172.21.15.89:0/2423801265 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f856800bae0 con 0x7f85700d2370 2024-04-02T07:52:19.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.206+0000 7f856ca00700 1 -- 172.21.15.89:0/2423801265 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856800be90 con 0x7f85700d2370 2024-04-02T07:52:19.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.207+0000 7f8574a00700 1 -- 172.21.15.89:0/2423801265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85700d2370 msgr2=0x7f85700ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:19.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.207+0000 7f8574a00700 1 --2- 172.21.15.89:0/2423801265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85700d2370 0x7f85700ce360 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8568009dc0 tx=0x7f856800b330 comp rx=0 tx=0).stop 2024-04-02T07:52:19.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.207+0000 7f8574a00700 1 -- 172.21.15.89:0/2423801265 shutdown_connections 2024-04-02T07:52:19.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.207+0000 7f8574a00700 1 --2- 172.21.15.89:0/2423801265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85700d2370 0x7f85700ce360 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.207+0000 7f8574a00700 1 --2- 172.21.15.89:0/2423801265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85700d16a0 0x7f85700d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.207+0000 7f8574a00700 1 -- 172.21.15.89:0/2423801265 >> 172.21.15.89:0/2423801265 conn(0x7f857008aff0 msgr2=0x7f85700d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:19.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.207+0000 7f8574a00700 1 -- 172.21.15.89:0/2423801265 shutdown_connections 2024-04-02T07:52:19.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.207+0000 7f8574a00700 1 -- 172.21.15.89:0/2423801265 wait complete. 2024-04-02T07:52:19.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.208+0000 7f8574a00700 1 Processor -- start 2024-04-02T07:52:19.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.209+0000 7f8574a00700 1 -- start start 2024-04-02T07:52:19.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.209+0000 7f8574a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85700d16a0 0x7f85701f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:19.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.209+0000 7f8574a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85700d2370 0x7f857015e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:19.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.209+0000 7f8574a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f857015d9d0 0x7f857014a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:19.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.209+0000 7f8574a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857014b510 con 0x7f85700d2370 2024-04-02T07:52:19.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.209+0000 7f8574a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857014af50 con 0x7f85700d16a0 2024-04-02T07:52:19.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.209+0000 7f8574a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857014b230 con 0x7f857015d9d0 2024-04-02T07:52:19.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.210+0000 7f856d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85700d2370 0x7f857015e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:19.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.210+0000 7f856e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f857015d9d0 0x7f857014a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:19.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.210+0000 7f856de00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85700d16a0 0x7f85701f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:19.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.210+0000 7f856d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85700d2370 0x7f857015e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39316/0 (socket says 172.21.15.89:39316) 2024-04-02T07:52:19.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.210+0000 7f856d400700 1 -- 172.21.15.89:0/426446450 learned_addr learned my addr 172.21.15.89:0/426446450 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:19.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.210+0000 7f856e800700 1 -- 172.21.15.89:0/426446450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85700d16a0 msgr2=0x7f85701f2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:19.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.210+0000 7f856e800700 1 --2- 172.21.15.89:0/426446450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85700d16a0 0x7f85701f2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.210+0000 7f856e800700 1 -- 172.21.15.89:0/426446450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85700d2370 msgr2=0x7f857015e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:19.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.210+0000 7f856e800700 1 --2- 172.21.15.89:0/426446450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85700d2370 0x7f857015e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.210+0000 7f856e800700 1 -- 172.21.15.89:0/426446450 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855c009430 con 0x7f857015d9d0 2024-04-02T07:52:19.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.210+0000 7f856e800700 1 --2- 172.21.15.89:0/426446450 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f857015d9d0 0x7f857014a500 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8568016000 tx=0x7f8568004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:19.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.211+0000 7f8566000700 1 -- 172.21.15.89:0/426446450 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8568027480 con 0x7f857015d9d0 2024-04-02T07:52:19.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.211+0000 7f8566000700 1 -- 172.21.15.89:0/426446450 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8568020070 con 0x7f857015d9d0 2024-04-02T07:52:19.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.211+0000 7f8566000700 1 -- 172.21.15.89:0/426446450 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8568027860 con 0x7f857015d9d0 2024-04-02T07:52:19.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.211+0000 7f8574a00700 1 -- 172.21.15.89:0/426446450 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8568009a70 con 0x7f857015d9d0 2024-04-02T07:52:19.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.211+0000 7f8574a00700 1 -- 172.21.15.89:0/426446450 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857015e230 con 0x7f857015d9d0 2024-04-02T07:52:19.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.212+0000 7f8574a00700 1 -- 172.21.15.89:0/426446450 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85700d1ca0 con 0x7f857015d9d0 2024-04-02T07:52:19.215 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:19.212+0000 7fbc7b540240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:52:19.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.213+0000 7f8566000700 1 -- 172.21.15.89:0/426446450 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 43) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f856801b400 con 0x7f857015d9d0 2024-04-02T07:52:19.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.214+0000 7f8566000700 1 --2- 172.21.15.89:0/426446450 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f855006ae00 0x7f855006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:19.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.214+0000 7f8566000700 1 -- 172.21.15.89:0/426446450 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8568022070 con 0x7f857015d9d0 2024-04-02T07:52:19.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.214+0000 7f856de00700 1 -- 172.21.15.89:0/426446450 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f855006ae00 msgr2=0x7f855006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:19.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.214+0000 7f856de00700 1 --2- 172.21.15.89:0/426446450 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f855006ae00 0x7f855006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:19.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.219+0000 7f8566000700 1 -- 172.21.15.89:0/426446450 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f85680561e0 con 0x7f857015d9d0 2024-04-02T07:52:19.378 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:19.376+0000 7fb57fdc7240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:52:19.402 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:19.401+0000 7fbc7b540240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:52:19.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.415+0000 7f856de00700 1 -- 172.21.15.89:0/426446450 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f855006ae00 msgr2=0x7f855006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:19.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.415+0000 7f856de00700 1 --2- 172.21.15.89:0/426446450 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f855006ae00 0x7f855006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:19.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.462+0000 7f8574a00700 1 -- 172.21.15.89:0/426446450 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f857015ddb0 con 0x7f857015d9d0 2024-04-02T07:52:19.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.470+0000 7f8566000700 1 -- 172.21.15.89:0/426446450 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f857015ddb0 con 0x7f857015d9d0 2024-04-02T07:52:19.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.472+0000 7f8574a00700 1 -- 172.21.15.89:0/426446450 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f855006ae00 msgr2=0x7f855006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:19.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.472+0000 7f8574a00700 1 --2- 172.21.15.89:0/426446450 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f855006ae00 0x7f855006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.472+0000 7f8574a00700 1 -- 172.21.15.89:0/426446450 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f857015d9d0 msgr2=0x7f857014a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:19.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.472+0000 7f8574a00700 1 --2- 172.21.15.89:0/426446450 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f857015d9d0 0x7f857014a500 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8568016000 tx=0x7f8568004ad0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.472+0000 7f8574a00700 1 -- 172.21.15.89:0/426446450 shutdown_connections 2024-04-02T07:52:19.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.472+0000 7f8574a00700 1 --2- 172.21.15.89:0/426446450 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f855006ae00 0x7f855006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.472+0000 7f8574a00700 1 --2- 172.21.15.89:0/426446450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85700d16a0 0x7f85701f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.472+0000 7f8574a00700 1 --2- 172.21.15.89:0/426446450 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f857015d9d0 0x7f857014a500 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.472+0000 7f8574a00700 1 --2- 172.21.15.89:0/426446450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85700d2370 0x7f857015e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.472+0000 7f8574a00700 1 -- 172.21.15.89:0/426446450 >> 172.21.15.89:0/426446450 conn(0x7f857008aff0 msgr2=0x7f857015b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:19.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.472+0000 7f8574a00700 1 -- 172.21.15.89:0/426446450 shutdown_connections 2024-04-02T07:52:19.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.472+0000 7f8574a00700 1 -- 172.21.15.89:0/426446450 wait complete. 2024-04-02T07:52:19.492 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T07:52:19.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.691+0000 7f1735600700 1 Processor -- start 2024-04-02T07:52:19.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.691+0000 7f1735600700 1 -- start start 2024-04-02T07:52:19.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.692+0000 7f1735600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f173015dad0 0x7f17300ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:19.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.692+0000 7f1735600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17300d1630 0x7f17300d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:19.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.692+0000 7f1735600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173015f820 con 0x7f17300d1fd0 2024-04-02T07:52:19.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.692+0000 7f1735600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17300cfcc0 con 0x7f173015dad0 2024-04-02T07:52:19.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.692+0000 7f1735600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17300cd9f0 con 0x7f17300d1630 2024-04-02T07:52:19.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.692+0000 7f172ea00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f173015dad0 0x7f17300ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:19.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.693+0000 7f172ea00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f173015dad0 0x7f17300ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:48304/0 (socket says 172.21.15.89:48304) 2024-04-02T07:52:19.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.693+0000 7f172ea00700 1 -- 172.21.15.89:0/2966459811 learned_addr learned my addr 172.21.15.89:0/2966459811 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:19.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.693+0000 7f172e000700 1 --2- 172.21.15.89:0/2966459811 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17300d1630 0x7f17300d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:19.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.693+0000 7f172ea00700 1 -- 172.21.15.89:0/2966459811 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17300d1630 msgr2=0x7f17300d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:19.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.693+0000 7f172ea00700 1 --2- 172.21.15.89:0/2966459811 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17300d1630 0x7f17300d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.693+0000 7f172ea00700 1 -- 172.21.15.89:0/2966459811 >> v1:172.21.15.113:6789/0 conn(0x7f17300d1fd0 legacy=0x7f173015deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:19.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.693+0000 7f172ea00700 1 -- 172.21.15.89:0/2966459811 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17300d2460 con 0x7f173015dad0 2024-04-02T07:52:19.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.693+0000 7f172ea00700 1 --2- 172.21.15.89:0/2966459811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173015dad0 0x7f17300ced80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f17200098f0 tx=0x7f1720009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdc7048eb2e054c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:19.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.693+0000 7f172d600700 1 -- 172.21.15.89:0/2966459811 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720011830 con 0x7f173015dad0 2024-04-02T07:52:19.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.694+0000 7f172d600700 1 -- 172.21.15.89:0/2966459811 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1720011990 con 0x7f173015dad0 2024-04-02T07:52:19.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.694+0000 7f172d600700 1 -- 172.21.15.89:0/2966459811 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720011d20 con 0x7f173015dad0 2024-04-02T07:52:19.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.694+0000 7f1735600700 1 -- 172.21.15.89:0/2966459811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173015dad0 msgr2=0x7f17300ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:19.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.694+0000 7f1735600700 1 --2- 172.21.15.89:0/2966459811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173015dad0 0x7f17300ced80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f17200098f0 tx=0x7f1720009c00 comp rx=0 tx=0).stop 2024-04-02T07:52:19.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.694+0000 7f1735600700 1 -- 172.21.15.89:0/2966459811 shutdown_connections 2024-04-02T07:52:19.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.694+0000 7f1735600700 1 --2- 172.21.15.89:0/2966459811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173015dad0 0x7f17300ced80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.694+0000 7f1735600700 1 --2- 172.21.15.89:0/2966459811 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17300d1630 0x7f17300d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.694+0000 7f1735600700 1 -- 172.21.15.89:0/2966459811 >> 172.21.15.89:0/2966459811 conn(0x7f173008aff0 msgr2=0x7f17300d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:19.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.694+0000 7f1735600700 1 -- 172.21.15.89:0/2966459811 shutdown_connections 2024-04-02T07:52:19.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.694+0000 7f1735600700 1 -- 172.21.15.89:0/2966459811 wait complete. 2024-04-02T07:52:19.699 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:19.697+0000 7fb57fdc7240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:52:19.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.696+0000 7f1735600700 1 Processor -- start 2024-04-02T07:52:19.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.696+0000 7f1735600700 1 -- start start 2024-04-02T07:52:19.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.697+0000 7f1735600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17300d1630 0x7f17301507f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:19.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.697+0000 7f1735600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17300d1fd0 0x7f17300b88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:19.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.697+0000 7f1735600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173015dad0 0x7f17300cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:19.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.697+0000 7f1735600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17300b84d0 con 0x7f17300d1fd0 2024-04-02T07:52:19.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.697+0000 7f1735600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17300c8480 con 0x7f173015dad0 2024-04-02T07:52:19.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.697+0000 7f1735600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1730150d30 con 0x7f17300d1630 2024-04-02T07:52:19.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.697+0000 7f172e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17300d1fd0 0x7f17300b88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:19.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.697+0000 7f172e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17300d1fd0 0x7f17300b88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39318/0 (socket says 172.21.15.89:39318) 2024-04-02T07:52:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.697+0000 7f172ea00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17300d1630 0x7f17301507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.697+0000 7f172e000700 1 -- 172.21.15.89:0/4203943690 learned_addr learned my addr 172.21.15.89:0/4203943690 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.697+0000 7f172f400700 1 --2- 172.21.15.89:0/4203943690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173015dad0 0x7f17300cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.698+0000 7f172e000700 1 -- 172.21.15.89:0/4203943690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17300d1630 msgr2=0x7f17301507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.698+0000 7f172e000700 1 --2- 172.21.15.89:0/4203943690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17300d1630 0x7f17301507f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.698+0000 7f172e000700 1 -- 172.21.15.89:0/4203943690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173015dad0 msgr2=0x7f17300cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.698+0000 7f172e000700 1 --2- 172.21.15.89:0/4203943690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173015dad0 0x7f17300cbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.698+0000 7f172e000700 1 -- 172.21.15.89:0/4203943690 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1720009550 con 0x7f17300d1fd0 2024-04-02T07:52:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.698+0000 7f172ea00700 1 --2- 172.21.15.89:0/4203943690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17300d1630 0x7f17301507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.698+0000 7f172e000700 1 --2- 172.21.15.89:0/4203943690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17300d1fd0 0x7f17300b88f0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f171c00e9e0 tx=0x7f171c00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:19.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.698+0000 7f1726a00700 1 -- 172.21.15.89:0/4203943690 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171c009710 con 0x7f17300d1fd0 2024-04-02T07:52:19.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.698+0000 7f1726a00700 1 -- 172.21.15.89:0/4203943690 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171c00ee80 con 0x7f17300d1fd0 2024-04-02T07:52:19.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.698+0000 7f1735600700 1 -- 172.21.15.89:0/4203943690 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17300b7850 con 0x7f17300d1fd0 2024-04-02T07:52:19.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.698+0000 7f1726a00700 1 -- 172.21.15.89:0/4203943690 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171c00ca50 con 0x7f17300d1fd0 2024-04-02T07:52:19.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.698+0000 7f1735600700 1 -- 172.21.15.89:0/4203943690 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17300b7e10 con 0x7f17300d1fd0 2024-04-02T07:52:19.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.699+0000 7f1735600700 1 -- 172.21.15.89:0/4203943690 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1730067720 con 0x7f17300d1fd0 2024-04-02T07:52:19.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.700+0000 7f1726a00700 1 -- 172.21.15.89:0/4203943690 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 43) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f171c00cbb0 con 0x7f17300d1fd0 2024-04-02T07:52:19.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.701+0000 7f1726a00700 1 --2- 172.21.15.89:0/4203943690 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f171006ae00 0x7f171006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:19.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.701+0000 7f172ea00700 1 -- 172.21.15.89:0/4203943690 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f171006ae00 msgr2=0x7f171006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:19.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.701+0000 7f1726a00700 1 -- 172.21.15.89:0/4203943690 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f171c089670 con 0x7f17300d1fd0 2024-04-02T07:52:19.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.701+0000 7f172ea00700 1 --2- 172.21.15.89:0/4203943690 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f171006ae00 0x7f171006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.707+0000 7f1726a00700 1 -- 172.21.15.89:0/4203943690 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f171c0545d0 con 0x7f17300d1fd0 2024-04-02T07:52:19.739 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:19.738+0000 7fbc7b540240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:52:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.902+0000 7f172ea00700 1 -- 172.21.15.89:0/4203943690 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f171006ae00 msgr2=0x7f171006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.902+0000 7f172ea00700 1 --2- 172.21.15.89:0/4203943690 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f171006ae00 0x7f171006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:19.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.918+0000 7f1735600700 1 -- 172.21.15.89:0/4203943690 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f173006d630 con 0x7f17300d1fd0 2024-04-02T07:52:19.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.924+0000 7f1726a00700 1 -- 172.21.15.89:0/4203943690 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f171c057bf0 con 0x7f17300d1fd0 2024-04-02T07:52:19.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.927+0000 7f1735600700 1 -- 172.21.15.89:0/4203943690 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f171006ae00 msgr2=0x7f171006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:19.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.927+0000 7f1735600700 1 --2- 172.21.15.89:0/4203943690 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f171006ae00 0x7f171006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.927+0000 7f1735600700 1 -- 172.21.15.89:0/4203943690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17300d1fd0 msgr2=0x7f17300b88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:19.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.927+0000 7f1735600700 1 --2- 172.21.15.89:0/4203943690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17300d1fd0 0x7f17300b88f0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f171c00e9e0 tx=0x7f171c00ecf0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.927+0000 7f1735600700 1 -- 172.21.15.89:0/4203943690 shutdown_connections 2024-04-02T07:52:19.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.927+0000 7f1735600700 1 --2- 172.21.15.89:0/4203943690 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f171006ae00 0x7f171006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.927+0000 7f1735600700 1 --2- 172.21.15.89:0/4203943690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173015dad0 0x7f17300cbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.927+0000 7f1735600700 1 --2- 172.21.15.89:0/4203943690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17300d1630 0x7f17301507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.927+0000 7f1735600700 1 --2- 172.21.15.89:0/4203943690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17300d1fd0 0x7f17300b88f0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:19.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.927+0000 7f1735600700 1 -- 172.21.15.89:0/4203943690 >> 172.21.15.89:0/4203943690 conn(0x7f173008aff0 msgr2=0x7f17300ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:19.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.927+0000 7f1735600700 1 -- 172.21.15.89:0/4203943690 shutdown_connections 2024-04-02T07:52:19.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:19.927+0000 7f1735600700 1 -- 172.21.15.89:0/4203943690 wait complete. 2024-04-02T07:52:19.947 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-04-02T07:52:20.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.155+0000 7f76fbe00700 1 Processor -- start 2024-04-02T07:52:20.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.156+0000 7f76fbe00700 1 -- start start 2024-04-02T07:52:20.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.157+0000 7f76fbe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f76fc15d2e0 0x7f76fc0b3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:20.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.157+0000 7f76fbe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76fc0cfcc0 0x7f76fc0aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:20.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.157+0000 7f76fbe00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76fc072230 con 0x7f76fc0af500 2024-04-02T07:52:20.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.157+0000 7f76fbe00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76fc0af9b0 con 0x7f76fc15d2e0 2024-04-02T07:52:20.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.157+0000 7f76fbe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc0726c0 con 0x7f76fc0cfcc0 2024-04-02T07:52:20.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.157+0000 7f76fb400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f76fc0af500 0x7f76fc160fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60160/0 (socket says 172.21.15.89:60160) 2024-04-02T07:52:20.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.157+0000 7f76fb400700 1 -- 172.21.15.89:0/1592216706 learned_addr learned my addr 172.21.15.89:0/1592216706 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:20.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.157+0000 7f76faa00700 1 --2- 172.21.15.89:0/1592216706 >> v2:172.21.15.113:3300/0 conn(0x7f76fc15d2e0 0x7f76fc0b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:20.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.157+0000 7f76fa000700 1 --2- 172.21.15.89:0/1592216706 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76fc0cfcc0 0x7f76fc0aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:20.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.158+0000 7f76f9600700 1 -- 172.21.15.89:0/1592216706 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144884933 0 0) 0x7f76fc072230 con 0x7f76fc0af500 2024-04-02T07:52:20.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.158+0000 7f76f9600700 1 -- 172.21.15.89:0/1592216706 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e00035b0 con 0x7f76fc0af500 2024-04-02T07:52:20.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.158+0000 7f76fa000700 1 -- 172.21.15.89:0/1592216706 >> v2:172.21.15.113:3300/0 conn(0x7f76fc15d2e0 msgr2=0x7f76fc0b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:20.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.158+0000 7f76fa000700 1 --2- 172.21.15.89:0/1592216706 >> v2:172.21.15.113:3300/0 conn(0x7f76fc15d2e0 0x7f76fc0b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:20.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.158+0000 7f76fa000700 1 -- 172.21.15.89:0/1592216706 >> v1:172.21.15.89:6789/0 conn(0x7f76fc0af500 legacy=0x7f76fc160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:20.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.158+0000 7f76fa000700 1 -- 172.21.15.89:0/1592216706 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76fc071900 con 0x7f76fc0cfcc0 2024-04-02T07:52:20.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.158+0000 7f76faa00700 1 --2- 172.21.15.89:0/1592216706 >> v2:172.21.15.113:3300/0 conn(0x7f76fc15d2e0 0x7f76fc0b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:20.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.158+0000 7f76fa000700 1 --2- 172.21.15.89:0/1592216706 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76fc0cfcc0 0x7f76fc0aefc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009840 tx=0x7f76e8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3c4dd70943c4fca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:20.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.158+0000 7f76f9600700 1 -- 172.21.15.89:0/1592216706 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e8011830 con 0x7f76fc0cfcc0 2024-04-02T07:52:20.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.158+0000 7f76f9600700 1 -- 172.21.15.89:0/1592216706 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e8011990 con 0x7f76fc0cfcc0 2024-04-02T07:52:20.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.158+0000 7f76f9600700 1 -- 172.21.15.89:0/1592216706 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e8011d20 con 0x7f76fc0cfcc0 2024-04-02T07:52:20.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.159+0000 7f76fbe00700 1 -- 172.21.15.89:0/1592216706 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76fc0cfcc0 msgr2=0x7f76fc0aefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:20.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.159+0000 7f76fbe00700 1 --2- 172.21.15.89:0/1592216706 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76fc0cfcc0 0x7f76fc0aefc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009840 tx=0x7f76e8009b50 comp rx=0 tx=0).stop 2024-04-02T07:52:20.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.159+0000 7f76fbe00700 1 -- 172.21.15.89:0/1592216706 shutdown_connections 2024-04-02T07:52:20.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.159+0000 7f76fbe00700 1 --2- 172.21.15.89:0/1592216706 >> v2:172.21.15.113:3300/0 conn(0x7f76fc15d2e0 0x7f76fc0b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:20.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.159+0000 7f76fbe00700 1 --2- 172.21.15.89:0/1592216706 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76fc0cfcc0 0x7f76fc0aefc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:20.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.159+0000 7f76fbe00700 1 -- 172.21.15.89:0/1592216706 >> 172.21.15.89:0/1592216706 conn(0x7f76fc0d00a0 msgr2=0x7f76fc0d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:20.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.159+0000 7f76fbe00700 1 -- 172.21.15.89:0/1592216706 shutdown_connections 2024-04-02T07:52:20.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.159+0000 7f76fbe00700 1 -- 172.21.15.89:0/1592216706 wait complete. 2024-04-02T07:52:20.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.161+0000 7f76fbe00700 1 Processor -- start 2024-04-02T07:52:20.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.161+0000 7f76fbe00700 1 -- start start 2024-04-02T07:52:20.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.162+0000 7f76fbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76fc0af500 0x7f76fc148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:20.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.162+0000 7f76fbe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76fc0cfcc0 0x7f76fc15f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:20.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.162+0000 7f76fbe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76fc15d2e0 0x7f76fc15fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:20.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.162+0000 7f76fbe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc15afa0 con 0x7f76fc0af500 2024-04-02T07:52:20.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.162+0000 7f76fbe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc15a390 con 0x7f76fc15d2e0 2024-04-02T07:52:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.162+0000 7f76fbe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc15a670 con 0x7f76fc0cfcc0 2024-04-02T07:52:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.162+0000 7f76faa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76fc0af500 0x7f76fc148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.162+0000 7f76faa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76fc0af500 0x7f76fc148b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39324/0 (socket says 172.21.15.89:39324) 2024-04-02T07:52:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.162+0000 7f76faa00700 1 -- 172.21.15.89:0/1733802419 learned_addr learned my addr 172.21.15.89:0/1733802419 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.162+0000 7f76fb400700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76fc15d2e0 0x7f76fc15fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.162+0000 7f76fa000700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76fc0cfcc0 0x7f76fc15f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.163+0000 7f76faa00700 1 -- 172.21.15.89:0/1733802419 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76fc0cfcc0 msgr2=0x7f76fc15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.163+0000 7f76faa00700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76fc0cfcc0 0x7f76fc15f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:20.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.163+0000 7f76faa00700 1 -- 172.21.15.89:0/1733802419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76fc15d2e0 msgr2=0x7f76fc15fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.163+0000 7f76faa00700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76fc15d2e0 0x7f76fc15fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.163+0000 7f76faa00700 1 -- 172.21.15.89:0/1733802419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec0094a0 con 0x7f76fc0af500 2024-04-02T07:52:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.163+0000 7f76fb400700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76fc15d2e0 0x7f76fc15fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.163+0000 7f76faa00700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76fc0af500 0x7f76fc148b50 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00ebe0 tx=0x7f76ec00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.163+0000 7f76f2a00700 1 -- 172.21.15.89:0/1733802419 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec00c850 con 0x7f76fc0af500 2024-04-02T07:52:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.163+0000 7f76f2a00700 1 -- 172.21.15.89:0/1733802419 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76ec00c9b0 con 0x7f76fc0af500 2024-04-02T07:52:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.163+0000 7f76fbe00700 1 -- 172.21.15.89:0/1733802419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e80094a0 con 0x7f76fc0af500 2024-04-02T07:52:20.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.163+0000 7f76f2a00700 1 -- 172.21.15.89:0/1733802419 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec00cd40 con 0x7f76fc0af500 2024-04-02T07:52:20.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.163+0000 7f76fbe00700 1 -- 172.21.15.89:0/1733802419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76fc15ac30 con 0x7f76fc0af500 2024-04-02T07:52:20.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.165+0000 7f76fbe00700 1 -- 172.21.15.89:0/1733802419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76fc04ea80 con 0x7f76fc0af500 2024-04-02T07:52:20.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.165+0000 7f76f2a00700 1 -- 172.21.15.89:0/1733802419 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 43) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f76ec013470 con 0x7f76fc0af500 2024-04-02T07:52:20.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.166+0000 7f76f2a00700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 0x7f76dc06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:20.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.166+0000 7f76f2a00700 1 -- 172.21.15.89:0/1733802419 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f76ec0897d0 con 0x7f76fc0af500 2024-04-02T07:52:20.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.166+0000 7f76fa000700 1 -- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 msgr2=0x7f76dc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:20.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.166+0000 7f76fa000700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 0x7f76dc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:20.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.174+0000 7f76f2a00700 1 -- 172.21.15.89:0/1733802419 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f76ec0546c0 con 0x7f76fc0af500 2024-04-02T07:52:20.223 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:20.222+0000 7fb57fdc7240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:52:20.303 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:20.302+0000 7fbc7b540240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:52:20.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.367+0000 7f76fa000700 1 -- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 msgr2=0x7f76dc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:20.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.367+0000 7f76fa000700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 0x7f76dc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:20.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.412+0000 7f76fbe00700 1 -- 172.21.15.89:0/1733802419 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f76fc0b3fe0 con 0x7f76dc06ae00 2024-04-02T07:52:20.476 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:20.474+0000 7fb57fdc7240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:52:20.561 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:20.560+0000 7fbc7b540240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:52:20.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.768+0000 7f76fa000700 1 -- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 msgr2=0x7f76dc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:20.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:20.768+0000 7f76fa000700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 0x7f76dc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T07:52:20.928 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:20.926+0000 7fb57fdc7240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:52:21.019 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:21.018+0000 7fbc7b540240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:52:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:21.569+0000 7f76fa000700 1 -- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 msgr2=0x7f76dc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:21.569+0000 7f76fa000700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 0x7f76dc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T07:52:21.632 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:21.631+0000 7fb57fdc7240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:52:21.720 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:21.719+0000 7fbc7b540240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:52:21.723 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:21.721+0000 7fb57fdc7240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:52:21.811 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:21.810+0000 7fbc7b540240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:52:21.881 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:21.878+0000 7fb57fdc7240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:52:21.969 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:21.968+0000 7fbc7b540240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:52:22.092 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:22.090+0000 7fb57fdc7240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:52:22.182 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:22.181+0000 7fbc7b540240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:52:22.182 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:22.181+0000 7fb57fdc7240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:52:22.274 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:22.272+0000 7fbc7b540240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:52:22.281 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:22.279+0000 7fb57fdc7240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:52:22.371 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:22.370+0000 7fbc7b540240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:52:22.378 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:22.376+0000 7fb57fdc7240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:52:22.470 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:22.469+0000 7fbc7b540240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:52:22.475 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:22.473+0000 7fb57fdc7240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:52:22.569 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:22.567+0000 7fbc7b540240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:52:22.685 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:22.683+0000 7fb57fdc7240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:52:22.781 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:22.779+0000 7fbc7b540240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:52:22.938 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:22.937+0000 7fb57fdc7240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:52:23.038 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:23.037+0000 7fbc7b540240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:52:23.128 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:23.126+0000 7fb57fdc7240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:52:23.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:23.171+0000 7f76fa000700 1 -- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 msgr2=0x7f76dc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:23.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:23.171+0000 7f76fa000700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 0x7f76dc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T07:52:23.220 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:23.219+0000 7fb57fdc7240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:52:23.231 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:23.230+0000 7fbc7b540240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:52:23.323 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:23.322+0000 7fbc7b540240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:52:23.943 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:23.941+0000 7fb57fdc7240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:52:24.049 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:24.047+0000 7fbc7b540240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:52:24.150 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:24.149+0000 7fb57fdc7240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:52:24.260 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:24.259+0000 7fbc7b540240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:52:24.357 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:24.355+0000 7fb57fdc7240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:52:24.467 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:24.465+0000 7fbc7b540240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:52:24.829 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:24.827+0000 7fb57fdc7240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:52:24.941 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:24.940+0000 7fbc7b540240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:52:25.032 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:25.031+0000 7fb57fdc7240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:52:25.151 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:25.150+0000 7fbc7b540240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:52:25.469 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:25.468+0000 7fb57fdc7240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:52:25.577 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:25.577+0000 7fb57fdc7240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:52:25.592 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:25.591+0000 7fbc7b540240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:52:25.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:25.637+0000 7f76f2a00700 1 -- 172.21.15.89:0/1733802419 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 44) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f76ec087b90 con 0x7f76fc0af500 2024-04-02T07:52:25.708 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:25.707+0000 7fbc7b540240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:52:25.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:25.717+0000 7f76f2a00700 1 -- 172.21.15.89:0/1733802419 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 45) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f76ec081990 con 0x7f76fc0af500 2024-04-02T07:52:25.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:25.718+0000 7f76f2a00700 1 -- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 msgr2=0x7f76dc06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:25.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:25.718+0000 7f76f2a00700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 0x7f76dc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:25.890 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:25.889+0000 7fbbe3c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:25.891 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:25.889+0000 7fbbe3c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:25.891 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:25.889+0000 7fbbe3c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:25.891 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:25.889+0000 7fbbe3c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:25.891 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:25.889+0000 7fbbe3c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:25.892 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:25.890+0000 7fbbe6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:25.893 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:25.890+0000 7fbbe6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:25.893 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:25.890+0000 7fbbe6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:25.893 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:25.890+0000 7fbbe6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:25.893 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:25.890+0000 7fbbe6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:26.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.722+0000 7f76f2a00700 1 -- 172.21.15.89:0/1733802419 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mgrmap(e 46) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f76ec055360 con 0x7f76fc0af500 2024-04-02T07:52:26.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.722+0000 7f76f2a00700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 0x7f76dc06d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:26.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.722+0000 7f76f2a00700 1 -- 172.21.15.89:0/1733802419 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f76dc01d0c0 con 0x7f76dc06ae00 2024-04-02T07:52:26.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.724+0000 7f76fa000700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 0x7f76dc06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:26.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.727+0000 7f76fa000700 1 --2- 172.21.15.89:0/1733802419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f76dc06ae00 0x7f76dc06d240 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f76fc06b500 tx=0x7f76e8008030 comp rx=0 tx=0).ready entity=mgr.7104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:26.737 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:26.736+0000 7fbbf0400700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-04-02T07:52:26.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.737+0000 7f76f2a00700 1 -- 172.21.15.89:0/1733802419 <== mgr.7104 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f76dc01d0c0 con 0x7f76dc06ae00 2024-04-02T07:52:26.740 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-04-02T07:52:26.748 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-02T07:52:26.749 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T07:52:26.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.942+0000 7fb441a00700 1 Processor -- start 2024-04-02T07:52:26.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.942+0000 7fb441a00700 1 -- start start 2024-04-02T07:52:26.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.943+0000 7fb441a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb43c15d9d0 0x7fb43c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:26.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.943+0000 7fb441a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb43c0d2540 0x7fb43c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:26.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.943+0000 7fb441a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43c161420 con 0x7fb43c0d16a0 2024-04-02T07:52:26.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.943+0000 7fb441a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43c15edc0 con 0x7fb43c15d9d0 2024-04-02T07:52:26.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.943+0000 7fb441a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c161700 con 0x7fb43c0d2540 2024-04-02T07:52:26.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.943+0000 7fb43a000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb43c0d16a0 0x7fb43c0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60168/0 (socket says 172.21.15.89:60168) 2024-04-02T07:52:26.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.943+0000 7fb43a000700 1 -- 172.21.15.89:0/2166955031 learned_addr learned my addr 172.21.15.89:0/2166955031 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:26.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.943+0000 7fb43aa00700 1 --2- 172.21.15.89:0/2166955031 >> v2:172.21.15.113:3300/0 conn(0x7fb43c15d9d0 0x7fb43c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:26.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.943+0000 7fb43b400700 1 --2- 172.21.15.89:0/2166955031 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb43c0d2540 0x7fb43c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 -- 172.21.15.89:0/2166955031 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786066113 0 0) 0x7fb43c161420 con 0x7fb43c0d16a0 2024-04-02T07:52:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 -- 172.21.15.89:0/2166955031 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4200035b0 con 0x7fb43c0d16a0 2024-04-02T07:52:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 -- 172.21.15.89:0/2166955031 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1850511772 0 0) 0x7fb4200035b0 con 0x7fb43c0d16a0 2024-04-02T07:52:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 -- 172.21.15.89:0/2166955031 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb43c0d2540 msgr2=0x7fb43c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 --2- 172.21.15.89:0/2166955031 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb43c0d2540 0x7fb43c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 -- 172.21.15.89:0/2166955031 >> v2:172.21.15.113:3300/0 conn(0x7fb43c15d9d0 msgr2=0x7fb43c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 --2- 172.21.15.89:0/2166955031 >> v2:172.21.15.113:3300/0 conn(0x7fb43c15d9d0 0x7fb43c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 -- 172.21.15.89:0/2166955031 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43c1619e0 con 0x7fb43c0d16a0 2024-04-02T07:52:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 -- 172.21.15.89:0/2166955031 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fb4280017f0 con 0x7fb43c0d16a0 2024-04-02T07:52:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 -- 172.21.15.89:0/2166955031 >> v1:172.21.15.89:6789/0 conn(0x7fb43c0d16a0 legacy=0x7fb43c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:26.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 --2- 172.21.15.89:0/2166955031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb420003a00 0x7fb420003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:26.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 --2- 172.21.15.89:0/2166955031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb420004850 0x7fb420004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:26.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 --2- 172.21.15.89:0/2166955031 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb43c0d2540 0x7fb420005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:26.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 -- 172.21.15.89:0/2166955031 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c161700 con 0x7fb420003a00 2024-04-02T07:52:26.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 -- 172.21.15.89:0/2166955031 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42000ba10 con 0x7fb420004850 2024-04-02T07:52:26.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb439600700 1 -- 172.21.15.89:0/2166955031 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42000bb50 con 0x7fb43c0d2540 2024-04-02T07:52:26.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb43aa00700 1 --2- 172.21.15.89:0/2166955031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb420003a00 0x7fb420003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:26.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb43a000700 1 --2- 172.21.15.89:0/2166955031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb420004850 0x7fb420004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:26.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.944+0000 7fb43b400700 1 --2- 172.21.15.89:0/2166955031 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb43c0d2540 0x7fb420005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:26.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.945+0000 7fb43aa00700 1 -- 172.21.15.89:0/2166955031 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb43c0d2540 msgr2=0x7fb420005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:26.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.945+0000 7fb43aa00700 1 --2- 172.21.15.89:0/2166955031 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb43c0d2540 0x7fb420005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:26.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.945+0000 7fb43aa00700 1 -- 172.21.15.89:0/2166955031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb420004850 msgr2=0x7fb420004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:26.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.945+0000 7fb43aa00700 1 --2- 172.21.15.89:0/2166955031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb420004850 0x7fb420004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.945+0000 7fb43aa00700 1 -- 172.21.15.89:0/2166955031 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4200035b0 con 0x7fb420003a00 2024-04-02T07:52:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.945+0000 7fb43b400700 1 --2- 172.21.15.89:0/2166955031 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb43c0d2540 0x7fb420005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.945+0000 7fb43a000700 1 --2- 172.21.15.89:0/2166955031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb420004850 0x7fb420004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.945+0000 7fb43aa00700 1 --2- 172.21.15.89:0/2166955031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb420003a00 0x7fb420003e20 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009b60 tx=0x7fb420005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8e2ddc4e9228aa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.945+0000 7fb439600700 1 -- 172.21.15.89:0/2166955031 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c00e6c0 con 0x7fb420003a00 2024-04-02T07:52:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.945+0000 7fb439600700 1 -- 172.21.15.89:0/2166955031 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c00e820 con 0x7fb420003a00 2024-04-02T07:52:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.945+0000 7fb439600700 1 -- 172.21.15.89:0/2166955031 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c00eb90 con 0x7fb420003a00 2024-04-02T07:52:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.946+0000 7fb441a00700 1 -- 172.21.15.89:0/2166955031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb420003a00 msgr2=0x7fb420003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.946+0000 7fb441a00700 1 --2- 172.21.15.89:0/2166955031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb420003a00 0x7fb420003e20 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009b60 tx=0x7fb420005880 comp rx=0 tx=0).stop 2024-04-02T07:52:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.946+0000 7fb43b400700 1 -- 172.21.15.89:0/2166955031 reap_dead start 2024-04-02T07:52:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.946+0000 7fb441a00700 1 -- 172.21.15.89:0/2166955031 shutdown_connections 2024-04-02T07:52:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.946+0000 7fb441a00700 1 -- 172.21.15.89:0/2166955031 >> 172.21.15.89:0/2166955031 conn(0x7fb43c08aff0 msgr2=0x7fb43c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.946+0000 7fb441a00700 1 -- 172.21.15.89:0/2166955031 shutdown_connections 2024-04-02T07:52:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.946+0000 7fb441a00700 1 -- 172.21.15.89:0/2166955031 wait complete. 2024-04-02T07:52:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.948+0000 7fb441a00700 1 Processor -- start 2024-04-02T07:52:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.948+0000 7fb441a00700 1 -- start start 2024-04-02T07:52:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.949+0000 7fb441a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb420003a00 0x7fb43c0b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb441a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb43c14e8e0 0x7fb43c0ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb441a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb43c15f8e0 0x7fb43c15fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb441a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c1602a0 con 0x7fb420003a00 2024-04-02T07:52:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb441a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c0cd870 con 0x7fb43c14e8e0 2024-04-02T07:52:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb441a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c0b7e10 con 0x7fb43c15f8e0 2024-04-02T07:52:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb43aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb420003a00 0x7fb43c0b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb43aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb420003a00 0x7fb43c0b8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39456/0 (socket says 172.21.15.89:39456) 2024-04-02T07:52:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb43aa00700 1 -- 172.21.15.89:0/1867650588 learned_addr learned my addr 172.21.15.89:0/1867650588 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb43a000700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb43c14e8e0 0x7fb43c0ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb43b400700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb43c15f8e0 0x7fb43c15fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb43aa00700 1 -- 172.21.15.89:0/1867650588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb43c15f8e0 msgr2=0x7fb43c15fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb43aa00700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb43c15f8e0 0x7fb43c15fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb43aa00700 1 -- 172.21.15.89:0/1867650588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb43c14e8e0 msgr2=0x7fb43c0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb43aa00700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb43c14e8e0 0x7fb43c0ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb43aa00700 1 -- 172.21.15.89:0/1867650588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43c0b80f0 con 0x7fb420003a00 2024-04-02T07:52:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb43a000700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb43c14e8e0 0x7fb43c0ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:26.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.950+0000 7fb43aa00700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb420003a00 0x7fb43c0b8eb0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb42c0090b0 tx=0x7fb42c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:26.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.951+0000 7fb432a00700 1 -- 172.21.15.89:0/1867650588 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c00e6c0 con 0x7fb420003a00 2024-04-02T07:52:26.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.951+0000 7fb432a00700 1 -- 172.21.15.89:0/1867650588 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c003ad0 con 0x7fb420003a00 2024-04-02T07:52:26.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.951+0000 7fb432a00700 1 -- 172.21.15.89:0/1867650588 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c003e80 con 0x7fb420003a00 2024-04-02T07:52:26.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.951+0000 7fb441a00700 1 -- 172.21.15.89:0/1867650588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb43c0b83d0 con 0x7fb420003a00 2024-04-02T07:52:26.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.951+0000 7fb441a00700 1 -- 172.21.15.89:0/1867650588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb43c0b8990 con 0x7fb420003a00 2024-04-02T07:52:26.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.952+0000 7fb441a00700 1 -- 172.21.15.89:0/1867650588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb43c0d1ca0 con 0x7fb420003a00 2024-04-02T07:52:26.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.953+0000 7fb432a00700 1 -- 172.21.15.89:0/1867650588 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 46) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fb42c013470 con 0x7fb420003a00 2024-04-02T07:52:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.954+0000 7fb432a00700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb41c06ab30 0x7fb41c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.954+0000 7fb432a00700 1 -- 172.21.15.89:0/1867650588 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fb42c08acb0 con 0x7fb420003a00 2024-04-02T07:52:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.954+0000 7fb43a000700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb41c06ab30 0x7fb41c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.955+0000 7fb43a000700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb41c06ab30 0x7fb41c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb4280011b0 tx=0x7fb428009040 comp rx=0 tx=0).ready entity=mgr.7104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:26.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:26.959+0000 7fb432a00700 1 -- 172.21.15.89:0/1867650588 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb42c056030 con 0x7fb420003a00 2024-04-02T07:52:27.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.219+0000 7fb441a00700 1 -- 172.21.15.89:0/1867650588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fb43c0b8b20 con 0x7fb420003a00 2024-04-02T07:52:27.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.730+0000 7fb432a00700 1 -- 172.21.15.89:0/1867650588 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 47) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fb42c056400 con 0x7fb420003a00 2024-04-02T07:52:27.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.731+0000 7fb432a00700 1 -- 172.21.15.89:0/1867650588 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v47) v1 ==== 93+0+0 (secure 0 0 0) 0x7fb42c05be50 con 0x7fb420003a00 2024-04-02T07:52:27.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.733+0000 7fb441a00700 1 -- 172.21.15.89:0/1867650588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb41c06ab30 msgr2=0x7fb41c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:27.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.733+0000 7fb441a00700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb41c06ab30 0x7fb41c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb4280011b0 tx=0x7fb428009040 comp rx=0 tx=0).stop 2024-04-02T07:52:27.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.734+0000 7fb441a00700 1 -- 172.21.15.89:0/1867650588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb420003a00 msgr2=0x7fb43c0b8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:27.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.734+0000 7fb441a00700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb420003a00 0x7fb43c0b8eb0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb42c0090b0 tx=0x7fb42c0047c0 comp rx=0 tx=0).stop 2024-04-02T07:52:27.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.734+0000 7fb441a00700 1 -- 172.21.15.89:0/1867650588 shutdown_connections 2024-04-02T07:52:27.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.734+0000 7fb441a00700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb41c06ab30 0x7fb41c06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:27.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.734+0000 7fb441a00700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb43c15f8e0 0x7fb43c15fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:27.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.734+0000 7fb441a00700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb420003a00 0x7fb43c0b8eb0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:27.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.734+0000 7fb441a00700 1 --2- 172.21.15.89:0/1867650588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb43c14e8e0 0x7fb43c0ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:27.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.734+0000 7fb441a00700 1 -- 172.21.15.89:0/1867650588 >> 172.21.15.89:0/1867650588 conn(0x7fb43c08aff0 msgr2=0x7fb43c06cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:27.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.734+0000 7fb441a00700 1 -- 172.21.15.89:0/1867650588 shutdown_connections 2024-04-02T07:52:27.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:27.734+0000 7fb441a00700 1 -- 172.21.15.89:0/1867650588 wait complete. 2024-04-02T07:52:27.755 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:52:27.755 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:52:27.844 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:52:27.933 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:52:27.933 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:52:27.933 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:27.957 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:27.957+0000 7fb9f298d240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:52:27.961 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:27.960+0000 7fcb1564a240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:52:27.978 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:52:27.978 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:28.064 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:52:28.143 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:28.141+0000 7fb9f298d240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:52:28.154 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:28.152+0000 7fcb1564a240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:52:28.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.320+0000 7fc9b6200700 1 Processor -- start 2024-04-02T07:52:28.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.320+0000 7fc9b6200700 1 -- start start 2024-04-02T07:52:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.321+0000 7fc9b6200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc9b015b5d0 0x7fc9b00ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.321+0000 7fc9b6200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc9b00cb3f0 0x7fc9b00ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.321+0000 7fc9b6200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b00ae930 0x7fc9b00ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.321+0000 7fc9b6200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b015b9b0 con 0x7fc9b015b5d0 2024-04-02T07:52:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.321+0000 7fc9b6200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b015fd50 con 0x7fc9b00cb3f0 2024-04-02T07:52:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.321+0000 7fc9b6200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b015f140 con 0x7fc9b00ae930 2024-04-02T07:52:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.322+0000 7fc9af400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc9b015b5d0 0x7fc9b00ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.322+0000 7fc9af400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc9b015b5d0 0x7fc9b00ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39472/0 (socket says 172.21.15.89:39472) 2024-04-02T07:52:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.322+0000 7fc9af400700 1 -- 172.21.15.89:0/1438252497 learned_addr learned my addr 172.21.15.89:0/1438252497 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.322+0000 7fc9aea00700 1 --2- 172.21.15.89:0/1438252497 >> v2:172.21.15.113:3300/0 conn(0x7fc9b00cb3f0 0x7fc9b00ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.322+0000 7fc9afe00700 1 --2- 172.21.15.89:0/1438252497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b00ae930 0x7fc9b00ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.322+0000 7fc9af400700 1 -- 172.21.15.89:0/1438252497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b00ae930 msgr2=0x7fc9b00ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:28.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.322+0000 7fc9af400700 1 --2- 172.21.15.89:0/1438252497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b00ae930 0x7fc9b00ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.322+0000 7fc9af400700 1 -- 172.21.15.89:0/1438252497 >> v2:172.21.15.113:3300/0 conn(0x7fc9b00cb3f0 msgr2=0x7fc9b00ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:28.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.322+0000 7fc9af400700 1 --2- 172.21.15.89:0/1438252497 >> v2:172.21.15.113:3300/0 conn(0x7fc9b00cb3f0 0x7fc9b00ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.322+0000 7fc9af400700 1 -- 172.21.15.89:0/1438252497 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b015f420 con 0x7fc9b015b5d0 2024-04-02T07:52:28.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.322+0000 7fc9aea00700 1 --2- 172.21.15.89:0/1438252497 >> v2:172.21.15.113:3300/0 conn(0x7fc9b00cb3f0 0x7fc9b00ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:28.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.322+0000 7fc9af400700 1 --2- 172.21.15.89:0/1438252497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9b015b5d0 0x7fc9b00ae3f0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc9b006c6c0 tx=0x7fc9a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d817995d8a509bac server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:28.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.323+0000 7fc9ae000700 1 -- 172.21.15.89:0/1438252497 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a000b870 con 0x7fc9b015b5d0 2024-04-02T07:52:28.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.323+0000 7fc9ae000700 1 -- 172.21.15.89:0/1438252497 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a000b9d0 con 0x7fc9b015b5d0 2024-04-02T07:52:28.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.323+0000 7fc9ae000700 1 -- 172.21.15.89:0/1438252497 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a000bd60 con 0x7fc9b015b5d0 2024-04-02T07:52:28.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.323+0000 7fc9b6200700 1 -- 172.21.15.89:0/1438252497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9b015b5d0 msgr2=0x7fc9b00ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:28.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.323+0000 7fc9b6200700 1 --2- 172.21.15.89:0/1438252497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9b015b5d0 0x7fc9b00ae3f0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc9b006c6c0 tx=0x7fc9a0009bd0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.323+0000 7fc9b6200700 1 -- 172.21.15.89:0/1438252497 shutdown_connections 2024-04-02T07:52:28.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.323+0000 7fc9b6200700 1 --2- 172.21.15.89:0/1438252497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9b015b5d0 0x7fc9b00ae3f0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.323+0000 7fc9b6200700 1 --2- 172.21.15.89:0/1438252497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b00ae930 0x7fc9b00ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.323+0000 7fc9b6200700 1 --2- 172.21.15.89:0/1438252497 >> v2:172.21.15.113:3300/0 conn(0x7fc9b00cb3f0 0x7fc9b00ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.323+0000 7fc9b6200700 1 -- 172.21.15.89:0/1438252497 >> 172.21.15.89:0/1438252497 conn(0x7fc9b00cf190 msgr2=0x7fc9b00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:28.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.323+0000 7fc9b6200700 1 -- 172.21.15.89:0/1438252497 shutdown_connections 2024-04-02T07:52:28.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.324+0000 7fc9b6200700 1 -- 172.21.15.89:0/1438252497 wait complete. 2024-04-02T07:52:28.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.325+0000 7fc9b6200700 1 Processor -- start 2024-04-02T07:52:28.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.325+0000 7fc9b6200700 1 -- start start 2024-04-02T07:52:28.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.326+0000 7fc9b6200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b00ae930 0x7fc9b01507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:28.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.326+0000 7fc9b6200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9b015b5d0 0x7fc9b014de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:28.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.326+0000 7fc9b6200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b0098830 0x7fc9b00cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:28.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.326+0000 7fc9b6200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b0152d90 con 0x7fc9b015b5d0 2024-04-02T07:52:28.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.326+0000 7fc9b6200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b0150450 con 0x7fc9b0098830 2024-04-02T07:52:28.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.326+0000 7fc9b6200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b01558e0 con 0x7fc9b00ae930 2024-04-02T07:52:28.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.327+0000 7fc9aea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9b015b5d0 0x7fc9b014de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:28.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.327+0000 7fc9aea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9b015b5d0 0x7fc9b014de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39480/0 (socket says 172.21.15.89:39480) 2024-04-02T07:52:28.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.327+0000 7fc9aea00700 1 -- 172.21.15.89:0/175744729 learned_addr learned my addr 172.21.15.89:0/175744729 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:28.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.327+0000 7fc9af400700 1 --2- 172.21.15.89:0/175744729 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b00ae930 0x7fc9b01507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:28.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.327+0000 7fc9afe00700 1 --2- 172.21.15.89:0/175744729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b0098830 0x7fc9b00cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:28.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.327+0000 7fc9aea00700 1 -- 172.21.15.89:0/175744729 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b00ae930 msgr2=0x7fc9b01507e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:28.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.327+0000 7fc9aea00700 1 --2- 172.21.15.89:0/175744729 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b00ae930 0x7fc9b01507e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.327+0000 7fc9aea00700 1 -- 172.21.15.89:0/175744729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b0098830 msgr2=0x7fc9b00cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:28.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.327+0000 7fc9aea00700 1 --2- 172.21.15.89:0/175744729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b0098830 0x7fc9b00cd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.327+0000 7fc9aea00700 1 -- 172.21.15.89:0/175744729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a0009550 con 0x7fc9b015b5d0 2024-04-02T07:52:28.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.327+0000 7fc9aea00700 1 --2- 172.21.15.89:0/175744729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9b015b5d0 0x7fc9b014de60 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00e840 tx=0x7fc99c00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:28.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.328+0000 7fc9a7400700 1 -- 172.21.15.89:0/175744729 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c00c070 con 0x7fc9b015b5d0 2024-04-02T07:52:28.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.328+0000 7fc9a7400700 1 -- 172.21.15.89:0/175744729 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99c004230 con 0x7fc9b015b5d0 2024-04-02T07:52:28.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.328+0000 7fc9a7400700 1 -- 172.21.15.89:0/175744729 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0036a0 con 0x7fc9b015b5d0 2024-04-02T07:52:28.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.328+0000 7fc9b6200700 1 -- 172.21.15.89:0/175744729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b0155600 con 0x7fc9b015b5d0 2024-04-02T07:52:28.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.328+0000 7fc9b6200700 1 -- 172.21.15.89:0/175744729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b00cc930 con 0x7fc9b015b5d0 2024-04-02T07:52:28.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.329+0000 7fc9b6200700 1 -- 172.21.15.89:0/175744729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b006b6d0 con 0x7fc9b015b5d0 2024-04-02T07:52:28.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.330+0000 7fc9a7400700 1 -- 172.21.15.89:0/175744729 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 47) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fc99c003800 con 0x7fc9b015b5d0 2024-04-02T07:52:28.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.331+0000 7fc9a7400700 1 --2- 172.21.15.89:0/175744729 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc99006adb0 0x7fc99006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:28.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.331+0000 7fc9a7400700 1 -- 172.21.15.89:0/175744729 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fc99c089550 con 0x7fc9b015b5d0 2024-04-02T07:52:28.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.331+0000 7fc9af400700 1 -- 172.21.15.89:0/175744729 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc99006adb0 msgr2=0x7fc99006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:28.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.331+0000 7fc9af400700 1 --2- 172.21.15.89:0/175744729 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc99006adb0 0x7fc99006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:28.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.335+0000 7fc9a7400700 1 -- 172.21.15.89:0/175744729 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc99c0541b0 con 0x7fc9b015b5d0 2024-04-02T07:52:28.460 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:28.459+0000 7fb9f298d240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:52:28.485 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:28.484+0000 7fcb1564a240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:52:28.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.532+0000 7fc9af400700 1 -- 172.21.15.89:0/175744729 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc99006adb0 msgr2=0x7fc99006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:28.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.532+0000 7fc9af400700 1 --2- 172.21.15.89:0/175744729 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc99006adb0 0x7fc99006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:28.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.547+0000 7fc9b6200700 1 -- 172.21.15.89:0/175744729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9b00adb70 con 0x7fc9b015b5d0 2024-04-02T07:52:28.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.548+0000 7fc9a7400700 1 -- 172.21.15.89:0/175744729 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+17058 (secure 0 0 0) 0x7fc99c0094c0 con 0x7fc9b015b5d0 2024-04-02T07:52:28.550 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:28.551 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":98,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:52:25.709797+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:52:06.991300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-02T07:52:08.054066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/1957897648":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/1991189445":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/4248118690":"2024-04-03T07:52:25.709764+0000","172.21.15.89:6833/43357":"2024-04-03T07:52:25.709764+0000","172.21.15.89:6832/43357":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/3231846713":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/3820568899":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/3513660462":"2024-04-03T07:52:25.709764+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:52:28.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.551+0000 7fc9b6200700 1 -- 172.21.15.89:0/175744729 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc99006adb0 msgr2=0x7fc99006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:28.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.551+0000 7fc9b6200700 1 --2- 172.21.15.89:0/175744729 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc99006adb0 0x7fc99006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.551+0000 7fc9b6200700 1 -- 172.21.15.89:0/175744729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9b015b5d0 msgr2=0x7fc9b014de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:28.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.551+0000 7fc9b6200700 1 --2- 172.21.15.89:0/175744729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9b015b5d0 0x7fc9b014de60 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00e840 tx=0x7fc99c00eb50 comp rx=0 tx=0).stop 2024-04-02T07:52:28.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.551+0000 7fc9b6200700 1 -- 172.21.15.89:0/175744729 shutdown_connections 2024-04-02T07:52:28.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.551+0000 7fc9b6200700 1 --2- 172.21.15.89:0/175744729 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc99006adb0 0x7fc99006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.551+0000 7fc9b6200700 1 --2- 172.21.15.89:0/175744729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b0098830 0x7fc9b00cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.551+0000 7fc9b6200700 1 --2- 172.21.15.89:0/175744729 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b00ae930 0x7fc9b01507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.551+0000 7fc9b6200700 1 --2- 172.21.15.89:0/175744729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9b015b5d0 0x7fc9b014de60 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.551+0000 7fc9b6200700 1 -- 172.21.15.89:0/175744729 >> 172.21.15.89:0/175744729 conn(0x7fc9b00cf190 msgr2=0x7fc9b015cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:28.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.551+0000 7fc9b6200700 1 -- 172.21.15.89:0/175744729 shutdown_connections 2024-04-02T07:52:28.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.551+0000 7fc9b6200700 1 -- 172.21.15.89:0/175744729 wait complete. 2024-04-02T07:52:28.571 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:52:06.991300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:52:08.054066+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:52:28.572 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:52:28.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.811+0000 7fb8c8600700 1 Processor -- start 2024-04-02T07:52:28.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.811+0000 7fb8c8600700 1 -- start start 2024-04-02T07:52:28.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.812+0000 7fb8c8600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8c0160d70 0x7fb8c00b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:28.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.812+0000 7fb8c8600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb8c015ef70 0x7fb8c015e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:28.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.812+0000 7fb8c8600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c0161150 con 0x7fb8c006a790 2024-04-02T07:52:28.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.812+0000 7fb8c8600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8c00b4cd0 con 0x7fb8c015ef70 2024-04-02T07:52:28.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.812+0000 7fb8c8600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c006dba0 con 0x7fb8c0160d70 2024-04-02T07:52:28.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.813+0000 7fb8c5200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb8c015ef70 0x7fb8c015e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:28.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.813+0000 7fb8c5200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb8c015ef70 0x7fb8c015e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54390/0 (socket says 172.21.15.89:54390) 2024-04-02T07:52:28.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.813+0000 7fb8c5200700 1 -- 172.21.15.89:0/2911002334 learned_addr learned my addr 172.21.15.89:0/2911002334 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:28.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.813+0000 7fb8c5c00700 1 --2- 172.21.15.89:0/2911002334 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8c0160d70 0x7fb8c00b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:28.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.813+0000 7fb8c5200700 1 -- 172.21.15.89:0/2911002334 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8c0160d70 msgr2=0x7fb8c00b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:28.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.813+0000 7fb8c5200700 1 --2- 172.21.15.89:0/2911002334 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8c0160d70 0x7fb8c00b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.813+0000 7fb8c5200700 1 -- 172.21.15.89:0/2911002334 >> v1:172.21.15.113:6789/0 conn(0x7fb8c006a790 legacy=0x7fb8c01613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:28.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.813+0000 7fb8c5200700 1 -- 172.21.15.89:0/2911002334 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c015ed30 con 0x7fb8c015ef70 2024-04-02T07:52:28.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.813+0000 7fb8c5200700 1 --2- 172.21.15.89:0/2911002334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8c015ef70 0x7fb8c015e7f0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0098f0 tx=0x7fb8ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1196dea74b81ff server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:28.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.813+0000 7fb8bfe00700 1 -- 172.21.15.89:0/2911002334 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac011830 con 0x7fb8c015ef70 2024-04-02T07:52:28.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.814+0000 7fb8bfe00700 1 -- 172.21.15.89:0/2911002334 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ac011990 con 0x7fb8c015ef70 2024-04-02T07:52:28.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.814+0000 7fb8bfe00700 1 -- 172.21.15.89:0/2911002334 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac011d20 con 0x7fb8c015ef70 2024-04-02T07:52:28.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.814+0000 7fb8c8600700 1 -- 172.21.15.89:0/2911002334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8c015ef70 msgr2=0x7fb8c015e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:28.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.814+0000 7fb8c8600700 1 --2- 172.21.15.89:0/2911002334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8c015ef70 0x7fb8c015e7f0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0098f0 tx=0x7fb8ac009c00 comp rx=0 tx=0).stop 2024-04-02T07:52:28.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.814+0000 7fb8c8600700 1 -- 172.21.15.89:0/2911002334 shutdown_connections 2024-04-02T07:52:28.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.814+0000 7fb8c8600700 1 --2- 172.21.15.89:0/2911002334 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8c0160d70 0x7fb8c00b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.814+0000 7fb8c8600700 1 --2- 172.21.15.89:0/2911002334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8c015ef70 0x7fb8c015e7f0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.814+0000 7fb8c8600700 1 -- 172.21.15.89:0/2911002334 >> 172.21.15.89:0/2911002334 conn(0x7fb8c00b5720 msgr2=0x7fb8c00b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:28.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.814+0000 7fb8c8600700 1 -- 172.21.15.89:0/2911002334 shutdown_connections 2024-04-02T07:52:28.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.815+0000 7fb8c8600700 1 -- 172.21.15.89:0/2911002334 wait complete. 2024-04-02T07:52:28.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.816+0000 7fb8c8600700 1 Processor -- start 2024-04-02T07:52:28.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.816+0000 7fb8c8600700 1 -- start start 2024-04-02T07:52:28.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.817+0000 7fb8c8600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8c006a790 0x7fb8c009b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.817+0000 7fb8c8600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8c015ef70 0x7fb8c0144300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.817+0000 7fb8c8600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8c0160d70 0x7fb8c00c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.817+0000 7fb8c8600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c009c5a0 con 0x7fb8c006a790 2024-04-02T07:52:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.817+0000 7fb8c8600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c00c9ac0 con 0x7fb8c015ef70 2024-04-02T07:52:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.817+0000 7fb8c8600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c009b920 con 0x7fb8c0160d70 2024-04-02T07:52:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.818+0000 7fb8c5c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8c006a790 0x7fb8c009b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.818+0000 7fb8c5c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8c006a790 0x7fb8c009b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54398/0 (socket says 172.21.15.89:54398) 2024-04-02T07:52:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.818+0000 7fb8c5c00700 1 -- 172.21.15.89:0/3423192368 learned_addr learned my addr 172.21.15.89:0/3423192368 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.818+0000 7fb8c5200700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8c015ef70 0x7fb8c0144300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.818+0000 7fb8c6600700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8c0160d70 0x7fb8c00c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.818+0000 7fb8c5c00700 1 -- 172.21.15.89:0/3423192368 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8c0160d70 msgr2=0x7fb8c00c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:28.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.818+0000 7fb8c5c00700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8c0160d70 0x7fb8c00c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.818+0000 7fb8c5c00700 1 -- 172.21.15.89:0/3423192368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8c015ef70 msgr2=0x7fb8c0144300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:28.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.818+0000 7fb8c5c00700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8c015ef70 0x7fb8c0144300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:28.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.818+0000 7fb8c5c00700 1 -- 172.21.15.89:0/3423192368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac009550 con 0x7fb8c006a790 2024-04-02T07:52:28.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.818+0000 7fb8c5200700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8c015ef70 0x7fb8c0144300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:28.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.818+0000 7fb8c5c00700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8c006a790 0x7fb8c009b340 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb8b400e870 tx=0x7fb8b400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:28.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.819+0000 7fb8bd600700 1 -- 172.21.15.89:0/3423192368 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4009590 con 0x7fb8c006a790 2024-04-02T07:52:28.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.819+0000 7fb8bd600700 1 -- 172.21.15.89:0/3423192368 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b4013680 con 0x7fb8c006a790 2024-04-02T07:52:28.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.819+0000 7fb8bd600700 1 -- 172.21.15.89:0/3423192368 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b40139f0 con 0x7fb8c006a790 2024-04-02T07:52:28.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.819+0000 7fb8c8600700 1 -- 172.21.15.89:0/3423192368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c009bc00 con 0x7fb8c006a790 2024-04-02T07:52:28.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.819+0000 7fb8c8600700 1 -- 172.21.15.89:0/3423192368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c009c1c0 con 0x7fb8c006a790 2024-04-02T07:52:28.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.820+0000 7fb8c8600700 1 -- 172.21.15.89:0/3423192368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c009c880 con 0x7fb8c006a790 2024-04-02T07:52:28.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.821+0000 7fb8bd600700 1 -- 172.21.15.89:0/3423192368 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 47) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fb8b4013b50 con 0x7fb8c006a790 2024-04-02T07:52:28.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.822+0000 7fb8bd600700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb8a006adb0 0x7fb8a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:28.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.822+0000 7fb8bd600700 1 -- 172.21.15.89:0/3423192368 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fb8b4089650 con 0x7fb8c006a790 2024-04-02T07:52:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.822+0000 7fb8c5200700 1 -- 172.21.15.89:0/3423192368 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb8a006adb0 msgr2=0x7fb8a006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.822+0000 7fb8c5200700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb8a006adb0 0x7fb8a006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:28.830+0000 7fb8bd600700 1 -- 172.21.15.89:0/3423192368 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb8b40543c0 con 0x7fb8c006a790 2024-04-02T07:52:28.983 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:28.983+0000 7fb9f298d240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:52:29.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.023+0000 7fb8c5200700 1 -- 172.21.15.89:0/3423192368 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb8a006adb0 msgr2=0x7fb8a006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:29.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.023+0000 7fb8c5200700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb8a006adb0 0x7fb8a006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:29.038 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:29.037+0000 7fcb1564a240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:52:29.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.041+0000 7fb8c8600700 1 -- 172.21.15.89:0/3423192368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb8c0067720 con 0x7fb8c006a790 2024-04-02T07:52:29.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.041+0000 7fb8bd600700 1 -- 172.21.15.89:0/3423192368 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb8b40579e0 con 0x7fb8c006a790 2024-04-02T07:52:29.043 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:52:29.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.044+0000 7fb8c8600700 1 -- 172.21.15.89:0/3423192368 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb8a006adb0 msgr2=0x7fb8a006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:29.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.044+0000 7fb8c8600700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb8a006adb0 0x7fb8a006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.044+0000 7fb8c8600700 1 -- 172.21.15.89:0/3423192368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8c006a790 msgr2=0x7fb8c009b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:29.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.044+0000 7fb8c8600700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8c006a790 0x7fb8c009b340 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb8b400e870 tx=0x7fb8b400eb80 comp rx=0 tx=0).stop 2024-04-02T07:52:29.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.044+0000 7fb8c8600700 1 -- 172.21.15.89:0/3423192368 shutdown_connections 2024-04-02T07:52:29.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.044+0000 7fb8c8600700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb8a006adb0 0x7fb8a006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.044+0000 7fb8c8600700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8c0160d70 0x7fb8c00c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.044+0000 7fb8c8600700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8c006a790 0x7fb8c009b340 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.044+0000 7fb8c8600700 1 --2- 172.21.15.89:0/3423192368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8c015ef70 0x7fb8c0144300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.044+0000 7fb8c8600700 1 -- 172.21.15.89:0/3423192368 >> 172.21.15.89:0/3423192368 conn(0x7fb8c00b5720 msgr2=0x7fb8c0146c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:29.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.044+0000 7fb8c8600700 1 -- 172.21.15.89:0/3423192368 shutdown_connections 2024-04-02T07:52:29.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.044+0000 7fb8c8600700 1 -- 172.21.15.89:0/3423192368 wait complete. 2024-04-02T07:52:29.064 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:52:29.236 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:29.235+0000 7fb9f298d240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:52:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.268+0000 7f147bc00700 1 Processor -- start 2024-04-02T07:52:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.268+0000 7f147bc00700 1 -- start start 2024-04-02T07:52:29.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.269+0000 7f147bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14740ced80 0x7f14740d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:29.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.269+0000 7f147bc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f147415fcb0 0x7f1474160140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:29.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.269+0000 7f147bc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14740cf1e0 con 0x7f147415dad0 2024-04-02T07:52:29.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.269+0000 7f147bc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14740cfcc0 con 0x7f147415fcb0 2024-04-02T07:52:29.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.269+0000 7f147bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14740cd9f0 con 0x7f14740ced80 2024-04-02T07:52:29.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.270+0000 7f1479c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f147415fcb0 0x7f1474160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:29.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.270+0000 7f1479c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f147415fcb0 0x7f1474160140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54414/0 (socket says 172.21.15.89:54414) 2024-04-02T07:52:29.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.270+0000 7f1479c00700 1 -- 172.21.15.89:0/4158498349 learned_addr learned my addr 172.21.15.89:0/4158498349 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:29.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.270+0000 7f1479200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f147415dad0 0x7f14740d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33076/0 (socket says 172.21.15.89:33076) 2024-04-02T07:52:29.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.270+0000 7f1473e00700 1 --2- 172.21.15.89:0/4158498349 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14740ced80 0x7f14740d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:29.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.270+0000 7f1479c00700 1 -- 172.21.15.89:0/4158498349 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14740ced80 msgr2=0x7f14740d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:29.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.270+0000 7f1479c00700 1 --2- 172.21.15.89:0/4158498349 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14740ced80 0x7f14740d1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.270+0000 7f1479c00700 1 -- 172.21.15.89:0/4158498349 >> v1:172.21.15.89:6789/0 conn(0x7f147415dad0 legacy=0x7f14740d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:29.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.270+0000 7f1479c00700 1 -- 172.21.15.89:0/4158498349 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147415f820 con 0x7f147415fcb0 2024-04-02T07:52:29.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.270+0000 7f1479c00700 1 --2- 172.21.15.89:0/4158498349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415fcb0 0x7f1474160140 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f146c009e50 tx=0x7f146c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ed970d8cf9e778 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:29.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.271+0000 7f1473400700 1 -- 172.21.15.89:0/4158498349 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c00e070 con 0x7f147415fcb0 2024-04-02T07:52:29.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.271+0000 7f1473400700 1 -- 172.21.15.89:0/4158498349 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146c00b910 con 0x7f147415fcb0 2024-04-02T07:52:29.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.271+0000 7f1473400700 1 -- 172.21.15.89:0/4158498349 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c00bcc0 con 0x7f147415fcb0 2024-04-02T07:52:29.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.271+0000 7f147bc00700 1 -- 172.21.15.89:0/4158498349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415fcb0 msgr2=0x7f1474160140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:29.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.271+0000 7f147bc00700 1 --2- 172.21.15.89:0/4158498349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415fcb0 0x7f1474160140 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f146c009e50 tx=0x7f146c00b330 comp rx=0 tx=0).stop 2024-04-02T07:52:29.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.271+0000 7f147bc00700 1 -- 172.21.15.89:0/4158498349 shutdown_connections 2024-04-02T07:52:29.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.271+0000 7f147bc00700 1 --2- 172.21.15.89:0/4158498349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415fcb0 0x7f1474160140 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.271+0000 7f147bc00700 1 --2- 172.21.15.89:0/4158498349 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14740ced80 0x7f14740d1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.271+0000 7f147bc00700 1 -- 172.21.15.89:0/4158498349 >> 172.21.15.89:0/4158498349 conn(0x7f147408aff0 msgr2=0x7f14740d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:29.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.271+0000 7f147bc00700 1 -- 172.21.15.89:0/4158498349 shutdown_connections 2024-04-02T07:52:29.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.272+0000 7f147bc00700 1 -- 172.21.15.89:0/4158498349 wait complete. 2024-04-02T07:52:29.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.273+0000 7f147bc00700 1 Processor -- start 2024-04-02T07:52:29.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.273+0000 7f147bc00700 1 -- start start 2024-04-02T07:52:29.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.274+0000 7f147bc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14740ced80 0x7f14740b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:29.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.274+0000 7f147bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f147415dad0 0x7f1474151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:29.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.274+0000 7f147bc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415fcb0 0x7f14740b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:29.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.274+0000 7f147bc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14740b6370 con 0x7f147415fcb0 2024-04-02T07:52:29.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.274+0000 7f147bc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14740b6c10 con 0x7f14740ced80 2024-04-02T07:52:29.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.274+0000 7f147bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14740b8230 con 0x7f147415dad0 2024-04-02T07:52:29.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.275+0000 7f1479c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415fcb0 0x7f14740b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:29.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.275+0000 7f1479c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415fcb0 0x7f14740b77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54430/0 (socket says 172.21.15.89:54430) 2024-04-02T07:52:29.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.275+0000 7f1479c00700 1 -- 172.21.15.89:0/1238760419 learned_addr learned my addr 172.21.15.89:0/1238760419 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:29.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.275+0000 7f1473e00700 1 --2- 172.21.15.89:0/1238760419 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f147415dad0 0x7f1474151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:29.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.275+0000 7f1479200700 1 --2- 172.21.15.89:0/1238760419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14740ced80 0x7f14740b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:29.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.275+0000 7f1479c00700 1 -- 172.21.15.89:0/1238760419 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f147415dad0 msgr2=0x7f1474151060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:29.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.275+0000 7f1479c00700 1 --2- 172.21.15.89:0/1238760419 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f147415dad0 0x7f1474151060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.275+0000 7f1479c00700 1 -- 172.21.15.89:0/1238760419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14740ced80 msgr2=0x7f14740b70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:29.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.275+0000 7f1479c00700 1 --2- 172.21.15.89:0/1238760419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14740ced80 0x7f14740b70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.275+0000 7f1479c00700 1 -- 172.21.15.89:0/1238760419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c009b30 con 0x7f147415fcb0 2024-04-02T07:52:29.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.275+0000 7f1479c00700 1 --2- 172.21.15.89:0/1238760419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415fcb0 0x7f14740b77b0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f147408ce40 tx=0x7f146c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:29.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.276+0000 7f1470c00700 1 -- 172.21.15.89:0/1238760419 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c004800 con 0x7f147415fcb0 2024-04-02T07:52:29.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.276+0000 7f1470c00700 1 -- 172.21.15.89:0/1238760419 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146c004960 con 0x7f147415fcb0 2024-04-02T07:52:29.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.276+0000 7f1470c00700 1 -- 172.21.15.89:0/1238760419 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c01b670 con 0x7f147415fcb0 2024-04-02T07:52:29.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.276+0000 7f147bc00700 1 -- 172.21.15.89:0/1238760419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14740b65c0 con 0x7f147415fcb0 2024-04-02T07:52:29.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.276+0000 7f147bc00700 1 -- 172.21.15.89:0/1238760419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14740c8f40 con 0x7f147415fcb0 2024-04-02T07:52:29.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.277+0000 7f147bc00700 1 -- 172.21.15.89:0/1238760419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14740b6d80 con 0x7f147415fcb0 2024-04-02T07:52:29.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.278+0000 7f1470c00700 1 -- 172.21.15.89:0/1238760419 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 47) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f146c01b7d0 con 0x7f147415fcb0 2024-04-02T07:52:29.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.279+0000 7f1470c00700 1 --2- 172.21.15.89:0/1238760419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f145406ade0 0x7f145406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:29.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.279+0000 7f1470c00700 1 -- 172.21.15.89:0/1238760419 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f146c08afa0 con 0x7f147415fcb0 2024-04-02T07:52:29.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.279+0000 7f1479200700 1 -- 172.21.15.89:0/1238760419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f145406ade0 msgr2=0x7f145406d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:29.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.279+0000 7f1479200700 1 --2- 172.21.15.89:0/1238760419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f145406ade0 0x7f145406d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:29.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.283+0000 7f1470c00700 1 -- 172.21.15.89:0/1238760419 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f146c055d30 con 0x7f147415fcb0 2024-04-02T07:52:29.312 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:29.311+0000 7fcb1564a240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:52:29.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.480+0000 7f1479200700 1 -- 172.21.15.89:0/1238760419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f145406ade0 msgr2=0x7f145406d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:29.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.480+0000 7f1479200700 1 --2- 172.21.15.89:0/1238760419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f145406ade0 0x7f145406d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:29.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.495+0000 7f147bc00700 1 -- 172.21.15.89:0/1238760419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1474067720 con 0x7f147415fcb0 2024-04-02T07:52:29.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.495+0000 7f1470c00700 1 -- 172.21.15.89:0/1238760419 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7f146c059350 con 0x7f147415fcb0 2024-04-02T07:52:29.497 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:52:29.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.498+0000 7f147bc00700 1 -- 172.21.15.89:0/1238760419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f145406ade0 msgr2=0x7f145406d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:29.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.498+0000 7f147bc00700 1 --2- 172.21.15.89:0/1238760419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f145406ade0 0x7f145406d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.498+0000 7f147bc00700 1 -- 172.21.15.89:0/1238760419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415fcb0 msgr2=0x7f14740b77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.498+0000 7f147bc00700 1 --2- 172.21.15.89:0/1238760419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415fcb0 0x7f14740b77b0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f147408ce40 tx=0x7f146c004ad0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.498+0000 7f147bc00700 1 -- 172.21.15.89:0/1238760419 shutdown_connections 2024-04-02T07:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.498+0000 7f147bc00700 1 --2- 172.21.15.89:0/1238760419 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f145406ade0 0x7f145406d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.498+0000 7f147bc00700 1 --2- 172.21.15.89:0/1238760419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14740ced80 0x7f14740b70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.498+0000 7f147bc00700 1 --2- 172.21.15.89:0/1238760419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415fcb0 0x7f14740b77b0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.498+0000 7f147bc00700 1 --2- 172.21.15.89:0/1238760419 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f147415dad0 0x7f1474151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.498+0000 7f147bc00700 1 -- 172.21.15.89:0/1238760419 >> 172.21.15.89:0/1238760419 conn(0x7f147408aff0 msgr2=0x7f147406d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.498+0000 7f147bc00700 1 -- 172.21.15.89:0/1238760419 shutdown_connections 2024-04-02T07:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.498+0000 7f147bc00700 1 -- 172.21.15.89:0/1238760419 wait complete. 2024-04-02T07:52:29.519 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:52:29.692 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:29.691+0000 7fb9f298d240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:52:29.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.719+0000 7fdde6e00700 1 Processor -- start 2024-04-02T07:52:29.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.719+0000 7fdde6e00700 1 -- start start 2024-04-02T07:52:29.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.720+0000 7fdde6e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdde00ced80 0x7fdde00d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:29.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.720+0000 7fdde6e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdde015fcb0 0x7fdde0160140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.720+0000 7fdde6e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde00cf1e0 con 0x7fdde015dad0 2024-04-02T07:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.720+0000 7fdde6e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde00cfcc0 con 0x7fdde015fcb0 2024-04-02T07:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.720+0000 7fdde6e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde00cd9f0 con 0x7fdde00ced80 2024-04-02T07:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.720+0000 7fdddfe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdde015dad0 0x7fdde00d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33078/0 (socket says 172.21.15.89:33078) 2024-04-02T07:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.720+0000 7fdddfe00700 1 -- 172.21.15.89:0/1539540352 learned_addr learned my addr 172.21.15.89:0/1539540352 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.721+0000 7fdddf400700 1 --2- 172.21.15.89:0/1539540352 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdde00ced80 0x7fdde00d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.721+0000 7fdde4e00700 1 --2- 172.21.15.89:0/1539540352 >> v2:172.21.15.113:3300/0 conn(0x7fdde015fcb0 0x7fdde0160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.721+0000 7fdddea00700 1 -- 172.21.15.89:0/1539540352 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180798370 0 0) 0x7fddd0001110 con 0x7fdde015dad0 2024-04-02T07:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.721+0000 7fdddea00700 1 -- 172.21.15.89:0/1539540352 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc40035b0 con 0x7fdde015dad0 2024-04-02T07:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.721+0000 7fdddf400700 1 -- 172.21.15.89:0/1539540352 >> v2:172.21.15.113:3300/0 conn(0x7fdde015fcb0 msgr2=0x7fdde0160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.721+0000 7fdddf400700 1 --2- 172.21.15.89:0/1539540352 >> v2:172.21.15.113:3300/0 conn(0x7fdde015fcb0 0x7fdde0160140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.721+0000 7fdddf400700 1 -- 172.21.15.89:0/1539540352 >> v1:172.21.15.89:6789/0 conn(0x7fdde015dad0 legacy=0x7fdde00d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.721+0000 7fdddf400700 1 -- 172.21.15.89:0/1539540352 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde015f820 con 0x7fdde00ced80 2024-04-02T07:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.721+0000 7fdde4e00700 1 --2- 172.21.15.89:0/1539540352 >> v2:172.21.15.113:3300/0 conn(0x7fdde015fcb0 0x7fdde0160140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:29.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.721+0000 7fdddf400700 1 --2- 172.21.15.89:0/1539540352 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdde00ced80 0x7fdde00d1d40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009840 tx=0x7fddcc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7e9f13bc013e720 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.722+0000 7fdddea00700 1 -- 172.21.15.89:0/1539540352 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc011830 con 0x7fdde00ced80 2024-04-02T07:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.722+0000 7fdddea00700 1 -- 172.21.15.89:0/1539540352 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc011990 con 0x7fdde00ced80 2024-04-02T07:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.722+0000 7fdddea00700 1 -- 172.21.15.89:0/1539540352 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc011d20 con 0x7fdde00ced80 2024-04-02T07:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.722+0000 7fdde6e00700 1 -- 172.21.15.89:0/1539540352 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdde00ced80 msgr2=0x7fdde00d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.722+0000 7fdde6e00700 1 --2- 172.21.15.89:0/1539540352 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdde00ced80 0x7fdde00d1d40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009840 tx=0x7fddcc009b50 comp rx=0 tx=0).stop 2024-04-02T07:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.722+0000 7fdde6e00700 1 -- 172.21.15.89:0/1539540352 shutdown_connections 2024-04-02T07:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.722+0000 7fdde6e00700 1 --2- 172.21.15.89:0/1539540352 >> v2:172.21.15.113:3300/0 conn(0x7fdde015fcb0 0x7fdde0160140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.722+0000 7fdde6e00700 1 --2- 172.21.15.89:0/1539540352 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdde00ced80 0x7fdde00d1d40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.723+0000 7fdde6e00700 1 -- 172.21.15.89:0/1539540352 >> 172.21.15.89:0/1539540352 conn(0x7fdde008aff0 msgr2=0x7fdde00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.723+0000 7fdde6e00700 1 -- 172.21.15.89:0/1539540352 shutdown_connections 2024-04-02T07:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.723+0000 7fdde6e00700 1 -- 172.21.15.89:0/1539540352 wait complete. 2024-04-02T07:52:29.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.724+0000 7fdde6e00700 1 Processor -- start 2024-04-02T07:52:29.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.725+0000 7fdde6e00700 1 -- start start 2024-04-02T07:52:29.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdde6e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdde00ced80 0x7fdde01f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:29.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdde6e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdde015dad0 0x7fdde00c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:29.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdde6e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde015fcb0 0x7fdde00c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:29.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdde6e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde00ccf80 con 0x7fdde015fcb0 2024-04-02T07:52:29.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdde6e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde00b6ca0 con 0x7fdde00ced80 2024-04-02T07:52:29.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdde6e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde00c8a50 con 0x7fdde015dad0 2024-04-02T07:52:29.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdddfe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdde00ced80 0x7fdde01f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:29.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdde4e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde015fcb0 0x7fdde00c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:29.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdddf400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdde015dad0 0x7fdde00c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:29.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdddfe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdde00ced80 0x7fdde01f2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:48490/0 (socket says 172.21.15.89:48490) 2024-04-02T07:52:29.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdddfe00700 1 -- 172.21.15.89:0/3399795905 learned_addr learned my addr 172.21.15.89:0/3399795905 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:29.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdddf400700 1 -- 172.21.15.89:0/3399795905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdde00ced80 msgr2=0x7fdde01f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:29.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdddf400700 1 --2- 172.21.15.89:0/3399795905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdde00ced80 0x7fdde01f2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdddf400700 1 -- 172.21.15.89:0/3399795905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde015fcb0 msgr2=0x7fdde00c8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:29.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdddf400700 1 --2- 172.21.15.89:0/3399795905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde015fcb0 0x7fdde00c8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.726+0000 7fdddf400700 1 -- 172.21.15.89:0/3399795905 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd8009a70 con 0x7fdde015dad0 2024-04-02T07:52:29.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.727+0000 7fdddf400700 1 --2- 172.21.15.89:0/3399795905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdde015dad0 0x7fdde00c7fd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fddcc016000 tx=0x7fddcc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:29.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.727+0000 7fddd7e00700 1 -- 172.21.15.89:0/3399795905 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc004550 con 0x7fdde015dad0 2024-04-02T07:52:29.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.727+0000 7fddd7e00700 1 -- 172.21.15.89:0/3399795905 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc01f070 con 0x7fdde015dad0 2024-04-02T07:52:29.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.727+0000 7fdde6e00700 1 -- 172.21.15.89:0/3399795905 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddcc0094a0 con 0x7fdde015dad0 2024-04-02T07:52:29.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.727+0000 7fddd7e00700 1 -- 172.21.15.89:0/3399795905 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc0038e0 con 0x7fdde015dad0 2024-04-02T07:52:29.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.727+0000 7fdde6e00700 1 -- 172.21.15.89:0/3399795905 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde00b6370 con 0x7fdde015dad0 2024-04-02T07:52:29.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.728+0000 7fdde6e00700 1 -- 172.21.15.89:0/3399795905 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde015d330 con 0x7fdde015dad0 2024-04-02T07:52:29.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.729+0000 7fddd7e00700 1 -- 172.21.15.89:0/3399795905 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 47) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fddcc003a40 con 0x7fdde015dad0 2024-04-02T07:52:29.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.730+0000 7fddd7e00700 1 --2- 172.21.15.89:0/3399795905 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fddc006ade0 0x7fddc006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:29.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.730+0000 7fddd7e00700 1 -- 172.21.15.89:0/3399795905 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fddcc021070 con 0x7fdde015dad0 2024-04-02T07:52:29.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.730+0000 7fdddfe00700 1 -- 172.21.15.89:0/3399795905 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fddc006ade0 msgr2=0x7fddc006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:29.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.730+0000 7fdddfe00700 1 --2- 172.21.15.89:0/3399795905 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fddc006ade0 0x7fddc006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:29.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.735+0000 7fddd7e00700 1 -- 172.21.15.89:0/3399795905 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fddcc055b70 con 0x7fdde015dad0 2024-04-02T07:52:29.790 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:29.789+0000 7fcb1564a240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:52:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.931+0000 7fdddfe00700 1 -- 172.21.15.89:0/3399795905 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fddc006ade0 msgr2=0x7fddc006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:29.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.931+0000 7fdddfe00700 1 --2- 172.21.15.89:0/3399795905 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fddc006ade0 0x7fddc006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:29.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.948+0000 7fdde6e00700 1 -- 172.21.15.89:0/3399795905 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdde00d2ba0 con 0x7fdde015dad0 2024-04-02T07:52:29.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.949+0000 7fddd7e00700 1 -- 172.21.15.89:0/3399795905 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7fddcc02b020 con 0x7fdde015dad0 2024-04-02T07:52:29.950 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:52:29.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.952+0000 7fdde6e00700 1 -- 172.21.15.89:0/3399795905 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fddc006ade0 msgr2=0x7fddc006d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:29.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.952+0000 7fdde6e00700 1 --2- 172.21.15.89:0/3399795905 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fddc006ade0 0x7fddc006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.952+0000 7fdde6e00700 1 -- 172.21.15.89:0/3399795905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdde015dad0 msgr2=0x7fdde00c7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:29.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.952+0000 7fdde6e00700 1 --2- 172.21.15.89:0/3399795905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdde015dad0 0x7fdde00c7fd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fddcc016000 tx=0x7fddcc004750 comp rx=0 tx=0).stop 2024-04-02T07:52:29.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.952+0000 7fdde6e00700 1 -- 172.21.15.89:0/3399795905 shutdown_connections 2024-04-02T07:52:29.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.952+0000 7fdde6e00700 1 --2- 172.21.15.89:0/3399795905 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fddc006ade0 0x7fddc006d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.952+0000 7fdde6e00700 1 --2- 172.21.15.89:0/3399795905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdde00ced80 0x7fdde01f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.952+0000 7fdde6e00700 1 --2- 172.21.15.89:0/3399795905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdde015fcb0 0x7fdde00c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.952+0000 7fdde6e00700 1 --2- 172.21.15.89:0/3399795905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdde015dad0 0x7fdde00c7fd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:29.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.952+0000 7fdde6e00700 1 -- 172.21.15.89:0/3399795905 >> 172.21.15.89:0/3399795905 conn(0x7fdde008aff0 msgr2=0x7fdde0153ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:29.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.952+0000 7fdde6e00700 1 -- 172.21.15.89:0/3399795905 shutdown_connections 2024-04-02T07:52:29.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:29.952+0000 7fdde6e00700 1 -- 172.21.15.89:0/3399795905 wait complete. 2024-04-02T07:52:29.971 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T07:52:30.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.183+0000 7f3383c00700 1 Processor -- start 2024-04-02T07:52:30.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.183+0000 7f3383c00700 1 -- start start 2024-04-02T07:52:30.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.184+0000 7f3383c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f337c15dad0 0x7f337c0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:30.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.184+0000 7f3383c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f337c0d21d0 0x7f337c15fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:30.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.184+0000 7f3383c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c15f820 con 0x7f337c0d1630 2024-04-02T07:52:30.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.184+0000 7f3383c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f337c0cfcc0 con 0x7f337c15dad0 2024-04-02T07:52:30.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.184+0000 7f3383c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337c0cd9f0 con 0x7f337c0d21d0 2024-04-02T07:52:30.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.184+0000 7f3381200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f337c15dad0 0x7f337c0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:30.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.184+0000 7f3381200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f337c15dad0 0x7f337c0ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54442/0 (socket says 172.21.15.89:54442) 2024-04-02T07:52:30.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.184+0000 7f3381200700 1 -- 172.21.15.89:0/1287914228 learned_addr learned my addr 172.21.15.89:0/1287914228 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:30.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.184+0000 7f337be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f337c0d1630 0x7f337c0d1ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33086/0 (socket says 172.21.15.89:33086) 2024-04-02T07:52:30.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.184+0000 7f3381c00700 1 --2- 172.21.15.89:0/1287914228 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f337c0d21d0 0x7f337c15fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:30.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.185+0000 7f3381200700 1 -- 172.21.15.89:0/1287914228 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f337c0d21d0 msgr2=0x7f337c15fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:30.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.185+0000 7f3381200700 1 --2- 172.21.15.89:0/1287914228 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f337c0d21d0 0x7f337c15fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.185+0000 7f3381200700 1 -- 172.21.15.89:0/1287914228 >> v1:172.21.15.89:6789/0 conn(0x7f337c0d1630 legacy=0x7f337c0d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:30.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.185+0000 7f3381200700 1 -- 172.21.15.89:0/1287914228 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c0cf420 con 0x7f337c15dad0 2024-04-02T07:52:30.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.185+0000 7f3381c00700 1 --2- 172.21.15.89:0/1287914228 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f337c0d21d0 0x7f337c15fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:30.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.185+0000 7f3381200700 1 --2- 172.21.15.89:0/1287914228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f337c15dad0 0x7f337c0ced80 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f33700098f0 tx=0x7f3370009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e392b12bd91434a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.185+0000 7f337b400700 1 -- 172.21.15.89:0/1287914228 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337000b8a0 con 0x7f337c15dad0 2024-04-02T07:52:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.185+0000 7f337b400700 1 -- 172.21.15.89:0/1287914228 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337000ba00 con 0x7f337c15dad0 2024-04-02T07:52:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.185+0000 7f337b400700 1 -- 172.21.15.89:0/1287914228 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337000bd90 con 0x7f337c15dad0 2024-04-02T07:52:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.186+0000 7f3383c00700 1 -- 172.21.15.89:0/1287914228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f337c15dad0 msgr2=0x7f337c0ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.186+0000 7f3383c00700 1 --2- 172.21.15.89:0/1287914228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f337c15dad0 0x7f337c0ced80 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f33700098f0 tx=0x7f3370009c00 comp rx=0 tx=0).stop 2024-04-02T07:52:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.186+0000 7f3383c00700 1 -- 172.21.15.89:0/1287914228 shutdown_connections 2024-04-02T07:52:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.186+0000 7f3383c00700 1 --2- 172.21.15.89:0/1287914228 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f337c0d21d0 0x7f337c15fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.186+0000 7f3383c00700 1 --2- 172.21.15.89:0/1287914228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f337c15dad0 0x7f337c0ced80 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.186+0000 7f3383c00700 1 -- 172.21.15.89:0/1287914228 >> 172.21.15.89:0/1287914228 conn(0x7f337c08aff0 msgr2=0x7f337c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.186+0000 7f3383c00700 1 -- 172.21.15.89:0/1287914228 shutdown_connections 2024-04-02T07:52:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.186+0000 7f3383c00700 1 -- 172.21.15.89:0/1287914228 wait complete. 2024-04-02T07:52:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.188+0000 7f3383c00700 1 Processor -- start 2024-04-02T07:52:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.188+0000 7f3383c00700 1 -- start start 2024-04-02T07:52:30.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f3383c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f337c0d1630 0x7f337c0cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:30.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f3383c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f337c0d21d0 0x7f337c0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:30.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f3383c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f337c15dad0 0x7f337c0b8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:30.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f3383c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337c0cab30 con 0x7f337c0d21d0 2024-04-02T07:52:30.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f3383c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337c0cadd0 con 0x7f337c15dad0 2024-04-02T07:52:30.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f3383c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337c0cb0b0 con 0x7f337c0d1630 2024-04-02T07:52:30.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f337be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f337c0d21d0 0x7f337c0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:30.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f337be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f337c0d21d0 0x7f337c0ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54450/0 (socket says 172.21.15.89:54450) 2024-04-02T07:52:30.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f337be00700 1 -- 172.21.15.89:0/4167603301 learned_addr learned my addr 172.21.15.89:0/4167603301 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:30.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f3381200700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f337c0d1630 0x7f337c0cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:30.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f3381c00700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f337c15dad0 0x7f337c0b8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:30.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f337be00700 1 -- 172.21.15.89:0/4167603301 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f337c0d1630 msgr2=0x7f337c0cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:30.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f337be00700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f337c0d1630 0x7f337c0cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f337be00700 1 -- 172.21.15.89:0/4167603301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f337c15dad0 msgr2=0x7f337c0b8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:30.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f337be00700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f337c15dad0 0x7f337c0b8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f337be00700 1 -- 172.21.15.89:0/4167603301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3370009550 con 0x7f337c0d21d0 2024-04-02T07:52:30.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.189+0000 7f3381c00700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f337c15dad0 0x7f337c0b8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:30.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.190+0000 7f337be00700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f337c0d21d0 0x7f337c0ca5f0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f336800c520 tx=0x7f336800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:30.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.190+0000 7f3378c00700 1 -- 172.21.15.89:0/4167603301 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368005cb0 con 0x7f337c0d21d0 2024-04-02T07:52:30.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.190+0000 7f3378c00700 1 -- 172.21.15.89:0/4167603301 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3368005e10 con 0x7f337c0d21d0 2024-04-02T07:52:30.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.190+0000 7f3383c00700 1 -- 172.21.15.89:0/4167603301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f337c0cb390 con 0x7f337c0d21d0 2024-04-02T07:52:30.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.190+0000 7f3378c00700 1 -- 172.21.15.89:0/4167603301 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33680072d0 con 0x7f337c0d21d0 2024-04-02T07:52:30.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.190+0000 7f3383c00700 1 -- 172.21.15.89:0/4167603301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f337c0cb950 con 0x7f337c0d21d0 2024-04-02T07:52:30.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.191+0000 7f3383c00700 1 -- 172.21.15.89:0/4167603301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337c067720 con 0x7f337c0d21d0 2024-04-02T07:52:30.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.192+0000 7f3378c00700 1 -- 172.21.15.89:0/4167603301 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 47) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f3368007430 con 0x7f337c0d21d0 2024-04-02T07:52:30.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.193+0000 7f3378c00700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f335c06adb0 0x7f335c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:30.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.193+0000 7f3378c00700 1 -- 172.21.15.89:0/4167603301 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f3368088ac0 con 0x7f337c0d21d0 2024-04-02T07:52:30.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.193+0000 7f3381200700 1 -- 172.21.15.89:0/4167603301 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f335c06adb0 msgr2=0x7f335c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:30.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.193+0000 7f3381200700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f335c06adb0 0x7f335c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:30.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.201+0000 7f3378c00700 1 -- 172.21.15.89:0/4167603301 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3368053970 con 0x7f337c0d21d0 2024-04-02T07:52:30.394 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:30.393+0000 7fb9f298d240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:52:30.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.394+0000 7f3381200700 1 -- 172.21.15.89:0/4167603301 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f335c06adb0 msgr2=0x7f335c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:30.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.394+0000 7f3381200700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f335c06adb0 0x7f335c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.420+0000 7f3383c00700 1 -- 172.21.15.89:0/4167603301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f337c0b6130 con 0x7f337c0d21d0 2024-04-02T07:52:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.420+0000 7f3378c00700 1 -- 172.21.15.89:0/4167603301 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v98) v1 ==== 80+0+460 (secure 0 0 0) 0x7f3368017020 con 0x7f337c0d21d0 2024-04-02T07:52:30.421 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1957897648 2024-04-03T07:52:25.709764+0000 2024-04-02T07:52:30.422 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1991189445 2024-04-03T07:52:25.709764+0000 2024-04-02T07:52:30.422 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/4248118690 2024-04-03T07:52:25.709764+0000 2024-04-02T07:52:30.422 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/43357 2024-04-03T07:52:25.709764+0000 2024-04-02T07:52:30.422 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/43357 2024-04-03T07:52:25.709764+0000 2024-04-02T07:52:30.422 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3231846713 2024-04-03T07:52:25.709764+0000 2024-04-02T07:52:30.422 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3820568899 2024-04-03T07:52:25.709764+0000 2024-04-02T07:52:30.422 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3513660462 2024-04-03T07:52:25.709764+0000 2024-04-02T07:52:30.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.424+0000 7f3383c00700 1 -- 172.21.15.89:0/4167603301 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f335c06adb0 msgr2=0x7f335c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:30.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.424+0000 7f3383c00700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f335c06adb0 0x7f335c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.424+0000 7f3383c00700 1 -- 172.21.15.89:0/4167603301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f337c0d21d0 msgr2=0x7f337c0ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:30.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.424+0000 7f3383c00700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f337c0d21d0 0x7f337c0ca5f0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f336800c520 tx=0x7f336800c830 comp rx=0 tx=0).stop 2024-04-02T07:52:30.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.424+0000 7f3383c00700 1 -- 172.21.15.89:0/4167603301 shutdown_connections 2024-04-02T07:52:30.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.424+0000 7f3383c00700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f335c06adb0 0x7f335c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.424+0000 7f3383c00700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f337c15dad0 0x7f337c0b8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.424+0000 7f3383c00700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f337c0d1630 0x7f337c0cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.424+0000 7f3383c00700 1 --2- 172.21.15.89:0/4167603301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f337c0d21d0 0x7f337c0ca5f0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.424+0000 7f3383c00700 1 -- 172.21.15.89:0/4167603301 >> 172.21.15.89:0/4167603301 conn(0x7f337c08aff0 msgr2=0x7f337c151e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:30.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.424+0000 7f3383c00700 1 -- 172.21.15.89:0/4167603301 shutdown_connections 2024-04-02T07:52:30.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.424+0000 7f3383c00700 1 -- 172.21.15.89:0/4167603301 wait complete. 2024-04-02T07:52:30.427 INFO:teuthology.orchestra.run.smithi089.stderr:listed 8 entries 2024-04-02T07:52:30.451 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T07:52:30.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:30.451 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T07:52:30.482 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:30.482+0000 7fb9f298d240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:52:30.525 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:52:30.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:30.525 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T07:52:30.569 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:30.568+0000 7fcb1564a240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:52:30.571 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:52:30.571 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:30.615 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:52:30.615 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:52:30.615 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:30.649 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:30.648+0000 7fb9f298d240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:52:30.664 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:52:30.664 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:52:30.664 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:30.671 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:30.670+0000 7fcb1564a240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:52:30.708 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:52:30.708 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:52:30.840 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:30.839+0000 7fcb1564a240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:52:30.867 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:30.867+0000 7fb9f298d240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:52:30.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.905+0000 7fd2e8600700 1 Processor -- start 2024-04-02T07:52:30.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.905+0000 7fd2e8600700 1 -- start start 2024-04-02T07:52:30.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.906+0000 7fd2e8600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd2e00afbe0 0x7fd2e006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:30.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.906+0000 7fd2e8600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2e00cebd0 0x7fd2e00cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:30.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.906+0000 7fd2e8600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e006f850 con 0x7fd2e006a790 2024-04-02T07:52:30.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.906+0000 7fd2e8600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e006fad0 con 0x7fd2e00afbe0 2024-04-02T07:52:30.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.906+0000 7fd2e8600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e00af970 con 0x7fd2e00cebd0 2024-04-02T07:52:30.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.906+0000 7fd2e5200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd2e00afbe0 0x7fd2e006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:30.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.906+0000 7fd2e6600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2e00cebd0 0x7fd2e00cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:30.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.906+0000 7fd2e5c00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fd2e006a790 0x7fd2e006ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:55018/0 (socket says 172.21.15.89:55018) 2024-04-02T07:52:30.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.906+0000 7fd2e5c00700 1 -- 172.21.15.89:0/2112078452 learned_addr learned my addr 172.21.15.89:0/2112078452 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:30.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.907+0000 7fd2dfe00700 1 -- 172.21.15.89:0/2112078452 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783982105 0 0) 0x7fd2e006f850 con 0x7fd2e006a790 2024-04-02T07:52:30.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.907+0000 7fd2dfe00700 1 -- 172.21.15.89:0/2112078452 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c80035b0 con 0x7fd2e006a790 2024-04-02T07:52:30.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.907+0000 7fd2e6600700 1 -- 172.21.15.89:0/2112078452 >> v2:172.21.15.113:3300/0 conn(0x7fd2e00afbe0 msgr2=0x7fd2e006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:30.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.907+0000 7fd2e6600700 1 --2- 172.21.15.89:0/2112078452 >> v2:172.21.15.113:3300/0 conn(0x7fd2e00afbe0 0x7fd2e006dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.907+0000 7fd2e6600700 1 -- 172.21.15.89:0/2112078452 >> v1:172.21.15.113:6789/0 conn(0x7fd2e006a790 legacy=0x7fd2e006ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:30.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.907+0000 7fd2e6600700 1 -- 172.21.15.89:0/2112078452 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e00af690 con 0x7fd2e00cebd0 2024-04-02T07:52:30.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.907+0000 7fd2e5200700 1 --2- 172.21.15.89:0/2112078452 >> v2:172.21.15.113:3300/0 conn(0x7fd2e00afbe0 0x7fd2e006dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:30.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.907+0000 7fd2e6600700 1 --2- 172.21.15.89:0/2112078452 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2e00cebd0 0x7fd2e00cf040 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009d90 tx=0x7fd2d800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da125bad6e3bb8d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:30.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.907+0000 7fd2dfe00700 1 -- 172.21.15.89:0/2112078452 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d800e070 con 0x7fd2e00cebd0 2024-04-02T07:52:30.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.907+0000 7fd2dfe00700 1 -- 172.21.15.89:0/2112078452 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d800bb20 con 0x7fd2e00cebd0 2024-04-02T07:52:30.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.907+0000 7fd2dfe00700 1 -- 172.21.15.89:0/2112078452 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8012430 con 0x7fd2e00cebd0 2024-04-02T07:52:30.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.908+0000 7fd2e8600700 1 -- 172.21.15.89:0/2112078452 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2e00cebd0 msgr2=0x7fd2e00cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:30.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.908+0000 7fd2e8600700 1 --2- 172.21.15.89:0/2112078452 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2e00cebd0 0x7fd2e00cf040 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009d90 tx=0x7fd2d800b330 comp rx=0 tx=0).stop 2024-04-02T07:52:30.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.908+0000 7fd2e8600700 1 -- 172.21.15.89:0/2112078452 shutdown_connections 2024-04-02T07:52:30.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.908+0000 7fd2e8600700 1 --2- 172.21.15.89:0/2112078452 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2e00cebd0 0x7fd2e00cf040 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.908+0000 7fd2e8600700 1 --2- 172.21.15.89:0/2112078452 >> v2:172.21.15.113:3300/0 conn(0x7fd2e00afbe0 0x7fd2e006dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.908+0000 7fd2e8600700 1 -- 172.21.15.89:0/2112078452 >> 172.21.15.89:0/2112078452 conn(0x7fd2e00b5ab0 msgr2=0x7fd2e00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:30.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.908+0000 7fd2e8600700 1 -- 172.21.15.89:0/2112078452 shutdown_connections 2024-04-02T07:52:30.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.908+0000 7fd2e8600700 1 -- 172.21.15.89:0/2112078452 wait complete. 2024-04-02T07:52:30.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.910+0000 7fd2e8600700 1 Processor -- start 2024-04-02T07:52:30.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.910+0000 7fd2e8600700 1 -- start start 2024-04-02T07:52:30.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.911+0000 7fd2e8600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e006a790 0x7fd2e009b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:30.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.911+0000 7fd2e8600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2e00afbe0 0x7fd2e009d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:30.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.911+0000 7fd2e8600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e00cebd0 0x7fd2e014cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:30.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.911+0000 7fd2e8600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e00cb4c0 con 0x7fd2e006a790 2024-04-02T07:52:30.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.911+0000 7fd2e8600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e009d890 con 0x7fd2e00cebd0 2024-04-02T07:52:30.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.911+0000 7fd2e8600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e009db60 con 0x7fd2e00afbe0 2024-04-02T07:52:30.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.911+0000 7fd2e5c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e006a790 0x7fd2e009b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:30.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.911+0000 7fd2e5c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e006a790 0x7fd2e009b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54458/0 (socket says 172.21.15.89:54458) 2024-04-02T07:52:30.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.911+0000 7fd2e5c00700 1 -- 172.21.15.89:0/4059962122 learned_addr learned my addr 172.21.15.89:0/4059962122 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:30.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.911+0000 7fd2e5200700 1 --2- 172.21.15.89:0/4059962122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2e00afbe0 0x7fd2e009d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:30.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.912+0000 7fd2e6600700 1 --2- 172.21.15.89:0/4059962122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e00cebd0 0x7fd2e014cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:30.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.912+0000 7fd2e5c00700 1 -- 172.21.15.89:0/4059962122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2e00afbe0 msgr2=0x7fd2e009d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:30.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.912+0000 7fd2e5c00700 1 --2- 172.21.15.89:0/4059962122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2e00afbe0 0x7fd2e009d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.912+0000 7fd2e5c00700 1 -- 172.21.15.89:0/4059962122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e00cebd0 msgr2=0x7fd2e014cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:30.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.912+0000 7fd2e5c00700 1 --2- 172.21.15.89:0/4059962122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e00cebd0 0x7fd2e014cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:30.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.912+0000 7fd2e5c00700 1 -- 172.21.15.89:0/4059962122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2cc0094a0 con 0x7fd2e006a790 2024-04-02T07:52:30.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.912+0000 7fd2e5c00700 1 --2- 172.21.15.89:0/4059962122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e006a790 0x7fd2e009b030 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4001eb0 tx=0x7fd2d400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:30.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.912+0000 7fd2dd600700 1 -- 172.21.15.89:0/4059962122 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4005d70 con 0x7fd2e006a790 2024-04-02T07:52:30.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.912+0000 7fd2dd600700 1 -- 172.21.15.89:0/4059962122 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d40070d0 con 0x7fd2e006a790 2024-04-02T07:52:30.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.912+0000 7fd2e8600700 1 -- 172.21.15.89:0/4059962122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d8009a70 con 0x7fd2e006a790 2024-04-02T07:52:30.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.912+0000 7fd2dd600700 1 -- 172.21.15.89:0/4059962122 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4007460 con 0x7fd2e006a790 2024-04-02T07:52:30.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.913+0000 7fd2e8600700 1 -- 172.21.15.89:0/4059962122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e009e120 con 0x7fd2e006a790 2024-04-02T07:52:30.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.914+0000 7fd2e8600700 1 -- 172.21.15.89:0/4059962122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e00cd750 con 0x7fd2e006a790 2024-04-02T07:52:30.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.915+0000 7fd2dd600700 1 -- 172.21.15.89:0/4059962122 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 47) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fd2d40075c0 con 0x7fd2e006a790 2024-04-02T07:52:30.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.916+0000 7fd2dd600700 1 --2- 172.21.15.89:0/4059962122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2c006adb0 0x7fd2c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:30.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.916+0000 7fd2dd600700 1 -- 172.21.15.89:0/4059962122 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd2d4088b70 con 0x7fd2e006a790 2024-04-02T07:52:30.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.916+0000 7fd2e5200700 1 -- 172.21.15.89:0/4059962122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2c006adb0 msgr2=0x7fd2c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:30.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.916+0000 7fd2e5200700 1 --2- 172.21.15.89:0/4059962122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2c006adb0 0x7fd2c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:30.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:30.921+0000 7fd2dd600700 1 -- 172.21.15.89:0/4059962122 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd2d4053990 con 0x7fd2e006a790 2024-04-02T07:52:30.958 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:30.958+0000 7fb9f298d240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:52:31.055 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:31.054+0000 7fb9f298d240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:52:31.059 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:31.058+0000 7fcb1564a240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:52:31.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.117+0000 7fd2e5200700 1 -- 172.21.15.89:0/4059962122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2c006adb0 msgr2=0x7fd2c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:31.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.117+0000 7fd2e5200700 1 --2- 172.21.15.89:0/4059962122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2c006adb0 0x7fd2c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:31.152 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:31.151+0000 7fcb1564a240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:52:31.153 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:31.152+0000 7fb9f298d240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:52:31.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.173+0000 7fd2e8600700 1 -- 172.21.15.89:0/4059962122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2e0067720 con 0x7fd2e006a790 2024-04-02T07:52:31.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.173+0000 7fd2dd600700 1 -- 172.21.15.89:0/4059962122 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fd2d4056fb0 con 0x7fd2e006a790 2024-04-02T07:52:31.175 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:31.175 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6556,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3307197948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3307197948},{"type":"v1","addr":"172.21.15.113:6835","nonce":3307197948}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1797655686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1797655686},{"type":"v1","addr":"172.21.15.113:6834","nonce":1797655686}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4203087300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4203087300},{"type":"v1","addr":"172.21.15.89:6836","nonce":4203087300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:52:09.066449+0000","modified":"2024-04-02T07:52:12.080422+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3228984565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3228984565},{"type":"v1","addr":"172.21.15.89:6837","nonce":3228984565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-02T07:52:31.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.176+0000 7fd2e8600700 1 -- 172.21.15.89:0/4059962122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2c006adb0 msgr2=0x7fd2c006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:31.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.176+0000 7fd2e8600700 1 --2- 172.21.15.89:0/4059962122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2c006adb0 0x7fd2c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.176+0000 7fd2e8600700 1 -- 172.21.15.89:0/4059962122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e006a790 msgr2=0x7fd2e009b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:31.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.176+0000 7fd2e8600700 1 --2- 172.21.15.89:0/4059962122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e006a790 0x7fd2e009b030 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4001eb0 tx=0x7fd2d400c7b0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.176+0000 7fd2e8600700 1 -- 172.21.15.89:0/4059962122 shutdown_connections 2024-04-02T07:52:31.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.176+0000 7fd2e8600700 1 --2- 172.21.15.89:0/4059962122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2c006adb0 0x7fd2c006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.176+0000 7fd2e8600700 1 --2- 172.21.15.89:0/4059962122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e00cebd0 0x7fd2e014cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.176+0000 7fd2e8600700 1 --2- 172.21.15.89:0/4059962122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2e006a790 0x7fd2e009b030 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.176+0000 7fd2e8600700 1 --2- 172.21.15.89:0/4059962122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2e00afbe0 0x7fd2e009d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.176+0000 7fd2e8600700 1 -- 172.21.15.89:0/4059962122 >> 172.21.15.89:0/4059962122 conn(0x7fd2e00b5ab0 msgr2=0x7fd2e00ad350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:31.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.176+0000 7fd2e8600700 1 -- 172.21.15.89:0/4059962122 shutdown_connections 2024-04-02T07:52:31.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.176+0000 7fd2e8600700 1 -- 172.21.15.89:0/4059962122 wait complete. 2024-04-02T07:52:31.180 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 41 2024-04-02T07:52:31.198 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:52:31.250 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:31.250+0000 7fb9f298d240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:52:31.261 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:31.260+0000 7fcb1564a240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:52:31.366 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:31.365+0000 7fcb1564a240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:52:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.395+0000 7f7693200700 1 Processor -- start 2024-04-02T07:52:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.395+0000 7f7693200700 1 -- start start 2024-04-02T07:52:31.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.396+0000 7f7693200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f768c15da50 0x7f768c0ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:31.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.396+0000 7f7693200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f768c0cec70 0x7f768c0d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:31.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.396+0000 7f7693200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c0d24e0 con 0x7f768c0d0b20 2024-04-02T07:52:31.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.396+0000 7f7693200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768c0d0fd0 con 0x7f768c15da50 2024-04-02T07:52:31.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.396+0000 7f7693200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768c0b95d0 con 0x7f768c0cec70 2024-04-02T07:52:31.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.396+0000 7f768be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f768c15da50 0x7f768c0ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:31.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.396+0000 7f7691200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f768c0d0b20 0x7f768c0b52e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33094/0 (socket says 172.21.15.89:33094) 2024-04-02T07:52:31.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.396+0000 7f7691200700 1 -- 172.21.15.89:0/1831984327 learned_addr learned my addr 172.21.15.89:0/1831984327 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:31.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.396+0000 7f768be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f768c15da50 0x7f768c0ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54460/0 (socket says 172.21.15.89:54460) 2024-04-02T07:52:31.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.396+0000 7f768b400700 1 --2- 172.21.15.89:0/1831984327 >> v2:172.21.15.113:3300/0 conn(0x7f768c0cec70 0x7f768c0d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:31.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.397+0000 7f768be00700 1 -- 172.21.15.89:0/1831984327 >> v2:172.21.15.113:3300/0 conn(0x7f768c0cec70 msgr2=0x7f768c0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:31.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.397+0000 7f768be00700 1 --2- 172.21.15.89:0/1831984327 >> v2:172.21.15.113:3300/0 conn(0x7f768c0cec70 0x7f768c0d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.397+0000 7f768be00700 1 -- 172.21.15.89:0/1831984327 >> v1:172.21.15.89:6789/0 conn(0x7f768c0d0b20 legacy=0x7f768c0b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:31.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.397+0000 7f768be00700 1 -- 172.21.15.89:0/1831984327 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c0d27c0 con 0x7f768c15da50 2024-04-02T07:52:31.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.397+0000 7f768aa00700 1 -- 172.21.15.89:0/1831984327 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436219677 0 0) 0x7f768c0d24e0 con 0x7f768c0d0b20 2024-04-02T07:52:31.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.397+0000 7f768be00700 1 --2- 172.21.15.89:0/1831984327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c15da50 0x7f768c0ce730 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f768c0740a0 tx=0x7f767c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04283277c59e1e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:31.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.397+0000 7f768aa00700 1 -- 172.21.15.89:0/1831984327 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c011830 con 0x7f768c15da50 2024-04-02T07:52:31.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.397+0000 7f768aa00700 1 -- 172.21.15.89:0/1831984327 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c011990 con 0x7f768c15da50 2024-04-02T07:52:31.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.397+0000 7f768aa00700 1 -- 172.21.15.89:0/1831984327 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c011d20 con 0x7f768c15da50 2024-04-02T07:52:31.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.398+0000 7f7693200700 1 -- 172.21.15.89:0/1831984327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c15da50 msgr2=0x7f768c0ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:31.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.398+0000 7f7693200700 1 --2- 172.21.15.89:0/1831984327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c15da50 0x7f768c0ce730 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f768c0740a0 tx=0x7f767c009b20 comp rx=0 tx=0).stop 2024-04-02T07:52:31.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.398+0000 7f7693200700 1 -- 172.21.15.89:0/1831984327 shutdown_connections 2024-04-02T07:52:31.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.398+0000 7f7693200700 1 --2- 172.21.15.89:0/1831984327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c15da50 0x7f768c0ce730 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.398+0000 7f7693200700 1 --2- 172.21.15.89:0/1831984327 >> v2:172.21.15.113:3300/0 conn(0x7f768c0cec70 0x7f768c0d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.398+0000 7f7693200700 1 -- 172.21.15.89:0/1831984327 >> 172.21.15.89:0/1831984327 conn(0x7f768c0666a0 msgr2=0x7f768c066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:31.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.398+0000 7f7693200700 1 -- 172.21.15.89:0/1831984327 shutdown_connections 2024-04-02T07:52:31.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.398+0000 7f7693200700 1 -- 172.21.15.89:0/1831984327 wait complete. 2024-04-02T07:52:31.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.400+0000 7f7693200700 1 Processor -- start 2024-04-02T07:52:31.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.400+0000 7f7693200700 1 -- start start 2024-04-02T07:52:31.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.401+0000 7f7693200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f768c0cec70 0x7f768c159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:31.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.401+0000 7f7693200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c0d0b20 0x7f768c152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:31.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.401+0000 7f7693200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f768c15da50 0x7f768c0cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:31.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.401+0000 7f7693200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0cb370 con 0x7f768c0d0b20 2024-04-02T07:52:31.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.401+0000 7f7693200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0cbcd0 con 0x7f768c15da50 2024-04-02T07:52:31.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.401+0000 7f7693200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c160f70 con 0x7f768c0cec70 2024-04-02T07:52:31.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.401+0000 7f768b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c0d0b20 0x7f768c152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:31.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.401+0000 7f768b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c0d0b20 0x7f768c152670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54462/0 (socket says 172.21.15.89:54462) 2024-04-02T07:52:31.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.402+0000 7f768b400700 1 -- 172.21.15.89:0/221261447 learned_addr learned my addr 172.21.15.89:0/221261447 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:31.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.402+0000 7f768be00700 1 --2- 172.21.15.89:0/221261447 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f768c0cec70 0x7f768c159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:31.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.402+0000 7f7691200700 1 --2- 172.21.15.89:0/221261447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f768c15da50 0x7f768c0cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:31.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.402+0000 7f768b400700 1 -- 172.21.15.89:0/221261447 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f768c0cec70 msgr2=0x7f768c159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:31.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.402+0000 7f768b400700 1 --2- 172.21.15.89:0/221261447 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f768c0cec70 0x7f768c159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.402+0000 7f768b400700 1 -- 172.21.15.89:0/221261447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f768c15da50 msgr2=0x7f768c0cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:31.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.402+0000 7f768b400700 1 --2- 172.21.15.89:0/221261447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f768c15da50 0x7f768c0cb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.402+0000 7f768b400700 1 -- 172.21.15.89:0/221261447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c0094a0 con 0x7f768c0d0b20 2024-04-02T07:52:31.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.402+0000 7f768b400700 1 --2- 172.21.15.89:0/221261447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c0d0b20 0x7f768c152670 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f767800e980 tx=0x7f767800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:31.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.402+0000 7f7683e00700 1 -- 172.21.15.89:0/221261447 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7678009710 con 0x7f768c0d0b20 2024-04-02T07:52:31.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.402+0000 7f7683e00700 1 -- 172.21.15.89:0/221261447 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767800c690 con 0x7f768c0d0b20 2024-04-02T07:52:31.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.402+0000 7f7693200700 1 -- 172.21.15.89:0/221261447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c15a4f0 con 0x7f768c0d0b20 2024-04-02T07:52:31.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.402+0000 7f7683e00700 1 -- 172.21.15.89:0/221261447 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767800ca20 con 0x7f768c0d0b20 2024-04-02T07:52:31.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.403+0000 7f7693200700 1 -- 172.21.15.89:0/221261447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c0cacb0 con 0x7f768c0d0b20 2024-04-02T07:52:31.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.404+0000 7f7693200700 1 -- 172.21.15.89:0/221261447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768c0cb5e0 con 0x7f768c0d0b20 2024-04-02T07:52:31.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.405+0000 7f7683e00700 1 -- 172.21.15.89:0/221261447 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 47) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f767800cb80 con 0x7f768c0d0b20 2024-04-02T07:52:31.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.406+0000 7f7683e00700 1 --2- 172.21.15.89:0/221261447 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f766c06adb0 0x7f766c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:31.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.406+0000 7f768be00700 1 -- 172.21.15.89:0/221261447 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f766c06adb0 msgr2=0x7f766c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:31.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.406+0000 7f7683e00700 1 -- 172.21.15.89:0/221261447 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f7678089630 con 0x7f768c0d0b20 2024-04-02T07:52:31.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.406+0000 7f768be00700 1 --2- 172.21.15.89:0/221261447 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f766c06adb0 0x7f766c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:31.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.414+0000 7f7683e00700 1 -- 172.21.15.89:0/221261447 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f76780544d0 con 0x7f768c0d0b20 2024-04-02T07:52:31.460 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:31.460+0000 7fb9f298d240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:52:31.474 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:31.473+0000 7fcb1564a240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:52:31.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.607+0000 7f768be00700 1 -- 172.21.15.89:0/221261447 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f766c06adb0 msgr2=0x7f766c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:31.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.607+0000 7f768be00700 1 --2- 172.21.15.89:0/221261447 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f766c06adb0 0x7f766c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.622+0000 7f7693200700 1 -- 172.21.15.89:0/221261447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f768c067720 con 0x7f768c0d0b20 2024-04-02T07:52:31.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.623+0000 7f7683e00700 1 -- 172.21.15.89:0/221261447 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+17058 (secure 0 0 0) 0x7f7678057af0 con 0x7f768c0d0b20 2024-04-02T07:52:31.625 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:31.625 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":98,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:52:25.709797+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:52:06.991300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-02T07:52:08.054066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/1957897648":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/1991189445":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/4248118690":"2024-04-03T07:52:25.709764+0000","172.21.15.89:6833/43357":"2024-04-03T07:52:25.709764+0000","172.21.15.89:6832/43357":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/3231846713":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/3820568899":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/3513660462":"2024-04-03T07:52:25.709764+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:52:31.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.626+0000 7f7693200700 1 -- 172.21.15.89:0/221261447 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f766c06adb0 msgr2=0x7f766c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:31.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.626+0000 7f7693200700 1 --2- 172.21.15.89:0/221261447 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f766c06adb0 0x7f766c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.626+0000 7f7693200700 1 -- 172.21.15.89:0/221261447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c0d0b20 msgr2=0x7f768c152670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:31.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.626+0000 7f7693200700 1 --2- 172.21.15.89:0/221261447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c0d0b20 0x7f768c152670 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f767800e980 tx=0x7f767800ec90 comp rx=0 tx=0).stop 2024-04-02T07:52:31.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.626+0000 7f7693200700 1 -- 172.21.15.89:0/221261447 shutdown_connections 2024-04-02T07:52:31.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.626+0000 7f7693200700 1 --2- 172.21.15.89:0/221261447 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f766c06adb0 0x7f766c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.626+0000 7f7693200700 1 --2- 172.21.15.89:0/221261447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f768c15da50 0x7f768c0cb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.626+0000 7f7693200700 1 --2- 172.21.15.89:0/221261447 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f768c0cec70 0x7f768c159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.626+0000 7f7693200700 1 --2- 172.21.15.89:0/221261447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c0d0b20 0x7f768c152670 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.626+0000 7f7693200700 1 -- 172.21.15.89:0/221261447 >> 172.21.15.89:0/221261447 conn(0x7f768c0666a0 msgr2=0x7f768c0b80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:31.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.626+0000 7f7693200700 1 -- 172.21.15.89:0/221261447 shutdown_connections 2024-04-02T07:52:31.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.626+0000 7f7693200700 1 -- 172.21.15.89:0/221261447 wait complete. 2024-04-02T07:52:31.646 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:52:06.991300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:52:08.054066+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:52:31.646 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:52:31.694 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:31.693+0000 7fcb1564a240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:52:31.713 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:31.712+0000 7fb9f298d240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:52:31.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.849+0000 7f9e67800700 1 Processor -- start 2024-04-02T07:52:31.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.849+0000 7f9e67800700 1 -- start start 2024-04-02T07:52:31.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.850+0000 7f9e67800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9e60160e50 0x7f9e600b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:31.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.850+0000 7f9e67800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e600b3cb0 con 0x7f9e600d3f20 2024-04-02T07:52:31.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.850+0000 7f9e67800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e600cef00 con 0x7f9e600b3f80 2024-04-02T07:52:31.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.850+0000 7f9e67800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e60161230 con 0x7f9e60160e50 2024-04-02T07:52:31.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.850+0000 7f9e5fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9e600d3f20 0x7f9e600b2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33102/0 (socket says 172.21.15.89:33102) 2024-04-02T07:52:31.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.850+0000 7f9e5fe00700 1 -- 172.21.15.89:0/296469482 learned_addr learned my addr 172.21.15.89:0/296469482 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:31.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.850+0000 7f9e64e00700 1 --2- 172.21.15.89:0/296469482 >> v2:172.21.15.113:3300/0 conn(0x7f9e60160e50 0x7f9e600b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:31.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.851+0000 7f9e64e00700 1 -- 172.21.15.89:0/296469482 >> v1:172.21.15.113:6789/0 conn(0x7f9e600b3f80 legacy=0x7f9e600b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:31.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.851+0000 7f9e64e00700 1 -- 172.21.15.89:0/296469482 >> v1:172.21.15.89:6789/0 conn(0x7f9e600d3f20 legacy=0x7f9e600b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:31.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.851+0000 7f9e64e00700 1 -- 172.21.15.89:0/296469482 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e600d2a80 con 0x7f9e60160e50 2024-04-02T07:52:31.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.851+0000 7f9e5f400700 1 -- 172.21.15.89:0/296469482 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321168580 0 0) 0x7f9e600b3cb0 con 0x7f9e600d3f20 2024-04-02T07:52:31.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.851+0000 7f9e64e00700 1 --2- 172.21.15.89:0/296469482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e60160e50 0x7f9e600b4e90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9e540097c0 tx=0x7f9e54009ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76baff7e82cd5819 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:31.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.851+0000 7f9e5f400700 1 -- 172.21.15.89:0/296469482 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54004230 con 0x7f9e60160e50 2024-04-02T07:52:31.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.851+0000 7f9e5f400700 1 -- 172.21.15.89:0/296469482 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e54004390 con 0x7f9e60160e50 2024-04-02T07:52:31.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.851+0000 7f9e5f400700 1 -- 172.21.15.89:0/296469482 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e540036a0 con 0x7f9e60160e50 2024-04-02T07:52:31.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.852+0000 7f9e67800700 1 -- 172.21.15.89:0/296469482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e60160e50 msgr2=0x7f9e600b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:31.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.852+0000 7f9e67800700 1 --2- 172.21.15.89:0/296469482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e60160e50 0x7f9e600b4e90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9e540097c0 tx=0x7f9e54009ad0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.852+0000 7f9e67800700 1 -- 172.21.15.89:0/296469482 shutdown_connections 2024-04-02T07:52:31.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.852+0000 7f9e67800700 1 --2- 172.21.15.89:0/296469482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e60160e50 0x7f9e600b4e90 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.852+0000 7f9e67800700 1 -- 172.21.15.89:0/296469482 >> 172.21.15.89:0/296469482 conn(0x7f9e6008aff0 msgr2=0x7f9e600d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:31.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.852+0000 7f9e67800700 1 -- 172.21.15.89:0/296469482 shutdown_connections 2024-04-02T07:52:31.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.852+0000 7f9e67800700 1 -- 172.21.15.89:0/296469482 wait complete. 2024-04-02T07:52:31.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.854+0000 7f9e67800700 1 Processor -- start 2024-04-02T07:52:31.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.854+0000 7f9e67800700 1 -- start start 2024-04-02T07:52:31.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.855+0000 7f9e67800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e600b3f80 0x7f9e600cc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:31.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.855+0000 7f9e67800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e60160e50 0x7f9e6015ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:31.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.855+0000 7f9e67800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e600b8060 0x7f9e600ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:31.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.855+0000 7f9e67800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e600b9070 con 0x7f9e600b8060 2024-04-02T07:52:31.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.855+0000 7f9e67800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e600b84f0 con 0x7f9e60160e50 2024-04-02T07:52:31.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.855+0000 7f9e67800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e600b87d0 con 0x7f9e600b3f80 2024-04-02T07:52:31.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.855+0000 7f9e65800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e600b8060 0x7f9e600ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:31.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.855+0000 7f9e64e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e600b3f80 0x7f9e600cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:31.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.855+0000 7f9e65800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e600b8060 0x7f9e600ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54472/0 (socket says 172.21.15.89:54472) 2024-04-02T07:52:31.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.855+0000 7f9e65800700 1 -- 172.21.15.89:0/2005691051 learned_addr learned my addr 172.21.15.89:0/2005691051 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:31.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.855+0000 7f9e5fe00700 1 --2- 172.21.15.89:0/2005691051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e60160e50 0x7f9e6015ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:31.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.856+0000 7f9e64e00700 1 -- 172.21.15.89:0/2005691051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e60160e50 msgr2=0x7f9e6015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:31.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.856+0000 7f9e64e00700 1 --2- 172.21.15.89:0/2005691051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e60160e50 0x7f9e6015ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.856+0000 7f9e64e00700 1 -- 172.21.15.89:0/2005691051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e600b8060 msgr2=0x7f9e600ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:31.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.856+0000 7f9e64e00700 1 --2- 172.21.15.89:0/2005691051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e600b8060 0x7f9e600ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:31.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.856+0000 7f9e64e00700 1 -- 172.21.15.89:0/2005691051 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e540094a0 con 0x7f9e600b3f80 2024-04-02T07:52:31.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.856+0000 7f9e65800700 1 --2- 172.21.15.89:0/2005691051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e600b8060 0x7f9e600ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:31.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.856+0000 7f9e64e00700 1 --2- 172.21.15.89:0/2005691051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e600b3f80 0x7f9e600cc550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9e6008b910 tx=0x7f9e54004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:31.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.856+0000 7f9e5cc00700 1 -- 172.21.15.89:0/2005691051 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54011630 con 0x7f9e600b3f80 2024-04-02T07:52:31.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.856+0000 7f9e5cc00700 1 -- 172.21.15.89:0/2005691051 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5401f070 con 0x7f9e600b3f80 2024-04-02T07:52:31.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.856+0000 7f9e67800700 1 -- 172.21.15.89:0/2005691051 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e600b8ab0 con 0x7f9e600b3f80 2024-04-02T07:52:31.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.856+0000 7f9e5cc00700 1 -- 172.21.15.89:0/2005691051 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e540119f0 con 0x7f9e600b3f80 2024-04-02T07:52:31.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.857+0000 7f9e67800700 1 -- 172.21.15.89:0/2005691051 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6015b080 con 0x7f9e600b3f80 2024-04-02T07:52:31.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.858+0000 7f9e67800700 1 -- 172.21.15.89:0/2005691051 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6015b210 con 0x7f9e600b3f80 2024-04-02T07:52:31.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.858+0000 7f9e5cc00700 1 -- 172.21.15.89:0/2005691051 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 47) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f9e54011b50 con 0x7f9e600b3f80 2024-04-02T07:52:31.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.859+0000 7f9e5cc00700 1 --2- 172.21.15.89:0/2005691051 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9e4006adb0 0x7f9e4006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:31.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.859+0000 7f9e5cc00700 1 -- 172.21.15.89:0/2005691051 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f9e54021070 con 0x7f9e600b3f80 2024-04-02T07:52:31.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.859+0000 7f9e5fe00700 1 -- 172.21.15.89:0/2005691051 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9e4006adb0 msgr2=0x7f9e4006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:31.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.859+0000 7f9e5fe00700 1 --2- 172.21.15.89:0/2005691051 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9e4006adb0 0x7f9e4006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:31.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:31.864+0000 7f9e5cc00700 1 -- 172.21.15.89:0/2005691051 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e6015b210 con 0x7f9e600b3f80 2024-04-02T07:52:31.906 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:31.904+0000 7fb9f298d240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:52:31.966 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:31.965+0000 7fcb1564a240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:52:31.996 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:31.996+0000 7fb9f298d240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:52:32.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.060+0000 7f9e5fe00700 1 -- 172.21.15.89:0/2005691051 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9e4006adb0 msgr2=0x7f9e4006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:32.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.060+0000 7f9e5fe00700 1 --2- 172.21.15.89:0/2005691051 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9e4006adb0 0x7f9e4006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:32.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.075+0000 7f9e67800700 1 -- 172.21.15.89:0/2005691051 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9e6015b210 con 0x7f9e600b3f80 2024-04-02T07:52:32.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.075+0000 7f9e5cc00700 1 -- 172.21.15.89:0/2005691051 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9e6015b210 con 0x7f9e600b3f80 2024-04-02T07:52:32.077 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:52:32.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.078+0000 7f9e67800700 1 -- 172.21.15.89:0/2005691051 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9e4006adb0 msgr2=0x7f9e4006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:32.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.078+0000 7f9e67800700 1 --2- 172.21.15.89:0/2005691051 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9e4006adb0 0x7f9e4006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.078+0000 7f9e67800700 1 -- 172.21.15.89:0/2005691051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e600b3f80 msgr2=0x7f9e600cc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:32.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.078+0000 7f9e67800700 1 --2- 172.21.15.89:0/2005691051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e600b3f80 0x7f9e600cc550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9e6008b910 tx=0x7f9e54004750 comp rx=0 tx=0).stop 2024-04-02T07:52:32.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.078+0000 7f9e67800700 1 -- 172.21.15.89:0/2005691051 shutdown_connections 2024-04-02T07:52:32.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.078+0000 7f9e67800700 1 --2- 172.21.15.89:0/2005691051 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9e4006adb0 0x7f9e4006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.078+0000 7f9e67800700 1 --2- 172.21.15.89:0/2005691051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e600b8060 0x7f9e600ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.078+0000 7f9e67800700 1 --2- 172.21.15.89:0/2005691051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e600b3f80 0x7f9e600cc550 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.078+0000 7f9e67800700 1 --2- 172.21.15.89:0/2005691051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e60160e50 0x7f9e6015ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.078+0000 7f9e67800700 1 -- 172.21.15.89:0/2005691051 >> 172.21.15.89:0/2005691051 conn(0x7f9e6008aff0 msgr2=0x7f9e6006cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:32.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.078+0000 7f9e67800700 1 -- 172.21.15.89:0/2005691051 shutdown_connections 2024-04-02T07:52:32.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.078+0000 7f9e67800700 1 -- 172.21.15.89:0/2005691051 wait complete. 2024-04-02T07:52:32.097 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:52:32.169 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:32.168+0000 7fcb1564a240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:52:32.268 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:32.267+0000 7fcb1564a240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:52:32.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.294+0000 7f82e7800700 1 Processor -- start 2024-04-02T07:52:32.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.294+0000 7f82e7800700 1 -- start start 2024-04-02T07:52:32.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.295+0000 7f82e7800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f82e015d9d0 0x7f82e00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:32.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.295+0000 7f82e7800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f82e00d2540 0x7f82e00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:32.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.295+0000 7f82e7800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e0161420 con 0x7f82e00d16a0 2024-04-02T07:52:32.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.295+0000 7f82e7800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e015edc0 con 0x7f82e00d2540 2024-04-02T07:52:32.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.295+0000 7f82e7800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e0161700 con 0x7f82e015d9d0 2024-04-02T07:52:32.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.295+0000 7f82e5800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f82e00d2540 0x7f82e00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:32.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.295+0000 7f82e5800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f82e00d2540 0x7f82e00ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54476/0 (socket says 172.21.15.89:54476) 2024-04-02T07:52:32.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.295+0000 7f82e5800700 1 -- 172.21.15.89:0/1684705524 learned_addr learned my addr 172.21.15.89:0/1684705524 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:32.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.295+0000 7f82e4e00700 1 --2- 172.21.15.89:0/1684705524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f82e015d9d0 0x7f82e00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:32.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.296+0000 7f82e5800700 1 -- 172.21.15.89:0/1684705524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f82e015d9d0 msgr2=0x7f82e00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:32.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.296+0000 7f82e5800700 1 --2- 172.21.15.89:0/1684705524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f82e015d9d0 0x7f82e00d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.296+0000 7f82e5800700 1 -- 172.21.15.89:0/1684705524 >> v1:172.21.15.89:6789/0 conn(0x7f82e00d16a0 legacy=0x7f82e00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:32.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.296+0000 7f82e5800700 1 -- 172.21.15.89:0/1684705524 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e01619e0 con 0x7f82e00d2540 2024-04-02T07:52:32.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.296+0000 7f82df400700 1 -- 172.21.15.89:0/1684705524 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703911583 0 0) 0x7f82e0161420 con 0x7f82e00d16a0 2024-04-02T07:52:32.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.296+0000 7f82e5800700 1 --2- 172.21.15.89:0/1684705524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82e00d2540 0x7f82e00ce360 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f82d8009e80 tx=0x7f82d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9881f83117953924 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:32.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.296+0000 7f82df400700 1 -- 172.21.15.89:0/1684705524 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d800e070 con 0x7f82e00d2540 2024-04-02T07:52:32.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.296+0000 7f82df400700 1 -- 172.21.15.89:0/1684705524 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d800b8d0 con 0x7f82e00d2540 2024-04-02T07:52:32.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.296+0000 7f82df400700 1 -- 172.21.15.89:0/1684705524 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d800bc80 con 0x7f82e00d2540 2024-04-02T07:52:32.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.297+0000 7f82e7800700 1 -- 172.21.15.89:0/1684705524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82e00d2540 msgr2=0x7f82e00ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:32.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.297+0000 7f82e7800700 1 --2- 172.21.15.89:0/1684705524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82e00d2540 0x7f82e00ce360 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f82d8009e80 tx=0x7f82d800b330 comp rx=0 tx=0).stop 2024-04-02T07:52:32.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.297+0000 7f82e7800700 1 -- 172.21.15.89:0/1684705524 shutdown_connections 2024-04-02T07:52:32.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.297+0000 7f82e7800700 1 --2- 172.21.15.89:0/1684705524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82e00d2540 0x7f82e00ce360 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.297+0000 7f82e7800700 1 --2- 172.21.15.89:0/1684705524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f82e015d9d0 0x7f82e00d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.297+0000 7f82e7800700 1 -- 172.21.15.89:0/1684705524 >> 172.21.15.89:0/1684705524 conn(0x7f82e008aff0 msgr2=0x7f82e00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:32.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.297+0000 7f82e7800700 1 -- 172.21.15.89:0/1684705524 shutdown_connections 2024-04-02T07:52:32.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.297+0000 7f82e7800700 1 -- 172.21.15.89:0/1684705524 wait complete. 2024-04-02T07:52:32.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.299+0000 7f82e7800700 1 Processor -- start 2024-04-02T07:52:32.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.299+0000 7f82e7800700 1 -- start start 2024-04-02T07:52:32.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.300+0000 7f82e7800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82e00d16a0 0x7f82e00b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:32.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.300+0000 7f82e7800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82e00d2540 0x7f82e00b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:32.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.300+0000 7f82e7800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f82e015d9d0 0x7f82e01f62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:32.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.300+0000 7f82e7800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e00b2430 con 0x7f82e00d2540 2024-04-02T07:52:32.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.300+0000 7f82e7800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e00b17b0 con 0x7f82e00d16a0 2024-04-02T07:52:32.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.300+0000 7f82e7800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e00b1a90 con 0x7f82e015d9d0 2024-04-02T07:52:32.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.300+0000 7f82dfe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82e00d2540 0x7f82e00b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:32.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.300+0000 7f82dfe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82e00d2540 0x7f82e00b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54484/0 (socket says 172.21.15.89:54484) 2024-04-02T07:52:32.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.300+0000 7f82dfe00700 1 -- 172.21.15.89:0/4292726008 learned_addr learned my addr 172.21.15.89:0/4292726008 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:32.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.300+0000 7f82e4e00700 1 --2- 172.21.15.89:0/4292726008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82e00d16a0 0x7f82e00b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:32.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.300+0000 7f82e5800700 1 --2- 172.21.15.89:0/4292726008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f82e015d9d0 0x7f82e01f62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:32.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.301+0000 7f82dfe00700 1 -- 172.21.15.89:0/4292726008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f82e015d9d0 msgr2=0x7f82e01f62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:32.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.301+0000 7f82dfe00700 1 --2- 172.21.15.89:0/4292726008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f82e015d9d0 0x7f82e01f62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.301+0000 7f82dfe00700 1 -- 172.21.15.89:0/4292726008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82e00d16a0 msgr2=0x7f82e00b7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:32.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.301+0000 7f82dfe00700 1 --2- 172.21.15.89:0/4292726008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82e00d16a0 0x7f82e00b7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.301+0000 7f82dfe00700 1 -- 172.21.15.89:0/4292726008 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d8009b30 con 0x7f82e00d2540 2024-04-02T07:52:32.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.301+0000 7f82dfe00700 1 --2- 172.21.15.89:0/4292726008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82e00d2540 0x7f82e00b0cf0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00c550 tx=0x7f82cc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:32.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.301+0000 7f82dcc00700 1 -- 172.21.15.89:0/4292726008 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc0054e0 con 0x7f82e00d2540 2024-04-02T07:52:32.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.301+0000 7f82dcc00700 1 -- 172.21.15.89:0/4292726008 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82cc003680 con 0x7f82e00d2540 2024-04-02T07:52:32.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.301+0000 7f82dcc00700 1 -- 172.21.15.89:0/4292726008 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc003a30 con 0x7f82e00d2540 2024-04-02T07:52:32.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.301+0000 7f82e7800700 1 -- 172.21.15.89:0/4292726008 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e00b1d70 con 0x7f82e00d2540 2024-04-02T07:52:32.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.301+0000 7f82e7800700 1 -- 172.21.15.89:0/4292726008 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e00b62b0 con 0x7f82e00d2540 2024-04-02T07:52:32.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.302+0000 7f82e7800700 1 -- 172.21.15.89:0/4292726008 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e00d1ca0 con 0x7f82e00d2540 2024-04-02T07:52:32.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.303+0000 7f82dcc00700 1 -- 172.21.15.89:0/4292726008 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 47) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f82cc003b90 con 0x7f82e00d2540 2024-04-02T07:52:32.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.304+0000 7f82dcc00700 1 --2- 172.21.15.89:0/4292726008 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f82c006adb0 0x7f82c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:32.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.304+0000 7f82e4e00700 1 -- 172.21.15.89:0/4292726008 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f82c006adb0 msgr2=0x7f82c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:32.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.304+0000 7f82dcc00700 1 -- 172.21.15.89:0/4292726008 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f82cc088b80 con 0x7f82e00d2540 2024-04-02T07:52:32.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.304+0000 7f82e4e00700 1 --2- 172.21.15.89:0/4292726008 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f82c006adb0 0x7f82c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:32.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.308+0000 7f82dcc00700 1 -- 172.21.15.89:0/4292726008 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f82cc053990 con 0x7f82e00d2540 2024-04-02T07:52:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.505+0000 7f82e4e00700 1 -- 172.21.15.89:0/4292726008 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f82c006adb0 msgr2=0x7f82c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.505+0000 7f82e4e00700 1 --2- 172.21.15.89:0/4292726008 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f82c006adb0 0x7f82c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:32.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.517+0000 7f82e7800700 1 -- 172.21.15.89:0/4292726008 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f82e0067720 con 0x7f82e00d2540 2024-04-02T07:52:32.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.518+0000 7f82dcc00700 1 -- 172.21.15.89:0/4292726008 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7f82cc0170a0 con 0x7f82e00d2540 2024-04-02T07:52:32.519 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:52:32.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.520+0000 7f82e7800700 1 -- 172.21.15.89:0/4292726008 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f82c006adb0 msgr2=0x7f82c006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:32.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.521+0000 7f82e7800700 1 --2- 172.21.15.89:0/4292726008 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f82c006adb0 0x7f82c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.521+0000 7f82e7800700 1 -- 172.21.15.89:0/4292726008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82e00d2540 msgr2=0x7f82e00b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:32.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.521+0000 7f82e7800700 1 --2- 172.21.15.89:0/4292726008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82e00d2540 0x7f82e00b0cf0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00c550 tx=0x7f82cc00c860 comp rx=0 tx=0).stop 2024-04-02T07:52:32.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.521+0000 7f82e7800700 1 -- 172.21.15.89:0/4292726008 shutdown_connections 2024-04-02T07:52:32.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.521+0000 7f82e7800700 1 --2- 172.21.15.89:0/4292726008 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f82c006adb0 0x7f82c006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.521+0000 7f82e7800700 1 --2- 172.21.15.89:0/4292726008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82e00d16a0 0x7f82e00b7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.521+0000 7f82e7800700 1 --2- 172.21.15.89:0/4292726008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f82e015d9d0 0x7f82e01f62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.521+0000 7f82e7800700 1 --2- 172.21.15.89:0/4292726008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82e00d2540 0x7f82e00b0cf0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.521+0000 7f82e7800700 1 -- 172.21.15.89:0/4292726008 >> 172.21.15.89:0/4292726008 conn(0x7f82e008aff0 msgr2=0x7f82e0160780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:32.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.521+0000 7f82e7800700 1 -- 172.21.15.89:0/4292726008 shutdown_connections 2024-04-02T07:52:32.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.521+0000 7f82e7800700 1 -- 172.21.15.89:0/4292726008 wait complete. 2024-04-02T07:52:32.540 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:52:32.720 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:32.719+0000 7fb9f298d240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:52:32.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.736+0000 7f97b8e00700 1 Processor -- start 2024-04-02T07:52:32.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.736+0000 7f97b8e00700 1 -- start start 2024-04-02T07:52:32.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.737+0000 7f97b8e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f97b40cbb30 0x7f97b40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:32.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.738+0000 7f97b8e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b40cc450 con 0x7f97b40cd020 2024-04-02T07:52:32.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.738+0000 7f97b8e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b40cdae0 con 0x7f97b40cd480 2024-04-02T07:52:32.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.738+0000 7f97b8e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97b40cd8f0 con 0x7f97b40cbb30 2024-04-02T07:52:32.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.738+0000 7f97b1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f97b40cbb30 0x7f97b40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:32.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.738+0000 7f97b1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f97b40cbb30 0x7f97b40cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54486/0 (socket says 172.21.15.89:54486) 2024-04-02T07:52:32.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.738+0000 7f97abe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f97b40cd020 0x7f97b40b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33118/0 (socket says 172.21.15.89:33118) 2024-04-02T07:52:32.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.738+0000 7f97b1e00700 1 -- 172.21.15.89:0/4252166325 learned_addr learned my addr 172.21.15.89:0/4252166325 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:32.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.738+0000 7f97b1e00700 1 -- 172.21.15.89:0/4252166325 >> v1:172.21.15.113:6789/0 conn(0x7f97b40cd480 legacy=0x7f97b406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:32.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.738+0000 7f97b1e00700 1 -- 172.21.15.89:0/4252166325 >> v1:172.21.15.89:6789/0 conn(0x7f97b40cd020 legacy=0x7f97b40b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:32.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.738+0000 7f97b1e00700 1 -- 172.21.15.89:0/4252166325 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b40cddc0 con 0x7f97b40cbb30 2024-04-02T07:52:32.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.739+0000 7f97b1e00700 1 --2- 172.21.15.89:0/4252166325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b40cbb30 0x7f97b40cbf10 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f97a40098f0 tx=0x7f97a4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1045305c1be8271 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:32.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.739+0000 7f97b1400700 1 -- 172.21.15.89:0/4252166325 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a400b8a0 con 0x7f97b40cbb30 2024-04-02T07:52:32.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.739+0000 7f97b1400700 1 -- 172.21.15.89:0/4252166325 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a400ba00 con 0x7f97b40cbb30 2024-04-02T07:52:32.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.739+0000 7f97b1400700 1 -- 172.21.15.89:0/4252166325 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a400bd90 con 0x7f97b40cbb30 2024-04-02T07:52:32.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.739+0000 7f97b8e00700 1 -- 172.21.15.89:0/4252166325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b40cbb30 msgr2=0x7f97b40cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:32.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.739+0000 7f97b8e00700 1 --2- 172.21.15.89:0/4252166325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b40cbb30 0x7f97b40cbf10 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f97a40098f0 tx=0x7f97a4009c00 comp rx=0 tx=0).stop 2024-04-02T07:52:32.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.740+0000 7f97b8e00700 1 -- 172.21.15.89:0/4252166325 shutdown_connections 2024-04-02T07:52:32.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.740+0000 7f97b8e00700 1 --2- 172.21.15.89:0/4252166325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b40cbb30 0x7f97b40cbf10 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.740+0000 7f97b8e00700 1 -- 172.21.15.89:0/4252166325 >> 172.21.15.89:0/4252166325 conn(0x7f97b40b0d70 msgr2=0x7f97b40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:32.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.740+0000 7f97b8e00700 1 -- 172.21.15.89:0/4252166325 shutdown_connections 2024-04-02T07:52:32.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.740+0000 7f97b8e00700 1 -- 172.21.15.89:0/4252166325 wait complete. 2024-04-02T07:52:32.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.741+0000 7f97b8e00700 1 Processor -- start 2024-04-02T07:52:32.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.742+0000 7f97b8e00700 1 -- start start 2024-04-02T07:52:32.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97b8e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97b40cbb30 0x7f97b4153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:32.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97b8e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b40cd020 0x7f97b40acec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:32.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97b8e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97b40cd480 0x7f97b40ad400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:32.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97b8e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b41e7070 con 0x7f97b40cd020 2024-04-02T07:52:32.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97b8e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b41e7350 con 0x7f97b40cbb30 2024-04-02T07:52:32.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97b8e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b41e7630 con 0x7f97b40cd480 2024-04-02T07:52:32.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97abe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b40cd020 0x7f97b40acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:32.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97abe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b40cd020 0x7f97b40acec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54492/0 (socket says 172.21.15.89:54492) 2024-04-02T07:52:32.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97abe00700 1 -- 172.21.15.89:0/4259861938 learned_addr learned my addr 172.21.15.89:0/4259861938 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:32.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97b1e00700 1 --2- 172.21.15.89:0/4259861938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97b40cbb30 0x7f97b4153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:32.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97b2800700 1 --2- 172.21.15.89:0/4259861938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97b40cd480 0x7f97b40ad400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:32.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97abe00700 1 -- 172.21.15.89:0/4259861938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97b40cd480 msgr2=0x7f97b40ad400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:32.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97abe00700 1 --2- 172.21.15.89:0/4259861938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97b40cd480 0x7f97b40ad400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97abe00700 1 -- 172.21.15.89:0/4259861938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97b40cbb30 msgr2=0x7f97b4153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:32.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97abe00700 1 --2- 172.21.15.89:0/4259861938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97b40cbb30 0x7f97b4153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.743+0000 7f97abe00700 1 -- 172.21.15.89:0/4259861938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a4009550 con 0x7f97b40cd020 2024-04-02T07:52:32.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.744+0000 7f97abe00700 1 --2- 172.21.15.89:0/4259861938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b40cd020 0x7f97b40acec0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f979c00c520 tx=0x7f979c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:32.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.744+0000 7f97aa000700 1 -- 172.21.15.89:0/4259861938 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c005cb0 con 0x7f97b40cd020 2024-04-02T07:52:32.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.744+0000 7f97aa000700 1 -- 172.21.15.89:0/4259861938 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979c005e10 con 0x7f97b40cd020 2024-04-02T07:52:32.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.744+0000 7f97aa000700 1 -- 172.21.15.89:0/4259861938 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c0072d0 con 0x7f97b40cd020 2024-04-02T07:52:32.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.744+0000 7f97b8e00700 1 -- 172.21.15.89:0/4259861938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b41e7910 con 0x7f97b40cd020 2024-04-02T07:52:32.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.744+0000 7f97b8e00700 1 -- 172.21.15.89:0/4259861938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b41e6520 con 0x7f97b40cd020 2024-04-02T07:52:32.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.745+0000 7f97b8e00700 1 -- 172.21.15.89:0/4259861938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b4067720 con 0x7f97b40cd020 2024-04-02T07:52:32.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.746+0000 7f97aa000700 1 -- 172.21.15.89:0/4259861938 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 47) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f979c007430 con 0x7f97b40cd020 2024-04-02T07:52:32.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.747+0000 7f97aa000700 1 --2- 172.21.15.89:0/4259861938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f979006adb0 0x7f979006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:32.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.747+0000 7f97aa000700 1 -- 172.21.15.89:0/4259861938 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f979c088a80 con 0x7f97b40cd020 2024-04-02T07:52:32.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.747+0000 7f97b1e00700 1 -- 172.21.15.89:0/4259861938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f979006adb0 msgr2=0x7f979006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:32.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.747+0000 7f97b1e00700 1 --2- 172.21.15.89:0/4259861938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f979006adb0 0x7f979006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:32.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.752+0000 7f97aa000700 1 -- 172.21.15.89:0/4259861938 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f979c0538b0 con 0x7f97b40cd020 2024-04-02T07:52:32.932 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:32.931+0000 7fb9f298d240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:52:32.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.948+0000 7f97b1e00700 1 -- 172.21.15.89:0/4259861938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f979006adb0 msgr2=0x7f979006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:32.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.948+0000 7f97b1e00700 1 --2- 172.21.15.89:0/4259861938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f979006adb0 0x7f979006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:32.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.958+0000 7f97b8e00700 1 -- 172.21.15.89:0/4259861938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f97b40ce2e0 con 0x7f97b40cd020 2024-04-02T07:52:32.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.959+0000 7f97aa000700 1 -- 172.21.15.89:0/4259861938 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7f979c017020 con 0x7f97b40cd020 2024-04-02T07:52:32.960 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:52:32.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.962+0000 7f97b8e00700 1 -- 172.21.15.89:0/4259861938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f979006adb0 msgr2=0x7f979006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:32.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.962+0000 7f97b8e00700 1 --2- 172.21.15.89:0/4259861938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f979006adb0 0x7f979006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.962+0000 7f97b8e00700 1 -- 172.21.15.89:0/4259861938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b40cd020 msgr2=0x7f97b40acec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:32.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.962+0000 7f97b8e00700 1 --2- 172.21.15.89:0/4259861938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b40cd020 0x7f97b40acec0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f979c00c520 tx=0x7f979c00c830 comp rx=0 tx=0).stop 2024-04-02T07:52:32.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.962+0000 7f97b8e00700 1 -- 172.21.15.89:0/4259861938 shutdown_connections 2024-04-02T07:52:32.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.962+0000 7f97b8e00700 1 --2- 172.21.15.89:0/4259861938 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f979006adb0 0x7f979006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.962+0000 7f97b8e00700 1 --2- 172.21.15.89:0/4259861938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97b40cbb30 0x7f97b4153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.962+0000 7f97b8e00700 1 --2- 172.21.15.89:0/4259861938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97b40cd480 0x7f97b40ad400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.962+0000 7f97b8e00700 1 --2- 172.21.15.89:0/4259861938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b40cd020 0x7f97b40acec0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:32.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.962+0000 7f97b8e00700 1 -- 172.21.15.89:0/4259861938 >> 172.21.15.89:0/4259861938 conn(0x7f97b40b0d70 msgr2=0x7f97b415c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:32.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.962+0000 7f97b8e00700 1 -- 172.21.15.89:0/4259861938 shutdown_connections 2024-04-02T07:52:32.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:32.962+0000 7f97b8e00700 1 -- 172.21.15.89:0/4259861938 wait complete. 2024-04-02T07:52:32.984 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:52:33.045 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:33.044+0000 7fcb1564a240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:52:33.141 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:33.140+0000 7fb9f298d240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:52:33.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.182+0000 7fc414e00700 1 Processor -- start 2024-04-02T07:52:33.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.182+0000 7fc414e00700 1 -- start start 2024-04-02T07:52:33.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.183+0000 7fc414e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc410160c90 0x7fc4100b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:33.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.183+0000 7fc414e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4100718e0 0x7fc4100b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:33.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.183+0000 7fc414e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4100ac4d0 con 0x7fc4100709c0 2024-04-02T07:52:33.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.183+0000 7fc414e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc410071e50 con 0x7fc4100718e0 2024-04-02T07:52:33.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.183+0000 7fc414e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc410072130 con 0x7fc410160c90 2024-04-02T07:52:33.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.184+0000 7fc40d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4100718e0 0x7fc4100b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:33.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.184+0000 7fc40d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4100718e0 0x7fc4100b2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54508/0 (socket says 172.21.15.89:54508) 2024-04-02T07:52:33.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.184+0000 7fc40e800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc4100709c0 0x7fc41006cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33134/0 (socket says 172.21.15.89:33134) 2024-04-02T07:52:33.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.184+0000 7fc40d400700 1 -- 172.21.15.89:0/900881196 learned_addr learned my addr 172.21.15.89:0/900881196 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:33.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.184+0000 7fc40de00700 1 --2- 172.21.15.89:0/900881196 >> v2:172.21.15.113:3300/0 conn(0x7fc410160c90 0x7fc4100b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:33.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.184+0000 7fc40d400700 1 -- 172.21.15.89:0/900881196 >> v2:172.21.15.113:3300/0 conn(0x7fc410160c90 msgr2=0x7fc4100b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:33.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.184+0000 7fc40d400700 1 --2- 172.21.15.89:0/900881196 >> v2:172.21.15.113:3300/0 conn(0x7fc410160c90 0x7fc4100b20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.184+0000 7fc40d400700 1 -- 172.21.15.89:0/900881196 >> v1:172.21.15.89:6789/0 conn(0x7fc4100709c0 legacy=0x7fc41006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:33.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.184+0000 7fc40d400700 1 -- 172.21.15.89:0/900881196 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc410072410 con 0x7fc4100718e0 2024-04-02T07:52:33.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.184+0000 7fc40ca00700 1 -- 172.21.15.89:0/900881196 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567318163 0 0) 0x7fc4100ac4d0 con 0x7fc4100709c0 2024-04-02T07:52:33.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.185+0000 7fc40d400700 1 --2- 172.21.15.89:0/900881196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4100718e0 0x7fc4100b2620 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fc410070090 tx=0x7fc3fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11515e6924d558c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:33.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.185+0000 7fc40ca00700 1 -- 172.21.15.89:0/900881196 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc011830 con 0x7fc4100718e0 2024-04-02T07:52:33.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.185+0000 7fc40ca00700 1 -- 172.21.15.89:0/900881196 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3fc011990 con 0x7fc4100718e0 2024-04-02T07:52:33.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.185+0000 7fc40ca00700 1 -- 172.21.15.89:0/900881196 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc011d20 con 0x7fc4100718e0 2024-04-02T07:52:33.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.185+0000 7fc414e00700 1 -- 172.21.15.89:0/900881196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4100718e0 msgr2=0x7fc4100b2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:33.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.185+0000 7fc414e00700 1 --2- 172.21.15.89:0/900881196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4100718e0 0x7fc4100b2620 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fc410070090 tx=0x7fc3fc009bd0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.185+0000 7fc414e00700 1 -- 172.21.15.89:0/900881196 shutdown_connections 2024-04-02T07:52:33.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.185+0000 7fc414e00700 1 --2- 172.21.15.89:0/900881196 >> v2:172.21.15.113:3300/0 conn(0x7fc410160c90 0x7fc4100b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.185+0000 7fc414e00700 1 --2- 172.21.15.89:0/900881196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4100718e0 0x7fc4100b2620 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.185+0000 7fc414e00700 1 -- 172.21.15.89:0/900881196 >> 172.21.15.89:0/900881196 conn(0x7fc4100cf190 msgr2=0x7fc4100cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:33.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.185+0000 7fc414e00700 1 -- 172.21.15.89:0/900881196 shutdown_connections 2024-04-02T07:52:33.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.186+0000 7fc414e00700 1 -- 172.21.15.89:0/900881196 wait complete. 2024-04-02T07:52:33.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.187+0000 7fc414e00700 1 Processor -- start 2024-04-02T07:52:33.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.187+0000 7fc414e00700 1 -- start start 2024-04-02T07:52:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.188+0000 7fc414e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4100709c0 0x7fc4101f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.188+0000 7fc414e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4100718e0 0x7fc41015a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.188+0000 7fc414e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc410160c90 0x7fc41015b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.188+0000 7fc414e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4101f6880 con 0x7fc4100718e0 2024-04-02T07:52:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.188+0000 7fc414e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4101f37c0 con 0x7fc410160c90 2024-04-02T07:52:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.188+0000 7fc414e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4101f6230 con 0x7fc4100709c0 2024-04-02T07:52:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.189+0000 7fc40d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4100718e0 0x7fc41015a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.189+0000 7fc40d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4100718e0 0x7fc41015a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54514/0 (socket says 172.21.15.89:54514) 2024-04-02T07:52:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.189+0000 7fc40d400700 1 -- 172.21.15.89:0/2257796316 learned_addr learned my addr 172.21.15.89:0/2257796316 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.189+0000 7fc40de00700 1 --2- 172.21.15.89:0/2257796316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4100709c0 0x7fc4101f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.189+0000 7fc40e800700 1 --2- 172.21.15.89:0/2257796316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc410160c90 0x7fc41015b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.189+0000 7fc40d400700 1 -- 172.21.15.89:0/2257796316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4100709c0 msgr2=0x7fc4101f6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:33.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.189+0000 7fc40d400700 1 --2- 172.21.15.89:0/2257796316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4100709c0 0x7fc4101f6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.189+0000 7fc40d400700 1 -- 172.21.15.89:0/2257796316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc410160c90 msgr2=0x7fc41015b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.189+0000 7fc40d400700 1 --2- 172.21.15.89:0/2257796316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc410160c90 0x7fc41015b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.189+0000 7fc40d400700 1 -- 172.21.15.89:0/2257796316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3fc009550 con 0x7fc4100718e0 2024-04-02T07:52:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.189+0000 7fc40d400700 1 --2- 172.21.15.89:0/2257796316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4100718e0 0x7fc41015a990 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc009c10 tx=0x7fc3fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.189+0000 7fc406000700 1 -- 172.21.15.89:0/2257796316 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc004510 con 0x7fc4100718e0 2024-04-02T07:52:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.190+0000 7fc406000700 1 -- 172.21.15.89:0/2257796316 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3fc0036a0 con 0x7fc4100718e0 2024-04-02T07:52:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.190+0000 7fc414e00700 1 -- 172.21.15.89:0/2257796316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4101f6510 con 0x7fc4100718e0 2024-04-02T07:52:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.190+0000 7fc406000700 1 -- 172.21.15.89:0/2257796316 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc003a10 con 0x7fc4100718e0 2024-04-02T07:52:33.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.190+0000 7fc414e00700 1 -- 172.21.15.89:0/2257796316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4101f3aa0 con 0x7fc4100718e0 2024-04-02T07:52:33.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.191+0000 7fc414e00700 1 -- 172.21.15.89:0/2257796316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4101f69f0 con 0x7fc4100718e0 2024-04-02T07:52:33.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.192+0000 7fc406000700 1 -- 172.21.15.89:0/2257796316 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 47) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fc3fc003b70 con 0x7fc4100718e0 2024-04-02T07:52:33.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.193+0000 7fc406000700 1 --2- 172.21.15.89:0/2257796316 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc3f006adb0 0x7fc3f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:33.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.193+0000 7fc406000700 1 -- 172.21.15.89:0/2257796316 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fc3fc021030 con 0x7fc4100718e0 2024-04-02T07:52:33.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.193+0000 7fc40de00700 1 -- 172.21.15.89:0/2257796316 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc3f006adb0 msgr2=0x7fc3f006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:33.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.193+0000 7fc40de00700 1 --2- 172.21.15.89:0/2257796316 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc3f006adb0 0x7fc3f006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:33.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.197+0000 7fc406000700 1 -- 172.21.15.89:0/2257796316 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc3fc055bd0 con 0x7fc4100718e0 2024-04-02T07:52:33.305 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:33.304+0000 7fcb1564a240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:52:33.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.394+0000 7fc40de00700 1 -- 172.21.15.89:0/2257796316 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc3f006adb0 msgr2=0x7fc3f006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:33.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.394+0000 7fc40de00700 1 --2- 172.21.15.89:0/2257796316 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc3f006adb0 0x7fc3f006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:33.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.491+0000 7fc414e00700 1 -- 172.21.15.89:0/2257796316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4101f66a0 con 0x7fc4100718e0 2024-04-02T07:52:33.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.492+0000 7fc406000700 1 -- 172.21.15.89:0/2257796316 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fc3fc015090 con 0x7fc4100718e0 2024-04-02T07:52:33.493 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:33.493 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6556,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3307197948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3307197948},{"type":"v1","addr":"172.21.15.113:6835","nonce":3307197948}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1797655686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1797655686},{"type":"v1","addr":"172.21.15.113:6834","nonce":1797655686}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4203087300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4203087300},{"type":"v1","addr":"172.21.15.89:6836","nonce":4203087300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:52:09.066449+0000","modified":"2024-04-02T07:52:12.080422+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3228984565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3228984565},{"type":"v1","addr":"172.21.15.89:6837","nonce":3228984565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-02T07:52:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.494+0000 7fc414e00700 1 -- 172.21.15.89:0/2257796316 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc3f006adb0 msgr2=0x7fc3f006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.494+0000 7fc414e00700 1 --2- 172.21.15.89:0/2257796316 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc3f006adb0 0x7fc3f006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.494+0000 7fc414e00700 1 -- 172.21.15.89:0/2257796316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4100718e0 msgr2=0x7fc41015a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.494+0000 7fc414e00700 1 --2- 172.21.15.89:0/2257796316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4100718e0 0x7fc41015a990 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc009c10 tx=0x7fc3fc004750 comp rx=0 tx=0).stop 2024-04-02T07:52:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.494+0000 7fc414e00700 1 -- 172.21.15.89:0/2257796316 shutdown_connections 2024-04-02T07:52:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.494+0000 7fc414e00700 1 --2- 172.21.15.89:0/2257796316 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc3f006adb0 0x7fc3f006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.494+0000 7fc414e00700 1 --2- 172.21.15.89:0/2257796316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc410160c90 0x7fc41015b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.494+0000 7fc414e00700 1 --2- 172.21.15.89:0/2257796316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4100709c0 0x7fc4101f6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.494+0000 7fc414e00700 1 --2- 172.21.15.89:0/2257796316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4100718e0 0x7fc41015a990 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.494+0000 7fc414e00700 1 -- 172.21.15.89:0/2257796316 >> 172.21.15.89:0/2257796316 conn(0x7fc4100cf190 msgr2=0x7fc4100b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:33.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.494+0000 7fc414e00700 1 -- 172.21.15.89:0/2257796316 shutdown_connections 2024-04-02T07:52:33.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.494+0000 7fc414e00700 1 -- 172.21.15.89:0/2257796316 wait complete. 2024-04-02T07:52:33.497 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 41 2024-04-02T07:52:33.514 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:52:33.521 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:33.520+0000 7fcb1564a240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:52:33.609 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:33.607+0000 7fb9f298d240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:52:33.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.708+0000 7fe71b200700 1 Processor -- start 2024-04-02T07:52:33.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.708+0000 7fe71b200700 1 -- start start 2024-04-02T07:52:33.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.710+0000 7fe71b200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe7140cebd0 0x7fe7140cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:33.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.710+0000 7fe71b200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71406f850 con 0x7fe71406a790 2024-04-02T07:52:33.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.710+0000 7fe71b200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71406e3f0 con 0x7fe7140afbe0 2024-04-02T07:52:33.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.710+0000 7fe71b200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe71406fad0 con 0x7fe7140cebd0 2024-04-02T07:52:33.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.710+0000 7fe713e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe71406a790 0x7fe71406ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33142/0 (socket says 172.21.15.89:33142) 2024-04-02T07:52:33.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.710+0000 7fe713e00700 1 -- 172.21.15.89:0/3059315911 learned_addr learned my addr 172.21.15.89:0/3059315911 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:33.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.710+0000 7fe719200700 1 --2- 172.21.15.89:0/3059315911 >> v2:172.21.15.113:3300/0 conn(0x7fe7140cebd0 0x7fe7140cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:33.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.710+0000 7fe712a00700 1 -- 172.21.15.89:0/3059315911 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610291837 0 0) 0x7fe71406f850 con 0x7fe71406a790 2024-04-02T07:52:33.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.711+0000 7fe712a00700 1 -- 172.21.15.89:0/3059315911 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6f80035b0 con 0x7fe71406a790 2024-04-02T07:52:33.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.711+0000 7fe719200700 1 -- 172.21.15.89:0/3059315911 >> v1:172.21.15.113:6789/0 conn(0x7fe7140afbe0 legacy=0x7fe71406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:33.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.711+0000 7fe719200700 1 -- 172.21.15.89:0/3059315911 >> v1:172.21.15.89:6789/0 conn(0x7fe71406a790 legacy=0x7fe71406ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:33.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.711+0000 7fe719200700 1 -- 172.21.15.89:0/3059315911 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7140af970 con 0x7fe7140cebd0 2024-04-02T07:52:33.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.711+0000 7fe719200700 1 --2- 172.21.15.89:0/3059315911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7140cebd0 0x7fe7140cf000 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009d90 tx=0x7fe70c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a0edae582510fff server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:33.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.711+0000 7fe712a00700 1 -- 172.21.15.89:0/3059315911 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c00e070 con 0x7fe7140cebd0 2024-04-02T07:52:33.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.711+0000 7fe712a00700 1 -- 172.21.15.89:0/3059315911 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c00bb20 con 0x7fe7140cebd0 2024-04-02T07:52:33.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.711+0000 7fe712a00700 1 -- 172.21.15.89:0/3059315911 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c00be90 con 0x7fe7140cebd0 2024-04-02T07:52:33.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.712+0000 7fe71b200700 1 -- 172.21.15.89:0/3059315911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7140cebd0 msgr2=0x7fe7140cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:33.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.712+0000 7fe71b200700 1 --2- 172.21.15.89:0/3059315911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7140cebd0 0x7fe7140cf000 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009d90 tx=0x7fe70c00b330 comp rx=0 tx=0).stop 2024-04-02T07:52:33.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.712+0000 7fe71b200700 1 -- 172.21.15.89:0/3059315911 shutdown_connections 2024-04-02T07:52:33.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.712+0000 7fe71b200700 1 --2- 172.21.15.89:0/3059315911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7140cebd0 0x7fe7140cf000 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.712+0000 7fe71b200700 1 -- 172.21.15.89:0/3059315911 >> 172.21.15.89:0/3059315911 conn(0x7fe7140b5ab0 msgr2=0x7fe7140b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:33.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.712+0000 7fe71b200700 1 -- 172.21.15.89:0/3059315911 shutdown_connections 2024-04-02T07:52:33.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.712+0000 7fe71b200700 1 -- 172.21.15.89:0/3059315911 wait complete. 2024-04-02T07:52:33.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.714+0000 7fe71b200700 1 Processor -- start 2024-04-02T07:52:33.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.715+0000 7fe71b200700 1 -- start start 2024-04-02T07:52:33.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.716+0000 7fe71b200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe71406a790 0x7fe714149b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:33.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.716+0000 7fe71b200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7140afbe0 0x7fe7141467b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:33.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.716+0000 7fe71b200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7140cebd0 0x7fe7141488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:33.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.716+0000 7fe71b200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7141484a0 con 0x7fe7140cebd0 2024-04-02T07:52:33.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.716+0000 7fe71b200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe714146cf0 con 0x7fe7140afbe0 2024-04-02T07:52:33.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.716+0000 7fe71b200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe714148e00 con 0x7fe71406a790 2024-04-02T07:52:33.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.716+0000 7fe719200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7140cebd0 0x7fe7141488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:33.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.716+0000 7fe719200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7140cebd0 0x7fe7141488c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54526/0 (socket says 172.21.15.89:54526) 2024-04-02T07:52:33.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.716+0000 7fe713e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe71406a790 0x7fe714149b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:33.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.716+0000 7fe719200700 1 -- 172.21.15.89:0/774663143 learned_addr learned my addr 172.21.15.89:0/774663143 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:33.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.716+0000 7fe713400700 1 --2- 172.21.15.89:0/774663143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7140afbe0 0x7fe7141467b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:33.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.717+0000 7fe719200700 1 -- 172.21.15.89:0/774663143 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe71406a790 msgr2=0x7fe714149b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:33.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.717+0000 7fe719200700 1 --2- 172.21.15.89:0/774663143 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe71406a790 0x7fe714149b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.717+0000 7fe719200700 1 -- 172.21.15.89:0/774663143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7140afbe0 msgr2=0x7fe7141467b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:33.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.717+0000 7fe719200700 1 --2- 172.21.15.89:0/774663143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7140afbe0 0x7fe7141467b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.717+0000 7fe719200700 1 -- 172.21.15.89:0/774663143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c009a70 con 0x7fe7140cebd0 2024-04-02T07:52:33.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.717+0000 7fe719200700 1 --2- 172.21.15.89:0/774663143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7140cebd0 0x7fe7141488c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fe70c00b370 tx=0x7fe70c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:33.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.717+0000 7fe70be00700 1 -- 172.21.15.89:0/774663143 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c012430 con 0x7fe7140cebd0 2024-04-02T07:52:33.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.717+0000 7fe70be00700 1 -- 172.21.15.89:0/774663143 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c012590 con 0x7fe7140cebd0 2024-04-02T07:52:33.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.717+0000 7fe71b200700 1 -- 172.21.15.89:0/774663143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe714147820 con 0x7fe7140cebd0 2024-04-02T07:52:33.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.717+0000 7fe70be00700 1 -- 172.21.15.89:0/774663143 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c012940 con 0x7fe7140cebd0 2024-04-02T07:52:33.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.717+0000 7fe71b200700 1 -- 172.21.15.89:0/774663143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe714147de0 con 0x7fe7140cebd0 2024-04-02T07:52:33.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.718+0000 7fe71b200700 1 -- 172.21.15.89:0/774663143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7141485e0 con 0x7fe7140cebd0 2024-04-02T07:52:33.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.719+0000 7fe70be00700 1 -- 172.21.15.89:0/774663143 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 47) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fe70c012aa0 con 0x7fe7140cebd0 2024-04-02T07:52:33.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.720+0000 7fe70be00700 1 --2- 172.21.15.89:0/774663143 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6f406adb0 0x7fe6f406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:33.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.720+0000 7fe70be00700 1 -- 172.21.15.89:0/774663143 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fe70c08b530 con 0x7fe7140cebd0 2024-04-02T07:52:33.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.720+0000 7fe713e00700 1 -- 172.21.15.89:0/774663143 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6f406adb0 msgr2=0x7fe6f406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:33.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.720+0000 7fe713e00700 1 --2- 172.21.15.89:0/774663143 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6f406adb0 0x7fe6f406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:33.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.724+0000 7fe70be00700 1 -- 172.21.15.89:0/774663143 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe70c056370 con 0x7fe7140cebd0 2024-04-02T07:52:33.817 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:33.816+0000 7fb9f298d240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:52:33.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.921+0000 7fe713e00700 1 -- 172.21.15.89:0/774663143 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6f406adb0 msgr2=0x7fe6f406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:33.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.921+0000 7fe713e00700 1 --2- 172.21.15.89:0/774663143 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6f406adb0 0x7fe6f406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:33.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.935+0000 7fe71b200700 1 -- 172.21.15.89:0/774663143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe714067720 con 0x7fe7140cebd0 2024-04-02T07:52:33.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.937+0000 7fe70be00700 1 -- 172.21.15.89:0/774663143 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+17058 (secure 0 0 0) 0x7fe70c059990 con 0x7fe7140cebd0 2024-04-02T07:52:33.938 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:33.938 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":98,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:52:25.709797+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:52:06.991300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-02T07:52:08.054066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/1957897648":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/1991189445":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/4248118690":"2024-04-03T07:52:25.709764+0000","172.21.15.89:6833/43357":"2024-04-03T07:52:25.709764+0000","172.21.15.89:6832/43357":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/3231846713":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/3820568899":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/3513660462":"2024-04-03T07:52:25.709764+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:52:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.939+0000 7fe71b200700 1 -- 172.21.15.89:0/774663143 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6f406adb0 msgr2=0x7fe6f406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.939+0000 7fe71b200700 1 --2- 172.21.15.89:0/774663143 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6f406adb0 0x7fe6f406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.939+0000 7fe71b200700 1 -- 172.21.15.89:0/774663143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7140cebd0 msgr2=0x7fe7141488c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.939+0000 7fe71b200700 1 --2- 172.21.15.89:0/774663143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7140cebd0 0x7fe7141488c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fe70c00b370 tx=0x7fe70c004ad0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.939+0000 7fe71b200700 1 -- 172.21.15.89:0/774663143 shutdown_connections 2024-04-02T07:52:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.939+0000 7fe71b200700 1 --2- 172.21.15.89:0/774663143 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6f406adb0 0x7fe6f406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.939+0000 7fe71b200700 1 --2- 172.21.15.89:0/774663143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7140cebd0 0x7fe7141488c0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.939+0000 7fe71b200700 1 --2- 172.21.15.89:0/774663143 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe71406a790 0x7fe714149b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.939+0000 7fe71b200700 1 --2- 172.21.15.89:0/774663143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7140afbe0 0x7fe7141467b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.939+0000 7fe71b200700 1 -- 172.21.15.89:0/774663143 >> 172.21.15.89:0/774663143 conn(0x7fe7140b5ab0 msgr2=0x7fe71415e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:33.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.939+0000 7fe71b200700 1 -- 172.21.15.89:0/774663143 shutdown_connections 2024-04-02T07:52:33.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:33.939+0000 7fe71b200700 1 -- 172.21.15.89:0/774663143 wait complete. 2024-04-02T07:52:33.959 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T07:52:33.959 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:52:34.027 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:34.026+0000 7fcb1564a240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:52:34.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.194+0000 7f6deac00700 1 Processor -- start 2024-04-02T07:52:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.194+0000 7f6deac00700 1 -- start start 2024-04-02T07:52:34.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.196+0000 7f6deac00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6de40ceec0 0x7f6de41612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.196+0000 7f6deac00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de4161c50 con 0x7f6de415d640 2024-04-02T07:52:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.196+0000 7f6deac00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de40d30d0 con 0x7f6de4161800 2024-04-02T07:52:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.196+0000 7f6deac00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6de40b6840 con 0x7f6de40ceec0 2024-04-02T07:52:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.196+0000 7f6ddbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6de40ceec0 0x7f6de41612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.196+0000 7f6de3e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6de415d640 0x7f6de40d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33148/0 (socket says 172.21.15.89:33148) 2024-04-02T07:52:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.196+0000 7f6de3e00700 1 -- 172.21.15.89:0/3599433878 learned_addr learned my addr 172.21.15.89:0/3599433878 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.196+0000 7f6ddbe00700 1 -- 172.21.15.89:0/3599433878 >> v1:172.21.15.113:6789/0 conn(0x7f6de4161800 legacy=0x7f6de40b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.196+0000 7f6ddbe00700 1 -- 172.21.15.89:0/3599433878 >> v1:172.21.15.89:6789/0 conn(0x7f6de415d640 legacy=0x7f6de40d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.196+0000 7f6ddbe00700 1 -- 172.21.15.89:0/3599433878 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de40d2b30 con 0x7f6de40ceec0 2024-04-02T07:52:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.196+0000 7f6de3400700 1 -- 172.21.15.89:0/3599433878 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579980070 0 0) 0x7f6dd4001110 con 0x7f6de415d640 2024-04-02T07:52:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.197+0000 7f6ddbe00700 1 --2- 172.21.15.89:0/3599433878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6de40ceec0 0x7f6de41612c0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009840 tx=0x7f6dcc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505b143b55afa4b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.197+0000 7f6de3400700 1 -- 172.21.15.89:0/3599433878 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc011830 con 0x7f6de40ceec0 2024-04-02T07:52:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.197+0000 7f6de3400700 1 -- 172.21.15.89:0/3599433878 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dcc011990 con 0x7f6de40ceec0 2024-04-02T07:52:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.197+0000 7f6de3400700 1 -- 172.21.15.89:0/3599433878 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc011d20 con 0x7f6de40ceec0 2024-04-02T07:52:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.198+0000 7f6deac00700 1 -- 172.21.15.89:0/3599433878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6de40ceec0 msgr2=0x7f6de41612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.198+0000 7f6deac00700 1 --2- 172.21.15.89:0/3599433878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6de40ceec0 0x7f6de41612c0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009840 tx=0x7f6dcc009b50 comp rx=0 tx=0).stop 2024-04-02T07:52:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.198+0000 7f6deac00700 1 -- 172.21.15.89:0/3599433878 shutdown_connections 2024-04-02T07:52:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.198+0000 7f6deac00700 1 --2- 172.21.15.89:0/3599433878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6de40ceec0 0x7f6de41612c0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.198+0000 7f6deac00700 1 -- 172.21.15.89:0/3599433878 >> 172.21.15.89:0/3599433878 conn(0x7f6de408aff0 msgr2=0x7f6de40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.198+0000 7f6deac00700 1 -- 172.21.15.89:0/3599433878 shutdown_connections 2024-04-02T07:52:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.198+0000 7f6deac00700 1 -- 172.21.15.89:0/3599433878 wait complete. 2024-04-02T07:52:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.200+0000 7f6deac00700 1 Processor -- start 2024-04-02T07:52:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.200+0000 7f6deac00700 1 -- start start 2024-04-02T07:52:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6deac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6de40ceec0 0x7f6de41f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6deac00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6de415d640 0x7f6de409f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6deac00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6de4161800 0x7f6de409fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6deac00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de40ca830 con 0x7f6de415d640 2024-04-02T07:52:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6deac00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de40ce580 con 0x7f6de4161800 2024-04-02T07:52:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6deac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de40ce7d0 con 0x7f6de40ceec0 2024-04-02T07:52:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6ddbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6de415d640 0x7f6de409f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6ddbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6de415d640 0x7f6de409f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54538/0 (socket says 172.21.15.89:54538) 2024-04-02T07:52:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6ddbe00700 1 -- 172.21.15.89:0/2230471133 learned_addr learned my addr 172.21.15.89:0/2230471133 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6de3e00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6de40ceec0 0x7f6de41f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6de8c00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6de4161800 0x7f6de409fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6ddbe00700 1 -- 172.21.15.89:0/2230471133 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6de40ceec0 msgr2=0x7f6de41f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6ddbe00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6de40ceec0 0x7f6de41f3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6ddbe00700 1 -- 172.21.15.89:0/2230471133 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6de4161800 msgr2=0x7f6de409fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6ddbe00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6de4161800 0x7f6de409fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6ddbe00700 1 -- 172.21.15.89:0/2230471133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc0094a0 con 0x7f6de415d640 2024-04-02T07:52:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.201+0000 7f6de3e00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6de40ceec0 0x7f6de41f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.202+0000 7f6ddbe00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6de415d640 0x7f6de409f6b0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc014040 tx=0x7f6dcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.202+0000 7f6de0c00700 1 -- 172.21.15.89:0/2230471133 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc01a7e0 con 0x7f6de415d640 2024-04-02T07:52:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.202+0000 7f6de0c00700 1 -- 172.21.15.89:0/2230471133 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dcc01a940 con 0x7f6de415d640 2024-04-02T07:52:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.202+0000 7f6deac00700 1 -- 172.21.15.89:0/2230471133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de40ceab0 con 0x7f6de415d640 2024-04-02T07:52:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.202+0000 7f6de0c00700 1 -- 172.21.15.89:0/2230471133 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc01acb0 con 0x7f6de415d640 2024-04-02T07:52:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.202+0000 7f6deac00700 1 -- 172.21.15.89:0/2230471133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de409e490 con 0x7f6de415d640 2024-04-02T07:52:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.203+0000 7f6deac00700 1 -- 172.21.15.89:0/2230471133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de406cec0 con 0x7f6de415d640 2024-04-02T07:52:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.205+0000 7f6de0c00700 1 -- 172.21.15.89:0/2230471133 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 47) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f6dcc022070 con 0x7f6de415d640 2024-04-02T07:52:34.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.206+0000 7f6de0c00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6dc006adb0 0x7f6dc006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:34.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.206+0000 7f6de0c00700 1 -- 172.21.15.89:0/2230471133 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f6dcc08a910 con 0x7f6de415d640 2024-04-02T07:52:34.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.206+0000 7f6de3e00700 1 -- 172.21.15.89:0/2230471133 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6dc006adb0 msgr2=0x7f6dc006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:34.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.206+0000 7f6de3e00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6dc006adb0 0x7f6dc006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:34.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.210+0000 7f6de0c00700 1 -- 172.21.15.89:0/2230471133 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6dcc055710 con 0x7f6de415d640 2024-04-02T07:52:34.252 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:34.250+0000 7fb9f298d240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:52:34.259 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:34.255+0000 7fcb1564a240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:52:34.360 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:34.359+0000 7fb9f298d240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:52:34.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.407+0000 7f6de3e00700 1 -- 172.21.15.89:0/2230471133 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6dc006adb0 msgr2=0x7f6dc006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:34.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.407+0000 7f6de3e00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6dc006adb0 0x7f6dc006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:34.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.473+0000 7f6deac00700 1 -- 172.21.15.89:0/2230471133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6de4067720 con 0x7f6de415d640 2024-04-02T07:52:34.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.474+0000 7f6de0c00700 1 -- 172.21.15.89:0/2230471133 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f6dcc058d30 con 0x7f6de415d640 2024-04-02T07:52:34.475 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:34.475 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6556,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3307197948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3307197948},{"type":"v1","addr":"172.21.15.113:6835","nonce":3307197948}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1797655686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1797655686},{"type":"v1","addr":"172.21.15.113:6834","nonce":1797655686}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4203087300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4203087300},{"type":"v1","addr":"172.21.15.89:6836","nonce":4203087300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:52:09.066449+0000","modified":"2024-04-02T07:52:12.080422+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3228984565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3228984565},{"type":"v1","addr":"172.21.15.89:6837","nonce":3228984565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-02T07:52:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.477+0000 7f6deac00700 1 -- 172.21.15.89:0/2230471133 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6dc006adb0 msgr2=0x7f6dc006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.477+0000 7f6deac00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6dc006adb0 0x7f6dc006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.477+0000 7f6deac00700 1 -- 172.21.15.89:0/2230471133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6de415d640 msgr2=0x7f6de409f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.477+0000 7f6deac00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6de415d640 0x7f6de409f6b0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc014040 tx=0x7f6dcc004750 comp rx=0 tx=0).stop 2024-04-02T07:52:34.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.477+0000 7f6deac00700 1 -- 172.21.15.89:0/2230471133 shutdown_connections 2024-04-02T07:52:34.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.477+0000 7f6deac00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6dc006adb0 0x7f6dc006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:34.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.477+0000 7f6deac00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6de4161800 0x7f6de409fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:34.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.477+0000 7f6deac00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6de40ceec0 0x7f6de41f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:34.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.477+0000 7f6deac00700 1 --2- 172.21.15.89:0/2230471133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6de415d640 0x7f6de409f6b0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:34.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.477+0000 7f6deac00700 1 -- 172.21.15.89:0/2230471133 >> 172.21.15.89:0/2230471133 conn(0x7f6de408aff0 msgr2=0x7f6de40d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:34.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.477+0000 7f6deac00700 1 -- 172.21.15.89:0/2230471133 shutdown_connections 2024-04-02T07:52:34.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.477+0000 7f6deac00700 1 -- 172.21.15.89:0/2230471133 wait complete. 2024-04-02T07:52:34.479 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 41 2024-04-02T07:52:34.499 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:52:34.714 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:34.713+0000 7fcb1564a240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:52:34.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.742+0000 7f85db000700 1 Processor -- start 2024-04-02T07:52:34.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.742+0000 7f85db000700 1 -- start start 2024-04-02T07:52:34.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.744+0000 7f85db000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85d40d05e0 0x7f85d40d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:34.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.744+0000 7f85db000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f85d40b6990 0x7f85d40b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:34.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.744+0000 7f85db000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d40d0f80 con 0x7f85d415da50 2024-04-02T07:52:34.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.744+0000 7f85db000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d40b95d0 con 0x7f85d40b6990 2024-04-02T07:52:34.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.744+0000 7f85db000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d415de30 con 0x7f85d40d05e0 2024-04-02T07:52:34.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.745+0000 7f85d9000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f85d40b6990 0x7f85d40b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:34.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.745+0000 7f85d3400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85d40d05e0 0x7f85d40d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:34.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.745+0000 7f85d9000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f85d40b6990 0x7f85d40b6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:48596/0 (socket says 172.21.15.89:48596) 2024-04-02T07:52:34.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.745+0000 7f85d9000700 1 -- 172.21.15.89:0/3333514400 learned_addr learned my addr 172.21.15.89:0/3333514400 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:34.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.745+0000 7f85d3e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f85d415da50 0x7f85d40ce730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:55068/0 (socket says 172.21.15.89:55068) 2024-04-02T07:52:34.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.745+0000 7f85d9000700 1 -- 172.21.15.89:0/3333514400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85d40d05e0 msgr2=0x7f85d40d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:34.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.745+0000 7f85d9000700 1 --2- 172.21.15.89:0/3333514400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85d40d05e0 0x7f85d40d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:34.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.745+0000 7f85d9000700 1 -- 172.21.15.89:0/3333514400 >> v1:172.21.15.113:6789/0 conn(0x7f85d415da50 legacy=0x7f85d40ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:34.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.745+0000 7f85d9000700 1 -- 172.21.15.89:0/3333514400 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d40d24e0 con 0x7f85d40b6990 2024-04-02T07:52:34.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.745+0000 7f85d3400700 1 --2- 172.21.15.89:0/3333514400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85d40d05e0 0x7f85d40d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:34.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.745+0000 7f85d9000700 1 --2- 172.21.15.89:0/3333514400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85d40b6990 0x7f85d40b6e10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f85cc009e50 tx=0x7f85cc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0d3220b4776ce75 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:34.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.753+0000 7f85d2a00700 1 -- 172.21.15.89:0/3333514400 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc00e070 con 0x7f85d40b6990 2024-04-02T07:52:34.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.753+0000 7f85d2a00700 1 -- 172.21.15.89:0/3333514400 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85cc00b910 con 0x7f85d40b6990 2024-04-02T07:52:34.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.753+0000 7f85d2a00700 1 -- 172.21.15.89:0/3333514400 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc00bcc0 con 0x7f85d40b6990 2024-04-02T07:52:34.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.754+0000 7f85db000700 1 -- 172.21.15.89:0/3333514400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85d40b6990 msgr2=0x7f85d40b6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:34.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.754+0000 7f85db000700 1 --2- 172.21.15.89:0/3333514400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85d40b6990 0x7f85d40b6e10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f85cc009e50 tx=0x7f85cc00b330 comp rx=0 tx=0).stop 2024-04-02T07:52:34.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.754+0000 7f85db000700 1 -- 172.21.15.89:0/3333514400 shutdown_connections 2024-04-02T07:52:34.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.754+0000 7f85db000700 1 --2- 172.21.15.89:0/3333514400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85d40b6990 0x7f85d40b6e10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:34.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.754+0000 7f85db000700 1 --2- 172.21.15.89:0/3333514400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85d40d05e0 0x7f85d40d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:34.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.754+0000 7f85db000700 1 -- 172.21.15.89:0/3333514400 >> 172.21.15.89:0/3333514400 conn(0x7f85d40666a0 msgr2=0x7f85d4066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:34.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.754+0000 7f85db000700 1 -- 172.21.15.89:0/3333514400 shutdown_connections 2024-04-02T07:52:34.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.754+0000 7f85db000700 1 -- 172.21.15.89:0/3333514400 wait complete. 2024-04-02T07:52:34.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.757+0000 7f85db000700 1 Processor -- start 2024-04-02T07:52:34.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.757+0000 7f85db000700 1 -- start start 2024-04-02T07:52:34.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.758+0000 7f85db000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85d40b6990 0x7f85d415c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:34.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.758+0000 7f85db000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85d40d05e0 0x7f85d4148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:34.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.758+0000 7f85db000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85d415da50 0x7f85d41490b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:34.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.758+0000 7f85db000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d40cc350 con 0x7f85d415da50 2024-04-02T07:52:34.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.758+0000 7f85db000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d40cba20 con 0x7f85d40b6990 2024-04-02T07:52:34.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.758+0000 7f85db000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d41495f0 con 0x7f85d40d05e0 2024-04-02T07:52:34.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.758+0000 7f85d9000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85d415da50 0x7f85d41490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:34.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.758+0000 7f85d9000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85d415da50 0x7f85d41490b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54546/0 (socket says 172.21.15.89:54546) 2024-04-02T07:52:34.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.758+0000 7f85d3e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85d40b6990 0x7f85d415c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:34.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.758+0000 7f85d3400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85d40d05e0 0x7f85d4148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:34.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.758+0000 7f85d9000700 1 -- 172.21.15.89:0/4122530894 learned_addr learned my addr 172.21.15.89:0/4122530894 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:34.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.758+0000 7f85d9000700 1 -- 172.21.15.89:0/4122530894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85d40d05e0 msgr2=0x7f85d4148b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:34.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.759+0000 7f85d9000700 1 --2- 172.21.15.89:0/4122530894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85d40d05e0 0x7f85d4148b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:34.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.759+0000 7f85d9000700 1 -- 172.21.15.89:0/4122530894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85d40b6990 msgr2=0x7f85d415c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:34.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.759+0000 7f85d9000700 1 --2- 172.21.15.89:0/4122530894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85d40b6990 0x7f85d415c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:34.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.759+0000 7f85d9000700 1 -- 172.21.15.89:0/4122530894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c00094a0 con 0x7f85d415da50 2024-04-02T07:52:34.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.759+0000 7f85d3400700 1 --2- 172.21.15.89:0/4122530894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85d40d05e0 0x7f85d4148b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:34.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.759+0000 7f85d3e00700 1 --2- 172.21.15.89:0/4122530894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85d40b6990 0x7f85d415c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:34.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.759+0000 7f85d9000700 1 --2- 172.21.15.89:0/4122530894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85d415da50 0x7f85d41490b0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00b680 tx=0x7f85cc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:34.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.759+0000 7f85cbe00700 1 -- 172.21.15.89:0/4122530894 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc004830 con 0x7f85d415da50 2024-04-02T07:52:34.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.759+0000 7f85cbe00700 1 -- 172.21.15.89:0/4122530894 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85cc0099b0 con 0x7f85d415da50 2024-04-02T07:52:34.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.759+0000 7f85db000700 1 -- 172.21.15.89:0/4122530894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85cc009b30 con 0x7f85d415da50 2024-04-02T07:52:34.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.759+0000 7f85cbe00700 1 -- 172.21.15.89:0/4122530894 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc01c680 con 0x7f85d415da50 2024-04-02T07:52:34.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.759+0000 7f85db000700 1 -- 172.21.15.89:0/4122530894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d40cbd40 con 0x7f85d415da50 2024-04-02T07:52:34.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.761+0000 7f85db000700 1 -- 172.21.15.89:0/4122530894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d41499c0 con 0x7f85d415da50 2024-04-02T07:52:34.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.761+0000 7f85cbe00700 1 -- 172.21.15.89:0/4122530894 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 48) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f85cc01c7e0 con 0x7f85d415da50 2024-04-02T07:52:34.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.763+0000 7f85cbe00700 1 --2- 172.21.15.89:0/4122530894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f85b406adb0 0x7f85b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:34.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.763+0000 7f85d3e00700 1 -- 172.21.15.89:0/4122530894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f85b406adb0 msgr2=0x7f85b406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:34.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.763+0000 7f85cbe00700 1 -- 172.21.15.89:0/4122530894 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f85cc08b330 con 0x7f85d415da50 2024-04-02T07:52:34.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.763+0000 7f85d3e00700 1 --2- 172.21.15.89:0/4122530894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f85b406adb0 0x7f85b406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:34.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.767+0000 7f85cbe00700 1 -- 172.21.15.89:0/4122530894 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f85cc0560c0 con 0x7f85d415da50 2024-04-02T07:52:34.836 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:34.835+0000 7fcb1564a240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:52:34.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.845+0000 7f85cbe00700 1 -- 172.21.15.89:0/4122530894 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 49) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f85cc089590 con 0x7f85d415da50 2024-04-02T07:52:34.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.845+0000 7f85cbe00700 1 -- 172.21.15.89:0/4122530894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f85b406adb0 msgr2=0x7f85b406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:34.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:34.845+0000 7f85cbe00700 1 --2- 172.21.15.89:0/4122530894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f85b406adb0 0x7f85b406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:34.915 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:34.913+0000 7fca84c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:34.915 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:34.913+0000 7fca84c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:34.915 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:34.913+0000 7fca84c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:34.915 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:34.913+0000 7fca84c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:34.915 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:34.913+0000 7fca84c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:34.916 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:34.914+0000 7fca88800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:34.916 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:34.914+0000 7fca88800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:34.916 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:34.914+0000 7fca88800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:34.916 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:34.914+0000 7fca88800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:34.916 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:34.914+0000 7fca88800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:52:35.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.041+0000 7f85db000700 1 -- 172.21.15.89:0/4122530894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85d40b6560 con 0x7f85d415da50 2024-04-02T07:52:35.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.042+0000 7f85cbe00700 1 -- 172.21.15.89:0/4122530894 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f85cc05d610 con 0x7f85d415da50 2024-04-02T07:52:35.045 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:35.045 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6556,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3307197948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3307197948},{"type":"v1","addr":"172.21.15.113:6835","nonce":3307197948}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1797655686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1797655686},{"type":"v1","addr":"172.21.15.113:6834","nonce":1797655686}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4203087300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4203087300},{"type":"v1","addr":"172.21.15.89:6836","nonce":4203087300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:52:09.066449+0000","modified":"2024-04-02T07:52:12.080422+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6918},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6918":{"gid":6918,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3228984565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3228984565},{"type":"v1","addr":"172.21.15.89:6837","nonce":3228984565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-02T07:52:35.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.044+0000 7f85db000700 1 -- 172.21.15.89:0/4122530894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85d415da50 msgr2=0x7f85d41490b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:35.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.044+0000 7f85db000700 1 --2- 172.21.15.89:0/4122530894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85d415da50 0x7f85d41490b0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00b680 tx=0x7f85cc004ad0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.045+0000 7f85db000700 1 -- 172.21.15.89:0/4122530894 shutdown_connections 2024-04-02T07:52:35.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.045+0000 7f85db000700 1 --2- 172.21.15.89:0/4122530894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f85b406adb0 0x7f85b406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.045+0000 7f85db000700 1 --2- 172.21.15.89:0/4122530894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85d40b6990 0x7f85d415c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.045+0000 7f85db000700 1 --2- 172.21.15.89:0/4122530894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85d415da50 0x7f85d41490b0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.045+0000 7f85db000700 1 --2- 172.21.15.89:0/4122530894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85d40d05e0 0x7f85d4148b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.045+0000 7f85db000700 1 -- 172.21.15.89:0/4122530894 >> 172.21.15.89:0/4122530894 conn(0x7f85d40666a0 msgr2=0x7f85d415a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:35.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.045+0000 7f85db000700 1 -- 172.21.15.89:0/4122530894 shutdown_connections 2024-04-02T07:52:35.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.045+0000 7f85db000700 1 -- 172.21.15.89:0/4122530894 wait complete. 2024-04-02T07:52:35.047 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 41 2024-04-02T07:52:35.064 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:52:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.255+0000 7f4855400700 1 Processor -- start 2024-04-02T07:52:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.256+0000 7f4855400700 1 -- start start 2024-04-02T07:52:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.257+0000 7f4855400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4850160f10 0x7f48500d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.257+0000 7f4855400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4850161590 0x7f4850161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.257+0000 7f4855400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48500ceda0 con 0x7f48500cf230 2024-04-02T07:52:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.257+0000 7f4855400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4850161f80 con 0x7f4850161590 2024-04-02T07:52:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.257+0000 7f4855400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48501612f0 con 0x7f4850160f10 2024-04-02T07:52:35.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.257+0000 7f484f200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4850161590 0x7f4850161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:35.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.257+0000 7f484e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4850160f10 0x7f48500d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:35.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.257+0000 7f484f200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4850161590 0x7f4850161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:48748/0 (socket says 172.21.15.89:48748) 2024-04-02T07:52:35.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.257+0000 7f484f200700 1 -- 172.21.15.89:0/2020211596 learned_addr learned my addr 172.21.15.89:0/2020211596 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.258+0000 7f484f200700 1 -- 172.21.15.89:0/2020211596 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4850160f10 msgr2=0x7f48500d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.258+0000 7f484f200700 1 --2- 172.21.15.89:0/2020211596 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4850160f10 0x7f48500d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.258+0000 7f484f200700 1 -- 172.21.15.89:0/2020211596 >> v1:172.21.15.113:6789/0 conn(0x7f48500cf230 legacy=0x7f48500d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.258+0000 7f484f200700 1 -- 172.21.15.89:0/2020211596 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48500ce470 con 0x7f4850161590 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.258+0000 7f484f200700 1 --2- 172.21.15.89:0/2020211596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4850161590 0x7f4850161a40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4848009e80 tx=0x7f484800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7799226bef20442 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.258+0000 7f484d400700 1 -- 172.21.15.89:0/2020211596 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484800e070 con 0x7f4850161590 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.258+0000 7f484d400700 1 -- 172.21.15.89:0/2020211596 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484800b8d0 con 0x7f4850161590 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.258+0000 7f484d400700 1 -- 172.21.15.89:0/2020211596 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484800bc80 con 0x7f4850161590 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.259+0000 7f4855400700 1 -- 172.21.15.89:0/2020211596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4850161590 msgr2=0x7f4850161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.259+0000 7f4855400700 1 --2- 172.21.15.89:0/2020211596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4850161590 0x7f4850161a40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4848009e80 tx=0x7f484800b330 comp rx=0 tx=0).stop 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.259+0000 7f4855400700 1 -- 172.21.15.89:0/2020211596 shutdown_connections 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.259+0000 7f4855400700 1 --2- 172.21.15.89:0/2020211596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4850161590 0x7f4850161a40 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.259+0000 7f4855400700 1 --2- 172.21.15.89:0/2020211596 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4850160f10 0x7f48500d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.259+0000 7f4855400700 1 -- 172.21.15.89:0/2020211596 >> 172.21.15.89:0/2020211596 conn(0x7f485008aff0 msgr2=0x7f48500d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:35.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.259+0000 7f4855400700 1 -- 172.21.15.89:0/2020211596 shutdown_connections 2024-04-02T07:52:35.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.259+0000 7f4855400700 1 -- 172.21.15.89:0/2020211596 wait complete. 2024-04-02T07:52:35.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.261+0000 7f4855400700 1 Processor -- start 2024-04-02T07:52:35.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.261+0000 7f4855400700 1 -- start start 2024-04-02T07:52:35.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f4855400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f48500cf230 0x7f48500b7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:35.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f4855400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4850160f10 0x7f485015f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:35.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f4855400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4850161590 0x7f48500b8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:35.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f4855400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48500b6680 con 0x7f4850160f10 2024-04-02T07:52:35.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f4855400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48500b6960 con 0x7f4850161590 2024-04-02T07:52:35.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f4855400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48500b6c40 con 0x7f48500cf230 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f484de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4850160f10 0x7f485015f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f484de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4850160f10 0x7f485015f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54644/0 (socket says 172.21.15.89:54644) 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f484de00700 1 -- 172.21.15.89:0/830027473 learned_addr learned my addr 172.21.15.89:0/830027473 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f484f200700 1 --2- 172.21.15.89:0/830027473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4850161590 0x7f48500b8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f484e800700 1 --2- 172.21.15.89:0/830027473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f48500cf230 0x7f48500b7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f484de00700 1 -- 172.21.15.89:0/830027473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f48500cf230 msgr2=0x7f48500b7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f484de00700 1 --2- 172.21.15.89:0/830027473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f48500cf230 0x7f48500b7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f484de00700 1 -- 172.21.15.89:0/830027473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4850161590 msgr2=0x7f48500b8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f484de00700 1 --2- 172.21.15.89:0/830027473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4850161590 0x7f48500b8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.262+0000 7f484de00700 1 -- 172.21.15.89:0/830027473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4848009b30 con 0x7f4850160f10 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.263+0000 7f484de00700 1 --2- 172.21.15.89:0/830027473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4850160f10 0x7f485015f5e0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f483c00c420 tx=0x7f483c00c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.263+0000 7f4846a00700 1 -- 172.21.15.89:0/830027473 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c0052d0 con 0x7f4850160f10 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.263+0000 7f4846a00700 1 -- 172.21.15.89:0/830027473 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483c005430 con 0x7f4850160f10 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.263+0000 7f4855400700 1 -- 172.21.15.89:0/830027473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48500b6f20 con 0x7f4850160f10 2024-04-02T07:52:35.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.263+0000 7f4846a00700 1 -- 172.21.15.89:0/830027473 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c0038c0 con 0x7f4850160f10 2024-04-02T07:52:35.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.263+0000 7f4855400700 1 -- 172.21.15.89:0/830027473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4850158bf0 con 0x7f4850160f10 2024-04-02T07:52:35.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.264+0000 7f4855400700 1 -- 172.21.15.89:0/830027473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4850067720 con 0x7f4850160f10 2024-04-02T07:52:35.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.265+0000 7f4846a00700 1 -- 172.21.15.89:0/830027473 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 49) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f483c003a20 con 0x7f4850160f10 2024-04-02T07:52:35.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.266+0000 7f4846a00700 1 -- 172.21.15.89:0/830027473 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f483c088530 con 0x7f4850160f10 2024-04-02T07:52:35.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.270+0000 7f4846a00700 1 -- 172.21.15.89:0/830027473 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f483c0536b0 con 0x7f4850160f10 2024-04-02T07:52:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.471+0000 7f4855400700 1 -- 172.21.15.89:0/830027473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48500ccc30 con 0x7f4850160f10 2024-04-02T07:52:35.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.473+0000 7f4846a00700 1 -- 172.21.15.89:0/830027473 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+17428 (secure 0 0 0) 0x7f483c056cd0 con 0x7f4850160f10 2024-04-02T07:52:35.475 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:35.476 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":99,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:52:34.839369+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:52:06.991300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-02T07:52:08.054066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/3995871086":"2024-04-03T07:52:34.839335+0000","172.21.15.89:0/1754750797":"2024-04-03T07:52:34.839335+0000","172.21.15.89:0/176013638":"2024-04-03T07:52:34.839335+0000","172.21.15.89:0/262403275":"2024-04-03T07:52:34.839335+0000","172.21.15.89:0/1957897648":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/2341355349":"2024-04-03T07:52:34.839335+0000","172.21.15.89:0/1991189445":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/4248118690":"2024-04-03T07:52:25.709764+0000","172.21.15.89:6833/43357":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/3800047871":"2024-04-03T07:52:34.839335+0000","172.21.15.89:6832/43357":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/3231846713":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/3820568899":"2024-04-03T07:52:25.709764+0000","172.21.15.89:0/3513660462":"2024-04-03T07:52:25.709764+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:52:35.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.475+0000 7f4855400700 1 -- 172.21.15.89:0/830027473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4850160f10 msgr2=0x7f485015f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:35.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.475+0000 7f4855400700 1 --2- 172.21.15.89:0/830027473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4850160f10 0x7f485015f5e0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f483c00c420 tx=0x7f483c00c730 comp rx=0 tx=0).stop 2024-04-02T07:52:35.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.475+0000 7f4855400700 1 -- 172.21.15.89:0/830027473 shutdown_connections 2024-04-02T07:52:35.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.475+0000 7f4855400700 1 --2- 172.21.15.89:0/830027473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4850161590 0x7f48500b8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.475+0000 7f4855400700 1 --2- 172.21.15.89:0/830027473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f48500cf230 0x7f48500b7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.475+0000 7f4855400700 1 --2- 172.21.15.89:0/830027473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4850160f10 0x7f485015f5e0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.475+0000 7f4855400700 1 -- 172.21.15.89:0/830027473 >> 172.21.15.89:0/830027473 conn(0x7f485008aff0 msgr2=0x7f48500b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:35.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.475+0000 7f4855400700 1 -- 172.21.15.89:0/830027473 shutdown_connections 2024-04-02T07:52:35.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.475+0000 7f4855400700 1 -- 172.21.15.89:0/830027473 wait complete. 2024-04-02T07:52:35.496 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T07:52:35.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.687+0000 7fcb35e00700 1 Processor -- start 2024-04-02T07:52:35.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.687+0000 7fcb35e00700 1 -- start start 2024-04-02T07:52:35.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.688+0000 7fcb35e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcb3015dfe0 0x7fcb300ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:35.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.688+0000 7fcb35e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb300ce670 0x7fcb300cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:35.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.688+0000 7fcb35e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb300cb310 con 0x7fcb3015a5c0 2024-04-02T07:52:35.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.689+0000 7fcb35e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb300ceb00 con 0x7fcb300ce670 2024-04-02T07:52:35.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.689+0000 7fcb35e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb300711a0 con 0x7fcb3015dfe0 2024-04-02T07:52:35.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.689+0000 7fcb2f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb300ce670 0x7fcb300cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:35.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.689+0000 7fcb2f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb300ce670 0x7fcb300cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54646/0 (socket says 172.21.15.89:54646) 2024-04-02T07:52:35.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.689+0000 7fcb2f400700 1 -- 172.21.15.89:0/4182965210 learned_addr learned my addr 172.21.15.89:0/4182965210 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:35.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.689+0000 7fcb2e000700 1 --2- 172.21.15.89:0/4182965210 >> v2:172.21.15.113:3300/0 conn(0x7fcb3015dfe0 0x7fcb300ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:35.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.689+0000 7fcb2f400700 1 -- 172.21.15.89:0/4182965210 >> v2:172.21.15.113:3300/0 conn(0x7fcb3015dfe0 msgr2=0x7fcb300ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:35.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.689+0000 7fcb2f400700 1 --2- 172.21.15.89:0/4182965210 >> v2:172.21.15.113:3300/0 conn(0x7fcb3015dfe0 0x7fcb300ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.689+0000 7fcb2f400700 1 -- 172.21.15.89:0/4182965210 >> v1:172.21.15.113:6789/0 conn(0x7fcb3015a5c0 legacy=0x7fcb300709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:35.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.689+0000 7fcb2f400700 1 -- 172.21.15.89:0/4182965210 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb30071480 con 0x7fcb300ce670 2024-04-02T07:52:35.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.689+0000 7fcb2f400700 1 --2- 172.21.15.89:0/4182965210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb300ce670 0x7fcb300cba80 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fcb300cb2c0 tx=0x7fcb2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da2b83ef55337ea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:35.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.690+0000 7fcb2d600700 1 -- 172.21.15.89:0/4182965210 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2800e070 con 0x7fcb300ce670 2024-04-02T07:52:35.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.690+0000 7fcb2d600700 1 -- 172.21.15.89:0/4182965210 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2800b910 con 0x7fcb300ce670 2024-04-02T07:52:35.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.690+0000 7fcb2d600700 1 -- 172.21.15.89:0/4182965210 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2800bcc0 con 0x7fcb300ce670 2024-04-02T07:52:35.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.690+0000 7fcb35e00700 1 -- 172.21.15.89:0/4182965210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb300ce670 msgr2=0x7fcb300cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:35.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.690+0000 7fcb35e00700 1 --2- 172.21.15.89:0/4182965210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb300ce670 0x7fcb300cba80 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fcb300cb2c0 tx=0x7fcb2800b330 comp rx=0 tx=0).stop 2024-04-02T07:52:35.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.690+0000 7fcb35e00700 1 -- 172.21.15.89:0/4182965210 shutdown_connections 2024-04-02T07:52:35.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.690+0000 7fcb35e00700 1 --2- 172.21.15.89:0/4182965210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb300ce670 0x7fcb300cba80 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.690+0000 7fcb35e00700 1 --2- 172.21.15.89:0/4182965210 >> v2:172.21.15.113:3300/0 conn(0x7fcb3015dfe0 0x7fcb300ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.690+0000 7fcb35e00700 1 -- 172.21.15.89:0/4182965210 >> 172.21.15.89:0/4182965210 conn(0x7fcb300b0d70 msgr2=0x7fcb300b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:35.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.690+0000 7fcb35e00700 1 -- 172.21.15.89:0/4182965210 shutdown_connections 2024-04-02T07:52:35.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.690+0000 7fcb35e00700 1 -- 172.21.15.89:0/4182965210 wait complete. 2024-04-02T07:52:35.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.692+0000 7fcb35e00700 1 Processor -- start 2024-04-02T07:52:35.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.692+0000 7fcb35e00700 1 -- start start 2024-04-02T07:52:35.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.693+0000 7fcb35e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb300ce670 0x7fcb3015bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:35.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.693+0000 7fcb35e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3015a5c0 0x7fcb3013f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:35.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.693+0000 7fcb35e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb3015dfe0 0x7fcb3015d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:35.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.693+0000 7fcb35e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3013f8e0 con 0x7fcb300ce670 2024-04-02T07:52:35.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.693+0000 7fcb35e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3015ae80 con 0x7fcb3015dfe0 2024-04-02T07:52:35.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.693+0000 7fcb35e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3015b160 con 0x7fcb3015a5c0 2024-04-02T07:52:35.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.693+0000 7fcb2ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb300ce670 0x7fcb3015bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:35.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.694+0000 7fcb2ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb300ce670 0x7fcb3015bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54656/0 (socket says 172.21.15.89:54656) 2024-04-02T07:52:35.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.694+0000 7fcb2ea00700 1 -- 172.21.15.89:0/1126596913 learned_addr learned my addr 172.21.15.89:0/1126596913 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:35.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.694+0000 7fcb2e000700 1 --2- 172.21.15.89:0/1126596913 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3015a5c0 0x7fcb3013f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:35.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.694+0000 7fcb2f400700 1 --2- 172.21.15.89:0/1126596913 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb3015dfe0 0x7fcb3015d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:35.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.694+0000 7fcb2ea00700 1 -- 172.21.15.89:0/1126596913 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3015a5c0 msgr2=0x7fcb3013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:35.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.694+0000 7fcb2ea00700 1 --2- 172.21.15.89:0/1126596913 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3015a5c0 0x7fcb3013f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.694+0000 7fcb2ea00700 1 -- 172.21.15.89:0/1126596913 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb3015dfe0 msgr2=0x7fcb3015d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:35.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.694+0000 7fcb2ea00700 1 --2- 172.21.15.89:0/1126596913 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb3015dfe0 0x7fcb3015d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.694+0000 7fcb2ea00700 1 -- 172.21.15.89:0/1126596913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb28009b30 con 0x7fcb300ce670 2024-04-02T07:52:35.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.694+0000 7fcb2ea00700 1 --2- 172.21.15.89:0/1126596913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb300ce670 0x7fcb3015bf20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fcb2000c4e0 tx=0x7fcb2000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:35.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.694+0000 7fcb27400700 1 -- 172.21.15.89:0/1126596913 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb200052d0 con 0x7fcb300ce670 2024-04-02T07:52:35.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.695+0000 7fcb27400700 1 -- 172.21.15.89:0/1126596913 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb20005430 con 0x7fcb300ce670 2024-04-02T07:52:35.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.695+0000 7fcb27400700 1 -- 172.21.15.89:0/1126596913 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb200038c0 con 0x7fcb300ce670 2024-04-02T07:52:35.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.695+0000 7fcb35e00700 1 -- 172.21.15.89:0/1126596913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3015b440 con 0x7fcb300ce670 2024-04-02T07:52:35.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.695+0000 7fcb35e00700 1 -- 172.21.15.89:0/1126596913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3015ba00 con 0x7fcb300ce670 2024-04-02T07:52:35.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.696+0000 7fcb35e00700 1 -- 172.21.15.89:0/1126596913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3015bbf0 con 0x7fcb300ce670 2024-04-02T07:52:35.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.697+0000 7fcb27400700 1 -- 172.21.15.89:0/1126596913 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 49) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fcb20003a20 con 0x7fcb300ce670 2024-04-02T07:52:35.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.697+0000 7fcb27400700 1 -- 172.21.15.89:0/1126596913 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fcb200885e0 con 0x7fcb300ce670 2024-04-02T07:52:35.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.702+0000 7fcb27400700 1 -- 172.21.15.89:0/1126596913 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcb20053760 con 0x7fcb300ce670 2024-04-02T07:52:35.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.852+0000 7fcb27400700 1 -- 172.21.15.89:0/1126596913 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 50) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fcb20053210 con 0x7fcb300ce670 2024-04-02T07:52:35.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.852+0000 7fcb27400700 1 --2- 172.21.15.89:0/1126596913 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb10036100 0x7fcb10072620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:35.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.854+0000 7fcb2e000700 1 --2- 172.21.15.89:0/1126596913 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb10036100 0x7fcb10072620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:35.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.857+0000 7fcb2e000700 1 --2- 172.21.15.89:0/1126596913 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb10036100 0x7fcb10072620 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcb3006cd70 tx=0x7fcb1c009210 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:35.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.958+0000 7fcb35e00700 1 -- 172.21.15.89:0/1126596913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcb30067720 con 0x7fcb300ce670 2024-04-02T07:52:35.960 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T07:52:35.959+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T07:52:35.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.974+0000 7fcb27400700 1 -- 172.21.15.89:0/1126596913 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v42) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcb200724b0 con 0x7fcb300ce670 2024-04-02T07:52:35.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.977+0000 7fcb35e00700 1 -- 172.21.15.89:0/1126596913 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb10036100 msgr2=0x7fcb10072620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:35.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.977+0000 7fcb35e00700 1 --2- 172.21.15.89:0/1126596913 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb10036100 0x7fcb10072620 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcb3006cd70 tx=0x7fcb1c009210 comp rx=0 tx=0).stop 2024-04-02T07:52:35.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.977+0000 7fcb35e00700 1 -- 172.21.15.89:0/1126596913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb300ce670 msgr2=0x7fcb3015bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:35.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.977+0000 7fcb35e00700 1 --2- 172.21.15.89:0/1126596913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb300ce670 0x7fcb3015bf20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fcb2000c4e0 tx=0x7fcb2000c7f0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.977+0000 7fcb35e00700 1 -- 172.21.15.89:0/1126596913 shutdown_connections 2024-04-02T07:52:35.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.977+0000 7fcb35e00700 1 --2- 172.21.15.89:0/1126596913 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb10036100 0x7fcb10072620 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.977+0000 7fcb35e00700 1 --2- 172.21.15.89:0/1126596913 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb3015dfe0 0x7fcb3015d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.977+0000 7fcb35e00700 1 --2- 172.21.15.89:0/1126596913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb300ce670 0x7fcb3015bf20 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.977+0000 7fcb35e00700 1 --2- 172.21.15.89:0/1126596913 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb3015a5c0 0x7fcb3013f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:35.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.977+0000 7fcb35e00700 1 -- 172.21.15.89:0/1126596913 >> 172.21.15.89:0/1126596913 conn(0x7fcb300b0d70 msgr2=0x7fcb300cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:35.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.978+0000 7fcb35e00700 1 -- 172.21.15.89:0/1126596913 shutdown_connections 2024-04-02T07:52:35.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:35.978+0000 7fcb35e00700 1 -- 172.21.15.89:0/1126596913 wait complete. 2024-04-02T07:52:35.980 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T07:52:35.984 INFO:tasks.ceph.mds.a.smithi089.stderr: -3915> 2024-04-02T07:52:14.559+0000 7fd7eee00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T07:52:35.997 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T07:52:36.038 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T07:52:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.194+0000 7fa3d8a00700 1 Processor -- start 2024-04-02T07:52:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.194+0000 7fa3d8a00700 1 -- start start 2024-04-02T07:52:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.195+0000 7fa3d8a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3d40cd480 0x7fa3d406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:36.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.195+0000 7fa3d8a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d40cdae0 con 0x7fa3d40cbb30 2024-04-02T07:52:36.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.195+0000 7fa3d8a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d40cddc0 con 0x7fa3d40cd020 2024-04-02T07:52:36.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.195+0000 7fa3d8a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40ce0a0 con 0x7fa3d40cd480 2024-04-02T07:52:36.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.196+0000 7fa3d1e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa3d40cbb30 0x7fa3d40cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33164/0 (socket says 172.21.15.89:33164) 2024-04-02T07:52:36.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.196+0000 7fa3d1e00700 1 -- 172.21.15.89:0/1162109328 learned_addr learned my addr 172.21.15.89:0/1162109328 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:36.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.196+0000 7fa3d2800700 1 --2- 172.21.15.89:0/1162109328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3d40cd480 0x7fa3d406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:36.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.196+0000 7fa3d0a00700 1 -- 172.21.15.89:0/1162109328 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244982314 0 0) 0x7fa3d40cdae0 con 0x7fa3d40cbb30 2024-04-02T07:52:36.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.196+0000 7fa3d0a00700 1 -- 172.21.15.89:0/1162109328 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b80035b0 con 0x7fa3d40cbb30 2024-04-02T07:52:36.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.197+0000 7fa3d0a00700 1 -- 172.21.15.89:0/1162109328 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124485403 0 0) 0x7fa3d40cddc0 con 0x7fa3d40cd020 2024-04-02T07:52:36.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.197+0000 7fa3d0a00700 1 -- 172.21.15.89:0/1162109328 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d40cdae0 con 0x7fa3d40cd020 2024-04-02T07:52:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.197+0000 7fa3d0a00700 1 -- 172.21.15.89:0/1162109328 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2525997878 0 0) 0x7fa3b80035b0 con 0x7fa3d40cbb30 2024-04-02T07:52:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.197+0000 7fa3d2800700 1 -- 172.21.15.89:0/1162109328 >> v1:172.21.15.113:6789/0 conn(0x7fa3d40cd020 legacy=0x7fa3d40b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.197+0000 7fa3d2800700 1 -- 172.21.15.89:0/1162109328 >> v1:172.21.15.89:6789/0 conn(0x7fa3d40cbb30 legacy=0x7fa3d40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.197+0000 7fa3d2800700 1 -- 172.21.15.89:0/1162109328 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d406dc20 con 0x7fa3d40cd480 2024-04-02T07:52:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.197+0000 7fa3d2800700 1 --2- 172.21.15.89:0/1162109328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3d40cd480 0x7fa3d406e420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009dc0 tx=0x7fa3cc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1397170e7a796b70 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.197+0000 7fa3d0a00700 1 -- 172.21.15.89:0/1162109328 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc00e070 con 0x7fa3d40cd480 2024-04-02T07:52:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.197+0000 7fa3d0a00700 1 -- 172.21.15.89:0/1162109328 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3cc003a20 con 0x7fa3d40cd480 2024-04-02T07:52:36.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.197+0000 7fa3d0a00700 1 -- 172.21.15.89:0/1162109328 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4001670 con 0x7fa3d40cd480 2024-04-02T07:52:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.198+0000 7fa3d8a00700 1 -- 172.21.15.89:0/1162109328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3d40cd480 msgr2=0x7fa3d406e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.198+0000 7fa3d8a00700 1 --2- 172.21.15.89:0/1162109328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3d40cd480 0x7fa3d406e420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009dc0 tx=0x7fa3cc00b330 comp rx=0 tx=0).stop 2024-04-02T07:52:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.198+0000 7fa3d8a00700 1 -- 172.21.15.89:0/1162109328 shutdown_connections 2024-04-02T07:52:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.198+0000 7fa3d8a00700 1 --2- 172.21.15.89:0/1162109328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3d40cd480 0x7fa3d406e420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.198+0000 7fa3d8a00700 1 -- 172.21.15.89:0/1162109328 >> 172.21.15.89:0/1162109328 conn(0x7fa3d40b0d70 msgr2=0x7fa3d40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.198+0000 7fa3d8a00700 1 -- 172.21.15.89:0/1162109328 shutdown_connections 2024-04-02T07:52:36.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.198+0000 7fa3d8a00700 1 -- 172.21.15.89:0/1162109328 wait complete. 2024-04-02T07:52:36.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.200+0000 7fa3d8a00700 1 Processor -- start 2024-04-02T07:52:36.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.200+0000 7fa3d8a00700 1 -- start start 2024-04-02T07:52:36.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.201+0000 7fa3d8a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d40cbb30 0x7fa3d41579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:36.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.201+0000 7fa3d8a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3d40cd020 0x7fa3d40ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:36.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.201+0000 7fa3d8a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3d40cd480 0x7fa3d415a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:36.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.201+0000 7fa3d8a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d4157f30 con 0x7fa3d40cbb30 2024-04-02T07:52:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.201+0000 7fa3d8a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40ac3a0 con 0x7fa3d40cd480 2024-04-02T07:52:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.201+0000 7fa3d8a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40ac680 con 0x7fa3d40cd020 2024-04-02T07:52:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.201+0000 7fa3d1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d40cbb30 0x7fa3d41579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.201+0000 7fa3d1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d40cbb30 0x7fa3d41579f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54686/0 (socket says 172.21.15.89:54686) 2024-04-02T07:52:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.201+0000 7fa3d1e00700 1 -- 172.21.15.89:0/3646605279 learned_addr learned my addr 172.21.15.89:0/3646605279 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.201+0000 7fa3d1400700 1 --2- 172.21.15.89:0/3646605279 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3d40cd020 0x7fa3d40ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.201+0000 7fa3d2800700 1 --2- 172.21.15.89:0/3646605279 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3d40cd480 0x7fa3d415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.201+0000 7fa3d1e00700 1 -- 172.21.15.89:0/3646605279 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3d40cd020 msgr2=0x7fa3d40ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.201+0000 7fa3d1e00700 1 --2- 172.21.15.89:0/3646605279 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3d40cd020 0x7fa3d40ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.202+0000 7fa3d1e00700 1 -- 172.21.15.89:0/3646605279 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3d40cd480 msgr2=0x7fa3d415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.202+0000 7fa3d1e00700 1 --2- 172.21.15.89:0/3646605279 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3d40cd480 0x7fa3d415a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.202+0000 7fa3d1e00700 1 -- 172.21.15.89:0/3646605279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc009a70 con 0x7fa3d40cbb30 2024-04-02T07:52:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.202+0000 7fa3d1e00700 1 --2- 172.21.15.89:0/3646605279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d40cbb30 0x7fa3d41579f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400c750 tx=0x7fa3c400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.202+0000 7fa3ca000700 1 -- 172.21.15.89:0/3646605279 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4004c80 con 0x7fa3d40cbb30 2024-04-02T07:52:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.202+0000 7fa3ca000700 1 -- 172.21.15.89:0/3646605279 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c4004de0 con 0x7fa3d40cbb30 2024-04-02T07:52:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.202+0000 7fa3d8a00700 1 -- 172.21.15.89:0/3646605279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d40ac960 con 0x7fa3d40cbb30 2024-04-02T07:52:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.202+0000 7fa3ca000700 1 -- 172.21.15.89:0/3646605279 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4007310 con 0x7fa3d40cbb30 2024-04-02T07:52:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.202+0000 7fa3d8a00700 1 -- 172.21.15.89:0/3646605279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d40acf20 con 0x7fa3d40cbb30 2024-04-02T07:52:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.203+0000 7fa3d8a00700 1 -- 172.21.15.89:0/3646605279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d40c83f0 con 0x7fa3d40cbb30 2024-04-02T07:52:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.204+0000 7fa3ca000700 1 -- 172.21.15.89:0/3646605279 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 50) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fa3c4007470 con 0x7fa3d40cbb30 2024-04-02T07:52:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.205+0000 7fa3ca000700 1 --2- 172.21.15.89:0/3646605279 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa3b406aae0 0x7fa3b406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.205+0000 7fa3ca000700 1 -- 172.21.15.89:0/3646605279 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fa3c40891f0 con 0x7fa3d40cbb30 2024-04-02T07:52:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.205+0000 7fa3d1400700 1 --2- 172.21.15.89:0/3646605279 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa3b406aae0 0x7fa3b406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.206+0000 7fa3d1400700 1 --2- 172.21.15.89:0/3646605279 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa3b406aae0 0x7fa3b406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0001220 tx=0x7fa3c0009040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.210+0000 7fa3ca000700 1 -- 172.21.15.89:0/3646605279 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa3c4054230 con 0x7fa3d40cbb30 2024-04-02T07:52:36.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.457+0000 7fa3d8a00700 1 -- 172.21.15.89:0/3646605279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa3d4067720 con 0x7fa3d40cbb30 2024-04-02T07:52:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.979+0000 7fa3ca000700 1 -- 172.21.15.89:0/3646605279 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 51) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fa3c4054600 con 0x7fa3d40cbb30 2024-04-02T07:52:36.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.985+0000 7fa3ca000700 1 -- 172.21.15.89:0/3646605279 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa3c4074500 con 0x7fa3d40cbb30 2024-04-02T07:52:36.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.989+0000 7fa3d8a00700 1 -- 172.21.15.89:0/3646605279 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa3b406aae0 msgr2=0x7fa3b406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:36.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.989+0000 7fa3d8a00700 1 --2- 172.21.15.89:0/3646605279 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa3b406aae0 0x7fa3b406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0001220 tx=0x7fa3c0009040 comp rx=0 tx=0).stop 2024-04-02T07:52:36.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.989+0000 7fa3d8a00700 1 -- 172.21.15.89:0/3646605279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d40cbb30 msgr2=0x7fa3d41579f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:36.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.989+0000 7fa3d8a00700 1 --2- 172.21.15.89:0/3646605279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d40cbb30 0x7fa3d41579f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400c750 tx=0x7fa3c400ca60 comp rx=0 tx=0).stop 2024-04-02T07:52:36.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.989+0000 7fa3d8a00700 1 -- 172.21.15.89:0/3646605279 shutdown_connections 2024-04-02T07:52:36.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.989+0000 7fa3d8a00700 1 --2- 172.21.15.89:0/3646605279 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa3b406aae0 0x7fa3b406cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:36.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.989+0000 7fa3d8a00700 1 --2- 172.21.15.89:0/3646605279 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3d40cd480 0x7fa3d415a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:36.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.989+0000 7fa3d8a00700 1 --2- 172.21.15.89:0/3646605279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3d40cbb30 0x7fa3d41579f0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:36.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.989+0000 7fa3d8a00700 1 --2- 172.21.15.89:0/3646605279 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3d40cd020 0x7fa3d40ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:36.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.989+0000 7fa3d8a00700 1 -- 172.21.15.89:0/3646605279 >> 172.21.15.89:0/3646605279 conn(0x7fa3d40b0d70 msgr2=0x7fa3d40ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:36.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.989+0000 7fa3d8a00700 1 -- 172.21.15.89:0/3646605279 shutdown_connections 2024-04-02T07:52:36.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:36.989+0000 7fa3d8a00700 1 -- 172.21.15.89:0/3646605279 wait complete. 2024-04-02T07:52:37.008 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T07:52:37.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.241+0000 7f43eec00700 1 Processor -- start 2024-04-02T07:52:37.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.241+0000 7f43eec00700 1 -- start start 2024-04-02T07:52:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.243+0000 7f43eec00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f43e80b3930 0x7f43e809a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.243+0000 7f43eec00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e80cbdf0 con 0x7f43e815db10 2024-04-02T07:52:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.243+0000 7f43eec00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e806bec0 con 0x7f43e80cc280 2024-04-02T07:52:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.243+0000 7f43eec00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e80b27a0 con 0x7f43e80b3930 2024-04-02T07:52:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.243+0000 7f43e7e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f43e815db10 0x7f43e8099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33176/0 (socket says 172.21.15.89:33176) 2024-04-02T07:52:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.243+0000 7f43e7e00700 1 -- 172.21.15.89:0/352532285 learned_addr learned my addr 172.21.15.89:0/352532285 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.243+0000 7f43e6a00700 1 -- 172.21.15.89:0/352532285 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644685406 0 0) 0x7f43e80cbdf0 con 0x7f43e815db10 2024-04-02T07:52:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.243+0000 7f43e6a00700 1 -- 172.21.15.89:0/352532285 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43cc0035b0 con 0x7f43e815db10 2024-04-02T07:52:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.243+0000 7f43e7400700 1 --2- 172.21.15.89:0/352532285 >> v2:172.21.15.113:3300/0 conn(0x7f43e80b3930 0x7f43e809a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e6a00700 1 -- 172.21.15.89:0/352532285 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154659145 0 0) 0x7f43cc0035b0 con 0x7f43e815db10 2024-04-02T07:52:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e6a00700 1 -- 172.21.15.89:0/352532285 >> v2:172.21.15.113:3300/0 conn(0x7f43e80b3930 msgr2=0x7f43e809a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:37.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e6a00700 1 --2- 172.21.15.89:0/352532285 >> v2:172.21.15.113:3300/0 conn(0x7f43e80b3930 0x7f43e809a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e6a00700 1 -- 172.21.15.89:0/352532285 >> v1:172.21.15.113:6789/0 conn(0x7f43e80cc280 legacy=0x7f43e80cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e6a00700 1 -- 172.21.15.89:0/352532285 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e806bbe0 con 0x7f43e815db10 2024-04-02T07:52:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e6a00700 1 -- 172.21.15.89:0/352532285 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f43d8001670 con 0x7f43e815db10 2024-04-02T07:52:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e6a00700 1 -- 172.21.15.89:0/352532285 >> v1:172.21.15.89:6789/0 conn(0x7f43e815db10 legacy=0x7f43e8099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e6a00700 1 --2- 172.21.15.89:0/352532285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43cc003a00 0x7f43cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e6a00700 1 --2- 172.21.15.89:0/352532285 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43cc004850 0x7f43cc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e6a00700 1 --2- 172.21.15.89:0/352532285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43cc005210 0x7f43cc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e6a00700 1 -- 172.21.15.89:0/352532285 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e80b27a0 con 0x7f43cc005210 2024-04-02T07:52:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e6a00700 1 -- 172.21.15.89:0/352532285 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc00c1b0 con 0x7f43cc003a00 2024-04-02T07:52:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e6a00700 1 -- 172.21.15.89:0/352532285 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc00c370 con 0x7f43cc004850 2024-04-02T07:52:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43ecc00700 1 --2- 172.21.15.89:0/352532285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43cc005210 0x7f43cc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e7e00700 1 --2- 172.21.15.89:0/352532285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43cc003a00 0x7f43cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:37.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43e7400700 1 --2- 172.21.15.89:0/352532285 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43cc004850 0x7f43cc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:37.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43ecc00700 1 -- 172.21.15.89:0/352532285 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43cc004850 msgr2=0x7f43cc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:37.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43ecc00700 1 --2- 172.21.15.89:0/352532285 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43cc004850 0x7f43cc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:37.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43ecc00700 1 -- 172.21.15.89:0/352532285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43cc003a00 msgr2=0x7f43cc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:37.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.244+0000 7f43ecc00700 1 --2- 172.21.15.89:0/352532285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43cc003a00 0x7f43cc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:37.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.245+0000 7f43ecc00700 1 -- 172.21.15.89:0/352532285 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43cc0035b0 con 0x7f43cc005210 2024-04-02T07:52:37.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.245+0000 7f43ecc00700 1 -- 172.21.15.89:0/352532285 reap_dead start 2024-04-02T07:52:37.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.245+0000 7f43ecc00700 1 --2- 172.21.15.89:0/352532285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43cc005210 0x7f43cc00ba10 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f43e000c190 tx=0x7f43cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=942d0bcce211b3cb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:37.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.245+0000 7f43e6a00700 1 -- 172.21.15.89:0/352532285 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0001db0 con 0x7f43cc005210 2024-04-02T07:52:37.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.245+0000 7f43e6a00700 1 -- 172.21.15.89:0/352532285 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e000ccf0 con 0x7f43cc005210 2024-04-02T07:52:37.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.245+0000 7f43e6a00700 1 -- 172.21.15.89:0/352532285 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0003c40 con 0x7f43cc005210 2024-04-02T07:52:37.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.245+0000 7f43eec00700 1 -- 172.21.15.89:0/352532285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43cc005210 msgr2=0x7f43cc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:37.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.245+0000 7f43eec00700 1 --2- 172.21.15.89:0/352532285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43cc005210 0x7f43cc00ba10 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f43e000c190 tx=0x7f43cc005880 comp rx=0 tx=0).stop 2024-04-02T07:52:37.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.246+0000 7f43eec00700 1 -- 172.21.15.89:0/352532285 shutdown_connections 2024-04-02T07:52:37.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.246+0000 7f43eec00700 1 --2- 172.21.15.89:0/352532285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43cc005210 0x7f43cc00ba10 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:37.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.246+0000 7f43eec00700 1 -- 172.21.15.89:0/352532285 >> 172.21.15.89:0/352532285 conn(0x7f43e80b2e10 msgr2=0x7f43e80b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:37.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.246+0000 7f43eec00700 1 -- 172.21.15.89:0/352532285 shutdown_connections 2024-04-02T07:52:37.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.246+0000 7f43eec00700 1 -- 172.21.15.89:0/352532285 wait complete. 2024-04-02T07:52:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.247+0000 7f43eec00700 1 Processor -- start 2024-04-02T07:52:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.247+0000 7f43eec00700 1 -- start start 2024-04-02T07:52:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.248+0000 7f43eec00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43cc005210 0x7f43e80c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.248+0000 7f43eec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43e8160c00 0x7f43e80c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.248+0000 7f43eec00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43e8158d80 0x7f43e815c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.248+0000 7f43eec00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e80c7490 con 0x7f43cc005210 2024-04-02T07:52:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.248+0000 7f43eec00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e8161080 con 0x7f43e8158d80 2024-04-02T07:52:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.248+0000 7f43eec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e815f6b0 con 0x7f43e8160c00 2024-04-02T07:52:37.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.249+0000 7f43e7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43cc005210 0x7f43e80c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.249+0000 7f43e7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43cc005210 0x7f43e80c78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54706/0 (socket says 172.21.15.89:54706) 2024-04-02T07:52:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.249+0000 7f43e7e00700 1 -- 172.21.15.89:0/457632565 learned_addr learned my addr 172.21.15.89:0/457632565 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.249+0000 7f43ecc00700 1 --2- 172.21.15.89:0/457632565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43e8158d80 0x7f43e815c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.249+0000 7f43e7400700 1 --2- 172.21.15.89:0/457632565 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43e8160c00 0x7f43e80c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.249+0000 7f43e7e00700 1 -- 172.21.15.89:0/457632565 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43e8160c00 msgr2=0x7f43e80c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.249+0000 7f43e7e00700 1 --2- 172.21.15.89:0/457632565 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43e8160c00 0x7f43e80c6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.249+0000 7f43e7e00700 1 -- 172.21.15.89:0/457632565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43e8158d80 msgr2=0x7f43e815c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.249+0000 7f43e7e00700 1 --2- 172.21.15.89:0/457632565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43e8158d80 0x7f43e815c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.249+0000 7f43e7e00700 1 -- 172.21.15.89:0/457632565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e80c6810 con 0x7f43cc005210 2024-04-02T07:52:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.249+0000 7f43e7e00700 1 --2- 172.21.15.89:0/457632565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43cc005210 0x7f43e80c78b0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f43d8007290 tx=0x7f43d80075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.250+0000 7f43dfe00700 1 -- 172.21.15.89:0/457632565 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d8012560 con 0x7f43cc005210 2024-04-02T07:52:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.250+0000 7f43dfe00700 1 -- 172.21.15.89:0/457632565 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d80126c0 con 0x7f43cc005210 2024-04-02T07:52:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.250+0000 7f43dfe00700 1 -- 172.21.15.89:0/457632565 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d8012a50 con 0x7f43cc005210 2024-04-02T07:52:37.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.250+0000 7f43eec00700 1 -- 172.21.15.89:0/457632565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e80c6af0 con 0x7f43cc005210 2024-04-02T07:52:37.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.250+0000 7f43eec00700 1 -- 172.21.15.89:0/457632565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e80c70b0 con 0x7f43cc005210 2024-04-02T07:52:37.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.251+0000 7f43eec00700 1 -- 172.21.15.89:0/457632565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e80c7270 con 0x7f43cc005210 2024-04-02T07:52:37.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.252+0000 7f43dfe00700 1 -- 172.21.15.89:0/457632565 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 51) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f43d8012bb0 con 0x7f43cc005210 2024-04-02T07:52:37.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.253+0000 7f43dfe00700 1 --2- 172.21.15.89:0/457632565 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f43c806ad20 0x7f43c806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:37.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.253+0000 7f43dfe00700 1 -- 172.21.15.89:0/457632565 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f43d8089510 con 0x7f43cc005210 2024-04-02T07:52:37.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.253+0000 7f43e7400700 1 --2- 172.21.15.89:0/457632565 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f43c806ad20 0x7f43c806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:37.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.254+0000 7f43e7400700 1 --2- 172.21.15.89:0/457632565 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f43c806ad20 0x7f43c806d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f43e80c7e10 tx=0x7f43d4009210 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:37.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.258+0000 7f43dfe00700 1 -- 172.21.15.89:0/457632565 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f43d8054380 con 0x7f43cc005210 2024-04-02T07:52:37.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.455+0000 7f43eec00700 1 -- 172.21.15.89:0/457632565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f43e8067720 con 0x7f43cc005210 2024-04-02T07:52:37.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.988+0000 7f43dfe00700 1 -- 172.21.15.89:0/457632565 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v101) v1 ==== 181+0+0 (secure 0 0 0) 0x7f43d80579a0 con 0x7f43cc005210 2024-04-02T07:52:37.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.992+0000 7f43eec00700 1 -- 172.21.15.89:0/457632565 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f43c806ad20 msgr2=0x7f43c806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:37.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.992+0000 7f43eec00700 1 --2- 172.21.15.89:0/457632565 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f43c806ad20 0x7f43c806d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f43e80c7e10 tx=0x7f43d4009210 comp rx=0 tx=0).stop 2024-04-02T07:52:37.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.992+0000 7f43eec00700 1 -- 172.21.15.89:0/457632565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43cc005210 msgr2=0x7f43e80c78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:37.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.992+0000 7f43eec00700 1 --2- 172.21.15.89:0/457632565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43cc005210 0x7f43e80c78b0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f43d8007290 tx=0x7f43d80075a0 comp rx=0 tx=0).stop 2024-04-02T07:52:37.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.992+0000 7f43eec00700 1 -- 172.21.15.89:0/457632565 shutdown_connections 2024-04-02T07:52:37.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.992+0000 7f43eec00700 1 --2- 172.21.15.89:0/457632565 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f43c806ad20 0x7f43c806d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:37.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.992+0000 7f43eec00700 1 --2- 172.21.15.89:0/457632565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43e8158d80 0x7f43e815c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:37.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.992+0000 7f43eec00700 1 --2- 172.21.15.89:0/457632565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43cc005210 0x7f43e80c78b0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:37.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.992+0000 7f43eec00700 1 --2- 172.21.15.89:0/457632565 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43e8160c00 0x7f43e80c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:37.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.992+0000 7f43eec00700 1 -- 172.21.15.89:0/457632565 >> 172.21.15.89:0/457632565 conn(0x7f43e80b2e10 msgr2=0x7f43e806fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:37.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.992+0000 7f43eec00700 1 -- 172.21.15.89:0/457632565 shutdown_connections 2024-04-02T07:52:37.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:37.992+0000 7f43eec00700 1 -- 172.21.15.89:0/457632565 wait complete. 2024-04-02T07:52:37.996 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T07:52:38.015 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T07:52:38.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.214+0000 7f4944200700 1 Processor -- start 2024-04-02T07:52:38.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.215+0000 7f4944200700 1 -- start start 2024-04-02T07:52:38.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.216+0000 7f4944200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f493c15ddf0 0x7f493c0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:38.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.216+0000 7f4944200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f493c15e1d0 0x7f493c0d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:38.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.216+0000 7f4944200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f493c0d2710 0x7f493c1612f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:38.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.216+0000 7f4944200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f493c0d2b80 con 0x7f493c15ddf0 2024-04-02T07:52:38.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.216+0000 7f4944200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f493c15ec60 con 0x7f493c0d2710 2024-04-02T07:52:38.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.216+0000 7f4944200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493c0ce8a0 con 0x7f493c15e1d0 2024-04-02T07:52:38.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.216+0000 7f4941800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f493c15ddf0 0x7f493c0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:38.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.216+0000 7f4941800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f493c15ddf0 0x7f493c0ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54714/0 (socket says 172.21.15.89:54714) 2024-04-02T07:52:38.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.216+0000 7f4941800700 1 -- 172.21.15.89:0/2082416883 learned_addr learned my addr 172.21.15.89:0/2082416883 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:38.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.216+0000 7f4941800700 1 -- 172.21.15.89:0/2082416883 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f493c15e1d0 msgr2=0x7f493c0d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:38.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.216+0000 7f4941800700 1 --2- 172.21.15.89:0/2082416883 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f493c15e1d0 0x7f493c0d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:38.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.216+0000 7f4941800700 1 -- 172.21.15.89:0/2082416883 >> v2:172.21.15.113:3300/0 conn(0x7f493c0d2710 msgr2=0x7f493c1612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:38.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.216+0000 7f4941800700 1 --2- 172.21.15.89:0/2082416883 >> v2:172.21.15.113:3300/0 conn(0x7f493c0d2710 0x7f493c1612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:38.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.217+0000 7f4941800700 1 -- 172.21.15.89:0/2082416883 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c0cdc90 con 0x7f493c15ddf0 2024-04-02T07:52:38.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.217+0000 7f4941800700 1 --2- 172.21.15.89:0/2082416883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f493c15ddf0 0x7f493c0ced30 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f493c0cff40 tx=0x7f4930009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d21fd4b4870f41b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:38.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.217+0000 7f493be00700 1 -- 172.21.15.89:0/2082416883 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493000b870 con 0x7f493c15ddf0 2024-04-02T07:52:38.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.217+0000 7f493be00700 1 -- 172.21.15.89:0/2082416883 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493000b9d0 con 0x7f493c15ddf0 2024-04-02T07:52:38.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.217+0000 7f493be00700 1 -- 172.21.15.89:0/2082416883 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493000bd60 con 0x7f493c15ddf0 2024-04-02T07:52:38.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.218+0000 7f4944200700 1 -- 172.21.15.89:0/2082416883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f493c15ddf0 msgr2=0x7f493c0ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:38.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.218+0000 7f4944200700 1 --2- 172.21.15.89:0/2082416883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f493c15ddf0 0x7f493c0ced30 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f493c0cff40 tx=0x7f4930009bd0 comp rx=0 tx=0).stop 2024-04-02T07:52:38.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.218+0000 7f4944200700 1 -- 172.21.15.89:0/2082416883 shutdown_connections 2024-04-02T07:52:38.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.218+0000 7f4944200700 1 --2- 172.21.15.89:0/2082416883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f493c15ddf0 0x7f493c0ced30 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:38.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.218+0000 7f4944200700 1 --2- 172.21.15.89:0/2082416883 >> v2:172.21.15.113:3300/0 conn(0x7f493c0d2710 0x7f493c1612f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:38.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.218+0000 7f4944200700 1 --2- 172.21.15.89:0/2082416883 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f493c15e1d0 0x7f493c0d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:38.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.218+0000 7f4944200700 1 -- 172.21.15.89:0/2082416883 >> 172.21.15.89:0/2082416883 conn(0x7f493c08aff0 msgr2=0x7f493c0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:38.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.218+0000 7f4944200700 1 -- 172.21.15.89:0/2082416883 shutdown_connections 2024-04-02T07:52:38.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.218+0000 7f4944200700 1 -- 172.21.15.89:0/2082416883 wait complete. 2024-04-02T07:52:38.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.219+0000 7f4944200700 1 Processor -- start 2024-04-02T07:52:38.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.220+0000 7f4944200700 1 -- start start 2024-04-02T07:52:38.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.220+0000 7f4944200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f493c0d2710 0x7f493c0c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:38.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.220+0000 7f4944200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f493c15ddf0 0x7f493c14f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:38.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.220+0000 7f4944200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f493c15e1d0 0x7f493c0b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:38.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.220+0000 7f4944200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c0b4780 con 0x7f493c15ddf0 2024-04-02T07:52:38.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.220+0000 7f4944200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c0b3430 con 0x7f493c15e1d0 2024-04-02T07:52:38.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.220+0000 7f4944200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493c0b40c0 con 0x7f493c0d2710 2024-04-02T07:52:38.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.221+0000 7f4940e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f493c15ddf0 0x7f493c14f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:38.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.221+0000 7f4940e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f493c15ddf0 0x7f493c14f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54726/0 (socket says 172.21.15.89:54726) 2024-04-02T07:52:38.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.221+0000 7f4940e00700 1 -- 172.21.15.89:0/2673869056 learned_addr learned my addr 172.21.15.89:0/2673869056 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:38.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.221+0000 7f4941800700 1 --2- 172.21.15.89:0/2673869056 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f493c0d2710 0x7f493c0c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:38.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.221+0000 7f4942200700 1 --2- 172.21.15.89:0/2673869056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f493c15e1d0 0x7f493c0b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:38.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.221+0000 7f4940e00700 1 -- 172.21.15.89:0/2673869056 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f493c0d2710 msgr2=0x7f493c0c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:38.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.221+0000 7f4940e00700 1 --2- 172.21.15.89:0/2673869056 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f493c0d2710 0x7f493c0c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:38.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.221+0000 7f4940e00700 1 -- 172.21.15.89:0/2673869056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f493c15e1d0 msgr2=0x7f493c0b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:38.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.221+0000 7f4940e00700 1 --2- 172.21.15.89:0/2673869056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f493c15e1d0 0x7f493c0b4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:38.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.221+0000 7f4940e00700 1 -- 172.21.15.89:0/2673869056 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4930009550 con 0x7f493c15ddf0 2024-04-02T07:52:38.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.221+0000 7f4940e00700 1 --2- 172.21.15.89:0/2673869056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f493c15ddf0 0x7f493c14f4d0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f492800b4a0 tx=0x7f492800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:38.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.222+0000 7f4939600700 1 -- 172.21.15.89:0/2673869056 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928011830 con 0x7f493c15ddf0 2024-04-02T07:52:38.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.222+0000 7f4939600700 1 -- 172.21.15.89:0/2673869056 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4928011990 con 0x7f493c15ddf0 2024-04-02T07:52:38.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.222+0000 7f4939600700 1 -- 172.21.15.89:0/2673869056 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928011d40 con 0x7f493c15ddf0 2024-04-02T07:52:38.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.222+0000 7f4944200700 1 -- 172.21.15.89:0/2673869056 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f493c0b43a0 con 0x7f493c15ddf0 2024-04-02T07:52:38.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.222+0000 7f4944200700 1 -- 172.21.15.89:0/2673869056 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f493c0c9b80 con 0x7f493c15ddf0 2024-04-02T07:52:38.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.223+0000 7f4944200700 1 -- 172.21.15.89:0/2673869056 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f493c0b48c0 con 0x7f493c15ddf0 2024-04-02T07:52:38.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.224+0000 7f4939600700 1 -- 172.21.15.89:0/2673869056 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 51) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f492801a400 con 0x7f493c15ddf0 2024-04-02T07:52:38.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.224+0000 7f4939600700 1 --2- 172.21.15.89:0/2673869056 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f491c06ad20 0x7f491c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:38.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.225+0000 7f4939600700 1 -- 172.21.15.89:0/2673869056 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6307+0+0 (secure 0 0 0) 0x7f49280894c0 con 0x7f493c15ddf0 2024-04-02T07:52:38.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.225+0000 7f4941800700 1 --2- 172.21.15.89:0/2673869056 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f491c06ad20 0x7f491c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:38.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.225+0000 7f4941800700 1 --2- 172.21.15.89:0/2673869056 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f491c06ad20 0x7f491c06d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4930000c00 tx=0x7f4930008030 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:38.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.229+0000 7f4939600700 1 -- 172.21.15.89:0/2673869056 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f493c0b48c0 con 0x7f493c15ddf0 2024-04-02T07:52:38.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:38.430+0000 7f4944200700 1 -- 172.21.15.89:0/2673869056 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f493c0c9b80 con 0x7f493c15ddf0 2024-04-02T07:52:39.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.000+0000 7f4939600700 1 -- 172.21.15.89:0/2673869056 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4928054730 con 0x7f493c15ddf0 2024-04-02T07:52:39.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.001+0000 7f4939600700 1 -- 172.21.15.89:0/2673869056 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v102) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4928074f10 con 0x7f493c15ddf0 2024-04-02T07:52:39.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.005+0000 7f4944200700 1 -- 172.21.15.89:0/2673869056 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f491c06ad20 msgr2=0x7f491c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:39.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.005+0000 7f4944200700 1 --2- 172.21.15.89:0/2673869056 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f491c06ad20 0x7f491c06d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4930000c00 tx=0x7f4930008030 comp rx=0 tx=0).stop 2024-04-02T07:52:39.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.005+0000 7f4944200700 1 -- 172.21.15.89:0/2673869056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f493c15ddf0 msgr2=0x7f493c14f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:39.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.005+0000 7f4944200700 1 --2- 172.21.15.89:0/2673869056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f493c15ddf0 0x7f493c14f4d0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f492800b4a0 tx=0x7f492800b860 comp rx=0 tx=0).stop 2024-04-02T07:52:39.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.005+0000 7f4944200700 1 -- 172.21.15.89:0/2673869056 shutdown_connections 2024-04-02T07:52:39.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.005+0000 7f4944200700 1 --2- 172.21.15.89:0/2673869056 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f491c06ad20 0x7f491c06d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.005+0000 7f4944200700 1 --2- 172.21.15.89:0/2673869056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f493c15e1d0 0x7f493c0b4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.005+0000 7f4944200700 1 --2- 172.21.15.89:0/2673869056 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f493c0d2710 0x7f493c0c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.005+0000 7f4944200700 1 --2- 172.21.15.89:0/2673869056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f493c15ddf0 0x7f493c14f4d0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.005+0000 7f4944200700 1 -- 172.21.15.89:0/2673869056 >> 172.21.15.89:0/2673869056 conn(0x7f493c08aff0 msgr2=0x7f493c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:39.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.005+0000 7f4944200700 1 -- 172.21.15.89:0/2673869056 shutdown_connections 2024-04-02T07:52:39.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.005+0000 7f4944200700 1 -- 172.21.15.89:0/2673869056 wait complete. 2024-04-02T07:52:39.008 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T07:52:39.026 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T07:52:39.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.219+0000 7f31dc800700 1 Processor -- start 2024-04-02T07:52:39.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.219+0000 7f31dc800700 1 -- start start 2024-04-02T07:52:39.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.220+0000 7f31dc800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31d40cbb30 0x7f31d40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:39.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.220+0000 7f31dc800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f31d40cd4a0 0x7f31d406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:39.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.220+0000 7f31dc800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d40cc450 con 0x7f31d40cd020 2024-04-02T07:52:39.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.220+0000 7f31dc800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d40cdae0 con 0x7f31d40cd4a0 2024-04-02T07:52:39.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.220+0000 7f31dc800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40cddc0 con 0x7f31d40cbb30 2024-04-02T07:52:39.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.220+0000 7f31da800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f31d40cd4a0 0x7f31d406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:39.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.220+0000 7f31d9400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f31d40cd020 0x7f31d40b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43662/0 (socket says 172.21.15.89:43662) 2024-04-02T07:52:39.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.220+0000 7f31d9400700 1 -- 172.21.15.89:0/3772159604 learned_addr learned my addr 172.21.15.89:0/3772159604 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:39.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.220+0000 7f31d9e00700 1 --2- 172.21.15.89:0/3772159604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31d40cbb30 0x7f31d40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:39.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.220+0000 7f31da800700 1 -- 172.21.15.89:0/3772159604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31d40cbb30 msgr2=0x7f31d40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:39.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.221+0000 7f31da800700 1 --2- 172.21.15.89:0/3772159604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31d40cbb30 0x7f31d40cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.221+0000 7f31da800700 1 -- 172.21.15.89:0/3772159604 >> v1:172.21.15.89:6789/0 conn(0x7f31d40cd020 legacy=0x7f31d40b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:39.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.221+0000 7f31da800700 1 -- 172.21.15.89:0/3772159604 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d40ce0a0 con 0x7f31d40cd4a0 2024-04-02T07:52:39.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.221+0000 7f31d8a00700 1 -- 172.21.15.89:0/3772159604 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771111464 0 0) 0x7f31d40cc450 con 0x7f31d40cd020 2024-04-02T07:52:39.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.221+0000 7f31d9e00700 1 --2- 172.21.15.89:0/3772159604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31d40cbb30 0x7f31d40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:39.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.221+0000 7f31da800700 1 --2- 172.21.15.89:0/3772159604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31d40cd4a0 0x7f31d406e420 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009d90 tx=0x7f31d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164df950591e9b4d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:39.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.221+0000 7f31d8a00700 1 -- 172.21.15.89:0/3772159604 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d000e070 con 0x7f31d40cd4a0 2024-04-02T07:52:39.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.221+0000 7f31d8a00700 1 -- 172.21.15.89:0/3772159604 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d000bbd0 con 0x7f31d40cd4a0 2024-04-02T07:52:39.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.221+0000 7f31d8a00700 1 -- 172.21.15.89:0/3772159604 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d0012430 con 0x7f31d40cd4a0 2024-04-02T07:52:39.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.221+0000 7f31dc800700 1 -- 172.21.15.89:0/3772159604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31d40cd4a0 msgr2=0x7f31d406e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:39.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.221+0000 7f31dc800700 1 --2- 172.21.15.89:0/3772159604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31d40cd4a0 0x7f31d406e420 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009d90 tx=0x7f31d000b330 comp rx=0 tx=0).stop 2024-04-02T07:52:39.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.222+0000 7f31dc800700 1 -- 172.21.15.89:0/3772159604 shutdown_connections 2024-04-02T07:52:39.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.222+0000 7f31dc800700 1 --2- 172.21.15.89:0/3772159604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31d40cd4a0 0x7f31d406e420 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.222+0000 7f31dc800700 1 --2- 172.21.15.89:0/3772159604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31d40cbb30 0x7f31d40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.222+0000 7f31dc800700 1 -- 172.21.15.89:0/3772159604 >> 172.21.15.89:0/3772159604 conn(0x7f31d40b0d70 msgr2=0x7f31d40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:39.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.222+0000 7f31dc800700 1 -- 172.21.15.89:0/3772159604 shutdown_connections 2024-04-02T07:52:39.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.222+0000 7f31dc800700 1 -- 172.21.15.89:0/3772159604 wait complete. 2024-04-02T07:52:39.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.223+0000 7f31dc800700 1 Processor -- start 2024-04-02T07:52:39.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.224+0000 7f31dc800700 1 -- start start 2024-04-02T07:52:39.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.224+0000 7f31dc800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31d40cbb30 0x7f31d414e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:39.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.224+0000 7f31dc800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31d40cd020 0x7f31d40ac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:39.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31dc800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31d40cd4a0 0x7f31d40add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:39.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31dc800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40acf30 con 0x7f31d40cd4a0 2024-04-02T07:52:39.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31dc800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40ad530 con 0x7f31d40cbb30 2024-04-02T07:52:39.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31dc800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40ac560 con 0x7f31d40cd020 2024-04-02T07:52:39.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31da800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31d40cd4a0 0x7f31d40add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:39.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31da800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31d40cd4a0 0x7f31d40add40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40456/0 (socket says 172.21.15.89:40456) 2024-04-02T07:52:39.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31da800700 1 -- 172.21.15.89:0/3521551939 learned_addr learned my addr 172.21.15.89:0/3521551939 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:39.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31d9400700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31d40cd020 0x7f31d40ac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:39.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31d9e00700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31d40cbb30 0x7f31d414e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:39.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31da800700 1 -- 172.21.15.89:0/3521551939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31d40cd020 msgr2=0x7f31d40ac9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:39.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31da800700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31d40cd020 0x7f31d40ac9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31da800700 1 -- 172.21.15.89:0/3521551939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31d40cbb30 msgr2=0x7f31d414e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:39.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31da800700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31d40cbb30 0x7f31d414e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31da800700 1 -- 172.21.15.89:0/3521551939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d0009a70 con 0x7f31d40cd4a0 2024-04-02T07:52:39.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.225+0000 7f31d9e00700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31d40cbb30 0x7f31d414e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:39.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.226+0000 7f31da800700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31d40cd4a0 0x7f31d40add40 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f31d406c130 tx=0x7f31d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:39.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.226+0000 7f31ce000700 1 -- 172.21.15.89:0/3521551939 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d0003ea0 con 0x7f31d40cd4a0 2024-04-02T07:52:39.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.226+0000 7f31ce000700 1 -- 172.21.15.89:0/3521551939 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d0020070 con 0x7f31d40cd4a0 2024-04-02T07:52:39.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.226+0000 7f31ce000700 1 -- 172.21.15.89:0/3521551939 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d0004280 con 0x7f31d40cd4a0 2024-04-02T07:52:39.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.226+0000 7f31dc800700 1 -- 172.21.15.89:0/3521551939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d40ae280 con 0x7f31d40cd4a0 2024-04-02T07:52:39.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.226+0000 7f31dc800700 1 -- 172.21.15.89:0/3521551939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d40ada60 con 0x7f31d40cd4a0 2024-04-02T07:52:39.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.227+0000 7f31dc800700 1 -- 172.21.15.89:0/3521551939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d4067720 con 0x7f31d40cd4a0 2024-04-02T07:52:39.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.228+0000 7f31ce000700 1 -- 172.21.15.89:0/3521551939 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f31d00043e0 con 0x7f31d40cd4a0 2024-04-02T07:52:39.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.229+0000 7f31ce000700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f31b806ade0 0x7f31b806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:39.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.229+0000 7f31ce000700 1 -- 172.21.15.89:0/3521551939 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f31d0022070 con 0x7f31d40cd4a0 2024-04-02T07:52:39.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.229+0000 7f31d9e00700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f31b806ade0 0x7f31b806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:39.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.230+0000 7f31d9e00700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f31b806ade0 0x7f31b806d2a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f31c800aa20 tx=0x7f31c8008040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:39.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.233+0000 7f31ce000700 1 -- 172.21.15.89:0/3521551939 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f31d0056390 con 0x7f31d40cd4a0 2024-04-02T07:52:39.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.440+0000 7f31dc800700 1 -- 172.21.15.89:0/3521551939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f31d415bda0 con 0x7f31d40cd4a0 2024-04-02T07:52:39.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.445+0000 7f31ce000700 1 -- 172.21.15.89:0/3521551939 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v31) v1 ==== 115+0+0 (secure 0 0 0) 0x7f31d00599b0 con 0x7f31d40cd4a0 2024-04-02T07:52:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.447+0000 7f31dc800700 1 -- 172.21.15.89:0/3521551939 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f31b806ade0 msgr2=0x7f31b806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.447+0000 7f31dc800700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f31b806ade0 0x7f31b806d2a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f31c800aa20 tx=0x7f31c8008040 comp rx=0 tx=0).stop 2024-04-02T07:52:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.447+0000 7f31dc800700 1 -- 172.21.15.89:0/3521551939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31d40cd4a0 msgr2=0x7f31d40add40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.447+0000 7f31dc800700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31d40cd4a0 0x7f31d40add40 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f31d406c130 tx=0x7f31d0004ad0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.448+0000 7f31dc800700 1 -- 172.21.15.89:0/3521551939 shutdown_connections 2024-04-02T07:52:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.448+0000 7f31dc800700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f31b806ade0 0x7f31b806d2a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.448+0000 7f31dc800700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31d40cbb30 0x7f31d414e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.448+0000 7f31dc800700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31d40cd4a0 0x7f31d40add40 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.448+0000 7f31dc800700 1 --2- 172.21.15.89:0/3521551939 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31d40cd020 0x7f31d40ac9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.448+0000 7f31dc800700 1 -- 172.21.15.89:0/3521551939 >> 172.21.15.89:0/3521551939 conn(0x7f31d40b0d70 msgr2=0x7f31d406e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.448+0000 7f31dc800700 1 -- 172.21.15.89:0/3521551939 shutdown_connections 2024-04-02T07:52:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.448+0000 7f31dc800700 1 -- 172.21.15.89:0/3521551939 wait complete. 2024-04-02T07:52:39.474 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T07:52:39.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.664+0000 7f49de000700 1 Processor -- start 2024-04-02T07:52:39.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.664+0000 7f49de000700 1 -- start start 2024-04-02T07:52:39.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.665+0000 7f49de000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49d815d9d0 0x7f49d80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:39.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.665+0000 7f49de000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f49d80d2540 0x7f49d80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.665+0000 7f49de000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d8161420 con 0x7f49d80d16a0 2024-04-02T07:52:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.665+0000 7f49de000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d815edc0 con 0x7f49d815d9d0 2024-04-02T07:52:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.665+0000 7f49de000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d8161700 con 0x7f49d80d2540 2024-04-02T07:52:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.665+0000 7f49d7400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49d815d9d0 0x7f49d80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.665+0000 7f49d7400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49d815d9d0 0x7f49d80d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40466/0 (socket says 172.21.15.89:40466) 2024-04-02T07:52:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.665+0000 7f49d7400700 1 -- 172.21.15.89:0/2606208995 learned_addr learned my addr 172.21.15.89:0/2606208995 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.665+0000 7f49d7e00700 1 --2- 172.21.15.89:0/2606208995 >> v2:172.21.15.113:3300/0 conn(0x7f49d80d2540 0x7f49d80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.666+0000 7f49d7400700 1 -- 172.21.15.89:0/2606208995 >> v2:172.21.15.113:3300/0 conn(0x7f49d80d2540 msgr2=0x7f49d80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.666+0000 7f49d7400700 1 --2- 172.21.15.89:0/2606208995 >> v2:172.21.15.113:3300/0 conn(0x7f49d80d2540 0x7f49d80ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.666+0000 7f49d7400700 1 -- 172.21.15.89:0/2606208995 >> v1:172.21.15.113:6789/0 conn(0x7f49d80d16a0 legacy=0x7f49d80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.666+0000 7f49d7400700 1 -- 172.21.15.89:0/2606208995 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d81619e0 con 0x7f49d815d9d0 2024-04-02T07:52:39.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.666+0000 7f49d7400700 1 --2- 172.21.15.89:0/2606208995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49d815d9d0 0x7f49d80d3c30 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f49d808ca10 tx=0x7f49c8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60907483d562a3ce server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:39.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.666+0000 7f49d6000700 1 -- 172.21.15.89:0/2606208995 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c8004230 con 0x7f49d815d9d0 2024-04-02T07:52:39.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.666+0000 7f49d6000700 1 -- 172.21.15.89:0/2606208995 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c8004390 con 0x7f49d815d9d0 2024-04-02T07:52:39.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.666+0000 7f49d6000700 1 -- 172.21.15.89:0/2606208995 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c80036a0 con 0x7f49d815d9d0 2024-04-02T07:52:39.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.667+0000 7f49de000700 1 -- 172.21.15.89:0/2606208995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49d815d9d0 msgr2=0x7f49d80d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:39.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.667+0000 7f49de000700 1 --2- 172.21.15.89:0/2606208995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49d815d9d0 0x7f49d80d3c30 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f49d808ca10 tx=0x7f49c8009bd0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.667+0000 7f49de000700 1 -- 172.21.15.89:0/2606208995 shutdown_connections 2024-04-02T07:52:39.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.667+0000 7f49de000700 1 --2- 172.21.15.89:0/2606208995 >> v2:172.21.15.113:3300/0 conn(0x7f49d80d2540 0x7f49d80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.667+0000 7f49de000700 1 --2- 172.21.15.89:0/2606208995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49d815d9d0 0x7f49d80d3c30 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.667+0000 7f49de000700 1 -- 172.21.15.89:0/2606208995 >> 172.21.15.89:0/2606208995 conn(0x7f49d808aff0 msgr2=0x7f49d80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:39.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.667+0000 7f49de000700 1 -- 172.21.15.89:0/2606208995 shutdown_connections 2024-04-02T07:52:39.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.667+0000 7f49de000700 1 -- 172.21.15.89:0/2606208995 wait complete. 2024-04-02T07:52:39.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.669+0000 7f49de000700 1 Processor -- start 2024-04-02T07:52:39.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.669+0000 7f49de000700 1 -- start start 2024-04-02T07:52:39.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.669+0000 7f49de000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49d80d16a0 0x7f49d81e8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:39.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49de000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49d80d2540 0x7f49d80b76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:39.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49de000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49d815d9d0 0x7f49d81e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:39.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49de000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d80b8d60 con 0x7f49d80d2540 2024-04-02T07:52:39.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49de000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d80b80e0 con 0x7f49d80d16a0 2024-04-02T07:52:39.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49de000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d80b83c0 con 0x7f49d815d9d0 2024-04-02T07:52:39.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49d6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49d80d2540 0x7f49d80b76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:39.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49d6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49d80d2540 0x7f49d80b76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40482/0 (socket says 172.21.15.89:40482) 2024-04-02T07:52:39.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49d6a00700 1 -- 172.21.15.89:0/231397323 learned_addr learned my addr 172.21.15.89:0/231397323 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:39.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49d6a00700 1 -- 172.21.15.89:0/231397323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49d815d9d0 msgr2=0x7f49d81e7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:39.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49d6a00700 1 --2- 172.21.15.89:0/231397323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49d815d9d0 0x7f49d81e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49d6a00700 1 -- 172.21.15.89:0/231397323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49d80d16a0 msgr2=0x7f49d81e8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:39.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49d7400700 1 --2- 172.21.15.89:0/231397323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49d80d16a0 0x7f49d81e8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:39.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49d6a00700 1 --2- 172.21.15.89:0/231397323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49d80d16a0 0x7f49d81e8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49d6a00700 1 -- 172.21.15.89:0/231397323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c8009550 con 0x7f49d80d2540 2024-04-02T07:52:39.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.670+0000 7f49d7400700 1 --2- 172.21.15.89:0/231397323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49d80d16a0 0x7f49d81e8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:39.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.671+0000 7f49d6a00700 1 --2- 172.21.15.89:0/231397323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49d80d2540 0x7f49d80b76b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f49c400c4e0 tx=0x7f49c400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:39.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.671+0000 7f49cf400700 1 -- 172.21.15.89:0/231397323 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c4011790 con 0x7f49d80d2540 2024-04-02T07:52:39.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.671+0000 7f49cf400700 1 -- 172.21.15.89:0/231397323 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c40118f0 con 0x7f49d80d2540 2024-04-02T07:52:39.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.671+0000 7f49de000700 1 -- 172.21.15.89:0/231397323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d80b86a0 con 0x7f49d80d2540 2024-04-02T07:52:39.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.671+0000 7f49cf400700 1 -- 172.21.15.89:0/231397323 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c4011ca0 con 0x7f49d80d2540 2024-04-02T07:52:39.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.671+0000 7f49de000700 1 -- 172.21.15.89:0/231397323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d8156d60 con 0x7f49d80d2540 2024-04-02T07:52:39.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.672+0000 7f49de000700 1 -- 172.21.15.89:0/231397323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d80d1ca0 con 0x7f49d80d2540 2024-04-02T07:52:39.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.673+0000 7f49cf400700 1 -- 172.21.15.89:0/231397323 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f49c401c460 con 0x7f49d80d2540 2024-04-02T07:52:39.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.673+0000 7f49cf400700 1 --2- 172.21.15.89:0/231397323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f49b806adb0 0x7f49b806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:39.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.674+0000 7f49cf400700 1 -- 172.21.15.89:0/231397323 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f49c4089480 con 0x7f49d80d2540 2024-04-02T07:52:39.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.674+0000 7f49d7400700 1 --2- 172.21.15.89:0/231397323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f49b806adb0 0x7f49b806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:39.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.674+0000 7f49d7400700 1 --2- 172.21.15.89:0/231397323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f49b806adb0 0x7f49b806d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009c10 tx=0x7f49c8008030 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:39.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.678+0000 7f49cf400700 1 -- 172.21.15.89:0/231397323 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f49c40543f0 con 0x7f49d80d2540 2024-04-02T07:52:39.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.881+0000 7f49de000700 1 -- 172.21.15.89:0/231397323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f49d80b8ea0 con 0x7f49d80d2540 2024-04-02T07:52:39.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.886+0000 7f49cf400700 1 -- 172.21.15.89:0/231397323 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v32) v1 ==== 120+0+0 (secure 0 0 0) 0x7f49d80b8ea0 con 0x7f49d80d2540 2024-04-02T07:52:39.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.888+0000 7f49de000700 1 -- 172.21.15.89:0/231397323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f49b806adb0 msgr2=0x7f49b806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:39.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.888+0000 7f49de000700 1 --2- 172.21.15.89:0/231397323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f49b806adb0 0x7f49b806d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009c10 tx=0x7f49c8008030 comp rx=0 tx=0).stop 2024-04-02T07:52:39.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.888+0000 7f49de000700 1 -- 172.21.15.89:0/231397323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49d80d2540 msgr2=0x7f49d80b76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:39.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.888+0000 7f49de000700 1 --2- 172.21.15.89:0/231397323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49d80d2540 0x7f49d80b76b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f49c400c4e0 tx=0x7f49c400c8a0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.888+0000 7f49de000700 1 -- 172.21.15.89:0/231397323 shutdown_connections 2024-04-02T07:52:39.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.888+0000 7f49de000700 1 --2- 172.21.15.89:0/231397323 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f49b806adb0 0x7f49b806d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.888+0000 7f49de000700 1 --2- 172.21.15.89:0/231397323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49d80d16a0 0x7f49d81e8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.888+0000 7f49de000700 1 --2- 172.21.15.89:0/231397323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49d815d9d0 0x7f49d81e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.888+0000 7f49de000700 1 --2- 172.21.15.89:0/231397323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49d80d2540 0x7f49d80b76b0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:39.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.888+0000 7f49de000700 1 -- 172.21.15.89:0/231397323 >> 172.21.15.89:0/231397323 conn(0x7f49d808aff0 msgr2=0x7f49d806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:39.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.889+0000 7f49de000700 1 -- 172.21.15.89:0/231397323 shutdown_connections 2024-04-02T07:52:39.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:39.889+0000 7f49de000700 1 -- 172.21.15.89:0/231397323 wait complete. 2024-04-02T07:52:39.910 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T07:52:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.100+0000 7f6a94000700 1 Processor -- start 2024-04-02T07:52:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.100+0000 7f6a94000700 1 -- start start 2024-04-02T07:52:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.101+0000 7f6a94000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a8c15d9d0 0x7f6a8c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.101+0000 7f6a94000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6a8c0d2540 0x7f6a8c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.101+0000 7f6a94000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a8c161420 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.101+0000 7f6a94000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a8c15edc0 con 0x7f6a8c0d2540 2024-04-02T07:52:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.102+0000 7f6a94000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c161700 con 0x7f6a8c15d9d0 2024-04-02T07:52:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.102+0000 7f6a92000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6a8c0d2540 0x7f6a8c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.102+0000 7f6a90c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6a8c0d16a0 0x7f6a8c0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43678/0 (socket says 172.21.15.89:43678) 2024-04-02T07:52:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.102+0000 7f6a90c00700 1 -- 172.21.15.89:0/2914082174 learned_addr learned my addr 172.21.15.89:0/2914082174 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.102+0000 7f6a91600700 1 --2- 172.21.15.89:0/2914082174 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a8c15d9d0 0x7f6a8c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.102+0000 7f6a92000700 1 -- 172.21.15.89:0/2914082174 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a8c15d9d0 msgr2=0x7f6a8c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.102+0000 7f6a92000700 1 --2- 172.21.15.89:0/2914082174 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a8c15d9d0 0x7f6a8c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.102+0000 7f6a92000700 1 -- 172.21.15.89:0/2914082174 >> v1:172.21.15.89:6789/0 conn(0x7f6a8c0d16a0 legacy=0x7f6a8c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.102+0000 7f6a92000700 1 -- 172.21.15.89:0/2914082174 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8c1619e0 con 0x7f6a8c0d2540 2024-04-02T07:52:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.102+0000 7f6a8be00700 1 -- 172.21.15.89:0/2914082174 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627713002 0 0) 0x7f6a8c161420 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.103+0000 7f6a92000700 1 --2- 172.21.15.89:0/2914082174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a8c0d2540 0x7f6a8c0ce360 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f6a84009dc0 tx=0x7f6a8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a42173b28fee8c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.103+0000 7f6a8be00700 1 -- 172.21.15.89:0/2914082174 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8400e070 con 0x7f6a8c0d2540 2024-04-02T07:52:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.103+0000 7f6a8be00700 1 -- 172.21.15.89:0/2914082174 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8400bb90 con 0x7f6a8c0d2540 2024-04-02T07:52:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.103+0000 7f6a8be00700 1 -- 172.21.15.89:0/2914082174 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a84012430 con 0x7f6a8c0d2540 2024-04-02T07:52:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.103+0000 7f6a94000700 1 -- 172.21.15.89:0/2914082174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a8c0d2540 msgr2=0x7f6a8c0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.103+0000 7f6a94000700 1 --2- 172.21.15.89:0/2914082174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a8c0d2540 0x7f6a8c0ce360 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f6a84009dc0 tx=0x7f6a8400b330 comp rx=0 tx=0).stop 2024-04-02T07:52:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.103+0000 7f6a94000700 1 -- 172.21.15.89:0/2914082174 shutdown_connections 2024-04-02T07:52:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.103+0000 7f6a94000700 1 --2- 172.21.15.89:0/2914082174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a8c0d2540 0x7f6a8c0ce360 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.103+0000 7f6a94000700 1 --2- 172.21.15.89:0/2914082174 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a8c15d9d0 0x7f6a8c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.103+0000 7f6a94000700 1 -- 172.21.15.89:0/2914082174 >> 172.21.15.89:0/2914082174 conn(0x7f6a8c08aff0 msgr2=0x7f6a8c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:40.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.103+0000 7f6a94000700 1 -- 172.21.15.89:0/2914082174 shutdown_connections 2024-04-02T07:52:40.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.103+0000 7f6a94000700 1 -- 172.21.15.89:0/2914082174 wait complete. 2024-04-02T07:52:40.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.105+0000 7f6a94000700 1 Processor -- start 2024-04-02T07:52:40.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.105+0000 7f6a94000700 1 -- start start 2024-04-02T07:52:40.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.106+0000 7f6a94000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a8c0d16a0 0x7f6a8c0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:40.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.106+0000 7f6a94000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a8c0d2540 0x7f6a8c0cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:40.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.106+0000 7f6a94000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a8c15d9d0 0x7f6a8c152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:40.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.106+0000 7f6a94000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c161390 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.106+0000 7f6a94000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c1610b0 con 0x7f6a8c0d2540 2024-04-02T07:52:40.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.106+0000 7f6a94000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c0b7b20 con 0x7f6a8c15d9d0 2024-04-02T07:52:40.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.106+0000 7f6a91600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a8c0d16a0 0x7f6a8c0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:40.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.106+0000 7f6a91600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a8c0d16a0 0x7f6a8c0b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40492/0 (socket says 172.21.15.89:40492) 2024-04-02T07:52:40.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.106+0000 7f6a91600700 1 -- 172.21.15.89:0/1024053864 learned_addr learned my addr 172.21.15.89:0/1024053864 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:40.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.106+0000 7f6a90c00700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a8c0d2540 0x7f6a8c0cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:40.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.106+0000 7f6a92000700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a8c15d9d0 0x7f6a8c152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:40.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.107+0000 7f6a91600700 1 -- 172.21.15.89:0/1024053864 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a8c15d9d0 msgr2=0x7f6a8c152410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:40.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.107+0000 7f6a91600700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a8c15d9d0 0x7f6a8c152410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.107+0000 7f6a91600700 1 -- 172.21.15.89:0/1024053864 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a8c0d2540 msgr2=0x7f6a8c0cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:40.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.107+0000 7f6a91600700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a8c0d2540 0x7f6a8c0cc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.107+0000 7f6a91600700 1 -- 172.21.15.89:0/1024053864 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a84009a70 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.107+0000 7f6a90c00700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a8c0d2540 0x7f6a8c0cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:40.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.107+0000 7f6a91600700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a8c0d16a0 0x7f6a8c0b8600 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f6a8000d800 tx=0x7f6a8000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:40.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.107+0000 7f6a89600700 1 -- 172.21.15.89:0/1024053864 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a800125f0 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.107+0000 7f6a89600700 1 -- 172.21.15.89:0/1024053864 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a80012750 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.107+0000 7f6a94000700 1 -- 172.21.15.89:0/1024053864 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a8c0b7e00 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.107+0000 7f6a89600700 1 -- 172.21.15.89:0/1024053864 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a80012b00 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.107+0000 7f6a94000700 1 -- 172.21.15.89:0/1024053864 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a8c15cf10 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.108+0000 7f6a94000700 1 -- 172.21.15.89:0/1024053864 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8c0d1ca0 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.109+0000 7f6a89600700 1 -- 172.21.15.89:0/1024053864 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6a80012c60 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.111+0000 7f6a89600700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6a6c06ade0 0x7f6a6c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:40.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.111+0000 7f6a89600700 1 -- 172.21.15.89:0/1024053864 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f6a800899e0 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.111+0000 7f6a90c00700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6a6c06ade0 0x7f6a6c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:40.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.111+0000 7f6a90c00700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6a6c06ade0 0x7f6a6c06d2a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c0ca580 tx=0x7f6a78009040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:40.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.115+0000 7f6a89600700 1 -- 172.21.15.89:0/1024053864 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a800548c0 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.320+0000 7f6a94000700 1 -- 172.21.15.89:0/1024053864 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6a8c067720 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.326+0000 7f6a89600700 1 -- 172.21.15.89:0/1024053864 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6a80057ee0 con 0x7f6a8c0d16a0 2024-04-02T07:52:40.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.329+0000 7f6a94000700 1 -- 172.21.15.89:0/1024053864 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6a6c06ade0 msgr2=0x7f6a6c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:40.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.329+0000 7f6a94000700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6a6c06ade0 0x7f6a6c06d2a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c0ca580 tx=0x7f6a78009040 comp rx=0 tx=0).stop 2024-04-02T07:52:40.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.329+0000 7f6a94000700 1 -- 172.21.15.89:0/1024053864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a8c0d16a0 msgr2=0x7f6a8c0b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:40.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.329+0000 7f6a94000700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a8c0d16a0 0x7f6a8c0b8600 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f6a8000d800 tx=0x7f6a8000dbc0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.329+0000 7f6a94000700 1 -- 172.21.15.89:0/1024053864 shutdown_connections 2024-04-02T07:52:40.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.329+0000 7f6a94000700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6a6c06ade0 0x7f6a6c06d2a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.329+0000 7f6a94000700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a8c15d9d0 0x7f6a8c152410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.329+0000 7f6a94000700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a8c0d16a0 0x7f6a8c0b8600 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.329+0000 7f6a94000700 1 --2- 172.21.15.89:0/1024053864 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a8c0d2540 0x7f6a8c0cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.329+0000 7f6a94000700 1 -- 172.21.15.89:0/1024053864 >> 172.21.15.89:0/1024053864 conn(0x7f6a8c08aff0 msgr2=0x7f6a8c15f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:40.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.329+0000 7f6a94000700 1 -- 172.21.15.89:0/1024053864 shutdown_connections 2024-04-02T07:52:40.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.329+0000 7f6a94000700 1 -- 172.21.15.89:0/1024053864 wait complete. 2024-04-02T07:52:40.350 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-04-02T07:52:40.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.539+0000 7fc47d000700 1 Processor -- start 2024-04-02T07:52:40.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.539+0000 7fc47d000700 1 -- start start 2024-04-02T07:52:40.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.540+0000 7fc47d000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc47815d640 0x7fc4780d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:40.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.540+0000 7fc47d000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4780ceec0 0x7fc4781612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.540+0000 7fc47d000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4780d30d0 con 0x7fc478161800 2024-04-02T07:52:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.540+0000 7fc47d000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4780b6840 con 0x7fc4780ceec0 2024-04-02T07:52:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.540+0000 7fc47d000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc478161c20 con 0x7fc47815d640 2024-04-02T07:52:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.540+0000 7fc475400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4780ceec0 0x7fc4781612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.540+0000 7fc476800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc478161800 0x7fc4780cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43680/0 (socket says 172.21.15.89:43680) 2024-04-02T07:52:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.540+0000 7fc476800700 1 -- 172.21.15.89:0/3947101091 learned_addr learned my addr 172.21.15.89:0/3947101091 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.541+0000 7fc475e00700 1 --2- 172.21.15.89:0/3947101091 >> v2:172.21.15.113:3300/0 conn(0x7fc47815d640 0x7fc4780d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.541+0000 7fc475400700 1 -- 172.21.15.89:0/3947101091 >> v2:172.21.15.113:3300/0 conn(0x7fc47815d640 msgr2=0x7fc4780d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.541+0000 7fc475400700 1 --2- 172.21.15.89:0/3947101091 >> v2:172.21.15.113:3300/0 conn(0x7fc47815d640 0x7fc4780d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.541+0000 7fc475400700 1 -- 172.21.15.89:0/3947101091 >> v1:172.21.15.89:6789/0 conn(0x7fc478161800 legacy=0x7fc4780cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.541+0000 7fc475400700 1 -- 172.21.15.89:0/3947101091 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4780d2880 con 0x7fc4780ceec0 2024-04-02T07:52:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.541+0000 7fc474a00700 1 -- 172.21.15.89:0/3947101091 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039501552 0 0) 0x7fc4780d30d0 con 0x7fc478161800 2024-04-02T07:52:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.541+0000 7fc475400700 1 --2- 172.21.15.89:0/3947101091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4780ceec0 0x7fc4781612c0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fc4780b5c20 tx=0x7fc464009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670b9719b7ad6bd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:40.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.541+0000 7fc474a00700 1 -- 172.21.15.89:0/3947101091 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc464011830 con 0x7fc4780ceec0 2024-04-02T07:52:40.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.541+0000 7fc474a00700 1 -- 172.21.15.89:0/3947101091 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc464011990 con 0x7fc4780ceec0 2024-04-02T07:52:40.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.541+0000 7fc474a00700 1 -- 172.21.15.89:0/3947101091 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc464011d20 con 0x7fc4780ceec0 2024-04-02T07:52:40.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.542+0000 7fc47d000700 1 -- 172.21.15.89:0/3947101091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4780ceec0 msgr2=0x7fc4781612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:40.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.542+0000 7fc47d000700 1 --2- 172.21.15.89:0/3947101091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4780ceec0 0x7fc4781612c0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fc4780b5c20 tx=0x7fc464009b20 comp rx=0 tx=0).stop 2024-04-02T07:52:40.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.542+0000 7fc47d000700 1 -- 172.21.15.89:0/3947101091 shutdown_connections 2024-04-02T07:52:40.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.542+0000 7fc47d000700 1 --2- 172.21.15.89:0/3947101091 >> v2:172.21.15.113:3300/0 conn(0x7fc47815d640 0x7fc4780d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.542+0000 7fc47d000700 1 --2- 172.21.15.89:0/3947101091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4780ceec0 0x7fc4781612c0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.542+0000 7fc47d000700 1 -- 172.21.15.89:0/3947101091 >> 172.21.15.89:0/3947101091 conn(0x7fc47808aff0 msgr2=0x7fc4780d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:40.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.542+0000 7fc47d000700 1 -- 172.21.15.89:0/3947101091 shutdown_connections 2024-04-02T07:52:40.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.542+0000 7fc47d000700 1 -- 172.21.15.89:0/3947101091 wait complete. 2024-04-02T07:52:40.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.544+0000 7fc47d000700 1 Processor -- start 2024-04-02T07:52:40.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.544+0000 7fc47d000700 1 -- start start 2024-04-02T07:52:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.545+0000 7fc47d000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4780ceec0 0x7fc4780a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.545+0000 7fc47d000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc47815d640 0x7fc47809fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.545+0000 7fc47d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc478161800 0x7fc4780c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.545+0000 7fc47d000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4780c8b30 con 0x7fc478161800 2024-04-02T07:52:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.545+0000 7fc47d000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4780c7b80 con 0x7fc4780ceec0 2024-04-02T07:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.545+0000 7fc47d000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4780c7e60 con 0x7fc47815d640 2024-04-02T07:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.545+0000 7fc476800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc478161800 0x7fc4780c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.545+0000 7fc476800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc478161800 0x7fc4780c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40510/0 (socket says 172.21.15.89:40510) 2024-04-02T07:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.545+0000 7fc476800700 1 -- 172.21.15.89:0/2769746292 learned_addr learned my addr 172.21.15.89:0/2769746292 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.545+0000 7fc475e00700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4780ceec0 0x7fc4780a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.545+0000 7fc475400700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc47815d640 0x7fc47809fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.546+0000 7fc476800700 1 -- 172.21.15.89:0/2769746292 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc47815d640 msgr2=0x7fc47809fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.546+0000 7fc476800700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc47815d640 0x7fc47809fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.546+0000 7fc476800700 1 -- 172.21.15.89:0/2769746292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4780ceec0 msgr2=0x7fc4780a0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.546+0000 7fc476800700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4780ceec0 0x7fc4780a0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.546+0000 7fc476800700 1 -- 172.21.15.89:0/2769746292 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4640094a0 con 0x7fc478161800 2024-04-02T07:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.546+0000 7fc475400700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc47815d640 0x7fc47809fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.546+0000 7fc476800700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc478161800 0x7fc4780c7080 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc470002230 tx=0x7fc47000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.546+0000 7fc46e000700 1 -- 172.21.15.89:0/2769746292 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47000ac20 con 0x7fc478161800 2024-04-02T07:52:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.546+0000 7fc46e000700 1 -- 172.21.15.89:0/2769746292 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47000ada0 con 0x7fc478161800 2024-04-02T07:52:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.546+0000 7fc46e000700 1 -- 172.21.15.89:0/2769746292 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470007310 con 0x7fc478161800 2024-04-02T07:52:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.546+0000 7fc47d000700 1 -- 172.21.15.89:0/2769746292 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4780c8df0 con 0x7fc478161800 2024-04-02T07:52:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.546+0000 7fc47d000700 1 -- 172.21.15.89:0/2769746292 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4780c86c0 con 0x7fc478161800 2024-04-02T07:52:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.547+0000 7fc47d000700 1 -- 172.21.15.89:0/2769746292 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4780c9310 con 0x7fc478161800 2024-04-02T07:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.548+0000 7fc46e000700 1 -- 172.21.15.89:0/2769746292 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc470007470 con 0x7fc478161800 2024-04-02T07:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.549+0000 7fc46e000700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc45806adb0 0x7fc45806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.549+0000 7fc46e000700 1 -- 172.21.15.89:0/2769746292 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5893+0+0 (secure 0 0 0) 0x7fc4700891d0 con 0x7fc478161800 2024-04-02T07:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.549+0000 7fc475e00700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc45806adb0 0x7fc45806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.550+0000 7fc475e00700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc45806adb0 0x7fc45806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc4780ceb80 tx=0x7fc468008040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:40.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.553+0000 7fc46e000700 1 -- 172.21.15.89:0/2769746292 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc4780c9310 con 0x7fc478161800 2024-04-02T07:52:40.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:40.834+0000 7fc47d000700 1 -- 172.21.15.89:0/2769746292 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fc4780c9310 con 0x7fc478161800 2024-04-02T07:52:41.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.002+0000 7fc46e000700 1 -- 172.21.15.89:0/2769746292 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v627) v1 ==== 166+0+0 (secure 0 0 0) 0x7fc4780c9310 con 0x7fc478161800 2024-04-02T07:52:41.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.004+0000 7fc47d000700 1 -- 172.21.15.89:0/2769746292 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc45806adb0 msgr2=0x7fc45806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:41.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.004+0000 7fc47d000700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc45806adb0 0x7fc45806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc4780ceb80 tx=0x7fc468008040 comp rx=0 tx=0).stop 2024-04-02T07:52:41.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.004+0000 7fc47d000700 1 -- 172.21.15.89:0/2769746292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc478161800 msgr2=0x7fc4780c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:41.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.004+0000 7fc47d000700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc478161800 0x7fc4780c7080 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc470002230 tx=0x7fc47000a060 comp rx=0 tx=0).stop 2024-04-02T07:52:41.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.004+0000 7fc47d000700 1 -- 172.21.15.89:0/2769746292 shutdown_connections 2024-04-02T07:52:41.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.004+0000 7fc47d000700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc45806adb0 0x7fc45806d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:41.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.004+0000 7fc47d000700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4780ceec0 0x7fc4780a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:41.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.004+0000 7fc47d000700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc478161800 0x7fc4780c7080 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:41.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.004+0000 7fc47d000700 1 --2- 172.21.15.89:0/2769746292 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc47815d640 0x7fc47809fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:41.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.004+0000 7fc47d000700 1 -- 172.21.15.89:0/2769746292 >> 172.21.15.89:0/2769746292 conn(0x7fc47808aff0 msgr2=0x7fc4780b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:41.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.004+0000 7fc47d000700 1 -- 172.21.15.89:0/2769746292 shutdown_connections 2024-04-02T07:52:41.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.004+0000 7fc47d000700 1 -- 172.21.15.89:0/2769746292 wait complete. 2024-04-02T07:52:41.025 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2024-04-02T07:52:41.025 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2024-04-02T07:52:41.025 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-02T07:52:41.026 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-04-02T07:52:41.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.215+0000 7f5b1ec00700 1 Processor -- start 2024-04-02T07:52:41.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.216+0000 7f5b1ec00700 1 -- start start 2024-04-02T07:52:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.217+0000 7f5b1ec00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b180d18b0 0x7f5b180cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.217+0000 7f5b1ec00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5b18161590 0x7f5b18161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.217+0000 7f5b1ec00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b180ceda0 con 0x7f5b18160f10 2024-04-02T07:52:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.217+0000 7f5b1ec00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b180d1d30 con 0x7f5b180d18b0 2024-04-02T07:52:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.217+0000 7f5b1ec00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b18161f60 con 0x7f5b18161590 2024-04-02T07:52:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.217+0000 7f5b17e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5b18160f10 0x7f5b180d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43696/0 (socket says 172.21.15.89:43696) 2024-04-02T07:52:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.217+0000 7f5b17e00700 1 -- 172.21.15.89:0/1921807172 learned_addr learned my addr 172.21.15.89:0/1921807172 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.217+0000 7f5b17400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b180d18b0 0x7f5b180cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.217+0000 7f5b16a00700 1 -- 172.21.15.89:0/1921807172 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645912509 0 0) 0x7f5b180ceda0 con 0x7f5b18160f10 2024-04-02T07:52:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.217+0000 7f5b16a00700 1 -- 172.21.15.89:0/1921807172 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5afc0035b0 con 0x7f5b18160f10 2024-04-02T07:52:41.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.217+0000 7f5b1cc00700 1 --2- 172.21.15.89:0/1921807172 >> v2:172.21.15.113:3300/0 conn(0x7f5b18161590 0x7f5b18161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.218+0000 7f5b17400700 1 -- 172.21.15.89:0/1921807172 >> v2:172.21.15.113:3300/0 conn(0x7f5b18161590 msgr2=0x7f5b18161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.218+0000 7f5b17400700 1 --2- 172.21.15.89:0/1921807172 >> v2:172.21.15.113:3300/0 conn(0x7f5b18161590 0x7f5b18161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.218+0000 7f5b17400700 1 -- 172.21.15.89:0/1921807172 >> v1:172.21.15.89:6789/0 conn(0x7f5b18160f10 legacy=0x7f5b180d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.218+0000 7f5b17400700 1 -- 172.21.15.89:0/1921807172 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b181612f0 con 0x7f5b180d18b0 2024-04-02T07:52:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.218+0000 7f5b1cc00700 1 --2- 172.21.15.89:0/1921807172 >> v2:172.21.15.113:3300/0 conn(0x7f5b18161590 0x7f5b18161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.218+0000 7f5b17400700 1 --2- 172.21.15.89:0/1921807172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b180d18b0 0x7f5b180cf230 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f5b180d1030 tx=0x7f5b04009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96d4aeba6f189ad server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.218+0000 7f5b16a00700 1 -- 172.21.15.89:0/1921807172 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04011830 con 0x7f5b180d18b0 2024-04-02T07:52:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.218+0000 7f5b16a00700 1 -- 172.21.15.89:0/1921807172 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b04011990 con 0x7f5b180d18b0 2024-04-02T07:52:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.218+0000 7f5b16a00700 1 -- 172.21.15.89:0/1921807172 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04011d00 con 0x7f5b180d18b0 2024-04-02T07:52:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.219+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1921807172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b180d18b0 msgr2=0x7f5b180cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.219+0000 7f5b1ec00700 1 --2- 172.21.15.89:0/1921807172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b180d18b0 0x7f5b180cf230 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f5b180d1030 tx=0x7f5b04009b20 comp rx=0 tx=0).stop 2024-04-02T07:52:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.219+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1921807172 shutdown_connections 2024-04-02T07:52:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.219+0000 7f5b1ec00700 1 --2- 172.21.15.89:0/1921807172 >> v2:172.21.15.113:3300/0 conn(0x7f5b18161590 0x7f5b18161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:41.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.219+0000 7f5b1ec00700 1 --2- 172.21.15.89:0/1921807172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b180d18b0 0x7f5b180cf230 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.219+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1921807172 >> 172.21.15.89:0/1921807172 conn(0x7f5b1808aff0 msgr2=0x7f5b180d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.219+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1921807172 shutdown_connections 2024-04-02T07:52:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.219+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1921807172 wait complete. 2024-04-02T07:52:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.220+0000 7f5b1ec00700 1 Processor -- start 2024-04-02T07:52:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.221+0000 7f5b1ec00700 1 -- start start 2024-04-02T07:52:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.221+0000 7f5b1ec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b180d18b0 0x7f5b1815b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.221+0000 7f5b1ec00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b18160f10 0x7f5b18153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b1ec00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b18161590 0x7f5b1815fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b1ec00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1815a120 con 0x7f5b18160f10 2024-04-02T07:52:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b1ec00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1815a400 con 0x7f5b18161590 2024-04-02T07:52:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b1ec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1815a6e0 con 0x7f5b180d18b0 2024-04-02T07:52:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b17400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b18160f10 0x7f5b18153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:41.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b17400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b18160f10 0x7f5b18153360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40518/0 (socket says 172.21.15.89:40518) 2024-04-02T07:52:41.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b17400700 1 -- 172.21.15.89:0/1755162296 learned_addr learned my addr 172.21.15.89:0/1755162296 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:41.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b1cc00700 1 --2- 172.21.15.89:0/1755162296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b18161590 0x7f5b1815fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:41.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b17e00700 1 --2- 172.21.15.89:0/1755162296 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b180d18b0 0x7f5b1815b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:41.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b17400700 1 -- 172.21.15.89:0/1755162296 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b180d18b0 msgr2=0x7f5b1815b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:41.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b17400700 1 --2- 172.21.15.89:0/1755162296 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b180d18b0 0x7f5b1815b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:41.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b17400700 1 -- 172.21.15.89:0/1755162296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b18161590 msgr2=0x7f5b1815fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:41.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b17400700 1 --2- 172.21.15.89:0/1755162296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b18161590 0x7f5b1815fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:41.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b17400700 1 -- 172.21.15.89:0/1755162296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b040094a0 con 0x7f5b18160f10 2024-04-02T07:52:41.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.222+0000 7f5b17400700 1 --2- 172.21.15.89:0/1755162296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b18160f10 0x7f5b18153360 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f5b040090b0 tx=0x7f5b04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:41.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.223+0000 7f5b0fe00700 1 -- 172.21.15.89:0/1755162296 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b040036a0 con 0x7f5b18160f10 2024-04-02T07:52:41.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.223+0000 7f5b0fe00700 1 -- 172.21.15.89:0/1755162296 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0401e070 con 0x7f5b18160f10 2024-04-02T07:52:41.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.223+0000 7f5b0fe00700 1 -- 172.21.15.89:0/1755162296 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04003a40 con 0x7f5b18160f10 2024-04-02T07:52:41.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.223+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1755162296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1815a9c0 con 0x7f5b18160f10 2024-04-02T07:52:41.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.223+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1755162296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1815b700 con 0x7f5b18160f10 2024-04-02T07:52:41.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.224+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1755162296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1815aee0 con 0x7f5b18160f10 2024-04-02T07:52:41.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.224+0000 7f5b0fe00700 1 -- 172.21.15.89:0/1755162296 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5b04003ba0 con 0x7f5b18160f10 2024-04-02T07:52:41.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.225+0000 7f5b0fe00700 1 --2- 172.21.15.89:0/1755162296 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5af806adb0 0x7f5af806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:41.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.225+0000 7f5b0fe00700 1 -- 172.21.15.89:0/1755162296 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f5b0408a5e0 con 0x7f5b18160f10 2024-04-02T07:52:41.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.226+0000 7f5b17e00700 1 --2- 172.21.15.89:0/1755162296 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5af806adb0 0x7f5af806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:41.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.226+0000 7f5b17e00700 1 --2- 172.21.15.89:0/1755162296 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5af806adb0 0x7f5af806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5b08005910 tx=0x7f5b08009040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:41.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.230+0000 7f5b0fe00700 1 -- 172.21.15.89:0/1755162296 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5b1815aee0 con 0x7f5b18160f10 2024-04-02T07:52:41.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:41.512+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1755162296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f5b1815aee0 con 0x7f5b18160f10 2024-04-02T07:52:42.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.002+0000 7f5b0fe00700 1 -- 172.21.15.89:0/1755162296 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v628) v1 ==== 186+0+0 (secure 0 0 0) 0x7f5b1815aee0 con 0x7f5b18160f10 2024-04-02T07:52:42.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.004+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1755162296 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5af806adb0 msgr2=0x7f5af806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:42.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.004+0000 7f5b1ec00700 1 --2- 172.21.15.89:0/1755162296 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5af806adb0 0x7f5af806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5b08005910 tx=0x7f5b08009040 comp rx=0 tx=0).stop 2024-04-02T07:52:42.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.004+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1755162296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b18160f10 msgr2=0x7f5b18153360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:42.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.004+0000 7f5b1ec00700 1 --2- 172.21.15.89:0/1755162296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b18160f10 0x7f5b18153360 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f5b040090b0 tx=0x7f5b04004750 comp rx=0 tx=0).stop 2024-04-02T07:52:42.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.004+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1755162296 shutdown_connections 2024-04-02T07:52:42.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.004+0000 7f5b1ec00700 1 --2- 172.21.15.89:0/1755162296 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5af806adb0 0x7f5af806d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:42.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.004+0000 7f5b1ec00700 1 --2- 172.21.15.89:0/1755162296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b18161590 0x7f5b1815fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:42.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.004+0000 7f5b1ec00700 1 --2- 172.21.15.89:0/1755162296 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b180d18b0 0x7f5b1815b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:42.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.004+0000 7f5b1ec00700 1 --2- 172.21.15.89:0/1755162296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b18160f10 0x7f5b18153360 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:42.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.004+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1755162296 >> 172.21.15.89:0/1755162296 conn(0x7f5b1808aff0 msgr2=0x7f5b180cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:42.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.004+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1755162296 shutdown_connections 2024-04-02T07:52:42.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.005+0000 7f5b1ec00700 1 -- 172.21.15.89:0/1755162296 wait complete. 2024-04-02T07:52:42.026 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T07:52:42.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.220+0000 7f4fb4a00700 1 Processor -- start 2024-04-02T07:52:42.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.221+0000 7f4fb4a00700 1 -- start start 2024-04-02T07:52:42.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.222+0000 7f4fb4a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4fb00d21f0 0x7f4fb00ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:42.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.222+0000 7f4fb4a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4fb00cec40 0x7f4fb0161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:42.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.222+0000 7f4fb4a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb00ce270 con 0x7f4fb01611b0 2024-04-02T07:52:42.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.222+0000 7f4fb4a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb015b600 con 0x7f4fb00d21f0 2024-04-02T07:52:42.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.222+0000 7f4fb4a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb00d2760 con 0x7f4fb00cec40 2024-04-02T07:52:42.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.222+0000 7f4fa7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4fb00d21f0 0x7f4fb00ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:42.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.222+0000 7f4fa7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4fb00d21f0 0x7f4fb00ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40526/0 (socket says 172.21.15.89:40526) 2024-04-02T07:52:42.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.222+0000 7f4fa7e00700 1 -- 172.21.15.89:0/562379459 learned_addr learned my addr 172.21.15.89:0/562379459 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:42.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.222+0000 7f4fae800700 1 --2- 172.21.15.89:0/562379459 >> v2:172.21.15.113:3300/0 conn(0x7f4fb00cec40 0x7f4fb0161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:42.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.222+0000 7f4fa7e00700 1 -- 172.21.15.89:0/562379459 >> v2:172.21.15.113:3300/0 conn(0x7f4fb00cec40 msgr2=0x7f4fb0161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:42.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.222+0000 7f4fa7e00700 1 --2- 172.21.15.89:0/562379459 >> v2:172.21.15.113:3300/0 conn(0x7f4fb00cec40 0x7f4fb0161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:42.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.223+0000 7f4fa7e00700 1 -- 172.21.15.89:0/562379459 >> v1:172.21.15.113:6789/0 conn(0x7f4fb01611b0 legacy=0x7f4fb015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:42.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.223+0000 7f4fa7e00700 1 -- 172.21.15.89:0/562379459 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb00d2a40 con 0x7f4fb00d21f0 2024-04-02T07:52:42.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.223+0000 7f4fae800700 1 --2- 172.21.15.89:0/562379459 >> v2:172.21.15.113:3300/0 conn(0x7f4fb00cec40 0x7f4fb0161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:42.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.223+0000 7f4fa7e00700 1 --2- 172.21.15.89:0/562379459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb00d21f0 0x7f4fb00ce700 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00b5cb0 tx=0x7f4f98009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99fe00b77dcc850b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:42.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.223+0000 7f4fad400700 1 -- 172.21.15.89:0/562379459 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9800b870 con 0x7f4fb00d21f0 2024-04-02T07:52:42.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.223+0000 7f4fad400700 1 -- 172.21.15.89:0/562379459 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9800b9d0 con 0x7f4fb00d21f0 2024-04-02T07:52:42.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.223+0000 7f4fad400700 1 -- 172.21.15.89:0/562379459 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9800bd60 con 0x7f4fb00d21f0 2024-04-02T07:52:42.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.224+0000 7f4fb4a00700 1 -- 172.21.15.89:0/562379459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb00d21f0 msgr2=0x7f4fb00ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:42.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.224+0000 7f4fb4a00700 1 --2- 172.21.15.89:0/562379459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb00d21f0 0x7f4fb00ce700 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00b5cb0 tx=0x7f4f98009bd0 comp rx=0 tx=0).stop 2024-04-02T07:52:42.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.224+0000 7f4fb4a00700 1 -- 172.21.15.89:0/562379459 shutdown_connections 2024-04-02T07:52:42.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.224+0000 7f4fb4a00700 1 --2- 172.21.15.89:0/562379459 >> v2:172.21.15.113:3300/0 conn(0x7f4fb00cec40 0x7f4fb0161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:42.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.224+0000 7f4fb4a00700 1 --2- 172.21.15.89:0/562379459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb00d21f0 0x7f4fb00ce700 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:42.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.224+0000 7f4fb4a00700 1 -- 172.21.15.89:0/562379459 >> 172.21.15.89:0/562379459 conn(0x7f4fb008aff0 msgr2=0x7f4fb00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:42.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.224+0000 7f4fb4a00700 1 -- 172.21.15.89:0/562379459 shutdown_connections 2024-04-02T07:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.224+0000 7f4fb4a00700 1 -- 172.21.15.89:0/562379459 wait complete. 2024-04-02T07:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.225+0000 7f4fb4a00700 1 Processor -- start 2024-04-02T07:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.226+0000 7f4fb4a00700 1 -- start start 2024-04-02T07:52:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.226+0000 7f4fb4a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb00cec40 0x7f4fb00c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.226+0000 7f4fb4a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4fb00d21f0 0x7f4fb00cb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fb4a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4fb01611b0 0x7f4fb00cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fb4a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb00cb9a0 con 0x7f4fb00cec40 2024-04-02T07:52:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fb4a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb00ca6a0 con 0x7f4fb01611b0 2024-04-02T07:52:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fb4a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb00ca980 con 0x7f4fb00d21f0 2024-04-02T07:52:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fade00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb00cec40 0x7f4fb00c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fade00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb00cec40 0x7f4fb00c9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40534/0 (socket says 172.21.15.89:40534) 2024-04-02T07:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fade00700 1 -- 172.21.15.89:0/4055180721 learned_addr learned my addr 172.21.15.89:0/4055180721 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fa7e00700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4fb00d21f0 0x7f4fb00cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fae800700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4fb01611b0 0x7f4fb00cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fade00700 1 -- 172.21.15.89:0/4055180721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4fb00d21f0 msgr2=0x7f4fb00cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fade00700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4fb00d21f0 0x7f4fb00cb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fade00700 1 -- 172.21.15.89:0/4055180721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4fb01611b0 msgr2=0x7f4fb00cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fade00700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4fb01611b0 0x7f4fb00cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fade00700 1 -- 172.21.15.89:0/4055180721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f98009550 con 0x7f4fb00cec40 2024-04-02T07:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fae800700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4fb01611b0 0x7f4fb00cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.227+0000 7f4fade00700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb00cec40 0x7f4fb00c9860 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f4fa000c450 tx=0x7f4fa000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.228+0000 7f4fa6000700 1 -- 172.21.15.89:0/4055180721 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa00052d0 con 0x7f4fb00cec40 2024-04-02T07:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.228+0000 7f4fa6000700 1 -- 172.21.15.89:0/4055180721 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa0005430 con 0x7f4fb00cec40 2024-04-02T07:52:42.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.228+0000 7f4fb4a00700 1 -- 172.21.15.89:0/4055180721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb00cac60 con 0x7f4fb00cec40 2024-04-02T07:52:42.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.228+0000 7f4fa6000700 1 -- 172.21.15.89:0/4055180721 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa00038c0 con 0x7f4fb00cec40 2024-04-02T07:52:42.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.228+0000 7f4fb4a00700 1 -- 172.21.15.89:0/4055180721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb00c9f60 con 0x7f4fb00cec40 2024-04-02T07:52:42.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.229+0000 7f4fb4a00700 1 -- 172.21.15.89:0/4055180721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb00cb180 con 0x7f4fb00cec40 2024-04-02T07:52:42.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.230+0000 7f4fa6000700 1 -- 172.21.15.89:0/4055180721 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4fa0003a20 con 0x7f4fb00cec40 2024-04-02T07:52:42.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.231+0000 7f4fa6000700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4f8c06adb0 0x7f4f8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:42.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.231+0000 7f4fa6000700 1 -- 172.21.15.89:0/4055180721 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f4fa0088950 con 0x7f4fb00cec40 2024-04-02T07:52:42.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.231+0000 7f4fa7e00700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4f8c06adb0 0x7f4f8c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:42.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.232+0000 7f4fa7e00700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4f8c06adb0 0x7f4f8c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4f98000c00 tx=0x7f4f98008030 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:42.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.236+0000 7f4fa6000700 1 -- 172.21.15.89:0/4055180721 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4fa0053940 con 0x7f4fb00cec40 2024-04-02T07:52:42.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.439+0000 7f4fb4a00700 1 -- 172.21.15.89:0/4055180721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4fb00ca0f0 con 0x7f4fb00cec40 2024-04-02T07:52:42.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.445+0000 7f4fa6000700 1 -- 172.21.15.89:0/4055180721 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4fa00170c0 con 0x7f4fb00cec40 2024-04-02T07:52:42.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.448+0000 7f4fb4a00700 1 -- 172.21.15.89:0/4055180721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4f8c06adb0 msgr2=0x7f4f8c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:42.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.448+0000 7f4fb4a00700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4f8c06adb0 0x7f4f8c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4f98000c00 tx=0x7f4f98008030 comp rx=0 tx=0).stop 2024-04-02T07:52:42.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.448+0000 7f4fb4a00700 1 -- 172.21.15.89:0/4055180721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb00cec40 msgr2=0x7f4fb00c9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:42.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.448+0000 7f4fb4a00700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb00cec40 0x7f4fb00c9860 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f4fa000c450 tx=0x7f4fa000c760 comp rx=0 tx=0).stop 2024-04-02T07:52:42.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.448+0000 7f4fb4a00700 1 -- 172.21.15.89:0/4055180721 shutdown_connections 2024-04-02T07:52:42.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.448+0000 7f4fb4a00700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4f8c06adb0 0x7f4f8c06d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:42.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.448+0000 7f4fb4a00700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4fb01611b0 0x7f4fb00cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:42.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.448+0000 7f4fb4a00700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4fb00cec40 0x7f4fb00c9860 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:42.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.448+0000 7f4fb4a00700 1 --2- 172.21.15.89:0/4055180721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4fb00d21f0 0x7f4fb00cb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:42.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.448+0000 7f4fb4a00700 1 -- 172.21.15.89:0/4055180721 >> 172.21.15.89:0/4055180721 conn(0x7f4fb008aff0 msgr2=0x7f4fb015ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:42.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.448+0000 7f4fb4a00700 1 -- 172.21.15.89:0/4055180721 shutdown_connections 2024-04-02T07:52:42.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:42.448+0000 7f4fb4a00700 1 -- 172.21.15.89:0/4055180721 wait complete. 2024-04-02T07:52:42.469 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:52:42.469 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:52:42.572 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:52:42.666 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:52:42.666 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:52:42.667 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:42.711 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:52:42.711 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:52:42.711 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:42.757 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:52:42.757 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:52:42.757 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:42.801 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:52:42.801 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:52:42.801 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:42.853 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:52:42.854 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T07:52:42.854 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:52:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.047+0000 7fc44d600700 1 Processor -- start 2024-04-02T07:52:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.047+0000 7fc44d600700 1 -- start start 2024-04-02T07:52:43.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.048+0000 7fc44d600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4480ceec0 0x7fc4481612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:43.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.048+0000 7fc44d600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc448161800 0x7fc4480b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:43.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.048+0000 7fc44d600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4480d30d0 con 0x7fc44815d640 2024-04-02T07:52:43.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.048+0000 7fc44d600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc448161c90 con 0x7fc448161800 2024-04-02T07:52:43.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.048+0000 7fc44d600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4480b6840 con 0x7fc4480ceec0 2024-04-02T07:52:43.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.048+0000 7fc447400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc448161800 0x7fc4480b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:43.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.048+0000 7fc446a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc44815d640 0x7fc4480d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43706/0 (socket says 172.21.15.89:43706) 2024-04-02T07:52:43.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.048+0000 7fc446a00700 1 -- 172.21.15.89:0/3403151385 learned_addr learned my addr 172.21.15.89:0/3403151385 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:43.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.048+0000 7fc446000700 1 --2- 172.21.15.89:0/3403151385 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4480ceec0 0x7fc4481612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:43.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.049+0000 7fc445600700 1 -- 172.21.15.89:0/3403151385 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559616967 0 0) 0x7fc4480d30d0 con 0x7fc44815d640 2024-04-02T07:52:43.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.049+0000 7fc445600700 1 -- 172.21.15.89:0/3403151385 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc42c0035b0 con 0x7fc44815d640 2024-04-02T07:52:43.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.049+0000 7fc447400700 1 -- 172.21.15.89:0/3403151385 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4480ceec0 msgr2=0x7fc4481612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:43.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.049+0000 7fc447400700 1 --2- 172.21.15.89:0/3403151385 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4480ceec0 0x7fc4481612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:43.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.049+0000 7fc447400700 1 -- 172.21.15.89:0/3403151385 >> v1:172.21.15.89:6789/0 conn(0x7fc44815d640 legacy=0x7fc4480d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:43.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.049+0000 7fc447400700 1 -- 172.21.15.89:0/3403151385 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4480d2b30 con 0x7fc448161800 2024-04-02T07:52:43.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.049+0000 7fc446000700 1 --2- 172.21.15.89:0/3403151385 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4480ceec0 0x7fc4481612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:43.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.049+0000 7fc447400700 1 --2- 172.21.15.89:0/3403151385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc448161800 0x7fc4480b6cd0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fc440009d90 tx=0x7fc44000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d84d2b780602a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:43.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.049+0000 7fc445600700 1 -- 172.21.15.89:0/3403151385 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44000e070 con 0x7fc448161800 2024-04-02T07:52:43.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.049+0000 7fc445600700 1 -- 172.21.15.89:0/3403151385 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44000bb20 con 0x7fc448161800 2024-04-02T07:52:43.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.049+0000 7fc445600700 1 -- 172.21.15.89:0/3403151385 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc440012430 con 0x7fc448161800 2024-04-02T07:52:43.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.050+0000 7fc44d600700 1 -- 172.21.15.89:0/3403151385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc448161800 msgr2=0x7fc4480b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:43.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.050+0000 7fc44d600700 1 --2- 172.21.15.89:0/3403151385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc448161800 0x7fc4480b6cd0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fc440009d90 tx=0x7fc44000b330 comp rx=0 tx=0).stop 2024-04-02T07:52:43.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.050+0000 7fc44d600700 1 -- 172.21.15.89:0/3403151385 shutdown_connections 2024-04-02T07:52:43.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.050+0000 7fc44d600700 1 --2- 172.21.15.89:0/3403151385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc448161800 0x7fc4480b6cd0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:43.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.050+0000 7fc44d600700 1 --2- 172.21.15.89:0/3403151385 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4480ceec0 0x7fc4481612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:43.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.050+0000 7fc44d600700 1 -- 172.21.15.89:0/3403151385 >> 172.21.15.89:0/3403151385 conn(0x7fc44808aff0 msgr2=0x7fc4480d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:43.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.050+0000 7fc44d600700 1 -- 172.21.15.89:0/3403151385 shutdown_connections 2024-04-02T07:52:43.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.050+0000 7fc44d600700 1 -- 172.21.15.89:0/3403151385 wait complete. 2024-04-02T07:52:43.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.052+0000 7fc44d600700 1 Processor -- start 2024-04-02T07:52:43.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.052+0000 7fc44d600700 1 -- start start 2024-04-02T07:52:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc44d600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4480ceec0 0x7fc4481f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc44d600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44815d640 0x7fc448155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc44d600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc448161800 0x7fc4481564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc44d600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc448153600 con 0x7fc44815d640 2024-04-02T07:52:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc44d600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4481538a0 con 0x7fc448161800 2024-04-02T07:52:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc44d600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc448153b80 con 0x7fc4480ceec0 2024-04-02T07:52:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc446000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44815d640 0x7fc448155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc446000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44815d640 0x7fc448155f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40554/0 (socket says 172.21.15.89:40554) 2024-04-02T07:52:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc446000700 1 -- 172.21.15.89:0/971009037 learned_addr learned my addr 172.21.15.89:0/971009037 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc447400700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc448161800 0x7fc4481564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc446a00700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4480ceec0 0x7fc4481f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc446000700 1 -- 172.21.15.89:0/971009037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4480ceec0 msgr2=0x7fc4481f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc446000700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4480ceec0 0x7fc4481f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc446000700 1 -- 172.21.15.89:0/971009037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc448161800 msgr2=0x7fc4481564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc446000700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc448161800 0x7fc4481564a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc446000700 1 -- 172.21.15.89:0/971009037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4340094a0 con 0x7fc44815d640 2024-04-02T07:52:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.053+0000 7fc446a00700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4480ceec0 0x7fc4481f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.054+0000 7fc446000700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44815d640 0x7fc448155f60 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc43400ebe0 tx=0x7fc434009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.054+0000 7fc43ea00700 1 -- 172.21.15.89:0/971009037 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43400c8b0 con 0x7fc44815d640 2024-04-02T07:52:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.054+0000 7fc43ea00700 1 -- 172.21.15.89:0/971009037 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43400ca10 con 0x7fc44815d640 2024-04-02T07:52:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.054+0000 7fc43ea00700 1 -- 172.21.15.89:0/971009037 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43400cda0 con 0x7fc44815d640 2024-04-02T07:52:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.054+0000 7fc44d600700 1 -- 172.21.15.89:0/971009037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc440009a70 con 0x7fc44815d640 2024-04-02T07:52:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.054+0000 7fc44d600700 1 -- 172.21.15.89:0/971009037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4481f2ce0 con 0x7fc44815d640 2024-04-02T07:52:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.055+0000 7fc44d600700 1 -- 172.21.15.89:0/971009037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc448067720 con 0x7fc44815d640 2024-04-02T07:52:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.056+0000 7fc43ea00700 1 -- 172.21.15.89:0/971009037 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc434013470 con 0x7fc44815d640 2024-04-02T07:52:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.057+0000 7fc43ea00700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc42806adb0 0x7fc42806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.057+0000 7fc43ea00700 1 -- 172.21.15.89:0/971009037 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5893+0+0 (secure 0 0 0) 0x7fc434089610 con 0x7fc44815d640 2024-04-02T07:52:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.057+0000 7fc446a00700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc42806adb0 0x7fc42806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.058+0000 7fc446a00700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc42806adb0 0x7fc42806d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc4380011e0 tx=0x7fc438009040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:43.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.062+0000 7fc43ea00700 1 -- 172.21.15.89:0/971009037 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc4340546b0 con 0x7fc44815d640 2024-04-02T07:52:43.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.303+0000 7fc44d600700 1 -- 172.21.15.89:0/971009037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4480b5c70 con 0x7fc44815d640 2024-04-02T07:52:43.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.303+0000 7fc43ea00700 1 -- 172.21.15.89:0/971009037 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+3535 (secure 0 0 0) 0x7fc434017de0 con 0x7fc44815d640 2024-04-02T07:52:43.305 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:43.306 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":43,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6556,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3307197948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3307197948},{"type":"v1","addr":"172.21.15.113:6835","nonce":3307197948}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1797655686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1797655686},{"type":"v1","addr":"172.21.15.113:6834","nonce":1797655686}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6915,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4203087300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4203087300},{"type":"v1","addr":"172.21.15.89:6836","nonce":4203087300}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2024400889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2024400889},{"type":"v1","addr":"172.21.15.89:6837","nonce":2024400889}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[]} 2024-04-02T07:52:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.305+0000 7fc44d600700 1 -- 172.21.15.89:0/971009037 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc42806adb0 msgr2=0x7fc42806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.305+0000 7fc44d600700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc42806adb0 0x7fc42806d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc4380011e0 tx=0x7fc438009040 comp rx=0 tx=0).stop 2024-04-02T07:52:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.305+0000 7fc44d600700 1 -- 172.21.15.89:0/971009037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44815d640 msgr2=0x7fc448155f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.305+0000 7fc44d600700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44815d640 0x7fc448155f60 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc43400ebe0 tx=0x7fc434009e20 comp rx=0 tx=0).stop 2024-04-02T07:52:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.306+0000 7fc44d600700 1 -- 172.21.15.89:0/971009037 shutdown_connections 2024-04-02T07:52:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.306+0000 7fc44d600700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc42806adb0 0x7fc42806d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.306+0000 7fc44d600700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc448161800 0x7fc4481564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:43.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.306+0000 7fc44d600700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4480ceec0 0x7fc4481f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.306+0000 7fc44d600700 1 --2- 172.21.15.89:0/971009037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc44815d640 0x7fc448155f60 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.306+0000 7fc44d600700 1 -- 172.21.15.89:0/971009037 >> 172.21.15.89:0/971009037 conn(0x7fc44808aff0 msgr2=0x7fc44815cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.306+0000 7fc44d600700 1 -- 172.21.15.89:0/971009037 shutdown_connections 2024-04-02T07:52:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.306+0000 7fc44d600700 1 -- 172.21.15.89:0/971009037 wait complete. 2024-04-02T07:52:43.309 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 43 2024-04-02T07:52:43.326 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T07:52:43.327 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T07:52:43.327 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T07:52:43.327 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:52:43.327 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T07:52:43.327 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T07:52:43.327 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T07:52:43.327 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:52:43.328 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T07:52:43.328 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T07:52:43.328 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T07:52:43.328 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:52:43.328 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T07:52:43.328 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T07:52:43.328 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T07:52:43.329 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:52:43.343 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T07:52:43.343 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T07:52:43.344 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T07:52:43.344 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T07:52:43.345 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T07:52:43.345 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T07:52:43.346 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T07:52:43.346 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T07:52:43.388 INFO:tasks.ceph.mds.b:Started 2024-04-02T07:52:43.388 DEBUG:teuthology.parallel:result is None 2024-04-02T07:52:43.390 INFO:tasks.ceph.mds.a:Started 2024-04-02T07:52:43.390 DEBUG:teuthology.parallel:result is None 2024-04-02T07:52:43.391 INFO:tasks.ceph.mds.d:Started 2024-04-02T07:52:43.391 DEBUG:teuthology.parallel:result is None 2024-04-02T07:52:43.392 INFO:tasks.ceph.mds.c:Started 2024-04-02T07:52:43.392 DEBUG:teuthology.parallel:result is None 2024-04-02T07:52:43.393 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T07:52:43.530 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T07:52:43.531 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T07:52:43.537 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T07:52:43.538 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T07:52:43.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.595+0000 7f2659c00700 1 Processor -- start 2024-04-02T07:52:43.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.596+0000 7f2659c00700 1 -- start start 2024-04-02T07:52:43.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.597+0000 7f2659c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f265415d9d0 0x7f26540d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:43.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.597+0000 7f2659c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f26540d16a0 0x7f26540d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:43.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.597+0000 7f2659c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2654161420 con 0x7f26540d2370 2024-04-02T07:52:43.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.597+0000 7f2659c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26540d27e0 con 0x7f26540d16a0 2024-04-02T07:52:43.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.597+0000 7f2659c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2654161700 con 0x7f265415d9d0 2024-04-02T07:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.597+0000 7f264be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f26540d16a0 0x7f26540d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.597+0000 7f2653400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f26540d2370 0x7f265415ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43742/0 (socket says 172.21.15.89:43742) 2024-04-02T07:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.597+0000 7f264be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f26540d16a0 0x7f26540d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40602/0 (socket says 172.21.15.89:40602) 2024-04-02T07:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.597+0000 7f2653400700 1 -- 172.21.15.89:0/463244438 learned_addr learned my addr 172.21.15.89:0/463244438 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.597+0000 7f2652a00700 1 --2- 172.21.15.89:0/463244438 >> v2:172.21.15.113:3300/0 conn(0x7f265415d9d0 0x7f26540d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.597+0000 7f264be00700 1 -- 172.21.15.89:0/463244438 >> v2:172.21.15.113:3300/0 conn(0x7f265415d9d0 msgr2=0x7f26540d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.597+0000 7f264be00700 1 --2- 172.21.15.89:0/463244438 >> v2:172.21.15.113:3300/0 conn(0x7f265415d9d0 0x7f26540d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.597+0000 7f264be00700 1 -- 172.21.15.89:0/463244438 >> v1:172.21.15.89:6789/0 conn(0x7f26540d2370 legacy=0x7f265415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.597+0000 7f264be00700 1 -- 172.21.15.89:0/463244438 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26541619e0 con 0x7f26540d16a0 2024-04-02T07:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.598+0000 7f264be00700 1 --2- 172.21.15.89:0/463244438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26540d16a0 0x7f26540d1e30 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f26540d0ff0 tx=0x7f263c009b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8a1020b22cab23 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.598+0000 7f2652000700 1 -- 172.21.15.89:0/463244438 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c011830 con 0x7f26540d16a0 2024-04-02T07:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.598+0000 7f2652000700 1 -- 172.21.15.89:0/463244438 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263c011990 con 0x7f26540d16a0 2024-04-02T07:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.598+0000 7f2652000700 1 -- 172.21.15.89:0/463244438 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c011d40 con 0x7f26540d16a0 2024-04-02T07:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.598+0000 7f2659c00700 1 -- 172.21.15.89:0/463244438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26540d16a0 msgr2=0x7f26540d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.598+0000 7f2659c00700 1 --2- 172.21.15.89:0/463244438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26540d16a0 0x7f26540d1e30 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f26540d0ff0 tx=0x7f263c009b10 comp rx=0 tx=0).stop 2024-04-02T07:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.598+0000 7f2659c00700 1 -- 172.21.15.89:0/463244438 shutdown_connections 2024-04-02T07:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.598+0000 7f2659c00700 1 --2- 172.21.15.89:0/463244438 >> v2:172.21.15.113:3300/0 conn(0x7f265415d9d0 0x7f26540d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.598+0000 7f2659c00700 1 --2- 172.21.15.89:0/463244438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26540d16a0 0x7f26540d1e30 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.598+0000 7f2659c00700 1 -- 172.21.15.89:0/463244438 >> 172.21.15.89:0/463244438 conn(0x7f265408aff0 msgr2=0x7f26540d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.599+0000 7f2659c00700 1 -- 172.21.15.89:0/463244438 shutdown_connections 2024-04-02T07:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.599+0000 7f2659c00700 1 -- 172.21.15.89:0/463244438 wait complete. 2024-04-02T07:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.601+0000 7f2659c00700 1 Processor -- start 2024-04-02T07:52:43.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.601+0000 7f2659c00700 1 -- start start 2024-04-02T07:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.602+0000 7f2659c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26540d16a0 0x7f26540caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.602+0000 7f2659c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26540d2370 0x7f26540b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.602+0000 7f2659c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f265415d9d0 0x7f26540b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.602+0000 7f2659c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2654155fa0 con 0x7f26540d16a0 2024-04-02T07:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.602+0000 7f2659c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26540cc980 con 0x7f265415d9d0 2024-04-02T07:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.602+0000 7f2659c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26540ccc60 con 0x7f26540d2370 2024-04-02T07:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.602+0000 7f2652a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26540d16a0 0x7f26540caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.602+0000 7f2652a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26540d16a0 0x7f26540caa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40618/0 (socket says 172.21.15.89:40618) 2024-04-02T07:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.602+0000 7f2652a00700 1 -- 172.21.15.89:0/15414440 learned_addr learned my addr 172.21.15.89:0/15414440 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.603+0000 7f2653400700 1 --2- 172.21.15.89:0/15414440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f265415d9d0 0x7f26540b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.603+0000 7f264be00700 1 --2- 172.21.15.89:0/15414440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26540d2370 0x7f26540b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.603+0000 7f2652a00700 1 -- 172.21.15.89:0/15414440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26540d2370 msgr2=0x7f26540b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.603+0000 7f2652a00700 1 --2- 172.21.15.89:0/15414440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26540d2370 0x7f26540b4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.603+0000 7f2652a00700 1 -- 172.21.15.89:0/15414440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f265415d9d0 msgr2=0x7f26540b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.603+0000 7f2652a00700 1 --2- 172.21.15.89:0/15414440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f265415d9d0 0x7f26540b4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.603+0000 7f2652a00700 1 -- 172.21.15.89:0/15414440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c009440 con 0x7f26540d16a0 2024-04-02T07:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.603+0000 7f2652a00700 1 --2- 172.21.15.89:0/15414440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26540d16a0 0x7f26540caa90 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f264400e870 tx=0x7f264400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.603+0000 7f264aa00700 1 -- 172.21.15.89:0/15414440 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2644009590 con 0x7f26540d16a0 2024-04-02T07:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.603+0000 7f264aa00700 1 -- 172.21.15.89:0/15414440 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264400ee70 con 0x7f26540d16a0 2024-04-02T07:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.603+0000 7f264aa00700 1 -- 172.21.15.89:0/15414440 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264400ca50 con 0x7f26540d16a0 2024-04-02T07:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.603+0000 7f2659c00700 1 -- 172.21.15.89:0/15414440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26540ccf40 con 0x7f26540d16a0 2024-04-02T07:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.604+0000 7f2659c00700 1 -- 172.21.15.89:0/15414440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26540cd500 con 0x7f26540d16a0 2024-04-02T07:52:43.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.604+0000 7f2659c00700 1 -- 172.21.15.89:0/15414440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26540d1ca0 con 0x7f26540d16a0 2024-04-02T07:52:43.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.605+0000 7f264aa00700 1 -- 172.21.15.89:0/15414440 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f264400cbb0 con 0x7f26540d16a0 2024-04-02T07:52:43.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.606+0000 7f264aa00700 1 --2- 172.21.15.89:0/15414440 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f263006adb0 0x7f263006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:43.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.606+0000 7f264aa00700 1 -- 172.21.15.89:0/15414440 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f2644089600 con 0x7f26540d16a0 2024-04-02T07:52:43.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.607+0000 7f264be00700 1 --2- 172.21.15.89:0/15414440 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f263006adb0 0x7f263006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:43.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.607+0000 7f264be00700 1 --2- 172.21.15.89:0/15414440 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f263006adb0 0x7f263006d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f26540d1ae0 tx=0x7f263c01c040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:43.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.611+0000 7f264aa00700 1 -- 172.21.15.89:0/15414440 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2644054570 con 0x7f26540d16a0 2024-04-02T07:52:43.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:43.828+0000 7f2659c00700 1 -- 172.21.15.89:0/15414440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2654067720 con 0x7f26540d16a0 2024-04-02T07:52:44.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.585+0000 7f264aa00700 1 -- 172.21.15.89:0/15414440 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v107) v1 ==== 97+0+0 (secure 0 0 0) 0x7f264400ce60 con 0x7f26540d16a0 2024-04-02T07:52:44.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.588+0000 7f2659c00700 1 -- 172.21.15.89:0/15414440 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f263006adb0 msgr2=0x7f263006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:44.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.588+0000 7f2659c00700 1 --2- 172.21.15.89:0/15414440 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f263006adb0 0x7f263006d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f26540d1ae0 tx=0x7f263c01c040 comp rx=0 tx=0).stop 2024-04-02T07:52:44.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.588+0000 7f2659c00700 1 -- 172.21.15.89:0/15414440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26540d16a0 msgr2=0x7f26540caa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:44.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.588+0000 7f2659c00700 1 --2- 172.21.15.89:0/15414440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26540d16a0 0x7f26540caa90 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f264400e870 tx=0x7f264400ec30 comp rx=0 tx=0).stop 2024-04-02T07:52:44.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.588+0000 7f2659c00700 1 -- 172.21.15.89:0/15414440 shutdown_connections 2024-04-02T07:52:44.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.588+0000 7f2659c00700 1 --2- 172.21.15.89:0/15414440 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f263006adb0 0x7f263006d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:44.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.588+0000 7f2659c00700 1 --2- 172.21.15.89:0/15414440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f265415d9d0 0x7f26540b4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:44.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.588+0000 7f2659c00700 1 --2- 172.21.15.89:0/15414440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26540d16a0 0x7f26540caa90 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:44.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.588+0000 7f2659c00700 1 --2- 172.21.15.89:0/15414440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26540d2370 0x7f26540b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:44.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.588+0000 7f2659c00700 1 -- 172.21.15.89:0/15414440 >> 172.21.15.89:0/15414440 conn(0x7f265408aff0 msgr2=0x7f265415be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:44.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.588+0000 7f2659c00700 1 -- 172.21.15.89:0/15414440 shutdown_connections 2024-04-02T07:52:44.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.589+0000 7f2659c00700 1 -- 172.21.15.89:0/15414440 wait complete. 2024-04-02T07:52:44.593 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T07:52:44.610 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T07:52:44.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.800+0000 7f9c77600700 1 Processor -- start 2024-04-02T07:52:44.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.800+0000 7f9c77600700 1 -- start start 2024-04-02T07:52:44.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.801+0000 7f9c77600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c7015d640 0x7f9c700d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:44.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.801+0000 7f9c77600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9c700ceec0 0x7f9c701612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:44.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.801+0000 7f9c77600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9c70161800 0x7f9c700cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:44.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.801+0000 7f9c77600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c70161c90 con 0x7f9c70161800 2024-04-02T07:52:44.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.801+0000 7f9c77600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c700d30d0 con 0x7f9c700ceec0 2024-04-02T07:52:44.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.801+0000 7f9c77600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c700b6840 con 0x7f9c7015d640 2024-04-02T07:52:44.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.801+0000 7f9c75600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9c70161800 0x7f9c700cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:44.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.801+0000 7f9c75600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9c70161800 0x7f9c700cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40634/0 (socket says 172.21.15.89:40634) 2024-04-02T07:52:44.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.801+0000 7f9c75600700 1 -- 172.21.15.89:0/1240642671 learned_addr learned my addr 172.21.15.89:0/1240642671 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:44.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.801+0000 7f9c74c00700 1 --2- 172.21.15.89:0/1240642671 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c7015d640 0x7f9c700d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:44.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.801+0000 7f9c6fe00700 1 --2- 172.21.15.89:0/1240642671 >> v2:172.21.15.113:3300/0 conn(0x7f9c700ceec0 0x7f9c701612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:44.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.802+0000 7f9c75600700 1 -- 172.21.15.89:0/1240642671 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c7015d640 msgr2=0x7f9c700d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:44.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.802+0000 7f9c75600700 1 --2- 172.21.15.89:0/1240642671 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c7015d640 0x7f9c700d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:44.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.802+0000 7f9c75600700 1 -- 172.21.15.89:0/1240642671 >> v2:172.21.15.113:3300/0 conn(0x7f9c700ceec0 msgr2=0x7f9c701612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:44.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.802+0000 7f9c75600700 1 --2- 172.21.15.89:0/1240642671 >> v2:172.21.15.113:3300/0 conn(0x7f9c700ceec0 0x7f9c701612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:44.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.802+0000 7f9c75600700 1 -- 172.21.15.89:0/1240642671 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c700d2880 con 0x7f9c70161800 2024-04-02T07:52:44.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.802+0000 7f9c75600700 1 --2- 172.21.15.89:0/1240642671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c70161800 0x7f9c700cdba0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f9c7008c1c0 tx=0x7f9c6800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be5a69b74a6167e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:44.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.802+0000 7f9c6f400700 1 -- 172.21.15.89:0/1240642671 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6800e070 con 0x7f9c70161800 2024-04-02T07:52:44.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.802+0000 7f9c6f400700 1 -- 172.21.15.89:0/1240642671 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6800b9c0 con 0x7f9c70161800 2024-04-02T07:52:44.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.802+0000 7f9c6f400700 1 -- 172.21.15.89:0/1240642671 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6800bd70 con 0x7f9c70161800 2024-04-02T07:52:44.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.803+0000 7f9c77600700 1 -- 172.21.15.89:0/1240642671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c70161800 msgr2=0x7f9c700cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:44.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.803+0000 7f9c77600700 1 --2- 172.21.15.89:0/1240642671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c70161800 0x7f9c700cdba0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f9c7008c1c0 tx=0x7f9c6800b330 comp rx=0 tx=0).stop 2024-04-02T07:52:44.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.803+0000 7f9c77600700 1 -- 172.21.15.89:0/1240642671 shutdown_connections 2024-04-02T07:52:44.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.803+0000 7f9c77600700 1 --2- 172.21.15.89:0/1240642671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c70161800 0x7f9c700cdba0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:44.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.803+0000 7f9c77600700 1 --2- 172.21.15.89:0/1240642671 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c7015d640 0x7f9c700d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:44.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.803+0000 7f9c77600700 1 --2- 172.21.15.89:0/1240642671 >> v2:172.21.15.113:3300/0 conn(0x7f9c700ceec0 0x7f9c701612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:44.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.803+0000 7f9c77600700 1 -- 172.21.15.89:0/1240642671 >> 172.21.15.89:0/1240642671 conn(0x7f9c7008aff0 msgr2=0x7f9c700d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:44.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.803+0000 7f9c77600700 1 -- 172.21.15.89:0/1240642671 shutdown_connections 2024-04-02T07:52:44.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.803+0000 7f9c77600700 1 -- 172.21.15.89:0/1240642671 wait complete. 2024-04-02T07:52:44.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.805+0000 7f9c77600700 1 Processor -- start 2024-04-02T07:52:44.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.805+0000 7f9c77600700 1 -- start start 2024-04-02T07:52:44.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.806+0000 7f9c77600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c700ceec0 0x7f9c70157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:44.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.806+0000 7f9c77600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c7015d640 0x7f9c700c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:44.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c77600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c70161800 0x7f9c7015f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:44.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c77600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c700c8df0 con 0x7f9c70161800 2024-04-02T07:52:44.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c77600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c70157fe0 con 0x7f9c700ceec0 2024-04-02T07:52:44.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c77600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c700c81e0 con 0x7f9c7015d640 2024-04-02T07:52:44.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c75600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c70161800 0x7f9c7015f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:44.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c75600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c70161800 0x7f9c7015f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40640/0 (socket says 172.21.15.89:40640) 2024-04-02T07:52:44.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c75600700 1 -- 172.21.15.89:0/3995660378 learned_addr learned my addr 172.21.15.89:0/3995660378 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:44.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c6fe00700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c7015d640 0x7f9c700c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:44.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c74c00700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c700ceec0 0x7f9c70157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:44.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c75600700 1 -- 172.21.15.89:0/3995660378 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c7015d640 msgr2=0x7f9c700c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:44.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c75600700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c7015d640 0x7f9c700c9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:44.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c75600700 1 -- 172.21.15.89:0/3995660378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c700ceec0 msgr2=0x7f9c70157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:44.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c75600700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c700ceec0 0x7f9c70157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:44.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c75600700 1 -- 172.21.15.89:0/3995660378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c68009b30 con 0x7f9c70161800 2024-04-02T07:52:44.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c6fe00700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c7015d640 0x7f9c700c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:44.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c74c00700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c700ceec0 0x7f9c70157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:44.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.807+0000 7f9c75600700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c70161800 0x7f9c7015f9a0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f9c7008c490 tx=0x7f9c68004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:44.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.808+0000 7f9c6cc00700 1 -- 172.21.15.89:0/3995660378 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c68004250 con 0x7f9c70161800 2024-04-02T07:52:44.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.808+0000 7f9c6cc00700 1 -- 172.21.15.89:0/3995660378 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c680043d0 con 0x7f9c70161800 2024-04-02T07:52:44.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.808+0000 7f9c6cc00700 1 -- 172.21.15.89:0/3995660378 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c68004780 con 0x7f9c70161800 2024-04-02T07:52:44.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.808+0000 7f9c77600700 1 -- 172.21.15.89:0/3995660378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c700c84c0 con 0x7f9c70161800 2024-04-02T07:52:44.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.808+0000 7f9c77600700 1 -- 172.21.15.89:0/3995660378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c700c8a80 con 0x7f9c70161800 2024-04-02T07:52:44.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.809+0000 7f9c77600700 1 -- 172.21.15.89:0/3995660378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c700c8fa0 con 0x7f9c70161800 2024-04-02T07:52:44.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.810+0000 7f9c6cc00700 1 -- 172.21.15.89:0/3995660378 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9c6801c440 con 0x7f9c70161800 2024-04-02T07:52:44.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.811+0000 7f9c6cc00700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c5006adb0 0x7f9c5006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:44.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.811+0000 7f9c6cc00700 1 -- 172.21.15.89:0/3995660378 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f9c6808a9f0 con 0x7f9c70161800 2024-04-02T07:52:44.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.811+0000 7f9c74c00700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c5006adb0 0x7f9c5006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:44.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.811+0000 7f9c74c00700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c5006adb0 0x7f9c5006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9c64009970 tx=0x7f9c64008040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:44.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:44.815+0000 7f9c6cc00700 1 -- 172.21.15.89:0/3995660378 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9c700c8fa0 con 0x7f9c70161800 2024-04-02T07:52:45.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.103+0000 7f9c77600700 1 -- 172.21.15.89:0/3995660378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9c70067720 con 0x7f9c70161800 2024-04-02T07:52:45.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.104+0000 7f9c6cc00700 1 -- 172.21.15.89:0/3995660378 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f9c6802a090 con 0x7f9c70161800 2024-04-02T07:52:45.106 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:45.106 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:52:45.106 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T07:52:45.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.106 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T07:52:45.106 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:52:45.107 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.108 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.108 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T07:52:45.108 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T07:52:45.108 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.108 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:45.108 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:52:45.108 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:52:45.108 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.108 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.109 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T07:52:45.109 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T07:52:45.109 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.109 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:45.109 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:52:45.109 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:52:45.109 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.109 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.109 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T07:52:45.109 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T07:52:45.110 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.110 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:45.110 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:45.110 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.110 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T07:52:45.110 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T07:52:45.110 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.110 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:45.110 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:45.110 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T07:52:45.111 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.112 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:45.112 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:45.112 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.112 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.112 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T07:52:45.112 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T07:52:45.112 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.112 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:45.112 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:45.112 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.112 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:45.113 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:45.114 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.114 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.114 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T07:52:45.114 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T07:52:45.114 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.114 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:52:45.114 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:52:45.114 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.114 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.114 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T07:52:45.115 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T07:52:45.115 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.115 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:45.115 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2024-04-02T07:52:45.115 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T07:52:45.115 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.115 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T07:52:45.115 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T07:52:45.115 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.115 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.117 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T07:52:45.117 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T07:52:45.117 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.117 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:52:45.117 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:52:45.117 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:52:45.117 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:52:45.117 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.117 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T07:52:45.117 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T07:52:45.117 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.118 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:52:45.118 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:52:45.118 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:52:45.118 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.118 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T07:52:45.118 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T07:52:45.118 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.118 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:52:45.118 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:52:45.118 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.119 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:52:45.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:45.119 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T07:52:45.119 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T07:52:45.119 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:52:45.119 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:52:45.119 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:52:45.119 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:52:45.119 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.119 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:45.119 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T07:52:45.119 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:52:45.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.106+0000 7f9c77600700 1 -- 172.21.15.89:0/3995660378 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c5006adb0 msgr2=0x7f9c5006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.106+0000 7f9c77600700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c5006adb0 0x7f9c5006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9c64009970 tx=0x7f9c64008040 comp rx=0 tx=0).stop 2024-04-02T07:52:45.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.106+0000 7f9c77600700 1 -- 172.21.15.89:0/3995660378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c70161800 msgr2=0x7f9c7015f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.106+0000 7f9c77600700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c70161800 0x7f9c7015f9a0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f9c7008c490 tx=0x7f9c68004ad0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.106+0000 7f9c77600700 1 -- 172.21.15.89:0/3995660378 shutdown_connections 2024-04-02T07:52:45.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.106+0000 7f9c77600700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c5006adb0 0x7f9c5006d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.106+0000 7f9c77600700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c700ceec0 0x7f9c70157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.106+0000 7f9c77600700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c70161800 0x7f9c7015f9a0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.106+0000 7f9c77600700 1 --2- 172.21.15.89:0/3995660378 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c7015d640 0x7f9c700c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.106+0000 7f9c77600700 1 -- 172.21.15.89:0/3995660378 >> 172.21.15.89:0/3995660378 conn(0x7f9c7008aff0 msgr2=0x7f9c7015aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:45.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.106+0000 7f9c77600700 1 -- 172.21.15.89:0/3995660378 shutdown_connections 2024-04-02T07:52:45.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.106+0000 7f9c77600700 1 -- 172.21.15.89:0/3995660378 wait complete. 2024-04-02T07:52:45.127 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:52:45.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.316+0000 7f575c400700 1 Processor -- start 2024-04-02T07:52:45.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.316+0000 7f575c400700 1 -- start start 2024-04-02T07:52:45.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.317+0000 7f575c400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5754160f10 0x7f57540d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:45.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.317+0000 7f575c400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57540ceda0 con 0x7f57540cf230 2024-04-02T07:52:45.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.318+0000 7f575c400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57541612f0 con 0x7f5754161590 2024-04-02T07:52:45.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.318+0000 7f575c400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57540ce470 con 0x7f5754160f10 2024-04-02T07:52:45.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.318+0000 7f5759000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f57540cf230 0x7f57540d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43754/0 (socket says 172.21.15.89:43754) 2024-04-02T07:52:45.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.318+0000 7f5759000700 1 -- 172.21.15.89:0/1908295020 learned_addr learned my addr 172.21.15.89:0/1908295020 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:45.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.318+0000 7f5759a00700 1 --2- 172.21.15.89:0/1908295020 >> v2:172.21.15.113:3300/0 conn(0x7f5754160f10 0x7f57540d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:45.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.318+0000 7f5753e00700 1 -- 172.21.15.89:0/1908295020 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082504352 0 0) 0x7f57540ceda0 con 0x7f57540cf230 2024-04-02T07:52:45.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.318+0000 7f5753e00700 1 -- 172.21.15.89:0/1908295020 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573c0035b0 con 0x7f57540cf230 2024-04-02T07:52:45.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.318+0000 7f5753e00700 1 -- 172.21.15.89:0/1908295020 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605360365 0 0) 0x7f57541612f0 con 0x7f5754161590 2024-04-02T07:52:45.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.318+0000 7f5753e00700 1 -- 172.21.15.89:0/1908295020 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57540ceda0 con 0x7f5754161590 2024-04-02T07:52:45.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.318+0000 7f5759a00700 1 -- 172.21.15.89:0/1908295020 >> v1:172.21.15.113:6789/0 conn(0x7f5754161590 legacy=0x7f57540b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.318+0000 7f5753e00700 1 -- 172.21.15.89:0/1908295020 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1846740317 0 0) 0x7f573c0035b0 con 0x7f57540cf230 2024-04-02T07:52:45.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.318+0000 7f5759a00700 1 -- 172.21.15.89:0/1908295020 >> v1:172.21.15.89:6789/0 conn(0x7f57540cf230 legacy=0x7f57540d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.319+0000 7f5759a00700 1 -- 172.21.15.89:0/1908295020 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57540ce750 con 0x7f5754160f10 2024-04-02T07:52:45.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.319+0000 7f5759a00700 1 --2- 172.21.15.89:0/1908295020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5754160f10 0x7f57540d18b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5748009840 tx=0x7f5748009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffdc58fdb73f8c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:45.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.319+0000 7f5753e00700 1 -- 172.21.15.89:0/1908295020 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740001670 con 0x7f5754160f10 2024-04-02T07:52:45.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.319+0000 7f5753e00700 1 -- 172.21.15.89:0/1908295020 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5748004030 con 0x7f5754160f10 2024-04-02T07:52:45.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.319+0000 7f5753e00700 1 -- 172.21.15.89:0/1908295020 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57480043a0 con 0x7f5754160f10 2024-04-02T07:52:45.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.320+0000 7f575c400700 1 -- 172.21.15.89:0/1908295020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5754160f10 msgr2=0x7f57540d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.320+0000 7f575c400700 1 --2- 172.21.15.89:0/1908295020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5754160f10 0x7f57540d18b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5748009840 tx=0x7f5748009b50 comp rx=0 tx=0).stop 2024-04-02T07:52:45.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.320+0000 7f575c400700 1 -- 172.21.15.89:0/1908295020 shutdown_connections 2024-04-02T07:52:45.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.320+0000 7f575c400700 1 --2- 172.21.15.89:0/1908295020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5754160f10 0x7f57540d18b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.320+0000 7f575c400700 1 -- 172.21.15.89:0/1908295020 >> 172.21.15.89:0/1908295020 conn(0x7f575408aff0 msgr2=0x7f57540d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:45.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.320+0000 7f575c400700 1 -- 172.21.15.89:0/1908295020 shutdown_connections 2024-04-02T07:52:45.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.320+0000 7f575c400700 1 -- 172.21.15.89:0/1908295020 wait complete. 2024-04-02T07:52:45.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.321+0000 7f575c400700 1 Processor -- start 2024-04-02T07:52:45.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.322+0000 7f575c400700 1 -- start start 2024-04-02T07:52:45.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.322+0000 7f575c400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f57540cf230 0x7f5754158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:45.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.322+0000 7f575c400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5754160f10 0x7f57540b6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:45.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.322+0000 7f575c400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5754161590 0x7f57541e88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:45.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.322+0000 7f575c400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575415e120 con 0x7f5754161590 2024-04-02T07:52:45.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.323+0000 7f575c400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57540b7cb0 con 0x7f5754160f10 2024-04-02T07:52:45.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.323+0000 7f575c400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57540b7410 con 0x7f57540cf230 2024-04-02T07:52:45.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.323+0000 7f575a400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5754161590 0x7f57541e88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:45.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.323+0000 7f575a400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5754161590 0x7f57541e88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40646/0 (socket says 172.21.15.89:40646) 2024-04-02T07:52:45.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.323+0000 7f575a400700 1 -- 172.21.15.89:0/1973680790 learned_addr learned my addr 172.21.15.89:0/1973680790 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:45.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.323+0000 7f5759a00700 1 --2- 172.21.15.89:0/1973680790 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f57540cf230 0x7f5754158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:45.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.323+0000 7f5759000700 1 --2- 172.21.15.89:0/1973680790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5754160f10 0x7f57540b6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:45.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.323+0000 7f575a400700 1 -- 172.21.15.89:0/1973680790 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f57540cf230 msgr2=0x7f5754158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.323+0000 7f575a400700 1 --2- 172.21.15.89:0/1973680790 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f57540cf230 0x7f5754158c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.323+0000 7f575a400700 1 -- 172.21.15.89:0/1973680790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5754160f10 msgr2=0x7f57540b6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.323+0000 7f575a400700 1 --2- 172.21.15.89:0/1973680790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5754160f10 0x7f57540b6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.323+0000 7f575a400700 1 -- 172.21.15.89:0/1973680790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57480094a0 con 0x7f5754161590 2024-04-02T07:52:45.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.323+0000 7f575a400700 1 --2- 172.21.15.89:0/1973680790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5754161590 0x7f57541e88e0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f574c00cbc0 tx=0x7f574c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:45.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.324+0000 7f5751600700 1 -- 172.21.15.89:0/1973680790 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c003c00 con 0x7f5754161590 2024-04-02T07:52:45.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.324+0000 7f5751600700 1 -- 172.21.15.89:0/1973680790 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574c003d60 con 0x7f5754161590 2024-04-02T07:52:45.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.324+0000 7f575c400700 1 -- 172.21.15.89:0/1973680790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57540b76f0 con 0x7f5754161590 2024-04-02T07:52:45.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.324+0000 7f5751600700 1 -- 172.21.15.89:0/1973680790 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c00aaa0 con 0x7f5754161590 2024-04-02T07:52:45.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.324+0000 7f575c400700 1 -- 172.21.15.89:0/1973680790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5754159140 con 0x7f5754161590 2024-04-02T07:52:45.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.325+0000 7f575c400700 1 -- 172.21.15.89:0/1973680790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57540cf690 con 0x7f5754161590 2024-04-02T07:52:45.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.326+0000 7f5751600700 1 -- 172.21.15.89:0/1973680790 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f574c00ac00 con 0x7f5754161590 2024-04-02T07:52:45.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.326+0000 7f5751600700 1 --2- 172.21.15.89:0/1973680790 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f573406adb0 0x7f573406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:45.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.327+0000 7f5751600700 1 -- 172.21.15.89:0/1973680790 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f574c089290 con 0x7f5754161590 2024-04-02T07:52:45.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.327+0000 7f5759a00700 1 --2- 172.21.15.89:0/1973680790 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f573406adb0 0x7f573406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:45.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.327+0000 7f5759a00700 1 --2- 172.21.15.89:0/1973680790 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f573406adb0 0x7f573406d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f57480090b0 tx=0x7f5748008030 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:45.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.331+0000 7f5751600700 1 -- 172.21.15.89:0/1973680790 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f57540cf690 con 0x7f5754161590 2024-04-02T07:52:45.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.537+0000 7f575c400700 1 -- 172.21.15.89:0/1973680790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57540cf690 con 0x7f5754161590 2024-04-02T07:52:45.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.538+0000 7f5751600700 1 -- 172.21.15.89:0/1973680790 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+12918 (secure 0 0 0) 0x7f57540cf690 con 0x7f5754161590 2024-04-02T07:52:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:45.541 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":107,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:52:44.582567+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:52:45.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.541+0000 7f575c400700 1 -- 172.21.15.89:0/1973680790 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f573406adb0 msgr2=0x7f573406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.541+0000 7f575c400700 1 --2- 172.21.15.89:0/1973680790 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f573406adb0 0x7f573406d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f57480090b0 tx=0x7f5748008030 comp rx=0 tx=0).stop 2024-04-02T07:52:45.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.541+0000 7f575c400700 1 -- 172.21.15.89:0/1973680790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5754161590 msgr2=0x7f57541e88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.541+0000 7f575c400700 1 --2- 172.21.15.89:0/1973680790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5754161590 0x7f57541e88e0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f574c00cbc0 tx=0x7f574c00ced0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.541+0000 7f575c400700 1 -- 172.21.15.89:0/1973680790 shutdown_connections 2024-04-02T07:52:45.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.541+0000 7f575c400700 1 --2- 172.21.15.89:0/1973680790 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f573406adb0 0x7f573406d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.541+0000 7f575c400700 1 --2- 172.21.15.89:0/1973680790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5754161590 0x7f57541e88e0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.541+0000 7f575c400700 1 --2- 172.21.15.89:0/1973680790 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f57540cf230 0x7f5754158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.541+0000 7f575c400700 1 --2- 172.21.15.89:0/1973680790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5754160f10 0x7f57540b6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.541+0000 7f575c400700 1 -- 172.21.15.89:0/1973680790 >> 172.21.15.89:0/1973680790 conn(0x7f575408aff0 msgr2=0x7f57540ccc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:45.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.541+0000 7f575c400700 1 -- 172.21.15.89:0/1973680790 shutdown_connections 2024-04-02T07:52:45.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.541+0000 7f575c400700 1 -- 172.21.15.89:0/1973680790 wait complete. 2024-04-02T07:52:45.561 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:52:45.561 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:52:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.753+0000 7feb57c00700 1 Processor -- start 2024-04-02T07:52:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.754+0000 7feb57c00700 1 -- start start 2024-04-02T07:52:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.755+0000 7feb57c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feb500ced80 0x7feb500d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.755+0000 7feb57c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7feb5015fcb0 0x7feb50160140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.755+0000 7feb57c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb500cf1e0 con 0x7feb5015dad0 2024-04-02T07:52:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.755+0000 7feb57c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb500cfcc0 con 0x7feb5015fcb0 2024-04-02T07:52:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.755+0000 7feb57c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb500cd9f0 con 0x7feb500ced80 2024-04-02T07:52:45.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.755+0000 7feb4fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feb500ced80 0x7feb500d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:45.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.755+0000 7feb55200700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7feb5015dad0 0x7feb500d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:58662/0 (socket says 172.21.15.89:58662) 2024-04-02T07:52:45.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.755+0000 7feb55200700 1 -- 172.21.15.89:0/4111274756 learned_addr learned my addr 172.21.15.89:0/4111274756 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:45.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.755+0000 7feb55c00700 1 --2- 172.21.15.89:0/4111274756 >> v2:172.21.15.113:3300/0 conn(0x7feb5015fcb0 0x7feb50160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:45.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.756+0000 7feb4fe00700 1 -- 172.21.15.89:0/4111274756 >> v2:172.21.15.113:3300/0 conn(0x7feb5015fcb0 msgr2=0x7feb50160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:45.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.756+0000 7feb4fe00700 1 --2- 172.21.15.89:0/4111274756 >> v2:172.21.15.113:3300/0 conn(0x7feb5015fcb0 0x7feb50160140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.756+0000 7feb4fe00700 1 -- 172.21.15.89:0/4111274756 >> v1:172.21.15.113:6789/0 conn(0x7feb5015dad0 legacy=0x7feb500d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.756+0000 7feb4fe00700 1 -- 172.21.15.89:0/4111274756 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5015f820 con 0x7feb500ced80 2024-04-02T07:52:45.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.756+0000 7feb55c00700 1 --2- 172.21.15.89:0/4111274756 >> v2:172.21.15.113:3300/0 conn(0x7feb5015fcb0 0x7feb50160140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:45.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.756+0000 7feb4fe00700 1 --2- 172.21.15.89:0/4111274756 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feb500ced80 0x7feb500d1d40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7feb3c009840 tx=0x7feb3c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f7781f2800c7297 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.756+0000 7feb4f400700 1 -- 172.21.15.89:0/4111274756 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c011830 con 0x7feb500ced80 2024-04-02T07:52:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.756+0000 7feb4f400700 1 -- 172.21.15.89:0/4111274756 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c011990 con 0x7feb500ced80 2024-04-02T07:52:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.756+0000 7feb4f400700 1 -- 172.21.15.89:0/4111274756 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c011d20 con 0x7feb500ced80 2024-04-02T07:52:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.757+0000 7feb57c00700 1 -- 172.21.15.89:0/4111274756 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feb500ced80 msgr2=0x7feb500d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.757+0000 7feb57c00700 1 --2- 172.21.15.89:0/4111274756 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feb500ced80 0x7feb500d1d40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7feb3c009840 tx=0x7feb3c009b50 comp rx=0 tx=0).stop 2024-04-02T07:52:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.757+0000 7feb57c00700 1 -- 172.21.15.89:0/4111274756 shutdown_connections 2024-04-02T07:52:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.757+0000 7feb57c00700 1 --2- 172.21.15.89:0/4111274756 >> v2:172.21.15.113:3300/0 conn(0x7feb5015fcb0 0x7feb50160140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.757+0000 7feb57c00700 1 --2- 172.21.15.89:0/4111274756 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feb500ced80 0x7feb500d1d40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.757+0000 7feb57c00700 1 -- 172.21.15.89:0/4111274756 >> 172.21.15.89:0/4111274756 conn(0x7feb5008aff0 msgr2=0x7feb500d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.757+0000 7feb57c00700 1 -- 172.21.15.89:0/4111274756 shutdown_connections 2024-04-02T07:52:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.757+0000 7feb57c00700 1 -- 172.21.15.89:0/4111274756 wait complete. 2024-04-02T07:52:45.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.759+0000 7feb57c00700 1 Processor -- start 2024-04-02T07:52:45.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.759+0000 7feb57c00700 1 -- start start 2024-04-02T07:52:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.760+0000 7feb57c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb500ced80 0x7feb500c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.760+0000 7feb57c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feb5015dad0 0x7feb500c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.760+0000 7feb57c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb5015fcb0 0x7feb500c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.760+0000 7feb57c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb500c9a90 con 0x7feb5015fcb0 2024-04-02T07:52:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.760+0000 7feb57c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5009ed40 con 0x7feb500ced80 2024-04-02T07:52:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.760+0000 7feb57c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5009d250 con 0x7feb5015dad0 2024-04-02T07:52:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.760+0000 7feb55c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb5015fcb0 0x7feb500c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.760+0000 7feb55200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb500ced80 0x7feb500c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.760+0000 7feb55c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb5015fcb0 0x7feb500c85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40648/0 (socket says 172.21.15.89:40648) 2024-04-02T07:52:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.760+0000 7feb55c00700 1 -- 172.21.15.89:0/3376752151 learned_addr learned my addr 172.21.15.89:0/3376752151 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.760+0000 7feb4fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feb5015dad0 0x7feb500c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.761+0000 7feb55c00700 1 -- 172.21.15.89:0/3376752151 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feb5015dad0 msgr2=0x7feb500c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.761+0000 7feb55c00700 1 --2- 172.21.15.89:0/3376752151 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feb5015dad0 0x7feb500c80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.761+0000 7feb55c00700 1 -- 172.21.15.89:0/3376752151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb500ced80 msgr2=0x7feb500c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.761+0000 7feb55c00700 1 --2- 172.21.15.89:0/3376752151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb500ced80 0x7feb500c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.761+0000 7feb55c00700 1 -- 172.21.15.89:0/3376752151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb48009a70 con 0x7feb5015fcb0 2024-04-02T07:52:45.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.761+0000 7feb55c00700 1 --2- 172.21.15.89:0/3376752151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb5015fcb0 0x7feb500c85e0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7feb4800efc0 tx=0x7feb4800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:45.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.761+0000 7feb4cc00700 1 -- 172.21.15.89:0/3376752151 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4800ca90 con 0x7feb5015fcb0 2024-04-02T07:52:45.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.761+0000 7feb4cc00700 1 -- 172.21.15.89:0/3376752151 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4800f040 con 0x7feb5015fcb0 2024-04-02T07:52:45.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.761+0000 7feb57c00700 1 -- 172.21.15.89:0/3376752151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb3c0094a0 con 0x7feb5015fcb0 2024-04-02T07:52:45.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.761+0000 7feb4cc00700 1 -- 172.21.15.89:0/3376752151 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4800ce30 con 0x7feb5015fcb0 2024-04-02T07:52:45.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.762+0000 7feb57c00700 1 -- 172.21.15.89:0/3376752151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb501f3150 con 0x7feb5015fcb0 2024-04-02T07:52:45.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.762+0000 7feb57c00700 1 -- 172.21.15.89:0/3376752151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb50067720 con 0x7feb5015fcb0 2024-04-02T07:52:45.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.763+0000 7feb4cc00700 1 -- 172.21.15.89:0/3376752151 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7feb48014480 con 0x7feb5015fcb0 2024-04-02T07:52:45.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.764+0000 7feb4cc00700 1 --2- 172.21.15.89:0/3376752151 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb3006adb0 0x7feb3006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:45.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.764+0000 7feb4cc00700 1 -- 172.21.15.89:0/3376752151 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5205+0+0 (secure 0 0 0) 0x7feb48089b30 con 0x7feb5015fcb0 2024-04-02T07:52:45.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.764+0000 7feb55200700 1 --2- 172.21.15.89:0/3376752151 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb3006adb0 0x7feb3006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:45.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.765+0000 7feb55200700 1 --2- 172.21.15.89:0/3376752151 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb3006adb0 0x7feb3006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feb44005910 tx=0x7feb44009040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:45.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.769+0000 7feb4cc00700 1 -- 172.21.15.89:0/3376752151 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feb48054bc0 con 0x7feb5015fcb0 2024-04-02T07:52:45.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.969+0000 7feb57c00700 1 -- 172.21.15.89:0/3376752151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feb50146e80 con 0x7feb5015fcb0 2024-04-02T07:52:45.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.970+0000 7feb4cc00700 1 -- 172.21.15.89:0/3376752151 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7feb48009cd0 con 0x7feb5015fcb0 2024-04-02T07:52:45.972 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:52:45.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.972+0000 7feb57c00700 1 -- 172.21.15.89:0/3376752151 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb3006adb0 msgr2=0x7feb3006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.972+0000 7feb57c00700 1 --2- 172.21.15.89:0/3376752151 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb3006adb0 0x7feb3006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feb44005910 tx=0x7feb44009040 comp rx=0 tx=0).stop 2024-04-02T07:52:45.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.972+0000 7feb57c00700 1 -- 172.21.15.89:0/3376752151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb5015fcb0 msgr2=0x7feb500c85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:45.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.972+0000 7feb57c00700 1 --2- 172.21.15.89:0/3376752151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb5015fcb0 0x7feb500c85e0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7feb4800efc0 tx=0x7feb4800c3a0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.972+0000 7feb57c00700 1 -- 172.21.15.89:0/3376752151 shutdown_connections 2024-04-02T07:52:45.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.972+0000 7feb57c00700 1 --2- 172.21.15.89:0/3376752151 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feb3006adb0 0x7feb3006d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.972+0000 7feb57c00700 1 --2- 172.21.15.89:0/3376752151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb500ced80 0x7feb500c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.972+0000 7feb57c00700 1 --2- 172.21.15.89:0/3376752151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb5015fcb0 0x7feb500c85e0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.972+0000 7feb57c00700 1 --2- 172.21.15.89:0/3376752151 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feb5015dad0 0x7feb500c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:45.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.973+0000 7feb57c00700 1 -- 172.21.15.89:0/3376752151 >> 172.21.15.89:0/3376752151 conn(0x7feb5008aff0 msgr2=0x7feb500cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:45.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.973+0000 7feb57c00700 1 -- 172.21.15.89:0/3376752151 shutdown_connections 2024-04-02T07:52:45.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:45.973+0000 7feb57c00700 1 -- 172.21.15.89:0/3376752151 wait complete. 2024-04-02T07:52:45.993 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T07:52:46.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.181+0000 7f93eae00700 1 Processor -- start 2024-04-02T07:52:46.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.181+0000 7f93eae00700 1 -- start start 2024-04-02T07:52:46.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.182+0000 7f93eae00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f93e4160f10 0x7f93e40d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:46.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.182+0000 7f93eae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93e40cf230 0x7f93e40d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:46.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.182+0000 7f93eae00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e40ceda0 con 0x7f93e4161590 2024-04-02T07:52:46.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.182+0000 7f93eae00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e40d23d0 con 0x7f93e4160f10 2024-04-02T07:52:46.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.182+0000 7f93eae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e41612f0 con 0x7f93e40cf230 2024-04-02T07:52:46.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.182+0000 7f93e8e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f93e4161590 0x7f93e40b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43770/0 (socket says 172.21.15.89:43770) 2024-04-02T07:52:46.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.182+0000 7f93e8e00700 1 -- 172.21.15.89:0/464254228 learned_addr learned my addr 172.21.15.89:0/464254228 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:46.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.182+0000 7f93e3e00700 1 --2- 172.21.15.89:0/464254228 >> v2:172.21.15.113:3300/0 conn(0x7f93e4160f10 0x7f93e40d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:46.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.182+0000 7f93e3400700 1 --2- 172.21.15.89:0/464254228 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93e40cf230 0x7f93e40d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:46.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.183+0000 7f93e2a00700 1 -- 172.21.15.89:0/464254228 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636893821 0 0) 0x7f93e40ceda0 con 0x7f93e4161590 2024-04-02T07:52:46.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.183+0000 7f93e2a00700 1 -- 172.21.15.89:0/464254228 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c80035b0 con 0x7f93e4161590 2024-04-02T07:52:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.183+0000 7f93e3e00700 1 -- 172.21.15.89:0/464254228 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93e40cf230 msgr2=0x7f93e40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.183+0000 7f93e3e00700 1 --2- 172.21.15.89:0/464254228 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93e40cf230 0x7f93e40d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.183+0000 7f93e3e00700 1 -- 172.21.15.89:0/464254228 >> v1:172.21.15.89:6789/0 conn(0x7f93e4161590 legacy=0x7f93e40b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.183+0000 7f93e3e00700 1 -- 172.21.15.89:0/464254228 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e40ce470 con 0x7f93e4160f10 2024-04-02T07:52:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.183+0000 7f93e3400700 1 --2- 172.21.15.89:0/464254228 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93e40cf230 0x7f93e40d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.183+0000 7f93e3e00700 1 --2- 172.21.15.89:0/464254228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93e4160f10 0x7f93e40d18b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f93d4009840 tx=0x7f93d4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fded5167d74e0d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.183+0000 7f93e2a00700 1 -- 172.21.15.89:0/464254228 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d4011830 con 0x7f93e4160f10 2024-04-02T07:52:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.183+0000 7f93e2a00700 1 -- 172.21.15.89:0/464254228 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d4011990 con 0x7f93e4160f10 2024-04-02T07:52:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.183+0000 7f93e2a00700 1 -- 172.21.15.89:0/464254228 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d4011d00 con 0x7f93e4160f10 2024-04-02T07:52:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.184+0000 7f93eae00700 1 -- 172.21.15.89:0/464254228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93e4160f10 msgr2=0x7f93e40d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.184+0000 7f93eae00700 1 --2- 172.21.15.89:0/464254228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93e4160f10 0x7f93e40d18b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f93d4009840 tx=0x7f93d4009b50 comp rx=0 tx=0).stop 2024-04-02T07:52:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.184+0000 7f93eae00700 1 -- 172.21.15.89:0/464254228 shutdown_connections 2024-04-02T07:52:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.184+0000 7f93eae00700 1 --2- 172.21.15.89:0/464254228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93e4160f10 0x7f93e40d18b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:46.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.184+0000 7f93eae00700 1 --2- 172.21.15.89:0/464254228 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93e40cf230 0x7f93e40d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:46.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.184+0000 7f93eae00700 1 -- 172.21.15.89:0/464254228 >> 172.21.15.89:0/464254228 conn(0x7f93e408aff0 msgr2=0x7f93e40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:46.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.184+0000 7f93eae00700 1 -- 172.21.15.89:0/464254228 shutdown_connections 2024-04-02T07:52:46.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.184+0000 7f93eae00700 1 -- 172.21.15.89:0/464254228 wait complete. 2024-04-02T07:52:46.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.186+0000 7f93eae00700 1 Processor -- start 2024-04-02T07:52:46.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.186+0000 7f93eae00700 1 -- start start 2024-04-02T07:52:46.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93eae00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93e40cf230 0x7f93e415f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:46.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93eae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93e4160f10 0x7f93e415a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:46.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93eae00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e4161590 0x7f93e40b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:46.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93eae00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e40b0290 con 0x7f93e4161590 2024-04-02T07:52:46.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93eae00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e415f830 con 0x7f93e40cf230 2024-04-02T07:52:46.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93eae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e4160080 con 0x7f93e4160f10 2024-04-02T07:52:46.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93e8e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e4161590 0x7f93e40b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:46.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93e8e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e4161590 0x7f93e40b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40654/0 (socket says 172.21.15.89:40654) 2024-04-02T07:52:46.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93e8e00700 1 -- 172.21.15.89:0/1563150170 learned_addr learned my addr 172.21.15.89:0/1563150170 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:46.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93e3400700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93e4160f10 0x7f93e415a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:46.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93e3e00700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93e40cf230 0x7f93e415f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:46.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93e8e00700 1 -- 172.21.15.89:0/1563150170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93e4160f10 msgr2=0x7f93e415a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:46.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93e8e00700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93e4160f10 0x7f93e415a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:46.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93e8e00700 1 -- 172.21.15.89:0/1563150170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93e40cf230 msgr2=0x7f93e415f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:46.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93e8e00700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93e40cf230 0x7f93e415f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:46.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.187+0000 7f93e8e00700 1 -- 172.21.15.89:0/1563150170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d00094a0 con 0x7f93e4161590 2024-04-02T07:52:46.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.188+0000 7f93e3e00700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93e40cf230 0x7f93e415f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:46.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.188+0000 7f93e8e00700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e4161590 0x7f93e40b32c0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00caa0 tx=0x7f93dc00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:46.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.188+0000 7f93dbe00700 1 -- 172.21.15.89:0/1563150170 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc00aa80 con 0x7f93e4161590 2024-04-02T07:52:46.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.188+0000 7f93dbe00700 1 -- 172.21.15.89:0/1563150170 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93dc00abe0 con 0x7f93e4161590 2024-04-02T07:52:46.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.188+0000 7f93eae00700 1 -- 172.21.15.89:0/1563150170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d40094a0 con 0x7f93e4161590 2024-04-02T07:52:46.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.188+0000 7f93dbe00700 1 -- 172.21.15.89:0/1563150170 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc012400 con 0x7f93e4161590 2024-04-02T07:52:46.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.188+0000 7f93eae00700 1 -- 172.21.15.89:0/1563150170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e4160320 con 0x7f93e4161590 2024-04-02T07:52:46.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.189+0000 7f93eae00700 1 -- 172.21.15.89:0/1563150170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e4067720 con 0x7f93e4161590 2024-04-02T07:52:46.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.190+0000 7f93dbe00700 1 -- 172.21.15.89:0/1563150170 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f93dc01b3b0 con 0x7f93e4161590 2024-04-02T07:52:46.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.191+0000 7f93dbe00700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f93c406adb0 0x7f93c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:46.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.191+0000 7f93dbe00700 1 -- 172.21.15.89:0/1563150170 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f93dc088f70 con 0x7f93e4161590 2024-04-02T07:52:46.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.191+0000 7f93e3e00700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f93c406adb0 0x7f93c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:46.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.192+0000 7f93e3e00700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f93c406adb0 0x7f93c406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f93d4009fd0 tx=0x7f93d4008030 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:46.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.195+0000 7f93dbe00700 1 -- 172.21.15.89:0/1563150170 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f93dc0bc580 con 0x7f93e4161590 2024-04-02T07:52:46.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.403+0000 7f93eae00700 1 -- 172.21.15.89:0/1563150170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f93e40b03d0 con 0x7f93e4161590 2024-04-02T07:52:46.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.404+0000 7f93dbe00700 1 -- 172.21.15.89:0/1563150170 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v107) v1 ==== 84+0+65 (secure 0 0 0) 0x7f93dc017060 con 0x7f93e4161590 2024-04-02T07:52:46.406 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:46.406 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:52:46.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:46.406 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T07:52:46.406 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T07:52:46.406 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:46.406 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:52:46.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.406+0000 7f93eae00700 1 -- 172.21.15.89:0/1563150170 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f93c406adb0 msgr2=0x7f93c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:46.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.406+0000 7f93eae00700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f93c406adb0 0x7f93c406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f93d4009fd0 tx=0x7f93d4008030 comp rx=0 tx=0).stop 2024-04-02T07:52:46.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.406+0000 7f93eae00700 1 -- 172.21.15.89:0/1563150170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e4161590 msgr2=0x7f93e40b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:46.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.406+0000 7f93eae00700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e4161590 0x7f93e40b32c0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00caa0 tx=0x7f93dc00ce60 comp rx=0 tx=0).stop 2024-04-02T07:52:46.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.406+0000 7f93eae00700 1 -- 172.21.15.89:0/1563150170 shutdown_connections 2024-04-02T07:52:46.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.406+0000 7f93eae00700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f93c406adb0 0x7f93c406d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:46.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.406+0000 7f93eae00700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93e40cf230 0x7f93e415f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:46.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.406+0000 7f93eae00700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e4161590 0x7f93e40b32c0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:46.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.406+0000 7f93eae00700 1 --2- 172.21.15.89:0/1563150170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93e4160f10 0x7f93e415a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:46.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.406+0000 7f93eae00700 1 -- 172.21.15.89:0/1563150170 >> 172.21.15.89:0/1563150170 conn(0x7f93e408aff0 msgr2=0x7f93e40cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:46.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.406+0000 7f93eae00700 1 -- 172.21.15.89:0/1563150170 shutdown_connections 2024-04-02T07:52:46.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.406+0000 7f93eae00700 1 -- 172.21.15.89:0/1563150170 wait complete. 2024-04-02T07:52:46.426 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T07:52:46.426 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T07:52:46.538 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:52:46.542 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:52:46.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.623+0000 7f63a9e00700 1 Processor -- start 2024-04-02T07:52:46.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.623+0000 7f63a9e00700 1 -- start start 2024-04-02T07:52:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.625+0000 7f63a9e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40b5430 0x7f63a40ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.625+0000 7f63a9e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a41623a0 con 0x7f63a40b4fa0 2024-04-02T07:52:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.625+0000 7f63a9e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a40b58c0 con 0x7f63a415d930 2024-04-02T07:52:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.625+0000 7f63a9e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a41614b0 con 0x7f63a40b5430 2024-04-02T07:52:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.625+0000 7f63a2000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f63a40b4fa0 0x7f63a40ce610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43784/0 (socket says 172.21.15.89:43784) 2024-04-02T07:52:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.625+0000 7f63a2000700 1 -- 172.21.15.89:0/1183209285 learned_addr learned my addr 172.21.15.89:0/1183209285 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.625+0000 7f63a3400700 1 --2- 172.21.15.89:0/1183209285 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40b5430 0x7f63a40ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.625+0000 7f63a1600700 1 -- 172.21.15.89:0/1183209285 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784071083 0 0) 0x7f63a41623a0 con 0x7f63a40b4fa0 2024-04-02T07:52:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.625+0000 7f63a1600700 1 -- 172.21.15.89:0/1183209285 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63880035b0 con 0x7f63a40b4fa0 2024-04-02T07:52:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.625+0000 7f63a1600700 1 -- 172.21.15.89:0/1183209285 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581511406 0 0) 0x7f63a40b58c0 con 0x7f63a415d930 2024-04-02T07:52:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.625+0000 7f63a1600700 1 -- 172.21.15.89:0/1183209285 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a41623a0 con 0x7f63a415d930 2024-04-02T07:52:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.626+0000 7f63a1600700 1 -- 172.21.15.89:0/1183209285 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 334017932 0 0) 0x7f63880035b0 con 0x7f63a40b4fa0 2024-04-02T07:52:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.626+0000 7f63a3400700 1 -- 172.21.15.89:0/1183209285 >> v1:172.21.15.113:6789/0 conn(0x7f63a415d930 legacy=0x7f63a40d3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.626+0000 7f63a3400700 1 -- 172.21.15.89:0/1183209285 >> v1:172.21.15.89:6789/0 conn(0x7f63a40b4fa0 legacy=0x7f63a40ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.626+0000 7f63a3400700 1 -- 172.21.15.89:0/1183209285 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a4161790 con 0x7f63a40b5430 2024-04-02T07:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.626+0000 7f63a3400700 1 --2- 172.21.15.89:0/1183209285 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40b5430 0x7f63a40ced20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f639c009dc0 tx=0x7f639c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb698a1dae32141a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.626+0000 7f63a1600700 1 -- 172.21.15.89:0/1183209285 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c00e070 con 0x7f63a40b5430 2024-04-02T07:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.626+0000 7f63a1600700 1 -- 172.21.15.89:0/1183209285 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c003a20 con 0x7f63a40b5430 2024-04-02T07:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.626+0000 7f63a1600700 1 -- 172.21.15.89:0/1183209285 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6390001670 con 0x7f63a40b5430 2024-04-02T07:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.627+0000 7f63a9e00700 1 -- 172.21.15.89:0/1183209285 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40b5430 msgr2=0x7f63a40ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.627+0000 7f63a9e00700 1 --2- 172.21.15.89:0/1183209285 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40b5430 0x7f63a40ced20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f639c009dc0 tx=0x7f639c00b330 comp rx=0 tx=0).stop 2024-04-02T07:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.627+0000 7f63a9e00700 1 -- 172.21.15.89:0/1183209285 shutdown_connections 2024-04-02T07:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.627+0000 7f63a9e00700 1 --2- 172.21.15.89:0/1183209285 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40b5430 0x7f63a40ced20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.627+0000 7f63a9e00700 1 -- 172.21.15.89:0/1183209285 >> 172.21.15.89:0/1183209285 conn(0x7f63a408aff0 msgr2=0x7f63a40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.627+0000 7f63a9e00700 1 -- 172.21.15.89:0/1183209285 shutdown_connections 2024-04-02T07:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.627+0000 7f63a9e00700 1 -- 172.21.15.89:0/1183209285 wait complete. 2024-04-02T07:52:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.629+0000 7f63a9e00700 1 Processor -- start 2024-04-02T07:52:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.629+0000 7f63a9e00700 1 -- start start 2024-04-02T07:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.629+0000 7f63a9e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63a40b4fa0 0x7f63a41f2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a9e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40b5430 0x7f63a40b8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a415d930 0x7f63a414cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a9e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40b89e0 con 0x7f63a415d930 2024-04-02T07:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a9e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40cd100 con 0x7f63a40b4fa0 2024-04-02T07:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a9e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a414d0e0 con 0x7f63a40b5430 2024-04-02T07:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a415d930 0x7f63a414cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a415d930 0x7f63a414cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40664/0 (socket says 172.21.15.89:40664) 2024-04-02T07:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a3400700 1 -- 172.21.15.89:0/1513807085 learned_addr learned my addr 172.21.15.89:0/1513807085 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a2000700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40b5430 0x7f63a40b8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a3400700 1 -- 172.21.15.89:0/1513807085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40b5430 msgr2=0x7f63a40b8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a2a00700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63a40b4fa0 0x7f63a41f2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a3400700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40b5430 0x7f63a40b8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a3400700 1 -- 172.21.15.89:0/1513807085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63a40b4fa0 msgr2=0x7f63a41f2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a3400700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63a40b4fa0 0x7f63a41f2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a3400700 1 -- 172.21.15.89:0/1513807085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c009a70 con 0x7f63a415d930 2024-04-02T07:52:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a2000700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40b5430 0x7f63a40b8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.630+0000 7f63a2a00700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63a40b4fa0 0x7f63a41f2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.631+0000 7f63a3400700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a415d930 0x7f63a414cba0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f639c009d90 tx=0x7f639c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.631+0000 7f639b400700 1 -- 172.21.15.89:0/1513807085 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c0046f0 con 0x7f63a415d930 2024-04-02T07:52:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.631+0000 7f639b400700 1 -- 172.21.15.89:0/1513807085 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c004850 con 0x7f63a415d930 2024-04-02T07:52:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.631+0000 7f63a9e00700 1 -- 172.21.15.89:0/1513807085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a40b7d60 con 0x7f63a415d930 2024-04-02T07:52:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.631+0000 7f639b400700 1 -- 172.21.15.89:0/1513807085 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c01d670 con 0x7f63a415d930 2024-04-02T07:52:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.631+0000 7f63a9e00700 1 -- 172.21.15.89:0/1513807085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a40b8320 con 0x7f63a415d930 2024-04-02T07:52:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.632+0000 7f63a9e00700 1 -- 172.21.15.89:0/1513807085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a40b8b20 con 0x7f63a415d930 2024-04-02T07:52:46.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.633+0000 7f639b400700 1 -- 172.21.15.89:0/1513807085 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f639c01d7d0 con 0x7f63a415d930 2024-04-02T07:52:46.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.634+0000 7f639b400700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f638406adb0 0x7f638406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:46.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.634+0000 7f639b400700 1 -- 172.21.15.89:0/1513807085 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f639c08b100 con 0x7f63a415d930 2024-04-02T07:52:46.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.634+0000 7f63a2a00700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f638406adb0 0x7f638406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:46.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.635+0000 7f63a2a00700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f638406adb0 0x7f638406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6394007fd0 tx=0x7f6394009040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:46.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.639+0000 7f639b400700 1 -- 172.21.15.89:0/1513807085 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f63a40b8b20 con 0x7f63a415d930 2024-04-02T07:52:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:46.843+0000 7f63a9e00700 1 -- 172.21.15.89:0/1513807085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f63a40b8b20 con 0x7f63a415d930 2024-04-02T07:52:47.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.465+0000 7f639b400700 1 -- 172.21.15.89:0/1513807085 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v108) v1 ==== 138+0+0 (secure 0 0 0) 0x7f63a40b8b20 con 0x7f63a415d930 2024-04-02T07:52:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.469+0000 7f63a9e00700 1 -- 172.21.15.89:0/1513807085 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f638406adb0 msgr2=0x7f638406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.469+0000 7f63a9e00700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f638406adb0 0x7f638406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6394007fd0 tx=0x7f6394009040 comp rx=0 tx=0).stop 2024-04-02T07:52:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.469+0000 7f63a9e00700 1 -- 172.21.15.89:0/1513807085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a415d930 msgr2=0x7f63a414cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.469+0000 7f63a9e00700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a415d930 0x7f63a414cba0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f639c009d90 tx=0x7f639c004ad0 comp rx=0 tx=0).stop 2024-04-02T07:52:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.469+0000 7f63a9e00700 1 -- 172.21.15.89:0/1513807085 shutdown_connections 2024-04-02T07:52:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.469+0000 7f63a9e00700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f638406adb0 0x7f638406d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:47.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.469+0000 7f63a9e00700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63a40b4fa0 0x7f63a41f2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:47.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.469+0000 7f63a9e00700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63a415d930 0x7f63a414cba0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:47.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.469+0000 7f63a9e00700 1 --2- 172.21.15.89:0/1513807085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63a40b5430 0x7f63a40b8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:47.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.469+0000 7f63a9e00700 1 -- 172.21.15.89:0/1513807085 >> 172.21.15.89:0/1513807085 conn(0x7f63a408aff0 msgr2=0x7f63a415cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:47.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.469+0000 7f63a9e00700 1 -- 172.21.15.89:0/1513807085 shutdown_connections 2024-04-02T07:52:47.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.469+0000 7f63a9e00700 1 -- 172.21.15.89:0/1513807085 wait complete. 2024-04-02T07:52:47.474 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T07:52:47.494 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T07:52:47.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.732+0000 7f5631200700 1 Processor -- start 2024-04-02T07:52:47.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.732+0000 7f5631200700 1 -- start start 2024-04-02T07:52:47.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.733+0000 7f5631200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f562c15d9d0 0x7f562c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:47.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.734+0000 7f5631200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562c161420 con 0x7f562c0d2540 2024-04-02T07:52:47.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.734+0000 7f5631200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562c161700 con 0x7f562c0d16a0 2024-04-02T07:52:47.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.734+0000 7f5631200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f562c15edc0 con 0x7f562c15d9d0 2024-04-02T07:52:47.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.734+0000 7f5629e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f562c15d9d0 0x7f562c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:47.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.734+0000 7f562a800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f562c0d2540 0x7f562c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43786/0 (socket says 172.21.15.89:43786) 2024-04-02T07:52:47.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.734+0000 7f562a800700 1 -- 172.21.15.89:0/4269870453 learned_addr learned my addr 172.21.15.89:0/4269870453 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:47.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.734+0000 7f5629e00700 1 -- 172.21.15.89:0/4269870453 >> v1:172.21.15.113:6789/0 conn(0x7f562c0d16a0 legacy=0x7f562c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:47.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.734+0000 7f5629e00700 1 -- 172.21.15.89:0/4269870453 >> v1:172.21.15.89:6789/0 conn(0x7f562c0d2540 legacy=0x7f562c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:47.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.734+0000 7f5629e00700 1 -- 172.21.15.89:0/4269870453 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c1619e0 con 0x7f562c15d9d0 2024-04-02T07:52:47.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.734+0000 7f5628a00700 1 -- 172.21.15.89:0/4269870453 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59059349 0 0) 0x7f562c161420 con 0x7f562c0d2540 2024-04-02T07:52:47.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.735+0000 7f5629e00700 1 --2- 172.21.15.89:0/4269870453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c15d9d0 0x7f562c0d3c30 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f561c009840 tx=0x7f561c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f9a4816f57efa7e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:47.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.735+0000 7f5628a00700 1 -- 172.21.15.89:0/4269870453 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c011830 con 0x7f562c15d9d0 2024-04-02T07:52:47.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.735+0000 7f5628a00700 1 -- 172.21.15.89:0/4269870453 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561c011990 con 0x7f562c15d9d0 2024-04-02T07:52:47.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.735+0000 7f5628a00700 1 -- 172.21.15.89:0/4269870453 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c011d20 con 0x7f562c15d9d0 2024-04-02T07:52:47.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.735+0000 7f5631200700 1 -- 172.21.15.89:0/4269870453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c15d9d0 msgr2=0x7f562c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:47.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.735+0000 7f5631200700 1 --2- 172.21.15.89:0/4269870453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c15d9d0 0x7f562c0d3c30 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f561c009840 tx=0x7f561c009b50 comp rx=0 tx=0).stop 2024-04-02T07:52:47.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.735+0000 7f5631200700 1 -- 172.21.15.89:0/4269870453 shutdown_connections 2024-04-02T07:52:47.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.735+0000 7f5631200700 1 --2- 172.21.15.89:0/4269870453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c15d9d0 0x7f562c0d3c30 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:47.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.735+0000 7f5631200700 1 -- 172.21.15.89:0/4269870453 >> 172.21.15.89:0/4269870453 conn(0x7f562c08aff0 msgr2=0x7f562c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:47.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.736+0000 7f5631200700 1 -- 172.21.15.89:0/4269870453 shutdown_connections 2024-04-02T07:52:47.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.736+0000 7f5631200700 1 -- 172.21.15.89:0/4269870453 wait complete. 2024-04-02T07:52:47.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.737+0000 7f5631200700 1 Processor -- start 2024-04-02T07:52:47.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.737+0000 7f5631200700 1 -- start start 2024-04-02T07:52:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.738+0000 7f5631200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f562c0d16a0 0x7f562c1601d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.738+0000 7f5631200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f562c0d2540 0x7f562c0b7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.738+0000 7f5631200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c15d9d0 0x7f562c0b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.738+0000 7f5631200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c1f7970 con 0x7f562c15d9d0 2024-04-02T07:52:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.738+0000 7f5631200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c0b8930 con 0x7f562c0d2540 2024-04-02T07:52:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.738+0000 7f5631200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562c0b8b70 con 0x7f562c0d16a0 2024-04-02T07:52:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.738+0000 7f562a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c15d9d0 0x7f562c0b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.739+0000 7f562a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c15d9d0 0x7f562c0b83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40682/0 (socket says 172.21.15.89:40682) 2024-04-02T07:52:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.739+0000 7f562a800700 1 -- 172.21.15.89:0/77390272 learned_addr learned my addr 172.21.15.89:0/77390272 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.739+0000 7f5629e00700 1 --2- 172.21.15.89:0/77390272 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f562c0d16a0 0x7f562c1601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:47.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.739+0000 7f5629400700 1 --2- 172.21.15.89:0/77390272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f562c0d2540 0x7f562c0b7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:47.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.739+0000 7f562a800700 1 -- 172.21.15.89:0/77390272 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f562c0d16a0 msgr2=0x7f562c1601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:47.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.739+0000 7f562a800700 1 --2- 172.21.15.89:0/77390272 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f562c0d16a0 0x7f562c1601d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:47.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.739+0000 7f562a800700 1 -- 172.21.15.89:0/77390272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f562c0d2540 msgr2=0x7f562c0b7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:47.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.739+0000 7f562a800700 1 --2- 172.21.15.89:0/77390272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f562c0d2540 0x7f562c0b7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:47.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.739+0000 7f562a800700 1 -- 172.21.15.89:0/77390272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c0094a0 con 0x7f562c15d9d0 2024-04-02T07:52:47.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.739+0000 7f562a800700 1 --2- 172.21.15.89:0/77390272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c15d9d0 0x7f562c0b83f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f562400cc70 tx=0x7f562400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:47.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.739+0000 7f5622000700 1 -- 172.21.15.89:0/77390272 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562400aa40 con 0x7f562c15d9d0 2024-04-02T07:52:47.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.740+0000 7f5622000700 1 -- 172.21.15.89:0/77390272 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562400abc0 con 0x7f562c15d9d0 2024-04-02T07:52:47.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.740+0000 7f5631200700 1 -- 172.21.15.89:0/77390272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562c1f6fd0 con 0x7f562c15d9d0 2024-04-02T07:52:47.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.740+0000 7f5622000700 1 -- 172.21.15.89:0/77390272 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56240070d0 con 0x7f562c15d9d0 2024-04-02T07:52:47.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.740+0000 7f5631200700 1 -- 172.21.15.89:0/77390272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562c1f7590 con 0x7f562c15d9d0 2024-04-02T07:52:47.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.741+0000 7f5631200700 1 -- 172.21.15.89:0/77390272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562c0d1ca0 con 0x7f562c15d9d0 2024-04-02T07:52:47.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.742+0000 7f5622000700 1 -- 172.21.15.89:0/77390272 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5624007230 con 0x7f562c15d9d0 2024-04-02T07:52:47.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.743+0000 7f5622000700 1 --2- 172.21.15.89:0/77390272 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f560c06adb0 0x7f560c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:47.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.743+0000 7f5622000700 1 -- 172.21.15.89:0/77390272 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f5624088de0 con 0x7f562c15d9d0 2024-04-02T07:52:47.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.743+0000 7f5629e00700 1 --2- 172.21.15.89:0/77390272 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f560c06adb0 0x7f560c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:47.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.744+0000 7f5629e00700 1 --2- 172.21.15.89:0/77390272 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f560c06adb0 0x7f560c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f561c0090b0 tx=0x7f561c01c040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:47.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.747+0000 7f5622000700 1 -- 172.21.15.89:0/77390272 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5624053f20 con 0x7f562c15d9d0 2024-04-02T07:52:47.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:47.947+0000 7f5631200700 1 -- 172.21.15.89:0/77390272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f562c067e30 con 0x7f562c15d9d0 2024-04-02T07:52:48.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.470+0000 7f5622000700 1 -- 172.21.15.89:0/77390272 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v109) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5624018020 con 0x7f562c15d9d0 2024-04-02T07:52:48.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.476+0000 7f5631200700 1 -- 172.21.15.89:0/77390272 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f560c06adb0 msgr2=0x7f560c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:48.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.476+0000 7f5631200700 1 --2- 172.21.15.89:0/77390272 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f560c06adb0 0x7f560c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f561c0090b0 tx=0x7f561c01c040 comp rx=0 tx=0).stop 2024-04-02T07:52:48.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.476+0000 7f5631200700 1 -- 172.21.15.89:0/77390272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c15d9d0 msgr2=0x7f562c0b83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:48.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.476+0000 7f5631200700 1 --2- 172.21.15.89:0/77390272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c15d9d0 0x7f562c0b83f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f562400cc70 tx=0x7f562400cf80 comp rx=0 tx=0).stop 2024-04-02T07:52:48.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.476+0000 7f5631200700 1 -- 172.21.15.89:0/77390272 shutdown_connections 2024-04-02T07:52:48.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.476+0000 7f5631200700 1 --2- 172.21.15.89:0/77390272 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f560c06adb0 0x7f560c06d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:48.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.476+0000 7f5631200700 1 --2- 172.21.15.89:0/77390272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c15d9d0 0x7f562c0b83f0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:48.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.476+0000 7f5631200700 1 --2- 172.21.15.89:0/77390272 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f562c0d16a0 0x7f562c1601d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:48.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.476+0000 7f5631200700 1 --2- 172.21.15.89:0/77390272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f562c0d2540 0x7f562c0b7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:48.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.476+0000 7f5631200700 1 -- 172.21.15.89:0/77390272 >> 172.21.15.89:0/77390272 conn(0x7f562c08aff0 msgr2=0x7f562c0b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:48.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.478+0000 7f5631200700 1 -- 172.21.15.89:0/77390272 shutdown_connections 2024-04-02T07:52:48.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.479+0000 7f5631200700 1 -- 172.21.15.89:0/77390272 wait complete. 2024-04-02T07:52:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T07:52:48.511 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T07:52:48.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.704+0000 7fedb2600700 1 Processor -- start 2024-04-02T07:52:48.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.704+0000 7fedb2600700 1 -- start start 2024-04-02T07:52:48.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.706+0000 7fedb2600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fedac15dfe0 0x7fedac0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:48.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.706+0000 7fedb2600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0ce670 0x7fedac0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:48.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.706+0000 7fedb2600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac0cb310 con 0x7fedac15a5c0 2024-04-02T07:52:48.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.706+0000 7fedb2600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedac0ceb00 con 0x7fedac15dfe0 2024-04-02T07:52:48.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.706+0000 7fedb2600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedac0711a0 con 0x7fedac0ce670 2024-04-02T07:52:48.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.706+0000 7fedaaa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fedac15dfe0 0x7fedac0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:48.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.707+0000 7fedaaa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fedac15dfe0 0x7fedac0ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40688/0 (socket says 172.21.15.89:40688) 2024-04-02T07:52:48.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.707+0000 7fedaaa00700 1 -- 172.21.15.89:0/4147111527 learned_addr learned my addr 172.21.15.89:0/4147111527 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:48.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.707+0000 7fedabe00700 1 --2- 172.21.15.89:0/4147111527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0ce670 0x7fedac0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:48.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.707+0000 7fedaaa00700 1 -- 172.21.15.89:0/4147111527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0ce670 msgr2=0x7fedac0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:48.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.707+0000 7fedaaa00700 1 --2- 172.21.15.89:0/4147111527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0ce670 0x7fedac0cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:48.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.707+0000 7fedaaa00700 1 -- 172.21.15.89:0/4147111527 >> v1:172.21.15.89:6789/0 conn(0x7fedac15a5c0 legacy=0x7fedac0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:48.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.707+0000 7fedaaa00700 1 -- 172.21.15.89:0/4147111527 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac071480 con 0x7fedac15dfe0 2024-04-02T07:52:48.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.707+0000 7fedaa000700 1 -- 172.21.15.89:0/4147111527 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127728138 0 0) 0x7fedac0cb310 con 0x7fedac15a5c0 2024-04-02T07:52:48.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.707+0000 7fedaaa00700 1 --2- 172.21.15.89:0/4147111527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac15dfe0 0x7fedac0ce130 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fed980098f0 tx=0x7fed98009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9788406c869910e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:48.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.707+0000 7fedaa000700 1 -- 172.21.15.89:0/4147111527 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9800b8a0 con 0x7fedac15dfe0 2024-04-02T07:52:48.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.707+0000 7fedaa000700 1 -- 172.21.15.89:0/4147111527 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9800ba00 con 0x7fedac15dfe0 2024-04-02T07:52:48.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.707+0000 7fedaa000700 1 -- 172.21.15.89:0/4147111527 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9800bd90 con 0x7fedac15dfe0 2024-04-02T07:52:48.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.708+0000 7fedb2600700 1 -- 172.21.15.89:0/4147111527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac15dfe0 msgr2=0x7fedac0ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:48.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.708+0000 7fedb2600700 1 --2- 172.21.15.89:0/4147111527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac15dfe0 0x7fedac0ce130 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fed980098f0 tx=0x7fed98009c00 comp rx=0 tx=0).stop 2024-04-02T07:52:48.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.708+0000 7fedb2600700 1 -- 172.21.15.89:0/4147111527 shutdown_connections 2024-04-02T07:52:48.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.708+0000 7fedb2600700 1 --2- 172.21.15.89:0/4147111527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0ce670 0x7fedac0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:48.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.708+0000 7fedb2600700 1 --2- 172.21.15.89:0/4147111527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac15dfe0 0x7fedac0ce130 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:48.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.708+0000 7fedb2600700 1 -- 172.21.15.89:0/4147111527 >> 172.21.15.89:0/4147111527 conn(0x7fedac0b0d70 msgr2=0x7fedac0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:48.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.708+0000 7fedb2600700 1 -- 172.21.15.89:0/4147111527 shutdown_connections 2024-04-02T07:52:48.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.708+0000 7fedb2600700 1 -- 172.21.15.89:0/4147111527 wait complete. 2024-04-02T07:52:48.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.710+0000 7fedb2600700 1 Processor -- start 2024-04-02T07:52:48.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.710+0000 7fedb2600700 1 -- start start 2024-04-02T07:52:48.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.711+0000 7fedb2600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0ce670 0x7fedac150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:48.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.711+0000 7fedb2600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac15a5c0 0x7fedac1f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:48.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.711+0000 7fedb2600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac15dfe0 0x7fedac0afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:48.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.711+0000 7fedb2600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac0c9790 con 0x7fedac15dfe0 2024-04-02T07:52:48.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.711+0000 7fedb2600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac150aa0 con 0x7fedac15a5c0 2024-04-02T07:52:48.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.711+0000 7fedb2600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedac1590f0 con 0x7fedac0ce670 2024-04-02T07:52:48.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.711+0000 7fedabe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac15dfe0 0x7fedac0afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:48.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.711+0000 7fedabe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac15dfe0 0x7fedac0afd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40702/0 (socket says 172.21.15.89:40702) 2024-04-02T07:52:48.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.711+0000 7fedabe00700 1 -- 172.21.15.89:0/3196409298 learned_addr learned my addr 172.21.15.89:0/3196409298 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:48.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.712+0000 7fedab400700 1 --2- 172.21.15.89:0/3196409298 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0ce670 0x7fedac150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:48.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.712+0000 7fedaaa00700 1 --2- 172.21.15.89:0/3196409298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac15a5c0 0x7fedac1f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:48.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.712+0000 7fedabe00700 1 -- 172.21.15.89:0/3196409298 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0ce670 msgr2=0x7fedac150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:48.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.712+0000 7fedabe00700 1 --2- 172.21.15.89:0/3196409298 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0ce670 0x7fedac150560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:48.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.712+0000 7fedabe00700 1 -- 172.21.15.89:0/3196409298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac15a5c0 msgr2=0x7fedac1f6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:48.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.712+0000 7fedabe00700 1 --2- 172.21.15.89:0/3196409298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac15a5c0 0x7fedac1f6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:48.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.712+0000 7fedabe00700 1 -- 172.21.15.89:0/3196409298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed98009550 con 0x7fedac15dfe0 2024-04-02T07:52:48.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.712+0000 7fedabe00700 1 --2- 172.21.15.89:0/3196409298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac15dfe0 0x7fedac0afd30 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7feda4007e40 tx=0x7feda400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:48.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.712+0000 7feda3400700 1 -- 172.21.15.89:0/3196409298 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda400eea0 con 0x7fedac15dfe0 2024-04-02T07:52:48.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.712+0000 7feda3400700 1 -- 172.21.15.89:0/3196409298 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda4014420 con 0x7fedac15dfe0 2024-04-02T07:52:48.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.712+0000 7feda3400700 1 -- 172.21.15.89:0/3196409298 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda40147b0 con 0x7fedac15dfe0 2024-04-02T07:52:48.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.712+0000 7fedb2600700 1 -- 172.21.15.89:0/3196409298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedac1593d0 con 0x7fedac15dfe0 2024-04-02T07:52:48.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.712+0000 7fedb2600700 1 -- 172.21.15.89:0/3196409298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedac159990 con 0x7fedac15dfe0 2024-04-02T07:52:48.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.713+0000 7fedb2600700 1 -- 172.21.15.89:0/3196409298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedac067720 con 0x7fedac15dfe0 2024-04-02T07:52:48.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.714+0000 7feda3400700 1 -- 172.21.15.89:0/3196409298 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7feda4014910 con 0x7fedac15dfe0 2024-04-02T07:52:48.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.716+0000 7feda3400700 1 --2- 172.21.15.89:0/3196409298 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fed8c06adb0 0x7fed8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:48.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.716+0000 7feda3400700 1 -- 172.21.15.89:0/3196409298 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5957+0+0 (secure 0 0 0) 0x7feda4089f50 con 0x7fedac15dfe0 2024-04-02T07:52:48.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.716+0000 7fedab400700 1 --2- 172.21.15.89:0/3196409298 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fed8c06adb0 0x7fed8c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:48.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.716+0000 7fedab400700 1 --2- 172.21.15.89:0/3196409298 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fed8c06adb0 0x7fed8c06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fed9c000f60 tx=0x7fed9c009040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:48.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.721+0000 7feda3400700 1 -- 172.21.15.89:0/3196409298 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feda4054e00 con 0x7fedac15dfe0 2024-04-02T07:52:48.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.967+0000 7fedb2600700 1 -- 172.21.15.89:0/3196409298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fedac0b1ad0 con 0x7fedac15dfe0 2024-04-02T07:52:48.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.974+0000 7feda3400700 1 -- 172.21.15.89:0/3196409298 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v45) v1 ==== 174+0+0 (secure 0 0 0) 0x7feda4058420 con 0x7fedac15dfe0 2024-04-02T07:52:48.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.981+0000 7fedb2600700 1 -- 172.21.15.89:0/3196409298 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fed8c06adb0 msgr2=0x7fed8c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:48.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.981+0000 7fedb2600700 1 --2- 172.21.15.89:0/3196409298 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fed8c06adb0 0x7fed8c06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fed9c000f60 tx=0x7fed9c009040 comp rx=0 tx=0).stop 2024-04-02T07:52:48.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.981+0000 7fedb2600700 1 -- 172.21.15.89:0/3196409298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac15dfe0 msgr2=0x7fedac0afd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:48.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.981+0000 7fedb2600700 1 --2- 172.21.15.89:0/3196409298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac15dfe0 0x7fedac0afd30 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7feda4007e40 tx=0x7feda400e3c0 comp rx=0 tx=0).stop 2024-04-02T07:52:48.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.981+0000 7fedb2600700 1 -- 172.21.15.89:0/3196409298 shutdown_connections 2024-04-02T07:52:48.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.981+0000 7fedb2600700 1 --2- 172.21.15.89:0/3196409298 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fed8c06adb0 0x7fed8c06d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:48.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.981+0000 7fedb2600700 1 --2- 172.21.15.89:0/3196409298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac15dfe0 0x7fedac0afd30 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:48.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.981+0000 7fedb2600700 1 --2- 172.21.15.89:0/3196409298 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0ce670 0x7fedac150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:48.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.981+0000 7fedb2600700 1 --2- 172.21.15.89:0/3196409298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac15a5c0 0x7fedac1f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:48.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.981+0000 7fedb2600700 1 -- 172.21.15.89:0/3196409298 >> 172.21.15.89:0/3196409298 conn(0x7fedac0b0d70 msgr2=0x7fedac0cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:48.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.983+0000 7fedb2600700 1 -- 172.21.15.89:0/3196409298 shutdown_connections 2024-04-02T07:52:48.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:48.983+0000 7fedb2600700 1 -- 172.21.15.89:0/3196409298 wait complete. 2024-04-02T07:52:48.994 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 12 and data pool 13 2024-04-02T07:52:49.020 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:52:49.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.208+0000 7fee91600700 1 Processor -- start 2024-04-02T07:52:49.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.209+0000 7fee91600700 1 -- start start 2024-04-02T07:52:49.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.210+0000 7fee91600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fee8c15d5d0 0x7fee8c0d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.210+0000 7fee91600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee8c0d2dc0 con 0x7fee8c0d2870 2024-04-02T07:52:49.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.210+0000 7fee91600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee8c0d30a0 con 0x7fee8c0d0a00 2024-04-02T07:52:49.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.210+0000 7fee91600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee8c15d9b0 con 0x7fee8c15d5d0 2024-04-02T07:52:49.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.210+0000 7fee8b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fee8c0d2870 0x7fee8c0d02f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33266/0 (socket says 172.21.15.89:33266) 2024-04-02T07:52:49.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.210+0000 7fee8b400700 1 -- 172.21.15.89:0/1052462880 learned_addr learned my addr 172.21.15.89:0/1052462880 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:49.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.210+0000 7fee8be00700 1 --2- 172.21.15.89:0/1052462880 >> v2:172.21.15.113:3300/0 conn(0x7fee8c15d5d0 0x7fee8c0d3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.210+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059582920 0 0) 0x7fee8c0d2dc0 con 0x7fee8c0d2870 2024-04-02T07:52:49.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.210+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee700035b0 con 0x7fee8c0d2870 2024-04-02T07:52:49.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62877335 0 0) 0x7fee8c0d30a0 con 0x7fee8c0d0a00 2024-04-02T07:52:49.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee8c0d2dc0 con 0x7fee8c0d0a00 2024-04-02T07:52:49.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2664649404 0 0) 0x7fee700035b0 con 0x7fee8c0d2870 2024-04-02T07:52:49.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 >> v2:172.21.15.113:3300/0 conn(0x7fee8c15d5d0 msgr2=0x7fee8c0d3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:49.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 --2- 172.21.15.89:0/1052462880 >> v2:172.21.15.113:3300/0 conn(0x7fee8c15d5d0 0x7fee8c0d3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 >> v1:172.21.15.113:6789/0 conn(0x7fee8c0d0a00 legacy=0x7fee8c0d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:49.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c0d3380 con 0x7fee8c0d2870 2024-04-02T07:52:49.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fee78001670 con 0x7fee8c0d2870 2024-04-02T07:52:49.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 >> v1:172.21.15.89:6789/0 conn(0x7fee8c0d2870 legacy=0x7fee8c0d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:49.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 --2- 172.21.15.89:0/1052462880 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fee70003be0 0x7fee70003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 --2- 172.21.15.89:0/1052462880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee70004850 0x7fee70004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 --2- 172.21.15.89:0/1052462880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee70005210 0x7fee7000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c15d9b0 con 0x7fee70005210 2024-04-02T07:52:49.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7000c0e0 con 0x7fee70004850 2024-04-02T07:52:49.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee7000c2a0 con 0x7fee70003be0 2024-04-02T07:52:49.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee90c00700 1 --2- 172.21.15.89:0/1052462880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee70005210 0x7fee7000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8be00700 1 --2- 172.21.15.89:0/1052462880 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fee70003be0 0x7fee70003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.211+0000 7fee8b400700 1 --2- 172.21.15.89:0/1052462880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee70004850 0x7fee70004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.212+0000 7fee90c00700 1 -- 172.21.15.89:0/1052462880 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fee70003be0 msgr2=0x7fee70003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.212+0000 7fee90c00700 1 --2- 172.21.15.89:0/1052462880 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fee70003be0 0x7fee70003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.212+0000 7fee90c00700 1 -- 172.21.15.89:0/1052462880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee70004850 msgr2=0x7fee70004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.212+0000 7fee90c00700 1 --2- 172.21.15.89:0/1052462880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee70004850 0x7fee70004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.212+0000 7fee90c00700 1 -- 172.21.15.89:0/1052462880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee700035b0 con 0x7fee70005210 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.212+0000 7fee90c00700 1 -- 172.21.15.89:0/1052462880 reap_dead start 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.212+0000 7fee90c00700 1 --2- 172.21.15.89:0/1052462880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee70005210 0x7fee7000ba10 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fee8400c2f0 tx=0x7fee70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6010b74652d592e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.212+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84001f30 con 0x7fee70005210 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.212+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee84003a00 con 0x7fee70005210 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.212+0000 7fee8aa00700 1 -- 172.21.15.89:0/1052462880 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84003db0 con 0x7fee70005210 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.213+0000 7fee91600700 1 -- 172.21.15.89:0/1052462880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee70005210 msgr2=0x7fee7000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.213+0000 7fee91600700 1 --2- 172.21.15.89:0/1052462880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee70005210 0x7fee7000ba10 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fee8400c2f0 tx=0x7fee70005880 comp rx=0 tx=0).stop 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.213+0000 7fee91600700 1 -- 172.21.15.89:0/1052462880 shutdown_connections 2024-04-02T07:52:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.213+0000 7fee91600700 1 --2- 172.21.15.89:0/1052462880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee70005210 0x7fee7000ba10 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.213+0000 7fee91600700 1 -- 172.21.15.89:0/1052462880 >> 172.21.15.89:0/1052462880 conn(0x7fee8c0668e0 msgr2=0x7fee8c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:49.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.213+0000 7fee91600700 1 -- 172.21.15.89:0/1052462880 shutdown_connections 2024-04-02T07:52:49.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.213+0000 7fee91600700 1 -- 172.21.15.89:0/1052462880 wait complete. 2024-04-02T07:52:49.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.215+0000 7fee91600700 1 Processor -- start 2024-04-02T07:52:49.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.215+0000 7fee91600700 1 -- start start 2024-04-02T07:52:49.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee91600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee70005210 0x7fee8c160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee91600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fee8c150fc0 0x7fee8c155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee91600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee8c1592e0 0x7fee8c155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee91600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c160ad0 con 0x7fee8c1592e0 2024-04-02T07:52:49.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee91600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c158f30 con 0x7fee70005210 2024-04-02T07:52:49.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee91600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c159c50 con 0x7fee8c150fc0 2024-04-02T07:52:49.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee90c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee8c1592e0 0x7fee8c155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee90c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee8c1592e0 0x7fee8c155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46442/0 (socket says 172.21.15.89:46442) 2024-04-02T07:52:49.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee90c00700 1 -- 172.21.15.89:0/2687522826 learned_addr learned my addr 172.21.15.89:0/2687522826 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:49.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee8b400700 1 --2- 172.21.15.89:0/2687522826 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fee8c150fc0 0x7fee8c155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee8be00700 1 --2- 172.21.15.89:0/2687522826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee70005210 0x7fee8c160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee90c00700 1 -- 172.21.15.89:0/2687522826 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fee8c150fc0 msgr2=0x7fee8c155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:49.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee90c00700 1 --2- 172.21.15.89:0/2687522826 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fee8c150fc0 0x7fee8c155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee90c00700 1 -- 172.21.15.89:0/2687522826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee70005210 msgr2=0x7fee8c160440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:49.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee90c00700 1 --2- 172.21.15.89:0/2687522826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee70005210 0x7fee8c160440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee90c00700 1 -- 172.21.15.89:0/2687522826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c158690 con 0x7fee8c1592e0 2024-04-02T07:52:49.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.216+0000 7fee90c00700 1 --2- 172.21.15.89:0/2687522826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee8c1592e0 0x7fee8c155990 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fee8c0b8f30 tx=0x7fee84001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:49.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.217+0000 7fee83e00700 1 -- 172.21.15.89:0/2687522826 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84005270 con 0x7fee8c1592e0 2024-04-02T07:52:49.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.217+0000 7fee83e00700 1 -- 172.21.15.89:0/2687522826 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee840053d0 con 0x7fee8c1592e0 2024-04-02T07:52:49.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.217+0000 7fee83e00700 1 -- 172.21.15.89:0/2687522826 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee840057c0 con 0x7fee8c1592e0 2024-04-02T07:52:49.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.217+0000 7fee91600700 1 -- 172.21.15.89:0/2687522826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8c158970 con 0x7fee8c1592e0 2024-04-02T07:52:49.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.217+0000 7fee91600700 1 -- 172.21.15.89:0/2687522826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8c15daf0 con 0x7fee8c1592e0 2024-04-02T07:52:49.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.218+0000 7fee91600700 1 -- 172.21.15.89:0/2687522826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8c0b2860 con 0x7fee8c1592e0 2024-04-02T07:52:49.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.219+0000 7fee83e00700 1 -- 172.21.15.89:0/2687522826 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fee8401e470 con 0x7fee8c1592e0 2024-04-02T07:52:49.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.219+0000 7fee83e00700 1 --2- 172.21.15.89:0/2687522826 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fee6c06adb0 0x7fee6c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.220+0000 7fee83e00700 1 -- 172.21.15.89:0/2687522826 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fee8408aee0 con 0x7fee8c1592e0 2024-04-02T07:52:49.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.220+0000 7fee8be00700 1 --2- 172.21.15.89:0/2687522826 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fee6c06adb0 0x7fee6c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.220+0000 7fee8be00700 1 --2- 172.21.15.89:0/2687522826 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fee6c06adb0 0x7fee6c06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fee7c009b20 tx=0x7fee7c008040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:49.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.224+0000 7fee83e00700 1 -- 172.21.15.89:0/2687522826 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fee84055f60 con 0x7fee8c1592e0 2024-04-02T07:52:49.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.426+0000 7fee91600700 1 -- 172.21.15.89:0/2687522826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee8c0b2340 con 0x7fee8c1592e0 2024-04-02T07:52:49.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.427+0000 7fee83e00700 1 -- 172.21.15.89:0/2687522826 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16655 (secure 0 0 0) 0x7fee8c0b2340 con 0x7fee8c1592e0 2024-04-02T07:52:49.430 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:49.430 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":110,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:52:48.969057+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:52:46.885836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-02T07:52:47.989359+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:52:49.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.430+0000 7fee91600700 1 -- 172.21.15.89:0/2687522826 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fee6c06adb0 msgr2=0x7fee6c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:49.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.430+0000 7fee91600700 1 --2- 172.21.15.89:0/2687522826 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fee6c06adb0 0x7fee6c06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fee7c009b20 tx=0x7fee7c008040 comp rx=0 tx=0).stop 2024-04-02T07:52:49.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.430+0000 7fee91600700 1 -- 172.21.15.89:0/2687522826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee8c1592e0 msgr2=0x7fee8c155990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:49.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.430+0000 7fee91600700 1 --2- 172.21.15.89:0/2687522826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee8c1592e0 0x7fee8c155990 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fee8c0b8f30 tx=0x7fee84001650 comp rx=0 tx=0).stop 2024-04-02T07:52:49.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.430+0000 7fee91600700 1 -- 172.21.15.89:0/2687522826 shutdown_connections 2024-04-02T07:52:49.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.430+0000 7fee91600700 1 --2- 172.21.15.89:0/2687522826 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fee6c06adb0 0x7fee6c06d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.430+0000 7fee91600700 1 --2- 172.21.15.89:0/2687522826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee70005210 0x7fee8c160440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.430+0000 7fee91600700 1 --2- 172.21.15.89:0/2687522826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee8c1592e0 0x7fee8c155990 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.430+0000 7fee91600700 1 --2- 172.21.15.89:0/2687522826 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fee8c150fc0 0x7fee8c155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.430+0000 7fee91600700 1 -- 172.21.15.89:0/2687522826 >> 172.21.15.89:0/2687522826 conn(0x7fee8c0668e0 msgr2=0x7fee8c0b68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:49.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.430+0000 7fee91600700 1 -- 172.21.15.89:0/2687522826 shutdown_connections 2024-04-02T07:52:49.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.430+0000 7fee91600700 1 -- 172.21.15.89:0/2687522826 wait complete. 2024-04-02T07:52:49.451 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:52:49.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.642+0000 7ff860600700 1 Processor -- start 2024-04-02T07:52:49.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.642+0000 7ff860600700 1 -- start start 2024-04-02T07:52:49.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.643+0000 7ff860600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8580ceec0 0x7ff8581612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.644+0000 7ff860600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff858161800 0x7ff8580b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.644+0000 7ff860600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8580d30d0 con 0x7ff85815d640 2024-04-02T07:52:49.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.644+0000 7ff860600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff858161c90 con 0x7ff858161800 2024-04-02T07:52:49.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.644+0000 7ff860600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8580b6840 con 0x7ff8580ceec0 2024-04-02T07:52:49.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.644+0000 7ff85dc00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff85815d640 0x7ff8580d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33272/0 (socket says 172.21.15.89:33272) 2024-04-02T07:52:49.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.644+0000 7ff85dc00700 1 -- 172.21.15.89:0/1641247577 learned_addr learned my addr 172.21.15.89:0/1641247577 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:49.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.644+0000 7ff85e600700 1 --2- 172.21.15.89:0/1641247577 >> v2:172.21.15.113:3300/0 conn(0x7ff858161800 0x7ff8580b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.644+0000 7ff85d200700 1 --2- 172.21.15.89:0/1641247577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8580ceec0 0x7ff8581612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.644+0000 7ff857e00700 1 -- 172.21.15.89:0/1641247577 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222847730 0 0) 0x7ff8580d30d0 con 0x7ff85815d640 2024-04-02T07:52:49.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.644+0000 7ff857e00700 1 -- 172.21.15.89:0/1641247577 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8400035b0 con 0x7ff85815d640 2024-04-02T07:52:49.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff857e00700 1 -- 172.21.15.89:0/1641247577 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2053178327 0 0) 0x7ff8400035b0 con 0x7ff85815d640 2024-04-02T07:52:49.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff857e00700 1 -- 172.21.15.89:0/1641247577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8580ceec0 msgr2=0x7ff8581612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:49.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff857e00700 1 --2- 172.21.15.89:0/1641247577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8580ceec0 0x7ff8581612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff857e00700 1 -- 172.21.15.89:0/1641247577 >> v2:172.21.15.113:3300/0 conn(0x7ff858161800 msgr2=0x7ff8580b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:49.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff857e00700 1 --2- 172.21.15.89:0/1641247577 >> v2:172.21.15.113:3300/0 conn(0x7ff858161800 0x7ff8580b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff857e00700 1 -- 172.21.15.89:0/1641247577 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8580d2b30 con 0x7ff85815d640 2024-04-02T07:52:49.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff857e00700 1 -- 172.21.15.89:0/1641247577 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7ff84c001670 con 0x7ff85815d640 2024-04-02T07:52:49.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff85e600700 1 --2- 172.21.15.89:0/1641247577 >> v2:172.21.15.113:3300/0 conn(0x7ff858161800 0x7ff8580b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:49.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff857e00700 1 -- 172.21.15.89:0/1641247577 >> v1:172.21.15.89:6789/0 conn(0x7ff85815d640 legacy=0x7ff8580d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:49.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff857e00700 1 --2- 172.21.15.89:0/1641247577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff840003a00 0x7ff840003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff857e00700 1 --2- 172.21.15.89:0/1641247577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff840004850 0x7ff840004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff857e00700 1 --2- 172.21.15.89:0/1641247577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff840005210 0x7ff84000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff857e00700 1 -- 172.21.15.89:0/1641247577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff858161c90 con 0x7ff840004850 2024-04-02T07:52:49.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff857e00700 1 -- 172.21.15.89:0/1641247577 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8580b6840 con 0x7ff840003a00 2024-04-02T07:52:49.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff857e00700 1 -- 172.21.15.89:0/1641247577 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84000bf80 con 0x7ff840005210 2024-04-02T07:52:49.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff85d200700 1 --2- 172.21.15.89:0/1641247577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8580ceec0 0x7ff8581612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:49.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff85dc00700 1 --2- 172.21.15.89:0/1641247577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff840003a00 0x7ff840003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff85d200700 1 --2- 172.21.15.89:0/1641247577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff840004850 0x7ff840004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.645+0000 7ff85e600700 1 --2- 172.21.15.89:0/1641247577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff840005210 0x7ff84000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.646+0000 7ff85d200700 1 -- 172.21.15.89:0/1641247577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff840005210 msgr2=0x7ff84000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:49.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.646+0000 7ff85d200700 1 --2- 172.21.15.89:0/1641247577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff840005210 0x7ff84000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.646+0000 7ff85d200700 1 -- 172.21.15.89:0/1641247577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff840003a00 msgr2=0x7ff840003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:49.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.646+0000 7ff85d200700 1 --2- 172.21.15.89:0/1641247577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff840003a00 0x7ff840003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.646+0000 7ff85d200700 1 -- 172.21.15.89:0/1641247577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8400035b0 con 0x7ff840004850 2024-04-02T07:52:49.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.646+0000 7ff85d200700 1 --2- 172.21.15.89:0/1641247577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff840004850 0x7ff840004cd0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7ff858161220 tx=0x7ff840005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b42fc70488af80c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:49.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.646+0000 7ff857e00700 1 -- 172.21.15.89:0/1641247577 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844004030 con 0x7ff840004850 2024-04-02T07:52:49.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.646+0000 7ff857e00700 1 -- 172.21.15.89:0/1641247577 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff844004190 con 0x7ff840004850 2024-04-02T07:52:49.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.646+0000 7ff857e00700 1 -- 172.21.15.89:0/1641247577 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844004540 con 0x7ff840004850 2024-04-02T07:52:49.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.646+0000 7ff860600700 1 -- 172.21.15.89:0/1641247577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff840004850 msgr2=0x7ff840004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:49.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.646+0000 7ff860600700 1 --2- 172.21.15.89:0/1641247577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff840004850 0x7ff840004cd0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7ff858161220 tx=0x7ff840005880 comp rx=0 tx=0).stop 2024-04-02T07:52:49.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.646+0000 7ff85e600700 1 -- 172.21.15.89:0/1641247577 reap_dead start 2024-04-02T07:52:49.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.647+0000 7ff860600700 1 -- 172.21.15.89:0/1641247577 shutdown_connections 2024-04-02T07:52:49.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.647+0000 7ff860600700 1 -- 172.21.15.89:0/1641247577 >> 172.21.15.89:0/1641247577 conn(0x7ff85808aff0 msgr2=0x7ff8580d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:49.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.647+0000 7ff860600700 1 -- 172.21.15.89:0/1641247577 shutdown_connections 2024-04-02T07:52:49.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.647+0000 7ff860600700 1 -- 172.21.15.89:0/1641247577 wait complete. 2024-04-02T07:52:49.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.648+0000 7ff860600700 1 Processor -- start 2024-04-02T07:52:49.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.649+0000 7ff860600700 1 -- start start 2024-04-02T07:52:49.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.649+0000 7ff860600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff840004850 0x7ff85814d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.649+0000 7ff860600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8580cdc00 0x7ff85814dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.649+0000 7ff860600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff85814e1c0 0x7ff8581f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.649+0000 7ff860600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85814bc30 con 0x7ff840004850 2024-04-02T07:52:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.649+0000 7ff860600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8580ce4b0 con 0x7ff8580cdc00 2024-04-02T07:52:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.649+0000 7ff860600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8580ce1d0 con 0x7ff85814e1c0 2024-04-02T07:52:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.650+0000 7ff85dc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff840004850 0x7ff85814d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.650+0000 7ff85dc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff840004850 0x7ff85814d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46454/0 (socket says 172.21.15.89:46454) 2024-04-02T07:52:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.650+0000 7ff85dc00700 1 -- 172.21.15.89:0/1475902535 learned_addr learned my addr 172.21.15.89:0/1475902535 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.650+0000 7ff85d200700 1 --2- 172.21.15.89:0/1475902535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8580cdc00 0x7ff85814dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.650+0000 7ff85e600700 1 --2- 172.21.15.89:0/1475902535 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff85814e1c0 0x7ff8581f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.650+0000 7ff85dc00700 1 -- 172.21.15.89:0/1475902535 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff85814e1c0 msgr2=0x7ff8581f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.650+0000 7ff85dc00700 1 --2- 172.21.15.89:0/1475902535 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff85814e1c0 0x7ff8581f2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.650+0000 7ff85dc00700 1 -- 172.21.15.89:0/1475902535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8580cdc00 msgr2=0x7ff85814dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.650+0000 7ff85dc00700 1 --2- 172.21.15.89:0/1475902535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8580cdc00 0x7ff85814dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.650+0000 7ff85dc00700 1 -- 172.21.15.89:0/1475902535 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85814b570 con 0x7ff840004850 2024-04-02T07:52:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.650+0000 7ff85dc00700 1 --2- 172.21.15.89:0/1475902535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff840004850 0x7ff85814d740 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7ff84c007310 tx=0x7ff84c007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.650+0000 7ff855600700 1 -- 172.21.15.89:0/1475902535 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c00f660 con 0x7ff840004850 2024-04-02T07:52:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.651+0000 7ff855600700 1 -- 172.21.15.89:0/1475902535 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c00f820 con 0x7ff840004850 2024-04-02T07:52:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.651+0000 7ff860600700 1 -- 172.21.15.89:0/1475902535 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85814b850 con 0x7ff840004850 2024-04-02T07:52:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.651+0000 7ff855600700 1 -- 172.21.15.89:0/1475902535 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c00fbd0 con 0x7ff840004850 2024-04-02T07:52:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.651+0000 7ff860600700 1 -- 172.21.15.89:0/1475902535 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8581f2690 con 0x7ff840004850 2024-04-02T07:52:49.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.652+0000 7ff860600700 1 -- 172.21.15.89:0/1475902535 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85814bd70 con 0x7ff840004850 2024-04-02T07:52:49.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.653+0000 7ff855600700 1 -- 172.21.15.89:0/1475902535 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff84c00fd30 con 0x7ff840004850 2024-04-02T07:52:49.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.653+0000 7ff855600700 1 --2- 172.21.15.89:0/1475902535 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff83806adb0 0x7ff83806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:49.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.654+0000 7ff855600700 1 -- 172.21.15.89:0/1475902535 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff84c089580 con 0x7ff840004850 2024-04-02T07:52:49.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.654+0000 7ff85d200700 1 --2- 172.21.15.89:0/1475902535 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff83806adb0 0x7ff83806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:49.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.654+0000 7ff85d200700 1 --2- 172.21.15.89:0/1475902535 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff83806adb0 0x7ff83806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff858161220 tx=0x7ff844008070 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:49.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.658+0000 7ff855600700 1 -- 172.21.15.89:0/1475902535 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff85814bd70 con 0x7ff840004850 2024-04-02T07:52:49.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.857+0000 7ff860600700 1 -- 172.21.15.89:0/1475902535 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff85814bd70 con 0x7ff840004850 2024-04-02T07:52:49.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.859+0000 7ff855600700 1 -- 172.21.15.89:0/1475902535 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16655 (secure 0 0 0) 0x7ff85814bd70 con 0x7ff840004850 2024-04-02T07:52:49.861 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:49.861 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":110,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:52:48.969057+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:52:46.885836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-02T07:52:47.989359+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:52:49.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.861+0000 7ff860600700 1 -- 172.21.15.89:0/1475902535 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff83806adb0 msgr2=0x7ff83806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:49.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.861+0000 7ff860600700 1 --2- 172.21.15.89:0/1475902535 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff83806adb0 0x7ff83806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff858161220 tx=0x7ff844008070 comp rx=0 tx=0).stop 2024-04-02T07:52:49.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.861+0000 7ff860600700 1 -- 172.21.15.89:0/1475902535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff840004850 msgr2=0x7ff85814d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:49.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.861+0000 7ff860600700 1 --2- 172.21.15.89:0/1475902535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff840004850 0x7ff85814d740 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7ff84c007310 tx=0x7ff84c007620 comp rx=0 tx=0).stop 2024-04-02T07:52:49.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.861+0000 7ff860600700 1 -- 172.21.15.89:0/1475902535 shutdown_connections 2024-04-02T07:52:49.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.861+0000 7ff860600700 1 --2- 172.21.15.89:0/1475902535 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff83806adb0 0x7ff83806d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.861+0000 7ff860600700 1 --2- 172.21.15.89:0/1475902535 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff85814e1c0 0x7ff8581f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.861+0000 7ff860600700 1 --2- 172.21.15.89:0/1475902535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff840004850 0x7ff85814d740 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.861+0000 7ff860600700 1 --2- 172.21.15.89:0/1475902535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8580cdc00 0x7ff85814dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:49.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.861+0000 7ff860600700 1 -- 172.21.15.89:0/1475902535 >> 172.21.15.89:0/1475902535 conn(0x7ff85808aff0 msgr2=0x7ff8580d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:49.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.861+0000 7ff860600700 1 -- 172.21.15.89:0/1475902535 shutdown_connections 2024-04-02T07:52:49.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:49.861+0000 7ff860600700 1 -- 172.21.15.89:0/1475902535 wait complete. 2024-04-02T07:52:49.882 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T07:52:50.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.075+0000 7f55a4e00700 1 Processor -- start 2024-04-02T07:52:50.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.075+0000 7f55a4e00700 1 -- start start 2024-04-02T07:52:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.076+0000 7f55a4e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f55a015d9d0 0x7f55a00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.076+0000 7f55a4e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f55a00d2540 0x7f55a00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.076+0000 7f55a4e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a0161420 con 0x7f55a00d16a0 2024-04-02T07:52:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.076+0000 7f55a4e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a015edc0 con 0x7f55a015d9d0 2024-04-02T07:52:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.076+0000 7f55a4e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a0161700 con 0x7f55a00d2540 2024-04-02T07:52:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.077+0000 7f559de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f55a015d9d0 0x7f55a00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.077+0000 7f559de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f55a015d9d0 0x7f55a00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46470/0 (socket says 172.21.15.89:46470) 2024-04-02T07:52:50.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.077+0000 7f559de00700 1 -- 172.21.15.89:0/2263929978 learned_addr learned my addr 172.21.15.89:0/2263929978 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:50.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.077+0000 7f559e800700 1 --2- 172.21.15.89:0/2263929978 >> v2:172.21.15.113:3300/0 conn(0x7f55a00d2540 0x7f55a00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:50.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.077+0000 7f559de00700 1 -- 172.21.15.89:0/2263929978 >> v2:172.21.15.113:3300/0 conn(0x7f55a00d2540 msgr2=0x7f55a00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:50.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.077+0000 7f559de00700 1 --2- 172.21.15.89:0/2263929978 >> v2:172.21.15.113:3300/0 conn(0x7f55a00d2540 0x7f55a00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:50.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.077+0000 7f559de00700 1 -- 172.21.15.89:0/2263929978 >> v1:172.21.15.113:6789/0 conn(0x7f55a00d16a0 legacy=0x7f55a00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:50.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.077+0000 7f559de00700 1 -- 172.21.15.89:0/2263929978 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a01619e0 con 0x7f55a015d9d0 2024-04-02T07:52:50.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.077+0000 7f559e800700 1 --2- 172.21.15.89:0/2263929978 >> v2:172.21.15.113:3300/0 conn(0x7f55a00d2540 0x7f55a00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:50.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.077+0000 7f559de00700 1 --2- 172.21.15.89:0/2263929978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a015d9d0 0x7f55a00d3c30 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f55a008ca10 tx=0x7f5590009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=421b0026ece30908 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:50.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.078+0000 7f559ca00700 1 -- 172.21.15.89:0/2263929978 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5590004230 con 0x7f55a015d9d0 2024-04-02T07:52:50.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.078+0000 7f559ca00700 1 -- 172.21.15.89:0/2263929978 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5590004390 con 0x7f55a015d9d0 2024-04-02T07:52:50.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.078+0000 7f559ca00700 1 -- 172.21.15.89:0/2263929978 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55900036a0 con 0x7f55a015d9d0 2024-04-02T07:52:50.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.078+0000 7f55a4e00700 1 -- 172.21.15.89:0/2263929978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a015d9d0 msgr2=0x7f55a00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:50.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.078+0000 7f55a4e00700 1 --2- 172.21.15.89:0/2263929978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a015d9d0 0x7f55a00d3c30 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f55a008ca10 tx=0x7f5590009bd0 comp rx=0 tx=0).stop 2024-04-02T07:52:50.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.078+0000 7f55a4e00700 1 -- 172.21.15.89:0/2263929978 shutdown_connections 2024-04-02T07:52:50.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.078+0000 7f55a4e00700 1 --2- 172.21.15.89:0/2263929978 >> v2:172.21.15.113:3300/0 conn(0x7f55a00d2540 0x7f55a00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:50.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.078+0000 7f55a4e00700 1 --2- 172.21.15.89:0/2263929978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a015d9d0 0x7f55a00d3c30 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:50.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.078+0000 7f55a4e00700 1 -- 172.21.15.89:0/2263929978 >> 172.21.15.89:0/2263929978 conn(0x7f55a008aff0 msgr2=0x7f55a00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:50.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.078+0000 7f55a4e00700 1 -- 172.21.15.89:0/2263929978 shutdown_connections 2024-04-02T07:52:50.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.078+0000 7f55a4e00700 1 -- 172.21.15.89:0/2263929978 wait complete. 2024-04-02T07:52:50.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.080+0000 7f55a4e00700 1 Processor -- start 2024-04-02T07:52:50.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.080+0000 7f55a4e00700 1 -- start start 2024-04-02T07:52:50.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.081+0000 7f55a4e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55a00d16a0 0x7f55a00b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:50.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.081+0000 7f55a4e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a00d2540 0x7f55a00b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:50.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.081+0000 7f55a4e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a015d9d0 0x7f55a01e8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:50.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.081+0000 7f55a4e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a00cb150 con 0x7f55a00d2540 2024-04-02T07:52:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.081+0000 7f55a4e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a00b7fc0 con 0x7f55a00d16a0 2024-04-02T07:52:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.081+0000 7f55a4e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a01e7840 con 0x7f55a015d9d0 2024-04-02T07:52:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.081+0000 7f559d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a00d2540 0x7f55a00b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.081+0000 7f559d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a00d2540 0x7f55a00b07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46482/0 (socket says 172.21.15.89:46482) 2024-04-02T07:52:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.081+0000 7f559d400700 1 -- 172.21.15.89:0/2164918149 learned_addr learned my addr 172.21.15.89:0/2164918149 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.081+0000 7f559de00700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55a00d16a0 0x7f55a00b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.081+0000 7f559e800700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a015d9d0 0x7f55a01e8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.082+0000 7f559d400700 1 -- 172.21.15.89:0/2164918149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a015d9d0 msgr2=0x7f55a01e8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.082+0000 7f559d400700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a015d9d0 0x7f55a01e8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.082+0000 7f559d400700 1 -- 172.21.15.89:0/2164918149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55a00d16a0 msgr2=0x7f55a00b7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.082+0000 7f559d400700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55a00d16a0 0x7f55a00b7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.082+0000 7f559d400700 1 -- 172.21.15.89:0/2164918149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5590009550 con 0x7f55a00d2540 2024-04-02T07:52:50.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.082+0000 7f559e800700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a015d9d0 0x7f55a01e8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.082+0000 7f559d400700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a00d2540 0x7f55a00b07d0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f558c00c480 tx=0x7f558c00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.082+0000 7f5596000700 1 -- 172.21.15.89:0/2164918149 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c011790 con 0x7f55a00d2540 2024-04-02T07:52:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.082+0000 7f5596000700 1 -- 172.21.15.89:0/2164918149 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558c011910 con 0x7f55a00d2540 2024-04-02T07:52:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.082+0000 7f55a4e00700 1 -- 172.21.15.89:0/2164918149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a01e7b20 con 0x7f55a00d2540 2024-04-02T07:52:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.082+0000 7f5596000700 1 -- 172.21.15.89:0/2164918149 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c011cc0 con 0x7f55a00d2540 2024-04-02T07:52:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.082+0000 7f55a4e00700 1 -- 172.21.15.89:0/2164918149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a01e80e0 con 0x7f55a00d2540 2024-04-02T07:52:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.083+0000 7f55a4e00700 1 -- 172.21.15.89:0/2164918149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a00d1ca0 con 0x7f55a00d2540 2024-04-02T07:52:50.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.084+0000 7f5596000700 1 -- 172.21.15.89:0/2164918149 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f558c012420 con 0x7f55a00d2540 2024-04-02T07:52:50.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.085+0000 7f5596000700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f558006ade0 0x7f558006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:50.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.085+0000 7f5596000700 1 -- 172.21.15.89:0/2164918149 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f558c0891d0 con 0x7f55a00d2540 2024-04-02T07:52:50.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.086+0000 7f559de00700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f558006ade0 0x7f558006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:50.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.086+0000 7f559de00700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f558006ade0 0x7f558006d2a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5590009130 tx=0x7f5590008030 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:50.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.089+0000 7f5596000700 1 -- 172.21.15.89:0/2164918149 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f558c0540d0 con 0x7f55a00d2540 2024-04-02T07:52:50.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.333+0000 7f55a4e00700 1 -- 172.21.15.89:0/2164918149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f55a0067720 con 0x7f55a00d2540 2024-04-02T07:52:50.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.980+0000 7f5596000700 1 -- 172.21.15.89:0/2164918149 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v48) v1 ==== 118+0+0 (secure 0 0 0) 0x7f558c0576f0 con 0x7f55a00d2540 2024-04-02T07:52:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.984+0000 7f55a4e00700 1 -- 172.21.15.89:0/2164918149 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f558006ade0 msgr2=0x7f558006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.984+0000 7f55a4e00700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f558006ade0 0x7f558006d2a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5590009130 tx=0x7f5590008030 comp rx=0 tx=0).stop 2024-04-02T07:52:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.984+0000 7f55a4e00700 1 -- 172.21.15.89:0/2164918149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a00d2540 msgr2=0x7f55a00b07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.984+0000 7f55a4e00700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a00d2540 0x7f55a00b07d0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f558c00c480 tx=0x7f558c00c790 comp rx=0 tx=0).stop 2024-04-02T07:52:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.984+0000 7f55a4e00700 1 -- 172.21.15.89:0/2164918149 shutdown_connections 2024-04-02T07:52:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.984+0000 7f55a4e00700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f558006ade0 0x7f558006d2a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.984+0000 7f55a4e00700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55a00d16a0 0x7f55a00b7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.984+0000 7f55a4e00700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a015d9d0 0x7f55a01e8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.984+0000 7f55a4e00700 1 --2- 172.21.15.89:0/2164918149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a00d2540 0x7f55a00b07d0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.984+0000 7f55a4e00700 1 -- 172.21.15.89:0/2164918149 >> 172.21.15.89:0/2164918149 conn(0x7f55a008aff0 msgr2=0x7f55a00b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.984+0000 7f55a4e00700 1 -- 172.21.15.89:0/2164918149 shutdown_connections 2024-04-02T07:52:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:50.984+0000 7f55a4e00700 1 -- 172.21.15.89:0/2164918149 wait complete. 2024-04-02T07:52:51.006 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T07:52:51.006 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T07:52:51.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.201+0000 7f3723200700 1 Processor -- start 2024-04-02T07:52:51.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.201+0000 7f3723200700 1 -- start start 2024-04-02T07:52:51.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.202+0000 7f3723200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f371c0d16a0 0x7f371c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:51.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.202+0000 7f3723200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371c161420 con 0x7f371c0d2370 2024-04-02T07:52:51.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.202+0000 7f3723200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371c161700 con 0x7f371c15d9d0 2024-04-02T07:52:51.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.202+0000 7f3723200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f371c0d4340 con 0x7f371c0d16a0 2024-04-02T07:52:51.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.202+0000 7f3721200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f371c0d2370 0x7f371c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33282/0 (socket says 172.21.15.89:33282) 2024-04-02T07:52:51.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.202+0000 7f3721200700 1 -- 172.21.15.89:0/1344867017 learned_addr learned my addr 172.21.15.89:0/1344867017 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:51.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371b400700 1 --2- 172.21.15.89:0/1344867017 >> v2:172.21.15.113:3300/0 conn(0x7f371c0d16a0 0x7f371c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:51.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 -- 172.21.15.89:0/1344867017 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319750140 0 0) 0x7f371c161420 con 0x7f371c0d2370 2024-04-02T07:52:51.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 -- 172.21.15.89:0/1344867017 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37000035b0 con 0x7f371c0d2370 2024-04-02T07:52:51.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 -- 172.21.15.89:0/1344867017 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 458160895 0 0) 0x7f37000035b0 con 0x7f371c0d2370 2024-04-02T07:52:51.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 -- 172.21.15.89:0/1344867017 >> v2:172.21.15.113:3300/0 conn(0x7f371c0d16a0 msgr2=0x7f371c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:51.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 --2- 172.21.15.89:0/1344867017 >> v2:172.21.15.113:3300/0 conn(0x7f371c0d16a0 0x7f371c0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:51.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 -- 172.21.15.89:0/1344867017 >> v1:172.21.15.113:6789/0 conn(0x7f371c15d9d0 legacy=0x7f371c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:51.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 -- 172.21.15.89:0/1344867017 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371c1619e0 con 0x7f371c0d2370 2024-04-02T07:52:51.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 -- 172.21.15.89:0/1344867017 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f37140019f0 con 0x7f371c0d2370 2024-04-02T07:52:51.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 -- 172.21.15.89:0/1344867017 >> v1:172.21.15.89:6789/0 conn(0x7f371c0d2370 legacy=0x7f371c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:51.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 --2- 172.21.15.89:0/1344867017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3700003a00 0x7f3700003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:51.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 --2- 172.21.15.89:0/1344867017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3700004400 0x7f37000048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:51.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 --2- 172.21.15.89:0/1344867017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37000050d0 0x7f370000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:51.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 -- 172.21.15.89:0/1344867017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370000c000 con 0x7f3700003a00 2024-04-02T07:52:51.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 -- 172.21.15.89:0/1344867017 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370000c170 con 0x7f37000050d0 2024-04-02T07:52:51.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371aa00700 1 -- 172.21.15.89:0/1344867017 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370000c330 con 0x7f3700004400 2024-04-02T07:52:51.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.203+0000 7f371be00700 1 --2- 172.21.15.89:0/1344867017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3700003a00 0x7f3700003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:51.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.204+0000 7f3721200700 1 --2- 172.21.15.89:0/1344867017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37000050d0 0x7f370000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:51.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.204+0000 7f371b400700 1 --2- 172.21.15.89:0/1344867017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3700004400 0x7f37000048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:51.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.204+0000 7f371be00700 1 -- 172.21.15.89:0/1344867017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3700004400 msgr2=0x7f37000048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:51.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.204+0000 7f371be00700 1 --2- 172.21.15.89:0/1344867017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3700004400 0x7f37000048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:51.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.204+0000 7f371be00700 1 -- 172.21.15.89:0/1344867017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37000050d0 msgr2=0x7f370000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:51.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.204+0000 7f371be00700 1 --2- 172.21.15.89:0/1344867017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37000050d0 0x7f370000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:51.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.204+0000 7f371be00700 1 -- 172.21.15.89:0/1344867017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37000035b0 con 0x7f3700003a00 2024-04-02T07:52:51.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.204+0000 7f3721200700 1 --2- 172.21.15.89:0/1344867017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37000050d0 0x7f370000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:51.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.204+0000 7f3721200700 1 -- 172.21.15.89:0/1344867017 reap_dead start 2024-04-02T07:52:51.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.204+0000 7f371be00700 1 --2- 172.21.15.89:0/1344867017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3700003a00 0x7f3700003de0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f370c0018d0 tx=0x7f3700005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af6480dd212688b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:51.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.204+0000 7f371aa00700 1 -- 172.21.15.89:0/1344867017 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c0052d0 con 0x7f3700003a00 2024-04-02T07:52:51.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.204+0000 7f371aa00700 1 -- 172.21.15.89:0/1344867017 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370c005430 con 0x7f3700003a00 2024-04-02T07:52:51.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.204+0000 7f371aa00700 1 -- 172.21.15.89:0/1344867017 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c0038c0 con 0x7f3700003a00 2024-04-02T07:52:51.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.205+0000 7f3723200700 1 -- 172.21.15.89:0/1344867017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3700003a00 msgr2=0x7f3700003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:51.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.205+0000 7f3723200700 1 --2- 172.21.15.89:0/1344867017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3700003a00 0x7f3700003de0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f370c0018d0 tx=0x7f3700005880 comp rx=0 tx=0).stop 2024-04-02T07:52:51.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.205+0000 7f3723200700 1 -- 172.21.15.89:0/1344867017 shutdown_connections 2024-04-02T07:52:51.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.205+0000 7f3723200700 1 --2- 172.21.15.89:0/1344867017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3700003a00 0x7f3700003de0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:51.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.205+0000 7f3723200700 1 -- 172.21.15.89:0/1344867017 >> 172.21.15.89:0/1344867017 conn(0x7f371c08aff0 msgr2=0x7f371c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:51.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.205+0000 7f3723200700 1 -- 172.21.15.89:0/1344867017 shutdown_connections 2024-04-02T07:52:51.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.205+0000 7f3723200700 1 -- 172.21.15.89:0/1344867017 wait complete. 2024-04-02T07:52:51.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.207+0000 7f3723200700 1 Processor -- start 2024-04-02T07:52:51.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.207+0000 7f3723200700 1 -- start start 2024-04-02T07:52:51.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f3723200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3700003a00 0x7f371c160510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:51.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f3723200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f371c159740 0x7f371c0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:51.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f3723200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f371c0cd060 0x7f371c0cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:51.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f3723200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371c0cda20 con 0x7f3700003a00 2024-04-02T07:52:51.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f3723200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371c0cb6f0 con 0x7f371c0cd060 2024-04-02T07:52:51.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f3723200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f371c160a50 con 0x7f371c159740 2024-04-02T07:52:51.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f371be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3700003a00 0x7f371c160510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:51.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f371be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3700003a00 0x7f371c160510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46502/0 (socket says 172.21.15.89:46502) 2024-04-02T07:52:51.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f371be00700 1 -- 172.21.15.89:0/335278667 learned_addr learned my addr 172.21.15.89:0/335278667 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:51.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f3721200700 1 --2- 172.21.15.89:0/335278667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f371c0cd060 0x7f371c0cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:51.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f371b400700 1 --2- 172.21.15.89:0/335278667 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f371c159740 0x7f371c0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:51.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f371be00700 1 -- 172.21.15.89:0/335278667 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f371c159740 msgr2=0x7f371c0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:51.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f371be00700 1 --2- 172.21.15.89:0/335278667 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f371c159740 0x7f371c0cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:51.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f371be00700 1 -- 172.21.15.89:0/335278667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f371c0cd060 msgr2=0x7f371c0cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:51.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f371be00700 1 --2- 172.21.15.89:0/335278667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f371c0cd060 0x7f371c0cd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:51.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.208+0000 7f371be00700 1 -- 172.21.15.89:0/335278667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371c0ccc70 con 0x7f3700003a00 2024-04-02T07:52:51.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.209+0000 7f371be00700 1 --2- 172.21.15.89:0/335278667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3700003a00 0x7f371c160510 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f370c00c0e0 tx=0x7f370c001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:51.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.209+0000 7f3713e00700 1 -- 172.21.15.89:0/335278667 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c006a60 con 0x7f3700003a00 2024-04-02T07:52:51.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.209+0000 7f3713e00700 1 -- 172.21.15.89:0/335278667 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370c006bc0 con 0x7f3700003a00 2024-04-02T07:52:51.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.209+0000 7f3723200700 1 -- 172.21.15.89:0/335278667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f371c0cb970 con 0x7f3700003a00 2024-04-02T07:52:51.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.209+0000 7f3713e00700 1 -- 172.21.15.89:0/335278667 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c007020 con 0x7f3700003a00 2024-04-02T07:52:51.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.209+0000 7f3723200700 1 -- 172.21.15.89:0/335278667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f371c0cbf30 con 0x7f3700003a00 2024-04-02T07:52:51.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.210+0000 7f3723200700 1 -- 172.21.15.89:0/335278667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f371c067720 con 0x7f3700003a00 2024-04-02T07:52:51.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.211+0000 7f3713e00700 1 -- 172.21.15.89:0/335278667 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f370c007180 con 0x7f3700003a00 2024-04-02T07:52:51.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.212+0000 7f3713e00700 1 --2- 172.21.15.89:0/335278667 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f36fc06adb0 0x7f36fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:51.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.212+0000 7f3713e00700 1 -- 172.21.15.89:0/335278667 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f370c08a700 con 0x7f3700003a00 2024-04-02T07:52:51.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.212+0000 7f371b400700 1 --2- 172.21.15.89:0/335278667 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f36fc06adb0 0x7f36fc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:51.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.212+0000 7f371b400700 1 --2- 172.21.15.89:0/335278667 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f36fc06adb0 0x7f36fc06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f371c0b0760 tx=0x7f3708008040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:51.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.216+0000 7f3713e00700 1 -- 172.21.15.89:0/335278667 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f370c055660 con 0x7f3700003a00 2024-04-02T07:52:51.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.469+0000 7f3723200700 1 -- 172.21.15.89:0/335278667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f371c08d690 con 0x7f3700003a00 2024-04-02T07:52:51.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.988+0000 7f3713e00700 1 -- 172.21.15.89:0/335278667 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v49) v1 ==== 122+0+0 (secure 0 0 0) 0x7f370c058c80 con 0x7f3700003a00 2024-04-02T07:52:51.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.990+0000 7f3723200700 1 -- 172.21.15.89:0/335278667 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f36fc06adb0 msgr2=0x7f36fc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:51.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.990+0000 7f3723200700 1 --2- 172.21.15.89:0/335278667 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f36fc06adb0 0x7f36fc06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f371c0b0760 tx=0x7f3708008040 comp rx=0 tx=0).stop 2024-04-02T07:52:51.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.990+0000 7f3723200700 1 -- 172.21.15.89:0/335278667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3700003a00 msgr2=0x7f371c160510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:51.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.990+0000 7f3723200700 1 --2- 172.21.15.89:0/335278667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3700003a00 0x7f371c160510 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f370c00c0e0 tx=0x7f370c001230 comp rx=0 tx=0).stop 2024-04-02T07:52:51.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.990+0000 7f3723200700 1 -- 172.21.15.89:0/335278667 shutdown_connections 2024-04-02T07:52:51.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.990+0000 7f3723200700 1 --2- 172.21.15.89:0/335278667 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f36fc06adb0 0x7f36fc06d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:51.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.990+0000 7f3723200700 1 --2- 172.21.15.89:0/335278667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f371c0cd060 0x7f371c0cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:51.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.990+0000 7f3723200700 1 --2- 172.21.15.89:0/335278667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3700003a00 0x7f371c160510 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:51.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.990+0000 7f3723200700 1 --2- 172.21.15.89:0/335278667 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f371c159740 0x7f371c0cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:51.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.990+0000 7f3723200700 1 -- 172.21.15.89:0/335278667 >> 172.21.15.89:0/335278667 conn(0x7f371c08aff0 msgr2=0x7f371c0cefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:51.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.991+0000 7f3723200700 1 -- 172.21.15.89:0/335278667 shutdown_connections 2024-04-02T07:52:51.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:51.991+0000 7f3723200700 1 -- 172.21.15.89:0/335278667 wait complete. 2024-04-02T07:52:52.011 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T07:52:52.011 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:52:52.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.200+0000 7f3f44200700 1 Processor -- start 2024-04-02T07:52:52.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.200+0000 7f3f44200700 1 -- start start 2024-04-02T07:52:52.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.201+0000 7f3f44200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3f3c0709c0 0x7f3f3c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:52.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.201+0000 7f3f44200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0b2180 con 0x7f3f3c0b3af0 2024-04-02T07:52:52.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.201+0000 7f3f44200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c15b100 con 0x7f3f3c072580 2024-04-02T07:52:52.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.201+0000 7f3f44200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f3c15b3e0 con 0x7f3f3c0709c0 2024-04-02T07:52:52.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f41800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3f3c0b3af0 0x7f3f3c15a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33296/0 (socket says 172.21.15.89:33296) 2024-04-02T07:52:52.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f41800700 1 -- 172.21.15.89:0/1203658765 learned_addr learned my addr 172.21.15.89:0/1203658765 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:52.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f42200700 1 --2- 172.21.15.89:0/1203658765 >> v2:172.21.15.113:3300/0 conn(0x7f3f3c0709c0 0x7f3f3c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361730502 0 0) 0x7f3f3c0b2180 con 0x7f3f3c0b3af0 2024-04-02T07:52:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f240035b0 con 0x7f3f3c0b3af0 2024-04-02T07:52:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553393061 0 0) 0x7f3f3c15b100 con 0x7f3f3c072580 2024-04-02T07:52:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f3c0b2180 con 0x7f3f3c072580 2024-04-02T07:52:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3180940914 0 0) 0x7f3f240035b0 con 0x7f3f3c0b3af0 2024-04-02T07:52:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 >> v2:172.21.15.113:3300/0 conn(0x7f3f3c0709c0 msgr2=0x7f3f3c070e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f3be00700 1 --2- 172.21.15.89:0/1203658765 >> v2:172.21.15.113:3300/0 conn(0x7f3f3c0709c0 0x7f3f3c070e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 >> v1:172.21.15.113:6789/0 conn(0x7f3f3c072580 legacy=0x7f3f3c0b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c0b23f0 con 0x7f3f3c0b3af0 2024-04-02T07:52:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f3f30001670 con 0x7f3f3c0b3af0 2024-04-02T07:52:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 >> v1:172.21.15.89:6789/0 conn(0x7f3f3c0b3af0 legacy=0x7f3f3c15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:52.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f3be00700 1 --2- 172.21.15.89:0/1203658765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f24003b90 0x7f3f24003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.202+0000 7f3f3be00700 1 --2- 172.21.15.89:0/1203658765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f24004850 0x7f3f24004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f3be00700 1 --2- 172.21.15.89:0/1203658765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f24005210 0x7f3f2400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c15b3e0 con 0x7f3f24003b90 2024-04-02T07:52:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2400c090 con 0x7f3f24004850 2024-04-02T07:52:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2400c250 con 0x7f3f24005210 2024-04-02T07:52:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f41800700 1 --2- 172.21.15.89:0/1203658765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f24003b90 0x7f3f24003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f40e00700 1 --2- 172.21.15.89:0/1203658765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f24004850 0x7f3f24004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f42200700 1 --2- 172.21.15.89:0/1203658765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f24005210 0x7f3f2400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f41800700 1 -- 172.21.15.89:0/1203658765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f24005210 msgr2=0x7f3f2400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f41800700 1 --2- 172.21.15.89:0/1203658765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f24005210 0x7f3f2400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f41800700 1 -- 172.21.15.89:0/1203658765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f24004850 msgr2=0x7f3f24004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f41800700 1 --2- 172.21.15.89:0/1203658765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f24004850 0x7f3f24004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f41800700 1 -- 172.21.15.89:0/1203658765 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f240035b0 con 0x7f3f24003b90 2024-04-02T07:52:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f42200700 1 --2- 172.21.15.89:0/1203658765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f24005210 0x7f3f2400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f40e00700 1 --2- 172.21.15.89:0/1203658765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f24004850 0x7f3f24004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f42200700 1 -- 172.21.15.89:0/1203658765 reap_dead start 2024-04-02T07:52:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.203+0000 7f3f41800700 1 --2- 172.21.15.89:0/1203658765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f24003b90 0x7f3f24003f90 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000c210 tx=0x7f3f24005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3b932d7b8c9714 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.204+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3000cdb0 con 0x7f3f24003b90 2024-04-02T07:52:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.204+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f30001af0 con 0x7f3f24003b90 2024-04-02T07:52:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.204+0000 7f3f3be00700 1 -- 172.21.15.89:0/1203658765 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f30005ba0 con 0x7f3f24003b90 2024-04-02T07:52:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.204+0000 7f3f44200700 1 -- 172.21.15.89:0/1203658765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f24003b90 msgr2=0x7f3f24003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.204+0000 7f3f44200700 1 --2- 172.21.15.89:0/1203658765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f24003b90 0x7f3f24003f90 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000c210 tx=0x7f3f24005880 comp rx=0 tx=0).stop 2024-04-02T07:52:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.204+0000 7f3f44200700 1 -- 172.21.15.89:0/1203658765 shutdown_connections 2024-04-02T07:52:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.204+0000 7f3f44200700 1 --2- 172.21.15.89:0/1203658765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f24003b90 0x7f3f24003f90 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.204+0000 7f3f44200700 1 -- 172.21.15.89:0/1203658765 >> 172.21.15.89:0/1203658765 conn(0x7f3f3c0cf190 msgr2=0x7f3f3c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.204+0000 7f3f44200700 1 -- 172.21.15.89:0/1203658765 shutdown_connections 2024-04-02T07:52:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.204+0000 7f3f44200700 1 -- 172.21.15.89:0/1203658765 wait complete. 2024-04-02T07:52:52.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.206+0000 7f3f44200700 1 Processor -- start 2024-04-02T07:52:52.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.206+0000 7f3f44200700 1 -- start start 2024-04-02T07:52:52.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.207+0000 7f3f44200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f24003b90 0x7f3f3c149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:52.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.207+0000 7f3f44200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f3c098330 0x7f3f3c094080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:52.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.207+0000 7f3f44200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f3c06caf0 0x7f3f3c0945c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:52.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.207+0000 7f3f44200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c148ac0 con 0x7f3f3c098330 2024-04-02T07:52:52.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.207+0000 7f3f44200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c06e8a0 con 0x7f3f24003b90 2024-04-02T07:52:52.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.207+0000 7f3f44200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c148d30 con 0x7f3f3c06caf0 2024-04-02T07:52:52.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.207+0000 7f3f40e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f3c098330 0x7f3f3c094080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:52.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.207+0000 7f3f40e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f3c098330 0x7f3f3c094080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46518/0 (socket says 172.21.15.89:46518) 2024-04-02T07:52:52.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f40e00700 1 -- 172.21.15.89:0/874173702 learned_addr learned my addr 172.21.15.89:0/874173702 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:52.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f42200700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f3c06caf0 0x7f3f3c0945c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:52.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f41800700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f24003b90 0x7f3f3c149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f40e00700 1 -- 172.21.15.89:0/874173702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f3c06caf0 msgr2=0x7f3f3c0945c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f40e00700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f3c06caf0 0x7f3f3c0945c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f40e00700 1 -- 172.21.15.89:0/874173702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f24003b90 msgr2=0x7f3f3c149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f40e00700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f24003b90 0x7f3f3c149890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f40e00700 1 -- 172.21.15.89:0/874173702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c1583f0 con 0x7f3f3c098330 2024-04-02T07:52:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f41800700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f24003b90 0x7f3f3c149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f40e00700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f3c098330 0x7f3f3c094080 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f3f280078e0 tx=0x7f3f2800e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f39600700 1 -- 172.21.15.89:0/874173702 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f280052e0 con 0x7f3f3c098330 2024-04-02T07:52:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f39600700 1 -- 172.21.15.89:0/874173702 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f28005440 con 0x7f3f3c098330 2024-04-02T07:52:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f39600700 1 -- 172.21.15.89:0/874173702 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f280038a0 con 0x7f3f3c098330 2024-04-02T07:52:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f44200700 1 -- 172.21.15.89:0/874173702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c094b00 con 0x7f3f3c098330 2024-04-02T07:52:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.208+0000 7f3f44200700 1 -- 172.21.15.89:0/874173702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c149dd0 con 0x7f3f3c098330 2024-04-02T07:52:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.210+0000 7f3f44200700 1 -- 172.21.15.89:0/874173702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3c067720 con 0x7f3f3c098330 2024-04-02T07:52:52.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.210+0000 7f3f39600700 1 -- 172.21.15.89:0/874173702 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3f28003a00 con 0x7f3f3c098330 2024-04-02T07:52:52.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.211+0000 7f3f39600700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f1c06adb0 0x7f3f1c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:52.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.211+0000 7f3f39600700 1 -- 172.21.15.89:0/874173702 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3f28088a60 con 0x7f3f3c098330 2024-04-02T07:52:52.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.211+0000 7f3f41800700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f1c06adb0 0x7f3f1c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:52.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.212+0000 7f3f41800700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f1c06adb0 0x7f3f1c06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3f30000c00 tx=0x7f3f3000a000 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.216+0000 7f3f39600700 1 -- 172.21.15.89:0/874173702 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3f280539c0 con 0x7f3f3c098330 2024-04-02T07:52:52.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.460+0000 7f3f44200700 1 -- 172.21.15.89:0/874173702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f3c0ad760 con 0x7f3f3c098330 2024-04-02T07:52:52.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.461+0000 7f3f39600700 1 -- 172.21.15.89:0/874173702 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f3f28019090 con 0x7f3f3c098330 2024-04-02T07:52:52.466 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:52.466 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6317,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1566757639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1566757639},{"type":"v1","addr":"172.21.15.113:6835","nonce":1566757639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":6985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4102221782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4102221782},{"type":"v1","addr":"172.21.15.89:6836","nonce":4102221782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3127911401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3127911401},{"type":"v1","addr":"172.21.15.113:6834","nonce":3127911401}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:52:48.969376+0000","modified":"2024-04-02T07:52:51.980647+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7419":{"gid":7419,"name":"a","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2016280781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2016280781},{"type":"v1","addr":"172.21.15.89:6837","nonce":2016280781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-02T07:52:52.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.464+0000 7f3f44200700 1 -- 172.21.15.89:0/874173702 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f1c06adb0 msgr2=0x7f3f1c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:52.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.464+0000 7f3f44200700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f1c06adb0 0x7f3f1c06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3f30000c00 tx=0x7f3f3000a000 comp rx=0 tx=0).stop 2024-04-02T07:52:52.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.464+0000 7f3f44200700 1 -- 172.21.15.89:0/874173702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f3c098330 msgr2=0x7f3f3c094080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:52.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.464+0000 7f3f44200700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f3c098330 0x7f3f3c094080 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f3f280078e0 tx=0x7f3f2800e830 comp rx=0 tx=0).stop 2024-04-02T07:52:52.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.464+0000 7f3f44200700 1 -- 172.21.15.89:0/874173702 shutdown_connections 2024-04-02T07:52:52.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.464+0000 7f3f44200700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f1c06adb0 0x7f3f1c06d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.464+0000 7f3f44200700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f24003b90 0x7f3f3c149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.464+0000 7f3f44200700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f3c06caf0 0x7f3f3c0945c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.464+0000 7f3f44200700 1 --2- 172.21.15.89:0/874173702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f3c098330 0x7f3f3c094080 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.464+0000 7f3f44200700 1 -- 172.21.15.89:0/874173702 >> 172.21.15.89:0/874173702 conn(0x7f3f3c0cf190 msgr2=0x7f3f3c06d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:52.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.464+0000 7f3f44200700 1 -- 172.21.15.89:0/874173702 shutdown_connections 2024-04-02T07:52:52.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.464+0000 7f3f44200700 1 -- 172.21.15.89:0/874173702 wait complete. 2024-04-02T07:52:52.468 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 49 2024-04-02T07:52:52.484 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:52:52.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.672+0000 7f751c800700 1 Processor -- start 2024-04-02T07:52:52.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.672+0000 7f751c800700 1 -- start start 2024-04-02T07:52:52.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.673+0000 7f751c800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f751415d640 0x7f75140d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:52.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.673+0000 7f751c800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f75140ceec0 0x7f75141612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:52.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.673+0000 7f751c800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75140d30d0 con 0x7f7514161800 2024-04-02T07:52:52.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.673+0000 7f751c800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7514161c90 con 0x7f75140ceec0 2024-04-02T07:52:52.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.673+0000 7f751c800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75140b6840 con 0x7f751415d640 2024-04-02T07:52:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.674+0000 7f7519e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f751415d640 0x7f75140d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.674+0000 7f7519400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f75140ceec0 0x7f75141612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.674+0000 7f751a800700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f7514161800 0x7f75140cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:43760/0 (socket says 172.21.15.89:43760) 2024-04-02T07:52:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.674+0000 7f751a800700 1 -- 172.21.15.89:0/3763666142 learned_addr learned my addr 172.21.15.89:0/3763666142 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.674+0000 7f7519e00700 1 -- 172.21.15.89:0/3763666142 >> v2:172.21.15.113:3300/0 conn(0x7f75140ceec0 msgr2=0x7f75141612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.674+0000 7f7519e00700 1 --2- 172.21.15.89:0/3763666142 >> v2:172.21.15.113:3300/0 conn(0x7f75140ceec0 0x7f75141612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.674+0000 7f7519e00700 1 -- 172.21.15.89:0/3763666142 >> v1:172.21.15.113:6789/0 conn(0x7f7514161800 legacy=0x7f75140cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.674+0000 7f7518a00700 1 -- 172.21.15.89:0/3763666142 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693335941 0 0) 0x7f75140d30d0 con 0x7f7514161800 2024-04-02T07:52:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.674+0000 7f7519e00700 1 -- 172.21.15.89:0/3763666142 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75140d2880 con 0x7f751415d640 2024-04-02T07:52:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.674+0000 7f7519400700 1 --2- 172.21.15.89:0/3763666142 >> v2:172.21.15.113:3300/0 conn(0x7f75140ceec0 0x7f75141612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.675+0000 7f7519e00700 1 --2- 172.21.15.89:0/3763666142 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f751415d640 0x7f75140d2340 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7508009840 tx=0x7f7508009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7924e67a669d136b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.675+0000 7f7518a00700 1 -- 172.21.15.89:0/3763666142 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508011830 con 0x7f751415d640 2024-04-02T07:52:52.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.675+0000 7f7518a00700 1 -- 172.21.15.89:0/3763666142 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7508011990 con 0x7f751415d640 2024-04-02T07:52:52.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.675+0000 7f7518a00700 1 -- 172.21.15.89:0/3763666142 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508011d20 con 0x7f751415d640 2024-04-02T07:52:52.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.675+0000 7f751c800700 1 -- 172.21.15.89:0/3763666142 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f751415d640 msgr2=0x7f75140d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:52.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.675+0000 7f751c800700 1 --2- 172.21.15.89:0/3763666142 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f751415d640 0x7f75140d2340 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7508009840 tx=0x7f7508009b50 comp rx=0 tx=0).stop 2024-04-02T07:52:52.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.676+0000 7f751c800700 1 -- 172.21.15.89:0/3763666142 shutdown_connections 2024-04-02T07:52:52.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.676+0000 7f751c800700 1 --2- 172.21.15.89:0/3763666142 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f751415d640 0x7f75140d2340 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.676+0000 7f751c800700 1 --2- 172.21.15.89:0/3763666142 >> v2:172.21.15.113:3300/0 conn(0x7f75140ceec0 0x7f75141612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.676+0000 7f751c800700 1 -- 172.21.15.89:0/3763666142 >> 172.21.15.89:0/3763666142 conn(0x7f751408aff0 msgr2=0x7f75140d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:52.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.676+0000 7f751c800700 1 -- 172.21.15.89:0/3763666142 shutdown_connections 2024-04-02T07:52:52.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.676+0000 7f751c800700 1 -- 172.21.15.89:0/3763666142 wait complete. 2024-04-02T07:52:52.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.677+0000 7f751c800700 1 Processor -- start 2024-04-02T07:52:52.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.678+0000 7f751c800700 1 -- start start 2024-04-02T07:52:52.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.678+0000 7f751c800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75140ceec0 0x7f75141f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:52.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.678+0000 7f751c800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f751415d640 0x7f75141532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.679+0000 7f751c800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7514161800 0x7f7514155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.679+0000 7f751c800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7514158ab0 con 0x7f7514161800 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.679+0000 7f751c800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7514157ea0 con 0x7f75140ceec0 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.679+0000 7f751c800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7514158180 con 0x7f751415d640 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.679+0000 7f751a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7514161800 0x7f7514155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.679+0000 7f7519400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f751415d640 0x7f75141532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.679+0000 7f751a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7514161800 0x7f7514155d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46522/0 (socket says 172.21.15.89:46522) 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.679+0000 7f751a800700 1 -- 172.21.15.89:0/3947131616 learned_addr learned my addr 172.21.15.89:0/3947131616 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.679+0000 7f7519e00700 1 --2- 172.21.15.89:0/3947131616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75140ceec0 0x7f75141f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.679+0000 7f751a800700 1 -- 172.21.15.89:0/3947131616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f751415d640 msgr2=0x7f75141532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.679+0000 7f751a800700 1 --2- 172.21.15.89:0/3947131616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f751415d640 0x7f75141532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.679+0000 7f751a800700 1 -- 172.21.15.89:0/3947131616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75140ceec0 msgr2=0x7f75141f2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.679+0000 7f751a800700 1 --2- 172.21.15.89:0/3947131616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75140ceec0 0x7f75141f2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.679+0000 7f751a800700 1 -- 172.21.15.89:0/3947131616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75040094a0 con 0x7f7514161800 2024-04-02T07:52:52.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.680+0000 7f751a800700 1 --2- 172.21.15.89:0/3947131616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7514161800 0x7f7514155d50 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f751000cca0 tx=0x7f751000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:52.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.680+0000 7f750e000700 1 -- 172.21.15.89:0/3947131616 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751000ab70 con 0x7f7514161800 2024-04-02T07:52:52.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.680+0000 7f750e000700 1 -- 172.21.15.89:0/3947131616 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751000acd0 con 0x7f7514161800 2024-04-02T07:52:52.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.680+0000 7f751c800700 1 -- 172.21.15.89:0/3947131616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75080094a0 con 0x7f7514161800 2024-04-02T07:52:52.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.680+0000 7f750e000700 1 -- 172.21.15.89:0/3947131616 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510007310 con 0x7f7514161800 2024-04-02T07:52:52.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.680+0000 7f751c800700 1 -- 172.21.15.89:0/3947131616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7514158740 con 0x7f7514161800 2024-04-02T07:52:52.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.681+0000 7f751c800700 1 -- 172.21.15.89:0/3947131616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7514067720 con 0x7f7514161800 2024-04-02T07:52:52.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.682+0000 7f750e000700 1 -- 172.21.15.89:0/3947131616 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7510007470 con 0x7f7514161800 2024-04-02T07:52:52.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.683+0000 7f750e000700 1 --2- 172.21.15.89:0/3947131616 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f74f806adb0 0x7f74f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:52.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.683+0000 7f750e000700 1 -- 172.21.15.89:0/3947131616 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7510089400 con 0x7f7514161800 2024-04-02T07:52:52.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.683+0000 7f7519e00700 1 --2- 172.21.15.89:0/3947131616 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f74f806adb0 0x7f74f806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:52.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.683+0000 7f7519e00700 1 --2- 172.21.15.89:0/3947131616 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f74f806adb0 0x7f74f806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7508000c00 tx=0x7f750801c040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:52.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.687+0000 7f750e000700 1 -- 172.21.15.89:0/3947131616 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7510054360 con 0x7f7514161800 2024-04-02T07:52:52.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.889+0000 7f751c800700 1 -- 172.21.15.89:0/3947131616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7514158bf0 con 0x7f7514161800 2024-04-02T07:52:52.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.890+0000 7f750e000700 1 -- 172.21.15.89:0/3947131616 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16642 (secure 0 0 0) 0x7f7510057980 con 0x7f7514161800 2024-04-02T07:52:52.892 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:52.892 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":111,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:52:49.972007+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:52:46.885836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-02T07:52:47.989359+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:52:52.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.893+0000 7f751c800700 1 -- 172.21.15.89:0/3947131616 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f74f806adb0 msgr2=0x7f74f806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:52.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.893+0000 7f751c800700 1 --2- 172.21.15.89:0/3947131616 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f74f806adb0 0x7f74f806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7508000c00 tx=0x7f750801c040 comp rx=0 tx=0).stop 2024-04-02T07:52:52.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.893+0000 7f751c800700 1 -- 172.21.15.89:0/3947131616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7514161800 msgr2=0x7f7514155d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:52.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.893+0000 7f751c800700 1 --2- 172.21.15.89:0/3947131616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7514161800 0x7f7514155d50 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f751000cca0 tx=0x7f751000a060 comp rx=0 tx=0).stop 2024-04-02T07:52:52.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.893+0000 7f751c800700 1 -- 172.21.15.89:0/3947131616 shutdown_connections 2024-04-02T07:52:52.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.893+0000 7f751c800700 1 --2- 172.21.15.89:0/3947131616 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f74f806adb0 0x7f74f806d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.893+0000 7f751c800700 1 --2- 172.21.15.89:0/3947131616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75140ceec0 0x7f75141f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.893+0000 7f751c800700 1 --2- 172.21.15.89:0/3947131616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7514161800 0x7f7514155d50 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.893+0000 7f751c800700 1 --2- 172.21.15.89:0/3947131616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f751415d640 0x7f75141532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:52.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.893+0000 7f751c800700 1 -- 172.21.15.89:0/3947131616 >> 172.21.15.89:0/3947131616 conn(0x7f751408aff0 msgr2=0x7f75140d3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:52.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.893+0000 7f751c800700 1 -- 172.21.15.89:0/3947131616 shutdown_connections 2024-04-02T07:52:52.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:52.893+0000 7f751c800700 1 -- 172.21.15.89:0/3947131616 wait complete. 2024-04-02T07:52:52.913 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T07:52:52.913 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:52:53.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.103+0000 7fa25fa00700 1 Processor -- start 2024-04-02T07:52:53.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.103+0000 7fa25fa00700 1 -- start start 2024-04-02T07:52:53.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.104+0000 7fa25fa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa25815b5d0 0x7fa2580ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:53.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.104+0000 7fa25fa00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25815fd50 con 0x7fa2580ac7d0 2024-04-02T07:52:53.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.104+0000 7fa25fa00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25815f140 con 0x7fa2580cb3f0 2024-04-02T07:52:53.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.104+0000 7fa25fa00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa25815b9b0 con 0x7fa25815b5d0 2024-04-02T07:52:53.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.105+0000 7fa25d000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa25815b5d0 0x7fa2580ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:53.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.105+0000 7fa25d000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa25815b5d0 0x7fa2580ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46526/0 (socket says 172.21.15.89:46526) 2024-04-02T07:52:53.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.105+0000 7fa25d000700 1 -- 172.21.15.89:0/3781928723 learned_addr learned my addr 172.21.15.89:0/3781928723 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:53.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.105+0000 7fa25d000700 1 -- 172.21.15.89:0/3781928723 >> v1:172.21.15.113:6789/0 conn(0x7fa2580cb3f0 legacy=0x7fa2580ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:53.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.105+0000 7fa25d000700 1 -- 172.21.15.89:0/3781928723 >> v1:172.21.15.89:6789/0 conn(0x7fa2580ac7d0 legacy=0x7fa25806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:53.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.105+0000 7fa25d000700 1 -- 172.21.15.89:0/3781928723 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25815f420 con 0x7fa25815b5d0 2024-04-02T07:52:53.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.105+0000 7fa257400700 1 -- 172.21.15.89:0/3781928723 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592572237 0 0) 0x7fa25815fd50 con 0x7fa2580ac7d0 2024-04-02T07:52:53.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.105+0000 7fa25d000700 1 --2- 172.21.15.89:0/3781928723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa25815b5d0 0x7fa2580ae3f0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0098f0 tx=0x7fa24c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f16ed6f18575f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:53.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.105+0000 7fa257400700 1 -- 172.21.15.89:0/3781928723 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c00b8a0 con 0x7fa25815b5d0 2024-04-02T07:52:53.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.106+0000 7fa257400700 1 -- 172.21.15.89:0/3781928723 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24c00ba00 con 0x7fa25815b5d0 2024-04-02T07:52:53.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.106+0000 7fa257400700 1 -- 172.21.15.89:0/3781928723 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c00bd90 con 0x7fa25815b5d0 2024-04-02T07:52:53.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.106+0000 7fa25fa00700 1 -- 172.21.15.89:0/3781928723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa25815b5d0 msgr2=0x7fa2580ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:53.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.106+0000 7fa25fa00700 1 --2- 172.21.15.89:0/3781928723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa25815b5d0 0x7fa2580ae3f0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0098f0 tx=0x7fa24c009c00 comp rx=0 tx=0).stop 2024-04-02T07:52:53.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.106+0000 7fa25fa00700 1 -- 172.21.15.89:0/3781928723 shutdown_connections 2024-04-02T07:52:53.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.106+0000 7fa25fa00700 1 --2- 172.21.15.89:0/3781928723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa25815b5d0 0x7fa2580ae3f0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.106+0000 7fa25fa00700 1 -- 172.21.15.89:0/3781928723 >> 172.21.15.89:0/3781928723 conn(0x7fa2580cf190 msgr2=0x7fa2580cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:53.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.106+0000 7fa25fa00700 1 -- 172.21.15.89:0/3781928723 shutdown_connections 2024-04-02T07:52:53.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.106+0000 7fa25fa00700 1 -- 172.21.15.89:0/3781928723 wait complete. 2024-04-02T07:52:53.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.108+0000 7fa25fa00700 1 Processor -- start 2024-04-02T07:52:53.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.108+0000 7fa25fa00700 1 -- start start 2024-04-02T07:52:53.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.109+0000 7fa25fa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2580ac7d0 0x7fa2580709c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:53.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.109+0000 7fa25fa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa25815b5d0 0x7fa2580ca570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:53.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.109+0000 7fa25fa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2580c9680 0x7fa2581f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:53.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.109+0000 7fa25fa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2580caab0 con 0x7fa25815b5d0 2024-04-02T07:52:53.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.109+0000 7fa25fa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2580ccf10 con 0x7fa2580c9680 2024-04-02T07:52:53.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.109+0000 7fa25fa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2580cd1f0 con 0x7fa2580ac7d0 2024-04-02T07:52:53.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.109+0000 7fa257e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa25815b5d0 0x7fa2580ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:53.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.109+0000 7fa257e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa25815b5d0 0x7fa2580ca570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46530/0 (socket says 172.21.15.89:46530) 2024-04-02T07:52:53.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.109+0000 7fa257e00700 1 -- 172.21.15.89:0/416185991 learned_addr learned my addr 172.21.15.89:0/416185991 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:53.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.109+0000 7fa25da00700 1 --2- 172.21.15.89:0/416185991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2580c9680 0x7fa2581f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:53.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.109+0000 7fa25d000700 1 --2- 172.21.15.89:0/416185991 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2580ac7d0 0x7fa2580709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:53.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.110+0000 7fa257e00700 1 -- 172.21.15.89:0/416185991 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2580ac7d0 msgr2=0x7fa2580709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:53.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.110+0000 7fa257e00700 1 --2- 172.21.15.89:0/416185991 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2580ac7d0 0x7fa2580709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.110+0000 7fa257e00700 1 -- 172.21.15.89:0/416185991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2580c9680 msgr2=0x7fa2581f5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:53.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.110+0000 7fa257e00700 1 --2- 172.21.15.89:0/416185991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2580c9680 0x7fa2581f5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.110+0000 7fa257e00700 1 -- 172.21.15.89:0/416185991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c009550 con 0x7fa25815b5d0 2024-04-02T07:52:53.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.110+0000 7fa257e00700 1 --2- 172.21.15.89:0/416185991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa25815b5d0 0x7fa2580ca570 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fa24400c4e0 tx=0x7fa24400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:53.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.110+0000 7fa254c00700 1 -- 172.21.15.89:0/416185991 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2440052d0 con 0x7fa25815b5d0 2024-04-02T07:52:53.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.110+0000 7fa254c00700 1 -- 172.21.15.89:0/416185991 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa244005430 con 0x7fa25815b5d0 2024-04-02T07:52:53.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.110+0000 7fa25fa00700 1 -- 172.21.15.89:0/416185991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2580cd4d0 con 0x7fa25815b5d0 2024-04-02T07:52:53.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.110+0000 7fa254c00700 1 -- 172.21.15.89:0/416185991 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2440038c0 con 0x7fa25815b5d0 2024-04-02T07:52:53.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.110+0000 7fa25fa00700 1 -- 172.21.15.89:0/416185991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2581f6470 con 0x7fa25815b5d0 2024-04-02T07:52:53.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.111+0000 7fa25fa00700 1 -- 172.21.15.89:0/416185991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2580b0bc0 con 0x7fa25815b5d0 2024-04-02T07:52:53.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.112+0000 7fa254c00700 1 -- 172.21.15.89:0/416185991 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa244003a20 con 0x7fa25815b5d0 2024-04-02T07:52:53.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.113+0000 7fa254c00700 1 --2- 172.21.15.89:0/416185991 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa23806adb0 0x7fa23806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:53.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.113+0000 7fa254c00700 1 -- 172.21.15.89:0/416185991 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa244088ad0 con 0x7fa25815b5d0 2024-04-02T07:52:53.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.113+0000 7fa25d000700 1 --2- 172.21.15.89:0/416185991 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa23806adb0 0x7fa23806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:53.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.114+0000 7fa25d000700 1 --2- 172.21.15.89:0/416185991 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa23806adb0 0x7fa23806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa24c014040 tx=0x7fa24c008030 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:53.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.118+0000 7fa254c00700 1 -- 172.21.15.89:0/416185991 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa244053a30 con 0x7fa25815b5d0 2024-04-02T07:52:53.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.313+0000 7fa25fa00700 1 -- 172.21.15.89:0/416185991 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2580cd080 con 0x7fa23806adb0 2024-04-02T07:52:53.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.323+0000 7fa254c00700 1 -- 172.21.15.89:0/416185991 <== mgr.6775 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289602 (secure 0 0 0) 0x7fa2580cd080 con 0x7fa23806adb0 2024-04-02T07:52:53.328 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:53.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.327+0000 7fa25fa00700 1 -- 172.21.15.89:0/416185991 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa23806adb0 msgr2=0x7fa23806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:53.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.327+0000 7fa25fa00700 1 --2- 172.21.15.89:0/416185991 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa23806adb0 0x7fa23806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa24c014040 tx=0x7fa24c008030 comp rx=0 tx=0).stop 2024-04-02T07:52:53.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.327+0000 7fa25fa00700 1 -- 172.21.15.89:0/416185991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa25815b5d0 msgr2=0x7fa2580ca570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:53.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.327+0000 7fa25fa00700 1 --2- 172.21.15.89:0/416185991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa25815b5d0 0x7fa2580ca570 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fa24400c4e0 tx=0x7fa24400c8a0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.328+0000 7fa25fa00700 1 -- 172.21.15.89:0/416185991 shutdown_connections 2024-04-02T07:52:53.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.328+0000 7fa25fa00700 1 --2- 172.21.15.89:0/416185991 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa23806adb0 0x7fa23806d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.328+0000 7fa25fa00700 1 --2- 172.21.15.89:0/416185991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2580c9680 0x7fa2581f5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.328+0000 7fa25fa00700 1 --2- 172.21.15.89:0/416185991 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2580ac7d0 0x7fa2580709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.328+0000 7fa25fa00700 1 --2- 172.21.15.89:0/416185991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa25815b5d0 0x7fa2580ca570 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.328+0000 7fa25fa00700 1 -- 172.21.15.89:0/416185991 >> 172.21.15.89:0/416185991 conn(0x7fa2580cf190 msgr2=0x7fa25806c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:53.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.328+0000 7fa25fa00700 1 -- 172.21.15.89:0/416185991 shutdown_connections 2024-04-02T07:52:53.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.328+0000 7fa25fa00700 1 -- 172.21.15.89:0/416185991 wait complete. 2024-04-02T07:52:53.332 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:52:53.348 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T07:52:52.861964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":321460,"kb_used_data":11812,"kb_used_omap":12,"kb_used_meta":309491,"kb_avail":754653260,"statfs":{"total":773094113280,"available":772764938240,"internally_reserved":0,"allocated":12095488,"data_stored":6497398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":316919352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.396620"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.010194+0000","last_change":"2024-04-02T07:52:49.008897+0000","last_active":"2024-04-02T07:52:49.010194+0000","last_peered":"2024-04-02T07:52:49.010194+0000","last_clean":"2024-04-02T07:52:49.010194+0000","last_became_active":"2024-04-02T07:52:49.008537+0000","last_became_peered":"2024-04-02T07:52:49.008537+0000","last_unstale":"2024-04-02T07:52:49.010194+0000","last_undegraded":"2024-04-02T07:52:49.010194+0000","last_fullsized":"2024-04-02T07:52:49.010194+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:29.124428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983043+0000","last_change":"2024-04-02T07:52:48.984704+0000","last_active":"2024-04-02T07:52:49.983043+0000","last_peered":"2024-04-02T07:52:49.983043+0000","last_clean":"2024-04-02T07:52:49.983043+0000","last_became_active":"2024-04-02T07:52:48.984443+0000","last_became_peered":"2024-04-02T07:52:48.984443+0000","last_unstale":"2024-04-02T07:52:49.983043+0000","last_undegraded":"2024-04-02T07:52:49.983043+0000","last_fullsized":"2024-04-02T07:52:49.983043+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:22:27.541973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.001271+0000","last_change":"2024-04-02T07:52:48.997229+0000","last_active":"2024-04-02T07:52:49.001271+0000","last_peered":"2024-04-02T07:52:49.001271+0000","last_clean":"2024-04-02T07:52:49.001271+0000","last_became_active":"2024-04-02T07:52:48.996919+0000","last_became_peered":"2024-04-02T07:52:48.996919+0000","last_unstale":"2024-04-02T07:52:49.001271+0000","last_undegraded":"2024-04-02T07:52:49.001271+0000","last_fullsized":"2024-04-02T07:52:49.001271+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:43:13.014987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982719+0000","last_change":"2024-04-02T07:52:48.988005+0000","last_active":"2024-04-02T07:52:49.982719+0000","last_peered":"2024-04-02T07:52:49.982719+0000","last_clean":"2024-04-02T07:52:49.982719+0000","last_became_active":"2024-04-02T07:52:48.987697+0000","last_became_peered":"2024-04-02T07:52:48.987697+0000","last_unstale":"2024-04-02T07:52:49.982719+0000","last_undegraded":"2024-04-02T07:52:49.982719+0000","last_fullsized":"2024-04-02T07:52:49.982719+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:43.696202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.990751+0000","last_change":"2024-04-02T07:52:48.986945+0000","last_active":"2024-04-02T07:52:48.990751+0000","last_peered":"2024-04-02T07:52:48.990751+0000","last_clean":"2024-04-02T07:52:48.990751+0000","last_became_active":"2024-04-02T07:52:48.986619+0000","last_became_peered":"2024-04-02T07:52:48.986619+0000","last_unstale":"2024-04-02T07:52:48.990751+0000","last_undegraded":"2024-04-02T07:52:48.990751+0000","last_fullsized":"2024-04-02T07:52:48.990751+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:43:38.553176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.006991+0000","last_change":"2024-04-02T07:52:49.003644+0000","last_active":"2024-04-02T07:52:49.006991+0000","last_peered":"2024-04-02T07:52:49.006991+0000","last_clean":"2024-04-02T07:52:49.006991+0000","last_became_active":"2024-04-02T07:52:49.003394+0000","last_became_peered":"2024-04-02T07:52:49.003394+0000","last_unstale":"2024-04-02T07:52:49.006991+0000","last_undegraded":"2024-04-02T07:52:49.006991+0000","last_fullsized":"2024-04-02T07:52:49.006991+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:13:13.336296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.981987+0000","last_change":"2024-04-02T07:52:48.981572+0000","last_active":"2024-04-02T07:52:49.981987+0000","last_peered":"2024-04-02T07:52:49.981987+0000","last_clean":"2024-04-02T07:52:49.981987+0000","last_became_active":"2024-04-02T07:52:48.981278+0000","last_became_peered":"2024-04-02T07:52:48.981278+0000","last_unstale":"2024-04-02T07:52:49.981987+0000","last_undegraded":"2024-04-02T07:52:49.981987+0000","last_fullsized":"2024-04-02T07:52:49.981987+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:29.124428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.010943+0000","last_change":"2024-04-02T07:52:49.008419+0000","last_active":"2024-04-02T07:52:49.010943+0000","last_peered":"2024-04-02T07:52:49.010943+0000","last_clean":"2024-04-02T07:52:49.010943+0000","last_became_active":"2024-04-02T07:52:49.008077+0000","last_became_peered":"2024-04-02T07:52:49.008077+0000","last_unstale":"2024-04-02T07:52:49.010943+0000","last_undegraded":"2024-04-02T07:52:49.010943+0000","last_fullsized":"2024-04-02T07:52:49.010943+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:59:01.177649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999866+0000","last_change":"2024-04-02T07:52:48.997488+0000","last_active":"2024-04-02T07:52:48.999866+0000","last_peered":"2024-04-02T07:52:48.999866+0000","last_clean":"2024-04-02T07:52:48.999866+0000","last_became_active":"2024-04-02T07:52:48.995310+0000","last_became_peered":"2024-04-02T07:52:48.995310+0000","last_unstale":"2024-04-02T07:52:48.999866+0000","last_undegraded":"2024-04-02T07:52:48.999866+0000","last_fullsized":"2024-04-02T07:52:48.999866+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:43.695576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.998749+0000","last_change":"2024-04-02T07:52:48.991820+0000","last_active":"2024-04-02T07:52:48.998749+0000","last_peered":"2024-04-02T07:52:48.998749+0000","last_clean":"2024-04-02T07:52:48.998749+0000","last_became_active":"2024-04-02T07:52:48.991514+0000","last_became_peered":"2024-04-02T07:52:48.991514+0000","last_unstale":"2024-04-02T07:52:48.998749+0000","last_undegraded":"2024-04-02T07:52:48.998749+0000","last_fullsized":"2024-04-02T07:52:48.998749+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:53:04.013403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.006028+0000","last_change":"2024-04-02T07:52:49.003710+0000","last_active":"2024-04-02T07:52:49.006028+0000","last_peered":"2024-04-02T07:52:49.006028+0000","last_clean":"2024-04-02T07:52:49.006028+0000","last_became_active":"2024-04-02T07:52:49.003283+0000","last_became_peered":"2024-04-02T07:52:49.003283+0000","last_unstale":"2024-04-02T07:52:49.006028+0000","last_undegraded":"2024-04-02T07:52:49.006028+0000","last_fullsized":"2024-04-02T07:52:49.006028+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:06:03.280191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.006526+0000","last_change":"2024-04-02T07:52:48.991768+0000","last_active":"2024-04-02T07:52:49.006526+0000","last_peered":"2024-04-02T07:52:49.006526+0000","last_clean":"2024-04-02T07:52:49.006526+0000","last_became_active":"2024-04-02T07:52:48.989320+0000","last_became_peered":"2024-04-02T07:52:48.989320+0000","last_unstale":"2024-04-02T07:52:49.006526+0000","last_undegraded":"2024-04-02T07:52:49.006526+0000","last_fullsized":"2024-04-02T07:52:49.006526+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:43.696202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.990525+0000","last_change":"2024-04-02T07:52:48.986001+0000","last_active":"2024-04-02T07:52:48.990525+0000","last_peered":"2024-04-02T07:52:48.990525+0000","last_clean":"2024-04-02T07:52:48.990525+0000","last_became_active":"2024-04-02T07:52:48.985705+0000","last_became_peered":"2024-04-02T07:52:48.985705+0000","last_unstale":"2024-04-02T07:52:48.990525+0000","last_undegraded":"2024-04-02T07:52:48.990525+0000","last_fullsized":"2024-04-02T07:52:48.990525+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:08:11.549996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983567+0000","last_change":"2024-04-02T07:52:48.990501+0000","last_active":"2024-04-02T07:52:49.983567+0000","last_peered":"2024-04-02T07:52:49.983567+0000","last_clean":"2024-04-02T07:52:49.983567+0000","last_became_active":"2024-04-02T07:52:48.990247+0000","last_became_peered":"2024-04-02T07:52:48.990247+0000","last_unstale":"2024-04-02T07:52:49.983567+0000","last_undegraded":"2024-04-02T07:52:49.983567+0000","last_fullsized":"2024-04-02T07:52:49.983567+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:41:35.832935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982096+0000","last_change":"2024-04-02T07:52:48.987296+0000","last_active":"2024-04-02T07:52:49.982096+0000","last_peered":"2024-04-02T07:52:49.982096+0000","last_clean":"2024-04-02T07:52:49.982096+0000","last_became_active":"2024-04-02T07:52:48.986131+0000","last_became_peered":"2024-04-02T07:52:48.986131+0000","last_unstale":"2024-04-02T07:52:49.982096+0000","last_undegraded":"2024-04-02T07:52:49.982096+0000","last_fullsized":"2024-04-02T07:52:49.982096+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:43.695576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.000556+0000","last_change":"2024-04-02T07:52:48.996381+0000","last_active":"2024-04-02T07:52:49.000556+0000","last_peered":"2024-04-02T07:52:49.000556+0000","last_clean":"2024-04-02T07:52:49.000556+0000","last_became_active":"2024-04-02T07:52:48.995157+0000","last_became_peered":"2024-04-02T07:52:48.995157+0000","last_unstale":"2024-04-02T07:52:49.000556+0000","last_undegraded":"2024-04-02T07:52:49.000556+0000","last_fullsized":"2024-04-02T07:52:49.000556+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:18.988774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.998192+0000","last_change":"2024-04-02T07:52:48.991127+0000","last_active":"2024-04-02T07:52:48.998192+0000","last_peered":"2024-04-02T07:52:48.998192+0000","last_clean":"2024-04-02T07:52:48.998192+0000","last_became_active":"2024-04-02T07:52:48.988157+0000","last_became_peered":"2024-04-02T07:52:48.988157+0000","last_unstale":"2024-04-02T07:52:48.998192+0000","last_undegraded":"2024-04-02T07:52:48.998192+0000","last_fullsized":"2024-04-02T07:52:48.998192+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:58.200859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.009404+0000","last_change":"2024-04-02T07:52:49.007749+0000","last_active":"2024-04-02T07:52:49.009404+0000","last_peered":"2024-04-02T07:52:49.009404+0000","last_clean":"2024-04-02T07:52:49.009404+0000","last_became_active":"2024-04-02T07:52:49.007467+0000","last_became_peered":"2024-04-02T07:52:49.007467+0000","last_unstale":"2024-04-02T07:52:49.009404+0000","last_undegraded":"2024-04-02T07:52:49.009404+0000","last_fullsized":"2024-04-02T07:52:49.009404+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:55:04.980012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983908+0000","last_change":"2024-04-02T07:52:49.002468+0000","last_active":"2024-04-02T07:52:49.983908+0000","last_peered":"2024-04-02T07:52:49.983908+0000","last_clean":"2024-04-02T07:52:49.983908+0000","last_became_active":"2024-04-02T07:52:49.001891+0000","last_became_peered":"2024-04-02T07:52:49.001891+0000","last_unstale":"2024-04-02T07:52:49.983908+0000","last_undegraded":"2024-04-02T07:52:49.983908+0000","last_fullsized":"2024-04-02T07:52:49.983908+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:59.387423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983799+0000","last_change":"2024-04-02T07:52:49.001894+0000","last_active":"2024-04-02T07:52:49.983799+0000","last_peered":"2024-04-02T07:52:49.983799+0000","last_clean":"2024-04-02T07:52:49.983799+0000","last_became_active":"2024-04-02T07:52:49.001022+0000","last_became_peered":"2024-04-02T07:52:49.001022+0000","last_unstale":"2024-04-02T07:52:49.983799+0000","last_undegraded":"2024-04-02T07:52:49.983799+0000","last_fullsized":"2024-04-02T07:52:49.983799+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:04:01.924124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982512+0000","last_change":"2024-04-02T07:52:48.990977+0000","last_active":"2024-04-02T07:52:49.982512+0000","last_peered":"2024-04-02T07:52:49.982512+0000","last_clean":"2024-04-02T07:52:49.982512+0000","last_became_active":"2024-04-02T07:52:48.990482+0000","last_became_peered":"2024-04-02T07:52:48.990482+0000","last_unstale":"2024-04-02T07:52:49.982512+0000","last_undegraded":"2024-04-02T07:52:49.982512+0000","last_fullsized":"2024-04-02T07:52:49.982512+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:38.860332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983658+0000","last_change":"2024-04-02T07:52:48.995621+0000","last_active":"2024-04-02T07:52:49.983658+0000","last_peered":"2024-04-02T07:52:49.983658+0000","last_clean":"2024-04-02T07:52:49.983658+0000","last_became_active":"2024-04-02T07:52:48.995157+0000","last_became_peered":"2024-04-02T07:52:48.995157+0000","last_unstale":"2024-04-02T07:52:49.983658+0000","last_undegraded":"2024-04-02T07:52:49.983658+0000","last_fullsized":"2024-04-02T07:52:49.983658+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:41.075062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999254+0000","last_change":"2024-04-02T07:52:48.994369+0000","last_active":"2024-04-02T07:52:48.999254+0000","last_peered":"2024-04-02T07:52:48.999254+0000","last_clean":"2024-04-02T07:52:48.999254+0000","last_became_active":"2024-04-02T07:52:48.989191+0000","last_became_peered":"2024-04-02T07:52:48.989191+0000","last_unstale":"2024-04-02T07:52:48.999254+0000","last_undegraded":"2024-04-02T07:52:48.999254+0000","last_fullsized":"2024-04-02T07:52:48.999254+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:16:27.175143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.005991+0000","last_change":"2024-04-02T07:52:49.003588+0000","last_active":"2024-04-02T07:52:49.005991+0000","last_peered":"2024-04-02T07:52:49.005991+0000","last_clean":"2024-04-02T07:52:49.005991+0000","last_became_active":"2024-04-02T07:52:48.998527+0000","last_became_peered":"2024-04-02T07:52:48.998527+0000","last_unstale":"2024-04-02T07:52:49.005991+0000","last_undegraded":"2024-04-02T07:52:49.005991+0000","last_fullsized":"2024-04-02T07:52:49.005991+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:57:04.305513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.003283+0000","last_change":"2024-04-02T07:52:48.993733+0000","last_active":"2024-04-02T07:52:49.003283+0000","last_peered":"2024-04-02T07:52:49.003283+0000","last_clean":"2024-04-02T07:52:49.003283+0000","last_became_active":"2024-04-02T07:52:48.993476+0000","last_became_peered":"2024-04-02T07:52:48.993476+0000","last_unstale":"2024-04-02T07:52:49.003283+0000","last_undegraded":"2024-04-02T07:52:49.003283+0000","last_fullsized":"2024-04-02T07:52:49.003283+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:11:34.231899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999414+0000","last_change":"2024-04-02T07:52:48.992292+0000","last_active":"2024-04-02T07:52:48.999414+0000","last_peered":"2024-04-02T07:52:48.999414+0000","last_clean":"2024-04-02T07:52:48.999414+0000","last_became_active":"2024-04-02T07:52:48.991798+0000","last_became_peered":"2024-04-02T07:52:48.991798+0000","last_unstale":"2024-04-02T07:52:48.999414+0000","last_undegraded":"2024-04-02T07:52:48.999414+0000","last_fullsized":"2024-04-02T07:52:48.999414+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:32:03.330012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999419+0000","last_change":"2024-04-02T07:52:48.993925+0000","last_active":"2024-04-02T07:52:48.999419+0000","last_peered":"2024-04-02T07:52:48.999419+0000","last_clean":"2024-04-02T07:52:48.999419+0000","last_became_active":"2024-04-02T07:52:48.993481+0000","last_became_peered":"2024-04-02T07:52:48.993481+0000","last_unstale":"2024-04-02T07:52:48.999419+0000","last_undegraded":"2024-04-02T07:52:48.999419+0000","last_fullsized":"2024-04-02T07:52:48.999419+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:13:58.964747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999892+0000","last_change":"2024-04-02T07:52:48.997911+0000","last_active":"2024-04-02T07:52:48.999892+0000","last_peered":"2024-04-02T07:52:48.999892+0000","last_clean":"2024-04-02T07:52:48.999892+0000","last_became_active":"2024-04-02T07:52:48.994626+0000","last_became_peered":"2024-04-02T07:52:48.994626+0000","last_unstale":"2024-04-02T07:52:48.999892+0000","last_undegraded":"2024-04-02T07:52:48.999892+0000","last_fullsized":"2024-04-02T07:52:48.999892+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:53:04.013403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982396+0000","last_change":"2024-04-02T07:52:48.995392+0000","last_active":"2024-04-02T07:52:49.982396+0000","last_peered":"2024-04-02T07:52:49.982396+0000","last_clean":"2024-04-02T07:52:49.982396+0000","last_became_active":"2024-04-02T07:52:48.995081+0000","last_became_peered":"2024-04-02T07:52:48.995081+0000","last_unstale":"2024-04-02T07:52:49.982396+0000","last_undegraded":"2024-04-02T07:52:49.982396+0000","last_fullsized":"2024-04-02T07:52:49.982396+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:36.306309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983656+0000","last_change":"2024-04-02T07:52:48.996745+0000","last_active":"2024-04-02T07:52:49.983656+0000","last_peered":"2024-04-02T07:52:49.983656+0000","last_clean":"2024-04-02T07:52:49.983656+0000","last_became_active":"2024-04-02T07:52:48.996456+0000","last_became_peered":"2024-04-02T07:52:48.996456+0000","last_unstale":"2024-04-02T07:52:49.983656+0000","last_undegraded":"2024-04-02T07:52:49.983656+0000","last_fullsized":"2024-04-02T07:52:49.983656+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:41:00.474374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.997638+0000","last_change":"2024-04-02T07:52:48.992361+0000","last_active":"2024-04-02T07:52:48.997638+0000","last_peered":"2024-04-02T07:52:48.997638+0000","last_clean":"2024-04-02T07:52:48.997638+0000","last_became_active":"2024-04-02T07:52:48.989481+0000","last_became_peered":"2024-04-02T07:52:48.989481+0000","last_unstale":"2024-04-02T07:52:48.997638+0000","last_undegraded":"2024-04-02T07:52:48.997638+0000","last_fullsized":"2024-04-02T07:52:48.997638+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:41.075062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.000884+0000","last_change":"2024-04-02T07:52:48.998651+0000","last_active":"2024-04-02T07:52:49.000884+0000","last_peered":"2024-04-02T07:52:49.000884+0000","last_clean":"2024-04-02T07:52:49.000884+0000","last_became_active":"2024-04-02T07:52:48.998413+0000","last_became_peered":"2024-04-02T07:52:48.998413+0000","last_unstale":"2024-04-02T07:52:49.000884+0000","last_undegraded":"2024-04-02T07:52:49.000884+0000","last_fullsized":"2024-04-02T07:52:49.000884+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:22:27.541973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.985234+0000","last_change":"2024-04-02T07:52:48.982883+0000","last_active":"2024-04-02T07:52:48.985234+0000","last_peered":"2024-04-02T07:52:48.985234+0000","last_clean":"2024-04-02T07:52:48.985234+0000","last_became_active":"2024-04-02T07:52:48.982631+0000","last_became_peered":"2024-04-02T07:52:48.982631+0000","last_unstale":"2024-04-02T07:52:48.985234+0000","last_undegraded":"2024-04-02T07:52:48.985234+0000","last_fullsized":"2024-04-02T07:52:48.985234+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:36.306309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"110'4","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.005532+0000","last_change":"2024-04-02T07:52:48.492212+0000","last_active":"2024-04-02T07:52:49.005532+0000","last_peered":"2024-04-02T07:52:49.005532+0000","last_clean":"2024-04-02T07:52:49.005532+0000","last_became_active":"2024-04-02T07:52:48.491899+0000","last_became_peered":"2024-04-02T07:52:48.491899+0000","last_unstale":"2024-04-02T07:52:49.005532+0000","last_undegraded":"2024-04-02T07:52:49.005532+0000","last_fullsized":"2024-04-02T07:52:49.005532+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:21.397120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.004800+0000","last_change":"2024-04-02T07:52:49.002096+0000","last_active":"2024-04-02T07:52:49.004800+0000","last_peered":"2024-04-02T07:52:49.004800+0000","last_clean":"2024-04-02T07:52:49.004800+0000","last_became_active":"2024-04-02T07:52:49.001857+0000","last_became_peered":"2024-04-02T07:52:49.001857+0000","last_unstale":"2024-04-02T07:52:49.004800+0000","last_undegraded":"2024-04-02T07:52:49.004800+0000","last_fullsized":"2024-04-02T07:52:49.004800+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:41:35.832935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1e","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.010031+0000","last_change":"2024-04-02T07:52:48.494135+0000","last_active":"2024-04-02T07:52:49.010031+0000","last_peered":"2024-04-02T07:52:49.010031+0000","last_clean":"2024-04-02T07:52:49.010031+0000","last_became_active":"2024-04-02T07:52:48.493882+0000","last_became_peered":"2024-04-02T07:52:48.493882+0000","last_unstale":"2024-04-02T07:52:49.010031+0000","last_undegraded":"2024-04-02T07:52:49.010031+0000","last_fullsized":"2024-04-02T07:52:49.010031+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:23:06.484479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.006545+0000","last_change":"2024-04-02T07:52:49.003117+0000","last_active":"2024-04-02T07:52:49.006545+0000","last_peered":"2024-04-02T07:52:49.006545+0000","last_clean":"2024-04-02T07:52:49.006545+0000","last_became_active":"2024-04-02T07:52:48.996406+0000","last_became_peered":"2024-04-02T07:52:48.996406+0000","last_unstale":"2024-04-02T07:52:49.006545+0000","last_undegraded":"2024-04-02T07:52:49.006545+0000","last_fullsized":"2024-04-02T07:52:49.006545+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:22.405783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1d","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.002010+0000","last_change":"2024-04-02T07:52:48.491033+0000","last_active":"2024-04-02T07:52:49.002010+0000","last_peered":"2024-04-02T07:52:49.002010+0000","last_clean":"2024-04-02T07:52:49.002010+0000","last_became_active":"2024-04-02T07:52:48.490465+0000","last_became_peered":"2024-04-02T07:52:48.490465+0000","last_unstale":"2024-04-02T07:52:49.002010+0000","last_undegraded":"2024-04-02T07:52:49.002010+0000","last_fullsized":"2024-04-02T07:52:49.002010+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:43:22.090288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.004136+0000","last_change":"2024-04-02T07:52:49.000838+0000","last_active":"2024-04-02T07:52:49.004136+0000","last_peered":"2024-04-02T07:52:49.004136+0000","last_clean":"2024-04-02T07:52:49.004136+0000","last_became_active":"2024-04-02T07:52:49.000177+0000","last_became_peered":"2024-04-02T07:52:49.000177+0000","last_unstale":"2024-04-02T07:52:49.004136+0000","last_undegraded":"2024-04-02T07:52:49.004136+0000","last_fullsized":"2024-04-02T07:52:49.004136+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:13:13.336296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1c","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982776+0000","last_change":"2024-04-02T07:52:48.495550+0000","last_active":"2024-04-02T07:52:49.982776+0000","last_peered":"2024-04-02T07:52:49.982776+0000","last_clean":"2024-04-02T07:52:49.982776+0000","last_became_active":"2024-04-02T07:52:48.495305+0000","last_became_peered":"2024-04-02T07:52:48.495305+0000","last_unstale":"2024-04-02T07:52:49.982776+0000","last_undegraded":"2024-04-02T07:52:49.982776+0000","last_fullsized":"2024-04-02T07:52:49.982776+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:41:54.857786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982644+0000","last_change":"2024-04-02T07:52:48.994969+0000","last_active":"2024-04-02T07:52:49.982644+0000","last_peered":"2024-04-02T07:52:49.982644+0000","last_clean":"2024-04-02T07:52:49.982644+0000","last_became_active":"2024-04-02T07:52:48.994519+0000","last_became_peered":"2024-04-02T07:52:48.994519+0000","last_unstale":"2024-04-02T07:52:49.982644+0000","last_undegraded":"2024-04-02T07:52:49.982644+0000","last_fullsized":"2024-04-02T07:52:49.982644+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:58.200859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1b","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.009246+0000","last_change":"2024-04-02T07:52:48.489078+0000","last_active":"2024-04-02T07:52:49.009246+0000","last_peered":"2024-04-02T07:52:49.009246+0000","last_clean":"2024-04-02T07:52:49.009246+0000","last_became_active":"2024-04-02T07:52:48.488812+0000","last_became_peered":"2024-04-02T07:52:48.488812+0000","last_unstale":"2024-04-02T07:52:49.009246+0000","last_undegraded":"2024-04-02T07:52:49.009246+0000","last_fullsized":"2024-04-02T07:52:49.009246+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:49:09.118777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.997467+0000","last_change":"2024-04-02T07:52:48.988912+0000","last_active":"2024-04-02T07:52:48.997467+0000","last_peered":"2024-04-02T07:52:48.997467+0000","last_clean":"2024-04-02T07:52:48.997467+0000","last_became_active":"2024-04-02T07:52:48.988649+0000","last_became_peered":"2024-04-02T07:52:48.988649+0000","last_unstale":"2024-04-02T07:52:48.997467+0000","last_undegraded":"2024-04-02T07:52:48.997467+0000","last_fullsized":"2024-04-02T07:52:48.997467+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:21:44.892940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1a","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.008106+0000","last_change":"2024-04-02T07:52:48.490637+0000","last_active":"2024-04-02T07:52:49.008106+0000","last_peered":"2024-04-02T07:52:49.008106+0000","last_clean":"2024-04-02T07:52:49.008106+0000","last_became_active":"2024-04-02T07:52:48.490038+0000","last_became_peered":"2024-04-02T07:52:48.490038+0000","last_unstale":"2024-04-02T07:52:49.008106+0000","last_undegraded":"2024-04-02T07:52:49.008106+0000","last_fullsized":"2024-04-02T07:52:49.008106+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:06:02.271528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.010066+0000","last_change":"2024-04-02T07:52:49.007001+0000","last_active":"2024-04-02T07:52:49.010066+0000","last_peered":"2024-04-02T07:52:49.010066+0000","last_clean":"2024-04-02T07:52:49.010066+0000","last_became_active":"2024-04-02T07:52:49.006738+0000","last_became_peered":"2024-04-02T07:52:49.006738+0000","last_unstale":"2024-04-02T07:52:49.010066+0000","last_undegraded":"2024-04-02T07:52:49.010066+0000","last_fullsized":"2024-04-02T07:52:49.010066+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:41.075062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983467+0000","last_change":"2024-04-02T07:52:48.484996+0000","last_active":"2024-04-02T07:52:49.983467+0000","last_peered":"2024-04-02T07:52:49.983467+0000","last_clean":"2024-04-02T07:52:49.983467+0000","last_became_active":"2024-04-02T07:52:48.484692+0000","last_became_peered":"2024-04-02T07:52:48.484692+0000","last_unstale":"2024-04-02T07:52:49.983467+0000","last_undegraded":"2024-04-02T07:52:49.983467+0000","last_fullsized":"2024-04-02T07:52:49.983467+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:57:46.918091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983420+0000","last_change":"2024-04-02T07:52:48.990452+0000","last_active":"2024-04-02T07:52:49.983420+0000","last_peered":"2024-04-02T07:52:49.983420+0000","last_clean":"2024-04-02T07:52:49.983420+0000","last_became_active":"2024-04-02T07:52:48.990155+0000","last_became_peered":"2024-04-02T07:52:48.990155+0000","last_unstale":"2024-04-02T07:52:49.983420+0000","last_undegraded":"2024-04-02T07:52:49.983420+0000","last_fullsized":"2024-04-02T07:52:49.983420+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:16.097343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983253+0000","last_change":"2024-04-02T07:52:48.495555+0000","last_active":"2024-04-02T07:52:49.983253+0000","last_peered":"2024-04-02T07:52:49.983253+0000","last_clean":"2024-04-02T07:52:49.983253+0000","last_became_active":"2024-04-02T07:52:48.495287+0000","last_became_peered":"2024-04-02T07:52:48.495287+0000","last_unstale":"2024-04-02T07:52:49.983253+0000","last_undegraded":"2024-04-02T07:52:49.983253+0000","last_fullsized":"2024-04-02T07:52:49.983253+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:31:10.337978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983992+0000","last_change":"2024-04-02T07:52:48.998856+0000","last_active":"2024-04-02T07:52:49.983992+0000","last_peered":"2024-04-02T07:52:49.983992+0000","last_clean":"2024-04-02T07:52:49.983992+0000","last_became_active":"2024-04-02T07:52:48.997924+0000","last_became_peered":"2024-04-02T07:52:48.997924+0000","last_unstale":"2024-04-02T07:52:49.983992+0000","last_undegraded":"2024-04-02T07:52:49.983992+0000","last_fullsized":"2024-04-02T07:52:49.983992+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:29.124428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.986991+0000","last_change":"2024-04-02T07:52:48.489345+0000","last_active":"2024-04-02T07:52:48.986991+0000","last_peered":"2024-04-02T07:52:48.986991+0000","last_clean":"2024-04-02T07:52:48.986991+0000","last_became_active":"2024-04-02T07:52:48.488890+0000","last_became_peered":"2024-04-02T07:52:48.488890+0000","last_unstale":"2024-04-02T07:52:48.986991+0000","last_undegraded":"2024-04-02T07:52:48.986991+0000","last_fullsized":"2024-04-02T07:52:48.986991+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:56:18.611413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.007917+0000","last_change":"2024-04-02T07:52:49.004722+0000","last_active":"2024-04-02T07:52:49.007917+0000","last_peered":"2024-04-02T07:52:49.007917+0000","last_clean":"2024-04-02T07:52:49.007917+0000","last_became_active":"2024-04-02T07:52:49.004378+0000","last_became_peered":"2024-04-02T07:52:49.004378+0000","last_unstale":"2024-04-02T07:52:49.007917+0000","last_undegraded":"2024-04-02T07:52:49.007917+0000","last_fullsized":"2024-04-02T07:52:49.007917+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:22:27.541973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.981561+0000","last_change":"2024-04-02T07:52:48.494163+0000","last_active":"2024-04-02T07:52:48.981561+0000","last_peered":"2024-04-02T07:52:48.981561+0000","last_clean":"2024-04-02T07:52:48.981561+0000","last_became_active":"2024-04-02T07:52:48.493111+0000","last_became_peered":"2024-04-02T07:52:48.493111+0000","last_unstale":"2024-04-02T07:52:48.981561+0000","last_undegraded":"2024-04-02T07:52:48.981561+0000","last_fullsized":"2024-04-02T07:52:48.981561+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:57:03.296849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.003552+0000","last_change":"2024-04-02T07:52:48.991086+0000","last_active":"2024-04-02T07:52:49.003552+0000","last_peered":"2024-04-02T07:52:49.003552+0000","last_clean":"2024-04-02T07:52:49.003552+0000","last_became_active":"2024-04-02T07:52:48.990733+0000","last_became_peered":"2024-04-02T07:52:48.990733+0000","last_unstale":"2024-04-02T07:52:49.003552+0000","last_undegraded":"2024-04-02T07:52:49.003552+0000","last_fullsized":"2024-04-02T07:52:49.003552+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:59.387423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.15","version":"110'3","reported_seq":13,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.009476+0000","last_change":"2024-04-02T07:52:48.494720+0000","last_active":"2024-04-02T07:52:49.009476+0000","last_peered":"2024-04-02T07:52:49.009476+0000","last_clean":"2024-04-02T07:52:49.009476+0000","last_became_active":"2024-04-02T07:52:48.494426+0000","last_became_peered":"2024-04-02T07:52:48.494426+0000","last_unstale":"2024-04-02T07:52:49.009476+0000","last_undegraded":"2024-04-02T07:52:49.009476+0000","last_fullsized":"2024-04-02T07:52:49.009476+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:31:10.337978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.997950+0000","last_change":"2024-04-02T07:52:48.991803+0000","last_active":"2024-04-02T07:52:48.997950+0000","last_peered":"2024-04-02T07:52:48.997950+0000","last_clean":"2024-04-02T07:52:48.997950+0000","last_became_active":"2024-04-02T07:52:48.991399+0000","last_became_peered":"2024-04-02T07:52:48.991399+0000","last_unstale":"2024-04-02T07:52:48.997950+0000","last_undegraded":"2024-04-02T07:52:48.997950+0000","last_fullsized":"2024-04-02T07:52:48.997950+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:10:20.627252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.14","version":"110'6","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.012826+0000","last_change":"2024-04-02T07:52:48.485211+0000","last_active":"2024-04-02T07:52:49.012826+0000","last_peered":"2024-04-02T07:52:49.012826+0000","last_clean":"2024-04-02T07:52:49.012826+0000","last_became_active":"2024-04-02T07:52:48.484715+0000","last_became_peered":"2024-04-02T07:52:48.484715+0000","last_unstale":"2024-04-02T07:52:49.012826+0000","last_undegraded":"2024-04-02T07:52:49.012826+0000","last_fullsized":"2024-04-02T07:52:49.012826+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:58:21.014343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.998161+0000","last_change":"2024-04-02T07:52:48.989787+0000","last_active":"2024-04-02T07:52:48.998161+0000","last_peered":"2024-04-02T07:52:48.998161+0000","last_clean":"2024-04-02T07:52:48.998161+0000","last_became_active":"2024-04-02T07:52:48.989488+0000","last_became_peered":"2024-04-02T07:52:48.989488+0000","last_unstale":"2024-04-02T07:52:48.998161+0000","last_undegraded":"2024-04-02T07:52:48.998161+0000","last_fullsized":"2024-04-02T07:52:48.998161+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:05:38.004241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.13","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982615+0000","last_change":"2024-04-02T07:52:48.491042+0000","last_active":"2024-04-02T07:52:49.982615+0000","last_peered":"2024-04-02T07:52:49.982615+0000","last_clean":"2024-04-02T07:52:49.982615+0000","last_became_active":"2024-04-02T07:52:48.490086+0000","last_became_peered":"2024-04-02T07:52:48.490086+0000","last_unstale":"2024-04-02T07:52:49.982615+0000","last_undegraded":"2024-04-02T07:52:49.982615+0000","last_fullsized":"2024-04-02T07:52:49.982615+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:09:28.390077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.002507+0000","last_change":"2024-04-02T07:52:49.001717+0000","last_active":"2024-04-02T07:52:49.002507+0000","last_peered":"2024-04-02T07:52:49.002507+0000","last_clean":"2024-04-02T07:52:49.002507+0000","last_became_active":"2024-04-02T07:52:49.000518+0000","last_became_peered":"2024-04-02T07:52:49.000518+0000","last_unstale":"2024-04-02T07:52:49.002507+0000","last_undegraded":"2024-04-02T07:52:49.002507+0000","last_fullsized":"2024-04-02T07:52:49.002507+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:21:44.892940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982075+0000","last_change":"2024-04-02T07:52:48.493071+0000","last_active":"2024-04-02T07:52:49.982075+0000","last_peered":"2024-04-02T07:52:49.982075+0000","last_clean":"2024-04-02T07:52:49.982075+0000","last_became_active":"2024-04-02T07:52:48.492221+0000","last_became_peered":"2024-04-02T07:52:48.492221+0000","last_unstale":"2024-04-02T07:52:49.982075+0000","last_undegraded":"2024-04-02T07:52:49.982075+0000","last_fullsized":"2024-04-02T07:52:49.982075+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:05:36.995577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983263+0000","last_change":"2024-04-02T07:52:49.005606+0000","last_active":"2024-04-02T07:52:49.983263+0000","last_peered":"2024-04-02T07:52:49.983263+0000","last_clean":"2024-04-02T07:52:49.983263+0000","last_became_active":"2024-04-02T07:52:49.005257+0000","last_became_peered":"2024-04-02T07:52:49.005257+0000","last_unstale":"2024-04-02T07:52:49.983263+0000","last_undegraded":"2024-04-02T07:52:49.983263+0000","last_fullsized":"2024-04-02T07:52:49.983263+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:43.695576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.979689+0000","last_change":"2024-04-02T07:52:48.496436+0000","last_active":"2024-04-02T07:52:48.979689+0000","last_peered":"2024-04-02T07:52:48.979689+0000","last_clean":"2024-04-02T07:52:48.979689+0000","last_became_active":"2024-04-02T07:52:48.496179+0000","last_became_peered":"2024-04-02T07:52:48.496179+0000","last_unstale":"2024-04-02T07:52:48.979689+0000","last_undegraded":"2024-04-02T07:52:48.979689+0000","last_fullsized":"2024-04-02T07:52:48.979689+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:07:13.484062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.005629+0000","last_change":"2024-04-02T07:52:49.002985+0000","last_active":"2024-04-02T07:52:49.005629+0000","last_peered":"2024-04-02T07:52:49.005629+0000","last_clean":"2024-04-02T07:52:49.005629+0000","last_became_active":"2024-04-02T07:52:48.997846+0000","last_became_peered":"2024-04-02T07:52:48.997846+0000","last_unstale":"2024-04-02T07:52:49.005629+0000","last_undegraded":"2024-04-02T07:52:49.005629+0000","last_fullsized":"2024-04-02T07:52:49.005629+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:22:39.816893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.10","version":"110'8","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.010883+0000","last_change":"2024-04-02T07:52:48.496481+0000","last_active":"2024-04-02T07:52:49.010883+0000","last_peered":"2024-04-02T07:52:49.010883+0000","last_clean":"2024-04-02T07:52:49.010883+0000","last_became_active":"2024-04-02T07:52:48.496149+0000","last_became_peered":"2024-04-02T07:52:48.496149+0000","last_unstale":"2024-04-02T07:52:49.010883+0000","last_undegraded":"2024-04-02T07:52:49.010883+0000","last_fullsized":"2024-04-02T07:52:49.010883+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:22:03.224804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.010241+0000","last_change":"2024-04-02T07:52:49.006851+0000","last_active":"2024-04-02T07:52:49.010241+0000","last_peered":"2024-04-02T07:52:49.010241+0000","last_clean":"2024-04-02T07:52:49.010241+0000","last_became_active":"2024-04-02T07:52:48.996690+0000","last_became_peered":"2024-04-02T07:52:48.996690+0000","last_unstale":"2024-04-02T07:52:49.010241+0000","last_undegraded":"2024-04-02T07:52:49.010241+0000","last_fullsized":"2024-04-02T07:52:49.010241+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:38.860332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.f","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.007149+0000","last_change":"2024-04-02T07:52:48.488279+0000","last_active":"2024-04-02T07:52:49.007149+0000","last_peered":"2024-04-02T07:52:49.007149+0000","last_clean":"2024-04-02T07:52:49.007149+0000","last_became_active":"2024-04-02T07:52:48.487450+0000","last_became_peered":"2024-04-02T07:52:48.487450+0000","last_unstale":"2024-04-02T07:52:49.007149+0000","last_undegraded":"2024-04-02T07:52:49.007149+0000","last_fullsized":"2024-04-02T07:52:49.007149+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:42.687539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.004212+0000","last_change":"2024-04-02T07:52:49.002363+0000","last_active":"2024-04-02T07:52:49.004212+0000","last_peered":"2024-04-02T07:52:49.004212+0000","last_clean":"2024-04-02T07:52:49.004212+0000","last_became_active":"2024-04-02T07:52:49.002094+0000","last_became_peered":"2024-04-02T07:52:49.002094+0000","last_unstale":"2024-04-02T07:52:49.004212+0000","last_undegraded":"2024-04-02T07:52:49.004212+0000","last_fullsized":"2024-04-02T07:52:49.004212+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:22:17.864289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.e","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983995+0000","last_change":"2024-04-02T07:52:48.490888+0000","last_active":"2024-04-02T07:52:49.983995+0000","last_peered":"2024-04-02T07:52:49.983995+0000","last_clean":"2024-04-02T07:52:49.983995+0000","last_became_active":"2024-04-02T07:52:48.490529+0000","last_became_peered":"2024-04-02T07:52:48.490529+0000","last_unstale":"2024-04-02T07:52:49.983995+0000","last_undegraded":"2024-04-02T07:52:49.983995+0000","last_fullsized":"2024-04-02T07:52:49.983995+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:26:50.809967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.004282+0000","last_change":"2024-04-02T07:52:49.002825+0000","last_active":"2024-04-02T07:52:49.004282+0000","last_peered":"2024-04-02T07:52:49.004282+0000","last_clean":"2024-04-02T07:52:49.004282+0000","last_became_active":"2024-04-02T07:52:49.002541+0000","last_became_peered":"2024-04-02T07:52:49.002541+0000","last_unstale":"2024-04-02T07:52:49.004282+0000","last_undegraded":"2024-04-02T07:52:49.004282+0000","last_fullsized":"2024-04-02T07:52:49.004282+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:31:35.288356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983770+0000","last_change":"2024-04-02T07:52:48.486374+0000","last_active":"2024-04-02T07:52:49.983770+0000","last_peered":"2024-04-02T07:52:49.983770+0000","last_clean":"2024-04-02T07:52:49.983770+0000","last_became_active":"2024-04-02T07:52:48.486083+0000","last_became_peered":"2024-04-02T07:52:48.486083+0000","last_unstale":"2024-04-02T07:52:49.983770+0000","last_undegraded":"2024-04-02T07:52:49.983770+0000","last_fullsized":"2024-04-02T07:52:49.983770+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:39:31.031890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982011+0000","last_change":"2024-04-02T07:52:48.995443+0000","last_active":"2024-04-02T07:52:49.982011+0000","last_peered":"2024-04-02T07:52:49.982011+0000","last_clean":"2024-04-02T07:52:49.982011+0000","last_became_active":"2024-04-02T07:52:48.995157+0000","last_became_peered":"2024-04-02T07:52:48.995157+0000","last_unstale":"2024-04-02T07:52:49.982011+0000","last_undegraded":"2024-04-02T07:52:49.982011+0000","last_fullsized":"2024-04-02T07:52:49.982011+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:41.075062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983764+0000","last_change":"2024-04-02T07:52:48.490860+0000","last_active":"2024-04-02T07:52:49.983764+0000","last_peered":"2024-04-02T07:52:49.983764+0000","last_clean":"2024-04-02T07:52:49.983764+0000","last_became_active":"2024-04-02T07:52:48.490102+0000","last_became_peered":"2024-04-02T07:52:48.490102+0000","last_unstale":"2024-04-02T07:52:49.983764+0000","last_undegraded":"2024-04-02T07:52:49.983764+0000","last_fullsized":"2024-04-02T07:52:49.983764+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:56:18.611413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.005345+0000","last_change":"2024-04-02T07:52:49.001085+0000","last_active":"2024-04-02T07:52:49.005345+0000","last_peered":"2024-04-02T07:52:49.005345+0000","last_clean":"2024-04-02T07:52:49.005345+0000","last_became_active":"2024-04-02T07:52:49.000813+0000","last_became_peered":"2024-04-02T07:52:49.000813+0000","last_unstale":"2024-04-02T07:52:49.005345+0000","last_undegraded":"2024-04-02T07:52:49.005345+0000","last_fullsized":"2024-04-02T07:52:49.005345+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:15.851298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982132+0000","last_change":"2024-04-02T07:52:48.493649+0000","last_active":"2024-04-02T07:52:49.982132+0000","last_peered":"2024-04-02T07:52:49.982132+0000","last_clean":"2024-04-02T07:52:49.982132+0000","last_became_active":"2024-04-02T07:52:48.493371+0000","last_became_peered":"2024-04-02T07:52:48.493371+0000","last_unstale":"2024-04-02T07:52:49.982132+0000","last_undegraded":"2024-04-02T07:52:49.982132+0000","last_fullsized":"2024-04-02T07:52:49.982132+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:17:03.086138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.981833+0000","last_change":"2024-04-02T07:52:48.989357+0000","last_active":"2024-04-02T07:52:49.981833+0000","last_peered":"2024-04-02T07:52:49.981833+0000","last_clean":"2024-04-02T07:52:49.981833+0000","last_became_active":"2024-04-02T07:52:48.988857+0000","last_became_peered":"2024-04-02T07:52:48.988857+0000","last_unstale":"2024-04-02T07:52:49.981833+0000","last_undegraded":"2024-04-02T07:52:49.981833+0000","last_fullsized":"2024-04-02T07:52:49.981833+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:22.405783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.984651+0000","last_change":"2024-04-02T07:52:48.497577+0000","last_active":"2024-04-02T07:52:48.984651+0000","last_peered":"2024-04-02T07:52:48.984651+0000","last_clean":"2024-04-02T07:52:48.984651+0000","last_became_active":"2024-04-02T07:52:48.497275+0000","last_became_peered":"2024-04-02T07:52:48.497275+0000","last_unstale":"2024-04-02T07:52:48.984651+0000","last_undegraded":"2024-04-02T07:52:48.984651+0000","last_fullsized":"2024-04-02T07:52:48.984651+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:23.927722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999274+0000","last_change":"2024-04-02T07:52:48.996785+0000","last_active":"2024-04-02T07:52:48.999274+0000","last_peered":"2024-04-02T07:52:48.999274+0000","last_clean":"2024-04-02T07:52:48.999274+0000","last_became_active":"2024-04-02T07:52:48.995473+0000","last_became_peered":"2024-04-02T07:52:48.995473+0000","last_unstale":"2024-04-02T07:52:48.999274+0000","last_undegraded":"2024-04-02T07:52:48.999274+0000","last_fullsized":"2024-04-02T07:52:48.999274+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:01.482800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983342+0000","last_change":"2024-04-02T07:52:48.491275+0000","last_active":"2024-04-02T07:52:49.983342+0000","last_peered":"2024-04-02T07:52:49.983342+0000","last_clean":"2024-04-02T07:52:49.983342+0000","last_became_active":"2024-04-02T07:52:48.489789+0000","last_became_peered":"2024-04-02T07:52:48.489789+0000","last_unstale":"2024-04-02T07:52:49.983342+0000","last_undegraded":"2024-04-02T07:52:49.983342+0000","last_fullsized":"2024-04-02T07:52:49.983342+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:34:54.355423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999391+0000","last_change":"2024-04-02T07:52:48.991923+0000","last_active":"2024-04-02T07:52:48.999391+0000","last_peered":"2024-04-02T07:52:48.999391+0000","last_clean":"2024-04-02T07:52:48.999391+0000","last_became_active":"2024-04-02T07:52:48.991659+0000","last_became_peered":"2024-04-02T07:52:48.991659+0000","last_unstale":"2024-04-02T07:52:48.999391+0000","last_undegraded":"2024-04-02T07:52:48.999391+0000","last_fullsized":"2024-04-02T07:52:48.999391+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:30:49.706989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.984019+0000","last_change":"2024-04-02T07:52:48.496716+0000","last_active":"2024-04-02T07:52:49.984019+0000","last_peered":"2024-04-02T07:52:49.984019+0000","last_clean":"2024-04-02T07:52:49.984019+0000","last_became_active":"2024-04-02T07:52:48.496269+0000","last_became_peered":"2024-04-02T07:52:48.496269+0000","last_unstale":"2024-04-02T07:52:49.984019+0000","last_undegraded":"2024-04-02T07:52:49.984019+0000","last_fullsized":"2024-04-02T07:52:49.984019+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:12.888674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.000974+0000","last_change":"2024-04-02T07:52:48.995412+0000","last_active":"2024-04-02T07:52:49.000974+0000","last_peered":"2024-04-02T07:52:49.000974+0000","last_clean":"2024-04-02T07:52:49.000974+0000","last_became_active":"2024-04-02T07:52:48.995173+0000","last_became_peered":"2024-04-02T07:52:48.995173+0000","last_unstale":"2024-04-02T07:52:49.000974+0000","last_undegraded":"2024-04-02T07:52:49.000974+0000","last_fullsized":"2024-04-02T07:52:49.000974+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:36:23.265060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.989609+0000","last_change":"2024-04-02T07:52:48.492490+0000","last_active":"2024-04-02T07:52:48.989609+0000","last_peered":"2024-04-02T07:52:48.989609+0000","last_clean":"2024-04-02T07:52:48.989609+0000","last_became_active":"2024-04-02T07:52:48.491324+0000","last_became_peered":"2024-04-02T07:52:48.491324+0000","last_unstale":"2024-04-02T07:52:48.989609+0000","last_undegraded":"2024-04-02T07:52:48.989609+0000","last_fullsized":"2024-04-02T07:52:48.989609+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:37:19.717506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999456+0000","last_change":"2024-04-02T07:52:48.993034+0000","last_active":"2024-04-02T07:52:48.999456+0000","last_peered":"2024-04-02T07:52:48.999456+0000","last_clean":"2024-04-02T07:52:48.999456+0000","last_became_active":"2024-04-02T07:52:48.992764+0000","last_became_peered":"2024-04-02T07:52:48.992764+0000","last_unstale":"2024-04-02T07:52:48.999456+0000","last_undegraded":"2024-04-02T07:52:48.999456+0000","last_fullsized":"2024-04-02T07:52:48.999456+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:47.131307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.975174+0000","last_change":"2024-04-02T07:52:48.493638+0000","last_active":"2024-04-02T07:52:48.975174+0000","last_peered":"2024-04-02T07:52:48.975174+0000","last_clean":"2024-04-02T07:52:48.975174+0000","last_became_active":"2024-04-02T07:52:48.493328+0000","last_became_peered":"2024-04-02T07:52:48.493328+0000","last_unstale":"2024-04-02T07:52:48.975174+0000","last_undegraded":"2024-04-02T07:52:48.975174+0000","last_fullsized":"2024-04-02T07:52:48.975174+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:02:56.034403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.996751+0000","last_change":"2024-04-02T07:52:48.987746+0000","last_active":"2024-04-02T07:52:48.996751+0000","last_peered":"2024-04-02T07:52:48.996751+0000","last_clean":"2024-04-02T07:52:48.996751+0000","last_became_active":"2024-04-02T07:52:48.987478+0000","last_became_peered":"2024-04-02T07:52:48.987478+0000","last_unstale":"2024-04-02T07:52:48.996751+0000","last_undegraded":"2024-04-02T07:52:48.996751+0000","last_fullsized":"2024-04-02T07:52:48.996751+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:12:19.526956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.978798+0000","last_change":"2024-04-02T07:52:48.489650+0000","last_active":"2024-04-02T07:52:48.978798+0000","last_peered":"2024-04-02T07:52:48.978798+0000","last_clean":"2024-04-02T07:52:48.978798+0000","last_became_active":"2024-04-02T07:52:48.488446+0000","last_became_peered":"2024-04-02T07:52:48.488446+0000","last_unstale":"2024-04-02T07:52:48.978798+0000","last_undegraded":"2024-04-02T07:52:48.978798+0000","last_fullsized":"2024-04-02T07:52:48.978798+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:39:31.031890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.004421+0000","last_change":"2024-04-02T07:52:49.003102+0000","last_active":"2024-04-02T07:52:49.004421+0000","last_peered":"2024-04-02T07:52:49.004421+0000","last_clean":"2024-04-02T07:52:49.004421+0000","last_became_active":"2024-04-02T07:52:49.000556+0000","last_became_peered":"2024-04-02T07:52:49.000556+0000","last_unstale":"2024-04-02T07:52:49.004421+0000","last_undegraded":"2024-04-02T07:52:49.004421+0000","last_fullsized":"2024-04-02T07:52:49.004421+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:16.097343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982539+0000","last_change":"2024-04-02T07:52:48.498143+0000","last_active":"2024-04-02T07:52:49.982539+0000","last_peered":"2024-04-02T07:52:49.982539+0000","last_clean":"2024-04-02T07:52:49.982539+0000","last_became_active":"2024-04-02T07:52:48.497870+0000","last_became_peered":"2024-04-02T07:52:48.497870+0000","last_unstale":"2024-04-02T07:52:49.982539+0000","last_undegraded":"2024-04-02T07:52:49.982539+0000","last_fullsized":"2024-04-02T07:52:49.982539+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:11:33.223235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.005077+0000","last_change":"2024-04-02T07:52:49.001571+0000","last_active":"2024-04-02T07:52:49.005077+0000","last_peered":"2024-04-02T07:52:49.005077+0000","last_clean":"2024-04-02T07:52:49.005077+0000","last_became_active":"2024-04-02T07:52:48.998760+0000","last_became_peered":"2024-04-02T07:52:48.998760+0000","last_unstale":"2024-04-02T07:52:49.005077+0000","last_undegraded":"2024-04-02T07:52:49.005077+0000","last_fullsized":"2024-04-02T07:52:49.005077+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:17:04.094801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.990340+0000","last_change":"2024-04-02T07:52:48.489893+0000","last_active":"2024-04-02T07:52:48.990340+0000","last_peered":"2024-04-02T07:52:48.990340+0000","last_clean":"2024-04-02T07:52:48.990340+0000","last_became_active":"2024-04-02T07:52:48.489617+0000","last_became_peered":"2024-04-02T07:52:48.489617+0000","last_unstale":"2024-04-02T07:52:48.990340+0000","last_undegraded":"2024-04-02T07:52:48.990340+0000","last_fullsized":"2024-04-02T07:52:48.990340+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:22:38.808229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.006903+0000","last_change":"2024-04-02T07:52:49.003591+0000","last_active":"2024-04-02T07:52:49.006903+0000","last_peered":"2024-04-02T07:52:49.006903+0000","last_clean":"2024-04-02T07:52:49.006903+0000","last_became_active":"2024-04-02T07:52:49.003331+0000","last_became_peered":"2024-04-02T07:52:49.003331+0000","last_unstale":"2024-04-02T07:52:49.006903+0000","last_undegraded":"2024-04-02T07:52:49.006903+0000","last_fullsized":"2024-04-02T07:52:49.006903+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:11:34.231899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.984669+0000","last_change":"2024-04-02T07:52:48.495565+0000","last_active":"2024-04-02T07:52:48.984669+0000","last_peered":"2024-04-02T07:52:48.984669+0000","last_clean":"2024-04-02T07:52:48.984669+0000","last_became_active":"2024-04-02T07:52:48.491504+0000","last_became_peered":"2024-04-02T07:52:48.491504+0000","last_unstale":"2024-04-02T07:52:48.984669+0000","last_undegraded":"2024-04-02T07:52:48.984669+0000","last_fullsized":"2024-04-02T07:52:48.984669+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:39:31.031890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.995261+0000","last_change":"2024-04-02T07:52:48.992426+0000","last_active":"2024-04-02T07:52:48.995261+0000","last_peered":"2024-04-02T07:52:48.995261+0000","last_clean":"2024-04-02T07:52:48.995261+0000","last_became_active":"2024-04-02T07:52:48.991349+0000","last_became_peered":"2024-04-02T07:52:48.991349+0000","last_unstale":"2024-04-02T07:52:48.995261+0000","last_undegraded":"2024-04-02T07:52:48.995261+0000","last_fullsized":"2024-04-02T07:52:48.995261+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:04:01.924124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.975086+0000","last_change":"2024-04-02T07:52:48.485630+0000","last_active":"2024-04-02T07:52:48.975086+0000","last_peered":"2024-04-02T07:52:48.975086+0000","last_clean":"2024-04-02T07:52:48.975086+0000","last_became_active":"2024-04-02T07:52:48.483333+0000","last_became_peered":"2024-04-02T07:52:48.483333+0000","last_unstale":"2024-04-02T07:52:48.975086+0000","last_undegraded":"2024-04-02T07:52:48.975086+0000","last_fullsized":"2024-04-02T07:52:48.975086+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:34:43.091007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.998524+0000","last_change":"2024-04-02T07:52:48.990862+0000","last_active":"2024-04-02T07:52:48.998524+0000","last_peered":"2024-04-02T07:52:48.998524+0000","last_clean":"2024-04-02T07:52:48.998524+0000","last_became_active":"2024-04-02T07:52:48.989925+0000","last_became_peered":"2024-04-02T07:52:48.989925+0000","last_unstale":"2024-04-02T07:52:48.998524+0000","last_undegraded":"2024-04-02T07:52:48.998524+0000","last_fullsized":"2024-04-02T07:52:48.998524+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:16.097343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.d","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.994105+0000","last_change":"2024-04-02T07:52:48.486362+0000","last_active":"2024-04-02T07:52:48.994105+0000","last_peered":"2024-04-02T07:52:48.994105+0000","last_clean":"2024-04-02T07:52:48.994105+0000","last_became_active":"2024-04-02T07:52:48.486053+0000","last_became_peered":"2024-04-02T07:52:48.486053+0000","last_unstale":"2024-04-02T07:52:48.994105+0000","last_undegraded":"2024-04-02T07:52:48.994105+0000","last_fullsized":"2024-04-02T07:52:48.994105+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:57.192196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"100'183","reported_seq":483,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.481485+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:52:48.481485+0000","last_peered":"2024-04-02T07:52:48.481485+0000","last_clean":"2024-04-02T07:52:48.481485+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:52:48.481485+0000","last_undegraded":"2024-04-02T07:52:48.481485+0000","last_fullsized":"2024-04-02T07:52:48.481485+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575049,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1368064,"data_stored":668472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.65000000000000002,"15min":0.67800000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.316},"max":{"1min":0.89400000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66400000000000003,"15min":0.68200000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.33200000000000002},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.53800000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65800000000000003,"15min":0.67900000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.65200000000000002,"15min":0.67800000000000005},"min":{"1min":0.42499999999999999,"5min":0.379,"15min":0.34699999999999998},"max":{"1min":0.98299999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.69199999999999995,"15min":0.70999999999999996},"min":{"1min":0.39500000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.94199999999999995,"5min":1.127,"15min":1.127},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.73599999999999999,"15min":0.76100000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":1.044,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.61799999999999999}]},{"osd":3,"last update":"Tue Apr 2 07:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.67300000000000004,"15min":0.70299999999999996},"min":{"1min":0.40600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93500000000000005,"5min":0.97799999999999998,"15min":1.026},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.624,"5min":0.69599999999999995,"15min":0.71999999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.071,"5min":1.196,"15min":1.196},"last":0.58099999999999996}]},{"osd":4,"last update":"Tue Apr 2 07:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56599999999999995,"15min":0.58499999999999996},"min":{"1min":0.36299999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0229999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.438},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.67200000000000004,"15min":0.68300000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.80500000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.627}]},{"osd":5,"last update":"Tue Apr 2 07:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.42999999999999999,"5min":0.37,"15min":0.37},"max":{"1min":1.0109999999999999,"5min":1.167,"15min":1.167},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.67200000000000004,"15min":0.68300000000000005},"min":{"1min":0.40200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.343,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.75}]},{"osd":6,"last update":"Tue Apr 2 07:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.66300000000000003,"15min":0.65500000000000003},"min":{"1min":0.435,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0580000000000001,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.64300000000000002,"15min":0.64800000000000002},"min":{"1min":0.45400000000000001,"5min":0.42299999999999999,"15min":0.39600000000000002},"max":{"1min":0.83399999999999996,"5min":0.93799999999999994,"15min":1.0509999999999999},"last":0.68300000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575047,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36548,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335292,"statfs":{"total":96636764160,"available":96599339008,"internally_reserved":0,"allocated":1359872,"data_stored":662426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.68899999999999995,"15min":0.70599999999999996},"min":{"1min":0.40899999999999997,"5min":0.38700000000000001,"15min":0.33400000000000002},"max":{"1min":0.80600000000000005,"5min":0.95699999999999996,"15min":0.98099999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.68700000000000006,"15min":0.70799999999999996},"min":{"1min":0.46100000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.84799999999999998,"5min":0.96899999999999997,"15min":1.0249999999999999},"last":0.502}]},{"osd":1,"last update":"Tue Apr 2 07:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.67300000000000004,"15min":0.69599999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.41199999999999998},"max":{"1min":0.76300000000000001,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.71399999999999997,"15min":0.71999999999999997},"min":{"1min":0.45300000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84399999999999997,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.625}]},{"osd":2,"last update":"Tue Apr 2 07:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.70599999999999996,"15min":0.71499999999999997},"min":{"1min":0.46600000000000003,"5min":0.45500000000000002,"15min":0.38200000000000001},"max":{"1min":0.72999999999999998,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46300000000000002},"max":{"1min":0.83799999999999997,"5min":0.96499999999999997,"15min":0.96599999999999997},"last":0.51900000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.72299999999999998,"15min":0.73799999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.91600000000000004,"5min":1.089,"15min":1.089},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.72599999999999998,"15min":0.73899999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.45300000000000001},"max":{"1min":0.82099999999999995,"5min":0.97799999999999998,"15min":1.0309999999999999},"last":0.60299999999999998}]},{"osd":4,"last update":"Tue Apr 2 07:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61799999999999999,"15min":0.60499999999999998},"min":{"1min":0.42799999999999999,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.82399999999999995,"5min":0.996,"15min":0.996},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63500000000000001,"15min":0.628},"min":{"1min":0.45200000000000001,"5min":0.40600000000000003,"15min":0.35499999999999998},"max":{"1min":0.86699999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.71799999999999997}]},{"osd":5,"last update":"Tue Apr 2 07:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.628,"15min":0.621},"min":{"1min":0.45200000000000001,"5min":0.379,"15min":0.35799999999999998},"max":{"1min":0.88100000000000001,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.67900000000000005,"15min":0.68799999999999994},"min":{"1min":0.51700000000000002,"5min":0.42099999999999999,"15min":0.40300000000000002},"max":{"1min":0.90200000000000002,"5min":1.01,"15min":1.01},"last":0.76300000000000001}]},{"osd":7,"last update":"Tue Apr 2 07:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.69599999999999995,"15min":0.68799999999999994},"min":{"1min":0.56000000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.91400000000000003,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.746},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69199999999999995,"15min":0.69499999999999995},"min":{"1min":0.48999999999999999,"5min":0.44,"15min":0.42099999999999999},"max":{"1min":0.92800000000000005,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.78800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575048,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55172,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316668,"statfs":{"total":96636764160,"available":96580268032,"internally_reserved":0,"allocated":1949696,"data_stored":1252212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.435,"15min":0.436},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":10.052},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51400000000000001,"15min":0.504},"min":{"1min":0.27500000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":9.0250000000000004}]},{"osd":2,"last update":"Tue Apr 2 07:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.29699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":7.6059999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.36699999999999999,"15min":0.35699999999999998},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":6.0709999999999997}]},{"osd":3,"last update":"Tue Apr 2 07:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.609,"5min":2.609,"15min":2.609},"last":1.194},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54000000000000004,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.72999999999999998}]},{"osd":4,"last update":"Tue Apr 2 07:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78300000000000003,"15min":0.77500000000000002},"min":{"1min":0.55100000000000005,"5min":0.46700000000000003,"15min":0.46300000000000002},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78000000000000003,"15min":0.77100000000000002},"min":{"1min":0.55100000000000005,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":6.3419999999999996}]},{"osd":5,"last update":"Tue Apr 2 07:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.77200000000000002,"15min":0.77500000000000002},"min":{"1min":0.58399999999999996,"5min":0.53800000000000003,"15min":0.41999999999999998},"max":{"1min":2.4830000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":6.5529999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.79700000000000004,"15min":0.78900000000000003},"min":{"1min":0.56699999999999995,"5min":0.51000000000000001,"15min":0.48799999999999999},"max":{"1min":2.544,"5min":2.544,"15min":2.544},"last":2.1000000000000001}]},{"osd":6,"last update":"Tue Apr 2 07:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.80200000000000005,"15min":0.79800000000000004},"min":{"1min":0.55900000000000005,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.2370000000000001,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":7.8029999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.80100000000000005,"15min":0.79400000000000004},"min":{"1min":0.58499999999999996,"5min":0.51400000000000001,"15min":0.38100000000000001},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":6.4329999999999998}]},{"osd":7,"last update":"Tue Apr 2 07:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.80700000000000005,"15min":0.80500000000000005},"min":{"1min":0.55000000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":1.242},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82499999999999996,"15min":0.82299999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.502},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":0.76000000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575048,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1363968,"data_stored":662321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.433,"15min":0.435},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.66000000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.26200000000000001},"max":{"1min":0.60099999999999998,"5min":0.72199999999999998,"15min":0.79100000000000004},"last":0.38700000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.48099999999999998,"15min":0.48699999999999999},"min":{"1min":0.312,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":0.749,"15min":0.76300000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.39400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.63,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46100000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.52900000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.28899999999999998},"max":{"1min":0.60899999999999999,"5min":0.73999999999999999,"15min":0.81799999999999995},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.40100000000000002},"max":{"1min":0.77700000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.40699999999999997}]},{"osd":4,"last update":"Tue Apr 2 07:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.75800000000000001,"15min":0.77700000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.35599999999999998},"max":{"1min":0.95899999999999996,"5min":1.0640000000000001,"15min":1.181},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.76500000000000001,"15min":0.77300000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":0.91800000000000004,"5min":1.095,"15min":1.331},"last":0.753}]},{"osd":5,"last update":"Tue Apr 2 07:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.79100000000000004,"15min":0.78800000000000003},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.42099999999999999},"max":{"1min":0.95199999999999996,"5min":2.1869999999999998,"15min":2.1869999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.79400000000000004,"15min":0.79500000000000004},"min":{"1min":0.59399999999999997,"5min":0.54100000000000004,"15min":0.51800000000000002},"max":{"1min":0.92500000000000004,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":0.81499999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.745,"15min":0.751},"min":{"1min":0.56399999999999995,"5min":0.53000000000000003,"15min":0.46800000000000003},"max":{"1min":0.82499999999999996,"5min":1.044,"15min":1.1200000000000001},"last":0.872},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.72499999999999998,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.76200000000000001,"5min":1.0609999999999999,"15min":1.1339999999999999},"last":0.73599999999999999}]},{"osd":7,"last update":"Tue Apr 2 07:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.82899999999999996,"15min":0.81499999999999995},"min":{"1min":0.60499999999999998,"5min":0.51600000000000001,"15min":0.40799999999999997},"max":{"1min":0.89500000000000002,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.78600000000000003,"15min":0.80500000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.88500000000000001,"5min":2.145,"15min":2.145},"last":0.77400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575047,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36540,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335300,"statfs":{"total":96636764160,"available":96599347200,"internally_reserved":0,"allocated":1351680,"data_stored":661754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.44,"15min":0.44800000000000001},"min":{"1min":0.26700000000000002,"5min":0.26400000000000001,"15min":0.25700000000000001},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46300000000000002,"15min":0.47699999999999998},"min":{"1min":0.36199999999999999,"5min":0.27900000000000003,"15min":0.26200000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.62}]},{"osd":1,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.49199999999999999,"15min":0.49099999999999999},"min":{"1min":0.33000000000000002,"5min":0.30599999999999999,"15min":0.30299999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.499},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.52400000000000002},"min":{"1min":0.39700000000000002,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.66300000000000003}]},{"osd":3,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.35999999999999999,"5min":0.316,"15min":0.27800000000000002},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59599999999999997,"15min":0.60999999999999999},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39000000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.64000000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76300000000000001,"15min":0.77000000000000002},"min":{"1min":0.5,"5min":0.5,"15min":0.47199999999999998},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.748,"15min":0.77200000000000002},"min":{"1min":0.56499999999999995,"5min":0.51600000000000001,"15min":0.48499999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.68500000000000005}]},{"osd":5,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.752,"15min":0.76100000000000001},"min":{"1min":0.56499999999999995,"5min":0.54300000000000004,"15min":0.42199999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.55600000000000005,"5min":0.53800000000000003,"15min":0.50900000000000001},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.67200000000000004}]},{"osd":6,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.79400000000000004,"15min":0.79200000000000004},"min":{"1min":0.55000000000000004,"5min":0.53600000000000003,"15min":0.442},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.752,"5min":0.75600000000000001,"15min":0.78700000000000003},"min":{"1min":0.53100000000000003,"5min":0.502,"15min":0.502},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.70099999999999996}]},{"osd":7,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.77200000000000002,"15min":0.79500000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.5},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.77100000000000002,"15min":0.79300000000000004},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.65500000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575047,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54600,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317240,"statfs":{"total":96636764160,"available":96580853760,"internally_reserved":0,"allocated":1363968,"data_stored":662428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.42399999999999999,"15min":0.42999999999999999},"min":{"1min":0.26500000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.72599999999999998,"5min":0.748,"15min":0.79600000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.435,"15min":0.46400000000000002},"min":{"1min":0.31,"5min":0.29699999999999999,"15min":0.28699999999999998},"max":{"1min":0.57099999999999995,"5min":0.70799999999999996,"15min":0.81899999999999995},"last":0.49399999999999999}]},{"osd":1,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.47899999999999998,"15min":0.49199999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61899999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.439,"5min":0.35799999999999998,"15min":0.34200000000000003},"max":{"1min":0.77500000000000002,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.51400000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59099999999999997,"15min":0.60699999999999998},"min":{"1min":0.439,"5min":0.38700000000000001,"15min":0.376},"max":{"1min":0.78400000000000003,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57699999999999996,"15min":0.60399999999999998},"min":{"1min":0.39500000000000002,"5min":0.38100000000000001,"15min":0.35899999999999999},"max":{"1min":0.70699999999999996,"5min":1.244,"15min":1.244},"last":0.44500000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.70899999999999996,"15min":0.73199999999999998},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.38400000000000001},"max":{"1min":0.83499999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73199999999999998,"15min":0.75700000000000001},"min":{"1min":0.52300000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":1.1759999999999999},"last":0.74299999999999999}]},{"osd":5,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.72399999999999998,"15min":0.754},"min":{"1min":0.51300000000000001,"5min":0.41699999999999998,"15min":0.40899999999999997},"max":{"1min":0.97299999999999998,"5min":1.079,"15min":1.151},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72099999999999997,"15min":0.749},"min":{"1min":0.49199999999999999,"5min":0.48899999999999999,"15min":0.40200000000000002},"max":{"1min":0.94199999999999995,"5min":0.97599999999999998,"15min":1.1439999999999999},"last":0.72599999999999998}]},{"osd":6,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72199999999999998,"15min":0.76000000000000001},"min":{"1min":0.52800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.86899999999999999,"5min":0.98999999999999999,"15min":1.0620000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.748,"15min":0.77400000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.90400000000000003,"5min":1.1140000000000001,"15min":1.1899999999999999},"last":0.68400000000000005}]},{"osd":7,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.73799999999999999,"15min":0.77000000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.439},"max":{"1min":0.91100000000000003,"5min":1.0409999999999999,"15min":1.0680000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.75,"15min":0.78500000000000003},"min":{"1min":0.60299999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0089999999999999,"5min":1.181,"15min":1.181},"last":0.78700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575047,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36552,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335288,"statfs":{"total":96636764160,"available":96599334912,"internally_reserved":0,"allocated":1368064,"data_stored":668436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.42499999999999999,"5min":0.38100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86599999999999999,"5min":0.91900000000000004,"15min":1.0189999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.66800000000000004,"15min":0.67500000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.879,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.53400000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.71599999999999997,"15min":0.72799999999999998},"min":{"1min":0.47599999999999998,"5min":0.438,"15min":0.433},"max":{"1min":0.90300000000000002,"5min":0.97299999999999998,"15min":1.036},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.73799999999999999},"min":{"1min":0.58099999999999996,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.91900000000000004,"5min":0.97899999999999998,"15min":0.99399999999999999},"last":0.496}]},{"osd":2,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.67400000000000004,"15min":0.69099999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.47299999999999998,"5min":0.47199999999999998,"15min":0.41899999999999998},"max":{"1min":0.874,"5min":0.97199999999999998,"15min":1.0620000000000001},"last":0.55300000000000005}]},{"osd":3,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72099999999999997},"min":{"1min":0.51500000000000001,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":0.95299999999999996,"5min":0.95699999999999996,"15min":1.002},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.71299999999999997,"15min":0.72999999999999998},"min":{"1min":0.49299999999999999,"5min":0.48799999999999999,"15min":0.41499999999999998},"max":{"1min":0.92100000000000004,"5min":0.93999999999999995,"15min":0.96999999999999997},"last":0.47299999999999998}]},{"osd":5,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56200000000000006,"15min":0.57399999999999995},"min":{"1min":0.45800000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.746,"5min":0.84799999999999998,"15min":0.93999999999999995},"last":0.435},{"interface":"front","average":{"1min":0.621,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.434,"5min":0.38800000000000001,"15min":0.33700000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.66800000000000004}]},{"osd":6,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.61199999999999999,"15min":0.60399999999999998},"min":{"1min":0.45200000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.51600000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.038},"last":0.65500000000000003}]},{"osd":7,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68000000000000005,"15min":0.66800000000000004},"min":{"1min":0.55700000000000005,"5min":0.432,"15min":0.38300000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.74099999999999999,"15min":0.72999999999999998},"min":{"1min":0.58999999999999997,"5min":0.47799999999999998,"15min":0.434},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.746}]}]},{"osd":5,"up_from":13,"seq":55834575050,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37140,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334700,"statfs":{"total":96636764160,"available":96598732800,"internally_reserved":0,"allocated":1970176,"data_stored":1259349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.68500000000000005},"min":{"1min":0.433,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83099999999999996,"5min":0.95299999999999996,"15min":0.99199999999999999},"last":0.499},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.504,"5min":0.378,"15min":0.378},"max":{"1min":0.92700000000000005,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.69399999999999995}]},{"osd":1,"last update":"Tue Apr 2 07:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.66500000000000004,"15min":0.69399999999999995},"min":{"1min":0.48399999999999999,"5min":0.40300000000000002,"15min":0.40200000000000002},"max":{"1min":0.84799999999999998,"5min":1.131,"15min":1.131},"last":0.44},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.68000000000000005,"15min":0.69199999999999995},"min":{"1min":0.48499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85599999999999998,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.40999999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.68899999999999995,"15min":0.70899999999999996},"min":{"1min":0.43099999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.78800000000000003,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.68000000000000005,"15min":0.70599999999999996},"min":{"1min":0.497,"5min":0.47399999999999998,"15min":0.372},"max":{"1min":0.82599999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.55700000000000005}]},{"osd":3,"last update":"Tue Apr 2 07:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.71799999999999997,"15min":0.73799999999999999},"min":{"1min":0.51500000000000001,"5min":0.45900000000000002,"15min":0.45100000000000001},"max":{"1min":0.85099999999999998,"5min":1.341,"15min":1.341},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.754,"15min":0.76400000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.44900000000000001},"max":{"1min":0.88100000000000001,"5min":1.484,"15min":1.484},"last":0.68000000000000005}]},{"osd":4,"last update":"Tue Apr 2 07:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60799999999999998,"15min":0.59799999999999998},"min":{"1min":0.43099999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.78300000000000003,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65000000000000002,"15min":0.64000000000000001},"min":{"1min":0.44,"5min":0.42499999999999999,"15min":0.35599999999999998},"max":{"1min":0.86199999999999999,"5min":1.504,"15min":1.504},"last":0.61799999999999999}]},{"osd":6,"last update":"Tue Apr 2 07:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.65600000000000003,"15min":0.64200000000000002},"min":{"1min":0.46999999999999997,"5min":0.44700000000000001,"15min":0.39900000000000002},"max":{"1min":0.94599999999999995,"5min":1.212,"15min":1.212},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70799999999999996,"15min":0.67700000000000005},"min":{"1min":0.53300000000000003,"5min":0.44900000000000001,"15min":0.41199999999999998},"max":{"1min":0.90300000000000002,"5min":1.774,"15min":1.774},"last":0.54300000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72199999999999998,"15min":0.70099999999999996},"min":{"1min":0.57999999999999996,"5min":0.47599999999999998,"15min":0.38900000000000001},"max":{"1min":0.96899999999999997,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.74199999999999999,"15min":0.72199999999999998},"min":{"1min":0.63600000000000001,"5min":0.46300000000000002,"15min":0.434},"max":{"1min":0.98099999999999998,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.70199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:52:53.353 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:52:53.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.577+0000 7ff188c00700 1 Processor -- start 2024-04-02T07:52:53.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.577+0000 7ff188c00700 1 -- start start 2024-04-02T07:52:53.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.578+0000 7ff188c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff1840d18b0 0x7ff1840cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:53.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.578+0000 7ff188c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff184161590 0x7ff184161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:53.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.578+0000 7ff188c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1840ceda0 con 0x7ff184160f10 2024-04-02T07:52:53.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.578+0000 7ff188c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1840d1d30 con 0x7ff184161590 2024-04-02T07:52:53.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.578+0000 7ff188c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff184161f60 con 0x7ff1840d18b0 2024-04-02T07:52:53.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.578+0000 7ff182800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff184161590 0x7ff184161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:53.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.578+0000 7ff182800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff184161590 0x7ff184161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46544/0 (socket says 172.21.15.89:46544) 2024-04-02T07:52:53.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.578+0000 7ff182800700 1 -- 172.21.15.89:0/2353057113 learned_addr learned my addr 172.21.15.89:0/2353057113 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:53.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.578+0000 7ff181e00700 1 --1- 172.21.15.89:0/2353057113 >> v1:172.21.15.89:6789/0 conn(0x7ff184160f10 0x7ff1840d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33306/0 (socket says 172.21.15.89:33306) 2024-04-02T07:52:53.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.579+0000 7ff17be00700 1 --2- 172.21.15.89:0/2353057113 >> v2:172.21.15.113:3300/0 conn(0x7ff1840d18b0 0x7ff1840cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:53.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.579+0000 7ff182800700 1 -- 172.21.15.89:0/2353057113 >> v2:172.21.15.113:3300/0 conn(0x7ff1840d18b0 msgr2=0x7ff1840cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:53.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.579+0000 7ff182800700 1 --2- 172.21.15.89:0/2353057113 >> v2:172.21.15.113:3300/0 conn(0x7ff1840d18b0 0x7ff1840cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.579+0000 7ff182800700 1 -- 172.21.15.89:0/2353057113 >> v1:172.21.15.89:6789/0 conn(0x7ff184160f10 legacy=0x7ff1840d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:53.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.579+0000 7ff182800700 1 -- 172.21.15.89:0/2353057113 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1841612f0 con 0x7ff184161590 2024-04-02T07:52:53.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.579+0000 7ff17be00700 1 --2- 172.21.15.89:0/2353057113 >> v2:172.21.15.113:3300/0 conn(0x7ff1840d18b0 0x7ff1840cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:52:53.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.579+0000 7ff181400700 1 -- 172.21.15.89:0/2353057113 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655222228 0 0) 0x7ff1840ceda0 con 0x7ff184160f10 2024-04-02T07:52:53.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.579+0000 7ff182800700 1 --2- 172.21.15.89:0/2353057113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff184161590 0x7ff184161a20 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ff18408ca30 tx=0x7ff17c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a76a17a5aedc04d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:53.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.579+0000 7ff181400700 1 -- 172.21.15.89:0/2353057113 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c00e070 con 0x7ff184161590 2024-04-02T07:52:53.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.579+0000 7ff181400700 1 -- 172.21.15.89:0/2353057113 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c00b9c0 con 0x7ff184161590 2024-04-02T07:52:53.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.579+0000 7ff181400700 1 -- 172.21.15.89:0/2353057113 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c00bd70 con 0x7ff184161590 2024-04-02T07:52:53.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.580+0000 7ff188c00700 1 -- 172.21.15.89:0/2353057113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff184161590 msgr2=0x7ff184161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:53.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.580+0000 7ff188c00700 1 --2- 172.21.15.89:0/2353057113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff184161590 0x7ff184161a20 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ff18408ca30 tx=0x7ff17c00b330 comp rx=0 tx=0).stop 2024-04-02T07:52:53.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.580+0000 7ff188c00700 1 -- 172.21.15.89:0/2353057113 shutdown_connections 2024-04-02T07:52:53.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.580+0000 7ff188c00700 1 --2- 172.21.15.89:0/2353057113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff184161590 0x7ff184161a20 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.580+0000 7ff188c00700 1 --2- 172.21.15.89:0/2353057113 >> v2:172.21.15.113:3300/0 conn(0x7ff1840d18b0 0x7ff1840cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.580+0000 7ff188c00700 1 -- 172.21.15.89:0/2353057113 >> 172.21.15.89:0/2353057113 conn(0x7ff18408aff0 msgr2=0x7ff1840d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:53.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.580+0000 7ff188c00700 1 -- 172.21.15.89:0/2353057113 shutdown_connections 2024-04-02T07:52:53.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.580+0000 7ff188c00700 1 -- 172.21.15.89:0/2353057113 wait complete. 2024-04-02T07:52:53.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.581+0000 7ff188c00700 1 Processor -- start 2024-04-02T07:52:53.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.582+0000 7ff188c00700 1 -- start start 2024-04-02T07:52:53.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.582+0000 7ff188c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1840d18b0 0x7ff1841f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:53.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.582+0000 7ff188c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff184160f10 0x7ff1840cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:53.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.582+0000 7ff188c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff184161590 0x7ff1840cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.582+0000 7ff188c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1841f33b0 con 0x7ff184160f10 2024-04-02T07:52:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.582+0000 7ff188c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1841f4190 con 0x7ff184161590 2024-04-02T07:52:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.582+0000 7ff188c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1841f35c0 con 0x7ff1840d18b0 2024-04-02T07:52:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.583+0000 7ff17be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff184160f10 0x7ff1840cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.583+0000 7ff17be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff184160f10 0x7ff1840cc110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46552/0 (socket says 172.21.15.89:46552) 2024-04-02T07:52:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.583+0000 7ff17be00700 1 -- 172.21.15.89:0/1551433364 learned_addr learned my addr 172.21.15.89:0/1551433364 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.583+0000 7ff181e00700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1840d18b0 0x7ff1841f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.583+0000 7ff182800700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff184161590 0x7ff1840cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.583+0000 7ff17be00700 1 -- 172.21.15.89:0/1551433364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1840d18b0 msgr2=0x7ff1841f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.583+0000 7ff17be00700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1840d18b0 0x7ff1841f6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.583+0000 7ff17be00700 1 -- 172.21.15.89:0/1551433364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff184161590 msgr2=0x7ff1840cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.583+0000 7ff17be00700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff184161590 0x7ff1840cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.583+0000 7ff17be00700 1 -- 172.21.15.89:0/1551433364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17c009b30 con 0x7ff184160f10 2024-04-02T07:52:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.583+0000 7ff182800700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff184161590 0x7ff1840cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.583+0000 7ff17be00700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff184160f10 0x7ff1840cc110 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7ff16c00c7f0 tx=0x7ff16c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.584+0000 7ff17a000700 1 -- 172.21.15.89:0/1551433364 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c011800 con 0x7ff184160f10 2024-04-02T07:52:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.584+0000 7ff17a000700 1 -- 172.21.15.89:0/1551433364 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16c011960 con 0x7ff184160f10 2024-04-02T07:52:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.584+0000 7ff188c00700 1 -- 172.21.15.89:0/1551433364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1841f3860 con 0x7ff184160f10 2024-04-02T07:52:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.584+0000 7ff17a000700 1 -- 172.21.15.89:0/1551433364 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c011cf0 con 0x7ff184160f10 2024-04-02T07:52:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.584+0000 7ff188c00700 1 -- 172.21.15.89:0/1551433364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1841f3e20 con 0x7ff184160f10 2024-04-02T07:52:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.585+0000 7ff188c00700 1 -- 172.21.15.89:0/1551433364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18406ceb0 con 0x7ff184160f10 2024-04-02T07:52:53.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.586+0000 7ff17a000700 1 -- 172.21.15.89:0/1551433364 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff16c018360 con 0x7ff184160f10 2024-04-02T07:52:53.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.587+0000 7ff17a000700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff16006ade0 0x7ff16006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:53.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.587+0000 7ff17a000700 1 -- 172.21.15.89:0/1551433364 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff16c0896e0 con 0x7ff184160f10 2024-04-02T07:52:53.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.587+0000 7ff181e00700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff16006ade0 0x7ff16006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:53.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.588+0000 7ff181e00700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff16006ade0 0x7ff16006d2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff174000f60 tx=0x7ff174007e70 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:53.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.592+0000 7ff17a000700 1 -- 172.21.15.89:0/1551433364 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff16c054420 con 0x7ff184160f10 2024-04-02T07:52:53.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.789+0000 7ff188c00700 1 -- 172.21.15.89:0/1551433364 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff1840ce750 con 0x7ff16006ade0 2024-04-02T07:52:53.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.798+0000 7ff17a000700 1 -- 172.21.15.89:0/1551433364 <== mgr.6775 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289602 (secure 0 0 0) 0x7ff174001290 con 0x7ff16006ade0 2024-04-02T07:52:53.805 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:53.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.802+0000 7ff188c00700 1 -- 172.21.15.89:0/1551433364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff16006ade0 msgr2=0x7ff16006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:53.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.802+0000 7ff188c00700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff16006ade0 0x7ff16006d2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff174000f60 tx=0x7ff174007e70 comp rx=0 tx=0).stop 2024-04-02T07:52:53.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.802+0000 7ff188c00700 1 -- 172.21.15.89:0/1551433364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff184160f10 msgr2=0x7ff1840cc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:53.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.802+0000 7ff188c00700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff184160f10 0x7ff1840cc110 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7ff16c00c7f0 tx=0x7ff16c00cb00 comp rx=0 tx=0).stop 2024-04-02T07:52:53.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.802+0000 7ff188c00700 1 -- 172.21.15.89:0/1551433364 shutdown_connections 2024-04-02T07:52:53.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.802+0000 7ff188c00700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff16006ade0 0x7ff16006d2a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.802+0000 7ff188c00700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff184161590 0x7ff1840cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.802+0000 7ff188c00700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1840d18b0 0x7ff1841f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.802+0000 7ff188c00700 1 --2- 172.21.15.89:0/1551433364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff184160f10 0x7ff1840cc110 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:53.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.802+0000 7ff188c00700 1 -- 172.21.15.89:0/1551433364 >> 172.21.15.89:0/1551433364 conn(0x7ff18408aff0 msgr2=0x7ff1840d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:53.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.802+0000 7ff188c00700 1 -- 172.21.15.89:0/1551433364 shutdown_connections 2024-04-02T07:52:53.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:53.802+0000 7ff188c00700 1 -- 172.21.15.89:0/1551433364 wait complete. 2024-04-02T07:52:53.808 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:52:53.821 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T07:52:52.861964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":321460,"kb_used_data":11812,"kb_used_omap":12,"kb_used_meta":309491,"kb_avail":754653260,"statfs":{"total":773094113280,"available":772764938240,"internally_reserved":0,"allocated":12095488,"data_stored":6497398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":316919352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.396620"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.010194+0000","last_change":"2024-04-02T07:52:49.008897+0000","last_active":"2024-04-02T07:52:49.010194+0000","last_peered":"2024-04-02T07:52:49.010194+0000","last_clean":"2024-04-02T07:52:49.010194+0000","last_became_active":"2024-04-02T07:52:49.008537+0000","last_became_peered":"2024-04-02T07:52:49.008537+0000","last_unstale":"2024-04-02T07:52:49.010194+0000","last_undegraded":"2024-04-02T07:52:49.010194+0000","last_fullsized":"2024-04-02T07:52:49.010194+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:29.124428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983043+0000","last_change":"2024-04-02T07:52:48.984704+0000","last_active":"2024-04-02T07:52:49.983043+0000","last_peered":"2024-04-02T07:52:49.983043+0000","last_clean":"2024-04-02T07:52:49.983043+0000","last_became_active":"2024-04-02T07:52:48.984443+0000","last_became_peered":"2024-04-02T07:52:48.984443+0000","last_unstale":"2024-04-02T07:52:49.983043+0000","last_undegraded":"2024-04-02T07:52:49.983043+0000","last_fullsized":"2024-04-02T07:52:49.983043+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:22:27.541973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.001271+0000","last_change":"2024-04-02T07:52:48.997229+0000","last_active":"2024-04-02T07:52:49.001271+0000","last_peered":"2024-04-02T07:52:49.001271+0000","last_clean":"2024-04-02T07:52:49.001271+0000","last_became_active":"2024-04-02T07:52:48.996919+0000","last_became_peered":"2024-04-02T07:52:48.996919+0000","last_unstale":"2024-04-02T07:52:49.001271+0000","last_undegraded":"2024-04-02T07:52:49.001271+0000","last_fullsized":"2024-04-02T07:52:49.001271+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:43:13.014987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982719+0000","last_change":"2024-04-02T07:52:48.988005+0000","last_active":"2024-04-02T07:52:49.982719+0000","last_peered":"2024-04-02T07:52:49.982719+0000","last_clean":"2024-04-02T07:52:49.982719+0000","last_became_active":"2024-04-02T07:52:48.987697+0000","last_became_peered":"2024-04-02T07:52:48.987697+0000","last_unstale":"2024-04-02T07:52:49.982719+0000","last_undegraded":"2024-04-02T07:52:49.982719+0000","last_fullsized":"2024-04-02T07:52:49.982719+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:43.696202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.990751+0000","last_change":"2024-04-02T07:52:48.986945+0000","last_active":"2024-04-02T07:52:48.990751+0000","last_peered":"2024-04-02T07:52:48.990751+0000","last_clean":"2024-04-02T07:52:48.990751+0000","last_became_active":"2024-04-02T07:52:48.986619+0000","last_became_peered":"2024-04-02T07:52:48.986619+0000","last_unstale":"2024-04-02T07:52:48.990751+0000","last_undegraded":"2024-04-02T07:52:48.990751+0000","last_fullsized":"2024-04-02T07:52:48.990751+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:43:38.553176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.006991+0000","last_change":"2024-04-02T07:52:49.003644+0000","last_active":"2024-04-02T07:52:49.006991+0000","last_peered":"2024-04-02T07:52:49.006991+0000","last_clean":"2024-04-02T07:52:49.006991+0000","last_became_active":"2024-04-02T07:52:49.003394+0000","last_became_peered":"2024-04-02T07:52:49.003394+0000","last_unstale":"2024-04-02T07:52:49.006991+0000","last_undegraded":"2024-04-02T07:52:49.006991+0000","last_fullsized":"2024-04-02T07:52:49.006991+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:13:13.336296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.981987+0000","last_change":"2024-04-02T07:52:48.981572+0000","last_active":"2024-04-02T07:52:49.981987+0000","last_peered":"2024-04-02T07:52:49.981987+0000","last_clean":"2024-04-02T07:52:49.981987+0000","last_became_active":"2024-04-02T07:52:48.981278+0000","last_became_peered":"2024-04-02T07:52:48.981278+0000","last_unstale":"2024-04-02T07:52:49.981987+0000","last_undegraded":"2024-04-02T07:52:49.981987+0000","last_fullsized":"2024-04-02T07:52:49.981987+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:29.124428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.010943+0000","last_change":"2024-04-02T07:52:49.008419+0000","last_active":"2024-04-02T07:52:49.010943+0000","last_peered":"2024-04-02T07:52:49.010943+0000","last_clean":"2024-04-02T07:52:49.010943+0000","last_became_active":"2024-04-02T07:52:49.008077+0000","last_became_peered":"2024-04-02T07:52:49.008077+0000","last_unstale":"2024-04-02T07:52:49.010943+0000","last_undegraded":"2024-04-02T07:52:49.010943+0000","last_fullsized":"2024-04-02T07:52:49.010943+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:59:01.177649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999866+0000","last_change":"2024-04-02T07:52:48.997488+0000","last_active":"2024-04-02T07:52:48.999866+0000","last_peered":"2024-04-02T07:52:48.999866+0000","last_clean":"2024-04-02T07:52:48.999866+0000","last_became_active":"2024-04-02T07:52:48.995310+0000","last_became_peered":"2024-04-02T07:52:48.995310+0000","last_unstale":"2024-04-02T07:52:48.999866+0000","last_undegraded":"2024-04-02T07:52:48.999866+0000","last_fullsized":"2024-04-02T07:52:48.999866+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:43.695576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.998749+0000","last_change":"2024-04-02T07:52:48.991820+0000","last_active":"2024-04-02T07:52:48.998749+0000","last_peered":"2024-04-02T07:52:48.998749+0000","last_clean":"2024-04-02T07:52:48.998749+0000","last_became_active":"2024-04-02T07:52:48.991514+0000","last_became_peered":"2024-04-02T07:52:48.991514+0000","last_unstale":"2024-04-02T07:52:48.998749+0000","last_undegraded":"2024-04-02T07:52:48.998749+0000","last_fullsized":"2024-04-02T07:52:48.998749+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:53:04.013403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.006028+0000","last_change":"2024-04-02T07:52:49.003710+0000","last_active":"2024-04-02T07:52:49.006028+0000","last_peered":"2024-04-02T07:52:49.006028+0000","last_clean":"2024-04-02T07:52:49.006028+0000","last_became_active":"2024-04-02T07:52:49.003283+0000","last_became_peered":"2024-04-02T07:52:49.003283+0000","last_unstale":"2024-04-02T07:52:49.006028+0000","last_undegraded":"2024-04-02T07:52:49.006028+0000","last_fullsized":"2024-04-02T07:52:49.006028+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:06:03.280191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.006526+0000","last_change":"2024-04-02T07:52:48.991768+0000","last_active":"2024-04-02T07:52:49.006526+0000","last_peered":"2024-04-02T07:52:49.006526+0000","last_clean":"2024-04-02T07:52:49.006526+0000","last_became_active":"2024-04-02T07:52:48.989320+0000","last_became_peered":"2024-04-02T07:52:48.989320+0000","last_unstale":"2024-04-02T07:52:49.006526+0000","last_undegraded":"2024-04-02T07:52:49.006526+0000","last_fullsized":"2024-04-02T07:52:49.006526+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:43.696202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.990525+0000","last_change":"2024-04-02T07:52:48.986001+0000","last_active":"2024-04-02T07:52:48.990525+0000","last_peered":"2024-04-02T07:52:48.990525+0000","last_clean":"2024-04-02T07:52:48.990525+0000","last_became_active":"2024-04-02T07:52:48.985705+0000","last_became_peered":"2024-04-02T07:52:48.985705+0000","last_unstale":"2024-04-02T07:52:48.990525+0000","last_undegraded":"2024-04-02T07:52:48.990525+0000","last_fullsized":"2024-04-02T07:52:48.990525+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:08:11.549996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983567+0000","last_change":"2024-04-02T07:52:48.990501+0000","last_active":"2024-04-02T07:52:49.983567+0000","last_peered":"2024-04-02T07:52:49.983567+0000","last_clean":"2024-04-02T07:52:49.983567+0000","last_became_active":"2024-04-02T07:52:48.990247+0000","last_became_peered":"2024-04-02T07:52:48.990247+0000","last_unstale":"2024-04-02T07:52:49.983567+0000","last_undegraded":"2024-04-02T07:52:49.983567+0000","last_fullsized":"2024-04-02T07:52:49.983567+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:41:35.832935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982096+0000","last_change":"2024-04-02T07:52:48.987296+0000","last_active":"2024-04-02T07:52:49.982096+0000","last_peered":"2024-04-02T07:52:49.982096+0000","last_clean":"2024-04-02T07:52:49.982096+0000","last_became_active":"2024-04-02T07:52:48.986131+0000","last_became_peered":"2024-04-02T07:52:48.986131+0000","last_unstale":"2024-04-02T07:52:49.982096+0000","last_undegraded":"2024-04-02T07:52:49.982096+0000","last_fullsized":"2024-04-02T07:52:49.982096+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:43.695576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.000556+0000","last_change":"2024-04-02T07:52:48.996381+0000","last_active":"2024-04-02T07:52:49.000556+0000","last_peered":"2024-04-02T07:52:49.000556+0000","last_clean":"2024-04-02T07:52:49.000556+0000","last_became_active":"2024-04-02T07:52:48.995157+0000","last_became_peered":"2024-04-02T07:52:48.995157+0000","last_unstale":"2024-04-02T07:52:49.000556+0000","last_undegraded":"2024-04-02T07:52:49.000556+0000","last_fullsized":"2024-04-02T07:52:49.000556+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:17:18.988774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.998192+0000","last_change":"2024-04-02T07:52:48.991127+0000","last_active":"2024-04-02T07:52:48.998192+0000","last_peered":"2024-04-02T07:52:48.998192+0000","last_clean":"2024-04-02T07:52:48.998192+0000","last_became_active":"2024-04-02T07:52:48.988157+0000","last_became_peered":"2024-04-02T07:52:48.988157+0000","last_unstale":"2024-04-02T07:52:48.998192+0000","last_undegraded":"2024-04-02T07:52:48.998192+0000","last_fullsized":"2024-04-02T07:52:48.998192+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:58.200859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.009404+0000","last_change":"2024-04-02T07:52:49.007749+0000","last_active":"2024-04-02T07:52:49.009404+0000","last_peered":"2024-04-02T07:52:49.009404+0000","last_clean":"2024-04-02T07:52:49.009404+0000","last_became_active":"2024-04-02T07:52:49.007467+0000","last_became_peered":"2024-04-02T07:52:49.007467+0000","last_unstale":"2024-04-02T07:52:49.009404+0000","last_undegraded":"2024-04-02T07:52:49.009404+0000","last_fullsized":"2024-04-02T07:52:49.009404+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:55:04.980012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983908+0000","last_change":"2024-04-02T07:52:49.002468+0000","last_active":"2024-04-02T07:52:49.983908+0000","last_peered":"2024-04-02T07:52:49.983908+0000","last_clean":"2024-04-02T07:52:49.983908+0000","last_became_active":"2024-04-02T07:52:49.001891+0000","last_became_peered":"2024-04-02T07:52:49.001891+0000","last_unstale":"2024-04-02T07:52:49.983908+0000","last_undegraded":"2024-04-02T07:52:49.983908+0000","last_fullsized":"2024-04-02T07:52:49.983908+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:59.387423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983799+0000","last_change":"2024-04-02T07:52:49.001894+0000","last_active":"2024-04-02T07:52:49.983799+0000","last_peered":"2024-04-02T07:52:49.983799+0000","last_clean":"2024-04-02T07:52:49.983799+0000","last_became_active":"2024-04-02T07:52:49.001022+0000","last_became_peered":"2024-04-02T07:52:49.001022+0000","last_unstale":"2024-04-02T07:52:49.983799+0000","last_undegraded":"2024-04-02T07:52:49.983799+0000","last_fullsized":"2024-04-02T07:52:49.983799+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:04:01.924124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982512+0000","last_change":"2024-04-02T07:52:48.990977+0000","last_active":"2024-04-02T07:52:49.982512+0000","last_peered":"2024-04-02T07:52:49.982512+0000","last_clean":"2024-04-02T07:52:49.982512+0000","last_became_active":"2024-04-02T07:52:48.990482+0000","last_became_peered":"2024-04-02T07:52:48.990482+0000","last_unstale":"2024-04-02T07:52:49.982512+0000","last_undegraded":"2024-04-02T07:52:49.982512+0000","last_fullsized":"2024-04-02T07:52:49.982512+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:38.860332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983658+0000","last_change":"2024-04-02T07:52:48.995621+0000","last_active":"2024-04-02T07:52:49.983658+0000","last_peered":"2024-04-02T07:52:49.983658+0000","last_clean":"2024-04-02T07:52:49.983658+0000","last_became_active":"2024-04-02T07:52:48.995157+0000","last_became_peered":"2024-04-02T07:52:48.995157+0000","last_unstale":"2024-04-02T07:52:49.983658+0000","last_undegraded":"2024-04-02T07:52:49.983658+0000","last_fullsized":"2024-04-02T07:52:49.983658+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:41.075062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999254+0000","last_change":"2024-04-02T07:52:48.994369+0000","last_active":"2024-04-02T07:52:48.999254+0000","last_peered":"2024-04-02T07:52:48.999254+0000","last_clean":"2024-04-02T07:52:48.999254+0000","last_became_active":"2024-04-02T07:52:48.989191+0000","last_became_peered":"2024-04-02T07:52:48.989191+0000","last_unstale":"2024-04-02T07:52:48.999254+0000","last_undegraded":"2024-04-02T07:52:48.999254+0000","last_fullsized":"2024-04-02T07:52:48.999254+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:16:27.175143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.005991+0000","last_change":"2024-04-02T07:52:49.003588+0000","last_active":"2024-04-02T07:52:49.005991+0000","last_peered":"2024-04-02T07:52:49.005991+0000","last_clean":"2024-04-02T07:52:49.005991+0000","last_became_active":"2024-04-02T07:52:48.998527+0000","last_became_peered":"2024-04-02T07:52:48.998527+0000","last_unstale":"2024-04-02T07:52:49.005991+0000","last_undegraded":"2024-04-02T07:52:49.005991+0000","last_fullsized":"2024-04-02T07:52:49.005991+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:57:04.305513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.003283+0000","last_change":"2024-04-02T07:52:48.993733+0000","last_active":"2024-04-02T07:52:49.003283+0000","last_peered":"2024-04-02T07:52:49.003283+0000","last_clean":"2024-04-02T07:52:49.003283+0000","last_became_active":"2024-04-02T07:52:48.993476+0000","last_became_peered":"2024-04-02T07:52:48.993476+0000","last_unstale":"2024-04-02T07:52:49.003283+0000","last_undegraded":"2024-04-02T07:52:49.003283+0000","last_fullsized":"2024-04-02T07:52:49.003283+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:11:34.231899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999414+0000","last_change":"2024-04-02T07:52:48.992292+0000","last_active":"2024-04-02T07:52:48.999414+0000","last_peered":"2024-04-02T07:52:48.999414+0000","last_clean":"2024-04-02T07:52:48.999414+0000","last_became_active":"2024-04-02T07:52:48.991798+0000","last_became_peered":"2024-04-02T07:52:48.991798+0000","last_unstale":"2024-04-02T07:52:48.999414+0000","last_undegraded":"2024-04-02T07:52:48.999414+0000","last_fullsized":"2024-04-02T07:52:48.999414+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:32:03.330012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999419+0000","last_change":"2024-04-02T07:52:48.993925+0000","last_active":"2024-04-02T07:52:48.999419+0000","last_peered":"2024-04-02T07:52:48.999419+0000","last_clean":"2024-04-02T07:52:48.999419+0000","last_became_active":"2024-04-02T07:52:48.993481+0000","last_became_peered":"2024-04-02T07:52:48.993481+0000","last_unstale":"2024-04-02T07:52:48.999419+0000","last_undegraded":"2024-04-02T07:52:48.999419+0000","last_fullsized":"2024-04-02T07:52:48.999419+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:13:58.964747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999892+0000","last_change":"2024-04-02T07:52:48.997911+0000","last_active":"2024-04-02T07:52:48.999892+0000","last_peered":"2024-04-02T07:52:48.999892+0000","last_clean":"2024-04-02T07:52:48.999892+0000","last_became_active":"2024-04-02T07:52:48.994626+0000","last_became_peered":"2024-04-02T07:52:48.994626+0000","last_unstale":"2024-04-02T07:52:48.999892+0000","last_undegraded":"2024-04-02T07:52:48.999892+0000","last_fullsized":"2024-04-02T07:52:48.999892+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:53:04.013403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982396+0000","last_change":"2024-04-02T07:52:48.995392+0000","last_active":"2024-04-02T07:52:49.982396+0000","last_peered":"2024-04-02T07:52:49.982396+0000","last_clean":"2024-04-02T07:52:49.982396+0000","last_became_active":"2024-04-02T07:52:48.995081+0000","last_became_peered":"2024-04-02T07:52:48.995081+0000","last_unstale":"2024-04-02T07:52:49.982396+0000","last_undegraded":"2024-04-02T07:52:49.982396+0000","last_fullsized":"2024-04-02T07:52:49.982396+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:36.306309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983656+0000","last_change":"2024-04-02T07:52:48.996745+0000","last_active":"2024-04-02T07:52:49.983656+0000","last_peered":"2024-04-02T07:52:49.983656+0000","last_clean":"2024-04-02T07:52:49.983656+0000","last_became_active":"2024-04-02T07:52:48.996456+0000","last_became_peered":"2024-04-02T07:52:48.996456+0000","last_unstale":"2024-04-02T07:52:49.983656+0000","last_undegraded":"2024-04-02T07:52:49.983656+0000","last_fullsized":"2024-04-02T07:52:49.983656+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:41:00.474374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.997638+0000","last_change":"2024-04-02T07:52:48.992361+0000","last_active":"2024-04-02T07:52:48.997638+0000","last_peered":"2024-04-02T07:52:48.997638+0000","last_clean":"2024-04-02T07:52:48.997638+0000","last_became_active":"2024-04-02T07:52:48.989481+0000","last_became_peered":"2024-04-02T07:52:48.989481+0000","last_unstale":"2024-04-02T07:52:48.997638+0000","last_undegraded":"2024-04-02T07:52:48.997638+0000","last_fullsized":"2024-04-02T07:52:48.997638+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:41.075062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.000884+0000","last_change":"2024-04-02T07:52:48.998651+0000","last_active":"2024-04-02T07:52:49.000884+0000","last_peered":"2024-04-02T07:52:49.000884+0000","last_clean":"2024-04-02T07:52:49.000884+0000","last_became_active":"2024-04-02T07:52:48.998413+0000","last_became_peered":"2024-04-02T07:52:48.998413+0000","last_unstale":"2024-04-02T07:52:49.000884+0000","last_undegraded":"2024-04-02T07:52:49.000884+0000","last_fullsized":"2024-04-02T07:52:49.000884+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:22:27.541973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.985234+0000","last_change":"2024-04-02T07:52:48.982883+0000","last_active":"2024-04-02T07:52:48.985234+0000","last_peered":"2024-04-02T07:52:48.985234+0000","last_clean":"2024-04-02T07:52:48.985234+0000","last_became_active":"2024-04-02T07:52:48.982631+0000","last_became_peered":"2024-04-02T07:52:48.982631+0000","last_unstale":"2024-04-02T07:52:48.985234+0000","last_undegraded":"2024-04-02T07:52:48.985234+0000","last_fullsized":"2024-04-02T07:52:48.985234+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:38:36.306309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"110'4","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.005532+0000","last_change":"2024-04-02T07:52:48.492212+0000","last_active":"2024-04-02T07:52:49.005532+0000","last_peered":"2024-04-02T07:52:49.005532+0000","last_clean":"2024-04-02T07:52:49.005532+0000","last_became_active":"2024-04-02T07:52:48.491899+0000","last_became_peered":"2024-04-02T07:52:48.491899+0000","last_unstale":"2024-04-02T07:52:49.005532+0000","last_undegraded":"2024-04-02T07:52:49.005532+0000","last_fullsized":"2024-04-02T07:52:49.005532+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:21.397120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.004800+0000","last_change":"2024-04-02T07:52:49.002096+0000","last_active":"2024-04-02T07:52:49.004800+0000","last_peered":"2024-04-02T07:52:49.004800+0000","last_clean":"2024-04-02T07:52:49.004800+0000","last_became_active":"2024-04-02T07:52:49.001857+0000","last_became_peered":"2024-04-02T07:52:49.001857+0000","last_unstale":"2024-04-02T07:52:49.004800+0000","last_undegraded":"2024-04-02T07:52:49.004800+0000","last_fullsized":"2024-04-02T07:52:49.004800+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:41:35.832935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1e","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.010031+0000","last_change":"2024-04-02T07:52:48.494135+0000","last_active":"2024-04-02T07:52:49.010031+0000","last_peered":"2024-04-02T07:52:49.010031+0000","last_clean":"2024-04-02T07:52:49.010031+0000","last_became_active":"2024-04-02T07:52:48.493882+0000","last_became_peered":"2024-04-02T07:52:48.493882+0000","last_unstale":"2024-04-02T07:52:49.010031+0000","last_undegraded":"2024-04-02T07:52:49.010031+0000","last_fullsized":"2024-04-02T07:52:49.010031+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:23:06.484479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.006545+0000","last_change":"2024-04-02T07:52:49.003117+0000","last_active":"2024-04-02T07:52:49.006545+0000","last_peered":"2024-04-02T07:52:49.006545+0000","last_clean":"2024-04-02T07:52:49.006545+0000","last_became_active":"2024-04-02T07:52:48.996406+0000","last_became_peered":"2024-04-02T07:52:48.996406+0000","last_unstale":"2024-04-02T07:52:49.006545+0000","last_undegraded":"2024-04-02T07:52:49.006545+0000","last_fullsized":"2024-04-02T07:52:49.006545+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:22.405783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1d","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.002010+0000","last_change":"2024-04-02T07:52:48.491033+0000","last_active":"2024-04-02T07:52:49.002010+0000","last_peered":"2024-04-02T07:52:49.002010+0000","last_clean":"2024-04-02T07:52:49.002010+0000","last_became_active":"2024-04-02T07:52:48.490465+0000","last_became_peered":"2024-04-02T07:52:48.490465+0000","last_unstale":"2024-04-02T07:52:49.002010+0000","last_undegraded":"2024-04-02T07:52:49.002010+0000","last_fullsized":"2024-04-02T07:52:49.002010+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:43:22.090288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.004136+0000","last_change":"2024-04-02T07:52:49.000838+0000","last_active":"2024-04-02T07:52:49.004136+0000","last_peered":"2024-04-02T07:52:49.004136+0000","last_clean":"2024-04-02T07:52:49.004136+0000","last_became_active":"2024-04-02T07:52:49.000177+0000","last_became_peered":"2024-04-02T07:52:49.000177+0000","last_unstale":"2024-04-02T07:52:49.004136+0000","last_undegraded":"2024-04-02T07:52:49.004136+0000","last_fullsized":"2024-04-02T07:52:49.004136+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:13:13.336296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1c","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982776+0000","last_change":"2024-04-02T07:52:48.495550+0000","last_active":"2024-04-02T07:52:49.982776+0000","last_peered":"2024-04-02T07:52:49.982776+0000","last_clean":"2024-04-02T07:52:49.982776+0000","last_became_active":"2024-04-02T07:52:48.495305+0000","last_became_peered":"2024-04-02T07:52:48.495305+0000","last_unstale":"2024-04-02T07:52:49.982776+0000","last_undegraded":"2024-04-02T07:52:49.982776+0000","last_fullsized":"2024-04-02T07:52:49.982776+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:41:54.857786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982644+0000","last_change":"2024-04-02T07:52:48.994969+0000","last_active":"2024-04-02T07:52:49.982644+0000","last_peered":"2024-04-02T07:52:49.982644+0000","last_clean":"2024-04-02T07:52:49.982644+0000","last_became_active":"2024-04-02T07:52:48.994519+0000","last_became_peered":"2024-04-02T07:52:48.994519+0000","last_unstale":"2024-04-02T07:52:49.982644+0000","last_undegraded":"2024-04-02T07:52:49.982644+0000","last_fullsized":"2024-04-02T07:52:49.982644+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:58.200859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1b","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.009246+0000","last_change":"2024-04-02T07:52:48.489078+0000","last_active":"2024-04-02T07:52:49.009246+0000","last_peered":"2024-04-02T07:52:49.009246+0000","last_clean":"2024-04-02T07:52:49.009246+0000","last_became_active":"2024-04-02T07:52:48.488812+0000","last_became_peered":"2024-04-02T07:52:48.488812+0000","last_unstale":"2024-04-02T07:52:49.009246+0000","last_undegraded":"2024-04-02T07:52:49.009246+0000","last_fullsized":"2024-04-02T07:52:49.009246+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:49:09.118777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.997467+0000","last_change":"2024-04-02T07:52:48.988912+0000","last_active":"2024-04-02T07:52:48.997467+0000","last_peered":"2024-04-02T07:52:48.997467+0000","last_clean":"2024-04-02T07:52:48.997467+0000","last_became_active":"2024-04-02T07:52:48.988649+0000","last_became_peered":"2024-04-02T07:52:48.988649+0000","last_unstale":"2024-04-02T07:52:48.997467+0000","last_undegraded":"2024-04-02T07:52:48.997467+0000","last_fullsized":"2024-04-02T07:52:48.997467+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:21:44.892940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1a","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.008106+0000","last_change":"2024-04-02T07:52:48.490637+0000","last_active":"2024-04-02T07:52:49.008106+0000","last_peered":"2024-04-02T07:52:49.008106+0000","last_clean":"2024-04-02T07:52:49.008106+0000","last_became_active":"2024-04-02T07:52:48.490038+0000","last_became_peered":"2024-04-02T07:52:48.490038+0000","last_unstale":"2024-04-02T07:52:49.008106+0000","last_undegraded":"2024-04-02T07:52:49.008106+0000","last_fullsized":"2024-04-02T07:52:49.008106+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:06:02.271528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.010066+0000","last_change":"2024-04-02T07:52:49.007001+0000","last_active":"2024-04-02T07:52:49.010066+0000","last_peered":"2024-04-02T07:52:49.010066+0000","last_clean":"2024-04-02T07:52:49.010066+0000","last_became_active":"2024-04-02T07:52:49.006738+0000","last_became_peered":"2024-04-02T07:52:49.006738+0000","last_unstale":"2024-04-02T07:52:49.010066+0000","last_undegraded":"2024-04-02T07:52:49.010066+0000","last_fullsized":"2024-04-02T07:52:49.010066+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:41.075062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983467+0000","last_change":"2024-04-02T07:52:48.484996+0000","last_active":"2024-04-02T07:52:49.983467+0000","last_peered":"2024-04-02T07:52:49.983467+0000","last_clean":"2024-04-02T07:52:49.983467+0000","last_became_active":"2024-04-02T07:52:48.484692+0000","last_became_peered":"2024-04-02T07:52:48.484692+0000","last_unstale":"2024-04-02T07:52:49.983467+0000","last_undegraded":"2024-04-02T07:52:49.983467+0000","last_fullsized":"2024-04-02T07:52:49.983467+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:57:46.918091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983420+0000","last_change":"2024-04-02T07:52:48.990452+0000","last_active":"2024-04-02T07:52:49.983420+0000","last_peered":"2024-04-02T07:52:49.983420+0000","last_clean":"2024-04-02T07:52:49.983420+0000","last_became_active":"2024-04-02T07:52:48.990155+0000","last_became_peered":"2024-04-02T07:52:48.990155+0000","last_unstale":"2024-04-02T07:52:49.983420+0000","last_undegraded":"2024-04-02T07:52:49.983420+0000","last_fullsized":"2024-04-02T07:52:49.983420+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:16.097343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983253+0000","last_change":"2024-04-02T07:52:48.495555+0000","last_active":"2024-04-02T07:52:49.983253+0000","last_peered":"2024-04-02T07:52:49.983253+0000","last_clean":"2024-04-02T07:52:49.983253+0000","last_became_active":"2024-04-02T07:52:48.495287+0000","last_became_peered":"2024-04-02T07:52:48.495287+0000","last_unstale":"2024-04-02T07:52:49.983253+0000","last_undegraded":"2024-04-02T07:52:49.983253+0000","last_fullsized":"2024-04-02T07:52:49.983253+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:31:10.337978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983992+0000","last_change":"2024-04-02T07:52:48.998856+0000","last_active":"2024-04-02T07:52:49.983992+0000","last_peered":"2024-04-02T07:52:49.983992+0000","last_clean":"2024-04-02T07:52:49.983992+0000","last_became_active":"2024-04-02T07:52:48.997924+0000","last_became_peered":"2024-04-02T07:52:48.997924+0000","last_unstale":"2024-04-02T07:52:49.983992+0000","last_undegraded":"2024-04-02T07:52:49.983992+0000","last_fullsized":"2024-04-02T07:52:49.983992+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:29.124428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.986991+0000","last_change":"2024-04-02T07:52:48.489345+0000","last_active":"2024-04-02T07:52:48.986991+0000","last_peered":"2024-04-02T07:52:48.986991+0000","last_clean":"2024-04-02T07:52:48.986991+0000","last_became_active":"2024-04-02T07:52:48.488890+0000","last_became_peered":"2024-04-02T07:52:48.488890+0000","last_unstale":"2024-04-02T07:52:48.986991+0000","last_undegraded":"2024-04-02T07:52:48.986991+0000","last_fullsized":"2024-04-02T07:52:48.986991+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:56:18.611413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.007917+0000","last_change":"2024-04-02T07:52:49.004722+0000","last_active":"2024-04-02T07:52:49.007917+0000","last_peered":"2024-04-02T07:52:49.007917+0000","last_clean":"2024-04-02T07:52:49.007917+0000","last_became_active":"2024-04-02T07:52:49.004378+0000","last_became_peered":"2024-04-02T07:52:49.004378+0000","last_unstale":"2024-04-02T07:52:49.007917+0000","last_undegraded":"2024-04-02T07:52:49.007917+0000","last_fullsized":"2024-04-02T07:52:49.007917+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:22:27.541973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.981561+0000","last_change":"2024-04-02T07:52:48.494163+0000","last_active":"2024-04-02T07:52:48.981561+0000","last_peered":"2024-04-02T07:52:48.981561+0000","last_clean":"2024-04-02T07:52:48.981561+0000","last_became_active":"2024-04-02T07:52:48.493111+0000","last_became_peered":"2024-04-02T07:52:48.493111+0000","last_unstale":"2024-04-02T07:52:48.981561+0000","last_undegraded":"2024-04-02T07:52:48.981561+0000","last_fullsized":"2024-04-02T07:52:48.981561+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:57:03.296849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.003552+0000","last_change":"2024-04-02T07:52:48.991086+0000","last_active":"2024-04-02T07:52:49.003552+0000","last_peered":"2024-04-02T07:52:49.003552+0000","last_clean":"2024-04-02T07:52:49.003552+0000","last_became_active":"2024-04-02T07:52:48.990733+0000","last_became_peered":"2024-04-02T07:52:48.990733+0000","last_unstale":"2024-04-02T07:52:49.003552+0000","last_undegraded":"2024-04-02T07:52:49.003552+0000","last_fullsized":"2024-04-02T07:52:49.003552+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:59.387423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.15","version":"110'3","reported_seq":13,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.009476+0000","last_change":"2024-04-02T07:52:48.494720+0000","last_active":"2024-04-02T07:52:49.009476+0000","last_peered":"2024-04-02T07:52:49.009476+0000","last_clean":"2024-04-02T07:52:49.009476+0000","last_became_active":"2024-04-02T07:52:48.494426+0000","last_became_peered":"2024-04-02T07:52:48.494426+0000","last_unstale":"2024-04-02T07:52:49.009476+0000","last_undegraded":"2024-04-02T07:52:49.009476+0000","last_fullsized":"2024-04-02T07:52:49.009476+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:31:10.337978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.997950+0000","last_change":"2024-04-02T07:52:48.991803+0000","last_active":"2024-04-02T07:52:48.997950+0000","last_peered":"2024-04-02T07:52:48.997950+0000","last_clean":"2024-04-02T07:52:48.997950+0000","last_became_active":"2024-04-02T07:52:48.991399+0000","last_became_peered":"2024-04-02T07:52:48.991399+0000","last_unstale":"2024-04-02T07:52:48.997950+0000","last_undegraded":"2024-04-02T07:52:48.997950+0000","last_fullsized":"2024-04-02T07:52:48.997950+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:10:20.627252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.14","version":"110'6","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.012826+0000","last_change":"2024-04-02T07:52:48.485211+0000","last_active":"2024-04-02T07:52:49.012826+0000","last_peered":"2024-04-02T07:52:49.012826+0000","last_clean":"2024-04-02T07:52:49.012826+0000","last_became_active":"2024-04-02T07:52:48.484715+0000","last_became_peered":"2024-04-02T07:52:48.484715+0000","last_unstale":"2024-04-02T07:52:49.012826+0000","last_undegraded":"2024-04-02T07:52:49.012826+0000","last_fullsized":"2024-04-02T07:52:49.012826+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:58:21.014343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.998161+0000","last_change":"2024-04-02T07:52:48.989787+0000","last_active":"2024-04-02T07:52:48.998161+0000","last_peered":"2024-04-02T07:52:48.998161+0000","last_clean":"2024-04-02T07:52:48.998161+0000","last_became_active":"2024-04-02T07:52:48.989488+0000","last_became_peered":"2024-04-02T07:52:48.989488+0000","last_unstale":"2024-04-02T07:52:48.998161+0000","last_undegraded":"2024-04-02T07:52:48.998161+0000","last_fullsized":"2024-04-02T07:52:48.998161+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:05:38.004241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.13","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982615+0000","last_change":"2024-04-02T07:52:48.491042+0000","last_active":"2024-04-02T07:52:49.982615+0000","last_peered":"2024-04-02T07:52:49.982615+0000","last_clean":"2024-04-02T07:52:49.982615+0000","last_became_active":"2024-04-02T07:52:48.490086+0000","last_became_peered":"2024-04-02T07:52:48.490086+0000","last_unstale":"2024-04-02T07:52:49.982615+0000","last_undegraded":"2024-04-02T07:52:49.982615+0000","last_fullsized":"2024-04-02T07:52:49.982615+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:09:28.390077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.002507+0000","last_change":"2024-04-02T07:52:49.001717+0000","last_active":"2024-04-02T07:52:49.002507+0000","last_peered":"2024-04-02T07:52:49.002507+0000","last_clean":"2024-04-02T07:52:49.002507+0000","last_became_active":"2024-04-02T07:52:49.000518+0000","last_became_peered":"2024-04-02T07:52:49.000518+0000","last_unstale":"2024-04-02T07:52:49.002507+0000","last_undegraded":"2024-04-02T07:52:49.002507+0000","last_fullsized":"2024-04-02T07:52:49.002507+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:21:44.892940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982075+0000","last_change":"2024-04-02T07:52:48.493071+0000","last_active":"2024-04-02T07:52:49.982075+0000","last_peered":"2024-04-02T07:52:49.982075+0000","last_clean":"2024-04-02T07:52:49.982075+0000","last_became_active":"2024-04-02T07:52:48.492221+0000","last_became_peered":"2024-04-02T07:52:48.492221+0000","last_unstale":"2024-04-02T07:52:49.982075+0000","last_undegraded":"2024-04-02T07:52:49.982075+0000","last_fullsized":"2024-04-02T07:52:49.982075+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:05:36.995577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983263+0000","last_change":"2024-04-02T07:52:49.005606+0000","last_active":"2024-04-02T07:52:49.983263+0000","last_peered":"2024-04-02T07:52:49.983263+0000","last_clean":"2024-04-02T07:52:49.983263+0000","last_became_active":"2024-04-02T07:52:49.005257+0000","last_became_peered":"2024-04-02T07:52:49.005257+0000","last_unstale":"2024-04-02T07:52:49.983263+0000","last_undegraded":"2024-04-02T07:52:49.983263+0000","last_fullsized":"2024-04-02T07:52:49.983263+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:43.695576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.979689+0000","last_change":"2024-04-02T07:52:48.496436+0000","last_active":"2024-04-02T07:52:48.979689+0000","last_peered":"2024-04-02T07:52:48.979689+0000","last_clean":"2024-04-02T07:52:48.979689+0000","last_became_active":"2024-04-02T07:52:48.496179+0000","last_became_peered":"2024-04-02T07:52:48.496179+0000","last_unstale":"2024-04-02T07:52:48.979689+0000","last_undegraded":"2024-04-02T07:52:48.979689+0000","last_fullsized":"2024-04-02T07:52:48.979689+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:07:13.484062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.005629+0000","last_change":"2024-04-02T07:52:49.002985+0000","last_active":"2024-04-02T07:52:49.005629+0000","last_peered":"2024-04-02T07:52:49.005629+0000","last_clean":"2024-04-02T07:52:49.005629+0000","last_became_active":"2024-04-02T07:52:48.997846+0000","last_became_peered":"2024-04-02T07:52:48.997846+0000","last_unstale":"2024-04-02T07:52:49.005629+0000","last_undegraded":"2024-04-02T07:52:49.005629+0000","last_fullsized":"2024-04-02T07:52:49.005629+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:22:39.816893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.10","version":"110'8","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.010883+0000","last_change":"2024-04-02T07:52:48.496481+0000","last_active":"2024-04-02T07:52:49.010883+0000","last_peered":"2024-04-02T07:52:49.010883+0000","last_clean":"2024-04-02T07:52:49.010883+0000","last_became_active":"2024-04-02T07:52:48.496149+0000","last_became_peered":"2024-04-02T07:52:48.496149+0000","last_unstale":"2024-04-02T07:52:49.010883+0000","last_undegraded":"2024-04-02T07:52:49.010883+0000","last_fullsized":"2024-04-02T07:52:49.010883+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:22:03.224804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.010241+0000","last_change":"2024-04-02T07:52:49.006851+0000","last_active":"2024-04-02T07:52:49.010241+0000","last_peered":"2024-04-02T07:52:49.010241+0000","last_clean":"2024-04-02T07:52:49.010241+0000","last_became_active":"2024-04-02T07:52:48.996690+0000","last_became_peered":"2024-04-02T07:52:48.996690+0000","last_unstale":"2024-04-02T07:52:49.010241+0000","last_undegraded":"2024-04-02T07:52:49.010241+0000","last_fullsized":"2024-04-02T07:52:49.010241+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:38.860332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.f","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.007149+0000","last_change":"2024-04-02T07:52:48.488279+0000","last_active":"2024-04-02T07:52:49.007149+0000","last_peered":"2024-04-02T07:52:49.007149+0000","last_clean":"2024-04-02T07:52:49.007149+0000","last_became_active":"2024-04-02T07:52:48.487450+0000","last_became_peered":"2024-04-02T07:52:48.487450+0000","last_unstale":"2024-04-02T07:52:49.007149+0000","last_undegraded":"2024-04-02T07:52:49.007149+0000","last_fullsized":"2024-04-02T07:52:49.007149+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:42.687539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.004212+0000","last_change":"2024-04-02T07:52:49.002363+0000","last_active":"2024-04-02T07:52:49.004212+0000","last_peered":"2024-04-02T07:52:49.004212+0000","last_clean":"2024-04-02T07:52:49.004212+0000","last_became_active":"2024-04-02T07:52:49.002094+0000","last_became_peered":"2024-04-02T07:52:49.002094+0000","last_unstale":"2024-04-02T07:52:49.004212+0000","last_undegraded":"2024-04-02T07:52:49.004212+0000","last_fullsized":"2024-04-02T07:52:49.004212+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:22:17.864289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.e","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983995+0000","last_change":"2024-04-02T07:52:48.490888+0000","last_active":"2024-04-02T07:52:49.983995+0000","last_peered":"2024-04-02T07:52:49.983995+0000","last_clean":"2024-04-02T07:52:49.983995+0000","last_became_active":"2024-04-02T07:52:48.490529+0000","last_became_peered":"2024-04-02T07:52:48.490529+0000","last_unstale":"2024-04-02T07:52:49.983995+0000","last_undegraded":"2024-04-02T07:52:49.983995+0000","last_fullsized":"2024-04-02T07:52:49.983995+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:26:50.809967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.004282+0000","last_change":"2024-04-02T07:52:49.002825+0000","last_active":"2024-04-02T07:52:49.004282+0000","last_peered":"2024-04-02T07:52:49.004282+0000","last_clean":"2024-04-02T07:52:49.004282+0000","last_became_active":"2024-04-02T07:52:49.002541+0000","last_became_peered":"2024-04-02T07:52:49.002541+0000","last_unstale":"2024-04-02T07:52:49.004282+0000","last_undegraded":"2024-04-02T07:52:49.004282+0000","last_fullsized":"2024-04-02T07:52:49.004282+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:31:35.288356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983770+0000","last_change":"2024-04-02T07:52:48.486374+0000","last_active":"2024-04-02T07:52:49.983770+0000","last_peered":"2024-04-02T07:52:49.983770+0000","last_clean":"2024-04-02T07:52:49.983770+0000","last_became_active":"2024-04-02T07:52:48.486083+0000","last_became_peered":"2024-04-02T07:52:48.486083+0000","last_unstale":"2024-04-02T07:52:49.983770+0000","last_undegraded":"2024-04-02T07:52:49.983770+0000","last_fullsized":"2024-04-02T07:52:49.983770+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:39:31.031890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982011+0000","last_change":"2024-04-02T07:52:48.995443+0000","last_active":"2024-04-02T07:52:49.982011+0000","last_peered":"2024-04-02T07:52:49.982011+0000","last_clean":"2024-04-02T07:52:49.982011+0000","last_became_active":"2024-04-02T07:52:48.995157+0000","last_became_peered":"2024-04-02T07:52:48.995157+0000","last_unstale":"2024-04-02T07:52:49.982011+0000","last_undegraded":"2024-04-02T07:52:49.982011+0000","last_fullsized":"2024-04-02T07:52:49.982011+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:41.075062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983764+0000","last_change":"2024-04-02T07:52:48.490860+0000","last_active":"2024-04-02T07:52:49.983764+0000","last_peered":"2024-04-02T07:52:49.983764+0000","last_clean":"2024-04-02T07:52:49.983764+0000","last_became_active":"2024-04-02T07:52:48.490102+0000","last_became_peered":"2024-04-02T07:52:48.490102+0000","last_unstale":"2024-04-02T07:52:49.983764+0000","last_undegraded":"2024-04-02T07:52:49.983764+0000","last_fullsized":"2024-04-02T07:52:49.983764+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:56:18.611413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.005345+0000","last_change":"2024-04-02T07:52:49.001085+0000","last_active":"2024-04-02T07:52:49.005345+0000","last_peered":"2024-04-02T07:52:49.005345+0000","last_clean":"2024-04-02T07:52:49.005345+0000","last_became_active":"2024-04-02T07:52:49.000813+0000","last_became_peered":"2024-04-02T07:52:49.000813+0000","last_unstale":"2024-04-02T07:52:49.005345+0000","last_undegraded":"2024-04-02T07:52:49.005345+0000","last_fullsized":"2024-04-02T07:52:49.005345+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:15.851298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982132+0000","last_change":"2024-04-02T07:52:48.493649+0000","last_active":"2024-04-02T07:52:49.982132+0000","last_peered":"2024-04-02T07:52:49.982132+0000","last_clean":"2024-04-02T07:52:49.982132+0000","last_became_active":"2024-04-02T07:52:48.493371+0000","last_became_peered":"2024-04-02T07:52:48.493371+0000","last_unstale":"2024-04-02T07:52:49.982132+0000","last_undegraded":"2024-04-02T07:52:49.982132+0000","last_fullsized":"2024-04-02T07:52:49.982132+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:17:03.086138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.981833+0000","last_change":"2024-04-02T07:52:48.989357+0000","last_active":"2024-04-02T07:52:49.981833+0000","last_peered":"2024-04-02T07:52:49.981833+0000","last_clean":"2024-04-02T07:52:49.981833+0000","last_became_active":"2024-04-02T07:52:48.988857+0000","last_became_peered":"2024-04-02T07:52:48.988857+0000","last_unstale":"2024-04-02T07:52:49.981833+0000","last_undegraded":"2024-04-02T07:52:49.981833+0000","last_fullsized":"2024-04-02T07:52:49.981833+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:33:22.405783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.984651+0000","last_change":"2024-04-02T07:52:48.497577+0000","last_active":"2024-04-02T07:52:48.984651+0000","last_peered":"2024-04-02T07:52:48.984651+0000","last_clean":"2024-04-02T07:52:48.984651+0000","last_became_active":"2024-04-02T07:52:48.497275+0000","last_became_peered":"2024-04-02T07:52:48.497275+0000","last_unstale":"2024-04-02T07:52:48.984651+0000","last_undegraded":"2024-04-02T07:52:48.984651+0000","last_fullsized":"2024-04-02T07:52:48.984651+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:23.927722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999274+0000","last_change":"2024-04-02T07:52:48.996785+0000","last_active":"2024-04-02T07:52:48.999274+0000","last_peered":"2024-04-02T07:52:48.999274+0000","last_clean":"2024-04-02T07:52:48.999274+0000","last_became_active":"2024-04-02T07:52:48.995473+0000","last_became_peered":"2024-04-02T07:52:48.995473+0000","last_unstale":"2024-04-02T07:52:48.999274+0000","last_undegraded":"2024-04-02T07:52:48.999274+0000","last_fullsized":"2024-04-02T07:52:48.999274+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:01.482800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.983342+0000","last_change":"2024-04-02T07:52:48.491275+0000","last_active":"2024-04-02T07:52:49.983342+0000","last_peered":"2024-04-02T07:52:49.983342+0000","last_clean":"2024-04-02T07:52:49.983342+0000","last_became_active":"2024-04-02T07:52:48.489789+0000","last_became_peered":"2024-04-02T07:52:48.489789+0000","last_unstale":"2024-04-02T07:52:49.983342+0000","last_undegraded":"2024-04-02T07:52:49.983342+0000","last_fullsized":"2024-04-02T07:52:49.983342+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:34:54.355423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999391+0000","last_change":"2024-04-02T07:52:48.991923+0000","last_active":"2024-04-02T07:52:48.999391+0000","last_peered":"2024-04-02T07:52:48.999391+0000","last_clean":"2024-04-02T07:52:48.999391+0000","last_became_active":"2024-04-02T07:52:48.991659+0000","last_became_peered":"2024-04-02T07:52:48.991659+0000","last_unstale":"2024-04-02T07:52:48.999391+0000","last_undegraded":"2024-04-02T07:52:48.999391+0000","last_fullsized":"2024-04-02T07:52:48.999391+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:30:49.706989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.984019+0000","last_change":"2024-04-02T07:52:48.496716+0000","last_active":"2024-04-02T07:52:49.984019+0000","last_peered":"2024-04-02T07:52:49.984019+0000","last_clean":"2024-04-02T07:52:49.984019+0000","last_became_active":"2024-04-02T07:52:48.496269+0000","last_became_peered":"2024-04-02T07:52:48.496269+0000","last_unstale":"2024-04-02T07:52:49.984019+0000","last_undegraded":"2024-04-02T07:52:49.984019+0000","last_fullsized":"2024-04-02T07:52:49.984019+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:21:12.888674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.000974+0000","last_change":"2024-04-02T07:52:48.995412+0000","last_active":"2024-04-02T07:52:49.000974+0000","last_peered":"2024-04-02T07:52:49.000974+0000","last_clean":"2024-04-02T07:52:49.000974+0000","last_became_active":"2024-04-02T07:52:48.995173+0000","last_became_peered":"2024-04-02T07:52:48.995173+0000","last_unstale":"2024-04-02T07:52:49.000974+0000","last_undegraded":"2024-04-02T07:52:49.000974+0000","last_fullsized":"2024-04-02T07:52:49.000974+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:36:23.265060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.989609+0000","last_change":"2024-04-02T07:52:48.492490+0000","last_active":"2024-04-02T07:52:48.989609+0000","last_peered":"2024-04-02T07:52:48.989609+0000","last_clean":"2024-04-02T07:52:48.989609+0000","last_became_active":"2024-04-02T07:52:48.491324+0000","last_became_peered":"2024-04-02T07:52:48.491324+0000","last_unstale":"2024-04-02T07:52:48.989609+0000","last_undegraded":"2024-04-02T07:52:48.989609+0000","last_fullsized":"2024-04-02T07:52:48.989609+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:37:19.717506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.999456+0000","last_change":"2024-04-02T07:52:48.993034+0000","last_active":"2024-04-02T07:52:48.999456+0000","last_peered":"2024-04-02T07:52:48.999456+0000","last_clean":"2024-04-02T07:52:48.999456+0000","last_became_active":"2024-04-02T07:52:48.992764+0000","last_became_peered":"2024-04-02T07:52:48.992764+0000","last_unstale":"2024-04-02T07:52:48.999456+0000","last_undegraded":"2024-04-02T07:52:48.999456+0000","last_fullsized":"2024-04-02T07:52:48.999456+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:47.131307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.975174+0000","last_change":"2024-04-02T07:52:48.493638+0000","last_active":"2024-04-02T07:52:48.975174+0000","last_peered":"2024-04-02T07:52:48.975174+0000","last_clean":"2024-04-02T07:52:48.975174+0000","last_became_active":"2024-04-02T07:52:48.493328+0000","last_became_peered":"2024-04-02T07:52:48.493328+0000","last_unstale":"2024-04-02T07:52:48.975174+0000","last_undegraded":"2024-04-02T07:52:48.975174+0000","last_fullsized":"2024-04-02T07:52:48.975174+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:02:56.034403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.996751+0000","last_change":"2024-04-02T07:52:48.987746+0000","last_active":"2024-04-02T07:52:48.996751+0000","last_peered":"2024-04-02T07:52:48.996751+0000","last_clean":"2024-04-02T07:52:48.996751+0000","last_became_active":"2024-04-02T07:52:48.987478+0000","last_became_peered":"2024-04-02T07:52:48.987478+0000","last_unstale":"2024-04-02T07:52:48.996751+0000","last_undegraded":"2024-04-02T07:52:48.996751+0000","last_fullsized":"2024-04-02T07:52:48.996751+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:12:19.526956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.978798+0000","last_change":"2024-04-02T07:52:48.489650+0000","last_active":"2024-04-02T07:52:48.978798+0000","last_peered":"2024-04-02T07:52:48.978798+0000","last_clean":"2024-04-02T07:52:48.978798+0000","last_became_active":"2024-04-02T07:52:48.488446+0000","last_became_peered":"2024-04-02T07:52:48.488446+0000","last_unstale":"2024-04-02T07:52:48.978798+0000","last_undegraded":"2024-04-02T07:52:48.978798+0000","last_fullsized":"2024-04-02T07:52:48.978798+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:39:31.031890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.004421+0000","last_change":"2024-04-02T07:52:49.003102+0000","last_active":"2024-04-02T07:52:49.004421+0000","last_peered":"2024-04-02T07:52:49.004421+0000","last_clean":"2024-04-02T07:52:49.004421+0000","last_became_active":"2024-04-02T07:52:49.000556+0000","last_became_peered":"2024-04-02T07:52:49.000556+0000","last_unstale":"2024-04-02T07:52:49.004421+0000","last_undegraded":"2024-04-02T07:52:49.004421+0000","last_fullsized":"2024-04-02T07:52:49.004421+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:16.097343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.982539+0000","last_change":"2024-04-02T07:52:48.498143+0000","last_active":"2024-04-02T07:52:49.982539+0000","last_peered":"2024-04-02T07:52:49.982539+0000","last_clean":"2024-04-02T07:52:49.982539+0000","last_became_active":"2024-04-02T07:52:48.497870+0000","last_became_peered":"2024-04-02T07:52:48.497870+0000","last_unstale":"2024-04-02T07:52:49.982539+0000","last_undegraded":"2024-04-02T07:52:49.982539+0000","last_fullsized":"2024-04-02T07:52:49.982539+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:11:33.223235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.005077+0000","last_change":"2024-04-02T07:52:49.001571+0000","last_active":"2024-04-02T07:52:49.005077+0000","last_peered":"2024-04-02T07:52:49.005077+0000","last_clean":"2024-04-02T07:52:49.005077+0000","last_became_active":"2024-04-02T07:52:48.998760+0000","last_became_peered":"2024-04-02T07:52:48.998760+0000","last_unstale":"2024-04-02T07:52:49.005077+0000","last_undegraded":"2024-04-02T07:52:49.005077+0000","last_fullsized":"2024-04-02T07:52:49.005077+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:17:04.094801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.990340+0000","last_change":"2024-04-02T07:52:48.489893+0000","last_active":"2024-04-02T07:52:48.990340+0000","last_peered":"2024-04-02T07:52:48.990340+0000","last_clean":"2024-04-02T07:52:48.990340+0000","last_became_active":"2024-04-02T07:52:48.489617+0000","last_became_peered":"2024-04-02T07:52:48.489617+0000","last_unstale":"2024-04-02T07:52:48.990340+0000","last_undegraded":"2024-04-02T07:52:48.990340+0000","last_fullsized":"2024-04-02T07:52:48.990340+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:22:38.808229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:49.006903+0000","last_change":"2024-04-02T07:52:49.003591+0000","last_active":"2024-04-02T07:52:49.006903+0000","last_peered":"2024-04-02T07:52:49.006903+0000","last_clean":"2024-04-02T07:52:49.006903+0000","last_became_active":"2024-04-02T07:52:49.003331+0000","last_became_peered":"2024-04-02T07:52:49.003331+0000","last_unstale":"2024-04-02T07:52:49.006903+0000","last_undegraded":"2024-04-02T07:52:49.006903+0000","last_fullsized":"2024-04-02T07:52:49.006903+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:11:34.231899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.984669+0000","last_change":"2024-04-02T07:52:48.495565+0000","last_active":"2024-04-02T07:52:48.984669+0000","last_peered":"2024-04-02T07:52:48.984669+0000","last_clean":"2024-04-02T07:52:48.984669+0000","last_became_active":"2024-04-02T07:52:48.491504+0000","last_became_peered":"2024-04-02T07:52:48.491504+0000","last_unstale":"2024-04-02T07:52:48.984669+0000","last_undegraded":"2024-04-02T07:52:48.984669+0000","last_fullsized":"2024-04-02T07:52:48.984669+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:39:31.031890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.995261+0000","last_change":"2024-04-02T07:52:48.992426+0000","last_active":"2024-04-02T07:52:48.995261+0000","last_peered":"2024-04-02T07:52:48.995261+0000","last_clean":"2024-04-02T07:52:48.995261+0000","last_became_active":"2024-04-02T07:52:48.991349+0000","last_became_peered":"2024-04-02T07:52:48.991349+0000","last_unstale":"2024-04-02T07:52:48.995261+0000","last_undegraded":"2024-04-02T07:52:48.995261+0000","last_fullsized":"2024-04-02T07:52:48.995261+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:04:01.924124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.975086+0000","last_change":"2024-04-02T07:52:48.485630+0000","last_active":"2024-04-02T07:52:48.975086+0000","last_peered":"2024-04-02T07:52:48.975086+0000","last_clean":"2024-04-02T07:52:48.975086+0000","last_became_active":"2024-04-02T07:52:48.483333+0000","last_became_peered":"2024-04-02T07:52:48.483333+0000","last_unstale":"2024-04-02T07:52:48.975086+0000","last_undegraded":"2024-04-02T07:52:48.975086+0000","last_fullsized":"2024-04-02T07:52:48.975086+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:34:43.091007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.998524+0000","last_change":"2024-04-02T07:52:48.990862+0000","last_active":"2024-04-02T07:52:48.998524+0000","last_peered":"2024-04-02T07:52:48.998524+0000","last_clean":"2024-04-02T07:52:48.998524+0000","last_became_active":"2024-04-02T07:52:48.989925+0000","last_became_peered":"2024-04-02T07:52:48.989925+0000","last_unstale":"2024-04-02T07:52:48.998524+0000","last_undegraded":"2024-04-02T07:52:48.998524+0000","last_fullsized":"2024-04-02T07:52:48.998524+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:48.464935+0000","last_clean_scrub_stamp":"2024-04-02T07:52:48.464935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:16.097343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.d","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.994105+0000","last_change":"2024-04-02T07:52:48.486362+0000","last_active":"2024-04-02T07:52:48.994105+0000","last_peered":"2024-04-02T07:52:48.994105+0000","last_clean":"2024-04-02T07:52:48.994105+0000","last_became_active":"2024-04-02T07:52:48.486053+0000","last_became_peered":"2024-04-02T07:52:48.486053+0000","last_unstale":"2024-04-02T07:52:48.994105+0000","last_undegraded":"2024-04-02T07:52:48.994105+0000","last_fullsized":"2024-04-02T07:52:48.994105+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:52:47.456272+0000","last_clean_scrub_stamp":"2024-04-02T07:52:47.456272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:57.192196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"100'183","reported_seq":483,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-02T07:52:48.481485+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:52:48.481485+0000","last_peered":"2024-04-02T07:52:48.481485+0000","last_clean":"2024-04-02T07:52:48.481485+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:52:48.481485+0000","last_undegraded":"2024-04-02T07:52:48.481485+0000","last_fullsized":"2024-04-02T07:52:48.481485+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575049,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1368064,"data_stored":668472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.65000000000000002,"15min":0.67800000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.316},"max":{"1min":0.89400000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66400000000000003,"15min":0.68200000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.33200000000000002},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.53800000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65800000000000003,"15min":0.67900000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.65200000000000002,"15min":0.67800000000000005},"min":{"1min":0.42499999999999999,"5min":0.379,"15min":0.34699999999999998},"max":{"1min":0.98299999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.69199999999999995,"15min":0.70999999999999996},"min":{"1min":0.39500000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.94199999999999995,"5min":1.127,"15min":1.127},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.73599999999999999,"15min":0.76100000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":1.044,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.61799999999999999}]},{"osd":3,"last update":"Tue Apr 2 07:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.67300000000000004,"15min":0.70299999999999996},"min":{"1min":0.40600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93500000000000005,"5min":0.97799999999999998,"15min":1.026},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.624,"5min":0.69599999999999995,"15min":0.71999999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.071,"5min":1.196,"15min":1.196},"last":0.58099999999999996}]},{"osd":4,"last update":"Tue Apr 2 07:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56599999999999995,"15min":0.58499999999999996},"min":{"1min":0.36299999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0229999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.438},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.67200000000000004,"15min":0.68300000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.80500000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.627}]},{"osd":5,"last update":"Tue Apr 2 07:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.42999999999999999,"5min":0.37,"15min":0.37},"max":{"1min":1.0109999999999999,"5min":1.167,"15min":1.167},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.67200000000000004,"15min":0.68300000000000005},"min":{"1min":0.40200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.343,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.75}]},{"osd":6,"last update":"Tue Apr 2 07:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.66300000000000003,"15min":0.65500000000000003},"min":{"1min":0.435,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0580000000000001,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.64300000000000002,"15min":0.64800000000000002},"min":{"1min":0.45400000000000001,"5min":0.42299999999999999,"15min":0.39600000000000002},"max":{"1min":0.83399999999999996,"5min":0.93799999999999994,"15min":1.0509999999999999},"last":0.68300000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575047,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36548,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335292,"statfs":{"total":96636764160,"available":96599339008,"internally_reserved":0,"allocated":1359872,"data_stored":662426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.68899999999999995,"15min":0.70599999999999996},"min":{"1min":0.40899999999999997,"5min":0.38700000000000001,"15min":0.33400000000000002},"max":{"1min":0.80600000000000005,"5min":0.95699999999999996,"15min":0.98099999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.68700000000000006,"15min":0.70799999999999996},"min":{"1min":0.46100000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.84799999999999998,"5min":0.96899999999999997,"15min":1.0249999999999999},"last":0.502}]},{"osd":1,"last update":"Tue Apr 2 07:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.67300000000000004,"15min":0.69599999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.41199999999999998},"max":{"1min":0.76300000000000001,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.71399999999999997,"15min":0.71999999999999997},"min":{"1min":0.45300000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84399999999999997,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.625}]},{"osd":2,"last update":"Tue Apr 2 07:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.70599999999999996,"15min":0.71499999999999997},"min":{"1min":0.46600000000000003,"5min":0.45500000000000002,"15min":0.38200000000000001},"max":{"1min":0.72999999999999998,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46300000000000002},"max":{"1min":0.83799999999999997,"5min":0.96499999999999997,"15min":0.96599999999999997},"last":0.51900000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.72299999999999998,"15min":0.73799999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.91600000000000004,"5min":1.089,"15min":1.089},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.72599999999999998,"15min":0.73899999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.45300000000000001},"max":{"1min":0.82099999999999995,"5min":0.97799999999999998,"15min":1.0309999999999999},"last":0.60299999999999998}]},{"osd":4,"last update":"Tue Apr 2 07:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61799999999999999,"15min":0.60499999999999998},"min":{"1min":0.42799999999999999,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.82399999999999995,"5min":0.996,"15min":0.996},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63500000000000001,"15min":0.628},"min":{"1min":0.45200000000000001,"5min":0.40600000000000003,"15min":0.35499999999999998},"max":{"1min":0.86699999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.71799999999999997}]},{"osd":5,"last update":"Tue Apr 2 07:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.628,"15min":0.621},"min":{"1min":0.45200000000000001,"5min":0.379,"15min":0.35799999999999998},"max":{"1min":0.88100000000000001,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.67900000000000005,"15min":0.68799999999999994},"min":{"1min":0.51700000000000002,"5min":0.42099999999999999,"15min":0.40300000000000002},"max":{"1min":0.90200000000000002,"5min":1.01,"15min":1.01},"last":0.76300000000000001}]},{"osd":7,"last update":"Tue Apr 2 07:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.69599999999999995,"15min":0.68799999999999994},"min":{"1min":0.56000000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.91400000000000003,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.746},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69199999999999995,"15min":0.69499999999999995},"min":{"1min":0.48999999999999999,"5min":0.44,"15min":0.42099999999999999},"max":{"1min":0.92800000000000005,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.78800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575048,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55172,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316668,"statfs":{"total":96636764160,"available":96580268032,"internally_reserved":0,"allocated":1949696,"data_stored":1252212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.435,"15min":0.436},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":10.052},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51400000000000001,"15min":0.504},"min":{"1min":0.27500000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":9.0250000000000004}]},{"osd":2,"last update":"Tue Apr 2 07:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.29699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":7.6059999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.36699999999999999,"15min":0.35699999999999998},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":6.0709999999999997}]},{"osd":3,"last update":"Tue Apr 2 07:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.609,"5min":2.609,"15min":2.609},"last":1.194},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54000000000000004,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.72999999999999998}]},{"osd":4,"last update":"Tue Apr 2 07:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78300000000000003,"15min":0.77500000000000002},"min":{"1min":0.55100000000000005,"5min":0.46700000000000003,"15min":0.46300000000000002},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78000000000000003,"15min":0.77100000000000002},"min":{"1min":0.55100000000000005,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":6.3419999999999996}]},{"osd":5,"last update":"Tue Apr 2 07:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.77200000000000002,"15min":0.77500000000000002},"min":{"1min":0.58399999999999996,"5min":0.53800000000000003,"15min":0.41999999999999998},"max":{"1min":2.4830000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":6.5529999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.79700000000000004,"15min":0.78900000000000003},"min":{"1min":0.56699999999999995,"5min":0.51000000000000001,"15min":0.48799999999999999},"max":{"1min":2.544,"5min":2.544,"15min":2.544},"last":2.1000000000000001}]},{"osd":6,"last update":"Tue Apr 2 07:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.80200000000000005,"15min":0.79800000000000004},"min":{"1min":0.55900000000000005,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.2370000000000001,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":7.8029999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.80100000000000005,"15min":0.79400000000000004},"min":{"1min":0.58499999999999996,"5min":0.51400000000000001,"15min":0.38100000000000001},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":6.4329999999999998}]},{"osd":7,"last update":"Tue Apr 2 07:52:14 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.80700000000000005,"15min":0.80500000000000005},"min":{"1min":0.55000000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":1.242},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82499999999999996,"15min":0.82299999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.502},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":0.76000000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575048,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1363968,"data_stored":662321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.433,"15min":0.435},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.66000000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.26200000000000001},"max":{"1min":0.60099999999999998,"5min":0.72199999999999998,"15min":0.79100000000000004},"last":0.38700000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.48099999999999998,"15min":0.48699999999999999},"min":{"1min":0.312,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":0.749,"15min":0.76300000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.39400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.63,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46100000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.52900000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.28899999999999998},"max":{"1min":0.60899999999999999,"5min":0.73999999999999999,"15min":0.81799999999999995},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.40100000000000002},"max":{"1min":0.77700000000000002,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.40699999999999997}]},{"osd":4,"last update":"Tue Apr 2 07:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.75800000000000001,"15min":0.77700000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.35599999999999998},"max":{"1min":0.95899999999999996,"5min":1.0640000000000001,"15min":1.181},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.76500000000000001,"15min":0.77300000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":0.91800000000000004,"5min":1.095,"15min":1.331},"last":0.753}]},{"osd":5,"last update":"Tue Apr 2 07:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.79100000000000004,"15min":0.78800000000000003},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.42099999999999999},"max":{"1min":0.95199999999999996,"5min":2.1869999999999998,"15min":2.1869999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.79400000000000004,"15min":0.79500000000000004},"min":{"1min":0.59399999999999997,"5min":0.54100000000000004,"15min":0.51800000000000002},"max":{"1min":0.92500000000000004,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":0.81499999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.745,"15min":0.751},"min":{"1min":0.56399999999999995,"5min":0.53000000000000003,"15min":0.46800000000000003},"max":{"1min":0.82499999999999996,"5min":1.044,"15min":1.1200000000000001},"last":0.872},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.72499999999999998,"15min":0.747},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.76200000000000001,"5min":1.0609999999999999,"15min":1.1339999999999999},"last":0.73599999999999999}]},{"osd":7,"last update":"Tue Apr 2 07:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.82899999999999996,"15min":0.81499999999999995},"min":{"1min":0.60499999999999998,"5min":0.51600000000000001,"15min":0.40799999999999997},"max":{"1min":0.89500000000000002,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.78600000000000003,"15min":0.80500000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.88500000000000001,"5min":2.145,"15min":2.145},"last":0.77400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575047,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36540,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335300,"statfs":{"total":96636764160,"available":96599347200,"internally_reserved":0,"allocated":1351680,"data_stored":661754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.44,"15min":0.44800000000000001},"min":{"1min":0.26700000000000002,"5min":0.26400000000000001,"15min":0.25700000000000001},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46300000000000002,"15min":0.47699999999999998},"min":{"1min":0.36199999999999999,"5min":0.27900000000000003,"15min":0.26200000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.62}]},{"osd":1,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.49199999999999999,"15min":0.49099999999999999},"min":{"1min":0.33000000000000002,"5min":0.30599999999999999,"15min":0.30299999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.499},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.52400000000000002},"min":{"1min":0.39700000000000002,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.66300000000000003}]},{"osd":3,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.35999999999999999,"5min":0.316,"15min":0.27800000000000002},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59599999999999997,"15min":0.60999999999999999},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39000000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.64000000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76300000000000001,"15min":0.77000000000000002},"min":{"1min":0.5,"5min":0.5,"15min":0.47199999999999998},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.748,"15min":0.77200000000000002},"min":{"1min":0.56499999999999995,"5min":0.51600000000000001,"15min":0.48499999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.68500000000000005}]},{"osd":5,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.752,"15min":0.76100000000000001},"min":{"1min":0.56499999999999995,"5min":0.54300000000000004,"15min":0.42199999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.55600000000000005,"5min":0.53800000000000003,"15min":0.50900000000000001},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.67200000000000004}]},{"osd":6,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.79400000000000004,"15min":0.79200000000000004},"min":{"1min":0.55000000000000004,"5min":0.53600000000000003,"15min":0.442},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.752,"5min":0.75600000000000001,"15min":0.78700000000000003},"min":{"1min":0.53100000000000003,"5min":0.502,"15min":0.502},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.70099999999999996}]},{"osd":7,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.77200000000000002,"15min":0.79500000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.5},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.77100000000000002,"15min":0.79300000000000004},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.65500000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575047,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54600,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317240,"statfs":{"total":96636764160,"available":96580853760,"internally_reserved":0,"allocated":1363968,"data_stored":662428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.42399999999999999,"15min":0.42999999999999999},"min":{"1min":0.26500000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.72599999999999998,"5min":0.748,"15min":0.79600000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.435,"15min":0.46400000000000002},"min":{"1min":0.31,"5min":0.29699999999999999,"15min":0.28699999999999998},"max":{"1min":0.57099999999999995,"5min":0.70799999999999996,"15min":0.81899999999999995},"last":0.49399999999999999}]},{"osd":1,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.47899999999999998,"15min":0.49199999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.61899999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.439,"5min":0.35799999999999998,"15min":0.34200000000000003},"max":{"1min":0.77500000000000002,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.51400000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59099999999999997,"15min":0.60699999999999998},"min":{"1min":0.439,"5min":0.38700000000000001,"15min":0.376},"max":{"1min":0.78400000000000003,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57699999999999996,"15min":0.60399999999999998},"min":{"1min":0.39500000000000002,"5min":0.38100000000000001,"15min":0.35899999999999999},"max":{"1min":0.70699999999999996,"5min":1.244,"15min":1.244},"last":0.44500000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.70899999999999996,"15min":0.73199999999999998},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.38400000000000001},"max":{"1min":0.83499999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73199999999999998,"15min":0.75700000000000001},"min":{"1min":0.52300000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":1.1759999999999999},"last":0.74299999999999999}]},{"osd":5,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.72399999999999998,"15min":0.754},"min":{"1min":0.51300000000000001,"5min":0.41699999999999998,"15min":0.40899999999999997},"max":{"1min":0.97299999999999998,"5min":1.079,"15min":1.151},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72099999999999997,"15min":0.749},"min":{"1min":0.49199999999999999,"5min":0.48899999999999999,"15min":0.40200000000000002},"max":{"1min":0.94199999999999995,"5min":0.97599999999999998,"15min":1.1439999999999999},"last":0.72599999999999998}]},{"osd":6,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72199999999999998,"15min":0.76000000000000001},"min":{"1min":0.52800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.86899999999999999,"5min":0.98999999999999999,"15min":1.0620000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.748,"15min":0.77400000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.90400000000000003,"5min":1.1140000000000001,"15min":1.1899999999999999},"last":0.68400000000000005}]},{"osd":7,"last update":"Tue Apr 2 07:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.73799999999999999,"15min":0.77000000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.439},"max":{"1min":0.91100000000000003,"5min":1.0409999999999999,"15min":1.0680000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.75,"15min":0.78500000000000003},"min":{"1min":0.60299999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0089999999999999,"5min":1.181,"15min":1.181},"last":0.78700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575047,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36552,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335288,"statfs":{"total":96636764160,"available":96599334912,"internally_reserved":0,"allocated":1368064,"data_stored":668436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.42499999999999999,"5min":0.38100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86599999999999999,"5min":0.91900000000000004,"15min":1.0189999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.66800000000000004,"15min":0.67500000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.879,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.53400000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.71599999999999997,"15min":0.72799999999999998},"min":{"1min":0.47599999999999998,"5min":0.438,"15min":0.433},"max":{"1min":0.90300000000000002,"5min":0.97299999999999998,"15min":1.036},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.73799999999999999},"min":{"1min":0.58099999999999996,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.91900000000000004,"5min":0.97899999999999998,"15min":0.99399999999999999},"last":0.496}]},{"osd":2,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.67400000000000004,"15min":0.69099999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.47299999999999998,"5min":0.47199999999999998,"15min":0.41899999999999998},"max":{"1min":0.874,"5min":0.97199999999999998,"15min":1.0620000000000001},"last":0.55300000000000005}]},{"osd":3,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72099999999999997},"min":{"1min":0.51500000000000001,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":0.95299999999999996,"5min":0.95699999999999996,"15min":1.002},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.71299999999999997,"15min":0.72999999999999998},"min":{"1min":0.49299999999999999,"5min":0.48799999999999999,"15min":0.41499999999999998},"max":{"1min":0.92100000000000004,"5min":0.93999999999999995,"15min":0.96999999999999997},"last":0.47299999999999998}]},{"osd":5,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56200000000000006,"15min":0.57399999999999995},"min":{"1min":0.45800000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.746,"5min":0.84799999999999998,"15min":0.93999999999999995},"last":0.435},{"interface":"front","average":{"1min":0.621,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.434,"5min":0.38800000000000001,"15min":0.33700000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.66800000000000004}]},{"osd":6,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.61199999999999999,"15min":0.60399999999999998},"min":{"1min":0.45200000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.51600000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.038},"last":0.65500000000000003}]},{"osd":7,"last update":"Tue Apr 2 07:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68000000000000005,"15min":0.66800000000000004},"min":{"1min":0.55700000000000005,"5min":0.432,"15min":0.38300000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.74099999999999999,"15min":0.72999999999999998},"min":{"1min":0.58999999999999997,"5min":0.47799999999999998,"15min":0.434},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.746}]}]},{"osd":5,"up_from":13,"seq":55834575050,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37140,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334700,"statfs":{"total":96636764160,"available":96598732800,"internally_reserved":0,"allocated":1970176,"data_stored":1259349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.68500000000000005},"min":{"1min":0.433,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83099999999999996,"5min":0.95299999999999996,"15min":0.99199999999999999},"last":0.499},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.504,"5min":0.378,"15min":0.378},"max":{"1min":0.92700000000000005,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.69399999999999995}]},{"osd":1,"last update":"Tue Apr 2 07:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.66500000000000004,"15min":0.69399999999999995},"min":{"1min":0.48399999999999999,"5min":0.40300000000000002,"15min":0.40200000000000002},"max":{"1min":0.84799999999999998,"5min":1.131,"15min":1.131},"last":0.44},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.68000000000000005,"15min":0.69199999999999995},"min":{"1min":0.48499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85599999999999998,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.40999999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.68899999999999995,"15min":0.70899999999999996},"min":{"1min":0.43099999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.78800000000000003,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.68000000000000005,"15min":0.70599999999999996},"min":{"1min":0.497,"5min":0.47399999999999998,"15min":0.372},"max":{"1min":0.82599999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.55700000000000005}]},{"osd":3,"last update":"Tue Apr 2 07:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.71799999999999997,"15min":0.73799999999999999},"min":{"1min":0.51500000000000001,"5min":0.45900000000000002,"15min":0.45100000000000001},"max":{"1min":0.85099999999999998,"5min":1.341,"15min":1.341},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.754,"15min":0.76400000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.44900000000000001},"max":{"1min":0.88100000000000001,"5min":1.484,"15min":1.484},"last":0.68000000000000005}]},{"osd":4,"last update":"Tue Apr 2 07:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60799999999999998,"15min":0.59799999999999998},"min":{"1min":0.43099999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.78300000000000003,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65000000000000002,"15min":0.64000000000000001},"min":{"1min":0.44,"5min":0.42499999999999999,"15min":0.35599999999999998},"max":{"1min":0.86199999999999999,"5min":1.504,"15min":1.504},"last":0.61799999999999999}]},{"osd":6,"last update":"Tue Apr 2 07:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.65600000000000003,"15min":0.64200000000000002},"min":{"1min":0.46999999999999997,"5min":0.44700000000000001,"15min":0.39900000000000002},"max":{"1min":0.94599999999999995,"5min":1.212,"15min":1.212},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70799999999999996,"15min":0.67700000000000005},"min":{"1min":0.53300000000000003,"5min":0.44900000000000001,"15min":0.41199999999999998},"max":{"1min":0.90300000000000002,"5min":1.774,"15min":1.774},"last":0.54300000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:52:12 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72199999999999998,"15min":0.70099999999999996},"min":{"1min":0.57999999999999996,"5min":0.47599999999999998,"15min":0.38900000000000001},"max":{"1min":0.96899999999999997,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.74199999999999999,"15min":0.72199999999999998},"min":{"1min":0.63600000000000001,"5min":0.46300000000000002,"15min":0.434},"max":{"1min":0.98099999999999998,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.70199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:52:53.826 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T07:52:53.827 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T07:52:54.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.060+0000 7fd16bc00700 1 Processor -- start 2024-04-02T07:52:54.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.060+0000 7fd16bc00700 1 -- start start 2024-04-02T07:52:54.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.061+0000 7fd16bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1640d2540 0x7fd1641687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:54.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.061+0000 7fd16bc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd164161420 con 0x7fd1640d16a0 2024-04-02T07:52:54.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.061+0000 7fd16bc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd164161700 con 0x7fd16415d9d0 2024-04-02T07:52:54.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.061+0000 7fd16bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640d4340 con 0x7fd1640d2540 2024-04-02T07:52:54.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.061+0000 7fd163e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd1640d16a0 0x7fd1640d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33322/0 (socket says 172.21.15.89:33322) 2024-04-02T07:52:54.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.061+0000 7fd163e00700 1 -- 172.21.15.89:0/1578208516 learned_addr learned my addr 172.21.15.89:0/1578208516 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:54.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd169c00700 1 --2- 172.21.15.89:0/1578208516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1640d2540 0x7fd1641687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:54.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423780124 0 0) 0x7fd164161420 con 0x7fd1640d16a0 2024-04-02T07:52:54.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd14c0035b0 con 0x7fd1640d16a0 2024-04-02T07:52:54.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649609803 0 0) 0x7fd164161700 con 0x7fd16415d9d0 2024-04-02T07:52:54.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd164161420 con 0x7fd16415d9d0 2024-04-02T07:52:54.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3287645402 0 0) 0x7fd14c0035b0 con 0x7fd1640d16a0 2024-04-02T07:52:54.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1640d2540 msgr2=0x7fd1641687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:54.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 --2- 172.21.15.89:0/1578208516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1640d2540 0x7fd1641687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 >> v1:172.21.15.113:6789/0 conn(0x7fd16415d9d0 legacy=0x7fd1640d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:54.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1641619e0 con 0x7fd1640d16a0 2024-04-02T07:52:54.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fd1500017f0 con 0x7fd1640d16a0 2024-04-02T07:52:54.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 >> v1:172.21.15.89:6789/0 conn(0x7fd1640d16a0 legacy=0x7fd1640d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:54.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 --2- 172.21.15.89:0/1578208516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd14c003bc0 0x7fd14c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:54.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 --2- 172.21.15.89:0/1578208516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1640d2540 0x7fd14c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:54.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 --2- 172.21.15.89:0/1578208516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd14c004d90 0x7fd14c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:54.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c00bac0 con 0x7fd14c003bc0 2024-04-02T07:52:54.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c00bc00 con 0x7fd14c004d90 2024-04-02T07:52:54.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c00bdc0 con 0x7fd1640d2540 2024-04-02T07:52:54.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.062+0000 7fd169200700 1 --2- 172.21.15.89:0/1578208516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd14c003bc0 0x7fd14c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:54.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.063+0000 7fd163e00700 1 --2- 172.21.15.89:0/1578208516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1640d2540 0x7fd14c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:54.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.063+0000 7fd169c00700 1 --2- 172.21.15.89:0/1578208516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd14c004d90 0x7fd14c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:54.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.063+0000 7fd169200700 1 -- 172.21.15.89:0/1578208516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1640d2540 msgr2=0x7fd14c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:54.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.063+0000 7fd169200700 1 --2- 172.21.15.89:0/1578208516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1640d2540 0x7fd14c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.063+0000 7fd169200700 1 -- 172.21.15.89:0/1578208516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd14c004d90 msgr2=0x7fd14c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:54.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.063+0000 7fd169200700 1 --2- 172.21.15.89:0/1578208516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd14c004d90 0x7fd14c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.063+0000 7fd169200700 1 -- 172.21.15.89:0/1578208516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14c0035b0 con 0x7fd14c003bc0 2024-04-02T07:52:54.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.063+0000 7fd169200700 1 --2- 172.21.15.89:0/1578208516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd14c003bc0 0x7fd14c003fc0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fd158001940 tx=0x7fd14c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55391d17dc670ef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:54.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.063+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158001430 con 0x7fd14c003bc0 2024-04-02T07:52:54.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.063+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd158001d70 con 0x7fd14c003bc0 2024-04-02T07:52:54.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.063+0000 7fd163400700 1 -- 172.21.15.89:0/1578208516 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158005510 con 0x7fd14c003bc0 2024-04-02T07:52:54.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.064+0000 7fd16bc00700 1 -- 172.21.15.89:0/1578208516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd14c003bc0 msgr2=0x7fd14c003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:54.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.064+0000 7fd16bc00700 1 --2- 172.21.15.89:0/1578208516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd14c003bc0 0x7fd14c003fc0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fd158001940 tx=0x7fd14c005880 comp rx=0 tx=0).stop 2024-04-02T07:52:54.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.064+0000 7fd169c00700 1 -- 172.21.15.89:0/1578208516 reap_dead start 2024-04-02T07:52:54.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.064+0000 7fd16bc00700 1 -- 172.21.15.89:0/1578208516 shutdown_connections 2024-04-02T07:52:54.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.064+0000 7fd16bc00700 1 -- 172.21.15.89:0/1578208516 >> 172.21.15.89:0/1578208516 conn(0x7fd16408aff0 msgr2=0x7fd1640d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:54.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.064+0000 7fd16bc00700 1 -- 172.21.15.89:0/1578208516 shutdown_connections 2024-04-02T07:52:54.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.064+0000 7fd16bc00700 1 -- 172.21.15.89:0/1578208516 wait complete. 2024-04-02T07:52:54.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.066+0000 7fd16bc00700 1 Processor -- start 2024-04-02T07:52:54.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.066+0000 7fd16bc00700 1 -- start start 2024-04-02T07:52:54.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd16bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd14c003bc0 0x7fd16415ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:54.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd16bc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1640b6020 0x7fd164157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:54.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd16bc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd164158310 0x7fd1640b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:54.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd16bc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16415fbf0 con 0x7fd1640b6020 2024-04-02T07:52:54.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd16bc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd164158b60 con 0x7fd164158310 2024-04-02T07:52:54.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd16bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641580c0 con 0x7fd14c003bc0 2024-04-02T07:52:54.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd163e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1640b6020 0x7fd164157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:54.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd163e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1640b6020 0x7fd164157b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46566/0 (socket says 172.21.15.89:46566) 2024-04-02T07:52:54.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd163e00700 1 -- 172.21.15.89:0/907692709 learned_addr learned my addr 172.21.15.89:0/907692709 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:54.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd169200700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd14c003bc0 0x7fd16415ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:54.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd169c00700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd164158310 0x7fd1640b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:54.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd163e00700 1 -- 172.21.15.89:0/907692709 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd14c003bc0 msgr2=0x7fd16415ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:54.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd163e00700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd14c003bc0 0x7fd16415ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd163e00700 1 -- 172.21.15.89:0/907692709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd164158310 msgr2=0x7fd1640b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:54.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd163e00700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd164158310 0x7fd1640b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd163e00700 1 -- 172.21.15.89:0/907692709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16415f250 con 0x7fd1640b6020 2024-04-02T07:52:54.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.067+0000 7fd169c00700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd164158310 0x7fd1640b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:54.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.068+0000 7fd163e00700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1640b6020 0x7fd164157b80 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fd15000e780 tx=0x7fd15000ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:54.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.068+0000 7fd160c00700 1 -- 172.21.15.89:0/907692709 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1500072e0 con 0x7fd1640b6020 2024-04-02T07:52:54.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.068+0000 7fd160c00700 1 -- 172.21.15.89:0/907692709 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd150007440 con 0x7fd1640b6020 2024-04-02T07:52:54.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.068+0000 7fd16bc00700 1 -- 172.21.15.89:0/907692709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16415f530 con 0x7fd1640b6020 2024-04-02T07:52:54.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.068+0000 7fd160c00700 1 -- 172.21.15.89:0/907692709 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150005830 con 0x7fd1640b6020 2024-04-02T07:52:54.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.068+0000 7fd16bc00700 1 -- 172.21.15.89:0/907692709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1640b64a0 con 0x7fd1640b6020 2024-04-02T07:52:54.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.069+0000 7fd16bc00700 1 -- 172.21.15.89:0/907692709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1640d1ca0 con 0x7fd1640b6020 2024-04-02T07:52:54.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.070+0000 7fd160c00700 1 -- 172.21.15.89:0/907692709 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd150005990 con 0x7fd1640b6020 2024-04-02T07:52:54.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.071+0000 7fd160c00700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd14406adb0 0x7fd14406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:54.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.071+0000 7fd160c00700 1 -- 172.21.15.89:0/907692709 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd150089440 con 0x7fd1640b6020 2024-04-02T07:52:54.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.071+0000 7fd169200700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd14406adb0 0x7fd14406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:54.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.071+0000 7fd169200700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd14406adb0 0x7fd14406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd158015040 tx=0x7fd15800a000 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:54.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.075+0000 7fd160c00700 1 -- 172.21.15.89:0/907692709 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd150057da0 con 0x7fd1640b6020 2024-04-02T07:52:54.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.355+0000 7fd16bc00700 1 -- 172.21.15.89:0/907692709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd164067720 con 0x7fd1640b6020 2024-04-02T07:52:54.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.361+0000 7fd160c00700 1 -- 172.21.15.89:0/907692709 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+0 (secure 0 0 0) 0x7fd150057930 con 0x7fd1640b6020 2024-04-02T07:52:54.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.364+0000 7fd16bc00700 1 -- 172.21.15.89:0/907692709 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd14406adb0 msgr2=0x7fd14406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:54.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.364+0000 7fd16bc00700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd14406adb0 0x7fd14406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd158015040 tx=0x7fd15800a000 comp rx=0 tx=0).stop 2024-04-02T07:52:54.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.364+0000 7fd16bc00700 1 -- 172.21.15.89:0/907692709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1640b6020 msgr2=0x7fd164157b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:54.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.364+0000 7fd16bc00700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1640b6020 0x7fd164157b80 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fd15000e780 tx=0x7fd15000ea90 comp rx=0 tx=0).stop 2024-04-02T07:52:54.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.364+0000 7fd16bc00700 1 -- 172.21.15.89:0/907692709 shutdown_connections 2024-04-02T07:52:54.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.364+0000 7fd16bc00700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd14406adb0 0x7fd14406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.364+0000 7fd16bc00700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd164158310 0x7fd1640b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.364+0000 7fd16bc00700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd14c003bc0 0x7fd16415ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.364+0000 7fd16bc00700 1 --2- 172.21.15.89:0/907692709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1640b6020 0x7fd164157b80 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.364+0000 7fd16bc00700 1 -- 172.21.15.89:0/907692709 >> 172.21.15.89:0/907692709 conn(0x7fd16408aff0 msgr2=0x7fd1640b21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:54.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.364+0000 7fd16bc00700 1 -- 172.21.15.89:0/907692709 shutdown_connections 2024-04-02T07:52:54.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.364+0000 7fd16bc00700 1 -- 172.21.15.89:0/907692709 wait complete. 2024-04-02T07:52:54.366 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T07:52:54.385 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:52:54.461 INFO:tasks.ceph.mds.a.smithi089.stderr:2024-04-02T07:52:54.460+0000 7fa6dc800700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T07:52:54.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.575+0000 7fe9e3e00700 1 Processor -- start 2024-04-02T07:52:54.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.575+0000 7fe9e3e00700 1 -- start start 2024-04-02T07:52:54.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.577+0000 7fe9e3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe9dc0b3af0 0x7fe9dc15a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:54.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.577+0000 7fe9e3e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9dc0b2180 con 0x7fe9dc0709c0 2024-04-02T07:52:54.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.577+0000 7fe9e3e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9dc15b100 con 0x7fe9dc072580 2024-04-02T07:52:54.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.577+0000 7fe9e3e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9dc15b3e0 con 0x7fe9dc0b3af0 2024-04-02T07:52:54.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.577+0000 7fe9e1400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe9dc0b3af0 0x7fe9dc15a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:54.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.577+0000 7fe9e1e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe9dc0709c0 0x7fe9dc06e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33332/0 (socket says 172.21.15.89:33332) 2024-04-02T07:52:54.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.577+0000 7fe9e1400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe9dc0b3af0 0x7fe9dc15a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46574/0 (socket says 172.21.15.89:46574) 2024-04-02T07:52:54.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.577+0000 7fe9e1e00700 1 -- 172.21.15.89:0/441862202 learned_addr learned my addr 172.21.15.89:0/441862202 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:54.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.577+0000 7fe9e1400700 1 -- 172.21.15.89:0/441862202 >> v1:172.21.15.113:6789/0 conn(0x7fe9dc072580 legacy=0x7fe9dc0b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:54.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.577+0000 7fe9e1400700 1 -- 172.21.15.89:0/441862202 >> v1:172.21.15.89:6789/0 conn(0x7fe9dc0709c0 legacy=0x7fe9dc06e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:54.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.577+0000 7fe9e1400700 1 -- 172.21.15.89:0/441862202 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc0b23f0 con 0x7fe9dc0b3af0 2024-04-02T07:52:54.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.577+0000 7fe9dbe00700 1 -- 172.21.15.89:0/441862202 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737534409 0 0) 0x7fe9dc0b2180 con 0x7fe9dc0709c0 2024-04-02T07:52:54.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.578+0000 7fe9e1400700 1 --2- 172.21.15.89:0/441862202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc0b3af0 0x7fe9dc15a860 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009840 tx=0x7fe9d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=920ce1184eeafbf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:54.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.578+0000 7fe9dbe00700 1 -- 172.21.15.89:0/441862202 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0011830 con 0x7fe9dc0b3af0 2024-04-02T07:52:54.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.578+0000 7fe9dbe00700 1 -- 172.21.15.89:0/441862202 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d0011990 con 0x7fe9dc0b3af0 2024-04-02T07:52:54.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.578+0000 7fe9dbe00700 1 -- 172.21.15.89:0/441862202 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0011d20 con 0x7fe9dc0b3af0 2024-04-02T07:52:54.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.578+0000 7fe9e3e00700 1 -- 172.21.15.89:0/441862202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc0b3af0 msgr2=0x7fe9dc15a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:54.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.578+0000 7fe9e3e00700 1 --2- 172.21.15.89:0/441862202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc0b3af0 0x7fe9dc15a860 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009840 tx=0x7fe9d0009b50 comp rx=0 tx=0).stop 2024-04-02T07:52:54.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.578+0000 7fe9e3e00700 1 -- 172.21.15.89:0/441862202 shutdown_connections 2024-04-02T07:52:54.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.578+0000 7fe9e3e00700 1 --2- 172.21.15.89:0/441862202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc0b3af0 0x7fe9dc15a860 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.578+0000 7fe9e3e00700 1 -- 172.21.15.89:0/441862202 >> 172.21.15.89:0/441862202 conn(0x7fe9dc0cf190 msgr2=0x7fe9dc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:54.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.579+0000 7fe9e3e00700 1 -- 172.21.15.89:0/441862202 shutdown_connections 2024-04-02T07:52:54.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.579+0000 7fe9e3e00700 1 -- 172.21.15.89:0/441862202 wait complete. 2024-04-02T07:52:54.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.580+0000 7fe9e3e00700 1 Processor -- start 2024-04-02T07:52:54.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.580+0000 7fe9e3e00700 1 -- start start 2024-04-02T07:52:54.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.581+0000 7fe9e3e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9dc0709c0 0x7fe9dc0c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:54.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.581+0000 7fe9e3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc072580 0x7fe9dc0caf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:54.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.581+0000 7fe9e3e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9dc0b3af0 0x7fe9dc156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:54.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.581+0000 7fe9e3e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc1569a0 con 0x7fe9dc072580 2024-04-02T07:52:54.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.581+0000 7fe9e3e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc0c89a0 con 0x7fe9dc0709c0 2024-04-02T07:52:54.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.581+0000 7fe9e3e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc0cb450 con 0x7fe9dc0b3af0 2024-04-02T07:52:54.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.581+0000 7fe9e0a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc072580 0x7fe9dc0caf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:54.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.582+0000 7fe9e0a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc072580 0x7fe9dc0caf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46588/0 (socket says 172.21.15.89:46588) 2024-04-02T07:52:54.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.582+0000 7fe9e0a00700 1 -- 172.21.15.89:0/1615791469 learned_addr learned my addr 172.21.15.89:0/1615791469 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:54.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.582+0000 7fe9e1400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9dc0709c0 0x7fe9dc0c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:54.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.582+0000 7fe9e1e00700 1 --2- 172.21.15.89:0/1615791469 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9dc0b3af0 0x7fe9dc156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:54.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.582+0000 7fe9e0a00700 1 -- 172.21.15.89:0/1615791469 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9dc0b3af0 msgr2=0x7fe9dc156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:54.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.582+0000 7fe9e0a00700 1 --2- 172.21.15.89:0/1615791469 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9dc0b3af0 0x7fe9dc156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.582+0000 7fe9e0a00700 1 -- 172.21.15.89:0/1615791469 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9dc0709c0 msgr2=0x7fe9dc0c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:54.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.582+0000 7fe9e0a00700 1 --2- 172.21.15.89:0/1615791469 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9dc0709c0 0x7fe9dc0c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.582+0000 7fe9e0a00700 1 -- 172.21.15.89:0/1615791469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d00094a0 con 0x7fe9dc072580 2024-04-02T07:52:54.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.582+0000 7fe9e0a00700 1 --2- 172.21.15.89:0/1615791469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc072580 0x7fe9dc0caf10 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800c4b0 tx=0x7fe9c800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:54.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.582+0000 7fe9d9600700 1 -- 172.21.15.89:0/1615791469 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c80052d0 con 0x7fe9dc072580 2024-04-02T07:52:54.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.582+0000 7fe9d9600700 1 -- 172.21.15.89:0/1615791469 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c8005430 con 0x7fe9dc072580 2024-04-02T07:52:54.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.582+0000 7fe9e3e00700 1 -- 172.21.15.89:0/1615791469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9dc0c8bd0 con 0x7fe9dc072580 2024-04-02T07:52:54.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.582+0000 7fe9d9600700 1 -- 172.21.15.89:0/1615791469 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c80038c0 con 0x7fe9dc072580 2024-04-02T07:52:54.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.583+0000 7fe9e3e00700 1 -- 172.21.15.89:0/1615791469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9dc1562e0 con 0x7fe9dc072580 2024-04-02T07:52:54.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.584+0000 7fe9e3e00700 1 -- 172.21.15.89:0/1615791469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9dc156ae0 con 0x7fe9dc072580 2024-04-02T07:52:54.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.585+0000 7fe9d9600700 1 -- 172.21.15.89:0/1615791469 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe9c8003a20 con 0x7fe9dc072580 2024-04-02T07:52:54.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.585+0000 7fe9d9600700 1 --2- 172.21.15.89:0/1615791469 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe9bc06adb0 0x7fe9bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:54.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.585+0000 7fe9d9600700 1 -- 172.21.15.89:0/1615791469 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe9c80889d0 con 0x7fe9dc072580 2024-04-02T07:52:54.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.586+0000 7fe9e1400700 1 --2- 172.21.15.89:0/1615791469 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe9bc06adb0 0x7fe9bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:54.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.586+0000 7fe9e1400700 1 --2- 172.21.15.89:0/1615791469 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe9bc06adb0 0x7fe9bc06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00090b0 tx=0x7fe9d001c040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:54.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.590+0000 7fe9d9600700 1 -- 172.21.15.89:0/1615791469 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe9dc156ae0 con 0x7fe9dc072580 2024-04-02T07:52:54.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.834+0000 7fe9e3e00700 1 -- 172.21.15.89:0/1615791469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9dc156ae0 con 0x7fe9dc072580 2024-04-02T07:52:54.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.835+0000 7fe9d9600700 1 -- 172.21.15.89:0/1615791469 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fe9dc156ae0 con 0x7fe9dc072580 2024-04-02T07:52:54.836 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:52:54.836 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6317,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1566757639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1566757639},{"type":"v1","addr":"172.21.15.113:6835","nonce":1566757639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":6985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4102221782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4102221782},{"type":"v1","addr":"172.21.15.89:6836","nonce":4102221782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3127911401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3127911401},{"type":"v1","addr":"172.21.15.113:6834","nonce":3127911401}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:52:48.969376+0000","modified":"2024-04-02T07:52:51.980647+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7419":{"gid":7419,"name":"a","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2016280781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2016280781},{"type":"v1","addr":"172.21.15.89:6837","nonce":2016280781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-02T07:52:54.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.837+0000 7fe9e3e00700 1 -- 172.21.15.89:0/1615791469 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe9bc06adb0 msgr2=0x7fe9bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:54.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.837+0000 7fe9e3e00700 1 --2- 172.21.15.89:0/1615791469 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe9bc06adb0 0x7fe9bc06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00090b0 tx=0x7fe9d001c040 comp rx=0 tx=0).stop 2024-04-02T07:52:54.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.837+0000 7fe9e3e00700 1 -- 172.21.15.89:0/1615791469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc072580 msgr2=0x7fe9dc0caf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:54.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.837+0000 7fe9e3e00700 1 --2- 172.21.15.89:0/1615791469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc072580 0x7fe9dc0caf10 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800c4b0 tx=0x7fe9c800c7c0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.837+0000 7fe9e3e00700 1 -- 172.21.15.89:0/1615791469 shutdown_connections 2024-04-02T07:52:54.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.837+0000 7fe9e3e00700 1 --2- 172.21.15.89:0/1615791469 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe9bc06adb0 0x7fe9bc06d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.837+0000 7fe9e3e00700 1 --2- 172.21.15.89:0/1615791469 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9dc0709c0 0x7fe9dc0c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.837+0000 7fe9e3e00700 1 --2- 172.21.15.89:0/1615791469 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9dc0b3af0 0x7fe9dc156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.837+0000 7fe9e3e00700 1 --2- 172.21.15.89:0/1615791469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9dc072580 0x7fe9dc0caf10 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:54.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.837+0000 7fe9e3e00700 1 -- 172.21.15.89:0/1615791469 >> 172.21.15.89:0/1615791469 conn(0x7fe9dc0cf190 msgr2=0x7fe9dc0ad7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:54.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.838+0000 7fe9e3e00700 1 -- 172.21.15.89:0/1615791469 shutdown_connections 2024-04-02T07:52:54.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:54.838+0000 7fe9e3e00700 1 -- 172.21.15.89:0/1615791469 wait complete. 2024-04-02T07:52:54.840 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 49 2024-04-02T07:52:55.261 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T07:52:48.969376+0000', 'modified': '2024-04-02T07:52:51.980647+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7419}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7419': {'gid': 7419, 'name': 'a', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6837/2016280781', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6834', 'nonce': 2016280781}, {'type': 'v1', 'addr': '172.21.15.89:6837', 'nonce': 2016280781}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T07:52:55.261 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T07:52:55.261 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T07:52:55.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.455+0000 7fd256600700 1 Processor -- start 2024-04-02T07:52:55.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.456+0000 7fd256600700 1 -- start start 2024-04-02T07:52:55.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.457+0000 7fd256600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd2500d16a0 0x7fd2500d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.457+0000 7fd256600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2500d2370 0x7fd2500ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.457+0000 7fd256600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd250161420 con 0x7fd25015d9d0 2024-04-02T07:52:55.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.457+0000 7fd256600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2500d4340 con 0x7fd2500d2370 2024-04-02T07:52:55.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.457+0000 7fd256600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd25015edc0 con 0x7fd2500d16a0 2024-04-02T07:52:55.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.457+0000 7fd24fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2500d2370 0x7fd2500ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.457+0000 7fd24fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2500d2370 0x7fd2500ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46600/0 (socket says 172.21.15.89:46600) 2024-04-02T07:52:55.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.457+0000 7fd24fe00700 1 -- 172.21.15.89:0/1992844376 learned_addr learned my addr 172.21.15.89:0/1992844376 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:55.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.457+0000 7fd24ea00700 1 --2- 172.21.15.89:0/1992844376 >> v2:172.21.15.113:3300/0 conn(0x7fd2500d16a0 0x7fd2500d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.457+0000 7fd24fe00700 1 -- 172.21.15.89:0/1992844376 >> v2:172.21.15.113:3300/0 conn(0x7fd2500d16a0 msgr2=0x7fd2500d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:55.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.457+0000 7fd24fe00700 1 --2- 172.21.15.89:0/1992844376 >> v2:172.21.15.113:3300/0 conn(0x7fd2500d16a0 0x7fd2500d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.458+0000 7fd24fe00700 1 -- 172.21.15.89:0/1992844376 >> v1:172.21.15.113:6789/0 conn(0x7fd25015d9d0 legacy=0x7fd2500d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.458+0000 7fd24fe00700 1 -- 172.21.15.89:0/1992844376 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd250161700 con 0x7fd2500d2370 2024-04-02T07:52:55.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.458+0000 7fd24fe00700 1 --2- 172.21.15.89:0/1992844376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2500d2370 0x7fd2500ce360 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fd2500d12f0 tx=0x7fd24800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7830de5abfea9ad6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:55.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.458+0000 7fd24e000700 1 -- 172.21.15.89:0/1992844376 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24800e070 con 0x7fd2500d2370 2024-04-02T07:52:55.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.458+0000 7fd24e000700 1 -- 172.21.15.89:0/1992844376 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24800b9c0 con 0x7fd2500d2370 2024-04-02T07:52:55.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.458+0000 7fd24e000700 1 -- 172.21.15.89:0/1992844376 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24800bd70 con 0x7fd2500d2370 2024-04-02T07:52:55.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.459+0000 7fd256600700 1 -- 172.21.15.89:0/1992844376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2500d2370 msgr2=0x7fd2500ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.459+0000 7fd256600700 1 --2- 172.21.15.89:0/1992844376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2500d2370 0x7fd2500ce360 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fd2500d12f0 tx=0x7fd24800b330 comp rx=0 tx=0).stop 2024-04-02T07:52:55.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.459+0000 7fd256600700 1 -- 172.21.15.89:0/1992844376 shutdown_connections 2024-04-02T07:52:55.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.459+0000 7fd256600700 1 --2- 172.21.15.89:0/1992844376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2500d2370 0x7fd2500ce360 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.459+0000 7fd256600700 1 --2- 172.21.15.89:0/1992844376 >> v2:172.21.15.113:3300/0 conn(0x7fd2500d16a0 0x7fd2500d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.459+0000 7fd256600700 1 -- 172.21.15.89:0/1992844376 >> 172.21.15.89:0/1992844376 conn(0x7fd25008aff0 msgr2=0x7fd2500d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:55.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.459+0000 7fd256600700 1 -- 172.21.15.89:0/1992844376 shutdown_connections 2024-04-02T07:52:55.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.459+0000 7fd256600700 1 -- 172.21.15.89:0/1992844376 wait complete. 2024-04-02T07:52:55.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.460+0000 7fd256600700 1 Processor -- start 2024-04-02T07:52:55.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.461+0000 7fd256600700 1 -- start start 2024-04-02T07:52:55.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.461+0000 7fd256600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2500d16a0 0x7fd25015e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.461+0000 7fd256600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2500d2370 0x7fd25015c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.461+0000 7fd256600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd25015d9d0 0x7fd2500b7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.461+0000 7fd256600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2500cc3d0 con 0x7fd2500d16a0 2024-04-02T07:52:55.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.461+0000 7fd256600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2500cc6b0 con 0x7fd25015d9d0 2024-04-02T07:52:55.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.461+0000 7fd256600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2500cc990 con 0x7fd2500d2370 2024-04-02T07:52:55.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.462+0000 7fd24f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2500d16a0 0x7fd25015e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.462+0000 7fd24f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2500d16a0 0x7fd25015e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46614/0 (socket says 172.21.15.89:46614) 2024-04-02T07:52:55.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.462+0000 7fd24f400700 1 -- 172.21.15.89:0/719743975 learned_addr learned my addr 172.21.15.89:0/719743975 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:55.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.462+0000 7fd24f400700 1 -- 172.21.15.89:0/719743975 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2500d2370 msgr2=0x7fd25015c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.462+0000 7fd24f400700 1 --2- 172.21.15.89:0/719743975 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2500d2370 0x7fd25015c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.462+0000 7fd24f400700 1 -- 172.21.15.89:0/719743975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd25015d9d0 msgr2=0x7fd2500b7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.462+0000 7fd24f400700 1 --2- 172.21.15.89:0/719743975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd25015d9d0 0x7fd2500b7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.462+0000 7fd24f400700 1 -- 172.21.15.89:0/719743975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd248009b30 con 0x7fd2500d16a0 2024-04-02T07:52:55.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.462+0000 7fd24f400700 1 --2- 172.21.15.89:0/719743975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2500d16a0 0x7fd25015e580 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fd24000c450 tx=0x7fd24000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:55.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.462+0000 7fd247400700 1 -- 172.21.15.89:0/719743975 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2400052d0 con 0x7fd2500d16a0 2024-04-02T07:52:55.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.463+0000 7fd247400700 1 -- 172.21.15.89:0/719743975 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd240005430 con 0x7fd2500d16a0 2024-04-02T07:52:55.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.463+0000 7fd247400700 1 -- 172.21.15.89:0/719743975 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2400038c0 con 0x7fd2500d16a0 2024-04-02T07:52:55.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.463+0000 7fd256600700 1 -- 172.21.15.89:0/719743975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2500ccc70 con 0x7fd2500d16a0 2024-04-02T07:52:55.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.463+0000 7fd256600700 1 -- 172.21.15.89:0/719743975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd25015cf20 con 0x7fd2500d16a0 2024-04-02T07:52:55.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.465+0000 7fd247400700 1 -- 172.21.15.89:0/719743975 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd240003a20 con 0x7fd2500d16a0 2024-04-02T07:52:55.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.465+0000 7fd247400700 1 --2- 172.21.15.89:0/719743975 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd23006adb0 0x7fd23006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.466+0000 7fd247400700 1 -- 172.21.15.89:0/719743975 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd240088a40 con 0x7fd2500d16a0 2024-04-02T07:52:55.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.466+0000 7fd24ea00700 1 --2- 172.21.15.89:0/719743975 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd23006adb0 0x7fd23006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.466+0000 7fd24ea00700 1 --2- 172.21.15.89:0/719743975 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd23006adb0 0x7fd23006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd23c006eb0 tx=0x7fd23c008040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:55.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.467+0000 7fd26a14bb80 1 Processor -- start 2024-04-02T07:52:55.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.468+0000 7fd26a14bb80 1 -- start start 2024-04-02T07:52:55.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.468+0000 7fd26a14bb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783eee830 0x555783eeec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.468+0000 7fd26a14bb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ec1be0 0x555783ec4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.468+0000 7fd26a14bb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783eeba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.468+0000 7fd26a14bb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555783e39650 con 0x555783ebfb60 2024-04-02T07:52:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.468+0000 7fd26a14bb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x555783e1ac30 con 0x555783ec1be0 2024-04-02T07:52:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.468+0000 7fd26a14bb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x555783e18430 con 0x555783eee830 2024-04-02T07:52:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.469+0000 7fd24fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783eeba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.469+0000 7fd24fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783eeba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46620/0 (socket says 172.21.15.89:46620) 2024-04-02T07:52:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.469+0000 7fd24fe00700 1 -- 172.21.15.89:0/3210246464 learned_addr learned my addr 172.21.15.89:0/3210246464 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.469+0000 7fd24ea00700 1 --2- 172.21.15.89:0/3210246464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ec1be0 0x555783ec4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.469+0000 7fd24f400700 1 --2- 172.21.15.89:0/3210246464 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783eee830 0x555783eeec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.469+0000 7fd24fe00700 1 -- 172.21.15.89:0/3210246464 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783eee830 msgr2=0x555783eeec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.469+0000 7fd24fe00700 1 --2- 172.21.15.89:0/3210246464 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783eee830 0x555783eeec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.469+0000 7fd24fe00700 1 -- 172.21.15.89:0/3210246464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ec1be0 msgr2=0x555783ec4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:55.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.469+0000 7fd24fe00700 1 --2- 172.21.15.89:0/3210246464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ec1be0 0x555783ec4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.469+0000 7fd24fe00700 1 -- 172.21.15.89:0/3210246464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555783c07790 con 0x555783ebfb60 2024-04-02T07:52:55.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.469+0000 7fd24fe00700 1 --2- 172.21.15.89:0/3210246464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783eeba20 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fd25008c230 tx=0x7fd24800e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0b78656def1b16 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:55.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.470+0000 7fd23be00700 1 -- 172.21.15.89:0/3210246464 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2480099b0 con 0x555783ebfb60 2024-04-02T07:52:55.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.470+0000 7fd23be00700 1 -- 172.21.15.89:0/3210246464 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24800bb30 con 0x555783ebfb60 2024-04-02T07:52:55.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.470+0000 7fd23be00700 1 -- 172.21.15.89:0/3210246464 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd248003a20 con 0x555783ebfb60 2024-04-02T07:52:55.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.470+0000 7fd26a14bb80 1 -- 172.21.15.89:0/3210246464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 msgr2=0x555783eeba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.470+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/3210246464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783eeba20 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fd25008c230 tx=0x7fd24800e000 comp rx=0 tx=0).stop 2024-04-02T07:52:55.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.470+0000 7fd26a14bb80 1 -- 172.21.15.89:0/3210246464 shutdown_connections 2024-04-02T07:52:55.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.470+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/3210246464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783eeba20 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.470+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/3210246464 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783eee830 0x555783eeec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.470+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/3210246464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ec1be0 0x555783ec4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.471+0000 7fd26a14bb80 1 -- 172.21.15.89:0/3210246464 >> 172.21.15.89:0/3210246464 conn(0x555783e226a0 msgr2=0x555783f09640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:55.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.471+0000 7fd26a14bb80 1 -- 172.21.15.89:0/3210246464 shutdown_connections 2024-04-02T07:52:55.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.471+0000 7fd26a14bb80 1 -- 172.21.15.89:0/3210246464 wait complete. 2024-04-02T07:52:55.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.471+0000 7fd26a14bb80 1 -- start start 2024-04-02T07:52:55.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.471+0000 7fd26a14bb80 1 Processor -- start 2024-04-02T07:52:55.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.472+0000 7fd26a14bb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783ee2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.472+0000 7fd26a14bb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ec1be0 0x555783ee9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.472+0000 7fd26a14bb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783eee830 0x555783eea370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.472+0000 7fd26a14bb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555783e13d20 con 0x555783ebfb60 2024-04-02T07:52:55.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.472+0000 7fd26a14bb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x555783daf770 con 0x555783ec1be0 2024-04-02T07:52:55.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.472+0000 7fd26a14bb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x555783dd26f0 con 0x555783eee830 2024-04-02T07:52:55.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.473+0000 7fd26a14bb80 10 client.0 _ll_register_callbacks cb 0x555783db95e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:52:55.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.473+0000 7fd24f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783ee2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.473+0000 7fd24f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783ee2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46622/0 (socket says 172.21.15.89:46622) 2024-04-02T07:52:55.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.473+0000 7fd24f400700 1 -- 172.21.15.89:0/429782169 learned_addr learned my addr 172.21.15.89:0/429782169 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:55.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.473+0000 7fd24ea00700 1 --2- 172.21.15.89:0/429782169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ec1be0 0x555783ee9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.473+0000 7fd24fe00700 1 --2- 172.21.15.89:0/429782169 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783eee830 0x555783eea370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.473+0000 7fd24f400700 1 -- 172.21.15.89:0/429782169 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783eee830 msgr2=0x555783eea370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.473+0000 7fd24f400700 1 --2- 172.21.15.89:0/429782169 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783eee830 0x555783eea370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.473+0000 7fd24f400700 1 -- 172.21.15.89:0/429782169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ec1be0 msgr2=0x555783ee9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.473+0000 7fd24f400700 1 --2- 172.21.15.89:0/429782169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ec1be0 0x555783ee9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.473+0000 7fd24f400700 1 -- 172.21.15.89:0/429782169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd250161700 con 0x555783ebfb60 2024-04-02T07:52:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.473+0000 7fd24fe00700 1 --2- 172.21.15.89:0/429782169 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783eee830 0x555783eea370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.473+0000 7fd24f400700 1 --2- 172.21.15.89:0/429782169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783ee2cf0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fd24000cc10 tx=0x7fd24005b400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8369f9a337a02e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.473+0000 7fd238c00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:52:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd238c00700 1 -- 172.21.15.89:0/429782169 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24006f020 con 0x555783ebfb60 2024-04-02T07:52:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd238c00700 1 -- 172.21.15.89:0/429782169 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd24005b170 con 0x555783ebfb60 2024-04-02T07:52:55.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd26a14bb80 1 -- 172.21.15.89:0/429782169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555783eb6920 con 0x555783ebfb60 2024-04-02T07:52:55.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd238c00700 1 -- 172.21.15.89:0/429782169 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd240057b00 con 0x555783ebfb60 2024-04-02T07:52:55.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd26a14bb80 10 client.7551 fetch_fsmap learned FSMap version 49 2024-04-02T07:52:55.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd26a14bb80 1 -- 172.21.15.89:0/429782169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x555783cf40a0 con 0x555783ebfb60 2024-04-02T07:52:55.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd238c00700 1 -- 172.21.15.89:0/429782169 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555783eb6920 con 0x555783ebfb60 2024-04-02T07:52:55.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd238c00700 1 -- 172.21.15.89:0/429782169 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd240056ac0 con 0x555783ebfb60 2024-04-02T07:52:55.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd26a14bb80 10 client.7551 fetch_fsmap finished waiting for FSMap version 49 2024-04-02T07:52:55.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd26a14bb80 10 client.7551 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T07:52:55.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd26a14bb80 20 client.7551 populate_metadata read hostname 'smithi089' 2024-04-02T07:52:55.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/429782169 >> [v2:172.21.15.89:6834/2016280781,v1:172.21.15.89:6837/2016280781] conn(0x555783ee3a60 0x555783ee3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd26a14bb80 4 client.7551 mds_command: new command op to 7419 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T07:52:55.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd26a14bb80 1 -- 172.21.15.89:0/429782169 --> [v2:172.21.15.89:6834/2016280781,v1:172.21.15.89:6837/2016280781] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555783e936f0 con 0x555783ee3a60 2024-04-02T07:52:55.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.474+0000 7fd24ea00700 1 --2- 172.21.15.89:0/429782169 >> [v2:172.21.15.89:6834/2016280781,v1:172.21.15.89:6837/2016280781] conn(0x555783ee3a60 0x555783ee3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.475+0000 7fd24ea00700 1 --2- 172.21.15.89:0/429782169 >> [v2:172.21.15.89:6834/2016280781,v1:172.21.15.89:6837/2016280781] conn(0x555783ee3a60 0x555783ee3e40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cd86b481579962 server_cookie=1638f1218fda30a1 in_seq=0 out_seq=0 2024-04-02T07:52:55.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.475+0000 7fd238c00700 10 client.7551 ms_handle_connect on v2:172.21.15.89:6834/2016280781 2024-04-02T07:52:55.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.475+0000 7fd238c00700 1 -- 172.21.15.89:0/429782169 <== mds.0 v2:172.21.15.89:6834/2016280781 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555783e936f0 con 0x555783ee3a60 2024-04-02T07:52:55.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.475+0000 7fd238c00700 10 client.7551 handle_command_reply: tid=0 2024-04-02T07:52:55.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.476+0000 7fd26a14bb80 1 client.7551 shutdown 2024-04-02T07:52:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.476+0000 7fd26a14bb80 1 -- 172.21.15.89:0/429782169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 msgr2=0x555783ee2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.476+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/429782169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783ee2cf0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fd24000cc10 tx=0x7fd24005b400 comp rx=0 tx=0).stop 2024-04-02T07:52:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.476+0000 7fd26a14bb80 1 -- 172.21.15.89:0/429782169 shutdown_connections 2024-04-02T07:52:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.476+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/429782169 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783eee830 0x555783eea370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.476+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/429782169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783ee2cf0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.476+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/429782169 >> [v2:172.21.15.89:6834/2016280781,v1:172.21.15.89:6837/2016280781] conn(0x555783ee3a60 0x555783ee3e40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.476+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/429782169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ec1be0 0x555783ee9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.476+0000 7fd26a14bb80 1 -- 172.21.15.89:0/429782169 >> 172.21.15.89:0/429782169 conn(0x555783e226a0 msgr2=0x555783ec3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.476+0000 7fd238c00700 0 client.7551 ms_handle_reset on v2:172.21.15.89:6834/2016280781 2024-04-02T07:52:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.476+0000 7fd26a14bb80 1 -- 172.21.15.89:0/429782169 shutdown_connections 2024-04-02T07:52:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.476+0000 7fd26a14bb80 1 -- 172.21.15.89:0/429782169 wait complete. 2024-04-02T07:52:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.476+0000 7fd26a14bb80 20 client.7551 trim_cache size 0 max 16384 2024-04-02T07:52:55.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.494+0000 7fd26a14bb80 1 Processor -- start 2024-04-02T07:52:55.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.494+0000 7fd26a14bb80 1 -- start start 2024-04-02T07:52:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.495+0000 7fd26a14bb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783ebe860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.495+0000 7fd26a14bb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783ec1be0 0x555783ec07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.495+0000 7fd26a14bb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783eee830 0x555783ec37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.495+0000 7fd26a14bb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555783c08390 con 0x555783ebfb60 2024-04-02T07:52:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.495+0000 7fd26a14bb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x555783ef2100 con 0x555783eee830 2024-04-02T07:52:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.495+0000 7fd26a14bb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x555783eed850 con 0x555783ec1be0 2024-04-02T07:52:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.495+0000 7fd24f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783ebe860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.495+0000 7fd24f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783ebe860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46632/0 (socket says 172.21.15.89:46632) 2024-04-02T07:52:55.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.495+0000 7fd24f400700 1 -- 172.21.15.89:0/3179300845 learned_addr learned my addr 172.21.15.89:0/3179300845 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:55.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.495+0000 7fd24fe00700 1 --2- 172.21.15.89:0/3179300845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783eee830 0x555783ec37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.495+0000 7fd24ea00700 1 --2- 172.21.15.89:0/3179300845 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783ec1be0 0x555783ec07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.496+0000 7fd24f400700 1 -- 172.21.15.89:0/3179300845 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783ec1be0 msgr2=0x555783ec07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:55.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.496+0000 7fd24f400700 1 --2- 172.21.15.89:0/3179300845 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783ec1be0 0x555783ec07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.496+0000 7fd24f400700 1 -- 172.21.15.89:0/3179300845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783eee830 msgr2=0x555783ec37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:55.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.496+0000 7fd24f400700 1 --2- 172.21.15.89:0/3179300845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783eee830 0x555783ec37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.496+0000 7fd24f400700 1 -- 172.21.15.89:0/3179300845 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555783dda010 con 0x555783ebfb60 2024-04-02T07:52:55.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.496+0000 7fd24ea00700 1 --2- 172.21.15.89:0/3179300845 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783ec1be0 0x555783ec07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:52:55.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.496+0000 7fd24f400700 1 --2- 172.21.15.89:0/3179300845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783ebe860 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fd24005f490 tx=0x7fd240059000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2b432b4be0cc05a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:55.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.496+0000 7fd244c00700 1 -- 172.21.15.89:0/3179300845 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240057b00 con 0x555783ebfb60 2024-04-02T07:52:55.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.496+0000 7fd244c00700 1 -- 172.21.15.89:0/3179300845 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd240001120 con 0x555783ebfb60 2024-04-02T07:52:55.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.496+0000 7fd244c00700 1 -- 172.21.15.89:0/3179300845 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240057110 con 0x555783ebfb60 2024-04-02T07:52:55.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.497+0000 7fd26a14bb80 1 -- 172.21.15.89:0/3179300845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 msgr2=0x555783ebe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.497+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/3179300845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783ebe860 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fd24005f490 tx=0x7fd240059000 comp rx=0 tx=0).stop 2024-04-02T07:52:55.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.497+0000 7fd26a14bb80 1 -- 172.21.15.89:0/3179300845 shutdown_connections 2024-04-02T07:52:55.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.497+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/3179300845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783eee830 0x555783ec37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.497+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/3179300845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783ebfb60 0x555783ebe860 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.497+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/3179300845 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783ec1be0 0x555783ec07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.497+0000 7fd26a14bb80 1 -- 172.21.15.89:0/3179300845 >> 172.21.15.89:0/3179300845 conn(0x555783e226a0 msgr2=0x555783ec5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:55.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.497+0000 7fd26a14bb80 1 -- 172.21.15.89:0/3179300845 shutdown_connections 2024-04-02T07:52:55.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.497+0000 7fd26a14bb80 1 -- 172.21.15.89:0/3179300845 wait complete. 2024-04-02T07:52:55.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.497+0000 7fd26a14bb80 1 -- start start 2024-04-02T07:52:55.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.498+0000 7fd26a14bb80 1 Processor -- start 2024-04-02T07:52:55.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.499+0000 7fd26a14bb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ebfb60 0x555783f56df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.499+0000 7fd26a14bb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783ec1be0 0x555783eff300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.499+0000 7fd26a14bb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783eee830 0x555783ee3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.499+0000 7fd26a14bb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555783f004c0 con 0x555783eee830 2024-04-02T07:52:55.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.499+0000 7fd26a14bb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x555783ee9730 con 0x555783ebfb60 2024-04-02T07:52:55.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.499+0000 7fd26a14bb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x555783efff90 con 0x555783ec1be0 2024-04-02T07:52:55.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.499+0000 7fd26a14bb80 10 client.0 _ll_register_callbacks cb 0x7fd240057cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:52:55.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.500+0000 7fd24fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783eee830 0x555783ee3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.500+0000 7fd24fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783eee830 0x555783ee3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46644/0 (socket says 172.21.15.89:46644) 2024-04-02T07:52:55.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.500+0000 7fd24fe00700 1 -- 172.21.15.89:0/111516982 learned_addr learned my addr 172.21.15.89:0/111516982 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:55.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.500+0000 7fd24ea00700 1 --2- 172.21.15.89:0/111516982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783ec1be0 0x555783eff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.500+0000 7fd24f400700 1 --2- 172.21.15.89:0/111516982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ebfb60 0x555783f56df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.500+0000 7fd24fe00700 1 -- 172.21.15.89:0/111516982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783ec1be0 msgr2=0x555783eff300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:55.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.500+0000 7fd24fe00700 1 --2- 172.21.15.89:0/111516982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783ec1be0 0x555783eff300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.500+0000 7fd24fe00700 1 -- 172.21.15.89:0/111516982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ebfb60 msgr2=0x555783f56df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:55.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.500+0000 7fd24fe00700 1 --2- 172.21.15.89:0/111516982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ebfb60 0x555783f56df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.500+0000 7fd24fe00700 1 -- 172.21.15.89:0/111516982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555783cf40a0 con 0x555783eee830 2024-04-02T07:52:55.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.500+0000 7fd24fe00700 1 --2- 172.21.15.89:0/111516982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783eee830 0x555783ee3e40 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x555783f00690 tx=0x7fd248004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa445d93d3a99ba1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:55.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.500+0000 7fd238c00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:52:55.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd238c00700 1 -- 172.21.15.89:0/111516982 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24800e040 con 0x555783eee830 2024-04-02T07:52:55.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd238c00700 1 -- 172.21.15.89:0/111516982 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd24800bca0 con 0x555783eee830 2024-04-02T07:52:55.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd26a14bb80 1 -- 172.21.15.89:0/111516982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd24800bb30 con 0x555783eee830 2024-04-02T07:52:55.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd238c00700 1 -- 172.21.15.89:0/111516982 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd248004600 con 0x555783eee830 2024-04-02T07:52:55.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd26a14bb80 10 client.7557 fetch_fsmap learned FSMap version 49 2024-04-02T07:52:55.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd26a14bb80 1 -- 172.21.15.89:0/111516982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x555783c681d0 con 0x555783eee830 2024-04-02T07:52:55.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd238c00700 1 -- 172.21.15.89:0/111516982 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24800bb30 con 0x555783eee830 2024-04-02T07:52:55.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd238c00700 1 -- 172.21.15.89:0/111516982 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd248019070 con 0x555783eee830 2024-04-02T07:52:55.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd26a14bb80 10 client.7557 fetch_fsmap finished waiting for FSMap version 49 2024-04-02T07:52:55.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd26a14bb80 10 client.7557 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T07:52:55.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd26a14bb80 20 client.7557 populate_metadata read hostname 'smithi089' 2024-04-02T07:52:55.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/111516982 >> [v2:172.21.15.89:6834/2016280781,v1:172.21.15.89:6837/2016280781] conn(0x555783ee8bb0 0x555783ee8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd26a14bb80 4 client.7557 mds_command: new command op to 7419 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T07:52:55.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd26a14bb80 1 -- 172.21.15.89:0/111516982 --> [v2:172.21.15.89:6834/2016280781,v1:172.21.15.89:6837/2016280781] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555783c4d040 con 0x555783ee8bb0 2024-04-02T07:52:55.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.501+0000 7fd24f400700 1 --2- 172.21.15.89:0/111516982 >> [v2:172.21.15.89:6834/2016280781,v1:172.21.15.89:6837/2016280781] conn(0x555783ee8bb0 0x555783ee8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.502+0000 7fd24f400700 1 --2- 172.21.15.89:0/111516982 >> [v2:172.21.15.89:6834/2016280781,v1:172.21.15.89:6837/2016280781] conn(0x555783ee8bb0 0x555783ee8f90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f99458190e3552 server_cookie=85fd615f6416e9a in_seq=0 out_seq=0 2024-04-02T07:52:55.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.502+0000 7fd238c00700 10 client.7557 ms_handle_connect on v2:172.21.15.89:6834/2016280781 2024-04-02T07:52:55.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.502+0000 7fd238c00700 1 -- 172.21.15.89:0/111516982 <== mds.0 v2:172.21.15.89:6834/2016280781 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x555783c4d040 con 0x555783ee8bb0 2024-04-02T07:52:55.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.502+0000 7fd238c00700 10 client.7557 handle_command_reply: tid=0 2024-04-02T07:52:55.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.502+0000 7fd26a14bb80 1 client.7557 shutdown 2024-04-02T07:52:55.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.502+0000 7fd26a14bb80 1 -- 172.21.15.89:0/111516982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783eee830 msgr2=0x555783ee3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.502+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/111516982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783eee830 0x555783ee3e40 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x555783f00690 tx=0x7fd248004b40 comp rx=0 tx=0).stop 2024-04-02T07:52:55.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.502+0000 7fd26a14bb80 1 -- 172.21.15.89:0/111516982 shutdown_connections 2024-04-02T07:52:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.502+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/111516982 >> [v2:172.21.15.89:6834/2016280781,v1:172.21.15.89:6837/2016280781] conn(0x555783ee8bb0 0x555783ee8f90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.502+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/111516982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555783ebfb60 0x555783f56df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.502+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/111516982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555783eee830 0x555783ee3e40 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.502+0000 7fd238c00700 0 client.7557 ms_handle_reset on v2:172.21.15.89:6834/2016280781 2024-04-02T07:52:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.502+0000 7fd26a14bb80 1 --2- 172.21.15.89:0/111516982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555783ec1be0 0x555783eff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.502+0000 7fd26a14bb80 1 -- 172.21.15.89:0/111516982 >> 172.21.15.89:0/111516982 conn(0x555783e226a0 msgr2=0x555783ec5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.503+0000 7fd26a14bb80 1 -- 172.21.15.89:0/111516982 shutdown_connections 2024-04-02T07:52:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.503+0000 7fd26a14bb80 1 -- 172.21.15.89:0/111516982 wait complete. 2024-04-02T07:52:55.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.503+0000 7fd26a14bb80 20 client.7557 trim_cache size 0 max 16384 2024-04-02T07:52:55.508 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:52:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T07:52:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T07:52:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 7419, 2024-04-02T07:52:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T07:52:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T07:52:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T07:52:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 6.5308765419999997, 2024-04-02T07:52:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 49, 2024-04-02T07:52:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 111, 2024-04-02T07:52:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 111, 2024-04-02T07:52:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 11.960520646999999 2024-04-02T07:52:55.510 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:52:55.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.503+0000 7fd256600700 1 -- 172.21.15.89:0/719743975 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd23006adb0 msgr2=0x7fd23006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.503+0000 7fd256600700 1 --2- 172.21.15.89:0/719743975 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd23006adb0 0x7fd23006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd23c006eb0 tx=0x7fd23c008040 comp rx=0 tx=0).stop 2024-04-02T07:52:55.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.503+0000 7fd256600700 1 -- 172.21.15.89:0/719743975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2500d16a0 msgr2=0x7fd25015e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.503+0000 7fd256600700 1 --2- 172.21.15.89:0/719743975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2500d16a0 0x7fd25015e580 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fd24000c450 tx=0x7fd24000c810 comp rx=0 tx=0).stop 2024-04-02T07:52:55.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.503+0000 7fd256600700 1 -- 172.21.15.89:0/719743975 shutdown_connections 2024-04-02T07:52:55.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.503+0000 7fd256600700 1 --2- 172.21.15.89:0/719743975 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd23006adb0 0x7fd23006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.503+0000 7fd256600700 1 --2- 172.21.15.89:0/719743975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd25015d9d0 0x7fd2500b7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.503+0000 7fd256600700 1 --2- 172.21.15.89:0/719743975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2500d16a0 0x7fd25015e580 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.503+0000 7fd256600700 1 --2- 172.21.15.89:0/719743975 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2500d2370 0x7fd25015c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.503+0000 7fd256600700 1 -- 172.21.15.89:0/719743975 >> 172.21.15.89:0/719743975 conn(0x7fd25008aff0 msgr2=0x7fd25006cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:55.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.503+0000 7fd256600700 1 -- 172.21.15.89:0/719743975 shutdown_connections 2024-04-02T07:52:55.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.504+0000 7fd256600700 1 -- 172.21.15.89:0/719743975 wait complete. 2024-04-02T07:52:55.523 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T07:52:55.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.713+0000 7fb84d800700 1 Processor -- start 2024-04-02T07:52:55.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.713+0000 7fb84d800700 1 -- start start 2024-04-02T07:52:55.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.714+0000 7fb84d800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb848160f10 0x7fb8480d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.714+0000 7fb84d800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb8480cf230 0x7fb8480d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.714+0000 7fb84d800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8480ceda0 con 0x7fb848161590 2024-04-02T07:52:55.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.714+0000 7fb84d800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8480d23d0 con 0x7fb848160f10 2024-04-02T07:52:55.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.714+0000 7fb84d800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8481612f0 con 0x7fb8480cf230 2024-04-02T07:52:55.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.714+0000 7fb846a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb848160f10 0x7fb8480d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.714+0000 7fb846a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb848160f10 0x7fb8480d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46658/0 (socket says 172.21.15.89:46658) 2024-04-02T07:52:55.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.714+0000 7fb846a00700 1 -- 172.21.15.89:0/316177067 learned_addr learned my addr 172.21.15.89:0/316177067 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:55.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.715+0000 7fb846a00700 1 -- 172.21.15.89:0/316177067 >> v2:172.21.15.113:3300/0 conn(0x7fb8480cf230 msgr2=0x7fb8480d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:55.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.715+0000 7fb846a00700 1 --2- 172.21.15.89:0/316177067 >> v2:172.21.15.113:3300/0 conn(0x7fb8480cf230 0x7fb8480d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.715+0000 7fb846a00700 1 -- 172.21.15.89:0/316177067 >> v1:172.21.15.113:6789/0 conn(0x7fb848161590 legacy=0x7fb8480b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:55.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.715+0000 7fb846a00700 1 -- 172.21.15.89:0/316177067 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8480ce470 con 0x7fb848160f10 2024-04-02T07:52:55.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.715+0000 7fb846a00700 1 --2- 172.21.15.89:0/316177067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb848160f10 0x7fb8480d18b0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fb84808cdb0 tx=0x7fb838009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53690d755e42609f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:55.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.715+0000 7fb845600700 1 -- 172.21.15.89:0/316177067 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838004230 con 0x7fb848160f10 2024-04-02T07:52:55.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.715+0000 7fb845600700 1 -- 172.21.15.89:0/316177067 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb838004390 con 0x7fb848160f10 2024-04-02T07:52:55.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.715+0000 7fb845600700 1 -- 172.21.15.89:0/316177067 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8380036a0 con 0x7fb848160f10 2024-04-02T07:52:55.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.716+0000 7fb84d800700 1 -- 172.21.15.89:0/316177067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb848160f10 msgr2=0x7fb8480d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.716+0000 7fb84d800700 1 --2- 172.21.15.89:0/316177067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb848160f10 0x7fb8480d18b0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fb84808cdb0 tx=0x7fb838009bd0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.716+0000 7fb84d800700 1 -- 172.21.15.89:0/316177067 shutdown_connections 2024-04-02T07:52:55.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.716+0000 7fb84d800700 1 --2- 172.21.15.89:0/316177067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb848160f10 0x7fb8480d18b0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.716+0000 7fb84d800700 1 --2- 172.21.15.89:0/316177067 >> v2:172.21.15.113:3300/0 conn(0x7fb8480cf230 0x7fb8480d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.716+0000 7fb84d800700 1 -- 172.21.15.89:0/316177067 >> 172.21.15.89:0/316177067 conn(0x7fb84808aff0 msgr2=0x7fb8480d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:55.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.716+0000 7fb84d800700 1 -- 172.21.15.89:0/316177067 shutdown_connections 2024-04-02T07:52:55.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.716+0000 7fb84d800700 1 -- 172.21.15.89:0/316177067 wait complete. 2024-04-02T07:52:55.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.718+0000 7fb84d800700 1 Processor -- start 2024-04-02T07:52:55.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.718+0000 7fb84d800700 1 -- start start 2024-04-02T07:52:55.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb84d800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8480cf230 0x7fb84815e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb84d800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb848160f10 0x7fb848145560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb84d800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb848161590 0x7fb8480b6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb84d800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84815d080 con 0x7fb848160f10 2024-04-02T07:52:55.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb84d800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84815d320 con 0x7fb848161590 2024-04-02T07:52:55.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb84d800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84815d600 con 0x7fb8480cf230 2024-04-02T07:52:55.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb846000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb848160f10 0x7fb848145560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb846000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb848160f10 0x7fb848145560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46660/0 (socket says 172.21.15.89:46660) 2024-04-02T07:52:55.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb846000700 1 -- 172.21.15.89:0/1835746654 learned_addr learned my addr 172.21.15.89:0/1835746654 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:55.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb846a00700 1 --2- 172.21.15.89:0/1835746654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8480cf230 0x7fb84815e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb847400700 1 --2- 172.21.15.89:0/1835746654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb848161590 0x7fb8480b6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb846000700 1 -- 172.21.15.89:0/1835746654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8480cf230 msgr2=0x7fb84815e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb846000700 1 --2- 172.21.15.89:0/1835746654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8480cf230 0x7fb84815e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb846000700 1 -- 172.21.15.89:0/1835746654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb848161590 msgr2=0x7fb8480b6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb846000700 1 --2- 172.21.15.89:0/1835746654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb848161590 0x7fb8480b6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.719+0000 7fb846000700 1 -- 172.21.15.89:0/1835746654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb838009550 con 0x7fb848160f10 2024-04-02T07:52:55.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.720+0000 7fb846000700 1 --2- 172.21.15.89:0/1835746654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb848160f10 0x7fb848145560 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fb83400b4a0 tx=0x7fb83400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:55.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.720+0000 7fb83ea00700 1 -- 172.21.15.89:0/1835746654 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834011830 con 0x7fb848160f10 2024-04-02T07:52:55.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.720+0000 7fb83ea00700 1 -- 172.21.15.89:0/1835746654 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb834011990 con 0x7fb848160f10 2024-04-02T07:52:55.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.720+0000 7fb83ea00700 1 -- 172.21.15.89:0/1835746654 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834011d40 con 0x7fb848160f10 2024-04-02T07:52:55.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.720+0000 7fb84d800700 1 -- 172.21.15.89:0/1835746654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84815d8e0 con 0x7fb848160f10 2024-04-02T07:52:55.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.720+0000 7fb84d800700 1 -- 172.21.15.89:0/1835746654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb84815dea0 con 0x7fb848160f10 2024-04-02T07:52:55.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.721+0000 7fb84d800700 1 -- 172.21.15.89:0/1835746654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84815e060 con 0x7fb848160f10 2024-04-02T07:52:55.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.722+0000 7fb83ea00700 1 -- 172.21.15.89:0/1835746654 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb83401a400 con 0x7fb848160f10 2024-04-02T07:52:55.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.723+0000 7fb83ea00700 1 --2- 172.21.15.89:0/1835746654 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb82806adb0 0x7fb82806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:55.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.723+0000 7fb83ea00700 1 -- 172.21.15.89:0/1835746654 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb834089660 con 0x7fb848160f10 2024-04-02T07:52:55.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.723+0000 7fb846a00700 1 --2- 172.21.15.89:0/1835746654 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb82806adb0 0x7fb82806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:55.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.724+0000 7fb846a00700 1 --2- 172.21.15.89:0/1835746654 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb82806adb0 0x7fb82806d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb838009130 tx=0x7fb838008030 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:55.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.727+0000 7fb83ea00700 1 -- 172.21.15.89:0/1835746654 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb834054540 con 0x7fb848160f10 2024-04-02T07:52:55.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.976+0000 7fb84d800700 1 -- 172.21.15.89:0/1835746654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb848067720 con 0x7fb848160f10 2024-04-02T07:52:55.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.976+0000 7fb83ea00700 1 -- 172.21.15.89:0/1835746654 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v49) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb834057b60 con 0x7fb848160f10 2024-04-02T07:52:55.978 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T07:52:55.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.979+0000 7fb84d800700 1 -- 172.21.15.89:0/1835746654 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb82806adb0 msgr2=0x7fb82806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.979+0000 7fb84d800700 1 --2- 172.21.15.89:0/1835746654 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb82806adb0 0x7fb82806d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb838009130 tx=0x7fb838008030 comp rx=0 tx=0).stop 2024-04-02T07:52:55.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.979+0000 7fb84d800700 1 -- 172.21.15.89:0/1835746654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb848160f10 msgr2=0x7fb848145560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:55.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.979+0000 7fb84d800700 1 --2- 172.21.15.89:0/1835746654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb848160f10 0x7fb848145560 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fb83400b4a0 tx=0x7fb83400b860 comp rx=0 tx=0).stop 2024-04-02T07:52:55.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.979+0000 7fb84d800700 1 -- 172.21.15.89:0/1835746654 shutdown_connections 2024-04-02T07:52:55.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.979+0000 7fb84d800700 1 --2- 172.21.15.89:0/1835746654 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb82806adb0 0x7fb82806d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.979+0000 7fb84d800700 1 --2- 172.21.15.89:0/1835746654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb848161590 0x7fb8480b6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.979+0000 7fb84d800700 1 --2- 172.21.15.89:0/1835746654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8480cf230 0x7fb84815e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.979+0000 7fb84d800700 1 --2- 172.21.15.89:0/1835746654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb848160f10 0x7fb848145560 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:55.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.979+0000 7fb84d800700 1 -- 172.21.15.89:0/1835746654 >> 172.21.15.89:0/1835746654 conn(0x7fb84808aff0 msgr2=0x7fb8480caf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:55.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.979+0000 7fb84d800700 1 -- 172.21.15.89:0/1835746654 shutdown_connections 2024-04-02T07:52:55.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:55.979+0000 7fb84d800700 1 -- 172.21.15.89:0/1835746654 wait complete. 2024-04-02T07:52:55.998 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T07:52:55.998 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T07:52:55.998 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T07:52:55.999 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T07:52:55.999 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T07:52:55.999 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T07:52:55.999 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T07:52:55.999 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T07:52:55.999 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:52:55.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:55.999 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T07:52:56.044 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:52:56.044 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:52:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:52:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:52:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T07:52:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:52:56.045 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:52:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:52:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T07:52:56.045 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:52:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:52:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T07:52:56.045 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:52:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:52:56.046 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T07:52:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:52:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T07:52:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 41453sec preferred_lft 41453sec 2024-04-02T07:52:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T07:52:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:52:56.046 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T07:52:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:52:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T07:52:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:52:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T07:52:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:52:56.047 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:52:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T07:52:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T07:52:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:52:56.047 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:52:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T07:52:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T07:52:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:52:56.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:56.048 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T07:52:56.150 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T07:52:56.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:56.152 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T07:52:56.199 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T07:52:56.199 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T07:52:56.199 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T07:52:56.199 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T07:52:56.199 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T07:52:56.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:56.200 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:52:56.200 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:52:56.201 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T07:52:56.201 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T07:52:56.201 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T07:52:56.201 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:52:56.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:56.360 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T07:52:56.405 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T07:52:56.405 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T07:52:56.406 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:52:56.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:56.406 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T07:52:56.514 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:52:56.514 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T07:52:56.558 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:52:56.559 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:52:56.559 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:52:56.642 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T07:52:56.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:56.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:52:56.782 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:52:56.782 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:52:56.809 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:52:56.809 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:52:56.810 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T07:52:56.810 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T07:52:56.810 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T07:52:56.814 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T07:52:56.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:56.814 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:52:56.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:52:56.881 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:52:56.978 INFO:teuthology.orchestra.run.smithi089.stdout:client7566 2024-04-02T07:52:56.978 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:52:56.978 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:52:57.024 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T07:52:57.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.256+0000 7f5435600700 1 Processor -- start 2024-04-02T07:52:57.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.257+0000 7f5435600700 1 -- start start 2024-04-02T07:52:57.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.258+0000 7f5435600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f543015a5c0 0x7f54300709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:57.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.258+0000 7f5435600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f54300ce840 0x7f54300cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:57.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.258+0000 7f5435600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54300cb310 con 0x7f543015dfe0 2024-04-02T07:52:57.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.258+0000 7f5435600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5430070f00 con 0x7f54300ce840 2024-04-02T07:52:57.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.258+0000 7f5435600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54300711a0 con 0x7f543015a5c0 2024-04-02T07:52:57.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.258+0000 7f542f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f54300ce840 0x7f54300cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:57.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.258+0000 7f542e000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f543015dfe0 0x7f54300ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33334/0 (socket says 172.21.15.89:33334) 2024-04-02T07:52:57.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.258+0000 7f542e000700 1 -- 172.21.15.89:0/2282319072 learned_addr learned my addr 172.21.15.89:0/2282319072 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:57.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.258+0000 7f542ea00700 1 --2- 172.21.15.89:0/2282319072 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f543015a5c0 0x7f54300709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:57.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.259+0000 7f542f400700 1 -- 172.21.15.89:0/2282319072 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f543015a5c0 msgr2=0x7f54300709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:57.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.259+0000 7f542f400700 1 --2- 172.21.15.89:0/2282319072 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f543015a5c0 0x7f54300709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.259+0000 7f542d600700 1 -- 172.21.15.89:0/2282319072 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842944862 0 0) 0x7f54300cb310 con 0x7f543015dfe0 2024-04-02T07:52:57.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.259+0000 7f542f400700 1 -- 172.21.15.89:0/2282319072 >> v1:172.21.15.89:6789/0 conn(0x7f543015dfe0 legacy=0x7f54300ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:57.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.259+0000 7f542f400700 1 -- 172.21.15.89:0/2282319072 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5430071480 con 0x7f54300ce840 2024-04-02T07:52:57.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.259+0000 7f542f400700 1 --2- 172.21.15.89:0/2282319072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54300ce840 0x7f54300cba80 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f5428009d90 tx=0x7f542800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09cbebd42ede439 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:57.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.259+0000 7f542d600700 1 -- 172.21.15.89:0/2282319072 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542800e070 con 0x7f54300ce840 2024-04-02T07:52:57.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.259+0000 7f542d600700 1 -- 172.21.15.89:0/2282319072 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542800bb20 con 0x7f54300ce840 2024-04-02T07:52:57.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.259+0000 7f542d600700 1 -- 172.21.15.89:0/2282319072 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5428012430 con 0x7f54300ce840 2024-04-02T07:52:57.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.260+0000 7f5435600700 1 -- 172.21.15.89:0/2282319072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54300ce840 msgr2=0x7f54300cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:57.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.260+0000 7f5435600700 1 --2- 172.21.15.89:0/2282319072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54300ce840 0x7f54300cba80 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f5428009d90 tx=0x7f542800b330 comp rx=0 tx=0).stop 2024-04-02T07:52:57.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.260+0000 7f5435600700 1 -- 172.21.15.89:0/2282319072 shutdown_connections 2024-04-02T07:52:57.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.260+0000 7f5435600700 1 --2- 172.21.15.89:0/2282319072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54300ce840 0x7f54300cba80 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.260+0000 7f5435600700 1 --2- 172.21.15.89:0/2282319072 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f543015a5c0 0x7f54300709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.260+0000 7f5435600700 1 -- 172.21.15.89:0/2282319072 >> 172.21.15.89:0/2282319072 conn(0x7f54300b0d70 msgr2=0x7f54300b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:57.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.260+0000 7f5435600700 1 -- 172.21.15.89:0/2282319072 shutdown_connections 2024-04-02T07:52:57.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.260+0000 7f5435600700 1 -- 172.21.15.89:0/2282319072 wait complete. 2024-04-02T07:52:57.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.261+0000 7f5435600700 1 Processor -- start 2024-04-02T07:52:57.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.262+0000 7f5435600700 1 -- start start 2024-04-02T07:52:57.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.262+0000 7f5435600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54300ce840 0x7f54301f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:57.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.262+0000 7f5435600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f543015a5c0 0x7f5430145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:57.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.263+0000 7f5435600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f543015dfe0 0x7f5430156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:57.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.263+0000 7f5435600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5430157fc0 con 0x7f54300ce840 2024-04-02T07:52:57.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.263+0000 7f5435600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54301582a0 con 0x7f543015a5c0 2024-04-02T07:52:57.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.263+0000 7f5435600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5430158580 con 0x7f543015dfe0 2024-04-02T07:52:57.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.263+0000 7f542ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54300ce840 0x7f54301f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:57.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.263+0000 7f542ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54300ce840 0x7f54301f2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46682/0 (socket says 172.21.15.89:46682) 2024-04-02T07:52:57.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.263+0000 7f542ea00700 1 -- 172.21.15.89:0/3186035249 learned_addr learned my addr 172.21.15.89:0/3186035249 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:57.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.263+0000 7f542ea00700 1 -- 172.21.15.89:0/3186035249 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f543015dfe0 msgr2=0x7f5430156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:57.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.263+0000 7f542ea00700 1 --2- 172.21.15.89:0/3186035249 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f543015dfe0 0x7f5430156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.263+0000 7f542ea00700 1 -- 172.21.15.89:0/3186035249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f543015a5c0 msgr2=0x7f5430145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:57.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.263+0000 7f542ea00700 1 --2- 172.21.15.89:0/3186035249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f543015a5c0 0x7f5430145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.263+0000 7f542ea00700 1 -- 172.21.15.89:0/3186035249 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5428009a70 con 0x7f54300ce840 2024-04-02T07:52:57.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.263+0000 7f542ea00700 1 --2- 172.21.15.89:0/3186035249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54300ce840 0x7f54301f2400 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f542000e9b0 tx=0x7f542000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:57.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.264+0000 7f5426a00700 1 -- 172.21.15.89:0/3186035249 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5420009710 con 0x7f54300ce840 2024-04-02T07:52:57.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.264+0000 7f5426a00700 1 -- 172.21.15.89:0/3186035249 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542000c850 con 0x7f54300ce840 2024-04-02T07:52:57.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.264+0000 7f5426a00700 1 -- 172.21.15.89:0/3186035249 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542000cbc0 con 0x7f54300ce840 2024-04-02T07:52:57.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.264+0000 7f5435600700 1 -- 172.21.15.89:0/3186035249 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5430158860 con 0x7f54300ce840 2024-04-02T07:52:57.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.264+0000 7f5435600700 1 -- 172.21.15.89:0/3186035249 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54301f2940 con 0x7f54300ce840 2024-04-02T07:52:57.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.265+0000 7f5435600700 1 -- 172.21.15.89:0/3186035249 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5430067720 con 0x7f54300ce840 2024-04-02T07:52:57.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.266+0000 7f5426a00700 1 -- 172.21.15.89:0/3186035249 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f542000cd20 con 0x7f54300ce840 2024-04-02T07:52:57.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.267+0000 7f5426a00700 1 --2- 172.21.15.89:0/3186035249 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f541006ade0 0x7f541006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:57.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.267+0000 7f5426a00700 1 -- 172.21.15.89:0/3186035249 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f54200896f0 con 0x7f54300ce840 2024-04-02T07:52:57.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.267+0000 7f542e000700 1 --2- 172.21.15.89:0/3186035249 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f541006ade0 0x7f541006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:57.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.268+0000 7f542e000700 1 --2- 172.21.15.89:0/3186035249 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f541006ade0 0x7f541006d2a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f54301f5ff0 tx=0x7f541c0079f0 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:57.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.271+0000 7f5426a00700 1 -- 172.21.15.89:0/3186035249 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5420054660 con 0x7f54300ce840 2024-04-02T07:52:57.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.503+0000 7f5435600700 1 -- 172.21.15.89:0/3186035249 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f54300cb560 con 0x7f541006ade0 2024-04-02T07:52:57.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.504+0000 7f5426a00700 1 -- 172.21.15.89:0/3186035249 <== mgr.6775 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f541c004870 con 0x7f541006ade0 2024-04-02T07:52:57.506 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:52:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:52:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T07:52:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:52:57.506 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:52:57.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.507+0000 7f5435600700 1 -- 172.21.15.89:0/3186035249 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f541006ade0 msgr2=0x7f541006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:57.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.507+0000 7f5435600700 1 --2- 172.21.15.89:0/3186035249 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f541006ade0 0x7f541006d2a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f54301f5ff0 tx=0x7f541c0079f0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.507+0000 7f5435600700 1 -- 172.21.15.89:0/3186035249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54300ce840 msgr2=0x7f54301f2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:57.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.507+0000 7f5435600700 1 --2- 172.21.15.89:0/3186035249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54300ce840 0x7f54301f2400 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f542000e9b0 tx=0x7f542000ed70 comp rx=0 tx=0).stop 2024-04-02T07:52:57.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.507+0000 7f5435600700 1 -- 172.21.15.89:0/3186035249 shutdown_connections 2024-04-02T07:52:57.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.507+0000 7f5435600700 1 --2- 172.21.15.89:0/3186035249 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f541006ade0 0x7f541006d2a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.507+0000 7f5435600700 1 --2- 172.21.15.89:0/3186035249 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f543015dfe0 0x7f5430156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.507+0000 7f5435600700 1 --2- 172.21.15.89:0/3186035249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54300ce840 0x7f54301f2400 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.507+0000 7f5435600700 1 --2- 172.21.15.89:0/3186035249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f543015a5c0 0x7f5430145d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.507+0000 7f5435600700 1 -- 172.21.15.89:0/3186035249 >> 172.21.15.89:0/3186035249 conn(0x7f54300b0d70 msgr2=0x7f54300b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:57.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.507+0000 7f5435600700 1 -- 172.21.15.89:0/3186035249 shutdown_connections 2024-04-02T07:52:57.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.507+0000 7f5435600700 1 -- 172.21.15.89:0/3186035249 wait complete. 2024-04-02T07:52:57.527 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T07:52:57.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.719+0000 7f0908000700 1 Processor -- start 2024-04-02T07:52:57.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.719+0000 7f0908000700 1 -- start start 2024-04-02T07:52:57.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.721+0000 7f0908000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09000cd020 0x7f09000cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:57.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.721+0000 7f0908000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f09000b2070 0x7f09000b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:57.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.721+0000 7f0908000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09000cdae0 con 0x7f09000cbb30 2024-04-02T07:52:57.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.721+0000 7f0908000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09000cddc0 con 0x7f09000b2070 2024-04-02T07:52:57.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.721+0000 7f0908000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09000ce0a0 con 0x7f09000cd020 2024-04-02T07:52:57.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.721+0000 7f0906000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f09000b2070 0x7f09000b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:57.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.721+0000 7f0905600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f09000cbb30 0x7f09000cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33350/0 (socket says 172.21.15.89:33350) 2024-04-02T07:52:57.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.721+0000 7f0905600700 1 -- 172.21.15.89:0/2056143064 learned_addr learned my addr 172.21.15.89:0/2056143064 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:57.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.721+0000 7f0904c00700 1 --2- 172.21.15.89:0/2056143064 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09000cd020 0x7f09000cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:57.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.721+0000 7f0906000700 1 -- 172.21.15.89:0/2056143064 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09000cd020 msgr2=0x7f09000cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:57.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.721+0000 7f0906000700 1 --2- 172.21.15.89:0/2056143064 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09000cd020 0x7f09000cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.721+0000 7f0906000700 1 -- 172.21.15.89:0/2056143064 >> v1:172.21.15.89:6789/0 conn(0x7f09000cbb30 legacy=0x7f09000cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:57.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.721+0000 7f0906000700 1 -- 172.21.15.89:0/2056143064 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090006dc20 con 0x7f09000b2070 2024-04-02T07:52:57.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.721+0000 7f08ffe00700 1 -- 172.21.15.89:0/2056143064 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907796549 0 0) 0x7f09000cdae0 con 0x7f09000cbb30 2024-04-02T07:52:57.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.722+0000 7f0906000700 1 --2- 172.21.15.89:0/2056143064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09000b2070 0x7f09000b2500 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f08f8009d90 tx=0x7f08f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e4bcf00cd139af server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:57.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.722+0000 7f08ffe00700 1 -- 172.21.15.89:0/2056143064 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f800e070 con 0x7f09000b2070 2024-04-02T07:52:57.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.722+0000 7f08ffe00700 1 -- 172.21.15.89:0/2056143064 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f800bbd0 con 0x7f09000b2070 2024-04-02T07:52:57.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.722+0000 7f08ffe00700 1 -- 172.21.15.89:0/2056143064 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f8012430 con 0x7f09000b2070 2024-04-02T07:52:57.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.722+0000 7f0908000700 1 -- 172.21.15.89:0/2056143064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09000b2070 msgr2=0x7f09000b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:57.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.722+0000 7f0908000700 1 --2- 172.21.15.89:0/2056143064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09000b2070 0x7f09000b2500 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f08f8009d90 tx=0x7f08f800b330 comp rx=0 tx=0).stop 2024-04-02T07:52:57.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.722+0000 7f0908000700 1 -- 172.21.15.89:0/2056143064 shutdown_connections 2024-04-02T07:52:57.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.722+0000 7f0908000700 1 --2- 172.21.15.89:0/2056143064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09000b2070 0x7f09000b2500 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.722+0000 7f0908000700 1 --2- 172.21.15.89:0/2056143064 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09000cd020 0x7f09000cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.722+0000 7f0908000700 1 -- 172.21.15.89:0/2056143064 >> 172.21.15.89:0/2056143064 conn(0x7f09000b0d70 msgr2=0x7f09000b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:57.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.723+0000 7f0908000700 1 -- 172.21.15.89:0/2056143064 shutdown_connections 2024-04-02T07:52:57.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.723+0000 7f0908000700 1 -- 172.21.15.89:0/2056143064 wait complete. 2024-04-02T07:52:57.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.724+0000 7f0908000700 1 Processor -- start 2024-04-02T07:52:57.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.724+0000 7f0908000700 1 -- start start 2024-04-02T07:52:57.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.725+0000 7f0908000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09000b2070 0x7f0900156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:57.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.725+0000 7f0908000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09000cbb30 0x7f09001f6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:57.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.725+0000 7f0908000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09000cd020 0x7f09001576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:57.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.725+0000 7f0908000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0900156200 con 0x7f09000b2070 2024-04-02T07:52:57.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.725+0000 7f0908000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09001f7260 con 0x7f09000cbb30 2024-04-02T07:52:57.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.725+0000 7f0908000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090015e9e0 con 0x7f09000cd020 2024-04-02T07:52:57.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.726+0000 7f0905600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09000b2070 0x7f0900156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:57.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.726+0000 7f0905600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09000b2070 0x7f0900156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46698/0 (socket says 172.21.15.89:46698) 2024-04-02T07:52:57.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.726+0000 7f0905600700 1 -- 172.21.15.89:0/3954789107 learned_addr learned my addr 172.21.15.89:0/3954789107 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:57.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.726+0000 7f0906000700 1 --2- 172.21.15.89:0/3954789107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09000cd020 0x7f09001576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:57.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.726+0000 7f0904c00700 1 --2- 172.21.15.89:0/3954789107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09000cbb30 0x7f09001f6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:57.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.726+0000 7f0905600700 1 -- 172.21.15.89:0/3954789107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09000cd020 msgr2=0x7f09001576c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:57.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.726+0000 7f0905600700 1 --2- 172.21.15.89:0/3954789107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09000cd020 0x7f09001576c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.726+0000 7f0905600700 1 -- 172.21.15.89:0/3954789107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09000cbb30 msgr2=0x7f09001f6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:57.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.726+0000 7f0905600700 1 --2- 172.21.15.89:0/3954789107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09000cbb30 0x7f09001f6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:57.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.726+0000 7f0905600700 1 -- 172.21.15.89:0/3954789107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f8009a70 con 0x7f09000b2070 2024-04-02T07:52:57.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.726+0000 7f0905600700 1 --2- 172.21.15.89:0/3954789107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09000b2070 0x7f0900156ed0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f08f40011e0 tx=0x7f08f400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:57.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.727+0000 7f08fd600700 1 -- 172.21.15.89:0/3954789107 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4004c80 con 0x7f09000b2070 2024-04-02T07:52:57.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.727+0000 7f08fd600700 1 -- 172.21.15.89:0/3954789107 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f4004e00 con 0x7f09000b2070 2024-04-02T07:52:57.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.727+0000 7f08fd600700 1 -- 172.21.15.89:0/3954789107 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4007310 con 0x7f09000b2070 2024-04-02T07:52:57.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.727+0000 7f0908000700 1 -- 172.21.15.89:0/3954789107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f090015dbe0 con 0x7f09000b2070 2024-04-02T07:52:57.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.727+0000 7f0908000700 1 -- 172.21.15.89:0/3954789107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09001566f0 con 0x7f09000b2070 2024-04-02T07:52:57.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.728+0000 7f0908000700 1 -- 172.21.15.89:0/3954789107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0900156d40 con 0x7f09000b2070 2024-04-02T07:52:57.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.729+0000 7f08fd600700 1 -- 172.21.15.89:0/3954789107 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 52) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f08f4007470 con 0x7f09000b2070 2024-04-02T07:52:57.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.730+0000 7f08fd600700 1 --2- 172.21.15.89:0/3954789107 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f08e006adb0 0x7f08e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:57.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.730+0000 7f08fd600700 1 -- 172.21.15.89:0/3954789107 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f08f4089230 con 0x7f09000b2070 2024-04-02T07:52:57.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.730+0000 7f0904c00700 1 --2- 172.21.15.89:0/3954789107 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f08e006adb0 0x7f08e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:57.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.731+0000 7f0904c00700 1 --2- 172.21.15.89:0/3954789107 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f08e006adb0 0x7f08e006d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f08ec006b20 tx=0x7f08ec008040 comp rx=0 tx=0).ready entity=mgr.6775 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:57.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.734+0000 7f08fd600700 1 -- 172.21.15.89:0/3954789107 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f08f40541a0 con 0x7f09000b2070 2024-04-02T07:52:57.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:57.994+0000 7f0908000700 1 -- 172.21.15.89:0/3954789107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f090015bd50 con 0x7f09000b2070 2024-04-02T07:52:59.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.005+0000 7f08fd600700 1 -- 172.21.15.89:0/3954789107 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 53) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f08f40875f0 con 0x7f09000b2070 2024-04-02T07:52:59.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.006+0000 7f08fd600700 1 -- 172.21.15.89:0/3954789107 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v53) v1 ==== 92+0+0 (secure 0 0 0) 0x7f08f4073d60 con 0x7f09000b2070 2024-04-02T07:52:59.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.009+0000 7f0908000700 1 -- 172.21.15.89:0/3954789107 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f08e006adb0 msgr2=0x7f08e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:59.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.009+0000 7f0908000700 1 --2- 172.21.15.89:0/3954789107 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f08e006adb0 0x7f08e006d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f08ec006b20 tx=0x7f08ec008040 comp rx=0 tx=0).stop 2024-04-02T07:52:59.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.009+0000 7f0908000700 1 -- 172.21.15.89:0/3954789107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09000b2070 msgr2=0x7f0900156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:59.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.009+0000 7f0908000700 1 --2- 172.21.15.89:0/3954789107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09000b2070 0x7f0900156ed0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f08f40011e0 tx=0x7f08f400ca30 comp rx=0 tx=0).stop 2024-04-02T07:52:59.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.009+0000 7f0908000700 1 -- 172.21.15.89:0/3954789107 shutdown_connections 2024-04-02T07:52:59.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.009+0000 7f0908000700 1 --2- 172.21.15.89:0/3954789107 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f08e006adb0 0x7f08e006d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.009+0000 7f0908000700 1 --2- 172.21.15.89:0/3954789107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09000cd020 0x7f09001576c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.009+0000 7f0908000700 1 --2- 172.21.15.89:0/3954789107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09000b2070 0x7f0900156ed0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.009+0000 7f0908000700 1 --2- 172.21.15.89:0/3954789107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09000cbb30 0x7f09001f6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.009+0000 7f0908000700 1 -- 172.21.15.89:0/3954789107 >> 172.21.15.89:0/3954789107 conn(0x7f09000b0d70 msgr2=0x7f09000cbfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:59.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.010+0000 7f0908000700 1 -- 172.21.15.89:0/3954789107 shutdown_connections 2024-04-02T07:52:59.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.010+0000 7f0908000700 1 -- 172.21.15.89:0/3954789107 wait complete. 2024-04-02T07:52:59.033 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T07:52:59.221 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:59.219+0000 7f31a1ed2240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:52:59.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.237+0000 7f5375c00700 1 Processor -- start 2024-04-02T07:52:59.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.237+0000 7f5375c00700 1 -- start start 2024-04-02T07:52:59.239 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:59.238+0000 7fb8bf273240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:52:59.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.238+0000 7f5375c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53700d16a0 0x7f53700d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:59.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.238+0000 7f5375c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f53700d2370 0x7f53700ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:59.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.238+0000 7f5375c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5370161420 con 0x7f537015d9d0 2024-04-02T07:52:59.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.238+0000 7f5375c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53700d2820 con 0x7f53700d2370 2024-04-02T07:52:59.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.239+0000 7f5375c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53700d4340 con 0x7f53700d16a0 2024-04-02T07:52:59.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.239+0000 7f536ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f537015d9d0 0x7f53700d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42422/0 (socket says 172.21.15.89:42422) 2024-04-02T07:52:59.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.239+0000 7f536ea00700 1 -- 172.21.15.89:0/233118243 learned_addr learned my addr 172.21.15.89:0/233118243 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:52:59.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.239+0000 7f5367e00700 1 --2- 172.21.15.89:0/233118243 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53700d16a0 0x7f53700d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:59.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.239+0000 7f536f400700 1 --2- 172.21.15.89:0/233118243 >> v2:172.21.15.113:3300/0 conn(0x7f53700d2370 0x7f53700ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:59.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.239+0000 7f536e000700 1 -- 172.21.15.89:0/233118243 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110137050 0 0) 0x7f5370161420 con 0x7f537015d9d0 2024-04-02T07:52:59.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.239+0000 7f536e000700 1 -- 172.21.15.89:0/233118243 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53540035b0 con 0x7f537015d9d0 2024-04-02T07:52:59.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.239+0000 7f5367e00700 1 -- 172.21.15.89:0/233118243 >> v2:172.21.15.113:3300/0 conn(0x7f53700d2370 msgr2=0x7f53700ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:59.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.239+0000 7f5367e00700 1 --2- 172.21.15.89:0/233118243 >> v2:172.21.15.113:3300/0 conn(0x7f53700d2370 0x7f53700ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.239+0000 7f5367e00700 1 -- 172.21.15.89:0/233118243 >> v1:172.21.15.89:6789/0 conn(0x7f537015d9d0 legacy=0x7f53700d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:59.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.239+0000 7f5367e00700 1 -- 172.21.15.89:0/233118243 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5370161700 con 0x7f53700d16a0 2024-04-02T07:52:59.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.240+0000 7f5367e00700 1 --2- 172.21.15.89:0/233118243 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53700d16a0 0x7f53700d1e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5358009820 tx=0x7f5358009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=798c03b6c8617cd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:59.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.240+0000 7f536e000700 1 -- 172.21.15.89:0/233118243 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358005e30 con 0x7f53700d16a0 2024-04-02T07:52:59.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.240+0000 7f536e000700 1 -- 172.21.15.89:0/233118243 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5358006030 con 0x7f53700d16a0 2024-04-02T07:52:59.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.240+0000 7f536e000700 1 -- 172.21.15.89:0/233118243 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53580063c0 con 0x7f53700d16a0 2024-04-02T07:52:59.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.240+0000 7f5375c00700 1 -- 172.21.15.89:0/233118243 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53700d16a0 msgr2=0x7f53700d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:59.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.240+0000 7f5375c00700 1 --2- 172.21.15.89:0/233118243 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53700d16a0 0x7f53700d1e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5358009820 tx=0x7f5358009b30 comp rx=0 tx=0).stop 2024-04-02T07:52:59.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.241+0000 7f5375c00700 1 -- 172.21.15.89:0/233118243 shutdown_connections 2024-04-02T07:52:59.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.241+0000 7f5375c00700 1 --2- 172.21.15.89:0/233118243 >> v2:172.21.15.113:3300/0 conn(0x7f53700d2370 0x7f53700ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.241+0000 7f5375c00700 1 --2- 172.21.15.89:0/233118243 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53700d16a0 0x7f53700d1e30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.241+0000 7f5375c00700 1 -- 172.21.15.89:0/233118243 >> 172.21.15.89:0/233118243 conn(0x7f537008aff0 msgr2=0x7f53700d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:59.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.241+0000 7f5375c00700 1 -- 172.21.15.89:0/233118243 shutdown_connections 2024-04-02T07:52:59.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.241+0000 7f5375c00700 1 -- 172.21.15.89:0/233118243 wait complete. 2024-04-02T07:52:59.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.242+0000 7f5375c00700 1 Processor -- start 2024-04-02T07:52:59.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.243+0000 7f5375c00700 1 -- start start 2024-04-02T07:52:59.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.243+0000 7f5375c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53700d16a0 0x7f53701f0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:59.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.243+0000 7f5375c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53700d2370 0x7f53700b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:59.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.243+0000 7f5375c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f537015d9d0 0x7f53700b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:59.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.243+0000 7f5375c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53701f3ec0 con 0x7f53700d16a0 2024-04-02T07:52:59.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.243+0000 7f5375c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53700cd510 con 0x7f53700d2370 2024-04-02T07:52:59.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.243+0000 7f5375c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53701f3340 con 0x7f537015d9d0 2024-04-02T07:52:59.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.244+0000 7f536ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53700d16a0 0x7f53701f0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:59.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.244+0000 7f536ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53700d16a0 0x7f53701f0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38228/0 (socket says 172.21.15.89:38228) 2024-04-02T07:52:59.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.244+0000 7f536ea00700 1 -- 172.21.15.89:0/3436713446 learned_addr learned my addr 172.21.15.89:0/3436713446 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:59.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.244+0000 7f5367e00700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53700d2370 0x7f53700b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:59.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.244+0000 7f536f400700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f537015d9d0 0x7f53700b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:59.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.244+0000 7f536f400700 1 -- 172.21.15.89:0/3436713446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53700d2370 msgr2=0x7f53700b8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:59.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.244+0000 7f536f400700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53700d2370 0x7f53700b8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.244+0000 7f536f400700 1 -- 172.21.15.89:0/3436713446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53700d16a0 msgr2=0x7f53701f0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:59.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.244+0000 7f536f400700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53700d16a0 0x7f53701f0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.244+0000 7f536f400700 1 -- 172.21.15.89:0/3436713446 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5358009430 con 0x7f537015d9d0 2024-04-02T07:52:59.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.244+0000 7f536ea00700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53700d16a0 0x7f53701f0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:52:59.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.245+0000 7f536f400700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f537015d9d0 0x7f53700b48e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f536800efc0 tx=0x7f536800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:59.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.245+0000 7f5366a00700 1 -- 172.21.15.89:0/3436713446 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536800cc90 con 0x7f537015d9d0 2024-04-02T07:52:59.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.245+0000 7f5366a00700 1 -- 172.21.15.89:0/3436713446 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536800f040 con 0x7f537015d9d0 2024-04-02T07:52:59.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.245+0000 7f5375c00700 1 -- 172.21.15.89:0/3436713446 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53701f3620 con 0x7f537015d9d0 2024-04-02T07:52:59.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.245+0000 7f5366a00700 1 -- 172.21.15.89:0/3436713446 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53680146c0 con 0x7f537015d9d0 2024-04-02T07:52:59.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.245+0000 7f5375c00700 1 -- 172.21.15.89:0/3436713446 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53701f3be0 con 0x7f537015d9d0 2024-04-02T07:52:59.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.246+0000 7f5375c00700 1 -- 172.21.15.89:0/3436713446 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53700d1ca0 con 0x7f537015d9d0 2024-04-02T07:52:59.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.247+0000 7f5366a00700 1 -- 172.21.15.89:0/3436713446 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 53) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f5368014820 con 0x7f537015d9d0 2024-04-02T07:52:59.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.248+0000 7f5366a00700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f534c06ae00 0x7f534c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:59.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.248+0000 7f536ea00700 1 -- 172.21.15.89:0/3436713446 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f534c06ae00 msgr2=0x7f534c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:59.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.248+0000 7f5366a00700 1 -- 172.21.15.89:0/3436713446 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f536808a240 con 0x7f537015d9d0 2024-04-02T07:52:59.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.248+0000 7f536ea00700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f534c06ae00 0x7f534c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:59.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.257+0000 7f5366a00700 1 -- 172.21.15.89:0/3436713446 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5368055090 con 0x7f537015d9d0 2024-04-02T07:52:59.414 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:59.410+0000 7f31a1ed2240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:52:59.431 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:59.430+0000 7fb8bf273240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:52:59.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.452+0000 7f536ea00700 1 -- 172.21.15.89:0/3436713446 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f534c06ae00 msgr2=0x7f534c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:59.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.452+0000 7f536ea00700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f534c06ae00 0x7f534c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:59.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.504+0000 7f5375c00700 1 -- 172.21.15.89:0/3436713446 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f537015de40 con 0x7f537015d9d0 2024-04-02T07:52:59.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.512+0000 7f5366a00700 1 -- 172.21.15.89:0/3436713446 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7f537015de40 con 0x7f537015d9d0 2024-04-02T07:52:59.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.515+0000 7f5375c00700 1 -- 172.21.15.89:0/3436713446 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f534c06ae00 msgr2=0x7f534c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:59.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.515+0000 7f5375c00700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f534c06ae00 0x7f534c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.515+0000 7f5375c00700 1 -- 172.21.15.89:0/3436713446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f537015d9d0 msgr2=0x7f53700b48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:59.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.515+0000 7f5375c00700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f537015d9d0 0x7f53700b48e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f536800efc0 tx=0x7f536800c3a0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.515+0000 7f5375c00700 1 -- 172.21.15.89:0/3436713446 shutdown_connections 2024-04-02T07:52:59.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.515+0000 7f5375c00700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f534c06ae00 0x7f534c06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.515+0000 7f5375c00700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f537015d9d0 0x7f53700b48e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.515+0000 7f5375c00700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53700d16a0 0x7f53701f0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.515+0000 7f5375c00700 1 --2- 172.21.15.89:0/3436713446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53700d2370 0x7f53700b8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.515+0000 7f5375c00700 1 -- 172.21.15.89:0/3436713446 >> 172.21.15.89:0/3436713446 conn(0x7f537008aff0 msgr2=0x7f537015b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:59.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.515+0000 7f5375c00700 1 -- 172.21.15.89:0/3436713446 shutdown_connections 2024-04-02T07:52:59.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.515+0000 7f5375c00700 1 -- 172.21.15.89:0/3436713446 wait complete. 2024-04-02T07:52:59.535 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T07:52:59.732 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:52:59.731+0000 7f31a1ed2240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:52:59.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.733+0000 7febeb800700 1 Processor -- start 2024-04-02T07:52:59.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.733+0000 7febeb800700 1 -- start start 2024-04-02T07:52:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.734+0000 7febeb800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febe415d9d0 0x7febe40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.734+0000 7febeb800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7febe40d2540 0x7febe40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.734+0000 7febeb800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe4161420 con 0x7febe40d16a0 2024-04-02T07:52:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.734+0000 7febeb800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe415edc0 con 0x7febe40d2540 2024-04-02T07:52:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.734+0000 7febeb800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe4161700 con 0x7febe415d9d0 2024-04-02T07:52:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.735+0000 7febe9800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7febe40d2540 0x7febe40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.735+0000 7febe9800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7febe40d2540 0x7febe40ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38242/0 (socket says 172.21.15.89:38242) 2024-04-02T07:52:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.735+0000 7febe9800700 1 -- 172.21.15.89:0/4100094117 learned_addr learned my addr 172.21.15.89:0/4100094117 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.735+0000 7febe8e00700 1 --2- 172.21.15.89:0/4100094117 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febe415d9d0 0x7febe40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.735+0000 7febe9800700 1 -- 172.21.15.89:0/4100094117 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febe415d9d0 msgr2=0x7febe40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:52:59.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.735+0000 7febe9800700 1 --2- 172.21.15.89:0/4100094117 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febe415d9d0 0x7febe40d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.735+0000 7febe9800700 1 -- 172.21.15.89:0/4100094117 >> v1:172.21.15.113:6789/0 conn(0x7febe40d16a0 legacy=0x7febe40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:59.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.735+0000 7febe9800700 1 -- 172.21.15.89:0/4100094117 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe41619e0 con 0x7febe40d2540 2024-04-02T07:52:59.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.735+0000 7febe9800700 1 --2- 172.21.15.89:0/4100094117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febe40d2540 0x7febe40ce360 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7febdc009e80 tx=0x7febdc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d63b718a5852870 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:59.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.735+0000 7febe3400700 1 -- 172.21.15.89:0/4100094117 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc00e070 con 0x7febe40d2540 2024-04-02T07:52:59.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.736+0000 7febe3400700 1 -- 172.21.15.89:0/4100094117 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febdc00b8d0 con 0x7febe40d2540 2024-04-02T07:52:59.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.736+0000 7febe3400700 1 -- 172.21.15.89:0/4100094117 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc00bc80 con 0x7febe40d2540 2024-04-02T07:52:59.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.736+0000 7febeb800700 1 -- 172.21.15.89:0/4100094117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febe40d2540 msgr2=0x7febe40ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:59.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.736+0000 7febeb800700 1 --2- 172.21.15.89:0/4100094117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febe40d2540 0x7febe40ce360 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7febdc009e80 tx=0x7febdc00b330 comp rx=0 tx=0).stop 2024-04-02T07:52:59.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.736+0000 7febeb800700 1 -- 172.21.15.89:0/4100094117 shutdown_connections 2024-04-02T07:52:59.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.736+0000 7febeb800700 1 --2- 172.21.15.89:0/4100094117 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febe415d9d0 0x7febe40d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.736+0000 7febeb800700 1 --2- 172.21.15.89:0/4100094117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febe40d2540 0x7febe40ce360 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.736+0000 7febeb800700 1 -- 172.21.15.89:0/4100094117 >> 172.21.15.89:0/4100094117 conn(0x7febe408aff0 msgr2=0x7febe40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:59.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.736+0000 7febeb800700 1 -- 172.21.15.89:0/4100094117 shutdown_connections 2024-04-02T07:52:59.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.736+0000 7febeb800700 1 -- 172.21.15.89:0/4100094117 wait complete. 2024-04-02T07:52:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.738+0000 7febeb800700 1 Processor -- start 2024-04-02T07:52:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.738+0000 7febeb800700 1 -- start start 2024-04-02T07:52:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.739+0000 7febeb800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febe40d16a0 0x7febe40b7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.739+0000 7febeb800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febe40d2540 0x7febe41f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.739+0000 7febeb800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febe415d9d0 0x7febe415caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.739+0000 7febeb800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe40b7010 con 0x7febe40d16a0 2024-04-02T07:52:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.739+0000 7febeb800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe40b7970 con 0x7febe415d9d0 2024-04-02T07:52:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.739+0000 7febeb800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe41f44b0 con 0x7febe40d2540 2024-04-02T07:52:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.739+0000 7febe8e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febe40d16a0 0x7febe40b7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.739+0000 7febe8e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febe40d16a0 0x7febe40b7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38252/0 (socket says 172.21.15.89:38252) 2024-04-02T07:52:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.739+0000 7febe8e00700 1 -- 172.21.15.89:0/3805584409 learned_addr learned my addr 172.21.15.89:0/3805584409 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:52:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.740+0000 7febe3e00700 1 --2- 172.21.15.89:0/3805584409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febe40d2540 0x7febe41f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.740+0000 7febe9800700 1 --2- 172.21.15.89:0/3805584409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febe415d9d0 0x7febe415caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:52:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.740+0000 7febe8e00700 1 -- 172.21.15.89:0/3805584409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febe40d2540 msgr2=0x7febe41f3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.740+0000 7febe8e00700 1 --2- 172.21.15.89:0/3805584409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febe40d2540 0x7febe41f3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.740+0000 7febe8e00700 1 -- 172.21.15.89:0/3805584409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febe415d9d0 msgr2=0x7febe415caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.740+0000 7febe8e00700 1 --2- 172.21.15.89:0/3805584409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febe415d9d0 0x7febe415caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.740+0000 7febe8e00700 1 -- 172.21.15.89:0/3805584409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc009b30 con 0x7febe40d16a0 2024-04-02T07:52:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.740+0000 7febe8e00700 1 --2- 172.21.15.89:0/3805584409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febe40d16a0 0x7febe40b7430 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7febd800e870 tx=0x7febd800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:52:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.740+0000 7febe0c00700 1 -- 172.21.15.89:0/3805584409 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd8009590 con 0x7febe40d16a0 2024-04-02T07:52:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.740+0000 7febe0c00700 1 -- 172.21.15.89:0/3805584409 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd8013680 con 0x7febe40d16a0 2024-04-02T07:52:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.740+0000 7febe0c00700 1 -- 172.21.15.89:0/3805584409 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd80139f0 con 0x7febe40d16a0 2024-04-02T07:52:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.740+0000 7febeb800700 1 -- 172.21.15.89:0/3805584409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe40b6390 con 0x7febe40d16a0 2024-04-02T07:52:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.741+0000 7febeb800700 1 -- 172.21.15.89:0/3805584409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe40b6950 con 0x7febe40d16a0 2024-04-02T07:52:59.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.742+0000 7febeb800700 1 -- 172.21.15.89:0/3805584409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe40d1ca0 con 0x7febe40d16a0 2024-04-02T07:52:59.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.743+0000 7febe0c00700 1 -- 172.21.15.89:0/3805584409 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 53) v1 ==== 89256+0+0 (secure 0 0 0) 0x7febd8013b50 con 0x7febe40d16a0 2024-04-02T07:52:59.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.743+0000 7febe0c00700 1 --2- 172.21.15.89:0/3805584409 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febc406ae00 0x7febc406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:52:59.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.744+0000 7febe0c00700 1 -- 172.21.15.89:0/3805584409 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7febd80895a0 con 0x7febe40d16a0 2024-04-02T07:52:59.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.744+0000 7febe3e00700 1 -- 172.21.15.89:0/3805584409 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febc406ae00 msgr2=0x7febc406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:59.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.744+0000 7febe3e00700 1 --2- 172.21.15.89:0/3805584409 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febc406ae00 0x7febc406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:52:59.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.748+0000 7febe0c00700 1 -- 172.21.15.89:0/3805584409 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7febd8054490 con 0x7febe40d16a0 2024-04-02T07:52:59.769 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:52:59.767+0000 7fb8bf273240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:52:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.945+0000 7febe3e00700 1 -- 172.21.15.89:0/3805584409 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febc406ae00 msgr2=0x7febc406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:52:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.945+0000 7febe3e00700 1 --2- 172.21.15.89:0/3805584409 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febc406ae00 0x7febc406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:52:59.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.959+0000 7febeb800700 1 -- 172.21.15.89:0/3805584409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7febe4067720 con 0x7febe40d16a0 2024-04-02T07:52:59.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.965+0000 7febe0c00700 1 -- 172.21.15.89:0/3805584409 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7febd8057ab0 con 0x7febe40d16a0 2024-04-02T07:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.967+0000 7febeb800700 1 -- 172.21.15.89:0/3805584409 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febc406ae00 msgr2=0x7febc406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.967+0000 7febeb800700 1 --2- 172.21.15.89:0/3805584409 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febc406ae00 0x7febc406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.967+0000 7febeb800700 1 -- 172.21.15.89:0/3805584409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febe40d16a0 msgr2=0x7febe40b7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.967+0000 7febeb800700 1 --2- 172.21.15.89:0/3805584409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febe40d16a0 0x7febe40b7430 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7febd800e870 tx=0x7febd800eb80 comp rx=0 tx=0).stop 2024-04-02T07:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.967+0000 7febeb800700 1 -- 172.21.15.89:0/3805584409 shutdown_connections 2024-04-02T07:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.967+0000 7febeb800700 1 --2- 172.21.15.89:0/3805584409 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7febc406ae00 0x7febc406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.967+0000 7febeb800700 1 --2- 172.21.15.89:0/3805584409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febe415d9d0 0x7febe415caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.967+0000 7febeb800700 1 --2- 172.21.15.89:0/3805584409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febe40d16a0 0x7febe40b7430 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.967+0000 7febeb800700 1 --2- 172.21.15.89:0/3805584409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7febe40d2540 0x7febe41f3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.967+0000 7febeb800700 1 -- 172.21.15.89:0/3805584409 >> 172.21.15.89:0/3805584409 conn(0x7febe408aff0 msgr2=0x7febe415f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.968+0000 7febeb800700 1 -- 172.21.15.89:0/3805584409 shutdown_connections 2024-04-02T07:52:59.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:52:59.968+0000 7febeb800700 1 -- 172.21.15.89:0/3805584409 wait complete. 2024-04-02T07:52:59.987 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2024-04-02T07:53:00.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.187+0000 7fac28600700 1 Processor -- start 2024-04-02T07:53:00.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.187+0000 7fac28600700 1 -- start start 2024-04-02T07:53:00.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.188+0000 7fac28600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac200cd480 0x7fac2006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:53:00.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.188+0000 7fac28600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac200cdae0 con 0x7fac200cd020 2024-04-02T07:53:00.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.188+0000 7fac28600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac200cddc0 con 0x7fac200cbb30 2024-04-02T07:53:00.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.188+0000 7fac28600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac200ce0a0 con 0x7fac200cd480 2024-04-02T07:53:00.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.188+0000 7fac25200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fac200cd020 0x7fac200b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42434/0 (socket says 172.21.15.89:42434) 2024-04-02T07:53:00.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.188+0000 7fac25200700 1 -- 172.21.15.89:0/4177398119 learned_addr learned my addr 172.21.15.89:0/4177398119 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:53:00.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.188+0000 7fac26600700 1 --2- 172.21.15.89:0/4177398119 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac200cd480 0x7fac2006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:53:00.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.189+0000 7fac1fe00700 1 -- 172.21.15.89:0/4177398119 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664150237 0 0) 0x7fac200cdae0 con 0x7fac200cd020 2024-04-02T07:53:00.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.189+0000 7fac1fe00700 1 -- 172.21.15.89:0/4177398119 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac080035b0 con 0x7fac200cd020 2024-04-02T07:53:00.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.189+0000 7fac1fe00700 1 -- 172.21.15.89:0/4177398119 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287404476 0 0) 0x7fac200cddc0 con 0x7fac200cbb30 2024-04-02T07:53:00.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.189+0000 7fac1fe00700 1 -- 172.21.15.89:0/4177398119 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac200cdae0 con 0x7fac200cbb30 2024-04-02T07:53:00.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.189+0000 7fac26600700 1 -- 172.21.15.89:0/4177398119 >> v1:172.21.15.113:6789/0 conn(0x7fac200cbb30 legacy=0x7fac200cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:53:00.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.189+0000 7fac26600700 1 -- 172.21.15.89:0/4177398119 >> v1:172.21.15.89:6789/0 conn(0x7fac200cd020 legacy=0x7fac200b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:53:00.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.189+0000 7fac26600700 1 -- 172.21.15.89:0/4177398119 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2006dc20 con 0x7fac200cd480 2024-04-02T07:53:00.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.189+0000 7fac26600700 1 --2- 172.21.15.89:0/4177398119 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac200cd480 0x7fac2006e420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fac18009dc0 tx=0x7fac1800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28615d5cfeec409a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:53:00.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.189+0000 7fac1fe00700 1 -- 172.21.15.89:0/4177398119 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1800e070 con 0x7fac200cd480 2024-04-02T07:53:00.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.189+0000 7fac1fe00700 1 -- 172.21.15.89:0/4177398119 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1800bb20 con 0x7fac200cd480 2024-04-02T07:53:00.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.189+0000 7fac1fe00700 1 -- 172.21.15.89:0/4177398119 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18012430 con 0x7fac200cd480 2024-04-02T07:53:00.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.190+0000 7fac28600700 1 -- 172.21.15.89:0/4177398119 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac200cd480 msgr2=0x7fac2006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:53:00.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.190+0000 7fac28600700 1 --2- 172.21.15.89:0/4177398119 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac200cd480 0x7fac2006e420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fac18009dc0 tx=0x7fac1800b330 comp rx=0 tx=0).stop 2024-04-02T07:53:00.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.190+0000 7fac28600700 1 -- 172.21.15.89:0/4177398119 shutdown_connections 2024-04-02T07:53:00.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.190+0000 7fac28600700 1 --2- 172.21.15.89:0/4177398119 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac200cd480 0x7fac2006e420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:00.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.190+0000 7fac28600700 1 -- 172.21.15.89:0/4177398119 >> 172.21.15.89:0/4177398119 conn(0x7fac200b0d70 msgr2=0x7fac200b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:53:00.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.190+0000 7fac28600700 1 -- 172.21.15.89:0/4177398119 shutdown_connections 2024-04-02T07:53:00.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.190+0000 7fac28600700 1 -- 172.21.15.89:0/4177398119 wait complete. 2024-04-02T07:53:00.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.192+0000 7fac28600700 1 Processor -- start 2024-04-02T07:53:00.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.192+0000 7fac28600700 1 -- start start 2024-04-02T07:53:00.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.193+0000 7fac28600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac200cbb30 0x7fac201444e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:53:00.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.193+0000 7fac28600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac200cd020 0x7fac201520e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:53:00.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.193+0000 7fac28600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac200cd480 0x7fac2015a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:53:00.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.193+0000 7fac28600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac20159620 con 0x7fac200cd020 2024-04-02T07:53:00.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.193+0000 7fac28600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac201f3290 con 0x7fac200cbb30 2024-04-02T07:53:00.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.193+0000 7fac28600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac20159870 con 0x7fac200cd480 2024-04-02T07:53:00.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.193+0000 7fac25200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac200cd020 0x7fac201520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:53:00.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.193+0000 7fac25200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac200cd020 0x7fac201520e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38260/0 (socket says 172.21.15.89:38260) 2024-04-02T07:53:00.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.193+0000 7fac25200700 1 -- 172.21.15.89:0/942559721 learned_addr learned my addr 172.21.15.89:0/942559721 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:53:00.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.193+0000 7fac26600700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac200cd480 0x7fac2015a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:53:00.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.193+0000 7fac25c00700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac200cbb30 0x7fac201444e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:53:00.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.194+0000 7fac25200700 1 -- 172.21.15.89:0/942559721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac200cd480 msgr2=0x7fac2015a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:53:00.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.194+0000 7fac25200700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac200cd480 0x7fac2015a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:00.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.194+0000 7fac25200700 1 -- 172.21.15.89:0/942559721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac200cbb30 msgr2=0x7fac201444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:53:00.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.194+0000 7fac25200700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac200cbb30 0x7fac201444e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:00.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.194+0000 7fac25200700 1 -- 172.21.15.89:0/942559721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac18009a70 con 0x7fac200cd020 2024-04-02T07:53:00.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.194+0000 7fac26600700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac200cd480 0x7fac2015a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:53:00.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.194+0000 7fac25c00700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac200cbb30 0x7fac201444e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:53:00.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.194+0000 7fac25200700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac200cd020 0x7fac201520e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fac0c00c700 tx=0x7fac0c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:53:00.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.194+0000 7fac1d600700 1 -- 172.21.15.89:0/942559721 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c005d70 con 0x7fac200cd020 2024-04-02T07:53:00.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.194+0000 7fac1d600700 1 -- 172.21.15.89:0/942559721 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0c0070d0 con 0x7fac200cd020 2024-04-02T07:53:00.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.194+0000 7fac1d600700 1 -- 172.21.15.89:0/942559721 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c007440 con 0x7fac200cd020 2024-04-02T07:53:00.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.194+0000 7fac28600700 1 -- 172.21.15.89:0/942559721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac20159b50 con 0x7fac200cd020 2024-04-02T07:53:00.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.195+0000 7fac28600700 1 -- 172.21.15.89:0/942559721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2015a110 con 0x7fac200cd020 2024-04-02T07:53:00.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.196+0000 7fac28600700 1 -- 172.21.15.89:0/942559721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac20067720 con 0x7fac200cd020 2024-04-02T07:53:00.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.197+0000 7fac1d600700 1 -- 172.21.15.89:0/942559721 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 53) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fac0c0075a0 con 0x7fac200cd020 2024-04-02T07:53:00.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.197+0000 7fac1d600700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 0x7fac0006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:53:00.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.198+0000 7fac1d600700 1 -- 172.21.15.89:0/942559721 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fac0c0889c0 con 0x7fac200cd020 2024-04-02T07:53:00.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.198+0000 7fac25c00700 1 -- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 msgr2=0x7fac0006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:53:00.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.198+0000 7fac25c00700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 0x7fac0006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:53:00.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.202+0000 7fac1d600700 1 -- 172.21.15.89:0/942559721 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac0c053940 con 0x7fac200cd020 2024-04-02T07:53:00.258 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:00.257+0000 7f31a1ed2240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:53:00.319 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:00.318+0000 7fb8bf273240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:53:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.399+0000 7fac25c00700 1 -- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 msgr2=0x7fac0006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:53:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.399+0000 7fac25c00700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 0x7fac0006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:53:00.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.434+0000 7fac28600700 1 -- 172.21.15.89:0/942559721 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fac2015bdd0 con 0x7fac0006ae00 2024-04-02T07:53:00.512 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:00.510+0000 7f31a1ed2240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:53:00.576 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:00.575+0000 7fb8bf273240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:53:00.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.800+0000 7fac25c00700 1 -- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 msgr2=0x7fac0006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:53:00.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:00.800+0000 7fac25c00700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 0x7fac0006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T07:53:00.964 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:00.962+0000 7f31a1ed2240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:53:01.033 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:01.032+0000 7fb8bf273240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:53:01.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:01.601+0000 7fac25c00700 1 -- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 msgr2=0x7fac0006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:53:01.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:01.601+0000 7fac25c00700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 0x7fac0006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T07:53:01.667 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:01.666+0000 7f31a1ed2240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:53:01.739 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:01.738+0000 7fb8bf273240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:53:01.759 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:01.757+0000 7f31a1ed2240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:53:01.830 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:01.829+0000 7fb8bf273240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:53:01.916 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:01.914+0000 7f31a1ed2240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:53:01.989 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:01.988+0000 7fb8bf273240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:53:02.127 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:02.126+0000 7f31a1ed2240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:53:02.202 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:02.201+0000 7fb8bf273240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:53:02.218 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:02.217+0000 7f31a1ed2240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:53:02.294 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:02.293+0000 7fb8bf273240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:53:02.317 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:02.315+0000 7f31a1ed2240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:53:02.392 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:02.391+0000 7fb8bf273240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:53:02.414 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:02.413+0000 7f31a1ed2240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:53:02.490 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:02.489+0000 7fb8bf273240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:53:02.511 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:02.510+0000 7f31a1ed2240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:53:02.589 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:02.588+0000 7fb8bf273240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:53:02.722 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:02.720+0000 7f31a1ed2240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:53:02.800 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:02.799+0000 7fb8bf273240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:53:02.976 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:02.974+0000 7f31a1ed2240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:53:03.056 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:03.055+0000 7fb8bf273240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:53:03.166 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:03.165+0000 7f31a1ed2240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:53:03.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:03.203+0000 7fac25c00700 1 -- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 msgr2=0x7fac0006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:53:03.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:03.203+0000 7fac25c00700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 0x7fac0006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T07:53:03.249 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:03.248+0000 7fb8bf273240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:53:03.257 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:03.256+0000 7f31a1ed2240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:53:03.342 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:03.341+0000 7fb8bf273240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:53:03.979 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:03.977+0000 7f31a1ed2240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:53:04.069 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:04.068+0000 7fb8bf273240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:53:04.188 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:04.186+0000 7f31a1ed2240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:53:04.281 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:04.280+0000 7fb8bf273240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:53:04.391 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:04.390+0000 7f31a1ed2240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:53:04.488 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:04.487+0000 7fb8bf273240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:53:04.864 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:04.863+0000 7f31a1ed2240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:53:04.964 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:04.963+0000 7fb8bf273240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:53:05.072 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:05.071+0000 7f31a1ed2240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:53:05.176 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:05.175+0000 7fb8bf273240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:53:05.506 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:05.505+0000 7f31a1ed2240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:53:05.615 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:53:05.614+0000 7f31a1ed2240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:53:05.616 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:05.615+0000 7fb8bf273240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:53:05.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:05.675+0000 7fac1d600700 1 -- 172.21.15.89:0/942559721 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 54) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fac0c053d10 con 0x7fac200cd020 2024-04-02T07:53:05.733 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:05.732+0000 7fb8bf273240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:53:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:05.742+0000 7fac1d600700 1 -- 172.21.15.89:0/942559721 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 55) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fac0c05f4b0 con 0x7fac200cd020 2024-04-02T07:53:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:05.743+0000 7fac1d600700 1 -- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 msgr2=0x7fac0006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:53:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:05.743+0000 7fac1d600700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 0x7fac0006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:05.921 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:05.920+0000 7fb827000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:53:05.921 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:05.920+0000 7fb827000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:53:05.921 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:05.920+0000 7fb827000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:53:05.921 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:05.920+0000 7fb827000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:53:05.921 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:05.920+0000 7fb827000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:53:05.921 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:05.920+0000 7fb82b600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:53:05.921 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:05.920+0000 7fb82b600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:53:05.922 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:05.920+0000 7fb82b600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:53:05.922 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:05.920+0000 7fb82b600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:53:05.922 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:53:05.920+0000 7fb82b600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:53:06.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.748+0000 7fac1d600700 1 -- 172.21.15.89:0/942559721 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mgrmap(e 56) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fac0c0545e0 con 0x7fac200cd020 2024-04-02T07:53:06.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.749+0000 7fac1d600700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 0x7fac0006d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:53:06.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.749+0000 7fac1d600700 1 -- 172.21.15.89:0/942559721 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fac000283c0 con 0x7fac0006ae00 2024-04-02T07:53:06.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.750+0000 7fac25c00700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 0x7fac0006d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:53:06.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.753+0000 7fac25c00700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 0x7fac0006d240 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fac14001220 tx=0x7fac14009040 comp rx=0 tx=0).ready entity=mgr.7596 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:53:06.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.825+0000 7fac1d600700 1 -- 172.21.15.89:0/942559721 <== mgr.7596 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fac000283c0 con 0x7fac0006ae00 2024-04-02T07:53:06.826 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path /bad-path 2024-04-02T07:53:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.827+0000 7fac28600700 1 -- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 msgr2=0x7fac0006d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:53:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.827+0000 7fac28600700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 0x7fac0006d240 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fac14001220 tx=0x7fac14009040 comp rx=0 tx=0).stop 2024-04-02T07:53:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.828+0000 7fac28600700 1 -- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac200cd020 msgr2=0x7fac201520e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:53:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.828+0000 7fac28600700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac200cd020 0x7fac201520e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fac0c00c700 tx=0x7fac0c00ca10 comp rx=0 tx=0).stop 2024-04-02T07:53:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.828+0000 7fac28600700 1 -- 172.21.15.89:0/942559721 shutdown_connections 2024-04-02T07:53:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.828+0000 7fac28600700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac0006ae00 0x7fac0006d240 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.828+0000 7fac28600700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac200cbb30 0x7fac201444e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.828+0000 7fac28600700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac200cd480 0x7fac2015a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.828+0000 7fac28600700 1 --2- 172.21.15.89:0/942559721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac200cd020 0x7fac201520e0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.828+0000 7fac28600700 1 -- 172.21.15.89:0/942559721 >> 172.21.15.89:0/942559721 conn(0x7fac200b0d70 msgr2=0x7fac200709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:53:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.828+0000 7fac28600700 1 -- 172.21.15.89:0/942559721 shutdown_connections 2024-04-02T07:53:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:06.828+0000 7fac28600700 1 -- 172.21.15.89:0/942559721 wait complete. 2024-04-02T07:53:06.848 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2024-04-02T07:53:06.848 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-04-02T07:53:07.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.040+0000 7fceb7e00700 1 Processor -- start 2024-04-02T07:53:07.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.041+0000 7fceb7e00700 1 -- start start 2024-04-02T07:53:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.042+0000 7fceb7e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fceb00cd020 0x7fceb00cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:53:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.042+0000 7fceb7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fceb00b2070 0x7fceb00b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:53:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.042+0000 7fceb7e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb00cdae0 con 0x7fceb00cbb30 2024-04-02T07:53:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.042+0000 7fceb7e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb00cddc0 con 0x7fceb00b2070 2024-04-02T07:53:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.042+0000 7fceb7e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb00ce0a0 con 0x7fceb00cd020 2024-04-02T07:53:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.042+0000 7fceb5e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fceb00b2070 0x7fceb00b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:53:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.042+0000 7fceb5e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fceb00b2070 0x7fceb00b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38394/0 (socket says 172.21.15.89:38394) 2024-04-02T07:53:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.042+0000 7fceb5e00700 1 -- 172.21.15.89:0/1232272416 learned_addr learned my addr 172.21.15.89:0/1232272416 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:53:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.042+0000 7fceb4a00700 1 --2- 172.21.15.89:0/1232272416 >> v2:172.21.15.113:3300/0 conn(0x7fceb00cd020 0x7fceb00cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:53:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.042+0000 7fceb5e00700 1 -- 172.21.15.89:0/1232272416 >> v2:172.21.15.113:3300/0 conn(0x7fceb00cd020 msgr2=0x7fceb00cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:53:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.043+0000 7fceb5e00700 1 --2- 172.21.15.89:0/1232272416 >> v2:172.21.15.113:3300/0 conn(0x7fceb00cd020 0x7fceb00cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.043+0000 7fceb5e00700 1 -- 172.21.15.89:0/1232272416 >> v1:172.21.15.113:6789/0 conn(0x7fceb00cbb30 legacy=0x7fceb00cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:53:07.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.043+0000 7fceb5e00700 1 -- 172.21.15.89:0/1232272416 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb006dc20 con 0x7fceb00b2070 2024-04-02T07:53:07.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.043+0000 7fceb4a00700 1 --2- 172.21.15.89:0/1232272416 >> v2:172.21.15.113:3300/0 conn(0x7fceb00cd020 0x7fceb00cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:53:07.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.043+0000 7fceb5e00700 1 --2- 172.21.15.89:0/1232272416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb00b2070 0x7fceb00b2500 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fceb00b1e30 tx=0x7fcea800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17cadb5a2ee311b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:53:07.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.043+0000 7fceafe00700 1 -- 172.21.15.89:0/1232272416 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea800e070 con 0x7fceb00b2070 2024-04-02T07:53:07.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.043+0000 7fceafe00700 1 -- 172.21.15.89:0/1232272416 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea8003a20 con 0x7fceb00b2070 2024-04-02T07:53:07.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.043+0000 7fceafe00700 1 -- 172.21.15.89:0/1232272416 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea8003dd0 con 0x7fceb00b2070 2024-04-02T07:53:07.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.044+0000 7fceb7e00700 1 -- 172.21.15.89:0/1232272416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb00b2070 msgr2=0x7fceb00b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:53:07.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.044+0000 7fceb7e00700 1 --2- 172.21.15.89:0/1232272416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb00b2070 0x7fceb00b2500 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fceb00b1e30 tx=0x7fcea800b330 comp rx=0 tx=0).stop 2024-04-02T07:53:07.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.044+0000 7fceb7e00700 1 -- 172.21.15.89:0/1232272416 shutdown_connections 2024-04-02T07:53:07.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.044+0000 7fceb7e00700 1 --2- 172.21.15.89:0/1232272416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb00b2070 0x7fceb00b2500 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:07.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.044+0000 7fceb7e00700 1 --2- 172.21.15.89:0/1232272416 >> v2:172.21.15.113:3300/0 conn(0x7fceb00cd020 0x7fceb00cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:07.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.044+0000 7fceb7e00700 1 -- 172.21.15.89:0/1232272416 >> 172.21.15.89:0/1232272416 conn(0x7fceb00b0d70 msgr2=0x7fceb00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:53:07.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.044+0000 7fceb7e00700 1 -- 172.21.15.89:0/1232272416 shutdown_connections 2024-04-02T07:53:07.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.044+0000 7fceb7e00700 1 -- 172.21.15.89:0/1232272416 wait complete. 2024-04-02T07:53:07.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.046+0000 7fceb7e00700 1 Processor -- start 2024-04-02T07:53:07.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.046+0000 7fceb7e00700 1 -- start start 2024-04-02T07:53:07.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.047+0000 7fceb7e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fceb00b2070 0x7fceb0072690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:53:07.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.047+0000 7fceb7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb00cbb30 0x7fceb00709c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:53:07.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.047+0000 7fceb7e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fceb00cd020 0x7fceb014da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:53:07.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.047+0000 7fceb7e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb0071e60 con 0x7fceb00cbb30 2024-04-02T07:53:07.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.047+0000 7fceb7e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb014dfa0 con 0x7fceb00cd020 2024-04-02T07:53:07.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.047+0000 7fceb7e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb0071270 con 0x7fceb00b2070 2024-04-02T07:53:07.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.048+0000 7fceb4a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb00cbb30 0x7fceb00709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:53:07.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.048+0000 7fceb4a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb00cbb30 0x7fceb00709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38400/0 (socket says 172.21.15.89:38400) 2024-04-02T07:53:07.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.048+0000 7fceb4a00700 1 -- 172.21.15.89:0/2888918506 learned_addr learned my addr 172.21.15.89:0/2888918506 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:53:07.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.048+0000 7fceb5e00700 1 --2- 172.21.15.89:0/2888918506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fceb00cd020 0x7fceb014da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:53:07.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.048+0000 7fceb5400700 1 --2- 172.21.15.89:0/2888918506 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fceb00b2070 0x7fceb0072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:53:07.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.048+0000 7fceb4a00700 1 -- 172.21.15.89:0/2888918506 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fceb00b2070 msgr2=0x7fceb0072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:53:07.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.048+0000 7fceb4a00700 1 --2- 172.21.15.89:0/2888918506 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fceb00b2070 0x7fceb0072690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:07.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.048+0000 7fceb4a00700 1 -- 172.21.15.89:0/2888918506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fceb00cd020 msgr2=0x7fceb014da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:53:07.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.048+0000 7fceb4a00700 1 --2- 172.21.15.89:0/2888918506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fceb00cd020 0x7fceb014da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:07.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.048+0000 7fceb4a00700 1 -- 172.21.15.89:0/2888918506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea8009b30 con 0x7fceb00cbb30 2024-04-02T07:53:07.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.048+0000 7fceb4a00700 1 --2- 172.21.15.89:0/2888918506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb00cbb30 0x7fceb00709c0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00c7f0 tx=0x7fce9c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:53:07.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.049+0000 7fcead600700 1 -- 172.21.15.89:0/2888918506 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c011800 con 0x7fceb00cbb30 2024-04-02T07:53:07.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.049+0000 7fcead600700 1 -- 172.21.15.89:0/2888918506 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c011960 con 0x7fceb00cbb30 2024-04-02T07:53:07.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.049+0000 7fceb7e00700 1 -- 172.21.15.89:0/2888918506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb0071530 con 0x7fceb00cbb30 2024-04-02T07:53:07.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.049+0000 7fcead600700 1 -- 172.21.15.89:0/2888918506 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c011cf0 con 0x7fceb00cbb30 2024-04-02T07:53:07.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.049+0000 7fceb7e00700 1 -- 172.21.15.89:0/2888918506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb0071af0 con 0x7fceb00cbb30 2024-04-02T07:53:07.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.050+0000 7fceb7e00700 1 -- 172.21.15.89:0/2888918506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb0072010 con 0x7fceb00cbb30 2024-04-02T07:53:07.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.051+0000 7fcead600700 1 -- 172.21.15.89:0/2888918506 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 56) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fce9c018360 con 0x7fceb00cbb30 2024-04-02T07:53:07.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.052+0000 7fcead600700 1 --2- 172.21.15.89:0/2888918506 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce9006ab30 0x7fce9006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:53:07.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.052+0000 7fcead600700 1 -- 172.21.15.89:0/2888918506 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fce9c089080 con 0x7fceb00cbb30 2024-04-02T07:53:07.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.052+0000 7fceb5400700 1 --2- 172.21.15.89:0/2888918506 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce9006ab30 0x7fce9006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:53:07.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.052+0000 7fceb5400700 1 --2- 172.21.15.89:0/2888918506 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce9006ab30 0x7fce9006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcea4007fd0 tx=0x7fcea4007ec0 comp rx=0 tx=0).ready entity=mgr.7596 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:53:07.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.057+0000 7fcead600700 1 -- 172.21.15.89:0/2888918506 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fce9c0542e0 con 0x7fceb00cbb30 2024-04-02T07:53:07.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.284+0000 7fceb7e00700 1 -- 172.21.15.89:0/2888918506 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fceb015bdd0 con 0x7fce9006ab30 2024-04-02T07:53:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.296+0000 7fcead600700 1 -- 172.21.15.89:0/2888918506 <== mgr.7596 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+302 (secure 0 0 0) 0x7fcea4003a90 con 0x7fce9006ab30 2024-04-02T07:53:07.298 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:53:07.298 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:53:06", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-04-02T07:53:07.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.298+0000 7fceb7e00700 1 -- 172.21.15.89:0/2888918506 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce9006ab30 msgr2=0x7fce9006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:53:07.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.298+0000 7fceb7e00700 1 --2- 172.21.15.89:0/2888918506 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce9006ab30 0x7fce9006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcea4007fd0 tx=0x7fcea4007ec0 comp rx=0 tx=0).stop 2024-04-02T07:53:07.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.298+0000 7fceb7e00700 1 -- 172.21.15.89:0/2888918506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb00cbb30 msgr2=0x7fceb00709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:53:07.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.298+0000 7fceb7e00700 1 --2- 172.21.15.89:0/2888918506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb00cbb30 0x7fceb00709c0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00c7f0 tx=0x7fce9c00cb00 comp rx=0 tx=0).stop 2024-04-02T07:53:07.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.299+0000 7fceb7e00700 1 -- 172.21.15.89:0/2888918506 shutdown_connections 2024-04-02T07:53:07.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.299+0000 7fceb7e00700 1 --2- 172.21.15.89:0/2888918506 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fce9006ab30 0x7fce9006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:07.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.299+0000 7fceb7e00700 1 --2- 172.21.15.89:0/2888918506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fceb00cd020 0x7fceb014da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:07.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.299+0000 7fceb7e00700 1 --2- 172.21.15.89:0/2888918506 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fceb00b2070 0x7fceb0072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:07.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.299+0000 7fceb7e00700 1 --2- 172.21.15.89:0/2888918506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fceb00cbb30 0x7fceb00709c0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:53:07.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.299+0000 7fceb7e00700 1 -- 172.21.15.89:0/2888918506 >> 172.21.15.89:0/2888918506 conn(0x7fceb00b0d70 msgr2=0x7fceb00cbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:53:07.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.299+0000 7fceb7e00700 1 -- 172.21.15.89:0/2888918506 shutdown_connections 2024-04-02T07:53:07.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:53:07.299+0000 7fceb7e00700 1 -- 172.21.15.89:0/2888918506 wait complete. 2024-04-02T07:53:07.318 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:53:06", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-04-02T07:53:16.649 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:53:16.655 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:53:46.762 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:53:46.767 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:54:00.841 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:00.839+0000 7fb816c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:00.841 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:00.839+0000 7fb816c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:00.841 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:00.839+0000 7fb816c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:00.841 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:00.839+0000 7fb816c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:00.841 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:00.839+0000 7fb816c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:07.319 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-04-02T07:54:07.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.519+0000 7fe9ef800700 1 Processor -- start 2024-04-02T07:54:07.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.519+0000 7fe9ef800700 1 -- start start 2024-04-02T07:54:07.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.520+0000 7fe9ef800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe9e815d800 0x7fe9e80725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:07.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.520+0000 7fe9ef800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe9e80ce6a0 0x7fe9e815dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:07.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.520+0000 7fe9ef800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e80709c0 con 0x7fe9e80ce230 2024-04-02T07:54:07.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.520+0000 7fe9ef800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e80b40f0 con 0x7fe9e815d800 2024-04-02T07:54:07.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.520+0000 7fe9ef800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e8070c80 con 0x7fe9e80ce6a0 2024-04-02T07:54:07.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.520+0000 7fe9ece00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe9e815d800 0x7fe9e80725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:07.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.520+0000 7fe9ece00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe9e815d800 0x7fe9e80725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52706/0 (socket says 172.21.15.89:52706) 2024-04-02T07:54:07.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.520+0000 7fe9ece00700 1 -- 172.21.15.89:0/1784981218 learned_addr learned my addr 172.21.15.89:0/1784981218 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:07.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.521+0000 7fe9ed800700 1 --2- 172.21.15.89:0/1784981218 >> v2:172.21.15.113:3300/0 conn(0x7fe9e80ce6a0 0x7fe9e815dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:07.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.521+0000 7fe9ece00700 1 -- 172.21.15.89:0/1784981218 >> v2:172.21.15.113:3300/0 conn(0x7fe9e80ce6a0 msgr2=0x7fe9e815dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:07.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.521+0000 7fe9ece00700 1 --2- 172.21.15.89:0/1784981218 >> v2:172.21.15.113:3300/0 conn(0x7fe9e80ce6a0 0x7fe9e815dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.521+0000 7fe9ece00700 1 -- 172.21.15.89:0/1784981218 >> v1:172.21.15.113:6789/0 conn(0x7fe9e80ce230 legacy=0x7fe9e80b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:07.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.521+0000 7fe9ece00700 1 -- 172.21.15.89:0/1784981218 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e8070f60 con 0x7fe9e815d800 2024-04-02T07:54:07.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.521+0000 7fe9ed800700 1 --2- 172.21.15.89:0/1784981218 >> v2:172.21.15.113:3300/0 conn(0x7fe9e80ce6a0 0x7fe9e815dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:07.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.521+0000 7fe9ece00700 1 --2- 172.21.15.89:0/1784981218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e815d800 0x7fe9e80725c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fe9e806b6d0 tx=0x7fe9dc00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43eef729eb8efac6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:07.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.521+0000 7fe9e7400700 1 -- 172.21.15.89:0/1784981218 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc00dcf0 con 0x7fe9e815d800 2024-04-02T07:54:07.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.521+0000 7fe9e7400700 1 -- 172.21.15.89:0/1784981218 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9dc00de50 con 0x7fe9e815d800 2024-04-02T07:54:07.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.521+0000 7fe9e7400700 1 -- 172.21.15.89:0/1784981218 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc004250 con 0x7fe9e815d800 2024-04-02T07:54:07.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.522+0000 7fe9ef800700 1 -- 172.21.15.89:0/1784981218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e815d800 msgr2=0x7fe9e80725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:07.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.522+0000 7fe9ef800700 1 --2- 172.21.15.89:0/1784981218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e815d800 0x7fe9e80725c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fe9e806b6d0 tx=0x7fe9dc00be70 comp rx=0 tx=0).stop 2024-04-02T07:54:07.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.522+0000 7fe9ef800700 1 -- 172.21.15.89:0/1784981218 shutdown_connections 2024-04-02T07:54:07.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.522+0000 7fe9ef800700 1 --2- 172.21.15.89:0/1784981218 >> v2:172.21.15.113:3300/0 conn(0x7fe9e80ce6a0 0x7fe9e815dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.522+0000 7fe9ef800700 1 --2- 172.21.15.89:0/1784981218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e815d800 0x7fe9e80725c0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.522+0000 7fe9ef800700 1 -- 172.21.15.89:0/1784981218 >> 172.21.15.89:0/1784981218 conn(0x7fe9e80b0d70 msgr2=0x7fe9e80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:07.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.522+0000 7fe9ef800700 1 -- 172.21.15.89:0/1784981218 shutdown_connections 2024-04-02T07:54:07.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.522+0000 7fe9ef800700 1 -- 172.21.15.89:0/1784981218 wait complete. 2024-04-02T07:54:07.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.524+0000 7fe9ef800700 1 Processor -- start 2024-04-02T07:54:07.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.524+0000 7fe9ef800700 1 -- start start 2024-04-02T07:54:07.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ef800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9e80ce230 0x7fe9e814c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:07.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ef800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9e80ce6a0 0x7fe9e8151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:07.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ef800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e815d800 0x7fe9e81e4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:07.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ef800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80ad790 con 0x7fe9e815d800 2024-04-02T07:54:07.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ef800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80ac1d0 con 0x7fe9e80ce6a0 2024-04-02T07:54:07.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ef800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80ac4b0 con 0x7fe9e80ce230 2024-04-02T07:54:07.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ed800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e815d800 0x7fe9e81e4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:07.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ed800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e815d800 0x7fe9e81e4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52708/0 (socket says 172.21.15.89:52708) 2024-04-02T07:54:07.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ed800700 1 -- 172.21.15.89:0/3876940093 learned_addr learned my addr 172.21.15.89:0/3876940093 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:07.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ece00700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9e80ce230 0x7fe9e814c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:07.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9e7e00700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9e80ce6a0 0x7fe9e8151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:07.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ed800700 1 -- 172.21.15.89:0/3876940093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9e80ce230 msgr2=0x7fe9e814c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:07.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ed800700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9e80ce230 0x7fe9e814c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ed800700 1 -- 172.21.15.89:0/3876940093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9e80ce6a0 msgr2=0x7fe9e8151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:07.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ed800700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9e80ce6a0 0x7fe9e8151b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9ed800700 1 -- 172.21.15.89:0/3876940093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc00b7f0 con 0x7fe9e815d800 2024-04-02T07:54:07.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.525+0000 7fe9e7e00700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9e80ce6a0 0x7fe9e8151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:07.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.526+0000 7fe9ed800700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e815d800 0x7fe9e81e4670 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fe9e000cef0 tx=0x7fe9e000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:07.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.526+0000 7fe9e4c00700 1 -- 172.21.15.89:0/3876940093 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e000adb0 con 0x7fe9e815d800 2024-04-02T07:54:07.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.526+0000 7fe9e4c00700 1 -- 172.21.15.89:0/3876940093 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e0015440 con 0x7fe9e815d800 2024-04-02T07:54:07.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.526+0000 7fe9e4c00700 1 -- 172.21.15.89:0/3876940093 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e00157f0 con 0x7fe9e815d800 2024-04-02T07:54:07.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.526+0000 7fe9ef800700 1 -- 172.21.15.89:0/3876940093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e80ac790 con 0x7fe9e815d800 2024-04-02T07:54:07.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.526+0000 7fe9ef800700 1 -- 172.21.15.89:0/3876940093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e80ad300 con 0x7fe9e815d800 2024-04-02T07:54:07.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.527+0000 7fe9ef800700 1 -- 172.21.15.89:0/3876940093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e80ac9b0 con 0x7fe9e815d800 2024-04-02T07:54:07.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.528+0000 7fe9e4c00700 1 -- 172.21.15.89:0/3876940093 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 59) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe9e000d430 con 0x7fe9e815d800 2024-04-02T07:54:07.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.529+0000 7fe9e4c00700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe9c806af20 0x7fe9c806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:07.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.529+0000 7fe9e4c00700 1 -- 172.21.15.89:0/3876940093 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fe9e008a080 con 0x7fe9e815d800 2024-04-02T07:54:07.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.529+0000 7fe9ece00700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe9c806af20 0x7fe9c806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:07.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.529+0000 7fe9ece00700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe9c806af20 0x7fe9c806d3e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00b340 tx=0x7fe9dc01d040 comp rx=0 tx=0).ready entity=mgr.7596 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:07.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.533+0000 7fe9e4c00700 1 -- 172.21.15.89:0/3876940093 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe9e00bd580 con 0x7fe9e815d800 2024-04-02T07:54:07.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.758+0000 7fe9ef800700 1 -- 172.21.15.89:0/3876940093 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe9e80ad580 con 0x7fe9c806af20 2024-04-02T07:54:07.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.766+0000 7fe9e4c00700 1 -- 172.21.15.89:0/3876940093 <== mgr.7596 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+303 (secure 0 0 0) 0x7fe9e80ad580 con 0x7fe9c806af20 2024-04-02T07:54:07.768 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:07.768 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:53:06", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-04-02T07:54:07.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.768+0000 7fe9ef800700 1 -- 172.21.15.89:0/3876940093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe9c806af20 msgr2=0x7fe9c806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:07.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.768+0000 7fe9ef800700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe9c806af20 0x7fe9c806d3e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00b340 tx=0x7fe9dc01d040 comp rx=0 tx=0).stop 2024-04-02T07:54:07.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.768+0000 7fe9ef800700 1 -- 172.21.15.89:0/3876940093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e815d800 msgr2=0x7fe9e81e4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:07.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.768+0000 7fe9ef800700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e815d800 0x7fe9e81e4670 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fe9e000cef0 tx=0x7fe9e000cf20 comp rx=0 tx=0).stop 2024-04-02T07:54:07.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.768+0000 7fe9ef800700 1 -- 172.21.15.89:0/3876940093 shutdown_connections 2024-04-02T07:54:07.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.768+0000 7fe9ef800700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe9c806af20 0x7fe9c806d3e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.768+0000 7fe9ef800700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e815d800 0x7fe9e81e4670 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.768+0000 7fe9ef800700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9e80ce230 0x7fe9e814c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.768+0000 7fe9ef800700 1 --2- 172.21.15.89:0/3876940093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9e80ce6a0 0x7fe9e8151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.768+0000 7fe9ef800700 1 -- 172.21.15.89:0/3876940093 >> 172.21.15.89:0/3876940093 conn(0x7fe9e80b0d70 msgr2=0x7fe9e80aecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:07.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.768+0000 7fe9ef800700 1 -- 172.21.15.89:0/3876940093 shutdown_connections 2024-04-02T07:54:07.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.768+0000 7fe9ef800700 1 -- 172.21.15.89:0/3876940093 wait complete. 2024-04-02T07:54:07.788 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:53:06", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-04-02T07:54:07.788 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2024-04-02T07:54:07.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.979+0000 7f058c400700 1 Processor -- start 2024-04-02T07:54:07.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.979+0000 7f058c400700 1 -- start start 2024-04-02T07:54:07.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.980+0000 7f058c400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f05841611b0 0x7f058415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:07.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.980+0000 7f058c400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f058415f4d0 0x7f05840ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:07.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.980+0000 7f058c400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05840d21f0 0x7f05840cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:07.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.980+0000 7f058c400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05840cf180 con 0x7f058415f4d0 2024-04-02T07:54:07.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.980+0000 7f058c400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05840ce270 con 0x7f05841611b0 2024-04-02T07:54:07.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.980+0000 7f058c400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058415b600 con 0x7f05840d21f0 2024-04-02T07:54:07.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.980+0000 7f0589000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f058415f4d0 0x7f05840ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:07.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.980+0000 7f0589000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f058415f4d0 0x7f05840ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52720/0 (socket says 172.21.15.89:52720) 2024-04-02T07:54:07.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.980+0000 7f0589000700 1 -- 172.21.15.89:0/192787763 learned_addr learned my addr 172.21.15.89:0/192787763 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:07.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.980+0000 7f0589a00700 1 --2- 172.21.15.89:0/192787763 >> v2:172.21.15.113:3300/0 conn(0x7f05841611b0 0x7f058415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:07.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.980+0000 7f0589000700 1 -- 172.21.15.89:0/192787763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05840d21f0 msgr2=0x7f05840cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:07.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.980+0000 7f058a400700 1 --2- 172.21.15.89:0/192787763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05840d21f0 0x7f05840cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:07.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.981+0000 7f0589000700 1 --2- 172.21.15.89:0/192787763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05840d21f0 0x7f05840cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.981+0000 7f0589000700 1 -- 172.21.15.89:0/192787763 >> v2:172.21.15.113:3300/0 conn(0x7f05841611b0 msgr2=0x7f058415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:07.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.981+0000 7f0589000700 1 --2- 172.21.15.89:0/192787763 >> v2:172.21.15.113:3300/0 conn(0x7f05841611b0 0x7f058415ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.981+0000 7f0589000700 1 -- 172.21.15.89:0/192787763 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05840d2760 con 0x7f058415f4d0 2024-04-02T07:54:07.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.981+0000 7f0589a00700 1 --2- 172.21.15.89:0/192787763 >> v2:172.21.15.113:3300/0 conn(0x7f05841611b0 0x7f058415ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:07.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.981+0000 7f058a400700 1 --2- 172.21.15.89:0/192787763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05840d21f0 0x7f05840cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:07.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.981+0000 7f0589000700 1 --2- 172.21.15.89:0/192787763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f058415f4d0 0x7f05840ce700 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f058408b520 tx=0x7f0570009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629f8d342762df7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:07.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.981+0000 7f0583e00700 1 -- 172.21.15.89:0/192787763 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057000b870 con 0x7f058415f4d0 2024-04-02T07:54:07.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.981+0000 7f0583e00700 1 -- 172.21.15.89:0/192787763 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057000b9d0 con 0x7f058415f4d0 2024-04-02T07:54:07.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.981+0000 7f0583e00700 1 -- 172.21.15.89:0/192787763 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057000bd60 con 0x7f058415f4d0 2024-04-02T07:54:07.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.982+0000 7f058c400700 1 -- 172.21.15.89:0/192787763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f058415f4d0 msgr2=0x7f05840ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:07.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.982+0000 7f058c400700 1 --2- 172.21.15.89:0/192787763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f058415f4d0 0x7f05840ce700 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f058408b520 tx=0x7f0570009bd0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.982+0000 7f058c400700 1 -- 172.21.15.89:0/192787763 shutdown_connections 2024-04-02T07:54:07.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.982+0000 7f058c400700 1 --2- 172.21.15.89:0/192787763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05840d21f0 0x7f05840cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.982+0000 7f058c400700 1 --2- 172.21.15.89:0/192787763 >> v2:172.21.15.113:3300/0 conn(0x7f05841611b0 0x7f058415ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.982+0000 7f058c400700 1 --2- 172.21.15.89:0/192787763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f058415f4d0 0x7f05840ce700 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.982+0000 7f058c400700 1 -- 172.21.15.89:0/192787763 >> 172.21.15.89:0/192787763 conn(0x7f058408aff0 msgr2=0x7f05840d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:07.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.982+0000 7f058c400700 1 -- 172.21.15.89:0/192787763 shutdown_connections 2024-04-02T07:54:07.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.982+0000 7f058c400700 1 -- 172.21.15.89:0/192787763 wait complete. 2024-04-02T07:54:07.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.983+0000 7f058c400700 1 Processor -- start 2024-04-02T07:54:07.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.984+0000 7f058c400700 1 -- start start 2024-04-02T07:54:07.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.984+0000 7f058c400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05840d21f0 0x7f05841ea700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:07.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.984+0000 7f058c400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f058415f4d0 0x7f05841e8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:07.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.984+0000 7f058c400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05841611b0 0x7f05840c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:07.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.984+0000 7f058c400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05841e9660 con 0x7f058415f4d0 2024-04-02T07:54:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.984+0000 7f058c400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05841e9940 con 0x7f05840d21f0 2024-04-02T07:54:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.984+0000 7f058c400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05841e9c20 con 0x7f05841611b0 2024-04-02T07:54:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.985+0000 7f0589000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f058415f4d0 0x7f05841e8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.985+0000 7f0589000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f058415f4d0 0x7f05841e8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52726/0 (socket says 172.21.15.89:52726) 2024-04-02T07:54:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.985+0000 7f0589000700 1 -- 172.21.15.89:0/426061185 learned_addr learned my addr 172.21.15.89:0/426061185 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.985+0000 7f0589a00700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05840d21f0 0x7f05841ea700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.985+0000 7f058a400700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05841611b0 0x7f05840c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.985+0000 7f0589000700 1 -- 172.21.15.89:0/426061185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05841611b0 msgr2=0x7f05840c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.985+0000 7f0589000700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05841611b0 0x7f05840c8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.985+0000 7f0589000700 1 -- 172.21.15.89:0/426061185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05840d21f0 msgr2=0x7f05841ea700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.985+0000 7f0589000700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05840d21f0 0x7f05841ea700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.985+0000 7f0589000700 1 -- 172.21.15.89:0/426061185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0570009550 con 0x7f058415f4d0 2024-04-02T07:54:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.985+0000 7f058a400700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05841611b0 0x7f05840c8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.985+0000 7f0589a00700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05840d21f0 0x7f05841ea700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:07.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.985+0000 7f0589000700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f058415f4d0 0x7f05841e8cf0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f0570013000 tx=0x7f0570004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:07.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.986+0000 7f0581600700 1 -- 172.21.15.89:0/426061185 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0570004310 con 0x7f058415f4d0 2024-04-02T07:54:07.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.986+0000 7f0581600700 1 -- 172.21.15.89:0/426061185 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057001f070 con 0x7f058415f4d0 2024-04-02T07:54:07.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.986+0000 7f058c400700 1 -- 172.21.15.89:0/426061185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05841e9f00 con 0x7f058415f4d0 2024-04-02T07:54:07.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.986+0000 7f0581600700 1 -- 172.21.15.89:0/426061185 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05700036a0 con 0x7f058415f4d0 2024-04-02T07:54:07.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.986+0000 7f058c400700 1 -- 172.21.15.89:0/426061185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05841eac40 con 0x7f058415f4d0 2024-04-02T07:54:07.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.987+0000 7f058c400700 1 -- 172.21.15.89:0/426061185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05841ea420 con 0x7f058415f4d0 2024-04-02T07:54:07.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.988+0000 7f0581600700 1 -- 172.21.15.89:0/426061185 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 59) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f0570021030 con 0x7f058415f4d0 2024-04-02T07:54:07.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.988+0000 7f0581600700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f056406af20 0x7f056406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:07.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.989+0000 7f0581600700 1 -- 172.21.15.89:0/426061185 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f057008af60 con 0x7f058415f4d0 2024-04-02T07:54:07.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.989+0000 7f0589a00700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f056406af20 0x7f056406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:07.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.989+0000 7f0589a00700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f056406af20 0x7f056406d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f057800aa20 tx=0x7f0578008040 comp rx=0 tx=0).ready entity=mgr.7596 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:07.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:07.993+0000 7f0581600700 1 -- 172.21.15.89:0/426061185 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f05841ea420 con 0x7f058415f4d0 2024-04-02T07:54:08.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.221+0000 7f058c400700 1 -- 172.21.15.89:0/426061185 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7f05840cdf00 con 0x7f056406af20 2024-04-02T07:54:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.305+0000 7f0581600700 1 -- 172.21.15.89:0/426061185 <== mgr.7596 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7f057800a290 con 0x7f056406af20 2024-04-02T07:54:08.307 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path /bad-path 2024-04-02T07:54:08.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.307+0000 7f058c400700 1 -- 172.21.15.89:0/426061185 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f056406af20 msgr2=0x7f056406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.307+0000 7f058c400700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f056406af20 0x7f056406d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f057800aa20 tx=0x7f0578008040 comp rx=0 tx=0).stop 2024-04-02T07:54:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.307+0000 7f058c400700 1 -- 172.21.15.89:0/426061185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f058415f4d0 msgr2=0x7f05841e8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.307+0000 7f058c400700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f058415f4d0 0x7f05841e8cf0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f0570013000 tx=0x7f0570004750 comp rx=0 tx=0).stop 2024-04-02T07:54:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.308+0000 7f058c400700 1 -- 172.21.15.89:0/426061185 shutdown_connections 2024-04-02T07:54:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.308+0000 7f058c400700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f056406af20 0x7f056406d3e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.308+0000 7f058c400700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05840d21f0 0x7f05841ea700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.308+0000 7f058c400700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05841611b0 0x7f05840c8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.308+0000 7f058c400700 1 --2- 172.21.15.89:0/426061185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f058415f4d0 0x7f05841e8cf0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.308+0000 7f058c400700 1 -- 172.21.15.89:0/426061185 >> 172.21.15.89:0/426061185 conn(0x7f058408aff0 msgr2=0x7f05840d16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.308+0000 7f058c400700 1 -- 172.21.15.89:0/426061185 shutdown_connections 2024-04-02T07:54:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.308+0000 7f058c400700 1 -- 172.21.15.89:0/426061185 wait complete. 2024-04-02T07:54:08.327 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2024-04-02T07:54:08.328 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T07:54:08.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.520+0000 7fd377600700 1 Processor -- start 2024-04-02T07:54:08.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.520+0000 7fd377600700 1 -- start start 2024-04-02T07:54:08.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.521+0000 7fd377600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd37015a5c0 0x7fd3700709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:08.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.521+0000 7fd377600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd37015dfe0 0x7fd3700ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:08.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.521+0000 7fd377600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3700cb310 con 0x7fd3700ce670 2024-04-02T07:54:08.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.521+0000 7fd377600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3700ceb00 con 0x7fd37015a5c0 2024-04-02T07:54:08.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.521+0000 7fd377600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd370070f00 con 0x7fd37015dfe0 2024-04-02T07:54:08.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.521+0000 7fd375600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd3700ce670 0x7fd3700b2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44350/0 (socket says 172.21.15.89:44350) 2024-04-02T07:54:08.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.521+0000 7fd375600700 1 -- 172.21.15.89:0/3941467806 learned_addr learned my addr 172.21.15.89:0/3941467806 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:08.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.522+0000 7fd36fe00700 1 --2- 172.21.15.89:0/3941467806 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd37015dfe0 0x7fd3700ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:08.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.522+0000 7fd374c00700 1 --2- 172.21.15.89:0/3941467806 >> v2:172.21.15.113:3300/0 conn(0x7fd37015a5c0 0x7fd3700709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:08.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.522+0000 7fd36f400700 1 -- 172.21.15.89:0/3941467806 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072527164 0 0) 0x7fd3700cb310 con 0x7fd3700ce670 2024-04-02T07:54:08.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.522+0000 7fd36f400700 1 -- 172.21.15.89:0/3941467806 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3580057c0 con 0x7fd3700ce670 2024-04-02T07:54:08.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.522+0000 7fd36fe00700 1 -- 172.21.15.89:0/3941467806 >> v2:172.21.15.113:3300/0 conn(0x7fd37015a5c0 msgr2=0x7fd3700709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:08.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.522+0000 7fd36f400700 1 -- 172.21.15.89:0/3941467806 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1104157700 0 0) 0x7fd3580057c0 con 0x7fd3700ce670 2024-04-02T07:54:08.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.522+0000 7fd36fe00700 1 --2- 172.21.15.89:0/3941467806 >> v2:172.21.15.113:3300/0 conn(0x7fd37015a5c0 0x7fd3700709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:08.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.522+0000 7fd36fe00700 1 -- 172.21.15.89:0/3941467806 >> v1:172.21.15.89:6789/0 conn(0x7fd3700ce670 legacy=0x7fd3700b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:08.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.522+0000 7fd36fe00700 1 -- 172.21.15.89:0/3941467806 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3700cca90 con 0x7fd37015dfe0 2024-04-02T07:54:08.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.522+0000 7fd374c00700 1 --2- 172.21.15.89:0/3941467806 >> v2:172.21.15.113:3300/0 conn(0x7fd37015a5c0 0x7fd3700709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:08.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.522+0000 7fd36fe00700 1 --2- 172.21.15.89:0/3941467806 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd37015dfe0 0x7fd3700ce130 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd35c009840 tx=0x7fd35c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b4e4216307264bb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:08.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.523+0000 7fd36f400700 1 -- 172.21.15.89:0/3941467806 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3680019f0 con 0x7fd37015dfe0 2024-04-02T07:54:08.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.523+0000 7fd36f400700 1 -- 172.21.15.89:0/3941467806 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35c004030 con 0x7fd37015dfe0 2024-04-02T07:54:08.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.523+0000 7fd36f400700 1 -- 172.21.15.89:0/3941467806 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c0043c0 con 0x7fd37015dfe0 2024-04-02T07:54:08.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.523+0000 7fd377600700 1 -- 172.21.15.89:0/3941467806 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd37015dfe0 msgr2=0x7fd3700ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:08.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.523+0000 7fd377600700 1 --2- 172.21.15.89:0/3941467806 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd37015dfe0 0x7fd3700ce130 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd35c009840 tx=0x7fd35c009b50 comp rx=0 tx=0).stop 2024-04-02T07:54:08.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.523+0000 7fd377600700 1 -- 172.21.15.89:0/3941467806 shutdown_connections 2024-04-02T07:54:08.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.523+0000 7fd377600700 1 --2- 172.21.15.89:0/3941467806 >> v2:172.21.15.113:3300/0 conn(0x7fd37015a5c0 0x7fd3700709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:08.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.523+0000 7fd377600700 1 --2- 172.21.15.89:0/3941467806 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd37015dfe0 0x7fd3700ce130 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:08.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.523+0000 7fd377600700 1 -- 172.21.15.89:0/3941467806 >> 172.21.15.89:0/3941467806 conn(0x7fd3700b0d70 msgr2=0x7fd3700b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:08.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.523+0000 7fd377600700 1 -- 172.21.15.89:0/3941467806 shutdown_connections 2024-04-02T07:54:08.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.523+0000 7fd377600700 1 -- 172.21.15.89:0/3941467806 wait complete. 2024-04-02T07:54:08.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.526+0000 7fd377600700 1 Processor -- start 2024-04-02T07:54:08.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.526+0000 7fd377600700 1 -- start start 2024-04-02T07:54:08.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.527+0000 7fd377600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3700ce670 0x7fd3701f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:08.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.527+0000 7fd377600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd37015a5c0 0x7fd37015aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:08.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.527+0000 7fd377600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd37015dfe0 0x7fd37015c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:08.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.527+0000 7fd377600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37015b330 con 0x7fd37015dfe0 2024-04-02T07:54:08.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.527+0000 7fd377600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3701f2da0 con 0x7fd3700ce670 2024-04-02T07:54:08.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.527+0000 7fd377600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3701544a0 con 0x7fd37015a5c0 2024-04-02T07:54:08.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.527+0000 7fd375600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd37015dfe0 0x7fd37015c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:08.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.527+0000 7fd375600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd37015dfe0 0x7fd37015c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52740/0 (socket says 172.21.15.89:52740) 2024-04-02T07:54:08.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.527+0000 7fd375600700 1 -- 172.21.15.89:0/3020901451 learned_addr learned my addr 172.21.15.89:0/3020901451 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:08.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.527+0000 7fd374c00700 1 --2- 172.21.15.89:0/3020901451 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3700ce670 0x7fd3701f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:08.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.527+0000 7fd36fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd37015a5c0 0x7fd37015aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:08.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.528+0000 7fd375600700 1 -- 172.21.15.89:0/3020901451 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd37015a5c0 msgr2=0x7fd37015aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:08.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.528+0000 7fd375600700 1 --2- 172.21.15.89:0/3020901451 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd37015a5c0 0x7fd37015aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:08.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.528+0000 7fd375600700 1 -- 172.21.15.89:0/3020901451 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3700ce670 msgr2=0x7fd3701f2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:08.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.528+0000 7fd375600700 1 --2- 172.21.15.89:0/3020901451 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3700ce670 0x7fd3701f2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:08.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.528+0000 7fd375600700 1 -- 172.21.15.89:0/3020901451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3640094a0 con 0x7fd37015dfe0 2024-04-02T07:54:08.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.528+0000 7fd375600700 1 --2- 172.21.15.89:0/3020901451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd37015dfe0 0x7fd37015c750 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fd36800cca0 tx=0x7fd36800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:08.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.528+0000 7fd36cc00700 1 -- 172.21.15.89:0/3020901451 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36800ab70 con 0x7fd37015dfe0 2024-04-02T07:54:08.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.528+0000 7fd36cc00700 1 -- 172.21.15.89:0/3020901451 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36800acf0 con 0x7fd37015dfe0 2024-04-02T07:54:08.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.528+0000 7fd377600700 1 -- 172.21.15.89:0/3020901451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c0094a0 con 0x7fd37015dfe0 2024-04-02T07:54:08.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.528+0000 7fd36cc00700 1 -- 172.21.15.89:0/3020901451 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd368007310 con 0x7fd37015dfe0 2024-04-02T07:54:08.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.529+0000 7fd377600700 1 -- 172.21.15.89:0/3020901451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd370154a60 con 0x7fd37015dfe0 2024-04-02T07:54:08.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.529+0000 7fd377600700 1 -- 172.21.15.89:0/3020901451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd370067720 con 0x7fd37015dfe0 2024-04-02T07:54:08.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.530+0000 7fd36cc00700 1 -- 172.21.15.89:0/3020901451 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 59) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fd368007470 con 0x7fd37015dfe0 2024-04-02T07:54:08.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.531+0000 7fd36cc00700 1 --2- 172.21.15.89:0/3020901451 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd35006af20 0x7fd35006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:08.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.531+0000 7fd36cc00700 1 -- 172.21.15.89:0/3020901451 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd368089900 con 0x7fd37015dfe0 2024-04-02T07:54:08.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.532+0000 7fd374c00700 1 --2- 172.21.15.89:0/3020901451 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd35006af20 0x7fd35006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:08.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.532+0000 7fd374c00700 1 --2- 172.21.15.89:0/3020901451 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd35006af20 0x7fd35006d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd3701f6430 tx=0x7fd364009210 comp rx=0 tx=0).ready entity=mgr.7596 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:08.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.536+0000 7fd36cc00700 1 -- 172.21.15.89:0/3020901451 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd3680544b0 con 0x7fd37015dfe0 2024-04-02T07:54:08.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:08.793+0000 7fd377600700 1 -- 172.21.15.89:0/3020901451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fd3700b1ad0 con 0x7fd37015dfe0 2024-04-02T07:54:09.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.131+0000 7fd36cc00700 1 -- 172.21.15.89:0/3020901451 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fd368054880 con 0x7fd37015dfe0 2024-04-02T07:54:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.132+0000 7fd36cc00700 1 -- 172.21.15.89:0/3020901451 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v60) v1 ==== 93+0+0 (secure 0 0 0) 0x7fd3680745b0 con 0x7fd37015dfe0 2024-04-02T07:54:09.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.135+0000 7fd377600700 1 -- 172.21.15.89:0/3020901451 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd35006af20 msgr2=0x7fd35006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:09.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.135+0000 7fd377600700 1 --2- 172.21.15.89:0/3020901451 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd35006af20 0x7fd35006d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd3701f6430 tx=0x7fd364009210 comp rx=0 tx=0).stop 2024-04-02T07:54:09.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.135+0000 7fd377600700 1 -- 172.21.15.89:0/3020901451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd37015dfe0 msgr2=0x7fd37015c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:09.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.135+0000 7fd377600700 1 --2- 172.21.15.89:0/3020901451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd37015dfe0 0x7fd37015c750 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fd36800cca0 tx=0x7fd36800a060 comp rx=0 tx=0).stop 2024-04-02T07:54:09.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.135+0000 7fd377600700 1 -- 172.21.15.89:0/3020901451 shutdown_connections 2024-04-02T07:54:09.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.135+0000 7fd377600700 1 --2- 172.21.15.89:0/3020901451 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd35006af20 0x7fd35006d3e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:09.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.135+0000 7fd377600700 1 --2- 172.21.15.89:0/3020901451 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3700ce670 0x7fd3701f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:09.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.135+0000 7fd377600700 1 --2- 172.21.15.89:0/3020901451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd37015dfe0 0x7fd37015c750 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:09.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.135+0000 7fd377600700 1 --2- 172.21.15.89:0/3020901451 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd37015a5c0 0x7fd37015aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:09.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.135+0000 7fd377600700 1 -- 172.21.15.89:0/3020901451 >> 172.21.15.89:0/3020901451 conn(0x7fd3700b0d70 msgr2=0x7fd3700ceef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:09.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.135+0000 7fd377600700 1 -- 172.21.15.89:0/3020901451 shutdown_connections 2024-04-02T07:54:09.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.135+0000 7fd377600700 1 -- 172.21.15.89:0/3020901451 wait complete. 2024-04-02T07:54:09.155 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:54:09.156 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:54:09.241 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:54:09.324 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:54:09.324 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:54:09.324 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:54:09.352 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:09.350+0000 7efd97554240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:54:09.364 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:09.362+0000 7f85b0efb240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:54:09.368 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:54:09.369 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:54:09.455 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:54:09.537 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:09.535+0000 7efd97554240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:54:09.556 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:09.555+0000 7f85b0efb240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:54:09.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.717+0000 7fa85a000700 1 Processor -- start 2024-04-02T07:54:09.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.717+0000 7fa85a000700 1 -- start start 2024-04-02T07:54:09.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.718+0000 7fa85a000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa85415d640 0x7fa8540d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:09.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.719+0000 7fa85a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa8541619d0 0x7fa8540b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:09.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.719+0000 7fa85a000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8540d30d0 con 0x7fa8540ceec0 2024-04-02T07:54:09.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.719+0000 7fa85a000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8540b6840 con 0x7fa8541619d0 2024-04-02T07:54:09.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.719+0000 7fa85a000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8540d2880 con 0x7fa85415d640 2024-04-02T07:54:09.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.719+0000 7fa853e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa8541619d0 0x7fa8540b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:09.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.719+0000 7fa853e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa8541619d0 0x7fa8540b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58440/0 (socket says 172.21.15.89:58440) 2024-04-02T07:54:09.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.719+0000 7fa853400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa85415d640 0x7fa8540d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:09.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.719+0000 7fa853e00700 1 -- 172.21.15.89:0/4103942784 learned_addr learned my addr 172.21.15.89:0/4103942784 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:09.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.719+0000 7fa853400700 1 -- 172.21.15.89:0/4103942784 >> v2:172.21.15.89:3300/0 conn(0x7fa8541619d0 msgr2=0x7fa8540b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:09.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.719+0000 7fa853400700 1 --2- 172.21.15.89:0/4103942784 >> v2:172.21.15.89:3300/0 conn(0x7fa8541619d0 0x7fa8540b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:09.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.719+0000 7fa853400700 1 -- 172.21.15.89:0/4103942784 >> v1:172.21.15.113:6789/0 conn(0x7fa8540ceec0 legacy=0x7fa8541612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:09.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.719+0000 7fa853400700 1 -- 172.21.15.89:0/4103942784 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8540d2b30 con 0x7fa85415d640 2024-04-02T07:54:09.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.720+0000 7fa853400700 1 --2- 172.21.15.89:0/4103942784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa85415d640 0x7fa8540d2340 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa8540b5c20 tx=0x7fa844009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fae58ee37e7d8bc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:09.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.720+0000 7fa852000700 1 -- 172.21.15.89:0/4103942784 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa844011830 con 0x7fa85415d640 2024-04-02T07:54:09.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.720+0000 7fa852000700 1 -- 172.21.15.89:0/4103942784 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa844011990 con 0x7fa85415d640 2024-04-02T07:54:09.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.720+0000 7fa852000700 1 -- 172.21.15.89:0/4103942784 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa844011d20 con 0x7fa85415d640 2024-04-02T07:54:09.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.721+0000 7fa85a000700 1 -- 172.21.15.89:0/4103942784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa85415d640 msgr2=0x7fa8540d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:09.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.721+0000 7fa85a000700 1 --2- 172.21.15.89:0/4103942784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa85415d640 0x7fa8540d2340 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa8540b5c20 tx=0x7fa844009b20 comp rx=0 tx=0).stop 2024-04-02T07:54:09.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.721+0000 7fa85a000700 1 -- 172.21.15.89:0/4103942784 shutdown_connections 2024-04-02T07:54:09.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.721+0000 7fa85a000700 1 --2- 172.21.15.89:0/4103942784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa85415d640 0x7fa8540d2340 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:09.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.721+0000 7fa85a000700 1 --2- 172.21.15.89:0/4103942784 >> v2:172.21.15.89:3300/0 conn(0x7fa8541619d0 0x7fa8540b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:09.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.721+0000 7fa85a000700 1 -- 172.21.15.89:0/4103942784 >> 172.21.15.89:0/4103942784 conn(0x7fa85408aff0 msgr2=0x7fa8540d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:09.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.721+0000 7fa85a000700 1 -- 172.21.15.89:0/4103942784 shutdown_connections 2024-04-02T07:54:09.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.721+0000 7fa85a000700 1 -- 172.21.15.89:0/4103942784 wait complete. 2024-04-02T07:54:09.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.722+0000 7fa85a000700 1 Processor -- start 2024-04-02T07:54:09.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.723+0000 7fa85a000700 1 -- start start 2024-04-02T07:54:09.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.723+0000 7fa85a000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8540ceec0 0x7fa8541585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:09.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.723+0000 7fa85a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85415d640 0x7fa85415ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:09.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.723+0000 7fa85a000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8541619d0 0x7fa85415f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:09.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.723+0000 7fa85a000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8540a1220 con 0x7fa85415d640 2024-04-02T07:54:09.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.723+0000 7fa85a000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8540a0c60 con 0x7fa8540ceec0 2024-04-02T07:54:09.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.723+0000 7fa85a000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8540a0f40 con 0x7fa8541619d0 2024-04-02T07:54:09.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.724+0000 7fa852a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85415d640 0x7fa85415ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:09.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.724+0000 7fa853400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8540ceec0 0x7fa8541585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:09.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.724+0000 7fa852a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85415d640 0x7fa85415ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58456/0 (socket says 172.21.15.89:58456) 2024-04-02T07:54:09.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.724+0000 7fa852a00700 1 -- 172.21.15.89:0/660320485 learned_addr learned my addr 172.21.15.89:0/660320485 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:09.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.724+0000 7fa853e00700 1 --2- 172.21.15.89:0/660320485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8541619d0 0x7fa85415f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:09.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.724+0000 7fa852a00700 1 -- 172.21.15.89:0/660320485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8541619d0 msgr2=0x7fa85415f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:09.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.724+0000 7fa852a00700 1 --2- 172.21.15.89:0/660320485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8541619d0 0x7fa85415f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:09.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.724+0000 7fa852a00700 1 -- 172.21.15.89:0/660320485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8540ceec0 msgr2=0x7fa8541585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:09.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.724+0000 7fa852a00700 1 --2- 172.21.15.89:0/660320485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8540ceec0 0x7fa8541585f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:09.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.724+0000 7fa852a00700 1 -- 172.21.15.89:0/660320485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8440094a0 con 0x7fa85415d640 2024-04-02T07:54:09.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.724+0000 7fa852a00700 1 --2- 172.21.15.89:0/660320485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85415d640 0x7fa85415ee10 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fa84000c450 tx=0x7fa84000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:09.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.725+0000 7fa84b400700 1 -- 172.21.15.89:0/660320485 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8400052d0 con 0x7fa85415d640 2024-04-02T07:54:09.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.725+0000 7fa84b400700 1 -- 172.21.15.89:0/660320485 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa840005430 con 0x7fa85415d640 2024-04-02T07:54:09.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.725+0000 7fa84b400700 1 -- 172.21.15.89:0/660320485 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8400038c0 con 0x7fa85415d640 2024-04-02T07:54:09.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.725+0000 7fa85a000700 1 -- 172.21.15.89:0/660320485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8540c86e0 con 0x7fa85415d640 2024-04-02T07:54:09.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.725+0000 7fa85a000700 1 -- 172.21.15.89:0/660320485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa854158b30 con 0x7fa85415d640 2024-04-02T07:54:09.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.726+0000 7fa85a000700 1 -- 172.21.15.89:0/660320485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8540c8c00 con 0x7fa85415d640 2024-04-02T07:54:09.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.727+0000 7fa84b400700 1 -- 172.21.15.89:0/660320485 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fa840003a20 con 0x7fa85415d640 2024-04-02T07:54:09.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.728+0000 7fa84b400700 1 --2- 172.21.15.89:0/660320485 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa83406aed0 0x7fa83406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:09.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.728+0000 7fa84b400700 1 -- 172.21.15.89:0/660320485 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fa840088d00 con 0x7fa85415d640 2024-04-02T07:54:09.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.728+0000 7fa853400700 1 -- 172.21.15.89:0/660320485 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa83406aed0 msgr2=0x7fa83406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:09.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.728+0000 7fa853400700 1 --2- 172.21.15.89:0/660320485 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa83406aed0 0x7fa83406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:09.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.732+0000 7fa84b400700 1 -- 172.21.15.89:0/660320485 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa8540c8c00 con 0x7fa85415d640 2024-04-02T07:54:09.857 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:09.855+0000 7efd97554240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:54:09.890 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:09.888+0000 7f85b0efb240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:54:09.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.929+0000 7fa853400700 1 -- 172.21.15.89:0/660320485 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa83406aed0 msgr2=0x7fa83406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:09.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.929+0000 7fa853400700 1 --2- 172.21.15.89:0/660320485 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa83406aed0 0x7fa83406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:09.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.939+0000 7fa85a000700 1 -- 172.21.15.89:0/660320485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8540c8c00 con 0x7fa85415d640 2024-04-02T07:54:09.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.940+0000 7fa84b400700 1 -- 172.21.15.89:0/660320485 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+17130 (secure 0 0 0) 0x7fa8540c8c00 con 0x7fa85415d640 2024-04-02T07:54:09.943 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:09.943 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":112,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:53:05.735192+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:52:46.885836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-02T07:52:47.989359+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/1115793129":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/458120672":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/2657927819":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/383704566":"2024-04-03T07:53:05.735155+0000","172.21.15.89:6832/43357":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/1938634818":"2024-04-03T07:53:05.735155+0000","172.21.15.89:6833/43357":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/552112415":"2024-04-03T07:53:05.735155+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:54:09.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.943+0000 7fa85a000700 1 -- 172.21.15.89:0/660320485 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa83406aed0 msgr2=0x7fa83406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:09.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.943+0000 7fa85a000700 1 --2- 172.21.15.89:0/660320485 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa83406aed0 0x7fa83406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:09.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.943+0000 7fa85a000700 1 -- 172.21.15.89:0/660320485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85415d640 msgr2=0x7fa85415ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:09.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.943+0000 7fa85a000700 1 --2- 172.21.15.89:0/660320485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85415d640 0x7fa85415ee10 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fa84000c450 tx=0x7fa84000c760 comp rx=0 tx=0).stop 2024-04-02T07:54:09.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.943+0000 7fa85a000700 1 -- 172.21.15.89:0/660320485 shutdown_connections 2024-04-02T07:54:09.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.943+0000 7fa85a000700 1 --2- 172.21.15.89:0/660320485 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa83406aed0 0x7fa83406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:09.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.943+0000 7fa85a000700 1 --2- 172.21.15.89:0/660320485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8540ceec0 0x7fa8541585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:09.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.943+0000 7fa85a000700 1 --2- 172.21.15.89:0/660320485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8541619d0 0x7fa85415f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:09.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.943+0000 7fa85a000700 1 --2- 172.21.15.89:0/660320485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa85415d640 0x7fa85415ee10 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:09.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.943+0000 7fa85a000700 1 -- 172.21.15.89:0/660320485 >> 172.21.15.89:0/660320485 conn(0x7fa85408aff0 msgr2=0x7fa8540b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:09.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.943+0000 7fa85a000700 1 -- 172.21.15.89:0/660320485 shutdown_connections 2024-04-02T07:54:09.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:09.943+0000 7fa85a000700 1 -- 172.21.15.89:0/660320485 wait complete. 2024-04-02T07:54:09.964 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:52:46.885836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:52:47.989359+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:54:09.964 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:54:10.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.162+0000 7f6024000700 1 Processor -- start 2024-04-02T07:54:10.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.162+0000 7f6024000700 1 -- start start 2024-04-02T07:54:10.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.164+0000 7f6024000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f601c0ce230 0x7f601c0ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:10.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.164+0000 7f6024000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f601c0b2070 0x7f601c0b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:10.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.164+0000 7f6024000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601c0709c0 con 0x7f601c15d800 2024-04-02T07:54:10.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.164+0000 7f6024000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f601c0b40f0 con 0x7f601c0ce230 2024-04-02T07:54:10.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.164+0000 7f6024000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f601c070c80 con 0x7f601c0b2070 2024-04-02T07:54:10.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.164+0000 7f6021600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f601c15d800 0x7f601c0725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33550/0 (socket says 172.21.15.89:33550) 2024-04-02T07:54:10.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.164+0000 7f6021600700 1 -- 172.21.15.89:0/3136179408 learned_addr learned my addr 172.21.15.89:0/3136179408 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:10.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.164+0000 7f6020c00700 1 --2- 172.21.15.89:0/3136179408 >> v2:172.21.15.113:3300/0 conn(0x7f601c0ce230 0x7f601c0ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:10.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.164+0000 7f6022000700 1 --2- 172.21.15.89:0/3136179408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f601c0b2070 0x7f601c0b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:10.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.165+0000 7f601be00700 1 -- 172.21.15.89:0/3136179408 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483083346 0 0) 0x7f601c0709c0 con 0x7f601c15d800 2024-04-02T07:54:10.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.165+0000 7f601be00700 1 -- 172.21.15.89:0/3136179408 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60040035b0 con 0x7f601c15d800 2024-04-02T07:54:10.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.165+0000 7f6020c00700 1 -- 172.21.15.89:0/3136179408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f601c0b2070 msgr2=0x7f601c0b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:10.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.165+0000 7f6020c00700 1 --2- 172.21.15.89:0/3136179408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f601c0b2070 0x7f601c0b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.165+0000 7f6020c00700 1 -- 172.21.15.89:0/3136179408 >> v1:172.21.15.89:6789/0 conn(0x7f601c15d800 legacy=0x7f601c0725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:10.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.165+0000 7f6020c00700 1 -- 172.21.15.89:0/3136179408 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601c070f60 con 0x7f601c0ce230 2024-04-02T07:54:10.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.165+0000 7f6022000700 1 --2- 172.21.15.89:0/3136179408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f601c0b2070 0x7f601c0b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:10.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.165+0000 7f6020c00700 1 --2- 172.21.15.89:0/3136179408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f601c0ce230 0x7f601c0ce6a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6008009840 tx=0x7f6008009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d81bddb8ba8730a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:10.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.166+0000 7f601be00700 1 -- 172.21.15.89:0/3136179408 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008011830 con 0x7f601c0ce230 2024-04-02T07:54:10.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.166+0000 7f601be00700 1 -- 172.21.15.89:0/3136179408 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6008011990 con 0x7f601c0ce230 2024-04-02T07:54:10.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.166+0000 7f601be00700 1 -- 172.21.15.89:0/3136179408 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008011d00 con 0x7f601c0ce230 2024-04-02T07:54:10.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.166+0000 7f6024000700 1 -- 172.21.15.89:0/3136179408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f601c0ce230 msgr2=0x7f601c0ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:10.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.166+0000 7f6024000700 1 --2- 172.21.15.89:0/3136179408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f601c0ce230 0x7f601c0ce6a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6008009840 tx=0x7f6008009b50 comp rx=0 tx=0).stop 2024-04-02T07:54:10.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.166+0000 7f6024000700 1 -- 172.21.15.89:0/3136179408 shutdown_connections 2024-04-02T07:54:10.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.166+0000 7f6024000700 1 --2- 172.21.15.89:0/3136179408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f601c0b2070 0x7f601c0b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.166+0000 7f6024000700 1 --2- 172.21.15.89:0/3136179408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f601c0ce230 0x7f601c0ce6a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.166+0000 7f6024000700 1 -- 172.21.15.89:0/3136179408 >> 172.21.15.89:0/3136179408 conn(0x7f601c0b0d70 msgr2=0x7f601c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:10.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.166+0000 7f6024000700 1 -- 172.21.15.89:0/3136179408 shutdown_connections 2024-04-02T07:54:10.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.166+0000 7f6024000700 1 -- 172.21.15.89:0/3136179408 wait complete. 2024-04-02T07:54:10.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.168+0000 7f6024000700 1 Processor -- start 2024-04-02T07:54:10.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.168+0000 7f6024000700 1 -- start start 2024-04-02T07:54:10.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.169+0000 7f6024000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f601c0b2070 0x7f601c158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:10.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.169+0000 7f6024000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f601c0ce230 0x7f601c140610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:10.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.169+0000 7f6024000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f601c15d800 0x7f601c15a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:10.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.169+0000 7f6024000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c15d0c0 con 0x7f601c0ce230 2024-04-02T07:54:10.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.169+0000 7f6024000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c15d3a0 con 0x7f601c0b2070 2024-04-02T07:54:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.169+0000 7f6024000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f601c159f90 con 0x7f601c15d800 2024-04-02T07:54:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.169+0000 7f6020c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f601c0ce230 0x7f601c140610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.169+0000 7f6020c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f601c0ce230 0x7f601c140610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58458/0 (socket says 172.21.15.89:58458) 2024-04-02T07:54:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.169+0000 7f6020c00700 1 -- 172.21.15.89:0/2657690895 learned_addr learned my addr 172.21.15.89:0/2657690895 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.169+0000 7f6021600700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f601c0b2070 0x7f601c158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.169+0000 7f6022000700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f601c15d800 0x7f601c15a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.170+0000 7f6020c00700 1 -- 172.21.15.89:0/2657690895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f601c15d800 msgr2=0x7f601c15a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.170+0000 7f6020c00700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f601c15d800 0x7f601c15a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.170+0000 7f6020c00700 1 -- 172.21.15.89:0/2657690895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f601c0b2070 msgr2=0x7f601c158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.170+0000 7f6020c00700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f601c0b2070 0x7f601c158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.170+0000 7f6020c00700 1 -- 172.21.15.89:0/2657690895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6014009a70 con 0x7f601c0ce230 2024-04-02T07:54:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.170+0000 7f6022000700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f601c15d800 0x7f601c15a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.170+0000 7f6020c00700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f601c0ce230 0x7f601c140610 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f60080090b0 tx=0x7f6008004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:10.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.170+0000 7f6019600700 1 -- 172.21.15.89:0/2657690895 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008004550 con 0x7f601c0ce230 2024-04-02T07:54:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.170+0000 7f6019600700 1 -- 172.21.15.89:0/2657690895 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600801f070 con 0x7f601c0ce230 2024-04-02T07:54:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.170+0000 7f6024000700 1 -- 172.21.15.89:0/2657690895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60080094a0 con 0x7f601c0ce230 2024-04-02T07:54:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.170+0000 7f6019600700 1 -- 172.21.15.89:0/2657690895 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60080038a0 con 0x7f601c0ce230 2024-04-02T07:54:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.170+0000 7f6024000700 1 -- 172.21.15.89:0/2657690895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601c158780 con 0x7f601c0ce230 2024-04-02T07:54:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.172+0000 7f6024000700 1 -- 172.21.15.89:0/2657690895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f601c067720 con 0x7f601c0ce230 2024-04-02T07:54:10.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.172+0000 7f6019600700 1 -- 172.21.15.89:0/2657690895 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f6008003a00 con 0x7f601c0ce230 2024-04-02T07:54:10.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.173+0000 7f6019600700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5ffc06aed0 0x7f5ffc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:10.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.173+0000 7f6019600700 1 -- 172.21.15.89:0/2657690895 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f6008021070 con 0x7f601c0ce230 2024-04-02T07:54:10.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.173+0000 7f6021600700 1 -- 172.21.15.89:0/2657690895 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5ffc06aed0 msgr2=0x7f5ffc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:10.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.173+0000 7f6021600700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5ffc06aed0 0x7f5ffc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:10.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.177+0000 7f6019600700 1 -- 172.21.15.89:0/2657690895 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6008055e10 con 0x7f601c0ce230 2024-04-02T07:54:10.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.374+0000 7f6021600700 1 -- 172.21.15.89:0/2657690895 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5ffc06aed0 msgr2=0x7f5ffc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:10.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.374+0000 7f6021600700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5ffc06aed0 0x7f5ffc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:10.387 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:10.386+0000 7efd97554240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:54:10.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.430+0000 7f6024000700 1 -- 172.21.15.89:0/2657690895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f601c0ae430 con 0x7f601c0ce230 2024-04-02T07:54:10.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.430+0000 7f6019600700 1 -- 172.21.15.89:0/2657690895 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6008003cb0 con 0x7f601c0ce230 2024-04-02T07:54:10.432 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:54:10.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.433+0000 7f6024000700 1 -- 172.21.15.89:0/2657690895 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5ffc06aed0 msgr2=0x7f5ffc06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:10.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.433+0000 7f6024000700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5ffc06aed0 0x7f5ffc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.433+0000 7f6024000700 1 -- 172.21.15.89:0/2657690895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f601c0ce230 msgr2=0x7f601c140610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:10.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.433+0000 7f6024000700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f601c0ce230 0x7f601c140610 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f60080090b0 tx=0x7f6008004750 comp rx=0 tx=0).stop 2024-04-02T07:54:10.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.433+0000 7f6024000700 1 -- 172.21.15.89:0/2657690895 shutdown_connections 2024-04-02T07:54:10.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.433+0000 7f6024000700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5ffc06aed0 0x7f5ffc06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.433+0000 7f6024000700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f601c0b2070 0x7f601c158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.433+0000 7f6024000700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f601c15d800 0x7f601c15a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.433+0000 7f6024000700 1 --2- 172.21.15.89:0/2657690895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f601c0ce230 0x7f601c140610 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.433+0000 7f6024000700 1 -- 172.21.15.89:0/2657690895 >> 172.21.15.89:0/2657690895 conn(0x7f601c0b0d70 msgr2=0x7f601c0cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:10.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.433+0000 7f6024000700 1 -- 172.21.15.89:0/2657690895 shutdown_connections 2024-04-02T07:54:10.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.433+0000 7f6024000700 1 -- 172.21.15.89:0/2657690895 wait complete. 2024-04-02T07:54:10.454 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:54:10.486 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:10.485+0000 7f85b0efb240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:54:10.642 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:10.640+0000 7efd97554240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:54:10.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.653+0000 7f6647200700 1 Processor -- start 2024-04-02T07:54:10.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.653+0000 7f6647200700 1 -- start start 2024-04-02T07:54:10.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.654+0000 7f6647200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f66400d21f0 0x7f66400ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:10.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.654+0000 7f6647200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66400cec40 0x7f6640161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:10.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.654+0000 7f6647200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66400ce270 con 0x7f66401611b0 2024-04-02T07:54:10.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.654+0000 7f6647200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664015b600 con 0x7f66400d21f0 2024-04-02T07:54:10.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.654+0000 7f6647200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66400d2760 con 0x7f66400cec40 2024-04-02T07:54:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.654+0000 7f663f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f66400d21f0 0x7f66400ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.655+0000 7f663f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f66400d21f0 0x7f66400ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58466/0 (socket says 172.21.15.89:58466) 2024-04-02T07:54:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.655+0000 7f663f400700 1 -- 172.21.15.89:0/3107468033 learned_addr learned my addr 172.21.15.89:0/3107468033 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.655+0000 7f6645200700 1 --2- 172.21.15.89:0/3107468033 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66400cec40 0x7f6640161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.655+0000 7f663f400700 1 -- 172.21.15.89:0/3107468033 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66400cec40 msgr2=0x7f6640161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.655+0000 7f663f400700 1 --2- 172.21.15.89:0/3107468033 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66400cec40 0x7f6640161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.655+0000 7f663f400700 1 -- 172.21.15.89:0/3107468033 >> v1:172.21.15.113:6789/0 conn(0x7f66401611b0 legacy=0x7f664015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.655+0000 7f663f400700 1 -- 172.21.15.89:0/3107468033 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66400d2a40 con 0x7f66400d21f0 2024-04-02T07:54:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.655+0000 7f663f400700 1 --2- 172.21.15.89:0/3107468033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66400d21f0 0x7f66400ce700 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f662c0098f0 tx=0x7f662c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1c36b52e277500 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.656+0000 7f663ea00700 1 -- 172.21.15.89:0/3107468033 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c011830 con 0x7f66400d21f0 2024-04-02T07:54:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.656+0000 7f663ea00700 1 -- 172.21.15.89:0/3107468033 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662c011990 con 0x7f66400d21f0 2024-04-02T07:54:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.656+0000 7f663ea00700 1 -- 172.21.15.89:0/3107468033 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c011d20 con 0x7f66400d21f0 2024-04-02T07:54:10.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.656+0000 7f6647200700 1 -- 172.21.15.89:0/3107468033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66400d21f0 msgr2=0x7f66400ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:10.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.656+0000 7f6647200700 1 --2- 172.21.15.89:0/3107468033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66400d21f0 0x7f66400ce700 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f662c0098f0 tx=0x7f662c009c00 comp rx=0 tx=0).stop 2024-04-02T07:54:10.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.656+0000 7f6647200700 1 -- 172.21.15.89:0/3107468033 shutdown_connections 2024-04-02T07:54:10.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.656+0000 7f6647200700 1 --2- 172.21.15.89:0/3107468033 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66400cec40 0x7f6640161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.656+0000 7f6647200700 1 --2- 172.21.15.89:0/3107468033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66400d21f0 0x7f66400ce700 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.656+0000 7f6647200700 1 -- 172.21.15.89:0/3107468033 >> 172.21.15.89:0/3107468033 conn(0x7f664008aff0 msgr2=0x7f66400d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:10.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.656+0000 7f6647200700 1 -- 172.21.15.89:0/3107468033 shutdown_connections 2024-04-02T07:54:10.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.656+0000 7f6647200700 1 -- 172.21.15.89:0/3107468033 wait complete. 2024-04-02T07:54:10.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.658+0000 7f6647200700 1 Processor -- start 2024-04-02T07:54:10.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.658+0000 7f6647200700 1 -- start start 2024-04-02T07:54:10.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.659+0000 7f6647200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66400cec40 0x7f6640159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:10.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.659+0000 7f6647200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66400d21f0 0x7f66401e9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:10.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.659+0000 7f6647200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66401611b0 0x7f66400b7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:10.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.659+0000 7f6647200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664015a530 con 0x7f66400d21f0 2024-04-02T07:54:10.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.659+0000 7f6647200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66401e7aa0 con 0x7f66400cec40 2024-04-02T07:54:10.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.659+0000 7f6647200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66401e7d40 con 0x7f66401611b0 2024-04-02T07:54:10.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.659+0000 7f663f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66400d21f0 0x7f66401e9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:10.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.659+0000 7f663f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66400d21f0 0x7f66401e9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58476/0 (socket says 172.21.15.89:58476) 2024-04-02T07:54:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.659+0000 7f663fe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66400cec40 0x7f6640159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.659+0000 7f663f400700 1 -- 172.21.15.89:0/3666305713 learned_addr learned my addr 172.21.15.89:0/3666305713 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.659+0000 7f6645200700 1 --2- 172.21.15.89:0/3666305713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66401611b0 0x7f66400b7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.660+0000 7f663f400700 1 -- 172.21.15.89:0/3666305713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66401611b0 msgr2=0x7f66400b7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.660+0000 7f663f400700 1 --2- 172.21.15.89:0/3666305713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66401611b0 0x7f66400b7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.660+0000 7f663f400700 1 -- 172.21.15.89:0/3666305713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66400cec40 msgr2=0x7f6640159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.660+0000 7f663f400700 1 --2- 172.21.15.89:0/3666305713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66400cec40 0x7f6640159ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.660+0000 7f663f400700 1 -- 172.21.15.89:0/3666305713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c009550 con 0x7f66400d21f0 2024-04-02T07:54:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.660+0000 7f663f400700 1 --2- 172.21.15.89:0/3666305713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66400d21f0 0x7f66401e9770 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f662c009c40 tx=0x7f662c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.660+0000 7f6637e00700 1 -- 172.21.15.89:0/3666305713 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c004380 con 0x7f66400d21f0 2024-04-02T07:54:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.660+0000 7f6637e00700 1 -- 172.21.15.89:0/3666305713 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662c01f070 con 0x7f66400d21f0 2024-04-02T07:54:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.660+0000 7f6647200700 1 -- 172.21.15.89:0/3666305713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66401e8020 con 0x7f66400d21f0 2024-04-02T07:54:10.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.660+0000 7f6637e00700 1 -- 172.21.15.89:0/3666305713 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c0036a0 con 0x7f66400d21f0 2024-04-02T07:54:10.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.661+0000 7f6647200700 1 -- 172.21.15.89:0/3666305713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66401e85e0 con 0x7f66400d21f0 2024-04-02T07:54:10.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.661+0000 7f6647200700 1 -- 172.21.15.89:0/3666305713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6640067720 con 0x7f66400d21f0 2024-04-02T07:54:10.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.663+0000 7f6637e00700 1 -- 172.21.15.89:0/3666305713 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f662c021030 con 0x7f66400d21f0 2024-04-02T07:54:10.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.663+0000 7f6637e00700 1 --2- 172.21.15.89:0/3666305713 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f662006aed0 0x7f662006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:10.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.664+0000 7f6637e00700 1 -- 172.21.15.89:0/3666305713 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f662c08ada0 con 0x7f66400d21f0 2024-04-02T07:54:10.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.664+0000 7f663fe00700 1 -- 172.21.15.89:0/3666305713 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f662006aed0 msgr2=0x7f662006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:10.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.664+0000 7f663fe00700 1 --2- 172.21.15.89:0/3666305713 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f662006aed0 0x7f662006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:10.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.668+0000 7f6637e00700 1 -- 172.21.15.89:0/3666305713 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f662c0558d0 con 0x7f66400d21f0 2024-04-02T07:54:10.762 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:10.760+0000 7f85b0efb240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:54:10.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.865+0000 7f663fe00700 1 -- 172.21.15.89:0/3666305713 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f662006aed0 msgr2=0x7f662006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:10.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.865+0000 7f663fe00700 1 --2- 172.21.15.89:0/3666305713 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f662006aed0 0x7f662006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:10.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.876+0000 7f6647200700 1 -- 172.21.15.89:0/3666305713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f664015d600 con 0x7f66400d21f0 2024-04-02T07:54:10.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.877+0000 7f6637e00700 1 -- 172.21.15.89:0/3666305713 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v112) v1 ==== 104+0+11 (secure 0 0 0) 0x7f662c058ef0 con 0x7f66400d21f0 2024-04-02T07:54:10.878 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:54:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.879+0000 7f6647200700 1 -- 172.21.15.89:0/3666305713 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f662006aed0 msgr2=0x7f662006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.879+0000 7f6647200700 1 --2- 172.21.15.89:0/3666305713 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f662006aed0 0x7f662006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.879+0000 7f6647200700 1 -- 172.21.15.89:0/3666305713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66400d21f0 msgr2=0x7f66401e9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.879+0000 7f6647200700 1 --2- 172.21.15.89:0/3666305713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66400d21f0 0x7f66401e9770 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f662c009c40 tx=0x7f662c004750 comp rx=0 tx=0).stop 2024-04-02T07:54:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.879+0000 7f6647200700 1 -- 172.21.15.89:0/3666305713 shutdown_connections 2024-04-02T07:54:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.879+0000 7f6647200700 1 --2- 172.21.15.89:0/3666305713 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f662006aed0 0x7f662006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.879+0000 7f6647200700 1 --2- 172.21.15.89:0/3666305713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66400cec40 0x7f6640159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.879+0000 7f6647200700 1 --2- 172.21.15.89:0/3666305713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66401611b0 0x7f66400b7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.879+0000 7f6647200700 1 --2- 172.21.15.89:0/3666305713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66400d21f0 0x7f66401e9770 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:10.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.879+0000 7f6647200700 1 -- 172.21.15.89:0/3666305713 >> 172.21.15.89:0/3666305713 conn(0x7f664008aff0 msgr2=0x7f664015f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:10.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.879+0000 7f6647200700 1 -- 172.21.15.89:0/3666305713 shutdown_connections 2024-04-02T07:54:10.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:10.879+0000 7f6647200700 1 -- 172.21.15.89:0/3666305713 wait complete. 2024-04-02T07:54:10.899 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:54:11.098 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:11.096+0000 7efd97554240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:54:11.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.137+0000 7f716d400700 1 Processor -- start 2024-04-02T07:54:11.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.138+0000 7f716d400700 1 -- start start 2024-04-02T07:54:11.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.139+0000 7f716d400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f71681611b0 0x7f716815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:11.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.139+0000 7f716d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f716815f4d0 0x7f71680ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:11.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.139+0000 7f716d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71680d21f0 0x7f71680cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:11.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.139+0000 7f716d400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71680cf180 con 0x7f716815f4d0 2024-04-02T07:54:11.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.139+0000 7f716d400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71680ce270 con 0x7f71681611b0 2024-04-02T07:54:11.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.139+0000 7f716d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716815b600 con 0x7f71680d21f0 2024-04-02T07:54:11.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.140+0000 7f715fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f716815f4d0 0x7f71680ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:11.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.140+0000 7f715fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f716815f4d0 0x7f71680ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58486/0 (socket says 172.21.15.89:58486) 2024-04-02T07:54:11.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.140+0000 7f715fe00700 1 -- 172.21.15.89:0/4215440371 learned_addr learned my addr 172.21.15.89:0/4215440371 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:11.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.140+0000 7f7166800700 1 --2- 172.21.15.89:0/4215440371 >> v2:172.21.15.113:3300/0 conn(0x7f71681611b0 0x7f716815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:11.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.140+0000 7f7167200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71680d21f0 0x7f71680cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:11.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.140+0000 7f715fe00700 1 -- 172.21.15.89:0/4215440371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71680d21f0 msgr2=0x7f71680cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:11.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.140+0000 7f715fe00700 1 --2- 172.21.15.89:0/4215440371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71680d21f0 0x7f71680cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.140+0000 7f715fe00700 1 -- 172.21.15.89:0/4215440371 >> v2:172.21.15.113:3300/0 conn(0x7f71681611b0 msgr2=0x7f716815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:11.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.140+0000 7f715fe00700 1 --2- 172.21.15.89:0/4215440371 >> v2:172.21.15.113:3300/0 conn(0x7f71681611b0 0x7f716815ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.140+0000 7f715fe00700 1 -- 172.21.15.89:0/4215440371 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71680d2760 con 0x7f716815f4d0 2024-04-02T07:54:11.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.140+0000 7f715fe00700 1 --2- 172.21.15.89:0/4215440371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f716815f4d0 0x7f71680ce700 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f716808b520 tx=0x7f7150009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2563d0dd62bc2088 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:11.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.141+0000 7f7165e00700 1 -- 172.21.15.89:0/4215440371 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150011830 con 0x7f716815f4d0 2024-04-02T07:54:11.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.141+0000 7f7165e00700 1 -- 172.21.15.89:0/4215440371 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7150011990 con 0x7f716815f4d0 2024-04-02T07:54:11.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.141+0000 7f7165e00700 1 -- 172.21.15.89:0/4215440371 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150011d20 con 0x7f716815f4d0 2024-04-02T07:54:11.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.142+0000 7f716d400700 1 -- 172.21.15.89:0/4215440371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f716815f4d0 msgr2=0x7f71680ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:11.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.142+0000 7f716d400700 1 --2- 172.21.15.89:0/4215440371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f716815f4d0 0x7f71680ce700 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f716808b520 tx=0x7f7150009bd0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.142+0000 7f716d400700 1 -- 172.21.15.89:0/4215440371 shutdown_connections 2024-04-02T07:54:11.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.142+0000 7f716d400700 1 --2- 172.21.15.89:0/4215440371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71680d21f0 0x7f71680cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.142+0000 7f716d400700 1 --2- 172.21.15.89:0/4215440371 >> v2:172.21.15.113:3300/0 conn(0x7f71681611b0 0x7f716815ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.142+0000 7f716d400700 1 --2- 172.21.15.89:0/4215440371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f716815f4d0 0x7f71680ce700 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.142+0000 7f716d400700 1 -- 172.21.15.89:0/4215440371 >> 172.21.15.89:0/4215440371 conn(0x7f716808aff0 msgr2=0x7f71680d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:11.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.142+0000 7f716d400700 1 -- 172.21.15.89:0/4215440371 shutdown_connections 2024-04-02T07:54:11.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.142+0000 7f716d400700 1 -- 172.21.15.89:0/4215440371 wait complete. 2024-04-02T07:54:11.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.144+0000 7f716d400700 1 Processor -- start 2024-04-02T07:54:11.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.144+0000 7f716d400700 1 -- start start 2024-04-02T07:54:11.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.146+0000 7f716d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71680d21f0 0x7f71681f2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:11.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.146+0000 7f716d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f716815f4d0 0x7f71680a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:11.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.146+0000 7f716d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71681611b0 0x7f716815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:11.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.146+0000 7f716d400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71680a1210 con 0x7f71681611b0 2024-04-02T07:54:11.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.146+0000 7f716d400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71681f74f0 con 0x7f716815f4d0 2024-04-02T07:54:11.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.146+0000 7f716d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71681f6730 con 0x7f71680d21f0 2024-04-02T07:54:11.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.146+0000 7f7167200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71681611b0 0x7f716815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:11.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.146+0000 7f715fe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f716815f4d0 0x7f71680a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:11.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.146+0000 7f7166800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71680d21f0 0x7f71681f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:11.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.146+0000 7f715fe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f716815f4d0 0x7f71680a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:56174/0 (socket says 172.21.15.89:56174) 2024-04-02T07:54:11.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.146+0000 7f715fe00700 1 -- 172.21.15.89:0/3604686230 learned_addr learned my addr 172.21.15.89:0/3604686230 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:11.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.147+0000 7f715fe00700 1 -- 172.21.15.89:0/3604686230 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71680d21f0 msgr2=0x7f71681f2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:11.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.147+0000 7f715fe00700 1 --2- 172.21.15.89:0/3604686230 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71680d21f0 0x7f71681f2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.147+0000 7f715fe00700 1 -- 172.21.15.89:0/3604686230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71681611b0 msgr2=0x7f716815bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:11.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.147+0000 7f715fe00700 1 --2- 172.21.15.89:0/3604686230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71681611b0 0x7f716815bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.147+0000 7f715fe00700 1 -- 172.21.15.89:0/3604686230 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7150009550 con 0x7f716815f4d0 2024-04-02T07:54:11.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.147+0000 7f715fe00700 1 --2- 172.21.15.89:0/3604686230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f716815f4d0 0x7f71680a0810 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7150000c00 tx=0x7f7150004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:11.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.147+0000 7f715ea00700 1 -- 172.21.15.89:0/3604686230 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150004240 con 0x7f716815f4d0 2024-04-02T07:54:11.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.147+0000 7f715ea00700 1 -- 172.21.15.89:0/3604686230 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715001f070 con 0x7f716815f4d0 2024-04-02T07:54:11.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.147+0000 7f715ea00700 1 -- 172.21.15.89:0/3604686230 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71500036a0 con 0x7f716815f4d0 2024-04-02T07:54:11.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.147+0000 7f716d400700 1 -- 172.21.15.89:0/3604686230 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71681f6a10 con 0x7f716815f4d0 2024-04-02T07:54:11.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.147+0000 7f716d400700 1 -- 172.21.15.89:0/3604686230 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71681f6fd0 con 0x7f716815f4d0 2024-04-02T07:54:11.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.148+0000 7f716d400700 1 -- 172.21.15.89:0/3604686230 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7168067720 con 0x7f716815f4d0 2024-04-02T07:54:11.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.149+0000 7f715ea00700 1 -- 172.21.15.89:0/3604686230 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f7150021030 con 0x7f716815f4d0 2024-04-02T07:54:11.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.150+0000 7f715ea00700 1 --2- 172.21.15.89:0/3604686230 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f714406aed0 0x7f714406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:11.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.150+0000 7f715ea00700 1 -- 172.21.15.89:0/3604686230 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f715008ab40 con 0x7f716815f4d0 2024-04-02T07:54:11.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.150+0000 7f7166800700 1 -- 172.21.15.89:0/3604686230 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f714406aed0 msgr2=0x7f714406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:11.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.150+0000 7f7166800700 1 --2- 172.21.15.89:0/3604686230 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f714406aed0 0x7f714406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:11.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.160+0000 7f715ea00700 1 -- 172.21.15.89:0/3604686230 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7150055710 con 0x7f716815f4d0 2024-04-02T07:54:11.246 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:11.245+0000 7f85b0efb240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:54:11.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.351+0000 7f7166800700 1 -- 172.21.15.89:0/3604686230 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f714406aed0 msgr2=0x7f714406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:11.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.351+0000 7f7166800700 1 --2- 172.21.15.89:0/3604686230 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f714406aed0 0x7f714406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:11.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.369+0000 7f716d400700 1 -- 172.21.15.89:0/3604686230 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f71680b5d00 con 0x7f716815f4d0 2024-04-02T07:54:11.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.370+0000 7f715ea00700 1 -- 172.21.15.89:0/3604686230 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v112) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7150015020 con 0x7f716815f4d0 2024-04-02T07:54:11.372 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:54:11.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.372+0000 7f716d400700 1 -- 172.21.15.89:0/3604686230 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f714406aed0 msgr2=0x7f714406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:11.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.372+0000 7f716d400700 1 --2- 172.21.15.89:0/3604686230 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f714406aed0 0x7f714406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.372+0000 7f716d400700 1 -- 172.21.15.89:0/3604686230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f716815f4d0 msgr2=0x7f71680a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:11.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.372+0000 7f716d400700 1 --2- 172.21.15.89:0/3604686230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f716815f4d0 0x7f71680a0810 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7150000c00 tx=0x7f7150004750 comp rx=0 tx=0).stop 2024-04-02T07:54:11.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.372+0000 7f716d400700 1 -- 172.21.15.89:0/3604686230 shutdown_connections 2024-04-02T07:54:11.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.372+0000 7f716d400700 1 --2- 172.21.15.89:0/3604686230 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f714406aed0 0x7f714406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.372+0000 7f716d400700 1 --2- 172.21.15.89:0/3604686230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71681611b0 0x7f716815bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.372+0000 7f716d400700 1 --2- 172.21.15.89:0/3604686230 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71680d21f0 0x7f71681f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.372+0000 7f716d400700 1 --2- 172.21.15.89:0/3604686230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f716815f4d0 0x7f71680a0810 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.372+0000 7f716d400700 1 -- 172.21.15.89:0/3604686230 >> 172.21.15.89:0/3604686230 conn(0x7f716808aff0 msgr2=0x7f716815da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:11.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.373+0000 7f716d400700 1 -- 172.21.15.89:0/3604686230 shutdown_connections 2024-04-02T07:54:11.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.373+0000 7f716d400700 1 -- 172.21.15.89:0/3604686230 wait complete. 2024-04-02T07:54:11.392 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T07:54:11.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.586+0000 7f15bc200700 1 Processor -- start 2024-04-02T07:54:11.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.586+0000 7f15bc200700 1 -- start start 2024-04-02T07:54:11.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.587+0000 7f15bc200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f15b40d16a0 0x7f15b40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:11.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.587+0000 7f15bc200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15b40d2370 0x7f15b40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:11.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.587+0000 7f15bc200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b4161420 con 0x7f15b415d9d0 2024-04-02T07:54:11.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.587+0000 7f15bc200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b40d2820 con 0x7f15b40d16a0 2024-04-02T07:54:11.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.587+0000 7f15bc200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b40d4340 con 0x7f15b40d2370 2024-04-02T07:54:11.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.587+0000 7f15b9800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f15b415d9d0 0x7f15b40d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33564/0 (socket says 172.21.15.89:33564) 2024-04-02T07:54:11.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.587+0000 7f15b9800700 1 -- 172.21.15.89:0/1848005725 learned_addr learned my addr 172.21.15.89:0/1848005725 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:11.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.587+0000 7f15b8e00700 1 --2- 172.21.15.89:0/1848005725 >> v2:172.21.15.113:3300/0 conn(0x7f15b40d16a0 0x7f15b40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:11.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.587+0000 7f15ba200700 1 --2- 172.21.15.89:0/1848005725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15b40d2370 0x7f15b40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:11.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.588+0000 7f15b3e00700 1 -- 172.21.15.89:0/1848005725 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988330808 0 0) 0x7f15b4161420 con 0x7f15b415d9d0 2024-04-02T07:54:11.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.588+0000 7f15b3e00700 1 -- 172.21.15.89:0/1848005725 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f159c0035b0 con 0x7f15b415d9d0 2024-04-02T07:54:11.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.588+0000 7f15b8e00700 1 -- 172.21.15.89:0/1848005725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15b40d2370 msgr2=0x7f15b40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:11.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.588+0000 7f15b8e00700 1 --2- 172.21.15.89:0/1848005725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15b40d2370 0x7f15b40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.588+0000 7f15b8e00700 1 -- 172.21.15.89:0/1848005725 >> v1:172.21.15.89:6789/0 conn(0x7f15b415d9d0 legacy=0x7f15b40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:11.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.588+0000 7f15b8e00700 1 -- 172.21.15.89:0/1848005725 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b4161700 con 0x7f15b40d16a0 2024-04-02T07:54:11.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.588+0000 7f15b8e00700 1 --2- 172.21.15.89:0/1848005725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15b40d16a0 0x7f15b40d1e30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009820 tx=0x7f15a0009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ec6d6ee857783d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:11.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.588+0000 7f15b3e00700 1 -- 172.21.15.89:0/1848005725 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0005e30 con 0x7f15b40d16a0 2024-04-02T07:54:11.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.588+0000 7f15b3e00700 1 -- 172.21.15.89:0/1848005725 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a0006030 con 0x7f15b40d16a0 2024-04-02T07:54:11.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.588+0000 7f15b3e00700 1 -- 172.21.15.89:0/1848005725 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a00063a0 con 0x7f15b40d16a0 2024-04-02T07:54:11.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.589+0000 7f15bc200700 1 -- 172.21.15.89:0/1848005725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15b40d16a0 msgr2=0x7f15b40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:11.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.589+0000 7f15bc200700 1 --2- 172.21.15.89:0/1848005725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15b40d16a0 0x7f15b40d1e30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009820 tx=0x7f15a0009b30 comp rx=0 tx=0).stop 2024-04-02T07:54:11.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.589+0000 7f15bc200700 1 -- 172.21.15.89:0/1848005725 shutdown_connections 2024-04-02T07:54:11.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.589+0000 7f15bc200700 1 --2- 172.21.15.89:0/1848005725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15b40d2370 0x7f15b40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.589+0000 7f15bc200700 1 --2- 172.21.15.89:0/1848005725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15b40d16a0 0x7f15b40d1e30 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.589+0000 7f15bc200700 1 -- 172.21.15.89:0/1848005725 >> 172.21.15.89:0/1848005725 conn(0x7f15b408aff0 msgr2=0x7f15b40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:11.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.589+0000 7f15bc200700 1 -- 172.21.15.89:0/1848005725 shutdown_connections 2024-04-02T07:54:11.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.589+0000 7f15bc200700 1 -- 172.21.15.89:0/1848005725 wait complete. 2024-04-02T07:54:11.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.591+0000 7f15bc200700 1 Processor -- start 2024-04-02T07:54:11.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.591+0000 7f15bc200700 1 -- start start 2024-04-02T07:54:11.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.592+0000 7f15bc200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15b40d16a0 0x7f15b414f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:11.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.592+0000 7f15bc200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15b40d2370 0x7f15b415bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:11.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.592+0000 7f15bc200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15b415d9d0 0x7f15b415c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:11.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.592+0000 7f15bc200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b40cb050 con 0x7f15b40d2370 2024-04-02T07:54:11.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.592+0000 7f15bc200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b4160bb0 con 0x7f15b415d9d0 2024-04-02T07:54:11.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.592+0000 7f15bc200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b40ca720 con 0x7f15b40d16a0 2024-04-02T07:54:11.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.592+0000 7f15b8e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15b40d2370 0x7f15b415bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:11.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.592+0000 7f15b8e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15b40d2370 0x7f15b415bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58512/0 (socket says 172.21.15.89:58512) 2024-04-02T07:54:11.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.592+0000 7f15b8e00700 1 -- 172.21.15.89:0/2126169890 learned_addr learned my addr 172.21.15.89:0/2126169890 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:11.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.592+0000 7f15b9800700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15b40d16a0 0x7f15b414f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:11.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.592+0000 7f15ba200700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15b415d9d0 0x7f15b415c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:11.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.593+0000 7f15b8e00700 1 -- 172.21.15.89:0/2126169890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15b40d16a0 msgr2=0x7f15b414f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:11.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.593+0000 7f15b8e00700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15b40d16a0 0x7f15b414f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.593+0000 7f15b8e00700 1 -- 172.21.15.89:0/2126169890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15b415d9d0 msgr2=0x7f15b415c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:11.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.593+0000 7f15b8e00700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15b415d9d0 0x7f15b415c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.593+0000 7f15b8e00700 1 -- 172.21.15.89:0/2126169890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a0009430 con 0x7f15b40d2370 2024-04-02T07:54:11.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.593+0000 7f15b9800700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15b40d16a0 0x7f15b414f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:11.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.593+0000 7f15ba200700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15b415d9d0 0x7f15b415c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:11.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.593+0000 7f15b8e00700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15b40d2370 0x7f15b415bc10 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f15a0000c00 tx=0x7f15a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:11.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.593+0000 7f15b1600700 1 -- 172.21.15.89:0/2126169890 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a00063a0 con 0x7f15b40d2370 2024-04-02T07:54:11.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.593+0000 7f15b1600700 1 -- 172.21.15.89:0/2126169890 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a0006500 con 0x7f15b40d2370 2024-04-02T07:54:11.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.593+0000 7f15bc200700 1 -- 172.21.15.89:0/2126169890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b40caa00 con 0x7f15b40d2370 2024-04-02T07:54:11.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.593+0000 7f15b1600700 1 -- 172.21.15.89:0/2126169890 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a00068b0 con 0x7f15b40d2370 2024-04-02T07:54:11.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.594+0000 7f15bc200700 1 -- 172.21.15.89:0/2126169890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b414fe50 con 0x7f15b40d2370 2024-04-02T07:54:11.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.595+0000 7f15bc200700 1 -- 172.21.15.89:0/2126169890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b40d1ca0 con 0x7f15b40d2370 2024-04-02T07:54:11.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.596+0000 7f15b1600700 1 -- 172.21.15.89:0/2126169890 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f15a0022070 con 0x7f15b40d2370 2024-04-02T07:54:11.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.597+0000 7f15b1600700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f159406aed0 0x7f159406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:11.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.597+0000 7f15b1600700 1 -- 172.21.15.89:0/2126169890 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f15a008afe0 con 0x7f15b40d2370 2024-04-02T07:54:11.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.597+0000 7f15b9800700 1 -- 172.21.15.89:0/2126169890 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f159406aed0 msgr2=0x7f159406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:11.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.597+0000 7f15b9800700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f159406aed0 0x7f159406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:11.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.601+0000 7f15b1600700 1 -- 172.21.15.89:0/2126169890 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15a0055bb0 con 0x7f15b40d2370 2024-04-02T07:54:11.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.798+0000 7f15b9800700 1 -- 172.21.15.89:0/2126169890 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f159406aed0 msgr2=0x7f159406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.798+0000 7f15b9800700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f159406aed0 0x7f159406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:11.813 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:11.812+0000 7efd97554240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:54:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.818+0000 7f15bc200700 1 -- 172.21.15.89:0/2126169890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f15b4067720 con 0x7f15b40d2370 2024-04-02T07:54:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.819+0000 7f15b1600700 1 -- 172.21.15.89:0/2126169890 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v112) v1 ==== 80+0+457 (secure 0 0 0) 0x7f15a00591d0 con 0x7f15b40d2370 2024-04-02T07:54:11.821 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1115793129 2024-04-03T07:53:05.735155+0000 2024-04-02T07:54:11.821 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/458120672 2024-04-03T07:53:05.735155+0000 2024-04-02T07:54:11.821 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2657927819 2024-04-03T07:53:05.735155+0000 2024-04-02T07:54:11.821 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/383704566 2024-04-03T07:53:05.735155+0000 2024-04-02T07:54:11.821 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/43357 2024-04-03T07:53:05.735155+0000 2024-04-02T07:54:11.821 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1938634818 2024-04-03T07:53:05.735155+0000 2024-04-02T07:54:11.821 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/43357 2024-04-03T07:53:05.735155+0000 2024-04-02T07:54:11.821 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/552112415 2024-04-03T07:53:05.735155+0000 2024-04-02T07:54:11.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.821+0000 7f15bc200700 1 -- 172.21.15.89:0/2126169890 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f159406aed0 msgr2=0x7f159406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.821+0000 7f15bc200700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f159406aed0 0x7f159406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.821+0000 7f15bc200700 1 -- 172.21.15.89:0/2126169890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15b40d2370 msgr2=0x7f15b415bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.821+0000 7f15bc200700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15b40d2370 0x7f15b415bc10 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f15a0000c00 tx=0x7f15a0004750 comp rx=0 tx=0).stop 2024-04-02T07:54:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.821+0000 7f15bc200700 1 -- 172.21.15.89:0/2126169890 shutdown_connections 2024-04-02T07:54:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.821+0000 7f15bc200700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f159406aed0 0x7f159406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.821+0000 7f15bc200700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15b415d9d0 0x7f15b415c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.821+0000 7f15bc200700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15b40d16a0 0x7f15b414f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.821+0000 7f15bc200700 1 --2- 172.21.15.89:0/2126169890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15b40d2370 0x7f15b415bc10 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.822+0000 7f15bc200700 1 -- 172.21.15.89:0/2126169890 >> 172.21.15.89:0/2126169890 conn(0x7f15b408aff0 msgr2=0x7f15b40cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.822+0000 7f15bc200700 1 -- 172.21.15.89:0/2126169890 shutdown_connections 2024-04-02T07:54:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:11.822+0000 7f15bc200700 1 -- 172.21.15.89:0/2126169890 wait complete. 2024-04-02T07:54:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:listed 8 entries 2024-04-02T07:54:11.842 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T07:54:11.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:11.843 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T07:54:11.904 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:11.903+0000 7efd97554240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:54:11.915 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:54:11.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:11.915 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T07:54:11.960 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:54:11.961 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:54:11.993 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:11.991+0000 7f85b0efb240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:54:12.004 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:54:12.005 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:54:12.005 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:54:12.049 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:54:12.049 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:54:12.049 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:54:12.070 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:12.069+0000 7efd97554240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:54:12.089 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:12.088+0000 7f85b0efb240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:54:12.093 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:54:12.093 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:54:12.260 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:12.258+0000 7f85b0efb240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:54:12.286 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:12.284+0000 7efd97554240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:54:12.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.288+0000 7fd24b400700 1 Processor -- start 2024-04-02T07:54:12.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.288+0000 7fd24b400700 1 -- start start 2024-04-02T07:54:12.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.289+0000 7fd24b400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd24415b5d0 0x7fd2440ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:12.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.289+0000 7fd24b400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24415fd50 con 0x7fd2440ac7d0 2024-04-02T07:54:12.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.289+0000 7fd24b400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24415f140 con 0x7fd2440cb3f0 2024-04-02T07:54:12.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.289+0000 7fd24b400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24415b9b0 con 0x7fd24415b5d0 2024-04-02T07:54:12.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.289+0000 7fd249400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd2440ac7d0 0x7fd24406c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33576/0 (socket says 172.21.15.89:33576) 2024-04-02T07:54:12.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.289+0000 7fd249400700 1 -- 172.21.15.89:0/2098688494 learned_addr learned my addr 172.21.15.89:0/2098688494 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:12.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.289+0000 7fd248a00700 1 --2- 172.21.15.89:0/2098688494 >> v2:172.21.15.113:3300/0 conn(0x7fd24415b5d0 0x7fd2440ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.290+0000 7fd243400700 1 -- 172.21.15.89:0/2098688494 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630512138 0 0) 0x7fd24415fd50 con 0x7fd2440ac7d0 2024-04-02T07:54:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.290+0000 7fd243400700 1 -- 172.21.15.89:0/2098688494 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd22c0035b0 con 0x7fd2440ac7d0 2024-04-02T07:54:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.290+0000 7fd248a00700 1 -- 172.21.15.89:0/2098688494 >> v1:172.21.15.113:6789/0 conn(0x7fd2440cb3f0 legacy=0x7fd2440ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.290+0000 7fd248a00700 1 -- 172.21.15.89:0/2098688494 >> v1:172.21.15.89:6789/0 conn(0x7fd2440ac7d0 legacy=0x7fd24406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.290+0000 7fd248a00700 1 -- 172.21.15.89:0/2098688494 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24415f420 con 0x7fd24415b5d0 2024-04-02T07:54:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.290+0000 7fd248a00700 1 --2- 172.21.15.89:0/2098688494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd24415b5d0 0x7fd2440ae3f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd238009840 tx=0x7fd238009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e80635963f3aff7e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.290+0000 7fd243400700 1 -- 172.21.15.89:0/2098688494 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd238011830 con 0x7fd24415b5d0 2024-04-02T07:54:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.290+0000 7fd243400700 1 -- 172.21.15.89:0/2098688494 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd238011990 con 0x7fd24415b5d0 2024-04-02T07:54:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.290+0000 7fd243400700 1 -- 172.21.15.89:0/2098688494 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd238011d20 con 0x7fd24415b5d0 2024-04-02T07:54:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.291+0000 7fd24b400700 1 -- 172.21.15.89:0/2098688494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd24415b5d0 msgr2=0x7fd2440ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.291+0000 7fd24b400700 1 --2- 172.21.15.89:0/2098688494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd24415b5d0 0x7fd2440ae3f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd238009840 tx=0x7fd238009b50 comp rx=0 tx=0).stop 2024-04-02T07:54:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.291+0000 7fd24b400700 1 -- 172.21.15.89:0/2098688494 shutdown_connections 2024-04-02T07:54:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.291+0000 7fd24b400700 1 --2- 172.21.15.89:0/2098688494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd24415b5d0 0x7fd2440ae3f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.291+0000 7fd24b400700 1 -- 172.21.15.89:0/2098688494 >> 172.21.15.89:0/2098688494 conn(0x7fd2440cf190 msgr2=0x7fd2440cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.291+0000 7fd24b400700 1 -- 172.21.15.89:0/2098688494 shutdown_connections 2024-04-02T07:54:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.291+0000 7fd24b400700 1 -- 172.21.15.89:0/2098688494 wait complete. 2024-04-02T07:54:12.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.293+0000 7fd24b400700 1 Processor -- start 2024-04-02T07:54:12.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.293+0000 7fd24b400700 1 -- start start 2024-04-02T07:54:12.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd24b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2440ac7d0 0x7fd244156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:12.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd24b400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd24415b5d0 0x7fd24415d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:12.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd24b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd24415d8a0 0x7fd244159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:12.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd24b400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24415aad0 con 0x7fd24415d8a0 2024-04-02T07:54:12.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd24b400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24415dd30 con 0x7fd24415b5d0 2024-04-02T07:54:12.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd24b400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2440cbf80 con 0x7fd2440ac7d0 2024-04-02T07:54:12.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd249400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd24415d8a0 0x7fd244159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:12.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd249400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd24415d8a0 0x7fd244159da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58528/0 (socket says 172.21.15.89:58528) 2024-04-02T07:54:12.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd249400700 1 -- 172.21.15.89:0/300273697 learned_addr learned my addr 172.21.15.89:0/300273697 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:12.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd248a00700 1 --2- 172.21.15.89:0/300273697 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2440ac7d0 0x7fd244156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:12.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd243e00700 1 --2- 172.21.15.89:0/300273697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd24415b5d0 0x7fd24415d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:12.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd249400700 1 -- 172.21.15.89:0/300273697 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2440ac7d0 msgr2=0x7fd244156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:12.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd249400700 1 --2- 172.21.15.89:0/300273697 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2440ac7d0 0x7fd244156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:12.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd249400700 1 -- 172.21.15.89:0/300273697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd24415b5d0 msgr2=0x7fd24415d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:12.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd249400700 1 --2- 172.21.15.89:0/300273697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd24415b5d0 0x7fd24415d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:12.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.294+0000 7fd249400700 1 -- 172.21.15.89:0/300273697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2380094a0 con 0x7fd24415d8a0 2024-04-02T07:54:12.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.295+0000 7fd249400700 1 --2- 172.21.15.89:0/300273697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd24415d8a0 0x7fd244159da0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00cb70 tx=0x7fd23c00ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:12.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.295+0000 7fd240c00700 1 -- 172.21.15.89:0/300273697 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c00aa60 con 0x7fd24415d8a0 2024-04-02T07:54:12.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.295+0000 7fd240c00700 1 -- 172.21.15.89:0/300273697 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c00abc0 con 0x7fd24415d8a0 2024-04-02T07:54:12.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.295+0000 7fd240c00700 1 -- 172.21.15.89:0/300273697 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c012400 con 0x7fd24415d8a0 2024-04-02T07:54:12.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.295+0000 7fd24b400700 1 -- 172.21.15.89:0/300273697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2440cc260 con 0x7fd24415d8a0 2024-04-02T07:54:12.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.295+0000 7fd24b400700 1 -- 172.21.15.89:0/300273697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2440cc820 con 0x7fd24415d8a0 2024-04-02T07:54:12.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.296+0000 7fd24b400700 1 -- 172.21.15.89:0/300273697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24415f9c0 con 0x7fd24415d8a0 2024-04-02T07:54:12.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.297+0000 7fd240c00700 1 -- 172.21.15.89:0/300273697 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fd23c01b3b0 con 0x7fd24415d8a0 2024-04-02T07:54:12.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.298+0000 7fd240c00700 1 --2- 172.21.15.89:0/300273697 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd22406aed0 0x7fd22406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:12.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.298+0000 7fd248a00700 1 -- 172.21.15.89:0/300273697 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd22406aed0 msgr2=0x7fd22406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:12.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.298+0000 7fd240c00700 1 -- 172.21.15.89:0/300273697 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd23c0897d0 con 0x7fd24415d8a0 2024-04-02T07:54:12.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.298+0000 7fd248a00700 1 --2- 172.21.15.89:0/300273697 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd22406aed0 0x7fd22406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:12.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.307+0000 7fd240c00700 1 -- 172.21.15.89:0/300273697 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd24415f9c0 con 0x7fd24415d8a0 2024-04-02T07:54:12.378 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:12.377+0000 7efd97554240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:54:12.477 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:12.475+0000 7efd97554240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:54:12.485 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:12.483+0000 7f85b0efb240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:54:12.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.499+0000 7fd248a00700 1 -- 172.21.15.89:0/300273697 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd22406aed0 msgr2=0x7fd22406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:12.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.499+0000 7fd248a00700 1 --2- 172.21.15.89:0/300273697 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd22406aed0 0x7fd22406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:12.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.561+0000 7fd24b400700 1 -- 172.21.15.89:0/300273697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd24415f9c0 con 0x7fd24415d8a0 2024-04-02T07:54:12.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.562+0000 7fd240c00700 1 -- 172.21.15.89:0/300273697 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fd24415f9c0 con 0x7fd24415d8a0 2024-04-02T07:54:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:12.564 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6317,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1566757639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1566757639},{"type":"v1","addr":"172.21.15.113:6835","nonce":1566757639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":6985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4102221782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4102221782},{"type":"v1","addr":"172.21.15.89:6836","nonce":4102221782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3127911401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3127911401},{"type":"v1","addr":"172.21.15.113:6834","nonce":3127911401}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:52:48.969376+0000","modified":"2024-04-02T07:52:51.980647+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7419":{"gid":7419,"name":"a","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2016280781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2016280781},{"type":"v1","addr":"172.21.15.89:6837","nonce":2016280781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-02T07:54:12.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.565+0000 7fd24b400700 1 -- 172.21.15.89:0/300273697 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd22406aed0 msgr2=0x7fd22406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:12.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.565+0000 7fd24b400700 1 --2- 172.21.15.89:0/300273697 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd22406aed0 0x7fd22406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:12.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.565+0000 7fd24b400700 1 -- 172.21.15.89:0/300273697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd24415d8a0 msgr2=0x7fd244159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:12.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.565+0000 7fd24b400700 1 --2- 172.21.15.89:0/300273697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd24415d8a0 0x7fd244159da0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00cb70 tx=0x7fd23c00ce80 comp rx=0 tx=0).stop 2024-04-02T07:54:12.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.565+0000 7fd24b400700 1 -- 172.21.15.89:0/300273697 shutdown_connections 2024-04-02T07:54:12.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.565+0000 7fd24b400700 1 --2- 172.21.15.89:0/300273697 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd22406aed0 0x7fd22406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:12.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.565+0000 7fd24b400700 1 --2- 172.21.15.89:0/300273697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd24415d8a0 0x7fd244159da0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:12.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.565+0000 7fd24b400700 1 --2- 172.21.15.89:0/300273697 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2440ac7d0 0x7fd244156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:12.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.565+0000 7fd24b400700 1 --2- 172.21.15.89:0/300273697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd24415b5d0 0x7fd24415d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:12.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.565+0000 7fd24b400700 1 -- 172.21.15.89:0/300273697 >> 172.21.15.89:0/300273697 conn(0x7fd2440cf190 msgr2=0x7fd24415cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:12.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.565+0000 7fd24b400700 1 -- 172.21.15.89:0/300273697 shutdown_connections 2024-04-02T07:54:12.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.565+0000 7fd24b400700 1 -- 172.21.15.89:0/300273697 wait complete. 2024-04-02T07:54:12.568 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 49 2024-04-02T07:54:12.574 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:12.573+0000 7efd97554240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:54:12.581 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:12.579+0000 7f85b0efb240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:54:12.585 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:54:12.672 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:12.671+0000 7efd97554240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:54:12.687 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:12.685+0000 7f85b0efb240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:54:12.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.779+0000 7fe08f600700 1 Processor -- start 2024-04-02T07:54:12.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.779+0000 7fe08f600700 1 -- start start 2024-04-02T07:54:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.780+0000 7fe08f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0880d05e0 0x7fe0880d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.780+0000 7fe08f600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe0880b6990 0x7fe0880b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.780+0000 7fe08f600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0880d0f80 con 0x7fe08815da50 2024-04-02T07:54:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.780+0000 7fe08f600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0880b95d0 con 0x7fe0880b6990 2024-04-02T07:54:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.780+0000 7fe08f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08815de30 con 0x7fe0880d05e0 2024-04-02T07:54:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.781+0000 7fe08d600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe0880b6990 0x7fe0880b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.781+0000 7fe08d600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe0880b6990 0x7fe0880b6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58542/0 (socket says 172.21.15.89:58542) 2024-04-02T07:54:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.781+0000 7fe08d600700 1 -- 172.21.15.89:0/1436442641 learned_addr learned my addr 172.21.15.89:0/1436442641 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.781+0000 7fe087e00700 1 --2- 172.21.15.89:0/1436442641 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0880d05e0 0x7fe0880d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.781+0000 7fe08d600700 1 -- 172.21.15.89:0/1436442641 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0880d05e0 msgr2=0x7fe0880d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.781+0000 7fe08d600700 1 --2- 172.21.15.89:0/1436442641 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0880d05e0 0x7fe0880d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.781+0000 7fe08d600700 1 -- 172.21.15.89:0/1436442641 >> v1:172.21.15.89:6789/0 conn(0x7fe08815da50 legacy=0x7fe0880ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.781+0000 7fe08d600700 1 -- 172.21.15.89:0/1436442641 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0880d24e0 con 0x7fe0880b6990 2024-04-02T07:54:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.781+0000 7fe08d600700 1 --2- 172.21.15.89:0/1436442641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0880b6990 0x7fe0880b6e10 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fe080009e50 tx=0x7fe08000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1538617ffc31066b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.781+0000 7fe087400700 1 -- 172.21.15.89:0/1436442641 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08000e070 con 0x7fe0880b6990 2024-04-02T07:54:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.782+0000 7fe087400700 1 -- 172.21.15.89:0/1436442641 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08000b910 con 0x7fe0880b6990 2024-04-02T07:54:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.782+0000 7fe087400700 1 -- 172.21.15.89:0/1436442641 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08000bcc0 con 0x7fe0880b6990 2024-04-02T07:54:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.782+0000 7fe08f600700 1 -- 172.21.15.89:0/1436442641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0880b6990 msgr2=0x7fe0880b6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.782+0000 7fe08f600700 1 --2- 172.21.15.89:0/1436442641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0880b6990 0x7fe0880b6e10 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fe080009e50 tx=0x7fe08000b330 comp rx=0 tx=0).stop 2024-04-02T07:54:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.782+0000 7fe08f600700 1 -- 172.21.15.89:0/1436442641 shutdown_connections 2024-04-02T07:54:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.782+0000 7fe08f600700 1 --2- 172.21.15.89:0/1436442641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0880b6990 0x7fe0880b6e10 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.782+0000 7fe08f600700 1 --2- 172.21.15.89:0/1436442641 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0880d05e0 0x7fe0880d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.782+0000 7fe08f600700 1 -- 172.21.15.89:0/1436442641 >> 172.21.15.89:0/1436442641 conn(0x7fe0880666a0 msgr2=0x7fe088066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.782+0000 7fe08f600700 1 -- 172.21.15.89:0/1436442641 shutdown_connections 2024-04-02T07:54:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.782+0000 7fe08f600700 1 -- 172.21.15.89:0/1436442641 wait complete. 2024-04-02T07:54:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.784+0000 7fe08f600700 1 Processor -- start 2024-04-02T07:54:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.784+0000 7fe08f600700 1 -- start start 2024-04-02T07:54:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.785+0000 7fe08f600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0880b6990 0x7fe0881f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.785+0000 7fe08f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0880d05e0 0x7fe0880b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.785+0000 7fe08f600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe08815da50 0x7fe088156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.785+0000 7fe08f600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0881565a0 con 0x7fe0880b6990 2024-04-02T07:54:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.785+0000 7fe08f600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe088156840 con 0x7fe08815da50 2024-04-02T07:54:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.785+0000 7fe08f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0881f31f0 con 0x7fe0880d05e0 2024-04-02T07:54:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.785+0000 7fe08cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0880b6990 0x7fe0881f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.785+0000 7fe08cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0880b6990 0x7fe0881f29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58552/0 (socket says 172.21.15.89:58552) 2024-04-02T07:54:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.785+0000 7fe08cc00700 1 -- 172.21.15.89:0/751634178 learned_addr learned my addr 172.21.15.89:0/751634178 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.785+0000 7fe087e00700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0880d05e0 0x7fe0880b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.785+0000 7fe08d600700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe08815da50 0x7fe088156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.786+0000 7fe08cc00700 1 -- 172.21.15.89:0/751634178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0880d05e0 msgr2=0x7fe0880b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.786+0000 7fe08cc00700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0880d05e0 0x7fe0880b4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.786+0000 7fe08cc00700 1 -- 172.21.15.89:0/751634178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe08815da50 msgr2=0x7fe088156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.786+0000 7fe08cc00700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe08815da50 0x7fe088156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.786+0000 7fe08cc00700 1 -- 172.21.15.89:0/751634178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe080009b30 con 0x7fe0880b6990 2024-04-02T07:54:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.786+0000 7fe087e00700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0880d05e0 0x7fe0880b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.786+0000 7fe08cc00700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0880b6990 0x7fe0881f29a0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00c450 tx=0x7fe07c00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.786+0000 7fe084c00700 1 -- 172.21.15.89:0/751634178 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c0052d0 con 0x7fe0880b6990 2024-04-02T07:54:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.786+0000 7fe084c00700 1 -- 172.21.15.89:0/751634178 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07c005430 con 0x7fe0880b6990 2024-04-02T07:54:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.786+0000 7fe084c00700 1 -- 172.21.15.89:0/751634178 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c0038c0 con 0x7fe0880b6990 2024-04-02T07:54:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.786+0000 7fe08f600700 1 -- 172.21.15.89:0/751634178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0881f34d0 con 0x7fe0880b6990 2024-04-02T07:54:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.786+0000 7fe08f600700 1 -- 172.21.15.89:0/751634178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0881f2ee0 con 0x7fe0880b6990 2024-04-02T07:54:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.788+0000 7fe08f600700 1 -- 172.21.15.89:0/751634178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe088156b20 con 0x7fe0880b6990 2024-04-02T07:54:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.789+0000 7fe084c00700 1 -- 172.21.15.89:0/751634178 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fe07c003a20 con 0x7fe0880b6990 2024-04-02T07:54:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.790+0000 7fe084c00700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe06806aed0 0x7fe06806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.790+0000 7fe084c00700 1 -- 172.21.15.89:0/751634178 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fe07c088c80 con 0x7fe0880b6990 2024-04-02T07:54:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.790+0000 7fe087e00700 1 -- 172.21.15.89:0/751634178 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe06806aed0 msgr2=0x7fe06806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.790+0000 7fe087e00700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe06806aed0 0x7fe06806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.795+0000 7fe084c00700 1 -- 172.21.15.89:0/751634178 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe07c053890 con 0x7fe0880b6990 2024-04-02T07:54:12.797 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:12.795+0000 7f85b0efb240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:54:12.886 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:12.885+0000 7efd97554240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:54:12.900 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:12.899+0000 7f85b0efb240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:54:12.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.991+0000 7fe087e00700 1 -- 172.21.15.89:0/751634178 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe06806aed0 msgr2=0x7fe06806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:12.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:12.991+0000 7fe087e00700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe06806aed0 0x7fe06806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:13.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.004+0000 7fe08f600700 1 -- 172.21.15.89:0/751634178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe088067720 con 0x7fe0880b6990 2024-04-02T07:54:13.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.006+0000 7fe084c00700 1 -- 172.21.15.89:0/751634178 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+17130 (secure 0 0 0) 0x7fe07c00ce60 con 0x7fe0880b6990 2024-04-02T07:54:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:13.008 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":112,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:53:05.735192+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:52:46.885836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-02T07:52:47.989359+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/1115793129":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/458120672":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/2657927819":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/383704566":"2024-04-03T07:53:05.735155+0000","172.21.15.89:6832/43357":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/1938634818":"2024-04-03T07:53:05.735155+0000","172.21.15.89:6833/43357":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/552112415":"2024-04-03T07:53:05.735155+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:54:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.008+0000 7fe08f600700 1 -- 172.21.15.89:0/751634178 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe06806aed0 msgr2=0x7fe06806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.008+0000 7fe08f600700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe06806aed0 0x7fe06806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.008+0000 7fe08f600700 1 -- 172.21.15.89:0/751634178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0880b6990 msgr2=0x7fe0881f29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.008+0000 7fe08f600700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0880b6990 0x7fe0881f29a0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00c450 tx=0x7fe07c00c760 comp rx=0 tx=0).stop 2024-04-02T07:54:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.009+0000 7fe08f600700 1 -- 172.21.15.89:0/751634178 shutdown_connections 2024-04-02T07:54:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.009+0000 7fe08f600700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe06806aed0 0x7fe06806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.009+0000 7fe08f600700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe08815da50 0x7fe088156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.009+0000 7fe08f600700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0880b6990 0x7fe0881f29a0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.009+0000 7fe08f600700 1 --2- 172.21.15.89:0/751634178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0880d05e0 0x7fe0880b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.009+0000 7fe08f600700 1 -- 172.21.15.89:0/751634178 >> 172.21.15.89:0/751634178 conn(0x7fe0880666a0 msgr2=0x7fe08815a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:13.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.009+0000 7fe08f600700 1 -- 172.21.15.89:0/751634178 shutdown_connections 2024-04-02T07:54:13.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.009+0000 7fe08f600700 1 -- 172.21.15.89:0/751634178 wait complete. 2024-04-02T07:54:13.030 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:52:46.885836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:52:47.989359+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:54:13.030 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:54:13.127 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:13.126+0000 7f85b0efb240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:54:13.142 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:13.141+0000 7efd97554240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:54:13.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.229+0000 7f496e200700 1 Processor -- start 2024-04-02T07:54:13.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.229+0000 7f496e200700 1 -- start start 2024-04-02T07:54:13.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.231+0000 7f496e200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4968160cc0 0x7f4968168730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:13.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.231+0000 7f496e200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496815f820 con 0x7f49680ced80 2024-04-02T07:54:13.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.231+0000 7f496e200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49680d2450 con 0x7f496815dad0 2024-04-02T07:54:13.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.231+0000 7f496e200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49680cfcc0 con 0x7f4968160cc0 2024-04-02T07:54:13.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.231+0000 7f4967e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4968160cc0 0x7f4968168730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:13.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.231+0000 7f4967e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4968160cc0 0x7f4968168730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58564/0 (socket says 172.21.15.89:58564) 2024-04-02T07:54:13.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.231+0000 7f4967e00700 1 -- 172.21.15.89:0/4121224265 learned_addr learned my addr 172.21.15.89:0/4121224265 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:13.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.231+0000 7f4967e00700 1 -- 172.21.15.89:0/4121224265 >> v1:172.21.15.113:6789/0 conn(0x7f496815dad0 legacy=0x7f49680d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:13.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.231+0000 7f4967e00700 1 -- 172.21.15.89:0/4121224265 >> v1:172.21.15.89:6789/0 conn(0x7f49680ced80 legacy=0x7f49680d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:13.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.231+0000 7f4966a00700 1 -- 172.21.15.89:0/4121224265 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451589188 0 0) 0x7f49680d2450 con 0x7f496815dad0 2024-04-02T07:54:13.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.231+0000 7f4967e00700 1 -- 172.21.15.89:0/4121224265 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49680cf420 con 0x7f4968160cc0 2024-04-02T07:54:13.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.232+0000 7f4967e00700 1 --2- 172.21.15.89:0/4121224265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968160cc0 0x7f4968168730 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f4960009e50 tx=0x7f496000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e17b0bb5282159 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:13.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.232+0000 7f4966a00700 1 -- 172.21.15.89:0/4121224265 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496000e070 con 0x7f4968160cc0 2024-04-02T07:54:13.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.232+0000 7f4966a00700 1 -- 172.21.15.89:0/4121224265 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f496000b9c0 con 0x7f4968160cc0 2024-04-02T07:54:13.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.232+0000 7f4966a00700 1 -- 172.21.15.89:0/4121224265 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496000bd70 con 0x7f4968160cc0 2024-04-02T07:54:13.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.233+0000 7f496e200700 1 -- 172.21.15.89:0/4121224265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968160cc0 msgr2=0x7f4968168730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:13.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.233+0000 7f496e200700 1 --2- 172.21.15.89:0/4121224265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968160cc0 0x7f4968168730 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f4960009e50 tx=0x7f496000b330 comp rx=0 tx=0).stop 2024-04-02T07:54:13.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.233+0000 7f496e200700 1 -- 172.21.15.89:0/4121224265 shutdown_connections 2024-04-02T07:54:13.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.233+0000 7f496e200700 1 --2- 172.21.15.89:0/4121224265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968160cc0 0x7f4968168730 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.233+0000 7f496e200700 1 -- 172.21.15.89:0/4121224265 >> 172.21.15.89:0/4121224265 conn(0x7f496808aff0 msgr2=0x7f49680d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:13.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.233+0000 7f496e200700 1 -- 172.21.15.89:0/4121224265 shutdown_connections 2024-04-02T07:54:13.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.233+0000 7f496e200700 1 -- 172.21.15.89:0/4121224265 wait complete. 2024-04-02T07:54:13.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.235+0000 7f496e200700 1 Processor -- start 2024-04-02T07:54:13.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.235+0000 7f496e200700 1 -- start start 2024-04-02T07:54:13.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f496e200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49680ced80 0x7f49680cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:13.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f496e200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f496815dad0 0x7f49680b6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:13.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f496e200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968160cc0 0x7f49680cff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:13.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f496e200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49680c8fa0 con 0x7f4968160cc0 2024-04-02T07:54:13.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f496e200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49680c8660 con 0x7f496815dad0 2024-04-02T07:54:13.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f496e200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49680b9010 con 0x7f49680ced80 2024-04-02T07:54:13.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f4967e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968160cc0 0x7f49680cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:13.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f4967e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968160cc0 0x7f49680cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58576/0 (socket says 172.21.15.89:58576) 2024-04-02T07:54:13.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f4967e00700 1 -- 172.21.15.89:0/604850483 learned_addr learned my addr 172.21.15.89:0/604850483 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:13.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f495fe00700 1 --2- 172.21.15.89:0/604850483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f496815dad0 0x7f49680b6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:13.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f4967400700 1 --2- 172.21.15.89:0/604850483 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49680ced80 0x7f49680cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:13.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f4967e00700 1 -- 172.21.15.89:0/604850483 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49680ced80 msgr2=0x7f49680cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:13.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f4967e00700 1 --2- 172.21.15.89:0/604850483 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49680ced80 0x7f49680cc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f4967e00700 1 -- 172.21.15.89:0/604850483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f496815dad0 msgr2=0x7f49680b6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:13.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f4967e00700 1 --2- 172.21.15.89:0/604850483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f496815dad0 0x7f49680b6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.236+0000 7f4967e00700 1 -- 172.21.15.89:0/604850483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4960009b30 con 0x7f4968160cc0 2024-04-02T07:54:13.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.237+0000 7f4967e00700 1 --2- 172.21.15.89:0/604850483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968160cc0 0x7f49680cff30 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f496000b730 tx=0x7f4960004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:13.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.237+0000 7f495f400700 1 -- 172.21.15.89:0/604850483 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496001b630 con 0x7f4968160cc0 2024-04-02T07:54:13.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.237+0000 7f495f400700 1 -- 172.21.15.89:0/604850483 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f496001b7b0 con 0x7f4968160cc0 2024-04-02T07:54:13.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.237+0000 7f495f400700 1 -- 172.21.15.89:0/604850483 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496001bb60 con 0x7f4968160cc0 2024-04-02T07:54:13.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.237+0000 7f496e200700 1 -- 172.21.15.89:0/604850483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49680c88e0 con 0x7f4968160cc0 2024-04-02T07:54:13.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.237+0000 7f496e200700 1 -- 172.21.15.89:0/604850483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49680cc640 con 0x7f4968160cc0 2024-04-02T07:54:13.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.238+0000 7f496e200700 1 -- 172.21.15.89:0/604850483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49680c90e0 con 0x7f4968160cc0 2024-04-02T07:54:13.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.240+0000 7f495f400700 1 -- 172.21.15.89:0/604850483 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f496001bcc0 con 0x7f4968160cc0 2024-04-02T07:54:13.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.241+0000 7f495f400700 1 --2- 172.21.15.89:0/604850483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f494406aed0 0x7f494406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:13.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.241+0000 7f495f400700 1 -- 172.21.15.89:0/604850483 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f496008b090 con 0x7f4968160cc0 2024-04-02T07:54:13.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.241+0000 7f4967400700 1 -- 172.21.15.89:0/604850483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f494406aed0 msgr2=0x7f494406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:13.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.242+0000 7f4967400700 1 --2- 172.21.15.89:0/604850483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f494406aed0 0x7f494406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:13.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.247+0000 7f495f400700 1 -- 172.21.15.89:0/604850483 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f49680c90e0 con 0x7f4968160cc0 2024-04-02T07:54:13.334 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:13.332+0000 7efd97554240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:54:13.398 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:13.397+0000 7f85b0efb240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:54:13.426 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:13.425+0000 7efd97554240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:54:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.443+0000 7f4967400700 1 -- 172.21.15.89:0/604850483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f494406aed0 msgr2=0x7f494406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.443+0000 7f4967400700 1 --2- 172.21.15.89:0/604850483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f494406aed0 0x7f494406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:13.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.458+0000 7f496e200700 1 -- 172.21.15.89:0/604850483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f49680c90e0 con 0x7f4968160cc0 2024-04-02T07:54:13.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.458+0000 7f495f400700 1 -- 172.21.15.89:0/604850483 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7f49680c90e0 con 0x7f4968160cc0 2024-04-02T07:54:13.460 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:54:13.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.461+0000 7f496e200700 1 -- 172.21.15.89:0/604850483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f494406aed0 msgr2=0x7f494406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:13.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.461+0000 7f496e200700 1 --2- 172.21.15.89:0/604850483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f494406aed0 0x7f494406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.461+0000 7f496e200700 1 -- 172.21.15.89:0/604850483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968160cc0 msgr2=0x7f49680cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:13.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.461+0000 7f496e200700 1 --2- 172.21.15.89:0/604850483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968160cc0 0x7f49680cff30 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f496000b730 tx=0x7f4960004ad0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.461+0000 7f496e200700 1 -- 172.21.15.89:0/604850483 shutdown_connections 2024-04-02T07:54:13.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.461+0000 7f496e200700 1 --2- 172.21.15.89:0/604850483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f494406aed0 0x7f494406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.461+0000 7f496e200700 1 --2- 172.21.15.89:0/604850483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968160cc0 0x7f49680cff30 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.461+0000 7f496e200700 1 --2- 172.21.15.89:0/604850483 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49680ced80 0x7f49680cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.461+0000 7f496e200700 1 --2- 172.21.15.89:0/604850483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f496815dad0 0x7f49680b6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.461+0000 7f496e200700 1 -- 172.21.15.89:0/604850483 >> 172.21.15.89:0/604850483 conn(0x7f496808aff0 msgr2=0x7f496806d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:13.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.461+0000 7f496e200700 1 -- 172.21.15.89:0/604850483 shutdown_connections 2024-04-02T07:54:13.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.461+0000 7f496e200700 1 -- 172.21.15.89:0/604850483 wait complete. 2024-04-02T07:54:13.481 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:54:13.603 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:13.602+0000 7f85b0efb240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:54:13.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.678+0000 7f02a1200700 1 Processor -- start 2024-04-02T07:54:13.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.679+0000 7f02a1200700 1 -- start start 2024-04-02T07:54:13.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.680+0000 7f02a1200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f029c0cbb30 0x7f029c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:13.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.680+0000 7f02a1200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f029c0cd020 0x7f029c0cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:13.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.680+0000 7f02a1200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029c0cc450 con 0x7f029c0b2070 2024-04-02T07:54:13.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.680+0000 7f02a1200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029c0cdae0 con 0x7f029c0cd020 2024-04-02T07:54:13.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.680+0000 7f02a1200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0cddc0 con 0x7f029c0cbb30 2024-04-02T07:54:13.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.680+0000 7f0299400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f029c0cd020 0x7f029c0cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:13.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.680+0000 7f0299400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f029c0cd020 0x7f029c0cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58588/0 (socket says 172.21.15.89:58588) 2024-04-02T07:54:13.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.680+0000 7f0299400700 1 -- 172.21.15.89:0/3242956242 learned_addr learned my addr 172.21.15.89:0/3242956242 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:13.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.680+0000 7f0299e00700 1 --2- 172.21.15.89:0/3242956242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f029c0cbb30 0x7f029c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:13.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.680+0000 7f0299400700 1 -- 172.21.15.89:0/3242956242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f029c0cbb30 msgr2=0x7f029c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:13.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.680+0000 7f0299400700 1 --2- 172.21.15.89:0/3242956242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f029c0cbb30 0x7f029c0cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.680+0000 7f0299400700 1 -- 172.21.15.89:0/3242956242 >> v1:172.21.15.89:6789/0 conn(0x7f029c0b2070 legacy=0x7f029c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:13.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.680+0000 7f0299400700 1 -- 172.21.15.89:0/3242956242 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c0ce0a0 con 0x7f029c0cd020 2024-04-02T07:54:13.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.680+0000 7f0299e00700 1 --2- 172.21.15.89:0/3242956242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f029c0cbb30 0x7f029c0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:13.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.681+0000 7f0299400700 1 --2- 172.21.15.89:0/3242956242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f029c0cd020 0x7f029c0cd4a0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f02880098f0 tx=0x7f0288009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbad19d314442376 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:13.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.681+0000 7f0298a00700 1 -- 172.21.15.89:0/3242956242 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028800b8a0 con 0x7f029c0cd020 2024-04-02T07:54:13.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.681+0000 7f0298a00700 1 -- 172.21.15.89:0/3242956242 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028800ba00 con 0x7f029c0cd020 2024-04-02T07:54:13.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.681+0000 7f0298a00700 1 -- 172.21.15.89:0/3242956242 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028800bd90 con 0x7f029c0cd020 2024-04-02T07:54:13.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.681+0000 7f02a1200700 1 -- 172.21.15.89:0/3242956242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f029c0cd020 msgr2=0x7f029c0cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:13.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.681+0000 7f02a1200700 1 --2- 172.21.15.89:0/3242956242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f029c0cd020 0x7f029c0cd4a0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f02880098f0 tx=0x7f0288009c00 comp rx=0 tx=0).stop 2024-04-02T07:54:13.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.682+0000 7f02a1200700 1 -- 172.21.15.89:0/3242956242 shutdown_connections 2024-04-02T07:54:13.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.682+0000 7f02a1200700 1 --2- 172.21.15.89:0/3242956242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f029c0cbb30 0x7f029c0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.682+0000 7f02a1200700 1 --2- 172.21.15.89:0/3242956242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f029c0cd020 0x7f029c0cd4a0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.682+0000 7f02a1200700 1 -- 172.21.15.89:0/3242956242 >> 172.21.15.89:0/3242956242 conn(0x7f029c0b0d70 msgr2=0x7f029c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:13.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.682+0000 7f02a1200700 1 -- 172.21.15.89:0/3242956242 shutdown_connections 2024-04-02T07:54:13.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.682+0000 7f02a1200700 1 -- 172.21.15.89:0/3242956242 wait complete. 2024-04-02T07:54:13.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.683+0000 7f02a1200700 1 Processor -- start 2024-04-02T07:54:13.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.684+0000 7f02a1200700 1 -- start start 2024-04-02T07:54:13.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.684+0000 7f02a1200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f029c0b2070 0x7f029c071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:13.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.684+0000 7f02a1200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f029c0cbb30 0x7f029c15de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:13.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.684+0000 7f02a1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f029c0cd020 0x7f029c14b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:13.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.684+0000 7f02a1200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0ac050 con 0x7f029c0cd020 2024-04-02T07:54:13.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.684+0000 7f02a1200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0721e0 con 0x7f029c0b2070 2024-04-02T07:54:13.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.684+0000 7f02a1200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0ca7e0 con 0x7f029c0cbb30 2024-04-02T07:54:13.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.685+0000 7f0299400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f029c0cbb30 0x7f029c15de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:13.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.685+0000 7f029a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f029c0cd020 0x7f029c14b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:13.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.685+0000 7f0299400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f029c0cbb30 0x7f029c15de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:40610/0 (socket says 172.21.15.89:40610) 2024-04-02T07:54:13.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.685+0000 7f0299e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f029c0b2070 0x7f029c071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:13.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.685+0000 7f0299400700 1 -- 172.21.15.89:0/120176561 learned_addr learned my addr 172.21.15.89:0/120176561 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:13.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.685+0000 7f0299400700 1 -- 172.21.15.89:0/120176561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f029c0b2070 msgr2=0x7f029c071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:13.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.685+0000 7f0299400700 1 --2- 172.21.15.89:0/120176561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f029c0b2070 0x7f029c071ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.685+0000 7f0299400700 1 -- 172.21.15.89:0/120176561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f029c0cd020 msgr2=0x7f029c14b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:13.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.685+0000 7f0299400700 1 --2- 172.21.15.89:0/120176561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f029c0cd020 0x7f029c14b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.685+0000 7f0299400700 1 -- 172.21.15.89:0/120176561 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0288009550 con 0x7f029c0cbb30 2024-04-02T07:54:13.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.685+0000 7f0299e00700 1 --2- 172.21.15.89:0/120176561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f029c0b2070 0x7f029c071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:13.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.686+0000 7f0299400700 1 --2- 172.21.15.89:0/120176561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f029c0cbb30 0x7f029c15de70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0288009c40 tx=0x7f0288004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:13.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.686+0000 7f0292000700 1 -- 172.21.15.89:0/120176561 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0288004510 con 0x7f029c0cbb30 2024-04-02T07:54:13.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.686+0000 7f0292000700 1 -- 172.21.15.89:0/120176561 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028801e070 con 0x7f029c0cbb30 2024-04-02T07:54:13.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.686+0000 7f02a1200700 1 -- 172.21.15.89:0/120176561 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029c0caac0 con 0x7f029c0cbb30 2024-04-02T07:54:13.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.686+0000 7f0292000700 1 -- 172.21.15.89:0/120176561 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02880038e0 con 0x7f029c0cbb30 2024-04-02T07:54:13.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.686+0000 7f02a1200700 1 -- 172.21.15.89:0/120176561 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029c0cb080 con 0x7f029c0cbb30 2024-04-02T07:54:13.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.687+0000 7f02a1200700 1 -- 172.21.15.89:0/120176561 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029c0ac1c0 con 0x7f029c0cbb30 2024-04-02T07:54:13.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.689+0000 7f0292000700 1 -- 172.21.15.89:0/120176561 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f0288003a40 con 0x7f029c0cbb30 2024-04-02T07:54:13.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.690+0000 7f0292000700 1 --2- 172.21.15.89:0/120176561 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f027c06aed0 0x7f027c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:13.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.690+0000 7f0292000700 1 -- 172.21.15.89:0/120176561 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f0288020070 con 0x7f029c0cbb30 2024-04-02T07:54:13.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.690+0000 7f0299e00700 1 -- 172.21.15.89:0/120176561 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f027c06aed0 msgr2=0x7f027c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:13.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.690+0000 7f0299e00700 1 --2- 172.21.15.89:0/120176561 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f027c06aed0 0x7f027c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:13.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.695+0000 7f0292000700 1 -- 172.21.15.89:0/120176561 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0288055970 con 0x7f029c0cbb30 2024-04-02T07:54:13.704 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:13.703+0000 7f85b0efb240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:54:13.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.891+0000 7f0299e00700 1 -- 172.21.15.89:0/120176561 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f027c06aed0 msgr2=0x7f027c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:13.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.891+0000 7f0299e00700 1 --2- 172.21.15.89:0/120176561 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f027c06aed0 0x7f027c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:13.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.903+0000 7f02a1200700 1 -- 172.21.15.89:0/120176561 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f029c15bda0 con 0x7f029c0cbb30 2024-04-02T07:54:13.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.904+0000 7f0292000700 1 -- 172.21.15.89:0/120176561 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v112) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0288058f90 con 0x7f029c0cbb30 2024-04-02T07:54:13.906 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:54:13.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.906+0000 7f02a1200700 1 -- 172.21.15.89:0/120176561 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f027c06aed0 msgr2=0x7f027c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:13.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.906+0000 7f02a1200700 1 --2- 172.21.15.89:0/120176561 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f027c06aed0 0x7f027c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.906+0000 7f02a1200700 1 -- 172.21.15.89:0/120176561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f029c0cbb30 msgr2=0x7f029c15de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:13.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.906+0000 7f02a1200700 1 --2- 172.21.15.89:0/120176561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f029c0cbb30 0x7f029c15de70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0288009c40 tx=0x7f0288004750 comp rx=0 tx=0).stop 2024-04-02T07:54:13.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.906+0000 7f02a1200700 1 -- 172.21.15.89:0/120176561 shutdown_connections 2024-04-02T07:54:13.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.906+0000 7f02a1200700 1 --2- 172.21.15.89:0/120176561 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f027c06aed0 0x7f027c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.906+0000 7f02a1200700 1 --2- 172.21.15.89:0/120176561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f029c0b2070 0x7f029c071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.906+0000 7f02a1200700 1 --2- 172.21.15.89:0/120176561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f029c0cd020 0x7f029c14b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.906+0000 7f02a1200700 1 --2- 172.21.15.89:0/120176561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f029c0cbb30 0x7f029c15de70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:13.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.906+0000 7f02a1200700 1 -- 172.21.15.89:0/120176561 >> 172.21.15.89:0/120176561 conn(0x7f029c0b0d70 msgr2=0x7f029c0afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:13.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.906+0000 7f02a1200700 1 -- 172.21.15.89:0/120176561 shutdown_connections 2024-04-02T07:54:13.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:13.907+0000 7f02a1200700 1 -- 172.21.15.89:0/120176561 wait complete. 2024-04-02T07:54:13.926 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:54:14.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.129+0000 7f6bf1000700 1 Processor -- start 2024-04-02T07:54:14.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.130+0000 7f6bf1000700 1 -- start start 2024-04-02T07:54:14.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.131+0000 7f6bf1000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6bec0cbb30 0x7f6bec0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:14.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.131+0000 7f6bf1000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec0cd480 0x7f6bec06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:14.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.131+0000 7f6bf1000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bec0cc450 con 0x7f6bec0cd020 2024-04-02T07:54:14.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.131+0000 7f6bf1000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bec0cdae0 con 0x7f6bec0cbb30 2024-04-02T07:54:14.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.131+0000 7f6bf1000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec0cddc0 con 0x7f6bec0cd480 2024-04-02T07:54:14.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.131+0000 7f6be9e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6bec0cbb30 0x7f6bec0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:14.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.132+0000 7f6be3e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f6bec0cd020 0x7f6bec0b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:52940/0 (socket says 172.21.15.89:52940) 2024-04-02T07:54:14.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.132+0000 7f6be9e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6bec0cbb30 0x7f6bec0cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:56258/0 (socket says 172.21.15.89:56258) 2024-04-02T07:54:14.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.132+0000 7f6be3e00700 1 -- 172.21.15.89:0/340144567 learned_addr learned my addr 172.21.15.89:0/340144567 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:14.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.132+0000 7f6bea800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec0cd480 0x7f6bec06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:14.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.132+0000 7f6be9e00700 1 -- 172.21.15.89:0/340144567 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec0cd480 msgr2=0x7f6bec06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:14.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.132+0000 7f6be9e00700 1 --2- 172.21.15.89:0/340144567 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec0cd480 0x7f6bec06e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.132+0000 7f6be9e00700 1 -- 172.21.15.89:0/340144567 >> v1:172.21.15.113:6789/0 conn(0x7f6bec0cd020 legacy=0x7f6bec0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:14.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.132+0000 7f6be9e00700 1 -- 172.21.15.89:0/340144567 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bec0ce0a0 con 0x7f6bec0cbb30 2024-04-02T07:54:14.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.132+0000 7f6bea800700 1 --2- 172.21.15.89:0/340144567 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec0cd480 0x7f6bec06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:14.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.132+0000 7f6be9e00700 1 --2- 172.21.15.89:0/340144567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bec0cbb30 0x7f6bec0cbf10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009840 tx=0x7f6bdc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=365e8db019d6bff1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:14.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.135+0000 7f6be9400700 1 -- 172.21.15.89:0/340144567 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc011830 con 0x7f6bec0cbb30 2024-04-02T07:54:14.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.135+0000 7f6be9400700 1 -- 172.21.15.89:0/340144567 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bdc011990 con 0x7f6bec0cbb30 2024-04-02T07:54:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.135+0000 7f6be9400700 1 -- 172.21.15.89:0/340144567 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc011d20 con 0x7f6bec0cbb30 2024-04-02T07:54:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.135+0000 7f6bf1000700 1 -- 172.21.15.89:0/340144567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bec0cbb30 msgr2=0x7f6bec0cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.135+0000 7f6bf1000700 1 --2- 172.21.15.89:0/340144567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bec0cbb30 0x7f6bec0cbf10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009840 tx=0x7f6bdc009b50 comp rx=0 tx=0).stop 2024-04-02T07:54:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.135+0000 7f6bf1000700 1 -- 172.21.15.89:0/340144567 shutdown_connections 2024-04-02T07:54:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.135+0000 7f6bf1000700 1 --2- 172.21.15.89:0/340144567 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec0cd480 0x7f6bec06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.135+0000 7f6bf1000700 1 --2- 172.21.15.89:0/340144567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bec0cbb30 0x7f6bec0cbf10 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.135+0000 7f6bf1000700 1 -- 172.21.15.89:0/340144567 >> 172.21.15.89:0/340144567 conn(0x7f6bec0b0d70 msgr2=0x7f6bec0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.136+0000 7f6bf1000700 1 -- 172.21.15.89:0/340144567 shutdown_connections 2024-04-02T07:54:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.136+0000 7f6bf1000700 1 -- 172.21.15.89:0/340144567 wait complete. 2024-04-02T07:54:14.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.137+0000 7f6bf1000700 1 Processor -- start 2024-04-02T07:54:14.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.137+0000 7f6bf1000700 1 -- start start 2024-04-02T07:54:14.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.138+0000 7f6bf1000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec0cbb30 0x7f6bec153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:14.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.138+0000 7f6bf1000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec0cd020 0x7f6bec1484c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:14.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.138+0000 7f6bf1000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bec0cd480 0x7f6bec0cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:14.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.138+0000 7f6bf1000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec0718d0 con 0x7f6bec0cbb30 2024-04-02T07:54:14.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.138+0000 7f6bf1000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec070c90 con 0x7f6bec0cd480 2024-04-02T07:54:14.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.138+0000 7f6bf1000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec070f30 con 0x7f6bec0cd020 2024-04-02T07:54:14.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.139+0000 7f6be9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec0cbb30 0x7f6bec153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:14.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.139+0000 7f6be9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec0cbb30 0x7f6bec153fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58606/0 (socket says 172.21.15.89:58606) 2024-04-02T07:54:14.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.139+0000 7f6be9e00700 1 -- 172.21.15.89:0/3914019117 learned_addr learned my addr 172.21.15.89:0/3914019117 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:14.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.139+0000 7f6bea800700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bec0cd480 0x7f6bec0cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:14.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.139+0000 7f6be3e00700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec0cd020 0x7f6bec1484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:14.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.139+0000 7f6be9e00700 1 -- 172.21.15.89:0/3914019117 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec0cd020 msgr2=0x7f6bec1484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:14.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.139+0000 7f6be9e00700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec0cd020 0x7f6bec1484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.139+0000 7f6be9e00700 1 -- 172.21.15.89:0/3914019117 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bec0cd480 msgr2=0x7f6bec0cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:14.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.139+0000 7f6be9e00700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bec0cd480 0x7f6bec0cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.139+0000 7f6be9e00700 1 -- 172.21.15.89:0/3914019117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be4009a70 con 0x7f6bec0cbb30 2024-04-02T07:54:14.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.139+0000 7f6be3e00700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec0cd020 0x7f6bec1484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:14.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.139+0000 7f6be9e00700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec0cbb30 0x7f6bec153fd0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009fd0 tx=0x7f6bdc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:14.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.140+0000 7f6be2000700 1 -- 172.21.15.89:0/3914019117 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc0036a0 con 0x7f6bec0cbb30 2024-04-02T07:54:14.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.140+0000 7f6be2000700 1 -- 172.21.15.89:0/3914019117 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bdc01e070 con 0x7f6bec0cbb30 2024-04-02T07:54:14.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.140+0000 7f6bf1000700 1 -- 172.21.15.89:0/3914019117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bdc0094a0 con 0x7f6bec0cbb30 2024-04-02T07:54:14.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.140+0000 7f6be2000700 1 -- 172.21.15.89:0/3914019117 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc003a60 con 0x7f6bec0cbb30 2024-04-02T07:54:14.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.140+0000 7f6bf1000700 1 -- 172.21.15.89:0/3914019117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bec0714f0 con 0x7f6bec0cbb30 2024-04-02T07:54:14.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.141+0000 7f6bf1000700 1 -- 172.21.15.89:0/3914019117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bec067720 con 0x7f6bec0cbb30 2024-04-02T07:54:14.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.142+0000 7f6be2000700 1 -- 172.21.15.89:0/3914019117 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f6bdc003bc0 con 0x7f6bec0cbb30 2024-04-02T07:54:14.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.143+0000 7f6be2000700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6bc806aed0 0x7f6bc806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:14.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.143+0000 7f6be2000700 1 -- 172.21.15.89:0/3914019117 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f6bdc08ac60 con 0x7f6bec0cbb30 2024-04-02T07:54:14.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.143+0000 7f6be3e00700 1 -- 172.21.15.89:0/3914019117 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6bc806aed0 msgr2=0x7f6bc806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:14.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.143+0000 7f6be3e00700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6bc806aed0 0x7f6bc806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.147+0000 7f6be2000700 1 -- 172.21.15.89:0/3914019117 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6bdc055990 con 0x7f6bec0cbb30 2024-04-02T07:54:14.154 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:14.153+0000 7efd97554240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:54:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.344+0000 7f6be3e00700 1 -- 172.21.15.89:0/3914019117 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6bc806aed0 msgr2=0x7f6bc806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.344+0000 7f6be3e00700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6bc806aed0 0x7f6bc806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.356+0000 7f6bf1000700 1 -- 172.21.15.89:0/3914019117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6bec15bda0 con 0x7f6bec0cbb30 2024-04-02T07:54:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.356+0000 7f6be2000700 1 -- 172.21.15.89:0/3914019117 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v112) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6bdc058fb0 con 0x7f6bec0cbb30 2024-04-02T07:54:14.358 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:54:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.359+0000 7f6bf1000700 1 -- 172.21.15.89:0/3914019117 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6bc806aed0 msgr2=0x7f6bc806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.359+0000 7f6bf1000700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6bc806aed0 0x7f6bc806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.359+0000 7f6bf1000700 1 -- 172.21.15.89:0/3914019117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec0cbb30 msgr2=0x7f6bec153fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:14.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.359+0000 7f6bf1000700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec0cbb30 0x7f6bec153fd0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009fd0 tx=0x7f6bdc004750 comp rx=0 tx=0).stop 2024-04-02T07:54:14.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.359+0000 7f6bf1000700 1 -- 172.21.15.89:0/3914019117 shutdown_connections 2024-04-02T07:54:14.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.359+0000 7f6bf1000700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6bc806aed0 0x7f6bc806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.359+0000 7f6bf1000700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bec0cd480 0x7f6bec0cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.359+0000 7f6bf1000700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec0cbb30 0x7f6bec153fd0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.359+0000 7f6bf1000700 1 --2- 172.21.15.89:0/3914019117 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec0cd020 0x7f6bec1484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.359+0000 7f6bf1000700 1 -- 172.21.15.89:0/3914019117 >> 172.21.15.89:0/3914019117 conn(0x7f6bec0b0d70 msgr2=0x7f6bec06e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:14.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.359+0000 7f6bf1000700 1 -- 172.21.15.89:0/3914019117 shutdown_connections 2024-04-02T07:54:14.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.360+0000 7f6bf1000700 1 -- 172.21.15.89:0/3914019117 wait complete. 2024-04-02T07:54:14.366 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:14.365+0000 7efd97554240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:54:14.382 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:54:14.489 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:14.487+0000 7f85b0efb240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:54:14.573 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:14.572+0000 7efd97554240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:54:14.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.578+0000 7ff244c00700 1 Processor -- start 2024-04-02T07:54:14.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.579+0000 7ff244c00700 1 -- start start 2024-04-02T07:54:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.580+0000 7ff244c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff24015b5d0 0x7ff2400ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.580+0000 7ff244c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2400cb3f0 0x7ff2400ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.580+0000 7ff244c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24015fd50 con 0x7ff2400ae930 2024-04-02T07:54:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.580+0000 7ff244c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff24015b9b0 con 0x7ff24015b5d0 2024-04-02T07:54:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.580+0000 7ff244c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24015f140 con 0x7ff2400cb3f0 2024-04-02T07:54:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.580+0000 7ff23de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff24015b5d0 0x7ff2400ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.580+0000 7ff23de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff24015b5d0 0x7ff2400ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58616/0 (socket says 172.21.15.89:58616) 2024-04-02T07:54:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.580+0000 7ff23de00700 1 -- 172.21.15.89:0/2553177624 learned_addr learned my addr 172.21.15.89:0/2553177624 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.580+0000 7ff23d400700 1 --2- 172.21.15.89:0/2553177624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2400cb3f0 0x7ff2400ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:14.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.581+0000 7ff23de00700 1 -- 172.21.15.89:0/2553177624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2400cb3f0 msgr2=0x7ff2400ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:14.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.581+0000 7ff23de00700 1 --2- 172.21.15.89:0/2553177624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2400cb3f0 0x7ff2400ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.581+0000 7ff23de00700 1 -- 172.21.15.89:0/2553177624 >> v1:172.21.15.89:6789/0 conn(0x7ff2400ae930 legacy=0x7ff240140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:14.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.581+0000 7ff23de00700 1 -- 172.21.15.89:0/2553177624 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24015f420 con 0x7ff24015b5d0 2024-04-02T07:54:14.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.581+0000 7ff23d400700 1 --2- 172.21.15.89:0/2553177624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2400cb3f0 0x7ff2400ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:14.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.581+0000 7ff23de00700 1 --2- 172.21.15.89:0/2553177624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24015b5d0 0x7ff2400ae3f0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7ff2300098f0 tx=0x7ff230009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee210fc7c0a6f68 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:14.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.581+0000 7ff23ca00700 1 -- 172.21.15.89:0/2553177624 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff230011830 con 0x7ff24015b5d0 2024-04-02T07:54:14.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.581+0000 7ff23ca00700 1 -- 172.21.15.89:0/2553177624 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff230011990 con 0x7ff24015b5d0 2024-04-02T07:54:14.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.581+0000 7ff23ca00700 1 -- 172.21.15.89:0/2553177624 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff230011d20 con 0x7ff24015b5d0 2024-04-02T07:54:14.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.582+0000 7ff244c00700 1 -- 172.21.15.89:0/2553177624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24015b5d0 msgr2=0x7ff2400ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:14.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.582+0000 7ff244c00700 1 --2- 172.21.15.89:0/2553177624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24015b5d0 0x7ff2400ae3f0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7ff2300098f0 tx=0x7ff230009c00 comp rx=0 tx=0).stop 2024-04-02T07:54:14.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.582+0000 7ff244c00700 1 -- 172.21.15.89:0/2553177624 shutdown_connections 2024-04-02T07:54:14.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.582+0000 7ff244c00700 1 --2- 172.21.15.89:0/2553177624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24015b5d0 0x7ff2400ae3f0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.582+0000 7ff244c00700 1 --2- 172.21.15.89:0/2553177624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2400cb3f0 0x7ff2400ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.582+0000 7ff244c00700 1 -- 172.21.15.89:0/2553177624 >> 172.21.15.89:0/2553177624 conn(0x7ff2400cf190 msgr2=0x7ff2400cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:14.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.582+0000 7ff244c00700 1 -- 172.21.15.89:0/2553177624 shutdown_connections 2024-04-02T07:54:14.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.582+0000 7ff244c00700 1 -- 172.21.15.89:0/2553177624 wait complete. 2024-04-02T07:54:14.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.584+0000 7ff244c00700 1 Processor -- start 2024-04-02T07:54:14.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.584+0000 7ff244c00700 1 -- start start 2024-04-02T07:54:14.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff244c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2400ae930 0x7ff24014dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:14.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff244c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24015b5d0 0x7ff2400ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:14.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff244c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff240155c90 0x7ff240156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:14.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff244c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff240156650 con 0x7ff24015b5d0 2024-04-02T07:54:14.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff244c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2401f65d0 con 0x7ff240155c90 2024-04-02T07:54:14.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff244c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2401f68b0 con 0x7ff2400ae930 2024-04-02T07:54:14.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff23d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24015b5d0 0x7ff2400ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:14.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff23d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24015b5d0 0x7ff2400ce250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58632/0 (socket says 172.21.15.89:58632) 2024-04-02T07:54:14.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff23d400700 1 -- 172.21.15.89:0/1915578732 learned_addr learned my addr 172.21.15.89:0/1915578732 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:14.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff23de00700 1 --2- 172.21.15.89:0/1915578732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2400ae930 0x7ff24014dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:14.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff23e800700 1 --2- 172.21.15.89:0/1915578732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff240155c90 0x7ff240156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:14.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff23d400700 1 -- 172.21.15.89:0/1915578732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2400ae930 msgr2=0x7ff24014dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:14.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff23d400700 1 --2- 172.21.15.89:0/1915578732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2400ae930 0x7ff24014dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff23d400700 1 -- 172.21.15.89:0/1915578732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff240155c90 msgr2=0x7ff240156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:14.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff23d400700 1 --2- 172.21.15.89:0/1915578732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff240155c90 0x7ff240156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.585+0000 7ff23d400700 1 -- 172.21.15.89:0/1915578732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22c0094a0 con 0x7ff24015b5d0 2024-04-02T07:54:14.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.586+0000 7ff23d400700 1 --2- 172.21.15.89:0/1915578732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24015b5d0 0x7ff2400ce250 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00ebe0 tx=0x7ff22c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:14.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.586+0000 7ff236000700 1 -- 172.21.15.89:0/1915578732 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c00c850 con 0x7ff24015b5d0 2024-04-02T07:54:14.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.586+0000 7ff236000700 1 -- 172.21.15.89:0/1915578732 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22c00c9b0 con 0x7ff24015b5d0 2024-04-02T07:54:14.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.586+0000 7ff244c00700 1 -- 172.21.15.89:0/1915578732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff230009550 con 0x7ff24015b5d0 2024-04-02T07:54:14.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.586+0000 7ff236000700 1 -- 172.21.15.89:0/1915578732 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c00cd40 con 0x7ff24015b5d0 2024-04-02T07:54:14.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.586+0000 7ff244c00700 1 -- 172.21.15.89:0/1915578732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2401f6e70 con 0x7ff24015b5d0 2024-04-02T07:54:14.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.587+0000 7ff244c00700 1 -- 172.21.15.89:0/1915578732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2400ca690 con 0x7ff24015b5d0 2024-04-02T07:54:14.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.588+0000 7ff236000700 1 -- 172.21.15.89:0/1915578732 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7ff22c013470 con 0x7ff24015b5d0 2024-04-02T07:54:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.589+0000 7ff236000700 1 --2- 172.21.15.89:0/1915578732 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff22006aed0 0x7ff22006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.589+0000 7ff236000700 1 -- 172.21.15.89:0/1915578732 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7ff22c089a30 con 0x7ff24015b5d0 2024-04-02T07:54:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.589+0000 7ff23de00700 1 -- 172.21.15.89:0/1915578732 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff22006aed0 msgr2=0x7ff22006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.590+0000 7ff23de00700 1 --2- 172.21.15.89:0/1915578732 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff22006aed0 0x7ff22006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:14.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.593+0000 7ff236000700 1 -- 172.21.15.89:0/1915578732 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff22c054680 con 0x7ff24015b5d0 2024-04-02T07:54:14.717 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:14.715+0000 7f85b0efb240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:54:14.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.791+0000 7ff23de00700 1 -- 172.21.15.89:0/1915578732 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff22006aed0 msgr2=0x7ff22006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:14.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.791+0000 7ff23de00700 1 --2- 172.21.15.89:0/1915578732 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff22006aed0 0x7ff22006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:14.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.849+0000 7ff244c00700 1 -- 172.21.15.89:0/1915578732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff240067720 con 0x7ff24015b5d0 2024-04-02T07:54:14.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.849+0000 7ff236000700 1 -- 172.21.15.89:0/1915578732 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4742 (secure 0 0 0) 0x7ff22c00cea0 con 0x7ff24015b5d0 2024-04-02T07:54:14.851 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:14.851 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6317,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1566757639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1566757639},{"type":"v1","addr":"172.21.15.113:6835","nonce":1566757639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":6985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4102221782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4102221782},{"type":"v1","addr":"172.21.15.89:6836","nonce":4102221782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3127911401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3127911401},{"type":"v1","addr":"172.21.15.113:6834","nonce":3127911401}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:52:48.969376+0000","modified":"2024-04-02T07:52:51.980647+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7419":{"gid":7419,"name":"a","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2016280781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2016280781},{"type":"v1","addr":"172.21.15.89:6837","nonce":2016280781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-02T07:54:14.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.852+0000 7ff244c00700 1 -- 172.21.15.89:0/1915578732 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff22006aed0 msgr2=0x7ff22006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:14.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.852+0000 7ff244c00700 1 --2- 172.21.15.89:0/1915578732 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff22006aed0 0x7ff22006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.852+0000 7ff244c00700 1 -- 172.21.15.89:0/1915578732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24015b5d0 msgr2=0x7ff2400ce250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:14.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.852+0000 7ff244c00700 1 --2- 172.21.15.89:0/1915578732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24015b5d0 0x7ff2400ce250 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00ebe0 tx=0x7ff22c00eef0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.852+0000 7ff244c00700 1 -- 172.21.15.89:0/1915578732 shutdown_connections 2024-04-02T07:54:14.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.852+0000 7ff244c00700 1 --2- 172.21.15.89:0/1915578732 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff22006aed0 0x7ff22006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.852+0000 7ff244c00700 1 --2- 172.21.15.89:0/1915578732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff240155c90 0x7ff240156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.852+0000 7ff244c00700 1 --2- 172.21.15.89:0/1915578732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2400ae930 0x7ff24014dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.852+0000 7ff244c00700 1 --2- 172.21.15.89:0/1915578732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff24015b5d0 0x7ff2400ce250 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:14.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.852+0000 7ff244c00700 1 -- 172.21.15.89:0/1915578732 >> 172.21.15.89:0/1915578732 conn(0x7ff2400cf190 msgr2=0x7ff24015c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:14.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.852+0000 7ff244c00700 1 -- 172.21.15.89:0/1915578732 shutdown_connections 2024-04-02T07:54:14.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:14.852+0000 7ff244c00700 1 -- 172.21.15.89:0/1915578732 wait complete. 2024-04-02T07:54:14.856 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 49 2024-04-02T07:54:14.872 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:54:14.938 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:14.936+0000 7f85b0efb240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:54:15.048 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:15.047+0000 7efd97554240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:54:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.067+0000 7fd340800700 1 Processor -- start 2024-04-02T07:54:15.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.067+0000 7fd340800700 1 -- start start 2024-04-02T07:54:15.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.068+0000 7fd340800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd33815d920 0x7fd33806c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:15.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.068+0000 7fd340800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd33809a910 0x7fd3380cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:15.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.068+0000 7fd340800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd338099a60 0x7fd338099e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:15.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.068+0000 7fd340800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3380d08d0 con 0x7fd33809a910 2024-04-02T07:54:15.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.068+0000 7fd340800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33806c300 con 0x7fd33815d920 2024-04-02T07:54:15.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.068+0000 7fd340800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33809a3d0 con 0x7fd338099a60 2024-04-02T07:54:15.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.068+0000 7fd33d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd33809a910 0x7fd3380cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:15.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.069+0000 7fd33d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd33809a910 0x7fd3380cfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58646/0 (socket says 172.21.15.89:58646) 2024-04-02T07:54:15.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.069+0000 7fd33d400700 1 -- 172.21.15.89:0/559502586 learned_addr learned my addr 172.21.15.89:0/559502586 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:15.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.069+0000 7fd33e800700 1 --2- 172.21.15.89:0/559502586 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd338099a60 0x7fd338099e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:15.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.069+0000 7fd33de00700 1 --2- 172.21.15.89:0/559502586 >> v2:172.21.15.113:3300/0 conn(0x7fd33815d920 0x7fd33806c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:15.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.069+0000 7fd33d400700 1 -- 172.21.15.89:0/559502586 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd338099a60 msgr2=0x7fd338099e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.069+0000 7fd33d400700 1 --2- 172.21.15.89:0/559502586 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd338099a60 0x7fd338099e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.069+0000 7fd33d400700 1 -- 172.21.15.89:0/559502586 >> v2:172.21.15.113:3300/0 conn(0x7fd33815d920 msgr2=0x7fd33806c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.069+0000 7fd33d400700 1 --2- 172.21.15.89:0/559502586 >> v2:172.21.15.113:3300/0 conn(0x7fd33815d920 0x7fd33806c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.069+0000 7fd33d400700 1 -- 172.21.15.89:0/559502586 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33806b9d0 con 0x7fd33809a910 2024-04-02T07:54:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.069+0000 7fd33de00700 1 --2- 172.21.15.89:0/559502586 >> v2:172.21.15.113:3300/0 conn(0x7fd33815d920 0x7fd33806c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.069+0000 7fd33d400700 1 --2- 172.21.15.89:0/559502586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd33809a910 0x7fd3380cfff0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fd33809a6b0 tx=0x7fd328009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5acfe4ed223f18e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.070+0000 7fd33ca00700 1 -- 172.21.15.89:0/559502586 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd328011830 con 0x7fd33809a910 2024-04-02T07:54:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.070+0000 7fd33ca00700 1 -- 172.21.15.89:0/559502586 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd328011990 con 0x7fd33809a910 2024-04-02T07:54:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.070+0000 7fd33ca00700 1 -- 172.21.15.89:0/559502586 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd328011d20 con 0x7fd33809a910 2024-04-02T07:54:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.070+0000 7fd340800700 1 -- 172.21.15.89:0/559502586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd33809a910 msgr2=0x7fd3380cfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.070+0000 7fd340800700 1 --2- 172.21.15.89:0/559502586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd33809a910 0x7fd3380cfff0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fd33809a6b0 tx=0x7fd328009bd0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.070+0000 7fd340800700 1 -- 172.21.15.89:0/559502586 shutdown_connections 2024-04-02T07:54:15.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.070+0000 7fd340800700 1 --2- 172.21.15.89:0/559502586 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd338099a60 0x7fd338099e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.070+0000 7fd340800700 1 --2- 172.21.15.89:0/559502586 >> v2:172.21.15.113:3300/0 conn(0x7fd33815d920 0x7fd33806c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.070+0000 7fd340800700 1 --2- 172.21.15.89:0/559502586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd33809a910 0x7fd3380cfff0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.070+0000 7fd340800700 1 -- 172.21.15.89:0/559502586 >> 172.21.15.89:0/559502586 conn(0x7fd3380d1350 msgr2=0x7fd3380d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.070+0000 7fd340800700 1 -- 172.21.15.89:0/559502586 shutdown_connections 2024-04-02T07:54:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.070+0000 7fd340800700 1 -- 172.21.15.89:0/559502586 wait complete. 2024-04-02T07:54:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.072+0000 7fd340800700 1 Processor -- start 2024-04-02T07:54:15.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.072+0000 7fd340800700 1 -- start start 2024-04-02T07:54:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.073+0000 7fd340800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd338099a60 0x7fd3381588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.073+0000 7fd340800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd33809a910 0x7fd3380b6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:15.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.073+0000 7fd340800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd33815d920 0x7fd3380b6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.073+0000 7fd340800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3380b6fe0 con 0x7fd33815d920 2024-04-02T07:54:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.073+0000 7fd340800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3380cade0 con 0x7fd338099a60 2024-04-02T07:54:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.073+0000 7fd340800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3380ca260 con 0x7fd33809a910 2024-04-02T07:54:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.073+0000 7fd33de00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd338099a60 0x7fd3381588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.073+0000 7fd33de00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd338099a60 0x7fd3381588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:56288/0 (socket says 172.21.15.89:56288) 2024-04-02T07:54:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.073+0000 7fd33e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd33815d920 0x7fd3380b6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.073+0000 7fd33de00700 1 -- 172.21.15.89:0/4263737894 learned_addr learned my addr 172.21.15.89:0/4263737894 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.073+0000 7fd33d400700 1 --2- 172.21.15.89:0/4263737894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd33809a910 0x7fd3380b6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.074+0000 7fd33de00700 1 -- 172.21.15.89:0/4263737894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd33809a910 msgr2=0x7fd3380b6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.074+0000 7fd33de00700 1 --2- 172.21.15.89:0/4263737894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd33809a910 0x7fd3380b6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.074+0000 7fd33de00700 1 -- 172.21.15.89:0/4263737894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd33815d920 msgr2=0x7fd3380b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.074+0000 7fd33de00700 1 --2- 172.21.15.89:0/4263737894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd33815d920 0x7fd3380b6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.074+0000 7fd33de00700 1 -- 172.21.15.89:0/4263737894 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c0094a0 con 0x7fd338099a60 2024-04-02T07:54:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.074+0000 7fd33de00700 1 --2- 172.21.15.89:0/4263737894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd338099a60 0x7fd3381588f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00ebe0 tx=0x7fd32c00eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.074+0000 7fd332000700 1 -- 172.21.15.89:0/4263737894 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c009760 con 0x7fd338099a60 2024-04-02T07:54:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.074+0000 7fd332000700 1 -- 172.21.15.89:0/4263737894 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c00c850 con 0x7fd338099a60 2024-04-02T07:54:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.075+0000 7fd340800700 1 -- 172.21.15.89:0/4263737894 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd328009550 con 0x7fd338099a60 2024-04-02T07:54:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.075+0000 7fd332000700 1 -- 172.21.15.89:0/4263737894 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c00cbe0 con 0x7fd338099a60 2024-04-02T07:54:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.075+0000 7fd340800700 1 -- 172.21.15.89:0/4263737894 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3380ca820 con 0x7fd338099a60 2024-04-02T07:54:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.076+0000 7fd340800700 1 -- 172.21.15.89:0/4263737894 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33815e9c0 con 0x7fd338099a60 2024-04-02T07:54:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.077+0000 7fd332000700 1 -- 172.21.15.89:0/4263737894 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fd32c00cd40 con 0x7fd338099a60 2024-04-02T07:54:15.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.077+0000 7fd332000700 1 --2- 172.21.15.89:0/4263737894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd31c06aed0 0x7fd31c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:15.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.077+0000 7fd332000700 1 -- 172.21.15.89:0/4263737894 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd32c089a00 con 0x7fd338099a60 2024-04-02T07:54:15.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.078+0000 7fd33d400700 1 -- 172.21.15.89:0/4263737894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd31c06aed0 msgr2=0x7fd31c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:15.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.078+0000 7fd33d400700 1 --2- 172.21.15.89:0/4263737894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd31c06aed0 0x7fd31c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:15.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.082+0000 7fd332000700 1 -- 172.21.15.89:0/4263737894 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd32c0545b0 con 0x7fd338099a60 2024-04-02T07:54:15.254 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:15.253+0000 7efd97554240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:54:15.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.279+0000 7fd33d400700 1 -- 172.21.15.89:0/4263737894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd31c06aed0 msgr2=0x7fd31c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:15.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.279+0000 7fd33d400700 1 --2- 172.21.15.89:0/4263737894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd31c06aed0 0x7fd31c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:15.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.294+0000 7fd340800700 1 -- 172.21.15.89:0/4263737894 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd338067720 con 0x7fd338099a60 2024-04-02T07:54:15.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.295+0000 7fd332000700 1 -- 172.21.15.89:0/4263737894 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+17130 (secure 0 0 0) 0x7fd32c057bd0 con 0x7fd338099a60 2024-04-02T07:54:15.297 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:15.298 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":112,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:53:05.735192+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:52:46.885836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-02T07:52:47.989359+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/1115793129":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/458120672":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/2657927819":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/383704566":"2024-04-03T07:53:05.735155+0000","172.21.15.89:6832/43357":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/1938634818":"2024-04-03T07:53:05.735155+0000","172.21.15.89:6833/43357":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/552112415":"2024-04-03T07:53:05.735155+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:54:15.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.298+0000 7fd340800700 1 -- 172.21.15.89:0/4263737894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd31c06aed0 msgr2=0x7fd31c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:15.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.298+0000 7fd340800700 1 --2- 172.21.15.89:0/4263737894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd31c06aed0 0x7fd31c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.298+0000 7fd340800700 1 -- 172.21.15.89:0/4263737894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd338099a60 msgr2=0x7fd3381588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:15.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.298+0000 7fd340800700 1 --2- 172.21.15.89:0/4263737894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd338099a60 0x7fd3381588f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00ebe0 tx=0x7fd32c00eef0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.298+0000 7fd340800700 1 -- 172.21.15.89:0/4263737894 shutdown_connections 2024-04-02T07:54:15.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.298+0000 7fd340800700 1 --2- 172.21.15.89:0/4263737894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd31c06aed0 0x7fd31c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.298+0000 7fd340800700 1 --2- 172.21.15.89:0/4263737894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd338099a60 0x7fd3381588f0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.298+0000 7fd340800700 1 --2- 172.21.15.89:0/4263737894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd33815d920 0x7fd3380b6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.298+0000 7fd340800700 1 --2- 172.21.15.89:0/4263737894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd33809a910 0x7fd3380b6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.298+0000 7fd340800700 1 -- 172.21.15.89:0/4263737894 >> 172.21.15.89:0/4263737894 conn(0x7fd3380d1350 msgr2=0x7fd3380cdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:15.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.298+0000 7fd340800700 1 -- 172.21.15.89:0/4263737894 shutdown_connections 2024-04-02T07:54:15.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.298+0000 7fd340800700 1 -- 172.21.15.89:0/4263737894 wait complete. 2024-04-02T07:54:15.320 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T07:54:15.320 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:54:15.449 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:15.448+0000 7f85b0efb240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:54:15.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.518+0000 7f6546200700 1 Processor -- start 2024-04-02T07:54:15.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.518+0000 7f6546200700 1 -- start start 2024-04-02T07:54:15.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f6546200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f654015d920 0x7f654006c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:15.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f6546200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f654009a910 0x7f65400cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:15.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f6546200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6540099a60 0x7f6540099eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:15.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f6546200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f654009a3f0 con 0x7f654015d920 2024-04-02T07:54:15.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f6546200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65400d08d0 con 0x7f6540099a60 2024-04-02T07:54:15.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f6546200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654006c300 con 0x7f654009a910 2024-04-02T07:54:15.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f653f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f654015d920 0x7f654006c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:15.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f653f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f654015d920 0x7f654006c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58660/0 (socket says 172.21.15.89:58660) 2024-04-02T07:54:15.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f653f400700 1 -- 172.21.15.89:0/4204126918 learned_addr learned my addr 172.21.15.89:0/4204126918 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:15.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f653ea00700 1 --2- 172.21.15.89:0/4204126918 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f654009a910 0x7f65400cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:15.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f653fe00700 1 --2- 172.21.15.89:0/4204126918 >> v2:172.21.15.113:3300/0 conn(0x7f6540099a60 0x7f6540099eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:15.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f653f400700 1 -- 172.21.15.89:0/4204126918 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f654009a910 msgr2=0x7f65400cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:15.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f653f400700 1 --2- 172.21.15.89:0/4204126918 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f654009a910 0x7f65400cfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f653f400700 1 -- 172.21.15.89:0/4204126918 >> v2:172.21.15.113:3300/0 conn(0x7f6540099a60 msgr2=0x7f6540099eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:15.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f653f400700 1 --2- 172.21.15.89:0/4204126918 >> v2:172.21.15.113:3300/0 conn(0x7f6540099a60 0x7f6540099eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f653f400700 1 -- 172.21.15.89:0/4204126918 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654006b9d0 con 0x7f654015d920 2024-04-02T07:54:15.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.520+0000 7f653fe00700 1 --2- 172.21.15.89:0/4204126918 >> v2:172.21.15.113:3300/0 conn(0x7f6540099a60 0x7f6540099eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:15.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.521+0000 7f653f400700 1 --2- 172.21.15.89:0/4204126918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654015d920 0x7f654006c790 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f6540074590 tx=0x7f6530009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c01ed71d511c066 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:15.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.521+0000 7f653e000700 1 -- 172.21.15.89:0/4204126918 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530011830 con 0x7f654015d920 2024-04-02T07:54:15.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.521+0000 7f653e000700 1 -- 172.21.15.89:0/4204126918 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6530011990 con 0x7f654015d920 2024-04-02T07:54:15.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.521+0000 7f653e000700 1 -- 172.21.15.89:0/4204126918 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530011d20 con 0x7f654015d920 2024-04-02T07:54:15.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.522+0000 7f6546200700 1 -- 172.21.15.89:0/4204126918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654015d920 msgr2=0x7f654006c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:15.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.522+0000 7f6546200700 1 --2- 172.21.15.89:0/4204126918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654015d920 0x7f654006c790 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f6540074590 tx=0x7f6530009bd0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.522+0000 7f6546200700 1 -- 172.21.15.89:0/4204126918 shutdown_connections 2024-04-02T07:54:15.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.522+0000 7f6546200700 1 --2- 172.21.15.89:0/4204126918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654015d920 0x7f654006c790 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.522+0000 7f6546200700 1 --2- 172.21.15.89:0/4204126918 >> v2:172.21.15.113:3300/0 conn(0x7f6540099a60 0x7f6540099eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.522+0000 7f6546200700 1 --2- 172.21.15.89:0/4204126918 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f654009a910 0x7f65400cfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.522+0000 7f6546200700 1 -- 172.21.15.89:0/4204126918 >> 172.21.15.89:0/4204126918 conn(0x7f65400d1350 msgr2=0x7f65400d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:15.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.522+0000 7f6546200700 1 -- 172.21.15.89:0/4204126918 shutdown_connections 2024-04-02T07:54:15.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.522+0000 7f6546200700 1 -- 172.21.15.89:0/4204126918 wait complete. 2024-04-02T07:54:15.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.524+0000 7f6546200700 1 Processor -- start 2024-04-02T07:54:15.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.524+0000 7f6546200700 1 -- start start 2024-04-02T07:54:15.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f6546200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6540099a60 0x7f65400b6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:15.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f6546200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654009a910 0x7f654009eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:15.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f6546200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f654015d920 0x7f654009b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:15.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f6546200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65400b56f0 con 0x7f654009a910 2024-04-02T07:54:15.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f6546200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65400b59d0 con 0x7f654015d920 2024-04-02T07:54:15.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f6546200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65400b5cb0 con 0x7f6540099a60 2024-04-02T07:54:15.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f653ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654009a910 0x7f654009eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:15.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f653ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654009a910 0x7f654009eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58666/0 (socket says 172.21.15.89:58666) 2024-04-02T07:54:15.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f653ea00700 1 -- 172.21.15.89:0/2914002545 learned_addr learned my addr 172.21.15.89:0/2914002545 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:15.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f653f400700 1 --2- 172.21.15.89:0/2914002545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6540099a60 0x7f65400b6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:15.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f653fe00700 1 --2- 172.21.15.89:0/2914002545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f654015d920 0x7f654009b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:15.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f653ea00700 1 -- 172.21.15.89:0/2914002545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6540099a60 msgr2=0x7f65400b6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:15.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f653ea00700 1 --2- 172.21.15.89:0/2914002545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6540099a60 0x7f65400b6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f653ea00700 1 -- 172.21.15.89:0/2914002545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f654015d920 msgr2=0x7f654009b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:15.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f653ea00700 1 --2- 172.21.15.89:0/2914002545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f654015d920 0x7f654009b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.525+0000 7f653ea00700 1 -- 172.21.15.89:0/2914002545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6530009550 con 0x7f654009a910 2024-04-02T07:54:15.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.526+0000 7f653ea00700 1 --2- 172.21.15.89:0/2914002545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654009a910 0x7f654009eb20 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f652c00e840 tx=0x7f652c00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:15.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.526+0000 7f6537400700 1 -- 172.21.15.89:0/2914002545 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c009590 con 0x7f654009a910 2024-04-02T07:54:15.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.526+0000 7f6537400700 1 -- 172.21.15.89:0/2914002545 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f652c013680 con 0x7f654009a910 2024-04-02T07:54:15.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.526+0000 7f6546200700 1 -- 172.21.15.89:0/2914002545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65400b5f90 con 0x7f654009a910 2024-04-02T07:54:15.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.526+0000 7f6537400700 1 -- 172.21.15.89:0/2914002545 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c013a30 con 0x7f654009a910 2024-04-02T07:54:15.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.526+0000 7f6546200700 1 -- 172.21.15.89:0/2914002545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f654009f060 con 0x7f654009a910 2024-04-02T07:54:15.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.527+0000 7f6546200700 1 -- 172.21.15.89:0/2914002545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65400b64b0 con 0x7f654009a910 2024-04-02T07:54:15.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.528+0000 7f6537400700 1 -- 172.21.15.89:0/2914002545 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f652c013b90 con 0x7f654009a910 2024-04-02T07:54:15.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.529+0000 7f6537400700 1 --2- 172.21.15.89:0/2914002545 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f652006aed0 0x7f652006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:15.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.529+0000 7f6537400700 1 -- 172.21.15.89:0/2914002545 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f652c089940 con 0x7f654009a910 2024-04-02T07:54:15.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.529+0000 7f653f400700 1 -- 172.21.15.89:0/2914002545 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f652006aed0 msgr2=0x7f652006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:15.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.529+0000 7f653f400700 1 --2- 172.21.15.89:0/2914002545 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f652006aed0 0x7f652006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:15.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.533+0000 7f6537400700 1 -- 172.21.15.89:0/2914002545 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f65400b64b0 con 0x7f654009a910 2024-04-02T07:54:15.670 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:15.668+0000 7f85b0efb240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:54:15.692 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:15.691+0000 7efd97554240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:54:15.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.730+0000 7f653f400700 1 -- 172.21.15.89:0/2914002545 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f652006aed0 msgr2=0x7f652006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:15.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.730+0000 7f653f400700 1 --2- 172.21.15.89:0/2914002545 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f652006aed0 0x7f652006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:15.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.789+0000 7f6546200700 1 -- 172.21.15.89:0/2914002545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65400b64b0 con 0x7f654009a910 2024-04-02T07:54:15.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.790+0000 7f6537400700 1 -- 172.21.15.89:0/2914002545 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f65400b64b0 con 0x7f654009a910 2024-04-02T07:54:15.792 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:15.792 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6317,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1566757639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1566757639},{"type":"v1","addr":"172.21.15.113:6835","nonce":1566757639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":6985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4102221782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4102221782},{"type":"v1","addr":"172.21.15.89:6836","nonce":4102221782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3127911401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3127911401},{"type":"v1","addr":"172.21.15.113:6834","nonce":3127911401}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:52:48.969376+0000","modified":"2024-04-02T07:52:51.980647+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7419":{"gid":7419,"name":"a","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2016280781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2016280781},{"type":"v1","addr":"172.21.15.89:6837","nonce":2016280781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-02T07:54:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.792+0000 7f6546200700 1 -- 172.21.15.89:0/2914002545 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f652006aed0 msgr2=0x7f652006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.792+0000 7f6546200700 1 --2- 172.21.15.89:0/2914002545 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f652006aed0 0x7f652006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.792+0000 7f6546200700 1 -- 172.21.15.89:0/2914002545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654009a910 msgr2=0x7f654009eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.792+0000 7f6546200700 1 --2- 172.21.15.89:0/2914002545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654009a910 0x7f654009eb20 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f652c00e840 tx=0x7f652c00eb50 comp rx=0 tx=0).stop 2024-04-02T07:54:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.793+0000 7f6546200700 1 -- 172.21.15.89:0/2914002545 shutdown_connections 2024-04-02T07:54:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.793+0000 7f6546200700 1 --2- 172.21.15.89:0/2914002545 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f652006aed0 0x7f652006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.793+0000 7f6546200700 1 --2- 172.21.15.89:0/2914002545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f654015d920 0x7f654009b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.793+0000 7f6546200700 1 --2- 172.21.15.89:0/2914002545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6540099a60 0x7f65400b6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.793+0000 7f6546200700 1 --2- 172.21.15.89:0/2914002545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654009a910 0x7f654009eb20 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.793+0000 7f6546200700 1 -- 172.21.15.89:0/2914002545 >> 172.21.15.89:0/2914002545 conn(0x7f65400d1350 msgr2=0x7f65400b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.793+0000 7f6546200700 1 -- 172.21.15.89:0/2914002545 shutdown_connections 2024-04-02T07:54:15.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:15.793+0000 7f6546200700 1 -- 172.21.15.89:0/2914002545 wait complete. 2024-04-02T07:54:15.796 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 49 2024-04-02T07:54:15.801 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:15.800+0000 7efd97554240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:54:15.813 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:54:16.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.012+0000 7f69e9c00700 1 Processor -- start 2024-04-02T07:54:16.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.012+0000 7f69e9c00700 1 -- start start 2024-04-02T07:54:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.013+0000 7f69e9c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f69e40718e0 0x7f69e40ad070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.013+0000 7f69e9c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e40ac4d0 con 0x7f69e40b27f0 2024-04-02T07:54:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.013+0000 7f69e9c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e4071e50 con 0x7f69e4160c90 2024-04-02T07:54:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.013+0000 7f69e9c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e4072130 con 0x7f69e40718e0 2024-04-02T07:54:16.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.014+0000 7f69e2000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f69e40b27f0 0x7f69e40ac960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33628/0 (socket says 172.21.15.89:33628) 2024-04-02T07:54:16.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.014+0000 7f69e2000700 1 -- 172.21.15.89:0/3750688896 learned_addr learned my addr 172.21.15.89:0/3750688896 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:16.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.014+0000 7f69e3400700 1 --2- 172.21.15.89:0/3750688896 >> v2:172.21.15.113:3300/0 conn(0x7f69e40718e0 0x7f69e40ad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:16.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.014+0000 7f69e1600700 1 -- 172.21.15.89:0/3750688896 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052369012 0 0) 0x7f69e40ac4d0 con 0x7f69e40b27f0 2024-04-02T07:54:16.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.014+0000 7f69e1600700 1 -- 172.21.15.89:0/3750688896 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c80035b0 con 0x7f69e40b27f0 2024-04-02T07:54:16.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.014+0000 7f69e1600700 1 -- 172.21.15.89:0/3750688896 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1650323377 0 0) 0x7f69c80035b0 con 0x7f69e40b27f0 2024-04-02T07:54:16.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.014+0000 7f69e1600700 1 -- 172.21.15.89:0/3750688896 >> v2:172.21.15.113:3300/0 conn(0x7f69e40718e0 msgr2=0x7f69e40ad070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:16.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.014+0000 7f69e1600700 1 --2- 172.21.15.89:0/3750688896 >> v2:172.21.15.113:3300/0 conn(0x7f69e40718e0 0x7f69e40ad070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.014+0000 7f69e1600700 1 -- 172.21.15.89:0/3750688896 >> v1:172.21.15.113:6789/0 conn(0x7f69e4160c90 legacy=0x7f69e40b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.014+0000 7f69e1600700 1 -- 172.21.15.89:0/3750688896 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e4072410 con 0x7f69e40b27f0 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.014+0000 7f69e1600700 1 -- 172.21.15.89:0/3750688896 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f69d0001670 con 0x7f69e40b27f0 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.014+0000 7f69e1600700 1 -- 172.21.15.89:0/3750688896 >> v1:172.21.15.89:6789/0 conn(0x7f69e40b27f0 legacy=0x7f69e40ac960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.014+0000 7f69e1600700 1 --2- 172.21.15.89:0/3750688896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c8003a00 0x7f69c8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e1600700 1 --2- 172.21.15.89:0/3750688896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c8004400 0x7f69c80048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e1600700 1 --2- 172.21.15.89:0/3750688896 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c80050d0 0x7f69c800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e1600700 1 -- 172.21.15.89:0/3750688896 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e4072130 con 0x7f69c8004400 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e1600700 1 -- 172.21.15.89:0/3750688896 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c800c030 con 0x7f69c8003a00 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e1600700 1 -- 172.21.15.89:0/3750688896 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c800c1f0 con 0x7f69c80050d0 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e2000700 1 --2- 172.21.15.89:0/3750688896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c8004400 0x7f69c80048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e3400700 1 --2- 172.21.15.89:0/3750688896 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c80050d0 0x7f69c800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e2a00700 1 --2- 172.21.15.89:0/3750688896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c8003a00 0x7f69c8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e2000700 1 -- 172.21.15.89:0/3750688896 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c80050d0 msgr2=0x7f69c800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e2000700 1 --2- 172.21.15.89:0/3750688896 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c80050d0 0x7f69c800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e2000700 1 -- 172.21.15.89:0/3750688896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c8003a00 msgr2=0x7f69c8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e2000700 1 --2- 172.21.15.89:0/3750688896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c8003a00 0x7f69c8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e2000700 1 -- 172.21.15.89:0/3750688896 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c80035b0 con 0x7f69c8004400 2024-04-02T07:54:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e3400700 1 -- 172.21.15.89:0/3750688896 reap_dead start 2024-04-02T07:54:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.015+0000 7f69e2000700 1 --2- 172.21.15.89:0/3750688896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c8004400 0x7f69c80048d0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f69d000c240 tx=0x7f69c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48b3037a9ebc10c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.016+0000 7f69e1600700 1 -- 172.21.15.89:0/3750688896 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d000cd80 con 0x7f69c8004400 2024-04-02T07:54:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.016+0000 7f69e1600700 1 -- 172.21.15.89:0/3750688896 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d0005960 con 0x7f69c8004400 2024-04-02T07:54:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.016+0000 7f69e1600700 1 -- 172.21.15.89:0/3750688896 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d0005d10 con 0x7f69c8004400 2024-04-02T07:54:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.016+0000 7f69e9c00700 1 -- 172.21.15.89:0/3750688896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c8004400 msgr2=0x7f69c80048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.016+0000 7f69e9c00700 1 --2- 172.21.15.89:0/3750688896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c8004400 0x7f69c80048d0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f69d000c240 tx=0x7f69c8005880 comp rx=0 tx=0).stop 2024-04-02T07:54:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.016+0000 7f69e9c00700 1 -- 172.21.15.89:0/3750688896 shutdown_connections 2024-04-02T07:54:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.016+0000 7f69e9c00700 1 --2- 172.21.15.89:0/3750688896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c8004400 0x7f69c80048d0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.016+0000 7f69e9c00700 1 -- 172.21.15.89:0/3750688896 >> 172.21.15.89:0/3750688896 conn(0x7f69e40cf190 msgr2=0x7f69e40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.016+0000 7f69e9c00700 1 -- 172.21.15.89:0/3750688896 shutdown_connections 2024-04-02T07:54:16.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.016+0000 7f69e9c00700 1 -- 172.21.15.89:0/3750688896 wait complete. 2024-04-02T07:54:16.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.018+0000 7f69e9c00700 1 Processor -- start 2024-04-02T07:54:16.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.018+0000 7f69e9c00700 1 -- start start 2024-04-02T07:54:16.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.019+0000 7f69e9c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c8004400 0x7f69e409bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:16.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.019+0000 7f69e9c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69e415ee40 0x7f69e414dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:16.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.019+0000 7f69e9c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40c9810 0x7f69e40c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:16.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.019+0000 7f69e9c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e414e010 con 0x7f69e40c9810 2024-04-02T07:54:16.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.019+0000 7f69e9c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e40ca200 con 0x7f69e415ee40 2024-04-02T07:54:16.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.019+0000 7f69e9c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e409cde0 con 0x7f69c8004400 2024-04-02T07:54:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.020+0000 7f69e2a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c8004400 0x7f69e409bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.020+0000 7f69e3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40c9810 0x7f69e40c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.020+0000 7f69e2a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c8004400 0x7f69e409bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:40720/0 (socket says 172.21.15.89:40720) 2024-04-02T07:54:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.020+0000 7f69e2a00700 1 -- 172.21.15.89:0/1440040034 learned_addr learned my addr 172.21.15.89:0/1440040034 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.020+0000 7f69e2000700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69e415ee40 0x7f69e414dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.020+0000 7f69e3400700 1 -- 172.21.15.89:0/1440040034 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c8004400 msgr2=0x7f69e409bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.020+0000 7f69e3400700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c8004400 0x7f69e409bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.020+0000 7f69e3400700 1 -- 172.21.15.89:0/1440040034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69e415ee40 msgr2=0x7f69e414dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.020+0000 7f69e3400700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69e415ee40 0x7f69e414dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.020+0000 7f69e3400700 1 -- 172.21.15.89:0/1440040034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e409c1d0 con 0x7f69e40c9810 2024-04-02T07:54:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.020+0000 7f69e2000700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69e415ee40 0x7f69e414dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.020+0000 7f69e2a00700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c8004400 0x7f69e409bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.020+0000 7f69e3400700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40c9810 0x7f69e40c9cc0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f69e40700f0 tx=0x7f69dc011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:16.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.021+0000 7f69daa00700 1 -- 172.21.15.89:0/1440040034 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69dc00e040 con 0x7f69e40c9810 2024-04-02T07:54:16.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.021+0000 7f69daa00700 1 -- 172.21.15.89:0/1440040034 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69dc00f040 con 0x7f69e40c9810 2024-04-02T07:54:16.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.021+0000 7f69daa00700 1 -- 172.21.15.89:0/1440040034 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69dc00fc40 con 0x7f69e40c9810 2024-04-02T07:54:16.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.021+0000 7f69e9c00700 1 -- 172.21.15.89:0/1440040034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e409c4b0 con 0x7f69e40c9810 2024-04-02T07:54:16.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.021+0000 7f69e9c00700 1 -- 172.21.15.89:0/1440040034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e409ca70 con 0x7f69e40c9810 2024-04-02T07:54:16.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.022+0000 7f69e9c00700 1 -- 172.21.15.89:0/1440040034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e4067720 con 0x7f69e40c9810 2024-04-02T07:54:16.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.024+0000 7f69daa00700 1 -- 172.21.15.89:0/1440040034 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 60) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f69dc022460 con 0x7f69e40c9810 2024-04-02T07:54:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.025+0000 7f69daa00700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69c406aed0 0x7f69c406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.025+0000 7f69daa00700 1 -- 172.21.15.89:0/1440040034 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f69dc089fc0 con 0x7f69e40c9810 2024-04-02T07:54:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.025+0000 7f69e2a00700 1 -- 172.21.15.89:0/1440040034 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69c406aed0 msgr2=0x7f69c406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.025+0000 7f69e2a00700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69c406aed0 0x7f69c406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:16.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.029+0000 7f69daa00700 1 -- 172.21.15.89:0/1440040034 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f69dc0549d0 con 0x7f69e40c9810 2024-04-02T07:54:16.138 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:16.136+0000 7f85b0efb240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:54:16.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.145+0000 7f69daa00700 1 -- 172.21.15.89:0/1440040034 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 61) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f69dc054da0 con 0x7f69e40c9810 2024-04-02T07:54:16.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.226+0000 7f69e2a00700 1 -- 172.21.15.89:0/1440040034 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69c406aed0 msgr2=0x7f69c406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:16.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.226+0000 7f69e2a00700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69c406aed0 0x7f69c406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:16.258 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:16.257+0000 7f85b0efb240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:54:16.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.267+0000 7f69daa00700 1 -- 172.21.15.89:0/1440040034 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 62) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f69dc081c20 con 0x7f69e40c9810 2024-04-02T07:54:16.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.267+0000 7f69daa00700 1 -- 172.21.15.89:0/1440040034 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69c406aed0 msgr2=0x7f69c406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:16.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.267+0000 7f69daa00700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69c406aed0 0x7f69c406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.302+0000 7f69e9c00700 1 -- 172.21.15.89:0/1440040034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69e409cc00 con 0x7f69e40c9810 2024-04-02T07:54:16.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.302+0000 7f69daa00700 1 -- 172.21.15.89:0/1440040034 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f69dc0559e0 con 0x7f69e40c9810 2024-04-02T07:54:16.304 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:16.305 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6317,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1566757639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1566757639},{"type":"v1","addr":"172.21.15.113:6835","nonce":1566757639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":6985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4102221782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4102221782},{"type":"v1","addr":"172.21.15.89:6836","nonce":4102221782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3127911401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3127911401},{"type":"v1","addr":"172.21.15.113:6834","nonce":3127911401}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:52:48.969376+0000","modified":"2024-04-02T07:52:51.980647+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7419":{"gid":7419,"name":"a","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2016280781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2016280781},{"type":"v1","addr":"172.21.15.89:6837","nonce":2016280781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-02T07:54:16.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.306+0000 7f69e9c00700 1 -- 172.21.15.89:0/1440040034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40c9810 msgr2=0x7f69e40c9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:16.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.306+0000 7f69e9c00700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40c9810 0x7f69e40c9cc0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f69e40700f0 tx=0x7f69dc011ed0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.306+0000 7f69e9c00700 1 -- 172.21.15.89:0/1440040034 shutdown_connections 2024-04-02T07:54:16.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.306+0000 7f69e9c00700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f69c406aed0 0x7f69c406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.306+0000 7f69e9c00700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69e40c9810 0x7f69e40c9cc0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.306+0000 7f69e9c00700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c8004400 0x7f69e409bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.306+0000 7f69e9c00700 1 --2- 172.21.15.89:0/1440040034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69e415ee40 0x7f69e414dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.306+0000 7f69e9c00700 1 -- 172.21.15.89:0/1440040034 >> 172.21.15.89:0/1440040034 conn(0x7f69e40cf190 msgr2=0x7f69e40adfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:16.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.306+0000 7f69e9c00700 1 -- 172.21.15.89:0/1440040034 shutdown_connections 2024-04-02T07:54:16.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.306+0000 7f69e9c00700 1 -- 172.21.15.89:0/1440040034 wait complete. 2024-04-02T07:54:16.309 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 49 2024-04-02T07:54:16.329 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:54:16.343 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:16.341+0000 7f8522c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:16.343 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:16.341+0000 7f8522c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:16.344 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:16.341+0000 7f8522c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:16.344 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:16.341+0000 7f8522c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:16.344 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:16.341+0000 7f8522c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:16.344 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:16.342+0000 7f851e600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:16.344 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:16.342+0000 7f851e600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:16.344 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:16.342+0000 7f851e600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:16.344 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:16.342+0000 7f851e600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:16.344 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:16.342+0000 7f851e600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:16.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.521+0000 7f78a7400700 1 Processor -- start 2024-04-02T07:54:16.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.522+0000 7f78a7400700 1 -- start start 2024-04-02T07:54:16.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.523+0000 7f78a7400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78a0160be0 0x7f78a00cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:16.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.523+0000 7f78a7400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78a00ce0e0 0x7f78a00ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:16.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.523+0000 7f78a7400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a0160fc0 con 0x7f78a00cead0 2024-04-02T07:54:16.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.523+0000 7f78a7400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a00cef60 con 0x7f78a0160be0 2024-04-02T07:54:16.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.523+0000 7f78a7400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00b65d0 con 0x7f78a00ce0e0 2024-04-02T07:54:16.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.523+0000 7f78a4a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78a0160be0 0x7f78a00cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:16.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.523+0000 7f78a4a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78a0160be0 0x7f78a00cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58816/0 (socket says 172.21.15.89:58816) 2024-04-02T07:54:16.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.523+0000 7f78a4a00700 1 -- 172.21.15.89:0/817996416 learned_addr learned my addr 172.21.15.89:0/817996416 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:16.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.523+0000 7f78a4a00700 1 -- 172.21.15.89:0/817996416 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78a00ce0e0 msgr2=0x7f78a00ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:16.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.523+0000 7f789fe00700 1 --2- 172.21.15.89:0/817996416 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78a00ce0e0 0x7f78a00ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:16.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.523+0000 7f78a4a00700 1 --2- 172.21.15.89:0/817996416 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78a00ce0e0 0x7f78a00ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.523+0000 7f78a4a00700 1 -- 172.21.15.89:0/817996416 >> v1:172.21.15.113:6789/0 conn(0x7f78a00cead0 legacy=0x7f78a0161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:16.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.524+0000 7f78a4a00700 1 -- 172.21.15.89:0/817996416 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00d21e0 con 0x7f78a0160be0 2024-04-02T07:54:16.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.524+0000 7f789fe00700 1 --2- 172.21.15.89:0/817996416 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78a00ce0e0 0x7f78a00ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:16.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.524+0000 7f78a4a00700 1 --2- 172.21.15.89:0/817996416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a0160be0 0x7f78a00cdba0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f78940098f0 tx=0x7f7894009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1decbdc4cc6bf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:16.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.524+0000 7f789f400700 1 -- 172.21.15.89:0/817996416 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789400b8a0 con 0x7f78a0160be0 2024-04-02T07:54:16.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.524+0000 7f789f400700 1 -- 172.21.15.89:0/817996416 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789400ba00 con 0x7f78a0160be0 2024-04-02T07:54:16.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.524+0000 7f789f400700 1 -- 172.21.15.89:0/817996416 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789400bd90 con 0x7f78a0160be0 2024-04-02T07:54:16.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.524+0000 7f78a7400700 1 -- 172.21.15.89:0/817996416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a0160be0 msgr2=0x7f78a00cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:16.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.524+0000 7f78a7400700 1 --2- 172.21.15.89:0/817996416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a0160be0 0x7f78a00cdba0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f78940098f0 tx=0x7f7894009c00 comp rx=0 tx=0).stop 2024-04-02T07:54:16.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.525+0000 7f78a7400700 1 -- 172.21.15.89:0/817996416 shutdown_connections 2024-04-02T07:54:16.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.525+0000 7f78a7400700 1 --2- 172.21.15.89:0/817996416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a0160be0 0x7f78a00cdba0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.525+0000 7f78a7400700 1 --2- 172.21.15.89:0/817996416 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78a00ce0e0 0x7f78a00ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.525+0000 7f78a7400700 1 -- 172.21.15.89:0/817996416 >> 172.21.15.89:0/817996416 conn(0x7f78a008aff0 msgr2=0x7f78a00d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:16.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.525+0000 7f78a7400700 1 -- 172.21.15.89:0/817996416 shutdown_connections 2024-04-02T07:54:16.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.525+0000 7f78a7400700 1 -- 172.21.15.89:0/817996416 wait complete. 2024-04-02T07:54:16.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.526+0000 7f78a7400700 1 Processor -- start 2024-04-02T07:54:16.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.527+0000 7f78a7400700 1 -- start start 2024-04-02T07:54:16.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.527+0000 7f78a7400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78a00ce0e0 0x7f78a00a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:16.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.527+0000 7f78a7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a00cead0 0x7f78a00c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:16.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.527+0000 7f78a7400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a0160be0 0x7f78a00cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:16.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.527+0000 7f78a7400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a009d3d0 con 0x7f78a00cead0 2024-04-02T07:54:16.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.527+0000 7f78a7400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00a0100 con 0x7f78a0160be0 2024-04-02T07:54:16.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.527+0000 7f78a7400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a01f34a0 con 0x7f78a00ce0e0 2024-04-02T07:54:16.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.528+0000 7f789fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a00cead0 0x7f78a00c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:16.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.528+0000 7f789fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a00cead0 0x7f78a00c9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58832/0 (socket says 172.21.15.89:58832) 2024-04-02T07:54:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.528+0000 7f789fe00700 1 -- 172.21.15.89:0/1381246989 learned_addr learned my addr 172.21.15.89:0/1381246989 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.528+0000 7f78a4a00700 1 --2- 172.21.15.89:0/1381246989 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78a00ce0e0 0x7f78a00a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.528+0000 7f78a5400700 1 --2- 172.21.15.89:0/1381246989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a0160be0 0x7f78a00cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.528+0000 7f789fe00700 1 -- 172.21.15.89:0/1381246989 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78a00ce0e0 msgr2=0x7f78a00a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.528+0000 7f789fe00700 1 --2- 172.21.15.89:0/1381246989 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78a00ce0e0 0x7f78a00a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.528+0000 7f789fe00700 1 -- 172.21.15.89:0/1381246989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a0160be0 msgr2=0x7f78a00cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.528+0000 7f789fe00700 1 --2- 172.21.15.89:0/1381246989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a0160be0 0x7f78a00cc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.528+0000 7f789fe00700 1 -- 172.21.15.89:0/1381246989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7894009550 con 0x7f78a00cead0 2024-04-02T07:54:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.528+0000 7f78a5400700 1 --2- 172.21.15.89:0/1381246989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a0160be0 0x7f78a00cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.528+0000 7f789fe00700 1 --2- 172.21.15.89:0/1381246989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a00cead0 0x7f78a00c9bb0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f788c00c750 tx=0x7f788c00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.529+0000 7f789cc00700 1 -- 172.21.15.89:0/1381246989 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c011800 con 0x7f78a00cead0 2024-04-02T07:54:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.529+0000 7f789cc00700 1 -- 172.21.15.89:0/1381246989 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788c011960 con 0x7f78a00cead0 2024-04-02T07:54:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.529+0000 7f789cc00700 1 -- 172.21.15.89:0/1381246989 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c011cd0 con 0x7f78a00cead0 2024-04-02T07:54:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.529+0000 7f78a7400700 1 -- 172.21.15.89:0/1381246989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a01f3780 con 0x7f78a00cead0 2024-04-02T07:54:16.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.529+0000 7f78a7400700 1 -- 172.21.15.89:0/1381246989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a01f3d40 con 0x7f78a00cead0 2024-04-02T07:54:16.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.530+0000 7f78a7400700 1 -- 172.21.15.89:0/1381246989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a009d510 con 0x7f78a00cead0 2024-04-02T07:54:16.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.531+0000 7f789cc00700 1 -- 172.21.15.89:0/1381246989 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 62) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f788c021350 con 0x7f78a00cead0 2024-04-02T07:54:16.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.532+0000 7f789cc00700 1 -- 172.21.15.89:0/1381246989 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f788c089170 con 0x7f78a00cead0 2024-04-02T07:54:16.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.537+0000 7f789cc00700 1 -- 172.21.15.89:0/1381246989 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f78a009d510 con 0x7f78a00cead0 2024-04-02T07:54:16.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.740+0000 7f78a7400700 1 -- 172.21.15.89:0/1381246989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78a009d510 con 0x7f78a00cead0 2024-04-02T07:54:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.742+0000 7f789cc00700 1 -- 172.21.15.89:0/1381246989 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+17622 (secure 0 0 0) 0x7f78a009d510 con 0x7f78a00cead0 2024-04-02T07:54:16.744 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:16.744 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":113,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:54:16.261655+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:52:46.885836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-02T07:52:47.989359+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/1829707496":"2024-04-03T07:54:16.261615+0000","172.21.15.89:0/311592371":"2024-04-03T07:54:16.261615+0000","172.21.15.89:0/21462376":"2024-04-03T07:54:16.261615+0000","172.21.15.89:0/1368039270":"2024-04-03T07:54:16.261615+0000","172.21.15.89:0/2117754594":"2024-04-03T07:54:16.261615+0000","172.21.15.89:0/3703441222":"2024-04-03T07:54:16.261615+0000","172.21.15.89:0/1115793129":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/814322436":"2024-04-03T07:54:16.261615+0000","172.21.15.89:0/458120672":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/2657927819":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/383704566":"2024-04-03T07:53:05.735155+0000","172.21.15.89:6832/43357":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/1938634818":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/4197840578":"2024-04-03T07:54:16.261615+0000","172.21.15.89:6833/43357":"2024-04-03T07:53:05.735155+0000","172.21.15.89:0/552112415":"2024-04-03T07:53:05.735155+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:54:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.744+0000 7f78a7400700 1 -- 172.21.15.89:0/1381246989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a00cead0 msgr2=0x7f78a00c9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.744+0000 7f78a7400700 1 --2- 172.21.15.89:0/1381246989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a00cead0 0x7f78a00c9bb0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f788c00c750 tx=0x7f788c00ca60 comp rx=0 tx=0).stop 2024-04-02T07:54:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.744+0000 7f78a7400700 1 -- 172.21.15.89:0/1381246989 shutdown_connections 2024-04-02T07:54:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.744+0000 7f78a7400700 1 --2- 172.21.15.89:0/1381246989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a0160be0 0x7f78a00cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.744+0000 7f78a7400700 1 --2- 172.21.15.89:0/1381246989 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78a00ce0e0 0x7f78a00a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.744+0000 7f78a7400700 1 --2- 172.21.15.89:0/1381246989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78a00cead0 0x7f78a00c9bb0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.744+0000 7f78a7400700 1 -- 172.21.15.89:0/1381246989 >> 172.21.15.89:0/1381246989 conn(0x7f78a008aff0 msgr2=0x7f78a015f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.744+0000 7f78a7400700 1 -- 172.21.15.89:0/1381246989 shutdown_connections 2024-04-02T07:54:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:16.744+0000 7f78a7400700 1 -- 172.21.15.89:0/1381246989 wait complete. 2024-04-02T07:54:16.764 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T07:54:16.867 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:54:16.871 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:54:17.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.002+0000 7f97b9800700 1 Processor -- start 2024-04-02T07:54:17.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.002+0000 7f97b9800700 1 -- start start 2024-04-02T07:54:17.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.003+0000 7f97b9800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f97b413fd80 0x7f97b40714c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:17.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.003+0000 7f97b9800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b41401f0 con 0x7f97b413d040 2024-04-02T07:54:17.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.003+0000 7f97b9800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b40ce650 con 0x7f97b413f8d0 2024-04-02T07:54:17.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.003+0000 7f97b9800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97b40ce930 con 0x7f97b413fd80 2024-04-02T07:54:17.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.004+0000 7f97b3e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f97b413d040 0x7f97b40cdb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33638/0 (socket says 172.21.15.89:33638) 2024-04-02T07:54:17.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.004+0000 7f97b3e00700 1 -- 172.21.15.89:0/27325858 learned_addr learned my addr 172.21.15.89:0/27325858 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:17.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.004+0000 7f97b8e00700 1 --2- 172.21.15.89:0/27325858 >> v2:172.21.15.89:3300/0 conn(0x7f97b413fd80 0x7f97b40714c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.004+0000 7f97b3400700 1 -- 172.21.15.89:0/27325858 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308236898 0 0) 0x7f97b41401f0 con 0x7f97b413d040 2024-04-02T07:54:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.004+0000 7f97b3400700 1 -- 172.21.15.89:0/27325858 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97980035b0 con 0x7f97b413d040 2024-04-02T07:54:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.004+0000 7f97b3400700 1 -- 172.21.15.89:0/27325858 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029354533 0 0) 0x7f97b40ce650 con 0x7f97b413f8d0 2024-04-02T07:54:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.004+0000 7f97b3400700 1 -- 172.21.15.89:0/27325858 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b41401f0 con 0x7f97b413f8d0 2024-04-02T07:54:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.004+0000 7f97b8e00700 1 -- 172.21.15.89:0/27325858 >> v1:172.21.15.113:6789/0 conn(0x7f97b413f8d0 legacy=0x7f97b4070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.004+0000 7f97b8e00700 1 -- 172.21.15.89:0/27325858 >> v1:172.21.15.89:6789/0 conn(0x7f97b413d040 legacy=0x7f97b40cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.004+0000 7f97b8e00700 1 -- 172.21.15.89:0/27325858 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b40cec10 con 0x7f97b413fd80 2024-04-02T07:54:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.004+0000 7f97b8e00700 1 --2- 172.21.15.89:0/27325858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b413fd80 0x7f97b40714c0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f97ac009d90 tx=0x7f97ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9389ada619d4d6e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.005+0000 7f97b3400700 1 -- 172.21.15.89:0/27325858 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac00e070 con 0x7f97b413fd80 2024-04-02T07:54:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.005+0000 7f97b3400700 1 -- 172.21.15.89:0/27325858 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ac00bb20 con 0x7f97b413fd80 2024-04-02T07:54:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.005+0000 7f97b3400700 1 -- 172.21.15.89:0/27325858 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac00be90 con 0x7f97b413fd80 2024-04-02T07:54:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.005+0000 7f97b9800700 1 -- 172.21.15.89:0/27325858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b413fd80 msgr2=0x7f97b40714c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.005+0000 7f97b9800700 1 --2- 172.21.15.89:0/27325858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b413fd80 0x7f97b40714c0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f97ac009d90 tx=0x7f97ac00b330 comp rx=0 tx=0).stop 2024-04-02T07:54:17.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.005+0000 7f97b9800700 1 -- 172.21.15.89:0/27325858 shutdown_connections 2024-04-02T07:54:17.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.005+0000 7f97b9800700 1 --2- 172.21.15.89:0/27325858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b413fd80 0x7f97b40714c0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:17.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.005+0000 7f97b9800700 1 -- 172.21.15.89:0/27325858 >> 172.21.15.89:0/27325858 conn(0x7f97b406df90 msgr2=0x7f97b406a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:17.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.005+0000 7f97b9800700 1 -- 172.21.15.89:0/27325858 shutdown_connections 2024-04-02T07:54:17.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.005+0000 7f97b9800700 1 -- 172.21.15.89:0/27325858 wait complete. 2024-04-02T07:54:17.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.007+0000 7f97b9800700 1 Processor -- start 2024-04-02T07:54:17.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.007+0000 7f97b9800700 1 -- start start 2024-04-02T07:54:17.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.008+0000 7f97b9800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97b413d040 0x7f97b413b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:17.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.008+0000 7f97b9800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97b413f8d0 0x7f97b413b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:17.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.008+0000 7f97b9800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b413fd80 0x7f97b40af570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:17.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.008+0000 7f97b9800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b4136ad0 con 0x7f97b413fd80 2024-04-02T07:54:17.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.008+0000 7f97b9800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b415c050 con 0x7f97b413d040 2024-04-02T07:54:17.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.008+0000 7f97b9800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b40b0150 con 0x7f97b413f8d0 2024-04-02T07:54:17.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.009+0000 7f97b8e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b413fd80 0x7f97b40af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:17.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.009+0000 7f97b8e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b413fd80 0x7f97b40af570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58856/0 (socket says 172.21.15.89:58856) 2024-04-02T07:54:17.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.009+0000 7f97b3e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97b413d040 0x7f97b413b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:17.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.009+0000 7f97b8e00700 1 -- 172.21.15.89:0/343320835 learned_addr learned my addr 172.21.15.89:0/343320835 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:17.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.009+0000 7f97abe00700 1 --2- 172.21.15.89:0/343320835 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97b413f8d0 0x7f97b413b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:17.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.009+0000 7f97b8e00700 1 -- 172.21.15.89:0/343320835 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97b413f8d0 msgr2=0x7f97b413b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:17.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.009+0000 7f97b8e00700 1 --2- 172.21.15.89:0/343320835 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97b413f8d0 0x7f97b413b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:17.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.009+0000 7f97b8e00700 1 -- 172.21.15.89:0/343320835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97b413d040 msgr2=0x7f97b413b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:17.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.009+0000 7f97b8e00700 1 --2- 172.21.15.89:0/343320835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97b413d040 0x7f97b413b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:17.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.009+0000 7f97b8e00700 1 -- 172.21.15.89:0/343320835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ac009a70 con 0x7f97b413fd80 2024-04-02T07:54:17.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.009+0000 7f97b8e00700 1 --2- 172.21.15.89:0/343320835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b413fd80 0x7f97b40af570 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f97b406af40 tx=0x7f97ac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:17.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.009+0000 7f97b0c00700 1 -- 172.21.15.89:0/343320835 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0048c0 con 0x7f97b413fd80 2024-04-02T07:54:17.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.010+0000 7f97b0c00700 1 -- 172.21.15.89:0/343320835 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ac020070 con 0x7f97b413fd80 2024-04-02T07:54:17.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.010+0000 7f97b0c00700 1 -- 172.21.15.89:0/343320835 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0276c0 con 0x7f97b413fd80 2024-04-02T07:54:17.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.010+0000 7f97b9800700 1 -- 172.21.15.89:0/343320835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b40b0430 con 0x7f97b413fd80 2024-04-02T07:54:17.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.010+0000 7f97b9800700 1 -- 172.21.15.89:0/343320835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b415bce0 con 0x7f97b413fd80 2024-04-02T07:54:17.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.011+0000 7f97b9800700 1 -- 172.21.15.89:0/343320835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b415c1c0 con 0x7f97b413fd80 2024-04-02T07:54:17.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.012+0000 7f97b0c00700 1 -- 172.21.15.89:0/343320835 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 62) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f97ac012430 con 0x7f97b413fd80 2024-04-02T07:54:17.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.013+0000 7f97b0c00700 1 -- 172.21.15.89:0/343320835 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f97ac022070 con 0x7f97b413fd80 2024-04-02T07:54:17.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.018+0000 7f97b0c00700 1 -- 172.21.15.89:0/343320835 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f97ac056040 con 0x7f97b413fd80 2024-04-02T07:54:17.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.262+0000 7f97b9800700 1 -- 172.21.15.89:0/343320835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f97b404ea80 con 0x7f97b413fd80 2024-04-02T07:54:17.264 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T07:54:17.263+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T07:54:17.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.275+0000 7f97b0c00700 1 -- 172.21.15.89:0/343320835 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 63) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f97ac055af0 con 0x7f97b413fd80 2024-04-02T07:54:17.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.275+0000 7f97b0c00700 1 --2- 172.21.15.89:0/343320835 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9790036150 0x7f97900726c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:17.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.277+0000 7f97b0c00700 1 -- 172.21.15.89:0/343320835 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v50) v1 ==== 161+0+0 (secure 0 0 0) 0x7f97ac0834c0 con 0x7f97b413fd80 2024-04-02T07:54:17.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.277+0000 7f97b3e00700 1 --2- 172.21.15.89:0/343320835 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9790036150 0x7f97900726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:17.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.281+0000 7f97b9800700 1 -- 172.21.15.89:0/343320835 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9790036150 msgr2=0x7f97900726c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:17.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.281+0000 7f97b9800700 1 --2- 172.21.15.89:0/343320835 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9790036150 0x7f97900726c0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f97a4007fd0 tx=0x7f97a4007e00 comp rx=0 tx=0).stop 2024-04-02T07:54:17.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.281+0000 7f97b9800700 1 -- 172.21.15.89:0/343320835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b413fd80 msgr2=0x7f97b40af570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:17.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.281+0000 7f97b9800700 1 --2- 172.21.15.89:0/343320835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b413fd80 0x7f97b40af570 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f97b406af40 tx=0x7f97ac004ad0 comp rx=0 tx=0).stop 2024-04-02T07:54:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.282+0000 7f97b9800700 1 -- 172.21.15.89:0/343320835 shutdown_connections 2024-04-02T07:54:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.282+0000 7f97b9800700 1 --2- 172.21.15.89:0/343320835 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9790036150 0x7f97900726c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.282+0000 7f97b9800700 1 --2- 172.21.15.89:0/343320835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97b413d040 0x7f97b413b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.282+0000 7f97b9800700 1 --2- 172.21.15.89:0/343320835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97b413fd80 0x7f97b40af570 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.282+0000 7f97b9800700 1 --2- 172.21.15.89:0/343320835 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97b413f8d0 0x7f97b413b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.282+0000 7f97b9800700 1 -- 172.21.15.89:0/343320835 >> 172.21.15.89:0/343320835 conn(0x7f97b406df90 msgr2=0x7f97b4159f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.283+0000 7f97b9800700 1 -- 172.21.15.89:0/343320835 shutdown_connections 2024-04-02T07:54:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.283+0000 7f97b9800700 1 -- 172.21.15.89:0/343320835 wait complete. 2024-04-02T07:54:17.286 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T07:54:17.294 INFO:tasks.ceph.mds.a.smithi089.stderr: -8240> 2024-04-02T07:52:54.460+0000 7fa6dc800700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T07:54:17.312 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T07:54:17.355 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T07:54:17.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.505+0000 7fdec1600700 1 Processor -- start 2024-04-02T07:54:17.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.505+0000 7fdec1600700 1 -- start start 2024-04-02T07:54:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.506+0000 7fdec1600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdebc0d16a0 0x7fdebc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.506+0000 7fdec1600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdebc161420 con 0x7fdebc15d9d0 2024-04-02T07:54:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.506+0000 7fdec1600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdebc161700 con 0x7fdebc0d2370 2024-04-02T07:54:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.506+0000 7fdec1600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdebc0d4340 con 0x7fdebc0d16a0 2024-04-02T07:54:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.506+0000 7fdebaa00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdebc15d9d0 0x7fdebc0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33642/0 (socket says 172.21.15.89:33642) 2024-04-02T07:54:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.506+0000 7fdebaa00700 1 -- 172.21.15.89:0/3493174723 learned_addr learned my addr 172.21.15.89:0/3493174723 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.506+0000 7fdeba000700 1 --2- 172.21.15.89:0/3493174723 >> v2:172.21.15.113:3300/0 conn(0x7fdebc0d16a0 0x7fdebc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:17.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.507+0000 7fdeba000700 1 -- 172.21.15.89:0/3493174723 >> v1:172.21.15.113:6789/0 conn(0x7fdebc0d2370 legacy=0x7fdebc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:17.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.507+0000 7fdeb9600700 1 -- 172.21.15.89:0/3493174723 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986680866 0 0) 0x7fdebc161420 con 0x7fdebc15d9d0 2024-04-02T07:54:17.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.507+0000 7fdeba000700 1 -- 172.21.15.89:0/3493174723 >> v1:172.21.15.89:6789/0 conn(0x7fdebc15d9d0 legacy=0x7fdebc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:17.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.507+0000 7fdeba000700 1 -- 172.21.15.89:0/3493174723 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdebc1619e0 con 0x7fdebc0d16a0 2024-04-02T07:54:17.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.507+0000 7fdeba000700 1 --2- 172.21.15.89:0/3493174723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc0d16a0 0x7fdebc0d1e30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdea80097a0 tx=0x7fdea8009ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a899b23362a09785 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:17.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.507+0000 7fdeb9600700 1 -- 172.21.15.89:0/3493174723 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8005e30 con 0x7fdebc0d16a0 2024-04-02T07:54:17.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.507+0000 7fdeb9600700 1 -- 172.21.15.89:0/3493174723 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea8004030 con 0x7fdebc0d16a0 2024-04-02T07:54:17.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.507+0000 7fdeb9600700 1 -- 172.21.15.89:0/3493174723 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea80043e0 con 0x7fdebc0d16a0 2024-04-02T07:54:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.508+0000 7fdec1600700 1 -- 172.21.15.89:0/3493174723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc0d16a0 msgr2=0x7fdebc0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.508+0000 7fdec1600700 1 --2- 172.21.15.89:0/3493174723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc0d16a0 0x7fdebc0d1e30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdea80097a0 tx=0x7fdea8009ab0 comp rx=0 tx=0).stop 2024-04-02T07:54:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.508+0000 7fdec1600700 1 -- 172.21.15.89:0/3493174723 shutdown_connections 2024-04-02T07:54:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.508+0000 7fdec1600700 1 --2- 172.21.15.89:0/3493174723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc0d16a0 0x7fdebc0d1e30 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.508+0000 7fdec1600700 1 -- 172.21.15.89:0/3493174723 >> 172.21.15.89:0/3493174723 conn(0x7fdebc08aff0 msgr2=0x7fdebc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.508+0000 7fdec1600700 1 -- 172.21.15.89:0/3493174723 shutdown_connections 2024-04-02T07:54:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.508+0000 7fdec1600700 1 -- 172.21.15.89:0/3493174723 wait complete. 2024-04-02T07:54:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.509+0000 7fdec1600700 1 Processor -- start 2024-04-02T07:54:17.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.510+0000 7fdec1600700 1 -- start start 2024-04-02T07:54:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.510+0000 7fdec1600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdebc0d16a0 0x7fdebc0b53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.510+0000 7fdec1600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdebc0d2370 0x7fdebc153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.510+0000 7fdec1600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc15d9d0 0x7fdebc1e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdec1600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc0cd490 con 0x7fdebc0d2370 2024-04-02T07:54:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdec1600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc1e8670 con 0x7fdebc15d9d0 2024-04-02T07:54:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdec1600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc1e77c0 con 0x7fdebc0d16a0 2024-04-02T07:54:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdeba000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdebc0d2370 0x7fdebc153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdeba000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdebc0d2370 0x7fdebc153340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58876/0 (socket says 172.21.15.89:58876) 2024-04-02T07:54:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdeba000700 1 -- 172.21.15.89:0/1169842370 learned_addr learned my addr 172.21.15.89:0/1169842370 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdebaa00700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdebc0d16a0 0x7fdebc0b53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdebb400700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc15d9d0 0x7fdebc1e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdeba000700 1 -- 172.21.15.89:0/1169842370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdebc0d16a0 msgr2=0x7fdebc0b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdeba000700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdebc0d16a0 0x7fdebc0b53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdeba000700 1 -- 172.21.15.89:0/1169842370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc15d9d0 msgr2=0x7fdebc1e7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdeba000700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc15d9d0 0x7fdebc1e7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdeba000700 1 -- 172.21.15.89:0/1169842370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea8009430 con 0x7fdebc0d2370 2024-04-02T07:54:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdebaa00700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdebc0d16a0 0x7fdebc0b53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdebb400700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc15d9d0 0x7fdebc1e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.511+0000 7fdeba000700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdebc0d2370 0x7fdebc153340 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fdea8014040 tx=0x7fdea8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.512+0000 7fdeb2a00700 1 -- 172.21.15.89:0/1169842370 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea801d630 con 0x7fdebc0d2370 2024-04-02T07:54:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.512+0000 7fdeb2a00700 1 -- 172.21.15.89:0/1169842370 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea801d790 con 0x7fdebc0d2370 2024-04-02T07:54:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.512+0000 7fdec1600700 1 -- 172.21.15.89:0/1169842370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdebc1e7a60 con 0x7fdebc0d2370 2024-04-02T07:54:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.512+0000 7fdeb2a00700 1 -- 172.21.15.89:0/1169842370 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea801db40 con 0x7fdebc0d2370 2024-04-02T07:54:17.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.512+0000 7fdec1600700 1 -- 172.21.15.89:0/1169842370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdebc1e8020 con 0x7fdebc0d2370 2024-04-02T07:54:17.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.513+0000 7fdec1600700 1 -- 172.21.15.89:0/1169842370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdebc0d1ca0 con 0x7fdebc0d2370 2024-04-02T07:54:17.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.514+0000 7fdeb2a00700 1 -- 172.21.15.89:0/1169842370 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 63) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fdea801dca0 con 0x7fdebc0d2370 2024-04-02T07:54:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.514+0000 7fdeb2a00700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fde9c06aae0 0x7fde9c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.515+0000 7fdeb2a00700 1 -- 172.21.15.89:0/1169842370 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fdea808a910 con 0x7fdebc0d2370 2024-04-02T07:54:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.515+0000 7fdebaa00700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fde9c06aae0 0x7fde9c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.515+0000 7fdebaa00700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fde9c06aae0 0x7fde9c06cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdeac007fd0 tx=0x7fdeac009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.519+0000 7fdeb2a00700 1 -- 172.21.15.89:0/1169842370 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdea8055900 con 0x7fdebc0d2370 2024-04-02T07:54:17.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:17.765+0000 7fdec1600700 1 -- 172.21.15.89:0/1169842370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdebc1e87e0 con 0x7fdebc0d2370 2024-04-02T07:54:18.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.276+0000 7fdeb2a00700 1 -- 172.21.15.89:0/1169842370 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v51) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdea8058f20 con 0x7fdebc0d2370 2024-04-02T07:54:18.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.280+0000 7fdec1600700 1 -- 172.21.15.89:0/1169842370 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fde9c06aae0 msgr2=0x7fde9c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:18.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.280+0000 7fdec1600700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fde9c06aae0 0x7fde9c06cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdeac007fd0 tx=0x7fdeac009040 comp rx=0 tx=0).stop 2024-04-02T07:54:18.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.280+0000 7fdec1600700 1 -- 172.21.15.89:0/1169842370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdebc0d2370 msgr2=0x7fdebc153340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:18.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.280+0000 7fdec1600700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdebc0d2370 0x7fdebc153340 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fdea8014040 tx=0x7fdea8004750 comp rx=0 tx=0).stop 2024-04-02T07:54:18.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.280+0000 7fdec1600700 1 -- 172.21.15.89:0/1169842370 shutdown_connections 2024-04-02T07:54:18.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.280+0000 7fdec1600700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fde9c06aae0 0x7fde9c06cfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:18.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.280+0000 7fdec1600700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc15d9d0 0x7fdebc1e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:18.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.280+0000 7fdec1600700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdebc0d16a0 0x7fdebc0b53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:18.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.280+0000 7fdec1600700 1 --2- 172.21.15.89:0/1169842370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdebc0d2370 0x7fdebc153340 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:18.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.280+0000 7fdec1600700 1 -- 172.21.15.89:0/1169842370 >> 172.21.15.89:0/1169842370 conn(0x7fdebc08aff0 msgr2=0x7fdebc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:18.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.280+0000 7fdec1600700 1 -- 172.21.15.89:0/1169842370 shutdown_connections 2024-04-02T07:54:18.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.280+0000 7fdec1600700 1 -- 172.21.15.89:0/1169842370 wait complete. 2024-04-02T07:54:18.302 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T07:54:18.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.536+0000 7f0fd6600700 1 Processor -- start 2024-04-02T07:54:18.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.536+0000 7f0fd6600700 1 -- start start 2024-04-02T07:54:18.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.537+0000 7f0fd6600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0fd015ddf0 0x7f0fd00ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:18.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.537+0000 7f0fd6600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0fd015e1d0 0x7f0fd00d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:18.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.537+0000 7f0fd6600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0fd00d2710 0x7f0fd01612f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:18.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.537+0000 7f0fd6600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd00d2b80 con 0x7f0fd00d2710 2024-04-02T07:54:18.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.538+0000 7f0fd6600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd015ec60 con 0x7f0fd015e1d0 2024-04-02T07:54:18.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.538+0000 7f0fd6600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd00ce8a0 con 0x7f0fd015ddf0 2024-04-02T07:54:18.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.538+0000 7f0fcfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0fd00d2710 0x7f0fd01612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:18.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.538+0000 7f0fcfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0fd00d2710 0x7f0fd01612f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58910/0 (socket says 172.21.15.89:58910) 2024-04-02T07:54:18.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.538+0000 7f0fcfe00700 1 -- 172.21.15.89:0/2877357828 learned_addr learned my addr 172.21.15.89:0/2877357828 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:18.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.538+0000 7f0fcea00700 1 --2- 172.21.15.89:0/2877357828 >> v2:172.21.15.113:3300/0 conn(0x7f0fd015e1d0 0x7f0fd00d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:18.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.538+0000 7f0fcf400700 1 --2- 172.21.15.89:0/2877357828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0fd015ddf0 0x7f0fd00ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:18.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.538+0000 7f0fcfe00700 1 -- 172.21.15.89:0/2877357828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0fd015ddf0 msgr2=0x7f0fd00ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:18.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.538+0000 7f0fcfe00700 1 --2- 172.21.15.89:0/2877357828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0fd015ddf0 0x7f0fd00ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:18.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.538+0000 7f0fcfe00700 1 -- 172.21.15.89:0/2877357828 >> v2:172.21.15.113:3300/0 conn(0x7f0fd015e1d0 msgr2=0x7f0fd00d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:18.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.538+0000 7f0fcfe00700 1 --2- 172.21.15.89:0/2877357828 >> v2:172.21.15.113:3300/0 conn(0x7f0fd015e1d0 0x7f0fd00d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:18.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.538+0000 7f0fcfe00700 1 -- 172.21.15.89:0/2877357828 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd00cdc90 con 0x7f0fd00d2710 2024-04-02T07:54:18.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.538+0000 7f0fcea00700 1 --2- 172.21.15.89:0/2877357828 >> v2:172.21.15.113:3300/0 conn(0x7f0fd015e1d0 0x7f0fd00d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:18.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.538+0000 7f0fcfe00700 1 --2- 172.21.15.89:0/2877357828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fd00d2710 0x7f0fd01612f0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f0fd008b840 tx=0x7f0fc800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f74cf519ddba818 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:18.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.539+0000 7f0fce000700 1 -- 172.21.15.89:0/2877357828 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc800e070 con 0x7f0fd00d2710 2024-04-02T07:54:18.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.539+0000 7f0fce000700 1 -- 172.21.15.89:0/2877357828 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc800b9c0 con 0x7f0fd00d2710 2024-04-02T07:54:18.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.539+0000 7f0fce000700 1 -- 172.21.15.89:0/2877357828 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc800bd70 con 0x7f0fd00d2710 2024-04-02T07:54:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.539+0000 7f0fd6600700 1 -- 172.21.15.89:0/2877357828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fd00d2710 msgr2=0x7f0fd01612f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.539+0000 7f0fd6600700 1 --2- 172.21.15.89:0/2877357828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fd00d2710 0x7f0fd01612f0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f0fd008b840 tx=0x7f0fc800b330 comp rx=0 tx=0).stop 2024-04-02T07:54:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.539+0000 7f0fd6600700 1 -- 172.21.15.89:0/2877357828 shutdown_connections 2024-04-02T07:54:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.539+0000 7f0fd6600700 1 --2- 172.21.15.89:0/2877357828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fd00d2710 0x7f0fd01612f0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:18.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.539+0000 7f0fd6600700 1 --2- 172.21.15.89:0/2877357828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0fd015ddf0 0x7f0fd00ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.539+0000 7f0fd6600700 1 --2- 172.21.15.89:0/2877357828 >> v2:172.21.15.113:3300/0 conn(0x7f0fd015e1d0 0x7f0fd00d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.539+0000 7f0fd6600700 1 -- 172.21.15.89:0/2877357828 >> 172.21.15.89:0/2877357828 conn(0x7f0fd008aff0 msgr2=0x7f0fd00d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.539+0000 7f0fd6600700 1 -- 172.21.15.89:0/2877357828 shutdown_connections 2024-04-02T07:54:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.540+0000 7f0fd6600700 1 -- 172.21.15.89:0/2877357828 wait complete. 2024-04-02T07:54:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.541+0000 7f0fd6600700 1 Processor -- start 2024-04-02T07:54:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.541+0000 7f0fd6600700 1 -- start start 2024-04-02T07:54:18.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.542+0000 7f0fd6600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0fd00d2710 0x7f0fd00c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:18.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.542+0000 7f0fd6600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fd015ddf0 0x7f0fd00c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:18.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.542+0000 7f0fd6600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fd015e1d0 0x7f0fd014b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.542+0000 7f0fd6600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd00c9eb0 con 0x7f0fd015ddf0 2024-04-02T07:54:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.542+0000 7f0fd6600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd015ff50 con 0x7f0fd015e1d0 2024-04-02T07:54:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.542+0000 7f0fd6600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd0160230 con 0x7f0fd00d2710 2024-04-02T07:54:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.542+0000 7f0fcea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fd015ddf0 0x7f0fd00c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.543+0000 7f0fcea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fd015ddf0 0x7f0fd00c9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58918/0 (socket says 172.21.15.89:58918) 2024-04-02T07:54:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.543+0000 7f0fcea00700 1 -- 172.21.15.89:0/397564415 learned_addr learned my addr 172.21.15.89:0/397564415 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.543+0000 7f0fcf400700 1 --2- 172.21.15.89:0/397564415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0fd00d2710 0x7f0fd00c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.543+0000 7f0fcfe00700 1 --2- 172.21.15.89:0/397564415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fd015e1d0 0x7f0fd014b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.543+0000 7f0fcea00700 1 -- 172.21.15.89:0/397564415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0fd00d2710 msgr2=0x7f0fd00c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.543+0000 7f0fcea00700 1 --2- 172.21.15.89:0/397564415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0fd00d2710 0x7f0fd00c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.543+0000 7f0fcea00700 1 -- 172.21.15.89:0/397564415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fd015e1d0 msgr2=0x7f0fd014b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.543+0000 7f0fcea00700 1 --2- 172.21.15.89:0/397564415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fd015e1d0 0x7f0fd014b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:18.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.543+0000 7f0fcea00700 1 -- 172.21.15.89:0/397564415 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc8009b30 con 0x7f0fd015ddf0 2024-04-02T07:54:18.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.543+0000 7f0fcea00700 1 --2- 172.21.15.89:0/397564415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fd015ddf0 0x7f0fd00c9970 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00e760 tx=0x7f0fbc00ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:18.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.543+0000 7f0fc7400700 1 -- 172.21.15.89:0/397564415 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc00c070 con 0x7f0fd015ddf0 2024-04-02T07:54:18.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.543+0000 7f0fc7400700 1 -- 172.21.15.89:0/397564415 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fbc013610 con 0x7f0fd015ddf0 2024-04-02T07:54:18.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.543+0000 7f0fd6600700 1 -- 172.21.15.89:0/397564415 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd0160510 con 0x7f0fd015ddf0 2024-04-02T07:54:18.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.543+0000 7f0fc7400700 1 -- 172.21.15.89:0/397564415 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc0139c0 con 0x7f0fd015ddf0 2024-04-02T07:54:18.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.544+0000 7f0fd6600700 1 -- 172.21.15.89:0/397564415 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd014be70 con 0x7f0fd015ddf0 2024-04-02T07:54:18.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.545+0000 7f0fd6600700 1 -- 172.21.15.89:0/397564415 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd00b24a0 con 0x7f0fd015ddf0 2024-04-02T07:54:18.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.545+0000 7f0fc7400700 1 -- 172.21.15.89:0/397564415 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 63) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f0fbc013b20 con 0x7f0fd015ddf0 2024-04-02T07:54:18.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.546+0000 7f0fc7400700 1 --2- 172.21.15.89:0/397564415 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0fb006aae0 0x7f0fb006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:18.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.546+0000 7f0fc7400700 1 -- 172.21.15.89:0/397564415 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f0fbc0890f0 con 0x7f0fd015ddf0 2024-04-02T07:54:18.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.547+0000 7f0fcf400700 1 --2- 172.21.15.89:0/397564415 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0fb006aae0 0x7f0fb006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:18.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.547+0000 7f0fcf400700 1 --2- 172.21.15.89:0/397564415 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0fb006aae0 0x7f0fb006cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009950 tx=0x7f0fc0008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:18.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.551+0000 7f0fc7400700 1 -- 172.21.15.89:0/397564415 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0fbc0540e0 con 0x7f0fd015ddf0 2024-04-02T07:54:18.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:18.750+0000 7f0fd6600700 1 -- 172.21.15.89:0/397564415 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0fd0067720 con 0x7f0fd015ddf0 2024-04-02T07:54:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.153+0000 7f0fc7400700 1 -- 172.21.15.89:0/397564415 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 64) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f0fbc0544b0 con 0x7f0fd015ddf0 2024-04-02T07:54:19.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.286+0000 7f0fc7400700 1 -- 172.21.15.89:0/397564415 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v115) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0fbc084b80 con 0x7f0fd015ddf0 2024-04-02T07:54:19.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.289+0000 7f0fd6600700 1 -- 172.21.15.89:0/397564415 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0fb006aae0 msgr2=0x7f0fb006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:19.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.289+0000 7f0fd6600700 1 --2- 172.21.15.89:0/397564415 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0fb006aae0 0x7f0fb006cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009950 tx=0x7f0fc0008040 comp rx=0 tx=0).stop 2024-04-02T07:54:19.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.289+0000 7f0fd6600700 1 -- 172.21.15.89:0/397564415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fd015ddf0 msgr2=0x7f0fd00c9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:19.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.289+0000 7f0fd6600700 1 --2- 172.21.15.89:0/397564415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fd015ddf0 0x7f0fd00c9970 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00e760 tx=0x7f0fbc00ea70 comp rx=0 tx=0).stop 2024-04-02T07:54:19.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.289+0000 7f0fd6600700 1 -- 172.21.15.89:0/397564415 shutdown_connections 2024-04-02T07:54:19.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.289+0000 7f0fd6600700 1 --2- 172.21.15.89:0/397564415 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0fb006aae0 0x7f0fb006cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:19.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.289+0000 7f0fd6600700 1 --2- 172.21.15.89:0/397564415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fd015e1d0 0x7f0fd014b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:19.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.289+0000 7f0fd6600700 1 --2- 172.21.15.89:0/397564415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0fd00d2710 0x7f0fd00c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:19.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.289+0000 7f0fd6600700 1 --2- 172.21.15.89:0/397564415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fd015ddf0 0x7f0fd00c9970 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:19.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.289+0000 7f0fd6600700 1 -- 172.21.15.89:0/397564415 >> 172.21.15.89:0/397564415 conn(0x7f0fd008aff0 msgr2=0x7f0fd00d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:19.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.289+0000 7f0fd6600700 1 -- 172.21.15.89:0/397564415 shutdown_connections 2024-04-02T07:54:19.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.289+0000 7f0fd6600700 1 -- 172.21.15.89:0/397564415 wait complete. 2024-04-02T07:54:19.292 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T07:54:19.310 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T07:54:19.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.503+0000 7f7819a00700 1 Processor -- start 2024-04-02T07:54:19.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.503+0000 7f7819a00700 1 -- start start 2024-04-02T07:54:19.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.504+0000 7f7819a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78140b56e0 0x7f78140d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:19.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.504+0000 7f7819a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140ceb70 con 0x7f781415d7f0 2024-04-02T07:54:19.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.504+0000 7f7819a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140cf000 con 0x7f78140d1770 2024-04-02T07:54:19.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.504+0000 7f7819a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f781415dbd0 con 0x7f78140b56e0 2024-04-02T07:54:19.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.505+0000 7f7812a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f781415d7f0 0x7f78140b4fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39140/0 (socket says 172.21.15.89:39140) 2024-04-02T07:54:19.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.505+0000 7f7812a00700 1 -- 172.21.15.89:0/752691334 learned_addr learned my addr 172.21.15.89:0/752691334 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:19.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.505+0000 7f7812000700 1 --2- 172.21.15.89:0/752691334 >> v2:172.21.15.89:3300/0 conn(0x7f78140b56e0 0x7f78140d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:19.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.505+0000 7f7811600700 1 -- 172.21.15.89:0/752691334 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444317749 0 0) 0x7f78140ceb70 con 0x7f781415d7f0 2024-04-02T07:54:19.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.505+0000 7f7811600700 1 -- 172.21.15.89:0/752691334 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77f80035b0 con 0x7f781415d7f0 2024-04-02T07:54:19.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.505+0000 7f7812000700 1 -- 172.21.15.89:0/752691334 >> v1:172.21.15.113:6789/0 conn(0x7f78140d1770 legacy=0x7f781406ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:19.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.505+0000 7f7812000700 1 -- 172.21.15.89:0/752691334 >> v1:172.21.15.89:6789/0 conn(0x7f781415d7f0 legacy=0x7f78140b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:19.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.505+0000 7f7812000700 1 -- 172.21.15.89:0/752691334 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78140cdf60 con 0x7f78140b56e0 2024-04-02T07:54:19.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.506+0000 7f7812000700 1 --2- 172.21.15.89:0/752691334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78140b56e0 0x7f78140d25d0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f7800009840 tx=0x7f7800009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b9a88e3eec5e13 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:19.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.506+0000 7f7811600700 1 -- 172.21.15.89:0/752691334 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7800011830 con 0x7f78140b56e0 2024-04-02T07:54:19.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.506+0000 7f7811600700 1 -- 172.21.15.89:0/752691334 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7800011990 con 0x7f78140b56e0 2024-04-02T07:54:19.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.506+0000 7f7811600700 1 -- 172.21.15.89:0/752691334 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7800011d20 con 0x7f78140b56e0 2024-04-02T07:54:19.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.506+0000 7f7819a00700 1 -- 172.21.15.89:0/752691334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78140b56e0 msgr2=0x7f78140d25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:19.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.506+0000 7f7819a00700 1 --2- 172.21.15.89:0/752691334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78140b56e0 0x7f78140d25d0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f7800009840 tx=0x7f7800009b50 comp rx=0 tx=0).stop 2024-04-02T07:54:19.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.506+0000 7f7819a00700 1 -- 172.21.15.89:0/752691334 shutdown_connections 2024-04-02T07:54:19.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.506+0000 7f7819a00700 1 --2- 172.21.15.89:0/752691334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78140b56e0 0x7f78140d25d0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:19.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.506+0000 7f7819a00700 1 -- 172.21.15.89:0/752691334 >> 172.21.15.89:0/752691334 conn(0x7f781408aff0 msgr2=0x7f78140d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:19.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.506+0000 7f7819a00700 1 -- 172.21.15.89:0/752691334 shutdown_connections 2024-04-02T07:54:19.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.506+0000 7f7819a00700 1 -- 172.21.15.89:0/752691334 wait complete. 2024-04-02T07:54:19.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.508+0000 7f7819a00700 1 Processor -- start 2024-04-02T07:54:19.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.508+0000 7f7819a00700 1 -- start start 2024-04-02T07:54:19.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.509+0000 7f7819a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78140b56e0 0x7f78140c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:19.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.509+0000 7f7819a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78140d1770 0x7f7814149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:19.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.509+0000 7f7819a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f781415d7f0 0x7f781415dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:19.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.509+0000 7f7819a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7814149d60 con 0x7f78140d1770 2024-04-02T07:54:19.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.509+0000 7f7819a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781415e260 con 0x7f781415d7f0 2024-04-02T07:54:19.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.509+0000 7f7819a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f781415e500 con 0x7f78140b56e0 2024-04-02T07:54:19.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.509+0000 7f7812000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78140d1770 0x7f7814149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:19.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.509+0000 7f7812000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78140d1770 0x7f7814149820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43684/0 (socket says 172.21.15.89:43684) 2024-04-02T07:54:19.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.509+0000 7f7812000700 1 -- 172.21.15.89:0/777399521 learned_addr learned my addr 172.21.15.89:0/777399521 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:19.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.510+0000 7f7812a00700 1 --2- 172.21.15.89:0/777399521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78140b56e0 0x7f78140c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:19.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.510+0000 7f7813400700 1 --2- 172.21.15.89:0/777399521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f781415d7f0 0x7f781415dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:19.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.510+0000 7f7812000700 1 -- 172.21.15.89:0/777399521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78140b56e0 msgr2=0x7f78140c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:19.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.510+0000 7f7812000700 1 --2- 172.21.15.89:0/777399521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78140b56e0 0x7f78140c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:19.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.510+0000 7f7812000700 1 -- 172.21.15.89:0/777399521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f781415d7f0 msgr2=0x7f781415dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:19.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.510+0000 7f7812000700 1 --2- 172.21.15.89:0/777399521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f781415d7f0 0x7f781415dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:19.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.510+0000 7f7812000700 1 -- 172.21.15.89:0/777399521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78000094a0 con 0x7f78140d1770 2024-04-02T07:54:19.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.510+0000 7f7812000700 1 --2- 172.21.15.89:0/777399521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78140d1770 0x7f7814149820 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f7800009b90 tx=0x7f7800004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:19.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.510+0000 7f780aa00700 1 -- 172.21.15.89:0/777399521 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78000038a0 con 0x7f78140d1770 2024-04-02T07:54:19.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.510+0000 7f780aa00700 1 -- 172.21.15.89:0/777399521 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780001e070 con 0x7f78140d1770 2024-04-02T07:54:19.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.510+0000 7f780aa00700 1 -- 172.21.15.89:0/777399521 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7800003c40 con 0x7f78140d1770 2024-04-02T07:54:19.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.510+0000 7f7819a00700 1 -- 172.21.15.89:0/777399521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781415e7e0 con 0x7f78140d1770 2024-04-02T07:54:19.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.510+0000 7f7819a00700 1 -- 172.21.15.89:0/777399521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781415eda0 con 0x7f78140d1770 2024-04-02T07:54:19.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.511+0000 7f7819a00700 1 -- 172.21.15.89:0/777399521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7814067720 con 0x7f78140d1770 2024-04-02T07:54:19.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.512+0000 7f780aa00700 1 -- 172.21.15.89:0/777399521 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 64) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f780002d450 con 0x7f78140d1770 2024-04-02T07:54:19.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.513+0000 7f780aa00700 1 --2- 172.21.15.89:0/777399521 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f77f406ad20 0x7f77f406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:19.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.513+0000 7f780aa00700 1 -- 172.21.15.89:0/777399521 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f780008a660 con 0x7f78140d1770 2024-04-02T07:54:19.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.513+0000 7f7812a00700 1 --2- 172.21.15.89:0/777399521 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f77f406ad20 0x7f77f406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:19.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.514+0000 7f7812a00700 1 --2- 172.21.15.89:0/777399521 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f77f406ad20 0x7f77f406d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f78040011e0 tx=0x7f7804009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:19.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.518+0000 7f780aa00700 1 -- 172.21.15.89:0/777399521 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7800055530 con 0x7f78140d1770 2024-04-02T07:54:19.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:19.719+0000 7f7819a00700 1 -- 172.21.15.89:0/777399521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f78140d32c0 con 0x7f78140d1770 2024-04-02T07:54:20.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.285+0000 7f780aa00700 1 -- 172.21.15.89:0/777399521 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v116) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7800058b50 con 0x7f78140d1770 2024-04-02T07:54:20.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.288+0000 7f7819a00700 1 -- 172.21.15.89:0/777399521 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f77f406ad20 msgr2=0x7f77f406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:20.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.288+0000 7f7819a00700 1 --2- 172.21.15.89:0/777399521 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f77f406ad20 0x7f77f406d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f78040011e0 tx=0x7f7804009040 comp rx=0 tx=0).stop 2024-04-02T07:54:20.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.288+0000 7f7819a00700 1 -- 172.21.15.89:0/777399521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78140d1770 msgr2=0x7f7814149820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:20.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.288+0000 7f7819a00700 1 --2- 172.21.15.89:0/777399521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78140d1770 0x7f7814149820 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f7800009b90 tx=0x7f7800004750 comp rx=0 tx=0).stop 2024-04-02T07:54:20.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.288+0000 7f7819a00700 1 -- 172.21.15.89:0/777399521 shutdown_connections 2024-04-02T07:54:20.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.288+0000 7f7819a00700 1 --2- 172.21.15.89:0/777399521 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f77f406ad20 0x7f77f406d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.288+0000 7f7819a00700 1 --2- 172.21.15.89:0/777399521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f781415d7f0 0x7f781415dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.288+0000 7f7819a00700 1 --2- 172.21.15.89:0/777399521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78140b56e0 0x7f78140c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.288+0000 7f7819a00700 1 --2- 172.21.15.89:0/777399521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78140d1770 0x7f7814149820 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.288+0000 7f7819a00700 1 -- 172.21.15.89:0/777399521 >> 172.21.15.89:0/777399521 conn(0x7f781408aff0 msgr2=0x7f78140b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:20.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.288+0000 7f7819a00700 1 -- 172.21.15.89:0/777399521 shutdown_connections 2024-04-02T07:54:20.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.288+0000 7f7819a00700 1 -- 172.21.15.89:0/777399521 wait complete. 2024-04-02T07:54:20.292 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T07:54:20.308 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T07:54:20.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.507+0000 7fc8bd800700 1 Processor -- start 2024-04-02T07:54:20.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.507+0000 7fc8bd800700 1 -- start start 2024-04-02T07:54:20.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.508+0000 7fc8bd800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8b80d2540 0x7fc8b81687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:20.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.508+0000 7fc8bd800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b8161420 con 0x7fc8b815d9d0 2024-04-02T07:54:20.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.508+0000 7fc8bd800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b8161700 con 0x7fc8b80d16a0 2024-04-02T07:54:20.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.508+0000 7fc8bd800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b80d4340 con 0x7fc8b80d2540 2024-04-02T07:54:20.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.509+0000 7fc8b7400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8b80d2540 0x7fc8b81687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:20.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.509+0000 7fc8b6a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc8b815d9d0 0x7fc8b80d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39156/0 (socket says 172.21.15.89:39156) 2024-04-02T07:54:20.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.509+0000 7fc8b6a00700 1 -- 172.21.15.89:0/1940480916 learned_addr learned my addr 172.21.15.89:0/1940480916 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:20.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.509+0000 7fc8b7400700 1 -- 172.21.15.89:0/1940480916 >> v1:172.21.15.113:6789/0 conn(0x7fc8b80d16a0 legacy=0x7fc8b80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:20.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.509+0000 7fc8b7400700 1 -- 172.21.15.89:0/1940480916 >> v1:172.21.15.89:6789/0 conn(0x7fc8b815d9d0 legacy=0x7fc8b80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:20.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.509+0000 7fc8b7400700 1 -- 172.21.15.89:0/1940480916 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b81619e0 con 0x7fc8b80d2540 2024-04-02T07:54:20.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.509+0000 7fc8b7400700 1 --2- 172.21.15.89:0/1940480916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8b80d2540 0x7fc8b81687b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0009d90 tx=0x7fc8b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ccfcd0d2a27d07 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:20.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.511+0000 7fc8b5600700 1 -- 172.21.15.89:0/1940480916 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b000e070 con 0x7fc8b80d2540 2024-04-02T07:54:20.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.511+0000 7fc8b5600700 1 -- 172.21.15.89:0/1940480916 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b000bbd0 con 0x7fc8b80d2540 2024-04-02T07:54:20.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.511+0000 7fc8b5600700 1 -- 172.21.15.89:0/1940480916 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0012430 con 0x7fc8b80d2540 2024-04-02T07:54:20.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.511+0000 7fc8bd800700 1 -- 172.21.15.89:0/1940480916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8b80d2540 msgr2=0x7fc8b81687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:20.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.511+0000 7fc8bd800700 1 --2- 172.21.15.89:0/1940480916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8b80d2540 0x7fc8b81687b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0009d90 tx=0x7fc8b000b330 comp rx=0 tx=0).stop 2024-04-02T07:54:20.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.511+0000 7fc8bd800700 1 -- 172.21.15.89:0/1940480916 shutdown_connections 2024-04-02T07:54:20.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.511+0000 7fc8bd800700 1 --2- 172.21.15.89:0/1940480916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8b80d2540 0x7fc8b81687b0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.511+0000 7fc8bd800700 1 -- 172.21.15.89:0/1940480916 >> 172.21.15.89:0/1940480916 conn(0x7fc8b808aff0 msgr2=0x7fc8b80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:20.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.511+0000 7fc8bd800700 1 -- 172.21.15.89:0/1940480916 shutdown_connections 2024-04-02T07:54:20.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.511+0000 7fc8bd800700 1 -- 172.21.15.89:0/1940480916 wait complete. 2024-04-02T07:54:20.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.513+0000 7fc8bd800700 1 Processor -- start 2024-04-02T07:54:20.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.513+0000 7fc8bd800700 1 -- start start 2024-04-02T07:54:20.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.514+0000 7fc8bd800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8b80d16a0 0x7fc8b81519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:20.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.514+0000 7fc8bd800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8b80d2540 0x7fc8b81f3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:20.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.514+0000 7fc8bd800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8b815d9d0 0x7fc8b80b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:20.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.514+0000 7fc8bd800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80b7650 con 0x7fc8b80d16a0 2024-04-02T07:54:20.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.514+0000 7fc8bd800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80b7930 con 0x7fc8b815d9d0 2024-04-02T07:54:20.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.514+0000 7fc8bd800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80b7c10 con 0x7fc8b80d2540 2024-04-02T07:54:20.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.514+0000 7fc8b6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8b80d16a0 0x7fc8b81519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:20.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.514+0000 7fc8b6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8b80d16a0 0x7fc8b81519a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43702/0 (socket says 172.21.15.89:43702) 2024-04-02T07:54:20.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.514+0000 7fc8b6a00700 1 -- 172.21.15.89:0/3035282112 learned_addr learned my addr 172.21.15.89:0/3035282112 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:20.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.514+0000 7fc8b7400700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8b815d9d0 0x7fc8b80b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:20.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.514+0000 7fc8b6000700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8b80d2540 0x7fc8b81f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:20.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.515+0000 7fc8b6a00700 1 -- 172.21.15.89:0/3035282112 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8b80d2540 msgr2=0x7fc8b81f3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:20.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.515+0000 7fc8b6a00700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8b80d2540 0x7fc8b81f3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.515+0000 7fc8b6a00700 1 -- 172.21.15.89:0/3035282112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8b815d9d0 msgr2=0x7fc8b80b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:20.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.515+0000 7fc8b6a00700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8b815d9d0 0x7fc8b80b89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.515+0000 7fc8b6a00700 1 -- 172.21.15.89:0/3035282112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b0009a70 con 0x7fc8b80d16a0 2024-04-02T07:54:20.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.515+0000 7fc8b6000700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8b80d2540 0x7fc8b81f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:20.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.515+0000 7fc8b6a00700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8b80d16a0 0x7fc8b81519a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fc8a800c480 tx=0x7fc8a800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:20.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.515+0000 7fc8aea00700 1 -- 172.21.15.89:0/3035282112 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8005d70 con 0x7fc8b80d16a0 2024-04-02T07:54:20.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.515+0000 7fc8aea00700 1 -- 172.21.15.89:0/3035282112 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a80070d0 con 0x7fc8b80d16a0 2024-04-02T07:54:20.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.515+0000 7fc8aea00700 1 -- 172.21.15.89:0/3035282112 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8007440 con 0x7fc8b80d16a0 2024-04-02T07:54:20.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.515+0000 7fc8bd800700 1 -- 172.21.15.89:0/3035282112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b80b7ef0 con 0x7fc8b80d16a0 2024-04-02T07:54:20.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.515+0000 7fc8bd800700 1 -- 172.21.15.89:0/3035282112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b80b84b0 con 0x7fc8b80d16a0 2024-04-02T07:54:20.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.517+0000 7fc8bd800700 1 -- 172.21.15.89:0/3035282112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b80d1ca0 con 0x7fc8b80d16a0 2024-04-02T07:54:20.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.518+0000 7fc8aea00700 1 -- 172.21.15.89:0/3035282112 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 64) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fc8a80075a0 con 0x7fc8b80d16a0 2024-04-02T07:54:20.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.518+0000 7fc8aea00700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc89806ad20 0x7fc89806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:20.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.518+0000 7fc8aea00700 1 -- 172.21.15.89:0/3035282112 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fc8a8088870 con 0x7fc8b80d16a0 2024-04-02T07:54:20.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.518+0000 7fc8b6000700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc89806ad20 0x7fc89806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:20.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.520+0000 7fc8b6000700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc89806ad20 0x7fc89806d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc8b80b75e0 tx=0x7fc8a4009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:20.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.523+0000 7fc8aea00700 1 -- 172.21.15.89:0/3035282112 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc8a8053a00 con 0x7fc8b80d16a0 2024-04-02T07:54:20.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.722+0000 7fc8bd800700 1 -- 172.21.15.89:0/3035282112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fc8b80b29d0 con 0x7fc8b80d16a0 2024-04-02T07:54:20.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.726+0000 7fc8aea00700 1 -- 172.21.15.89:0/3035282112 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v37) v1 ==== 115+0+0 (secure 0 0 0) 0x7fc8a8057020 con 0x7fc8b80d16a0 2024-04-02T07:54:20.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.729+0000 7fc8bd800700 1 -- 172.21.15.89:0/3035282112 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc89806ad20 msgr2=0x7fc89806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:20.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.729+0000 7fc8bd800700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc89806ad20 0x7fc89806d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc8b80b75e0 tx=0x7fc8a4009040 comp rx=0 tx=0).stop 2024-04-02T07:54:20.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.729+0000 7fc8bd800700 1 -- 172.21.15.89:0/3035282112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8b80d16a0 msgr2=0x7fc8b81519a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:20.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.729+0000 7fc8bd800700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8b80d16a0 0x7fc8b81519a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fc8a800c480 tx=0x7fc8a800c840 comp rx=0 tx=0).stop 2024-04-02T07:54:20.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.729+0000 7fc8bd800700 1 -- 172.21.15.89:0/3035282112 shutdown_connections 2024-04-02T07:54:20.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.729+0000 7fc8bd800700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc89806ad20 0x7fc89806d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.729+0000 7fc8bd800700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8b815d9d0 0x7fc8b80b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.729+0000 7fc8bd800700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8b80d16a0 0x7fc8b81519a0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.729+0000 7fc8bd800700 1 --2- 172.21.15.89:0/3035282112 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8b80d2540 0x7fc8b81f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.729+0000 7fc8bd800700 1 -- 172.21.15.89:0/3035282112 >> 172.21.15.89:0/3035282112 conn(0x7fc8b808aff0 msgr2=0x7fc8b815c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:20.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.729+0000 7fc8bd800700 1 -- 172.21.15.89:0/3035282112 shutdown_connections 2024-04-02T07:54:20.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.729+0000 7fc8bd800700 1 -- 172.21.15.89:0/3035282112 wait complete. 2024-04-02T07:54:20.748 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T07:54:20.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.940+0000 7fc6be600700 1 Processor -- start 2024-04-02T07:54:20.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.941+0000 7fc6be600700 1 -- start start 2024-04-02T07:54:20.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.942+0000 7fc6be600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc6b815d9d0 0x7fc6b80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:20.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.942+0000 7fc6be600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80d2540 0x7fc6b80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:20.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.942+0000 7fc6be600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b8161420 con 0x7fc6b80d16a0 2024-04-02T07:54:20.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.942+0000 7fc6be600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b815edc0 con 0x7fc6b815d9d0 2024-04-02T07:54:20.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.942+0000 7fc6be600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b8161700 con 0x7fc6b80d2540 2024-04-02T07:54:20.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.942+0000 7fc6b6a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc6b80d16a0 0x7fc6b80d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39166/0 (socket says 172.21.15.89:39166) 2024-04-02T07:54:20.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.942+0000 7fc6b6a00700 1 -- 172.21.15.89:0/2930944437 learned_addr learned my addr 172.21.15.89:0/2930944437 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:20.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.942+0000 7fc6b6000700 1 -- 172.21.15.89:0/2930944437 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134252057 0 0) 0x7fc6b8161420 con 0x7fc6b80d16a0 2024-04-02T07:54:20.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.942+0000 7fc6b6000700 1 -- 172.21.15.89:0/2930944437 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc69c0035b0 con 0x7fc6b80d16a0 2024-04-02T07:54:20.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.942+0000 7fc6b7400700 1 --2- 172.21.15.89:0/2930944437 >> v2:172.21.15.113:3300/0 conn(0x7fc6b815d9d0 0x7fc6b80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:20.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.942+0000 7fc6b7e00700 1 --2- 172.21.15.89:0/2930944437 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80d2540 0x7fc6b80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:20.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6000700 1 -- 172.21.15.89:0/2930944437 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4292418877 0 0) 0x7fc69c0035b0 con 0x7fc6b80d16a0 2024-04-02T07:54:20.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6000700 1 -- 172.21.15.89:0/2930944437 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80d2540 msgr2=0x7fc6b80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:20.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6000700 1 --2- 172.21.15.89:0/2930944437 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80d2540 0x7fc6b80ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6000700 1 -- 172.21.15.89:0/2930944437 >> v2:172.21.15.113:3300/0 conn(0x7fc6b815d9d0 msgr2=0x7fc6b80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:20.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6000700 1 --2- 172.21.15.89:0/2930944437 >> v2:172.21.15.113:3300/0 conn(0x7fc6b815d9d0 0x7fc6b80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6000700 1 -- 172.21.15.89:0/2930944437 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b81619e0 con 0x7fc6b80d16a0 2024-04-02T07:54:20.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6000700 1 -- 172.21.15.89:0/2930944437 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fc6a40017f0 con 0x7fc6b80d16a0 2024-04-02T07:54:20.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6000700 1 -- 172.21.15.89:0/2930944437 >> v1:172.21.15.89:6789/0 conn(0x7fc6b80d16a0 legacy=0x7fc6b80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:20.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6000700 1 --2- 172.21.15.89:0/2930944437 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80d2540 0x7fc69c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:20.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6000700 1 --2- 172.21.15.89:0/2930944437 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc69c003fd0 0x7fc69c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:20.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6000700 1 --2- 172.21.15.89:0/2930944437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c004d90 0x7fc69c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:20.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6000700 1 -- 172.21.15.89:0/2930944437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b8161700 con 0x7fc69c004d90 2024-04-02T07:54:20.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6000700 1 -- 172.21.15.89:0/2930944437 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c00ba40 con 0x7fc69c003fd0 2024-04-02T07:54:20.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6000700 1 -- 172.21.15.89:0/2930944437 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c00bc00 con 0x7fc6b80d2540 2024-04-02T07:54:20.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b7e00700 1 --2- 172.21.15.89:0/2930944437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c004d90 0x7fc69c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:20.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b7400700 1 --2- 172.21.15.89:0/2930944437 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80d2540 0x7fc69c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:20.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b6a00700 1 --2- 172.21.15.89:0/2930944437 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc69c003fd0 0x7fc69c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.943+0000 7fc6b7e00700 1 -- 172.21.15.89:0/2930944437 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80d2540 msgr2=0x7fc69c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.944+0000 7fc6b7e00700 1 --2- 172.21.15.89:0/2930944437 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80d2540 0x7fc69c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.944+0000 7fc6b7e00700 1 -- 172.21.15.89:0/2930944437 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc69c003fd0 msgr2=0x7fc69c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.944+0000 7fc6b7e00700 1 --2- 172.21.15.89:0/2930944437 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc69c003fd0 0x7fc69c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.944+0000 7fc6b7e00700 1 -- 172.21.15.89:0/2930944437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c0035b0 con 0x7fc69c004d90 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.944+0000 7fc6b6a00700 1 --2- 172.21.15.89:0/2930944437 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc69c003fd0 0x7fc69c004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.944+0000 7fc6b7e00700 1 --2- 172.21.15.89:0/2930944437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c004d90 0x7fc69c005210 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fc6b808c320 tx=0x7fc69c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ec77202b50cd8f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.944+0000 7fc6b6000700 1 -- 172.21.15.89:0/2930944437 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0007df0 con 0x7fc69c004d90 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.944+0000 7fc6b6000700 1 -- 172.21.15.89:0/2930944437 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b0003a20 con 0x7fc69c004d90 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.944+0000 7fc6b6000700 1 -- 172.21.15.89:0/2930944437 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0003dd0 con 0x7fc69c004d90 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.944+0000 7fc6be600700 1 -- 172.21.15.89:0/2930944437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c004d90 msgr2=0x7fc69c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.945+0000 7fc6be600700 1 --2- 172.21.15.89:0/2930944437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c004d90 0x7fc69c005210 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fc6b808c320 tx=0x7fc69c005880 comp rx=0 tx=0).stop 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.945+0000 7fc6b7e00700 1 -- 172.21.15.89:0/2930944437 reap_dead start 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.945+0000 7fc6be600700 1 -- 172.21.15.89:0/2930944437 shutdown_connections 2024-04-02T07:54:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.945+0000 7fc6be600700 1 -- 172.21.15.89:0/2930944437 >> 172.21.15.89:0/2930944437 conn(0x7fc6b808aff0 msgr2=0x7fc6b80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:20.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.945+0000 7fc6be600700 1 -- 172.21.15.89:0/2930944437 shutdown_connections 2024-04-02T07:54:20.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.945+0000 7fc6be600700 1 -- 172.21.15.89:0/2930944437 wait complete. 2024-04-02T07:54:20.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.946+0000 7fc6be600700 1 Processor -- start 2024-04-02T07:54:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.947+0000 7fc6be600700 1 -- start start 2024-04-02T07:54:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.947+0000 7fc6be600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c004d90 0x7fc6b8151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.947+0000 7fc6be600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80ca680 0x7fc6b80b7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.947+0000 7fc6be600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b815ff80 0x7fc6b8160400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.947+0000 7fc6be600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80b8a60 con 0x7fc69c004d90 2024-04-02T07:54:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.947+0000 7fc6be600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b8160940 con 0x7fc6b815ff80 2024-04-02T07:54:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.947+0000 7fc6be600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80b7de0 con 0x7fc6b80ca680 2024-04-02T07:54:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.948+0000 7fc6b7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c004d90 0x7fc6b8151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.948+0000 7fc6b7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c004d90 0x7fc6b8151a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43714/0 (socket says 172.21.15.89:43714) 2024-04-02T07:54:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.948+0000 7fc6b7400700 1 -- 172.21.15.89:0/2933222173 learned_addr learned my addr 172.21.15.89:0/2933222173 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.948+0000 7fc6b6a00700 1 --2- 172.21.15.89:0/2933222173 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80ca680 0x7fc6b80b7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.948+0000 7fc6b7e00700 1 --2- 172.21.15.89:0/2933222173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b815ff80 0x7fc6b8160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.948+0000 7fc6b7400700 1 -- 172.21.15.89:0/2933222173 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80ca680 msgr2=0x7fc6b80b7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.948+0000 7fc6b7400700 1 --2- 172.21.15.89:0/2933222173 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80ca680 0x7fc6b80b7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.948+0000 7fc6b7400700 1 -- 172.21.15.89:0/2933222173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b815ff80 msgr2=0x7fc6b8160400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.948+0000 7fc6b7400700 1 --2- 172.21.15.89:0/2933222173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b815ff80 0x7fc6b8160400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.948+0000 7fc6b7400700 1 -- 172.21.15.89:0/2933222173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b80b80c0 con 0x7fc69c004d90 2024-04-02T07:54:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.948+0000 7fc6b7400700 1 --2- 172.21.15.89:0/2933222173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c004d90 0x7fc6b8151a10 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8011aa0 tx=0x7fc6a8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.948+0000 7fc6af400700 1 -- 172.21.15.89:0/2933222173 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a800f9e0 con 0x7fc69c004d90 2024-04-02T07:54:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.949+0000 7fc6af400700 1 -- 172.21.15.89:0/2933222173 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a800fb40 con 0x7fc69c004d90 2024-04-02T07:54:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.949+0000 7fc6be600700 1 -- 172.21.15.89:0/2933222173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b80b83a0 con 0x7fc69c004d90 2024-04-02T07:54:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.949+0000 7fc6af400700 1 -- 172.21.15.89:0/2933222173 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8014440 con 0x7fc69c004d90 2024-04-02T07:54:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.949+0000 7fc6be600700 1 -- 172.21.15.89:0/2933222173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b80b7a40 con 0x7fc69c004d90 2024-04-02T07:54:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.950+0000 7fc6be600700 1 -- 172.21.15.89:0/2933222173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b80b8cd0 con 0x7fc69c004d90 2024-04-02T07:54:20.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.950+0000 7fc6af400700 1 -- 172.21.15.89:0/2933222173 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 64) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fc6a80145a0 con 0x7fc69c004d90 2024-04-02T07:54:20.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.951+0000 7fc6af400700 1 --2- 172.21.15.89:0/2933222173 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc69806ad20 0x7fc69806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:20.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.951+0000 7fc6af400700 1 -- 172.21.15.89:0/2933222173 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fc6a8089cd0 con 0x7fc69c004d90 2024-04-02T07:54:20.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.952+0000 7fc6b6a00700 1 --2- 172.21.15.89:0/2933222173 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc69806ad20 0x7fc69806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:20.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.952+0000 7fc6b6a00700 1 --2- 172.21.15.89:0/2933222173 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc69806ad20 0x7fc69806d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4001170 tx=0x7fc6a4009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:20.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:20.956+0000 7fc6af400700 1 -- 172.21.15.89:0/2933222173 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc6a8054d40 con 0x7fc69c004d90 2024-04-02T07:54:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.156+0000 7fc6be600700 1 -- 172.21.15.89:0/2933222173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fc6b80d1ca0 con 0x7fc69c004d90 2024-04-02T07:54:21.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.161+0000 7fc6af400700 1 -- 172.21.15.89:0/2933222173 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v38) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc6a8058360 con 0x7fc69c004d90 2024-04-02T07:54:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.164+0000 7fc6be600700 1 -- 172.21.15.89:0/2933222173 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc69806ad20 msgr2=0x7fc69806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.164+0000 7fc6be600700 1 --2- 172.21.15.89:0/2933222173 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc69806ad20 0x7fc69806d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4001170 tx=0x7fc6a4009040 comp rx=0 tx=0).stop 2024-04-02T07:54:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.164+0000 7fc6be600700 1 -- 172.21.15.89:0/2933222173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c004d90 msgr2=0x7fc6b8151a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.164+0000 7fc6be600700 1 --2- 172.21.15.89:0/2933222173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c004d90 0x7fc6b8151a10 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8011aa0 tx=0x7fc6a8011e60 comp rx=0 tx=0).stop 2024-04-02T07:54:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.164+0000 7fc6be600700 1 -- 172.21.15.89:0/2933222173 shutdown_connections 2024-04-02T07:54:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.164+0000 7fc6be600700 1 --2- 172.21.15.89:0/2933222173 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc69806ad20 0x7fc69806d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.164+0000 7fc6be600700 1 --2- 172.21.15.89:0/2933222173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b815ff80 0x7fc6b8160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.164+0000 7fc6be600700 1 --2- 172.21.15.89:0/2933222173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c004d90 0x7fc6b8151a10 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.164+0000 7fc6be600700 1 --2- 172.21.15.89:0/2933222173 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80ca680 0x7fc6b80b7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.164+0000 7fc6be600700 1 -- 172.21.15.89:0/2933222173 >> 172.21.15.89:0/2933222173 conn(0x7fc6b808aff0 msgr2=0x7fc6b80b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.164+0000 7fc6be600700 1 -- 172.21.15.89:0/2933222173 shutdown_connections 2024-04-02T07:54:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.164+0000 7fc6be600700 1 -- 172.21.15.89:0/2933222173 wait complete. 2024-04-02T07:54:21.187 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T07:54:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.384+0000 7fdf1e200700 1 Processor -- start 2024-04-02T07:54:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.384+0000 7fdf1e200700 1 -- start start 2024-04-02T07:54:21.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.385+0000 7fdf1e200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdf180b3f80 0x7fdf180ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:21.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.385+0000 7fdf1e200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf18161230 con 0x7fdf180d3f20 2024-04-02T07:54:21.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.385+0000 7fdf1e200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf180d2a80 con 0x7fdf18160e50 2024-04-02T07:54:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.385+0000 7fdf1e200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf180d43b0 con 0x7fdf180b3f80 2024-04-02T07:54:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.386+0000 7fdf16a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdf180d3f20 0x7fdf180cdef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39178/0 (socket says 172.21.15.89:39178) 2024-04-02T07:54:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.386+0000 7fdf16a00700 1 -- 172.21.15.89:0/3289522121 learned_addr learned my addr 172.21.15.89:0/3289522121 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.386+0000 7fdf17e00700 1 --2- 172.21.15.89:0/3289522121 >> v2:172.21.15.89:3300/0 conn(0x7fdf180b3f80 0x7fdf180ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.386+0000 7fdf16000700 1 -- 172.21.15.89:0/3289522121 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313595113 0 0) 0x7fdf18161230 con 0x7fdf180d3f20 2024-04-02T07:54:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.386+0000 7fdf16000700 1 -- 172.21.15.89:0/3289522121 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdefc0035b0 con 0x7fdf180d3f20 2024-04-02T07:54:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.386+0000 7fdf17e00700 1 -- 172.21.15.89:0/3289522121 >> v1:172.21.15.113:6789/0 conn(0x7fdf18160e50 legacy=0x7fdf180b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.386+0000 7fdf17e00700 1 -- 172.21.15.89:0/3289522121 >> v1:172.21.15.89:6789/0 conn(0x7fdf180d3f20 legacy=0x7fdf180cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.386+0000 7fdf17e00700 1 -- 172.21.15.89:0/3289522121 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf180d21e0 con 0x7fdf180b3f80 2024-04-02T07:54:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.386+0000 7fdf17e00700 1 --2- 172.21.15.89:0/3289522121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf180b3f80 0x7fdf180ce600 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fdf10009d90 tx=0x7fdf1000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=257ae6c9b9a07c7d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.387+0000 7fdf16000700 1 -- 172.21.15.89:0/3289522121 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1000e070 con 0x7fdf180b3f80 2024-04-02T07:54:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.387+0000 7fdf16000700 1 -- 172.21.15.89:0/3289522121 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1000bb20 con 0x7fdf180b3f80 2024-04-02T07:54:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.387+0000 7fdf16000700 1 -- 172.21.15.89:0/3289522121 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf10012430 con 0x7fdf180b3f80 2024-04-02T07:54:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.387+0000 7fdf1e200700 1 -- 172.21.15.89:0/3289522121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf180b3f80 msgr2=0x7fdf180ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.387+0000 7fdf1e200700 1 --2- 172.21.15.89:0/3289522121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf180b3f80 0x7fdf180ce600 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fdf10009d90 tx=0x7fdf1000b330 comp rx=0 tx=0).stop 2024-04-02T07:54:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.387+0000 7fdf1e200700 1 -- 172.21.15.89:0/3289522121 shutdown_connections 2024-04-02T07:54:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.387+0000 7fdf1e200700 1 --2- 172.21.15.89:0/3289522121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf180b3f80 0x7fdf180ce600 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.387+0000 7fdf1e200700 1 -- 172.21.15.89:0/3289522121 >> 172.21.15.89:0/3289522121 conn(0x7fdf1808aff0 msgr2=0x7fdf180d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.388+0000 7fdf1e200700 1 -- 172.21.15.89:0/3289522121 shutdown_connections 2024-04-02T07:54:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.388+0000 7fdf1e200700 1 -- 172.21.15.89:0/3289522121 wait complete. 2024-04-02T07:54:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.389+0000 7fdf1e200700 1 Processor -- start 2024-04-02T07:54:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.390+0000 7fdf1e200700 1 -- start start 2024-04-02T07:54:21.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.390+0000 7fdf1e200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf180b3f80 0x7fdf180b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:21.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.390+0000 7fdf1e200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf18160e50 0x7fdf181509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:21.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.390+0000 7fdf1e200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf1815d050 0x7fdf180c6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:21.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.390+0000 7fdf1e200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf180cafd0 con 0x7fdf1815d050 2024-04-02T07:54:21.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.391+0000 7fdf1e200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf180cb270 con 0x7fdf18160e50 2024-04-02T07:54:21.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.391+0000 7fdf1e200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf180b7fc0 con 0x7fdf180b3f80 2024-04-02T07:54:21.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.391+0000 7fdf17e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf1815d050 0x7fdf180c6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:21.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.391+0000 7fdf17e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf1815d050 0x7fdf180c6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43724/0 (socket says 172.21.15.89:43724) 2024-04-02T07:54:21.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.391+0000 7fdf17e00700 1 -- 172.21.15.89:0/1277072952 learned_addr learned my addr 172.21.15.89:0/1277072952 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:21.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.391+0000 7fdf17400700 1 --2- 172.21.15.89:0/1277072952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf180b3f80 0x7fdf180b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.391+0000 7fdf16a00700 1 --2- 172.21.15.89:0/1277072952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf18160e50 0x7fdf181509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.391+0000 7fdf17e00700 1 -- 172.21.15.89:0/1277072952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf180b3f80 msgr2=0x7fdf180b7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.391+0000 7fdf17e00700 1 --2- 172.21.15.89:0/1277072952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf180b3f80 0x7fdf180b7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.391+0000 7fdf17e00700 1 -- 172.21.15.89:0/1277072952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf18160e50 msgr2=0x7fdf181509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.391+0000 7fdf17e00700 1 --2- 172.21.15.89:0/1277072952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf18160e50 0x7fdf181509a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.391+0000 7fdf17e00700 1 -- 172.21.15.89:0/1277072952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf10009a70 con 0x7fdf1815d050 2024-04-02T07:54:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.391+0000 7fdf17e00700 1 --2- 172.21.15.89:0/1277072952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf1815d050 0x7fdf180c6f90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fdf1808ba30 tx=0x7fdf10004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.392+0000 7fdf0f400700 1 -- 172.21.15.89:0/1277072952 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf10012430 con 0x7fdf1815d050 2024-04-02T07:54:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.392+0000 7fdf0f400700 1 -- 172.21.15.89:0/1277072952 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf10020070 con 0x7fdf1815d050 2024-04-02T07:54:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.392+0000 7fdf0f400700 1 -- 172.21.15.89:0/1277072952 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf10004280 con 0x7fdf1815d050 2024-04-02T07:54:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.392+0000 7fdf1e200700 1 -- 172.21.15.89:0/1277072952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf180b8290 con 0x7fdf1815d050 2024-04-02T07:54:21.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.392+0000 7fdf1e200700 1 -- 172.21.15.89:0/1277072952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf180b8850 con 0x7fdf1815d050 2024-04-02T07:54:21.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.393+0000 7fdf1e200700 1 -- 172.21.15.89:0/1277072952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf180cddf0 con 0x7fdf1815d050 2024-04-02T07:54:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.394+0000 7fdf0f400700 1 -- 172.21.15.89:0/1277072952 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdf100043e0 con 0x7fdf1815d050 2024-04-02T07:54:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.394+0000 7fdf0f400700 1 --2- 172.21.15.89:0/1277072952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdef806adb0 0x7fdef806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.395+0000 7fdf0f400700 1 -- 172.21.15.89:0/1277072952 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fdf10022070 con 0x7fdf1815d050 2024-04-02T07:54:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.395+0000 7fdf17400700 1 --2- 172.21.15.89:0/1277072952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdef806adb0 0x7fdef806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.395+0000 7fdf17400700 1 --2- 172.21.15.89:0/1277072952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdef806adb0 0x7fdef806d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf08000f60 tx=0x7fdf08009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:21.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.399+0000 7fdf0f400700 1 -- 172.21.15.89:0/1277072952 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdf100562c0 con 0x7fdf1815d050 2024-04-02T07:54:21.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.601+0000 7fdf1e200700 1 -- 172.21.15.89:0/1277072952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fdf18067720 con 0x7fdf1815d050 2024-04-02T07:54:21.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.606+0000 7fdf0f400700 1 -- 172.21.15.89:0/1277072952 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v39) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdf100598e0 con 0x7fdf1815d050 2024-04-02T07:54:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.608+0000 7fdf1e200700 1 -- 172.21.15.89:0/1277072952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdef806adb0 msgr2=0x7fdef806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.608+0000 7fdf1e200700 1 --2- 172.21.15.89:0/1277072952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdef806adb0 0x7fdef806d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf08000f60 tx=0x7fdf08009040 comp rx=0 tx=0).stop 2024-04-02T07:54:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.608+0000 7fdf1e200700 1 -- 172.21.15.89:0/1277072952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf1815d050 msgr2=0x7fdf180c6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.608+0000 7fdf1e200700 1 --2- 172.21.15.89:0/1277072952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf1815d050 0x7fdf180c6f90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fdf1808ba30 tx=0x7fdf10004ad0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.608+0000 7fdf1e200700 1 -- 172.21.15.89:0/1277072952 shutdown_connections 2024-04-02T07:54:21.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.608+0000 7fdf1e200700 1 --2- 172.21.15.89:0/1277072952 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdef806adb0 0x7fdef806d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.608+0000 7fdf1e200700 1 --2- 172.21.15.89:0/1277072952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf1815d050 0x7fdf180c6f90 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.608+0000 7fdf1e200700 1 --2- 172.21.15.89:0/1277072952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf180b3f80 0x7fdf180b7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.608+0000 7fdf1e200700 1 --2- 172.21.15.89:0/1277072952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf18160e50 0x7fdf181509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.608+0000 7fdf1e200700 1 -- 172.21.15.89:0/1277072952 >> 172.21.15.89:0/1277072952 conn(0x7fdf1808aff0 msgr2=0x7fdf1815f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:21.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.608+0000 7fdf1e200700 1 -- 172.21.15.89:0/1277072952 shutdown_connections 2024-04-02T07:54:21.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.608+0000 7fdf1e200700 1 -- 172.21.15.89:0/1277072952 wait complete. 2024-04-02T07:54:21.627 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-04-02T07:54:21.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.815+0000 7f66ffe00700 1 Processor -- start 2024-04-02T07:54:21.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.815+0000 7f66ffe00700 1 -- start start 2024-04-02T07:54:21.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.816+0000 7f66ffe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f670015d5d0 0x7f67000d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:21.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.817+0000 7f66ffe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f67000d0a00 0x7f67000d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.817+0000 7f66ffe00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67000d2dc0 con 0x7f67000d2870 2024-04-02T07:54:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.817+0000 7f66ffe00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f670015d9b0 con 0x7f67000d0a00 2024-04-02T07:54:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.817+0000 7f66ffe00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67000b2860 con 0x7f670015d5d0 2024-04-02T07:54:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.817+0000 7f66ff400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f67000d0a00 0x7f67000d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.817+0000 7f66fe000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f67000d2870 0x7f67000d02f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39194/0 (socket says 172.21.15.89:39194) 2024-04-02T07:54:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.817+0000 7f66fe000700 1 -- 172.21.15.89:0/2494535840 learned_addr learned my addr 172.21.15.89:0/2494535840 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.817+0000 7f66fea00700 1 --2- 172.21.15.89:0/2494535840 >> v2:172.21.15.113:3300/0 conn(0x7f670015d5d0 0x7f67000d3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.817+0000 7f66fd600700 1 -- 172.21.15.89:0/2494535840 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496828768 0 0) 0x7f67000d2dc0 con 0x7f67000d2870 2024-04-02T07:54:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.817+0000 7f66fd600700 1 -- 172.21.15.89:0/2494535840 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66e40035b0 con 0x7f67000d2870 2024-04-02T07:54:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.817+0000 7f66ff400700 1 -- 172.21.15.89:0/2494535840 >> v2:172.21.15.113:3300/0 conn(0x7f670015d5d0 msgr2=0x7f67000d3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.817+0000 7f66ff400700 1 --2- 172.21.15.89:0/2494535840 >> v2:172.21.15.113:3300/0 conn(0x7f670015d5d0 0x7f67000d3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.817+0000 7f66ff400700 1 -- 172.21.15.89:0/2494535840 >> v1:172.21.15.89:6789/0 conn(0x7f67000d2870 legacy=0x7f67000d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.817+0000 7f66ff400700 1 -- 172.21.15.89:0/2494535840 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67000d2470 con 0x7f67000d0a00 2024-04-02T07:54:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.818+0000 7f66ff400700 1 --2- 172.21.15.89:0/2494535840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67000d0a00 0x7f67000d0f20 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f670006c7b0 tx=0x7f66f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9548a02ed0f8bf42 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.818+0000 7f66fd600700 1 -- 172.21.15.89:0/2494535840 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f800e070 con 0x7f67000d0a00 2024-04-02T07:54:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.818+0000 7f66fd600700 1 -- 172.21.15.89:0/2494535840 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f800bb20 con 0x7f67000d0a00 2024-04-02T07:54:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.818+0000 7f66fd600700 1 -- 172.21.15.89:0/2494535840 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f8012430 con 0x7f67000d0a00 2024-04-02T07:54:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.818+0000 7f66ffe00700 1 -- 172.21.15.89:0/2494535840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67000d0a00 msgr2=0x7f67000d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.818+0000 7f66ffe00700 1 --2- 172.21.15.89:0/2494535840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67000d0a00 0x7f67000d0f20 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f670006c7b0 tx=0x7f66f800b330 comp rx=0 tx=0).stop 2024-04-02T07:54:21.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.818+0000 7f66ffe00700 1 -- 172.21.15.89:0/2494535840 shutdown_connections 2024-04-02T07:54:21.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.818+0000 7f66ffe00700 1 --2- 172.21.15.89:0/2494535840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67000d0a00 0x7f67000d0f20 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.818+0000 7f66ffe00700 1 --2- 172.21.15.89:0/2494535840 >> v2:172.21.15.113:3300/0 conn(0x7f670015d5d0 0x7f67000d3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.818+0000 7f66ffe00700 1 -- 172.21.15.89:0/2494535840 >> 172.21.15.89:0/2494535840 conn(0x7f67000668e0 msgr2=0x7f6700066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:21.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.818+0000 7f66ffe00700 1 -- 172.21.15.89:0/2494535840 shutdown_connections 2024-04-02T07:54:21.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.819+0000 7f66ffe00700 1 -- 172.21.15.89:0/2494535840 wait complete. 2024-04-02T07:54:21.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.820+0000 7f66ffe00700 1 Processor -- start 2024-04-02T07:54:21.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.820+0000 7f66ffe00700 1 -- start start 2024-04-02T07:54:21.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.821+0000 7f66ffe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f67000d0a00 0x7f67001f57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.821+0000 7f66ffe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67000d2870 0x7f67000cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.821+0000 7f66ffe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f670015d5d0 0x7f67000cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.821+0000 7f66ffe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670014a9f0 con 0x7f67000d2870 2024-04-02T07:54:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.821+0000 7f66ffe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6700149b20 con 0x7f670015d5d0 2024-04-02T07:54:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.821+0000 7f66ffe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6700149d70 con 0x7f67000d0a00 2024-04-02T07:54:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.821+0000 7f66fe000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67000d2870 0x7f67000cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.822+0000 7f66fe000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67000d2870 0x7f67000cb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43732/0 (socket says 172.21.15.89:43732) 2024-04-02T07:54:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.822+0000 7f66fe000700 1 -- 172.21.15.89:0/1714978976 learned_addr learned my addr 172.21.15.89:0/1714978976 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.822+0000 7f66fea00700 1 --2- 172.21.15.89:0/1714978976 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f67000d0a00 0x7f67001f57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.822+0000 7f66ff400700 1 --2- 172.21.15.89:0/1714978976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f670015d5d0 0x7f67000cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.822+0000 7f66fe000700 1 -- 172.21.15.89:0/1714978976 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f67000d0a00 msgr2=0x7f67001f57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.822+0000 7f66fe000700 1 --2- 172.21.15.89:0/1714978976 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f67000d0a00 0x7f67001f57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.822+0000 7f66fe000700 1 -- 172.21.15.89:0/1714978976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f670015d5d0 msgr2=0x7f67000cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.822+0000 7f66fe000700 1 --2- 172.21.15.89:0/1714978976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f670015d5d0 0x7f67000cbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:21.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.822+0000 7f66fe000700 1 -- 172.21.15.89:0/1714978976 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f8009a70 con 0x7f67000d2870 2024-04-02T07:54:21.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.822+0000 7f66fe000700 1 --2- 172.21.15.89:0/1714978976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67000d2870 0x7f67000cb620 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f66ec001eb0 tx=0x7f66ec00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:21.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.822+0000 7f66f6a00700 1 -- 172.21.15.89:0/1714978976 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ec005d70 con 0x7f67000d2870 2024-04-02T07:54:21.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.822+0000 7f66f6a00700 1 -- 172.21.15.89:0/1714978976 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66ec0070d0 con 0x7f67000d2870 2024-04-02T07:54:21.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.822+0000 7f66f6a00700 1 -- 172.21.15.89:0/1714978976 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ec007460 con 0x7f67000d2870 2024-04-02T07:54:21.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.822+0000 7f66ffe00700 1 -- 172.21.15.89:0/1714978976 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f670014a050 con 0x7f67000d2870 2024-04-02T07:54:21.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.823+0000 7f66ffe00700 1 -- 172.21.15.89:0/1714978976 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f670014a610 con 0x7f67000d2870 2024-04-02T07:54:21.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.823+0000 7f66ffe00700 1 -- 172.21.15.89:0/1714978976 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f670015f240 con 0x7f67000d2870 2024-04-02T07:54:21.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.824+0000 7f66f6a00700 1 -- 172.21.15.89:0/1714978976 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f66ec0075c0 con 0x7f67000d2870 2024-04-02T07:54:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.825+0000 7f66f6a00700 1 --2- 172.21.15.89:0/1714978976 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f66e006adb0 0x7f66e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.825+0000 7f66f6a00700 1 -- 172.21.15.89:0/1714978976 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f66ec088980 con 0x7f67000d2870 2024-04-02T07:54:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.825+0000 7f66fea00700 1 --2- 172.21.15.89:0/1714978976 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f66e006adb0 0x7f66e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:21.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.826+0000 7f66fea00700 1 --2- 172.21.15.89:0/1714978976 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f66e006adb0 0x7f66e006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f66f0006b20 tx=0x7f66f0008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:21.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:21.829+0000 7f66f6a00700 1 -- 172.21.15.89:0/1714978976 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f66ec053930 con 0x7f67000d2870 2024-04-02T07:54:22.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.117+0000 7f66ffe00700 1 -- 172.21.15.89:0/1714978976 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f670004ea80 con 0x7f67000d2870 2024-04-02T07:54:22.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.295+0000 7f66f6a00700 1 -- 172.21.15.89:0/1714978976 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v690) v1 ==== 183+0+0 (secure 0 0 0) 0x7f66ec0170f0 con 0x7f67000d2870 2024-04-02T07:54:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.297+0000 7f66ffe00700 1 -- 172.21.15.89:0/1714978976 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f66e006adb0 msgr2=0x7f66e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.297+0000 7f66ffe00700 1 --2- 172.21.15.89:0/1714978976 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f66e006adb0 0x7f66e006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f66f0006b20 tx=0x7f66f0008040 comp rx=0 tx=0).stop 2024-04-02T07:54:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.297+0000 7f66ffe00700 1 -- 172.21.15.89:0/1714978976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67000d2870 msgr2=0x7f67000cb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.297+0000 7f66ffe00700 1 --2- 172.21.15.89:0/1714978976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67000d2870 0x7f67000cb620 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f66ec001eb0 tx=0x7f66ec00c7b0 comp rx=0 tx=0).stop 2024-04-02T07:54:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.297+0000 7f66ffe00700 1 -- 172.21.15.89:0/1714978976 shutdown_connections 2024-04-02T07:54:22.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.297+0000 7f66ffe00700 1 --2- 172.21.15.89:0/1714978976 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f66e006adb0 0x7f66e006d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:22.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.297+0000 7f66ffe00700 1 --2- 172.21.15.89:0/1714978976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f670015d5d0 0x7f67000cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:22.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.297+0000 7f66ffe00700 1 --2- 172.21.15.89:0/1714978976 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f67000d0a00 0x7f67001f57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:22.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.297+0000 7f66ffe00700 1 --2- 172.21.15.89:0/1714978976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67000d2870 0x7f67000cb620 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:22.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.297+0000 7f66ffe00700 1 -- 172.21.15.89:0/1714978976 >> 172.21.15.89:0/1714978976 conn(0x7f67000668e0 msgr2=0x7f6700066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:22.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.297+0000 7f66ffe00700 1 -- 172.21.15.89:0/1714978976 shutdown_connections 2024-04-02T07:54:22.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.297+0000 7f66ffe00700 1 -- 172.21.15.89:0/1714978976 wait complete. 2024-04-02T07:54:22.316 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2024-04-02T07:54:22.316 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2024-04-02T07:54:22.316 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-02T07:54:22.316 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-04-02T07:54:22.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.507+0000 7f7d51600700 1 Processor -- start 2024-04-02T07:54:22.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.507+0000 7f7d51600700 1 -- start start 2024-04-02T07:54:22.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.508+0000 7f7d51600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7d4c15b5d0 0x7f7d4c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:22.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.508+0000 7f7d51600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7d4c0ac7d0 0x7f7d4c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:22.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.508+0000 7f7d51600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c15fd50 con 0x7f7d4c0cb3f0 2024-04-02T07:54:22.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.508+0000 7f7d51600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c15b9b0 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.508+0000 7f7d51600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c15f140 con 0x7f7d4c15b5d0 2024-04-02T07:54:22.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.508+0000 7f7d4b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7d4c0ac7d0 0x7f7d4c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:22.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.508+0000 7f7d4b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7d4c0ac7d0 0x7f7d4c06c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43740/0 (socket says 172.21.15.89:43740) 2024-04-02T07:54:22.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.508+0000 7f7d4b400700 1 -- 172.21.15.89:0/3492376534 learned_addr learned my addr 172.21.15.89:0/3492376534 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:22.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.508+0000 7f7d4aa00700 1 --2- 172.21.15.89:0/3492376534 >> v2:172.21.15.113:3300/0 conn(0x7f7d4c15b5d0 0x7f7d4c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:22.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.509+0000 7f7d4b400700 1 -- 172.21.15.89:0/3492376534 >> v2:172.21.15.113:3300/0 conn(0x7f7d4c15b5d0 msgr2=0x7f7d4c0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:22.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.509+0000 7f7d4b400700 1 --2- 172.21.15.89:0/3492376534 >> v2:172.21.15.113:3300/0 conn(0x7f7d4c15b5d0 0x7f7d4c0ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:22.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.509+0000 7f7d4b400700 1 -- 172.21.15.89:0/3492376534 >> v1:172.21.15.113:6789/0 conn(0x7f7d4c0cb3f0 legacy=0x7f7d4c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:22.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.509+0000 7f7d4b400700 1 -- 172.21.15.89:0/3492376534 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c15f420 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.509+0000 7f7d4b400700 1 --2- 172.21.15.89:0/3492376534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d4c0ac7d0 0x7f7d4c06c9d0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c06b8b0 tx=0x7f7d4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161fd10c2d2c2b76 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:22.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.509+0000 7f7d4a000700 1 -- 172.21.15.89:0/3492376534 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4400e070 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.509+0000 7f7d4a000700 1 -- 172.21.15.89:0/3492376534 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4400b910 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.509+0000 7f7d4a000700 1 -- 172.21.15.89:0/3492376534 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4400bcc0 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.510+0000 7f7d51600700 1 -- 172.21.15.89:0/3492376534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d4c0ac7d0 msgr2=0x7f7d4c06c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:22.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.510+0000 7f7d51600700 1 --2- 172.21.15.89:0/3492376534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d4c0ac7d0 0x7f7d4c06c9d0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c06b8b0 tx=0x7f7d4400b330 comp rx=0 tx=0).stop 2024-04-02T07:54:22.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.510+0000 7f7d51600700 1 -- 172.21.15.89:0/3492376534 shutdown_connections 2024-04-02T07:54:22.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.510+0000 7f7d51600700 1 --2- 172.21.15.89:0/3492376534 >> v2:172.21.15.113:3300/0 conn(0x7f7d4c15b5d0 0x7f7d4c0ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:22.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.510+0000 7f7d51600700 1 --2- 172.21.15.89:0/3492376534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d4c0ac7d0 0x7f7d4c06c9d0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:22.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.510+0000 7f7d51600700 1 -- 172.21.15.89:0/3492376534 >> 172.21.15.89:0/3492376534 conn(0x7f7d4c0cf190 msgr2=0x7f7d4c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:22.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.510+0000 7f7d51600700 1 -- 172.21.15.89:0/3492376534 shutdown_connections 2024-04-02T07:54:22.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.510+0000 7f7d51600700 1 -- 172.21.15.89:0/3492376534 wait complete. 2024-04-02T07:54:22.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.512+0000 7f7d51600700 1 Processor -- start 2024-04-02T07:54:22.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.512+0000 7f7d51600700 1 -- start start 2024-04-02T07:54:22.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.513+0000 7f7d51600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d4c0ac7d0 0x7f7d4c0718e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:22.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.513+0000 7f7d51600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d4c15b5d0 0x7f7d4c0c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:22.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.513+0000 7f7d51600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d4c0709c0 0x7f7d4c0ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:22.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.513+0000 7f7d51600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c071e20 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.513+0000 7f7d51600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c0720c0 con 0x7f7d4c15b5d0 2024-04-02T07:54:22.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.513+0000 7f7d51600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c0723a0 con 0x7f7d4c0709c0 2024-04-02T07:54:22.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.513+0000 7f7d4aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d4c0ac7d0 0x7f7d4c0718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.513+0000 7f7d4aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d4c0ac7d0 0x7f7d4c0718e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43748/0 (socket says 172.21.15.89:43748) 2024-04-02T07:54:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.513+0000 7f7d4aa00700 1 -- 172.21.15.89:0/2276080623 learned_addr learned my addr 172.21.15.89:0/2276080623 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.513+0000 7f7d43e00700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d4c15b5d0 0x7f7d4c0c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.513+0000 7f7d4b400700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d4c0709c0 0x7f7d4c0ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.514+0000 7f7d4aa00700 1 -- 172.21.15.89:0/2276080623 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d4c0709c0 msgr2=0x7f7d4c0ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.514+0000 7f7d4aa00700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d4c0709c0 0x7f7d4c0ca080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.514+0000 7f7d4aa00700 1 -- 172.21.15.89:0/2276080623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d4c15b5d0 msgr2=0x7f7d4c0c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.514+0000 7f7d4aa00700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d4c15b5d0 0x7f7d4c0c9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.514+0000 7f7d4aa00700 1 -- 172.21.15.89:0/2276080623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d44009b30 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.514+0000 7f7d4b400700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d4c0709c0 0x7f7d4c0ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.514+0000 7f7d4aa00700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d4c0ac7d0 0x7f7d4c0718e0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00e870 tx=0x7f7d3c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.514+0000 7f7d42a00700 1 -- 172.21.15.89:0/2276080623 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c009590 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.514+0000 7f7d42a00700 1 -- 172.21.15.89:0/2276080623 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c013680 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.514+0000 7f7d51600700 1 -- 172.21.15.89:0/2276080623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4c072680 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.514+0000 7f7d42a00700 1 -- 172.21.15.89:0/2276080623 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c0139f0 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.514+0000 7f7d51600700 1 -- 172.21.15.89:0/2276080623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4c1f44c0 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.515+0000 7f7d51600700 1 -- 172.21.15.89:0/2276080623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4c067720 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.516+0000 7f7d42a00700 1 -- 172.21.15.89:0/2276080623 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7d3c013b50 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.517+0000 7f7d42a00700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7d2806adb0 0x7f7d2806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:22.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.517+0000 7f7d42a00700 1 -- 172.21.15.89:0/2276080623 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f7d3c0898b0 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.517+0000 7f7d43e00700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7d2806adb0 0x7f7d2806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:22.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.518+0000 7f7d43e00700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7d2806adb0 0x7f7d2806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c1538a0 tx=0x7f7d34009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:22.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.521+0000 7f7d42a00700 1 -- 172.21.15.89:0/2276080623 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7d3c0547e0 con 0x7f7d4c0ac7d0 2024-04-02T07:54:22.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:22.801+0000 7f7d51600700 1 -- 172.21.15.89:0/2276080623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f7d4c1f4830 con 0x7f7d4c0ac7d0 2024-04-02T07:54:23.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.296+0000 7f7d42a00700 1 -- 172.21.15.89:0/2276080623 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v691) v1 ==== 151+0+0 (secure 0 0 0) 0x7f7d3c057e00 con 0x7f7d4c0ac7d0 2024-04-02T07:54:23.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.298+0000 7f7d51600700 1 -- 172.21.15.89:0/2276080623 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7d2806adb0 msgr2=0x7f7d2806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:23.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.298+0000 7f7d51600700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7d2806adb0 0x7f7d2806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c1538a0 tx=0x7f7d34009040 comp rx=0 tx=0).stop 2024-04-02T07:54:23.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.298+0000 7f7d51600700 1 -- 172.21.15.89:0/2276080623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d4c0ac7d0 msgr2=0x7f7d4c0718e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.298+0000 7f7d51600700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d4c0ac7d0 0x7f7d4c0718e0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00e870 tx=0x7f7d3c00eb80 comp rx=0 tx=0).stop 2024-04-02T07:54:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.298+0000 7f7d51600700 1 -- 172.21.15.89:0/2276080623 shutdown_connections 2024-04-02T07:54:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.298+0000 7f7d51600700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7d2806adb0 0x7f7d2806d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.298+0000 7f7d51600700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d4c0709c0 0x7f7d4c0ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.298+0000 7f7d51600700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d4c0ac7d0 0x7f7d4c0718e0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.298+0000 7f7d51600700 1 --2- 172.21.15.89:0/2276080623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d4c15b5d0 0x7f7d4c0c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.298+0000 7f7d51600700 1 -- 172.21.15.89:0/2276080623 >> 172.21.15.89:0/2276080623 conn(0x7f7d4c0cf190 msgr2=0x7f7d4c0af090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.298+0000 7f7d51600700 1 -- 172.21.15.89:0/2276080623 shutdown_connections 2024-04-02T07:54:23.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.299+0000 7f7d51600700 1 -- 172.21.15.89:0/2276080623 wait complete. 2024-04-02T07:54:23.317 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T07:54:23.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.509+0000 7f9272000700 1 Processor -- start 2024-04-02T07:54:23.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.510+0000 7f9272000700 1 -- start start 2024-04-02T07:54:23.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.511+0000 7f9272000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f926c0ced80 0x7f926c0d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.511+0000 7f9272000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f926c15fcb0 0x7f926c160120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.511+0000 7f9272000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c15f820 con 0x7f926c15dad0 2024-04-02T07:54:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.511+0000 7f9272000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f926c0cfcc0 con 0x7f926c0ced80 2024-04-02T07:54:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.511+0000 7f9272000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0cd9f0 con 0x7f926c15fcb0 2024-04-02T07:54:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.511+0000 7f926b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f926c15dad0 0x7f926c0d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39198/0 (socket says 172.21.15.89:39198) 2024-04-02T07:54:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.511+0000 7f926b400700 1 -- 172.21.15.89:0/3889923732 learned_addr learned my addr 172.21.15.89:0/3889923732 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.511+0000 7f926aa00700 1 --2- 172.21.15.89:0/3889923732 >> v2:172.21.15.113:3300/0 conn(0x7f926c0ced80 0x7f926c0d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.511+0000 7f926be00700 1 --2- 172.21.15.89:0/3889923732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f926c15fcb0 0x7f926c160120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.511+0000 7f926a000700 1 -- 172.21.15.89:0/3889923732 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404883766 0 0) 0x7f926c15f820 con 0x7f926c15dad0 2024-04-02T07:54:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.511+0000 7f926a000700 1 -- 172.21.15.89:0/3889923732 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92500035b0 con 0x7f926c15dad0 2024-04-02T07:54:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926a000700 1 -- 172.21.15.89:0/3889923732 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2928004307 0 0) 0x7f92500035b0 con 0x7f926c15dad0 2024-04-02T07:54:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926a000700 1 -- 172.21.15.89:0/3889923732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f926c15fcb0 msgr2=0x7f926c160120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926a000700 1 --2- 172.21.15.89:0/3889923732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f926c15fcb0 0x7f926c160120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926a000700 1 -- 172.21.15.89:0/3889923732 >> v2:172.21.15.113:3300/0 conn(0x7f926c0ced80 msgr2=0x7f926c0d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926a000700 1 --2- 172.21.15.89:0/3889923732 >> v2:172.21.15.113:3300/0 conn(0x7f926c0ced80 0x7f926c0d1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926a000700 1 -- 172.21.15.89:0/3889923732 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c0cf420 con 0x7f926c15dad0 2024-04-02T07:54:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926a000700 1 -- 172.21.15.89:0/3889923732 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f925c001670 con 0x7f926c15dad0 2024-04-02T07:54:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926a000700 1 -- 172.21.15.89:0/3889923732 >> v1:172.21.15.89:6789/0 conn(0x7f926c15dad0 legacy=0x7f926c0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926a000700 1 --2- 172.21.15.89:0/3889923732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9250003a00 0x7f9250003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926a000700 1 --2- 172.21.15.89:0/3889923732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9250004850 0x7f9250004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926a000700 1 --2- 172.21.15.89:0/3889923732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f926c15fcb0 0x7f9250005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926a000700 1 -- 172.21.15.89:0/3889923732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0cfcc0 con 0x7f9250004850 2024-04-02T07:54:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926a000700 1 -- 172.21.15.89:0/3889923732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0cd9f0 con 0x7f9250003a00 2024-04-02T07:54:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926a000700 1 -- 172.21.15.89:0/3889923732 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925000ba10 con 0x7f926c15fcb0 2024-04-02T07:54:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926aa00700 1 --2- 172.21.15.89:0/3889923732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9250004850 0x7f9250004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926be00700 1 --2- 172.21.15.89:0/3889923732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f926c15fcb0 0x7f9250005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.512+0000 7f926b400700 1 --2- 172.21.15.89:0/3889923732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9250003a00 0x7f9250003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.513+0000 7f926aa00700 1 -- 172.21.15.89:0/3889923732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f926c15fcb0 msgr2=0x7f9250005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.513+0000 7f926aa00700 1 --2- 172.21.15.89:0/3889923732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f926c15fcb0 0x7f9250005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.513+0000 7f926aa00700 1 -- 172.21.15.89:0/3889923732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9250003a00 msgr2=0x7f9250003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.513+0000 7f926aa00700 1 --2- 172.21.15.89:0/3889923732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9250003a00 0x7f9250003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.513+0000 7f926aa00700 1 -- 172.21.15.89:0/3889923732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92500035b0 con 0x7f9250004850 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.513+0000 7f926b400700 1 --2- 172.21.15.89:0/3889923732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9250003a00 0x7f9250003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.513+0000 7f926aa00700 1 --2- 172.21.15.89:0/3889923732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9250004850 0x7f9250004cd0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f9258009b60 tx=0x7f9250005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db4e290163971060 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.513+0000 7f926a000700 1 -- 172.21.15.89:0/3889923732 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925800e6c0 con 0x7f9250004850 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.513+0000 7f926a000700 1 -- 172.21.15.89:0/3889923732 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925800e820 con 0x7f9250004850 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.513+0000 7f926a000700 1 -- 172.21.15.89:0/3889923732 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925800eb90 con 0x7f9250004850 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.513+0000 7f9272000700 1 -- 172.21.15.89:0/3889923732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9250004850 msgr2=0x7f9250004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.513+0000 7f9272000700 1 --2- 172.21.15.89:0/3889923732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9250004850 0x7f9250004cd0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f9258009b60 tx=0x7f9250005880 comp rx=0 tx=0).stop 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.513+0000 7f926be00700 1 -- 172.21.15.89:0/3889923732 reap_dead start 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.514+0000 7f9272000700 1 -- 172.21.15.89:0/3889923732 shutdown_connections 2024-04-02T07:54:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.514+0000 7f9272000700 1 -- 172.21.15.89:0/3889923732 >> 172.21.15.89:0/3889923732 conn(0x7f926c08aff0 msgr2=0x7f926c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.514+0000 7f9272000700 1 -- 172.21.15.89:0/3889923732 shutdown_connections 2024-04-02T07:54:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.514+0000 7f9272000700 1 -- 172.21.15.89:0/3889923732 wait complete. 2024-04-02T07:54:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.515+0000 7f9272000700 1 Processor -- start 2024-04-02T07:54:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.516+0000 7f9272000700 1 -- start start 2024-04-02T07:54:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.516+0000 7f9272000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9250004850 0x7f926c0b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.516+0000 7f9272000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f926c1f2ee0 0x7f926c0c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.516+0000 7f9272000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f926c0c7050 0x7f926c159260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.516+0000 7f9272000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0b73c0 con 0x7f926c0c7050 2024-04-02T07:54:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.516+0000 7f9272000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0b9250 con 0x7f9250004850 2024-04-02T07:54:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.516+0000 7f9272000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c1f33c0 con 0x7f926c1f2ee0 2024-04-02T07:54:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.517+0000 7f926be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f926c0c7050 0x7f926c159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.517+0000 7f926be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f926c0c7050 0x7f926c159260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43758/0 (socket says 172.21.15.89:43758) 2024-04-02T07:54:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.517+0000 7f926be00700 1 -- 172.21.15.89:0/3741519816 learned_addr learned my addr 172.21.15.89:0/3741519816 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.517+0000 7f926b400700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9250004850 0x7f926c0b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.517+0000 7f926aa00700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f926c1f2ee0 0x7f926c0c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.517+0000 7f926be00700 1 -- 172.21.15.89:0/3741519816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f926c1f2ee0 msgr2=0x7f926c0c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.517+0000 7f926be00700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f926c1f2ee0 0x7f926c0c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.517+0000 7f926be00700 1 -- 172.21.15.89:0/3741519816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9250004850 msgr2=0x7f926c0b8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.517+0000 7f926be00700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9250004850 0x7f926c0b8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.517+0000 7f926be00700 1 -- 172.21.15.89:0/3741519816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c1f36a0 con 0x7f926c0c7050 2024-04-02T07:54:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.517+0000 7f926aa00700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f926c1f2ee0 0x7f926c0c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.517+0000 7f926be00700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f926c0c7050 0x7f926c159260 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f926c08d4f0 tx=0x7f92640114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.517+0000 7f9263400700 1 -- 172.21.15.89:0/3741519816 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926400e070 con 0x7f926c0c7050 2024-04-02T07:54:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.517+0000 7f9263400700 1 -- 172.21.15.89:0/3741519816 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9264003cf0 con 0x7f926c0c7050 2024-04-02T07:54:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.518+0000 7f9272000700 1 -- 172.21.15.89:0/3741519816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926c1f3980 con 0x7f926c0c7050 2024-04-02T07:54:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.518+0000 7f9263400700 1 -- 172.21.15.89:0/3741519816 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264003ea0 con 0x7f926c0c7050 2024-04-02T07:54:23.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.518+0000 7f9272000700 1 -- 172.21.15.89:0/3741519816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926c0b8980 con 0x7f926c0c7050 2024-04-02T07:54:23.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.519+0000 7f9272000700 1 -- 172.21.15.89:0/3741519816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926c067720 con 0x7f926c0c7050 2024-04-02T07:54:23.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.520+0000 7f9263400700 1 -- 172.21.15.89:0/3741519816 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9264011a10 con 0x7f926c0c7050 2024-04-02T07:54:23.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.520+0000 7f9263400700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f924c06adb0 0x7f924c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:23.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.520+0000 7f9263400700 1 -- 172.21.15.89:0/3741519816 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f9264089b60 con 0x7f926c0c7050 2024-04-02T07:54:23.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.521+0000 7f926b400700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f924c06adb0 0x7f924c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:23.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.521+0000 7f926b400700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f924c06adb0 0x7f924c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f925c000ff0 tx=0x7f925c009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:23.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.525+0000 7f9263400700 1 -- 172.21.15.89:0/3741519816 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9264054970 con 0x7f926c0c7050 2024-04-02T07:54:23.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.727+0000 7f9272000700 1 -- 172.21.15.89:0/3741519816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f926c0cd400 con 0x7f926c0c7050 2024-04-02T07:54:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.731+0000 7f9263400700 1 -- 172.21.15.89:0/3741519816 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9264057f90 con 0x7f926c0c7050 2024-04-02T07:54:23.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.734+0000 7f9272000700 1 -- 172.21.15.89:0/3741519816 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f924c06adb0 msgr2=0x7f924c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:23.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.734+0000 7f9272000700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f924c06adb0 0x7f924c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f925c000ff0 tx=0x7f925c009040 comp rx=0 tx=0).stop 2024-04-02T07:54:23.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.734+0000 7f9272000700 1 -- 172.21.15.89:0/3741519816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f926c0c7050 msgr2=0x7f926c159260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:23.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.734+0000 7f9272000700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f926c0c7050 0x7f926c159260 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f926c08d4f0 tx=0x7f92640114b0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.734+0000 7f9272000700 1 -- 172.21.15.89:0/3741519816 shutdown_connections 2024-04-02T07:54:23.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.734+0000 7f9272000700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f924c06adb0 0x7f924c06d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.734+0000 7f9272000700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9250004850 0x7f926c0b8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.734+0000 7f9272000700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f926c0c7050 0x7f926c159260 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.734+0000 7f9272000700 1 --2- 172.21.15.89:0/3741519816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f926c1f2ee0 0x7f926c0c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:23.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.734+0000 7f9272000700 1 -- 172.21.15.89:0/3741519816 >> 172.21.15.89:0/3741519816 conn(0x7f926c08aff0 msgr2=0x7f926c0cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:23.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.734+0000 7f9272000700 1 -- 172.21.15.89:0/3741519816 shutdown_connections 2024-04-02T07:54:23.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:23.734+0000 7f9272000700 1 -- 172.21.15.89:0/3741519816 wait complete. 2024-04-02T07:54:23.752 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:54:23.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:54:23.839 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:54:23.933 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:54:23.933 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:54:23.933 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:54:23.978 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:54:23.978 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:54:23.978 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:54:24.025 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:54:24.025 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:54:24.025 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:54:24.071 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:54:24.071 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:54:24.071 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:54:24.117 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:54:24.117 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T07:54:24.117 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:54:24.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.309+0000 7f1c14e00700 1 Processor -- start 2024-04-02T07:54:24.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.309+0000 7f1c14e00700 1 -- start start 2024-04-02T07:54:24.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.310+0000 7f1c14e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1c1015b5d0 0x7f1c100ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:24.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.310+0000 7f1c14e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c100cb3f0 0x7f1c100ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:24.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.310+0000 7f1c14e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1c100ae930 0x7f1c100ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:24.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.310+0000 7f1c14e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c1015b9b0 con 0x7f1c100ae930 2024-04-02T07:54:24.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.310+0000 7f1c14e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c1015fd50 con 0x7f1c1015b5d0 2024-04-02T07:54:24.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.310+0000 7f1c14e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1015f140 con 0x7f1c100cb3f0 2024-04-02T07:54:24.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.311+0000 7f1c0e800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1c100ae930 0x7f1c100ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:24.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.311+0000 7f1c0e800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1c100ae930 0x7f1c100ac600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43772/0 (socket says 172.21.15.89:43772) 2024-04-02T07:54:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.311+0000 7f1c0e800700 1 -- 172.21.15.89:0/849285624 learned_addr learned my addr 172.21.15.89:0/849285624 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.311+0000 7f1c0de00700 1 --2- 172.21.15.89:0/849285624 >> v2:172.21.15.113:3300/0 conn(0x7f1c1015b5d0 0x7f1c100ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.311+0000 7f1c0d400700 1 --2- 172.21.15.89:0/849285624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c100cb3f0 0x7f1c100ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.311+0000 7f1c0e800700 1 -- 172.21.15.89:0/849285624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c100cb3f0 msgr2=0x7f1c100ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.311+0000 7f1c0e800700 1 --2- 172.21.15.89:0/849285624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c100cb3f0 0x7f1c100ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.311+0000 7f1c0e800700 1 -- 172.21.15.89:0/849285624 >> v2:172.21.15.113:3300/0 conn(0x7f1c1015b5d0 msgr2=0x7f1c100ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.311+0000 7f1c0e800700 1 --2- 172.21.15.89:0/849285624 >> v2:172.21.15.113:3300/0 conn(0x7f1c1015b5d0 0x7f1c100ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.311+0000 7f1c0e800700 1 -- 172.21.15.89:0/849285624 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1015f420 con 0x7f1c100ae930 2024-04-02T07:54:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.311+0000 7f1c0e800700 1 --2- 172.21.15.89:0/849285624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c100ae930 0x7f1c100ac600 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f1c1006b8b0 tx=0x7f1c0800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820026a3a7d3aa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.312+0000 7f1c0ca00700 1 -- 172.21.15.89:0/849285624 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0800e070 con 0x7f1c100ae930 2024-04-02T07:54:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.312+0000 7f1c0ca00700 1 -- 172.21.15.89:0/849285624 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0800b910 con 0x7f1c100ae930 2024-04-02T07:54:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.312+0000 7f1c0ca00700 1 -- 172.21.15.89:0/849285624 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0800bcc0 con 0x7f1c100ae930 2024-04-02T07:54:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.312+0000 7f1c14e00700 1 -- 172.21.15.89:0/849285624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c100ae930 msgr2=0x7f1c100ac600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.312+0000 7f1c14e00700 1 --2- 172.21.15.89:0/849285624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c100ae930 0x7f1c100ac600 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f1c1006b8b0 tx=0x7f1c0800b330 comp rx=0 tx=0).stop 2024-04-02T07:54:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.312+0000 7f1c14e00700 1 -- 172.21.15.89:0/849285624 shutdown_connections 2024-04-02T07:54:24.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.312+0000 7f1c14e00700 1 --2- 172.21.15.89:0/849285624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c100ae930 0x7f1c100ac600 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:24.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.312+0000 7f1c14e00700 1 --2- 172.21.15.89:0/849285624 >> v2:172.21.15.113:3300/0 conn(0x7f1c1015b5d0 0x7f1c100ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:24.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.312+0000 7f1c14e00700 1 --2- 172.21.15.89:0/849285624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c100cb3f0 0x7f1c100ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:24.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.312+0000 7f1c14e00700 1 -- 172.21.15.89:0/849285624 >> 172.21.15.89:0/849285624 conn(0x7f1c100cf190 msgr2=0x7f1c100cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:24.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.312+0000 7f1c14e00700 1 -- 172.21.15.89:0/849285624 shutdown_connections 2024-04-02T07:54:24.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.312+0000 7f1c14e00700 1 -- 172.21.15.89:0/849285624 wait complete. 2024-04-02T07:54:24.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.314+0000 7f1c14e00700 1 Processor -- start 2024-04-02T07:54:24.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.314+0000 7f1c14e00700 1 -- start start 2024-04-02T07:54:24.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.315+0000 7f1c14e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c100ae930 0x7f1c100721a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:24.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.315+0000 7f1c14e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c1015b5d0 0x7f1c100c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:24.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.315+0000 7f1c14e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c1015d810 0x7f1c100ca210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:24.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.315+0000 7f1c14e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c100726e0 con 0x7f1c100ae930 2024-04-02T07:54:24.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.315+0000 7f1c14e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c100ae540 con 0x7f1c1015d810 2024-04-02T07:54:24.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.315+0000 7f1c14e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c101f3cd0 con 0x7f1c1015b5d0 2024-04-02T07:54:24.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.315+0000 7f1c0de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c100ae930 0x7f1c100721a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:24.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.315+0000 7f1c0de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c100ae930 0x7f1c100721a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43776/0 (socket says 172.21.15.89:43776) 2024-04-02T07:54:24.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.315+0000 7f1c0de00700 1 -- 172.21.15.89:0/2200092516 learned_addr learned my addr 172.21.15.89:0/2200092516 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:24.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.315+0000 7f1c0d400700 1 --2- 172.21.15.89:0/2200092516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c1015b5d0 0x7f1c100c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:24.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.315+0000 7f1c0e800700 1 --2- 172.21.15.89:0/2200092516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c1015d810 0x7f1c100ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:24.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.316+0000 7f1c0de00700 1 -- 172.21.15.89:0/2200092516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c1015b5d0 msgr2=0x7f1c100c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:24.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.316+0000 7f1c0de00700 1 --2- 172.21.15.89:0/2200092516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c1015b5d0 0x7f1c100c9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:24.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.316+0000 7f1c0de00700 1 -- 172.21.15.89:0/2200092516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c1015d810 msgr2=0x7f1c100ca210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:24.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.316+0000 7f1c0de00700 1 --2- 172.21.15.89:0/2200092516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c1015d810 0x7f1c100ca210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:24.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.316+0000 7f1c0de00700 1 -- 172.21.15.89:0/2200092516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c08009b30 con 0x7f1c100ae930 2024-04-02T07:54:24.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.316+0000 7f1c0de00700 1 --2- 172.21.15.89:0/2200092516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c100ae930 0x7f1c100721a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000e870 tx=0x7f1c0000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:24.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.316+0000 7f1c06000700 1 -- 172.21.15.89:0/2200092516 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00009590 con 0x7f1c100ae930 2024-04-02T07:54:24.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.316+0000 7f1c06000700 1 -- 172.21.15.89:0/2200092516 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c00013680 con 0x7f1c100ae930 2024-04-02T07:54:24.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.316+0000 7f1c14e00700 1 -- 172.21.15.89:0/2200092516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c101f39f0 con 0x7f1c100ae930 2024-04-02T07:54:24.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.316+0000 7f1c06000700 1 -- 172.21.15.89:0/2200092516 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c000139f0 con 0x7f1c100ae930 2024-04-02T07:54:24.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.316+0000 7f1c14e00700 1 -- 172.21.15.89:0/2200092516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c101f67d0 con 0x7f1c100ae930 2024-04-02T07:54:24.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.317+0000 7f1c14e00700 1 -- 172.21.15.89:0/2200092516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c10067720 con 0x7f1c100ae930 2024-04-02T07:54:24.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.318+0000 7f1c06000700 1 -- 172.21.15.89:0/2200092516 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1c00013b50 con 0x7f1c100ae930 2024-04-02T07:54:24.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.319+0000 7f1c06000700 1 --2- 172.21.15.89:0/2200092516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1bf006adb0 0x7f1bf006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:24.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.319+0000 7f1c06000700 1 -- 172.21.15.89:0/2200092516 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f1c000897d0 con 0x7f1c100ae930 2024-04-02T07:54:24.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.320+0000 7f1c0d400700 1 --2- 172.21.15.89:0/2200092516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1bf006adb0 0x7f1bf006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:24.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.320+0000 7f1c0d400700 1 --2- 172.21.15.89:0/2200092516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1bf006adb0 0x7f1bf006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1c1015dd30 tx=0x7f1bfc009210 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:24.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.323+0000 7f1c06000700 1 -- 172.21.15.89:0/2200092516 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1c00054780 con 0x7f1c100ae930 2024-04-02T07:54:24.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.566+0000 7f1c14e00700 1 -- 172.21.15.89:0/2200092516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c10072820 con 0x7f1c100ae930 2024-04-02T07:54:24.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.566+0000 7f1c06000700 1 -- 172.21.15.89:0/2200092516 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f1c00017de0 con 0x7f1c100ae930 2024-04-02T07:54:24.567 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:24.567 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":51,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6317,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1566757639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1566757639},{"type":"v1","addr":"172.21.15.113:6835","nonce":1566757639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":6985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/4102221782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":4102221782},{"type":"v1","addr":"172.21.15.89:6836","nonce":4102221782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/3127911401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3127911401},{"type":"v1","addr":"172.21.15.113:6834","nonce":3127911401}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2912087818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2912087818},{"type":"v1","addr":"172.21.15.89:6837","nonce":2912087818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[]} 2024-04-02T07:54:24.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.569+0000 7f1c14e00700 1 -- 172.21.15.89:0/2200092516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1bf006adb0 msgr2=0x7f1bf006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:24.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.569+0000 7f1c14e00700 1 --2- 172.21.15.89:0/2200092516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1bf006adb0 0x7f1bf006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1c1015dd30 tx=0x7f1bfc009210 comp rx=0 tx=0).stop 2024-04-02T07:54:24.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.569+0000 7f1c14e00700 1 -- 172.21.15.89:0/2200092516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c100ae930 msgr2=0x7f1c100721a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:24.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.569+0000 7f1c14e00700 1 --2- 172.21.15.89:0/2200092516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c100ae930 0x7f1c100721a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000e870 tx=0x7f1c0000eb80 comp rx=0 tx=0).stop 2024-04-02T07:54:24.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.569+0000 7f1c14e00700 1 -- 172.21.15.89:0/2200092516 shutdown_connections 2024-04-02T07:54:24.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.569+0000 7f1c14e00700 1 --2- 172.21.15.89:0/2200092516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1bf006adb0 0x7f1bf006d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:24.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.569+0000 7f1c14e00700 1 --2- 172.21.15.89:0/2200092516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c1015d810 0x7f1c100ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:24.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.569+0000 7f1c14e00700 1 --2- 172.21.15.89:0/2200092516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c100ae930 0x7f1c100721a0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:24.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.569+0000 7f1c14e00700 1 --2- 172.21.15.89:0/2200092516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c1015b5d0 0x7f1c100c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:24.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.569+0000 7f1c14e00700 1 -- 172.21.15.89:0/2200092516 >> 172.21.15.89:0/2200092516 conn(0x7f1c100cf190 msgr2=0x7f1c100af480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:24.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.569+0000 7f1c14e00700 1 -- 172.21.15.89:0/2200092516 shutdown_connections 2024-04-02T07:54:24.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.569+0000 7f1c14e00700 1 -- 172.21.15.89:0/2200092516 wait complete. 2024-04-02T07:54:24.571 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 51 2024-04-02T07:54:24.588 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T07:54:24.588 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T07:54:24.588 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T07:54:24.588 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:54:24.589 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T07:54:24.589 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T07:54:24.589 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T07:54:24.589 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:54:24.589 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T07:54:24.589 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T07:54:24.589 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T07:54:24.589 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:54:24.590 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T07:54:24.590 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T07:54:24.590 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T07:54:24.590 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:54:24.604 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T07:54:24.605 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T07:54:24.605 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T07:54:24.605 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T07:54:24.606 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T07:54:24.607 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T07:54:24.607 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T07:54:24.608 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T07:54:24.649 INFO:tasks.ceph.mds.a:Started 2024-04-02T07:54:24.649 DEBUG:teuthology.parallel:result is None 2024-04-02T07:54:24.651 INFO:tasks.ceph.mds.c:Started 2024-04-02T07:54:24.651 INFO:tasks.ceph.mds.b:Started 2024-04-02T07:54:24.652 DEBUG:teuthology.parallel:result is None 2024-04-02T07:54:24.652 DEBUG:teuthology.parallel:result is None 2024-04-02T07:54:24.653 INFO:tasks.ceph.mds.d:Started 2024-04-02T07:54:24.653 DEBUG:teuthology.parallel:result is None 2024-04-02T07:54:24.654 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T07:54:24.796 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T07:54:24.797 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T07:54:24.797 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T07:54:24.797 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T07:54:24.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.855+0000 7f949dc00700 1 Processor -- start 2024-04-02T07:54:24.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.855+0000 7f949dc00700 1 -- start start 2024-04-02T07:54:24.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.856+0000 7f949dc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f94980719c0 0x7f9498159530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:24.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.856+0000 7f949dc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f949806d120 con 0x7f94980b31d0 2024-04-02T07:54:24.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.856+0000 7f949dc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f949806ebd0 con 0x7f94980715e0 2024-04-02T07:54:24.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.856+0000 7f949dc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94980b3770 con 0x7f94980719c0 2024-04-02T07:54:24.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.856+0000 7f9496000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f94980719c0 0x7f9498159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:24.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.856+0000 7f9496000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f94980719c0 0x7f9498159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43810/0 (socket says 172.21.15.89:43810) 2024-04-02T07:54:24.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.856+0000 7f9496000700 1 -- 172.21.15.89:0/8876360 learned_addr learned my addr 172.21.15.89:0/8876360 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:24.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.856+0000 7f9497400700 1 --1- 172.21.15.89:0/8876360 >> v1:172.21.15.89:6789/0 conn(0x7f94980b31d0 0x7f9498159a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39228/0 (socket says 172.21.15.89:39228) 2024-04-02T07:54:24.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.857+0000 7f9496000700 1 -- 172.21.15.89:0/8876360 >> v1:172.21.15.113:6789/0 conn(0x7f94980715e0 legacy=0x7f94980b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:24.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.857+0000 7f9496000700 1 -- 172.21.15.89:0/8876360 >> v1:172.21.15.89:6789/0 conn(0x7f94980b31d0 legacy=0x7f9498159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:24.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.857+0000 7f9496000700 1 -- 172.21.15.89:0/8876360 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94980b3a50 con 0x7f94980719c0 2024-04-02T07:54:24.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.857+0000 7f9495600700 1 -- 172.21.15.89:0/8876360 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105869991 0 0) 0x7f949806d120 con 0x7f94980b31d0 2024-04-02T07:54:24.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.857+0000 7f9496000700 1 --2- 172.21.15.89:0/8876360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94980719c0 0x7f9498159530 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f948400aaf0 tx=0x7f948400ae00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85a73af335b5b96 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:24.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.857+0000 7f9495600700 1 -- 172.21.15.89:0/8876360 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948400cbb0 con 0x7f94980719c0 2024-04-02T07:54:24.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.857+0000 7f9495600700 1 -- 172.21.15.89:0/8876360 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948400cd10 con 0x7f94980719c0 2024-04-02T07:54:24.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.857+0000 7f9495600700 1 -- 172.21.15.89:0/8876360 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9484018620 con 0x7f94980719c0 2024-04-02T07:54:24.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.858+0000 7f949dc00700 1 -- 172.21.15.89:0/8876360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94980719c0 msgr2=0x7f9498159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:24.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.858+0000 7f949dc00700 1 --2- 172.21.15.89:0/8876360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94980719c0 0x7f9498159530 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f948400aaf0 tx=0x7f948400ae00 comp rx=0 tx=0).stop 2024-04-02T07:54:24.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.858+0000 7f949dc00700 1 -- 172.21.15.89:0/8876360 shutdown_connections 2024-04-02T07:54:24.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.858+0000 7f949dc00700 1 --2- 172.21.15.89:0/8876360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94980719c0 0x7f9498159530 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:24.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.858+0000 7f949dc00700 1 -- 172.21.15.89:0/8876360 >> 172.21.15.89:0/8876360 conn(0x7f94980b1230 msgr2=0x7f94980b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:24.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.858+0000 7f949dc00700 1 -- 172.21.15.89:0/8876360 shutdown_connections 2024-04-02T07:54:24.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.858+0000 7f949dc00700 1 -- 172.21.15.89:0/8876360 wait complete. 2024-04-02T07:54:24.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.860+0000 7f949dc00700 1 Processor -- start 2024-04-02T07:54:24.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.860+0000 7f949dc00700 1 -- start start 2024-04-02T07:54:24.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f949dc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f94980715e0 0x7f9498149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:24.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f949dc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94980719c0 0x7f94980aff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:24.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f949dc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94980b31d0 0x7f94980b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:24.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f949dc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94980cdb00 con 0x7f94980b31d0 2024-04-02T07:54:24.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f949dc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94980cdde0 con 0x7f94980719c0 2024-04-02T07:54:24.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f949dc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94980ce0c0 con 0x7f94980715e0 2024-04-02T07:54:24.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f9497400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94980b31d0 0x7f94980b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:24.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f9497400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94980b31d0 0x7f94980b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43824/0 (socket says 172.21.15.89:43824) 2024-04-02T07:54:24.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f9497400700 1 -- 172.21.15.89:0/1225680 learned_addr learned my addr 172.21.15.89:0/1225680 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:24.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f9496a00700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f94980715e0 0x7f9498149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f9496000700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94980719c0 0x7f94980aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f9497400700 1 -- 172.21.15.89:0/1225680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f94980715e0 msgr2=0x7f9498149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f9497400700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f94980715e0 0x7f9498149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f9497400700 1 -- 172.21.15.89:0/1225680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94980719c0 msgr2=0x7f94980aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f9497400700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94980719c0 0x7f94980aff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f9497400700 1 -- 172.21.15.89:0/1225680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948400a6f0 con 0x7f94980b31d0 2024-04-02T07:54:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f9496a00700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f94980715e0 0x7f9498149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.861+0000 7f9496000700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94980719c0 0x7f94980aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.862+0000 7f9497400700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94980b31d0 0x7f94980b0490 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f949000cb60 tx=0x7f949000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.862+0000 7f948ea00700 1 -- 172.21.15.89:0/1225680 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949000ab50 con 0x7f94980b31d0 2024-04-02T07:54:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.862+0000 7f948ea00700 1 -- 172.21.15.89:0/1225680 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949000acb0 con 0x7f94980b31d0 2024-04-02T07:54:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.862+0000 7f948ea00700 1 -- 172.21.15.89:0/1225680 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9490007310 con 0x7f94980b31d0 2024-04-02T07:54:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.862+0000 7f949dc00700 1 -- 172.21.15.89:0/1225680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94980ce3a0 con 0x7f94980b31d0 2024-04-02T07:54:24.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.862+0000 7f949dc00700 1 -- 172.21.15.89:0/1225680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94980cd720 con 0x7f94980b31d0 2024-04-02T07:54:24.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.863+0000 7f949dc00700 1 -- 172.21.15.89:0/1225680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94980ce8c0 con 0x7f94980b31d0 2024-04-02T07:54:24.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.864+0000 7f948ea00700 1 -- 172.21.15.89:0/1225680 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9490007470 con 0x7f94980b31d0 2024-04-02T07:54:24.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.865+0000 7f948ea00700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f947806adb0 0x7f947806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:24.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.865+0000 7f948ea00700 1 -- 172.21.15.89:0/1225680 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f9490089150 con 0x7f94980b31d0 2024-04-02T07:54:24.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.865+0000 7f9496a00700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f947806adb0 0x7f947806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:24.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.866+0000 7f9496a00700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f947806adb0 0x7f947806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f949814df80 tx=0x7f9488009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:24.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:24.870+0000 7f948ea00700 1 -- 172.21.15.89:0/1225680 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f94980ce8c0 con 0x7f94980b31d0 2024-04-02T07:54:25.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:25.086+0000 7f949dc00700 1 -- 172.21.15.89:0/1225680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f94980ce8c0 con 0x7f94980b31d0 2024-04-02T07:54:25.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:25.842+0000 7f948ea00700 1 -- 172.21.15.89:0/1225680 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v121) v1 ==== 97+0+0 (secure 0 0 0) 0x7f94980ce8c0 con 0x7f94980b31d0 2024-04-02T07:54:25.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:25.846+0000 7f949dc00700 1 -- 172.21.15.89:0/1225680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f947806adb0 msgr2=0x7f947806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:25.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:25.846+0000 7f949dc00700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f947806adb0 0x7f947806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f949814df80 tx=0x7f9488009040 comp rx=0 tx=0).stop 2024-04-02T07:54:25.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:25.846+0000 7f949dc00700 1 -- 172.21.15.89:0/1225680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94980b31d0 msgr2=0x7f94980b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:25.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:25.846+0000 7f949dc00700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94980b31d0 0x7f94980b0490 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f949000cb60 tx=0x7f949000cf20 comp rx=0 tx=0).stop 2024-04-02T07:54:25.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:25.846+0000 7f949dc00700 1 -- 172.21.15.89:0/1225680 shutdown_connections 2024-04-02T07:54:25.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:25.846+0000 7f949dc00700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f947806adb0 0x7f947806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:25.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:25.846+0000 7f949dc00700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94980b31d0 0x7f94980b0490 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:25.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:25.846+0000 7f949dc00700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f94980715e0 0x7f9498149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:25.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:25.846+0000 7f949dc00700 1 --2- 172.21.15.89:0/1225680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94980719c0 0x7f94980aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:25.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:25.846+0000 7f949dc00700 1 -- 172.21.15.89:0/1225680 >> 172.21.15.89:0/1225680 conn(0x7f94980b1230 msgr2=0x7f9498154a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:25.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:25.846+0000 7f949dc00700 1 -- 172.21.15.89:0/1225680 shutdown_connections 2024-04-02T07:54:25.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:25.846+0000 7f949dc00700 1 -- 172.21.15.89:0/1225680 wait complete. 2024-04-02T07:54:25.848 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T07:54:25.867 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T07:54:26.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.061+0000 7fd169800700 1 Processor -- start 2024-04-02T07:54:26.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.062+0000 7fd169800700 1 -- start start 2024-04-02T07:54:26.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.063+0000 7fd169800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd164160f10 0x7fd1640d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.063+0000 7fd169800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd1640cf230 0x7fd1640d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.063+0000 7fd169800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1640ceda0 con 0x7fd164161590 2024-04-02T07:54:26.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.063+0000 7fd169800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1640d23d0 con 0x7fd1640cf230 2024-04-02T07:54:26.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.063+0000 7fd169800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641612f0 con 0x7fd164160f10 2024-04-02T07:54:26.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.063+0000 7fd163400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd164161590 0x7fd1640b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39244/0 (socket says 172.21.15.89:39244) 2024-04-02T07:54:26.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.063+0000 7fd163400700 1 -- 172.21.15.89:0/1530345786 learned_addr learned my addr 172.21.15.89:0/1530345786 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:26.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.063+0000 7fd162a00700 1 --2- 172.21.15.89:0/1530345786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd164160f10 0x7fd1640d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.063+0000 7fd162000700 1 --2- 172.21.15.89:0/1530345786 >> v2:172.21.15.113:3300/0 conn(0x7fd1640cf230 0x7fd1640d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.063+0000 7fd161600700 1 -- 172.21.15.89:0/1530345786 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242235540 0 0) 0x7fd1640ceda0 con 0x7fd164161590 2024-04-02T07:54:26.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.063+0000 7fd161600700 1 -- 172.21.15.89:0/1530345786 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1480035b0 con 0x7fd164161590 2024-04-02T07:54:26.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd161600700 1 -- 172.21.15.89:0/1530345786 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2544383322 0 0) 0x7fd1480035b0 con 0x7fd164161590 2024-04-02T07:54:26.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd161600700 1 -- 172.21.15.89:0/1530345786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd164160f10 msgr2=0x7fd1640d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:26.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd161600700 1 --2- 172.21.15.89:0/1530345786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd164160f10 0x7fd1640d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd161600700 1 -- 172.21.15.89:0/1530345786 >> v2:172.21.15.113:3300/0 conn(0x7fd1640cf230 msgr2=0x7fd1640d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:26.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd161600700 1 --2- 172.21.15.89:0/1530345786 >> v2:172.21.15.113:3300/0 conn(0x7fd1640cf230 0x7fd1640d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd161600700 1 -- 172.21.15.89:0/1530345786 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1640ce470 con 0x7fd164161590 2024-04-02T07:54:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd161600700 1 -- 172.21.15.89:0/1530345786 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fd15c0019f0 con 0x7fd164161590 2024-04-02T07:54:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd161600700 1 -- 172.21.15.89:0/1530345786 >> v1:172.21.15.89:6789/0 conn(0x7fd164161590 legacy=0x7fd1640b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd161600700 1 --2- 172.21.15.89:0/1530345786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd164160f10 0x7fd148003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd161600700 1 --2- 172.21.15.89:0/1530345786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd148003f80 0x7fd148004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd161600700 1 --2- 172.21.15.89:0/1530345786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd148004d90 0x7fd148005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd161600700 1 -- 172.21.15.89:0/1530345786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640d23d0 con 0x7fd148003f80 2024-04-02T07:54:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd161600700 1 -- 172.21.15.89:0/1530345786 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641612f0 con 0x7fd148004d90 2024-04-02T07:54:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd161600700 1 -- 172.21.15.89:0/1530345786 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14800ba60 con 0x7fd164160f10 2024-04-02T07:54:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd162000700 1 --2- 172.21.15.89:0/1530345786 >> v2:172.21.15.113:3300/0 conn(0x7fd1640cf230 0x7fd1640d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd162000700 1 --2- 172.21.15.89:0/1530345786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd148003f80 0x7fd148004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd163400700 1 --2- 172.21.15.89:0/1530345786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd148004d90 0x7fd148005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.064+0000 7fd162a00700 1 --2- 172.21.15.89:0/1530345786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd164160f10 0x7fd148003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.065+0000 7fd162000700 1 -- 172.21.15.89:0/1530345786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd164160f10 msgr2=0x7fd148003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:26.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.065+0000 7fd162000700 1 --2- 172.21.15.89:0/1530345786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd164160f10 0x7fd148003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.065+0000 7fd162000700 1 -- 172.21.15.89:0/1530345786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd148004d90 msgr2=0x7fd148005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:26.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.065+0000 7fd162000700 1 --2- 172.21.15.89:0/1530345786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd148004d90 0x7fd148005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.065+0000 7fd162000700 1 -- 172.21.15.89:0/1530345786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1480035b0 con 0x7fd148003f80 2024-04-02T07:54:26.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.065+0000 7fd162000700 1 --2- 172.21.15.89:0/1530345786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd148003f80 0x7fd148004850 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fd150009d40 tx=0x7fd148005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50cef076c0dfc0e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:26.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.065+0000 7fd161600700 1 -- 172.21.15.89:0/1530345786 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150009e90 con 0x7fd148003f80 2024-04-02T07:54:26.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.065+0000 7fd161600700 1 -- 172.21.15.89:0/1530345786 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15000e9a0 con 0x7fd148003f80 2024-04-02T07:54:26.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.065+0000 7fd161600700 1 -- 172.21.15.89:0/1530345786 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15000ed10 con 0x7fd148003f80 2024-04-02T07:54:26.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.066+0000 7fd169800700 1 -- 172.21.15.89:0/1530345786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd148003f80 msgr2=0x7fd148004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:26.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.066+0000 7fd169800700 1 --2- 172.21.15.89:0/1530345786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd148003f80 0x7fd148004850 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fd150009d40 tx=0x7fd148005880 comp rx=0 tx=0).stop 2024-04-02T07:54:26.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.066+0000 7fd163400700 1 -- 172.21.15.89:0/1530345786 reap_dead start 2024-04-02T07:54:26.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.066+0000 7fd169800700 1 -- 172.21.15.89:0/1530345786 shutdown_connections 2024-04-02T07:54:26.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.066+0000 7fd169800700 1 -- 172.21.15.89:0/1530345786 >> 172.21.15.89:0/1530345786 conn(0x7fd16408aff0 msgr2=0x7fd1640d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:26.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.066+0000 7fd169800700 1 -- 172.21.15.89:0/1530345786 shutdown_connections 2024-04-02T07:54:26.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.066+0000 7fd169800700 1 -- 172.21.15.89:0/1530345786 wait complete. 2024-04-02T07:54:26.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.068+0000 7fd169800700 1 Processor -- start 2024-04-02T07:54:26.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.068+0000 7fd169800700 1 -- start start 2024-04-02T07:54:26.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd169800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd148003f80 0x7fd1641f3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd169800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1640b7ce0 0x7fd1640c9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd169800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1641f2fc0 0x7fd1641f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd169800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641f3980 con 0x7fd148003f80 2024-04-02T07:54:26.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd169800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16415ece0 con 0x7fd1640b7ce0 2024-04-02T07:54:26.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd169800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640b84a0 con 0x7fd1641f2fc0 2024-04-02T07:54:26.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd162a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd148003f80 0x7fd1641f3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd162a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd148003f80 0x7fd1641f3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43846/0 (socket says 172.21.15.89:43846) 2024-04-02T07:54:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd162a00700 1 -- 172.21.15.89:0/2553638802 learned_addr learned my addr 172.21.15.89:0/2553638802 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd162000700 1 --2- 172.21.15.89:0/2553638802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1640b7ce0 0x7fd1640c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd163400700 1 --2- 172.21.15.89:0/2553638802 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1641f2fc0 0x7fd1641f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd162a00700 1 -- 172.21.15.89:0/2553638802 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1641f2fc0 msgr2=0x7fd1641f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd162a00700 1 --2- 172.21.15.89:0/2553638802 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1641f2fc0 0x7fd1641f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd162a00700 1 -- 172.21.15.89:0/2553638802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1640b7ce0 msgr2=0x7fd1640c9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd162a00700 1 --2- 172.21.15.89:0/2553638802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1640b7ce0 0x7fd1640c9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.069+0000 7fd162a00700 1 -- 172.21.15.89:0/2553638802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1640b8780 con 0x7fd148003f80 2024-04-02T07:54:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.070+0000 7fd162a00700 1 --2- 172.21.15.89:0/2553638802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd148003f80 0x7fd1641f3b60 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fd154011aa0 tx=0x7fd154011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.070+0000 7fd15aa00700 1 -- 172.21.15.89:0/2553638802 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15400f980 con 0x7fd148003f80 2024-04-02T07:54:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.070+0000 7fd15aa00700 1 -- 172.21.15.89:0/2553638802 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15400fae0 con 0x7fd148003f80 2024-04-02T07:54:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.070+0000 7fd169800700 1 -- 172.21.15.89:0/2553638802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1640b8a60 con 0x7fd148003f80 2024-04-02T07:54:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.070+0000 7fd15aa00700 1 -- 172.21.15.89:0/2553638802 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15400fe90 con 0x7fd148003f80 2024-04-02T07:54:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.070+0000 7fd169800700 1 -- 172.21.15.89:0/2553638802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1640b8160 con 0x7fd148003f80 2024-04-02T07:54:26.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.071+0000 7fd169800700 1 -- 172.21.15.89:0/2553638802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16415efc0 con 0x7fd148003f80 2024-04-02T07:54:26.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.072+0000 7fd15aa00700 1 -- 172.21.15.89:0/2553638802 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd154014440 con 0x7fd148003f80 2024-04-02T07:54:26.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.072+0000 7fd15aa00700 1 --2- 172.21.15.89:0/2553638802 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd14406adb0 0x7fd14406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.073+0000 7fd15aa00700 1 -- 172.21.15.89:0/2553638802 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fd1540899c0 con 0x7fd148003f80 2024-04-02T07:54:26.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.073+0000 7fd162000700 1 --2- 172.21.15.89:0/2553638802 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd14406adb0 0x7fd14406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.073+0000 7fd162000700 1 --2- 172.21.15.89:0/2553638802 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd14406adb0 0x7fd14406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd1500090b0 tx=0x7fd150008030 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:26.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.077+0000 7fd15aa00700 1 -- 172.21.15.89:0/2553638802 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd154054d40 con 0x7fd148003f80 2024-04-02T07:54:26.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.363+0000 7fd169800700 1 -- 172.21.15.89:0/2553638802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd16415eb20 con 0x7fd148003f80 2024-04-02T07:54:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.363+0000 7fd15aa00700 1 -- 172.21.15.89:0/2553638802 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd16415eb20 con 0x7fd148003f80 2024-04-02T07:54:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:26.364 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:54:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T07:54:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T07:54:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T07:54:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:54:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:54:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:54:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T07:54:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T07:54:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:54:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:54:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:54:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T07:54:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T07:54:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:54:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:54:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:54:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T07:54:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T07:54:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:54:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:54:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:54:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:54:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T07:54:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T07:54:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:54:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:54:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T07:54:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T07:54:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:54:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:54:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T07:54:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T07:54:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:54:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:54:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T07:54:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T07:54:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:54:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:54:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T07:54:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T07:54:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T07:54:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:54:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2024-04-02T07:54:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T07:54:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T07:54:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T07:54:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T07:54:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T07:54:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:54:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:54:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:54:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:54:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T07:54:26.377 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:54:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.365+0000 7fd169800700 1 -- 172.21.15.89:0/2553638802 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd14406adb0 msgr2=0x7fd14406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.365+0000 7fd169800700 1 --2- 172.21.15.89:0/2553638802 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd14406adb0 0x7fd14406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd1500090b0 tx=0x7fd150008030 comp rx=0 tx=0).stop 2024-04-02T07:54:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.365+0000 7fd169800700 1 -- 172.21.15.89:0/2553638802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd148003f80 msgr2=0x7fd1641f3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.365+0000 7fd169800700 1 --2- 172.21.15.89:0/2553638802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd148003f80 0x7fd1641f3b60 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fd154011aa0 tx=0x7fd154011e60 comp rx=0 tx=0).stop 2024-04-02T07:54:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.366+0000 7fd169800700 1 -- 172.21.15.89:0/2553638802 shutdown_connections 2024-04-02T07:54:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.366+0000 7fd169800700 1 --2- 172.21.15.89:0/2553638802 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd14406adb0 0x7fd14406d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.366+0000 7fd169800700 1 --2- 172.21.15.89:0/2553638802 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1641f2fc0 0x7fd1641f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.366+0000 7fd169800700 1 --2- 172.21.15.89:0/2553638802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd148003f80 0x7fd1641f3b60 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.366+0000 7fd169800700 1 --2- 172.21.15.89:0/2553638802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1640b7ce0 0x7fd1640c9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.366+0000 7fd169800700 1 -- 172.21.15.89:0/2553638802 >> 172.21.15.89:0/2553638802 conn(0x7fd16408aff0 msgr2=0x7fd1640cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:26.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.366+0000 7fd169800700 1 -- 172.21.15.89:0/2553638802 shutdown_connections 2024-04-02T07:54:26.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.366+0000 7fd169800700 1 -- 172.21.15.89:0/2553638802 wait complete. 2024-04-02T07:54:26.385 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:54:26.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.576+0000 7f31f9400700 1 Processor -- start 2024-04-02T07:54:26.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.576+0000 7f31f9400700 1 -- start start 2024-04-02T07:54:26.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.577+0000 7f31f9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31f40b3af0 0x7f31f415a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.577+0000 7f31f9400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f31f4072580 0x7f31f40b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.577+0000 7f31f9400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f31f40b2e50 0x7f31f40709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.577+0000 7f31f9400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f40b2180 con 0x7f31f4072580 2024-04-02T07:54:26.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.577+0000 7f31f9400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f415b100 con 0x7f31f40b2e50 2024-04-02T07:54:26.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.577+0000 7f31f9400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f415b3e0 con 0x7f31f40b3af0 2024-04-02T07:54:26.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.578+0000 7f31f1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f31f4072580 0x7f31f40b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.578+0000 7f31f1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f31f4072580 0x7f31f40b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43856/0 (socket says 172.21.15.89:43856) 2024-04-02T07:54:26.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.578+0000 7f31f1e00700 1 -- 172.21.15.89:0/798101895 learned_addr learned my addr 172.21.15.89:0/798101895 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:26.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.578+0000 7f31f3200700 1 --2- 172.21.15.89:0/798101895 >> v2:172.21.15.113:3300/0 conn(0x7f31f40b2e50 0x7f31f40709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.578+0000 7f31f2800700 1 --2- 172.21.15.89:0/798101895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31f40b3af0 0x7f31f415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.578+0000 7f31f1e00700 1 -- 172.21.15.89:0/798101895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31f40b3af0 msgr2=0x7f31f415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:26.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.578+0000 7f31f1e00700 1 --2- 172.21.15.89:0/798101895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31f40b3af0 0x7f31f415a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.578+0000 7f31f1e00700 1 -- 172.21.15.89:0/798101895 >> v2:172.21.15.113:3300/0 conn(0x7f31f40b2e50 msgr2=0x7f31f40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:26.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.578+0000 7f31f1e00700 1 --2- 172.21.15.89:0/798101895 >> v2:172.21.15.113:3300/0 conn(0x7f31f40b2e50 0x7f31f40709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.578+0000 7f31f1e00700 1 -- 172.21.15.89:0/798101895 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f40b23f0 con 0x7f31f4072580 2024-04-02T07:54:26.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.578+0000 7f31f1e00700 1 --2- 172.21.15.89:0/798101895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f4072580 0x7f31f40b2910 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f31f415dd90 tx=0x7f31e000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8167c0c97fe0150 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:26.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.579+0000 7f31f1400700 1 -- 172.21.15.89:0/798101895 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e000dc70 con 0x7f31f4072580 2024-04-02T07:54:26.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.579+0000 7f31f1400700 1 -- 172.21.15.89:0/798101895 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e000ddd0 con 0x7f31f4072580 2024-04-02T07:54:26.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.579+0000 7f31f1400700 1 -- 172.21.15.89:0/798101895 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0010690 con 0x7f31f4072580 2024-04-02T07:54:26.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.579+0000 7f31f9400700 1 -- 172.21.15.89:0/798101895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f4072580 msgr2=0x7f31f40b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:26.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.579+0000 7f31f9400700 1 --2- 172.21.15.89:0/798101895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f4072580 0x7f31f40b2910 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f31f415dd90 tx=0x7f31e000be70 comp rx=0 tx=0).stop 2024-04-02T07:54:26.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.579+0000 7f31f9400700 1 -- 172.21.15.89:0/798101895 shutdown_connections 2024-04-02T07:54:26.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.579+0000 7f31f9400700 1 --2- 172.21.15.89:0/798101895 >> v2:172.21.15.113:3300/0 conn(0x7f31f40b2e50 0x7f31f40709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.579+0000 7f31f9400700 1 --2- 172.21.15.89:0/798101895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31f40b3af0 0x7f31f415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.579+0000 7f31f9400700 1 --2- 172.21.15.89:0/798101895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f4072580 0x7f31f40b2910 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.579+0000 7f31f9400700 1 -- 172.21.15.89:0/798101895 >> 172.21.15.89:0/798101895 conn(0x7f31f40cf190 msgr2=0x7f31f40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:26.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.579+0000 7f31f9400700 1 -- 172.21.15.89:0/798101895 shutdown_connections 2024-04-02T07:54:26.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.580+0000 7f31f9400700 1 -- 172.21.15.89:0/798101895 wait complete. 2024-04-02T07:54:26.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.581+0000 7f31f9400700 1 Processor -- start 2024-04-02T07:54:26.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.581+0000 7f31f9400700 1 -- start start 2024-04-02T07:54:26.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.582+0000 7f31f9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31f4072580 0x7f31f4147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.582+0000 7f31f9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f40b2e50 0x7f31f4159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.582+0000 7f31f9400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f40b3af0 0x7f31f4149230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.582+0000 7f31f9400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f40cbd40 con 0x7f31f40b2e50 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.582+0000 7f31f9400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f40cbfe0 con 0x7f31f40b3af0 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.582+0000 7f31f9400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f40cc2c0 con 0x7f31f4072580 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.582+0000 7f31f1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f40b2e50 0x7f31f4159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.582+0000 7f31f1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f40b2e50 0x7f31f4159e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43870/0 (socket says 172.21.15.89:43870) 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.582+0000 7f31f1e00700 1 -- 172.21.15.89:0/2013195207 learned_addr learned my addr 172.21.15.89:0/2013195207 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.582+0000 7f31f2800700 1 --2- 172.21.15.89:0/2013195207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31f4072580 0x7f31f4147350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.582+0000 7f31f3200700 1 --2- 172.21.15.89:0/2013195207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f40b3af0 0x7f31f4149230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.583+0000 7f31f1e00700 1 -- 172.21.15.89:0/2013195207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31f4072580 msgr2=0x7f31f4147350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.583+0000 7f31f1e00700 1 --2- 172.21.15.89:0/2013195207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31f4072580 0x7f31f4147350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.583+0000 7f31f1e00700 1 -- 172.21.15.89:0/2013195207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f40b3af0 msgr2=0x7f31f4149230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.583+0000 7f31f1e00700 1 --2- 172.21.15.89:0/2013195207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f40b3af0 0x7f31f4149230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.583+0000 7f31f1e00700 1 -- 172.21.15.89:0/2013195207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e000b7f0 con 0x7f31f40b2e50 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.583+0000 7f31f1e00700 1 --2- 172.21.15.89:0/2013195207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f40b2e50 0x7f31f4159e50 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f31e000beb0 tx=0x7f31e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.583+0000 7f31eaa00700 1 -- 172.21.15.89:0/2013195207 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0003ab0 con 0x7f31f40b2e50 2024-04-02T07:54:26.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.583+0000 7f31eaa00700 1 -- 172.21.15.89:0/2013195207 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e0003c30 con 0x7f31f40b2e50 2024-04-02T07:54:26.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.583+0000 7f31f9400700 1 -- 172.21.15.89:0/2013195207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f40cc5a0 con 0x7f31f40b2e50 2024-04-02T07:54:26.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.583+0000 7f31eaa00700 1 -- 172.21.15.89:0/2013195207 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0010470 con 0x7f31f40b2e50 2024-04-02T07:54:26.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.583+0000 7f31f9400700 1 -- 172.21.15.89:0/2013195207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f4147b60 con 0x7f31f40b2e50 2024-04-02T07:54:26.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.584+0000 7f31f9400700 1 -- 172.21.15.89:0/2013195207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f4067720 con 0x7f31f40b2e50 2024-04-02T07:54:26.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.585+0000 7f31eaa00700 1 -- 172.21.15.89:0/2013195207 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f31e0019400 con 0x7f31f40b2e50 2024-04-02T07:54:26.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.586+0000 7f31eaa00700 1 --2- 172.21.15.89:0/2013195207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f31d406adb0 0x7f31d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:26.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.586+0000 7f31eaa00700 1 -- 172.21.15.89:0/2013195207 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f31e008a4d0 con 0x7f31f40b2e50 2024-04-02T07:54:26.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.586+0000 7f31f2800700 1 --2- 172.21.15.89:0/2013195207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f31d406adb0 0x7f31d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:26.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.587+0000 7f31f2800700 1 --2- 172.21.15.89:0/2013195207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f31d406adb0 0x7f31d406d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f31f41480d0 tx=0x7f31e4009210 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:26.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.590+0000 7f31eaa00700 1 -- 172.21.15.89:0/2013195207 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f31e0055670 con 0x7f31f40b2e50 2024-04-02T07:54:26.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.793+0000 7f31f9400700 1 -- 172.21.15.89:0/2013195207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31f406e970 con 0x7f31f40b2e50 2024-04-02T07:54:26.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.794+0000 7f31eaa00700 1 -- 172.21.15.89:0/2013195207 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+12926 (secure 0 0 0) 0x7f31e0058c90 con 0x7f31f40b2e50 2024-04-02T07:54:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:26.796 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":121,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:54:25.836754+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:54:26.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.797+0000 7f31f9400700 1 -- 172.21.15.89:0/2013195207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f31d406adb0 msgr2=0x7f31d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:26.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.797+0000 7f31f9400700 1 --2- 172.21.15.89:0/2013195207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f31d406adb0 0x7f31d406d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f31f41480d0 tx=0x7f31e4009210 comp rx=0 tx=0).stop 2024-04-02T07:54:26.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.797+0000 7f31f9400700 1 -- 172.21.15.89:0/2013195207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f40b2e50 msgr2=0x7f31f4159e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:26.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.797+0000 7f31f9400700 1 --2- 172.21.15.89:0/2013195207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f40b2e50 0x7f31f4159e50 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f31e000beb0 tx=0x7f31e0004750 comp rx=0 tx=0).stop 2024-04-02T07:54:26.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.797+0000 7f31f9400700 1 -- 172.21.15.89:0/2013195207 shutdown_connections 2024-04-02T07:54:26.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.797+0000 7f31f9400700 1 --2- 172.21.15.89:0/2013195207 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f31d406adb0 0x7f31d406d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.797+0000 7f31f9400700 1 --2- 172.21.15.89:0/2013195207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f40b3af0 0x7f31f4149230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.797+0000 7f31f9400700 1 --2- 172.21.15.89:0/2013195207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31f4072580 0x7f31f4147350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.797+0000 7f31f9400700 1 --2- 172.21.15.89:0/2013195207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f40b2e50 0x7f31f4159e50 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:26.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.797+0000 7f31f9400700 1 -- 172.21.15.89:0/2013195207 >> 172.21.15.89:0/2013195207 conn(0x7f31f40cf190 msgr2=0x7f31f40aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:26.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.797+0000 7f31f9400700 1 -- 172.21.15.89:0/2013195207 shutdown_connections 2024-04-02T07:54:26.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:26.797+0000 7f31f9400700 1 -- 172.21.15.89:0/2013195207 wait complete. 2024-04-02T07:54:26.818 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:54:26.818 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:54:27.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.010+0000 7f8e1b200700 1 Processor -- start 2024-04-02T07:54:27.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.010+0000 7f8e1b200700 1 -- start start 2024-04-02T07:54:27.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.011+0000 7f8e1b200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8e140b3f80 0x7f8e140ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.011+0000 7f8e1b200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e14161230 con 0x7f8e14160e50 2024-04-02T07:54:27.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.011+0000 7f8e1b200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e140d2a80 con 0x7f8e140d3f20 2024-04-02T07:54:27.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.012+0000 7f8e1b200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e140d43b0 con 0x7f8e140b3f80 2024-04-02T07:54:27.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.012+0000 7f8e19200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8e140b3f80 0x7f8e140ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.012+0000 7f8e13e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8e14160e50 0x7f8e140b4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39254/0 (socket says 172.21.15.89:39254) 2024-04-02T07:54:27.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.012+0000 7f8e13e00700 1 -- 172.21.15.89:0/1314033142 learned_addr learned my addr 172.21.15.89:0/1314033142 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:27.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.012+0000 7f8e19200700 1 -- 172.21.15.89:0/1314033142 >> v1:172.21.15.113:6789/0 conn(0x7f8e140d3f20 legacy=0x7f8e140cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:27.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.012+0000 7f8e19200700 1 -- 172.21.15.89:0/1314033142 >> v1:172.21.15.89:6789/0 conn(0x7f8e14160e50 legacy=0x7f8e140b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.012+0000 7f8e19200700 1 -- 172.21.15.89:0/1314033142 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e140d21e0 con 0x7f8e140b3f80 2024-04-02T07:54:27.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.012+0000 7f8e12a00700 1 -- 172.21.15.89:0/1314033142 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647995364 0 0) 0x7f8e14161230 con 0x7f8e14160e50 2024-04-02T07:54:27.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.012+0000 7f8e19200700 1 --2- 172.21.15.89:0/1314033142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e140b3f80 0x7f8e140ce600 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c009d90 tx=0x7f8e0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616596bca381b423 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.013+0000 7f8e12a00700 1 -- 172.21.15.89:0/1314033142 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c00e070 con 0x7f8e140b3f80 2024-04-02T07:54:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.013+0000 7f8e12a00700 1 -- 172.21.15.89:0/1314033142 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c00bbd0 con 0x7f8e140b3f80 2024-04-02T07:54:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.013+0000 7f8e12a00700 1 -- 172.21.15.89:0/1314033142 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c012430 con 0x7f8e140b3f80 2024-04-02T07:54:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.013+0000 7f8e1b200700 1 -- 172.21.15.89:0/1314033142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e140b3f80 msgr2=0x7f8e140ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.013+0000 7f8e1b200700 1 --2- 172.21.15.89:0/1314033142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e140b3f80 0x7f8e140ce600 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c009d90 tx=0x7f8e0c00b330 comp rx=0 tx=0).stop 2024-04-02T07:54:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.013+0000 7f8e1b200700 1 -- 172.21.15.89:0/1314033142 shutdown_connections 2024-04-02T07:54:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.013+0000 7f8e1b200700 1 --2- 172.21.15.89:0/1314033142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e140b3f80 0x7f8e140ce600 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.013+0000 7f8e1b200700 1 -- 172.21.15.89:0/1314033142 >> 172.21.15.89:0/1314033142 conn(0x7f8e1408aff0 msgr2=0x7f8e140d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.013+0000 7f8e1b200700 1 -- 172.21.15.89:0/1314033142 shutdown_connections 2024-04-02T07:54:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.013+0000 7f8e1b200700 1 -- 172.21.15.89:0/1314033142 wait complete. 2024-04-02T07:54:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.015+0000 7f8e1b200700 1 Processor -- start 2024-04-02T07:54:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.015+0000 7f8e1b200700 1 -- start start 2024-04-02T07:54:27.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.016+0000 7f8e1b200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e140b3f80 0x7f8e1406c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.016+0000 7f8e1b200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e14160e50 0x7f8e1415b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.016+0000 7f8e1b200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e140c7580 0x7f8e1415baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.016+0000 7f8e1b200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1415c030 con 0x7f8e14160e50 2024-04-02T07:54:27.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.016+0000 7f8e1b200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140cc260 con 0x7f8e140c7580 2024-04-02T07:54:27.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.016+0000 7f8e1b200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140cb650 con 0x7f8e140b3f80 2024-04-02T07:54:27.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.016+0000 7f8e13400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e14160e50 0x7f8e1415b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.016+0000 7f8e13400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e14160e50 0x7f8e1415b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43878/0 (socket says 172.21.15.89:43878) 2024-04-02T07:54:27.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.016+0000 7f8e13400700 1 -- 172.21.15.89:0/3989670807 learned_addr learned my addr 172.21.15.89:0/3989670807 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:27.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.017+0000 7f8e19200700 1 --2- 172.21.15.89:0/3989670807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e140c7580 0x7f8e1415baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.017+0000 7f8e13e00700 1 --2- 172.21.15.89:0/3989670807 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e140b3f80 0x7f8e1406c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.017+0000 7f8e13400700 1 -- 172.21.15.89:0/3989670807 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e140b3f80 msgr2=0x7f8e1406c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.017+0000 7f8e13400700 1 --2- 172.21.15.89:0/3989670807 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e140b3f80 0x7f8e1406c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.017+0000 7f8e13400700 1 -- 172.21.15.89:0/3989670807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e140c7580 msgr2=0x7f8e1415baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.017+0000 7f8e13400700 1 --2- 172.21.15.89:0/3989670807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e140c7580 0x7f8e1415baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.017+0000 7f8e13400700 1 -- 172.21.15.89:0/3989670807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c009a70 con 0x7f8e14160e50 2024-04-02T07:54:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.017+0000 7f8e13400700 1 --2- 172.21.15.89:0/3989670807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e14160e50 0x7f8e1415b5b0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000c4e0 tx=0x7f8e0000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.017+0000 7f8e0be00700 1 -- 172.21.15.89:0/3989670807 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e000052d0 con 0x7f8e14160e50 2024-04-02T07:54:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.017+0000 7f8e0be00700 1 -- 172.21.15.89:0/3989670807 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e00005430 con 0x7f8e14160e50 2024-04-02T07:54:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.017+0000 7f8e1b200700 1 -- 172.21.15.89:0/3989670807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e140cb930 con 0x7f8e14160e50 2024-04-02T07:54:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.017+0000 7f8e0be00700 1 -- 172.21.15.89:0/3989670807 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e000038c0 con 0x7f8e14160e50 2024-04-02T07:54:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.017+0000 7f8e1b200700 1 -- 172.21.15.89:0/3989670807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e140cbef0 con 0x7f8e14160e50 2024-04-02T07:54:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.018+0000 7f8e1b200700 1 -- 172.21.15.89:0/3989670807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e140cc410 con 0x7f8e14160e50 2024-04-02T07:54:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.019+0000 7f8e0be00700 1 -- 172.21.15.89:0/3989670807 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8e00003a20 con 0x7f8e14160e50 2024-04-02T07:54:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.020+0000 7f8e0be00700 1 --2- 172.21.15.89:0/3989670807 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8df406adb0 0x7f8df406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.020+0000 7f8e0be00700 1 -- 172.21.15.89:0/3989670807 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f8e00088770 con 0x7f8e14160e50 2024-04-02T07:54:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.020+0000 7f8e13e00700 1 --2- 172.21.15.89:0/3989670807 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8df406adb0 0x7f8df406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.021+0000 7f8e13e00700 1 --2- 172.21.15.89:0/3989670807 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8df406adb0 0x7f8df406d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e04000f60 tx=0x7f8e04007d90 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:27.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.024+0000 7f8e0be00700 1 -- 172.21.15.89:0/3989670807 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8e140cc410 con 0x7f8e14160e50 2024-04-02T07:54:27.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.220+0000 7f8e1b200700 1 -- 172.21.15.89:0/3989670807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8e140cc3d0 con 0x7f8e14160e50 2024-04-02T07:54:27.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.220+0000 7f8e0be00700 1 -- 172.21.15.89:0/3989670807 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8e00017020 con 0x7f8e14160e50 2024-04-02T07:54:27.221 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:54:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.223+0000 7f8e1b200700 1 -- 172.21.15.89:0/3989670807 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8df406adb0 msgr2=0x7f8df406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.223+0000 7f8e1b200700 1 --2- 172.21.15.89:0/3989670807 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8df406adb0 0x7f8df406d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e04000f60 tx=0x7f8e04007d90 comp rx=0 tx=0).stop 2024-04-02T07:54:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.223+0000 7f8e1b200700 1 -- 172.21.15.89:0/3989670807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e14160e50 msgr2=0x7f8e1415b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.223+0000 7f8e1b200700 1 --2- 172.21.15.89:0/3989670807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e14160e50 0x7f8e1415b5b0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000c4e0 tx=0x7f8e0000c8a0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.223+0000 7f8e1b200700 1 -- 172.21.15.89:0/3989670807 shutdown_connections 2024-04-02T07:54:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.223+0000 7f8e1b200700 1 --2- 172.21.15.89:0/3989670807 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8df406adb0 0x7f8df406d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.223+0000 7f8e1b200700 1 --2- 172.21.15.89:0/3989670807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e140c7580 0x7f8e1415baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.223+0000 7f8e1b200700 1 --2- 172.21.15.89:0/3989670807 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e140b3f80 0x7f8e1406c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.223+0000 7f8e1b200700 1 --2- 172.21.15.89:0/3989670807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e14160e50 0x7f8e1415b5b0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.223+0000 7f8e1b200700 1 -- 172.21.15.89:0/3989670807 >> 172.21.15.89:0/3989670807 conn(0x7f8e1408aff0 msgr2=0x7f8e140d3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.223+0000 7f8e1b200700 1 -- 172.21.15.89:0/3989670807 shutdown_connections 2024-04-02T07:54:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.223+0000 7f8e1b200700 1 -- 172.21.15.89:0/3989670807 wait complete. 2024-04-02T07:54:27.242 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T07:54:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.433+0000 7f8f3d800700 1 Processor -- start 2024-04-02T07:54:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.433+0000 7f8f3d800700 1 -- start start 2024-04-02T07:54:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.434+0000 7f8f3d800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8f38160f10 0x7f8f380d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.435+0000 7f8f3d800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f380ceda0 con 0x7f8f380cf230 2024-04-02T07:54:27.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.435+0000 7f8f3d800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f381612f0 con 0x7f8f38161590 2024-04-02T07:54:27.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.435+0000 7f8f3d800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f380ce470 con 0x7f8f38160f10 2024-04-02T07:54:27.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.435+0000 7f8f36000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8f380cf230 0x7f8f380d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39266/0 (socket says 172.21.15.89:39266) 2024-04-02T07:54:27.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.435+0000 7f8f36000700 1 -- 172.21.15.89:0/2786036374 learned_addr learned my addr 172.21.15.89:0/2786036374 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:27.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.435+0000 7f8f36a00700 1 --2- 172.21.15.89:0/2786036374 >> v2:172.21.15.113:3300/0 conn(0x7f8f38160f10 0x7f8f380d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.435+0000 7f8f35600700 1 -- 172.21.15.89:0/2786036374 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497972186 0 0) 0x7f8f380ceda0 con 0x7f8f380cf230 2024-04-02T07:54:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.435+0000 7f8f35600700 1 -- 172.21.15.89:0/2786036374 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c0035b0 con 0x7f8f380cf230 2024-04-02T07:54:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.436+0000 7f8f35600700 1 -- 172.21.15.89:0/2786036374 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3194261627 0 0) 0x7f8f1c0035b0 con 0x7f8f380cf230 2024-04-02T07:54:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.436+0000 7f8f36a00700 1 -- 172.21.15.89:0/2786036374 >> v1:172.21.15.113:6789/0 conn(0x7f8f38161590 legacy=0x7f8f380b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.436+0000 7f8f36a00700 1 -- 172.21.15.89:0/2786036374 >> v1:172.21.15.89:6789/0 conn(0x7f8f380cf230 legacy=0x7f8f380d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.436+0000 7f8f36a00700 1 -- 172.21.15.89:0/2786036374 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f380ce750 con 0x7f8f38160f10 2024-04-02T07:54:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.436+0000 7f8f36a00700 1 --2- 172.21.15.89:0/2786036374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f38160f10 0x7f8f380d18b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009840 tx=0x7f8f28009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f48fe01451588dc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.436+0000 7f8f35600700 1 -- 172.21.15.89:0/2786036374 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f24001670 con 0x7f8f38160f10 2024-04-02T07:54:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.436+0000 7f8f35600700 1 -- 172.21.15.89:0/2786036374 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f28004030 con 0x7f8f38160f10 2024-04-02T07:54:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.436+0000 7f8f35600700 1 -- 172.21.15.89:0/2786036374 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f280043c0 con 0x7f8f38160f10 2024-04-02T07:54:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.437+0000 7f8f3d800700 1 -- 172.21.15.89:0/2786036374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f38160f10 msgr2=0x7f8f380d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.437+0000 7f8f3d800700 1 --2- 172.21.15.89:0/2786036374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f38160f10 0x7f8f380d18b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009840 tx=0x7f8f28009b50 comp rx=0 tx=0).stop 2024-04-02T07:54:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.437+0000 7f8f3d800700 1 -- 172.21.15.89:0/2786036374 shutdown_connections 2024-04-02T07:54:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.437+0000 7f8f3d800700 1 --2- 172.21.15.89:0/2786036374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f38160f10 0x7f8f380d18b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.437+0000 7f8f3d800700 1 -- 172.21.15.89:0/2786036374 >> 172.21.15.89:0/2786036374 conn(0x7f8f3808aff0 msgr2=0x7f8f380d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.437+0000 7f8f3d800700 1 -- 172.21.15.89:0/2786036374 shutdown_connections 2024-04-02T07:54:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.437+0000 7f8f3d800700 1 -- 172.21.15.89:0/2786036374 wait complete. 2024-04-02T07:54:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.439+0000 7f8f3d800700 1 Processor -- start 2024-04-02T07:54:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.439+0000 7f8f3d800700 1 -- start start 2024-04-02T07:54:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f3d800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f380cf230 0x7f8f380c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f3d800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f38160f10 0x7f8f3815f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f3d800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f38161590 0x7f8f3815fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f3d800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3815d020 con 0x7f8f38160f10 2024-04-02T07:54:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f3d800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f38160160 con 0x7f8f38161590 2024-04-02T07:54:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f3d800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f381e7d00 con 0x7f8f380cf230 2024-04-02T07:54:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f36000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f38160f10 0x7f8f3815f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f36000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f38160f10 0x7f8f3815f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43894/0 (socket says 172.21.15.89:43894) 2024-04-02T07:54:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f36000700 1 -- 172.21.15.89:0/375225805 learned_addr learned my addr 172.21.15.89:0/375225805 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f36a00700 1 --2- 172.21.15.89:0/375225805 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f380cf230 0x7f8f380c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f37400700 1 --2- 172.21.15.89:0/375225805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f38161590 0x7f8f3815fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f36000700 1 -- 172.21.15.89:0/375225805 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f380cf230 msgr2=0x7f8f380c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f36000700 1 --2- 172.21.15.89:0/375225805 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f380cf230 0x7f8f380c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f36000700 1 -- 172.21.15.89:0/375225805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f38161590 msgr2=0x7f8f3815fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f36000700 1 --2- 172.21.15.89:0/375225805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f38161590 0x7f8f3815fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.440+0000 7f8f36000700 1 -- 172.21.15.89:0/375225805 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f280094a0 con 0x7f8f38160f10 2024-04-02T07:54:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.441+0000 7f8f36000700 1 --2- 172.21.15.89:0/375225805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f38160f10 0x7f8f3815f6e0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f8f2400c750 tx=0x7f8f2400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.441+0000 7f8f2ea00700 1 -- 172.21.15.89:0/375225805 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f24005d70 con 0x7f8f38160f10 2024-04-02T07:54:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.441+0000 7f8f2ea00700 1 -- 172.21.15.89:0/375225805 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f240070d0 con 0x7f8f38160f10 2024-04-02T07:54:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.441+0000 7f8f2ea00700 1 -- 172.21.15.89:0/375225805 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f240074a0 con 0x7f8f38160f10 2024-04-02T07:54:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.441+0000 7f8f3d800700 1 -- 172.21.15.89:0/375225805 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f381e7fe0 con 0x7f8f38160f10 2024-04-02T07:54:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.441+0000 7f8f3d800700 1 -- 172.21.15.89:0/375225805 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f381e85a0 con 0x7f8f38160f10 2024-04-02T07:54:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.442+0000 7f8f3d800700 1 -- 172.21.15.89:0/375225805 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f380cf690 con 0x7f8f38160f10 2024-04-02T07:54:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.443+0000 7f8f2ea00700 1 -- 172.21.15.89:0/375225805 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8f24007600 con 0x7f8f38160f10 2024-04-02T07:54:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.444+0000 7f8f2ea00700 1 --2- 172.21.15.89:0/375225805 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8f1806adb0 0x7f8f1806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.444+0000 7f8f2ea00700 1 -- 172.21.15.89:0/375225805 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f8f240887b0 con 0x7f8f38160f10 2024-04-02T07:54:27.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.444+0000 7f8f36a00700 1 --2- 172.21.15.89:0/375225805 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8f1806adb0 0x7f8f1806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.444+0000 7f8f36a00700 1 --2- 172.21.15.89:0/375225805 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8f1806adb0 0x7f8f1806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8f2800a010 tx=0x7f8f28008000 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:27.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.448+0000 7f8f2ea00700 1 -- 172.21.15.89:0/375225805 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8f380cf690 con 0x7f8f38160f10 2024-04-02T07:54:27.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.652+0000 7f8f3d800700 1 -- 172.21.15.89:0/375225805 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8f380cf690 con 0x7f8f38160f10 2024-04-02T07:54:27.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.653+0000 7f8f2ea00700 1 -- 172.21.15.89:0/375225805 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v121) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8f380cf690 con 0x7f8f38160f10 2024-04-02T07:54:27.653 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:27.654 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:54:27.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:27.654 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T07:54:27.654 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T07:54:27.654 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:27.654 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:54:27.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.655+0000 7f8f3d800700 1 -- 172.21.15.89:0/375225805 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8f1806adb0 msgr2=0x7f8f1806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.655+0000 7f8f3d800700 1 --2- 172.21.15.89:0/375225805 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8f1806adb0 0x7f8f1806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8f2800a010 tx=0x7f8f28008000 comp rx=0 tx=0).stop 2024-04-02T07:54:27.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.655+0000 7f8f3d800700 1 -- 172.21.15.89:0/375225805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f38160f10 msgr2=0x7f8f3815f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.655+0000 7f8f3d800700 1 --2- 172.21.15.89:0/375225805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f38160f10 0x7f8f3815f6e0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f8f2400c750 tx=0x7f8f2400cb10 comp rx=0 tx=0).stop 2024-04-02T07:54:27.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.655+0000 7f8f3d800700 1 -- 172.21.15.89:0/375225805 shutdown_connections 2024-04-02T07:54:27.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.655+0000 7f8f3d800700 1 --2- 172.21.15.89:0/375225805 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8f1806adb0 0x7f8f1806d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.655+0000 7f8f3d800700 1 --2- 172.21.15.89:0/375225805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f38161590 0x7f8f3815fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.655+0000 7f8f3d800700 1 --2- 172.21.15.89:0/375225805 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f380cf230 0x7f8f380c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.655+0000 7f8f3d800700 1 --2- 172.21.15.89:0/375225805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f38160f10 0x7f8f3815f6e0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.655+0000 7f8f3d800700 1 -- 172.21.15.89:0/375225805 >> 172.21.15.89:0/375225805 conn(0x7f8f3808aff0 msgr2=0x7f8f3815c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:27.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.655+0000 7f8f3d800700 1 -- 172.21.15.89:0/375225805 shutdown_connections 2024-04-02T07:54:27.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.655+0000 7f8f3d800700 1 -- 172.21.15.89:0/375225805 wait complete. 2024-04-02T07:54:27.674 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T07:54:27.674 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T07:54:27.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.869+0000 7f63cbc00700 1 Processor -- start 2024-04-02T07:54:27.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.870+0000 7f63cbc00700 1 -- start start 2024-04-02T07:54:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63cbc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63c4155ac0 0x7f63c415c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63cbc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f63c406c910 0x7f63c40adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63cbc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f63c40ae4a0 0x7f63c415f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63cbc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c406dd00 con 0x7f63c406c910 2024-04-02T07:54:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63cbc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c406d0f0 con 0x7f63c40ae4a0 2024-04-02T07:54:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63cbc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c406d3d0 con 0x7f63c4155ac0 2024-04-02T07:54:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63c3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f63c406c910 0x7f63c40adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63c3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f63c406c910 0x7f63c40adf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43910/0 (socket says 172.21.15.89:43910) 2024-04-02T07:54:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63c3e00700 1 -- 172.21.15.89:0/3286785 learned_addr learned my addr 172.21.15.89:0/3286785 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63c9200700 1 --2- 172.21.15.89:0/3286785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63c4155ac0 0x7f63c415c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63c9c00700 1 --2- 172.21.15.89:0/3286785 >> v2:172.21.15.113:3300/0 conn(0x7f63c40ae4a0 0x7f63c415f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63c3e00700 1 -- 172.21.15.89:0/3286785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63c4155ac0 msgr2=0x7f63c415c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:27.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63c3e00700 1 --2- 172.21.15.89:0/3286785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63c4155ac0 0x7f63c415c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63c3e00700 1 -- 172.21.15.89:0/3286785 >> v2:172.21.15.113:3300/0 conn(0x7f63c40ae4a0 msgr2=0x7f63c415f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:27.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63c3e00700 1 --2- 172.21.15.89:0/3286785 >> v2:172.21.15.113:3300/0 conn(0x7f63c40ae4a0 0x7f63c415f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63c3e00700 1 -- 172.21.15.89:0/3286785 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c406d6b0 con 0x7f63c406c910 2024-04-02T07:54:27.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.871+0000 7f63c9c00700 1 --2- 172.21.15.89:0/3286785 >> v2:172.21.15.113:3300/0 conn(0x7f63c40ae4a0 0x7f63c415f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:27.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.872+0000 7f63c3e00700 1 --2- 172.21.15.89:0/3286785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c406c910 0x7f63c40adf60 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f63c415cc40 tx=0x7f63b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be626135e0318c16 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:27.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.872+0000 7f63c3400700 1 -- 172.21.15.89:0/3286785 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b000b870 con 0x7f63c406c910 2024-04-02T07:54:27.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.872+0000 7f63c3400700 1 -- 172.21.15.89:0/3286785 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b000b9d0 con 0x7f63c406c910 2024-04-02T07:54:27.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.872+0000 7f63c3400700 1 -- 172.21.15.89:0/3286785 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b000bd60 con 0x7f63c406c910 2024-04-02T07:54:27.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.872+0000 7f63cbc00700 1 -- 172.21.15.89:0/3286785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c406c910 msgr2=0x7f63c40adf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.872+0000 7f63cbc00700 1 --2- 172.21.15.89:0/3286785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c406c910 0x7f63c40adf60 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f63c415cc40 tx=0x7f63b0009bd0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.873+0000 7f63cbc00700 1 -- 172.21.15.89:0/3286785 shutdown_connections 2024-04-02T07:54:27.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.873+0000 7f63cbc00700 1 --2- 172.21.15.89:0/3286785 >> v2:172.21.15.113:3300/0 conn(0x7f63c40ae4a0 0x7f63c415f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.873+0000 7f63cbc00700 1 --2- 172.21.15.89:0/3286785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63c4155ac0 0x7f63c415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.873+0000 7f63cbc00700 1 --2- 172.21.15.89:0/3286785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c406c910 0x7f63c40adf60 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.873+0000 7f63cbc00700 1 -- 172.21.15.89:0/3286785 >> 172.21.15.89:0/3286785 conn(0x7f63c40cf210 msgr2=0x7f63c40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:27.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.873+0000 7f63cbc00700 1 -- 172.21.15.89:0/3286785 shutdown_connections 2024-04-02T07:54:27.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.873+0000 7f63cbc00700 1 -- 172.21.15.89:0/3286785 wait complete. 2024-04-02T07:54:27.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.874+0000 7f63cbc00700 1 Processor -- start 2024-04-02T07:54:27.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.875+0000 7f63cbc00700 1 -- start start 2024-04-02T07:54:27.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.875+0000 7f63cbc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c406c910 0x7f63c4144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.875+0000 7f63cbc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63c40ae4a0 0x7f63c4072500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.875+0000 7f63cbc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c4155ac0 0x7f63c415a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.875+0000 7f63cbc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4071a10 con 0x7f63c406c910 2024-04-02T07:54:27.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.875+0000 7f63cbc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4070e00 con 0x7f63c4155ac0 2024-04-02T07:54:27.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.875+0000 7f63cbc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40710e0 con 0x7f63c40ae4a0 2024-04-02T07:54:27.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.876+0000 7f63c9200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c406c910 0x7f63c4144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.876+0000 7f63c9200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c406c910 0x7f63c4144570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43924/0 (socket says 172.21.15.89:43924) 2024-04-02T07:54:27.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.876+0000 7f63c9200700 1 -- 172.21.15.89:0/3185374277 learned_addr learned my addr 172.21.15.89:0/3185374277 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:27.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.876+0000 7f63c9c00700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c4155ac0 0x7f63c415a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.876+0000 7f63c3e00700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63c40ae4a0 0x7f63c4072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.876+0000 7f63c9200700 1 -- 172.21.15.89:0/3185374277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63c40ae4a0 msgr2=0x7f63c4072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.876+0000 7f63c9200700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63c40ae4a0 0x7f63c4072500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.876+0000 7f63c9200700 1 -- 172.21.15.89:0/3185374277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c4155ac0 msgr2=0x7f63c415a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:27.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.876+0000 7f63c9200700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c4155ac0 0x7f63c415a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:27.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.876+0000 7f63c9200700 1 -- 172.21.15.89:0/3185374277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b0009550 con 0x7f63c406c910 2024-04-02T07:54:27.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.876+0000 7f63c9c00700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c4155ac0 0x7f63c415a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:27.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.876+0000 7f63c9200700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c406c910 0x7f63c4144570 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f63b800d800 tx=0x7f63b800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:27.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.876+0000 7f63c0c00700 1 -- 172.21.15.89:0/3185374277 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b80125f0 con 0x7f63c406c910 2024-04-02T07:54:27.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.877+0000 7f63c0c00700 1 -- 172.21.15.89:0/3185374277 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b8012750 con 0x7f63c406c910 2024-04-02T07:54:27.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.877+0000 7f63cbc00700 1 -- 172.21.15.89:0/3185374277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c40713c0 con 0x7f63c406c910 2024-04-02T07:54:27.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.877+0000 7f63c0c00700 1 -- 172.21.15.89:0/3185374277 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8012b00 con 0x7f63c406c910 2024-04-02T07:54:27.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.877+0000 7f63cbc00700 1 -- 172.21.15.89:0/3185374277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c4144ab0 con 0x7f63c406c910 2024-04-02T07:54:27.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.878+0000 7f63cbc00700 1 -- 172.21.15.89:0/3185374277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c4067720 con 0x7f63c406c910 2024-04-02T07:54:27.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.879+0000 7f63c0c00700 1 -- 172.21.15.89:0/3185374277 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f63b8012c60 con 0x7f63c406c910 2024-04-02T07:54:27.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.880+0000 7f63c0c00700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f63a406adb0 0x7f63a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:27.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.880+0000 7f63c0c00700 1 -- 172.21.15.89:0/3185374277 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f63b8089630 con 0x7f63c406c910 2024-04-02T07:54:27.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.880+0000 7f63c3e00700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f63a406adb0 0x7f63a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:27.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.880+0000 7f63c3e00700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f63a406adb0 0x7f63a406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f63c4152c20 tx=0x7f63b0008030 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:27.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:27.884+0000 7f63c0c00700 1 -- 172.21.15.89:0/3185374277 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f63b80548f0 con 0x7f63c406c910 2024-04-02T07:54:28.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:28.089+0000 7f63cbc00700 1 -- 172.21.15.89:0/3185374277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f63c406e950 con 0x7f63c406c910 2024-04-02T07:54:29.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.028+0000 7f63c0c00700 1 -- 172.21.15.89:0/3185374277 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v122) v1 ==== 138+0+0 (secure 0 0 0) 0x7f63b8089070 con 0x7f63c406c910 2024-04-02T07:54:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.032+0000 7f63cbc00700 1 -- 172.21.15.89:0/3185374277 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f63a406adb0 msgr2=0x7f63a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.032+0000 7f63cbc00700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f63a406adb0 0x7f63a406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f63c4152c20 tx=0x7f63b0008030 comp rx=0 tx=0).stop 2024-04-02T07:54:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.032+0000 7f63cbc00700 1 -- 172.21.15.89:0/3185374277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c406c910 msgr2=0x7f63c4144570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.032+0000 7f63cbc00700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c406c910 0x7f63c4144570 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f63b800d800 tx=0x7f63b800dbc0 comp rx=0 tx=0).stop 2024-04-02T07:54:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.033+0000 7f63cbc00700 1 -- 172.21.15.89:0/3185374277 shutdown_connections 2024-04-02T07:54:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.033+0000 7f63cbc00700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f63a406adb0 0x7f63a406d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.033+0000 7f63cbc00700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c4155ac0 0x7f63c415a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.033+0000 7f63cbc00700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63c406c910 0x7f63c4144570 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.033+0000 7f63cbc00700 1 --2- 172.21.15.89:0/3185374277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63c40ae4a0 0x7f63c4072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.033+0000 7f63cbc00700 1 -- 172.21.15.89:0/3185374277 >> 172.21.15.89:0/3185374277 conn(0x7f63c40cf210 msgr2=0x7f63c415ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.033+0000 7f63cbc00700 1 -- 172.21.15.89:0/3185374277 shutdown_connections 2024-04-02T07:54:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.033+0000 7f63cbc00700 1 -- 172.21.15.89:0/3185374277 wait complete. 2024-04-02T07:54:29.036 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T07:54:29.057 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T07:54:29.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.258+0000 7f0853000700 1 Processor -- start 2024-04-02T07:54:29.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.258+0000 7f0853000700 1 -- start start 2024-04-02T07:54:29.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.259+0000 7f0853000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f084c0d21f0 0x7f084c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:29.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.259+0000 7f0853000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f084c0cec40 0x7f084c161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:29.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.259+0000 7f0853000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084c0ce270 con 0x7f084c1611b0 2024-04-02T07:54:29.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.259+0000 7f0853000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084c15b600 con 0x7f084c0d21f0 2024-04-02T07:54:29.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.259+0000 7f0853000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084c0d2760 con 0x7f084c0cec40 2024-04-02T07:54:29.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.259+0000 7f084be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f084c1611b0 0x7f084c15ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36664/0 (socket says 172.21.15.89:36664) 2024-04-02T07:54:29.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.259+0000 7f084be00700 1 -- 172.21.15.89:0/2622464783 learned_addr learned my addr 172.21.15.89:0/2622464783 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.259+0000 7f0851000700 1 --2- 172.21.15.89:0/2622464783 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f084c0cec40 0x7f084c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.259+0000 7f084b400700 1 --2- 172.21.15.89:0/2622464783 >> v2:172.21.15.113:3300/0 conn(0x7f084c0d21f0 0x7f084c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.260+0000 7f084aa00700 1 -- 172.21.15.89:0/2622464783 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458668269 0 0) 0x7f084c0ce270 con 0x7f084c1611b0 2024-04-02T07:54:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.260+0000 7f084aa00700 1 -- 172.21.15.89:0/2622464783 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08300035b0 con 0x7f084c1611b0 2024-04-02T07:54:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.260+0000 7f084aa00700 1 -- 172.21.15.89:0/2622464783 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2298588714 0 0) 0x7f08300035b0 con 0x7f084c1611b0 2024-04-02T07:54:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.260+0000 7f084b400700 1 -- 172.21.15.89:0/2622464783 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f084c0cec40 msgr2=0x7f084c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.260+0000 7f084b400700 1 --2- 172.21.15.89:0/2622464783 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f084c0cec40 0x7f084c161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:29.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.260+0000 7f084b400700 1 -- 172.21.15.89:0/2622464783 >> v1:172.21.15.89:6789/0 conn(0x7f084c1611b0 legacy=0x7f084c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.260+0000 7f084b400700 1 -- 172.21.15.89:0/2622464783 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c0d2a40 con 0x7f084c0d21f0 2024-04-02T07:54:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.260+0000 7f0851000700 1 --2- 172.21.15.89:0/2622464783 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f084c0cec40 0x7f084c161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.260+0000 7f084b400700 1 --2- 172.21.15.89:0/2622464783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f084c0d21f0 0x7f084c0ce700 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0838009840 tx=0x7f0838009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65a3d01034d6d75 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.261+0000 7f084aa00700 1 -- 172.21.15.89:0/2622464783 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c001670 con 0x7f084c0d21f0 2024-04-02T07:54:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.261+0000 7f084aa00700 1 -- 172.21.15.89:0/2622464783 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0838004030 con 0x7f084c0d21f0 2024-04-02T07:54:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.261+0000 7f084aa00700 1 -- 172.21.15.89:0/2622464783 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08380043a0 con 0x7f084c0d21f0 2024-04-02T07:54:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.261+0000 7f0853000700 1 -- 172.21.15.89:0/2622464783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f084c0d21f0 msgr2=0x7f084c0ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.261+0000 7f0853000700 1 --2- 172.21.15.89:0/2622464783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f084c0d21f0 0x7f084c0ce700 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0838009840 tx=0x7f0838009b50 comp rx=0 tx=0).stop 2024-04-02T07:54:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.261+0000 7f0853000700 1 -- 172.21.15.89:0/2622464783 shutdown_connections 2024-04-02T07:54:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.261+0000 7f0853000700 1 --2- 172.21.15.89:0/2622464783 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f084c0cec40 0x7f084c161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.261+0000 7f0853000700 1 --2- 172.21.15.89:0/2622464783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f084c0d21f0 0x7f084c0ce700 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.261+0000 7f0853000700 1 -- 172.21.15.89:0/2622464783 >> 172.21.15.89:0/2622464783 conn(0x7f084c08aff0 msgr2=0x7f084c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.261+0000 7f0853000700 1 -- 172.21.15.89:0/2622464783 shutdown_connections 2024-04-02T07:54:29.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.261+0000 7f0853000700 1 -- 172.21.15.89:0/2622464783 wait complete. 2024-04-02T07:54:29.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.263+0000 7f0853000700 1 Processor -- start 2024-04-02T07:54:29.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.263+0000 7f0853000700 1 -- start start 2024-04-02T07:54:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.264+0000 7f0853000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f084c0cec40 0x7f084c158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.264+0000 7f0853000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f084c0d21f0 0x7f084c0b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.264+0000 7f0853000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f084c1611b0 0x7f084c06be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.264+0000 7f0853000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c09e540 con 0x7f084c1611b0 2024-04-02T07:54:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.264+0000 7f0853000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c0b6ca0 con 0x7f084c0d21f0 2024-04-02T07:54:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.264+0000 7f0853000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084c0b6f80 con 0x7f084c0cec40 2024-04-02T07:54:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.264+0000 7f0851000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f084c1611b0 0x7f084c06be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.264+0000 7f0851000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f084c1611b0 0x7f084c06be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35916/0 (socket says 172.21.15.89:35916) 2024-04-02T07:54:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.264+0000 7f0851000700 1 -- 172.21.15.89:0/2226998386 learned_addr learned my addr 172.21.15.89:0/2226998386 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:29.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.264+0000 7f084be00700 1 --2- 172.21.15.89:0/2226998386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f084c0cec40 0x7f084c158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:29.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.264+0000 7f084b400700 1 --2- 172.21.15.89:0/2226998386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f084c0d21f0 0x7f084c0b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:29.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.265+0000 7f0851000700 1 -- 172.21.15.89:0/2226998386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f084c0cec40 msgr2=0x7f084c158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:29.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.265+0000 7f0851000700 1 --2- 172.21.15.89:0/2226998386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f084c0cec40 0x7f084c158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:29.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.265+0000 7f0851000700 1 -- 172.21.15.89:0/2226998386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f084c0d21f0 msgr2=0x7f084c0b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:29.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.265+0000 7f0851000700 1 --2- 172.21.15.89:0/2226998386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f084c0d21f0 0x7f084c0b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:29.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.265+0000 7f0851000700 1 -- 172.21.15.89:0/2226998386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0844009a70 con 0x7f084c1611b0 2024-04-02T07:54:29.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.265+0000 7f0851000700 1 --2- 172.21.15.89:0/2226998386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f084c1611b0 0x7f084c06be60 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f084400efc0 tx=0x7f084400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:29.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.265+0000 7f0843e00700 1 -- 172.21.15.89:0/2226998386 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084400ca90 con 0x7f084c1611b0 2024-04-02T07:54:29.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.265+0000 7f0843e00700 1 -- 172.21.15.89:0/2226998386 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084400f040 con 0x7f084c1611b0 2024-04-02T07:54:29.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.265+0000 7f0843e00700 1 -- 172.21.15.89:0/2226998386 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084400ce10 con 0x7f084c1611b0 2024-04-02T07:54:29.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.265+0000 7f0853000700 1 -- 172.21.15.89:0/2226998386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08380094a0 con 0x7f084c1611b0 2024-04-02T07:54:29.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.266+0000 7f0853000700 1 -- 172.21.15.89:0/2226998386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084c0b7540 con 0x7f084c1611b0 2024-04-02T07:54:29.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.266+0000 7f0853000700 1 -- 172.21.15.89:0/2226998386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084c09e780 con 0x7f084c1611b0 2024-04-02T07:54:29.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.268+0000 7f0843e00700 1 -- 172.21.15.89:0/2226998386 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0844003a20 con 0x7f084c1611b0 2024-04-02T07:54:29.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.269+0000 7f0843e00700 1 --2- 172.21.15.89:0/2226998386 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f082c06adb0 0x7f082c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:29.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.269+0000 7f0843e00700 1 -- 172.21.15.89:0/2226998386 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f084408a5a0 con 0x7f084c1611b0 2024-04-02T07:54:29.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.269+0000 7f084be00700 1 --2- 172.21.15.89:0/2226998386 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f082c06adb0 0x7f082c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:29.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.270+0000 7f084be00700 1 --2- 172.21.15.89:0/2226998386 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f082c06adb0 0x7f082c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f083c0011e0 tx=0x7f083c001da0 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:29.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.273+0000 7f0843e00700 1 -- 172.21.15.89:0/2226998386 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0844055650 con 0x7f084c1611b0 2024-04-02T07:54:29.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:29.474+0000 7f0853000700 1 -- 172.21.15.89:0/2226998386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f084c067720 con 0x7f084c1611b0 2024-04-02T07:54:30.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.034+0000 7f0843e00700 1 -- 172.21.15.89:0/2226998386 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v123) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0844009d70 con 0x7f084c1611b0 2024-04-02T07:54:30.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.043+0000 7f0853000700 1 -- 172.21.15.89:0/2226998386 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f082c06adb0 msgr2=0x7f082c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.043+0000 7f0853000700 1 --2- 172.21.15.89:0/2226998386 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f082c06adb0 0x7f082c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f083c0011e0 tx=0x7f083c001da0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.043+0000 7f0853000700 1 -- 172.21.15.89:0/2226998386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f084c1611b0 msgr2=0x7f084c06be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.043+0000 7f0853000700 1 --2- 172.21.15.89:0/2226998386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f084c1611b0 0x7f084c06be60 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f084400efc0 tx=0x7f084400c3a0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.044+0000 7f0853000700 1 -- 172.21.15.89:0/2226998386 shutdown_connections 2024-04-02T07:54:30.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.044+0000 7f0853000700 1 --2- 172.21.15.89:0/2226998386 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f082c06adb0 0x7f082c06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.044+0000 7f0853000700 1 --2- 172.21.15.89:0/2226998386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f084c1611b0 0x7f084c06be60 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.044+0000 7f0853000700 1 --2- 172.21.15.89:0/2226998386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f084c0cec40 0x7f084c158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.044+0000 7f0853000700 1 --2- 172.21.15.89:0/2226998386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f084c0d21f0 0x7f084c0b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.044+0000 7f0853000700 1 -- 172.21.15.89:0/2226998386 >> 172.21.15.89:0/2226998386 conn(0x7f084c08aff0 msgr2=0x7f084c161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:30.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.045+0000 7f0853000700 1 -- 172.21.15.89:0/2226998386 shutdown_connections 2024-04-02T07:54:30.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.046+0000 7f0853000700 1 -- 172.21.15.89:0/2226998386 wait complete. 2024-04-02T07:54:30.050 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T07:54:30.070 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T07:54:30.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.262+0000 7f6c0bc00700 1 Processor -- start 2024-04-02T07:54:30.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.263+0000 7f6c0bc00700 1 -- start start 2024-04-02T07:54:30.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.264+0000 7f6c0bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c040d16a0 0x7f6c040d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.264+0000 7f6c0bc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c04161420 con 0x7f6c0415d9d0 2024-04-02T07:54:30.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.264+0000 7f6c0bc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c04161700 con 0x7f6c040d2370 2024-04-02T07:54:30.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.264+0000 7f6c0bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040d2820 con 0x7f6c040d16a0 2024-04-02T07:54:30.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.264+0000 7f6c09200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6c0415d9d0 0x7f6c040d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36672/0 (socket says 172.21.15.89:36672) 2024-04-02T07:54:30.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.264+0000 7f6c09200700 1 -- 172.21.15.89:0/54455200 learned_addr learned my addr 172.21.15.89:0/54455200 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:30.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03e00700 1 --2- 172.21.15.89:0/54455200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c040d16a0 0x7f6c040d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 -- 172.21.15.89:0/54455200 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816086926 0 0) 0x7f6c04161420 con 0x7f6c0415d9d0 2024-04-02T07:54:30.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 -- 172.21.15.89:0/54455200 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bec0035b0 con 0x7f6c0415d9d0 2024-04-02T07:54:30.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 -- 172.21.15.89:0/54455200 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 864183176 0 0) 0x7f6bec0035b0 con 0x7f6c0415d9d0 2024-04-02T07:54:30.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 -- 172.21.15.89:0/54455200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c040d16a0 msgr2=0x7f6c040d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:30.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 --2- 172.21.15.89:0/54455200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c040d16a0 0x7f6c040d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 -- 172.21.15.89:0/54455200 >> v1:172.21.15.113:6789/0 conn(0x7f6c040d2370 legacy=0x7f6c040ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 -- 172.21.15.89:0/54455200 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c041619e0 con 0x7f6c0415d9d0 2024-04-02T07:54:30.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 -- 172.21.15.89:0/54455200 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f6bf8001670 con 0x7f6c0415d9d0 2024-04-02T07:54:30.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 -- 172.21.15.89:0/54455200 >> v1:172.21.15.89:6789/0 conn(0x7f6c0415d9d0 legacy=0x7f6c040d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 --2- 172.21.15.89:0/54455200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec003a30 0x7f6bec003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 --2- 172.21.15.89:0/54455200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bec004850 0x7f6bec004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 --2- 172.21.15.89:0/54455200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c040d16a0 0x7f6bec005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 -- 172.21.15.89:0/54455200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040d2820 con 0x7f6bec003a30 2024-04-02T07:54:30.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 -- 172.21.15.89:0/54455200 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec00bac0 con 0x7f6bec004850 2024-04-02T07:54:30.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.265+0000 7f6c03400700 1 -- 172.21.15.89:0/54455200 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec00bc80 con 0x7f6c040d16a0 2024-04-02T07:54:30.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.266+0000 7f6c09200700 1 --2- 172.21.15.89:0/54455200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec003a30 0x7f6bec003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.266+0000 7f6c03e00700 1 --2- 172.21.15.89:0/54455200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bec004850 0x7f6bec004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.266+0000 7f6c09c00700 1 --2- 172.21.15.89:0/54455200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c040d16a0 0x7f6bec005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.266+0000 7f6c03e00700 1 -- 172.21.15.89:0/54455200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c040d16a0 msgr2=0x7f6bec005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:30.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.266+0000 7f6c03e00700 1 --2- 172.21.15.89:0/54455200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c040d16a0 0x7f6bec005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.266+0000 7f6c03e00700 1 -- 172.21.15.89:0/54455200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec003a30 msgr2=0x7f6bec003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:30.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.266+0000 7f6c03e00700 1 --2- 172.21.15.89:0/54455200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec003a30 0x7f6bec003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.266+0000 7f6c03e00700 1 -- 172.21.15.89:0/54455200 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bec0035b0 con 0x7f6bec004850 2024-04-02T07:54:30.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.266+0000 7f6c09200700 1 --2- 172.21.15.89:0/54455200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bec003a30 0x7f6bec003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:30.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.266+0000 7f6c03e00700 1 --2- 172.21.15.89:0/54455200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bec004850 0x7f6bec004cd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6c040d1ae0 tx=0x7f6bec005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aba1da126f633e17 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:30.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.267+0000 7f6c03400700 1 -- 172.21.15.89:0/54455200 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf0005a20 con 0x7f6bec004850 2024-04-02T07:54:30.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.267+0000 7f6c03400700 1 -- 172.21.15.89:0/54455200 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf0005b80 con 0x7f6bec004850 2024-04-02T07:54:30.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.267+0000 7f6c03400700 1 -- 172.21.15.89:0/54455200 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf000e6c0 con 0x7f6bec004850 2024-04-02T07:54:30.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.267+0000 7f6c0bc00700 1 -- 172.21.15.89:0/54455200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bec004850 msgr2=0x7f6bec004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.267+0000 7f6c0bc00700 1 --2- 172.21.15.89:0/54455200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bec004850 0x7f6bec004cd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6c040d1ae0 tx=0x7f6bec005880 comp rx=0 tx=0).stop 2024-04-02T07:54:30.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.267+0000 7f6c09c00700 1 -- 172.21.15.89:0/54455200 reap_dead start 2024-04-02T07:54:30.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.267+0000 7f6c0bc00700 1 -- 172.21.15.89:0/54455200 shutdown_connections 2024-04-02T07:54:30.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.267+0000 7f6c0bc00700 1 -- 172.21.15.89:0/54455200 >> 172.21.15.89:0/54455200 conn(0x7f6c0408aff0 msgr2=0x7f6c040d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:30.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.267+0000 7f6c0bc00700 1 -- 172.21.15.89:0/54455200 shutdown_connections 2024-04-02T07:54:30.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.267+0000 7f6c0bc00700 1 -- 172.21.15.89:0/54455200 wait complete. 2024-04-02T07:54:30.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.269+0000 7f6c0bc00700 1 Processor -- start 2024-04-02T07:54:30.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.269+0000 7f6c0bc00700 1 -- start start 2024-04-02T07:54:30.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.270+0000 7f6c0bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec004850 0x7f6c040cc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.270+0000 7f6c0bc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c041505d0 0x7f6c0414a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.270+0000 7f6c0bc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c04150a50 0x7f6c0414a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.270+0000 7f6c0bc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040cc1d0 con 0x7f6c04150a50 2024-04-02T07:54:30.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.270+0000 7f6c0bc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040ccba0 con 0x7f6c041505d0 2024-04-02T07:54:30.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.270+0000 7f6c0bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040cb5c0 con 0x7f6bec004850 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.270+0000 7f6c09c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c04150a50 0x7f6c0414a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.270+0000 7f6c09c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c04150a50 0x7f6c0414a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35938/0 (socket says 172.21.15.89:35938) 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.270+0000 7f6c09c00700 1 -- 172.21.15.89:0/2164633096 learned_addr learned my addr 172.21.15.89:0/2164633096 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.270+0000 7f6c03e00700 1 --2- 172.21.15.89:0/2164633096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c041505d0 0x7f6c0414a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.270+0000 7f6c09200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec004850 0x7f6c040cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.271+0000 7f6c09c00700 1 -- 172.21.15.89:0/2164633096 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec004850 msgr2=0x7f6c040cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.271+0000 7f6c09c00700 1 --2- 172.21.15.89:0/2164633096 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec004850 0x7f6c040cc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.271+0000 7f6c09c00700 1 -- 172.21.15.89:0/2164633096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c041505d0 msgr2=0x7f6c0414a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.271+0000 7f6c09c00700 1 --2- 172.21.15.89:0/2164633096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c041505d0 0x7f6c0414a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.271+0000 7f6c09c00700 1 -- 172.21.15.89:0/2164633096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c040cb8a0 con 0x7f6c04150a50 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.271+0000 7f6c09c00700 1 --2- 172.21.15.89:0/2164633096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c04150a50 0x7f6c0414a580 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f6c0406b5b0 tx=0x7f6bfc00d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.271+0000 7f6c00c00700 1 -- 172.21.15.89:0/2164633096 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc003c10 con 0x7f6c04150a50 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.271+0000 7f6c00c00700 1 -- 172.21.15.89:0/2164633096 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bfc003d90 con 0x7f6c04150a50 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.271+0000 7f6c00c00700 1 -- 172.21.15.89:0/2164633096 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc00d960 con 0x7f6c04150a50 2024-04-02T07:54:30.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.271+0000 7f6c0bc00700 1 -- 172.21.15.89:0/2164633096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c040cbb80 con 0x7f6c04150a50 2024-04-02T07:54:30.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.271+0000 7f6c0bc00700 1 -- 172.21.15.89:0/2164633096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0414aac0 con 0x7f6c04150a50 2024-04-02T07:54:30.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.272+0000 7f6c0bc00700 1 -- 172.21.15.89:0/2164633096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c040cc340 con 0x7f6c04150a50 2024-04-02T07:54:30.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.274+0000 7f6c00c00700 1 -- 172.21.15.89:0/2164633096 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6bfc00dac0 con 0x7f6c04150a50 2024-04-02T07:54:30.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.274+0000 7f6c00c00700 1 --2- 172.21.15.89:0/2164633096 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6be406adb0 0x7f6be406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.275+0000 7f6c00c00700 1 -- 172.21.15.89:0/2164633096 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f6bfc089ff0 con 0x7f6c04150a50 2024-04-02T07:54:30.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.275+0000 7f6c09200700 1 --2- 172.21.15.89:0/2164633096 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6be406adb0 0x7f6be406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.275+0000 7f6c09200700 1 --2- 172.21.15.89:0/2164633096 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6be406adb0 0x7f6be406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8001590 tx=0x7f6bf80011c0 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:30.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.279+0000 7f6c00c00700 1 -- 172.21.15.89:0/2164633096 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6bfc055010 con 0x7f6c04150a50 2024-04-02T07:54:30.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.521+0000 7f6c0bc00700 1 -- 172.21.15.89:0/2164633096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6c04067720 con 0x7f6c04150a50 2024-04-02T07:54:30.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.528+0000 7f6c00c00700 1 -- 172.21.15.89:0/2164633096 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v53) v1 ==== 174+0+0 (secure 0 0 0) 0x7f6bfc058630 con 0x7f6c04150a50 2024-04-02T07:54:30.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.532+0000 7f6c0bc00700 1 -- 172.21.15.89:0/2164633096 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6be406adb0 msgr2=0x7f6be406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.532+0000 7f6c0bc00700 1 --2- 172.21.15.89:0/2164633096 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6be406adb0 0x7f6be406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8001590 tx=0x7f6bf80011c0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.532+0000 7f6c0bc00700 1 -- 172.21.15.89:0/2164633096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c04150a50 msgr2=0x7f6c0414a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.532+0000 7f6c0bc00700 1 --2- 172.21.15.89:0/2164633096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c04150a50 0x7f6c0414a580 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f6c0406b5b0 tx=0x7f6bfc00d150 comp rx=0 tx=0).stop 2024-04-02T07:54:30.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.533+0000 7f6c0bc00700 1 -- 172.21.15.89:0/2164633096 shutdown_connections 2024-04-02T07:54:30.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.533+0000 7f6c0bc00700 1 --2- 172.21.15.89:0/2164633096 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6be406adb0 0x7f6be406d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.533+0000 7f6c0bc00700 1 --2- 172.21.15.89:0/2164633096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c04150a50 0x7f6c0414a580 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.533+0000 7f6c0bc00700 1 --2- 172.21.15.89:0/2164633096 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bec004850 0x7f6c040cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.533+0000 7f6c0bc00700 1 --2- 172.21.15.89:0/2164633096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c041505d0 0x7f6c0414a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.533+0000 7f6c0bc00700 1 -- 172.21.15.89:0/2164633096 >> 172.21.15.89:0/2164633096 conn(0x7f6c0408aff0 msgr2=0x7f6c040ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:30.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.534+0000 7f6c0bc00700 1 -- 172.21.15.89:0/2164633096 shutdown_connections 2024-04-02T07:54:30.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.535+0000 7f6c0bc00700 1 -- 172.21.15.89:0/2164633096 wait complete. 2024-04-02T07:54:30.541 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 14 and data pool 15 2024-04-02T07:54:30.567 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:54:30.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.758+0000 7fa365c00700 1 Processor -- start 2024-04-02T07:54:30.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.758+0000 7fa365c00700 1 -- start start 2024-04-02T07:54:30.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.759+0000 7fa365c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa360160f10 0x7fa3600d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.759+0000 7fa365c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa360161590 0x7fa360161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.759+0000 7fa365c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3600ceda0 con 0x7fa3600cf230 2024-04-02T07:54:30.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.759+0000 7fa365c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa360161f80 con 0x7fa360160f10 2024-04-02T07:54:30.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.759+0000 7fa365c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3601612f0 con 0x7fa360161590 2024-04-02T07:54:30.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.759+0000 7fa35e000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa3600cf230 0x7fa3600d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36680/0 (socket says 172.21.15.89:36680) 2024-04-02T07:54:30.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35e000700 1 -- 172.21.15.89:0/4047927768 learned_addr learned my addr 172.21.15.89:0/4047927768 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:30.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35f400700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa360161590 0x7fa360161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35ea00700 1 --2- 172.21.15.89:0/4047927768 >> v2:172.21.15.113:3300/0 conn(0x7fa360160f10 0x7fa3600d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 -- 172.21.15.89:0/4047927768 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425153681 0 0) 0x7fa3600ceda0 con 0x7fa3600cf230 2024-04-02T07:54:30.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 -- 172.21.15.89:0/4047927768 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3440035b0 con 0x7fa3600cf230 2024-04-02T07:54:30.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 -- 172.21.15.89:0/4047927768 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2123708268 0 0) 0x7fa3440035b0 con 0x7fa3600cf230 2024-04-02T07:54:30.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 -- 172.21.15.89:0/4047927768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa360161590 msgr2=0x7fa360161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:30.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa360161590 0x7fa360161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 -- 172.21.15.89:0/4047927768 >> v2:172.21.15.113:3300/0 conn(0x7fa360160f10 msgr2=0x7fa3600d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:30.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 --2- 172.21.15.89:0/4047927768 >> v2:172.21.15.113:3300/0 conn(0x7fa360160f10 0x7fa3600d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 -- 172.21.15.89:0/4047927768 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3600ce470 con 0x7fa3600cf230 2024-04-02T07:54:30.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 -- 172.21.15.89:0/4047927768 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fa34c001670 con 0x7fa3600cf230 2024-04-02T07:54:30.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 -- 172.21.15.89:0/4047927768 >> v1:172.21.15.89:6789/0 conn(0x7fa3600cf230 legacy=0x7fa3600d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa344003a00 0x7fa344003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa344004850 0x7fa344004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344005210 0x7fa34400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 -- 172.21.15.89:0/4047927768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa360161f80 con 0x7fa344005210 2024-04-02T07:54:30.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 -- 172.21.15.89:0/4047927768 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3601612f0 con 0x7fa344003a00 2024-04-02T07:54:30.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35d600700 1 -- 172.21.15.89:0/4047927768 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34400bfa0 con 0x7fa344004850 2024-04-02T07:54:30.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35f400700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa360161590 0x7fa360161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:30.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.760+0000 7fa35ea00700 1 --2- 172.21.15.89:0/4047927768 >> v2:172.21.15.113:3300/0 conn(0x7fa360160f10 0x7fa3600d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:30.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.761+0000 7fa35f400700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344005210 0x7fa34400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.761+0000 7fa35e000700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa344004850 0x7fa344004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.761+0000 7fa35ea00700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa344003a00 0x7fa344003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.761+0000 7fa35f400700 1 -- 172.21.15.89:0/4047927768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa344004850 msgr2=0x7fa344004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:30.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.761+0000 7fa35f400700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa344004850 0x7fa344004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.761+0000 7fa35f400700 1 -- 172.21.15.89:0/4047927768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa344003a00 msgr2=0x7fa344003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:30.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.761+0000 7fa35f400700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa344003a00 0x7fa344003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.761+0000 7fa35f400700 1 -- 172.21.15.89:0/4047927768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3440035b0 con 0x7fa344005210 2024-04-02T07:54:30.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.761+0000 7fa35e000700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa344004850 0x7fa344004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:30.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.761+0000 7fa35ea00700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa344003a00 0x7fa344003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:30.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.761+0000 7fa35f400700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344005210 0x7fa34400ba10 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fa35800e220 tx=0x7fa344005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212902ba12409d14 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:30.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.762+0000 7fa35d600700 1 -- 172.21.15.89:0/4047927768 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35800ecc0 con 0x7fa344005210 2024-04-02T07:54:30.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.762+0000 7fa35d600700 1 -- 172.21.15.89:0/4047927768 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35800f040 con 0x7fa344005210 2024-04-02T07:54:30.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.762+0000 7fa35d600700 1 -- 172.21.15.89:0/4047927768 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358003a20 con 0x7fa344005210 2024-04-02T07:54:30.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.762+0000 7fa365c00700 1 -- 172.21.15.89:0/4047927768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344005210 msgr2=0x7fa34400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.762+0000 7fa365c00700 1 --2- 172.21.15.89:0/4047927768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344005210 0x7fa34400ba10 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fa35800e220 tx=0x7fa344005880 comp rx=0 tx=0).stop 2024-04-02T07:54:30.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.762+0000 7fa35f400700 1 -- 172.21.15.89:0/4047927768 reap_dead start 2024-04-02T07:54:30.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.762+0000 7fa365c00700 1 -- 172.21.15.89:0/4047927768 shutdown_connections 2024-04-02T07:54:30.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.762+0000 7fa365c00700 1 -- 172.21.15.89:0/4047927768 >> 172.21.15.89:0/4047927768 conn(0x7fa36008aff0 msgr2=0x7fa3600d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:30.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.762+0000 7fa365c00700 1 -- 172.21.15.89:0/4047927768 shutdown_connections 2024-04-02T07:54:30.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.762+0000 7fa365c00700 1 -- 172.21.15.89:0/4047927768 wait complete. 2024-04-02T07:54:30.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.764+0000 7fa365c00700 1 Processor -- start 2024-04-02T07:54:30.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.764+0000 7fa365c00700 1 -- start start 2024-04-02T07:54:30.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.765+0000 7fa365c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344005210 0x7fa3601553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.765+0000 7fa365c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3600d00a0 0x7fa3600d0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.765+0000 7fa365c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3600b8830 0x7fa3600b8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.765+0000 7fa365c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3600b7f30 con 0x7fa344005210 2024-04-02T07:54:30.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.765+0000 7fa365c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3600d0a60 con 0x7fa3600b8830 2024-04-02T07:54:30.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.765+0000 7fa365c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3600b72b0 con 0x7fa3600d00a0 2024-04-02T07:54:30.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.765+0000 7fa35ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344005210 0x7fa3601553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.765+0000 7fa35ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344005210 0x7fa3601553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35952/0 (socket says 172.21.15.89:35952) 2024-04-02T07:54:30.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.765+0000 7fa35ea00700 1 -- 172.21.15.89:0/3877944214 learned_addr learned my addr 172.21.15.89:0/3877944214 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:30.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.765+0000 7fa35e000700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3600d00a0 0x7fa3600d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.765+0000 7fa35f400700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3600b8830 0x7fa3600b8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.766+0000 7fa35ea00700 1 -- 172.21.15.89:0/3877944214 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3600d00a0 msgr2=0x7fa3600d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.766+0000 7fa35ea00700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3600d00a0 0x7fa3600d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.766+0000 7fa35ea00700 1 -- 172.21.15.89:0/3877944214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3600b8830 msgr2=0x7fa3600b8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.766+0000 7fa35ea00700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3600b8830 0x7fa3600b8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.766+0000 7fa35ea00700 1 -- 172.21.15.89:0/3877944214 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3600b7590 con 0x7fa344005210 2024-04-02T07:54:30.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.766+0000 7fa35e000700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3600d00a0 0x7fa3600d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:30.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.766+0000 7fa35ea00700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344005210 0x7fa3601553c0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fa350010c30 tx=0x7fa350010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:30.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.766+0000 7fa356a00700 1 -- 172.21.15.89:0/3877944214 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35000e8d0 con 0x7fa344005210 2024-04-02T07:54:30.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.766+0000 7fa356a00700 1 -- 172.21.15.89:0/3877944214 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35000ea30 con 0x7fa344005210 2024-04-02T07:54:30.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.766+0000 7fa356a00700 1 -- 172.21.15.89:0/3877944214 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35000eda0 con 0x7fa344005210 2024-04-02T07:54:30.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.766+0000 7fa365c00700 1 -- 172.21.15.89:0/3877944214 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3600b7870 con 0x7fa344005210 2024-04-02T07:54:30.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.766+0000 7fa365c00700 1 -- 172.21.15.89:0/3877944214 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3600b91f0 con 0x7fa344005210 2024-04-02T07:54:30.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.767+0000 7fa365c00700 1 -- 172.21.15.89:0/3877944214 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa360067720 con 0x7fa344005210 2024-04-02T07:54:30.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.768+0000 7fa356a00700 1 -- 172.21.15.89:0/3877944214 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa350016460 con 0x7fa344005210 2024-04-02T07:54:30.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.769+0000 7fa356a00700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa34006adb0 0x7fa34006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:30.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.769+0000 7fa356a00700 1 -- 172.21.15.89:0/3877944214 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa350089ac0 con 0x7fa344005210 2024-04-02T07:54:30.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.769+0000 7fa35e000700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa34006adb0 0x7fa34006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:30.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.770+0000 7fa35e000700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa34006adb0 0x7fa34006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa34c001010 tx=0x7fa34c009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:30.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.774+0000 7fa356a00700 1 -- 172.21.15.89:0/3877944214 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa350054a20 con 0x7fa344005210 2024-04-02T07:54:30.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.974+0000 7fa365c00700 1 -- 172.21.15.89:0/3877944214 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3600cf6b0 con 0x7fa344005210 2024-04-02T07:54:30.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.975+0000 7fa356a00700 1 -- 172.21.15.89:0/3877944214 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+16591 (secure 0 0 0) 0x7fa350058040 con 0x7fa344005210 2024-04-02T07:54:30.977 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:30.977 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":124,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:54:30.522109+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:54:28.130987+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-02T07:54:29.515956+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:54:30.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.978+0000 7fa365c00700 1 -- 172.21.15.89:0/3877944214 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa34006adb0 msgr2=0x7fa34006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.978+0000 7fa365c00700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa34006adb0 0x7fa34006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa34c001010 tx=0x7fa34c009040 comp rx=0 tx=0).stop 2024-04-02T07:54:30.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.978+0000 7fa365c00700 1 -- 172.21.15.89:0/3877944214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344005210 msgr2=0x7fa3601553c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:30.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.978+0000 7fa365c00700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344005210 0x7fa3601553c0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fa350010c30 tx=0x7fa350010f40 comp rx=0 tx=0).stop 2024-04-02T07:54:30.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.978+0000 7fa365c00700 1 -- 172.21.15.89:0/3877944214 shutdown_connections 2024-04-02T07:54:30.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.978+0000 7fa365c00700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa34006adb0 0x7fa34006d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.978+0000 7fa365c00700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3600b8830 0x7fa3600b8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.978+0000 7fa365c00700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa344005210 0x7fa3601553c0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.978+0000 7fa365c00700 1 --2- 172.21.15.89:0/3877944214 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3600d00a0 0x7fa3600d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:30.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.978+0000 7fa365c00700 1 -- 172.21.15.89:0/3877944214 >> 172.21.15.89:0/3877944214 conn(0x7fa36008aff0 msgr2=0x7fa3600ca750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:30.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.978+0000 7fa365c00700 1 -- 172.21.15.89:0/3877944214 shutdown_connections 2024-04-02T07:54:30.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:30.978+0000 7fa365c00700 1 -- 172.21.15.89:0/3877944214 wait complete. 2024-04-02T07:54:30.999 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:54:31.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.192+0000 7faf1ea00700 1 Processor -- start 2024-04-02T07:54:31.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.192+0000 7faf1ea00700 1 -- start start 2024-04-02T07:54:31.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.193+0000 7faf1ea00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7faf180d18b0 0x7faf180cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.193+0000 7faf1ea00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf18161590 0x7faf18161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.194+0000 7faf1ea00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf180ceda0 con 0x7faf18160f10 2024-04-02T07:54:31.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.194+0000 7faf1ea00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf180d1d30 con 0x7faf180d18b0 2024-04-02T07:54:31.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.194+0000 7faf1ea00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf18161f60 con 0x7faf18161590 2024-04-02T07:54:31.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.194+0000 7faf17e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7faf18160f10 0x7faf180d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36694/0 (socket says 172.21.15.89:36694) 2024-04-02T07:54:31.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.194+0000 7faf17e00700 1 -- 172.21.15.89:0/99847649 learned_addr learned my addr 172.21.15.89:0/99847649 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:31.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.194+0000 7faf1ca00700 1 --2- 172.21.15.89:0/99847649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf18161590 0x7faf18161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:31.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.194+0000 7faf17400700 1 --2- 172.21.15.89:0/99847649 >> v2:172.21.15.113:3300/0 conn(0x7faf180d18b0 0x7faf180cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:31.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.194+0000 7faf16a00700 1 -- 172.21.15.89:0/99847649 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408426252 0 0) 0x7faf180ceda0 con 0x7faf18160f10 2024-04-02T07:54:31.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.194+0000 7faf16a00700 1 -- 172.21.15.89:0/99847649 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faefc0035b0 con 0x7faf18160f10 2024-04-02T07:54:31.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf16a00700 1 -- 172.21.15.89:0/99847649 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1605678117 0 0) 0x7faefc0035b0 con 0x7faf18160f10 2024-04-02T07:54:31.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf16a00700 1 -- 172.21.15.89:0/99847649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf18161590 msgr2=0x7faf18161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:31.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf16a00700 1 --2- 172.21.15.89:0/99847649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf18161590 0x7faf18161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf16a00700 1 -- 172.21.15.89:0/99847649 >> v2:172.21.15.113:3300/0 conn(0x7faf180d18b0 msgr2=0x7faf180cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:31.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf16a00700 1 --2- 172.21.15.89:0/99847649 >> v2:172.21.15.113:3300/0 conn(0x7faf180d18b0 0x7faf180cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf16a00700 1 -- 172.21.15.89:0/99847649 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf181612f0 con 0x7faf18160f10 2024-04-02T07:54:31.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf16a00700 1 -- 172.21.15.89:0/99847649 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7faf08001670 con 0x7faf18160f10 2024-04-02T07:54:31.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf16a00700 1 -- 172.21.15.89:0/99847649 >> v1:172.21.15.89:6789/0 conn(0x7faf18160f10 legacy=0x7faf180d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:31.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf16a00700 1 --2- 172.21.15.89:0/99847649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faefc003a30 0x7faefc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf16a00700 1 --2- 172.21.15.89:0/99847649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf18161590 0x7faefc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf16a00700 1 --2- 172.21.15.89:0/99847649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faefc004d90 0x7faefc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf16a00700 1 -- 172.21.15.89:0/99847649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf180d1d30 con 0x7faefc004d90 2024-04-02T07:54:31.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf16a00700 1 -- 172.21.15.89:0/99847649 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf18161f60 con 0x7faefc003a30 2024-04-02T07:54:31.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf16a00700 1 -- 172.21.15.89:0/99847649 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faefc00ba60 con 0x7faf18161590 2024-04-02T07:54:31.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf1ca00700 1 --2- 172.21.15.89:0/99847649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faefc004d90 0x7faefc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:31.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf17e00700 1 --2- 172.21.15.89:0/99847649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faefc003a30 0x7faefc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:31.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf17400700 1 --2- 172.21.15.89:0/99847649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf18161590 0x7faefc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:31.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf1ca00700 1 -- 172.21.15.89:0/99847649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf18161590 msgr2=0x7faefc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:31.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf1ca00700 1 --2- 172.21.15.89:0/99847649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf18161590 0x7faefc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf1ca00700 1 -- 172.21.15.89:0/99847649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faefc003a30 msgr2=0x7faefc003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:31.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf1ca00700 1 --2- 172.21.15.89:0/99847649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faefc003a30 0x7faefc003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.195+0000 7faf1ca00700 1 -- 172.21.15.89:0/99847649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faefc0035b0 con 0x7faefc004d90 2024-04-02T07:54:31.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.196+0000 7faf1ca00700 1 --2- 172.21.15.89:0/99847649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faefc004d90 0x7faefc005210 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7faf1000e170 tx=0x7faefc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f4b506755ebff3c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:31.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.196+0000 7faf16a00700 1 -- 172.21.15.89:0/99847649 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1000ec90 con 0x7faefc004d90 2024-04-02T07:54:31.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.196+0000 7faf16a00700 1 -- 172.21.15.89:0/99847649 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1000f040 con 0x7faefc004d90 2024-04-02T07:54:31.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.196+0000 7faf16a00700 1 -- 172.21.15.89:0/99847649 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf10014480 con 0x7faefc004d90 2024-04-02T07:54:31.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.196+0000 7faf1ea00700 1 -- 172.21.15.89:0/99847649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faefc004d90 msgr2=0x7faefc005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:31.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.196+0000 7faf1ea00700 1 --2- 172.21.15.89:0/99847649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faefc004d90 0x7faefc005210 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7faf1000e170 tx=0x7faefc005880 comp rx=0 tx=0).stop 2024-04-02T07:54:31.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.196+0000 7faf1ca00700 1 -- 172.21.15.89:0/99847649 reap_dead start 2024-04-02T07:54:31.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.196+0000 7faf1ea00700 1 -- 172.21.15.89:0/99847649 shutdown_connections 2024-04-02T07:54:31.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.196+0000 7faf1ea00700 1 -- 172.21.15.89:0/99847649 >> 172.21.15.89:0/99847649 conn(0x7faf1808aff0 msgr2=0x7faf180d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:31.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.197+0000 7faf1ea00700 1 -- 172.21.15.89:0/99847649 shutdown_connections 2024-04-02T07:54:31.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.197+0000 7faf1ea00700 1 -- 172.21.15.89:0/99847649 wait complete. 2024-04-02T07:54:31.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.199+0000 7faf1ea00700 1 Processor -- start 2024-04-02T07:54:31.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.199+0000 7faf1ea00700 1 -- start start 2024-04-02T07:54:31.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.200+0000 7faf1ea00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf181e92b0 0x7faf181e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.200+0000 7faf1ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf181e8340 0x7faf181e87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.200+0000 7faf1ea00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf18149f20 0x7faf1814a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.200+0000 7faf1ea00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf181600e0 con 0x7faf181e8340 2024-04-02T07:54:31.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.200+0000 7faf1ea00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf18149440 con 0x7faf181e92b0 2024-04-02T07:54:31.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.200+0000 7faf1ea00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf18149720 con 0x7faf18149f20 2024-04-02T07:54:31.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf17400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf181e8340 0x7faf181e87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf17400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf181e8340 0x7faf181e87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35984/0 (socket says 172.21.15.89:35984) 2024-04-02T07:54:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf17400700 1 -- 172.21.15.89:0/2561118393 learned_addr learned my addr 172.21.15.89:0/2561118393 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf17e00700 1 --2- 172.21.15.89:0/2561118393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf181e92b0 0x7faf181e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf1ca00700 1 --2- 172.21.15.89:0/2561118393 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf18149f20 0x7faf1814a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf17400700 1 -- 172.21.15.89:0/2561118393 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf18149f20 msgr2=0x7faf1814a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf17400700 1 --2- 172.21.15.89:0/2561118393 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf18149f20 0x7faf1814a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf17400700 1 -- 172.21.15.89:0/2561118393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf181e92b0 msgr2=0x7faf181e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf17400700 1 --2- 172.21.15.89:0/2561118393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf181e92b0 0x7faf181e7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf17400700 1 -- 172.21.15.89:0/2561118393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf18149a00 con 0x7faf181e8340 2024-04-02T07:54:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf17400700 1 --2- 172.21.15.89:0/2561118393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf181e8340 0x7faf181e87c0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7faf04011aa0 tx=0x7faf04011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf0fe00700 1 -- 172.21.15.89:0/2561118393 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0400f940 con 0x7faf181e8340 2024-04-02T07:54:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf0fe00700 1 -- 172.21.15.89:0/2561118393 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0400faa0 con 0x7faf181e8340 2024-04-02T07:54:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf0fe00700 1 -- 172.21.15.89:0/2561118393 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0400fe50 con 0x7faf181e8340 2024-04-02T07:54:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.201+0000 7faf1ea00700 1 -- 172.21.15.89:0/2561118393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf1814c080 con 0x7faf181e8340 2024-04-02T07:54:31.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.202+0000 7faf1ea00700 1 -- 172.21.15.89:0/2561118393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf181e9870 con 0x7faf181e8340 2024-04-02T07:54:31.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.202+0000 7faf1ea00700 1 -- 172.21.15.89:0/2561118393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf181492a0 con 0x7faf181e8340 2024-04-02T07:54:31.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.203+0000 7faf0fe00700 1 -- 172.21.15.89:0/2561118393 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7faf04014440 con 0x7faf181e8340 2024-04-02T07:54:31.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.204+0000 7faf0fe00700 1 --2- 172.21.15.89:0/2561118393 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faef806adb0 0x7faef806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.204+0000 7faf0fe00700 1 -- 172.21.15.89:0/2561118393 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6057+0+0 (secure 0 0 0) 0x7faf04089e00 con 0x7faf181e8340 2024-04-02T07:54:31.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.204+0000 7faf17e00700 1 --2- 172.21.15.89:0/2561118393 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faef806adb0 0x7faef806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:31.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.205+0000 7faf17e00700 1 --2- 172.21.15.89:0/2561118393 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faef806adb0 0x7faef806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faf080017a0 tx=0x7faf080015d0 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:31.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.209+0000 7faf0fe00700 1 -- 172.21.15.89:0/2561118393 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faf181492a0 con 0x7faf181e8340 2024-04-02T07:54:31.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.413+0000 7faf1ea00700 1 -- 172.21.15.89:0/2561118393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf18149580 con 0x7faf181e8340 2024-04-02T07:54:31.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.415+0000 7faf0fe00700 1 -- 172.21.15.89:0/2561118393 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+16591 (secure 0 0 0) 0x7faf18149580 con 0x7faf181e8340 2024-04-02T07:54:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:31.418 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":124,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:54:30.522109+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:54:28.130987+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-02T07:54:29.515956+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:54:31.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.417+0000 7faf1ea00700 1 -- 172.21.15.89:0/2561118393 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faef806adb0 msgr2=0x7faef806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:31.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.417+0000 7faf1ea00700 1 --2- 172.21.15.89:0/2561118393 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faef806adb0 0x7faef806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faf080017a0 tx=0x7faf080015d0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.417+0000 7faf1ea00700 1 -- 172.21.15.89:0/2561118393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf181e8340 msgr2=0x7faf181e87c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:31.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.417+0000 7faf1ea00700 1 --2- 172.21.15.89:0/2561118393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf181e8340 0x7faf181e87c0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7faf04011aa0 tx=0x7faf04011db0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.418+0000 7faf1ea00700 1 -- 172.21.15.89:0/2561118393 shutdown_connections 2024-04-02T07:54:31.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.418+0000 7faf1ea00700 1 --2- 172.21.15.89:0/2561118393 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faef806adb0 0x7faef806d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.418+0000 7faf1ea00700 1 --2- 172.21.15.89:0/2561118393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf181e92b0 0x7faf181e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.418+0000 7faf1ea00700 1 --2- 172.21.15.89:0/2561118393 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf18149f20 0x7faf1814a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.418+0000 7faf1ea00700 1 --2- 172.21.15.89:0/2561118393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf181e8340 0x7faf181e87c0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.418+0000 7faf1ea00700 1 -- 172.21.15.89:0/2561118393 >> 172.21.15.89:0/2561118393 conn(0x7faf1808aff0 msgr2=0x7faf180ca850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:31.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.418+0000 7faf1ea00700 1 -- 172.21.15.89:0/2561118393 shutdown_connections 2024-04-02T07:54:31.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.418+0000 7faf1ea00700 1 -- 172.21.15.89:0/2561118393 wait complete. 2024-04-02T07:54:31.437 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T07:54:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.635+0000 7fe683c00700 1 Processor -- start 2024-04-02T07:54:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.635+0000 7fe683c00700 1 -- start start 2024-04-02T07:54:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.636+0000 7fe683c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe67c0cd5c0 0x7fe67c15e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.636+0000 7fe683c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c06f310 0x7fe67c06f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.636+0000 7fe683c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c15eda0 con 0x7fe67c15cfe0 2024-04-02T07:54:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.636+0000 7fe683c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67c06fce0 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.636+0000 7fe683c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c099dd0 con 0x7fe67c06f310 2024-04-02T07:54:31.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.637+0000 7fe681c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe67c15cfe0 0x7fe67c15d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36700/0 (socket says 172.21.15.89:36700) 2024-04-02T07:54:31.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.637+0000 7fe681c00700 1 -- 172.21.15.89:0/395893170 learned_addr learned my addr 172.21.15.89:0/395893170 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:31.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.637+0000 7fe67be00700 1 --2- 172.21.15.89:0/395893170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c06f310 0x7fe67c06f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.637+0000 7fe681200700 1 --2- 172.21.15.89:0/395893170 >> v2:172.21.15.113:3300/0 conn(0x7fe67c0cd5c0 0x7fe67c15e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.637+0000 7fe67b400700 1 -- 172.21.15.89:0/395893170 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162159907 0 0) 0x7fe674001490 con 0x7fe67c15cfe0 2024-04-02T07:54:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.637+0000 7fe67b400700 1 -- 172.21.15.89:0/395893170 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6640035b0 con 0x7fe67c15cfe0 2024-04-02T07:54:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.637+0000 7fe681200700 1 -- 172.21.15.89:0/395893170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c06f310 msgr2=0x7fe67c06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.637+0000 7fe681200700 1 --2- 172.21.15.89:0/395893170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c06f310 0x7fe67c06f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.637+0000 7fe67b400700 1 -- 172.21.15.89:0/395893170 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2429677563 0 0) 0x7fe6640035b0 con 0x7fe67c15cfe0 2024-04-02T07:54:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.637+0000 7fe681200700 1 -- 172.21.15.89:0/395893170 >> v1:172.21.15.89:6789/0 conn(0x7fe67c15cfe0 legacy=0x7fe67c15d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.637+0000 7fe681200700 1 -- 172.21.15.89:0/395893170 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c09a0b0 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.637+0000 7fe67be00700 1 --2- 172.21.15.89:0/395893170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c06f310 0x7fe67c06f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.638+0000 7fe681200700 1 --2- 172.21.15.89:0/395893170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c0cd5c0 0x7fe67c15e860 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe670009840 tx=0x7fe670009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e72230814eb88c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.638+0000 7fe67b400700 1 -- 172.21.15.89:0/395893170 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe674001b80 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.638+0000 7fe67b400700 1 -- 172.21.15.89:0/395893170 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe670004030 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.638+0000 7fe67b400700 1 -- 172.21.15.89:0/395893170 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6700043a0 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.639+0000 7fe683c00700 1 -- 172.21.15.89:0/395893170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c0cd5c0 msgr2=0x7fe67c15e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:31.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.639+0000 7fe683c00700 1 --2- 172.21.15.89:0/395893170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c0cd5c0 0x7fe67c15e860 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe670009840 tx=0x7fe670009b50 comp rx=0 tx=0).stop 2024-04-02T07:54:31.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.639+0000 7fe683c00700 1 -- 172.21.15.89:0/395893170 shutdown_connections 2024-04-02T07:54:31.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.639+0000 7fe683c00700 1 --2- 172.21.15.89:0/395893170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c0cd5c0 0x7fe67c15e860 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.639+0000 7fe683c00700 1 --2- 172.21.15.89:0/395893170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c06f310 0x7fe67c06f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.639+0000 7fe683c00700 1 -- 172.21.15.89:0/395893170 >> 172.21.15.89:0/395893170 conn(0x7fe67c0d1370 msgr2=0x7fe67c0d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:31.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.639+0000 7fe683c00700 1 -- 172.21.15.89:0/395893170 shutdown_connections 2024-04-02T07:54:31.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.639+0000 7fe683c00700 1 -- 172.21.15.89:0/395893170 wait complete. 2024-04-02T07:54:31.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.641+0000 7fe683c00700 1 Processor -- start 2024-04-02T07:54:31.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.641+0000 7fe683c00700 1 -- start start 2024-04-02T07:54:31.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.641+0000 7fe683c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c06f310 0x7fe67c1f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe683c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0cd5c0 0x7fe67c1f0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe683c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c15cfe0 0x7fe67c0cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe683c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c0b63b0 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe683c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c1f4330 con 0x7fe67c06f310 2024-04-02T07:54:31.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe683c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c1f0d90 con 0x7fe67c15cfe0 2024-04-02T07:54:31.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe67be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0cd5c0 0x7fe67c1f0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:31.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe67be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0cd5c0 0x7fe67c1f0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35998/0 (socket says 172.21.15.89:35998) 2024-04-02T07:54:31.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe67be00700 1 -- 172.21.15.89:0/1110373053 learned_addr learned my addr 172.21.15.89:0/1110373053 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:31.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe681200700 1 --2- 172.21.15.89:0/1110373053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c06f310 0x7fe67c1f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:31.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe67be00700 1 -- 172.21.15.89:0/1110373053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c15cfe0 msgr2=0x7fe67c0cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:31.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe67be00700 1 --2- 172.21.15.89:0/1110373053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c15cfe0 0x7fe67c0cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe67be00700 1 -- 172.21.15.89:0/1110373053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c06f310 msgr2=0x7fe67c1f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:31.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe67be00700 1 --2- 172.21.15.89:0/1110373053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c06f310 0x7fe67c1f3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:31.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe67be00700 1 -- 172.21.15.89:0/1110373053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6680094a0 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe681200700 1 --2- 172.21.15.89:0/1110373053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c06f310 0x7fe67c1f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:31.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.642+0000 7fe67be00700 1 --2- 172.21.15.89:0/1110373053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0cd5c0 0x7fe67c1f0850 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fe66800ec10 tx=0x7fe668009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:31.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.643+0000 7fe678c00700 1 -- 172.21.15.89:0/1110373053 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66800c940 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.643+0000 7fe678c00700 1 -- 172.21.15.89:0/1110373053 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66800caa0 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.643+0000 7fe678c00700 1 -- 172.21.15.89:0/1110373053 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66800ce10 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.643+0000 7fe683c00700 1 -- 172.21.15.89:0/1110373053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6700094a0 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.643+0000 7fe683c00700 1 -- 172.21.15.89:0/1110373053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67c0b5730 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.644+0000 7fe683c00700 1 -- 172.21.15.89:0/1110373053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67c0b64f0 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.645+0000 7fe678c00700 1 -- 172.21.15.89:0/1110373053 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe668013470 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.646+0000 7fe678c00700 1 --2- 172.21.15.89:0/1110373053 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe65c06adb0 0x7fe65c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:31.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.646+0000 7fe678c00700 1 -- 172.21.15.89:0/1110373053 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe668089860 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.646+0000 7fe681200700 1 --2- 172.21.15.89:0/1110373053 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe65c06adb0 0x7fe65c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:31.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.646+0000 7fe681200700 1 --2- 172.21.15.89:0/1110373053 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe65c06adb0 0x7fe65c06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe670009b90 tx=0x7fe670008030 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:31.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.650+0000 7fe678c00700 1 -- 172.21.15.89:0/1110373053 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe67c0b64f0 con 0x7fe67c0cd5c0 2024-04-02T07:54:31.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:31.898+0000 7fe683c00700 1 -- 172.21.15.89:0/1110373053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe67c0b64f0 con 0x7fe67c0cd5c0 2024-04-02T07:54:32.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.537+0000 7fe678c00700 1 -- 172.21.15.89:0/1110373053 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v56) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe67c0b64f0 con 0x7fe67c0cd5c0 2024-04-02T07:54:32.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.540+0000 7fe683c00700 1 -- 172.21.15.89:0/1110373053 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe65c06adb0 msgr2=0x7fe65c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:32.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.540+0000 7fe683c00700 1 --2- 172.21.15.89:0/1110373053 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe65c06adb0 0x7fe65c06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe670009b90 tx=0x7fe670008030 comp rx=0 tx=0).stop 2024-04-02T07:54:32.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.540+0000 7fe683c00700 1 -- 172.21.15.89:0/1110373053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0cd5c0 msgr2=0x7fe67c1f0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:32.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.540+0000 7fe683c00700 1 --2- 172.21.15.89:0/1110373053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0cd5c0 0x7fe67c1f0850 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fe66800ec10 tx=0x7fe668009ec0 comp rx=0 tx=0).stop 2024-04-02T07:54:32.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.540+0000 7fe683c00700 1 -- 172.21.15.89:0/1110373053 shutdown_connections 2024-04-02T07:54:32.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.540+0000 7fe683c00700 1 --2- 172.21.15.89:0/1110373053 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe65c06adb0 0x7fe65c06d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:32.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.540+0000 7fe683c00700 1 --2- 172.21.15.89:0/1110373053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe67c06f310 0x7fe67c1f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:32.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.540+0000 7fe683c00700 1 --2- 172.21.15.89:0/1110373053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe67c15cfe0 0x7fe67c0cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:32.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.540+0000 7fe683c00700 1 --2- 172.21.15.89:0/1110373053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0cd5c0 0x7fe67c1f0850 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:32.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.540+0000 7fe683c00700 1 -- 172.21.15.89:0/1110373053 >> 172.21.15.89:0/1110373053 conn(0x7fe67c0d1370 msgr2=0x7fe67c0b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:32.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.540+0000 7fe683c00700 1 -- 172.21.15.89:0/1110373053 shutdown_connections 2024-04-02T07:54:32.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.540+0000 7fe683c00700 1 -- 172.21.15.89:0/1110373053 wait complete. 2024-04-02T07:54:32.560 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T07:54:32.561 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T07:54:32.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.755+0000 7f4c15200700 1 Processor -- start 2024-04-02T07:54:32.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.755+0000 7f4c15200700 1 -- start start 2024-04-02T07:54:32.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.757+0000 7f4c15200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c100d2540 0x7f4c101687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:32.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.757+0000 7f4c15200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c10161420 con 0x7f4c1015d9d0 2024-04-02T07:54:32.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.757+0000 7f4c15200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c10161700 con 0x7f4c100d16a0 2024-04-02T07:54:32.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.757+0000 7f4c15200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c100d4340 con 0x7f4c100d2540 2024-04-02T07:54:32.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.757+0000 7f4c0de00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4c1015d9d0 0x7f4c100d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36704/0 (socket says 172.21.15.89:36704) 2024-04-02T07:54:32.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.757+0000 7f4c0de00700 1 -- 172.21.15.89:0/1812139311 learned_addr learned my addr 172.21.15.89:0/1812139311 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:32.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.757+0000 7f4c0e800700 1 --2- 172.21.15.89:0/1812139311 >> v2:172.21.15.113:3300/0 conn(0x7f4c100d2540 0x7f4c101687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:32.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.757+0000 7f4c0ca00700 1 -- 172.21.15.89:0/1812139311 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771464170 0 0) 0x7f4c10161420 con 0x7f4c1015d9d0 2024-04-02T07:54:32.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.757+0000 7f4c0ca00700 1 -- 172.21.15.89:0/1812139311 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf40035b0 con 0x7f4c1015d9d0 2024-04-02T07:54:32.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0ca00700 1 -- 172.21.15.89:0/1812139311 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1941838174 0 0) 0x7f4bf40035b0 con 0x7f4c1015d9d0 2024-04-02T07:54:32.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0ca00700 1 -- 172.21.15.89:0/1812139311 >> v2:172.21.15.113:3300/0 conn(0x7f4c100d2540 msgr2=0x7f4c101687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:32.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0ca00700 1 --2- 172.21.15.89:0/1812139311 >> v2:172.21.15.113:3300/0 conn(0x7f4c100d2540 0x7f4c101687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:32.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0ca00700 1 -- 172.21.15.89:0/1812139311 >> v1:172.21.15.113:6789/0 conn(0x7f4c100d16a0 legacy=0x7f4c100d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:32.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0ca00700 1 -- 172.21.15.89:0/1812139311 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c101619e0 con 0x7f4c1015d9d0 2024-04-02T07:54:32.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0ca00700 1 -- 172.21.15.89:0/1812139311 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f4c00001670 con 0x7f4c1015d9d0 2024-04-02T07:54:32.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0ca00700 1 -- 172.21.15.89:0/1812139311 >> v1:172.21.15.89:6789/0 conn(0x7f4c1015d9d0 legacy=0x7f4c100d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:32.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0ca00700 1 --2- 172.21.15.89:0/1812139311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bf4003a00 0x7f4bf4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:32.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0ca00700 1 --2- 172.21.15.89:0/1812139311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bf4004850 0x7f4bf4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:32.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0ca00700 1 --2- 172.21.15.89:0/1812139311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bf4005210 0x7f4bf400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:32.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0ca00700 1 -- 172.21.15.89:0/1812139311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf400c180 con 0x7f4bf4003a00 2024-04-02T07:54:32.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0ca00700 1 -- 172.21.15.89:0/1812139311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf400c2f0 con 0x7f4bf4004850 2024-04-02T07:54:32.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0ca00700 1 -- 172.21.15.89:0/1812139311 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf400c4b0 con 0x7f4bf4005210 2024-04-02T07:54:32.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0de00700 1 --2- 172.21.15.89:0/1812139311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bf4003a00 0x7f4bf4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:32.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0e800700 1 --2- 172.21.15.89:0/1812139311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bf4005210 0x7f4bf400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:32.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0d400700 1 --2- 172.21.15.89:0/1812139311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bf4004850 0x7f4bf4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:32.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0de00700 1 -- 172.21.15.89:0/1812139311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bf4005210 msgr2=0x7f4bf400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:32.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0de00700 1 --2- 172.21.15.89:0/1812139311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bf4005210 0x7f4bf400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:32.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0de00700 1 -- 172.21.15.89:0/1812139311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bf4004850 msgr2=0x7f4bf4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:32.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.758+0000 7f4c0de00700 1 --2- 172.21.15.89:0/1812139311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bf4004850 0x7f4bf4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:32.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.759+0000 7f4c0de00700 1 -- 172.21.15.89:0/1812139311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf40035b0 con 0x7f4bf4003a00 2024-04-02T07:54:32.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.759+0000 7f4c0e800700 1 --2- 172.21.15.89:0/1812139311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bf4005210 0x7f4bf400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:32.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.759+0000 7f4c0d400700 1 --2- 172.21.15.89:0/1812139311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bf4004850 0x7f4bf4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:32.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.759+0000 7f4c0e800700 1 -- 172.21.15.89:0/1812139311 reap_dead start 2024-04-02T07:54:32.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.759+0000 7f4c0de00700 1 --2- 172.21.15.89:0/1812139311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bf4003a00 0x7f4bf4003e20 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f4c0000c240 tx=0x7f4bf4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c09a65202e63f14d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:32.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.759+0000 7f4c0ca00700 1 -- 172.21.15.89:0/1812139311 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c00007c10 con 0x7f4bf4003a00 2024-04-02T07:54:32.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.759+0000 7f4c0ca00700 1 -- 172.21.15.89:0/1812139311 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0000ce00 con 0x7f4bf4003a00 2024-04-02T07:54:32.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.759+0000 7f4c0ca00700 1 -- 172.21.15.89:0/1812139311 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c00005ba0 con 0x7f4bf4003a00 2024-04-02T07:54:32.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.759+0000 7f4c15200700 1 -- 172.21.15.89:0/1812139311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bf4003a00 msgr2=0x7f4bf4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:32.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.759+0000 7f4c15200700 1 --2- 172.21.15.89:0/1812139311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bf4003a00 0x7f4bf4003e20 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f4c0000c240 tx=0x7f4bf4005880 comp rx=0 tx=0).stop 2024-04-02T07:54:32.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.760+0000 7f4c15200700 1 -- 172.21.15.89:0/1812139311 shutdown_connections 2024-04-02T07:54:32.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.760+0000 7f4c15200700 1 --2- 172.21.15.89:0/1812139311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bf4003a00 0x7f4bf4003e20 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:32.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.760+0000 7f4c15200700 1 -- 172.21.15.89:0/1812139311 >> 172.21.15.89:0/1812139311 conn(0x7f4c1008aff0 msgr2=0x7f4c100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:32.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.760+0000 7f4c15200700 1 -- 172.21.15.89:0/1812139311 shutdown_connections 2024-04-02T07:54:32.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.760+0000 7f4c15200700 1 -- 172.21.15.89:0/1812139311 wait complete. 2024-04-02T07:54:32.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.761+0000 7f4c15200700 1 Processor -- start 2024-04-02T07:54:32.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.762+0000 7f4c15200700 1 -- start start 2024-04-02T07:54:32.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.762+0000 7f4c15200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bf4003a00 0x7f4c101f45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:32.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.762+0000 7f4c15200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c10158b80 0x7f4c100b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:32.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.762+0000 7f4c15200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c100b4ad0 0x7f4c1015f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:32.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.762+0000 7f4c15200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c100b6b60 con 0x7f4c100b4ad0 2024-04-02T07:54:32.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.762+0000 7f4c15200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101601c0 con 0x7f4c10158b80 2024-04-02T07:54:32.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.762+0000 7f4c15200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c100b4f50 con 0x7f4bf4003a00 2024-04-02T07:54:32.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.763+0000 7f4c0e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c100b4ad0 0x7f4c1015f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:32.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.763+0000 7f4c0de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bf4003a00 0x7f4c101f45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:32.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.763+0000 7f4c0e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c100b4ad0 0x7f4c1015f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36012/0 (socket says 172.21.15.89:36012) 2024-04-02T07:54:32.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.763+0000 7f4c0e800700 1 -- 172.21.15.89:0/1323624510 learned_addr learned my addr 172.21.15.89:0/1323624510 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:32.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.763+0000 7f4c0d400700 1 --2- 172.21.15.89:0/1323624510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c10158b80 0x7f4c100b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:32.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.763+0000 7f4c0e800700 1 -- 172.21.15.89:0/1323624510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bf4003a00 msgr2=0x7f4c101f45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:32.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.763+0000 7f4c0e800700 1 --2- 172.21.15.89:0/1323624510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bf4003a00 0x7f4c101f45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:32.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.763+0000 7f4c0e800700 1 -- 172.21.15.89:0/1323624510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c10158b80 msgr2=0x7f4c100b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:32.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.763+0000 7f4c0e800700 1 --2- 172.21.15.89:0/1323624510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c10158b80 0x7f4c100b4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:32.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.763+0000 7f4c0e800700 1 -- 172.21.15.89:0/1323624510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c100b61c0 con 0x7f4c100b4ad0 2024-04-02T07:54:32.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.763+0000 7f4c0e800700 1 --2- 172.21.15.89:0/1323624510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c100b4ad0 0x7f4c1015f540 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f4c100d2f40 tx=0x7f4c080104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:32.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.764+0000 7f4c06000700 1 -- 172.21.15.89:0/1323624510 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0801b070 con 0x7f4c100b4ad0 2024-04-02T07:54:32.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.764+0000 7f4c06000700 1 -- 172.21.15.89:0/1323624510 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0800f040 con 0x7f4c100b4ad0 2024-04-02T07:54:32.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.764+0000 7f4c06000700 1 -- 172.21.15.89:0/1323624510 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0800e050 con 0x7f4c100b4ad0 2024-04-02T07:54:32.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.764+0000 7f4c15200700 1 -- 172.21.15.89:0/1323624510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c100b64a0 con 0x7f4c100b4ad0 2024-04-02T07:54:32.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.764+0000 7f4c15200700 1 -- 172.21.15.89:0/1323624510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c10159000 con 0x7f4c100b4ad0 2024-04-02T07:54:32.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.765+0000 7f4c15200700 1 -- 172.21.15.89:0/1323624510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c100b6ca0 con 0x7f4c100b4ad0 2024-04-02T07:54:32.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.766+0000 7f4c06000700 1 -- 172.21.15.89:0/1323624510 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4c08010d40 con 0x7f4c100b4ad0 2024-04-02T07:54:32.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.767+0000 7f4c06000700 1 --2- 172.21.15.89:0/1323624510 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4bf006adb0 0x7f4bf006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:32.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.767+0000 7f4c06000700 1 -- 172.21.15.89:0/1323624510 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4c08089860 con 0x7f4c100b4ad0 2024-04-02T07:54:32.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.767+0000 7f4c0de00700 1 --2- 172.21.15.89:0/1323624510 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4bf006adb0 0x7f4bf006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:32.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.768+0000 7f4c0de00700 1 --2- 172.21.15.89:0/1323624510 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4bf006adb0 0x7f4bf006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4c00000c00 tx=0x7f4c00007000 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:32.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:32.772+0000 7f4c06000700 1 -- 172.21.15.89:0/1323624510 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c100b6ca0 con 0x7f4c100b4ad0 2024-04-02T07:54:33.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.018+0000 7f4c15200700 1 -- 172.21.15.89:0/1323624510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4c100b6ca0 con 0x7f4c100b4ad0 2024-04-02T07:54:33.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.539+0000 7f4c06000700 1 -- 172.21.15.89:0/1323624510 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v57) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4c100b6ca0 con 0x7f4c100b4ad0 2024-04-02T07:54:33.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.542+0000 7f4c15200700 1 -- 172.21.15.89:0/1323624510 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4bf006adb0 msgr2=0x7f4bf006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:33.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.542+0000 7f4c15200700 1 --2- 172.21.15.89:0/1323624510 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4bf006adb0 0x7f4bf006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4c00000c00 tx=0x7f4c00007000 comp rx=0 tx=0).stop 2024-04-02T07:54:33.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.542+0000 7f4c15200700 1 -- 172.21.15.89:0/1323624510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c100b4ad0 msgr2=0x7f4c1015f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:33.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.542+0000 7f4c15200700 1 --2- 172.21.15.89:0/1323624510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c100b4ad0 0x7f4c1015f540 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f4c100d2f40 tx=0x7f4c080104a0 comp rx=0 tx=0).stop 2024-04-02T07:54:33.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.542+0000 7f4c15200700 1 -- 172.21.15.89:0/1323624510 shutdown_connections 2024-04-02T07:54:33.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.542+0000 7f4c15200700 1 --2- 172.21.15.89:0/1323624510 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4bf006adb0 0x7f4bf006d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:33.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.542+0000 7f4c15200700 1 --2- 172.21.15.89:0/1323624510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c100b4ad0 0x7f4c1015f540 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:33.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.542+0000 7f4c15200700 1 --2- 172.21.15.89:0/1323624510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bf4003a00 0x7f4c101f45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:33.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.542+0000 7f4c15200700 1 --2- 172.21.15.89:0/1323624510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c10158b80 0x7f4c100b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:33.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.542+0000 7f4c15200700 1 -- 172.21.15.89:0/1323624510 >> 172.21.15.89:0/1323624510 conn(0x7f4c1008aff0 msgr2=0x7f4c100b1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:33.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.542+0000 7f4c15200700 1 -- 172.21.15.89:0/1323624510 shutdown_connections 2024-04-02T07:54:33.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.542+0000 7f4c15200700 1 -- 172.21.15.89:0/1323624510 wait complete. 2024-04-02T07:54:33.563 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T07:54:33.563 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:54:33.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.754+0000 7f5f98c00700 1 Processor -- start 2024-04-02T07:54:33.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.754+0000 7f5f98c00700 1 -- start start 2024-04-02T07:54:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.755+0000 7f5f98c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5f940d05e0 0x7f5f940d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.755+0000 7f5f98c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5f940b6990 0x7f5f940b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.755+0000 7f5f98c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f940d0f80 con 0x7f5f9415da50 2024-04-02T07:54:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.755+0000 7f5f98c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f940b95d0 con 0x7f5f940d05e0 2024-04-02T07:54:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.755+0000 7f5f98c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f9415de30 con 0x7f5f940b6990 2024-04-02T07:54:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.755+0000 7f5f91400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5f940d05e0 0x7f5f940d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.756+0000 7f5f91400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5f940d05e0 0x7f5f940d0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36026/0 (socket says 172.21.15.89:36026) 2024-04-02T07:54:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.756+0000 7f5f91400700 1 -- 172.21.15.89:0/2025785388 learned_addr learned my addr 172.21.15.89:0/2025785388 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.756+0000 7f5f92800700 1 --2- 172.21.15.89:0/2025785388 >> v2:172.21.15.113:3300/0 conn(0x7f5f940b6990 0x7f5f940b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.756+0000 7f5f91400700 1 -- 172.21.15.89:0/2025785388 >> v2:172.21.15.113:3300/0 conn(0x7f5f940b6990 msgr2=0x7f5f940b6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.756+0000 7f5f91400700 1 --2- 172.21.15.89:0/2025785388 >> v2:172.21.15.113:3300/0 conn(0x7f5f940b6990 0x7f5f940b6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.756+0000 7f5f91400700 1 -- 172.21.15.89:0/2025785388 >> v1:172.21.15.113:6789/0 conn(0x7f5f9415da50 legacy=0x7f5f940ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.756+0000 7f5f91400700 1 -- 172.21.15.89:0/2025785388 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f940d24e0 con 0x7f5f940d05e0 2024-04-02T07:54:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.756+0000 7f5f91400700 1 --2- 172.21.15.89:0/2025785388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f940d05e0 0x7f5f940d0a40 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f5f940cde40 tx=0x7f5f80009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87506eee46063123 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.756+0000 7f5f90a00700 1 -- 172.21.15.89:0/2025785388 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f80011830 con 0x7f5f940d05e0 2024-04-02T07:54:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.757+0000 7f5f90a00700 1 -- 172.21.15.89:0/2025785388 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f80011990 con 0x7f5f940d05e0 2024-04-02T07:54:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.757+0000 7f5f90a00700 1 -- 172.21.15.89:0/2025785388 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f80011d20 con 0x7f5f940d05e0 2024-04-02T07:54:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.757+0000 7f5f98c00700 1 -- 172.21.15.89:0/2025785388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f940d05e0 msgr2=0x7f5f940d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.757+0000 7f5f98c00700 1 --2- 172.21.15.89:0/2025785388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f940d05e0 0x7f5f940d0a40 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f5f940cde40 tx=0x7f5f80009bd0 comp rx=0 tx=0).stop 2024-04-02T07:54:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.757+0000 7f5f98c00700 1 -- 172.21.15.89:0/2025785388 shutdown_connections 2024-04-02T07:54:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.757+0000 7f5f98c00700 1 --2- 172.21.15.89:0/2025785388 >> v2:172.21.15.113:3300/0 conn(0x7f5f940b6990 0x7f5f940b6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.757+0000 7f5f98c00700 1 --2- 172.21.15.89:0/2025785388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f940d05e0 0x7f5f940d0a40 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.757+0000 7f5f98c00700 1 -- 172.21.15.89:0/2025785388 >> 172.21.15.89:0/2025785388 conn(0x7f5f940666a0 msgr2=0x7f5f94066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.757+0000 7f5f98c00700 1 -- 172.21.15.89:0/2025785388 shutdown_connections 2024-04-02T07:54:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.757+0000 7f5f98c00700 1 -- 172.21.15.89:0/2025785388 wait complete. 2024-04-02T07:54:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.759+0000 7f5f98c00700 1 Processor -- start 2024-04-02T07:54:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.759+0000 7f5f98c00700 1 -- start start 2024-04-02T07:54:33.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.760+0000 7f5f98c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5f940b6990 0x7f5f940c7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:33.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.760+0000 7f5f98c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f940d05e0 0x7f5f941f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:33.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.760+0000 7f5f98c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f9415da50 0x7f5f94160140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:33.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.760+0000 7f5f98c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f941f73e0 con 0x7f5f9415da50 2024-04-02T07:54:33.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.760+0000 7f5f98c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9414b140 con 0x7f5f940d05e0 2024-04-02T07:54:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.760+0000 7f5f98c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f941f6760 con 0x7f5f940b6990 2024-04-02T07:54:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.760+0000 7f5f92800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f9415da50 0x7f5f94160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.760+0000 7f5f92800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f9415da50 0x7f5f94160140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36038/0 (socket says 172.21.15.89:36038) 2024-04-02T07:54:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.760+0000 7f5f92800700 1 -- 172.21.15.89:0/635565015 learned_addr learned my addr 172.21.15.89:0/635565015 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.760+0000 7f5f91400700 1 --2- 172.21.15.89:0/635565015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f940d05e0 0x7f5f941f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.760+0000 7f5f91e00700 1 --2- 172.21.15.89:0/635565015 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5f940b6990 0x7f5f940c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.761+0000 7f5f92800700 1 -- 172.21.15.89:0/635565015 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5f940b6990 msgr2=0x7f5f940c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.761+0000 7f5f92800700 1 --2- 172.21.15.89:0/635565015 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5f940b6990 0x7f5f940c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.761+0000 7f5f92800700 1 -- 172.21.15.89:0/635565015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f940d05e0 msgr2=0x7f5f941f2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.761+0000 7f5f92800700 1 --2- 172.21.15.89:0/635565015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f940d05e0 0x7f5f941f2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.761+0000 7f5f92800700 1 -- 172.21.15.89:0/635565015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f80009550 con 0x7f5f9415da50 2024-04-02T07:54:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.761+0000 7f5f92800700 1 --2- 172.21.15.89:0/635565015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f9415da50 0x7f5f94160140 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c00efc0 tx=0x7f5f8c00c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.761+0000 7f5f8a000700 1 -- 172.21.15.89:0/635565015 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c00caa0 con 0x7f5f9415da50 2024-04-02T07:54:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.761+0000 7f5f8a000700 1 -- 172.21.15.89:0/635565015 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8c00f040 con 0x7f5f9415da50 2024-04-02T07:54:33.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.761+0000 7f5f8a000700 1 -- 172.21.15.89:0/635565015 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c00ce40 con 0x7f5f9415da50 2024-04-02T07:54:33.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.761+0000 7f5f98c00700 1 -- 172.21.15.89:0/635565015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f941f6a40 con 0x7f5f9415da50 2024-04-02T07:54:33.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.761+0000 7f5f98c00700 1 -- 172.21.15.89:0/635565015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f941f7000 con 0x7f5f9415da50 2024-04-02T07:54:33.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.762+0000 7f5f98c00700 1 -- 172.21.15.89:0/635565015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f941f7520 con 0x7f5f9415da50 2024-04-02T07:54:33.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.763+0000 7f5f8a000700 1 -- 172.21.15.89:0/635565015 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5f8c014480 con 0x7f5f9415da50 2024-04-02T07:54:33.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.764+0000 7f5f8a000700 1 --2- 172.21.15.89:0/635565015 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5f7406adb0 0x7f5f7406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:33.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.764+0000 7f5f8a000700 1 -- 172.21.15.89:0/635565015 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5f8c089f30 con 0x7f5f9415da50 2024-04-02T07:54:33.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.764+0000 7f5f91e00700 1 --2- 172.21.15.89:0/635565015 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5f7406adb0 0x7f5f7406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:33.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.765+0000 7f5f91e00700 1 --2- 172.21.15.89:0/635565015 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5f7406adb0 0x7f5f7406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5f84007fd0 tx=0x7f5f84009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:33.769+0000 7f5f8a000700 1 -- 172.21.15.89:0/635565015 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5f941f7520 con 0x7f5f9415da50 2024-04-02T07:54:34.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.014+0000 7f5f98c00700 1 -- 172.21.15.89:0/635565015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f941f7520 con 0x7f5f9415da50 2024-04-02T07:54:34.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.015+0000 7f5f8a000700 1 -- 172.21.15.89:0/635565015 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f5f941f7520 con 0x7f5f9415da50 2024-04-02T07:54:34.016 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:34.016 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1967870793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1967870793},{"type":"v1","addr":"172.21.15.89:6837","nonce":1967870793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1050024225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1050024225},{"type":"v1","addr":"172.21.15.113:6835","nonce":1050024225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7447,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/235320877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":235320877},{"type":"v1","addr":"172.21.15.113:6834","nonce":235320877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:54:30.522457+0000","modified":"2024-04-02T07:54:33.536401+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/1949161483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1949161483},{"type":"v1","addr":"172.21.15.89:6836","nonce":1949161483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-02T07:54:34.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.017+0000 7f5f98c00700 1 -- 172.21.15.89:0/635565015 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5f7406adb0 msgr2=0x7f5f7406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.017+0000 7f5f98c00700 1 --2- 172.21.15.89:0/635565015 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5f7406adb0 0x7f5f7406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5f84007fd0 tx=0x7f5f84009040 comp rx=0 tx=0).stop 2024-04-02T07:54:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.017+0000 7f5f98c00700 1 -- 172.21.15.89:0/635565015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f9415da50 msgr2=0x7f5f94160140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.017+0000 7f5f98c00700 1 --2- 172.21.15.89:0/635565015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f9415da50 0x7f5f94160140 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c00efc0 tx=0x7f5f8c00c0c0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.017+0000 7f5f98c00700 1 -- 172.21.15.89:0/635565015 shutdown_connections 2024-04-02T07:54:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.017+0000 7f5f98c00700 1 --2- 172.21.15.89:0/635565015 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5f7406adb0 0x7f5f7406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.017+0000 7f5f98c00700 1 --2- 172.21.15.89:0/635565015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f9415da50 0x7f5f94160140 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.017+0000 7f5f98c00700 1 --2- 172.21.15.89:0/635565015 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5f940b6990 0x7f5f940c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.017+0000 7f5f98c00700 1 --2- 172.21.15.89:0/635565015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f940d05e0 0x7f5f941f2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.017+0000 7f5f98c00700 1 -- 172.21.15.89:0/635565015 >> 172.21.15.89:0/635565015 conn(0x7f5f940666a0 msgr2=0x7f5f9415c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.018+0000 7f5f98c00700 1 -- 172.21.15.89:0/635565015 shutdown_connections 2024-04-02T07:54:34.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.018+0000 7f5f98c00700 1 -- 172.21.15.89:0/635565015 wait complete. 2024-04-02T07:54:34.020 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 57 2024-04-02T07:54:34.036 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:54:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.225+0000 7f9ab9800700 1 Processor -- start 2024-04-02T07:54:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.226+0000 7f9ab9800700 1 -- start start 2024-04-02T07:54:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.227+0000 7f9ab9800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab415b5d0 0x7f9ab40ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.227+0000 7f9ab9800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab415fd50 con 0x7f9ab40ac7d0 2024-04-02T07:54:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.227+0000 7f9ab9800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab415f140 con 0x7f9ab40cb3f0 2024-04-02T07:54:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.227+0000 7f9ab9800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab415b9b0 con 0x7f9ab415b5d0 2024-04-02T07:54:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.227+0000 7f9ab3400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9ab40ac7d0 0x7f9ab406c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36718/0 (socket says 172.21.15.89:36718) 2024-04-02T07:54:34.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.227+0000 7f9ab3400700 1 -- 172.21.15.89:0/3256469534 learned_addr learned my addr 172.21.15.89:0/3256469534 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.227+0000 7f9ab2a00700 1 --2- 172.21.15.89:0/3256469534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab415b5d0 0x7f9ab40ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.227+0000 7f9ab2000700 1 -- 172.21.15.89:0/3256469534 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364490684 0 0) 0x7f9ab415fd50 con 0x7f9ab40ac7d0 2024-04-02T07:54:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.227+0000 7f9ab2000700 1 -- 172.21.15.89:0/3256469534 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a980035b0 con 0x7f9ab40ac7d0 2024-04-02T07:54:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab2000700 1 -- 172.21.15.89:0/3256469534 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2575770323 0 0) 0x7f9a980035b0 con 0x7f9ab40ac7d0 2024-04-02T07:54:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab2000700 1 -- 172.21.15.89:0/3256469534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab415b5d0 msgr2=0x7f9ab40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab2000700 1 --2- 172.21.15.89:0/3256469534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab415b5d0 0x7f9ab40ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab2000700 1 -- 172.21.15.89:0/3256469534 >> v1:172.21.15.113:6789/0 conn(0x7f9ab40cb3f0 legacy=0x7f9ab40ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab2000700 1 -- 172.21.15.89:0/3256469534 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab415f420 con 0x7f9ab40ac7d0 2024-04-02T07:54:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab2000700 1 -- 172.21.15.89:0/3256469534 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f9aac0019f0 con 0x7f9ab40ac7d0 2024-04-02T07:54:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab2000700 1 -- 172.21.15.89:0/3256469534 >> v1:172.21.15.89:6789/0 conn(0x7f9ab40ac7d0 legacy=0x7f9ab406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab2000700 1 --2- 172.21.15.89:0/3256469534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a98003a30 0x7f9a98003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab2000700 1 --2- 172.21.15.89:0/3256469534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab415b5d0 0x7f9a98004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab2000700 1 --2- 172.21.15.89:0/3256469534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a98004d90 0x7f9a98005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab2000700 1 -- 172.21.15.89:0/3256469534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab415b9b0 con 0x7f9a98004d90 2024-04-02T07:54:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab2000700 1 -- 172.21.15.89:0/3256469534 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9800bb40 con 0x7f9a98003a30 2024-04-02T07:54:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab2000700 1 -- 172.21.15.89:0/3256469534 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9800bd00 con 0x7f9ab415b5d0 2024-04-02T07:54:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab3400700 1 --2- 172.21.15.89:0/3256469534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a98004d90 0x7f9a98005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab2a00700 1 --2- 172.21.15.89:0/3256469534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a98003a30 0x7f9a98003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9aabe00700 1 --2- 172.21.15.89:0/3256469534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab415b5d0 0x7f9a98004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab3400700 1 -- 172.21.15.89:0/3256469534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab415b5d0 msgr2=0x7f9a98004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab3400700 1 --2- 172.21.15.89:0/3256469534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab415b5d0 0x7f9a98004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab3400700 1 -- 172.21.15.89:0/3256469534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a98003a30 msgr2=0x7f9a98003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab3400700 1 --2- 172.21.15.89:0/3256469534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a98003a30 0x7f9a98003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.228+0000 7f9ab3400700 1 -- 172.21.15.89:0/3256469534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a980035b0 con 0x7f9a98004d90 2024-04-02T07:54:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.229+0000 7f9ab3400700 1 --2- 172.21.15.89:0/3256469534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a98004d90 0x7f9a98005210 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4070970 tx=0x7f9a98005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a9c889a6b4a5d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.229+0000 7f9ab2000700 1 -- 172.21.15.89:0/3256469534 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac00a2f0 con 0x7f9a98004d90 2024-04-02T07:54:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.229+0000 7f9ab2000700 1 -- 172.21.15.89:0/3256469534 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aac012620 con 0x7f9a98004d90 2024-04-02T07:54:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.229+0000 7f9ab2000700 1 -- 172.21.15.89:0/3256469534 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac0129d0 con 0x7f9a98004d90 2024-04-02T07:54:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.229+0000 7f9ab9800700 1 -- 172.21.15.89:0/3256469534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a98004d90 msgr2=0x7f9a98005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.229+0000 7f9ab9800700 1 --2- 172.21.15.89:0/3256469534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a98004d90 0x7f9a98005210 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4070970 tx=0x7f9a98005880 comp rx=0 tx=0).stop 2024-04-02T07:54:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.229+0000 7f9ab3400700 1 -- 172.21.15.89:0/3256469534 reap_dead start 2024-04-02T07:54:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.230+0000 7f9ab9800700 1 -- 172.21.15.89:0/3256469534 shutdown_connections 2024-04-02T07:54:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.230+0000 7f9ab9800700 1 -- 172.21.15.89:0/3256469534 >> 172.21.15.89:0/3256469534 conn(0x7f9ab40cf190 msgr2=0x7f9ab40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.230+0000 7f9ab9800700 1 -- 172.21.15.89:0/3256469534 shutdown_connections 2024-04-02T07:54:34.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.230+0000 7f9ab9800700 1 -- 172.21.15.89:0/3256469534 wait complete. 2024-04-02T07:54:34.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.231+0000 7f9ab9800700 1 Processor -- start 2024-04-02T07:54:34.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.232+0000 7f9ab9800700 1 -- start start 2024-04-02T07:54:34.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.232+0000 7f9ab9800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a98004d90 0x7f9ab4072220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.232+0000 7f9ab9800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ab409c880 0x7f9ab409cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.232+0000 7f9ab9800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab40b2280 0x7f9ab40b2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.232+0000 7f9ab9800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab409d240 con 0x7f9ab40b2280 2024-04-02T07:54:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.232+0000 7f9ab9800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab40b2fb0 con 0x7f9ab409c880 2024-04-02T07:54:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.232+0000 7f9ab9800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab40b3290 con 0x7f9a98004d90 2024-04-02T07:54:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.233+0000 7f9ab3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab40b2280 0x7f9ab40b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.233+0000 7f9ab3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab40b2280 0x7f9ab40b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36054/0 (socket says 172.21.15.89:36054) 2024-04-02T07:54:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.233+0000 7f9ab3400700 1 -- 172.21.15.89:0/761423629 learned_addr learned my addr 172.21.15.89:0/761423629 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.233+0000 7f9ab2a00700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a98004d90 0x7f9ab4072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.233+0000 7f9aabe00700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ab409c880 0x7f9ab409cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.233+0000 7f9ab3400700 1 -- 172.21.15.89:0/761423629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a98004d90 msgr2=0x7f9ab4072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.233+0000 7f9ab3400700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a98004d90 0x7f9ab4072220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.233+0000 7f9ab3400700 1 -- 172.21.15.89:0/761423629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ab409c880 msgr2=0x7f9ab409cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.233+0000 7f9ab3400700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ab409c880 0x7f9ab409cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.233+0000 7f9ab3400700 1 -- 172.21.15.89:0/761423629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab40b3570 con 0x7f9ab40b2280 2024-04-02T07:54:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.233+0000 7f9ab2a00700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a98004d90 0x7f9ab4072220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.233+0000 7f9aabe00700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ab409c880 0x7f9ab409cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.233+0000 7f9ab3400700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab40b2280 0x7f9ab40b2700 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f9aac012000 tx=0x7f9aac001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.234+0000 7f9aaaa00700 1 -- 172.21.15.89:0/761423629 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac00a210 con 0x7f9ab40b2280 2024-04-02T07:54:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.234+0000 7f9aaaa00700 1 -- 172.21.15.89:0/761423629 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aac0017b0 con 0x7f9ab40b2280 2024-04-02T07:54:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.234+0000 7f9ab9800700 1 -- 172.21.15.89:0/761423629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab40b3850 con 0x7f9ab40b2280 2024-04-02T07:54:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.234+0000 7f9aaaa00700 1 -- 172.21.15.89:0/761423629 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac01e430 con 0x7f9ab40b2280 2024-04-02T07:54:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.234+0000 7f9ab9800700 1 -- 172.21.15.89:0/761423629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab40b2c40 con 0x7f9ab40b2280 2024-04-02T07:54:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.235+0000 7f9ab9800700 1 -- 172.21.15.89:0/761423629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab40cba10 con 0x7f9ab40b2280 2024-04-02T07:54:34.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.236+0000 7f9aaaa00700 1 -- 172.21.15.89:0/761423629 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9aac01e5b0 con 0x7f9ab40b2280 2024-04-02T07:54:34.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.237+0000 7f9aaaa00700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a9006ade0 0x7f9a9006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:34.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.237+0000 7f9aaaa00700 1 -- 172.21.15.89:0/761423629 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9aac08b420 con 0x7f9ab40b2280 2024-04-02T07:54:34.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.237+0000 7f9ab2a00700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a9006ade0 0x7f9a9006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.237+0000 7f9ab2a00700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a9006ade0 0x7f9a9006d2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009b50 tx=0x7f9aa4008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.241+0000 7f9aaaa00700 1 -- 172.21.15.89:0/761423629 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9aac056540 con 0x7f9ab40b2280 2024-04-02T07:54:34.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.442+0000 7f9ab9800700 1 -- 172.21.15.89:0/761423629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ab415f9a0 con 0x7f9ab40b2280 2024-04-02T07:54:34.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.443+0000 7f9aaaa00700 1 -- 172.21.15.89:0/761423629 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16578 (secure 0 0 0) 0x7f9aac059b60 con 0x7f9ab40b2280 2024-04-02T07:54:34.445 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:34.445 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":125,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:54:31.525087+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:54:28.130987+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-02T07:54:29.515956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:54:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.446+0000 7f9ab9800700 1 -- 172.21.15.89:0/761423629 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a9006ade0 msgr2=0x7f9a9006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.446+0000 7f9ab9800700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a9006ade0 0x7f9a9006d2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009b50 tx=0x7f9aa4008040 comp rx=0 tx=0).stop 2024-04-02T07:54:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.446+0000 7f9ab9800700 1 -- 172.21.15.89:0/761423629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab40b2280 msgr2=0x7f9ab40b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.446+0000 7f9ab9800700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab40b2280 0x7f9ab40b2700 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f9aac012000 tx=0x7f9aac001d80 comp rx=0 tx=0).stop 2024-04-02T07:54:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.446+0000 7f9ab9800700 1 -- 172.21.15.89:0/761423629 shutdown_connections 2024-04-02T07:54:34.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.446+0000 7f9ab9800700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a9006ade0 0x7f9a9006d2a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.446+0000 7f9ab9800700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab40b2280 0x7f9ab40b2700 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.446+0000 7f9ab9800700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a98004d90 0x7f9ab4072220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.446+0000 7f9ab9800700 1 --2- 172.21.15.89:0/761423629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ab409c880 0x7f9ab409cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.446+0000 7f9ab9800700 1 -- 172.21.15.89:0/761423629 >> 172.21.15.89:0/761423629 conn(0x7f9ab40cf190 msgr2=0x7f9ab406de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:34.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.446+0000 7f9ab9800700 1 -- 172.21.15.89:0/761423629 shutdown_connections 2024-04-02T07:54:34.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.446+0000 7f9ab9800700 1 -- 172.21.15.89:0/761423629 wait complete. 2024-04-02T07:54:34.466 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T07:54:34.466 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:54:34.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.658+0000 7f499fa00700 1 Processor -- start 2024-04-02T07:54:34.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.658+0000 7f499fa00700 1 -- start start 2024-04-02T07:54:34.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.659+0000 7f499fa00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4998160f10 0x7f49980d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:34.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.659+0000 7f499fa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49980cf230 0x7f49980d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:34.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.659+0000 7f499fa00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49980ceda0 con 0x7f4998161590 2024-04-02T07:54:34.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.659+0000 7f499fa00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49980d23d0 con 0x7f4998160f10 2024-04-02T07:54:34.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.659+0000 7f499fa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49981612f0 con 0x7f49980cf230 2024-04-02T07:54:34.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.660+0000 7f499da00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4998161590 0x7f49980b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36722/0 (socket says 172.21.15.89:36722) 2024-04-02T07:54:34.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.660+0000 7f499da00700 1 -- 172.21.15.89:0/3226709956 learned_addr learned my addr 172.21.15.89:0/3226709956 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:34.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.660+0000 7f4997e00700 1 --2- 172.21.15.89:0/3226709956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49980cf230 0x7f49980d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:34.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.660+0000 7f499d000700 1 --2- 172.21.15.89:0/3226709956 >> v2:172.21.15.113:3300/0 conn(0x7f4998160f10 0x7f49980d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:34.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.660+0000 7f4997400700 1 -- 172.21.15.89:0/3226709956 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211815163 0 0) 0x7f49980ceda0 con 0x7f4998161590 2024-04-02T07:54:34.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.660+0000 7f4997400700 1 -- 172.21.15.89:0/3226709956 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49800035b0 con 0x7f4998161590 2024-04-02T07:54:34.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.660+0000 7f4997e00700 1 -- 172.21.15.89:0/3226709956 >> v2:172.21.15.113:3300/0 conn(0x7f4998160f10 msgr2=0x7f49980d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:34.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.660+0000 7f4997e00700 1 --2- 172.21.15.89:0/3226709956 >> v2:172.21.15.113:3300/0 conn(0x7f4998160f10 0x7f49980d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.660+0000 7f4997e00700 1 -- 172.21.15.89:0/3226709956 >> v1:172.21.15.89:6789/0 conn(0x7f4998161590 legacy=0x7f49980b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.660+0000 7f4997e00700 1 -- 172.21.15.89:0/3226709956 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49980ce470 con 0x7f49980cf230 2024-04-02T07:54:34.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.661+0000 7f4997e00700 1 --2- 172.21.15.89:0/3226709956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49980cf230 0x7f49980d1e90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4984009840 tx=0x7f4984009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c40970c8ff74cead server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.661+0000 7f4997400700 1 -- 172.21.15.89:0/3226709956 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4984011830 con 0x7f49980cf230 2024-04-02T07:54:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.661+0000 7f4997400700 1 -- 172.21.15.89:0/3226709956 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4984011990 con 0x7f49980cf230 2024-04-02T07:54:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.661+0000 7f4997400700 1 -- 172.21.15.89:0/3226709956 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4984011d20 con 0x7f49980cf230 2024-04-02T07:54:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.662+0000 7f499fa00700 1 -- 172.21.15.89:0/3226709956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49980cf230 msgr2=0x7f49980d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.662+0000 7f499fa00700 1 --2- 172.21.15.89:0/3226709956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49980cf230 0x7f49980d1e90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4984009840 tx=0x7f4984009b50 comp rx=0 tx=0).stop 2024-04-02T07:54:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.662+0000 7f499fa00700 1 -- 172.21.15.89:0/3226709956 shutdown_connections 2024-04-02T07:54:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.662+0000 7f499fa00700 1 --2- 172.21.15.89:0/3226709956 >> v2:172.21.15.113:3300/0 conn(0x7f4998160f10 0x7f49980d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.662+0000 7f499fa00700 1 --2- 172.21.15.89:0/3226709956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49980cf230 0x7f49980d1e90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.662+0000 7f499fa00700 1 -- 172.21.15.89:0/3226709956 >> 172.21.15.89:0/3226709956 conn(0x7f499808aff0 msgr2=0x7f49980d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.662+0000 7f499fa00700 1 -- 172.21.15.89:0/3226709956 shutdown_connections 2024-04-02T07:54:34.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.662+0000 7f499fa00700 1 -- 172.21.15.89:0/3226709956 wait complete. 2024-04-02T07:54:34.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.664+0000 7f499fa00700 1 Processor -- start 2024-04-02T07:54:34.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.664+0000 7f499fa00700 1 -- start start 2024-04-02T07:54:34.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.665+0000 7f499fa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49980cf230 0x7f4998156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:34.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.665+0000 7f499fa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4998160f10 0x7f49981526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:34.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.665+0000 7f499fa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4998161590 0x7f49980b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:34.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.665+0000 7f499fa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4998152c20 con 0x7f49980cf230 2024-04-02T07:54:34.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.665+0000 7f499fa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499815ea60 con 0x7f4998160f10 2024-04-02T07:54:34.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.665+0000 7f499fa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49980b6e50 con 0x7f4998161590 2024-04-02T07:54:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.665+0000 7f499d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49980cf230 0x7f4998156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.665+0000 7f499d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49980cf230 0x7f4998156120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36058/0 (socket says 172.21.15.89:36058) 2024-04-02T07:54:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.665+0000 7f499d000700 1 -- 172.21.15.89:0/3959135930 learned_addr learned my addr 172.21.15.89:0/3959135930 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.665+0000 7f4997e00700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4998160f10 0x7f49981526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.665+0000 7f499da00700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4998161590 0x7f49980b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.665+0000 7f499d000700 1 -- 172.21.15.89:0/3959135930 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4998161590 msgr2=0x7f49980b6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.665+0000 7f499d000700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4998161590 0x7f49980b6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.665+0000 7f499d000700 1 -- 172.21.15.89:0/3959135930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4998160f10 msgr2=0x7f49981526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.666+0000 7f499d000700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4998160f10 0x7f49981526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.666+0000 7f499d000700 1 -- 172.21.15.89:0/3959135930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49840094a0 con 0x7f49980cf230 2024-04-02T07:54:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.666+0000 7f4997e00700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4998160f10 0x7f49981526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.666+0000 7f499d000700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49980cf230 0x7f4998156120 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f498c00e870 tx=0x7f498c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.666+0000 7f4994c00700 1 -- 172.21.15.89:0/3959135930 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c009590 con 0x7f49980cf230 2024-04-02T07:54:34.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.666+0000 7f4994c00700 1 -- 172.21.15.89:0/3959135930 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c013680 con 0x7f49980cf230 2024-04-02T07:54:34.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.666+0000 7f499fa00700 1 -- 172.21.15.89:0/3959135930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49980b70f0 con 0x7f49980cf230 2024-04-02T07:54:34.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.666+0000 7f4994c00700 1 -- 172.21.15.89:0/3959135930 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c0139f0 con 0x7f49980cf230 2024-04-02T07:54:34.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.666+0000 7f499fa00700 1 -- 172.21.15.89:0/3959135930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499815e6f0 con 0x7f49980cf230 2024-04-02T07:54:34.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.667+0000 7f499fa00700 1 -- 172.21.15.89:0/3959135930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f499815ed40 con 0x7f49980cf230 2024-04-02T07:54:34.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.668+0000 7f4994c00700 1 -- 172.21.15.89:0/3959135930 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f498c013b50 con 0x7f49980cf230 2024-04-02T07:54:34.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.669+0000 7f4994c00700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f497806adb0 0x7f497806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:34.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.669+0000 7f4994c00700 1 -- 172.21.15.89:0/3959135930 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f498c089460 con 0x7f49980cf230 2024-04-02T07:54:34.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.669+0000 7f4997e00700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f497806adb0 0x7f497806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:34.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.670+0000 7f4997e00700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f497806adb0 0x7f497806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4984000c00 tx=0x7f498401c040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:34.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.674+0000 7f4994c00700 1 -- 172.21.15.89:0/3959135930 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f498c0543c0 con 0x7f49980cf230 2024-04-02T07:54:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.873+0000 7f499fa00700 1 -- 172.21.15.89:0/3959135930 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f49980cf6b0 con 0x7f497806adb0 2024-04-02T07:54:34.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.883+0000 7f4994c00700 1 -- 172.21.15.89:0/3959135930 <== mgr.7225 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289718 (secure 0 0 0) 0x7f4984011b80 con 0x7f497806adb0 2024-04-02T07:54:34.888 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.887+0000 7f499fa00700 1 -- 172.21.15.89:0/3959135930 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f497806adb0 msgr2=0x7f497806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.887+0000 7f499fa00700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f497806adb0 0x7f497806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4984000c00 tx=0x7f498401c040 comp rx=0 tx=0).stop 2024-04-02T07:54:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.887+0000 7f499fa00700 1 -- 172.21.15.89:0/3959135930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49980cf230 msgr2=0x7f4998156120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.887+0000 7f499fa00700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49980cf230 0x7f4998156120 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f498c00e870 tx=0x7f498c00eb80 comp rx=0 tx=0).stop 2024-04-02T07:54:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.887+0000 7f499fa00700 1 -- 172.21.15.89:0/3959135930 shutdown_connections 2024-04-02T07:54:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.887+0000 7f499fa00700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f497806adb0 0x7f497806d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.887+0000 7f499fa00700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4998161590 0x7f49980b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.888+0000 7f499fa00700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49980cf230 0x7f4998156120 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.888+0000 7f499fa00700 1 --2- 172.21.15.89:0/3959135930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4998160f10 0x7f49981526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.888+0000 7f499fa00700 1 -- 172.21.15.89:0/3959135930 >> 172.21.15.89:0/3959135930 conn(0x7f499808aff0 msgr2=0x7f49980ca650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.888+0000 7f499fa00700 1 -- 172.21.15.89:0/3959135930 shutdown_connections 2024-04-02T07:54:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:34.888+0000 7f499fa00700 1 -- 172.21.15.89:0/3959135930 wait complete. 2024-04-02T07:54:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:54:34.907 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T07:54:34.288323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":151,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331112,"kb_used_data":13268,"kb_used_omap":12,"kb_used_meta":317683,"kb_avail":754643608,"statfs":{"total":773094113280,"available":772755054592,"internally_reserved":0,"allocated":13586432,"data_stored":7263086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":325307960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.261969"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593956+0000","last_change":"2024-04-02T07:54:30.543010+0000","last_active":"2024-04-02T07:54:31.593956+0000","last_peered":"2024-04-02T07:54:31.593956+0000","last_clean":"2024-04-02T07:54:31.593956+0000","last_became_active":"2024-04-02T07:54:30.542690+0000","last_became_peered":"2024-04-02T07:54:30.542690+0000","last_unstale":"2024-04-02T07:54:31.593956+0000","last_undegraded":"2024-04-02T07:54:31.593956+0000","last_fullsized":"2024-04-02T07:54:31.593956+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:45.999065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593702+0000","last_change":"2024-04-02T07:54:30.555286+0000","last_active":"2024-04-02T07:54:31.593702+0000","last_peered":"2024-04-02T07:54:31.593702+0000","last_clean":"2024-04-02T07:54:31.593702+0000","last_became_active":"2024-04-02T07:54:30.555004+0000","last_became_peered":"2024-04-02T07:54:30.555004+0000","last_unstale":"2024-04-02T07:54:31.593702+0000","last_undegraded":"2024-04-02T07:54:31.593702+0000","last_fullsized":"2024-04-02T07:54:31.593702+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:07:59.484212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539836+0000","last_change":"2024-04-02T07:54:30.558159+0000","last_active":"2024-04-02T07:54:31.539836+0000","last_peered":"2024-04-02T07:54:31.539836+0000","last_clean":"2024-04-02T07:54:31.539836+0000","last_became_active":"2024-04-02T07:54:30.557097+0000","last_became_peered":"2024-04-02T07:54:30.557097+0000","last_unstale":"2024-04-02T07:54:31.539836+0000","last_undegraded":"2024-04-02T07:54:31.539836+0000","last_fullsized":"2024-04-02T07:54:31.539836+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:46.886600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594013+0000","last_change":"2024-04-02T07:54:30.554983+0000","last_active":"2024-04-02T07:54:31.594013+0000","last_peered":"2024-04-02T07:54:31.594013+0000","last_clean":"2024-04-02T07:54:31.594013+0000","last_became_active":"2024-04-02T07:54:30.554691+0000","last_became_peered":"2024-04-02T07:54:30.554691+0000","last_unstale":"2024-04-02T07:54:31.594013+0000","last_undegraded":"2024-04-02T07:54:31.594013+0000","last_fullsized":"2024-04-02T07:54:31.594013+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:04.865135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592302+0000","last_change":"2024-04-02T07:54:30.551458+0000","last_active":"2024-04-02T07:54:31.592302+0000","last_peered":"2024-04-02T07:54:31.592302+0000","last_clean":"2024-04-02T07:54:31.592302+0000","last_became_active":"2024-04-02T07:54:30.551184+0000","last_became_peered":"2024-04-02T07:54:30.551184+0000","last_unstale":"2024-04-02T07:54:31.592302+0000","last_undegraded":"2024-04-02T07:54:31.592302+0000","last_fullsized":"2024-04-02T07:54:31.592302+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:43.308795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592835+0000","last_change":"2024-04-02T07:54:30.555269+0000","last_active":"2024-04-02T07:54:31.592835+0000","last_peered":"2024-04-02T07:54:31.592835+0000","last_clean":"2024-04-02T07:54:31.592835+0000","last_became_active":"2024-04-02T07:54:30.555033+0000","last_became_peered":"2024-04-02T07:54:30.555033+0000","last_unstale":"2024-04-02T07:54:31.592835+0000","last_undegraded":"2024-04-02T07:54:31.592835+0000","last_fullsized":"2024-04-02T07:54:31.592835+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:17:53.275607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592071+0000","last_change":"2024-04-02T07:54:30.539954+0000","last_active":"2024-04-02T07:54:31.592071+0000","last_peered":"2024-04-02T07:54:31.592071+0000","last_clean":"2024-04-02T07:54:31.592071+0000","last_became_active":"2024-04-02T07:54:30.539027+0000","last_became_peered":"2024-04-02T07:54:30.539027+0000","last_unstale":"2024-04-02T07:54:31.592071+0000","last_undegraded":"2024-04-02T07:54:31.592071+0000","last_fullsized":"2024-04-02T07:54:31.592071+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:06:12.744373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539089+0000","last_change":"2024-04-02T07:54:30.551965+0000","last_active":"2024-04-02T07:54:31.539089+0000","last_peered":"2024-04-02T07:54:31.539089+0000","last_clean":"2024-04-02T07:54:31.539089+0000","last_became_active":"2024-04-02T07:54:30.551594+0000","last_became_peered":"2024-04-02T07:54:30.551594+0000","last_unstale":"2024-04-02T07:54:31.539089+0000","last_undegraded":"2024-04-02T07:54:31.539089+0000","last_fullsized":"2024-04-02T07:54:31.539089+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:07:59.484212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592466+0000","last_change":"2024-04-02T07:54:30.547886+0000","last_active":"2024-04-02T07:54:31.592466+0000","last_peered":"2024-04-02T07:54:31.592466+0000","last_clean":"2024-04-02T07:54:31.592466+0000","last_became_active":"2024-04-02T07:54:30.547616+0000","last_became_peered":"2024-04-02T07:54:30.547616+0000","last_unstale":"2024-04-02T07:54:31.592466+0000","last_undegraded":"2024-04-02T07:54:31.592466+0000","last_fullsized":"2024-04-02T07:54:31.592466+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:55:03.097106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592367+0000","last_change":"2024-04-02T07:54:30.545914+0000","last_active":"2024-04-02T07:54:31.592367+0000","last_peered":"2024-04-02T07:54:31.592367+0000","last_clean":"2024-04-02T07:54:31.592367+0000","last_became_active":"2024-04-02T07:54:30.545686+0000","last_became_peered":"2024-04-02T07:54:30.545686+0000","last_unstale":"2024-04-02T07:54:31.592367+0000","last_undegraded":"2024-04-02T07:54:31.592367+0000","last_fullsized":"2024-04-02T07:54:31.592367+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:21.025069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592455+0000","last_change":"2024-04-02T07:54:30.557600+0000","last_active":"2024-04-02T07:54:31.592455+0000","last_peered":"2024-04-02T07:54:31.592455+0000","last_clean":"2024-04-02T07:54:31.592455+0000","last_became_active":"2024-04-02T07:54:30.557320+0000","last_became_peered":"2024-04-02T07:54:30.557320+0000","last_unstale":"2024-04-02T07:54:31.592455+0000","last_undegraded":"2024-04-02T07:54:31.592455+0000","last_fullsized":"2024-04-02T07:54:31.592455+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:49.546265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539335+0000","last_change":"2024-04-02T07:54:30.557533+0000","last_active":"2024-04-02T07:54:31.539335+0000","last_peered":"2024-04-02T07:54:31.539335+0000","last_clean":"2024-04-02T07:54:31.539335+0000","last_became_active":"2024-04-02T07:54:30.557289+0000","last_became_peered":"2024-04-02T07:54:30.557289+0000","last_unstale":"2024-04-02T07:54:31.539335+0000","last_undegraded":"2024-04-02T07:54:31.539335+0000","last_fullsized":"2024-04-02T07:54:31.539335+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:07:59.484212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593010+0000","last_change":"2024-04-02T07:54:30.551440+0000","last_active":"2024-04-02T07:54:31.593010+0000","last_peered":"2024-04-02T07:54:31.593010+0000","last_clean":"2024-04-02T07:54:31.593010+0000","last_became_active":"2024-04-02T07:54:30.551167+0000","last_became_peered":"2024-04-02T07:54:30.551167+0000","last_unstale":"2024-04-02T07:54:31.593010+0000","last_undegraded":"2024-04-02T07:54:31.593010+0000","last_fullsized":"2024-04-02T07:54:31.593010+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:17:18.604476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593433+0000","last_change":"2024-04-02T07:54:30.556468+0000","last_active":"2024-04-02T07:54:31.593433+0000","last_peered":"2024-04-02T07:54:31.593433+0000","last_clean":"2024-04-02T07:54:31.593433+0000","last_became_active":"2024-04-02T07:54:30.556179+0000","last_became_peered":"2024-04-02T07:54:30.556179+0000","last_unstale":"2024-04-02T07:54:31.593433+0000","last_undegraded":"2024-04-02T07:54:31.593433+0000","last_fullsized":"2024-04-02T07:54:31.593433+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:46.886600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593297+0000","last_change":"2024-04-02T07:54:30.552284+0000","last_active":"2024-04-02T07:54:31.593297+0000","last_peered":"2024-04-02T07:54:31.593297+0000","last_clean":"2024-04-02T07:54:31.593297+0000","last_became_active":"2024-04-02T07:54:30.551081+0000","last_became_peered":"2024-04-02T07:54:30.551081+0000","last_unstale":"2024-04-02T07:54:31.593297+0000","last_undegraded":"2024-04-02T07:54:31.593297+0000","last_fullsized":"2024-04-02T07:54:31.593297+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:49:19.684498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.591999+0000","last_change":"2024-04-02T07:54:30.549056+0000","last_active":"2024-04-02T07:54:31.591999+0000","last_peered":"2024-04-02T07:54:31.591999+0000","last_clean":"2024-04-02T07:54:31.591999+0000","last_became_active":"2024-04-02T07:54:30.548815+0000","last_became_peered":"2024-04-02T07:54:30.548815+0000","last_unstale":"2024-04-02T07:54:31.591999+0000","last_undegraded":"2024-04-02T07:54:31.591999+0000","last_fullsized":"2024-04-02T07:54:31.591999+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:27:06.489753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594456+0000","last_change":"2024-04-02T07:54:30.553528+0000","last_active":"2024-04-02T07:54:31.594456+0000","last_peered":"2024-04-02T07:54:31.594456+0000","last_clean":"2024-04-02T07:54:31.594456+0000","last_became_active":"2024-04-02T07:54:30.553196+0000","last_became_peered":"2024-04-02T07:54:30.553196+0000","last_unstale":"2024-04-02T07:54:31.594456+0000","last_undegraded":"2024-04-02T07:54:31.594456+0000","last_fullsized":"2024-04-02T07:54:31.594456+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:25:53.954726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593811+0000","last_change":"2024-04-02T07:54:30.562626+0000","last_active":"2024-04-02T07:54:31.593811+0000","last_peered":"2024-04-02T07:54:31.593811+0000","last_clean":"2024-04-02T07:54:31.593811+0000","last_became_active":"2024-04-02T07:54:30.559567+0000","last_became_peered":"2024-04-02T07:54:30.559567+0000","last_unstale":"2024-04-02T07:54:31.593811+0000","last_undegraded":"2024-04-02T07:54:31.593811+0000","last_fullsized":"2024-04-02T07:54:31.593811+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:54.425891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594034+0000","last_change":"2024-04-02T07:54:30.558683+0000","last_active":"2024-04-02T07:54:31.594034+0000","last_peered":"2024-04-02T07:54:31.594034+0000","last_clean":"2024-04-02T07:54:31.594034+0000","last_became_active":"2024-04-02T07:54:30.556568+0000","last_became_peered":"2024-04-02T07:54:30.556568+0000","last_unstale":"2024-04-02T07:54:31.594034+0000","last_undegraded":"2024-04-02T07:54:31.594034+0000","last_fullsized":"2024-04-02T07:54:31.594034+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:55:56.043668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539097+0000","last_change":"2024-04-02T07:54:30.557143+0000","last_active":"2024-04-02T07:54:31.539097+0000","last_peered":"2024-04-02T07:54:31.539097+0000","last_clean":"2024-04-02T07:54:31.539097+0000","last_became_active":"2024-04-02T07:54:30.556078+0000","last_became_peered":"2024-04-02T07:54:30.556078+0000","last_unstale":"2024-04-02T07:54:31.539097+0000","last_undegraded":"2024-04-02T07:54:31.539097+0000","last_fullsized":"2024-04-02T07:54:31.539097+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:02.505038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539960+0000","last_change":"2024-04-02T07:54:30.550725+0000","last_active":"2024-04-02T07:54:31.539960+0000","last_peered":"2024-04-02T07:54:31.539960+0000","last_clean":"2024-04-02T07:54:31.539960+0000","last_became_active":"2024-04-02T07:54:30.550383+0000","last_became_peered":"2024-04-02T07:54:30.550383+0000","last_unstale":"2024-04-02T07:54:31.539960+0000","last_undegraded":"2024-04-02T07:54:31.539960+0000","last_fullsized":"2024-04-02T07:54:31.539960+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:37:29.009138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539253+0000","last_change":"2024-04-02T07:54:30.546488+0000","last_active":"2024-04-02T07:54:31.539253+0000","last_peered":"2024-04-02T07:54:31.539253+0000","last_clean":"2024-04-02T07:54:31.539253+0000","last_became_active":"2024-04-02T07:54:30.543130+0000","last_became_peered":"2024-04-02T07:54:30.543130+0000","last_unstale":"2024-04-02T07:54:31.539253+0000","last_undegraded":"2024-04-02T07:54:31.539253+0000","last_fullsized":"2024-04-02T07:54:31.539253+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:04.865135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593387+0000","last_change":"2024-04-02T07:54:30.557275+0000","last_active":"2024-04-02T07:54:31.593387+0000","last_peered":"2024-04-02T07:54:31.593387+0000","last_clean":"2024-04-02T07:54:31.593387+0000","last_became_active":"2024-04-02T07:54:30.556925+0000","last_became_peered":"2024-04-02T07:54:30.556925+0000","last_unstale":"2024-04-02T07:54:31.593387+0000","last_undegraded":"2024-04-02T07:54:31.593387+0000","last_fullsized":"2024-04-02T07:54:31.593387+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:23:38.420697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.537549+0000","last_change":"2024-04-02T07:54:30.556437+0000","last_active":"2024-04-02T07:54:31.537549+0000","last_peered":"2024-04-02T07:54:31.537549+0000","last_clean":"2024-04-02T07:54:31.537549+0000","last_became_active":"2024-04-02T07:54:30.555167+0000","last_became_peered":"2024-04-02T07:54:30.555167+0000","last_unstale":"2024-04-02T07:54:31.537549+0000","last_undegraded":"2024-04-02T07:54:31.537549+0000","last_fullsized":"2024-04-02T07:54:31.537549+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:01:03.970262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592606+0000","last_change":"2024-04-02T07:54:30.544288+0000","last_active":"2024-04-02T07:54:31.592606+0000","last_peered":"2024-04-02T07:54:31.592606+0000","last_clean":"2024-04-02T07:54:31.592606+0000","last_became_active":"2024-04-02T07:54:30.544053+0000","last_became_peered":"2024-04-02T07:54:30.544053+0000","last_unstale":"2024-04-02T07:54:31.592606+0000","last_undegraded":"2024-04-02T07:54:31.592606+0000","last_fullsized":"2024-04-02T07:54:31.592606+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:40:39.992792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593418+0000","last_change":"2024-04-02T07:54:30.548292+0000","last_active":"2024-04-02T07:54:31.593418+0000","last_peered":"2024-04-02T07:54:31.593418+0000","last_clean":"2024-04-02T07:54:31.593418+0000","last_became_active":"2024-04-02T07:54:30.546897+0000","last_became_peered":"2024-04-02T07:54:30.546897+0000","last_unstale":"2024-04-02T07:54:31.593418+0000","last_undegraded":"2024-04-02T07:54:31.593418+0000","last_fullsized":"2024-04-02T07:54:31.593418+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:05.024378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538421+0000","last_change":"2024-04-02T07:54:30.548044+0000","last_active":"2024-04-02T07:54:31.538421+0000","last_peered":"2024-04-02T07:54:31.538421+0000","last_clean":"2024-04-02T07:54:31.538421+0000","last_became_active":"2024-04-02T07:54:30.547763+0000","last_became_peered":"2024-04-02T07:54:30.547763+0000","last_unstale":"2024-04-02T07:54:31.538421+0000","last_undegraded":"2024-04-02T07:54:31.538421+0000","last_fullsized":"2024-04-02T07:54:31.538421+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:07:59.484212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538212+0000","last_change":"2024-04-02T07:54:30.549092+0000","last_active":"2024-04-02T07:54:31.538212+0000","last_peered":"2024-04-02T07:54:31.538212+0000","last_clean":"2024-04-02T07:54:31.538212+0000","last_became_active":"2024-04-02T07:54:30.548099+0000","last_became_peered":"2024-04-02T07:54:30.548099+0000","last_unstale":"2024-04-02T07:54:31.538212+0000","last_undegraded":"2024-04-02T07:54:31.538212+0000","last_fullsized":"2024-04-02T07:54:31.538212+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:04.865135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593129+0000","last_change":"2024-04-02T07:54:30.548114+0000","last_active":"2024-04-02T07:54:31.593129+0000","last_peered":"2024-04-02T07:54:31.593129+0000","last_clean":"2024-04-02T07:54:31.593129+0000","last_became_active":"2024-04-02T07:54:30.547551+0000","last_became_peered":"2024-04-02T07:54:30.547551+0000","last_unstale":"2024-04-02T07:54:31.593129+0000","last_undegraded":"2024-04-02T07:54:31.593129+0000","last_fullsized":"2024-04-02T07:54:31.593129+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:34:44.239761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592859+0000","last_change":"2024-04-02T07:54:30.550274+0000","last_active":"2024-04-02T07:54:31.592859+0000","last_peered":"2024-04-02T07:54:31.592859+0000","last_clean":"2024-04-02T07:54:31.592859+0000","last_became_active":"2024-04-02T07:54:30.549924+0000","last_became_peered":"2024-04-02T07:54:30.549924+0000","last_unstale":"2024-04-02T07:54:31.592859+0000","last_undegraded":"2024-04-02T07:54:31.592859+0000","last_fullsized":"2024-04-02T07:54:31.592859+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:42:05.610304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539490+0000","last_change":"2024-04-02T07:54:30.545185+0000","last_active":"2024-04-02T07:54:31.539490+0000","last_peered":"2024-04-02T07:54:31.539490+0000","last_clean":"2024-04-02T07:54:31.539490+0000","last_became_active":"2024-04-02T07:54:30.544265+0000","last_became_peered":"2024-04-02T07:54:30.544265+0000","last_unstale":"2024-04-02T07:54:31.539490+0000","last_undegraded":"2024-04-02T07:54:31.539490+0000","last_fullsized":"2024-04-02T07:54:31.539490+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:59:37.327187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594119+0000","last_change":"2024-04-02T07:54:30.553233+0000","last_active":"2024-04-02T07:54:31.594119+0000","last_peered":"2024-04-02T07:54:31.594119+0000","last_clean":"2024-04-02T07:54:31.594119+0000","last_became_active":"2024-04-02T07:54:30.552587+0000","last_became_peered":"2024-04-02T07:54:30.552587+0000","last_unstale":"2024-04-02T07:54:31.594119+0000","last_undegraded":"2024-04-02T07:54:31.594119+0000","last_fullsized":"2024-04-02T07:54:31.594119+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:02.505038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594338+0000","last_change":"2024-04-02T07:54:30.563460+0000","last_active":"2024-04-02T07:54:31.594338+0000","last_peered":"2024-04-02T07:54:31.594338+0000","last_clean":"2024-04-02T07:54:31.594338+0000","last_became_active":"2024-04-02T07:54:30.562337+0000","last_became_peered":"2024-04-02T07:54:30.562337+0000","last_unstale":"2024-04-02T07:54:31.594338+0000","last_undegraded":"2024-04-02T07:54:31.594338+0000","last_fullsized":"2024-04-02T07:54:31.594338+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:26.433783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1f","version":"124'4","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593294+0000","last_change":"2024-04-02T07:54:30.050477+0000","last_active":"2024-04-02T07:54:31.593294+0000","last_peered":"2024-04-02T07:54:31.593294+0000","last_clean":"2024-04-02T07:54:31.593294+0000","last_became_active":"2024-04-02T07:54:30.050163+0000","last_became_peered":"2024-04-02T07:54:30.050163+0000","last_unstale":"2024-04-02T07:54:31.593294+0000","last_undegraded":"2024-04-02T07:54:31.593294+0000","last_fullsized":"2024-04-02T07:54:31.593294+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:17.245349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593191+0000","last_change":"2024-04-02T07:54:30.547234+0000","last_active":"2024-04-02T07:54:31.593191+0000","last_peered":"2024-04-02T07:54:31.593191+0000","last_clean":"2024-04-02T07:54:31.593191+0000","last_became_active":"2024-04-02T07:54:30.546928+0000","last_became_peered":"2024-04-02T07:54:30.546928+0000","last_unstale":"2024-04-02T07:54:31.593191+0000","last_undegraded":"2024-04-02T07:54:31.593191+0000","last_fullsized":"2024-04-02T07:54:31.593191+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:45:59.236866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1e","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592362+0000","last_change":"2024-04-02T07:54:30.052507+0000","last_active":"2024-04-02T07:54:31.592362+0000","last_peered":"2024-04-02T07:54:31.592362+0000","last_clean":"2024-04-02T07:54:31.592362+0000","last_became_active":"2024-04-02T07:54:30.052239+0000","last_became_peered":"2024-04-02T07:54:30.052239+0000","last_unstale":"2024-04-02T07:54:31.592362+0000","last_undegraded":"2024-04-02T07:54:31.592362+0000","last_fullsized":"2024-04-02T07:54:31.592362+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:19:33.356465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594027+0000","last_change":"2024-04-02T07:54:30.547257+0000","last_active":"2024-04-02T07:54:31.594027+0000","last_peered":"2024-04-02T07:54:31.594027+0000","last_clean":"2024-04-02T07:54:31.594027+0000","last_became_active":"2024-04-02T07:54:30.542861+0000","last_became_peered":"2024-04-02T07:54:30.542861+0000","last_unstale":"2024-04-02T07:54:31.594027+0000","last_undegraded":"2024-04-02T07:54:31.594027+0000","last_fullsized":"2024-04-02T07:54:31.594027+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:04:38.775670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539834+0000","last_change":"2024-04-02T07:54:30.056814+0000","last_active":"2024-04-02T07:54:31.539834+0000","last_peered":"2024-04-02T07:54:31.539834+0000","last_clean":"2024-04-02T07:54:31.539834+0000","last_became_active":"2024-04-02T07:54:30.056564+0000","last_became_peered":"2024-04-02T07:54:30.056564+0000","last_unstale":"2024-04-02T07:54:31.539834+0000","last_undegraded":"2024-04-02T07:54:31.539834+0000","last_fullsized":"2024-04-02T07:54:31.539834+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:35.636951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592148+0000","last_change":"2024-04-02T07:54:30.545754+0000","last_active":"2024-04-02T07:54:31.592148+0000","last_peered":"2024-04-02T07:54:31.592148+0000","last_clean":"2024-04-02T07:54:31.592148+0000","last_became_active":"2024-04-02T07:54:30.545507+0000","last_became_peered":"2024-04-02T07:54:30.545507+0000","last_unstale":"2024-04-02T07:54:31.592148+0000","last_undegraded":"2024-04-02T07:54:31.592148+0000","last_fullsized":"2024-04-02T07:54:31.592148+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:26:33.418627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594119+0000","last_change":"2024-04-02T07:54:30.060771+0000","last_active":"2024-04-02T07:54:31.594119+0000","last_peered":"2024-04-02T07:54:31.594119+0000","last_clean":"2024-04-02T07:54:31.594119+0000","last_became_active":"2024-04-02T07:54:30.060435+0000","last_became_peered":"2024-04-02T07:54:30.060435+0000","last_unstale":"2024-04-02T07:54:31.594119+0000","last_undegraded":"2024-04-02T07:54:31.594119+0000","last_fullsized":"2024-04-02T07:54:31.594119+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:33:07.449281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538928+0000","last_change":"2024-04-02T07:54:30.549044+0000","last_active":"2024-04-02T07:54:31.538928+0000","last_peered":"2024-04-02T07:54:31.538928+0000","last_clean":"2024-04-02T07:54:31.538928+0000","last_became_active":"2024-04-02T07:54:30.548799+0000","last_became_peered":"2024-04-02T07:54:30.548799+0000","last_unstale":"2024-04-02T07:54:31.538928+0000","last_undegraded":"2024-04-02T07:54:31.538928+0000","last_fullsized":"2024-04-02T07:54:31.538928+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:54:19.338399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1b","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593338+0000","last_change":"2024-04-02T07:54:30.053735+0000","last_active":"2024-04-02T07:54:31.593338+0000","last_peered":"2024-04-02T07:54:31.593338+0000","last_clean":"2024-04-02T07:54:31.593338+0000","last_became_active":"2024-04-02T07:54:30.050754+0000","last_became_peered":"2024-04-02T07:54:30.050754+0000","last_unstale":"2024-04-02T07:54:31.593338+0000","last_undegraded":"2024-04-02T07:54:31.593338+0000","last_fullsized":"2024-04-02T07:54:31.593338+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:17.245349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592473+0000","last_change":"2024-04-02T07:54:30.555339+0000","last_active":"2024-04-02T07:54:31.592473+0000","last_peered":"2024-04-02T07:54:31.592473+0000","last_clean":"2024-04-02T07:54:31.592473+0000","last_became_active":"2024-04-02T07:54:30.555072+0000","last_became_peered":"2024-04-02T07:54:30.555072+0000","last_unstale":"2024-04-02T07:54:31.592473+0000","last_undegraded":"2024-04-02T07:54:31.592473+0000","last_fullsized":"2024-04-02T07:54:31.592473+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:08:21.920480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1a","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539348+0000","last_change":"2024-04-02T07:54:30.050501+0000","last_active":"2024-04-02T07:54:31.539348+0000","last_peered":"2024-04-02T07:54:31.539348+0000","last_clean":"2024-04-02T07:54:31.539348+0000","last_became_active":"2024-04-02T07:54:30.050225+0000","last_became_peered":"2024-04-02T07:54:30.050225+0000","last_unstale":"2024-04-02T07:54:31.539348+0000","last_undegraded":"2024-04-02T07:54:31.539348+0000","last_fullsized":"2024-04-02T07:54:31.539348+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:00:49.879036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592528+0000","last_change":"2024-04-02T07:54:30.561628+0000","last_active":"2024-04-02T07:54:31.592528+0000","last_peered":"2024-04-02T07:54:31.592528+0000","last_clean":"2024-04-02T07:54:31.592528+0000","last_became_active":"2024-04-02T07:54:30.561355+0000","last_became_peered":"2024-04-02T07:54:30.561355+0000","last_unstale":"2024-04-02T07:54:31.592528+0000","last_undegraded":"2024-04-02T07:54:31.592528+0000","last_fullsized":"2024-04-02T07:54:31.592528+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:47.419730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592149+0000","last_change":"2024-04-02T07:54:30.043077+0000","last_active":"2024-04-02T07:54:31.592149+0000","last_peered":"2024-04-02T07:54:31.592149+0000","last_clean":"2024-04-02T07:54:31.592149+0000","last_became_active":"2024-04-02T07:54:30.042794+0000","last_became_peered":"2024-04-02T07:54:30.042794+0000","last_unstale":"2024-04-02T07:54:31.592149+0000","last_undegraded":"2024-04-02T07:54:31.592149+0000","last_fullsized":"2024-04-02T07:54:31.592149+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:25.427335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538803+0000","last_change":"2024-04-02T07:54:30.544448+0000","last_active":"2024-04-02T07:54:31.538803+0000","last_peered":"2024-04-02T07:54:31.538803+0000","last_clean":"2024-04-02T07:54:31.538803+0000","last_became_active":"2024-04-02T07:54:30.543985+0000","last_became_peered":"2024-04-02T07:54:30.543985+0000","last_unstale":"2024-04-02T07:54:31.538803+0000","last_undegraded":"2024-04-02T07:54:31.538803+0000","last_fullsized":"2024-04-02T07:54:31.538803+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:04.865135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593676+0000","last_change":"2024-04-02T07:54:30.054090+0000","last_active":"2024-04-02T07:54:31.593676+0000","last_peered":"2024-04-02T07:54:31.593676+0000","last_clean":"2024-04-02T07:54:31.593676+0000","last_became_active":"2024-04-02T07:54:30.053814+0000","last_became_peered":"2024-04-02T07:54:30.053814+0000","last_unstale":"2024-04-02T07:54:31.593676+0000","last_undegraded":"2024-04-02T07:54:31.593676+0000","last_fullsized":"2024-04-02T07:54:31.593676+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:45:31.896403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593888+0000","last_change":"2024-04-02T07:54:30.555846+0000","last_active":"2024-04-02T07:54:31.593888+0000","last_peered":"2024-04-02T07:54:31.593888+0000","last_clean":"2024-04-02T07:54:31.593888+0000","last_became_active":"2024-04-02T07:54:30.555556+0000","last_became_peered":"2024-04-02T07:54:30.555556+0000","last_unstale":"2024-04-02T07:54:31.593888+0000","last_undegraded":"2024-04-02T07:54:31.593888+0000","last_fullsized":"2024-04-02T07:54:31.593888+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:37:29.009138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.540066+0000","last_change":"2024-04-02T07:54:30.046428+0000","last_active":"2024-04-02T07:54:31.540066+0000","last_peered":"2024-04-02T07:54:31.540066+0000","last_clean":"2024-04-02T07:54:31.540066+0000","last_became_active":"2024-04-02T07:54:30.045802+0000","last_became_peered":"2024-04-02T07:54:30.045802+0000","last_unstale":"2024-04-02T07:54:31.540066+0000","last_undegraded":"2024-04-02T07:54:31.540066+0000","last_fullsized":"2024-04-02T07:54:31.540066+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:33:07.449281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538913+0000","last_change":"2024-04-02T07:54:30.541154+0000","last_active":"2024-04-02T07:54:31.538913+0000","last_peered":"2024-04-02T07:54:31.538913+0000","last_clean":"2024-04-02T07:54:31.538913+0000","last_became_active":"2024-04-02T07:54:30.540786+0000","last_became_peered":"2024-04-02T07:54:30.540786+0000","last_unstale":"2024-04-02T07:54:31.538913+0000","last_undegraded":"2024-04-02T07:54:31.538913+0000","last_fullsized":"2024-04-02T07:54:31.538913+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:04:38.775670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593890+0000","last_change":"2024-04-02T07:54:30.056988+0000","last_active":"2024-04-02T07:54:31.593890+0000","last_peered":"2024-04-02T07:54:31.593890+0000","last_clean":"2024-04-02T07:54:31.593890+0000","last_became_active":"2024-04-02T07:54:30.056721+0000","last_became_peered":"2024-04-02T07:54:30.056721+0000","last_unstale":"2024-04-02T07:54:31.593890+0000","last_undegraded":"2024-04-02T07:54:31.593890+0000","last_fullsized":"2024-04-02T07:54:31.593890+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:52.812327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592529+0000","last_change":"2024-04-02T07:54:30.546859+0000","last_active":"2024-04-02T07:54:31.592529+0000","last_peered":"2024-04-02T07:54:31.592529+0000","last_clean":"2024-04-02T07:54:31.592529+0000","last_became_active":"2024-04-02T07:54:30.544420+0000","last_became_peered":"2024-04-02T07:54:30.544420+0000","last_unstale":"2024-04-02T07:54:31.592529+0000","last_undegraded":"2024-04-02T07:54:31.592529+0000","last_fullsized":"2024-04-02T07:54:31.592529+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:50:40.545354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.15","version":"124'3","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592546+0000","last_change":"2024-04-02T07:54:30.047690+0000","last_active":"2024-04-02T07:54:31.592546+0000","last_peered":"2024-04-02T07:54:31.592546+0000","last_clean":"2024-04-02T07:54:31.592546+0000","last_became_active":"2024-04-02T07:54:30.047423+0000","last_became_peered":"2024-04-02T07:54:30.047423+0000","last_unstale":"2024-04-02T07:54:31.592546+0000","last_undegraded":"2024-04-02T07:54:31.592546+0000","last_fullsized":"2024-04-02T07:54:31.592546+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:44.349638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538433+0000","last_change":"2024-04-02T07:54:30.546888+0000","last_active":"2024-04-02T07:54:31.538433+0000","last_peered":"2024-04-02T07:54:31.538433+0000","last_clean":"2024-04-02T07:54:31.538433+0000","last_became_active":"2024-04-02T07:54:30.545311+0000","last_became_peered":"2024-04-02T07:54:30.545311+0000","last_unstale":"2024-04-02T07:54:31.538433+0000","last_undegraded":"2024-04-02T07:54:31.538433+0000","last_fullsized":"2024-04-02T07:54:31.538433+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:46.886600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.14","version":"124'6","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539287+0000","last_change":"2024-04-02T07:54:30.052792+0000","last_active":"2024-04-02T07:54:31.539287+0000","last_peered":"2024-04-02T07:54:31.539287+0000","last_clean":"2024-04-02T07:54:31.539287+0000","last_became_active":"2024-04-02T07:54:30.052544+0000","last_became_peered":"2024-04-02T07:54:30.052544+0000","last_unstale":"2024-04-02T07:54:31.539287+0000","last_undegraded":"2024-04-02T07:54:31.539287+0000","last_fullsized":"2024-04-02T07:54:31.539287+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:34:43.233313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594015+0000","last_change":"2024-04-02T07:54:30.544160+0000","last_active":"2024-04-02T07:54:31.594015+0000","last_peered":"2024-04-02T07:54:31.594015+0000","last_clean":"2024-04-02T07:54:31.594015+0000","last_became_active":"2024-04-02T07:54:30.543902+0000","last_became_peered":"2024-04-02T07:54:30.543902+0000","last_unstale":"2024-04-02T07:54:31.594015+0000","last_undegraded":"2024-04-02T07:54:31.594015+0000","last_fullsized":"2024-04-02T07:54:31.594015+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:59:37.327187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.13","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593607+0000","last_change":"2024-04-02T07:54:30.053291+0000","last_active":"2024-04-02T07:54:31.593607+0000","last_peered":"2024-04-02T07:54:31.593607+0000","last_clean":"2024-04-02T07:54:31.593607+0000","last_became_active":"2024-04-02T07:54:30.052011+0000","last_became_peered":"2024-04-02T07:54:30.052011+0000","last_unstale":"2024-04-02T07:54:31.593607+0000","last_undegraded":"2024-04-02T07:54:31.593607+0000","last_fullsized":"2024-04-02T07:54:31.593607+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:52:21.754612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539925+0000","last_change":"2024-04-02T07:54:30.550714+0000","last_active":"2024-04-02T07:54:31.539925+0000","last_peered":"2024-04-02T07:54:31.539925+0000","last_clean":"2024-04-02T07:54:31.539925+0000","last_became_active":"2024-04-02T07:54:30.549286+0000","last_became_peered":"2024-04-02T07:54:30.549286+0000","last_unstale":"2024-04-02T07:54:31.539925+0000","last_undegraded":"2024-04-02T07:54:31.539925+0000","last_fullsized":"2024-04-02T07:54:31.539925+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:54:19.338399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594116+0000","last_change":"2024-04-02T07:54:30.059972+0000","last_active":"2024-04-02T07:54:31.594116+0000","last_peered":"2024-04-02T07:54:31.594116+0000","last_clean":"2024-04-02T07:54:31.594116+0000","last_became_active":"2024-04-02T07:54:30.059250+0000","last_became_peered":"2024-04-02T07:54:30.059250+0000","last_unstale":"2024-04-02T07:54:31.594116+0000","last_undegraded":"2024-04-02T07:54:31.594116+0000","last_fullsized":"2024-04-02T07:54:31.594116+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:54.278160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592074+0000","last_change":"2024-04-02T07:54:30.550612+0000","last_active":"2024-04-02T07:54:31.592074+0000","last_peered":"2024-04-02T07:54:31.592074+0000","last_clean":"2024-04-02T07:54:31.592074+0000","last_became_active":"2024-04-02T07:54:30.550355+0000","last_became_peered":"2024-04-02T07:54:30.550355+0000","last_unstale":"2024-04-02T07:54:31.592074+0000","last_undegraded":"2024-04-02T07:54:31.592074+0000","last_fullsized":"2024-04-02T07:54:31.592074+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:06:21.694201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538892+0000","last_change":"2024-04-02T07:54:30.045344+0000","last_active":"2024-04-02T07:54:31.538892+0000","last_peered":"2024-04-02T07:54:31.538892+0000","last_clean":"2024-04-02T07:54:31.538892+0000","last_became_active":"2024-04-02T07:54:30.045114+0000","last_became_peered":"2024-04-02T07:54:30.045114+0000","last_unstale":"2024-04-02T07:54:31.538892+0000","last_undegraded":"2024-04-02T07:54:31.538892+0000","last_fullsized":"2024-04-02T07:54:31.538892+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:04.017930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539538+0000","last_change":"2024-04-02T07:54:30.545113+0000","last_active":"2024-04-02T07:54:31.539538+0000","last_peered":"2024-04-02T07:54:31.539538+0000","last_clean":"2024-04-02T07:54:31.539538+0000","last_became_active":"2024-04-02T07:54:30.543617+0000","last_became_peered":"2024-04-02T07:54:30.543617+0000","last_unstale":"2024-04-02T07:54:31.539538+0000","last_undegraded":"2024-04-02T07:54:31.539538+0000","last_fullsized":"2024-04-02T07:54:31.539538+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:02.505038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.10","version":"124'8","reported_seq":19,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593871+0000","last_change":"2024-04-02T07:54:30.055148+0000","last_active":"2024-04-02T07:54:31.593871+0000","last_peered":"2024-04-02T07:54:31.593871+0000","last_clean":"2024-04-02T07:54:31.593871+0000","last_became_active":"2024-04-02T07:54:30.054904+0000","last_became_peered":"2024-04-02T07:54:30.054904+0000","last_unstale":"2024-04-02T07:54:31.593871+0000","last_undegraded":"2024-04-02T07:54:31.593871+0000","last_fullsized":"2024-04-02T07:54:31.593871+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:16:23.501686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539064+0000","last_change":"2024-04-02T07:54:30.551645+0000","last_active":"2024-04-02T07:54:31.539064+0000","last_peered":"2024-04-02T07:54:31.539064+0000","last_clean":"2024-04-02T07:54:31.539064+0000","last_became_active":"2024-04-02T07:54:30.551365+0000","last_became_peered":"2024-04-02T07:54:30.551365+0000","last_unstale":"2024-04-02T07:54:31.539064+0000","last_undegraded":"2024-04-02T07:54:31.539064+0000","last_fullsized":"2024-04-02T07:54:31.539064+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:02.505038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.e","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539318+0000","last_change":"2024-04-02T07:54:30.053094+0000","last_active":"2024-04-02T07:54:31.539318+0000","last_peered":"2024-04-02T07:54:31.539318+0000","last_clean":"2024-04-02T07:54:31.539318+0000","last_became_active":"2024-04-02T07:54:30.052016+0000","last_became_peered":"2024-04-02T07:54:30.052016+0000","last_unstale":"2024-04-02T07:54:31.539318+0000","last_undegraded":"2024-04-02T07:54:31.539318+0000","last_fullsized":"2024-04-02T07:54:31.539318+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:46.413282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538261+0000","last_change":"2024-04-02T07:54:30.558623+0000","last_active":"2024-04-02T07:54:31.538261+0000","last_peered":"2024-04-02T07:54:31.538261+0000","last_clean":"2024-04-02T07:54:31.538261+0000","last_became_active":"2024-04-02T07:54:30.556706+0000","last_became_peered":"2024-04-02T07:54:30.556706+0000","last_unstale":"2024-04-02T07:54:31.538261+0000","last_undegraded":"2024-04-02T07:54:31.538261+0000","last_fullsized":"2024-04-02T07:54:31.538261+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:54:19.338399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.d","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592740+0000","last_change":"2024-04-02T07:54:30.049156+0000","last_active":"2024-04-02T07:54:31.592740+0000","last_peered":"2024-04-02T07:54:31.592740+0000","last_clean":"2024-04-02T07:54:31.592740+0000","last_became_active":"2024-04-02T07:54:30.048701+0000","last_became_peered":"2024-04-02T07:54:30.048701+0000","last_unstale":"2024-04-02T07:54:31.592740+0000","last_undegraded":"2024-04-02T07:54:31.592740+0000","last_fullsized":"2024-04-02T07:54:31.592740+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:55:55.037220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539225+0000","last_change":"2024-04-02T07:54:30.553351+0000","last_active":"2024-04-02T07:54:31.539225+0000","last_peered":"2024-04-02T07:54:31.539225+0000","last_clean":"2024-04-02T07:54:31.539225+0000","last_became_active":"2024-04-02T07:54:30.552907+0000","last_became_peered":"2024-04-02T07:54:30.552907+0000","last_unstale":"2024-04-02T07:54:31.539225+0000","last_undegraded":"2024-04-02T07:54:31.539225+0000","last_fullsized":"2024-04-02T07:54:31.539225+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:04:38.775670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593783+0000","last_change":"2024-04-02T07:54:30.052611+0000","last_active":"2024-04-02T07:54:31.593783+0000","last_peered":"2024-04-02T07:54:31.593783+0000","last_clean":"2024-04-02T07:54:31.593783+0000","last_became_active":"2024-04-02T07:54:30.052373+0000","last_became_peered":"2024-04-02T07:54:30.052373+0000","last_unstale":"2024-04-02T07:54:31.593783+0000","last_undegraded":"2024-04-02T07:54:31.593783+0000","last_fullsized":"2024-04-02T07:54:31.593783+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:29:03.725237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592786+0000","last_change":"2024-04-02T07:54:30.551933+0000","last_active":"2024-04-02T07:54:31.592786+0000","last_peered":"2024-04-02T07:54:31.592786+0000","last_clean":"2024-04-02T07:54:31.592786+0000","last_became_active":"2024-04-02T07:54:30.551519+0000","last_became_peered":"2024-04-02T07:54:30.551519+0000","last_unstale":"2024-04-02T07:54:31.592786+0000","last_undegraded":"2024-04-02T07:54:31.592786+0000","last_fullsized":"2024-04-02T07:54:31.592786+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:02:19.300367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593436+0000","last_change":"2024-04-02T07:54:30.060771+0000","last_active":"2024-04-02T07:54:31.593436+0000","last_peered":"2024-04-02T07:54:31.593436+0000","last_clean":"2024-04-02T07:54:31.593436+0000","last_became_active":"2024-04-02T07:54:30.060518+0000","last_became_peered":"2024-04-02T07:54:30.060518+0000","last_unstale":"2024-04-02T07:54:31.593436+0000","last_undegraded":"2024-04-02T07:54:31.593436+0000","last_fullsized":"2024-04-02T07:54:31.593436+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:35:18.551498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592845+0000","last_change":"2024-04-02T07:54:30.554540+0000","last_active":"2024-04-02T07:54:31.592845+0000","last_peered":"2024-04-02T07:54:31.592845+0000","last_clean":"2024-04-02T07:54:31.592845+0000","last_became_active":"2024-04-02T07:54:30.554239+0000","last_became_peered":"2024-04-02T07:54:30.554239+0000","last_unstale":"2024-04-02T07:54:31.592845+0000","last_undegraded":"2024-04-02T07:54:31.592845+0000","last_fullsized":"2024-04-02T07:54:31.592845+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:45.356086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.537968+0000","last_change":"2024-04-02T07:54:30.045190+0000","last_active":"2024-04-02T07:54:31.537968+0000","last_peered":"2024-04-02T07:54:31.537968+0000","last_clean":"2024-04-02T07:54:31.537968+0000","last_became_active":"2024-04-02T07:54:30.044075+0000","last_became_peered":"2024-04-02T07:54:30.044075+0000","last_unstale":"2024-04-02T07:54:31.537968+0000","last_undegraded":"2024-04-02T07:54:31.537968+0000","last_fullsized":"2024-04-02T07:54:31.537968+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:04.017930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593692+0000","last_change":"2024-04-02T07:54:30.553411+0000","last_active":"2024-04-02T07:54:31.593692+0000","last_peered":"2024-04-02T07:54:31.593692+0000","last_clean":"2024-04-02T07:54:31.593692+0000","last_became_active":"2024-04-02T07:54:30.552913+0000","last_became_peered":"2024-04-02T07:54:30.552913+0000","last_unstale":"2024-04-02T07:54:31.593692+0000","last_undegraded":"2024-04-02T07:54:31.593692+0000","last_fullsized":"2024-04-02T07:54:31.593692+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:58:10.742469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592224+0000","last_change":"2024-04-02T07:54:30.043594+0000","last_active":"2024-04-02T07:54:31.592224+0000","last_peered":"2024-04-02T07:54:31.592224+0000","last_clean":"2024-04-02T07:54:31.592224+0000","last_became_active":"2024-04-02T07:54:30.043350+0000","last_became_peered":"2024-04-02T07:54:30.043350+0000","last_unstale":"2024-04-02T07:54:31.592224+0000","last_undegraded":"2024-04-02T07:54:31.592224+0000","last_fullsized":"2024-04-02T07:54:31.592224+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:58:42.614095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.537732+0000","last_change":"2024-04-02T07:54:30.542011+0000","last_active":"2024-04-02T07:54:31.537732+0000","last_peered":"2024-04-02T07:54:31.537732+0000","last_clean":"2024-04-02T07:54:31.537732+0000","last_became_active":"2024-04-02T07:54:30.541726+0000","last_became_peered":"2024-04-02T07:54:30.541726+0000","last_unstale":"2024-04-02T07:54:31.537732+0000","last_undegraded":"2024-04-02T07:54:31.537732+0000","last_fullsized":"2024-04-02T07:54:31.537732+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:37:29.009138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539180+0000","last_change":"2024-04-02T07:54:30.053908+0000","last_active":"2024-04-02T07:54:31.539180+0000","last_peered":"2024-04-02T07:54:31.539180+0000","last_clean":"2024-04-02T07:54:31.539180+0000","last_became_active":"2024-04-02T07:54:30.052545+0000","last_became_peered":"2024-04-02T07:54:30.052545+0000","last_unstale":"2024-04-02T07:54:31.539180+0000","last_undegraded":"2024-04-02T07:54:31.539180+0000","last_fullsized":"2024-04-02T07:54:31.539180+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:54:28.788127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594350+0000","last_change":"2024-04-02T07:54:30.546063+0000","last_active":"2024-04-02T07:54:31.594350+0000","last_peered":"2024-04-02T07:54:31.594350+0000","last_clean":"2024-04-02T07:54:31.594350+0000","last_became_active":"2024-04-02T07:54:30.544123+0000","last_became_peered":"2024-04-02T07:54:30.544123+0000","last_unstale":"2024-04-02T07:54:31.594350+0000","last_undegraded":"2024-04-02T07:54:31.594350+0000","last_fullsized":"2024-04-02T07:54:31.594350+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:55.892015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.7","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539384+0000","last_change":"2024-04-02T07:54:30.053485+0000","last_active":"2024-04-02T07:54:31.539384+0000","last_peered":"2024-04-02T07:54:31.539384+0000","last_clean":"2024-04-02T07:54:31.539384+0000","last_became_active":"2024-04-02T07:54:30.051982+0000","last_became_peered":"2024-04-02T07:54:30.051982+0000","last_unstale":"2024-04-02T07:54:31.539384+0000","last_undegraded":"2024-04-02T07:54:31.539384+0000","last_fullsized":"2024-04-02T07:54:31.539384+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:35.636951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594524+0000","last_change":"2024-04-02T07:54:30.555465+0000","last_active":"2024-04-02T07:54:31.594524+0000","last_peered":"2024-04-02T07:54:31.594524+0000","last_clean":"2024-04-02T07:54:31.594524+0000","last_became_active":"2024-04-02T07:54:30.555139+0000","last_became_peered":"2024-04-02T07:54:30.555139+0000","last_unstale":"2024-04-02T07:54:31.594524+0000","last_undegraded":"2024-04-02T07:54:31.594524+0000","last_fullsized":"2024-04-02T07:54:31.594524+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:19:34.362913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594456+0000","last_change":"2024-04-02T07:54:30.056940+0000","last_active":"2024-04-02T07:54:31.594456+0000","last_peered":"2024-04-02T07:54:31.594456+0000","last_clean":"2024-04-02T07:54:31.594456+0000","last_became_active":"2024-04-02T07:54:30.056590+0000","last_became_peered":"2024-04-02T07:54:30.056590+0000","last_unstale":"2024-04-02T07:54:31.594456+0000","last_undegraded":"2024-04-02T07:54:31.594456+0000","last_fullsized":"2024-04-02T07:54:31.594456+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:26:21.428369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539877+0000","last_change":"2024-04-02T07:54:30.557689+0000","last_active":"2024-04-02T07:54:31.539877+0000","last_peered":"2024-04-02T07:54:31.539877+0000","last_clean":"2024-04-02T07:54:31.539877+0000","last_became_active":"2024-04-02T07:54:30.557430+0000","last_became_peered":"2024-04-02T07:54:30.557430+0000","last_unstale":"2024-04-02T07:54:31.539877+0000","last_undegraded":"2024-04-02T07:54:31.539877+0000","last_fullsized":"2024-04-02T07:54:31.539877+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:01:03.970262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594164+0000","last_change":"2024-04-02T07:54:30.053073+0000","last_active":"2024-04-02T07:54:31.594164+0000","last_peered":"2024-04-02T07:54:31.594164+0000","last_clean":"2024-04-02T07:54:31.594164+0000","last_became_active":"2024-04-02T07:54:30.052801+0000","last_became_peered":"2024-04-02T07:54:30.052801+0000","last_unstale":"2024-04-02T07:54:31.594164+0000","last_undegraded":"2024-04-02T07:54:31.594164+0000","last_fullsized":"2024-04-02T07:54:31.594164+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:57.107798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592643+0000","last_change":"2024-04-02T07:54:30.541572+0000","last_active":"2024-04-02T07:54:31.592643+0000","last_peered":"2024-04-02T07:54:31.592643+0000","last_clean":"2024-04-02T07:54:31.592643+0000","last_became_active":"2024-04-02T07:54:30.541262+0000","last_became_peered":"2024-04-02T07:54:30.541262+0000","last_unstale":"2024-04-02T07:54:31.592643+0000","last_undegraded":"2024-04-02T07:54:31.592643+0000","last_fullsized":"2024-04-02T07:54:31.592643+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:36:29.743050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538294+0000","last_change":"2024-04-02T07:54:30.050096+0000","last_active":"2024-04-02T07:54:31.538294+0000","last_peered":"2024-04-02T07:54:31.538294+0000","last_clean":"2024-04-02T07:54:31.538294+0000","last_became_active":"2024-04-02T07:54:30.049652+0000","last_became_peered":"2024-04-02T07:54:30.049652+0000","last_unstale":"2024-04-02T07:54:31.538294+0000","last_undegraded":"2024-04-02T07:54:31.538294+0000","last_fullsized":"2024-04-02T07:54:31.538294+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:54:28.788127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592844+0000","last_change":"2024-04-02T07:54:30.547960+0000","last_active":"2024-04-02T07:54:31.592844+0000","last_peered":"2024-04-02T07:54:31.592844+0000","last_clean":"2024-04-02T07:54:31.592844+0000","last_became_active":"2024-04-02T07:54:30.547684+0000","last_became_peered":"2024-04-02T07:54:30.547684+0000","last_unstale":"2024-04-02T07:54:31.592844+0000","last_undegraded":"2024-04-02T07:54:31.592844+0000","last_fullsized":"2024-04-02T07:54:31.592844+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:36:35.379755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.540052+0000","last_change":"2024-04-02T07:54:30.058287+0000","last_active":"2024-04-02T07:54:31.540052+0000","last_peered":"2024-04-02T07:54:31.540052+0000","last_clean":"2024-04-02T07:54:31.540052+0000","last_became_active":"2024-04-02T07:54:30.055165+0000","last_became_peered":"2024-04-02T07:54:30.055165+0000","last_unstale":"2024-04-02T07:54:31.540052+0000","last_undegraded":"2024-04-02T07:54:31.540052+0000","last_fullsized":"2024-04-02T07:54:31.540052+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:35:18.551498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592568+0000","last_change":"2024-04-02T07:54:30.552203+0000","last_active":"2024-04-02T07:54:31.592568+0000","last_peered":"2024-04-02T07:54:31.592568+0000","last_clean":"2024-04-02T07:54:31.592568+0000","last_became_active":"2024-04-02T07:54:30.551933+0000","last_became_peered":"2024-04-02T07:54:30.551933+0000","last_unstale":"2024-04-02T07:54:31.592568+0000","last_undegraded":"2024-04-02T07:54:31.592568+0000","last_fullsized":"2024-04-02T07:54:31.592568+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:10:35.288763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538019+0000","last_change":"2024-04-02T07:54:30.052250+0000","last_active":"2024-04-02T07:54:31.538019+0000","last_peered":"2024-04-02T07:54:31.538019+0000","last_clean":"2024-04-02T07:54:31.538019+0000","last_became_active":"2024-04-02T07:54:30.051980+0000","last_became_peered":"2024-04-02T07:54:30.051980+0000","last_unstale":"2024-04-02T07:54:31.538019+0000","last_undegraded":"2024-04-02T07:54:31.538019+0000","last_fullsized":"2024-04-02T07:54:31.538019+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:00:49.879036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593574+0000","last_change":"2024-04-02T07:54:30.546191+0000","last_active":"2024-04-02T07:54:31.593574+0000","last_peered":"2024-04-02T07:54:31.593574+0000","last_clean":"2024-04-02T07:54:31.593574+0000","last_became_active":"2024-04-02T07:54:30.545517+0000","last_became_peered":"2024-04-02T07:54:30.545517+0000","last_unstale":"2024-04-02T07:54:31.593574+0000","last_undegraded":"2024-04-02T07:54:31.593574+0000","last_fullsized":"2024-04-02T07:54:31.593574+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:03:09.202047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592755+0000","last_change":"2024-04-02T07:54:30.058271+0000","last_active":"2024-04-02T07:54:31.592755+0000","last_peered":"2024-04-02T07:54:31.592755+0000","last_clean":"2024-04-02T07:54:31.592755+0000","last_became_active":"2024-04-02T07:54:30.057968+0000","last_became_peered":"2024-04-02T07:54:30.057968+0000","last_unstale":"2024-04-02T07:54:31.592755+0000","last_undegraded":"2024-04-02T07:54:31.592755+0000","last_fullsized":"2024-04-02T07:54:31.592755+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:54.278160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538867+0000","last_change":"2024-04-02T07:54:30.558497+0000","last_active":"2024-04-02T07:54:31.538867+0000","last_peered":"2024-04-02T07:54:31.538867+0000","last_clean":"2024-04-02T07:54:31.538867+0000","last_became_active":"2024-04-02T07:54:30.557865+0000","last_became_peered":"2024-04-02T07:54:30.557865+0000","last_unstale":"2024-04-02T07:54:31.538867+0000","last_undegraded":"2024-04-02T07:54:31.538867+0000","last_fullsized":"2024-04-02T07:54:31.538867+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:01:03.970262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539200+0000","last_change":"2024-04-02T07:54:30.056409+0000","last_active":"2024-04-02T07:54:31.539200+0000","last_peered":"2024-04-02T07:54:31.539200+0000","last_clean":"2024-04-02T07:54:31.539200+0000","last_became_active":"2024-04-02T07:54:30.056116+0000","last_became_peered":"2024-04-02T07:54:30.056116+0000","last_unstale":"2024-04-02T07:54:31.539200+0000","last_undegraded":"2024-04-02T07:54:31.539200+0000","last_fullsized":"2024-04-02T07:54:31.539200+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:52.812327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.540063+0000","last_change":"2024-04-02T07:54:30.555467+0000","last_active":"2024-04-02T07:54:31.540063+0000","last_peered":"2024-04-02T07:54:31.540063+0000","last_clean":"2024-04-02T07:54:31.540063+0000","last_became_active":"2024-04-02T07:54:30.552874+0000","last_became_peered":"2024-04-02T07:54:30.552874+0000","last_unstale":"2024-04-02T07:54:31.540063+0000","last_undegraded":"2024-04-02T07:54:31.540063+0000","last_fullsized":"2024-04-02T07:54:31.540063+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:54.425891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.f","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539356+0000","last_change":"2024-04-02T07:54:30.056911+0000","last_active":"2024-04-02T07:54:31.539356+0000","last_peered":"2024-04-02T07:54:31.539356+0000","last_clean":"2024-04-02T07:54:31.539356+0000","last_became_active":"2024-04-02T07:54:30.056710+0000","last_became_peered":"2024-04-02T07:54:30.056710+0000","last_unstale":"2024-04-02T07:54:31.539356+0000","last_undegraded":"2024-04-02T07:54:31.539356+0000","last_fullsized":"2024-04-02T07:54:31.539356+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:45:58.230418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"114'205","reported_seq":577,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-02T07:54:30.543107+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:54:30.543107+0000","last_peered":"2024-04-02T07:54:30.543107+0000","last_clean":"2024-04-02T07:54:30.543107+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:54:30.543107+0000","last_undegraded":"2024-04-02T07:54:30.543107+0000","last_fullsized":"2024-04-02T07:54:30.543107+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575067,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1536000,"data_stored":754723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65400000000000003,"15min":0.67300000000000004},"min":{"1min":0.46700000000000003,"5min":0.34699999999999998,"15min":0.316},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.66200000000000003,"15min":0.68000000000000005},"min":{"1min":0.433,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.93000000000000005,"5min":1.024,"15min":1.024},"last":0.53300000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.66500000000000004,"15min":0.67800000000000005},"min":{"1min":0.46700000000000003,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.96099999999999997,"5min":1.097,"15min":1.097},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.65300000000000002,"15min":0.67400000000000004},"min":{"1min":0.53400000000000003,"5min":0.42399999999999999,"15min":0.34699999999999998},"max":{"1min":0.93999999999999995,"5min":0.98299999999999998,"15min":1.2190000000000001},"last":0.63100000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.68600000000000005,"15min":0.70799999999999996},"min":{"1min":0.48599999999999999,"5min":0.39500000000000002,"15min":0.34000000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":1.127},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.55500000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96499999999999997,"5min":1.044,"15min":1.1259999999999999},"last":0.70799999999999996}]},{"osd":3,"last update":"Tue Apr 2 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.68999999999999995,"15min":0.70399999999999996},"min":{"1min":0.502,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.95799999999999996,"5min":0.97799999999999998,"15min":1.026},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.69699999999999995,"15min":0.71799999999999997},"min":{"1min":0.51500000000000001,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.98899999999999999,"5min":1.071,"15min":1.196},"last":0.76200000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57299999999999995,"15min":0.58999999999999997},"min":{"1min":0.39900000000000002,"5min":0.35799999999999998,"15min":0.32000000000000001},"max":{"1min":0.79900000000000004,"5min":1.0229999999999999,"15min":1.0389999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.69599999999999995,"15min":0.69099999999999995},"min":{"1min":0.52700000000000002,"5min":0.308,"15min":0.308},"max":{"1min":1.016,"5min":1.016,"15min":1.0669999999999999},"last":0.72799999999999998}]},{"osd":5,"last update":"Tue Apr 2 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.65600000000000003,"15min":0.65100000000000002},"min":{"1min":0.497,"5min":0.42999999999999999,"15min":0.37},"max":{"1min":0.99199999999999999,"5min":1.0109999999999999,"15min":1.167},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":1.343,"15min":1.4219999999999999},"last":0.748}]},{"osd":6,"last update":"Tue Apr 2 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.66400000000000003,"15min":0.66200000000000003},"min":{"1min":0.53200000000000003,"5min":0.435,"15min":0.36199999999999999},"max":{"1min":0.94499999999999995,"5min":1.0580000000000001,"15min":2.5369999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.67000000000000004,"15min":0.65800000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.39600000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.0509999999999999},"last":0.65100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575065,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36732,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335108,"statfs":{"total":96636764160,"available":96599150592,"internally_reserved":0,"allocated":1544192,"data_stored":760789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.68999999999999995,"15min":0.70699999999999996},"min":{"1min":0.50800000000000001,"5min":0.40899999999999997,"15min":0.33400000000000002},"max":{"1min":0.94299999999999995,"5min":0.95899999999999996,"15min":0.98099999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68200000000000005,"15min":0.70699999999999996},"min":{"1min":0.42399999999999999,"5min":0.40500000000000003,"15min":0.38300000000000001},"max":{"1min":0.94299999999999995,"5min":0.96899999999999997,"15min":1.0249999999999999},"last":0.57199999999999995}]},{"osd":1,"last update":"Tue Apr 2 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.67300000000000004,"15min":0.69399999999999995},"min":{"1min":0.47899999999999998,"5min":0.433,"15min":0.41799999999999998},"max":{"1min":0.89300000000000002,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.71599999999999997,"15min":0.71899999999999997},"min":{"1min":0.48899999999999999,"5min":0.45300000000000001,"15min":0.42599999999999999},"max":{"1min":0.96399999999999997,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.54000000000000004}]},{"osd":2,"last update":"Tue Apr 2 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68899999999999995,"15min":0.71299999999999997},"min":{"1min":0.54100000000000004,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83199999999999996,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.69099999999999995,"15min":0.70799999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93600000000000005,"5min":0.96499999999999997,"15min":0.96599999999999997},"last":0.66800000000000004}]},{"osd":3,"last update":"Tue Apr 2 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71599999999999997,"15min":0.73199999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90600000000000003,"5min":1.089,"15min":1.089},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.71799999999999997,"15min":0.73799999999999999},"min":{"1min":0.57099999999999995,"5min":0.46600000000000003,"15min":0.45300000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0309999999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61699999999999999,"15min":0.60499999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.372},"max":{"1min":0.99399999999999999,"5min":0.996,"15min":0.996},"last":0.622},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.41699999999999998,"5min":0.40600000000000003,"15min":0.35499999999999998},"max":{"1min":0.98099999999999998,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.70599999999999996}]},{"osd":5,"last update":"Tue Apr 2 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.61599999999999999,"15min":0.61799999999999999},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.88,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68000000000000005,"15min":0.68200000000000005},"min":{"1min":0.41199999999999998,"5min":0.40899999999999997,"15min":0.40300000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.81100000000000005}]},{"osd":7,"last update":"Tue Apr 2 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.68700000000000006,"15min":0.68300000000000005},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.40200000000000002},"max":{"1min":0.95199999999999996,"5min":0.96199999999999997,"15min":1.0389999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68700000000000006,"15min":0.69099999999999995},"min":{"1min":0.48199999999999998,"5min":0.42799999999999999,"15min":0.42099999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.0149999999999999},"last":0.80200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575066,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59472,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312368,"statfs":{"total":96636764160,"available":96575864832,"internally_reserved":0,"allocated":2158592,"data_stored":1352833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.55000000000000004,"15min":0.47599999999999998},"min":{"1min":0.309,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.627,"5min":10.052,"15min":10.052},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.60799999999999998,"15min":0.53800000000000003},"min":{"1min":0.33900000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.92700000000000005,"5min":9.0250000000000004,"15min":9.0250000000000004},"last":0.36199999999999999}]},{"osd":2,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59499999999999997,"15min":0.53400000000000003},"min":{"1min":0.45300000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.74199999999999999,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64500000000000002,"15min":0.59899999999999998},"min":{"1min":0.503,"5min":0.314,"15min":0.314},"max":{"1min":1.1719999999999999,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.503}]},{"osd":3,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58299999999999996,"15min":0.57399999999999995},"min":{"1min":0.40600000000000003,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.137,"5min":2.609,"15min":2.609},"last":0.443},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.54800000000000004,"15min":0.54300000000000004},"min":{"1min":0.45400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81499999999999995,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.32300000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.77500000000000002,"15min":0.77400000000000002},"min":{"1min":0.59499999999999997,"5min":0.50900000000000001,"15min":0.46300000000000002},"max":{"1min":0.95899999999999996,"5min":2.21,"15min":2.21},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.82999999999999996,"15min":0.78800000000000003},"min":{"1min":0.53100000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1100000000000001,"5min":6.3419999999999996,"15min":6.3419999999999996},"last":0.65400000000000003}]},{"osd":5,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.84099999999999997,"15min":0.79400000000000004},"min":{"1min":0.59399999999999997,"5min":0.53800000000000003,"15min":0.41999999999999998},"max":{"1min":1.0189999999999999,"5min":6.5529999999999999,"15min":6.5529999999999999},"last":0.625},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.79300000000000004},"min":{"1min":0.55100000000000005,"5min":0.51000000000000001,"15min":0.48799999999999999},"max":{"1min":1.0669999999999999,"5min":2.544,"15min":2.544},"last":0.52900000000000003}]},{"osd":6,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":1.863,"15min":1.1539999999999999},"min":{"1min":0.61299999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.1259999999999999,"5min":78.903000000000006,"15min":78.903000000000006},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.84499999999999997,"5min":1.847,"15min":1.143},"min":{"1min":0.56899999999999995,"5min":0.49199999999999999,"15min":0.38100000000000001},"max":{"1min":1.9750000000000001,"5min":78.837000000000003,"15min":78.837000000000003},"last":0.57899999999999996}]},{"osd":7,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79700000000000004,"15min":0.80300000000000005},"min":{"1min":0.622,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.139,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.82299999999999995,"15min":0.82499999999999996},"min":{"1min":0.65000000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.073,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":0.55800000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575066,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32628,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339212,"statfs":{"total":96636764160,"available":96603353088,"internally_reserved":0,"allocated":1540096,"data_stored":759770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:54:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.435,"15min":0.439},"min":{"1min":0.34899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.68000000000000005,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.379},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.47199999999999998,"15min":0.48099999999999998},"min":{"1min":0.28799999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.64200000000000002,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.47099999999999997}]},{"osd":2,"last update":"Tue Apr 2 07:54:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.30399999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63400000000000001,"5min":1.597,"15min":1.597},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.53000000000000003},"min":{"1min":0.32600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.75800000000000001,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.434}]},{"osd":3,"last update":"Tue Apr 2 07:54:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.50900000000000001,"15min":0.52300000000000002},"min":{"1min":0.34599999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73699999999999999,"5min":0.78700000000000003,"15min":0.81799999999999995},"last":0.496},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58699999999999997,"15min":0.60399999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.84499999999999997,"5min":1.492,"15min":1.492},"last":0.61299999999999999}]},{"osd":4,"last update":"Tue Apr 2 07:54:22 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.75800000000000001,"15min":0.77800000000000002},"min":{"1min":0.49299999999999999,"5min":0.39400000000000002,"15min":0.35599999999999998},"max":{"1min":1.1000000000000001,"5min":1.1579999999999999,"15min":1.181},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.76800000000000002,"15min":0.77300000000000002},"min":{"1min":0.48499999999999999,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":1.0629999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.54300000000000004}]},{"osd":5,"last update":"Tue Apr 2 07:54:22 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.77400000000000002,"15min":0.79200000000000004},"min":{"1min":0.53300000000000003,"5min":0.48699999999999999,"15min":0.42099999999999999},"max":{"1min":1.117,"5min":1.361,"15min":2.1869999999999998},"last":0.622},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.77800000000000002,"15min":0.79800000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.079,"5min":1.546,"15min":2.6230000000000002},"last":0.68600000000000005}]},{"osd":6,"last update":"Tue Apr 2 07:54:22 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.74399999999999999,"15min":0.752},"min":{"1min":0.52800000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.056,"5min":1.52,"15min":1.52},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.72099999999999997,"15min":0.745},"min":{"1min":0.50800000000000001,"5min":0.46700000000000003,"15min":0.376},"max":{"1min":1.0389999999999999,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.86099999999999999}]},{"osd":7,"last update":"Tue Apr 2 07:54:22 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.80100000000000005,"15min":0.81399999999999995},"min":{"1min":0.57299999999999995,"5min":0.51600000000000001,"15min":0.40799999999999997},"max":{"1min":1.139,"5min":1.139,"15min":2.3959999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76200000000000001,"15min":0.80400000000000005},"min":{"1min":0.59999999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.073,"5min":1.3959999999999999,"15min":2.145},"last":0.58699999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575065,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36736,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335104,"statfs":{"total":96636764160,"available":96599146496,"internally_reserved":0,"allocated":1548288,"data_stored":760859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.44600000000000001,"15min":0.44700000000000001},"min":{"1min":0.30499999999999999,"5min":0.26700000000000002,"15min":0.26400000000000001},"max":{"1min":0.56899999999999995,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.47499999999999998,"15min":0.47899999999999998},"min":{"1min":0.34200000000000003,"5min":0.29099999999999998,"15min":0.26200000000000001},"max":{"1min":0.73699999999999999,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.45300000000000001}]},{"osd":1,"last update":"Tue Apr 2 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.49099999999999999},"min":{"1min":0.375,"5min":0.32700000000000001,"15min":0.30499999999999999},"max":{"1min":0.64700000000000002,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.436},{"interface":"front","average":{"1min":0.504,"5min":0.51100000000000001,"15min":0.52300000000000002},"min":{"1min":0.375,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":0.65900000000000003,"5min":0.996,"15min":0.996},"last":0.39500000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53400000000000003,"15min":0.53600000000000003},"min":{"1min":0.39600000000000002,"5min":0.33500000000000002,"15min":0.316},"max":{"1min":0.69499999999999995,"5min":1.472,"15min":1.472},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59899999999999998,"15min":0.60699999999999998},"min":{"1min":0.47999999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.69699999999999995,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.497}]},{"osd":4,"last update":"Tue Apr 2 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.73599999999999999,"15min":0.76800000000000002},"min":{"1min":0.54600000000000004,"5min":0.5,"15min":0.47199999999999998},"max":{"1min":0.82799999999999996,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73899999999999999,"15min":0.76900000000000002},"min":{"1min":0.55000000000000004,"5min":0.54800000000000004,"15min":0.48499999999999999},"max":{"1min":0.96499999999999997,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.627}]},{"osd":5,"last update":"Tue Apr 2 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72699999999999998,"15min":0.75600000000000001},"min":{"1min":0.53200000000000003,"5min":0.51000000000000001,"15min":0.46200000000000002},"max":{"1min":0.86599999999999999,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.74199999999999999,"15min":0.77500000000000002},"min":{"1min":0.627,"5min":0.55600000000000005,"15min":0.50900000000000001},"max":{"1min":0.92400000000000004,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.61499999999999999}]},{"osd":6,"last update":"Tue Apr 2 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.76200000000000001,"15min":0.78600000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.442},"max":{"1min":0.86899999999999999,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.78000000000000003},"min":{"1min":0.622,"5min":0.502,"15min":0.502},"max":{"1min":0.84399999999999997,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.70499999999999996}]},{"osd":7,"last update":"Tue Apr 2 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.79300000000000004},"min":{"1min":0.58199999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.99199999999999999,"5min":1.278,"15min":1.278},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.74099999999999999,"15min":0.78700000000000003},"min":{"1min":0.56100000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93300000000000005,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.67800000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575065,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58884,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312956,"statfs":{"total":96636764160,"available":96576466944,"internally_reserved":0,"allocated":1556480,"data_stored":760983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42499999999999999,"15min":0.42799999999999999},"min":{"1min":0.308,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.65100000000000002,"5min":0.748,"15min":0.79600000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.45400000000000001,"15min":0.46500000000000002},"min":{"1min":0.33600000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.84099999999999997,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.40799999999999997}]},{"osd":1,"last update":"Tue Apr 2 07:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.47599999999999998,"15min":0.48999999999999999},"min":{"1min":0.32900000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.63,"5min":0.83799999999999997,"15min":1.2809999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57699999999999996,"15min":0.56699999999999995},"min":{"1min":0.42199999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.85799999999999998,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":0.47599999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.61499999999999999,"15min":0.61299999999999999},"min":{"1min":0.44400000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.871,"5min":2.214,"15min":2.214},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.58099999999999996,"15min":0.60499999999999998},"min":{"1min":0.46200000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":1.244},"last":0.441}]},{"osd":4,"last update":"Tue Apr 2 07:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.73799999999999999,"15min":0.74099999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.38400000000000001},"max":{"1min":1.022,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.73499999999999999,"15min":0.75700000000000001},"min":{"1min":0.58699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.052,"5min":1.052,"15min":1.1759999999999999},"last":0.67500000000000004}]},{"osd":5,"last update":"Tue Apr 2 07:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.74199999999999999,"15min":0.75900000000000001},"min":{"1min":0.61099999999999999,"5min":0.41699999999999998,"15min":0.40899999999999997},"max":{"1min":0.90500000000000003,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.748,"15min":0.75600000000000001},"min":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.40200000000000002},"max":{"1min":1.147,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.66100000000000003}]},{"osd":6,"last update":"Tue Apr 2 07:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.76400000000000001,"15min":0.77000000000000002},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.45800000000000002},"max":{"1min":1.1779999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.75,"15min":0.77400000000000002},"min":{"1min":0.497,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92700000000000005,"5min":2.286,"15min":2.286},"last":0.72199999999999998}]},{"osd":7,"last update":"Tue Apr 2 07:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.75600000000000001,"15min":0.77400000000000002},"min":{"1min":0.58199999999999996,"5min":0.52500000000000002,"15min":0.439},"max":{"1min":1.014,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.77100000000000002,"15min":0.79000000000000004},"min":{"1min":0.64100000000000001,"5min":0.54000000000000004,"15min":0.48999999999999999},"max":{"1min":1.022,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.63400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575065,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36724,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335116,"statfs":{"total":96636764160,"available":96599158784,"internally_reserved":0,"allocated":1548288,"data_stored":761371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.66600000000000004,"15min":0.67100000000000004},"min":{"1min":0.58599999999999997,"5min":0.41699999999999998,"15min":0.32100000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":1.0189999999999999},"last":0.754},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.65400000000000003,"15min":0.67100000000000004},"min":{"1min":0.60399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.94999999999999996,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.53200000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.70999999999999996,"15min":0.72599999999999998},"min":{"1min":0.56399999999999995,"5min":0.438,"15min":0.433},"max":{"1min":0.91700000000000004,"5min":1.014,"15min":1.036},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.71799999999999997,"15min":0.73599999999999999},"min":{"1min":0.55000000000000004,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.68400000000000005,"15min":0.69399999999999995},"min":{"1min":0.47399999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84199999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.70199999999999996,"15min":0.71499999999999997},"min":{"1min":0.63600000000000001,"5min":0.378,"15min":0.378},"max":{"1min":1.026,"5min":1.026,"15min":1.0620000000000001},"last":0.67800000000000005}]},{"osd":3,"last update":"Tue Apr 2 07:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.72999999999999998,"15min":0.72399999999999998},"min":{"1min":0.63500000000000001,"5min":0.505,"15min":0.46800000000000003},"max":{"1min":0.96399999999999997,"5min":0.97799999999999998,"15min":1.002},"last":0.628},{"interface":"front","average":{"1min":0.751,"5min":0.70699999999999996,"15min":0.72999999999999998},"min":{"1min":0.53900000000000003,"5min":0.47299999999999998,"15min":0.41499999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.70599999999999996}]},{"osd":5,"last update":"Tue Apr 2 07:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.437,"5min":0.38600000000000001,"15min":0.29599999999999999},"max":{"1min":0.85499999999999998,"5min":0.90800000000000003,"15min":0.93999999999999995},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59599999999999997,"15min":0.58599999999999997},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.33700000000000002},"max":{"1min":0.747,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.76600000000000001}]},{"osd":6,"last update":"Tue Apr 2 07:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.61599999999999999,"15min":0.60199999999999998},"min":{"1min":0.47399999999999998,"5min":0.433,"15min":0.374},"max":{"1min":0.84099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.65200000000000002},"min":{"1min":0.496,"5min":0.44400000000000001,"15min":0.42399999999999999},"max":{"1min":0.93100000000000005,"5min":1.0129999999999999,"15min":1.038},"last":0.84099999999999997}]},{"osd":7,"last update":"Tue Apr 2 07:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68100000000000005,"15min":0.67200000000000004},"min":{"1min":0.57999999999999996,"5min":0.39600000000000002,"15min":0.38300000000000001},"max":{"1min":0.89600000000000002,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.747,"5min":0.74099999999999999,"15min":0.73099999999999998},"min":{"1min":0.55400000000000005,"5min":0.47799999999999998,"15min":0.434},"max":{"1min":0.91700000000000004,"5min":1.095,"15min":1.095},"last":0.97499999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575068,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37316,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334524,"statfs":{"total":96636764160,"available":96598552576,"internally_reserved":0,"allocated":2154496,"data_stored":1351758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.65400000000000003,"15min":0.68200000000000005},"min":{"1min":0.443,"5min":0.35299999999999998,"15min":0.29899999999999999},"max":{"1min":0.92300000000000004,"5min":0.95299999999999996,"15min":0.99199999999999999},"last":0.629},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.67200000000000004,"15min":0.68799999999999994},"min":{"1min":0.45900000000000002,"5min":0.40000000000000002,"15min":0.378},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.4450000000000001},"last":0.53300000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.66700000000000004,"15min":0.69399999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.88100000000000001,"5min":1.0620000000000001,"15min":1.131},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.67600000000000005,"15min":0.69299999999999995},"min":{"1min":0.51800000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.5149999999999999},"last":0.56399999999999995}]},{"osd":2,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.67800000000000005,"15min":0.70499999999999996},"min":{"1min":0.47699999999999998,"5min":0.43099999999999999,"15min":0.41799999999999998},"max":{"1min":0.90700000000000003,"5min":0.98099999999999998,"15min":1.5449999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68500000000000005,"15min":0.70699999999999996},"min":{"1min":0.51900000000000002,"5min":0.47399999999999998,"15min":0.372},"max":{"1min":0.88500000000000001,"5min":1.111,"15min":1.3959999999999999},"last":0.51600000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.71899999999999997,"15min":0.73799999999999999},"min":{"1min":0.55900000000000005,"5min":0.46200000000000002,"15min":0.45100000000000001},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.341},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.44900000000000001},"max":{"1min":0.97499999999999998,"5min":1.1839999999999999,"15min":1.484},"last":0.64700000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58799999999999997,"15min":0.59199999999999997},"min":{"1min":0.44800000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.78100000000000003,"5min":0.80700000000000005,"15min":1.3109999999999999},"last":0.496},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64300000000000002,"15min":0.63500000000000001},"min":{"1min":0.52200000000000002,"5min":0.39600000000000002,"15min":0.35599999999999998},"max":{"1min":0.94199999999999995,"5min":0.96199999999999997,"15min":1.504},"last":0.69299999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66500000000000004,"15min":0.64100000000000001},"min":{"1min":0.42599999999999999,"5min":0.41899999999999998,"15min":0.39900000000000002},"max":{"1min":0.93000000000000005,"5min":1.212,"15min":1.212},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70099999999999996,"15min":0.67600000000000005},"min":{"1min":0.59899999999999998,"5min":0.45500000000000002,"15min":0.41199999999999998},"max":{"1min":0.87,"5min":1.774,"15min":1.774},"last":0.42999999999999999}]},{"osd":7,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72499999999999998,"15min":0.69999999999999996},"min":{"1min":0.55600000000000005,"5min":0.45800000000000002,"15min":0.38900000000000001},"max":{"1min":0.98999999999999999,"5min":1.0129999999999999,"15min":1.6890000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.745,"15min":0.72399999999999998},"min":{"1min":0.60899999999999999,"5min":0.47799999999999998,"15min":0.434},"max":{"1min":0.999,"5min":1.149,"15min":1.5309999999999999},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:54:34.913 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:54:35.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.137+0000 7f9ec5600700 1 Processor -- start 2024-04-02T07:54:35.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.137+0000 7f9ec5600700 1 -- start start 2024-04-02T07:54:35.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.138+0000 7f9ec5600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ec006a790 0x7f9ec006ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:35.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.138+0000 7f9ec5600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec006f850 con 0x7f9ec00afbe0 2024-04-02T07:54:35.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.138+0000 7f9ec5600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec006e3f0 con 0x7f9ec00cebd0 2024-04-02T07:54:35.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.138+0000 7f9ec5600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec00cf6f0 con 0x7f9ec006a790 2024-04-02T07:54:35.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.138+0000 7f9ebea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ec006a790 0x7f9ec006ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:35.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.138+0000 7f9ebe000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9ec00afbe0 0x7f9ec006dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36726/0 (socket says 172.21.15.89:36726) 2024-04-02T07:54:35.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.138+0000 7f9ebea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ec006a790 0x7f9ec006ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36064/0 (socket says 172.21.15.89:36064) 2024-04-02T07:54:35.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.138+0000 7f9ebe000700 1 -- 172.21.15.89:0/2058772434 learned_addr learned my addr 172.21.15.89:0/2058772434 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:35.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.139+0000 7f9ebea00700 1 -- 172.21.15.89:0/2058772434 >> v1:172.21.15.113:6789/0 conn(0x7f9ec00cebd0 legacy=0x7f9ec00cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:35.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.139+0000 7f9ebea00700 1 -- 172.21.15.89:0/2058772434 >> v1:172.21.15.89:6789/0 conn(0x7f9ec00afbe0 legacy=0x7f9ec006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:35.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.139+0000 7f9ebea00700 1 -- 172.21.15.89:0/2058772434 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec006b0b0 con 0x7f9ec006a790 2024-04-02T07:54:35.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.139+0000 7f9ebea00700 1 --2- 172.21.15.89:0/2058772434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ec006a790 0x7f9ec006ab70 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009840 tx=0x7f9eb0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a45b4c00ba4aef6c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:35.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.139+0000 7f9ebd600700 1 -- 172.21.15.89:0/2058772434 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb0011830 con 0x7f9ec006a790 2024-04-02T07:54:35.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.139+0000 7f9ebd600700 1 -- 172.21.15.89:0/2058772434 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb0011990 con 0x7f9ec006a790 2024-04-02T07:54:35.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.139+0000 7f9ebd600700 1 -- 172.21.15.89:0/2058772434 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb0011d20 con 0x7f9ec006a790 2024-04-02T07:54:35.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.140+0000 7f9ec5600700 1 -- 172.21.15.89:0/2058772434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ec006a790 msgr2=0x7f9ec006ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:35.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.140+0000 7f9ec5600700 1 --2- 172.21.15.89:0/2058772434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ec006a790 0x7f9ec006ab70 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009840 tx=0x7f9eb0009b50 comp rx=0 tx=0).stop 2024-04-02T07:54:35.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.140+0000 7f9ec5600700 1 -- 172.21.15.89:0/2058772434 shutdown_connections 2024-04-02T07:54:35.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.140+0000 7f9ec5600700 1 --2- 172.21.15.89:0/2058772434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ec006a790 0x7f9ec006ab70 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.140+0000 7f9ec5600700 1 -- 172.21.15.89:0/2058772434 >> 172.21.15.89:0/2058772434 conn(0x7f9ec00b5ab0 msgr2=0x7f9ec00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:35.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.140+0000 7f9ec5600700 1 -- 172.21.15.89:0/2058772434 shutdown_connections 2024-04-02T07:54:35.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.140+0000 7f9ec5600700 1 -- 172.21.15.89:0/2058772434 wait complete. 2024-04-02T07:54:35.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.142+0000 7f9ec5600700 1 Processor -- start 2024-04-02T07:54:35.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.142+0000 7f9ec5600700 1 -- start start 2024-04-02T07:54:35.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ec5600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ec006a790 0x7f9ec0154a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:35.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ec5600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ec00afbe0 0x7f9ec00ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:35.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ec5600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ec00cebd0 0x7f9ec00ae810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:35.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ec5600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec00c96f0 con 0x7f9ec00afbe0 2024-04-02T07:54:35.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ec5600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec00c9990 con 0x7f9ec006a790 2024-04-02T07:54:35.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ec5600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec00c9c70 con 0x7f9ec00cebd0 2024-04-02T07:54:35.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ebe000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ec00afbe0 0x7f9ec00ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:35.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ebe000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ec00afbe0 0x7f9ec00ae2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36078/0 (socket says 172.21.15.89:36078) 2024-04-02T07:54:35.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ebe000700 1 -- 172.21.15.89:0/3881817503 learned_addr learned my addr 172.21.15.89:0/3881817503 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:35.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ebea00700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ec006a790 0x7f9ec0154a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:35.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ebf400700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ec00cebd0 0x7f9ec00ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:35.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ebe000700 1 -- 172.21.15.89:0/3881817503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ec00cebd0 msgr2=0x7f9ec00ae810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:35.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ebe000700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ec00cebd0 0x7f9ec00ae810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ebe000700 1 -- 172.21.15.89:0/3881817503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ec006a790 msgr2=0x7f9ec0154a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:35.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ebe000700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ec006a790 0x7f9ec0154a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ebe000700 1 -- 172.21.15.89:0/3881817503 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb00094a0 con 0x7f9ec00afbe0 2024-04-02T07:54:35.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.143+0000 7f9ebf400700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ec00cebd0 0x7f9ec00ae810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:35.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.144+0000 7f9ebe000700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ec00afbe0 0x7f9ec00ae2d0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f9eac00c520 tx=0x7f9eac00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:35.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.144+0000 7f9eb6a00700 1 -- 172.21.15.89:0/3881817503 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eac005d70 con 0x7f9ec00afbe0 2024-04-02T07:54:35.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.144+0000 7f9eb6a00700 1 -- 172.21.15.89:0/3881817503 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eac0070d0 con 0x7f9ec00afbe0 2024-04-02T07:54:35.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.144+0000 7f9ec5600700 1 -- 172.21.15.89:0/3881817503 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec00c9f50 con 0x7f9ec00afbe0 2024-04-02T07:54:35.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.144+0000 7f9eb6a00700 1 -- 172.21.15.89:0/3881817503 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eac007440 con 0x7f9ec00afbe0 2024-04-02T07:54:35.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.144+0000 7f9ec5600700 1 -- 172.21.15.89:0/3881817503 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec0154fd0 con 0x7f9ec00afbe0 2024-04-02T07:54:35.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.145+0000 7f9ec5600700 1 -- 172.21.15.89:0/3881817503 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec00ca470 con 0x7f9ec00afbe0 2024-04-02T07:54:35.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.146+0000 7f9eb6a00700 1 -- 172.21.15.89:0/3881817503 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9eac0075a0 con 0x7f9ec00afbe0 2024-04-02T07:54:35.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.147+0000 7f9eb6a00700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9ea006adb0 0x7f9ea006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:35.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.147+0000 7f9eb6a00700 1 -- 172.21.15.89:0/3881817503 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9eac088ab0 con 0x7f9ec00afbe0 2024-04-02T07:54:35.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.147+0000 7f9ebea00700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9ea006adb0 0x7f9ea006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:35.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.148+0000 7f9ebea00700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9ea006adb0 0x7f9ea006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9eb00090b0 tx=0x7f9eb001c040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:35.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.151+0000 7f9eb6a00700 1 -- 172.21.15.89:0/3881817503 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9ec00ca470 con 0x7f9ec00afbe0 2024-04-02T07:54:35.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.349+0000 7f9ec5600700 1 -- 172.21.15.89:0/3881817503 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9ec00af690 con 0x7f9ea006adb0 2024-04-02T07:54:35.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.357+0000 7f9eb6a00700 1 -- 172.21.15.89:0/3881817503 <== mgr.7225 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289718 (secure 0 0 0) 0x7f9eb0011c80 con 0x7f9ea006adb0 2024-04-02T07:54:35.361 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:35.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.361+0000 7f9ec5600700 1 -- 172.21.15.89:0/3881817503 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9ea006adb0 msgr2=0x7f9ea006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:35.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.361+0000 7f9ec5600700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9ea006adb0 0x7f9ea006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9eb00090b0 tx=0x7f9eb001c040 comp rx=0 tx=0).stop 2024-04-02T07:54:35.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.361+0000 7f9ec5600700 1 -- 172.21.15.89:0/3881817503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ec00afbe0 msgr2=0x7f9ec00ae2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:35.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.361+0000 7f9ec5600700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ec00afbe0 0x7f9ec00ae2d0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f9eac00c520 tx=0x7f9eac00c8e0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.361+0000 7f9ec5600700 1 -- 172.21.15.89:0/3881817503 shutdown_connections 2024-04-02T07:54:35.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.361+0000 7f9ec5600700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9ea006adb0 0x7f9ea006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.361+0000 7f9ec5600700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ec006a790 0x7f9ec0154a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.361+0000 7f9ec5600700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ec00cebd0 0x7f9ec00ae810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.361+0000 7f9ec5600700 1 --2- 172.21.15.89:0/3881817503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ec00afbe0 0x7f9ec00ae2d0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.361+0000 7f9ec5600700 1 -- 172.21.15.89:0/3881817503 >> 172.21.15.89:0/3881817503 conn(0x7f9ec00b5ab0 msgr2=0x7f9ec00b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:35.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.361+0000 7f9ec5600700 1 -- 172.21.15.89:0/3881817503 shutdown_connections 2024-04-02T07:54:35.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.361+0000 7f9ec5600700 1 -- 172.21.15.89:0/3881817503 wait complete. 2024-04-02T07:54:35.365 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:54:35.380 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T07:54:34.288323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":151,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331112,"kb_used_data":13268,"kb_used_omap":12,"kb_used_meta":317683,"kb_avail":754643608,"statfs":{"total":773094113280,"available":772755054592,"internally_reserved":0,"allocated":13586432,"data_stored":7263086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":325307960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.261969"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593956+0000","last_change":"2024-04-02T07:54:30.543010+0000","last_active":"2024-04-02T07:54:31.593956+0000","last_peered":"2024-04-02T07:54:31.593956+0000","last_clean":"2024-04-02T07:54:31.593956+0000","last_became_active":"2024-04-02T07:54:30.542690+0000","last_became_peered":"2024-04-02T07:54:30.542690+0000","last_unstale":"2024-04-02T07:54:31.593956+0000","last_undegraded":"2024-04-02T07:54:31.593956+0000","last_fullsized":"2024-04-02T07:54:31.593956+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:45.999065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593702+0000","last_change":"2024-04-02T07:54:30.555286+0000","last_active":"2024-04-02T07:54:31.593702+0000","last_peered":"2024-04-02T07:54:31.593702+0000","last_clean":"2024-04-02T07:54:31.593702+0000","last_became_active":"2024-04-02T07:54:30.555004+0000","last_became_peered":"2024-04-02T07:54:30.555004+0000","last_unstale":"2024-04-02T07:54:31.593702+0000","last_undegraded":"2024-04-02T07:54:31.593702+0000","last_fullsized":"2024-04-02T07:54:31.593702+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:07:59.484212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539836+0000","last_change":"2024-04-02T07:54:30.558159+0000","last_active":"2024-04-02T07:54:31.539836+0000","last_peered":"2024-04-02T07:54:31.539836+0000","last_clean":"2024-04-02T07:54:31.539836+0000","last_became_active":"2024-04-02T07:54:30.557097+0000","last_became_peered":"2024-04-02T07:54:30.557097+0000","last_unstale":"2024-04-02T07:54:31.539836+0000","last_undegraded":"2024-04-02T07:54:31.539836+0000","last_fullsized":"2024-04-02T07:54:31.539836+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:46.886600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594013+0000","last_change":"2024-04-02T07:54:30.554983+0000","last_active":"2024-04-02T07:54:31.594013+0000","last_peered":"2024-04-02T07:54:31.594013+0000","last_clean":"2024-04-02T07:54:31.594013+0000","last_became_active":"2024-04-02T07:54:30.554691+0000","last_became_peered":"2024-04-02T07:54:30.554691+0000","last_unstale":"2024-04-02T07:54:31.594013+0000","last_undegraded":"2024-04-02T07:54:31.594013+0000","last_fullsized":"2024-04-02T07:54:31.594013+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:04.865135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592302+0000","last_change":"2024-04-02T07:54:30.551458+0000","last_active":"2024-04-02T07:54:31.592302+0000","last_peered":"2024-04-02T07:54:31.592302+0000","last_clean":"2024-04-02T07:54:31.592302+0000","last_became_active":"2024-04-02T07:54:30.551184+0000","last_became_peered":"2024-04-02T07:54:30.551184+0000","last_unstale":"2024-04-02T07:54:31.592302+0000","last_undegraded":"2024-04-02T07:54:31.592302+0000","last_fullsized":"2024-04-02T07:54:31.592302+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:43.308795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592835+0000","last_change":"2024-04-02T07:54:30.555269+0000","last_active":"2024-04-02T07:54:31.592835+0000","last_peered":"2024-04-02T07:54:31.592835+0000","last_clean":"2024-04-02T07:54:31.592835+0000","last_became_active":"2024-04-02T07:54:30.555033+0000","last_became_peered":"2024-04-02T07:54:30.555033+0000","last_unstale":"2024-04-02T07:54:31.592835+0000","last_undegraded":"2024-04-02T07:54:31.592835+0000","last_fullsized":"2024-04-02T07:54:31.592835+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:17:53.275607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592071+0000","last_change":"2024-04-02T07:54:30.539954+0000","last_active":"2024-04-02T07:54:31.592071+0000","last_peered":"2024-04-02T07:54:31.592071+0000","last_clean":"2024-04-02T07:54:31.592071+0000","last_became_active":"2024-04-02T07:54:30.539027+0000","last_became_peered":"2024-04-02T07:54:30.539027+0000","last_unstale":"2024-04-02T07:54:31.592071+0000","last_undegraded":"2024-04-02T07:54:31.592071+0000","last_fullsized":"2024-04-02T07:54:31.592071+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:06:12.744373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539089+0000","last_change":"2024-04-02T07:54:30.551965+0000","last_active":"2024-04-02T07:54:31.539089+0000","last_peered":"2024-04-02T07:54:31.539089+0000","last_clean":"2024-04-02T07:54:31.539089+0000","last_became_active":"2024-04-02T07:54:30.551594+0000","last_became_peered":"2024-04-02T07:54:30.551594+0000","last_unstale":"2024-04-02T07:54:31.539089+0000","last_undegraded":"2024-04-02T07:54:31.539089+0000","last_fullsized":"2024-04-02T07:54:31.539089+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:07:59.484212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592466+0000","last_change":"2024-04-02T07:54:30.547886+0000","last_active":"2024-04-02T07:54:31.592466+0000","last_peered":"2024-04-02T07:54:31.592466+0000","last_clean":"2024-04-02T07:54:31.592466+0000","last_became_active":"2024-04-02T07:54:30.547616+0000","last_became_peered":"2024-04-02T07:54:30.547616+0000","last_unstale":"2024-04-02T07:54:31.592466+0000","last_undegraded":"2024-04-02T07:54:31.592466+0000","last_fullsized":"2024-04-02T07:54:31.592466+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:55:03.097106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592367+0000","last_change":"2024-04-02T07:54:30.545914+0000","last_active":"2024-04-02T07:54:31.592367+0000","last_peered":"2024-04-02T07:54:31.592367+0000","last_clean":"2024-04-02T07:54:31.592367+0000","last_became_active":"2024-04-02T07:54:30.545686+0000","last_became_peered":"2024-04-02T07:54:30.545686+0000","last_unstale":"2024-04-02T07:54:31.592367+0000","last_undegraded":"2024-04-02T07:54:31.592367+0000","last_fullsized":"2024-04-02T07:54:31.592367+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:21.025069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592455+0000","last_change":"2024-04-02T07:54:30.557600+0000","last_active":"2024-04-02T07:54:31.592455+0000","last_peered":"2024-04-02T07:54:31.592455+0000","last_clean":"2024-04-02T07:54:31.592455+0000","last_became_active":"2024-04-02T07:54:30.557320+0000","last_became_peered":"2024-04-02T07:54:30.557320+0000","last_unstale":"2024-04-02T07:54:31.592455+0000","last_undegraded":"2024-04-02T07:54:31.592455+0000","last_fullsized":"2024-04-02T07:54:31.592455+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:17:49.546265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539335+0000","last_change":"2024-04-02T07:54:30.557533+0000","last_active":"2024-04-02T07:54:31.539335+0000","last_peered":"2024-04-02T07:54:31.539335+0000","last_clean":"2024-04-02T07:54:31.539335+0000","last_became_active":"2024-04-02T07:54:30.557289+0000","last_became_peered":"2024-04-02T07:54:30.557289+0000","last_unstale":"2024-04-02T07:54:31.539335+0000","last_undegraded":"2024-04-02T07:54:31.539335+0000","last_fullsized":"2024-04-02T07:54:31.539335+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:07:59.484212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593010+0000","last_change":"2024-04-02T07:54:30.551440+0000","last_active":"2024-04-02T07:54:31.593010+0000","last_peered":"2024-04-02T07:54:31.593010+0000","last_clean":"2024-04-02T07:54:31.593010+0000","last_became_active":"2024-04-02T07:54:30.551167+0000","last_became_peered":"2024-04-02T07:54:30.551167+0000","last_unstale":"2024-04-02T07:54:31.593010+0000","last_undegraded":"2024-04-02T07:54:31.593010+0000","last_fullsized":"2024-04-02T07:54:31.593010+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:17:18.604476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593433+0000","last_change":"2024-04-02T07:54:30.556468+0000","last_active":"2024-04-02T07:54:31.593433+0000","last_peered":"2024-04-02T07:54:31.593433+0000","last_clean":"2024-04-02T07:54:31.593433+0000","last_became_active":"2024-04-02T07:54:30.556179+0000","last_became_peered":"2024-04-02T07:54:30.556179+0000","last_unstale":"2024-04-02T07:54:31.593433+0000","last_undegraded":"2024-04-02T07:54:31.593433+0000","last_fullsized":"2024-04-02T07:54:31.593433+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:46.886600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593297+0000","last_change":"2024-04-02T07:54:30.552284+0000","last_active":"2024-04-02T07:54:31.593297+0000","last_peered":"2024-04-02T07:54:31.593297+0000","last_clean":"2024-04-02T07:54:31.593297+0000","last_became_active":"2024-04-02T07:54:30.551081+0000","last_became_peered":"2024-04-02T07:54:30.551081+0000","last_unstale":"2024-04-02T07:54:31.593297+0000","last_undegraded":"2024-04-02T07:54:31.593297+0000","last_fullsized":"2024-04-02T07:54:31.593297+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:49:19.684498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.591999+0000","last_change":"2024-04-02T07:54:30.549056+0000","last_active":"2024-04-02T07:54:31.591999+0000","last_peered":"2024-04-02T07:54:31.591999+0000","last_clean":"2024-04-02T07:54:31.591999+0000","last_became_active":"2024-04-02T07:54:30.548815+0000","last_became_peered":"2024-04-02T07:54:30.548815+0000","last_unstale":"2024-04-02T07:54:31.591999+0000","last_undegraded":"2024-04-02T07:54:31.591999+0000","last_fullsized":"2024-04-02T07:54:31.591999+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:27:06.489753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594456+0000","last_change":"2024-04-02T07:54:30.553528+0000","last_active":"2024-04-02T07:54:31.594456+0000","last_peered":"2024-04-02T07:54:31.594456+0000","last_clean":"2024-04-02T07:54:31.594456+0000","last_became_active":"2024-04-02T07:54:30.553196+0000","last_became_peered":"2024-04-02T07:54:30.553196+0000","last_unstale":"2024-04-02T07:54:31.594456+0000","last_undegraded":"2024-04-02T07:54:31.594456+0000","last_fullsized":"2024-04-02T07:54:31.594456+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:25:53.954726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593811+0000","last_change":"2024-04-02T07:54:30.562626+0000","last_active":"2024-04-02T07:54:31.593811+0000","last_peered":"2024-04-02T07:54:31.593811+0000","last_clean":"2024-04-02T07:54:31.593811+0000","last_became_active":"2024-04-02T07:54:30.559567+0000","last_became_peered":"2024-04-02T07:54:30.559567+0000","last_unstale":"2024-04-02T07:54:31.593811+0000","last_undegraded":"2024-04-02T07:54:31.593811+0000","last_fullsized":"2024-04-02T07:54:31.593811+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:54.425891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594034+0000","last_change":"2024-04-02T07:54:30.558683+0000","last_active":"2024-04-02T07:54:31.594034+0000","last_peered":"2024-04-02T07:54:31.594034+0000","last_clean":"2024-04-02T07:54:31.594034+0000","last_became_active":"2024-04-02T07:54:30.556568+0000","last_became_peered":"2024-04-02T07:54:30.556568+0000","last_unstale":"2024-04-02T07:54:31.594034+0000","last_undegraded":"2024-04-02T07:54:31.594034+0000","last_fullsized":"2024-04-02T07:54:31.594034+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:55:56.043668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539097+0000","last_change":"2024-04-02T07:54:30.557143+0000","last_active":"2024-04-02T07:54:31.539097+0000","last_peered":"2024-04-02T07:54:31.539097+0000","last_clean":"2024-04-02T07:54:31.539097+0000","last_became_active":"2024-04-02T07:54:30.556078+0000","last_became_peered":"2024-04-02T07:54:30.556078+0000","last_unstale":"2024-04-02T07:54:31.539097+0000","last_undegraded":"2024-04-02T07:54:31.539097+0000","last_fullsized":"2024-04-02T07:54:31.539097+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:02.505038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539960+0000","last_change":"2024-04-02T07:54:30.550725+0000","last_active":"2024-04-02T07:54:31.539960+0000","last_peered":"2024-04-02T07:54:31.539960+0000","last_clean":"2024-04-02T07:54:31.539960+0000","last_became_active":"2024-04-02T07:54:30.550383+0000","last_became_peered":"2024-04-02T07:54:30.550383+0000","last_unstale":"2024-04-02T07:54:31.539960+0000","last_undegraded":"2024-04-02T07:54:31.539960+0000","last_fullsized":"2024-04-02T07:54:31.539960+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:37:29.009138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539253+0000","last_change":"2024-04-02T07:54:30.546488+0000","last_active":"2024-04-02T07:54:31.539253+0000","last_peered":"2024-04-02T07:54:31.539253+0000","last_clean":"2024-04-02T07:54:31.539253+0000","last_became_active":"2024-04-02T07:54:30.543130+0000","last_became_peered":"2024-04-02T07:54:30.543130+0000","last_unstale":"2024-04-02T07:54:31.539253+0000","last_undegraded":"2024-04-02T07:54:31.539253+0000","last_fullsized":"2024-04-02T07:54:31.539253+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:04.865135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593387+0000","last_change":"2024-04-02T07:54:30.557275+0000","last_active":"2024-04-02T07:54:31.593387+0000","last_peered":"2024-04-02T07:54:31.593387+0000","last_clean":"2024-04-02T07:54:31.593387+0000","last_became_active":"2024-04-02T07:54:30.556925+0000","last_became_peered":"2024-04-02T07:54:30.556925+0000","last_unstale":"2024-04-02T07:54:31.593387+0000","last_undegraded":"2024-04-02T07:54:31.593387+0000","last_fullsized":"2024-04-02T07:54:31.593387+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:23:38.420697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.537549+0000","last_change":"2024-04-02T07:54:30.556437+0000","last_active":"2024-04-02T07:54:31.537549+0000","last_peered":"2024-04-02T07:54:31.537549+0000","last_clean":"2024-04-02T07:54:31.537549+0000","last_became_active":"2024-04-02T07:54:30.555167+0000","last_became_peered":"2024-04-02T07:54:30.555167+0000","last_unstale":"2024-04-02T07:54:31.537549+0000","last_undegraded":"2024-04-02T07:54:31.537549+0000","last_fullsized":"2024-04-02T07:54:31.537549+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:01:03.970262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592606+0000","last_change":"2024-04-02T07:54:30.544288+0000","last_active":"2024-04-02T07:54:31.592606+0000","last_peered":"2024-04-02T07:54:31.592606+0000","last_clean":"2024-04-02T07:54:31.592606+0000","last_became_active":"2024-04-02T07:54:30.544053+0000","last_became_peered":"2024-04-02T07:54:30.544053+0000","last_unstale":"2024-04-02T07:54:31.592606+0000","last_undegraded":"2024-04-02T07:54:31.592606+0000","last_fullsized":"2024-04-02T07:54:31.592606+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:40:39.992792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593418+0000","last_change":"2024-04-02T07:54:30.548292+0000","last_active":"2024-04-02T07:54:31.593418+0000","last_peered":"2024-04-02T07:54:31.593418+0000","last_clean":"2024-04-02T07:54:31.593418+0000","last_became_active":"2024-04-02T07:54:30.546897+0000","last_became_peered":"2024-04-02T07:54:30.546897+0000","last_unstale":"2024-04-02T07:54:31.593418+0000","last_undegraded":"2024-04-02T07:54:31.593418+0000","last_fullsized":"2024-04-02T07:54:31.593418+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:05.024378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538421+0000","last_change":"2024-04-02T07:54:30.548044+0000","last_active":"2024-04-02T07:54:31.538421+0000","last_peered":"2024-04-02T07:54:31.538421+0000","last_clean":"2024-04-02T07:54:31.538421+0000","last_became_active":"2024-04-02T07:54:30.547763+0000","last_became_peered":"2024-04-02T07:54:30.547763+0000","last_unstale":"2024-04-02T07:54:31.538421+0000","last_undegraded":"2024-04-02T07:54:31.538421+0000","last_fullsized":"2024-04-02T07:54:31.538421+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:07:59.484212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538212+0000","last_change":"2024-04-02T07:54:30.549092+0000","last_active":"2024-04-02T07:54:31.538212+0000","last_peered":"2024-04-02T07:54:31.538212+0000","last_clean":"2024-04-02T07:54:31.538212+0000","last_became_active":"2024-04-02T07:54:30.548099+0000","last_became_peered":"2024-04-02T07:54:30.548099+0000","last_unstale":"2024-04-02T07:54:31.538212+0000","last_undegraded":"2024-04-02T07:54:31.538212+0000","last_fullsized":"2024-04-02T07:54:31.538212+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:04.865135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593129+0000","last_change":"2024-04-02T07:54:30.548114+0000","last_active":"2024-04-02T07:54:31.593129+0000","last_peered":"2024-04-02T07:54:31.593129+0000","last_clean":"2024-04-02T07:54:31.593129+0000","last_became_active":"2024-04-02T07:54:30.547551+0000","last_became_peered":"2024-04-02T07:54:30.547551+0000","last_unstale":"2024-04-02T07:54:31.593129+0000","last_undegraded":"2024-04-02T07:54:31.593129+0000","last_fullsized":"2024-04-02T07:54:31.593129+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:34:44.239761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592859+0000","last_change":"2024-04-02T07:54:30.550274+0000","last_active":"2024-04-02T07:54:31.592859+0000","last_peered":"2024-04-02T07:54:31.592859+0000","last_clean":"2024-04-02T07:54:31.592859+0000","last_became_active":"2024-04-02T07:54:30.549924+0000","last_became_peered":"2024-04-02T07:54:30.549924+0000","last_unstale":"2024-04-02T07:54:31.592859+0000","last_undegraded":"2024-04-02T07:54:31.592859+0000","last_fullsized":"2024-04-02T07:54:31.592859+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:42:05.610304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539490+0000","last_change":"2024-04-02T07:54:30.545185+0000","last_active":"2024-04-02T07:54:31.539490+0000","last_peered":"2024-04-02T07:54:31.539490+0000","last_clean":"2024-04-02T07:54:31.539490+0000","last_became_active":"2024-04-02T07:54:30.544265+0000","last_became_peered":"2024-04-02T07:54:30.544265+0000","last_unstale":"2024-04-02T07:54:31.539490+0000","last_undegraded":"2024-04-02T07:54:31.539490+0000","last_fullsized":"2024-04-02T07:54:31.539490+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:59:37.327187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594119+0000","last_change":"2024-04-02T07:54:30.553233+0000","last_active":"2024-04-02T07:54:31.594119+0000","last_peered":"2024-04-02T07:54:31.594119+0000","last_clean":"2024-04-02T07:54:31.594119+0000","last_became_active":"2024-04-02T07:54:30.552587+0000","last_became_peered":"2024-04-02T07:54:30.552587+0000","last_unstale":"2024-04-02T07:54:31.594119+0000","last_undegraded":"2024-04-02T07:54:31.594119+0000","last_fullsized":"2024-04-02T07:54:31.594119+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:02.505038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594338+0000","last_change":"2024-04-02T07:54:30.563460+0000","last_active":"2024-04-02T07:54:31.594338+0000","last_peered":"2024-04-02T07:54:31.594338+0000","last_clean":"2024-04-02T07:54:31.594338+0000","last_became_active":"2024-04-02T07:54:30.562337+0000","last_became_peered":"2024-04-02T07:54:30.562337+0000","last_unstale":"2024-04-02T07:54:31.594338+0000","last_undegraded":"2024-04-02T07:54:31.594338+0000","last_fullsized":"2024-04-02T07:54:31.594338+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:26.433783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1f","version":"124'4","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593294+0000","last_change":"2024-04-02T07:54:30.050477+0000","last_active":"2024-04-02T07:54:31.593294+0000","last_peered":"2024-04-02T07:54:31.593294+0000","last_clean":"2024-04-02T07:54:31.593294+0000","last_became_active":"2024-04-02T07:54:30.050163+0000","last_became_peered":"2024-04-02T07:54:30.050163+0000","last_unstale":"2024-04-02T07:54:31.593294+0000","last_undegraded":"2024-04-02T07:54:31.593294+0000","last_fullsized":"2024-04-02T07:54:31.593294+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:17.245349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593191+0000","last_change":"2024-04-02T07:54:30.547234+0000","last_active":"2024-04-02T07:54:31.593191+0000","last_peered":"2024-04-02T07:54:31.593191+0000","last_clean":"2024-04-02T07:54:31.593191+0000","last_became_active":"2024-04-02T07:54:30.546928+0000","last_became_peered":"2024-04-02T07:54:30.546928+0000","last_unstale":"2024-04-02T07:54:31.593191+0000","last_undegraded":"2024-04-02T07:54:31.593191+0000","last_fullsized":"2024-04-02T07:54:31.593191+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:45:59.236866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1e","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592362+0000","last_change":"2024-04-02T07:54:30.052507+0000","last_active":"2024-04-02T07:54:31.592362+0000","last_peered":"2024-04-02T07:54:31.592362+0000","last_clean":"2024-04-02T07:54:31.592362+0000","last_became_active":"2024-04-02T07:54:30.052239+0000","last_became_peered":"2024-04-02T07:54:30.052239+0000","last_unstale":"2024-04-02T07:54:31.592362+0000","last_undegraded":"2024-04-02T07:54:31.592362+0000","last_fullsized":"2024-04-02T07:54:31.592362+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:19:33.356465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594027+0000","last_change":"2024-04-02T07:54:30.547257+0000","last_active":"2024-04-02T07:54:31.594027+0000","last_peered":"2024-04-02T07:54:31.594027+0000","last_clean":"2024-04-02T07:54:31.594027+0000","last_became_active":"2024-04-02T07:54:30.542861+0000","last_became_peered":"2024-04-02T07:54:30.542861+0000","last_unstale":"2024-04-02T07:54:31.594027+0000","last_undegraded":"2024-04-02T07:54:31.594027+0000","last_fullsized":"2024-04-02T07:54:31.594027+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:04:38.775670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539834+0000","last_change":"2024-04-02T07:54:30.056814+0000","last_active":"2024-04-02T07:54:31.539834+0000","last_peered":"2024-04-02T07:54:31.539834+0000","last_clean":"2024-04-02T07:54:31.539834+0000","last_became_active":"2024-04-02T07:54:30.056564+0000","last_became_peered":"2024-04-02T07:54:30.056564+0000","last_unstale":"2024-04-02T07:54:31.539834+0000","last_undegraded":"2024-04-02T07:54:31.539834+0000","last_fullsized":"2024-04-02T07:54:31.539834+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:35.636951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592148+0000","last_change":"2024-04-02T07:54:30.545754+0000","last_active":"2024-04-02T07:54:31.592148+0000","last_peered":"2024-04-02T07:54:31.592148+0000","last_clean":"2024-04-02T07:54:31.592148+0000","last_became_active":"2024-04-02T07:54:30.545507+0000","last_became_peered":"2024-04-02T07:54:30.545507+0000","last_unstale":"2024-04-02T07:54:31.592148+0000","last_undegraded":"2024-04-02T07:54:31.592148+0000","last_fullsized":"2024-04-02T07:54:31.592148+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:26:33.418627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594119+0000","last_change":"2024-04-02T07:54:30.060771+0000","last_active":"2024-04-02T07:54:31.594119+0000","last_peered":"2024-04-02T07:54:31.594119+0000","last_clean":"2024-04-02T07:54:31.594119+0000","last_became_active":"2024-04-02T07:54:30.060435+0000","last_became_peered":"2024-04-02T07:54:30.060435+0000","last_unstale":"2024-04-02T07:54:31.594119+0000","last_undegraded":"2024-04-02T07:54:31.594119+0000","last_fullsized":"2024-04-02T07:54:31.594119+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:33:07.449281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538928+0000","last_change":"2024-04-02T07:54:30.549044+0000","last_active":"2024-04-02T07:54:31.538928+0000","last_peered":"2024-04-02T07:54:31.538928+0000","last_clean":"2024-04-02T07:54:31.538928+0000","last_became_active":"2024-04-02T07:54:30.548799+0000","last_became_peered":"2024-04-02T07:54:30.548799+0000","last_unstale":"2024-04-02T07:54:31.538928+0000","last_undegraded":"2024-04-02T07:54:31.538928+0000","last_fullsized":"2024-04-02T07:54:31.538928+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:54:19.338399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1b","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593338+0000","last_change":"2024-04-02T07:54:30.053735+0000","last_active":"2024-04-02T07:54:31.593338+0000","last_peered":"2024-04-02T07:54:31.593338+0000","last_clean":"2024-04-02T07:54:31.593338+0000","last_became_active":"2024-04-02T07:54:30.050754+0000","last_became_peered":"2024-04-02T07:54:30.050754+0000","last_unstale":"2024-04-02T07:54:31.593338+0000","last_undegraded":"2024-04-02T07:54:31.593338+0000","last_fullsized":"2024-04-02T07:54:31.593338+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:17.245349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592473+0000","last_change":"2024-04-02T07:54:30.555339+0000","last_active":"2024-04-02T07:54:31.592473+0000","last_peered":"2024-04-02T07:54:31.592473+0000","last_clean":"2024-04-02T07:54:31.592473+0000","last_became_active":"2024-04-02T07:54:30.555072+0000","last_became_peered":"2024-04-02T07:54:30.555072+0000","last_unstale":"2024-04-02T07:54:31.592473+0000","last_undegraded":"2024-04-02T07:54:31.592473+0000","last_fullsized":"2024-04-02T07:54:31.592473+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:08:21.920480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1a","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539348+0000","last_change":"2024-04-02T07:54:30.050501+0000","last_active":"2024-04-02T07:54:31.539348+0000","last_peered":"2024-04-02T07:54:31.539348+0000","last_clean":"2024-04-02T07:54:31.539348+0000","last_became_active":"2024-04-02T07:54:30.050225+0000","last_became_peered":"2024-04-02T07:54:30.050225+0000","last_unstale":"2024-04-02T07:54:31.539348+0000","last_undegraded":"2024-04-02T07:54:31.539348+0000","last_fullsized":"2024-04-02T07:54:31.539348+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:00:49.879036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592528+0000","last_change":"2024-04-02T07:54:30.561628+0000","last_active":"2024-04-02T07:54:31.592528+0000","last_peered":"2024-04-02T07:54:31.592528+0000","last_clean":"2024-04-02T07:54:31.592528+0000","last_became_active":"2024-04-02T07:54:30.561355+0000","last_became_peered":"2024-04-02T07:54:30.561355+0000","last_unstale":"2024-04-02T07:54:31.592528+0000","last_undegraded":"2024-04-02T07:54:31.592528+0000","last_fullsized":"2024-04-02T07:54:31.592528+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:47.419730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592149+0000","last_change":"2024-04-02T07:54:30.043077+0000","last_active":"2024-04-02T07:54:31.592149+0000","last_peered":"2024-04-02T07:54:31.592149+0000","last_clean":"2024-04-02T07:54:31.592149+0000","last_became_active":"2024-04-02T07:54:30.042794+0000","last_became_peered":"2024-04-02T07:54:30.042794+0000","last_unstale":"2024-04-02T07:54:31.592149+0000","last_undegraded":"2024-04-02T07:54:31.592149+0000","last_fullsized":"2024-04-02T07:54:31.592149+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:38:25.427335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538803+0000","last_change":"2024-04-02T07:54:30.544448+0000","last_active":"2024-04-02T07:54:31.538803+0000","last_peered":"2024-04-02T07:54:31.538803+0000","last_clean":"2024-04-02T07:54:31.538803+0000","last_became_active":"2024-04-02T07:54:30.543985+0000","last_became_peered":"2024-04-02T07:54:30.543985+0000","last_unstale":"2024-04-02T07:54:31.538803+0000","last_undegraded":"2024-04-02T07:54:31.538803+0000","last_fullsized":"2024-04-02T07:54:31.538803+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:04.865135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593676+0000","last_change":"2024-04-02T07:54:30.054090+0000","last_active":"2024-04-02T07:54:31.593676+0000","last_peered":"2024-04-02T07:54:31.593676+0000","last_clean":"2024-04-02T07:54:31.593676+0000","last_became_active":"2024-04-02T07:54:30.053814+0000","last_became_peered":"2024-04-02T07:54:30.053814+0000","last_unstale":"2024-04-02T07:54:31.593676+0000","last_undegraded":"2024-04-02T07:54:31.593676+0000","last_fullsized":"2024-04-02T07:54:31.593676+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:45:31.896403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593888+0000","last_change":"2024-04-02T07:54:30.555846+0000","last_active":"2024-04-02T07:54:31.593888+0000","last_peered":"2024-04-02T07:54:31.593888+0000","last_clean":"2024-04-02T07:54:31.593888+0000","last_became_active":"2024-04-02T07:54:30.555556+0000","last_became_peered":"2024-04-02T07:54:30.555556+0000","last_unstale":"2024-04-02T07:54:31.593888+0000","last_undegraded":"2024-04-02T07:54:31.593888+0000","last_fullsized":"2024-04-02T07:54:31.593888+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:37:29.009138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.540066+0000","last_change":"2024-04-02T07:54:30.046428+0000","last_active":"2024-04-02T07:54:31.540066+0000","last_peered":"2024-04-02T07:54:31.540066+0000","last_clean":"2024-04-02T07:54:31.540066+0000","last_became_active":"2024-04-02T07:54:30.045802+0000","last_became_peered":"2024-04-02T07:54:30.045802+0000","last_unstale":"2024-04-02T07:54:31.540066+0000","last_undegraded":"2024-04-02T07:54:31.540066+0000","last_fullsized":"2024-04-02T07:54:31.540066+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:33:07.449281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538913+0000","last_change":"2024-04-02T07:54:30.541154+0000","last_active":"2024-04-02T07:54:31.538913+0000","last_peered":"2024-04-02T07:54:31.538913+0000","last_clean":"2024-04-02T07:54:31.538913+0000","last_became_active":"2024-04-02T07:54:30.540786+0000","last_became_peered":"2024-04-02T07:54:30.540786+0000","last_unstale":"2024-04-02T07:54:31.538913+0000","last_undegraded":"2024-04-02T07:54:31.538913+0000","last_fullsized":"2024-04-02T07:54:31.538913+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:04:38.775670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593890+0000","last_change":"2024-04-02T07:54:30.056988+0000","last_active":"2024-04-02T07:54:31.593890+0000","last_peered":"2024-04-02T07:54:31.593890+0000","last_clean":"2024-04-02T07:54:31.593890+0000","last_became_active":"2024-04-02T07:54:30.056721+0000","last_became_peered":"2024-04-02T07:54:30.056721+0000","last_unstale":"2024-04-02T07:54:31.593890+0000","last_undegraded":"2024-04-02T07:54:31.593890+0000","last_fullsized":"2024-04-02T07:54:31.593890+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:52.812327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592529+0000","last_change":"2024-04-02T07:54:30.546859+0000","last_active":"2024-04-02T07:54:31.592529+0000","last_peered":"2024-04-02T07:54:31.592529+0000","last_clean":"2024-04-02T07:54:31.592529+0000","last_became_active":"2024-04-02T07:54:30.544420+0000","last_became_peered":"2024-04-02T07:54:30.544420+0000","last_unstale":"2024-04-02T07:54:31.592529+0000","last_undegraded":"2024-04-02T07:54:31.592529+0000","last_fullsized":"2024-04-02T07:54:31.592529+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:50:40.545354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.15","version":"124'3","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592546+0000","last_change":"2024-04-02T07:54:30.047690+0000","last_active":"2024-04-02T07:54:31.592546+0000","last_peered":"2024-04-02T07:54:31.592546+0000","last_clean":"2024-04-02T07:54:31.592546+0000","last_became_active":"2024-04-02T07:54:30.047423+0000","last_became_peered":"2024-04-02T07:54:30.047423+0000","last_unstale":"2024-04-02T07:54:31.592546+0000","last_undegraded":"2024-04-02T07:54:31.592546+0000","last_fullsized":"2024-04-02T07:54:31.592546+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:44.349638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538433+0000","last_change":"2024-04-02T07:54:30.546888+0000","last_active":"2024-04-02T07:54:31.538433+0000","last_peered":"2024-04-02T07:54:31.538433+0000","last_clean":"2024-04-02T07:54:31.538433+0000","last_became_active":"2024-04-02T07:54:30.545311+0000","last_became_peered":"2024-04-02T07:54:30.545311+0000","last_unstale":"2024-04-02T07:54:31.538433+0000","last_undegraded":"2024-04-02T07:54:31.538433+0000","last_fullsized":"2024-04-02T07:54:31.538433+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:46.886600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.14","version":"124'6","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539287+0000","last_change":"2024-04-02T07:54:30.052792+0000","last_active":"2024-04-02T07:54:31.539287+0000","last_peered":"2024-04-02T07:54:31.539287+0000","last_clean":"2024-04-02T07:54:31.539287+0000","last_became_active":"2024-04-02T07:54:30.052544+0000","last_became_peered":"2024-04-02T07:54:30.052544+0000","last_unstale":"2024-04-02T07:54:31.539287+0000","last_undegraded":"2024-04-02T07:54:31.539287+0000","last_fullsized":"2024-04-02T07:54:31.539287+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:34:43.233313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594015+0000","last_change":"2024-04-02T07:54:30.544160+0000","last_active":"2024-04-02T07:54:31.594015+0000","last_peered":"2024-04-02T07:54:31.594015+0000","last_clean":"2024-04-02T07:54:31.594015+0000","last_became_active":"2024-04-02T07:54:30.543902+0000","last_became_peered":"2024-04-02T07:54:30.543902+0000","last_unstale":"2024-04-02T07:54:31.594015+0000","last_undegraded":"2024-04-02T07:54:31.594015+0000","last_fullsized":"2024-04-02T07:54:31.594015+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:59:37.327187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.13","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593607+0000","last_change":"2024-04-02T07:54:30.053291+0000","last_active":"2024-04-02T07:54:31.593607+0000","last_peered":"2024-04-02T07:54:31.593607+0000","last_clean":"2024-04-02T07:54:31.593607+0000","last_became_active":"2024-04-02T07:54:30.052011+0000","last_became_peered":"2024-04-02T07:54:30.052011+0000","last_unstale":"2024-04-02T07:54:31.593607+0000","last_undegraded":"2024-04-02T07:54:31.593607+0000","last_fullsized":"2024-04-02T07:54:31.593607+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:52:21.754612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539925+0000","last_change":"2024-04-02T07:54:30.550714+0000","last_active":"2024-04-02T07:54:31.539925+0000","last_peered":"2024-04-02T07:54:31.539925+0000","last_clean":"2024-04-02T07:54:31.539925+0000","last_became_active":"2024-04-02T07:54:30.549286+0000","last_became_peered":"2024-04-02T07:54:30.549286+0000","last_unstale":"2024-04-02T07:54:31.539925+0000","last_undegraded":"2024-04-02T07:54:31.539925+0000","last_fullsized":"2024-04-02T07:54:31.539925+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:54:19.338399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594116+0000","last_change":"2024-04-02T07:54:30.059972+0000","last_active":"2024-04-02T07:54:31.594116+0000","last_peered":"2024-04-02T07:54:31.594116+0000","last_clean":"2024-04-02T07:54:31.594116+0000","last_became_active":"2024-04-02T07:54:30.059250+0000","last_became_peered":"2024-04-02T07:54:30.059250+0000","last_unstale":"2024-04-02T07:54:31.594116+0000","last_undegraded":"2024-04-02T07:54:31.594116+0000","last_fullsized":"2024-04-02T07:54:31.594116+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:54.278160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592074+0000","last_change":"2024-04-02T07:54:30.550612+0000","last_active":"2024-04-02T07:54:31.592074+0000","last_peered":"2024-04-02T07:54:31.592074+0000","last_clean":"2024-04-02T07:54:31.592074+0000","last_became_active":"2024-04-02T07:54:30.550355+0000","last_became_peered":"2024-04-02T07:54:30.550355+0000","last_unstale":"2024-04-02T07:54:31.592074+0000","last_undegraded":"2024-04-02T07:54:31.592074+0000","last_fullsized":"2024-04-02T07:54:31.592074+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:06:21.694201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538892+0000","last_change":"2024-04-02T07:54:30.045344+0000","last_active":"2024-04-02T07:54:31.538892+0000","last_peered":"2024-04-02T07:54:31.538892+0000","last_clean":"2024-04-02T07:54:31.538892+0000","last_became_active":"2024-04-02T07:54:30.045114+0000","last_became_peered":"2024-04-02T07:54:30.045114+0000","last_unstale":"2024-04-02T07:54:31.538892+0000","last_undegraded":"2024-04-02T07:54:31.538892+0000","last_fullsized":"2024-04-02T07:54:31.538892+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:04.017930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539538+0000","last_change":"2024-04-02T07:54:30.545113+0000","last_active":"2024-04-02T07:54:31.539538+0000","last_peered":"2024-04-02T07:54:31.539538+0000","last_clean":"2024-04-02T07:54:31.539538+0000","last_became_active":"2024-04-02T07:54:30.543617+0000","last_became_peered":"2024-04-02T07:54:30.543617+0000","last_unstale":"2024-04-02T07:54:31.539538+0000","last_undegraded":"2024-04-02T07:54:31.539538+0000","last_fullsized":"2024-04-02T07:54:31.539538+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:02.505038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.10","version":"124'8","reported_seq":19,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593871+0000","last_change":"2024-04-02T07:54:30.055148+0000","last_active":"2024-04-02T07:54:31.593871+0000","last_peered":"2024-04-02T07:54:31.593871+0000","last_clean":"2024-04-02T07:54:31.593871+0000","last_became_active":"2024-04-02T07:54:30.054904+0000","last_became_peered":"2024-04-02T07:54:30.054904+0000","last_unstale":"2024-04-02T07:54:31.593871+0000","last_undegraded":"2024-04-02T07:54:31.593871+0000","last_fullsized":"2024-04-02T07:54:31.593871+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:16:23.501686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539064+0000","last_change":"2024-04-02T07:54:30.551645+0000","last_active":"2024-04-02T07:54:31.539064+0000","last_peered":"2024-04-02T07:54:31.539064+0000","last_clean":"2024-04-02T07:54:31.539064+0000","last_became_active":"2024-04-02T07:54:30.551365+0000","last_became_peered":"2024-04-02T07:54:30.551365+0000","last_unstale":"2024-04-02T07:54:31.539064+0000","last_undegraded":"2024-04-02T07:54:31.539064+0000","last_fullsized":"2024-04-02T07:54:31.539064+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:07:02.505038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.e","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539318+0000","last_change":"2024-04-02T07:54:30.053094+0000","last_active":"2024-04-02T07:54:31.539318+0000","last_peered":"2024-04-02T07:54:31.539318+0000","last_clean":"2024-04-02T07:54:31.539318+0000","last_became_active":"2024-04-02T07:54:30.052016+0000","last_became_peered":"2024-04-02T07:54:30.052016+0000","last_unstale":"2024-04-02T07:54:31.539318+0000","last_undegraded":"2024-04-02T07:54:31.539318+0000","last_fullsized":"2024-04-02T07:54:31.539318+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:47:46.413282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538261+0000","last_change":"2024-04-02T07:54:30.558623+0000","last_active":"2024-04-02T07:54:31.538261+0000","last_peered":"2024-04-02T07:54:31.538261+0000","last_clean":"2024-04-02T07:54:31.538261+0000","last_became_active":"2024-04-02T07:54:30.556706+0000","last_became_peered":"2024-04-02T07:54:30.556706+0000","last_unstale":"2024-04-02T07:54:31.538261+0000","last_undegraded":"2024-04-02T07:54:31.538261+0000","last_fullsized":"2024-04-02T07:54:31.538261+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:54:19.338399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.d","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592740+0000","last_change":"2024-04-02T07:54:30.049156+0000","last_active":"2024-04-02T07:54:31.592740+0000","last_peered":"2024-04-02T07:54:31.592740+0000","last_clean":"2024-04-02T07:54:31.592740+0000","last_became_active":"2024-04-02T07:54:30.048701+0000","last_became_peered":"2024-04-02T07:54:30.048701+0000","last_unstale":"2024-04-02T07:54:31.592740+0000","last_undegraded":"2024-04-02T07:54:31.592740+0000","last_fullsized":"2024-04-02T07:54:31.592740+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:55:55.037220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539225+0000","last_change":"2024-04-02T07:54:30.553351+0000","last_active":"2024-04-02T07:54:31.539225+0000","last_peered":"2024-04-02T07:54:31.539225+0000","last_clean":"2024-04-02T07:54:31.539225+0000","last_became_active":"2024-04-02T07:54:30.552907+0000","last_became_peered":"2024-04-02T07:54:30.552907+0000","last_unstale":"2024-04-02T07:54:31.539225+0000","last_undegraded":"2024-04-02T07:54:31.539225+0000","last_fullsized":"2024-04-02T07:54:31.539225+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:04:38.775670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593783+0000","last_change":"2024-04-02T07:54:30.052611+0000","last_active":"2024-04-02T07:54:31.593783+0000","last_peered":"2024-04-02T07:54:31.593783+0000","last_clean":"2024-04-02T07:54:31.593783+0000","last_became_active":"2024-04-02T07:54:30.052373+0000","last_became_peered":"2024-04-02T07:54:30.052373+0000","last_unstale":"2024-04-02T07:54:31.593783+0000","last_undegraded":"2024-04-02T07:54:31.593783+0000","last_fullsized":"2024-04-02T07:54:31.593783+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:29:03.725237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592786+0000","last_change":"2024-04-02T07:54:30.551933+0000","last_active":"2024-04-02T07:54:31.592786+0000","last_peered":"2024-04-02T07:54:31.592786+0000","last_clean":"2024-04-02T07:54:31.592786+0000","last_became_active":"2024-04-02T07:54:30.551519+0000","last_became_peered":"2024-04-02T07:54:30.551519+0000","last_unstale":"2024-04-02T07:54:31.592786+0000","last_undegraded":"2024-04-02T07:54:31.592786+0000","last_fullsized":"2024-04-02T07:54:31.592786+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:02:19.300367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593436+0000","last_change":"2024-04-02T07:54:30.060771+0000","last_active":"2024-04-02T07:54:31.593436+0000","last_peered":"2024-04-02T07:54:31.593436+0000","last_clean":"2024-04-02T07:54:31.593436+0000","last_became_active":"2024-04-02T07:54:30.060518+0000","last_became_peered":"2024-04-02T07:54:30.060518+0000","last_unstale":"2024-04-02T07:54:31.593436+0000","last_undegraded":"2024-04-02T07:54:31.593436+0000","last_fullsized":"2024-04-02T07:54:31.593436+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:35:18.551498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592845+0000","last_change":"2024-04-02T07:54:30.554540+0000","last_active":"2024-04-02T07:54:31.592845+0000","last_peered":"2024-04-02T07:54:31.592845+0000","last_clean":"2024-04-02T07:54:31.592845+0000","last_became_active":"2024-04-02T07:54:30.554239+0000","last_became_peered":"2024-04-02T07:54:30.554239+0000","last_unstale":"2024-04-02T07:54:31.592845+0000","last_undegraded":"2024-04-02T07:54:31.592845+0000","last_fullsized":"2024-04-02T07:54:31.592845+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:45.356086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.537968+0000","last_change":"2024-04-02T07:54:30.045190+0000","last_active":"2024-04-02T07:54:31.537968+0000","last_peered":"2024-04-02T07:54:31.537968+0000","last_clean":"2024-04-02T07:54:31.537968+0000","last_became_active":"2024-04-02T07:54:30.044075+0000","last_became_peered":"2024-04-02T07:54:30.044075+0000","last_unstale":"2024-04-02T07:54:31.537968+0000","last_undegraded":"2024-04-02T07:54:31.537968+0000","last_fullsized":"2024-04-02T07:54:31.537968+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:04.017930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593692+0000","last_change":"2024-04-02T07:54:30.553411+0000","last_active":"2024-04-02T07:54:31.593692+0000","last_peered":"2024-04-02T07:54:31.593692+0000","last_clean":"2024-04-02T07:54:31.593692+0000","last_became_active":"2024-04-02T07:54:30.552913+0000","last_became_peered":"2024-04-02T07:54:30.552913+0000","last_unstale":"2024-04-02T07:54:31.593692+0000","last_undegraded":"2024-04-02T07:54:31.593692+0000","last_fullsized":"2024-04-02T07:54:31.593692+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:58:10.742469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592224+0000","last_change":"2024-04-02T07:54:30.043594+0000","last_active":"2024-04-02T07:54:31.592224+0000","last_peered":"2024-04-02T07:54:31.592224+0000","last_clean":"2024-04-02T07:54:31.592224+0000","last_became_active":"2024-04-02T07:54:30.043350+0000","last_became_peered":"2024-04-02T07:54:30.043350+0000","last_unstale":"2024-04-02T07:54:31.592224+0000","last_undegraded":"2024-04-02T07:54:31.592224+0000","last_fullsized":"2024-04-02T07:54:31.592224+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:58:42.614095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.537732+0000","last_change":"2024-04-02T07:54:30.542011+0000","last_active":"2024-04-02T07:54:31.537732+0000","last_peered":"2024-04-02T07:54:31.537732+0000","last_clean":"2024-04-02T07:54:31.537732+0000","last_became_active":"2024-04-02T07:54:30.541726+0000","last_became_peered":"2024-04-02T07:54:30.541726+0000","last_unstale":"2024-04-02T07:54:31.537732+0000","last_undegraded":"2024-04-02T07:54:31.537732+0000","last_fullsized":"2024-04-02T07:54:31.537732+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:37:29.009138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539180+0000","last_change":"2024-04-02T07:54:30.053908+0000","last_active":"2024-04-02T07:54:31.539180+0000","last_peered":"2024-04-02T07:54:31.539180+0000","last_clean":"2024-04-02T07:54:31.539180+0000","last_became_active":"2024-04-02T07:54:30.052545+0000","last_became_peered":"2024-04-02T07:54:30.052545+0000","last_unstale":"2024-04-02T07:54:31.539180+0000","last_undegraded":"2024-04-02T07:54:31.539180+0000","last_fullsized":"2024-04-02T07:54:31.539180+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:54:28.788127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594350+0000","last_change":"2024-04-02T07:54:30.546063+0000","last_active":"2024-04-02T07:54:31.594350+0000","last_peered":"2024-04-02T07:54:31.594350+0000","last_clean":"2024-04-02T07:54:31.594350+0000","last_became_active":"2024-04-02T07:54:30.544123+0000","last_became_peered":"2024-04-02T07:54:30.544123+0000","last_unstale":"2024-04-02T07:54:31.594350+0000","last_undegraded":"2024-04-02T07:54:31.594350+0000","last_fullsized":"2024-04-02T07:54:31.594350+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:55.892015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.7","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539384+0000","last_change":"2024-04-02T07:54:30.053485+0000","last_active":"2024-04-02T07:54:31.539384+0000","last_peered":"2024-04-02T07:54:31.539384+0000","last_clean":"2024-04-02T07:54:31.539384+0000","last_became_active":"2024-04-02T07:54:30.051982+0000","last_became_peered":"2024-04-02T07:54:30.051982+0000","last_unstale":"2024-04-02T07:54:31.539384+0000","last_undegraded":"2024-04-02T07:54:31.539384+0000","last_fullsized":"2024-04-02T07:54:31.539384+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:35.636951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594524+0000","last_change":"2024-04-02T07:54:30.555465+0000","last_active":"2024-04-02T07:54:31.594524+0000","last_peered":"2024-04-02T07:54:31.594524+0000","last_clean":"2024-04-02T07:54:31.594524+0000","last_became_active":"2024-04-02T07:54:30.555139+0000","last_became_peered":"2024-04-02T07:54:30.555139+0000","last_unstale":"2024-04-02T07:54:31.594524+0000","last_undegraded":"2024-04-02T07:54:31.594524+0000","last_fullsized":"2024-04-02T07:54:31.594524+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:19:34.362913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594456+0000","last_change":"2024-04-02T07:54:30.056940+0000","last_active":"2024-04-02T07:54:31.594456+0000","last_peered":"2024-04-02T07:54:31.594456+0000","last_clean":"2024-04-02T07:54:31.594456+0000","last_became_active":"2024-04-02T07:54:30.056590+0000","last_became_peered":"2024-04-02T07:54:30.056590+0000","last_unstale":"2024-04-02T07:54:31.594456+0000","last_undegraded":"2024-04-02T07:54:31.594456+0000","last_fullsized":"2024-04-02T07:54:31.594456+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:26:21.428369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539877+0000","last_change":"2024-04-02T07:54:30.557689+0000","last_active":"2024-04-02T07:54:31.539877+0000","last_peered":"2024-04-02T07:54:31.539877+0000","last_clean":"2024-04-02T07:54:31.539877+0000","last_became_active":"2024-04-02T07:54:30.557430+0000","last_became_peered":"2024-04-02T07:54:30.557430+0000","last_unstale":"2024-04-02T07:54:31.539877+0000","last_undegraded":"2024-04-02T07:54:31.539877+0000","last_fullsized":"2024-04-02T07:54:31.539877+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:01:03.970262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.594164+0000","last_change":"2024-04-02T07:54:30.053073+0000","last_active":"2024-04-02T07:54:31.594164+0000","last_peered":"2024-04-02T07:54:31.594164+0000","last_clean":"2024-04-02T07:54:31.594164+0000","last_became_active":"2024-04-02T07:54:30.052801+0000","last_became_peered":"2024-04-02T07:54:30.052801+0000","last_unstale":"2024-04-02T07:54:31.594164+0000","last_undegraded":"2024-04-02T07:54:31.594164+0000","last_fullsized":"2024-04-02T07:54:31.594164+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:57.107798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592643+0000","last_change":"2024-04-02T07:54:30.541572+0000","last_active":"2024-04-02T07:54:31.592643+0000","last_peered":"2024-04-02T07:54:31.592643+0000","last_clean":"2024-04-02T07:54:31.592643+0000","last_became_active":"2024-04-02T07:54:30.541262+0000","last_became_peered":"2024-04-02T07:54:30.541262+0000","last_unstale":"2024-04-02T07:54:31.592643+0000","last_undegraded":"2024-04-02T07:54:31.592643+0000","last_fullsized":"2024-04-02T07:54:31.592643+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:36:29.743050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538294+0000","last_change":"2024-04-02T07:54:30.050096+0000","last_active":"2024-04-02T07:54:31.538294+0000","last_peered":"2024-04-02T07:54:31.538294+0000","last_clean":"2024-04-02T07:54:31.538294+0000","last_became_active":"2024-04-02T07:54:30.049652+0000","last_became_peered":"2024-04-02T07:54:30.049652+0000","last_unstale":"2024-04-02T07:54:31.538294+0000","last_undegraded":"2024-04-02T07:54:31.538294+0000","last_fullsized":"2024-04-02T07:54:31.538294+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:54:28.788127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592844+0000","last_change":"2024-04-02T07:54:30.547960+0000","last_active":"2024-04-02T07:54:31.592844+0000","last_peered":"2024-04-02T07:54:31.592844+0000","last_clean":"2024-04-02T07:54:31.592844+0000","last_became_active":"2024-04-02T07:54:30.547684+0000","last_became_peered":"2024-04-02T07:54:30.547684+0000","last_unstale":"2024-04-02T07:54:31.592844+0000","last_undegraded":"2024-04-02T07:54:31.592844+0000","last_fullsized":"2024-04-02T07:54:31.592844+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:36:35.379755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.540052+0000","last_change":"2024-04-02T07:54:30.058287+0000","last_active":"2024-04-02T07:54:31.540052+0000","last_peered":"2024-04-02T07:54:31.540052+0000","last_clean":"2024-04-02T07:54:31.540052+0000","last_became_active":"2024-04-02T07:54:30.055165+0000","last_became_peered":"2024-04-02T07:54:30.055165+0000","last_unstale":"2024-04-02T07:54:31.540052+0000","last_undegraded":"2024-04-02T07:54:31.540052+0000","last_fullsized":"2024-04-02T07:54:31.540052+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:35:18.551498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592568+0000","last_change":"2024-04-02T07:54:30.552203+0000","last_active":"2024-04-02T07:54:31.592568+0000","last_peered":"2024-04-02T07:54:31.592568+0000","last_clean":"2024-04-02T07:54:31.592568+0000","last_became_active":"2024-04-02T07:54:30.551933+0000","last_became_peered":"2024-04-02T07:54:30.551933+0000","last_unstale":"2024-04-02T07:54:31.592568+0000","last_undegraded":"2024-04-02T07:54:31.592568+0000","last_fullsized":"2024-04-02T07:54:31.592568+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:10:35.288763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538019+0000","last_change":"2024-04-02T07:54:30.052250+0000","last_active":"2024-04-02T07:54:31.538019+0000","last_peered":"2024-04-02T07:54:31.538019+0000","last_clean":"2024-04-02T07:54:31.538019+0000","last_became_active":"2024-04-02T07:54:30.051980+0000","last_became_peered":"2024-04-02T07:54:30.051980+0000","last_unstale":"2024-04-02T07:54:31.538019+0000","last_undegraded":"2024-04-02T07:54:31.538019+0000","last_fullsized":"2024-04-02T07:54:31.538019+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:00:49.879036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.593574+0000","last_change":"2024-04-02T07:54:30.546191+0000","last_active":"2024-04-02T07:54:31.593574+0000","last_peered":"2024-04-02T07:54:31.593574+0000","last_clean":"2024-04-02T07:54:31.593574+0000","last_became_active":"2024-04-02T07:54:30.545517+0000","last_became_peered":"2024-04-02T07:54:30.545517+0000","last_unstale":"2024-04-02T07:54:31.593574+0000","last_undegraded":"2024-04-02T07:54:31.593574+0000","last_fullsized":"2024-04-02T07:54:31.593574+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:03:09.202047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.592755+0000","last_change":"2024-04-02T07:54:30.058271+0000","last_active":"2024-04-02T07:54:31.592755+0000","last_peered":"2024-04-02T07:54:31.592755+0000","last_clean":"2024-04-02T07:54:31.592755+0000","last_became_active":"2024-04-02T07:54:30.057968+0000","last_became_peered":"2024-04-02T07:54:30.057968+0000","last_unstale":"2024-04-02T07:54:31.592755+0000","last_undegraded":"2024-04-02T07:54:31.592755+0000","last_fullsized":"2024-04-02T07:54:31.592755+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:54.278160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.538867+0000","last_change":"2024-04-02T07:54:30.558497+0000","last_active":"2024-04-02T07:54:31.538867+0000","last_peered":"2024-04-02T07:54:31.538867+0000","last_clean":"2024-04-02T07:54:31.538867+0000","last_became_active":"2024-04-02T07:54:30.557865+0000","last_became_peered":"2024-04-02T07:54:30.557865+0000","last_unstale":"2024-04-02T07:54:31.538867+0000","last_undegraded":"2024-04-02T07:54:31.538867+0000","last_fullsized":"2024-04-02T07:54:31.538867+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:01:03.970262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539200+0000","last_change":"2024-04-02T07:54:30.056409+0000","last_active":"2024-04-02T07:54:31.539200+0000","last_peered":"2024-04-02T07:54:31.539200+0000","last_clean":"2024-04-02T07:54:31.539200+0000","last_became_active":"2024-04-02T07:54:30.056116+0000","last_became_peered":"2024-04-02T07:54:30.056116+0000","last_unstale":"2024-04-02T07:54:31.539200+0000","last_undegraded":"2024-04-02T07:54:31.539200+0000","last_fullsized":"2024-04-02T07:54:31.539200+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:49:52.812327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.540063+0000","last_change":"2024-04-02T07:54:30.555467+0000","last_active":"2024-04-02T07:54:31.540063+0000","last_peered":"2024-04-02T07:54:31.540063+0000","last_clean":"2024-04-02T07:54:31.540063+0000","last_became_active":"2024-04-02T07:54:30.552874+0000","last_became_peered":"2024-04-02T07:54:30.552874+0000","last_unstale":"2024-04-02T07:54:31.540063+0000","last_undegraded":"2024-04-02T07:54:31.540063+0000","last_fullsized":"2024-04-02T07:54:31.540063+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:30.025981+0000","last_clean_scrub_stamp":"2024-04-02T07:54:30.025981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:54.425891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.f","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-02T07:54:31.539356+0000","last_change":"2024-04-02T07:54:30.056911+0000","last_active":"2024-04-02T07:54:31.539356+0000","last_peered":"2024-04-02T07:54:31.539356+0000","last_clean":"2024-04-02T07:54:31.539356+0000","last_became_active":"2024-04-02T07:54:30.056710+0000","last_became_peered":"2024-04-02T07:54:30.056710+0000","last_unstale":"2024-04-02T07:54:31.539356+0000","last_undegraded":"2024-04-02T07:54:31.539356+0000","last_fullsized":"2024-04-02T07:54:31.539356+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:54:29.019533+0000","last_clean_scrub_stamp":"2024-04-02T07:54:29.019533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:45:58.230418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"114'205","reported_seq":577,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-02T07:54:30.543107+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:54:30.543107+0000","last_peered":"2024-04-02T07:54:30.543107+0000","last_clean":"2024-04-02T07:54:30.543107+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:54:30.543107+0000","last_undegraded":"2024-04-02T07:54:30.543107+0000","last_fullsized":"2024-04-02T07:54:30.543107+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575067,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1536000,"data_stored":754723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65400000000000003,"15min":0.67300000000000004},"min":{"1min":0.46700000000000003,"5min":0.34699999999999998,"15min":0.316},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.66200000000000003,"15min":0.68000000000000005},"min":{"1min":0.433,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.93000000000000005,"5min":1.024,"15min":1.024},"last":0.53300000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.66500000000000004,"15min":0.67800000000000005},"min":{"1min":0.46700000000000003,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.96099999999999997,"5min":1.097,"15min":1.097},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.65300000000000002,"15min":0.67400000000000004},"min":{"1min":0.53400000000000003,"5min":0.42399999999999999,"15min":0.34699999999999998},"max":{"1min":0.93999999999999995,"5min":0.98299999999999998,"15min":1.2190000000000001},"last":0.63100000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.68600000000000005,"15min":0.70799999999999996},"min":{"1min":0.48599999999999999,"5min":0.39500000000000002,"15min":0.34000000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":1.127},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.55500000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96499999999999997,"5min":1.044,"15min":1.1259999999999999},"last":0.70799999999999996}]},{"osd":3,"last update":"Tue Apr 2 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.68999999999999995,"15min":0.70399999999999996},"min":{"1min":0.502,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.95799999999999996,"5min":0.97799999999999998,"15min":1.026},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.69699999999999995,"15min":0.71799999999999997},"min":{"1min":0.51500000000000001,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.98899999999999999,"5min":1.071,"15min":1.196},"last":0.76200000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57299999999999995,"15min":0.58999999999999997},"min":{"1min":0.39900000000000002,"5min":0.35799999999999998,"15min":0.32000000000000001},"max":{"1min":0.79900000000000004,"5min":1.0229999999999999,"15min":1.0389999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.69599999999999995,"15min":0.69099999999999995},"min":{"1min":0.52700000000000002,"5min":0.308,"15min":0.308},"max":{"1min":1.016,"5min":1.016,"15min":1.0669999999999999},"last":0.72799999999999998}]},{"osd":5,"last update":"Tue Apr 2 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.65600000000000003,"15min":0.65100000000000002},"min":{"1min":0.497,"5min":0.42999999999999999,"15min":0.37},"max":{"1min":0.99199999999999999,"5min":1.0109999999999999,"15min":1.167},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.38400000000000001},"max":{"1min":0.90200000000000002,"5min":1.343,"15min":1.4219999999999999},"last":0.748}]},{"osd":6,"last update":"Tue Apr 2 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.66400000000000003,"15min":0.66200000000000003},"min":{"1min":0.53200000000000003,"5min":0.435,"15min":0.36199999999999999},"max":{"1min":0.94499999999999995,"5min":1.0580000000000001,"15min":2.5369999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.67000000000000004,"15min":0.65800000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.39600000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.0509999999999999},"last":0.65100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575065,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36732,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335108,"statfs":{"total":96636764160,"available":96599150592,"internally_reserved":0,"allocated":1544192,"data_stored":760789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.68999999999999995,"15min":0.70699999999999996},"min":{"1min":0.50800000000000001,"5min":0.40899999999999997,"15min":0.33400000000000002},"max":{"1min":0.94299999999999995,"5min":0.95899999999999996,"15min":0.98099999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68200000000000005,"15min":0.70699999999999996},"min":{"1min":0.42399999999999999,"5min":0.40500000000000003,"15min":0.38300000000000001},"max":{"1min":0.94299999999999995,"5min":0.96899999999999997,"15min":1.0249999999999999},"last":0.57199999999999995}]},{"osd":1,"last update":"Tue Apr 2 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.67300000000000004,"15min":0.69399999999999995},"min":{"1min":0.47899999999999998,"5min":0.433,"15min":0.41799999999999998},"max":{"1min":0.89300000000000002,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.71599999999999997,"15min":0.71899999999999997},"min":{"1min":0.48899999999999999,"5min":0.45300000000000001,"15min":0.42599999999999999},"max":{"1min":0.96399999999999997,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.54000000000000004}]},{"osd":2,"last update":"Tue Apr 2 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68899999999999995,"15min":0.71299999999999997},"min":{"1min":0.54100000000000004,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83199999999999996,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.69099999999999995,"15min":0.70799999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93600000000000005,"5min":0.96499999999999997,"15min":0.96599999999999997},"last":0.66800000000000004}]},{"osd":3,"last update":"Tue Apr 2 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71599999999999997,"15min":0.73199999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90600000000000003,"5min":1.089,"15min":1.089},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.71799999999999997,"15min":0.73799999999999999},"min":{"1min":0.57099999999999995,"5min":0.46600000000000003,"15min":0.45300000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0309999999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61699999999999999,"15min":0.60499999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.372},"max":{"1min":0.99399999999999999,"5min":0.996,"15min":0.996},"last":0.622},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.41699999999999998,"5min":0.40600000000000003,"15min":0.35499999999999998},"max":{"1min":0.98099999999999998,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.70599999999999996}]},{"osd":5,"last update":"Tue Apr 2 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.61599999999999999,"15min":0.61799999999999999},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.88,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68000000000000005,"15min":0.68200000000000005},"min":{"1min":0.41199999999999998,"5min":0.40899999999999997,"15min":0.40300000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.81100000000000005}]},{"osd":7,"last update":"Tue Apr 2 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.68700000000000006,"15min":0.68300000000000005},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.40200000000000002},"max":{"1min":0.95199999999999996,"5min":0.96199999999999997,"15min":1.0389999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68700000000000006,"15min":0.69099999999999995},"min":{"1min":0.48199999999999998,"5min":0.42799999999999999,"15min":0.42099999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.0149999999999999},"last":0.80200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575066,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59472,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312368,"statfs":{"total":96636764160,"available":96575864832,"internally_reserved":0,"allocated":2158592,"data_stored":1352833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.55000000000000004,"15min":0.47599999999999998},"min":{"1min":0.309,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.627,"5min":10.052,"15min":10.052},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.60799999999999998,"15min":0.53800000000000003},"min":{"1min":0.33900000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.92700000000000005,"5min":9.0250000000000004,"15min":9.0250000000000004},"last":0.36199999999999999}]},{"osd":2,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59499999999999997,"15min":0.53400000000000003},"min":{"1min":0.45300000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.74199999999999999,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64500000000000002,"15min":0.59899999999999998},"min":{"1min":0.503,"5min":0.314,"15min":0.314},"max":{"1min":1.1719999999999999,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.503}]},{"osd":3,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58299999999999996,"15min":0.57399999999999995},"min":{"1min":0.40600000000000003,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.137,"5min":2.609,"15min":2.609},"last":0.443},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.54800000000000004,"15min":0.54300000000000004},"min":{"1min":0.45400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81499999999999995,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.32300000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.77500000000000002,"15min":0.77400000000000002},"min":{"1min":0.59499999999999997,"5min":0.50900000000000001,"15min":0.46300000000000002},"max":{"1min":0.95899999999999996,"5min":2.21,"15min":2.21},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.82999999999999996,"15min":0.78800000000000003},"min":{"1min":0.53100000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1100000000000001,"5min":6.3419999999999996,"15min":6.3419999999999996},"last":0.65400000000000003}]},{"osd":5,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.84099999999999997,"15min":0.79400000000000004},"min":{"1min":0.59399999999999997,"5min":0.53800000000000003,"15min":0.41999999999999998},"max":{"1min":1.0189999999999999,"5min":6.5529999999999999,"15min":6.5529999999999999},"last":0.625},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.79300000000000004},"min":{"1min":0.55100000000000005,"5min":0.51000000000000001,"15min":0.48799999999999999},"max":{"1min":1.0669999999999999,"5min":2.544,"15min":2.544},"last":0.52900000000000003}]},{"osd":6,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":1.863,"15min":1.1539999999999999},"min":{"1min":0.61299999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.1259999999999999,"5min":78.903000000000006,"15min":78.903000000000006},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.84499999999999997,"5min":1.847,"15min":1.143},"min":{"1min":0.56899999999999995,"5min":0.49199999999999999,"15min":0.38100000000000001},"max":{"1min":1.9750000000000001,"5min":78.837000000000003,"15min":78.837000000000003},"last":0.57899999999999996}]},{"osd":7,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79700000000000004,"15min":0.80300000000000005},"min":{"1min":0.622,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.139,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.82299999999999995,"15min":0.82499999999999996},"min":{"1min":0.65000000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.073,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":0.55800000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575066,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32628,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339212,"statfs":{"total":96636764160,"available":96603353088,"internally_reserved":0,"allocated":1540096,"data_stored":759770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:54:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.435,"15min":0.439},"min":{"1min":0.34899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.68000000000000005,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.379},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.47199999999999998,"15min":0.48099999999999998},"min":{"1min":0.28799999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.64200000000000002,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.47099999999999997}]},{"osd":2,"last update":"Tue Apr 2 07:54:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.30399999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63400000000000001,"5min":1.597,"15min":1.597},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.53000000000000003},"min":{"1min":0.32600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.75800000000000001,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.434}]},{"osd":3,"last update":"Tue Apr 2 07:54:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.50900000000000001,"15min":0.52300000000000002},"min":{"1min":0.34599999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73699999999999999,"5min":0.78700000000000003,"15min":0.81799999999999995},"last":0.496},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58699999999999997,"15min":0.60399999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.84499999999999997,"5min":1.492,"15min":1.492},"last":0.61299999999999999}]},{"osd":4,"last update":"Tue Apr 2 07:54:22 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.75800000000000001,"15min":0.77800000000000002},"min":{"1min":0.49299999999999999,"5min":0.39400000000000002,"15min":0.35599999999999998},"max":{"1min":1.1000000000000001,"5min":1.1579999999999999,"15min":1.181},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.76800000000000002,"15min":0.77300000000000002},"min":{"1min":0.48499999999999999,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":1.0629999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.54300000000000004}]},{"osd":5,"last update":"Tue Apr 2 07:54:22 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.77400000000000002,"15min":0.79200000000000004},"min":{"1min":0.53300000000000003,"5min":0.48699999999999999,"15min":0.42099999999999999},"max":{"1min":1.117,"5min":1.361,"15min":2.1869999999999998},"last":0.622},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.77800000000000002,"15min":0.79800000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.079,"5min":1.546,"15min":2.6230000000000002},"last":0.68600000000000005}]},{"osd":6,"last update":"Tue Apr 2 07:54:22 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.74399999999999999,"15min":0.752},"min":{"1min":0.52800000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.056,"5min":1.52,"15min":1.52},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.72099999999999997,"15min":0.745},"min":{"1min":0.50800000000000001,"5min":0.46700000000000003,"15min":0.376},"max":{"1min":1.0389999999999999,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.86099999999999999}]},{"osd":7,"last update":"Tue Apr 2 07:54:22 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.80100000000000005,"15min":0.81399999999999995},"min":{"1min":0.57299999999999995,"5min":0.51600000000000001,"15min":0.40799999999999997},"max":{"1min":1.139,"5min":1.139,"15min":2.3959999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76200000000000001,"15min":0.80400000000000005},"min":{"1min":0.59999999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.073,"5min":1.3959999999999999,"15min":2.145},"last":0.58699999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575065,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36736,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335104,"statfs":{"total":96636764160,"available":96599146496,"internally_reserved":0,"allocated":1548288,"data_stored":760859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.44600000000000001,"15min":0.44700000000000001},"min":{"1min":0.30499999999999999,"5min":0.26700000000000002,"15min":0.26400000000000001},"max":{"1min":0.56899999999999995,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.47499999999999998,"15min":0.47899999999999998},"min":{"1min":0.34200000000000003,"5min":0.29099999999999998,"15min":0.26200000000000001},"max":{"1min":0.73699999999999999,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.45300000000000001}]},{"osd":1,"last update":"Tue Apr 2 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.49099999999999999},"min":{"1min":0.375,"5min":0.32700000000000001,"15min":0.30499999999999999},"max":{"1min":0.64700000000000002,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.436},{"interface":"front","average":{"1min":0.504,"5min":0.51100000000000001,"15min":0.52300000000000002},"min":{"1min":0.375,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":0.65900000000000003,"5min":0.996,"15min":0.996},"last":0.39500000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53400000000000003,"15min":0.53600000000000003},"min":{"1min":0.39600000000000002,"5min":0.33500000000000002,"15min":0.316},"max":{"1min":0.69499999999999995,"5min":1.472,"15min":1.472},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59899999999999998,"15min":0.60699999999999998},"min":{"1min":0.47999999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.69699999999999995,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.497}]},{"osd":4,"last update":"Tue Apr 2 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.73599999999999999,"15min":0.76800000000000002},"min":{"1min":0.54600000000000004,"5min":0.5,"15min":0.47199999999999998},"max":{"1min":0.82799999999999996,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73899999999999999,"15min":0.76900000000000002},"min":{"1min":0.55000000000000004,"5min":0.54800000000000004,"15min":0.48499999999999999},"max":{"1min":0.96499999999999997,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.627}]},{"osd":5,"last update":"Tue Apr 2 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72699999999999998,"15min":0.75600000000000001},"min":{"1min":0.53200000000000003,"5min":0.51000000000000001,"15min":0.46200000000000002},"max":{"1min":0.86599999999999999,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.74199999999999999,"15min":0.77500000000000002},"min":{"1min":0.627,"5min":0.55600000000000005,"15min":0.50900000000000001},"max":{"1min":0.92400000000000004,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.61499999999999999}]},{"osd":6,"last update":"Tue Apr 2 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.76200000000000001,"15min":0.78600000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.442},"max":{"1min":0.86899999999999999,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.78000000000000003},"min":{"1min":0.622,"5min":0.502,"15min":0.502},"max":{"1min":0.84399999999999997,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.70499999999999996}]},{"osd":7,"last update":"Tue Apr 2 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.79300000000000004},"min":{"1min":0.58199999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.99199999999999999,"5min":1.278,"15min":1.278},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.74099999999999999,"15min":0.78700000000000003},"min":{"1min":0.56100000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93300000000000005,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.67800000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575065,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58884,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312956,"statfs":{"total":96636764160,"available":96576466944,"internally_reserved":0,"allocated":1556480,"data_stored":760983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42499999999999999,"15min":0.42799999999999999},"min":{"1min":0.308,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.65100000000000002,"5min":0.748,"15min":0.79600000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.45400000000000001,"15min":0.46500000000000002},"min":{"1min":0.33600000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.84099999999999997,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.40799999999999997}]},{"osd":1,"last update":"Tue Apr 2 07:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.47599999999999998,"15min":0.48999999999999999},"min":{"1min":0.32900000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.63,"5min":0.83799999999999997,"15min":1.2809999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57699999999999996,"15min":0.56699999999999995},"min":{"1min":0.42199999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.85799999999999998,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":0.47599999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.61499999999999999,"15min":0.61299999999999999},"min":{"1min":0.44400000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.871,"5min":2.214,"15min":2.214},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.58099999999999996,"15min":0.60499999999999998},"min":{"1min":0.46200000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":1.244},"last":0.441}]},{"osd":4,"last update":"Tue Apr 2 07:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.73799999999999999,"15min":0.74099999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.38400000000000001},"max":{"1min":1.022,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.73499999999999999,"15min":0.75700000000000001},"min":{"1min":0.58699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.052,"5min":1.052,"15min":1.1759999999999999},"last":0.67500000000000004}]},{"osd":5,"last update":"Tue Apr 2 07:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.74199999999999999,"15min":0.75900000000000001},"min":{"1min":0.61099999999999999,"5min":0.41699999999999998,"15min":0.40899999999999997},"max":{"1min":0.90500000000000003,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.748,"15min":0.75600000000000001},"min":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.40200000000000002},"max":{"1min":1.147,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.66100000000000003}]},{"osd":6,"last update":"Tue Apr 2 07:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.76400000000000001,"15min":0.77000000000000002},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.45800000000000002},"max":{"1min":1.1779999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.75,"15min":0.77400000000000002},"min":{"1min":0.497,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92700000000000005,"5min":2.286,"15min":2.286},"last":0.72199999999999998}]},{"osd":7,"last update":"Tue Apr 2 07:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.75600000000000001,"15min":0.77400000000000002},"min":{"1min":0.58199999999999996,"5min":0.52500000000000002,"15min":0.439},"max":{"1min":1.014,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.77100000000000002,"15min":0.79000000000000004},"min":{"1min":0.64100000000000001,"5min":0.54000000000000004,"15min":0.48999999999999999},"max":{"1min":1.022,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.63400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575065,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36724,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335116,"statfs":{"total":96636764160,"available":96599158784,"internally_reserved":0,"allocated":1548288,"data_stored":761371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.66600000000000004,"15min":0.67100000000000004},"min":{"1min":0.58599999999999997,"5min":0.41699999999999998,"15min":0.32100000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":1.0189999999999999},"last":0.754},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.65400000000000003,"15min":0.67100000000000004},"min":{"1min":0.60399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.94999999999999996,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.53200000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.70999999999999996,"15min":0.72599999999999998},"min":{"1min":0.56399999999999995,"5min":0.438,"15min":0.433},"max":{"1min":0.91700000000000004,"5min":1.014,"15min":1.036},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.71799999999999997,"15min":0.73599999999999999},"min":{"1min":0.55000000000000004,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.68400000000000005,"15min":0.69399999999999995},"min":{"1min":0.47399999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84199999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.70199999999999996,"15min":0.71499999999999997},"min":{"1min":0.63600000000000001,"5min":0.378,"15min":0.378},"max":{"1min":1.026,"5min":1.026,"15min":1.0620000000000001},"last":0.67800000000000005}]},{"osd":3,"last update":"Tue Apr 2 07:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.72999999999999998,"15min":0.72399999999999998},"min":{"1min":0.63500000000000001,"5min":0.505,"15min":0.46800000000000003},"max":{"1min":0.96399999999999997,"5min":0.97799999999999998,"15min":1.002},"last":0.628},{"interface":"front","average":{"1min":0.751,"5min":0.70699999999999996,"15min":0.72999999999999998},"min":{"1min":0.53900000000000003,"5min":0.47299999999999998,"15min":0.41499999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.70599999999999996}]},{"osd":5,"last update":"Tue Apr 2 07:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.437,"5min":0.38600000000000001,"15min":0.29599999999999999},"max":{"1min":0.85499999999999998,"5min":0.90800000000000003,"15min":0.93999999999999995},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59599999999999997,"15min":0.58599999999999997},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.33700000000000002},"max":{"1min":0.747,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.76600000000000001}]},{"osd":6,"last update":"Tue Apr 2 07:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.61599999999999999,"15min":0.60199999999999998},"min":{"1min":0.47399999999999998,"5min":0.433,"15min":0.374},"max":{"1min":0.84099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.65200000000000002},"min":{"1min":0.496,"5min":0.44400000000000001,"15min":0.42399999999999999},"max":{"1min":0.93100000000000005,"5min":1.0129999999999999,"15min":1.038},"last":0.84099999999999997}]},{"osd":7,"last update":"Tue Apr 2 07:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68100000000000005,"15min":0.67200000000000004},"min":{"1min":0.57999999999999996,"5min":0.39600000000000002,"15min":0.38300000000000001},"max":{"1min":0.89600000000000002,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.747,"5min":0.74099999999999999,"15min":0.73099999999999998},"min":{"1min":0.55400000000000005,"5min":0.47799999999999998,"15min":0.434},"max":{"1min":0.91700000000000004,"5min":1.095,"15min":1.095},"last":0.97499999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575068,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37316,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334524,"statfs":{"total":96636764160,"available":96598552576,"internally_reserved":0,"allocated":2154496,"data_stored":1351758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.65400000000000003,"15min":0.68200000000000005},"min":{"1min":0.443,"5min":0.35299999999999998,"15min":0.29899999999999999},"max":{"1min":0.92300000000000004,"5min":0.95299999999999996,"15min":0.99199999999999999},"last":0.629},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.67200000000000004,"15min":0.68799999999999994},"min":{"1min":0.45900000000000002,"5min":0.40000000000000002,"15min":0.378},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.4450000000000001},"last":0.53300000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.66700000000000004,"15min":0.69399999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.88100000000000001,"5min":1.0620000000000001,"15min":1.131},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.67600000000000005,"15min":0.69299999999999995},"min":{"1min":0.51800000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.5149999999999999},"last":0.56399999999999995}]},{"osd":2,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.67800000000000005,"15min":0.70499999999999996},"min":{"1min":0.47699999999999998,"5min":0.43099999999999999,"15min":0.41799999999999998},"max":{"1min":0.90700000000000003,"5min":0.98099999999999998,"15min":1.5449999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68500000000000005,"15min":0.70699999999999996},"min":{"1min":0.51900000000000002,"5min":0.47399999999999998,"15min":0.372},"max":{"1min":0.88500000000000001,"5min":1.111,"15min":1.3959999999999999},"last":0.51600000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.71899999999999997,"15min":0.73799999999999999},"min":{"1min":0.55900000000000005,"5min":0.46200000000000002,"15min":0.45100000000000001},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.341},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.44900000000000001},"max":{"1min":0.97499999999999998,"5min":1.1839999999999999,"15min":1.484},"last":0.64700000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58799999999999997,"15min":0.59199999999999997},"min":{"1min":0.44800000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.78100000000000003,"5min":0.80700000000000005,"15min":1.3109999999999999},"last":0.496},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64300000000000002,"15min":0.63500000000000001},"min":{"1min":0.52200000000000002,"5min":0.39600000000000002,"15min":0.35599999999999998},"max":{"1min":0.94199999999999995,"5min":0.96199999999999997,"15min":1.504},"last":0.69299999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66500000000000004,"15min":0.64100000000000001},"min":{"1min":0.42599999999999999,"5min":0.41899999999999998,"15min":0.39900000000000002},"max":{"1min":0.93000000000000005,"5min":1.212,"15min":1.212},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70099999999999996,"15min":0.67600000000000005},"min":{"1min":0.59899999999999998,"5min":0.45500000000000002,"15min":0.41199999999999998},"max":{"1min":0.87,"5min":1.774,"15min":1.774},"last":0.42999999999999999}]},{"osd":7,"last update":"Tue Apr 2 07:54:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72499999999999998,"15min":0.69999999999999996},"min":{"1min":0.55600000000000005,"5min":0.45800000000000002,"15min":0.38900000000000001},"max":{"1min":0.98999999999999999,"5min":1.0129999999999999,"15min":1.6890000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.745,"15min":0.72399999999999998},"min":{"1min":0.60899999999999999,"5min":0.47799999999999998,"15min":0.434},"max":{"1min":0.999,"5min":1.149,"15min":1.5309999999999999},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:54:35.384 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T07:54:35.385 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T07:54:35.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.619+0000 7f81ea600700 1 Processor -- start 2024-04-02T07:54:35.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.619+0000 7f81ea600700 1 -- start start 2024-04-02T07:54:35.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81ea600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81e40ceb60 0x7f81e40d32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:35.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81ea600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f81e415dcd0 0x7f81e415eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:35.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81ea600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f81e415f430 0x7f81e40d0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:35.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81ea600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81e40cef40 con 0x7f81e415f430 2024-04-02T07:54:35.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81ea600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81e40d3060 con 0x7f81e415dcd0 2024-04-02T07:54:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81ea600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e415ed00 con 0x7f81e40ceb60 2024-04-02T07:54:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81e3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f81e415f430 0x7f81e40d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81e3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f81e415f430 0x7f81e40d0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36082/0 (socket says 172.21.15.89:36082) 2024-04-02T07:54:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81e3e00700 1 -- 172.21.15.89:0/3616762390 learned_addr learned my addr 172.21.15.89:0/3616762390 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81e2a00700 1 --2- 172.21.15.89:0/3616762390 >> v2:172.21.15.113:3300/0 conn(0x7f81e415dcd0 0x7f81e415eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81e3400700 1 --2- 172.21.15.89:0/3616762390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81e40ceb60 0x7f81e40d32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81e3e00700 1 -- 172.21.15.89:0/3616762390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81e40ceb60 msgr2=0x7f81e40d32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81e3e00700 1 --2- 172.21.15.89:0/3616762390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81e40ceb60 0x7f81e40d32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81e3e00700 1 -- 172.21.15.89:0/3616762390 >> v2:172.21.15.113:3300/0 conn(0x7f81e415dcd0 msgr2=0x7f81e415eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81e3e00700 1 --2- 172.21.15.89:0/3616762390 >> v2:172.21.15.113:3300/0 conn(0x7f81e415dcd0 0x7f81e415eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.621+0000 7f81e3e00700 1 -- 172.21.15.89:0/3616762390 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e415e460 con 0x7f81e415f430 2024-04-02T07:54:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.622+0000 7f81e3e00700 1 --2- 172.21.15.89:0/3616762390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e415f430 0x7f81e40d0240 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f81e406b610 tx=0x7f81dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607098fefed861e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.622+0000 7f81e2000700 1 -- 172.21.15.89:0/3616762390 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc00e070 con 0x7f81e415f430 2024-04-02T07:54:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.622+0000 7f81e2000700 1 -- 172.21.15.89:0/3616762390 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81dc00b980 con 0x7f81e415f430 2024-04-02T07:54:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.622+0000 7f81e2000700 1 -- 172.21.15.89:0/3616762390 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc00bd30 con 0x7f81e415f430 2024-04-02T07:54:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.622+0000 7f81ea600700 1 -- 172.21.15.89:0/3616762390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e415f430 msgr2=0x7f81e40d0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:35.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.622+0000 7f81ea600700 1 --2- 172.21.15.89:0/3616762390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e415f430 0x7f81e40d0240 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f81e406b610 tx=0x7f81dc00b330 comp rx=0 tx=0).stop 2024-04-02T07:54:35.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.622+0000 7f81ea600700 1 -- 172.21.15.89:0/3616762390 shutdown_connections 2024-04-02T07:54:35.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.623+0000 7f81ea600700 1 --2- 172.21.15.89:0/3616762390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e415f430 0x7f81e40d0240 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.623+0000 7f81ea600700 1 --2- 172.21.15.89:0/3616762390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81e40ceb60 0x7f81e40d32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.623+0000 7f81ea600700 1 --2- 172.21.15.89:0/3616762390 >> v2:172.21.15.113:3300/0 conn(0x7f81e415dcd0 0x7f81e415eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.623+0000 7f81ea600700 1 -- 172.21.15.89:0/3616762390 >> 172.21.15.89:0/3616762390 conn(0x7f81e40664c0 msgr2=0x7f81e40668d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:35.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.623+0000 7f81ea600700 1 -- 172.21.15.89:0/3616762390 shutdown_connections 2024-04-02T07:54:35.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.623+0000 7f81ea600700 1 -- 172.21.15.89:0/3616762390 wait complete. 2024-04-02T07:54:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.624+0000 7f81ea600700 1 Processor -- start 2024-04-02T07:54:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.624+0000 7f81ea600700 1 -- start start 2024-04-02T07:54:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.625+0000 7f81ea600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e40ceb60 0x7f81e40b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.625+0000 7f81ea600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e415dcd0 0x7f81e40ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.625+0000 7f81ea600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81e415f430 0x7f81e40cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.625+0000 7f81ea600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e414b8d0 con 0x7f81e40ceb60 2024-04-02T07:54:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.625+0000 7f81ea600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e414a830 con 0x7f81e415dcd0 2024-04-02T07:54:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.625+0000 7f81ea600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e414ab10 con 0x7f81e415f430 2024-04-02T07:54:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81e3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e40ceb60 0x7f81e40b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81e3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e40ceb60 0x7f81e40b5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36088/0 (socket says 172.21.15.89:36088) 2024-04-02T07:54:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81e3400700 1 -- 172.21.15.89:0/2096818364 learned_addr learned my addr 172.21.15.89:0/2096818364 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81e2a00700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e415dcd0 0x7f81e40ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81e3e00700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81e415f430 0x7f81e40cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81e3400700 1 -- 172.21.15.89:0/2096818364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81e415f430 msgr2=0x7f81e40cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:35.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81e3400700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81e415f430 0x7f81e40cd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81e3400700 1 -- 172.21.15.89:0/2096818364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e415dcd0 msgr2=0x7f81e40ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:35.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81e3400700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e415dcd0 0x7f81e40ccb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81e3400700 1 -- 172.21.15.89:0/2096818364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc009b30 con 0x7f81e40ceb60 2024-04-02T07:54:35.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81e3e00700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81e415f430 0x7f81e40cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:35.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81e2a00700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e415dcd0 0x7f81e40ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:35.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81e3400700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e40ceb60 0x7f81e40b5b50 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f81d400e810 tx=0x7f81d400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:35.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81db400700 1 -- 172.21.15.89:0/2096818364 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d4013610 con 0x7f81e40ceb60 2024-04-02T07:54:35.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81db400700 1 -- 172.21.15.89:0/2096818364 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d4013770 con 0x7f81e40ceb60 2024-04-02T07:54:35.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81ea600700 1 -- 172.21.15.89:0/2096818364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e414adf0 con 0x7f81e40ceb60 2024-04-02T07:54:35.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81db400700 1 -- 172.21.15.89:0/2096818364 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d4013b00 con 0x7f81e40ceb60 2024-04-02T07:54:35.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.626+0000 7f81ea600700 1 -- 172.21.15.89:0/2096818364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e414b3b0 con 0x7f81e40ceb60 2024-04-02T07:54:35.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.628+0000 7f81ea600700 1 -- 172.21.15.89:0/2096818364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e414b720 con 0x7f81e40ceb60 2024-04-02T07:54:35.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.628+0000 7f81db400700 1 -- 172.21.15.89:0/2096818364 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f81d4013c60 con 0x7f81e40ceb60 2024-04-02T07:54:35.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.629+0000 7f81db400700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f81c406adb0 0x7f81c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:35.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.629+0000 7f81db400700 1 -- 172.21.15.89:0/2096818364 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f81d40892f0 con 0x7f81e40ceb60 2024-04-02T07:54:35.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.629+0000 7f81e2a00700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f81c406adb0 0x7f81c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:35.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.630+0000 7f81e2a00700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f81c406adb0 0x7f81c406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f81e40cfa60 tx=0x7f81d0008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.633+0000 7f81db400700 1 -- 172.21.15.89:0/2096818364 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f81d4054150 con 0x7f81e40ceb60 2024-04-02T07:54:35.742 INFO:tasks.ceph.mds.c.smithi089.stderr:2024-04-02T07:54:35.742+0000 7f3b57e00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T07:54:35.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.915+0000 7f81ea600700 1 -- 172.21.15.89:0/2096818364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f81e4067720 con 0x7f81e40ceb60 2024-04-02T07:54:35.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.919+0000 7f81db400700 1 -- 172.21.15.89:0/2096818364 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+0 (secure 0 0 0) 0x7f81d4009440 con 0x7f81e40ceb60 2024-04-02T07:54:35.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.921+0000 7f81ea600700 1 -- 172.21.15.89:0/2096818364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f81c406adb0 msgr2=0x7f81c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:35.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.921+0000 7f81ea600700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f81c406adb0 0x7f81c406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f81e40cfa60 tx=0x7f81d0008040 comp rx=0 tx=0).stop 2024-04-02T07:54:35.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.921+0000 7f81ea600700 1 -- 172.21.15.89:0/2096818364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e40ceb60 msgr2=0x7f81e40b5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:35.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.921+0000 7f81ea600700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e40ceb60 0x7f81e40b5b50 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f81d400e810 tx=0x7f81d400ebd0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.922+0000 7f81ea600700 1 -- 172.21.15.89:0/2096818364 shutdown_connections 2024-04-02T07:54:35.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.922+0000 7f81ea600700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f81c406adb0 0x7f81c406d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.922+0000 7f81ea600700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81e415f430 0x7f81e40cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.922+0000 7f81ea600700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e40ceb60 0x7f81e40b5b50 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.922+0000 7f81ea600700 1 --2- 172.21.15.89:0/2096818364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81e415dcd0 0x7f81e40ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:35.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.922+0000 7f81ea600700 1 -- 172.21.15.89:0/2096818364 >> 172.21.15.89:0/2096818364 conn(0x7f81e40664c0 msgr2=0x7f81e40668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:35.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.922+0000 7f81ea600700 1 -- 172.21.15.89:0/2096818364 shutdown_connections 2024-04-02T07:54:35.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:35.922+0000 7f81ea600700 1 -- 172.21.15.89:0/2096818364 wait complete. 2024-04-02T07:54:35.924 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T07:54:35.940 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:54:36.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.129+0000 7f2b4e200700 1 Processor -- start 2024-04-02T07:54:36.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.130+0000 7f2b4e200700 1 -- start start 2024-04-02T07:54:36.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.131+0000 7f2b4e200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b4815d640 0x7f2b480d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:36.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.131+0000 7f2b4e200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2b480ceec0 0x7f2b481612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:36.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.131+0000 7f2b4e200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b480d30d0 con 0x7f2b48161800 2024-04-02T07:54:36.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.131+0000 7f2b4e200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b48161c90 con 0x7f2b480ceec0 2024-04-02T07:54:36.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.131+0000 7f2b4e200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b480b6840 con 0x7f2b4815d640 2024-04-02T07:54:36.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.131+0000 7f2b46a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2b480ceec0 0x7f2b481612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:36.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.131+0000 7f2b46a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2b480ceec0 0x7f2b481612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36090/0 (socket says 172.21.15.89:36090) 2024-04-02T07:54:36.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.131+0000 7f2b46a00700 1 -- 172.21.15.89:0/3149289828 learned_addr learned my addr 172.21.15.89:0/3149289828 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:36.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.131+0000 7f2b47400700 1 --2- 172.21.15.89:0/3149289828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b4815d640 0x7f2b480d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:36.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.131+0000 7f2b46a00700 1 -- 172.21.15.89:0/3149289828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b4815d640 msgr2=0x7f2b480d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:36.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.131+0000 7f2b46a00700 1 --2- 172.21.15.89:0/3149289828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b4815d640 0x7f2b480d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:36.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.132+0000 7f2b46a00700 1 -- 172.21.15.89:0/3149289828 >> v1:172.21.15.113:6789/0 conn(0x7f2b48161800 legacy=0x7f2b480cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:36.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.132+0000 7f2b46a00700 1 -- 172.21.15.89:0/3149289828 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b480d2880 con 0x7f2b480ceec0 2024-04-02T07:54:36.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.132+0000 7f2b46a00700 1 --2- 172.21.15.89:0/3149289828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b480ceec0 0x7f2b481612c0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f2b340098f0 tx=0x7f2b34009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a3f3c93524a5d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:36.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.132+0000 7f2b46000700 1 -- 172.21.15.89:0/3149289828 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3400b8a0 con 0x7f2b480ceec0 2024-04-02T07:54:36.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.132+0000 7f2b46000700 1 -- 172.21.15.89:0/3149289828 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3400ba00 con 0x7f2b480ceec0 2024-04-02T07:54:36.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.132+0000 7f2b46000700 1 -- 172.21.15.89:0/3149289828 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3400bd90 con 0x7f2b480ceec0 2024-04-02T07:54:36.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.133+0000 7f2b4e200700 1 -- 172.21.15.89:0/3149289828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b480ceec0 msgr2=0x7f2b481612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:36.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.133+0000 7f2b4e200700 1 --2- 172.21.15.89:0/3149289828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b480ceec0 0x7f2b481612c0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f2b340098f0 tx=0x7f2b34009c00 comp rx=0 tx=0).stop 2024-04-02T07:54:36.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.133+0000 7f2b4e200700 1 -- 172.21.15.89:0/3149289828 shutdown_connections 2024-04-02T07:54:36.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.133+0000 7f2b4e200700 1 --2- 172.21.15.89:0/3149289828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b4815d640 0x7f2b480d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:36.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.133+0000 7f2b4e200700 1 --2- 172.21.15.89:0/3149289828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b480ceec0 0x7f2b481612c0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:36.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.133+0000 7f2b4e200700 1 -- 172.21.15.89:0/3149289828 >> 172.21.15.89:0/3149289828 conn(0x7f2b4808aff0 msgr2=0x7f2b480d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:36.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.133+0000 7f2b4e200700 1 -- 172.21.15.89:0/3149289828 shutdown_connections 2024-04-02T07:54:36.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.133+0000 7f2b4e200700 1 -- 172.21.15.89:0/3149289828 wait complete. 2024-04-02T07:54:36.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.134+0000 7f2b4e200700 1 Processor -- start 2024-04-02T07:54:36.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.135+0000 7f2b4e200700 1 -- start start 2024-04-02T07:54:36.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.135+0000 7f2b4e200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b480ceec0 0x7f2b480c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:36.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.135+0000 7f2b4e200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b4815d640 0x7f2b480c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:36.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.135+0000 7f2b4e200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b48161800 0x7f2b4815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.135+0000 7f2b4e200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b480c6670 con 0x7f2b48161800 2024-04-02T07:54:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.135+0000 7f2b4e200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4815ed90 con 0x7f2b480ceec0 2024-04-02T07:54:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.135+0000 7f2b4e200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4815f050 con 0x7f2b4815d640 2024-04-02T07:54:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.136+0000 7f2b47e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b48161800 0x7f2b4815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.136+0000 7f2b46a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b4815d640 0x7f2b480c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.136+0000 7f2b47400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b480ceec0 0x7f2b480c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.136+0000 7f2b47e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b48161800 0x7f2b4815fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36096/0 (socket says 172.21.15.89:36096) 2024-04-02T07:54:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.136+0000 7f2b47e00700 1 -- 172.21.15.89:0/1417919313 learned_addr learned my addr 172.21.15.89:0/1417919313 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.136+0000 7f2b47e00700 1 -- 172.21.15.89:0/1417919313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b4815d640 msgr2=0x7f2b480c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.136+0000 7f2b47e00700 1 --2- 172.21.15.89:0/1417919313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b4815d640 0x7f2b480c9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.136+0000 7f2b47e00700 1 -- 172.21.15.89:0/1417919313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b480ceec0 msgr2=0x7f2b480c6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.136+0000 7f2b47e00700 1 --2- 172.21.15.89:0/1417919313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b480ceec0 0x7f2b480c6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:36.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.136+0000 7f2b47e00700 1 -- 172.21.15.89:0/1417919313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b34009550 con 0x7f2b48161800 2024-04-02T07:54:36.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.136+0000 7f2b47e00700 1 --2- 172.21.15.89:0/1417919313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b48161800 0x7f2b4815fe10 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f2b4000cc50 tx=0x7f2b4000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:36.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.137+0000 7f2b3f400700 1 -- 172.21.15.89:0/1417919313 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b40003c00 con 0x7f2b48161800 2024-04-02T07:54:36.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.137+0000 7f2b3f400700 1 -- 172.21.15.89:0/1417919313 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b40003d60 con 0x7f2b48161800 2024-04-02T07:54:36.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.137+0000 7f2b3f400700 1 -- 172.21.15.89:0/1417919313 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4000aab0 con 0x7f2b48161800 2024-04-02T07:54:36.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.137+0000 7f2b4e200700 1 -- 172.21.15.89:0/1417919313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b4815f330 con 0x7f2b48161800 2024-04-02T07:54:36.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.137+0000 7f2b4e200700 1 -- 172.21.15.89:0/1417919313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b4815f8f0 con 0x7f2b48161800 2024-04-02T07:54:36.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.138+0000 7f2b4e200700 1 -- 172.21.15.89:0/1417919313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b48067720 con 0x7f2b48161800 2024-04-02T07:54:36.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.139+0000 7f2b3f400700 1 -- 172.21.15.89:0/1417919313 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2b4000ac10 con 0x7f2b48161800 2024-04-02T07:54:36.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.140+0000 7f2b3f400700 1 --2- 172.21.15.89:0/1417919313 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2b2806adb0 0x7f2b2806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:36.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.140+0000 7f2b3f400700 1 -- 172.21.15.89:0/1417919313 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2b40089670 con 0x7f2b48161800 2024-04-02T07:54:36.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.140+0000 7f2b47400700 1 --2- 172.21.15.89:0/1417919313 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2b2806adb0 0x7f2b2806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:36.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.141+0000 7f2b47400700 1 --2- 172.21.15.89:0/1417919313 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2b2806adb0 0x7f2b2806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2b380099d0 tx=0x7f2b38008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:36.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.145+0000 7f2b3f400700 1 -- 172.21.15.89:0/1417919313 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2b40054540 con 0x7f2b48161800 2024-04-02T07:54:36.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.390+0000 7f2b4e200700 1 -- 172.21.15.89:0/1417919313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b480d4040 con 0x7f2b48161800 2024-04-02T07:54:36.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.390+0000 7f2b3f400700 1 -- 172.21.15.89:0/1417919313 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f2b40057b60 con 0x7f2b48161800 2024-04-02T07:54:36.392 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:36.392 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1967870793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1967870793},{"type":"v1","addr":"172.21.15.89:6837","nonce":1967870793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1050024225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1050024225},{"type":"v1","addr":"172.21.15.113:6835","nonce":1050024225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7447,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/235320877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":235320877},{"type":"v1","addr":"172.21.15.113:6834","nonce":235320877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:54:30.522457+0000","modified":"2024-04-02T07:54:33.536401+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/1949161483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1949161483},{"type":"v1","addr":"172.21.15.89:6836","nonce":1949161483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-02T07:54:36.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.393+0000 7f2b4e200700 1 -- 172.21.15.89:0/1417919313 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2b2806adb0 msgr2=0x7f2b2806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:36.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.393+0000 7f2b4e200700 1 --2- 172.21.15.89:0/1417919313 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2b2806adb0 0x7f2b2806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2b380099d0 tx=0x7f2b38008040 comp rx=0 tx=0).stop 2024-04-02T07:54:36.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.393+0000 7f2b4e200700 1 -- 172.21.15.89:0/1417919313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b48161800 msgr2=0x7f2b4815fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:36.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.393+0000 7f2b4e200700 1 --2- 172.21.15.89:0/1417919313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b48161800 0x7f2b4815fe10 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f2b4000cc50 tx=0x7f2b4000cf60 comp rx=0 tx=0).stop 2024-04-02T07:54:36.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.393+0000 7f2b4e200700 1 -- 172.21.15.89:0/1417919313 shutdown_connections 2024-04-02T07:54:36.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.393+0000 7f2b4e200700 1 --2- 172.21.15.89:0/1417919313 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2b2806adb0 0x7f2b2806d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:36.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.393+0000 7f2b4e200700 1 --2- 172.21.15.89:0/1417919313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b480ceec0 0x7f2b480c6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:36.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.393+0000 7f2b4e200700 1 --2- 172.21.15.89:0/1417919313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b48161800 0x7f2b4815fe10 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:36.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.393+0000 7f2b4e200700 1 --2- 172.21.15.89:0/1417919313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b4815d640 0x7f2b480c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:36.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.393+0000 7f2b4e200700 1 -- 172.21.15.89:0/1417919313 >> 172.21.15.89:0/1417919313 conn(0x7f2b4808aff0 msgr2=0x7f2b480b8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:36.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.393+0000 7f2b4e200700 1 -- 172.21.15.89:0/1417919313 shutdown_connections 2024-04-02T07:54:36.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:36.393+0000 7f2b4e200700 1 -- 172.21.15.89:0/1417919313 wait complete. 2024-04-02T07:54:36.395 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 57 2024-04-02T07:54:36.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T07:54:30.522457+0000', 'modified': '2024-04-02T07:54:33.536401+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7932}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7932': {'gid': 7932, 'name': 'c', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6836/1949161483', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6834', 'nonce': 1949161483}, {'type': 'v1', 'addr': '172.21.15.89:6836', 'nonce': 1949161483}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T07:54:36.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T07:54:36.815 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-02T07:54:37.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.008+0000 7fa777c00700 1 Processor -- start 2024-04-02T07:54:37.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.008+0000 7fa777c00700 1 -- start start 2024-04-02T07:54:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.009+0000 7fa777c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa77015a5c0 0x7fa7700709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.010+0000 7fa777c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7700ce840 0x7fa7700cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.010+0000 7fa777c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7700cb310 con 0x7fa77015dfe0 2024-04-02T07:54:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.010+0000 7fa777c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa770070f00 con 0x7fa77015a5c0 2024-04-02T07:54:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.010+0000 7fa777c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7700711a0 con 0x7fa7700ce840 2024-04-02T07:54:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.010+0000 7fa76fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa77015dfe0 0x7fa7700ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36730/0 (socket says 172.21.15.89:36730) 2024-04-02T07:54:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.010+0000 7fa76fe00700 1 -- 172.21.15.89:0/3380466796 learned_addr learned my addr 172.21.15.89:0/3380466796 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.010+0000 7fa775200700 1 --2- 172.21.15.89:0/3380466796 >> v2:172.21.15.113:3300/0 conn(0x7fa77015a5c0 0x7fa7700709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.010+0000 7fa775c00700 1 --2- 172.21.15.89:0/3380466796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7700ce840 0x7fa7700cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.010+0000 7fa76f400700 1 -- 172.21.15.89:0/3380466796 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722116920 0 0) 0x7fa7700cb310 con 0x7fa77015dfe0 2024-04-02T07:54:37.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.010+0000 7fa76f400700 1 -- 172.21.15.89:0/3380466796 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7580057c0 con 0x7fa77015dfe0 2024-04-02T07:54:37.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.011+0000 7fa775200700 1 -- 172.21.15.89:0/3380466796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7700ce840 msgr2=0x7fa7700cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:37.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.011+0000 7fa775200700 1 --2- 172.21.15.89:0/3380466796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7700ce840 0x7fa7700cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.011+0000 7fa775200700 1 -- 172.21.15.89:0/3380466796 >> v1:172.21.15.89:6789/0 conn(0x7fa77015dfe0 legacy=0x7fa7700ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.011+0000 7fa775200700 1 -- 172.21.15.89:0/3380466796 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa770071480 con 0x7fa77015a5c0 2024-04-02T07:54:37.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.011+0000 7fa775200700 1 --2- 172.21.15.89:0/3380466796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa77015a5c0 0x7fa7700709c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa764009840 tx=0x7fa764009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2dc389f04a51354 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:37.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.011+0000 7fa76f400700 1 -- 172.21.15.89:0/3380466796 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa764011830 con 0x7fa77015a5c0 2024-04-02T07:54:37.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.011+0000 7fa76f400700 1 -- 172.21.15.89:0/3380466796 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa764011990 con 0x7fa77015a5c0 2024-04-02T07:54:37.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.011+0000 7fa76f400700 1 -- 172.21.15.89:0/3380466796 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa764011d20 con 0x7fa77015a5c0 2024-04-02T07:54:37.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.012+0000 7fa777c00700 1 -- 172.21.15.89:0/3380466796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa77015a5c0 msgr2=0x7fa7700709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.012+0000 7fa777c00700 1 --2- 172.21.15.89:0/3380466796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa77015a5c0 0x7fa7700709c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa764009840 tx=0x7fa764009b50 comp rx=0 tx=0).stop 2024-04-02T07:54:37.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.012+0000 7fa777c00700 1 -- 172.21.15.89:0/3380466796 shutdown_connections 2024-04-02T07:54:37.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.012+0000 7fa777c00700 1 --2- 172.21.15.89:0/3380466796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7700ce840 0x7fa7700cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.012+0000 7fa777c00700 1 --2- 172.21.15.89:0/3380466796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa77015a5c0 0x7fa7700709c0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.012+0000 7fa777c00700 1 -- 172.21.15.89:0/3380466796 >> 172.21.15.89:0/3380466796 conn(0x7fa7700b0d70 msgr2=0x7fa7700b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:37.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.012+0000 7fa777c00700 1 -- 172.21.15.89:0/3380466796 shutdown_connections 2024-04-02T07:54:37.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.012+0000 7fa777c00700 1 -- 172.21.15.89:0/3380466796 wait complete. 2024-04-02T07:54:37.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.014+0000 7fa777c00700 1 Processor -- start 2024-04-02T07:54:37.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.014+0000 7fa777c00700 1 -- start start 2024-04-02T07:54:37.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa777c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7700ce840 0x7fa77014e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa777c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa77015a5c0 0x7fa7700afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa777c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa77015dfe0 0x7fa7700b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa777c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77015b4c0 con 0x7fa7700ce840 2024-04-02T07:54:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa777c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77015abd0 con 0x7fa77015a5c0 2024-04-02T07:54:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa777c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa770157f20 con 0x7fa77015dfe0 2024-04-02T07:54:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa775200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7700ce840 0x7fa77014e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa775200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7700ce840 0x7fa77014e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36112/0 (socket says 172.21.15.89:36112) 2024-04-02T07:54:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa775200700 1 -- 172.21.15.89:0/535868288 learned_addr learned my addr 172.21.15.89:0/535868288 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa775c00700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa77015dfe0 0x7fa7700b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa76fe00700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa77015a5c0 0x7fa7700afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa775200700 1 -- 172.21.15.89:0/535868288 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa77015dfe0 msgr2=0x7fa7700b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa775200700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa77015dfe0 0x7fa7700b0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa775200700 1 -- 172.21.15.89:0/535868288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa77015a5c0 msgr2=0x7fa7700afd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa775200700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa77015a5c0 0x7fa7700afd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa775200700 1 -- 172.21.15.89:0/535868288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7640094a0 con 0x7fa7700ce840 2024-04-02T07:54:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa76fe00700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa77015a5c0 0x7fa7700afd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.015+0000 7fa775200700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7700ce840 0x7fa77014e960 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fa7640090b0 tx=0x7fa764004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.016+0000 7fa76cc00700 1 -- 172.21.15.89:0/535868288 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7640036a0 con 0x7fa7700ce840 2024-04-02T07:54:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.016+0000 7fa76cc00700 1 -- 172.21.15.89:0/535868288 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76401f070 con 0x7fa7700ce840 2024-04-02T07:54:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.016+0000 7fa777c00700 1 -- 172.21.15.89:0/535868288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77015ae70 con 0x7fa7700ce840 2024-04-02T07:54:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.016+0000 7fa76cc00700 1 -- 172.21.15.89:0/535868288 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa764003a40 con 0x7fa7700ce840 2024-04-02T07:54:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.016+0000 7fa777c00700 1 -- 172.21.15.89:0/535868288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77014eea0 con 0x7fa7700ce840 2024-04-02T07:54:37.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.018+0000 7fa76cc00700 1 -- 172.21.15.89:0/535868288 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa764003ba0 con 0x7fa7700ce840 2024-04-02T07:54:37.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.019+0000 7fa76cc00700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa75006adb0 0x7fa75006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.019+0000 7fa76cc00700 1 -- 172.21.15.89:0/535868288 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa764021070 con 0x7fa7700ce840 2024-04-02T07:54:37.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.019+0000 7fa76fe00700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa75006adb0 0x7fa75006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.020+0000 7fa76fe00700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa75006adb0 0x7fa75006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa77015dc10 tx=0x7fa75c009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:37.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.021+0000 7fa78b6b3b80 1 Processor -- start 2024-04-02T07:54:37.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.021+0000 7fa78b6b3b80 1 -- start start 2024-04-02T07:54:37.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.022+0000 7fa78b6b3b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 0x563e19c1bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.022+0000 7fa78b6b3b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19beebe0 0x563e19bf1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.022+0000 7fa78b6b3b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 0x563e19c18a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.022+0000 7fa78b6b3b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e19b66650 con 0x563e19beebe0 2024-04-02T07:54:37.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.022+0000 7fa78b6b3b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e19b47c30 con 0x563e19becb60 2024-04-02T07:54:37.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.022+0000 7fa78b6b3b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x563e19b45430 con 0x563e19c1b830 2024-04-02T07:54:37.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.022+0000 7fa76fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19beebe0 0x563e19bf1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.023+0000 7fa76fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19beebe0 0x563e19bf1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36128/0 (socket says 172.21.15.89:36128) 2024-04-02T07:54:37.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.023+0000 7fa76fe00700 1 -- 172.21.15.89:0/1232245890 learned_addr learned my addr 172.21.15.89:0/1232245890 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:37.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.023+0000 7fa775c00700 1 --2- 172.21.15.89:0/1232245890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 0x563e19c18a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.023+0000 7fa775200700 1 --2- 172.21.15.89:0/1232245890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 0x563e19c1bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.023+0000 7fa76fe00700 1 -- 172.21.15.89:0/1232245890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 msgr2=0x563e19c1bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:37.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.023+0000 7fa76fe00700 1 --2- 172.21.15.89:0/1232245890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 0x563e19c1bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.023+0000 7fa76fe00700 1 -- 172.21.15.89:0/1232245890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 msgr2=0x563e19c18a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:37.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.023+0000 7fa76fe00700 1 --2- 172.21.15.89:0/1232245890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 0x563e19c18a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.023+0000 7fa76fe00700 1 -- 172.21.15.89:0/1232245890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e19934790 con 0x563e19beebe0 2024-04-02T07:54:37.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.023+0000 7fa76fe00700 1 --2- 172.21.15.89:0/1232245890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19beebe0 0x563e19bf1a10 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fa75c005b40 tx=0x7fa75c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c76fa8d7c9b071da server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:37.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.023+0000 7fa761600700 1 -- 172.21.15.89:0/1232245890 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c005e50 con 0x563e19beebe0 2024-04-02T07:54:37.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.023+0000 7fa761600700 1 -- 172.21.15.89:0/1232245890 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75c020450 con 0x563e19beebe0 2024-04-02T07:54:37.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.023+0000 7fa761600700 1 -- 172.21.15.89:0/1232245890 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c0207c0 con 0x563e19beebe0 2024-04-02T07:54:37.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.024+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/1232245890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19beebe0 msgr2=0x563e19bf1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.024+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/1232245890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19beebe0 0x563e19bf1a10 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fa75c005b40 tx=0x7fa75c010000 comp rx=0 tx=0).stop 2024-04-02T07:54:37.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.024+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/1232245890 shutdown_connections 2024-04-02T07:54:37.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.024+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/1232245890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 0x563e19c18a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.024+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/1232245890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 0x563e19c1bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.024+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/1232245890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19beebe0 0x563e19bf1a10 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.024+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/1232245890 >> 172.21.15.89:0/1232245890 conn(0x563e19b4f6a0 msgr2=0x563e19c36640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:37.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.024+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/1232245890 shutdown_connections 2024-04-02T07:54:37.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.024+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/1232245890 wait complete. 2024-04-02T07:54:37.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.025+0000 7fa78b6b3b80 1 -- start start 2024-04-02T07:54:37.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.025+0000 7fa78b6b3b80 1 Processor -- start 2024-04-02T07:54:37.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.026+0000 7fa78b6b3b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19becb60 0x563e19c33000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.026+0000 7fa78b6b3b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19beebe0 0x563e19c1dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.026+0000 7fa78b6b3b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 0x563e19c39ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.026+0000 7fa78b6b3b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e19b40d20 con 0x563e19becb60 2024-04-02T07:54:37.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.026+0000 7fa78b6b3b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e19adc770 con 0x563e19beebe0 2024-04-02T07:54:37.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.026+0000 7fa78b6b3b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x563e19aff6f0 con 0x563e19c1b830 2024-04-02T07:54:37.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.026+0000 7fa78b6b3b80 10 client.0 _ll_register_callbacks cb 0x563e19ae65e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:54:37.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.026+0000 7fa775200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19becb60 0x563e19c33000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.026+0000 7fa775200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19becb60 0x563e19c33000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36130/0 (socket says 172.21.15.89:36130) 2024-04-02T07:54:37.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.026+0000 7fa775200700 1 -- 172.21.15.89:0/388484639 learned_addr learned my addr 172.21.15.89:0/388484639 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:37.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.026+0000 7fa76fe00700 1 --2- 172.21.15.89:0/388484639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19beebe0 0x563e19c1dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa775c00700 1 --2- 172.21.15.89:0/388484639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 0x563e19c39ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa775200700 1 -- 172.21.15.89:0/388484639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 msgr2=0x563e19c39ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:37.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa775200700 1 --2- 172.21.15.89:0/388484639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 0x563e19c39ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa775200700 1 -- 172.21.15.89:0/388484639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19beebe0 msgr2=0x563e19c1dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:37.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa775200700 1 --2- 172.21.15.89:0/388484639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19beebe0 0x563e19c1dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa775200700 1 -- 172.21.15.89:0/388484639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7580057c0 con 0x563e19becb60 2024-04-02T07:54:37.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa76fe00700 1 --2- 172.21.15.89:0/388484639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19beebe0 0x563e19c1dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:37.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa775200700 1 --2- 172.21.15.89:0/388484639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19becb60 0x563e19c33000 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fa764009b90 tx=0x7fa764059b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d0becd79fa56e4c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:37.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa756000700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:54:37.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa756000700 1 -- 172.21.15.89:0/388484639 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa764061ed0 con 0x563e19becb60 2024-04-02T07:54:37.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa756000700 1 -- 172.21.15.89:0/388484639 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa76402fe30 con 0x563e19becb60 2024-04-02T07:54:37.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/388484639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563e19be3920 con 0x563e19becb60 2024-04-02T07:54:37.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa756000700 1 -- 172.21.15.89:0/388484639 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa764058fd0 con 0x563e19becb60 2024-04-02T07:54:37.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa78b6b3b80 10 client.8070 fetch_fsmap learned FSMap version 57 2024-04-02T07:54:37.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.027+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/388484639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x563e19a210a0 con 0x563e19becb60 2024-04-02T07:54:37.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.028+0000 7fa756000700 1 -- 172.21.15.89:0/388484639 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563e19be3920 con 0x563e19becb60 2024-04-02T07:54:37.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.028+0000 7fa756000700 1 -- 172.21.15.89:0/388484639 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 57) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa7640782f0 con 0x563e19becb60 2024-04-02T07:54:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.028+0000 7fa78b6b3b80 10 client.8070 fetch_fsmap finished waiting for FSMap version 57 2024-04-02T07:54:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.028+0000 7fa78b6b3b80 10 client.8070 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T07:54:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.028+0000 7fa78b6b3b80 20 client.8070 populate_metadata read hostname 'smithi089' 2024-04-02T07:54:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.028+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/388484639 >> [v2:172.21.15.89:6834/1949161483,v1:172.21.15.89:6836/1949161483] conn(0x563e19c459d0 0x563e19c19910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.028+0000 7fa78b6b3b80 4 client.8070 mds_command: new command op to 7932 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T07:54:37.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.028+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/388484639 --> [v2:172.21.15.89:6834/1949161483,v1:172.21.15.89:6836/1949161483] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563e19bc06f0 con 0x563e19c459d0 2024-04-02T07:54:37.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.028+0000 7fa76fe00700 1 --2- 172.21.15.89:0/388484639 >> [v2:172.21.15.89:6834/1949161483,v1:172.21.15.89:6836/1949161483] conn(0x563e19c459d0 0x563e19c19910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.028+0000 7fa76fe00700 1 --2- 172.21.15.89:0/388484639 >> [v2:172.21.15.89:6834/1949161483,v1:172.21.15.89:6836/1949161483] conn(0x563e19c459d0 0x563e19c19910 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f6bb68013879910 server_cookie=f565d6fc8c772857 in_seq=0 out_seq=0 2024-04-02T07:54:37.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.028+0000 7fa756000700 10 client.8070 ms_handle_connect on v2:172.21.15.89:6834/1949161483 2024-04-02T07:54:37.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.029+0000 7fa756000700 1 -- 172.21.15.89:0/388484639 <== mds.0 v2:172.21.15.89:6834/1949161483 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563e19bc06f0 con 0x563e19c459d0 2024-04-02T07:54:37.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.029+0000 7fa756000700 10 client.8070 handle_command_reply: tid=0 2024-04-02T07:54:37.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.029+0000 7fa78b6b3b80 1 client.8070 shutdown 2024-04-02T07:54:37.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.029+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/388484639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19becb60 msgr2=0x563e19c33000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.029+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/388484639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19becb60 0x563e19c33000 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fa764009b90 tx=0x7fa764059b90 comp rx=0 tx=0).stop 2024-04-02T07:54:37.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.029+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/388484639 shutdown_connections 2024-04-02T07:54:37.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.029+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/388484639 >> [v2:172.21.15.89:6834/1949161483,v1:172.21.15.89:6836/1949161483] conn(0x563e19c459d0 0x563e19c19910 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.029+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/388484639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 0x563e19c39ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.029+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/388484639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19becb60 0x563e19c33000 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.029+0000 7fa756000700 0 client.8070 ms_handle_reset on v2:172.21.15.89:6834/1949161483 2024-04-02T07:54:37.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.029+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/388484639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19beebe0 0x563e19c1dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.030+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/388484639 >> 172.21.15.89:0/388484639 conn(0x563e19b4f6a0 msgr2=0x563e19c17ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:37.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.030+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/388484639 shutdown_connections 2024-04-02T07:54:37.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.030+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/388484639 wait complete. 2024-04-02T07:54:37.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.030+0000 7fa78b6b3b80 20 client.8070 trim_cache size 0 max 16384 2024-04-02T07:54:37.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.048+0000 7fa78b6b3b80 1 Processor -- start 2024-04-02T07:54:37.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.048+0000 7fa78b6b3b80 1 -- start start 2024-04-02T07:54:37.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.048+0000 7fa78b6b3b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 0x563e19c392c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.048+0000 7fa78b6b3b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19beebe0 0x563e19c1c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.048+0000 7fa78b6b3b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19c1b830 0x563e19c19890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.048+0000 7fa78b6b3b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e19935390 con 0x563e19c1b830 2024-04-02T07:54:37.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.048+0000 7fa78b6b3b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e19c39800 con 0x563e19becb60 2024-04-02T07:54:37.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.048+0000 7fa78b6b3b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x563e19c39ef0 con 0x563e19beebe0 2024-04-02T07:54:37.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.049+0000 7fa775c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19c1b830 0x563e19c19890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.049+0000 7fa775c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19c1b830 0x563e19c19890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36144/0 (socket says 172.21.15.89:36144) 2024-04-02T07:54:37.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.049+0000 7fa775c00700 1 -- 172.21.15.89:0/517418289 learned_addr learned my addr 172.21.15.89:0/517418289 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:37.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.049+0000 7fa775200700 1 --2- 172.21.15.89:0/517418289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 0x563e19c392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.049+0000 7fa76fe00700 1 --2- 172.21.15.89:0/517418289 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19beebe0 0x563e19c1c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.049+0000 7fa775c00700 1 -- 172.21.15.89:0/517418289 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19beebe0 msgr2=0x563e19c1c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:37.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.049+0000 7fa775c00700 1 --2- 172.21.15.89:0/517418289 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19beebe0 0x563e19c1c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.049+0000 7fa775c00700 1 -- 172.21.15.89:0/517418289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 msgr2=0x563e19c392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:37.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.049+0000 7fa775c00700 1 --2- 172.21.15.89:0/517418289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 0x563e19c392c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.049+0000 7fa775c00700 1 -- 172.21.15.89:0/517418289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e19b07010 con 0x563e19c1b830 2024-04-02T07:54:37.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.049+0000 7fa775c00700 1 --2- 172.21.15.89:0/517418289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19c1b830 0x563e19c19890 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fa768009fd0 tx=0x7fa768009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d993a9a777d757 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:37.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.050+0000 7fa762000700 1 -- 172.21.15.89:0/517418289 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76800e040 con 0x563e19c1b830 2024-04-02T07:54:37.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.050+0000 7fa762000700 1 -- 172.21.15.89:0/517418289 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76800f040 con 0x563e19c1b830 2024-04-02T07:54:37.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.050+0000 7fa762000700 1 -- 172.21.15.89:0/517418289 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa768018880 con 0x563e19c1b830 2024-04-02T07:54:37.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.050+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/517418289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19c1b830 msgr2=0x563e19c19890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.050+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/517418289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19c1b830 0x563e19c19890 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fa768009fd0 tx=0x7fa768009750 comp rx=0 tx=0).stop 2024-04-02T07:54:37.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.050+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/517418289 shutdown_connections 2024-04-02T07:54:37.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.050+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/517418289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 0x563e19c392c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.050+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/517418289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19c1b830 0x563e19c19890 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.050+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/517418289 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19beebe0 0x563e19c1c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.050+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/517418289 >> 172.21.15.89:0/517418289 conn(0x563e19b4f6a0 msgr2=0x563e19bea5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.050+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/517418289 shutdown_connections 2024-04-02T07:54:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.050+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/517418289 wait complete. 2024-04-02T07:54:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.051+0000 7fa78b6b3b80 1 -- start start 2024-04-02T07:54:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.051+0000 7fa78b6b3b80 1 Processor -- start 2024-04-02T07:54:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.052+0000 7fa78b6b3b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 0x563e19c86ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.052+0000 7fa78b6b3b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19beebe0 0x563e19c37ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.052+0000 7fa78b6b3b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 0x563e19c36440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.052+0000 7fa78b6b3b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e19c87220 con 0x563e19beebe0 2024-04-02T07:54:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.052+0000 7fa78b6b3b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e19c38210 con 0x563e19becb60 2024-04-02T07:54:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.052+0000 7fa78b6b3b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x563e19c16cc0 con 0x563e19c1b830 2024-04-02T07:54:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.052+0000 7fa78b6b3b80 10 client.0 _ll_register_callbacks cb 0x563e19ae65e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:54:37.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.052+0000 7fa76fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19beebe0 0x563e19c37ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.053+0000 7fa76fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19beebe0 0x563e19c37ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36154/0 (socket says 172.21.15.89:36154) 2024-04-02T07:54:37.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.053+0000 7fa76fe00700 1 -- 172.21.15.89:0/839608786 learned_addr learned my addr 172.21.15.89:0/839608786 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:37.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.053+0000 7fa775200700 1 --2- 172.21.15.89:0/839608786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 0x563e19c86ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.053+0000 7fa775c00700 1 --2- 172.21.15.89:0/839608786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 0x563e19c36440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.053+0000 7fa76fe00700 1 -- 172.21.15.89:0/839608786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 msgr2=0x563e19c36440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:37.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.053+0000 7fa76fe00700 1 --2- 172.21.15.89:0/839608786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 0x563e19c36440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.053+0000 7fa76fe00700 1 -- 172.21.15.89:0/839608786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 msgr2=0x563e19c86ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:37.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.053+0000 7fa76fe00700 1 --2- 172.21.15.89:0/839608786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 0x563e19c86ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.053+0000 7fa76fe00700 1 -- 172.21.15.89:0/839608786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7680152d0 con 0x563e19beebe0 2024-04-02T07:54:37.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.053+0000 7fa76fe00700 1 --2- 172.21.15.89:0/839608786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19beebe0 0x563e19c37ac0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x563e19c2d690 tx=0x7fa75c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7e227df41f3ee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:37.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.053+0000 7fa756000700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:54:37.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.053+0000 7fa756000700 1 -- 172.21.15.89:0/839608786 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c00d150 con 0x563e19beebe0 2024-04-02T07:54:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.053+0000 7fa756000700 1 -- 172.21.15.89:0/839608786 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa75c02d910 con 0x563e19beebe0 2024-04-02T07:54:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.053+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/839608786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa770157c40 con 0x563e19beebe0 2024-04-02T07:54:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.054+0000 7fa756000700 1 -- 172.21.15.89:0/839608786 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa75c005e50 con 0x563e19beebe0 2024-04-02T07:54:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.054+0000 7fa78b6b3b80 10 client.8076 fetch_fsmap learned FSMap version 57 2024-04-02T07:54:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.054+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/839608786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x563e199951d0 con 0x563e19beebe0 2024-04-02T07:54:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.054+0000 7fa756000700 1 -- 172.21.15.89:0/839608786 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa770157c40 con 0x563e19beebe0 2024-04-02T07:54:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.054+0000 7fa756000700 1 -- 172.21.15.89:0/839608786 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 57) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa75c005b70 con 0x563e19beebe0 2024-04-02T07:54:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.054+0000 7fa78b6b3b80 10 client.8076 fetch_fsmap finished waiting for FSMap version 57 2024-04-02T07:54:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.054+0000 7fa78b6b3b80 10 client.8076 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T07:54:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.054+0000 7fa78b6b3b80 20 client.8076 populate_metadata read hostname 'smithi089' 2024-04-02T07:54:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.054+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/839608786 >> [v2:172.21.15.89:6834/1949161483,v1:172.21.15.89:6836/1949161483] conn(0x563e19c1d4f0 0x563e19c1d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.054+0000 7fa78b6b3b80 4 client.8076 mds_command: new command op to 7932 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T07:54:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.054+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/839608786 --> [v2:172.21.15.89:6834/1949161483,v1:172.21.15.89:6836/1949161483] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fa75c020450 con 0x563e19c1d4f0 2024-04-02T07:54:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.054+0000 7fa775200700 1 --2- 172.21.15.89:0/839608786 >> [v2:172.21.15.89:6834/1949161483,v1:172.21.15.89:6836/1949161483] conn(0x563e19c1d4f0 0x563e19c1d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa775200700 1 --2- 172.21.15.89:0/839608786 >> [v2:172.21.15.89:6834/1949161483,v1:172.21.15.89:6836/1949161483] conn(0x563e19c1d4f0 0x563e19c1d8d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5df16b7e31cc3507 server_cookie=5ba31cec74f4c675 in_seq=0 out_seq=0 2024-04-02T07:54:37.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa756000700 10 client.8076 ms_handle_connect on v2:172.21.15.89:6834/1949161483 2024-04-02T07:54:37.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa756000700 1 -- 172.21.15.89:0/839608786 <== mds.0 v2:172.21.15.89:6834/1949161483 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x7fa75c020450 con 0x563e19c1d4f0 2024-04-02T07:54:37.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa756000700 10 client.8076 handle_command_reply: tid=0 2024-04-02T07:54:37.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa78b6b3b80 1 client.8076 shutdown 2024-04-02T07:54:37.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/839608786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19beebe0 msgr2=0x563e19c37ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/839608786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19beebe0 0x563e19c37ac0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x563e19c2d690 tx=0x7fa75c000ed0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/839608786 shutdown_connections 2024-04-02T07:54:37.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/839608786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563e19becb60 0x563e19c86ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/839608786 >> [v2:172.21.15.89:6834/1949161483,v1:172.21.15.89:6836/1949161483] conn(0x563e19c1d4f0 0x563e19c1d8d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/839608786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563e19c1b830 0x563e19c36440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa78b6b3b80 1 --2- 172.21.15.89:0/839608786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e19beebe0 0x563e19c37ac0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa756000700 0 client.8076 ms_handle_reset on v2:172.21.15.89:6834/1949161483 2024-04-02T07:54:37.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/839608786 >> 172.21.15.89:0/839608786 conn(0x563e19b4f6a0 msgr2=0x563e19c1c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:37.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/839608786 shutdown_connections 2024-04-02T07:54:37.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.055+0000 7fa78b6b3b80 1 -- 172.21.15.89:0/839608786 wait complete. 2024-04-02T07:54:37.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.056+0000 7fa78b6b3b80 20 client.8076 trim_cache size 0 max 16384 2024-04-02T07:54:37.063 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:54:37.063 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T07:54:37.063 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T07:54:37.063 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 7932, 2024-04-02T07:54:37.063 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T07:54:37.063 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T07:54:37.064 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T07:54:37.064 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 6.5306999689999996, 2024-04-02T07:54:37.064 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 57, 2024-04-02T07:54:37.064 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 125, 2024-04-02T07:54:37.064 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 125, 2024-04-02T07:54:37.064 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 12.255898673000001 2024-04-02T07:54:37.064 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:54:37.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.056+0000 7fa777c00700 1 -- 172.21.15.89:0/535868288 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa75006adb0 msgr2=0x7fa75006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.056+0000 7fa777c00700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa75006adb0 0x7fa75006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa77015dc10 tx=0x7fa75c009040 comp rx=0 tx=0).stop 2024-04-02T07:54:37.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.056+0000 7fa777c00700 1 -- 172.21.15.89:0/535868288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7700ce840 msgr2=0x7fa77014e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.056+0000 7fa777c00700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7700ce840 0x7fa77014e960 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fa7640090b0 tx=0x7fa764004750 comp rx=0 tx=0).stop 2024-04-02T07:54:37.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.056+0000 7fa777c00700 1 -- 172.21.15.89:0/535868288 shutdown_connections 2024-04-02T07:54:37.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.056+0000 7fa777c00700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa75006adb0 0x7fa75006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.056+0000 7fa777c00700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa77015dfe0 0x7fa7700b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.056+0000 7fa777c00700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7700ce840 0x7fa77014e960 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.056+0000 7fa777c00700 1 --2- 172.21.15.89:0/535868288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa77015a5c0 0x7fa7700afd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.056+0000 7fa777c00700 1 -- 172.21.15.89:0/535868288 >> 172.21.15.89:0/535868288 conn(0x7fa7700b0d70 msgr2=0x7fa7701554c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:37.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.056+0000 7fa777c00700 1 -- 172.21.15.89:0/535868288 shutdown_connections 2024-04-02T07:54:37.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.056+0000 7fa777c00700 1 -- 172.21.15.89:0/535868288 wait complete. 2024-04-02T07:54:37.074 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T07:54:37.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.264+0000 7f4133000700 1 Processor -- start 2024-04-02T07:54:37.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.264+0000 7f4133000700 1 -- start start 2024-04-02T07:54:37.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.265+0000 7f4133000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f412c160f10 0x7f412c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.265+0000 7f4133000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f412c161590 0x7f412c161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.265+0000 7f4133000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412c0ceda0 con 0x7f412c0cf230 2024-04-02T07:54:37.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.265+0000 7f4133000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412c161f80 con 0x7f412c161590 2024-04-02T07:54:37.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.265+0000 7f4133000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412c1612f0 con 0x7f412c160f10 2024-04-02T07:54:37.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.266+0000 7f4131000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f412c161590 0x7f412c161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.266+0000 7f4131000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f412c161590 0x7f412c161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36160/0 (socket says 172.21.15.89:36160) 2024-04-02T07:54:37.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.266+0000 7f4131000700 1 -- 172.21.15.89:0/1987536958 learned_addr learned my addr 172.21.15.89:0/1987536958 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:37.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.266+0000 7f412be00700 1 --2- 172.21.15.89:0/1987536958 >> v2:172.21.15.113:3300/0 conn(0x7f412c160f10 0x7f412c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.266+0000 7f4131000700 1 -- 172.21.15.89:0/1987536958 >> v2:172.21.15.113:3300/0 conn(0x7f412c160f10 msgr2=0x7f412c0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:37.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.266+0000 7f4131000700 1 --2- 172.21.15.89:0/1987536958 >> v2:172.21.15.113:3300/0 conn(0x7f412c160f10 0x7f412c0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.266+0000 7f4131000700 1 -- 172.21.15.89:0/1987536958 >> v1:172.21.15.113:6789/0 conn(0x7f412c0cf230 legacy=0x7f412c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.266+0000 7f4131000700 1 -- 172.21.15.89:0/1987536958 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c0ce470 con 0x7f412c161590 2024-04-02T07:54:37.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.266+0000 7f4131000700 1 --2- 172.21.15.89:0/1987536958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c161590 0x7f412c161a40 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f412c0d1080 tx=0x7f412400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f965f2fec2d48cab server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:37.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.267+0000 7f412aa00700 1 -- 172.21.15.89:0/1987536958 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412400e070 con 0x7f412c161590 2024-04-02T07:54:37.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.267+0000 7f412aa00700 1 -- 172.21.15.89:0/1987536958 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412400b9c0 con 0x7f412c161590 2024-04-02T07:54:37.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.267+0000 7f412aa00700 1 -- 172.21.15.89:0/1987536958 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412400bd70 con 0x7f412c161590 2024-04-02T07:54:37.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.267+0000 7f4133000700 1 -- 172.21.15.89:0/1987536958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c161590 msgr2=0x7f412c161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.267+0000 7f4133000700 1 --2- 172.21.15.89:0/1987536958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c161590 0x7f412c161a40 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f412c0d1080 tx=0x7f412400b330 comp rx=0 tx=0).stop 2024-04-02T07:54:37.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.267+0000 7f4133000700 1 -- 172.21.15.89:0/1987536958 shutdown_connections 2024-04-02T07:54:37.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.267+0000 7f4133000700 1 --2- 172.21.15.89:0/1987536958 >> v2:172.21.15.113:3300/0 conn(0x7f412c160f10 0x7f412c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.267+0000 7f4133000700 1 --2- 172.21.15.89:0/1987536958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c161590 0x7f412c161a40 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.267+0000 7f4133000700 1 -- 172.21.15.89:0/1987536958 >> 172.21.15.89:0/1987536958 conn(0x7f412c08aff0 msgr2=0x7f412c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:37.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.267+0000 7f4133000700 1 -- 172.21.15.89:0/1987536958 shutdown_connections 2024-04-02T07:54:37.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.267+0000 7f4133000700 1 -- 172.21.15.89:0/1987536958 wait complete. 2024-04-02T07:54:37.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.269+0000 7f4133000700 1 Processor -- start 2024-04-02T07:54:37.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.269+0000 7f4133000700 1 -- start start 2024-04-02T07:54:37.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.270+0000 7f4133000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0cf230 0x7f412c157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.270+0000 7f4133000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f412c160f10 0x7f412c15a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.270+0000 7f4133000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f412c161590 0x7f412c15af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.270+0000 7f4133000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c158250 con 0x7f412c0cf230 2024-04-02T07:54:37.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.270+0000 7f4133000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c15f360 con 0x7f412c161590 2024-04-02T07:54:37.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.270+0000 7f4133000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412c15f640 con 0x7f412c160f10 2024-04-02T07:54:37.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.270+0000 7f412be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0cf230 0x7f412c157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.270+0000 7f412be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0cf230 0x7f412c157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36164/0 (socket says 172.21.15.89:36164) 2024-04-02T07:54:37.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.270+0000 7f412be00700 1 -- 172.21.15.89:0/2651903870 learned_addr learned my addr 172.21.15.89:0/2651903870 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:37.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.271+0000 7f4131000700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f412c161590 0x7f412c15af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.270+0000 7f412b400700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f412c160f10 0x7f412c15a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.271+0000 7f412be00700 1 -- 172.21.15.89:0/2651903870 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f412c160f10 msgr2=0x7f412c15a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.271+0000 7f412be00700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f412c160f10 0x7f412c15a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.271+0000 7f412be00700 1 -- 172.21.15.89:0/2651903870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f412c161590 msgr2=0x7f412c15af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.271+0000 7f412be00700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f412c161590 0x7f412c15af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.271+0000 7f412be00700 1 -- 172.21.15.89:0/2651903870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4124009b30 con 0x7f412c0cf230 2024-04-02T07:54:37.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.271+0000 7f412b400700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f412c160f10 0x7f412c15a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:54:37.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.271+0000 7f412be00700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0cf230 0x7f412c157d10 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f411c00d800 tx=0x7f411c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:37.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.271+0000 7f4123e00700 1 -- 172.21.15.89:0/2651903870 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c0125f0 con 0x7f412c0cf230 2024-04-02T07:54:37.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.271+0000 7f4123e00700 1 -- 172.21.15.89:0/2651903870 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411c012750 con 0x7f412c0cf230 2024-04-02T07:54:37.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.271+0000 7f4133000700 1 -- 172.21.15.89:0/2651903870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f412c15f920 con 0x7f412c0cf230 2024-04-02T07:54:37.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.271+0000 7f4123e00700 1 -- 172.21.15.89:0/2651903870 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c012b00 con 0x7f412c0cf230 2024-04-02T07:54:37.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.271+0000 7f4133000700 1 -- 172.21.15.89:0/2651903870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412c15fee0 con 0x7f412c0cf230 2024-04-02T07:54:37.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.272+0000 7f4133000700 1 -- 172.21.15.89:0/2651903870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412c15d0b0 con 0x7f412c0cf230 2024-04-02T07:54:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.273+0000 7f4123e00700 1 -- 172.21.15.89:0/2651903870 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f411c012c60 con 0x7f412c0cf230 2024-04-02T07:54:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.274+0000 7f4123e00700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f410c06adb0 0x7f410c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.274+0000 7f4123e00700 1 -- 172.21.15.89:0/2651903870 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f411c089860 con 0x7f412c0cf230 2024-04-02T07:54:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.274+0000 7f412b400700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f410c06adb0 0x7f410c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:37.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.275+0000 7f412b400700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f410c06adb0 0x7f410c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f412c0c9f70 tx=0x7f4118009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:37.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.278+0000 7f4123e00700 1 -- 172.21.15.89:0/2651903870 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f411c054860 con 0x7f412c0cf230 2024-04-02T07:54:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.524+0000 7f4133000700 1 -- 172.21.15.89:0/2651903870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f412c067720 con 0x7f412c0cf230 2024-04-02T07:54:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.524+0000 7f4123e00700 1 -- 172.21.15.89:0/2651903870 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v57) v1 ==== 53+0+73 (secure 0 0 0) 0x7f411c057e80 con 0x7f412c0cf230 2024-04-02T07:54:37.525 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T07:54:37.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.526+0000 7f4133000700 1 -- 172.21.15.89:0/2651903870 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f410c06adb0 msgr2=0x7f410c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.526+0000 7f4133000700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f410c06adb0 0x7f410c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f412c0c9f70 tx=0x7f4118009040 comp rx=0 tx=0).stop 2024-04-02T07:54:37.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.526+0000 7f4133000700 1 -- 172.21.15.89:0/2651903870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0cf230 msgr2=0x7f412c157d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:37.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.526+0000 7f4133000700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0cf230 0x7f412c157d10 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f411c00d800 tx=0x7f411c00db10 comp rx=0 tx=0).stop 2024-04-02T07:54:37.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.527+0000 7f4133000700 1 -- 172.21.15.89:0/2651903870 shutdown_connections 2024-04-02T07:54:37.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.527+0000 7f4133000700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f410c06adb0 0x7f410c06d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.527+0000 7f4133000700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f412c161590 0x7f412c15af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.527+0000 7f4133000700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0cf230 0x7f412c157d10 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.527+0000 7f4133000700 1 --2- 172.21.15.89:0/2651903870 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f412c160f10 0x7f412c15a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:37.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.527+0000 7f4133000700 1 -- 172.21.15.89:0/2651903870 >> 172.21.15.89:0/2651903870 conn(0x7f412c08aff0 msgr2=0x7f412c0cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:37.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.527+0000 7f4133000700 1 -- 172.21.15.89:0/2651903870 shutdown_connections 2024-04-02T07:54:37.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:37.527+0000 7f4133000700 1 -- 172.21.15.89:0/2651903870 wait complete. 2024-04-02T07:54:37.545 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T07:54:37.545 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T07:54:37.545 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T07:54:37.545 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T07:54:37.545 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T07:54:37.545 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T07:54:37.545 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T07:54:37.545 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T07:54:37.546 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:54:37.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:37.546 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T07:54:37.591 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:54:37.591 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:54:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:54:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:54:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T07:54:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:54:37.592 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:54:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:54:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T07:54:37.592 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:54:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:54:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T07:54:37.592 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:54:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:54:37.592 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T07:54:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 41351sec preferred_lft 41351sec 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:54:37.593 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:54:37.594 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T07:54:37.594 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T07:54:37.594 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:54:37.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:37.594 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T07:54:37.695 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T07:54:37.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:37.698 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T07:54:37.746 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T07:54:37.746 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T07:54:37.746 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T07:54:37.746 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T07:54:37.746 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T07:54:37.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:37.747 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:54:37.747 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:54:37.747 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T07:54:37.747 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T07:54:37.747 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T07:54:37.747 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:54:37.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:37.907 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T07:54:37.952 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T07:54:37.952 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T07:54:37.952 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:54:37.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:37.952 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T07:54:38.060 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:54:38.060 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T07:54:38.104 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:54:38.104 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:54:38.104 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:54:38.188 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T07:54:38.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:38.188 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:54:38.328 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:54:38.328 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:54:38.355 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:54:38.355 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:54:38.355 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T07:54:38.355 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T07:54:38.355 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T07:54:38.360 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T07:54:38.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:38.360 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:54:38.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:38.426 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:54:38.524 INFO:teuthology.orchestra.run.smithi089.stdout:client8085 2024-04-02T07:54:38.524 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:54:38.524 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:54:38.570 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T07:54:38.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.800+0000 7f4eb1400700 1 Processor -- start 2024-04-02T07:54:38.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.800+0000 7f4eb1400700 1 -- start start 2024-04-02T07:54:38.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.801+0000 7f4eb1400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4eac15d9d0 0x7f4eac0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:38.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.801+0000 7f4eb1400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4eac0d2540 0x7f4eac0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:38.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.801+0000 7f4eb1400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eac161420 con 0x7f4eac0d16a0 2024-04-02T07:54:38.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.801+0000 7f4eb1400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eac15edc0 con 0x7f4eac0d2540 2024-04-02T07:54:38.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.801+0000 7f4eb1400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac161700 con 0x7f4eac15d9d0 2024-04-02T07:54:38.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.802+0000 7f4eab200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4eac0d2540 0x7f4eac0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:38.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.802+0000 7f4eaa800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4eac15d9d0 0x7f4eac0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:38.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.802+0000 7f4eaa800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4eac15d9d0 0x7f4eac0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:33768/0 (socket says 172.21.15.89:33768) 2024-04-02T07:54:38.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.802+0000 7f4eab200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4eac0d2540 0x7f4eac0ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:47168/0 (socket says 172.21.15.89:47168) 2024-04-02T07:54:38.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.802+0000 7f4eaa800700 1 -- 172.21.15.89:0/179322745 learned_addr learned my addr 172.21.15.89:0/179322745 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:38.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.802+0000 7f4eab200700 1 -- 172.21.15.89:0/179322745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4eac15d9d0 msgr2=0x7f4eac0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:38.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.802+0000 7f4eab200700 1 --2- 172.21.15.89:0/179322745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4eac15d9d0 0x7f4eac0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:38.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.802+0000 7f4eab200700 1 -- 172.21.15.89:0/179322745 >> v1:172.21.15.113:6789/0 conn(0x7f4eac0d16a0 legacy=0x7f4eac0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:38.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.802+0000 7f4eab200700 1 -- 172.21.15.89:0/179322745 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac1619e0 con 0x7f4eac0d2540 2024-04-02T07:54:38.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.802+0000 7f4ea9e00700 1 -- 172.21.15.89:0/179322745 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324383174 0 0) 0x7f4eac161420 con 0x7f4eac0d16a0 2024-04-02T07:54:38.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.803+0000 7f4eaa800700 1 --2- 172.21.15.89:0/179322745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4eac15d9d0 0x7f4eac0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:38.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.803+0000 7f4eab200700 1 --2- 172.21.15.89:0/179322745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4eac0d2540 0x7f4eac0ce360 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009dc0 tx=0x7f4ea400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3fc64b80e83012e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:38.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.803+0000 7f4ea9e00700 1 -- 172.21.15.89:0/179322745 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea400e070 con 0x7f4eac0d2540 2024-04-02T07:54:38.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.803+0000 7f4ea9e00700 1 -- 172.21.15.89:0/179322745 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea400bae0 con 0x7f4eac0d2540 2024-04-02T07:54:38.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.803+0000 7f4ea9e00700 1 -- 172.21.15.89:0/179322745 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea400be90 con 0x7f4eac0d2540 2024-04-02T07:54:38.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.804+0000 7f4eb1400700 1 -- 172.21.15.89:0/179322745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4eac0d2540 msgr2=0x7f4eac0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:38.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.804+0000 7f4eb1400700 1 --2- 172.21.15.89:0/179322745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4eac0d2540 0x7f4eac0ce360 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009dc0 tx=0x7f4ea400b330 comp rx=0 tx=0).stop 2024-04-02T07:54:38.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.804+0000 7f4eb1400700 1 -- 172.21.15.89:0/179322745 shutdown_connections 2024-04-02T07:54:38.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.804+0000 7f4eb1400700 1 --2- 172.21.15.89:0/179322745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4eac0d2540 0x7f4eac0ce360 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:38.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.804+0000 7f4eb1400700 1 --2- 172.21.15.89:0/179322745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4eac15d9d0 0x7f4eac0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:38.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.804+0000 7f4eb1400700 1 -- 172.21.15.89:0/179322745 >> 172.21.15.89:0/179322745 conn(0x7f4eac08aff0 msgr2=0x7f4eac0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:38.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.804+0000 7f4eb1400700 1 -- 172.21.15.89:0/179322745 shutdown_connections 2024-04-02T07:54:38.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.804+0000 7f4eb1400700 1 -- 172.21.15.89:0/179322745 wait complete. 2024-04-02T07:54:38.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.805+0000 7f4eb1400700 1 Processor -- start 2024-04-02T07:54:38.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.806+0000 7f4eb1400700 1 -- start start 2024-04-02T07:54:38.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.806+0000 7f4eb1400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4eac0d16a0 0x7f4eac15aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:38.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.806+0000 7f4eb1400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eac0d2540 0x7f4eac0b1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:38.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.806+0000 7f4eb1400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4eac15d9d0 0x7f4eac0b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:38.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.806+0000 7f4eb1400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0b6ef0 con 0x7f4eac0d2540 2024-04-02T07:54:38.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.806+0000 7f4eb1400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0b6020 con 0x7f4eac0d16a0 2024-04-02T07:54:38.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.806+0000 7f4eb1400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac15c510 con 0x7f4eac15d9d0 2024-04-02T07:54:38.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.807+0000 7f4ea1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eac0d2540 0x7f4eac0b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:38.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.807+0000 7f4ea1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eac0d2540 0x7f4eac0b1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37606/0 (socket says 172.21.15.89:37606) 2024-04-02T07:54:38.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.807+0000 7f4ea1e00700 1 -- 172.21.15.89:0/1425739619 learned_addr learned my addr 172.21.15.89:0/1425739619 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:38.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.807+0000 7f4eaa800700 1 --2- 172.21.15.89:0/1425739619 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4eac0d16a0 0x7f4eac15aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:38.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.807+0000 7f4eab200700 1 --2- 172.21.15.89:0/1425739619 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4eac15d9d0 0x7f4eac0b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:38.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.807+0000 7f4ea1e00700 1 -- 172.21.15.89:0/1425739619 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4eac15d9d0 msgr2=0x7f4eac0b19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:38.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.807+0000 7f4ea1e00700 1 --2- 172.21.15.89:0/1425739619 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4eac15d9d0 0x7f4eac0b19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:38.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.807+0000 7f4ea1e00700 1 -- 172.21.15.89:0/1425739619 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4eac0d16a0 msgr2=0x7f4eac15aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:38.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.807+0000 7f4ea1e00700 1 --2- 172.21.15.89:0/1425739619 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4eac0d16a0 0x7f4eac15aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:38.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.807+0000 7f4ea1e00700 1 -- 172.21.15.89:0/1425739619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c009550 con 0x7f4eac0d2540 2024-04-02T07:54:38.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.808+0000 7f4ea1e00700 1 --2- 172.21.15.89:0/1425739619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eac0d2540 0x7f4eac0b1480 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400c4a0 tx=0x7f4e9400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:38.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.808+0000 7f4ea3400700 1 -- 172.21.15.89:0/1425739619 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e940054e0 con 0x7f4eac0d2540 2024-04-02T07:54:38.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.808+0000 7f4ea3400700 1 -- 172.21.15.89:0/1425739619 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e940036d0 con 0x7f4eac0d2540 2024-04-02T07:54:38.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.808+0000 7f4eb1400700 1 -- 172.21.15.89:0/1425739619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea4009a70 con 0x7f4eac0d2540 2024-04-02T07:54:38.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.808+0000 7f4ea3400700 1 -- 172.21.15.89:0/1425739619 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94003a80 con 0x7f4eac0d2540 2024-04-02T07:54:38.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.808+0000 7f4eb1400700 1 -- 172.21.15.89:0/1425739619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eac0b6550 con 0x7f4eac0d2540 2024-04-02T07:54:38.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.809+0000 7f4eb1400700 1 -- 172.21.15.89:0/1425739619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eac0d1ca0 con 0x7f4eac0d2540 2024-04-02T07:54:38.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.810+0000 7f4ea3400700 1 -- 172.21.15.89:0/1425739619 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4e94003be0 con 0x7f4eac0d2540 2024-04-02T07:54:38.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.811+0000 7f4ea3400700 1 --2- 172.21.15.89:0/1425739619 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4e8806adb0 0x7f4e8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:38.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.811+0000 7f4ea3400700 1 -- 172.21.15.89:0/1425739619 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4e94088950 con 0x7f4eac0d2540 2024-04-02T07:54:38.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.811+0000 7f4eaa800700 1 --2- 172.21.15.89:0/1425739619 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4e8806adb0 0x7f4e8806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:38.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.812+0000 7f4eaa800700 1 --2- 172.21.15.89:0/1425739619 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4e8806adb0 0x7f4e8806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4eac0cd000 tx=0x7f4e9c0092c0 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:38.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:38.815+0000 7f4ea3400700 1 -- 172.21.15.89:0/1425739619 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4e94053950 con 0x7f4eac0d2540 2024-04-02T07:54:39.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.051+0000 7f4eb1400700 1 -- 172.21.15.89:0/1425739619 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4eac161fa0 con 0x7f4e8806adb0 2024-04-02T07:54:39.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.053+0000 7f4ea3400700 1 -- 172.21.15.89:0/1425739619 <== mgr.7225 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f4e9c009960 con 0x7f4e8806adb0 2024-04-02T07:54:39.054 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:54:39.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:54:39.054 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T07:54:39.054 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:54:39.054 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:54:39.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.055+0000 7f4eb1400700 1 -- 172.21.15.89:0/1425739619 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4e8806adb0 msgr2=0x7f4e8806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:39.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.055+0000 7f4eb1400700 1 --2- 172.21.15.89:0/1425739619 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4e8806adb0 0x7f4e8806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4eac0cd000 tx=0x7f4e9c0092c0 comp rx=0 tx=0).stop 2024-04-02T07:54:39.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.055+0000 7f4eb1400700 1 -- 172.21.15.89:0/1425739619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eac0d2540 msgr2=0x7f4eac0b1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:39.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.055+0000 7f4eb1400700 1 --2- 172.21.15.89:0/1425739619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eac0d2540 0x7f4eac0b1480 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400c4a0 tx=0x7f4e9400c7b0 comp rx=0 tx=0).stop 2024-04-02T07:54:39.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.055+0000 7f4eb1400700 1 -- 172.21.15.89:0/1425739619 shutdown_connections 2024-04-02T07:54:39.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.055+0000 7f4eb1400700 1 --2- 172.21.15.89:0/1425739619 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4e8806adb0 0x7f4e8806d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:39.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.055+0000 7f4eb1400700 1 --2- 172.21.15.89:0/1425739619 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4eac0d16a0 0x7f4eac15aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:39.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.055+0000 7f4eb1400700 1 --2- 172.21.15.89:0/1425739619 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4eac15d9d0 0x7f4eac0b19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:39.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.055+0000 7f4eb1400700 1 --2- 172.21.15.89:0/1425739619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eac0d2540 0x7f4eac0b1480 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:39.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.055+0000 7f4eb1400700 1 -- 172.21.15.89:0/1425739619 >> 172.21.15.89:0/1425739619 conn(0x7f4eac08aff0 msgr2=0x7f4eac0b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:39.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.055+0000 7f4eb1400700 1 -- 172.21.15.89:0/1425739619 shutdown_connections 2024-04-02T07:54:39.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.055+0000 7f4eb1400700 1 -- 172.21.15.89:0/1425739619 wait complete. 2024-04-02T07:54:39.074 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T07:54:39.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.267+0000 7f6d0c400700 1 Processor -- start 2024-04-02T07:54:39.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.268+0000 7f6d0c400700 1 -- start start 2024-04-02T07:54:39.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.269+0000 7f6d0c400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6d040cbb30 0x7f6d040cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:39.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.269+0000 7f6d0c400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d040cd480 0x7f6d0406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:39.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.269+0000 7f6d0c400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d040cc450 con 0x7f6d040cd020 2024-04-02T07:54:39.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.269+0000 7f6d0c400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d040cdae0 con 0x7f6d040cbb30 2024-04-02T07:54:39.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.269+0000 7f6d0c400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d040cddc0 con 0x7f6d040cd480 2024-04-02T07:54:39.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.269+0000 7f6d09a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6d040cbb30 0x7f6d040cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:39.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.269+0000 7f6d09a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6d040cbb30 0x7f6d040cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37612/0 (socket says 172.21.15.89:37612) 2024-04-02T07:54:39.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.269+0000 7f6d09a00700 1 -- 172.21.15.89:0/3871008896 learned_addr learned my addr 172.21.15.89:0/3871008896 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:39.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.269+0000 7f6d09a00700 1 -- 172.21.15.89:0/3871008896 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d040cd480 msgr2=0x7f6d0406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:39.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.269+0000 7f6d0a400700 1 --2- 172.21.15.89:0/3871008896 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d040cd480 0x7f6d0406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:39.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.269+0000 7f6d09a00700 1 --2- 172.21.15.89:0/3871008896 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d040cd480 0x7f6d0406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:39.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.269+0000 7f6d09a00700 1 -- 172.21.15.89:0/3871008896 >> v1:172.21.15.113:6789/0 conn(0x7f6d040cd020 legacy=0x7f6d040b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:39.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.269+0000 7f6d09a00700 1 -- 172.21.15.89:0/3871008896 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d040ce0a0 con 0x7f6d040cbb30 2024-04-02T07:54:39.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.269+0000 7f6d0a400700 1 --2- 172.21.15.89:0/3871008896 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d040cd480 0x7f6d0406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:54:39.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.270+0000 7f6d09a00700 1 --2- 172.21.15.89:0/3871008896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d040cbb30 0x7f6d040cbf10 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80098f0 tx=0x7f6cf8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db9a3679648dacd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:39.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.270+0000 7f6d03e00700 1 -- 172.21.15.89:0/3871008896 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8004230 con 0x7f6d040cbb30 2024-04-02T07:54:39.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.270+0000 7f6d03e00700 1 -- 172.21.15.89:0/3871008896 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf8004390 con 0x7f6d040cbb30 2024-04-02T07:54:39.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.270+0000 7f6d03e00700 1 -- 172.21.15.89:0/3871008896 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf80036a0 con 0x7f6d040cbb30 2024-04-02T07:54:39.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.270+0000 7f6d0c400700 1 -- 172.21.15.89:0/3871008896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d040cbb30 msgr2=0x7f6d040cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:39.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.270+0000 7f6d0c400700 1 --2- 172.21.15.89:0/3871008896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d040cbb30 0x7f6d040cbf10 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80098f0 tx=0x7f6cf8009c00 comp rx=0 tx=0).stop 2024-04-02T07:54:39.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.271+0000 7f6d0c400700 1 -- 172.21.15.89:0/3871008896 shutdown_connections 2024-04-02T07:54:39.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.271+0000 7f6d0c400700 1 --2- 172.21.15.89:0/3871008896 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d040cd480 0x7f6d0406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:39.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.271+0000 7f6d0c400700 1 --2- 172.21.15.89:0/3871008896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d040cbb30 0x7f6d040cbf10 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:39.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.271+0000 7f6d0c400700 1 -- 172.21.15.89:0/3871008896 >> 172.21.15.89:0/3871008896 conn(0x7f6d040b0d70 msgr2=0x7f6d040b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:39.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.271+0000 7f6d0c400700 1 -- 172.21.15.89:0/3871008896 shutdown_connections 2024-04-02T07:54:39.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.271+0000 7f6d0c400700 1 -- 172.21.15.89:0/3871008896 wait complete. 2024-04-02T07:54:39.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.272+0000 7f6d0c400700 1 Processor -- start 2024-04-02T07:54:39.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.272+0000 7f6d0c400700 1 -- start start 2024-04-02T07:54:39.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.273+0000 7f6d0c400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d040cbb30 0x7f6d041485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:39.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.273+0000 7f6d0c400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d040cd020 0x7f6d04071d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:39.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.273+0000 7f6d0c400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d040cd480 0x7f6d04155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:39.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.273+0000 7f6d0c400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d040722b0 con 0x7f6d040cd020 2024-04-02T07:54:39.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.273+0000 7f6d0c400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d04071a30 con 0x7f6d040cbb30 2024-04-02T07:54:39.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.273+0000 7f6d0c400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d04071750 con 0x7f6d040cd480 2024-04-02T07:54:39.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.274+0000 7f6d09000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d040cd020 0x7f6d04071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:39.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.274+0000 7f6d09000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d040cd020 0x7f6d04071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37618/0 (socket says 172.21.15.89:37618) 2024-04-02T07:54:39.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.274+0000 7f6d09000700 1 -- 172.21.15.89:0/2178957185 learned_addr learned my addr 172.21.15.89:0/2178957185 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:39.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.274+0000 7f6d09000700 1 -- 172.21.15.89:0/2178957185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d040cd480 msgr2=0x7f6d04155770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:39.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.274+0000 7f6d09a00700 1 --2- 172.21.15.89:0/2178957185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d040cbb30 0x7f6d041485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:39.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.274+0000 7f6d09000700 1 --2- 172.21.15.89:0/2178957185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d040cd480 0x7f6d04155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:39.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.274+0000 7f6d09000700 1 -- 172.21.15.89:0/2178957185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d040cbb30 msgr2=0x7f6d041485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:39.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.274+0000 7f6d09000700 1 --2- 172.21.15.89:0/2178957185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d040cbb30 0x7f6d041485e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:39.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.274+0000 7f6d09000700 1 -- 172.21.15.89:0/2178957185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf8009550 con 0x7f6d040cd020 2024-04-02T07:54:39.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.274+0000 7f6d09000700 1 --2- 172.21.15.89:0/2178957185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d040cd020 0x7f6d04071d70 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000c4d0 tx=0x7f6cf000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:39.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.274+0000 7f6d01600700 1 -- 172.21.15.89:0/2178957185 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0011790 con 0x7f6d040cd020 2024-04-02T07:54:39.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.274+0000 7f6d01600700 1 -- 172.21.15.89:0/2178957185 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf00118f0 con 0x7f6d040cd020 2024-04-02T07:54:39.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.274+0000 7f6d0c400700 1 -- 172.21.15.89:0/2178957185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d04154c90 con 0x7f6d040cd020 2024-04-02T07:54:39.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.275+0000 7f6d01600700 1 -- 172.21.15.89:0/2178957185 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0011ca0 con 0x7f6d040cd020 2024-04-02T07:54:39.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.275+0000 7f6d0c400700 1 -- 172.21.15.89:0/2178957185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d04155250 con 0x7f6d040cd020 2024-04-02T07:54:39.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.275+0000 7f6d0c400700 1 -- 172.21.15.89:0/2178957185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d04067720 con 0x7f6d040cd020 2024-04-02T07:54:39.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.277+0000 7f6d01600700 1 -- 172.21.15.89:0/2178957185 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 65) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6cf001c460 con 0x7f6d040cd020 2024-04-02T07:54:39.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.278+0000 7f6d01600700 1 --2- 172.21.15.89:0/2178957185 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ce406adb0 0x7f6ce406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:39.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.278+0000 7f6d01600700 1 -- 172.21.15.89:0/2178957185 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6cf0089490 con 0x7f6d040cd020 2024-04-02T07:54:39.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.278+0000 7f6d09a00700 1 --2- 172.21.15.89:0/2178957185 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ce406adb0 0x7f6ce406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:39.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.278+0000 7f6d09a00700 1 --2- 172.21.15.89:0/2178957185 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ce406adb0 0x7f6ce406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009c40 tx=0x7f6cf8008030 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:39.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.282+0000 7f6d01600700 1 -- 172.21.15.89:0/2178957185 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6cf0054370 con 0x7f6d040cd020 2024-04-02T07:54:39.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:39.544+0000 7f6d0c400700 1 -- 172.21.15.89:0/2178957185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f6d040b4020 con 0x7f6d040cd020 2024-04-02T07:54:40.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.033+0000 7f6d01600700 1 -- 172.21.15.89:0/2178957185 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 66) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f6cf0087850 con 0x7f6d040cd020 2024-04-02T07:54:40.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.034+0000 7f6d01600700 1 -- 172.21.15.89:0/2178957185 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v66) v1 ==== 92+0+0 (secure 0 0 0) 0x7f6cf0071330 con 0x7f6d040cd020 2024-04-02T07:54:40.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.036+0000 7f6d0c400700 1 -- 172.21.15.89:0/2178957185 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ce406adb0 msgr2=0x7f6ce406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:40.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.036+0000 7f6d0c400700 1 --2- 172.21.15.89:0/2178957185 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ce406adb0 0x7f6ce406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009c40 tx=0x7f6cf8008030 comp rx=0 tx=0).stop 2024-04-02T07:54:40.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.036+0000 7f6d0c400700 1 -- 172.21.15.89:0/2178957185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d040cd020 msgr2=0x7f6d04071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:40.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.037+0000 7f6d0c400700 1 --2- 172.21.15.89:0/2178957185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d040cd020 0x7f6d04071d70 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000c4d0 tx=0x7f6cf000c890 comp rx=0 tx=0).stop 2024-04-02T07:54:40.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.037+0000 7f6d0c400700 1 -- 172.21.15.89:0/2178957185 shutdown_connections 2024-04-02T07:54:40.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.037+0000 7f6d0c400700 1 --2- 172.21.15.89:0/2178957185 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ce406adb0 0x7f6ce406d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.037+0000 7f6d0c400700 1 --2- 172.21.15.89:0/2178957185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d040cbb30 0x7f6d041485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.037+0000 7f6d0c400700 1 --2- 172.21.15.89:0/2178957185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d040cd480 0x7f6d04155770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.037+0000 7f6d0c400700 1 --2- 172.21.15.89:0/2178957185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d040cd020 0x7f6d04071d70 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.037+0000 7f6d0c400700 1 -- 172.21.15.89:0/2178957185 >> 172.21.15.89:0/2178957185 conn(0x7f6d040b0d70 msgr2=0x7f6d04159830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:40.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.037+0000 7f6d0c400700 1 -- 172.21.15.89:0/2178957185 shutdown_connections 2024-04-02T07:54:40.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.037+0000 7f6d0c400700 1 -- 172.21.15.89:0/2178957185 wait complete. 2024-04-02T07:54:40.058 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T07:54:40.250 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:40.249+0000 7f5e92b53240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:54:40.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.257+0000 7fc208c00700 1 Processor -- start 2024-04-02T07:54:40.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.257+0000 7fc208c00700 1 -- start start 2024-04-02T07:54:40.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.259+0000 7fc208c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc2040cef40 0x7fc204099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:40.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.259+0000 7fc208c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc2040cf3f0 0x7fc204099fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:40.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.259+0000 7fc208c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2040ccdd0 con 0x7fc20415f1e0 2024-04-02T07:54:40.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.259+0000 7fc208c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2040cd260 con 0x7fc2040cf3f0 2024-04-02T07:54:40.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.259+0000 7fc208c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2041600c0 con 0x7fc2040cef40 2024-04-02T07:54:40.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.259+0000 7fc202800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc2040cf3f0 0x7fc204099fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:40.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.259+0000 7fc202800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc2040cf3f0 0x7fc204099fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37648/0 (socket says 172.21.15.89:37648) 2024-04-02T07:54:40.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.259+0000 7fc201400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc2040cef40 0x7fc204099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:40.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.259+0000 7fc202800700 1 -- 172.21.15.89:0/2359023771 learned_addr learned my addr 172.21.15.89:0/2359023771 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:40.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.259+0000 7fc202800700 1 -- 172.21.15.89:0/2359023771 >> v2:172.21.15.113:3300/0 conn(0x7fc2040cef40 msgr2=0x7fc204099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:40.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.259+0000 7fc202800700 1 --2- 172.21.15.89:0/2359023771 >> v2:172.21.15.113:3300/0 conn(0x7fc2040cef40 0x7fc204099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.259+0000 7fc202800700 1 -- 172.21.15.89:0/2359023771 >> v1:172.21.15.113:6789/0 conn(0x7fc20415f1e0 legacy=0x7fc20415f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:40.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.259+0000 7fc202800700 1 -- 172.21.15.89:0/2359023771 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20415cae0 con 0x7fc2040cf3f0 2024-04-02T07:54:40.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.260+0000 7fc202800700 1 --2- 172.21.15.89:0/2359023771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2040cf3f0 0x7fc204099fa0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc009e50 tx=0x7fc1fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c45e7df71dcc3c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:40.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.260+0000 7fc200a00700 1 -- 172.21.15.89:0/2359023771 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc00e070 con 0x7fc2040cf3f0 2024-04-02T07:54:40.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.260+0000 7fc200a00700 1 -- 172.21.15.89:0/2359023771 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1fc00b8d0 con 0x7fc2040cf3f0 2024-04-02T07:54:40.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.260+0000 7fc200a00700 1 -- 172.21.15.89:0/2359023771 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc00bc80 con 0x7fc2040cf3f0 2024-04-02T07:54:40.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.261+0000 7fc208c00700 1 -- 172.21.15.89:0/2359023771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2040cf3f0 msgr2=0x7fc204099fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:40.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.261+0000 7fc208c00700 1 --2- 172.21.15.89:0/2359023771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2040cf3f0 0x7fc204099fa0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc009e50 tx=0x7fc1fc00b330 comp rx=0 tx=0).stop 2024-04-02T07:54:40.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.261+0000 7fc208c00700 1 -- 172.21.15.89:0/2359023771 shutdown_connections 2024-04-02T07:54:40.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.261+0000 7fc208c00700 1 --2- 172.21.15.89:0/2359023771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2040cf3f0 0x7fc204099fa0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.261+0000 7fc208c00700 1 --2- 172.21.15.89:0/2359023771 >> v2:172.21.15.113:3300/0 conn(0x7fc2040cef40 0x7fc204099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.261+0000 7fc208c00700 1 -- 172.21.15.89:0/2359023771 >> 172.21.15.89:0/2359023771 conn(0x7fc2040d1370 msgr2=0x7fc2040d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:40.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.261+0000 7fc208c00700 1 -- 172.21.15.89:0/2359023771 shutdown_connections 2024-04-02T07:54:40.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.261+0000 7fc208c00700 1 -- 172.21.15.89:0/2359023771 wait complete. 2024-04-02T07:54:40.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.262+0000 7fc208c00700 1 Processor -- start 2024-04-02T07:54:40.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.263+0000 7fc208c00700 1 -- start start 2024-04-02T07:54:40.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.263+0000 7fc208c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2040cef40 0x7fc2040af420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.263+0000 7fc208c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2040cf3f0 0x7fc20415d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.263+0000 7fc208c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20415f1e0 0x7fc20415dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.264+0000 7fc208c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20415ebd0 con 0x7fc20415f1e0 2024-04-02T07:54:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.264+0000 7fc208c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2040c4bf0 con 0x7fc2040cef40 2024-04-02T07:54:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.264+0000 7fc208c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2040b4680 con 0x7fc2040cf3f0 2024-04-02T07:54:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.264+0000 7fc202800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20415f1e0 0x7fc20415dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.264+0000 7fc201e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2040cef40 0x7fc2040af420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.264+0000 7fc202800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20415f1e0 0x7fc20415dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37664/0 (socket says 172.21.15.89:37664) 2024-04-02T07:54:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.264+0000 7fc201400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2040cf3f0 0x7fc20415d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.264+0000 7fc202800700 1 -- 172.21.15.89:0/564702364 learned_addr learned my addr 172.21.15.89:0/564702364 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.264+0000 7fc202800700 1 -- 172.21.15.89:0/564702364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2040cf3f0 msgr2=0x7fc20415d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.264+0000 7fc202800700 1 --2- 172.21.15.89:0/564702364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2040cf3f0 0x7fc20415d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.264+0000 7fc202800700 1 -- 172.21.15.89:0/564702364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2040cef40 msgr2=0x7fc2040af420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:40.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.264+0000 7fc202800700 1 --2- 172.21.15.89:0/564702364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2040cef40 0x7fc2040af420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.264+0000 7fc202800700 1 -- 172.21.15.89:0/564702364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc009b30 con 0x7fc20415f1e0 2024-04-02T07:54:40.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.264+0000 7fc201e00700 1 --2- 172.21.15.89:0/564702364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2040cef40 0x7fc2040af420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:54:40.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.265+0000 7fc202800700 1 --2- 172.21.15.89:0/564702364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20415f1e0 0x7fc20415dc20 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fc204072450 tx=0x7fc1fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:40.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.265+0000 7fc1fa000700 1 -- 172.21.15.89:0/564702364 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc004800 con 0x7fc20415f1e0 2024-04-02T07:54:40.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.265+0000 7fc1fa000700 1 -- 172.21.15.89:0/564702364 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1fc0092e0 con 0x7fc20415f1e0 2024-04-02T07:54:40.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.265+0000 7fc1fa000700 1 -- 172.21.15.89:0/564702364 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc01c680 con 0x7fc20415f1e0 2024-04-02T07:54:40.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.265+0000 7fc208c00700 1 -- 172.21.15.89:0/564702364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc20415e2a0 con 0x7fc20415f1e0 2024-04-02T07:54:40.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.265+0000 7fc208c00700 1 -- 172.21.15.89:0/564702364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2040c5120 con 0x7fc20415f1e0 2024-04-02T07:54:40.268 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:40.266+0000 7fce09bbd240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:54:40.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.266+0000 7fc208c00700 1 -- 172.21.15.89:0/564702364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2040d0ca0 con 0x7fc20415f1e0 2024-04-02T07:54:40.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.267+0000 7fc1fa000700 1 -- 172.21.15.89:0/564702364 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 66) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fc1fc01c7e0 con 0x7fc20415f1e0 2024-04-02T07:54:40.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.268+0000 7fc1fa000700 1 --2- 172.21.15.89:0/564702364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1e406ae00 0x7fc1e406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:40.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.268+0000 7fc1fa000700 1 -- 172.21.15.89:0/564702364 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc1fc08b390 con 0x7fc20415f1e0 2024-04-02T07:54:40.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.268+0000 7fc201e00700 1 -- 172.21.15.89:0/564702364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1e406ae00 msgr2=0x7fc1e406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:40.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.268+0000 7fc201e00700 1 --2- 172.21.15.89:0/564702364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1e406ae00 0x7fc1e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:40.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.272+0000 7fc1fa000700 1 -- 172.21.15.89:0/564702364 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc1fc0561e0 con 0x7fc20415f1e0 2024-04-02T07:54:40.436 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:40.435+0000 7f5e92b53240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:54:40.455 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:40.454+0000 7fce09bbd240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:54:40.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.469+0000 7fc201e00700 1 -- 172.21.15.89:0/564702364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1e406ae00 msgr2=0x7fc1e406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:40.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.469+0000 7fc201e00700 1 --2- 172.21.15.89:0/564702364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1e406ae00 0x7fc1e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:40.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.516+0000 7fc208c00700 1 -- 172.21.15.89:0/564702364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fc204067720 con 0x7fc20415f1e0 2024-04-02T07:54:40.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.521+0000 7fc1fa000700 1 -- 172.21.15.89:0/564702364 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc1fc059800 con 0x7fc20415f1e0 2024-04-02T07:54:40.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.523+0000 7fc208c00700 1 -- 172.21.15.89:0/564702364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1e406ae00 msgr2=0x7fc1e406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:40.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.523+0000 7fc208c00700 1 --2- 172.21.15.89:0/564702364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1e406ae00 0x7fc1e406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.523+0000 7fc208c00700 1 -- 172.21.15.89:0/564702364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20415f1e0 msgr2=0x7fc20415dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:40.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.523+0000 7fc208c00700 1 --2- 172.21.15.89:0/564702364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20415f1e0 0x7fc20415dc20 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fc204072450 tx=0x7fc1fc004ad0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.524+0000 7fc208c00700 1 -- 172.21.15.89:0/564702364 shutdown_connections 2024-04-02T07:54:40.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.524+0000 7fc208c00700 1 --2- 172.21.15.89:0/564702364 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1e406ae00 0x7fc1e406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.524+0000 7fc208c00700 1 --2- 172.21.15.89:0/564702364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2040cef40 0x7fc2040af420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.524+0000 7fc208c00700 1 --2- 172.21.15.89:0/564702364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc20415f1e0 0x7fc20415dc20 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.524+0000 7fc208c00700 1 --2- 172.21.15.89:0/564702364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2040cf3f0 0x7fc20415d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.524+0000 7fc208c00700 1 -- 172.21.15.89:0/564702364 >> 172.21.15.89:0/564702364 conn(0x7fc2040d1370 msgr2=0x7fc20415a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:40.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.524+0000 7fc208c00700 1 -- 172.21.15.89:0/564702364 shutdown_connections 2024-04-02T07:54:40.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.524+0000 7fc208c00700 1 -- 172.21.15.89:0/564702364 wait complete. 2024-04-02T07:54:40.544 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T07:54:40.752 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:40.751+0000 7f5e92b53240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:54:40.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.753+0000 7f527ba00700 1 Processor -- start 2024-04-02T07:54:40.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.753+0000 7f527ba00700 1 -- start start 2024-04-02T07:54:40.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.754+0000 7f527ba00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f527415d8a0 0x7f52740b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:40.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.754+0000 7f527ba00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f52740cf1b0 0x7f52740cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:40.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.754+0000 7f527ba00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52740ced20 con 0x7f52740d3eb0 2024-04-02T07:54:40.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.754+0000 7f527ba00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52740d4340 con 0x7f52740cf1b0 2024-04-02T07:54:40.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.755+0000 7f527ba00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52740ce6d0 con 0x7f527415d8a0 2024-04-02T07:54:40.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.755+0000 7f5279a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f52740cf1b0 0x7f52740cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:40.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.755+0000 7f5273e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f52740d3eb0 0x7f527406b890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38792/0 (socket says 172.21.15.89:38792) 2024-04-02T07:54:40.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.755+0000 7f5279a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f52740cf1b0 0x7f52740cf640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37674/0 (socket says 172.21.15.89:37674) 2024-04-02T07:54:40.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.755+0000 7f5273e00700 1 -- 172.21.15.89:0/213726322 learned_addr learned my addr 172.21.15.89:0/213726322 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:40.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.755+0000 7f5279000700 1 --2- 172.21.15.89:0/213726322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f527415d8a0 0x7f52740b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:40.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.755+0000 7f5279a00700 1 -- 172.21.15.89:0/213726322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f527415d8a0 msgr2=0x7f52740b2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:40.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.755+0000 7f5279a00700 1 --2- 172.21.15.89:0/213726322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f527415d8a0 0x7f52740b2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.755+0000 7f5279a00700 1 -- 172.21.15.89:0/213726322 >> v1:172.21.15.89:6789/0 conn(0x7f52740d3eb0 legacy=0x7f527406b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:40.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.755+0000 7f5279a00700 1 -- 172.21.15.89:0/213726322 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52740ce9b0 con 0x7f52740cf1b0 2024-04-02T07:54:40.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.756+0000 7f5279a00700 1 --2- 172.21.15.89:0/213726322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52740cf1b0 0x7f52740cf640 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f526c009d90 tx=0x7f526c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c836d6dbefeb3c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:40.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.756+0000 7f5273400700 1 -- 172.21.15.89:0/213726322 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c00e070 con 0x7f52740cf1b0 2024-04-02T07:54:40.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.756+0000 7f5273400700 1 -- 172.21.15.89:0/213726322 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526c00bb20 con 0x7f52740cf1b0 2024-04-02T07:54:40.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.756+0000 7f5273400700 1 -- 172.21.15.89:0/213726322 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c012430 con 0x7f52740cf1b0 2024-04-02T07:54:40.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.756+0000 7f527ba00700 1 -- 172.21.15.89:0/213726322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52740cf1b0 msgr2=0x7f52740cf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:40.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.756+0000 7f527ba00700 1 --2- 172.21.15.89:0/213726322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52740cf1b0 0x7f52740cf640 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f526c009d90 tx=0x7f526c00b330 comp rx=0 tx=0).stop 2024-04-02T07:54:40.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.756+0000 7f527ba00700 1 -- 172.21.15.89:0/213726322 shutdown_connections 2024-04-02T07:54:40.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.756+0000 7f527ba00700 1 --2- 172.21.15.89:0/213726322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52740cf1b0 0x7f52740cf640 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.756+0000 7f527ba00700 1 --2- 172.21.15.89:0/213726322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f527415d8a0 0x7f52740b2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.756+0000 7f527ba00700 1 -- 172.21.15.89:0/213726322 >> 172.21.15.89:0/213726322 conn(0x7f527408aff0 msgr2=0x7f52740d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:40.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.757+0000 7f527ba00700 1 -- 172.21.15.89:0/213726322 shutdown_connections 2024-04-02T07:54:40.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.757+0000 7f527ba00700 1 -- 172.21.15.89:0/213726322 wait complete. 2024-04-02T07:54:40.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.759+0000 7f527ba00700 1 Processor -- start 2024-04-02T07:54:40.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.759+0000 7f527ba00700 1 -- start start 2024-04-02T07:54:40.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.760+0000 7f527ba00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52740cf1b0 0x7f52741f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:40.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.760+0000 7f527ba00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f527415d8a0 0x7f52740b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:40.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.760+0000 7f527ba00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52740cc590 0x7f52740b78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:40.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.760+0000 7f527ba00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52741f2d20 con 0x7f52740cc590 2024-04-02T07:54:40.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.760+0000 7f527ba00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52741542e0 con 0x7f52740cf1b0 2024-04-02T07:54:40.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.760+0000 7f527ba00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52740b7540 con 0x7f527415d8a0 2024-04-02T07:54:40.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.761+0000 7f5279a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52740cc590 0x7f52740b78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:40.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.761+0000 7f5279a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52740cc590 0x7f52740b78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37682/0 (socket says 172.21.15.89:37682) 2024-04-02T07:54:40.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.761+0000 7f5279a00700 1 -- 172.21.15.89:0/3952048948 learned_addr learned my addr 172.21.15.89:0/3952048948 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:40.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.761+0000 7f5279000700 1 --2- 172.21.15.89:0/3952048948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52740cf1b0 0x7f52741f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:40.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.761+0000 7f5273e00700 1 --2- 172.21.15.89:0/3952048948 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f527415d8a0 0x7f52740b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:40.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.761+0000 7f5279a00700 1 -- 172.21.15.89:0/3952048948 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f527415d8a0 msgr2=0x7f52740b6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:40.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.761+0000 7f5279a00700 1 --2- 172.21.15.89:0/3952048948 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f527415d8a0 0x7f52740b6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.761+0000 7f5279a00700 1 -- 172.21.15.89:0/3952048948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52740cf1b0 msgr2=0x7f52741f27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:40.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.761+0000 7f5279a00700 1 --2- 172.21.15.89:0/3952048948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52740cf1b0 0x7f52741f27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:40.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.761+0000 7f5279a00700 1 -- 172.21.15.89:0/3952048948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c009a70 con 0x7f52740cc590 2024-04-02T07:54:40.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.761+0000 7f5279a00700 1 --2- 172.21.15.89:0/3952048948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52740cc590 0x7f52740b78b0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f526c014040 tx=0x7f526c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:40.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.761+0000 7f5270c00700 1 -- 172.21.15.89:0/3952048948 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c012430 con 0x7f52740cc590 2024-04-02T07:54:40.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.762+0000 7f5270c00700 1 -- 172.21.15.89:0/3952048948 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526c020070 con 0x7f52740cc590 2024-04-02T07:54:40.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.762+0000 7f527ba00700 1 -- 172.21.15.89:0/3952048948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52740b6690 con 0x7f52740cc590 2024-04-02T07:54:40.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.762+0000 7f5270c00700 1 -- 172.21.15.89:0/3952048948 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c003ea0 con 0x7f52740cc590 2024-04-02T07:54:40.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.762+0000 7f527ba00700 1 -- 172.21.15.89:0/3952048948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52740b6c10 con 0x7f52740cc590 2024-04-02T07:54:40.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.763+0000 7f527ba00700 1 -- 172.21.15.89:0/3952048948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52740ca660 con 0x7f52740cc590 2024-04-02T07:54:40.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.764+0000 7f5270c00700 1 -- 172.21.15.89:0/3952048948 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 66) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f526c004040 con 0x7f52740cc590 2024-04-02T07:54:40.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.764+0000 7f5270c00700 1 --2- 172.21.15.89:0/3952048948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f525406ae00 0x7f525406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:40.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.765+0000 7f5270c00700 1 -- 172.21.15.89:0/3952048948 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f526c022070 con 0x7f52740cc590 2024-04-02T07:54:40.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.765+0000 7f5279000700 1 -- 172.21.15.89:0/3952048948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f525406ae00 msgr2=0x7f525406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:40.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.765+0000 7f5279000700 1 --2- 172.21.15.89:0/3952048948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f525406ae00 0x7f525406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:40.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.769+0000 7f5270c00700 1 -- 172.21.15.89:0/3952048948 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f52740ca660 con 0x7f52740cc590 2024-04-02T07:54:40.824 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:40.824+0000 7fce09bbd240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:54:40.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.966+0000 7f5279000700 1 -- 172.21.15.89:0/3952048948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f525406ae00 msgr2=0x7f525406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:40.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:40.966+0000 7f5279000700 1 --2- 172.21.15.89:0/3952048948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f525406ae00 0x7f525406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:41.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.012+0000 7f527ba00700 1 -- 172.21.15.89:0/3952048948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f52740ca660 con 0x7f52740cc590 2024-04-02T07:54:41.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.016+0000 7f5270c00700 1 -- 172.21.15.89:0/3952048948 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7f52740ca660 con 0x7f52740cc590 2024-04-02T07:54:41.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.019+0000 7f527ba00700 1 -- 172.21.15.89:0/3952048948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f525406ae00 msgr2=0x7f525406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:41.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.019+0000 7f527ba00700 1 --2- 172.21.15.89:0/3952048948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f525406ae00 0x7f525406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:41.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.019+0000 7f527ba00700 1 -- 172.21.15.89:0/3952048948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52740cc590 msgr2=0x7f52740b78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:41.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.019+0000 7f527ba00700 1 --2- 172.21.15.89:0/3952048948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52740cc590 0x7f52740b78b0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f526c014040 tx=0x7f526c004ad0 comp rx=0 tx=0).stop 2024-04-02T07:54:41.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.019+0000 7f527ba00700 1 -- 172.21.15.89:0/3952048948 shutdown_connections 2024-04-02T07:54:41.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.019+0000 7f527ba00700 1 --2- 172.21.15.89:0/3952048948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f525406ae00 0x7f525406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:41.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.019+0000 7f527ba00700 1 --2- 172.21.15.89:0/3952048948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52740cf1b0 0x7f52741f27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:41.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.019+0000 7f527ba00700 1 --2- 172.21.15.89:0/3952048948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52740cc590 0x7f52740b78b0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:41.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.019+0000 7f527ba00700 1 --2- 172.21.15.89:0/3952048948 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f527415d8a0 0x7f52740b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:41.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.019+0000 7f527ba00700 1 -- 172.21.15.89:0/3952048948 >> 172.21.15.89:0/3952048948 conn(0x7f527408aff0 msgr2=0x7f527415df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:41.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.019+0000 7f527ba00700 1 -- 172.21.15.89:0/3952048948 shutdown_connections 2024-04-02T07:54:41.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.019+0000 7f527ba00700 1 -- 172.21.15.89:0/3952048948 wait complete. 2024-04-02T07:54:41.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:41.038 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-02T07:54:41.087 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-04-02T07:54:41.280 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:41.278+0000 7f5e92b53240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:54:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.296+0000 7ff119a00700 1 Processor -- start 2024-04-02T07:54:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.296+0000 7ff119a00700 1 -- start start 2024-04-02T07:54:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.297+0000 7ff119a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff11415f1e0 0x7ff11415f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.297+0000 7ff119a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff1140cf3f0 0x7ff11415c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.297+0000 7ff119a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1140ccdd0 con 0x7ff1140cef40 2024-04-02T07:54:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.297+0000 7ff119a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1140cd260 con 0x7ff11415f1e0 2024-04-02T07:54:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.297+0000 7ff119a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1141600c0 con 0x7ff1140cf3f0 2024-04-02T07:54:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.297+0000 7ff112a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff11415f1e0 0x7ff11415f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.297+0000 7ff112a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff11415f1e0 0x7ff11415f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37696/0 (socket says 172.21.15.89:37696) 2024-04-02T07:54:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.297+0000 7ff112a00700 1 -- 172.21.15.89:0/1545230207 learned_addr learned my addr 172.21.15.89:0/1545230207 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.297+0000 7ff113400700 1 --2- 172.21.15.89:0/1545230207 >> v2:172.21.15.113:3300/0 conn(0x7ff1140cf3f0 0x7ff11415c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.298+0000 7ff112a00700 1 -- 172.21.15.89:0/1545230207 >> v2:172.21.15.113:3300/0 conn(0x7ff1140cf3f0 msgr2=0x7ff11415c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.298+0000 7ff112a00700 1 --2- 172.21.15.89:0/1545230207 >> v2:172.21.15.113:3300/0 conn(0x7ff1140cf3f0 0x7ff11415c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.298+0000 7ff112a00700 1 -- 172.21.15.89:0/1545230207 >> v1:172.21.15.113:6789/0 conn(0x7ff1140cef40 legacy=0x7ff11415bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.298+0000 7ff112a00700 1 -- 172.21.15.89:0/1545230207 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11415fb00 con 0x7ff11415f1e0 2024-04-02T07:54:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.298+0000 7ff112a00700 1 --2- 172.21.15.89:0/1545230207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff11415f1e0 0x7ff11415f5c0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff1040098f0 tx=0x7ff104009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cfbf9eff1bdc769 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.298+0000 7ff111600700 1 -- 172.21.15.89:0/1545230207 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104011830 con 0x7ff11415f1e0 2024-04-02T07:54:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.298+0000 7ff111600700 1 -- 172.21.15.89:0/1545230207 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff104011990 con 0x7ff11415f1e0 2024-04-02T07:54:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.298+0000 7ff111600700 1 -- 172.21.15.89:0/1545230207 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104011d20 con 0x7ff11415f1e0 2024-04-02T07:54:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.299+0000 7ff119a00700 1 -- 172.21.15.89:0/1545230207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff11415f1e0 msgr2=0x7ff11415f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.299+0000 7ff119a00700 1 --2- 172.21.15.89:0/1545230207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff11415f1e0 0x7ff11415f5c0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff1040098f0 tx=0x7ff104009c00 comp rx=0 tx=0).stop 2024-04-02T07:54:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.299+0000 7ff119a00700 1 -- 172.21.15.89:0/1545230207 shutdown_connections 2024-04-02T07:54:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.299+0000 7ff119a00700 1 --2- 172.21.15.89:0/1545230207 >> v2:172.21.15.113:3300/0 conn(0x7ff1140cf3f0 0x7ff11415c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.299+0000 7ff119a00700 1 --2- 172.21.15.89:0/1545230207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff11415f1e0 0x7ff11415f5c0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.299+0000 7ff119a00700 1 -- 172.21.15.89:0/1545230207 >> 172.21.15.89:0/1545230207 conn(0x7ff1140d1370 msgr2=0x7ff1140d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.299+0000 7ff119a00700 1 -- 172.21.15.89:0/1545230207 shutdown_connections 2024-04-02T07:54:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.299+0000 7ff119a00700 1 -- 172.21.15.89:0/1545230207 wait complete. 2024-04-02T07:54:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.301+0000 7ff119a00700 1 Processor -- start 2024-04-02T07:54:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.301+0000 7ff119a00700 1 -- start start 2024-04-02T07:54:41.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff119a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1140cef40 0x7ff11414b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:41.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff119a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1140cf3f0 0x7ff1140c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:41.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff119a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff11415f1e0 0x7ff1140c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:41.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff119a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1140afd70 con 0x7ff1140cef40 2024-04-02T07:54:41.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff119a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1140b6fa0 con 0x7ff11415f1e0 2024-04-02T07:54:41.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff119a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1141f45f0 con 0x7ff1140cf3f0 2024-04-02T07:54:41.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff112a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1140cef40 0x7ff11414b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:41.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff112a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1140cef40 0x7ff11414b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37704/0 (socket says 172.21.15.89:37704) 2024-04-02T07:54:41.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff112a00700 1 -- 172.21.15.89:0/244972794 learned_addr learned my addr 172.21.15.89:0/244972794 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:41.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff113400700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff11415f1e0 0x7ff1140c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:41.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff112000700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1140cf3f0 0x7ff1140c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:41.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff112a00700 1 -- 172.21.15.89:0/244972794 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1140cf3f0 msgr2=0x7ff1140c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:41.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff112a00700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1140cf3f0 0x7ff1140c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:41.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff112a00700 1 -- 172.21.15.89:0/244972794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff11415f1e0 msgr2=0x7ff1140c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:41.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff112a00700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff11415f1e0 0x7ff1140c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:41.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.302+0000 7ff112a00700 1 -- 172.21.15.89:0/244972794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff104009550 con 0x7ff1140cef40 2024-04-02T07:54:41.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.303+0000 7ff112a00700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1140cef40 0x7ff11414b460 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7ff104014040 tx=0x7ff104004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:41.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.303+0000 7ff10aa00700 1 -- 172.21.15.89:0/244972794 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104004590 con 0x7ff1140cef40 2024-04-02T07:54:41.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.303+0000 7ff10aa00700 1 -- 172.21.15.89:0/244972794 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10401e070 con 0x7ff1140cef40 2024-04-02T07:54:41.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.303+0000 7ff10aa00700 1 -- 172.21.15.89:0/244972794 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040038e0 con 0x7ff1140cef40 2024-04-02T07:54:41.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.303+0000 7ff119a00700 1 -- 172.21.15.89:0/244972794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1141f48d0 con 0x7ff1140cef40 2024-04-02T07:54:41.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.303+0000 7ff119a00700 1 -- 172.21.15.89:0/244972794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1141f4200 con 0x7ff1140cef40 2024-04-02T07:54:41.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.304+0000 7ff119a00700 1 -- 172.21.15.89:0/244972794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1141f4450 con 0x7ff1140cef40 2024-04-02T07:54:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.305+0000 7ff10aa00700 1 -- 172.21.15.89:0/244972794 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 66) v1 ==== 89256+0+0 (secure 0 0 0) 0x7ff104003a40 con 0x7ff1140cef40 2024-04-02T07:54:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.306+0000 7ff10aa00700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 0x7ff0f406d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.306+0000 7ff10aa00700 1 -- 172.21.15.89:0/244972794 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff10408a560 con 0x7ff1140cef40 2024-04-02T07:54:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.306+0000 7ff112000700 1 -- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 msgr2=0x7ff0f406d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.306+0000 7ff112000700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 0x7ff0f406d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:54:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.315+0000 7ff10aa00700 1 -- 172.21.15.89:0/244972794 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff1141f4450 con 0x7ff1140cef40 2024-04-02T07:54:41.395 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:41.394+0000 7fce09bbd240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:54:41.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.507+0000 7ff112000700 1 -- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 msgr2=0x7ff0f406d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:41.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.507+0000 7ff112000700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 0x7ff0f406d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:54:41.534 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:41.533+0000 7f5e92b53240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:54:41.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.542+0000 7ff119a00700 1 -- 172.21.15.89:0/244972794 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1140afc00 con 0x7ff0f406ae30 2024-04-02T07:54:41.651 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:41.650+0000 7fce09bbd240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:54:41.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.908+0000 7ff112000700 1 -- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 msgr2=0x7ff0f406d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:41.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:41.908+0000 7ff112000700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 0x7ff0f406d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T07:54:41.990 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:41.989+0000 7f5e92b53240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:54:42.107 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:42.106+0000 7fce09bbd240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:54:42.692 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:42.691+0000 7f5e92b53240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:54:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:42.709+0000 7ff112000700 1 -- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 msgr2=0x7ff0f406d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:42.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:42.709+0000 7ff112000700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 0x7ff0f406d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T07:54:42.784 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:42.783+0000 7f5e92b53240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:54:42.815 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:42.815+0000 7fce09bbd240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:54:42.907 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:42.906+0000 7fce09bbd240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:54:42.941 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:42.940+0000 7f5e92b53240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:54:43.067 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:43.066+0000 7fce09bbd240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:54:43.153 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:43.152+0000 7f5e92b53240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:54:43.244 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:43.243+0000 7f5e92b53240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:54:43.282 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:43.281+0000 7fce09bbd240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:54:43.341 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:43.340+0000 7f5e92b53240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:54:43.374 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:43.374+0000 7fce09bbd240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:54:43.440 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:43.439+0000 7f5e92b53240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:54:43.474 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:43.473+0000 7fce09bbd240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:54:43.538 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:43.537+0000 7f5e92b53240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:54:43.572 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:43.572+0000 7fce09bbd240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:54:43.671 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:43.670+0000 7fce09bbd240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:54:43.749 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:43.748+0000 7f5e92b53240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:54:43.884 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:43.884+0000 7fce09bbd240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:54:44.003 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:44.002+0000 7f5e92b53240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:54:44.140 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:44.140+0000 7fce09bbd240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:54:44.195 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:44.193+0000 7f5e92b53240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:54:44.286 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:44.285+0000 7f5e92b53240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:54:44.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:44.310+0000 7ff112000700 1 -- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 msgr2=0x7ff0f406d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:54:44.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:44.310+0000 7ff112000700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 0x7ff0f406d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T07:54:44.332 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:44.332+0000 7fce09bbd240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:54:44.427 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:44.426+0000 7fce09bbd240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:54:45.009 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:45.007+0000 7f5e92b53240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:54:45.160 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:45.159+0000 7fce09bbd240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:54:45.220 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:45.219+0000 7f5e92b53240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:54:45.374 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:45.373+0000 7fce09bbd240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:54:45.425 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:45.424+0000 7f5e92b53240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:54:45.583 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:45.582+0000 7fce09bbd240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:54:45.895 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:45.893+0000 7f5e92b53240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:54:46.061 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:46.060+0000 7fce09bbd240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:54:46.101 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:46.100+0000 7f5e92b53240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:54:46.270 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:46.270+0000 7fce09bbd240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:54:46.534 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:46.533+0000 7f5e92b53240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:54:46.649 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:54:46.647+0000 7f5e92b53240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:54:46.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:46.708+0000 7ff10aa00700 1 -- 172.21.15.89:0/244972794 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 67) v1 ==== 89256+0+0 (secure 0 0 0) 0x7ff104003cf0 con 0x7ff1140cef40 2024-04-02T07:54:46.722 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:46.721+0000 7fce09bbd240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:54:46.834 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:46.833+0000 7fce09bbd240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:54:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:46.843+0000 7ff10aa00700 1 -- 172.21.15.89:0/244972794 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 68) v1 ==== 88865+0+0 (secure 0 0 0) 0x7ff104055930 con 0x7ff1140cef40 2024-04-02T07:54:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:46.843+0000 7ff10aa00700 1 -- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 msgr2=0x7ff0f406d2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:54:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:46.843+0000 7ff10aa00700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 0x7ff0f406d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:47.025 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:47.024+0000 7fcd76000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:47.025 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:47.024+0000 7fcd76000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:47.025 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:47.024+0000 7fcd76000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:47.025 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:47.024+0000 7fcd76000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:47.025 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:47.024+0000 7fcd76000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:47.025 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:47.024+0000 7fcd71a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:47.026 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:47.024+0000 7fcd71a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:47.026 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:47.024+0000 7fcd71a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:47.026 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:47.024+0000 7fcd71a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:47.026 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:54:47.024+0000 7fcd71a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:54:47.077 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:54:47.081 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:54:47.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.847+0000 7ff10aa00700 1 -- 172.21.15.89:0/244972794 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mgrmap(e 69) v1 ==== 88992+0+0 (secure 0 0 0) 0x7ff10408ed70 con 0x7ff1140cef40 2024-04-02T07:54:47.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.847+0000 7ff10aa00700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 0x7ff0f406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:47.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.847+0000 7ff10aa00700 1 -- 172.21.15.89:0/244972794 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1140afc00 con 0x7ff0f406ae30 2024-04-02T07:54:47.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.852+0000 7ff112000700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 0x7ff0f406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:47.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.856+0000 7ff112000700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 0x7ff0f406d270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff1140b1610 tx=0x7ff1000011c0 comp rx=0 tx=0).ready entity=mgr.8103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:47.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.956+0000 7ff10aa00700 1 -- 172.21.15.89:0/244972794 <== mgr.8103 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff1140afc00 con 0x7ff0f406ae30 2024-04-02T07:54:47.956 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path snap_test_dir1 2024-04-02T07:54:47.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.958+0000 7ff119a00700 1 -- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 msgr2=0x7ff0f406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:47.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.958+0000 7ff119a00700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 0x7ff0f406d270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff1140b1610 tx=0x7ff1000011c0 comp rx=0 tx=0).stop 2024-04-02T07:54:47.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.958+0000 7ff119a00700 1 -- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1140cef40 msgr2=0x7ff11414b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:47.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.958+0000 7ff119a00700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1140cef40 0x7ff11414b460 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7ff104014040 tx=0x7ff104004750 comp rx=0 tx=0).stop 2024-04-02T07:54:47.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.959+0000 7ff119a00700 1 -- 172.21.15.89:0/244972794 shutdown_connections 2024-04-02T07:54:47.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.959+0000 7ff119a00700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff0f406ae30 0x7ff0f406d270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:47.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.959+0000 7ff119a00700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff11415f1e0 0x7ff1140c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:47.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.959+0000 7ff119a00700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1140cef40 0x7ff11414b460 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:47.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.959+0000 7ff119a00700 1 --2- 172.21.15.89:0/244972794 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1140cf3f0 0x7ff1140c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:47.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.959+0000 7ff119a00700 1 -- 172.21.15.89:0/244972794 >> 172.21.15.89:0/244972794 conn(0x7ff1140d1370 msgr2=0x7ff1140cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:47.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.959+0000 7ff119a00700 1 -- 172.21.15.89:0/244972794 shutdown_connections 2024-04-02T07:54:47.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:47.959+0000 7ff119a00700 1 -- 172.21.15.89:0/244972794 wait complete. 2024-04-02T07:54:47.979 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-02T07:54:47.979 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-02-07_55_00 in ~13s... 2024-04-02T07:54:47.979 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-04-02T07:54:47.979 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-04-02T07:54:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.172+0000 7f1336800700 1 Processor -- start 2024-04-02T07:54:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.172+0000 7f1336800700 1 -- start start 2024-04-02T07:54:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.173+0000 7f1336800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f13300cd020 0x7f13300cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.173+0000 7f1336800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13300b2070 0x7f13300b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.173+0000 7f1336800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13300cdae0 con 0x7f13300cbb30 2024-04-02T07:54:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.173+0000 7f1336800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13300cddc0 con 0x7f13300cd020 2024-04-02T07:54:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.173+0000 7f1336800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13300ce0a0 con 0x7f13300b2070 2024-04-02T07:54:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.174+0000 7f132f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f13300cbb30 0x7f13300cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38798/0 (socket says 172.21.15.89:38798) 2024-04-02T07:54:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.174+0000 7f132f400700 1 -- 172.21.15.89:0/2704238071 learned_addr learned my addr 172.21.15.89:0/2704238071 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:54:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.174+0000 7f132fe00700 1 --2- 172.21.15.89:0/2704238071 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13300b2070 0x7f13300b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.174+0000 7f1327e00700 1 --2- 172.21.15.89:0/2704238071 >> v2:172.21.15.113:3300/0 conn(0x7f13300cd020 0x7f13300cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.174+0000 7f132ea00700 1 -- 172.21.15.89:0/2704238071 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061325768 0 0) 0x7f13300cdae0 con 0x7f13300cbb30 2024-04-02T07:54:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.174+0000 7f132ea00700 1 -- 172.21.15.89:0/2704238071 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13140035b0 con 0x7f13300cbb30 2024-04-02T07:54:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.174+0000 7f132fe00700 1 -- 172.21.15.89:0/2704238071 >> v2:172.21.15.113:3300/0 conn(0x7f13300cd020 msgr2=0x7f13300cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:54:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.174+0000 7f132fe00700 1 --2- 172.21.15.89:0/2704238071 >> v2:172.21.15.113:3300/0 conn(0x7f13300cd020 0x7f13300cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.174+0000 7f132fe00700 1 -- 172.21.15.89:0/2704238071 >> v1:172.21.15.89:6789/0 conn(0x7f13300cbb30 legacy=0x7f13300cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.174+0000 7f132fe00700 1 -- 172.21.15.89:0/2704238071 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133006dc20 con 0x7f13300b2070 2024-04-02T07:54:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.175+0000 7f132fe00700 1 --2- 172.21.15.89:0/2704238071 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13300b2070 0x7f13300b2500 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1328009d90 tx=0x7f132800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e8b161540832cc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.175+0000 7f132ea00700 1 -- 172.21.15.89:0/2704238071 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132800e070 con 0x7f13300b2070 2024-04-02T07:54:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.175+0000 7f132ea00700 1 -- 172.21.15.89:0/2704238071 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132800bb20 con 0x7f13300b2070 2024-04-02T07:54:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.175+0000 7f132ea00700 1 -- 172.21.15.89:0/2704238071 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328012430 con 0x7f13300b2070 2024-04-02T07:54:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.175+0000 7f1336800700 1 -- 172.21.15.89:0/2704238071 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13300b2070 msgr2=0x7f13300b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.175+0000 7f1336800700 1 --2- 172.21.15.89:0/2704238071 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13300b2070 0x7f13300b2500 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1328009d90 tx=0x7f132800b330 comp rx=0 tx=0).stop 2024-04-02T07:54:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.175+0000 7f1336800700 1 -- 172.21.15.89:0/2704238071 shutdown_connections 2024-04-02T07:54:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.175+0000 7f1336800700 1 --2- 172.21.15.89:0/2704238071 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13300b2070 0x7f13300b2500 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.175+0000 7f1336800700 1 --2- 172.21.15.89:0/2704238071 >> v2:172.21.15.113:3300/0 conn(0x7f13300cd020 0x7f13300cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.175+0000 7f1336800700 1 -- 172.21.15.89:0/2704238071 >> 172.21.15.89:0/2704238071 conn(0x7f13300b0d70 msgr2=0x7f13300b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.176+0000 7f1336800700 1 -- 172.21.15.89:0/2704238071 shutdown_connections 2024-04-02T07:54:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.176+0000 7f1336800700 1 -- 172.21.15.89:0/2704238071 wait complete. 2024-04-02T07:54:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.177+0000 7f1336800700 1 Processor -- start 2024-04-02T07:54:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.177+0000 7f1336800700 1 -- start start 2024-04-02T07:54:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.178+0000 7f1336800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13300b2070 0x7f133014b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.178+0000 7f1336800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13300cbb30 0x7f13300c3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.178+0000 7f1336800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13300cd020 0x7f13301f05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.178+0000 7f1336800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13300c4520 con 0x7f13300cd020 2024-04-02T07:54:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.178+0000 7f1336800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13301f4400 con 0x7f13300cbb30 2024-04-02T07:54:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.178+0000 7f1336800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13301f46e0 con 0x7f13300b2070 2024-04-02T07:54:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.179+0000 7f132fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13300cd020 0x7f13301f05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.179+0000 7f1327e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13300cbb30 0x7f13300c3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.179+0000 7f132fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13300cd020 0x7f13301f05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37836/0 (socket says 172.21.15.89:37836) 2024-04-02T07:54:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.179+0000 7f132fe00700 1 -- 172.21.15.89:0/2902443628 learned_addr learned my addr 172.21.15.89:0/2902443628 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:54:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.179+0000 7f132f400700 1 --2- 172.21.15.89:0/2902443628 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13300b2070 0x7f133014b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.179+0000 7f132fe00700 1 -- 172.21.15.89:0/2902443628 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13300b2070 msgr2=0x7f133014b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.179+0000 7f132fe00700 1 --2- 172.21.15.89:0/2902443628 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13300b2070 0x7f133014b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.179+0000 7f132fe00700 1 -- 172.21.15.89:0/2902443628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13300cbb30 msgr2=0x7f13300c3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.179+0000 7f132fe00700 1 --2- 172.21.15.89:0/2902443628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13300cbb30 0x7f13300c3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.179+0000 7f132fe00700 1 -- 172.21.15.89:0/2902443628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1328009a70 con 0x7f13300cd020 2024-04-02T07:54:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.179+0000 7f132fe00700 1 --2- 172.21.15.89:0/2902443628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13300cd020 0x7f13301f05d0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f1328014040 tx=0x7f1328004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:48.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.179+0000 7f1327400700 1 -- 172.21.15.89:0/2902443628 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13280129f0 con 0x7f13300cd020 2024-04-02T07:54:48.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.180+0000 7f1327400700 1 -- 172.21.15.89:0/2902443628 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1328020070 con 0x7f13300cd020 2024-04-02T07:54:48.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.180+0000 7f1336800700 1 -- 172.21.15.89:0/2902443628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13301f49c0 con 0x7f13300cd020 2024-04-02T07:54:48.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.180+0000 7f1327400700 1 -- 172.21.15.89:0/2902443628 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328012db0 con 0x7f13300cd020 2024-04-02T07:54:48.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.180+0000 7f1336800700 1 -- 172.21.15.89:0/2902443628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13301f4f80 con 0x7f13300cd020 2024-04-02T07:54:48.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.181+0000 7f1336800700 1 -- 172.21.15.89:0/2902443628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1330067720 con 0x7f13300cd020 2024-04-02T07:54:48.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.181+0000 7f1327400700 1 -- 172.21.15.89:0/2902443628 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 69) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f1328004040 con 0x7f13300cd020 2024-04-02T07:54:48.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.182+0000 7f1327400700 1 --2- 172.21.15.89:0/2902443628 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f130c06ab30 0x7f130c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:54:48.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.182+0000 7f1327400700 1 -- 172.21.15.89:0/2902443628 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f1328022070 con 0x7f13300cd020 2024-04-02T07:54:48.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.182+0000 7f132f400700 1 --2- 172.21.15.89:0/2902443628 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f130c06ab30 0x7f130c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:54:48.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.183+0000 7f132f400700 1 --2- 172.21.15.89:0/2902443628 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f130c06ab30 0x7f130c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1320007fd0 tx=0x7f1320007dc0 comp rx=0 tx=0).ready entity=mgr.8103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:54:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.187+0000 7f1327400700 1 -- 172.21.15.89:0/2902443628 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1328056030 con 0x7f13300cd020 2024-04-02T07:54:48.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.413+0000 7f1336800700 1 -- 172.21.15.89:0/2902443628 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f133015bdd0 con 0x7f130c06ab30 2024-04-02T07:54:48.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.425+0000 7f1327400700 1 -- 172.21.15.89:0/2902443628 <== mgr.8103 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f1320006bd0 con 0x7f130c06ab30 2024-04-02T07:54:48.426 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:54:48.426 INFO:teuthology.orchestra.run.smithi089.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-04-02T07:54:48.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.427+0000 7f1336800700 1 -- 172.21.15.89:0/2902443628 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f130c06ab30 msgr2=0x7f130c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:48.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.427+0000 7f1336800700 1 --2- 172.21.15.89:0/2902443628 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f130c06ab30 0x7f130c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1320007fd0 tx=0x7f1320007dc0 comp rx=0 tx=0).stop 2024-04-02T07:54:48.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.427+0000 7f1336800700 1 -- 172.21.15.89:0/2902443628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13300cd020 msgr2=0x7f13301f05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:54:48.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.427+0000 7f1336800700 1 --2- 172.21.15.89:0/2902443628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13300cd020 0x7f13301f05d0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f1328014040 tx=0x7f1328004ad0 comp rx=0 tx=0).stop 2024-04-02T07:54:48.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.427+0000 7f1336800700 1 -- 172.21.15.89:0/2902443628 shutdown_connections 2024-04-02T07:54:48.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.427+0000 7f1336800700 1 --2- 172.21.15.89:0/2902443628 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f130c06ab30 0x7f130c06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:48.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.427+0000 7f1336800700 1 --2- 172.21.15.89:0/2902443628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13300cd020 0x7f13301f05d0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:48.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.427+0000 7f1336800700 1 --2- 172.21.15.89:0/2902443628 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13300b2070 0x7f133014b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:48.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.427+0000 7f1336800700 1 --2- 172.21.15.89:0/2902443628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13300cbb30 0x7f13300c3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:54:48.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.427+0000 7f1336800700 1 -- 172.21.15.89:0/2902443628 >> 172.21.15.89:0/2902443628 conn(0x7f13300b0d70 msgr2=0x7f13300ce540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:54:48.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.427+0000 7f1336800700 1 -- 172.21.15.89:0/2902443628 shutdown_connections 2024-04-02T07:54:48.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:54:48.428+0000 7f1336800700 1 -- 172.21.15.89:0/2902443628 wait complete. 2024-04-02T07:54:48.446 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-04-02T07:54:48.446 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2024-04-02T07:54:48.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:48.446 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:54:48.493 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:54:48.493 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:54:48.493 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:54:49.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:49.494 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:54:49.542 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:54:49.542 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:54:49.542 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:54:50.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:50.543 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:54:50.595 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:54:50.595 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:54:50.595 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:54:51.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:51.596 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:54:51.648 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:54:51.648 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:54:51.648 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:54:52.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:52.903 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:54:52.955 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:54:52.955 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:54:52.955 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:54:53.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:53.956 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:54:54.007 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:54:54.008 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:54:54.008 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:54:55.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:55.009 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:54:55.061 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:54:55.061 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:54:55.061 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:54:56.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:56.062 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:54:56.114 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:54:56.114 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:54:56.114 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:54:57.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:57.115 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:54:57.167 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:54:57.167 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:54:57.167 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:54:58.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:58.168 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:54:58.219 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:54:58.219 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:54:58.219 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:54:59.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:54:59.221 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:54:59.273 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:54:59.273 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:54:59.273 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:55:00.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:00.274 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:55:00.325 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T07:55:00.325 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T07:55:00.325 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:55:00.970 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:00.969+0000 7fcd61600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:00.971 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:00.969+0000 7fcd61600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:00.971 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:00.969+0000 7fcd61600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:00.971 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:00.969+0000 7fcd61600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:00.971 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:00.969+0000 7fcd61600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:01.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:01.327 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:55:01.377 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_55_00_UTC 2024-04-02T07:55:01.378 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-07_55_00_UTC'} 2024-04-02T07:55:01.378 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-02-07_55_00_UTC'} 2024-04-02T07:55:01.378 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T07:55:01.378 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-04-02-07_55_00_UTC'] 2024-04-02T07:55:01.378 DEBUG:tasks.cephfs.test_snap_schedules:exec=1712044487.979236, now = 1712044501.378916, timo = 13 2024-04-02T07:55:02.379 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-02T07:55:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.585+0000 7f1081a00700 1 Processor -- start 2024-04-02T07:55:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.585+0000 7f1081a00700 1 -- start start 2024-04-02T07:55:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.586+0000 7f1081a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f107c06fa20 0x7f107c15e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.586+0000 7f1081a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c15eda0 con 0x7f107c0cd5c0 2024-04-02T07:55:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.586+0000 7f1081a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c099dd0 con 0x7f107c15cfe0 2024-04-02T07:55:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.586+0000 7f1081a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f107c09a0b0 con 0x7f107c06fa20 2024-04-02T07:55:02.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.586+0000 7f107aa00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f107c0cd5c0 0x7f107c06f310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35650/0 (socket says 172.21.15.89:35650) 2024-04-02T07:55:02.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f107aa00700 1 -- 172.21.15.89:0/1185624644 learned_addr learned my addr 172.21.15.89:0/1185624644 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:02.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f107a000700 1 --2- 172.21.15.89:0/1185624644 >> v2:172.21.15.113:3300/0 conn(0x7f107c06fa20 0x7f107c15e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:02.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 -- 172.21.15.89:0/1185624644 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500957539 0 0) 0x7f106c001110 con 0x7f107c0cd5c0 2024-04-02T07:55:02.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 -- 172.21.15.89:0/1185624644 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10600035b0 con 0x7f107c0cd5c0 2024-04-02T07:55:02.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 -- 172.21.15.89:0/1185624644 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2359373885 0 0) 0x7f10600035b0 con 0x7f107c0cd5c0 2024-04-02T07:55:02.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 -- 172.21.15.89:0/1185624644 >> v2:172.21.15.113:3300/0 conn(0x7f107c06fa20 msgr2=0x7f107c15e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:02.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 --2- 172.21.15.89:0/1185624644 >> v2:172.21.15.113:3300/0 conn(0x7f107c06fa20 0x7f107c15e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:02.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 -- 172.21.15.89:0/1185624644 >> v1:172.21.15.113:6789/0 conn(0x7f107c15cfe0 legacy=0x7f107c15d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 -- 172.21.15.89:0/1185624644 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c09a390 con 0x7f107c0cd5c0 2024-04-02T07:55:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 -- 172.21.15.89:0/1185624644 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f106c001800 con 0x7f107c0cd5c0 2024-04-02T07:55:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 -- 172.21.15.89:0/1185624644 >> v1:172.21.15.89:6789/0 conn(0x7f107c0cd5c0 legacy=0x7f107c06f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 --2- 172.21.15.89:0/1185624644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1060003a30 0x7f1060003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 --2- 172.21.15.89:0/1185624644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1060004850 0x7f1060004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 --2- 172.21.15.89:0/1185624644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1060005210 0x7f106000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 -- 172.21.15.89:0/1185624644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c09a0b0 con 0x7f1060003a30 2024-04-02T07:55:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 -- 172.21.15.89:0/1185624644 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106000c0c0 con 0x7f1060005210 2024-04-02T07:55:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.587+0000 7f1079600700 1 -- 172.21.15.89:0/1185624644 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106000c280 con 0x7f1060004850 2024-04-02T07:55:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.588+0000 7f107aa00700 1 --2- 172.21.15.89:0/1185624644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1060003a30 0x7f1060003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.588+0000 7f107a000700 1 --2- 172.21.15.89:0/1185624644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1060004850 0x7f1060004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.588+0000 7f107b400700 1 --2- 172.21.15.89:0/1185624644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1060005210 0x7f106000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.588+0000 7f107aa00700 1 -- 172.21.15.89:0/1185624644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1060004850 msgr2=0x7f1060004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.588+0000 7f107aa00700 1 --2- 172.21.15.89:0/1185624644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1060004850 0x7f1060004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.588+0000 7f107aa00700 1 -- 172.21.15.89:0/1185624644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1060005210 msgr2=0x7f106000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.588+0000 7f107aa00700 1 --2- 172.21.15.89:0/1185624644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1060005210 0x7f106000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.588+0000 7f107aa00700 1 -- 172.21.15.89:0/1185624644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10600035b0 con 0x7f1060003a30 2024-04-02T07:55:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.588+0000 7f107a000700 1 --2- 172.21.15.89:0/1185624644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1060004850 0x7f1060004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:55:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.588+0000 7f107b400700 1 --2- 172.21.15.89:0/1185624644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1060005210 0x7f106000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:55:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.588+0000 7f107b400700 1 -- 172.21.15.89:0/1185624644 reap_dead start 2024-04-02T07:55:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.588+0000 7f107aa00700 1 --2- 172.21.15.89:0/1185624644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1060003a30 0x7f1060003e50 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f106c00c210 tx=0x7f1060005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c4911bf913fc71 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.589+0000 7f1079600700 1 -- 172.21.15.89:0/1185624644 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c00cdb0 con 0x7f1060003a30 2024-04-02T07:55:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.589+0000 7f1079600700 1 -- 172.21.15.89:0/1185624644 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c005960 con 0x7f1060003a30 2024-04-02T07:55:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.589+0000 7f1079600700 1 -- 172.21.15.89:0/1185624644 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c005d10 con 0x7f1060003a30 2024-04-02T07:55:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.589+0000 7f1081a00700 1 -- 172.21.15.89:0/1185624644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1060003a30 msgr2=0x7f1060003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.589+0000 7f1081a00700 1 --2- 172.21.15.89:0/1185624644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1060003a30 0x7f1060003e50 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f106c00c210 tx=0x7f1060005880 comp rx=0 tx=0).stop 2024-04-02T07:55:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.589+0000 7f1081a00700 1 -- 172.21.15.89:0/1185624644 shutdown_connections 2024-04-02T07:55:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.589+0000 7f1081a00700 1 --2- 172.21.15.89:0/1185624644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1060003a30 0x7f1060003e50 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:02.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.589+0000 7f1081a00700 1 -- 172.21.15.89:0/1185624644 >> 172.21.15.89:0/1185624644 conn(0x7f107c0d1370 msgr2=0x7f107c0d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:02.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.589+0000 7f1081a00700 1 -- 172.21.15.89:0/1185624644 shutdown_connections 2024-04-02T07:55:02.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.589+0000 7f1081a00700 1 -- 172.21.15.89:0/1185624644 wait complete. 2024-04-02T07:55:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.592+0000 7f1081a00700 1 Processor -- start 2024-04-02T07:55:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.592+0000 7f1081a00700 1 -- start start 2024-04-02T07:55:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.593+0000 7f1081a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1060003a30 0x7f107c06ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.593+0000 7f1081a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f107c06cf60 0x7f107c0c5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.593+0000 7f1081a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f107c0c56d0 0x7f107c06b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.593+0000 7f1081a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c0c5b50 con 0x7f1060003a30 2024-04-02T07:55:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.593+0000 7f1081a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c15fdd0 con 0x7f107c0c56d0 2024-04-02T07:55:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.593+0000 7f1081a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107c1600b0 con 0x7f107c06cf60 2024-04-02T07:55:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.593+0000 7f107aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1060003a30 0x7f107c06ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.593+0000 7f107aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1060003a30 0x7f107c06ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38478/0 (socket says 172.21.15.89:38478) 2024-04-02T07:55:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.593+0000 7f107aa00700 1 -- 172.21.15.89:0/3480600537 learned_addr learned my addr 172.21.15.89:0/3480600537 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.593+0000 7f107a000700 1 --2- 172.21.15.89:0/3480600537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f107c06cf60 0x7f107c0c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.593+0000 7f107b400700 1 --2- 172.21.15.89:0/3480600537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f107c0c56d0 0x7f107c06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.594+0000 7f107aa00700 1 -- 172.21.15.89:0/3480600537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f107c06cf60 msgr2=0x7f107c0c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.594+0000 7f107aa00700 1 --2- 172.21.15.89:0/3480600537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f107c06cf60 0x7f107c0c5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.594+0000 7f107aa00700 1 -- 172.21.15.89:0/3480600537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f107c0c56d0 msgr2=0x7f107c06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.594+0000 7f107aa00700 1 --2- 172.21.15.89:0/3480600537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f107c0c56d0 0x7f107c06b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.594+0000 7f107aa00700 1 -- 172.21.15.89:0/3480600537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c160390 con 0x7f1060003a30 2024-04-02T07:55:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.594+0000 7f107aa00700 1 --2- 172.21.15.89:0/3480600537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1060003a30 0x7f107c06ca20 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f106000c090 tx=0x7f106c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.594+0000 7f1072a00700 1 -- 172.21.15.89:0/3480600537 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c004870 con 0x7f1060003a30 2024-04-02T07:55:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.594+0000 7f1072a00700 1 -- 172.21.15.89:0/3480600537 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c0049f0 con 0x7f1060003a30 2024-04-02T07:55:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.594+0000 7f1081a00700 1 -- 172.21.15.89:0/3480600537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107c160670 con 0x7f1060003a30 2024-04-02T07:55:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.594+0000 7f1072a00700 1 -- 172.21.15.89:0/3480600537 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c004da0 con 0x7f1060003a30 2024-04-02T07:55:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.594+0000 7f1081a00700 1 -- 172.21.15.89:0/3480600537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107c15fa60 con 0x7f1060003a30 2024-04-02T07:55:02.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.595+0000 7f1081a00700 1 -- 172.21.15.89:0/3480600537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107c067720 con 0x7f1060003a30 2024-04-02T07:55:02.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.596+0000 7f1072a00700 1 -- 172.21.15.89:0/3480600537 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 71) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f106c005040 con 0x7f1060003a30 2024-04-02T07:55:02.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.597+0000 7f1072a00700 1 --2- 172.21.15.89:0/3480600537 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f105c06ae90 0x7f105c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:02.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.597+0000 7f1072a00700 1 -- 172.21.15.89:0/3480600537 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f106c08a4d0 con 0x7f1060003a30 2024-04-02T07:55:02.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.597+0000 7f107a000700 1 --2- 172.21.15.89:0/3480600537 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f105c06ae90 0x7f105c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:02.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.598+0000 7f107a000700 1 --2- 172.21.15.89:0/3480600537 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f105c06ae90 0x7f105c06d350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f107c0b1530 tx=0x7f1068009210 comp rx=0 tx=0).ready entity=mgr.8103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:02.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.602+0000 7f1072a00700 1 -- 172.21.15.89:0/3480600537 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f106c055220 con 0x7f1060003a30 2024-04-02T07:55:02.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.827+0000 7f1081a00700 1 -- 172.21.15.89:0/3480600537 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f107c0c72f0 con 0x7f105c06ae90 2024-04-02T07:55:02.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.916+0000 7f1072a00700 1 -- 172.21.15.89:0/3480600537 <== mgr.8103 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f107c0c72f0 con 0x7f105c06ae90 2024-04-02T07:55:02.918 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path snap_test_dir1 2024-04-02T07:55:02.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.919+0000 7f1081a00700 1 -- 172.21.15.89:0/3480600537 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f105c06ae90 msgr2=0x7f105c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:02.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.919+0000 7f1081a00700 1 --2- 172.21.15.89:0/3480600537 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f105c06ae90 0x7f105c06d350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f107c0b1530 tx=0x7f1068009210 comp rx=0 tx=0).stop 2024-04-02T07:55:02.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.919+0000 7f1081a00700 1 -- 172.21.15.89:0/3480600537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1060003a30 msgr2=0x7f107c06ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:02.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.919+0000 7f1081a00700 1 --2- 172.21.15.89:0/3480600537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1060003a30 0x7f107c06ca20 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f106000c090 tx=0x7f106c000fe0 comp rx=0 tx=0).stop 2024-04-02T07:55:02.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.919+0000 7f1081a00700 1 -- 172.21.15.89:0/3480600537 shutdown_connections 2024-04-02T07:55:02.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.919+0000 7f1081a00700 1 --2- 172.21.15.89:0/3480600537 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f105c06ae90 0x7f105c06d350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:02.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.919+0000 7f1081a00700 1 --2- 172.21.15.89:0/3480600537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f107c0c56d0 0x7f107c06b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:02.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.919+0000 7f1081a00700 1 --2- 172.21.15.89:0/3480600537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1060003a30 0x7f107c06ca20 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:02.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.919+0000 7f1081a00700 1 --2- 172.21.15.89:0/3480600537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f107c06cf60 0x7f107c0c5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:02.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.919+0000 7f1081a00700 1 -- 172.21.15.89:0/3480600537 >> 172.21.15.89:0/3480600537 conn(0x7f107c0d1370 msgr2=0x7f107c15b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:02.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.919+0000 7f1081a00700 1 -- 172.21.15.89:0/3480600537 shutdown_connections 2024-04-02T07:55:02.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:02.919+0000 7f1081a00700 1 -- 172.21.15.89:0/3480600537 wait complete. 2024-04-02T07:55:02.939 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-02T07:55:02.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:02.939 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T07:55:02.983 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_55_00_UTC 2024-04-02T07:55:02.985 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-07_55_00_UTC 2024-04-02T07:55:02.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:02.985 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-07_55_00_UTC) 2024-04-02T07:55:03.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:03.077 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-02T07:55:03.124 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T07:55:03.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.314+0000 7f7491000700 1 Processor -- start 2024-04-02T07:55:03.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.315+0000 7f7491000700 1 -- start start 2024-04-02T07:55:03.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.316+0000 7f7491000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f748c0cbb30 0x7f748c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:03.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.316+0000 7f7491000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f748c0cd020 0x7f748c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:03.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.316+0000 7f7491000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748c0cc450 con 0x7f748c0b2070 2024-04-02T07:55:03.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.316+0000 7f7491000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f748c0cdae0 con 0x7f748c0cbb30 2024-04-02T07:55:03.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.316+0000 7f7491000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f748c0cddc0 con 0x7f748c0cd020 2024-04-02T07:55:03.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.316+0000 7f748a800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f748c0b2070 0x7f748c06e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35662/0 (socket says 172.21.15.89:35662) 2024-04-02T07:55:03.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.316+0000 7f748a800700 1 -- 172.21.15.89:0/90100620 learned_addr learned my addr 172.21.15.89:0/90100620 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:03.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.316+0000 7f7489e00700 1 --2- 172.21.15.89:0/90100620 >> v2:172.21.15.89:3300/0 conn(0x7f748c0cbb30 0x7f748c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:03.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.316+0000 7f7489400700 1 --2- 172.21.15.89:0/90100620 >> v2:172.21.15.113:3300/0 conn(0x7f748c0cd020 0x7f748c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:03.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.316+0000 7f7489e00700 1 -- 172.21.15.89:0/90100620 >> v2:172.21.15.113:3300/0 conn(0x7f748c0cd020 msgr2=0x7f748c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:03.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.316+0000 7f7489e00700 1 --2- 172.21.15.89:0/90100620 >> v2:172.21.15.113:3300/0 conn(0x7f748c0cd020 0x7f748c0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:03.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.316+0000 7f7488a00700 1 -- 172.21.15.89:0/90100620 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854994070 0 0) 0x7f748c0cc450 con 0x7f748c0b2070 2024-04-02T07:55:03.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.316+0000 7f7489e00700 1 -- 172.21.15.89:0/90100620 >> v1:172.21.15.89:6789/0 conn(0x7f748c0b2070 legacy=0x7f748c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:03.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.316+0000 7f7489e00700 1 -- 172.21.15.89:0/90100620 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748c0ce0a0 con 0x7f748c0cbb30 2024-04-02T07:55:03.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.317+0000 7f7489e00700 1 --2- 172.21.15.89:0/90100620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f748c0cbb30 0x7f748c0cbf10 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f748c0b1e30 tx=0x7f747c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773b28f0bd0eeb36 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:03.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.317+0000 7f7488a00700 1 -- 172.21.15.89:0/90100620 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c011830 con 0x7f748c0cbb30 2024-04-02T07:55:03.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.317+0000 7f7488a00700 1 -- 172.21.15.89:0/90100620 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747c011990 con 0x7f748c0cbb30 2024-04-02T07:55:03.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.317+0000 7f7488a00700 1 -- 172.21.15.89:0/90100620 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c011d20 con 0x7f748c0cbb30 2024-04-02T07:55:03.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.317+0000 7f7491000700 1 -- 172.21.15.89:0/90100620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f748c0cbb30 msgr2=0x7f748c0cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:03.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.317+0000 7f7491000700 1 --2- 172.21.15.89:0/90100620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f748c0cbb30 0x7f748c0cbf10 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f748c0b1e30 tx=0x7f747c009b20 comp rx=0 tx=0).stop 2024-04-02T07:55:03.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.318+0000 7f7491000700 1 -- 172.21.15.89:0/90100620 shutdown_connections 2024-04-02T07:55:03.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.318+0000 7f7491000700 1 --2- 172.21.15.89:0/90100620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f748c0cbb30 0x7f748c0cbf10 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:03.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.318+0000 7f7491000700 1 --2- 172.21.15.89:0/90100620 >> v2:172.21.15.113:3300/0 conn(0x7f748c0cd020 0x7f748c0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:03.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.318+0000 7f7491000700 1 -- 172.21.15.89:0/90100620 >> 172.21.15.89:0/90100620 conn(0x7f748c0b0d70 msgr2=0x7f748c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:03.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.318+0000 7f7491000700 1 -- 172.21.15.89:0/90100620 shutdown_connections 2024-04-02T07:55:03.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.318+0000 7f7491000700 1 -- 172.21.15.89:0/90100620 wait complete. 2024-04-02T07:55:03.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.319+0000 7f7491000700 1 Processor -- start 2024-04-02T07:55:03.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.320+0000 7f7491000700 1 -- start start 2024-04-02T07:55:03.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.320+0000 7f7491000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f748c0b2070 0x7f748c1503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:03.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7491000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f748c0cbb30 0x7f748c0720c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:03.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7491000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f748c0cd020 0x7f748c072600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:03.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7491000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c0c9610 con 0x7f748c0cbb30 2024-04-02T07:55:03.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7491000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c0713d0 con 0x7f748c0b2070 2024-04-02T07:55:03.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7491000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748c071c30 con 0x7f748c0cd020 2024-04-02T07:55:03.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7489400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f748c0cbb30 0x7f748c0720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:03.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7489400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f748c0cbb30 0x7f748c0720c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38496/0 (socket says 172.21.15.89:38496) 2024-04-02T07:55:03.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7489400700 1 -- 172.21.15.89:0/3208043381 learned_addr learned my addr 172.21.15.89:0/3208043381 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:03.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7489e00700 1 --2- 172.21.15.89:0/3208043381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f748c0b2070 0x7f748c1503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:03.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f748a800700 1 --2- 172.21.15.89:0/3208043381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f748c0cd020 0x7f748c072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:03.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7489400700 1 -- 172.21.15.89:0/3208043381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f748c0cd020 msgr2=0x7f748c072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:03.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7489400700 1 --2- 172.21.15.89:0/3208043381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f748c0cd020 0x7f748c072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:03.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7489400700 1 -- 172.21.15.89:0/3208043381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f748c0b2070 msgr2=0x7f748c1503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:03.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7489400700 1 --2- 172.21.15.89:0/3208043381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f748c0b2070 0x7f748c1503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:03.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7489400700 1 -- 172.21.15.89:0/3208043381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c0094a0 con 0x7f748c0cbb30 2024-04-02T07:55:03.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.321+0000 7f7489400700 1 --2- 172.21.15.89:0/3208043381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f748c0cbb30 0x7f748c0720c0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f747800e810 tx=0x7f747800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:03.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.322+0000 7f7482000700 1 -- 172.21.15.89:0/3208043381 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478013610 con 0x7f748c0cbb30 2024-04-02T07:55:03.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.322+0000 7f7482000700 1 -- 172.21.15.89:0/3208043381 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7478013770 con 0x7f748c0cbb30 2024-04-02T07:55:03.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.322+0000 7f7491000700 1 -- 172.21.15.89:0/3208043381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748c0c9330 con 0x7f748c0cbb30 2024-04-02T07:55:03.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.322+0000 7f7482000700 1 -- 172.21.15.89:0/3208043381 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478013b20 con 0x7f748c0cbb30 2024-04-02T07:55:03.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.322+0000 7f7491000700 1 -- 172.21.15.89:0/3208043381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748c0718c0 con 0x7f748c0cbb30 2024-04-02T07:55:03.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.323+0000 7f7491000700 1 -- 172.21.15.89:0/3208043381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748c071dc0 con 0x7f748c0cbb30 2024-04-02T07:55:03.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.324+0000 7f7482000700 1 -- 172.21.15.89:0/3208043381 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 72) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7478013c80 con 0x7f748c0cbb30 2024-04-02T07:55:03.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.325+0000 7f7482000700 1 --2- 172.21.15.89:0/3208043381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f746c06af50 0x7f746c06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:03.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.325+0000 7f7482000700 1 -- 172.21.15.89:0/3208043381 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f7478089890 con 0x7f748c0cbb30 2024-04-02T07:55:03.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.325+0000 7f7489e00700 1 --2- 172.21.15.89:0/3208043381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f746c06af50 0x7f746c06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:03.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.326+0000 7f7489e00700 1 --2- 172.21.15.89:0/3208043381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f746c06af50 0x7f746c06d410 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f747c0090b0 tx=0x7f747c01c040 comp rx=0 tx=0).ready entity=mgr.8103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:03.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.329+0000 7f7482000700 1 -- 172.21.15.89:0/3208043381 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f74780542b0 con 0x7f748c0cbb30 2024-04-02T07:55:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:03.587+0000 7f7491000700 1 -- 172.21.15.89:0/3208043381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f748c066ff0 con 0x7f748c0cbb30 2024-04-02T07:55:04.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.073+0000 7f7482000700 1 -- 172.21.15.89:0/3208043381 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f7478054680 con 0x7f748c0cbb30 2024-04-02T07:55:04.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.074+0000 7f7482000700 1 -- 172.21.15.89:0/3208043381 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v73) v1 ==== 93+0+0 (secure 0 0 0) 0x7f74780723f0 con 0x7f748c0cbb30 2024-04-02T07:55:04.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.077+0000 7f7491000700 1 -- 172.21.15.89:0/3208043381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f746c06af50 msgr2=0x7f746c06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.077+0000 7f7491000700 1 --2- 172.21.15.89:0/3208043381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f746c06af50 0x7f746c06d410 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f747c0090b0 tx=0x7f747c01c040 comp rx=0 tx=0).stop 2024-04-02T07:55:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.077+0000 7f7491000700 1 -- 172.21.15.89:0/3208043381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f748c0cbb30 msgr2=0x7f748c0720c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.077+0000 7f7491000700 1 --2- 172.21.15.89:0/3208043381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f748c0cbb30 0x7f748c0720c0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f747800e810 tx=0x7f747800ebd0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.077+0000 7f7491000700 1 -- 172.21.15.89:0/3208043381 shutdown_connections 2024-04-02T07:55:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.077+0000 7f7491000700 1 --2- 172.21.15.89:0/3208043381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f746c06af50 0x7f746c06d410 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.077+0000 7f7491000700 1 --2- 172.21.15.89:0/3208043381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f748c0b2070 0x7f748c1503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.077+0000 7f7491000700 1 --2- 172.21.15.89:0/3208043381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f748c0cd020 0x7f748c072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.077+0000 7f7491000700 1 --2- 172.21.15.89:0/3208043381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f748c0cbb30 0x7f748c0720c0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.077+0000 7f7491000700 1 -- 172.21.15.89:0/3208043381 >> 172.21.15.89:0/3208043381 conn(0x7f748c0b0d70 msgr2=0x7f748c0ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.078+0000 7f7491000700 1 -- 172.21.15.89:0/3208043381 shutdown_connections 2024-04-02T07:55:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.078+0000 7f7491000700 1 -- 172.21.15.89:0/3208043381 wait complete. 2024-04-02T07:55:04.098 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:55:04.098 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:55:04.197 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:55:04.281 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:55:04.282 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:55:04.282 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:04.297 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:04.295+0000 7f9aa6846240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:55:04.305 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:04.303+0000 7fd8340a5240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:55:04.330 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:55:04.330 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:04.415 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:55:04.480 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:04.479+0000 7f9aa6846240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:55:04.495 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:04.494+0000 7fd8340a5240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:55:04.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.669+0000 7fd132000700 1 Processor -- start 2024-04-02T07:55:04.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.669+0000 7fd132000700 1 -- start start 2024-04-02T07:55:04.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.670+0000 7fd132000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd12c15a460 0x7fd12c0ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:04.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.670+0000 7fd132000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd12c15b150 0x7fd12c15b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:04.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.671+0000 7fd132000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c15a840 con 0x7fd12c06e3b0 2024-04-02T07:55:04.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.671+0000 7fd132000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd12c0cbf50 con 0x7fd12c15a460 2024-04-02T07:55:04.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.671+0000 7fd132000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c0cc230 con 0x7fd12c15b150 2024-04-02T07:55:04.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.671+0000 7fd12be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd12c06e3b0 0x7fd12c15bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35674/0 (socket says 172.21.15.89:35674) 2024-04-02T07:55:04.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.671+0000 7fd12be00700 1 -- 172.21.15.89:0/580281759 learned_addr learned my addr 172.21.15.89:0/580281759 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:04.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.671+0000 7fd12b400700 1 --2- 172.21.15.89:0/580281759 >> v2:172.21.15.89:3300/0 conn(0x7fd12c15a460 0x7fd12c0ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:04.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.671+0000 7fd12aa00700 1 --2- 172.21.15.89:0/580281759 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd12c15b150 0x7fd12c15b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:04.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.671+0000 7fd12b400700 1 -- 172.21.15.89:0/580281759 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd12c15b150 msgr2=0x7fd12c15b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:04.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.671+0000 7fd12b400700 1 --2- 172.21.15.89:0/580281759 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd12c15b150 0x7fd12c15b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.671+0000 7fd12a000700 1 -- 172.21.15.89:0/580281759 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214091370 0 0) 0x7fd12c15a840 con 0x7fd12c06e3b0 2024-04-02T07:55:04.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.671+0000 7fd12b400700 1 -- 172.21.15.89:0/580281759 >> v1:172.21.15.89:6789/0 conn(0x7fd12c06e3b0 legacy=0x7fd12c15bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:04.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.671+0000 7fd12b400700 1 -- 172.21.15.89:0/580281759 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c0cc510 con 0x7fd12c15a460 2024-04-02T07:55:04.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.672+0000 7fd12b400700 1 --2- 172.21.15.89:0/580281759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12c15a460 0x7fd12c0ccff0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fd11c009840 tx=0x7fd11c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceee8b98e26709fa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:04.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.672+0000 7fd12a000700 1 -- 172.21.15.89:0/580281759 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c011830 con 0x7fd12c15a460 2024-04-02T07:55:04.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.672+0000 7fd12a000700 1 -- 172.21.15.89:0/580281759 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11c011990 con 0x7fd12c15a460 2024-04-02T07:55:04.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.672+0000 7fd12a000700 1 -- 172.21.15.89:0/580281759 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c011d20 con 0x7fd12c15a460 2024-04-02T07:55:04.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.673+0000 7fd132000700 1 -- 172.21.15.89:0/580281759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12c15a460 msgr2=0x7fd12c0ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:04.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.673+0000 7fd132000700 1 --2- 172.21.15.89:0/580281759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12c15a460 0x7fd12c0ccff0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fd11c009840 tx=0x7fd11c009b50 comp rx=0 tx=0).stop 2024-04-02T07:55:04.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.673+0000 7fd132000700 1 -- 172.21.15.89:0/580281759 shutdown_connections 2024-04-02T07:55:04.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.673+0000 7fd132000700 1 --2- 172.21.15.89:0/580281759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12c15a460 0x7fd12c0ccff0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.673+0000 7fd132000700 1 --2- 172.21.15.89:0/580281759 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd12c15b150 0x7fd12c15b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.673+0000 7fd132000700 1 -- 172.21.15.89:0/580281759 >> 172.21.15.89:0/580281759 conn(0x7fd12c072170 msgr2=0x7fd12c06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:04.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.673+0000 7fd132000700 1 -- 172.21.15.89:0/580281759 shutdown_connections 2024-04-02T07:55:04.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.673+0000 7fd132000700 1 -- 172.21.15.89:0/580281759 wait complete. 2024-04-02T07:55:04.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.675+0000 7fd132000700 1 Processor -- start 2024-04-02T07:55:04.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.675+0000 7fd132000700 1 -- start start 2024-04-02T07:55:04.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd132000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12c06e3b0 0x7fd12c13e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:04.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd132000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd12c15a460 0x7fd12c140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:04.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd132000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd12c15b150 0x7fd12c1411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:04.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd132000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c1f3060 con 0x7fd12c06e3b0 2024-04-02T07:55:04.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd132000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c0cb2c0 con 0x7fd12c15a460 2024-04-02T07:55:04.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd132000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c0ca6b0 con 0x7fd12c15b150 2024-04-02T07:55:04.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd12b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12c06e3b0 0x7fd12c13e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:04.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd12b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12c06e3b0 0x7fd12c13e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38528/0 (socket says 172.21.15.89:38528) 2024-04-02T07:55:04.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd12b400700 1 -- 172.21.15.89:0/4213115919 learned_addr learned my addr 172.21.15.89:0/4213115919 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:04.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd12be00700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd12c15b150 0x7fd12c1411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd12aa00700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd12c15a460 0x7fd12c140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd12b400700 1 -- 172.21.15.89:0/4213115919 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd12c15b150 msgr2=0x7fd12c1411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd12b400700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd12c15b150 0x7fd12c1411e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd12b400700 1 -- 172.21.15.89:0/4213115919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd12c15a460 msgr2=0x7fd12c140ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd12b400700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd12c15a460 0x7fd12c140ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd12b400700 1 -- 172.21.15.89:0/4213115919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c0094a0 con 0x7fd12c06e3b0 2024-04-02T07:55:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.676+0000 7fd12aa00700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd12c15a460 0x7fd12c140ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.677+0000 7fd12b400700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12c06e3b0 0x7fd12c13e450 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fd11c009fd0 tx=0x7fd11c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.677+0000 7fd123400700 1 -- 172.21.15.89:0/4213115919 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c0036a0 con 0x7fd12c06e3b0 2024-04-02T07:55:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.677+0000 7fd123400700 1 -- 172.21.15.89:0/4213115919 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11c01f070 con 0x7fd12c06e3b0 2024-04-02T07:55:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.677+0000 7fd123400700 1 -- 172.21.15.89:0/4213115919 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c003a60 con 0x7fd12c06e3b0 2024-04-02T07:55:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.677+0000 7fd132000700 1 -- 172.21.15.89:0/4213115919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12c0ca990 con 0x7fd12c06e3b0 2024-04-02T07:55:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.677+0000 7fd132000700 1 -- 172.21.15.89:0/4213115919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12c0caf50 con 0x7fd12c06e3b0 2024-04-02T07:55:04.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.678+0000 7fd132000700 1 -- 172.21.15.89:0/4213115919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12c066e90 con 0x7fd12c06e3b0 2024-04-02T07:55:04.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.679+0000 7fd123400700 1 -- 172.21.15.89:0/4213115919 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fd11c003bc0 con 0x7fd12c06e3b0 2024-04-02T07:55:04.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.680+0000 7fd123400700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd10c06aed0 0x7fd10c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:04.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.680+0000 7fd12aa00700 1 -- 172.21.15.89:0/4213115919 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd10c06aed0 msgr2=0x7fd10c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:04.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.680+0000 7fd123400700 1 -- 172.21.15.89:0/4213115919 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd11c021070 con 0x7fd12c06e3b0 2024-04-02T07:55:04.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.680+0000 7fd12aa00700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd10c06aed0 0x7fd10c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.684+0000 7fd123400700 1 -- 172.21.15.89:0/4213115919 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd11c055f30 con 0x7fd12c06e3b0 2024-04-02T07:55:04.799 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:04.798+0000 7f9aa6846240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:55:04.834 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:04.832+0000 7fd8340a5240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:55:04.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.881+0000 7fd12aa00700 1 -- 172.21.15.89:0/4213115919 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd10c06aed0 msgr2=0x7fd10c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:04.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.881+0000 7fd12aa00700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd10c06aed0 0x7fd10c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:04.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.896+0000 7fd132000700 1 -- 172.21.15.89:0/4213115919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd12c067720 con 0x7fd12c06e3b0 2024-04-02T07:55:04.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.897+0000 7fd123400700 1 -- 172.21.15.89:0/4213115919 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+17063 (secure 0 0 0) 0x7fd11c059550 con 0x7fd12c06e3b0 2024-04-02T07:55:04.903 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:04.903 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":126,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:54:46.836305+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:54:28.130987+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-02T07:54:29.515956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/49649361":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/51116008":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/236471161":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/709589358":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/1096941421":"2024-04-03T07:54:46.836267+0000","172.21.15.89:6833/43357":"2024-04-03T07:54:46.836267+0000","172.21.15.89:6832/43357":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/3407101260":"2024-04-03T07:54:46.836267+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:55:04.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.900+0000 7fd132000700 1 -- 172.21.15.89:0/4213115919 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd10c06aed0 msgr2=0x7fd10c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:04.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.900+0000 7fd132000700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd10c06aed0 0x7fd10c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.900+0000 7fd132000700 1 -- 172.21.15.89:0/4213115919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12c06e3b0 msgr2=0x7fd12c13e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:04.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.900+0000 7fd132000700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12c06e3b0 0x7fd12c13e450 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fd11c009fd0 tx=0x7fd11c004750 comp rx=0 tx=0).stop 2024-04-02T07:55:04.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.900+0000 7fd132000700 1 -- 172.21.15.89:0/4213115919 shutdown_connections 2024-04-02T07:55:04.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.900+0000 7fd132000700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd10c06aed0 0x7fd10c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.900+0000 7fd132000700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd12c15b150 0x7fd12c1411e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.900+0000 7fd132000700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd12c06e3b0 0x7fd12c13e450 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.900+0000 7fd132000700 1 --2- 172.21.15.89:0/4213115919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd12c15a460 0x7fd12c140ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:04.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.900+0000 7fd132000700 1 -- 172.21.15.89:0/4213115919 >> 172.21.15.89:0/4213115919 conn(0x7fd12c072170 msgr2=0x7fd12c0b35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:04.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.900+0000 7fd132000700 1 -- 172.21.15.89:0/4213115919 shutdown_connections 2024-04-02T07:55:04.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:04.900+0000 7fd132000700 1 -- 172.21.15.89:0/4213115919 wait complete. 2024-04-02T07:55:04.921 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:54:28.130987+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:54:29.515956+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:55:04.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:55:05.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.158+0000 7f2a45600700 1 Processor -- start 2024-04-02T07:55:05.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.159+0000 7f2a45600700 1 -- start start 2024-04-02T07:55:05.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.160+0000 7f2a45600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a40160d70 0x7f2a400b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:05.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.160+0000 7f2a45600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2a4015ef70 0x7f2a4015e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:05.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.160+0000 7f2a45600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a4006a790 0x7f2a4006ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:05.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.160+0000 7f2a45600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a4006b180 con 0x7f2a40160d70 2024-04-02T07:55:05.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.160+0000 7f2a45600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a400b4cd0 con 0x7f2a4015ef70 2024-04-02T07:55:05.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.160+0000 7f2a45600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a40161150 con 0x7f2a4006a790 2024-04-02T07:55:05.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.160+0000 7f2a3ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a40160d70 0x7f2a400b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:05.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.160+0000 7f2a3ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a40160d70 0x7f2a400b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38532/0 (socket says 172.21.15.89:38532) 2024-04-02T07:55:05.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.160+0000 7f2a3ea00700 1 -- 172.21.15.89:0/3490422141 learned_addr learned my addr 172.21.15.89:0/3490422141 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:05.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.160+0000 7f2a3e000700 1 --2- 172.21.15.89:0/3490422141 >> v2:172.21.15.113:3300/0 conn(0x7f2a4015ef70 0x7f2a4015e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:05.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.160+0000 7f2a3f400700 1 --2- 172.21.15.89:0/3490422141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a4006a790 0x7f2a4006ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:05.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.160+0000 7f2a3ea00700 1 -- 172.21.15.89:0/3490422141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a4006a790 msgr2=0x7f2a4006ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:05.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.160+0000 7f2a3ea00700 1 --2- 172.21.15.89:0/3490422141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a4006a790 0x7f2a4006ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.161+0000 7f2a3ea00700 1 -- 172.21.15.89:0/3490422141 >> v2:172.21.15.113:3300/0 conn(0x7f2a4015ef70 msgr2=0x7f2a4015e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:05.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.161+0000 7f2a3ea00700 1 --2- 172.21.15.89:0/3490422141 >> v2:172.21.15.113:3300/0 conn(0x7f2a4015ef70 0x7f2a4015e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.161+0000 7f2a3ea00700 1 -- 172.21.15.89:0/3490422141 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4006dba0 con 0x7f2a40160d70 2024-04-02T07:55:05.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.161+0000 7f2a3f400700 1 --2- 172.21.15.89:0/3490422141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a4006a790 0x7f2a4006ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:05.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.161+0000 7f2a3ea00700 1 --2- 172.21.15.89:0/3490422141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a40160d70 0x7f2a400b3200 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f2a400d1370 tx=0x7f2a30009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eefbe941fd58300 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:05.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.161+0000 7f2a3d600700 1 -- 172.21.15.89:0/3490422141 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a30011830 con 0x7f2a40160d70 2024-04-02T07:55:05.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.161+0000 7f2a3d600700 1 -- 172.21.15.89:0/3490422141 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a30011990 con 0x7f2a40160d70 2024-04-02T07:55:05.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.161+0000 7f2a3d600700 1 -- 172.21.15.89:0/3490422141 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a30011d20 con 0x7f2a40160d70 2024-04-02T07:55:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.162+0000 7f2a45600700 1 -- 172.21.15.89:0/3490422141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a40160d70 msgr2=0x7f2a400b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.162+0000 7f2a45600700 1 --2- 172.21.15.89:0/3490422141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a40160d70 0x7f2a400b3200 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f2a400d1370 tx=0x7f2a30009bd0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.162+0000 7f2a45600700 1 -- 172.21.15.89:0/3490422141 shutdown_connections 2024-04-02T07:55:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.162+0000 7f2a45600700 1 --2- 172.21.15.89:0/3490422141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a40160d70 0x7f2a400b3200 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.162+0000 7f2a45600700 1 --2- 172.21.15.89:0/3490422141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a4006a790 0x7f2a4006ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.162+0000 7f2a45600700 1 --2- 172.21.15.89:0/3490422141 >> v2:172.21.15.113:3300/0 conn(0x7f2a4015ef70 0x7f2a4015e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.162+0000 7f2a45600700 1 -- 172.21.15.89:0/3490422141 >> 172.21.15.89:0/3490422141 conn(0x7f2a400b5720 msgr2=0x7f2a400b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:05.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.162+0000 7f2a45600700 1 -- 172.21.15.89:0/3490422141 shutdown_connections 2024-04-02T07:55:05.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.162+0000 7f2a45600700 1 -- 172.21.15.89:0/3490422141 wait complete. 2024-04-02T07:55:05.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.164+0000 7f2a45600700 1 Processor -- start 2024-04-02T07:55:05.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.164+0000 7f2a45600700 1 -- start start 2024-04-02T07:55:05.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.165+0000 7f2a45600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4006a790 0x7f2a40144dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:05.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.165+0000 7f2a45600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a4015ef70 0x7f2a4009ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:05.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.165+0000 7f2a45600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a40160d70 0x7f2a400aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:05.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.165+0000 7f2a45600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a40145300 con 0x7f2a4006a790 2024-04-02T07:55:05.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.165+0000 7f2a45600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a400ad5f0 con 0x7f2a4015ef70 2024-04-02T07:55:05.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.165+0000 7f2a45600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4009bd80 con 0x7f2a40160d70 2024-04-02T07:55:05.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.165+0000 7f2a3ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4006a790 0x7f2a40144dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.165+0000 7f2a3ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4006a790 0x7f2a40144dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38548/0 (socket says 172.21.15.89:38548) 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.165+0000 7f2a3ea00700 1 -- 172.21.15.89:0/1913256479 learned_addr learned my addr 172.21.15.89:0/1913256479 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.165+0000 7f2a3f400700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a40160d70 0x7f2a400aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.165+0000 7f2a3e000700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a4015ef70 0x7f2a4009ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.166+0000 7f2a3e000700 1 -- 172.21.15.89:0/1913256479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a40160d70 msgr2=0x7f2a400aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.166+0000 7f2a3e000700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a40160d70 0x7f2a400aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.166+0000 7f2a3e000700 1 -- 172.21.15.89:0/1913256479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4006a790 msgr2=0x7f2a40144dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.166+0000 7f2a3e000700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4006a790 0x7f2a40144dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.166+0000 7f2a3e000700 1 -- 172.21.15.89:0/1913256479 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a30009550 con 0x7f2a4015ef70 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.166+0000 7f2a3ea00700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4006a790 0x7f2a40144dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.166+0000 7f2a3f400700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a40160d70 0x7f2a400aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.166+0000 7f2a3e000700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a4015ef70 0x7f2a4009ad00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c00e900 tx=0x7f2a2c00ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.166+0000 7f2a36a00700 1 -- 172.21.15.89:0/1913256479 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c013680 con 0x7f2a4015ef70 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.166+0000 7f2a36a00700 1 -- 172.21.15.89:0/1913256479 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2c0137e0 con 0x7f2a4015ef70 2024-04-02T07:55:05.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.166+0000 7f2a45600700 1 -- 172.21.15.89:0/1913256479 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a400b0f30 con 0x7f2a4015ef70 2024-04-02T07:55:05.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.166+0000 7f2a36a00700 1 -- 172.21.15.89:0/1913256479 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c013b50 con 0x7f2a4015ef70 2024-04-02T07:55:05.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.166+0000 7f2a45600700 1 -- 172.21.15.89:0/1913256479 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a4009baa0 con 0x7f2a4015ef70 2024-04-02T07:55:05.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.168+0000 7f2a45600700 1 -- 172.21.15.89:0/1913256479 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a40067720 con 0x7f2a4015ef70 2024-04-02T07:55:05.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.168+0000 7f2a36a00700 1 -- 172.21.15.89:0/1913256479 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f2a2c013cb0 con 0x7f2a4015ef70 2024-04-02T07:55:05.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.169+0000 7f2a36a00700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2a2006aed0 0x7f2a2006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:05.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.169+0000 7f2a36a00700 1 -- 172.21.15.89:0/1913256479 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f2a2c089a40 con 0x7f2a4015ef70 2024-04-02T07:55:05.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.169+0000 7f2a3ea00700 1 -- 172.21.15.89:0/1913256479 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2a2006aed0 msgr2=0x7f2a2006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:05.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.169+0000 7f2a3ea00700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2a2006aed0 0x7f2a2006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:05.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.175+0000 7f2a36a00700 1 -- 172.21.15.89:0/1913256479 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2a2c054560 con 0x7f2a4015ef70 2024-04-02T07:55:05.325 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:05.324+0000 7f9aa6846240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:55:05.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.370+0000 7f2a3ea00700 1 -- 172.21.15.89:0/1913256479 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2a2006aed0 msgr2=0x7f2a2006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:05.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.370+0000 7f2a3ea00700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2a2006aed0 0x7f2a2006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:05.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.382+0000 7f2a45600700 1 -- 172.21.15.89:0/1913256479 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2a400ad730 con 0x7f2a4015ef70 2024-04-02T07:55:05.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.383+0000 7f2a36a00700 1 -- 172.21.15.89:0/1913256479 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2a2c0170a0 con 0x7f2a4015ef70 2024-04-02T07:55:05.384 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:55:05.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.386+0000 7f2a45600700 1 -- 172.21.15.89:0/1913256479 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2a2006aed0 msgr2=0x7f2a2006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:05.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.386+0000 7f2a45600700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2a2006aed0 0x7f2a2006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.386+0000 7f2a45600700 1 -- 172.21.15.89:0/1913256479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a4015ef70 msgr2=0x7f2a4009ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:05.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.386+0000 7f2a45600700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a4015ef70 0x7f2a4009ad00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c00e900 tx=0x7f2a2c00ec10 comp rx=0 tx=0).stop 2024-04-02T07:55:05.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.386+0000 7f2a45600700 1 -- 172.21.15.89:0/1913256479 shutdown_connections 2024-04-02T07:55:05.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.386+0000 7f2a45600700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2a2006aed0 0x7f2a2006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.386+0000 7f2a45600700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a40160d70 0x7f2a400aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.386+0000 7f2a45600700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a4006a790 0x7f2a40144dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.386+0000 7f2a45600700 1 --2- 172.21.15.89:0/1913256479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a4015ef70 0x7f2a4009ad00 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.386+0000 7f2a45600700 1 -- 172.21.15.89:0/1913256479 >> 172.21.15.89:0/1913256479 conn(0x7f2a400b5720 msgr2=0x7f2a4006df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:05.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.386+0000 7f2a45600700 1 -- 172.21.15.89:0/1913256479 shutdown_connections 2024-04-02T07:55:05.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.386+0000 7f2a45600700 1 -- 172.21.15.89:0/1913256479 wait complete. 2024-04-02T07:55:05.393 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:05.393+0000 7fd8340a5240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:55:05.405 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:55:05.578 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:05.577+0000 7f9aa6846240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:55:05.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.603+0000 7f7787600700 1 Processor -- start 2024-04-02T07:55:05.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.603+0000 7f7787600700 1 -- start start 2024-04-02T07:55:05.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.604+0000 7f7787600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f778015dad0 0x7f77800ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:05.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.604+0000 7f7787600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f77800d1630 0x7f77800d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:05.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.605+0000 7f7787600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778015f820 con 0x7f77800d1fd0 2024-04-02T07:55:05.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.605+0000 7f7787600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77800cfcc0 con 0x7f77800d1630 2024-04-02T07:55:05.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.605+0000 7f7787600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77800cd9f0 con 0x7f778015dad0 2024-04-02T07:55:05.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.605+0000 7f777fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f77800d1630 0x7f77800d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:05.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.605+0000 7f7785600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f77800d1fd0 0x7f778015deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35690/0 (socket says 172.21.15.89:35690) 2024-04-02T07:55:05.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.605+0000 7f7785600700 1 -- 172.21.15.89:0/3708331792 learned_addr learned my addr 172.21.15.89:0/3708331792 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:05.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.605+0000 7f7784c00700 1 --2- 172.21.15.89:0/3708331792 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f778015dad0 0x7f77800ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:05.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.605+0000 7f777f400700 1 -- 172.21.15.89:0/3708331792 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254640743 0 0) 0x7f778015f820 con 0x7f77800d1fd0 2024-04-02T07:55:05.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.605+0000 7f777f400700 1 -- 172.21.15.89:0/3708331792 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77680035b0 con 0x7f77800d1fd0 2024-04-02T07:55:05.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.605+0000 7f777fe00700 1 -- 172.21.15.89:0/3708331792 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f778015dad0 msgr2=0x7f77800ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:05.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.605+0000 7f777fe00700 1 --2- 172.21.15.89:0/3708331792 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f778015dad0 0x7f77800ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.605+0000 7f777fe00700 1 -- 172.21.15.89:0/3708331792 >> v1:172.21.15.89:6789/0 conn(0x7f77800d1fd0 legacy=0x7f778015deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:05.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.605+0000 7f777fe00700 1 -- 172.21.15.89:0/3708331792 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77800d2460 con 0x7f77800d1630 2024-04-02T07:55:05.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.606+0000 7f777fe00700 1 --2- 172.21.15.89:0/3708331792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77800d1630 0x7f77800d1a90 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f776c009840 tx=0x7f776c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee88789bc92e77e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:05.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.606+0000 7f777f400700 1 -- 172.21.15.89:0/3708331792 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c011830 con 0x7f77800d1630 2024-04-02T07:55:05.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.606+0000 7f777f400700 1 -- 172.21.15.89:0/3708331792 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776c011990 con 0x7f77800d1630 2024-04-02T07:55:05.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.606+0000 7f777f400700 1 -- 172.21.15.89:0/3708331792 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c011d00 con 0x7f77800d1630 2024-04-02T07:55:05.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.607+0000 7f7787600700 1 -- 172.21.15.89:0/3708331792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77800d1630 msgr2=0x7f77800d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:05.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.607+0000 7f7787600700 1 --2- 172.21.15.89:0/3708331792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77800d1630 0x7f77800d1a90 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f776c009840 tx=0x7f776c009b50 comp rx=0 tx=0).stop 2024-04-02T07:55:05.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.607+0000 7f7787600700 1 -- 172.21.15.89:0/3708331792 shutdown_connections 2024-04-02T07:55:05.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.607+0000 7f7787600700 1 --2- 172.21.15.89:0/3708331792 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f778015dad0 0x7f77800ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.607+0000 7f7787600700 1 --2- 172.21.15.89:0/3708331792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77800d1630 0x7f77800d1a90 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.607+0000 7f7787600700 1 -- 172.21.15.89:0/3708331792 >> 172.21.15.89:0/3708331792 conn(0x7f778008aff0 msgr2=0x7f77800d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:05.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.607+0000 7f7787600700 1 -- 172.21.15.89:0/3708331792 shutdown_connections 2024-04-02T07:55:05.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.607+0000 7f7787600700 1 -- 172.21.15.89:0/3708331792 wait complete. 2024-04-02T07:55:05.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.609+0000 7f7787600700 1 Processor -- start 2024-04-02T07:55:05.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.609+0000 7f7787600700 1 -- start start 2024-04-02T07:55:05.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.610+0000 7f7787600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77800d1630 0x7f77800cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:05.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.610+0000 7f7787600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77800d1fd0 0x7f7780147740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:05.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.610+0000 7f7787600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778015dad0 0x7f77800c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:05.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.610+0000 7f7787600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77800cd060 con 0x7f77800d1fd0 2024-04-02T07:55:05.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.610+0000 7f7787600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77800cc590 con 0x7f778015dad0 2024-04-02T07:55:05.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.610+0000 7f7787600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77800cc7c0 con 0x7f77800d1630 2024-04-02T07:55:05.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.610+0000 7f777fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77800d1fd0 0x7f7780147740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:05.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.610+0000 7f7784c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77800d1630 0x7f77800cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:05.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.610+0000 7f777fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77800d1fd0 0x7f7780147740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38566/0 (socket says 172.21.15.89:38566) 2024-04-02T07:55:05.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.610+0000 7f777fe00700 1 -- 172.21.15.89:0/501284911 learned_addr learned my addr 172.21.15.89:0/501284911 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:05.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.610+0000 7f7785600700 1 --2- 172.21.15.89:0/501284911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778015dad0 0x7f77800c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:05.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.611+0000 7f777fe00700 1 -- 172.21.15.89:0/501284911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77800d1630 msgr2=0x7f77800cc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:05.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.611+0000 7f777fe00700 1 --2- 172.21.15.89:0/501284911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77800d1630 0x7f77800cc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.611+0000 7f777fe00700 1 -- 172.21.15.89:0/501284911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778015dad0 msgr2=0x7f77800c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:05.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.611+0000 7f777fe00700 1 --2- 172.21.15.89:0/501284911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778015dad0 0x7f77800c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.611+0000 7f777fe00700 1 -- 172.21.15.89:0/501284911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776c0094a0 con 0x7f77800d1fd0 2024-04-02T07:55:05.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.611+0000 7f7784c00700 1 --2- 172.21.15.89:0/501284911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77800d1630 0x7f77800cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:55:05.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.611+0000 7f777fe00700 1 --2- 172.21.15.89:0/501284911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77800d1fd0 0x7f7780147740 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f776c0090b0 tx=0x7f776c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:05.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.611+0000 7f777cc00700 1 -- 172.21.15.89:0/501284911 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c004550 con 0x7f77800d1fd0 2024-04-02T07:55:05.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.611+0000 7f777cc00700 1 -- 172.21.15.89:0/501284911 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776c01f070 con 0x7f77800d1fd0 2024-04-02T07:55:05.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.611+0000 7f7787600700 1 -- 172.21.15.89:0/501284911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77800ccaa0 con 0x7f77800d1fd0 2024-04-02T07:55:05.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.611+0000 7f777cc00700 1 -- 172.21.15.89:0/501284911 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c0038c0 con 0x7f77800d1fd0 2024-04-02T07:55:05.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.611+0000 7f7787600700 1 -- 172.21.15.89:0/501284911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77800b6e50 con 0x7f77800d1fd0 2024-04-02T07:55:05.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.613+0000 7f7787600700 1 -- 172.21.15.89:0/501284911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77800b6cb0 con 0x7f77800d1fd0 2024-04-02T07:55:05.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.613+0000 7f777cc00700 1 -- 172.21.15.89:0/501284911 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f776c003a20 con 0x7f77800d1fd0 2024-04-02T07:55:05.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.614+0000 7f777cc00700 1 --2- 172.21.15.89:0/501284911 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f776006aed0 0x7f776006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:05.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.614+0000 7f777cc00700 1 -- 172.21.15.89:0/501284911 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f776c021070 con 0x7f77800d1fd0 2024-04-02T07:55:05.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.614+0000 7f7784c00700 1 -- 172.21.15.89:0/501284911 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f776006aed0 msgr2=0x7f776006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:05.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.614+0000 7f7784c00700 1 --2- 172.21.15.89:0/501284911 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f776006aed0 0x7f776006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:05.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.619+0000 7f777cc00700 1 -- 172.21.15.89:0/501284911 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f776c0be580 con 0x7f77800d1fd0 2024-04-02T07:55:05.668 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:05.667+0000 7fd8340a5240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:55:05.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.815+0000 7f7784c00700 1 -- 172.21.15.89:0/501284911 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f776006aed0 msgr2=0x7f776006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:05.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.815+0000 7f7784c00700 1 --2- 172.21.15.89:0/501284911 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f776006aed0 0x7f776006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:05.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.853+0000 7f7787600700 1 -- 172.21.15.89:0/501284911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f77800b7070 con 0x7f77800d1fd0 2024-04-02T07:55:05.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.854+0000 7f777cc00700 1 -- 172.21.15.89:0/501284911 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v126) v1 ==== 104+0+11 (secure 0 0 0) 0x7f77800b6cb0 con 0x7f77800d1fd0 2024-04-02T07:55:05.854 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:55:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.856+0000 7f7787600700 1 -- 172.21.15.89:0/501284911 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f776006aed0 msgr2=0x7f776006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.856+0000 7f7787600700 1 --2- 172.21.15.89:0/501284911 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f776006aed0 0x7f776006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.856+0000 7f7787600700 1 -- 172.21.15.89:0/501284911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77800d1fd0 msgr2=0x7f7780147740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.856+0000 7f7787600700 1 --2- 172.21.15.89:0/501284911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77800d1fd0 0x7f7780147740 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f776c0090b0 tx=0x7f776c004750 comp rx=0 tx=0).stop 2024-04-02T07:55:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.856+0000 7f7787600700 1 -- 172.21.15.89:0/501284911 shutdown_connections 2024-04-02T07:55:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.856+0000 7f7787600700 1 --2- 172.21.15.89:0/501284911 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f776006aed0 0x7f776006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.856+0000 7f7787600700 1 --2- 172.21.15.89:0/501284911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778015dad0 0x7f77800c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.856+0000 7f7787600700 1 --2- 172.21.15.89:0/501284911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77800d1630 0x7f77800cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.856+0000 7f7787600700 1 --2- 172.21.15.89:0/501284911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77800d1fd0 0x7f7780147740 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.856+0000 7f7787600700 1 -- 172.21.15.89:0/501284911 >> 172.21.15.89:0/501284911 conn(0x7f778008aff0 msgr2=0x7f778015b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:05.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.856+0000 7f7787600700 1 -- 172.21.15.89:0/501284911 shutdown_connections 2024-04-02T07:55:05.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:05.856+0000 7f7787600700 1 -- 172.21.15.89:0/501284911 wait complete. 2024-04-02T07:55:05.876 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:55:06.034 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:06.033+0000 7f9aa6846240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:55:06.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.075+0000 7f0040000700 1 Processor -- start 2024-04-02T07:55:06.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.075+0000 7f0040000700 1 -- start start 2024-04-02T07:55:06.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.076+0000 7f0040000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f003815d9d0 0x7f00380d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:06.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.076+0000 7f0040000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f00380d16a0 0x7f00380d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:06.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.076+0000 7f0040000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0038161420 con 0x7f00380d2370 2024-04-02T07:55:06.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.076+0000 7f0040000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00380d27e0 con 0x7f00380d16a0 2024-04-02T07:55:06.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.076+0000 7f0040000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0038161700 con 0x7f003815d9d0 2024-04-02T07:55:06.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.077+0000 7f003e000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f00380d2370 0x7f003815ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35698/0 (socket says 172.21.15.89:35698) 2024-04-02T07:55:06.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.077+0000 7f003e000700 1 -- 172.21.15.89:0/2734589762 learned_addr learned my addr 172.21.15.89:0/2734589762 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:06.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.077+0000 7f003cc00700 1 --2- 172.21.15.89:0/2734589762 >> v2:172.21.15.89:3300/0 conn(0x7f00380d16a0 0x7f00380d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:06.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.077+0000 7f003d600700 1 --2- 172.21.15.89:0/2734589762 >> v2:172.21.15.113:3300/0 conn(0x7f003815d9d0 0x7f00380d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:06.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.077+0000 7f003cc00700 1 -- 172.21.15.89:0/2734589762 >> v2:172.21.15.113:3300/0 conn(0x7f003815d9d0 msgr2=0x7f00380d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:06.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.077+0000 7f003cc00700 1 --2- 172.21.15.89:0/2734589762 >> v2:172.21.15.113:3300/0 conn(0x7f003815d9d0 0x7f00380d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.077+0000 7f003cc00700 1 -- 172.21.15.89:0/2734589762 >> v1:172.21.15.89:6789/0 conn(0x7f00380d2370 legacy=0x7f003815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.077+0000 7f003cc00700 1 -- 172.21.15.89:0/2734589762 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00381619e0 con 0x7f00380d16a0 2024-04-02T07:55:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.077+0000 7f0037e00700 1 -- 172.21.15.89:0/2734589762 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981406287 0 0) 0x7f0038161420 con 0x7f00380d2370 2024-04-02T07:55:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.078+0000 7f003cc00700 1 --2- 172.21.15.89:0/2734589762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00380d16a0 0x7f00380d1e30 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f00380d0ff0 tx=0x7f0024009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527260375c4d91b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.078+0000 7f0037e00700 1 -- 172.21.15.89:0/2734589762 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024005e30 con 0x7f00380d16a0 2024-04-02T07:55:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.078+0000 7f0037e00700 1 -- 172.21.15.89:0/2734589762 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0024006030 con 0x7f00380d16a0 2024-04-02T07:55:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.078+0000 7f0037e00700 1 -- 172.21.15.89:0/2734589762 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00240063c0 con 0x7f00380d16a0 2024-04-02T07:55:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.078+0000 7f0040000700 1 -- 172.21.15.89:0/2734589762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00380d16a0 msgr2=0x7f00380d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.078+0000 7f0040000700 1 --2- 172.21.15.89:0/2734589762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00380d16a0 0x7f00380d1e30 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f00380d0ff0 tx=0x7f0024009b00 comp rx=0 tx=0).stop 2024-04-02T07:55:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.079+0000 7f0040000700 1 -- 172.21.15.89:0/2734589762 shutdown_connections 2024-04-02T07:55:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.079+0000 7f0040000700 1 --2- 172.21.15.89:0/2734589762 >> v2:172.21.15.113:3300/0 conn(0x7f003815d9d0 0x7f00380d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.079+0000 7f0040000700 1 --2- 172.21.15.89:0/2734589762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00380d16a0 0x7f00380d1e30 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.079+0000 7f0040000700 1 -- 172.21.15.89:0/2734589762 >> 172.21.15.89:0/2734589762 conn(0x7f003808aff0 msgr2=0x7f00380d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.079+0000 7f0040000700 1 -- 172.21.15.89:0/2734589762 shutdown_connections 2024-04-02T07:55:06.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.079+0000 7f0040000700 1 -- 172.21.15.89:0/2734589762 wait complete. 2024-04-02T07:55:06.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.081+0000 7f0040000700 1 Processor -- start 2024-04-02T07:55:06.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.081+0000 7f0040000700 1 -- start start 2024-04-02T07:55:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.082+0000 7f0040000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00380d16a0 0x7f0038158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.082+0000 7f0040000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f00380d2370 0x7f003815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.082+0000 7f0040000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f003815d9d0 0x7f003815e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.082+0000 7f0040000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00380cc900 con 0x7f00380d16a0 2024-04-02T07:55:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.082+0000 7f0040000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003815ef90 con 0x7f003815d9d0 2024-04-02T07:55:06.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.082+0000 7f0040000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003815e920 con 0x7f00380d2370 2024-04-02T07:55:06.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.082+0000 7f003d600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00380d16a0 0x7f0038158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:06.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.082+0000 7f003d600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00380d16a0 0x7f0038158d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38590/0 (socket says 172.21.15.89:38590) 2024-04-02T07:55:06.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.082+0000 7f003d600700 1 -- 172.21.15.89:0/3454817406 learned_addr learned my addr 172.21.15.89:0/3454817406 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:06.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.082+0000 7f003cc00700 1 --2- 172.21.15.89:0/3454817406 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f00380d2370 0x7f003815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:06.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.082+0000 7f003e000700 1 --2- 172.21.15.89:0/3454817406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f003815d9d0 0x7f003815e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:06.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.083+0000 7f003d600700 1 -- 172.21.15.89:0/3454817406 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f00380d2370 msgr2=0x7f003815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:06.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.083+0000 7f003d600700 1 --2- 172.21.15.89:0/3454817406 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f00380d2370 0x7f003815ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.083+0000 7f003d600700 1 -- 172.21.15.89:0/3454817406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f003815d9d0 msgr2=0x7f003815e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:06.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.083+0000 7f003d600700 1 --2- 172.21.15.89:0/3454817406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f003815d9d0 0x7f003815e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.083+0000 7f003d600700 1 -- 172.21.15.89:0/3454817406 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0024009430 con 0x7f00380d16a0 2024-04-02T07:55:06.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.083+0000 7f003d600700 1 --2- 172.21.15.89:0/3454817406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00380d16a0 0x7f0038158d10 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f002c00e870 tx=0x7f002c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.083+0000 7f0035600700 1 -- 172.21.15.89:0/3454817406 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c009590 con 0x7f00380d16a0 2024-04-02T07:55:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.083+0000 7f0035600700 1 -- 172.21.15.89:0/3454817406 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002c013680 con 0x7f00380d16a0 2024-04-02T07:55:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.083+0000 7f0040000700 1 -- 172.21.15.89:0/3454817406 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003815ec00 con 0x7f00380d16a0 2024-04-02T07:55:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.083+0000 7f0035600700 1 -- 172.21.15.89:0/3454817406 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c0139f0 con 0x7f00380d16a0 2024-04-02T07:55:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.083+0000 7f0040000700 1 -- 172.21.15.89:0/3454817406 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f003815f4c0 con 0x7f00380d16a0 2024-04-02T07:55:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.085+0000 7f0040000700 1 -- 172.21.15.89:0/3454817406 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00380d1ca0 con 0x7f00380d16a0 2024-04-02T07:55:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.086+0000 7f0035600700 1 -- 172.21.15.89:0/3454817406 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f002c013b50 con 0x7f00380d16a0 2024-04-02T07:55:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.086+0000 7f0035600700 1 --2- 172.21.15.89:0/3454817406 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f001806aed0 0x7f001806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.086+0000 7f0035600700 1 -- 172.21.15.89:0/3454817406 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f002c089880 con 0x7f00380d16a0 2024-04-02T07:55:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.086+0000 7f003cc00700 1 -- 172.21.15.89:0/3454817406 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f001806aed0 msgr2=0x7f001806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.087+0000 7f003cc00700 1 --2- 172.21.15.89:0/3454817406 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f001806aed0 0x7f001806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:06.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.094+0000 7f0035600700 1 -- 172.21.15.89:0/3454817406 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f002c054430 con 0x7f00380d16a0 2024-04-02T07:55:06.168 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:06.167+0000 7fd8340a5240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:55:06.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.288+0000 7f003cc00700 1 -- 172.21.15.89:0/3454817406 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f001806aed0 msgr2=0x7f001806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:06.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.288+0000 7f003cc00700 1 --2- 172.21.15.89:0/3454817406 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f001806aed0 0x7f001806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:06.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.305+0000 7f0040000700 1 -- 172.21.15.89:0/3454817406 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f003815f6b0 con 0x7f00380d16a0 2024-04-02T07:55:06.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.305+0000 7f0035600700 1 -- 172.21.15.89:0/3454817406 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v126) v1 ==== 100+0+11 (secure 0 0 0) 0x7f002c057a50 con 0x7f00380d16a0 2024-04-02T07:55:06.306 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:55:06.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.308+0000 7f0040000700 1 -- 172.21.15.89:0/3454817406 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f001806aed0 msgr2=0x7f001806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:06.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.308+0000 7f0040000700 1 --2- 172.21.15.89:0/3454817406 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f001806aed0 0x7f001806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.308+0000 7f0040000700 1 -- 172.21.15.89:0/3454817406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00380d16a0 msgr2=0x7f0038158d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:06.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.308+0000 7f0040000700 1 --2- 172.21.15.89:0/3454817406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00380d16a0 0x7f0038158d10 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f002c00e870 tx=0x7f002c00eb80 comp rx=0 tx=0).stop 2024-04-02T07:55:06.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.308+0000 7f0040000700 1 -- 172.21.15.89:0/3454817406 shutdown_connections 2024-04-02T07:55:06.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.308+0000 7f0040000700 1 --2- 172.21.15.89:0/3454817406 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f001806aed0 0x7f001806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.308+0000 7f0040000700 1 --2- 172.21.15.89:0/3454817406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f003815d9d0 0x7f003815e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.308+0000 7f0040000700 1 --2- 172.21.15.89:0/3454817406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00380d16a0 0x7f0038158d10 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.308+0000 7f0040000700 1 --2- 172.21.15.89:0/3454817406 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f00380d2370 0x7f003815ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.308+0000 7f0040000700 1 -- 172.21.15.89:0/3454817406 >> 172.21.15.89:0/3454817406 conn(0x7f003808aff0 msgr2=0x7f00380b13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:06.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.308+0000 7f0040000700 1 -- 172.21.15.89:0/3454817406 shutdown_connections 2024-04-02T07:55:06.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.308+0000 7f0040000700 1 -- 172.21.15.89:0/3454817406 wait complete. 2024-04-02T07:55:06.328 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T07:55:06.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.521+0000 7f9677000700 1 Processor -- start 2024-04-02T07:55:06.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.522+0000 7f9677000700 1 -- start start 2024-04-02T07:55:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.523+0000 7f9677000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f967006aea0 0x7f96700b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:06.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.523+0000 7f9677000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9670161150 con 0x7f9670160d70 2024-04-02T07:55:06.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.523+0000 7f9677000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967006dba0 con 0x7f967015ef70 2024-04-02T07:55:06.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.523+0000 7f9677000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96700b4cd0 con 0x7f967006aea0 2024-04-02T07:55:06.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.523+0000 7f966fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9670160d70 0x7f967015e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35706/0 (socket says 172.21.15.89:35706) 2024-04-02T07:55:06.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.523+0000 7f9675000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f967006aea0 0x7f96700b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:06.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.523+0000 7f966fe00700 1 -- 172.21.15.89:0/3512910663 learned_addr learned my addr 172.21.15.89:0/3512910663 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.524+0000 7f966ea00700 1 -- 172.21.15.89:0/3512910663 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379425735 0 0) 0x7f9670161150 con 0x7f9670160d70 2024-04-02T07:55:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.524+0000 7f966ea00700 1 -- 172.21.15.89:0/3512910663 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96540035b0 con 0x7f9670160d70 2024-04-02T07:55:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.524+0000 7f966ea00700 1 -- 172.21.15.89:0/3512910663 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579819038 0 0) 0x7f967006dba0 con 0x7f967015ef70 2024-04-02T07:55:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.524+0000 7f966ea00700 1 -- 172.21.15.89:0/3512910663 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9670161150 con 0x7f967015ef70 2024-04-02T07:55:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.524+0000 7f9675000700 1 -- 172.21.15.89:0/3512910663 >> v1:172.21.15.113:6789/0 conn(0x7f967015ef70 legacy=0x7f967006a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.524+0000 7f9675000700 1 -- 172.21.15.89:0/3512910663 >> v1:172.21.15.89:6789/0 conn(0x7f9670160d70 legacy=0x7f967015e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.524+0000 7f9675000700 1 -- 172.21.15.89:0/3512910663 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96700b4ee0 con 0x7f967006aea0 2024-04-02T07:55:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.524+0000 7f9675000700 1 --2- 172.21.15.89:0/3512910663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f967006aea0 0x7f96700b3200 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f9668009d90 tx=0x7f966800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70bdcf3937e9cd5d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.524+0000 7f966ea00700 1 -- 172.21.15.89:0/3512910663 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966800e070 con 0x7f967006aea0 2024-04-02T07:55:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.524+0000 7f966ea00700 1 -- 172.21.15.89:0/3512910663 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966800bb20 con 0x7f967006aea0 2024-04-02T07:55:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.524+0000 7f966ea00700 1 -- 172.21.15.89:0/3512910663 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966800be90 con 0x7f967006aea0 2024-04-02T07:55:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.525+0000 7f9677000700 1 -- 172.21.15.89:0/3512910663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f967006aea0 msgr2=0x7f96700b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.525+0000 7f9677000700 1 --2- 172.21.15.89:0/3512910663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f967006aea0 0x7f96700b3200 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f9668009d90 tx=0x7f966800b330 comp rx=0 tx=0).stop 2024-04-02T07:55:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.525+0000 7f9677000700 1 -- 172.21.15.89:0/3512910663 shutdown_connections 2024-04-02T07:55:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.525+0000 7f9677000700 1 --2- 172.21.15.89:0/3512910663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f967006aea0 0x7f96700b3200 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.525+0000 7f9677000700 1 -- 172.21.15.89:0/3512910663 >> 172.21.15.89:0/3512910663 conn(0x7f96700b5720 msgr2=0x7f96700b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:06.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.525+0000 7f9677000700 1 -- 172.21.15.89:0/3512910663 shutdown_connections 2024-04-02T07:55:06.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.525+0000 7f9677000700 1 -- 172.21.15.89:0/3512910663 wait complete. 2024-04-02T07:55:06.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.527+0000 7f9677000700 1 Processor -- start 2024-04-02T07:55:06.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.527+0000 7f9677000700 1 -- start start 2024-04-02T07:55:06.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f9677000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f967006aea0 0x7f9670153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:06.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f9677000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f967015ef70 0x7f9670152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:06.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f9677000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9670160d70 0x7f9670152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:06.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f9677000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96701540d0 con 0x7f9670160d70 2024-04-02T07:55:06.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f9677000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967014c780 con 0x7f967015ef70 2024-04-02T07:55:06.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f9677000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967014ca60 con 0x7f967006aea0 2024-04-02T07:55:06.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f9675000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9670160d70 0x7f9670152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:06.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f9675000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9670160d70 0x7f9670152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38602/0 (socket says 172.21.15.89:38602) 2024-04-02T07:55:06.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f9675000700 1 -- 172.21.15.89:0/2869170234 learned_addr learned my addr 172.21.15.89:0/2869170234 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:06.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f966fe00700 1 --2- 172.21.15.89:0/2869170234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f967006aea0 0x7f9670153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:06.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f966f400700 1 --2- 172.21.15.89:0/2869170234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f967015ef70 0x7f9670152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:06.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f9675000700 1 -- 172.21.15.89:0/2869170234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f967006aea0 msgr2=0x7f9670153b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:06.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f9675000700 1 --2- 172.21.15.89:0/2869170234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f967006aea0 0x7f9670153b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f9675000700 1 -- 172.21.15.89:0/2869170234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f967015ef70 msgr2=0x7f9670152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:06.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f9675000700 1 --2- 172.21.15.89:0/2869170234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f967015ef70 0x7f9670152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.528+0000 7f9675000700 1 -- 172.21.15.89:0/2869170234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9668009a70 con 0x7f9670160d70 2024-04-02T07:55:06.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.529+0000 7f9675000700 1 --2- 172.21.15.89:0/2869170234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9670160d70 0x7f9670152b90 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f9668013000 tx=0x7f9668004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:06.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.529+0000 7f9667e00700 1 -- 172.21.15.89:0/2869170234 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96680048c0 con 0x7f9670160d70 2024-04-02T07:55:06.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.529+0000 7f9667e00700 1 -- 172.21.15.89:0/2869170234 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9668020070 con 0x7f9670160d70 2024-04-02T07:55:06.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.529+0000 7f9677000700 1 -- 172.21.15.89:0/2869170234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967014cd40 con 0x7f9670160d70 2024-04-02T07:55:06.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.529+0000 7f9667e00700 1 -- 172.21.15.89:0/2869170234 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96680276c0 con 0x7f9670160d70 2024-04-02T07:55:06.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.529+0000 7f9677000700 1 -- 172.21.15.89:0/2869170234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967014d300 con 0x7f9670160d70 2024-04-02T07:55:06.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.531+0000 7f9677000700 1 -- 172.21.15.89:0/2869170234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9670067720 con 0x7f9670160d70 2024-04-02T07:55:06.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.531+0000 7f9667e00700 1 -- 172.21.15.89:0/2869170234 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f9668012430 con 0x7f9670160d70 2024-04-02T07:55:06.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.532+0000 7f9667e00700 1 --2- 172.21.15.89:0/2869170234 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f965006aed0 0x7f965006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:06.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.532+0000 7f9667e00700 1 -- 172.21.15.89:0/2869170234 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f9668022070 con 0x7f9670160d70 2024-04-02T07:55:06.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.532+0000 7f966fe00700 1 -- 172.21.15.89:0/2869170234 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f965006aed0 msgr2=0x7f965006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:06.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.532+0000 7f966fe00700 1 --2- 172.21.15.89:0/2869170234 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f965006aed0 0x7f965006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:06.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.536+0000 7f9667e00700 1 -- 172.21.15.89:0/2869170234 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9668056550 con 0x7f9670160d70 2024-04-02T07:55:06.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.733+0000 7f966fe00700 1 -- 172.21.15.89:0/2869170234 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f965006aed0 msgr2=0x7f965006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:06.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.733+0000 7f966fe00700 1 --2- 172.21.15.89:0/2869170234 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f965006aed0 0x7f965006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:06.742 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:06.738+0000 7f9aa6846240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:55:06.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.792+0000 7f9677000700 1 -- 172.21.15.89:0/2869170234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f967015f630 con 0x7f9670160d70 2024-04-02T07:55:06.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.792+0000 7f9667e00700 1 -- 172.21.15.89:0/2869170234 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v126) v1 ==== 80+0+454 (secure 0 0 0) 0x7f9668059b70 con 0x7f9670160d70 2024-04-02T07:55:06.793 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/49649361 2024-04-03T07:54:46.836267+0000 2024-04-02T07:55:06.793 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/51116008 2024-04-03T07:54:46.836267+0000 2024-04-02T07:55:06.793 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/236471161 2024-04-03T07:54:46.836267+0000 2024-04-02T07:55:06.793 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/709589358 2024-04-03T07:54:46.836267+0000 2024-04-02T07:55:06.793 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1096941421 2024-04-03T07:54:46.836267+0000 2024-04-02T07:55:06.793 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/43357 2024-04-03T07:54:46.836267+0000 2024-04-02T07:55:06.794 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/43357 2024-04-03T07:54:46.836267+0000 2024-04-02T07:55:06.794 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3407101260 2024-04-03T07:54:46.836267+0000 2024-04-02T07:55:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.795+0000 7f9677000700 1 -- 172.21.15.89:0/2869170234 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f965006aed0 msgr2=0x7f965006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.795+0000 7f9677000700 1 --2- 172.21.15.89:0/2869170234 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f965006aed0 0x7f965006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.795+0000 7f9677000700 1 -- 172.21.15.89:0/2869170234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9670160d70 msgr2=0x7f9670152b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.795+0000 7f9677000700 1 --2- 172.21.15.89:0/2869170234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9670160d70 0x7f9670152b90 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f9668013000 tx=0x7f9668004ad0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.795+0000 7f9677000700 1 -- 172.21.15.89:0/2869170234 shutdown_connections 2024-04-02T07:55:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.795+0000 7f9677000700 1 --2- 172.21.15.89:0/2869170234 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f965006aed0 0x7f965006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.795+0000 7f9677000700 1 --2- 172.21.15.89:0/2869170234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9670160d70 0x7f9670152b90 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.795+0000 7f9677000700 1 --2- 172.21.15.89:0/2869170234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f967006aea0 0x7f9670153b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.795+0000 7f9677000700 1 --2- 172.21.15.89:0/2869170234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f967015ef70 0x7f9670152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.795+0000 7f9677000700 1 -- 172.21.15.89:0/2869170234 >> 172.21.15.89:0/2869170234 conn(0x7f96700b5720 msgr2=0x7f967009a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.795+0000 7f9677000700 1 -- 172.21.15.89:0/2869170234 shutdown_connections 2024-04-02T07:55:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:06.795+0000 7f9677000700 1 -- 172.21.15.89:0/2869170234 wait complete. 2024-04-02T07:55:06.797 INFO:teuthology.orchestra.run.smithi089.stderr:listed 8 entries 2024-04-02T07:55:06.814 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T07:55:06.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:06.814 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T07:55:06.831 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:06.830+0000 7f9aa6846240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:55:06.888 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:55:06.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:06.889 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T07:55:06.933 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:55:06.933 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:06.958 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:06.958+0000 7fd8340a5240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:55:06.977 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:55:06.978 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:55:06.978 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:06.996 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:06.996+0000 7f9aa6846240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:55:07.022 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:55:07.022 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:55:07.022 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:07.053 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:07.052+0000 7fd8340a5240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:55:07.068 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:55:07.069 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:55:07.215 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:07.214+0000 7f9aa6846240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:55:07.223 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:07.223+0000 7fd8340a5240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:55:07.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.264+0000 7f680ee00700 1 Processor -- start 2024-04-02T07:55:07.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.264+0000 7f680ee00700 1 -- start start 2024-04-02T07:55:07.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.265+0000 7f680ee00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f680815d640 0x7f68080d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:07.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.265+0000 7f680ee00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f68081619d0 0x7f68080b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:07.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.265+0000 7f680ee00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68080d30d0 con 0x7f68080ceec0 2024-04-02T07:55:07.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.265+0000 7f680ee00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68080b6840 con 0x7f68081619d0 2024-04-02T07:55:07.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.265+0000 7f680ee00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68080d2880 con 0x7f680815d640 2024-04-02T07:55:07.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.266+0000 7f6807400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f68080ceec0 0x7f68081612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35708/0 (socket says 172.21.15.89:35708) 2024-04-02T07:55:07.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.266+0000 7f6807400700 1 -- 172.21.15.89:0/3551820170 learned_addr learned my addr 172.21.15.89:0/3551820170 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:07.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.266+0000 7f680ce00700 1 --2- 172.21.15.89:0/3551820170 >> v2:172.21.15.113:3300/0 conn(0x7f68081619d0 0x7f68080b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:07.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.266+0000 7f6807e00700 1 --2- 172.21.15.89:0/3551820170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f680815d640 0x7f68080d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:07.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.266+0000 7f6806a00700 1 -- 172.21.15.89:0/3551820170 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2390730378 0 0) 0x7f68080d30d0 con 0x7f68080ceec0 2024-04-02T07:55:07.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.266+0000 7f6806a00700 1 -- 172.21.15.89:0/3551820170 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67ec0035b0 con 0x7f68080ceec0 2024-04-02T07:55:07.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.266+0000 7f680ce00700 1 -- 172.21.15.89:0/3551820170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f680815d640 msgr2=0x7f68080d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:07.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.266+0000 7f680ce00700 1 --2- 172.21.15.89:0/3551820170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f680815d640 0x7f68080d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.266+0000 7f680ce00700 1 -- 172.21.15.89:0/3551820170 >> v1:172.21.15.89:6789/0 conn(0x7f68080ceec0 legacy=0x7f68081612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:07.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.266+0000 7f680ce00700 1 -- 172.21.15.89:0/3551820170 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68080d2b30 con 0x7f68081619d0 2024-04-02T07:55:07.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.266+0000 7f680ce00700 1 --2- 172.21.15.89:0/3551820170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68081619d0 0x7f68080b6cd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6800009d90 tx=0x7f680000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af71513d335673c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:07.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.267+0000 7f6806a00700 1 -- 172.21.15.89:0/3551820170 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680000e070 con 0x7f68081619d0 2024-04-02T07:55:07.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.267+0000 7f6806a00700 1 -- 172.21.15.89:0/3551820170 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680000bb20 con 0x7f68081619d0 2024-04-02T07:55:07.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.267+0000 7f6806a00700 1 -- 172.21.15.89:0/3551820170 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680000be90 con 0x7f68081619d0 2024-04-02T07:55:07.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.267+0000 7f680ee00700 1 -- 172.21.15.89:0/3551820170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68081619d0 msgr2=0x7f68080b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:07.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.267+0000 7f680ee00700 1 --2- 172.21.15.89:0/3551820170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68081619d0 0x7f68080b6cd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6800009d90 tx=0x7f680000b330 comp rx=0 tx=0).stop 2024-04-02T07:55:07.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.267+0000 7f680ee00700 1 -- 172.21.15.89:0/3551820170 shutdown_connections 2024-04-02T07:55:07.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.267+0000 7f680ee00700 1 --2- 172.21.15.89:0/3551820170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68081619d0 0x7f68080b6cd0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.267+0000 7f680ee00700 1 --2- 172.21.15.89:0/3551820170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f680815d640 0x7f68080d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.267+0000 7f680ee00700 1 -- 172.21.15.89:0/3551820170 >> 172.21.15.89:0/3551820170 conn(0x7f680808aff0 msgr2=0x7f68080d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:07.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.268+0000 7f680ee00700 1 -- 172.21.15.89:0/3551820170 shutdown_connections 2024-04-02T07:55:07.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.268+0000 7f680ee00700 1 -- 172.21.15.89:0/3551820170 wait complete. 2024-04-02T07:55:07.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.269+0000 7f680ee00700 1 Processor -- start 2024-04-02T07:55:07.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.270+0000 7f680ee00700 1 -- start start 2024-04-02T07:55:07.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.271+0000 7f680ee00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68080ceec0 0x7f680815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:07.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.271+0000 7f680ee00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f680815d640 0x7f680809da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:07.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.271+0000 7f680ee00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68081619d0 0x7f680809df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:07.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.271+0000 7f680ee00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68080c7f50 con 0x7f680815d640 2024-04-02T07:55:07.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.271+0000 7f680ee00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68080c8230 con 0x7f68080ceec0 2024-04-02T07:55:07.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.271+0000 7f680ee00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68080c8510 con 0x7f68081619d0 2024-04-02T07:55:07.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.271+0000 7f6807400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f680815d640 0x7f680809da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:07.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.271+0000 7f6807e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68080ceec0 0x7f680815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:07.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.271+0000 7f6807400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f680815d640 0x7f680809da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38612/0 (socket says 172.21.15.89:38612) 2024-04-02T07:55:07.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.271+0000 7f6807400700 1 -- 172.21.15.89:0/1922602674 learned_addr learned my addr 172.21.15.89:0/1922602674 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:07.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.271+0000 7f680ce00700 1 --2- 172.21.15.89:0/1922602674 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68081619d0 0x7f680809df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:07.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.272+0000 7f6807400700 1 -- 172.21.15.89:0/1922602674 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68081619d0 msgr2=0x7f680809df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:07.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.272+0000 7f6807400700 1 --2- 172.21.15.89:0/1922602674 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68081619d0 0x7f680809df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.272+0000 7f6807400700 1 -- 172.21.15.89:0/1922602674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68080ceec0 msgr2=0x7f680815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:07.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.272+0000 7f6807400700 1 --2- 172.21.15.89:0/1922602674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68080ceec0 0x7f680815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.272+0000 7f6807400700 1 -- 172.21.15.89:0/1922602674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6800009a70 con 0x7f680815d640 2024-04-02T07:55:07.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.272+0000 7f6807400700 1 --2- 172.21.15.89:0/1922602674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f680815d640 0x7f680809da20 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f67f400c420 tx=0x7f67f400c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:07.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.272+0000 7f67ffe00700 1 -- 172.21.15.89:0/1922602674 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4005d70 con 0x7f680815d640 2024-04-02T07:55:07.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.272+0000 7f67ffe00700 1 -- 172.21.15.89:0/1922602674 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f400ce30 con 0x7f680815d640 2024-04-02T07:55:07.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.272+0000 7f67ffe00700 1 -- 172.21.15.89:0/1922602674 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f40072d0 con 0x7f680815d640 2024-04-02T07:55:07.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.272+0000 7f680ee00700 1 -- 172.21.15.89:0/1922602674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68080c87f0 con 0x7f680815d640 2024-04-02T07:55:07.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.273+0000 7f680ee00700 1 -- 172.21.15.89:0/1922602674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680815b4f0 con 0x7f680815d640 2024-04-02T07:55:07.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.274+0000 7f680ee00700 1 -- 172.21.15.89:0/1922602674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68080c8d10 con 0x7f680815d640 2024-04-02T07:55:07.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.274+0000 7f67ffe00700 1 -- 172.21.15.89:0/1922602674 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f67f4007430 con 0x7f680815d640 2024-04-02T07:55:07.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.275+0000 7f67ffe00700 1 --2- 172.21.15.89:0/1922602674 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f67e806aed0 0x7f67e806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:07.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.275+0000 7f67ffe00700 1 -- 172.21.15.89:0/1922602674 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f67f4088c30 con 0x7f680815d640 2024-04-02T07:55:07.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.275+0000 7f6807e00700 1 -- 172.21.15.89:0/1922602674 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f67e806aed0 msgr2=0x7f67e806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:07.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.275+0000 7f6807e00700 1 --2- 172.21.15.89:0/1922602674 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f67e806aed0 0x7f67e806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:07.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.279+0000 7f67ffe00700 1 -- 172.21.15.89:0/1922602674 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f68080c8d10 con 0x7f680815d640 2024-04-02T07:55:07.307 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:07.306+0000 7f9aa6846240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:55:07.404 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:07.403+0000 7f9aa6846240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:55:07.442 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:07.441+0000 7fd8340a5240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:55:07.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.476+0000 7f6807e00700 1 -- 172.21.15.89:0/1922602674 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f67e806aed0 msgr2=0x7f67e806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:07.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.476+0000 7f6807e00700 1 --2- 172.21.15.89:0/1922602674 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f67e806aed0 0x7f67e806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:07.501 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:07.500+0000 7f9aa6846240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:55:07.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.528+0000 7f680ee00700 1 -- 172.21.15.89:0/1922602674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68080c8d10 con 0x7f680815d640 2024-04-02T07:55:07.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.529+0000 7f67ffe00700 1 -- 172.21.15.89:0/1922602674 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f68080c8d10 con 0x7f680815d640 2024-04-02T07:55:07.529 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:07.529 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1967870793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1967870793},{"type":"v1","addr":"172.21.15.89:6837","nonce":1967870793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1050024225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1050024225},{"type":"v1","addr":"172.21.15.113:6835","nonce":1050024225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7447,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/235320877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":235320877},{"type":"v1","addr":"172.21.15.113:6834","nonce":235320877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:54:30.522457+0000","modified":"2024-04-02T07:54:33.536401+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/1949161483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1949161483},{"type":"v1","addr":"172.21.15.89:6836","nonce":1949161483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-02T07:55:07.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.531+0000 7f680ee00700 1 -- 172.21.15.89:0/1922602674 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f67e806aed0 msgr2=0x7f67e806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:07.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.531+0000 7f680ee00700 1 --2- 172.21.15.89:0/1922602674 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f67e806aed0 0x7f67e806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.531+0000 7f680ee00700 1 -- 172.21.15.89:0/1922602674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f680815d640 msgr2=0x7f680809da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:07.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.531+0000 7f680ee00700 1 --2- 172.21.15.89:0/1922602674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f680815d640 0x7f680809da20 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f67f400c420 tx=0x7f67f400c730 comp rx=0 tx=0).stop 2024-04-02T07:55:07.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.531+0000 7f680ee00700 1 -- 172.21.15.89:0/1922602674 shutdown_connections 2024-04-02T07:55:07.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.531+0000 7f680ee00700 1 --2- 172.21.15.89:0/1922602674 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f67e806aed0 0x7f67e806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.531+0000 7f680ee00700 1 --2- 172.21.15.89:0/1922602674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68080ceec0 0x7f680815afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.531+0000 7f680ee00700 1 --2- 172.21.15.89:0/1922602674 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68081619d0 0x7f680809df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.531+0000 7f680ee00700 1 --2- 172.21.15.89:0/1922602674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f680815d640 0x7f680809da20 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.531+0000 7f680ee00700 1 -- 172.21.15.89:0/1922602674 >> 172.21.15.89:0/1922602674 conn(0x7f680808aff0 msgr2=0x7f68080b82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:07.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.531+0000 7f680ee00700 1 -- 172.21.15.89:0/1922602674 shutdown_connections 2024-04-02T07:55:07.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.532+0000 7f680ee00700 1 -- 172.21.15.89:0/1922602674 wait complete. 2024-04-02T07:55:07.533 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 57 2024-04-02T07:55:07.536 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:07.535+0000 7fd8340a5240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:55:07.551 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:55:07.597 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:07.596+0000 7f9aa6846240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:55:07.643 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:07.643+0000 7fd8340a5240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:55:07.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.746+0000 7f0f90400700 1 Processor -- start 2024-04-02T07:55:07.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.746+0000 7f0f90400700 1 -- start start 2024-04-02T07:55:07.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.747+0000 7f0f90400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0f880718e0 0x7f0f880ad070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:07.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.747+0000 7f0f90400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f880ac4d0 con 0x7f0f88160c90 2024-04-02T07:55:07.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.747+0000 7f0f90400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f88071e50 con 0x7f0f880b27f0 2024-04-02T07:55:07.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.747+0000 7f0f90400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f88072130 con 0x7f0f880718e0 2024-04-02T07:55:07.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.747+0000 7f0f8da00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0f88160c90 0x7f0f880b20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35712/0 (socket says 172.21.15.89:35712) 2024-04-02T07:55:07.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.747+0000 7f0f8da00700 1 -- 172.21.15.89:0/52733644 learned_addr learned my addr 172.21.15.89:0/52733644 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:07.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.747+0000 7f0f8e400700 1 --2- 172.21.15.89:0/52733644 >> v2:172.21.15.113:3300/0 conn(0x7f0f880718e0 0x7f0f880ad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.748+0000 7f0f8e400700 1 -- 172.21.15.89:0/52733644 >> v1:172.21.15.113:6789/0 conn(0x7f0f880b27f0 legacy=0x7f0f880ac960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.748+0000 7f0f87e00700 1 -- 172.21.15.89:0/52733644 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051362761 0 0) 0x7f0f880ac4d0 con 0x7f0f88160c90 2024-04-02T07:55:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.748+0000 7f0f8e400700 1 -- 172.21.15.89:0/52733644 >> v1:172.21.15.89:6789/0 conn(0x7f0f88160c90 legacy=0x7f0f880b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:07.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.748+0000 7f0f8e400700 1 -- 172.21.15.89:0/52733644 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f88072410 con 0x7f0f880718e0 2024-04-02T07:55:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.748+0000 7f0f8e400700 1 --2- 172.21.15.89:0/52733644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f880718e0 0x7f0f880ad070 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0f80009d90 tx=0x7f0f8000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b23804362e5eb17 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.748+0000 7f0f87e00700 1 -- 172.21.15.89:0/52733644 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8000e070 con 0x7f0f880718e0 2024-04-02T07:55:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.749+0000 7f0f87e00700 1 -- 172.21.15.89:0/52733644 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8000bbd0 con 0x7f0f880718e0 2024-04-02T07:55:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.749+0000 7f0f87e00700 1 -- 172.21.15.89:0/52733644 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f80012430 con 0x7f0f880718e0 2024-04-02T07:55:07.750 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:07.749+0000 7fd8340a5240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:55:07.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.749+0000 7f0f90400700 1 -- 172.21.15.89:0/52733644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f880718e0 msgr2=0x7f0f880ad070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:07.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.749+0000 7f0f90400700 1 --2- 172.21.15.89:0/52733644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f880718e0 0x7f0f880ad070 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0f80009d90 tx=0x7f0f8000b330 comp rx=0 tx=0).stop 2024-04-02T07:55:07.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.749+0000 7f0f90400700 1 -- 172.21.15.89:0/52733644 shutdown_connections 2024-04-02T07:55:07.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.749+0000 7f0f90400700 1 --2- 172.21.15.89:0/52733644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f880718e0 0x7f0f880ad070 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.749+0000 7f0f90400700 1 -- 172.21.15.89:0/52733644 >> 172.21.15.89:0/52733644 conn(0x7f0f880cf190 msgr2=0x7f0f880cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:07.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.749+0000 7f0f90400700 1 -- 172.21.15.89:0/52733644 shutdown_connections 2024-04-02T07:55:07.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.749+0000 7f0f90400700 1 -- 172.21.15.89:0/52733644 wait complete. 2024-04-02T07:55:07.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.751+0000 7f0f90400700 1 Processor -- start 2024-04-02T07:55:07.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.751+0000 7f0f90400700 1 -- start start 2024-04-02T07:55:07.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.752+0000 7f0f90400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f880718e0 0x7f0f88159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:07.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.752+0000 7f0f90400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f880b27f0 0x7f0f88157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:07.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.752+0000 7f0f90400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f88160c90 0x7f0f8809bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:07.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.752+0000 7f0f90400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f88159b80 con 0x7f0f880b27f0 2024-04-02T07:55:07.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.752+0000 7f0f90400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f881581b0 con 0x7f0f88160c90 2024-04-02T07:55:07.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.752+0000 7f0f90400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f88158490 con 0x7f0f880718e0 2024-04-02T07:55:07.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.752+0000 7f0f8d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f880b27f0 0x7f0f88157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:07.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.752+0000 7f0f8e400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f88160c90 0x7f0f8809bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:07.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.752+0000 7f0f8da00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f880718e0 0x7f0f88159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:07.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.752+0000 7f0f8d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f880b27f0 0x7f0f88157840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38618/0 (socket says 172.21.15.89:38618) 2024-04-02T07:55:07.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.752+0000 7f0f8d000700 1 -- 172.21.15.89:0/616567415 learned_addr learned my addr 172.21.15.89:0/616567415 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:07.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.752+0000 7f0f8e400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f88160c90 0x7f0f8809bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:51736/0 (socket says 172.21.15.89:51736) 2024-04-02T07:55:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.753+0000 7f0f8e400700 1 -- 172.21.15.89:0/616567415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f880718e0 msgr2=0x7f0f88159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.753+0000 7f0f8e400700 1 --2- 172.21.15.89:0/616567415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f880718e0 0x7f0f88159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.753+0000 7f0f8e400700 1 -- 172.21.15.89:0/616567415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f880b27f0 msgr2=0x7f0f88157840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.753+0000 7f0f8e400700 1 --2- 172.21.15.89:0/616567415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f880b27f0 0x7f0f88157840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.753+0000 7f0f8e400700 1 -- 172.21.15.89:0/616567415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f80009a70 con 0x7f0f88160c90 2024-04-02T07:55:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.753+0000 7f0f8e400700 1 --2- 172.21.15.89:0/616567415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f88160c90 0x7f0f8809bdc0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0f80016000 tx=0x7f0f80004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.753+0000 7f0f85600700 1 -- 172.21.15.89:0/616567415 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8001b810 con 0x7f0f88160c90 2024-04-02T07:55:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.753+0000 7f0f85600700 1 -- 172.21.15.89:0/616567415 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f80020070 con 0x7f0f88160c90 2024-04-02T07:55:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.753+0000 7f0f90400700 1 -- 172.21.15.89:0/616567415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f88158770 con 0x7f0f88160c90 2024-04-02T07:55:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.753+0000 7f0f85600700 1 -- 172.21.15.89:0/616567415 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8001bbf0 con 0x7f0f88160c90 2024-04-02T07:55:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.754+0000 7f0f90400700 1 -- 172.21.15.89:0/616567415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f881f2fd0 con 0x7f0f88160c90 2024-04-02T07:55:07.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.755+0000 7f0f90400700 1 -- 172.21.15.89:0/616567415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f88067720 con 0x7f0f88160c90 2024-04-02T07:55:07.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.756+0000 7f0f85600700 1 -- 172.21.15.89:0/616567415 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f0f8001bd50 con 0x7f0f88160c90 2024-04-02T07:55:07.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.756+0000 7f0f85600700 1 --2- 172.21.15.89:0/616567415 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f6806aed0 0x7f0f6806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:07.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.756+0000 7f0f85600700 1 -- 172.21.15.89:0/616567415 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f0f80022070 con 0x7f0f88160c90 2024-04-02T07:55:07.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.756+0000 7f0f8da00700 1 -- 172.21.15.89:0/616567415 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f6806aed0 msgr2=0x7f0f6806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:07.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.756+0000 7f0f8da00700 1 --2- 172.21.15.89:0/616567415 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f6806aed0 0x7f0f6806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:07.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.761+0000 7f0f85600700 1 -- 172.21.15.89:0/616567415 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0f800568f0 con 0x7f0f88160c90 2024-04-02T07:55:07.808 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:07.807+0000 7f9aa6846240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:55:07.854 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:07.854+0000 7fd8340a5240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:55:07.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.957+0000 7f0f8da00700 1 -- 172.21.15.89:0/616567415 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f6806aed0 msgr2=0x7f0f6806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:07.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.957+0000 7f0f8da00700 1 --2- 172.21.15.89:0/616567415 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f6806aed0 0x7f0f6806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:07.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.973+0000 7f0f90400700 1 -- 172.21.15.89:0/616567415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f8806daf0 con 0x7f0f88160c90 2024-04-02T07:55:07.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.975+0000 7f0f85600700 1 -- 172.21.15.89:0/616567415 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+17063 (secure 0 0 0) 0x7f0f8002c090 con 0x7f0f88160c90 2024-04-02T07:55:07.976 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:07.976 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":126,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:54:46.836305+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:54:28.130987+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-02T07:54:29.515956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/49649361":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/51116008":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/236471161":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/709589358":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/1096941421":"2024-04-03T07:54:46.836267+0000","172.21.15.89:6833/43357":"2024-04-03T07:54:46.836267+0000","172.21.15.89:6832/43357":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/3407101260":"2024-04-03T07:54:46.836267+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:55:07.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.977+0000 7f0f90400700 1 -- 172.21.15.89:0/616567415 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f6806aed0 msgr2=0x7f0f6806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:07.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.977+0000 7f0f90400700 1 --2- 172.21.15.89:0/616567415 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f6806aed0 0x7f0f6806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.977+0000 7f0f90400700 1 -- 172.21.15.89:0/616567415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f88160c90 msgr2=0x7f0f8809bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:07.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.977+0000 7f0f90400700 1 --2- 172.21.15.89:0/616567415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f88160c90 0x7f0f8809bdc0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0f80016000 tx=0x7f0f80004ad0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.977+0000 7f0f90400700 1 -- 172.21.15.89:0/616567415 shutdown_connections 2024-04-02T07:55:07.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.977+0000 7f0f90400700 1 --2- 172.21.15.89:0/616567415 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f6806aed0 0x7f0f6806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.977+0000 7f0f90400700 1 --2- 172.21.15.89:0/616567415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f88160c90 0x7f0f8809bdc0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.977+0000 7f0f90400700 1 --2- 172.21.15.89:0/616567415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f880718e0 0x7f0f88159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.977+0000 7f0f90400700 1 --2- 172.21.15.89:0/616567415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f880b27f0 0x7f0f88157840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:07.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.977+0000 7f0f90400700 1 -- 172.21.15.89:0/616567415 >> 172.21.15.89:0/616567415 conn(0x7f0f880cf190 msgr2=0x7f0f8806e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:07.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.977+0000 7f0f90400700 1 -- 172.21.15.89:0/616567415 shutdown_connections 2024-04-02T07:55:07.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:07.978+0000 7f0f90400700 1 -- 172.21.15.89:0/616567415 wait complete. 2024-04-02T07:55:07.997 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:54:28.130987+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:54:29.515956+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:55:07.997 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:55:08.064 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:08.063+0000 7f9aa6846240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:55:08.082 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:08.081+0000 7fd8340a5240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:55:08.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.199+0000 7f7238400700 1 Processor -- start 2024-04-02T07:55:08.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.200+0000 7f7238400700 1 -- start start 2024-04-02T07:55:08.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.201+0000 7f7238400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72300d16a0 0x7f72300d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:08.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.201+0000 7f7238400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f72300d2370 0x7f72300ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:08.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.201+0000 7f7238400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7230161420 con 0x7f723015d9d0 2024-04-02T07:55:08.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.201+0000 7f7238400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72300d2820 con 0x7f72300d2370 2024-04-02T07:55:08.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.201+0000 7f7238400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72300d4340 con 0x7f72300d16a0 2024-04-02T07:55:08.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.201+0000 7f7235a00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f723015d9d0 0x7f72300d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:35618/0 (socket says 172.21.15.89:35618) 2024-04-02T07:55:08.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.201+0000 7f7236400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f72300d2370 0x7f72300ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:08.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.201+0000 7f7235000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72300d16a0 0x7f72300d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:08.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.201+0000 7f7235a00700 1 -- 172.21.15.89:0/1847541016 learned_addr learned my addr 172.21.15.89:0/1847541016 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:08.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.202+0000 7f722fe00700 1 -- 172.21.15.89:0/1847541016 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220758372 0 0) 0x7f7230161420 con 0x7f723015d9d0 2024-04-02T07:55:08.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.202+0000 7f722fe00700 1 -- 172.21.15.89:0/1847541016 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72180035b0 con 0x7f723015d9d0 2024-04-02T07:55:08.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.202+0000 7f7236400700 1 -- 172.21.15.89:0/1847541016 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72300d16a0 msgr2=0x7f72300d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:08.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.202+0000 7f7236400700 1 --2- 172.21.15.89:0/1847541016 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72300d16a0 0x7f72300d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.202+0000 7f7236400700 1 -- 172.21.15.89:0/1847541016 >> v1:172.21.15.113:6789/0 conn(0x7f723015d9d0 legacy=0x7f72300d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:08.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.202+0000 7f7236400700 1 -- 172.21.15.89:0/1847541016 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7230161700 con 0x7f72300d2370 2024-04-02T07:55:08.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.202+0000 7f7235000700 1 --2- 172.21.15.89:0/1847541016 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72300d16a0 0x7f72300d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:55:08.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.202+0000 7f7236400700 1 --2- 172.21.15.89:0/1847541016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72300d2370 0x7f72300ce360 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7228009dc0 tx=0x7f722800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=781a6a9f3c34340c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:08.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.202+0000 7f722fe00700 1 -- 172.21.15.89:0/1847541016 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722800e070 con 0x7f72300d2370 2024-04-02T07:55:08.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.202+0000 7f722fe00700 1 -- 172.21.15.89:0/1847541016 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722800bae0 con 0x7f72300d2370 2024-04-02T07:55:08.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.202+0000 7f722fe00700 1 -- 172.21.15.89:0/1847541016 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722800be90 con 0x7f72300d2370 2024-04-02T07:55:08.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.203+0000 7f7238400700 1 -- 172.21.15.89:0/1847541016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72300d2370 msgr2=0x7f72300ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:08.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.203+0000 7f7238400700 1 --2- 172.21.15.89:0/1847541016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72300d2370 0x7f72300ce360 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7228009dc0 tx=0x7f722800b330 comp rx=0 tx=0).stop 2024-04-02T07:55:08.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.203+0000 7f7238400700 1 -- 172.21.15.89:0/1847541016 shutdown_connections 2024-04-02T07:55:08.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.203+0000 7f7238400700 1 --2- 172.21.15.89:0/1847541016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72300d2370 0x7f72300ce360 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.203+0000 7f7238400700 1 --2- 172.21.15.89:0/1847541016 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72300d16a0 0x7f72300d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.203+0000 7f7238400700 1 -- 172.21.15.89:0/1847541016 >> 172.21.15.89:0/1847541016 conn(0x7f723008aff0 msgr2=0x7f72300d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:08.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.203+0000 7f7238400700 1 -- 172.21.15.89:0/1847541016 shutdown_connections 2024-04-02T07:55:08.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.203+0000 7f7238400700 1 -- 172.21.15.89:0/1847541016 wait complete. 2024-04-02T07:55:08.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.205+0000 7f7238400700 1 Processor -- start 2024-04-02T07:55:08.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.205+0000 7f7238400700 1 -- start start 2024-04-02T07:55:08.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.206+0000 7f7238400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72300d16a0 0x7f723015eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:08.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.206+0000 7f7238400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72300d2370 0x7f72300ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:08.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.206+0000 7f7238400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f723015d9d0 0x7f72300cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:08.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.206+0000 7f7238400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723015e6c0 con 0x7f72300d2370 2024-04-02T07:55:08.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.206+0000 7f7238400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723015ddb0 con 0x7f72300d16a0 2024-04-02T07:55:08.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.206+0000 7f7238400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72300b7f10 con 0x7f723015d9d0 2024-04-02T07:55:08.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.206+0000 7f7235000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72300d2370 0x7f72300ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:08.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.207+0000 7f7235a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72300d16a0 0x7f723015eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:08.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.207+0000 7f7235000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72300d2370 0x7f72300ccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38632/0 (socket says 172.21.15.89:38632) 2024-04-02T07:55:08.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.207+0000 7f7235000700 1 -- 172.21.15.89:0/3952121216 learned_addr learned my addr 172.21.15.89:0/3952121216 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:08.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.207+0000 7f7236400700 1 --2- 172.21.15.89:0/3952121216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f723015d9d0 0x7f72300cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:08.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.207+0000 7f7235a00700 1 -- 172.21.15.89:0/3952121216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f723015d9d0 msgr2=0x7f72300cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:08.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.207+0000 7f7235a00700 1 --2- 172.21.15.89:0/3952121216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f723015d9d0 0x7f72300cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.207+0000 7f7235a00700 1 -- 172.21.15.89:0/3952121216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72300d2370 msgr2=0x7f72300ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:08.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.207+0000 7f7235a00700 1 --2- 172.21.15.89:0/3952121216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72300d2370 0x7f72300ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.207+0000 7f7235a00700 1 -- 172.21.15.89:0/3952121216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c009430 con 0x7f72300d16a0 2024-04-02T07:55:08.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.207+0000 7f7235000700 1 --2- 172.21.15.89:0/3952121216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72300d2370 0x7f72300ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:55:08.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.207+0000 7f7235a00700 1 --2- 172.21.15.89:0/3952121216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72300d16a0 0x7f723015eae0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f722400c430 tx=0x7f722400c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:08.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.207+0000 7f722d600700 1 -- 172.21.15.89:0/3952121216 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7224005d70 con 0x7f72300d16a0 2024-04-02T07:55:08.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.208+0000 7f722d600700 1 -- 172.21.15.89:0/3952121216 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72240070d0 con 0x7f72300d16a0 2024-04-02T07:55:08.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.208+0000 7f722d600700 1 -- 172.21.15.89:0/3952121216 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7224007460 con 0x7f72300d16a0 2024-04-02T07:55:08.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.208+0000 7f7238400700 1 -- 172.21.15.89:0/3952121216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7228009a70 con 0x7f72300d16a0 2024-04-02T07:55:08.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.208+0000 7f7238400700 1 -- 172.21.15.89:0/3952121216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723015e2e0 con 0x7f72300d16a0 2024-04-02T07:55:08.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.209+0000 7f7238400700 1 -- 172.21.15.89:0/3952121216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72300d1ca0 con 0x7f72300d16a0 2024-04-02T07:55:08.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.210+0000 7f722d600700 1 -- 172.21.15.89:0/3952121216 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f72240075c0 con 0x7f72300d16a0 2024-04-02T07:55:08.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.210+0000 7f722d600700 1 --2- 172.21.15.89:0/3952121216 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f721006aed0 0x7f721006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:08.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.210+0000 7f722d600700 1 -- 172.21.15.89:0/3952121216 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f7224088df0 con 0x7f72300d16a0 2024-04-02T07:55:08.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.210+0000 7f7235000700 1 -- 172.21.15.89:0/3952121216 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f721006aed0 msgr2=0x7f721006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:08.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.211+0000 7f7235000700 1 --2- 172.21.15.89:0/3952121216 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f721006aed0 0x7f721006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:08.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.215+0000 7f722d600700 1 -- 172.21.15.89:0/3952121216 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7224053910 con 0x7f72300d16a0 2024-04-02T07:55:08.256 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:08.255+0000 7f9aa6846240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:55:08.347 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:08.346+0000 7f9aa6846240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:55:08.353 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:08.353+0000 7fd8340a5240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:55:08.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.412+0000 7f7235000700 1 -- 172.21.15.89:0/3952121216 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f721006aed0 msgr2=0x7f721006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:08.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.412+0000 7f7235000700 1 --2- 172.21.15.89:0/3952121216 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f721006aed0 0x7f721006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.424+0000 7f7238400700 1 -- 172.21.15.89:0/3952121216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f72300d4080 con 0x7f72300d16a0 2024-04-02T07:55:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.425+0000 7f722d600700 1 -- 172.21.15.89:0/3952121216 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7f72240170a0 con 0x7f72300d16a0 2024-04-02T07:55:08.426 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:55:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.427+0000 7f7238400700 1 -- 172.21.15.89:0/3952121216 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f721006aed0 msgr2=0x7f721006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.427+0000 7f7238400700 1 --2- 172.21.15.89:0/3952121216 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f721006aed0 0x7f721006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.428+0000 7f7238400700 1 -- 172.21.15.89:0/3952121216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72300d16a0 msgr2=0x7f723015eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.428+0000 7f7238400700 1 --2- 172.21.15.89:0/3952121216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72300d16a0 0x7f723015eae0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f722400c430 tx=0x7f722400c740 comp rx=0 tx=0).stop 2024-04-02T07:55:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.428+0000 7f7238400700 1 -- 172.21.15.89:0/3952121216 shutdown_connections 2024-04-02T07:55:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.428+0000 7f7238400700 1 --2- 172.21.15.89:0/3952121216 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f721006aed0 0x7f721006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.428+0000 7f7238400700 1 --2- 172.21.15.89:0/3952121216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72300d16a0 0x7f723015eae0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.428+0000 7f7238400700 1 --2- 172.21.15.89:0/3952121216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f723015d9d0 0x7f72300cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.428+0000 7f7238400700 1 --2- 172.21.15.89:0/3952121216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72300d2370 0x7f72300ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.428+0000 7f7238400700 1 -- 172.21.15.89:0/3952121216 >> 172.21.15.89:0/3952121216 conn(0x7f723008aff0 msgr2=0x7f72300b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.428+0000 7f7238400700 1 -- 172.21.15.89:0/3952121216 shutdown_connections 2024-04-02T07:55:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.428+0000 7f7238400700 1 -- 172.21.15.89:0/3952121216 wait complete. 2024-04-02T07:55:08.447 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:55:08.558 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:08.557+0000 7fd8340a5240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:55:08.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.645+0000 7fcaeec00700 1 Processor -- start 2024-04-02T07:55:08.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.645+0000 7fcaeec00700 1 -- start start 2024-04-02T07:55:08.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.646+0000 7fcaeec00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcae8160e50 0x7fcae80b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:08.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.646+0000 7fcaeec00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcae80b3f80 0x7fcae80b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:08.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.646+0000 7fcaeec00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae80cef00 con 0x7fcae80d3f20 2024-04-02T07:55:08.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.646+0000 7fcaeec00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae8161230 con 0x7fcae8160e50 2024-04-02T07:55:08.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.646+0000 7fcaeec00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae80d2a80 con 0x7fcae80b3f80 2024-04-02T07:55:08.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.647+0000 7fcae7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcae8160e50 0x7fcae80b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:08.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.647+0000 7fcae7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcae8160e50 0x7fcae80b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38646/0 (socket says 172.21.15.89:38646) 2024-04-02T07:55:08.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.647+0000 7fcae7e00700 1 -- 172.21.15.89:0/3863032263 learned_addr learned my addr 172.21.15.89:0/3863032263 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:08.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.647+0000 7fcaecc00700 1 --2- 172.21.15.89:0/3863032263 >> v2:172.21.15.113:3300/0 conn(0x7fcae80b3f80 0x7fcae80b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:08.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.647+0000 7fcae7e00700 1 -- 172.21.15.89:0/3863032263 >> v2:172.21.15.113:3300/0 conn(0x7fcae80b3f80 msgr2=0x7fcae80b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:08.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.647+0000 7fcae7e00700 1 --2- 172.21.15.89:0/3863032263 >> v2:172.21.15.113:3300/0 conn(0x7fcae80b3f80 0x7fcae80b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.647+0000 7fcae7e00700 1 -- 172.21.15.89:0/3863032263 >> v1:172.21.15.89:6789/0 conn(0x7fcae80d3f20 legacy=0x7fcae80b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:08.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.647+0000 7fcae7e00700 1 -- 172.21.15.89:0/3863032263 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae80d43b0 con 0x7fcae8160e50 2024-04-02T07:55:08.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.648+0000 7fcae7e00700 1 --2- 172.21.15.89:0/3863032263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae8160e50 0x7fcae80b4e90 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fcae80b6100 tx=0x7fcad8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3a77991e1be792 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.648+0000 7fcae6a00700 1 -- 172.21.15.89:0/3863032263 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad8011830 con 0x7fcae8160e50 2024-04-02T07:55:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.648+0000 7fcae6a00700 1 -- 172.21.15.89:0/3863032263 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad8011990 con 0x7fcae8160e50 2024-04-02T07:55:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.648+0000 7fcae6a00700 1 -- 172.21.15.89:0/3863032263 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad8011d20 con 0x7fcae8160e50 2024-04-02T07:55:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.648+0000 7fcaeec00700 1 -- 172.21.15.89:0/3863032263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae8160e50 msgr2=0x7fcae80b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.648+0000 7fcaeec00700 1 --2- 172.21.15.89:0/3863032263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae8160e50 0x7fcae80b4e90 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fcae80b6100 tx=0x7fcad8009bd0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.649+0000 7fcaeec00700 1 -- 172.21.15.89:0/3863032263 shutdown_connections 2024-04-02T07:55:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.649+0000 7fcaeec00700 1 --2- 172.21.15.89:0/3863032263 >> v2:172.21.15.113:3300/0 conn(0x7fcae80b3f80 0x7fcae80b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.649+0000 7fcaeec00700 1 --2- 172.21.15.89:0/3863032263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae8160e50 0x7fcae80b4e90 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.649+0000 7fcaeec00700 1 -- 172.21.15.89:0/3863032263 >> 172.21.15.89:0/3863032263 conn(0x7fcae808aff0 msgr2=0x7fcae80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:08.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.649+0000 7fcaeec00700 1 -- 172.21.15.89:0/3863032263 shutdown_connections 2024-04-02T07:55:08.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.649+0000 7fcaeec00700 1 -- 172.21.15.89:0/3863032263 wait complete. 2024-04-02T07:55:08.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.650+0000 7fcaeec00700 1 Processor -- start 2024-04-02T07:55:08.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.651+0000 7fcaeec00700 1 -- start start 2024-04-02T07:55:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.651+0000 7fcaeec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcae80b3f80 0x7fcae814eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.651+0000 7fcaeec00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae8160e50 0x7fcae81f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.651+0000 7fcaeec00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcae81f6360 0x7fcae80b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.652+0000 7fcaeec00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae814f050 con 0x7fcae8160e50 2024-04-02T07:55:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.652+0000 7fcaeec00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae80cc1f0 con 0x7fcae81f6360 2024-04-02T07:55:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.652+0000 7fcaeec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae80cc4d0 con 0x7fcae80b3f80 2024-04-02T07:55:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.652+0000 7fcae7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae8160e50 0x7fcae81f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.652+0000 7fcae7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae8160e50 0x7fcae81f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38652/0 (socket says 172.21.15.89:38652) 2024-04-02T07:55:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.652+0000 7fcae7400700 1 -- 172.21.15.89:0/858657392 learned_addr learned my addr 172.21.15.89:0/858657392 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.652+0000 7fcaecc00700 1 --2- 172.21.15.89:0/858657392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcae81f6360 0x7fcae80b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.652+0000 7fcae7e00700 1 --2- 172.21.15.89:0/858657392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcae80b3f80 0x7fcae814eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:08.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.652+0000 7fcae7400700 1 -- 172.21.15.89:0/858657392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcae80b3f80 msgr2=0x7fcae814eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:08.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.652+0000 7fcae7400700 1 --2- 172.21.15.89:0/858657392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcae80b3f80 0x7fcae814eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.652+0000 7fcae7400700 1 -- 172.21.15.89:0/858657392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcae81f6360 msgr2=0x7fcae80b1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:08.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.652+0000 7fcae7400700 1 --2- 172.21.15.89:0/858657392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcae81f6360 0x7fcae80b1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.652+0000 7fcae7400700 1 -- 172.21.15.89:0/858657392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad8009550 con 0x7fcae8160e50 2024-04-02T07:55:08.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.652+0000 7fcae7400700 1 --2- 172.21.15.89:0/858657392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae8160e50 0x7fcae81f6be0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fcad400c4e0 tx=0x7fcad400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:08.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.653+0000 7fcadfe00700 1 -- 172.21.15.89:0/858657392 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad40052d0 con 0x7fcae8160e50 2024-04-02T07:55:08.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.653+0000 7fcadfe00700 1 -- 172.21.15.89:0/858657392 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad4005430 con 0x7fcae8160e50 2024-04-02T07:55:08.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.653+0000 7fcaeec00700 1 -- 172.21.15.89:0/858657392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae80cc7b0 con 0x7fcae8160e50 2024-04-02T07:55:08.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.653+0000 7fcadfe00700 1 -- 172.21.15.89:0/858657392 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad40038c0 con 0x7fcae8160e50 2024-04-02T07:55:08.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.653+0000 7fcaeec00700 1 -- 172.21.15.89:0/858657392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae80ccd70 con 0x7fcae8160e50 2024-04-02T07:55:08.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.654+0000 7fcaeec00700 1 -- 172.21.15.89:0/858657392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae80ccf60 con 0x7fcae8160e50 2024-04-02T07:55:08.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.655+0000 7fcadfe00700 1 -- 172.21.15.89:0/858657392 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fcad4003a20 con 0x7fcae8160e50 2024-04-02T07:55:08.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.656+0000 7fcadfe00700 1 --2- 172.21.15.89:0/858657392 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcac806aed0 0x7fcac806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:08.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.656+0000 7fcadfe00700 1 -- 172.21.15.89:0/858657392 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fcad4088e10 con 0x7fcae8160e50 2024-04-02T07:55:08.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.656+0000 7fcae7e00700 1 -- 172.21.15.89:0/858657392 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcac806aed0 msgr2=0x7fcac806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:08.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.656+0000 7fcae7e00700 1 --2- 172.21.15.89:0/858657392 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcac806aed0 0x7fcac806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:08.659 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:08.659+0000 7fd8340a5240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:55:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.661+0000 7fcadfe00700 1 -- 172.21.15.89:0/858657392 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcad408c030 con 0x7fcae8160e50 2024-04-02T07:55:08.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.857+0000 7fcae7e00700 1 -- 172.21.15.89:0/858657392 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcac806aed0 msgr2=0x7fcac806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:08.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.857+0000 7fcae7e00700 1 --2- 172.21.15.89:0/858657392 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcac806aed0 0x7fcac806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:08.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.870+0000 7fcaeec00700 1 -- 172.21.15.89:0/858657392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcae8067720 con 0x7fcae8160e50 2024-04-02T07:55:08.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.871+0000 7fcadfe00700 1 -- 172.21.15.89:0/858657392 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v126) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcad40170a0 con 0x7fcae8160e50 2024-04-02T07:55:08.872 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:55:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.873+0000 7fcaeec00700 1 -- 172.21.15.89:0/858657392 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcac806aed0 msgr2=0x7fcac806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.873+0000 7fcaeec00700 1 --2- 172.21.15.89:0/858657392 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcac806aed0 0x7fcac806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.873+0000 7fcaeec00700 1 -- 172.21.15.89:0/858657392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae8160e50 msgr2=0x7fcae81f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.873+0000 7fcaeec00700 1 --2- 172.21.15.89:0/858657392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae8160e50 0x7fcae81f6be0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fcad400c4e0 tx=0x7fcad400c7f0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.874+0000 7fcaeec00700 1 -- 172.21.15.89:0/858657392 shutdown_connections 2024-04-02T07:55:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.874+0000 7fcaeec00700 1 --2- 172.21.15.89:0/858657392 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcac806aed0 0x7fcac806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.874+0000 7fcaeec00700 1 --2- 172.21.15.89:0/858657392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcae81f6360 0x7fcae80b1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.874+0000 7fcaeec00700 1 --2- 172.21.15.89:0/858657392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcae80b3f80 0x7fcae814eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.874+0000 7fcaeec00700 1 --2- 172.21.15.89:0/858657392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcae8160e50 0x7fcae81f6be0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.874+0000 7fcaeec00700 1 -- 172.21.15.89:0/858657392 >> 172.21.15.89:0/858657392 conn(0x7fcae808aff0 msgr2=0x7fcae80b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.874+0000 7fcaeec00700 1 -- 172.21.15.89:0/858657392 shutdown_connections 2024-04-02T07:55:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:08.874+0000 7fcaeec00700 1 -- 172.21.15.89:0/858657392 wait complete. 2024-04-02T07:55:08.903 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:55:09.073 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:09.072+0000 7f9aa6846240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:55:09.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.099+0000 7ff6f0a00700 1 Processor -- start 2024-04-02T07:55:09.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.100+0000 7ff6f0a00700 1 -- start start 2024-04-02T07:55:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.101+0000 7ff6f0a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff6ec0d16a0 0x7ff6ec0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.101+0000 7ff6f0a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff6ec0d2370 0x7ff6ec0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.101+0000 7ff6f0a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec161420 con 0x7ff6ec15d9d0 2024-04-02T07:55:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.101+0000 7ff6f0a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6ec0d4340 con 0x7ff6ec0d2370 2024-04-02T07:55:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.101+0000 7ff6f0a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6ec15edc0 con 0x7ff6ec0d16a0 2024-04-02T07:55:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.101+0000 7ff6ea800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff6ec0d2370 0x7ff6ec0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.101+0000 7ff6ea800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff6ec0d2370 0x7ff6ec0ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47876/0 (socket says 172.21.15.89:47876) 2024-04-02T07:55:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.101+0000 7ff6ea800700 1 -- 172.21.15.89:0/402376260 learned_addr learned my addr 172.21.15.89:0/402376260 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.101+0000 7ff6e9e00700 1 --1- 172.21.15.89:0/402376260 >> v1:172.21.15.113:6789/0 conn(0x7ff6ec15d9d0 0x7ff6ec0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:46052/0 (socket says 172.21.15.89:46052) 2024-04-02T07:55:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.101+0000 7ff6e9400700 1 --2- 172.21.15.89:0/402376260 >> v2:172.21.15.113:3300/0 conn(0x7ff6ec0d16a0 0x7ff6ec0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.102+0000 7ff6ea800700 1 -- 172.21.15.89:0/402376260 >> v2:172.21.15.113:3300/0 conn(0x7ff6ec0d16a0 msgr2=0x7ff6ec0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.102+0000 7ff6e8a00700 1 -- 172.21.15.89:0/402376260 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696714498 0 0) 0x7ff6ec161420 con 0x7ff6ec15d9d0 2024-04-02T07:55:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.102+0000 7ff6ea800700 1 --2- 172.21.15.89:0/402376260 >> v2:172.21.15.113:3300/0 conn(0x7ff6ec0d16a0 0x7ff6ec0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.102+0000 7ff6ea800700 1 -- 172.21.15.89:0/402376260 >> v1:172.21.15.113:6789/0 conn(0x7ff6ec15d9d0 legacy=0x7ff6ec0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.102+0000 7ff6ea800700 1 -- 172.21.15.89:0/402376260 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec161700 con 0x7ff6ec0d2370 2024-04-02T07:55:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.102+0000 7ff6e9400700 1 --2- 172.21.15.89:0/402376260 >> v2:172.21.15.113:3300/0 conn(0x7ff6ec0d16a0 0x7ff6ec0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:55:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.102+0000 7ff6ea800700 1 --2- 172.21.15.89:0/402376260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6ec0d2370 0x7ff6ec0ce360 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec0d12f0 tx=0x7ff6e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609cb3e99d5e2957 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.102+0000 7ff6e8a00700 1 -- 172.21.15.89:0/402376260 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e400e070 con 0x7ff6ec0d2370 2024-04-02T07:55:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.102+0000 7ff6e8a00700 1 -- 172.21.15.89:0/402376260 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e400b910 con 0x7ff6ec0d2370 2024-04-02T07:55:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.102+0000 7ff6e8a00700 1 -- 172.21.15.89:0/402376260 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e400bcc0 con 0x7ff6ec0d2370 2024-04-02T07:55:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.103+0000 7ff6f0a00700 1 -- 172.21.15.89:0/402376260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6ec0d2370 msgr2=0x7ff6ec0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.103+0000 7ff6f0a00700 1 --2- 172.21.15.89:0/402376260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6ec0d2370 0x7ff6ec0ce360 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec0d12f0 tx=0x7ff6e400b330 comp rx=0 tx=0).stop 2024-04-02T07:55:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.103+0000 7ff6f0a00700 1 -- 172.21.15.89:0/402376260 shutdown_connections 2024-04-02T07:55:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.103+0000 7ff6f0a00700 1 --2- 172.21.15.89:0/402376260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6ec0d2370 0x7ff6ec0ce360 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.103+0000 7ff6f0a00700 1 --2- 172.21.15.89:0/402376260 >> v2:172.21.15.113:3300/0 conn(0x7ff6ec0d16a0 0x7ff6ec0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.103+0000 7ff6f0a00700 1 -- 172.21.15.89:0/402376260 >> 172.21.15.89:0/402376260 conn(0x7ff6ec08aff0 msgr2=0x7ff6ec0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.103+0000 7ff6f0a00700 1 -- 172.21.15.89:0/402376260 shutdown_connections 2024-04-02T07:55:09.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.103+0000 7ff6f0a00700 1 -- 172.21.15.89:0/402376260 wait complete. 2024-04-02T07:55:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.105+0000 7ff6f0a00700 1 Processor -- start 2024-04-02T07:55:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.105+0000 7ff6f0a00700 1 -- start start 2024-04-02T07:55:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.106+0000 7ff6f0a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6ec0d16a0 0x7ff6ec15c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.106+0000 7ff6f0a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6ec0d2370 0x7ff6ec0cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.106+0000 7ff6f0a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6ec15d9d0 0x7ff6ec15bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.106+0000 7ff6f0a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec0b6390 con 0x7ff6ec15d9d0 2024-04-02T07:55:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.106+0000 7ff6f0a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec0b6670 con 0x7ff6ec0d2370 2024-04-02T07:55:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.106+0000 7ff6f0a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec0b6950 con 0x7ff6ec0d16a0 2024-04-02T07:55:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.106+0000 7ff6ea800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6ec15d9d0 0x7ff6ec15bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.106+0000 7ff6e9400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6ec0d2370 0x7ff6ec0cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.106+0000 7ff6e9e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6ec0d16a0 0x7ff6ec15c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.106+0000 7ff6e9400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6ec0d2370 0x7ff6ec0cacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:59082/0 (socket says 172.21.15.89:59082) 2024-04-02T07:55:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.106+0000 7ff6e9400700 1 -- 172.21.15.89:0/3871710845 learned_addr learned my addr 172.21.15.89:0/3871710845 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.107+0000 7ff6e9400700 1 -- 172.21.15.89:0/3871710845 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6ec0d16a0 msgr2=0x7ff6ec15c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.107+0000 7ff6e9400700 1 --2- 172.21.15.89:0/3871710845 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6ec0d16a0 0x7ff6ec15c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.107+0000 7ff6e9400700 1 -- 172.21.15.89:0/3871710845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6ec15d9d0 msgr2=0x7ff6ec15bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.107+0000 7ff6e9400700 1 --2- 172.21.15.89:0/3871710845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6ec15d9d0 0x7ff6ec15bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.107+0000 7ff6e9400700 1 -- 172.21.15.89:0/3871710845 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d8009430 con 0x7ff6ec0d2370 2024-04-02T07:55:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.107+0000 7ff6e9400700 1 --2- 172.21.15.89:0/3871710845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6ec0d2370 0x7ff6ec0cacf0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800ebe0 tx=0x7ff6d800eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.107+0000 7ff6e2000700 1 -- 172.21.15.89:0/3871710845 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8009780 con 0x7ff6ec0d2370 2024-04-02T07:55:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.107+0000 7ff6e2000700 1 -- 172.21.15.89:0/3871710845 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d8005a20 con 0x7ff6ec0d2370 2024-04-02T07:55:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.107+0000 7ff6f0a00700 1 -- 172.21.15.89:0/3871710845 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e4009b30 con 0x7ff6ec0d2370 2024-04-02T07:55:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.107+0000 7ff6e2000700 1 -- 172.21.15.89:0/3871710845 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8005dd0 con 0x7ff6ec0d2370 2024-04-02T07:55:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.107+0000 7ff6f0a00700 1 -- 172.21.15.89:0/3871710845 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ec0b6020 con 0x7ff6ec0d2370 2024-04-02T07:55:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.108+0000 7ff6f0a00700 1 -- 172.21.15.89:0/3871710845 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6ec0d1ca0 con 0x7ff6ec0d2370 2024-04-02T07:55:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.109+0000 7ff6e2000700 1 -- 172.21.15.89:0/3871710845 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7ff6d800c850 con 0x7ff6ec0d2370 2024-04-02T07:55:09.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.111+0000 7ff6e2000700 1 --2- 172.21.15.89:0/3871710845 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff6cc06aed0 0x7ff6cc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:09.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.111+0000 7ff6e2000700 1 -- 172.21.15.89:0/3871710845 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7ff6d8089ca0 con 0x7ff6ec0d2370 2024-04-02T07:55:09.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.111+0000 7ff6e9e00700 1 -- 172.21.15.89:0/3871710845 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff6cc06aed0 msgr2=0x7ff6cc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:09.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.111+0000 7ff6e9e00700 1 --2- 172.21.15.89:0/3871710845 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff6cc06aed0 0x7ff6cc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:09.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.115+0000 7ff6e2000700 1 -- 172.21.15.89:0/3871710845 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff6d80547c0 con 0x7ff6ec0d2370 2024-04-02T07:55:09.287 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:09.286+0000 7f9aa6846240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:55:09.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.312+0000 7ff6e9e00700 1 -- 172.21.15.89:0/3871710845 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff6cc06aed0 msgr2=0x7ff6cc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:09.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.312+0000 7ff6e9e00700 1 --2- 172.21.15.89:0/3871710845 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff6cc06aed0 0x7ff6cc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:09.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.326+0000 7ff6f0a00700 1 -- 172.21.15.89:0/3871710845 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff6ec067720 con 0x7ff6ec0d2370 2024-04-02T07:55:09.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.327+0000 7ff6e2000700 1 -- 172.21.15.89:0/3871710845 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v126) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff6d80200a0 con 0x7ff6ec0d2370 2024-04-02T07:55:09.327 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:55:09.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.329+0000 7ff6f0a00700 1 -- 172.21.15.89:0/3871710845 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff6cc06aed0 msgr2=0x7ff6cc06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:09.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.329+0000 7ff6f0a00700 1 --2- 172.21.15.89:0/3871710845 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff6cc06aed0 0x7ff6cc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.329+0000 7ff6f0a00700 1 -- 172.21.15.89:0/3871710845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6ec0d2370 msgr2=0x7ff6ec0cacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:09.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.329+0000 7ff6f0a00700 1 --2- 172.21.15.89:0/3871710845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6ec0d2370 0x7ff6ec0cacf0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800ebe0 tx=0x7ff6d800eef0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.329+0000 7ff6f0a00700 1 -- 172.21.15.89:0/3871710845 shutdown_connections 2024-04-02T07:55:09.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.329+0000 7ff6f0a00700 1 --2- 172.21.15.89:0/3871710845 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff6cc06aed0 0x7ff6cc06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.329+0000 7ff6f0a00700 1 --2- 172.21.15.89:0/3871710845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6ec15d9d0 0x7ff6ec15bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.329+0000 7ff6f0a00700 1 --2- 172.21.15.89:0/3871710845 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6ec0d16a0 0x7ff6ec15c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.329+0000 7ff6f0a00700 1 --2- 172.21.15.89:0/3871710845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6ec0d2370 0x7ff6ec0cacf0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.329+0000 7ff6f0a00700 1 -- 172.21.15.89:0/3871710845 >> 172.21.15.89:0/3871710845 conn(0x7ff6ec08aff0 msgr2=0x7ff6ec0b1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:09.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.329+0000 7ff6f0a00700 1 -- 172.21.15.89:0/3871710845 shutdown_connections 2024-04-02T07:55:09.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.329+0000 7ff6f0a00700 1 -- 172.21.15.89:0/3871710845 wait complete. 2024-04-02T07:55:09.349 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:55:09.456 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:09.455+0000 7fd8340a5240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:55:09.491 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:09.490+0000 7f9aa6846240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:55:09.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.556+0000 7f12a8600700 1 Processor -- start 2024-04-02T07:55:09.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.556+0000 7f12a8600700 1 -- start start 2024-04-02T07:55:09.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.557+0000 7f12a8600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f12a015da50 0x7f12a00ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:09.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.557+0000 7f12a8600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12a00cec70 0x7f12a00d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:09.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.558+0000 7f12a8600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a00d24e0 con 0x7f12a00d0b20 2024-04-02T07:55:09.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.558+0000 7f12a8600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a00b95d0 con 0x7f12a015da50 2024-04-02T07:55:09.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.558+0000 7f12a8600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a015de30 con 0x7f12a00cec70 2024-04-02T07:55:09.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.558+0000 7f12a6600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f12a00d0b20 0x7f12a00b52e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59334/0 (socket says 172.21.15.89:59334) 2024-04-02T07:55:09.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.558+0000 7f12a6600700 1 -- 172.21.15.89:0/123986894 learned_addr learned my addr 172.21.15.89:0/123986894 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:09.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.558+0000 7f12a5c00700 1 --2- 172.21.15.89:0/123986894 >> v2:172.21.15.113:3300/0 conn(0x7f12a015da50 0x7f12a00ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:09.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.558+0000 7f12a5200700 1 --2- 172.21.15.89:0/123986894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12a00cec70 0x7f12a00d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:09.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.558+0000 7f129fe00700 1 -- 172.21.15.89:0/123986894 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598597602 0 0) 0x7f12a00d24e0 con 0x7f12a00d0b20 2024-04-02T07:55:09.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.558+0000 7f12a5200700 1 -- 172.21.15.89:0/123986894 >> v2:172.21.15.113:3300/0 conn(0x7f12a015da50 msgr2=0x7f12a00ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:09.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.558+0000 7f12a5200700 1 --2- 172.21.15.89:0/123986894 >> v2:172.21.15.113:3300/0 conn(0x7f12a015da50 0x7f12a00ce730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.558+0000 7f12a5200700 1 -- 172.21.15.89:0/123986894 >> v1:172.21.15.89:6789/0 conn(0x7f12a00d0b20 legacy=0x7f12a00b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:09.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.558+0000 7f12a5200700 1 -- 172.21.15.89:0/123986894 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a00d27c0 con 0x7f12a00cec70 2024-04-02T07:55:09.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.558+0000 7f12a5c00700 1 --2- 172.21.15.89:0/123986894 >> v2:172.21.15.113:3300/0 conn(0x7f12a015da50 0x7f12a00ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:55:09.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.559+0000 7f12a5200700 1 --2- 172.21.15.89:0/123986894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12a00cec70 0x7f12a00d05e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f128c009840 tx=0x7f128c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44caacbd18fbf2cb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:09.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.559+0000 7f129fe00700 1 -- 172.21.15.89:0/123986894 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c011830 con 0x7f12a00cec70 2024-04-02T07:55:09.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.559+0000 7f129fe00700 1 -- 172.21.15.89:0/123986894 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c011990 con 0x7f12a00cec70 2024-04-02T07:55:09.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.559+0000 7f129fe00700 1 -- 172.21.15.89:0/123986894 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c011d20 con 0x7f12a00cec70 2024-04-02T07:55:09.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.559+0000 7f12a8600700 1 -- 172.21.15.89:0/123986894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12a00cec70 msgr2=0x7f12a00d05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:09.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.559+0000 7f12a8600700 1 --2- 172.21.15.89:0/123986894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12a00cec70 0x7f12a00d05e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f128c009840 tx=0x7f128c009b50 comp rx=0 tx=0).stop 2024-04-02T07:55:09.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.560+0000 7f12a8600700 1 -- 172.21.15.89:0/123986894 shutdown_connections 2024-04-02T07:55:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.560+0000 7f12a8600700 1 --2- 172.21.15.89:0/123986894 >> v2:172.21.15.113:3300/0 conn(0x7f12a015da50 0x7f12a00ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.560+0000 7f12a8600700 1 --2- 172.21.15.89:0/123986894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12a00cec70 0x7f12a00d05e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.560+0000 7f12a8600700 1 -- 172.21.15.89:0/123986894 >> 172.21.15.89:0/123986894 conn(0x7f12a00666a0 msgr2=0x7f12a0066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.560+0000 7f12a8600700 1 -- 172.21.15.89:0/123986894 shutdown_connections 2024-04-02T07:55:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.560+0000 7f12a8600700 1 -- 172.21.15.89:0/123986894 wait complete. 2024-04-02T07:55:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.561+0000 7f12a8600700 1 Processor -- start 2024-04-02T07:55:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.562+0000 7f12a8600700 1 -- start start 2024-04-02T07:55:09.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.562+0000 7f12a8600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12a00cec70 0x7f12a00b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:09.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.562+0000 7f12a8600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12a00d0b20 0x7f12a014ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.562+0000 7f12a8600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12a015da50 0x7f12a00c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.562+0000 7f12a8600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a00b3800 con 0x7f12a00cec70 2024-04-02T07:55:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.562+0000 7f12a8600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a00b39f0 con 0x7f12a015da50 2024-04-02T07:55:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.562+0000 7f12a8600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a00b3cd0 con 0x7f12a00d0b20 2024-04-02T07:55:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.563+0000 7f12a5c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12a00cec70 0x7f12a00b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.563+0000 7f12a6600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12a015da50 0x7f12a00c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.563+0000 7f12a5c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12a00cec70 0x7f12a00b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47896/0 (socket says 172.21.15.89:47896) 2024-04-02T07:55:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.563+0000 7f12a5c00700 1 -- 172.21.15.89:0/1025682841 learned_addr learned my addr 172.21.15.89:0/1025682841 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.563+0000 7f12a5200700 1 --2- 172.21.15.89:0/1025682841 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12a00d0b20 0x7f12a014ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.563+0000 7f12a6600700 1 -- 172.21.15.89:0/1025682841 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12a00d0b20 msgr2=0x7f12a014ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.563+0000 7f12a6600700 1 --2- 172.21.15.89:0/1025682841 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12a00d0b20 0x7f12a014ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.563+0000 7f12a6600700 1 -- 172.21.15.89:0/1025682841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12a00cec70 msgr2=0x7f12a00b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.563+0000 7f12a6600700 1 --2- 172.21.15.89:0/1025682841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12a00cec70 0x7f12a00b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.563+0000 7f12a6600700 1 -- 172.21.15.89:0/1025682841 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12940094a0 con 0x7f12a015da50 2024-04-02T07:55:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.563+0000 7f12a5c00700 1 --2- 172.21.15.89:0/1025682841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12a00cec70 0x7f12a00b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:55:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.563+0000 7f12a6600700 1 --2- 172.21.15.89:0/1025682841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12a015da50 0x7f12a00c7720 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f129800cac0 tx=0x7f129800cdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.564+0000 7f129d600700 1 -- 172.21.15.89:0/1025682841 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129800a9a0 con 0x7f12a015da50 2024-04-02T07:55:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.564+0000 7f129d600700 1 -- 172.21.15.89:0/1025682841 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129800ab00 con 0x7f12a015da50 2024-04-02T07:55:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.564+0000 7f12a8600700 1 -- 172.21.15.89:0/1025682841 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128c0094a0 con 0x7f12a015da50 2024-04-02T07:55:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.564+0000 7f129d600700 1 -- 172.21.15.89:0/1025682841 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12980070d0 con 0x7f12a015da50 2024-04-02T07:55:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.564+0000 7f12a8600700 1 -- 172.21.15.89:0/1025682841 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a00b4290 con 0x7f12a015da50 2024-04-02T07:55:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.565+0000 7f12a8600700 1 -- 172.21.15.89:0/1025682841 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a0067720 con 0x7f12a015da50 2024-04-02T07:55:09.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.566+0000 7f129d600700 1 -- 172.21.15.89:0/1025682841 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f1298007230 con 0x7f12a015da50 2024-04-02T07:55:09.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.567+0000 7f129d600700 1 --2- 172.21.15.89:0/1025682841 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f128006aed0 0x7f128006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:09.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.567+0000 7f12a5c00700 1 -- 172.21.15.89:0/1025682841 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f128006aed0 msgr2=0x7f128006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:09.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.567+0000 7f129d600700 1 -- 172.21.15.89:0/1025682841 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f1298089630 con 0x7f12a015da50 2024-04-02T07:55:09.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.567+0000 7f12a5c00700 1 --2- 172.21.15.89:0/1025682841 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f128006aed0 0x7f128006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:09.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.571+0000 7f129d600700 1 -- 172.21.15.89:0/1025682841 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f12980541e0 con 0x7f12a015da50 2024-04-02T07:55:09.685 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:09.684+0000 7fd8340a5240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:55:09.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.768+0000 7f12a5c00700 1 -- 172.21.15.89:0/1025682841 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f128006aed0 msgr2=0x7f128006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:09.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.768+0000 7f12a5c00700 1 --2- 172.21.15.89:0/1025682841 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f128006aed0 0x7f128006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:09.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.828+0000 7f12a8600700 1 -- 172.21.15.89:0/1025682841 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12a0067fc0 con 0x7f12a015da50 2024-04-02T07:55:09.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.829+0000 7f129d600700 1 -- 172.21.15.89:0/1025682841 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f1298057800 con 0x7f12a015da50 2024-04-02T07:55:09.830 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:09.830 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1967870793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1967870793},{"type":"v1","addr":"172.21.15.89:6837","nonce":1967870793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1050024225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1050024225},{"type":"v1","addr":"172.21.15.113:6835","nonce":1050024225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7447,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/235320877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":235320877},{"type":"v1","addr":"172.21.15.113:6834","nonce":235320877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:54:30.522457+0000","modified":"2024-04-02T07:54:33.536401+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/1949161483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1949161483},{"type":"v1","addr":"172.21.15.89:6836","nonce":1949161483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-02T07:55:09.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.831+0000 7f12a8600700 1 -- 172.21.15.89:0/1025682841 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f128006aed0 msgr2=0x7f128006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:09.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.831+0000 7f12a8600700 1 --2- 172.21.15.89:0/1025682841 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f128006aed0 0x7f128006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.831+0000 7f12a8600700 1 -- 172.21.15.89:0/1025682841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12a015da50 msgr2=0x7f12a00c7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:09.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.831+0000 7f12a8600700 1 --2- 172.21.15.89:0/1025682841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12a015da50 0x7f12a00c7720 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f129800cac0 tx=0x7f129800cdd0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.831+0000 7f12a8600700 1 -- 172.21.15.89:0/1025682841 shutdown_connections 2024-04-02T07:55:09.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.831+0000 7f12a8600700 1 --2- 172.21.15.89:0/1025682841 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f128006aed0 0x7f128006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.832+0000 7f12a8600700 1 --2- 172.21.15.89:0/1025682841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12a015da50 0x7f12a00c7720 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.832+0000 7f12a8600700 1 --2- 172.21.15.89:0/1025682841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12a00cec70 0x7f12a00b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.832+0000 7f12a8600700 1 --2- 172.21.15.89:0/1025682841 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12a00d0b20 0x7f12a014ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:09.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.832+0000 7f12a8600700 1 -- 172.21.15.89:0/1025682841 >> 172.21.15.89:0/1025682841 conn(0x7f12a00666a0 msgr2=0x7f12a00d18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:09.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.832+0000 7f12a8600700 1 -- 172.21.15.89:0/1025682841 shutdown_connections 2024-04-02T07:55:09.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:09.832+0000 7f12a8600700 1 -- 172.21.15.89:0/1025682841 wait complete. 2024-04-02T07:55:09.833 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 57 2024-04-02T07:55:09.851 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:55:09.913 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:09.912+0000 7fd8340a5240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:55:09.968 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:09.967+0000 7f9aa6846240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:55:10.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.052+0000 7f0dbae00700 1 Processor -- start 2024-04-02T07:55:10.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.052+0000 7f0dbae00700 1 -- start start 2024-04-02T07:55:10.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.054+0000 7f0dbae00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0db406ca50 0x7f0db40b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:10.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.054+0000 7f0dbae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0db40b4320 0x7f0db415f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:10.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.054+0000 7f0dbae00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db415c940 con 0x7f0db4155b00 2024-04-02T07:55:10.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.054+0000 7f0dbae00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db40b4790 con 0x7f0db406ca50 2024-04-02T07:55:10.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.054+0000 7f0dbae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db415bd30 con 0x7f0db40b4320 2024-04-02T07:55:10.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.054+0000 7f0db3400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0db406ca50 0x7f0db40b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:10.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.054+0000 7f0db3400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0db406ca50 0x7f0db40b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47912/0 (socket says 172.21.15.89:47912) 2024-04-02T07:55:10.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.054+0000 7f0db8e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0db40b4320 0x7f0db415f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:10.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.054+0000 7f0db3400700 1 -- 172.21.15.89:0/2573430842 learned_addr learned my addr 172.21.15.89:0/2573430842 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:10.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.054+0000 7f0db3e00700 1 --1- 172.21.15.89:0/2573430842 >> v1:172.21.15.113:6789/0 conn(0x7f0db4155b00 0x7f0db40ac1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:46066/0 (socket says 172.21.15.89:46066) 2024-04-02T07:55:10.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.054+0000 7f0db3400700 1 -- 172.21.15.89:0/2573430842 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0db40b4320 msgr2=0x7f0db415f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:10.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.054+0000 7f0db3400700 1 --2- 172.21.15.89:0/2573430842 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0db40b4320 0x7f0db415f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.054+0000 7f0db3400700 1 -- 172.21.15.89:0/2573430842 >> v1:172.21.15.113:6789/0 conn(0x7f0db4155b00 legacy=0x7f0db40ac1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:10.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.054+0000 7f0db3400700 1 -- 172.21.15.89:0/2573430842 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db415c010 con 0x7f0db406ca50 2024-04-02T07:55:10.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.055+0000 7f0db3400700 1 --2- 172.21.15.89:0/2573430842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db406ca50 0x7f0db40b3de0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f0da00098f0 tx=0x7f0da0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=165046fc0787d162 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:10.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.055+0000 7f0db2a00700 1 -- 172.21.15.89:0/2573430842 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da0011830 con 0x7f0db406ca50 2024-04-02T07:55:10.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.055+0000 7f0db2a00700 1 -- 172.21.15.89:0/2573430842 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da0011990 con 0x7f0db406ca50 2024-04-02T07:55:10.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.055+0000 7f0db2a00700 1 -- 172.21.15.89:0/2573430842 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da0011d20 con 0x7f0db406ca50 2024-04-02T07:55:10.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.055+0000 7f0dbae00700 1 -- 172.21.15.89:0/2573430842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db406ca50 msgr2=0x7f0db40b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:10.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.055+0000 7f0dbae00700 1 --2- 172.21.15.89:0/2573430842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db406ca50 0x7f0db40b3de0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f0da00098f0 tx=0x7f0da0009c00 comp rx=0 tx=0).stop 2024-04-02T07:55:10.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.056+0000 7f0dbae00700 1 -- 172.21.15.89:0/2573430842 shutdown_connections 2024-04-02T07:55:10.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.056+0000 7f0dbae00700 1 --2- 172.21.15.89:0/2573430842 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0db40b4320 0x7f0db415f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.056+0000 7f0dbae00700 1 --2- 172.21.15.89:0/2573430842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db406ca50 0x7f0db40b3de0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.056+0000 7f0dbae00700 1 -- 172.21.15.89:0/2573430842 >> 172.21.15.89:0/2573430842 conn(0x7f0db40b0ef0 msgr2=0x7f0db40b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:10.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.056+0000 7f0dbae00700 1 -- 172.21.15.89:0/2573430842 shutdown_connections 2024-04-02T07:55:10.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.056+0000 7f0dbae00700 1 -- 172.21.15.89:0/2573430842 wait complete. 2024-04-02T07:55:10.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.058+0000 7f0dbae00700 1 Processor -- start 2024-04-02T07:55:10.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.059+0000 7f0dbae00700 1 -- start start 2024-04-02T07:55:10.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.060+0000 7f0dbae00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0db406ca50 0x7f0db409a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:10.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.060+0000 7f0dbae00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db40b4320 0x7f0db4071800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:10.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.060+0000 7f0dbae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0db4155b00 0x7f0db4099470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:10.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.060+0000 7f0dbae00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db4153910 con 0x7f0db40b4320 2024-04-02T07:55:10.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.060+0000 7f0dbae00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db409ab20 con 0x7f0db406ca50 2024-04-02T07:55:10.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.060+0000 7f0dbae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db41f3300 con 0x7f0db4155b00 2024-04-02T07:55:10.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.061+0000 7f0db3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db40b4320 0x7f0db4071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:10.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.061+0000 7f0db3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db40b4320 0x7f0db4071800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47916/0 (socket says 172.21.15.89:47916) 2024-04-02T07:55:10.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.061+0000 7f0db3400700 1 -- 172.21.15.89:0/487067321 learned_addr learned my addr 172.21.15.89:0/487067321 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:10.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.061+0000 7f0db3e00700 1 --2- 172.21.15.89:0/487067321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0db406ca50 0x7f0db409a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:10.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.061+0000 7f0db8e00700 1 --2- 172.21.15.89:0/487067321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0db4155b00 0x7f0db4099470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:10.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.061+0000 7f0db3400700 1 -- 172.21.15.89:0/487067321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0db4155b00 msgr2=0x7f0db4099470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:10.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.061+0000 7f0db3400700 1 --2- 172.21.15.89:0/487067321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0db4155b00 0x7f0db4099470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.061+0000 7f0db3400700 1 -- 172.21.15.89:0/487067321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0db406ca50 msgr2=0x7f0db409a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:10.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.061+0000 7f0db3400700 1 --2- 172.21.15.89:0/487067321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0db406ca50 0x7f0db409a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.061+0000 7f0db3400700 1 -- 172.21.15.89:0/487067321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da0009550 con 0x7f0db40b4320 2024-04-02T07:55:10.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.061+0000 7f0db3400700 1 --2- 172.21.15.89:0/487067321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db40b4320 0x7f0db4071800 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f0da0009c40 tx=0x7f0da0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:10.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.062+0000 7f0dabe00700 1 -- 172.21.15.89:0/487067321 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da0004340 con 0x7f0db40b4320 2024-04-02T07:55:10.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.062+0000 7f0dabe00700 1 -- 172.21.15.89:0/487067321 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da001f070 con 0x7f0db40b4320 2024-04-02T07:55:10.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.062+0000 7f0dabe00700 1 -- 172.21.15.89:0/487067321 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da00036a0 con 0x7f0db40b4320 2024-04-02T07:55:10.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.062+0000 7f0dbae00700 1 -- 172.21.15.89:0/487067321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db41f35e0 con 0x7f0db40b4320 2024-04-02T07:55:10.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.062+0000 7f0dbae00700 1 -- 172.21.15.89:0/487067321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db41f3ba0 con 0x7f0db40b4320 2024-04-02T07:55:10.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.063+0000 7f0dbae00700 1 -- 172.21.15.89:0/487067321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db4153a80 con 0x7f0db40b4320 2024-04-02T07:55:10.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.064+0000 7f0dabe00700 1 -- 172.21.15.89:0/487067321 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f0da0021030 con 0x7f0db40b4320 2024-04-02T07:55:10.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.064+0000 7f0dabe00700 1 --2- 172.21.15.89:0/487067321 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0d9406aed0 0x7f0d9406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:10.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.065+0000 7f0dabe00700 1 -- 172.21.15.89:0/487067321 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f0da008ad40 con 0x7f0db40b4320 2024-04-02T07:55:10.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.065+0000 7f0db3e00700 1 -- 172.21.15.89:0/487067321 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0d9406aed0 msgr2=0x7f0d9406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:10.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.065+0000 7f0db3e00700 1 --2- 172.21.15.89:0/487067321 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0d9406aed0 0x7f0d9406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:10.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.069+0000 7f0dabe00700 1 -- 172.21.15.89:0/487067321 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0db4153a80 con 0x7f0db40b4320 2024-04-02T07:55:10.183 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:10.182+0000 7f9aa6846240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:55:10.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.266+0000 7f0db3e00700 1 -- 172.21.15.89:0/487067321 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0d9406aed0 msgr2=0x7f0d9406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:10.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.266+0000 7f0db3e00700 1 --2- 172.21.15.89:0/487067321 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0d9406aed0 0x7f0d9406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:10.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.297+0000 7f0dbae00700 1 -- 172.21.15.89:0/487067321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0db4153a50 con 0x7f0db40b4320 2024-04-02T07:55:10.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.298+0000 7f0dabe00700 1 -- 172.21.15.89:0/487067321 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+17175 (secure 0 0 0) 0x7f0da0055940 con 0x7f0db40b4320 2024-04-02T07:55:10.299 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:10.299 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":127,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:55:10.085149+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:54:28.130987+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-02T07:54:29.515956+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":127,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/49649361":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/51116008":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/236471161":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/709589358":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/1096941421":"2024-04-03T07:54:46.836267+0000","172.21.15.89:6833/43357":"2024-04-03T07:54:46.836267+0000","172.21.15.89:6832/43357":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/3407101260":"2024-04-03T07:54:46.836267+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:55:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.301+0000 7f0dbae00700 1 -- 172.21.15.89:0/487067321 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0d9406aed0 msgr2=0x7f0d9406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.301+0000 7f0dbae00700 1 --2- 172.21.15.89:0/487067321 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0d9406aed0 0x7f0d9406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.301+0000 7f0dbae00700 1 -- 172.21.15.89:0/487067321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db40b4320 msgr2=0x7f0db4071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.301+0000 7f0dbae00700 1 --2- 172.21.15.89:0/487067321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db40b4320 0x7f0db4071800 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f0da0009c40 tx=0x7f0da0004750 comp rx=0 tx=0).stop 2024-04-02T07:55:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.301+0000 7f0dbae00700 1 -- 172.21.15.89:0/487067321 shutdown_connections 2024-04-02T07:55:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.301+0000 7f0dbae00700 1 --2- 172.21.15.89:0/487067321 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0d9406aed0 0x7f0d9406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.301+0000 7f0dbae00700 1 --2- 172.21.15.89:0/487067321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0db406ca50 0x7f0db409a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.301+0000 7f0dbae00700 1 --2- 172.21.15.89:0/487067321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0db4155b00 0x7f0db4099470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.301+0000 7f0dbae00700 1 --2- 172.21.15.89:0/487067321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db40b4320 0x7f0db4071800 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.301+0000 7f0dbae00700 1 -- 172.21.15.89:0/487067321 >> 172.21.15.89:0/487067321 conn(0x7f0db40b0ef0 msgr2=0x7f0db40b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.301+0000 7f0dbae00700 1 -- 172.21.15.89:0/487067321 shutdown_connections 2024-04-02T07:55:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.301+0000 7f0dbae00700 1 -- 172.21.15.89:0/487067321 wait complete. 2024-04-02T07:55:10.320 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T07:55:10.320 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:55:10.436 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:10.435+0000 7fd8340a5240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:55:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.520+0000 7f5333e00700 1 Processor -- start 2024-04-02T07:55:10.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.520+0000 7f5333e00700 1 -- start start 2024-04-02T07:55:10.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.521+0000 7f5333e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f532c0b27f0 0x7f532c0ac960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:10.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.521+0000 7f5333e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f532c0718e0 0x7f532c0acea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:10.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.521+0000 7f5333e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532c0ac4d0 con 0x7f532c160c90 2024-04-02T07:55:10.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.521+0000 7f5333e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f532c071e50 con 0x7f532c0718e0 2024-04-02T07:55:10.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.521+0000 7f5333e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532c072130 con 0x7f532c0b27f0 2024-04-02T07:55:10.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.521+0000 7f5331e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f532c0718e0 0x7f532c0acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:10.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.521+0000 7f5331e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f532c0718e0 0x7f532c0acea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47928/0 (socket says 172.21.15.89:47928) 2024-04-02T07:55:10.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.521+0000 7f5331e00700 1 -- 172.21.15.89:0/485304890 learned_addr learned my addr 172.21.15.89:0/485304890 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:10.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.521+0000 7f5331400700 1 --1- 172.21.15.89:0/485304890 >> v1:172.21.15.113:6789/0 conn(0x7f532c160c90 0x7f532c0b20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:46074/0 (socket says 172.21.15.89:46074) 2024-04-02T07:55:10.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.522+0000 7f5330a00700 1 --2- 172.21.15.89:0/485304890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f532c0b27f0 0x7f532c0ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:10.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.522+0000 7f5331e00700 1 -- 172.21.15.89:0/485304890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f532c0b27f0 msgr2=0x7f532c0ac960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:10.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.522+0000 7f5331e00700 1 --2- 172.21.15.89:0/485304890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f532c0b27f0 0x7f532c0ac960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.522+0000 7f5331e00700 1 -- 172.21.15.89:0/485304890 >> v1:172.21.15.113:6789/0 conn(0x7f532c160c90 legacy=0x7f532c0b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:10.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.522+0000 7f5331e00700 1 -- 172.21.15.89:0/485304890 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532c072410 con 0x7f532c0718e0 2024-04-02T07:55:10.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.522+0000 7f532be00700 1 -- 172.21.15.89:0/485304890 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604578238 0 0) 0x7f532c0ac4d0 con 0x7f532c160c90 2024-04-02T07:55:10.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.522+0000 7f5331e00700 1 --2- 172.21.15.89:0/485304890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c0718e0 0x7f532c0acea0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f5324009e50 tx=0x7f532400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551dddf1e3365bdc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:10.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.522+0000 7f532be00700 1 -- 172.21.15.89:0/485304890 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532400e070 con 0x7f532c0718e0 2024-04-02T07:55:10.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.522+0000 7f532be00700 1 -- 172.21.15.89:0/485304890 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532400b910 con 0x7f532c0718e0 2024-04-02T07:55:10.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.522+0000 7f532be00700 1 -- 172.21.15.89:0/485304890 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532400bcc0 con 0x7f532c0718e0 2024-04-02T07:55:10.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.523+0000 7f5333e00700 1 -- 172.21.15.89:0/485304890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c0718e0 msgr2=0x7f532c0acea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:10.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.523+0000 7f5333e00700 1 --2- 172.21.15.89:0/485304890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c0718e0 0x7f532c0acea0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f5324009e50 tx=0x7f532400b330 comp rx=0 tx=0).stop 2024-04-02T07:55:10.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.523+0000 7f5333e00700 1 -- 172.21.15.89:0/485304890 shutdown_connections 2024-04-02T07:55:10.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.523+0000 7f5333e00700 1 --2- 172.21.15.89:0/485304890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c0718e0 0x7f532c0acea0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.523+0000 7f5333e00700 1 --2- 172.21.15.89:0/485304890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f532c0b27f0 0x7f532c0ac960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.523+0000 7f5333e00700 1 -- 172.21.15.89:0/485304890 >> 172.21.15.89:0/485304890 conn(0x7f532c0cf190 msgr2=0x7f532c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:10.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.523+0000 7f5333e00700 1 -- 172.21.15.89:0/485304890 shutdown_connections 2024-04-02T07:55:10.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.523+0000 7f5333e00700 1 -- 172.21.15.89:0/485304890 wait complete. 2024-04-02T07:55:10.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.525+0000 7f5333e00700 1 Processor -- start 2024-04-02T07:55:10.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.525+0000 7f5333e00700 1 -- start start 2024-04-02T07:55:10.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.526+0000 7f5333e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f532c0718e0 0x7f532c147c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.526+0000 7f5333e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c0b27f0 0x7f532c1e7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.526+0000 7f5333e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f532c160c90 0x7f532c1e8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.526+0000 7f5333e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c148990 con 0x7f532c0b27f0 2024-04-02T07:55:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.526+0000 7f5333e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c1e8690 con 0x7f532c160c90 2024-04-02T07:55:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.526+0000 7f5333e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532c1e8970 con 0x7f532c0718e0 2024-04-02T07:55:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.526+0000 7f5330a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c0b27f0 0x7f532c1e7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.526+0000 7f5330a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c0b27f0 0x7f532c1e7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47944/0 (socket says 172.21.15.89:47944) 2024-04-02T07:55:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.526+0000 7f5330a00700 1 -- 172.21.15.89:0/33650723 learned_addr learned my addr 172.21.15.89:0/33650723 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.526+0000 7f5331400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f532c0718e0 0x7f532c147c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.526+0000 7f5331e00700 1 --2- 172.21.15.89:0/33650723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f532c160c90 0x7f532c1e8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.527+0000 7f5330a00700 1 -- 172.21.15.89:0/33650723 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f532c0718e0 msgr2=0x7f532c147c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.527+0000 7f5330a00700 1 --2- 172.21.15.89:0/33650723 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f532c0718e0 0x7f532c147c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.527+0000 7f5330a00700 1 -- 172.21.15.89:0/33650723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f532c160c90 msgr2=0x7f532c1e8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:10.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.527+0000 7f5330a00700 1 --2- 172.21.15.89:0/33650723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f532c160c90 0x7f532c1e8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.527+0000 7f5330a00700 1 -- 172.21.15.89:0/33650723 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5324009b30 con 0x7f532c0b27f0 2024-04-02T07:55:10.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.527+0000 7f5330a00700 1 --2- 172.21.15.89:0/33650723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c0b27f0 0x7f532c1e7b20 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f531800e9b0 tx=0x7f531800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:10.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.527+0000 7f5329600700 1 -- 172.21.15.89:0/33650723 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318009710 con 0x7f532c0b27f0 2024-04-02T07:55:10.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.527+0000 7f5329600700 1 -- 172.21.15.89:0/33650723 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531800ee50 con 0x7f532c0b27f0 2024-04-02T07:55:10.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.527+0000 7f5329600700 1 -- 172.21.15.89:0/33650723 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531800ca50 con 0x7f532c0b27f0 2024-04-02T07:55:10.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.527+0000 7f5333e00700 1 -- 172.21.15.89:0/33650723 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532c148340 con 0x7f532c0b27f0 2024-04-02T07:55:10.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.527+0000 7f5333e00700 1 -- 172.21.15.89:0/33650723 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532c15cc50 con 0x7f532c0b27f0 2024-04-02T07:55:10.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.528+0000 7f5333e00700 1 -- 172.21.15.89:0/33650723 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f532c148b40 con 0x7f532c0b27f0 2024-04-02T07:55:10.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.529+0000 7f5329600700 1 -- 172.21.15.89:0/33650723 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f531800cbb0 con 0x7f532c0b27f0 2024-04-02T07:55:10.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.530+0000 7f5329600700 1 --2- 172.21.15.89:0/33650723 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f530c06aed0 0x7f530c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:10.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.530+0000 7f5331400700 1 -- 172.21.15.89:0/33650723 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f530c06aed0 msgr2=0x7f530c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:10.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.530+0000 7f5329600700 1 -- 172.21.15.89:0/33650723 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6457+0+0 (secure 0 0 0) 0x7f5318089b00 con 0x7f532c0b27f0 2024-04-02T07:55:10.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.530+0000 7f5331400700 1 --2- 172.21.15.89:0/33650723 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f530c06aed0 0x7f530c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:10.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.539+0000 7f5329600700 1 -- 172.21.15.89:0/33650723 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f532c148b40 con 0x7f532c0b27f0 2024-04-02T07:55:10.631 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:10.630+0000 7f9aa6846240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:55:10.665 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:10.664+0000 7fd8340a5240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:55:10.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.731+0000 7f5331400700 1 -- 172.21.15.89:0/33650723 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f530c06aed0 msgr2=0x7f530c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:10.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.731+0000 7f5331400700 1 --2- 172.21.15.89:0/33650723 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f530c06aed0 0x7f530c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:10.741 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:10.740+0000 7f9aa6846240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:55:10.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.796+0000 7f5333e00700 1 -- 172.21.15.89:0/33650723 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f532c067720 con 0x7f532c0b27f0 2024-04-02T07:55:10.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.797+0000 7f5329600700 1 -- 172.21.15.89:0/33650723 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f531808be60 con 0x7f532c0b27f0 2024-04-02T07:55:10.797 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:10.797 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1967870793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1967870793},{"type":"v1","addr":"172.21.15.89:6837","nonce":1967870793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1050024225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1050024225},{"type":"v1","addr":"172.21.15.113:6835","nonce":1050024225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7447,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/235320877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":235320877},{"type":"v1","addr":"172.21.15.113:6834","nonce":235320877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:54:30.522457+0000","modified":"2024-04-02T07:54:33.536401+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/1949161483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1949161483},{"type":"v1","addr":"172.21.15.89:6836","nonce":1949161483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-02T07:55:10.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.799+0000 7f5333e00700 1 -- 172.21.15.89:0/33650723 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f530c06aed0 msgr2=0x7f530c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:10.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.799+0000 7f5333e00700 1 --2- 172.21.15.89:0/33650723 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f530c06aed0 0x7f530c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.799+0000 7f5333e00700 1 -- 172.21.15.89:0/33650723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c0b27f0 msgr2=0x7f532c1e7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:10.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.799+0000 7f5333e00700 1 --2- 172.21.15.89:0/33650723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c0b27f0 0x7f532c1e7b20 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f531800e9b0 tx=0x7f531800ecc0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.799+0000 7f5333e00700 1 -- 172.21.15.89:0/33650723 shutdown_connections 2024-04-02T07:55:10.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.799+0000 7f5333e00700 1 --2- 172.21.15.89:0/33650723 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f530c06aed0 0x7f530c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.799+0000 7f5333e00700 1 --2- 172.21.15.89:0/33650723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f532c160c90 0x7f532c1e8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.799+0000 7f5333e00700 1 --2- 172.21.15.89:0/33650723 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f532c0718e0 0x7f532c147c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.799+0000 7f5333e00700 1 --2- 172.21.15.89:0/33650723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c0b27f0 0x7f532c1e7b20 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:10.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.799+0000 7f5333e00700 1 -- 172.21.15.89:0/33650723 >> 172.21.15.89:0/33650723 conn(0x7f532c0cf190 msgr2=0x7f532c0b1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:10.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.799+0000 7f5333e00700 1 -- 172.21.15.89:0/33650723 shutdown_connections 2024-04-02T07:55:10.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:10.799+0000 7f5333e00700 1 -- 172.21.15.89:0/33650723 wait complete. 2024-04-02T07:55:10.801 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 57 2024-04-02T07:55:10.818 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:55:11.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.023+0000 7f5416600700 1 Processor -- start 2024-04-02T07:55:11.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.023+0000 7f5416600700 1 -- start start 2024-04-02T07:55:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.024+0000 7f5416600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f541015da50 0x7f54100ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.024+0000 7f5416600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f54100d0cf0 0x7f54100b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.024+0000 7f5416600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54100d24e0 con 0x7f54100cec70 2024-04-02T07:55:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.024+0000 7f5416600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54100b95d0 con 0x7f541015da50 2024-04-02T07:55:11.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.024+0000 7f5416600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f541015de30 con 0x7f54100d0cf0 2024-04-02T07:55:11.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.025+0000 7f540f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f541015da50 0x7f54100ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:11.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.025+0000 7f540f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f541015da50 0x7f54100ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47946/0 (socket says 172.21.15.89:47946) 2024-04-02T07:55:11.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.025+0000 7f540f400700 1 -- 172.21.15.89:0/3944729755 learned_addr learned my addr 172.21.15.89:0/3944729755 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:11.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.025+0000 7f540ea00700 1 --1- 172.21.15.89:0/3944729755 >> v1:172.21.15.89:6789/0 conn(0x7f54100cec70 0x7f54100d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59348/0 (socket says 172.21.15.89:59348) 2024-04-02T07:55:11.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.025+0000 7f540fe00700 1 --2- 172.21.15.89:0/3944729755 >> v2:172.21.15.113:3300/0 conn(0x7f54100d0cf0 0x7f54100b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:11.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.025+0000 7f540f400700 1 -- 172.21.15.89:0/3944729755 >> v2:172.21.15.113:3300/0 conn(0x7f54100d0cf0 msgr2=0x7f54100b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:11.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.025+0000 7f540f400700 1 --2- 172.21.15.89:0/3944729755 >> v2:172.21.15.113:3300/0 conn(0x7f54100d0cf0 0x7f54100b6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.025+0000 7f540f400700 1 -- 172.21.15.89:0/3944729755 >> v1:172.21.15.89:6789/0 conn(0x7f54100cec70 legacy=0x7f54100d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:11.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.025+0000 7f540f400700 1 -- 172.21.15.89:0/3944729755 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54100d27c0 con 0x7f541015da50 2024-04-02T07:55:11.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.025+0000 7f540fe00700 1 --2- 172.21.15.89:0/3944729755 >> v2:172.21.15.113:3300/0 conn(0x7f54100d0cf0 0x7f54100b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:11.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.025+0000 7f540f400700 1 --2- 172.21.15.89:0/3944729755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f541015da50 0x7f54100ce730 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f54100d1660 tx=0x7f5400009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3211a7da01ffc9ac server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:11.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.026+0000 7f540e000700 1 -- 172.21.15.89:0/3944729755 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540000b870 con 0x7f541015da50 2024-04-02T07:55:11.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.026+0000 7f540e000700 1 -- 172.21.15.89:0/3944729755 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540000b9d0 con 0x7f541015da50 2024-04-02T07:55:11.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.026+0000 7f540e000700 1 -- 172.21.15.89:0/3944729755 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540000bd60 con 0x7f541015da50 2024-04-02T07:55:11.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.026+0000 7f5416600700 1 -- 172.21.15.89:0/3944729755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f541015da50 msgr2=0x7f54100ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:11.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.026+0000 7f5416600700 1 --2- 172.21.15.89:0/3944729755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f541015da50 0x7f54100ce730 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f54100d1660 tx=0x7f5400009bd0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.026+0000 7f5416600700 1 -- 172.21.15.89:0/3944729755 shutdown_connections 2024-04-02T07:55:11.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.026+0000 7f5416600700 1 --2- 172.21.15.89:0/3944729755 >> v2:172.21.15.113:3300/0 conn(0x7f54100d0cf0 0x7f54100b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.026+0000 7f5416600700 1 --2- 172.21.15.89:0/3944729755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f541015da50 0x7f54100ce730 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.026+0000 7f5416600700 1 -- 172.21.15.89:0/3944729755 >> 172.21.15.89:0/3944729755 conn(0x7f54100666a0 msgr2=0x7f5410066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:11.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.026+0000 7f5416600700 1 -- 172.21.15.89:0/3944729755 shutdown_connections 2024-04-02T07:55:11.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.026+0000 7f5416600700 1 -- 172.21.15.89:0/3944729755 wait complete. 2024-04-02T07:55:11.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.028+0000 7f5416600700 1 Processor -- start 2024-04-02T07:55:11.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.028+0000 7f5416600700 1 -- start start 2024-04-02T07:55:11.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.029+0000 7f5416600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54100cec70 0x7f54101f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.029+0000 7f5416600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54100d0cf0 0x7f5410146440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.029+0000 7f5416600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f541015da50 0x7f54100c9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.029+0000 7f5416600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541015e6e0 con 0x7f54100d0cf0 2024-04-02T07:55:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.029+0000 7f5416600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541015e400 con 0x7f54100cec70 2024-04-02T07:55:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.029+0000 7f5416600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54100cc4a0 con 0x7f541015da50 2024-04-02T07:55:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.029+0000 7f540ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54100d0cf0 0x7f5410146440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.029+0000 7f540ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54100d0cf0 0x7f5410146440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47958/0 (socket says 172.21.15.89:47958) 2024-04-02T07:55:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.029+0000 7f540ea00700 1 -- 172.21.15.89:0/2550031483 learned_addr learned my addr 172.21.15.89:0/2550031483 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.030+0000 7f540f400700 1 --2- 172.21.15.89:0/2550031483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54100cec70 0x7f54101f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.030+0000 7f540fe00700 1 --2- 172.21.15.89:0/2550031483 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f541015da50 0x7f54100c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.030+0000 7f540ea00700 1 -- 172.21.15.89:0/2550031483 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f541015da50 msgr2=0x7f54100c9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.030+0000 7f540ea00700 1 --2- 172.21.15.89:0/2550031483 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f541015da50 0x7f54100c9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.030+0000 7f540ea00700 1 -- 172.21.15.89:0/2550031483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54100cec70 msgr2=0x7f54101f6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:11.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.030+0000 7f540ea00700 1 --2- 172.21.15.89:0/2550031483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54100cec70 0x7f54101f6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.030+0000 7f540ea00700 1 -- 172.21.15.89:0/2550031483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5400009550 con 0x7f54100d0cf0 2024-04-02T07:55:11.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.030+0000 7f540ea00700 1 --2- 172.21.15.89:0/2550031483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54100d0cf0 0x7f5410146440 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00c4f0 tx=0x7f53fc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:11.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.030+0000 7f5407400700 1 -- 172.21.15.89:0/2550031483 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc005cb0 con 0x7f54100d0cf0 2024-04-02T07:55:11.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.031+0000 7f5407400700 1 -- 172.21.15.89:0/2550031483 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc005e10 con 0x7f54100d0cf0 2024-04-02T07:55:11.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.031+0000 7f5416600700 1 -- 172.21.15.89:0/2550031483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54100cc780 con 0x7f54100d0cf0 2024-04-02T07:55:11.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.031+0000 7f5407400700 1 -- 172.21.15.89:0/2550031483 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc0072d0 con 0x7f54100d0cf0 2024-04-02T07:55:11.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.031+0000 7f5416600700 1 -- 172.21.15.89:0/2550031483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54101f68d0 con 0x7f54100d0cf0 2024-04-02T07:55:11.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.032+0000 7f5416600700 1 -- 172.21.15.89:0/2550031483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54100ccc40 con 0x7f54100d0cf0 2024-04-02T07:55:11.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.033+0000 7f5407400700 1 -- 172.21.15.89:0/2550031483 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 73) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f53fc007430 con 0x7f54100d0cf0 2024-04-02T07:55:11.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.034+0000 7f5407400700 1 --2- 172.21.15.89:0/2550031483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f53f006aed0 0x7f53f006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.034+0000 7f5407400700 1 -- 172.21.15.89:0/2550031483 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6457+0+0 (secure 0 0 0) 0x7f53fc088c20 con 0x7f54100d0cf0 2024-04-02T07:55:11.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.034+0000 7f540f400700 1 -- 172.21.15.89:0/2550031483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f53f006aed0 msgr2=0x7f53f006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:11.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.034+0000 7f540f400700 1 --2- 172.21.15.89:0/2550031483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f53f006aed0 0x7f53f006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:11.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.039+0000 7f5407400700 1 -- 172.21.15.89:0/2550031483 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f53fc0538f0 con 0x7f54100d0cf0 2024-04-02T07:55:11.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.111+0000 7f5407400700 1 -- 172.21.15.89:0/2550031483 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 74) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f53fc053cc0 con 0x7f54100d0cf0 2024-04-02T07:55:11.137 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:11.137+0000 7fd8340a5240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:55:11.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.235+0000 7f540f400700 1 -- 172.21.15.89:0/2550031483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f53f006aed0 msgr2=0x7f53f006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:11.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.235+0000 7f540f400700 1 --2- 172.21.15.89:0/2550031483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f53f006aed0 0x7f53f006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:11.252 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:11.252+0000 7fd8340a5240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:55:11.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.265+0000 7f5407400700 1 -- 172.21.15.89:0/2550031483 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 75) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f53fc086d00 con 0x7f54100d0cf0 2024-04-02T07:55:11.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.266+0000 7f5407400700 1 -- 172.21.15.89:0/2550031483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f53f006aed0 msgr2=0x7f53f006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:11.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.266+0000 7f5407400700 1 --2- 172.21.15.89:0/2550031483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f53f006aed0 0x7f53f006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.308+0000 7f5416600700 1 -- 172.21.15.89:0/2550031483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5410067720 con 0x7f54100d0cf0 2024-04-02T07:55:11.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.309+0000 7f5407400700 1 -- 172.21.15.89:0/2550031483 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f53fc054900 con 0x7f54100d0cf0 2024-04-02T07:55:11.310 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:11.310 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1967870793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1967870793},{"type":"v1","addr":"172.21.15.89:6837","nonce":1967870793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1050024225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1050024225},{"type":"v1","addr":"172.21.15.113:6835","nonce":1050024225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7447,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/235320877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":235320877},{"type":"v1","addr":"172.21.15.113:6834","nonce":235320877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:54:30.522457+0000","modified":"2024-04-02T07:54:33.536401+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/1949161483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":1949161483},{"type":"v1","addr":"172.21.15.89:6836","nonce":1949161483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-02T07:55:11.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.311+0000 7f5416600700 1 -- 172.21.15.89:0/2550031483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54100d0cf0 msgr2=0x7f5410146440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:11.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.311+0000 7f5416600700 1 --2- 172.21.15.89:0/2550031483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54100d0cf0 0x7f5410146440 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00c4f0 tx=0x7f53fc00c800 comp rx=0 tx=0).stop 2024-04-02T07:55:11.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.311+0000 7f5416600700 1 -- 172.21.15.89:0/2550031483 shutdown_connections 2024-04-02T07:55:11.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.311+0000 7f5416600700 1 --2- 172.21.15.89:0/2550031483 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f53f006aed0 0x7f53f006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.311+0000 7f5416600700 1 --2- 172.21.15.89:0/2550031483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54100cec70 0x7f54101f6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.311+0000 7f5416600700 1 --2- 172.21.15.89:0/2550031483 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f541015da50 0x7f54100c9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.311+0000 7f5416600700 1 --2- 172.21.15.89:0/2550031483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54100d0cf0 0x7f5410146440 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.311+0000 7f5416600700 1 -- 172.21.15.89:0/2550031483 >> 172.21.15.89:0/2550031483 conn(0x7f54100666a0 msgr2=0x7f54100ce270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:11.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.312+0000 7f5416600700 1 -- 172.21.15.89:0/2550031483 shutdown_connections 2024-04-02T07:55:11.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.312+0000 7f5416600700 1 -- 172.21.15.89:0/2550031483 wait complete. 2024-04-02T07:55:11.313 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 57 2024-04-02T07:55:11.334 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:55:11.339 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:11.335+0000 7fd7a3600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:11.339 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:11.335+0000 7fd7a3600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:11.339 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:11.335+0000 7fd7a3600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:11.339 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:11.335+0000 7fd7a3600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:11.339 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:11.335+0000 7fd7a3600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:11.339 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:11.335+0000 7fd7a7200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:11.339 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:11.335+0000 7fd7a7200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:11.339 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:11.335+0000 7fd7a7200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:11.340 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:11.335+0000 7fd7a7200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:11.340 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:11.335+0000 7fd7a7200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:11.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.525+0000 7f23d6e00700 1 Processor -- start 2024-04-02T07:55:11.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.526+0000 7f23d6e00700 1 -- start start 2024-04-02T07:55:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.527+0000 7f23d6e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f23d00b3af0 0x7f23d015a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.527+0000 7f23d6e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23d0072580 0x7f23d00b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.527+0000 7f23d6e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d00b2180 con 0x7f23d00b2e50 2024-04-02T07:55:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.527+0000 7f23d6e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d015b100 con 0x7f23d0072580 2024-04-02T07:55:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.527+0000 7f23d6e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d015b3e0 con 0x7f23d00b3af0 2024-04-02T07:55:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.527+0000 7f23cf400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23d0072580 0x7f23d00b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.527+0000 7f23cf400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23d0072580 0x7f23d00b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48052/0 (socket says 172.21.15.89:48052) 2024-04-02T07:55:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.527+0000 7f23cf400700 1 -- 172.21.15.89:0/2437703555 learned_addr learned my addr 172.21.15.89:0/2437703555 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:11.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.527+0000 7f23cfe00700 1 --2- 172.21.15.89:0/2437703555 >> v2:172.21.15.113:3300/0 conn(0x7f23d00b3af0 0x7f23d015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:11.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.528+0000 7f23cf400700 1 -- 172.21.15.89:0/2437703555 >> v2:172.21.15.113:3300/0 conn(0x7f23d00b3af0 msgr2=0x7f23d015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:11.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.528+0000 7f23cf400700 1 --2- 172.21.15.89:0/2437703555 >> v2:172.21.15.113:3300/0 conn(0x7f23d00b3af0 0x7f23d015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.528+0000 7f23cf400700 1 -- 172.21.15.89:0/2437703555 >> v1:172.21.15.113:6789/0 conn(0x7f23d00b2e50 legacy=0x7f23d00709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:11.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.528+0000 7f23cf400700 1 -- 172.21.15.89:0/2437703555 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d00b23f0 con 0x7f23d0072580 2024-04-02T07:55:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.528+0000 7f23cf400700 1 --2- 172.21.15.89:0/2437703555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d0072580 0x7f23d00b2910 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f23d006b860 tx=0x7f23bc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee7ed7a7d91ac72 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.528+0000 7f23cea00700 1 -- 172.21.15.89:0/2437703555 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc011830 con 0x7f23d0072580 2024-04-02T07:55:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.528+0000 7f23cea00700 1 -- 172.21.15.89:0/2437703555 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23bc011990 con 0x7f23d0072580 2024-04-02T07:55:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.528+0000 7f23cea00700 1 -- 172.21.15.89:0/2437703555 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc011d20 con 0x7f23d0072580 2024-04-02T07:55:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.529+0000 7f23d6e00700 1 -- 172.21.15.89:0/2437703555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d0072580 msgr2=0x7f23d00b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.529+0000 7f23d6e00700 1 --2- 172.21.15.89:0/2437703555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d0072580 0x7f23d00b2910 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f23d006b860 tx=0x7f23bc009bd0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.529+0000 7f23d6e00700 1 -- 172.21.15.89:0/2437703555 shutdown_connections 2024-04-02T07:55:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.529+0000 7f23d6e00700 1 --2- 172.21.15.89:0/2437703555 >> v2:172.21.15.113:3300/0 conn(0x7f23d00b3af0 0x7f23d015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.529+0000 7f23d6e00700 1 --2- 172.21.15.89:0/2437703555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d0072580 0x7f23d00b2910 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.529+0000 7f23d6e00700 1 -- 172.21.15.89:0/2437703555 >> 172.21.15.89:0/2437703555 conn(0x7f23d00cf190 msgr2=0x7f23d00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.529+0000 7f23d6e00700 1 -- 172.21.15.89:0/2437703555 shutdown_connections 2024-04-02T07:55:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.529+0000 7f23d6e00700 1 -- 172.21.15.89:0/2437703555 wait complete. 2024-04-02T07:55:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.531+0000 7f23d6e00700 1 Processor -- start 2024-04-02T07:55:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.531+0000 7f23d6e00700 1 -- start start 2024-04-02T07:55:11.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23d6e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d0072580 0x7f23d014e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23d6e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d00b2e50 0x7f23d01f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23d6e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d00b3af0 0x7f23d01f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23d6e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d01598a0 con 0x7f23d00b2e50 2024-04-02T07:55:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23d6e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d0159b80 con 0x7f23d0072580 2024-04-02T07:55:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23d6e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d0159e60 con 0x7f23d00b3af0 2024-04-02T07:55:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23cf400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d00b2e50 0x7f23d01f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23cf400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d00b2e50 0x7f23d01f4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48056/0 (socket says 172.21.15.89:48056) 2024-04-02T07:55:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23cf400700 1 -- 172.21.15.89:0/3774574183 learned_addr learned my addr 172.21.15.89:0/3774574183 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23cfe00700 1 --2- 172.21.15.89:0/3774574183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d0072580 0x7f23d014e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23d4e00700 1 --2- 172.21.15.89:0/3774574183 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d00b3af0 0x7f23d01f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23cf400700 1 -- 172.21.15.89:0/3774574183 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d00b3af0 msgr2=0x7f23d01f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23cf400700 1 --2- 172.21.15.89:0/3774574183 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d00b3af0 0x7f23d01f3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23cf400700 1 -- 172.21.15.89:0/3774574183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d0072580 msgr2=0x7f23d014e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23cf400700 1 --2- 172.21.15.89:0/3774574183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d0072580 0x7f23d014e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23cf400700 1 -- 172.21.15.89:0/3774574183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc009550 con 0x7f23d00b2e50 2024-04-02T07:55:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.532+0000 7f23d4e00700 1 --2- 172.21.15.89:0/3774574183 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d00b3af0 0x7f23d01f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:55:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.533+0000 7f23cf400700 1 --2- 172.21.15.89:0/3774574183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d00b2e50 0x7f23d01f4760 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f23bc000c00 tx=0x7f23bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.533+0000 7f23c7e00700 1 -- 172.21.15.89:0/3774574183 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc004570 con 0x7f23d00b2e50 2024-04-02T07:55:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.533+0000 7f23c7e00700 1 -- 172.21.15.89:0/3774574183 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23bc01f070 con 0x7f23d00b2e50 2024-04-02T07:55:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.533+0000 7f23d6e00700 1 -- 172.21.15.89:0/3774574183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d00cbba0 con 0x7f23d00b2e50 2024-04-02T07:55:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.533+0000 7f23c7e00700 1 -- 172.21.15.89:0/3774574183 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc0038e0 con 0x7f23d00b2e50 2024-04-02T07:55:11.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.533+0000 7f23d6e00700 1 -- 172.21.15.89:0/3774574183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d01f4ca0 con 0x7f23d00b2e50 2024-04-02T07:55:11.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.534+0000 7f23d6e00700 1 -- 172.21.15.89:0/3774574183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d0067720 con 0x7f23d00b2e50 2024-04-02T07:55:11.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.535+0000 7f23c7e00700 1 -- 172.21.15.89:0/3774574183 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 75) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f23bc021030 con 0x7f23d00b2e50 2024-04-02T07:55:11.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.536+0000 7f23c7e00700 1 -- 172.21.15.89:0/3774574183 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6773+0+0 (secure 0 0 0) 0x7f23bc08a650 con 0x7f23d00b2e50 2024-04-02T07:55:11.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.540+0000 7f23c7e00700 1 -- 172.21.15.89:0/3774574183 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f23bc055970 con 0x7f23d00b2e50 2024-04-02T07:55:11.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.743+0000 7f23d6e00700 1 -- 172.21.15.89:0/3774574183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23d00cef70 con 0x7f23d00b2e50 2024-04-02T07:55:11.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.744+0000 7f23c7e00700 1 -- 172.21.15.89:0/3774574183 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+17624 (secure 0 0 0) 0x7f23bc015090 con 0x7f23d00b2e50 2024-04-02T07:55:11.745 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:11.745 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":128,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:55:11.254729+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:54:28.130987+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-02T07:54:29.515956+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":127,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/1098548839":"2024-04-03T07:55:11.254669+0000","172.21.15.89:0/4283832910":"2024-04-03T07:55:11.254669+0000","172.21.15.89:0/61382477":"2024-04-03T07:55:11.254669+0000","172.21.15.89:0/933659199":"2024-04-03T07:55:11.254669+0000","172.21.15.89:0/2142404772":"2024-04-03T07:55:11.254669+0000","172.21.15.89:0/3703296012":"2024-04-03T07:55:11.254669+0000","172.21.15.89:0/49649361":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/51116008":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/236471161":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/709589358":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/3908822617":"2024-04-03T07:55:11.254669+0000","172.21.15.89:0/1096941421":"2024-04-03T07:54:46.836267+0000","172.21.15.89:6833/43357":"2024-04-03T07:54:46.836267+0000","172.21.15.89:6832/43357":"2024-04-03T07:54:46.836267+0000","172.21.15.89:0/4022787599":"2024-04-03T07:55:11.254669+0000","172.21.15.89:0/3407101260":"2024-04-03T07:54:46.836267+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:55:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.747+0000 7f23d6e00700 1 -- 172.21.15.89:0/3774574183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d00b2e50 msgr2=0x7f23d01f4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.747+0000 7f23d6e00700 1 --2- 172.21.15.89:0/3774574183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d00b2e50 0x7f23d01f4760 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f23bc000c00 tx=0x7f23bc004750 comp rx=0 tx=0).stop 2024-04-02T07:55:11.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.747+0000 7f23d6e00700 1 -- 172.21.15.89:0/3774574183 shutdown_connections 2024-04-02T07:55:11.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.747+0000 7f23d6e00700 1 --2- 172.21.15.89:0/3774574183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23d0072580 0x7f23d014e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.747+0000 7f23d6e00700 1 --2- 172.21.15.89:0/3774574183 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23d00b3af0 0x7f23d01f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.747+0000 7f23d6e00700 1 --2- 172.21.15.89:0/3774574183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23d00b2e50 0x7f23d01f4760 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.747+0000 7f23d6e00700 1 -- 172.21.15.89:0/3774574183 >> 172.21.15.89:0/3774574183 conn(0x7f23d00cf190 msgr2=0x7f23d015dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:11.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.747+0000 7f23d6e00700 1 -- 172.21.15.89:0/3774574183 shutdown_connections 2024-04-02T07:55:11.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.747+0000 7f23d6e00700 1 -- 172.21.15.89:0/3774574183 wait complete. 2024-04-02T07:55:11.765 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T07:55:11.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.956+0000 7f1c6ae00700 1 Processor -- start 2024-04-02T07:55:11.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.956+0000 7f1c6ae00700 1 -- start start 2024-04-02T07:55:11.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.957+0000 7f1c6ae00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1c640cbb30 0x7f1c640cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.957+0000 7f1c6ae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c640cd480 0x7f1c6406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.957+0000 7f1c6ae00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c640cc450 con 0x7f1c640cd020 2024-04-02T07:55:11.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.957+0000 7f1c6ae00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c640cdae0 con 0x7f1c640cbb30 2024-04-02T07:55:11.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.957+0000 7f1c6ae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c640cddc0 con 0x7f1c640cd480 2024-04-02T07:55:11.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.957+0000 7f1c63e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1c640cbb30 0x7f1c640cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:11.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.957+0000 7f1c63e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1c640cbb30 0x7f1c640cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48058/0 (socket says 172.21.15.89:48058) 2024-04-02T07:55:11.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.957+0000 7f1c63e00700 1 -- 172.21.15.89:0/288323479 learned_addr learned my addr 172.21.15.89:0/288323479 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:11.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.957+0000 7f1c68e00700 1 --2- 172.21.15.89:0/288323479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c640cd480 0x7f1c6406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:11.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.958+0000 7f1c63e00700 1 -- 172.21.15.89:0/288323479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c640cd480 msgr2=0x7f1c6406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:11.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.958+0000 7f1c63e00700 1 --2- 172.21.15.89:0/288323479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c640cd480 0x7f1c6406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.958+0000 7f1c63e00700 1 -- 172.21.15.89:0/288323479 >> v1:172.21.15.113:6789/0 conn(0x7f1c640cd020 legacy=0x7f1c640b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:11.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.958+0000 7f1c63e00700 1 -- 172.21.15.89:0/288323479 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640ce0a0 con 0x7f1c640cbb30 2024-04-02T07:55:11.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.958+0000 7f1c63e00700 1 --2- 172.21.15.89:0/288323479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c640cbb30 0x7f1c640cbf10 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f1c540098f0 tx=0x7f1c54009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4176345468f36606 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:11.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.958+0000 7f1c62a00700 1 -- 172.21.15.89:0/288323479 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5400b8a0 con 0x7f1c640cbb30 2024-04-02T07:55:11.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.958+0000 7f1c62a00700 1 -- 172.21.15.89:0/288323479 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5400ba00 con 0x7f1c640cbb30 2024-04-02T07:55:11.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.958+0000 7f1c62a00700 1 -- 172.21.15.89:0/288323479 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5400bd90 con 0x7f1c640cbb30 2024-04-02T07:55:11.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.959+0000 7f1c6ae00700 1 -- 172.21.15.89:0/288323479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c640cbb30 msgr2=0x7f1c640cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:11.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.959+0000 7f1c6ae00700 1 --2- 172.21.15.89:0/288323479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c640cbb30 0x7f1c640cbf10 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f1c540098f0 tx=0x7f1c54009c00 comp rx=0 tx=0).stop 2024-04-02T07:55:11.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.959+0000 7f1c6ae00700 1 -- 172.21.15.89:0/288323479 shutdown_connections 2024-04-02T07:55:11.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.959+0000 7f1c6ae00700 1 --2- 172.21.15.89:0/288323479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c640cd480 0x7f1c6406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.959+0000 7f1c6ae00700 1 --2- 172.21.15.89:0/288323479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c640cbb30 0x7f1c640cbf10 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.959+0000 7f1c6ae00700 1 -- 172.21.15.89:0/288323479 >> 172.21.15.89:0/288323479 conn(0x7f1c640b0d70 msgr2=0x7f1c640b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:11.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.959+0000 7f1c6ae00700 1 -- 172.21.15.89:0/288323479 shutdown_connections 2024-04-02T07:55:11.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.959+0000 7f1c6ae00700 1 -- 172.21.15.89:0/288323479 wait complete. 2024-04-02T07:55:11.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.960+0000 7f1c6ae00700 1 Processor -- start 2024-04-02T07:55:11.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.961+0000 7f1c6ae00700 1 -- start start 2024-04-02T07:55:11.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c6ae00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c640cbb30 0x7f1c640c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c6ae00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c640cd020 0x7f1c6414c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c6ae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c640cd480 0x7f1c640c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:11.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c6ae00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c64156cf0 con 0x7f1c640cbb30 2024-04-02T07:55:11.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c6ae00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c64071010 con 0x7f1c640cd020 2024-04-02T07:55:11.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c6ae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c640cb550 con 0x7f1c640cd480 2024-04-02T07:55:11.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c63e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c640cbb30 0x7f1c640c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:11.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c63e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c640cbb30 0x7f1c640c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48070/0 (socket says 172.21.15.89:48070) 2024-04-02T07:55:11.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c63e00700 1 -- 172.21.15.89:0/1578351637 learned_addr learned my addr 172.21.15.89:0/1578351637 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:11.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c63e00700 1 -- 172.21.15.89:0/1578351637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c640cd480 msgr2=0x7f1c640c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:11.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c63e00700 1 --2- 172.21.15.89:0/1578351637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c640cd480 0x7f1c640c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c63e00700 1 -- 172.21.15.89:0/1578351637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c640cd020 msgr2=0x7f1c6414c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:11.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c63e00700 1 --2- 172.21.15.89:0/1578351637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c640cd020 0x7f1c6414c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:11.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c63e00700 1 -- 172.21.15.89:0/1578351637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c54009550 con 0x7f1c640cbb30 2024-04-02T07:55:11.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.962+0000 7f1c63e00700 1 --2- 172.21.15.89:0/1578351637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c640cbb30 0x7f1c640c83f0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f1c54009c40 tx=0x7f1c54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:11.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.963+0000 7f1c5be00700 1 -- 172.21.15.89:0/1578351637 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c540036a0 con 0x7f1c640cbb30 2024-04-02T07:55:11.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.963+0000 7f1c5be00700 1 -- 172.21.15.89:0/1578351637 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5401e070 con 0x7f1c640cbb30 2024-04-02T07:55:11.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.963+0000 7f1c5be00700 1 -- 172.21.15.89:0/1578351637 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c54003a60 con 0x7f1c640cbb30 2024-04-02T07:55:11.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.963+0000 7f1c6ae00700 1 -- 172.21.15.89:0/1578351637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c640cb830 con 0x7f1c640cbb30 2024-04-02T07:55:11.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.963+0000 7f1c6ae00700 1 -- 172.21.15.89:0/1578351637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c64070ca0 con 0x7f1c640cbb30 2024-04-02T07:55:11.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.964+0000 7f1c6ae00700 1 -- 172.21.15.89:0/1578351637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c64067720 con 0x7f1c640cbb30 2024-04-02T07:55:11.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.965+0000 7f1c5be00700 1 -- 172.21.15.89:0/1578351637 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 75) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f1c54003bc0 con 0x7f1c640cbb30 2024-04-02T07:55:11.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.965+0000 7f1c5be00700 1 -- 172.21.15.89:0/1578351637 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6773+0+0 (secure 0 0 0) 0x7f1c54020070 con 0x7f1c640cbb30 2024-04-02T07:55:11.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:11.970+0000 7f1c5be00700 1 -- 172.21.15.89:0/1578351637 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1c540558d0 con 0x7f1c640cbb30 2024-04-02T07:55:12.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.220+0000 7f1c6ae00700 1 -- 172.21.15.89:0/1578351637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1c6415bda0 con 0x7f1c640cbb30 2024-04-02T07:55:12.221 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T07:55:12.221+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T07:55:12.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.230+0000 7f1c5be00700 1 -- 172.21.15.89:0/1578351637 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v58) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1c54015090 con 0x7f1c640cbb30 2024-04-02T07:55:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.234+0000 7f1c6ae00700 1 -- 172.21.15.89:0/1578351637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c640cbb30 msgr2=0x7f1c640c83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.234+0000 7f1c6ae00700 1 --2- 172.21.15.89:0/1578351637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c640cbb30 0x7f1c640c83f0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f1c54009c40 tx=0x7f1c54004750 comp rx=0 tx=0).stop 2024-04-02T07:55:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.234+0000 7f1c6ae00700 1 -- 172.21.15.89:0/1578351637 shutdown_connections 2024-04-02T07:55:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.234+0000 7f1c6ae00700 1 --2- 172.21.15.89:0/1578351637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1c640cd480 0x7f1c640c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.234+0000 7f1c6ae00700 1 --2- 172.21.15.89:0/1578351637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c640cbb30 0x7f1c640c83f0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.234+0000 7f1c6ae00700 1 --2- 172.21.15.89:0/1578351637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c640cd020 0x7f1c6414c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.234+0000 7f1c6ae00700 1 -- 172.21.15.89:0/1578351637 >> 172.21.15.89:0/1578351637 conn(0x7f1c640b0d70 msgr2=0x7f1c6415ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.234+0000 7f1c6ae00700 1 -- 172.21.15.89:0/1578351637 shutdown_connections 2024-04-02T07:55:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.234+0000 7f1c6ae00700 1 -- 172.21.15.89:0/1578351637 wait complete. 2024-04-02T07:55:12.236 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T07:55:12.245 INFO:tasks.ceph.mds.c.smithi089.stderr: -7658> 2024-04-02T07:54:35.742+0000 7f3b57e00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T07:55:12.254 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T07:55:12.305 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T07:55:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.456+0000 7fec34200700 1 Processor -- start 2024-04-02T07:55:12.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.456+0000 7fec34200700 1 -- start start 2024-04-02T07:55:12.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.457+0000 7fec34200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2c0d16a0 0x7fec2c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:12.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.458+0000 7fec34200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c161420 con 0x7fec2c15d9d0 2024-04-02T07:55:12.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.458+0000 7fec34200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c161700 con 0x7fec2c0d2370 2024-04-02T07:55:12.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.458+0000 7fec34200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c0d2820 con 0x7fec2c0d16a0 2024-04-02T07:55:12.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.458+0000 7fec31800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fec2c15d9d0 0x7fec2c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59380/0 (socket says 172.21.15.89:59380) 2024-04-02T07:55:12.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.458+0000 7fec31800700 1 -- 172.21.15.89:0/1169197602 learned_addr learned my addr 172.21.15.89:0/1169197602 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:12.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.458+0000 7fec30e00700 1 --2- 172.21.15.89:0/1169197602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2c0d16a0 0x7fec2c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:12.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.458+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746305660 0 0) 0x7fec2c161420 con 0x7fec2c15d9d0 2024-04-02T07:55:12.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.458+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec140035b0 con 0x7fec2c15d9d0 2024-04-02T07:55:12.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.458+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031011299 0 0) 0x7fec2c161700 con 0x7fec2c0d2370 2024-04-02T07:55:12.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.458+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec2c161420 con 0x7fec2c0d2370 2024-04-02T07:55:12.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.458+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1661195835 0 0) 0x7fec140035b0 con 0x7fec2c15d9d0 2024-04-02T07:55:12.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.458+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2c0d16a0 msgr2=0x7fec2c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:12.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.458+0000 7fec2be00700 1 --2- 172.21.15.89:0/1169197602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2c0d16a0 0x7fec2c0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:12.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 >> v1:172.21.15.113:6789/0 conn(0x7fec2c0d2370 legacy=0x7fec2c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:12.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c1619e0 con 0x7fec2c15d9d0 2024-04-02T07:55:12.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fec20001670 con 0x7fec2c15d9d0 2024-04-02T07:55:12.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 >> v1:172.21.15.89:6789/0 conn(0x7fec2c15d9d0 legacy=0x7fec2c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:12.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec2be00700 1 --2- 172.21.15.89:0/1169197602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec14003bc0 0x7fec14003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:12.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec2be00700 1 --2- 172.21.15.89:0/1169197602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2c0d16a0 0x7fec14004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:12.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec2be00700 1 --2- 172.21.15.89:0/1169197602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec14004d90 0x7fec14005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:12.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c0d2820 con 0x7fec14003bc0 2024-04-02T07:55:12.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1400bac0 con 0x7fec14004d90 2024-04-02T07:55:12.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1400bc30 con 0x7fec2c0d16a0 2024-04-02T07:55:12.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec31800700 1 --2- 172.21.15.89:0/1169197602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec14003bc0 0x7fec14003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:12.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec30e00700 1 --2- 172.21.15.89:0/1169197602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2c0d16a0 0x7fec14004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:12.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec32200700 1 --2- 172.21.15.89:0/1169197602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec14004d90 0x7fec14005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:12.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec31800700 1 -- 172.21.15.89:0/1169197602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2c0d16a0 msgr2=0x7fec14004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:12.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec31800700 1 --2- 172.21.15.89:0/1169197602 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2c0d16a0 0x7fec14004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:12.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec31800700 1 -- 172.21.15.89:0/1169197602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec14004d90 msgr2=0x7fec14005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:12.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec31800700 1 --2- 172.21.15.89:0/1169197602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec14004d90 0x7fec14005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:12.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.459+0000 7fec31800700 1 -- 172.21.15.89:0/1169197602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec140035b0 con 0x7fec14003bc0 2024-04-02T07:55:12.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.460+0000 7fec31800700 1 --2- 172.21.15.89:0/1169197602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec14003bc0 0x7fec14003fc0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fec2000c210 tx=0x7fec14005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bccef6dd7c298363 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:12.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.460+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2000cd50 con 0x7fec14003bc0 2024-04-02T07:55:12.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.460+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec20005960 con 0x7fec14003bc0 2024-04-02T07:55:12.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.460+0000 7fec2be00700 1 -- 172.21.15.89:0/1169197602 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec20005d10 con 0x7fec14003bc0 2024-04-02T07:55:12.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.460+0000 7fec34200700 1 -- 172.21.15.89:0/1169197602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec14003bc0 msgr2=0x7fec14003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.460+0000 7fec34200700 1 --2- 172.21.15.89:0/1169197602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec14003bc0 0x7fec14003fc0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fec2000c210 tx=0x7fec14005880 comp rx=0 tx=0).stop 2024-04-02T07:55:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.460+0000 7fec32200700 1 -- 172.21.15.89:0/1169197602 reap_dead start 2024-04-02T07:55:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.460+0000 7fec34200700 1 -- 172.21.15.89:0/1169197602 shutdown_connections 2024-04-02T07:55:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.460+0000 7fec34200700 1 -- 172.21.15.89:0/1169197602 >> 172.21.15.89:0/1169197602 conn(0x7fec2c08aff0 msgr2=0x7fec2c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.461+0000 7fec34200700 1 -- 172.21.15.89:0/1169197602 shutdown_connections 2024-04-02T07:55:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.461+0000 7fec34200700 1 -- 172.21.15.89:0/1169197602 wait complete. 2024-04-02T07:55:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.462+0000 7fec34200700 1 Processor -- start 2024-04-02T07:55:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.462+0000 7fec34200700 1 -- start start 2024-04-02T07:55:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.463+0000 7fec34200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec14003bc0 0x7fec2c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.463+0000 7fec34200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec2c15f730 0x7fec2c0b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.463+0000 7fec34200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2c0cd150 0x7fec2c1585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.463+0000 7fec34200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c15e690 con 0x7fec14003bc0 2024-04-02T07:55:12.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.463+0000 7fec34200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c15e970 con 0x7fec2c15f730 2024-04-02T07:55:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.463+0000 7fec34200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c15ec50 con 0x7fec2c0cd150 2024-04-02T07:55:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.464+0000 7fec31800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec14003bc0 0x7fec2c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.464+0000 7fec31800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec14003bc0 0x7fec2c15ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48100/0 (socket says 172.21.15.89:48100) 2024-04-02T07:55:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.464+0000 7fec31800700 1 -- 172.21.15.89:0/4227154763 learned_addr learned my addr 172.21.15.89:0/4227154763 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.464+0000 7fec32200700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2c0cd150 0x7fec2c1585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.464+0000 7fec30e00700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec2c15f730 0x7fec2c0b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.464+0000 7fec31800700 1 -- 172.21.15.89:0/4227154763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2c0cd150 msgr2=0x7fec2c1585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.464+0000 7fec31800700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2c0cd150 0x7fec2c1585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.464+0000 7fec31800700 1 -- 172.21.15.89:0/4227154763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec2c15f730 msgr2=0x7fec2c0b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.464+0000 7fec31800700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec2c15f730 0x7fec2c0b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.464+0000 7fec31800700 1 -- 172.21.15.89:0/4227154763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c15ef30 con 0x7fec14003bc0 2024-04-02T07:55:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.464+0000 7fec32200700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2c0cd150 0x7fec2c1585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:12.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.464+0000 7fec30e00700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec2c15f730 0x7fec2c0b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.464+0000 7fec31800700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec14003bc0 0x7fec2c15ddb0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fec14005830 tx=0x7fec20001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.465+0000 7fec29600700 1 -- 172.21.15.89:0/4227154763 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec20004c80 con 0x7fec14003bc0 2024-04-02T07:55:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.465+0000 7fec29600700 1 -- 172.21.15.89:0/4227154763 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec20004de0 con 0x7fec14003bc0 2024-04-02T07:55:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.465+0000 7fec29600700 1 -- 172.21.15.89:0/4227154763 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec20007310 con 0x7fec14003bc0 2024-04-02T07:55:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.465+0000 7fec34200700 1 -- 172.21.15.89:0/4227154763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c15f210 con 0x7fec14003bc0 2024-04-02T07:55:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.465+0000 7fec34200700 1 -- 172.21.15.89:0/4227154763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2c0b4b40 con 0x7fec14003bc0 2024-04-02T07:55:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.466+0000 7fec34200700 1 -- 172.21.15.89:0/4227154763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2c0b4d30 con 0x7fec14003bc0 2024-04-02T07:55:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.466+0000 7fec29600700 1 -- 172.21.15.89:0/4227154763 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 76) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fec20007470 con 0x7fec14003bc0 2024-04-02T07:55:12.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.467+0000 7fec29600700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fec0c06aae0 0x7fec0c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:12.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.467+0000 7fec29600700 1 -- 172.21.15.89:0/4227154763 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fec2008a430 con 0x7fec14003bc0 2024-04-02T07:55:12.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.468+0000 7fec30e00700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fec0c06aae0 0x7fec0c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:12.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.468+0000 7fec30e00700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fec0c06aae0 0x7fec0c06cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fec2c0c9f70 tx=0x7fec18008040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:12.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.472+0000 7fec29600700 1 -- 172.21.15.89:0/4227154763 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fec20055450 con 0x7fec14003bc0 2024-04-02T07:55:12.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:12.715+0000 7fec34200700 1 -- 172.21.15.89:0/4227154763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fec2c067720 con 0x7fec14003bc0 2024-04-02T07:55:13.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.271+0000 7fec29600700 1 -- 172.21.15.89:0/4227154763 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v59) v1 ==== 104+0+0 (secure 0 0 0) 0x7fec2005de60 con 0x7fec14003bc0 2024-04-02T07:55:13.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.274+0000 7fec34200700 1 -- 172.21.15.89:0/4227154763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fec0c06aae0 msgr2=0x7fec0c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:13.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.274+0000 7fec34200700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fec0c06aae0 0x7fec0c06cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fec2c0c9f70 tx=0x7fec18008040 comp rx=0 tx=0).stop 2024-04-02T07:55:13.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.274+0000 7fec34200700 1 -- 172.21.15.89:0/4227154763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec14003bc0 msgr2=0x7fec2c15ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:13.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.274+0000 7fec34200700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec14003bc0 0x7fec2c15ddb0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fec14005830 tx=0x7fec20001980 comp rx=0 tx=0).stop 2024-04-02T07:55:13.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.274+0000 7fec34200700 1 -- 172.21.15.89:0/4227154763 shutdown_connections 2024-04-02T07:55:13.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.274+0000 7fec34200700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fec0c06aae0 0x7fec0c06cfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:13.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.274+0000 7fec34200700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2c0cd150 0x7fec2c1585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:13.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.274+0000 7fec34200700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec14003bc0 0x7fec2c15ddb0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:13.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.274+0000 7fec34200700 1 --2- 172.21.15.89:0/4227154763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec2c15f730 0x7fec2c0b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:13.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.274+0000 7fec34200700 1 -- 172.21.15.89:0/4227154763 >> 172.21.15.89:0/4227154763 conn(0x7fec2c08aff0 msgr2=0x7fec2c15ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:13.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.274+0000 7fec34200700 1 -- 172.21.15.89:0/4227154763 shutdown_connections 2024-04-02T07:55:13.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.274+0000 7fec34200700 1 -- 172.21.15.89:0/4227154763 wait complete. 2024-04-02T07:55:13.293 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T07:55:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.493+0000 7f8a59e00700 1 Processor -- start 2024-04-02T07:55:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.493+0000 7f8a59e00700 1 -- start start 2024-04-02T07:55:13.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.494+0000 7f8a59e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a5415d760 0x7f8a540b8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:13.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.494+0000 7f8a59e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a540d30d0 con 0x7f8a540d2c50 2024-04-02T07:55:13.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.494+0000 7f8a59e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a540d1bb0 con 0x7f8a540ce7a0 2024-04-02T07:55:13.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.494+0000 7f8a59e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540d1e90 con 0x7f8a5415d760 2024-04-02T07:55:13.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.495+0000 7f8a53e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8a540d2c50 0x7f8a540ce090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59386/0 (socket says 172.21.15.89:59386) 2024-04-02T07:55:13.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.495+0000 7f8a53e00700 1 -- 172.21.15.89:0/1039146775 learned_addr learned my addr 172.21.15.89:0/1039146775 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:13.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.495+0000 7f8a58a00700 1 --2- 172.21.15.89:0/1039146775 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a5415d760 0x7f8a540b8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:13.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.495+0000 7f8a53400700 1 -- 172.21.15.89:0/1039146775 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540759714 0 0) 0x7f8a540d30d0 con 0x7f8a540d2c50 2024-04-02T07:55:13.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.495+0000 7f8a53400700 1 -- 172.21.15.89:0/1039146775 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a3c0035b0 con 0x7f8a540d2c50 2024-04-02T07:55:13.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.495+0000 7f8a53400700 1 -- 172.21.15.89:0/1039146775 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010651958 0 0) 0x7f8a540d1bb0 con 0x7f8a540ce7a0 2024-04-02T07:55:13.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.495+0000 7f8a58a00700 1 -- 172.21.15.89:0/1039146775 >> v1:172.21.15.113:6789/0 conn(0x7f8a540ce7a0 legacy=0x7f8a540d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:13.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.495+0000 7f8a58a00700 1 -- 172.21.15.89:0/1039146775 >> v1:172.21.15.89:6789/0 conn(0x7f8a540d2c50 legacy=0x7f8a540ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:13.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.495+0000 7f8a58a00700 1 -- 172.21.15.89:0/1039146775 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a540d2170 con 0x7f8a5415d760 2024-04-02T07:55:13.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.496+0000 7f8a58a00700 1 --2- 172.21.15.89:0/1039146775 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a5415d760 0x7f8a540b8ea0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8a48009870 tx=0x7f8a48009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70cef09f80f4e890 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:13.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.496+0000 7f8a53400700 1 -- 172.21.15.89:0/1039146775 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a48011830 con 0x7f8a5415d760 2024-04-02T07:55:13.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.496+0000 7f8a53400700 1 -- 172.21.15.89:0/1039146775 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a48011990 con 0x7f8a5415d760 2024-04-02T07:55:13.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.496+0000 7f8a53400700 1 -- 172.21.15.89:0/1039146775 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a48011d20 con 0x7f8a5415d760 2024-04-02T07:55:13.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.496+0000 7f8a59e00700 1 -- 172.21.15.89:0/1039146775 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a5415d760 msgr2=0x7f8a540b8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:13.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.496+0000 7f8a59e00700 1 --2- 172.21.15.89:0/1039146775 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a5415d760 0x7f8a540b8ea0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8a48009870 tx=0x7f8a48009b80 comp rx=0 tx=0).stop 2024-04-02T07:55:13.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.496+0000 7f8a59e00700 1 -- 172.21.15.89:0/1039146775 shutdown_connections 2024-04-02T07:55:13.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.496+0000 7f8a59e00700 1 --2- 172.21.15.89:0/1039146775 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a5415d760 0x7f8a540b8ea0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:13.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.496+0000 7f8a59e00700 1 -- 172.21.15.89:0/1039146775 >> 172.21.15.89:0/1039146775 conn(0x7f8a540668e0 msgr2=0x7f8a54066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:13.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.497+0000 7f8a59e00700 1 -- 172.21.15.89:0/1039146775 shutdown_connections 2024-04-02T07:55:13.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.497+0000 7f8a59e00700 1 -- 172.21.15.89:0/1039146775 wait complete. 2024-04-02T07:55:13.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.498+0000 7f8a59e00700 1 Processor -- start 2024-04-02T07:55:13.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.498+0000 7f8a59e00700 1 -- start start 2024-04-02T07:55:13.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.499+0000 7f8a59e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a540ce7a0 0x7f8a54151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:13.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.499+0000 7f8a59e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a540d2c50 0x7f8a540c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:13.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.499+0000 7f8a59e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a5415d760 0x7f8a5415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:13.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.499+0000 7f8a59e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540c9d40 con 0x7f8a5415d760 2024-04-02T07:55:13.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.499+0000 7f8a59e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540cac10 con 0x7f8a540d2c50 2024-04-02T07:55:13.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.499+0000 7f8a59e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a54152510 con 0x7f8a540ce7a0 2024-04-02T07:55:13.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.500+0000 7f8a59400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a5415d760 0x7f8a5415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:13.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.500+0000 7f8a59400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a5415d760 0x7f8a5415ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48128/0 (socket says 172.21.15.89:48128) 2024-04-02T07:55:13.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.500+0000 7f8a59400700 1 -- 172.21.15.89:0/2000073506 learned_addr learned my addr 172.21.15.89:0/2000073506 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:13.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.500+0000 7f8a53e00700 1 --2- 172.21.15.89:0/2000073506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a540d2c50 0x7f8a540c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:13.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.500+0000 7f8a58a00700 1 --2- 172.21.15.89:0/2000073506 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a540ce7a0 0x7f8a54151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:13.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.500+0000 7f8a59400700 1 -- 172.21.15.89:0/2000073506 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a540ce7a0 msgr2=0x7f8a54151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:13.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.500+0000 7f8a59400700 1 --2- 172.21.15.89:0/2000073506 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a540ce7a0 0x7f8a54151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:13.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.500+0000 7f8a59400700 1 -- 172.21.15.89:0/2000073506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a540d2c50 msgr2=0x7f8a540c9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:13.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.500+0000 7f8a59400700 1 --2- 172.21.15.89:0/2000073506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a540d2c50 0x7f8a540c9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:13.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.500+0000 7f8a59400700 1 -- 172.21.15.89:0/2000073506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a480094a0 con 0x7f8a5415d760 2024-04-02T07:55:13.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.500+0000 7f8a59400700 1 --2- 172.21.15.89:0/2000073506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a5415d760 0x7f8a5415ac00 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c001560 tx=0x7f8a4c00cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:13.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.500+0000 7f8a50c00700 1 -- 172.21.15.89:0/2000073506 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c003e30 con 0x7f8a5415d760 2024-04-02T07:55:13.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.500+0000 7f8a50c00700 1 -- 172.21.15.89:0/2000073506 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4c005070 con 0x7f8a5415d760 2024-04-02T07:55:13.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.500+0000 7f8a50c00700 1 -- 172.21.15.89:0/2000073506 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c005420 con 0x7f8a5415d760 2024-04-02T07:55:13.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.501+0000 7f8a59e00700 1 -- 172.21.15.89:0/2000073506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a540c9a60 con 0x7f8a5415d760 2024-04-02T07:55:13.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.501+0000 7f8a59e00700 1 -- 172.21.15.89:0/2000073506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a540ca8a0 con 0x7f8a5415d760 2024-04-02T07:55:13.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.502+0000 7f8a59e00700 1 -- 172.21.15.89:0/2000073506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a540ca420 con 0x7f8a5415d760 2024-04-02T07:55:13.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.503+0000 7f8a50c00700 1 -- 172.21.15.89:0/2000073506 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 76) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f8a4c005580 con 0x7f8a5415d760 2024-04-02T07:55:13.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.503+0000 7f8a50c00700 1 --2- 172.21.15.89:0/2000073506 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a3406aae0 0x7f8a3406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:13.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.503+0000 7f8a50c00700 1 -- 172.21.15.89:0/2000073506 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f8a4c088ac0 con 0x7f8a5415d760 2024-04-02T07:55:13.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.504+0000 7f8a58a00700 1 --2- 172.21.15.89:0/2000073506 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a3406aae0 0x7f8a3406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:13.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.504+0000 7f8a58a00700 1 --2- 172.21.15.89:0/2000073506 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a3406aae0 0x7f8a3406cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8a480097c0 tx=0x7f8a4801c040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:13.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.508+0000 7f8a50c00700 1 -- 172.21.15.89:0/2000073506 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8a540ca420 con 0x7f8a5415d760 2024-04-02T07:55:13.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:13.710+0000 7f8a59e00700 1 -- 172.21.15.89:0/2000073506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8a540ca420 con 0x7f8a5415d760 2024-04-02T07:55:14.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.119+0000 7f8a50c00700 1 -- 172.21.15.89:0/2000073506 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 77) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f8a4c053e60 con 0x7f8a5415d760 2024-04-02T07:55:14.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.279+0000 7f8a50c00700 1 -- 172.21.15.89:0/2000073506 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v130) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8a4c0737c0 con 0x7f8a5415d760 2024-04-02T07:55:14.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.282+0000 7f8a59e00700 1 -- 172.21.15.89:0/2000073506 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a3406aae0 msgr2=0x7f8a3406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:14.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.282+0000 7f8a59e00700 1 --2- 172.21.15.89:0/2000073506 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a3406aae0 0x7f8a3406cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8a480097c0 tx=0x7f8a4801c040 comp rx=0 tx=0).stop 2024-04-02T07:55:14.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.282+0000 7f8a59e00700 1 -- 172.21.15.89:0/2000073506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a5415d760 msgr2=0x7f8a5415ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:14.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.282+0000 7f8a59e00700 1 --2- 172.21.15.89:0/2000073506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a5415d760 0x7f8a5415ac00 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c001560 tx=0x7f8a4c00cb50 comp rx=0 tx=0).stop 2024-04-02T07:55:14.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.282+0000 7f8a59e00700 1 -- 172.21.15.89:0/2000073506 shutdown_connections 2024-04-02T07:55:14.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.282+0000 7f8a59e00700 1 --2- 172.21.15.89:0/2000073506 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a3406aae0 0x7f8a3406cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:14.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.282+0000 7f8a59e00700 1 --2- 172.21.15.89:0/2000073506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a5415d760 0x7f8a5415ac00 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:14.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.282+0000 7f8a59e00700 1 --2- 172.21.15.89:0/2000073506 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a540ce7a0 0x7f8a54151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:14.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.282+0000 7f8a59e00700 1 --2- 172.21.15.89:0/2000073506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a540d2c50 0x7f8a540c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:14.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.282+0000 7f8a59e00700 1 -- 172.21.15.89:0/2000073506 >> 172.21.15.89:0/2000073506 conn(0x7f8a540668e0 msgr2=0x7f8a5415c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:14.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.282+0000 7f8a59e00700 1 -- 172.21.15.89:0/2000073506 shutdown_connections 2024-04-02T07:55:14.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.282+0000 7f8a59e00700 1 -- 172.21.15.89:0/2000073506 wait complete. 2024-04-02T07:55:14.284 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T07:55:14.301 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T07:55:14.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.498+0000 7f6337400700 1 Processor -- start 2024-04-02T07:55:14.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.498+0000 7f6337400700 1 -- start start 2024-04-02T07:55:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.499+0000 7f6337400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f633015c700 0x7f63300adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.499+0000 7f6337400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f63300ae4a0 0x7f633015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.499+0000 7f6337400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633006dd00 con 0x7f6330155ac0 2024-04-02T07:55:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.499+0000 7f6337400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f633006d0f0 con 0x7f633015c700 2024-04-02T07:55:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.499+0000 7f6337400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f633006d3d0 con 0x7f63300ae4a0 2024-04-02T07:55:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.499+0000 7f632fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f633015c700 0x7f63300adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.499+0000 7f632fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f633015c700 0x7f63300adf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48130/0 (socket says 172.21.15.89:48130) 2024-04-02T07:55:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.499+0000 7f632fe00700 1 -- 172.21.15.89:0/1295232295 learned_addr learned my addr 172.21.15.89:0/1295232295 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.500+0000 7f632fe00700 1 -- 172.21.15.89:0/1295232295 >> v2:172.21.15.113:3300/0 conn(0x7f63300ae4a0 msgr2=0x7f633015ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.500+0000 7f6335400700 1 --2- 172.21.15.89:0/1295232295 >> v2:172.21.15.113:3300/0 conn(0x7f63300ae4a0 0x7f633015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.500+0000 7f632fe00700 1 --2- 172.21.15.89:0/1295232295 >> v2:172.21.15.113:3300/0 conn(0x7f63300ae4a0 0x7f633015ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:14.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.500+0000 7f632fe00700 1 -- 172.21.15.89:0/1295232295 >> v1:172.21.15.113:6789/0 conn(0x7f6330155ac0 legacy=0x7f633006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:14.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.500+0000 7f632fe00700 1 -- 172.21.15.89:0/1295232295 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633006d6b0 con 0x7f633015c700 2024-04-02T07:55:14.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.500+0000 7f632fe00700 1 --2- 172.21.15.89:0/1295232295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f633015c700 0x7f63300adf60 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f63300aec90 tx=0x7f631c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24340ad2989580c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:14.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.500+0000 7f632f400700 1 -- 172.21.15.89:0/1295232295 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c004230 con 0x7f633015c700 2024-04-02T07:55:14.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.500+0000 7f632f400700 1 -- 172.21.15.89:0/1295232295 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631c004390 con 0x7f633015c700 2024-04-02T07:55:14.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.500+0000 7f632f400700 1 -- 172.21.15.89:0/1295232295 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c0036a0 con 0x7f633015c700 2024-04-02T07:55:14.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.501+0000 7f6337400700 1 -- 172.21.15.89:0/1295232295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f633015c700 msgr2=0x7f63300adf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:14.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.501+0000 7f6337400700 1 --2- 172.21.15.89:0/1295232295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f633015c700 0x7f63300adf60 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f63300aec90 tx=0x7f631c009bd0 comp rx=0 tx=0).stop 2024-04-02T07:55:14.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.501+0000 7f6337400700 1 -- 172.21.15.89:0/1295232295 shutdown_connections 2024-04-02T07:55:14.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.501+0000 7f6337400700 1 --2- 172.21.15.89:0/1295232295 >> v2:172.21.15.113:3300/0 conn(0x7f63300ae4a0 0x7f633015ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:14.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.501+0000 7f6337400700 1 --2- 172.21.15.89:0/1295232295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f633015c700 0x7f63300adf60 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:14.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.501+0000 7f6337400700 1 -- 172.21.15.89:0/1295232295 >> 172.21.15.89:0/1295232295 conn(0x7f63300cf210 msgr2=0x7f63300cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:14.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.501+0000 7f6337400700 1 -- 172.21.15.89:0/1295232295 shutdown_connections 2024-04-02T07:55:14.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.501+0000 7f6337400700 1 -- 172.21.15.89:0/1295232295 wait complete. 2024-04-02T07:55:14.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.503+0000 7f6337400700 1 Processor -- start 2024-04-02T07:55:14.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.503+0000 7f6337400700 1 -- start start 2024-04-02T07:55:14.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f6337400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63300ae4a0 0x7f633009bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:14.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f6337400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6330155ac0 0x7f6330098a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:14.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f6337400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f633015c700 0x7f6330099390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:14.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f6337400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6330098f60 con 0x7f6330155ac0 2024-04-02T07:55:14.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f6337400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633009d270 con 0x7f63300ae4a0 2024-04-02T07:55:14.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f6337400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633009c660 con 0x7f633015c700 2024-04-02T07:55:14.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f632fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6330155ac0 0x7f6330098a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:14.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f632fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6330155ac0 0x7f6330098a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48142/0 (socket says 172.21.15.89:48142) 2024-04-02T07:55:14.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f632fe00700 1 -- 172.21.15.89:0/2181568700 learned_addr learned my addr 172.21.15.89:0/2181568700 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:14.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f6335400700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f633015c700 0x7f6330099390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:14.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f6334a00700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63300ae4a0 0x7f633009bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:14.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f632fe00700 1 -- 172.21.15.89:0/2181568700 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f633015c700 msgr2=0x7f6330099390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:14.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f632fe00700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f633015c700 0x7f6330099390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:14.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f632fe00700 1 -- 172.21.15.89:0/2181568700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63300ae4a0 msgr2=0x7f633009bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:14.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f632fe00700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63300ae4a0 0x7f633009bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:14.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f632fe00700 1 -- 172.21.15.89:0/2181568700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631c009550 con 0x7f6330155ac0 2024-04-02T07:55:14.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.504+0000 7f6335400700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f633015c700 0x7f6330099390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:14.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.505+0000 7f632fe00700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6330155ac0 0x7f6330098a20 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f631c009fd0 tx=0x7f631c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:14.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.505+0000 7f632cc00700 1 -- 172.21.15.89:0/2181568700 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c011830 con 0x7f6330155ac0 2024-04-02T07:55:14.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.505+0000 7f632cc00700 1 -- 172.21.15.89:0/2181568700 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631c011990 con 0x7f6330155ac0 2024-04-02T07:55:14.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.505+0000 7f6337400700 1 -- 172.21.15.89:0/2181568700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6330156020 con 0x7f6330155ac0 2024-04-02T07:55:14.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.505+0000 7f632cc00700 1 -- 172.21.15.89:0/2181568700 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c011d40 con 0x7f6330155ac0 2024-04-02T07:55:14.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.505+0000 7f6337400700 1 -- 172.21.15.89:0/2181568700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f633009cc20 con 0x7f6330155ac0 2024-04-02T07:55:14.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.506+0000 7f6337400700 1 -- 172.21.15.89:0/2181568700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6330150680 con 0x7f6330155ac0 2024-04-02T07:55:14.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.507+0000 7f632cc00700 1 -- 172.21.15.89:0/2181568700 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 77) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f631c01b440 con 0x7f6330155ac0 2024-04-02T07:55:14.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.508+0000 7f632cc00700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f631006ad20 0x7f631006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:14.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.508+0000 7f632cc00700 1 -- 172.21.15.89:0/2181568700 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6421+0+0 (secure 0 0 0) 0x7f631c08ac70 con 0x7f6330155ac0 2024-04-02T07:55:14.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.508+0000 7f6334a00700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f631006ad20 0x7f631006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:14.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.509+0000 7f6334a00700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f631006ad20 0x7f631006d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6324001730 tx=0x7f6324009040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:14.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.512+0000 7f632cc00700 1 -- 172.21.15.89:0/2181568700 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6330150680 con 0x7f6330155ac0 2024-04-02T07:55:14.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:14.712+0000 7f6337400700 1 -- 172.21.15.89:0/2181568700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6330150680 con 0x7f6330155ac0 2024-04-02T07:55:15.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.282+0000 7f632cc00700 1 -- 172.21.15.89:0/2181568700 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v131) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6330150680 con 0x7f6330155ac0 2024-04-02T07:55:15.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.284+0000 7f6337400700 1 -- 172.21.15.89:0/2181568700 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f631006ad20 msgr2=0x7f631006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:15.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.284+0000 7f6337400700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f631006ad20 0x7f631006d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6324001730 tx=0x7f6324009040 comp rx=0 tx=0).stop 2024-04-02T07:55:15.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.284+0000 7f6337400700 1 -- 172.21.15.89:0/2181568700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6330155ac0 msgr2=0x7f6330098a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:15.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.284+0000 7f6337400700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6330155ac0 0x7f6330098a20 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f631c009fd0 tx=0x7f631c004750 comp rx=0 tx=0).stop 2024-04-02T07:55:15.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.284+0000 7f6337400700 1 -- 172.21.15.89:0/2181568700 shutdown_connections 2024-04-02T07:55:15.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.284+0000 7f6337400700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f631006ad20 0x7f631006d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.284+0000 7f6337400700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63300ae4a0 0x7f633009bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.284+0000 7f6337400700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f633015c700 0x7f6330099390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.284+0000 7f6337400700 1 --2- 172.21.15.89:0/2181568700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6330155ac0 0x7f6330098a20 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.284+0000 7f6337400700 1 -- 172.21.15.89:0/2181568700 >> 172.21.15.89:0/2181568700 conn(0x7f63300cf210 msgr2=0x7f633015e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:15.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.285+0000 7f6337400700 1 -- 172.21.15.89:0/2181568700 shutdown_connections 2024-04-02T07:55:15.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.285+0000 7f6337400700 1 -- 172.21.15.89:0/2181568700 wait complete. 2024-04-02T07:55:15.286 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T07:55:15.304 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T07:55:15.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.508+0000 7f73ab200700 1 Processor -- start 2024-04-02T07:55:15.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.508+0000 7f73ab200700 1 -- start start 2024-04-02T07:55:15.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.510+0000 7f73ab200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f73a41611b0 0x7f73a415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:15.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.510+0000 7f73ab200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73a40cee10 0x7f73a4161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:15.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.510+0000 7f73ab200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a40ce270 con 0x7f73a415f4d0 2024-04-02T07:55:15.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.510+0000 7f73ab200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73a415b600 con 0x7f73a41611b0 2024-04-02T07:55:15.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.510+0000 7f73ab200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a40d2760 con 0x7f73a40cee10 2024-04-02T07:55:15.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.510+0000 7f73a3400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f73a415f4d0 0x7f73a40ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59388/0 (socket says 172.21.15.89:59388) 2024-04-02T07:55:15.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.510+0000 7f73a3400700 1 -- 172.21.15.89:0/2179110946 learned_addr learned my addr 172.21.15.89:0/2179110946 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:15.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.510+0000 7f73a3e00700 1 --2- 172.21.15.89:0/2179110946 >> v2:172.21.15.89:3300/0 conn(0x7f73a41611b0 0x7f73a415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:15.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.510+0000 7f73a9200700 1 --2- 172.21.15.89:0/2179110946 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73a40cee10 0x7f73a4161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:15.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.510+0000 7f73a3e00700 1 -- 172.21.15.89:0/2179110946 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73a40cee10 msgr2=0x7f73a4161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:15.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.510+0000 7f73a3e00700 1 --2- 172.21.15.89:0/2179110946 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73a40cee10 0x7f73a4161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.510+0000 7f73a3e00700 1 -- 172.21.15.89:0/2179110946 >> v1:172.21.15.89:6789/0 conn(0x7f73a415f4d0 legacy=0x7f73a40ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:15.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.510+0000 7f73a3e00700 1 -- 172.21.15.89:0/2179110946 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a40d2a40 con 0x7f73a41611b0 2024-04-02T07:55:15.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.510+0000 7f73a2a00700 1 -- 172.21.15.89:0/2179110946 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322676209 0 0) 0x7f73a40ce270 con 0x7f73a415f4d0 2024-04-02T07:55:15.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.511+0000 7f73a3e00700 1 --2- 172.21.15.89:0/2179110946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a41611b0 0x7f73a415ef90 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f7394009840 tx=0x7f7394009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e2d56d52ca588 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:15.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.511+0000 7f73a2a00700 1 -- 172.21.15.89:0/2179110946 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7394011830 con 0x7f73a41611b0 2024-04-02T07:55:15.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.511+0000 7f73a2a00700 1 -- 172.21.15.89:0/2179110946 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7394011990 con 0x7f73a41611b0 2024-04-02T07:55:15.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.511+0000 7f73a2a00700 1 -- 172.21.15.89:0/2179110946 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7394011d20 con 0x7f73a41611b0 2024-04-02T07:55:15.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.512+0000 7f73ab200700 1 -- 172.21.15.89:0/2179110946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a41611b0 msgr2=0x7f73a415ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:15.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.512+0000 7f73ab200700 1 --2- 172.21.15.89:0/2179110946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a41611b0 0x7f73a415ef90 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f7394009840 tx=0x7f7394009b50 comp rx=0 tx=0).stop 2024-04-02T07:55:15.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.512+0000 7f73ab200700 1 -- 172.21.15.89:0/2179110946 shutdown_connections 2024-04-02T07:55:15.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.512+0000 7f73ab200700 1 --2- 172.21.15.89:0/2179110946 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73a40cee10 0x7f73a4161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.512+0000 7f73ab200700 1 --2- 172.21.15.89:0/2179110946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a41611b0 0x7f73a415ef90 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.512+0000 7f73ab200700 1 -- 172.21.15.89:0/2179110946 >> 172.21.15.89:0/2179110946 conn(0x7f73a408aff0 msgr2=0x7f73a40d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:15.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.512+0000 7f73ab200700 1 -- 172.21.15.89:0/2179110946 shutdown_connections 2024-04-02T07:55:15.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.512+0000 7f73ab200700 1 -- 172.21.15.89:0/2179110946 wait complete. 2024-04-02T07:55:15.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.513+0000 7f73ab200700 1 Processor -- start 2024-04-02T07:55:15.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.514+0000 7f73ab200700 1 -- start start 2024-04-02T07:55:15.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.514+0000 7f73ab200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a40cee10 0x7f73a40c7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:15.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.514+0000 7f73ab200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73a415f4d0 0x7f73a4156160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:15.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.514+0000 7f73ab200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73a41611b0 0x7f73a4160280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:15.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.514+0000 7f73ab200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a40ca600 con 0x7f73a40cee10 2024-04-02T07:55:15.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.514+0000 7f73ab200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a40b74c0 con 0x7f73a415f4d0 2024-04-02T07:55:15.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.514+0000 7f73ab200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a40c75a0 con 0x7f73a41611b0 2024-04-02T07:55:15.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.515+0000 7f73a3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a40cee10 0x7f73a40c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:15.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.515+0000 7f73a3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a40cee10 0x7f73a40c7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48172/0 (socket says 172.21.15.89:48172) 2024-04-02T07:55:15.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.515+0000 7f73a3e00700 1 -- 172.21.15.89:0/4021333099 learned_addr learned my addr 172.21.15.89:0/4021333099 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:15.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.515+0000 7f73a9200700 1 --2- 172.21.15.89:0/4021333099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73a41611b0 0x7f73a4160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:15.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.515+0000 7f73a3400700 1 --2- 172.21.15.89:0/4021333099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73a415f4d0 0x7f73a4156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:15.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.515+0000 7f73a3e00700 1 -- 172.21.15.89:0/4021333099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73a41611b0 msgr2=0x7f73a4160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:15.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.515+0000 7f73a3e00700 1 --2- 172.21.15.89:0/4021333099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73a41611b0 0x7f73a4160280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.515+0000 7f73a3e00700 1 -- 172.21.15.89:0/4021333099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73a415f4d0 msgr2=0x7f73a4156160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:15.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.515+0000 7f73a3e00700 1 --2- 172.21.15.89:0/4021333099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73a415f4d0 0x7f73a4156160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.515+0000 7f73a3e00700 1 -- 172.21.15.89:0/4021333099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73940094a0 con 0x7f73a40cee10 2024-04-02T07:55:15.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.515+0000 7f73a3e00700 1 --2- 172.21.15.89:0/4021333099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a40cee10 0x7f73a40c7060 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f7394009b90 tx=0x7f7394004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:15.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.516+0000 7f739be00700 1 -- 172.21.15.89:0/4021333099 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73940038a0 con 0x7f73a40cee10 2024-04-02T07:55:15.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.516+0000 7f739be00700 1 -- 172.21.15.89:0/4021333099 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739401e070 con 0x7f73a40cee10 2024-04-02T07:55:15.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.516+0000 7f73ab200700 1 -- 172.21.15.89:0/4021333099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a40cd100 con 0x7f73a40cee10 2024-04-02T07:55:15.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.516+0000 7f739be00700 1 -- 172.21.15.89:0/4021333099 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7394003c00 con 0x7f73a40cee10 2024-04-02T07:55:15.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.516+0000 7f73ab200700 1 -- 172.21.15.89:0/4021333099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a40b7710 con 0x7f73a40cee10 2024-04-02T07:55:15.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.517+0000 7f73ab200700 1 -- 172.21.15.89:0/4021333099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a4067720 con 0x7f73a40cee10 2024-04-02T07:55:15.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.518+0000 7f739be00700 1 -- 172.21.15.89:0/4021333099 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 77) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f739402d450 con 0x7f73a40cee10 2024-04-02T07:55:15.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.518+0000 7f739be00700 1 --2- 172.21.15.89:0/4021333099 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f738406ad20 0x7f738406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:15.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.519+0000 7f739be00700 1 -- 172.21.15.89:0/4021333099 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6007+0+0 (secure 0 0 0) 0x7f739408a430 con 0x7f73a40cee10 2024-04-02T07:55:15.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.519+0000 7f73a3400700 1 --2- 172.21.15.89:0/4021333099 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f738406ad20 0x7f738406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:15.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.519+0000 7f73a3400700 1 --2- 172.21.15.89:0/4021333099 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f738406ad20 0x7f738406d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73a4161600 tx=0x7f7390007be0 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:15.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.523+0000 7f739be00700 1 -- 172.21.15.89:0/4021333099 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7394055580 con 0x7f73a40cee10 2024-04-02T07:55:15.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.726+0000 7f73ab200700 1 -- 172.21.15.89:0/4021333099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f73a40b5d00 con 0x7f73a40cee10 2024-04-02T07:55:15.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.730+0000 7f739be00700 1 -- 172.21.15.89:0/4021333099 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v43) v1 ==== 115+0+0 (secure 0 0 0) 0x7f7394058ba0 con 0x7f73a40cee10 2024-04-02T07:55:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.733+0000 7f73ab200700 1 -- 172.21.15.89:0/4021333099 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f738406ad20 msgr2=0x7f738406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.733+0000 7f73ab200700 1 --2- 172.21.15.89:0/4021333099 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f738406ad20 0x7f738406d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73a4161600 tx=0x7f7390007be0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.733+0000 7f73ab200700 1 -- 172.21.15.89:0/4021333099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a40cee10 msgr2=0x7f73a40c7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.733+0000 7f73ab200700 1 --2- 172.21.15.89:0/4021333099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a40cee10 0x7f73a40c7060 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f7394009b90 tx=0x7f7394004750 comp rx=0 tx=0).stop 2024-04-02T07:55:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.733+0000 7f73ab200700 1 -- 172.21.15.89:0/4021333099 shutdown_connections 2024-04-02T07:55:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.733+0000 7f73ab200700 1 --2- 172.21.15.89:0/4021333099 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f738406ad20 0x7f738406d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.733+0000 7f73ab200700 1 --2- 172.21.15.89:0/4021333099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73a41611b0 0x7f73a4160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.733+0000 7f73ab200700 1 --2- 172.21.15.89:0/4021333099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a40cee10 0x7f73a40c7060 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.733+0000 7f73ab200700 1 --2- 172.21.15.89:0/4021333099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73a415f4d0 0x7f73a4156160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.733+0000 7f73ab200700 1 -- 172.21.15.89:0/4021333099 >> 172.21.15.89:0/4021333099 conn(0x7f73a408aff0 msgr2=0x7f73a406c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.733+0000 7f73ab200700 1 -- 172.21.15.89:0/4021333099 shutdown_connections 2024-04-02T07:55:15.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.733+0000 7f73ab200700 1 -- 172.21.15.89:0/4021333099 wait complete. 2024-04-02T07:55:15.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T07:55:15.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.945+0000 7fa740400700 1 Processor -- start 2024-04-02T07:55:15.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.946+0000 7fa740400700 1 -- start start 2024-04-02T07:55:15.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.947+0000 7fa740400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa7381611b0 0x7fa73815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:15.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.947+0000 7fa740400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa73815f4d0 0x7fa7380ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:15.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.947+0000 7fa740400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7380ce270 con 0x7fa7380d21f0 2024-04-02T07:55:15.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.947+0000 7fa740400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa73815b600 con 0x7fa7381611b0 2024-04-02T07:55:15.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.947+0000 7fa740400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7380d2760 con 0x7fa73815f4d0 2024-04-02T07:55:15.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.947+0000 7fa73d000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa73815f4d0 0x7fa7380ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:15.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.947+0000 7fa73da00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa7381611b0 0x7fa73815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:15.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.947+0000 7fa73da00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa7381611b0 0x7fa73815ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:59360/0 (socket says 172.21.15.89:59360) 2024-04-02T07:55:15.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.947+0000 7fa73da00700 1 -- 172.21.15.89:0/459828435 learned_addr learned my addr 172.21.15.89:0/459828435 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:15.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.947+0000 7fa73d000700 1 --2- 172.21.15.89:0/459828435 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa73815f4d0 0x7fa7380ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:45608/0 (socket says 172.21.15.89:45608) 2024-04-02T07:55:15.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.948+0000 7fa73da00700 1 -- 172.21.15.89:0/459828435 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa73815f4d0 msgr2=0x7fa7380ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:15.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.948+0000 7fa73da00700 1 --2- 172.21.15.89:0/459828435 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa73815f4d0 0x7fa7380ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.948+0000 7fa73da00700 1 -- 172.21.15.89:0/459828435 >> v1:172.21.15.113:6789/0 conn(0x7fa7380d21f0 legacy=0x7fa738161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:15.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.948+0000 7fa73da00700 1 -- 172.21.15.89:0/459828435 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7380d2a40 con 0x7fa7381611b0 2024-04-02T07:55:15.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.948+0000 7fa73da00700 1 --2- 172.21.15.89:0/459828435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7381611b0 0x7fa73815ef90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa72c0098f0 tx=0x7fa72c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bf06753785eed21 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:15.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.949+0000 7fa737e00700 1 -- 172.21.15.89:0/459828435 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c011830 con 0x7fa7381611b0 2024-04-02T07:55:15.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.949+0000 7fa737e00700 1 -- 172.21.15.89:0/459828435 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c011990 con 0x7fa7381611b0 2024-04-02T07:55:15.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.949+0000 7fa737e00700 1 -- 172.21.15.89:0/459828435 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c011d20 con 0x7fa7381611b0 2024-04-02T07:55:15.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.949+0000 7fa740400700 1 -- 172.21.15.89:0/459828435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7381611b0 msgr2=0x7fa73815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:15.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.949+0000 7fa740400700 1 --2- 172.21.15.89:0/459828435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7381611b0 0x7fa73815ef90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa72c0098f0 tx=0x7fa72c009c00 comp rx=0 tx=0).stop 2024-04-02T07:55:15.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.949+0000 7fa740400700 1 -- 172.21.15.89:0/459828435 shutdown_connections 2024-04-02T07:55:15.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.949+0000 7fa740400700 1 --2- 172.21.15.89:0/459828435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7381611b0 0x7fa73815ef90 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.949+0000 7fa740400700 1 --2- 172.21.15.89:0/459828435 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa73815f4d0 0x7fa7380ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.949+0000 7fa740400700 1 -- 172.21.15.89:0/459828435 >> 172.21.15.89:0/459828435 conn(0x7fa73808aff0 msgr2=0x7fa7380d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:15.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.949+0000 7fa740400700 1 -- 172.21.15.89:0/459828435 shutdown_connections 2024-04-02T07:55:15.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.949+0000 7fa740400700 1 -- 172.21.15.89:0/459828435 wait complete. 2024-04-02T07:55:15.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.951+0000 7fa740400700 1 Processor -- start 2024-04-02T07:55:15.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.951+0000 7fa740400700 1 -- start start 2024-04-02T07:55:15.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.952+0000 7fa740400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7380d21f0 0x7fa7381f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:15.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.952+0000 7fa740400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa73815f4d0 0x7fa7381f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:15.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.952+0000 7fa740400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7381611b0 0x7fa73809d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:15.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.952+0000 7fa740400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7380caf10 con 0x7fa73815f4d0 2024-04-02T07:55:15.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.952+0000 7fa740400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7380ca600 con 0x7fa7380d21f0 2024-04-02T07:55:15.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.952+0000 7fa740400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa73809e130 con 0x7fa7381611b0 2024-04-02T07:55:15.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.952+0000 7fa73d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa73815f4d0 0x7fa7381f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:15.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.953+0000 7fa73d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa73815f4d0 0x7fa7381f69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48180/0 (socket says 172.21.15.89:48180) 2024-04-02T07:55:15.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.953+0000 7fa73d000700 1 -- 172.21.15.89:0/582831898 learned_addr learned my addr 172.21.15.89:0/582831898 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:15.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.953+0000 7fa73da00700 1 --2- 172.21.15.89:0/582831898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7380d21f0 0x7fa7381f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:15.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.953+0000 7fa73e400700 1 --2- 172.21.15.89:0/582831898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7381611b0 0x7fa73809d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:15.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.953+0000 7fa73d000700 1 -- 172.21.15.89:0/582831898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7381611b0 msgr2=0x7fa73809d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:15.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.953+0000 7fa73d000700 1 --2- 172.21.15.89:0/582831898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7381611b0 0x7fa73809d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.953+0000 7fa73d000700 1 -- 172.21.15.89:0/582831898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7380d21f0 msgr2=0x7fa7381f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:15.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.953+0000 7fa73d000700 1 --2- 172.21.15.89:0/582831898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7380d21f0 0x7fa7381f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:15.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.953+0000 7fa73d000700 1 -- 172.21.15.89:0/582831898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c009550 con 0x7fa73815f4d0 2024-04-02T07:55:15.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.953+0000 7fa73d000700 1 --2- 172.21.15.89:0/582831898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa73815f4d0 0x7fa7381f69c0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fa72400e9b0 tx=0x7fa72400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:15.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.953+0000 7fa735600700 1 -- 172.21.15.89:0/582831898 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72400ee50 con 0x7fa73815f4d0 2024-04-02T07:55:15.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.953+0000 7fa735600700 1 -- 172.21.15.89:0/582831898 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72400c7f0 con 0x7fa73815f4d0 2024-04-02T07:55:15.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.953+0000 7fa740400700 1 -- 172.21.15.89:0/582831898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7380ca850 con 0x7fa73815f4d0 2024-04-02T07:55:15.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.954+0000 7fa735600700 1 -- 172.21.15.89:0/582831898 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72400cb60 con 0x7fa73815f4d0 2024-04-02T07:55:15.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.954+0000 7fa740400700 1 -- 172.21.15.89:0/582831898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa73809d920 con 0x7fa73815f4d0 2024-04-02T07:55:15.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.955+0000 7fa740400700 1 -- 172.21.15.89:0/582831898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa73815d610 con 0x7fa73815f4d0 2024-04-02T07:55:15.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.955+0000 7fa735600700 1 -- 172.21.15.89:0/582831898 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 77) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fa72400ccc0 con 0x7fa73815f4d0 2024-04-02T07:55:15.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.956+0000 7fa735600700 1 --2- 172.21.15.89:0/582831898 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa71806ad20 0x7fa71806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:15.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.956+0000 7fa735600700 1 -- 172.21.15.89:0/582831898 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6007+0+0 (secure 0 0 0) 0x7fa7240894a0 con 0x7fa73815f4d0 2024-04-02T07:55:15.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.957+0000 7fa73da00700 1 --2- 172.21.15.89:0/582831898 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa71806ad20 0x7fa71806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:15.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.957+0000 7fa73da00700 1 --2- 172.21.15.89:0/582831898 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa71806ad20 0x7fa71806d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa72c014040 tx=0x7fa72c008030 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:15.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:15.961+0000 7fa735600700 1 -- 172.21.15.89:0/582831898 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa7240544f0 con 0x7fa73815f4d0 2024-04-02T07:55:16.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.140+0000 7fa735600700 1 -- 172.21.15.89:0/582831898 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa7240548c0 con 0x7fa73815f4d0 2024-04-02T07:55:16.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.172+0000 7fa740400700 1 -- 172.21.15.89:0/582831898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fa738067720 con 0x7fa73815f4d0 2024-04-02T07:55:16.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.177+0000 7fa735600700 1 -- 172.21.15.89:0/582831898 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v44) v1 ==== 120+0+0 (secure 0 0 0) 0x7fa724057f80 con 0x7fa73815f4d0 2024-04-02T07:55:16.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.180+0000 7fa740400700 1 -- 172.21.15.89:0/582831898 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa71806ad20 msgr2=0x7fa71806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:16.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.180+0000 7fa740400700 1 --2- 172.21.15.89:0/582831898 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa71806ad20 0x7fa71806d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa72c014040 tx=0x7fa72c008030 comp rx=0 tx=0).stop 2024-04-02T07:55:16.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.180+0000 7fa740400700 1 -- 172.21.15.89:0/582831898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa73815f4d0 msgr2=0x7fa7381f69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:16.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.180+0000 7fa740400700 1 --2- 172.21.15.89:0/582831898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa73815f4d0 0x7fa7381f69c0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fa72400e9b0 tx=0x7fa72400ecc0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.180+0000 7fa740400700 1 -- 172.21.15.89:0/582831898 shutdown_connections 2024-04-02T07:55:16.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.180+0000 7fa740400700 1 --2- 172.21.15.89:0/582831898 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa71806ad20 0x7fa71806d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.180+0000 7fa740400700 1 --2- 172.21.15.89:0/582831898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7380d21f0 0x7fa7381f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.180+0000 7fa740400700 1 --2- 172.21.15.89:0/582831898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7381611b0 0x7fa73809d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.180+0000 7fa740400700 1 --2- 172.21.15.89:0/582831898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa73815f4d0 0x7fa7381f69c0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.180+0000 7fa740400700 1 -- 172.21.15.89:0/582831898 >> 172.21.15.89:0/582831898 conn(0x7fa73808aff0 msgr2=0x7fa73815da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:16.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.180+0000 7fa740400700 1 -- 172.21.15.89:0/582831898 shutdown_connections 2024-04-02T07:55:16.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.180+0000 7fa740400700 1 -- 172.21.15.89:0/582831898 wait complete. 2024-04-02T07:55:16.199 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T07:55:16.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.393+0000 7f8ae3200700 1 Processor -- start 2024-04-02T07:55:16.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.393+0000 7f8ae3200700 1 -- start start 2024-04-02T07:55:16.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.394+0000 7f8ae3200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8adc160d70 0x7f8adc0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.394+0000 7f8ae3200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8adc161150 con 0x7f8adc15ef70 2024-04-02T07:55:16.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.394+0000 7f8ae3200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8adc06dba0 con 0x7f8adc06a790 2024-04-02T07:55:16.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.394+0000 7f8ae3200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc0b4cd0 con 0x7f8adc160d70 2024-04-02T07:55:16.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.394+0000 7f8adb400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8adc15ef70 0x7f8adc15e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59404/0 (socket says 172.21.15.89:59404) 2024-04-02T07:55:16.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.394+0000 7f8adb400700 1 -- 172.21.15.89:0/85703005 learned_addr learned my addr 172.21.15.89:0/85703005 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:16.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adbe00700 1 --2- 172.21.15.89:0/85703005 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8adc160d70 0x7f8adc0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064225905 0 0) 0x7f8adc161150 con 0x7f8adc15ef70 2024-04-02T07:55:16.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac00035b0 con 0x7f8adc15ef70 2024-04-02T07:55:16.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193970071 0 0) 0x7f8adc06dba0 con 0x7f8adc06a790 2024-04-02T07:55:16.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8adc161150 con 0x7f8adc06a790 2024-04-02T07:55:16.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 245862175 0 0) 0x7f8ac00035b0 con 0x7f8adc15ef70 2024-04-02T07:55:16.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8adc160d70 msgr2=0x7f8adc0b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:16.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 --2- 172.21.15.89:0/85703005 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8adc160d70 0x7f8adc0b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 >> v1:172.21.15.113:6789/0 conn(0x7f8adc06a790 legacy=0x7f8adc06e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:16.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc0b4ee0 con 0x7f8adc15ef70 2024-04-02T07:55:16.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f8ac8001670 con 0x7f8adc15ef70 2024-04-02T07:55:16.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 >> v1:172.21.15.89:6789/0 conn(0x7f8adc15ef70 legacy=0x7f8adc15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:16.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 --2- 172.21.15.89:0/85703005 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac0003bc0 0x7f8ac0003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 --2- 172.21.15.89:0/85703005 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8adc160d70 0x7f8ac0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 --2- 172.21.15.89:0/85703005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ac0004d90 0x7f8ac0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc0b4cd0 con 0x7f8ac0004d90 2024-04-02T07:55:16.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac000bac0 con 0x7f8ac0003bc0 2024-04-02T07:55:16.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.395+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac000bc80 con 0x7f8adc160d70 2024-04-02T07:55:16.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.396+0000 7f8adbe00700 1 --2- 172.21.15.89:0/85703005 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac0003bc0 0x7f8ac0003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.396+0000 7f8adb400700 1 --2- 172.21.15.89:0/85703005 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8adc160d70 0x7f8ac0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.396+0000 7f8ae1200700 1 --2- 172.21.15.89:0/85703005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ac0004d90 0x7f8ac0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.396+0000 7f8adbe00700 1 -- 172.21.15.89:0/85703005 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8adc160d70 msgr2=0x7f8ac0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:16.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.396+0000 7f8adbe00700 1 --2- 172.21.15.89:0/85703005 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8adc160d70 0x7f8ac0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.396+0000 7f8adbe00700 1 -- 172.21.15.89:0/85703005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ac0004d90 msgr2=0x7f8ac0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:16.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.396+0000 7f8adbe00700 1 --2- 172.21.15.89:0/85703005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ac0004d90 0x7f8ac0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.396+0000 7f8adbe00700 1 -- 172.21.15.89:0/85703005 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac00035b0 con 0x7f8ac0003bc0 2024-04-02T07:55:16.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.397+0000 7f8adbe00700 1 --2- 172.21.15.89:0/85703005 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac0003bc0 0x7f8ac0003fc0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009b60 tx=0x7f8ac0005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbcd13223a14fd4d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:16.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.397+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc00e6c0 con 0x7f8ac0003bc0 2024-04-02T07:55:16.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.397+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8acc00e820 con 0x7f8ac0003bc0 2024-04-02T07:55:16.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.397+0000 7f8adaa00700 1 -- 172.21.15.89:0/85703005 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc00eb90 con 0x7f8ac0003bc0 2024-04-02T07:55:16.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.397+0000 7f8ae3200700 1 -- 172.21.15.89:0/85703005 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac0003bc0 msgr2=0x7f8ac0003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:16.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.397+0000 7f8ae3200700 1 --2- 172.21.15.89:0/85703005 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac0003bc0 0x7f8ac0003fc0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009b60 tx=0x7f8ac0005880 comp rx=0 tx=0).stop 2024-04-02T07:55:16.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.397+0000 7f8ae1200700 1 -- 172.21.15.89:0/85703005 reap_dead start 2024-04-02T07:55:16.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.397+0000 7f8ae3200700 1 -- 172.21.15.89:0/85703005 shutdown_connections 2024-04-02T07:55:16.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.397+0000 7f8ae3200700 1 -- 172.21.15.89:0/85703005 >> 172.21.15.89:0/85703005 conn(0x7f8adc0b5720 msgr2=0x7f8adc0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:16.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.397+0000 7f8ae3200700 1 -- 172.21.15.89:0/85703005 shutdown_connections 2024-04-02T07:55:16.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.397+0000 7f8ae3200700 1 -- 172.21.15.89:0/85703005 wait complete. 2024-04-02T07:55:16.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.399+0000 7f8ae3200700 1 Processor -- start 2024-04-02T07:55:16.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.399+0000 7f8ae3200700 1 -- start start 2024-04-02T07:55:16.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.400+0000 7f8ae3200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8ac0003bc0 0x7f8adc15da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.400+0000 7f8ae3200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc143600 0x7f8adc13f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.400+0000 7f8ae3200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8adc13fe40 0x7f8adc1402f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.400+0000 7f8ae3200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc13ef20 con 0x7f8adc143600 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.400+0000 7f8ae3200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc140830 con 0x7f8adc13fe40 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.400+0000 7f8ae3200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc15dfd0 con 0x7f8ac0003bc0 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.400+0000 7f8adb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc143600 0x7f8adc13f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.400+0000 7f8adb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc143600 0x7f8adc13f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48194/0 (socket says 172.21.15.89:48194) 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.400+0000 7f8adb400700 1 -- 172.21.15.89:0/4092372930 learned_addr learned my addr 172.21.15.89:0/4092372930 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.401+0000 7f8adbe00700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8ac0003bc0 0x7f8adc15da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.401+0000 7f8ae1200700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8adc13fe40 0x7f8adc1402f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.401+0000 7f8adb400700 1 -- 172.21.15.89:0/4092372930 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8ac0003bc0 msgr2=0x7f8adc15da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.401+0000 7f8adb400700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8ac0003bc0 0x7f8adc15da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.401+0000 7f8adb400700 1 -- 172.21.15.89:0/4092372930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8adc13fe40 msgr2=0x7f8adc1402f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.401+0000 7f8adb400700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8adc13fe40 0x7f8adc1402f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.401+0000 7f8adb400700 1 -- 172.21.15.89:0/4092372930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc13e680 con 0x7f8adc143600 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.401+0000 7f8ae1200700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8adc13fe40 0x7f8adc1402f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:16.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.401+0000 7f8adb400700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc143600 0x7f8adc13f590 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8001050 tx=0x7f8ac8007570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:16.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.401+0000 7f8ad3e00700 1 -- 172.21.15.89:0/4092372930 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8011580 con 0x7f8adc143600 2024-04-02T07:55:16.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.401+0000 7f8ad3e00700 1 -- 172.21.15.89:0/4092372930 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac8011700 con 0x7f8adc143600 2024-04-02T07:55:16.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.401+0000 7f8ae3200700 1 -- 172.21.15.89:0/4092372930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8adc13e960 con 0x7f8adc143600 2024-04-02T07:55:16.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.401+0000 7f8ad3e00700 1 -- 172.21.15.89:0/4092372930 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8011ab0 con 0x7f8adc143600 2024-04-02T07:55:16.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.402+0000 7f8ae3200700 1 -- 172.21.15.89:0/4092372930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8adc1414a0 con 0x7f8adc143600 2024-04-02T07:55:16.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.402+0000 7f8ae3200700 1 -- 172.21.15.89:0/4092372930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8adc141300 con 0x7f8adc143600 2024-04-02T07:55:16.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.403+0000 7f8ad3e00700 1 -- 172.21.15.89:0/4092372930 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8ac8011c10 con 0x7f8adc143600 2024-04-02T07:55:16.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.405+0000 7f8ad3e00700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8abc06adb0 0x7f8abc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.405+0000 7f8ad3e00700 1 -- 172.21.15.89:0/4092372930 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6007+0+0 (secure 0 0 0) 0x7f8ac8089590 con 0x7f8adc143600 2024-04-02T07:55:16.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.405+0000 7f8adbe00700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8abc06adb0 0x7f8abc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.405+0000 7f8adbe00700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8abc06adb0 0x7f8abc06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8acc0090b0 tx=0x7f8acc01f040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:16.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.409+0000 7f8ad3e00700 1 -- 172.21.15.89:0/4092372930 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8adc141300 con 0x7f8adc143600 2024-04-02T07:55:16.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.611+0000 7f8ae3200700 1 -- 172.21.15.89:0/4092372930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f8adc141300 con 0x7f8adc143600 2024-04-02T07:55:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.617+0000 7f8ad3e00700 1 -- 172.21.15.89:0/4092372930 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v45) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8adc141300 con 0x7f8adc143600 2024-04-02T07:55:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.619+0000 7f8ae3200700 1 -- 172.21.15.89:0/4092372930 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8abc06adb0 msgr2=0x7f8abc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.619+0000 7f8ae3200700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8abc06adb0 0x7f8abc06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8acc0090b0 tx=0x7f8acc01f040 comp rx=0 tx=0).stop 2024-04-02T07:55:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.619+0000 7f8ae3200700 1 -- 172.21.15.89:0/4092372930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc143600 msgr2=0x7f8adc13f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.619+0000 7f8ae3200700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc143600 0x7f8adc13f590 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8001050 tx=0x7f8ac8007570 comp rx=0 tx=0).stop 2024-04-02T07:55:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.619+0000 7f8ae3200700 1 -- 172.21.15.89:0/4092372930 shutdown_connections 2024-04-02T07:55:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.619+0000 7f8ae3200700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8abc06adb0 0x7f8abc06d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.619+0000 7f8ae3200700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8adc13fe40 0x7f8adc1402f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.619+0000 7f8ae3200700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8ac0003bc0 0x7f8adc15da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.619+0000 7f8ae3200700 1 --2- 172.21.15.89:0/4092372930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8adc143600 0x7f8adc13f590 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.619+0000 7f8ae3200700 1 -- 172.21.15.89:0/4092372930 >> 172.21.15.89:0/4092372930 conn(0x7f8adc0b5720 msgr2=0x7f8adc06ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.620+0000 7f8ae3200700 1 -- 172.21.15.89:0/4092372930 shutdown_connections 2024-04-02T07:55:16.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.620+0000 7f8ae3200700 1 -- 172.21.15.89:0/4092372930 wait complete. 2024-04-02T07:55:16.638 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-04-02T07:55:16.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.827+0000 7f16d1400700 1 Processor -- start 2024-04-02T07:55:16.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.827+0000 7f16d1400700 1 -- start start 2024-04-02T07:55:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.828+0000 7f16d1400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f16cc15d640 0x7f16cc0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.828+0000 7f16d1400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f16cc0ceec0 0x7f16cc1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.828+0000 7f16d1400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16cc161800 0x7f16cc0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.828+0000 7f16d1400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16cc161c70 con 0x7f16cc0ceec0 2024-04-02T07:55:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.828+0000 7f16d1400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16cc0d30d0 con 0x7f16cc15d640 2024-04-02T07:55:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.828+0000 7f16d1400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16cc0b6840 con 0x7f16cc161800 2024-04-02T07:55:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.829+0000 7f16c9e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f16cc0ceec0 0x7f16cc1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.829+0000 7f16c9e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f16cc0ceec0 0x7f16cc1612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48206/0 (socket says 172.21.15.89:48206) 2024-04-02T07:55:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.829+0000 7f16c9e00700 1 -- 172.21.15.89:0/2807112172 learned_addr learned my addr 172.21.15.89:0/2807112172 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.829+0000 7f16ca800700 1 --2- 172.21.15.89:0/2807112172 >> v2:172.21.15.113:3300/0 conn(0x7f16cc15d640 0x7f16cc0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.829+0000 7f16cb200700 1 --2- 172.21.15.89:0/2807112172 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16cc161800 0x7f16cc0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.829+0000 7f16c9e00700 1 -- 172.21.15.89:0/2807112172 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16cc161800 msgr2=0x7f16cc0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.829+0000 7f16c9e00700 1 --2- 172.21.15.89:0/2807112172 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16cc161800 0x7f16cc0cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.829+0000 7f16c9e00700 1 -- 172.21.15.89:0/2807112172 >> v2:172.21.15.113:3300/0 conn(0x7f16cc15d640 msgr2=0x7f16cc0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.829+0000 7f16c9e00700 1 --2- 172.21.15.89:0/2807112172 >> v2:172.21.15.113:3300/0 conn(0x7f16cc15d640 0x7f16cc0d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.829+0000 7f16c9e00700 1 -- 172.21.15.89:0/2807112172 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16cc0d2880 con 0x7f16cc0ceec0 2024-04-02T07:55:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.829+0000 7f16c9e00700 1 --2- 172.21.15.89:0/2807112172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16cc0ceec0 0x7f16cc1612c0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f16cc0b5c20 tx=0x7f16b8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb094407aebb387 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.830+0000 7f16c9400700 1 -- 172.21.15.89:0/2807112172 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b800b870 con 0x7f16cc0ceec0 2024-04-02T07:55:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.830+0000 7f16c9400700 1 -- 172.21.15.89:0/2807112172 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b800b9d0 con 0x7f16cc0ceec0 2024-04-02T07:55:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.830+0000 7f16c9400700 1 -- 172.21.15.89:0/2807112172 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b800bd60 con 0x7f16cc0ceec0 2024-04-02T07:55:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.830+0000 7f16d1400700 1 -- 172.21.15.89:0/2807112172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16cc0ceec0 msgr2=0x7f16cc1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.830+0000 7f16d1400700 1 --2- 172.21.15.89:0/2807112172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16cc0ceec0 0x7f16cc1612c0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f16cc0b5c20 tx=0x7f16b8009bd0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.830+0000 7f16d1400700 1 -- 172.21.15.89:0/2807112172 shutdown_connections 2024-04-02T07:55:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.830+0000 7f16d1400700 1 --2- 172.21.15.89:0/2807112172 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16cc161800 0x7f16cc0cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.830+0000 7f16d1400700 1 --2- 172.21.15.89:0/2807112172 >> v2:172.21.15.113:3300/0 conn(0x7f16cc15d640 0x7f16cc0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.830+0000 7f16d1400700 1 --2- 172.21.15.89:0/2807112172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16cc0ceec0 0x7f16cc1612c0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.830+0000 7f16d1400700 1 -- 172.21.15.89:0/2807112172 >> 172.21.15.89:0/2807112172 conn(0x7f16cc08aff0 msgr2=0x7f16cc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.830+0000 7f16d1400700 1 -- 172.21.15.89:0/2807112172 shutdown_connections 2024-04-02T07:55:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.830+0000 7f16d1400700 1 -- 172.21.15.89:0/2807112172 wait complete. 2024-04-02T07:55:16.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.832+0000 7f16d1400700 1 Processor -- start 2024-04-02T07:55:16.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.832+0000 7f16d1400700 1 -- start start 2024-04-02T07:55:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.833+0000 7f16d1400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16cc0ceec0 0x7f16cc15a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.833+0000 7f16d1400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16cc15d640 0x7f16cc0b7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.833+0000 7f16d1400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16cc161800 0x7f16cc0b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.833+0000 7f16d1400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16cc15ad00 con 0x7f16cc0ceec0 2024-04-02T07:55:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.833+0000 7f16d1400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16cc0cad10 con 0x7f16cc161800 2024-04-02T07:55:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.833+0000 7f16d1400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16cc0a0dd0 con 0x7f16cc15d640 2024-04-02T07:55:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.833+0000 7f16ca800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16cc0ceec0 0x7f16cc15a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.833+0000 7f16ca800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16cc0ceec0 0x7f16cc15a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48212/0 (socket says 172.21.15.89:48212) 2024-04-02T07:55:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.833+0000 7f16ca800700 1 -- 172.21.15.89:0/299011856 learned_addr learned my addr 172.21.15.89:0/299011856 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.833+0000 7f16cb200700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16cc161800 0x7f16cc0b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.833+0000 7f16c9e00700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16cc15d640 0x7f16cc0b7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.834+0000 7f16ca800700 1 -- 172.21.15.89:0/299011856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16cc15d640 msgr2=0x7f16cc0b7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:16.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.834+0000 7f16ca800700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16cc15d640 0x7f16cc0b7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.834+0000 7f16ca800700 1 -- 172.21.15.89:0/299011856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16cc161800 msgr2=0x7f16cc0b81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:16.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.834+0000 7f16ca800700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16cc161800 0x7f16cc0b81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:16.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.834+0000 7f16ca800700 1 -- 172.21.15.89:0/299011856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b8009550 con 0x7f16cc0ceec0 2024-04-02T07:55:16.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.834+0000 7f16c9e00700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16cc15d640 0x7f16cc0b7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:16.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.834+0000 7f16ca800700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16cc0ceec0 0x7f16cc15a7c0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00e810 tx=0x7f16bc00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:16.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.834+0000 7f16c2a00700 1 -- 172.21.15.89:0/299011856 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc013610 con 0x7f16cc0ceec0 2024-04-02T07:55:16.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.834+0000 7f16c2a00700 1 -- 172.21.15.89:0/299011856 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16bc013770 con 0x7f16cc0ceec0 2024-04-02T07:55:16.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.834+0000 7f16d1400700 1 -- 172.21.15.89:0/299011856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16cc0a10b0 con 0x7f16cc0ceec0 2024-04-02T07:55:16.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.834+0000 7f16c2a00700 1 -- 172.21.15.89:0/299011856 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc013b20 con 0x7f16cc0ceec0 2024-04-02T07:55:16.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.834+0000 7f16d1400700 1 -- 172.21.15.89:0/299011856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16cc0ca9a0 con 0x7f16cc0ceec0 2024-04-02T07:55:16.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.835+0000 7f16d1400700 1 -- 172.21.15.89:0/299011856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16cc072380 con 0x7f16cc0ceec0 2024-04-02T07:55:16.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.836+0000 7f16c2a00700 1 -- 172.21.15.89:0/299011856 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f16bc013c80 con 0x7f16cc0ceec0 2024-04-02T07:55:16.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.837+0000 7f16c2a00700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f16ac06ade0 0x7f16ac06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:16.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.837+0000 7f16c2a00700 1 -- 172.21.15.89:0/299011856 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6007+0+0 (secure 0 0 0) 0x7f16bc0893a0 con 0x7f16cc0ceec0 2024-04-02T07:55:16.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.837+0000 7f16c9e00700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f16ac06ade0 0x7f16ac06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:16.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.838+0000 7f16c9e00700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f16ac06ade0 0x7f16ac06d2a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f16b8014040 tx=0x7f16b8008030 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:16.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:16.841+0000 7f16c2a00700 1 -- 172.21.15.89:0/299011856 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f16bc0541a0 con 0x7f16cc0ceec0 2024-04-02T07:55:17.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.120+0000 7f16d1400700 1 -- 172.21.15.89:0/299011856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f16cc067720 con 0x7f16cc0ceec0 2024-04-02T07:55:17.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.290+0000 7f16c2a00700 1 -- 172.21.15.89:0/299011856 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v730) v1 ==== 148+0+0 (secure 0 0 0) 0x7f16bc0577c0 con 0x7f16cc0ceec0 2024-04-02T07:55:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.292+0000 7f16d1400700 1 -- 172.21.15.89:0/299011856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f16ac06ade0 msgr2=0x7f16ac06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.292+0000 7f16d1400700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f16ac06ade0 0x7f16ac06d2a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f16b8014040 tx=0x7f16b8008030 comp rx=0 tx=0).stop 2024-04-02T07:55:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.292+0000 7f16d1400700 1 -- 172.21.15.89:0/299011856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16cc0ceec0 msgr2=0x7f16cc15a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.292+0000 7f16d1400700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16cc0ceec0 0x7f16cc15a7c0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00e810 tx=0x7f16bc00ebd0 comp rx=0 tx=0).stop 2024-04-02T07:55:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.292+0000 7f16d1400700 1 -- 172.21.15.89:0/299011856 shutdown_connections 2024-04-02T07:55:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.292+0000 7f16d1400700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f16ac06ade0 0x7f16ac06d2a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.292+0000 7f16d1400700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16cc161800 0x7f16cc0b81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.292+0000 7f16d1400700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16cc0ceec0 0x7f16cc15a7c0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.292+0000 7f16d1400700 1 --2- 172.21.15.89:0/299011856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f16cc15d640 0x7f16cc0b7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:17.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.292+0000 7f16d1400700 1 -- 172.21.15.89:0/299011856 >> 172.21.15.89:0/299011856 conn(0x7f16cc08aff0 msgr2=0x7f16cc15cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.292+0000 7f16d1400700 1 -- 172.21.15.89:0/299011856 shutdown_connections 2024-04-02T07:55:17.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.292+0000 7f16d1400700 1 -- 172.21.15.89:0/299011856 wait complete. 2024-04-02T07:55:17.311 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2024-04-02T07:55:17.311 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-02T07:55:17.311 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-04-02T07:55:17.413 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:55:17.419 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:55:17.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.512+0000 7f3066400700 1 Processor -- start 2024-04-02T07:55:17.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.512+0000 7f3066400700 1 -- start start 2024-04-02T07:55:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.513+0000 7f3066400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3060160f10 0x7f30600d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.513+0000 7f3066400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3060161590 0x7f3060161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.513+0000 7f3066400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30600ceda0 con 0x7f30600cf230 2024-04-02T07:55:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.513+0000 7f3066400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3060161f80 con 0x7f3060161590 2024-04-02T07:55:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.513+0000 7f3066400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30601612f0 con 0x7f3060160f10 2024-04-02T07:55:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.513+0000 7f305ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f30600cf230 0x7f30600d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59410/0 (socket says 172.21.15.89:59410) 2024-04-02T07:55:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.513+0000 7f305ea00700 1 -- 172.21.15.89:0/2820474999 learned_addr learned my addr 172.21.15.89:0/2820474999 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.513+0000 7f305fe00700 1 --2- 172.21.15.89:0/2820474999 >> v2:172.21.15.89:3300/0 conn(0x7f3060161590 0x7f3060161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.514+0000 7f305f400700 1 --2- 172.21.15.89:0/2820474999 >> v2:172.21.15.113:3300/0 conn(0x7f3060160f10 0x7f30600d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:17.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.514+0000 7f305e000700 1 -- 172.21.15.89:0/2820474999 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709035495 0 0) 0x7f30600ceda0 con 0x7f30600cf230 2024-04-02T07:55:17.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.514+0000 7f305fe00700 1 -- 172.21.15.89:0/2820474999 >> v2:172.21.15.113:3300/0 conn(0x7f3060160f10 msgr2=0x7f30600d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:17.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.514+0000 7f305fe00700 1 --2- 172.21.15.89:0/2820474999 >> v2:172.21.15.113:3300/0 conn(0x7f3060160f10 0x7f30600d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:17.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.514+0000 7f305fe00700 1 -- 172.21.15.89:0/2820474999 >> v1:172.21.15.89:6789/0 conn(0x7f30600cf230 legacy=0x7f30600d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:17.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.514+0000 7f305fe00700 1 -- 172.21.15.89:0/2820474999 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30600ce470 con 0x7f3060161590 2024-04-02T07:55:17.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.514+0000 7f305fe00700 1 --2- 172.21.15.89:0/2820474999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3060161590 0x7f3060161a40 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f30600d1080 tx=0x7f305800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1d104a95c0779 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:17.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.515+0000 7f305e000700 1 -- 172.21.15.89:0/2820474999 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305800e070 con 0x7f3060161590 2024-04-02T07:55:17.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.515+0000 7f305e000700 1 -- 172.21.15.89:0/2820474999 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305800bb20 con 0x7f3060161590 2024-04-02T07:55:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.515+0000 7f305e000700 1 -- 172.21.15.89:0/2820474999 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3058012430 con 0x7f3060161590 2024-04-02T07:55:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.515+0000 7f3066400700 1 -- 172.21.15.89:0/2820474999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3060161590 msgr2=0x7f3060161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.515+0000 7f3066400700 1 --2- 172.21.15.89:0/2820474999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3060161590 0x7f3060161a40 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f30600d1080 tx=0x7f305800b330 comp rx=0 tx=0).stop 2024-04-02T07:55:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.515+0000 7f3066400700 1 -- 172.21.15.89:0/2820474999 shutdown_connections 2024-04-02T07:55:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.515+0000 7f3066400700 1 --2- 172.21.15.89:0/2820474999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3060161590 0x7f3060161a40 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.515+0000 7f3066400700 1 --2- 172.21.15.89:0/2820474999 >> v2:172.21.15.113:3300/0 conn(0x7f3060160f10 0x7f30600d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.515+0000 7f3066400700 1 -- 172.21.15.89:0/2820474999 >> 172.21.15.89:0/2820474999 conn(0x7f306008aff0 msgr2=0x7f30600d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.515+0000 7f3066400700 1 -- 172.21.15.89:0/2820474999 shutdown_connections 2024-04-02T07:55:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.515+0000 7f3066400700 1 -- 172.21.15.89:0/2820474999 wait complete. 2024-04-02T07:55:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.517+0000 7f3066400700 1 Processor -- start 2024-04-02T07:55:17.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.517+0000 7f3066400700 1 -- start start 2024-04-02T07:55:17.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.518+0000 7f3066400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30600cf230 0x7f306015f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:17.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.518+0000 7f3066400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3060160f10 0x7f30601e7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.518+0000 7f3066400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3060161590 0x7f30601e7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.518+0000 7f3066400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30601f3440 con 0x7f3060160f10 2024-04-02T07:55:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.518+0000 7f3066400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30601593c0 con 0x7f30600cf230 2024-04-02T07:55:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.518+0000 7f3066400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30600b6120 con 0x7f3060161590 2024-04-02T07:55:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.518+0000 7f305ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3060160f10 0x7f30601e7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.518+0000 7f305ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3060160f10 0x7f30601e7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48222/0 (socket says 172.21.15.89:48222) 2024-04-02T07:55:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.518+0000 7f305ea00700 1 -- 172.21.15.89:0/1683533788 learned_addr learned my addr 172.21.15.89:0/1683533788 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.518+0000 7f305f400700 1 --2- 172.21.15.89:0/1683533788 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30600cf230 0x7f306015f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.518+0000 7f305fe00700 1 --2- 172.21.15.89:0/1683533788 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3060161590 0x7f30601e7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.519+0000 7f305ea00700 1 -- 172.21.15.89:0/1683533788 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3060161590 msgr2=0x7f30601e7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.519+0000 7f305ea00700 1 --2- 172.21.15.89:0/1683533788 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3060161590 0x7f30601e7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.519+0000 7f305ea00700 1 -- 172.21.15.89:0/1683533788 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30600cf230 msgr2=0x7f306015f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.519+0000 7f305ea00700 1 --2- 172.21.15.89:0/1683533788 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30600cf230 0x7f306015f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:17.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.519+0000 7f305ea00700 1 -- 172.21.15.89:0/1683533788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3058009a70 con 0x7f3060160f10 2024-04-02T07:55:17.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.519+0000 7f305ea00700 1 --2- 172.21.15.89:0/1683533788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3060160f10 0x7f30601e7500 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f304c00c520 tx=0x7f304c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:17.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.519+0000 7f3057400700 1 -- 172.21.15.89:0/1683533788 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c004c80 con 0x7f3060160f10 2024-04-02T07:55:17.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.519+0000 7f3057400700 1 -- 172.21.15.89:0/1683533788 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304c004de0 con 0x7f3060160f10 2024-04-02T07:55:17.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.519+0000 7f3057400700 1 -- 172.21.15.89:0/1683533788 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c0072d0 con 0x7f3060160f10 2024-04-02T07:55:17.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.519+0000 7f3066400700 1 -- 172.21.15.89:0/1683533788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306015f9c0 con 0x7f3060160f10 2024-04-02T07:55:17.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.520+0000 7f3066400700 1 -- 172.21.15.89:0/1683533788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306015ff80 con 0x7f3060160f10 2024-04-02T07:55:17.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.520+0000 7f3066400700 1 -- 172.21.15.89:0/1683533788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30600b4e20 con 0x7f3060160f10 2024-04-02T07:55:17.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.521+0000 7f3057400700 1 -- 172.21.15.89:0/1683533788 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f304c007430 con 0x7f3060160f10 2024-04-02T07:55:17.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.522+0000 7f3057400700 1 --2- 172.21.15.89:0/1683533788 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f304006adb0 0x7f304006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:17.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.522+0000 7f3057400700 1 -- 172.21.15.89:0/1683533788 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6007+0+0 (secure 0 0 0) 0x7f304c089150 con 0x7f3060160f10 2024-04-02T07:55:17.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.523+0000 7f305f400700 1 --2- 172.21.15.89:0/1683533788 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f304006adb0 0x7f304006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:17.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.523+0000 7f305f400700 1 --2- 172.21.15.89:0/1683533788 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f304006adb0 0x7f304006d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3050009b40 tx=0x7f3050009210 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:17.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.527+0000 7f3057400700 1 -- 172.21.15.89:0/1683533788 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f304c0540e0 con 0x7f3060160f10 2024-04-02T07:55:17.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:17.810+0000 7f3066400700 1 -- 172.21.15.89:0/1683533788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7f3060067720 con 0x7f3060160f10 2024-04-02T07:55:18.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.289+0000 7f3057400700 1 -- 172.21.15.89:0/1683533788 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v731) v1 ==== 163+0+0 (secure 0 0 0) 0x7f304c057700 con 0x7f3060160f10 2024-04-02T07:55:18.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.292+0000 7f3066400700 1 -- 172.21.15.89:0/1683533788 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f304006adb0 msgr2=0x7f304006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:18.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.292+0000 7f3066400700 1 --2- 172.21.15.89:0/1683533788 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f304006adb0 0x7f304006d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3050009b40 tx=0x7f3050009210 comp rx=0 tx=0).stop 2024-04-02T07:55:18.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.292+0000 7f3066400700 1 -- 172.21.15.89:0/1683533788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3060160f10 msgr2=0x7f30601e7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:18.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.292+0000 7f3066400700 1 --2- 172.21.15.89:0/1683533788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3060160f10 0x7f30601e7500 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f304c00c520 tx=0x7f304c00c830 comp rx=0 tx=0).stop 2024-04-02T07:55:18.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.292+0000 7f3066400700 1 -- 172.21.15.89:0/1683533788 shutdown_connections 2024-04-02T07:55:18.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.292+0000 7f3066400700 1 --2- 172.21.15.89:0/1683533788 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f304006adb0 0x7f304006d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:18.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.292+0000 7f3066400700 1 --2- 172.21.15.89:0/1683533788 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30600cf230 0x7f306015f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:18.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.292+0000 7f3066400700 1 --2- 172.21.15.89:0/1683533788 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3060161590 0x7f30601e7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:18.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.292+0000 7f3066400700 1 --2- 172.21.15.89:0/1683533788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3060160f10 0x7f30601e7500 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:18.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.292+0000 7f3066400700 1 -- 172.21.15.89:0/1683533788 >> 172.21.15.89:0/1683533788 conn(0x7f306008aff0 msgr2=0x7f30600ca930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:18.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.292+0000 7f3066400700 1 -- 172.21.15.89:0/1683533788 shutdown_connections 2024-04-02T07:55:18.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.292+0000 7f3066400700 1 -- 172.21.15.89:0/1683533788 wait complete. 2024-04-02T07:55:18.313 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T07:55:18.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.550+0000 7eff3e600700 1 Processor -- start 2024-04-02T07:55:18.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.550+0000 7eff3e600700 1 -- start start 2024-04-02T07:55:18.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.552+0000 7eff3e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff380d2540 0x7eff381687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:18.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.552+0000 7eff3e600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff38161420 con 0x7eff3815d9d0 2024-04-02T07:55:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.552+0000 7eff3e600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff38161700 con 0x7eff380d16a0 2024-04-02T07:55:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.552+0000 7eff3e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff380d4340 con 0x7eff380d2540 2024-04-02T07:55:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.552+0000 7eff37400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7eff3815d9d0 0x7eff380d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59418/0 (socket says 172.21.15.89:59418) 2024-04-02T07:55:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.552+0000 7eff37400700 1 -- 172.21.15.89:0/3799985101 learned_addr learned my addr 172.21.15.89:0/3799985101 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.552+0000 7eff37e00700 1 --2- 172.21.15.89:0/3799985101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff380d2540 0x7eff381687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399932725 0 0) 0x7eff38161420 con 0x7eff3815d9d0 2024-04-02T07:55:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff1c0035b0 con 0x7eff3815d9d0 2024-04-02T07:55:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327257770 0 0) 0x7eff38161700 con 0x7eff380d16a0 2024-04-02T07:55:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff38161420 con 0x7eff380d16a0 2024-04-02T07:55:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3500130000 0 0) 0x7eff1c0035b0 con 0x7eff3815d9d0 2024-04-02T07:55:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff380d2540 msgr2=0x7eff381687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 --2- 172.21.15.89:0/3799985101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff380d2540 0x7eff381687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 >> v1:172.21.15.113:6789/0 conn(0x7eff380d16a0 legacy=0x7eff380d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff381619e0 con 0x7eff3815d9d0 2024-04-02T07:55:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7eff28001670 con 0x7eff3815d9d0 2024-04-02T07:55:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff37e00700 1 --2- 172.21.15.89:0/3799985101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff380d2540 0x7eff381687b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:55:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 >> v1:172.21.15.89:6789/0 conn(0x7eff3815d9d0 legacy=0x7eff380d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 --2- 172.21.15.89:0/3799985101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff1c003bc0 0x7eff1c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 --2- 172.21.15.89:0/3799985101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff1c004850 0x7eff1c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 --2- 172.21.15.89:0/3799985101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff380d2540 0x7eff1c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c00bac0 con 0x7eff1c004850 2024-04-02T07:55:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c00bc00 con 0x7eff1c003bc0 2024-04-02T07:55:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c00bdc0 con 0x7eff380d2540 2024-04-02T07:55:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff36a00700 1 --2- 172.21.15.89:0/3799985101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff1c004850 0x7eff1c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.553+0000 7eff37400700 1 --2- 172.21.15.89:0/3799985101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff1c003bc0 0x7eff1c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.554+0000 7eff37e00700 1 --2- 172.21.15.89:0/3799985101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff380d2540 0x7eff1c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.554+0000 7eff36a00700 1 -- 172.21.15.89:0/3799985101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff380d2540 msgr2=0x7eff1c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.554+0000 7eff36a00700 1 --2- 172.21.15.89:0/3799985101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff380d2540 0x7eff1c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.554+0000 7eff36a00700 1 -- 172.21.15.89:0/3799985101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff1c003bc0 msgr2=0x7eff1c003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.554+0000 7eff36a00700 1 --2- 172.21.15.89:0/3799985101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff1c003bc0 0x7eff1c003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.554+0000 7eff36a00700 1 -- 172.21.15.89:0/3799985101 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff30009a70 con 0x7eff1c004850 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.554+0000 7eff37e00700 1 --2- 172.21.15.89:0/3799985101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff380d2540 0x7eff1c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.554+0000 7eff36a00700 1 --2- 172.21.15.89:0/3799985101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff1c004850 0x7eff1c004cd0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7eff380d1ae0 tx=0x7eff1c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6829a53d6bd04ce server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.554+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff24001460 con 0x7eff1c004850 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.554+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff240052d0 con 0x7eff1c004850 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.554+0000 7eff36000700 1 -- 172.21.15.89:0/3799985101 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff240036d0 con 0x7eff1c004850 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.555+0000 7eff3e600700 1 -- 172.21.15.89:0/3799985101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff1c004850 msgr2=0x7eff1c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.555+0000 7eff3e600700 1 --2- 172.21.15.89:0/3799985101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff1c004850 0x7eff1c004cd0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7eff380d1ae0 tx=0x7eff1c005880 comp rx=0 tx=0).stop 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.555+0000 7eff37e00700 1 -- 172.21.15.89:0/3799985101 reap_dead start 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.555+0000 7eff3e600700 1 -- 172.21.15.89:0/3799985101 shutdown_connections 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.555+0000 7eff3e600700 1 -- 172.21.15.89:0/3799985101 >> 172.21.15.89:0/3799985101 conn(0x7eff3808aff0 msgr2=0x7eff380d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.555+0000 7eff3e600700 1 -- 172.21.15.89:0/3799985101 shutdown_connections 2024-04-02T07:55:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.555+0000 7eff3e600700 1 -- 172.21.15.89:0/3799985101 wait complete. 2024-04-02T07:55:18.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.557+0000 7eff3e600700 1 Processor -- start 2024-04-02T07:55:18.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.557+0000 7eff3e600700 1 -- start start 2024-04-02T07:55:18.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.558+0000 7eff3e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff1c004850 0x7eff380cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:18.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.558+0000 7eff3e600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff380b6020 0x7eff38158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:18.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.558+0000 7eff3e600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff380b44c0 0x7eff380b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:18.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.558+0000 7eff3e600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff380b4e80 con 0x7eff380b6020 2024-04-02T07:55:18.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.558+0000 7eff3e600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff380b6900 con 0x7eff380b44c0 2024-04-02T07:55:18.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.558+0000 7eff3e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff380b6be0 con 0x7eff1c004850 2024-04-02T07:55:18.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.558+0000 7eff36a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff380b6020 0x7eff38158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:18.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.558+0000 7eff36a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff380b6020 0x7eff38158960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48246/0 (socket says 172.21.15.89:48246) 2024-04-02T07:55:18.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.558+0000 7eff36a00700 1 -- 172.21.15.89:0/2447058611 learned_addr learned my addr 172.21.15.89:0/2447058611 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.558+0000 7eff37e00700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff380b44c0 0x7eff380b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.558+0000 7eff37400700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff1c004850 0x7eff380cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.559+0000 7eff36a00700 1 -- 172.21.15.89:0/2447058611 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff1c004850 msgr2=0x7eff380cbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.559+0000 7eff36a00700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff1c004850 0x7eff380cbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.559+0000 7eff36a00700 1 -- 172.21.15.89:0/2447058611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff380b44c0 msgr2=0x7eff380b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.559+0000 7eff36a00700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff380b44c0 0x7eff380b4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.559+0000 7eff36a00700 1 -- 172.21.15.89:0/2447058611 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff380b6ec0 con 0x7eff380b6020 2024-04-02T07:55:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.559+0000 7eff37400700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff1c004850 0x7eff380cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.559+0000 7eff36a00700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff380b6020 0x7eff38158960 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7eff2400c0e0 tx=0x7eff24001320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.559+0000 7eff2f400700 1 -- 172.21.15.89:0/2447058611 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff24007040 con 0x7eff380b6020 2024-04-02T07:55:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.559+0000 7eff2f400700 1 -- 172.21.15.89:0/2447058611 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff240071a0 con 0x7eff380b6020 2024-04-02T07:55:18.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.559+0000 7eff2f400700 1 -- 172.21.15.89:0/2447058611 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff24007330 con 0x7eff380b6020 2024-04-02T07:55:18.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.559+0000 7eff3e600700 1 -- 172.21.15.89:0/2447058611 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff380b71a0 con 0x7eff380b6020 2024-04-02T07:55:18.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.559+0000 7eff3e600700 1 -- 172.21.15.89:0/2447058611 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff381f3bc0 con 0x7eff380b6020 2024-04-02T07:55:18.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.560+0000 7eff3e600700 1 -- 172.21.15.89:0/2447058611 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff380d1ca0 con 0x7eff380b6020 2024-04-02T07:55:18.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.561+0000 7eff2f400700 1 -- 172.21.15.89:0/2447058611 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7eff24007490 con 0x7eff380b6020 2024-04-02T07:55:18.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.562+0000 7eff2f400700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff1806ade0 0x7eff1806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:18.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.562+0000 7eff2f400700 1 -- 172.21.15.89:0/2447058611 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6007+0+0 (secure 0 0 0) 0x7eff2408a5c0 con 0x7eff380b6020 2024-04-02T07:55:18.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.562+0000 7eff37400700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff1806ade0 0x7eff1806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:18.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.563+0000 7eff37400700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff1806ade0 0x7eff1806d2a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7eff28007c10 tx=0x7eff28009040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:18.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.566+0000 7eff2f400700 1 -- 172.21.15.89:0/2447058611 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff24055550 con 0x7eff380b6020 2024-04-02T07:55:18.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.768+0000 7eff3e600700 1 -- 172.21.15.89:0/2447058611 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7eff38067720 con 0x7eff380b6020 2024-04-02T07:55:18.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.772+0000 7eff2f400700 1 -- 172.21.15.89:0/2447058611 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7eff24058b70 con 0x7eff380b6020 2024-04-02T07:55:18.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.774+0000 7eff3e600700 1 -- 172.21.15.89:0/2447058611 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff1806ade0 msgr2=0x7eff1806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:18.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.774+0000 7eff3e600700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff1806ade0 0x7eff1806d2a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7eff28007c10 tx=0x7eff28009040 comp rx=0 tx=0).stop 2024-04-02T07:55:18.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.774+0000 7eff3e600700 1 -- 172.21.15.89:0/2447058611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff380b6020 msgr2=0x7eff38158960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:18.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.774+0000 7eff3e600700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff380b6020 0x7eff38158960 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7eff2400c0e0 tx=0x7eff24001320 comp rx=0 tx=0).stop 2024-04-02T07:55:18.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.775+0000 7eff3e600700 1 -- 172.21.15.89:0/2447058611 shutdown_connections 2024-04-02T07:55:18.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.775+0000 7eff3e600700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7eff1806ade0 0x7eff1806d2a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:18.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.775+0000 7eff3e600700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff380b44c0 0x7eff380b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:18.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.775+0000 7eff3e600700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff1c004850 0x7eff380cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:18.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.775+0000 7eff3e600700 1 --2- 172.21.15.89:0/2447058611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff380b6020 0x7eff38158960 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:18.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.775+0000 7eff3e600700 1 -- 172.21.15.89:0/2447058611 >> 172.21.15.89:0/2447058611 conn(0x7eff3808aff0 msgr2=0x7eff380b13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:18.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.775+0000 7eff3e600700 1 -- 172.21.15.89:0/2447058611 shutdown_connections 2024-04-02T07:55:18.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:18.775+0000 7eff3e600700 1 -- 172.21.15.89:0/2447058611 wait complete. 2024-04-02T07:55:18.793 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:55:18.793 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:55:18.887 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:55:18.977 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:55:18.978 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:55:18.978 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:19.022 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:55:19.023 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:55:19.023 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:19.069 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:55:19.069 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:55:19.069 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:19.113 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:55:19.113 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:55:19.113 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:19.158 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:55:19.158 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T07:55:19.159 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:55:19.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.352+0000 7fd2dde00700 1 Processor -- start 2024-04-02T07:55:19.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.352+0000 7fd2dde00700 1 -- start start 2024-04-02T07:55:19.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.353+0000 7fd2dde00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2d815d640 0x7fd2d80d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:19.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.353+0000 7fd2dde00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd2d80ceec0 0x7fd2d81612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:19.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.353+0000 7fd2dde00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2d8161800 0x7fd2d80cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:19.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.353+0000 7fd2dde00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d8161c90 con 0x7fd2d8161800 2024-04-02T07:55:19.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.353+0000 7fd2dde00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d80d30d0 con 0x7fd2d80ceec0 2024-04-02T07:55:19.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.353+0000 7fd2dde00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80b6840 con 0x7fd2d815d640 2024-04-02T07:55:19.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.354+0000 7fd2d7400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2d8161800 0x7fd2d80cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:19.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.354+0000 7fd2d7400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2d8161800 0x7fd2d80cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41116/0 (socket says 172.21.15.89:41116) 2024-04-02T07:55:19.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.354+0000 7fd2d7400700 1 -- 172.21.15.89:0/3893813521 learned_addr learned my addr 172.21.15.89:0/3893813521 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:19.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.354+0000 7fd2cfe00700 1 --2- 172.21.15.89:0/3893813521 >> v2:172.21.15.113:3300/0 conn(0x7fd2d80ceec0 0x7fd2d81612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:19.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.354+0000 7fd2d6a00700 1 --2- 172.21.15.89:0/3893813521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2d815d640 0x7fd2d80d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:19.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.354+0000 7fd2d7400700 1 -- 172.21.15.89:0/3893813521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2d815d640 msgr2=0x7fd2d80d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:19.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.354+0000 7fd2d7400700 1 --2- 172.21.15.89:0/3893813521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2d815d640 0x7fd2d80d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:19.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.354+0000 7fd2d7400700 1 -- 172.21.15.89:0/3893813521 >> v2:172.21.15.113:3300/0 conn(0x7fd2d80ceec0 msgr2=0x7fd2d81612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:19.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.354+0000 7fd2d7400700 1 --2- 172.21.15.89:0/3893813521 >> v2:172.21.15.113:3300/0 conn(0x7fd2d80ceec0 0x7fd2d81612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:19.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.354+0000 7fd2d7400700 1 -- 172.21.15.89:0/3893813521 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d80d2880 con 0x7fd2d8161800 2024-04-02T07:55:19.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.354+0000 7fd2d7400700 1 --2- 172.21.15.89:0/3893813521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2d8161800 0x7fd2d80cdba0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fd2d808c1c0 tx=0x7fd2d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8927e0430d73342 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:19.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.355+0000 7fd2d6000700 1 -- 172.21.15.89:0/3893813521 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d000e070 con 0x7fd2d8161800 2024-04-02T07:55:19.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.355+0000 7fd2d6000700 1 -- 172.21.15.89:0/3893813521 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d000b9c0 con 0x7fd2d8161800 2024-04-02T07:55:19.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.355+0000 7fd2d6000700 1 -- 172.21.15.89:0/3893813521 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d000bd70 con 0x7fd2d8161800 2024-04-02T07:55:19.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.355+0000 7fd2dde00700 1 -- 172.21.15.89:0/3893813521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2d8161800 msgr2=0x7fd2d80cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:19.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.355+0000 7fd2dde00700 1 --2- 172.21.15.89:0/3893813521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2d8161800 0x7fd2d80cdba0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fd2d808c1c0 tx=0x7fd2d000b330 comp rx=0 tx=0).stop 2024-04-02T07:55:19.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.355+0000 7fd2dde00700 1 -- 172.21.15.89:0/3893813521 shutdown_connections 2024-04-02T07:55:19.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.355+0000 7fd2dde00700 1 --2- 172.21.15.89:0/3893813521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2d8161800 0x7fd2d80cdba0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:19.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.355+0000 7fd2dde00700 1 --2- 172.21.15.89:0/3893813521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2d815d640 0x7fd2d80d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:19.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.355+0000 7fd2dde00700 1 --2- 172.21.15.89:0/3893813521 >> v2:172.21.15.113:3300/0 conn(0x7fd2d80ceec0 0x7fd2d81612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:19.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.355+0000 7fd2dde00700 1 -- 172.21.15.89:0/3893813521 >> 172.21.15.89:0/3893813521 conn(0x7fd2d808aff0 msgr2=0x7fd2d80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:19.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.355+0000 7fd2dde00700 1 -- 172.21.15.89:0/3893813521 shutdown_connections 2024-04-02T07:55:19.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.355+0000 7fd2dde00700 1 -- 172.21.15.89:0/3893813521 wait complete. 2024-04-02T07:55:19.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.357+0000 7fd2dde00700 1 Processor -- start 2024-04-02T07:55:19.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.357+0000 7fd2dde00700 1 -- start start 2024-04-02T07:55:19.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.358+0000 7fd2dde00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2d80ceec0 0x7fd2d8157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:19.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.358+0000 7fd2dde00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2d815d640 0x7fd2d80c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:19.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.358+0000 7fd2dde00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2d8161800 0x7fd2d815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:19.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.358+0000 7fd2dde00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80c8df0 con 0x7fd2d8161800 2024-04-02T07:55:19.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.358+0000 7fd2dde00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d8157fe0 con 0x7fd2d80ceec0 2024-04-02T07:55:19.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.358+0000 7fd2dde00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80c81e0 con 0x7fd2d815d640 2024-04-02T07:55:19.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.358+0000 7fd2d7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2d8161800 0x7fd2d815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:19.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.358+0000 7fd2cfe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2d815d640 0x7fd2d80c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:19.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.358+0000 7fd2d6a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2d80ceec0 0x7fd2d8157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:19.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.358+0000 7fd2cfe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2d815d640 0x7fd2d80c9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:38698/0 (socket says 172.21.15.89:38698) 2024-04-02T07:55:19.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.358+0000 7fd2cfe00700 1 -- 172.21.15.89:0/3894127727 learned_addr learned my addr 172.21.15.89:0/3894127727 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:19.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.358+0000 7fd2d7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2d8161800 0x7fd2d815f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41128/0 (socket says 172.21.15.89:41128) 2024-04-02T07:55:19.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.358+0000 7fd2d6a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2d80ceec0 0x7fd2d8157aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:34800/0 (socket says 172.21.15.89:34800) 2024-04-02T07:55:19.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.359+0000 7fd2cfe00700 1 -- 172.21.15.89:0/3894127727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2d80ceec0 msgr2=0x7fd2d8157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:19.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.359+0000 7fd2cfe00700 1 --2- 172.21.15.89:0/3894127727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2d80ceec0 0x7fd2d8157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:19.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.359+0000 7fd2cfe00700 1 -- 172.21.15.89:0/3894127727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2d8161800 msgr2=0x7fd2d815f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:19.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.359+0000 7fd2cfe00700 1 --2- 172.21.15.89:0/3894127727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2d8161800 0x7fd2d815f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:19.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.359+0000 7fd2cfe00700 1 -- 172.21.15.89:0/3894127727 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d0009b30 con 0x7fd2d815d640 2024-04-02T07:55:19.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.359+0000 7fd2d7400700 1 --2- 172.21.15.89:0/3894127727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2d8161800 0x7fd2d815f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:55:19.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.359+0000 7fd2cfe00700 1 --2- 172.21.15.89:0/3894127727 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2d815d640 0x7fd2d80c9280 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000d800 tx=0x7fd2c000db10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:19.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.359+0000 7fd2cf400700 1 -- 172.21.15.89:0/3894127727 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c00125f0 con 0x7fd2d815d640 2024-04-02T07:55:19.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.359+0000 7fd2cf400700 1 -- 172.21.15.89:0/3894127727 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c0012750 con 0x7fd2d815d640 2024-04-02T07:55:19.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.359+0000 7fd2dde00700 1 -- 172.21.15.89:0/3894127727 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d80c84c0 con 0x7fd2d815d640 2024-04-02T07:55:19.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.359+0000 7fd2cf400700 1 -- 172.21.15.89:0/3894127727 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0012b00 con 0x7fd2d815d640 2024-04-02T07:55:19.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.359+0000 7fd2dde00700 1 -- 172.21.15.89:0/3894127727 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d80c8a80 con 0x7fd2d815d640 2024-04-02T07:55:19.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.360+0000 7fd2dde00700 1 -- 172.21.15.89:0/3894127727 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d80c8f60 con 0x7fd2d815d640 2024-04-02T07:55:19.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.361+0000 7fd2cf400700 1 -- 172.21.15.89:0/3894127727 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd2c0012c60 con 0x7fd2d815d640 2024-04-02T07:55:19.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.362+0000 7fd2cf400700 1 --2- 172.21.15.89:0/3894127727 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2b406adb0 0x7fd2b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:19.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.362+0000 7fd2cf400700 1 -- 172.21.15.89:0/3894127727 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6007+0+0 (secure 0 0 0) 0x7fd2c0089700 con 0x7fd2d815d640 2024-04-02T07:55:19.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.362+0000 7fd2d6a00700 1 --2- 172.21.15.89:0/3894127727 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2b406adb0 0x7fd2b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:19.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.363+0000 7fd2d6a00700 1 --2- 172.21.15.89:0/3894127727 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2b406adb0 0x7fd2b406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8006b20 tx=0x7fd2c8008040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:19.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.367+0000 7fd2cf400700 1 -- 172.21.15.89:0/3894127727 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd2c0054600 con 0x7fd2d815d640 2024-04-02T07:55:19.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.612+0000 7fd2dde00700 1 -- 172.21.15.89:0/3894127727 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2d8067720 con 0x7fd2d815d640 2024-04-02T07:55:19.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.612+0000 7fd2cf400700 1 -- 172.21.15.89:0/3894127727 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+3529 (secure 0 0 0) 0x7fd2c0057c20 con 0x7fd2d815d640 2024-04-02T07:55:19.613 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:19.614 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":59,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1967870793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1967870793},{"type":"v1","addr":"172.21.15.89:6837","nonce":1967870793}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1050024225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1050024225},{"type":"v1","addr":"172.21.15.113:6835","nonce":1050024225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7447,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/235320877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":235320877},{"type":"v1","addr":"172.21.15.113:6834","nonce":235320877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/919993773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":919993773},{"type":"v1","addr":"172.21.15.89:6836","nonce":919993773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[]} 2024-04-02T07:55:19.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.615+0000 7fd2dde00700 1 -- 172.21.15.89:0/3894127727 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2b406adb0 msgr2=0x7fd2b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.615+0000 7fd2dde00700 1 --2- 172.21.15.89:0/3894127727 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2b406adb0 0x7fd2b406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8006b20 tx=0x7fd2c8008040 comp rx=0 tx=0).stop 2024-04-02T07:55:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.615+0000 7fd2dde00700 1 -- 172.21.15.89:0/3894127727 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2d815d640 msgr2=0x7fd2d80c9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.615+0000 7fd2dde00700 1 --2- 172.21.15.89:0/3894127727 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2d815d640 0x7fd2d80c9280 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000d800 tx=0x7fd2c000db10 comp rx=0 tx=0).stop 2024-04-02T07:55:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.615+0000 7fd2dde00700 1 -- 172.21.15.89:0/3894127727 shutdown_connections 2024-04-02T07:55:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.615+0000 7fd2dde00700 1 --2- 172.21.15.89:0/3894127727 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2b406adb0 0x7fd2b406d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.615+0000 7fd2dde00700 1 --2- 172.21.15.89:0/3894127727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2d80ceec0 0x7fd2d8157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.615+0000 7fd2dde00700 1 --2- 172.21.15.89:0/3894127727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2d8161800 0x7fd2d815f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.615+0000 7fd2dde00700 1 --2- 172.21.15.89:0/3894127727 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2d815d640 0x7fd2d80c9280 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.615+0000 7fd2dde00700 1 -- 172.21.15.89:0/3894127727 >> 172.21.15.89:0/3894127727 conn(0x7fd2d808aff0 msgr2=0x7fd2d815aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.615+0000 7fd2dde00700 1 -- 172.21.15.89:0/3894127727 shutdown_connections 2024-04-02T07:55:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:19.615+0000 7fd2dde00700 1 -- 172.21.15.89:0/3894127727 wait complete. 2024-04-02T07:55:19.617 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 59 2024-04-02T07:55:19.634 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T07:55:19.634 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T07:55:19.634 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T07:55:19.634 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:55:19.635 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T07:55:19.635 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T07:55:19.635 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T07:55:19.635 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:55:19.635 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T07:55:19.635 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T07:55:19.636 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T07:55:19.636 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:55:19.636 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T07:55:19.636 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T07:55:19.636 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T07:55:19.636 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:55:19.652 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T07:55:19.652 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T07:55:19.653 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T07:55:19.653 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T07:55:19.654 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T07:55:19.654 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T07:55:19.695 INFO:tasks.ceph.mds.d:Started 2024-04-02T07:55:19.696 DEBUG:teuthology.parallel:result is None 2024-04-02T07:55:19.697 INFO:tasks.ceph.mds.a:Started 2024-04-02T07:55:19.697 DEBUG:teuthology.parallel:result is None 2024-04-02T07:55:19.699 INFO:tasks.ceph.mds.c:Started 2024-04-02T07:55:19.700 DEBUG:teuthology.parallel:result is None 2024-04-02T07:55:19.835 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T07:55:19.839 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T07:55:19.840 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T07:55:21.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-04-02T07:55:25.651 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T07:55:25.652 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T07:55:25.656 INFO:tasks.ceph.mds.b:Started 2024-04-02T07:55:25.656 DEBUG:teuthology.parallel:result is None 2024-04-02T07:55:25.657 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T07:55:25.803 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T07:55:25.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.862+0000 7f1cb4a00700 1 Processor -- start 2024-04-02T07:55:25.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.863+0000 7f1cb4a00700 1 -- start start 2024-04-02T07:55:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.864+0000 7f1cb4a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1cb00d06a0 0x7f1cb015e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.864+0000 7f1cb4a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb015eda0 con 0x7f1cb00cd5c0 2024-04-02T07:55:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.864+0000 7f1cb4a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb0099dd0 con 0x7f1cb006fa20 2024-04-02T07:55:25.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.864+0000 7f1cb4a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cb009a0b0 con 0x7f1cb00d06a0 2024-04-02T07:55:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.864+0000 7f1cade00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1cb00cd5c0 0x7f1cb006f310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40866/0 (socket says 172.21.15.89:40866) 2024-04-02T07:55:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.864+0000 7f1cae800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1cb00d06a0 0x7f1cb015e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.864+0000 7f1cade00700 1 -- 172.21.15.89:0/587768692 learned_addr learned my addr 172.21.15.89:0/587768692 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.864+0000 7f1caca00700 1 -- 172.21.15.89:0/587768692 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224181030 0 0) 0x7f1ca0001110 con 0x7f1cb00cd5c0 2024-04-02T07:55:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.864+0000 7f1cae800700 1 -- 172.21.15.89:0/587768692 >> v1:172.21.15.113:6789/0 conn(0x7f1cb006fa20 legacy=0x7f1cb015cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.865+0000 7f1cae800700 1 -- 172.21.15.89:0/587768692 >> v1:172.21.15.89:6789/0 conn(0x7f1cb00cd5c0 legacy=0x7f1cb006f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.865+0000 7f1cae800700 1 -- 172.21.15.89:0/587768692 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb009a390 con 0x7f1cb00d06a0 2024-04-02T07:55:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.865+0000 7f1cae800700 1 --2- 172.21.15.89:0/587768692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cb00d06a0 0x7f1cb015e860 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8009dc0 tx=0x7f1ca800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec25e277a5f27c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.865+0000 7f1caca00700 1 -- 172.21.15.89:0/587768692 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca800e070 con 0x7f1cb00d06a0 2024-04-02T07:55:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.865+0000 7f1caca00700 1 -- 172.21.15.89:0/587768692 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca800bad0 con 0x7f1cb00d06a0 2024-04-02T07:55:25.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.865+0000 7f1caca00700 1 -- 172.21.15.89:0/587768692 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca800be60 con 0x7f1cb00d06a0 2024-04-02T07:55:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.865+0000 7f1cb4a00700 1 -- 172.21.15.89:0/587768692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cb00d06a0 msgr2=0x7f1cb015e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.865+0000 7f1cb4a00700 1 --2- 172.21.15.89:0/587768692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cb00d06a0 0x7f1cb015e860 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8009dc0 tx=0x7f1ca800b330 comp rx=0 tx=0).stop 2024-04-02T07:55:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.866+0000 7f1cb4a00700 1 -- 172.21.15.89:0/587768692 shutdown_connections 2024-04-02T07:55:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.866+0000 7f1cb4a00700 1 --2- 172.21.15.89:0/587768692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cb00d06a0 0x7f1cb015e860 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.866+0000 7f1cb4a00700 1 -- 172.21.15.89:0/587768692 >> 172.21.15.89:0/587768692 conn(0x7f1cb00d1370 msgr2=0x7f1cb00d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.866+0000 7f1cb4a00700 1 -- 172.21.15.89:0/587768692 shutdown_connections 2024-04-02T07:55:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.866+0000 7f1cb4a00700 1 -- 172.21.15.89:0/587768692 wait complete. 2024-04-02T07:55:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.867+0000 7f1cb4a00700 1 Processor -- start 2024-04-02T07:55:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.868+0000 7f1cb4a00700 1 -- start start 2024-04-02T07:55:25.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.868+0000 7f1cb4a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cb006fa20 0x7f1cb01f7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:25.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.868+0000 7f1cb4a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cb00cd5c0 0x7f1cb009e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.868+0000 7f1cb4a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cb00d06a0 0x7f1cb00cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cb4a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb01f7db0 con 0x7f1cb006fa20 2024-04-02T07:55:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cb4a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb015ba60 con 0x7f1cb00d06a0 2024-04-02T07:55:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cb4a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb015a6b0 con 0x7f1cb00cd5c0 2024-04-02T07:55:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cade00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cb006fa20 0x7f1cb01f7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cade00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cb006fa20 0x7f1cb01f7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41174/0 (socket says 172.21.15.89:41174) 2024-04-02T07:55:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cade00700 1 -- 172.21.15.89:0/2398281057 learned_addr learned my addr 172.21.15.89:0/2398281057 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cae800700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cb00d06a0 0x7f1cb00cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cad400700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cb00cd5c0 0x7f1cb009e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cade00700 1 -- 172.21.15.89:0/2398281057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cb00cd5c0 msgr2=0x7f1cb009e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cade00700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cb00cd5c0 0x7f1cb009e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cade00700 1 -- 172.21.15.89:0/2398281057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cb00d06a0 msgr2=0x7f1cb00cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cade00700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cb00d06a0 0x7f1cb00cdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cade00700 1 -- 172.21.15.89:0/2398281057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca8009a70 con 0x7f1cb006fa20 2024-04-02T07:55:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cae800700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cb00d06a0 0x7f1cb00cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:55:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cad400700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cb00cd5c0 0x7f1cb009e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:55:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1cade00700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cb006fa20 0x7f1cb01f7870 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000c730 tx=0x7f1ca000caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1ca6000700 1 -- 172.21.15.89:0/2398281057 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0004c80 con 0x7f1cb006fa20 2024-04-02T07:55:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.869+0000 7f1ca6000700 1 -- 172.21.15.89:0/2398281057 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca0004e00 con 0x7f1cb006fa20 2024-04-02T07:55:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.870+0000 7f1ca6000700 1 -- 172.21.15.89:0/2398281057 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0007310 con 0x7f1cb006fa20 2024-04-02T07:55:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.870+0000 7f1cb4a00700 1 -- 172.21.15.89:0/2398281057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb015a990 con 0x7f1cb006fa20 2024-04-02T07:55:25.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.870+0000 7f1cb4a00700 1 -- 172.21.15.89:0/2398281057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb015af50 con 0x7f1cb006fa20 2024-04-02T07:55:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.871+0000 7f1cb4a00700 1 -- 172.21.15.89:0/2398281057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb00b3f70 con 0x7f1cb006fa20 2024-04-02T07:55:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.872+0000 7f1ca6000700 1 -- 172.21.15.89:0/2398281057 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1ca0007470 con 0x7f1cb006fa20 2024-04-02T07:55:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.872+0000 7f1ca6000700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1c9006adb0 0x7f1c9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.873+0000 7f1ca6000700 1 -- 172.21.15.89:0/2398281057 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6007+0+0 (secure 0 0 0) 0x7f1ca0089230 con 0x7f1cb006fa20 2024-04-02T07:55:25.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.873+0000 7f1cad400700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1c9006adb0 0x7f1c9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:25.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.873+0000 7f1cad400700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1c9006adb0 0x7f1c9006d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1cb015c2b0 tx=0x7f1c9c009040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:25.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:25.877+0000 7f1ca6000700 1 -- 172.21.15.89:0/2398281057 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ca0054130 con 0x7f1cb006fa20 2024-04-02T07:55:26.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:26.099+0000 7f1cb4a00700 1 -- 172.21.15.89:0/2398281057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1cb015eda0 con 0x7f1cb006fa20 2024-04-02T07:55:27.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.048+0000 7f1ca6000700 1 -- 172.21.15.89:0/2398281057 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v136) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1cb015eda0 con 0x7f1cb006fa20 2024-04-02T07:55:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.051+0000 7f1cb4a00700 1 -- 172.21.15.89:0/2398281057 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1c9006adb0 msgr2=0x7f1c9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.051+0000 7f1cb4a00700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1c9006adb0 0x7f1c9006d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1cb015c2b0 tx=0x7f1c9c009040 comp rx=0 tx=0).stop 2024-04-02T07:55:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.051+0000 7f1cb4a00700 1 -- 172.21.15.89:0/2398281057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cb006fa20 msgr2=0x7f1cb01f7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.051+0000 7f1cb4a00700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cb006fa20 0x7f1cb01f7870 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000c730 tx=0x7f1ca000caf0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.051+0000 7f1cb4a00700 1 -- 172.21.15.89:0/2398281057 shutdown_connections 2024-04-02T07:55:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.051+0000 7f1cb4a00700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1c9006adb0 0x7f1c9006d270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.051+0000 7f1cb4a00700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cb00d06a0 0x7f1cb00cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.051+0000 7f1cb4a00700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cb006fa20 0x7f1cb01f7870 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.051+0000 7f1cb4a00700 1 --2- 172.21.15.89:0/2398281057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cb00cd5c0 0x7f1cb009e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.051+0000 7f1cb4a00700 1 -- 172.21.15.89:0/2398281057 >> 172.21.15.89:0/2398281057 conn(0x7f1cb00d1370 msgr2=0x7f1cb006d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:27.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.051+0000 7f1cb4a00700 1 -- 172.21.15.89:0/2398281057 shutdown_connections 2024-04-02T07:55:27.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.051+0000 7f1cb4a00700 1 -- 172.21.15.89:0/2398281057 wait complete. 2024-04-02T07:55:27.053 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T07:55:27.074 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T07:55:27.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.266+0000 7ff070800700 1 Processor -- start 2024-04-02T07:55:27.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.267+0000 7ff070800700 1 -- start start 2024-04-02T07:55:27.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.268+0000 7ff070800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff068160f10 0x7ff0680d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:27.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.268+0000 7ff070800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff068161590 0x7ff068161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:27.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.268+0000 7ff070800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0680ceda0 con 0x7ff0680cf230 2024-04-02T07:55:27.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.268+0000 7ff070800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff068161f80 con 0x7ff068160f10 2024-04-02T07:55:27.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.268+0000 7ff070800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0681612f0 con 0x7ff068161590 2024-04-02T07:55:27.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.268+0000 7ff06de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff068160f10 0x7ff0680d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:27.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.268+0000 7ff06d400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff0680cf230 0x7ff0680d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40882/0 (socket says 172.21.15.89:40882) 2024-04-02T07:55:27.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.268+0000 7ff06d400700 1 -- 172.21.15.89:0/188557191 learned_addr learned my addr 172.21.15.89:0/188557191 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:27.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.268+0000 7ff06e800700 1 --2- 172.21.15.89:0/188557191 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff068161590 0x7ff068161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:27.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.269+0000 7ff06de00700 1 -- 172.21.15.89:0/188557191 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff068161590 msgr2=0x7ff068161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:27.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.269+0000 7ff06de00700 1 --2- 172.21.15.89:0/188557191 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff068161590 0x7ff068161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.269+0000 7ff06de00700 1 -- 172.21.15.89:0/188557191 >> v1:172.21.15.89:6789/0 conn(0x7ff0680cf230 legacy=0x7ff0680d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:27.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.269+0000 7ff06de00700 1 -- 172.21.15.89:0/188557191 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0680ce470 con 0x7ff068160f10 2024-04-02T07:55:27.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.269+0000 7ff06ca00700 1 -- 172.21.15.89:0/188557191 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361997107 0 0) 0x7ff0680ceda0 con 0x7ff0680cf230 2024-04-02T07:55:27.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.269+0000 7ff06de00700 1 --2- 172.21.15.89:0/188557191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff068160f10 0x7ff0680d18b0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009840 tx=0x7ff05c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf2a7ad5ff3f7a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:27.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.269+0000 7ff06ca00700 1 -- 172.21.15.89:0/188557191 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c011830 con 0x7ff068160f10 2024-04-02T07:55:27.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.269+0000 7ff06ca00700 1 -- 172.21.15.89:0/188557191 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05c011990 con 0x7ff068160f10 2024-04-02T07:55:27.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.269+0000 7ff06ca00700 1 -- 172.21.15.89:0/188557191 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c011d20 con 0x7ff068160f10 2024-04-02T07:55:27.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.270+0000 7ff070800700 1 -- 172.21.15.89:0/188557191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff068160f10 msgr2=0x7ff0680d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:27.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.270+0000 7ff070800700 1 --2- 172.21.15.89:0/188557191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff068160f10 0x7ff0680d18b0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009840 tx=0x7ff05c009b50 comp rx=0 tx=0).stop 2024-04-02T07:55:27.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.270+0000 7ff070800700 1 -- 172.21.15.89:0/188557191 shutdown_connections 2024-04-02T07:55:27.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.270+0000 7ff070800700 1 --2- 172.21.15.89:0/188557191 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff068161590 0x7ff068161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.270+0000 7ff070800700 1 --2- 172.21.15.89:0/188557191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff068160f10 0x7ff0680d18b0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.270+0000 7ff070800700 1 -- 172.21.15.89:0/188557191 >> 172.21.15.89:0/188557191 conn(0x7ff06808aff0 msgr2=0x7ff0680d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:27.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.270+0000 7ff070800700 1 -- 172.21.15.89:0/188557191 shutdown_connections 2024-04-02T07:55:27.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.270+0000 7ff070800700 1 -- 172.21.15.89:0/188557191 wait complete. 2024-04-02T07:55:27.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.272+0000 7ff070800700 1 Processor -- start 2024-04-02T07:55:27.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.272+0000 7ff070800700 1 -- start start 2024-04-02T07:55:27.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.273+0000 7ff070800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0680cf230 0x7ff0681589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:27.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.273+0000 7ff070800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff068160f10 0x7ff068157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:27.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.273+0000 7ff070800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff068161590 0x7ff06815e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:27.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.273+0000 7ff070800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0681582c0 con 0x7ff0680cf230 2024-04-02T07:55:27.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.274+0000 7ff070800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06815f2b0 con 0x7ff068161590 2024-04-02T07:55:27.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.274+0000 7ff070800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06815f590 con 0x7ff068160f10 2024-04-02T07:55:27.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.274+0000 7ff06de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0680cf230 0x7ff0681589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:27.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.274+0000 7ff06de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0680cf230 0x7ff0681589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41178/0 (socket says 172.21.15.89:41178) 2024-04-02T07:55:27.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.274+0000 7ff06de00700 1 -- 172.21.15.89:0/1173954826 learned_addr learned my addr 172.21.15.89:0/1173954826 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:27.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.274+0000 7ff06d400700 1 --2- 172.21.15.89:0/1173954826 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff068160f10 0x7ff068157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:27.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.274+0000 7ff06e800700 1 --2- 172.21.15.89:0/1173954826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff068161590 0x7ff06815e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:27.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.274+0000 7ff06de00700 1 -- 172.21.15.89:0/1173954826 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff068160f10 msgr2=0x7ff068157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:27.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.274+0000 7ff06de00700 1 --2- 172.21.15.89:0/1173954826 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff068160f10 0x7ff068157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.274+0000 7ff06de00700 1 -- 172.21.15.89:0/1173954826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff068161590 msgr2=0x7ff06815e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:27.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.274+0000 7ff06de00700 1 --2- 172.21.15.89:0/1173954826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff068161590 0x7ff06815e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.274+0000 7ff06de00700 1 -- 172.21.15.89:0/1173954826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c0094a0 con 0x7ff0680cf230 2024-04-02T07:55:27.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.275+0000 7ff06de00700 1 --2- 172.21.15.89:0/1173954826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0680cf230 0x7ff0681589b0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009b90 tx=0x7ff05c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:27.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.275+0000 7ff062000700 1 -- 172.21.15.89:0/1173954826 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c0036a0 con 0x7ff0680cf230 2024-04-02T07:55:27.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.275+0000 7ff062000700 1 -- 172.21.15.89:0/1173954826 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05c01e070 con 0x7ff0680cf230 2024-04-02T07:55:27.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.275+0000 7ff062000700 1 -- 172.21.15.89:0/1173954826 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c003a60 con 0x7ff0680cf230 2024-04-02T07:55:27.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.275+0000 7ff070800700 1 -- 172.21.15.89:0/1173954826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff06815f870 con 0x7ff0680cf230 2024-04-02T07:55:27.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.275+0000 7ff070800700 1 -- 172.21.15.89:0/1173954826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0681e70b0 con 0x7ff0680cf230 2024-04-02T07:55:27.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.276+0000 7ff070800700 1 -- 172.21.15.89:0/1173954826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff068067720 con 0x7ff0680cf230 2024-04-02T07:55:27.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.277+0000 7ff062000700 1 -- 172.21.15.89:0/1173954826 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff05c003bc0 con 0x7ff0680cf230 2024-04-02T07:55:27.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.278+0000 7ff062000700 1 --2- 172.21.15.89:0/1173954826 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff04c06adb0 0x7ff04c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:27.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.278+0000 7ff062000700 1 -- 172.21.15.89:0/1173954826 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5233+0+0 (secure 0 0 0) 0x7ff05c08a120 con 0x7ff0680cf230 2024-04-02T07:55:27.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.278+0000 7ff06d400700 1 --2- 172.21.15.89:0/1173954826 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff04c06adb0 0x7ff04c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:27.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.279+0000 7ff06d400700 1 --2- 172.21.15.89:0/1173954826 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff04c06adb0 0x7ff04c06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff068161b60 tx=0x7ff058007e70 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:27.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.283+0000 7ff062000700 1 -- 172.21.15.89:0/1173954826 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff05c055450 con 0x7ff0680cf230 2024-04-02T07:55:27.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.562+0000 7ff070800700 1 -- 172.21.15.89:0/1173954826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff06815d140 con 0x7ff0680cf230 2024-04-02T07:55:27.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.562+0000 7ff062000700 1 -- 172.21.15.89:0/1173954826 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff05c05de60 con 0x7ff0680cf230 2024-04-02T07:55:27.563 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:27.563 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:55:27.563 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T07:55:27.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.563 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T07:55:27.563 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T07:55:27.564 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.564 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:55:27.564 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:55:27.564 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:55:27.564 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.564 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.564 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T07:55:27.564 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T07:55:27.564 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.564 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:55:27.564 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:55:27.564 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.565 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.569 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T07:55:27.569 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T07:55:27.569 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.569 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:55:27.569 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:55:27.569 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.569 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2024-04-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T07:55:27.576 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T07:55:27.576 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:55:27.576 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:55:27.576 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:55:27.576 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:55:27.576 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.576 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:27.576 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T07:55:27.576 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:55:27.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.565+0000 7ff070800700 1 -- 172.21.15.89:0/1173954826 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff04c06adb0 msgr2=0x7ff04c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:27.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.565+0000 7ff070800700 1 --2- 172.21.15.89:0/1173954826 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff04c06adb0 0x7ff04c06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff068161b60 tx=0x7ff058007e70 comp rx=0 tx=0).stop 2024-04-02T07:55:27.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.565+0000 7ff070800700 1 -- 172.21.15.89:0/1173954826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0680cf230 msgr2=0x7ff0681589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:27.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.565+0000 7ff070800700 1 --2- 172.21.15.89:0/1173954826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0680cf230 0x7ff0681589b0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009b90 tx=0x7ff05c004750 comp rx=0 tx=0).stop 2024-04-02T07:55:27.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.565+0000 7ff070800700 1 -- 172.21.15.89:0/1173954826 shutdown_connections 2024-04-02T07:55:27.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.565+0000 7ff070800700 1 --2- 172.21.15.89:0/1173954826 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff04c06adb0 0x7ff04c06d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.565+0000 7ff070800700 1 --2- 172.21.15.89:0/1173954826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff068161590 0x7ff06815e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.565+0000 7ff070800700 1 --2- 172.21.15.89:0/1173954826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0680cf230 0x7ff0681589b0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.565+0000 7ff070800700 1 --2- 172.21.15.89:0/1173954826 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff068160f10 0x7ff068157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.565+0000 7ff070800700 1 -- 172.21.15.89:0/1173954826 >> 172.21.15.89:0/1173954826 conn(0x7ff06808aff0 msgr2=0x7ff0680b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:27.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.565+0000 7ff070800700 1 -- 172.21.15.89:0/1173954826 shutdown_connections 2024-04-02T07:55:27.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.565+0000 7ff070800700 1 -- 172.21.15.89:0/1173954826 wait complete. 2024-04-02T07:55:27.584 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:55:27.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.775+0000 7f172e200700 1 Processor -- start 2024-04-02T07:55:27.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.775+0000 7f172e200700 1 -- start start 2024-04-02T07:55:27.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.776+0000 7f172e200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1728160f10 0x7f17280d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:27.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.776+0000 7f172e200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f17280cf230 0x7f17280d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:27.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.776+0000 7f172e200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17280ceda0 con 0x7f1728161590 2024-04-02T07:55:27.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.776+0000 7f172e200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17280d23d0 con 0x7f17280cf230 2024-04-02T07:55:27.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.776+0000 7f172e200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17281612f0 con 0x7f1728160f10 2024-04-02T07:55:27.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.776+0000 7f1727e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1728161590 0x7f17280b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40886/0 (socket says 172.21.15.89:40886) 2024-04-02T07:55:27.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.776+0000 7f1727e00700 1 -- 172.21.15.89:0/3381796118 learned_addr learned my addr 172.21.15.89:0/3381796118 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:27.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.776+0000 7f1726a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f17280cf230 0x7f17280d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:27.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.777+0000 7f1727400700 1 --2- 172.21.15.89:0/3381796118 >> v2:172.21.15.113:3300/0 conn(0x7f1728160f10 0x7f17280d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:27.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.777+0000 7f1726a00700 1 -- 172.21.15.89:0/3381796118 >> v2:172.21.15.113:3300/0 conn(0x7f1728160f10 msgr2=0x7f17280d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:27.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.777+0000 7f1726a00700 1 --2- 172.21.15.89:0/3381796118 >> v2:172.21.15.113:3300/0 conn(0x7f1728160f10 0x7f17280d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.777+0000 7f1726a00700 1 -- 172.21.15.89:0/3381796118 >> v1:172.21.15.89:6789/0 conn(0x7f1728161590 legacy=0x7f17280b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:27.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.777+0000 7f1726a00700 1 -- 172.21.15.89:0/3381796118 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17280ce470 con 0x7f17280cf230 2024-04-02T07:55:27.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.777+0000 7f1727400700 1 --2- 172.21.15.89:0/3381796118 >> v2:172.21.15.113:3300/0 conn(0x7f1728160f10 0x7f17280d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:27.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.777+0000 7f1726a00700 1 --2- 172.21.15.89:0/3381796118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17280cf230 0x7f17280d1e90 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f17280d1080 tx=0x7f1714009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d163f182acd1ce server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:27.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.777+0000 7f1726000700 1 -- 172.21.15.89:0/3381796118 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1714011830 con 0x7f17280cf230 2024-04-02T07:55:27.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.777+0000 7f1726000700 1 -- 172.21.15.89:0/3381796118 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1714011990 con 0x7f17280cf230 2024-04-02T07:55:27.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.777+0000 7f1726000700 1 -- 172.21.15.89:0/3381796118 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1714011d20 con 0x7f17280cf230 2024-04-02T07:55:27.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.778+0000 7f172e200700 1 -- 172.21.15.89:0/3381796118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17280cf230 msgr2=0x7f17280d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:27.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.778+0000 7f172e200700 1 --2- 172.21.15.89:0/3381796118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17280cf230 0x7f17280d1e90 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f17280d1080 tx=0x7f1714009b20 comp rx=0 tx=0).stop 2024-04-02T07:55:27.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.778+0000 7f172e200700 1 -- 172.21.15.89:0/3381796118 shutdown_connections 2024-04-02T07:55:27.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.778+0000 7f172e200700 1 --2- 172.21.15.89:0/3381796118 >> v2:172.21.15.113:3300/0 conn(0x7f1728160f10 0x7f17280d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.778+0000 7f172e200700 1 --2- 172.21.15.89:0/3381796118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17280cf230 0x7f17280d1e90 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.778+0000 7f172e200700 1 -- 172.21.15.89:0/3381796118 >> 172.21.15.89:0/3381796118 conn(0x7f172808aff0 msgr2=0x7f17280d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:27.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.778+0000 7f172e200700 1 -- 172.21.15.89:0/3381796118 shutdown_connections 2024-04-02T07:55:27.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.778+0000 7f172e200700 1 -- 172.21.15.89:0/3381796118 wait complete. 2024-04-02T07:55:27.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.780+0000 7f172e200700 1 Processor -- start 2024-04-02T07:55:27.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.780+0000 7f172e200700 1 -- start start 2024-04-02T07:55:27.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f172e200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1728160f10 0x7f172809f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:27.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f172e200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1728161590 0x7f17280c9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:27.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f172e200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f172814feb0 0x7f17280a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:27.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f172e200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1728150730 con 0x7f1728160f10 2024-04-02T07:55:27.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f172e200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172809f980 con 0x7f172814feb0 2024-04-02T07:55:27.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f172e200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f172809fc40 con 0x7f1728161590 2024-04-02T07:55:27.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f1727400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1728160f10 0x7f172809f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:27.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f1727400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1728160f10 0x7f172809f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41194/0 (socket says 172.21.15.89:41194) 2024-04-02T07:55:27.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f1727400700 1 -- 172.21.15.89:0/2322406533 learned_addr learned my addr 172.21.15.89:0/2322406533 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:27.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f1726a00700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1728161590 0x7f17280c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:27.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f1727e00700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f172814feb0 0x7f17280a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:27.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f1727400700 1 -- 172.21.15.89:0/2322406533 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1728161590 msgr2=0x7f17280c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:27.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f1727400700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1728161590 0x7f17280c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f1727400700 1 -- 172.21.15.89:0/2322406533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f172814feb0 msgr2=0x7f17280a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:27.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f1727400700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f172814feb0 0x7f17280a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f1727400700 1 -- 172.21.15.89:0/2322406533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17140094a0 con 0x7f1728160f10 2024-04-02T07:55:27.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.781+0000 7f1726a00700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1728161590 0x7f17280c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:27.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.782+0000 7f1727400700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1728160f10 0x7f172809f440 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f171800e760 tx=0x7f171800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:27.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.782+0000 7f171f400700 1 -- 172.21.15.89:0/2322406533 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718013610 con 0x7f1728160f10 2024-04-02T07:55:27.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.782+0000 7f171f400700 1 -- 172.21.15.89:0/2322406533 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1718013770 con 0x7f1728160f10 2024-04-02T07:55:27.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.782+0000 7f172e200700 1 -- 172.21.15.89:0/2322406533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f172809ff20 con 0x7f1728160f10 2024-04-02T07:55:27.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.782+0000 7f171f400700 1 -- 172.21.15.89:0/2322406533 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718013b20 con 0x7f1728160f10 2024-04-02T07:55:27.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.782+0000 7f172e200700 1 -- 172.21.15.89:0/2322406533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17280a04e0 con 0x7f1728160f10 2024-04-02T07:55:27.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.783+0000 7f172e200700 1 -- 172.21.15.89:0/2322406533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1728067720 con 0x7f1728160f10 2024-04-02T07:55:27.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.784+0000 7f171f400700 1 -- 172.21.15.89:0/2322406533 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1718013c80 con 0x7f1728160f10 2024-04-02T07:55:27.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.785+0000 7f171f400700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f170806adb0 0x7f170806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:27.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.785+0000 7f171f400700 1 -- 172.21.15.89:0/2322406533 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5233+0+0 (secure 0 0 0) 0x7f1718089040 con 0x7f1728160f10 2024-04-02T07:55:27.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.785+0000 7f1726a00700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f170806adb0 0x7f170806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:27.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.786+0000 7f1726a00700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f170806adb0 0x7f170806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f172808d240 tx=0x7f1714008030 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:27.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.789+0000 7f171f400700 1 -- 172.21.15.89:0/2322406533 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f17180541d0 con 0x7f1728160f10 2024-04-02T07:55:27.982 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-04-02T07:55:27.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.993+0000 7f172e200700 1 -- 172.21.15.89:0/2322406533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f172806ba00 con 0x7f1728160f10 2024-04-02T07:55:27.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.994+0000 7f171f400700 1 -- 172.21.15.89:0/2322406533 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+12971 (secure 0 0 0) 0x7f17180577f0 con 0x7f1728160f10 2024-04-02T07:55:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:27.995 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":136,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:55:27.042996+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:55:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.996+0000 7f172e200700 1 -- 172.21.15.89:0/2322406533 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f170806adb0 msgr2=0x7f170806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.996+0000 7f172e200700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f170806adb0 0x7f170806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f172808d240 tx=0x7f1714008030 comp rx=0 tx=0).stop 2024-04-02T07:55:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.996+0000 7f172e200700 1 -- 172.21.15.89:0/2322406533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1728160f10 msgr2=0x7f172809f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.996+0000 7f172e200700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1728160f10 0x7f172809f440 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f171800e760 tx=0x7f171800eb20 comp rx=0 tx=0).stop 2024-04-02T07:55:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.996+0000 7f172e200700 1 -- 172.21.15.89:0/2322406533 shutdown_connections 2024-04-02T07:55:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.996+0000 7f172e200700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f170806adb0 0x7f170806d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.996+0000 7f172e200700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f172814feb0 0x7f17280a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.996+0000 7f172e200700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1728160f10 0x7f172809f440 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.996+0000 7f172e200700 1 --2- 172.21.15.89:0/2322406533 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1728161590 0x7f17280c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.996+0000 7f172e200700 1 -- 172.21.15.89:0/2322406533 >> 172.21.15.89:0/2322406533 conn(0x7f172808aff0 msgr2=0x7f17280cb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.997+0000 7f172e200700 1 -- 172.21.15.89:0/2322406533 shutdown_connections 2024-04-02T07:55:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:27.997+0000 7f172e200700 1 -- 172.21.15.89:0/2322406533 wait complete. 2024-04-02T07:55:28.015 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:55:28.015 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:55:28.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.209+0000 7f9d67a00700 1 Processor -- start 2024-04-02T07:55:28.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.210+0000 7f9d67a00700 1 -- start start 2024-04-02T07:55:28.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.211+0000 7f9d67a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9d6015d800 0x7f9d600725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:28.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.211+0000 7f9d67a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9d600ce230 0x7f9d600ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:28.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.211+0000 7f9d67a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d600709c0 con 0x7f9d600b2070 2024-04-02T07:55:28.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.211+0000 7f9d67a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d600b40f0 con 0x7f9d600ce230 2024-04-02T07:55:28.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.211+0000 7f9d67a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d60070c80 con 0x7f9d6015d800 2024-04-02T07:55:28.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.211+0000 7f9d5fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9d600ce230 0x7f9d600ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:28.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.211+0000 7f9d65a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9d600b2070 0x7f9d6015dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40900/0 (socket says 172.21.15.89:40900) 2024-04-02T07:55:28.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.211+0000 7f9d65a00700 1 -- 172.21.15.89:0/2473386016 learned_addr learned my addr 172.21.15.89:0/2473386016 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:28.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.211+0000 7f9d65000700 1 --2- 172.21.15.89:0/2473386016 >> v2:172.21.15.113:3300/0 conn(0x7f9d6015d800 0x7f9d600725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:28.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.211+0000 7f9d5f400700 1 -- 172.21.15.89:0/2473386016 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373494345 0 0) 0x7f9d600709c0 con 0x7f9d600b2070 2024-04-02T07:55:28.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.212+0000 7f9d5fe00700 1 -- 172.21.15.89:0/2473386016 >> v2:172.21.15.113:3300/0 conn(0x7f9d6015d800 msgr2=0x7f9d600725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:28.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.212+0000 7f9d5fe00700 1 --2- 172.21.15.89:0/2473386016 >> v2:172.21.15.113:3300/0 conn(0x7f9d6015d800 0x7f9d600725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.212+0000 7f9d5fe00700 1 -- 172.21.15.89:0/2473386016 >> v1:172.21.15.89:6789/0 conn(0x7f9d600b2070 legacy=0x7f9d6015dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:28.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.212+0000 7f9d5fe00700 1 -- 172.21.15.89:0/2473386016 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d60070f60 con 0x7f9d600ce230 2024-04-02T07:55:28.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.212+0000 7f9d5fe00700 1 --2- 172.21.15.89:0/2473386016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d600ce230 0x7f9d600ce6a0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f9d6006c100 tx=0x7f9d4c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402cdde68286608e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:28.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.212+0000 7f9d5f400700 1 -- 172.21.15.89:0/2473386016 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c010800 con 0x7f9d600ce230 2024-04-02T07:55:28.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.212+0000 7f9d5f400700 1 -- 172.21.15.89:0/2473386016 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4c010960 con 0x7f9d600ce230 2024-04-02T07:55:28.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.212+0000 7f9d5f400700 1 -- 172.21.15.89:0/2473386016 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c010cf0 con 0x7f9d600ce230 2024-04-02T07:55:28.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.212+0000 7f9d67a00700 1 -- 172.21.15.89:0/2473386016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d600ce230 msgr2=0x7f9d600ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:28.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.213+0000 7f9d67a00700 1 --2- 172.21.15.89:0/2473386016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d600ce230 0x7f9d600ce6a0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f9d6006c100 tx=0x7f9d4c009b20 comp rx=0 tx=0).stop 2024-04-02T07:55:28.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.213+0000 7f9d67a00700 1 -- 172.21.15.89:0/2473386016 shutdown_connections 2024-04-02T07:55:28.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.213+0000 7f9d67a00700 1 --2- 172.21.15.89:0/2473386016 >> v2:172.21.15.113:3300/0 conn(0x7f9d6015d800 0x7f9d600725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.213+0000 7f9d67a00700 1 --2- 172.21.15.89:0/2473386016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d600ce230 0x7f9d600ce6a0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.213+0000 7f9d67a00700 1 -- 172.21.15.89:0/2473386016 >> 172.21.15.89:0/2473386016 conn(0x7f9d600b0d70 msgr2=0x7f9d600b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:28.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.213+0000 7f9d67a00700 1 -- 172.21.15.89:0/2473386016 shutdown_connections 2024-04-02T07:55:28.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.213+0000 7f9d67a00700 1 -- 172.21.15.89:0/2473386016 wait complete. 2024-04-02T07:55:28.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.214+0000 7f9d67a00700 1 Processor -- start 2024-04-02T07:55:28.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.215+0000 7f9d67a00700 1 -- start start 2024-04-02T07:55:28.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.215+0000 7f9d67a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d600b2070 0x7f9d60152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:28.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.215+0000 7f9d67a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d600ce230 0x7f9d600ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:28.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d67a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d6015d800 0x7f9d6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:28.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d67a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d60152da0 con 0x7f9d600ce230 2024-04-02T07:55:28.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d67a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d600ac8a0 con 0x7f9d600b2070 2024-04-02T07:55:28.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d67a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d600acb80 con 0x7f9d6015d800 2024-04-02T07:55:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d5fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d600ce230 0x7f9d600ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d5fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d600ce230 0x7f9d600ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41202/0 (socket says 172.21.15.89:41202) 2024-04-02T07:55:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d5fe00700 1 -- 172.21.15.89:0/2048474126 learned_addr learned my addr 172.21.15.89:0/2048474126 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d65000700 1 --2- 172.21.15.89:0/2048474126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d600b2070 0x7f9d60152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d65a00700 1 --2- 172.21.15.89:0/2048474126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d6015d800 0x7f9d6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d5fe00700 1 -- 172.21.15.89:0/2048474126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d6015d800 msgr2=0x7f9d6006d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d5fe00700 1 --2- 172.21.15.89:0/2048474126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d6015d800 0x7f9d6006d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d5fe00700 1 -- 172.21.15.89:0/2048474126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d600b2070 msgr2=0x7f9d60152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d5fe00700 1 --2- 172.21.15.89:0/2048474126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d600b2070 0x7f9d60152860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d5fe00700 1 -- 172.21.15.89:0/2048474126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c0094a0 con 0x7f9d600ce230 2024-04-02T07:55:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.216+0000 7f9d5fe00700 1 --2- 172.21.15.89:0/2048474126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d600ce230 0x7f9d600ac050 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0090b0 tx=0x7f9d4c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.217+0000 7f9d5cc00700 1 -- 172.21.15.89:0/2048474126 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c003ab0 con 0x7f9d600ce230 2024-04-02T07:55:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.217+0000 7f9d5cc00700 1 -- 172.21.15.89:0/2048474126 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4c003c10 con 0x7f9d600ce230 2024-04-02T07:55:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.217+0000 7f9d67a00700 1 -- 172.21.15.89:0/2048474126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d600ace60 con 0x7f9d600ce230 2024-04-02T07:55:28.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.217+0000 7f9d5cc00700 1 -- 172.21.15.89:0/2048474126 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c018400 con 0x7f9d600ce230 2024-04-02T07:55:28.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.217+0000 7f9d67a00700 1 -- 172.21.15.89:0/2048474126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d600ac590 con 0x7f9d600ce230 2024-04-02T07:55:28.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.218+0000 7f9d67a00700 1 -- 172.21.15.89:0/2048474126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d600c9dd0 con 0x7f9d600ce230 2024-04-02T07:55:28.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.219+0000 7f9d5cc00700 1 -- 172.21.15.89:0/2048474126 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9d4c018560 con 0x7f9d600ce230 2024-04-02T07:55:28.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.219+0000 7f9d5cc00700 1 --2- 172.21.15.89:0/2048474126 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d4006adb0 0x7f9d4006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:28.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.220+0000 7f9d5cc00700 1 -- 172.21.15.89:0/2048474126 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5233+0+0 (secure 0 0 0) 0x7f9d4c08a7a0 con 0x7f9d600ce230 2024-04-02T07:55:28.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.220+0000 7f9d65000700 1 --2- 172.21.15.89:0/2048474126 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d4006adb0 0x7f9d4006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:28.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.220+0000 7f9d65000700 1 --2- 172.21.15.89:0/2048474126 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d4006adb0 0x7f9d4006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9d600ae950 tx=0x7f9d54008040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:28.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.224+0000 7f9d5cc00700 1 -- 172.21.15.89:0/2048474126 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9d600c9dd0 con 0x7f9d600ce230 2024-04-02T07:55:28.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.423+0000 7f9d67a00700 1 -- 172.21.15.89:0/2048474126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9d600c9dd0 con 0x7f9d600ce230 2024-04-02T07:55:28.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.424+0000 7f9d5cc00700 1 -- 172.21.15.89:0/2048474126 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v136) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9d600c9dd0 con 0x7f9d600ce230 2024-04-02T07:55:28.425 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:55:28.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.426+0000 7f9d67a00700 1 -- 172.21.15.89:0/2048474126 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d4006adb0 msgr2=0x7f9d4006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:28.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.426+0000 7f9d67a00700 1 --2- 172.21.15.89:0/2048474126 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d4006adb0 0x7f9d4006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9d600ae950 tx=0x7f9d54008040 comp rx=0 tx=0).stop 2024-04-02T07:55:28.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.426+0000 7f9d67a00700 1 -- 172.21.15.89:0/2048474126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d600ce230 msgr2=0x7f9d600ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:28.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.426+0000 7f9d67a00700 1 --2- 172.21.15.89:0/2048474126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d600ce230 0x7f9d600ac050 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0090b0 tx=0x7f9d4c004750 comp rx=0 tx=0).stop 2024-04-02T07:55:28.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.426+0000 7f9d67a00700 1 -- 172.21.15.89:0/2048474126 shutdown_connections 2024-04-02T07:55:28.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.426+0000 7f9d67a00700 1 --2- 172.21.15.89:0/2048474126 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9d4006adb0 0x7f9d4006d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.426+0000 7f9d67a00700 1 --2- 172.21.15.89:0/2048474126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d600b2070 0x7f9d60152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.426+0000 7f9d67a00700 1 --2- 172.21.15.89:0/2048474126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d6015d800 0x7f9d6006d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.426+0000 7f9d67a00700 1 --2- 172.21.15.89:0/2048474126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d600ce230 0x7f9d600ac050 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.426+0000 7f9d67a00700 1 -- 172.21.15.89:0/2048474126 >> 172.21.15.89:0/2048474126 conn(0x7f9d600b0d70 msgr2=0x7f9d600aee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:28.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.426+0000 7f9d67a00700 1 -- 172.21.15.89:0/2048474126 shutdown_connections 2024-04-02T07:55:28.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.426+0000 7f9d67a00700 1 -- 172.21.15.89:0/2048474126 wait complete. 2024-04-02T07:55:28.445 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T07:55:28.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.635+0000 7fc4f2600700 1 Processor -- start 2024-04-02T07:55:28.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.635+0000 7fc4f2600700 1 -- start start 2024-04-02T07:55:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.636+0000 7fc4f2600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4ec160f10 0x7fc4ec0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.636+0000 7fc4f2600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc4ec161590 0x7fc4ec161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.636+0000 7fc4f2600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ec0ceda0 con 0x7fc4ec0cf230 2024-04-02T07:55:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.636+0000 7fc4f2600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ec161f80 con 0x7fc4ec160f10 2024-04-02T07:55:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.636+0000 7fc4f2600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ec1612f0 con 0x7fc4ec161590 2024-04-02T07:55:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.636+0000 7fc4eb400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4ec160f10 0x7fc4ec0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.636+0000 7fc4eaa00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc4ec0cf230 0x7fc4ec0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40906/0 (socket says 172.21.15.89:40906) 2024-04-02T07:55:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.636+0000 7fc4eaa00700 1 -- 172.21.15.89:0/2379356936 learned_addr learned my addr 172.21.15.89:0/2379356936 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.637+0000 7fc4ebe00700 1 --2- 172.21.15.89:0/2379356936 >> v2:172.21.15.113:3300/0 conn(0x7fc4ec161590 0x7fc4ec161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.637+0000 7fc4eb400700 1 -- 172.21.15.89:0/2379356936 >> v2:172.21.15.113:3300/0 conn(0x7fc4ec161590 msgr2=0x7fc4ec161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.637+0000 7fc4eb400700 1 --2- 172.21.15.89:0/2379356936 >> v2:172.21.15.113:3300/0 conn(0x7fc4ec161590 0x7fc4ec161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.637+0000 7fc4eb400700 1 -- 172.21.15.89:0/2379356936 >> v1:172.21.15.89:6789/0 conn(0x7fc4ec0cf230 legacy=0x7fc4ec0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.637+0000 7fc4eb400700 1 -- 172.21.15.89:0/2379356936 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec0ce470 con 0x7fc4ec160f10 2024-04-02T07:55:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.637+0000 7fc4ea000700 1 -- 172.21.15.89:0/2379356936 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445949152 0 0) 0x7fc4ec0ceda0 con 0x7fc4ec0cf230 2024-04-02T07:55:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.637+0000 7fc4eb400700 1 --2- 172.21.15.89:0/2379356936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ec160f10 0x7fc4ec0d18b0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0d1030 tx=0x7fc4dc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac1250951c950d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.637+0000 7fc4ea000700 1 -- 172.21.15.89:0/2379356936 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc011830 con 0x7fc4ec160f10 2024-04-02T07:55:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.637+0000 7fc4ea000700 1 -- 172.21.15.89:0/2379356936 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4dc011990 con 0x7fc4ec160f10 2024-04-02T07:55:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.637+0000 7fc4ea000700 1 -- 172.21.15.89:0/2379356936 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc011d20 con 0x7fc4ec160f10 2024-04-02T07:55:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.638+0000 7fc4f2600700 1 -- 172.21.15.89:0/2379356936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ec160f10 msgr2=0x7fc4ec0d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.638+0000 7fc4f2600700 1 --2- 172.21.15.89:0/2379356936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ec160f10 0x7fc4ec0d18b0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0d1030 tx=0x7fc4dc009b20 comp rx=0 tx=0).stop 2024-04-02T07:55:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.638+0000 7fc4f2600700 1 -- 172.21.15.89:0/2379356936 shutdown_connections 2024-04-02T07:55:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.638+0000 7fc4f2600700 1 --2- 172.21.15.89:0/2379356936 >> v2:172.21.15.113:3300/0 conn(0x7fc4ec161590 0x7fc4ec161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.638+0000 7fc4f2600700 1 --2- 172.21.15.89:0/2379356936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ec160f10 0x7fc4ec0d18b0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.638+0000 7fc4f2600700 1 -- 172.21.15.89:0/2379356936 >> 172.21.15.89:0/2379356936 conn(0x7fc4ec08aff0 msgr2=0x7fc4ec0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.638+0000 7fc4f2600700 1 -- 172.21.15.89:0/2379356936 shutdown_connections 2024-04-02T07:55:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.638+0000 7fc4f2600700 1 -- 172.21.15.89:0/2379356936 wait complete. 2024-04-02T07:55:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.640+0000 7fc4f2600700 1 Processor -- start 2024-04-02T07:55:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.640+0000 7fc4f2600700 1 -- start start 2024-04-02T07:55:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.641+0000 7fc4f2600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4ec0cf230 0x7fc4ec15f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.641+0000 7fc4f2600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4ec160f10 0x7fc4ec0b8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.641+0000 7fc4f2600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ec161590 0x7fc4ec0a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.641+0000 7fc4f2600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec15fb90 con 0x7fc4ec161590 2024-04-02T07:55:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.641+0000 7fc4f2600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec15fe70 con 0x7fc4ec0cf230 2024-04-02T07:55:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.641+0000 7fc4f2600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec160150 con 0x7fc4ec160f10 2024-04-02T07:55:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.641+0000 7fc4ebe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ec161590 0x7fc4ec0a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.641+0000 7fc4ebe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ec161590 0x7fc4ec0a0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41218/0 (socket says 172.21.15.89:41218) 2024-04-02T07:55:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.641+0000 7fc4ebe00700 1 -- 172.21.15.89:0/1572079170 learned_addr learned my addr 172.21.15.89:0/1572079170 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.642+0000 7fc4ebe00700 1 -- 172.21.15.89:0/1572079170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4ec160f10 msgr2=0x7fc4ec0b8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.642+0000 7fc4eaa00700 1 --2- 172.21.15.89:0/1572079170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4ec160f10 0x7fc4ec0b8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.642+0000 7fc4ebe00700 1 --2- 172.21.15.89:0/1572079170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4ec160f10 0x7fc4ec0b8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.642+0000 7fc4ebe00700 1 -- 172.21.15.89:0/1572079170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4ec0cf230 msgr2=0x7fc4ec15f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.642+0000 7fc4ebe00700 1 --2- 172.21.15.89:0/1572079170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4ec0cf230 0x7fc4ec15f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.642+0000 7fc4ebe00700 1 -- 172.21.15.89:0/1572079170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc0094a0 con 0x7fc4ec161590 2024-04-02T07:55:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.642+0000 7fc4eaa00700 1 --2- 172.21.15.89:0/1572079170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4ec160f10 0x7fc4ec0b8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:55:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.642+0000 7fc4ebe00700 1 --2- 172.21.15.89:0/1572079170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ec161590 0x7fc4ec0a0a70 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4007e60 tx=0x7fc4e400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.642+0000 7fc4e3400700 1 -- 172.21.15.89:0/1572079170 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e400ede0 con 0x7fc4ec161590 2024-04-02T07:55:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.642+0000 7fc4e3400700 1 -- 172.21.15.89:0/1572079170 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e4014420 con 0x7fc4ec161590 2024-04-02T07:55:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.642+0000 7fc4e3400700 1 -- 172.21.15.89:0/1572079170 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e40147d0 con 0x7fc4ec161590 2024-04-02T07:55:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.642+0000 7fc4f2600700 1 -- 172.21.15.89:0/1572079170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ec09d440 con 0x7fc4ec161590 2024-04-02T07:55:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.642+0000 7fc4f2600700 1 -- 172.21.15.89:0/1572079170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ec15f7f0 con 0x7fc4ec161590 2024-04-02T07:55:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.643+0000 7fc4f2600700 1 -- 172.21.15.89:0/1572079170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ec067720 con 0x7fc4ec161590 2024-04-02T07:55:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.644+0000 7fc4e3400700 1 -- 172.21.15.89:0/1572079170 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc4e4014930 con 0x7fc4ec161590 2024-04-02T07:55:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.645+0000 7fc4e3400700 1 --2- 172.21.15.89:0/1572079170 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4cc06ade0 0x7fc4cc06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.645+0000 7fc4e3400700 1 -- 172.21.15.89:0/1572079170 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5233+0+0 (secure 0 0 0) 0x7fc4e40899d0 con 0x7fc4ec161590 2024-04-02T07:55:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.645+0000 7fc4eb400700 1 --2- 172.21.15.89:0/1572079170 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4cc06ade0 0x7fc4cc06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:28.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.646+0000 7fc4eb400700 1 --2- 172.21.15.89:0/1572079170 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4cc06ade0 0x7fc4cc06d2a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc000c00 tx=0x7fc4dc010000 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:28.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.649+0000 7fc4e3400700 1 -- 172.21.15.89:0/1572079170 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc4e4054c70 con 0x7fc4ec161590 2024-04-02T07:55:28.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.854+0000 7fc4f2600700 1 -- 172.21.15.89:0/1572079170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc4ec15c7c0 con 0x7fc4ec161590 2024-04-02T07:55:28.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.854+0000 7fc4e3400700 1 -- 172.21.15.89:0/1572079170 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v136) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc4e4058290 con 0x7fc4ec161590 2024-04-02T07:55:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:28.855 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:55:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T07:55:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T07:55:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:28.855 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:55:28.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.856+0000 7fc4f2600700 1 -- 172.21.15.89:0/1572079170 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4cc06ade0 msgr2=0x7fc4cc06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:28.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.856+0000 7fc4f2600700 1 --2- 172.21.15.89:0/1572079170 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4cc06ade0 0x7fc4cc06d2a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc000c00 tx=0x7fc4dc010000 comp rx=0 tx=0).stop 2024-04-02T07:55:28.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.856+0000 7fc4f2600700 1 -- 172.21.15.89:0/1572079170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ec161590 msgr2=0x7fc4ec0a0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:28.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.856+0000 7fc4f2600700 1 --2- 172.21.15.89:0/1572079170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ec161590 0x7fc4ec0a0a70 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4007e60 tx=0x7fc4e400e3c0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.857+0000 7fc4f2600700 1 -- 172.21.15.89:0/1572079170 shutdown_connections 2024-04-02T07:55:28.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.857+0000 7fc4f2600700 1 --2- 172.21.15.89:0/1572079170 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4cc06ade0 0x7fc4cc06d2a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.857+0000 7fc4f2600700 1 --2- 172.21.15.89:0/1572079170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4ec0cf230 0x7fc4ec15f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.857+0000 7fc4f2600700 1 --2- 172.21.15.89:0/1572079170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4ec161590 0x7fc4ec0a0a70 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.857+0000 7fc4f2600700 1 --2- 172.21.15.89:0/1572079170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4ec160f10 0x7fc4ec0b8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:28.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.857+0000 7fc4f2600700 1 -- 172.21.15.89:0/1572079170 >> 172.21.15.89:0/1572079170 conn(0x7fc4ec08aff0 msgr2=0x7fc4ec0cc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:28.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.857+0000 7fc4f2600700 1 -- 172.21.15.89:0/1572079170 shutdown_connections 2024-04-02T07:55:28.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:28.857+0000 7fc4f2600700 1 -- 172.21.15.89:0/1572079170 wait complete. 2024-04-02T07:55:28.876 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T07:55:28.876 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T07:55:29.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.070+0000 7fa6d6800700 1 Processor -- start 2024-04-02T07:55:29.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.070+0000 7fa6d6800700 1 -- start start 2024-04-02T07:55:29.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.071+0000 7fa6d6800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa6d00ced30 0x7fa6d00d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:29.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.071+0000 7fa6d6800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa6d00d2710 0x7fa6d015f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:29.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.071+0000 7fa6d6800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d00ce8a0 con 0x7fa6d015ddf0 2024-04-02T07:55:29.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.071+0000 7fa6d6800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d015ec60 con 0x7fa6d00d2710 2024-04-02T07:55:29.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.071+0000 7fa6d6800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d00d2b60 con 0x7fa6d00ced30 2024-04-02T07:55:29.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.072+0000 7fa6cfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa6d00d2710 0x7fa6d015f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:29.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.072+0000 7fa6cfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa6d00d2710 0x7fa6d015f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54724/0 (socket says 172.21.15.89:54724) 2024-04-02T07:55:29.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.072+0000 7fa6cfe00700 1 -- 172.21.15.89:0/2716249411 learned_addr learned my addr 172.21.15.89:0/2716249411 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:29.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.072+0000 7fa6cea00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa6d00ced30 0x7fa6d00d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:29.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.072+0000 7fa6cfe00700 1 -- 172.21.15.89:0/2716249411 >> v2:172.21.15.113:3300/0 conn(0x7fa6d00ced30 msgr2=0x7fa6d00d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:29.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.072+0000 7fa6cfe00700 1 --2- 172.21.15.89:0/2716249411 >> v2:172.21.15.113:3300/0 conn(0x7fa6d00ced30 0x7fa6d00d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:29.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.072+0000 7fa6cfe00700 1 -- 172.21.15.89:0/2716249411 >> v1:172.21.15.113:6789/0 conn(0x7fa6d015ddf0 legacy=0x7fa6d015e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:29.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.072+0000 7fa6cfe00700 1 -- 172.21.15.89:0/2716249411 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d00cdc90 con 0x7fa6d00d2710 2024-04-02T07:55:29.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.073+0000 7fa6cfe00700 1 --2- 172.21.15.89:0/2716249411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6d00d2710 0x7fa6d015f0f0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00d0fe0 tx=0x7fa6c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9385bcca147df90 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:29.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.073+0000 7fa6ce000700 1 -- 172.21.15.89:0/2716249411 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c800e070 con 0x7fa6d00d2710 2024-04-02T07:55:29.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.073+0000 7fa6ce000700 1 -- 172.21.15.89:0/2716249411 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c800b910 con 0x7fa6d00d2710 2024-04-02T07:55:29.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.073+0000 7fa6ce000700 1 -- 172.21.15.89:0/2716249411 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c800bcc0 con 0x7fa6d00d2710 2024-04-02T07:55:29.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.073+0000 7fa6d6800700 1 -- 172.21.15.89:0/2716249411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6d00d2710 msgr2=0x7fa6d015f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:29.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.073+0000 7fa6d6800700 1 --2- 172.21.15.89:0/2716249411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6d00d2710 0x7fa6d015f0f0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00d0fe0 tx=0x7fa6c800b330 comp rx=0 tx=0).stop 2024-04-02T07:55:29.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.073+0000 7fa6d6800700 1 -- 172.21.15.89:0/2716249411 shutdown_connections 2024-04-02T07:55:29.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.073+0000 7fa6d6800700 1 --2- 172.21.15.89:0/2716249411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6d00d2710 0x7fa6d015f0f0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:29.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.073+0000 7fa6d6800700 1 --2- 172.21.15.89:0/2716249411 >> v2:172.21.15.113:3300/0 conn(0x7fa6d00ced30 0x7fa6d00d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:29.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.073+0000 7fa6d6800700 1 -- 172.21.15.89:0/2716249411 >> 172.21.15.89:0/2716249411 conn(0x7fa6d008aff0 msgr2=0x7fa6d00d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:29.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.073+0000 7fa6d6800700 1 -- 172.21.15.89:0/2716249411 shutdown_connections 2024-04-02T07:55:29.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.073+0000 7fa6d6800700 1 -- 172.21.15.89:0/2716249411 wait complete. 2024-04-02T07:55:29.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.075+0000 7fa6d6800700 1 Processor -- start 2024-04-02T07:55:29.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.075+0000 7fa6d6800700 1 -- start start 2024-04-02T07:55:29.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.076+0000 7fa6d6800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6d00ced30 0x7fa6d0156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:29.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.076+0000 7fa6d6800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6d00d2710 0x7fa6d00c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:29.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.076+0000 7fa6d6800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6d015ddf0 0x7fa6d00cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:29.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.076+0000 7fa6d6800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d01f3f70 con 0x7fa6d00d2710 2024-04-02T07:55:29.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.076+0000 7fa6d6800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d01f32f0 con 0x7fa6d00ced30 2024-04-02T07:55:29.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.076+0000 7fa6d6800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d01f35d0 con 0x7fa6d015ddf0 2024-04-02T07:55:29.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.076+0000 7fa6cea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6d00d2710 0x7fa6d00c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:29.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.076+0000 7fa6cea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6d00d2710 0x7fa6d00c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54726/0 (socket says 172.21.15.89:54726) 2024-04-02T07:55:29.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.076+0000 7fa6cea00700 1 -- 172.21.15.89:0/2124354741 learned_addr learned my addr 172.21.15.89:0/2124354741 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:29.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.077+0000 7fa6cf400700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6d00ced30 0x7fa6d0156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:29.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.077+0000 7fa6cfe00700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6d015ddf0 0x7fa6d00cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:29.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.077+0000 7fa6cea00700 1 -- 172.21.15.89:0/2124354741 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6d015ddf0 msgr2=0x7fa6d00cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:29.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.077+0000 7fa6cea00700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6d015ddf0 0x7fa6d00cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:29.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.077+0000 7fa6cea00700 1 -- 172.21.15.89:0/2124354741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6d00ced30 msgr2=0x7fa6d0156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:29.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.077+0000 7fa6cea00700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6d00ced30 0x7fa6d0156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:29.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.077+0000 7fa6cea00700 1 -- 172.21.15.89:0/2124354741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c8009b30 con 0x7fa6d00d2710 2024-04-02T07:55:29.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.077+0000 7fa6cfe00700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6d015ddf0 0x7fa6d00cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:29.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.077+0000 7fa6cea00700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6d00d2710 0x7fa6d00c9a90 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00e9b0 tx=0x7fa6bc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:29.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.077+0000 7fa6c7400700 1 -- 172.21.15.89:0/2124354741 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc009710 con 0x7fa6d00d2710 2024-04-02T07:55:29.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.077+0000 7fa6c7400700 1 -- 172.21.15.89:0/2124354741 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc00c850 con 0x7fa6d00d2710 2024-04-02T07:55:29.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.077+0000 7fa6c7400700 1 -- 172.21.15.89:0/2124354741 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc00cbc0 con 0x7fa6d00d2710 2024-04-02T07:55:29.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.077+0000 7fa6d6800700 1 -- 172.21.15.89:0/2124354741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d01f38b0 con 0x7fa6d00d2710 2024-04-02T07:55:29.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.078+0000 7fa6d6800700 1 -- 172.21.15.89:0/2124354741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d0160190 con 0x7fa6d00d2710 2024-04-02T07:55:29.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.079+0000 7fa6d6800700 1 -- 172.21.15.89:0/2124354741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d0067720 con 0x7fa6d00d2710 2024-04-02T07:55:29.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.080+0000 7fa6c7400700 1 -- 172.21.15.89:0/2124354741 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa6bc00cd20 con 0x7fa6d00d2710 2024-04-02T07:55:29.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.080+0000 7fa6c7400700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa6b006adb0 0x7fa6b006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:29.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.081+0000 7fa6c7400700 1 -- 172.21.15.89:0/2124354741 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5233+0+0 (secure 0 0 0) 0x7fa6bc089310 con 0x7fa6d00d2710 2024-04-02T07:55:29.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.081+0000 7fa6cf400700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa6b006adb0 0x7fa6b006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:29.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.081+0000 7fa6cf400700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa6b006adb0 0x7fa6b006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0007fd0 tx=0x7fa6c0009040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:29.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.085+0000 7fa6c7400700 1 -- 172.21.15.89:0/2124354741 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa6bc054640 con 0x7fa6d00d2710 2024-04-02T07:55:29.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:29.288+0000 7fa6d6800700 1 -- 172.21.15.89:0/2124354741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa6d00b6780 con 0x7fa6d00d2710 2024-04-02T07:55:30.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.068+0000 7fa6c7400700 1 -- 172.21.15.89:0/2124354741 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v137) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa6bc057c60 con 0x7fa6d00d2710 2024-04-02T07:55:30.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.072+0000 7fa6d6800700 1 -- 172.21.15.89:0/2124354741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa6b006adb0 msgr2=0x7fa6b006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:30.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.072+0000 7fa6d6800700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa6b006adb0 0x7fa6b006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0007fd0 tx=0x7fa6c0009040 comp rx=0 tx=0).stop 2024-04-02T07:55:30.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.072+0000 7fa6d6800700 1 -- 172.21.15.89:0/2124354741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6d00d2710 msgr2=0x7fa6d00c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:30.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.072+0000 7fa6d6800700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6d00d2710 0x7fa6d00c9a90 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00e9b0 tx=0x7fa6bc00ecc0 comp rx=0 tx=0).stop 2024-04-02T07:55:30.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.072+0000 7fa6d6800700 1 -- 172.21.15.89:0/2124354741 shutdown_connections 2024-04-02T07:55:30.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.072+0000 7fa6d6800700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa6b006adb0 0x7fa6b006d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:30.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.072+0000 7fa6d6800700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6d00ced30 0x7fa6d0156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:30.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.072+0000 7fa6d6800700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6d015ddf0 0x7fa6d00cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:30.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.072+0000 7fa6d6800700 1 --2- 172.21.15.89:0/2124354741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6d00d2710 0x7fa6d00c9a90 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:30.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.072+0000 7fa6d6800700 1 -- 172.21.15.89:0/2124354741 >> 172.21.15.89:0/2124354741 conn(0x7fa6d008aff0 msgr2=0x7fa6d00b8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:30.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.073+0000 7fa6d6800700 1 -- 172.21.15.89:0/2124354741 shutdown_connections 2024-04-02T07:55:30.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.073+0000 7fa6d6800700 1 -- 172.21.15.89:0/2124354741 wait complete. 2024-04-02T07:55:30.076 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T07:55:30.095 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T07:55:30.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.299+0000 7fc8d7800700 1 Processor -- start 2024-04-02T07:55:30.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.299+0000 7fc8d7800700 1 -- start start 2024-04-02T07:55:30.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.300+0000 7fc8d7800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d015d9d0 0x7fc8d00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:30.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.300+0000 7fc8d7800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8d00d16a0 0x7fc8d00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:30.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.301+0000 7fc8d7800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d0161420 con 0x7fc8d00d2370 2024-04-02T07:55:30.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.301+0000 7fc8d7800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d00d2820 con 0x7fc8d00d16a0 2024-04-02T07:55:30.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.301+0000 7fc8d7800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d0161700 con 0x7fc8d015d9d0 2024-04-02T07:55:30.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.301+0000 7fc8cfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8d00d16a0 0x7fc8d00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:30.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.301+0000 7fc8d5800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc8d00d2370 0x7fc8d015ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44732/0 (socket says 172.21.15.89:44732) 2024-04-02T07:55:30.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.301+0000 7fc8d5800700 1 -- 172.21.15.89:0/4149772592 learned_addr learned my addr 172.21.15.89:0/4149772592 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:30.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.301+0000 7fc8d4e00700 1 --2- 172.21.15.89:0/4149772592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d015d9d0 0x7fc8d00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:30.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.301+0000 7fc8cf400700 1 -- 172.21.15.89:0/4149772592 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757153093 0 0) 0x7fc8d0161420 con 0x7fc8d00d2370 2024-04-02T07:55:30.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.301+0000 7fc8cfe00700 1 -- 172.21.15.89:0/4149772592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d015d9d0 msgr2=0x7fc8d00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:30.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.301+0000 7fc8cfe00700 1 --2- 172.21.15.89:0/4149772592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d015d9d0 0x7fc8d00d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:30.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.301+0000 7fc8cfe00700 1 -- 172.21.15.89:0/4149772592 >> v1:172.21.15.89:6789/0 conn(0x7fc8d00d2370 legacy=0x7fc8d015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:30.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.301+0000 7fc8cfe00700 1 -- 172.21.15.89:0/4149772592 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d01619e0 con 0x7fc8d00d16a0 2024-04-02T07:55:30.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.302+0000 7fc8cfe00700 1 --2- 172.21.15.89:0/4149772592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d00d16a0 0x7fc8d00d1e30 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009820 tx=0x7fc8bc009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b96e2e097b6af6a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:30.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.302+0000 7fc8cf400700 1 -- 172.21.15.89:0/4149772592 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc005e30 con 0x7fc8d00d16a0 2024-04-02T07:55:30.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.302+0000 7fc8cf400700 1 -- 172.21.15.89:0/4149772592 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8bc006030 con 0x7fc8d00d16a0 2024-04-02T07:55:30.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.302+0000 7fc8cf400700 1 -- 172.21.15.89:0/4149772592 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc0063e0 con 0x7fc8d00d16a0 2024-04-02T07:55:30.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.302+0000 7fc8d7800700 1 -- 172.21.15.89:0/4149772592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d00d16a0 msgr2=0x7fc8d00d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:30.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.302+0000 7fc8d7800700 1 --2- 172.21.15.89:0/4149772592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d00d16a0 0x7fc8d00d1e30 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009820 tx=0x7fc8bc009b30 comp rx=0 tx=0).stop 2024-04-02T07:55:30.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.302+0000 7fc8d7800700 1 -- 172.21.15.89:0/4149772592 shutdown_connections 2024-04-02T07:55:30.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.302+0000 7fc8d7800700 1 --2- 172.21.15.89:0/4149772592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d015d9d0 0x7fc8d00d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:30.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.302+0000 7fc8d7800700 1 --2- 172.21.15.89:0/4149772592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d00d16a0 0x7fc8d00d1e30 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:30.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.302+0000 7fc8d7800700 1 -- 172.21.15.89:0/4149772592 >> 172.21.15.89:0/4149772592 conn(0x7fc8d008aff0 msgr2=0x7fc8d00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:30.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.302+0000 7fc8d7800700 1 -- 172.21.15.89:0/4149772592 shutdown_connections 2024-04-02T07:55:30.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.303+0000 7fc8d7800700 1 -- 172.21.15.89:0/4149772592 wait complete. 2024-04-02T07:55:30.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.304+0000 7fc8d7800700 1 Processor -- start 2024-04-02T07:55:30.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.304+0000 7fc8d7800700 1 -- start start 2024-04-02T07:55:30.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.305+0000 7fc8d7800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d00d16a0 0x7fc8d00cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:30.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.305+0000 7fc8d7800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8d00d2370 0x7fc8d015e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:30.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.305+0000 7fc8d7800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d015d9d0 0x7fc8d00ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:30.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.305+0000 7fc8d7800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d015eb10 con 0x7fc8d015d9d0 2024-04-02T07:55:30.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.305+0000 7fc8d7800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00cda00 con 0x7fc8d00d2370 2024-04-02T07:55:30.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.305+0000 7fc8d7800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00cd720 con 0x7fc8d00d16a0 2024-04-02T07:55:30.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.305+0000 7fc8d5800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d015d9d0 0x7fc8d00ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:30.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.305+0000 7fc8d5800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d015d9d0 0x7fc8d00ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54742/0 (socket says 172.21.15.89:54742) 2024-04-02T07:55:30.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.305+0000 7fc8d5800700 1 -- 172.21.15.89:0/793356883 learned_addr learned my addr 172.21.15.89:0/793356883 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:30.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.305+0000 7fc8d4e00700 1 --2- 172.21.15.89:0/793356883 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d00d16a0 0x7fc8d00cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:30.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.306+0000 7fc8cfe00700 1 --2- 172.21.15.89:0/793356883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8d00d2370 0x7fc8d015e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:30.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.306+0000 7fc8d5800700 1 -- 172.21.15.89:0/793356883 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d00d16a0 msgr2=0x7fc8d00cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:30.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.306+0000 7fc8d5800700 1 --2- 172.21.15.89:0/793356883 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d00d16a0 0x7fc8d00cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:30.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.306+0000 7fc8d5800700 1 -- 172.21.15.89:0/793356883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8d00d2370 msgr2=0x7fc8d015e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:30.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.306+0000 7fc8d5800700 1 --2- 172.21.15.89:0/793356883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8d00d2370 0x7fc8d015e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:30.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.306+0000 7fc8d5800700 1 -- 172.21.15.89:0/793356883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc009430 con 0x7fc8d015d9d0 2024-04-02T07:55:30.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.306+0000 7fc8d5800700 1 --2- 172.21.15.89:0/793356883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d015d9d0 0x7fc8d00ca5f0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8002230 tx=0x7fc8c800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:30.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.306+0000 7fc8ccc00700 1 -- 172.21.15.89:0/793356883 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c800aac0 con 0x7fc8d015d9d0 2024-04-02T07:55:30.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.306+0000 7fc8ccc00700 1 -- 172.21.15.89:0/793356883 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c800ac40 con 0x7fc8d015d9d0 2024-04-02T07:55:30.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.306+0000 7fc8d7800700 1 -- 172.21.15.89:0/793356883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d00b6940 con 0x7fc8d015d9d0 2024-04-02T07:55:30.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.306+0000 7fc8ccc00700 1 -- 172.21.15.89:0/793356883 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c80070d0 con 0x7fc8d015d9d0 2024-04-02T07:55:30.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.307+0000 7fc8d7800700 1 -- 172.21.15.89:0/793356883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d00b6f00 con 0x7fc8d015d9d0 2024-04-02T07:55:30.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.308+0000 7fc8d7800700 1 -- 172.21.15.89:0/793356883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d00d1ca0 con 0x7fc8d015d9d0 2024-04-02T07:55:30.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.309+0000 7fc8ccc00700 1 -- 172.21.15.89:0/793356883 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc8c8007250 con 0x7fc8d015d9d0 2024-04-02T07:55:30.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.310+0000 7fc8ccc00700 1 --2- 172.21.15.89:0/793356883 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc8b006adb0 0x7fc8b006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:30.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.310+0000 7fc8ccc00700 1 -- 172.21.15.89:0/793356883 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5603+0+0 (secure 0 0 0) 0x7fc8c80894e0 con 0x7fc8d015d9d0 2024-04-02T07:55:30.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.310+0000 7fc8d4e00700 1 --2- 172.21.15.89:0/793356883 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc8b006adb0 0x7fc8b006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:30.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.311+0000 7fc8d4e00700 1 --2- 172.21.15.89:0/793356883 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc8b006adb0 0x7fc8b006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc8d01f6b90 tx=0x7fc8c4008040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:30.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.314+0000 7fc8ccc00700 1 -- 172.21.15.89:0/793356883 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc8c8054680 con 0x7fc8d015d9d0 2024-04-02T07:55:30.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:30.514+0000 7fc8d7800700 1 -- 172.21.15.89:0/793356883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc8d0067720 con 0x7fc8d015d9d0 2024-04-02T07:55:31.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.075+0000 7fc8ccc00700 1 -- 172.21.15.89:0/793356883 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v138) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc8c80180d0 con 0x7fc8d015d9d0 2024-04-02T07:55:31.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.080+0000 7fc8d7800700 1 -- 172.21.15.89:0/793356883 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc8b006adb0 msgr2=0x7fc8b006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:31.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.080+0000 7fc8d7800700 1 --2- 172.21.15.89:0/793356883 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc8b006adb0 0x7fc8b006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc8d01f6b90 tx=0x7fc8c4008040 comp rx=0 tx=0).stop 2024-04-02T07:55:31.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.080+0000 7fc8d7800700 1 -- 172.21.15.89:0/793356883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d015d9d0 msgr2=0x7fc8d00ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:31.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.080+0000 7fc8d7800700 1 --2- 172.21.15.89:0/793356883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d015d9d0 0x7fc8d00ca5f0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8002230 tx=0x7fc8c800cf50 comp rx=0 tx=0).stop 2024-04-02T07:55:31.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.084+0000 7fc8d7800700 1 -- 172.21.15.89:0/793356883 shutdown_connections 2024-04-02T07:55:31.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.084+0000 7fc8d7800700 1 --2- 172.21.15.89:0/793356883 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc8b006adb0 0x7fc8b006d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.084+0000 7fc8d7800700 1 --2- 172.21.15.89:0/793356883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d015d9d0 0x7fc8d00ca5f0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.084+0000 7fc8d7800700 1 --2- 172.21.15.89:0/793356883 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d00d16a0 0x7fc8d00cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.084+0000 7fc8d7800700 1 --2- 172.21.15.89:0/793356883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8d00d2370 0x7fc8d015e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.084+0000 7fc8d7800700 1 -- 172.21.15.89:0/793356883 >> 172.21.15.89:0/793356883 conn(0x7fc8d008aff0 msgr2=0x7fc8d00b5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:31.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.085+0000 7fc8d7800700 1 -- 172.21.15.89:0/793356883 shutdown_connections 2024-04-02T07:55:31.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.087+0000 7fc8d7800700 1 -- 172.21.15.89:0/793356883 wait complete. 2024-04-02T07:55:31.092 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T07:55:31.113 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T07:55:31.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.307+0000 7f0f6dc00700 1 Processor -- start 2024-04-02T07:55:31.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.308+0000 7f0f6dc00700 1 -- start start 2024-04-02T07:55:31.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.309+0000 7f0f6dc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f680cbb30 0x7f0f680cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:31.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.309+0000 7f0f6dc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0f680cd480 0x7f0f6806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:31.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.309+0000 7f0f6dc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f680cc450 con 0x7f0f680cd020 2024-04-02T07:55:31.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.309+0000 7f0f6dc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f680cd8f0 con 0x7f0f680cbb30 2024-04-02T07:55:31.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.309+0000 7f0f6dc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f680cdae0 con 0x7f0f680cd480 2024-04-02T07:55:31.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.309+0000 7f0f66a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f680cbb30 0x7f0f680cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:31.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.309+0000 7f0f66a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f680cbb30 0x7f0f680cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54756/0 (socket says 172.21.15.89:54756) 2024-04-02T07:55:31.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.309+0000 7f0f66a00700 1 -- 172.21.15.89:0/2071507934 learned_addr learned my addr 172.21.15.89:0/2071507934 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:31.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.309+0000 7f0f67400700 1 --2- 172.21.15.89:0/2071507934 >> v2:172.21.15.113:3300/0 conn(0x7f0f680cd480 0x7f0f6806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:31.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.309+0000 7f0f66a00700 1 -- 172.21.15.89:0/2071507934 >> v2:172.21.15.113:3300/0 conn(0x7f0f680cd480 msgr2=0x7f0f6806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:31.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.309+0000 7f0f66a00700 1 --2- 172.21.15.89:0/2071507934 >> v2:172.21.15.113:3300/0 conn(0x7f0f680cd480 0x7f0f6806e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.309+0000 7f0f66a00700 1 -- 172.21.15.89:0/2071507934 >> v1:172.21.15.113:6789/0 conn(0x7f0f680cd020 legacy=0x7f0f680b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:31.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.309+0000 7f0f66a00700 1 -- 172.21.15.89:0/2071507934 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f680cddc0 con 0x7f0f680cbb30 2024-04-02T07:55:31.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.310+0000 7f0f66a00700 1 --2- 172.21.15.89:0/2071507934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f680cbb30 0x7f0f680cbf10 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f0f6806b6d0 tx=0x7f0f58009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8349fba9cf2ccdb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:31.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.310+0000 7f0f65600700 1 -- 172.21.15.89:0/2071507934 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f58004230 con 0x7f0f680cbb30 2024-04-02T07:55:31.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.310+0000 7f0f65600700 1 -- 172.21.15.89:0/2071507934 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f58004390 con 0x7f0f680cbb30 2024-04-02T07:55:31.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.310+0000 7f0f65600700 1 -- 172.21.15.89:0/2071507934 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f580036a0 con 0x7f0f680cbb30 2024-04-02T07:55:31.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.310+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2071507934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f680cbb30 msgr2=0x7f0f680cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:31.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.310+0000 7f0f6dc00700 1 --2- 172.21.15.89:0/2071507934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f680cbb30 0x7f0f680cbf10 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f0f6806b6d0 tx=0x7f0f58009bd0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.311+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2071507934 shutdown_connections 2024-04-02T07:55:31.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.311+0000 7f0f6dc00700 1 --2- 172.21.15.89:0/2071507934 >> v2:172.21.15.113:3300/0 conn(0x7f0f680cd480 0x7f0f6806e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.311+0000 7f0f6dc00700 1 --2- 172.21.15.89:0/2071507934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f680cbb30 0x7f0f680cbf10 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.311+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2071507934 >> 172.21.15.89:0/2071507934 conn(0x7f0f680b0d70 msgr2=0x7f0f680b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:31.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.311+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2071507934 shutdown_connections 2024-04-02T07:55:31.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.311+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2071507934 wait complete. 2024-04-02T07:55:31.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.312+0000 7f0f6dc00700 1 Processor -- start 2024-04-02T07:55:31.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.312+0000 7f0f6dc00700 1 -- start start 2024-04-02T07:55:31.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.313+0000 7f0f6dc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f680cbb30 0x7f0f681f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:31.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.313+0000 7f0f6dc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f680cd020 0x7f0f681f1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:31.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.313+0000 7f0f6dc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680cd480 0x7f0f6809b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:31.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.313+0000 7f0f6dc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f68099370 con 0x7f0f680cd020 2024-04-02T07:55:31.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.313+0000 7f0f6dc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f68099650 con 0x7f0f680cd480 2024-04-02T07:55:31.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.313+0000 7f0f6dc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f68099930 con 0x7f0f680cbb30 2024-04-02T07:55:31.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.313+0000 7f0f66000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f680cd020 0x7f0f681f1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:31.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.314+0000 7f0f66000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f680cd020 0x7f0f681f1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54770/0 (socket says 172.21.15.89:54770) 2024-04-02T07:55:31.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.314+0000 7f0f66000700 1 -- 172.21.15.89:0/2837176199 learned_addr learned my addr 172.21.15.89:0/2837176199 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:31.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.314+0000 7f0f66a00700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f680cbb30 0x7f0f681f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:31.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.314+0000 7f0f67400700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680cd480 0x7f0f6809b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:31.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.314+0000 7f0f66000700 1 -- 172.21.15.89:0/2837176199 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f680cbb30 msgr2=0x7f0f681f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:31.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.314+0000 7f0f66000700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f680cbb30 0x7f0f681f2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.314+0000 7f0f66000700 1 -- 172.21.15.89:0/2837176199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680cd480 msgr2=0x7f0f6809b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:31.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.314+0000 7f0f66000700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680cd480 0x7f0f6809b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.314+0000 7f0f66000700 1 -- 172.21.15.89:0/2837176199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f58009550 con 0x7f0f680cd020 2024-04-02T07:55:31.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.314+0000 7f0f66a00700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f680cbb30 0x7f0f681f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:55:31.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.314+0000 7f0f67400700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680cd480 0x7f0f6809b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:55:31.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.314+0000 7f0f66000700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f680cd020 0x7f0f681f1770 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400b4a0 tx=0x7f0f5400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:31.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.314+0000 7f0f5ea00700 1 -- 172.21.15.89:0/2837176199 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f54011830 con 0x7f0f680cd020 2024-04-02T07:55:31.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.315+0000 7f0f5ea00700 1 -- 172.21.15.89:0/2837176199 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f54011990 con 0x7f0f680cd020 2024-04-02T07:55:31.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.315+0000 7f0f5ea00700 1 -- 172.21.15.89:0/2837176199 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f54011d40 con 0x7f0f680cd020 2024-04-02T07:55:31.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.315+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2837176199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f68099c10 con 0x7f0f680cd020 2024-04-02T07:55:31.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.315+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2837176199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f681f1cb0 con 0x7f0f680cd020 2024-04-02T07:55:31.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.316+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2837176199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f68099e00 con 0x7f0f680cd020 2024-04-02T07:55:31.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.317+0000 7f0f5ea00700 1 -- 172.21.15.89:0/2837176199 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0f5401a400 con 0x7f0f680cd020 2024-04-02T07:55:31.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.318+0000 7f0f5ea00700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f4806adb0 0x7f0f4806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:31.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.318+0000 7f0f5ea00700 1 -- 172.21.15.89:0/2837176199 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5985+0+0 (secure 0 0 0) 0x7f0f54089380 con 0x7f0f680cd020 2024-04-02T07:55:31.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.318+0000 7f0f66a00700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f4806adb0 0x7f0f4806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:31.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.318+0000 7f0f66a00700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f4806adb0 0x7f0f4806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009c10 tx=0x7f0f58008030 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:31.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.322+0000 7f0f5ea00700 1 -- 172.21.15.89:0/2837176199 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0f54054290 con 0x7f0f680cd020 2024-04-02T07:55:31.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.570+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2837176199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0f680994b0 con 0x7f0f680cd020 2024-04-02T07:55:31.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.577+0000 7f0f5ea00700 1 -- 172.21.15.89:0/2837176199 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v62) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0f680994b0 con 0x7f0f680cd020 2024-04-02T07:55:31.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.583+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2837176199 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f4806adb0 msgr2=0x7f0f4806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:31.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.583+0000 7f0f6dc00700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f4806adb0 0x7f0f4806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009c10 tx=0x7f0f58008030 comp rx=0 tx=0).stop 2024-04-02T07:55:31.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.583+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2837176199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f680cd020 msgr2=0x7f0f681f1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:31.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.583+0000 7f0f6dc00700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f680cd020 0x7f0f681f1770 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400b4a0 tx=0x7f0f5400b860 comp rx=0 tx=0).stop 2024-04-02T07:55:31.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.584+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2837176199 shutdown_connections 2024-04-02T07:55:31.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.584+0000 7f0f6dc00700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0f4806adb0 0x7f0f4806d270 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009c10 tx=0x7f0f58008030 comp rx=0 tx=0).stop 2024-04-02T07:55:31.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.584+0000 7f0f6dc00700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680cd480 0x7f0f6809b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.584+0000 7f0f6dc00700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f680cbb30 0x7f0f681f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.584+0000 7f0f6dc00700 1 --2- 172.21.15.89:0/2837176199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f680cd020 0x7f0f681f1770 secure :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400b4a0 tx=0x7f0f5400b860 comp rx=0 tx=0).stop 2024-04-02T07:55:31.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.584+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2837176199 >> 172.21.15.89:0/2837176199 conn(0x7f0f680b0d70 msgr2=0x7f0f680ce890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:31.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.587+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2837176199 shutdown_connections 2024-04-02T07:55:31.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.587+0000 7f0f6dc00700 1 -- 172.21.15.89:0/2837176199 wait complete. 2024-04-02T07:55:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 16 and data pool 17 2024-04-02T07:55:31.619 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:55:31.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.810+0000 7f3fe0800700 1 Processor -- start 2024-04-02T07:55:31.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.810+0000 7f3fe0800700 1 -- start start 2024-04-02T07:55:31.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.811+0000 7f3fe0800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3fd80cb3f0 0x7f3fd80ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:31.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.811+0000 7f3fe0800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3fd80ac600 0x7f3fd806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:31.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.811+0000 7f3fe0800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd815fd50 con 0x7f3fd815b5d0 2024-04-02T07:55:31.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.811+0000 7f3fe0800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd815b9b0 con 0x7f3fd80cb3f0 2024-04-02T07:55:31.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.811+0000 7f3fe0800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd80aca10 con 0x7f3fd80ac600 2024-04-02T07:55:31.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.812+0000 7f3fdd400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3fd80cb3f0 0x7f3fd80ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:31.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.812+0000 7f3fdd400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3fd80cb3f0 0x7f3fd80ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54782/0 (socket says 172.21.15.89:54782) 2024-04-02T07:55:31.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.812+0000 7f3fdd400700 1 -- 172.21.15.89:0/2139224886 learned_addr learned my addr 172.21.15.89:0/2139224886 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:31.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.812+0000 7f3fde800700 1 --2- 172.21.15.89:0/2139224886 >> v2:172.21.15.113:3300/0 conn(0x7f3fd80ac600 0x7f3fd806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:31.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.812+0000 7f3fdd400700 1 -- 172.21.15.89:0/2139224886 >> v2:172.21.15.113:3300/0 conn(0x7f3fd80ac600 msgr2=0x7f3fd806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:31.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.812+0000 7f3fdd400700 1 --2- 172.21.15.89:0/2139224886 >> v2:172.21.15.113:3300/0 conn(0x7f3fd80ac600 0x7f3fd806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.812+0000 7f3fdd400700 1 -- 172.21.15.89:0/2139224886 >> v1:172.21.15.113:6789/0 conn(0x7f3fd815b5d0 legacy=0x7f3fd80ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:31.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.812+0000 7f3fdd400700 1 -- 172.21.15.89:0/2139224886 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd815f140 con 0x7f3fd80cb3f0 2024-04-02T07:55:31.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.812+0000 7f3fdd400700 1 --2- 172.21.15.89:0/2139224886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd80cb3f0 0x7f3fd80ac0c0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80cfdc0 tx=0x7f3fc8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e9d646323deb2ca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:31.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.812+0000 7f3fdca00700 1 -- 172.21.15.89:0/2139224886 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8011830 con 0x7f3fd80cb3f0 2024-04-02T07:55:31.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.813+0000 7f3fdca00700 1 -- 172.21.15.89:0/2139224886 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc8011990 con 0x7f3fd80cb3f0 2024-04-02T07:55:31.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.813+0000 7f3fdca00700 1 -- 172.21.15.89:0/2139224886 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8011d20 con 0x7f3fd80cb3f0 2024-04-02T07:55:31.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.813+0000 7f3fe0800700 1 -- 172.21.15.89:0/2139224886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd80cb3f0 msgr2=0x7f3fd80ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:31.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.813+0000 7f3fe0800700 1 --2- 172.21.15.89:0/2139224886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd80cb3f0 0x7f3fd80ac0c0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80cfdc0 tx=0x7f3fc8009bd0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.813+0000 7f3fe0800700 1 -- 172.21.15.89:0/2139224886 shutdown_connections 2024-04-02T07:55:31.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.813+0000 7f3fe0800700 1 --2- 172.21.15.89:0/2139224886 >> v2:172.21.15.113:3300/0 conn(0x7f3fd80ac600 0x7f3fd806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.813+0000 7f3fe0800700 1 --2- 172.21.15.89:0/2139224886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd80cb3f0 0x7f3fd80ac0c0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.813+0000 7f3fe0800700 1 -- 172.21.15.89:0/2139224886 >> 172.21.15.89:0/2139224886 conn(0x7f3fd80cf190 msgr2=0x7f3fd80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:31.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.813+0000 7f3fe0800700 1 -- 172.21.15.89:0/2139224886 shutdown_connections 2024-04-02T07:55:31.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.813+0000 7f3fe0800700 1 -- 172.21.15.89:0/2139224886 wait complete. 2024-04-02T07:55:31.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.815+0000 7f3fe0800700 1 Processor -- start 2024-04-02T07:55:31.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.815+0000 7f3fe0800700 1 -- start start 2024-04-02T07:55:31.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.816+0000 7f3fe0800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd80ac600 0x7f3fd8153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:31.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.816+0000 7f3fe0800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fd815b5d0 0x7f3fd8150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:31.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.816+0000 7f3fe0800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fd81536d0 0x7f3fd8151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:31.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.816+0000 7f3fe0800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd80cca70 con 0x7f3fd80ac600 2024-04-02T07:55:31.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.816+0000 7f3fe0800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd80ccd50 con 0x7f3fd81536d0 2024-04-02T07:55:31.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.816+0000 7f3fe0800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd80cd030 con 0x7f3fd815b5d0 2024-04-02T07:55:31.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.816+0000 7f3fdde00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd80ac600 0x7f3fd8153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:31.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.816+0000 7f3fdde00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd80ac600 0x7f3fd8153190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54794/0 (socket says 172.21.15.89:54794) 2024-04-02T07:55:31.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.816+0000 7f3fdde00700 1 -- 172.21.15.89:0/4079324278 learned_addr learned my addr 172.21.15.89:0/4079324278 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:31.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.816+0000 7f3fde800700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fd81536d0 0x7f3fd8151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:31.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.816+0000 7f3fdd400700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fd815b5d0 0x7f3fd8150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:31.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.817+0000 7f3fdde00700 1 -- 172.21.15.89:0/4079324278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fd815b5d0 msgr2=0x7f3fd8150cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:31.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.817+0000 7f3fdde00700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fd815b5d0 0x7f3fd8150cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.817+0000 7f3fdde00700 1 -- 172.21.15.89:0/4079324278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fd81536d0 msgr2=0x7f3fd8151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:31.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.817+0000 7f3fdde00700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fd81536d0 0x7f3fd8151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:31.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.817+0000 7f3fdde00700 1 -- 172.21.15.89:0/4079324278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc8009550 con 0x7f3fd80ac600 2024-04-02T07:55:31.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.817+0000 7f3fdd400700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fd815b5d0 0x7f3fd8150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:31.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.817+0000 7f3fdde00700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd80ac600 0x7f3fd8153190 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc00c4e0 tx=0x7f3fcc00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:31.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.817+0000 7f3fd2000700 1 -- 172.21.15.89:0/4079324278 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc0052d0 con 0x7f3fd80ac600 2024-04-02T07:55:31.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.817+0000 7f3fd2000700 1 -- 172.21.15.89:0/4079324278 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fcc005430 con 0x7f3fd80ac600 2024-04-02T07:55:31.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.817+0000 7f3fd2000700 1 -- 172.21.15.89:0/4079324278 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc0038c0 con 0x7f3fd80ac600 2024-04-02T07:55:31.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.817+0000 7f3fe0800700 1 -- 172.21.15.89:0/4079324278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd80cd310 con 0x7f3fd80ac600 2024-04-02T07:55:31.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.817+0000 7f3fe0800700 1 -- 172.21.15.89:0/4079324278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd8144a60 con 0x7f3fd80ac600 2024-04-02T07:55:31.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.818+0000 7f3fe0800700 1 -- 172.21.15.89:0/4079324278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd80cba10 con 0x7f3fd80ac600 2024-04-02T07:55:31.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.820+0000 7f3fd2000700 1 -- 172.21.15.89:0/4079324278 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3fcc003a20 con 0x7f3fd80ac600 2024-04-02T07:55:31.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.820+0000 7f3fd2000700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3fbc06adb0 0x7f3fbc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:31.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.820+0000 7f3fd2000700 1 -- 172.21.15.89:0/4079324278 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f3fcc088a60 con 0x7f3fd80ac600 2024-04-02T07:55:31.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.821+0000 7f3fdd400700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3fbc06adb0 0x7f3fbc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:31.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.821+0000 7f3fdd400700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3fbc06adb0 0x7f3fbc06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009c10 tx=0x7f3fc8008030 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:31.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:31.825+0000 7f3fd2000700 1 -- 172.21.15.89:0/4079324278 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3fcc0539b0 con 0x7f3fd80ac600 2024-04-02T07:55:32.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.028+0000 7f3fe0800700 1 -- 172.21.15.89:0/4079324278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fd8067720 con 0x7f3fd80ac600 2024-04-02T07:55:32.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.030+0000 7f3fd2000700 1 -- 172.21.15.89:0/4079324278 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16700 (secure 0 0 0) 0x7f3fcc017090 con 0x7f3fd80ac600 2024-04-02T07:55:32.031 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:32.031 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":139,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:55:31.571370+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:55:29.329907+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-02T07:55:30.555350+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:55:32.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.032+0000 7f3fe0800700 1 -- 172.21.15.89:0/4079324278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3fbc06adb0 msgr2=0x7f3fbc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:32.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.032+0000 7f3fe0800700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3fbc06adb0 0x7f3fbc06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009c10 tx=0x7f3fc8008030 comp rx=0 tx=0).stop 2024-04-02T07:55:32.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.032+0000 7f3fe0800700 1 -- 172.21.15.89:0/4079324278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd80ac600 msgr2=0x7f3fd8153190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:32.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.032+0000 7f3fe0800700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd80ac600 0x7f3fd8153190 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc00c4e0 tx=0x7f3fcc00c7f0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.032+0000 7f3fe0800700 1 -- 172.21.15.89:0/4079324278 shutdown_connections 2024-04-02T07:55:32.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.032+0000 7f3fe0800700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3fbc06adb0 0x7f3fbc06d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.032+0000 7f3fe0800700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fd81536d0 0x7f3fd8151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.032+0000 7f3fe0800700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd80ac600 0x7f3fd8153190 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.032+0000 7f3fe0800700 1 --2- 172.21.15.89:0/4079324278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3fd815b5d0 0x7f3fd8150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.032+0000 7f3fe0800700 1 -- 172.21.15.89:0/4079324278 >> 172.21.15.89:0/4079324278 conn(0x7f3fd80cf190 msgr2=0x7f3fd8155f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:32.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.032+0000 7f3fe0800700 1 -- 172.21.15.89:0/4079324278 shutdown_connections 2024-04-02T07:55:32.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.033+0000 7f3fe0800700 1 -- 172.21.15.89:0/4079324278 wait complete. 2024-04-02T07:55:32.051 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:55:32.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.243+0000 7f8757e00700 1 Processor -- start 2024-04-02T07:55:32.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.243+0000 7f8757e00700 1 -- start start 2024-04-02T07:55:32.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.244+0000 7f8757e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f875015d710 0x7f87500d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.244+0000 7f8757e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f875009aba0 0x7f87500d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.244+0000 7f8757e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875006bb40 con 0x7f875006bf40 2024-04-02T07:55:32.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.244+0000 7f8757e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8750099b00 con 0x7f875009aba0 2024-04-02T07:55:32.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.244+0000 7f8757e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8750099de0 con 0x7f875015d710 2024-04-02T07:55:32.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.245+0000 7f8755e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f875006bf40 0x7f87500ccbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44748/0 (socket says 172.21.15.89:44748) 2024-04-02T07:55:32.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.245+0000 7f8755e00700 1 -- 172.21.15.89:0/2314881476 learned_addr learned my addr 172.21.15.89:0/2314881476 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.245+0000 7f8754a00700 1 --2- 172.21.15.89:0/2314881476 >> v2:172.21.15.113:3300/0 conn(0x7f875009aba0 0x7f87500d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.245+0000 7f8755400700 1 --2- 172.21.15.89:0/2314881476 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f875015d710 0x7f87500d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.245+0000 7f874fe00700 1 -- 172.21.15.89:0/2314881476 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992584024 0 0) 0x7f875006bb40 con 0x7f875006bf40 2024-04-02T07:55:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.245+0000 7f874fe00700 1 -- 172.21.15.89:0/2314881476 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87380035b0 con 0x7f875006bf40 2024-04-02T07:55:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.245+0000 7f874fe00700 1 -- 172.21.15.89:0/2314881476 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3999924758 0 0) 0x7f87380035b0 con 0x7f875006bf40 2024-04-02T07:55:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.245+0000 7f874fe00700 1 -- 172.21.15.89:0/2314881476 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f875015d710 msgr2=0x7f87500d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.245+0000 7f874fe00700 1 --2- 172.21.15.89:0/2314881476 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f875015d710 0x7f87500d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.245+0000 7f874fe00700 1 -- 172.21.15.89:0/2314881476 >> v2:172.21.15.113:3300/0 conn(0x7f875009aba0 msgr2=0x7f87500d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.245+0000 7f874fe00700 1 --2- 172.21.15.89:0/2314881476 >> v2:172.21.15.113:3300/0 conn(0x7f875009aba0 0x7f87500d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.245+0000 7f874fe00700 1 -- 172.21.15.89:0/2314881476 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875009a0c0 con 0x7f875006bf40 2024-04-02T07:55:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.245+0000 7f874fe00700 1 -- 172.21.15.89:0/2314881476 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f87480019f0 con 0x7f875006bf40 2024-04-02T07:55:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f874fe00700 1 -- 172.21.15.89:0/2314881476 >> v1:172.21.15.89:6789/0 conn(0x7f875006bf40 legacy=0x7f87500ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f874fe00700 1 --2- 172.21.15.89:0/2314881476 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f875015d710 0x7f8738003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f874fe00700 1 --2- 172.21.15.89:0/2314881476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8738003f80 0x7f8738004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f874fe00700 1 --2- 172.21.15.89:0/2314881476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8738004d90 0x7f8738005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f874fe00700 1 -- 172.21.15.89:0/2314881476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8750099b00 con 0x7f8738004d90 2024-04-02T07:55:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f874fe00700 1 -- 172.21.15.89:0/2314881476 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8750099de0 con 0x7f8738003f80 2024-04-02T07:55:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f874fe00700 1 -- 172.21.15.89:0/2314881476 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873800ba10 con 0x7f875015d710 2024-04-02T07:55:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f8755e00700 1 --2- 172.21.15.89:0/2314881476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8738004d90 0x7f8738005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f8754a00700 1 --2- 172.21.15.89:0/2314881476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8738003f80 0x7f8738004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f8755400700 1 --2- 172.21.15.89:0/2314881476 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f875015d710 0x7f8738003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f8755e00700 1 -- 172.21.15.89:0/2314881476 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f875015d710 msgr2=0x7f8738003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f8755e00700 1 --2- 172.21.15.89:0/2314881476 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f875015d710 0x7f8738003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f8755e00700 1 -- 172.21.15.89:0/2314881476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8738003f80 msgr2=0x7f8738004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f8755e00700 1 --2- 172.21.15.89:0/2314881476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8738003f80 0x7f8738004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.246+0000 7f8755e00700 1 -- 172.21.15.89:0/2314881476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87380035b0 con 0x7f8738004d90 2024-04-02T07:55:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.247+0000 7f8755e00700 1 --2- 172.21.15.89:0/2314881476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8738004d90 0x7f8738005210 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f874800a860 tx=0x7f8738005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172565d253ea7fbe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.247+0000 7f874fe00700 1 -- 172.21.15.89:0/2314881476 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8748012600 con 0x7f8738004d90 2024-04-02T07:55:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.247+0000 7f874fe00700 1 -- 172.21.15.89:0/2314881476 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8748012780 con 0x7f8738004d90 2024-04-02T07:55:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.247+0000 7f874fe00700 1 -- 172.21.15.89:0/2314881476 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8748012b30 con 0x7f8738004d90 2024-04-02T07:55:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.247+0000 7f8757e00700 1 -- 172.21.15.89:0/2314881476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8738004d90 msgr2=0x7f8738005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.247+0000 7f8757e00700 1 --2- 172.21.15.89:0/2314881476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8738004d90 0x7f8738005210 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f874800a860 tx=0x7f8738005880 comp rx=0 tx=0).stop 2024-04-02T07:55:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.247+0000 7f8755e00700 1 -- 172.21.15.89:0/2314881476 reap_dead start 2024-04-02T07:55:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.247+0000 7f8757e00700 1 -- 172.21.15.89:0/2314881476 shutdown_connections 2024-04-02T07:55:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.247+0000 7f8757e00700 1 -- 172.21.15.89:0/2314881476 >> 172.21.15.89:0/2314881476 conn(0x7f87500d13e0 msgr2=0x7f87500d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.247+0000 7f8757e00700 1 -- 172.21.15.89:0/2314881476 shutdown_connections 2024-04-02T07:55:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.248+0000 7f8757e00700 1 -- 172.21.15.89:0/2314881476 wait complete. 2024-04-02T07:55:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.249+0000 7f8757e00700 1 Processor -- start 2024-04-02T07:55:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.249+0000 7f8757e00700 1 -- start start 2024-04-02T07:55:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.250+0000 7f8757e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8738004d90 0x7f87500c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.250+0000 7f8757e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87500c84b0 0x7f87501f66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.250+0000 7f8757e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87501f6c10 0x7f87501f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.250+0000 7f8757e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87500cb3b0 con 0x7f87501f6c10 2024-04-02T07:55:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.250+0000 7f8757e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87500cb7d0 con 0x7f87500c84b0 2024-04-02T07:55:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.250+0000 7f8757e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87500caa10 con 0x7f8738004d90 2024-04-02T07:55:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.250+0000 7f8755e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87501f6c10 0x7f87501f70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f8755e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87501f6c10 0x7f87501f70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54810/0 (socket says 172.21.15.89:54810) 2024-04-02T07:55:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f8755e00700 1 -- 172.21.15.89:0/2110250886 learned_addr learned my addr 172.21.15.89:0/2110250886 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f8754a00700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87500c84b0 0x7f87501f66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f8755400700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8738004d90 0x7f87500c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f8755e00700 1 -- 172.21.15.89:0/2110250886 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8738004d90 msgr2=0x7f87500c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f8755e00700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8738004d90 0x7f87500c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f8755e00700 1 -- 172.21.15.89:0/2110250886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87500c84b0 msgr2=0x7f87501f66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f8755e00700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87500c84b0 0x7f87501f66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f8755e00700 1 -- 172.21.15.89:0/2110250886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87500cacf0 con 0x7f87501f6c10 2024-04-02T07:55:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f8755400700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8738004d90 0x7f87500c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f8754a00700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87500c84b0 0x7f87501f66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f8755e00700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87501f6c10 0x7f87501f70c0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f8748012000 tx=0x7f8748001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f874d600700 1 -- 172.21.15.89:0/2110250886 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874801e610 con 0x7f87501f6c10 2024-04-02T07:55:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f874d600700 1 -- 172.21.15.89:0/2110250886 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874801e770 con 0x7f87501f6c10 2024-04-02T07:55:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f8757e00700 1 -- 172.21.15.89:0/2110250886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87500cafd0 con 0x7f87501f6c10 2024-04-02T07:55:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.251+0000 7f874d600700 1 -- 172.21.15.89:0/2110250886 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874801eb00 con 0x7f87501f6c10 2024-04-02T07:55:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.252+0000 7f8757e00700 1 -- 172.21.15.89:0/2110250886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87500c8930 con 0x7f87501f6c10 2024-04-02T07:55:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.252+0000 7f8757e00700 1 -- 172.21.15.89:0/2110250886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8750067720 con 0x7f87501f6c10 2024-04-02T07:55:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.253+0000 7f874d600700 1 -- 172.21.15.89:0/2110250886 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8748019030 con 0x7f87501f6c10 2024-04-02T07:55:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.254+0000 7f874d600700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f873006adb0 0x7f873006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.254+0000 7f874d600700 1 -- 172.21.15.89:0/2110250886 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f874808b540 con 0x7f87501f6c10 2024-04-02T07:55:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.254+0000 7f8755400700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f873006adb0 0x7f873006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.255+0000 7f8755400700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f873006adb0 0x7f873006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8744009c60 tx=0x7f8744008040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:32.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.259+0000 7f874d600700 1 -- 172.21.15.89:0/2110250886 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8748056480 con 0x7f87501f6c10 2024-04-02T07:55:32.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.460+0000 7f8757e00700 1 -- 172.21.15.89:0/2110250886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f875015b2b0 con 0x7f87501f6c10 2024-04-02T07:55:32.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.462+0000 7f874d600700 1 -- 172.21.15.89:0/2110250886 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16700 (secure 0 0 0) 0x7f8748059aa0 con 0x7f87501f6c10 2024-04-02T07:55:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:32.463 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":139,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:55:31.571370+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:55:29.329907+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-02T07:55:30.555350+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:55:32.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.464+0000 7f8757e00700 1 -- 172.21.15.89:0/2110250886 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f873006adb0 msgr2=0x7f873006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:32.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.464+0000 7f8757e00700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f873006adb0 0x7f873006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8744009c60 tx=0x7f8744008040 comp rx=0 tx=0).stop 2024-04-02T07:55:32.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.464+0000 7f8757e00700 1 -- 172.21.15.89:0/2110250886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87501f6c10 msgr2=0x7f87501f70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:32.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.464+0000 7f8757e00700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87501f6c10 0x7f87501f70c0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f8748012000 tx=0x7f8748001d80 comp rx=0 tx=0).stop 2024-04-02T07:55:32.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.464+0000 7f8757e00700 1 -- 172.21.15.89:0/2110250886 shutdown_connections 2024-04-02T07:55:32.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.464+0000 7f8757e00700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f873006adb0 0x7f873006d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.464+0000 7f8757e00700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87501f6c10 0x7f87501f70c0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.464+0000 7f8757e00700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8738004d90 0x7f87500c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.464+0000 7f8757e00700 1 --2- 172.21.15.89:0/2110250886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87500c84b0 0x7f87501f66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.464+0000 7f8757e00700 1 -- 172.21.15.89:0/2110250886 >> 172.21.15.89:0/2110250886 conn(0x7f87500d13e0 msgr2=0x7f875006c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:32.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.464+0000 7f8757e00700 1 -- 172.21.15.89:0/2110250886 shutdown_connections 2024-04-02T07:55:32.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.464+0000 7f8757e00700 1 -- 172.21.15.89:0/2110250886 wait complete. 2024-04-02T07:55:32.483 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T07:55:32.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.679+0000 7f9b48400700 1 Processor -- start 2024-04-02T07:55:32.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.680+0000 7f9b48400700 1 -- start start 2024-04-02T07:55:32.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.681+0000 7f9b48400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b4015a5c0 0x7f9b400709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.681+0000 7f9b48400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9b400ce840 0x7f9b400cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.681+0000 7f9b48400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b400cb310 con 0x7f9b4015dfe0 2024-04-02T07:55:32.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.681+0000 7f9b48400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b40070f00 con 0x7f9b400ce840 2024-04-02T07:55:32.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.681+0000 7f9b48400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b400711a0 con 0x7f9b4015a5c0 2024-04-02T07:55:32.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.681+0000 7f9b46400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9b400ce840 0x7f9b400cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.681+0000 7f9b45000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9b4015dfe0 0x7f9b400ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44764/0 (socket says 172.21.15.89:44764) 2024-04-02T07:55:32.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.681+0000 7f9b45000700 1 -- 172.21.15.89:0/1501325304 learned_addr learned my addr 172.21.15.89:0/1501325304 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:32.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.681+0000 7f9b45a00700 1 --2- 172.21.15.89:0/1501325304 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b4015a5c0 0x7f9b400709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.682+0000 7f9b3fe00700 1 -- 172.21.15.89:0/1501325304 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008484506 0 0) 0x7f9b400cb310 con 0x7f9b4015dfe0 2024-04-02T07:55:32.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.682+0000 7f9b3fe00700 1 -- 172.21.15.89:0/1501325304 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b280057c0 con 0x7f9b4015dfe0 2024-04-02T07:55:32.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.682+0000 7f9b46400700 1 -- 172.21.15.89:0/1501325304 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b4015a5c0 msgr2=0x7f9b400709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:32.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.682+0000 7f9b46400700 1 --2- 172.21.15.89:0/1501325304 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b4015a5c0 0x7f9b400709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.682+0000 7f9b46400700 1 -- 172.21.15.89:0/1501325304 >> v1:172.21.15.89:6789/0 conn(0x7f9b4015dfe0 legacy=0x7f9b400ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:32.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.682+0000 7f9b46400700 1 -- 172.21.15.89:0/1501325304 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b40071480 con 0x7f9b400ce840 2024-04-02T07:55:32.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.682+0000 7f9b46400700 1 --2- 172.21.15.89:0/1501325304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b400ce840 0x7f9b400cba80 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009d90 tx=0x7f9b3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca79065c754b100 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:32.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.682+0000 7f9b3fe00700 1 -- 172.21.15.89:0/1501325304 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3800e070 con 0x7f9b400ce840 2024-04-02T07:55:32.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.682+0000 7f9b3fe00700 1 -- 172.21.15.89:0/1501325304 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3800bb20 con 0x7f9b400ce840 2024-04-02T07:55:32.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.682+0000 7f9b3fe00700 1 -- 172.21.15.89:0/1501325304 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b38012430 con 0x7f9b400ce840 2024-04-02T07:55:32.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.683+0000 7f9b48400700 1 -- 172.21.15.89:0/1501325304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b400ce840 msgr2=0x7f9b400cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:32.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.683+0000 7f9b48400700 1 --2- 172.21.15.89:0/1501325304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b400ce840 0x7f9b400cba80 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009d90 tx=0x7f9b3800b330 comp rx=0 tx=0).stop 2024-04-02T07:55:32.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.683+0000 7f9b48400700 1 -- 172.21.15.89:0/1501325304 shutdown_connections 2024-04-02T07:55:32.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.683+0000 7f9b48400700 1 --2- 172.21.15.89:0/1501325304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b400ce840 0x7f9b400cba80 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.683+0000 7f9b48400700 1 --2- 172.21.15.89:0/1501325304 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b4015a5c0 0x7f9b400709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.683+0000 7f9b48400700 1 -- 172.21.15.89:0/1501325304 >> 172.21.15.89:0/1501325304 conn(0x7f9b400b0d70 msgr2=0x7f9b400b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:32.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.683+0000 7f9b48400700 1 -- 172.21.15.89:0/1501325304 shutdown_connections 2024-04-02T07:55:32.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.683+0000 7f9b48400700 1 -- 172.21.15.89:0/1501325304 wait complete. 2024-04-02T07:55:32.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.685+0000 7f9b48400700 1 Processor -- start 2024-04-02T07:55:32.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.685+0000 7f9b48400700 1 -- start start 2024-04-02T07:55:32.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b48400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b400ce840 0x7f9b4015bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b48400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b4015a5c0 0x7f9b400ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b48400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4015dfe0 0x7f9b401f3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b48400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b401f4210 con 0x7f9b4015dfe0 2024-04-02T07:55:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b48400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b400ac710 con 0x7f9b400ce840 2024-04-02T07:55:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b48400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b400ac9f0 con 0x7f9b4015a5c0 2024-04-02T07:55:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b46400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4015dfe0 0x7f9b401f3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b46400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4015dfe0 0x7f9b401f3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54832/0 (socket says 172.21.15.89:54832) 2024-04-02T07:55:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b46400700 1 -- 172.21.15.89:0/1736510087 learned_addr learned my addr 172.21.15.89:0/1736510087 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b45000700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b4015a5c0 0x7f9b400ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b45a00700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b400ce840 0x7f9b4015bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b46400700 1 -- 172.21.15.89:0/1736510087 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b4015a5c0 msgr2=0x7f9b400ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b46400700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b4015a5c0 0x7f9b400ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b46400700 1 -- 172.21.15.89:0/1736510087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b400ce840 msgr2=0x7f9b4015bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b46400700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b400ce840 0x7f9b4015bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:32.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b46400700 1 -- 172.21.15.89:0/1736510087 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b38009a70 con 0x7f9b4015dfe0 2024-04-02T07:55:32.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b45000700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b4015a5c0 0x7f9b400ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:55:32.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.686+0000 7f9b45a00700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b400ce840 0x7f9b4015bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:55:32.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.687+0000 7f9b46400700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4015dfe0 0x7f9b401f3cd0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f9b38014040 tx=0x7f9b38004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:32.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.687+0000 7f9b3d600700 1 -- 172.21.15.89:0/1736510087 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b380129f0 con 0x7f9b4015dfe0 2024-04-02T07:55:32.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.687+0000 7f9b3d600700 1 -- 172.21.15.89:0/1736510087 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b38020070 con 0x7f9b4015dfe0 2024-04-02T07:55:32.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.687+0000 7f9b48400700 1 -- 172.21.15.89:0/1736510087 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b400accd0 con 0x7f9b4015dfe0 2024-04-02T07:55:32.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.687+0000 7f9b3d600700 1 -- 172.21.15.89:0/1736510087 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b38012db0 con 0x7f9b4015dfe0 2024-04-02T07:55:32.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.687+0000 7f9b48400700 1 -- 172.21.15.89:0/1736510087 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b400c99d0 con 0x7f9b4015dfe0 2024-04-02T07:55:32.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.688+0000 7f9b48400700 1 -- 172.21.15.89:0/1736510087 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b40067720 con 0x7f9b4015dfe0 2024-04-02T07:55:32.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.689+0000 7f9b3d600700 1 -- 172.21.15.89:0/1736510087 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9b38004040 con 0x7f9b4015dfe0 2024-04-02T07:55:32.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.690+0000 7f9b3d600700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9b2006adb0 0x7f9b2006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:32.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.690+0000 7f9b3d600700 1 -- 172.21.15.89:0/1736510087 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f9b38022070 con 0x7f9b4015dfe0 2024-04-02T07:55:32.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.690+0000 7f9b45a00700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9b2006adb0 0x7f9b2006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:32.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.691+0000 7f9b45a00700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9b2006adb0 0x7f9b2006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9b34009b60 tx=0x7f9b34008040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:32.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.695+0000 7f9b3d600700 1 -- 172.21.15.89:0/1736510087 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9b38056270 con 0x7f9b4015dfe0 2024-04-02T07:55:32.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:32.942+0000 7f9b48400700 1 -- 172.21.15.89:0/1736510087 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9b40150b60 con 0x7f9b4015dfe0 2024-04-02T07:55:33.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.584+0000 7f9b3d600700 1 -- 172.21.15.89:0/1736510087 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v65) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9b38059890 con 0x7f9b4015dfe0 2024-04-02T07:55:33.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.586+0000 7f9b48400700 1 -- 172.21.15.89:0/1736510087 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9b2006adb0 msgr2=0x7f9b2006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:33.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.586+0000 7f9b48400700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9b2006adb0 0x7f9b2006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9b34009b60 tx=0x7f9b34008040 comp rx=0 tx=0).stop 2024-04-02T07:55:33.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.586+0000 7f9b48400700 1 -- 172.21.15.89:0/1736510087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4015dfe0 msgr2=0x7f9b401f3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:33.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.586+0000 7f9b48400700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4015dfe0 0x7f9b401f3cd0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f9b38014040 tx=0x7f9b38004ad0 comp rx=0 tx=0).stop 2024-04-02T07:55:33.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.586+0000 7f9b48400700 1 -- 172.21.15.89:0/1736510087 shutdown_connections 2024-04-02T07:55:33.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.586+0000 7f9b48400700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9b2006adb0 0x7f9b2006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:33.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.586+0000 7f9b48400700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b400ce840 0x7f9b4015bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:33.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.586+0000 7f9b48400700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4015dfe0 0x7f9b401f3cd0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:33.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.586+0000 7f9b48400700 1 --2- 172.21.15.89:0/1736510087 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b4015a5c0 0x7f9b400ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:33.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.586+0000 7f9b48400700 1 -- 172.21.15.89:0/1736510087 >> 172.21.15.89:0/1736510087 conn(0x7f9b400b0d70 msgr2=0x7f9b4006e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:33.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.586+0000 7f9b48400700 1 -- 172.21.15.89:0/1736510087 shutdown_connections 2024-04-02T07:55:33.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.586+0000 7f9b48400700 1 -- 172.21.15.89:0/1736510087 wait complete. 2024-04-02T07:55:33.605 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T07:55:33.606 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T07:55:33.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.799+0000 7fd176600700 1 Processor -- start 2024-04-02T07:55:33.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.800+0000 7fd176600700 1 -- start start 2024-04-02T07:55:33.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.801+0000 7fd176600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1701611b0 0x7fd17015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:33.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.801+0000 7fd176600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd1700cee10 0x7fd170161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:33.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.801+0000 7fd176600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1700ce270 con 0x7fd17015f4d0 2024-04-02T07:55:33.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.801+0000 7fd176600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17015b600 con 0x7fd1700cee10 2024-04-02T07:55:33.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.801+0000 7fd176600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1700d2760 con 0x7fd1701611b0 2024-04-02T07:55:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.801+0000 7fd16f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1701611b0 0x7fd17015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.801+0000 7fd16fe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd1700cee10 0x7fd170161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.801+0000 7fd16f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1701611b0 0x7fd17015ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:56930/0 (socket says 172.21.15.89:56930) 2024-04-02T07:55:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.801+0000 7fd16f400700 1 -- 172.21.15.89:0/2039726157 learned_addr learned my addr 172.21.15.89:0/2039726157 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.801+0000 7fd16fe00700 1 --2- 172.21.15.89:0/2039726157 >> v2:172.21.15.113:3300/0 conn(0x7fd1700cee10 0x7fd170161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:46946/0 (socket says 172.21.15.89:46946) 2024-04-02T07:55:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.802+0000 7fd16f400700 1 -- 172.21.15.89:0/2039726157 >> v2:172.21.15.113:3300/0 conn(0x7fd1700cee10 msgr2=0x7fd170161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.802+0000 7fd16f400700 1 --2- 172.21.15.89:0/2039726157 >> v2:172.21.15.113:3300/0 conn(0x7fd1700cee10 0x7fd170161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.802+0000 7fd16f400700 1 -- 172.21.15.89:0/2039726157 >> v1:172.21.15.113:6789/0 conn(0x7fd17015f4d0 legacy=0x7fd1700ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.802+0000 7fd16f400700 1 -- 172.21.15.89:0/2039726157 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1700d2a40 con 0x7fd1701611b0 2024-04-02T07:55:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.802+0000 7fd16f400700 1 --2- 172.21.15.89:0/2039726157 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1701611b0 0x7fd17015ef90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd1600098f0 tx=0x7fd160009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54bea4a40d5ce0ee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:33.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.802+0000 7fd16e000700 1 -- 172.21.15.89:0/2039726157 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd160011830 con 0x7fd1701611b0 2024-04-02T07:55:33.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.802+0000 7fd16e000700 1 -- 172.21.15.89:0/2039726157 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd160011990 con 0x7fd1701611b0 2024-04-02T07:55:33.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.802+0000 7fd16e000700 1 -- 172.21.15.89:0/2039726157 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd160011d20 con 0x7fd1701611b0 2024-04-02T07:55:33.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.803+0000 7fd176600700 1 -- 172.21.15.89:0/2039726157 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1701611b0 msgr2=0x7fd17015ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:33.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.803+0000 7fd176600700 1 --2- 172.21.15.89:0/2039726157 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1701611b0 0x7fd17015ef90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd1600098f0 tx=0x7fd160009c00 comp rx=0 tx=0).stop 2024-04-02T07:55:33.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.803+0000 7fd176600700 1 -- 172.21.15.89:0/2039726157 shutdown_connections 2024-04-02T07:55:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.803+0000 7fd176600700 1 --2- 172.21.15.89:0/2039726157 >> v2:172.21.15.113:3300/0 conn(0x7fd1700cee10 0x7fd170161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.803+0000 7fd176600700 1 --2- 172.21.15.89:0/2039726157 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1701611b0 0x7fd17015ef90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.803+0000 7fd176600700 1 -- 172.21.15.89:0/2039726157 >> 172.21.15.89:0/2039726157 conn(0x7fd17008aff0 msgr2=0x7fd1700d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.803+0000 7fd176600700 1 -- 172.21.15.89:0/2039726157 shutdown_connections 2024-04-02T07:55:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.803+0000 7fd176600700 1 -- 172.21.15.89:0/2039726157 wait complete. 2024-04-02T07:55:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.805+0000 7fd176600700 1 Processor -- start 2024-04-02T07:55:33.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.805+0000 7fd176600700 1 -- start start 2024-04-02T07:55:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd176600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1700cee10 0x7fd170159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd176600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd17015f4d0 0x7fd1700cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd176600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1701611b0 0x7fd17006b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:33.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd176600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17015e630 con 0x7fd1700cee10 2024-04-02T07:55:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd176600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17015a430 con 0x7fd17015f4d0 2024-04-02T07:55:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd176600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1700cb010 con 0x7fd1701611b0 2024-04-02T07:55:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd16f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1700cee10 0x7fd170159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd16f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1700cee10 0x7fd170159ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54846/0 (socket says 172.21.15.89:54846) 2024-04-02T07:55:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd16f400700 1 -- 172.21.15.89:0/3303713078 learned_addr learned my addr 172.21.15.89:0/3303713078 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd16fe00700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1701611b0 0x7fd17006b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd16ea00700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd17015f4d0 0x7fd1700cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd16f400700 1 -- 172.21.15.89:0/3303713078 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1701611b0 msgr2=0x7fd17006b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd16f400700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1701611b0 0x7fd17006b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd16f400700 1 -- 172.21.15.89:0/3303713078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd17015f4d0 msgr2=0x7fd1700cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd16f400700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd17015f4d0 0x7fd1700cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd16f400700 1 -- 172.21.15.89:0/3303713078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd160009550 con 0x7fd1700cee10 2024-04-02T07:55:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.806+0000 7fd16ea00700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd17015f4d0 0x7fd1700cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:33.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.807+0000 7fd16f400700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1700cee10 0x7fd170159ef0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fd160009c40 tx=0x7fd160004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:33.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.807+0000 7fd167400700 1 -- 172.21.15.89:0/3303713078 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd160004590 con 0x7fd1700cee10 2024-04-02T07:55:33.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.807+0000 7fd167400700 1 -- 172.21.15.89:0/3303713078 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16001e070 con 0x7fd1700cee10 2024-04-02T07:55:33.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.807+0000 7fd176600700 1 -- 172.21.15.89:0/3303713078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1700cb260 con 0x7fd1700cee10 2024-04-02T07:55:33.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.807+0000 7fd167400700 1 -- 172.21.15.89:0/3303713078 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1600038e0 con 0x7fd1700cee10 2024-04-02T07:55:33.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.807+0000 7fd176600700 1 -- 172.21.15.89:0/3303713078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1700cb820 con 0x7fd1700cee10 2024-04-02T07:55:33.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.808+0000 7fd176600700 1 -- 172.21.15.89:0/3303713078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd170067720 con 0x7fd1700cee10 2024-04-02T07:55:33.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.809+0000 7fd167400700 1 -- 172.21.15.89:0/3303713078 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd160003a40 con 0x7fd1700cee10 2024-04-02T07:55:33.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.810+0000 7fd167400700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd15006adb0 0x7fd15006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:33.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.810+0000 7fd167400700 1 -- 172.21.15.89:0/3303713078 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fd16008a4a0 con 0x7fd1700cee10 2024-04-02T07:55:33.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.810+0000 7fd16ea00700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd15006adb0 0x7fd15006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:33.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.811+0000 7fd16ea00700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd15006adb0 0x7fd15006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd1700b58a0 tx=0x7fd15c009040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:33.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:33.814+0000 7fd167400700 1 -- 172.21.15.89:0/3303713078 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd160055480 con 0x7fd1700cee10 2024-04-02T07:55:34.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.064+0000 7fd176600700 1 -- 172.21.15.89:0/3303713078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd1700b5d00 con 0x7fd1700cee10 2024-04-02T07:55:34.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.589+0000 7fd167400700 1 -- 172.21.15.89:0/3303713078 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v66) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd160058aa0 con 0x7fd1700cee10 2024-04-02T07:55:34.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.592+0000 7fd176600700 1 -- 172.21.15.89:0/3303713078 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd15006adb0 msgr2=0x7fd15006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:34.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.592+0000 7fd176600700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd15006adb0 0x7fd15006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd1700b58a0 tx=0x7fd15c009040 comp rx=0 tx=0).stop 2024-04-02T07:55:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.592+0000 7fd176600700 1 -- 172.21.15.89:0/3303713078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1700cee10 msgr2=0x7fd170159ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.592+0000 7fd176600700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1700cee10 0x7fd170159ef0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fd160009c40 tx=0x7fd160004750 comp rx=0 tx=0).stop 2024-04-02T07:55:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.592+0000 7fd176600700 1 -- 172.21.15.89:0/3303713078 shutdown_connections 2024-04-02T07:55:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.592+0000 7fd176600700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd15006adb0 0x7fd15006d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.592+0000 7fd176600700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1701611b0 0x7fd17006b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.592+0000 7fd176600700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1700cee10 0x7fd170159ef0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.592+0000 7fd176600700 1 --2- 172.21.15.89:0/3303713078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd17015f4d0 0x7fd1700cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.592+0000 7fd176600700 1 -- 172.21.15.89:0/3303713078 >> 172.21.15.89:0/3303713078 conn(0x7fd17008aff0 msgr2=0x7fd170161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.592+0000 7fd176600700 1 -- 172.21.15.89:0/3303713078 shutdown_connections 2024-04-02T07:55:34.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.592+0000 7fd176600700 1 -- 172.21.15.89:0/3303713078 wait complete. 2024-04-02T07:55:34.612 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T07:55:34.612 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:55:34.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.804+0000 7fb776600700 1 Processor -- start 2024-04-02T07:55:34.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.804+0000 7fb776600700 1 -- start start 2024-04-02T07:55:34.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.805+0000 7fb776600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb77015d640 0x7fb7700d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:34.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.805+0000 7fb776600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7700ceec0 0x7fb7701612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:34.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.805+0000 7fb776600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7700d30d0 con 0x7fb770161800 2024-04-02T07:55:34.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.805+0000 7fb776600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7700b6840 con 0x7fb7700ceec0 2024-04-02T07:55:34.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.805+0000 7fb776600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb770161c20 con 0x7fb77015d640 2024-04-02T07:55:34.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.806+0000 7fb767e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7700ceec0 0x7fb7701612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:34.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.806+0000 7fb767e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7700ceec0 0x7fb7701612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54848/0 (socket says 172.21.15.89:54848) 2024-04-02T07:55:34.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.806+0000 7fb767e00700 1 -- 172.21.15.89:0/2250789702 learned_addr learned my addr 172.21.15.89:0/2250789702 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:34.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.806+0000 7fb76f400700 1 --2- 172.21.15.89:0/2250789702 >> v2:172.21.15.113:3300/0 conn(0x7fb77015d640 0x7fb7700d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:34.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.806+0000 7fb767e00700 1 -- 172.21.15.89:0/2250789702 >> v2:172.21.15.113:3300/0 conn(0x7fb77015d640 msgr2=0x7fb7700d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:34.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.806+0000 7fb767e00700 1 --2- 172.21.15.89:0/2250789702 >> v2:172.21.15.113:3300/0 conn(0x7fb77015d640 0x7fb7700d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:34.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.806+0000 7fb767e00700 1 -- 172.21.15.89:0/2250789702 >> v1:172.21.15.89:6789/0 conn(0x7fb770161800 legacy=0x7fb7700cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:34.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.806+0000 7fb767e00700 1 -- 172.21.15.89:0/2250789702 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7700d2880 con 0x7fb7700ceec0 2024-04-02T07:55:34.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.806+0000 7fb76ea00700 1 -- 172.21.15.89:0/2250789702 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291428884 0 0) 0x7fb7700d30d0 con 0x7fb770161800 2024-04-02T07:55:34.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.806+0000 7fb767e00700 1 --2- 172.21.15.89:0/2250789702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7700ceec0 0x7fb7701612c0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fb7700b5c20 tx=0x7fb758009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ada40b3f26c9e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:34.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.807+0000 7fb76ea00700 1 -- 172.21.15.89:0/2250789702 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb758011830 con 0x7fb7700ceec0 2024-04-02T07:55:34.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.807+0000 7fb76ea00700 1 -- 172.21.15.89:0/2250789702 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb758011990 con 0x7fb7700ceec0 2024-04-02T07:55:34.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.807+0000 7fb76ea00700 1 -- 172.21.15.89:0/2250789702 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb758011d20 con 0x7fb7700ceec0 2024-04-02T07:55:34.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.807+0000 7fb776600700 1 -- 172.21.15.89:0/2250789702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7700ceec0 msgr2=0x7fb7701612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:34.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.807+0000 7fb776600700 1 --2- 172.21.15.89:0/2250789702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7700ceec0 0x7fb7701612c0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fb7700b5c20 tx=0x7fb758009bd0 comp rx=0 tx=0).stop 2024-04-02T07:55:34.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.807+0000 7fb776600700 1 -- 172.21.15.89:0/2250789702 shutdown_connections 2024-04-02T07:55:34.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.807+0000 7fb776600700 1 --2- 172.21.15.89:0/2250789702 >> v2:172.21.15.113:3300/0 conn(0x7fb77015d640 0x7fb7700d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:34.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.807+0000 7fb776600700 1 --2- 172.21.15.89:0/2250789702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7700ceec0 0x7fb7701612c0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:34.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.807+0000 7fb776600700 1 -- 172.21.15.89:0/2250789702 >> 172.21.15.89:0/2250789702 conn(0x7fb77008aff0 msgr2=0x7fb7700d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:34.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.807+0000 7fb776600700 1 -- 172.21.15.89:0/2250789702 shutdown_connections 2024-04-02T07:55:34.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.807+0000 7fb776600700 1 -- 172.21.15.89:0/2250789702 wait complete. 2024-04-02T07:55:34.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.809+0000 7fb776600700 1 Processor -- start 2024-04-02T07:55:34.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.809+0000 7fb776600700 1 -- start start 2024-04-02T07:55:34.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.810+0000 7fb776600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7700ceec0 0x7fb7700a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:34.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.810+0000 7fb776600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb77015d640 0x7fb77009fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:34.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.810+0000 7fb776600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb770161800 0x7fb7700c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:34.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.810+0000 7fb776600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7700c8b30 con 0x7fb770161800 2024-04-02T07:55:34.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.810+0000 7fb776600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7700c7b80 con 0x7fb7700ceec0 2024-04-02T07:55:34.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.810+0000 7fb776600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7700c7e60 con 0x7fb77015d640 2024-04-02T07:55:34.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.810+0000 7fb76fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb770161800 0x7fb7700c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:34.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.810+0000 7fb76f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7700ceec0 0x7fb7700a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:34.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.810+0000 7fb76fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb770161800 0x7fb7700c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54860/0 (socket says 172.21.15.89:54860) 2024-04-02T07:55:34.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.810+0000 7fb76fe00700 1 -- 172.21.15.89:0/1468310965 learned_addr learned my addr 172.21.15.89:0/1468310965 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:34.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.810+0000 7fb767e00700 1 --2- 172.21.15.89:0/1468310965 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb77015d640 0x7fb77009fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:34.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.811+0000 7fb76fe00700 1 -- 172.21.15.89:0/1468310965 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb77015d640 msgr2=0x7fb77009fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:34.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.811+0000 7fb76fe00700 1 --2- 172.21.15.89:0/1468310965 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb77015d640 0x7fb77009fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:34.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.811+0000 7fb76fe00700 1 -- 172.21.15.89:0/1468310965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7700ceec0 msgr2=0x7fb7700a0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:34.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.811+0000 7fb76fe00700 1 --2- 172.21.15.89:0/1468310965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7700ceec0 0x7fb7700a0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:34.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.811+0000 7fb76fe00700 1 -- 172.21.15.89:0/1468310965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb758009550 con 0x7fb770161800 2024-04-02T07:55:34.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.811+0000 7fb76fe00700 1 --2- 172.21.15.89:0/1468310965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb770161800 0x7fb7700c7080 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fb768001560 tx=0x7fb76800cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:34.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.811+0000 7fb767400700 1 -- 172.21.15.89:0/1468310965 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb768003e30 con 0x7fb770161800 2024-04-02T07:55:34.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.811+0000 7fb767400700 1 -- 172.21.15.89:0/1468310965 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb768005070 con 0x7fb770161800 2024-04-02T07:55:34.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.811+0000 7fb776600700 1 -- 172.21.15.89:0/1468310965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7700c8df0 con 0x7fb770161800 2024-04-02T07:55:34.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.811+0000 7fb767400700 1 -- 172.21.15.89:0/1468310965 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb768005420 con 0x7fb770161800 2024-04-02T07:55:34.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.811+0000 7fb776600700 1 -- 172.21.15.89:0/1468310965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7700c86c0 con 0x7fb770161800 2024-04-02T07:55:34.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.812+0000 7fb776600700 1 -- 172.21.15.89:0/1468310965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7700c9310 con 0x7fb770161800 2024-04-02T07:55:34.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.813+0000 7fb767400700 1 -- 172.21.15.89:0/1468310965 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb768005580 con 0x7fb770161800 2024-04-02T07:55:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.814+0000 7fb767400700 1 --2- 172.21.15.89:0/1468310965 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb74c06adb0 0x7fb74c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.814+0000 7fb767400700 1 -- 172.21.15.89:0/1468310965 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fb768088d70 con 0x7fb770161800 2024-04-02T07:55:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.814+0000 7fb76f400700 1 --2- 172.21.15.89:0/1468310965 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb74c06adb0 0x7fb74c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.815+0000 7fb76f400700 1 --2- 172.21.15.89:0/1468310965 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb74c06adb0 0x7fb74c06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb7700ceb80 tx=0x7fb760009210 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:34.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:34.818+0000 7fb767400700 1 -- 172.21.15.89:0/1468310965 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb7700c9310 con 0x7fb770161800 2024-04-02T07:55:35.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.064+0000 7fb776600700 1 -- 172.21.15.89:0/1468310965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7700c9310 con 0x7fb770161800 2024-04-02T07:55:35.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.065+0000 7fb767400700 1 -- 172.21.15.89:0/1468310965 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4733 (secure 0 0 0) 0x7fb7700c9310 con 0x7fb770161800 2024-04-02T07:55:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:35.067 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/66826967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":66826967},{"type":"v1","addr":"172.21.15.113:6835","nonce":66826967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":7915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/1669765650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1669765650},{"type":"v1","addr":"172.21.15.113:6833","nonce":1669765650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":7918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3321722319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3321722319},{"type":"v1","addr":"172.21.15.89:6837","nonce":3321722319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:55:31.571731+0000","modified":"2024-04-02T07:55:34.583055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8436":{"gid":8436,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":4,"addr":"172.21.15.89:6836/417215494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":417215494},{"type":"v1","addr":"172.21.15.89:6836","nonce":417215494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-02T07:55:35.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.067+0000 7fb776600700 1 -- 172.21.15.89:0/1468310965 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb74c06adb0 msgr2=0x7fb74c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.067+0000 7fb776600700 1 --2- 172.21.15.89:0/1468310965 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb74c06adb0 0x7fb74c06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb7700ceb80 tx=0x7fb760009210 comp rx=0 tx=0).stop 2024-04-02T07:55:35.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.067+0000 7fb776600700 1 -- 172.21.15.89:0/1468310965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb770161800 msgr2=0x7fb7700c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.067+0000 7fb776600700 1 --2- 172.21.15.89:0/1468310965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb770161800 0x7fb7700c7080 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fb768001560 tx=0x7fb76800cb50 comp rx=0 tx=0).stop 2024-04-02T07:55:35.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.067+0000 7fb776600700 1 -- 172.21.15.89:0/1468310965 shutdown_connections 2024-04-02T07:55:35.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.067+0000 7fb776600700 1 --2- 172.21.15.89:0/1468310965 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb74c06adb0 0x7fb74c06d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.067+0000 7fb776600700 1 --2- 172.21.15.89:0/1468310965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7700ceec0 0x7fb7700a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.067+0000 7fb776600700 1 --2- 172.21.15.89:0/1468310965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb770161800 0x7fb7700c7080 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.067+0000 7fb776600700 1 --2- 172.21.15.89:0/1468310965 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb77015d640 0x7fb77009fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.067+0000 7fb776600700 1 -- 172.21.15.89:0/1468310965 >> 172.21.15.89:0/1468310965 conn(0x7fb77008aff0 msgr2=0x7fb7700b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:35.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.068+0000 7fb776600700 1 -- 172.21.15.89:0/1468310965 shutdown_connections 2024-04-02T07:55:35.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.068+0000 7fb776600700 1 -- 172.21.15.89:0/1468310965 wait complete. 2024-04-02T07:55:35.070 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 66 2024-04-02T07:55:35.088 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:55:35.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.278+0000 7f72fb000700 1 Processor -- start 2024-04-02T07:55:35.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.278+0000 7f72fb000700 1 -- start start 2024-04-02T07:55:35.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.279+0000 7f72fb000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f72f40d18b0 0x7f72f40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:35.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.279+0000 7f72fb000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f40ceda0 con 0x7f72f4161590 2024-04-02T07:55:35.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.279+0000 7f72fb000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f41612f0 con 0x7f72f4160f10 2024-04-02T07:55:35.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.279+0000 7f72fb000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f40d1d30 con 0x7f72f40d18b0 2024-04-02T07:55:35.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.279+0000 7f72f9000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f72f4161590 0x7f72f40b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44778/0 (socket says 172.21.15.89:44778) 2024-04-02T07:55:35.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.279+0000 7f72f9000700 1 -- 172.21.15.89:0/2225724236 learned_addr learned my addr 172.21.15.89:0/2225724236 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:35.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.279+0000 7f72f3400700 1 --2- 172.21.15.89:0/2225724236 >> v2:172.21.15.89:3300/0 conn(0x7f72f40d18b0 0x7f72f40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:35.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.280+0000 7f72f2a00700 1 -- 172.21.15.89:0/2225724236 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806377040 0 0) 0x7f72f40ceda0 con 0x7f72f4161590 2024-04-02T07:55:35.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.280+0000 7f72f2a00700 1 -- 172.21.15.89:0/2225724236 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d80035b0 con 0x7f72f4161590 2024-04-02T07:55:35.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.280+0000 7f72f3400700 1 -- 172.21.15.89:0/2225724236 >> v1:172.21.15.113:6789/0 conn(0x7f72f4160f10 legacy=0x7f72f40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.280+0000 7f72f2a00700 1 -- 172.21.15.89:0/2225724236 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343380024 0 0) 0x7f72f41612f0 con 0x7f72f4160f10 2024-04-02T07:55:35.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.280+0000 7f72f3400700 1 -- 172.21.15.89:0/2225724236 >> v1:172.21.15.89:6789/0 conn(0x7f72f4161590 legacy=0x7f72f40b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.280+0000 7f72f3400700 1 -- 172.21.15.89:0/2225724236 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f40ce470 con 0x7f72f40d18b0 2024-04-02T07:55:35.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.280+0000 7f72f3400700 1 --2- 172.21.15.89:0/2225724236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f40d18b0 0x7f72f40cf230 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f72e0009840 tx=0x7f72e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64397a36e228e64d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:35.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.280+0000 7f72f2a00700 1 -- 172.21.15.89:0/2225724236 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0011830 con 0x7f72f40d18b0 2024-04-02T07:55:35.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.280+0000 7f72f2a00700 1 -- 172.21.15.89:0/2225724236 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e0011990 con 0x7f72f40d18b0 2024-04-02T07:55:35.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.280+0000 7f72f2a00700 1 -- 172.21.15.89:0/2225724236 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0011d20 con 0x7f72f40d18b0 2024-04-02T07:55:35.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.281+0000 7f72fb000700 1 -- 172.21.15.89:0/2225724236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f40d18b0 msgr2=0x7f72f40cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.281+0000 7f72fb000700 1 --2- 172.21.15.89:0/2225724236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f40d18b0 0x7f72f40cf230 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f72e0009840 tx=0x7f72e0009b50 comp rx=0 tx=0).stop 2024-04-02T07:55:35.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.281+0000 7f72fb000700 1 -- 172.21.15.89:0/2225724236 shutdown_connections 2024-04-02T07:55:35.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.281+0000 7f72fb000700 1 --2- 172.21.15.89:0/2225724236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f40d18b0 0x7f72f40cf230 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.281+0000 7f72fb000700 1 -- 172.21.15.89:0/2225724236 >> 172.21.15.89:0/2225724236 conn(0x7f72f408aff0 msgr2=0x7f72f40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:35.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.281+0000 7f72fb000700 1 -- 172.21.15.89:0/2225724236 shutdown_connections 2024-04-02T07:55:35.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.281+0000 7f72fb000700 1 -- 172.21.15.89:0/2225724236 wait complete. 2024-04-02T07:55:35.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.282+0000 7f72fb000700 1 Processor -- start 2024-04-02T07:55:35.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.283+0000 7f72fb000700 1 -- start start 2024-04-02T07:55:35.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.283+0000 7f72fb000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f40d18b0 0x7f72f4158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:35.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.283+0000 7f72fb000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72f4160f10 0x7f72f41f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:35.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.283+0000 7f72fb000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72f4161590 0x7f72f40b8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:35.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.283+0000 7f72fb000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f41594d0 con 0x7f72f40d18b0 2024-04-02T07:55:35.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.283+0000 7f72fb000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f40b8440 con 0x7f72f4161590 2024-04-02T07:55:35.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.283+0000 7f72fb000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f40b8720 con 0x7f72f4160f10 2024-04-02T07:55:35.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.284+0000 7f72f3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f40d18b0 0x7f72f4158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.284+0000 7f72f3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f40d18b0 0x7f72f4158f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54892/0 (socket says 172.21.15.89:54892) 2024-04-02T07:55:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.284+0000 7f72f3e00700 1 -- 172.21.15.89:0/2514353144 learned_addr learned my addr 172.21.15.89:0/2514353144 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.284+0000 7f72f3400700 1 --2- 172.21.15.89:0/2514353144 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72f4160f10 0x7f72f41f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.284+0000 7f72f9000700 1 --2- 172.21.15.89:0/2514353144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72f4161590 0x7f72f40b8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.284+0000 7f72f3e00700 1 -- 172.21.15.89:0/2514353144 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72f4160f10 msgr2=0x7f72f41f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.284+0000 7f72f3e00700 1 --2- 172.21.15.89:0/2514353144 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72f4160f10 0x7f72f41f33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.284+0000 7f72f3e00700 1 -- 172.21.15.89:0/2514353144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72f4161590 msgr2=0x7f72f40b8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.284+0000 7f72f3e00700 1 --2- 172.21.15.89:0/2514353144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72f4161590 0x7f72f40b8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.284+0000 7f72f3e00700 1 -- 172.21.15.89:0/2514353144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e00094a0 con 0x7f72f40d18b0 2024-04-02T07:55:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.284+0000 7f72f3e00700 1 --2- 172.21.15.89:0/2514353144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f40d18b0 0x7f72f4158f90 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f72e400c550 tx=0x7f72e400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.285+0000 7f72ebe00700 1 -- 172.21.15.89:0/2514353144 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e40054e0 con 0x7f72f40d18b0 2024-04-02T07:55:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.285+0000 7f72ebe00700 1 -- 172.21.15.89:0/2514353144 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e4003680 con 0x7f72f40d18b0 2024-04-02T07:55:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.285+0000 7f72ebe00700 1 -- 172.21.15.89:0/2514353144 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4003a30 con 0x7f72f40d18b0 2024-04-02T07:55:35.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.285+0000 7f72fb000700 1 -- 172.21.15.89:0/2514353144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f4158790 con 0x7f72f40d18b0 2024-04-02T07:55:35.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.285+0000 7f72fb000700 1 -- 172.21.15.89:0/2514353144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f41f3ed0 con 0x7f72f40d18b0 2024-04-02T07:55:35.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.286+0000 7f72fb000700 1 -- 172.21.15.89:0/2514353144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f4158cb0 con 0x7f72f40d18b0 2024-04-02T07:55:35.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.287+0000 7f72ebe00700 1 -- 172.21.15.89:0/2514353144 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f72e4003b90 con 0x7f72f40d18b0 2024-04-02T07:55:35.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.288+0000 7f72ebe00700 1 --2- 172.21.15.89:0/2514353144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f72d406adb0 0x7f72d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:35.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.288+0000 7f72ebe00700 1 -- 172.21.15.89:0/2514353144 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f72e4088ac0 con 0x7f72f40d18b0 2024-04-02T07:55:35.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.288+0000 7f72f3400700 1 --2- 172.21.15.89:0/2514353144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f72d406adb0 0x7f72d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:35.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.289+0000 7f72f3400700 1 --2- 172.21.15.89:0/2514353144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f72d406adb0 0x7f72d406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f72e0009fd0 tx=0x7f72e001c040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:35.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.292+0000 7f72ebe00700 1 -- 172.21.15.89:0/2514353144 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f72f4158cb0 con 0x7f72f40d18b0 2024-04-02T07:55:35.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.495+0000 7f72fb000700 1 -- 172.21.15.89:0/2514353144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72f4158cb0 con 0x7f72f40d18b0 2024-04-02T07:55:35.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.496+0000 7f72ebe00700 1 -- 172.21.15.89:0/2514353144 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16687 (secure 0 0 0) 0x7f72f4158cb0 con 0x7f72f40d18b0 2024-04-02T07:55:35.498 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:35.498 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":140,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:55:32.574255+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:55:29.329907+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-02T07:55:30.555350+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:55:35.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.498+0000 7f72fb000700 1 -- 172.21.15.89:0/2514353144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f72d406adb0 msgr2=0x7f72d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.498+0000 7f72fb000700 1 --2- 172.21.15.89:0/2514353144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f72d406adb0 0x7f72d406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f72e0009fd0 tx=0x7f72e001c040 comp rx=0 tx=0).stop 2024-04-02T07:55:35.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.498+0000 7f72fb000700 1 -- 172.21.15.89:0/2514353144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f40d18b0 msgr2=0x7f72f4158f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.498+0000 7f72fb000700 1 --2- 172.21.15.89:0/2514353144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f40d18b0 0x7f72f4158f90 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f72e400c550 tx=0x7f72e400c910 comp rx=0 tx=0).stop 2024-04-02T07:55:35.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.499+0000 7f72fb000700 1 -- 172.21.15.89:0/2514353144 shutdown_connections 2024-04-02T07:55:35.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.499+0000 7f72fb000700 1 --2- 172.21.15.89:0/2514353144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f72d406adb0 0x7f72d406d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.499+0000 7f72fb000700 1 --2- 172.21.15.89:0/2514353144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72f4161590 0x7f72f40b8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.499+0000 7f72fb000700 1 --2- 172.21.15.89:0/2514353144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f40d18b0 0x7f72f4158f90 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.499+0000 7f72fb000700 1 --2- 172.21.15.89:0/2514353144 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72f4160f10 0x7f72f41f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.499+0000 7f72fb000700 1 -- 172.21.15.89:0/2514353144 >> 172.21.15.89:0/2514353144 conn(0x7f72f408aff0 msgr2=0x7f72f40cc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:35.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.499+0000 7f72fb000700 1 -- 172.21.15.89:0/2514353144 shutdown_connections 2024-04-02T07:55:35.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.499+0000 7f72fb000700 1 -- 172.21.15.89:0/2514353144 wait complete. 2024-04-02T07:55:35.519 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T07:55:35.519 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:55:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.709+0000 7f6bade00700 1 Processor -- start 2024-04-02T07:55:35.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.709+0000 7f6bade00700 1 -- start start 2024-04-02T07:55:35.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.710+0000 7f6bade00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6ba80d18b0 0x7f6ba80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:35.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.710+0000 7f6bade00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba80ceda0 con 0x7f6ba8161590 2024-04-02T07:55:35.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.710+0000 7f6bade00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba81612f0 con 0x7f6ba8160f10 2024-04-02T07:55:35.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.710+0000 7f6bade00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba80d1d30 con 0x7f6ba80d18b0 2024-04-02T07:55:35.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.711+0000 7f6ba6000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6ba80d18b0 0x7f6ba80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:35.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.711+0000 7f6ba7400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6ba8161590 0x7f6ba80b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44790/0 (socket says 172.21.15.89:44790) 2024-04-02T07:55:35.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.711+0000 7f6ba7400700 1 -- 172.21.15.89:0/868932091 learned_addr learned my addr 172.21.15.89:0/868932091 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:35.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.711+0000 7f6ba5600700 1 -- 172.21.15.89:0/868932091 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400184656 0 0) 0x7f6ba80ceda0 con 0x7f6ba8161590 2024-04-02T07:55:35.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.711+0000 7f6ba6000700 1 -- 172.21.15.89:0/868932091 >> v1:172.21.15.113:6789/0 conn(0x7f6ba8160f10 legacy=0x7f6ba80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.711+0000 7f6ba6000700 1 -- 172.21.15.89:0/868932091 >> v1:172.21.15.89:6789/0 conn(0x7f6ba8161590 legacy=0x7f6ba80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.711+0000 7f6ba6000700 1 -- 172.21.15.89:0/868932091 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba80ce470 con 0x7f6ba80d18b0 2024-04-02T07:55:35.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.711+0000 7f6ba6000700 1 --2- 172.21.15.89:0/868932091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba80d18b0 0x7f6ba80cf230 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009840 tx=0x7f6b94009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa92be2985efc78 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:35.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.712+0000 7f6ba5600700 1 -- 172.21.15.89:0/868932091 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94011830 con 0x7f6ba80d18b0 2024-04-02T07:55:35.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.712+0000 7f6ba5600700 1 -- 172.21.15.89:0/868932091 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b94011990 con 0x7f6ba80d18b0 2024-04-02T07:55:35.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.712+0000 7f6ba5600700 1 -- 172.21.15.89:0/868932091 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94011d20 con 0x7f6ba80d18b0 2024-04-02T07:55:35.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.712+0000 7f6bade00700 1 -- 172.21.15.89:0/868932091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba80d18b0 msgr2=0x7f6ba80cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.712+0000 7f6bade00700 1 --2- 172.21.15.89:0/868932091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba80d18b0 0x7f6ba80cf230 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009840 tx=0x7f6b94009b50 comp rx=0 tx=0).stop 2024-04-02T07:55:35.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.712+0000 7f6bade00700 1 -- 172.21.15.89:0/868932091 shutdown_connections 2024-04-02T07:55:35.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.712+0000 7f6bade00700 1 --2- 172.21.15.89:0/868932091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba80d18b0 0x7f6ba80cf230 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.712+0000 7f6bade00700 1 -- 172.21.15.89:0/868932091 >> 172.21.15.89:0/868932091 conn(0x7f6ba808aff0 msgr2=0x7f6ba80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:35.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.712+0000 7f6bade00700 1 -- 172.21.15.89:0/868932091 shutdown_connections 2024-04-02T07:55:35.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.712+0000 7f6bade00700 1 -- 172.21.15.89:0/868932091 wait complete. 2024-04-02T07:55:35.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.714+0000 7f6bade00700 1 Processor -- start 2024-04-02T07:55:35.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.714+0000 7f6bade00700 1 -- start start 2024-04-02T07:55:35.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.715+0000 7f6bade00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ba80d18b0 0x7f6ba80b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:35.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.715+0000 7f6bade00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba8160f10 0x7f6ba80b6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:35.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.715+0000 7f6bade00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba8161590 0x7f6ba81515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:35.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.715+0000 7f6bade00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba80b9520 con 0x7f6ba8161590 2024-04-02T07:55:35.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.715+0000 7f6bade00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba8157e90 con 0x7f6ba8160f10 2024-04-02T07:55:35.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.715+0000 7f6bade00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba80b8d10 con 0x7f6ba80d18b0 2024-04-02T07:55:35.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.715+0000 7f6ba7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba8161590 0x7f6ba81515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:35.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.715+0000 7f6ba7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba8161590 0x7f6ba81515b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54912/0 (socket says 172.21.15.89:54912) 2024-04-02T07:55:35.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.715+0000 7f6ba7400700 1 -- 172.21.15.89:0/2495608589 learned_addr learned my addr 172.21.15.89:0/2495608589 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:35.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.715+0000 7f6ba6000700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba8160f10 0x7f6ba80b6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.715+0000 7f6ba6a00700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ba80d18b0 0x7f6ba80b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.716+0000 7f6ba7400700 1 -- 172.21.15.89:0/2495608589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ba80d18b0 msgr2=0x7f6ba80b87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.716+0000 7f6ba7400700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ba80d18b0 0x7f6ba80b87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.716+0000 7f6ba7400700 1 -- 172.21.15.89:0/2495608589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba8160f10 msgr2=0x7f6ba80b6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.716+0000 7f6ba7400700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba8160f10 0x7f6ba80b6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.716+0000 7f6ba7400700 1 -- 172.21.15.89:0/2495608589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b940094a0 con 0x7f6ba8161590 2024-04-02T07:55:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.716+0000 7f6ba6a00700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ba80d18b0 0x7f6ba80b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.716+0000 7f6ba7400700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba8161590 0x7f6ba81515b0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f6ba000cb90 tx=0x7f6ba000cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.716+0000 7f6b9f400700 1 -- 172.21.15.89:0/2495608589 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba000aa80 con 0x7f6ba8161590 2024-04-02T07:55:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.716+0000 7f6b9f400700 1 -- 172.21.15.89:0/2495608589 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba000abe0 con 0x7f6ba8161590 2024-04-02T07:55:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.716+0000 7f6bade00700 1 -- 172.21.15.89:0/2495608589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba80b7340 con 0x7f6ba8161590 2024-04-02T07:55:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.716+0000 7f6b9f400700 1 -- 172.21.15.89:0/2495608589 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba00070d0 con 0x7f6ba8161590 2024-04-02T07:55:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.716+0000 7f6bade00700 1 -- 172.21.15.89:0/2495608589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba80b9240 con 0x7f6ba8161590 2024-04-02T07:55:35.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.717+0000 7f6bade00700 1 -- 172.21.15.89:0/2495608589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba8157980 con 0x7f6ba8161590 2024-04-02T07:55:35.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.718+0000 7f6b9f400700 1 -- 172.21.15.89:0/2495608589 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6ba0007230 con 0x7f6ba8161590 2024-04-02T07:55:35.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.719+0000 7f6b9f400700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b8806ade0 0x7f6b8806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:35.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.719+0000 7f6b9f400700 1 -- 172.21.15.89:0/2495608589 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f6ba00893f0 con 0x7f6ba8161590 2024-04-02T07:55:35.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.719+0000 7f6ba6a00700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b8806ade0 0x7f6b8806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:35.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.720+0000 7f6ba6a00700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b8806ade0 0x7f6b8806d2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6b98000f60 tx=0x7f6b98009040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:35.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.724+0000 7f6b9f400700 1 -- 172.21.15.89:0/2495608589 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ba8157980 con 0x7f6ba8161590 2024-04-02T07:55:35.782 INFO:tasks.ceph.mds.c.smithi089.stderr:2024-04-02T07:55:35.781+0000 7f5d5dc00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T07:55:35.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.922+0000 7f6bade00700 1 -- 172.21.15.89:0/2495608589 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6ba80cea30 con 0x7f6b8806ade0 2024-04-02T07:55:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.932+0000 7f6b9f400700 1 -- 172.21.15.89:0/2495608589 <== mgr.8205 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289576 (secure 0 0 0) 0x7f6b98001450 con 0x7f6b8806ade0 2024-04-02T07:55:35.937 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.936+0000 7f6bade00700 1 -- 172.21.15.89:0/2495608589 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b8806ade0 msgr2=0x7f6b8806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.936+0000 7f6bade00700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b8806ade0 0x7f6b8806d2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6b98000f60 tx=0x7f6b98009040 comp rx=0 tx=0).stop 2024-04-02T07:55:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.936+0000 7f6bade00700 1 -- 172.21.15.89:0/2495608589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba8161590 msgr2=0x7f6ba81515b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.936+0000 7f6bade00700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba8161590 0x7f6ba81515b0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f6ba000cb90 tx=0x7f6ba000cea0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.936+0000 7f6bade00700 1 -- 172.21.15.89:0/2495608589 shutdown_connections 2024-04-02T07:55:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.936+0000 7f6bade00700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b8806ade0 0x7f6b8806d2a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.936+0000 7f6bade00700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba8161590 0x7f6ba81515b0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.936+0000 7f6bade00700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ba80d18b0 0x7f6ba80b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.936+0000 7f6bade00700 1 --2- 172.21.15.89:0/2495608589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba8160f10 0x7f6ba80b6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.936+0000 7f6bade00700 1 -- 172.21.15.89:0/2495608589 >> 172.21.15.89:0/2495608589 conn(0x7f6ba808aff0 msgr2=0x7f6ba80cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.936+0000 7f6bade00700 1 -- 172.21.15.89:0/2495608589 shutdown_connections 2024-04-02T07:55:35.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:35.936+0000 7f6bade00700 1 -- 172.21.15.89:0/2495608589 wait complete. 2024-04-02T07:55:35.941 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:55:35.956 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-02T07:55:35.280047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":354628,"kb_used_data":14656,"kb_used_omap":12,"kb_used_meta":339827,"kb_avail":754620092,"statfs":{"total":773094113280,"available":772730974208,"internally_reserved":0,"allocated":15007744,"data_stored":7999988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":347983416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.212286"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.606601+0000","last_change":"2024-04-02T07:55:31.603558+0000","last_active":"2024-04-02T07:55:31.606601+0000","last_peered":"2024-04-02T07:55:31.606601+0000","last_clean":"2024-04-02T07:55:31.606601+0000","last_became_active":"2024-04-02T07:55:31.603272+0000","last_became_peered":"2024-04-02T07:55:31.603272+0000","last_unstale":"2024-04-02T07:55:31.606601+0000","last_undegraded":"2024-04-02T07:55:31.606601+0000","last_fullsized":"2024-04-02T07:55:31.606601+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:07:51.776847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.610246+0000","last_change":"2024-04-02T07:55:31.608040+0000","last_active":"2024-04-02T07:55:31.610246+0000","last_peered":"2024-04-02T07:55:31.610246+0000","last_clean":"2024-04-02T07:55:31.610246+0000","last_became_active":"2024-04-02T07:55:31.607582+0000","last_became_peered":"2024-04-02T07:55:31.607582+0000","last_unstale":"2024-04-02T07:55:31.610246+0000","last_undegraded":"2024-04-02T07:55:31.610246+0000","last_fullsized":"2024-04-02T07:55:31.610246+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:48:15.120406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.602607+0000","last_change":"2024-04-02T07:55:31.597008+0000","last_active":"2024-04-02T07:55:31.602607+0000","last_peered":"2024-04-02T07:55:31.602607+0000","last_clean":"2024-04-02T07:55:31.602607+0000","last_became_active":"2024-04-02T07:55:31.595009+0000","last_became_peered":"2024-04-02T07:55:31.595009+0000","last_unstale":"2024-04-02T07:55:31.602607+0000","last_undegraded":"2024-04-02T07:55:31.602607+0000","last_fullsized":"2024-04-02T07:55:31.602607+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:05:45.742380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.607588+0000","last_change":"2024-04-02T07:55:31.596385+0000","last_active":"2024-04-02T07:55:31.607588+0000","last_peered":"2024-04-02T07:55:31.607588+0000","last_clean":"2024-04-02T07:55:31.607588+0000","last_became_active":"2024-04-02T07:55:31.596105+0000","last_became_peered":"2024-04-02T07:55:31.596105+0000","last_unstale":"2024-04-02T07:55:31.607588+0000","last_undegraded":"2024-04-02T07:55:31.607588+0000","last_fullsized":"2024-04-02T07:55:31.607588+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:35:52.948729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.602560+0000","last_change":"2024-04-02T07:55:31.593771+0000","last_active":"2024-04-02T07:55:31.602560+0000","last_peered":"2024-04-02T07:55:31.602560+0000","last_clean":"2024-04-02T07:55:31.602560+0000","last_became_active":"2024-04-02T07:55:31.593310+0000","last_became_peered":"2024-04-02T07:55:31.593310+0000","last_unstale":"2024-04-02T07:55:31.602560+0000","last_undegraded":"2024-04-02T07:55:31.602560+0000","last_fullsized":"2024-04-02T07:55:31.602560+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:15.292010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.598295+0000","last_change":"2024-04-02T07:55:31.594948+0000","last_active":"2024-04-02T07:55:31.598295+0000","last_peered":"2024-04-02T07:55:31.598295+0000","last_clean":"2024-04-02T07:55:31.598295+0000","last_became_active":"2024-04-02T07:55:31.594685+0000","last_became_peered":"2024-04-02T07:55:31.594685+0000","last_unstale":"2024-04-02T07:55:31.598295+0000","last_undegraded":"2024-04-02T07:55:31.598295+0000","last_fullsized":"2024-04-02T07:55:31.598295+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:44:37.486456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605147+0000","last_change":"2024-04-02T07:55:31.600345+0000","last_active":"2024-04-02T07:55:31.605147+0000","last_peered":"2024-04-02T07:55:31.605147+0000","last_clean":"2024-04-02T07:55:31.605147+0000","last_became_active":"2024-04-02T07:55:31.600089+0000","last_became_peered":"2024-04-02T07:55:31.600089+0000","last_unstale":"2024-04-02T07:55:31.605147+0000","last_undegraded":"2024-04-02T07:55:31.605147+0000","last_fullsized":"2024-04-02T07:55:31.605147+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:18.774950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.595313+0000","last_change":"2024-04-02T07:55:31.592269+0000","last_active":"2024-04-02T07:55:31.595313+0000","last_peered":"2024-04-02T07:55:31.595313+0000","last_clean":"2024-04-02T07:55:31.595313+0000","last_became_active":"2024-04-02T07:55:31.591995+0000","last_became_peered":"2024-04-02T07:55:31.591995+0000","last_unstale":"2024-04-02T07:55:31.595313+0000","last_undegraded":"2024-04-02T07:55:31.595313+0000","last_fullsized":"2024-04-02T07:55:31.595313+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:07:42.638732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.610808+0000","last_change":"2024-04-02T07:55:31.606687+0000","last_active":"2024-04-02T07:55:31.610808+0000","last_peered":"2024-04-02T07:55:31.610808+0000","last_clean":"2024-04-02T07:55:31.610808+0000","last_became_active":"2024-04-02T07:55:31.605673+0000","last_became_peered":"2024-04-02T07:55:31.605673+0000","last_unstale":"2024-04-02T07:55:31.610808+0000","last_undegraded":"2024-04-02T07:55:31.610808+0000","last_fullsized":"2024-04-02T07:55:31.610808+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:28:07.810972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.600456+0000","last_change":"2024-04-02T07:55:31.590469+0000","last_active":"2024-04-02T07:55:31.600456+0000","last_peered":"2024-04-02T07:55:31.600456+0000","last_clean":"2024-04-02T07:55:31.600456+0000","last_became_active":"2024-04-02T07:55:31.589916+0000","last_became_peered":"2024-04-02T07:55:31.589916+0000","last_unstale":"2024-04-02T07:55:31.600456+0000","last_undegraded":"2024-04-02T07:55:31.600456+0000","last_fullsized":"2024-04-02T07:55:31.600456+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:53.426593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605776+0000","last_change":"2024-04-02T07:55:31.604573+0000","last_active":"2024-04-02T07:55:31.605776+0000","last_peered":"2024-04-02T07:55:31.605776+0000","last_clean":"2024-04-02T07:55:31.605776+0000","last_became_active":"2024-04-02T07:55:31.604322+0000","last_became_peered":"2024-04-02T07:55:31.604322+0000","last_unstale":"2024-04-02T07:55:31.605776+0000","last_undegraded":"2024-04-02T07:55:31.605776+0000","last_fullsized":"2024-04-02T07:55:31.605776+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:02:40.059301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.600915+0000","last_change":"2024-04-02T07:55:31.591751+0000","last_active":"2024-04-02T07:55:31.600915+0000","last_peered":"2024-04-02T07:55:31.600915+0000","last_clean":"2024-04-02T07:55:31.600915+0000","last_became_active":"2024-04-02T07:55:31.591445+0000","last_became_peered":"2024-04-02T07:55:31.591445+0000","last_unstale":"2024-04-02T07:55:31.600915+0000","last_undegraded":"2024-04-02T07:55:31.600915+0000","last_fullsized":"2024-04-02T07:55:31.600915+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:50:15.964841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.599205+0000","last_change":"2024-04-02T07:55:31.594243+0000","last_active":"2024-04-02T07:55:31.599205+0000","last_peered":"2024-04-02T07:55:31.599205+0000","last_clean":"2024-04-02T07:55:31.599205+0000","last_became_active":"2024-04-02T07:55:31.593477+0000","last_became_peered":"2024-04-02T07:55:31.593477+0000","last_unstale":"2024-04-02T07:55:31.599205+0000","last_undegraded":"2024-04-02T07:55:31.599205+0000","last_fullsized":"2024-04-02T07:55:31.599205+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:15.292010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.604651+0000","last_change":"2024-04-02T07:55:31.600044+0000","last_active":"2024-04-02T07:55:31.604651+0000","last_peered":"2024-04-02T07:55:31.604651+0000","last_clean":"2024-04-02T07:55:31.604651+0000","last_became_active":"2024-04-02T07:55:31.599743+0000","last_became_peered":"2024-04-02T07:55:31.599743+0000","last_unstale":"2024-04-02T07:55:31.604651+0000","last_undegraded":"2024-04-02T07:55:31.604651+0000","last_fullsized":"2024-04-02T07:55:31.604651+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:31:48.360153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.592434+0000","last_change":"2024-04-02T07:55:31.587033+0000","last_active":"2024-04-02T07:55:31.592434+0000","last_peered":"2024-04-02T07:55:31.592434+0000","last_clean":"2024-04-02T07:55:31.592434+0000","last_became_active":"2024-04-02T07:55:31.586772+0000","last_became_peered":"2024-04-02T07:55:31.586772+0000","last_unstale":"2024-04-02T07:55:31.592434+0000","last_undegraded":"2024-04-02T07:55:31.592434+0000","last_fullsized":"2024-04-02T07:55:31.592434+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:55.262988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.599484+0000","last_change":"2024-04-02T07:55:31.591692+0000","last_active":"2024-04-02T07:55:31.599484+0000","last_peered":"2024-04-02T07:55:31.599484+0000","last_clean":"2024-04-02T07:55:31.599484+0000","last_became_active":"2024-04-02T07:55:31.591415+0000","last_became_peered":"2024-04-02T07:55:31.591415+0000","last_unstale":"2024-04-02T07:55:31.599484+0000","last_undegraded":"2024-04-02T07:55:31.599484+0000","last_fullsized":"2024-04-02T07:55:31.599484+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:48:20.542068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.603320+0000","last_change":"2024-04-02T07:55:31.601169+0000","last_active":"2024-04-02T07:55:31.603320+0000","last_peered":"2024-04-02T07:55:31.603320+0000","last_clean":"2024-04-02T07:55:31.603320+0000","last_became_active":"2024-04-02T07:55:31.600930+0000","last_became_peered":"2024-04-02T07:55:31.600930+0000","last_unstale":"2024-04-02T07:55:31.603320+0000","last_undegraded":"2024-04-02T07:55:31.603320+0000","last_fullsized":"2024-04-02T07:55:31.603320+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:13:07.562470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.594758+0000","last_change":"2024-04-02T07:55:31.591154+0000","last_active":"2024-04-02T07:55:31.594758+0000","last_peered":"2024-04-02T07:55:31.594758+0000","last_clean":"2024-04-02T07:55:31.594758+0000","last_became_active":"2024-04-02T07:55:31.589651+0000","last_became_peered":"2024-04-02T07:55:31.589651+0000","last_unstale":"2024-04-02T07:55:31.594758+0000","last_undegraded":"2024-04-02T07:55:31.594758+0000","last_fullsized":"2024-04-02T07:55:31.594758+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:07:49.631127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.602645+0000","last_change":"2024-04-02T07:55:31.597181+0000","last_active":"2024-04-02T07:55:31.602645+0000","last_peered":"2024-04-02T07:55:31.602645+0000","last_clean":"2024-04-02T07:55:31.602645+0000","last_became_active":"2024-04-02T07:55:31.596854+0000","last_became_peered":"2024-04-02T07:55:31.596854+0000","last_unstale":"2024-04-02T07:55:31.602645+0000","last_undegraded":"2024-04-02T07:55:31.602645+0000","last_fullsized":"2024-04-02T07:55:31.602645+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:32:06.630259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.603026+0000","last_change":"2024-04-02T07:55:31.600923+0000","last_active":"2024-04-02T07:55:31.603026+0000","last_peered":"2024-04-02T07:55:31.603026+0000","last_clean":"2024-04-02T07:55:31.603026+0000","last_became_active":"2024-04-02T07:55:31.600658+0000","last_became_peered":"2024-04-02T07:55:31.600658+0000","last_unstale":"2024-04-02T07:55:31.603026+0000","last_undegraded":"2024-04-02T07:55:31.603026+0000","last_fullsized":"2024-04-02T07:55:31.603026+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:15.292010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.602717+0000","last_change":"2024-04-02T07:55:31.598595+0000","last_active":"2024-04-02T07:55:31.602717+0000","last_peered":"2024-04-02T07:55:31.602717+0000","last_clean":"2024-04-02T07:55:31.602717+0000","last_became_active":"2024-04-02T07:55:31.598345+0000","last_became_peered":"2024-04-02T07:55:31.598345+0000","last_unstale":"2024-04-02T07:55:31.602717+0000","last_undegraded":"2024-04-02T07:55:31.602717+0000","last_fullsized":"2024-04-02T07:55:31.602717+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:07:05.102587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.591831+0000","last_change":"2024-04-02T07:55:31.587979+0000","last_active":"2024-04-02T07:55:31.591831+0000","last_peered":"2024-04-02T07:55:31.591831+0000","last_clean":"2024-04-02T07:55:31.591831+0000","last_became_active":"2024-04-02T07:55:31.587691+0000","last_became_peered":"2024-04-02T07:55:31.587691+0000","last_unstale":"2024-04-02T07:55:31.591831+0000","last_undegraded":"2024-04-02T07:55:31.591831+0000","last_fullsized":"2024-04-02T07:55:31.591831+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:31:48.360153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605114+0000","last_change":"2024-04-02T07:55:31.600353+0000","last_active":"2024-04-02T07:55:31.605114+0000","last_peered":"2024-04-02T07:55:31.605114+0000","last_clean":"2024-04-02T07:55:31.605114+0000","last_became_active":"2024-04-02T07:55:31.600116+0000","last_became_peered":"2024-04-02T07:55:31.600116+0000","last_unstale":"2024-04-02T07:55:31.605114+0000","last_undegraded":"2024-04-02T07:55:31.605114+0000","last_fullsized":"2024-04-02T07:55:31.605114+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:08:01.923416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.610766+0000","last_change":"2024-04-02T07:55:31.605709+0000","last_active":"2024-04-02T07:55:31.610766+0000","last_peered":"2024-04-02T07:55:31.610766+0000","last_clean":"2024-04-02T07:55:31.610766+0000","last_became_active":"2024-04-02T07:55:31.603901+0000","last_became_peered":"2024-04-02T07:55:31.603901+0000","last_unstale":"2024-04-02T07:55:31.610766+0000","last_undegraded":"2024-04-02T07:55:31.610766+0000","last_fullsized":"2024-04-02T07:55:31.610766+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:26:40.691653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.602891+0000","last_change":"2024-04-02T07:55:31.597522+0000","last_active":"2024-04-02T07:55:31.602891+0000","last_peered":"2024-04-02T07:55:31.602891+0000","last_clean":"2024-04-02T07:55:31.602891+0000","last_became_active":"2024-04-02T07:55:31.597286+0000","last_became_peered":"2024-04-02T07:55:31.597286+0000","last_unstale":"2024-04-02T07:55:31.602891+0000","last_undegraded":"2024-04-02T07:55:31.602891+0000","last_fullsized":"2024-04-02T07:55:31.602891+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:00:44.301410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.593283+0000","last_change":"2024-04-02T07:55:31.591600+0000","last_active":"2024-04-02T07:55:31.593283+0000","last_peered":"2024-04-02T07:55:31.593283+0000","last_clean":"2024-04-02T07:55:31.593283+0000","last_became_active":"2024-04-02T07:55:31.591185+0000","last_became_peered":"2024-04-02T07:55:31.591185+0000","last_unstale":"2024-04-02T07:55:31.593283+0000","last_undegraded":"2024-04-02T07:55:31.593283+0000","last_fullsized":"2024-04-02T07:55:31.593283+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:16:03.608355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.609043+0000","last_change":"2024-04-02T07:55:31.602205+0000","last_active":"2024-04-02T07:55:31.609043+0000","last_peered":"2024-04-02T07:55:31.609043+0000","last_clean":"2024-04-02T07:55:31.609043+0000","last_became_active":"2024-04-02T07:55:31.599108+0000","last_became_peered":"2024-04-02T07:55:31.599108+0000","last_unstale":"2024-04-02T07:55:31.609043+0000","last_undegraded":"2024-04-02T07:55:31.609043+0000","last_fullsized":"2024-04-02T07:55:31.609043+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:36:31.822093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.598237+0000","last_change":"2024-04-02T07:55:31.597220+0000","last_active":"2024-04-02T07:55:31.598237+0000","last_peered":"2024-04-02T07:55:31.598237+0000","last_clean":"2024-04-02T07:55:31.598237+0000","last_became_active":"2024-04-02T07:55:31.596939+0000","last_became_peered":"2024-04-02T07:55:31.596939+0000","last_unstale":"2024-04-02T07:55:31.598237+0000","last_undegraded":"2024-04-02T07:55:31.598237+0000","last_fullsized":"2024-04-02T07:55:31.598237+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:49:10.266042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.604207+0000","last_change":"2024-04-02T07:55:31.599075+0000","last_active":"2024-04-02T07:55:31.604207+0000","last_peered":"2024-04-02T07:55:31.604207+0000","last_clean":"2024-04-02T07:55:31.604207+0000","last_became_active":"2024-04-02T07:55:31.597668+0000","last_became_peered":"2024-04-02T07:55:31.597668+0000","last_unstale":"2024-04-02T07:55:31.604207+0000","last_undegraded":"2024-04-02T07:55:31.604207+0000","last_fullsized":"2024-04-02T07:55:31.604207+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:57:26.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.593415+0000","last_change":"2024-04-02T07:55:31.589050+0000","last_active":"2024-04-02T07:55:31.593415+0000","last_peered":"2024-04-02T07:55:31.593415+0000","last_clean":"2024-04-02T07:55:31.593415+0000","last_became_active":"2024-04-02T07:55:31.588794+0000","last_became_peered":"2024-04-02T07:55:31.588794+0000","last_unstale":"2024-04-02T07:55:31.593415+0000","last_undegraded":"2024-04-02T07:55:31.593415+0000","last_fullsized":"2024-04-02T07:55:31.593415+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:45:05.161189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.606230+0000","last_change":"2024-04-02T07:55:31.602104+0000","last_active":"2024-04-02T07:55:31.606230+0000","last_peered":"2024-04-02T07:55:31.606230+0000","last_clean":"2024-04-02T07:55:31.606230+0000","last_became_active":"2024-04-02T07:55:31.601851+0000","last_became_peered":"2024-04-02T07:55:31.601851+0000","last_unstale":"2024-04-02T07:55:31.606230+0000","last_undegraded":"2024-04-02T07:55:31.606230+0000","last_fullsized":"2024-04-02T07:55:31.606230+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:26:40.691653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.606167+0000","last_change":"2024-04-02T07:55:31.601900+0000","last_active":"2024-04-02T07:55:31.606167+0000","last_peered":"2024-04-02T07:55:31.606167+0000","last_clean":"2024-04-02T07:55:31.606167+0000","last_became_active":"2024-04-02T07:55:31.601649+0000","last_became_peered":"2024-04-02T07:55:31.601649+0000","last_unstale":"2024-04-02T07:55:31.606167+0000","last_undegraded":"2024-04-02T07:55:31.606167+0000","last_fullsized":"2024-04-02T07:55:31.606167+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:48:15.120406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.601970+0000","last_change":"2024-04-02T07:55:31.599893+0000","last_active":"2024-04-02T07:55:31.601970+0000","last_peered":"2024-04-02T07:55:31.601970+0000","last_clean":"2024-04-02T07:55:31.601970+0000","last_became_active":"2024-04-02T07:55:31.599601+0000","last_became_peered":"2024-04-02T07:55:31.599601+0000","last_unstale":"2024-04-02T07:55:31.601970+0000","last_undegraded":"2024-04-02T07:55:31.601970+0000","last_fullsized":"2024-04-02T07:55:31.601970+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:56.353145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"139'4","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.607883+0000","last_change":"2024-04-02T07:55:31.094257+0000","last_active":"2024-04-02T07:55:31.607883+0000","last_peered":"2024-04-02T07:55:31.607883+0000","last_clean":"2024-04-02T07:55:31.607883+0000","last_became_active":"2024-04-02T07:55:31.093843+0000","last_became_peered":"2024-04-02T07:55:31.093843+0000","last_unstale":"2024-04-02T07:55:31.607883+0000","last_undegraded":"2024-04-02T07:55:31.607883+0000","last_fullsized":"2024-04-02T07:55:31.607883+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:07:50.769069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605899+0000","last_change":"2024-04-02T07:55:31.602503+0000","last_active":"2024-04-02T07:55:31.605899+0000","last_peered":"2024-04-02T07:55:31.605899+0000","last_clean":"2024-04-02T07:55:31.605899+0000","last_became_active":"2024-04-02T07:55:31.602265+0000","last_became_peered":"2024-04-02T07:55:31.602265+0000","last_unstale":"2024-04-02T07:55:31.605899+0000","last_undegraded":"2024-04-02T07:55:31.605899+0000","last_fullsized":"2024-04-02T07:55:31.605899+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:13.861818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1e","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.603571+0000","last_change":"2024-04-02T07:55:31.088483+0000","last_active":"2024-04-02T07:55:31.603571+0000","last_peered":"2024-04-02T07:55:31.603571+0000","last_clean":"2024-04-02T07:55:31.603571+0000","last_became_active":"2024-04-02T07:55:31.088242+0000","last_became_peered":"2024-04-02T07:55:31.088242+0000","last_unstale":"2024-04-02T07:55:31.603571+0000","last_undegraded":"2024-04-02T07:55:31.603571+0000","last_fullsized":"2024-04-02T07:55:31.603571+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:47:16.589118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.592958+0000","last_change":"2024-04-02T07:55:31.588766+0000","last_active":"2024-04-02T07:55:31.592958+0000","last_peered":"2024-04-02T07:55:31.592958+0000","last_clean":"2024-04-02T07:55:31.592958+0000","last_became_active":"2024-04-02T07:55:31.588384+0000","last_became_peered":"2024-04-02T07:55:31.588384+0000","last_unstale":"2024-04-02T07:55:31.592958+0000","last_undegraded":"2024-04-02T07:55:31.592958+0000","last_fullsized":"2024-04-02T07:55:31.592958+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:57:26.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1d","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.608654+0000","last_change":"2024-04-02T07:55:31.094859+0000","last_active":"2024-04-02T07:55:31.608654+0000","last_peered":"2024-04-02T07:55:31.608654+0000","last_clean":"2024-04-02T07:55:31.608654+0000","last_became_active":"2024-04-02T07:55:31.094585+0000","last_became_peered":"2024-04-02T07:55:31.094585+0000","last_unstale":"2024-04-02T07:55:31.608654+0000","last_undegraded":"2024-04-02T07:55:31.608654+0000","last_fullsized":"2024-04-02T07:55:31.608654+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:13:06.554691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.602629+0000","last_change":"2024-04-02T07:55:31.597100+0000","last_active":"2024-04-02T07:55:31.602629+0000","last_peered":"2024-04-02T07:55:31.602629+0000","last_clean":"2024-04-02T07:55:31.602629+0000","last_became_active":"2024-04-02T07:55:31.595358+0000","last_became_peered":"2024-04-02T07:55:31.595358+0000","last_unstale":"2024-04-02T07:55:31.602629+0000","last_undegraded":"2024-04-02T07:55:31.602629+0000","last_fullsized":"2024-04-02T07:55:31.602629+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:50:15.964841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.609130+0000","last_change":"2024-04-02T07:55:31.093302+0000","last_active":"2024-04-02T07:55:31.609130+0000","last_peered":"2024-04-02T07:55:31.609130+0000","last_clean":"2024-04-02T07:55:31.609130+0000","last_became_active":"2024-04-02T07:55:31.091025+0000","last_became_peered":"2024-04-02T07:55:31.091025+0000","last_unstale":"2024-04-02T07:55:31.609130+0000","last_undegraded":"2024-04-02T07:55:31.609130+0000","last_fullsized":"2024-04-02T07:55:31.609130+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:46.618508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.610727+0000","last_change":"2024-04-02T07:55:31.604592+0000","last_active":"2024-04-02T07:55:31.610727+0000","last_peered":"2024-04-02T07:55:31.610727+0000","last_clean":"2024-04-02T07:55:31.610727+0000","last_became_active":"2024-04-02T07:55:31.603197+0000","last_became_peered":"2024-04-02T07:55:31.603197+0000","last_unstale":"2024-04-02T07:55:31.610727+0000","last_undegraded":"2024-04-02T07:55:31.610727+0000","last_fullsized":"2024-04-02T07:55:31.610727+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:32:23.897820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1b","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.603082+0000","last_change":"2024-04-02T07:55:31.084659+0000","last_active":"2024-04-02T07:55:31.603082+0000","last_peered":"2024-04-02T07:55:31.603082+0000","last_clean":"2024-04-02T07:55:31.603082+0000","last_became_active":"2024-04-02T07:55:31.083757+0000","last_became_peered":"2024-04-02T07:55:31.083757+0000","last_unstale":"2024-04-02T07:55:31.603082+0000","last_undegraded":"2024-04-02T07:55:31.603082+0000","last_fullsized":"2024-04-02T07:55:31.603082+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:47:16.589118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.603694+0000","last_change":"2024-04-02T07:55:31.601478+0000","last_active":"2024-04-02T07:55:31.603694+0000","last_peered":"2024-04-02T07:55:31.603694+0000","last_clean":"2024-04-02T07:55:31.603694+0000","last_became_active":"2024-04-02T07:55:31.601237+0000","last_became_peered":"2024-04-02T07:55:31.601237+0000","last_unstale":"2024-04-02T07:55:31.603694+0000","last_undegraded":"2024-04-02T07:55:31.603694+0000","last_fullsized":"2024-04-02T07:55:31.603694+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:52:38.844963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.606307+0000","last_change":"2024-04-02T07:55:31.094674+0000","last_active":"2024-04-02T07:55:31.606307+0000","last_peered":"2024-04-02T07:55:31.606307+0000","last_clean":"2024-04-02T07:55:31.606307+0000","last_became_active":"2024-04-02T07:55:31.094215+0000","last_became_peered":"2024-04-02T07:55:31.094215+0000","last_unstale":"2024-04-02T07:55:31.606307+0000","last_undegraded":"2024-04-02T07:55:31.606307+0000","last_fullsized":"2024-04-02T07:55:31.606307+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:48:14.112628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.607633+0000","last_change":"2024-04-02T07:55:31.604907+0000","last_active":"2024-04-02T07:55:31.607633+0000","last_peered":"2024-04-02T07:55:31.607633+0000","last_clean":"2024-04-02T07:55:31.607633+0000","last_became_active":"2024-04-02T07:55:31.602161+0000","last_became_peered":"2024-04-02T07:55:31.602161+0000","last_unstale":"2024-04-02T07:55:31.607633+0000","last_undegraded":"2024-04-02T07:55:31.607633+0000","last_fullsized":"2024-04-02T07:55:31.607633+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:05:45.742380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.586906+0000","last_change":"2024-04-02T07:55:31.090631+0000","last_active":"2024-04-02T07:55:31.586906+0000","last_peered":"2024-04-02T07:55:31.586906+0000","last_clean":"2024-04-02T07:55:31.586906+0000","last_became_active":"2024-04-02T07:55:31.089242+0000","last_became_peered":"2024-04-02T07:55:31.089242+0000","last_unstale":"2024-04-02T07:55:31.586906+0000","last_undegraded":"2024-04-02T07:55:31.586906+0000","last_fullsized":"2024-04-02T07:55:31.586906+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:45:35.627867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.594814+0000","last_change":"2024-04-02T07:55:31.591684+0000","last_active":"2024-04-02T07:55:31.594814+0000","last_peered":"2024-04-02T07:55:31.594814+0000","last_clean":"2024-04-02T07:55:31.594814+0000","last_became_active":"2024-04-02T07:55:31.591419+0000","last_became_peered":"2024-04-02T07:55:31.591419+0000","last_unstale":"2024-04-02T07:55:31.594814+0000","last_undegraded":"2024-04-02T07:55:31.594814+0000","last_fullsized":"2024-04-02T07:55:31.594814+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:56:56.050365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.582889+0000","last_change":"2024-04-02T07:55:31.096341+0000","last_active":"2024-04-02T07:55:31.582889+0000","last_peered":"2024-04-02T07:55:31.582889+0000","last_clean":"2024-04-02T07:55:31.582889+0000","last_became_active":"2024-04-02T07:55:31.096037+0000","last_became_peered":"2024-04-02T07:55:31.096037+0000","last_unstale":"2024-04-02T07:55:31.582889+0000","last_undegraded":"2024-04-02T07:55:31.582889+0000","last_fullsized":"2024-04-02T07:55:31.582889+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:08:36.938918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.596124+0000","last_change":"2024-04-02T07:55:31.591858+0000","last_active":"2024-04-02T07:55:31.596124+0000","last_peered":"2024-04-02T07:55:31.596124+0000","last_clean":"2024-04-02T07:55:31.596124+0000","last_became_active":"2024-04-02T07:55:31.590772+0000","last_became_peered":"2024-04-02T07:55:31.590772+0000","last_unstale":"2024-04-02T07:55:31.596124+0000","last_undegraded":"2024-04-02T07:55:31.596124+0000","last_fullsized":"2024-04-02T07:55:31.596124+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:48:20.542068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.598384+0000","last_change":"2024-04-02T07:55:31.095060+0000","last_active":"2024-04-02T07:55:31.598384+0000","last_peered":"2024-04-02T07:55:31.598384+0000","last_clean":"2024-04-02T07:55:31.598384+0000","last_became_active":"2024-04-02T07:55:31.094769+0000","last_became_peered":"2024-04-02T07:55:31.094769+0000","last_unstale":"2024-04-02T07:55:31.598384+0000","last_undegraded":"2024-04-02T07:55:31.598384+0000","last_fullsized":"2024-04-02T07:55:31.598384+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:11:14.811238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.596093+0000","last_change":"2024-04-02T07:55:31.592762+0000","last_active":"2024-04-02T07:55:31.596093+0000","last_peered":"2024-04-02T07:55:31.596093+0000","last_clean":"2024-04-02T07:55:31.596093+0000","last_became_active":"2024-04-02T07:55:31.592275+0000","last_became_peered":"2024-04-02T07:55:31.592275+0000","last_unstale":"2024-04-02T07:55:31.596093+0000","last_undegraded":"2024-04-02T07:55:31.596093+0000","last_fullsized":"2024-04-02T07:55:31.596093+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:36:21.155941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.583258+0000","last_change":"2024-04-02T07:55:31.094042+0000","last_active":"2024-04-02T07:55:31.583258+0000","last_peered":"2024-04-02T07:55:31.583258+0000","last_clean":"2024-04-02T07:55:31.583258+0000","last_became_active":"2024-04-02T07:55:31.093747+0000","last_became_peered":"2024-04-02T07:55:31.093747+0000","last_unstale":"2024-04-02T07:55:31.583258+0000","last_undegraded":"2024-04-02T07:55:31.583258+0000","last_fullsized":"2024-04-02T07:55:31.583258+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:40.125366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.585937+0000","last_change":"2024-04-02T07:55:31.584753+0000","last_active":"2024-04-02T07:55:31.585937+0000","last_peered":"2024-04-02T07:55:31.585937+0000","last_clean":"2024-04-02T07:55:31.585937+0000","last_became_active":"2024-04-02T07:55:31.584418+0000","last_became_peered":"2024-04-02T07:55:31.584418+0000","last_unstale":"2024-04-02T07:55:31.585937+0000","last_undegraded":"2024-04-02T07:55:31.585937+0000","last_fullsized":"2024-04-02T07:55:31.585937+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:21:18.710733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.15","version":"139'3","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.611044+0000","last_change":"2024-04-02T07:55:31.098639+0000","last_active":"2024-04-02T07:55:31.611044+0000","last_peered":"2024-04-02T07:55:31.611044+0000","last_clean":"2024-04-02T07:55:31.611044+0000","last_became_active":"2024-04-02T07:55:31.098308+0000","last_became_peered":"2024-04-02T07:55:31.098308+0000","last_unstale":"2024-04-02T07:55:31.611044+0000","last_undegraded":"2024-04-02T07:55:31.611044+0000","last_fullsized":"2024-04-02T07:55:31.611044+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:22:28.833417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.602484+0000","last_change":"2024-04-02T07:55:31.596850+0000","last_active":"2024-04-02T07:55:31.602484+0000","last_peered":"2024-04-02T07:55:31.602484+0000","last_clean":"2024-04-02T07:55:31.602484+0000","last_became_active":"2024-04-02T07:55:31.596585+0000","last_became_peered":"2024-04-02T07:55:31.596585+0000","last_unstale":"2024-04-02T07:55:31.602484+0000","last_undegraded":"2024-04-02T07:55:31.602484+0000","last_fullsized":"2024-04-02T07:55:31.602484+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:31:25.345799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"139'6","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.613370+0000","last_change":"2024-04-02T07:55:31.093614+0000","last_active":"2024-04-02T07:55:31.613370+0000","last_peered":"2024-04-02T07:55:31.613370+0000","last_clean":"2024-04-02T07:55:31.613370+0000","last_became_active":"2024-04-02T07:55:31.093295+0000","last_became_peered":"2024-04-02T07:55:31.093295+0000","last_unstale":"2024-04-02T07:55:31.613370+0000","last_undegraded":"2024-04-02T07:55:31.613370+0000","last_fullsized":"2024-04-02T07:55:31.613370+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:02:06.434801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.600040+0000","last_change":"2024-04-02T07:55:31.595431+0000","last_active":"2024-04-02T07:55:31.600040+0000","last_peered":"2024-04-02T07:55:31.600040+0000","last_clean":"2024-04-02T07:55:31.600040+0000","last_became_active":"2024-04-02T07:55:31.594193+0000","last_became_peered":"2024-04-02T07:55:31.594193+0000","last_unstale":"2024-04-02T07:55:31.600040+0000","last_undegraded":"2024-04-02T07:55:31.600040+0000","last_fullsized":"2024-04-02T07:55:31.600040+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:36:21.155941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.610061+0000","last_change":"2024-04-02T07:55:31.094912+0000","last_active":"2024-04-02T07:55:31.610061+0000","last_peered":"2024-04-02T07:55:31.610061+0000","last_clean":"2024-04-02T07:55:31.610061+0000","last_became_active":"2024-04-02T07:55:31.094666+0000","last_became_peered":"2024-04-02T07:55:31.094666+0000","last_unstale":"2024-04-02T07:55:31.610061+0000","last_undegraded":"2024-04-02T07:55:31.610061+0000","last_fullsized":"2024-04-02T07:55:31.610061+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:58:01.791478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.609104+0000","last_change":"2024-04-02T07:55:31.600576+0000","last_active":"2024-04-02T07:55:31.609104+0000","last_peered":"2024-04-02T07:55:31.609104+0000","last_clean":"2024-04-02T07:55:31.609104+0000","last_became_active":"2024-04-02T07:55:31.600239+0000","last_became_peered":"2024-04-02T07:55:31.600239+0000","last_unstale":"2024-04-02T07:55:31.609104+0000","last_undegraded":"2024-04-02T07:55:31.609104+0000","last_fullsized":"2024-04-02T07:55:31.609104+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:10:09.268501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.582982+0000","last_change":"2024-04-02T07:55:31.085739+0000","last_active":"2024-04-02T07:55:31.582982+0000","last_peered":"2024-04-02T07:55:31.582982+0000","last_clean":"2024-04-02T07:55:31.582982+0000","last_became_active":"2024-04-02T07:55:31.084489+0000","last_became_peered":"2024-04-02T07:55:31.084489+0000","last_unstale":"2024-04-02T07:55:31.582982+0000","last_undegraded":"2024-04-02T07:55:31.582982+0000","last_fullsized":"2024-04-02T07:55:31.582982+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:58:33.545600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.606691+0000","last_change":"2024-04-02T07:55:31.594685+0000","last_active":"2024-04-02T07:55:31.606691+0000","last_peered":"2024-04-02T07:55:31.606691+0000","last_clean":"2024-04-02T07:55:31.606691+0000","last_became_active":"2024-04-02T07:55:31.594435+0000","last_became_peered":"2024-04-02T07:55:31.594435+0000","last_unstale":"2024-04-02T07:55:31.606691+0000","last_undegraded":"2024-04-02T07:55:31.606691+0000","last_fullsized":"2024-04-02T07:55:31.606691+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:54:15.998051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"139'8","reported_seq":18,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.608180+0000","last_change":"2024-04-02T07:55:31.096347+0000","last_active":"2024-04-02T07:55:31.608180+0000","last_peered":"2024-04-02T07:55:31.608180+0000","last_clean":"2024-04-02T07:55:31.608180+0000","last_became_active":"2024-04-02T07:55:31.095739+0000","last_became_peered":"2024-04-02T07:55:31.095739+0000","last_unstale":"2024-04-02T07:55:31.608180+0000","last_undegraded":"2024-04-02T07:55:31.608180+0000","last_fullsized":"2024-04-02T07:55:31.608180+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:48:19.534289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.597735+0000","last_change":"2024-04-02T07:55:31.589081+0000","last_active":"2024-04-02T07:55:31.597735+0000","last_peered":"2024-04-02T07:55:31.597735+0000","last_clean":"2024-04-02T07:55:31.597735+0000","last_became_active":"2024-04-02T07:55:31.588764+0000","last_became_peered":"2024-04-02T07:55:31.588764+0000","last_unstale":"2024-04-02T07:55:31.597735+0000","last_undegraded":"2024-04-02T07:55:31.597735+0000","last_fullsized":"2024-04-02T07:55:31.597735+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:44:06.419071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.f","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.606341+0000","last_change":"2024-04-02T07:55:31.087654+0000","last_active":"2024-04-02T07:55:31.606341+0000","last_peered":"2024-04-02T07:55:31.606341+0000","last_clean":"2024-04-02T07:55:31.606341+0000","last_became_active":"2024-04-02T07:55:31.087292+0000","last_became_peered":"2024-04-02T07:55:31.087292+0000","last_unstale":"2024-04-02T07:55:31.606341+0000","last_undegraded":"2024-04-02T07:55:31.606341+0000","last_fullsized":"2024-04-02T07:55:31.606341+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:54:14.990272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.600632+0000","last_change":"2024-04-02T07:55:31.596098+0000","last_active":"2024-04-02T07:55:31.600632+0000","last_peered":"2024-04-02T07:55:31.600632+0000","last_clean":"2024-04-02T07:55:31.600632+0000","last_became_active":"2024-04-02T07:55:31.594047+0000","last_became_peered":"2024-04-02T07:55:31.594047+0000","last_unstale":"2024-04-02T07:55:31.600632+0000","last_undegraded":"2024-04-02T07:55:31.600632+0000","last_fullsized":"2024-04-02T07:55:31.600632+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:55:29.860904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.e","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.608662+0000","last_change":"2024-04-02T07:55:31.098430+0000","last_active":"2024-04-02T07:55:31.608662+0000","last_peered":"2024-04-02T07:55:31.608662+0000","last_clean":"2024-04-02T07:55:31.608662+0000","last_became_active":"2024-04-02T07:55:31.097972+0000","last_became_peered":"2024-04-02T07:55:31.097972+0000","last_unstale":"2024-04-02T07:55:31.608662+0000","last_undegraded":"2024-04-02T07:55:31.608662+0000","last_fullsized":"2024-04-02T07:55:31.608662+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:35:14.025296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.594950+0000","last_change":"2024-04-02T07:55:31.591539+0000","last_active":"2024-04-02T07:55:31.594950+0000","last_peered":"2024-04-02T07:55:31.594950+0000","last_clean":"2024-04-02T07:55:31.594950+0000","last_became_active":"2024-04-02T07:55:31.591259+0000","last_became_peered":"2024-04-02T07:55:31.591259+0000","last_unstale":"2024-04-02T07:55:31.594950+0000","last_undegraded":"2024-04-02T07:55:31.594950+0000","last_fullsized":"2024-04-02T07:55:31.594950+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:02.593914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605795+0000","last_change":"2024-04-02T07:55:31.091627+0000","last_active":"2024-04-02T07:55:31.605795+0000","last_peered":"2024-04-02T07:55:31.605795+0000","last_clean":"2024-04-02T07:55:31.605795+0000","last_became_active":"2024-04-02T07:55:31.091371+0000","last_became_peered":"2024-04-02T07:55:31.091371+0000","last_unstale":"2024-04-02T07:55:31.605795+0000","last_undegraded":"2024-04-02T07:55:31.605795+0000","last_fullsized":"2024-04-02T07:55:31.605795+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:54.255209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605663+0000","last_change":"2024-04-02T07:55:31.598883+0000","last_active":"2024-04-02T07:55:31.605663+0000","last_peered":"2024-04-02T07:55:31.605663+0000","last_clean":"2024-04-02T07:55:31.605663+0000","last_became_active":"2024-04-02T07:55:31.598338+0000","last_became_peered":"2024-04-02T07:55:31.598338+0000","last_unstale":"2024-04-02T07:55:31.605663+0000","last_undegraded":"2024-04-02T07:55:31.605663+0000","last_fullsized":"2024-04-02T07:55:31.605663+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:38.406238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.591313+0000","last_change":"2024-04-02T07:55:31.096067+0000","last_active":"2024-04-02T07:55:31.591313+0000","last_peered":"2024-04-02T07:55:31.591313+0000","last_clean":"2024-04-02T07:55:31.591313+0000","last_became_active":"2024-04-02T07:55:31.095770+0000","last_became_peered":"2024-04-02T07:55:31.095770+0000","last_unstale":"2024-04-02T07:55:31.591313+0000","last_undegraded":"2024-04-02T07:55:31.591313+0000","last_fullsized":"2024-04-02T07:55:31.591313+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:26:39.683874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.593747+0000","last_change":"2024-04-02T07:55:31.590071+0000","last_active":"2024-04-02T07:55:31.593747+0000","last_peered":"2024-04-02T07:55:31.593747+0000","last_clean":"2024-04-02T07:55:31.593747+0000","last_became_active":"2024-04-02T07:55:31.589540+0000","last_became_peered":"2024-04-02T07:55:31.589540+0000","last_unstale":"2024-04-02T07:55:31.593747+0000","last_undegraded":"2024-04-02T07:55:31.593747+0000","last_fullsized":"2024-04-02T07:55:31.593747+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:36:26.431812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.607795+0000","last_change":"2024-04-02T07:55:31.095524+0000","last_active":"2024-04-02T07:55:31.607795+0000","last_peered":"2024-04-02T07:55:31.607795+0000","last_clean":"2024-04-02T07:55:31.607795+0000","last_became_active":"2024-04-02T07:55:31.095273+0000","last_became_peered":"2024-04-02T07:55:31.095273+0000","last_unstale":"2024-04-02T07:55:31.607795+0000","last_undegraded":"2024-04-02T07:55:31.607795+0000","last_fullsized":"2024-04-02T07:55:31.607795+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:14:07.054260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.598228+0000","last_change":"2024-04-02T07:55:31.594315+0000","last_active":"2024-04-02T07:55:31.598228+0000","last_peered":"2024-04-02T07:55:31.598228+0000","last_clean":"2024-04-02T07:55:31.598228+0000","last_became_active":"2024-04-02T07:55:31.594052+0000","last_became_peered":"2024-04-02T07:55:31.594052+0000","last_unstale":"2024-04-02T07:55:31.598228+0000","last_undegraded":"2024-04-02T07:55:31.598228+0000","last_fullsized":"2024-04-02T07:55:31.598228+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:06:19.358325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.579780+0000","last_change":"2024-04-02T07:55:31.099701+0000","last_active":"2024-04-02T07:55:31.579780+0000","last_peered":"2024-04-02T07:55:31.579780+0000","last_clean":"2024-04-02T07:55:31.579780+0000","last_became_active":"2024-04-02T07:55:31.099109+0000","last_became_peered":"2024-04-02T07:55:31.099109+0000","last_unstale":"2024-04-02T07:55:31.579780+0000","last_undegraded":"2024-04-02T07:55:31.579780+0000","last_fullsized":"2024-04-02T07:55:31.579780+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:14.284231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.604220+0000","last_change":"2024-04-02T07:55:31.596467+0000","last_active":"2024-04-02T07:55:31.604220+0000","last_peered":"2024-04-02T07:55:31.604220+0000","last_clean":"2024-04-02T07:55:31.604220+0000","last_became_active":"2024-04-02T07:55:31.596169+0000","last_became_peered":"2024-04-02T07:55:31.596169+0000","last_unstale":"2024-04-02T07:55:31.604220+0000","last_undegraded":"2024-04-02T07:55:31.604220+0000","last_fullsized":"2024-04-02T07:55:31.604220+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:44:37.486456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.593209+0000","last_change":"2024-04-02T07:55:31.096911+0000","last_active":"2024-04-02T07:55:31.593209+0000","last_peered":"2024-04-02T07:55:31.593209+0000","last_clean":"2024-04-02T07:55:31.593209+0000","last_became_active":"2024-04-02T07:55:31.096516+0000","last_became_peered":"2024-04-02T07:55:31.096516+0000","last_unstale":"2024-04-02T07:55:31.593209+0000","last_undegraded":"2024-04-02T07:55:31.593209+0000","last_fullsized":"2024-04-02T07:55:31.593209+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:42:21.973647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.601869+0000","last_change":"2024-04-02T07:55:31.596758+0000","last_active":"2024-04-02T07:55:31.601869+0000","last_peered":"2024-04-02T07:55:31.601869+0000","last_clean":"2024-04-02T07:55:31.601869+0000","last_became_active":"2024-04-02T07:55:31.596021+0000","last_became_peered":"2024-04-02T07:55:31.596021+0000","last_unstale":"2024-04-02T07:55:31.601869+0000","last_undegraded":"2024-04-02T07:55:31.601869+0000","last_fullsized":"2024-04-02T07:55:31.601869+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:36:26.431812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.592778+0000","last_change":"2024-04-02T07:55:31.084846+0000","last_active":"2024-04-02T07:55:31.592778+0000","last_peered":"2024-04-02T07:55:31.592778+0000","last_clean":"2024-04-02T07:55:31.592778+0000","last_became_active":"2024-04-02T07:55:31.084273+0000","last_became_peered":"2024-04-02T07:55:31.084273+0000","last_unstale":"2024-04-02T07:55:31.592778+0000","last_undegraded":"2024-04-02T07:55:31.592778+0000","last_fullsized":"2024-04-02T07:55:31.592778+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:50:14.957062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605903+0000","last_change":"2024-04-02T07:55:31.601740+0000","last_active":"2024-04-02T07:55:31.605903+0000","last_peered":"2024-04-02T07:55:31.605903+0000","last_clean":"2024-04-02T07:55:31.605903+0000","last_became_active":"2024-04-02T07:55:31.601484+0000","last_became_peered":"2024-04-02T07:55:31.601484+0000","last_unstale":"2024-04-02T07:55:31.605903+0000","last_undegraded":"2024-04-02T07:55:31.605903+0000","last_fullsized":"2024-04-02T07:55:31.605903+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:16:03.608355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.7","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.610760+0000","last_change":"2024-04-02T07:55:31.091390+0000","last_active":"2024-04-02T07:55:31.610760+0000","last_peered":"2024-04-02T07:55:31.610760+0000","last_clean":"2024-04-02T07:55:31.610760+0000","last_became_active":"2024-04-02T07:55:31.091115+0000","last_became_peered":"2024-04-02T07:55:31.091115+0000","last_unstale":"2024-04-02T07:55:31.610760+0000","last_undegraded":"2024-04-02T07:55:31.610760+0000","last_fullsized":"2024-04-02T07:55:31.610760+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:32:05.622480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.598084+0000","last_change":"2024-04-02T07:55:31.592775+0000","last_active":"2024-04-02T07:55:31.598084+0000","last_peered":"2024-04-02T07:55:31.598084+0000","last_clean":"2024-04-02T07:55:31.598084+0000","last_became_active":"2024-04-02T07:55:31.590036+0000","last_became_peered":"2024-04-02T07:55:31.590036+0000","last_unstale":"2024-04-02T07:55:31.598084+0000","last_undegraded":"2024-04-02T07:55:31.598084+0000","last_fullsized":"2024-04-02T07:55:31.598084+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:34:54.181448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.587947+0000","last_change":"2024-04-02T07:55:31.086115+0000","last_active":"2024-04-02T07:55:31.587947+0000","last_peered":"2024-04-02T07:55:31.587947+0000","last_clean":"2024-04-02T07:55:31.587947+0000","last_became_active":"2024-04-02T07:55:31.085548+0000","last_became_peered":"2024-04-02T07:55:31.085548+0000","last_unstale":"2024-04-02T07:55:31.587947+0000","last_undegraded":"2024-04-02T07:55:31.587947+0000","last_fullsized":"2024-04-02T07:55:31.587947+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:37:34.798163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.599734+0000","last_change":"2024-04-02T07:55:31.588374+0000","last_active":"2024-04-02T07:55:31.599734+0000","last_peered":"2024-04-02T07:55:31.599734+0000","last_clean":"2024-04-02T07:55:31.599734+0000","last_became_active":"2024-04-02T07:55:31.588077+0000","last_became_peered":"2024-04-02T07:55:31.588077+0000","last_unstale":"2024-04-02T07:55:31.599734+0000","last_undegraded":"2024-04-02T07:55:31.599734+0000","last_fullsized":"2024-04-02T07:55:31.599734+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:35:52.948729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.596579+0000","last_change":"2024-04-02T07:55:31.082634+0000","last_active":"2024-04-02T07:55:31.596579+0000","last_peered":"2024-04-02T07:55:31.596579+0000","last_clean":"2024-04-02T07:55:31.596579+0000","last_became_active":"2024-04-02T07:55:31.082321+0000","last_became_peered":"2024-04-02T07:55:31.082321+0000","last_unstale":"2024-04-02T07:55:31.596579+0000","last_undegraded":"2024-04-02T07:55:31.596579+0000","last_fullsized":"2024-04-02T07:55:31.596579+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:36:25.424033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605691+0000","last_change":"2024-04-02T07:55:31.599385+0000","last_active":"2024-04-02T07:55:31.605691+0000","last_peered":"2024-04-02T07:55:31.605691+0000","last_clean":"2024-04-02T07:55:31.605691+0000","last_became_active":"2024-04-02T07:55:31.598037+0000","last_became_peered":"2024-04-02T07:55:31.598037+0000","last_unstale":"2024-04-02T07:55:31.605691+0000","last_undegraded":"2024-04-02T07:55:31.605691+0000","last_fullsized":"2024-04-02T07:55:31.605691+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.274339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.595618+0000","last_change":"2024-04-02T07:55:31.088474+0000","last_active":"2024-04-02T07:55:31.595618+0000","last_peered":"2024-04-02T07:55:31.595618+0000","last_clean":"2024-04-02T07:55:31.595618+0000","last_became_active":"2024-04-02T07:55:31.088223+0000","last_became_peered":"2024-04-02T07:55:31.088223+0000","last_unstale":"2024-04-02T07:55:31.595618+0000","last_undegraded":"2024-04-02T07:55:31.595618+0000","last_fullsized":"2024-04-02T07:55:31.595618+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:05:44.734602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.598328+0000","last_change":"2024-04-02T07:55:31.593370+0000","last_active":"2024-04-02T07:55:31.598328+0000","last_peered":"2024-04-02T07:55:31.598328+0000","last_clean":"2024-04-02T07:55:31.598328+0000","last_became_active":"2024-04-02T07:55:31.593128+0000","last_became_peered":"2024-04-02T07:55:31.593128+0000","last_unstale":"2024-04-02T07:55:31.598328+0000","last_undegraded":"2024-04-02T07:55:31.598328+0000","last_fullsized":"2024-04-02T07:55:31.598328+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:48:20.542068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.587273+0000","last_change":"2024-04-02T07:55:31.091389+0000","last_active":"2024-04-02T07:55:31.587273+0000","last_peered":"2024-04-02T07:55:31.587273+0000","last_clean":"2024-04-02T07:55:31.587273+0000","last_became_active":"2024-04-02T07:55:31.090908+0000","last_became_peered":"2024-04-02T07:55:31.090908+0000","last_unstale":"2024-04-02T07:55:31.587273+0000","last_undegraded":"2024-04-02T07:55:31.587273+0000","last_fullsized":"2024-04-02T07:55:31.587273+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:37:39.394344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.604234+0000","last_change":"2024-04-02T07:55:31.599657+0000","last_active":"2024-04-02T07:55:31.604234+0000","last_peered":"2024-04-02T07:55:31.604234+0000","last_clean":"2024-04-02T07:55:31.604234+0000","last_became_active":"2024-04-02T07:55:31.599390+0000","last_became_peered":"2024-04-02T07:55:31.599390+0000","last_unstale":"2024-04-02T07:55:31.604234+0000","last_undegraded":"2024-04-02T07:55:31.604234+0000","last_fullsized":"2024-04-02T07:55:31.604234+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:49:10.266042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.590744+0000","last_change":"2024-04-02T07:55:31.095182+0000","last_active":"2024-04-02T07:55:31.590744+0000","last_peered":"2024-04-02T07:55:31.590744+0000","last_clean":"2024-04-02T07:55:31.590744+0000","last_became_active":"2024-04-02T07:55:31.094864+0000","last_became_peered":"2024-04-02T07:55:31.094864+0000","last_unstale":"2024-04-02T07:55:31.590744+0000","last_undegraded":"2024-04-02T07:55:31.590744+0000","last_fullsized":"2024-04-02T07:55:31.590744+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:44:05.411292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.604197+0000","last_change":"2024-04-02T07:55:31.595253+0000","last_active":"2024-04-02T07:55:31.604197+0000","last_peered":"2024-04-02T07:55:31.604197+0000","last_clean":"2024-04-02T07:55:31.604197+0000","last_became_active":"2024-04-02T07:55:31.595001+0000","last_became_peered":"2024-04-02T07:55:31.595001+0000","last_unstale":"2024-04-02T07:55:31.604197+0000","last_undegraded":"2024-04-02T07:55:31.604197+0000","last_fullsized":"2024-04-02T07:55:31.604197+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:31:33.770811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.578593+0000","last_change":"2024-04-02T07:55:31.093524+0000","last_active":"2024-04-02T07:55:31.578593+0000","last_peered":"2024-04-02T07:55:31.578593+0000","last_clean":"2024-04-02T07:55:31.578593+0000","last_became_active":"2024-04-02T07:55:31.093258+0000","last_became_peered":"2024-04-02T07:55:31.093258+0000","last_unstale":"2024-04-02T07:55:31.578593+0000","last_undegraded":"2024-04-02T07:55:31.578593+0000","last_fullsized":"2024-04-02T07:55:31.578593+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:36:20.148162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.585504+0000","last_change":"2024-04-02T07:55:31.583678+0000","last_active":"2024-04-02T07:55:31.585504+0000","last_peered":"2024-04-02T07:55:31.585504+0000","last_clean":"2024-04-02T07:55:31.585504+0000","last_became_active":"2024-04-02T07:55:31.583411+0000","last_became_peered":"2024-04-02T07:55:31.583411+0000","last_unstale":"2024-04-02T07:55:31.585504+0000","last_undegraded":"2024-04-02T07:55:31.585504+0000","last_fullsized":"2024-04-02T07:55:31.585504+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:13.861818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.577289+0000","last_change":"2024-04-02T07:55:31.087591+0000","last_active":"2024-04-02T07:55:31.577289+0000","last_peered":"2024-04-02T07:55:31.577289+0000","last_clean":"2024-04-02T07:55:31.577289+0000","last_became_active":"2024-04-02T07:55:31.087278+0000","last_became_peered":"2024-04-02T07:55:31.087278+0000","last_unstale":"2024-04-02T07:55:31.577289+0000","last_undegraded":"2024-04-02T07:55:31.577289+0000","last_fullsized":"2024-04-02T07:55:31.577289+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:10:08.260722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.606809+0000","last_change":"2024-04-02T07:55:31.596742+0000","last_active":"2024-04-02T07:55:31.606809+0000","last_peered":"2024-04-02T07:55:31.606809+0000","last_clean":"2024-04-02T07:55:31.606809+0000","last_became_active":"2024-04-02T07:55:31.596422+0000","last_became_peered":"2024-04-02T07:55:31.596422+0000","last_unstale":"2024-04-02T07:55:31.606809+0000","last_undegraded":"2024-04-02T07:55:31.606809+0000","last_fullsized":"2024-04-02T07:55:31.606809+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:55.262988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.585327+0000","last_change":"2024-04-02T07:55:31.096221+0000","last_active":"2024-04-02T07:55:31.585327+0000","last_peered":"2024-04-02T07:55:31.585327+0000","last_clean":"2024-04-02T07:55:31.585327+0000","last_became_active":"2024-04-02T07:55:31.095900+0000","last_became_peered":"2024-04-02T07:55:31.095900+0000","last_unstale":"2024-04-02T07:55:31.585327+0000","last_undegraded":"2024-04-02T07:55:31.585327+0000","last_fullsized":"2024-04-02T07:55:31.585327+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:23:05.365858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"129'227","reported_seq":669,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.076505+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:55:31.076505+0000","last_peered":"2024-04-02T07:55:31.076505+0000","last_clean":"2024-04-02T07:55:31.076505+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:55:31.076505+0000","last_undegraded":"2024-04-02T07:55:31.076505+0000","last_fullsized":"2024-04-02T07:55:31.076505+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575077,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32824,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339016,"statfs":{"total":96636764160,"available":96603152384,"internally_reserved":0,"allocated":1744896,"data_stored":855628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63200000000000001,"15min":0.66500000000000004},"min":{"1min":0.38600000000000001,"5min":0.34699999999999998,"15min":0.316},"max":{"1min":0.80700000000000005,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.41499999999999998,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.53400000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65100000000000002,"15min":0.67500000000000004},"min":{"1min":0.48699999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88100000000000001,"5min":1.097,"15min":1.097},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63800000000000001,"15min":0.66500000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.34699999999999998},"max":{"1min":0.78000000000000003,"5min":0.98299999999999998,"15min":1.2190000000000001},"last":0.55700000000000005}]},{"osd":2,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66200000000000003,"15min":0.69899999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.34000000000000002},"max":{"1min":0.88700000000000001,"5min":0.98999999999999999,"15min":1.127},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.72499999999999998,"15min":0.75900000000000001},"min":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95299999999999996,"5min":1.044,"15min":1.1259999999999999},"last":0.58499999999999996}]},{"osd":3,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.69899999999999995},"min":{"1min":0.45800000000000002,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.94199999999999995,"5min":0.97799999999999998,"15min":1.026},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68200000000000005,"15min":0.71299999999999997},"min":{"1min":0.52700000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.83399999999999996,"5min":1.071,"15min":1.196},"last":0.63}]},{"osd":4,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58799999999999997},"min":{"1min":0.377,"5min":0.35799999999999998,"15min":0.32000000000000001},"max":{"1min":0.85899999999999999,"5min":1.0229999999999999,"15min":1.0389999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68700000000000006,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.004,"5min":1.016,"15min":1.0669999999999999},"last":0.73899999999999999}]},{"osd":5,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.64400000000000002,"15min":0.65000000000000002},"min":{"1min":0.45500000000000002,"5min":0.42999999999999999,"15min":0.37},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.167},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.022,"5min":1.343,"15min":1.4219999999999999},"last":0.77900000000000003}]},{"osd":6,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.36199999999999999},"max":{"1min":0.92000000000000004,"5min":1.0580000000000001,"15min":2.5369999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.66600000000000004,"15min":0.66000000000000003},"min":{"1min":0.48399999999999999,"5min":0.41299999999999998,"15min":0.39600000000000002},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.70999999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575075,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36904,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334936,"statfs":{"total":96636764160,"available":96598974464,"internally_reserved":0,"allocated":1724416,"data_stored":853940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.67000000000000004,"15min":0.70099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.33400000000000002},"max":{"1min":0.89000000000000001,"5min":0.95899999999999996,"15min":0.98099999999999998},"last":0.628},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66700000000000004,"15min":0.70399999999999996},"min":{"1min":0.46200000000000002,"5min":0.40500000000000003,"15min":0.38300000000000001},"max":{"1min":0.88600000000000001,"5min":0.94299999999999995,"15min":1.0249999999999999},"last":0.64600000000000002}]},{"osd":1,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67000000000000004,"15min":0.69199999999999995},"min":{"1min":0.505,"5min":0.433,"15min":0.41799999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69899999999999995,"15min":0.71499999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":0.95799999999999996,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.50900000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68000000000000005,"15min":0.70999999999999996},"min":{"1min":0.46500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68899999999999995,"15min":0.70699999999999996},"min":{"1min":0.52600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.67200000000000004}]},{"osd":3,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71199999999999997,"15min":0.72899999999999998},"min":{"1min":0.53200000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.73899999999999999,"15min":0.745},"min":{"1min":0.52800000000000002,"5min":0.46600000000000003,"15min":0.45300000000000001},"max":{"1min":3.339,"5min":3.339,"15min":3.339},"last":0.68500000000000005}]},{"osd":4,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.625,"15min":0.60699999999999998},"min":{"1min":0.39100000000000001,"5min":0.377,"15min":0.372},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64100000000000001,"15min":0.63300000000000001},"min":{"1min":0.44500000000000001,"5min":0.41499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.71299999999999997}]},{"osd":5,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.629,"15min":0.625},"min":{"1min":0.47599999999999998,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.89200000000000002,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.69999999999999996,"15min":0.68600000000000005},"min":{"1min":0.45500000000000002,"5min":0.40899999999999997,"15min":0.40300000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.77600000000000002}]},{"osd":7,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.69999999999999996,"15min":0.68999999999999995},"min":{"1min":0.501,"5min":0.46999999999999997,"15min":0.40200000000000002},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.749},{"interface":"front","average":{"1min":0.754,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.52000000000000002,"5min":0.42799999999999999,"15min":0.42099999999999999},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.76200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575076,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59612,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312228,"statfs":{"total":96636764160,"available":96575721472,"internally_reserved":0,"allocated":2306048,"data_stored":1437157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.54400000000000004,"15min":0.47599999999999998},"min":{"1min":0.30199999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.57599999999999996,"5min":10.052,"15min":10.052},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.436,"5min":0.59399999999999997,"15min":0.53300000000000003},"min":{"1min":0.30399999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61799999999999999,"5min":9.0250000000000004,"15min":9.0250000000000004},"last":0.47999999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.58799999999999997,"15min":0.53000000000000003},"min":{"1min":0.36099999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.55500000000000005,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.64100000000000001,"15min":0.59899999999999998},"min":{"1min":0.45500000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.68899999999999995,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.46500000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64500000000000002,"5min":2.609,"15min":2.609},"last":0.433},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66800000000000004,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.36899999999999999}]},{"osd":4,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.75800000000000001,"15min":0.77000000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86599999999999999,"5min":2.21,"15min":2.21},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.81000000000000005,"15min":0.78500000000000003},"min":{"1min":0.47899999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.96299999999999997,"5min":6.3419999999999996,"15min":6.3419999999999996},"last":0.92600000000000005}]},{"osd":5,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.82499999999999996,"15min":0.78900000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.41999999999999998},"max":{"1min":0.96299999999999997,"5min":6.5529999999999999,"15min":6.5529999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.78200000000000003,"15min":0.79000000000000004},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.48799999999999999},"max":{"1min":1.0189999999999999,"5min":2.544,"15min":2.544},"last":0.94299999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":1.849,"15min":1.1519999999999999},"min":{"1min":0.55200000000000005,"5min":0.53200000000000003,"15min":0.47699999999999998},"max":{"1min":0.95399999999999996,"5min":78.903000000000006,"15min":78.903000000000006},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":1.8260000000000001,"15min":1.139},"min":{"1min":0.52400000000000002,"5min":0.49199999999999999,"15min":0.38100000000000001},"max":{"1min":0.89600000000000002,"5min":78.837000000000003,"15min":78.837000000000003},"last":0.73299999999999998}]},{"osd":7,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78500000000000003,"15min":0.80000000000000004},"min":{"1min":0.55400000000000005,"5min":0.52800000000000002,"15min":0.48799999999999999},"max":{"1min":0.93200000000000005,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.79100000000000004,"15min":0.81799999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.97699999999999998,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":0.80100000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575076,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32788,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":1708032,"data_stored":845770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.42199999999999999,"15min":0.437},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67800000000000005,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.45900000000000002,"15min":0.48199999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.629,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.55500000000000005}]},{"osd":2,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47299999999999998,"15min":0.48599999999999999},"min":{"1min":0.32200000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":1.597,"15min":1.597},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51100000000000001,"15min":0.52700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72699999999999998,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.41399999999999998}]},{"osd":3,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.50700000000000001,"15min":0.52300000000000002},"min":{"1min":0.38700000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81799999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.60199999999999998},"min":{"1min":0.40300000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.78500000000000003,"5min":1.492,"15min":1.492},"last":0.59999999999999998}]},{"osd":4,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.753,"15min":0.77600000000000002},"min":{"1min":0.51500000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0409999999999999,"5min":1.1579999999999999,"15min":1.181},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76300000000000001,"15min":0.77300000000000002},"min":{"1min":0.54300000000000004,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":1.097,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.82499999999999996}]},{"osd":5,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.76600000000000001,"15min":0.79100000000000004},"min":{"1min":0.56599999999999995,"5min":0.48699999999999999,"15min":0.47299999999999998},"max":{"1min":0.998,"5min":1.361,"15min":2.1869999999999998},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.76900000000000002,"15min":0.79500000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.026,"5min":1.546,"15min":2.6230000000000002},"last":0.89700000000000002}]},{"osd":6,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.73099999999999998,"15min":0.747},"min":{"1min":0.47999999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.93700000000000006,"5min":1.52,"15min":1.52},"last":0.748},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.71799999999999997,"15min":0.745},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.41299999999999998},"max":{"1min":1.026,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.69799999999999995}]},{"osd":7,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.78800000000000003,"15min":0.81200000000000006},"min":{"1min":0.53600000000000003,"5min":0.51600000000000001,"15min":0.505},"max":{"1min":1.002,"5min":1.139,"15min":2.3959999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.76200000000000001,"15min":0.80100000000000005},"min":{"1min":0.50900000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0960000000000001,"5min":1.3959999999999999,"15min":2.145},"last":0.86799999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575075,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36912,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334928,"statfs":{"total":96636764160,"available":96598966272,"internally_reserved":0,"allocated":1736704,"data_stored":854578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.29399999999999998,"5min":0.26700000000000002,"15min":0.26400000000000001},"max":{"1min":0.70499999999999996,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46700000000000003,"15min":0.47299999999999998},"min":{"1min":0.33600000000000002,"5min":0.29099999999999998,"15min":0.26200000000000001},"max":{"1min":0.58899999999999997,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.65000000000000002}]},{"osd":1,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.33800000000000002,"5min":0.32700000000000001,"15min":0.30499999999999999},"max":{"1min":0.58699999999999997,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.5,"15min":0.51700000000000002},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":0.58499999999999996,"5min":0.996,"15min":0.996},"last":0.59099999999999997}]},{"osd":3,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.316},"max":{"1min":0.82399999999999995,"5min":1.472,"15min":1.472},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.59199999999999997,"15min":0.60299999999999998},"min":{"1min":0.42899999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.70999999999999996,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.67200000000000004}]},{"osd":4,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.72699999999999998,"15min":0.76000000000000001},"min":{"1min":0.52300000000000002,"5min":0.5,"15min":0.47199999999999998},"max":{"1min":1.046,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.873},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.73199999999999998,"15min":0.76200000000000001},"min":{"1min":0.56000000000000005,"5min":0.55000000000000004,"15min":0.48499999999999999},"max":{"1min":1.1040000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.78000000000000003}]},{"osd":5,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.72199999999999998,"15min":0.751},"min":{"1min":0.54300000000000004,"5min":0.51000000000000001,"15min":0.46200000000000002},"max":{"1min":0.91500000000000004,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.76900000000000002},"min":{"1min":0.61499999999999999,"5min":0.55600000000000005,"15min":0.50900000000000001},"max":{"1min":0.95199999999999996,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.79800000000000004}]},{"osd":6,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.747,"15min":0.77800000000000002},"min":{"1min":0.58399999999999996,"5min":0.501,"15min":0.442},"max":{"1min":1.079,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.77100000000000002},"min":{"1min":0.56499999999999995,"5min":0.502,"15min":0.502},"max":{"1min":0.871,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.73499999999999999}]},{"osd":7,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.78400000000000003},"min":{"1min":0.55900000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0249999999999999,"5min":1.278,"15min":1.278},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73299999999999998,"15min":0.77900000000000003},"min":{"1min":0.504,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.99099999999999999,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.76600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575075,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59044,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312796,"statfs":{"total":96636764160,"available":96576303104,"internally_reserved":0,"allocated":1724416,"data_stored":853940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.437,"15min":0.42899999999999999},"min":{"1min":0.314,"5min":0.26500000000000001,"15min":0.23499999999999999},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.44600000000000001,"15min":0.46000000000000002},"min":{"1min":0.308,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58599999999999997,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.442}]},{"osd":1,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46999999999999997,"15min":0.48699999999999999},"min":{"1min":0.33900000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69599999999999995,"5min":0.78000000000000003,"15min":1.2809999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56899999999999995,"15min":0.56299999999999994},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71099999999999997,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":0.45800000000000002}]},{"osd":2,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.60799999999999998},"min":{"1min":0.42599999999999999,"5min":0.40899999999999997,"15min":0.38700000000000001},"max":{"1min":0.88900000000000001,"5min":2.214,"15min":2.214},"last":0.499},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58499999999999996,"15min":0.60099999999999998},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.50600000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.72999999999999998,"15min":0.73299999999999998},"min":{"1min":0.53600000000000003,"5min":0.47699999999999998,"15min":0.38400000000000001},"max":{"1min":0.86499999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.39600000000000002},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.61599999999999999}]},{"osd":5,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72799999999999998,"15min":0.751},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.40899999999999997},"max":{"1min":0.84399999999999997,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.74399999999999999,"15min":0.749},"min":{"1min":0.51100000000000001,"5min":0.49199999999999999,"15min":0.40200000000000002},"max":{"1min":0.90000000000000002,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.77600000000000002}]},{"osd":6,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76900000000000002,"15min":0.76500000000000001},"min":{"1min":0.55300000000000005,"5min":0.51100000000000001,"15min":0.45800000000000002},"max":{"1min":1.4359999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73899999999999999,"15min":0.76800000000000002},"min":{"1min":0.51700000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.90600000000000003,"5min":2.286,"15min":2.286},"last":0.70499999999999996}]},{"osd":7,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.746,"15min":0.76700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.439},"max":{"1min":0.82799999999999996,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77400000000000002,"15min":0.78500000000000003},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.48999999999999999},"max":{"1min":1.3200000000000001,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.68700000000000006}]}]},{"osd":4,"up_from":13,"seq":55834575075,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41012,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330828,"statfs":{"total":96636764160,"available":96594767872,"internally_reserved":0,"allocated":1740800,"data_stored":855594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64700000000000002,"15min":0.66900000000000004},"min":{"1min":0.42199999999999999,"5min":0.41699999999999998,"15min":0.32100000000000001},"max":{"1min":0.81999999999999995,"5min":0.97399999999999998,"15min":1.0189999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.64100000000000001,"15min":0.66900000000000004},"min":{"1min":0.44600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91800000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.54200000000000004}]},{"osd":1,"last update":"Tue Apr 2 07:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.70299999999999996,"15min":0.72299999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.433},"max":{"1min":1.012,"5min":1.014,"15min":1.036},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.70899999999999996,"15min":0.73499999999999999},"min":{"1min":0.47899999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.95799999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.72199999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.68000000000000005,"15min":0.69399999999999995},"min":{"1min":0.44500000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.93799999999999994,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.69299999999999995,"15min":0.71199999999999997},"min":{"1min":0.497,"5min":0.378,"15min":0.378},"max":{"1min":0.91800000000000004,"5min":1.026,"15min":1.0620000000000001},"last":0.67000000000000004}]},{"osd":3,"last update":"Tue Apr 2 07:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.53400000000000003,"5min":0.505,"15min":0.46800000000000003},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70199999999999996,"15min":0.72699999999999998},"min":{"1min":0.504,"5min":0.47299999999999998,"15min":0.41499999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.61399999999999999}]},{"osd":5,"last update":"Tue Apr 2 07:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58899999999999997,"15min":0.58099999999999996},"min":{"1min":0.39400000000000002,"5min":0.38600000000000001,"15min":0.29599999999999999},"max":{"1min":0.89600000000000002,"5min":0.90800000000000003,"15min":0.93999999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60199999999999998,"15min":0.58799999999999997},"min":{"1min":0.41699999999999998,"5min":0.41399999999999998,"15min":0.33700000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.69199999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.628,"15min":0.60599999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.374},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67300000000000004,"15min":0.65500000000000003},"min":{"1min":0.48099999999999998,"5min":0.44400000000000001,"15min":0.42399999999999999},"max":{"1min":0.97999999999999998,"5min":1.0129999999999999,"15min":1.038},"last":0.60099999999999998}]},{"osd":7,"last update":"Tue Apr 2 07:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.46500000000000002,"5min":0.39600000000000002,"15min":0.38300000000000001},"max":{"1min":0.98199999999999998,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.748,"15min":0.73699999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.434},"max":{"1min":1.0509999999999999,"5min":1.095,"15min":1.095},"last":0.77800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575078,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55532,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316308,"statfs":{"total":96636764160,"available":96579899392,"internally_reserved":0,"allocated":2322432,"data_stored":1443381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64200000000000002,"15min":0.67800000000000005},"min":{"1min":0.45200000000000001,"5min":0.35299999999999998,"15min":0.29899999999999999},"max":{"1min":0.90300000000000002,"5min":0.92300000000000004,"15min":0.99199999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.65100000000000002,"15min":0.68300000000000005},"min":{"1min":0.40500000000000003,"5min":0.40000000000000002,"15min":0.378},"max":{"1min":0.79000000000000004,"5min":1.0249999999999999,"15min":1.4450000000000001},"last":0.71299999999999997}]},{"osd":1,"last update":"Tue Apr 2 07:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.64600000000000002,"15min":0.68700000000000006},"min":{"1min":0.40999999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.875,"5min":1.0620000000000001,"15min":1.131},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66100000000000003,"15min":0.68899999999999995},"min":{"1min":0.48899999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.872,"5min":0.96399999999999997,"15min":1.5149999999999999},"last":0.78300000000000003}]},{"osd":2,"last update":"Tue Apr 2 07:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66500000000000004,"15min":0.70199999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.41799999999999998},"max":{"1min":0.88800000000000001,"5min":0.90700000000000003,"15min":1.5449999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.67300000000000004,"15min":0.70299999999999996},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.372},"max":{"1min":0.89300000000000002,"5min":1.111,"15min":1.3959999999999999},"last":0.879}]},{"osd":3,"last update":"Tue Apr 2 07:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.73199999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.92500000000000004,"5min":1.0649999999999999,"15min":1.341},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.54500000000000004,"5min":0.48299999999999998,"15min":0.44900000000000001},"max":{"1min":0.94099999999999995,"5min":1.1839999999999999,"15min":1.484},"last":0.92000000000000004}]},{"osd":4,"last update":"Tue Apr 2 07:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.40600000000000003,"5min":0.39500000000000002,"15min":0.372},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":1.3109999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64600000000000002,"15min":0.64000000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.35599999999999998},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":1.504},"last":0.59899999999999998}]},{"osd":6,"last update":"Tue Apr 2 07:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.39900000000000002},"max":{"1min":1.004,"5min":1.212,"15min":1.212},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69799999999999995,"15min":0.67700000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41199999999999998},"max":{"1min":0.92800000000000005,"5min":1.774,"15min":1.774},"last":0.57499999999999996}]},{"osd":7,"last update":"Tue Apr 2 07:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.72799999999999998,"15min":0.70499999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.38900000000000001},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.6890000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.745,"15min":0.72799999999999998},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":1.139,"5min":1.149,"15min":1.5309999999999999},"last":0.67500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:55:35.962 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:55:36.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.186+0000 7f3ab8600700 1 Processor -- start 2024-04-02T07:55:36.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.186+0000 7f3ab8600700 1 -- start start 2024-04-02T07:55:36.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.187+0000 7f3ab8600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3ab015da50 0x7f3ab00ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:36.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.187+0000 7f3ab8600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ab00d0cf0 0x7f3ab00b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:36.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.187+0000 7f3ab8600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab00d24e0 con 0x7f3ab00cec70 2024-04-02T07:55:36.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.187+0000 7f3ab8600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab00b95d0 con 0x7f3ab015da50 2024-04-02T07:55:36.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.187+0000 7f3ab8600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab015de30 con 0x7f3ab00d0cf0 2024-04-02T07:55:36.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.188+0000 7f3ab5c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3ab015da50 0x7f3ab00ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:36.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.188+0000 7f3ab5200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3ab00cec70 0x7f3ab00d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44792/0 (socket says 172.21.15.89:44792) 2024-04-02T07:55:36.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.188+0000 7f3ab5200700 1 -- 172.21.15.89:0/746654037 learned_addr learned my addr 172.21.15.89:0/746654037 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:36.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.188+0000 7f3ab6600700 1 --2- 172.21.15.89:0/746654037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ab00d0cf0 0x7f3ab00b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:36.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.188+0000 7f3ab5c00700 1 -- 172.21.15.89:0/746654037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ab00d0cf0 msgr2=0x7f3ab00b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:36.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.188+0000 7f3ab5c00700 1 --2- 172.21.15.89:0/746654037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ab00d0cf0 0x7f3ab00b6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.188+0000 7f3ab5c00700 1 -- 172.21.15.89:0/746654037 >> v1:172.21.15.89:6789/0 conn(0x7f3ab00cec70 legacy=0x7f3ab00d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:36.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.188+0000 7f3ab5c00700 1 -- 172.21.15.89:0/746654037 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab00d27c0 con 0x7f3ab015da50 2024-04-02T07:55:36.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.188+0000 7f3ab5c00700 1 --2- 172.21.15.89:0/746654037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ab015da50 0x7f3ab00ce730 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4009840 tx=0x7f3aa4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bceb510f86832bf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:36.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.188+0000 7f3aafe00700 1 -- 172.21.15.89:0/746654037 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4011830 con 0x7f3ab015da50 2024-04-02T07:55:36.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.188+0000 7f3aafe00700 1 -- 172.21.15.89:0/746654037 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa4011990 con 0x7f3ab015da50 2024-04-02T07:55:36.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.188+0000 7f3aafe00700 1 -- 172.21.15.89:0/746654037 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4011d20 con 0x7f3ab015da50 2024-04-02T07:55:36.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.189+0000 7f3ab8600700 1 -- 172.21.15.89:0/746654037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ab015da50 msgr2=0x7f3ab00ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:36.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.189+0000 7f3ab8600700 1 --2- 172.21.15.89:0/746654037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ab015da50 0x7f3ab00ce730 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4009840 tx=0x7f3aa4009b50 comp rx=0 tx=0).stop 2024-04-02T07:55:36.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.189+0000 7f3ab8600700 1 -- 172.21.15.89:0/746654037 shutdown_connections 2024-04-02T07:55:36.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.189+0000 7f3ab8600700 1 --2- 172.21.15.89:0/746654037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ab00d0cf0 0x7f3ab00b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.189+0000 7f3ab8600700 1 --2- 172.21.15.89:0/746654037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ab015da50 0x7f3ab00ce730 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.189+0000 7f3ab8600700 1 -- 172.21.15.89:0/746654037 >> 172.21.15.89:0/746654037 conn(0x7f3ab00666a0 msgr2=0x7f3ab0066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:36.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.189+0000 7f3ab8600700 1 -- 172.21.15.89:0/746654037 shutdown_connections 2024-04-02T07:55:36.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.189+0000 7f3ab8600700 1 -- 172.21.15.89:0/746654037 wait complete. 2024-04-02T07:55:36.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.191+0000 7f3ab8600700 1 Processor -- start 2024-04-02T07:55:36.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.191+0000 7f3ab8600700 1 -- start start 2024-04-02T07:55:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.192+0000 7f3ab8600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ab00cec70 0x7f3ab00c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.192+0000 7f3ab8600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ab00d0cf0 0x7f3ab0155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.192+0000 7f3ab8600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ab015da50 0x7f3ab0155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.192+0000 7f3ab8600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab00cc050 con 0x7f3ab00d0cf0 2024-04-02T07:55:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.192+0000 7f3ab8600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab01f2fc0 con 0x7f3ab015da50 2024-04-02T07:55:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.192+0000 7f3ab8600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab00cb6b0 con 0x7f3ab00cec70 2024-04-02T07:55:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.192+0000 7f3ab5200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ab00d0cf0 0x7f3ab0155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:36.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.192+0000 7f3ab5200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ab00d0cf0 0x7f3ab0155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54930/0 (socket says 172.21.15.89:54930) 2024-04-02T07:55:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.192+0000 7f3ab5200700 1 -- 172.21.15.89:0/1296274873 learned_addr learned my addr 172.21.15.89:0/1296274873 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.192+0000 7f3ab6600700 1 --2- 172.21.15.89:0/1296274873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ab015da50 0x7f3ab0155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.192+0000 7f3ab5c00700 1 --2- 172.21.15.89:0/1296274873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ab00cec70 0x7f3ab00c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.193+0000 7f3ab5200700 1 -- 172.21.15.89:0/1296274873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ab00cec70 msgr2=0x7f3ab00c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.193+0000 7f3ab5200700 1 --2- 172.21.15.89:0/1296274873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ab00cec70 0x7f3ab00c9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.193+0000 7f3ab5200700 1 -- 172.21.15.89:0/1296274873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ab015da50 msgr2=0x7f3ab0155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.193+0000 7f3ab5200700 1 --2- 172.21.15.89:0/1296274873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ab015da50 0x7f3ab0155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.193+0000 7f3ab5200700 1 -- 172.21.15.89:0/1296274873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa40094a0 con 0x7f3ab00d0cf0 2024-04-02T07:55:36.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.193+0000 7f3ab5200700 1 --2- 172.21.15.89:0/1296274873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ab00d0cf0 0x7f3ab0155550 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000c500 tx=0x7f3aa000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.193+0000 7f3aad600700 1 -- 172.21.15.89:0/1296274873 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0005d70 con 0x7f3ab00d0cf0 2024-04-02T07:55:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.193+0000 7f3aad600700 1 -- 172.21.15.89:0/1296274873 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa00070d0 con 0x7f3ab00d0cf0 2024-04-02T07:55:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.193+0000 7f3aad600700 1 -- 172.21.15.89:0/1296274873 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0007440 con 0x7f3ab00d0cf0 2024-04-02T07:55:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.193+0000 7f3ab8600700 1 -- 172.21.15.89:0/1296274873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab00cb990 con 0x7f3ab00d0cf0 2024-04-02T07:55:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.193+0000 7f3ab8600700 1 -- 172.21.15.89:0/1296274873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab015fc70 con 0x7f3ab00d0cf0 2024-04-02T07:55:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.194+0000 7f3ab8600700 1 -- 172.21.15.89:0/1296274873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab00cfd00 con 0x7f3ab00d0cf0 2024-04-02T07:55:36.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.195+0000 7f3aad600700 1 -- 172.21.15.89:0/1296274873 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3aa00075a0 con 0x7f3ab00d0cf0 2024-04-02T07:55:36.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.196+0000 7f3aad600700 1 --2- 172.21.15.89:0/1296274873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3a9406adb0 0x7f3a9406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:36.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.196+0000 7f3aad600700 1 -- 172.21.15.89:0/1296274873 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f3aa0088950 con 0x7f3ab00d0cf0 2024-04-02T07:55:36.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.196+0000 7f3ab5c00700 1 --2- 172.21.15.89:0/1296274873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3a9406adb0 0x7f3a9406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:36.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.197+0000 7f3ab5c00700 1 --2- 172.21.15.89:0/1296274873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3a9406adb0 0x7f3a9406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3aa40090b0 tx=0x7f3aa401c040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:36.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.200+0000 7f3aad600700 1 -- 172.21.15.89:0/1296274873 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3aa00538b0 con 0x7f3ab00d0cf0 2024-04-02T07:55:36.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.399+0000 7f3ab8600700 1 -- 172.21.15.89:0/1296274873 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ab00d3060 con 0x7f3a9406adb0 2024-04-02T07:55:36.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.407+0000 7f3aad600700 1 -- 172.21.15.89:0/1296274873 <== mgr.8205 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289576 (secure 0 0 0) 0x7f3aa4011c30 con 0x7f3a9406adb0 2024-04-02T07:55:36.413 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:36.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.412+0000 7f3ab8600700 1 -- 172.21.15.89:0/1296274873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3a9406adb0 msgr2=0x7f3a9406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:36.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.412+0000 7f3ab8600700 1 --2- 172.21.15.89:0/1296274873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3a9406adb0 0x7f3a9406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3aa40090b0 tx=0x7f3aa401c040 comp rx=0 tx=0).stop 2024-04-02T07:55:36.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.412+0000 7f3ab8600700 1 -- 172.21.15.89:0/1296274873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ab00d0cf0 msgr2=0x7f3ab0155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:36.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.412+0000 7f3ab8600700 1 --2- 172.21.15.89:0/1296274873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ab00d0cf0 0x7f3ab0155550 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000c500 tx=0x7f3aa000c810 comp rx=0 tx=0).stop 2024-04-02T07:55:36.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.412+0000 7f3ab8600700 1 -- 172.21.15.89:0/1296274873 shutdown_connections 2024-04-02T07:55:36.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.412+0000 7f3ab8600700 1 --2- 172.21.15.89:0/1296274873 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3a9406adb0 0x7f3a9406d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.412+0000 7f3ab8600700 1 --2- 172.21.15.89:0/1296274873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ab015da50 0x7f3ab0155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.412+0000 7f3ab8600700 1 --2- 172.21.15.89:0/1296274873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ab00cec70 0x7f3ab00c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.412+0000 7f3ab8600700 1 --2- 172.21.15.89:0/1296274873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ab00d0cf0 0x7f3ab0155550 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.412+0000 7f3ab8600700 1 -- 172.21.15.89:0/1296274873 >> 172.21.15.89:0/1296274873 conn(0x7f3ab00666a0 msgr2=0x7f3ab00b7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:36.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.412+0000 7f3ab8600700 1 -- 172.21.15.89:0/1296274873 shutdown_connections 2024-04-02T07:55:36.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.412+0000 7f3ab8600700 1 -- 172.21.15.89:0/1296274873 wait complete. 2024-04-02T07:55:36.416 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:55:36.432 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-02T07:55:35.280047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":354628,"kb_used_data":14656,"kb_used_omap":12,"kb_used_meta":339827,"kb_avail":754620092,"statfs":{"total":773094113280,"available":772730974208,"internally_reserved":0,"allocated":15007744,"data_stored":7999988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":347983416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.212286"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.606601+0000","last_change":"2024-04-02T07:55:31.603558+0000","last_active":"2024-04-02T07:55:31.606601+0000","last_peered":"2024-04-02T07:55:31.606601+0000","last_clean":"2024-04-02T07:55:31.606601+0000","last_became_active":"2024-04-02T07:55:31.603272+0000","last_became_peered":"2024-04-02T07:55:31.603272+0000","last_unstale":"2024-04-02T07:55:31.606601+0000","last_undegraded":"2024-04-02T07:55:31.606601+0000","last_fullsized":"2024-04-02T07:55:31.606601+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:07:51.776847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.610246+0000","last_change":"2024-04-02T07:55:31.608040+0000","last_active":"2024-04-02T07:55:31.610246+0000","last_peered":"2024-04-02T07:55:31.610246+0000","last_clean":"2024-04-02T07:55:31.610246+0000","last_became_active":"2024-04-02T07:55:31.607582+0000","last_became_peered":"2024-04-02T07:55:31.607582+0000","last_unstale":"2024-04-02T07:55:31.610246+0000","last_undegraded":"2024-04-02T07:55:31.610246+0000","last_fullsized":"2024-04-02T07:55:31.610246+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:48:15.120406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.602607+0000","last_change":"2024-04-02T07:55:31.597008+0000","last_active":"2024-04-02T07:55:31.602607+0000","last_peered":"2024-04-02T07:55:31.602607+0000","last_clean":"2024-04-02T07:55:31.602607+0000","last_became_active":"2024-04-02T07:55:31.595009+0000","last_became_peered":"2024-04-02T07:55:31.595009+0000","last_unstale":"2024-04-02T07:55:31.602607+0000","last_undegraded":"2024-04-02T07:55:31.602607+0000","last_fullsized":"2024-04-02T07:55:31.602607+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:05:45.742380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.607588+0000","last_change":"2024-04-02T07:55:31.596385+0000","last_active":"2024-04-02T07:55:31.607588+0000","last_peered":"2024-04-02T07:55:31.607588+0000","last_clean":"2024-04-02T07:55:31.607588+0000","last_became_active":"2024-04-02T07:55:31.596105+0000","last_became_peered":"2024-04-02T07:55:31.596105+0000","last_unstale":"2024-04-02T07:55:31.607588+0000","last_undegraded":"2024-04-02T07:55:31.607588+0000","last_fullsized":"2024-04-02T07:55:31.607588+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:35:52.948729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.602560+0000","last_change":"2024-04-02T07:55:31.593771+0000","last_active":"2024-04-02T07:55:31.602560+0000","last_peered":"2024-04-02T07:55:31.602560+0000","last_clean":"2024-04-02T07:55:31.602560+0000","last_became_active":"2024-04-02T07:55:31.593310+0000","last_became_peered":"2024-04-02T07:55:31.593310+0000","last_unstale":"2024-04-02T07:55:31.602560+0000","last_undegraded":"2024-04-02T07:55:31.602560+0000","last_fullsized":"2024-04-02T07:55:31.602560+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:15.292010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.598295+0000","last_change":"2024-04-02T07:55:31.594948+0000","last_active":"2024-04-02T07:55:31.598295+0000","last_peered":"2024-04-02T07:55:31.598295+0000","last_clean":"2024-04-02T07:55:31.598295+0000","last_became_active":"2024-04-02T07:55:31.594685+0000","last_became_peered":"2024-04-02T07:55:31.594685+0000","last_unstale":"2024-04-02T07:55:31.598295+0000","last_undegraded":"2024-04-02T07:55:31.598295+0000","last_fullsized":"2024-04-02T07:55:31.598295+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:44:37.486456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605147+0000","last_change":"2024-04-02T07:55:31.600345+0000","last_active":"2024-04-02T07:55:31.605147+0000","last_peered":"2024-04-02T07:55:31.605147+0000","last_clean":"2024-04-02T07:55:31.605147+0000","last_became_active":"2024-04-02T07:55:31.600089+0000","last_became_peered":"2024-04-02T07:55:31.600089+0000","last_unstale":"2024-04-02T07:55:31.605147+0000","last_undegraded":"2024-04-02T07:55:31.605147+0000","last_fullsized":"2024-04-02T07:55:31.605147+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:18.774950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.595313+0000","last_change":"2024-04-02T07:55:31.592269+0000","last_active":"2024-04-02T07:55:31.595313+0000","last_peered":"2024-04-02T07:55:31.595313+0000","last_clean":"2024-04-02T07:55:31.595313+0000","last_became_active":"2024-04-02T07:55:31.591995+0000","last_became_peered":"2024-04-02T07:55:31.591995+0000","last_unstale":"2024-04-02T07:55:31.595313+0000","last_undegraded":"2024-04-02T07:55:31.595313+0000","last_fullsized":"2024-04-02T07:55:31.595313+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:07:42.638732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.610808+0000","last_change":"2024-04-02T07:55:31.606687+0000","last_active":"2024-04-02T07:55:31.610808+0000","last_peered":"2024-04-02T07:55:31.610808+0000","last_clean":"2024-04-02T07:55:31.610808+0000","last_became_active":"2024-04-02T07:55:31.605673+0000","last_became_peered":"2024-04-02T07:55:31.605673+0000","last_unstale":"2024-04-02T07:55:31.610808+0000","last_undegraded":"2024-04-02T07:55:31.610808+0000","last_fullsized":"2024-04-02T07:55:31.610808+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:28:07.810972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.600456+0000","last_change":"2024-04-02T07:55:31.590469+0000","last_active":"2024-04-02T07:55:31.600456+0000","last_peered":"2024-04-02T07:55:31.600456+0000","last_clean":"2024-04-02T07:55:31.600456+0000","last_became_active":"2024-04-02T07:55:31.589916+0000","last_became_peered":"2024-04-02T07:55:31.589916+0000","last_unstale":"2024-04-02T07:55:31.600456+0000","last_undegraded":"2024-04-02T07:55:31.600456+0000","last_fullsized":"2024-04-02T07:55:31.600456+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:53.426593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605776+0000","last_change":"2024-04-02T07:55:31.604573+0000","last_active":"2024-04-02T07:55:31.605776+0000","last_peered":"2024-04-02T07:55:31.605776+0000","last_clean":"2024-04-02T07:55:31.605776+0000","last_became_active":"2024-04-02T07:55:31.604322+0000","last_became_peered":"2024-04-02T07:55:31.604322+0000","last_unstale":"2024-04-02T07:55:31.605776+0000","last_undegraded":"2024-04-02T07:55:31.605776+0000","last_fullsized":"2024-04-02T07:55:31.605776+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:02:40.059301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.600915+0000","last_change":"2024-04-02T07:55:31.591751+0000","last_active":"2024-04-02T07:55:31.600915+0000","last_peered":"2024-04-02T07:55:31.600915+0000","last_clean":"2024-04-02T07:55:31.600915+0000","last_became_active":"2024-04-02T07:55:31.591445+0000","last_became_peered":"2024-04-02T07:55:31.591445+0000","last_unstale":"2024-04-02T07:55:31.600915+0000","last_undegraded":"2024-04-02T07:55:31.600915+0000","last_fullsized":"2024-04-02T07:55:31.600915+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:50:15.964841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.599205+0000","last_change":"2024-04-02T07:55:31.594243+0000","last_active":"2024-04-02T07:55:31.599205+0000","last_peered":"2024-04-02T07:55:31.599205+0000","last_clean":"2024-04-02T07:55:31.599205+0000","last_became_active":"2024-04-02T07:55:31.593477+0000","last_became_peered":"2024-04-02T07:55:31.593477+0000","last_unstale":"2024-04-02T07:55:31.599205+0000","last_undegraded":"2024-04-02T07:55:31.599205+0000","last_fullsized":"2024-04-02T07:55:31.599205+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:15.292010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.604651+0000","last_change":"2024-04-02T07:55:31.600044+0000","last_active":"2024-04-02T07:55:31.604651+0000","last_peered":"2024-04-02T07:55:31.604651+0000","last_clean":"2024-04-02T07:55:31.604651+0000","last_became_active":"2024-04-02T07:55:31.599743+0000","last_became_peered":"2024-04-02T07:55:31.599743+0000","last_unstale":"2024-04-02T07:55:31.604651+0000","last_undegraded":"2024-04-02T07:55:31.604651+0000","last_fullsized":"2024-04-02T07:55:31.604651+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:31:48.360153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.592434+0000","last_change":"2024-04-02T07:55:31.587033+0000","last_active":"2024-04-02T07:55:31.592434+0000","last_peered":"2024-04-02T07:55:31.592434+0000","last_clean":"2024-04-02T07:55:31.592434+0000","last_became_active":"2024-04-02T07:55:31.586772+0000","last_became_peered":"2024-04-02T07:55:31.586772+0000","last_unstale":"2024-04-02T07:55:31.592434+0000","last_undegraded":"2024-04-02T07:55:31.592434+0000","last_fullsized":"2024-04-02T07:55:31.592434+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:55.262988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.599484+0000","last_change":"2024-04-02T07:55:31.591692+0000","last_active":"2024-04-02T07:55:31.599484+0000","last_peered":"2024-04-02T07:55:31.599484+0000","last_clean":"2024-04-02T07:55:31.599484+0000","last_became_active":"2024-04-02T07:55:31.591415+0000","last_became_peered":"2024-04-02T07:55:31.591415+0000","last_unstale":"2024-04-02T07:55:31.599484+0000","last_undegraded":"2024-04-02T07:55:31.599484+0000","last_fullsized":"2024-04-02T07:55:31.599484+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:48:20.542068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.603320+0000","last_change":"2024-04-02T07:55:31.601169+0000","last_active":"2024-04-02T07:55:31.603320+0000","last_peered":"2024-04-02T07:55:31.603320+0000","last_clean":"2024-04-02T07:55:31.603320+0000","last_became_active":"2024-04-02T07:55:31.600930+0000","last_became_peered":"2024-04-02T07:55:31.600930+0000","last_unstale":"2024-04-02T07:55:31.603320+0000","last_undegraded":"2024-04-02T07:55:31.603320+0000","last_fullsized":"2024-04-02T07:55:31.603320+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:13:07.562470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.594758+0000","last_change":"2024-04-02T07:55:31.591154+0000","last_active":"2024-04-02T07:55:31.594758+0000","last_peered":"2024-04-02T07:55:31.594758+0000","last_clean":"2024-04-02T07:55:31.594758+0000","last_became_active":"2024-04-02T07:55:31.589651+0000","last_became_peered":"2024-04-02T07:55:31.589651+0000","last_unstale":"2024-04-02T07:55:31.594758+0000","last_undegraded":"2024-04-02T07:55:31.594758+0000","last_fullsized":"2024-04-02T07:55:31.594758+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:07:49.631127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.602645+0000","last_change":"2024-04-02T07:55:31.597181+0000","last_active":"2024-04-02T07:55:31.602645+0000","last_peered":"2024-04-02T07:55:31.602645+0000","last_clean":"2024-04-02T07:55:31.602645+0000","last_became_active":"2024-04-02T07:55:31.596854+0000","last_became_peered":"2024-04-02T07:55:31.596854+0000","last_unstale":"2024-04-02T07:55:31.602645+0000","last_undegraded":"2024-04-02T07:55:31.602645+0000","last_fullsized":"2024-04-02T07:55:31.602645+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:32:06.630259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.603026+0000","last_change":"2024-04-02T07:55:31.600923+0000","last_active":"2024-04-02T07:55:31.603026+0000","last_peered":"2024-04-02T07:55:31.603026+0000","last_clean":"2024-04-02T07:55:31.603026+0000","last_became_active":"2024-04-02T07:55:31.600658+0000","last_became_peered":"2024-04-02T07:55:31.600658+0000","last_unstale":"2024-04-02T07:55:31.603026+0000","last_undegraded":"2024-04-02T07:55:31.603026+0000","last_fullsized":"2024-04-02T07:55:31.603026+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:15.292010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.602717+0000","last_change":"2024-04-02T07:55:31.598595+0000","last_active":"2024-04-02T07:55:31.602717+0000","last_peered":"2024-04-02T07:55:31.602717+0000","last_clean":"2024-04-02T07:55:31.602717+0000","last_became_active":"2024-04-02T07:55:31.598345+0000","last_became_peered":"2024-04-02T07:55:31.598345+0000","last_unstale":"2024-04-02T07:55:31.602717+0000","last_undegraded":"2024-04-02T07:55:31.602717+0000","last_fullsized":"2024-04-02T07:55:31.602717+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:07:05.102587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.591831+0000","last_change":"2024-04-02T07:55:31.587979+0000","last_active":"2024-04-02T07:55:31.591831+0000","last_peered":"2024-04-02T07:55:31.591831+0000","last_clean":"2024-04-02T07:55:31.591831+0000","last_became_active":"2024-04-02T07:55:31.587691+0000","last_became_peered":"2024-04-02T07:55:31.587691+0000","last_unstale":"2024-04-02T07:55:31.591831+0000","last_undegraded":"2024-04-02T07:55:31.591831+0000","last_fullsized":"2024-04-02T07:55:31.591831+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:31:48.360153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605114+0000","last_change":"2024-04-02T07:55:31.600353+0000","last_active":"2024-04-02T07:55:31.605114+0000","last_peered":"2024-04-02T07:55:31.605114+0000","last_clean":"2024-04-02T07:55:31.605114+0000","last_became_active":"2024-04-02T07:55:31.600116+0000","last_became_peered":"2024-04-02T07:55:31.600116+0000","last_unstale":"2024-04-02T07:55:31.605114+0000","last_undegraded":"2024-04-02T07:55:31.605114+0000","last_fullsized":"2024-04-02T07:55:31.605114+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:08:01.923416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.610766+0000","last_change":"2024-04-02T07:55:31.605709+0000","last_active":"2024-04-02T07:55:31.610766+0000","last_peered":"2024-04-02T07:55:31.610766+0000","last_clean":"2024-04-02T07:55:31.610766+0000","last_became_active":"2024-04-02T07:55:31.603901+0000","last_became_peered":"2024-04-02T07:55:31.603901+0000","last_unstale":"2024-04-02T07:55:31.610766+0000","last_undegraded":"2024-04-02T07:55:31.610766+0000","last_fullsized":"2024-04-02T07:55:31.610766+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:26:40.691653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.602891+0000","last_change":"2024-04-02T07:55:31.597522+0000","last_active":"2024-04-02T07:55:31.602891+0000","last_peered":"2024-04-02T07:55:31.602891+0000","last_clean":"2024-04-02T07:55:31.602891+0000","last_became_active":"2024-04-02T07:55:31.597286+0000","last_became_peered":"2024-04-02T07:55:31.597286+0000","last_unstale":"2024-04-02T07:55:31.602891+0000","last_undegraded":"2024-04-02T07:55:31.602891+0000","last_fullsized":"2024-04-02T07:55:31.602891+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:00:44.301410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.593283+0000","last_change":"2024-04-02T07:55:31.591600+0000","last_active":"2024-04-02T07:55:31.593283+0000","last_peered":"2024-04-02T07:55:31.593283+0000","last_clean":"2024-04-02T07:55:31.593283+0000","last_became_active":"2024-04-02T07:55:31.591185+0000","last_became_peered":"2024-04-02T07:55:31.591185+0000","last_unstale":"2024-04-02T07:55:31.593283+0000","last_undegraded":"2024-04-02T07:55:31.593283+0000","last_fullsized":"2024-04-02T07:55:31.593283+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:16:03.608355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.609043+0000","last_change":"2024-04-02T07:55:31.602205+0000","last_active":"2024-04-02T07:55:31.609043+0000","last_peered":"2024-04-02T07:55:31.609043+0000","last_clean":"2024-04-02T07:55:31.609043+0000","last_became_active":"2024-04-02T07:55:31.599108+0000","last_became_peered":"2024-04-02T07:55:31.599108+0000","last_unstale":"2024-04-02T07:55:31.609043+0000","last_undegraded":"2024-04-02T07:55:31.609043+0000","last_fullsized":"2024-04-02T07:55:31.609043+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:36:31.822093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.598237+0000","last_change":"2024-04-02T07:55:31.597220+0000","last_active":"2024-04-02T07:55:31.598237+0000","last_peered":"2024-04-02T07:55:31.598237+0000","last_clean":"2024-04-02T07:55:31.598237+0000","last_became_active":"2024-04-02T07:55:31.596939+0000","last_became_peered":"2024-04-02T07:55:31.596939+0000","last_unstale":"2024-04-02T07:55:31.598237+0000","last_undegraded":"2024-04-02T07:55:31.598237+0000","last_fullsized":"2024-04-02T07:55:31.598237+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:49:10.266042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.604207+0000","last_change":"2024-04-02T07:55:31.599075+0000","last_active":"2024-04-02T07:55:31.604207+0000","last_peered":"2024-04-02T07:55:31.604207+0000","last_clean":"2024-04-02T07:55:31.604207+0000","last_became_active":"2024-04-02T07:55:31.597668+0000","last_became_peered":"2024-04-02T07:55:31.597668+0000","last_unstale":"2024-04-02T07:55:31.604207+0000","last_undegraded":"2024-04-02T07:55:31.604207+0000","last_fullsized":"2024-04-02T07:55:31.604207+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:57:26.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.593415+0000","last_change":"2024-04-02T07:55:31.589050+0000","last_active":"2024-04-02T07:55:31.593415+0000","last_peered":"2024-04-02T07:55:31.593415+0000","last_clean":"2024-04-02T07:55:31.593415+0000","last_became_active":"2024-04-02T07:55:31.588794+0000","last_became_peered":"2024-04-02T07:55:31.588794+0000","last_unstale":"2024-04-02T07:55:31.593415+0000","last_undegraded":"2024-04-02T07:55:31.593415+0000","last_fullsized":"2024-04-02T07:55:31.593415+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:45:05.161189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.606230+0000","last_change":"2024-04-02T07:55:31.602104+0000","last_active":"2024-04-02T07:55:31.606230+0000","last_peered":"2024-04-02T07:55:31.606230+0000","last_clean":"2024-04-02T07:55:31.606230+0000","last_became_active":"2024-04-02T07:55:31.601851+0000","last_became_peered":"2024-04-02T07:55:31.601851+0000","last_unstale":"2024-04-02T07:55:31.606230+0000","last_undegraded":"2024-04-02T07:55:31.606230+0000","last_fullsized":"2024-04-02T07:55:31.606230+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:26:40.691653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.606167+0000","last_change":"2024-04-02T07:55:31.601900+0000","last_active":"2024-04-02T07:55:31.606167+0000","last_peered":"2024-04-02T07:55:31.606167+0000","last_clean":"2024-04-02T07:55:31.606167+0000","last_became_active":"2024-04-02T07:55:31.601649+0000","last_became_peered":"2024-04-02T07:55:31.601649+0000","last_unstale":"2024-04-02T07:55:31.606167+0000","last_undegraded":"2024-04-02T07:55:31.606167+0000","last_fullsized":"2024-04-02T07:55:31.606167+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:48:15.120406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.601970+0000","last_change":"2024-04-02T07:55:31.599893+0000","last_active":"2024-04-02T07:55:31.601970+0000","last_peered":"2024-04-02T07:55:31.601970+0000","last_clean":"2024-04-02T07:55:31.601970+0000","last_became_active":"2024-04-02T07:55:31.599601+0000","last_became_peered":"2024-04-02T07:55:31.599601+0000","last_unstale":"2024-04-02T07:55:31.601970+0000","last_undegraded":"2024-04-02T07:55:31.601970+0000","last_fullsized":"2024-04-02T07:55:31.601970+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:24:56.353145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"139'4","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.607883+0000","last_change":"2024-04-02T07:55:31.094257+0000","last_active":"2024-04-02T07:55:31.607883+0000","last_peered":"2024-04-02T07:55:31.607883+0000","last_clean":"2024-04-02T07:55:31.607883+0000","last_became_active":"2024-04-02T07:55:31.093843+0000","last_became_peered":"2024-04-02T07:55:31.093843+0000","last_unstale":"2024-04-02T07:55:31.607883+0000","last_undegraded":"2024-04-02T07:55:31.607883+0000","last_fullsized":"2024-04-02T07:55:31.607883+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:07:50.769069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605899+0000","last_change":"2024-04-02T07:55:31.602503+0000","last_active":"2024-04-02T07:55:31.605899+0000","last_peered":"2024-04-02T07:55:31.605899+0000","last_clean":"2024-04-02T07:55:31.605899+0000","last_became_active":"2024-04-02T07:55:31.602265+0000","last_became_peered":"2024-04-02T07:55:31.602265+0000","last_unstale":"2024-04-02T07:55:31.605899+0000","last_undegraded":"2024-04-02T07:55:31.605899+0000","last_fullsized":"2024-04-02T07:55:31.605899+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:13.861818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1e","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.603571+0000","last_change":"2024-04-02T07:55:31.088483+0000","last_active":"2024-04-02T07:55:31.603571+0000","last_peered":"2024-04-02T07:55:31.603571+0000","last_clean":"2024-04-02T07:55:31.603571+0000","last_became_active":"2024-04-02T07:55:31.088242+0000","last_became_peered":"2024-04-02T07:55:31.088242+0000","last_unstale":"2024-04-02T07:55:31.603571+0000","last_undegraded":"2024-04-02T07:55:31.603571+0000","last_fullsized":"2024-04-02T07:55:31.603571+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:47:16.589118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.592958+0000","last_change":"2024-04-02T07:55:31.588766+0000","last_active":"2024-04-02T07:55:31.592958+0000","last_peered":"2024-04-02T07:55:31.592958+0000","last_clean":"2024-04-02T07:55:31.592958+0000","last_became_active":"2024-04-02T07:55:31.588384+0000","last_became_peered":"2024-04-02T07:55:31.588384+0000","last_unstale":"2024-04-02T07:55:31.592958+0000","last_undegraded":"2024-04-02T07:55:31.592958+0000","last_fullsized":"2024-04-02T07:55:31.592958+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T07:57:26.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1d","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.608654+0000","last_change":"2024-04-02T07:55:31.094859+0000","last_active":"2024-04-02T07:55:31.608654+0000","last_peered":"2024-04-02T07:55:31.608654+0000","last_clean":"2024-04-02T07:55:31.608654+0000","last_became_active":"2024-04-02T07:55:31.094585+0000","last_became_peered":"2024-04-02T07:55:31.094585+0000","last_unstale":"2024-04-02T07:55:31.608654+0000","last_undegraded":"2024-04-02T07:55:31.608654+0000","last_fullsized":"2024-04-02T07:55:31.608654+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:13:06.554691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.602629+0000","last_change":"2024-04-02T07:55:31.597100+0000","last_active":"2024-04-02T07:55:31.602629+0000","last_peered":"2024-04-02T07:55:31.602629+0000","last_clean":"2024-04-02T07:55:31.602629+0000","last_became_active":"2024-04-02T07:55:31.595358+0000","last_became_peered":"2024-04-02T07:55:31.595358+0000","last_unstale":"2024-04-02T07:55:31.602629+0000","last_undegraded":"2024-04-02T07:55:31.602629+0000","last_fullsized":"2024-04-02T07:55:31.602629+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:50:15.964841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.609130+0000","last_change":"2024-04-02T07:55:31.093302+0000","last_active":"2024-04-02T07:55:31.609130+0000","last_peered":"2024-04-02T07:55:31.609130+0000","last_clean":"2024-04-02T07:55:31.609130+0000","last_became_active":"2024-04-02T07:55:31.091025+0000","last_became_peered":"2024-04-02T07:55:31.091025+0000","last_unstale":"2024-04-02T07:55:31.609130+0000","last_undegraded":"2024-04-02T07:55:31.609130+0000","last_fullsized":"2024-04-02T07:55:31.609130+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:46.618508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.610727+0000","last_change":"2024-04-02T07:55:31.604592+0000","last_active":"2024-04-02T07:55:31.610727+0000","last_peered":"2024-04-02T07:55:31.610727+0000","last_clean":"2024-04-02T07:55:31.610727+0000","last_became_active":"2024-04-02T07:55:31.603197+0000","last_became_peered":"2024-04-02T07:55:31.603197+0000","last_unstale":"2024-04-02T07:55:31.610727+0000","last_undegraded":"2024-04-02T07:55:31.610727+0000","last_fullsized":"2024-04-02T07:55:31.610727+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:32:23.897820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1b","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.603082+0000","last_change":"2024-04-02T07:55:31.084659+0000","last_active":"2024-04-02T07:55:31.603082+0000","last_peered":"2024-04-02T07:55:31.603082+0000","last_clean":"2024-04-02T07:55:31.603082+0000","last_became_active":"2024-04-02T07:55:31.083757+0000","last_became_peered":"2024-04-02T07:55:31.083757+0000","last_unstale":"2024-04-02T07:55:31.603082+0000","last_undegraded":"2024-04-02T07:55:31.603082+0000","last_fullsized":"2024-04-02T07:55:31.603082+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:47:16.589118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.603694+0000","last_change":"2024-04-02T07:55:31.601478+0000","last_active":"2024-04-02T07:55:31.603694+0000","last_peered":"2024-04-02T07:55:31.603694+0000","last_clean":"2024-04-02T07:55:31.603694+0000","last_became_active":"2024-04-02T07:55:31.601237+0000","last_became_peered":"2024-04-02T07:55:31.601237+0000","last_unstale":"2024-04-02T07:55:31.603694+0000","last_undegraded":"2024-04-02T07:55:31.603694+0000","last_fullsized":"2024-04-02T07:55:31.603694+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:52:38.844963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.606307+0000","last_change":"2024-04-02T07:55:31.094674+0000","last_active":"2024-04-02T07:55:31.606307+0000","last_peered":"2024-04-02T07:55:31.606307+0000","last_clean":"2024-04-02T07:55:31.606307+0000","last_became_active":"2024-04-02T07:55:31.094215+0000","last_became_peered":"2024-04-02T07:55:31.094215+0000","last_unstale":"2024-04-02T07:55:31.606307+0000","last_undegraded":"2024-04-02T07:55:31.606307+0000","last_fullsized":"2024-04-02T07:55:31.606307+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:48:14.112628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.607633+0000","last_change":"2024-04-02T07:55:31.604907+0000","last_active":"2024-04-02T07:55:31.607633+0000","last_peered":"2024-04-02T07:55:31.607633+0000","last_clean":"2024-04-02T07:55:31.607633+0000","last_became_active":"2024-04-02T07:55:31.602161+0000","last_became_peered":"2024-04-02T07:55:31.602161+0000","last_unstale":"2024-04-02T07:55:31.607633+0000","last_undegraded":"2024-04-02T07:55:31.607633+0000","last_fullsized":"2024-04-02T07:55:31.607633+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:05:45.742380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.586906+0000","last_change":"2024-04-02T07:55:31.090631+0000","last_active":"2024-04-02T07:55:31.586906+0000","last_peered":"2024-04-02T07:55:31.586906+0000","last_clean":"2024-04-02T07:55:31.586906+0000","last_became_active":"2024-04-02T07:55:31.089242+0000","last_became_peered":"2024-04-02T07:55:31.089242+0000","last_unstale":"2024-04-02T07:55:31.586906+0000","last_undegraded":"2024-04-02T07:55:31.586906+0000","last_fullsized":"2024-04-02T07:55:31.586906+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:45:35.627867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.594814+0000","last_change":"2024-04-02T07:55:31.591684+0000","last_active":"2024-04-02T07:55:31.594814+0000","last_peered":"2024-04-02T07:55:31.594814+0000","last_clean":"2024-04-02T07:55:31.594814+0000","last_became_active":"2024-04-02T07:55:31.591419+0000","last_became_peered":"2024-04-02T07:55:31.591419+0000","last_unstale":"2024-04-02T07:55:31.594814+0000","last_undegraded":"2024-04-02T07:55:31.594814+0000","last_fullsized":"2024-04-02T07:55:31.594814+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:56:56.050365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.582889+0000","last_change":"2024-04-02T07:55:31.096341+0000","last_active":"2024-04-02T07:55:31.582889+0000","last_peered":"2024-04-02T07:55:31.582889+0000","last_clean":"2024-04-02T07:55:31.582889+0000","last_became_active":"2024-04-02T07:55:31.096037+0000","last_became_peered":"2024-04-02T07:55:31.096037+0000","last_unstale":"2024-04-02T07:55:31.582889+0000","last_undegraded":"2024-04-02T07:55:31.582889+0000","last_fullsized":"2024-04-02T07:55:31.582889+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:08:36.938918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.596124+0000","last_change":"2024-04-02T07:55:31.591858+0000","last_active":"2024-04-02T07:55:31.596124+0000","last_peered":"2024-04-02T07:55:31.596124+0000","last_clean":"2024-04-02T07:55:31.596124+0000","last_became_active":"2024-04-02T07:55:31.590772+0000","last_became_peered":"2024-04-02T07:55:31.590772+0000","last_unstale":"2024-04-02T07:55:31.596124+0000","last_undegraded":"2024-04-02T07:55:31.596124+0000","last_fullsized":"2024-04-02T07:55:31.596124+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:48:20.542068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.598384+0000","last_change":"2024-04-02T07:55:31.095060+0000","last_active":"2024-04-02T07:55:31.598384+0000","last_peered":"2024-04-02T07:55:31.598384+0000","last_clean":"2024-04-02T07:55:31.598384+0000","last_became_active":"2024-04-02T07:55:31.094769+0000","last_became_peered":"2024-04-02T07:55:31.094769+0000","last_unstale":"2024-04-02T07:55:31.598384+0000","last_undegraded":"2024-04-02T07:55:31.598384+0000","last_fullsized":"2024-04-02T07:55:31.598384+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:11:14.811238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.596093+0000","last_change":"2024-04-02T07:55:31.592762+0000","last_active":"2024-04-02T07:55:31.596093+0000","last_peered":"2024-04-02T07:55:31.596093+0000","last_clean":"2024-04-02T07:55:31.596093+0000","last_became_active":"2024-04-02T07:55:31.592275+0000","last_became_peered":"2024-04-02T07:55:31.592275+0000","last_unstale":"2024-04-02T07:55:31.596093+0000","last_undegraded":"2024-04-02T07:55:31.596093+0000","last_fullsized":"2024-04-02T07:55:31.596093+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:36:21.155941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.583258+0000","last_change":"2024-04-02T07:55:31.094042+0000","last_active":"2024-04-02T07:55:31.583258+0000","last_peered":"2024-04-02T07:55:31.583258+0000","last_clean":"2024-04-02T07:55:31.583258+0000","last_became_active":"2024-04-02T07:55:31.093747+0000","last_became_peered":"2024-04-02T07:55:31.093747+0000","last_unstale":"2024-04-02T07:55:31.583258+0000","last_undegraded":"2024-04-02T07:55:31.583258+0000","last_fullsized":"2024-04-02T07:55:31.583258+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:40.125366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.585937+0000","last_change":"2024-04-02T07:55:31.584753+0000","last_active":"2024-04-02T07:55:31.585937+0000","last_peered":"2024-04-02T07:55:31.585937+0000","last_clean":"2024-04-02T07:55:31.585937+0000","last_became_active":"2024-04-02T07:55:31.584418+0000","last_became_peered":"2024-04-02T07:55:31.584418+0000","last_unstale":"2024-04-02T07:55:31.585937+0000","last_undegraded":"2024-04-02T07:55:31.585937+0000","last_fullsized":"2024-04-02T07:55:31.585937+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:21:18.710733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.15","version":"139'3","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.611044+0000","last_change":"2024-04-02T07:55:31.098639+0000","last_active":"2024-04-02T07:55:31.611044+0000","last_peered":"2024-04-02T07:55:31.611044+0000","last_clean":"2024-04-02T07:55:31.611044+0000","last_became_active":"2024-04-02T07:55:31.098308+0000","last_became_peered":"2024-04-02T07:55:31.098308+0000","last_unstale":"2024-04-02T07:55:31.611044+0000","last_undegraded":"2024-04-02T07:55:31.611044+0000","last_fullsized":"2024-04-02T07:55:31.611044+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:22:28.833417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.602484+0000","last_change":"2024-04-02T07:55:31.596850+0000","last_active":"2024-04-02T07:55:31.602484+0000","last_peered":"2024-04-02T07:55:31.602484+0000","last_clean":"2024-04-02T07:55:31.602484+0000","last_became_active":"2024-04-02T07:55:31.596585+0000","last_became_peered":"2024-04-02T07:55:31.596585+0000","last_unstale":"2024-04-02T07:55:31.602484+0000","last_undegraded":"2024-04-02T07:55:31.602484+0000","last_fullsized":"2024-04-02T07:55:31.602484+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:31:25.345799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"139'6","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.613370+0000","last_change":"2024-04-02T07:55:31.093614+0000","last_active":"2024-04-02T07:55:31.613370+0000","last_peered":"2024-04-02T07:55:31.613370+0000","last_clean":"2024-04-02T07:55:31.613370+0000","last_became_active":"2024-04-02T07:55:31.093295+0000","last_became_peered":"2024-04-02T07:55:31.093295+0000","last_unstale":"2024-04-02T07:55:31.613370+0000","last_undegraded":"2024-04-02T07:55:31.613370+0000","last_fullsized":"2024-04-02T07:55:31.613370+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:02:06.434801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.600040+0000","last_change":"2024-04-02T07:55:31.595431+0000","last_active":"2024-04-02T07:55:31.600040+0000","last_peered":"2024-04-02T07:55:31.600040+0000","last_clean":"2024-04-02T07:55:31.600040+0000","last_became_active":"2024-04-02T07:55:31.594193+0000","last_became_peered":"2024-04-02T07:55:31.594193+0000","last_unstale":"2024-04-02T07:55:31.600040+0000","last_undegraded":"2024-04-02T07:55:31.600040+0000","last_fullsized":"2024-04-02T07:55:31.600040+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:36:21.155941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.610061+0000","last_change":"2024-04-02T07:55:31.094912+0000","last_active":"2024-04-02T07:55:31.610061+0000","last_peered":"2024-04-02T07:55:31.610061+0000","last_clean":"2024-04-02T07:55:31.610061+0000","last_became_active":"2024-04-02T07:55:31.094666+0000","last_became_peered":"2024-04-02T07:55:31.094666+0000","last_unstale":"2024-04-02T07:55:31.610061+0000","last_undegraded":"2024-04-02T07:55:31.610061+0000","last_fullsized":"2024-04-02T07:55:31.610061+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:58:01.791478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.609104+0000","last_change":"2024-04-02T07:55:31.600576+0000","last_active":"2024-04-02T07:55:31.609104+0000","last_peered":"2024-04-02T07:55:31.609104+0000","last_clean":"2024-04-02T07:55:31.609104+0000","last_became_active":"2024-04-02T07:55:31.600239+0000","last_became_peered":"2024-04-02T07:55:31.600239+0000","last_unstale":"2024-04-02T07:55:31.609104+0000","last_undegraded":"2024-04-02T07:55:31.609104+0000","last_fullsized":"2024-04-02T07:55:31.609104+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:10:09.268501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.582982+0000","last_change":"2024-04-02T07:55:31.085739+0000","last_active":"2024-04-02T07:55:31.582982+0000","last_peered":"2024-04-02T07:55:31.582982+0000","last_clean":"2024-04-02T07:55:31.582982+0000","last_became_active":"2024-04-02T07:55:31.084489+0000","last_became_peered":"2024-04-02T07:55:31.084489+0000","last_unstale":"2024-04-02T07:55:31.582982+0000","last_undegraded":"2024-04-02T07:55:31.582982+0000","last_fullsized":"2024-04-02T07:55:31.582982+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:58:33.545600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.606691+0000","last_change":"2024-04-02T07:55:31.594685+0000","last_active":"2024-04-02T07:55:31.606691+0000","last_peered":"2024-04-02T07:55:31.606691+0000","last_clean":"2024-04-02T07:55:31.606691+0000","last_became_active":"2024-04-02T07:55:31.594435+0000","last_became_peered":"2024-04-02T07:55:31.594435+0000","last_unstale":"2024-04-02T07:55:31.606691+0000","last_undegraded":"2024-04-02T07:55:31.606691+0000","last_fullsized":"2024-04-02T07:55:31.606691+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:54:15.998051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"139'8","reported_seq":18,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.608180+0000","last_change":"2024-04-02T07:55:31.096347+0000","last_active":"2024-04-02T07:55:31.608180+0000","last_peered":"2024-04-02T07:55:31.608180+0000","last_clean":"2024-04-02T07:55:31.608180+0000","last_became_active":"2024-04-02T07:55:31.095739+0000","last_became_peered":"2024-04-02T07:55:31.095739+0000","last_unstale":"2024-04-02T07:55:31.608180+0000","last_undegraded":"2024-04-02T07:55:31.608180+0000","last_fullsized":"2024-04-02T07:55:31.608180+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:48:19.534289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.597735+0000","last_change":"2024-04-02T07:55:31.589081+0000","last_active":"2024-04-02T07:55:31.597735+0000","last_peered":"2024-04-02T07:55:31.597735+0000","last_clean":"2024-04-02T07:55:31.597735+0000","last_became_active":"2024-04-02T07:55:31.588764+0000","last_became_peered":"2024-04-02T07:55:31.588764+0000","last_unstale":"2024-04-02T07:55:31.597735+0000","last_undegraded":"2024-04-02T07:55:31.597735+0000","last_fullsized":"2024-04-02T07:55:31.597735+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:44:06.419071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.f","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.606341+0000","last_change":"2024-04-02T07:55:31.087654+0000","last_active":"2024-04-02T07:55:31.606341+0000","last_peered":"2024-04-02T07:55:31.606341+0000","last_clean":"2024-04-02T07:55:31.606341+0000","last_became_active":"2024-04-02T07:55:31.087292+0000","last_became_peered":"2024-04-02T07:55:31.087292+0000","last_unstale":"2024-04-02T07:55:31.606341+0000","last_undegraded":"2024-04-02T07:55:31.606341+0000","last_fullsized":"2024-04-02T07:55:31.606341+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:54:14.990272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.600632+0000","last_change":"2024-04-02T07:55:31.596098+0000","last_active":"2024-04-02T07:55:31.600632+0000","last_peered":"2024-04-02T07:55:31.600632+0000","last_clean":"2024-04-02T07:55:31.600632+0000","last_became_active":"2024-04-02T07:55:31.594047+0000","last_became_peered":"2024-04-02T07:55:31.594047+0000","last_unstale":"2024-04-02T07:55:31.600632+0000","last_undegraded":"2024-04-02T07:55:31.600632+0000","last_fullsized":"2024-04-02T07:55:31.600632+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:55:29.860904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.e","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.608662+0000","last_change":"2024-04-02T07:55:31.098430+0000","last_active":"2024-04-02T07:55:31.608662+0000","last_peered":"2024-04-02T07:55:31.608662+0000","last_clean":"2024-04-02T07:55:31.608662+0000","last_became_active":"2024-04-02T07:55:31.097972+0000","last_became_peered":"2024-04-02T07:55:31.097972+0000","last_unstale":"2024-04-02T07:55:31.608662+0000","last_undegraded":"2024-04-02T07:55:31.608662+0000","last_fullsized":"2024-04-02T07:55:31.608662+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:35:14.025296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.594950+0000","last_change":"2024-04-02T07:55:31.591539+0000","last_active":"2024-04-02T07:55:31.594950+0000","last_peered":"2024-04-02T07:55:31.594950+0000","last_clean":"2024-04-02T07:55:31.594950+0000","last_became_active":"2024-04-02T07:55:31.591259+0000","last_became_peered":"2024-04-02T07:55:31.591259+0000","last_unstale":"2024-04-02T07:55:31.594950+0000","last_undegraded":"2024-04-02T07:55:31.594950+0000","last_fullsized":"2024-04-02T07:55:31.594950+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:02.593914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"139'1","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605795+0000","last_change":"2024-04-02T07:55:31.091627+0000","last_active":"2024-04-02T07:55:31.605795+0000","last_peered":"2024-04-02T07:55:31.605795+0000","last_clean":"2024-04-02T07:55:31.605795+0000","last_became_active":"2024-04-02T07:55:31.091371+0000","last_became_peered":"2024-04-02T07:55:31.091371+0000","last_unstale":"2024-04-02T07:55:31.605795+0000","last_undegraded":"2024-04-02T07:55:31.605795+0000","last_fullsized":"2024-04-02T07:55:31.605795+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:54.255209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605663+0000","last_change":"2024-04-02T07:55:31.598883+0000","last_active":"2024-04-02T07:55:31.605663+0000","last_peered":"2024-04-02T07:55:31.605663+0000","last_clean":"2024-04-02T07:55:31.605663+0000","last_became_active":"2024-04-02T07:55:31.598338+0000","last_became_peered":"2024-04-02T07:55:31.598338+0000","last_unstale":"2024-04-02T07:55:31.605663+0000","last_undegraded":"2024-04-02T07:55:31.605663+0000","last_fullsized":"2024-04-02T07:55:31.605663+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:38.406238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.591313+0000","last_change":"2024-04-02T07:55:31.096067+0000","last_active":"2024-04-02T07:55:31.591313+0000","last_peered":"2024-04-02T07:55:31.591313+0000","last_clean":"2024-04-02T07:55:31.591313+0000","last_became_active":"2024-04-02T07:55:31.095770+0000","last_became_peered":"2024-04-02T07:55:31.095770+0000","last_unstale":"2024-04-02T07:55:31.591313+0000","last_undegraded":"2024-04-02T07:55:31.591313+0000","last_fullsized":"2024-04-02T07:55:31.591313+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:26:39.683874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.593747+0000","last_change":"2024-04-02T07:55:31.590071+0000","last_active":"2024-04-02T07:55:31.593747+0000","last_peered":"2024-04-02T07:55:31.593747+0000","last_clean":"2024-04-02T07:55:31.593747+0000","last_became_active":"2024-04-02T07:55:31.589540+0000","last_became_peered":"2024-04-02T07:55:31.589540+0000","last_unstale":"2024-04-02T07:55:31.593747+0000","last_undegraded":"2024-04-02T07:55:31.593747+0000","last_fullsized":"2024-04-02T07:55:31.593747+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:36:26.431812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.607795+0000","last_change":"2024-04-02T07:55:31.095524+0000","last_active":"2024-04-02T07:55:31.607795+0000","last_peered":"2024-04-02T07:55:31.607795+0000","last_clean":"2024-04-02T07:55:31.607795+0000","last_became_active":"2024-04-02T07:55:31.095273+0000","last_became_peered":"2024-04-02T07:55:31.095273+0000","last_unstale":"2024-04-02T07:55:31.607795+0000","last_undegraded":"2024-04-02T07:55:31.607795+0000","last_fullsized":"2024-04-02T07:55:31.607795+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:14:07.054260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.598228+0000","last_change":"2024-04-02T07:55:31.594315+0000","last_active":"2024-04-02T07:55:31.598228+0000","last_peered":"2024-04-02T07:55:31.598228+0000","last_clean":"2024-04-02T07:55:31.598228+0000","last_became_active":"2024-04-02T07:55:31.594052+0000","last_became_peered":"2024-04-02T07:55:31.594052+0000","last_unstale":"2024-04-02T07:55:31.598228+0000","last_undegraded":"2024-04-02T07:55:31.598228+0000","last_fullsized":"2024-04-02T07:55:31.598228+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:06:19.358325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.579780+0000","last_change":"2024-04-02T07:55:31.099701+0000","last_active":"2024-04-02T07:55:31.579780+0000","last_peered":"2024-04-02T07:55:31.579780+0000","last_clean":"2024-04-02T07:55:31.579780+0000","last_became_active":"2024-04-02T07:55:31.099109+0000","last_became_peered":"2024-04-02T07:55:31.099109+0000","last_unstale":"2024-04-02T07:55:31.579780+0000","last_undegraded":"2024-04-02T07:55:31.579780+0000","last_fullsized":"2024-04-02T07:55:31.579780+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:14.284231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.604220+0000","last_change":"2024-04-02T07:55:31.596467+0000","last_active":"2024-04-02T07:55:31.604220+0000","last_peered":"2024-04-02T07:55:31.604220+0000","last_clean":"2024-04-02T07:55:31.604220+0000","last_became_active":"2024-04-02T07:55:31.596169+0000","last_became_peered":"2024-04-02T07:55:31.596169+0000","last_unstale":"2024-04-02T07:55:31.604220+0000","last_undegraded":"2024-04-02T07:55:31.604220+0000","last_fullsized":"2024-04-02T07:55:31.604220+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:44:37.486456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.593209+0000","last_change":"2024-04-02T07:55:31.096911+0000","last_active":"2024-04-02T07:55:31.593209+0000","last_peered":"2024-04-02T07:55:31.593209+0000","last_clean":"2024-04-02T07:55:31.593209+0000","last_became_active":"2024-04-02T07:55:31.096516+0000","last_became_peered":"2024-04-02T07:55:31.096516+0000","last_unstale":"2024-04-02T07:55:31.593209+0000","last_undegraded":"2024-04-02T07:55:31.593209+0000","last_fullsized":"2024-04-02T07:55:31.593209+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:42:21.973647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.601869+0000","last_change":"2024-04-02T07:55:31.596758+0000","last_active":"2024-04-02T07:55:31.601869+0000","last_peered":"2024-04-02T07:55:31.601869+0000","last_clean":"2024-04-02T07:55:31.601869+0000","last_became_active":"2024-04-02T07:55:31.596021+0000","last_became_peered":"2024-04-02T07:55:31.596021+0000","last_unstale":"2024-04-02T07:55:31.601869+0000","last_undegraded":"2024-04-02T07:55:31.601869+0000","last_fullsized":"2024-04-02T07:55:31.601869+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:36:26.431812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.592778+0000","last_change":"2024-04-02T07:55:31.084846+0000","last_active":"2024-04-02T07:55:31.592778+0000","last_peered":"2024-04-02T07:55:31.592778+0000","last_clean":"2024-04-02T07:55:31.592778+0000","last_became_active":"2024-04-02T07:55:31.084273+0000","last_became_peered":"2024-04-02T07:55:31.084273+0000","last_unstale":"2024-04-02T07:55:31.592778+0000","last_undegraded":"2024-04-02T07:55:31.592778+0000","last_fullsized":"2024-04-02T07:55:31.592778+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:50:14.957062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605903+0000","last_change":"2024-04-02T07:55:31.601740+0000","last_active":"2024-04-02T07:55:31.605903+0000","last_peered":"2024-04-02T07:55:31.605903+0000","last_clean":"2024-04-02T07:55:31.605903+0000","last_became_active":"2024-04-02T07:55:31.601484+0000","last_became_peered":"2024-04-02T07:55:31.601484+0000","last_unstale":"2024-04-02T07:55:31.605903+0000","last_undegraded":"2024-04-02T07:55:31.605903+0000","last_fullsized":"2024-04-02T07:55:31.605903+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:16:03.608355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.7","version":"139'2","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.610760+0000","last_change":"2024-04-02T07:55:31.091390+0000","last_active":"2024-04-02T07:55:31.610760+0000","last_peered":"2024-04-02T07:55:31.610760+0000","last_clean":"2024-04-02T07:55:31.610760+0000","last_became_active":"2024-04-02T07:55:31.091115+0000","last_became_peered":"2024-04-02T07:55:31.091115+0000","last_unstale":"2024-04-02T07:55:31.610760+0000","last_undegraded":"2024-04-02T07:55:31.610760+0000","last_fullsized":"2024-04-02T07:55:31.610760+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:32:05.622480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.598084+0000","last_change":"2024-04-02T07:55:31.592775+0000","last_active":"2024-04-02T07:55:31.598084+0000","last_peered":"2024-04-02T07:55:31.598084+0000","last_clean":"2024-04-02T07:55:31.598084+0000","last_became_active":"2024-04-02T07:55:31.590036+0000","last_became_peered":"2024-04-02T07:55:31.590036+0000","last_unstale":"2024-04-02T07:55:31.598084+0000","last_undegraded":"2024-04-02T07:55:31.598084+0000","last_fullsized":"2024-04-02T07:55:31.598084+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:34:54.181448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.587947+0000","last_change":"2024-04-02T07:55:31.086115+0000","last_active":"2024-04-02T07:55:31.587947+0000","last_peered":"2024-04-02T07:55:31.587947+0000","last_clean":"2024-04-02T07:55:31.587947+0000","last_became_active":"2024-04-02T07:55:31.085548+0000","last_became_peered":"2024-04-02T07:55:31.085548+0000","last_unstale":"2024-04-02T07:55:31.587947+0000","last_undegraded":"2024-04-02T07:55:31.587947+0000","last_fullsized":"2024-04-02T07:55:31.587947+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:37:34.798163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.599734+0000","last_change":"2024-04-02T07:55:31.588374+0000","last_active":"2024-04-02T07:55:31.599734+0000","last_peered":"2024-04-02T07:55:31.599734+0000","last_clean":"2024-04-02T07:55:31.599734+0000","last_became_active":"2024-04-02T07:55:31.588077+0000","last_became_peered":"2024-04-02T07:55:31.588077+0000","last_unstale":"2024-04-02T07:55:31.599734+0000","last_undegraded":"2024-04-02T07:55:31.599734+0000","last_fullsized":"2024-04-02T07:55:31.599734+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:35:52.948729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.596579+0000","last_change":"2024-04-02T07:55:31.082634+0000","last_active":"2024-04-02T07:55:31.596579+0000","last_peered":"2024-04-02T07:55:31.596579+0000","last_clean":"2024-04-02T07:55:31.596579+0000","last_became_active":"2024-04-02T07:55:31.082321+0000","last_became_peered":"2024-04-02T07:55:31.082321+0000","last_unstale":"2024-04-02T07:55:31.596579+0000","last_undegraded":"2024-04-02T07:55:31.596579+0000","last_fullsized":"2024-04-02T07:55:31.596579+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:36:25.424033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.605691+0000","last_change":"2024-04-02T07:55:31.599385+0000","last_active":"2024-04-02T07:55:31.605691+0000","last_peered":"2024-04-02T07:55:31.605691+0000","last_clean":"2024-04-02T07:55:31.605691+0000","last_became_active":"2024-04-02T07:55:31.598037+0000","last_became_peered":"2024-04-02T07:55:31.598037+0000","last_unstale":"2024-04-02T07:55:31.605691+0000","last_undegraded":"2024-04-02T07:55:31.605691+0000","last_fullsized":"2024-04-02T07:55:31.605691+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:40.274339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.595618+0000","last_change":"2024-04-02T07:55:31.088474+0000","last_active":"2024-04-02T07:55:31.595618+0000","last_peered":"2024-04-02T07:55:31.595618+0000","last_clean":"2024-04-02T07:55:31.595618+0000","last_became_active":"2024-04-02T07:55:31.088223+0000","last_became_peered":"2024-04-02T07:55:31.088223+0000","last_unstale":"2024-04-02T07:55:31.595618+0000","last_undegraded":"2024-04-02T07:55:31.595618+0000","last_fullsized":"2024-04-02T07:55:31.595618+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:05:44.734602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.598328+0000","last_change":"2024-04-02T07:55:31.593370+0000","last_active":"2024-04-02T07:55:31.598328+0000","last_peered":"2024-04-02T07:55:31.598328+0000","last_clean":"2024-04-02T07:55:31.598328+0000","last_became_active":"2024-04-02T07:55:31.593128+0000","last_became_peered":"2024-04-02T07:55:31.593128+0000","last_unstale":"2024-04-02T07:55:31.598328+0000","last_undegraded":"2024-04-02T07:55:31.598328+0000","last_fullsized":"2024-04-02T07:55:31.598328+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:48:20.542068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.587273+0000","last_change":"2024-04-02T07:55:31.091389+0000","last_active":"2024-04-02T07:55:31.587273+0000","last_peered":"2024-04-02T07:55:31.587273+0000","last_clean":"2024-04-02T07:55:31.587273+0000","last_became_active":"2024-04-02T07:55:31.090908+0000","last_became_peered":"2024-04-02T07:55:31.090908+0000","last_unstale":"2024-04-02T07:55:31.587273+0000","last_undegraded":"2024-04-02T07:55:31.587273+0000","last_fullsized":"2024-04-02T07:55:31.587273+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:37:39.394344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.604234+0000","last_change":"2024-04-02T07:55:31.599657+0000","last_active":"2024-04-02T07:55:31.604234+0000","last_peered":"2024-04-02T07:55:31.604234+0000","last_clean":"2024-04-02T07:55:31.604234+0000","last_became_active":"2024-04-02T07:55:31.599390+0000","last_became_peered":"2024-04-02T07:55:31.599390+0000","last_unstale":"2024-04-02T07:55:31.604234+0000","last_undegraded":"2024-04-02T07:55:31.604234+0000","last_fullsized":"2024-04-02T07:55:31.604234+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:49:10.266042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.590744+0000","last_change":"2024-04-02T07:55:31.095182+0000","last_active":"2024-04-02T07:55:31.590744+0000","last_peered":"2024-04-02T07:55:31.590744+0000","last_clean":"2024-04-02T07:55:31.590744+0000","last_became_active":"2024-04-02T07:55:31.094864+0000","last_became_peered":"2024-04-02T07:55:31.094864+0000","last_unstale":"2024-04-02T07:55:31.590744+0000","last_undegraded":"2024-04-02T07:55:31.590744+0000","last_fullsized":"2024-04-02T07:55:31.590744+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:44:05.411292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.604197+0000","last_change":"2024-04-02T07:55:31.595253+0000","last_active":"2024-04-02T07:55:31.604197+0000","last_peered":"2024-04-02T07:55:31.604197+0000","last_clean":"2024-04-02T07:55:31.604197+0000","last_became_active":"2024-04-02T07:55:31.595001+0000","last_became_peered":"2024-04-02T07:55:31.595001+0000","last_unstale":"2024-04-02T07:55:31.604197+0000","last_undegraded":"2024-04-02T07:55:31.604197+0000","last_fullsized":"2024-04-02T07:55:31.604197+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:31:33.770811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.578593+0000","last_change":"2024-04-02T07:55:31.093524+0000","last_active":"2024-04-02T07:55:31.578593+0000","last_peered":"2024-04-02T07:55:31.578593+0000","last_clean":"2024-04-02T07:55:31.578593+0000","last_became_active":"2024-04-02T07:55:31.093258+0000","last_became_peered":"2024-04-02T07:55:31.093258+0000","last_unstale":"2024-04-02T07:55:31.578593+0000","last_undegraded":"2024-04-02T07:55:31.578593+0000","last_fullsized":"2024-04-02T07:55:31.578593+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:36:20.148162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.585504+0000","last_change":"2024-04-02T07:55:31.583678+0000","last_active":"2024-04-02T07:55:31.585504+0000","last_peered":"2024-04-02T07:55:31.585504+0000","last_clean":"2024-04-02T07:55:31.585504+0000","last_became_active":"2024-04-02T07:55:31.583411+0000","last_became_peered":"2024-04-02T07:55:31.583411+0000","last_unstale":"2024-04-02T07:55:31.585504+0000","last_undegraded":"2024-04-02T07:55:31.585504+0000","last_fullsized":"2024-04-02T07:55:31.585504+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:13.861818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.577289+0000","last_change":"2024-04-02T07:55:31.087591+0000","last_active":"2024-04-02T07:55:31.577289+0000","last_peered":"2024-04-02T07:55:31.577289+0000","last_clean":"2024-04-02T07:55:31.577289+0000","last_became_active":"2024-04-02T07:55:31.087278+0000","last_became_peered":"2024-04-02T07:55:31.087278+0000","last_unstale":"2024-04-02T07:55:31.577289+0000","last_undegraded":"2024-04-02T07:55:31.577289+0000","last_fullsized":"2024-04-02T07:55:31.577289+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:10:08.260722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.606809+0000","last_change":"2024-04-02T07:55:31.596742+0000","last_active":"2024-04-02T07:55:31.606809+0000","last_peered":"2024-04-02T07:55:31.606809+0000","last_clean":"2024-04-02T07:55:31.606809+0000","last_became_active":"2024-04-02T07:55:31.596422+0000","last_became_peered":"2024-04-02T07:55:31.596422+0000","last_unstale":"2024-04-02T07:55:31.606809+0000","last_undegraded":"2024-04-02T07:55:31.606809+0000","last_fullsized":"2024-04-02T07:55:31.606809+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:31.067219+0000","last_clean_scrub_stamp":"2024-04-02T07:55:31.067219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:55.262988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":10,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.585327+0000","last_change":"2024-04-02T07:55:31.096221+0000","last_active":"2024-04-02T07:55:31.585327+0000","last_peered":"2024-04-02T07:55:31.585327+0000","last_clean":"2024-04-02T07:55:31.585327+0000","last_became_active":"2024-04-02T07:55:31.095900+0000","last_became_peered":"2024-04-02T07:55:31.095900+0000","last_unstale":"2024-04-02T07:55:31.585327+0000","last_undegraded":"2024-04-02T07:55:31.585327+0000","last_fullsized":"2024-04-02T07:55:31.585327+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:55:30.059440+0000","last_clean_scrub_stamp":"2024-04-02T07:55:30.059440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:23:05.365858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"129'227","reported_seq":669,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-02T07:55:31.076505+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:55:31.076505+0000","last_peered":"2024-04-02T07:55:31.076505+0000","last_clean":"2024-04-02T07:55:31.076505+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:55:31.076505+0000","last_undegraded":"2024-04-02T07:55:31.076505+0000","last_fullsized":"2024-04-02T07:55:31.076505+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575077,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32824,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339016,"statfs":{"total":96636764160,"available":96603152384,"internally_reserved":0,"allocated":1744896,"data_stored":855628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63200000000000001,"15min":0.66500000000000004},"min":{"1min":0.38600000000000001,"5min":0.34699999999999998,"15min":0.316},"max":{"1min":0.80700000000000005,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.41499999999999998,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.53400000000000003}]},{"osd":1,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65100000000000002,"15min":0.67500000000000004},"min":{"1min":0.48699999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88100000000000001,"5min":1.097,"15min":1.097},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63800000000000001,"15min":0.66500000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.34699999999999998},"max":{"1min":0.78000000000000003,"5min":0.98299999999999998,"15min":1.2190000000000001},"last":0.55700000000000005}]},{"osd":2,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66200000000000003,"15min":0.69899999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.34000000000000002},"max":{"1min":0.88700000000000001,"5min":0.98999999999999999,"15min":1.127},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.72499999999999998,"15min":0.75900000000000001},"min":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95299999999999996,"5min":1.044,"15min":1.1259999999999999},"last":0.58499999999999996}]},{"osd":3,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.69899999999999995},"min":{"1min":0.45800000000000002,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.94199999999999995,"5min":0.97799999999999998,"15min":1.026},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68200000000000005,"15min":0.71299999999999997},"min":{"1min":0.52700000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.83399999999999996,"5min":1.071,"15min":1.196},"last":0.63}]},{"osd":4,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58799999999999997},"min":{"1min":0.377,"5min":0.35799999999999998,"15min":0.32000000000000001},"max":{"1min":0.85899999999999999,"5min":1.0229999999999999,"15min":1.0389999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68700000000000006,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.004,"5min":1.016,"15min":1.0669999999999999},"last":0.73899999999999999}]},{"osd":5,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.64400000000000002,"15min":0.65000000000000002},"min":{"1min":0.45500000000000002,"5min":0.42999999999999999,"15min":0.37},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.167},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.022,"5min":1.343,"15min":1.4219999999999999},"last":0.77900000000000003}]},{"osd":6,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.36199999999999999},"max":{"1min":0.92000000000000004,"5min":1.0580000000000001,"15min":2.5369999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.66600000000000004,"15min":0.66000000000000003},"min":{"1min":0.48399999999999999,"5min":0.41299999999999998,"15min":0.39600000000000002},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.70999999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575075,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36904,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334936,"statfs":{"total":96636764160,"available":96598974464,"internally_reserved":0,"allocated":1724416,"data_stored":853940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.67000000000000004,"15min":0.70099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.33400000000000002},"max":{"1min":0.89000000000000001,"5min":0.95899999999999996,"15min":0.98099999999999998},"last":0.628},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66700000000000004,"15min":0.70399999999999996},"min":{"1min":0.46200000000000002,"5min":0.40500000000000003,"15min":0.38300000000000001},"max":{"1min":0.88600000000000001,"5min":0.94299999999999995,"15min":1.0249999999999999},"last":0.64600000000000002}]},{"osd":1,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67000000000000004,"15min":0.69199999999999995},"min":{"1min":0.505,"5min":0.433,"15min":0.41799999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69899999999999995,"15min":0.71499999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":0.95799999999999996,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.50900000000000001}]},{"osd":2,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68000000000000005,"15min":0.70999999999999996},"min":{"1min":0.46500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68899999999999995,"15min":0.70699999999999996},"min":{"1min":0.52600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.67200000000000004}]},{"osd":3,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71199999999999997,"15min":0.72899999999999998},"min":{"1min":0.53200000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.73899999999999999,"15min":0.745},"min":{"1min":0.52800000000000002,"5min":0.46600000000000003,"15min":0.45300000000000001},"max":{"1min":3.339,"5min":3.339,"15min":3.339},"last":0.68500000000000005}]},{"osd":4,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.625,"15min":0.60699999999999998},"min":{"1min":0.39100000000000001,"5min":0.377,"15min":0.372},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64100000000000001,"15min":0.63300000000000001},"min":{"1min":0.44500000000000001,"5min":0.41499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.71299999999999997}]},{"osd":5,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.629,"15min":0.625},"min":{"1min":0.47599999999999998,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.89200000000000002,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.69999999999999996,"15min":0.68600000000000005},"min":{"1min":0.45500000000000002,"5min":0.40899999999999997,"15min":0.40300000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.77600000000000002}]},{"osd":7,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.69999999999999996,"15min":0.68999999999999995},"min":{"1min":0.501,"5min":0.46999999999999997,"15min":0.40200000000000002},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.749},{"interface":"front","average":{"1min":0.754,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.52000000000000002,"5min":0.42799999999999999,"15min":0.42099999999999999},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.76200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575076,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59612,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312228,"statfs":{"total":96636764160,"available":96575721472,"internally_reserved":0,"allocated":2306048,"data_stored":1437157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.54400000000000004,"15min":0.47599999999999998},"min":{"1min":0.30199999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.57599999999999996,"5min":10.052,"15min":10.052},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.436,"5min":0.59399999999999997,"15min":0.53300000000000003},"min":{"1min":0.30399999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61799999999999999,"5min":9.0250000000000004,"15min":9.0250000000000004},"last":0.47999999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.58799999999999997,"15min":0.53000000000000003},"min":{"1min":0.36099999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.55500000000000005,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.64100000000000001,"15min":0.59899999999999998},"min":{"1min":0.45500000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.68899999999999995,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.46500000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64500000000000002,"5min":2.609,"15min":2.609},"last":0.433},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66800000000000004,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.36899999999999999}]},{"osd":4,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.75800000000000001,"15min":0.77000000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86599999999999999,"5min":2.21,"15min":2.21},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.81000000000000005,"15min":0.78500000000000003},"min":{"1min":0.47899999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.96299999999999997,"5min":6.3419999999999996,"15min":6.3419999999999996},"last":0.92600000000000005}]},{"osd":5,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.82499999999999996,"15min":0.78900000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.41999999999999998},"max":{"1min":0.96299999999999997,"5min":6.5529999999999999,"15min":6.5529999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.78200000000000003,"15min":0.79000000000000004},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.48799999999999999},"max":{"1min":1.0189999999999999,"5min":2.544,"15min":2.544},"last":0.94299999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":1.849,"15min":1.1519999999999999},"min":{"1min":0.55200000000000005,"5min":0.53200000000000003,"15min":0.47699999999999998},"max":{"1min":0.95399999999999996,"5min":78.903000000000006,"15min":78.903000000000006},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":1.8260000000000001,"15min":1.139},"min":{"1min":0.52400000000000002,"5min":0.49199999999999999,"15min":0.38100000000000001},"max":{"1min":0.89600000000000002,"5min":78.837000000000003,"15min":78.837000000000003},"last":0.73299999999999998}]},{"osd":7,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78500000000000003,"15min":0.80000000000000004},"min":{"1min":0.55400000000000005,"5min":0.52800000000000002,"15min":0.48799999999999999},"max":{"1min":0.93200000000000005,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.79100000000000004,"15min":0.81799999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.97699999999999998,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":0.80100000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575076,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32788,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339052,"statfs":{"total":96636764160,"available":96603189248,"internally_reserved":0,"allocated":1708032,"data_stored":845770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.42199999999999999,"15min":0.437},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67800000000000005,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.45900000000000002,"15min":0.48199999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.629,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.55500000000000005}]},{"osd":2,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47299999999999998,"15min":0.48599999999999999},"min":{"1min":0.32200000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":1.597,"15min":1.597},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51100000000000001,"15min":0.52700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72699999999999998,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.41399999999999998}]},{"osd":3,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.50700000000000001,"15min":0.52300000000000002},"min":{"1min":0.38700000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81799999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.60199999999999998},"min":{"1min":0.40300000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.78500000000000003,"5min":1.492,"15min":1.492},"last":0.59999999999999998}]},{"osd":4,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.753,"15min":0.77600000000000002},"min":{"1min":0.51500000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0409999999999999,"5min":1.1579999999999999,"15min":1.181},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76300000000000001,"15min":0.77300000000000002},"min":{"1min":0.54300000000000004,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":1.097,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.82499999999999996}]},{"osd":5,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.76600000000000001,"15min":0.79100000000000004},"min":{"1min":0.56599999999999995,"5min":0.48699999999999999,"15min":0.47299999999999998},"max":{"1min":0.998,"5min":1.361,"15min":2.1869999999999998},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.76900000000000002,"15min":0.79500000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.026,"5min":1.546,"15min":2.6230000000000002},"last":0.89700000000000002}]},{"osd":6,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.73099999999999998,"15min":0.747},"min":{"1min":0.47999999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.93700000000000006,"5min":1.52,"15min":1.52},"last":0.748},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.71799999999999997,"15min":0.745},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.41299999999999998},"max":{"1min":1.026,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.69799999999999995}]},{"osd":7,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.78800000000000003,"15min":0.81200000000000006},"min":{"1min":0.53600000000000003,"5min":0.51600000000000001,"15min":0.505},"max":{"1min":1.002,"5min":1.139,"15min":2.3959999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.76200000000000001,"15min":0.80100000000000005},"min":{"1min":0.50900000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0960000000000001,"5min":1.3959999999999999,"15min":2.145},"last":0.86799999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575075,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36912,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334928,"statfs":{"total":96636764160,"available":96598966272,"internally_reserved":0,"allocated":1736704,"data_stored":854578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.29399999999999998,"5min":0.26700000000000002,"15min":0.26400000000000001},"max":{"1min":0.70499999999999996,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46700000000000003,"15min":0.47299999999999998},"min":{"1min":0.33600000000000002,"5min":0.29099999999999998,"15min":0.26200000000000001},"max":{"1min":0.58899999999999997,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.65000000000000002}]},{"osd":1,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.33800000000000002,"5min":0.32700000000000001,"15min":0.30499999999999999},"max":{"1min":0.58699999999999997,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.5,"15min":0.51700000000000002},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":0.58499999999999996,"5min":0.996,"15min":0.996},"last":0.59099999999999997}]},{"osd":3,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.316},"max":{"1min":0.82399999999999995,"5min":1.472,"15min":1.472},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.59199999999999997,"15min":0.60299999999999998},"min":{"1min":0.42899999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.70999999999999996,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.67200000000000004}]},{"osd":4,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.72699999999999998,"15min":0.76000000000000001},"min":{"1min":0.52300000000000002,"5min":0.5,"15min":0.47199999999999998},"max":{"1min":1.046,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.873},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.73199999999999998,"15min":0.76200000000000001},"min":{"1min":0.56000000000000005,"5min":0.55000000000000004,"15min":0.48499999999999999},"max":{"1min":1.1040000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.78000000000000003}]},{"osd":5,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.72199999999999998,"15min":0.751},"min":{"1min":0.54300000000000004,"5min":0.51000000000000001,"15min":0.46200000000000002},"max":{"1min":0.91500000000000004,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.76900000000000002},"min":{"1min":0.61499999999999999,"5min":0.55600000000000005,"15min":0.50900000000000001},"max":{"1min":0.95199999999999996,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.79800000000000004}]},{"osd":6,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.747,"15min":0.77800000000000002},"min":{"1min":0.58399999999999996,"5min":0.501,"15min":0.442},"max":{"1min":1.079,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.77100000000000002},"min":{"1min":0.56499999999999995,"5min":0.502,"15min":0.502},"max":{"1min":0.871,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.73499999999999999}]},{"osd":7,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.78400000000000003},"min":{"1min":0.55900000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0249999999999999,"5min":1.278,"15min":1.278},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73299999999999998,"15min":0.77900000000000003},"min":{"1min":0.504,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.99099999999999999,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.76600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575075,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59044,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312796,"statfs":{"total":96636764160,"available":96576303104,"internally_reserved":0,"allocated":1724416,"data_stored":853940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.437,"15min":0.42899999999999999},"min":{"1min":0.314,"5min":0.26500000000000001,"15min":0.23499999999999999},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.44600000000000001,"15min":0.46000000000000002},"min":{"1min":0.308,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58599999999999997,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.442}]},{"osd":1,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46999999999999997,"15min":0.48699999999999999},"min":{"1min":0.33900000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69599999999999995,"5min":0.78000000000000003,"15min":1.2809999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56899999999999995,"15min":0.56299999999999994},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71099999999999997,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":0.45800000000000002}]},{"osd":2,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.60799999999999998},"min":{"1min":0.42599999999999999,"5min":0.40899999999999997,"15min":0.38700000000000001},"max":{"1min":0.88900000000000001,"5min":2.214,"15min":2.214},"last":0.499},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58499999999999996,"15min":0.60099999999999998},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.50600000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.72999999999999998,"15min":0.73299999999999998},"min":{"1min":0.53600000000000003,"5min":0.47699999999999998,"15min":0.38400000000000001},"max":{"1min":0.86499999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.39600000000000002},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.61599999999999999}]},{"osd":5,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72799999999999998,"15min":0.751},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.40899999999999997},"max":{"1min":0.84399999999999997,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.74399999999999999,"15min":0.749},"min":{"1min":0.51100000000000001,"5min":0.49199999999999999,"15min":0.40200000000000002},"max":{"1min":0.90000000000000002,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.77600000000000002}]},{"osd":6,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76900000000000002,"15min":0.76500000000000001},"min":{"1min":0.55300000000000005,"5min":0.51100000000000001,"15min":0.45800000000000002},"max":{"1min":1.4359999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73899999999999999,"15min":0.76800000000000002},"min":{"1min":0.51700000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.90600000000000003,"5min":2.286,"15min":2.286},"last":0.70499999999999996}]},{"osd":7,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.746,"15min":0.76700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.439},"max":{"1min":0.82799999999999996,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77400000000000002,"15min":0.78500000000000003},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.48999999999999999},"max":{"1min":1.3200000000000001,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.68700000000000006}]}]},{"osd":4,"up_from":13,"seq":55834575075,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41012,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330828,"statfs":{"total":96636764160,"available":96594767872,"internally_reserved":0,"allocated":1740800,"data_stored":855594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64700000000000002,"15min":0.66900000000000004},"min":{"1min":0.42199999999999999,"5min":0.41699999999999998,"15min":0.32100000000000001},"max":{"1min":0.81999999999999995,"5min":0.97399999999999998,"15min":1.0189999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.64100000000000001,"15min":0.66900000000000004},"min":{"1min":0.44600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91800000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.54200000000000004}]},{"osd":1,"last update":"Tue Apr 2 07:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.70299999999999996,"15min":0.72299999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.433},"max":{"1min":1.012,"5min":1.014,"15min":1.036},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.70899999999999996,"15min":0.73499999999999999},"min":{"1min":0.47899999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.95799999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.72199999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.68000000000000005,"15min":0.69399999999999995},"min":{"1min":0.44500000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.93799999999999994,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.69299999999999995,"15min":0.71199999999999997},"min":{"1min":0.497,"5min":0.378,"15min":0.378},"max":{"1min":0.91800000000000004,"5min":1.026,"15min":1.0620000000000001},"last":0.67000000000000004}]},{"osd":3,"last update":"Tue Apr 2 07:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.53400000000000003,"5min":0.505,"15min":0.46800000000000003},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70199999999999996,"15min":0.72699999999999998},"min":{"1min":0.504,"5min":0.47299999999999998,"15min":0.41499999999999998},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.61399999999999999}]},{"osd":5,"last update":"Tue Apr 2 07:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58899999999999997,"15min":0.58099999999999996},"min":{"1min":0.39400000000000002,"5min":0.38600000000000001,"15min":0.29599999999999999},"max":{"1min":0.89600000000000002,"5min":0.90800000000000003,"15min":0.93999999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.60199999999999998,"15min":0.58799999999999997},"min":{"1min":0.41699999999999998,"5min":0.41399999999999998,"15min":0.33700000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.69199999999999995}]},{"osd":6,"last update":"Tue Apr 2 07:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.628,"15min":0.60599999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.374},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67300000000000004,"15min":0.65500000000000003},"min":{"1min":0.48099999999999998,"5min":0.44400000000000001,"15min":0.42399999999999999},"max":{"1min":0.97999999999999998,"5min":1.0129999999999999,"15min":1.038},"last":0.60099999999999998}]},{"osd":7,"last update":"Tue Apr 2 07:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.46500000000000002,"5min":0.39600000000000002,"15min":0.38300000000000001},"max":{"1min":0.98199999999999998,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.748,"15min":0.73699999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.434},"max":{"1min":1.0509999999999999,"5min":1.095,"15min":1.095},"last":0.77800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575078,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55532,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316308,"statfs":{"total":96636764160,"available":96579899392,"internally_reserved":0,"allocated":2322432,"data_stored":1443381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64200000000000002,"15min":0.67800000000000005},"min":{"1min":0.45200000000000001,"5min":0.35299999999999998,"15min":0.29899999999999999},"max":{"1min":0.90300000000000002,"5min":0.92300000000000004,"15min":0.99199999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.65100000000000002,"15min":0.68300000000000005},"min":{"1min":0.40500000000000003,"5min":0.40000000000000002,"15min":0.378},"max":{"1min":0.79000000000000004,"5min":1.0249999999999999,"15min":1.4450000000000001},"last":0.71299999999999997}]},{"osd":1,"last update":"Tue Apr 2 07:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.64600000000000002,"15min":0.68700000000000006},"min":{"1min":0.40999999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.875,"5min":1.0620000000000001,"15min":1.131},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66100000000000003,"15min":0.68899999999999995},"min":{"1min":0.48899999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.872,"5min":0.96399999999999997,"15min":1.5149999999999999},"last":0.78300000000000003}]},{"osd":2,"last update":"Tue Apr 2 07:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66500000000000004,"15min":0.70199999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.41799999999999998},"max":{"1min":0.88800000000000001,"5min":0.90700000000000003,"15min":1.5449999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.67300000000000004,"15min":0.70299999999999996},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.372},"max":{"1min":0.89300000000000002,"5min":1.111,"15min":1.3959999999999999},"last":0.879}]},{"osd":3,"last update":"Tue Apr 2 07:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.73199999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.92500000000000004,"5min":1.0649999999999999,"15min":1.341},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.54500000000000004,"5min":0.48299999999999998,"15min":0.44900000000000001},"max":{"1min":0.94099999999999995,"5min":1.1839999999999999,"15min":1.484},"last":0.92000000000000004}]},{"osd":4,"last update":"Tue Apr 2 07:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.40600000000000003,"5min":0.39500000000000002,"15min":0.372},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":1.3109999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64600000000000002,"15min":0.64000000000000001},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.35599999999999998},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":1.504},"last":0.59899999999999998}]},{"osd":6,"last update":"Tue Apr 2 07:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.39900000000000002},"max":{"1min":1.004,"5min":1.212,"15min":1.212},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69799999999999995,"15min":0.67700000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41199999999999998},"max":{"1min":0.92800000000000005,"5min":1.774,"15min":1.774},"last":0.57499999999999996}]},{"osd":7,"last update":"Tue Apr 2 07:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.72799999999999998,"15min":0.70499999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.38900000000000001},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.6890000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.745,"15min":0.72799999999999998},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":1.139,"5min":1.149,"15min":1.5309999999999999},"last":0.67500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:55:36.437 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T07:55:36.437 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T07:55:36.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.671+0000 7f6f09c00700 1 Processor -- start 2024-04-02T07:55:36.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.672+0000 7f6f09c00700 1 -- start start 2024-04-02T07:55:36.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.673+0000 7f6f09c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6f040d21f0 0x7f6f040ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:36.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.673+0000 7f6f09c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6f040cec40 0x7f6f04161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:36.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.673+0000 7f6f09c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f040ce270 con 0x7f6f041611b0 2024-04-02T07:55:36.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.673+0000 7f6f09c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f0415b600 con 0x7f6f040cec40 2024-04-02T07:55:36.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.673+0000 7f6f09c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f040d2760 con 0x7f6f040d21f0 2024-04-02T07:55:36.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.673+0000 7f6f03400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6f040cec40 0x7f6f04161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:36.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.673+0000 7f6f02000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6f040d21f0 0x7f6f040ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:36.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.673+0000 7f6f03400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6f040cec40 0x7f6f04161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54944/0 (socket says 172.21.15.89:54944) 2024-04-02T07:55:36.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.673+0000 7f6f03400700 1 -- 172.21.15.89:0/4204514539 learned_addr learned my addr 172.21.15.89:0/4204514539 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:36.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.674+0000 7f6f03400700 1 -- 172.21.15.89:0/4204514539 >> v2:172.21.15.113:3300/0 conn(0x7f6f040d21f0 msgr2=0x7f6f040ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:36.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.674+0000 7f6f03400700 1 --2- 172.21.15.89:0/4204514539 >> v2:172.21.15.113:3300/0 conn(0x7f6f040d21f0 0x7f6f040ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.674+0000 7f6f03400700 1 -- 172.21.15.89:0/4204514539 >> v1:172.21.15.113:6789/0 conn(0x7f6f041611b0 legacy=0x7f6f0415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:36.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.674+0000 7f6f03400700 1 -- 172.21.15.89:0/4204514539 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f040d2a40 con 0x7f6f040cec40 2024-04-02T07:55:36.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.674+0000 7f6f03400700 1 --2- 172.21.15.89:0/4204514539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f040cec40 0x7f6f04161590 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f6f0408c030 tx=0x7f6efc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baab7d9736d9a0f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:36.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.674+0000 7f6f01600700 1 -- 172.21.15.89:0/4204514539 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc00e070 con 0x7f6f040cec40 2024-04-02T07:55:36.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.674+0000 7f6f01600700 1 -- 172.21.15.89:0/4204514539 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6efc00b910 con 0x7f6f040cec40 2024-04-02T07:55:36.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.674+0000 7f6f01600700 1 -- 172.21.15.89:0/4204514539 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc00bcc0 con 0x7f6f040cec40 2024-04-02T07:55:36.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.675+0000 7f6f09c00700 1 -- 172.21.15.89:0/4204514539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f040cec40 msgr2=0x7f6f04161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:36.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.675+0000 7f6f09c00700 1 --2- 172.21.15.89:0/4204514539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f040cec40 0x7f6f04161590 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f6f0408c030 tx=0x7f6efc00b330 comp rx=0 tx=0).stop 2024-04-02T07:55:36.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.675+0000 7f6f09c00700 1 -- 172.21.15.89:0/4204514539 shutdown_connections 2024-04-02T07:55:36.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.675+0000 7f6f09c00700 1 --2- 172.21.15.89:0/4204514539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f040cec40 0x7f6f04161590 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.675+0000 7f6f09c00700 1 --2- 172.21.15.89:0/4204514539 >> v2:172.21.15.113:3300/0 conn(0x7f6f040d21f0 0x7f6f040ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.675+0000 7f6f09c00700 1 -- 172.21.15.89:0/4204514539 >> 172.21.15.89:0/4204514539 conn(0x7f6f0408aff0 msgr2=0x7f6f040d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:36.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.675+0000 7f6f09c00700 1 -- 172.21.15.89:0/4204514539 shutdown_connections 2024-04-02T07:55:36.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.675+0000 7f6f09c00700 1 -- 172.21.15.89:0/4204514539 wait complete. 2024-04-02T07:55:36.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.677+0000 7f6f09c00700 1 Processor -- start 2024-04-02T07:55:36.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.677+0000 7f6f09c00700 1 -- start start 2024-04-02T07:55:36.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.678+0000 7f6f09c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f040cec40 0x7f6f041e8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:36.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f09c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f040d21f0 0x7f6f041e8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:36.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f09c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f041611b0 0x7f6f041e8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:36.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f09c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f04154540 con 0x7f6f040d21f0 2024-04-02T07:55:36.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f09c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040cafd0 con 0x7f6f040cec40 2024-04-02T07:55:36.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f09c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f04154f40 con 0x7f6f041611b0 2024-04-02T07:55:36.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f02000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f040d21f0 0x7f6f041e8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:36.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f02000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f040d21f0 0x7f6f041e8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54946/0 (socket says 172.21.15.89:54946) 2024-04-02T07:55:36.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f02000700 1 -- 172.21.15.89:0/4063845039 learned_addr learned my addr 172.21.15.89:0/4063845039 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:36.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f02a00700 1 --2- 172.21.15.89:0/4063845039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f040cec40 0x7f6f041e8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:36.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f03400700 1 --2- 172.21.15.89:0/4063845039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f041611b0 0x7f6f041e8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:36.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f02000700 1 -- 172.21.15.89:0/4063845039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f041611b0 msgr2=0x7f6f041e8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:36.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f02000700 1 --2- 172.21.15.89:0/4063845039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f041611b0 0x7f6f041e8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f02000700 1 -- 172.21.15.89:0/4063845039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f040cec40 msgr2=0x7f6f041e8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:36.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f02000700 1 --2- 172.21.15.89:0/4063845039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f040cec40 0x7f6f041e8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.679+0000 7f6f02000700 1 -- 172.21.15.89:0/4063845039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc009b30 con 0x7f6f040d21f0 2024-04-02T07:55:36.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.680+0000 7f6f02000700 1 --2- 172.21.15.89:0/4063845039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f040d21f0 0x7f6f041e8580 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f6ef000e9b0 tx=0x7f6ef000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:36.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.680+0000 7f6efaa00700 1 -- 172.21.15.89:0/4063845039 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0009710 con 0x7f6f040d21f0 2024-04-02T07:55:36.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.680+0000 7f6efaa00700 1 -- 172.21.15.89:0/4063845039 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef000ee50 con 0x7f6f040d21f0 2024-04-02T07:55:36.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.680+0000 7f6efaa00700 1 -- 172.21.15.89:0/4063845039 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef000ca50 con 0x7f6f040d21f0 2024-04-02T07:55:36.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.680+0000 7f6f09c00700 1 -- 172.21.15.89:0/4063845039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f040ca6a0 con 0x7f6f040d21f0 2024-04-02T07:55:36.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.680+0000 7f6f09c00700 1 -- 172.21.15.89:0/4063845039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f040cac60 con 0x7f6f040d21f0 2024-04-02T07:55:36.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.681+0000 7f6f09c00700 1 -- 172.21.15.89:0/4063845039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f040723b0 con 0x7f6f040d21f0 2024-04-02T07:55:36.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.682+0000 7f6efaa00700 1 -- 172.21.15.89:0/4063845039 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6ef000cbb0 con 0x7f6f040d21f0 2024-04-02T07:55:36.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.683+0000 7f6efaa00700 1 --2- 172.21.15.89:0/4063845039 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ee406adb0 0x7f6ee406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:36.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.683+0000 7f6efaa00700 1 -- 172.21.15.89:0/4063845039 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f6ef0089a40 con 0x7f6f040d21f0 2024-04-02T07:55:36.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.683+0000 7f6f02a00700 1 --2- 172.21.15.89:0/4063845039 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ee406adb0 0x7f6ee406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:36.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.684+0000 7f6f02a00700 1 --2- 172.21.15.89:0/4063845039 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ee406adb0 0x7f6ee406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6f041e9ea0 tx=0x7f6ef4009040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:36.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.688+0000 7f6efaa00700 1 -- 172.21.15.89:0/4063845039 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ef00548f0 con 0x7f6f040d21f0 2024-04-02T07:55:36.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.968+0000 7f6f09c00700 1 -- 172.21.15.89:0/4063845039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6f040cda40 con 0x7f6f040d21f0 2024-04-02T07:55:36.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.972+0000 7f6efaa00700 1 -- 172.21.15.89:0/4063845039 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v23) v1 ==== 190+0+0 (secure 0 0 0) 0x7f6ef00170e0 con 0x7f6f040d21f0 2024-04-02T07:55:36.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.975+0000 7f6f09c00700 1 -- 172.21.15.89:0/4063845039 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ee406adb0 msgr2=0x7f6ee406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:36.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.975+0000 7f6f09c00700 1 --2- 172.21.15.89:0/4063845039 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ee406adb0 0x7f6ee406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6f041e9ea0 tx=0x7f6ef4009040 comp rx=0 tx=0).stop 2024-04-02T07:55:36.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.975+0000 7f6f09c00700 1 -- 172.21.15.89:0/4063845039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f040d21f0 msgr2=0x7f6f041e8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:36.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.975+0000 7f6f09c00700 1 --2- 172.21.15.89:0/4063845039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f040d21f0 0x7f6f041e8580 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f6ef000e9b0 tx=0x7f6ef000ecc0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.975+0000 7f6f09c00700 1 -- 172.21.15.89:0/4063845039 shutdown_connections 2024-04-02T07:55:36.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.975+0000 7f6f09c00700 1 --2- 172.21.15.89:0/4063845039 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6ee406adb0 0x7f6ee406d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.975+0000 7f6f09c00700 1 --2- 172.21.15.89:0/4063845039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f040cec40 0x7f6f041e8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.975+0000 7f6f09c00700 1 --2- 172.21.15.89:0/4063845039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f041611b0 0x7f6f041e8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.975+0000 7f6f09c00700 1 --2- 172.21.15.89:0/4063845039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f040d21f0 0x7f6f041e8580 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:36.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.975+0000 7f6f09c00700 1 -- 172.21.15.89:0/4063845039 >> 172.21.15.89:0/4063845039 conn(0x7f6f0408aff0 msgr2=0x7f6f0415da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:36.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.975+0000 7f6f09c00700 1 -- 172.21.15.89:0/4063845039 shutdown_connections 2024-04-02T07:55:36.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:36.975+0000 7f6f09c00700 1 -- 172.21.15.89:0/4063845039 wait complete. 2024-04-02T07:55:36.979 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T07:55:36.995 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:55:37.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.185+0000 7f313a400700 1 Processor -- start 2024-04-02T07:55:37.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.185+0000 7f313a400700 1 -- start start 2024-04-02T07:55:37.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.186+0000 7f313a400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f313415d670 0x7f31340ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:37.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.186+0000 7f313a400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f313406dee0 0x7f313406e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:37.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.186+0000 7f313a400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31340b3890 con 0x7f313415da50 2024-04-02T07:55:37.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.186+0000 7f313a400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31340acac0 con 0x7f313415d670 2024-04-02T07:55:37.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.186+0000 7f313a400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3134159f50 con 0x7f313406dee0 2024-04-02T07:55:37.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.186+0000 7f3133400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f313415d670 0x7f31340ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:37.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.186+0000 7f3133400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f313415d670 0x7f31340ac580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54956/0 (socket says 172.21.15.89:54956) 2024-04-02T07:55:37.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.186+0000 7f3133400700 1 -- 172.21.15.89:0/400823106 learned_addr learned my addr 172.21.15.89:0/400823106 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:37.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.186+0000 7f3132a00700 1 --1- 172.21.15.89:0/400823106 >> v1:172.21.15.113:6789/0 conn(0x7f313415da50 0x7f31340b2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:52046/0 (socket says 172.21.15.89:52046) 2024-04-02T07:55:37.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.186+0000 7f3133e00700 1 --2- 172.21.15.89:0/400823106 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f313406dee0 0x7f313406e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:37.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.187+0000 7f3133400700 1 -- 172.21.15.89:0/400823106 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f313406dee0 msgr2=0x7f313406e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:37.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.187+0000 7f3133400700 1 --2- 172.21.15.89:0/400823106 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f313406dee0 0x7f313406e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:37.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.187+0000 7f3133400700 1 -- 172.21.15.89:0/400823106 >> v1:172.21.15.113:6789/0 conn(0x7f313415da50 legacy=0x7f31340b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:37.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.187+0000 7f3133400700 1 -- 172.21.15.89:0/400823106 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31340b35b0 con 0x7f313415d670 2024-04-02T07:55:37.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.187+0000 7f3133400700 1 --2- 172.21.15.89:0/400823106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313415d670 0x7f31340ac580 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f31240098f0 tx=0x7f3124009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c060f4fb3b26d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:37.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.187+0000 7f3132000700 1 -- 172.21.15.89:0/400823106 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3124011830 con 0x7f313415d670 2024-04-02T07:55:37.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.187+0000 7f3132000700 1 -- 172.21.15.89:0/400823106 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3124011990 con 0x7f313415d670 2024-04-02T07:55:37.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.187+0000 7f3132000700 1 -- 172.21.15.89:0/400823106 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3124011d20 con 0x7f313415d670 2024-04-02T07:55:37.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.188+0000 7f313a400700 1 -- 172.21.15.89:0/400823106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313415d670 msgr2=0x7f31340ac580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:37.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.188+0000 7f313a400700 1 --2- 172.21.15.89:0/400823106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313415d670 0x7f31340ac580 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f31240098f0 tx=0x7f3124009c00 comp rx=0 tx=0).stop 2024-04-02T07:55:37.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.188+0000 7f313a400700 1 -- 172.21.15.89:0/400823106 shutdown_connections 2024-04-02T07:55:37.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.188+0000 7f313a400700 1 --2- 172.21.15.89:0/400823106 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f313406dee0 0x7f313406e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:37.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.188+0000 7f313a400700 1 --2- 172.21.15.89:0/400823106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313415d670 0x7f31340ac580 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:37.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.188+0000 7f313a400700 1 -- 172.21.15.89:0/400823106 >> 172.21.15.89:0/400823106 conn(0x7f31340cf190 msgr2=0x7f31340cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:37.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.188+0000 7f313a400700 1 -- 172.21.15.89:0/400823106 shutdown_connections 2024-04-02T07:55:37.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.188+0000 7f313a400700 1 -- 172.21.15.89:0/400823106 wait complete. 2024-04-02T07:55:37.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.190+0000 7f313a400700 1 Processor -- start 2024-04-02T07:55:37.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.190+0000 7f313a400700 1 -- start start 2024-04-02T07:55:37.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f313a400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f313406dee0 0x7f31341f4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:37.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f313a400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f313415d670 0x7f3134158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:37.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f313a400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313415da50 0x7f31340b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:37.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f313a400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3134158f90 con 0x7f313415da50 2024-04-02T07:55:37.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f313a400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31341578e0 con 0x7f313406dee0 2024-04-02T07:55:37.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f313a400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3134157bc0 con 0x7f313415d670 2024-04-02T07:55:37.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f3133e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313415da50 0x7f31340b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:37.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f3133e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313415da50 0x7f31340b02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54972/0 (socket says 172.21.15.89:54972) 2024-04-02T07:55:37.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f3133e00700 1 -- 172.21.15.89:0/1757208825 learned_addr learned my addr 172.21.15.89:0/1757208825 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:37.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f3133400700 1 --2- 172.21.15.89:0/1757208825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f313406dee0 0x7f31341f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:37.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f3133e00700 1 -- 172.21.15.89:0/1757208825 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f313415d670 msgr2=0x7f3134158a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:37.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f3133e00700 1 --2- 172.21.15.89:0/1757208825 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f313415d670 0x7f3134158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:37.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f3133e00700 1 -- 172.21.15.89:0/1757208825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f313406dee0 msgr2=0x7f31341f4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:37.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f3133e00700 1 --2- 172.21.15.89:0/1757208825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f313406dee0 0x7f31341f4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:37.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f3133e00700 1 -- 172.21.15.89:0/1757208825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3124009550 con 0x7f313415da50 2024-04-02T07:55:37.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f3133400700 1 --2- 172.21.15.89:0/1757208825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f313406dee0 0x7f31341f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:55:37.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.191+0000 7f3133e00700 1 --2- 172.21.15.89:0/1757208825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313415da50 0x7f31340b02b0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f312c009e00 tx=0x7f312c0074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:37.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.192+0000 7f312b400700 1 -- 172.21.15.89:0/1757208825 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c00add0 con 0x7f313415da50 2024-04-02T07:55:37.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.192+0000 7f312b400700 1 -- 172.21.15.89:0/1757208825 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312c010040 con 0x7f313415da50 2024-04-02T07:55:37.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.192+0000 7f313a400700 1 -- 172.21.15.89:0/1757208825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3134157ea0 con 0x7f313415da50 2024-04-02T07:55:37.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.192+0000 7f312b400700 1 -- 172.21.15.89:0/1757208825 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c003c60 con 0x7f313415da50 2024-04-02T07:55:37.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.192+0000 7f313a400700 1 -- 172.21.15.89:0/1757208825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3134158460 con 0x7f313415da50 2024-04-02T07:55:37.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.193+0000 7f313a400700 1 -- 172.21.15.89:0/1757208825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313415deb0 con 0x7f313415da50 2024-04-02T07:55:37.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.194+0000 7f312b400700 1 -- 172.21.15.89:0/1757208825 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f312c004030 con 0x7f313415da50 2024-04-02T07:55:37.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.195+0000 7f312b400700 1 --2- 172.21.15.89:0/1757208825 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f311406adb0 0x7f311406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:37.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.195+0000 7f312b400700 1 -- 172.21.15.89:0/1757208825 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f312c089660 con 0x7f313415da50 2024-04-02T07:55:37.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.195+0000 7f3133400700 1 --2- 172.21.15.89:0/1757208825 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f311406adb0 0x7f311406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:37.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.196+0000 7f3133400700 1 --2- 172.21.15.89:0/1757208825 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f311406adb0 0x7f311406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f313415ce60 tx=0x7f3124008030 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:37.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.199+0000 7f312b400700 1 -- 172.21.15.89:0/1757208825 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f312c054490 con 0x7f313415da50 2024-04-02T07:55:37.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.447+0000 7f313a400700 1 -- 172.21.15.89:0/1757208825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3134066ff0 con 0x7f313415da50 2024-04-02T07:55:37.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.447+0000 7f312b400700 1 -- 172.21.15.89:0/1757208825 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4733 (secure 0 0 0) 0x7f312c057ab0 con 0x7f313415da50 2024-04-02T07:55:37.449 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:37.449 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/66826967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":66826967},{"type":"v1","addr":"172.21.15.113:6835","nonce":66826967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":7915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/1669765650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1669765650},{"type":"v1","addr":"172.21.15.113:6833","nonce":1669765650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":7918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3321722319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3321722319},{"type":"v1","addr":"172.21.15.89:6837","nonce":3321722319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:55:31.571731+0000","modified":"2024-04-02T07:55:34.583055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8436":{"gid":8436,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":4,"addr":"172.21.15.89:6836/417215494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":417215494},{"type":"v1","addr":"172.21.15.89:6836","nonce":417215494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-02T07:55:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.450+0000 7f313a400700 1 -- 172.21.15.89:0/1757208825 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f311406adb0 msgr2=0x7f311406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.450+0000 7f313a400700 1 --2- 172.21.15.89:0/1757208825 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f311406adb0 0x7f311406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f313415ce60 tx=0x7f3124008030 comp rx=0 tx=0).stop 2024-04-02T07:55:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.450+0000 7f313a400700 1 -- 172.21.15.89:0/1757208825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313415da50 msgr2=0x7f31340b02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.450+0000 7f313a400700 1 --2- 172.21.15.89:0/1757208825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313415da50 0x7f31340b02b0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f312c009e00 tx=0x7f312c0074f0 comp rx=0 tx=0).stop 2024-04-02T07:55:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.450+0000 7f313a400700 1 -- 172.21.15.89:0/1757208825 shutdown_connections 2024-04-02T07:55:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.450+0000 7f313a400700 1 --2- 172.21.15.89:0/1757208825 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f311406adb0 0x7f311406d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.450+0000 7f313a400700 1 --2- 172.21.15.89:0/1757208825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f313406dee0 0x7f31341f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.450+0000 7f313a400700 1 --2- 172.21.15.89:0/1757208825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f313415da50 0x7f31340b02b0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.450+0000 7f313a400700 1 --2- 172.21.15.89:0/1757208825 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f313415d670 0x7f3134158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.450+0000 7f313a400700 1 -- 172.21.15.89:0/1757208825 >> 172.21.15.89:0/1757208825 conn(0x7f31340cf190 msgr2=0x7f31341535e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.450+0000 7f313a400700 1 -- 172.21.15.89:0/1757208825 shutdown_connections 2024-04-02T07:55:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:37.450+0000 7f313a400700 1 -- 172.21.15.89:0/1757208825 wait complete. 2024-04-02T07:55:37.453 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 66 2024-04-02T07:55:37.871 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T07:55:31.571731+0000', 'modified': '2024-04-02T07:55:34.583055+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8436}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8436': {'gid': 8436, 'name': 'c', 'rank': 0, 'incarnation': 62, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.89:6836/417215494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6834', 'nonce': 417215494}, {'type': 'v1', 'addr': '172.21.15.89:6836', 'nonce': 417215494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T07:55:37.872 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T07:55:37.872 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-02T07:55:38.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.067+0000 7effe5400700 1 Processor -- start 2024-04-02T07:55:38.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.067+0000 7effe5400700 1 -- start start 2024-04-02T07:55:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.068+0000 7effe5400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7effe00d16a0 0x7effe00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.068+0000 7effe5400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effe00d2370 0x7effe00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.068+0000 7effe5400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe0161420 con 0x7effe015d9d0 2024-04-02T07:55:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.068+0000 7effe5400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe00d2820 con 0x7effe00d16a0 2024-04-02T07:55:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.068+0000 7effe5400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe00d4340 con 0x7effe00d2370 2024-04-02T07:55:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.068+0000 7effde800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7effe015d9d0 0x7effe00d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44794/0 (socket says 172.21.15.89:44794) 2024-04-02T07:55:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.068+0000 7effde800700 1 -- 172.21.15.89:0/2634157808 learned_addr learned my addr 172.21.15.89:0/2634157808 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.068+0000 7effdde00700 1 --2- 172.21.15.89:0/2634157808 >> v2:172.21.15.113:3300/0 conn(0x7effe00d16a0 0x7effe00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.068+0000 7effdf200700 1 --2- 172.21.15.89:0/2634157808 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effe00d2370 0x7effe00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.068+0000 7effdd400700 1 -- 172.21.15.89:0/2634157808 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232621053 0 0) 0x7effe0161420 con 0x7effe015d9d0 2024-04-02T07:55:38.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 -- 172.21.15.89:0/2634157808 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc40035b0 con 0x7effe015d9d0 2024-04-02T07:55:38.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 -- 172.21.15.89:0/2634157808 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3400582782 0 0) 0x7effc40035b0 con 0x7effe015d9d0 2024-04-02T07:55:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 -- 172.21.15.89:0/2634157808 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effe00d2370 msgr2=0x7effe00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 --2- 172.21.15.89:0/2634157808 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effe00d2370 0x7effe00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 -- 172.21.15.89:0/2634157808 >> v2:172.21.15.113:3300/0 conn(0x7effe00d16a0 msgr2=0x7effe00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 --2- 172.21.15.89:0/2634157808 >> v2:172.21.15.113:3300/0 conn(0x7effe00d16a0 0x7effe00d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 -- 172.21.15.89:0/2634157808 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe0161700 con 0x7effe015d9d0 2024-04-02T07:55:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 -- 172.21.15.89:0/2634157808 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7effd0001670 con 0x7effe015d9d0 2024-04-02T07:55:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 -- 172.21.15.89:0/2634157808 >> v1:172.21.15.89:6789/0 conn(0x7effe015d9d0 legacy=0x7effe00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 --2- 172.21.15.89:0/2634157808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effc4003a00 0x7effc4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 --2- 172.21.15.89:0/2634157808 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effe00d2370 0x7effc4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 --2- 172.21.15.89:0/2634157808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc4004d90 0x7effc4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 -- 172.21.15.89:0/2634157808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe00d2820 con 0x7effc4004d90 2024-04-02T07:55:38.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 -- 172.21.15.89:0/2634157808 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc400ba10 con 0x7effc4003a00 2024-04-02T07:55:38.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdd400700 1 -- 172.21.15.89:0/2634157808 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc400bb50 con 0x7effe00d2370 2024-04-02T07:55:38.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdf200700 1 --2- 172.21.15.89:0/2634157808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc4004d90 0x7effc4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effde800700 1 --2- 172.21.15.89:0/2634157808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effc4003a00 0x7effc4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.069+0000 7effdde00700 1 --2- 172.21.15.89:0/2634157808 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effe00d2370 0x7effc4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.070+0000 7effdf200700 1 -- 172.21.15.89:0/2634157808 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effe00d2370 msgr2=0x7effc4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:38.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.070+0000 7effdf200700 1 --2- 172.21.15.89:0/2634157808 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effe00d2370 0x7effc4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.070+0000 7effdf200700 1 -- 172.21.15.89:0/2634157808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effc4003a00 msgr2=0x7effc4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:38.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.070+0000 7effdf200700 1 --2- 172.21.15.89:0/2634157808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effc4003a00 0x7effc4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.070+0000 7effdf200700 1 -- 172.21.15.89:0/2634157808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc40035b0 con 0x7effc4004d90 2024-04-02T07:55:38.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.070+0000 7effdf200700 1 --2- 172.21.15.89:0/2634157808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc4004d90 0x7effc4005210 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7effd8009fc0 tx=0x7effc4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a10a1ed2bad0c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:38.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.070+0000 7effdd400700 1 -- 172.21.15.89:0/2634157808 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd800ec30 con 0x7effc4004d90 2024-04-02T07:55:38.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.070+0000 7effdd400700 1 -- 172.21.15.89:0/2634157808 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd800f040 con 0x7effc4004d90 2024-04-02T07:55:38.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.070+0000 7effdd400700 1 -- 172.21.15.89:0/2634157808 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8014480 con 0x7effc4004d90 2024-04-02T07:55:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.071+0000 7effe5400700 1 -- 172.21.15.89:0/2634157808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc4004d90 msgr2=0x7effc4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.071+0000 7effe5400700 1 --2- 172.21.15.89:0/2634157808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc4004d90 0x7effc4005210 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7effd8009fc0 tx=0x7effc4005880 comp rx=0 tx=0).stop 2024-04-02T07:55:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.071+0000 7effdf200700 1 -- 172.21.15.89:0/2634157808 reap_dead start 2024-04-02T07:55:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.071+0000 7effe5400700 1 -- 172.21.15.89:0/2634157808 shutdown_connections 2024-04-02T07:55:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.071+0000 7effe5400700 1 -- 172.21.15.89:0/2634157808 >> 172.21.15.89:0/2634157808 conn(0x7effe008aff0 msgr2=0x7effe00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.071+0000 7effe5400700 1 -- 172.21.15.89:0/2634157808 shutdown_connections 2024-04-02T07:55:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.071+0000 7effe5400700 1 -- 172.21.15.89:0/2634157808 wait complete. 2024-04-02T07:55:38.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.073+0000 7effe5400700 1 Processor -- start 2024-04-02T07:55:38.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.073+0000 7effe5400700 1 -- start start 2024-04-02T07:55:38.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effe5400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effc4004d90 0x7effe00b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effe5400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effe014bad0 0x7effe00b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effe5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe015ddb0 0x7effe015e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effe5400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe015fb80 con 0x7effe015ddb0 2024-04-02T07:55:38.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effe5400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe015eae0 con 0x7effe014bad0 2024-04-02T07:55:38.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effe5400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe015edc0 con 0x7effc4004d90 2024-04-02T07:55:38.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effdf200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe015ddb0 0x7effe015e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effdf200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe015ddb0 0x7effe015e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54982/0 (socket says 172.21.15.89:54982) 2024-04-02T07:55:38.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effdf200700 1 -- 172.21.15.89:0/4160030641 learned_addr learned my addr 172.21.15.89:0/4160030641 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:38.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effde800700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effc4004d90 0x7effe00b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effdde00700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effe014bad0 0x7effe00b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effdf200700 1 -- 172.21.15.89:0/4160030641 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effc4004d90 msgr2=0x7effe00b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effdf200700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effc4004d90 0x7effe00b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effdf200700 1 -- 172.21.15.89:0/4160030641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effe014bad0 msgr2=0x7effe00b4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effdf200700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effe014bad0 0x7effe00b4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effdf200700 1 -- 172.21.15.89:0/4160030641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe015f0a0 con 0x7effe015ddb0 2024-04-02T07:55:38.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effde800700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effc4004d90 0x7effe00b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:38.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.075+0000 7effdde00700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effe014bad0 0x7effe00b4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:38.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.076+0000 7effdf200700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe015ddb0 0x7effe015e260 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7effd8017040 tx=0x7effd8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:38.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.076+0000 7effd6a00700 1 -- 172.21.15.89:0/4160030641 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8007000 con 0x7effe015ddb0 2024-04-02T07:55:38.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.076+0000 7effd6a00700 1 -- 172.21.15.89:0/4160030641 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd8003e50 con 0x7effe015ddb0 2024-04-02T07:55:38.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.076+0000 7effe5400700 1 -- 172.21.15.89:0/4160030641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe015f380 con 0x7effe015ddb0 2024-04-02T07:55:38.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.076+0000 7effd6a00700 1 -- 172.21.15.89:0/4160030641 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8014b50 con 0x7effe015ddb0 2024-04-02T07:55:38.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.076+0000 7effe5400700 1 -- 172.21.15.89:0/4160030641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe015e7a0 con 0x7effe015ddb0 2024-04-02T07:55:38.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.078+0000 7effd6a00700 1 -- 172.21.15.89:0/4160030641 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7effd8023020 con 0x7effe015ddb0 2024-04-02T07:55:38.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.080+0000 7effd6a00700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7effc006adb0 0x7effc006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.080+0000 7effd6a00700 1 -- 172.21.15.89:0/4160030641 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7effd808ac70 con 0x7effe015ddb0 2024-04-02T07:55:38.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.080+0000 7effde800700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7effc006adb0 0x7effc006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.080+0000 7effde800700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7effc006adb0 0x7effc006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7effd0000c00 tx=0x7effd0009040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:38.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.082+0000 7efff8e6db80 1 Processor -- start 2024-04-02T07:55:38.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.082+0000 7efff8e6db80 1 -- start start 2024-04-02T07:55:38.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.083+0000 7efff8e6db80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78063830 0x55ea78063c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.083+0000 7efff8e6db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78036be0 0x55ea78039a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.083+0000 7efff8e6db80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78034b60 0x55ea78060a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.083+0000 7efff8e6db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea77fae650 con 0x55ea78036be0 2024-04-02T07:55:38.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.083+0000 7efff8e6db80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea77f8fc30 con 0x55ea78034b60 2024-04-02T07:55:38.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.083+0000 7efff8e6db80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea77f8d430 con 0x55ea78063830 2024-04-02T07:55:38.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.083+0000 7effdde00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78036be0 0x55ea78039a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.083+0000 7effdde00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78036be0 0x55ea78039a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54998/0 (socket says 172.21.15.89:54998) 2024-04-02T07:55:38.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.083+0000 7effdde00700 1 -- 172.21.15.89:0/1362116150 learned_addr learned my addr 172.21.15.89:0/1362116150 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:38.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.083+0000 7effde800700 1 --2- 172.21.15.89:0/1362116150 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78063830 0x55ea78063c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.083+0000 7effdf200700 1 --2- 172.21.15.89:0/1362116150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78034b60 0x55ea78060a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.084+0000 7effdde00700 1 -- 172.21.15.89:0/1362116150 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78063830 msgr2=0x55ea78063c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:38.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.084+0000 7effdde00700 1 --2- 172.21.15.89:0/1362116150 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78063830 0x55ea78063c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.084+0000 7effdde00700 1 -- 172.21.15.89:0/1362116150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78034b60 msgr2=0x55ea78060a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:38.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.084+0000 7effdde00700 1 --2- 172.21.15.89:0/1362116150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78034b60 0x55ea78060a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.084+0000 7effdde00700 1 -- 172.21.15.89:0/1362116150 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea77d7c790 con 0x55ea78036be0 2024-04-02T07:55:38.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.084+0000 7effdde00700 1 --2- 172.21.15.89:0/1362116150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78036be0 0x55ea78039a10 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7effcc009cb0 tx=0x7effcc0091c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca59602aba0eebd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:38.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.084+0000 7effcb000700 1 -- 172.21.15.89:0/1362116150 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc005c30 con 0x55ea78036be0 2024-04-02T07:55:38.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.084+0000 7effcb000700 1 -- 172.21.15.89:0/1362116150 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effcc005d90 con 0x55ea78036be0 2024-04-02T07:55:38.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.084+0000 7effcb000700 1 -- 172.21.15.89:0/1362116150 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc014b60 con 0x55ea78036be0 2024-04-02T07:55:38.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.085+0000 7efff8e6db80 1 -- 172.21.15.89:0/1362116150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78036be0 msgr2=0x55ea78039a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.085+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1362116150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78036be0 0x55ea78039a10 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7effcc009cb0 tx=0x7effcc0091c0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.085+0000 7efff8e6db80 1 -- 172.21.15.89:0/1362116150 shutdown_connections 2024-04-02T07:55:38.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.085+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1362116150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78034b60 0x55ea78060a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.085+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1362116150 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78063830 0x55ea78063c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.085+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1362116150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78036be0 0x55ea78039a10 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.085+0000 7efff8e6db80 1 -- 172.21.15.89:0/1362116150 >> 172.21.15.89:0/1362116150 conn(0x55ea77f976a0 msgr2=0x55ea7807e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:38.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.085+0000 7efff8e6db80 1 -- 172.21.15.89:0/1362116150 shutdown_connections 2024-04-02T07:55:38.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.085+0000 7efff8e6db80 1 -- 172.21.15.89:0/1362116150 wait complete. 2024-04-02T07:55:38.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.086+0000 7efff8e6db80 1 -- start start 2024-04-02T07:55:38.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.086+0000 7efff8e6db80 1 Processor -- start 2024-04-02T07:55:38.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7efff8e6db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea7805bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7efff8e6db80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78036be0 0x55ea78062440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7efff8e6db80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78063830 0x55ea7805dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7efff8e6db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea77f88d20 con 0x55ea78034b60 2024-04-02T07:55:38.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7efff8e6db80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea77f24770 con 0x55ea78036be0 2024-04-02T07:55:38.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7efff8e6db80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea77f476f0 con 0x55ea78063830 2024-04-02T07:55:38.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7efff8e6db80 10 client.0 _ll_register_callbacks cb 0x55ea77f2e5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:55:38.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7effde800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea7805bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7effde800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea7805bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55002/0 (socket says 172.21.15.89:55002) 2024-04-02T07:55:38.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7effde800700 1 -- 172.21.15.89:0/1738030894 learned_addr learned my addr 172.21.15.89:0/1738030894 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:38.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7effde800700 1 -- 172.21.15.89:0/1738030894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78063830 msgr2=0x55ea7805dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:38.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7effde800700 1 --2- 172.21.15.89:0/1738030894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78063830 0x55ea7805dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7effde800700 1 -- 172.21.15.89:0/1738030894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78036be0 msgr2=0x55ea78062440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:38.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7effde800700 1 --2- 172.21.15.89:0/1738030894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78036be0 0x55ea78062440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.088+0000 7effde800700 1 -- 172.21.15.89:0/1738030894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effcc009910 con 0x55ea78034b60 2024-04-02T07:55:38.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.089+0000 7effde800700 1 --2- 172.21.15.89:0/1738030894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea7805bdb0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7effd00157b0 tx=0x7effd0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5344ccfec6cb1ab9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:38.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.089+0000 7effbf400700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:55:38.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.089+0000 7effbf400700 1 -- 172.21.15.89:0/1738030894 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0015e90 con 0x55ea78034b60 2024-04-02T07:55:38.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.089+0000 7effbf400700 1 -- 172.21.15.89:0/1738030894 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7effd001a040 con 0x55ea78034b60 2024-04-02T07:55:38.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.089+0000 7efff8e6db80 1 -- 172.21.15.89:0/1738030894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ea7802b920 con 0x55ea78034b60 2024-04-02T07:55:38.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.089+0000 7effbf400700 1 -- 172.21.15.89:0/1738030894 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7effd0022640 con 0x55ea78034b60 2024-04-02T07:55:38.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.089+0000 7efff8e6db80 10 client.8595 fetch_fsmap learned FSMap version 66 2024-04-02T07:55:38.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.089+0000 7efff8e6db80 1 -- 172.21.15.89:0/1738030894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x55ea77e690a0 con 0x55ea78034b60 2024-04-02T07:55:38.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.089+0000 7effbf400700 1 -- 172.21.15.89:0/1738030894 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd0031020 con 0x55ea78034b60 2024-04-02T07:55:38.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.089+0000 7effbf400700 1 -- 172.21.15.89:0/1738030894 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 66) v1 ==== 3114+0+0 (secure 0 0 0) 0x7effd00227a0 con 0x55ea78034b60 2024-04-02T07:55:38.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.089+0000 7efff8e6db80 10 client.8595 fetch_fsmap finished waiting for FSMap version 66 2024-04-02T07:55:38.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.090+0000 7efff8e6db80 10 client.8595 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T07:55:38.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.090+0000 7efff8e6db80 20 client.8595 populate_metadata read hostname 'smithi089' 2024-04-02T07:55:38.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.090+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1738030894 >> [v2:172.21.15.89:6834/417215494,v1:172.21.15.89:6836/417215494] conn(0x55ea780916a0 0x55ea78091a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.090+0000 7efff8e6db80 4 client.8595 mds_command: new command op to 8436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T07:55:38.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.090+0000 7efff8e6db80 1 -- 172.21.15.89:0/1738030894 --> [v2:172.21.15.89:6834/417215494,v1:172.21.15.89:6836/417215494] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ea780086f0 con 0x55ea780916a0 2024-04-02T07:55:38.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.090+0000 7effdde00700 1 --2- 172.21.15.89:0/1738030894 >> [v2:172.21.15.89:6834/417215494,v1:172.21.15.89:6836/417215494] conn(0x55ea780916a0 0x55ea78091a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.090+0000 7effdde00700 1 --2- 172.21.15.89:0/1738030894 >> [v2:172.21.15.89:6834/417215494,v1:172.21.15.89:6836/417215494] conn(0x55ea780916a0 0x55ea78091a80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=180a600842247645 server_cookie=b52df041e13c001d in_seq=0 out_seq=0 2024-04-02T07:55:38.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.090+0000 7effbf400700 10 client.8595 ms_handle_connect on v2:172.21.15.89:6834/417215494 2024-04-02T07:55:38.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.091+0000 7effbf400700 1 -- 172.21.15.89:0/1738030894 <== mds.0 v2:172.21.15.89:6834/417215494 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ea780086f0 con 0x55ea780916a0 2024-04-02T07:55:38.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.091+0000 7effbf400700 10 client.8595 handle_command_reply: tid=0 2024-04-02T07:55:38.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.091+0000 7efff8e6db80 1 client.8595 shutdown 2024-04-02T07:55:38.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.091+0000 7efff8e6db80 1 -- 172.21.15.89:0/1738030894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 msgr2=0x55ea7805bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.091+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1738030894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea7805bdb0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7effd00157b0 tx=0x7effd0001a00 comp rx=0 tx=0).stop 2024-04-02T07:55:38.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.091+0000 7efff8e6db80 1 -- 172.21.15.89:0/1738030894 shutdown_connections 2024-04-02T07:55:38.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.091+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1738030894 >> [v2:172.21.15.89:6834/417215494,v1:172.21.15.89:6836/417215494] conn(0x55ea780916a0 0x55ea78091a80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.091+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1738030894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78063830 0x55ea7805dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.091+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1738030894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea7805bdb0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.091+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1738030894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78036be0 0x55ea78062440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.091+0000 7effbf400700 0 client.8595 ms_handle_reset on v2:172.21.15.89:6834/417215494 2024-04-02T07:55:38.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.091+0000 7efff8e6db80 1 -- 172.21.15.89:0/1738030894 >> 172.21.15.89:0/1738030894 conn(0x55ea77f976a0 msgr2=0x55ea78038d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:38.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.092+0000 7efff8e6db80 1 -- 172.21.15.89:0/1738030894 shutdown_connections 2024-04-02T07:55:38.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.092+0000 7efff8e6db80 1 -- 172.21.15.89:0/1738030894 wait complete. 2024-04-02T07:55:38.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.092+0000 7efff8e6db80 20 client.8595 trim_cache size 0 max 16384 2024-04-02T07:55:38.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.110+0000 7efff8e6db80 1 Processor -- start 2024-04-02T07:55:38.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.110+0000 7efff8e6db80 1 -- start start 2024-04-02T07:55:38.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.111+0000 7efff8e6db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea78081c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.111+0000 7efff8e6db80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78036be0 0x55ea78082170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.111+0000 7efff8e6db80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78063830 0x55ea7805d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.111+0000 7efff8e6db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea77d7d390 con 0x55ea78034b60 2024-04-02T07:55:38.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.111+0000 7efff8e6db80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea7805bf20 con 0x55ea78063830 2024-04-02T07:55:38.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.111+0000 7efff8e6db80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea78063460 con 0x55ea78036be0 2024-04-02T07:55:38.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.111+0000 7effde800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea78081c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.111+0000 7effde800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea78081c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55010/0 (socket says 172.21.15.89:55010) 2024-04-02T07:55:38.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.111+0000 7effde800700 1 -- 172.21.15.89:0/3914696166 learned_addr learned my addr 172.21.15.89:0/3914696166 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:38.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.111+0000 7effde800700 1 -- 172.21.15.89:0/3914696166 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78036be0 msgr2=0x55ea78082170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:38.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.111+0000 7effde800700 1 --2- 172.21.15.89:0/3914696166 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78036be0 0x55ea78082170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.111+0000 7effde800700 1 -- 172.21.15.89:0/3914696166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78063830 msgr2=0x55ea7805d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:38.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.111+0000 7effde800700 1 --2- 172.21.15.89:0/3914696166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78063830 0x55ea7805d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.111+0000 7effde800700 1 -- 172.21.15.89:0/3914696166 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea77f4f010 con 0x55ea78034b60 2024-04-02T07:55:38.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.112+0000 7effde800700 1 --2- 172.21.15.89:0/3914696166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea78081c30 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7effd0007bd0 tx=0x7effd0022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785e56d411b558fe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:38.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.112+0000 7effcba00700 1 -- 172.21.15.89:0/3914696166 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0022640 con 0x55ea78034b60 2024-04-02T07:55:38.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.112+0000 7effcba00700 1 -- 172.21.15.89:0/3914696166 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd0031070 con 0x55ea78034b60 2024-04-02T07:55:38.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.112+0000 7effcba00700 1 -- 172.21.15.89:0/3914696166 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd002d460 con 0x55ea78034b60 2024-04-02T07:55:38.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.112+0000 7efff8e6db80 1 -- 172.21.15.89:0/3914696166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 msgr2=0x55ea78081c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.112+0000 7efff8e6db80 1 --2- 172.21.15.89:0/3914696166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea78081c30 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7effd0007bd0 tx=0x7effd0022000 comp rx=0 tx=0).stop 2024-04-02T07:55:38.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.112+0000 7efff8e6db80 1 -- 172.21.15.89:0/3914696166 shutdown_connections 2024-04-02T07:55:38.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.112+0000 7efff8e6db80 1 --2- 172.21.15.89:0/3914696166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78063830 0x55ea7805d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.112+0000 7efff8e6db80 1 --2- 172.21.15.89:0/3914696166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea78081c30 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.112+0000 7efff8e6db80 1 --2- 172.21.15.89:0/3914696166 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78036be0 0x55ea78082170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.112+0000 7efff8e6db80 1 -- 172.21.15.89:0/3914696166 >> 172.21.15.89:0/3914696166 conn(0x55ea77f976a0 msgr2=0x55ea7803a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:38.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.112+0000 7efff8e6db80 1 -- 172.21.15.89:0/3914696166 shutdown_connections 2024-04-02T07:55:38.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.113+0000 7efff8e6db80 1 -- 172.21.15.89:0/3914696166 wait complete. 2024-04-02T07:55:38.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.113+0000 7efff8e6db80 1 -- start start 2024-04-02T07:55:38.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.113+0000 7efff8e6db80 1 Processor -- start 2024-04-02T07:55:38.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.114+0000 7efff8e6db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea78060170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.114+0000 7efff8e6db80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78036be0 0x55ea78073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.114+0000 7efff8e6db80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78063830 0x55ea78074510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.114+0000 7efff8e6db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea7805cc20 con 0x55ea78034b60 2024-04-02T07:55:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.114+0000 7efff8e6db80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea7805f490 con 0x55ea78036be0 2024-04-02T07:55:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.114+0000 7efff8e6db80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea780cf370 con 0x55ea78063830 2024-04-02T07:55:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.114+0000 7efff8e6db80 10 client.0 _ll_register_callbacks cb 0x7effd0015430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:55:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.114+0000 7effde800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea78060170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.115+0000 7effde800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea78060170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55012/0 (socket says 172.21.15.89:55012) 2024-04-02T07:55:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.115+0000 7effde800700 1 -- 172.21.15.89:0/1109935574 learned_addr learned my addr 172.21.15.89:0/1109935574 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.115+0000 7effdde00700 1 --2- 172.21.15.89:0/1109935574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78036be0 0x55ea78073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.115+0000 7effdf200700 1 --2- 172.21.15.89:0/1109935574 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78063830 0x55ea78074510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.115+0000 7effde800700 1 -- 172.21.15.89:0/1109935574 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78063830 msgr2=0x55ea78074510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.115+0000 7effde800700 1 --2- 172.21.15.89:0/1109935574 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78063830 0x55ea78074510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.115+0000 7effde800700 1 -- 172.21.15.89:0/1109935574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78036be0 msgr2=0x55ea78073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.115+0000 7effde800700 1 --2- 172.21.15.89:0/1109935574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78036be0 0x55ea78073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.115+0000 7effde800700 1 -- 172.21.15.89:0/1109935574 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea77e690a0 con 0x55ea78034b60 2024-04-02T07:55:38.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.115+0000 7effde800700 1 --2- 172.21.15.89:0/1109935574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea78060170 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7effd0001930 tx=0x7effd000c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d3ef7d40d077bb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:38.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.115+0000 7effbf400700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:55:38.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.115+0000 7effbf400700 1 -- 172.21.15.89:0/1109935574 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd002d460 con 0x55ea78034b60 2024-04-02T07:55:38.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.115+0000 7effbf400700 1 -- 172.21.15.89:0/1109935574 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7effd001a050 con 0x55ea78034b60 2024-04-02T07:55:38.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.115+0000 7efff8e6db80 1 -- 172.21.15.89:0/1109935574 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7effd0015e90 con 0x55ea78034b60 2024-04-02T07:55:38.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.116+0000 7effbf400700 1 -- 172.21.15.89:0/1109935574 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7effd002d5c0 con 0x55ea78034b60 2024-04-02T07:55:38.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.116+0000 7efff8e6db80 10 client.8601 fetch_fsmap learned FSMap version 66 2024-04-02T07:55:38.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.116+0000 7efff8e6db80 1 -- 172.21.15.89:0/1109935574 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x55ea77ddd1d0 con 0x55ea78034b60 2024-04-02T07:55:38.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.116+0000 7effbf400700 1 -- 172.21.15.89:0/1109935574 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd0015e90 con 0x55ea78034b60 2024-04-02T07:55:38.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.116+0000 7effbf400700 1 -- 172.21.15.89:0/1109935574 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 66) v1 ==== 3114+0+0 (secure 0 0 0) 0x7effd002d720 con 0x55ea78034b60 2024-04-02T07:55:38.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.116+0000 7efff8e6db80 10 client.8601 fetch_fsmap finished waiting for FSMap version 66 2024-04-02T07:55:38.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.116+0000 7efff8e6db80 10 client.8601 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T07:55:38.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.116+0000 7efff8e6db80 20 client.8601 populate_metadata read hostname 'smithi089' 2024-04-02T07:55:38.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.116+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1109935574 >> [v2:172.21.15.89:6834/417215494,v1:172.21.15.89:6836/417215494] conn(0x55ea7807e560 0x55ea7807e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.116+0000 7efff8e6db80 4 client.8601 mds_command: new command op to 8436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T07:55:38.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.116+0000 7efff8e6db80 1 -- 172.21.15.89:0/1109935574 --> [v2:172.21.15.89:6834/417215494,v1:172.21.15.89:6836/417215494] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7effcc005d90 con 0x55ea7807e560 2024-04-02T07:55:38.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.116+0000 7effdde00700 1 --2- 172.21.15.89:0/1109935574 >> [v2:172.21.15.89:6834/417215494,v1:172.21.15.89:6836/417215494] conn(0x55ea7807e560 0x55ea7807e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7effdde00700 1 --2- 172.21.15.89:0/1109935574 >> [v2:172.21.15.89:6834/417215494,v1:172.21.15.89:6836/417215494] conn(0x55ea7807e560 0x55ea7807e940 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df5c3f96c5381c40 server_cookie=3c5023e07187018f in_seq=0 out_seq=0 2024-04-02T07:55:38.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7effbf400700 10 client.8601 ms_handle_connect on v2:172.21.15.89:6834/417215494 2024-04-02T07:55:38.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7effbf400700 1 -- 172.21.15.89:0/1109935574 <== mds.0 v2:172.21.15.89:6834/417215494 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x7effcc005d90 con 0x55ea7807e560 2024-04-02T07:55:38.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7effbf400700 10 client.8601 handle_command_reply: tid=0 2024-04-02T07:55:38.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7efff8e6db80 1 client.8601 shutdown 2024-04-02T07:55:38.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7efff8e6db80 1 -- 172.21.15.89:0/1109935574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 msgr2=0x55ea78060170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1109935574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea78060170 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7effd0001930 tx=0x7effd000c000 comp rx=0 tx=0).stop 2024-04-02T07:55:38.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7efff8e6db80 1 -- 172.21.15.89:0/1109935574 shutdown_connections 2024-04-02T07:55:38.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1109935574 >> [v2:172.21.15.89:6834/417215494,v1:172.21.15.89:6836/417215494] conn(0x55ea7807e560 0x55ea7807e940 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1109935574 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55ea78063830 0x55ea78074510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1109935574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea78034b60 0x55ea78060170 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7efff8e6db80 1 --2- 172.21.15.89:0/1109935574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ea78036be0 0x55ea78073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7efff8e6db80 1 -- 172.21.15.89:0/1109935574 >> 172.21.15.89:0/1109935574 conn(0x55ea77f976a0 msgr2=0x55ea7805f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:38.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7effbf400700 0 client.8601 ms_handle_reset on v2:172.21.15.89:6834/417215494 2024-04-02T07:55:38.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7efff8e6db80 1 -- 172.21.15.89:0/1109935574 shutdown_connections 2024-04-02T07:55:38.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.117+0000 7efff8e6db80 1 -- 172.21.15.89:0/1109935574 wait complete. 2024-04-02T07:55:38.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.118+0000 7efff8e6db80 20 client.8601 trim_cache size 0 max 16384 2024-04-02T07:55:38.123 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:55:38.123 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T07:55:38.124 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T07:55:38.124 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 8436, 2024-04-02T07:55:38.124 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T07:55:38.124 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T07:55:38.124 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T07:55:38.124 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 6.5445956709999997, 2024-04-02T07:55:38.124 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 66, 2024-04-02T07:55:38.124 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 140, 2024-04-02T07:55:38.124 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 140, 2024-04-02T07:55:38.124 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 18.275261152999999 2024-04-02T07:55:38.124 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:55:38.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.118+0000 7effe5400700 1 -- 172.21.15.89:0/4160030641 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7effc006adb0 msgr2=0x7effc006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.118+0000 7effe5400700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7effc006adb0 0x7effc006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7effd0000c00 tx=0x7effd0009040 comp rx=0 tx=0).stop 2024-04-02T07:55:38.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.118+0000 7effe5400700 1 -- 172.21.15.89:0/4160030641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe015ddb0 msgr2=0x7effe015e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.118+0000 7effe5400700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe015ddb0 0x7effe015e260 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7effd8017040 tx=0x7effd8009750 comp rx=0 tx=0).stop 2024-04-02T07:55:38.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.118+0000 7effe5400700 1 -- 172.21.15.89:0/4160030641 shutdown_connections 2024-04-02T07:55:38.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.118+0000 7effe5400700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7effc006adb0 0x7effc006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.118+0000 7effe5400700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe015ddb0 0x7effe015e260 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.118+0000 7effe5400700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effc4004d90 0x7effe00b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.118+0000 7effe5400700 1 --2- 172.21.15.89:0/4160030641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effe014bad0 0x7effe00b4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.118+0000 7effe5400700 1 -- 172.21.15.89:0/4160030641 >> 172.21.15.89:0/4160030641 conn(0x7effe008aff0 msgr2=0x7effe00b02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:38.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.118+0000 7effe5400700 1 -- 172.21.15.89:0/4160030641 shutdown_connections 2024-04-02T07:55:38.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.118+0000 7effe5400700 1 -- 172.21.15.89:0/4160030641 wait complete. 2024-04-02T07:55:38.138 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T07:55:38.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.326+0000 7f1312800700 1 Processor -- start 2024-04-02T07:55:38.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.326+0000 7f1312800700 1 -- start start 2024-04-02T07:55:38.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.327+0000 7f1312800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f130c160f10 0x7f130c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.327+0000 7f1312800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f130c161590 0x7f130c161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.328+0000 7f1312800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130c0ceda0 con 0x7f130c0cf230 2024-04-02T07:55:38.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.328+0000 7f1312800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f130c161f80 con 0x7f130c160f10 2024-04-02T07:55:38.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.328+0000 7f1312800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c1612f0 con 0x7f130c161590 2024-04-02T07:55:38.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.328+0000 7f130b400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f130c160f10 0x7f130c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.328+0000 7f130be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f130c161590 0x7f130c161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.328+0000 7f130b400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f130c160f10 0x7f130c0d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:47106/0 (socket says 172.21.15.89:47106) 2024-04-02T07:55:38.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.328+0000 7f130b400700 1 -- 172.21.15.89:0/2419864448 learned_addr learned my addr 172.21.15.89:0/2419864448 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:38.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.328+0000 7f130be00700 1 -- 172.21.15.89:0/2419864448 >> v2:172.21.15.113:3300/0 conn(0x7f130c160f10 msgr2=0x7f130c0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:38.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.328+0000 7f130be00700 1 --2- 172.21.15.89:0/2419864448 >> v2:172.21.15.113:3300/0 conn(0x7f130c160f10 0x7f130c0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.328+0000 7f130be00700 1 -- 172.21.15.89:0/2419864448 >> v1:172.21.15.113:6789/0 conn(0x7f130c0cf230 legacy=0x7f130c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.328+0000 7f130be00700 1 -- 172.21.15.89:0/2419864448 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c0ce470 con 0x7f130c161590 2024-04-02T07:55:38.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.329+0000 7f130be00700 1 --2- 172.21.15.89:0/2419864448 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f130c161590 0x7f130c161a40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1304009d90 tx=0x7f130400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9d9839627392ae5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:38.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.329+0000 7f130a000700 1 -- 172.21.15.89:0/2419864448 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130400e070 con 0x7f130c161590 2024-04-02T07:55:38.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.329+0000 7f130a000700 1 -- 172.21.15.89:0/2419864448 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130400bb20 con 0x7f130c161590 2024-04-02T07:55:38.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.329+0000 7f130a000700 1 -- 172.21.15.89:0/2419864448 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304012430 con 0x7f130c161590 2024-04-02T07:55:38.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.330+0000 7f1312800700 1 -- 172.21.15.89:0/2419864448 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f130c161590 msgr2=0x7f130c161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.330+0000 7f1312800700 1 --2- 172.21.15.89:0/2419864448 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f130c161590 0x7f130c161a40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1304009d90 tx=0x7f130400b330 comp rx=0 tx=0).stop 2024-04-02T07:55:38.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.330+0000 7f1312800700 1 -- 172.21.15.89:0/2419864448 shutdown_connections 2024-04-02T07:55:38.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.330+0000 7f1312800700 1 --2- 172.21.15.89:0/2419864448 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f130c161590 0x7f130c161a40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.330+0000 7f1312800700 1 --2- 172.21.15.89:0/2419864448 >> v2:172.21.15.113:3300/0 conn(0x7f130c160f10 0x7f130c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.330+0000 7f1312800700 1 -- 172.21.15.89:0/2419864448 >> 172.21.15.89:0/2419864448 conn(0x7f130c08aff0 msgr2=0x7f130c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:38.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.330+0000 7f1312800700 1 -- 172.21.15.89:0/2419864448 shutdown_connections 2024-04-02T07:55:38.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.330+0000 7f1312800700 1 -- 172.21.15.89:0/2419864448 wait complete. 2024-04-02T07:55:38.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.331+0000 7f1312800700 1 Processor -- start 2024-04-02T07:55:38.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.332+0000 7f1312800700 1 -- start start 2024-04-02T07:55:38.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f1312800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f130c0cf230 0x7f130c15f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f1312800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f130c160f10 0x7f130c15e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f1312800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130c161590 0x7f130c1586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f1312800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c0b7120 con 0x7f130c161590 2024-04-02T07:55:38.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f1312800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c15e860 con 0x7f130c160f10 2024-04-02T07:55:38.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f1312800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c15f670 con 0x7f130c0cf230 2024-04-02T07:55:38.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f130be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130c161590 0x7f130c1586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f130be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130c161590 0x7f130c1586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55014/0 (socket says 172.21.15.89:55014) 2024-04-02T07:55:38.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f130be00700 1 -- 172.21.15.89:0/1733769521 learned_addr learned my addr 172.21.15.89:0/1733769521 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:38.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f130b400700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f130c0cf230 0x7f130c15f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f130aa00700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f130c160f10 0x7f130c15e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f130be00700 1 -- 172.21.15.89:0/1733769521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f130c0cf230 msgr2=0x7f130c15f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f130be00700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f130c0cf230 0x7f130c15f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f130be00700 1 -- 172.21.15.89:0/1733769521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f130c160f10 msgr2=0x7f130c15e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f130be00700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f130c160f10 0x7f130c15e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f130be00700 1 -- 172.21.15.89:0/1733769521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1304009a70 con 0x7f130c161590 2024-04-02T07:55:38.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.333+0000 7f130b400700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f130c0cf230 0x7f130c15f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:38.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.334+0000 7f130be00700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130c161590 0x7f130c1586b0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f1304014040 tx=0x7f1304004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:38.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.334+0000 7f1303400700 1 -- 172.21.15.89:0/1733769521 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304012430 con 0x7f130c161590 2024-04-02T07:55:38.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.334+0000 7f1303400700 1 -- 172.21.15.89:0/1733769521 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1304020070 con 0x7f130c161590 2024-04-02T07:55:38.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.334+0000 7f1312800700 1 -- 172.21.15.89:0/1733769521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130c0b7370 con 0x7f130c161590 2024-04-02T07:55:38.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.334+0000 7f1303400700 1 -- 172.21.15.89:0/1733769521 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304003ea0 con 0x7f130c161590 2024-04-02T07:55:38.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.334+0000 7f1312800700 1 -- 172.21.15.89:0/1733769521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130c0b7930 con 0x7f130c161590 2024-04-02T07:55:38.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.335+0000 7f1312800700 1 -- 172.21.15.89:0/1733769521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130c067720 con 0x7f130c161590 2024-04-02T07:55:38.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.336+0000 7f1303400700 1 -- 172.21.15.89:0/1733769521 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1304004280 con 0x7f130c161590 2024-04-02T07:55:38.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.337+0000 7f1303400700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f12ec06adb0 0x7f12ec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:38.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.337+0000 7f1303400700 1 -- 172.21.15.89:0/1733769521 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f1304022070 con 0x7f130c161590 2024-04-02T07:55:38.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.337+0000 7f130b400700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f12ec06adb0 0x7f12ec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:38.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.338+0000 7f130b400700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f12ec06adb0 0x7f12ec06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009b80 tx=0x7f12fc008040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:38.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.342+0000 7f1303400700 1 -- 172.21.15.89:0/1733769521 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1304056340 con 0x7f130c161590 2024-04-02T07:55:38.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.588+0000 7f1312800700 1 -- 172.21.15.89:0/1733769521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f130c0cce20 con 0x7f130c161590 2024-04-02T07:55:38.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.589+0000 7f1303400700 1 -- 172.21.15.89:0/1733769521 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v66) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1304059960 con 0x7f130c161590 2024-04-02T07:55:38.591 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T07:55:38.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.591+0000 7f1312800700 1 -- 172.21.15.89:0/1733769521 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f12ec06adb0 msgr2=0x7f12ec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.591+0000 7f1312800700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f12ec06adb0 0x7f12ec06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009b80 tx=0x7f12fc008040 comp rx=0 tx=0).stop 2024-04-02T07:55:38.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.591+0000 7f1312800700 1 -- 172.21.15.89:0/1733769521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130c161590 msgr2=0x7f130c1586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:38.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.591+0000 7f1312800700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130c161590 0x7f130c1586b0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f1304014040 tx=0x7f1304004ad0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.591+0000 7f1312800700 1 -- 172.21.15.89:0/1733769521 shutdown_connections 2024-04-02T07:55:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.591+0000 7f1312800700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f12ec06adb0 0x7f12ec06d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.591+0000 7f1312800700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130c161590 0x7f130c1586b0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.591+0000 7f1312800700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f130c0cf230 0x7f130c15f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.591+0000 7f1312800700 1 --2- 172.21.15.89:0/1733769521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f130c160f10 0x7f130c15e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.591+0000 7f1312800700 1 -- 172.21.15.89:0/1733769521 >> 172.21.15.89:0/1733769521 conn(0x7f130c08aff0 msgr2=0x7f130c15c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.591+0000 7f1312800700 1 -- 172.21.15.89:0/1733769521 shutdown_connections 2024-04-02T07:55:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:38.591+0000 7f1312800700 1 -- 172.21.15.89:0/1733769521 wait complete. 2024-04-02T07:55:38.611 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T07:55:38.611 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T07:55:38.611 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T07:55:38.611 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T07:55:38.611 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T07:55:38.611 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T07:55:38.611 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T07:55:38.611 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T07:55:38.611 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:55:38.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:38.612 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T07:55:38.657 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:55:38.657 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:55:38.657 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:55:38.657 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:55:38.657 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T07:55:38.657 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:55:38.657 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:55:38.657 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:55:38.658 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T07:55:38.658 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:55:38.658 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:55:38.658 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T07:55:38.658 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:55:38.658 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:55:38.658 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T07:55:38.658 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:55:38.658 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T07:55:38.658 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 41290sec preferred_lft 41290sec 2024-04-02T07:55:38.658 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T07:55:38.658 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:55:38.658 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T07:55:38.659 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:55:38.659 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T07:55:38.659 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:55:38.805 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T07:55:38.805 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:55:38.805 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:55:38.805 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T07:55:38.805 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T07:55:38.805 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:55:38.805 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:55:38.805 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T07:55:38.805 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T07:55:38.805 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:55:38.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:38.806 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T07:55:38.870 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T07:55:38.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:38.873 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T07:55:38.919 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T07:55:38.920 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T07:55:38.920 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T07:55:38.920 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T07:55:38.920 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T07:55:38.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:38.921 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:55:38.921 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:55:38.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T07:55:38.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T07:55:38.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T07:55:38.921 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:55:39.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:39.124 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T07:55:39.168 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T07:55:39.168 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T07:55:39.169 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:55:39.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:39.169 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T07:55:39.276 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:55:39.276 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T07:55:39.320 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:55:39.321 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:55:39.321 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:55:39.403 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T07:55:39.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:39.403 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:55:39.543 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:55:39.543 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:55:39.570 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:55:39.570 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:55:39.570 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T07:55:39.570 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T07:55:39.570 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T07:55:39.575 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T07:55:39.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:39.575 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:55:39.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:39.643 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:55:39.741 INFO:teuthology.orchestra.run.smithi089.stdout:client8041 2024-04-02T07:55:39.741 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:55:39.741 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:39.785 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T07:55:40.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.017+0000 7f8a27200700 1 Processor -- start 2024-04-02T07:55:40.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.017+0000 7f8a27200700 1 -- start start 2024-04-02T07:55:40.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.018+0000 7f8a27200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8a201611b0 0x7f8a2015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:40.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.018+0000 7f8a27200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a2015f4d0 0x7f8a200ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:40.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.018+0000 7f8a27200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a200ce270 con 0x7f8a200d21f0 2024-04-02T07:55:40.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.018+0000 7f8a27200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a2015b600 con 0x7f8a201611b0 2024-04-02T07:55:40.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.018+0000 7f8a27200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a200d2760 con 0x7f8a2015f4d0 2024-04-02T07:55:40.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.018+0000 7f8a1fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8a201611b0 0x7f8a2015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:40.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.019+0000 7f8a1fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8a201611b0 0x7f8a2015ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55158/0 (socket says 172.21.15.89:55158) 2024-04-02T07:55:40.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.019+0000 7f8a1fe00700 1 -- 172.21.15.89:0/1007099766 learned_addr learned my addr 172.21.15.89:0/1007099766 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:40.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.019+0000 7f8a1f400700 1 --2- 172.21.15.89:0/1007099766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a2015f4d0 0x7f8a200ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:40.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.019+0000 7f8a1fe00700 1 -- 172.21.15.89:0/1007099766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a2015f4d0 msgr2=0x7f8a200ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:40.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.019+0000 7f8a1fe00700 1 --2- 172.21.15.89:0/1007099766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a2015f4d0 0x7f8a200ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:40.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.019+0000 7f8a1fe00700 1 -- 172.21.15.89:0/1007099766 >> v1:172.21.15.113:6789/0 conn(0x7f8a200d21f0 legacy=0x7f8a20161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:40.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.019+0000 7f8a1fe00700 1 -- 172.21.15.89:0/1007099766 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a200d2a40 con 0x7f8a201611b0 2024-04-02T07:55:40.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.019+0000 7f8a1fe00700 1 --2- 172.21.15.89:0/1007099766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a201611b0 0x7f8a2015ef90 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f8a100098f0 tx=0x7f8a10009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1870c56cc6fc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:40.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.019+0000 7f8a1ea00700 1 -- 172.21.15.89:0/1007099766 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a10011830 con 0x7f8a201611b0 2024-04-02T07:55:40.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.020+0000 7f8a1ea00700 1 -- 172.21.15.89:0/1007099766 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a10011990 con 0x7f8a201611b0 2024-04-02T07:55:40.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.020+0000 7f8a1ea00700 1 -- 172.21.15.89:0/1007099766 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a10011d20 con 0x7f8a201611b0 2024-04-02T07:55:40.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.020+0000 7f8a27200700 1 -- 172.21.15.89:0/1007099766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a201611b0 msgr2=0x7f8a2015ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:40.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.020+0000 7f8a27200700 1 --2- 172.21.15.89:0/1007099766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a201611b0 0x7f8a2015ef90 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f8a100098f0 tx=0x7f8a10009c00 comp rx=0 tx=0).stop 2024-04-02T07:55:40.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.020+0000 7f8a27200700 1 -- 172.21.15.89:0/1007099766 shutdown_connections 2024-04-02T07:55:40.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.020+0000 7f8a27200700 1 --2- 172.21.15.89:0/1007099766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a201611b0 0x7f8a2015ef90 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:40.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.020+0000 7f8a27200700 1 --2- 172.21.15.89:0/1007099766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a2015f4d0 0x7f8a200ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:40.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.020+0000 7f8a27200700 1 -- 172.21.15.89:0/1007099766 >> 172.21.15.89:0/1007099766 conn(0x7f8a2008aff0 msgr2=0x7f8a200d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:40.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.020+0000 7f8a27200700 1 -- 172.21.15.89:0/1007099766 shutdown_connections 2024-04-02T07:55:40.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.020+0000 7f8a27200700 1 -- 172.21.15.89:0/1007099766 wait complete. 2024-04-02T07:55:40.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.022+0000 7f8a27200700 1 Processor -- start 2024-04-02T07:55:40.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.022+0000 7f8a27200700 1 -- start start 2024-04-02T07:55:40.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.023+0000 7f8a27200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200d21f0 0x7f8a201f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:40.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.023+0000 7f8a27200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a2015f4d0 0x7f8a201f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:40.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.023+0000 7f8a27200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a201611b0 0x7f8a2009d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:40.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.023+0000 7f8a27200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a200caf10 con 0x7f8a200d21f0 2024-04-02T07:55:40.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.023+0000 7f8a27200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a200ca600 con 0x7f8a201611b0 2024-04-02T07:55:40.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.023+0000 7f8a27200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2009e130 con 0x7f8a2015f4d0 2024-04-02T07:55:40.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.023+0000 7f8a1fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200d21f0 0x7f8a201f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:40.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.023+0000 7f8a1fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200d21f0 0x7f8a201f6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55168/0 (socket says 172.21.15.89:55168) 2024-04-02T07:55:40.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.023+0000 7f8a1fe00700 1 -- 172.21.15.89:0/3468012894 learned_addr learned my addr 172.21.15.89:0/3468012894 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:40.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.023+0000 7f8a1f400700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a2015f4d0 0x7f8a201f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:40.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.023+0000 7f8a25200700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a201611b0 0x7f8a2009d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:40.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.023+0000 7f8a1fe00700 1 -- 172.21.15.89:0/3468012894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a2015f4d0 msgr2=0x7f8a201f69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:40.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.024+0000 7f8a1fe00700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a2015f4d0 0x7f8a201f69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:40.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.024+0000 7f8a1fe00700 1 -- 172.21.15.89:0/3468012894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a201611b0 msgr2=0x7f8a2009d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.024+0000 7f8a1fe00700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a201611b0 0x7f8a2009d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.024+0000 7f8a1fe00700 1 -- 172.21.15.89:0/3468012894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a10009550 con 0x7f8a200d21f0 2024-04-02T07:55:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.024+0000 7f8a1f400700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a2015f4d0 0x7f8a201f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.024+0000 7f8a25200700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a201611b0 0x7f8a2009d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.024+0000 7f8a1fe00700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200d21f0 0x7f8a201f6480 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009c40 tx=0x7f8a10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.024+0000 7f8a17e00700 1 -- 172.21.15.89:0/3468012894 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a100036a0 con 0x7f8a200d21f0 2024-04-02T07:55:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.024+0000 7f8a17e00700 1 -- 172.21.15.89:0/3468012894 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1001e070 con 0x7f8a200d21f0 2024-04-02T07:55:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.024+0000 7f8a17e00700 1 -- 172.21.15.89:0/3468012894 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a10003a40 con 0x7f8a200d21f0 2024-04-02T07:55:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.024+0000 7f8a27200700 1 -- 172.21.15.89:0/3468012894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a200ca850 con 0x7f8a200d21f0 2024-04-02T07:55:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.024+0000 7f8a27200700 1 -- 172.21.15.89:0/3468012894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2009d920 con 0x7f8a200d21f0 2024-04-02T07:55:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.025+0000 7f8a27200700 1 -- 172.21.15.89:0/3468012894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2015d610 con 0x7f8a200d21f0 2024-04-02T07:55:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.026+0000 7f8a17e00700 1 -- 172.21.15.89:0/3468012894 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8a10003ba0 con 0x7f8a200d21f0 2024-04-02T07:55:40.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.027+0000 7f8a17e00700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a0006adb0 0x7f8a0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:40.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.027+0000 7f8a17e00700 1 -- 172.21.15.89:0/3468012894 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f8a1008a640 con 0x7f8a200d21f0 2024-04-02T07:55:40.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.028+0000 7f8a1f400700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a0006adb0 0x7f8a0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:40.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.028+0000 7f8a1f400700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a0006adb0 0x7f8a0006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8a201f63e0 tx=0x7f8a0c008040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:40.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.032+0000 7f8a17e00700 1 -- 172.21.15.89:0/3468012894 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8a10055580 con 0x7f8a200d21f0 2024-04-02T07:55:40.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.260+0000 7f8a27200700 1 -- 172.21.15.89:0/3468012894 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a200cdf00 con 0x7f8a0006adb0 2024-04-02T07:55:40.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.262+0000 7f8a17e00700 1 -- 172.21.15.89:0/3468012894 <== mgr.8205 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f8a0c009980 con 0x7f8a0006adb0 2024-04-02T07:55:40.263 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:55:40.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:55:40.264 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T07:55:40.264 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:55:40.264 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:55:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.264+0000 7f8a27200700 1 -- 172.21.15.89:0/3468012894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a0006adb0 msgr2=0x7f8a0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.264+0000 7f8a27200700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a0006adb0 0x7f8a0006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8a201f63e0 tx=0x7f8a0c008040 comp rx=0 tx=0).stop 2024-04-02T07:55:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.264+0000 7f8a27200700 1 -- 172.21.15.89:0/3468012894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200d21f0 msgr2=0x7f8a201f6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.264+0000 7f8a27200700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200d21f0 0x7f8a201f6480 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009c40 tx=0x7f8a10004750 comp rx=0 tx=0).stop 2024-04-02T07:55:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.264+0000 7f8a27200700 1 -- 172.21.15.89:0/3468012894 shutdown_connections 2024-04-02T07:55:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.264+0000 7f8a27200700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8a0006adb0 0x7f8a0006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.264+0000 7f8a27200700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8a201611b0 0x7f8a2009d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.264+0000 7f8a27200700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a200d21f0 0x7f8a201f6480 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:40.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.264+0000 7f8a27200700 1 --2- 172.21.15.89:0/3468012894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8a2015f4d0 0x7f8a201f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:40.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.264+0000 7f8a27200700 1 -- 172.21.15.89:0/3468012894 >> 172.21.15.89:0/3468012894 conn(0x7f8a2008aff0 msgr2=0x7f8a2015da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:40.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.264+0000 7f8a27200700 1 -- 172.21.15.89:0/3468012894 shutdown_connections 2024-04-02T07:55:40.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.265+0000 7f8a27200700 1 -- 172.21.15.89:0/3468012894 wait complete. 2024-04-02T07:55:40.284 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T07:55:40.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.478+0000 7fc303400700 1 Processor -- start 2024-04-02T07:55:40.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.478+0000 7fc303400700 1 -- start start 2024-04-02T07:55:40.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.480+0000 7fc303400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc2fc15dfe0 0x7fc2fc0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:40.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.480+0000 7fc303400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2fc0cb310 con 0x7fc2fc0ce670 2024-04-02T07:55:40.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.480+0000 7fc303400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2fc0711a0 con 0x7fc2fc15a5c0 2024-04-02T07:55:40.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.480+0000 7fc303400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2fc0ceb00 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.480+0000 7fc301400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc2fc0ce670 0x7fc2fc0cba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:54046/0 (socket says 172.21.15.89:54046) 2024-04-02T07:55:40.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.480+0000 7fc301400700 1 -- 172.21.15.89:0/1411953183 learned_addr learned my addr 172.21.15.89:0/1411953183 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:40.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.480+0000 7fc2fbe00700 1 --2- 172.21.15.89:0/1411953183 >> v2:172.21.15.113:3300/0 conn(0x7fc2fc15dfe0 0x7fc2fc0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:40.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.480+0000 7fc2fb400700 1 -- 172.21.15.89:0/1411953183 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120736368 0 0) 0x7fc2fc0cb310 con 0x7fc2fc0ce670 2024-04-02T07:55:40.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.480+0000 7fc2fb400700 1 -- 172.21.15.89:0/1411953183 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2e40035b0 con 0x7fc2fc0ce670 2024-04-02T07:55:40.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.481+0000 7fc2fbe00700 1 -- 172.21.15.89:0/1411953183 >> v1:172.21.15.113:6789/0 conn(0x7fc2fc15a5c0 legacy=0x7fc2fc0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:40.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.481+0000 7fc2fb400700 1 -- 172.21.15.89:0/1411953183 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3198732763 0 0) 0x7fc2e40035b0 con 0x7fc2fc0ce670 2024-04-02T07:55:40.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.481+0000 7fc2fbe00700 1 -- 172.21.15.89:0/1411953183 >> v1:172.21.15.89:6789/0 conn(0x7fc2fc0ce670 legacy=0x7fc2fc0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:40.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.481+0000 7fc2fbe00700 1 -- 172.21.15.89:0/1411953183 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2fc071480 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.481+0000 7fc2fbe00700 1 --2- 172.21.15.89:0/1411953183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2fc15dfe0 0x7fc2fc0ce130 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009840 tx=0x7fc2e8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e18579807de773c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:40.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.481+0000 7fc2fb400700 1 -- 172.21.15.89:0/1411953183 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f40019f0 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.481+0000 7fc2fb400700 1 -- 172.21.15.89:0/1411953183 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e8004030 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.481+0000 7fc2fb400700 1 -- 172.21.15.89:0/1411953183 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e80043c0 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.482+0000 7fc303400700 1 -- 172.21.15.89:0/1411953183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2fc15dfe0 msgr2=0x7fc2fc0ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:40.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.482+0000 7fc303400700 1 --2- 172.21.15.89:0/1411953183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2fc15dfe0 0x7fc2fc0ce130 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009840 tx=0x7fc2e8009b50 comp rx=0 tx=0).stop 2024-04-02T07:55:40.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.482+0000 7fc303400700 1 -- 172.21.15.89:0/1411953183 shutdown_connections 2024-04-02T07:55:40.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.482+0000 7fc303400700 1 --2- 172.21.15.89:0/1411953183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2fc15dfe0 0x7fc2fc0ce130 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:40.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.482+0000 7fc303400700 1 -- 172.21.15.89:0/1411953183 >> 172.21.15.89:0/1411953183 conn(0x7fc2fc0b0d70 msgr2=0x7fc2fc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:40.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.482+0000 7fc303400700 1 -- 172.21.15.89:0/1411953183 shutdown_connections 2024-04-02T07:55:40.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.482+0000 7fc303400700 1 -- 172.21.15.89:0/1411953183 wait complete. 2024-04-02T07:55:40.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.484+0000 7fc303400700 1 Processor -- start 2024-04-02T07:55:40.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.484+0000 7fc303400700 1 -- start start 2024-04-02T07:55:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc303400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2fc0ce670 0x7fc2fc14b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc303400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2fc15a5c0 0x7fc2fc14a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc303400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2fc15dfe0 0x7fc2fc155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc303400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc15bce0 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc303400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc15d980 con 0x7fc2fc15a5c0 2024-04-02T07:55:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc303400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc14add0 con 0x7fc2fc0ce670 2024-04-02T07:55:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc301400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2fc15dfe0 0x7fc2fc155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc301400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2fc15dfe0 0x7fc2fc155630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55172/0 (socket says 172.21.15.89:55172) 2024-04-02T07:55:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc301400700 1 -- 172.21.15.89:0/3217363763 learned_addr learned my addr 172.21.15.89:0/3217363763 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc2fbe00700 1 --2- 172.21.15.89:0/3217363763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2fc15a5c0 0x7fc2fc14a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc300a00700 1 --2- 172.21.15.89:0/3217363763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2fc0ce670 0x7fc2fc14b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc301400700 1 -- 172.21.15.89:0/3217363763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2fc0ce670 msgr2=0x7fc2fc14b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc301400700 1 --2- 172.21.15.89:0/3217363763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2fc0ce670 0x7fc2fc14b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc301400700 1 -- 172.21.15.89:0/3217363763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2fc15a5c0 msgr2=0x7fc2fc14a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc301400700 1 --2- 172.21.15.89:0/3217363763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2fc15a5c0 0x7fc2fc14a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.485+0000 7fc301400700 1 -- 172.21.15.89:0/3217363763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e80094a0 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.486+0000 7fc301400700 1 --2- 172.21.15.89:0/3217363763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2fc15dfe0 0x7fc2fc155630 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400cc70 tx=0x7fc2f400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.486+0000 7fc2f8c00700 1 -- 172.21.15.89:0/3217363763 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f400ab70 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.486+0000 7fc2f8c00700 1 -- 172.21.15.89:0/3217363763 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f400acd0 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.486+0000 7fc303400700 1 -- 172.21.15.89:0/3217363763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2fc15ba00 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.486+0000 7fc2f8c00700 1 -- 172.21.15.89:0/3217363763 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4007310 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.486+0000 7fc303400700 1 -- 172.21.15.89:0/3217363763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2fc15d6a0 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.487+0000 7fc303400700 1 -- 172.21.15.89:0/3217363763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2fc067720 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.488+0000 7fc2f8c00700 1 -- 172.21.15.89:0/3217363763 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 78) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc2f4007470 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.489+0000 7fc2f8c00700 1 --2- 172.21.15.89:0/3217363763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc2dc06adb0 0x7fc2dc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:40.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.489+0000 7fc2f8c00700 1 -- 172.21.15.89:0/3217363763 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fc2f40891e0 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.489+0000 7fc300a00700 1 --2- 172.21.15.89:0/3217363763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc2dc06adb0 0x7fc2dc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:40.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.489+0000 7fc300a00700 1 --2- 172.21.15.89:0/3217363763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc2dc06adb0 0x7fc2dc06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc154860 tx=0x7fc2f0009040 comp rx=0 tx=0).ready entity=mgr.8205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:40.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.493+0000 7fc2f8c00700 1 -- 172.21.15.89:0/3217363763 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc2f4054120 con 0x7fc2fc15dfe0 2024-04-02T07:55:40.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:40.752+0000 7fc303400700 1 -- 172.21.15.89:0/3217363763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fc2fc0b1ad0 con 0x7fc2fc15dfe0 2024-04-02T07:55:41.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.048+0000 7fc2f8c00700 1 -- 172.21.15.89:0/3217363763 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 79) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fc2f40544f0 con 0x7fc2fc15dfe0 2024-04-02T07:55:41.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.049+0000 7fc2f8c00700 1 -- 172.21.15.89:0/3217363763 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v79) v1 ==== 92+0+0 (secure 0 0 0) 0x7fc2f4074270 con 0x7fc2fc15dfe0 2024-04-02T07:55:41.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.052+0000 7fc303400700 1 -- 172.21.15.89:0/3217363763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc2dc06adb0 msgr2=0x7fc2dc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:41.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.052+0000 7fc303400700 1 --2- 172.21.15.89:0/3217363763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc2dc06adb0 0x7fc2dc06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc154860 tx=0x7fc2f0009040 comp rx=0 tx=0).stop 2024-04-02T07:55:41.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.052+0000 7fc303400700 1 -- 172.21.15.89:0/3217363763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2fc15dfe0 msgr2=0x7fc2fc155630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:41.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.052+0000 7fc303400700 1 --2- 172.21.15.89:0/3217363763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2fc15dfe0 0x7fc2fc155630 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400cc70 tx=0x7fc2f400cf80 comp rx=0 tx=0).stop 2024-04-02T07:55:41.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.052+0000 7fc303400700 1 -- 172.21.15.89:0/3217363763 shutdown_connections 2024-04-02T07:55:41.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.052+0000 7fc303400700 1 --2- 172.21.15.89:0/3217363763 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc2dc06adb0 0x7fc2dc06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.052+0000 7fc303400700 1 --2- 172.21.15.89:0/3217363763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2fc15dfe0 0x7fc2fc155630 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.052+0000 7fc303400700 1 --2- 172.21.15.89:0/3217363763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2fc0ce670 0x7fc2fc14b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.052+0000 7fc303400700 1 --2- 172.21.15.89:0/3217363763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2fc15a5c0 0x7fc2fc14a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.052+0000 7fc303400700 1 -- 172.21.15.89:0/3217363763 >> 172.21.15.89:0/3217363763 conn(0x7fc2fc0b0d70 msgr2=0x7fc2fc0cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:41.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.052+0000 7fc303400700 1 -- 172.21.15.89:0/3217363763 shutdown_connections 2024-04-02T07:55:41.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.052+0000 7fc303400700 1 -- 172.21.15.89:0/3217363763 wait complete. 2024-04-02T07:55:41.073 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T07:55:41.269 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:41.269+0000 7fc2c906d240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:55:41.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.274+0000 7f345d400700 1 Processor -- start 2024-04-02T07:55:41.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.274+0000 7f345d400700 1 -- start start 2024-04-02T07:55:41.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.276+0000 7f345d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f345815d990 0x7f34580d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:41.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.276+0000 7f345d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345809a750 0x7f345806c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:41.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.276+0000 7f345d400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34580d0970 con 0x7f3458099a60 2024-04-02T07:55:41.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.276+0000 7f345d400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f345806c020 con 0x7f345815d990 2024-04-02T07:55:41.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.276+0000 7f345d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34580c81a0 con 0x7f345809a750 2024-04-02T07:55:41.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.276+0000 7f3455e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f345815d990 0x7f34580d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:41.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.276+0000 7f3455e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f345815d990 0x7f34580d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55204/0 (socket says 172.21.15.89:55204) 2024-04-02T07:55:41.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.276+0000 7f3455e00700 1 -- 172.21.15.89:0/2607027749 learned_addr learned my addr 172.21.15.89:0/2607027749 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:41.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.276+0000 7f3457200700 1 --2- 172.21.15.89:0/2607027749 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345809a750 0x7f345806c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:41.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.276+0000 7f3455e00700 1 -- 172.21.15.89:0/2607027749 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345809a750 msgr2=0x7f345806c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:41.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.276+0000 7f3455e00700 1 --2- 172.21.15.89:0/2607027749 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345809a750 0x7f345806c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.276+0000 7f3455e00700 1 -- 172.21.15.89:0/2607027749 >> v1:172.21.15.113:6789/0 conn(0x7f3458099a60 legacy=0x7f3458099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:41.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.276+0000 7f3455e00700 1 -- 172.21.15.89:0/2607027749 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34580d0690 con 0x7f345815d990 2024-04-02T07:55:41.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.277+0000 7f3455e00700 1 --2- 172.21.15.89:0/2607027749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345815d990 0x7f34580d0060 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f34440098f0 tx=0x7f3444009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bae48ff8fbe332d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:41.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.277+0000 7f3455400700 1 -- 172.21.15.89:0/2607027749 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444011830 con 0x7f345815d990 2024-04-02T07:55:41.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.277+0000 7f3455400700 1 -- 172.21.15.89:0/2607027749 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3444011990 con 0x7f345815d990 2024-04-02T07:55:41.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.277+0000 7f3455400700 1 -- 172.21.15.89:0/2607027749 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444011d20 con 0x7f345815d990 2024-04-02T07:55:41.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.277+0000 7f345d400700 1 -- 172.21.15.89:0/2607027749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345815d990 msgr2=0x7f34580d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:41.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.277+0000 7f345d400700 1 --2- 172.21.15.89:0/2607027749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345815d990 0x7f34580d0060 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f34440098f0 tx=0x7f3444009c00 comp rx=0 tx=0).stop 2024-04-02T07:55:41.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.277+0000 7f345d400700 1 -- 172.21.15.89:0/2607027749 shutdown_connections 2024-04-02T07:55:41.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.277+0000 7f345d400700 1 --2- 172.21.15.89:0/2607027749 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345809a750 0x7f345806c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.277+0000 7f345d400700 1 --2- 172.21.15.89:0/2607027749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345815d990 0x7f34580d0060 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.278+0000 7f345d400700 1 -- 172.21.15.89:0/2607027749 >> 172.21.15.89:0/2607027749 conn(0x7f34580b2e20 msgr2=0x7f34580b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:41.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.278+0000 7f345d400700 1 -- 172.21.15.89:0/2607027749 shutdown_connections 2024-04-02T07:55:41.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.278+0000 7f345d400700 1 -- 172.21.15.89:0/2607027749 wait complete. 2024-04-02T07:55:41.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.279+0000 7f345d400700 1 Processor -- start 2024-04-02T07:55:41.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.279+0000 7f345d400700 1 -- start start 2024-04-02T07:55:41.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.280+0000 7f345d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3458099a60 0x7f34580ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:41.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.280+0000 7f345d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345809a750 0x7f3458159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:41.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.280+0000 7f345d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345815d990 0x7f34580c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:41.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.280+0000 7f345d400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3458159e70 con 0x7f345815d990 2024-04-02T07:55:41.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.280+0000 7f345d400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34580c9640 con 0x7f3458099a60 2024-04-02T07:55:41.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.280+0000 7f345d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34580c9920 con 0x7f345809a750 2024-04-02T07:55:41.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.281+0000 7f3457200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345815d990 0x7f34580c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:41.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.281+0000 7f3457200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345815d990 0x7f34580c8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55218/0 (socket says 172.21.15.89:55218) 2024-04-02T07:55:41.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.281+0000 7f3457200700 1 -- 172.21.15.89:0/4268598888 learned_addr learned my addr 172.21.15.89:0/4268598888 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:41.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.281+0000 7f3455e00700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345809a750 0x7f3458159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:41.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.281+0000 7f3456800700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3458099a60 0x7f34580ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:41.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.281+0000 7f3457200700 1 -- 172.21.15.89:0/4268598888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345809a750 msgr2=0x7f3458159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:41.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.281+0000 7f3457200700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345809a750 0x7f3458159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.281+0000 7f3457200700 1 -- 172.21.15.89:0/4268598888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3458099a60 msgr2=0x7f34580ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:41.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.281+0000 7f3457200700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3458099a60 0x7f34580ca800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.281+0000 7f3457200700 1 -- 172.21.15.89:0/4268598888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3444009550 con 0x7f345815d990 2024-04-02T07:55:41.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.281+0000 7f3455e00700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345809a750 0x7f3458159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:55:41.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.281+0000 7f3457200700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345815d990 0x7f34580c8e10 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f345000efc0 tx=0x7f345000c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:41.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.281+0000 7f344ea00700 1 -- 172.21.15.89:0/4268598888 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345000cad0 con 0x7f345815d990 2024-04-02T07:55:41.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.282+0000 7f344ea00700 1 -- 172.21.15.89:0/4268598888 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345000f040 con 0x7f345815d990 2024-04-02T07:55:41.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.282+0000 7f345d400700 1 -- 172.21.15.89:0/4268598888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34580c9c00 con 0x7f345815d990 2024-04-02T07:55:41.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.282+0000 7f344ea00700 1 -- 172.21.15.89:0/4268598888 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345000ce70 con 0x7f345815d990 2024-04-02T07:55:41.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.282+0000 7f345d400700 1 -- 172.21.15.89:0/4268598888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34580c9350 con 0x7f345815d990 2024-04-02T07:55:41.285 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:41.282+0000 7fe7a12e9240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:55:41.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.283+0000 7f345d400700 1 -- 172.21.15.89:0/4268598888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3458067720 con 0x7f345815d990 2024-04-02T07:55:41.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.284+0000 7f344ea00700 1 -- 172.21.15.89:0/4268598888 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 79) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f3450014480 con 0x7f345815d990 2024-04-02T07:55:41.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.285+0000 7f344ea00700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f343806ae00 0x7f343806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:41.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.285+0000 7f3456800700 1 -- 172.21.15.89:0/4268598888 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f343806ae00 msgr2=0x7f343806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:41.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.285+0000 7f344ea00700 1 -- 172.21.15.89:0/4268598888 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f3450089f80 con 0x7f345815d990 2024-04-02T07:55:41.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.285+0000 7f3456800700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f343806ae00 0x7f343806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:41.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.290+0000 7f344ea00700 1 -- 172.21.15.89:0/4268598888 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3450054d30 con 0x7f345815d990 2024-04-02T07:55:41.455 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:41.454+0000 7fc2c906d240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:55:41.474 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:41.472+0000 7fe7a12e9240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:55:41.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.486+0000 7f3456800700 1 -- 172.21.15.89:0/4268598888 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f343806ae00 msgr2=0x7f343806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:41.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.486+0000 7f3456800700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f343806ae00 0x7f343806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:41.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.536+0000 7f345d400700 1 -- 172.21.15.89:0/4268598888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f34580707c0 con 0x7f345815d990 2024-04-02T07:55:41.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.540+0000 7f344ea00700 1 -- 172.21.15.89:0/4268598888 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3450058350 con 0x7f345815d990 2024-04-02T07:55:41.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.542+0000 7f345d400700 1 -- 172.21.15.89:0/4268598888 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f343806ae00 msgr2=0x7f343806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:41.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.542+0000 7f345d400700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f343806ae00 0x7f343806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.542+0000 7f345d400700 1 -- 172.21.15.89:0/4268598888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345815d990 msgr2=0x7f34580c8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:41.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.542+0000 7f345d400700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345815d990 0x7f34580c8e10 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f345000efc0 tx=0x7f345000c0c0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.542+0000 7f345d400700 1 -- 172.21.15.89:0/4268598888 shutdown_connections 2024-04-02T07:55:41.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.542+0000 7f345d400700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f343806ae00 0x7f343806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.542+0000 7f345d400700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3458099a60 0x7f34580ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.542+0000 7f345d400700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345815d990 0x7f34580c8e10 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.542+0000 7f345d400700 1 --2- 172.21.15.89:0/4268598888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345809a750 0x7f3458159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.542+0000 7f345d400700 1 -- 172.21.15.89:0/4268598888 >> 172.21.15.89:0/4268598888 conn(0x7f34580b2e20 msgr2=0x7f34580b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:41.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.542+0000 7f345d400700 1 -- 172.21.15.89:0/4268598888 shutdown_connections 2024-04-02T07:55:41.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.542+0000 7f345d400700 1 -- 172.21.15.89:0/4268598888 wait complete. 2024-04-02T07:55:41.563 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T07:55:41.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.759+0000 7f7b7fa00700 1 Processor -- start 2024-04-02T07:55:41.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.759+0000 7f7b7fa00700 1 -- start start 2024-04-02T07:55:41.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.760+0000 7f7b7fa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b781611b0 0x7f7b7815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:41.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.760+0000 7f7b7fa00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b780ce270 con 0x7f7b7815f4d0 2024-04-02T07:55:41.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.760+0000 7f7b7fa00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7815b600 con 0x7f7b780cee10 2024-04-02T07:55:41.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.760+0000 7f7b7fa00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b780d2760 con 0x7f7b781611b0 2024-04-02T07:55:41.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.760+0000 7f7b7d000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b781611b0 0x7f7b7815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:41.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.760+0000 7f7b7d000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b781611b0 0x7f7b7815ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55232/0 (socket says 172.21.15.89:55232) 2024-04-02T07:55:41.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.760+0000 7f7b7d000700 1 -- 172.21.15.89:0/2147991429 learned_addr learned my addr 172.21.15.89:0/2147991429 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:41.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.761+0000 7f7b7d000700 1 -- 172.21.15.89:0/2147991429 >> v1:172.21.15.113:6789/0 conn(0x7f7b780cee10 legacy=0x7f7b78161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:41.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.761+0000 7f7b7d000700 1 -- 172.21.15.89:0/2147991429 >> v1:172.21.15.89:6789/0 conn(0x7f7b7815f4d0 legacy=0x7f7b780ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:41.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.761+0000 7f7b7d000700 1 -- 172.21.15.89:0/2147991429 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b780d2a40 con 0x7f7b781611b0 2024-04-02T07:55:41.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.761+0000 7f7b77400700 1 -- 172.21.15.89:0/2147991429 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735533747 0 0) 0x7f7b7815b600 con 0x7f7b780cee10 2024-04-02T07:55:41.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.761+0000 7f7b7d000700 1 --2- 172.21.15.89:0/2147991429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b781611b0 0x7f7b7815ef90 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0098f0 tx=0x7f7b6c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5cc700b8a87cfd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:41.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.761+0000 7f7b77400700 1 -- 172.21.15.89:0/2147991429 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c00b8a0 con 0x7f7b781611b0 2024-04-02T07:55:41.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.761+0000 7f7b77400700 1 -- 172.21.15.89:0/2147991429 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c00ba00 con 0x7f7b781611b0 2024-04-02T07:55:41.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.761+0000 7f7b77400700 1 -- 172.21.15.89:0/2147991429 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c00bd90 con 0x7f7b781611b0 2024-04-02T07:55:41.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.762+0000 7f7b7fa00700 1 -- 172.21.15.89:0/2147991429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b781611b0 msgr2=0x7f7b7815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:41.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.762+0000 7f7b7fa00700 1 --2- 172.21.15.89:0/2147991429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b781611b0 0x7f7b7815ef90 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0098f0 tx=0x7f7b6c009c00 comp rx=0 tx=0).stop 2024-04-02T07:55:41.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.762+0000 7f7b7fa00700 1 -- 172.21.15.89:0/2147991429 shutdown_connections 2024-04-02T07:55:41.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.762+0000 7f7b7fa00700 1 --2- 172.21.15.89:0/2147991429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b781611b0 0x7f7b7815ef90 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.762+0000 7f7b7fa00700 1 -- 172.21.15.89:0/2147991429 >> 172.21.15.89:0/2147991429 conn(0x7f7b7808aff0 msgr2=0x7f7b780d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:41.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.762+0000 7f7b7fa00700 1 -- 172.21.15.89:0/2147991429 shutdown_connections 2024-04-02T07:55:41.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.762+0000 7f7b7fa00700 1 -- 172.21.15.89:0/2147991429 wait complete. 2024-04-02T07:55:41.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.764+0000 7f7b7fa00700 1 Processor -- start 2024-04-02T07:55:41.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.764+0000 7f7b7fa00700 1 -- start start 2024-04-02T07:55:41.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.765+0000 7f7b7fa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b780cee10 0x7f7b78159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:41.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.765+0000 7f7b7fa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b7815f4d0 0x7f7b781f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:41.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.765+0000 7f7b7fa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b781611b0 0x7f7b780c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:41.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.765+0000 7f7b7fa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780cc0b0 con 0x7f7b780cee10 2024-04-02T07:55:41.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.765+0000 7f7b7fa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780cb7a0 con 0x7f7b7815f4d0 2024-04-02T07:55:41.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.765+0000 7f7b7fa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780c8470 con 0x7f7b781611b0 2024-04-02T07:55:41.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.765+0000 7f7b7d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b780cee10 0x7f7b78159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:41.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.765+0000 7f7b7d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b780cee10 0x7f7b78159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55236/0 (socket says 172.21.15.89:55236) 2024-04-02T07:55:41.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.765+0000 7f7b7d000700 1 -- 172.21.15.89:0/580903335 learned_addr learned my addr 172.21.15.89:0/580903335 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:41.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.765+0000 7f7b77e00700 1 --2- 172.21.15.89:0/580903335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b7815f4d0 0x7f7b781f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:41.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.765+0000 7f7b7da00700 1 --2- 172.21.15.89:0/580903335 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b781611b0 0x7f7b780c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:41.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.766+0000 7f7b7d000700 1 -- 172.21.15.89:0/580903335 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b781611b0 msgr2=0x7f7b780c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:41.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.766+0000 7f7b7d000700 1 --2- 172.21.15.89:0/580903335 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b781611b0 0x7f7b780c7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.766+0000 7f7b7d000700 1 -- 172.21.15.89:0/580903335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b7815f4d0 msgr2=0x7f7b781f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:41.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.766+0000 7f7b7d000700 1 --2- 172.21.15.89:0/580903335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b7815f4d0 0x7f7b781f3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.766+0000 7f7b7d000700 1 -- 172.21.15.89:0/580903335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c009550 con 0x7f7b780cee10 2024-04-02T07:55:41.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.766+0000 7f7b7d000700 1 --2- 172.21.15.89:0/580903335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b780cee10 0x7f7b78159200 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c000c00 tx=0x7f7b6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:41.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.766+0000 7f7b74c00700 1 -- 172.21.15.89:0/580903335 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c01a900 con 0x7f7b780cee10 2024-04-02T07:55:41.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.766+0000 7f7b74c00700 1 -- 172.21.15.89:0/580903335 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c01f070 con 0x7f7b780cee10 2024-04-02T07:55:41.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.766+0000 7f7b74c00700 1 -- 172.21.15.89:0/580903335 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c01aca0 con 0x7f7b780cee10 2024-04-02T07:55:41.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.766+0000 7f7b7fa00700 1 -- 172.21.15.89:0/580903335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b780cb9f0 con 0x7f7b780cee10 2024-04-02T07:55:41.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.767+0000 7f7b7fa00700 1 -- 172.21.15.89:0/580903335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b781f3cd0 con 0x7f7b780cee10 2024-04-02T07:55:41.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.768+0000 7f7b7fa00700 1 -- 172.21.15.89:0/580903335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b781f3e60 con 0x7f7b780cee10 2024-04-02T07:55:41.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.769+0000 7f7b74c00700 1 -- 172.21.15.89:0/580903335 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 79) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f7b6c026480 con 0x7f7b780cee10 2024-04-02T07:55:41.770 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:41.771+0000 7fc2c906d240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:55:41.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.769+0000 7f7b74c00700 1 --2- 172.21.15.89:0/580903335 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7b5806ae00 0x7f7b5806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:41.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.770+0000 7f7b77e00700 1 -- 172.21.15.89:0/580903335 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7b5806ae00 msgr2=0x7f7b5806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:41.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.770+0000 7f7b74c00700 1 -- 172.21.15.89:0/580903335 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f7b6c021070 con 0x7f7b780cee10 2024-04-02T07:55:41.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.770+0000 7f7b77e00700 1 --2- 172.21.15.89:0/580903335 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7b5806ae00 0x7f7b5806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:41.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.774+0000 7f7b74c00700 1 -- 172.21.15.89:0/580903335 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7b6c055f40 con 0x7f7b780cee10 2024-04-02T07:55:41.806 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:41.805+0000 7fe7a12e9240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:55:41.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.971+0000 7f7b77e00700 1 -- 172.21.15.89:0/580903335 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7b5806ae00 msgr2=0x7f7b5806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:41.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.971+0000 7f7b77e00700 1 --2- 172.21.15.89:0/580903335 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7b5806ae00 0x7f7b5806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:41.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.988+0000 7f7b7fa00700 1 -- 172.21.15.89:0/580903335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f7b78067720 con 0x7f7b780cee10 2024-04-02T07:55:41.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.992+0000 7f7b74c00700 1 -- 172.21.15.89:0/580903335 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7b6c006d80 con 0x7f7b780cee10 2024-04-02T07:55:41.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.995+0000 7f7b7fa00700 1 -- 172.21.15.89:0/580903335 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7b5806ae00 msgr2=0x7f7b5806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:41.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.995+0000 7f7b7fa00700 1 --2- 172.21.15.89:0/580903335 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7b5806ae00 0x7f7b5806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.995+0000 7f7b7fa00700 1 -- 172.21.15.89:0/580903335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b780cee10 msgr2=0x7f7b78159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:41.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.995+0000 7f7b7fa00700 1 --2- 172.21.15.89:0/580903335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b780cee10 0x7f7b78159200 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c000c00 tx=0x7f7b6c004750 comp rx=0 tx=0).stop 2024-04-02T07:55:41.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.995+0000 7f7b7fa00700 1 -- 172.21.15.89:0/580903335 shutdown_connections 2024-04-02T07:55:41.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.995+0000 7f7b7fa00700 1 --2- 172.21.15.89:0/580903335 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7b5806ae00 0x7f7b5806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.995+0000 7f7b7fa00700 1 --2- 172.21.15.89:0/580903335 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b781611b0 0x7f7b780c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.995+0000 7f7b7fa00700 1 --2- 172.21.15.89:0/580903335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b780cee10 0x7f7b78159200 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.995+0000 7f7b7fa00700 1 --2- 172.21.15.89:0/580903335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b7815f4d0 0x7f7b781f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:41.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.995+0000 7f7b7fa00700 1 -- 172.21.15.89:0/580903335 >> 172.21.15.89:0/580903335 conn(0x7f7b7808aff0 msgr2=0x7f7b7806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:41.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.995+0000 7f7b7fa00700 1 -- 172.21.15.89:0/580903335 shutdown_connections 2024-04-02T07:55:41.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:41.995+0000 7f7b7fa00700 1 -- 172.21.15.89:0/580903335 wait complete. 2024-04-02T07:55:42.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:42.015 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2024-04-02T07:55:42.064 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2024-04-02T07:55:42.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.263+0000 7fa55d200700 1 Processor -- start 2024-04-02T07:55:42.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.263+0000 7fa55d200700 1 -- start start 2024-04-02T07:55:42.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.264+0000 7fa55d200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa5580cbb30 0x7fa5580cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:42.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.264+0000 7fa55d200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5580cc450 con 0x7fa5580cd020 2024-04-02T07:55:42.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.264+0000 7fa55d200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5580cdae0 con 0x7fa5580cd480 2024-04-02T07:55:42.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.264+0000 7fa55d200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5580cd8f0 con 0x7fa5580cbb30 2024-04-02T07:55:42.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.264+0000 7fa555400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa5580cd020 0x7fa5580b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:54066/0 (socket says 172.21.15.89:54066) 2024-04-02T07:55:42.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.264+0000 7fa555400700 1 -- 172.21.15.89:0/1756158146 learned_addr learned my addr 172.21.15.89:0/1756158146 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:42.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.264+0000 7fa555e00700 1 --2- 172.21.15.89:0/1756158146 >> v2:172.21.15.113:3300/0 conn(0x7fa5580cbb30 0x7fa5580cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:42.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.265+0000 7fa554a00700 1 -- 172.21.15.89:0/1756158146 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005877144 0 0) 0x7fa5580cc450 con 0x7fa5580cd020 2024-04-02T07:55:42.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.265+0000 7fa554a00700 1 -- 172.21.15.89:0/1756158146 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa53c0035b0 con 0x7fa5580cd020 2024-04-02T07:55:42.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.265+0000 7fa554a00700 1 -- 172.21.15.89:0/1756158146 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856234308 0 0) 0x7fa5580cdae0 con 0x7fa5580cd480 2024-04-02T07:55:42.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.265+0000 7fa554a00700 1 -- 172.21.15.89:0/1756158146 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5580cc450 con 0x7fa5580cd480 2024-04-02T07:55:42.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.265+0000 7fa555e00700 1 -- 172.21.15.89:0/1756158146 >> v1:172.21.15.113:6789/0 conn(0x7fa5580cd480 legacy=0x7fa55806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:42.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.265+0000 7fa555e00700 1 -- 172.21.15.89:0/1756158146 >> v1:172.21.15.89:6789/0 conn(0x7fa5580cd020 legacy=0x7fa5580b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:42.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.265+0000 7fa555e00700 1 -- 172.21.15.89:0/1756158146 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5580cddc0 con 0x7fa5580cbb30 2024-04-02T07:55:42.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.265+0000 7fa555e00700 1 --2- 172.21.15.89:0/1756158146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5580cbb30 0x7fa5580cbf10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa548009840 tx=0x7fa548009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abc4c7da5929520b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:42.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.265+0000 7fa554a00700 1 -- 172.21.15.89:0/1756158146 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa548011830 con 0x7fa5580cbb30 2024-04-02T07:55:42.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.265+0000 7fa554a00700 1 -- 172.21.15.89:0/1756158146 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa548011990 con 0x7fa5580cbb30 2024-04-02T07:55:42.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.265+0000 7fa554a00700 1 -- 172.21.15.89:0/1756158146 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa548011d20 con 0x7fa5580cbb30 2024-04-02T07:55:42.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.266+0000 7fa55d200700 1 -- 172.21.15.89:0/1756158146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5580cbb30 msgr2=0x7fa5580cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:42.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.266+0000 7fa55d200700 1 --2- 172.21.15.89:0/1756158146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5580cbb30 0x7fa5580cbf10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa548009840 tx=0x7fa548009b50 comp rx=0 tx=0).stop 2024-04-02T07:55:42.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.266+0000 7fa55d200700 1 -- 172.21.15.89:0/1756158146 shutdown_connections 2024-04-02T07:55:42.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.266+0000 7fa55d200700 1 --2- 172.21.15.89:0/1756158146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5580cbb30 0x7fa5580cbf10 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:42.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.266+0000 7fa55d200700 1 -- 172.21.15.89:0/1756158146 >> 172.21.15.89:0/1756158146 conn(0x7fa5580b0d70 msgr2=0x7fa5580b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:42.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.266+0000 7fa55d200700 1 -- 172.21.15.89:0/1756158146 shutdown_connections 2024-04-02T07:55:42.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.266+0000 7fa55d200700 1 -- 172.21.15.89:0/1756158146 wait complete. 2024-04-02T07:55:42.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.268+0000 7fa55d200700 1 Processor -- start 2024-04-02T07:55:42.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.268+0000 7fa55d200700 1 -- start start 2024-04-02T07:55:42.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.269+0000 7fa55d200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5580cbb30 0x7fa5580afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:42.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.269+0000 7fa55d200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5580cd020 0x7fa5580c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:42.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.269+0000 7fa55d200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5580cd480 0x7fa55814e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:42.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.269+0000 7fa55d200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5580c8d60 con 0x7fa5580cbb30 2024-04-02T07:55:42.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.269+0000 7fa55d200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5580c9040 con 0x7fa5580cd480 2024-04-02T07:55:42.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.269+0000 7fa55d200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5580c9320 con 0x7fa5580cd020 2024-04-02T07:55:42.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.269+0000 7fa555e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5580cbb30 0x7fa5580afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:42.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.269+0000 7fa555e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5580cbb30 0x7fa5580afea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55248/0 (socket says 172.21.15.89:55248) 2024-04-02T07:55:42.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.269+0000 7fa555e00700 1 -- 172.21.15.89:0/906460717 learned_addr learned my addr 172.21.15.89:0/906460717 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:42.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.269+0000 7fa555400700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5580cd020 0x7fa5580c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:42.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.269+0000 7fa556800700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5580cd480 0x7fa55814e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:42.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.270+0000 7fa555e00700 1 -- 172.21.15.89:0/906460717 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5580cd020 msgr2=0x7fa5580c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:42.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.270+0000 7fa555e00700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5580cd020 0x7fa5580c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:42.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.270+0000 7fa555e00700 1 -- 172.21.15.89:0/906460717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5580cd480 msgr2=0x7fa55814e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:42.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.270+0000 7fa555e00700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5580cd480 0x7fa55814e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:42.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.270+0000 7fa555e00700 1 -- 172.21.15.89:0/906460717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5480094a0 con 0x7fa5580cbb30 2024-04-02T07:55:42.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.270+0000 7fa555400700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5580cd020 0x7fa5580c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:42.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.270+0000 7fa555e00700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5580cbb30 0x7fa5580afea0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fa548014040 tx=0x7fa548004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:42.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.270+0000 7fa54e000700 1 -- 172.21.15.89:0/906460717 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54801a760 con 0x7fa5580cbb30 2024-04-02T07:55:42.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.270+0000 7fa54e000700 1 -- 172.21.15.89:0/906460717 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54801a8c0 con 0x7fa5580cbb30 2024-04-02T07:55:42.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.270+0000 7fa54e000700 1 -- 172.21.15.89:0/906460717 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54801ac30 con 0x7fa5580cbb30 2024-04-02T07:55:42.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.270+0000 7fa55d200700 1 -- 172.21.15.89:0/906460717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5580c9600 con 0x7fa5580cbb30 2024-04-02T07:55:42.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.270+0000 7fa55d200700 1 -- 172.21.15.89:0/906460717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5580b03e0 con 0x7fa5580cbb30 2024-04-02T07:55:42.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.271+0000 7fa55d200700 1 -- 172.21.15.89:0/906460717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5580c9b20 con 0x7fa5580cbb30 2024-04-02T07:55:42.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.272+0000 7fa54e000700 1 -- 172.21.15.89:0/906460717 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 79) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fa548022070 con 0x7fa5580cbb30 2024-04-02T07:55:42.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.273+0000 7fa54e000700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 0x7fa53806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:42.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.273+0000 7fa555400700 1 -- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 msgr2=0x7fa53806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:42.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.273+0000 7fa54e000700 1 -- 172.21.15.89:0/906460717 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fa54808a270 con 0x7fa5580cbb30 2024-04-02T07:55:42.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.273+0000 7fa555400700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 0x7fa53806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:42.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.282+0000 7fa54e000700 1 -- 172.21.15.89:0/906460717 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa548055260 con 0x7fa5580cbb30 2024-04-02T07:55:42.298 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:42.298+0000 7fc2c906d240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:55:42.365 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:42.363+0000 7fe7a12e9240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:55:42.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.474+0000 7fa555400700 1 -- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 msgr2=0x7fa53806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:42.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.474+0000 7fa555400700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 0x7fa53806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:42.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.518+0000 7fa55d200700 1 -- 172.21.15.89:0/906460717 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa55806df00 con 0x7fa53806ae00 2024-04-02T07:55:42.552 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:42.552+0000 7fc2c906d240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:55:42.625 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:42.624+0000 7fe7a12e9240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:55:42.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.875+0000 7fa555400700 1 -- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 msgr2=0x7fa53806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:42.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:42.875+0000 7fa555400700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 0x7fa53806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T07:55:43.006 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:43.006+0000 7fc2c906d240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:55:43.085 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:43.084+0000 7fe7a12e9240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:55:43.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:43.676+0000 7fa555400700 1 -- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 msgr2=0x7fa53806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:43.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:43.676+0000 7fa555400700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 0x7fa53806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T07:55:43.716 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:43.716+0000 7fc2c906d240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:55:43.787 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:43.786+0000 7fe7a12e9240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:55:43.807 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:43.807+0000 7fc2c906d240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:55:43.880 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:43.878+0000 7fe7a12e9240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:55:43.966 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:43.966+0000 7fc2c906d240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:55:44.037 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:44.035+0000 7fe7a12e9240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:55:44.180 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:44.180+0000 7fc2c906d240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:55:44.249 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:44.247+0000 7fe7a12e9240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:55:44.270 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:44.271+0000 7fc2c906d240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:55:44.341 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:44.339+0000 7fe7a12e9240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:55:44.368 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:44.369+0000 7fc2c906d240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:55:44.439 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:44.438+0000 7fe7a12e9240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:55:44.465 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:44.465+0000 7fc2c906d240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:55:44.538 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:44.536+0000 7fe7a12e9240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:55:44.562 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:44.562+0000 7fc2c906d240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:55:44.636 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:44.635+0000 7fe7a12e9240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:55:44.772 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:44.772+0000 7fc2c906d240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:55:44.849 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:44.848+0000 7fe7a12e9240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:55:45.027 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:45.027+0000 7fc2c906d240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:55:45.104 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:45.102+0000 7fe7a12e9240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:55:45.218 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:45.217+0000 7fc2c906d240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:55:45.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:45.278+0000 7fa555400700 1 -- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 msgr2=0x7fa53806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:45.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:45.278+0000 7fa555400700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 0x7fa53806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T07:55:45.295 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:45.293+0000 7fe7a12e9240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:55:45.309 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:45.309+0000 7fc2c906d240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:55:45.387 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:45.386+0000 7fe7a12e9240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:55:46.032 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:46.032+0000 7fc2c906d240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:55:46.116 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:46.114+0000 7fe7a12e9240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:55:46.243 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:46.242+0000 7fc2c906d240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:55:46.327 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:46.326+0000 7fe7a12e9240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:55:46.451 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:46.450+0000 7fc2c906d240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:55:46.534 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:46.533+0000 7fe7a12e9240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:55:46.923 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:46.922+0000 7fc2c906d240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:55:47.009 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:47.008+0000 7fe7a12e9240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:55:47.131 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:47.130+0000 7fc2c906d240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:55:47.219 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:47.218+0000 7fe7a12e9240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:55:47.570 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:47.570+0000 7fc2c906d240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:55:47.575 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:55:47.579 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:55:47.666 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:47.665+0000 7fe7a12e9240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:55:47.689 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:47.688+0000 7fc2c906d240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:55:47.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:47.747+0000 7fa54e000700 1 -- 172.21.15.89:0/906460717 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 80) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fa548055630 con 0x7fa5580cbb30 2024-04-02T07:55:47.783 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:47.781+0000 7fe7a12e9240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:55:47.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:47.791+0000 7fa54e000700 1 -- 172.21.15.89:0/906460717 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 81) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fa548088570 con 0x7fa5580cbb30 2024-04-02T07:55:47.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:47.792+0000 7fa54e000700 1 -- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 msgr2=0x7fa53806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:47.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:47.792+0000 7fa54e000700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 0x7fa53806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:47.956 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:47.954+0000 7fe70d600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:47.956 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:47.954+0000 7fe70d600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:47.956 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:47.954+0000 7fe70d600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:47.956 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:47.954+0000 7fe70d600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:47.956 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:47.954+0000 7fe70d600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:47.957 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:47.954+0000 7fe709a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:47.957 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:47.954+0000 7fe709a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:47.957 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:47.954+0000 7fe709a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:47.957 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:47.954+0000 7fe709a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:47.957 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:47.954+0000 7fe709a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:55:48.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.796+0000 7fa54e000700 1 -- 172.21.15.89:0/906460717 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mgrmap(e 82) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fa548055f00 con 0x7fa5580cbb30 2024-04-02T07:55:48.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.796+0000 7fa54e000700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 0x7fa53806d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:48.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.796+0000 7fa54e000700 1 -- 172.21.15.89:0/906460717 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5380283c0 con 0x7fa53806ae00 2024-04-02T07:55:48.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.797+0000 7fa555400700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 0x7fa53806d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:48.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.801+0000 7fa555400700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 0x7fa53806d240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa544007fd0 tx=0x7fa544009040 comp rx=0 tx=0).ready entity=mgr.7172 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:48.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.877+0000 7fa54e000700 1 -- 172.21.15.89:0/906460717 <== mgr.7172 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7fa5380283c0 con 0x7fa53806ae00 2024-04-02T07:55:48.878 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path snap_test_dir1/minutes 2024-04-02T07:55:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.879+0000 7fa55d200700 1 -- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 msgr2=0x7fa53806d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.879+0000 7fa55d200700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 0x7fa53806d240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa544007fd0 tx=0x7fa544009040 comp rx=0 tx=0).stop 2024-04-02T07:55:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.879+0000 7fa55d200700 1 -- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5580cbb30 msgr2=0x7fa5580afea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.879+0000 7fa55d200700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5580cbb30 0x7fa5580afea0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fa548014040 tx=0x7fa548004750 comp rx=0 tx=0).stop 2024-04-02T07:55:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.879+0000 7fa55d200700 1 -- 172.21.15.89:0/906460717 shutdown_connections 2024-04-02T07:55:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.879+0000 7fa55d200700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa53806ae00 0x7fa53806d240 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.879+0000 7fa55d200700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5580cd480 0x7fa55814e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.879+0000 7fa55d200700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5580cbb30 0x7fa5580afea0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.879+0000 7fa55d200700 1 --2- 172.21.15.89:0/906460717 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5580cd020 0x7fa5580c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.879+0000 7fa55d200700 1 -- 172.21.15.89:0/906460717 >> 172.21.15.89:0/906460717 conn(0x7fa5580b0d70 msgr2=0x7fa55806e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:48.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.879+0000 7fa55d200700 1 -- 172.21.15.89:0/906460717 shutdown_connections 2024-04-02T07:55:48.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:48.879+0000 7fa55d200700 1 -- 172.21.15.89:0/906460717 wait complete. 2024-04-02T07:55:48.900 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2024-04-02T07:55:48.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:48.900 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2024-04-02T07:55:48.946 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2024-04-02T07:55:49.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.180+0000 7f6d8a200700 1 Processor -- start 2024-04-02T07:55:49.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.180+0000 7f6d8a200700 1 -- start start 2024-04-02T07:55:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.181+0000 7f6d8a200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6d840ce230 0x7f6d840ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.181+0000 7f6d8a200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d840709c0 con 0x7f6d840b2070 2024-04-02T07:55:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.181+0000 7f6d8a200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d84070c80 con 0x7f6d8415d800 2024-04-02T07:55:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.181+0000 7f6d8a200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d840b40f0 con 0x7f6d840ce230 2024-04-02T07:55:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.181+0000 7f6d82a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6d840ce230 0x7f6d840ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.182+0000 7f6d82a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6d840ce230 0x7f6d840ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55570/0 (socket says 172.21.15.89:55570) 2024-04-02T07:55:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.182+0000 7f6d82a00700 1 -- 172.21.15.89:0/1988189657 learned_addr learned my addr 172.21.15.89:0/1988189657 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.182+0000 7f6d82a00700 1 -- 172.21.15.89:0/1988189657 >> v1:172.21.15.113:6789/0 conn(0x7f6d8415d800 legacy=0x7f6d840725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.182+0000 7f6d82a00700 1 -- 172.21.15.89:0/1988189657 >> v1:172.21.15.89:6789/0 conn(0x7f6d840b2070 legacy=0x7f6d8415dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.182+0000 7f6d82a00700 1 -- 172.21.15.89:0/1988189657 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d84070f60 con 0x7f6d840ce230 2024-04-02T07:55:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.182+0000 7f6d82a00700 1 --2- 172.21.15.89:0/1988189657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d840ce230 0x7f6d840ce6a0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000bb90 tx=0x7f6d7000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a853ca6788e7123 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.182+0000 7f6d82000700 1 -- 172.21.15.89:0/1988189657 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7000dc40 con 0x7f6d840ce230 2024-04-02T07:55:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.182+0000 7f6d82000700 1 -- 172.21.15.89:0/1988189657 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7000dda0 con 0x7f6d840ce230 2024-04-02T07:55:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.182+0000 7f6d82000700 1 -- 172.21.15.89:0/1988189657 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d70017630 con 0x7f6d840ce230 2024-04-02T07:55:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.183+0000 7f6d8a200700 1 -- 172.21.15.89:0/1988189657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d840ce230 msgr2=0x7f6d840ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.183+0000 7f6d8a200700 1 --2- 172.21.15.89:0/1988189657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d840ce230 0x7f6d840ce6a0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000bb90 tx=0x7f6d7000bea0 comp rx=0 tx=0).stop 2024-04-02T07:55:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.183+0000 7f6d8a200700 1 -- 172.21.15.89:0/1988189657 shutdown_connections 2024-04-02T07:55:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.183+0000 7f6d8a200700 1 --2- 172.21.15.89:0/1988189657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d840ce230 0x7f6d840ce6a0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.183+0000 7f6d8a200700 1 -- 172.21.15.89:0/1988189657 >> 172.21.15.89:0/1988189657 conn(0x7f6d840b0d70 msgr2=0x7f6d840b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.183+0000 7f6d8a200700 1 -- 172.21.15.89:0/1988189657 shutdown_connections 2024-04-02T07:55:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.183+0000 7f6d8a200700 1 -- 172.21.15.89:0/1988189657 wait complete. 2024-04-02T07:55:49.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.185+0000 7f6d8a200700 1 Processor -- start 2024-04-02T07:55:49.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.185+0000 7f6d8a200700 1 -- start start 2024-04-02T07:55:49.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.186+0000 7f6d8a200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d840b2070 0x7f6d840c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.186+0000 7f6d8a200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d840ce230 0x7f6d84152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.186+0000 7f6d8a200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d8415d800 0x7f6d84157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.186+0000 7f6d8a200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d841571e0 con 0x7f6d840ce230 2024-04-02T07:55:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.186+0000 7f6d8a200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d841589f0 con 0x7f6d840b2070 2024-04-02T07:55:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.186+0000 7f6d8a200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d84158cd0 con 0x7f6d8415d800 2024-04-02T07:55:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.186+0000 7f6d82a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d840ce230 0x7f6d84152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.186+0000 7f6d82a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d840ce230 0x7f6d84152a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55584/0 (socket says 172.21.15.89:55584) 2024-04-02T07:55:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.186+0000 7f6d82a00700 1 -- 172.21.15.89:0/2208123791 learned_addr learned my addr 172.21.15.89:0/2208123791 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.186+0000 7f6d83e00700 1 --2- 172.21.15.89:0/2208123791 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d8415d800 0x7f6d84157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.186+0000 7f6d83400700 1 --2- 172.21.15.89:0/2208123791 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d840b2070 0x7f6d840c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.186+0000 7f6d82a00700 1 -- 172.21.15.89:0/2208123791 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d8415d800 msgr2=0x7f6d84157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.186+0000 7f6d82a00700 1 --2- 172.21.15.89:0/2208123791 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d8415d800 0x7f6d84157d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.187+0000 7f6d82a00700 1 -- 172.21.15.89:0/2208123791 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d840b2070 msgr2=0x7f6d840c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.187+0000 7f6d82a00700 1 --2- 172.21.15.89:0/2208123791 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d840b2070 0x7f6d840c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:49.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.187+0000 7f6d82a00700 1 -- 172.21.15.89:0/2208123791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7000b7f0 con 0x7f6d840ce230 2024-04-02T07:55:49.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.187+0000 7f6d82a00700 1 --2- 172.21.15.89:0/2208123791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d840ce230 0x7f6d84152a10 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f6d8406c130 tx=0x7f6d70004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:49.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.187+0000 7f6d7b400700 1 -- 172.21.15.89:0/2208123791 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d70004050 con 0x7f6d840ce230 2024-04-02T07:55:49.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.187+0000 7f6d7b400700 1 -- 172.21.15.89:0/2208123791 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d700041b0 con 0x7f6d840ce230 2024-04-02T07:55:49.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.187+0000 7f6d7b400700 1 -- 172.21.15.89:0/2208123791 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d70004560 con 0x7f6d840ce230 2024-04-02T07:55:49.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.187+0000 7f6d8a200700 1 -- 172.21.15.89:0/2208123791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d840cac80 con 0x7f6d840ce230 2024-04-02T07:55:49.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.187+0000 7f6d8a200700 1 -- 172.21.15.89:0/2208123791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d840cb240 con 0x7f6d840ce230 2024-04-02T07:55:49.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.188+0000 7f6d8a200700 1 -- 172.21.15.89:0/2208123791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d84067720 con 0x7f6d840ce230 2024-04-02T07:55:49.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.189+0000 7f6d7b400700 1 -- 172.21.15.89:0/2208123791 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 82) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f6d70015070 con 0x7f6d840ce230 2024-04-02T07:55:49.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.190+0000 7f6d7b400700 1 --2- 172.21.15.89:0/2208123791 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6d6406ab30 0x7f6d6406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:49.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.191+0000 7f6d7b400700 1 -- 172.21.15.89:0/2208123791 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f6d7008aa60 con 0x7f6d840ce230 2024-04-02T07:55:49.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.191+0000 7f6d83400700 1 --2- 172.21.15.89:0/2208123791 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6d6406ab30 0x7f6d6406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:49.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.191+0000 7f6d83400700 1 --2- 172.21.15.89:0/2208123791 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6d6406ab30 0x7f6d6406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6d74000f60 tx=0x7f6d74009040 comp rx=0 tx=0).ready entity=mgr.7172 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:49.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.195+0000 7f6d7b400700 1 -- 172.21.15.89:0/2208123791 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6d70055d00 con 0x7f6d840ce230 2024-04-02T07:55:49.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.422+0000 7f6d8a200700 1 -- 172.21.15.89:0/2208123791 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d840cd9e0 con 0x7f6d6406ab30 2024-04-02T07:55:49.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.488+0000 7f6d7b400700 1 -- 172.21.15.89:0/2208123791 <== mgr.7172 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f6d74001340 con 0x7f6d6406ab30 2024-04-02T07:55:49.490 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path snap_test_dir1/hourly 2024-04-02T07:55:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.492+0000 7f6d8a200700 1 -- 172.21.15.89:0/2208123791 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6d6406ab30 msgr2=0x7f6d6406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.492+0000 7f6d8a200700 1 --2- 172.21.15.89:0/2208123791 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6d6406ab30 0x7f6d6406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6d74000f60 tx=0x7f6d74009040 comp rx=0 tx=0).stop 2024-04-02T07:55:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.492+0000 7f6d8a200700 1 -- 172.21.15.89:0/2208123791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d840ce230 msgr2=0x7f6d84152a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.492+0000 7f6d8a200700 1 --2- 172.21.15.89:0/2208123791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d840ce230 0x7f6d84152a10 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f6d8406c130 tx=0x7f6d70004750 comp rx=0 tx=0).stop 2024-04-02T07:55:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.492+0000 7f6d8a200700 1 -- 172.21.15.89:0/2208123791 shutdown_connections 2024-04-02T07:55:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.492+0000 7f6d8a200700 1 --2- 172.21.15.89:0/2208123791 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6d6406ab30 0x7f6d6406cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.492+0000 7f6d8a200700 1 --2- 172.21.15.89:0/2208123791 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d840b2070 0x7f6d840c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.492+0000 7f6d8a200700 1 --2- 172.21.15.89:0/2208123791 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d8415d800 0x7f6d84157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.492+0000 7f6d8a200700 1 --2- 172.21.15.89:0/2208123791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d840ce230 0x7f6d84152a10 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.492+0000 7f6d8a200700 1 -- 172.21.15.89:0/2208123791 >> 172.21.15.89:0/2208123791 conn(0x7f6d840b0d70 msgr2=0x7f6d840cbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.492+0000 7f6d8a200700 1 -- 172.21.15.89:0/2208123791 shutdown_connections 2024-04-02T07:55:49.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.492+0000 7f6d8a200700 1 -- 172.21.15.89:0/2208123791 wait complete. 2024-04-02T07:55:49.511 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2024-04-02T07:55:49.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:49.511 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2024-04-02T07:55:49.556 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2024-04-02T07:55:49.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.791+0000 7f84f9200700 1 Processor -- start 2024-04-02T07:55:49.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.791+0000 7f84f9200700 1 -- start start 2024-04-02T07:55:49.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.792+0000 7f84f9200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f84f40d16a0 0x7f84f40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:49.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.792+0000 7f84f9200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f4161420 con 0x7f84f415d9d0 2024-04-02T07:55:49.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.792+0000 7f84f9200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f4161700 con 0x7f84f40d2370 2024-04-02T07:55:49.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.792+0000 7f84f9200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f40d4340 con 0x7f84f40d16a0 2024-04-02T07:55:49.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.793+0000 7f84f1e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f84f415d9d0 0x7f84f40d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59482/0 (socket says 172.21.15.89:59482) 2024-04-02T07:55:49.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.793+0000 7f84f1e00700 1 -- 172.21.15.89:0/3944145878 learned_addr learned my addr 172.21.15.89:0/3944145878 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:49.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.793+0000 7f84ebe00700 1 --2- 172.21.15.89:0/3944145878 >> v2:172.21.15.113:3300/0 conn(0x7f84f40d16a0 0x7f84f40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:49.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.793+0000 7f84ebe00700 1 -- 172.21.15.89:0/3944145878 >> v1:172.21.15.113:6789/0 conn(0x7f84f40d2370 legacy=0x7f84f40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:49.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.793+0000 7f84ebe00700 1 -- 172.21.15.89:0/3944145878 >> v1:172.21.15.89:6789/0 conn(0x7f84f415d9d0 legacy=0x7f84f40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:49.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.793+0000 7f84ebe00700 1 -- 172.21.15.89:0/3944145878 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f41619e0 con 0x7f84f40d16a0 2024-04-02T07:55:49.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.794+0000 7f84ebe00700 1 --2- 172.21.15.89:0/3944145878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84f40d16a0 0x7f84f40d1e30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0097a0 tx=0x7f84dc009ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a40b40842d46e79d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:49.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.794+0000 7f84f1400700 1 -- 172.21.15.89:0/3944145878 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc005e30 con 0x7f84f40d16a0 2024-04-02T07:55:49.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.794+0000 7f84f1400700 1 -- 172.21.15.89:0/3944145878 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84dc004030 con 0x7f84f40d16a0 2024-04-02T07:55:49.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.794+0000 7f84f1400700 1 -- 172.21.15.89:0/3944145878 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc0043e0 con 0x7f84f40d16a0 2024-04-02T07:55:49.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.795+0000 7f84f9200700 1 -- 172.21.15.89:0/3944145878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84f40d16a0 msgr2=0x7f84f40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:49.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.795+0000 7f84f9200700 1 --2- 172.21.15.89:0/3944145878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84f40d16a0 0x7f84f40d1e30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0097a0 tx=0x7f84dc009ab0 comp rx=0 tx=0).stop 2024-04-02T07:55:49.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.795+0000 7f84f9200700 1 -- 172.21.15.89:0/3944145878 shutdown_connections 2024-04-02T07:55:49.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.795+0000 7f84f9200700 1 --2- 172.21.15.89:0/3944145878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84f40d16a0 0x7f84f40d1e30 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:49.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.795+0000 7f84f9200700 1 -- 172.21.15.89:0/3944145878 >> 172.21.15.89:0/3944145878 conn(0x7f84f408aff0 msgr2=0x7f84f40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:49.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.795+0000 7f84f9200700 1 -- 172.21.15.89:0/3944145878 shutdown_connections 2024-04-02T07:55:49.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.795+0000 7f84f9200700 1 -- 172.21.15.89:0/3944145878 wait complete. 2024-04-02T07:55:49.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.797+0000 7f84f9200700 1 Processor -- start 2024-04-02T07:55:49.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.797+0000 7f84f9200700 1 -- start start 2024-04-02T07:55:49.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.799+0000 7f84f9200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84f40d16a0 0x7f84f40c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:49.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.799+0000 7f84f9200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84f40d2370 0x7f84f40b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:49.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.799+0000 7f84f9200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84f415d9d0 0x7f84f40b4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:49.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.799+0000 7f84f9200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f40b77a0 con 0x7f84f415d9d0 2024-04-02T07:55:49.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.799+0000 7f84f9200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f40c9cc0 con 0x7f84f40d16a0 2024-04-02T07:55:49.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.799+0000 7f84f9200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f40b6b20 con 0x7f84f40d2370 2024-04-02T07:55:49.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.799+0000 7f84ebe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84f40d2370 0x7f84f40b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:49.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.799+0000 7f84f2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84f415d9d0 0x7f84f40b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:49.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.799+0000 7f84ebe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84f40d2370 0x7f84f40b6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:56936/0 (socket says 172.21.15.89:56936) 2024-04-02T07:55:49.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.799+0000 7f84ebe00700 1 -- 172.21.15.89:0/2965988587 learned_addr learned my addr 172.21.15.89:0/2965988587 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:49.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.800+0000 7f84f2800700 1 -- 172.21.15.89:0/2965988587 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84f40d2370 msgr2=0x7f84f40b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:49.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.800+0000 7f84f2800700 1 --2- 172.21.15.89:0/2965988587 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84f40d2370 0x7f84f40b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:49.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.800+0000 7f84f2800700 1 -- 172.21.15.89:0/2965988587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84f40d16a0 msgr2=0x7f84f40c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:49.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.800+0000 7f84f2800700 1 --2- 172.21.15.89:0/2965988587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84f40d16a0 0x7f84f40c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:49.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.800+0000 7f84f2800700 1 -- 172.21.15.89:0/2965988587 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc009430 con 0x7f84f415d9d0 2024-04-02T07:55:49.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.801+0000 7f84f2800700 1 --2- 172.21.15.89:0/2965988587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84f415d9d0 0x7f84f40b4320 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00c890 tx=0x7f84ec00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:49.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.802+0000 7f84ea000700 1 -- 172.21.15.89:0/2965988587 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec003e30 con 0x7f84f415d9d0 2024-04-02T07:55:49.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.802+0000 7f84ea000700 1 -- 172.21.15.89:0/2965988587 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ec005070 con 0x7f84f415d9d0 2024-04-02T07:55:49.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.802+0000 7f84ea000700 1 -- 172.21.15.89:0/2965988587 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec005400 con 0x7f84f415d9d0 2024-04-02T07:55:49.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.802+0000 7f84f9200700 1 -- 172.21.15.89:0/2965988587 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f40b6e00 con 0x7f84f415d9d0 2024-04-02T07:55:49.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.803+0000 7f84f9200700 1 -- 172.21.15.89:0/2965988587 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f40b73c0 con 0x7f84f415d9d0 2024-04-02T07:55:49.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.804+0000 7f84f9200700 1 -- 172.21.15.89:0/2965988587 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f40d1ca0 con 0x7f84f415d9d0 2024-04-02T07:55:49.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.804+0000 7f84ea000700 1 -- 172.21.15.89:0/2965988587 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 83) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f84ec005560 con 0x7f84f415d9d0 2024-04-02T07:55:49.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.805+0000 7f84ea000700 1 --2- 172.21.15.89:0/2965988587 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84d006ae00 0x7f84d006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:49.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.805+0000 7f84ea000700 1 -- 172.21.15.89:0/2965988587 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f84ec088c00 con 0x7f84f415d9d0 2024-04-02T07:55:49.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.806+0000 7f84f1e00700 1 --2- 172.21.15.89:0/2965988587 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84d006ae00 0x7f84d006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:49.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.806+0000 7f84f1e00700 1 --2- 172.21.15.89:0/2965988587 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84d006ae00 0x7f84d006d2c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f84e40015d0 tx=0x7f84e4007ad0 comp rx=0 tx=0).ready entity=mgr.7172 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:49.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:49.811+0000 7f84ea000700 1 -- 172.21.15.89:0/2965988587 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f84ec08f080 con 0x7f84f415d9d0 2024-04-02T07:55:50.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.050+0000 7f84f9200700 1 -- 172.21.15.89:0/2965988587 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) v1 -- 0x7f84f4161fa0 con 0x7f84d006ae00 2024-04-02T07:55:50.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.115+0000 7f84ea000700 1 -- 172.21.15.89:0/2965988587 <== mgr.7172 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f84e4001210 con 0x7f84d006ae00 2024-04-02T07:55:50.115 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path snap_test_dir1/daily 2024-04-02T07:55:50.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.117+0000 7f84f9200700 1 -- 172.21.15.89:0/2965988587 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84d006ae00 msgr2=0x7f84d006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:50.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.117+0000 7f84f9200700 1 --2- 172.21.15.89:0/2965988587 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84d006ae00 0x7f84d006d2c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f84e40015d0 tx=0x7f84e4007ad0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.117+0000 7f84f9200700 1 -- 172.21.15.89:0/2965988587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84f415d9d0 msgr2=0x7f84f40b4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:50.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.117+0000 7f84f9200700 1 --2- 172.21.15.89:0/2965988587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84f415d9d0 0x7f84f40b4320 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00c890 tx=0x7f84ec00cba0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.117+0000 7f84f9200700 1 -- 172.21.15.89:0/2965988587 shutdown_connections 2024-04-02T07:55:50.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.117+0000 7f84f9200700 1 --2- 172.21.15.89:0/2965988587 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f84d006ae00 0x7f84d006d2c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.117+0000 7f84f9200700 1 --2- 172.21.15.89:0/2965988587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84f40d16a0 0x7f84f40c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.117+0000 7f84f9200700 1 --2- 172.21.15.89:0/2965988587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84f415d9d0 0x7f84f40b4320 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.117+0000 7f84f9200700 1 --2- 172.21.15.89:0/2965988587 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f84f40d2370 0x7f84f40b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.117+0000 7f84f9200700 1 -- 172.21.15.89:0/2965988587 >> 172.21.15.89:0/2965988587 conn(0x7f84f408aff0 msgr2=0x7f84f40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:50.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.117+0000 7f84f9200700 1 -- 172.21.15.89:0/2965988587 shutdown_connections 2024-04-02T07:55:50.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.117+0000 7f84f9200700 1 -- 172.21.15.89:0/2965988587 wait complete. 2024-04-02T07:55:50.135 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2024-04-02T07:55:50.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:50.136 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2024-04-02T07:55:50.181 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2024-04-02T07:55:50.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.415+0000 7ff01de00700 1 Processor -- start 2024-04-02T07:55:50.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.415+0000 7ff01de00700 1 -- start start 2024-04-02T07:55:50.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.416+0000 7ff01de00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff01815d8a0 0x7ff0180b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:50.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.416+0000 7ff01de00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0180ced20 con 0x7ff0180d3eb0 2024-04-02T07:55:50.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.416+0000 7ff01de00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0180d4340 con 0x7ff0180cf1b0 2024-04-02T07:55:50.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.416+0000 7ff01de00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0180ce6d0 con 0x7ff01815d8a0 2024-04-02T07:55:50.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.416+0000 7ff016000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff0180d3eb0 0x7ff01806b890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59488/0 (socket says 172.21.15.89:59488) 2024-04-02T07:55:50.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.416+0000 7ff016000700 1 -- 172.21.15.89:0/1635325518 learned_addr learned my addr 172.21.15.89:0/1635325518 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:50.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.416+0000 7ff016a00700 1 --2- 172.21.15.89:0/1635325518 >> v2:172.21.15.113:3300/0 conn(0x7ff01815d8a0 0x7ff0180b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:50.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.417+0000 7ff015600700 1 -- 172.21.15.89:0/1635325518 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468078851 0 0) 0x7ff0180ced20 con 0x7ff0180d3eb0 2024-04-02T07:55:50.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.417+0000 7ff015600700 1 -- 172.21.15.89:0/1635325518 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feffc0035b0 con 0x7ff0180d3eb0 2024-04-02T07:55:50.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.417+0000 7ff016a00700 1 -- 172.21.15.89:0/1635325518 >> v1:172.21.15.113:6789/0 conn(0x7ff0180cf1b0 legacy=0x7ff0180d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:50.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.417+0000 7ff016a00700 1 -- 172.21.15.89:0/1635325518 >> v1:172.21.15.89:6789/0 conn(0x7ff0180d3eb0 legacy=0x7ff01806b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:50.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.417+0000 7ff016a00700 1 -- 172.21.15.89:0/1635325518 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0180ce9b0 con 0x7ff01815d8a0 2024-04-02T07:55:50.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.417+0000 7ff016a00700 1 --2- 172.21.15.89:0/1635325518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff01815d8a0 0x7ff0180b2990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff008009840 tx=0x7ff008009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46a772441be2296 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:50.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.417+0000 7ff015600700 1 -- 172.21.15.89:0/1635325518 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff008011830 con 0x7ff01815d8a0 2024-04-02T07:55:50.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.417+0000 7ff015600700 1 -- 172.21.15.89:0/1635325518 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff008011990 con 0x7ff01815d8a0 2024-04-02T07:55:50.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.417+0000 7ff015600700 1 -- 172.21.15.89:0/1635325518 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff008011d20 con 0x7ff01815d8a0 2024-04-02T07:55:50.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.418+0000 7ff01de00700 1 -- 172.21.15.89:0/1635325518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff01815d8a0 msgr2=0x7ff0180b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:50.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.418+0000 7ff01de00700 1 --2- 172.21.15.89:0/1635325518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff01815d8a0 0x7ff0180b2990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff008009840 tx=0x7ff008009b50 comp rx=0 tx=0).stop 2024-04-02T07:55:50.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.418+0000 7ff01de00700 1 -- 172.21.15.89:0/1635325518 shutdown_connections 2024-04-02T07:55:50.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.418+0000 7ff01de00700 1 --2- 172.21.15.89:0/1635325518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff01815d8a0 0x7ff0180b2990 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.418+0000 7ff01de00700 1 -- 172.21.15.89:0/1635325518 >> 172.21.15.89:0/1635325518 conn(0x7ff01808aff0 msgr2=0x7ff0180d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:50.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.418+0000 7ff01de00700 1 -- 172.21.15.89:0/1635325518 shutdown_connections 2024-04-02T07:55:50.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.418+0000 7ff01de00700 1 -- 172.21.15.89:0/1635325518 wait complete. 2024-04-02T07:55:50.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.420+0000 7ff01de00700 1 Processor -- start 2024-04-02T07:55:50.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.420+0000 7ff01de00700 1 -- start start 2024-04-02T07:55:50.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff01de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff0180cf1b0 0x7ff0180c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:50.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff01de00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff01815d8a0 0x7ff0180b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:50.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff01de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff018154920 0x7ff0180c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:50.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff01de00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0180b34a0 con 0x7ff018154920 2024-04-02T07:55:50.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff01de00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0180b7cf0 con 0x7ff01815d8a0 2024-04-02T07:55:50.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff01de00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0180c6ae0 con 0x7ff0180cf1b0 2024-04-02T07:55:50.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff017400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff018154920 0x7ff0180c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:50.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff017400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff018154920 0x7ff0180c65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55614/0 (socket says 172.21.15.89:55614) 2024-04-02T07:55:50.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff017400700 1 -- 172.21.15.89:0/786186014 learned_addr learned my addr 172.21.15.89:0/786186014 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:50.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff016000700 1 --2- 172.21.15.89:0/786186014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff01815d8a0 0x7ff0180b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:50.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff016a00700 1 --2- 172.21.15.89:0/786186014 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff0180cf1b0 0x7ff0180c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:50.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff017400700 1 -- 172.21.15.89:0/786186014 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff0180cf1b0 msgr2=0x7ff0180c7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:50.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff017400700 1 --2- 172.21.15.89:0/786186014 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff0180cf1b0 0x7ff0180c7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff017400700 1 -- 172.21.15.89:0/786186014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff01815d8a0 msgr2=0x7ff0180b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:50.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff017400700 1 --2- 172.21.15.89:0/786186014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff01815d8a0 0x7ff0180b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.421+0000 7ff017400700 1 -- 172.21.15.89:0/786186014 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0080094a0 con 0x7ff018154920 2024-04-02T07:55:50.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.422+0000 7ff017400700 1 --2- 172.21.15.89:0/786186014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff018154920 0x7ff0180c65a0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7ff01000cb80 tx=0x7ff01000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:50.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.422+0000 7ff00f400700 1 -- 172.21.15.89:0/786186014 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff010003c00 con 0x7ff018154920 2024-04-02T07:55:50.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.422+0000 7ff00f400700 1 -- 172.21.15.89:0/786186014 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff010003d60 con 0x7ff018154920 2024-04-02T07:55:50.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.422+0000 7ff01de00700 1 -- 172.21.15.89:0/786186014 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0180b36d0 con 0x7ff018154920 2024-04-02T07:55:50.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.422+0000 7ff00f400700 1 -- 172.21.15.89:0/786186014 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01000abb0 con 0x7ff018154920 2024-04-02T07:55:50.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.422+0000 7ff01de00700 1 -- 172.21.15.89:0/786186014 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0180b3c90 con 0x7ff018154920 2024-04-02T07:55:50.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.423+0000 7ff01de00700 1 -- 172.21.15.89:0/786186014 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0180b7e60 con 0x7ff018154920 2024-04-02T07:55:50.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.424+0000 7ff00f400700 1 -- 172.21.15.89:0/786186014 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 83) v1 ==== 89257+0+0 (secure 0 0 0) 0x7ff01000ad10 con 0x7ff018154920 2024-04-02T07:55:50.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.425+0000 7ff00f400700 1 --2- 172.21.15.89:0/786186014 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feff806ae30 0x7feff806d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:50.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.425+0000 7ff00f400700 1 -- 172.21.15.89:0/786186014 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7ff0100898d0 con 0x7ff018154920 2024-04-02T07:55:50.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.425+0000 7ff016a00700 1 --2- 172.21.15.89:0/786186014 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feff806ae30 0x7feff806d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:50.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.426+0000 7ff016a00700 1 --2- 172.21.15.89:0/786186014 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feff806ae30 0x7feff806d2f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff0080090b0 tx=0x7ff008008030 comp rx=0 tx=0).ready entity=mgr.7172 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:50.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.429+0000 7ff00f400700 1 -- 172.21.15.89:0/786186014 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff010054640 con 0x7ff018154920 2024-04-02T07:55:50.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.656+0000 7ff01de00700 1 -- 172.21.15.89:0/786186014 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0180ce380 con 0x7feff806ae30 2024-04-02T07:55:50.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.725+0000 7ff00f400700 1 -- 172.21.15.89:0/786186014 <== mgr.7172 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7ff008011b00 con 0x7feff806ae30 2024-04-02T07:55:50.725 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path snap_test_dir1/weekly 2024-04-02T07:55:50.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.727+0000 7ff01de00700 1 -- 172.21.15.89:0/786186014 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feff806ae30 msgr2=0x7feff806d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:50.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.727+0000 7ff01de00700 1 --2- 172.21.15.89:0/786186014 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feff806ae30 0x7feff806d2f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff0080090b0 tx=0x7ff008008030 comp rx=0 tx=0).stop 2024-04-02T07:55:50.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.727+0000 7ff01de00700 1 -- 172.21.15.89:0/786186014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff018154920 msgr2=0x7ff0180c65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:50.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.727+0000 7ff01de00700 1 --2- 172.21.15.89:0/786186014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff018154920 0x7ff0180c65a0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7ff01000cb80 tx=0x7ff01000cf40 comp rx=0 tx=0).stop 2024-04-02T07:55:50.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.727+0000 7ff01de00700 1 -- 172.21.15.89:0/786186014 shutdown_connections 2024-04-02T07:55:50.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.727+0000 7ff01de00700 1 --2- 172.21.15.89:0/786186014 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feff806ae30 0x7feff806d2f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.727+0000 7ff01de00700 1 --2- 172.21.15.89:0/786186014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff018154920 0x7ff0180c65a0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.727+0000 7ff01de00700 1 --2- 172.21.15.89:0/786186014 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff0180cf1b0 0x7ff0180c7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.727+0000 7ff01de00700 1 --2- 172.21.15.89:0/786186014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff01815d8a0 0x7ff0180b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.727+0000 7ff01de00700 1 -- 172.21.15.89:0/786186014 >> 172.21.15.89:0/786186014 conn(0x7ff01808aff0 msgr2=0x7ff01815df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:50.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.727+0000 7ff01de00700 1 -- 172.21.15.89:0/786186014 shutdown_connections 2024-04-02T07:55:50.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.728+0000 7ff01de00700 1 -- 172.21.15.89:0/786186014 wait complete. 2024-04-02T07:55:50.745 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2024-04-02T07:55:50.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:50.746 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2024-04-02T07:55:50.792 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2024-04-02T07:55:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.987+0000 7f189a800700 1 Processor -- start 2024-04-02T07:55:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.987+0000 7f189a800700 1 -- start start 2024-04-02T07:55:50.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.988+0000 7f189a800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f189415d9d0 0x7f18940d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:50.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.988+0000 7f189a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f18940d2540 0x7f18940ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:50.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.988+0000 7f189a800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1894161420 con 0x7f18940d16a0 2024-04-02T07:55:50.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.988+0000 7f189a800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f189415edc0 con 0x7f18940d2540 2024-04-02T07:55:50.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.988+0000 7f189a800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1894161700 con 0x7f189415d9d0 2024-04-02T07:55:50.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.988+0000 7f1893e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f18940d2540 0x7f18940ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:50.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.988+0000 7f1893e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f18940d2540 0x7f18940ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55616/0 (socket says 172.21.15.89:55616) 2024-04-02T07:55:50.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.988+0000 7f1893e00700 1 -- 172.21.15.89:0/2895120733 learned_addr learned my addr 172.21.15.89:0/2895120733 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:50.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.988+0000 7f1893400700 1 --2- 172.21.15.89:0/2895120733 >> v2:172.21.15.113:3300/0 conn(0x7f189415d9d0 0x7f18940d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:50.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.989+0000 7f1893e00700 1 -- 172.21.15.89:0/2895120733 >> v2:172.21.15.113:3300/0 conn(0x7f189415d9d0 msgr2=0x7f18940d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:50.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.989+0000 7f1893e00700 1 --2- 172.21.15.89:0/2895120733 >> v2:172.21.15.113:3300/0 conn(0x7f189415d9d0 0x7f18940d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.989+0000 7f1893e00700 1 -- 172.21.15.89:0/2895120733 >> v1:172.21.15.89:6789/0 conn(0x7f18940d16a0 legacy=0x7f18940d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:50.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.989+0000 7f1893e00700 1 -- 172.21.15.89:0/2895120733 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18941619e0 con 0x7f18940d2540 2024-04-02T07:55:50.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.989+0000 7f1893e00700 1 --2- 172.21.15.89:0/2895120733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18940d2540 0x7f18940ce360 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f18940d0ff0 tx=0x7f188c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28ce22e9479e0de server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:50.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.989+0000 7f1892000700 1 -- 172.21.15.89:0/2895120733 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c00e070 con 0x7f18940d2540 2024-04-02T07:55:50.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.989+0000 7f1892000700 1 -- 172.21.15.89:0/2895120733 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188c00b910 con 0x7f18940d2540 2024-04-02T07:55:50.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.989+0000 7f1892000700 1 -- 172.21.15.89:0/2895120733 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c00bcc0 con 0x7f18940d2540 2024-04-02T07:55:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.990+0000 7f189a800700 1 -- 172.21.15.89:0/2895120733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18940d2540 msgr2=0x7f18940ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.990+0000 7f189a800700 1 --2- 172.21.15.89:0/2895120733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18940d2540 0x7f18940ce360 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f18940d0ff0 tx=0x7f188c00b330 comp rx=0 tx=0).stop 2024-04-02T07:55:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.990+0000 7f189a800700 1 -- 172.21.15.89:0/2895120733 shutdown_connections 2024-04-02T07:55:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.990+0000 7f189a800700 1 --2- 172.21.15.89:0/2895120733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18940d2540 0x7f18940ce360 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.990+0000 7f189a800700 1 --2- 172.21.15.89:0/2895120733 >> v2:172.21.15.113:3300/0 conn(0x7f189415d9d0 0x7f18940d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.990+0000 7f189a800700 1 -- 172.21.15.89:0/2895120733 >> 172.21.15.89:0/2895120733 conn(0x7f189408aff0 msgr2=0x7f18940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.990+0000 7f189a800700 1 -- 172.21.15.89:0/2895120733 shutdown_connections 2024-04-02T07:55:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.990+0000 7f189a800700 1 -- 172.21.15.89:0/2895120733 wait complete. 2024-04-02T07:55:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.992+0000 7f189a800700 1 Processor -- start 2024-04-02T07:55:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.992+0000 7f189a800700 1 -- start start 2024-04-02T07:55:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f189a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f18940d16a0 0x7f189415bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f189a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18940d2540 0x7f18940b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f189a800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f189415d9d0 0x7f189415ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f189a800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189415c440 con 0x7f18940d2540 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f189a800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18940b6990 con 0x7f189415d9d0 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f189a800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18940b6c70 con 0x7f18940d16a0 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f1892a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18940d2540 0x7f18940b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f1892a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18940d2540 0x7f18940b6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55630/0 (socket says 172.21.15.89:55630) 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f1892a00700 1 -- 172.21.15.89:0/444836622 learned_addr learned my addr 172.21.15.89:0/444836622 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f1892a00700 1 -- 172.21.15.89:0/444836622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f18940d16a0 msgr2=0x7f189415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f1892a00700 1 --2- 172.21.15.89:0/444836622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f18940d16a0 0x7f189415bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f1892a00700 1 -- 172.21.15.89:0/444836622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f189415d9d0 msgr2=0x7f189415ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f1892a00700 1 --2- 172.21.15.89:0/444836622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f189415d9d0 0x7f189415ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f1892a00700 1 -- 172.21.15.89:0/444836622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188c009b30 con 0x7f18940d2540 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.993+0000 7f1892a00700 1 --2- 172.21.15.89:0/444836622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18940d2540 0x7f18940b6020 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f1880001eb0 tx=0x7f188000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.994+0000 7f188b400700 1 -- 172.21.15.89:0/444836622 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18800054e0 con 0x7f18940d2540 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.994+0000 7f188b400700 1 -- 172.21.15.89:0/444836622 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1880003680 con 0x7f18940d2540 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.994+0000 7f188b400700 1 -- 172.21.15.89:0/444836622 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18800039f0 con 0x7f18940d2540 2024-04-02T07:55:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.994+0000 7f189a800700 1 -- 172.21.15.89:0/444836622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18940b6f50 con 0x7f18940d2540 2024-04-02T07:55:50.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.994+0000 7f189a800700 1 -- 172.21.15.89:0/444836622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18940cb220 con 0x7f18940d2540 2024-04-02T07:55:50.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.995+0000 7f189a800700 1 -- 172.21.15.89:0/444836622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18940d1ca0 con 0x7f18940d2540 2024-04-02T07:55:50.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.996+0000 7f188b400700 1 -- 172.21.15.89:0/444836622 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 83) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f1880003b50 con 0x7f18940d2540 2024-04-02T07:55:50.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.997+0000 7f188b400700 1 --2- 172.21.15.89:0/444836622 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f187406ae00 0x7f187406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:50.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.997+0000 7f188b400700 1 -- 172.21.15.89:0/444836622 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f1880088bc0 con 0x7f18940d2540 2024-04-02T07:55:50.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.997+0000 7f1893400700 1 --2- 172.21.15.89:0/444836622 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f187406ae00 0x7f187406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:50.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:50.998+0000 7f1893400700 1 --2- 172.21.15.89:0/444836622 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f187406ae00 0x7f187406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1884006b20 tx=0x7f1884006ab0 comp rx=0 tx=0).ready entity=mgr.7172 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:51.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.001+0000 7f188b400700 1 -- 172.21.15.89:0/444836622 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f18800539d0 con 0x7f18940d2540 2024-04-02T07:55:51.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.231+0000 7f189a800700 1 -- 172.21.15.89:0/444836622 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f1894161fa0 con 0x7f187406ae00 2024-04-02T07:55:51.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.301+0000 7f188b400700 1 -- 172.21.15.89:0/444836622 <== mgr.7172 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f18840096c0 con 0x7f187406ae00 2024-04-02T07:55:51.301 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path snap_test_dir1/monthly 2024-04-02T07:55:51.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.303+0000 7f189a800700 1 -- 172.21.15.89:0/444836622 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f187406ae00 msgr2=0x7f187406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:51.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.303+0000 7f189a800700 1 --2- 172.21.15.89:0/444836622 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f187406ae00 0x7f187406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1884006b20 tx=0x7f1884006ab0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.303+0000 7f189a800700 1 -- 172.21.15.89:0/444836622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18940d2540 msgr2=0x7f18940b6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:51.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.303+0000 7f189a800700 1 --2- 172.21.15.89:0/444836622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18940d2540 0x7f18940b6020 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f1880001eb0 tx=0x7f188000c860 comp rx=0 tx=0).stop 2024-04-02T07:55:51.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.303+0000 7f189a800700 1 -- 172.21.15.89:0/444836622 shutdown_connections 2024-04-02T07:55:51.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.303+0000 7f189a800700 1 --2- 172.21.15.89:0/444836622 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f187406ae00 0x7f187406d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.303+0000 7f189a800700 1 --2- 172.21.15.89:0/444836622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f189415d9d0 0x7f189415ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.303+0000 7f189a800700 1 --2- 172.21.15.89:0/444836622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f18940d16a0 0x7f189415bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.303+0000 7f189a800700 1 --2- 172.21.15.89:0/444836622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18940d2540 0x7f18940b6020 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.303+0000 7f189a800700 1 -- 172.21.15.89:0/444836622 >> 172.21.15.89:0/444836622 conn(0x7f189408aff0 msgr2=0x7f1894160780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:51.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.303+0000 7f189a800700 1 -- 172.21.15.89:0/444836622 shutdown_connections 2024-04-02T07:55:51.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.303+0000 7f189a800700 1 -- 172.21.15.89:0/444836622 wait complete. 2024-04-02T07:55:51.322 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2024-04-02T07:55:51.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:51.322 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2024-04-02T07:55:51.368 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1Y 2024-04-02T07:55:51.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.561+0000 7f1293600700 1 Processor -- start 2024-04-02T07:55:51.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.562+0000 7f1293600700 1 -- start start 2024-04-02T07:55:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.563+0000 7f1293600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f128c15d9d0 0x7f128c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.563+0000 7f1293600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f128c0d16a0 0x7f128c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.563+0000 7f1293600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c161420 con 0x7f128c0d2370 2024-04-02T07:55:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.563+0000 7f1293600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f128c0d2820 con 0x7f128c0d16a0 2024-04-02T07:55:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.563+0000 7f1293600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128c161700 con 0x7f128c15d9d0 2024-04-02T07:55:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.563+0000 7f128be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f128c0d16a0 0x7f128c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:51.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.563+0000 7f128be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f128c0d16a0 0x7f128c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55644/0 (socket says 172.21.15.89:55644) 2024-04-02T07:55:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.563+0000 7f128be00700 1 -- 172.21.15.89:0/3867938342 learned_addr learned my addr 172.21.15.89:0/3867938342 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.563+0000 7f1290c00700 1 --2- 172.21.15.89:0/3867938342 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f128c15d9d0 0x7f128c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.563+0000 7f128be00700 1 -- 172.21.15.89:0/3867938342 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f128c15d9d0 msgr2=0x7f128c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.563+0000 7f128be00700 1 --2- 172.21.15.89:0/3867938342 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f128c15d9d0 0x7f128c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.563+0000 7f128be00700 1 -- 172.21.15.89:0/3867938342 >> v1:172.21.15.89:6789/0 conn(0x7f128c0d2370 legacy=0x7f128c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.563+0000 7f128be00700 1 -- 172.21.15.89:0/3867938342 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c1619e0 con 0x7f128c0d16a0 2024-04-02T07:55:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.563+0000 7f1290c00700 1 --2- 172.21.15.89:0/3867938342 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f128c15d9d0 0x7f128c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.564+0000 7f128be00700 1 --2- 172.21.15.89:0/3867938342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f128c0d16a0 0x7f128c0d1e30 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f1278006fd0 tx=0x7f1278009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a2c6e4a0d5cc0a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.564+0000 7f128b400700 1 -- 172.21.15.89:0/3867938342 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1278005e30 con 0x7f128c0d16a0 2024-04-02T07:55:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.564+0000 7f128b400700 1 -- 172.21.15.89:0/3867938342 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1278004030 con 0x7f128c0d16a0 2024-04-02T07:55:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.564+0000 7f128b400700 1 -- 172.21.15.89:0/3867938342 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12780043e0 con 0x7f128c0d16a0 2024-04-02T07:55:51.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.564+0000 7f1293600700 1 -- 172.21.15.89:0/3867938342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f128c0d16a0 msgr2=0x7f128c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:51.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.564+0000 7f1293600700 1 --2- 172.21.15.89:0/3867938342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f128c0d16a0 0x7f128c0d1e30 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f1278006fd0 tx=0x7f1278009bd0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.564+0000 7f1293600700 1 -- 172.21.15.89:0/3867938342 shutdown_connections 2024-04-02T07:55:51.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.564+0000 7f1293600700 1 --2- 172.21.15.89:0/3867938342 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f128c15d9d0 0x7f128c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.564+0000 7f1293600700 1 --2- 172.21.15.89:0/3867938342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f128c0d16a0 0x7f128c0d1e30 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.564+0000 7f1293600700 1 -- 172.21.15.89:0/3867938342 >> 172.21.15.89:0/3867938342 conn(0x7f128c08aff0 msgr2=0x7f128c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:51.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.565+0000 7f1293600700 1 -- 172.21.15.89:0/3867938342 shutdown_connections 2024-04-02T07:55:51.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.565+0000 7f1293600700 1 -- 172.21.15.89:0/3867938342 wait complete. 2024-04-02T07:55:51.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.566+0000 7f1293600700 1 Processor -- start 2024-04-02T07:55:51.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.566+0000 7f1293600700 1 -- start start 2024-04-02T07:55:51.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.567+0000 7f1293600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f128c0d16a0 0x7f128c151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:51.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.567+0000 7f1293600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f128c0d2370 0x7f128c0b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:51.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.567+0000 7f1293600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f128c15d9d0 0x7f128c0b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:51.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.567+0000 7f1293600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128c153830 con 0x7f128c0d16a0 2024-04-02T07:55:51.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.567+0000 7f1293600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128c153b10 con 0x7f128c15d9d0 2024-04-02T07:55:51.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.567+0000 7f1293600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128c153df0 con 0x7f128c0d2370 2024-04-02T07:55:51.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.568+0000 7f1290c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f128c0d16a0 0x7f128c151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:51.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.568+0000 7f1290c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f128c0d16a0 0x7f128c151dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55658/0 (socket says 172.21.15.89:55658) 2024-04-02T07:55:51.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.568+0000 7f1290c00700 1 -- 172.21.15.89:0/2346319262 learned_addr learned my addr 172.21.15.89:0/2346319262 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:51.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.568+0000 7f1290c00700 1 -- 172.21.15.89:0/2346319262 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f128c0d2370 msgr2=0x7f128c0b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:51.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.568+0000 7f1290c00700 1 --2- 172.21.15.89:0/2346319262 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f128c0d2370 0x7f128c0b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.568+0000 7f1290c00700 1 -- 172.21.15.89:0/2346319262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f128c15d9d0 msgr2=0x7f128c0b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:51.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.568+0000 7f1290c00700 1 --2- 172.21.15.89:0/2346319262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f128c15d9d0 0x7f128c0b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.568+0000 7f1290c00700 1 -- 172.21.15.89:0/2346319262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1278009500 con 0x7f128c0d16a0 2024-04-02T07:55:51.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.568+0000 7f1290c00700 1 --2- 172.21.15.89:0/2346319262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f128c0d16a0 0x7f128c151dc0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f128000e810 tx=0x7f128000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:51.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.568+0000 7f1288c00700 1 -- 172.21.15.89:0/2346319262 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128000c020 con 0x7f128c0d16a0 2024-04-02T07:55:51.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.568+0000 7f1288c00700 1 -- 172.21.15.89:0/2346319262 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1280004250 con 0x7f128c0d16a0 2024-04-02T07:55:51.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.568+0000 7f1288c00700 1 -- 172.21.15.89:0/2346319262 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12800036a0 con 0x7f128c0d16a0 2024-04-02T07:55:51.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.568+0000 7f1293600700 1 -- 172.21.15.89:0/2346319262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128c1540d0 con 0x7f128c0d16a0 2024-04-02T07:55:51.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.569+0000 7f1293600700 1 -- 172.21.15.89:0/2346319262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f128c0b84a0 con 0x7f128c0d16a0 2024-04-02T07:55:51.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.570+0000 7f1293600700 1 -- 172.21.15.89:0/2346319262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f128c0d1ca0 con 0x7f128c0d16a0 2024-04-02T07:55:51.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.571+0000 7f1288c00700 1 -- 172.21.15.89:0/2346319262 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 83) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f1280003800 con 0x7f128c0d16a0 2024-04-02T07:55:51.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.572+0000 7f1288c00700 1 --2- 172.21.15.89:0/2346319262 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f126c06ae00 0x7f126c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:51.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.572+0000 7f1288c00700 1 -- 172.21.15.89:0/2346319262 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f1280089340 con 0x7f128c0d16a0 2024-04-02T07:55:51.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.572+0000 7f128be00700 1 --2- 172.21.15.89:0/2346319262 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f126c06ae00 0x7f126c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:51.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.573+0000 7f128be00700 1 --2- 172.21.15.89:0/2346319262 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f126c06ae00 0x7f126c06d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f128c0d1ae0 tx=0x7f1278010000 comp rx=0 tx=0).ready entity=mgr.7172 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:51.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.576+0000 7f1288c00700 1 -- 172.21.15.89:0/2346319262 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1280054020 con 0x7f128c0d16a0 2024-04-02T07:55:51.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.803+0000 7f1293600700 1 -- 172.21.15.89:0/2346319262 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1Y", "target": ["mon-mgr", ""]}) v1 -- 0x7f128c153c80 con 0x7f126c06ae00 2024-04-02T07:55:51.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.823+0000 7f1288c00700 1 -- 172.21.15.89:0/2346319262 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 84) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f12800543f0 con 0x7f128c0d16a0 2024-04-02T07:55:51.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.872+0000 7f1288c00700 1 -- 172.21.15.89:0/2346319262 <== mgr.7172 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f128c153c80 con 0x7f126c06ae00 2024-04-02T07:55:51.873 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path snap_test_dir1/yearly 2024-04-02T07:55:51.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.874+0000 7f1293600700 1 -- 172.21.15.89:0/2346319262 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f126c06ae00 msgr2=0x7f126c06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:51.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.874+0000 7f1293600700 1 --2- 172.21.15.89:0/2346319262 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f126c06ae00 0x7f126c06d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f128c0d1ae0 tx=0x7f1278010000 comp rx=0 tx=0).stop 2024-04-02T07:55:51.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.874+0000 7f1293600700 1 -- 172.21.15.89:0/2346319262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f128c0d16a0 msgr2=0x7f128c151dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:51.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.874+0000 7f1293600700 1 --2- 172.21.15.89:0/2346319262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f128c0d16a0 0x7f128c151dc0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f128000e810 tx=0x7f128000ebd0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.875+0000 7f1293600700 1 -- 172.21.15.89:0/2346319262 shutdown_connections 2024-04-02T07:55:51.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.875+0000 7f1293600700 1 --2- 172.21.15.89:0/2346319262 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f126c06ae00 0x7f126c06d2c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.875+0000 7f1293600700 1 --2- 172.21.15.89:0/2346319262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f128c15d9d0 0x7f128c0b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.875+0000 7f1293600700 1 --2- 172.21.15.89:0/2346319262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f128c0d16a0 0x7f128c151dc0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.875+0000 7f1293600700 1 --2- 172.21.15.89:0/2346319262 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f128c0d2370 0x7f128c0b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:51.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.875+0000 7f1293600700 1 -- 172.21.15.89:0/2346319262 >> 172.21.15.89:0/2346319262 conn(0x7f128c08aff0 msgr2=0x7f128c0b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:51.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.875+0000 7f1293600700 1 -- 172.21.15.89:0/2346319262 shutdown_connections 2024-04-02T07:55:51.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:51.875+0000 7f1293600700 1 -- 172.21.15.89:0/2346319262 wait complete. 2024-04-02T07:55:51.893 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2024-04-02T07:55:51.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:51.893 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2024-04-02T07:55:51.939 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2024-04-02T07:55:52.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.174+0000 7fb78cc00700 1 Processor -- start 2024-04-02T07:55:52.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.175+0000 7fb78cc00700 1 -- start start 2024-04-02T07:55:52.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.176+0000 7fb78cc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb78815d9d0 0x7fb7880d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:52.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.176+0000 7fb78cc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7880d16a0 0x7fb7880d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:52.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.176+0000 7fb78cc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb788161420 con 0x7fb7880d2370 2024-04-02T07:55:52.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.176+0000 7fb78cc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7880d2820 con 0x7fb7880d16a0 2024-04-02T07:55:52.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.176+0000 7fb78cc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb788161700 con 0x7fb78815d9d0 2024-04-02T07:55:52.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.176+0000 7fb785400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7880d16a0 0x7fb7880d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:52.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.176+0000 7fb785400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7880d16a0 0x7fb7880d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55668/0 (socket says 172.21.15.89:55668) 2024-04-02T07:55:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.176+0000 7fb785400700 1 -- 172.21.15.89:0/2877306887 learned_addr learned my addr 172.21.15.89:0/2877306887 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.176+0000 7fb785e00700 1 --2- 172.21.15.89:0/2877306887 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb78815d9d0 0x7fb7880d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.176+0000 7fb785400700 1 -- 172.21.15.89:0/2877306887 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb78815d9d0 msgr2=0x7fb7880d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.176+0000 7fb785400700 1 --2- 172.21.15.89:0/2877306887 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb78815d9d0 0x7fb7880d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.176+0000 7fb785400700 1 -- 172.21.15.89:0/2877306887 >> v1:172.21.15.113:6789/0 conn(0x7fb7880d2370 legacy=0x7fb78815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.176+0000 7fb785400700 1 -- 172.21.15.89:0/2877306887 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7881619e0 con 0x7fb7880d16a0 2024-04-02T07:55:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.177+0000 7fb785400700 1 --2- 172.21.15.89:0/2877306887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7880d16a0 0x7fb7880d1e30 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fb7740098e0 tx=0x7fb774009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2c2cfe1dcb9eb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.177+0000 7fb784a00700 1 -- 172.21.15.89:0/2877306887 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774004230 con 0x7fb7880d16a0 2024-04-02T07:55:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.177+0000 7fb784a00700 1 -- 172.21.15.89:0/2877306887 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb774004390 con 0x7fb7880d16a0 2024-04-02T07:55:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.177+0000 7fb784a00700 1 -- 172.21.15.89:0/2877306887 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7740036a0 con 0x7fb7880d16a0 2024-04-02T07:55:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.177+0000 7fb78cc00700 1 -- 172.21.15.89:0/2877306887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7880d16a0 msgr2=0x7fb7880d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.177+0000 7fb78cc00700 1 --2- 172.21.15.89:0/2877306887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7880d16a0 0x7fb7880d1e30 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fb7740098e0 tx=0x7fb774009bf0 comp rx=0 tx=0).stop 2024-04-02T07:55:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.177+0000 7fb78cc00700 1 -- 172.21.15.89:0/2877306887 shutdown_connections 2024-04-02T07:55:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.177+0000 7fb78cc00700 1 --2- 172.21.15.89:0/2877306887 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb78815d9d0 0x7fb7880d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:52.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.177+0000 7fb78cc00700 1 --2- 172.21.15.89:0/2877306887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7880d16a0 0x7fb7880d1e30 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:52.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.177+0000 7fb78cc00700 1 -- 172.21.15.89:0/2877306887 >> 172.21.15.89:0/2877306887 conn(0x7fb78808aff0 msgr2=0x7fb7880d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:52.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.177+0000 7fb78cc00700 1 -- 172.21.15.89:0/2877306887 shutdown_connections 2024-04-02T07:55:52.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.178+0000 7fb78cc00700 1 -- 172.21.15.89:0/2877306887 wait complete. 2024-04-02T07:55:52.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.179+0000 7fb78cc00700 1 Processor -- start 2024-04-02T07:55:52.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.179+0000 7fb78cc00700 1 -- start start 2024-04-02T07:55:52.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.180+0000 7fb78cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7880d16a0 0x7fb7880b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:52.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.180+0000 7fb78cc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7880d2370 0x7fb78815f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:52.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.180+0000 7fb78cc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb78815d9d0 0x7fb788155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:52.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.180+0000 7fb78cc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78815ebd0 con 0x7fb7880d16a0 2024-04-02T07:55:52.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.180+0000 7fb78cc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78815df50 con 0x7fb78815d9d0 2024-04-02T07:55:52.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.180+0000 7fb78cc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78815e230 con 0x7fb7880d2370 2024-04-02T07:55:52.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.180+0000 7fb785e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7880d16a0 0x7fb7880b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:52.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.181+0000 7fb785e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7880d16a0 0x7fb7880b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55684/0 (socket says 172.21.15.89:55684) 2024-04-02T07:55:52.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.181+0000 7fb785e00700 1 -- 172.21.15.89:0/822919543 learned_addr learned my addr 172.21.15.89:0/822919543 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:52.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.181+0000 7fb785400700 1 --2- 172.21.15.89:0/822919543 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7880d2370 0x7fb78815f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:52.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.181+0000 7fb786800700 1 --2- 172.21.15.89:0/822919543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb78815d9d0 0x7fb788155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:52.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.181+0000 7fb785e00700 1 -- 172.21.15.89:0/822919543 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7880d2370 msgr2=0x7fb78815f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:52.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.181+0000 7fb785e00700 1 --2- 172.21.15.89:0/822919543 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7880d2370 0x7fb78815f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:52.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.181+0000 7fb785e00700 1 -- 172.21.15.89:0/822919543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb78815d9d0 msgr2=0x7fb788155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:52.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.181+0000 7fb785e00700 1 --2- 172.21.15.89:0/822919543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb78815d9d0 0x7fb788155520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:52.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.181+0000 7fb785e00700 1 -- 172.21.15.89:0/822919543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7740094f0 con 0x7fb7880d16a0 2024-04-02T07:55:52.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.181+0000 7fb785e00700 1 --2- 172.21.15.89:0/822919543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7880d16a0 0x7fb7880b4b80 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fb77800d800 tx=0x7fb77800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:52.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.181+0000 7fb77e000700 1 -- 172.21.15.89:0/822919543 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7780125f0 con 0x7fb7880d16a0 2024-04-02T07:55:52.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.181+0000 7fb77e000700 1 -- 172.21.15.89:0/822919543 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb778012750 con 0x7fb7880d16a0 2024-04-02T07:55:52.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.181+0000 7fb77e000700 1 -- 172.21.15.89:0/822919543 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778012b00 con 0x7fb7880d16a0 2024-04-02T07:55:52.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.181+0000 7fb78cc00700 1 -- 172.21.15.89:0/822919543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78815e510 con 0x7fb7880d16a0 2024-04-02T07:55:52.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.182+0000 7fb78cc00700 1 -- 172.21.15.89:0/822919543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7881f4410 con 0x7fb7880d16a0 2024-04-02T07:55:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.183+0000 7fb78cc00700 1 -- 172.21.15.89:0/822919543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7880d1ca0 con 0x7fb7880d16a0 2024-04-02T07:55:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.184+0000 7fb77e000700 1 -- 172.21.15.89:0/822919543 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 84) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fb778012c60 con 0x7fb7880d16a0 2024-04-02T07:55:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.184+0000 7fb77e000700 1 --2- 172.21.15.89:0/822919543 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb76806ae90 0x7fb76806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.184+0000 7fb77e000700 1 -- 172.21.15.89:0/822919543 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fb778089c90 con 0x7fb7880d16a0 2024-04-02T07:55:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.185+0000 7fb785400700 1 --2- 172.21.15.89:0/822919543 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb76806ae90 0x7fb76806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.185+0000 7fb785400700 1 --2- 172.21.15.89:0/822919543 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb76806ae90 0x7fb76806d350 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb7880d1ae0 tx=0x7fb774008030 comp rx=0 tx=0).ready entity=mgr.7172 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:52.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.189+0000 7fb77e000700 1 -- 172.21.15.89:0/822919543 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb7780549b0 con 0x7fb7880d16a0 2024-04-02T07:55:52.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.417+0000 7fb78cc00700 1 -- 172.21.15.89:0/822919543 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) v1 -- 0x7fb788161fa0 con 0x7fb76806ae90 2024-04-02T07:55:52.418 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:52.418+0000 7fe716200700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,Y 2024-04-02T07:55:52.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.418+0000 7fb77e000700 1 -- 172.21.15.89:0/822919543 <== mgr.7172 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,Y) v1 ==== 74+0+0 (secure 0 0 0) 0x7fb7740041b0 con 0x7fb76806ae90 2024-04-02T07:55:52.421 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,Y 2024-04-02T07:55:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-02T07:55:52.429 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2024-04-02T07:55:52.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.622+0000 7f9036800700 1 Processor -- start 2024-04-02T07:55:52.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.622+0000 7f9036800700 1 -- start start 2024-04-02T07:55:52.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.623+0000 7f9036800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f90300d16a0 0x7f90300d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:52.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.623+0000 7f9036800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f90300d2370 0x7f90300ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:52.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.623+0000 7f9036800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9030161420 con 0x7f903015d9d0 2024-04-02T07:55:52.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.623+0000 7f9036800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90300d4340 con 0x7f90300d2370 2024-04-02T07:55:52.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.623+0000 7f9036800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f903015edc0 con 0x7f90300d16a0 2024-04-02T07:55:52.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.623+0000 7f902f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f903015d9d0 0x7f90300d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59508/0 (socket says 172.21.15.89:59508) 2024-04-02T07:55:52.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.623+0000 7f902f400700 1 -- 172.21.15.89:0/2516965564 learned_addr learned my addr 172.21.15.89:0/2516965564 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:52.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.623+0000 7f902fe00700 1 --2- 172.21.15.89:0/2516965564 >> v2:172.21.15.89:3300/0 conn(0x7f90300d2370 0x7f90300ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:52.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.624+0000 7f902ea00700 1 --2- 172.21.15.89:0/2516965564 >> v2:172.21.15.113:3300/0 conn(0x7f90300d16a0 0x7f90300d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:52.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.624+0000 7f902e000700 1 -- 172.21.15.89:0/2516965564 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688603553 0 0) 0x7f9030161420 con 0x7f903015d9d0 2024-04-02T07:55:52.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.624+0000 7f902e000700 1 -- 172.21.15.89:0/2516965564 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90140035b0 con 0x7f903015d9d0 2024-04-02T07:55:52.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.624+0000 7f902fe00700 1 -- 172.21.15.89:0/2516965564 >> v2:172.21.15.113:3300/0 conn(0x7f90300d16a0 msgr2=0x7f90300d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:52.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.624+0000 7f902fe00700 1 --2- 172.21.15.89:0/2516965564 >> v2:172.21.15.113:3300/0 conn(0x7f90300d16a0 0x7f90300d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:52.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.624+0000 7f902fe00700 1 -- 172.21.15.89:0/2516965564 >> v1:172.21.15.89:6789/0 conn(0x7f903015d9d0 legacy=0x7f90300d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:52.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.624+0000 7f902fe00700 1 -- 172.21.15.89:0/2516965564 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9030161700 con 0x7f90300d2370 2024-04-02T07:55:52.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.624+0000 7f902fe00700 1 --2- 172.21.15.89:0/2516965564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90300d2370 0x7f90300ce360 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f90300d12f0 tx=0x7f902800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e740cd7901e40f52 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:52.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.624+0000 7f902e000700 1 -- 172.21.15.89:0/2516965564 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902800e070 con 0x7f90300d2370 2024-04-02T07:55:52.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.625+0000 7f902e000700 1 -- 172.21.15.89:0/2516965564 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902800bb20 con 0x7f90300d2370 2024-04-02T07:55:52.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.625+0000 7f902e000700 1 -- 172.21.15.89:0/2516965564 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9028012430 con 0x7f90300d2370 2024-04-02T07:55:52.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.625+0000 7f9036800700 1 -- 172.21.15.89:0/2516965564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90300d2370 msgr2=0x7f90300ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:52.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.625+0000 7f9036800700 1 --2- 172.21.15.89:0/2516965564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90300d2370 0x7f90300ce360 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f90300d12f0 tx=0x7f902800b330 comp rx=0 tx=0).stop 2024-04-02T07:55:52.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.625+0000 7f9036800700 1 -- 172.21.15.89:0/2516965564 shutdown_connections 2024-04-02T07:55:52.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.625+0000 7f9036800700 1 --2- 172.21.15.89:0/2516965564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90300d2370 0x7f90300ce360 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:52.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.625+0000 7f9036800700 1 --2- 172.21.15.89:0/2516965564 >> v2:172.21.15.113:3300/0 conn(0x7f90300d16a0 0x7f90300d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:52.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.625+0000 7f9036800700 1 -- 172.21.15.89:0/2516965564 >> 172.21.15.89:0/2516965564 conn(0x7f903008aff0 msgr2=0x7f90300d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:52.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.625+0000 7f9036800700 1 -- 172.21.15.89:0/2516965564 shutdown_connections 2024-04-02T07:55:52.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.625+0000 7f9036800700 1 -- 172.21.15.89:0/2516965564 wait complete. 2024-04-02T07:55:52.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.627+0000 7f9036800700 1 Processor -- start 2024-04-02T07:55:52.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.627+0000 7f9036800700 1 -- start start 2024-04-02T07:55:52.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.628+0000 7f9036800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90300d16a0 0x7f903015cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:52.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.628+0000 7f9036800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90300d2370 0x7f903006b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:52.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.628+0000 7f9036800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f903015d9d0 0x7f903006be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:52.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.628+0000 7f9036800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90300b70f0 con 0x7f90300d2370 2024-04-02T07:55:52.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.628+0000 7f9036800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903015d000 con 0x7f90300d16a0 2024-04-02T07:55:52.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.628+0000 7f9036800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90300b64e0 con 0x7f903015d9d0 2024-04-02T07:55:52.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f902ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90300d2370 0x7f903006b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:52.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f902ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90300d2370 0x7f903006b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55708/0 (socket says 172.21.15.89:55708) 2024-04-02T07:55:52.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f902ea00700 1 -- 172.21.15.89:0/2972053599 learned_addr learned my addr 172.21.15.89:0/2972053599 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:52.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f902f400700 1 --2- 172.21.15.89:0/2972053599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90300d16a0 0x7f903015cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:52.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f902fe00700 1 --2- 172.21.15.89:0/2972053599 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f903015d9d0 0x7f903006be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:52.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f902ea00700 1 -- 172.21.15.89:0/2972053599 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f903015d9d0 msgr2=0x7f903006be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:52.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f902ea00700 1 --2- 172.21.15.89:0/2972053599 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f903015d9d0 0x7f903006be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:52.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f902ea00700 1 -- 172.21.15.89:0/2972053599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90300d16a0 msgr2=0x7f903015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:52.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f902ea00700 1 --2- 172.21.15.89:0/2972053599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90300d16a0 0x7f903015cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:52.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f902ea00700 1 -- 172.21.15.89:0/2972053599 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9028009a70 con 0x7f90300d2370 2024-04-02T07:55:52.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f902fe00700 1 --2- 172.21.15.89:0/2972053599 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f903015d9d0 0x7f903006be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:52.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f902ea00700 1 --2- 172.21.15.89:0/2972053599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90300d2370 0x7f903006b900 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f901c00e960 tx=0x7f901c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:52.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f9027400700 1 -- 172.21.15.89:0/2972053599 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c0096f0 con 0x7f90300d2370 2024-04-02T07:55:52.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f9027400700 1 -- 172.21.15.89:0/2972053599 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901c00eeb0 con 0x7f90300d2370 2024-04-02T07:55:52.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.629+0000 7f9036800700 1 -- 172.21.15.89:0/2972053599 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90300b67c0 con 0x7f90300d2370 2024-04-02T07:55:52.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.630+0000 7f9027400700 1 -- 172.21.15.89:0/2972053599 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c00ca70 con 0x7f90300d2370 2024-04-02T07:55:52.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.630+0000 7f9036800700 1 -- 172.21.15.89:0/2972053599 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90300b6d80 con 0x7f90300d2370 2024-04-02T07:55:52.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.631+0000 7f9036800700 1 -- 172.21.15.89:0/2972053599 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90300d1ca0 con 0x7f90300d2370 2024-04-02T07:55:52.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.631+0000 7f9027400700 1 -- 172.21.15.89:0/2972053599 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 84) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f901c00cbd0 con 0x7f90300d2370 2024-04-02T07:55:52.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.632+0000 7f9027400700 1 --2- 172.21.15.89:0/2972053599 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f901006ae90 0x7f901006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:52.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.632+0000 7f9027400700 1 -- 172.21.15.89:0/2972053599 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f901c089e20 con 0x7f90300d2370 2024-04-02T07:55:52.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.633+0000 7f902f400700 1 --2- 172.21.15.89:0/2972053599 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f901006ae90 0x7f901006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:52.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.633+0000 7f902f400700 1 --2- 172.21.15.89:0/2972053599 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f901006ae90 0x7f901006d350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9020007fd0 tx=0x7f9020007dc0 comp rx=0 tx=0).ready entity=mgr.7172 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:52.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.637+0000 7f9027400700 1 -- 172.21.15.89:0/2972053599 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f901c054a10 con 0x7f90300d2370 2024-04-02T07:55:52.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.866+0000 7f9036800700 1 -- 172.21.15.89:0/2972053599 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) v1 -- 0x7f9030161fa0 con 0x7f901006ae90 2024-04-02T07:55:52.868 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:52.867+0000 7fe716200700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-02T07:55:52.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:52.868+0000 7f9027400700 1 -- 172.21.15.89:0/2972053599 <== mgr.7172 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f90200058e0 con 0x7f901006ae90 2024-04-02T07:55:52.870 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-02T07:55:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-02T07:55:52.879 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2024-04-02T07:55:53.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.074+0000 7f6e83e00700 1 Processor -- start 2024-04-02T07:55:53.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.074+0000 7f6e83e00700 1 -- start start 2024-04-02T07:55:53.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.075+0000 7f6e83e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0d21f0 0x7f6e7c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.075+0000 7f6e83e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6e7c0cec40 0x7f6e7c161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.075+0000 7f6e83e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7c0ce270 con 0x7f6e7c1611b0 2024-04-02T07:55:53.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.075+0000 7f6e83e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7c15b600 con 0x7f6e7c0cec40 2024-04-02T07:55:53.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.075+0000 7f6e83e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0cf0b0 con 0x7f6e7c0d21f0 2024-04-02T07:55:53.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.075+0000 7f6e81e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6e7c0cec40 0x7f6e7c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.075+0000 7f6e81400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6e7c1611b0 0x7f6e7c15ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59512/0 (socket says 172.21.15.89:59512) 2024-04-02T07:55:53.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.076+0000 7f6e81400700 1 -- 172.21.15.89:0/3033402840 learned_addr learned my addr 172.21.15.89:0/3033402840 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:53.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.076+0000 7f6e80a00700 1 --2- 172.21.15.89:0/3033402840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0d21f0 0x7f6e7c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.076+0000 7f6e81e00700 1 -- 172.21.15.89:0/3033402840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0d21f0 msgr2=0x7f6e7c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:53.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.076+0000 7f6e81e00700 1 --2- 172.21.15.89:0/3033402840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0d21f0 0x7f6e7c0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:53.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.076+0000 7f6e81e00700 1 -- 172.21.15.89:0/3033402840 >> v1:172.21.15.89:6789/0 conn(0x7f6e7c1611b0 legacy=0x7f6e7c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:53.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.076+0000 7f6e81e00700 1 -- 172.21.15.89:0/3033402840 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7c0d2760 con 0x7f6e7c0cec40 2024-04-02T07:55:53.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.076+0000 7f6e81e00700 1 --2- 172.21.15.89:0/3033402840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c0cec40 0x7f6e7c161590 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f6e74009d90 tx=0x7f6e7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e640a797a489b0a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:53.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.076+0000 7f6e7be00700 1 -- 172.21.15.89:0/3033402840 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7400e070 con 0x7f6e7c0cec40 2024-04-02T07:55:53.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.076+0000 7f6e7be00700 1 -- 172.21.15.89:0/3033402840 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7400bbd0 con 0x7f6e7c0cec40 2024-04-02T07:55:53.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.076+0000 7f6e7be00700 1 -- 172.21.15.89:0/3033402840 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74012430 con 0x7f6e7c0cec40 2024-04-02T07:55:53.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.077+0000 7f6e83e00700 1 -- 172.21.15.89:0/3033402840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c0cec40 msgr2=0x7f6e7c161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:53.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.077+0000 7f6e83e00700 1 --2- 172.21.15.89:0/3033402840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c0cec40 0x7f6e7c161590 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f6e74009d90 tx=0x7f6e7400b330 comp rx=0 tx=0).stop 2024-04-02T07:55:53.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.077+0000 7f6e83e00700 1 -- 172.21.15.89:0/3033402840 shutdown_connections 2024-04-02T07:55:53.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.077+0000 7f6e83e00700 1 --2- 172.21.15.89:0/3033402840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c0cec40 0x7f6e7c161590 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:53.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.077+0000 7f6e83e00700 1 --2- 172.21.15.89:0/3033402840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0d21f0 0x7f6e7c0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:53.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.077+0000 7f6e83e00700 1 -- 172.21.15.89:0/3033402840 >> 172.21.15.89:0/3033402840 conn(0x7f6e7c08aff0 msgr2=0x7f6e7c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:53.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.077+0000 7f6e83e00700 1 -- 172.21.15.89:0/3033402840 shutdown_connections 2024-04-02T07:55:53.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.077+0000 7f6e83e00700 1 -- 172.21.15.89:0/3033402840 wait complete. 2024-04-02T07:55:53.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.079+0000 7f6e83e00700 1 Processor -- start 2024-04-02T07:55:53.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.079+0000 7f6e83e00700 1 -- start start 2024-04-02T07:55:53.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e83e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0cec40 0x7f6e7c0ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e83e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c0d21f0 0x7f6e7c1f26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e83e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c1611b0 0x7f6e7c158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e83e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c1577c0 con 0x7f6e7c0d21f0 2024-04-02T07:55:53.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e83e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c155240 con 0x7f6e7c1611b0 2024-04-02T07:55:53.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e83e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c155490 con 0x7f6e7c0cec40 2024-04-02T07:55:53.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e80a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c0d21f0 0x7f6e7c1f26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e80a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c0d21f0 0x7f6e7c1f26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55722/0 (socket says 172.21.15.89:55722) 2024-04-02T07:55:53.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e80a00700 1 -- 172.21.15.89:0/1619885964 learned_addr learned my addr 172.21.15.89:0/1619885964 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:53.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e81400700 1 --2- 172.21.15.89:0/1619885964 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0cec40 0x7f6e7c0ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e81e00700 1 --2- 172.21.15.89:0/1619885964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c1611b0 0x7f6e7c158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e80a00700 1 -- 172.21.15.89:0/1619885964 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0cec40 msgr2=0x7f6e7c0ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:53.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e80a00700 1 --2- 172.21.15.89:0/1619885964 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e7c0cec40 0x7f6e7c0ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:53.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e80a00700 1 -- 172.21.15.89:0/1619885964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c1611b0 msgr2=0x7f6e7c158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:53.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e80a00700 1 --2- 172.21.15.89:0/1619885964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e7c1611b0 0x7f6e7c158340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:53.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.080+0000 7f6e80a00700 1 -- 172.21.15.89:0/1619885964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e74009a70 con 0x7f6e7c0d21f0 2024-04-02T07:55:53.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.081+0000 7f6e80a00700 1 --2- 172.21.15.89:0/1619885964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e7c0d21f0 0x7f6e7c1f26d0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800e810 tx=0x7f6e6800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:53.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.081+0000 7f6e79600700 1 -- 172.21.15.89:0/1619885964 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68013610 con 0x7f6e7c0d21f0 2024-04-02T07:55:53.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.081+0000 7f6e79600700 1 -- 172.21.15.89:0/1619885964 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e68013770 con 0x7f6e7c0d21f0 2024-04-02T07:55:53.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.081+0000 7f6e83e00700 1 -- 172.21.15.89:0/1619885964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e7c155770 con 0x7f6e7c0d21f0 2024-04-02T07:55:53.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.081+0000 7f6e79600700 1 -- 172.21.15.89:0/1619885964 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68013b20 con 0x7f6e7c0d21f0 2024-04-02T07:55:53.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.081+0000 7f6e83e00700 1 -- 172.21.15.89:0/1619885964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7c155d30 con 0x7f6e7c0d21f0 2024-04-02T07:55:53.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.082+0000 7f6e83e00700 1 -- 172.21.15.89:0/1619885964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7c155f50 con 0x7f6e7c0d21f0 2024-04-02T07:55:53.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.083+0000 7f6e79600700 1 -- 172.21.15.89:0/1619885964 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 84) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f6e68013c80 con 0x7f6e7c0d21f0 2024-04-02T07:55:53.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.084+0000 7f6e79600700 1 --2- 172.21.15.89:0/1619885964 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6e5c06ae90 0x7f6e5c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.084+0000 7f6e79600700 1 -- 172.21.15.89:0/1619885964 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f6e68089820 con 0x7f6e7c0d21f0 2024-04-02T07:55:53.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.084+0000 7f6e81400700 1 --2- 172.21.15.89:0/1619885964 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6e5c06ae90 0x7f6e5c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.085+0000 7f6e81400700 1 --2- 172.21.15.89:0/1619885964 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6e5c06ae90 0x7f6e5c06d350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6e70005910 tx=0x7f6e70009040 comp rx=0 tx=0).ready entity=mgr.7172 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:53.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.089+0000 7f6e79600700 1 -- 172.21.15.89:0/1619885964 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6e680543b0 con 0x7f6e7c0d21f0 2024-04-02T07:55:53.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.318+0000 7f6e83e00700 1 -- 172.21.15.89:0/1619885964 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e7c0cdf00 con 0x7f6e5c06ae90 2024-04-02T07:55:53.319 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:53.319+0000 7fe716200700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-02T07:55:53.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.319+0000 7f6e79600700 1 -- 172.21.15.89:0/1619885964 <== mgr.7172 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6e70005940 con 0x7f6e5c06ae90 2024-04-02T07:55:53.321 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-02T07:55:53.330 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-02T07:55:53.330 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2024-04-02T07:55:53.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.524+0000 7f93c0000700 1 Processor -- start 2024-04-02T07:55:53.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.525+0000 7f93c0000700 1 -- start start 2024-04-02T07:55:53.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.526+0000 7f93c0000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f93b8160d70 0x7f93b80b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.526+0000 7f93c0000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f93b806a790 0x7f93b806ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.526+0000 7f93c0000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b8161150 con 0x7f93b815ef70 2024-04-02T07:55:53.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.526+0000 7f93c0000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b80b4cd0 con 0x7f93b8160d70 2024-04-02T07:55:53.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.526+0000 7f93c0000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b806b140 con 0x7f93b806a790 2024-04-02T07:55:53.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.527+0000 7f93bd600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f93b8160d70 0x7f93b80b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.527+0000 7f93bd600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f93b8160d70 0x7f93b80b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55728/0 (socket says 172.21.15.89:55728) 2024-04-02T07:55:53.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.527+0000 7f93bd600700 1 -- 172.21.15.89:0/4281234340 learned_addr learned my addr 172.21.15.89:0/4281234340 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:53.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.527+0000 7f93bcc00700 1 --1- 172.21.15.89:0/4281234340 >> v1:172.21.15.89:6789/0 conn(0x7f93b815ef70 0x7f93b815e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59518/0 (socket says 172.21.15.89:59518) 2024-04-02T07:55:53.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.527+0000 7f93be000700 1 --2- 172.21.15.89:0/4281234340 >> v2:172.21.15.113:3300/0 conn(0x7f93b806a790 0x7f93b806ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.527+0000 7f93bd600700 1 -- 172.21.15.89:0/4281234340 >> v2:172.21.15.113:3300/0 conn(0x7f93b806a790 msgr2=0x7f93b806ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:53.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.527+0000 7f93bd600700 1 --2- 172.21.15.89:0/4281234340 >> v2:172.21.15.113:3300/0 conn(0x7f93b806a790 0x7f93b806ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:53.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.527+0000 7f93bd600700 1 -- 172.21.15.89:0/4281234340 >> v1:172.21.15.89:6789/0 conn(0x7f93b815ef70 legacy=0x7f93b815e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:53.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.527+0000 7f93bd600700 1 -- 172.21.15.89:0/4281234340 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b806dba0 con 0x7f93b8160d70 2024-04-02T07:55:53.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.527+0000 7f93bd600700 1 --2- 172.21.15.89:0/4281234340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b8160d70 0x7f93b80b3200 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f93b806b820 tx=0x7f93ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c92f42dd58b682 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:53.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.527+0000 7f93b7e00700 1 -- 172.21.15.89:0/4281234340 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac00b870 con 0x7f93b8160d70 2024-04-02T07:55:53.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.528+0000 7f93b7e00700 1 -- 172.21.15.89:0/4281234340 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac00b9d0 con 0x7f93b8160d70 2024-04-02T07:55:53.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.528+0000 7f93b7e00700 1 -- 172.21.15.89:0/4281234340 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac00bd60 con 0x7f93b8160d70 2024-04-02T07:55:53.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.528+0000 7f93c0000700 1 -- 172.21.15.89:0/4281234340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b8160d70 msgr2=0x7f93b80b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:53.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.528+0000 7f93c0000700 1 --2- 172.21.15.89:0/4281234340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b8160d70 0x7f93b80b3200 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f93b806b820 tx=0x7f93ac009bd0 comp rx=0 tx=0).stop 2024-04-02T07:55:53.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.528+0000 7f93c0000700 1 -- 172.21.15.89:0/4281234340 shutdown_connections 2024-04-02T07:55:53.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.528+0000 7f93c0000700 1 --2- 172.21.15.89:0/4281234340 >> v2:172.21.15.113:3300/0 conn(0x7f93b806a790 0x7f93b806ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:53.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.528+0000 7f93c0000700 1 --2- 172.21.15.89:0/4281234340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b8160d70 0x7f93b80b3200 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:53.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.528+0000 7f93c0000700 1 -- 172.21.15.89:0/4281234340 >> 172.21.15.89:0/4281234340 conn(0x7f93b80b5720 msgr2=0x7f93b80b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:53.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.528+0000 7f93c0000700 1 -- 172.21.15.89:0/4281234340 shutdown_connections 2024-04-02T07:55:53.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.528+0000 7f93c0000700 1 -- 172.21.15.89:0/4281234340 wait complete. 2024-04-02T07:55:53.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.530+0000 7f93c0000700 1 Processor -- start 2024-04-02T07:55:53.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.531+0000 7f93c0000700 1 -- start start 2024-04-02T07:55:53.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.532+0000 7f93c0000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93b806a790 0x7f93b809b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.532+0000 7f93c0000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b815ef70 0x7f93b809ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.532+0000 7f93c0000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b8160d70 0x7f93b80ad170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.532+0000 7f93c0000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b80afa30 con 0x7f93b8160d70 2024-04-02T07:55:53.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.532+0000 7f93c0000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b814ab90 con 0x7f93b815ef70 2024-04-02T07:55:53.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.532+0000 7f93c0000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b814a540 con 0x7f93b806a790 2024-04-02T07:55:53.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.532+0000 7f93be000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b8160d70 0x7f93b80ad170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.532+0000 7f93be000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b8160d70 0x7f93b80ad170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55730/0 (socket says 172.21.15.89:55730) 2024-04-02T07:55:53.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.532+0000 7f93be000700 1 -- 172.21.15.89:0/3915502328 learned_addr learned my addr 172.21.15.89:0/3915502328 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:53.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.532+0000 7f93bcc00700 1 --2- 172.21.15.89:0/3915502328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b815ef70 0x7f93b809ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.532+0000 7f93bd600700 1 --2- 172.21.15.89:0/3915502328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93b806a790 0x7f93b809b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.533+0000 7f93be000700 1 -- 172.21.15.89:0/3915502328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93b806a790 msgr2=0x7f93b809b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:53.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.533+0000 7f93be000700 1 --2- 172.21.15.89:0/3915502328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93b806a790 0x7f93b809b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:53.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.533+0000 7f93be000700 1 -- 172.21.15.89:0/3915502328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b815ef70 msgr2=0x7f93b809ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:53.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.533+0000 7f93be000700 1 --2- 172.21.15.89:0/3915502328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b815ef70 0x7f93b809ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:53.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.533+0000 7f93be000700 1 -- 172.21.15.89:0/3915502328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac009550 con 0x7f93b8160d70 2024-04-02T07:55:53.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.533+0000 7f93bd600700 1 --2- 172.21.15.89:0/3915502328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f93b806a790 0x7f93b809b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:53.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.533+0000 7f93be000700 1 --2- 172.21.15.89:0/3915502328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93b8160d70 0x7f93b80ad170 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f93b0007e40 tx=0x7f93b000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:53.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.533+0000 7f93b5600700 1 -- 172.21.15.89:0/3915502328 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b000ede0 con 0x7f93b8160d70 2024-04-02T07:55:53.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.533+0000 7f93b5600700 1 -- 172.21.15.89:0/3915502328 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b0014420 con 0x7f93b8160d70 2024-04-02T07:55:53.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.533+0000 7f93c0000700 1 -- 172.21.15.89:0/3915502328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b80afcd0 con 0x7f93b8160d70 2024-04-02T07:55:53.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.533+0000 7f93b5600700 1 -- 172.21.15.89:0/3915502328 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b00147d0 con 0x7f93b8160d70 2024-04-02T07:55:53.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.533+0000 7f93c0000700 1 -- 172.21.15.89:0/3915502328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b80b0290 con 0x7f93b8160d70 2024-04-02T07:55:53.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.534+0000 7f93c0000700 1 -- 172.21.15.89:0/3915502328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b8067720 con 0x7f93b8160d70 2024-04-02T07:55:53.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.535+0000 7f93b5600700 1 -- 172.21.15.89:0/3915502328 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 84) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f93b0014930 con 0x7f93b8160d70 2024-04-02T07:55:53.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.536+0000 7f93b5600700 1 --2- 172.21.15.89:0/3915502328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f939806ae90 0x7f939806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.536+0000 7f93b5600700 1 -- 172.21.15.89:0/3915502328 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f93b008a180 con 0x7f93b8160d70 2024-04-02T07:55:53.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.536+0000 7f93bd600700 1 --2- 172.21.15.89:0/3915502328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f939806ae90 0x7f939806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.537+0000 7f93bd600700 1 --2- 172.21.15.89:0/3915502328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f939806ae90 0x7f939806d350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f93ac014040 tx=0x7f93ac008030 comp rx=0 tx=0).ready entity=mgr.7172 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:53.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.541+0000 7f93b5600700 1 -- 172.21.15.89:0/3915502328 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f93b0054d70 con 0x7f93b8160d70 2024-04-02T07:55:53.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.771+0000 7f93c0000700 1 -- 172.21.15.89:0/3915502328 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) v1 -- 0x7f93b80b54a0 con 0x7f939806ae90 2024-04-02T07:55:53.772 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:53.772+0000 7fe716200700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-02T07:55:53.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.772+0000 7f93b5600700 1 -- 172.21.15.89:0/3915502328 <== mgr.7172 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f93ac00bbc0 con 0x7f939806ae90 2024-04-02T07:55:53.774 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-02T07:55:53.782 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-02T07:55:53.783 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2024-04-02T07:55:53.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.977+0000 7f963e000700 1 Processor -- start 2024-04-02T07:55:53.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.977+0000 7f963e000700 1 -- start start 2024-04-02T07:55:53.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.978+0000 7f963e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f96380d21f0 0x7f96380ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.979+0000 7f963e000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96380ce270 con 0x7f96380cec40 2024-04-02T07:55:53.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.979+0000 7f963e000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963815b600 con 0x7f96381611b0 2024-04-02T07:55:53.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.979+0000 7f963e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96380cf0b0 con 0x7f96380d21f0 2024-04-02T07:55:53.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.979+0000 7f9637e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f96380cec40 0x7f9638161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59532/0 (socket says 172.21.15.89:59532) 2024-04-02T07:55:53.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.979+0000 7f9637e00700 1 -- 172.21.15.89:0/3332247188 learned_addr learned my addr 172.21.15.89:0/3332247188 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:53.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.979+0000 7f9636a00700 1 --2- 172.21.15.89:0/3332247188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f96380d21f0 0x7f96380ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.979+0000 7f9636000700 1 -- 172.21.15.89:0/3332247188 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102112013 0 0) 0x7f96380ce270 con 0x7f96380cec40 2024-04-02T07:55:53.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.979+0000 7f9636000700 1 -- 172.21.15.89:0/3332247188 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f961c0035b0 con 0x7f96380cec40 2024-04-02T07:55:53.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.979+0000 7f9636000700 1 -- 172.21.15.89:0/3332247188 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283557762 0 0) 0x7f963815b600 con 0x7f96381611b0 2024-04-02T07:55:53.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.979+0000 7f9636000700 1 -- 172.21.15.89:0/3332247188 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96380ce270 con 0x7f96381611b0 2024-04-02T07:55:53.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.980+0000 7f9636000700 1 -- 172.21.15.89:0/3332247188 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2579815914 0 0) 0x7f961c0035b0 con 0x7f96380cec40 2024-04-02T07:55:53.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.980+0000 7f9636a00700 1 -- 172.21.15.89:0/3332247188 >> v1:172.21.15.113:6789/0 conn(0x7f96381611b0 legacy=0x7f963815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:53.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.980+0000 7f9636a00700 1 -- 172.21.15.89:0/3332247188 >> v1:172.21.15.89:6789/0 conn(0x7f96380cec40 legacy=0x7f9638161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:53.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.980+0000 7f9636a00700 1 -- 172.21.15.89:0/3332247188 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96380d2760 con 0x7f96380d21f0 2024-04-02T07:55:53.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.980+0000 7f9636a00700 1 --2- 172.21.15.89:0/3332247188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f96380d21f0 0x7f96380ce700 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9624009870 tx=0x7f9624009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cbbbcb47fd6a174 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:53.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.980+0000 7f9636000700 1 -- 172.21.15.89:0/3332247188 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96300019f0 con 0x7f96380d21f0 2024-04-02T07:55:53.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.980+0000 7f9636000700 1 -- 172.21.15.89:0/3332247188 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9624004030 con 0x7f96380d21f0 2024-04-02T07:55:53.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.980+0000 7f9636000700 1 -- 172.21.15.89:0/3332247188 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96240043c0 con 0x7f96380d21f0 2024-04-02T07:55:53.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.981+0000 7f963e000700 1 -- 172.21.15.89:0/3332247188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f96380d21f0 msgr2=0x7f96380ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:53.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.981+0000 7f963e000700 1 --2- 172.21.15.89:0/3332247188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f96380d21f0 0x7f96380ce700 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9624009870 tx=0x7f9624009b80 comp rx=0 tx=0).stop 2024-04-02T07:55:53.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.981+0000 7f963e000700 1 -- 172.21.15.89:0/3332247188 shutdown_connections 2024-04-02T07:55:53.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.981+0000 7f963e000700 1 --2- 172.21.15.89:0/3332247188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f96380d21f0 0x7f96380ce700 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:53.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.981+0000 7f963e000700 1 -- 172.21.15.89:0/3332247188 >> 172.21.15.89:0/3332247188 conn(0x7f963808aff0 msgr2=0x7f96380d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:53.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.981+0000 7f963e000700 1 -- 172.21.15.89:0/3332247188 shutdown_connections 2024-04-02T07:55:53.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.981+0000 7f963e000700 1 -- 172.21.15.89:0/3332247188 wait complete. 2024-04-02T07:55:53.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.983+0000 7f963e000700 1 Processor -- start 2024-04-02T07:55:53.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.983+0000 7f963e000700 1 -- start start 2024-04-02T07:55:53.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.984+0000 7f963e000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96380cec40 0x7f96380c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.984+0000 7f963e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96380d21f0 0x7f96380c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.984+0000 7f963e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f96381611b0 0x7f9638158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.985+0000 7f963e000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96380c9b20 con 0x7f96380d21f0 2024-04-02T07:55:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.985+0000 7f963e000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96380b73e0 con 0x7f96380cec40 2024-04-02T07:55:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.985+0000 7f963e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96380b76c0 con 0x7f96381611b0 2024-04-02T07:55:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.985+0000 7f9636a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96380d21f0 0x7f96380c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.985+0000 7f9636a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96380d21f0 0x7f96380c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55740/0 (socket says 172.21.15.89:55740) 2024-04-02T07:55:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.985+0000 7f9636a00700 1 -- 172.21.15.89:0/713012747 learned_addr learned my addr 172.21.15.89:0/713012747 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.985+0000 7f9637e00700 1 --2- 172.21.15.89:0/713012747 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f96381611b0 0x7f9638158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.985+0000 7f9637400700 1 --2- 172.21.15.89:0/713012747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96380cec40 0x7f96380c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.985+0000 7f9636a00700 1 -- 172.21.15.89:0/713012747 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f96381611b0 msgr2=0x7f9638158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.985+0000 7f9636a00700 1 --2- 172.21.15.89:0/713012747 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f96381611b0 0x7f9638158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.985+0000 7f9636a00700 1 -- 172.21.15.89:0/713012747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96380cec40 msgr2=0x7f96380c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:53.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.985+0000 7f9636a00700 1 --2- 172.21.15.89:0/713012747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96380cec40 0x7f96380c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:53.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.985+0000 7f9636a00700 1 -- 172.21.15.89:0/713012747 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96240094a0 con 0x7f96380d21f0 2024-04-02T07:55:53.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.986+0000 7f9636a00700 1 --2- 172.21.15.89:0/713012747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96380d21f0 0x7f96380c95e0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f9624000c00 tx=0x7f9624004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:53.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.986+0000 7f962f400700 1 -- 172.21.15.89:0/713012747 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624003e60 con 0x7f96380d21f0 2024-04-02T07:55:53.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.986+0000 7f962f400700 1 -- 172.21.15.89:0/713012747 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962401f070 con 0x7f96380d21f0 2024-04-02T07:55:53.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.986+0000 7f962f400700 1 -- 172.21.15.89:0/713012747 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624011850 con 0x7f96380d21f0 2024-04-02T07:55:53.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.986+0000 7f963e000700 1 -- 172.21.15.89:0/713012747 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96380b79a0 con 0x7f96380d21f0 2024-04-02T07:55:53.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.986+0000 7f963e000700 1 -- 172.21.15.89:0/713012747 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96380b7f60 con 0x7f96380d21f0 2024-04-02T07:55:53.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.987+0000 7f963e000700 1 -- 172.21.15.89:0/713012747 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96380b81a0 con 0x7f96380d21f0 2024-04-02T07:55:53.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.988+0000 7f962f400700 1 -- 172.21.15.89:0/713012747 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 84) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f96240119b0 con 0x7f96380d21f0 2024-04-02T07:55:53.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.989+0000 7f962f400700 1 --2- 172.21.15.89:0/713012747 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f961806ae90 0x7f961806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:53.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.989+0000 7f962f400700 1 -- 172.21.15.89:0/713012747 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f9624021070 con 0x7f96380d21f0 2024-04-02T07:55:53.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.989+0000 7f9637400700 1 --2- 172.21.15.89:0/713012747 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f961806ae90 0x7f961806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:53.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.989+0000 7f9637400700 1 --2- 172.21.15.89:0/713012747 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f961806ae90 0x7f961806d350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9628001220 tx=0x7f9628009040 comp rx=0 tx=0).ready entity=mgr.7172 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:53.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:53.993+0000 7f962f400700 1 -- 172.21.15.89:0/713012747 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f96240be580 con 0x7f96380d21f0 2024-04-02T07:55:54.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.225+0000 7f963e000700 1 -- 172.21.15.89:0/713012747 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f96380cdf00 con 0x7f961806ae90 2024-04-02T07:55:54.227 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:54.226+0000 7fe716200700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-04-02T07:55:54.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.226+0000 7f962f400700 1 -- 172.21.15.89:0/713012747 <== mgr.7172 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) v1 ==== 96+0+0 (secure 0 0 0) 0x7f9628001cb0 con 0x7f961806ae90 2024-04-02T07:55:54.228 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-04-02T07:55:54.236 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-02T07:55:54.237 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2024-04-02T07:55:54.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.430+0000 7f494be00700 1 Processor -- start 2024-04-02T07:55:54.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.430+0000 7f494be00700 1 -- start start 2024-04-02T07:55:54.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.432+0000 7f494be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49440d16a0 0x7f49440d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:54.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.432+0000 7f494be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49440d2370 0x7f49440ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:54.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.432+0000 7f494be00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4944161420 con 0x7f494415d9d0 2024-04-02T07:55:54.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.432+0000 7f494be00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49440d2820 con 0x7f49440d16a0 2024-04-02T07:55:54.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.432+0000 7f494be00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49440d4340 con 0x7f49440d2370 2024-04-02T07:55:54.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.432+0000 7f4948a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49440d16a0 0x7f49440d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:54.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.432+0000 7f4948a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49440d16a0 0x7f49440d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55750/0 (socket says 172.21.15.89:55750) 2024-04-02T07:55:54.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.432+0000 7f4948a00700 1 -- 172.21.15.89:0/408982053 learned_addr learned my addr 172.21.15.89:0/408982053 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:54.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.432+0000 7f4949e00700 1 --2- 172.21.15.89:0/408982053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49440d2370 0x7f49440ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:54.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.432+0000 7f4948a00700 1 -- 172.21.15.89:0/408982053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49440d2370 msgr2=0x7f49440ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:54.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.432+0000 7f4948a00700 1 --2- 172.21.15.89:0/408982053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49440d2370 0x7f49440ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:54.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.432+0000 7f4948a00700 1 -- 172.21.15.89:0/408982053 >> v1:172.21.15.113:6789/0 conn(0x7f494415d9d0 legacy=0x7f49440d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:54.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.432+0000 7f4948a00700 1 -- 172.21.15.89:0/408982053 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4944161700 con 0x7f49440d16a0 2024-04-02T07:55:54.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.433+0000 7f4948a00700 1 --2- 172.21.15.89:0/408982053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440d16a0 0x7f49440d1e30 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f4930006fd0 tx=0x7f4930009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ecac1ebc7efdb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:54.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.433+0000 7f4943e00700 1 -- 172.21.15.89:0/408982053 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4930005e30 con 0x7f49440d16a0 2024-04-02T07:55:54.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.433+0000 7f4943e00700 1 -- 172.21.15.89:0/408982053 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4930004030 con 0x7f49440d16a0 2024-04-02T07:55:54.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.433+0000 7f4943e00700 1 -- 172.21.15.89:0/408982053 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49300043e0 con 0x7f49440d16a0 2024-04-02T07:55:54.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.433+0000 7f494be00700 1 -- 172.21.15.89:0/408982053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440d16a0 msgr2=0x7f49440d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:54.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.433+0000 7f494be00700 1 --2- 172.21.15.89:0/408982053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440d16a0 0x7f49440d1e30 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f4930006fd0 tx=0x7f4930009bd0 comp rx=0 tx=0).stop 2024-04-02T07:55:54.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.434+0000 7f494be00700 1 -- 172.21.15.89:0/408982053 shutdown_connections 2024-04-02T07:55:54.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.434+0000 7f494be00700 1 --2- 172.21.15.89:0/408982053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49440d2370 0x7f49440ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:54.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.434+0000 7f494be00700 1 --2- 172.21.15.89:0/408982053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440d16a0 0x7f49440d1e30 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:54.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.434+0000 7f494be00700 1 -- 172.21.15.89:0/408982053 >> 172.21.15.89:0/408982053 conn(0x7f494408aff0 msgr2=0x7f49440d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:54.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.434+0000 7f494be00700 1 -- 172.21.15.89:0/408982053 shutdown_connections 2024-04-02T07:55:54.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.434+0000 7f494be00700 1 -- 172.21.15.89:0/408982053 wait complete. 2024-04-02T07:55:54.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.435+0000 7f494be00700 1 Processor -- start 2024-04-02T07:55:54.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.436+0000 7f494be00700 1 -- start start 2024-04-02T07:55:54.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.436+0000 7f494be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440d16a0 0x7f49440ca860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:54.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.436+0000 7f494be00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49440d2370 0x7f49440b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:54.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.436+0000 7f494be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f494415d9d0 0x7f4944160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:54.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.436+0000 7f494be00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49440c9d10 con 0x7f49440d16a0 2024-04-02T07:55:54.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.436+0000 7f494be00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49441475b0 con 0x7f49440d2370 2024-04-02T07:55:54.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.437+0000 7f494be00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4944149a50 con 0x7f494415d9d0 2024-04-02T07:55:54.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.437+0000 7f4949400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440d16a0 0x7f49440ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:54.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.437+0000 7f4949400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440d16a0 0x7f49440ca860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55766/0 (socket says 172.21.15.89:55766) 2024-04-02T07:55:54.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.437+0000 7f4949400700 1 -- 172.21.15.89:0/324322749 learned_addr learned my addr 172.21.15.89:0/324322749 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:54.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.437+0000 7f4949e00700 1 --2- 172.21.15.89:0/324322749 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f494415d9d0 0x7f4944160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:54.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.437+0000 7f4948a00700 1 --2- 172.21.15.89:0/324322749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49440d2370 0x7f49440b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:54.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.437+0000 7f4949400700 1 -- 172.21.15.89:0/324322749 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f494415d9d0 msgr2=0x7f4944160350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:54.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.437+0000 7f4949400700 1 --2- 172.21.15.89:0/324322749 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f494415d9d0 0x7f4944160350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:54.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.437+0000 7f4949400700 1 -- 172.21.15.89:0/324322749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49440d2370 msgr2=0x7f49440b7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:54.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.437+0000 7f4949400700 1 --2- 172.21.15.89:0/324322749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49440d2370 0x7f49440b7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:54.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.437+0000 7f4949400700 1 -- 172.21.15.89:0/324322749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4930009500 con 0x7f49440d16a0 2024-04-02T07:55:54.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.437+0000 7f4949400700 1 --2- 172.21.15.89:0/324322749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440d16a0 0x7f49440ca860 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f493800c480 tx=0x7f493800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:54.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.438+0000 7f4941600700 1 -- 172.21.15.89:0/324322749 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49380052d0 con 0x7f49440d16a0 2024-04-02T07:55:54.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.438+0000 7f4941600700 1 -- 172.21.15.89:0/324322749 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4938005430 con 0x7f49440d16a0 2024-04-02T07:55:54.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.438+0000 7f494be00700 1 -- 172.21.15.89:0/324322749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49440b89b0 con 0x7f49440d16a0 2024-04-02T07:55:54.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.438+0000 7f4941600700 1 -- 172.21.15.89:0/324322749 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49380038c0 con 0x7f49440d16a0 2024-04-02T07:55:54.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.438+0000 7f494be00700 1 -- 172.21.15.89:0/324322749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49440b86d0 con 0x7f49440d16a0 2024-04-02T07:55:54.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.439+0000 7f494be00700 1 -- 172.21.15.89:0/324322749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49440d1ca0 con 0x7f49440d16a0 2024-04-02T07:55:54.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.440+0000 7f4941600700 1 -- 172.21.15.89:0/324322749 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 84) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f4938003a20 con 0x7f49440d16a0 2024-04-02T07:55:54.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.440+0000 7f4941600700 1 --2- 172.21.15.89:0/324322749 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f492406ae90 0x7f492406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:54.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.441+0000 7f4941600700 1 -- 172.21.15.89:0/324322749 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f4938088dc0 con 0x7f49440d16a0 2024-04-02T07:55:54.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.441+0000 7f4948a00700 1 --2- 172.21.15.89:0/324322749 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f492406ae90 0x7f492406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:54.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.441+0000 7f4948a00700 1 --2- 172.21.15.89:0/324322749 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f492406ae90 0x7f492406d350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4930006fd0 tx=0x7f4930008070 comp rx=0 tx=0).ready entity=mgr.7172 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:54.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.445+0000 7f4941600700 1 -- 172.21.15.89:0/324322749 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4938053a50 con 0x7f49440d16a0 2024-04-02T07:55:54.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.670+0000 7f494be00700 1 -- 172.21.15.89:0/324322749 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f4944161fa0 con 0x7f492406ae90 2024-04-02T07:55:54.672 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:54.671+0000 7fe716200700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-02T07:55:54.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:54.671+0000 7f4941600700 1 -- 172.21.15.89:0/324322749 <== mgr.7172 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4930004520 con 0x7f492406ae90 2024-04-02T07:55:54.673 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-04-02T07:55:54.681 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-02T07:55:54.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:54.681 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2024-04-02T07:55:54.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:54.728 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2024-04-02T07:55:54.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:54.773 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2024-04-02T07:55:54.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:54.859 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2024-04-02T07:55:54.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:54.947 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2024-04-02T07:55:55.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:55.034 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2024-04-02T07:55:55.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:55.079 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2024-04-02T07:55:55.165 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T07:55:55.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.357+0000 7f231e800700 1 Processor -- start 2024-04-02T07:55:55.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.357+0000 7f231e800700 1 -- start start 2024-04-02T07:55:55.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.358+0000 7f231e800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f23180cbb30 0x7f23180cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:55.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.358+0000 7f231e800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23180cd020 0x7f23180cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:55.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.358+0000 7f231e800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23180cc450 con 0x7f23180b2070 2024-04-02T07:55:55.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.358+0000 7f231e800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23180cdae0 con 0x7f23180cd020 2024-04-02T07:55:55.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.358+0000 7f231e800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23180cddc0 con 0x7f23180cbb30 2024-04-02T07:55:55.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.359+0000 7f2316a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23180cd020 0x7f23180cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:55.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.359+0000 7f2316a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23180cd020 0x7f23180cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55780/0 (socket says 172.21.15.89:55780) 2024-04-02T07:55:55.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.359+0000 7f2316a00700 1 -- 172.21.15.89:0/2004483796 learned_addr learned my addr 172.21.15.89:0/2004483796 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:55.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.359+0000 7f2317400700 1 --2- 172.21.15.89:0/2004483796 >> v2:172.21.15.113:3300/0 conn(0x7f23180cbb30 0x7f23180cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:55.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.359+0000 7f2316a00700 1 -- 172.21.15.89:0/2004483796 >> v2:172.21.15.113:3300/0 conn(0x7f23180cbb30 msgr2=0x7f23180cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:55.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.359+0000 7f2316a00700 1 --2- 172.21.15.89:0/2004483796 >> v2:172.21.15.113:3300/0 conn(0x7f23180cbb30 0x7f23180cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:55.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.359+0000 7f2316a00700 1 -- 172.21.15.89:0/2004483796 >> v1:172.21.15.113:6789/0 conn(0x7f23180b2070 legacy=0x7f231806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:55.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.359+0000 7f2316a00700 1 -- 172.21.15.89:0/2004483796 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23180ce0a0 con 0x7f23180cd020 2024-04-02T07:55:55.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.359+0000 7f2316a00700 1 --2- 172.21.15.89:0/2004483796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23180cd020 0x7f23180cd480 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f231806fd00 tx=0x7f2304009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2985a5b3e46dafec server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:55.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.359+0000 7f2316000700 1 -- 172.21.15.89:0/2004483796 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230400b870 con 0x7f23180cd020 2024-04-02T07:55:55.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.360+0000 7f2316000700 1 -- 172.21.15.89:0/2004483796 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230400b9d0 con 0x7f23180cd020 2024-04-02T07:55:55.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.360+0000 7f2316000700 1 -- 172.21.15.89:0/2004483796 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230400bd60 con 0x7f23180cd020 2024-04-02T07:55:55.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.360+0000 7f231e800700 1 -- 172.21.15.89:0/2004483796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23180cd020 msgr2=0x7f23180cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:55.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.360+0000 7f231e800700 1 --2- 172.21.15.89:0/2004483796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23180cd020 0x7f23180cd480 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f231806fd00 tx=0x7f2304009bd0 comp rx=0 tx=0).stop 2024-04-02T07:55:55.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.360+0000 7f231e800700 1 -- 172.21.15.89:0/2004483796 shutdown_connections 2024-04-02T07:55:55.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.360+0000 7f231e800700 1 --2- 172.21.15.89:0/2004483796 >> v2:172.21.15.113:3300/0 conn(0x7f23180cbb30 0x7f23180cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:55.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.360+0000 7f231e800700 1 --2- 172.21.15.89:0/2004483796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23180cd020 0x7f23180cd480 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:55.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.360+0000 7f231e800700 1 -- 172.21.15.89:0/2004483796 >> 172.21.15.89:0/2004483796 conn(0x7f23180b0d70 msgr2=0x7f23180b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:55.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.360+0000 7f231e800700 1 -- 172.21.15.89:0/2004483796 shutdown_connections 2024-04-02T07:55:55.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.360+0000 7f231e800700 1 -- 172.21.15.89:0/2004483796 wait complete. 2024-04-02T07:55:55.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.362+0000 7f231e800700 1 Processor -- start 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.362+0000 7f231e800700 1 -- start start 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.363+0000 7f231e800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23180b2070 0x7f23180ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.363+0000 7f231e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23180cbb30 0x7f231814eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.363+0000 7f231e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23180cd020 0x7f23180caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.363+0000 7f231e800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23180c9c20 con 0x7f23180cbb30 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.363+0000 7f231e800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23180c92f0 con 0x7f23180b2070 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.363+0000 7f231e800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23180c95d0 con 0x7f23180cd020 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.363+0000 7f2316a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23180cbb30 0x7f231814eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.363+0000 7f2316a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23180cbb30 0x7f231814eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55796/0 (socket says 172.21.15.89:55796) 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.363+0000 7f2316a00700 1 -- 172.21.15.89:0/2624366755 learned_addr learned my addr 172.21.15.89:0/2624366755 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.363+0000 7f2317400700 1 --2- 172.21.15.89:0/2624366755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23180b2070 0x7f23180ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.363+0000 7f2317e00700 1 --2- 172.21.15.89:0/2624366755 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23180cd020 0x7f23180caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.364+0000 7f2316a00700 1 -- 172.21.15.89:0/2624366755 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23180cd020 msgr2=0x7f23180caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.364+0000 7f2316a00700 1 --2- 172.21.15.89:0/2624366755 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23180cd020 0x7f23180caa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.364+0000 7f2316a00700 1 -- 172.21.15.89:0/2624366755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23180b2070 msgr2=0x7f23180ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:55.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.364+0000 7f2316a00700 1 --2- 172.21.15.89:0/2624366755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23180b2070 0x7f23180ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:55.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.364+0000 7f2316a00700 1 -- 172.21.15.89:0/2624366755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2304009550 con 0x7f23180cbb30 2024-04-02T07:55:55.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.364+0000 7f2316a00700 1 --2- 172.21.15.89:0/2624366755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23180cbb30 0x7f231814eb00 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f2304000c00 tx=0x7f2304004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:55.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.364+0000 7f230f400700 1 -- 172.21.15.89:0/2624366755 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2304004540 con 0x7f23180cbb30 2024-04-02T07:55:55.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.364+0000 7f230f400700 1 -- 172.21.15.89:0/2624366755 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23040036a0 con 0x7f23180cbb30 2024-04-02T07:55:55.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.364+0000 7f231e800700 1 -- 172.21.15.89:0/2624366755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23180c98b0 con 0x7f23180cbb30 2024-04-02T07:55:55.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.364+0000 7f230f400700 1 -- 172.21.15.89:0/2624366755 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2304003a10 con 0x7f23180cbb30 2024-04-02T07:55:55.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.364+0000 7f231e800700 1 -- 172.21.15.89:0/2624366755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231814f040 con 0x7f23180cbb30 2024-04-02T07:55:55.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.365+0000 7f231e800700 1 -- 172.21.15.89:0/2624366755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2318067720 con 0x7f23180cbb30 2024-04-02T07:55:55.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.366+0000 7f230f400700 1 -- 172.21.15.89:0/2624366755 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 84) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f2304021070 con 0x7f23180cbb30 2024-04-02T07:55:55.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.367+0000 7f230f400700 1 --2- 172.21.15.89:0/2624366755 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22f806ae90 0x7f22f806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:55.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.367+0000 7f230f400700 1 -- 172.21.15.89:0/2624366755 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f230408b0e0 con 0x7f23180cbb30 2024-04-02T07:55:55.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.367+0000 7f2317400700 1 --2- 172.21.15.89:0/2624366755 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22f806ae90 0x7f22f806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:55.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.368+0000 7f2317400700 1 --2- 172.21.15.89:0/2624366755 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22f806ae90 0x7f22f806d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2308007a30 tx=0x7f2308008040 comp rx=0 tx=0).ready entity=mgr.7172 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:55.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.371+0000 7f230f400700 1 -- 172.21.15.89:0/2624366755 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2304055d60 con 0x7f23180cbb30 2024-04-02T07:55:55.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:55.635+0000 7f231e800700 1 -- 172.21.15.89:0/2624366755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f231815bda0 con 0x7f23180cbb30 2024-04-02T07:55:56.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.065+0000 7f230f400700 1 -- 172.21.15.89:0/2624366755 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f2304056130 con 0x7f23180cbb30 2024-04-02T07:55:56.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.067+0000 7f230f400700 1 -- 172.21.15.89:0/2624366755 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v85) v1 ==== 93+0+0 (secure 0 0 0) 0x7f230406eed0 con 0x7f23180cbb30 2024-04-02T07:55:56.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.070+0000 7f231e800700 1 -- 172.21.15.89:0/2624366755 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22f806ae90 msgr2=0x7f22f806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:56.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.070+0000 7f231e800700 1 --2- 172.21.15.89:0/2624366755 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22f806ae90 0x7f22f806d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2308007a30 tx=0x7f2308008040 comp rx=0 tx=0).stop 2024-04-02T07:55:56.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.070+0000 7f231e800700 1 -- 172.21.15.89:0/2624366755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23180cbb30 msgr2=0x7f231814eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:56.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.070+0000 7f231e800700 1 --2- 172.21.15.89:0/2624366755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23180cbb30 0x7f231814eb00 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f2304000c00 tx=0x7f2304004750 comp rx=0 tx=0).stop 2024-04-02T07:55:56.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.070+0000 7f231e800700 1 -- 172.21.15.89:0/2624366755 shutdown_connections 2024-04-02T07:55:56.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.070+0000 7f231e800700 1 --2- 172.21.15.89:0/2624366755 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22f806ae90 0x7f22f806d350 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.070+0000 7f231e800700 1 --2- 172.21.15.89:0/2624366755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23180b2070 0x7f23180ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.070+0000 7f231e800700 1 --2- 172.21.15.89:0/2624366755 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23180cd020 0x7f23180caa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.070+0000 7f231e800700 1 --2- 172.21.15.89:0/2624366755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23180cbb30 0x7f231814eb00 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.070+0000 7f231e800700 1 -- 172.21.15.89:0/2624366755 >> 172.21.15.89:0/2624366755 conn(0x7f23180b0d70 msgr2=0x7f23180ce820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:56.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.070+0000 7f231e800700 1 -- 172.21.15.89:0/2624366755 shutdown_connections 2024-04-02T07:55:56.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.070+0000 7f231e800700 1 -- 172.21.15.89:0/2624366755 wait complete. 2024-04-02T07:55:56.090 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:55:56.090 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:55:56.183 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:55:56.276 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:55:56.276 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:55:56.276 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:56.288 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:56.287+0000 7f231fc35240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:55:56.303 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:56.302+0000 7f3e94351240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:55:56.320 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:55:56.320 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:56.405 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:55:56.472 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:56.472+0000 7f231fc35240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:55:56.496 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:56.496+0000 7f3e94351240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:55:56.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.665+0000 7f4f21e00700 1 Processor -- start 2024-04-02T07:55:56.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.665+0000 7f4f21e00700 1 -- start start 2024-04-02T07:55:56.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.666+0000 7f4f21e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4f1c09aba0 0x7f4f1c06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.666+0000 7f4f21e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4f1c06c480 0x7f4f1c0ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.666+0000 7f4f21e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c06c8d0 con 0x7f4f1c15d710 2024-04-02T07:55:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.666+0000 7f4f21e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1c06bb40 con 0x7f4f1c09aba0 2024-04-02T07:55:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.666+0000 7f4f21e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1c099b00 con 0x7f4f1c06c480 2024-04-02T07:55:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.666+0000 7f4f1a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4f1c09aba0 0x7f4f1c06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.666+0000 7f4f1a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4f1c09aba0 0x7f4f1c06bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55826/0 (socket says 172.21.15.89:55826) 2024-04-02T07:55:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.666+0000 7f4f1aa00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4f1c15d710 0x7f4f1c0d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59560/0 (socket says 172.21.15.89:59560) 2024-04-02T07:55:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.666+0000 7f4f1a000700 1 -- 172.21.15.89:0/1263969976 learned_addr learned my addr 172.21.15.89:0/1263969976 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.666+0000 7f4f1b400700 1 --2- 172.21.15.89:0/1263969976 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c06c480 0x7f4f1c0ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.667+0000 7f4f1a000700 1 -- 172.21.15.89:0/1263969976 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c06c480 msgr2=0x7f4f1c0ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.667+0000 7f4f1a000700 1 --2- 172.21.15.89:0/1263969976 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c06c480 0x7f4f1c0ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.667+0000 7f4f1a000700 1 -- 172.21.15.89:0/1263969976 >> v1:172.21.15.89:6789/0 conn(0x7f4f1c15d710 legacy=0x7f4f1c0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.667+0000 7f4f1a000700 1 -- 172.21.15.89:0/1263969976 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c099de0 con 0x7f4f1c09aba0 2024-04-02T07:55:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.667+0000 7f4f19600700 1 -- 172.21.15.89:0/1263969976 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209690338 0 0) 0x7f4f1c06c8d0 con 0x7f4f1c15d710 2024-04-02T07:55:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.667+0000 7f4f1a000700 1 --2- 172.21.15.89:0/1263969976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f1c09aba0 0x7f4f1c06bf40 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0722c0 tx=0x7f4f08009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebaa1385272a7750 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.667+0000 7f4f19600700 1 -- 172.21.15.89:0/1263969976 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08011830 con 0x7f4f1c09aba0 2024-04-02T07:55:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.667+0000 7f4f19600700 1 -- 172.21.15.89:0/1263969976 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f08011990 con 0x7f4f1c09aba0 2024-04-02T07:55:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.667+0000 7f4f19600700 1 -- 172.21.15.89:0/1263969976 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08011d20 con 0x7f4f1c09aba0 2024-04-02T07:55:56.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.668+0000 7f4f21e00700 1 -- 172.21.15.89:0/1263969976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f1c09aba0 msgr2=0x7f4f1c06bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:56.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.668+0000 7f4f21e00700 1 --2- 172.21.15.89:0/1263969976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f1c09aba0 0x7f4f1c06bf40 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0722c0 tx=0x7f4f08009bd0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.668+0000 7f4f21e00700 1 -- 172.21.15.89:0/1263969976 shutdown_connections 2024-04-02T07:55:56.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.668+0000 7f4f21e00700 1 --2- 172.21.15.89:0/1263969976 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c06c480 0x7f4f1c0ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.668+0000 7f4f21e00700 1 --2- 172.21.15.89:0/1263969976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f1c09aba0 0x7f4f1c06bf40 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.668+0000 7f4f21e00700 1 -- 172.21.15.89:0/1263969976 >> 172.21.15.89:0/1263969976 conn(0x7f4f1c0d13e0 msgr2=0x7f4f1c0d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:56.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.668+0000 7f4f21e00700 1 -- 172.21.15.89:0/1263969976 shutdown_connections 2024-04-02T07:55:56.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.668+0000 7f4f21e00700 1 -- 172.21.15.89:0/1263969976 wait complete. 2024-04-02T07:55:56.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.670+0000 7f4f21e00700 1 Processor -- start 2024-04-02T07:55:56.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.670+0000 7f4f21e00700 1 -- start start 2024-04-02T07:55:56.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f21e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c06c480 0x7f4f1c09e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:56.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f21e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f1c09aba0 0x7f4f1c15ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f21e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f1c15d710 0x7f4f1c06ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f21e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c15b170 con 0x7f4f1c15d710 2024-04-02T07:55:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f21e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c15dd00 con 0x7f4f1c06c480 2024-04-02T07:55:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f21e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c160b90 con 0x7f4f1c09aba0 2024-04-02T07:55:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f1b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f1c15d710 0x7f4f1c06ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f1aa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c06c480 0x7f4f1c09e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f1a000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f1c09aba0 0x7f4f1c15ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f1b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f1c15d710 0x7f4f1c06ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55840/0 (socket says 172.21.15.89:55840) 2024-04-02T07:55:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f1b400700 1 -- 172.21.15.89:0/868174377 learned_addr learned my addr 172.21.15.89:0/868174377 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f1a000700 1 -- 172.21.15.89:0/868174377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c06c480 msgr2=0x7f4f1c09e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f1a000700 1 --2- 172.21.15.89:0/868174377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c06c480 0x7f4f1c09e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f1a000700 1 -- 172.21.15.89:0/868174377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f1c15d710 msgr2=0x7f4f1c06ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.671+0000 7f4f1a000700 1 --2- 172.21.15.89:0/868174377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f1c15d710 0x7f4f1c06ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.672+0000 7f4f1a000700 1 -- 172.21.15.89:0/868174377 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f08009550 con 0x7f4f1c09aba0 2024-04-02T07:55:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.672+0000 7f4f1a000700 1 --2- 172.21.15.89:0/868174377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f1c09aba0 0x7f4f1c15ac30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4f08016000 tx=0x7f4f08004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.672+0000 7f4f13400700 1 -- 172.21.15.89:0/868174377 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08004510 con 0x7f4f1c09aba0 2024-04-02T07:55:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.672+0000 7f4f13400700 1 -- 172.21.15.89:0/868174377 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0801f070 con 0x7f4f1c09aba0 2024-04-02T07:55:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.672+0000 7f4f21e00700 1 -- 172.21.15.89:0/868174377 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1c09ea20 con 0x7f4f1c09aba0 2024-04-02T07:55:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.672+0000 7f4f13400700 1 -- 172.21.15.89:0/868174377 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f080038a0 con 0x7f4f1c09aba0 2024-04-02T07:55:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.672+0000 7f4f21e00700 1 -- 172.21.15.89:0/868174377 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1c09efe0 con 0x7f4f1c09aba0 2024-04-02T07:55:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.673+0000 7f4f21e00700 1 -- 172.21.15.89:0/868174377 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1c067720 con 0x7f4f1c09aba0 2024-04-02T07:55:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.674+0000 7f4f13400700 1 -- 172.21.15.89:0/868174377 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f4f08003a00 con 0x7f4f1c09aba0 2024-04-02T07:55:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.675+0000 7f4f13400700 1 --2- 172.21.15.89:0/868174377 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4efc06ae40 0x7f4efc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.675+0000 7f4f1aa00700 1 -- 172.21.15.89:0/868174377 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4efc06ae40 msgr2=0x7f4efc06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.675+0000 7f4f13400700 1 -- 172.21.15.89:0/868174377 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f4f08021070 con 0x7f4f1c09aba0 2024-04-02T07:55:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.675+0000 7f4f1aa00700 1 --2- 172.21.15.89:0/868174377 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4efc06ae40 0x7f4efc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:56.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.680+0000 7f4f13400700 1 -- 172.21.15.89:0/868174377 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4f080559d0 con 0x7f4f1c09aba0 2024-04-02T07:55:56.794 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:56.793+0000 7f231fc35240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:55:56.833 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:56.832+0000 7f3e94351240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:55:56.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.876+0000 7f4f1aa00700 1 -- 172.21.15.89:0/868174377 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4efc06ae40 msgr2=0x7f4efc06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:56.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.876+0000 7f4f1aa00700 1 --2- 172.21.15.89:0/868174377 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4efc06ae40 0x7f4efc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:56.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.891+0000 7f4f21e00700 1 -- 172.21.15.89:0/868174377 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f1c15de40 con 0x7f4f1c09aba0 2024-04-02T07:55:56.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.892+0000 7f4f13400700 1 -- 172.21.15.89:0/868174377 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+17175 (secure 0 0 0) 0x7f4f08015090 con 0x7f4f1c09aba0 2024-04-02T07:55:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:56.893 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":141,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:55:47.785904+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:55:29.329907+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-02T07:55:30.555350+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/836334016":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/1824838019":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/2769562366":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/224945013":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/732038760":"2024-04-03T07:55:47.785863+0000","172.21.15.89:6833/43357":"2024-04-03T07:55:47.785863+0000","172.21.15.89:6832/43357":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/3727408218":"2024-04-03T07:55:47.785863+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:55:56.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.895+0000 7f4f21e00700 1 -- 172.21.15.89:0/868174377 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4efc06ae40 msgr2=0x7f4efc06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:56.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.895+0000 7f4f21e00700 1 --2- 172.21.15.89:0/868174377 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4efc06ae40 0x7f4efc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.895+0000 7f4f21e00700 1 -- 172.21.15.89:0/868174377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f1c09aba0 msgr2=0x7f4f1c15ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:56.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.895+0000 7f4f21e00700 1 --2- 172.21.15.89:0/868174377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f1c09aba0 0x7f4f1c15ac30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4f08016000 tx=0x7f4f08004750 comp rx=0 tx=0).stop 2024-04-02T07:55:56.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.895+0000 7f4f21e00700 1 -- 172.21.15.89:0/868174377 shutdown_connections 2024-04-02T07:55:56.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.895+0000 7f4f21e00700 1 --2- 172.21.15.89:0/868174377 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4efc06ae40 0x7f4efc06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.895+0000 7f4f21e00700 1 --2- 172.21.15.89:0/868174377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c06c480 0x7f4f1c09e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.895+0000 7f4f21e00700 1 --2- 172.21.15.89:0/868174377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f1c15d710 0x7f4f1c06ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.895+0000 7f4f21e00700 1 --2- 172.21.15.89:0/868174377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f1c09aba0 0x7f4f1c15ac30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:56.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.895+0000 7f4f21e00700 1 -- 172.21.15.89:0/868174377 >> 172.21.15.89:0/868174377 conn(0x7f4f1c0d13e0 msgr2=0x7f4f1c0cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:56.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.895+0000 7f4f21e00700 1 -- 172.21.15.89:0/868174377 shutdown_connections 2024-04-02T07:55:56.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:56.895+0000 7f4f21e00700 1 -- 172.21.15.89:0/868174377 wait complete. 2024-04-02T07:55:56.915 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:55:29.329907+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:55:30.555350+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:55:56.915 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:55:57.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.112+0000 7fae80200700 1 Processor -- start 2024-04-02T07:55:57.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.112+0000 7fae80200700 1 -- start start 2024-04-02T07:55:57.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.113+0000 7fae80200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fae780cbb30 0x7fae780cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:57.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.113+0000 7fae80200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae780cd480 0x7fae7806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:57.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.113+0000 7fae80200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae780cc450 con 0x7fae780cd020 2024-04-02T07:55:57.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.113+0000 7fae80200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae780cdae0 con 0x7fae780cbb30 2024-04-02T07:55:57.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.113+0000 7fae80200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae780cddc0 con 0x7fae780cd480 2024-04-02T07:55:57.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.113+0000 7fae7d800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fae780cbb30 0x7fae780cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:57.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.113+0000 7fae7ce00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fae780cd020 0x7fae780b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59570/0 (socket says 172.21.15.89:59570) 2024-04-02T07:55:57.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.113+0000 7fae7ce00700 1 -- 172.21.15.89:0/4156558889 learned_addr learned my addr 172.21.15.89:0/4156558889 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:57.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.114+0000 7fae7e200700 1 --2- 172.21.15.89:0/4156558889 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae780cd480 0x7fae7806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:57.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.114+0000 7fae77e00700 1 -- 172.21.15.89:0/4156558889 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238896572 0 0) 0x7fae780cc450 con 0x7fae780cd020 2024-04-02T07:55:57.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.114+0000 7fae77e00700 1 -- 172.21.15.89:0/4156558889 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae600035b0 con 0x7fae780cd020 2024-04-02T07:55:57.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.114+0000 7fae7d800700 1 -- 172.21.15.89:0/4156558889 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae780cd480 msgr2=0x7fae7806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:57.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.114+0000 7fae7d800700 1 --2- 172.21.15.89:0/4156558889 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae780cd480 0x7fae7806e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.114+0000 7fae7d800700 1 -- 172.21.15.89:0/4156558889 >> v1:172.21.15.89:6789/0 conn(0x7fae780cd020 legacy=0x7fae780b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.114+0000 7fae7d800700 1 -- 172.21.15.89:0/4156558889 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae780ce0a0 con 0x7fae780cbb30 2024-04-02T07:55:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.114+0000 7fae7d800700 1 --2- 172.21.15.89:0/4156558889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae780cbb30 0x7fae780cbf10 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009840 tx=0x7fae6c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b4c55b333f34d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.114+0000 7fae77e00700 1 -- 172.21.15.89:0/4156558889 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c011830 con 0x7fae780cbb30 2024-04-02T07:55:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.115+0000 7fae77e00700 1 -- 172.21.15.89:0/4156558889 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6c011990 con 0x7fae780cbb30 2024-04-02T07:55:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.115+0000 7fae77e00700 1 -- 172.21.15.89:0/4156558889 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c011d20 con 0x7fae780cbb30 2024-04-02T07:55:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.115+0000 7fae80200700 1 -- 172.21.15.89:0/4156558889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae780cbb30 msgr2=0x7fae780cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.115+0000 7fae80200700 1 --2- 172.21.15.89:0/4156558889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae780cbb30 0x7fae780cbf10 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009840 tx=0x7fae6c009b50 comp rx=0 tx=0).stop 2024-04-02T07:55:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.115+0000 7fae80200700 1 -- 172.21.15.89:0/4156558889 shutdown_connections 2024-04-02T07:55:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.115+0000 7fae80200700 1 --2- 172.21.15.89:0/4156558889 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae780cd480 0x7fae7806e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.115+0000 7fae80200700 1 --2- 172.21.15.89:0/4156558889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae780cbb30 0x7fae780cbf10 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.115+0000 7fae80200700 1 -- 172.21.15.89:0/4156558889 >> 172.21.15.89:0/4156558889 conn(0x7fae780b0d70 msgr2=0x7fae780b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.115+0000 7fae80200700 1 -- 172.21.15.89:0/4156558889 shutdown_connections 2024-04-02T07:55:57.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.115+0000 7fae80200700 1 -- 172.21.15.89:0/4156558889 wait complete. 2024-04-02T07:55:57.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.117+0000 7fae80200700 1 Processor -- start 2024-04-02T07:55:57.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.117+0000 7fae80200700 1 -- start start 2024-04-02T07:55:57.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.118+0000 7fae80200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae780cbb30 0x7fae781f4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:57.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.118+0000 7fae80200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae780cd020 0x7fae781f05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:57.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.118+0000 7fae80200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae780cd480 0x7fae78147330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.118+0000 7fae80200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae780aec40 con 0x7fae780cbb30 2024-04-02T07:55:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.118+0000 7fae80200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae781470a0 con 0x7fae780cd480 2024-04-02T07:55:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.118+0000 7fae80200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae781f4ab0 con 0x7fae780cd020 2024-04-02T07:55:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.118+0000 7fae7d800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae780cbb30 0x7fae781f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.118+0000 7fae7d800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae780cbb30 0x7fae781f4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55858/0 (socket says 172.21.15.89:55858) 2024-04-02T07:55:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.118+0000 7fae7d800700 1 -- 172.21.15.89:0/1026949204 learned_addr learned my addr 172.21.15.89:0/1026949204 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.118+0000 7fae7ce00700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae780cd020 0x7fae781f05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.118+0000 7fae7e200700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae780cd480 0x7fae78147330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.119+0000 7fae7d800700 1 -- 172.21.15.89:0/1026949204 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae780cd020 msgr2=0x7fae781f05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.119+0000 7fae7d800700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae780cd020 0x7fae781f05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.119+0000 7fae7d800700 1 -- 172.21.15.89:0/1026949204 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae780cd480 msgr2=0x7fae78147330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.119+0000 7fae7d800700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae780cd480 0x7fae78147330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.119+0000 7fae7d800700 1 -- 172.21.15.89:0/1026949204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6c0094a0 con 0x7fae780cbb30 2024-04-02T07:55:57.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.119+0000 7fae7ce00700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae780cd020 0x7fae781f05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:57.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.119+0000 7fae7d800700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae780cbb30 0x7fae781f4280 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0090b0 tx=0x7fae6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:57.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.119+0000 7fae75600700 1 -- 172.21.15.89:0/1026949204 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c0036a0 con 0x7fae780cbb30 2024-04-02T07:55:57.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.119+0000 7fae75600700 1 -- 172.21.15.89:0/1026949204 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6c01e070 con 0x7fae780cbb30 2024-04-02T07:55:57.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.119+0000 7fae75600700 1 -- 172.21.15.89:0/1026949204 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c003a40 con 0x7fae780cbb30 2024-04-02T07:55:57.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.119+0000 7fae80200700 1 -- 172.21.15.89:0/1026949204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae781f4d90 con 0x7fae780cbb30 2024-04-02T07:55:57.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.120+0000 7fae80200700 1 -- 172.21.15.89:0/1026949204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae780ae860 con 0x7fae780cbb30 2024-04-02T07:55:57.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.121+0000 7fae80200700 1 -- 172.21.15.89:0/1026949204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae780aed80 con 0x7fae780cbb30 2024-04-02T07:55:57.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.122+0000 7fae75600700 1 -- 172.21.15.89:0/1026949204 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fae6c003ba0 con 0x7fae780cbb30 2024-04-02T07:55:57.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.123+0000 7fae75600700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae5806ae40 0x7fae5806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:57.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.123+0000 7fae7ce00700 1 -- 172.21.15.89:0/1026949204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae5806ae40 msgr2=0x7fae5806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:57.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.123+0000 7fae75600700 1 -- 172.21.15.89:0/1026949204 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fae6c08a7b0 con 0x7fae780cbb30 2024-04-02T07:55:57.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.123+0000 7fae7ce00700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae5806ae40 0x7fae5806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:57.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.127+0000 7fae75600700 1 -- 172.21.15.89:0/1026949204 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae780aed80 con 0x7fae780cbb30 2024-04-02T07:55:57.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.324+0000 7fae7ce00700 1 -- 172.21.15.89:0/1026949204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae5806ae40 msgr2=0x7fae5806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:57.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.324+0000 7fae7ce00700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae5806ae40 0x7fae5806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:57.325 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:57.324+0000 7f231fc35240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:55:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.334+0000 7fae80200700 1 -- 172.21.15.89:0/1026949204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fae780aed80 con 0x7fae780cbb30 2024-04-02T07:55:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.335+0000 7fae75600700 1 -- 172.21.15.89:0/1026949204 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7fae780aed80 con 0x7fae780cbb30 2024-04-02T07:55:57.335 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:55:57.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.338+0000 7fae80200700 1 -- 172.21.15.89:0/1026949204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae5806ae40 msgr2=0x7fae5806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:57.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.338+0000 7fae80200700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae5806ae40 0x7fae5806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.338+0000 7fae80200700 1 -- 172.21.15.89:0/1026949204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae780cbb30 msgr2=0x7fae781f4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:57.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.338+0000 7fae80200700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae780cbb30 0x7fae781f4280 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0090b0 tx=0x7fae6c004750 comp rx=0 tx=0).stop 2024-04-02T07:55:57.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.338+0000 7fae80200700 1 -- 172.21.15.89:0/1026949204 shutdown_connections 2024-04-02T07:55:57.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.338+0000 7fae80200700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae5806ae40 0x7fae5806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.338+0000 7fae80200700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae780cd480 0x7fae78147330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.338+0000 7fae80200700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae780cbb30 0x7fae781f4280 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.338+0000 7fae80200700 1 --2- 172.21.15.89:0/1026949204 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae780cd020 0x7fae781f05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.338+0000 7fae80200700 1 -- 172.21.15.89:0/1026949204 >> 172.21.15.89:0/1026949204 conn(0x7fae780b0d70 msgr2=0x7fae78159620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:57.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.338+0000 7fae80200700 1 -- 172.21.15.89:0/1026949204 shutdown_connections 2024-04-02T07:55:57.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.338+0000 7fae80200700 1 -- 172.21.15.89:0/1026949204 wait complete. 2024-04-02T07:55:57.357 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:55:57.384 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:57.384+0000 7f3e94351240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:55:57.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.555+0000 7f0a14200700 1 Processor -- start 2024-04-02T07:55:57.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.556+0000 7f0a14200700 1 -- start start 2024-04-02T07:55:57.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.557+0000 7f0a14200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0a0c160e50 0x7f0a0c0b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:57.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.557+0000 7f0a14200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0a0c0d3f20 0x7f0a0c0b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:57.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.557+0000 7f0a14200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0c0cef00 con 0x7f0a0c0b3f80 2024-04-02T07:55:57.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.557+0000 7f0a14200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a0c161230 con 0x7f0a0c0d3f20 2024-04-02T07:55:57.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.557+0000 7f0a14200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a0c0d2a80 con 0x7f0a0c160e50 2024-04-02T07:55:57.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.557+0000 7f0a10e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0a0c0d3f20 0x7f0a0c0b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:57.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.557+0000 7f0a11800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0a0c160e50 0x7f0a0c0b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:57.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.557+0000 7f0a10e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0a0c0d3f20 0x7f0a0c0b53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55870/0 (socket says 172.21.15.89:55870) 2024-04-02T07:55:57.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.557+0000 7f0a10e00700 1 -- 172.21.15.89:0/974893268 learned_addr learned my addr 172.21.15.89:0/974893268 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:57.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.558+0000 7f0a11800700 1 -- 172.21.15.89:0/974893268 >> v2:172.21.15.89:3300/0 conn(0x7f0a0c0d3f20 msgr2=0x7f0a0c0b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:57.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.558+0000 7f0a11800700 1 --2- 172.21.15.89:0/974893268 >> v2:172.21.15.89:3300/0 conn(0x7f0a0c0d3f20 0x7f0a0c0b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.558+0000 7f0a11800700 1 -- 172.21.15.89:0/974893268 >> v1:172.21.15.113:6789/0 conn(0x7f0a0c0b3f80 legacy=0x7f0a0c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:57.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.558+0000 7f0a11800700 1 -- 172.21.15.89:0/974893268 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0c0d43b0 con 0x7f0a0c160e50 2024-04-02T07:55:57.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.558+0000 7f0a11800700 1 --2- 172.21.15.89:0/974893268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a0c160e50 0x7f0a0c0b4e90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c0cff10 tx=0x7f0a00009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18b6bf71a8cb07f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:57.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.558+0000 7f0a0be00700 1 -- 172.21.15.89:0/974893268 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a00011830 con 0x7f0a0c160e50 2024-04-02T07:55:57.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.558+0000 7f0a0be00700 1 -- 172.21.15.89:0/974893268 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a00011990 con 0x7f0a0c160e50 2024-04-02T07:55:57.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.558+0000 7f0a0be00700 1 -- 172.21.15.89:0/974893268 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a00011d20 con 0x7f0a0c160e50 2024-04-02T07:55:57.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.559+0000 7f0a14200700 1 -- 172.21.15.89:0/974893268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a0c160e50 msgr2=0x7f0a0c0b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:57.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.559+0000 7f0a14200700 1 --2- 172.21.15.89:0/974893268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a0c160e50 0x7f0a0c0b4e90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c0cff10 tx=0x7f0a00009b20 comp rx=0 tx=0).stop 2024-04-02T07:55:57.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.559+0000 7f0a14200700 1 -- 172.21.15.89:0/974893268 shutdown_connections 2024-04-02T07:55:57.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.559+0000 7f0a14200700 1 --2- 172.21.15.89:0/974893268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a0c160e50 0x7f0a0c0b4e90 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.559+0000 7f0a14200700 1 --2- 172.21.15.89:0/974893268 >> v2:172.21.15.89:3300/0 conn(0x7f0a0c0d3f20 0x7f0a0c0b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.559+0000 7f0a14200700 1 -- 172.21.15.89:0/974893268 >> 172.21.15.89:0/974893268 conn(0x7f0a0c08aff0 msgr2=0x7f0a0c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:57.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.559+0000 7f0a14200700 1 -- 172.21.15.89:0/974893268 shutdown_connections 2024-04-02T07:55:57.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.559+0000 7f0a14200700 1 -- 172.21.15.89:0/974893268 wait complete. 2024-04-02T07:55:57.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.561+0000 7f0a14200700 1 Processor -- start 2024-04-02T07:55:57.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.561+0000 7f0a14200700 1 -- start start 2024-04-02T07:55:57.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a14200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a0c0b3f80 0x7f0a0c15bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:57.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a14200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c160e50 0x7f0a0c0b8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:57.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a14200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a0c0b6130 0x7f0a0c06c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:57.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a14200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c0b9130 con 0x7f0a0c160e50 2024-04-02T07:55:57.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a14200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c0cbe00 con 0x7f0a0c0b6130 2024-04-02T07:55:57.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a14200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c0cb560 con 0x7f0a0c0b3f80 2024-04-02T07:55:57.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a10e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c160e50 0x7f0a0c0b8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:57.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a10e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c160e50 0x7f0a0c0b8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55872/0 (socket says 172.21.15.89:55872) 2024-04-02T07:55:57.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a10e00700 1 -- 172.21.15.89:0/3352759563 learned_addr learned my addr 172.21.15.89:0/3352759563 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:57.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a11800700 1 --2- 172.21.15.89:0/3352759563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a0c0b3f80 0x7f0a0c15bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:57.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a12200700 1 --2- 172.21.15.89:0/3352759563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a0c0b6130 0x7f0a0c06c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:57.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a10e00700 1 -- 172.21.15.89:0/3352759563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a0c0b3f80 msgr2=0x7f0a0c15bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:57.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a10e00700 1 --2- 172.21.15.89:0/3352759563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a0c0b3f80 0x7f0a0c15bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a10e00700 1 -- 172.21.15.89:0/3352759563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a0c0b6130 msgr2=0x7f0a0c06c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:57.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a10e00700 1 --2- 172.21.15.89:0/3352759563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a0c0b6130 0x7f0a0c06c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.562+0000 7f0a10e00700 1 -- 172.21.15.89:0/3352759563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a000094a0 con 0x7f0a0c160e50 2024-04-02T07:55:57.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.563+0000 7f0a10e00700 1 --2- 172.21.15.89:0/3352759563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c160e50 0x7f0a0c0b8bf0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f09f800e910 tx=0x7f09f800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:57.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.563+0000 7f0a09600700 1 -- 172.21.15.89:0/3352759563 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f80096a0 con 0x7f0a0c160e50 2024-04-02T07:55:57.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.563+0000 7f0a09600700 1 -- 172.21.15.89:0/3352759563 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f8013680 con 0x7f0a0c160e50 2024-04-02T07:55:57.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.563+0000 7f0a14200700 1 -- 172.21.15.89:0/3352759563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0c0cb840 con 0x7f0a0c160e50 2024-04-02T07:55:57.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.563+0000 7f0a09600700 1 -- 172.21.15.89:0/3352759563 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f8013a30 con 0x7f0a0c160e50 2024-04-02T07:55:57.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.563+0000 7f0a14200700 1 -- 172.21.15.89:0/3352759563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0c0c8bc0 con 0x7f0a0c160e50 2024-04-02T07:55:57.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.564+0000 7f0a14200700 1 -- 172.21.15.89:0/3352759563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0c0c8de0 con 0x7f0a0c160e50 2024-04-02T07:55:57.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.565+0000 7f0a09600700 1 -- 172.21.15.89:0/3352759563 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f09f8013b90 con 0x7f0a0c160e50 2024-04-02T07:55:57.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.566+0000 7f0a09600700 1 --2- 172.21.15.89:0/3352759563 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09ec06ae70 0x7f09ec06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:57.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.566+0000 7f0a11800700 1 -- 172.21.15.89:0/3352759563 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09ec06ae70 msgr2=0x7f09ec06d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:57.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.566+0000 7f0a09600700 1 -- 172.21.15.89:0/3352759563 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f09f8089be0 con 0x7f0a0c160e50 2024-04-02T07:55:57.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.566+0000 7f0a11800700 1 --2- 172.21.15.89:0/3352759563 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09ec06ae70 0x7f09ec06d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:57.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.574+0000 7f0a09600700 1 -- 172.21.15.89:0/3352759563 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09f80547d0 con 0x7f0a0c160e50 2024-04-02T07:55:57.579 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:57.578+0000 7f231fc35240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:55:57.654 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:57.654+0000 7f3e94351240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:55:57.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.767+0000 7f0a11800700 1 -- 172.21.15.89:0/3352759563 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09ec06ae70 msgr2=0x7f09ec06d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:57.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.767+0000 7f0a11800700 1 --2- 172.21.15.89:0/3352759563 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09ec06ae70 0x7f09ec06d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:57.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.782+0000 7f0a14200700 1 -- 172.21.15.89:0/3352759563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0a0c067720 con 0x7f0a0c160e50 2024-04-02T07:55:57.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.782+0000 7f0a09600700 1 -- 172.21.15.89:0/3352759563 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v141) v1 ==== 104+0+11 (secure 0 0 0) 0x7f09f8017020 con 0x7f0a0c160e50 2024-04-02T07:55:57.783 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:55:57.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.785+0000 7f0a14200700 1 -- 172.21.15.89:0/3352759563 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09ec06ae70 msgr2=0x7f09ec06d330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:57.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.785+0000 7f0a14200700 1 --2- 172.21.15.89:0/3352759563 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09ec06ae70 0x7f09ec06d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.785+0000 7f0a14200700 1 -- 172.21.15.89:0/3352759563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c160e50 msgr2=0x7f0a0c0b8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:57.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.785+0000 7f0a14200700 1 --2- 172.21.15.89:0/3352759563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c160e50 0x7f0a0c0b8bf0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f09f800e910 tx=0x7f09f800ec20 comp rx=0 tx=0).stop 2024-04-02T07:55:57.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.785+0000 7f0a14200700 1 -- 172.21.15.89:0/3352759563 shutdown_connections 2024-04-02T07:55:57.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.785+0000 7f0a14200700 1 --2- 172.21.15.89:0/3352759563 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09ec06ae70 0x7f09ec06d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.785+0000 7f0a14200700 1 --2- 172.21.15.89:0/3352759563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a0c0b6130 0x7f0a0c06c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.785+0000 7f0a14200700 1 --2- 172.21.15.89:0/3352759563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a0c0b3f80 0x7f0a0c15bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.785+0000 7f0a14200700 1 --2- 172.21.15.89:0/3352759563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c160e50 0x7f0a0c0b8bf0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:57.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.785+0000 7f0a14200700 1 -- 172.21.15.89:0/3352759563 >> 172.21.15.89:0/3352759563 conn(0x7f0a0c08aff0 msgr2=0x7f0a0c06b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:57.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.785+0000 7f0a14200700 1 -- 172.21.15.89:0/3352759563 shutdown_connections 2024-04-02T07:55:57.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:57.785+0000 7f0a14200700 1 -- 172.21.15.89:0/3352759563 wait complete. 2024-04-02T07:55:57.805 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:55:58.037 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:58.036+0000 7f231fc35240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:55:58.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.045+0000 7f052a000700 1 Processor -- start 2024-04-02T07:55:58.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.045+0000 7f052a000700 1 -- start start 2024-04-02T07:55:58.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.047+0000 7f052a000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f05240d2540 0x7f05241687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:58.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.047+0000 7f052a000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0524161420 con 0x7f05240d16a0 2024-04-02T07:55:58.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.047+0000 7f052a000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0524161700 con 0x7f052415d9d0 2024-04-02T07:55:58.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.047+0000 7f052a000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05240d4340 con 0x7f05240d2540 2024-04-02T07:55:58.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.047+0000 7f0522a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f05240d16a0 0x7f05240d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59574/0 (socket says 172.21.15.89:59574) 2024-04-02T07:55:58.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.047+0000 7f0522a00700 1 -- 172.21.15.89:0/3328276580 learned_addr learned my addr 172.21.15.89:0/3328276580 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:58.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.047+0000 7f0523e00700 1 --2- 172.21.15.89:0/3328276580 >> v2:172.21.15.113:3300/0 conn(0x7f05240d2540 0x7f05241687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:58.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.048+0000 7f0522000700 1 -- 172.21.15.89:0/3328276580 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861630118 0 0) 0x7f0524161700 con 0x7f052415d9d0 2024-04-02T07:55:58.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.048+0000 7f0522000700 1 -- 172.21.15.89:0/3328276580 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05080035b0 con 0x7f052415d9d0 2024-04-02T07:55:58.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.048+0000 7f0522000700 1 -- 172.21.15.89:0/3328276580 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198154192 0 0) 0x7f0524161420 con 0x7f05240d16a0 2024-04-02T07:55:58.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.048+0000 7f0522000700 1 -- 172.21.15.89:0/3328276580 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0524161700 con 0x7f05240d16a0 2024-04-02T07:55:58.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.048+0000 7f0523e00700 1 -- 172.21.15.89:0/3328276580 >> v1:172.21.15.113:6789/0 conn(0x7f052415d9d0 legacy=0x7f05240d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:58.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.048+0000 7f0522000700 1 -- 172.21.15.89:0/3328276580 <== mon.1 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2105391235 0 0) 0x7f05080035b0 con 0x7f052415d9d0 2024-04-02T07:55:58.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.048+0000 7f0523e00700 1 -- 172.21.15.89:0/3328276580 >> v1:172.21.15.89:6789/0 conn(0x7f05240d16a0 legacy=0x7f05240d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:58.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.048+0000 7f0523e00700 1 -- 172.21.15.89:0/3328276580 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05241619e0 con 0x7f05240d2540 2024-04-02T07:55:58.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.048+0000 7f0523e00700 1 --2- 172.21.15.89:0/3328276580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05240d2540 0x7f05241687b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f051c009d90 tx=0x7f051c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffaf3d5b1511675b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:58.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.048+0000 7f0522000700 1 -- 172.21.15.89:0/3328276580 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c00e070 con 0x7f05240d2540 2024-04-02T07:55:58.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.049+0000 7f0522000700 1 -- 172.21.15.89:0/3328276580 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051c003a20 con 0x7f05240d2540 2024-04-02T07:55:58.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.049+0000 7f0522000700 1 -- 172.21.15.89:0/3328276580 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0514001670 con 0x7f05240d2540 2024-04-02T07:55:58.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.049+0000 7f052a000700 1 -- 172.21.15.89:0/3328276580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05240d2540 msgr2=0x7f05241687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:58.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.049+0000 7f052a000700 1 --2- 172.21.15.89:0/3328276580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05240d2540 0x7f05241687b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f051c009d90 tx=0x7f051c00b330 comp rx=0 tx=0).stop 2024-04-02T07:55:58.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.049+0000 7f052a000700 1 -- 172.21.15.89:0/3328276580 shutdown_connections 2024-04-02T07:55:58.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.049+0000 7f052a000700 1 --2- 172.21.15.89:0/3328276580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05240d2540 0x7f05241687b0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.049+0000 7f052a000700 1 -- 172.21.15.89:0/3328276580 >> 172.21.15.89:0/3328276580 conn(0x7f052408aff0 msgr2=0x7f05240d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:58.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.049+0000 7f052a000700 1 -- 172.21.15.89:0/3328276580 shutdown_connections 2024-04-02T07:55:58.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.050+0000 7f052a000700 1 -- 172.21.15.89:0/3328276580 wait complete. 2024-04-02T07:55:58.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.051+0000 7f052a000700 1 Processor -- start 2024-04-02T07:55:58.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.051+0000 7f052a000700 1 -- start start 2024-04-02T07:55:58.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.052+0000 7f052a000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05240d16a0 0x7f052415c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:58.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.052+0000 7f052a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05240d2540 0x7f05240cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:58.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.052+0000 7f052a000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f052415d9d0 0x7f05240cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:58.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.052+0000 7f052a000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05240cb9e0 con 0x7f05240d2540 2024-04-02T07:55:58.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.052+0000 7f052a000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05240ca8c0 con 0x7f052415d9d0 2024-04-02T07:55:58.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.052+0000 7f052a000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05240caa80 con 0x7f05240d16a0 2024-04-02T07:55:58.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.053+0000 7f0522a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05240d2540 0x7f05240cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:58.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.053+0000 7f0523e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f052415d9d0 0x7f05240cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:58.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.053+0000 7f0522a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05240d2540 0x7f05240cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55880/0 (socket says 172.21.15.89:55880) 2024-04-02T07:55:58.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.053+0000 7f0522a00700 1 -- 172.21.15.89:0/1446586218 learned_addr learned my addr 172.21.15.89:0/1446586218 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:58.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.053+0000 7f0523400700 1 --2- 172.21.15.89:0/1446586218 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05240d16a0 0x7f052415c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:58.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.053+0000 7f0523e00700 1 -- 172.21.15.89:0/1446586218 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05240d16a0 msgr2=0x7f052415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:58.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.053+0000 7f0523e00700 1 --2- 172.21.15.89:0/1446586218 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05240d16a0 0x7f052415c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.053+0000 7f0523e00700 1 -- 172.21.15.89:0/1446586218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05240d2540 msgr2=0x7f05240cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:58.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.053+0000 7f0523e00700 1 --2- 172.21.15.89:0/1446586218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05240d2540 0x7f05240cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.053+0000 7f0523e00700 1 -- 172.21.15.89:0/1446586218 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c009a70 con 0x7f052415d9d0 2024-04-02T07:55:58.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.053+0000 7f0523400700 1 --2- 172.21.15.89:0/1446586218 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05240d16a0 0x7f052415c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:55:58.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.053+0000 7f0523e00700 1 --2- 172.21.15.89:0/1446586218 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f052415d9d0 0x7f05240cc840 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f051c014000 tx=0x7f051c004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:58.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.054+0000 7f051b400700 1 -- 172.21.15.89:0/1446586218 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c004440 con 0x7f052415d9d0 2024-04-02T07:55:58.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.054+0000 7f051b400700 1 -- 172.21.15.89:0/1446586218 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051c0045a0 con 0x7f052415d9d0 2024-04-02T07:55:58.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.054+0000 7f052a000700 1 -- 172.21.15.89:0/1446586218 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05240cad60 con 0x7f052415d9d0 2024-04-02T07:55:58.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.054+0000 7f051b400700 1 -- 172.21.15.89:0/1446586218 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c004910 con 0x7f052415d9d0 2024-04-02T07:55:58.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.054+0000 7f052a000700 1 -- 172.21.15.89:0/1446586218 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05240cb320 con 0x7f052415d9d0 2024-04-02T07:55:58.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.055+0000 7f052a000700 1 -- 172.21.15.89:0/1446586218 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05240d1ca0 con 0x7f052415d9d0 2024-04-02T07:55:58.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.056+0000 7f051b400700 1 -- 172.21.15.89:0/1446586218 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f051c01d470 con 0x7f052415d9d0 2024-04-02T07:55:58.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.057+0000 7f051b400700 1 --2- 172.21.15.89:0/1446586218 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f050406ae40 0x7f050406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:58.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.057+0000 7f0523400700 1 -- 172.21.15.89:0/1446586218 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f050406ae40 msgr2=0x7f050406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:58.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.057+0000 7f051b400700 1 -- 172.21.15.89:0/1446586218 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f051c08b490 con 0x7f052415d9d0 2024-04-02T07:55:58.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.057+0000 7f0523400700 1 --2- 172.21.15.89:0/1446586218 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f050406ae40 0x7f050406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:58.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.062+0000 7f051b400700 1 -- 172.21.15.89:0/1446586218 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f051c056000 con 0x7f052415d9d0 2024-04-02T07:55:58.132 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:58.132+0000 7f3e94351240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:55:58.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.258+0000 7f0523400700 1 -- 172.21.15.89:0/1446586218 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f050406ae40 msgr2=0x7f050406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:58.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.258+0000 7f0523400700 1 --2- 172.21.15.89:0/1446586218 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f050406ae40 0x7f050406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:58.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.273+0000 7f052a000700 1 -- 172.21.15.89:0/1446586218 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f05240cbb20 con 0x7f052415d9d0 2024-04-02T07:55:58.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.274+0000 7f051b400700 1 -- 172.21.15.89:0/1446586218 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v141) v1 ==== 100+0+11 (secure 0 0 0) 0x7f05240cbb20 con 0x7f052415d9d0 2024-04-02T07:55:58.274 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:55:58.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.276+0000 7f052a000700 1 -- 172.21.15.89:0/1446586218 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f050406ae40 msgr2=0x7f050406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:58.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.276+0000 7f052a000700 1 --2- 172.21.15.89:0/1446586218 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f050406ae40 0x7f050406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.276+0000 7f052a000700 1 -- 172.21.15.89:0/1446586218 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f052415d9d0 msgr2=0x7f05240cc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:58.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.276+0000 7f052a000700 1 --2- 172.21.15.89:0/1446586218 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f052415d9d0 0x7f05240cc840 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f051c014000 tx=0x7f051c004ad0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.276+0000 7f052a000700 1 -- 172.21.15.89:0/1446586218 shutdown_connections 2024-04-02T07:55:58.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.276+0000 7f052a000700 1 --2- 172.21.15.89:0/1446586218 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f050406ae40 0x7f050406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.276+0000 7f052a000700 1 --2- 172.21.15.89:0/1446586218 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f052415d9d0 0x7f05240cc840 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.276+0000 7f052a000700 1 --2- 172.21.15.89:0/1446586218 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05240d16a0 0x7f052415c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.276+0000 7f052a000700 1 --2- 172.21.15.89:0/1446586218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05240d2540 0x7f05240cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.276+0000 7f052a000700 1 -- 172.21.15.89:0/1446586218 >> 172.21.15.89:0/1446586218 conn(0x7f052408aff0 msgr2=0x7f052406d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:58.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.277+0000 7f052a000700 1 -- 172.21.15.89:0/1446586218 shutdown_connections 2024-04-02T07:55:58.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.277+0000 7f052a000700 1 -- 172.21.15.89:0/1446586218 wait complete. 2024-04-02T07:55:58.295 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T07:55:58.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.491+0000 7f0ea3800700 1 Processor -- start 2024-04-02T07:55:58.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.491+0000 7f0ea3800700 1 -- start start 2024-04-02T07:55:58.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.492+0000 7f0ea3800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e9c0cbb30 0x7f0e9c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:58.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.492+0000 7f0ea3800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0e9c0cd4a0 0x7f0e9c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:58.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.492+0000 7f0ea3800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9c0cc450 con 0x7f0e9c0cd020 2024-04-02T07:55:58.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.492+0000 7f0ea3800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e9c0cdae0 con 0x7f0e9c0cd4a0 2024-04-02T07:55:58.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.492+0000 7f0ea3800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c0cddc0 con 0x7f0e9c0cbb30 2024-04-02T07:55:58.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.493+0000 7f0ea1800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0e9c0cd4a0 0x7f0e9c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:58.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.493+0000 7f0ea1800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0e9c0cd4a0 0x7f0e9c06e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55888/0 (socket says 172.21.15.89:55888) 2024-04-02T07:55:58.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.493+0000 7f0ea1800700 1 -- 172.21.15.89:0/1151487802 learned_addr learned my addr 172.21.15.89:0/1151487802 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:58.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.493+0000 7f0ea0e00700 1 --2- 172.21.15.89:0/1151487802 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e9c0cbb30 0x7f0e9c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:58.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.493+0000 7f0ea1800700 1 -- 172.21.15.89:0/1151487802 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e9c0cbb30 msgr2=0x7f0e9c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:58.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.493+0000 7f0ea1800700 1 --2- 172.21.15.89:0/1151487802 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e9c0cbb30 0x7f0e9c0cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.493+0000 7f0ea1800700 1 -- 172.21.15.89:0/1151487802 >> v1:172.21.15.113:6789/0 conn(0x7f0e9c0cd020 legacy=0x7f0e9c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:58.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.493+0000 7f0ea1800700 1 -- 172.21.15.89:0/1151487802 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c0ce0a0 con 0x7f0e9c0cd4a0 2024-04-02T07:55:58.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.493+0000 7f0e9b400700 1 -- 172.21.15.89:0/1151487802 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636866200 0 0) 0x7f0e9c0cc450 con 0x7f0e9c0cd020 2024-04-02T07:55:58.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.494+0000 7f0ea1800700 1 --2- 172.21.15.89:0/1151487802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e9c0cd4a0 0x7f0e9c06e420 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009e50 tx=0x7f0e9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925dee67e02883be server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:58.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.494+0000 7f0e9b400700 1 -- 172.21.15.89:0/1151487802 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9400e070 con 0x7f0e9c0cd4a0 2024-04-02T07:55:58.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.494+0000 7f0e9b400700 1 -- 172.21.15.89:0/1151487802 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9400b910 con 0x7f0e9c0cd4a0 2024-04-02T07:55:58.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.494+0000 7f0e9b400700 1 -- 172.21.15.89:0/1151487802 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9400bcc0 con 0x7f0e9c0cd4a0 2024-04-02T07:55:58.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.494+0000 7f0ea3800700 1 -- 172.21.15.89:0/1151487802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e9c0cd4a0 msgr2=0x7f0e9c06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:58.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.494+0000 7f0ea3800700 1 --2- 172.21.15.89:0/1151487802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e9c0cd4a0 0x7f0e9c06e420 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009e50 tx=0x7f0e9400b330 comp rx=0 tx=0).stop 2024-04-02T07:55:58.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.495+0000 7f0ea3800700 1 -- 172.21.15.89:0/1151487802 shutdown_connections 2024-04-02T07:55:58.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.495+0000 7f0ea3800700 1 --2- 172.21.15.89:0/1151487802 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e9c0cbb30 0x7f0e9c0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.495+0000 7f0ea3800700 1 --2- 172.21.15.89:0/1151487802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e9c0cd4a0 0x7f0e9c06e420 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.495+0000 7f0ea3800700 1 -- 172.21.15.89:0/1151487802 >> 172.21.15.89:0/1151487802 conn(0x7f0e9c0b0d70 msgr2=0x7f0e9c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:58.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.495+0000 7f0ea3800700 1 -- 172.21.15.89:0/1151487802 shutdown_connections 2024-04-02T07:55:58.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.495+0000 7f0ea3800700 1 -- 172.21.15.89:0/1151487802 wait complete. 2024-04-02T07:55:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.496+0000 7f0ea3800700 1 Processor -- start 2024-04-02T07:55:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.496+0000 7f0ea3800700 1 -- start start 2024-04-02T07:55:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.497+0000 7f0ea3800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e9c0cbb30 0x7f0e9c0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.497+0000 7f0ea3800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e9c0cd020 0x7f0e9c14e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.497+0000 7f0ea3800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e9c0cd4a0 0x7f0e9c0b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.497+0000 7f0ea3800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c0b3ca0 con 0x7f0e9c0cd020 2024-04-02T07:55:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.497+0000 7f0ea3800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c14eb60 con 0x7f0e9c0cd4a0 2024-04-02T07:55:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.497+0000 7f0ea3800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c0b3090 con 0x7f0e9c0cbb30 2024-04-02T07:55:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.498+0000 7f0e9be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e9c0cd020 0x7f0e9c14e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.498+0000 7f0e9be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e9c0cd020 0x7f0e9c14e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55890/0 (socket says 172.21.15.89:55890) 2024-04-02T07:55:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.498+0000 7f0e9be00700 1 -- 172.21.15.89:0/691923708 learned_addr learned my addr 172.21.15.89:0/691923708 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.498+0000 7f0ea1800700 1 --2- 172.21.15.89:0/691923708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e9c0cd4a0 0x7f0e9c0b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.498+0000 7f0ea0e00700 1 --2- 172.21.15.89:0/691923708 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e9c0cbb30 0x7f0e9c0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:58.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.498+0000 7f0e9be00700 1 -- 172.21.15.89:0/691923708 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e9c0cbb30 msgr2=0x7f0e9c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:58.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.498+0000 7f0e9be00700 1 --2- 172.21.15.89:0/691923708 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e9c0cbb30 0x7f0e9c0ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.498+0000 7f0e9be00700 1 -- 172.21.15.89:0/691923708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e9c0cd4a0 msgr2=0x7f0e9c0b2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:58.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.498+0000 7f0e9be00700 1 --2- 172.21.15.89:0/691923708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e9c0cd4a0 0x7f0e9c0b2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.498+0000 7f0e9be00700 1 -- 172.21.15.89:0/691923708 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e94009b30 con 0x7f0e9c0cd020 2024-04-02T07:55:58.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.498+0000 7f0e9be00700 1 --2- 172.21.15.89:0/691923708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e9c0cd020 0x7f0e9c14e620 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800c580 tx=0x7f0e8800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:58.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.499+0000 7f0e98c00700 1 -- 172.21.15.89:0/691923708 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e880054e0 con 0x7f0e9c0cd020 2024-04-02T07:55:58.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.499+0000 7f0e98c00700 1 -- 172.21.15.89:0/691923708 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e88003680 con 0x7f0e9c0cd020 2024-04-02T07:55:58.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.499+0000 7f0e98c00700 1 -- 172.21.15.89:0/691923708 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e88003a30 con 0x7f0e9c0cd020 2024-04-02T07:55:58.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.499+0000 7f0ea3800700 1 -- 172.21.15.89:0/691923708 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e9c0b3370 con 0x7f0e9c0cd020 2024-04-02T07:55:58.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.499+0000 7f0ea3800700 1 -- 172.21.15.89:0/691923708 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e9c0b3930 con 0x7f0e9c0cd020 2024-04-02T07:55:58.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.500+0000 7f0ea3800700 1 -- 172.21.15.89:0/691923708 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e9c0b3e50 con 0x7f0e9c0cd020 2024-04-02T07:55:58.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.501+0000 7f0e98c00700 1 -- 172.21.15.89:0/691923708 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f0e88003b90 con 0x7f0e9c0cd020 2024-04-02T07:55:58.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.502+0000 7f0e98c00700 1 --2- 172.21.15.89:0/691923708 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e7c06ae40 0x7f0e7c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:58.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.502+0000 7f0ea0e00700 1 -- 172.21.15.89:0/691923708 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e7c06ae40 msgr2=0x7f0e7c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:58.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.502+0000 7f0e98c00700 1 -- 172.21.15.89:0/691923708 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f0e88088cb0 con 0x7f0e9c0cd020 2024-04-02T07:55:58.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.502+0000 7f0ea0e00700 1 --2- 172.21.15.89:0/691923708 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e7c06ae40 0x7f0e7c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:58.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.511+0000 7f0e98c00700 1 -- 172.21.15.89:0/691923708 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0e880538a0 con 0x7f0e9c0cd020 2024-04-02T07:55:58.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.703+0000 7f0ea0e00700 1 -- 172.21.15.89:0/691923708 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e7c06ae40 msgr2=0x7f0e7c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:58.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.703+0000 7f0ea0e00700 1 --2- 172.21.15.89:0/691923708 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e7c06ae40 0x7f0e7c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:58.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.735+0000 7f0ea3800700 1 -- 172.21.15.89:0/691923708 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0e9c15bda0 con 0x7f0e9c0cd020 2024-04-02T07:55:58.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.736+0000 7f0e98c00700 1 -- 172.21.15.89:0/691923708 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v141) v1 ==== 80+0+457 (secure 0 0 0) 0x7f0e88017070 con 0x7f0e9c0cd020 2024-04-02T07:55:58.737 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/836334016 2024-04-03T07:55:47.785863+0000 2024-04-02T07:55:58.737 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1824838019 2024-04-03T07:55:47.785863+0000 2024-04-02T07:55:58.737 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2769562366 2024-04-03T07:55:47.785863+0000 2024-04-02T07:55:58.737 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/224945013 2024-04-03T07:55:47.785863+0000 2024-04-02T07:55:58.737 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/732038760 2024-04-03T07:55:47.785863+0000 2024-04-02T07:55:58.737 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/43357 2024-04-03T07:55:47.785863+0000 2024-04-02T07:55:58.737 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/43357 2024-04-03T07:55:47.785863+0000 2024-04-02T07:55:58.737 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3727408218 2024-04-03T07:55:47.785863+0000 2024-04-02T07:55:58.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.739+0000 7f0ea3800700 1 -- 172.21.15.89:0/691923708 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e7c06ae40 msgr2=0x7f0e7c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:58.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.739+0000 7f0ea3800700 1 --2- 172.21.15.89:0/691923708 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e7c06ae40 0x7f0e7c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.739+0000 7f0ea3800700 1 -- 172.21.15.89:0/691923708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e9c0cd020 msgr2=0x7f0e9c14e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:58.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.739+0000 7f0ea3800700 1 --2- 172.21.15.89:0/691923708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e9c0cd020 0x7f0e9c14e620 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800c580 tx=0x7f0e8800c890 comp rx=0 tx=0).stop 2024-04-02T07:55:58.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.739+0000 7f0ea3800700 1 -- 172.21.15.89:0/691923708 shutdown_connections 2024-04-02T07:55:58.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.739+0000 7f0ea3800700 1 --2- 172.21.15.89:0/691923708 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e7c06ae40 0x7f0e7c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.739+0000 7f0ea3800700 1 --2- 172.21.15.89:0/691923708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e9c0cd4a0 0x7f0e9c0b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.739+0000 7f0ea3800700 1 --2- 172.21.15.89:0/691923708 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e9c0cbb30 0x7f0e9c0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.739+0000 7f0ea3800700 1 --2- 172.21.15.89:0/691923708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e9c0cd020 0x7f0e9c14e620 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:58.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.739+0000 7f0ea3800700 1 -- 172.21.15.89:0/691923708 >> 172.21.15.89:0/691923708 conn(0x7f0e9c0b0d70 msgr2=0x7f0e9c15d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:58.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.739+0000 7f0ea3800700 1 -- 172.21.15.89:0/691923708 shutdown_connections 2024-04-02T07:55:58.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:58.739+0000 7f0ea3800700 1 -- 172.21.15.89:0/691923708 wait complete. 2024-04-02T07:55:58.741 INFO:teuthology.orchestra.run.smithi089.stderr:listed 8 entries 2024-04-02T07:55:58.748 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:58.748+0000 7f231fc35240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:55:58.757 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T07:55:58.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:58.758 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T07:55:58.836 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T07:55:58.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:55:58.837 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T07:55:58.841 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:58.840+0000 7f231fc35240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:55:58.882 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:55:58.882 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:58.888 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:58.888+0000 7f3e94351240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:55:58.926 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:55:58.926 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:55:58.926 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:58.971 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:55:58.971 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:55:58.971 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:55:58.985 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:58.985+0000 7f3e94351240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:55:59.015 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:59.013+0000 7f231fc35240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:55:59.015 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:55:59.015 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:55:59.154 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:59.153+0000 7f3e94351240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:55:59.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.211+0000 7fe6d9e00700 1 Processor -- start 2024-04-02T07:55:59.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.211+0000 7fe6d9e00700 1 -- start start 2024-04-02T07:55:59.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.213+0000 7fe6d9e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6d415d920 0x7fe6d406c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:59.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.213+0000 7fe6d9e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6d409a910 0x7fe6d40cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:59.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.213+0000 7fe6d9e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d40d08d0 con 0x7fe6d4099a60 2024-04-02T07:55:59.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.213+0000 7fe6d9e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d406c300 con 0x7fe6d415d920 2024-04-02T07:55:59.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.213+0000 7fe6d9e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d406b9d0 con 0x7fe6d409a910 2024-04-02T07:55:59.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.214+0000 7fe6d2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6d415d920 0x7fe6d406c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:59.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.214+0000 7fe6d2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6d415d920 0x7fe6d406c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54118/0 (socket says 172.21.15.89:54118) 2024-04-02T07:55:59.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.214+0000 7fe6d2a00700 1 -- 172.21.15.89:0/2901623860 learned_addr learned my addr 172.21.15.89:0/2901623860 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:59.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.214+0000 7fe6d3400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe6d4099a60 0x7fe6d40cb9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55512/0 (socket says 172.21.15.89:55512) 2024-04-02T07:55:59.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.214+0000 7fe6cbe00700 1 --2- 172.21.15.89:0/2901623860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6d409a910 0x7fe6d40cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:59.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.214+0000 7fe6d2a00700 1 -- 172.21.15.89:0/2901623860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6d409a910 msgr2=0x7fe6d40cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:55:59.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.214+0000 7fe6d2a00700 1 --2- 172.21.15.89:0/2901623860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6d409a910 0x7fe6d40cfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.214+0000 7fe6d2a00700 1 -- 172.21.15.89:0/2901623860 >> v1:172.21.15.89:6789/0 conn(0x7fe6d4099a60 legacy=0x7fe6d40cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:59.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.214+0000 7fe6d2a00700 1 -- 172.21.15.89:0/2901623860 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d406bcb0 con 0x7fe6d415d920 2024-04-02T07:55:59.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.214+0000 7fe6d2000700 1 -- 172.21.15.89:0/2901623860 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365158544 0 0) 0x7fe6d40d08d0 con 0x7fe6d4099a60 2024-04-02T07:55:59.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.214+0000 7fe6d2a00700 1 --2- 172.21.15.89:0/2901623860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d415d920 0x7fe6d406c790 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40098f0 tx=0x7fe6c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f80c9c002793bb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:59.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.215+0000 7fe6d2000700 1 -- 172.21.15.89:0/2901623860 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4011830 con 0x7fe6d415d920 2024-04-02T07:55:59.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.215+0000 7fe6d2000700 1 -- 172.21.15.89:0/2901623860 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c4011990 con 0x7fe6d415d920 2024-04-02T07:55:59.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.215+0000 7fe6d2000700 1 -- 172.21.15.89:0/2901623860 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4011d20 con 0x7fe6d415d920 2024-04-02T07:55:59.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.216+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2901623860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d415d920 msgr2=0x7fe6d406c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:59.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.216+0000 7fe6d9e00700 1 --2- 172.21.15.89:0/2901623860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d415d920 0x7fe6d406c790 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40098f0 tx=0x7fe6c4009c00 comp rx=0 tx=0).stop 2024-04-02T07:55:59.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.216+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2901623860 shutdown_connections 2024-04-02T07:55:59.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.216+0000 7fe6d9e00700 1 --2- 172.21.15.89:0/2901623860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d415d920 0x7fe6d406c790 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.216+0000 7fe6d9e00700 1 --2- 172.21.15.89:0/2901623860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6d409a910 0x7fe6d40cfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.216+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2901623860 >> 172.21.15.89:0/2901623860 conn(0x7fe6d40d1350 msgr2=0x7fe6d40d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:59.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.216+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2901623860 shutdown_connections 2024-04-02T07:55:59.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.216+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2901623860 wait complete. 2024-04-02T07:55:59.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.217+0000 7fe6d9e00700 1 Processor -- start 2024-04-02T07:55:59.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.218+0000 7fe6d9e00700 1 -- start start 2024-04-02T07:55:59.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.218+0000 7fe6d9e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6d4099a60 0x7fe6d41f37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:59.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.218+0000 7fe6d9e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6d409a910 0x7fe6d41f6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:59.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6d9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d415d920 0x7fe6d40af170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:59.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6d9e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d41f3d10 con 0x7fe6d415d920 2024-04-02T07:55:59.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6d9e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d415f980 con 0x7fe6d409a910 2024-04-02T07:55:59.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6d9e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d415fc60 con 0x7fe6d4099a60 2024-04-02T07:55:59.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6d3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d415d920 0x7fe6d40af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:59.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6d3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d415d920 0x7fe6d40af170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54126/0 (socket says 172.21.15.89:54126) 2024-04-02T07:55:59.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6d3400700 1 -- 172.21.15.89:0/2341753736 learned_addr learned my addr 172.21.15.89:0/2341753736 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:59.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6d2a00700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6d4099a60 0x7fe6d41f37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:59.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6cbe00700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6d409a910 0x7fe6d41f6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:59.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6d3400700 1 -- 172.21.15.89:0/2341753736 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6d4099a60 msgr2=0x7fe6d41f37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:59.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6d3400700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6d4099a60 0x7fe6d41f37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6d3400700 1 -- 172.21.15.89:0/2341753736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6d409a910 msgr2=0x7fe6d41f6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:59.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6d3400700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6d409a910 0x7fe6d41f6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6d3400700 1 -- 172.21.15.89:0/2341753736 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c4009550 con 0x7fe6d415d920 2024-04-02T07:55:59.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6cbe00700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6d409a910 0x7fe6d41f6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:55:59.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.219+0000 7fe6d3400700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d415d920 0x7fe6d40af170 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc002230 tx=0x7fe6cc00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:59.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.220+0000 7fe6cb400700 1 -- 172.21.15.89:0/2341753736 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc00ab70 con 0x7fe6d415d920 2024-04-02T07:55:59.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.220+0000 7fe6cb400700 1 -- 172.21.15.89:0/2341753736 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6cc00acf0 con 0x7fe6d415d920 2024-04-02T07:55:59.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.220+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2341753736 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d415ff40 con 0x7fe6d415d920 2024-04-02T07:55:59.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.220+0000 7fe6cb400700 1 -- 172.21.15.89:0/2341753736 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc007310 con 0x7fe6d415d920 2024-04-02T07:55:59.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.220+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2341753736 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d4160500 con 0x7fe6d415d920 2024-04-02T07:55:59.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.221+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2341753736 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d4160720 con 0x7fe6d415d920 2024-04-02T07:55:59.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.222+0000 7fe6cb400700 1 -- 172.21.15.89:0/2341753736 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fe6cc007470 con 0x7fe6d415d920 2024-04-02T07:55:59.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.223+0000 7fe6cb400700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6b006ae40 0x7fe6b006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:59.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.223+0000 7fe6cb400700 1 -- 172.21.15.89:0/2341753736 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fe6cc089760 con 0x7fe6d415d920 2024-04-02T07:55:59.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.223+0000 7fe6d2a00700 1 -- 172.21.15.89:0/2341753736 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6b006ae40 msgr2=0x7fe6b006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:59.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.223+0000 7fe6d2a00700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6b006ae40 0x7fe6b006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:59.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.227+0000 7fe6cb400700 1 -- 172.21.15.89:0/2341753736 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6cc0543f0 con 0x7fe6d415d920 2024-04-02T07:55:59.230 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:59.229+0000 7f231fc35240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:55:59.323 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:59.322+0000 7f231fc35240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:55:59.378 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:59.378+0000 7f3e94351240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:55:59.421 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:59.420+0000 7f231fc35240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:55:59.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.424+0000 7fe6d2a00700 1 -- 172.21.15.89:0/2341753736 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6b006ae40 msgr2=0x7fe6b006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:59.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.424+0000 7fe6d2a00700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6b006ae40 0x7fe6b006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:59.474 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:59.474+0000 7f3e94351240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:55:59.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.481+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2341753736 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6d4067720 con 0x7fe6d415d920 2024-04-02T07:55:59.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.481+0000 7fe6cb400700 1 -- 172.21.15.89:0/2341753736 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4733 (secure 0 0 0) 0x7fe6cc00ae60 con 0x7fe6d415d920 2024-04-02T07:55:59.482 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:59.482 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/66826967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":66826967},{"type":"v1","addr":"172.21.15.113:6835","nonce":66826967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":7915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/1669765650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1669765650},{"type":"v1","addr":"172.21.15.113:6833","nonce":1669765650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":7918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3321722319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3321722319},{"type":"v1","addr":"172.21.15.89:6837","nonce":3321722319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:55:31.571731+0000","modified":"2024-04-02T07:55:34.583055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8436":{"gid":8436,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":4,"addr":"172.21.15.89:6836/417215494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":417215494},{"type":"v1","addr":"172.21.15.89:6836","nonce":417215494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-02T07:55:59.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.484+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2341753736 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6b006ae40 msgr2=0x7fe6b006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:59.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.484+0000 7fe6d9e00700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6b006ae40 0x7fe6b006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.484+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2341753736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d415d920 msgr2=0x7fe6d40af170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:59.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.484+0000 7fe6d9e00700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d415d920 0x7fe6d40af170 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc002230 tx=0x7fe6cc00cf50 comp rx=0 tx=0).stop 2024-04-02T07:55:59.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.484+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2341753736 shutdown_connections 2024-04-02T07:55:59.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.484+0000 7fe6d9e00700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe6b006ae40 0x7fe6b006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.484+0000 7fe6d9e00700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6d415d920 0x7fe6d40af170 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.484+0000 7fe6d9e00700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6d4099a60 0x7fe6d41f37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.484+0000 7fe6d9e00700 1 --2- 172.21.15.89:0/2341753736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6d409a910 0x7fe6d41f6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.484+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2341753736 >> 172.21.15.89:0/2341753736 conn(0x7fe6d40d1350 msgr2=0x7fe6d40b12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:59.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.484+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2341753736 shutdown_connections 2024-04-02T07:55:59.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.484+0000 7fe6d9e00700 1 -- 172.21.15.89:0/2341753736 wait complete. 2024-04-02T07:55:59.486 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 66 2024-04-02T07:55:59.503 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:55:59.518 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:59.517+0000 7f231fc35240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:55:59.582 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:59.581+0000 7f3e94351240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:55:59.618 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:59.617+0000 7f231fc35240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:55:59.687 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:59.686+0000 7f3e94351240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:55:59.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.699+0000 7fd569a00700 1 Processor -- start 2024-04-02T07:55:59.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.699+0000 7fd569a00700 1 -- start start 2024-04-02T07:55:59.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.700+0000 7fd569a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd564160f10 0x7fd5640d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:59.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.700+0000 7fd569a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5640ceda0 con 0x7fd5640cf230 2024-04-02T07:55:59.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.700+0000 7fd569a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5641612f0 con 0x7fd564161590 2024-04-02T07:55:59.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.700+0000 7fd569a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5640ce470 con 0x7fd564160f10 2024-04-02T07:55:59.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.701+0000 7fd562000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd5640cf230 0x7fd5640d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55518/0 (socket says 172.21.15.89:55518) 2024-04-02T07:55:59.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.701+0000 7fd562000700 1 -- 172.21.15.89:0/2442141596 learned_addr learned my addr 172.21.15.89:0/2442141596 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:55:59.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.701+0000 7fd562a00700 1 --2- 172.21.15.89:0/2442141596 >> v2:172.21.15.113:3300/0 conn(0x7fd564160f10 0x7fd5640d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:59.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.701+0000 7fd561600700 1 -- 172.21.15.89:0/2442141596 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513652143 0 0) 0x7fd5640ceda0 con 0x7fd5640cf230 2024-04-02T07:55:59.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.701+0000 7fd561600700 1 -- 172.21.15.89:0/2442141596 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5480035b0 con 0x7fd5640cf230 2024-04-02T07:55:59.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.701+0000 7fd561600700 1 -- 172.21.15.89:0/2442141596 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904142038 0 0) 0x7fd5641612f0 con 0x7fd564161590 2024-04-02T07:55:59.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.701+0000 7fd561600700 1 -- 172.21.15.89:0/2442141596 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5640ceda0 con 0x7fd564161590 2024-04-02T07:55:59.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.701+0000 7fd562a00700 1 -- 172.21.15.89:0/2442141596 >> v1:172.21.15.113:6789/0 conn(0x7fd564161590 legacy=0x7fd5640b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:59.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.701+0000 7fd562a00700 1 -- 172.21.15.89:0/2442141596 >> v1:172.21.15.89:6789/0 conn(0x7fd5640cf230 legacy=0x7fd5640d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:59.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.701+0000 7fd562a00700 1 -- 172.21.15.89:0/2442141596 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5640ce750 con 0x7fd564160f10 2024-04-02T07:55:59.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.701+0000 7fd562a00700 1 --2- 172.21.15.89:0/2442141596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd564160f10 0x7fd5640d18b0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd554009840 tx=0x7fd554009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55a01821aca3cb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:59.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.702+0000 7fd561600700 1 -- 172.21.15.89:0/2442141596 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd554011830 con 0x7fd564160f10 2024-04-02T07:55:59.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.702+0000 7fd561600700 1 -- 172.21.15.89:0/2442141596 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd554011990 con 0x7fd564160f10 2024-04-02T07:55:59.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.702+0000 7fd561600700 1 -- 172.21.15.89:0/2442141596 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd554011d20 con 0x7fd564160f10 2024-04-02T07:55:59.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.702+0000 7fd569a00700 1 -- 172.21.15.89:0/2442141596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd564160f10 msgr2=0x7fd5640d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:59.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.702+0000 7fd569a00700 1 --2- 172.21.15.89:0/2442141596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd564160f10 0x7fd5640d18b0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd554009840 tx=0x7fd554009b50 comp rx=0 tx=0).stop 2024-04-02T07:55:59.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.702+0000 7fd569a00700 1 -- 172.21.15.89:0/2442141596 shutdown_connections 2024-04-02T07:55:59.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.702+0000 7fd569a00700 1 --2- 172.21.15.89:0/2442141596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd564160f10 0x7fd5640d18b0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.702+0000 7fd569a00700 1 -- 172.21.15.89:0/2442141596 >> 172.21.15.89:0/2442141596 conn(0x7fd56408aff0 msgr2=0x7fd5640d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:59.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.702+0000 7fd569a00700 1 -- 172.21.15.89:0/2442141596 shutdown_connections 2024-04-02T07:55:59.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.703+0000 7fd569a00700 1 -- 172.21.15.89:0/2442141596 wait complete. 2024-04-02T07:55:59.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.704+0000 7fd569a00700 1 Processor -- start 2024-04-02T07:55:59.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.704+0000 7fd569a00700 1 -- start start 2024-04-02T07:55:59.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.705+0000 7fd569a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5640cf230 0x7fd5640b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:59.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.705+0000 7fd569a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd564160f10 0x7fd5640cc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:59.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.705+0000 7fd569a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd564161590 0x7fd5640ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:59.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.705+0000 7fd569a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5640b0290 con 0x7fd5640cf230 2024-04-02T07:55:59.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.705+0000 7fd569a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56415fd10 con 0x7fd564161590 2024-04-02T07:55:59.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.705+0000 7fd569a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56415fff0 con 0x7fd564160f10 2024-04-02T07:55:59.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.705+0000 7fd562a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5640cf230 0x7fd5640b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:59.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.706+0000 7fd562a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5640cf230 0x7fd5640b3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54140/0 (socket says 172.21.15.89:54140) 2024-04-02T07:55:59.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.706+0000 7fd562a00700 1 -- 172.21.15.89:0/2105096673 learned_addr learned my addr 172.21.15.89:0/2105096673 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:55:59.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.706+0000 7fd562000700 1 --2- 172.21.15.89:0/2105096673 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd564160f10 0x7fd5640cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:59.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.706+0000 7fd563400700 1 --2- 172.21.15.89:0/2105096673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd564161590 0x7fd5640ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:55:59.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.706+0000 7fd562a00700 1 -- 172.21.15.89:0/2105096673 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd564160f10 msgr2=0x7fd5640cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:59.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.706+0000 7fd562a00700 1 --2- 172.21.15.89:0/2105096673 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd564160f10 0x7fd5640cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.706+0000 7fd562a00700 1 -- 172.21.15.89:0/2105096673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd564161590 msgr2=0x7fd5640ccaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:59.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.706+0000 7fd562a00700 1 --2- 172.21.15.89:0/2105096673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd564161590 0x7fd5640ccaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.706+0000 7fd562a00700 1 -- 172.21.15.89:0/2105096673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5540094a0 con 0x7fd5640cf230 2024-04-02T07:55:59.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.706+0000 7fd562a00700 1 --2- 172.21.15.89:0/2105096673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5640cf230 0x7fd5640b3bf0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fd554009b90 tx=0x7fd554004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:55:59.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.707+0000 7fd55aa00700 1 -- 172.21.15.89:0/2105096673 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55401a760 con 0x7fd5640cf230 2024-04-02T07:55:59.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.707+0000 7fd55aa00700 1 -- 172.21.15.89:0/2105096673 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55401a8c0 con 0x7fd5640cf230 2024-04-02T07:55:59.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.707+0000 7fd569a00700 1 -- 172.21.15.89:0/2105096673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5641602d0 con 0x7fd5640cf230 2024-04-02T07:55:59.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.707+0000 7fd55aa00700 1 -- 172.21.15.89:0/2105096673 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55401ac70 con 0x7fd5640cf230 2024-04-02T07:55:59.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.707+0000 7fd569a00700 1 -- 172.21.15.89:0/2105096673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5640b4130 con 0x7fd5640cf230 2024-04-02T07:55:59.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.708+0000 7fd569a00700 1 -- 172.21.15.89:0/2105096673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5640b0720 con 0x7fd5640cf230 2024-04-02T07:55:59.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.710+0000 7fd55aa00700 1 -- 172.21.15.89:0/2105096673 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fd554022070 con 0x7fd5640cf230 2024-04-02T07:55:59.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.710+0000 7fd55aa00700 1 --2- 172.21.15.89:0/2105096673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd54406ae40 0x7fd54406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:55:59.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.711+0000 7fd55aa00700 1 -- 172.21.15.89:0/2105096673 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fd55408a510 con 0x7fd5640cf230 2024-04-02T07:55:59.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.711+0000 7fd562000700 1 -- 172.21.15.89:0/2105096673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd54406ae40 msgr2=0x7fd54406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:59.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.711+0000 7fd562000700 1 --2- 172.21.15.89:0/2105096673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd54406ae40 0x7fd54406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:55:59.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.715+0000 7fd55aa00700 1 -- 172.21.15.89:0/2105096673 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd5640b0720 con 0x7fd5640cf230 2024-04-02T07:55:59.792 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:55:59.792+0000 7f3e94351240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:55:59.833 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:55:59.832+0000 7f231fc35240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:55:59.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.912+0000 7fd562000700 1 -- 172.21.15.89:0/2105096673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd54406ae40 msgr2=0x7fd54406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:55:59.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.912+0000 7fd562000700 1 --2- 172.21.15.89:0/2105096673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd54406ae40 0x7fd54406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:55:59.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.925+0000 7fd569a00700 1 -- 172.21.15.89:0/2105096673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5640b0720 con 0x7fd5640cf230 2024-04-02T07:55:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.927+0000 7fd55aa00700 1 -- 172.21.15.89:0/2105096673 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+17175 (secure 0 0 0) 0x7fd5640b0720 con 0x7fd5640cf230 2024-04-02T07:55:59.927 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:55:59.928 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":141,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:55:47.785904+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:55:29.329907+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-02T07:55:30.555350+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/836334016":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/1824838019":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/2769562366":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/224945013":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/732038760":"2024-04-03T07:55:47.785863+0000","172.21.15.89:6833/43357":"2024-04-03T07:55:47.785863+0000","172.21.15.89:6832/43357":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/3727408218":"2024-04-03T07:55:47.785863+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:55:59.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.929+0000 7fd569a00700 1 -- 172.21.15.89:0/2105096673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd54406ae40 msgr2=0x7fd54406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:55:59.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.929+0000 7fd569a00700 1 --2- 172.21.15.89:0/2105096673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd54406ae40 0x7fd54406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.929+0000 7fd569a00700 1 -- 172.21.15.89:0/2105096673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5640cf230 msgr2=0x7fd5640b3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:55:59.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.929+0000 7fd569a00700 1 --2- 172.21.15.89:0/2105096673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5640cf230 0x7fd5640b3bf0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fd554009b90 tx=0x7fd554004750 comp rx=0 tx=0).stop 2024-04-02T07:55:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.929+0000 7fd569a00700 1 -- 172.21.15.89:0/2105096673 shutdown_connections 2024-04-02T07:55:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.929+0000 7fd569a00700 1 --2- 172.21.15.89:0/2105096673 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd54406ae40 0x7fd54406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.929+0000 7fd569a00700 1 --2- 172.21.15.89:0/2105096673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd564161590 0x7fd5640ccaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.929+0000 7fd569a00700 1 --2- 172.21.15.89:0/2105096673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5640cf230 0x7fd5640b3bf0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.929+0000 7fd569a00700 1 --2- 172.21.15.89:0/2105096673 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd564160f10 0x7fd5640cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:55:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.930+0000 7fd569a00700 1 -- 172.21.15.89:0/2105096673 >> 172.21.15.89:0/2105096673 conn(0x7fd56408aff0 msgr2=0x7fd5640cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:55:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.930+0000 7fd569a00700 1 -- 172.21.15.89:0/2105096673 shutdown_connections 2024-04-02T07:55:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:55:59.930+0000 7fd569a00700 1 -- 172.21.15.89:0/2105096673 wait complete. 2024-04-02T07:55:59.950 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:55:29.329907+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:55:30.555350+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:55:59.950 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:56:00.016 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:00.015+0000 7f3e94351240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:56:00.089 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:00.088+0000 7f231fc35240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:56:00.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.149+0000 7f068c000700 1 Processor -- start 2024-04-02T07:56:00.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.149+0000 7f068c000700 1 -- start start 2024-04-02T07:56:00.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.150+0000 7f068c000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0684160d70 0x7f06840b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:00.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.150+0000 7f068c000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f068406a790 0x7f068406ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:00.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.150+0000 7f068c000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0684161150 con 0x7f068415ef70 2024-04-02T07:56:00.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.150+0000 7f068c000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f068406b180 con 0x7f0684160d70 2024-04-02T07:56:00.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.150+0000 7f068c000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06840b4cd0 con 0x7f068406a790 2024-04-02T07:56:00.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.150+0000 7f0688c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f068415ef70 0x7f068415e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55520/0 (socket says 172.21.15.89:55520) 2024-04-02T07:56:00.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.150+0000 7f0688c00700 1 -- 172.21.15.89:0/4273740743 learned_addr learned my addr 172.21.15.89:0/4273740743 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:00.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.150+0000 7f068a000700 1 --2- 172.21.15.89:0/4273740743 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f068406a790 0x7f068406ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:00.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.151+0000 7f0689600700 1 --2- 172.21.15.89:0/4273740743 >> v2:172.21.15.113:3300/0 conn(0x7f0684160d70 0x7f06840b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:00.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.151+0000 7f0683e00700 1 -- 172.21.15.89:0/4273740743 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266168887 0 0) 0x7f0684161150 con 0x7f068415ef70 2024-04-02T07:56:00.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.151+0000 7f0683e00700 1 -- 172.21.15.89:0/4273740743 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f066c0035b0 con 0x7f068415ef70 2024-04-02T07:56:00.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.151+0000 7f068a000700 1 -- 172.21.15.89:0/4273740743 >> v2:172.21.15.113:3300/0 conn(0x7f0684160d70 msgr2=0x7f06840b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:00.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.151+0000 7f068a000700 1 --2- 172.21.15.89:0/4273740743 >> v2:172.21.15.113:3300/0 conn(0x7f0684160d70 0x7f06840b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.151+0000 7f068a000700 1 -- 172.21.15.89:0/4273740743 >> v1:172.21.15.89:6789/0 conn(0x7f068415ef70 legacy=0x7f068415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:00.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.151+0000 7f068a000700 1 -- 172.21.15.89:0/4273740743 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068406dba0 con 0x7f068406a790 2024-04-02T07:56:00.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.151+0000 7f068a000700 1 --2- 172.21.15.89:0/4273740743 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f068406a790 0x7f068406ac40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f067c009d90 tx=0x7f067c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ac76f370b900a7d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.151+0000 7f0683e00700 1 -- 172.21.15.89:0/4273740743 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c00e070 con 0x7f068406a790 2024-04-02T07:56:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.152+0000 7f0683e00700 1 -- 172.21.15.89:0/4273740743 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067c00bb20 con 0x7f068406a790 2024-04-02T07:56:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.152+0000 7f0683e00700 1 -- 172.21.15.89:0/4273740743 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c012430 con 0x7f068406a790 2024-04-02T07:56:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.152+0000 7f068c000700 1 -- 172.21.15.89:0/4273740743 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f068406a790 msgr2=0x7f068406ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.152+0000 7f068c000700 1 --2- 172.21.15.89:0/4273740743 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f068406a790 0x7f068406ac40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f067c009d90 tx=0x7f067c00b330 comp rx=0 tx=0).stop 2024-04-02T07:56:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.152+0000 7f068c000700 1 -- 172.21.15.89:0/4273740743 shutdown_connections 2024-04-02T07:56:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.152+0000 7f068c000700 1 --2- 172.21.15.89:0/4273740743 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f068406a790 0x7f068406ac40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.152+0000 7f068c000700 1 --2- 172.21.15.89:0/4273740743 >> v2:172.21.15.113:3300/0 conn(0x7f0684160d70 0x7f06840b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.152+0000 7f068c000700 1 -- 172.21.15.89:0/4273740743 >> 172.21.15.89:0/4273740743 conn(0x7f06840b5720 msgr2=0x7f06840b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.153+0000 7f068c000700 1 -- 172.21.15.89:0/4273740743 shutdown_connections 2024-04-02T07:56:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.153+0000 7f068c000700 1 -- 172.21.15.89:0/4273740743 wait complete. 2024-04-02T07:56:00.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.154+0000 7f068c000700 1 Processor -- start 2024-04-02T07:56:00.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.155+0000 7f068c000700 1 -- start start 2024-04-02T07:56:00.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.155+0000 7f068c000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f068406a790 0x7f0684146d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:00.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.155+0000 7f068c000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068415ef70 0x7f06840b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:00.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f068c000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0684160d70 0x7f0684148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:00.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f068c000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06840b3020 con 0x7f068415ef70 2024-04-02T07:56:00.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f068c000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06841478c0 con 0x7f068406a790 2024-04-02T07:56:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f068c000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0684147ba0 con 0x7f0684160d70 2024-04-02T07:56:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f0688c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068415ef70 0x7f06840b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f0688c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068415ef70 0x7f06840b2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54148/0 (socket says 172.21.15.89:54148) 2024-04-02T07:56:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f0688c00700 1 -- 172.21.15.89:0/150612492 learned_addr learned my addr 172.21.15.89:0/150612492 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f0689600700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f068406a790 0x7f0684146d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f068a000700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0684160d70 0x7f0684148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f0688c00700 1 -- 172.21.15.89:0/150612492 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0684160d70 msgr2=0x7f0684148960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f0688c00700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0684160d70 0x7f0684148960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f0688c00700 1 -- 172.21.15.89:0/150612492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f068406a790 msgr2=0x7f0684146d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f0688c00700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f068406a790 0x7f0684146d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f0688c00700 1 -- 172.21.15.89:0/150612492 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c009a70 con 0x7f068415ef70 2024-04-02T07:56:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.156+0000 7f0689600700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f068406a790 0x7f0684146d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:56:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.157+0000 7f0688c00700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068415ef70 0x7f06840b2ae0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f0670005910 tx=0x7f067000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.157+0000 7f0681600700 1 -- 172.21.15.89:0/150612492 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670005d70 con 0x7f068415ef70 2024-04-02T07:56:00.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.157+0000 7f0681600700 1 -- 172.21.15.89:0/150612492 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06700070d0 con 0x7f068415ef70 2024-04-02T07:56:00.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.157+0000 7f0681600700 1 -- 172.21.15.89:0/150612492 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670007460 con 0x7f068415ef70 2024-04-02T07:56:00.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.157+0000 7f068c000700 1 -- 172.21.15.89:0/150612492 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0684147e80 con 0x7f068415ef70 2024-04-02T07:56:00.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.157+0000 7f068c000700 1 -- 172.21.15.89:0/150612492 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0684148440 con 0x7f068415ef70 2024-04-02T07:56:00.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.158+0000 7f068c000700 1 -- 172.21.15.89:0/150612492 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0684067720 con 0x7f068415ef70 2024-04-02T07:56:00.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.159+0000 7f0681600700 1 -- 172.21.15.89:0/150612492 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f06700075c0 con 0x7f068415ef70 2024-04-02T07:56:00.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.160+0000 7f0681600700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f066406ae70 0x7f066406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:00.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.160+0000 7f0681600700 1 -- 172.21.15.89:0/150612492 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f0670088d80 con 0x7f068415ef70 2024-04-02T07:56:00.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.160+0000 7f0689600700 1 -- 172.21.15.89:0/150612492 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f066406ae70 msgr2=0x7f066406d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:00.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.160+0000 7f0689600700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f066406ae70 0x7f066406d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:56:00.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.170+0000 7f0681600700 1 -- 172.21.15.89:0/150612492 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0670053970 con 0x7f068415ef70 2024-04-02T07:56:00.281 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:00.281+0000 7f3e94351240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:56:00.283 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:00.282+0000 7f231fc35240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:56:00.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.361+0000 7f0689600700 1 -- 172.21.15.89:0/150612492 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f066406ae70 msgr2=0x7f066406d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:00.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.361+0000 7f0689600700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f066406ae70 0x7f066406d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:56:00.375 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:00.375+0000 7f231fc35240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:56:00.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.376+0000 7f068c000700 1 -- 172.21.15.89:0/150612492 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f06840b3500 con 0x7f068415ef70 2024-04-02T07:56:00.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.377+0000 7f0681600700 1 -- 172.21.15.89:0/150612492 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7f06700170a0 con 0x7f068415ef70 2024-04-02T07:56:00.377 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:56:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.379+0000 7f068c000700 1 -- 172.21.15.89:0/150612492 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f066406ae70 msgr2=0x7f066406d330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:56:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.379+0000 7f068c000700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f066406ae70 0x7f066406d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.379+0000 7f068c000700 1 -- 172.21.15.89:0/150612492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068415ef70 msgr2=0x7f06840b2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.379+0000 7f068c000700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068415ef70 0x7f06840b2ae0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f0670005910 tx=0x7f067000c9e0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.379+0000 7f068c000700 1 -- 172.21.15.89:0/150612492 shutdown_connections 2024-04-02T07:56:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.379+0000 7f068c000700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f066406ae70 0x7f066406d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.379+0000 7f068c000700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f068406a790 0x7f0684146d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.379+0000 7f068c000700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0684160d70 0x7f0684148960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.379+0000 7f068c000700 1 --2- 172.21.15.89:0/150612492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068415ef70 0x7f06840b2ae0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.379+0000 7f068c000700 1 -- 172.21.15.89:0/150612492 >> 172.21.15.89:0/150612492 conn(0x7f06840b5720 msgr2=0x7f068409b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:00.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.379+0000 7f068c000700 1 -- 172.21.15.89:0/150612492 shutdown_connections 2024-04-02T07:56:00.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.379+0000 7f068c000700 1 -- 172.21.15.89:0/150612492 wait complete. 2024-04-02T07:56:00.397 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T07:56:00.489 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:00.488+0000 7f3e94351240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:56:00.587 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:00.586+0000 7f3e94351240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:56:00.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.593+0000 7f5b94800700 1 Processor -- start 2024-04-02T07:56:00.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.594+0000 7f5b94800700 1 -- start start 2024-04-02T07:56:00.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.595+0000 7f5b94800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b8c15d9d0 0x7f5b8c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:00.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.595+0000 7f5b94800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b8c0d16a0 0x7f5b8c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:00.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.595+0000 7f5b94800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8c161420 con 0x7f5b8c0d2370 2024-04-02T07:56:00.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.595+0000 7f5b94800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b8c0d2820 con 0x7f5b8c0d16a0 2024-04-02T07:56:00.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.595+0000 7f5b94800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c161700 con 0x7f5b8c15d9d0 2024-04-02T07:56:00.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.595+0000 7f5b91400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b8c0d16a0 0x7f5b8c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:00.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.595+0000 7f5b91400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b8c0d16a0 0x7f5b8c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54162/0 (socket says 172.21.15.89:54162) 2024-04-02T07:56:00.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.595+0000 7f5b91400700 1 -- 172.21.15.89:0/3206686471 learned_addr learned my addr 172.21.15.89:0/3206686471 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:00.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.595+0000 7f5b91e00700 1 --2- 172.21.15.89:0/3206686471 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b8c15d9d0 0x7f5b8c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:00.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.596+0000 7f5b91400700 1 -- 172.21.15.89:0/3206686471 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b8c15d9d0 msgr2=0x7f5b8c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:00.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.596+0000 7f5b91400700 1 --2- 172.21.15.89:0/3206686471 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b8c15d9d0 0x7f5b8c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.596+0000 7f5b91400700 1 -- 172.21.15.89:0/3206686471 >> v1:172.21.15.113:6789/0 conn(0x7f5b8c0d2370 legacy=0x7f5b8c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:00.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.595+0000 7f5b90a00700 1 -- 172.21.15.89:0/3206686471 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746960303 0 0) 0x7f5b8c161420 con 0x7f5b8c0d2370 2024-04-02T07:56:00.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.596+0000 7f5b91400700 1 -- 172.21.15.89:0/3206686471 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8c1619e0 con 0x7f5b8c0d16a0 2024-04-02T07:56:00.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.596+0000 7f5b91400700 1 --2- 172.21.15.89:0/3206686471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b8c0d16a0 0x7f5b8c0d1e30 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c006fd0 tx=0x7f5b7c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ca073fc21d3926 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.596+0000 7f5b90a00700 1 -- 172.21.15.89:0/3206686471 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c005e30 con 0x7f5b8c0d16a0 2024-04-02T07:56:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.596+0000 7f5b90a00700 1 -- 172.21.15.89:0/3206686471 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c004030 con 0x7f5b8c0d16a0 2024-04-02T07:56:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.596+0000 7f5b90a00700 1 -- 172.21.15.89:0/3206686471 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c0043e0 con 0x7f5b8c0d16a0 2024-04-02T07:56:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.597+0000 7f5b94800700 1 -- 172.21.15.89:0/3206686471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b8c0d16a0 msgr2=0x7f5b8c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.597+0000 7f5b94800700 1 --2- 172.21.15.89:0/3206686471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b8c0d16a0 0x7f5b8c0d1e30 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c006fd0 tx=0x7f5b7c009bd0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.597+0000 7f5b94800700 1 -- 172.21.15.89:0/3206686471 shutdown_connections 2024-04-02T07:56:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.597+0000 7f5b94800700 1 --2- 172.21.15.89:0/3206686471 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b8c15d9d0 0x7f5b8c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.597+0000 7f5b94800700 1 --2- 172.21.15.89:0/3206686471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b8c0d16a0 0x7f5b8c0d1e30 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.597+0000 7f5b94800700 1 -- 172.21.15.89:0/3206686471 >> 172.21.15.89:0/3206686471 conn(0x7f5b8c08aff0 msgr2=0x7f5b8c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:00.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.597+0000 7f5b94800700 1 -- 172.21.15.89:0/3206686471 shutdown_connections 2024-04-02T07:56:00.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.597+0000 7f5b94800700 1 -- 172.21.15.89:0/3206686471 wait complete. 2024-04-02T07:56:00.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.599+0000 7f5b94800700 1 Processor -- start 2024-04-02T07:56:00.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.599+0000 7f5b94800700 1 -- start start 2024-04-02T07:56:00.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b94800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b8c0d16a0 0x7f5b8c158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:00.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b94800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b8c0d2370 0x7f5b8c0b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:00.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b94800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b8c15d9d0 0x7f5b8c0b7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:00.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b94800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c15ee10 con 0x7f5b8c15d9d0 2024-04-02T07:56:00.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b94800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c0b9050 con 0x7f5b8c0d2370 2024-04-02T07:56:00.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b94800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c15e190 con 0x7f5b8c0d16a0 2024-04-02T07:56:00.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b92800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b8c15d9d0 0x7f5b8c0b7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:00.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b91e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b8c0d16a0 0x7f5b8c158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:00.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b91400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b8c0d2370 0x7f5b8c0b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:00.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b91e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b8c0d16a0 0x7f5b8c158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:49686/0 (socket says 172.21.15.89:49686) 2024-04-02T07:56:00.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b92800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b8c15d9d0 0x7f5b8c0b7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54176/0 (socket says 172.21.15.89:54176) 2024-04-02T07:56:00.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b91e00700 1 -- 172.21.15.89:0/3309720017 learned_addr learned my addr 172.21.15.89:0/3309720017 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:00.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b92800700 1 -- 172.21.15.89:0/3309720017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b8c0d16a0 msgr2=0x7f5b8c158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:00.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b92800700 1 --2- 172.21.15.89:0/3309720017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b8c0d16a0 0x7f5b8c158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b92800700 1 -- 172.21.15.89:0/3309720017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b8c0d2370 msgr2=0x7f5b8c0b7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:00.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b92800700 1 --2- 172.21.15.89:0/3309720017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b8c0d2370 0x7f5b8c0b7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b92800700 1 -- 172.21.15.89:0/3309720017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c009500 con 0x7f5b8c15d9d0 2024-04-02T07:56:00.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b91e00700 1 --2- 172.21.15.89:0/3309720017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b8c0d16a0 0x7f5b8c158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:56:00.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.600+0000 7f5b91400700 1 --2- 172.21.15.89:0/3309720017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b8c0d2370 0x7f5b8c0b7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:56:00.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.601+0000 7f5b92800700 1 --2- 172.21.15.89:0/3309720017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b8c15d9d0 0x7f5b8c0b7840 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f5b8800c730 tx=0x7f5b8800ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:00.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.601+0000 7f5b86000700 1 -- 172.21.15.89:0/3309720017 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b88003e30 con 0x7f5b8c15d9d0 2024-04-02T07:56:00.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.601+0000 7f5b86000700 1 -- 172.21.15.89:0/3309720017 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b88005070 con 0x7f5b8c15d9d0 2024-04-02T07:56:00.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.601+0000 7f5b94800700 1 -- 172.21.15.89:0/3309720017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b8c15e470 con 0x7f5b8c15d9d0 2024-04-02T07:56:00.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.601+0000 7f5b86000700 1 -- 172.21.15.89:0/3309720017 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b88005420 con 0x7f5b8c15d9d0 2024-04-02T07:56:00.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.601+0000 7f5b94800700 1 -- 172.21.15.89:0/3309720017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8c15ea30 con 0x7f5b8c15d9d0 2024-04-02T07:56:00.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.602+0000 7f5b94800700 1 -- 172.21.15.89:0/3309720017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8c0d1ca0 con 0x7f5b8c15d9d0 2024-04-02T07:56:00.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.604+0000 7f5b86000700 1 -- 172.21.15.89:0/3309720017 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f5b88005580 con 0x7f5b8c15d9d0 2024-04-02T07:56:00.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.605+0000 7f5b86000700 1 --2- 172.21.15.89:0/3309720017 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5b7006ae40 0x7f5b7006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:00.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.605+0000 7f5b86000700 1 -- 172.21.15.89:0/3309720017 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f5b88089410 con 0x7f5b8c15d9d0 2024-04-02T07:56:00.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.605+0000 7f5b91e00700 1 -- 172.21.15.89:0/3309720017 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5b7006ae40 msgr2=0x7f5b7006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:00.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.605+0000 7f5b91e00700 1 --2- 172.21.15.89:0/3309720017 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5b7006ae40 0x7f5b7006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:56:00.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.609+0000 7f5b86000700 1 -- 172.21.15.89:0/3309720017 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5b880540a0 con 0x7f5b8c15d9d0 2024-04-02T07:56:00.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.806+0000 7f5b91e00700 1 -- 172.21.15.89:0/3309720017 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5b7006ae40 msgr2=0x7f5b7006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:00.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.806+0000 7f5b91e00700 1 --2- 172.21.15.89:0/3309720017 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5b7006ae40 0x7f5b7006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:56:00.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.814+0000 7f5b94800700 1 -- 172.21.15.89:0/3309720017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5b8c067720 con 0x7f5b8c15d9d0 2024-04-02T07:56:00.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.815+0000 7f5b86000700 1 -- 172.21.15.89:0/3309720017 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v141) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5b880576c0 con 0x7f5b8c15d9d0 2024-04-02T07:56:00.815 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T07:56:00.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.817+0000 7f5b94800700 1 -- 172.21.15.89:0/3309720017 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5b7006ae40 msgr2=0x7f5b7006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:56:00.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.817+0000 7f5b94800700 1 --2- 172.21.15.89:0/3309720017 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5b7006ae40 0x7f5b7006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.817+0000 7f5b94800700 1 -- 172.21.15.89:0/3309720017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b8c15d9d0 msgr2=0x7f5b8c0b7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:00.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.817+0000 7f5b94800700 1 --2- 172.21.15.89:0/3309720017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b8c15d9d0 0x7f5b8c0b7840 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f5b8800c730 tx=0x7f5b8800ca40 comp rx=0 tx=0).stop 2024-04-02T07:56:00.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.817+0000 7f5b94800700 1 -- 172.21.15.89:0/3309720017 shutdown_connections 2024-04-02T07:56:00.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.817+0000 7f5b94800700 1 --2- 172.21.15.89:0/3309720017 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5b7006ae40 0x7f5b7006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.817+0000 7f5b94800700 1 --2- 172.21.15.89:0/3309720017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b8c15d9d0 0x7f5b8c0b7840 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.817+0000 7f5b94800700 1 --2- 172.21.15.89:0/3309720017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b8c0d16a0 0x7f5b8c158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.817+0000 7f5b94800700 1 --2- 172.21.15.89:0/3309720017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b8c0d2370 0x7f5b8c0b7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:00.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.817+0000 7f5b94800700 1 -- 172.21.15.89:0/3309720017 >> 172.21.15.89:0/3309720017 conn(0x7f5b8c08aff0 msgr2=0x7f5b8c0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:00.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.818+0000 7f5b94800700 1 -- 172.21.15.89:0/3309720017 shutdown_connections 2024-04-02T07:56:00.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:00.818+0000 7f5b94800700 1 -- 172.21.15.89:0/3309720017 wait complete. 2024-04-02T07:56:00.837 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T07:56:01.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.033+0000 7f5383200700 1 Processor -- start 2024-04-02T07:56:01.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.034+0000 7f5383200700 1 -- start start 2024-04-02T07:56:01.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.035+0000 7f5383200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f53840b8fe0 0x7f53840b6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.035+0000 7f5383200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53840b6870 0x7f53840d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.035+0000 7f5383200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53840d1aa0 con 0x7f538415d230 2024-04-02T07:56:01.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.035+0000 7f5383200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53840b6cf0 con 0x7f53840b8fe0 2024-04-02T07:56:01.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.035+0000 7f5383200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53840b6fd0 con 0x7f53840b6870 2024-04-02T07:56:01.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.035+0000 7f5381400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f53840b8fe0 0x7f53840b6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:01.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.035+0000 7f5381400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f53840b8fe0 0x7f53840b6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54190/0 (socket says 172.21.15.89:54190) 2024-04-02T07:56:01.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.035+0000 7f5381400700 1 -- 172.21.15.89:0/2837336137 learned_addr learned my addr 172.21.15.89:0/2837336137 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:01.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.035+0000 7f5381e00700 1 --1- 172.21.15.89:0/2837336137 >> v1:172.21.15.89:6789/0 conn(0x7f538415d230 0x7f53840d1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55534/0 (socket says 172.21.15.89:55534) 2024-04-02T07:56:01.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.035+0000 7f5382800700 1 --2- 172.21.15.89:0/2837336137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53840b6870 0x7f53840d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:01.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.036+0000 7f5381400700 1 -- 172.21.15.89:0/2837336137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53840b6870 msgr2=0x7f53840d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:01.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.036+0000 7f5381400700 1 --2- 172.21.15.89:0/2837336137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53840b6870 0x7f53840d02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.036+0000 7f5381400700 1 -- 172.21.15.89:0/2837336137 >> v1:172.21.15.89:6789/0 conn(0x7f538415d230 legacy=0x7f53840d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.036+0000 7f5380a00700 1 -- 172.21.15.89:0/2837336137 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314300734 0 0) 0x7f53840d1aa0 con 0x7f538415d230 2024-04-02T07:56:01.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.036+0000 7f5381400700 1 -- 172.21.15.89:0/2837336137 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53840b72b0 con 0x7f53840b8fe0 2024-04-02T07:56:01.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.036+0000 7f5381400700 1 --2- 172.21.15.89:0/2837336137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53840b8fe0 0x7f53840b6330 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f5370009920 tx=0x7f5370009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1a1af1fb55fe6e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:01.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.036+0000 7f5380a00700 1 -- 172.21.15.89:0/2837336137 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5370011830 con 0x7f53840b8fe0 2024-04-02T07:56:01.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.036+0000 7f5380a00700 1 -- 172.21.15.89:0/2837336137 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5370011990 con 0x7f53840b8fe0 2024-04-02T07:56:01.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.036+0000 7f5380a00700 1 -- 172.21.15.89:0/2837336137 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5370011d20 con 0x7f53840b8fe0 2024-04-02T07:56:01.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.037+0000 7f5383200700 1 -- 172.21.15.89:0/2837336137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53840b8fe0 msgr2=0x7f53840b6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.037+0000 7f5383200700 1 --2- 172.21.15.89:0/2837336137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53840b8fe0 0x7f53840b6330 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f5370009920 tx=0x7f5370009c30 comp rx=0 tx=0).stop 2024-04-02T07:56:01.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.037+0000 7f5383200700 1 -- 172.21.15.89:0/2837336137 shutdown_connections 2024-04-02T07:56:01.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.037+0000 7f5383200700 1 --2- 172.21.15.89:0/2837336137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53840b6870 0x7f53840d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.037+0000 7f5383200700 1 --2- 172.21.15.89:0/2837336137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53840b8fe0 0x7f53840b6330 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.037+0000 7f5383200700 1 -- 172.21.15.89:0/2837336137 >> 172.21.15.89:0/2837336137 conn(0x7f53840668e0 msgr2=0x7f5384066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:01.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.037+0000 7f5383200700 1 -- 172.21.15.89:0/2837336137 shutdown_connections 2024-04-02T07:56:01.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.037+0000 7f5383200700 1 -- 172.21.15.89:0/2837336137 wait complete. 2024-04-02T07:56:01.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.039+0000 7f5383200700 1 Processor -- start 2024-04-02T07:56:01.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.039+0000 7f5383200700 1 -- start start 2024-04-02T07:56:01.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5383200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53840b6870 0x7f53840ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5383200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53840b8fe0 0x7f53840cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5383200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f538415d230 0x7f538414e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5383200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53840cb630 con 0x7f53840b6870 2024-04-02T07:56:01.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5383200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53840cc1d0 con 0x7f53840b8fe0 2024-04-02T07:56:01.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5383200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53840caa20 con 0x7f538415d230 2024-04-02T07:56:01.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5381e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53840b6870 0x7f53840ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:01.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5381e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53840b6870 0x7f53840ca440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54204/0 (socket says 172.21.15.89:54204) 2024-04-02T07:56:01.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5381e00700 1 -- 172.21.15.89:0/992750122 learned_addr learned my addr 172.21.15.89:0/992750122 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:01.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5382800700 1 --2- 172.21.15.89:0/992750122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f538415d230 0x7f538414e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:01.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5381400700 1 --2- 172.21.15.89:0/992750122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53840b8fe0 0x7f53840cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:01.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5381e00700 1 -- 172.21.15.89:0/992750122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f538415d230 msgr2=0x7f538414e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5381e00700 1 --2- 172.21.15.89:0/992750122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f538415d230 0x7f538414e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5381e00700 1 -- 172.21.15.89:0/992750122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53840b8fe0 msgr2=0x7f53840cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5381e00700 1 --2- 172.21.15.89:0/992750122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53840b8fe0 0x7f53840cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.040+0000 7f5381e00700 1 -- 172.21.15.89:0/992750122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5370009550 con 0x7f53840b6870 2024-04-02T07:56:01.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.041+0000 7f5381e00700 1 --2- 172.21.15.89:0/992750122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53840b6870 0x7f53840ca440 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f537400c550 tx=0x7f537400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:01.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.041+0000 7f537a000700 1 -- 172.21.15.89:0/992750122 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374004c80 con 0x7f53840b6870 2024-04-02T07:56:01.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.041+0000 7f537a000700 1 -- 172.21.15.89:0/992750122 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5374004e00 con 0x7f53840b6870 2024-04-02T07:56:01.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.041+0000 7f537a000700 1 -- 172.21.15.89:0/992750122 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374007310 con 0x7f53840b6870 2024-04-02T07:56:01.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.041+0000 7f5383200700 1 -- 172.21.15.89:0/992750122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53840cad00 con 0x7f53840b6870 2024-04-02T07:56:01.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.041+0000 7f5383200700 1 -- 172.21.15.89:0/992750122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53840cb2c0 con 0x7f53840b6870 2024-04-02T07:56:01.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.043+0000 7f5383200700 1 -- 172.21.15.89:0/992750122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538404ea80 con 0x7f53840b6870 2024-04-02T07:56:01.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.043+0000 7f537a000700 1 -- 172.21.15.89:0/992750122 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f5374007470 con 0x7f53840b6870 2024-04-02T07:56:01.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.044+0000 7f537a000700 1 --2- 172.21.15.89:0/992750122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f536406ae40 0x7f536406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.044+0000 7f537a000700 1 -- 172.21.15.89:0/992750122 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f5374089560 con 0x7f53840b6870 2024-04-02T07:56:01.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.044+0000 7f5381400700 1 -- 172.21.15.89:0/992750122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f536406ae40 msgr2=0x7f536406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:01.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.044+0000 7f5381400700 1 --2- 172.21.15.89:0/992750122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f536406ae40 0x7f536406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:56:01.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.048+0000 7f537a000700 1 -- 172.21.15.89:0/992750122 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5374054160 con 0x7f53840b6870 2024-04-02T07:56:01.109 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:01.108+0000 7f231fc35240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:56:01.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.245+0000 7f5381400700 1 -- 172.21.15.89:0/992750122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f536406ae40 msgr2=0x7f536406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:01.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.245+0000 7f5381400700 1 --2- 172.21.15.89:0/992750122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f536406ae40 0x7f536406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:56:01.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.272+0000 7f5383200700 1 -- 172.21.15.89:0/992750122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f53840b3780 con 0x7f53840b6870 2024-04-02T07:56:01.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.272+0000 7f537a000700 1 -- 172.21.15.89:0/992750122 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v141) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5374057780 con 0x7f53840b6870 2024-04-02T07:56:01.273 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T07:56:01.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.275+0000 7f5383200700 1 -- 172.21.15.89:0/992750122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f536406ae40 msgr2=0x7f536406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:56:01.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.275+0000 7f5383200700 1 --2- 172.21.15.89:0/992750122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f536406ae40 0x7f536406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.275+0000 7f5383200700 1 -- 172.21.15.89:0/992750122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53840b6870 msgr2=0x7f53840ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.275+0000 7f5383200700 1 --2- 172.21.15.89:0/992750122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53840b6870 0x7f53840ca440 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f537400c550 tx=0x7f537400c860 comp rx=0 tx=0).stop 2024-04-02T07:56:01.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.275+0000 7f5383200700 1 -- 172.21.15.89:0/992750122 shutdown_connections 2024-04-02T07:56:01.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.275+0000 7f5383200700 1 --2- 172.21.15.89:0/992750122 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f536406ae40 0x7f536406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.275+0000 7f5383200700 1 --2- 172.21.15.89:0/992750122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f538415d230 0x7f538414e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.275+0000 7f5383200700 1 --2- 172.21.15.89:0/992750122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53840b6870 0x7f53840ca440 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.275+0000 7f5383200700 1 --2- 172.21.15.89:0/992750122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53840b8fe0 0x7f53840cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.275+0000 7f5383200700 1 -- 172.21.15.89:0/992750122 >> 172.21.15.89:0/992750122 conn(0x7f53840668e0 msgr2=0x7f53840d02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:01.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.275+0000 7f5383200700 1 -- 172.21.15.89:0/992750122 shutdown_connections 2024-04-02T07:56:01.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.275+0000 7f5383200700 1 -- 172.21.15.89:0/992750122 wait complete. 2024-04-02T07:56:01.295 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:56:01.322 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:01.321+0000 7f231fc35240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:56:01.369 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:01.369+0000 7f3e94351240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:56:01.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.489+0000 7fba8da00700 1 Processor -- start 2024-04-02T07:56:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.489+0000 7fba8da00700 1 -- start start 2024-04-02T07:56:01.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.491+0000 7fba8da00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fba8815d670 0x7fba880ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.491+0000 7fba8da00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba880b3890 con 0x7fba8806dee0 2024-04-02T07:56:01.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.491+0000 7fba8da00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba880acac0 con 0x7fba8815da50 2024-04-02T07:56:01.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.491+0000 7fba8da00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba88159f50 con 0x7fba8815d670 2024-04-02T07:56:01.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.492+0000 7fba87400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fba8806dee0 0x7fba8806e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55542/0 (socket says 172.21.15.89:55542) 2024-04-02T07:56:01.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.492+0000 7fba87400700 1 -- 172.21.15.89:0/359322749 learned_addr learned my addr 172.21.15.89:0/359322749 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:01.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.492+0000 7fba86a00700 1 --2- 172.21.15.89:0/359322749 >> v2:172.21.15.113:3300/0 conn(0x7fba8815d670 0x7fba880ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.492+0000 7fba85600700 1 -- 172.21.15.89:0/359322749 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163136757 0 0) 0x7fba880b3890 con 0x7fba8806dee0 2024-04-02T07:56:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.492+0000 7fba85600700 1 -- 172.21.15.89:0/359322749 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba6c0034d0 con 0x7fba8806dee0 2024-04-02T07:56:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.492+0000 7fba86a00700 1 -- 172.21.15.89:0/359322749 >> v1:172.21.15.113:6789/0 conn(0x7fba8815da50 legacy=0x7fba880b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.492+0000 7fba85600700 1 -- 172.21.15.89:0/359322749 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174918877 0 0) 0x7fba880acac0 con 0x7fba8815da50 2024-04-02T07:56:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.492+0000 7fba86a00700 1 -- 172.21.15.89:0/359322749 >> v1:172.21.15.89:6789/0 conn(0x7fba8806dee0 legacy=0x7fba8806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.492+0000 7fba86a00700 1 -- 172.21.15.89:0/359322749 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba880b35b0 con 0x7fba8815d670 2024-04-02T07:56:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.493+0000 7fba86a00700 1 --2- 172.21.15.89:0/359322749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba8815d670 0x7fba880ac580 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fba78009840 tx=0x7fba78009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9d5f262b2e3856e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.493+0000 7fba85600700 1 -- 172.21.15.89:0/359322749 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78011830 con 0x7fba8815d670 2024-04-02T07:56:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.493+0000 7fba85600700 1 -- 172.21.15.89:0/359322749 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba78011990 con 0x7fba8815d670 2024-04-02T07:56:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.493+0000 7fba85600700 1 -- 172.21.15.89:0/359322749 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78011d20 con 0x7fba8815d670 2024-04-02T07:56:01.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.494+0000 7fba8da00700 1 -- 172.21.15.89:0/359322749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba8815d670 msgr2=0x7fba880ac580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.494+0000 7fba8da00700 1 --2- 172.21.15.89:0/359322749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba8815d670 0x7fba880ac580 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fba78009840 tx=0x7fba78009b50 comp rx=0 tx=0).stop 2024-04-02T07:56:01.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.494+0000 7fba8da00700 1 -- 172.21.15.89:0/359322749 shutdown_connections 2024-04-02T07:56:01.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.494+0000 7fba8da00700 1 --2- 172.21.15.89:0/359322749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba8815d670 0x7fba880ac580 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.494+0000 7fba8da00700 1 -- 172.21.15.89:0/359322749 >> 172.21.15.89:0/359322749 conn(0x7fba880cf190 msgr2=0x7fba880cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:01.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.494+0000 7fba8da00700 1 -- 172.21.15.89:0/359322749 shutdown_connections 2024-04-02T07:56:01.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.494+0000 7fba8da00700 1 -- 172.21.15.89:0/359322749 wait complete. 2024-04-02T07:56:01.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.497+0000 7fba8da00700 1 Processor -- start 2024-04-02T07:56:01.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.497+0000 7fba8da00700 1 -- start start 2024-04-02T07:56:01.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba8da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8806dee0 0x7fba880cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba8da00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8815d670 0x7fba880b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba8da00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba8815da50 0x7fba88153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba8da00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba88151ca0 con 0x7fba8806dee0 2024-04-02T07:56:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba8da00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba88099a50 con 0x7fba8815da50 2024-04-02T07:56:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba8da00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba880b2de0 con 0x7fba8815d670 2024-04-02T07:56:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba86a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8806dee0 0x7fba880cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba86a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8806dee0 0x7fba880cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54214/0 (socket says 172.21.15.89:54214) 2024-04-02T07:56:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba86a00700 1 -- 172.21.15.89:0/1417854831 learned_addr learned my addr 172.21.15.89:0/1417854831 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba87400700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba8815da50 0x7fba88153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba86000700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8815d670 0x7fba880b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba86a00700 1 -- 172.21.15.89:0/1417854831 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8815d670 msgr2=0x7fba880b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba86a00700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8815d670 0x7fba880b28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba86a00700 1 -- 172.21.15.89:0/1417854831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba8815da50 msgr2=0x7fba88153960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba86a00700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba8815da50 0x7fba88153960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba86a00700 1 -- 172.21.15.89:0/1417854831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba780094a0 con 0x7fba8806dee0 2024-04-02T07:56:01.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.499+0000 7fba86000700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8815d670 0x7fba880b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:01.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.500+0000 7fba86a00700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8806dee0 0x7fba880cde30 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fba78009fd0 tx=0x7fba78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:01.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.500+0000 7fba7ea00700 1 -- 172.21.15.89:0/1417854831 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7801aa70 con 0x7fba8806dee0 2024-04-02T07:56:01.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.500+0000 7fba7ea00700 1 -- 172.21.15.89:0/1417854831 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7801abd0 con 0x7fba8806dee0 2024-04-02T07:56:01.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.500+0000 7fba8da00700 1 -- 172.21.15.89:0/1417854831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba88151ef0 con 0x7fba8806dee0 2024-04-02T07:56:01.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.500+0000 7fba7ea00700 1 -- 172.21.15.89:0/1417854831 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba780243f0 con 0x7fba8806dee0 2024-04-02T07:56:01.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.500+0000 7fba8da00700 1 -- 172.21.15.89:0/1417854831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba881524b0 con 0x7fba8806dee0 2024-04-02T07:56:01.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.501+0000 7fba8da00700 1 -- 172.21.15.89:0/1417854831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba88067720 con 0x7fba8806dee0 2024-04-02T07:56:01.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.502+0000 7fba7ea00700 1 -- 172.21.15.89:0/1417854831 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fba78024550 con 0x7fba8806dee0 2024-04-02T07:56:01.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.503+0000 7fba7ea00700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fba6806ae40 0x7fba6806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.503+0000 7fba7ea00700 1 -- 172.21.15.89:0/1417854831 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fba7808ad40 con 0x7fba8806dee0 2024-04-02T07:56:01.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.503+0000 7fba86000700 1 -- 172.21.15.89:0/1417854831 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fba6806ae40 msgr2=0x7fba6806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:01.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.503+0000 7fba86000700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fba6806ae40 0x7fba6806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:56:01.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.507+0000 7fba7ea00700 1 -- 172.21.15.89:0/1417854831 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fba78055b80 con 0x7fba8806dee0 2024-04-02T07:56:01.528 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:01.527+0000 7f231fc35240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:56:01.613 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:01.613+0000 7f3e94351240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:56:01.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.704+0000 7fba86000700 1 -- 172.21.15.89:0/1417854831 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fba6806ae40 msgr2=0x7fba6806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:01.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.704+0000 7fba86000700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fba6806ae40 0x7fba6806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:56:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.774+0000 7fba8da00700 1 -- 172.21.15.89:0/1417854831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba880acef0 con 0x7fba8806dee0 2024-04-02T07:56:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.775+0000 7fba7ea00700 1 -- 172.21.15.89:0/1417854831 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4733 (secure 0 0 0) 0x7fba78015070 con 0x7fba8806dee0 2024-04-02T07:56:01.776 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:01.776 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/66826967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":66826967},{"type":"v1","addr":"172.21.15.113:6835","nonce":66826967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":7915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/1669765650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1669765650},{"type":"v1","addr":"172.21.15.113:6833","nonce":1669765650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":7918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3321722319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3321722319},{"type":"v1","addr":"172.21.15.89:6837","nonce":3321722319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:55:31.571731+0000","modified":"2024-04-02T07:55:34.583055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8436":{"gid":8436,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":4,"addr":"172.21.15.89:6836/417215494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":417215494},{"type":"v1","addr":"172.21.15.89:6836","nonce":417215494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-02T07:56:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.777+0000 7fba8da00700 1 -- 172.21.15.89:0/1417854831 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fba6806ae40 msgr2=0x7fba6806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:56:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.777+0000 7fba8da00700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fba6806ae40 0x7fba6806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.777+0000 7fba8da00700 1 -- 172.21.15.89:0/1417854831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8806dee0 msgr2=0x7fba880cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.777+0000 7fba8da00700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8806dee0 0x7fba880cde30 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fba78009fd0 tx=0x7fba78004750 comp rx=0 tx=0).stop 2024-04-02T07:56:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.778+0000 7fba8da00700 1 -- 172.21.15.89:0/1417854831 shutdown_connections 2024-04-02T07:56:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.778+0000 7fba8da00700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fba6806ae40 0x7fba6806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.778+0000 7fba8da00700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba8815da50 0x7fba88153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.778+0000 7fba8da00700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8806dee0 0x7fba880cde30 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.778+0000 7fba8da00700 1 --2- 172.21.15.89:0/1417854831 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8815d670 0x7fba880b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.778+0000 7fba8da00700 1 -- 172.21.15.89:0/1417854831 >> 172.21.15.89:0/1417854831 conn(0x7fba880cf190 msgr2=0x7fba880aec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.778+0000 7fba8da00700 1 -- 172.21.15.89:0/1417854831 shutdown_connections 2024-04-02T07:56:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.778+0000 7fba8da00700 1 -- 172.21.15.89:0/1417854831 wait complete. 2024-04-02T07:56:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 66 2024-04-02T07:56:01.796 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:56:01.830 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:01.829+0000 7f3e94351240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:56:01.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.990+0000 7fc4c4000700 1 Processor -- start 2024-04-02T07:56:01.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.991+0000 7fc4c4000700 1 -- start start 2024-04-02T07:56:01.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c4000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4bc15d640 0x7fc4bc0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c4000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4bc0ceec0 0x7fc4bc1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c4000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc0d30d0 con 0x7fc4bc161800 2024-04-02T07:56:01.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c4000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4bc161c90 con 0x7fc4bc0ceec0 2024-04-02T07:56:01.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c4000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0b6840 con 0x7fc4bc15d640 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c0c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4bc0ceec0 0x7fc4bc1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c0c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4bc0ceec0 0x7fc4bc1612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54218/0 (socket says 172.21.15.89:54218) 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c2000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc4bc161800 0x7fc4bc0cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55544/0 (socket says 172.21.15.89:55544) 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c0c00700 1 -- 172.21.15.89:0/2483925229 learned_addr learned my addr 172.21.15.89:0/2483925229 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c1600700 1 --2- 172.21.15.89:0/2483925229 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4bc15d640 0x7fc4bc0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c0c00700 1 -- 172.21.15.89:0/2483925229 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4bc15d640 msgr2=0x7fc4bc0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c0c00700 1 --2- 172.21.15.89:0/2483925229 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4bc15d640 0x7fc4bc0d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c0c00700 1 -- 172.21.15.89:0/2483925229 >> v1:172.21.15.89:6789/0 conn(0x7fc4bc161800 legacy=0x7fc4bc0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c0c00700 1 -- 172.21.15.89:0/2483925229 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc0d2880 con 0x7fc4bc0ceec0 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.992+0000 7fc4c1600700 1 --2- 172.21.15.89:0/2483925229 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4bc15d640 0x7fc4bc0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.993+0000 7fc4c0c00700 1 --2- 172.21.15.89:0/2483925229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0ceec0 0x7fc4bc1612c0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80098f0 tx=0x7fc4a8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e55d33ef7e884f23 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.993+0000 7fc4bbe00700 1 -- 172.21.15.89:0/2483925229 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a800b8a0 con 0x7fc4bc0ceec0 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.993+0000 7fc4bbe00700 1 -- 172.21.15.89:0/2483925229 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a800ba00 con 0x7fc4bc0ceec0 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.993+0000 7fc4bbe00700 1 -- 172.21.15.89:0/2483925229 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a800bd90 con 0x7fc4bc0ceec0 2024-04-02T07:56:01.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.994+0000 7fc4c4000700 1 -- 172.21.15.89:0/2483925229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0ceec0 msgr2=0x7fc4bc1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.994+0000 7fc4c4000700 1 --2- 172.21.15.89:0/2483925229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0ceec0 0x7fc4bc1612c0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80098f0 tx=0x7fc4a8009c00 comp rx=0 tx=0).stop 2024-04-02T07:56:01.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.994+0000 7fc4c4000700 1 -- 172.21.15.89:0/2483925229 shutdown_connections 2024-04-02T07:56:01.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.994+0000 7fc4c4000700 1 --2- 172.21.15.89:0/2483925229 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4bc15d640 0x7fc4bc0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.994+0000 7fc4c4000700 1 --2- 172.21.15.89:0/2483925229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc0ceec0 0x7fc4bc1612c0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.994+0000 7fc4c4000700 1 -- 172.21.15.89:0/2483925229 >> 172.21.15.89:0/2483925229 conn(0x7fc4bc08aff0 msgr2=0x7fc4bc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:01.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.994+0000 7fc4c4000700 1 -- 172.21.15.89:0/2483925229 shutdown_connections 2024-04-02T07:56:01.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.994+0000 7fc4c4000700 1 -- 172.21.15.89:0/2483925229 wait complete. 2024-04-02T07:56:01.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.995+0000 7fc4c4000700 1 Processor -- start 2024-04-02T07:56:01.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.996+0000 7fc4c4000700 1 -- start start 2024-04-02T07:56:01.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.996+0000 7fc4c4000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4bc0ceec0 0x7fc4bc1f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.996+0000 7fc4c4000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4bc15d640 0x7fc4bc1463d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.996+0000 7fc4c4000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc161800 0x7fc4bc0c8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:01.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.996+0000 7fc4c4000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0c8170 con 0x7fc4bc161800 2024-04-02T07:56:01.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.996+0000 7fc4c4000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc146cb0 con 0x7fc4bc15d640 2024-04-02T07:56:01.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.996+0000 7fc4c4000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc146f90 con 0x7fc4bc0ceec0 2024-04-02T07:56:01.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.997+0000 7fc4c2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc161800 0x7fc4bc0c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:01.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.997+0000 7fc4c1600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4bc0ceec0 0x7fc4bc1f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:01.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.997+0000 7fc4c0c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4bc15d640 0x7fc4bc1463d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:01.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.997+0000 7fc4c2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc161800 0x7fc4bc0c8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54224/0 (socket says 172.21.15.89:54224) 2024-04-02T07:56:01.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.997+0000 7fc4c2000700 1 -- 172.21.15.89:0/3650938516 learned_addr learned my addr 172.21.15.89:0/3650938516 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:01.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.997+0000 7fc4c1600700 1 --2- 172.21.15.89:0/3650938516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4bc0ceec0 0x7fc4bc1f2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:49738/0 (socket says 172.21.15.89:49738) 2024-04-02T07:56:01.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.997+0000 7fc4c2000700 1 -- 172.21.15.89:0/3650938516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4bc0ceec0 msgr2=0x7fc4bc1f2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.997+0000 7fc4c2000700 1 --2- 172.21.15.89:0/3650938516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4bc0ceec0 0x7fc4bc1f2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.997+0000 7fc4c2000700 1 -- 172.21.15.89:0/3650938516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4bc15d640 msgr2=0x7fc4bc1463d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:01.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.997+0000 7fc4c2000700 1 --2- 172.21.15.89:0/3650938516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4bc15d640 0x7fc4bc1463d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:01.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.997+0000 7fc4c2000700 1 -- 172.21.15.89:0/3650938516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a8009550 con 0x7fc4bc161800 2024-04-02T07:56:01.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.997+0000 7fc4c2000700 1 --2- 172.21.15.89:0/3650938516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc161800 0x7fc4bc0c8900 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400cbd0 tx=0x7fc4b400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:02.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.998+0000 7fc4b9600700 1 -- 172.21.15.89:0/3650938516 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b400aaa0 con 0x7fc4bc161800 2024-04-02T07:56:02.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.998+0000 7fc4b9600700 1 -- 172.21.15.89:0/3650938516 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b400ac00 con 0x7fc4bc161800 2024-04-02T07:56:02.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.998+0000 7fc4c4000700 1 -- 172.21.15.89:0/3650938516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4bc0c83e0 con 0x7fc4bc161800 2024-04-02T07:56:02.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.998+0000 7fc4b9600700 1 -- 172.21.15.89:0/3650938516 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4007070 con 0x7fc4bc161800 2024-04-02T07:56:02.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.998+0000 7fc4c4000700 1 -- 172.21.15.89:0/3650938516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc147270 con 0x7fc4bc161800 2024-04-02T07:56:02.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:01.999+0000 7fc4c4000700 1 -- 172.21.15.89:0/3650938516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4bc0c85d0 con 0x7fc4bc161800 2024-04-02T07:56:02.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.000+0000 7fc4b9600700 1 -- 172.21.15.89:0/3650938516 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fc4b40071d0 con 0x7fc4bc161800 2024-04-02T07:56:02.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.001+0000 7fc4b9600700 1 --2- 172.21.15.89:0/3650938516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc49c06ae40 0x7fc49c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:02.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.001+0000 7fc4c1600700 1 -- 172.21.15.89:0/3650938516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc49c06ae40 msgr2=0x7fc49c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:02.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.001+0000 7fc4b9600700 1 -- 172.21.15.89:0/3650938516 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fc4b4089590 con 0x7fc4bc161800 2024-04-02T07:56:02.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.001+0000 7fc4c1600700 1 --2- 172.21.15.89:0/3650938516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc49c06ae40 0x7fc49c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:56:02.004 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:02.003+0000 7f231fc35240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:56:02.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.005+0000 7fc4b9600700 1 -- 172.21.15.89:0/3650938516 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc4b4054230 con 0x7fc4bc161800 2024-04-02T07:56:02.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.202+0000 7fc4c1600700 1 -- 172.21.15.89:0/3650938516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc49c06ae40 msgr2=0x7fc49c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:02.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.202+0000 7fc4c1600700 1 --2- 172.21.15.89:0/3650938516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc49c06ae40 0x7fc49c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:56:02.213 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:02.213+0000 7f231fc35240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:56:02.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.219+0000 7fc4c4000700 1 -- 172.21.15.89:0/3650938516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4bc067720 con 0x7fc4bc161800 2024-04-02T07:56:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.221+0000 7fc4b9600700 1 -- 172.21.15.89:0/3650938516 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+17175 (secure 0 0 0) 0x7fc4b4057850 con 0x7fc4bc161800 2024-04-02T07:56:02.221 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:02.221 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":141,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:55:47.785904+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:55:29.329907+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-02T07:55:30.555350+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/836334016":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/1824838019":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/2769562366":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/224945013":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/732038760":"2024-04-03T07:55:47.785863+0000","172.21.15.89:6833/43357":"2024-04-03T07:55:47.785863+0000","172.21.15.89:6832/43357":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/3727408218":"2024-04-03T07:55:47.785863+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:56:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.223+0000 7fc4c4000700 1 -- 172.21.15.89:0/3650938516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc49c06ae40 msgr2=0x7fc49c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:56:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.223+0000 7fc4c4000700 1 --2- 172.21.15.89:0/3650938516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc49c06ae40 0x7fc49c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.223+0000 7fc4c4000700 1 -- 172.21.15.89:0/3650938516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc161800 msgr2=0x7fc4bc0c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.223+0000 7fc4c4000700 1 --2- 172.21.15.89:0/3650938516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc161800 0x7fc4bc0c8900 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400cbd0 tx=0x7fc4b400cee0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.223+0000 7fc4c4000700 1 -- 172.21.15.89:0/3650938516 shutdown_connections 2024-04-02T07:56:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.223+0000 7fc4c4000700 1 --2- 172.21.15.89:0/3650938516 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc49c06ae40 0x7fc49c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.223+0000 7fc4c4000700 1 --2- 172.21.15.89:0/3650938516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4bc161800 0x7fc4bc0c8900 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.223+0000 7fc4c4000700 1 --2- 172.21.15.89:0/3650938516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4bc0ceec0 0x7fc4bc1f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.223+0000 7fc4c4000700 1 --2- 172.21.15.89:0/3650938516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4bc15d640 0x7fc4bc1463d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.223+0000 7fc4c4000700 1 -- 172.21.15.89:0/3650938516 >> 172.21.15.89:0/3650938516 conn(0x7fc4bc08aff0 msgr2=0x7fc4bc0b7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.223+0000 7fc4c4000700 1 -- 172.21.15.89:0/3650938516 shutdown_connections 2024-04-02T07:56:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.223+0000 7fc4c4000700 1 -- 172.21.15.89:0/3650938516 wait complete. 2024-04-02T07:56:02.242 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T07:56:02.242 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:56:02.347 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:02.346+0000 7f3e94351240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:56:02.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.476+0000 7f0126800700 1 Processor -- start 2024-04-02T07:56:02.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.476+0000 7f0126800700 1 -- start start 2024-04-02T07:56:02.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.477+0000 7f0126800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01200ceec0 0x7f01201612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:02.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.477+0000 7f0126800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01200d30d0 con 0x7f0120161800 2024-04-02T07:56:02.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.477+0000 7f0126800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01200b6840 con 0x7f012015d640 2024-04-02T07:56:02.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.477+0000 7f0126800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0120161c90 con 0x7f01200ceec0 2024-04-02T07:56:02.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.477+0000 7f011fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0120161800 0x7f01200b6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55552/0 (socket says 172.21.15.89:55552) 2024-04-02T07:56:02.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.477+0000 7f011fe00700 1 -- 172.21.15.89:0/331894540 learned_addr learned my addr 172.21.15.89:0/331894540 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:02.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.477+0000 7f011ea00700 1 --2- 172.21.15.89:0/331894540 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01200ceec0 0x7f01201612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:02.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.478+0000 7f011e000700 1 -- 172.21.15.89:0/331894540 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528133556 0 0) 0x7f01200d30d0 con 0x7f0120161800 2024-04-02T07:56:02.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.478+0000 7f011e000700 1 -- 172.21.15.89:0/331894540 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01040035b0 con 0x7f0120161800 2024-04-02T07:56:02.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.478+0000 7f011e000700 1 -- 172.21.15.89:0/331894540 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835304536 0 0) 0x7f01200b6840 con 0x7f012015d640 2024-04-02T07:56:02.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.478+0000 7f011e000700 1 -- 172.21.15.89:0/331894540 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01200d30d0 con 0x7f012015d640 2024-04-02T07:56:02.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.478+0000 7f011ea00700 1 -- 172.21.15.89:0/331894540 >> v1:172.21.15.113:6789/0 conn(0x7f012015d640 legacy=0x7f01200d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:02.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.478+0000 7f011ea00700 1 -- 172.21.15.89:0/331894540 >> v1:172.21.15.89:6789/0 conn(0x7f0120161800 legacy=0x7f01200b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:02.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.478+0000 7f011ea00700 1 -- 172.21.15.89:0/331894540 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01200d2b30 con 0x7f01200ceec0 2024-04-02T07:56:02.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.478+0000 7f011ea00700 1 --2- 172.21.15.89:0/331894540 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01200ceec0 0x7f01201612c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f010c009870 tx=0x7f010c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec463a8d3c46d840 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:02.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.479+0000 7f011e000700 1 -- 172.21.15.89:0/331894540 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c011830 con 0x7f01200ceec0 2024-04-02T07:56:02.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.479+0000 7f011e000700 1 -- 172.21.15.89:0/331894540 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010c011990 con 0x7f01200ceec0 2024-04-02T07:56:02.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.479+0000 7f011e000700 1 -- 172.21.15.89:0/331894540 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c011d20 con 0x7f01200ceec0 2024-04-02T07:56:02.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.479+0000 7f0126800700 1 -- 172.21.15.89:0/331894540 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01200ceec0 msgr2=0x7f01201612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:02.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.479+0000 7f0126800700 1 --2- 172.21.15.89:0/331894540 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01200ceec0 0x7f01201612c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f010c009870 tx=0x7f010c009b80 comp rx=0 tx=0).stop 2024-04-02T07:56:02.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.479+0000 7f0126800700 1 -- 172.21.15.89:0/331894540 shutdown_connections 2024-04-02T07:56:02.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.479+0000 7f0126800700 1 --2- 172.21.15.89:0/331894540 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01200ceec0 0x7f01201612c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.479+0000 7f0126800700 1 -- 172.21.15.89:0/331894540 >> 172.21.15.89:0/331894540 conn(0x7f012008aff0 msgr2=0x7f01200d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:02.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.479+0000 7f0126800700 1 -- 172.21.15.89:0/331894540 shutdown_connections 2024-04-02T07:56:02.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.479+0000 7f0126800700 1 -- 172.21.15.89:0/331894540 wait complete. 2024-04-02T07:56:02.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.481+0000 7f0126800700 1 Processor -- start 2024-04-02T07:56:02.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.481+0000 7f0126800700 1 -- start start 2024-04-02T07:56:02.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.482+0000 7f0126800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01200ceec0 0x7f0120158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:02.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.482+0000 7f0126800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f012015d640 0x7f01201f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:02.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.482+0000 7f0126800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0120161800 0x7f01200cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:02.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.482+0000 7f0126800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01200cb780 con 0x7f01200ceec0 2024-04-02T07:56:02.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.482+0000 7f0126800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01200cd160 con 0x7f012015d640 2024-04-02T07:56:02.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.482+0000 7f0126800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01200cd440 con 0x7f0120161800 2024-04-02T07:56:02.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.482+0000 7f011f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01200ceec0 0x7f0120158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:02.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.482+0000 7f011f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01200ceec0 0x7f0120158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54236/0 (socket says 172.21.15.89:54236) 2024-04-02T07:56:02.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.483+0000 7f011f400700 1 -- 172.21.15.89:0/2690792370 learned_addr learned my addr 172.21.15.89:0/2690792370 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:02.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.483+0000 7f011ea00700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f012015d640 0x7f01201f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:02.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.483+0000 7f011fe00700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0120161800 0x7f01200cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:02.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.483+0000 7f011f400700 1 -- 172.21.15.89:0/2690792370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0120161800 msgr2=0x7f01200cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:02.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.483+0000 7f011f400700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0120161800 0x7f01200cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.483+0000 7f011f400700 1 -- 172.21.15.89:0/2690792370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f012015d640 msgr2=0x7f01201f4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:02.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.483+0000 7f011f400700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f012015d640 0x7f01201f4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.483+0000 7f011f400700 1 -- 172.21.15.89:0/2690792370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010c0094a0 con 0x7f01200ceec0 2024-04-02T07:56:02.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.483+0000 7f011ea00700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f012015d640 0x7f01201f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:56:02.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.483+0000 7f011f400700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01200ceec0 0x7f0120158a20 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f011000c520 tx=0x7f011000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:02.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.484+0000 7f0117400700 1 -- 172.21.15.89:0/2690792370 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01100052d0 con 0x7f01200ceec0 2024-04-02T07:56:02.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.484+0000 7f0117400700 1 -- 172.21.15.89:0/2690792370 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0110005430 con 0x7f01200ceec0 2024-04-02T07:56:02.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.484+0000 7f0126800700 1 -- 172.21.15.89:0/2690792370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01200cd720 con 0x7f01200ceec0 2024-04-02T07:56:02.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.484+0000 7f0117400700 1 -- 172.21.15.89:0/2690792370 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01100038c0 con 0x7f01200ceec0 2024-04-02T07:56:02.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.484+0000 7f0126800700 1 -- 172.21.15.89:0/2690792370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01201f4560 con 0x7f01200ceec0 2024-04-02T07:56:02.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.485+0000 7f0126800700 1 -- 172.21.15.89:0/2690792370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01200cd960 con 0x7f01200ceec0 2024-04-02T07:56:02.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.486+0000 7f0117400700 1 -- 172.21.15.89:0/2690792370 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f0110003a20 con 0x7f01200ceec0 2024-04-02T07:56:02.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.487+0000 7f0117400700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f010006ae40 0x7f010006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:02.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.487+0000 7f0117400700 1 -- 172.21.15.89:0/2690792370 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f0110088d20 con 0x7f01200ceec0 2024-04-02T07:56:02.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.487+0000 7f011ea00700 1 -- 172.21.15.89:0/2690792370 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f010006ae40 msgr2=0x7f010006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:02.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.487+0000 7f011ea00700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f010006ae40 0x7f010006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:56:02.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.491+0000 7f0117400700 1 -- 172.21.15.89:0/2690792370 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f01200cd960 con 0x7f01200ceec0 2024-04-02T07:56:02.571 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:02.570+0000 7f3e94351240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:56:02.651 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:02.650+0000 7f231fc35240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:56:02.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.688+0000 7f011ea00700 1 -- 172.21.15.89:0/2690792370 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f010006ae40 msgr2=0x7f010006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:02.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.688+0000 7f011ea00700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f010006ae40 0x7f010006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:56:02.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.745+0000 7f0126800700 1 -- 172.21.15.89:0/2690792370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01200cd960 con 0x7f01200ceec0 2024-04-02T07:56:02.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.746+0000 7f0117400700 1 -- 172.21.15.89:0/2690792370 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4733 (secure 0 0 0) 0x7f01200cd960 con 0x7f01200ceec0 2024-04-02T07:56:02.747 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:02.747 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/66826967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":66826967},{"type":"v1","addr":"172.21.15.113:6835","nonce":66826967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":7915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/1669765650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1669765650},{"type":"v1","addr":"172.21.15.113:6833","nonce":1669765650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":7918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3321722319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3321722319},{"type":"v1","addr":"172.21.15.89:6837","nonce":3321722319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:55:31.571731+0000","modified":"2024-04-02T07:55:34.583055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8436":{"gid":8436,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":4,"addr":"172.21.15.89:6836/417215494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":417215494},{"type":"v1","addr":"172.21.15.89:6836","nonce":417215494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-02T07:56:02.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.749+0000 7f0126800700 1 -- 172.21.15.89:0/2690792370 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f010006ae40 msgr2=0x7f010006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:56:02.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.749+0000 7f0126800700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f010006ae40 0x7f010006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.749+0000 7f0126800700 1 -- 172.21.15.89:0/2690792370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01200ceec0 msgr2=0x7f0120158a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:02.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.749+0000 7f0126800700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01200ceec0 0x7f0120158a20 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f011000c520 tx=0x7f011000c830 comp rx=0 tx=0).stop 2024-04-02T07:56:02.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.749+0000 7f0126800700 1 -- 172.21.15.89:0/2690792370 shutdown_connections 2024-04-02T07:56:02.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.749+0000 7f0126800700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f010006ae40 0x7f010006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.749+0000 7f0126800700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0120161800 0x7f01200cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.749+0000 7f0126800700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01200ceec0 0x7f0120158a20 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.749+0000 7f0126800700 1 --2- 172.21.15.89:0/2690792370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f012015d640 0x7f01201f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.749+0000 7f0126800700 1 -- 172.21.15.89:0/2690792370 >> 172.21.15.89:0/2690792370 conn(0x7f012008aff0 msgr2=0x7f012006ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:02.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.749+0000 7f0126800700 1 -- 172.21.15.89:0/2690792370 shutdown_connections 2024-04-02T07:56:02.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.749+0000 7f0126800700 1 -- 172.21.15.89:0/2690792370 wait complete. 2024-04-02T07:56:02.751 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 66 2024-04-02T07:56:02.761 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:02.760+0000 7f231fc35240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:56:02.770 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:56:02.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.973+0000 7feefbc00700 1 Processor -- start 2024-04-02T07:56:02.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.973+0000 7feefbc00700 1 -- start start 2024-04-02T07:56:02.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.974+0000 7feefbc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feef406a790 0x7feef406ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:02.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.974+0000 7feefbc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7feef40cebd0 0x7feef40cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:02.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.974+0000 7feefbc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef406f850 con 0x7feef40afbe0 2024-04-02T07:56:02.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.974+0000 7feefbc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef406e3f0 con 0x7feef40cebd0 2024-04-02T07:56:02.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.974+0000 7feefbc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef406b0b0 con 0x7feef406a790 2024-04-02T07:56:02.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.974+0000 7feef3e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7feef40afbe0 0x7feef406dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55564/0 (socket says 172.21.15.89:55564) 2024-04-02T07:56:02.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.975+0000 7feef3e00700 1 -- 172.21.15.89:0/1049044330 learned_addr learned my addr 172.21.15.89:0/1049044330 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:02.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.975+0000 7feef9200700 1 --2- 172.21.15.89:0/1049044330 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feef406a790 0x7feef406ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:02.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.975+0000 7feef9c00700 1 --2- 172.21.15.89:0/1049044330 >> v2:172.21.15.113:3300/0 conn(0x7feef40cebd0 0x7feef40cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.975+0000 7feef3400700 1 -- 172.21.15.89:0/1049044330 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813431879 0 0) 0x7feef406f850 con 0x7feef40afbe0 2024-04-02T07:56:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.975+0000 7feef3400700 1 -- 172.21.15.89:0/1049044330 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feedc0035b0 con 0x7feef40afbe0 2024-04-02T07:56:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.975+0000 7feef9200700 1 -- 172.21.15.89:0/1049044330 >> v2:172.21.15.113:3300/0 conn(0x7feef40cebd0 msgr2=0x7feef40cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.975+0000 7feef9200700 1 --2- 172.21.15.89:0/1049044330 >> v2:172.21.15.113:3300/0 conn(0x7feef40cebd0 0x7feef40cf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.975+0000 7feef9200700 1 -- 172.21.15.89:0/1049044330 >> v1:172.21.15.89:6789/0 conn(0x7feef40afbe0 legacy=0x7feef406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.975+0000 7feef9200700 1 -- 172.21.15.89:0/1049044330 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef406fad0 con 0x7feef406a790 2024-04-02T07:56:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.975+0000 7feef9c00700 1 --2- 172.21.15.89:0/1049044330 >> v2:172.21.15.113:3300/0 conn(0x7feef40cebd0 0x7feef40cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:56:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.975+0000 7feef9200700 1 --2- 172.21.15.89:0/1049044330 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feef406a790 0x7feef406ab70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feee8009840 tx=0x7feee8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f37ef494d9debdf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.976+0000 7feef3400700 1 -- 172.21.15.89:0/1049044330 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8011830 con 0x7feef406a790 2024-04-02T07:56:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.976+0000 7feef3400700 1 -- 172.21.15.89:0/1049044330 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee8011990 con 0x7feef406a790 2024-04-02T07:56:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.976+0000 7feef3400700 1 -- 172.21.15.89:0/1049044330 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8011d20 con 0x7feef406a790 2024-04-02T07:56:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.976+0000 7feefbc00700 1 -- 172.21.15.89:0/1049044330 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feef406a790 msgr2=0x7feef406ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.976+0000 7feefbc00700 1 --2- 172.21.15.89:0/1049044330 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feef406a790 0x7feef406ab70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feee8009840 tx=0x7feee8009b50 comp rx=0 tx=0).stop 2024-04-02T07:56:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.976+0000 7feefbc00700 1 -- 172.21.15.89:0/1049044330 shutdown_connections 2024-04-02T07:56:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.976+0000 7feefbc00700 1 --2- 172.21.15.89:0/1049044330 >> v2:172.21.15.113:3300/0 conn(0x7feef40cebd0 0x7feef40cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.976+0000 7feefbc00700 1 --2- 172.21.15.89:0/1049044330 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feef406a790 0x7feef406ab70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.976+0000 7feefbc00700 1 -- 172.21.15.89:0/1049044330 >> 172.21.15.89:0/1049044330 conn(0x7feef40b5ab0 msgr2=0x7feef40b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.976+0000 7feefbc00700 1 -- 172.21.15.89:0/1049044330 shutdown_connections 2024-04-02T07:56:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.976+0000 7feefbc00700 1 -- 172.21.15.89:0/1049044330 wait complete. 2024-04-02T07:56:02.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.978+0000 7feefbc00700 1 Processor -- start 2024-04-02T07:56:02.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.978+0000 7feefbc00700 1 -- start start 2024-04-02T07:56:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.979+0000 7feefbc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feef406a790 0x7feef41f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.979+0000 7feefbc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feef40afbe0 0x7feef414c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.979+0000 7feefbc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef40cebd0 0x7feef414cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.979+0000 7feefbc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef414d960 con 0x7feef40afbe0 2024-04-02T07:56:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.979+0000 7feefbc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef41f2f20 con 0x7feef40cebd0 2024-04-02T07:56:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.979+0000 7feefbc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef414d440 con 0x7feef406a790 2024-04-02T07:56:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.979+0000 7feef3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feef40afbe0 0x7feef414c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.979+0000 7feef3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feef40afbe0 0x7feef414c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54240/0 (socket says 172.21.15.89:54240) 2024-04-02T07:56:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.979+0000 7feef3e00700 1 -- 172.21.15.89:0/3505644508 learned_addr learned my addr 172.21.15.89:0/3505644508 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.980+0000 7feef9200700 1 --2- 172.21.15.89:0/3505644508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feef406a790 0x7feef41f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.980+0000 7feef9c00700 1 --2- 172.21.15.89:0/3505644508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef40cebd0 0x7feef414cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.980+0000 7feef3e00700 1 -- 172.21.15.89:0/3505644508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feef406a790 msgr2=0x7feef41f29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.980+0000 7feef3e00700 1 --2- 172.21.15.89:0/3505644508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feef406a790 0x7feef41f29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.980+0000 7feef3e00700 1 -- 172.21.15.89:0/3505644508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef40cebd0 msgr2=0x7feef414cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.980+0000 7feef3e00700 1 --2- 172.21.15.89:0/3505644508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef40cebd0 0x7feef414cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.980+0000 7feef3e00700 1 -- 172.21.15.89:0/3505644508 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec009a70 con 0x7feef40afbe0 2024-04-02T07:56:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.980+0000 7feef3e00700 1 --2- 172.21.15.89:0/3505644508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feef40afbe0 0x7feef414c9c0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7feee0001eb0 tx=0x7feee000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.980+0000 7feef0c00700 1 -- 172.21.15.89:0/3505644508 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee0005d70 con 0x7feef40afbe0 2024-04-02T07:56:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.980+0000 7feef0c00700 1 -- 172.21.15.89:0/3505644508 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee000ce40 con 0x7feef40afbe0 2024-04-02T07:56:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.980+0000 7feefbc00700 1 -- 172.21.15.89:0/3505644508 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee80094a0 con 0x7feef40afbe0 2024-04-02T07:56:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.980+0000 7feef0c00700 1 -- 172.21.15.89:0/3505644508 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee00072f0 con 0x7feef40afbe0 2024-04-02T07:56:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.980+0000 7feefbc00700 1 -- 172.21.15.89:0/3505644508 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef41f6dd0 con 0x7feef40afbe0 2024-04-02T07:56:02.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.981+0000 7feefbc00700 1 -- 172.21.15.89:0/3505644508 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef41f6f60 con 0x7feef40afbe0 2024-04-02T07:56:02.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.982+0000 7feef0c00700 1 -- 172.21.15.89:0/3505644508 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 85) v1 ==== 89296+0+0 (secure 0 0 0) 0x7feee0007450 con 0x7feef40afbe0 2024-04-02T07:56:02.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.983+0000 7feef0c00700 1 --2- 172.21.15.89:0/3505644508 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feed406ae40 0x7feed406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:02.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.983+0000 7feef0c00700 1 -- 172.21.15.89:0/3505644508 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6429+0+0 (secure 0 0 0) 0x7feee0088c10 con 0x7feef40afbe0 2024-04-02T07:56:02.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.983+0000 7feef9200700 1 -- 172.21.15.89:0/3505644508 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feed406ae40 msgr2=0x7feed406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:02.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.983+0000 7feef9200700 1 --2- 172.21.15.89:0/3505644508 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feed406ae40 0x7feed406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:56:02.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:02.988+0000 7feef0c00700 1 -- 172.21.15.89:0/3505644508 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feee0053890 con 0x7feef40afbe0 2024-04-02T07:56:03.042 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:03.042+0000 7f3e94351240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:56:03.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.081+0000 7feef0c00700 1 -- 172.21.15.89:0/3505644508 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 86) v1 ==== 89296+0+0 (secure 0 0 0) 0x7feee0053c60 con 0x7feef40afbe0 2024-04-02T07:56:03.166 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:03.165+0000 7f3e94351240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:56:03.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.176+0000 7feef0c00700 1 -- 172.21.15.89:0/3505644508 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 87) v1 ==== 88848+0+0 (secure 0 0 0) 0x7feee0086d30 con 0x7feef40afbe0 2024-04-02T07:56:03.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.176+0000 7feef0c00700 1 -- 172.21.15.89:0/3505644508 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feed406ae40 msgr2=0x7feed406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:56:03.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.176+0000 7feef0c00700 1 --2- 172.21.15.89:0/3505644508 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feed406ae40 0x7feed406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.245 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:03.244+0000 7f3e03800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:03.245 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:03.244+0000 7f3e03800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:03.245 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:03.244+0000 7f3e03800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:03.245 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:03.244+0000 7f3e03800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:03.245 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:03.244+0000 7f3e03800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:03.245 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:03.244+0000 7f3e07400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:03.245 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:03.244+0000 7f3e07400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:03.245 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:03.244+0000 7f3e07400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:03.245 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:03.244+0000 7f3e07400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:03.245 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:03.244+0000 7f3e07400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:03.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.275+0000 7feefbc00700 1 -- 172.21.15.89:0/3505644508 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feef4067720 con 0x7feef40afbe0 2024-04-02T07:56:03.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.276+0000 7feef0c00700 1 -- 172.21.15.89:0/3505644508 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4733 (secure 0 0 0) 0x7feee00548a0 con 0x7feef40afbe0 2024-04-02T07:56:03.276 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:03.277 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/66826967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":66826967},{"type":"v1","addr":"172.21.15.113:6835","nonce":66826967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":7915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/1669765650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1669765650},{"type":"v1","addr":"172.21.15.113:6833","nonce":1669765650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":7918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3321722319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3321722319},{"type":"v1","addr":"172.21.15.89:6837","nonce":3321722319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:55:31.571731+0000","modified":"2024-04-02T07:55:34.583055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8436":{"gid":8436,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":4,"addr":"172.21.15.89:6836/417215494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":417215494},{"type":"v1","addr":"172.21.15.89:6836","nonce":417215494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-02T07:56:03.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.278+0000 7feefbc00700 1 -- 172.21.15.89:0/3505644508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feef40afbe0 msgr2=0x7feef414c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:03.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.278+0000 7feefbc00700 1 --2- 172.21.15.89:0/3505644508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feef40afbe0 0x7feef414c9c0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7feee0001eb0 tx=0x7feee000c7b0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.278+0000 7feefbc00700 1 -- 172.21.15.89:0/3505644508 shutdown_connections 2024-04-02T07:56:03.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.278+0000 7feefbc00700 1 --2- 172.21.15.89:0/3505644508 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feed406ae40 0x7feed406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.278+0000 7feefbc00700 1 --2- 172.21.15.89:0/3505644508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef40cebd0 0x7feef414cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.278+0000 7feefbc00700 1 --2- 172.21.15.89:0/3505644508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feef406a790 0x7feef41f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.278+0000 7feefbc00700 1 --2- 172.21.15.89:0/3505644508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feef40afbe0 0x7feef414c9c0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.278+0000 7feefbc00700 1 -- 172.21.15.89:0/3505644508 >> 172.21.15.89:0/3505644508 conn(0x7feef40b5ab0 msgr2=0x7feef40ad350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:03.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.279+0000 7feefbc00700 1 -- 172.21.15.89:0/3505644508 shutdown_connections 2024-04-02T07:56:03.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.279+0000 7feefbc00700 1 -- 172.21.15.89:0/3505644508 wait complete. 2024-04-02T07:56:03.280 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 66 2024-04-02T07:56:03.298 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:56:03.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.489+0000 7efda8600700 1 Processor -- start 2024-04-02T07:56:03.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.489+0000 7efda8600700 1 -- start start 2024-04-02T07:56:03.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.490+0000 7efda8600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efda00d18b0 0x7efda00cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:03.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.490+0000 7efda8600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda00ceda0 con 0x7efda0160f10 2024-04-02T07:56:03.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.490+0000 7efda8600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda01612f0 con 0x7efda0161590 2024-04-02T07:56:03.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.490+0000 7efda8600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda00d1d30 con 0x7efda00d18b0 2024-04-02T07:56:03.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.490+0000 7efda5c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efda0160f10 0x7efda00d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55570/0 (socket says 172.21.15.89:55570) 2024-04-02T07:56:03.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.490+0000 7efda5c00700 1 -- 172.21.15.89:0/648908449 learned_addr learned my addr 172.21.15.89:0/648908449 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:03.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.490+0000 7efda5200700 1 --2- 172.21.15.89:0/648908449 >> v2:172.21.15.89:3300/0 conn(0x7efda00d18b0 0x7efda00cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:03.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.491+0000 7efd9fe00700 1 -- 172.21.15.89:0/648908449 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195949410 0 0) 0x7efda01612f0 con 0x7efda0161590 2024-04-02T07:56:03.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.491+0000 7efd9fe00700 1 -- 172.21.15.89:0/648908449 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd880035b0 con 0x7efda0161590 2024-04-02T07:56:03.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.491+0000 7efda5200700 1 -- 172.21.15.89:0/648908449 >> v1:172.21.15.113:6789/0 conn(0x7efda0161590 legacy=0x7efda00b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:03.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.491+0000 7efda5200700 1 -- 172.21.15.89:0/648908449 >> v1:172.21.15.89:6789/0 conn(0x7efda0160f10 legacy=0x7efda00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:03.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.491+0000 7efd9fe00700 1 -- 172.21.15.89:0/648908449 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137598273 0 0) 0x7efda00ceda0 con 0x7efda0160f10 2024-04-02T07:56:03.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.491+0000 7efda5200700 1 -- 172.21.15.89:0/648908449 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda00ce470 con 0x7efda00d18b0 2024-04-02T07:56:03.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.491+0000 7efda5200700 1 --2- 172.21.15.89:0/648908449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efda00d18b0 0x7efda00cf230 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7efd8c009840 tx=0x7efd8c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eba5bdcfee26f1e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:03.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.491+0000 7efd9fe00700 1 -- 172.21.15.89:0/648908449 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c011830 con 0x7efda00d18b0 2024-04-02T07:56:03.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.491+0000 7efd9fe00700 1 -- 172.21.15.89:0/648908449 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8c011990 con 0x7efda00d18b0 2024-04-02T07:56:03.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.491+0000 7efd9fe00700 1 -- 172.21.15.89:0/648908449 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c011d20 con 0x7efda00d18b0 2024-04-02T07:56:03.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.492+0000 7efda8600700 1 -- 172.21.15.89:0/648908449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efda00d18b0 msgr2=0x7efda00cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:03.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.492+0000 7efda8600700 1 --2- 172.21.15.89:0/648908449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efda00d18b0 0x7efda00cf230 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7efd8c009840 tx=0x7efd8c009b50 comp rx=0 tx=0).stop 2024-04-02T07:56:03.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.492+0000 7efda8600700 1 -- 172.21.15.89:0/648908449 shutdown_connections 2024-04-02T07:56:03.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.492+0000 7efda8600700 1 --2- 172.21.15.89:0/648908449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efda00d18b0 0x7efda00cf230 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.492+0000 7efda8600700 1 -- 172.21.15.89:0/648908449 >> 172.21.15.89:0/648908449 conn(0x7efda008aff0 msgr2=0x7efda00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:03.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.492+0000 7efda8600700 1 -- 172.21.15.89:0/648908449 shutdown_connections 2024-04-02T07:56:03.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.492+0000 7efda8600700 1 -- 172.21.15.89:0/648908449 wait complete. 2024-04-02T07:56:03.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.494+0000 7efda8600700 1 Processor -- start 2024-04-02T07:56:03.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.494+0000 7efda8600700 1 -- start start 2024-04-02T07:56:03.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda8600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efda00d18b0 0x7efda015e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:03.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda8600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efda0160f10 0x7efda00b86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:03.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda8600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efda0161590 0x7efda00c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:03.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda8600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda015e960 con 0x7efda00d18b0 2024-04-02T07:56:03.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda8600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda00b8c00 con 0x7efda0161590 2024-04-02T07:56:03.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda8600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda015f7a0 con 0x7efda0160f10 2024-04-02T07:56:03.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda5c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efda00d18b0 0x7efda015e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:03.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda5c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efda00d18b0 0x7efda015e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54366/0 (socket says 172.21.15.89:54366) 2024-04-02T07:56:03.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda5c00700 1 -- 172.21.15.89:0/269876567 learned_addr learned my addr 172.21.15.89:0/269876567 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:03.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda5c00700 1 -- 172.21.15.89:0/269876567 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efda0160f10 msgr2=0x7efda00b86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:03.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda5c00700 1 --2- 172.21.15.89:0/269876567 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efda0160f10 0x7efda00b86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda5c00700 1 -- 172.21.15.89:0/269876567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efda0161590 msgr2=0x7efda00c9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:03.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda6600700 1 --2- 172.21.15.89:0/269876567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efda0161590 0x7efda00c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:03.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda5c00700 1 --2- 172.21.15.89:0/269876567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efda0161590 0x7efda00c9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda5c00700 1 -- 172.21.15.89:0/269876567 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c0094a0 con 0x7efda00d18b0 2024-04-02T07:56:03.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.495+0000 7efda6600700 1 --2- 172.21.15.89:0/269876567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efda0161590 0x7efda00c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:03.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.496+0000 7efda5c00700 1 --2- 172.21.15.89:0/269876567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efda00d18b0 0x7efda015e420 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7efd9400c520 tx=0x7efd9400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:03.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.496+0000 7efd9d600700 1 -- 172.21.15.89:0/269876567 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd94004c80 con 0x7efda00d18b0 2024-04-02T07:56:03.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.496+0000 7efd9d600700 1 -- 172.21.15.89:0/269876567 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd94004e20 con 0x7efda00d18b0 2024-04-02T07:56:03.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.496+0000 7efda8600700 1 -- 172.21.15.89:0/269876567 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda015eb90 con 0x7efda00d18b0 2024-04-02T07:56:03.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.496+0000 7efd9d600700 1 -- 172.21.15.89:0/269876567 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd94007310 con 0x7efda00d18b0 2024-04-02T07:56:03.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.496+0000 7efda8600700 1 -- 172.21.15.89:0/269876567 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda015f150 con 0x7efda00d18b0 2024-04-02T07:56:03.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.497+0000 7efda8600700 1 -- 172.21.15.89:0/269876567 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda00729f0 con 0x7efda00d18b0 2024-04-02T07:56:03.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.498+0000 7efd9d600700 1 -- 172.21.15.89:0/269876567 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 87) v1 ==== 88848+0+0 (secure 0 0 0) 0x7efd94007470 con 0x7efda00d18b0 2024-04-02T07:56:03.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.499+0000 7efd9d600700 1 -- 172.21.15.89:0/269876567 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6730+0+0 (secure 0 0 0) 0x7efd94088f20 con 0x7efda00d18b0 2024-04-02T07:56:03.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.503+0000 7efd9d600700 1 -- 172.21.15.89:0/269876567 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efd94054060 con 0x7efda00d18b0 2024-04-02T07:56:03.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.710+0000 7efda8600700 1 -- 172.21.15.89:0/269876567 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efda0067720 con 0x7efda00d18b0 2024-04-02T07:56:03.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.711+0000 7efd9d600700 1 -- 172.21.15.89:0/269876567 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+17608 (secure 0 0 0) 0x7efd94057680 con 0x7efda00d18b0 2024-04-02T07:56:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:03.712 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":142,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:56:03.168328+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:55:29.329907+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-02T07:55:30.555350+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/3697974226":"2024-04-03T07:56:03.168290+0000","172.21.15.89:0/339054279":"2024-04-03T07:56:03.168290+0000","172.21.15.89:0/1901641921":"2024-04-03T07:56:03.168290+0000","172.21.15.89:0/2953791885":"2024-04-03T07:56:03.168290+0000","172.21.15.89:0/3060505378":"2024-04-03T07:56:03.168290+0000","172.21.15.89:0/2623780123":"2024-04-03T07:56:03.168290+0000","172.21.15.89:0/836334016":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/1824838019":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/3970678073":"2024-04-03T07:56:03.168290+0000","172.21.15.89:0/2769562366":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/224945013":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/732038760":"2024-04-03T07:55:47.785863+0000","172.21.15.89:6833/43357":"2024-04-03T07:55:47.785863+0000","172.21.15.89:6832/43357":"2024-04-03T07:55:47.785863+0000","172.21.15.89:0/3727408218":"2024-04-03T07:55:47.785863+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:56:03.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.713+0000 7efda8600700 1 -- 172.21.15.89:0/269876567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efda00d18b0 msgr2=0x7efda015e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:03.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.713+0000 7efda8600700 1 --2- 172.21.15.89:0/269876567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efda00d18b0 0x7efda015e420 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7efd9400c520 tx=0x7efd9400c8e0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.713+0000 7efda8600700 1 -- 172.21.15.89:0/269876567 shutdown_connections 2024-04-02T07:56:03.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.713+0000 7efda8600700 1 --2- 172.21.15.89:0/269876567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efda0161590 0x7efda00c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.713+0000 7efda8600700 1 --2- 172.21.15.89:0/269876567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efda00d18b0 0x7efda015e420 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.713+0000 7efda8600700 1 --2- 172.21.15.89:0/269876567 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efda0160f10 0x7efda00b86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.713+0000 7efda8600700 1 -- 172.21.15.89:0/269876567 >> 172.21.15.89:0/269876567 conn(0x7efda008aff0 msgr2=0x7efda00cac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:03.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.714+0000 7efda8600700 1 -- 172.21.15.89:0/269876567 shutdown_connections 2024-04-02T07:56:03.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.714+0000 7efda8600700 1 -- 172.21.15.89:0/269876567 wait complete. 2024-04-02T07:56:03.732 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T07:56:03.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.924+0000 7fae53400700 1 Processor -- start 2024-04-02T07:56:03.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.924+0000 7fae53400700 1 -- start start 2024-04-02T07:56:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.925+0000 7fae53400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fae4c160d70 0x7fae4c0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.925+0000 7fae53400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fae4c15ef70 0x7fae4c15e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.925+0000 7fae53400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae4c06a790 0x7fae4c06ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.925+0000 7fae53400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c06b180 con 0x7fae4c160d70 2024-04-02T07:56:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.925+0000 7fae53400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c0b4cd0 con 0x7fae4c15ef70 2024-04-02T07:56:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.925+0000 7fae53400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c161150 con 0x7fae4c06a790 2024-04-02T07:56:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.925+0000 7fae50a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fae4c160d70 0x7fae4c0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.925+0000 7fae50a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fae4c160d70 0x7fae4c0b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54368/0 (socket says 172.21.15.89:54368) 2024-04-02T07:56:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.925+0000 7fae50a00700 1 -- 172.21.15.89:0/1469566182 learned_addr learned my addr 172.21.15.89:0/1469566182 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.925+0000 7fae51400700 1 --2- 172.21.15.89:0/1469566182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae4c06a790 0x7fae4c06ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.925+0000 7fae4be00700 1 --2- 172.21.15.89:0/1469566182 >> v2:172.21.15.113:3300/0 conn(0x7fae4c15ef70 0x7fae4c15e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.926+0000 7fae50a00700 1 -- 172.21.15.89:0/1469566182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae4c06a790 msgr2=0x7fae4c06ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.926+0000 7fae50a00700 1 --2- 172.21.15.89:0/1469566182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae4c06a790 0x7fae4c06ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.926+0000 7fae50a00700 1 -- 172.21.15.89:0/1469566182 >> v2:172.21.15.113:3300/0 conn(0x7fae4c15ef70 msgr2=0x7fae4c15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.926+0000 7fae50a00700 1 --2- 172.21.15.89:0/1469566182 >> v2:172.21.15.113:3300/0 conn(0x7fae4c15ef70 0x7fae4c15e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.926+0000 7fae50a00700 1 -- 172.21.15.89:0/1469566182 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c06dba0 con 0x7fae4c160d70 2024-04-02T07:56:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.926+0000 7fae50a00700 1 --2- 172.21.15.89:0/1469566182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae4c160d70 0x7fae4c0b3200 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0d1370 tx=0x7fae40009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e252bb1fc1c8117 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:03.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.926+0000 7fae4b400700 1 -- 172.21.15.89:0/1469566182 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae40011830 con 0x7fae4c160d70 2024-04-02T07:56:03.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.926+0000 7fae4b400700 1 -- 172.21.15.89:0/1469566182 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae40011990 con 0x7fae4c160d70 2024-04-02T07:56:03.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.926+0000 7fae4b400700 1 -- 172.21.15.89:0/1469566182 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae40011d20 con 0x7fae4c160d70 2024-04-02T07:56:03.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.927+0000 7fae53400700 1 -- 172.21.15.89:0/1469566182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae4c160d70 msgr2=0x7fae4c0b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:03.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.927+0000 7fae53400700 1 --2- 172.21.15.89:0/1469566182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae4c160d70 0x7fae4c0b3200 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0d1370 tx=0x7fae40009bd0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.927+0000 7fae53400700 1 -- 172.21.15.89:0/1469566182 shutdown_connections 2024-04-02T07:56:03.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.927+0000 7fae53400700 1 --2- 172.21.15.89:0/1469566182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae4c160d70 0x7fae4c0b3200 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.927+0000 7fae53400700 1 --2- 172.21.15.89:0/1469566182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae4c06a790 0x7fae4c06ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.927+0000 7fae53400700 1 --2- 172.21.15.89:0/1469566182 >> v2:172.21.15.113:3300/0 conn(0x7fae4c15ef70 0x7fae4c15e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.927+0000 7fae53400700 1 -- 172.21.15.89:0/1469566182 >> 172.21.15.89:0/1469566182 conn(0x7fae4c0b5720 msgr2=0x7fae4c0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:03.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.927+0000 7fae53400700 1 -- 172.21.15.89:0/1469566182 shutdown_connections 2024-04-02T07:56:03.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.927+0000 7fae53400700 1 -- 172.21.15.89:0/1469566182 wait complete. 2024-04-02T07:56:03.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.929+0000 7fae53400700 1 Processor -- start 2024-04-02T07:56:03.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.929+0000 7fae53400700 1 -- start start 2024-04-02T07:56:03.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.930+0000 7fae53400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae4c06a790 0x7fae4c14d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:03.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.930+0000 7fae53400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae4c15ef70 0x7fae4c1495c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:03.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.930+0000 7fae53400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae4c160d70 0x7fae4c149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:03.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.930+0000 7fae53400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0aecb0 con 0x7fae4c15ef70 2024-04-02T07:56:03.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.930+0000 7fae53400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c14aa60 con 0x7fae4c06a790 2024-04-02T07:56:03.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.930+0000 7fae53400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0ae9d0 con 0x7fae4c160d70 2024-04-02T07:56:03.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.930+0000 7fae4be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae4c15ef70 0x7fae4c1495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.930+0000 7fae4be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae4c15ef70 0x7fae4c1495c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54376/0 (socket says 172.21.15.89:54376) 2024-04-02T07:56:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.930+0000 7fae4be00700 1 -- 172.21.15.89:0/1891893618 learned_addr learned my addr 172.21.15.89:0/1891893618 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.930+0000 7fae50a00700 1 --2- 172.21.15.89:0/1891893618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae4c06a790 0x7fae4c14d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.930+0000 7fae51400700 1 --2- 172.21.15.89:0/1891893618 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae4c160d70 0x7fae4c149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.930+0000 7fae4be00700 1 -- 172.21.15.89:0/1891893618 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae4c160d70 msgr2=0x7fae4c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.931+0000 7fae4be00700 1 --2- 172.21.15.89:0/1891893618 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae4c160d70 0x7fae4c149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.931+0000 7fae4be00700 1 -- 172.21.15.89:0/1891893618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae4c06a790 msgr2=0x7fae4c14d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.931+0000 7fae4be00700 1 --2- 172.21.15.89:0/1891893618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae4c06a790 0x7fae4c14d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.931+0000 7fae4be00700 1 -- 172.21.15.89:0/1891893618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae40009550 con 0x7fae4c15ef70 2024-04-02T07:56:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.931+0000 7fae4be00700 1 --2- 172.21.15.89:0/1891893618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae4c15ef70 0x7fae4c1495c0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fae3800e9b0 tx=0x7fae3800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.931+0000 7fae48c00700 1 -- 172.21.15.89:0/1891893618 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38009710 con 0x7fae4c15ef70 2024-04-02T07:56:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.931+0000 7fae48c00700 1 -- 172.21.15.89:0/1891893618 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3800c850 con 0x7fae4c15ef70 2024-04-02T07:56:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.931+0000 7fae53400700 1 -- 172.21.15.89:0/1891893618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4c147710 con 0x7fae4c15ef70 2024-04-02T07:56:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.931+0000 7fae48c00700 1 -- 172.21.15.89:0/1891893618 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3800cc00 con 0x7fae4c15ef70 2024-04-02T07:56:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.931+0000 7fae53400700 1 -- 172.21.15.89:0/1891893618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4c147cd0 con 0x7fae4c15ef70 2024-04-02T07:56:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.932+0000 7fae53400700 1 -- 172.21.15.89:0/1891893618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4c147570 con 0x7fae4c15ef70 2024-04-02T07:56:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.933+0000 7fae48c00700 1 -- 172.21.15.89:0/1891893618 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 87) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fae3800cd60 con 0x7fae4c15ef70 2024-04-02T07:56:03.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.934+0000 7fae48c00700 1 -- 172.21.15.89:0/1891893618 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6730+0+0 (secure 0 0 0) 0x7fae38089250 con 0x7fae4c15ef70 2024-04-02T07:56:03.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:03.938+0000 7fae48c00700 1 -- 172.21.15.89:0/1891893618 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae4c147570 con 0x7fae4c15ef70 2024-04-02T07:56:04.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.180+0000 7fae48c00700 1 -- 172.21.15.89:0/1891893618 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 88) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fae38053e40 con 0x7fae4c15ef70 2024-04-02T07:56:04.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.180+0000 7fae48c00700 1 --2- 172.21.15.89:0/1891893618 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae2c036150 0x7fae2c0726d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:04.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.183+0000 7fae50a00700 1 --2- 172.21.15.89:0/1891893618 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae2c036150 0x7fae2c0726d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:04.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.186+0000 7fae50a00700 1 --2- 172.21.15.89:0/1891893618 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae2c036150 0x7fae2c0726d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fae40014040 tx=0x7fae40008030 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:04.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.194+0000 7fae53400700 1 -- 172.21.15.89:0/1891893618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fae4c147570 con 0x7fae4c15ef70 2024-04-02T07:56:04.195 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T07:56:04.195+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T07:56:04.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.204+0000 7fae48c00700 1 -- 172.21.15.89:0/1891893618 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v67) v1 ==== 161+0+0 (secure 0 0 0) 0x7fae4c147570 con 0x7fae4c15ef70 2024-04-02T07:56:04.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.207+0000 7fae53400700 1 -- 172.21.15.89:0/1891893618 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae2c036150 msgr2=0x7fae2c0726d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:04.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.207+0000 7fae53400700 1 --2- 172.21.15.89:0/1891893618 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae2c036150 0x7fae2c0726d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fae40014040 tx=0x7fae40008030 comp rx=0 tx=0).stop 2024-04-02T07:56:04.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.207+0000 7fae53400700 1 -- 172.21.15.89:0/1891893618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae4c15ef70 msgr2=0x7fae4c1495c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:04.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.207+0000 7fae53400700 1 --2- 172.21.15.89:0/1891893618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae4c15ef70 0x7fae4c1495c0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fae3800e9b0 tx=0x7fae3800ecc0 comp rx=0 tx=0).stop 2024-04-02T07:56:04.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.207+0000 7fae53400700 1 -- 172.21.15.89:0/1891893618 shutdown_connections 2024-04-02T07:56:04.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.207+0000 7fae53400700 1 --2- 172.21.15.89:0/1891893618 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae2c036150 0x7fae2c0726d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:04.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.207+0000 7fae53400700 1 --2- 172.21.15.89:0/1891893618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae4c06a790 0x7fae4c14d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:04.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.207+0000 7fae53400700 1 --2- 172.21.15.89:0/1891893618 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae4c160d70 0x7fae4c149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:04.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.207+0000 7fae53400700 1 --2- 172.21.15.89:0/1891893618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae4c15ef70 0x7fae4c1495c0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:04.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.207+0000 7fae53400700 1 -- 172.21.15.89:0/1891893618 >> 172.21.15.89:0/1891893618 conn(0x7fae4c0b5720 msgr2=0x7fae4c159020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:04.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.207+0000 7fae53400700 1 -- 172.21.15.89:0/1891893618 shutdown_connections 2024-04-02T07:56:04.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.207+0000 7fae53400700 1 -- 172.21.15.89:0/1891893618 wait complete. 2024-04-02T07:56:04.209 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T07:56:04.215 INFO:tasks.ceph.mds.c.smithi089.stderr: -9800> 2024-04-02T07:55:35.781+0000 7f5d5dc00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T07:56:04.226 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T07:56:04.270 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T07:56:04.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.418+0000 7f6c73200700 1 Processor -- start 2024-04-02T07:56:04.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.419+0000 7f6c73200700 1 -- start start 2024-04-02T07:56:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.420+0000 7f6c73200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6c6c15d6d0 0x7f6c6c161430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.420+0000 7f6c73200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c6c15dab0 0x7f6c6c161970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.420+0000 7f6c73200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c6c161eb0 con 0x7f6c6c0d40f0 2024-04-02T07:56:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.420+0000 7f6c73200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c6c0b2d20 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.420+0000 7f6c73200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c1623e0 con 0x7f6c6c15dab0 2024-04-02T07:56:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.420+0000 7f6c6be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6c6c15d6d0 0x7f6c6c161430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.420+0000 7f6c6b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6c6c0d40f0 0x7f6c6c0d1830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55590/0 (socket says 172.21.15.89:55590) 2024-04-02T07:56:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.420+0000 7f6c6b400700 1 -- 172.21.15.89:0/3375296739 learned_addr learned my addr 172.21.15.89:0/3375296739 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.420+0000 7f6c71200700 1 --2- 172.21.15.89:0/3375296739 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c6c15dab0 0x7f6c6c161970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.420+0000 7f6c6be00700 1 -- 172.21.15.89:0/3375296739 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c6c15dab0 msgr2=0x7f6c6c161970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.420+0000 7f6c6be00700 1 --2- 172.21.15.89:0/3375296739 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c6c15dab0 0x7f6c6c161970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.420+0000 7f6c6be00700 1 -- 172.21.15.89:0/3375296739 >> v1:172.21.15.89:6789/0 conn(0x7f6c6c0d40f0 legacy=0x7f6c6c0d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.420+0000 7f6c6be00700 1 -- 172.21.15.89:0/3375296739 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c162100 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.420+0000 7f6c6aa00700 1 -- 172.21.15.89:0/3375296739 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728953480 0 0) 0x7f6c6c161eb0 con 0x7f6c6c0d40f0 2024-04-02T07:56:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.421+0000 7f6c6be00700 1 --2- 172.21.15.89:0/3375296739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c15d6d0 0x7f6c6c161430 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c009840 tx=0x7f6c5c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f262e568ac04bc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.421+0000 7f6c6aa00700 1 -- 172.21.15.89:0/3375296739 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c011830 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.421+0000 7f6c6aa00700 1 -- 172.21.15.89:0/3375296739 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5c011990 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.421+0000 7f6c6aa00700 1 -- 172.21.15.89:0/3375296739 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c011d20 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.421+0000 7f6c73200700 1 -- 172.21.15.89:0/3375296739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c15d6d0 msgr2=0x7f6c6c161430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.421+0000 7f6c73200700 1 --2- 172.21.15.89:0/3375296739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c15d6d0 0x7f6c6c161430 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c009840 tx=0x7f6c5c009b50 comp rx=0 tx=0).stop 2024-04-02T07:56:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.421+0000 7f6c73200700 1 -- 172.21.15.89:0/3375296739 shutdown_connections 2024-04-02T07:56:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.421+0000 7f6c73200700 1 --2- 172.21.15.89:0/3375296739 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c6c15dab0 0x7f6c6c161970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.421+0000 7f6c73200700 1 --2- 172.21.15.89:0/3375296739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c15d6d0 0x7f6c6c161430 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.421+0000 7f6c73200700 1 -- 172.21.15.89:0/3375296739 >> 172.21.15.89:0/3375296739 conn(0x7f6c6c08aff0 msgr2=0x7f6c6c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.422+0000 7f6c73200700 1 -- 172.21.15.89:0/3375296739 shutdown_connections 2024-04-02T07:56:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.422+0000 7f6c73200700 1 -- 172.21.15.89:0/3375296739 wait complete. 2024-04-02T07:56:04.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.423+0000 7f6c73200700 1 Processor -- start 2024-04-02T07:56:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.423+0000 7f6c73200700 1 -- start start 2024-04-02T07:56:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.424+0000 7f6c73200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c0d40f0 0x7f6c6c15a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.424+0000 7f6c73200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c15d6d0 0x7f6c6c06bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.424+0000 7f6c73200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c6c15dab0 0x7f6c6c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.424+0000 7f6c73200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0b8620 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.424+0000 7f6c73200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0cd410 con 0x7f6c6c0d40f0 2024-04-02T07:56:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.424+0000 7f6c73200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0b7a70 con 0x7f6c6c15dab0 2024-04-02T07:56:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.424+0000 7f6c6b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c15d6d0 0x7f6c6c06bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.424+0000 7f6c6b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c15d6d0 0x7f6c6c06bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54418/0 (socket says 172.21.15.89:54418) 2024-04-02T07:56:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.424+0000 7f6c6b400700 1 -- 172.21.15.89:0/3001034240 learned_addr learned my addr 172.21.15.89:0/3001034240 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.425+0000 7f6c6be00700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c0d40f0 0x7f6c6c15a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.425+0000 7f6c71200700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c6c15dab0 0x7f6c6c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.425+0000 7f6c6b400700 1 -- 172.21.15.89:0/3001034240 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c6c15dab0 msgr2=0x7f6c6c06c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:04.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.425+0000 7f6c6b400700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c6c15dab0 0x7f6c6c06c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:04.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.425+0000 7f6c6b400700 1 -- 172.21.15.89:0/3001034240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c0d40f0 msgr2=0x7f6c6c15a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:04.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.425+0000 7f6c6b400700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c0d40f0 0x7f6c6c15a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:04.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.425+0000 7f6c6b400700 1 -- 172.21.15.89:0/3001034240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c0094a0 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.425+0000 7f6c6be00700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c0d40f0 0x7f6c6c15a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:04.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.425+0000 7f6c6b400700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c15d6d0 0x7f6c6c06bef0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f6c580011e0 tx=0x7f6c5800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:04.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.425+0000 7f6c63e00700 1 -- 172.21.15.89:0/3001034240 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c58004c80 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.425+0000 7f6c63e00700 1 -- 172.21.15.89:0/3001034240 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c58004e00 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.425+0000 7f6c73200700 1 -- 172.21.15.89:0/3001034240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6c0b7c80 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.425+0000 7f6c63e00700 1 -- 172.21.15.89:0/3001034240 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c58007310 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.426+0000 7f6c73200700 1 -- 172.21.15.89:0/3001034240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6c0b8240 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.427+0000 7f6c73200700 1 -- 172.21.15.89:0/3001034240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6c0b8760 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.427+0000 7f6c63e00700 1 -- 172.21.15.89:0/3001034240 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 88) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f6c58007470 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.428+0000 7f6c63e00700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6c4c06aae0 0x7f6c4c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:04.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.428+0000 7f6c63e00700 1 -- 172.21.15.89:0/3001034240 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6816+0+0 (secure 0 0 0) 0x7f6c580891c0 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.428+0000 7f6c6be00700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6c4c06aae0 0x7f6c4c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:04.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.429+0000 7f6c6be00700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6c4c06aae0 0x7f6c4c06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0090b0 tx=0x7f6c5c008030 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:04.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.432+0000 7f6c63e00700 1 -- 172.21.15.89:0/3001034240 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c6c0b8760 con 0x7f6c6c15d6d0 2024-04-02T07:56:04.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:04.674+0000 7f6c73200700 1 -- 172.21.15.89:0/3001034240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6c6c0b8760 con 0x7f6c6c15d6d0 2024-04-02T07:56:05.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.211+0000 7f6c63e00700 1 -- 172.21.15.89:0/3001034240 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 89) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f6c580545b0 con 0x7f6c6c15d6d0 2024-04-02T07:56:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.217+0000 7f6c63e00700 1 -- 172.21.15.89:0/3001034240 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6c58073d50 con 0x7f6c6c15d6d0 2024-04-02T07:56:05.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.220+0000 7f6c73200700 1 -- 172.21.15.89:0/3001034240 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6c4c06aae0 msgr2=0x7f6c4c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:05.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.220+0000 7f6c73200700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6c4c06aae0 0x7f6c4c06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0090b0 tx=0x7f6c5c008030 comp rx=0 tx=0).stop 2024-04-02T07:56:05.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.220+0000 7f6c73200700 1 -- 172.21.15.89:0/3001034240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c15d6d0 msgr2=0x7f6c6c06bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:05.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.220+0000 7f6c73200700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c15d6d0 0x7f6c6c06bef0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f6c580011e0 tx=0x7f6c5800cae0 comp rx=0 tx=0).stop 2024-04-02T07:56:05.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.220+0000 7f6c73200700 1 -- 172.21.15.89:0/3001034240 shutdown_connections 2024-04-02T07:56:05.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.220+0000 7f6c73200700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6c4c06aae0 0x7f6c4c06cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:05.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.220+0000 7f6c73200700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c0d40f0 0x7f6c6c15a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:05.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.220+0000 7f6c73200700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c6c15dab0 0x7f6c6c06c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:05.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.220+0000 7f6c73200700 1 --2- 172.21.15.89:0/3001034240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c15d6d0 0x7f6c6c06bef0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:05.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.220+0000 7f6c73200700 1 -- 172.21.15.89:0/3001034240 >> 172.21.15.89:0/3001034240 conn(0x7f6c6c08aff0 msgr2=0x7f6c6c15ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:05.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.221+0000 7f6c73200700 1 -- 172.21.15.89:0/3001034240 shutdown_connections 2024-04-02T07:56:05.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.221+0000 7f6c73200700 1 -- 172.21.15.89:0/3001034240 wait complete. 2024-04-02T07:56:05.242 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T07:56:05.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.477+0000 7f3cb1800700 1 Processor -- start 2024-04-02T07:56:05.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.477+0000 7f3cb1800700 1 -- start start 2024-04-02T07:56:05.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.479+0000 7f3cb1800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3cac15ddf0 0x7f3cac0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:05.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.479+0000 7f3cb1800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3cac0d28e0 0x7f3cac15f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:05.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.479+0000 7f3cb1800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac0ce8a0 con 0x7f3cac15e1d0 2024-04-02T07:56:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.479+0000 7f3cb1800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cac15ec60 con 0x7f3cac15ddf0 2024-04-02T07:56:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.479+0000 7f3cb1800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac0cdc90 con 0x7f3cac0d28e0 2024-04-02T07:56:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.479+0000 7f3caa000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3cac15e1d0 0x7f3cac0d21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55598/0 (socket says 172.21.15.89:55598) 2024-04-02T07:56:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.479+0000 7f3caa000700 1 -- 172.21.15.89:0/4094453916 learned_addr learned my addr 172.21.15.89:0/4094453916 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.479+0000 7f3caaa00700 1 --2- 172.21.15.89:0/4094453916 >> v2:172.21.15.89:3300/0 conn(0x7f3cac15ddf0 0x7f3cac0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.479+0000 7f3cab400700 1 --2- 172.21.15.89:0/4094453916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3cac0d28e0 0x7f3cac15f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.479+0000 7f3caaa00700 1 -- 172.21.15.89:0/4094453916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3cac0d28e0 msgr2=0x7f3cac15f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.480+0000 7f3caaa00700 1 --2- 172.21.15.89:0/4094453916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3cac0d28e0 0x7f3cac15f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.480+0000 7f3caaa00700 1 -- 172.21.15.89:0/4094453916 >> v1:172.21.15.89:6789/0 conn(0x7f3cac15e1d0 legacy=0x7f3cac0d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.480+0000 7f3caaa00700 1 -- 172.21.15.89:0/4094453916 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac0cdf70 con 0x7f3cac15ddf0 2024-04-02T07:56:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.480+0000 7f3ca9600700 1 -- 172.21.15.89:0/4094453916 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846962241 0 0) 0x7f3cac0ce8a0 con 0x7f3cac15e1d0 2024-04-02T07:56:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.480+0000 7f3cab400700 1 --2- 172.21.15.89:0/4094453916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3cac0d28e0 0x7f3cac15f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.480+0000 7f3caaa00700 1 --2- 172.21.15.89:0/4094453916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cac15ddf0 0x7f3cac0ced30 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c009840 tx=0x7f3c9c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85538543475822ab server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:05.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.480+0000 7f3ca9600700 1 -- 172.21.15.89:0/4094453916 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c011830 con 0x7f3cac15ddf0 2024-04-02T07:56:05.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.480+0000 7f3ca9600700 1 -- 172.21.15.89:0/4094453916 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9c011990 con 0x7f3cac15ddf0 2024-04-02T07:56:05.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.480+0000 7f3ca9600700 1 -- 172.21.15.89:0/4094453916 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c011d20 con 0x7f3cac15ddf0 2024-04-02T07:56:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.481+0000 7f3cb1800700 1 -- 172.21.15.89:0/4094453916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cac15ddf0 msgr2=0x7f3cac0ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.481+0000 7f3cb1800700 1 --2- 172.21.15.89:0/4094453916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cac15ddf0 0x7f3cac0ced30 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c009840 tx=0x7f3c9c009b50 comp rx=0 tx=0).stop 2024-04-02T07:56:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.481+0000 7f3cb1800700 1 -- 172.21.15.89:0/4094453916 shutdown_connections 2024-04-02T07:56:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.481+0000 7f3cb1800700 1 --2- 172.21.15.89:0/4094453916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3cac0d28e0 0x7f3cac15f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.481+0000 7f3cb1800700 1 --2- 172.21.15.89:0/4094453916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cac15ddf0 0x7f3cac0ced30 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.481+0000 7f3cb1800700 1 -- 172.21.15.89:0/4094453916 >> 172.21.15.89:0/4094453916 conn(0x7f3cac08aff0 msgr2=0x7f3cac0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.481+0000 7f3cb1800700 1 -- 172.21.15.89:0/4094453916 shutdown_connections 2024-04-02T07:56:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.481+0000 7f3cb1800700 1 -- 172.21.15.89:0/4094453916 wait complete. 2024-04-02T07:56:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.482+0000 7f3cb1800700 1 Processor -- start 2024-04-02T07:56:05.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.483+0000 7f3cb1800700 1 -- start start 2024-04-02T07:56:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.483+0000 7f3cb1800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cac15ddf0 0x7f3cac0c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.483+0000 7f3cb1800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cac15e1d0 0x7f3cac0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.483+0000 7f3cb1800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3cac0c8600 0x7f3cac0b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:05.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.483+0000 7f3cb1800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac156f30 con 0x7f3cac15ddf0 2024-04-02T07:56:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.483+0000 7f3cb1800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac0c7020 con 0x7f3cac15e1d0 2024-04-02T07:56:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.483+0000 7f3cb1800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac0c7300 con 0x7f3cac0c8600 2024-04-02T07:56:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.484+0000 7f3caaa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cac15ddf0 0x7f3cac0c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.484+0000 7f3caaa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cac15ddf0 0x7f3cac0c80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54456/0 (socket says 172.21.15.89:54456) 2024-04-02T07:56:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.484+0000 7f3caaa00700 1 -- 172.21.15.89:0/2362740908 learned_addr learned my addr 172.21.15.89:0/2362740908 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.484+0000 7f3cab400700 1 --2- 172.21.15.89:0/2362740908 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3cac0c8600 0x7f3cac0b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.484+0000 7f3caa000700 1 --2- 172.21.15.89:0/2362740908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cac15e1d0 0x7f3cac0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.484+0000 7f3caaa00700 1 -- 172.21.15.89:0/2362740908 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3cac0c8600 msgr2=0x7f3cac0b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.484+0000 7f3caaa00700 1 --2- 172.21.15.89:0/2362740908 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3cac0c8600 0x7f3cac0b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.484+0000 7f3caaa00700 1 -- 172.21.15.89:0/2362740908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cac15e1d0 msgr2=0x7f3cac0b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.484+0000 7f3caaa00700 1 --2- 172.21.15.89:0/2362740908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cac15e1d0 0x7f3cac0b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.484+0000 7f3caaa00700 1 -- 172.21.15.89:0/2362740908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9c0094a0 con 0x7f3cac15ddf0 2024-04-02T07:56:05.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.484+0000 7f3caaa00700 1 --2- 172.21.15.89:0/2362740908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cac15ddf0 0x7f3cac0c80c0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c000c00 tx=0x7f3c9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:05.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.485+0000 7f3ca2a00700 1 -- 172.21.15.89:0/2362740908 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c0045f0 con 0x7f3cac15ddf0 2024-04-02T07:56:05.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.485+0000 7f3ca2a00700 1 -- 172.21.15.89:0/2362740908 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9c01f070 con 0x7f3cac15ddf0 2024-04-02T07:56:05.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.485+0000 7f3ca2a00700 1 -- 172.21.15.89:0/2362740908 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c003ae0 con 0x7f3cac15ddf0 2024-04-02T07:56:05.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.485+0000 7f3cb1800700 1 -- 172.21.15.89:0/2362740908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cac0c75e0 con 0x7f3cac15ddf0 2024-04-02T07:56:05.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.485+0000 7f3cb1800700 1 -- 172.21.15.89:0/2362740908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cac0c7ba0 con 0x7f3cac15ddf0 2024-04-02T07:56:05.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.486+0000 7f3cb1800700 1 -- 172.21.15.89:0/2362740908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cac067720 con 0x7f3cac15ddf0 2024-04-02T07:56:05.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.487+0000 7f3ca2a00700 1 -- 172.21.15.89:0/2362740908 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 89) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f3c9c003c40 con 0x7f3cac15ddf0 2024-04-02T07:56:05.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.488+0000 7f3ca2a00700 1 --2- 172.21.15.89:0/2362740908 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3c8c06ad20 0x7f3c8c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:05.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.488+0000 7f3ca2a00700 1 -- 172.21.15.89:0/2362740908 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6816+0+0 (secure 0 0 0) 0x7f3c9c021070 con 0x7f3cac15ddf0 2024-04-02T07:56:05.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.488+0000 7f3caa000700 1 --2- 172.21.15.89:0/2362740908 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3c8c06ad20 0x7f3c8c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:05.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.488+0000 7f3caa000700 1 --2- 172.21.15.89:0/2362740908 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3c8c06ad20 0x7f3c8c06d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3c98000f60 tx=0x7f3c98007be0 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:05.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.492+0000 7f3ca2a00700 1 -- 172.21.15.89:0/2362740908 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3c9c056030 con 0x7f3cac15ddf0 2024-04-02T07:56:05.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:05.693+0000 7f3cb1800700 1 -- 172.21.15.89:0/2362740908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3cac0b0b80 con 0x7f3cac15ddf0 2024-04-02T07:56:06.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.223+0000 7f3ca2a00700 1 -- 172.21.15.89:0/2362740908 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v144) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3c9c0150d0 con 0x7f3cac15ddf0 2024-04-02T07:56:06.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.226+0000 7f3cb1800700 1 -- 172.21.15.89:0/2362740908 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3c8c06ad20 msgr2=0x7f3c8c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:06.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.226+0000 7f3cb1800700 1 --2- 172.21.15.89:0/2362740908 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3c8c06ad20 0x7f3c8c06d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3c98000f60 tx=0x7f3c98007be0 comp rx=0 tx=0).stop 2024-04-02T07:56:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.227+0000 7f3cb1800700 1 -- 172.21.15.89:0/2362740908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cac15ddf0 msgr2=0x7f3cac0c80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.227+0000 7f3cb1800700 1 --2- 172.21.15.89:0/2362740908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cac15ddf0 0x7f3cac0c80c0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c000c00 tx=0x7f3c9c004750 comp rx=0 tx=0).stop 2024-04-02T07:56:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.227+0000 7f3cb1800700 1 -- 172.21.15.89:0/2362740908 shutdown_connections 2024-04-02T07:56:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.227+0000 7f3cb1800700 1 --2- 172.21.15.89:0/2362740908 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3c8c06ad20 0x7f3c8c06d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.227+0000 7f3cb1800700 1 --2- 172.21.15.89:0/2362740908 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3cac0c8600 0x7f3cac0b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.227+0000 7f3cb1800700 1 --2- 172.21.15.89:0/2362740908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cac15ddf0 0x7f3cac0c80c0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.227+0000 7f3cb1800700 1 --2- 172.21.15.89:0/2362740908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cac15e1d0 0x7f3cac0b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.227+0000 7f3cb1800700 1 -- 172.21.15.89:0/2362740908 >> 172.21.15.89:0/2362740908 conn(0x7f3cac08aff0 msgr2=0x7f3cac0b66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.227+0000 7f3cb1800700 1 -- 172.21.15.89:0/2362740908 shutdown_connections 2024-04-02T07:56:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.227+0000 7f3cb1800700 1 -- 172.21.15.89:0/2362740908 wait complete. 2024-04-02T07:56:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T07:56:06.247 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T07:56:06.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.441+0000 7fd100c00700 1 Processor -- start 2024-04-02T07:56:06.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.441+0000 7fd100c00700 1 -- start start 2024-04-02T07:56:06.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.443+0000 7fd100c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0fc0b3930 0x7fd0fc09a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:06.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.443+0000 7fd100c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0fc0cc280 0x7fd0fc0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:06.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.443+0000 7fd100c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0fc0cbdf0 con 0x7fd0fc15db10 2024-04-02T07:56:06.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.443+0000 7fd100c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0fc06bec0 con 0x7fd0fc0cc280 2024-04-02T07:56:06.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.443+0000 7fd100c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0b27a0 con 0x7fd0fc0b3930 2024-04-02T07:56:06.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.443+0000 7fd0fa800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0fc0cc280 0x7fd0fc0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:06.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.443+0000 7fd0fa800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0fc0cc280 0x7fd0fc0cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54462/0 (socket says 172.21.15.89:54462) 2024-04-02T07:56:06.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.443+0000 7fd0fa800700 1 -- 172.21.15.89:0/893653142 learned_addr learned my addr 172.21.15.89:0/893653142 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:06.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.444+0000 7fd0f9400700 1 --2- 172.21.15.89:0/893653142 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0fc0b3930 0x7fd0fc09a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:06.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.444+0000 7fd0fa800700 1 -- 172.21.15.89:0/893653142 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0fc0b3930 msgr2=0x7fd0fc09a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:06.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.444+0000 7fd0fa800700 1 --2- 172.21.15.89:0/893653142 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0fc0b3930 0x7fd0fc09a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:06.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.444+0000 7fd0fa800700 1 -- 172.21.15.89:0/893653142 >> v1:172.21.15.113:6789/0 conn(0x7fd0fc15db10 legacy=0x7fd0fc099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:06.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.444+0000 7fd0fa800700 1 -- 172.21.15.89:0/893653142 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc06bbe0 con 0x7fd0fc0cc280 2024-04-02T07:56:06.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.444+0000 7fd0fa800700 1 --2- 172.21.15.89:0/893653142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0cc280 0x7fd0fc0cc710 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009e50 tx=0x7fd0f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aaee370d4f0ff19 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:06.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.444+0000 7fd0f8a00700 1 -- 172.21.15.89:0/893653142 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f400e070 con 0x7fd0fc0cc280 2024-04-02T07:56:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.444+0000 7fd0f8a00700 1 -- 172.21.15.89:0/893653142 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f4003a20 con 0x7fd0fc0cc280 2024-04-02T07:56:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.444+0000 7fd0f8a00700 1 -- 172.21.15.89:0/893653142 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4003dd0 con 0x7fd0fc0cc280 2024-04-02T07:56:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.445+0000 7fd100c00700 1 -- 172.21.15.89:0/893653142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0cc280 msgr2=0x7fd0fc0cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.445+0000 7fd100c00700 1 --2- 172.21.15.89:0/893653142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0cc280 0x7fd0fc0cc710 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009e50 tx=0x7fd0f400b330 comp rx=0 tx=0).stop 2024-04-02T07:56:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.445+0000 7fd100c00700 1 -- 172.21.15.89:0/893653142 shutdown_connections 2024-04-02T07:56:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.445+0000 7fd100c00700 1 --2- 172.21.15.89:0/893653142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0cc280 0x7fd0fc0cc710 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.445+0000 7fd100c00700 1 --2- 172.21.15.89:0/893653142 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0fc0b3930 0x7fd0fc09a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.445+0000 7fd100c00700 1 -- 172.21.15.89:0/893653142 >> 172.21.15.89:0/893653142 conn(0x7fd0fc0b2e10 msgr2=0x7fd0fc0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.445+0000 7fd100c00700 1 -- 172.21.15.89:0/893653142 shutdown_connections 2024-04-02T07:56:06.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.445+0000 7fd100c00700 1 -- 172.21.15.89:0/893653142 wait complete. 2024-04-02T07:56:06.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.447+0000 7fd100c00700 1 Processor -- start 2024-04-02T07:56:06.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.447+0000 7fd100c00700 1 -- start start 2024-04-02T07:56:06.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd100c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0b3930 0x7fd0fc150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:06.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd100c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0fc0cc280 0x7fd0fc0c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:06.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd100c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0fc15db10 0x7fd0fc0c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:06.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd100c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0c9130 con 0x7fd0fc0b3930 2024-04-02T07:56:06.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd100c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0c8520 con 0x7fd0fc15db10 2024-04-02T07:56:06.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd100c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0c8800 con 0x7fd0fc0cc280 2024-04-02T07:56:06.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd0f9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0b3930 0x7fd0fc150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:06.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd0f9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0b3930 0x7fd0fc150840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54478/0 (socket says 172.21.15.89:54478) 2024-04-02T07:56:06.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd0f9e00700 1 -- 172.21.15.89:0/3250188948 learned_addr learned my addr 172.21.15.89:0/3250188948 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:06.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd0f9400700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0fc0cc280 0x7fd0fc0c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:06.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd0fa800700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0fc15db10 0x7fd0fc0c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:06.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd0f9e00700 1 -- 172.21.15.89:0/3250188948 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0fc0cc280 msgr2=0x7fd0fc0c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:06.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd0f9e00700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0fc0cc280 0x7fd0fc0c9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:06.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd0f9e00700 1 -- 172.21.15.89:0/3250188948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0fc15db10 msgr2=0x7fd0fc0c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:06.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd0f9e00700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0fc15db10 0x7fd0fc0c99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:06.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd0f9e00700 1 -- 172.21.15.89:0/3250188948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f4009b30 con 0x7fd0fc0b3930 2024-04-02T07:56:06.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.448+0000 7fd0fa800700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0fc15db10 0x7fd0fc0c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:06.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.449+0000 7fd0f9e00700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0b3930 0x7fd0fc150840 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00b4a0 tx=0x7fd0ec00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:06.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.449+0000 7fd0f2000700 1 -- 172.21.15.89:0/3250188948 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec011830 con 0x7fd0fc0b3930 2024-04-02T07:56:06.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.449+0000 7fd0f2000700 1 -- 172.21.15.89:0/3250188948 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ec011990 con 0x7fd0fc0b3930 2024-04-02T07:56:06.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.449+0000 7fd100c00700 1 -- 172.21.15.89:0/3250188948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0fc0c8ae0 con 0x7fd0fc0b3930 2024-04-02T07:56:06.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.449+0000 7fd0f2000700 1 -- 172.21.15.89:0/3250188948 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec011d40 con 0x7fd0fc0b3930 2024-04-02T07:56:06.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.449+0000 7fd100c00700 1 -- 172.21.15.89:0/3250188948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0fc150d80 con 0x7fd0fc0b3930 2024-04-02T07:56:06.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.450+0000 7fd100c00700 1 -- 172.21.15.89:0/3250188948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0fc067720 con 0x7fd0fc0b3930 2024-04-02T07:56:06.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.451+0000 7fd0f2000700 1 -- 172.21.15.89:0/3250188948 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 89) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fd0ec01a400 con 0x7fd0fc0b3930 2024-04-02T07:56:06.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.453+0000 7fd0f2000700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd0dc06ad20 0x7fd0dc06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:06.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.453+0000 7fd0f2000700 1 -- 172.21.15.89:0/3250188948 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fd0ec089390 con 0x7fd0fc0b3930 2024-04-02T07:56:06.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.453+0000 7fd0f9400700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd0dc06ad20 0x7fd0dc06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:06.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.454+0000 7fd0f9400700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd0dc06ad20 0x7fd0dc06d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc15c670 tx=0x7fd0e8008040 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:06.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.457+0000 7fd0f2000700 1 -- 172.21.15.89:0/3250188948 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd0ec054320 con 0x7fd0fc0b3930 2024-04-02T07:56:06.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:06.661+0000 7fd100c00700 1 -- 172.21.15.89:0/3250188948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd0fc06c090 con 0x7fd0fc0b3930 2024-04-02T07:56:07.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.237+0000 7fd0f2000700 1 -- 172.21.15.89:0/3250188948 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd0ec0546f0 con 0x7fd0fc0b3930 2024-04-02T07:56:07.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.238+0000 7fd0f2000700 1 -- 172.21.15.89:0/3250188948 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v145) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd0ec050f40 con 0x7fd0fc0b3930 2024-04-02T07:56:07.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.242+0000 7fd100c00700 1 -- 172.21.15.89:0/3250188948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd0dc06ad20 msgr2=0x7fd0dc06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:07.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.242+0000 7fd100c00700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd0dc06ad20 0x7fd0dc06d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc15c670 tx=0x7fd0e8008040 comp rx=0 tx=0).stop 2024-04-02T07:56:07.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.242+0000 7fd100c00700 1 -- 172.21.15.89:0/3250188948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0b3930 msgr2=0x7fd0fc150840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:07.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.242+0000 7fd100c00700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0b3930 0x7fd0fc150840 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00b4a0 tx=0x7fd0ec00b860 comp rx=0 tx=0).stop 2024-04-02T07:56:07.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.242+0000 7fd100c00700 1 -- 172.21.15.89:0/3250188948 shutdown_connections 2024-04-02T07:56:07.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.242+0000 7fd100c00700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd0dc06ad20 0x7fd0dc06d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.242+0000 7fd100c00700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0fc15db10 0x7fd0fc0c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.242+0000 7fd100c00700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0b3930 0x7fd0fc150840 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.242+0000 7fd100c00700 1 --2- 172.21.15.89:0/3250188948 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0fc0cc280 0x7fd0fc0c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.242+0000 7fd100c00700 1 -- 172.21.15.89:0/3250188948 >> 172.21.15.89:0/3250188948 conn(0x7fd0fc0b2e10 msgr2=0x7fd0fc070300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:07.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.242+0000 7fd100c00700 1 -- 172.21.15.89:0/3250188948 shutdown_connections 2024-04-02T07:56:07.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.243+0000 7fd100c00700 1 -- 172.21.15.89:0/3250188948 wait complete. 2024-04-02T07:56:07.244 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T07:56:07.261 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T07:56:07.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.453+0000 7fa42ac00700 1 Processor -- start 2024-04-02T07:56:07.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.453+0000 7fa42ac00700 1 -- start start 2024-04-02T07:56:07.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.455+0000 7fa42ac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4240cbb30 0x7fa4240cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:07.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.455+0000 7fa42ac00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa4240cd4a0 0x7fa42406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:07.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.455+0000 7fa42ac00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4240cc450 con 0x7fa4240cd020 2024-04-02T07:56:07.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.455+0000 7fa42ac00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4240cdae0 con 0x7fa4240cd4a0 2024-04-02T07:56:07.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.455+0000 7fa42ac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4240cddc0 con 0x7fa4240cbb30 2024-04-02T07:56:07.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.455+0000 7fa428c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa4240cd4a0 0x7fa42406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:07.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.455+0000 7fa428c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa4240cd4a0 0x7fa42406e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54488/0 (socket says 172.21.15.89:54488) 2024-04-02T07:56:07.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.455+0000 7fa428c00700 1 -- 172.21.15.89:0/1005295982 learned_addr learned my addr 172.21.15.89:0/1005295982 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:07.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.455+0000 7fa423e00700 1 --2- 172.21.15.89:0/1005295982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4240cbb30 0x7fa4240cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:07.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.455+0000 7fa428c00700 1 -- 172.21.15.89:0/1005295982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4240cbb30 msgr2=0x7fa4240cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:07.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.455+0000 7fa428c00700 1 --2- 172.21.15.89:0/1005295982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4240cbb30 0x7fa4240cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.455+0000 7fa428c00700 1 -- 172.21.15.89:0/1005295982 >> v1:172.21.15.113:6789/0 conn(0x7fa4240cd020 legacy=0x7fa4240b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:07.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.455+0000 7fa428c00700 1 -- 172.21.15.89:0/1005295982 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4240ce0a0 con 0x7fa4240cd4a0 2024-04-02T07:56:07.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.456+0000 7fa428c00700 1 --2- 172.21.15.89:0/1005295982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4240cd4a0 0x7fa42406e420 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009e50 tx=0x7fa41c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12f51e46e985c7d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:07.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.456+0000 7fa423400700 1 -- 172.21.15.89:0/1005295982 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c00e070 con 0x7fa4240cd4a0 2024-04-02T07:56:07.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.456+0000 7fa423400700 1 -- 172.21.15.89:0/1005295982 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa41c00b9c0 con 0x7fa4240cd4a0 2024-04-02T07:56:07.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.456+0000 7fa423400700 1 -- 172.21.15.89:0/1005295982 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c00bd70 con 0x7fa4240cd4a0 2024-04-02T07:56:07.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.456+0000 7fa42ac00700 1 -- 172.21.15.89:0/1005295982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4240cd4a0 msgr2=0x7fa42406e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:07.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.456+0000 7fa42ac00700 1 --2- 172.21.15.89:0/1005295982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4240cd4a0 0x7fa42406e420 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009e50 tx=0x7fa41c00b330 comp rx=0 tx=0).stop 2024-04-02T07:56:07.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.456+0000 7fa42ac00700 1 -- 172.21.15.89:0/1005295982 shutdown_connections 2024-04-02T07:56:07.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.456+0000 7fa42ac00700 1 --2- 172.21.15.89:0/1005295982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4240cbb30 0x7fa4240cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.456+0000 7fa42ac00700 1 --2- 172.21.15.89:0/1005295982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4240cd4a0 0x7fa42406e420 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.456+0000 7fa42ac00700 1 -- 172.21.15.89:0/1005295982 >> 172.21.15.89:0/1005295982 conn(0x7fa4240b0d70 msgr2=0x7fa4240b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:07.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.457+0000 7fa42ac00700 1 -- 172.21.15.89:0/1005295982 shutdown_connections 2024-04-02T07:56:07.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.457+0000 7fa42ac00700 1 -- 172.21.15.89:0/1005295982 wait complete. 2024-04-02T07:56:07.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.458+0000 7fa42ac00700 1 Processor -- start 2024-04-02T07:56:07.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.458+0000 7fa42ac00700 1 -- start start 2024-04-02T07:56:07.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.459+0000 7fa42ac00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4240cbb30 0x7fa4240accb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:07.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.459+0000 7fa42ac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4240cd020 0x7fa42414ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:07.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.459+0000 7fa42ac00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4240cd4a0 0x7fa4240ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:07.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.459+0000 7fa42ac00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42414f3d0 con 0x7fa4240cd4a0 2024-04-02T07:56:07.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.459+0000 7fa42ac00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4240add00 con 0x7fa4240cbb30 2024-04-02T07:56:07.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.459+0000 7fa42ac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4240adf30 con 0x7fa4240cd020 2024-04-02T07:56:07.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.460+0000 7fa428c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4240cd4a0 0x7fa4240ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:07.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.460+0000 7fa428c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4240cd4a0 0x7fa4240ad7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54500/0 (socket says 172.21.15.89:54500) 2024-04-02T07:56:07.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.460+0000 7fa428c00700 1 -- 172.21.15.89:0/1293763806 learned_addr learned my addr 172.21.15.89:0/1293763806 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:07.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.460+0000 7fa423e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4240cbb30 0x7fa4240accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:07.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.460+0000 7fa41be00700 1 --2- 172.21.15.89:0/1293763806 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4240cd020 0x7fa42414ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:07.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.460+0000 7fa428c00700 1 -- 172.21.15.89:0/1293763806 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4240cd020 msgr2=0x7fa42414ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:07.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.460+0000 7fa428c00700 1 --2- 172.21.15.89:0/1293763806 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4240cd020 0x7fa42414ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.460+0000 7fa428c00700 1 -- 172.21.15.89:0/1293763806 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4240cbb30 msgr2=0x7fa4240accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:07.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.460+0000 7fa428c00700 1 --2- 172.21.15.89:0/1293763806 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4240cbb30 0x7fa4240accb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.460+0000 7fa428c00700 1 -- 172.21.15.89:0/1293763806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c009b30 con 0x7fa4240cd4a0 2024-04-02T07:56:07.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.460+0000 7fa428c00700 1 --2- 172.21.15.89:0/1293763806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4240cd4a0 0x7fa4240ad7c0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fa41c00b730 tx=0x7fa41c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:07.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.461+0000 7fa420c00700 1 -- 172.21.15.89:0/1293763806 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c004830 con 0x7fa4240cd4a0 2024-04-02T07:56:07.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.461+0000 7fa420c00700 1 -- 172.21.15.89:0/1293763806 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa41c0099b0 con 0x7fa4240cd4a0 2024-04-02T07:56:07.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.461+0000 7fa420c00700 1 -- 172.21.15.89:0/1293763806 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c01c680 con 0x7fa4240cd4a0 2024-04-02T07:56:07.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.461+0000 7fa42ac00700 1 -- 172.21.15.89:0/1293763806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa424070a60 con 0x7fa4240cd4a0 2024-04-02T07:56:07.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.461+0000 7fa42ac00700 1 -- 172.21.15.89:0/1293763806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa424071020 con 0x7fa4240cd4a0 2024-04-02T07:56:07.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.462+0000 7fa42ac00700 1 -- 172.21.15.89:0/1293763806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4240ce540 con 0x7fa4240cd4a0 2024-04-02T07:56:07.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.463+0000 7fa420c00700 1 -- 172.21.15.89:0/1293763806 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa41c01c7e0 con 0x7fa4240cd4a0 2024-04-02T07:56:07.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.463+0000 7fa420c00700 1 --2- 172.21.15.89:0/1293763806 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa40006adb0 0x7fa40006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:07.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.464+0000 7fa420c00700 1 -- 172.21.15.89:0/1293763806 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5964+0+0 (secure 0 0 0) 0x7fa41c08b2c0 con 0x7fa4240cd4a0 2024-04-02T07:56:07.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.464+0000 7fa423e00700 1 --2- 172.21.15.89:0/1293763806 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa40006adb0 0x7fa40006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:07.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.464+0000 7fa423e00700 1 --2- 172.21.15.89:0/1293763806 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa40006adb0 0x7fa40006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa414007a30 tx=0x7fa414008040 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:07.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.468+0000 7fa420c00700 1 -- 172.21.15.89:0/1293763806 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa41c056170 con 0x7fa4240cd4a0 2024-04-02T07:56:07.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.670+0000 7fa42ac00700 1 -- 172.21.15.89:0/1293763806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fa424067720 con 0x7fa4240cd4a0 2024-04-02T07:56:07.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.675+0000 7fa420c00700 1 -- 172.21.15.89:0/1293763806 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v49) v1 ==== 115+0+0 (secure 0 0 0) 0x7fa41c059790 con 0x7fa4240cd4a0 2024-04-02T07:56:07.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.677+0000 7fa42ac00700 1 -- 172.21.15.89:0/1293763806 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa40006adb0 msgr2=0x7fa40006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:07.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.677+0000 7fa42ac00700 1 --2- 172.21.15.89:0/1293763806 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa40006adb0 0x7fa40006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa414007a30 tx=0x7fa414008040 comp rx=0 tx=0).stop 2024-04-02T07:56:07.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.677+0000 7fa42ac00700 1 -- 172.21.15.89:0/1293763806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4240cd4a0 msgr2=0x7fa4240ad7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:07.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.677+0000 7fa42ac00700 1 --2- 172.21.15.89:0/1293763806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4240cd4a0 0x7fa4240ad7c0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fa41c00b730 tx=0x7fa41c004ad0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.677+0000 7fa42ac00700 1 -- 172.21.15.89:0/1293763806 shutdown_connections 2024-04-02T07:56:07.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.677+0000 7fa42ac00700 1 --2- 172.21.15.89:0/1293763806 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa40006adb0 0x7fa40006d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.677+0000 7fa42ac00700 1 --2- 172.21.15.89:0/1293763806 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4240cbb30 0x7fa4240accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.677+0000 7fa42ac00700 1 --2- 172.21.15.89:0/1293763806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4240cd4a0 0x7fa4240ad7c0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.677+0000 7fa42ac00700 1 --2- 172.21.15.89:0/1293763806 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4240cd020 0x7fa42414ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.678+0000 7fa42ac00700 1 -- 172.21.15.89:0/1293763806 >> 172.21.15.89:0/1293763806 conn(0x7fa4240b0d70 msgr2=0x7fa42406e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:07.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.678+0000 7fa42ac00700 1 -- 172.21.15.89:0/1293763806 shutdown_connections 2024-04-02T07:56:07.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.678+0000 7fa42ac00700 1 -- 172.21.15.89:0/1293763806 wait complete. 2024-04-02T07:56:07.696 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T07:56:07.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.889+0000 7f65b7c00700 1 Processor -- start 2024-04-02T07:56:07.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.890+0000 7f65b7c00700 1 -- start start 2024-04-02T07:56:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.891+0000 7f65b7c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f65b00d2540 0x7f65b01687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.891+0000 7f65b7c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b0161420 con 0x7f65b015d9d0 2024-04-02T07:56:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.891+0000 7f65b7c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b0161700 con 0x7f65b00d16a0 2024-04-02T07:56:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.891+0000 7f65b7c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b00d4340 con 0x7f65b00d2540 2024-04-02T07:56:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.891+0000 7f65b5200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f65b015d9d0 0x7f65b00d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55614/0 (socket says 172.21.15.89:55614) 2024-04-02T07:56:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.891+0000 7f65b5200700 1 -- 172.21.15.89:0/340011011 learned_addr learned my addr 172.21.15.89:0/340011011 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.891+0000 7f65b5c00700 1 --2- 172.21.15.89:0/340011011 >> v2:172.21.15.113:3300/0 conn(0x7f65b00d2540 0x7f65b01687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.891+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773763922 0 0) 0x7f65b0161420 con 0x7f65b015d9d0 2024-04-02T07:56:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.891+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65980035b0 con 0x7f65b015d9d0 2024-04-02T07:56:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021756048 0 0) 0x7f65b0161700 con 0x7f65b00d16a0 2024-04-02T07:56:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b0161420 con 0x7f65b00d16a0 2024-04-02T07:56:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3584789800 0 0) 0x7f65980035b0 con 0x7f65b015d9d0 2024-04-02T07:56:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 >> v2:172.21.15.113:3300/0 conn(0x7f65b00d2540 msgr2=0x7f65b01687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 --2- 172.21.15.89:0/340011011 >> v2:172.21.15.113:3300/0 conn(0x7f65b00d2540 0x7f65b01687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 >> v1:172.21.15.113:6789/0 conn(0x7f65b00d16a0 legacy=0x7f65b00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b01619e0 con 0x7f65b015d9d0 2024-04-02T07:56:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65b5c00700 1 --2- 172.21.15.89:0/340011011 >> v2:172.21.15.113:3300/0 conn(0x7f65b00d2540 0x7f65b01687b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:56:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f65a4001670 con 0x7f65b015d9d0 2024-04-02T07:56:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 >> v1:172.21.15.89:6789/0 conn(0x7f65b015d9d0 legacy=0x7f65b00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 --2- 172.21.15.89:0/340011011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6598003b90 0x7f6598003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 --2- 172.21.15.89:0/340011011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6598004850 0x7f6598004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 --2- 172.21.15.89:0/340011011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6598005210 0x7f659800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659800c110 con 0x7f6598004850 2024-04-02T07:56:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659800c280 con 0x7f6598005210 2024-04-02T07:56:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659800c440 con 0x7f6598003b90 2024-04-02T07:56:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65afe00700 1 --2- 172.21.15.89:0/340011011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6598004850 0x7f6598004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65b5200700 1 --2- 172.21.15.89:0/340011011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6598003b90 0x7f6598003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65b5c00700 1 --2- 172.21.15.89:0/340011011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6598005210 0x7f659800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65afe00700 1 -- 172.21.15.89:0/340011011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6598003b90 msgr2=0x7f6598003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65afe00700 1 --2- 172.21.15.89:0/340011011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6598003b90 0x7f6598003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65afe00700 1 -- 172.21.15.89:0/340011011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6598005210 msgr2=0x7f659800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65afe00700 1 --2- 172.21.15.89:0/340011011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6598005210 0x7f659800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65afe00700 1 -- 172.21.15.89:0/340011011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65980035b0 con 0x7f6598004850 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65b5c00700 1 --2- 172.21.15.89:0/340011011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6598005210 0x7f659800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.892+0000 7f65b5c00700 1 -- 172.21.15.89:0/340011011 reap_dead start 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.893+0000 7f65afe00700 1 --2- 172.21.15.89:0/340011011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6598004850 0x7f6598004cd0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f659c001970 tx=0x7f6598005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2315338f7d83fd44 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.893+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c001460 con 0x7f6598004850 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.893+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659c0052d0 con 0x7f6598004850 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.893+0000 7f65af400700 1 -- 172.21.15.89:0/340011011 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c001da0 con 0x7f6598004850 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.893+0000 7f65b7c00700 1 -- 172.21.15.89:0/340011011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6598004850 msgr2=0x7f6598004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.893+0000 7f65b7c00700 1 --2- 172.21.15.89:0/340011011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6598004850 0x7f6598004cd0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f659c001970 tx=0x7f6598005880 comp rx=0 tx=0).stop 2024-04-02T07:56:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.893+0000 7f65b7c00700 1 -- 172.21.15.89:0/340011011 shutdown_connections 2024-04-02T07:56:07.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.893+0000 7f65b7c00700 1 --2- 172.21.15.89:0/340011011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6598004850 0x7f6598004cd0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.893+0000 7f65b7c00700 1 -- 172.21.15.89:0/340011011 >> 172.21.15.89:0/340011011 conn(0x7f65b008aff0 msgr2=0x7f65b00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:07.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.894+0000 7f65b7c00700 1 -- 172.21.15.89:0/340011011 shutdown_connections 2024-04-02T07:56:07.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.894+0000 7f65b7c00700 1 -- 172.21.15.89:0/340011011 wait complete. 2024-04-02T07:56:07.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.895+0000 7f65b7c00700 1 Processor -- start 2024-04-02T07:56:07.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.895+0000 7f65b7c00700 1 -- start start 2024-04-02T07:56:07.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.896+0000 7f65b7c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6598004850 0x7f65b014fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:07.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.896+0000 7f65b7c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65b01605d0 0x7f65b015f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:07.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.896+0000 7f65b7c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65b00b7f40 0x7f65b00b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:07.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.896+0000 7f65b7c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b00b7840 con 0x7f6598004850 2024-04-02T07:56:07.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.896+0000 7f65b7c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b00b8930 con 0x7f65b00b7f40 2024-04-02T07:56:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.896+0000 7f65b7c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b015ecd0 con 0x7f65b01605d0 2024-04-02T07:56:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.896+0000 7f65b5200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6598004850 0x7f65b014fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65b5200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6598004850 0x7f65b014fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54522/0 (socket says 172.21.15.89:54522) 2024-04-02T07:56:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65b5200700 1 -- 172.21.15.89:0/4210057445 learned_addr learned my addr 172.21.15.89:0/4210057445 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65afe00700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65b01605d0 0x7f65b015f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65b5c00700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65b00b7f40 0x7f65b00b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65b5200700 1 -- 172.21.15.89:0/4210057445 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65b01605d0 msgr2=0x7f65b015f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65b5200700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65b01605d0 0x7f65b015f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65b5200700 1 -- 172.21.15.89:0/4210057445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65b00b7f40 msgr2=0x7f65b00b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65b5200700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65b00b7f40 0x7f65b00b83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65b5200700 1 -- 172.21.15.89:0/4210057445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b00b7280 con 0x7f6598004850 2024-04-02T07:56:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65b5c00700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65b00b7f40 0x7f65b00b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65b5200700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6598004850 0x7f65b014fa80 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f65a400e810 tx=0x7f65a400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65acc00700 1 -- 172.21.15.89:0/4210057445 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a4012590 con 0x7f6598004850 2024-04-02T07:56:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65acc00700 1 -- 172.21.15.89:0/4210057445 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a4012710 con 0x7f6598004850 2024-04-02T07:56:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65acc00700 1 -- 172.21.15.89:0/4210057445 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a4012ac0 con 0x7f6598004850 2024-04-02T07:56:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.897+0000 7f65b7c00700 1 -- 172.21.15.89:0/4210057445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b00b7560 con 0x7f6598004850 2024-04-02T07:56:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.898+0000 7f65b7c00700 1 -- 172.21.15.89:0/4210057445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b0159d80 con 0x7f6598004850 2024-04-02T07:56:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.899+0000 7f65b7c00700 1 -- 172.21.15.89:0/4210057445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b015e7c0 con 0x7f6598004850 2024-04-02T07:56:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.900+0000 7f65acc00700 1 -- 172.21.15.89:0/4210057445 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f65a4012c20 con 0x7f6598004850 2024-04-02T07:56:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.901+0000 7f65acc00700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f659006adb0 0x7f659006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.901+0000 7f65acc00700 1 -- 172.21.15.89:0/4210057445 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5964+0+0 (secure 0 0 0) 0x7f65a40892b0 con 0x7f6598004850 2024-04-02T07:56:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.901+0000 7f65afe00700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f659006adb0 0x7f659006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:07.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.902+0000 7f65afe00700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f659006adb0 0x7f659006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f659c000c00 tx=0x7f659c00a000 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:07.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:07.905+0000 7f65acc00700 1 -- 172.21.15.89:0/4210057445 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f65b015e7c0 con 0x7f6598004850 2024-04-02T07:56:08.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.106+0000 7f65b7c00700 1 -- 172.21.15.89:0/4210057445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f65b015e7c0 con 0x7f6598004850 2024-04-02T07:56:08.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.112+0000 7f65acc00700 1 -- 172.21.15.89:0/4210057445 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v50) v1 ==== 120+0+0 (secure 0 0 0) 0x7f65b015e7c0 con 0x7f6598004850 2024-04-02T07:56:08.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.114+0000 7f65b7c00700 1 -- 172.21.15.89:0/4210057445 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f659006adb0 msgr2=0x7f659006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:08.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.114+0000 7f65b7c00700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f659006adb0 0x7f659006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f659c000c00 tx=0x7f659c00a000 comp rx=0 tx=0).stop 2024-04-02T07:56:08.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.114+0000 7f65b7c00700 1 -- 172.21.15.89:0/4210057445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6598004850 msgr2=0x7f65b014fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:08.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.114+0000 7f65b7c00700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6598004850 0x7f65b014fa80 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f65a400e810 tx=0x7f65a400ebd0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.115+0000 7f65b7c00700 1 -- 172.21.15.89:0/4210057445 shutdown_connections 2024-04-02T07:56:08.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.115+0000 7f65b7c00700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f659006adb0 0x7f659006d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.115+0000 7f65b7c00700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65b00b7f40 0x7f65b00b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.115+0000 7f65b7c00700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6598004850 0x7f65b014fa80 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.115+0000 7f65b7c00700 1 --2- 172.21.15.89:0/4210057445 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65b01605d0 0x7f65b015f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.115+0000 7f65b7c00700 1 -- 172.21.15.89:0/4210057445 >> 172.21.15.89:0/4210057445 conn(0x7f65b008aff0 msgr2=0x7f65b00ce360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:08.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.115+0000 7f65b7c00700 1 -- 172.21.15.89:0/4210057445 shutdown_connections 2024-04-02T07:56:08.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.115+0000 7f65b7c00700 1 -- 172.21.15.89:0/4210057445 wait complete. 2024-04-02T07:56:08.134 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T07:56:08.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.329+0000 7fe3c9800700 1 Processor -- start 2024-04-02T07:56:08.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.329+0000 7fe3c9800700 1 -- start start 2024-04-02T07:56:08.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.330+0000 7fe3c9800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe3c415d9d0 0x7fe3c40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.330+0000 7fe3c9800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3c40d2540 0x7fe3c40ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.330+0000 7fe3c9800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c4161420 con 0x7fe3c40d16a0 2024-04-02T07:56:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.330+0000 7fe3c9800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c415edc0 con 0x7fe3c415d9d0 2024-04-02T07:56:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.330+0000 7fe3c9800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c4161700 con 0x7fe3c40d2540 2024-04-02T07:56:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.331+0000 7fe3c2a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe3c415d9d0 0x7fe3c40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.331+0000 7fe3c3400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3c40d2540 0x7fe3c40ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.331+0000 7fe3c2a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe3c415d9d0 0x7fe3c40d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:39774/0 (socket says 172.21.15.89:39774) 2024-04-02T07:56:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.331+0000 7fe3c2a00700 1 -- 172.21.15.89:0/55846636 learned_addr learned my addr 172.21.15.89:0/55846636 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.331+0000 7fe3c2000700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fe3c40d16a0 0x7fe3c40d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:60828/0 (socket says 172.21.15.89:60828) 2024-04-02T07:56:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.331+0000 7fe3c2a00700 1 -- 172.21.15.89:0/55846636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3c40d2540 msgr2=0x7fe3c40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.331+0000 7fe3c2a00700 1 --2- 172.21.15.89:0/55846636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3c40d2540 0x7fe3c40ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.331+0000 7fe3c2a00700 1 -- 172.21.15.89:0/55846636 >> v1:172.21.15.113:6789/0 conn(0x7fe3c40d16a0 legacy=0x7fe3c40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.331+0000 7fe3c2a00700 1 -- 172.21.15.89:0/55846636 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c41619e0 con 0x7fe3c415d9d0 2024-04-02T07:56:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.331+0000 7fe3c2a00700 1 --2- 172.21.15.89:0/55846636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3c415d9d0 0x7fe3c40d3c30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe3b40098f0 tx=0x7fe3b4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeac0a5507073bee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.332+0000 7fe3c1600700 1 -- 172.21.15.89:0/55846636 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4011830 con 0x7fe3c415d9d0 2024-04-02T07:56:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.332+0000 7fe3c1600700 1 -- 172.21.15.89:0/55846636 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b4011990 con 0x7fe3c415d9d0 2024-04-02T07:56:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.332+0000 7fe3c1600700 1 -- 172.21.15.89:0/55846636 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4011d20 con 0x7fe3c415d9d0 2024-04-02T07:56:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.332+0000 7fe3c9800700 1 -- 172.21.15.89:0/55846636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3c415d9d0 msgr2=0x7fe3c40d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.332+0000 7fe3c9800700 1 --2- 172.21.15.89:0/55846636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3c415d9d0 0x7fe3c40d3c30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe3b40098f0 tx=0x7fe3b4009c00 comp rx=0 tx=0).stop 2024-04-02T07:56:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.332+0000 7fe3c9800700 1 -- 172.21.15.89:0/55846636 shutdown_connections 2024-04-02T07:56:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.332+0000 7fe3c9800700 1 --2- 172.21.15.89:0/55846636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3c40d2540 0x7fe3c40ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.332+0000 7fe3c9800700 1 --2- 172.21.15.89:0/55846636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3c415d9d0 0x7fe3c40d3c30 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.332+0000 7fe3c9800700 1 -- 172.21.15.89:0/55846636 >> 172.21.15.89:0/55846636 conn(0x7fe3c408aff0 msgr2=0x7fe3c40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.332+0000 7fe3c9800700 1 -- 172.21.15.89:0/55846636 shutdown_connections 2024-04-02T07:56:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.332+0000 7fe3c9800700 1 -- 172.21.15.89:0/55846636 wait complete. 2024-04-02T07:56:08.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.334+0000 7fe3c9800700 1 Processor -- start 2024-04-02T07:56:08.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.334+0000 7fe3c9800700 1 -- start start 2024-04-02T07:56:08.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.335+0000 7fe3c9800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3c40d16a0 0x7fe3c41e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:08.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.335+0000 7fe3c9800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c40d2540 0x7fe3c414f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:08.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.335+0000 7fe3c9800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3c415d9d0 0x7fe3c40b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:08.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.335+0000 7fe3c9800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c40cb5a0 con 0x7fe3c40d2540 2024-04-02T07:56:08.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.335+0000 7fe3c9800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c41e82e0 con 0x7fe3c415d9d0 2024-04-02T07:56:08.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.335+0000 7fe3c9800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c41e85c0 con 0x7fe3c40d16a0 2024-04-02T07:56:08.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.335+0000 7fe3c2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c40d2540 0x7fe3c414f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:08.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.335+0000 7fe3c2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c40d2540 0x7fe3c414f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54526/0 (socket says 172.21.15.89:54526) 2024-04-02T07:56:08.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.335+0000 7fe3c2000700 1 -- 172.21.15.89:0/1787212182 learned_addr learned my addr 172.21.15.89:0/1787212182 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:08.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.336+0000 7fe3c3400700 1 --2- 172.21.15.89:0/1787212182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3c415d9d0 0x7fe3c40b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:08.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.336+0000 7fe3c2a00700 1 --2- 172.21.15.89:0/1787212182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3c40d16a0 0x7fe3c41e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:08.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.336+0000 7fe3c2000700 1 -- 172.21.15.89:0/1787212182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3c40d16a0 msgr2=0x7fe3c41e7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:08.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.336+0000 7fe3c2000700 1 --2- 172.21.15.89:0/1787212182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3c40d16a0 0x7fe3c41e7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.336+0000 7fe3c2000700 1 -- 172.21.15.89:0/1787212182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3c415d9d0 msgr2=0x7fe3c40b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:08.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.336+0000 7fe3c2000700 1 --2- 172.21.15.89:0/1787212182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3c415d9d0 0x7fe3c40b7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.336+0000 7fe3c2000700 1 -- 172.21.15.89:0/1787212182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b4009550 con 0x7fe3c40d2540 2024-04-02T07:56:08.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.336+0000 7fe3c2000700 1 --2- 172.21.15.89:0/1787212182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c40d2540 0x7fe3c414f850 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000c4d0 tx=0x7fe3b000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:08.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.336+0000 7fe3baa00700 1 -- 172.21.15.89:0/1787212182 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0005d00 con 0x7fe3c40d2540 2024-04-02T07:56:08.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.336+0000 7fe3baa00700 1 -- 172.21.15.89:0/1787212182 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b0005e60 con 0x7fe3c40d2540 2024-04-02T07:56:08.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.336+0000 7fe3c9800700 1 -- 172.21.15.89:0/1787212182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c41e88a0 con 0x7fe3c40d2540 2024-04-02T07:56:08.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.336+0000 7fe3baa00700 1 -- 172.21.15.89:0/1787212182 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0007310 con 0x7fe3c40d2540 2024-04-02T07:56:08.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.336+0000 7fe3c9800700 1 -- 172.21.15.89:0/1787212182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c414fd90 con 0x7fe3c40d2540 2024-04-02T07:56:08.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.337+0000 7fe3c9800700 1 -- 172.21.15.89:0/1787212182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c40d1ca0 con 0x7fe3c40d2540 2024-04-02T07:56:08.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.338+0000 7fe3baa00700 1 -- 172.21.15.89:0/1787212182 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe3b0007470 con 0x7fe3c40d2540 2024-04-02T07:56:08.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.339+0000 7fe3baa00700 1 --2- 172.21.15.89:0/1787212182 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe3a406adb0 0x7fe3a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:08.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.339+0000 7fe3baa00700 1 -- 172.21.15.89:0/1787212182 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5964+0+0 (secure 0 0 0) 0x7fe3b00889e0 con 0x7fe3c40d2540 2024-04-02T07:56:08.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.339+0000 7fe3c2a00700 1 --2- 172.21.15.89:0/1787212182 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe3a406adb0 0x7fe3a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:08.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.340+0000 7fe3c2a00700 1 --2- 172.21.15.89:0/1787212182 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe3a406adb0 0x7fe3a406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4014040 tx=0x7fe3b4008030 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:08.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.344+0000 7fe3baa00700 1 -- 172.21.15.89:0/1787212182 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe3b0053910 con 0x7fe3c40d2540 2024-04-02T07:56:08.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.545+0000 7fe3c9800700 1 -- 172.21.15.89:0/1787212182 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe3c40cb3e0 con 0x7fe3c40d2540 2024-04-02T07:56:08.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.551+0000 7fe3baa00700 1 -- 172.21.15.89:0/1787212182 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v51) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe3c40cb3e0 con 0x7fe3c40d2540 2024-04-02T07:56:08.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.553+0000 7fe3c9800700 1 -- 172.21.15.89:0/1787212182 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe3a406adb0 msgr2=0x7fe3a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:08.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.553+0000 7fe3c9800700 1 --2- 172.21.15.89:0/1787212182 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe3a406adb0 0x7fe3a406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4014040 tx=0x7fe3b4008030 comp rx=0 tx=0).stop 2024-04-02T07:56:08.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.553+0000 7fe3c9800700 1 -- 172.21.15.89:0/1787212182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c40d2540 msgr2=0x7fe3c414f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:08.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.553+0000 7fe3c9800700 1 --2- 172.21.15.89:0/1787212182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c40d2540 0x7fe3c414f850 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000c4d0 tx=0x7fe3b000c7e0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.553+0000 7fe3c9800700 1 -- 172.21.15.89:0/1787212182 shutdown_connections 2024-04-02T07:56:08.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.553+0000 7fe3c9800700 1 --2- 172.21.15.89:0/1787212182 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe3a406adb0 0x7fe3a406d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.553+0000 7fe3c9800700 1 --2- 172.21.15.89:0/1787212182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3c415d9d0 0x7fe3c40b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.553+0000 7fe3c9800700 1 --2- 172.21.15.89:0/1787212182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3c40d16a0 0x7fe3c41e7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.553+0000 7fe3c9800700 1 --2- 172.21.15.89:0/1787212182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c40d2540 0x7fe3c414f850 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.553+0000 7fe3c9800700 1 -- 172.21.15.89:0/1787212182 >> 172.21.15.89:0/1787212182 conn(0x7fe3c408aff0 msgr2=0x7fe3c40b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:08.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.554+0000 7fe3c9800700 1 -- 172.21.15.89:0/1787212182 shutdown_connections 2024-04-02T07:56:08.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.554+0000 7fe3c9800700 1 -- 172.21.15.89:0/1787212182 wait complete. 2024-04-02T07:56:08.572 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-04-02T07:56:08.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.763+0000 7fcfa8e00700 1 Processor -- start 2024-04-02T07:56:08.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.763+0000 7fcfa8e00700 1 -- start start 2024-04-02T07:56:08.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.764+0000 7fcfa8e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcfa4160f10 0x7fcfa40d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:08.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.764+0000 7fcfa8e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcfa4161590 0x7fcfa4161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:08.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.764+0000 7fcfa8e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa40ceda0 con 0x7fcfa40cf230 2024-04-02T07:56:08.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.764+0000 7fcfa8e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa4161f80 con 0x7fcfa4161590 2024-04-02T07:56:08.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.764+0000 7fcfa8e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa41612f0 con 0x7fcfa4160f10 2024-04-02T07:56:08.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.764+0000 7fcfa2800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcfa4161590 0x7fcfa4161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:08.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.764+0000 7fcfa2800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcfa4161590 0x7fcfa4161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41064/0 (socket says 172.21.15.89:41064) 2024-04-02T07:56:08.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.764+0000 7fcfa2800700 1 -- 172.21.15.89:0/1550112100 learned_addr learned my addr 172.21.15.89:0/1550112100 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:08.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.764+0000 7fcfa1e00700 1 --2- 172.21.15.89:0/1550112100 >> v2:172.21.15.113:3300/0 conn(0x7fcfa4160f10 0x7fcfa40d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:08.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.765+0000 7fcfa2800700 1 -- 172.21.15.89:0/1550112100 >> v2:172.21.15.113:3300/0 conn(0x7fcfa4160f10 msgr2=0x7fcfa40d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:08.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.765+0000 7fcfa2800700 1 --2- 172.21.15.89:0/1550112100 >> v2:172.21.15.113:3300/0 conn(0x7fcfa4160f10 0x7fcfa40d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.765+0000 7fcfa2800700 1 -- 172.21.15.89:0/1550112100 >> v1:172.21.15.113:6789/0 conn(0x7fcfa40cf230 legacy=0x7fcfa40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:08.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.765+0000 7fcfa2800700 1 -- 172.21.15.89:0/1550112100 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa40ce470 con 0x7fcfa4161590 2024-04-02T07:56:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.765+0000 7fcfa2800700 1 --2- 172.21.15.89:0/1550112100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4161590 0x7fcfa4161a40 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fcfa40d1080 tx=0x7fcf9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913f05813c0b3c77 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.765+0000 7fcfa0a00700 1 -- 172.21.15.89:0/1550112100 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c00e070 con 0x7fcfa4161590 2024-04-02T07:56:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.765+0000 7fcfa0a00700 1 -- 172.21.15.89:0/1550112100 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c00b910 con 0x7fcfa4161590 2024-04-02T07:56:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.765+0000 7fcfa0a00700 1 -- 172.21.15.89:0/1550112100 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c00bcc0 con 0x7fcfa4161590 2024-04-02T07:56:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.766+0000 7fcfa8e00700 1 -- 172.21.15.89:0/1550112100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4161590 msgr2=0x7fcfa4161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.766+0000 7fcfa8e00700 1 --2- 172.21.15.89:0/1550112100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4161590 0x7fcfa4161a40 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fcfa40d1080 tx=0x7fcf9c00b330 comp rx=0 tx=0).stop 2024-04-02T07:56:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.766+0000 7fcfa8e00700 1 -- 172.21.15.89:0/1550112100 shutdown_connections 2024-04-02T07:56:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.766+0000 7fcfa8e00700 1 --2- 172.21.15.89:0/1550112100 >> v2:172.21.15.113:3300/0 conn(0x7fcfa4160f10 0x7fcfa40d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.766+0000 7fcfa8e00700 1 --2- 172.21.15.89:0/1550112100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4161590 0x7fcfa4161a40 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.766+0000 7fcfa8e00700 1 -- 172.21.15.89:0/1550112100 >> 172.21.15.89:0/1550112100 conn(0x7fcfa408aff0 msgr2=0x7fcfa40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.766+0000 7fcfa8e00700 1 -- 172.21.15.89:0/1550112100 shutdown_connections 2024-04-02T07:56:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.766+0000 7fcfa8e00700 1 -- 172.21.15.89:0/1550112100 wait complete. 2024-04-02T07:56:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.768+0000 7fcfa8e00700 1 Processor -- start 2024-04-02T07:56:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.768+0000 7fcfa8e00700 1 -- start start 2024-04-02T07:56:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa8e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa40cf230 0x7fcfa4157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa8e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa4160f10 0x7fcfa415a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa8e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4161590 0x7fcfa415af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa8e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa4158250 con 0x7fcfa4161590 2024-04-02T07:56:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa8e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa415f360 con 0x7fcfa4160f10 2024-04-02T07:56:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa8e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa415f640 con 0x7fcfa40cf230 2024-04-02T07:56:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4161590 0x7fcfa415af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4161590 0x7fcfa415af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41066/0 (socket says 172.21.15.89:41066) 2024-04-02T07:56:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa2800700 1 -- 172.21.15.89:0/2583135818 learned_addr learned my addr 172.21.15.89:0/2583135818 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa1e00700 1 --2- 172.21.15.89:0/2583135818 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa40cf230 0x7fcfa4157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa1400700 1 --2- 172.21.15.89:0/2583135818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa4160f10 0x7fcfa415a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa2800700 1 -- 172.21.15.89:0/2583135818 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa40cf230 msgr2=0x7fcfa4157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa2800700 1 --2- 172.21.15.89:0/2583135818 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa40cf230 0x7fcfa4157d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa2800700 1 -- 172.21.15.89:0/2583135818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa4160f10 msgr2=0x7fcfa415a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa2800700 1 --2- 172.21.15.89:0/2583135818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa4160f10 0x7fcfa415a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.769+0000 7fcfa2800700 1 -- 172.21.15.89:0/2583135818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c009b30 con 0x7fcfa4161590 2024-04-02T07:56:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.770+0000 7fcfa2800700 1 --2- 172.21.15.89:0/2583135818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4161590 0x7fcfa415af00 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c00b370 tx=0x7fcf9c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.770+0000 7fcf9a000700 1 -- 172.21.15.89:0/2583135818 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c004880 con 0x7fcfa4161590 2024-04-02T07:56:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.770+0000 7fcf9a000700 1 -- 172.21.15.89:0/2583135818 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c0099b0 con 0x7fcfa4161590 2024-04-02T07:56:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.770+0000 7fcfa8e00700 1 -- 172.21.15.89:0/2583135818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa415f920 con 0x7fcfa4161590 2024-04-02T07:56:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.770+0000 7fcf9a000700 1 -- 172.21.15.89:0/2583135818 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c01c680 con 0x7fcfa4161590 2024-04-02T07:56:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.770+0000 7fcfa8e00700 1 -- 172.21.15.89:0/2583135818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa415fee0 con 0x7fcfa4161590 2024-04-02T07:56:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.771+0000 7fcfa8e00700 1 -- 172.21.15.89:0/2583135818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa415d0b0 con 0x7fcfa4161590 2024-04-02T07:56:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.772+0000 7fcf9a000700 1 -- 172.21.15.89:0/2583135818 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcf9c01c7e0 con 0x7fcfa4161590 2024-04-02T07:56:08.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.773+0000 7fcf9a000700 1 --2- 172.21.15.89:0/2583135818 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8406adb0 0x7fcf8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:08.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.773+0000 7fcf9a000700 1 -- 172.21.15.89:0/2583135818 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5964+0+0 (secure 0 0 0) 0x7fcf9c08b220 con 0x7fcfa4161590 2024-04-02T07:56:08.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.774+0000 7fcfa1e00700 1 --2- 172.21.15.89:0/2583135818 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8406adb0 0x7fcf8406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:08.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.774+0000 7fcfa1e00700 1 --2- 172.21.15.89:0/2583135818 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8406adb0 0x7fcf8406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009b40 tx=0x7fcf94009210 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:08.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:08.778+0000 7fcf9a000700 1 -- 172.21.15.89:0/2583135818 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf9c0560d0 con 0x7fcfa4161590 2024-04-02T07:56:09.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.063+0000 7fcfa8e00700 1 -- 172.21.15.89:0/2583135818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7fcfa4067720 con 0x7fcfa4161590 2024-04-02T07:56:09.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.234+0000 7fcf9a000700 1 -- 172.21.15.89:0/2583135818 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v773) v1 ==== 160+0+0 (secure 0 0 0) 0x7fcf9c0596f0 con 0x7fcfa4161590 2024-04-02T07:56:09.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.237+0000 7fcfa8e00700 1 -- 172.21.15.89:0/2583135818 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8406adb0 msgr2=0x7fcf8406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:09.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.237+0000 7fcfa8e00700 1 --2- 172.21.15.89:0/2583135818 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8406adb0 0x7fcf8406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009b40 tx=0x7fcf94009210 comp rx=0 tx=0).stop 2024-04-02T07:56:09.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.237+0000 7fcfa8e00700 1 -- 172.21.15.89:0/2583135818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4161590 msgr2=0x7fcfa415af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:09.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.237+0000 7fcfa8e00700 1 --2- 172.21.15.89:0/2583135818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4161590 0x7fcfa415af00 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c00b370 tx=0x7fcf9c004ad0 comp rx=0 tx=0).stop 2024-04-02T07:56:09.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.237+0000 7fcfa8e00700 1 -- 172.21.15.89:0/2583135818 shutdown_connections 2024-04-02T07:56:09.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.237+0000 7fcfa8e00700 1 --2- 172.21.15.89:0/2583135818 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8406adb0 0x7fcf8406d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:09.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.237+0000 7fcfa8e00700 1 --2- 172.21.15.89:0/2583135818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4161590 0x7fcfa415af00 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:09.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.237+0000 7fcfa8e00700 1 --2- 172.21.15.89:0/2583135818 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa40cf230 0x7fcfa4157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:09.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.237+0000 7fcfa8e00700 1 --2- 172.21.15.89:0/2583135818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa4160f10 0x7fcfa415a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:09.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.237+0000 7fcfa8e00700 1 -- 172.21.15.89:0/2583135818 >> 172.21.15.89:0/2583135818 conn(0x7fcfa408aff0 msgr2=0x7fcfa40cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:09.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.237+0000 7fcfa8e00700 1 -- 172.21.15.89:0/2583135818 shutdown_connections 2024-04-02T07:56:09.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.237+0000 7fcfa8e00700 1 -- 172.21.15.89:0/2583135818 wait complete. 2024-04-02T07:56:09.256 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2024-04-02T07:56:09.256 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2024-04-02T07:56:09.256 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-02T07:56:09.256 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-04-02T07:56:09.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.447+0000 7f6c8b400700 1 Processor -- start 2024-04-02T07:56:09.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.447+0000 7f6c8b400700 1 -- start start 2024-04-02T07:56:09.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.448+0000 7f6c8b400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6c8415d640 0x7f6c840d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:09.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.448+0000 7f6c8b400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c840d30d0 con 0x7f6c840ceec0 2024-04-02T07:56:09.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.448+0000 7f6c8b400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c840b6840 con 0x7f6c841619d0 2024-04-02T07:56:09.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.448+0000 7f6c8b400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c840d2880 con 0x7f6c8415d640 2024-04-02T07:56:09.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.448+0000 7f6c83e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6c840ceec0 0x7f6c841612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49868/0 (socket says 172.21.15.89:49868) 2024-04-02T07:56:09.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.448+0000 7f6c83e00700 1 -- 172.21.15.89:0/943842081 learned_addr learned my addr 172.21.15.89:0/943842081 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:09.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c88a00700 1 --2- 172.21.15.89:0/943842081 >> v2:172.21.15.113:3300/0 conn(0x7f6c8415d640 0x7f6c840d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:09.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 -- 172.21.15.89:0/943842081 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570558464 0 0) 0x7f6c840d30d0 con 0x7f6c840ceec0 2024-04-02T07:56:09.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 -- 172.21.15.89:0/943842081 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c6c0035b0 con 0x7f6c840ceec0 2024-04-02T07:56:09.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 -- 172.21.15.89:0/943842081 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3459179771 0 0) 0x7f6c6c0035b0 con 0x7f6c840ceec0 2024-04-02T07:56:09.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 -- 172.21.15.89:0/943842081 >> v2:172.21.15.113:3300/0 conn(0x7f6c8415d640 msgr2=0x7f6c840d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:09.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 --2- 172.21.15.89:0/943842081 >> v2:172.21.15.113:3300/0 conn(0x7f6c8415d640 0x7f6c840d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:09.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 -- 172.21.15.89:0/943842081 >> v1:172.21.15.113:6789/0 conn(0x7f6c841619d0 legacy=0x7f6c840b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:09.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 -- 172.21.15.89:0/943842081 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c840d2b30 con 0x7f6c840ceec0 2024-04-02T07:56:09.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 -- 172.21.15.89:0/943842081 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f6c70001670 con 0x7f6c840ceec0 2024-04-02T07:56:09.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 -- 172.21.15.89:0/943842081 >> v1:172.21.15.89:6789/0 conn(0x7f6c840ceec0 legacy=0x7f6c841612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:09.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 --2- 172.21.15.89:0/943842081 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c6c003a00 0x7f6c6c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:09.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 --2- 172.21.15.89:0/943842081 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c004850 0x7f6c6c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:09.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 --2- 172.21.15.89:0/943842081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c005210 0x7f6c6c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:09.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 -- 172.21.15.89:0/943842081 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c840d2880 con 0x7f6c6c005210 2024-04-02T07:56:09.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 -- 172.21.15.89:0/943842081 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c00c1b0 con 0x7f6c6c004850 2024-04-02T07:56:09.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.449+0000 7f6c83400700 1 -- 172.21.15.89:0/943842081 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c00c370 con 0x7f6c6c003a00 2024-04-02T07:56:09.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.450+0000 7f6c89400700 1 --2- 172.21.15.89:0/943842081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c005210 0x7f6c6c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:09.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.450+0000 7f6c83e00700 1 --2- 172.21.15.89:0/943842081 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c004850 0x7f6c6c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:09.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.450+0000 7f6c88a00700 1 --2- 172.21.15.89:0/943842081 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c6c003a00 0x7f6c6c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:09.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.450+0000 7f6c89400700 1 -- 172.21.15.89:0/943842081 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c6c003a00 msgr2=0x7f6c6c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:09.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.450+0000 7f6c89400700 1 --2- 172.21.15.89:0/943842081 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c6c003a00 0x7f6c6c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:09.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.450+0000 7f6c89400700 1 -- 172.21.15.89:0/943842081 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c004850 msgr2=0x7f6c6c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:09.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.450+0000 7f6c89400700 1 --2- 172.21.15.89:0/943842081 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c004850 0x7f6c6c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:09.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.450+0000 7f6c89400700 1 -- 172.21.15.89:0/943842081 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c0035b0 con 0x7f6c6c005210 2024-04-02T07:56:09.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.450+0000 7f6c89400700 1 -- 172.21.15.89:0/943842081 reap_dead start 2024-04-02T07:56:09.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.450+0000 7f6c89400700 1 --2- 172.21.15.89:0/943842081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c005210 0x7f6c6c00ba10 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00c190 tx=0x7f6c6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e584139c6e3a7e4d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:09.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.451+0000 7f6c83400700 1 -- 172.21.15.89:0/943842081 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c001db0 con 0x7f6c6c005210 2024-04-02T07:56:09.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.451+0000 7f6c83400700 1 -- 172.21.15.89:0/943842081 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7c00ccd0 con 0x7f6c6c005210 2024-04-02T07:56:09.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.451+0000 7f6c83400700 1 -- 172.21.15.89:0/943842081 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c003c40 con 0x7f6c6c005210 2024-04-02T07:56:09.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.451+0000 7f6c8b400700 1 -- 172.21.15.89:0/943842081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c005210 msgr2=0x7f6c6c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:09.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.451+0000 7f6c8b400700 1 --2- 172.21.15.89:0/943842081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c005210 0x7f6c6c00ba10 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00c190 tx=0x7f6c6c005880 comp rx=0 tx=0).stop 2024-04-02T07:56:09.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.451+0000 7f6c8b400700 1 -- 172.21.15.89:0/943842081 shutdown_connections 2024-04-02T07:56:09.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.451+0000 7f6c8b400700 1 --2- 172.21.15.89:0/943842081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c6c005210 0x7f6c6c00ba10 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:09.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.451+0000 7f6c8b400700 1 -- 172.21.15.89:0/943842081 >> 172.21.15.89:0/943842081 conn(0x7f6c8408aff0 msgr2=0x7f6c840d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:09.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.451+0000 7f6c8b400700 1 -- 172.21.15.89:0/943842081 shutdown_connections 2024-04-02T07:56:09.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.451+0000 7f6c8b400700 1 -- 172.21.15.89:0/943842081 wait complete. 2024-04-02T07:56:09.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.453+0000 7f6c8b400700 1 Processor -- start 2024-04-02T07:56:09.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.453+0000 7f6c8b400700 1 -- start start 2024-04-02T07:56:09.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.454+0000 7f6c8b400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c005210 0x7f6c8415edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:09.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.454+0000 7f6c8b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c84158b40 0x7f6c8415f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:09.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.454+0000 7f6c8b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c8409fb50 0x7f6c8409ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:09.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.454+0000 7f6c8b400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8409d780 con 0x7f6c84158b40 2024-04-02T07:56:09.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.454+0000 7f6c8b400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c840c8590 con 0x7f6c6c005210 2024-04-02T07:56:09.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.454+0000 7f6c8b400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c840a0880 con 0x7f6c8409fb50 2024-04-02T07:56:09.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.454+0000 7f6c83e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c84158b40 0x7f6c8415f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.454+0000 7f6c83e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c84158b40 0x7f6c8415f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41088/0 (socket says 172.21.15.89:41088) 2024-04-02T07:56:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.454+0000 7f6c83e00700 1 -- 172.21.15.89:0/2012641093 learned_addr learned my addr 172.21.15.89:0/2012641093 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.455+0000 7f6c88a00700 1 --2- 172.21.15.89:0/2012641093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c005210 0x7f6c8415edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.455+0000 7f6c89400700 1 --2- 172.21.15.89:0/2012641093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c8409fb50 0x7f6c8409ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.455+0000 7f6c83e00700 1 -- 172.21.15.89:0/2012641093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c8409fb50 msgr2=0x7f6c8409ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.455+0000 7f6c83e00700 1 --2- 172.21.15.89:0/2012641093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c8409fb50 0x7f6c8409ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.455+0000 7f6c83e00700 1 -- 172.21.15.89:0/2012641093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c005210 msgr2=0x7f6c8415edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.455+0000 7f6c83e00700 1 --2- 172.21.15.89:0/2012641093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c005210 0x7f6c8415edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.455+0000 7f6c83e00700 1 -- 172.21.15.89:0/2012641093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c840a0b60 con 0x7f6c84158b40 2024-04-02T07:56:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.455+0000 7f6c83e00700 1 --2- 172.21.15.89:0/2012641093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c84158b40 0x7f6c8415f310 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f6c700072c0 tx=0x7f6c70007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.455+0000 7f6c80c00700 1 -- 172.21.15.89:0/2012641093 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c700077a0 con 0x7f6c84158b40 2024-04-02T07:56:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.455+0000 7f6c80c00700 1 -- 172.21.15.89:0/2012641093 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7000f8d0 con 0x7f6c84158b40 2024-04-02T07:56:09.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.455+0000 7f6c8b400700 1 -- 172.21.15.89:0/2012641093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c840a0e40 con 0x7f6c84158b40 2024-04-02T07:56:09.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.455+0000 7f6c80c00700 1 -- 172.21.15.89:0/2012641093 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7000fc80 con 0x7f6c84158b40 2024-04-02T07:56:09.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.455+0000 7f6c8b400700 1 -- 172.21.15.89:0/2012641093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c840a0510 con 0x7f6c84158b40 2024-04-02T07:56:09.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.456+0000 7f6c8b400700 1 -- 172.21.15.89:0/2012641093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c840c8740 con 0x7f6c84158b40 2024-04-02T07:56:09.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.457+0000 7f6c80c00700 1 -- 172.21.15.89:0/2012641093 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6c7001d460 con 0x7f6c84158b40 2024-04-02T07:56:09.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.458+0000 7f6c80c00700 1 --2- 172.21.15.89:0/2012641093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6c6406adb0 0x7f6c6406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:09.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.458+0000 7f6c80c00700 1 -- 172.21.15.89:0/2012641093 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5964+0+0 (secure 0 0 0) 0x7f6c70089640 con 0x7f6c84158b40 2024-04-02T07:56:09.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.458+0000 7f6c88a00700 1 --2- 172.21.15.89:0/2012641093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6c6406adb0 0x7f6c6406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:09.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.459+0000 7f6c88a00700 1 --2- 172.21.15.89:0/2012641093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6c6406adb0 0x7f6c6406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6c78009b20 tx=0x7f6c78008040 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:09.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.462+0000 7f6c80c00700 1 -- 172.21.15.89:0/2012641093 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c840c8740 con 0x7f6c84158b40 2024-04-02T07:56:09.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:09.753+0000 7f6c8b400700 1 -- 172.21.15.89:0/2012641093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f6c84067720 con 0x7f6c84158b40 2024-04-02T07:56:10.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.234+0000 7f6c80c00700 1 -- 172.21.15.89:0/2012641093 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v774) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6c70058000 con 0x7f6c84158b40 2024-04-02T07:56:10.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.237+0000 7f6c8b400700 1 -- 172.21.15.89:0/2012641093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6c6406adb0 msgr2=0x7f6c6406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:10.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.237+0000 7f6c8b400700 1 --2- 172.21.15.89:0/2012641093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6c6406adb0 0x7f6c6406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6c78009b20 tx=0x7f6c78008040 comp rx=0 tx=0).stop 2024-04-02T07:56:10.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.237+0000 7f6c8b400700 1 -- 172.21.15.89:0/2012641093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c84158b40 msgr2=0x7f6c8415f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:10.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.237+0000 7f6c8b400700 1 --2- 172.21.15.89:0/2012641093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c84158b40 0x7f6c8415f310 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f6c700072c0 tx=0x7f6c70007680 comp rx=0 tx=0).stop 2024-04-02T07:56:10.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.237+0000 7f6c8b400700 1 -- 172.21.15.89:0/2012641093 shutdown_connections 2024-04-02T07:56:10.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.237+0000 7f6c8b400700 1 --2- 172.21.15.89:0/2012641093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6c6406adb0 0x7f6c6406d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:10.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.237+0000 7f6c8b400700 1 --2- 172.21.15.89:0/2012641093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c6c005210 0x7f6c8415edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:10.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.237+0000 7f6c8b400700 1 --2- 172.21.15.89:0/2012641093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c8409fb50 0x7f6c8409ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:10.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.237+0000 7f6c8b400700 1 --2- 172.21.15.89:0/2012641093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c84158b40 0x7f6c8415f310 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:10.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.237+0000 7f6c8b400700 1 -- 172.21.15.89:0/2012641093 >> 172.21.15.89:0/2012641093 conn(0x7f6c8408aff0 msgr2=0x7f6c8415b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:10.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.237+0000 7f6c8b400700 1 -- 172.21.15.89:0/2012641093 shutdown_connections 2024-04-02T07:56:10.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.237+0000 7f6c8b400700 1 -- 172.21.15.89:0/2012641093 wait complete. 2024-04-02T07:56:10.259 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T07:56:10.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.452+0000 7f2db0400700 1 Processor -- start 2024-04-02T07:56:10.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.453+0000 7f2db0400700 1 -- start start 2024-04-02T07:56:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.454+0000 7f2db0400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2da815d8a0 0x7f2da80b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.454+0000 7f2db0400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da80ced20 con 0x7f2da80d3eb0 2024-04-02T07:56:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.454+0000 7f2db0400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da80d4340 con 0x7f2da80cf1b0 2024-04-02T07:56:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.454+0000 7f2db0400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da80ce6d0 con 0x7f2da815d8a0 2024-04-02T07:56:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.454+0000 7f2dad000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2da80d3eb0 0x7f2da806b890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49884/0 (socket says 172.21.15.89:49884) 2024-04-02T07:56:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.454+0000 7f2dad000700 1 -- 172.21.15.89:0/2678759245 learned_addr learned my addr 172.21.15.89:0/2678759245 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.454+0000 7f2dada00700 1 --2- 172.21.15.89:0/2678759245 >> v2:172.21.15.113:3300/0 conn(0x7f2da815d8a0 0x7f2da80b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.454+0000 7f2da7e00700 1 -- 172.21.15.89:0/2678759245 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726346639 0 0) 0x7f2da80ced20 con 0x7f2da80d3eb0 2024-04-02T07:56:10.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.455+0000 7f2da7e00700 1 -- 172.21.15.89:0/2678759245 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d900035b0 con 0x7f2da80d3eb0 2024-04-02T07:56:10.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.455+0000 7f2da7e00700 1 -- 172.21.15.89:0/2678759245 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792115947 0 0) 0x7f2da80d4340 con 0x7f2da80cf1b0 2024-04-02T07:56:10.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.455+0000 7f2da7e00700 1 -- 172.21.15.89:0/2678759245 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da80ced20 con 0x7f2da80cf1b0 2024-04-02T07:56:10.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.455+0000 7f2da7e00700 1 -- 172.21.15.89:0/2678759245 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1446508633 0 0) 0x7f2d900035b0 con 0x7f2da80d3eb0 2024-04-02T07:56:10.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.455+0000 7f2dada00700 1 -- 172.21.15.89:0/2678759245 >> v1:172.21.15.113:6789/0 conn(0x7f2da80cf1b0 legacy=0x7f2da80d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:10.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.455+0000 7f2dada00700 1 -- 172.21.15.89:0/2678759245 >> v1:172.21.15.89:6789/0 conn(0x7f2da80d3eb0 legacy=0x7f2da806b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:10.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.455+0000 7f2dada00700 1 -- 172.21.15.89:0/2678759245 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da80ce9b0 con 0x7f2da815d8a0 2024-04-02T07:56:10.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.455+0000 7f2dada00700 1 --2- 172.21.15.89:0/2678759245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2da815d8a0 0x7f2da80b2990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009840 tx=0x7f2d9c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9063fadce678f57 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:10.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.455+0000 7f2da7e00700 1 -- 172.21.15.89:0/2678759245 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d940017f0 con 0x7f2da815d8a0 2024-04-02T07:56:10.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.455+0000 7f2da7e00700 1 -- 172.21.15.89:0/2678759245 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9c004030 con 0x7f2da815d8a0 2024-04-02T07:56:10.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.455+0000 7f2da7e00700 1 -- 172.21.15.89:0/2678759245 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c0043c0 con 0x7f2da815d8a0 2024-04-02T07:56:10.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.456+0000 7f2db0400700 1 -- 172.21.15.89:0/2678759245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2da815d8a0 msgr2=0x7f2da80b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:10.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.456+0000 7f2db0400700 1 --2- 172.21.15.89:0/2678759245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2da815d8a0 0x7f2da80b2990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009840 tx=0x7f2d9c009b50 comp rx=0 tx=0).stop 2024-04-02T07:56:10.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.456+0000 7f2db0400700 1 -- 172.21.15.89:0/2678759245 shutdown_connections 2024-04-02T07:56:10.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.456+0000 7f2db0400700 1 --2- 172.21.15.89:0/2678759245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2da815d8a0 0x7f2da80b2990 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:10.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.456+0000 7f2db0400700 1 -- 172.21.15.89:0/2678759245 >> 172.21.15.89:0/2678759245 conn(0x7f2da808aff0 msgr2=0x7f2da80d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:10.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.456+0000 7f2db0400700 1 -- 172.21.15.89:0/2678759245 shutdown_connections 2024-04-02T07:56:10.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.456+0000 7f2db0400700 1 -- 172.21.15.89:0/2678759245 wait complete. 2024-04-02T07:56:10.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.458+0000 7f2db0400700 1 Processor -- start 2024-04-02T07:56:10.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.459+0000 7f2db0400700 1 -- start start 2024-04-02T07:56:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2db0400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2da80cf1b0 0x7f2da8154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2db0400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da815d8a0 0x7f2da80b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2db0400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2da80b1c30 0x7f2da80cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2db0400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da80b20c0 con 0x7f2da815d8a0 2024-04-02T07:56:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2db0400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da8155520 con 0x7f2da80b1c30 2024-04-02T07:56:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2db0400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da81f6a30 con 0x7f2da80cf1b0 2024-04-02T07:56:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2dad000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da815d8a0 0x7f2da80b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2dad000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da815d8a0 0x7f2da80b16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41092/0 (socket says 172.21.15.89:41092) 2024-04-02T07:56:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2dad000700 1 -- 172.21.15.89:0/3914163569 learned_addr learned my addr 172.21.15.89:0/3914163569 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2dada00700 1 --2- 172.21.15.89:0/3914163569 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2da80cf1b0 0x7f2da8154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2dae400700 1 --2- 172.21.15.89:0/3914163569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2da80b1c30 0x7f2da80cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2dad000700 1 -- 172.21.15.89:0/3914163569 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2da80cf1b0 msgr2=0x7f2da8154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2dad000700 1 --2- 172.21.15.89:0/3914163569 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2da80cf1b0 0x7f2da8154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:10.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2dad000700 1 -- 172.21.15.89:0/3914163569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2da80b1c30 msgr2=0x7f2da80cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:10.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2dad000700 1 --2- 172.21.15.89:0/3914163569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2da80b1c30 0x7f2da80cc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:10.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.460+0000 7f2dad000700 1 -- 172.21.15.89:0/3914163569 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9c0094a0 con 0x7f2da815d8a0 2024-04-02T07:56:10.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.461+0000 7f2dad000700 1 --2- 172.21.15.89:0/3914163569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da815d8a0 0x7f2da80b16f0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400c730 tx=0x7f2d9400caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:10.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.461+0000 7f2da5600700 1 -- 172.21.15.89:0/3914163569 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d940074e0 con 0x7f2da815d8a0 2024-04-02T07:56:10.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.461+0000 7f2da5600700 1 -- 172.21.15.89:0/3914163569 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d94007640 con 0x7f2da815d8a0 2024-04-02T07:56:10.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.461+0000 7f2db0400700 1 -- 172.21.15.89:0/3914163569 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da81f6ce0 con 0x7f2da815d8a0 2024-04-02T07:56:10.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.461+0000 7f2da5600700 1 -- 172.21.15.89:0/3914163569 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d94005a50 con 0x7f2da815d8a0 2024-04-02T07:56:10.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.461+0000 7f2db0400700 1 -- 172.21.15.89:0/3914163569 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da81f7520 con 0x7f2da815d8a0 2024-04-02T07:56:10.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.462+0000 7f2db0400700 1 -- 172.21.15.89:0/3914163569 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da8067720 con 0x7f2da815d8a0 2024-04-02T07:56:10.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.463+0000 7f2da5600700 1 -- 172.21.15.89:0/3914163569 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2d94005bb0 con 0x7f2da815d8a0 2024-04-02T07:56:10.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.464+0000 7f2da5600700 1 --2- 172.21.15.89:0/3914163569 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2d8806adb0 0x7f2d8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:10.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.464+0000 7f2da5600700 1 -- 172.21.15.89:0/3914163569 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5964+0+0 (secure 0 0 0) 0x7f2d94088bf0 con 0x7f2da815d8a0 2024-04-02T07:56:10.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.464+0000 7f2dada00700 1 --2- 172.21.15.89:0/3914163569 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2d8806adb0 0x7f2d8806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:10.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.465+0000 7f2dada00700 1 --2- 172.21.15.89:0/3914163569 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2d8806adb0 0x7f2d8806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c0090b0 tx=0x7f2d9c008030 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:10.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.468+0000 7f2da5600700 1 -- 172.21.15.89:0/3914163569 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2d94053b20 con 0x7f2da815d8a0 2024-04-02T07:56:10.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.672+0000 7f2db0400700 1 -- 172.21.15.89:0/3914163569 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2da80d1950 con 0x7f2da815d8a0 2024-04-02T07:56:10.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.676+0000 7f2da5600700 1 -- 172.21.15.89:0/3914163569 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2d94017020 con 0x7f2da815d8a0 2024-04-02T07:56:10.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.679+0000 7f2db0400700 1 -- 172.21.15.89:0/3914163569 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2d8806adb0 msgr2=0x7f2d8806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:10.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.679+0000 7f2db0400700 1 --2- 172.21.15.89:0/3914163569 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2d8806adb0 0x7f2d8806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c0090b0 tx=0x7f2d9c008030 comp rx=0 tx=0).stop 2024-04-02T07:56:10.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.679+0000 7f2db0400700 1 -- 172.21.15.89:0/3914163569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da815d8a0 msgr2=0x7f2da80b16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:10.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.679+0000 7f2db0400700 1 --2- 172.21.15.89:0/3914163569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da815d8a0 0x7f2da80b16f0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400c730 tx=0x7f2d9400caf0 comp rx=0 tx=0).stop 2024-04-02T07:56:10.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.679+0000 7f2db0400700 1 -- 172.21.15.89:0/3914163569 shutdown_connections 2024-04-02T07:56:10.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.679+0000 7f2db0400700 1 --2- 172.21.15.89:0/3914163569 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2d8806adb0 0x7f2d8806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:10.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.679+0000 7f2db0400700 1 --2- 172.21.15.89:0/3914163569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2da80b1c30 0x7f2da80cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:10.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.679+0000 7f2db0400700 1 --2- 172.21.15.89:0/3914163569 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2da80cf1b0 0x7f2da8154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:10.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.679+0000 7f2db0400700 1 --2- 172.21.15.89:0/3914163569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da815d8a0 0x7f2da80b16f0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:10.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.679+0000 7f2db0400700 1 -- 172.21.15.89:0/3914163569 >> 172.21.15.89:0/3914163569 conn(0x7f2da808aff0 msgr2=0x7f2da806cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:10.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.679+0000 7f2db0400700 1 -- 172.21.15.89:0/3914163569 shutdown_connections 2024-04-02T07:56:10.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:10.679+0000 7f2db0400700 1 -- 172.21.15.89:0/3914163569 wait complete. 2024-04-02T07:56:10.698 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T07:56:10.698 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:56:10.780 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T07:56:10.871 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T07:56:10.872 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:56:10.872 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:56:10.916 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:56:10.916 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:56:10.916 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:56:10.961 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:56:10.961 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:56:10.962 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:56:11.005 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T07:56:11.005 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T07:56:11.005 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:56:11.049 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T07:56:11.049 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T07:56:11.049 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:56:11.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.243+0000 7fcfaa800700 1 Processor -- start 2024-04-02T07:56:11.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.244+0000 7fcfaa800700 1 -- start start 2024-04-02T07:56:11.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.245+0000 7fcfaa800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcfa40d18b0 0x7fcfa40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:11.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.245+0000 7fcfaa800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa4161590 0x7fcfa4161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:11.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.245+0000 7fcfaa800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa40ceda0 con 0x7fcfa4160f10 2024-04-02T07:56:11.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.245+0000 7fcfaa800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa40d1d30 con 0x7fcfa40d18b0 2024-04-02T07:56:11.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.245+0000 7fcfaa800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa4161f60 con 0x7fcfa4161590 2024-04-02T07:56:11.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.245+0000 7fcfa3e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa4161590 0x7fcfa4161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:11.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.245+0000 7fcfa3e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa4161590 0x7fcfa4161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:58220/0 (socket says 172.21.15.89:58220) 2024-04-02T07:56:11.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.245+0000 7fcfa3e00700 1 -- 172.21.15.89:0/2220442222 learned_addr learned my addr 172.21.15.89:0/2220442222 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:11.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.245+0000 7fcfa3400700 1 --1- 172.21.15.89:0/2220442222 >> v1:172.21.15.113:6789/0 conn(0x7fcfa4160f10 0x7fcfa40d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:40370/0 (socket says 172.21.15.89:40370) 2024-04-02T07:56:11.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.245+0000 7fcfa2a00700 1 --2- 172.21.15.89:0/2220442222 >> v2:172.21.15.113:3300/0 conn(0x7fcfa40d18b0 0x7fcfa40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.246+0000 7fcfa2000700 1 -- 172.21.15.89:0/2220442222 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429033827 0 0) 0x7fcfa40ceda0 con 0x7fcfa4160f10 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.246+0000 7fcfa2a00700 1 -- 172.21.15.89:0/2220442222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa4161590 msgr2=0x7fcfa4161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.246+0000 7fcfa2a00700 1 --2- 172.21.15.89:0/2220442222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa4161590 0x7fcfa4161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.246+0000 7fcfa2a00700 1 -- 172.21.15.89:0/2220442222 >> v1:172.21.15.113:6789/0 conn(0x7fcfa4160f10 legacy=0x7fcfa40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.246+0000 7fcfa2a00700 1 -- 172.21.15.89:0/2220442222 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa41612f0 con 0x7fcfa40d18b0 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.246+0000 7fcfa3e00700 1 --2- 172.21.15.89:0/2220442222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa4161590 0x7fcfa4161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.246+0000 7fcfa2a00700 1 --2- 172.21.15.89:0/2220442222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa40d18b0 0x7fcfa40cf230 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009870 tx=0x7fcf90009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15c3a9996a58ead9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.246+0000 7fcfa2000700 1 -- 172.21.15.89:0/2220442222 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90011830 con 0x7fcfa40d18b0 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.246+0000 7fcfa2000700 1 -- 172.21.15.89:0/2220442222 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf90011990 con 0x7fcfa40d18b0 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.246+0000 7fcfa2000700 1 -- 172.21.15.89:0/2220442222 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90011d20 con 0x7fcfa40d18b0 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.247+0000 7fcfaa800700 1 -- 172.21.15.89:0/2220442222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa40d18b0 msgr2=0x7fcfa40cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.247+0000 7fcfaa800700 1 --2- 172.21.15.89:0/2220442222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa40d18b0 0x7fcfa40cf230 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009870 tx=0x7fcf90009b80 comp rx=0 tx=0).stop 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.247+0000 7fcfaa800700 1 -- 172.21.15.89:0/2220442222 shutdown_connections 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.247+0000 7fcfaa800700 1 --2- 172.21.15.89:0/2220442222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa4161590 0x7fcfa4161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.247+0000 7fcfaa800700 1 --2- 172.21.15.89:0/2220442222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa40d18b0 0x7fcfa40cf230 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.247+0000 7fcfaa800700 1 -- 172.21.15.89:0/2220442222 >> 172.21.15.89:0/2220442222 conn(0x7fcfa408aff0 msgr2=0x7fcfa40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:11.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.247+0000 7fcfaa800700 1 -- 172.21.15.89:0/2220442222 shutdown_connections 2024-04-02T07:56:11.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.247+0000 7fcfaa800700 1 -- 172.21.15.89:0/2220442222 wait complete. 2024-04-02T07:56:11.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.248+0000 7fcfaa800700 1 Processor -- start 2024-04-02T07:56:11.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.249+0000 7fcfaa800700 1 -- start start 2024-04-02T07:56:11.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.249+0000 7fcfaa800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa40d18b0 0x7fcfa415a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:11.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.249+0000 7fcfaa800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4160f10 0x7fcfa409fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:11.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.249+0000 7fcfaa800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa4161590 0x7fcfa40a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:11.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.250+0000 7fcfaa800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa409ea30 con 0x7fcfa4160f10 2024-04-02T07:56:11.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.250+0000 7fcfaa800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa409ec80 con 0x7fcfa40d18b0 2024-04-02T07:56:11.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.250+0000 7fcfaa800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa409ef60 con 0x7fcfa4161590 2024-04-02T07:56:11.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.250+0000 7fcfa2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4160f10 0x7fcfa409fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:11.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.250+0000 7fcfa2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4160f10 0x7fcfa409fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41098/0 (socket says 172.21.15.89:41098) 2024-04-02T07:56:11.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.250+0000 7fcfa2a00700 1 -- 172.21.15.89:0/2128781192 learned_addr learned my addr 172.21.15.89:0/2128781192 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:11.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.250+0000 7fcfa3400700 1 --2- 172.21.15.89:0/2128781192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa40d18b0 0x7fcfa415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:11.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.250+0000 7fcfa3e00700 1 --2- 172.21.15.89:0/2128781192 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa4161590 0x7fcfa40a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:11.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.250+0000 7fcfa2a00700 1 -- 172.21.15.89:0/2128781192 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa4161590 msgr2=0x7fcfa40a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:11.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.250+0000 7fcfa2a00700 1 --2- 172.21.15.89:0/2128781192 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa4161590 0x7fcfa40a0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.250+0000 7fcfa2a00700 1 -- 172.21.15.89:0/2128781192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa40d18b0 msgr2=0x7fcfa415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:11.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.250+0000 7fcfa2a00700 1 --2- 172.21.15.89:0/2128781192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa40d18b0 0x7fcfa415a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.250+0000 7fcfa2a00700 1 -- 172.21.15.89:0/2128781192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c009b30 con 0x7fcfa4160f10 2024-04-02T07:56:11.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.250+0000 7fcfa2a00700 1 --2- 172.21.15.89:0/2128781192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4160f10 0x7fcfa409fd20 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fcf90014040 tx=0x7fcf90004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:11.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.251+0000 7fcf9b400700 1 -- 172.21.15.89:0/2128781192 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf900036a0 con 0x7fcfa4160f10 2024-04-02T07:56:11.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.251+0000 7fcf9b400700 1 -- 172.21.15.89:0/2128781192 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9001e070 con 0x7fcfa4160f10 2024-04-02T07:56:11.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.251+0000 7fcfaa800700 1 -- 172.21.15.89:0/2128781192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf900094a0 con 0x7fcfa4160f10 2024-04-02T07:56:11.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.251+0000 7fcf9b400700 1 -- 172.21.15.89:0/2128781192 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90003a00 con 0x7fcfa4160f10 2024-04-02T07:56:11.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.251+0000 7fcfaa800700 1 -- 172.21.15.89:0/2128781192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa409f520 con 0x7fcfa4160f10 2024-04-02T07:56:11.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.252+0000 7fcfaa800700 1 -- 172.21.15.89:0/2128781192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa4067720 con 0x7fcfa4160f10 2024-04-02T07:56:11.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.253+0000 7fcf9b400700 1 -- 172.21.15.89:0/2128781192 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcf90003b60 con 0x7fcfa4160f10 2024-04-02T07:56:11.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.254+0000 7fcf9b400700 1 --2- 172.21.15.89:0/2128781192 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8406adb0 0x7fcf8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:11.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.254+0000 7fcf9b400700 1 -- 172.21.15.89:0/2128781192 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5964+0+0 (secure 0 0 0) 0x7fcf9008a5f0 con 0x7fcfa4160f10 2024-04-02T07:56:11.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.254+0000 7fcfa3400700 1 --2- 172.21.15.89:0/2128781192 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8406adb0 0x7fcf8406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:11.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.254+0000 7fcfa3400700 1 --2- 172.21.15.89:0/2128781192 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8406adb0 0x7fcf8406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcf94000f60 tx=0x7fcf94007c90 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:11.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.258+0000 7fcf9b400700 1 -- 172.21.15.89:0/2128781192 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf900555b0 con 0x7fcfa4160f10 2024-04-02T07:56:11.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.498+0000 7fcfaa800700 1 -- 172.21.15.89:0/2128781192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfa415ada0 con 0x7fcfa4160f10 2024-04-02T07:56:11.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.499+0000 7fcf9b400700 1 -- 172.21.15.89:0/2128781192 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+3529 (secure 0 0 0) 0x7fcf90020de0 con 0x7fcfa4160f10 2024-04-02T07:56:11.499 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:11.499 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":68,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/66826967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":66826967},{"type":"v1","addr":"172.21.15.113:6835","nonce":66826967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":7915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/1669765650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1669765650},{"type":"v1","addr":"172.21.15.113:6833","nonce":1669765650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":7918,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3321722319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3321722319},{"type":"v1","addr":"172.21.15.89:6837","nonce":3321722319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3990908761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3990908761},{"type":"v1","addr":"172.21.15.89:6836","nonce":3990908761}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[]} 2024-04-02T07:56:11.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.501+0000 7fcfaa800700 1 -- 172.21.15.89:0/2128781192 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8406adb0 msgr2=0x7fcf8406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:11.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.501+0000 7fcfaa800700 1 --2- 172.21.15.89:0/2128781192 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8406adb0 0x7fcf8406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcf94000f60 tx=0x7fcf94007c90 comp rx=0 tx=0).stop 2024-04-02T07:56:11.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.501+0000 7fcfaa800700 1 -- 172.21.15.89:0/2128781192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4160f10 msgr2=0x7fcfa409fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:11.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.501+0000 7fcfaa800700 1 --2- 172.21.15.89:0/2128781192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4160f10 0x7fcfa409fd20 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fcf90014040 tx=0x7fcf90004750 comp rx=0 tx=0).stop 2024-04-02T07:56:11.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.501+0000 7fcfaa800700 1 -- 172.21.15.89:0/2128781192 shutdown_connections 2024-04-02T07:56:11.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.501+0000 7fcfaa800700 1 --2- 172.21.15.89:0/2128781192 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8406adb0 0x7fcf8406d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.501+0000 7fcfaa800700 1 --2- 172.21.15.89:0/2128781192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa40d18b0 0x7fcfa415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.501+0000 7fcfaa800700 1 --2- 172.21.15.89:0/2128781192 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa4161590 0x7fcfa40a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.501+0000 7fcfaa800700 1 --2- 172.21.15.89:0/2128781192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa4160f10 0x7fcfa409fd20 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.501+0000 7fcfaa800700 1 -- 172.21.15.89:0/2128781192 >> 172.21.15.89:0/2128781192 conn(0x7fcfa408aff0 msgr2=0x7fcfa40cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:11.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.502+0000 7fcfaa800700 1 -- 172.21.15.89:0/2128781192 shutdown_connections 2024-04-02T07:56:11.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.502+0000 7fcfaa800700 1 -- 172.21.15.89:0/2128781192 wait complete. 2024-04-02T07:56:11.503 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 68 2024-04-02T07:56:11.520 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T07:56:11.520 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T07:56:11.520 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T07:56:11.520 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:56:11.521 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T07:56:11.521 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T07:56:11.521 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T07:56:11.521 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:56:11.521 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T07:56:11.521 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T07:56:11.522 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T07:56:11.522 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:56:11.522 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T07:56:11.522 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T07:56:11.522 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T07:56:11.522 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T07:56:11.534 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T07:56:11.534 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T07:56:11.536 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T07:56:11.536 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T07:56:11.537 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T07:56:11.537 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T07:56:11.538 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T07:56:11.538 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T07:56:11.580 INFO:tasks.ceph.mds.a:Started 2024-04-02T07:56:11.580 DEBUG:teuthology.parallel:result is None 2024-04-02T07:56:11.580 INFO:tasks.ceph.mds.b:Started 2024-04-02T07:56:11.581 DEBUG:teuthology.parallel:result is None 2024-04-02T07:56:11.583 INFO:tasks.ceph.mds.c:Started 2024-04-02T07:56:11.583 DEBUG:teuthology.parallel:result is None 2024-04-02T07:56:11.583 INFO:tasks.ceph.mds.d:Started 2024-04-02T07:56:11.584 DEBUG:teuthology.parallel:result is None 2024-04-02T07:56:11.584 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T07:56:11.726 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T07:56:11.726 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T07:56:11.727 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T07:56:11.728 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T07:56:11.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.791+0000 7fb149600700 1 Processor -- start 2024-04-02T07:56:11.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.791+0000 7fb149600700 1 -- start start 2024-04-02T07:56:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.792+0000 7fb149600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb14415d800 0x7fb1440725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.792+0000 7fb149600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb1440ce230 0x7fb1440ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb149600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1440b2070 0x7fb1440b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb149600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1440b40f0 con 0x7fb1440ce230 2024-04-02T07:56:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb149600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1440709c0 con 0x7fb14415d800 2024-04-02T07:56:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb149600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb144070c80 con 0x7fb1440b2070 2024-04-02T07:56:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb142000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb1440ce230 0x7fb1440ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb142000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb1440ce230 0x7fb1440ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41140/0 (socket says 172.21.15.89:41140) 2024-04-02T07:56:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb142000700 1 -- 172.21.15.89:0/27834400 learned_addr learned my addr 172.21.15.89:0/27834400 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb142a00700 1 --2- 172.21.15.89:0/27834400 >> v2:172.21.15.113:3300/0 conn(0x7fb14415d800 0x7fb1440725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:11.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb143400700 1 --2- 172.21.15.89:0/27834400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1440b2070 0x7fb1440b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:11.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb142000700 1 -- 172.21.15.89:0/27834400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1440b2070 msgr2=0x7fb1440b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:11.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb142000700 1 --2- 172.21.15.89:0/27834400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1440b2070 0x7fb1440b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb142000700 1 -- 172.21.15.89:0/27834400 >> v2:172.21.15.113:3300/0 conn(0x7fb14415d800 msgr2=0x7fb1440725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:11.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb142000700 1 --2- 172.21.15.89:0/27834400 >> v2:172.21.15.113:3300/0 conn(0x7fb14415d800 0x7fb1440725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb142000700 1 -- 172.21.15.89:0/27834400 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb144070f60 con 0x7fb1440ce230 2024-04-02T07:56:11.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.793+0000 7fb142a00700 1 --2- 172.21.15.89:0/27834400 >> v2:172.21.15.113:3300/0 conn(0x7fb14415d800 0x7fb1440725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:11.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.794+0000 7fb142000700 1 --2- 172.21.15.89:0/27834400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1440ce230 0x7fb1440ce6a0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fb14406c100 tx=0x7fb130009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6094e1fe1e24f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:11.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.794+0000 7fb141600700 1 -- 172.21.15.89:0/27834400 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13000b870 con 0x7fb1440ce230 2024-04-02T07:56:11.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.794+0000 7fb141600700 1 -- 172.21.15.89:0/27834400 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13000b9d0 con 0x7fb1440ce230 2024-04-02T07:56:11.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.794+0000 7fb141600700 1 -- 172.21.15.89:0/27834400 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13000bd60 con 0x7fb1440ce230 2024-04-02T07:56:11.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.794+0000 7fb149600700 1 -- 172.21.15.89:0/27834400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1440ce230 msgr2=0x7fb1440ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:11.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.794+0000 7fb149600700 1 --2- 172.21.15.89:0/27834400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1440ce230 0x7fb1440ce6a0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fb14406c100 tx=0x7fb130009bd0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.794+0000 7fb149600700 1 -- 172.21.15.89:0/27834400 shutdown_connections 2024-04-02T07:56:11.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.795+0000 7fb149600700 1 --2- 172.21.15.89:0/27834400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1440b2070 0x7fb1440b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.795+0000 7fb149600700 1 --2- 172.21.15.89:0/27834400 >> v2:172.21.15.113:3300/0 conn(0x7fb14415d800 0x7fb1440725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.795+0000 7fb149600700 1 --2- 172.21.15.89:0/27834400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1440ce230 0x7fb1440ce6a0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.795+0000 7fb149600700 1 -- 172.21.15.89:0/27834400 >> 172.21.15.89:0/27834400 conn(0x7fb1440b0d70 msgr2=0x7fb1440b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:11.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.795+0000 7fb149600700 1 -- 172.21.15.89:0/27834400 shutdown_connections 2024-04-02T07:56:11.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.795+0000 7fb149600700 1 -- 172.21.15.89:0/27834400 wait complete. 2024-04-02T07:56:11.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.797+0000 7fb149600700 1 Processor -- start 2024-04-02T07:56:11.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.797+0000 7fb149600700 1 -- start start 2024-04-02T07:56:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.798+0000 7fb149600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1440b2070 0x7fb14414e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.798+0000 7fb149600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1440ce230 0x7fb144150e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.798+0000 7fb149600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb14415d800 0x7fb144150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.798+0000 7fb149600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144099d20 con 0x7fb14415d800 2024-04-02T07:56:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.798+0000 7fb149600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144151780 con 0x7fb1440ce230 2024-04-02T07:56:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.798+0000 7fb149600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb144151a60 con 0x7fb1440b2070 2024-04-02T07:56:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb143400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb14415d800 0x7fb144150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb143400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb14415d800 0x7fb144150070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41154/0 (socket says 172.21.15.89:41154) 2024-04-02T07:56:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb143400700 1 -- 172.21.15.89:0/4203132832 learned_addr learned my addr 172.21.15.89:0/4203132832 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb142a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1440b2070 0x7fb14414e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb142000700 1 --2- 172.21.15.89:0/4203132832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1440ce230 0x7fb144150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb143400700 1 -- 172.21.15.89:0/4203132832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1440b2070 msgr2=0x7fb14414e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb143400700 1 --2- 172.21.15.89:0/4203132832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1440b2070 0x7fb14414e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb143400700 1 -- 172.21.15.89:0/4203132832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1440ce230 msgr2=0x7fb144150e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb143400700 1 --2- 172.21.15.89:0/4203132832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1440ce230 0x7fb144150e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb143400700 1 -- 172.21.15.89:0/4203132832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb130009550 con 0x7fb14415d800 2024-04-02T07:56:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb143400700 1 --2- 172.21.15.89:0/4203132832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb14415d800 0x7fb144150070 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fb13c007e60 tx=0x7fb13c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb13aa00700 1 -- 172.21.15.89:0/4203132832 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c00ed30 con 0x7fb14415d800 2024-04-02T07:56:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb13aa00700 1 -- 172.21.15.89:0/4203132832 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13c014420 con 0x7fb14415d800 2024-04-02T07:56:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb149600700 1 -- 172.21.15.89:0/4203132832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb144151d40 con 0x7fb14415d800 2024-04-02T07:56:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.799+0000 7fb13aa00700 1 -- 172.21.15.89:0/4203132832 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c0147d0 con 0x7fb14415d800 2024-04-02T07:56:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.800+0000 7fb149600700 1 -- 172.21.15.89:0/4203132832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb144152300 con 0x7fb14415d800 2024-04-02T07:56:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.801+0000 7fb149600700 1 -- 172.21.15.89:0/4203132832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb144067720 con 0x7fb14415d800 2024-04-02T07:56:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.801+0000 7fb13aa00700 1 -- 172.21.15.89:0/4203132832 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb13c014930 con 0x7fb14415d800 2024-04-02T07:56:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.803+0000 7fb13aa00700 1 --2- 172.21.15.89:0/4203132832 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb12406adb0 0x7fb12406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.803+0000 7fb13aa00700 1 -- 172.21.15.89:0/4203132832 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5964+0+0 (secure 0 0 0) 0x7fb13c089bb0 con 0x7fb14415d800 2024-04-02T07:56:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.803+0000 7fb142a00700 1 --2- 172.21.15.89:0/4203132832 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb12406adb0 0x7fb12406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:11.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.804+0000 7fb142a00700 1 --2- 172.21.15.89:0/4203132832 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb12406adb0 0x7fb12406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb13400c940 tx=0x7fb134008040 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:11.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:11.808+0000 7fb13aa00700 1 -- 172.21.15.89:0/4203132832 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb13c054b70 con 0x7fb14415d800 2024-04-02T07:56:12.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.024+0000 7fb149600700 1 -- 172.21.15.89:0/4203132832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb14414ea40 con 0x7fb14415d800 2024-04-02T07:56:12.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.758+0000 7fb13aa00700 1 -- 172.21.15.89:0/4203132832 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v150) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb13c058190 con 0x7fb14415d800 2024-04-02T07:56:12.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.762+0000 7fb149600700 1 -- 172.21.15.89:0/4203132832 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb12406adb0 msgr2=0x7fb12406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:12.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.762+0000 7fb149600700 1 --2- 172.21.15.89:0/4203132832 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb12406adb0 0x7fb12406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb13400c940 tx=0x7fb134008040 comp rx=0 tx=0).stop 2024-04-02T07:56:12.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.762+0000 7fb149600700 1 -- 172.21.15.89:0/4203132832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb14415d800 msgr2=0x7fb144150070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:12.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.762+0000 7fb149600700 1 --2- 172.21.15.89:0/4203132832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb14415d800 0x7fb144150070 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fb13c007e60 tx=0x7fb13c00e3c0 comp rx=0 tx=0).stop 2024-04-02T07:56:12.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.762+0000 7fb149600700 1 -- 172.21.15.89:0/4203132832 shutdown_connections 2024-04-02T07:56:12.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.762+0000 7fb149600700 1 --2- 172.21.15.89:0/4203132832 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb12406adb0 0x7fb12406d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:12.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.762+0000 7fb149600700 1 --2- 172.21.15.89:0/4203132832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb14415d800 0x7fb144150070 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:12.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.762+0000 7fb149600700 1 --2- 172.21.15.89:0/4203132832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1440b2070 0x7fb14414e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:12.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.762+0000 7fb149600700 1 --2- 172.21.15.89:0/4203132832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1440ce230 0x7fb144150e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:12.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.762+0000 7fb149600700 1 -- 172.21.15.89:0/4203132832 >> 172.21.15.89:0/4203132832 conn(0x7fb1440b0d70 msgr2=0x7fb1440af450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:12.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.762+0000 7fb149600700 1 -- 172.21.15.89:0/4203132832 shutdown_connections 2024-04-02T07:56:12.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.762+0000 7fb149600700 1 -- 172.21.15.89:0/4203132832 wait complete. 2024-04-02T07:56:12.764 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T07:56:12.783 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T07:56:12.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.978+0000 7fbec0c00700 1 Processor -- start 2024-04-02T07:56:12.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.978+0000 7fbec0c00700 1 -- start start 2024-04-02T07:56:12.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.979+0000 7fbec0c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbebc15d760 0x7fbebc0d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:12.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.979+0000 7fbec0c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbebc15db40 0x7fbebc0cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:12.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.979+0000 7fbec0c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc0d1330 con 0x7fbebc0cf1f0 2024-04-02T07:56:12.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.979+0000 7fbec0c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbebc0d1fb0 con 0x7fbebc15d760 2024-04-02T07:56:12.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.979+0000 7fbec0c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbebc0cf660 con 0x7fbebc15db40 2024-04-02T07:56:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.979+0000 7fbeb9e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbebc15d760 0x7fbebc0d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.979+0000 7fbeba800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbebc0cf1f0 0x7fbebc0cffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49908/0 (socket says 172.21.15.89:49908) 2024-04-02T07:56:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.979+0000 7fbeba800700 1 -- 172.21.15.89:0/1728958226 learned_addr learned my addr 172.21.15.89:0/1728958226 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.979+0000 7fbeb9400700 1 --2- 172.21.15.89:0/1728958226 >> v2:172.21.15.113:3300/0 conn(0x7fbebc15db40 0x7fbebc0cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.979+0000 7fbeb9e00700 1 -- 172.21.15.89:0/1728958226 >> v2:172.21.15.113:3300/0 conn(0x7fbebc15db40 msgr2=0x7fbebc0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.980+0000 7fbeb9e00700 1 --2- 172.21.15.89:0/1728958226 >> v2:172.21.15.113:3300/0 conn(0x7fbebc15db40 0x7fbebc0cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.980+0000 7fbeb9e00700 1 -- 172.21.15.89:0/1728958226 >> v1:172.21.15.89:6789/0 conn(0x7fbebc0cf1f0 legacy=0x7fbebc0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.980+0000 7fbeb9e00700 1 -- 172.21.15.89:0/1728958226 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc0d1610 con 0x7fbebc15d760 2024-04-02T07:56:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.980+0000 7fbeb8a00700 1 -- 172.21.15.89:0/1728958226 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3045464825 0 0) 0x7fbebc0d1330 con 0x7fbebc0cf1f0 2024-04-02T07:56:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.980+0000 7fbeb9e00700 1 --2- 172.21.15.89:0/1728958226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc15d760 0x7fbebc0d23d0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fbebc08c480 tx=0x7fbeac009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a7103c0d9982bbf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.980+0000 7fbeb8a00700 1 -- 172.21.15.89:0/1728958226 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac011830 con 0x7fbebc15d760 2024-04-02T07:56:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.980+0000 7fbeb8a00700 1 -- 172.21.15.89:0/1728958226 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeac011990 con 0x7fbebc15d760 2024-04-02T07:56:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.980+0000 7fbeb8a00700 1 -- 172.21.15.89:0/1728958226 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac011d20 con 0x7fbebc15d760 2024-04-02T07:56:12.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.981+0000 7fbec0c00700 1 -- 172.21.15.89:0/1728958226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc15d760 msgr2=0x7fbebc0d23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:12.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.981+0000 7fbec0c00700 1 --2- 172.21.15.89:0/1728958226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc15d760 0x7fbebc0d23d0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fbebc08c480 tx=0x7fbeac009b20 comp rx=0 tx=0).stop 2024-04-02T07:56:12.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.981+0000 7fbec0c00700 1 -- 172.21.15.89:0/1728958226 shutdown_connections 2024-04-02T07:56:12.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.981+0000 7fbec0c00700 1 --2- 172.21.15.89:0/1728958226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc15d760 0x7fbebc0d23d0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:12.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.981+0000 7fbec0c00700 1 --2- 172.21.15.89:0/1728958226 >> v2:172.21.15.113:3300/0 conn(0x7fbebc15db40 0x7fbebc0cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:12.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.981+0000 7fbec0c00700 1 -- 172.21.15.89:0/1728958226 >> 172.21.15.89:0/1728958226 conn(0x7fbebc08aff0 msgr2=0x7fbebc0d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:12.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.981+0000 7fbec0c00700 1 -- 172.21.15.89:0/1728958226 shutdown_connections 2024-04-02T07:56:12.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.981+0000 7fbec0c00700 1 -- 172.21.15.89:0/1728958226 wait complete. 2024-04-02T07:56:12.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.982+0000 7fbec0c00700 1 Processor -- start 2024-04-02T07:56:12.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.983+0000 7fbec0c00700 1 -- start start 2024-04-02T07:56:12.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbec0c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc0cf1f0 0x7fbebc0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:12.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbec0c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc15d760 0x7fbebc1468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:12.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbec0c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc15db40 0x7fbebc0b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:12.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbec0c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc0b82b0 con 0x7fbebc15db40 2024-04-02T07:56:12.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbec0c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc1f31b0 con 0x7fbebc15d760 2024-04-02T07:56:12.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbec0c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc0b7950 con 0x7fbebc0cf1f0 2024-04-02T07:56:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbeba800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc15db40 0x7fbebc0b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbeba800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc15db40 0x7fbebc0b50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41180/0 (socket says 172.21.15.89:41180) 2024-04-02T07:56:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbeba800700 1 -- 172.21.15.89:0/1803904664 learned_addr learned my addr 172.21.15.89:0/1803904664 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbeb9e00700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc0cf1f0 0x7fbebc0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbeb9400700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc15d760 0x7fbebc1468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbeba800700 1 -- 172.21.15.89:0/1803904664 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc0cf1f0 msgr2=0x7fbebc0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbeba800700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc0cf1f0 0x7fbebc0ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbeba800700 1 -- 172.21.15.89:0/1803904664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc15d760 msgr2=0x7fbebc1468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbeba800700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc15d760 0x7fbebc1468b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbeba800700 1 -- 172.21.15.89:0/1803904664 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac0094a0 con 0x7fbebc15db40 2024-04-02T07:56:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.984+0000 7fbeb9e00700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc0cf1f0 0x7fbebc0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.985+0000 7fbeba800700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc15db40 0x7fbebc0b50b0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400cc70 tx=0x7fbeb400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.985+0000 7fbeb2000700 1 -- 172.21.15.89:0/1803904664 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb400ada0 con 0x7fbebc15db40 2024-04-02T07:56:12.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.985+0000 7fbeb2000700 1 -- 172.21.15.89:0/1803904664 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb400ce70 con 0x7fbebc15db40 2024-04-02T07:56:12.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.985+0000 7fbec0c00700 1 -- 172.21.15.89:0/1803904664 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbebc0b7bf0 con 0x7fbebc15db40 2024-04-02T07:56:12.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.985+0000 7fbeb2000700 1 -- 172.21.15.89:0/1803904664 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40072f0 con 0x7fbebc15db40 2024-04-02T07:56:12.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.985+0000 7fbec0c00700 1 -- 172.21.15.89:0/1803904664 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbebc0cd480 con 0x7fbebc15db40 2024-04-02T07:56:12.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.986+0000 7fbec0c00700 1 -- 172.21.15.89:0/1803904664 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbebc0b83f0 con 0x7fbebc15db40 2024-04-02T07:56:12.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.987+0000 7fbeb2000700 1 -- 172.21.15.89:0/1803904664 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbeb4007450 con 0x7fbebc15db40 2024-04-02T07:56:12.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.988+0000 7fbeb2000700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbe9c06adb0 0x7fbe9c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:12.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.988+0000 7fbeb2000700 1 -- 172.21.15.89:0/1803904664 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5233+0+0 (secure 0 0 0) 0x7fbeb4089250 con 0x7fbebc15db40 2024-04-02T07:56:12.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.988+0000 7fbeb9e00700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbe9c06adb0 0x7fbe9c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:12.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.989+0000 7fbeb9e00700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbe9c06adb0 0x7fbe9c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbeac0090b0 tx=0x7fbeac008030 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:12.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:12.992+0000 7fbeb2000700 1 -- 172.21.15.89:0/1803904664 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbebc0b83f0 con 0x7fbebc15db40 2024-04-02T07:56:13.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.272+0000 7fbec0c00700 1 -- 172.21.15.89:0/1803904664 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbebc0b83f0 con 0x7fbebc15db40 2024-04-02T07:56:13.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.273+0000 7fbeb2000700 1 -- 172.21.15.89:0/1803904664 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v23) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fbebc0b83f0 con 0x7fbebc15db40 2024-04-02T07:56:13.273 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:13.274 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:56:13.274 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T07:56:13.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.274 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T07:56:13.274 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T07:56:13.274 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.274 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:56:13.274 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:56:13.274 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:56:13.274 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.274 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:56:13.275 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:56:13.276 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:56:13.276 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.276 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.276 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T07:56:13.276 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T07:56:13.276 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.276 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:56:13.276 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:56:13.276 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:56:13.276 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.276 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:56:13.277 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.278 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.278 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T07:56:13.278 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T07:56:13.278 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.278 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:56:13.278 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:56:13.278 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.278 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.278 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T07:56:13.278 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.280 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T07:56:13.280 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T07:56:13.280 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.280 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:56:13.280 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:56:13.280 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.280 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.280 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T07:56:13.280 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T07:56:13.280 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.280 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:56:13.280 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:56:13.281 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.281 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.281 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T07:56:13.281 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T07:56:13.281 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.281 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T07:56:13.281 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T07:56:13.281 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.281 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.282 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T07:56:13.282 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T07:56:13.282 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.282 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:56:13.282 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2024-04-02T07:56:13.282 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T07:56:13.282 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.282 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T07:56:13.282 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T07:56:13.282 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.282 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:56:13.283 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:56:13.283 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:56:13.283 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:56:13.283 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.283 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T07:56:13.283 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T07:56:13.283 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.283 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:56:13.283 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:56:13.283 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:56:13.283 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:56:13.283 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T07:56:13.284 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.285 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:56:13.285 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T07:56:13.285 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T07:56:13.285 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.285 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T07:56:13.285 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T07:56:13.285 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.285 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:56:13.285 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:56:13.285 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T07:56:13.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:13.286 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T07:56:13.286 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T07:56:13.286 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T07:56:13.286 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T07:56:13.286 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T07:56:13.286 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T07:56:13.286 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.286 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:13.286 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T07:56:13.286 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:56:13.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.275+0000 7fbec0c00700 1 -- 172.21.15.89:0/1803904664 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbe9c06adb0 msgr2=0x7fbe9c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:13.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.275+0000 7fbec0c00700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbe9c06adb0 0x7fbe9c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbeac0090b0 tx=0x7fbeac008030 comp rx=0 tx=0).stop 2024-04-02T07:56:13.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.275+0000 7fbec0c00700 1 -- 172.21.15.89:0/1803904664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc15db40 msgr2=0x7fbebc0b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:13.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.275+0000 7fbec0c00700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc15db40 0x7fbebc0b50b0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400cc70 tx=0x7fbeb400a340 comp rx=0 tx=0).stop 2024-04-02T07:56:13.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.275+0000 7fbec0c00700 1 -- 172.21.15.89:0/1803904664 shutdown_connections 2024-04-02T07:56:13.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.275+0000 7fbec0c00700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbe9c06adb0 0x7fbe9c06d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.275+0000 7fbec0c00700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc15db40 0x7fbebc0b50b0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.275+0000 7fbec0c00700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc0cf1f0 0x7fbebc0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.275+0000 7fbec0c00700 1 --2- 172.21.15.89:0/1803904664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc15d760 0x7fbebc1468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.275+0000 7fbec0c00700 1 -- 172.21.15.89:0/1803904664 >> 172.21.15.89:0/1803904664 conn(0x7fbebc08aff0 msgr2=0x7fbebc06c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:13.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.276+0000 7fbec0c00700 1 -- 172.21.15.89:0/1803904664 shutdown_connections 2024-04-02T07:56:13.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.276+0000 7fbec0c00700 1 -- 172.21.15.89:0/1803904664 wait complete. 2024-04-02T07:56:13.294 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:56:13.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.485+0000 7f4733a00700 1 Processor -- start 2024-04-02T07:56:13.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.485+0000 7f4733a00700 1 -- start start 2024-04-02T07:56:13.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.486+0000 7f4733a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f472c160f10 0x7f472c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:13.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.486+0000 7f4733a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f472c0cf230 0x7f472c0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:13.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.486+0000 7f4733a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472c0ceda0 con 0x7f472c161590 2024-04-02T07:56:13.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.486+0000 7f4733a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472c0d23d0 con 0x7f472c160f10 2024-04-02T07:56:13.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.486+0000 7f4733a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472c1612f0 con 0x7f472c0cf230 2024-04-02T07:56:13.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.487+0000 7f4731000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f472c160f10 0x7f472c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:13.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.487+0000 7f4731000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f472c160f10 0x7f472c0d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41192/0 (socket says 172.21.15.89:41192) 2024-04-02T07:56:13.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.487+0000 7f4731000700 1 -- 172.21.15.89:0/392262238 learned_addr learned my addr 172.21.15.89:0/392262238 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:13.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.487+0000 7f472be00700 1 --2- 172.21.15.89:0/392262238 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f472c0cf230 0x7f472c0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:13.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.487+0000 7f4731000700 1 -- 172.21.15.89:0/392262238 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f472c0cf230 msgr2=0x7f472c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:13.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.487+0000 7f4731000700 1 --2- 172.21.15.89:0/392262238 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f472c0cf230 0x7f472c0d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.487+0000 7f4731000700 1 -- 172.21.15.89:0/392262238 >> v1:172.21.15.113:6789/0 conn(0x7f472c161590 legacy=0x7f472c0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:13.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.487+0000 7f4731000700 1 -- 172.21.15.89:0/392262238 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c0ce470 con 0x7f472c160f10 2024-04-02T07:56:13.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.487+0000 7f472be00700 1 --2- 172.21.15.89:0/392262238 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f472c0cf230 0x7f472c0d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:13.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.487+0000 7f4731000700 1 --2- 172.21.15.89:0/392262238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c160f10 0x7f472c0d18b0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f47200098f0 tx=0x7f4720009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14d144571aad3c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:13.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.488+0000 7f472b400700 1 -- 172.21.15.89:0/392262238 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472000b8a0 con 0x7f472c160f10 2024-04-02T07:56:13.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.488+0000 7f472b400700 1 -- 172.21.15.89:0/392262238 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472000ba00 con 0x7f472c160f10 2024-04-02T07:56:13.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.488+0000 7f472b400700 1 -- 172.21.15.89:0/392262238 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472000bd90 con 0x7f472c160f10 2024-04-02T07:56:13.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.488+0000 7f4733a00700 1 -- 172.21.15.89:0/392262238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c160f10 msgr2=0x7f472c0d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:13.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.488+0000 7f4733a00700 1 --2- 172.21.15.89:0/392262238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c160f10 0x7f472c0d18b0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f47200098f0 tx=0x7f4720009c00 comp rx=0 tx=0).stop 2024-04-02T07:56:13.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.488+0000 7f4733a00700 1 -- 172.21.15.89:0/392262238 shutdown_connections 2024-04-02T07:56:13.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.488+0000 7f4733a00700 1 --2- 172.21.15.89:0/392262238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c160f10 0x7f472c0d18b0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.488+0000 7f4733a00700 1 --2- 172.21.15.89:0/392262238 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f472c0cf230 0x7f472c0d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.488+0000 7f4733a00700 1 -- 172.21.15.89:0/392262238 >> 172.21.15.89:0/392262238 conn(0x7f472c08aff0 msgr2=0x7f472c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:13.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.488+0000 7f4733a00700 1 -- 172.21.15.89:0/392262238 shutdown_connections 2024-04-02T07:56:13.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.488+0000 7f4733a00700 1 -- 172.21.15.89:0/392262238 wait complete. 2024-04-02T07:56:13.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.490+0000 7f4733a00700 1 Processor -- start 2024-04-02T07:56:13.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.490+0000 7f4733a00700 1 -- start start 2024-04-02T07:56:13.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.491+0000 7f4733a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f472c0cf230 0x7f472c153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:13.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.491+0000 7f4733a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f472c160f10 0x7f472c1551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:13.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.491+0000 7f4733a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c161590 0x7f472c1556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:13.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.491+0000 7f4733a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0b72a0 con 0x7f472c161590 2024-04-02T07:56:13.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.491+0000 7f4733a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c156520 con 0x7f472c160f10 2024-04-02T07:56:13.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.491+0000 7f4733a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472c156240 con 0x7f472c0cf230 2024-04-02T07:56:13.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.491+0000 7f4731000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f472c0cf230 0x7f472c153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:13.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.491+0000 7f4731a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c161590 0x7f472c1556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:13.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.491+0000 7f4731000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f472c0cf230 0x7f472c153580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:58286/0 (socket says 172.21.15.89:58286) 2024-04-02T07:56:13.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.491+0000 7f4731000700 1 -- 172.21.15.89:0/4237692808 learned_addr learned my addr 172.21.15.89:0/4237692808 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:13.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.492+0000 7f472be00700 1 --2- 172.21.15.89:0/4237692808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f472c160f10 0x7f472c1551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:13.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.492+0000 7f4731a00700 1 -- 172.21.15.89:0/4237692808 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f472c0cf230 msgr2=0x7f472c153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.492+0000 7f4731a00700 1 --2- 172.21.15.89:0/4237692808 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f472c0cf230 0x7f472c153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.492+0000 7f4731a00700 1 -- 172.21.15.89:0/4237692808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f472c160f10 msgr2=0x7f472c1551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.492+0000 7f4731a00700 1 --2- 172.21.15.89:0/4237692808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f472c160f10 0x7f472c1551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.492+0000 7f4731a00700 1 -- 172.21.15.89:0/4237692808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4720009550 con 0x7f472c161590 2024-04-02T07:56:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.492+0000 7f4731000700 1 --2- 172.21.15.89:0/4237692808 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f472c0cf230 0x7f472c153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:56:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.492+0000 7f4731a00700 1 --2- 172.21.15.89:0/4237692808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c161590 0x7f472c1556e0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f472400cae0 tx=0x7f472400cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.492+0000 7f4728c00700 1 -- 172.21.15.89:0/4237692808 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4724003c00 con 0x7f472c161590 2024-04-02T07:56:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.492+0000 7f4728c00700 1 -- 172.21.15.89:0/4237692808 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4724003d60 con 0x7f472c161590 2024-04-02T07:56:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.492+0000 7f4733a00700 1 -- 172.21.15.89:0/4237692808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472c0b6c50 con 0x7f472c161590 2024-04-02T07:56:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.492+0000 7f4728c00700 1 -- 172.21.15.89:0/4237692808 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472400aa20 con 0x7f472c161590 2024-04-02T07:56:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.493+0000 7f4733a00700 1 -- 172.21.15.89:0/4237692808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c155c20 con 0x7f472c161590 2024-04-02T07:56:13.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.493+0000 7f4733a00700 1 -- 172.21.15.89:0/4237692808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472c0b7450 con 0x7f472c161590 2024-04-02T07:56:13.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.494+0000 7f4728c00700 1 -- 172.21.15.89:0/4237692808 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f472400ab80 con 0x7f472c161590 2024-04-02T07:56:13.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.495+0000 7f4728c00700 1 --2- 172.21.15.89:0/4237692808 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f470c06adb0 0x7f470c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:13.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.495+0000 7f4728c00700 1 -- 172.21.15.89:0/4237692808 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5233+0+0 (secure 0 0 0) 0x7f47240890e0 con 0x7f472c161590 2024-04-02T07:56:13.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.495+0000 7f4731000700 1 --2- 172.21.15.89:0/4237692808 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f470c06adb0 0x7f470c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:13.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.496+0000 7f4731000700 1 --2- 172.21.15.89:0/4237692808 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f470c06adb0 0x7f470c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4720009130 tx=0x7f4720008030 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:13.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.500+0000 7f4728c00700 1 -- 172.21.15.89:0/4237692808 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4724054370 con 0x7f472c161590 2024-04-02T07:56:13.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.702+0000 7f4733a00700 1 -- 172.21.15.89:0/4237692808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f472c067720 con 0x7f472c161590 2024-04-02T07:56:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.703+0000 7f4728c00700 1 -- 172.21.15.89:0/4237692808 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+12971 (secure 0 0 0) 0x7f4724057990 con 0x7f472c161590 2024-04-02T07:56:13.704 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:13.704 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":150,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:56:12.754846+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:56:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.705+0000 7f4733a00700 1 -- 172.21.15.89:0/4237692808 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f470c06adb0 msgr2=0x7f470c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.705+0000 7f4733a00700 1 --2- 172.21.15.89:0/4237692808 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f470c06adb0 0x7f470c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4720009130 tx=0x7f4720008030 comp rx=0 tx=0).stop 2024-04-02T07:56:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.705+0000 7f4733a00700 1 -- 172.21.15.89:0/4237692808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c161590 msgr2=0x7f472c1556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.705+0000 7f4733a00700 1 --2- 172.21.15.89:0/4237692808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c161590 0x7f472c1556e0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f472400cae0 tx=0x7f472400cdf0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.706+0000 7f4733a00700 1 -- 172.21.15.89:0/4237692808 shutdown_connections 2024-04-02T07:56:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.706+0000 7f4733a00700 1 --2- 172.21.15.89:0/4237692808 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f470c06adb0 0x7f470c06d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.706+0000 7f4733a00700 1 --2- 172.21.15.89:0/4237692808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f472c161590 0x7f472c1556e0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.706+0000 7f4733a00700 1 --2- 172.21.15.89:0/4237692808 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f472c0cf230 0x7f472c153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.706+0000 7f4733a00700 1 --2- 172.21.15.89:0/4237692808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f472c160f10 0x7f472c1551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.706+0000 7f4733a00700 1 -- 172.21.15.89:0/4237692808 >> 172.21.15.89:0/4237692808 conn(0x7f472c08aff0 msgr2=0x7f472c0cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.706+0000 7f4733a00700 1 -- 172.21.15.89:0/4237692808 shutdown_connections 2024-04-02T07:56:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.706+0000 7f4733a00700 1 -- 172.21.15.89:0/4237692808 wait complete. 2024-04-02T07:56:13.725 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T07:56:13.725 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T07:56:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.918+0000 7f069aa00700 1 Processor -- start 2024-04-02T07:56:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.919+0000 7f069aa00700 1 -- start start 2024-04-02T07:56:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.920+0000 7f069aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f069415a5c0 0x7f06940709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.920+0000 7f069aa00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f06940ce840 0x7f06940cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.920+0000 7f069aa00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06940cb310 con 0x7f069415dfe0 2024-04-02T07:56:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.920+0000 7f069aa00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0694070f00 con 0x7f069415a5c0 2024-04-02T07:56:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.920+0000 7f069aa00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06940711a0 con 0x7f06940ce840 2024-04-02T07:56:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.920+0000 7f0693e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f069415a5c0 0x7f06940709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.920+0000 7f0693e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f069415a5c0 0x7f06940709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41206/0 (socket says 172.21.15.89:41206) 2024-04-02T07:56:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.920+0000 7f0693e00700 1 -- 172.21.15.89:0/1577397362 learned_addr learned my addr 172.21.15.89:0/1577397362 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.920+0000 7f0693e00700 1 -- 172.21.15.89:0/1577397362 >> v2:172.21.15.113:3300/0 conn(0x7f06940ce840 msgr2=0x7f06940cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.920+0000 7f0698a00700 1 --2- 172.21.15.89:0/1577397362 >> v2:172.21.15.113:3300/0 conn(0x7f06940ce840 0x7f06940cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.920+0000 7f0693e00700 1 --2- 172.21.15.89:0/1577397362 >> v2:172.21.15.113:3300/0 conn(0x7f06940ce840 0x7f06940cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.920+0000 7f0693e00700 1 -- 172.21.15.89:0/1577397362 >> v1:172.21.15.113:6789/0 conn(0x7f069415dfe0 legacy=0x7f06940ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.920+0000 7f0693e00700 1 -- 172.21.15.89:0/1577397362 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0694071480 con 0x7f069415a5c0 2024-04-02T07:56:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.921+0000 7f0693e00700 1 --2- 172.21.15.89:0/1577397362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f069415a5c0 0x7f06940709c0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f069406b630 tx=0x7f0684009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fda44e16658b69a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.921+0000 7f0692a00700 1 -- 172.21.15.89:0/1577397362 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684004230 con 0x7f069415a5c0 2024-04-02T07:56:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.921+0000 7f0692a00700 1 -- 172.21.15.89:0/1577397362 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0684004390 con 0x7f069415a5c0 2024-04-02T07:56:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.921+0000 7f0692a00700 1 -- 172.21.15.89:0/1577397362 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06840036a0 con 0x7f069415a5c0 2024-04-02T07:56:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.921+0000 7f069aa00700 1 -- 172.21.15.89:0/1577397362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f069415a5c0 msgr2=0x7f06940709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.921+0000 7f069aa00700 1 --2- 172.21.15.89:0/1577397362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f069415a5c0 0x7f06940709c0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f069406b630 tx=0x7f0684009bd0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.921+0000 7f069aa00700 1 -- 172.21.15.89:0/1577397362 shutdown_connections 2024-04-02T07:56:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.922+0000 7f069aa00700 1 --2- 172.21.15.89:0/1577397362 >> v2:172.21.15.113:3300/0 conn(0x7f06940ce840 0x7f06940cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.922+0000 7f069aa00700 1 --2- 172.21.15.89:0/1577397362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f069415a5c0 0x7f06940709c0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.922+0000 7f069aa00700 1 -- 172.21.15.89:0/1577397362 >> 172.21.15.89:0/1577397362 conn(0x7f06940b0d70 msgr2=0x7f06940b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:13.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.922+0000 7f069aa00700 1 -- 172.21.15.89:0/1577397362 shutdown_connections 2024-04-02T07:56:13.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.922+0000 7f069aa00700 1 -- 172.21.15.89:0/1577397362 wait complete. 2024-04-02T07:56:13.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.923+0000 7f069aa00700 1 Processor -- start 2024-04-02T07:56:13.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.923+0000 7f069aa00700 1 -- start start 2024-04-02T07:56:13.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.924+0000 7f069aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06940ce840 0x7f069415c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:13.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.924+0000 7f069aa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f069415a5c0 0x7f069414f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:13.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.924+0000 7f069aa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069415dfe0 0x7f069414f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:13.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.924+0000 7f069aa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069415cce0 con 0x7f06940ce840 2024-04-02T07:56:13.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.924+0000 7f069aa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069415cfc0 con 0x7f069415dfe0 2024-04-02T07:56:13.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.924+0000 7f069aa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069415d2a0 con 0x7f069415a5c0 2024-04-02T07:56:13.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.924+0000 7f0693e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06940ce840 0x7f069415c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:13.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f0693e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06940ce840 0x7f069415c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41212/0 (socket says 172.21.15.89:41212) 2024-04-02T07:56:13.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f0693e00700 1 -- 172.21.15.89:0/3504579335 learned_addr learned my addr 172.21.15.89:0/3504579335 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:13.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f0698a00700 1 --2- 172.21.15.89:0/3504579335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069415dfe0 0x7f069414f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:13.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f0693e00700 1 -- 172.21.15.89:0/3504579335 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f069415a5c0 msgr2=0x7f069414f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:13.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f0693e00700 1 --2- 172.21.15.89:0/3504579335 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f069415a5c0 0x7f069414f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f0693e00700 1 -- 172.21.15.89:0/3504579335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069415dfe0 msgr2=0x7f069414f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:13.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f0693e00700 1 --2- 172.21.15.89:0/3504579335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069415dfe0 0x7f069414f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:13.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f0693e00700 1 -- 172.21.15.89:0/3504579335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0684009550 con 0x7f06940ce840 2024-04-02T07:56:13.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f0698a00700 1 --2- 172.21.15.89:0/3504579335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069415dfe0 0x7f069414f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:13.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f0693e00700 1 --2- 172.21.15.89:0/3504579335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06940ce840 0x7f069415c490 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f0684009130 tx=0x7f0684004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:13.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f068be00700 1 -- 172.21.15.89:0/3504579335 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684003dc0 con 0x7f06940ce840 2024-04-02T07:56:13.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f068be00700 1 -- 172.21.15.89:0/3504579335 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068401f070 con 0x7f06940ce840 2024-04-02T07:56:13.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f068be00700 1 -- 172.21.15.89:0/3504579335 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684011870 con 0x7f06940ce840 2024-04-02T07:56:13.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f069aa00700 1 -- 172.21.15.89:0/3504579335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069409a290 con 0x7f06940ce840 2024-04-02T07:56:13.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.925+0000 7f069aa00700 1 -- 172.21.15.89:0/3504579335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0694099ef0 con 0x7f06940ce840 2024-04-02T07:56:13.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.926+0000 7f069aa00700 1 -- 172.21.15.89:0/3504579335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069409a0e0 con 0x7f06940ce840 2024-04-02T07:56:13.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.927+0000 7f068be00700 1 -- 172.21.15.89:0/3504579335 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0684021030 con 0x7f06940ce840 2024-04-02T07:56:13.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.929+0000 7f068be00700 1 --2- 172.21.15.89:0/3504579335 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f067406adb0 0x7f067406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:13.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.929+0000 7f068be00700 1 -- 172.21.15.89:0/3504579335 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5233+0+0 (secure 0 0 0) 0x7f068408a950 con 0x7f06940ce840 2024-04-02T07:56:13.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.929+0000 7f0693400700 1 --2- 172.21.15.89:0/3504579335 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f067406adb0 0x7f067406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:13.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.929+0000 7f0693400700 1 --2- 172.21.15.89:0/3504579335 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f067406adb0 0x7f067406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f06941f66d0 tx=0x7f0680006c30 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:13.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:13.933+0000 7f068be00700 1 -- 172.21.15.89:0/3504579335 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0684055c80 con 0x7f06940ce840 2024-04-02T07:56:14.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.134+0000 7f069aa00700 1 -- 172.21.15.89:0/3504579335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f069409a4a0 con 0x7f06940ce840 2024-04-02T07:56:14.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.135+0000 7f068be00700 1 -- 172.21.15.89:0/3504579335 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v150) v1 ==== 93+0+10 (secure 0 0 0) 0x7f06840290a0 con 0x7f06940ce840 2024-04-02T07:56:14.135 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T07:56:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.137+0000 7f069aa00700 1 -- 172.21.15.89:0/3504579335 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f067406adb0 msgr2=0x7f067406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.137+0000 7f069aa00700 1 --2- 172.21.15.89:0/3504579335 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f067406adb0 0x7f067406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f06941f66d0 tx=0x7f0680006c30 comp rx=0 tx=0).stop 2024-04-02T07:56:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.137+0000 7f069aa00700 1 -- 172.21.15.89:0/3504579335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06940ce840 msgr2=0x7f069415c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.137+0000 7f069aa00700 1 --2- 172.21.15.89:0/3504579335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06940ce840 0x7f069415c490 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f0684009130 tx=0x7f0684004750 comp rx=0 tx=0).stop 2024-04-02T07:56:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.137+0000 7f069aa00700 1 -- 172.21.15.89:0/3504579335 shutdown_connections 2024-04-02T07:56:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.137+0000 7f069aa00700 1 --2- 172.21.15.89:0/3504579335 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f067406adb0 0x7f067406d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.137+0000 7f069aa00700 1 --2- 172.21.15.89:0/3504579335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069415dfe0 0x7f069414f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.137+0000 7f069aa00700 1 --2- 172.21.15.89:0/3504579335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06940ce840 0x7f069415c490 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.137+0000 7f069aa00700 1 --2- 172.21.15.89:0/3504579335 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f069415a5c0 0x7f069414f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.137+0000 7f069aa00700 1 -- 172.21.15.89:0/3504579335 >> 172.21.15.89:0/3504579335 conn(0x7f06940b0d70 msgr2=0x7f06940cd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:14.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.137+0000 7f069aa00700 1 -- 172.21.15.89:0/3504579335 shutdown_connections 2024-04-02T07:56:14.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.137+0000 7f069aa00700 1 -- 172.21.15.89:0/3504579335 wait complete. 2024-04-02T07:56:14.156 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T07:56:14.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.348+0000 7f2dfa800700 1 Processor -- start 2024-04-02T07:56:14.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.348+0000 7f2dfa800700 1 -- start start 2024-04-02T07:56:14.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.349+0000 7f2dfa800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2df40d18b0 0x7f2df40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:14.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.349+0000 7f2dfa800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2df4161590 0x7f2df4161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:14.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.349+0000 7f2dfa800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df40ceda0 con 0x7f2df4160f10 2024-04-02T07:56:14.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.349+0000 7f2dfa800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df40d1d30 con 0x7f2df4161590 2024-04-02T07:56:14.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.349+0000 7f2dfa800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df4161f60 con 0x7f2df40d18b0 2024-04-02T07:56:14.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.349+0000 7f2df3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2df4161590 0x7f2df4161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:14.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.350+0000 7f2df3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2df4161590 0x7f2df4161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41226/0 (socket says 172.21.15.89:41226) 2024-04-02T07:56:14.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.350+0000 7f2df3e00700 1 -- 172.21.15.89:0/1321137053 learned_addr learned my addr 172.21.15.89:0/1321137053 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:14.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.350+0000 7f2df2a00700 1 --2- 172.21.15.89:0/1321137053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2df40d18b0 0x7f2df40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:14.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.350+0000 7f2df3e00700 1 -- 172.21.15.89:0/1321137053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2df40d18b0 msgr2=0x7f2df40cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:14.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.350+0000 7f2df3e00700 1 --2- 172.21.15.89:0/1321137053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2df40d18b0 0x7f2df40cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.350+0000 7f2df3e00700 1 -- 172.21.15.89:0/1321137053 >> v1:172.21.15.113:6789/0 conn(0x7f2df4160f10 legacy=0x7f2df40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:14.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.350+0000 7f2df3e00700 1 -- 172.21.15.89:0/1321137053 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df41612f0 con 0x7f2df4161590 2024-04-02T07:56:14.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.350+0000 7f2df3e00700 1 --2- 172.21.15.89:0/1321137053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2df4161590 0x7f2df4161a20 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009e80 tx=0x7f2dec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3d3ce313df78b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:14.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.350+0000 7f2df2000700 1 -- 172.21.15.89:0/1321137053 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec00e070 con 0x7f2df4161590 2024-04-02T07:56:14.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.350+0000 7f2df2000700 1 -- 172.21.15.89:0/1321137053 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dec00b980 con 0x7f2df4161590 2024-04-02T07:56:14.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.350+0000 7f2df2000700 1 -- 172.21.15.89:0/1321137053 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec00bd30 con 0x7f2df4161590 2024-04-02T07:56:14.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.351+0000 7f2dfa800700 1 -- 172.21.15.89:0/1321137053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2df4161590 msgr2=0x7f2df4161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:14.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.351+0000 7f2dfa800700 1 --2- 172.21.15.89:0/1321137053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2df4161590 0x7f2df4161a20 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009e80 tx=0x7f2dec00b330 comp rx=0 tx=0).stop 2024-04-02T07:56:14.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.351+0000 7f2dfa800700 1 -- 172.21.15.89:0/1321137053 shutdown_connections 2024-04-02T07:56:14.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.351+0000 7f2dfa800700 1 --2- 172.21.15.89:0/1321137053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2df4161590 0x7f2df4161a20 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.351+0000 7f2dfa800700 1 --2- 172.21.15.89:0/1321137053 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2df40d18b0 0x7f2df40cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.351+0000 7f2dfa800700 1 -- 172.21.15.89:0/1321137053 >> 172.21.15.89:0/1321137053 conn(0x7f2df408aff0 msgr2=0x7f2df40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:14.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.351+0000 7f2dfa800700 1 -- 172.21.15.89:0/1321137053 shutdown_connections 2024-04-02T07:56:14.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.351+0000 7f2dfa800700 1 -- 172.21.15.89:0/1321137053 wait complete. 2024-04-02T07:56:14.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.353+0000 7f2dfa800700 1 Processor -- start 2024-04-02T07:56:14.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.353+0000 7f2dfa800700 1 -- start start 2024-04-02T07:56:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.354+0000 7f2dfa800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2df40d18b0 0x7f2df415cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.354+0000 7f2dfa800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2df4160f10 0x7f2df415e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.354+0000 7f2dfa800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2df4161590 0x7f2df414b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.354+0000 7f2dfa800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df415faf0 con 0x7f2df4161590 2024-04-02T07:56:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.354+0000 7f2dfa800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df41594c0 con 0x7f2df40d18b0 2024-04-02T07:56:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.354+0000 7f2dfa800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df415f4a0 con 0x7f2df4160f10 2024-04-02T07:56:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.354+0000 7f2df3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2df4161590 0x7f2df414b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.354+0000 7f2df3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2df4161590 0x7f2df414b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41238/0 (socket says 172.21.15.89:41238) 2024-04-02T07:56:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.354+0000 7f2df3e00700 1 -- 172.21.15.89:0/2184822339 learned_addr learned my addr 172.21.15.89:0/2184822339 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.354+0000 7f2df2a00700 1 --2- 172.21.15.89:0/2184822339 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2df4160f10 0x7f2df415e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.354+0000 7f2df3400700 1 --2- 172.21.15.89:0/2184822339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2df40d18b0 0x7f2df415cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.355+0000 7f2df3e00700 1 -- 172.21.15.89:0/2184822339 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2df4160f10 msgr2=0x7f2df415e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.355+0000 7f2df3e00700 1 --2- 172.21.15.89:0/2184822339 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2df4160f10 0x7f2df415e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.355+0000 7f2df3e00700 1 -- 172.21.15.89:0/2184822339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2df40d18b0 msgr2=0x7f2df415cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.355+0000 7f2df3e00700 1 --2- 172.21.15.89:0/2184822339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2df40d18b0 0x7f2df415cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.355+0000 7f2df3e00700 1 -- 172.21.15.89:0/2184822339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec009b30 con 0x7f2df4161590 2024-04-02T07:56:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.355+0000 7f2df3e00700 1 --2- 172.21.15.89:0/2184822339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2df4161590 0x7f2df414b490 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f2df408c570 tx=0x7f2dec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.355+0000 7f2deb400700 1 -- 172.21.15.89:0/2184822339 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec004800 con 0x7f2df4161590 2024-04-02T07:56:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.355+0000 7f2deb400700 1 -- 172.21.15.89:0/2184822339 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dec004960 con 0x7f2df4161590 2024-04-02T07:56:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.355+0000 7f2dfa800700 1 -- 172.21.15.89:0/2184822339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df415f780 con 0x7f2df4161590 2024-04-02T07:56:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.355+0000 7f2deb400700 1 -- 172.21.15.89:0/2184822339 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec01c680 con 0x7f2df4161590 2024-04-02T07:56:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.355+0000 7f2dfa800700 1 -- 172.21.15.89:0/2184822339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df4158f40 con 0x7f2df4161590 2024-04-02T07:56:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.356+0000 7f2dfa800700 1 -- 172.21.15.89:0/2184822339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df4067720 con 0x7f2df4161590 2024-04-02T07:56:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.357+0000 7f2deb400700 1 -- 172.21.15.89:0/2184822339 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2dec01c7e0 con 0x7f2df4161590 2024-04-02T07:56:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.358+0000 7f2deb400700 1 --2- 172.21.15.89:0/2184822339 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2dd406adb0 0x7f2dd406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.358+0000 7f2deb400700 1 -- 172.21.15.89:0/2184822339 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5233+0+0 (secure 0 0 0) 0x7f2dec08aba0 con 0x7f2df4161590 2024-04-02T07:56:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.358+0000 7f2df3400700 1 --2- 172.21.15.89:0/2184822339 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2dd406adb0 0x7f2dd406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.359+0000 7f2df3400700 1 --2- 172.21.15.89:0/2184822339 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2dd406adb0 0x7f2dd406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2de4007fd0 tx=0x7f2de4009040 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:14.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.362+0000 7f2deb400700 1 -- 172.21.15.89:0/2184822339 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2dec055d20 con 0x7f2df4161590 2024-04-02T07:56:14.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.564+0000 7f2dfa800700 1 -- 172.21.15.89:0/2184822339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2df415fc30 con 0x7f2df4161590 2024-04-02T07:56:14.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.565+0000 7f2deb400700 1 -- 172.21.15.89:0/2184822339 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v150) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2dec059340 con 0x7f2df4161590 2024-04-02T07:56:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:14.565 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:56:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T07:56:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T07:56:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:14.566 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:56:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.567+0000 7f2dfa800700 1 -- 172.21.15.89:0/2184822339 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2dd406adb0 msgr2=0x7f2dd406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.567+0000 7f2dfa800700 1 --2- 172.21.15.89:0/2184822339 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2dd406adb0 0x7f2dd406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2de4007fd0 tx=0x7f2de4009040 comp rx=0 tx=0).stop 2024-04-02T07:56:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.567+0000 7f2dfa800700 1 -- 172.21.15.89:0/2184822339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2df4161590 msgr2=0x7f2df414b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.567+0000 7f2dfa800700 1 --2- 172.21.15.89:0/2184822339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2df4161590 0x7f2df414b490 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f2df408c570 tx=0x7f2dec004ad0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.567+0000 7f2dfa800700 1 -- 172.21.15.89:0/2184822339 shutdown_connections 2024-04-02T07:56:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.567+0000 7f2dfa800700 1 --2- 172.21.15.89:0/2184822339 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2dd406adb0 0x7f2dd406d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.567+0000 7f2dfa800700 1 --2- 172.21.15.89:0/2184822339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2df40d18b0 0x7f2df415cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.567+0000 7f2dfa800700 1 --2- 172.21.15.89:0/2184822339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2df4161590 0x7f2df414b490 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.567+0000 7f2dfa800700 1 --2- 172.21.15.89:0/2184822339 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2df4160f10 0x7f2df415e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.567+0000 7f2dfa800700 1 -- 172.21.15.89:0/2184822339 >> 172.21.15.89:0/2184822339 conn(0x7f2df408aff0 msgr2=0x7f2df406ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.567+0000 7f2dfa800700 1 -- 172.21.15.89:0/2184822339 shutdown_connections 2024-04-02T07:56:14.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.567+0000 7f2dfa800700 1 -- 172.21.15.89:0/2184822339 wait complete. 2024-04-02T07:56:14.585 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T07:56:14.585 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T07:56:14.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.782+0000 7fdc1a800700 1 Processor -- start 2024-04-02T07:56:14.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.782+0000 7fdc1a800700 1 -- start start 2024-04-02T07:56:14.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.783+0000 7fdc1a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdc141580c0 0x7fdc1406e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:14.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.783+0000 7fdc1a800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdc1406e7f0 0x7fdc1415ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:14.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.783+0000 7fdc1a800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc14070980 con 0x7fdc1415b160 2024-04-02T07:56:14.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.783+0000 7fdc1a800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc14071570 con 0x7fdc141580c0 2024-04-02T07:56:14.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.783+0000 7fdc1a800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc14071990 con 0x7fdc1406e7f0 2024-04-02T07:56:14.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.783+0000 7fdc13400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdc141580c0 0x7fdc1406e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:14.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.783+0000 7fdc13e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdc1415b160 0x7fdc1415df60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49916/0 (socket says 172.21.15.89:49916) 2024-04-02T07:56:14.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.783+0000 7fdc13e00700 1 -- 172.21.15.89:0/2036429490 learned_addr learned my addr 172.21.15.89:0/2036429490 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:14.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.784+0000 7fdc12a00700 1 --2- 172.21.15.89:0/2036429490 >> v2:172.21.15.113:3300/0 conn(0x7fdc1406e7f0 0x7fdc1415ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:14.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.784+0000 7fdc12000700 1 -- 172.21.15.89:0/2036429490 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879146054 0 0) 0x7fdc14070980 con 0x7fdc1415b160 2024-04-02T07:56:14.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.784+0000 7fdc13400700 1 -- 172.21.15.89:0/2036429490 >> v2:172.21.15.113:3300/0 conn(0x7fdc1406e7f0 msgr2=0x7fdc1415ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:14.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.784+0000 7fdc13400700 1 --2- 172.21.15.89:0/2036429490 >> v2:172.21.15.113:3300/0 conn(0x7fdc1406e7f0 0x7fdc1415ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.784+0000 7fdc13400700 1 -- 172.21.15.89:0/2036429490 >> v1:172.21.15.89:6789/0 conn(0x7fdc1415b160 legacy=0x7fdc1415df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:14.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.784+0000 7fdc13400700 1 -- 172.21.15.89:0/2036429490 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc14070bd0 con 0x7fdc141580c0 2024-04-02T07:56:14.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.784+0000 7fdc13400700 1 --2- 172.21.15.89:0/2036429490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc141580c0 0x7fdc1406e2b0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fdc1406ee00 tx=0x7fdc04009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af76e6bf90757307 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:14.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.784+0000 7fdc12000700 1 -- 172.21.15.89:0/2036429490 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04011830 con 0x7fdc141580c0 2024-04-02T07:56:14.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.784+0000 7fdc12000700 1 -- 172.21.15.89:0/2036429490 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc04011990 con 0x7fdc141580c0 2024-04-02T07:56:14.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.784+0000 7fdc12000700 1 -- 172.21.15.89:0/2036429490 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04011d20 con 0x7fdc141580c0 2024-04-02T07:56:14.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.785+0000 7fdc1a800700 1 -- 172.21.15.89:0/2036429490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc141580c0 msgr2=0x7fdc1406e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:14.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.785+0000 7fdc1a800700 1 --2- 172.21.15.89:0/2036429490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc141580c0 0x7fdc1406e2b0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fdc1406ee00 tx=0x7fdc04009b20 comp rx=0 tx=0).stop 2024-04-02T07:56:14.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.785+0000 7fdc1a800700 1 -- 172.21.15.89:0/2036429490 shutdown_connections 2024-04-02T07:56:14.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.785+0000 7fdc1a800700 1 --2- 172.21.15.89:0/2036429490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc141580c0 0x7fdc1406e2b0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.785+0000 7fdc1a800700 1 --2- 172.21.15.89:0/2036429490 >> v2:172.21.15.113:3300/0 conn(0x7fdc1406e7f0 0x7fdc1415ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.785+0000 7fdc1a800700 1 -- 172.21.15.89:0/2036429490 >> 172.21.15.89:0/2036429490 conn(0x7fdc140cf210 msgr2=0x7fdc140cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:14.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.785+0000 7fdc1a800700 1 -- 172.21.15.89:0/2036429490 shutdown_connections 2024-04-02T07:56:14.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.785+0000 7fdc1a800700 1 -- 172.21.15.89:0/2036429490 wait complete. 2024-04-02T07:56:14.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.787+0000 7fdc1a800700 1 Processor -- start 2024-04-02T07:56:14.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.788+0000 7fdc1a800700 1 -- start start 2024-04-02T07:56:14.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc1a800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc1406e7f0 0x7fdc14155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:14.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc1a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc141580c0 0x7fdc140cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:14.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc1a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc1415b160 0x7fdc14159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:14.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc1a800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1415a8e0 con 0x7fdc141580c0 2024-04-02T07:56:14.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc1a800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc14156130 con 0x7fdc1406e7f0 2024-04-02T07:56:14.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc1a800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1415ae10 con 0x7fdc1415b160 2024-04-02T07:56:14.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc12a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc141580c0 0x7fdc140cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:14.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc12a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc141580c0 0x7fdc140cc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41270/0 (socket says 172.21.15.89:41270) 2024-04-02T07:56:14.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc12a00700 1 -- 172.21.15.89:0/1304078025 learned_addr learned my addr 172.21.15.89:0/1304078025 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:14.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc13400700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc1406e7f0 0x7fdc14155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:14.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc13e00700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc1415b160 0x7fdc14159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:14.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc12a00700 1 -- 172.21.15.89:0/1304078025 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc1415b160 msgr2=0x7fdc14159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:14.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc12a00700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc1415b160 0x7fdc14159550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc12a00700 1 -- 172.21.15.89:0/1304078025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc1406e7f0 msgr2=0x7fdc14155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:14.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc12a00700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc1406e7f0 0x7fdc14155bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:14.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc12a00700 1 -- 172.21.15.89:0/1304078025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc040094a0 con 0x7fdc141580c0 2024-04-02T07:56:14.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.789+0000 7fdc13e00700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc1415b160 0x7fdc14159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:14.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.790+0000 7fdc12a00700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc141580c0 0x7fdc140cc1a0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fdc0000e9b0 tx=0x7fdc0000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:14.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.790+0000 7fdc0b400700 1 -- 172.21.15.89:0/1304078025 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc00009710 con 0x7fdc141580c0 2024-04-02T07:56:14.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.790+0000 7fdc0b400700 1 -- 172.21.15.89:0/1304078025 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0000c850 con 0x7fdc141580c0 2024-04-02T07:56:14.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.790+0000 7fdc1a800700 1 -- 172.21.15.89:0/1304078025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1415ab30 con 0x7fdc141580c0 2024-04-02T07:56:14.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.790+0000 7fdc0b400700 1 -- 172.21.15.89:0/1304078025 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0000cbc0 con 0x7fdc141580c0 2024-04-02T07:56:14.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.790+0000 7fdc1a800700 1 -- 172.21.15.89:0/1304078025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc140cbc80 con 0x7fdc141580c0 2024-04-02T07:56:14.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.791+0000 7fdc1a800700 1 -- 172.21.15.89:0/1304078025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc140b4710 con 0x7fdc141580c0 2024-04-02T07:56:14.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.792+0000 7fdc0b400700 1 -- 172.21.15.89:0/1304078025 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdc0000cd20 con 0x7fdc141580c0 2024-04-02T07:56:14.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.793+0000 7fdc0b400700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdbf406adb0 0x7fdbf406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:14.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.793+0000 7fdc0b400700 1 -- 172.21.15.89:0/1304078025 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5233+0+0 (secure 0 0 0) 0x7fdc00089440 con 0x7fdc141580c0 2024-04-02T07:56:14.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.793+0000 7fdc13400700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdbf406adb0 0x7fdbf406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:14.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.794+0000 7fdc13400700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdbf406adb0 0x7fdbf406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdc040090b0 tx=0x7fdc04008030 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:14.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.797+0000 7fdc0b400700 1 -- 172.21.15.89:0/1304078025 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdc000546d0 con 0x7fdc141580c0 2024-04-02T07:56:14.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:14.998+0000 7fdc1a800700 1 -- 172.21.15.89:0/1304078025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdc140b2f10 con 0x7fdc141580c0 2024-04-02T07:56:15.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.692+0000 7fdc0b400700 1 -- 172.21.15.89:0/1304078025 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v151) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdc00057cf0 con 0x7fdc141580c0 2024-04-02T07:56:15.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.697+0000 7fdc1a800700 1 -- 172.21.15.89:0/1304078025 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdbf406adb0 msgr2=0x7fdbf406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:15.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.697+0000 7fdc1a800700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdbf406adb0 0x7fdbf406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdc040090b0 tx=0x7fdc04008030 comp rx=0 tx=0).stop 2024-04-02T07:56:15.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.697+0000 7fdc1a800700 1 -- 172.21.15.89:0/1304078025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc141580c0 msgr2=0x7fdc140cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:15.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.697+0000 7fdc1a800700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc141580c0 0x7fdc140cc1a0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fdc0000e9b0 tx=0x7fdc0000ed70 comp rx=0 tx=0).stop 2024-04-02T07:56:15.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.697+0000 7fdc1a800700 1 -- 172.21.15.89:0/1304078025 shutdown_connections 2024-04-02T07:56:15.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.697+0000 7fdc1a800700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdbf406adb0 0x7fdbf406d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:15.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.697+0000 7fdc1a800700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc1406e7f0 0x7fdc14155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:15.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.697+0000 7fdc1a800700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdc1415b160 0x7fdc14159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:15.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.697+0000 7fdc1a800700 1 --2- 172.21.15.89:0/1304078025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc141580c0 0x7fdc140cc1a0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:15.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.697+0000 7fdc1a800700 1 -- 172.21.15.89:0/1304078025 >> 172.21.15.89:0/1304078025 conn(0x7fdc140cf210 msgr2=0x7fdc140b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:15.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.698+0000 7fdc1a800700 1 -- 172.21.15.89:0/1304078025 shutdown_connections 2024-04-02T07:56:15.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.698+0000 7fdc1a800700 1 -- 172.21.15.89:0/1304078025 wait complete. 2024-04-02T07:56:15.701 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T07:56:15.722 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T07:56:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.922+0000 7feedfe00700 1 Processor -- start 2024-04-02T07:56:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.923+0000 7feedfe00700 1 -- start start 2024-04-02T07:56:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.924+0000 7feedfe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7feed815d9d0 0x7feed80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.924+0000 7feedfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7feed80d2540 0x7feed80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.924+0000 7feedfe00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed8161420 con 0x7feed80d16a0 2024-04-02T07:56:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.924+0000 7feedfe00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed815edc0 con 0x7feed80d2540 2024-04-02T07:56:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.924+0000 7feedfe00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed8161700 con 0x7feed815d9d0 2024-04-02T07:56:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.924+0000 7feedca00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7feed80d16a0 0x7feed80d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49924/0 (socket says 172.21.15.89:49924) 2024-04-02T07:56:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.924+0000 7feedca00700 1 -- 172.21.15.89:0/4209114147 learned_addr learned my addr 172.21.15.89:0/4209114147 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.924+0000 7feedde00700 1 --2- 172.21.15.89:0/4209114147 >> v2:172.21.15.89:3300/0 conn(0x7feed80d2540 0x7feed80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.924+0000 7feedd400700 1 --2- 172.21.15.89:0/4209114147 >> v2:172.21.15.113:3300/0 conn(0x7feed815d9d0 0x7feed80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.925+0000 7feedde00700 1 -- 172.21.15.89:0/4209114147 >> v2:172.21.15.113:3300/0 conn(0x7feed815d9d0 msgr2=0x7feed80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.924+0000 7feed7e00700 1 -- 172.21.15.89:0/4209114147 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201913981 0 0) 0x7feed8161420 con 0x7feed80d16a0 2024-04-02T07:56:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.925+0000 7feedde00700 1 --2- 172.21.15.89:0/4209114147 >> v2:172.21.15.113:3300/0 conn(0x7feed815d9d0 0x7feed80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.925+0000 7feedde00700 1 -- 172.21.15.89:0/4209114147 >> v1:172.21.15.89:6789/0 conn(0x7feed80d16a0 legacy=0x7feed80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.925+0000 7feedde00700 1 -- 172.21.15.89:0/4209114147 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed81619e0 con 0x7feed80d2540 2024-04-02T07:56:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.925+0000 7feedde00700 1 --2- 172.21.15.89:0/4209114147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feed80d2540 0x7feed80ce360 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7feed80d0ff0 tx=0x7feed000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d51fdefeeafd91 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.925+0000 7feed7e00700 1 -- 172.21.15.89:0/4209114147 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed000e070 con 0x7feed80d2540 2024-04-02T07:56:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.925+0000 7feed7e00700 1 -- 172.21.15.89:0/4209114147 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed000bbd0 con 0x7feed80d2540 2024-04-02T07:56:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.925+0000 7feed7e00700 1 -- 172.21.15.89:0/4209114147 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0012430 con 0x7feed80d2540 2024-04-02T07:56:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.925+0000 7feedfe00700 1 -- 172.21.15.89:0/4209114147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feed80d2540 msgr2=0x7feed80ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.925+0000 7feedfe00700 1 --2- 172.21.15.89:0/4209114147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feed80d2540 0x7feed80ce360 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7feed80d0ff0 tx=0x7feed000b330 comp rx=0 tx=0).stop 2024-04-02T07:56:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.926+0000 7feedfe00700 1 -- 172.21.15.89:0/4209114147 shutdown_connections 2024-04-02T07:56:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.926+0000 7feedfe00700 1 --2- 172.21.15.89:0/4209114147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feed80d2540 0x7feed80ce360 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.926+0000 7feedfe00700 1 --2- 172.21.15.89:0/4209114147 >> v2:172.21.15.113:3300/0 conn(0x7feed815d9d0 0x7feed80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.926+0000 7feedfe00700 1 -- 172.21.15.89:0/4209114147 >> 172.21.15.89:0/4209114147 conn(0x7feed808aff0 msgr2=0x7feed80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.926+0000 7feedfe00700 1 -- 172.21.15.89:0/4209114147 shutdown_connections 2024-04-02T07:56:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.926+0000 7feedfe00700 1 -- 172.21.15.89:0/4209114147 wait complete. 2024-04-02T07:56:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.927+0000 7feedfe00700 1 Processor -- start 2024-04-02T07:56:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.927+0000 7feedfe00700 1 -- start start 2024-04-02T07:56:15.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.928+0000 7feedfe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feed80d16a0 0x7feed8158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:15.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.928+0000 7feedfe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feed80d2540 0x7feed815a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:15.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.928+0000 7feedfe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feed815d9d0 0x7feed80cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:15.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.928+0000 7feedfe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed80b0fa0 con 0x7feed80d16a0 2024-04-02T07:56:15.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.928+0000 7feedfe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed815b060 con 0x7feed815d9d0 2024-04-02T07:56:15.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.928+0000 7feedfe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed815aaa0 con 0x7feed80d2540 2024-04-02T07:56:15.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.929+0000 7feedd400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feed80d16a0 0x7feed8158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:15.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.929+0000 7feedd400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feed80d16a0 0x7feed8158c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41286/0 (socket says 172.21.15.89:41286) 2024-04-02T07:56:15.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.929+0000 7feedd400700 1 -- 172.21.15.89:0/4054339417 learned_addr learned my addr 172.21.15.89:0/4054339417 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:15.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.929+0000 7feedde00700 1 --2- 172.21.15.89:0/4054339417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feed815d9d0 0x7feed80cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:15.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.929+0000 7feedd400700 1 -- 172.21.15.89:0/4054339417 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feed80d2540 msgr2=0x7feed815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.929+0000 7feedd400700 1 --2- 172.21.15.89:0/4054339417 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feed80d2540 0x7feed815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.929+0000 7feedd400700 1 -- 172.21.15.89:0/4054339417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feed815d9d0 msgr2=0x7feed80cb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.929+0000 7feedd400700 1 --2- 172.21.15.89:0/4054339417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feed815d9d0 0x7feed80cb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.929+0000 7feedd400700 1 -- 172.21.15.89:0/4054339417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed0009a70 con 0x7feed80d16a0 2024-04-02T07:56:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.929+0000 7feedde00700 1 --2- 172.21.15.89:0/4054339417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feed815d9d0 0x7feed80cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.929+0000 7feedd400700 1 --2- 172.21.15.89:0/4054339417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feed80d16a0 0x7feed8158c30 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7feecc00d800 tx=0x7feecc00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.930+0000 7feed5600700 1 -- 172.21.15.89:0/4054339417 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc0125f0 con 0x7feed80d16a0 2024-04-02T07:56:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.930+0000 7feed5600700 1 -- 172.21.15.89:0/4054339417 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feecc012750 con 0x7feed80d16a0 2024-04-02T07:56:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.930+0000 7feed5600700 1 -- 172.21.15.89:0/4054339417 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc012b00 con 0x7feed80d16a0 2024-04-02T07:56:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.930+0000 7feedfe00700 1 -- 172.21.15.89:0/4054339417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed815ad80 con 0x7feed80d16a0 2024-04-02T07:56:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.930+0000 7feedfe00700 1 -- 172.21.15.89:0/4054339417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed80cd310 con 0x7feed80d16a0 2024-04-02T07:56:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.931+0000 7feedfe00700 1 -- 172.21.15.89:0/4054339417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed80d1ca0 con 0x7feed80d16a0 2024-04-02T07:56:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.932+0000 7feed5600700 1 -- 172.21.15.89:0/4054339417 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7feecc012c60 con 0x7feed80d16a0 2024-04-02T07:56:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.933+0000 7feed5600700 1 --2- 172.21.15.89:0/4054339417 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feeb806adb0 0x7feeb806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.933+0000 7feed5600700 1 -- 172.21.15.89:0/4054339417 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5603+0+0 (secure 0 0 0) 0x7feecc0897f0 con 0x7feed80d16a0 2024-04-02T07:56:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.933+0000 7feedca00700 1 --2- 172.21.15.89:0/4054339417 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feeb806adb0 0x7feeb806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:15.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.934+0000 7feedca00700 1 --2- 172.21.15.89:0/4054339417 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feeb806adb0 0x7feeb806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7feec40010e0 tx=0x7feec4007850 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:15.938+0000 7feed5600700 1 -- 172.21.15.89:0/4054339417 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feecc054910 con 0x7feed80d16a0 2024-04-02T07:56:16.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.138+0000 7feedfe00700 1 -- 172.21.15.89:0/4054339417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7feed80b1210 con 0x7feed80d16a0 2024-04-02T07:56:16.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.696+0000 7feed5600700 1 -- 172.21.15.89:0/4054339417 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v152) v1 ==== 170+0+0 (secure 0 0 0) 0x7feecc057f30 con 0x7feed80d16a0 2024-04-02T07:56:16.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.702+0000 7feedfe00700 1 -- 172.21.15.89:0/4054339417 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feeb806adb0 msgr2=0x7feeb806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:16.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.702+0000 7feedfe00700 1 --2- 172.21.15.89:0/4054339417 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feeb806adb0 0x7feeb806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7feec40010e0 tx=0x7feec4007850 comp rx=0 tx=0).stop 2024-04-02T07:56:16.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.702+0000 7feedfe00700 1 -- 172.21.15.89:0/4054339417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feed80d16a0 msgr2=0x7feed8158c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:16.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.702+0000 7feedfe00700 1 --2- 172.21.15.89:0/4054339417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feed80d16a0 0x7feed8158c30 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7feecc00d800 tx=0x7feecc00dbc0 comp rx=0 tx=0).stop 2024-04-02T07:56:16.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.707+0000 7feedfe00700 1 -- 172.21.15.89:0/4054339417 shutdown_connections 2024-04-02T07:56:16.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.707+0000 7feedfe00700 1 --2- 172.21.15.89:0/4054339417 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7feeb806adb0 0x7feeb806d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:16.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.707+0000 7feedfe00700 1 --2- 172.21.15.89:0/4054339417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feed815d9d0 0x7feed80cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:16.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.707+0000 7feedfe00700 1 --2- 172.21.15.89:0/4054339417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feed80d16a0 0x7feed8158c30 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:16.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.707+0000 7feedfe00700 1 --2- 172.21.15.89:0/4054339417 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feed80d2540 0x7feed815a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:16.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.707+0000 7feedfe00700 1 -- 172.21.15.89:0/4054339417 >> 172.21.15.89:0/4054339417 conn(0x7feed808aff0 msgr2=0x7feed8160780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:16.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.707+0000 7feedfe00700 1 -- 172.21.15.89:0/4054339417 shutdown_connections 2024-04-02T07:56:16.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.708+0000 7feedfe00700 1 -- 172.21.15.89:0/4054339417 wait complete. 2024-04-02T07:56:16.710 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T07:56:16.734 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T07:56:16.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.928+0000 7f5ccba00700 1 Processor -- start 2024-04-02T07:56:16.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.928+0000 7f5ccba00700 1 -- start start 2024-04-02T07:56:16.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.929+0000 7f5ccba00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5cc415d9d0 0x7f5cc40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:16.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.929+0000 7f5ccba00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5cc40d16a0 0x7f5cc40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:16.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.929+0000 7f5ccba00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc4161420 con 0x7f5cc40d2370 2024-04-02T07:56:16.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.929+0000 7f5ccba00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc40d27e0 con 0x7f5cc40d16a0 2024-04-02T07:56:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.929+0000 7f5ccba00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc4161700 con 0x7f5cc415d9d0 2024-04-02T07:56:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.929+0000 7f5cc3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5cc40d16a0 0x7f5cc40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.929+0000 7f5cc3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5cc40d16a0 0x7f5cc40d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41298/0 (socket says 172.21.15.89:41298) 2024-04-02T07:56:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.929+0000 7f5cc3e00700 1 -- 172.21.15.89:0/1380782809 learned_addr learned my addr 172.21.15.89:0/1380782809 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.930+0000 7f5cc9000700 1 --2- 172.21.15.89:0/1380782809 >> v2:172.21.15.113:3300/0 conn(0x7f5cc415d9d0 0x7f5cc40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.930+0000 7f5cc3e00700 1 -- 172.21.15.89:0/1380782809 >> v2:172.21.15.113:3300/0 conn(0x7f5cc415d9d0 msgr2=0x7f5cc40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.930+0000 7f5cc3e00700 1 --2- 172.21.15.89:0/1380782809 >> v2:172.21.15.113:3300/0 conn(0x7f5cc415d9d0 0x7f5cc40d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.930+0000 7f5cc3e00700 1 -- 172.21.15.89:0/1380782809 >> v1:172.21.15.113:6789/0 conn(0x7f5cc40d2370 legacy=0x7f5cc415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.930+0000 7f5cc3e00700 1 -- 172.21.15.89:0/1380782809 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc41619e0 con 0x7f5cc40d16a0 2024-04-02T07:56:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.930+0000 7f5cc3e00700 1 --2- 172.21.15.89:0/1380782809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cc40d16a0 0x7f5cc40d1e30 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40d0ff0 tx=0x7f5cb0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=983265d6c8b0dccf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.930+0000 7f5cc3400700 1 -- 172.21.15.89:0/1380782809 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb0005e30 con 0x7f5cc40d16a0 2024-04-02T07:56:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.930+0000 7f5cc3400700 1 -- 172.21.15.89:0/1380782809 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb0006030 con 0x7f5cc40d16a0 2024-04-02T07:56:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.930+0000 7f5cc3400700 1 -- 172.21.15.89:0/1380782809 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb00063e0 con 0x7f5cc40d16a0 2024-04-02T07:56:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.931+0000 7f5ccba00700 1 -- 172.21.15.89:0/1380782809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cc40d16a0 msgr2=0x7f5cc40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:16.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.931+0000 7f5ccba00700 1 --2- 172.21.15.89:0/1380782809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cc40d16a0 0x7f5cc40d1e30 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40d0ff0 tx=0x7f5cb0009bd0 comp rx=0 tx=0).stop 2024-04-02T07:56:16.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.931+0000 7f5ccba00700 1 -- 172.21.15.89:0/1380782809 shutdown_connections 2024-04-02T07:56:16.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.931+0000 7f5ccba00700 1 --2- 172.21.15.89:0/1380782809 >> v2:172.21.15.113:3300/0 conn(0x7f5cc415d9d0 0x7f5cc40d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:16.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.931+0000 7f5ccba00700 1 --2- 172.21.15.89:0/1380782809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cc40d16a0 0x7f5cc40d1e30 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:16.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.931+0000 7f5ccba00700 1 -- 172.21.15.89:0/1380782809 >> 172.21.15.89:0/1380782809 conn(0x7f5cc408aff0 msgr2=0x7f5cc40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:16.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.931+0000 7f5ccba00700 1 -- 172.21.15.89:0/1380782809 shutdown_connections 2024-04-02T07:56:16.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.931+0000 7f5ccba00700 1 -- 172.21.15.89:0/1380782809 wait complete. 2024-04-02T07:56:16.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.933+0000 7f5ccba00700 1 Processor -- start 2024-04-02T07:56:16.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.934+0000 7f5ccba00700 1 -- start start 2024-04-02T07:56:16.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5ccba00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cc40d16a0 0x7f5cc40c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5ccba00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc40d2370 0x7f5cc40b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5ccba00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cc415d9d0 0x7f5cc40b8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5ccba00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc41e77d0 con 0x7f5cc415d9d0 2024-04-02T07:56:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5ccba00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc41e7a70 con 0x7f5cc40d2370 2024-04-02T07:56:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5ccba00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc4158270 con 0x7f5cc40d16a0 2024-04-02T07:56:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5cc9a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cc415d9d0 0x7f5cc40b8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5cc9000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cc40d16a0 0x7f5cc40c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5cc9a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cc415d9d0 0x7f5cc40b8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41306/0 (socket says 172.21.15.89:41306) 2024-04-02T07:56:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5cc9a00700 1 -- 172.21.15.89:0/3693145328 learned_addr learned my addr 172.21.15.89:0/3693145328 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5cc3e00700 1 --2- 172.21.15.89:0/3693145328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc40d2370 0x7f5cc40b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5cc9a00700 1 -- 172.21.15.89:0/3693145328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cc40d16a0 msgr2=0x7f5cc40c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:16.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5cc9a00700 1 --2- 172.21.15.89:0/3693145328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cc40d16a0 0x7f5cc40c9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:16.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5cc9a00700 1 -- 172.21.15.89:0/3693145328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc40d2370 msgr2=0x7f5cc40b7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:16.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5cc9a00700 1 --2- 172.21.15.89:0/3693145328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc40d2370 0x7f5cc40b7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:16.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5cc9a00700 1 -- 172.21.15.89:0/3693145328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb0009500 con 0x7f5cc415d9d0 2024-04-02T07:56:16.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.935+0000 7f5cc3e00700 1 --2- 172.21.15.89:0/3693145328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc40d2370 0x7f5cc40b7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:16.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.936+0000 7f5cc9a00700 1 --2- 172.21.15.89:0/3693145328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cc415d9d0 0x7f5cc40b8360 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00cc10 tx=0x7f5cbc00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:16.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.936+0000 7f5cc0c00700 1 -- 172.21.15.89:0/3693145328 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00ab10 con 0x7f5cc415d9d0 2024-04-02T07:56:16.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.936+0000 7f5cc0c00700 1 -- 172.21.15.89:0/3693145328 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc00ac70 con 0x7f5cc415d9d0 2024-04-02T07:56:16.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.936+0000 7f5cc0c00700 1 -- 172.21.15.89:0/3693145328 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc012420 con 0x7f5cc415d9d0 2024-04-02T07:56:16.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.936+0000 7f5ccba00700 1 -- 172.21.15.89:0/3693145328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc41579d0 con 0x7f5cc415d9d0 2024-04-02T07:56:16.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.936+0000 7f5ccba00700 1 -- 172.21.15.89:0/3693145328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc4157f90 con 0x7f5cc415d9d0 2024-04-02T07:56:16.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.937+0000 7f5ccba00700 1 -- 172.21.15.89:0/3693145328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc40d1ca0 con 0x7f5cc415d9d0 2024-04-02T07:56:16.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.938+0000 7f5cc0c00700 1 -- 172.21.15.89:0/3693145328 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5cbc012580 con 0x7f5cc415d9d0 2024-04-02T07:56:16.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.939+0000 7f5cc0c00700 1 --2- 172.21.15.89:0/3693145328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5ca406adb0 0x7f5ca406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:16.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.940+0000 7f5cc0c00700 1 -- 172.21.15.89:0/3693145328 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5985+0+0 (secure 0 0 0) 0x7f5cbc0898c0 con 0x7f5cc415d9d0 2024-04-02T07:56:16.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.940+0000 7f5cc9000700 1 --2- 172.21.15.89:0/3693145328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5ca406adb0 0x7f5ca406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:16.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.940+0000 7f5cc9000700 1 --2- 172.21.15.89:0/3693145328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5ca406adb0 0x7f5ca406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8006b20 tx=0x7f5cb8008040 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:16.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:16.945+0000 7f5cc0c00700 1 -- 172.21.15.89:0/3693145328 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5cbc054900 con 0x7f5cc415d9d0 2024-04-02T07:56:17.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.191+0000 7f5ccba00700 1 -- 172.21.15.89:0/3693145328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f5cc4067720 con 0x7f5cc415d9d0 2024-04-02T07:56:17.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.198+0000 7f5cc0c00700 1 -- 172.21.15.89:0/3693145328 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v70) v1 ==== 174+0+0 (secure 0 0 0) 0x7f5cbc057f20 con 0x7f5cc415d9d0 2024-04-02T07:56:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.204+0000 7f5ccba00700 1 -- 172.21.15.89:0/3693145328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5ca406adb0 msgr2=0x7f5ca406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.204+0000 7f5ccba00700 1 --2- 172.21.15.89:0/3693145328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5ca406adb0 0x7f5ca406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8006b20 tx=0x7f5cb8008040 comp rx=0 tx=0).stop 2024-04-02T07:56:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.204+0000 7f5ccba00700 1 -- 172.21.15.89:0/3693145328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cc415d9d0 msgr2=0x7f5cc40b8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.204+0000 7f5ccba00700 1 --2- 172.21.15.89:0/3693145328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cc415d9d0 0x7f5cc40b8360 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00cc10 tx=0x7f5cbc00cf20 comp rx=0 tx=0).stop 2024-04-02T07:56:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.206+0000 7f5ccba00700 1 -- 172.21.15.89:0/3693145328 shutdown_connections 2024-04-02T07:56:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.206+0000 7f5ccba00700 1 --2- 172.21.15.89:0/3693145328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5ca406adb0 0x7f5ca406d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.206+0000 7f5ccba00700 1 --2- 172.21.15.89:0/3693145328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cc415d9d0 0x7f5cc40b8360 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.206+0000 7f5ccba00700 1 --2- 172.21.15.89:0/3693145328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cc40d16a0 0x7f5cc40c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.206+0000 7f5ccba00700 1 --2- 172.21.15.89:0/3693145328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc40d2370 0x7f5cc40b7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.206+0000 7f5ccba00700 1 -- 172.21.15.89:0/3693145328 >> 172.21.15.89:0/3693145328 conn(0x7f5cc408aff0 msgr2=0x7f5cc406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.207+0000 7f5ccba00700 1 -- 172.21.15.89:0/3693145328 shutdown_connections 2024-04-02T07:56:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.207+0000 7f5ccba00700 1 -- 172.21.15.89:0/3693145328 wait complete. 2024-04-02T07:56:17.212 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 18 and data pool 19 2024-04-02T07:56:17.242 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:56:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.432+0000 7fd513800700 1 Processor -- start 2024-04-02T07:56:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.432+0000 7fd513800700 1 -- start start 2024-04-02T07:56:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.433+0000 7fd513800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd50c0ac7d0 0x7fd50c166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.433+0000 7fd513800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c15fd50 con 0x7fd50c0cb3f0 2024-04-02T07:56:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.433+0000 7fd513800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c15f140 con 0x7fd50c15b5d0 2024-04-02T07:56:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.433+0000 7fd513800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd50c15b9b0 con 0x7fd50c0ac7d0 2024-04-02T07:56:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.434+0000 7fd50be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd50c0cb3f0 0x7fd50c0ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49936/0 (socket says 172.21.15.89:49936) 2024-04-02T07:56:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.434+0000 7fd50be00700 1 -- 172.21.15.89:0/3434223725 learned_addr learned my addr 172.21.15.89:0/3434223725 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.434+0000 7fd511800700 1 --2- 172.21.15.89:0/3434223725 >> v2:172.21.15.113:3300/0 conn(0x7fd50c0ac7d0 0x7fd50c166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.434+0000 7fd50b400700 1 -- 172.21.15.89:0/3434223725 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054629550 0 0) 0x7fd50c15fd50 con 0x7fd50c0cb3f0 2024-04-02T07:56:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.434+0000 7fd50b400700 1 -- 172.21.15.89:0/3434223725 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f40035b0 con 0x7fd50c0cb3f0 2024-04-02T07:56:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.434+0000 7fd50b400700 1 -- 172.21.15.89:0/3434223725 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2059620790 0 0) 0x7fd4f40035b0 con 0x7fd50c0cb3f0 2024-04-02T07:56:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.434+0000 7fd50b400700 1 -- 172.21.15.89:0/3434223725 >> v2:172.21.15.113:3300/0 conn(0x7fd50c0ac7d0 msgr2=0x7fd50c166430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.434+0000 7fd50b400700 1 --2- 172.21.15.89:0/3434223725 >> v2:172.21.15.113:3300/0 conn(0x7fd50c0ac7d0 0x7fd50c166430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.434+0000 7fd50b400700 1 -- 172.21.15.89:0/3434223725 >> v1:172.21.15.113:6789/0 conn(0x7fd50c15b5d0 legacy=0x7fd50c0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.434+0000 7fd50b400700 1 -- 172.21.15.89:0/3434223725 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c15f420 con 0x7fd50c0cb3f0 2024-04-02T07:56:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.434+0000 7fd50b400700 1 -- 172.21.15.89:0/3434223725 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fd4f8001670 con 0x7fd50c0cb3f0 2024-04-02T07:56:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.434+0000 7fd50b400700 1 -- 172.21.15.89:0/3434223725 >> v1:172.21.15.89:6789/0 conn(0x7fd50c0cb3f0 legacy=0x7fd50c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.434+0000 7fd50b400700 1 --2- 172.21.15.89:0/3434223725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd4f4003a00 0x7fd4f4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.434+0000 7fd50b400700 1 --2- 172.21.15.89:0/3434223725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4f4004850 0x7fd4f4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.435+0000 7fd50b400700 1 --2- 172.21.15.89:0/3434223725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4f4005210 0x7fd4f400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.435+0000 7fd50b400700 1 -- 172.21.15.89:0/3434223725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c15b9b0 con 0x7fd4f4004850 2024-04-02T07:56:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.435+0000 7fd50b400700 1 -- 172.21.15.89:0/3434223725 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f400c1b0 con 0x7fd4f4005210 2024-04-02T07:56:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.435+0000 7fd50b400700 1 -- 172.21.15.89:0/3434223725 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f400c370 con 0x7fd4f4003a00 2024-04-02T07:56:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.435+0000 7fd50be00700 1 --2- 172.21.15.89:0/3434223725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4f4004850 0x7fd4f4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.435+0000 7fd510e00700 1 --2- 172.21.15.89:0/3434223725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd4f4003a00 0x7fd4f4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.435+0000 7fd511800700 1 --2- 172.21.15.89:0/3434223725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4f4005210 0x7fd4f400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.435+0000 7fd50be00700 1 -- 172.21.15.89:0/3434223725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd4f4003a00 msgr2=0x7fd4f4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.435+0000 7fd50be00700 1 --2- 172.21.15.89:0/3434223725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd4f4003a00 0x7fd4f4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.435+0000 7fd50be00700 1 -- 172.21.15.89:0/3434223725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4f4005210 msgr2=0x7fd4f400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.435+0000 7fd50be00700 1 --2- 172.21.15.89:0/3434223725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4f4005210 0x7fd4f400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.435+0000 7fd50be00700 1 -- 172.21.15.89:0/3434223725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f40035b0 con 0x7fd4f4004850 2024-04-02T07:56:17.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.435+0000 7fd511800700 1 -- 172.21.15.89:0/3434223725 reap_dead start 2024-04-02T07:56:17.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.435+0000 7fd50be00700 1 --2- 172.21.15.89:0/3434223725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4f4004850 0x7fd4f4004cd0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800c240 tx=0x7fd4f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a58453728fa7a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:17.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.436+0000 7fd50b400700 1 -- 172.21.15.89:0/3434223725 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8007c10 con 0x7fd4f4004850 2024-04-02T07:56:17.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.436+0000 7fd50b400700 1 -- 172.21.15.89:0/3434223725 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f800ce00 con 0x7fd4f4004850 2024-04-02T07:56:17.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.436+0000 7fd50b400700 1 -- 172.21.15.89:0/3434223725 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8005ba0 con 0x7fd4f4004850 2024-04-02T07:56:17.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.436+0000 7fd513800700 1 -- 172.21.15.89:0/3434223725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4f4004850 msgr2=0x7fd4f4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:17.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.436+0000 7fd513800700 1 --2- 172.21.15.89:0/3434223725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4f4004850 0x7fd4f4004cd0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800c240 tx=0x7fd4f4005880 comp rx=0 tx=0).stop 2024-04-02T07:56:17.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.436+0000 7fd513800700 1 -- 172.21.15.89:0/3434223725 shutdown_connections 2024-04-02T07:56:17.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.436+0000 7fd513800700 1 --2- 172.21.15.89:0/3434223725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4f4004850 0x7fd4f4004cd0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.436+0000 7fd513800700 1 -- 172.21.15.89:0/3434223725 >> 172.21.15.89:0/3434223725 conn(0x7fd50c0cf190 msgr2=0x7fd50c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:17.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.436+0000 7fd513800700 1 -- 172.21.15.89:0/3434223725 shutdown_connections 2024-04-02T07:56:17.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.436+0000 7fd513800700 1 -- 172.21.15.89:0/3434223725 wait complete. 2024-04-02T07:56:17.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.438+0000 7fd513800700 1 Processor -- start 2024-04-02T07:56:17.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.438+0000 7fd513800700 1 -- start start 2024-04-02T07:56:17.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.439+0000 7fd513800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4f4004850 0x7fd50c1552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:17.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.439+0000 7fd513800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd50c09a4f0 0x7fd50c157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:17.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.439+0000 7fd513800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd50c14aad0 0x7fd50c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:17.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.439+0000 7fd513800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c09a970 con 0x7fd4f4004850 2024-04-02T07:56:17.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.439+0000 7fd513800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c157ac0 con 0x7fd50c09a4f0 2024-04-02T07:56:17.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.439+0000 7fd513800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c157cf0 con 0x7fd50c14aad0 2024-04-02T07:56:17.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.439+0000 7fd510e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4f4004850 0x7fd50c1552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:17.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.439+0000 7fd510e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4f4004850 0x7fd50c1552e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41310/0 (socket says 172.21.15.89:41310) 2024-04-02T07:56:17.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.439+0000 7fd510e00700 1 -- 172.21.15.89:0/2472158093 learned_addr learned my addr 172.21.15.89:0/2472158093 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:17.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.440+0000 7fd50be00700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd50c09a4f0 0x7fd50c157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:17.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.440+0000 7fd511800700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd50c14aad0 0x7fd50c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:17.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.440+0000 7fd510e00700 1 -- 172.21.15.89:0/2472158093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd50c14aad0 msgr2=0x7fd50c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:17.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.440+0000 7fd510e00700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd50c14aad0 0x7fd50c06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.440+0000 7fd510e00700 1 -- 172.21.15.89:0/2472158093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd50c09a4f0 msgr2=0x7fd50c157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:17.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.440+0000 7fd510e00700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd50c09a4f0 0x7fd50c157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.440+0000 7fd510e00700 1 -- 172.21.15.89:0/2472158093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c157fd0 con 0x7fd4f4004850 2024-04-02T07:56:17.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.440+0000 7fd50be00700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd50c09a4f0 0x7fd50c157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:17.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.440+0000 7fd510e00700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4f4004850 0x7fd50c1552e0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fd50000f7a0 tx=0x7fd50000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:17.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.440+0000 7fd508c00700 1 -- 172.21.15.89:0/2472158093 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5000052e0 con 0x7fd4f4004850 2024-04-02T07:56:17.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.440+0000 7fd508c00700 1 -- 172.21.15.89:0/2472158093 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd500005440 con 0x7fd4f4004850 2024-04-02T07:56:17.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.440+0000 7fd513800700 1 -- 172.21.15.89:0/2472158093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd50c1582b0 con 0x7fd4f4004850 2024-04-02T07:56:17.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.440+0000 7fd508c00700 1 -- 172.21.15.89:0/2472158093 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5000038a0 con 0x7fd4f4004850 2024-04-02T07:56:17.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.440+0000 7fd513800700 1 -- 172.21.15.89:0/2472158093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd50c158870 con 0x7fd4f4004850 2024-04-02T07:56:17.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.441+0000 7fd513800700 1 -- 172.21.15.89:0/2472158093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50c067720 con 0x7fd4f4004850 2024-04-02T07:56:17.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.442+0000 7fd508c00700 1 -- 172.21.15.89:0/2472158093 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd500003a00 con 0x7fd4f4004850 2024-04-02T07:56:17.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.443+0000 7fd508c00700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd4ec06ade0 0x7fd4ec06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:17.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.443+0000 7fd508c00700 1 -- 172.21.15.89:0/2472158093 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fd5000893b0 con 0x7fd4f4004850 2024-04-02T07:56:17.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.444+0000 7fd50be00700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd4ec06ade0 0x7fd4ec06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:17.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.444+0000 7fd50be00700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd4ec06ade0 0x7fd4ec06d2a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd50c144500 tx=0x7fd4f8007000 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:17.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.448+0000 7fd508c00700 1 -- 172.21.15.89:0/2472158093 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd500054260 con 0x7fd4f4004850 2024-04-02T07:56:17.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.650+0000 7fd513800700 1 -- 172.21.15.89:0/2472158093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd50c06ea60 con 0x7fd4f4004850 2024-04-02T07:56:17.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.651+0000 7fd508c00700 1 -- 172.21.15.89:0/2472158093 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+16644 (secure 0 0 0) 0x7fd500003cb0 con 0x7fd4f4004850 2024-04-02T07:56:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:17.652 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":153,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:56:17.192539+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:56:15.040512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-02T07:56:16.179443+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:56:17.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.654+0000 7fd513800700 1 -- 172.21.15.89:0/2472158093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd4ec06ade0 msgr2=0x7fd4ec06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:17.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.654+0000 7fd513800700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd4ec06ade0 0x7fd4ec06d2a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd50c144500 tx=0x7fd4f8007000 comp rx=0 tx=0).stop 2024-04-02T07:56:17.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.654+0000 7fd513800700 1 -- 172.21.15.89:0/2472158093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4f4004850 msgr2=0x7fd50c1552e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:17.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.654+0000 7fd513800700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4f4004850 0x7fd50c1552e0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fd50000f7a0 tx=0x7fd50000fab0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.654+0000 7fd513800700 1 -- 172.21.15.89:0/2472158093 shutdown_connections 2024-04-02T07:56:17.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.654+0000 7fd513800700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd4ec06ade0 0x7fd4ec06d2a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.654+0000 7fd513800700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd50c14aad0 0x7fd50c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.654+0000 7fd513800700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4f4004850 0x7fd50c1552e0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.654+0000 7fd513800700 1 --2- 172.21.15.89:0/2472158093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd50c09a4f0 0x7fd50c157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.654+0000 7fd513800700 1 -- 172.21.15.89:0/2472158093 >> 172.21.15.89:0/2472158093 conn(0x7fd50c0cf190 msgr2=0x7fd50c155d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:17.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.654+0000 7fd513800700 1 -- 172.21.15.89:0/2472158093 shutdown_connections 2024-04-02T07:56:17.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.654+0000 7fd513800700 1 -- 172.21.15.89:0/2472158093 wait complete. 2024-04-02T07:56:17.673 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:56:17.720 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:56:17.724 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:56:17.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.869+0000 7f7145200700 1 Processor -- start 2024-04-02T07:56:17.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.869+0000 7f7145200700 1 -- start start 2024-04-02T07:56:17.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.871+0000 7f7145200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f714015da50 0x7f71400ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:17.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.871+0000 7f7145200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71400d0cf0 0x7f71400b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.871+0000 7f7145200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71400d24e0 con 0x7f71400cec70 2024-04-02T07:56:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.871+0000 7f7145200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71400b95d0 con 0x7f714015da50 2024-04-02T07:56:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.871+0000 7f7145200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714015de30 con 0x7f71400d0cf0 2024-04-02T07:56:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.871+0000 7f713de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f714015da50 0x7f71400ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.871+0000 7f713de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f714015da50 0x7f71400ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41312/0 (socket says 172.21.15.89:41312) 2024-04-02T07:56:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.871+0000 7f713de00700 1 -- 172.21.15.89:0/2157406083 learned_addr learned my addr 172.21.15.89:0/2157406083 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.872+0000 7f713de00700 1 -- 172.21.15.89:0/2157406083 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71400d0cf0 msgr2=0x7f71400b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.872+0000 7f713e800700 1 --2- 172.21.15.89:0/2157406083 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71400d0cf0 0x7f71400b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.872+0000 7f713de00700 1 --2- 172.21.15.89:0/2157406083 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71400d0cf0 0x7f71400b6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.872+0000 7f713de00700 1 -- 172.21.15.89:0/2157406083 >> v1:172.21.15.89:6789/0 conn(0x7f71400cec70 legacy=0x7f71400d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.872+0000 7f713ca00700 1 -- 172.21.15.89:0/2157406083 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132860449 0 0) 0x7f71400d24e0 con 0x7f71400cec70 2024-04-02T07:56:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.872+0000 7f713de00700 1 -- 172.21.15.89:0/2157406083 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71400d27c0 con 0x7f714015da50 2024-04-02T07:56:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.872+0000 7f713e800700 1 --2- 172.21.15.89:0/2157406083 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71400d0cf0 0x7f71400b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.872+0000 7f713de00700 1 --2- 172.21.15.89:0/2157406083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f714015da50 0x7f71400ce730 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f71300098f0 tx=0x7f7130009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f9eea361600ea9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:17.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.872+0000 7f713ca00700 1 -- 172.21.15.89:0/2157406083 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713000b8a0 con 0x7f714015da50 2024-04-02T07:56:17.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.872+0000 7f713ca00700 1 -- 172.21.15.89:0/2157406083 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713000ba00 con 0x7f714015da50 2024-04-02T07:56:17.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.872+0000 7f713ca00700 1 -- 172.21.15.89:0/2157406083 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713000bd90 con 0x7f714015da50 2024-04-02T07:56:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.873+0000 7f7145200700 1 -- 172.21.15.89:0/2157406083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f714015da50 msgr2=0x7f71400ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.873+0000 7f7145200700 1 --2- 172.21.15.89:0/2157406083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f714015da50 0x7f71400ce730 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f71300098f0 tx=0x7f7130009c00 comp rx=0 tx=0).stop 2024-04-02T07:56:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.873+0000 7f7145200700 1 -- 172.21.15.89:0/2157406083 shutdown_connections 2024-04-02T07:56:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.873+0000 7f7145200700 1 --2- 172.21.15.89:0/2157406083 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71400d0cf0 0x7f71400b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.873+0000 7f7145200700 1 --2- 172.21.15.89:0/2157406083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f714015da50 0x7f71400ce730 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.873+0000 7f7145200700 1 -- 172.21.15.89:0/2157406083 >> 172.21.15.89:0/2157406083 conn(0x7f71400666a0 msgr2=0x7f7140066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.873+0000 7f7145200700 1 -- 172.21.15.89:0/2157406083 shutdown_connections 2024-04-02T07:56:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.873+0000 7f7145200700 1 -- 172.21.15.89:0/2157406083 wait complete. 2024-04-02T07:56:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.875+0000 7f7145200700 1 Processor -- start 2024-04-02T07:56:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.875+0000 7f7145200700 1 -- start start 2024-04-02T07:56:17.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f7145200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71400cec70 0x7f71401f7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:17.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f7145200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71400d0cf0 0x7f7140159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f7145200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f714015da50 0x7f714015f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f7145200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71401f76d0 con 0x7f714015da50 2024-04-02T07:56:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f7145200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714015df70 con 0x7f71400d0cf0 2024-04-02T07:56:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f7145200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714015e210 con 0x7f71400cec70 2024-04-02T07:56:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f713e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f714015da50 0x7f714015f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f713e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f714015da50 0x7f714015f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41314/0 (socket says 172.21.15.89:41314) 2024-04-02T07:56:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f713e800700 1 -- 172.21.15.89:0/394972998 learned_addr learned my addr 172.21.15.89:0/394972998 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f713de00700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71400cec70 0x7f71401f7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f713d400700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71400d0cf0 0x7f7140159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f713e800700 1 -- 172.21.15.89:0/394972998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71400cec70 msgr2=0x7f71401f7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f713e800700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71400cec70 0x7f71401f7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f713e800700 1 -- 172.21.15.89:0/394972998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71400d0cf0 msgr2=0x7f7140159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f713e800700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71400d0cf0 0x7f7140159dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:17.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f713e800700 1 -- 172.21.15.89:0/394972998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7130009550 con 0x7f714015da50 2024-04-02T07:56:17.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f713de00700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71400cec70 0x7f71401f7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:17.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f713d400700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71400d0cf0 0x7f7140159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:17.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.876+0000 7f713e800700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f714015da50 0x7f714015f2b0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f713800ce40 tx=0x7f713800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:17.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.877+0000 7f7136000700 1 -- 172.21.15.89:0/394972998 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713800adb0 con 0x7f714015da50 2024-04-02T07:56:17.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.877+0000 7f7136000700 1 -- 172.21.15.89:0/394972998 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7138015440 con 0x7f714015da50 2024-04-02T07:56:17.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.877+0000 7f7145200700 1 -- 172.21.15.89:0/394972998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714015e4f0 con 0x7f714015da50 2024-04-02T07:56:17.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.877+0000 7f7136000700 1 -- 172.21.15.89:0/394972998 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71380157f0 con 0x7f714015da50 2024-04-02T07:56:17.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.877+0000 7f7145200700 1 -- 172.21.15.89:0/394972998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714015eab0 con 0x7f714015da50 2024-04-02T07:56:17.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.878+0000 7f7145200700 1 -- 172.21.15.89:0/394972998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714015d290 con 0x7f714015da50 2024-04-02T07:56:17.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.879+0000 7f7136000700 1 -- 172.21.15.89:0/394972998 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f713800d430 con 0x7f714015da50 2024-04-02T07:56:17.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.880+0000 7f7136000700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f712006adb0 0x7f712006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:17.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.880+0000 7f7136000700 1 -- 172.21.15.89:0/394972998 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f7138089b90 con 0x7f714015da50 2024-04-02T07:56:17.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.880+0000 7f713de00700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f712006adb0 0x7f712006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:17.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.881+0000 7f713de00700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f712006adb0 0x7f712006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7130000c00 tx=0x7f7130008030 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:17.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:17.884+0000 7f7136000700 1 -- 172.21.15.89:0/394972998 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7138054ad0 con 0x7f714015da50 2024-04-02T07:56:18.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.088+0000 7f7145200700 1 -- 172.21.15.89:0/394972998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7140067720 con 0x7f714015da50 2024-04-02T07:56:18.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.090+0000 7f7136000700 1 -- 172.21.15.89:0/394972998 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+16644 (secure 0 0 0) 0x7f713801a020 con 0x7f714015da50 2024-04-02T07:56:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:18.091 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":153,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:56:17.192539+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:56:15.040512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-02T07:56:16.179443+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:56:18.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.092+0000 7f7145200700 1 -- 172.21.15.89:0/394972998 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f712006adb0 msgr2=0x7f712006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:18.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.092+0000 7f7145200700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f712006adb0 0x7f712006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7130000c00 tx=0x7f7130008030 comp rx=0 tx=0).stop 2024-04-02T07:56:18.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.092+0000 7f7145200700 1 -- 172.21.15.89:0/394972998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f714015da50 msgr2=0x7f714015f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:18.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.092+0000 7f7145200700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f714015da50 0x7f714015f2b0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f713800ce40 tx=0x7f713800cf20 comp rx=0 tx=0).stop 2024-04-02T07:56:18.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.092+0000 7f7145200700 1 -- 172.21.15.89:0/394972998 shutdown_connections 2024-04-02T07:56:18.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.092+0000 7f7145200700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f712006adb0 0x7f712006d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:18.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.092+0000 7f7145200700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f714015da50 0x7f714015f2b0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:18.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.092+0000 7f7145200700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71400cec70 0x7f71401f7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:18.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.092+0000 7f7145200700 1 --2- 172.21.15.89:0/394972998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71400d0cf0 0x7f7140159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:18.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.092+0000 7f7145200700 1 -- 172.21.15.89:0/394972998 >> 172.21.15.89:0/394972998 conn(0x7f71400666a0 msgr2=0x7f71400b32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:18.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.092+0000 7f7145200700 1 -- 172.21.15.89:0/394972998 shutdown_connections 2024-04-02T07:56:18.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.092+0000 7f7145200700 1 -- 172.21.15.89:0/394972998 wait complete. 2024-04-02T07:56:18.111 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T07:56:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.307+0000 7fc9eb000700 1 Processor -- start 2024-04-02T07:56:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.307+0000 7fc9eb000700 1 -- start start 2024-04-02T07:56:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.308+0000 7fc9eb000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc9e415dad0 0x7fc9e40ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.308+0000 7fc9eb000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc9e40d1630 0x7fc9e40d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.308+0000 7fc9eb000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e415f820 con 0x7fc9e40d2000 2024-04-02T07:56:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.308+0000 7fc9eb000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e40cfcc0 con 0x7fc9e40d1630 2024-04-02T07:56:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.308+0000 7fc9eb000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e40cd9f0 con 0x7fc9e415dad0 2024-04-02T07:56:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.308+0000 7fc9e3400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc9e40d1630 0x7fc9e40d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.308+0000 7fc9e3400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc9e40d1630 0x7fc9e40d1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41330/0 (socket says 172.21.15.89:41330) 2024-04-02T07:56:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.308+0000 7fc9e3400700 1 -- 172.21.15.89:0/1346925455 learned_addr learned my addr 172.21.15.89:0/1346925455 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.309+0000 7fc9e3400700 1 -- 172.21.15.89:0/1346925455 >> v2:172.21.15.113:3300/0 conn(0x7fc9e415dad0 msgr2=0x7fc9e40ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.309+0000 7fc9e3e00700 1 --2- 172.21.15.89:0/1346925455 >> v2:172.21.15.113:3300/0 conn(0x7fc9e415dad0 0x7fc9e40ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.309+0000 7fc9e3400700 1 --2- 172.21.15.89:0/1346925455 >> v2:172.21.15.113:3300/0 conn(0x7fc9e415dad0 0x7fc9e40ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.309+0000 7fc9e3400700 1 -- 172.21.15.89:0/1346925455 >> v1:172.21.15.113:6789/0 conn(0x7fc9e40d2000 legacy=0x7fc9e415deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.309+0000 7fc9e3400700 1 -- 172.21.15.89:0/1346925455 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e40d2430 con 0x7fc9e40d1630 2024-04-02T07:56:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.309+0000 7fc9e3e00700 1 --2- 172.21.15.89:0/1346925455 >> v2:172.21.15.113:3300/0 conn(0x7fc9e415dad0 0x7fc9e40ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.309+0000 7fc9e3400700 1 --2- 172.21.15.89:0/1346925455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9e40d1630 0x7fc9e40d1ac0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fc9e408d380 tx=0x7fc9d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc95d6f770543ccc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.309+0000 7fc9e2a00700 1 -- 172.21.15.89:0/1346925455 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0004230 con 0x7fc9e40d1630 2024-04-02T07:56:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.309+0000 7fc9e2a00700 1 -- 172.21.15.89:0/1346925455 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d0004390 con 0x7fc9e40d1630 2024-04-02T07:56:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.309+0000 7fc9e2a00700 1 -- 172.21.15.89:0/1346925455 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d00036a0 con 0x7fc9e40d1630 2024-04-02T07:56:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.310+0000 7fc9eb000700 1 -- 172.21.15.89:0/1346925455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9e40d1630 msgr2=0x7fc9e40d1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.310+0000 7fc9eb000700 1 --2- 172.21.15.89:0/1346925455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9e40d1630 0x7fc9e40d1ac0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fc9e408d380 tx=0x7fc9d0009bd0 comp rx=0 tx=0).stop 2024-04-02T07:56:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.310+0000 7fc9eb000700 1 -- 172.21.15.89:0/1346925455 shutdown_connections 2024-04-02T07:56:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.310+0000 7fc9eb000700 1 --2- 172.21.15.89:0/1346925455 >> v2:172.21.15.113:3300/0 conn(0x7fc9e415dad0 0x7fc9e40ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.310+0000 7fc9eb000700 1 --2- 172.21.15.89:0/1346925455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9e40d1630 0x7fc9e40d1ac0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.310+0000 7fc9eb000700 1 -- 172.21.15.89:0/1346925455 >> 172.21.15.89:0/1346925455 conn(0x7fc9e408aff0 msgr2=0x7fc9e40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:18.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.310+0000 7fc9eb000700 1 -- 172.21.15.89:0/1346925455 shutdown_connections 2024-04-02T07:56:18.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.310+0000 7fc9eb000700 1 -- 172.21.15.89:0/1346925455 wait complete. 2024-04-02T07:56:18.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.312+0000 7fc9eb000700 1 Processor -- start 2024-04-02T07:56:18.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.312+0000 7fc9eb000700 1 -- start start 2024-04-02T07:56:18.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.313+0000 7fc9eb000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9e40d1630 0x7fc9e4150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:18.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.313+0000 7fc9eb000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9e40d2000 0x7fc9e40b6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:18.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.313+0000 7fc9eb000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9e415dad0 0x7fc9e40b8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:18.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.313+0000 7fc9eb000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e40b8330 con 0x7fc9e40d2000 2024-04-02T07:56:18.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.313+0000 7fc9eb000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e40a0b60 con 0x7fc9e415dad0 2024-04-02T07:56:18.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.313+0000 7fc9eb000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e40a0e40 con 0x7fc9e40d1630 2024-04-02T07:56:18.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.313+0000 7fc9e3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9e40d2000 0x7fc9e40b6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:18.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.313+0000 7fc9e3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9e40d2000 0x7fc9e40b6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41346/0 (socket says 172.21.15.89:41346) 2024-04-02T07:56:18.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.313+0000 7fc9e3400700 1 -- 172.21.15.89:0/4051777851 learned_addr learned my addr 172.21.15.89:0/4051777851 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:18.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.313+0000 7fc9e3e00700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9e40d1630 0x7fc9e4150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:18.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.313+0000 7fc9e9000700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9e415dad0 0x7fc9e40b8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:18.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.314+0000 7fc9e3400700 1 -- 172.21.15.89:0/4051777851 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9e40d1630 msgr2=0x7fc9e4150310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:18.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.314+0000 7fc9e3400700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9e40d1630 0x7fc9e4150310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:18.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.314+0000 7fc9e3400700 1 -- 172.21.15.89:0/4051777851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9e415dad0 msgr2=0x7fc9e40b8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.314+0000 7fc9e3400700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9e415dad0 0x7fc9e40b8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.314+0000 7fc9e3400700 1 -- 172.21.15.89:0/4051777851 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d0009550 con 0x7fc9e40d2000 2024-04-02T07:56:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.314+0000 7fc9e3e00700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9e40d1630 0x7fc9e4150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.314+0000 7fc9e3400700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9e40d2000 0x7fc9e40b6bb0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fc9e408ce40 tx=0x7fc9d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.314+0000 7fc9dbe00700 1 -- 172.21.15.89:0/4051777851 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0003cd0 con 0x7fc9e40d2000 2024-04-02T07:56:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.314+0000 7fc9dbe00700 1 -- 172.21.15.89:0/4051777851 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d001f070 con 0x7fc9e40d2000 2024-04-02T07:56:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.314+0000 7fc9eb000700 1 -- 172.21.15.89:0/4051777851 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e40a1120 con 0x7fc9e40d2000 2024-04-02T07:56:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.314+0000 7fc9dbe00700 1 -- 172.21.15.89:0/4051777851 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0011870 con 0x7fc9e40d2000 2024-04-02T07:56:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.314+0000 7fc9eb000700 1 -- 172.21.15.89:0/4051777851 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e40b70f0 con 0x7fc9e40d2000 2024-04-02T07:56:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.315+0000 7fc9eb000700 1 -- 172.21.15.89:0/4051777851 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9e40b8470 con 0x7fc9e40d2000 2024-04-02T07:56:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.316+0000 7fc9dbe00700 1 -- 172.21.15.89:0/4051777851 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc9d0021030 con 0x7fc9e40d2000 2024-04-02T07:56:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.318+0000 7fc9dbe00700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc9c406adb0 0x7fc9c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.318+0000 7fc9dbe00700 1 -- 172.21.15.89:0/4051777851 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fc9d008aa00 con 0x7fc9e40d2000 2024-04-02T07:56:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.318+0000 7fc9e3e00700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc9c406adb0 0x7fc9c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:18.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.318+0000 7fc9e3e00700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc9c406adb0 0x7fc9c406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc9d400aa20 tx=0x7fc9d4008040 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:18.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.323+0000 7fc9dbe00700 1 -- 172.21.15.89:0/4051777851 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc9e40b8470 con 0x7fc9e40d2000 2024-04-02T07:56:18.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:18.573+0000 7fc9eb000700 1 -- 172.21.15.89:0/4051777851 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc9e40b8470 con 0x7fc9e40d2000 2024-04-02T07:56:19.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.201+0000 7fc9dbe00700 1 -- 172.21.15.89:0/4051777851 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v72) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc9e40b8470 con 0x7fc9e40d2000 2024-04-02T07:56:19.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.204+0000 7fc9eb000700 1 -- 172.21.15.89:0/4051777851 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc9c406adb0 msgr2=0x7fc9c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:19.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.204+0000 7fc9eb000700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc9c406adb0 0x7fc9c406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc9d400aa20 tx=0x7fc9d4008040 comp rx=0 tx=0).stop 2024-04-02T07:56:19.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.204+0000 7fc9eb000700 1 -- 172.21.15.89:0/4051777851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9e40d2000 msgr2=0x7fc9e40b6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:19.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.204+0000 7fc9eb000700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9e40d2000 0x7fc9e40b6bb0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fc9e408ce40 tx=0x7fc9d0004750 comp rx=0 tx=0).stop 2024-04-02T07:56:19.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.204+0000 7fc9eb000700 1 -- 172.21.15.89:0/4051777851 shutdown_connections 2024-04-02T07:56:19.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.204+0000 7fc9eb000700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc9c406adb0 0x7fc9c406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:19.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.204+0000 7fc9eb000700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9e415dad0 0x7fc9e40b8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:19.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.204+0000 7fc9eb000700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9e40d1630 0x7fc9e4150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:19.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.204+0000 7fc9eb000700 1 --2- 172.21.15.89:0/4051777851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9e40d2000 0x7fc9e40b6bb0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:19.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.204+0000 7fc9eb000700 1 -- 172.21.15.89:0/4051777851 >> 172.21.15.89:0/4051777851 conn(0x7fc9e408aff0 msgr2=0x7fc9e415ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:19.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.205+0000 7fc9eb000700 1 -- 172.21.15.89:0/4051777851 shutdown_connections 2024-04-02T07:56:19.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.205+0000 7fc9eb000700 1 -- 172.21.15.89:0/4051777851 wait complete. 2024-04-02T07:56:19.225 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T07:56:19.225 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T07:56:19.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.421+0000 7f369a600700 1 Processor -- start 2024-04-02T07:56:19.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.421+0000 7f369a600700 1 -- start start 2024-04-02T07:56:19.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.422+0000 7f369a600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36941611b0 0x7f369415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:19.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.422+0000 7f369a600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36940ce270 con 0x7f36940cee10 2024-04-02T07:56:19.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.422+0000 7f369a600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369415b600 con 0x7f369415f4d0 2024-04-02T07:56:19.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.422+0000 7f369a600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36940d2760 con 0x7f36941611b0 2024-04-02T07:56:19.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.422+0000 7f3693e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f36940cee10 0x7f3694161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58238/0 (socket says 172.21.15.89:58238) 2024-04-02T07:56:19.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.422+0000 7f3693e00700 1 -- 172.21.15.89:0/6827316 learned_addr learned my addr 172.21.15.89:0/6827316 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:19.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3693400700 1 --2- 172.21.15.89:0/6827316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36941611b0 0x7f369415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:19.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319574909 0 0) 0x7f36940ce270 con 0x7f36940cee10 2024-04-02T07:56:19.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36780035b0 con 0x7f36940cee10 2024-04-02T07:56:19.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148375096 0 0) 0x7f369415b600 con 0x7f369415f4d0 2024-04-02T07:56:19.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36940ce270 con 0x7f369415f4d0 2024-04-02T07:56:19.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3866162803 0 0) 0x7f36780035b0 con 0x7f36940cee10 2024-04-02T07:56:19.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36941611b0 msgr2=0x7f369415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:19.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 --2- 172.21.15.89:0/6827316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36941611b0 0x7f369415ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 >> v1:172.21.15.113:6789/0 conn(0x7f369415f4d0 legacy=0x7f36940ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36940d2a40 con 0x7f36940cee10 2024-04-02T07:56:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f368c0019f0 con 0x7f36940cee10 2024-04-02T07:56:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 >> v1:172.21.15.89:6789/0 conn(0x7f36940cee10 legacy=0x7f3694161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 --2- 172.21.15.89:0/6827316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36941611b0 0x7f3678003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 --2- 172.21.15.89:0/6827316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3678004120 0x7f3678004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 --2- 172.21.15.89:0/6827316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3678004d90 0x7f3678005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36940d2760 con 0x7f3678004120 2024-04-02T07:56:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367800baf0 con 0x7f3678004d90 2024-04-02T07:56:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.423+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367800bcb0 con 0x7f36941611b0 2024-04-02T07:56:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.424+0000 7f3692a00700 1 --2- 172.21.15.89:0/6827316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3678004120 0x7f3678004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.424+0000 7f3693400700 1 --2- 172.21.15.89:0/6827316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36941611b0 0x7f3678003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.424+0000 7f3693e00700 1 --2- 172.21.15.89:0/6827316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3678004d90 0x7f3678005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.424+0000 7f3692a00700 1 -- 172.21.15.89:0/6827316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36941611b0 msgr2=0x7f3678003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.424+0000 7f3692a00700 1 --2- 172.21.15.89:0/6827316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36941611b0 0x7f3678003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.424+0000 7f3692a00700 1 -- 172.21.15.89:0/6827316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3678004d90 msgr2=0x7f3678005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.424+0000 7f3692a00700 1 --2- 172.21.15.89:0/6827316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3678004d90 0x7f3678005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.424+0000 7f3692a00700 1 -- 172.21.15.89:0/6827316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36780035b0 con 0x7f3678004120 2024-04-02T07:56:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.424+0000 7f3692a00700 1 --2- 172.21.15.89:0/6827316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3678004120 0x7f3678004850 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f369415f8b0 tx=0x7f3678005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=371e25ec5c21000 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.424+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3680001d80 con 0x7f3678004120 2024-04-02T07:56:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.424+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36800052d0 con 0x7f3678004120 2024-04-02T07:56:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.424+0000 7f3692000700 1 -- 172.21.15.89:0/6827316 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3680001470 con 0x7f3678004120 2024-04-02T07:56:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.425+0000 7f369a600700 1 -- 172.21.15.89:0/6827316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3678004120 msgr2=0x7f3678004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.425+0000 7f369a600700 1 --2- 172.21.15.89:0/6827316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3678004120 0x7f3678004850 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f369415f8b0 tx=0x7f3678005880 comp rx=0 tx=0).stop 2024-04-02T07:56:19.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.425+0000 7f3693e00700 1 -- 172.21.15.89:0/6827316 reap_dead start 2024-04-02T07:56:19.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.425+0000 7f369a600700 1 -- 172.21.15.89:0/6827316 shutdown_connections 2024-04-02T07:56:19.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.425+0000 7f369a600700 1 -- 172.21.15.89:0/6827316 >> 172.21.15.89:0/6827316 conn(0x7f369408aff0 msgr2=0x7f36940d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:19.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.425+0000 7f369a600700 1 -- 172.21.15.89:0/6827316 shutdown_connections 2024-04-02T07:56:19.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.425+0000 7f369a600700 1 -- 172.21.15.89:0/6827316 wait complete. 2024-04-02T07:56:19.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.427+0000 7f369a600700 1 Processor -- start 2024-04-02T07:56:19.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.427+0000 7f369a600700 1 -- start start 2024-04-02T07:56:19.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.428+0000 7f369a600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3678004120 0x7f3694157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:19.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.428+0000 7f369a600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36940cbc30 0x7f36940c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:19.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.428+0000 7f369a600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36941f6800 0x7f3694159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:19.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.428+0000 7f369a600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36940c7280 con 0x7f3678004120 2024-04-02T07:56:19.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.428+0000 7f369a600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36940c9530 con 0x7f36940cbc30 2024-04-02T07:56:19.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.428+0000 7f369a600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36940c9810 con 0x7f36941f6800 2024-04-02T07:56:19.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.428+0000 7f3693400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3678004120 0x7f3694157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.428+0000 7f3693400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3678004120 0x7f3694157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42274/0 (socket says 172.21.15.89:42274) 2024-04-02T07:56:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.428+0000 7f3693400700 1 -- 172.21.15.89:0/1723077314 learned_addr learned my addr 172.21.15.89:0/1723077314 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.428+0000 7f3692a00700 1 --2- 172.21.15.89:0/1723077314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36940cbc30 0x7f36940c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.428+0000 7f3693e00700 1 --2- 172.21.15.89:0/1723077314 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36941f6800 0x7f3694159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.429+0000 7f3693400700 1 -- 172.21.15.89:0/1723077314 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36941f6800 msgr2=0x7f3694159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.429+0000 7f3693400700 1 --2- 172.21.15.89:0/1723077314 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36941f6800 0x7f3694159ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.429+0000 7f3693400700 1 -- 172.21.15.89:0/1723077314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36940cbc30 msgr2=0x7f36940c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.429+0000 7f3693400700 1 --2- 172.21.15.89:0/1723077314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36940cbc30 0x7f36940c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.429+0000 7f3693400700 1 -- 172.21.15.89:0/1723077314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36940c9af0 con 0x7f3678004120 2024-04-02T07:56:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.429+0000 7f3693400700 1 --2- 172.21.15.89:0/1723077314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3678004120 0x7f3694157fd0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f3684011aa0 tx=0x7f3684011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.429+0000 7f368b400700 1 -- 172.21.15.89:0/1723077314 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3684011ee0 con 0x7f3678004120 2024-04-02T07:56:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.429+0000 7f368b400700 1 -- 172.21.15.89:0/1723077314 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368400f940 con 0x7f3678004120 2024-04-02T07:56:19.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.429+0000 7f369a600700 1 -- 172.21.15.89:0/1723077314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36940c9dd0 con 0x7f3678004120 2024-04-02T07:56:19.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.429+0000 7f368b400700 1 -- 172.21.15.89:0/1723077314 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368400fcf0 con 0x7f3678004120 2024-04-02T07:56:19.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.429+0000 7f369a600700 1 -- 172.21.15.89:0/1723077314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36940c91e0 con 0x7f3678004120 2024-04-02T07:56:19.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.430+0000 7f369a600700 1 -- 172.21.15.89:0/1723077314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36940c7560 con 0x7f3678004120 2024-04-02T07:56:19.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.431+0000 7f368b400700 1 -- 172.21.15.89:0/1723077314 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3684014440 con 0x7f3678004120 2024-04-02T07:56:19.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.432+0000 7f368b400700 1 --2- 172.21.15.89:0/1723077314 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f367406adb0 0x7f367406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:19.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.432+0000 7f368b400700 1 -- 172.21.15.89:0/1723077314 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f3684089c90 con 0x7f3678004120 2024-04-02T07:56:19.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.432+0000 7f3692a00700 1 --2- 172.21.15.89:0/1723077314 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f367406adb0 0x7f367406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:19.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.433+0000 7f3692a00700 1 --2- 172.21.15.89:0/1723077314 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f367406adb0 0x7f367406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3680005040 tx=0x7f368000a000 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:19.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.436+0000 7f368b400700 1 -- 172.21.15.89:0/1723077314 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3684054c00 con 0x7f3678004120 2024-04-02T07:56:19.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:19.679+0000 7f369a600700 1 -- 172.21.15.89:0/1723077314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3694067720 con 0x7f3678004120 2024-04-02T07:56:20.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.204+0000 7f368b400700 1 -- 172.21.15.89:0/1723077314 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v73) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3684058220 con 0x7f3678004120 2024-04-02T07:56:20.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.206+0000 7f369a600700 1 -- 172.21.15.89:0/1723077314 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f367406adb0 msgr2=0x7f367406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:20.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.206+0000 7f369a600700 1 --2- 172.21.15.89:0/1723077314 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f367406adb0 0x7f367406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3680005040 tx=0x7f368000a000 comp rx=0 tx=0).stop 2024-04-02T07:56:20.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.207+0000 7f369a600700 1 -- 172.21.15.89:0/1723077314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3678004120 msgr2=0x7f3694157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:20.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.207+0000 7f369a600700 1 --2- 172.21.15.89:0/1723077314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3678004120 0x7f3694157fd0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f3684011aa0 tx=0x7f3684011db0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.207+0000 7f369a600700 1 -- 172.21.15.89:0/1723077314 shutdown_connections 2024-04-02T07:56:20.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.207+0000 7f369a600700 1 --2- 172.21.15.89:0/1723077314 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f367406adb0 0x7f367406d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.207+0000 7f369a600700 1 --2- 172.21.15.89:0/1723077314 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36941f6800 0x7f3694159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.207+0000 7f369a600700 1 --2- 172.21.15.89:0/1723077314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3678004120 0x7f3694157fd0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.207+0000 7f369a600700 1 --2- 172.21.15.89:0/1723077314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36940cbc30 0x7f36940c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.207+0000 7f369a600700 1 -- 172.21.15.89:0/1723077314 >> 172.21.15.89:0/1723077314 conn(0x7f369408aff0 msgr2=0x7f369406c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:20.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.207+0000 7f369a600700 1 -- 172.21.15.89:0/1723077314 shutdown_connections 2024-04-02T07:56:20.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.207+0000 7f369a600700 1 -- 172.21.15.89:0/1723077314 wait complete. 2024-04-02T07:56:20.228 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T07:56:20.228 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:56:20.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.417+0000 7fdd58c00700 1 Processor -- start 2024-04-02T07:56:20.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.417+0000 7fdd58c00700 1 -- start start 2024-04-02T07:56:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.418+0000 7fdd58c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 0x7fdd540d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.418+0000 7fdd58c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdd540cf230 0x7fdd540d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.419+0000 7fdd58c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd540ceda0 con 0x7fdd54161590 2024-04-02T07:56:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.419+0000 7fdd58c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd540d23d0 con 0x7fdd540cf230 2024-04-02T07:56:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.419+0000 7fdd58c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd541612f0 con 0x7fdd54160f10 2024-04-02T07:56:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.419+0000 7fdd52800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdd54161590 0x7fdd540b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58252/0 (socket says 172.21.15.89:58252) 2024-04-02T07:56:20.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.419+0000 7fdd52800700 1 -- 172.21.15.89:0/848907822 learned_addr learned my addr 172.21.15.89:0/848907822 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:20.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.419+0000 7fdd51e00700 1 --2- 172.21.15.89:0/848907822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 0x7fdd540d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.419+0000 7fdd4be00700 1 --2- 172.21.15.89:0/848907822 >> v2:172.21.15.113:3300/0 conn(0x7fdd540cf230 0x7fdd540d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.419+0000 7fdd51400700 1 -- 172.21.15.89:0/848907822 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884433052 0 0) 0x7fdd540ceda0 con 0x7fdd54161590 2024-04-02T07:56:20.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.419+0000 7fdd51400700 1 -- 172.21.15.89:0/848907822 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd380035b0 con 0x7fdd54161590 2024-04-02T07:56:20.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51400700 1 -- 172.21.15.89:0/848907822 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2756742732 0 0) 0x7fdd380035b0 con 0x7fdd54161590 2024-04-02T07:56:20.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51400700 1 -- 172.21.15.89:0/848907822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 msgr2=0x7fdd540d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:20.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51400700 1 --2- 172.21.15.89:0/848907822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 0x7fdd540d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51400700 1 -- 172.21.15.89:0/848907822 >> v2:172.21.15.113:3300/0 conn(0x7fdd540cf230 msgr2=0x7fdd540d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:20.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51400700 1 --2- 172.21.15.89:0/848907822 >> v2:172.21.15.113:3300/0 conn(0x7fdd540cf230 0x7fdd540d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51400700 1 -- 172.21.15.89:0/848907822 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd540ce470 con 0x7fdd54161590 2024-04-02T07:56:20.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51400700 1 -- 172.21.15.89:0/848907822 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fdd4c0019f0 con 0x7fdd54161590 2024-04-02T07:56:20.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51400700 1 -- 172.21.15.89:0/848907822 >> v1:172.21.15.89:6789/0 conn(0x7fdd54161590 legacy=0x7fdd540b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:20.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51400700 1 --2- 172.21.15.89:0/848907822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd38003a00 0x7fdd38003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51400700 1 --2- 172.21.15.89:0/848907822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 0x7fdd38004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51400700 1 --2- 172.21.15.89:0/848907822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd38004d90 0x7fdd38005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51400700 1 -- 172.21.15.89:0/848907822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd540d23d0 con 0x7fdd38003a00 2024-04-02T07:56:20.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51400700 1 -- 172.21.15.89:0/848907822 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd541612f0 con 0x7fdd38004d90 2024-04-02T07:56:20.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51400700 1 -- 172.21.15.89:0/848907822 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3800ba10 con 0x7fdd54160f10 2024-04-02T07:56:20.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51e00700 1 --2- 172.21.15.89:0/848907822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd38003a00 0x7fdd38003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd4be00700 1 --2- 172.21.15.89:0/848907822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 0x7fdd38004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd52800700 1 --2- 172.21.15.89:0/848907822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd38004d90 0x7fdd38005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51e00700 1 -- 172.21.15.89:0/848907822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 msgr2=0x7fdd38004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:20.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51e00700 1 --2- 172.21.15.89:0/848907822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 0x7fdd38004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51e00700 1 -- 172.21.15.89:0/848907822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd38004d90 msgr2=0x7fdd38005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:20.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51e00700 1 --2- 172.21.15.89:0/848907822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd38004d90 0x7fdd38005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd51e00700 1 -- 172.21.15.89:0/848907822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd380035b0 con 0x7fdd38003a00 2024-04-02T07:56:20.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd52800700 1 --2- 172.21.15.89:0/848907822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd38004d90 0x7fdd38005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:20.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.420+0000 7fdd4be00700 1 --2- 172.21.15.89:0/848907822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 0x7fdd38004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:20.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.421+0000 7fdd51e00700 1 --2- 172.21.15.89:0/848907822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd38003a00 0x7fdd38003e20 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009b60 tx=0x7fdd38005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79872fbfce84d267 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:20.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.421+0000 7fdd51400700 1 -- 172.21.15.89:0/848907822 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4400e6c0 con 0x7fdd38003a00 2024-04-02T07:56:20.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.421+0000 7fdd51400700 1 -- 172.21.15.89:0/848907822 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4400e820 con 0x7fdd38003a00 2024-04-02T07:56:20.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.421+0000 7fdd51400700 1 -- 172.21.15.89:0/848907822 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4400eb90 con 0x7fdd38003a00 2024-04-02T07:56:20.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.421+0000 7fdd58c00700 1 -- 172.21.15.89:0/848907822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd38003a00 msgr2=0x7fdd38003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:20.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.421+0000 7fdd58c00700 1 --2- 172.21.15.89:0/848907822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd38003a00 0x7fdd38003e20 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009b60 tx=0x7fdd38005880 comp rx=0 tx=0).stop 2024-04-02T07:56:20.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.421+0000 7fdd52800700 1 -- 172.21.15.89:0/848907822 reap_dead start 2024-04-02T07:56:20.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.422+0000 7fdd58c00700 1 -- 172.21.15.89:0/848907822 shutdown_connections 2024-04-02T07:56:20.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.422+0000 7fdd58c00700 1 -- 172.21.15.89:0/848907822 >> 172.21.15.89:0/848907822 conn(0x7fdd5408aff0 msgr2=0x7fdd540d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:20.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.422+0000 7fdd58c00700 1 -- 172.21.15.89:0/848907822 shutdown_connections 2024-04-02T07:56:20.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.422+0000 7fdd58c00700 1 -- 172.21.15.89:0/848907822 wait complete. 2024-04-02T07:56:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.423+0000 7fdd58c00700 1 Processor -- start 2024-04-02T07:56:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.423+0000 7fdd58c00700 1 -- start start 2024-04-02T07:56:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.424+0000 7fdd58c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd38003a00 0x7fdd541f2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.424+0000 7fdd58c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd5415dd50 0x7fdd541e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.424+0000 7fdd58c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd541e7e30 0x7fdd540b6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.424+0000 7fdd58c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd541e7460 con 0x7fdd38003a00 2024-04-02T07:56:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.424+0000 7fdd58c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd541e7160 con 0x7fdd5415dd50 2024-04-02T07:56:20.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.424+0000 7fdd58c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5415f970 con 0x7fdd541e7e30 2024-04-02T07:56:20.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.425+0000 7fdd51e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd38003a00 0x7fdd541f2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.425+0000 7fdd51e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd38003a00 0x7fdd541f2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42298/0 (socket says 172.21.15.89:42298) 2024-04-02T07:56:20.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.425+0000 7fdd51e00700 1 -- 172.21.15.89:0/3817623355 learned_addr learned my addr 172.21.15.89:0/3817623355 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:20.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.425+0000 7fdd52800700 1 --2- 172.21.15.89:0/3817623355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd541e7e30 0x7fdd540b6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.425+0000 7fdd4be00700 1 --2- 172.21.15.89:0/3817623355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd5415dd50 0x7fdd541e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.425+0000 7fdd51e00700 1 -- 172.21.15.89:0/3817623355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd541e7e30 msgr2=0x7fdd540b6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:20.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.425+0000 7fdd51e00700 1 --2- 172.21.15.89:0/3817623355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd541e7e30 0x7fdd540b6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.425+0000 7fdd51e00700 1 -- 172.21.15.89:0/3817623355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd5415dd50 msgr2=0x7fdd541e78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:20.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.425+0000 7fdd51e00700 1 --2- 172.21.15.89:0/3817623355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd5415dd50 0x7fdd541e78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.425+0000 7fdd51e00700 1 -- 172.21.15.89:0/3817623355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5415fbf0 con 0x7fdd38003a00 2024-04-02T07:56:20.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.425+0000 7fdd51e00700 1 --2- 172.21.15.89:0/3817623355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd38003a00 0x7fdd541f2b40 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400e4b0 tx=0x7fdd440047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:20.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.425+0000 7fdd4a000700 1 -- 172.21.15.89:0/3817623355 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4400e6c0 con 0x7fdd38003a00 2024-04-02T07:56:20.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.426+0000 7fdd4a000700 1 -- 172.21.15.89:0/3817623355 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd44003ab0 con 0x7fdd38003a00 2024-04-02T07:56:20.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.426+0000 7fdd58c00700 1 -- 172.21.15.89:0/3817623355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5415fed0 con 0x7fdd38003a00 2024-04-02T07:56:20.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.426+0000 7fdd4a000700 1 -- 172.21.15.89:0/3817623355 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44003e60 con 0x7fdd38003a00 2024-04-02T07:56:20.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.426+0000 7fdd58c00700 1 -- 172.21.15.89:0/3817623355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd541601b0 con 0x7fdd38003a00 2024-04-02T07:56:20.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.427+0000 7fdd58c00700 1 -- 172.21.15.89:0/3817623355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd540d0e30 con 0x7fdd38003a00 2024-04-02T07:56:20.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.428+0000 7fdd4a000700 1 -- 172.21.15.89:0/3817623355 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdd44013470 con 0x7fdd38003a00 2024-04-02T07:56:20.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.429+0000 7fdd4a000700 1 --2- 172.21.15.89:0/3817623355 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd3006ade0 0x7fdd3006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.429+0000 7fdd4a000700 1 -- 172.21.15.89:0/3817623355 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fdd4408afe0 con 0x7fdd38003a00 2024-04-02T07:56:20.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.429+0000 7fdd4be00700 1 --2- 172.21.15.89:0/3817623355 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd3006ade0 0x7fdd3006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.430+0000 7fdd4be00700 1 --2- 172.21.15.89:0/3817623355 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd3006ade0 0x7fdd3006d2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdd541e8430 tx=0x7fdd3c008040 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:20.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.434+0000 7fdd4a000700 1 -- 172.21.15.89:0/3817623355 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdd440560e0 con 0x7fdd38003a00 2024-04-02T07:56:20.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.677+0000 7fdd58c00700 1 -- 172.21.15.89:0/3817623355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd54067720 con 0x7fdd38003a00 2024-04-02T07:56:20.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.678+0000 7fdd4a000700 1 -- 172.21.15.89:0/3817623355 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fdd44059700 con 0x7fdd38003a00 2024-04-02T07:56:20.680 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:20.680 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7445,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/4112144997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":4112144997},{"type":"v1","addr":"172.21.15.113:6834","nonce":4112144997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":8416,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2047642384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2047642384},{"type":"v1","addr":"172.21.15.113:6835","nonce":2047642384}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":9030,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2201184833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2201184833},{"type":"v1","addr":"172.21.15.89:6835","nonce":2201184833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:56:17.192894+0000","modified":"2024-04-02T07:56:20.200137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9033},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9033":{"gid":9033,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3861363328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3861363328},{"type":"v1","addr":"172.21.15.89:6837","nonce":3861363328}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-02T07:56:20.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.680+0000 7fdd58c00700 1 -- 172.21.15.89:0/3817623355 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd3006ade0 msgr2=0x7fdd3006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:20.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.680+0000 7fdd58c00700 1 --2- 172.21.15.89:0/3817623355 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd3006ade0 0x7fdd3006d2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdd541e8430 tx=0x7fdd3c008040 comp rx=0 tx=0).stop 2024-04-02T07:56:20.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.680+0000 7fdd58c00700 1 -- 172.21.15.89:0/3817623355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd38003a00 msgr2=0x7fdd541f2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:20.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.680+0000 7fdd58c00700 1 --2- 172.21.15.89:0/3817623355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd38003a00 0x7fdd541f2b40 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400e4b0 tx=0x7fdd440047c0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.681+0000 7fdd58c00700 1 -- 172.21.15.89:0/3817623355 shutdown_connections 2024-04-02T07:56:20.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.681+0000 7fdd58c00700 1 --2- 172.21.15.89:0/3817623355 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd3006ade0 0x7fdd3006d2a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.681+0000 7fdd58c00700 1 --2- 172.21.15.89:0/3817623355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd541e7e30 0x7fdd540b6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.681+0000 7fdd58c00700 1 --2- 172.21.15.89:0/3817623355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd38003a00 0x7fdd541f2b40 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.681+0000 7fdd58c00700 1 --2- 172.21.15.89:0/3817623355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd5415dd50 0x7fdd541e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.681+0000 7fdd58c00700 1 -- 172.21.15.89:0/3817623355 >> 172.21.15.89:0/3817623355 conn(0x7fdd5408aff0 msgr2=0x7fdd540cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:20.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.681+0000 7fdd58c00700 1 -- 172.21.15.89:0/3817623355 shutdown_connections 2024-04-02T07:56:20.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.681+0000 7fdd58c00700 1 -- 172.21.15.89:0/3817623355 wait complete. 2024-04-02T07:56:20.683 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 73 2024-04-02T07:56:20.700 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T07:56:20.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.891+0000 7f7c55a00700 1 Processor -- start 2024-04-02T07:56:20.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.891+0000 7f7c55a00700 1 -- start start 2024-04-02T07:56:20.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.892+0000 7f7c55a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c500d18b0 0x7f7c500cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.892+0000 7f7c55a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7c50161590 0x7f7c50161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.892+0000 7f7c55a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c500ceda0 con 0x7f7c50160f10 2024-04-02T07:56:20.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.892+0000 7f7c55a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c500d1d30 con 0x7f7c50161590 2024-04-02T07:56:20.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.892+0000 7f7c55a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c50161f60 con 0x7f7c500d18b0 2024-04-02T07:56:20.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.893+0000 7f7c4ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7c50160f10 0x7f7c500d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58266/0 (socket says 172.21.15.89:58266) 2024-04-02T07:56:20.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.893+0000 7f7c4ea00700 1 -- 172.21.15.89:0/2669836422 learned_addr learned my addr 172.21.15.89:0/2669836422 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:20.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.893+0000 7f7c4f400700 1 --2- 172.21.15.89:0/2669836422 >> v2:172.21.15.113:3300/0 conn(0x7f7c50161590 0x7f7c50161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.893+0000 7f7c47e00700 1 --2- 172.21.15.89:0/2669836422 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c500d18b0 0x7f7c500cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.893+0000 7f7c4e000700 1 -- 172.21.15.89:0/2669836422 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394041089 0 0) 0x7f7c500ceda0 con 0x7f7c50160f10 2024-04-02T07:56:20.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.893+0000 7f7c4e000700 1 -- 172.21.15.89:0/2669836422 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c340035b0 con 0x7f7c50160f10 2024-04-02T07:56:20.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.893+0000 7f7c4e000700 1 -- 172.21.15.89:0/2669836422 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1279950418 0 0) 0x7f7c340035b0 con 0x7f7c50160f10 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.894+0000 7f7c4f400700 1 -- 172.21.15.89:0/2669836422 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c500d18b0 msgr2=0x7f7c500cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.894+0000 7f7c4f400700 1 --2- 172.21.15.89:0/2669836422 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c500d18b0 0x7f7c500cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.894+0000 7f7c4f400700 1 -- 172.21.15.89:0/2669836422 >> v1:172.21.15.89:6789/0 conn(0x7f7c50160f10 legacy=0x7f7c500d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.894+0000 7f7c4f400700 1 -- 172.21.15.89:0/2669836422 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c501612f0 con 0x7f7c50161590 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.894+0000 7f7c47e00700 1 --2- 172.21.15.89:0/2669836422 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c500d18b0 0x7f7c500cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.894+0000 7f7c4f400700 1 --2- 172.21.15.89:0/2669836422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c50161590 0x7f7c50161a20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7c48009dc0 tx=0x7f7c4800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2ab9941f509d78a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.894+0000 7f7c4e000700 1 -- 172.21.15.89:0/2669836422 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4800e070 con 0x7f7c50161590 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.894+0000 7f7c4e000700 1 -- 172.21.15.89:0/2669836422 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c48003a20 con 0x7f7c50161590 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.894+0000 7f7c4e000700 1 -- 172.21.15.89:0/2669836422 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40001670 con 0x7f7c50161590 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.895+0000 7f7c55a00700 1 -- 172.21.15.89:0/2669836422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c50161590 msgr2=0x7f7c50161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.895+0000 7f7c55a00700 1 --2- 172.21.15.89:0/2669836422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c50161590 0x7f7c50161a20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7c48009dc0 tx=0x7f7c4800b330 comp rx=0 tx=0).stop 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.895+0000 7f7c55a00700 1 -- 172.21.15.89:0/2669836422 shutdown_connections 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.895+0000 7f7c55a00700 1 --2- 172.21.15.89:0/2669836422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c50161590 0x7f7c50161a20 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.895+0000 7f7c55a00700 1 --2- 172.21.15.89:0/2669836422 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c500d18b0 0x7f7c500cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.895+0000 7f7c55a00700 1 -- 172.21.15.89:0/2669836422 >> 172.21.15.89:0/2669836422 conn(0x7f7c5008aff0 msgr2=0x7f7c500d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:20.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.895+0000 7f7c55a00700 1 -- 172.21.15.89:0/2669836422 shutdown_connections 2024-04-02T07:56:20.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.895+0000 7f7c55a00700 1 -- 172.21.15.89:0/2669836422 wait complete. 2024-04-02T07:56:20.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.896+0000 7f7c55a00700 1 Processor -- start 2024-04-02T07:56:20.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.897+0000 7f7c55a00700 1 -- start start 2024-04-02T07:56:20.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.897+0000 7f7c55a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c500d18b0 0x7f7c50158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.897+0000 7f7c55a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c50160f10 0x7f7c50147630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.897+0000 7f7c55a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c50161590 0x7f7c50147b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.897+0000 7f7c55a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c50158fa0 con 0x7f7c50160f10 2024-04-02T07:56:20.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.897+0000 7f7c55a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c501461b0 con 0x7f7c50161590 2024-04-02T07:56:20.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.897+0000 7f7c55a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c50146490 con 0x7f7c500d18b0 2024-04-02T07:56:20.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.898+0000 7f7c47e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c50160f10 0x7f7c50147630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.898+0000 7f7c47e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c50160f10 0x7f7c50147630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42306/0 (socket says 172.21.15.89:42306) 2024-04-02T07:56:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.898+0000 7f7c47e00700 1 -- 172.21.15.89:0/3407044520 learned_addr learned my addr 172.21.15.89:0/3407044520 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.898+0000 7f7c4ea00700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c500d18b0 0x7f7c50158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.898+0000 7f7c4f400700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c50161590 0x7f7c50147b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.898+0000 7f7c47e00700 1 -- 172.21.15.89:0/3407044520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c500d18b0 msgr2=0x7f7c50158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.898+0000 7f7c47e00700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c500d18b0 0x7f7c50158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.898+0000 7f7c47e00700 1 -- 172.21.15.89:0/3407044520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c50161590 msgr2=0x7f7c50147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.898+0000 7f7c47e00700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c50161590 0x7f7c50147b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.898+0000 7f7c47e00700 1 -- 172.21.15.89:0/3407044520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c380094a0 con 0x7f7c50160f10 2024-04-02T07:56:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.898+0000 7f7c4f400700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c50161590 0x7f7c50147b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.898+0000 7f7c4ea00700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c500d18b0 0x7f7c50158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.898+0000 7f7c47e00700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c50160f10 0x7f7c50147630 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800ebe0 tx=0x7f7c38009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.899+0000 7f7c46a00700 1 -- 172.21.15.89:0/3407044520 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3800c8b0 con 0x7f7c50160f10 2024-04-02T07:56:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.899+0000 7f7c46a00700 1 -- 172.21.15.89:0/3407044520 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3800ca10 con 0x7f7c50160f10 2024-04-02T07:56:20.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.899+0000 7f7c46a00700 1 -- 172.21.15.89:0/3407044520 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3800cda0 con 0x7f7c50160f10 2024-04-02T07:56:20.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.899+0000 7f7c55a00700 1 -- 172.21.15.89:0/3407044520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c48009a70 con 0x7f7c50160f10 2024-04-02T07:56:20.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.899+0000 7f7c55a00700 1 -- 172.21.15.89:0/3407044520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c50146a50 con 0x7f7c50160f10 2024-04-02T07:56:20.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.900+0000 7f7c55a00700 1 -- 172.21.15.89:0/3407044520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c50067720 con 0x7f7c50160f10 2024-04-02T07:56:20.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.901+0000 7f7c46a00700 1 -- 172.21.15.89:0/3407044520 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7c38013470 con 0x7f7c50160f10 2024-04-02T07:56:20.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.902+0000 7f7c46a00700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c2c06adb0 0x7f7c2c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:20.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.902+0000 7f7c46a00700 1 -- 172.21.15.89:0/3407044520 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f7c38089690 con 0x7f7c50160f10 2024-04-02T07:56:20.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.902+0000 7f7c4ea00700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c2c06adb0 0x7f7c2c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:20.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.903+0000 7f7c4ea00700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c2c06adb0 0x7f7c2c06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7c400017a0 tx=0x7f7c400015f0 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:20.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:20.906+0000 7f7c46a00700 1 -- 172.21.15.89:0/3407044520 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7c38054670 con 0x7f7c50160f10 2024-04-02T07:56:21.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.109+0000 7f7c55a00700 1 -- 172.21.15.89:0/3407044520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c50158990 con 0x7f7c50160f10 2024-04-02T07:56:21.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.110+0000 7f7c46a00700 1 -- 172.21.15.89:0/3407044520 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+16631 (secure 0 0 0) 0x7f7c38087e60 con 0x7f7c50160f10 2024-04-02T07:56:21.112 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:21.112 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":154,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T07:56:18.194811+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:56:15.040512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-02T07:56:16.179443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T07:56:21.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.113+0000 7f7c55a00700 1 -- 172.21.15.89:0/3407044520 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c2c06adb0 msgr2=0x7f7c2c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:21.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.113+0000 7f7c55a00700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c2c06adb0 0x7f7c2c06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7c400017a0 tx=0x7f7c400015f0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.113+0000 7f7c55a00700 1 -- 172.21.15.89:0/3407044520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c50160f10 msgr2=0x7f7c50147630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:21.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.113+0000 7f7c55a00700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c50160f10 0x7f7c50147630 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800ebe0 tx=0x7f7c38009e20 comp rx=0 tx=0).stop 2024-04-02T07:56:21.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.113+0000 7f7c55a00700 1 -- 172.21.15.89:0/3407044520 shutdown_connections 2024-04-02T07:56:21.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.113+0000 7f7c55a00700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c2c06adb0 0x7f7c2c06d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.113+0000 7f7c55a00700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c50161590 0x7f7c50147b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.113+0000 7f7c55a00700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c500d18b0 0x7f7c50158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.113+0000 7f7c55a00700 1 --2- 172.21.15.89:0/3407044520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c50160f10 0x7f7c50147630 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.113+0000 7f7c55a00700 1 -- 172.21.15.89:0/3407044520 >> 172.21.15.89:0/3407044520 conn(0x7f7c5008aff0 msgr2=0x7f7c5006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:21.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.113+0000 7f7c55a00700 1 -- 172.21.15.89:0/3407044520 shutdown_connections 2024-04-02T07:56:21.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.113+0000 7f7c55a00700 1 -- 172.21.15.89:0/3407044520 wait complete. 2024-04-02T07:56:21.133 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T07:56:21.133 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:56:21.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.324+0000 7f0e47800700 1 Processor -- start 2024-04-02T07:56:21.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.324+0000 7f0e47800700 1 -- start start 2024-04-02T07:56:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.325+0000 7f0e47800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e40160f10 0x7f0e400d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.325+0000 7f0e47800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0e400cf230 0x7f0e400d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.325+0000 7f0e47800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0e40161590 0x7f0e40161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.325+0000 7f0e47800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e40161f60 con 0x7f0e40161590 2024-04-02T07:56:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.325+0000 7f0e47800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e400ceda0 con 0x7f0e400cf230 2024-04-02T07:56:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.325+0000 7f0e47800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e400d23d0 con 0x7f0e40160f10 2024-04-02T07:56:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.325+0000 7f0e45800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0e40161590 0x7f0e40161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.325+0000 7f0e45800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0e40161590 0x7f0e40161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42316/0 (socket says 172.21.15.89:42316) 2024-04-02T07:56:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.325+0000 7f0e45800700 1 -- 172.21.15.89:0/1378409487 learned_addr learned my addr 172.21.15.89:0/1378409487 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.325+0000 7f0e44e00700 1 --2- 172.21.15.89:0/1378409487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e40160f10 0x7f0e400d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.325+0000 7f0e3fe00700 1 --2- 172.21.15.89:0/1378409487 >> v2:172.21.15.113:3300/0 conn(0x7f0e400cf230 0x7f0e400d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.326+0000 7f0e45800700 1 -- 172.21.15.89:0/1378409487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e40160f10 msgr2=0x7f0e400d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.326+0000 7f0e45800700 1 --2- 172.21.15.89:0/1378409487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e40160f10 0x7f0e400d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.326+0000 7f0e45800700 1 -- 172.21.15.89:0/1378409487 >> v2:172.21.15.113:3300/0 conn(0x7f0e400cf230 msgr2=0x7f0e400d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.326+0000 7f0e45800700 1 --2- 172.21.15.89:0/1378409487 >> v2:172.21.15.113:3300/0 conn(0x7f0e400cf230 0x7f0e400d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.326+0000 7f0e45800700 1 -- 172.21.15.89:0/1378409487 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e401612f0 con 0x7f0e40161590 2024-04-02T07:56:21.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.326+0000 7f0e45800700 1 --2- 172.21.15.89:0/1378409487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e40161590 0x7f0e40161a20 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f0e4008c1d0 tx=0x7f0e3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fca58937bd1d9de server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:21.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.326+0000 7f0e3f400700 1 -- 172.21.15.89:0/1378409487 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3800e070 con 0x7f0e40161590 2024-04-02T07:56:21.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.326+0000 7f0e3f400700 1 -- 172.21.15.89:0/1378409487 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3800b910 con 0x7f0e40161590 2024-04-02T07:56:21.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.326+0000 7f0e3f400700 1 -- 172.21.15.89:0/1378409487 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3800bcc0 con 0x7f0e40161590 2024-04-02T07:56:21.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.327+0000 7f0e47800700 1 -- 172.21.15.89:0/1378409487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e40161590 msgr2=0x7f0e40161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:21.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.327+0000 7f0e47800700 1 --2- 172.21.15.89:0/1378409487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e40161590 0x7f0e40161a20 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f0e4008c1d0 tx=0x7f0e3800b330 comp rx=0 tx=0).stop 2024-04-02T07:56:21.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.327+0000 7f0e47800700 1 -- 172.21.15.89:0/1378409487 shutdown_connections 2024-04-02T07:56:21.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.327+0000 7f0e47800700 1 --2- 172.21.15.89:0/1378409487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e40161590 0x7f0e40161a20 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.327+0000 7f0e47800700 1 --2- 172.21.15.89:0/1378409487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e40160f10 0x7f0e400d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.327+0000 7f0e47800700 1 --2- 172.21.15.89:0/1378409487 >> v2:172.21.15.113:3300/0 conn(0x7f0e400cf230 0x7f0e400d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.327+0000 7f0e47800700 1 -- 172.21.15.89:0/1378409487 >> 172.21.15.89:0/1378409487 conn(0x7f0e4008aff0 msgr2=0x7f0e400d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:21.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.327+0000 7f0e47800700 1 -- 172.21.15.89:0/1378409487 shutdown_connections 2024-04-02T07:56:21.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.327+0000 7f0e47800700 1 -- 172.21.15.89:0/1378409487 wait complete. 2024-04-02T07:56:21.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.329+0000 7f0e47800700 1 Processor -- start 2024-04-02T07:56:21.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.329+0000 7f0e47800700 1 -- start start 2024-04-02T07:56:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.329+0000 7f0e47800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e400cf230 0x7f0e40158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e47800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e40160f10 0x7f0e400b6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e47800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e40161590 0x7f0e400b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e47800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4015e560 con 0x7f0e40160f10 2024-04-02T07:56:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e47800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e401e84b0 con 0x7f0e40161590 2024-04-02T07:56:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e47800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e401e8790 con 0x7f0e400cf230 2024-04-02T07:56:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e3fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e40160f10 0x7f0e400b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e3fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e40160f10 0x7f0e400b6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42326/0 (socket says 172.21.15.89:42326) 2024-04-02T07:56:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1639171811 learned_addr learned my addr 172.21.15.89:0/1639171811 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e44e00700 1 --2- 172.21.15.89:0/1639171811 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e400cf230 0x7f0e40158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e45800700 1 --2- 172.21.15.89:0/1639171811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e40161590 0x7f0e400b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1639171811 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e400cf230 msgr2=0x7f0e40158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e3fe00700 1 --2- 172.21.15.89:0/1639171811 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e400cf230 0x7f0e40158c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1639171811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e40161590 msgr2=0x7f0e400b66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e3fe00700 1 --2- 172.21.15.89:0/1639171811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e40161590 0x7f0e400b66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e3fe00700 1 -- 172.21.15.89:0/1639171811 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e38009b30 con 0x7f0e40160f10 2024-04-02T07:56:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.330+0000 7f0e3fe00700 1 --2- 172.21.15.89:0/1639171811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e40160f10 0x7f0e400b6180 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00e870 tx=0x7f0e2c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.331+0000 7f0e3cc00700 1 -- 172.21.15.89:0/1639171811 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c009590 con 0x7f0e40160f10 2024-04-02T07:56:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.331+0000 7f0e3cc00700 1 -- 172.21.15.89:0/1639171811 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2c00ee70 con 0x7f0e40160f10 2024-04-02T07:56:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.331+0000 7f0e3cc00700 1 -- 172.21.15.89:0/1639171811 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c00ca50 con 0x7f0e40160f10 2024-04-02T07:56:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.331+0000 7f0e47800700 1 -- 172.21.15.89:0/1639171811 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4015e1f0 con 0x7f0e40160f10 2024-04-02T07:56:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.331+0000 7f0e47800700 1 -- 172.21.15.89:0/1639171811 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e401e77c0 con 0x7f0e40160f10 2024-04-02T07:56:21.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.332+0000 7f0e47800700 1 -- 172.21.15.89:0/1639171811 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e401e8960 con 0x7f0e40160f10 2024-04-02T07:56:21.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.333+0000 7f0e3cc00700 1 -- 172.21.15.89:0/1639171811 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0e2c00cbb0 con 0x7f0e40160f10 2024-04-02T07:56:21.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.334+0000 7f0e3cc00700 1 --2- 172.21.15.89:0/1639171811 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e2006adb0 0x7f0e2006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:21.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.334+0000 7f0e3cc00700 1 -- 172.21.15.89:0/1639171811 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f0e2c089670 con 0x7f0e40160f10 2024-04-02T07:56:21.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.334+0000 7f0e44e00700 1 --2- 172.21.15.89:0/1639171811 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e2006adb0 0x7f0e2006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:21.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.335+0000 7f0e44e00700 1 --2- 172.21.15.89:0/1639171811 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e2006adb0 0x7f0e2006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009a50 tx=0x7f0e34008040 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:21.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.338+0000 7f0e3cc00700 1 -- 172.21.15.89:0/1639171811 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0e2c054520 con 0x7f0e40160f10 2024-04-02T07:56:21.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.532+0000 7f0e47800700 1 -- 172.21.15.89:0/1639171811 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0e400cea30 con 0x7f0e2006adb0 2024-04-02T07:56:21.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.543+0000 7f0e3cc00700 1 -- 172.21.15.89:0/1639171811 <== mgr.8790 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289736 (secure 0 0 0) 0x7f0e340096a0 con 0x7f0e2006adb0 2024-04-02T07:56:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:21.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.547+0000 7f0e47800700 1 -- 172.21.15.89:0/1639171811 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e2006adb0 msgr2=0x7f0e2006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:21.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.547+0000 7f0e47800700 1 --2- 172.21.15.89:0/1639171811 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e2006adb0 0x7f0e2006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009a50 tx=0x7f0e34008040 comp rx=0 tx=0).stop 2024-04-02T07:56:21.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.547+0000 7f0e47800700 1 -- 172.21.15.89:0/1639171811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e40160f10 msgr2=0x7f0e400b6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:21.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.547+0000 7f0e47800700 1 --2- 172.21.15.89:0/1639171811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e40160f10 0x7f0e400b6180 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00e870 tx=0x7f0e2c00ec30 comp rx=0 tx=0).stop 2024-04-02T07:56:21.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.547+0000 7f0e47800700 1 -- 172.21.15.89:0/1639171811 shutdown_connections 2024-04-02T07:56:21.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.547+0000 7f0e47800700 1 --2- 172.21.15.89:0/1639171811 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0e2006adb0 0x7f0e2006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.547+0000 7f0e47800700 1 --2- 172.21.15.89:0/1639171811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e40161590 0x7f0e400b66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.547+0000 7f0e47800700 1 --2- 172.21.15.89:0/1639171811 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e400cf230 0x7f0e40158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.547+0000 7f0e47800700 1 --2- 172.21.15.89:0/1639171811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e40160f10 0x7f0e400b6180 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.547+0000 7f0e47800700 1 -- 172.21.15.89:0/1639171811 >> 172.21.15.89:0/1639171811 conn(0x7f0e4008aff0 msgr2=0x7f0e400caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:21.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.548+0000 7f0e47800700 1 -- 172.21.15.89:0/1639171811 shutdown_connections 2024-04-02T07:56:21.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.548+0000 7f0e47800700 1 -- 172.21.15.89:0/1639171811 wait complete. 2024-04-02T07:56:21.552 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:56:21.567 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T07:56:21.189838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":356124,"kb_used_data":16132,"kb_used_omap":12,"kb_used_meta":339827,"kb_avail":754618596,"statfs":{"total":773094113280,"available":772729442304,"internally_reserved":0,"allocated":16519168,"data_stored":8773826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":347983416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.499619"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201272+0000","last_change":"2024-04-02T07:56:17.228217+0000","last_active":"2024-04-02T07:56:18.201272+0000","last_peered":"2024-04-02T07:56:18.201272+0000","last_clean":"2024-04-02T07:56:18.201272+0000","last_became_active":"2024-04-02T07:56:17.227934+0000","last_became_peered":"2024-04-02T07:56:17.227934+0000","last_unstale":"2024-04-02T07:56:18.201272+0000","last_undegraded":"2024-04-02T07:56:18.201272+0000","last_fullsized":"2024-04-02T07:56:18.201272+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:17:27.958914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201588+0000","last_change":"2024-04-02T07:56:17.220920+0000","last_active":"2024-04-02T07:56:18.201588+0000","last_peered":"2024-04-02T07:56:18.201588+0000","last_clean":"2024-04-02T07:56:18.201588+0000","last_became_active":"2024-04-02T07:56:17.220658+0000","last_became_peered":"2024-04-02T07:56:17.220658+0000","last_unstale":"2024-04-02T07:56:18.201588+0000","last_undegraded":"2024-04-02T07:56:18.201588+0000","last_fullsized":"2024-04-02T07:56:18.201588+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:34.929140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.212627+0000","last_change":"2024-04-02T07:56:17.208856+0000","last_active":"2024-04-02T07:56:17.212627+0000","last_peered":"2024-04-02T07:56:17.212627+0000","last_clean":"2024-04-02T07:56:17.212627+0000","last_became_active":"2024-04-02T07:56:17.208458+0000","last_became_peered":"2024-04-02T07:56:17.208458+0000","last_unstale":"2024-04-02T07:56:17.212627+0000","last_undegraded":"2024-04-02T07:56:17.212627+0000","last_fullsized":"2024-04-02T07:56:17.212627+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:15:20.199042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.221804+0000","last_change":"2024-04-02T07:56:17.219167+0000","last_active":"2024-04-02T07:56:17.221804+0000","last_peered":"2024-04-02T07:56:17.221804+0000","last_clean":"2024-04-02T07:56:17.221804+0000","last_became_active":"2024-04-02T07:56:17.218904+0000","last_became_peered":"2024-04-02T07:56:17.218904+0000","last_unstale":"2024-04-02T07:56:17.221804+0000","last_undegraded":"2024-04-02T07:56:17.221804+0000","last_fullsized":"2024-04-02T07:56:17.221804+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:31.191815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.199830+0000","last_change":"2024-04-02T07:56:17.217244+0000","last_active":"2024-04-02T07:56:18.199830+0000","last_peered":"2024-04-02T07:56:18.199830+0000","last_clean":"2024-04-02T07:56:18.199830+0000","last_became_active":"2024-04-02T07:56:17.216765+0000","last_became_peered":"2024-04-02T07:56:17.216765+0000","last_unstale":"2024-04-02T07:56:18.199830+0000","last_undegraded":"2024-04-02T07:56:18.199830+0000","last_fullsized":"2024-04-02T07:56:18.199830+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:58:19.065313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751139+0000","last_change":"2024-04-02T07:56:17.226865+0000","last_active":"2024-04-02T07:56:18.751139+0000","last_peered":"2024-04-02T07:56:18.751139+0000","last_clean":"2024-04-02T07:56:18.751139+0000","last_became_active":"2024-04-02T07:56:17.226581+0000","last_became_peered":"2024-04-02T07:56:17.226581+0000","last_unstale":"2024-04-02T07:56:18.751139+0000","last_undegraded":"2024-04-02T07:56:18.751139+0000","last_fullsized":"2024-04-02T07:56:18.751139+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:56:04.841946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.199574+0000","last_change":"2024-04-02T07:56:17.212002+0000","last_active":"2024-04-02T07:56:18.199574+0000","last_peered":"2024-04-02T07:56:18.199574+0000","last_clean":"2024-04-02T07:56:18.199574+0000","last_became_active":"2024-04-02T07:56:17.211724+0000","last_became_peered":"2024-04-02T07:56:17.211724+0000","last_unstale":"2024-04-02T07:56:18.199574+0000","last_undegraded":"2024-04-02T07:56:18.199574+0000","last_fullsized":"2024-04-02T07:56:18.199574+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:44:51.037155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.199430+0000","last_change":"2024-04-02T07:56:17.215116+0000","last_active":"2024-04-02T07:56:18.199430+0000","last_peered":"2024-04-02T07:56:18.199430+0000","last_clean":"2024-04-02T07:56:18.199430+0000","last_became_active":"2024-04-02T07:56:17.213978+0000","last_became_peered":"2024-04-02T07:56:17.213978+0000","last_unstale":"2024-04-02T07:56:18.199430+0000","last_undegraded":"2024-04-02T07:56:18.199430+0000","last_fullsized":"2024-04-02T07:56:18.199430+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:52.462023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.208904+0000","last_change":"2024-04-02T07:56:17.203201+0000","last_active":"2024-04-02T07:56:17.208904+0000","last_peered":"2024-04-02T07:56:17.208904+0000","last_clean":"2024-04-02T07:56:17.208904+0000","last_became_active":"2024-04-02T07:56:17.202926+0000","last_became_peered":"2024-04-02T07:56:17.202926+0000","last_unstale":"2024-04-02T07:56:17.208904+0000","last_undegraded":"2024-04-02T07:56:17.208904+0000","last_fullsized":"2024-04-02T07:56:17.208904+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:36:17.721861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201702+0000","last_change":"2024-04-02T07:56:17.221434+0000","last_active":"2024-04-02T07:56:18.201702+0000","last_peered":"2024-04-02T07:56:18.201702+0000","last_clean":"2024-04-02T07:56:18.201702+0000","last_became_active":"2024-04-02T07:56:17.221181+0000","last_became_peered":"2024-04-02T07:56:17.221181+0000","last_unstale":"2024-04-02T07:56:18.201702+0000","last_undegraded":"2024-04-02T07:56:18.201702+0000","last_fullsized":"2024-04-02T07:56:18.201702+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:25:01.575042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.214726+0000","last_change":"2024-04-02T07:56:17.213301+0000","last_active":"2024-04-02T07:56:17.214726+0000","last_peered":"2024-04-02T07:56:17.214726+0000","last_clean":"2024-04-02T07:56:17.214726+0000","last_became_active":"2024-04-02T07:56:17.212433+0000","last_became_peered":"2024-04-02T07:56:17.212433+0000","last_unstale":"2024-04-02T07:56:17.214726+0000","last_undegraded":"2024-04-02T07:56:17.214726+0000","last_fullsized":"2024-04-02T07:56:17.214726+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:07:38.125893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751344+0000","last_change":"2024-04-02T07:56:17.224697+0000","last_active":"2024-04-02T07:56:18.751344+0000","last_peered":"2024-04-02T07:56:18.751344+0000","last_clean":"2024-04-02T07:56:18.751344+0000","last_became_active":"2024-04-02T07:56:17.223662+0000","last_became_peered":"2024-04-02T07:56:17.223662+0000","last_unstale":"2024-04-02T07:56:18.751344+0000","last_undegraded":"2024-04-02T07:56:18.751344+0000","last_fullsized":"2024-04-02T07:56:18.751344+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:16:19.108293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.225851+0000","last_change":"2024-04-02T07:56:17.223145+0000","last_active":"2024-04-02T07:56:17.225851+0000","last_peered":"2024-04-02T07:56:17.225851+0000","last_clean":"2024-04-02T07:56:17.225851+0000","last_became_active":"2024-04-02T07:56:17.222732+0000","last_became_peered":"2024-04-02T07:56:17.222732+0000","last_unstale":"2024-04-02T07:56:17.225851+0000","last_undegraded":"2024-04-02T07:56:17.225851+0000","last_fullsized":"2024-04-02T07:56:17.225851+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:01:41.298193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801079+0000","last_change":"2024-04-02T07:56:17.216980+0000","last_active":"2024-04-02T07:56:18.801079+0000","last_peered":"2024-04-02T07:56:18.801079+0000","last_clean":"2024-04-02T07:56:18.801079+0000","last_became_active":"2024-04-02T07:56:17.216689+0000","last_became_peered":"2024-04-02T07:56:17.216689+0000","last_unstale":"2024-04-02T07:56:18.801079+0000","last_undegraded":"2024-04-02T07:56:18.801079+0000","last_fullsized":"2024-04-02T07:56:18.801079+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:59:49.826346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201354+0000","last_change":"2024-04-02T07:56:17.216532+0000","last_active":"2024-04-02T07:56:18.201354+0000","last_peered":"2024-04-02T07:56:18.201354+0000","last_clean":"2024-04-02T07:56:18.201354+0000","last_became_active":"2024-04-02T07:56:17.216264+0000","last_became_peered":"2024-04-02T07:56:17.216264+0000","last_unstale":"2024-04-02T07:56:18.201354+0000","last_undegraded":"2024-04-02T07:56:18.201354+0000","last_fullsized":"2024-04-02T07:56:18.201354+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:13:40.138497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801012+0000","last_change":"2024-04-02T07:56:17.216064+0000","last_active":"2024-04-02T07:56:18.801012+0000","last_peered":"2024-04-02T07:56:18.801012+0000","last_clean":"2024-04-02T07:56:18.801012+0000","last_became_active":"2024-04-02T07:56:17.215729+0000","last_became_peered":"2024-04-02T07:56:17.215729+0000","last_unstale":"2024-04-02T07:56:18.801012+0000","last_undegraded":"2024-04-02T07:56:18.801012+0000","last_fullsized":"2024-04-02T07:56:18.801012+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:26:51.859488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.214434+0000","last_change":"2024-04-02T07:56:17.209022+0000","last_active":"2024-04-02T07:56:17.214434+0000","last_peered":"2024-04-02T07:56:17.214434+0000","last_clean":"2024-04-02T07:56:17.214434+0000","last_became_active":"2024-04-02T07:56:17.208739+0000","last_became_peered":"2024-04-02T07:56:17.208739+0000","last_unstale":"2024-04-02T07:56:17.214434+0000","last_undegraded":"2024-04-02T07:56:17.214434+0000","last_fullsized":"2024-04-02T07:56:17.214434+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:37:36.640185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.202163+0000","last_change":"2024-04-02T07:56:17.210431+0000","last_active":"2024-04-02T07:56:18.202163+0000","last_peered":"2024-04-02T07:56:18.202163+0000","last_clean":"2024-04-02T07:56:18.202163+0000","last_became_active":"2024-04-02T07:56:17.209098+0000","last_became_peered":"2024-04-02T07:56:17.209098+0000","last_unstale":"2024-04-02T07:56:18.202163+0000","last_undegraded":"2024-04-02T07:56:18.202163+0000","last_fullsized":"2024-04-02T07:56:18.202163+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:49:12.293116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201104+0000","last_change":"2024-04-02T07:56:17.226280+0000","last_active":"2024-04-02T07:56:18.201104+0000","last_peered":"2024-04-02T07:56:18.201104+0000","last_clean":"2024-04-02T07:56:18.201104+0000","last_became_active":"2024-04-02T07:56:17.225114+0000","last_became_peered":"2024-04-02T07:56:17.225114+0000","last_unstale":"2024-04-02T07:56:18.201104+0000","last_undegraded":"2024-04-02T07:56:18.201104+0000","last_fullsized":"2024-04-02T07:56:18.201104+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:20.856155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.202062+0000","last_change":"2024-04-02T07:56:17.214951+0000","last_active":"2024-04-02T07:56:18.202062+0000","last_peered":"2024-04-02T07:56:18.202062+0000","last_clean":"2024-04-02T07:56:18.202062+0000","last_became_active":"2024-04-02T07:56:17.213069+0000","last_became_peered":"2024-04-02T07:56:17.213069+0000","last_unstale":"2024-04-02T07:56:18.202062+0000","last_undegraded":"2024-04-02T07:56:18.202062+0000","last_fullsized":"2024-04-02T07:56:18.202062+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:28:09.719123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751165+0000","last_change":"2024-04-02T07:56:17.219528+0000","last_active":"2024-04-02T07:56:18.751165+0000","last_peered":"2024-04-02T07:56:18.751165+0000","last_clean":"2024-04-02T07:56:18.751165+0000","last_became_active":"2024-04-02T07:56:17.216203+0000","last_became_peered":"2024-04-02T07:56:17.216203+0000","last_unstale":"2024-04-02T07:56:18.751165+0000","last_undegraded":"2024-04-02T07:56:18.751165+0000","last_fullsized":"2024-04-02T07:56:18.751165+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:57:35.607653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201755+0000","last_change":"2024-04-02T07:56:17.216412+0000","last_active":"2024-04-02T07:56:18.201755+0000","last_peered":"2024-04-02T07:56:18.201755+0000","last_clean":"2024-04-02T07:56:18.201755+0000","last_became_active":"2024-04-02T07:56:17.216122+0000","last_became_peered":"2024-04-02T07:56:17.216122+0000","last_unstale":"2024-04-02T07:56:18.201755+0000","last_undegraded":"2024-04-02T07:56:18.201755+0000","last_fullsized":"2024-04-02T07:56:18.201755+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:59:00.949515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.221052+0000","last_change":"2024-04-02T07:56:17.219286+0000","last_active":"2024-04-02T07:56:17.221052+0000","last_peered":"2024-04-02T07:56:17.221052+0000","last_clean":"2024-04-02T07:56:17.221052+0000","last_became_active":"2024-04-02T07:56:17.219016+0000","last_became_peered":"2024-04-02T07:56:17.219016+0000","last_unstale":"2024-04-02T07:56:17.221052+0000","last_undegraded":"2024-04-02T07:56:17.221052+0000","last_fullsized":"2024-04-02T07:56:17.221052+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:54:45.468728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.200717+0000","last_change":"2024-04-02T07:56:17.216205+0000","last_active":"2024-04-02T07:56:18.200717+0000","last_peered":"2024-04-02T07:56:18.200717+0000","last_clean":"2024-04-02T07:56:18.200717+0000","last_became_active":"2024-04-02T07:56:17.215546+0000","last_became_peered":"2024-04-02T07:56:17.215546+0000","last_unstale":"2024-04-02T07:56:18.200717+0000","last_undegraded":"2024-04-02T07:56:18.200717+0000","last_fullsized":"2024-04-02T07:56:18.200717+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:26:06.353668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.216622+0000","last_change":"2024-04-02T07:56:17.215236+0000","last_active":"2024-04-02T07:56:17.216622+0000","last_peered":"2024-04-02T07:56:17.216622+0000","last_clean":"2024-04-02T07:56:17.216622+0000","last_became_active":"2024-04-02T07:56:17.214457+0000","last_became_peered":"2024-04-02T07:56:17.214457+0000","last_unstale":"2024-04-02T07:56:17.216622+0000","last_undegraded":"2024-04-02T07:56:17.216622+0000","last_fullsized":"2024-04-02T07:56:17.216622+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:14:42.522106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201755+0000","last_change":"2024-04-02T07:56:17.220389+0000","last_active":"2024-04-02T07:56:18.201755+0000","last_peered":"2024-04-02T07:56:18.201755+0000","last_clean":"2024-04-02T07:56:18.201755+0000","last_became_active":"2024-04-02T07:56:17.220025+0000","last_became_peered":"2024-04-02T07:56:17.220025+0000","last_unstale":"2024-04-02T07:56:18.201755+0000","last_undegraded":"2024-04-02T07:56:18.201755+0000","last_fullsized":"2024-04-02T07:56:18.201755+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:31.191815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801432+0000","last_change":"2024-04-02T07:56:17.221197+0000","last_active":"2024-04-02T07:56:18.801432+0000","last_peered":"2024-04-02T07:56:18.801432+0000","last_clean":"2024-04-02T07:56:18.801432+0000","last_became_active":"2024-04-02T07:56:17.220796+0000","last_became_peered":"2024-04-02T07:56:17.220796+0000","last_unstale":"2024-04-02T07:56:18.801432+0000","last_undegraded":"2024-04-02T07:56:18.801432+0000","last_fullsized":"2024-04-02T07:56:18.801432+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:57:07.303322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.223346+0000","last_change":"2024-04-02T07:56:17.222232+0000","last_active":"2024-04-02T07:56:17.223346+0000","last_peered":"2024-04-02T07:56:17.223346+0000","last_clean":"2024-04-02T07:56:17.223346+0000","last_became_active":"2024-04-02T07:56:17.220361+0000","last_became_peered":"2024-04-02T07:56:17.220361+0000","last_unstale":"2024-04-02T07:56:17.223346+0000","last_undegraded":"2024-04-02T07:56:17.223346+0000","last_fullsized":"2024-04-02T07:56:17.223346+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:44:57.200725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201763+0000","last_change":"2024-04-02T07:56:17.212364+0000","last_active":"2024-04-02T07:56:18.201763+0000","last_peered":"2024-04-02T07:56:18.201763+0000","last_clean":"2024-04-02T07:56:18.201763+0000","last_became_active":"2024-04-02T07:56:17.212072+0000","last_became_peered":"2024-04-02T07:56:17.212072+0000","last_unstale":"2024-04-02T07:56:18.201763+0000","last_undegraded":"2024-04-02T07:56:18.201763+0000","last_fullsized":"2024-04-02T07:56:18.201763+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:56:11.399042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201555+0000","last_change":"2024-04-02T07:56:17.213872+0000","last_active":"2024-04-02T07:56:18.201555+0000","last_peered":"2024-04-02T07:56:18.201555+0000","last_clean":"2024-04-02T07:56:18.201555+0000","last_became_active":"2024-04-02T07:56:17.213595+0000","last_became_peered":"2024-04-02T07:56:17.213595+0000","last_unstale":"2024-04-02T07:56:18.201555+0000","last_undegraded":"2024-04-02T07:56:18.201555+0000","last_fullsized":"2024-04-02T07:56:18.201555+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:46:52.515384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.217474+0000","last_change":"2024-04-02T07:56:17.212353+0000","last_active":"2024-04-02T07:56:17.217474+0000","last_peered":"2024-04-02T07:56:17.217474+0000","last_clean":"2024-04-02T07:56:17.217474+0000","last_became_active":"2024-04-02T07:56:17.212003+0000","last_became_peered":"2024-04-02T07:56:17.212003+0000","last_unstale":"2024-04-02T07:56:17.217474+0000","last_undegraded":"2024-04-02T07:56:17.217474+0000","last_fullsized":"2024-04-02T07:56:17.217474+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:44:51.037155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.221589+0000","last_change":"2024-04-02T07:56:17.215278+0000","last_active":"2024-04-02T07:56:17.221589+0000","last_peered":"2024-04-02T07:56:17.221589+0000","last_clean":"2024-04-02T07:56:17.221589+0000","last_became_active":"2024-04-02T07:56:17.215021+0000","last_became_peered":"2024-04-02T07:56:17.215021+0000","last_unstale":"2024-04-02T07:56:17.221589+0000","last_undegraded":"2024-04-02T07:56:17.221589+0000","last_fullsized":"2024-04-02T07:56:17.221589+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:43:50.657790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801439+0000","last_change":"2024-04-02T07:56:17.223954+0000","last_active":"2024-04-02T07:56:18.801439+0000","last_peered":"2024-04-02T07:56:18.801439+0000","last_clean":"2024-04-02T07:56:18.801439+0000","last_became_active":"2024-04-02T07:56:17.223497+0000","last_became_peered":"2024-04-02T07:56:17.223497+0000","last_unstale":"2024-04-02T07:56:18.801439+0000","last_undegraded":"2024-04-02T07:56:18.801439+0000","last_fullsized":"2024-04-02T07:56:18.801439+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:36.562378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1f","version":"153'4","reported_seq":15,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.200228+0000","last_change":"2024-04-02T07:56:16.707369+0000","last_active":"2024-04-02T07:56:18.200228+0000","last_peered":"2024-04-02T07:56:18.200228+0000","last_clean":"2024-04-02T07:56:18.200228+0000","last_became_active":"2024-04-02T07:56:16.705817+0000","last_became_peered":"2024-04-02T07:56:16.705817+0000","last_unstale":"2024-04-02T07:56:18.200228+0000","last_undegraded":"2024-04-02T07:56:18.200228+0000","last_fullsized":"2024-04-02T07:56:18.200228+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:56:03.390568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.214456+0000","last_change":"2024-04-02T07:56:17.209750+0000","last_active":"2024-04-02T07:56:17.214456+0000","last_peered":"2024-04-02T07:56:17.214456+0000","last_clean":"2024-04-02T07:56:17.214456+0000","last_became_active":"2024-04-02T07:56:17.209235+0000","last_became_peered":"2024-04-02T07:56:17.209235+0000","last_unstale":"2024-04-02T07:56:17.214456+0000","last_undegraded":"2024-04-02T07:56:17.214456+0000","last_fullsized":"2024-04-02T07:56:17.214456+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:15:21.320878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1e","version":"153'1","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.232854+0000","last_change":"2024-04-02T07:56:16.714403+0000","last_active":"2024-04-02T07:56:17.232854+0000","last_peered":"2024-04-02T07:56:17.232854+0000","last_clean":"2024-04-02T07:56:17.232854+0000","last_became_active":"2024-04-02T07:56:16.714141+0000","last_became_peered":"2024-04-02T07:56:16.714141+0000","last_unstale":"2024-04-02T07:56:17.232854+0000","last_undegraded":"2024-04-02T07:56:17.232854+0000","last_fullsized":"2024-04-02T07:56:17.232854+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:31.606349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.221245+0000","last_change":"2024-04-02T07:56:17.215667+0000","last_active":"2024-04-02T07:56:17.221245+0000","last_peered":"2024-04-02T07:56:17.221245+0000","last_clean":"2024-04-02T07:56:17.221245+0000","last_became_active":"2024-04-02T07:56:17.215422+0000","last_became_peered":"2024-04-02T07:56:17.215422+0000","last_unstale":"2024-04-02T07:56:17.221245+0000","last_undegraded":"2024-04-02T07:56:17.221245+0000","last_fullsized":"2024-04-02T07:56:17.221245+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:10:08.386400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1d","version":"153'2","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.228765+0000","last_change":"2024-04-02T07:56:16.718433+0000","last_active":"2024-04-02T07:56:17.228765+0000","last_peered":"2024-04-02T07:56:17.228765+0000","last_clean":"2024-04-02T07:56:17.228765+0000","last_became_active":"2024-04-02T07:56:16.718060+0000","last_became_peered":"2024-04-02T07:56:16.718060+0000","last_unstale":"2024-04-02T07:56:17.228765+0000","last_undegraded":"2024-04-02T07:56:17.228765+0000","last_fullsized":"2024-04-02T07:56:17.228765+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:55.044510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751510+0000","last_change":"2024-04-02T07:56:17.213206+0000","last_active":"2024-04-02T07:56:18.751510+0000","last_peered":"2024-04-02T07:56:18.751510+0000","last_clean":"2024-04-02T07:56:18.751510+0000","last_became_active":"2024-04-02T07:56:17.212941+0000","last_became_peered":"2024-04-02T07:56:17.212941+0000","last_unstale":"2024-04-02T07:56:18.751510+0000","last_undegraded":"2024-04-02T07:56:18.751510+0000","last_fullsized":"2024-04-02T07:56:18.751510+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:16:04.779564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1c","version":"153'2","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201600+0000","last_change":"2024-04-02T07:56:16.715903+0000","last_active":"2024-04-02T07:56:18.201600+0000","last_peered":"2024-04-02T07:56:18.201600+0000","last_clean":"2024-04-02T07:56:18.201600+0000","last_became_active":"2024-04-02T07:56:16.715622+0000","last_became_peered":"2024-04-02T07:56:16.715622+0000","last_unstale":"2024-04-02T07:56:18.201600+0000","last_undegraded":"2024-04-02T07:56:18.201600+0000","last_fullsized":"2024-04-02T07:56:18.201600+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:43:01.550275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201363+0000","last_change":"2024-04-02T07:56:17.213727+0000","last_active":"2024-04-02T07:56:18.201363+0000","last_peered":"2024-04-02T07:56:18.201363+0000","last_clean":"2024-04-02T07:56:18.201363+0000","last_became_active":"2024-04-02T07:56:17.213446+0000","last_became_peered":"2024-04-02T07:56:17.213446+0000","last_unstale":"2024-04-02T07:56:18.201363+0000","last_undegraded":"2024-04-02T07:56:18.201363+0000","last_fullsized":"2024-04-02T07:56:18.201363+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:06:06.229996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1b","version":"153'1","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201715+0000","last_change":"2024-04-02T07:56:16.712283+0000","last_active":"2024-04-02T07:56:18.201715+0000","last_peered":"2024-04-02T07:56:18.201715+0000","last_clean":"2024-04-02T07:56:18.201715+0000","last_became_active":"2024-04-02T07:56:16.711971+0000","last_became_peered":"2024-04-02T07:56:16.711971+0000","last_unstale":"2024-04-02T07:56:18.201715+0000","last_undegraded":"2024-04-02T07:56:18.201715+0000","last_fullsized":"2024-04-02T07:56:18.201715+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:19:49.151260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751282+0000","last_change":"2024-04-02T07:56:17.219746+0000","last_active":"2024-04-02T07:56:18.751282+0000","last_peered":"2024-04-02T07:56:18.751282+0000","last_clean":"2024-04-02T07:56:18.751282+0000","last_became_active":"2024-04-02T07:56:17.218551+0000","last_became_peered":"2024-04-02T07:56:17.218551+0000","last_unstale":"2024-04-02T07:56:18.751282+0000","last_undegraded":"2024-04-02T07:56:18.751282+0000","last_fullsized":"2024-04-02T07:56:18.751282+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:54:11.802163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"153'1","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201923+0000","last_change":"2024-04-02T07:56:16.721064+0000","last_active":"2024-04-02T07:56:18.201923+0000","last_peered":"2024-04-02T07:56:18.201923+0000","last_clean":"2024-04-02T07:56:18.201923+0000","last_became_active":"2024-04-02T07:56:16.720572+0000","last_became_peered":"2024-04-02T07:56:16.720572+0000","last_unstale":"2024-04-02T07:56:18.201923+0000","last_undegraded":"2024-04-02T07:56:18.201923+0000","last_fullsized":"2024-04-02T07:56:18.201923+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:31:36.461451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751088+0000","last_change":"2024-04-02T07:56:17.222686+0000","last_active":"2024-04-02T07:56:18.751088+0000","last_peered":"2024-04-02T07:56:18.751088+0000","last_clean":"2024-04-02T07:56:18.751088+0000","last_became_active":"2024-04-02T07:56:17.221832+0000","last_became_peered":"2024-04-02T07:56:17.221832+0000","last_unstale":"2024-04-02T07:56:18.751088+0000","last_undegraded":"2024-04-02T07:56:18.751088+0000","last_fullsized":"2024-04-02T07:56:18.751088+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:46:52.515384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.197127+0000","last_change":"2024-04-02T07:56:16.718802+0000","last_active":"2024-04-02T07:56:17.197127+0000","last_peered":"2024-04-02T07:56:17.197127+0000","last_clean":"2024-04-02T07:56:17.197127+0000","last_became_active":"2024-04-02T07:56:16.718497+0000","last_became_peered":"2024-04-02T07:56:16.718497+0000","last_unstale":"2024-04-02T07:56:17.197127+0000","last_undegraded":"2024-04-02T07:56:17.197127+0000","last_fullsized":"2024-04-02T07:56:17.197127+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:42:33.236077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.200928+0000","last_change":"2024-04-02T07:56:17.224386+0000","last_active":"2024-04-02T07:56:18.200928+0000","last_peered":"2024-04-02T07:56:18.200928+0000","last_clean":"2024-04-02T07:56:18.200928+0000","last_became_active":"2024-04-02T07:56:17.224093+0000","last_became_peered":"2024-04-02T07:56:17.224093+0000","last_unstale":"2024-04-02T07:56:18.200928+0000","last_undegraded":"2024-04-02T07:56:18.200928+0000","last_fullsized":"2024-04-02T07:56:18.200928+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:57:07.303322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.200963+0000","last_change":"2024-04-02T07:56:16.716294+0000","last_active":"2024-04-02T07:56:18.200963+0000","last_peered":"2024-04-02T07:56:18.200963+0000","last_clean":"2024-04-02T07:56:18.200963+0000","last_became_active":"2024-04-02T07:56:16.714501+0000","last_became_peered":"2024-04-02T07:56:16.714501+0000","last_unstale":"2024-04-02T07:56:18.200963+0000","last_undegraded":"2024-04-02T07:56:18.200963+0000","last_fullsized":"2024-04-02T07:56:18.200963+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:25:27.947017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201807+0000","last_change":"2024-04-02T07:56:17.227002+0000","last_active":"2024-04-02T07:56:18.201807+0000","last_peered":"2024-04-02T07:56:18.201807+0000","last_clean":"2024-04-02T07:56:18.201807+0000","last_became_active":"2024-04-02T07:56:17.226716+0000","last_became_peered":"2024-04-02T07:56:17.226716+0000","last_unstale":"2024-04-02T07:56:18.201807+0000","last_undegraded":"2024-04-02T07:56:18.201807+0000","last_fullsized":"2024-04-02T07:56:18.201807+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:07:25.934705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751549+0000","last_change":"2024-04-02T07:56:16.714383+0000","last_active":"2024-04-02T07:56:18.751549+0000","last_peered":"2024-04-02T07:56:18.751549+0000","last_clean":"2024-04-02T07:56:18.751549+0000","last_became_active":"2024-04-02T07:56:16.713929+0000","last_became_peered":"2024-04-02T07:56:16.713929+0000","last_unstale":"2024-04-02T07:56:18.751549+0000","last_undegraded":"2024-04-02T07:56:18.751549+0000","last_fullsized":"2024-04-02T07:56:18.751549+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:55.044510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.200005+0000","last_change":"2024-04-02T07:56:17.212537+0000","last_active":"2024-04-02T07:56:18.200005+0000","last_peered":"2024-04-02T07:56:18.200005+0000","last_clean":"2024-04-02T07:56:18.200005+0000","last_became_active":"2024-04-02T07:56:17.212291+0000","last_became_peered":"2024-04-02T07:56:17.212291+0000","last_unstale":"2024-04-02T07:56:18.200005+0000","last_undegraded":"2024-04-02T07:56:18.200005+0000","last_fullsized":"2024-04-02T07:56:18.200005+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:44:57.200725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.202310+0000","last_change":"2024-04-02T07:56:16.719851+0000","last_active":"2024-04-02T07:56:18.202310+0000","last_peered":"2024-04-02T07:56:18.202310+0000","last_clean":"2024-04-02T07:56:18.202310+0000","last_became_active":"2024-04-02T07:56:16.718004+0000","last_became_peered":"2024-04-02T07:56:16.718004+0000","last_unstale":"2024-04-02T07:56:18.202310+0000","last_undegraded":"2024-04-02T07:56:18.202310+0000","last_fullsized":"2024-04-02T07:56:18.202310+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:27.254350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751383+0000","last_change":"2024-04-02T07:56:17.220141+0000","last_active":"2024-04-02T07:56:18.751383+0000","last_peered":"2024-04-02T07:56:18.751383+0000","last_clean":"2024-04-02T07:56:18.751383+0000","last_became_active":"2024-04-02T07:56:17.219894+0000","last_became_peered":"2024-04-02T07:56:17.219894+0000","last_unstale":"2024-04-02T07:56:18.751383+0000","last_undegraded":"2024-04-02T07:56:18.751383+0000","last_fullsized":"2024-04-02T07:56:18.751383+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:52:31.704949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.15","version":"153'3","reported_seq":14,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.199770+0000","last_change":"2024-04-02T07:56:16.705712+0000","last_active":"2024-04-02T07:56:18.199770+0000","last_peered":"2024-04-02T07:56:18.199770+0000","last_clean":"2024-04-02T07:56:18.199770+0000","last_became_active":"2024-04-02T07:56:16.705288+0000","last_became_peered":"2024-04-02T07:56:16.705288+0000","last_unstale":"2024-04-02T07:56:18.199770+0000","last_undegraded":"2024-04-02T07:56:18.199770+0000","last_fullsized":"2024-04-02T07:56:18.199770+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:11:37.420301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.230517+0000","last_change":"2024-04-02T07:56:17.229449+0000","last_active":"2024-04-02T07:56:17.230517+0000","last_peered":"2024-04-02T07:56:17.230517+0000","last_clean":"2024-04-02T07:56:17.230517+0000","last_became_active":"2024-04-02T07:56:17.229159+0000","last_became_peered":"2024-04-02T07:56:17.229159+0000","last_unstale":"2024-04-02T07:56:17.230517+0000","last_undegraded":"2024-04-02T07:56:17.230517+0000","last_fullsized":"2024-04-02T07:56:17.230517+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:39:22.649036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.14","version":"153'6","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.235307+0000","last_change":"2024-04-02T07:56:16.711712+0000","last_active":"2024-04-02T07:56:17.235307+0000","last_peered":"2024-04-02T07:56:17.235307+0000","last_clean":"2024-04-02T07:56:17.235307+0000","last_became_active":"2024-04-02T07:56:16.711361+0000","last_became_peered":"2024-04-02T07:56:16.711361+0000","last_unstale":"2024-04-02T07:56:17.235307+0000","last_undegraded":"2024-04-02T07:56:17.235307+0000","last_fullsized":"2024-04-02T07:56:17.235307+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:57:40.665983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801232+0000","last_change":"2024-04-02T07:56:17.217638+0000","last_active":"2024-04-02T07:56:18.801232+0000","last_peered":"2024-04-02T07:56:18.801232+0000","last_clean":"2024-04-02T07:56:18.801232+0000","last_became_active":"2024-04-02T07:56:17.217380+0000","last_became_peered":"2024-04-02T07:56:17.217380+0000","last_unstale":"2024-04-02T07:56:18.801232+0000","last_undegraded":"2024-04-02T07:56:18.801232+0000","last_fullsized":"2024-04-02T07:56:18.801232+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:06:58.818408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801137+0000","last_change":"2024-04-02T07:56:16.718975+0000","last_active":"2024-04-02T07:56:18.801137+0000","last_peered":"2024-04-02T07:56:18.801137+0000","last_clean":"2024-04-02T07:56:18.801137+0000","last_became_active":"2024-04-02T07:56:16.718267+0000","last_became_peered":"2024-04-02T07:56:16.718267+0000","last_unstale":"2024-04-02T07:56:18.801137+0000","last_undegraded":"2024-04-02T07:56:18.801137+0000","last_fullsized":"2024-04-02T07:56:18.801137+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:06.225724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801020+0000","last_change":"2024-04-02T07:56:17.219206+0000","last_active":"2024-04-02T07:56:18.801020+0000","last_peered":"2024-04-02T07:56:18.801020+0000","last_clean":"2024-04-02T07:56:18.801020+0000","last_became_active":"2024-04-02T07:56:17.218814+0000","last_became_peered":"2024-04-02T07:56:17.218814+0000","last_unstale":"2024-04-02T07:56:18.801020+0000","last_undegraded":"2024-04-02T07:56:18.801020+0000","last_fullsized":"2024-04-02T07:56:18.801020+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:56.051002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.202146+0000","last_change":"2024-04-02T07:56:16.714035+0000","last_active":"2024-04-02T07:56:17.202146+0000","last_peered":"2024-04-02T07:56:17.202146+0000","last_clean":"2024-04-02T07:56:17.202146+0000","last_became_active":"2024-04-02T07:56:16.713683+0000","last_became_peered":"2024-04-02T07:56:16.713683+0000","last_unstale":"2024-04-02T07:56:17.202146+0000","last_undegraded":"2024-04-02T07:56:17.202146+0000","last_fullsized":"2024-04-02T07:56:17.202146+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:06:57.811916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751037+0000","last_change":"2024-04-02T07:56:17.215732+0000","last_active":"2024-04-02T07:56:18.751037+0000","last_peered":"2024-04-02T07:56:18.751037+0000","last_clean":"2024-04-02T07:56:18.751037+0000","last_became_active":"2024-04-02T07:56:17.215450+0000","last_became_peered":"2024-04-02T07:56:17.215450+0000","last_unstale":"2024-04-02T07:56:18.751037+0000","last_undegraded":"2024-04-02T07:56:18.751037+0000","last_fullsized":"2024-04-02T07:56:18.751037+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:40:31.418887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.10","version":"153'8","reported_seq":19,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751074+0000","last_change":"2024-04-02T07:56:16.714558+0000","last_active":"2024-04-02T07:56:18.751074+0000","last_peered":"2024-04-02T07:56:18.751074+0000","last_clean":"2024-04-02T07:56:18.751074+0000","last_became_active":"2024-04-02T07:56:16.714276+0000","last_became_peered":"2024-04-02T07:56:16.714276+0000","last_unstale":"2024-04-02T07:56:18.751074+0000","last_undegraded":"2024-04-02T07:56:18.751074+0000","last_fullsized":"2024-04-02T07:56:18.751074+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:58:37.517431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.199776+0000","last_change":"2024-04-02T07:56:17.218056+0000","last_active":"2024-04-02T07:56:18.199776+0000","last_peered":"2024-04-02T07:56:18.199776+0000","last_clean":"2024-04-02T07:56:18.199776+0000","last_became_active":"2024-04-02T07:56:17.214883+0000","last_became_peered":"2024-04-02T07:56:17.214883+0000","last_unstale":"2024-04-02T07:56:18.199776+0000","last_undegraded":"2024-04-02T07:56:18.199776+0000","last_fullsized":"2024-04-02T07:56:18.199776+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:54:45.468728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.f","version":"153'1","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751591+0000","last_change":"2024-04-02T07:56:16.714205+0000","last_active":"2024-04-02T07:56:18.751591+0000","last_peered":"2024-04-02T07:56:18.751591+0000","last_clean":"2024-04-02T07:56:18.751591+0000","last_became_active":"2024-04-02T07:56:16.713576+0000","last_became_peered":"2024-04-02T07:56:16.713576+0000","last_unstale":"2024-04-02T07:56:18.751591+0000","last_undegraded":"2024-04-02T07:56:18.751591+0000","last_fullsized":"2024-04-02T07:56:18.751591+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:26:50.852997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.200088+0000","last_change":"2024-04-02T07:56:17.213711+0000","last_active":"2024-04-02T07:56:18.200088+0000","last_peered":"2024-04-02T07:56:18.200088+0000","last_clean":"2024-04-02T07:56:18.200088+0000","last_became_active":"2024-04-02T07:56:17.211615+0000","last_became_peered":"2024-04-02T07:56:17.211615+0000","last_unstale":"2024-04-02T07:56:18.200088+0000","last_undegraded":"2024-04-02T07:56:18.200088+0000","last_fullsized":"2024-04-02T07:56:18.200088+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:25.681411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"153'2","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.230887+0000","last_change":"2024-04-02T07:56:16.711626+0000","last_active":"2024-04-02T07:56:17.230887+0000","last_peered":"2024-04-02T07:56:17.230887+0000","last_clean":"2024-04-02T07:56:17.230887+0000","last_became_active":"2024-04-02T07:56:16.711311+0000","last_became_peered":"2024-04-02T07:56:16.711311+0000","last_unstale":"2024-04-02T07:56:17.230887+0000","last_undegraded":"2024-04-02T07:56:17.230887+0000","last_fullsized":"2024-04-02T07:56:17.230887+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:16.738817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201489+0000","last_change":"2024-04-02T07:56:17.226039+0000","last_active":"2024-04-02T07:56:18.201489+0000","last_peered":"2024-04-02T07:56:18.201489+0000","last_clean":"2024-04-02T07:56:18.201489+0000","last_became_active":"2024-04-02T07:56:17.225641+0000","last_became_peered":"2024-04-02T07:56:17.225641+0000","last_unstale":"2024-04-02T07:56:18.201489+0000","last_undegraded":"2024-04-02T07:56:18.201489+0000","last_fullsized":"2024-04-02T07:56:18.201489+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:07:38.125893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.d","version":"153'1","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.231298+0000","last_change":"2024-04-02T07:56:16.720235+0000","last_active":"2024-04-02T07:56:17.231298+0000","last_peered":"2024-04-02T07:56:17.231298+0000","last_clean":"2024-04-02T07:56:17.231298+0000","last_became_active":"2024-04-02T07:56:16.719994+0000","last_became_peered":"2024-04-02T07:56:16.719994+0000","last_unstale":"2024-04-02T07:56:17.231298+0000","last_undegraded":"2024-04-02T07:56:17.231298+0000","last_fullsized":"2024-04-02T07:56:17.231298+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:09:58.680838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201244+0000","last_change":"2024-04-02T07:56:17.216379+0000","last_active":"2024-04-02T07:56:18.201244+0000","last_peered":"2024-04-02T07:56:18.201244+0000","last_clean":"2024-04-02T07:56:18.201244+0000","last_became_active":"2024-04-02T07:56:17.216079+0000","last_became_peered":"2024-04-02T07:56:17.216079+0000","last_unstale":"2024-04-02T07:56:18.201244+0000","last_undegraded":"2024-04-02T07:56:18.201244+0000","last_fullsized":"2024-04-02T07:56:18.201244+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:22:34.417939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.199681+0000","last_change":"2024-04-02T07:56:16.716547+0000","last_active":"2024-04-02T07:56:18.199681+0000","last_peered":"2024-04-02T07:56:18.199681+0000","last_clean":"2024-04-02T07:56:18.199681+0000","last_became_active":"2024-04-02T07:56:16.716322+0000","last_became_peered":"2024-04-02T07:56:16.716322+0000","last_unstale":"2024-04-02T07:56:18.199681+0000","last_undegraded":"2024-04-02T07:56:18.199681+0000","last_fullsized":"2024-04-02T07:56:18.199681+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:08:34.246745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201599+0000","last_change":"2024-04-02T07:56:17.228292+0000","last_active":"2024-04-02T07:56:18.201599+0000","last_peered":"2024-04-02T07:56:18.201599+0000","last_clean":"2024-04-02T07:56:18.201599+0000","last_became_active":"2024-04-02T07:56:17.226480+0000","last_became_peered":"2024-04-02T07:56:17.226480+0000","last_unstale":"2024-04-02T07:56:18.201599+0000","last_undegraded":"2024-04-02T07:56:18.201599+0000","last_fullsized":"2024-04-02T07:56:18.201599+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:01:41.298193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.b","version":"153'2","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.230064+0000","last_change":"2024-04-02T07:56:16.718277+0000","last_active":"2024-04-02T07:56:17.230064+0000","last_peered":"2024-04-02T07:56:17.230064+0000","last_clean":"2024-04-02T07:56:17.230064+0000","last_became_active":"2024-04-02T07:56:16.717167+0000","last_became_peered":"2024-04-02T07:56:16.717167+0000","last_unstale":"2024-04-02T07:56:17.230064+0000","last_undegraded":"2024-04-02T07:56:17.230064+0000","last_fullsized":"2024-04-02T07:56:17.230064+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:56.568169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.216922+0000","last_change":"2024-04-02T07:56:17.211469+0000","last_active":"2024-04-02T07:56:17.216922+0000","last_peered":"2024-04-02T07:56:17.216922+0000","last_clean":"2024-04-02T07:56:17.216922+0000","last_became_active":"2024-04-02T07:56:17.211165+0000","last_became_peered":"2024-04-02T07:56:17.211165+0000","last_unstale":"2024-04-02T07:56:17.216922+0000","last_undegraded":"2024-04-02T07:56:17.216922+0000","last_fullsized":"2024-04-02T07:56:17.216922+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:58:19.065313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201985+0000","last_change":"2024-04-02T07:56:16.716277+0000","last_active":"2024-04-02T07:56:18.201985+0000","last_peered":"2024-04-02T07:56:18.201985+0000","last_clean":"2024-04-02T07:56:18.201985+0000","last_became_active":"2024-04-02T07:56:16.715806+0000","last_became_peered":"2024-04-02T07:56:16.715806+0000","last_unstale":"2024-04-02T07:56:18.201985+0000","last_undegraded":"2024-04-02T07:56:18.201985+0000","last_fullsized":"2024-04-02T07:56:18.201985+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:53:23.460581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801259+0000","last_change":"2024-04-02T07:56:17.212227+0000","last_active":"2024-04-02T07:56:18.801259+0000","last_peered":"2024-04-02T07:56:18.801259+0000","last_clean":"2024-04-02T07:56:18.801259+0000","last_became_active":"2024-04-02T07:56:17.211525+0000","last_became_peered":"2024-04-02T07:56:17.211525+0000","last_unstale":"2024-04-02T07:56:18.801259+0000","last_undegraded":"2024-04-02T07:56:18.801259+0000","last_fullsized":"2024-04-02T07:56:18.801259+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:36:20.278308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801048+0000","last_change":"2024-04-02T07:56:16.716753+0000","last_active":"2024-04-02T07:56:18.801048+0000","last_peered":"2024-04-02T07:56:18.801048+0000","last_clean":"2024-04-02T07:56:18.801048+0000","last_became_active":"2024-04-02T07:56:16.714406+0000","last_became_peered":"2024-04-02T07:56:16.714406+0000","last_unstale":"2024-04-02T07:56:18.801048+0000","last_undegraded":"2024-04-02T07:56:18.801048+0000","last_fullsized":"2024-04-02T07:56:18.801048+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:30:15.057311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751172+0000","last_change":"2024-04-02T07:56:17.219225+0000","last_active":"2024-04-02T07:56:18.751172+0000","last_peered":"2024-04-02T07:56:18.751172+0000","last_clean":"2024-04-02T07:56:18.751172+0000","last_became_active":"2024-04-02T07:56:17.218978+0000","last_became_peered":"2024-04-02T07:56:17.218978+0000","last_unstale":"2024-04-02T07:56:18.751172+0000","last_undegraded":"2024-04-02T07:56:18.751172+0000","last_fullsized":"2024-04-02T07:56:18.751172+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:07:25.934705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.202466+0000","last_change":"2024-04-02T07:56:16.711503+0000","last_active":"2024-04-02T07:56:17.202466+0000","last_peered":"2024-04-02T07:56:17.202466+0000","last_clean":"2024-04-02T07:56:17.202466+0000","last_became_active":"2024-04-02T07:56:16.711132+0000","last_became_peered":"2024-04-02T07:56:16.711132+0000","last_unstale":"2024-04-02T07:56:17.202466+0000","last_undegraded":"2024-04-02T07:56:17.202466+0000","last_fullsized":"2024-04-02T07:56:17.202466+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:58:37.517431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.223430+0000","last_change":"2024-04-02T07:56:17.222334+0000","last_active":"2024-04-02T07:56:17.223430+0000","last_peered":"2024-04-02T07:56:17.223430+0000","last_clean":"2024-04-02T07:56:17.223430+0000","last_became_active":"2024-04-02T07:56:17.220969+0000","last_became_peered":"2024-04-02T07:56:17.220969+0000","last_unstale":"2024-04-02T07:56:17.223430+0000","last_undegraded":"2024-04-02T07:56:17.223430+0000","last_fullsized":"2024-04-02T07:56:17.223430+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:29:03.752049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.7","version":"153'2","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.202351+0000","last_change":"2024-04-02T07:56:16.719121+0000","last_active":"2024-04-02T07:56:18.202351+0000","last_peered":"2024-04-02T07:56:18.202351+0000","last_clean":"2024-04-02T07:56:18.202351+0000","last_became_active":"2024-04-02T07:56:16.718846+0000","last_became_peered":"2024-04-02T07:56:16.718846+0000","last_unstale":"2024-04-02T07:56:18.202351+0000","last_undegraded":"2024-04-02T07:56:18.202351+0000","last_fullsized":"2024-04-02T07:56:18.202351+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:24.674919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201380+0000","last_change":"2024-04-02T07:56:17.225390+0000","last_active":"2024-04-02T07:56:18.201380+0000","last_peered":"2024-04-02T07:56:18.201380+0000","last_clean":"2024-04-02T07:56:18.201380+0000","last_became_active":"2024-04-02T07:56:17.225094+0000","last_became_peered":"2024-04-02T07:56:17.225094+0000","last_unstale":"2024-04-02T07:56:18.201380+0000","last_undegraded":"2024-04-02T07:56:18.201380+0000","last_fullsized":"2024-04-02T07:56:18.201380+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:56.051002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.199837+0000","last_change":"2024-04-02T07:56:16.718650+0000","last_active":"2024-04-02T07:56:18.199837+0000","last_peered":"2024-04-02T07:56:18.199837+0000","last_clean":"2024-04-02T07:56:18.199837+0000","last_became_active":"2024-04-02T07:56:16.718344+0000","last_became_peered":"2024-04-02T07:56:16.718344+0000","last_unstale":"2024-04-02T07:56:18.199837+0000","last_undegraded":"2024-04-02T07:56:18.199837+0000","last_fullsized":"2024-04-02T07:56:18.199837+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:47.014632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801352+0000","last_change":"2024-04-02T07:56:17.221890+0000","last_active":"2024-04-02T07:56:18.801352+0000","last_peered":"2024-04-02T07:56:18.801352+0000","last_clean":"2024-04-02T07:56:18.801352+0000","last_became_active":"2024-04-02T07:56:17.220807+0000","last_became_peered":"2024-04-02T07:56:17.220807+0000","last_unstale":"2024-04-02T07:56:18.801352+0000","last_undegraded":"2024-04-02T07:56:18.801352+0000","last_fullsized":"2024-04-02T07:56:18.801352+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:58:38.523923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.202157+0000","last_change":"2024-04-02T07:56:16.718812+0000","last_active":"2024-04-02T07:56:18.202157+0000","last_peered":"2024-04-02T07:56:18.202157+0000","last_clean":"2024-04-02T07:56:18.202157+0000","last_became_active":"2024-04-02T07:56:16.718529+0000","last_became_peered":"2024-04-02T07:56:16.718529+0000","last_unstale":"2024-04-02T07:56:18.202157+0000","last_undegraded":"2024-04-02T07:56:18.202157+0000","last_fullsized":"2024-04-02T07:56:18.202157+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:28.917028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.212608+0000","last_change":"2024-04-02T07:56:17.206998+0000","last_active":"2024-04-02T07:56:17.212608+0000","last_peered":"2024-04-02T07:56:17.212608+0000","last_clean":"2024-04-02T07:56:17.212608+0000","last_became_active":"2024-04-02T07:56:17.206732+0000","last_became_peered":"2024-04-02T07:56:17.206732+0000","last_unstale":"2024-04-02T07:56:17.212608+0000","last_undegraded":"2024-04-02T07:56:17.212608+0000","last_fullsized":"2024-04-02T07:56:17.212608+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:16:29.631645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801329+0000","last_change":"2024-04-02T07:56:16.717912+0000","last_active":"2024-04-02T07:56:18.801329+0000","last_peered":"2024-04-02T07:56:18.801329+0000","last_clean":"2024-04-02T07:56:18.801329+0000","last_became_active":"2024-04-02T07:56:16.717252+0000","last_became_peered":"2024-04-02T07:56:16.717252+0000","last_unstale":"2024-04-02T07:56:18.801329+0000","last_undegraded":"2024-04-02T07:56:18.801329+0000","last_fullsized":"2024-04-02T07:56:18.801329+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:28:08.712631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751145+0000","last_change":"2024-04-02T07:56:17.218043+0000","last_active":"2024-04-02T07:56:18.751145+0000","last_peered":"2024-04-02T07:56:18.751145+0000","last_clean":"2024-04-02T07:56:18.751145+0000","last_became_active":"2024-04-02T07:56:17.217410+0000","last_became_peered":"2024-04-02T07:56:17.217410+0000","last_unstale":"2024-04-02T07:56:18.751145+0000","last_undegraded":"2024-04-02T07:56:18.751145+0000","last_fullsized":"2024-04-02T07:56:18.751145+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:22:34.417939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201831+0000","last_change":"2024-04-02T07:56:16.711117+0000","last_active":"2024-04-02T07:56:18.201831+0000","last_peered":"2024-04-02T07:56:18.201831+0000","last_clean":"2024-04-02T07:56:18.201831+0000","last_became_active":"2024-04-02T07:56:16.710454+0000","last_became_peered":"2024-04-02T07:56:16.710454+0000","last_unstale":"2024-04-02T07:56:18.201831+0000","last_undegraded":"2024-04-02T07:56:18.201831+0000","last_fullsized":"2024-04-02T07:56:18.201831+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:07:30.218500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751282+0000","last_change":"2024-04-02T07:56:17.224241+0000","last_active":"2024-04-02T07:56:18.751282+0000","last_peered":"2024-04-02T07:56:18.751282+0000","last_clean":"2024-04-02T07:56:18.751282+0000","last_became_active":"2024-04-02T07:56:17.215798+0000","last_became_peered":"2024-04-02T07:56:17.215798+0000","last_unstale":"2024-04-02T07:56:18.751282+0000","last_undegraded":"2024-04-02T07:56:18.751282+0000","last_fullsized":"2024-04-02T07:56:18.751282+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:01:29.623806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751223+0000","last_change":"2024-04-02T07:56:16.714562+0000","last_active":"2024-04-02T07:56:18.751223+0000","last_peered":"2024-04-02T07:56:18.751223+0000","last_clean":"2024-04-02T07:56:18.751223+0000","last_became_active":"2024-04-02T07:56:16.714300+0000","last_became_peered":"2024-04-02T07:56:16.714300+0000","last_unstale":"2024-04-02T07:56:18.751223+0000","last_undegraded":"2024-04-02T07:56:18.751223+0000","last_fullsized":"2024-04-02T07:56:18.751223+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:35.555887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201389+0000","last_change":"2024-04-02T07:56:17.216422+0000","last_active":"2024-04-02T07:56:18.201389+0000","last_peered":"2024-04-02T07:56:18.201389+0000","last_clean":"2024-04-02T07:56:18.201389+0000","last_became_active":"2024-04-02T07:56:17.215923+0000","last_became_peered":"2024-04-02T07:56:17.215923+0000","last_unstale":"2024-04-02T07:56:18.201389+0000","last_undegraded":"2024-04-02T07:56:18.201389+0000","last_fullsized":"2024-04-02T07:56:18.201389+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:40:10.320276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.202217+0000","last_change":"2024-04-02T07:56:16.714825+0000","last_active":"2024-04-02T07:56:17.202217+0000","last_peered":"2024-04-02T07:56:17.202217+0000","last_clean":"2024-04-02T07:56:17.202217+0000","last_became_active":"2024-04-02T07:56:16.714539+0000","last_became_peered":"2024-04-02T07:56:16.714539+0000","last_unstale":"2024-04-02T07:56:17.202217+0000","last_undegraded":"2024-04-02T07:56:17.202217+0000","last_fullsized":"2024-04-02T07:56:17.202217+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:36:19.271816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801064+0000","last_change":"2024-04-02T07:56:17.213134+0000","last_active":"2024-04-02T07:56:18.801064+0000","last_peered":"2024-04-02T07:56:18.801064+0000","last_clean":"2024-04-02T07:56:18.801064+0000","last_became_active":"2024-04-02T07:56:17.212837+0000","last_became_peered":"2024-04-02T07:56:17.212837+0000","last_unstale":"2024-04-02T07:56:18.801064+0000","last_undegraded":"2024-04-02T07:56:18.801064+0000","last_fullsized":"2024-04-02T07:56:18.801064+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:17.745309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801260+0000","last_change":"2024-04-02T07:56:16.713716+0000","last_active":"2024-04-02T07:56:18.801260+0000","last_peered":"2024-04-02T07:56:18.801260+0000","last_clean":"2024-04-02T07:56:18.801260+0000","last_became_active":"2024-04-02T07:56:16.713404+0000","last_became_peered":"2024-04-02T07:56:16.713404+0000","last_unstale":"2024-04-02T07:56:18.801260+0000","last_undegraded":"2024-04-02T07:56:18.801260+0000","last_fullsized":"2024-04-02T07:56:18.801260+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:13:39.132006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.220862+0000","last_change":"2024-04-02T07:56:17.216374+0000","last_active":"2024-04-02T07:56:17.220862+0000","last_peered":"2024-04-02T07:56:17.220862+0000","last_clean":"2024-04-02T07:56:17.220862+0000","last_became_active":"2024-04-02T07:56:17.216117+0000","last_became_peered":"2024-04-02T07:56:17.216117+0000","last_unstale":"2024-04-02T07:56:17.220862+0000","last_undegraded":"2024-04-02T07:56:17.220862+0000","last_fullsized":"2024-04-02T07:56:17.220862+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:13:15.311857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.13","version":"153'2","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751356+0000","last_change":"2024-04-02T07:56:16.715391+0000","last_active":"2024-04-02T07:56:18.751356+0000","last_peered":"2024-04-02T07:56:18.751356+0000","last_clean":"2024-04-02T07:56:18.751356+0000","last_became_active":"2024-04-02T07:56:16.715062+0000","last_became_peered":"2024-04-02T07:56:16.715062+0000","last_unstale":"2024-04-02T07:56:18.751356+0000","last_undegraded":"2024-04-02T07:56:18.751356+0000","last_fullsized":"2024-04-02T07:56:18.751356+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:57:06.296830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"143'249","reported_seq":759,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-04-02T07:56:16.697590+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:56:16.697590+0000","last_peered":"2024-04-02T07:56:16.697590+0000","last_clean":"2024-04-02T07:56:16.697590+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:56:16.697590+0000","last_undegraded":"2024-04-02T07:56:16.697590+0000","last_fullsized":"2024-04-02T07:56:16.697590+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575084,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1921024,"data_stored":953007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63200000000000001,"15min":0.66500000000000004},"min":{"1min":0.38600000000000001,"5min":0.34699999999999998,"15min":0.316},"max":{"1min":0.80700000000000005,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.41499999999999998,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.68600000000000005}]},{"osd":1,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65100000000000002,"15min":0.67500000000000004},"min":{"1min":0.48699999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88100000000000001,"5min":1.097,"15min":1.097},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63800000000000001,"15min":0.66500000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.34699999999999998},"max":{"1min":0.78000000000000003,"5min":0.98299999999999998,"15min":1.2190000000000001},"last":0.91900000000000004}]},{"osd":2,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66200000000000003,"15min":0.69899999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.34000000000000002},"max":{"1min":0.88700000000000001,"5min":0.98999999999999999,"15min":1.127},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.72499999999999998,"15min":0.75900000000000001},"min":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95299999999999996,"5min":1.044,"15min":1.1259999999999999},"last":0.88200000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.69899999999999995},"min":{"1min":0.45800000000000002,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.94199999999999995,"5min":0.97799999999999998,"15min":1.026},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68200000000000005,"15min":0.71299999999999997},"min":{"1min":0.52700000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.83399999999999996,"5min":1.071,"15min":1.196},"last":0.83599999999999997}]},{"osd":4,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58799999999999997},"min":{"1min":0.377,"5min":0.35799999999999998,"15min":0.32000000000000001},"max":{"1min":0.85899999999999999,"5min":1.0229999999999999,"15min":1.0389999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68700000000000006,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.004,"5min":1.016,"15min":1.0669999999999999},"last":0.89700000000000002}]},{"osd":5,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.64400000000000002,"15min":0.65000000000000002},"min":{"1min":0.45500000000000002,"5min":0.42999999999999999,"15min":0.37},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.167},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.022,"5min":1.343,"15min":1.4219999999999999},"last":0.95099999999999996}]},{"osd":6,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.36199999999999999},"max":{"1min":0.92000000000000004,"5min":1.0580000000000001,"15min":2.5369999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.66600000000000004,"15min":0.66000000000000003},"min":{"1min":0.48399999999999999,"5min":0.41299999999999998,"15min":0.39600000000000002},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.94199999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575082,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37084,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":1904640,"data_stored":946297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.67000000000000004,"15min":0.70099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.33400000000000002},"max":{"1min":0.89000000000000001,"5min":0.95899999999999996,"15min":0.98099999999999998},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66700000000000004,"15min":0.70399999999999996},"min":{"1min":0.46200000000000002,"5min":0.40500000000000003,"15min":0.38300000000000001},"max":{"1min":0.88600000000000001,"5min":0.94299999999999995,"15min":1.0249999999999999},"last":0.83799999999999997}]},{"osd":1,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67000000000000004,"15min":0.69199999999999995},"min":{"1min":0.505,"5min":0.433,"15min":0.41799999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69899999999999995,"15min":0.71499999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":0.95799999999999996,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.67200000000000004}]},{"osd":2,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68000000000000005,"15min":0.70999999999999996},"min":{"1min":0.46500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68899999999999995,"15min":0.70699999999999996},"min":{"1min":0.52600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.70899999999999996}]},{"osd":3,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71199999999999997,"15min":0.72899999999999998},"min":{"1min":0.53200000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.73899999999999999,"15min":0.745},"min":{"1min":0.52800000000000002,"5min":0.46600000000000003,"15min":0.45300000000000001},"max":{"1min":3.339,"5min":3.339,"15min":3.339},"last":0.68799999999999994}]},{"osd":4,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.625,"15min":0.60699999999999998},"min":{"1min":0.39100000000000001,"5min":0.377,"15min":0.372},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64100000000000001,"15min":0.63300000000000001},"min":{"1min":0.44500000000000001,"5min":0.41499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.59899999999999998}]},{"osd":5,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.629,"15min":0.625},"min":{"1min":0.47599999999999998,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.89200000000000002,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.69999999999999996,"15min":0.68600000000000005},"min":{"1min":0.45500000000000002,"5min":0.40899999999999997,"15min":0.40300000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.72299999999999998}]},{"osd":7,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.69999999999999996,"15min":0.68999999999999995},"min":{"1min":0.501,"5min":0.46999999999999997,"15min":0.40200000000000002},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.52000000000000002,"5min":0.42799999999999999,"15min":0.42099999999999999},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.81899999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575083,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59820,"kb_used_data":2456,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312020,"statfs":{"total":96636764160,"available":96575508480,"internally_reserved":0,"allocated":2514944,"data_stored":1538251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.54400000000000004,"15min":0.47599999999999998},"min":{"1min":0.30199999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.57599999999999996,"5min":10.052,"15min":10.052},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.59399999999999997,"15min":0.53300000000000003},"min":{"1min":0.30399999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61799999999999999,"5min":9.0250000000000004,"15min":9.0250000000000004},"last":0.33900000000000002}]},{"osd":2,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.58799999999999997,"15min":0.53000000000000003},"min":{"1min":0.36099999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.55500000000000005,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.64100000000000001,"15min":0.59899999999999998},"min":{"1min":0.45500000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.68899999999999995,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.63300000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64500000000000002,"5min":2.609,"15min":2.609},"last":0.436},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66800000000000004,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.51400000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.75800000000000001,"15min":0.77000000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86599999999999999,"5min":2.21,"15min":2.21},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.81000000000000005,"15min":0.78500000000000003},"min":{"1min":0.47899999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.96299999999999997,"5min":6.3419999999999996,"15min":6.3419999999999996},"last":0.879}]},{"osd":5,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.82499999999999996,"15min":0.78900000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.41999999999999998},"max":{"1min":0.96299999999999997,"5min":6.5529999999999999,"15min":6.5529999999999999},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.78200000000000003,"15min":0.79000000000000004},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.48799999999999999},"max":{"1min":1.0189999999999999,"5min":2.544,"15min":2.544},"last":0.83999999999999997}]},{"osd":6,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":1.849,"15min":1.1519999999999999},"min":{"1min":0.55200000000000005,"5min":0.53200000000000003,"15min":0.47699999999999998},"max":{"1min":0.95399999999999996,"5min":78.903000000000006,"15min":78.903000000000006},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":1.8260000000000001,"15min":1.139},"min":{"1min":0.52400000000000002,"5min":0.49199999999999999,"15min":0.38100000000000001},"max":{"1min":0.89600000000000002,"5min":78.837000000000003,"15min":78.837000000000003},"last":0.79500000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78500000000000003,"15min":0.80000000000000004},"min":{"1min":0.55400000000000005,"5min":0.52800000000000002,"15min":0.48799999999999999},"max":{"1min":0.93200000000000005,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.79100000000000004,"15min":0.81799999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.97699999999999998,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":0.995}]}]},{"osd":0,"up_from":13,"seq":55834575083,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1921024,"data_stored":946898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.42199999999999999,"15min":0.437},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67800000000000005,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.45900000000000002,"15min":0.48199999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.629,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.377}]},{"osd":2,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47299999999999998,"15min":0.48599999999999999},"min":{"1min":0.32200000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":1.597,"15min":1.597},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51100000000000001,"15min":0.52700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72699999999999998,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.438}]},{"osd":3,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.50700000000000001,"15min":0.52300000000000002},"min":{"1min":0.38700000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81799999999999995},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.60199999999999998},"min":{"1min":0.40300000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.78500000000000003,"5min":1.492,"15min":1.492},"last":0.52100000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.753,"15min":0.77600000000000002},"min":{"1min":0.51500000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0409999999999999,"5min":1.1579999999999999,"15min":1.181},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76300000000000001,"15min":0.77300000000000002},"min":{"1min":0.54300000000000004,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":1.097,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.78100000000000003}]},{"osd":5,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.76600000000000001,"15min":0.79100000000000004},"min":{"1min":0.56599999999999995,"5min":0.48699999999999999,"15min":0.47299999999999998},"max":{"1min":0.998,"5min":1.361,"15min":2.1869999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.76900000000000002,"15min":0.79500000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.026,"5min":1.546,"15min":2.6230000000000002},"last":0.71299999999999997}]},{"osd":6,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.73099999999999998,"15min":0.747},"min":{"1min":0.47999999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.93700000000000006,"5min":1.52,"15min":1.52},"last":0.628},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.71799999999999997,"15min":0.745},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.41299999999999998},"max":{"1min":1.026,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.79500000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.78800000000000003,"15min":0.81200000000000006},"min":{"1min":0.53600000000000003,"5min":0.51600000000000001,"15min":0.505},"max":{"1min":1.002,"5min":1.139,"15min":2.3959999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.76200000000000001,"15min":0.80100000000000005},"min":{"1min":0.50900000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0960000000000001,"5min":1.3959999999999999,"15min":2.145},"last":0.81799999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575082,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37100,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1925120,"data_stored":953645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.29399999999999998,"5min":0.26700000000000002,"15min":0.26400000000000001},"max":{"1min":0.70499999999999996,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46700000000000003,"15min":0.47299999999999998},"min":{"1min":0.33600000000000002,"5min":0.29099999999999998,"15min":0.26200000000000001},"max":{"1min":0.58899999999999997,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.41099999999999998}]},{"osd":1,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.33800000000000002,"5min":0.32700000000000001,"15min":0.30499999999999999},"max":{"1min":0.58699999999999997,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.5,"15min":0.51700000000000002},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":0.58499999999999996,"5min":0.996,"15min":0.996},"last":0.45500000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.316},"max":{"1min":0.82399999999999995,"5min":1.472,"15min":1.472},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.59199999999999997,"15min":0.60299999999999998},"min":{"1min":0.42899999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.70999999999999996,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.57899999999999996}]},{"osd":4,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.72699999999999998,"15min":0.76000000000000001},"min":{"1min":0.52300000000000002,"5min":0.5,"15min":0.47199999999999998},"max":{"1min":1.046,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.62},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.73199999999999998,"15min":0.76200000000000001},"min":{"1min":0.56000000000000005,"5min":0.55000000000000004,"15min":0.48499999999999999},"max":{"1min":1.1040000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.753}]},{"osd":5,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.72199999999999998,"15min":0.751},"min":{"1min":0.54300000000000004,"5min":0.51000000000000001,"15min":0.46200000000000002},"max":{"1min":0.91500000000000004,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.76900000000000002},"min":{"1min":0.61499999999999999,"5min":0.55600000000000005,"15min":0.50900000000000001},"max":{"1min":0.95199999999999996,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.70499999999999996}]},{"osd":6,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.747,"15min":0.77800000000000002},"min":{"1min":0.58399999999999996,"5min":0.501,"15min":0.442},"max":{"1min":1.079,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.77100000000000002},"min":{"1min":0.56499999999999995,"5min":0.502,"15min":0.502},"max":{"1min":0.871,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.64600000000000002}]},{"osd":7,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.78400000000000003},"min":{"1min":0.55900000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0249999999999999,"5min":1.278,"15min":1.278},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73299999999999998,"15min":0.77900000000000003},"min":{"1min":0.504,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.99099999999999999,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.67800000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59224,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312616,"statfs":{"total":96636764160,"available":96576118784,"internally_reserved":0,"allocated":1908736,"data_stored":946367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.437,"15min":0.42899999999999999},"min":{"1min":0.314,"5min":0.26500000000000001,"15min":0.23499999999999999},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.44600000000000001,"15min":0.46000000000000002},"min":{"1min":0.308,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58599999999999997,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.38600000000000001}]},{"osd":1,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46999999999999997,"15min":0.48699999999999999},"min":{"1min":0.33900000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69599999999999995,"5min":0.78000000000000003,"15min":1.2809999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56899999999999995,"15min":0.56299999999999994},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71099999999999997,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":0.47799999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.60799999999999998},"min":{"1min":0.42599999999999999,"5min":0.40899999999999997,"15min":0.38700000000000001},"max":{"1min":0.88900000000000001,"5min":2.214,"15min":2.214},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58499999999999996,"15min":0.60099999999999998},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.46000000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.72999999999999998,"15min":0.73299999999999998},"min":{"1min":0.53600000000000003,"5min":0.47699999999999998,"15min":0.38400000000000001},"max":{"1min":0.86499999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.39600000000000002},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.63200000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72799999999999998,"15min":0.751},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.40899999999999997},"max":{"1min":0.84399999999999997,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.74399999999999999,"15min":0.749},"min":{"1min":0.51100000000000001,"5min":0.49199999999999999,"15min":0.40200000000000002},"max":{"1min":0.90000000000000002,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.76000000000000001}]},{"osd":6,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76900000000000002,"15min":0.76500000000000001},"min":{"1min":0.55300000000000005,"5min":0.51100000000000001,"15min":0.45800000000000002},"max":{"1min":1.4359999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73899999999999999,"15min":0.76800000000000002},"min":{"1min":0.51700000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.90600000000000003,"5min":2.286,"15min":2.286},"last":0.64400000000000002}]},{"osd":7,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.746,"15min":0.76700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.439},"max":{"1min":0.82799999999999996,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77400000000000002,"15min":0.78500000000000003},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.48999999999999999},"max":{"1min":1.3200000000000001,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.71199999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575082,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41188,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330652,"statfs":{"total":96636764160,"available":96594587648,"internally_reserved":0,"allocated":1916928,"data_stored":952973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.63800000000000001,"15min":0.66300000000000003},"min":{"1min":0.45300000000000001,"5min":0.41699999999999998,"15min":0.38100000000000001},"max":{"1min":0.82599999999999996,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.61499999999999999,"15min":0.66000000000000003},"min":{"1min":0.432,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.77700000000000002,"5min":0.94999999999999996,"15min":0.97599999999999998},"last":0.68999999999999995}]},{"osd":1,"last update":"Tue Apr 2 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.69299999999999995,"15min":0.71599999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.438},"max":{"1min":0.876,"5min":1.014,"15min":1.0349999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70399999999999996,"15min":0.72999999999999998},"min":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.36799999999999999},"max":{"1min":0.89000000000000001,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.81899999999999995}]},{"osd":2,"last update":"Tue Apr 2 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.66700000000000004,"15min":0.68600000000000005},"min":{"1min":0.44700000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.78800000000000003,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.68700000000000006,"15min":0.70699999999999996},"min":{"1min":0.48799999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.92200000000000004,"5min":1.026,"15min":1.026},"last":0.91100000000000003}]},{"osd":3,"last update":"Tue Apr 2 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.46800000000000003},"max":{"1min":0.89800000000000002,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.68999999999999995,"15min":0.71899999999999997},"min":{"1min":0.54600000000000004,"5min":0.47299999999999998,"15min":0.41499999999999998},"max":{"1min":0.85399999999999998,"5min":1.046,"15min":1.046},"last":0.73299999999999998}]},{"osd":5,"last update":"Tue Apr 2 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.621,"15min":0.58699999999999997},"min":{"1min":0.41799999999999998,"5min":0.39400000000000002,"15min":0.29599999999999999},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61099999999999999,"15min":0.59099999999999997},"min":{"1min":0.46700000000000003,"5min":0.41699999999999998,"15min":0.33700000000000002},"max":{"1min":0.95699999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.63900000000000001}]},{"osd":6,"last update":"Tue Apr 2 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.64000000000000001,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.374},"max":{"1min":0.95199999999999996,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.622},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.68600000000000005,"15min":0.65800000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.88600000000000001}]},{"osd":7,"last update":"Tue Apr 2 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.68899999999999995,"15min":0.67900000000000005},"min":{"1min":0.5,"5min":0.39600000000000002,"15min":0.38300000000000001},"max":{"1min":0.88800000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.75900000000000001,"15min":0.73999999999999999},"min":{"1min":0.56399999999999995,"5min":0.52200000000000002,"15min":0.434},"max":{"1min":1.004,"5min":1.095,"15min":1.095},"last":0.76800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575085,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55712,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316128,"statfs":{"total":96636764160,"available":96579715072,"internally_reserved":0,"allocated":2506752,"data_stored":1536388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63700000000000001,"15min":0.67500000000000004},"min":{"1min":0.41999999999999998,"5min":0.35299999999999998,"15min":0.29899999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.99199999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.64800000000000002,"15min":0.68100000000000005},"min":{"1min":0.51000000000000001,"5min":0.40500000000000003,"15min":0.378},"max":{"1min":0.85599999999999998,"5min":1.0249999999999999,"15min":1.4450000000000001},"last":0.69099999999999995}]},{"osd":1,"last update":"Tue Apr 2 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.63600000000000001,"15min":0.67800000000000005},"min":{"1min":0.45300000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.81399999999999995,"5min":0.88100000000000001,"15min":1.131},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.66200000000000003,"15min":0.68700000000000006},"min":{"1min":0.47099999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.78300000000000003,"5min":0.96399999999999997,"15min":1.5149999999999999},"last":0.73699999999999999}]},{"osd":2,"last update":"Tue Apr 2 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.66000000000000003,"15min":0.69999999999999996},"min":{"1min":0.437,"5min":0.42799999999999999,"15min":0.41799999999999998},"max":{"1min":0.84699999999999998,"5min":0.90700000000000003,"15min":1.5449999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67000000000000004,"15min":0.69999999999999996},"min":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.372},"max":{"1min":0.879,"5min":0.89700000000000002,"15min":1.3959999999999999},"last":0.75600000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.72999999999999998},"min":{"1min":0.49199999999999999,"5min":0.436,"15min":0.436},"max":{"1min":0.89500000000000002,"5min":1.0649999999999999,"15min":1.341},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.73199999999999998,"15min":0.75900000000000001},"min":{"1min":0.53700000000000003,"5min":0.48299999999999998,"15min":0.44900000000000001},"max":{"1min":0.92000000000000004,"5min":0.97499999999999998,"15min":1.484},"last":0.79100000000000004}]},{"osd":4,"last update":"Tue Apr 2 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84399999999999997,"5min":0.84599999999999997,"15min":1.3109999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64400000000000002,"15min":0.64100000000000001},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.35599999999999998},"max":{"1min":0.81899999999999995,"5min":0.97999999999999998,"15min":1.504},"last":0.77400000000000002}]},{"osd":6,"last update":"Tue Apr 2 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.65200000000000002,"15min":0.64300000000000002},"min":{"1min":0.42899999999999999,"5min":0.41599999999999998,"15min":0.39900000000000002},"max":{"1min":0.871,"5min":1.004,"15min":1.212},"last":0.63},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.67800000000000005,"15min":0.67400000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.88500000000000001,"5min":0.92800000000000005,"15min":1.774},"last":0.65800000000000003}]},{"osd":7,"last update":"Tue Apr 2 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.71699999999999997,"15min":0.70499999999999996},"min":{"1min":0.53900000000000003,"5min":0.45000000000000001,"15min":0.38900000000000001},"max":{"1min":0.80700000000000005,"5min":1.1020000000000001,"15min":1.6890000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.497,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":0.93700000000000006,"5min":1.139,"15min":1.5309999999999999},"last":0.81599999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:56:21.572 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T07:56:21.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.798+0000 7f170ce00700 1 Processor -- start 2024-04-02T07:56:21.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.798+0000 7f170ce00700 1 -- start start 2024-04-02T07:56:21.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.799+0000 7f170ce00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1708160f10 0x7f17080d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:21.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.799+0000 7f170ce00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17080cf230 0x7f17080d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:21.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.799+0000 7f170ce00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1708161590 0x7f1708161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:21.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.799+0000 7f170ce00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1708161f60 con 0x7f1708160f10 2024-04-02T07:56:21.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.799+0000 7f170ce00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17080ceda0 con 0x7f1708161590 2024-04-02T07:56:21.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.799+0000 7f170ce00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17080d23d0 con 0x7f17080cf230 2024-04-02T07:56:21.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.800+0000 7f1705e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1708160f10 0x7f17080d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:21.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.800+0000 7f1705e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1708160f10 0x7f17080d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42330/0 (socket says 172.21.15.89:42330) 2024-04-02T07:56:21.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.800+0000 7f1705e00700 1 -- 172.21.15.89:0/3730004967 learned_addr learned my addr 172.21.15.89:0/3730004967 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:21.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.800+0000 7f1706800700 1 --2- 172.21.15.89:0/3730004967 >> v2:172.21.15.113:3300/0 conn(0x7f1708161590 0x7f1708161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:21.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.800+0000 7f1705400700 1 --2- 172.21.15.89:0/3730004967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17080cf230 0x7f17080d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:21.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.800+0000 7f1705e00700 1 -- 172.21.15.89:0/3730004967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17080cf230 msgr2=0x7f17080d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:21.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.800+0000 7f1705e00700 1 --2- 172.21.15.89:0/3730004967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17080cf230 0x7f17080d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.800+0000 7f1705e00700 1 -- 172.21.15.89:0/3730004967 >> v2:172.21.15.113:3300/0 conn(0x7f1708161590 msgr2=0x7f1708161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:21.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.800+0000 7f1705e00700 1 --2- 172.21.15.89:0/3730004967 >> v2:172.21.15.113:3300/0 conn(0x7f1708161590 0x7f1708161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.800+0000 7f1705e00700 1 -- 172.21.15.89:0/3730004967 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17081612f0 con 0x7f1708160f10 2024-04-02T07:56:21.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.800+0000 7f1705400700 1 --2- 172.21.15.89:0/3730004967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17080cf230 0x7f17080d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:21.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.800+0000 7f1705e00700 1 --2- 172.21.15.89:0/3730004967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1708160f10 0x7f17080d18b0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f17080cfad0 tx=0x7f16f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dbdf06b6af03a63 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.801+0000 7f1704a00700 1 -- 172.21.15.89:0/3730004967 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f800b870 con 0x7f1708160f10 2024-04-02T07:56:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.801+0000 7f1704a00700 1 -- 172.21.15.89:0/3730004967 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f800b9d0 con 0x7f1708160f10 2024-04-02T07:56:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.801+0000 7f1704a00700 1 -- 172.21.15.89:0/3730004967 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f800bd60 con 0x7f1708160f10 2024-04-02T07:56:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.801+0000 7f170ce00700 1 -- 172.21.15.89:0/3730004967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1708160f10 msgr2=0x7f17080d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.801+0000 7f170ce00700 1 --2- 172.21.15.89:0/3730004967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1708160f10 0x7f17080d18b0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f17080cfad0 tx=0x7f16f8009bd0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.801+0000 7f170ce00700 1 -- 172.21.15.89:0/3730004967 shutdown_connections 2024-04-02T07:56:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.801+0000 7f170ce00700 1 --2- 172.21.15.89:0/3730004967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1708160f10 0x7f17080d18b0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.801+0000 7f170ce00700 1 --2- 172.21.15.89:0/3730004967 >> v2:172.21.15.113:3300/0 conn(0x7f1708161590 0x7f1708161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.801+0000 7f170ce00700 1 --2- 172.21.15.89:0/3730004967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17080cf230 0x7f17080d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.801+0000 7f170ce00700 1 -- 172.21.15.89:0/3730004967 >> 172.21.15.89:0/3730004967 conn(0x7f170808aff0 msgr2=0x7f17080d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.801+0000 7f170ce00700 1 -- 172.21.15.89:0/3730004967 shutdown_connections 2024-04-02T07:56:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.801+0000 7f170ce00700 1 -- 172.21.15.89:0/3730004967 wait complete. 2024-04-02T07:56:21.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.803+0000 7f170ce00700 1 Processor -- start 2024-04-02T07:56:21.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.803+0000 7f170ce00700 1 -- start start 2024-04-02T07:56:21.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.804+0000 7f170ce00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17080cf230 0x7f17080b8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:21.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.804+0000 7f170ce00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1708160f10 0x7f1708158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:21.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.804+0000 7f170ce00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1708161590 0x7f17081585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:21.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.804+0000 7f170ce00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17081f3fb0 con 0x7f1708160f10 2024-04-02T07:56:21.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.804+0000 7f170ce00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1708157170 con 0x7f17080cf230 2024-04-02T07:56:21.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.804+0000 7f170ce00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1708156560 con 0x7f1708161590 2024-04-02T07:56:21.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.804+0000 7f1705400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1708160f10 0x7f1708158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:21.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.804+0000 7f1705400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1708160f10 0x7f1708158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42332/0 (socket says 172.21.15.89:42332) 2024-04-02T07:56:21.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.804+0000 7f1705400700 1 -- 172.21.15.89:0/1170441839 learned_addr learned my addr 172.21.15.89:0/1170441839 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:21.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.804+0000 7f1705e00700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17080cf230 0x7f17080b8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:21.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.805+0000 7f1706800700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1708161590 0x7f17081585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:21.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.805+0000 7f1705400700 1 -- 172.21.15.89:0/1170441839 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1708161590 msgr2=0x7f17081585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:21.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.805+0000 7f1705400700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1708161590 0x7f17081585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.805+0000 7f1705400700 1 -- 172.21.15.89:0/1170441839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17080cf230 msgr2=0x7f17080b8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:21.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.805+0000 7f1705400700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17080cf230 0x7f17080b8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:21.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.805+0000 7f1705400700 1 -- 172.21.15.89:0/1170441839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f8009550 con 0x7f1708160f10 2024-04-02T07:56:21.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.805+0000 7f1706800700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1708161590 0x7f17081585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:21.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.805+0000 7f1705400700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1708160f10 0x7f1708158060 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f16f400c7f0 tx=0x7f16f400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:21.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.805+0000 7f16fe000700 1 -- 172.21.15.89:0/1170441839 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4011800 con 0x7f1708160f10 2024-04-02T07:56:21.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.805+0000 7f16fe000700 1 -- 172.21.15.89:0/1170441839 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f4011960 con 0x7f1708160f10 2024-04-02T07:56:21.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.805+0000 7f170ce00700 1 -- 172.21.15.89:0/1170441839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1708156840 con 0x7f1708160f10 2024-04-02T07:56:21.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.805+0000 7f16fe000700 1 -- 172.21.15.89:0/1170441839 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4011cf0 con 0x7f1708160f10 2024-04-02T07:56:21.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.805+0000 7f170ce00700 1 -- 172.21.15.89:0/1170441839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1708156e00 con 0x7f1708160f10 2024-04-02T07:56:21.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.807+0000 7f170ce00700 1 -- 172.21.15.89:0/1170441839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1708067720 con 0x7f1708160f10 2024-04-02T07:56:21.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.807+0000 7f16fe000700 1 -- 172.21.15.89:0/1170441839 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f16f4018360 con 0x7f1708160f10 2024-04-02T07:56:21.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.808+0000 7f16fe000700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f16e806adb0 0x7f16e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:21.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.809+0000 7f16fe000700 1 -- 172.21.15.89:0/1170441839 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f16f4089670 con 0x7f1708160f10 2024-04-02T07:56:21.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.809+0000 7f1705e00700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f16e806adb0 0x7f16e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:21.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.809+0000 7f1705e00700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f16e806adb0 0x7f16e806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f16f8014040 tx=0x7f16f8008030 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:21.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:21.812+0000 7f16fe000700 1 -- 172.21.15.89:0/1170441839 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f16f40544c0 con 0x7f1708160f10 2024-04-02T07:56:22.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.010+0000 7f170ce00700 1 -- 172.21.15.89:0/1170441839 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17080cea30 con 0x7f16e806adb0 2024-04-02T07:56:22.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.018+0000 7f16fe000700 1 -- 172.21.15.89:0/1170441839 <== mgr.8790 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289736 (secure 0 0 0) 0x7f16f800bc70 con 0x7f16e806adb0 2024-04-02T07:56:22.024 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:22.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.022+0000 7f170ce00700 1 -- 172.21.15.89:0/1170441839 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f16e806adb0 msgr2=0x7f16e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:22.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.022+0000 7f170ce00700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f16e806adb0 0x7f16e806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f16f8014040 tx=0x7f16f8008030 comp rx=0 tx=0).stop 2024-04-02T07:56:22.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.022+0000 7f170ce00700 1 -- 172.21.15.89:0/1170441839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1708160f10 msgr2=0x7f1708158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:22.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.022+0000 7f170ce00700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1708160f10 0x7f1708158060 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f16f400c7f0 tx=0x7f16f400cb00 comp rx=0 tx=0).stop 2024-04-02T07:56:22.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.023+0000 7f170ce00700 1 -- 172.21.15.89:0/1170441839 shutdown_connections 2024-04-02T07:56:22.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.023+0000 7f170ce00700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f16e806adb0 0x7f16e806d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.023+0000 7f170ce00700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17080cf230 0x7f17080b8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.023+0000 7f170ce00700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1708161590 0x7f17081585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.023+0000 7f170ce00700 1 --2- 172.21.15.89:0/1170441839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1708160f10 0x7f1708158060 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.023+0000 7f170ce00700 1 -- 172.21.15.89:0/1170441839 >> 172.21.15.89:0/1170441839 conn(0x7f170808aff0 msgr2=0x7f170806ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:22.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.023+0000 7f170ce00700 1 -- 172.21.15.89:0/1170441839 shutdown_connections 2024-04-02T07:56:22.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.023+0000 7f170ce00700 1 -- 172.21.15.89:0/1170441839 wait complete. 2024-04-02T07:56:22.027 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T07:56:22.041 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T07:56:21.189838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":356124,"kb_used_data":16132,"kb_used_omap":12,"kb_used_meta":339827,"kb_avail":754618596,"statfs":{"total":773094113280,"available":772729442304,"internally_reserved":0,"allocated":16519168,"data_stored":8773826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":347983416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.499619"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201272+0000","last_change":"2024-04-02T07:56:17.228217+0000","last_active":"2024-04-02T07:56:18.201272+0000","last_peered":"2024-04-02T07:56:18.201272+0000","last_clean":"2024-04-02T07:56:18.201272+0000","last_became_active":"2024-04-02T07:56:17.227934+0000","last_became_peered":"2024-04-02T07:56:17.227934+0000","last_unstale":"2024-04-02T07:56:18.201272+0000","last_undegraded":"2024-04-02T07:56:18.201272+0000","last_fullsized":"2024-04-02T07:56:18.201272+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:17:27.958914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201588+0000","last_change":"2024-04-02T07:56:17.220920+0000","last_active":"2024-04-02T07:56:18.201588+0000","last_peered":"2024-04-02T07:56:18.201588+0000","last_clean":"2024-04-02T07:56:18.201588+0000","last_became_active":"2024-04-02T07:56:17.220658+0000","last_became_peered":"2024-04-02T07:56:17.220658+0000","last_unstale":"2024-04-02T07:56:18.201588+0000","last_undegraded":"2024-04-02T07:56:18.201588+0000","last_fullsized":"2024-04-02T07:56:18.201588+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:34.929140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.212627+0000","last_change":"2024-04-02T07:56:17.208856+0000","last_active":"2024-04-02T07:56:17.212627+0000","last_peered":"2024-04-02T07:56:17.212627+0000","last_clean":"2024-04-02T07:56:17.212627+0000","last_became_active":"2024-04-02T07:56:17.208458+0000","last_became_peered":"2024-04-02T07:56:17.208458+0000","last_unstale":"2024-04-02T07:56:17.212627+0000","last_undegraded":"2024-04-02T07:56:17.212627+0000","last_fullsized":"2024-04-02T07:56:17.212627+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:15:20.199042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.221804+0000","last_change":"2024-04-02T07:56:17.219167+0000","last_active":"2024-04-02T07:56:17.221804+0000","last_peered":"2024-04-02T07:56:17.221804+0000","last_clean":"2024-04-02T07:56:17.221804+0000","last_became_active":"2024-04-02T07:56:17.218904+0000","last_became_peered":"2024-04-02T07:56:17.218904+0000","last_unstale":"2024-04-02T07:56:17.221804+0000","last_undegraded":"2024-04-02T07:56:17.221804+0000","last_fullsized":"2024-04-02T07:56:17.221804+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:31.191815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.199830+0000","last_change":"2024-04-02T07:56:17.217244+0000","last_active":"2024-04-02T07:56:18.199830+0000","last_peered":"2024-04-02T07:56:18.199830+0000","last_clean":"2024-04-02T07:56:18.199830+0000","last_became_active":"2024-04-02T07:56:17.216765+0000","last_became_peered":"2024-04-02T07:56:17.216765+0000","last_unstale":"2024-04-02T07:56:18.199830+0000","last_undegraded":"2024-04-02T07:56:18.199830+0000","last_fullsized":"2024-04-02T07:56:18.199830+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:58:19.065313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751139+0000","last_change":"2024-04-02T07:56:17.226865+0000","last_active":"2024-04-02T07:56:18.751139+0000","last_peered":"2024-04-02T07:56:18.751139+0000","last_clean":"2024-04-02T07:56:18.751139+0000","last_became_active":"2024-04-02T07:56:17.226581+0000","last_became_peered":"2024-04-02T07:56:17.226581+0000","last_unstale":"2024-04-02T07:56:18.751139+0000","last_undegraded":"2024-04-02T07:56:18.751139+0000","last_fullsized":"2024-04-02T07:56:18.751139+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:56:04.841946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.199574+0000","last_change":"2024-04-02T07:56:17.212002+0000","last_active":"2024-04-02T07:56:18.199574+0000","last_peered":"2024-04-02T07:56:18.199574+0000","last_clean":"2024-04-02T07:56:18.199574+0000","last_became_active":"2024-04-02T07:56:17.211724+0000","last_became_peered":"2024-04-02T07:56:17.211724+0000","last_unstale":"2024-04-02T07:56:18.199574+0000","last_undegraded":"2024-04-02T07:56:18.199574+0000","last_fullsized":"2024-04-02T07:56:18.199574+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:44:51.037155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.199430+0000","last_change":"2024-04-02T07:56:17.215116+0000","last_active":"2024-04-02T07:56:18.199430+0000","last_peered":"2024-04-02T07:56:18.199430+0000","last_clean":"2024-04-02T07:56:18.199430+0000","last_became_active":"2024-04-02T07:56:17.213978+0000","last_became_peered":"2024-04-02T07:56:17.213978+0000","last_unstale":"2024-04-02T07:56:18.199430+0000","last_undegraded":"2024-04-02T07:56:18.199430+0000","last_fullsized":"2024-04-02T07:56:18.199430+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:52.462023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.208904+0000","last_change":"2024-04-02T07:56:17.203201+0000","last_active":"2024-04-02T07:56:17.208904+0000","last_peered":"2024-04-02T07:56:17.208904+0000","last_clean":"2024-04-02T07:56:17.208904+0000","last_became_active":"2024-04-02T07:56:17.202926+0000","last_became_peered":"2024-04-02T07:56:17.202926+0000","last_unstale":"2024-04-02T07:56:17.208904+0000","last_undegraded":"2024-04-02T07:56:17.208904+0000","last_fullsized":"2024-04-02T07:56:17.208904+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:36:17.721861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201702+0000","last_change":"2024-04-02T07:56:17.221434+0000","last_active":"2024-04-02T07:56:18.201702+0000","last_peered":"2024-04-02T07:56:18.201702+0000","last_clean":"2024-04-02T07:56:18.201702+0000","last_became_active":"2024-04-02T07:56:17.221181+0000","last_became_peered":"2024-04-02T07:56:17.221181+0000","last_unstale":"2024-04-02T07:56:18.201702+0000","last_undegraded":"2024-04-02T07:56:18.201702+0000","last_fullsized":"2024-04-02T07:56:18.201702+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:25:01.575042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.214726+0000","last_change":"2024-04-02T07:56:17.213301+0000","last_active":"2024-04-02T07:56:17.214726+0000","last_peered":"2024-04-02T07:56:17.214726+0000","last_clean":"2024-04-02T07:56:17.214726+0000","last_became_active":"2024-04-02T07:56:17.212433+0000","last_became_peered":"2024-04-02T07:56:17.212433+0000","last_unstale":"2024-04-02T07:56:17.214726+0000","last_undegraded":"2024-04-02T07:56:17.214726+0000","last_fullsized":"2024-04-02T07:56:17.214726+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:07:38.125893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751344+0000","last_change":"2024-04-02T07:56:17.224697+0000","last_active":"2024-04-02T07:56:18.751344+0000","last_peered":"2024-04-02T07:56:18.751344+0000","last_clean":"2024-04-02T07:56:18.751344+0000","last_became_active":"2024-04-02T07:56:17.223662+0000","last_became_peered":"2024-04-02T07:56:17.223662+0000","last_unstale":"2024-04-02T07:56:18.751344+0000","last_undegraded":"2024-04-02T07:56:18.751344+0000","last_fullsized":"2024-04-02T07:56:18.751344+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:16:19.108293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.225851+0000","last_change":"2024-04-02T07:56:17.223145+0000","last_active":"2024-04-02T07:56:17.225851+0000","last_peered":"2024-04-02T07:56:17.225851+0000","last_clean":"2024-04-02T07:56:17.225851+0000","last_became_active":"2024-04-02T07:56:17.222732+0000","last_became_peered":"2024-04-02T07:56:17.222732+0000","last_unstale":"2024-04-02T07:56:17.225851+0000","last_undegraded":"2024-04-02T07:56:17.225851+0000","last_fullsized":"2024-04-02T07:56:17.225851+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:01:41.298193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801079+0000","last_change":"2024-04-02T07:56:17.216980+0000","last_active":"2024-04-02T07:56:18.801079+0000","last_peered":"2024-04-02T07:56:18.801079+0000","last_clean":"2024-04-02T07:56:18.801079+0000","last_became_active":"2024-04-02T07:56:17.216689+0000","last_became_peered":"2024-04-02T07:56:17.216689+0000","last_unstale":"2024-04-02T07:56:18.801079+0000","last_undegraded":"2024-04-02T07:56:18.801079+0000","last_fullsized":"2024-04-02T07:56:18.801079+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:59:49.826346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201354+0000","last_change":"2024-04-02T07:56:17.216532+0000","last_active":"2024-04-02T07:56:18.201354+0000","last_peered":"2024-04-02T07:56:18.201354+0000","last_clean":"2024-04-02T07:56:18.201354+0000","last_became_active":"2024-04-02T07:56:17.216264+0000","last_became_peered":"2024-04-02T07:56:17.216264+0000","last_unstale":"2024-04-02T07:56:18.201354+0000","last_undegraded":"2024-04-02T07:56:18.201354+0000","last_fullsized":"2024-04-02T07:56:18.201354+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:13:40.138497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801012+0000","last_change":"2024-04-02T07:56:17.216064+0000","last_active":"2024-04-02T07:56:18.801012+0000","last_peered":"2024-04-02T07:56:18.801012+0000","last_clean":"2024-04-02T07:56:18.801012+0000","last_became_active":"2024-04-02T07:56:17.215729+0000","last_became_peered":"2024-04-02T07:56:17.215729+0000","last_unstale":"2024-04-02T07:56:18.801012+0000","last_undegraded":"2024-04-02T07:56:18.801012+0000","last_fullsized":"2024-04-02T07:56:18.801012+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:26:51.859488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.214434+0000","last_change":"2024-04-02T07:56:17.209022+0000","last_active":"2024-04-02T07:56:17.214434+0000","last_peered":"2024-04-02T07:56:17.214434+0000","last_clean":"2024-04-02T07:56:17.214434+0000","last_became_active":"2024-04-02T07:56:17.208739+0000","last_became_peered":"2024-04-02T07:56:17.208739+0000","last_unstale":"2024-04-02T07:56:17.214434+0000","last_undegraded":"2024-04-02T07:56:17.214434+0000","last_fullsized":"2024-04-02T07:56:17.214434+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:37:36.640185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.202163+0000","last_change":"2024-04-02T07:56:17.210431+0000","last_active":"2024-04-02T07:56:18.202163+0000","last_peered":"2024-04-02T07:56:18.202163+0000","last_clean":"2024-04-02T07:56:18.202163+0000","last_became_active":"2024-04-02T07:56:17.209098+0000","last_became_peered":"2024-04-02T07:56:17.209098+0000","last_unstale":"2024-04-02T07:56:18.202163+0000","last_undegraded":"2024-04-02T07:56:18.202163+0000","last_fullsized":"2024-04-02T07:56:18.202163+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:49:12.293116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201104+0000","last_change":"2024-04-02T07:56:17.226280+0000","last_active":"2024-04-02T07:56:18.201104+0000","last_peered":"2024-04-02T07:56:18.201104+0000","last_clean":"2024-04-02T07:56:18.201104+0000","last_became_active":"2024-04-02T07:56:17.225114+0000","last_became_peered":"2024-04-02T07:56:17.225114+0000","last_unstale":"2024-04-02T07:56:18.201104+0000","last_undegraded":"2024-04-02T07:56:18.201104+0000","last_fullsized":"2024-04-02T07:56:18.201104+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:20.856155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.202062+0000","last_change":"2024-04-02T07:56:17.214951+0000","last_active":"2024-04-02T07:56:18.202062+0000","last_peered":"2024-04-02T07:56:18.202062+0000","last_clean":"2024-04-02T07:56:18.202062+0000","last_became_active":"2024-04-02T07:56:17.213069+0000","last_became_peered":"2024-04-02T07:56:17.213069+0000","last_unstale":"2024-04-02T07:56:18.202062+0000","last_undegraded":"2024-04-02T07:56:18.202062+0000","last_fullsized":"2024-04-02T07:56:18.202062+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:28:09.719123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751165+0000","last_change":"2024-04-02T07:56:17.219528+0000","last_active":"2024-04-02T07:56:18.751165+0000","last_peered":"2024-04-02T07:56:18.751165+0000","last_clean":"2024-04-02T07:56:18.751165+0000","last_became_active":"2024-04-02T07:56:17.216203+0000","last_became_peered":"2024-04-02T07:56:17.216203+0000","last_unstale":"2024-04-02T07:56:18.751165+0000","last_undegraded":"2024-04-02T07:56:18.751165+0000","last_fullsized":"2024-04-02T07:56:18.751165+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:57:35.607653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201755+0000","last_change":"2024-04-02T07:56:17.216412+0000","last_active":"2024-04-02T07:56:18.201755+0000","last_peered":"2024-04-02T07:56:18.201755+0000","last_clean":"2024-04-02T07:56:18.201755+0000","last_became_active":"2024-04-02T07:56:17.216122+0000","last_became_peered":"2024-04-02T07:56:17.216122+0000","last_unstale":"2024-04-02T07:56:18.201755+0000","last_undegraded":"2024-04-02T07:56:18.201755+0000","last_fullsized":"2024-04-02T07:56:18.201755+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:59:00.949515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.221052+0000","last_change":"2024-04-02T07:56:17.219286+0000","last_active":"2024-04-02T07:56:17.221052+0000","last_peered":"2024-04-02T07:56:17.221052+0000","last_clean":"2024-04-02T07:56:17.221052+0000","last_became_active":"2024-04-02T07:56:17.219016+0000","last_became_peered":"2024-04-02T07:56:17.219016+0000","last_unstale":"2024-04-02T07:56:17.221052+0000","last_undegraded":"2024-04-02T07:56:17.221052+0000","last_fullsized":"2024-04-02T07:56:17.221052+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:54:45.468728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.200717+0000","last_change":"2024-04-02T07:56:17.216205+0000","last_active":"2024-04-02T07:56:18.200717+0000","last_peered":"2024-04-02T07:56:18.200717+0000","last_clean":"2024-04-02T07:56:18.200717+0000","last_became_active":"2024-04-02T07:56:17.215546+0000","last_became_peered":"2024-04-02T07:56:17.215546+0000","last_unstale":"2024-04-02T07:56:18.200717+0000","last_undegraded":"2024-04-02T07:56:18.200717+0000","last_fullsized":"2024-04-02T07:56:18.200717+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:26:06.353668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.216622+0000","last_change":"2024-04-02T07:56:17.215236+0000","last_active":"2024-04-02T07:56:17.216622+0000","last_peered":"2024-04-02T07:56:17.216622+0000","last_clean":"2024-04-02T07:56:17.216622+0000","last_became_active":"2024-04-02T07:56:17.214457+0000","last_became_peered":"2024-04-02T07:56:17.214457+0000","last_unstale":"2024-04-02T07:56:17.216622+0000","last_undegraded":"2024-04-02T07:56:17.216622+0000","last_fullsized":"2024-04-02T07:56:17.216622+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:14:42.522106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201755+0000","last_change":"2024-04-02T07:56:17.220389+0000","last_active":"2024-04-02T07:56:18.201755+0000","last_peered":"2024-04-02T07:56:18.201755+0000","last_clean":"2024-04-02T07:56:18.201755+0000","last_became_active":"2024-04-02T07:56:17.220025+0000","last_became_peered":"2024-04-02T07:56:17.220025+0000","last_unstale":"2024-04-02T07:56:18.201755+0000","last_undegraded":"2024-04-02T07:56:18.201755+0000","last_fullsized":"2024-04-02T07:56:18.201755+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:31.191815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801432+0000","last_change":"2024-04-02T07:56:17.221197+0000","last_active":"2024-04-02T07:56:18.801432+0000","last_peered":"2024-04-02T07:56:18.801432+0000","last_clean":"2024-04-02T07:56:18.801432+0000","last_became_active":"2024-04-02T07:56:17.220796+0000","last_became_peered":"2024-04-02T07:56:17.220796+0000","last_unstale":"2024-04-02T07:56:18.801432+0000","last_undegraded":"2024-04-02T07:56:18.801432+0000","last_fullsized":"2024-04-02T07:56:18.801432+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:57:07.303322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.223346+0000","last_change":"2024-04-02T07:56:17.222232+0000","last_active":"2024-04-02T07:56:17.223346+0000","last_peered":"2024-04-02T07:56:17.223346+0000","last_clean":"2024-04-02T07:56:17.223346+0000","last_became_active":"2024-04-02T07:56:17.220361+0000","last_became_peered":"2024-04-02T07:56:17.220361+0000","last_unstale":"2024-04-02T07:56:17.223346+0000","last_undegraded":"2024-04-02T07:56:17.223346+0000","last_fullsized":"2024-04-02T07:56:17.223346+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:44:57.200725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201763+0000","last_change":"2024-04-02T07:56:17.212364+0000","last_active":"2024-04-02T07:56:18.201763+0000","last_peered":"2024-04-02T07:56:18.201763+0000","last_clean":"2024-04-02T07:56:18.201763+0000","last_became_active":"2024-04-02T07:56:17.212072+0000","last_became_peered":"2024-04-02T07:56:17.212072+0000","last_unstale":"2024-04-02T07:56:18.201763+0000","last_undegraded":"2024-04-02T07:56:18.201763+0000","last_fullsized":"2024-04-02T07:56:18.201763+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:56:11.399042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201555+0000","last_change":"2024-04-02T07:56:17.213872+0000","last_active":"2024-04-02T07:56:18.201555+0000","last_peered":"2024-04-02T07:56:18.201555+0000","last_clean":"2024-04-02T07:56:18.201555+0000","last_became_active":"2024-04-02T07:56:17.213595+0000","last_became_peered":"2024-04-02T07:56:17.213595+0000","last_unstale":"2024-04-02T07:56:18.201555+0000","last_undegraded":"2024-04-02T07:56:18.201555+0000","last_fullsized":"2024-04-02T07:56:18.201555+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:46:52.515384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.217474+0000","last_change":"2024-04-02T07:56:17.212353+0000","last_active":"2024-04-02T07:56:17.217474+0000","last_peered":"2024-04-02T07:56:17.217474+0000","last_clean":"2024-04-02T07:56:17.217474+0000","last_became_active":"2024-04-02T07:56:17.212003+0000","last_became_peered":"2024-04-02T07:56:17.212003+0000","last_unstale":"2024-04-02T07:56:17.217474+0000","last_undegraded":"2024-04-02T07:56:17.217474+0000","last_fullsized":"2024-04-02T07:56:17.217474+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:44:51.037155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.221589+0000","last_change":"2024-04-02T07:56:17.215278+0000","last_active":"2024-04-02T07:56:17.221589+0000","last_peered":"2024-04-02T07:56:17.221589+0000","last_clean":"2024-04-02T07:56:17.221589+0000","last_became_active":"2024-04-02T07:56:17.215021+0000","last_became_peered":"2024-04-02T07:56:17.215021+0000","last_unstale":"2024-04-02T07:56:17.221589+0000","last_undegraded":"2024-04-02T07:56:17.221589+0000","last_fullsized":"2024-04-02T07:56:17.221589+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:43:50.657790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801439+0000","last_change":"2024-04-02T07:56:17.223954+0000","last_active":"2024-04-02T07:56:18.801439+0000","last_peered":"2024-04-02T07:56:18.801439+0000","last_clean":"2024-04-02T07:56:18.801439+0000","last_became_active":"2024-04-02T07:56:17.223497+0000","last_became_peered":"2024-04-02T07:56:17.223497+0000","last_unstale":"2024-04-02T07:56:18.801439+0000","last_undegraded":"2024-04-02T07:56:18.801439+0000","last_fullsized":"2024-04-02T07:56:18.801439+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:36.562378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1f","version":"153'4","reported_seq":15,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.200228+0000","last_change":"2024-04-02T07:56:16.707369+0000","last_active":"2024-04-02T07:56:18.200228+0000","last_peered":"2024-04-02T07:56:18.200228+0000","last_clean":"2024-04-02T07:56:18.200228+0000","last_became_active":"2024-04-02T07:56:16.705817+0000","last_became_peered":"2024-04-02T07:56:16.705817+0000","last_unstale":"2024-04-02T07:56:18.200228+0000","last_undegraded":"2024-04-02T07:56:18.200228+0000","last_fullsized":"2024-04-02T07:56:18.200228+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:56:03.390568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.214456+0000","last_change":"2024-04-02T07:56:17.209750+0000","last_active":"2024-04-02T07:56:17.214456+0000","last_peered":"2024-04-02T07:56:17.214456+0000","last_clean":"2024-04-02T07:56:17.214456+0000","last_became_active":"2024-04-02T07:56:17.209235+0000","last_became_peered":"2024-04-02T07:56:17.209235+0000","last_unstale":"2024-04-02T07:56:17.214456+0000","last_undegraded":"2024-04-02T07:56:17.214456+0000","last_fullsized":"2024-04-02T07:56:17.214456+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:15:21.320878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1e","version":"153'1","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.232854+0000","last_change":"2024-04-02T07:56:16.714403+0000","last_active":"2024-04-02T07:56:17.232854+0000","last_peered":"2024-04-02T07:56:17.232854+0000","last_clean":"2024-04-02T07:56:17.232854+0000","last_became_active":"2024-04-02T07:56:16.714141+0000","last_became_peered":"2024-04-02T07:56:16.714141+0000","last_unstale":"2024-04-02T07:56:17.232854+0000","last_undegraded":"2024-04-02T07:56:17.232854+0000","last_fullsized":"2024-04-02T07:56:17.232854+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:31.606349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.221245+0000","last_change":"2024-04-02T07:56:17.215667+0000","last_active":"2024-04-02T07:56:17.221245+0000","last_peered":"2024-04-02T07:56:17.221245+0000","last_clean":"2024-04-02T07:56:17.221245+0000","last_became_active":"2024-04-02T07:56:17.215422+0000","last_became_peered":"2024-04-02T07:56:17.215422+0000","last_unstale":"2024-04-02T07:56:17.221245+0000","last_undegraded":"2024-04-02T07:56:17.221245+0000","last_fullsized":"2024-04-02T07:56:17.221245+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:10:08.386400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1d","version":"153'2","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.228765+0000","last_change":"2024-04-02T07:56:16.718433+0000","last_active":"2024-04-02T07:56:17.228765+0000","last_peered":"2024-04-02T07:56:17.228765+0000","last_clean":"2024-04-02T07:56:17.228765+0000","last_became_active":"2024-04-02T07:56:16.718060+0000","last_became_peered":"2024-04-02T07:56:16.718060+0000","last_unstale":"2024-04-02T07:56:17.228765+0000","last_undegraded":"2024-04-02T07:56:17.228765+0000","last_fullsized":"2024-04-02T07:56:17.228765+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:55.044510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751510+0000","last_change":"2024-04-02T07:56:17.213206+0000","last_active":"2024-04-02T07:56:18.751510+0000","last_peered":"2024-04-02T07:56:18.751510+0000","last_clean":"2024-04-02T07:56:18.751510+0000","last_became_active":"2024-04-02T07:56:17.212941+0000","last_became_peered":"2024-04-02T07:56:17.212941+0000","last_unstale":"2024-04-02T07:56:18.751510+0000","last_undegraded":"2024-04-02T07:56:18.751510+0000","last_fullsized":"2024-04-02T07:56:18.751510+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:16:04.779564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1c","version":"153'2","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201600+0000","last_change":"2024-04-02T07:56:16.715903+0000","last_active":"2024-04-02T07:56:18.201600+0000","last_peered":"2024-04-02T07:56:18.201600+0000","last_clean":"2024-04-02T07:56:18.201600+0000","last_became_active":"2024-04-02T07:56:16.715622+0000","last_became_peered":"2024-04-02T07:56:16.715622+0000","last_unstale":"2024-04-02T07:56:18.201600+0000","last_undegraded":"2024-04-02T07:56:18.201600+0000","last_fullsized":"2024-04-02T07:56:18.201600+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:43:01.550275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201363+0000","last_change":"2024-04-02T07:56:17.213727+0000","last_active":"2024-04-02T07:56:18.201363+0000","last_peered":"2024-04-02T07:56:18.201363+0000","last_clean":"2024-04-02T07:56:18.201363+0000","last_became_active":"2024-04-02T07:56:17.213446+0000","last_became_peered":"2024-04-02T07:56:17.213446+0000","last_unstale":"2024-04-02T07:56:18.201363+0000","last_undegraded":"2024-04-02T07:56:18.201363+0000","last_fullsized":"2024-04-02T07:56:18.201363+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:06:06.229996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1b","version":"153'1","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201715+0000","last_change":"2024-04-02T07:56:16.712283+0000","last_active":"2024-04-02T07:56:18.201715+0000","last_peered":"2024-04-02T07:56:18.201715+0000","last_clean":"2024-04-02T07:56:18.201715+0000","last_became_active":"2024-04-02T07:56:16.711971+0000","last_became_peered":"2024-04-02T07:56:16.711971+0000","last_unstale":"2024-04-02T07:56:18.201715+0000","last_undegraded":"2024-04-02T07:56:18.201715+0000","last_fullsized":"2024-04-02T07:56:18.201715+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:19:49.151260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751282+0000","last_change":"2024-04-02T07:56:17.219746+0000","last_active":"2024-04-02T07:56:18.751282+0000","last_peered":"2024-04-02T07:56:18.751282+0000","last_clean":"2024-04-02T07:56:18.751282+0000","last_became_active":"2024-04-02T07:56:17.218551+0000","last_became_peered":"2024-04-02T07:56:17.218551+0000","last_unstale":"2024-04-02T07:56:18.751282+0000","last_undegraded":"2024-04-02T07:56:18.751282+0000","last_fullsized":"2024-04-02T07:56:18.751282+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:54:11.802163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"153'1","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201923+0000","last_change":"2024-04-02T07:56:16.721064+0000","last_active":"2024-04-02T07:56:18.201923+0000","last_peered":"2024-04-02T07:56:18.201923+0000","last_clean":"2024-04-02T07:56:18.201923+0000","last_became_active":"2024-04-02T07:56:16.720572+0000","last_became_peered":"2024-04-02T07:56:16.720572+0000","last_unstale":"2024-04-02T07:56:18.201923+0000","last_undegraded":"2024-04-02T07:56:18.201923+0000","last_fullsized":"2024-04-02T07:56:18.201923+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:31:36.461451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751088+0000","last_change":"2024-04-02T07:56:17.222686+0000","last_active":"2024-04-02T07:56:18.751088+0000","last_peered":"2024-04-02T07:56:18.751088+0000","last_clean":"2024-04-02T07:56:18.751088+0000","last_became_active":"2024-04-02T07:56:17.221832+0000","last_became_peered":"2024-04-02T07:56:17.221832+0000","last_unstale":"2024-04-02T07:56:18.751088+0000","last_undegraded":"2024-04-02T07:56:18.751088+0000","last_fullsized":"2024-04-02T07:56:18.751088+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:46:52.515384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.197127+0000","last_change":"2024-04-02T07:56:16.718802+0000","last_active":"2024-04-02T07:56:17.197127+0000","last_peered":"2024-04-02T07:56:17.197127+0000","last_clean":"2024-04-02T07:56:17.197127+0000","last_became_active":"2024-04-02T07:56:16.718497+0000","last_became_peered":"2024-04-02T07:56:16.718497+0000","last_unstale":"2024-04-02T07:56:17.197127+0000","last_undegraded":"2024-04-02T07:56:17.197127+0000","last_fullsized":"2024-04-02T07:56:17.197127+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:42:33.236077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.200928+0000","last_change":"2024-04-02T07:56:17.224386+0000","last_active":"2024-04-02T07:56:18.200928+0000","last_peered":"2024-04-02T07:56:18.200928+0000","last_clean":"2024-04-02T07:56:18.200928+0000","last_became_active":"2024-04-02T07:56:17.224093+0000","last_became_peered":"2024-04-02T07:56:17.224093+0000","last_unstale":"2024-04-02T07:56:18.200928+0000","last_undegraded":"2024-04-02T07:56:18.200928+0000","last_fullsized":"2024-04-02T07:56:18.200928+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:57:07.303322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.200963+0000","last_change":"2024-04-02T07:56:16.716294+0000","last_active":"2024-04-02T07:56:18.200963+0000","last_peered":"2024-04-02T07:56:18.200963+0000","last_clean":"2024-04-02T07:56:18.200963+0000","last_became_active":"2024-04-02T07:56:16.714501+0000","last_became_peered":"2024-04-02T07:56:16.714501+0000","last_unstale":"2024-04-02T07:56:18.200963+0000","last_undegraded":"2024-04-02T07:56:18.200963+0000","last_fullsized":"2024-04-02T07:56:18.200963+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:25:27.947017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201807+0000","last_change":"2024-04-02T07:56:17.227002+0000","last_active":"2024-04-02T07:56:18.201807+0000","last_peered":"2024-04-02T07:56:18.201807+0000","last_clean":"2024-04-02T07:56:18.201807+0000","last_became_active":"2024-04-02T07:56:17.226716+0000","last_became_peered":"2024-04-02T07:56:17.226716+0000","last_unstale":"2024-04-02T07:56:18.201807+0000","last_undegraded":"2024-04-02T07:56:18.201807+0000","last_fullsized":"2024-04-02T07:56:18.201807+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:07:25.934705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751549+0000","last_change":"2024-04-02T07:56:16.714383+0000","last_active":"2024-04-02T07:56:18.751549+0000","last_peered":"2024-04-02T07:56:18.751549+0000","last_clean":"2024-04-02T07:56:18.751549+0000","last_became_active":"2024-04-02T07:56:16.713929+0000","last_became_peered":"2024-04-02T07:56:16.713929+0000","last_unstale":"2024-04-02T07:56:18.751549+0000","last_undegraded":"2024-04-02T07:56:18.751549+0000","last_fullsized":"2024-04-02T07:56:18.751549+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:55.044510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.200005+0000","last_change":"2024-04-02T07:56:17.212537+0000","last_active":"2024-04-02T07:56:18.200005+0000","last_peered":"2024-04-02T07:56:18.200005+0000","last_clean":"2024-04-02T07:56:18.200005+0000","last_became_active":"2024-04-02T07:56:17.212291+0000","last_became_peered":"2024-04-02T07:56:17.212291+0000","last_unstale":"2024-04-02T07:56:18.200005+0000","last_undegraded":"2024-04-02T07:56:18.200005+0000","last_fullsized":"2024-04-02T07:56:18.200005+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:44:57.200725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.202310+0000","last_change":"2024-04-02T07:56:16.719851+0000","last_active":"2024-04-02T07:56:18.202310+0000","last_peered":"2024-04-02T07:56:18.202310+0000","last_clean":"2024-04-02T07:56:18.202310+0000","last_became_active":"2024-04-02T07:56:16.718004+0000","last_became_peered":"2024-04-02T07:56:16.718004+0000","last_unstale":"2024-04-02T07:56:18.202310+0000","last_undegraded":"2024-04-02T07:56:18.202310+0000","last_fullsized":"2024-04-02T07:56:18.202310+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:27.254350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751383+0000","last_change":"2024-04-02T07:56:17.220141+0000","last_active":"2024-04-02T07:56:18.751383+0000","last_peered":"2024-04-02T07:56:18.751383+0000","last_clean":"2024-04-02T07:56:18.751383+0000","last_became_active":"2024-04-02T07:56:17.219894+0000","last_became_peered":"2024-04-02T07:56:17.219894+0000","last_unstale":"2024-04-02T07:56:18.751383+0000","last_undegraded":"2024-04-02T07:56:18.751383+0000","last_fullsized":"2024-04-02T07:56:18.751383+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:52:31.704949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.15","version":"153'3","reported_seq":14,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.199770+0000","last_change":"2024-04-02T07:56:16.705712+0000","last_active":"2024-04-02T07:56:18.199770+0000","last_peered":"2024-04-02T07:56:18.199770+0000","last_clean":"2024-04-02T07:56:18.199770+0000","last_became_active":"2024-04-02T07:56:16.705288+0000","last_became_peered":"2024-04-02T07:56:16.705288+0000","last_unstale":"2024-04-02T07:56:18.199770+0000","last_undegraded":"2024-04-02T07:56:18.199770+0000","last_fullsized":"2024-04-02T07:56:18.199770+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:11:37.420301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.230517+0000","last_change":"2024-04-02T07:56:17.229449+0000","last_active":"2024-04-02T07:56:17.230517+0000","last_peered":"2024-04-02T07:56:17.230517+0000","last_clean":"2024-04-02T07:56:17.230517+0000","last_became_active":"2024-04-02T07:56:17.229159+0000","last_became_peered":"2024-04-02T07:56:17.229159+0000","last_unstale":"2024-04-02T07:56:17.230517+0000","last_undegraded":"2024-04-02T07:56:17.230517+0000","last_fullsized":"2024-04-02T07:56:17.230517+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:39:22.649036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.14","version":"153'6","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.235307+0000","last_change":"2024-04-02T07:56:16.711712+0000","last_active":"2024-04-02T07:56:17.235307+0000","last_peered":"2024-04-02T07:56:17.235307+0000","last_clean":"2024-04-02T07:56:17.235307+0000","last_became_active":"2024-04-02T07:56:16.711361+0000","last_became_peered":"2024-04-02T07:56:16.711361+0000","last_unstale":"2024-04-02T07:56:17.235307+0000","last_undegraded":"2024-04-02T07:56:17.235307+0000","last_fullsized":"2024-04-02T07:56:17.235307+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:57:40.665983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801232+0000","last_change":"2024-04-02T07:56:17.217638+0000","last_active":"2024-04-02T07:56:18.801232+0000","last_peered":"2024-04-02T07:56:18.801232+0000","last_clean":"2024-04-02T07:56:18.801232+0000","last_became_active":"2024-04-02T07:56:17.217380+0000","last_became_peered":"2024-04-02T07:56:17.217380+0000","last_unstale":"2024-04-02T07:56:18.801232+0000","last_undegraded":"2024-04-02T07:56:18.801232+0000","last_fullsized":"2024-04-02T07:56:18.801232+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:06:58.818408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801137+0000","last_change":"2024-04-02T07:56:16.718975+0000","last_active":"2024-04-02T07:56:18.801137+0000","last_peered":"2024-04-02T07:56:18.801137+0000","last_clean":"2024-04-02T07:56:18.801137+0000","last_became_active":"2024-04-02T07:56:16.718267+0000","last_became_peered":"2024-04-02T07:56:16.718267+0000","last_unstale":"2024-04-02T07:56:18.801137+0000","last_undegraded":"2024-04-02T07:56:18.801137+0000","last_fullsized":"2024-04-02T07:56:18.801137+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:06.225724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801020+0000","last_change":"2024-04-02T07:56:17.219206+0000","last_active":"2024-04-02T07:56:18.801020+0000","last_peered":"2024-04-02T07:56:18.801020+0000","last_clean":"2024-04-02T07:56:18.801020+0000","last_became_active":"2024-04-02T07:56:17.218814+0000","last_became_peered":"2024-04-02T07:56:17.218814+0000","last_unstale":"2024-04-02T07:56:18.801020+0000","last_undegraded":"2024-04-02T07:56:18.801020+0000","last_fullsized":"2024-04-02T07:56:18.801020+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:56.051002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.202146+0000","last_change":"2024-04-02T07:56:16.714035+0000","last_active":"2024-04-02T07:56:17.202146+0000","last_peered":"2024-04-02T07:56:17.202146+0000","last_clean":"2024-04-02T07:56:17.202146+0000","last_became_active":"2024-04-02T07:56:16.713683+0000","last_became_peered":"2024-04-02T07:56:16.713683+0000","last_unstale":"2024-04-02T07:56:17.202146+0000","last_undegraded":"2024-04-02T07:56:17.202146+0000","last_fullsized":"2024-04-02T07:56:17.202146+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:06:57.811916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751037+0000","last_change":"2024-04-02T07:56:17.215732+0000","last_active":"2024-04-02T07:56:18.751037+0000","last_peered":"2024-04-02T07:56:18.751037+0000","last_clean":"2024-04-02T07:56:18.751037+0000","last_became_active":"2024-04-02T07:56:17.215450+0000","last_became_peered":"2024-04-02T07:56:17.215450+0000","last_unstale":"2024-04-02T07:56:18.751037+0000","last_undegraded":"2024-04-02T07:56:18.751037+0000","last_fullsized":"2024-04-02T07:56:18.751037+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:40:31.418887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.10","version":"153'8","reported_seq":19,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751074+0000","last_change":"2024-04-02T07:56:16.714558+0000","last_active":"2024-04-02T07:56:18.751074+0000","last_peered":"2024-04-02T07:56:18.751074+0000","last_clean":"2024-04-02T07:56:18.751074+0000","last_became_active":"2024-04-02T07:56:16.714276+0000","last_became_peered":"2024-04-02T07:56:16.714276+0000","last_unstale":"2024-04-02T07:56:18.751074+0000","last_undegraded":"2024-04-02T07:56:18.751074+0000","last_fullsized":"2024-04-02T07:56:18.751074+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:58:37.517431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.199776+0000","last_change":"2024-04-02T07:56:17.218056+0000","last_active":"2024-04-02T07:56:18.199776+0000","last_peered":"2024-04-02T07:56:18.199776+0000","last_clean":"2024-04-02T07:56:18.199776+0000","last_became_active":"2024-04-02T07:56:17.214883+0000","last_became_peered":"2024-04-02T07:56:17.214883+0000","last_unstale":"2024-04-02T07:56:18.199776+0000","last_undegraded":"2024-04-02T07:56:18.199776+0000","last_fullsized":"2024-04-02T07:56:18.199776+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:54:45.468728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.f","version":"153'1","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751591+0000","last_change":"2024-04-02T07:56:16.714205+0000","last_active":"2024-04-02T07:56:18.751591+0000","last_peered":"2024-04-02T07:56:18.751591+0000","last_clean":"2024-04-02T07:56:18.751591+0000","last_became_active":"2024-04-02T07:56:16.713576+0000","last_became_peered":"2024-04-02T07:56:16.713576+0000","last_unstale":"2024-04-02T07:56:18.751591+0000","last_undegraded":"2024-04-02T07:56:18.751591+0000","last_fullsized":"2024-04-02T07:56:18.751591+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:26:50.852997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.200088+0000","last_change":"2024-04-02T07:56:17.213711+0000","last_active":"2024-04-02T07:56:18.200088+0000","last_peered":"2024-04-02T07:56:18.200088+0000","last_clean":"2024-04-02T07:56:18.200088+0000","last_became_active":"2024-04-02T07:56:17.211615+0000","last_became_peered":"2024-04-02T07:56:17.211615+0000","last_unstale":"2024-04-02T07:56:18.200088+0000","last_undegraded":"2024-04-02T07:56:18.200088+0000","last_fullsized":"2024-04-02T07:56:18.200088+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:25.681411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"153'2","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.230887+0000","last_change":"2024-04-02T07:56:16.711626+0000","last_active":"2024-04-02T07:56:17.230887+0000","last_peered":"2024-04-02T07:56:17.230887+0000","last_clean":"2024-04-02T07:56:17.230887+0000","last_became_active":"2024-04-02T07:56:16.711311+0000","last_became_peered":"2024-04-02T07:56:16.711311+0000","last_unstale":"2024-04-02T07:56:17.230887+0000","last_undegraded":"2024-04-02T07:56:17.230887+0000","last_fullsized":"2024-04-02T07:56:17.230887+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:16.738817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201489+0000","last_change":"2024-04-02T07:56:17.226039+0000","last_active":"2024-04-02T07:56:18.201489+0000","last_peered":"2024-04-02T07:56:18.201489+0000","last_clean":"2024-04-02T07:56:18.201489+0000","last_became_active":"2024-04-02T07:56:17.225641+0000","last_became_peered":"2024-04-02T07:56:17.225641+0000","last_unstale":"2024-04-02T07:56:18.201489+0000","last_undegraded":"2024-04-02T07:56:18.201489+0000","last_fullsized":"2024-04-02T07:56:18.201489+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:07:38.125893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.d","version":"153'1","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.231298+0000","last_change":"2024-04-02T07:56:16.720235+0000","last_active":"2024-04-02T07:56:17.231298+0000","last_peered":"2024-04-02T07:56:17.231298+0000","last_clean":"2024-04-02T07:56:17.231298+0000","last_became_active":"2024-04-02T07:56:16.719994+0000","last_became_peered":"2024-04-02T07:56:16.719994+0000","last_unstale":"2024-04-02T07:56:17.231298+0000","last_undegraded":"2024-04-02T07:56:17.231298+0000","last_fullsized":"2024-04-02T07:56:17.231298+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:09:58.680838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201244+0000","last_change":"2024-04-02T07:56:17.216379+0000","last_active":"2024-04-02T07:56:18.201244+0000","last_peered":"2024-04-02T07:56:18.201244+0000","last_clean":"2024-04-02T07:56:18.201244+0000","last_became_active":"2024-04-02T07:56:17.216079+0000","last_became_peered":"2024-04-02T07:56:17.216079+0000","last_unstale":"2024-04-02T07:56:18.201244+0000","last_undegraded":"2024-04-02T07:56:18.201244+0000","last_fullsized":"2024-04-02T07:56:18.201244+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:22:34.417939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.199681+0000","last_change":"2024-04-02T07:56:16.716547+0000","last_active":"2024-04-02T07:56:18.199681+0000","last_peered":"2024-04-02T07:56:18.199681+0000","last_clean":"2024-04-02T07:56:18.199681+0000","last_became_active":"2024-04-02T07:56:16.716322+0000","last_became_peered":"2024-04-02T07:56:16.716322+0000","last_unstale":"2024-04-02T07:56:18.199681+0000","last_undegraded":"2024-04-02T07:56:18.199681+0000","last_fullsized":"2024-04-02T07:56:18.199681+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:08:34.246745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201599+0000","last_change":"2024-04-02T07:56:17.228292+0000","last_active":"2024-04-02T07:56:18.201599+0000","last_peered":"2024-04-02T07:56:18.201599+0000","last_clean":"2024-04-02T07:56:18.201599+0000","last_became_active":"2024-04-02T07:56:17.226480+0000","last_became_peered":"2024-04-02T07:56:17.226480+0000","last_unstale":"2024-04-02T07:56:18.201599+0000","last_undegraded":"2024-04-02T07:56:18.201599+0000","last_fullsized":"2024-04-02T07:56:18.201599+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:01:41.298193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.b","version":"153'2","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.230064+0000","last_change":"2024-04-02T07:56:16.718277+0000","last_active":"2024-04-02T07:56:17.230064+0000","last_peered":"2024-04-02T07:56:17.230064+0000","last_clean":"2024-04-02T07:56:17.230064+0000","last_became_active":"2024-04-02T07:56:16.717167+0000","last_became_peered":"2024-04-02T07:56:16.717167+0000","last_unstale":"2024-04-02T07:56:17.230064+0000","last_undegraded":"2024-04-02T07:56:17.230064+0000","last_fullsized":"2024-04-02T07:56:17.230064+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:56.568169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.216922+0000","last_change":"2024-04-02T07:56:17.211469+0000","last_active":"2024-04-02T07:56:17.216922+0000","last_peered":"2024-04-02T07:56:17.216922+0000","last_clean":"2024-04-02T07:56:17.216922+0000","last_became_active":"2024-04-02T07:56:17.211165+0000","last_became_peered":"2024-04-02T07:56:17.211165+0000","last_unstale":"2024-04-02T07:56:17.216922+0000","last_undegraded":"2024-04-02T07:56:17.216922+0000","last_fullsized":"2024-04-02T07:56:17.216922+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:58:19.065313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201985+0000","last_change":"2024-04-02T07:56:16.716277+0000","last_active":"2024-04-02T07:56:18.201985+0000","last_peered":"2024-04-02T07:56:18.201985+0000","last_clean":"2024-04-02T07:56:18.201985+0000","last_became_active":"2024-04-02T07:56:16.715806+0000","last_became_peered":"2024-04-02T07:56:16.715806+0000","last_unstale":"2024-04-02T07:56:18.201985+0000","last_undegraded":"2024-04-02T07:56:18.201985+0000","last_fullsized":"2024-04-02T07:56:18.201985+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:53:23.460581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801259+0000","last_change":"2024-04-02T07:56:17.212227+0000","last_active":"2024-04-02T07:56:18.801259+0000","last_peered":"2024-04-02T07:56:18.801259+0000","last_clean":"2024-04-02T07:56:18.801259+0000","last_became_active":"2024-04-02T07:56:17.211525+0000","last_became_peered":"2024-04-02T07:56:17.211525+0000","last_unstale":"2024-04-02T07:56:18.801259+0000","last_undegraded":"2024-04-02T07:56:18.801259+0000","last_fullsized":"2024-04-02T07:56:18.801259+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:36:20.278308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801048+0000","last_change":"2024-04-02T07:56:16.716753+0000","last_active":"2024-04-02T07:56:18.801048+0000","last_peered":"2024-04-02T07:56:18.801048+0000","last_clean":"2024-04-02T07:56:18.801048+0000","last_became_active":"2024-04-02T07:56:16.714406+0000","last_became_peered":"2024-04-02T07:56:16.714406+0000","last_unstale":"2024-04-02T07:56:18.801048+0000","last_undegraded":"2024-04-02T07:56:18.801048+0000","last_fullsized":"2024-04-02T07:56:18.801048+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:30:15.057311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751172+0000","last_change":"2024-04-02T07:56:17.219225+0000","last_active":"2024-04-02T07:56:18.751172+0000","last_peered":"2024-04-02T07:56:18.751172+0000","last_clean":"2024-04-02T07:56:18.751172+0000","last_became_active":"2024-04-02T07:56:17.218978+0000","last_became_peered":"2024-04-02T07:56:17.218978+0000","last_unstale":"2024-04-02T07:56:18.751172+0000","last_undegraded":"2024-04-02T07:56:18.751172+0000","last_fullsized":"2024-04-02T07:56:18.751172+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:07:25.934705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.202466+0000","last_change":"2024-04-02T07:56:16.711503+0000","last_active":"2024-04-02T07:56:17.202466+0000","last_peered":"2024-04-02T07:56:17.202466+0000","last_clean":"2024-04-02T07:56:17.202466+0000","last_became_active":"2024-04-02T07:56:16.711132+0000","last_became_peered":"2024-04-02T07:56:16.711132+0000","last_unstale":"2024-04-02T07:56:17.202466+0000","last_undegraded":"2024-04-02T07:56:17.202466+0000","last_fullsized":"2024-04-02T07:56:17.202466+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:58:37.517431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.223430+0000","last_change":"2024-04-02T07:56:17.222334+0000","last_active":"2024-04-02T07:56:17.223430+0000","last_peered":"2024-04-02T07:56:17.223430+0000","last_clean":"2024-04-02T07:56:17.223430+0000","last_became_active":"2024-04-02T07:56:17.220969+0000","last_became_peered":"2024-04-02T07:56:17.220969+0000","last_unstale":"2024-04-02T07:56:17.223430+0000","last_undegraded":"2024-04-02T07:56:17.223430+0000","last_fullsized":"2024-04-02T07:56:17.223430+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:29:03.752049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.7","version":"153'2","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.202351+0000","last_change":"2024-04-02T07:56:16.719121+0000","last_active":"2024-04-02T07:56:18.202351+0000","last_peered":"2024-04-02T07:56:18.202351+0000","last_clean":"2024-04-02T07:56:18.202351+0000","last_became_active":"2024-04-02T07:56:16.718846+0000","last_became_peered":"2024-04-02T07:56:16.718846+0000","last_unstale":"2024-04-02T07:56:18.202351+0000","last_undegraded":"2024-04-02T07:56:18.202351+0000","last_fullsized":"2024-04-02T07:56:18.202351+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:24.674919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201380+0000","last_change":"2024-04-02T07:56:17.225390+0000","last_active":"2024-04-02T07:56:18.201380+0000","last_peered":"2024-04-02T07:56:18.201380+0000","last_clean":"2024-04-02T07:56:18.201380+0000","last_became_active":"2024-04-02T07:56:17.225094+0000","last_became_peered":"2024-04-02T07:56:17.225094+0000","last_unstale":"2024-04-02T07:56:18.201380+0000","last_undegraded":"2024-04-02T07:56:18.201380+0000","last_fullsized":"2024-04-02T07:56:18.201380+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:56.051002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.199837+0000","last_change":"2024-04-02T07:56:16.718650+0000","last_active":"2024-04-02T07:56:18.199837+0000","last_peered":"2024-04-02T07:56:18.199837+0000","last_clean":"2024-04-02T07:56:18.199837+0000","last_became_active":"2024-04-02T07:56:16.718344+0000","last_became_peered":"2024-04-02T07:56:16.718344+0000","last_unstale":"2024-04-02T07:56:18.199837+0000","last_undegraded":"2024-04-02T07:56:18.199837+0000","last_fullsized":"2024-04-02T07:56:18.199837+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:47.014632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801352+0000","last_change":"2024-04-02T07:56:17.221890+0000","last_active":"2024-04-02T07:56:18.801352+0000","last_peered":"2024-04-02T07:56:18.801352+0000","last_clean":"2024-04-02T07:56:18.801352+0000","last_became_active":"2024-04-02T07:56:17.220807+0000","last_became_peered":"2024-04-02T07:56:17.220807+0000","last_unstale":"2024-04-02T07:56:18.801352+0000","last_undegraded":"2024-04-02T07:56:18.801352+0000","last_fullsized":"2024-04-02T07:56:18.801352+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:58:38.523923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.202157+0000","last_change":"2024-04-02T07:56:16.718812+0000","last_active":"2024-04-02T07:56:18.202157+0000","last_peered":"2024-04-02T07:56:18.202157+0000","last_clean":"2024-04-02T07:56:18.202157+0000","last_became_active":"2024-04-02T07:56:16.718529+0000","last_became_peered":"2024-04-02T07:56:16.718529+0000","last_unstale":"2024-04-02T07:56:18.202157+0000","last_undegraded":"2024-04-02T07:56:18.202157+0000","last_fullsized":"2024-04-02T07:56:18.202157+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:28.917028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.212608+0000","last_change":"2024-04-02T07:56:17.206998+0000","last_active":"2024-04-02T07:56:17.212608+0000","last_peered":"2024-04-02T07:56:17.212608+0000","last_clean":"2024-04-02T07:56:17.212608+0000","last_became_active":"2024-04-02T07:56:17.206732+0000","last_became_peered":"2024-04-02T07:56:17.206732+0000","last_unstale":"2024-04-02T07:56:17.212608+0000","last_undegraded":"2024-04-02T07:56:17.212608+0000","last_fullsized":"2024-04-02T07:56:17.212608+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:16:29.631645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801329+0000","last_change":"2024-04-02T07:56:16.717912+0000","last_active":"2024-04-02T07:56:18.801329+0000","last_peered":"2024-04-02T07:56:18.801329+0000","last_clean":"2024-04-02T07:56:18.801329+0000","last_became_active":"2024-04-02T07:56:16.717252+0000","last_became_peered":"2024-04-02T07:56:16.717252+0000","last_unstale":"2024-04-02T07:56:18.801329+0000","last_undegraded":"2024-04-02T07:56:18.801329+0000","last_fullsized":"2024-04-02T07:56:18.801329+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:28:08.712631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751145+0000","last_change":"2024-04-02T07:56:17.218043+0000","last_active":"2024-04-02T07:56:18.751145+0000","last_peered":"2024-04-02T07:56:18.751145+0000","last_clean":"2024-04-02T07:56:18.751145+0000","last_became_active":"2024-04-02T07:56:17.217410+0000","last_became_peered":"2024-04-02T07:56:17.217410+0000","last_unstale":"2024-04-02T07:56:18.751145+0000","last_undegraded":"2024-04-02T07:56:18.751145+0000","last_fullsized":"2024-04-02T07:56:18.751145+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:22:34.417939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201831+0000","last_change":"2024-04-02T07:56:16.711117+0000","last_active":"2024-04-02T07:56:18.201831+0000","last_peered":"2024-04-02T07:56:18.201831+0000","last_clean":"2024-04-02T07:56:18.201831+0000","last_became_active":"2024-04-02T07:56:16.710454+0000","last_became_peered":"2024-04-02T07:56:16.710454+0000","last_unstale":"2024-04-02T07:56:18.201831+0000","last_undegraded":"2024-04-02T07:56:18.201831+0000","last_fullsized":"2024-04-02T07:56:18.201831+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:07:30.218500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751282+0000","last_change":"2024-04-02T07:56:17.224241+0000","last_active":"2024-04-02T07:56:18.751282+0000","last_peered":"2024-04-02T07:56:18.751282+0000","last_clean":"2024-04-02T07:56:18.751282+0000","last_became_active":"2024-04-02T07:56:17.215798+0000","last_became_peered":"2024-04-02T07:56:17.215798+0000","last_unstale":"2024-04-02T07:56:18.751282+0000","last_undegraded":"2024-04-02T07:56:18.751282+0000","last_fullsized":"2024-04-02T07:56:18.751282+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:01:29.623806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751223+0000","last_change":"2024-04-02T07:56:16.714562+0000","last_active":"2024-04-02T07:56:18.751223+0000","last_peered":"2024-04-02T07:56:18.751223+0000","last_clean":"2024-04-02T07:56:18.751223+0000","last_became_active":"2024-04-02T07:56:16.714300+0000","last_became_peered":"2024-04-02T07:56:16.714300+0000","last_unstale":"2024-04-02T07:56:18.751223+0000","last_undegraded":"2024-04-02T07:56:18.751223+0000","last_fullsized":"2024-04-02T07:56:18.751223+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:35.555887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.201389+0000","last_change":"2024-04-02T07:56:17.216422+0000","last_active":"2024-04-02T07:56:18.201389+0000","last_peered":"2024-04-02T07:56:18.201389+0000","last_clean":"2024-04-02T07:56:18.201389+0000","last_became_active":"2024-04-02T07:56:17.215923+0000","last_became_peered":"2024-04-02T07:56:17.215923+0000","last_unstale":"2024-04-02T07:56:18.201389+0000","last_undegraded":"2024-04-02T07:56:18.201389+0000","last_fullsized":"2024-04-02T07:56:18.201389+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:40:10.320276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.202217+0000","last_change":"2024-04-02T07:56:16.714825+0000","last_active":"2024-04-02T07:56:17.202217+0000","last_peered":"2024-04-02T07:56:17.202217+0000","last_clean":"2024-04-02T07:56:17.202217+0000","last_became_active":"2024-04-02T07:56:16.714539+0000","last_became_peered":"2024-04-02T07:56:16.714539+0000","last_unstale":"2024-04-02T07:56:17.202217+0000","last_undegraded":"2024-04-02T07:56:17.202217+0000","last_fullsized":"2024-04-02T07:56:17.202217+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:36:19.271816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":10,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801064+0000","last_change":"2024-04-02T07:56:17.213134+0000","last_active":"2024-04-02T07:56:18.801064+0000","last_peered":"2024-04-02T07:56:18.801064+0000","last_clean":"2024-04-02T07:56:18.801064+0000","last_became_active":"2024-04-02T07:56:17.212837+0000","last_became_peered":"2024-04-02T07:56:17.212837+0000","last_unstale":"2024-04-02T07:56:18.801064+0000","last_undegraded":"2024-04-02T07:56:18.801064+0000","last_fullsized":"2024-04-02T07:56:18.801064+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:17.745309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.801260+0000","last_change":"2024-04-02T07:56:16.713716+0000","last_active":"2024-04-02T07:56:18.801260+0000","last_peered":"2024-04-02T07:56:18.801260+0000","last_clean":"2024-04-02T07:56:18.801260+0000","last_became_active":"2024-04-02T07:56:16.713404+0000","last_became_peered":"2024-04-02T07:56:16.713404+0000","last_unstale":"2024-04-02T07:56:18.801260+0000","last_undegraded":"2024-04-02T07:56:18.801260+0000","last_fullsized":"2024-04-02T07:56:18.801260+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:13:39.132006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-02T07:56:17.220862+0000","last_change":"2024-04-02T07:56:17.216374+0000","last_active":"2024-04-02T07:56:17.220862+0000","last_peered":"2024-04-02T07:56:17.220862+0000","last_clean":"2024-04-02T07:56:17.220862+0000","last_became_active":"2024-04-02T07:56:17.216117+0000","last_became_peered":"2024-04-02T07:56:17.216117+0000","last_unstale":"2024-04-02T07:56:17.220862+0000","last_undegraded":"2024-04-02T07:56:17.220862+0000","last_fullsized":"2024-04-02T07:56:17.220862+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:16.689329+0000","last_clean_scrub_stamp":"2024-04-02T07:56:16.689329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:13:15.311857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.13","version":"153'2","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-04-02T07:56:18.751356+0000","last_change":"2024-04-02T07:56:16.715391+0000","last_active":"2024-04-02T07:56:18.751356+0000","last_peered":"2024-04-02T07:56:18.751356+0000","last_clean":"2024-04-02T07:56:18.751356+0000","last_became_active":"2024-04-02T07:56:16.715062+0000","last_became_peered":"2024-04-02T07:56:16.715062+0000","last_unstale":"2024-04-02T07:56:18.751356+0000","last_undegraded":"2024-04-02T07:56:18.751356+0000","last_fullsized":"2024-04-02T07:56:18.751356+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:56:15.682837+0000","last_clean_scrub_stamp":"2024-04-02T07:56:15.682837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:57:06.296830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"143'249","reported_seq":759,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-04-02T07:56:16.697590+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T07:56:16.697590+0000","last_peered":"2024-04-02T07:56:16.697590+0000","last_clean":"2024-04-02T07:56:16.697590+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T07:56:16.697590+0000","last_undegraded":"2024-04-02T07:56:16.697590+0000","last_fullsized":"2024-04-02T07:56:16.697590+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575084,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1921024,"data_stored":953007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63200000000000001,"15min":0.66500000000000004},"min":{"1min":0.38600000000000001,"5min":0.34699999999999998,"15min":0.316},"max":{"1min":0.80700000000000005,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.67900000000000005},"min":{"1min":0.41499999999999998,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.68600000000000005}]},{"osd":1,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65100000000000002,"15min":0.67500000000000004},"min":{"1min":0.48699999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88100000000000001,"5min":1.097,"15min":1.097},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63800000000000001,"15min":0.66500000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.34699999999999998},"max":{"1min":0.78000000000000003,"5min":0.98299999999999998,"15min":1.2190000000000001},"last":0.91900000000000004}]},{"osd":2,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66200000000000003,"15min":0.69899999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.34000000000000002},"max":{"1min":0.88700000000000001,"5min":0.98999999999999999,"15min":1.127},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.72499999999999998,"15min":0.75900000000000001},"min":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95299999999999996,"5min":1.044,"15min":1.1259999999999999},"last":0.88200000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.69899999999999995},"min":{"1min":0.45800000000000002,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.94199999999999995,"5min":0.97799999999999998,"15min":1.026},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68200000000000005,"15min":0.71299999999999997},"min":{"1min":0.52700000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.83399999999999996,"5min":1.071,"15min":1.196},"last":0.83599999999999997}]},{"osd":4,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58799999999999997},"min":{"1min":0.377,"5min":0.35799999999999998,"15min":0.32000000000000001},"max":{"1min":0.85899999999999999,"5min":1.0229999999999999,"15min":1.0389999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68700000000000006,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.004,"5min":1.016,"15min":1.0669999999999999},"last":0.89700000000000002}]},{"osd":5,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.64400000000000002,"15min":0.65000000000000002},"min":{"1min":0.45500000000000002,"5min":0.42999999999999999,"15min":0.37},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.167},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.022,"5min":1.343,"15min":1.4219999999999999},"last":0.95099999999999996}]},{"osd":6,"last update":"Tue Apr 2 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.36199999999999999},"max":{"1min":0.92000000000000004,"5min":1.0580000000000001,"15min":2.5369999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.66600000000000004,"15min":0.66000000000000003},"min":{"1min":0.48399999999999999,"5min":0.41299999999999998,"15min":0.39600000000000002},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.94199999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575082,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37084,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":1904640,"data_stored":946297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.67000000000000004,"15min":0.70099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.33400000000000002},"max":{"1min":0.89000000000000001,"5min":0.95899999999999996,"15min":0.98099999999999998},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66700000000000004,"15min":0.70399999999999996},"min":{"1min":0.46200000000000002,"5min":0.40500000000000003,"15min":0.38300000000000001},"max":{"1min":0.88600000000000001,"5min":0.94299999999999995,"15min":1.0249999999999999},"last":0.83799999999999997}]},{"osd":1,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67000000000000004,"15min":0.69199999999999995},"min":{"1min":0.505,"5min":0.433,"15min":0.41799999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69899999999999995,"15min":0.71499999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":0.95799999999999996,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.67200000000000004}]},{"osd":2,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68000000000000005,"15min":0.70999999999999996},"min":{"1min":0.46500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68899999999999995,"15min":0.70699999999999996},"min":{"1min":0.52600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.70899999999999996}]},{"osd":3,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71199999999999997,"15min":0.72899999999999998},"min":{"1min":0.53200000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.73899999999999999,"15min":0.745},"min":{"1min":0.52800000000000002,"5min":0.46600000000000003,"15min":0.45300000000000001},"max":{"1min":3.339,"5min":3.339,"15min":3.339},"last":0.68799999999999994}]},{"osd":4,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.625,"15min":0.60699999999999998},"min":{"1min":0.39100000000000001,"5min":0.377,"15min":0.372},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64100000000000001,"15min":0.63300000000000001},"min":{"1min":0.44500000000000001,"5min":0.41499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.59899999999999998}]},{"osd":5,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.629,"15min":0.625},"min":{"1min":0.47599999999999998,"5min":0.39000000000000001,"15min":0.35799999999999998},"max":{"1min":0.89200000000000002,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.69999999999999996,"15min":0.68600000000000005},"min":{"1min":0.45500000000000002,"5min":0.40899999999999997,"15min":0.40300000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.72299999999999998}]},{"osd":7,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.69999999999999996,"15min":0.68999999999999995},"min":{"1min":0.501,"5min":0.46999999999999997,"15min":0.40200000000000002},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.52000000000000002,"5min":0.42799999999999999,"15min":0.42099999999999999},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.81899999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575083,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59820,"kb_used_data":2456,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312020,"statfs":{"total":96636764160,"available":96575508480,"internally_reserved":0,"allocated":2514944,"data_stored":1538251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.54400000000000004,"15min":0.47599999999999998},"min":{"1min":0.30199999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.57599999999999996,"5min":10.052,"15min":10.052},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.59399999999999997,"15min":0.53300000000000003},"min":{"1min":0.30399999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61799999999999999,"5min":9.0250000000000004,"15min":9.0250000000000004},"last":0.33900000000000002}]},{"osd":2,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.58799999999999997,"15min":0.53000000000000003},"min":{"1min":0.36099999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.55500000000000005,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.64100000000000001,"15min":0.59899999999999998},"min":{"1min":0.45500000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.68899999999999995,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.63300000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64500000000000002,"5min":2.609,"15min":2.609},"last":0.436},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66800000000000004,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.51400000000000001}]},{"osd":4,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.75800000000000001,"15min":0.77000000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86599999999999999,"5min":2.21,"15min":2.21},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.81000000000000005,"15min":0.78500000000000003},"min":{"1min":0.47899999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.96299999999999997,"5min":6.3419999999999996,"15min":6.3419999999999996},"last":0.879}]},{"osd":5,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.82499999999999996,"15min":0.78900000000000003},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.41999999999999998},"max":{"1min":0.96299999999999997,"5min":6.5529999999999999,"15min":6.5529999999999999},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.78200000000000003,"15min":0.79000000000000004},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.48799999999999999},"max":{"1min":1.0189999999999999,"5min":2.544,"15min":2.544},"last":0.83999999999999997}]},{"osd":6,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":1.849,"15min":1.1519999999999999},"min":{"1min":0.55200000000000005,"5min":0.53200000000000003,"15min":0.47699999999999998},"max":{"1min":0.95399999999999996,"5min":78.903000000000006,"15min":78.903000000000006},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":1.8260000000000001,"15min":1.139},"min":{"1min":0.52400000000000002,"5min":0.49199999999999999,"15min":0.38100000000000001},"max":{"1min":0.89600000000000002,"5min":78.837000000000003,"15min":78.837000000000003},"last":0.79500000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78500000000000003,"15min":0.80000000000000004},"min":{"1min":0.55400000000000005,"5min":0.52800000000000002,"15min":0.48799999999999999},"max":{"1min":0.93200000000000005,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.79100000000000004,"15min":0.81799999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.97699999999999998,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":0.995}]}]},{"osd":0,"up_from":13,"seq":55834575083,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1921024,"data_stored":946898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.42199999999999999,"15min":0.437},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67800000000000005,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.45900000000000002,"15min":0.48199999999999998},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.629,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.377}]},{"osd":2,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47299999999999998,"15min":0.48599999999999999},"min":{"1min":0.32200000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":1.597,"15min":1.597},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51100000000000001,"15min":0.52700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72699999999999998,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.438}]},{"osd":3,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.50700000000000001,"15min":0.52300000000000002},"min":{"1min":0.38700000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81799999999999995},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.60199999999999998},"min":{"1min":0.40300000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.78500000000000003,"5min":1.492,"15min":1.492},"last":0.52100000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.753,"15min":0.77600000000000002},"min":{"1min":0.51500000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0409999999999999,"5min":1.1579999999999999,"15min":1.181},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76300000000000001,"15min":0.77300000000000002},"min":{"1min":0.54300000000000004,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":1.097,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.78100000000000003}]},{"osd":5,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.76600000000000001,"15min":0.79100000000000004},"min":{"1min":0.56599999999999995,"5min":0.48699999999999999,"15min":0.47299999999999998},"max":{"1min":0.998,"5min":1.361,"15min":2.1869999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.76900000000000002,"15min":0.79500000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.026,"5min":1.546,"15min":2.6230000000000002},"last":0.71299999999999997}]},{"osd":6,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.73099999999999998,"15min":0.747},"min":{"1min":0.47999999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.93700000000000006,"5min":1.52,"15min":1.52},"last":0.628},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.71799999999999997,"15min":0.745},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.41299999999999998},"max":{"1min":1.026,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.79500000000000004}]},{"osd":7,"last update":"Tue Apr 2 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.78800000000000003,"15min":0.81200000000000006},"min":{"1min":0.53600000000000003,"5min":0.51600000000000001,"15min":0.505},"max":{"1min":1.002,"5min":1.139,"15min":2.3959999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.76200000000000001,"15min":0.80100000000000005},"min":{"1min":0.50900000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0960000000000001,"5min":1.3959999999999999,"15min":2.145},"last":0.81799999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575082,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37100,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1925120,"data_stored":953645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.29399999999999998,"5min":0.26700000000000002,"15min":0.26400000000000001},"max":{"1min":0.70499999999999996,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46700000000000003,"15min":0.47299999999999998},"min":{"1min":0.33600000000000002,"5min":0.29099999999999998,"15min":0.26200000000000001},"max":{"1min":0.58899999999999997,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.41099999999999998}]},{"osd":1,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.33800000000000002,"5min":0.32700000000000001,"15min":0.30499999999999999},"max":{"1min":0.58699999999999997,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.5,"15min":0.51700000000000002},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":0.58499999999999996,"5min":0.996,"15min":0.996},"last":0.45500000000000002}]},{"osd":3,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.316},"max":{"1min":0.82399999999999995,"5min":1.472,"15min":1.472},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.59199999999999997,"15min":0.60299999999999998},"min":{"1min":0.42899999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.70999999999999996,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.57899999999999996}]},{"osd":4,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.72699999999999998,"15min":0.76000000000000001},"min":{"1min":0.52300000000000002,"5min":0.5,"15min":0.47199999999999998},"max":{"1min":1.046,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.62},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.73199999999999998,"15min":0.76200000000000001},"min":{"1min":0.56000000000000005,"5min":0.55000000000000004,"15min":0.48499999999999999},"max":{"1min":1.1040000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.753}]},{"osd":5,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.72199999999999998,"15min":0.751},"min":{"1min":0.54300000000000004,"5min":0.51000000000000001,"15min":0.46200000000000002},"max":{"1min":0.91500000000000004,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.76900000000000002},"min":{"1min":0.61499999999999999,"5min":0.55600000000000005,"15min":0.50900000000000001},"max":{"1min":0.95199999999999996,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.70499999999999996}]},{"osd":6,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.747,"15min":0.77800000000000002},"min":{"1min":0.58399999999999996,"5min":0.501,"15min":0.442},"max":{"1min":1.079,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.77100000000000002},"min":{"1min":0.56499999999999995,"5min":0.502,"15min":0.502},"max":{"1min":0.871,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.64600000000000002}]},{"osd":7,"last update":"Tue Apr 2 07:55:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.78400000000000003},"min":{"1min":0.55900000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0249999999999999,"5min":1.278,"15min":1.278},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73299999999999998,"15min":0.77900000000000003},"min":{"1min":0.504,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.99099999999999999,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.67800000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59224,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312616,"statfs":{"total":96636764160,"available":96576118784,"internally_reserved":0,"allocated":1908736,"data_stored":946367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.437,"15min":0.42899999999999999},"min":{"1min":0.314,"5min":0.26500000000000001,"15min":0.23499999999999999},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.44600000000000001,"15min":0.46000000000000002},"min":{"1min":0.308,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58599999999999997,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.38600000000000001}]},{"osd":1,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46999999999999997,"15min":0.48699999999999999},"min":{"1min":0.33900000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69599999999999995,"5min":0.78000000000000003,"15min":1.2809999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56899999999999995,"15min":0.56299999999999994},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71099999999999997,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":0.47799999999999998}]},{"osd":2,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.60799999999999998},"min":{"1min":0.42599999999999999,"5min":0.40899999999999997,"15min":0.38700000000000001},"max":{"1min":0.88900000000000001,"5min":2.214,"15min":2.214},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58499999999999996,"15min":0.60099999999999998},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.46000000000000002}]},{"osd":4,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.72999999999999998,"15min":0.73299999999999998},"min":{"1min":0.53600000000000003,"5min":0.47699999999999998,"15min":0.38400000000000001},"max":{"1min":0.86499999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.39600000000000002},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.63200000000000001}]},{"osd":5,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72799999999999998,"15min":0.751},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.40899999999999997},"max":{"1min":0.84399999999999997,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.74399999999999999,"15min":0.749},"min":{"1min":0.51100000000000001,"5min":0.49199999999999999,"15min":0.40200000000000002},"max":{"1min":0.90000000000000002,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.76000000000000001}]},{"osd":6,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76900000000000002,"15min":0.76500000000000001},"min":{"1min":0.55300000000000005,"5min":0.51100000000000001,"15min":0.45800000000000002},"max":{"1min":1.4359999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73899999999999999,"15min":0.76800000000000002},"min":{"1min":0.51700000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.90600000000000003,"5min":2.286,"15min":2.286},"last":0.64400000000000002}]},{"osd":7,"last update":"Tue Apr 2 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.746,"15min":0.76700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.439},"max":{"1min":0.82799999999999996,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77400000000000002,"15min":0.78500000000000003},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.48999999999999999},"max":{"1min":1.3200000000000001,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.71199999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575082,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41188,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330652,"statfs":{"total":96636764160,"available":96594587648,"internally_reserved":0,"allocated":1916928,"data_stored":952973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.63800000000000001,"15min":0.66300000000000003},"min":{"1min":0.45300000000000001,"5min":0.41699999999999998,"15min":0.38100000000000001},"max":{"1min":0.82599999999999996,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.61499999999999999,"15min":0.66000000000000003},"min":{"1min":0.432,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.77700000000000002,"5min":0.94999999999999996,"15min":0.97599999999999998},"last":0.68999999999999995}]},{"osd":1,"last update":"Tue Apr 2 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.69299999999999995,"15min":0.71599999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.438},"max":{"1min":0.876,"5min":1.014,"15min":1.0349999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70399999999999996,"15min":0.72999999999999998},"min":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.36799999999999999},"max":{"1min":0.89000000000000001,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.81899999999999995}]},{"osd":2,"last update":"Tue Apr 2 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.66700000000000004,"15min":0.68600000000000005},"min":{"1min":0.44700000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.78800000000000003,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.68700000000000006,"15min":0.70699999999999996},"min":{"1min":0.48799999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.92200000000000004,"5min":1.026,"15min":1.026},"last":0.91100000000000003}]},{"osd":3,"last update":"Tue Apr 2 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.46800000000000003},"max":{"1min":0.89800000000000002,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.68999999999999995,"15min":0.71899999999999997},"min":{"1min":0.54600000000000004,"5min":0.47299999999999998,"15min":0.41499999999999998},"max":{"1min":0.85399999999999998,"5min":1.046,"15min":1.046},"last":0.73299999999999998}]},{"osd":5,"last update":"Tue Apr 2 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.621,"15min":0.58699999999999997},"min":{"1min":0.41799999999999998,"5min":0.39400000000000002,"15min":0.29599999999999999},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61099999999999999,"15min":0.59099999999999997},"min":{"1min":0.46700000000000003,"5min":0.41699999999999998,"15min":0.33700000000000002},"max":{"1min":0.95699999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.63900000000000001}]},{"osd":6,"last update":"Tue Apr 2 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.64000000000000001,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.374},"max":{"1min":0.95199999999999996,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.622},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.68600000000000005,"15min":0.65800000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.88600000000000001}]},{"osd":7,"last update":"Tue Apr 2 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.68899999999999995,"15min":0.67900000000000005},"min":{"1min":0.5,"5min":0.39600000000000002,"15min":0.38300000000000001},"max":{"1min":0.88800000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.75900000000000001,"15min":0.73999999999999999},"min":{"1min":0.56399999999999995,"5min":0.52200000000000002,"15min":0.434},"max":{"1min":1.004,"5min":1.095,"15min":1.095},"last":0.76800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575085,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55712,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316128,"statfs":{"total":96636764160,"available":96579715072,"internally_reserved":0,"allocated":2506752,"data_stored":1536388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63700000000000001,"15min":0.67500000000000004},"min":{"1min":0.41999999999999998,"5min":0.35299999999999998,"15min":0.29899999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.99199999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.64800000000000002,"15min":0.68100000000000005},"min":{"1min":0.51000000000000001,"5min":0.40500000000000003,"15min":0.378},"max":{"1min":0.85599999999999998,"5min":1.0249999999999999,"15min":1.4450000000000001},"last":0.69099999999999995}]},{"osd":1,"last update":"Tue Apr 2 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.63600000000000001,"15min":0.67800000000000005},"min":{"1min":0.45300000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.81399999999999995,"5min":0.88100000000000001,"15min":1.131},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.66200000000000003,"15min":0.68700000000000006},"min":{"1min":0.47099999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.78300000000000003,"5min":0.96399999999999997,"15min":1.5149999999999999},"last":0.73699999999999999}]},{"osd":2,"last update":"Tue Apr 2 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.66000000000000003,"15min":0.69999999999999996},"min":{"1min":0.437,"5min":0.42799999999999999,"15min":0.41799999999999998},"max":{"1min":0.84699999999999998,"5min":0.90700000000000003,"15min":1.5449999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67000000000000004,"15min":0.69999999999999996},"min":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.372},"max":{"1min":0.879,"5min":0.89700000000000002,"15min":1.3959999999999999},"last":0.75600000000000001}]},{"osd":3,"last update":"Tue Apr 2 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.72999999999999998},"min":{"1min":0.49199999999999999,"5min":0.436,"15min":0.436},"max":{"1min":0.89500000000000002,"5min":1.0649999999999999,"15min":1.341},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.73199999999999998,"15min":0.75900000000000001},"min":{"1min":0.53700000000000003,"5min":0.48299999999999998,"15min":0.44900000000000001},"max":{"1min":0.92000000000000004,"5min":0.97499999999999998,"15min":1.484},"last":0.79100000000000004}]},{"osd":4,"last update":"Tue Apr 2 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84399999999999997,"5min":0.84599999999999997,"15min":1.3109999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64400000000000002,"15min":0.64100000000000001},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.35599999999999998},"max":{"1min":0.81899999999999995,"5min":0.97999999999999998,"15min":1.504},"last":0.77400000000000002}]},{"osd":6,"last update":"Tue Apr 2 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.65200000000000002,"15min":0.64300000000000002},"min":{"1min":0.42899999999999999,"5min":0.41599999999999998,"15min":0.39900000000000002},"max":{"1min":0.871,"5min":1.004,"15min":1.212},"last":0.63},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.67800000000000005,"15min":0.67400000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.88500000000000001,"5min":0.92800000000000005,"15min":1.774},"last":0.65800000000000003}]},{"osd":7,"last update":"Tue Apr 2 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.71699999999999997,"15min":0.70499999999999996},"min":{"1min":0.53900000000000003,"5min":0.45000000000000001,"15min":0.38900000000000001},"max":{"1min":0.80700000000000005,"5min":1.1020000000000001,"15min":1.6890000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.497,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":0.93700000000000006,"5min":1.139,"15min":1.5309999999999999},"last":0.81599999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T07:56:22.046 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T07:56:22.046 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T07:56:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.280+0000 7fe2d9a00700 1 Processor -- start 2024-04-02T07:56:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.281+0000 7fe2d9a00700 1 -- start start 2024-04-02T07:56:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.282+0000 7fe2d9a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe2d415d9d0 0x7fe2d40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.282+0000 7fe2d9a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d40d16a0 0x7fe2d40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.282+0000 7fe2d9a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d4161420 con 0x7fe2d40d2370 2024-04-02T07:56:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.282+0000 7fe2d9a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2d40d2820 con 0x7fe2d415d9d0 2024-04-02T07:56:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.282+0000 7fe2d9a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d4161700 con 0x7fe2d40d16a0 2024-04-02T07:56:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.282+0000 7fe2d2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe2d415d9d0 0x7fe2d40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.282+0000 7fe2d2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe2d415d9d0 0x7fe2d40d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42336/0 (socket says 172.21.15.89:42336) 2024-04-02T07:56:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.282+0000 7fe2d2a00700 1 -- 172.21.15.89:0/979192610 learned_addr learned my addr 172.21.15.89:0/979192610 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.282+0000 7fe2d2a00700 1 -- 172.21.15.89:0/979192610 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d40d16a0 msgr2=0x7fe2d40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.282+0000 7fe2cbe00700 1 --2- 172.21.15.89:0/979192610 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d40d16a0 0x7fe2d40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.282+0000 7fe2d2a00700 1 --2- 172.21.15.89:0/979192610 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d40d16a0 0x7fe2d40d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.282+0000 7fe2d2a00700 1 -- 172.21.15.89:0/979192610 >> v1:172.21.15.113:6789/0 conn(0x7fe2d40d2370 legacy=0x7fe2d415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.282+0000 7fe2d2a00700 1 -- 172.21.15.89:0/979192610 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d41619e0 con 0x7fe2d415d9d0 2024-04-02T07:56:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.282+0000 7fe2cbe00700 1 --2- 172.21.15.89:0/979192610 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d40d16a0 0x7fe2d40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.283+0000 7fe2d2a00700 1 --2- 172.21.15.89:0/979192610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415d9d0 0x7fe2d40d3c30 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fe2c40098f0 tx=0x7fe2c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9cf3322bd13004 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.283+0000 7fe2d2000700 1 -- 172.21.15.89:0/979192610 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4004230 con 0x7fe2d415d9d0 2024-04-02T07:56:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.283+0000 7fe2d2000700 1 -- 172.21.15.89:0/979192610 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c4004390 con 0x7fe2d415d9d0 2024-04-02T07:56:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.283+0000 7fe2d2000700 1 -- 172.21.15.89:0/979192610 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c40036a0 con 0x7fe2d415d9d0 2024-04-02T07:56:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.283+0000 7fe2d9a00700 1 -- 172.21.15.89:0/979192610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415d9d0 msgr2=0x7fe2d40d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.283+0000 7fe2d9a00700 1 --2- 172.21.15.89:0/979192610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415d9d0 0x7fe2d40d3c30 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fe2c40098f0 tx=0x7fe2c4009c00 comp rx=0 tx=0).stop 2024-04-02T07:56:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.283+0000 7fe2d9a00700 1 -- 172.21.15.89:0/979192610 shutdown_connections 2024-04-02T07:56:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.283+0000 7fe2d9a00700 1 --2- 172.21.15.89:0/979192610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415d9d0 0x7fe2d40d3c30 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.283+0000 7fe2d9a00700 1 --2- 172.21.15.89:0/979192610 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d40d16a0 0x7fe2d40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.284+0000 7fe2d9a00700 1 -- 172.21.15.89:0/979192610 >> 172.21.15.89:0/979192610 conn(0x7fe2d408aff0 msgr2=0x7fe2d40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:22.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.284+0000 7fe2d9a00700 1 -- 172.21.15.89:0/979192610 shutdown_connections 2024-04-02T07:56:22.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.284+0000 7fe2d9a00700 1 -- 172.21.15.89:0/979192610 wait complete. 2024-04-02T07:56:22.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.285+0000 7fe2d9a00700 1 Processor -- start 2024-04-02T07:56:22.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.285+0000 7fe2d9a00700 1 -- start start 2024-04-02T07:56:22.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.286+0000 7fe2d9a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2d40d16a0 0x7fe2d415aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:22.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.286+0000 7fe2d9a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d40d2370 0x7fe2d415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:22.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.286+0000 7fe2d9a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415d9d0 0x7fe2d41588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:22.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.286+0000 7fe2d9a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d40b6250 con 0x7fe2d415d9d0 2024-04-02T07:56:22.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.286+0000 7fe2d9a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d415efc0 con 0x7fe2d40d16a0 2024-04-02T07:56:22.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.286+0000 7fe2d9a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d415e720 con 0x7fe2d40d2370 2024-04-02T07:56:22.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2d3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415d9d0 0x7fe2d41588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:22.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2d3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415d9d0 0x7fe2d41588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42340/0 (socket says 172.21.15.89:42340) 2024-04-02T07:56:22.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2d3400700 1 -- 172.21.15.89:0/2177531878 learned_addr learned my addr 172.21.15.89:0/2177531878 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:22.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2d2a00700 1 --2- 172.21.15.89:0/2177531878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2d40d16a0 0x7fe2d415aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:22.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2cbe00700 1 --2- 172.21.15.89:0/2177531878 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d40d2370 0x7fe2d415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:22.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2d3400700 1 -- 172.21.15.89:0/2177531878 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d40d2370 msgr2=0x7fe2d415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:22.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2d3400700 1 --2- 172.21.15.89:0/2177531878 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d40d2370 0x7fe2d415ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2d3400700 1 -- 172.21.15.89:0/2177531878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2d40d16a0 msgr2=0x7fe2d415aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:22.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2d3400700 1 --2- 172.21.15.89:0/2177531878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2d40d16a0 0x7fe2d415aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2d3400700 1 -- 172.21.15.89:0/2177531878 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c4009550 con 0x7fe2d415d9d0 2024-04-02T07:56:22.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2d3400700 1 --2- 172.21.15.89:0/2177531878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415d9d0 0x7fe2d41588a0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00cc40 tx=0x7fe2cc00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:22.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2caa00700 1 -- 172.21.15.89:0/2177531878 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc00aa80 con 0x7fe2d415d9d0 2024-04-02T07:56:22.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2caa00700 1 -- 172.21.15.89:0/2177531878 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2cc00abe0 con 0x7fe2d415d9d0 2024-04-02T07:56:22.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2d9a00700 1 -- 172.21.15.89:0/2177531878 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d415ea00 con 0x7fe2d415d9d0 2024-04-02T07:56:22.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.287+0000 7fe2caa00700 1 -- 172.21.15.89:0/2177531878 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc012420 con 0x7fe2d415d9d0 2024-04-02T07:56:22.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.288+0000 7fe2d9a00700 1 -- 172.21.15.89:0/2177531878 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d415b400 con 0x7fe2d415d9d0 2024-04-02T07:56:22.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.289+0000 7fe2d9a00700 1 -- 172.21.15.89:0/2177531878 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d40d1ca0 con 0x7fe2d415d9d0 2024-04-02T07:56:22.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.289+0000 7fe2caa00700 1 -- 172.21.15.89:0/2177531878 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe2cc012580 con 0x7fe2d415d9d0 2024-04-02T07:56:22.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.290+0000 7fe2caa00700 1 --2- 172.21.15.89:0/2177531878 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe2b006adb0 0x7fe2b006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:22.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.290+0000 7fe2caa00700 1 -- 172.21.15.89:0/2177531878 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fe2cc089cc0 con 0x7fe2d415d9d0 2024-04-02T07:56:22.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.291+0000 7fe2d2a00700 1 --2- 172.21.15.89:0/2177531878 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe2b006adb0 0x7fe2b006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:22.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.291+0000 7fe2d2a00700 1 --2- 172.21.15.89:0/2177531878 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe2b006adb0 0x7fe2b006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009130 tx=0x7fe2c4008030 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:22.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.295+0000 7fe2caa00700 1 -- 172.21.15.89:0/2177531878 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe2cc090080 con 0x7fe2d415d9d0 2024-04-02T07:56:22.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.575+0000 7fe2d9a00700 1 -- 172.21.15.89:0/2177531878 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe2d40b6020 con 0x7fe2d415d9d0 2024-04-02T07:56:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.579+0000 7fe2caa00700 1 -- 172.21.15.89:0/2177531878 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v24) v1 ==== 190+0+0 (secure 0 0 0) 0x7fe2d40b6020 con 0x7fe2d415d9d0 2024-04-02T07:56:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.581+0000 7fe2d9a00700 1 -- 172.21.15.89:0/2177531878 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe2b006adb0 msgr2=0x7fe2b006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.581+0000 7fe2d9a00700 1 --2- 172.21.15.89:0/2177531878 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe2b006adb0 0x7fe2b006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009130 tx=0x7fe2c4008030 comp rx=0 tx=0).stop 2024-04-02T07:56:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.581+0000 7fe2d9a00700 1 -- 172.21.15.89:0/2177531878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415d9d0 msgr2=0x7fe2d41588a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.581+0000 7fe2d9a00700 1 --2- 172.21.15.89:0/2177531878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415d9d0 0x7fe2d41588a0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00cc40 tx=0x7fe2cc00cf50 comp rx=0 tx=0).stop 2024-04-02T07:56:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.581+0000 7fe2d9a00700 1 -- 172.21.15.89:0/2177531878 shutdown_connections 2024-04-02T07:56:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.581+0000 7fe2d9a00700 1 --2- 172.21.15.89:0/2177531878 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe2b006adb0 0x7fe2b006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.581+0000 7fe2d9a00700 1 --2- 172.21.15.89:0/2177531878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2d40d16a0 0x7fe2d415aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.581+0000 7fe2d9a00700 1 --2- 172.21.15.89:0/2177531878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d415d9d0 0x7fe2d41588a0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.581+0000 7fe2d9a00700 1 --2- 172.21.15.89:0/2177531878 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2d40d2370 0x7fe2d415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.581+0000 7fe2d9a00700 1 -- 172.21.15.89:0/2177531878 >> 172.21.15.89:0/2177531878 conn(0x7fe2d408aff0 msgr2=0x7fe2d40b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.582+0000 7fe2d9a00700 1 -- 172.21.15.89:0/2177531878 shutdown_connections 2024-04-02T07:56:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.582+0000 7fe2d9a00700 1 -- 172.21.15.89:0/2177531878 wait complete. 2024-04-02T07:56:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T07:56:22.601 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T07:56:22.687 INFO:tasks.ceph.mds.c.smithi089.stderr:2024-04-02T07:56:22.686+0000 7f91bc800700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T07:56:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.790+0000 7fe1e4200700 1 Processor -- start 2024-04-02T07:56:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.791+0000 7fe1e4200700 1 -- start start 2024-04-02T07:56:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.792+0000 7fe1e4200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc0ceec0 0x7fe1dc1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.792+0000 7fe1e4200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe1dc161800 0x7fe1dc0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.792+0000 7fe1e4200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc0d30d0 con 0x7fe1dc15d640 2024-04-02T07:56:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.792+0000 7fe1e4200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1dc161c90 con 0x7fe1dc161800 2024-04-02T07:56:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.792+0000 7fe1e4200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0b6840 con 0x7fe1dc0ceec0 2024-04-02T07:56:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.792+0000 7fe1e2200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe1dc161800 0x7fe1dc0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.792+0000 7fe1e1800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe1dc15d640 0x7fe1dc0d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58276/0 (socket says 172.21.15.89:58276) 2024-04-02T07:56:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.792+0000 7fe1e1800700 1 -- 172.21.15.89:0/652187365 learned_addr learned my addr 172.21.15.89:0/652187365 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.792+0000 7fe1e0e00700 1 --2- 172.21.15.89:0/652187365 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc0ceec0 0x7fe1dc1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.792+0000 7fe1e2200700 1 -- 172.21.15.89:0/652187365 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc0ceec0 msgr2=0x7fe1dc1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.792+0000 7fe1e2200700 1 --2- 172.21.15.89:0/652187365 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc0ceec0 0x7fe1dc1612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.792+0000 7fe1dbe00700 1 -- 172.21.15.89:0/652187365 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145188106 0 0) 0x7fe1dc0d30d0 con 0x7fe1dc15d640 2024-04-02T07:56:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.792+0000 7fe1e2200700 1 -- 172.21.15.89:0/652187365 >> v1:172.21.15.89:6789/0 conn(0x7fe1dc15d640 legacy=0x7fe1dc0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.792+0000 7fe1e2200700 1 -- 172.21.15.89:0/652187365 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc0d2b30 con 0x7fe1dc161800 2024-04-02T07:56:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.793+0000 7fe1e2200700 1 --2- 172.21.15.89:0/652187365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc161800 0x7fe1dc0b6cd0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4009d90 tx=0x7fe1d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea75a677fa37983 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.793+0000 7fe1dbe00700 1 -- 172.21.15.89:0/652187365 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d400e070 con 0x7fe1dc161800 2024-04-02T07:56:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.793+0000 7fe1dbe00700 1 -- 172.21.15.89:0/652187365 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d400bbd0 con 0x7fe1dc161800 2024-04-02T07:56:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.793+0000 7fe1dbe00700 1 -- 172.21.15.89:0/652187365 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4012430 con 0x7fe1dc161800 2024-04-02T07:56:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.793+0000 7fe1e4200700 1 -- 172.21.15.89:0/652187365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc161800 msgr2=0x7fe1dc0b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.793+0000 7fe1e4200700 1 --2- 172.21.15.89:0/652187365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc161800 0x7fe1dc0b6cd0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4009d90 tx=0x7fe1d400b330 comp rx=0 tx=0).stop 2024-04-02T07:56:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.794+0000 7fe1e4200700 1 -- 172.21.15.89:0/652187365 shutdown_connections 2024-04-02T07:56:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.794+0000 7fe1e4200700 1 --2- 172.21.15.89:0/652187365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc161800 0x7fe1dc0b6cd0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.794+0000 7fe1e4200700 1 --2- 172.21.15.89:0/652187365 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc0ceec0 0x7fe1dc1612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.794+0000 7fe1e4200700 1 -- 172.21.15.89:0/652187365 >> 172.21.15.89:0/652187365 conn(0x7fe1dc08aff0 msgr2=0x7fe1dc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.794+0000 7fe1e4200700 1 -- 172.21.15.89:0/652187365 shutdown_connections 2024-04-02T07:56:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.794+0000 7fe1e4200700 1 -- 172.21.15.89:0/652187365 wait complete. 2024-04-02T07:56:22.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.795+0000 7fe1e4200700 1 Processor -- start 2024-04-02T07:56:22.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.796+0000 7fe1e4200700 1 -- start start 2024-04-02T07:56:22.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.796+0000 7fe1e4200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc0ceec0 0x7fe1dc14c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:22.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.796+0000 7fe1e4200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1dc15d640 0x7fe1dc1f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:22.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.796+0000 7fe1e4200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc161800 0x7fe1dc1f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1e4200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc1f4770 con 0x7fe1dc0ceec0 2024-04-02T07:56:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1e4200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc1f3e60 con 0x7fe1dc15d640 2024-04-02T07:56:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1e4200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0c98c0 con 0x7fe1dc161800 2024-04-02T07:56:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1e1800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc0ceec0 0x7fe1dc14c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1e1800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc0ceec0 0x7fe1dc14c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42354/0 (socket says 172.21.15.89:42354) 2024-04-02T07:56:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1e1800700 1 -- 172.21.15.89:0/3186608530 learned_addr learned my addr 172.21.15.89:0/3186608530 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1e2200700 1 --2- 172.21.15.89:0/3186608530 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc161800 0x7fe1dc1f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1e0e00700 1 --2- 172.21.15.89:0/3186608530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1dc15d640 0x7fe1dc1f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1e1800700 1 -- 172.21.15.89:0/3186608530 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc161800 msgr2=0x7fe1dc1f32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1e1800700 1 --2- 172.21.15.89:0/3186608530 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc161800 0x7fe1dc1f32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1e1800700 1 -- 172.21.15.89:0/3186608530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1dc15d640 msgr2=0x7fe1dc1f2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1e1800700 1 --2- 172.21.15.89:0/3186608530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1dc15d640 0x7fe1dc1f2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1e1800700 1 -- 172.21.15.89:0/3186608530 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d4009a70 con 0x7fe1dc0ceec0 2024-04-02T07:56:22.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1e1800700 1 --2- 172.21.15.89:0/3186608530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc0ceec0 0x7fe1dc14c910 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00011e0 tx=0x7fe1d000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:22.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.797+0000 7fe1d9600700 1 -- 172.21.15.89:0/3186608530 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0004c80 con 0x7fe1dc0ceec0 2024-04-02T07:56:22.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.798+0000 7fe1d9600700 1 -- 172.21.15.89:0/3186608530 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d0004e00 con 0x7fe1dc0ceec0 2024-04-02T07:56:22.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.798+0000 7fe1e4200700 1 -- 172.21.15.89:0/3186608530 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc1f4120 con 0x7fe1dc0ceec0 2024-04-02T07:56:22.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.798+0000 7fe1d9600700 1 -- 172.21.15.89:0/3186608530 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0007310 con 0x7fe1dc0ceec0 2024-04-02T07:56:22.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.798+0000 7fe1e4200700 1 -- 172.21.15.89:0/3186608530 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc14ce50 con 0x7fe1dc0ceec0 2024-04-02T07:56:22.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.799+0000 7fe1e4200700 1 -- 172.21.15.89:0/3186608530 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc067720 con 0x7fe1dc0ceec0 2024-04-02T07:56:22.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.800+0000 7fe1d9600700 1 -- 172.21.15.89:0/3186608530 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe1d0007470 con 0x7fe1dc0ceec0 2024-04-02T07:56:22.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.800+0000 7fe1d9600700 1 --2- 172.21.15.89:0/3186608530 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe1bc06adb0 0x7fe1bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:22.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.801+0000 7fe1d9600700 1 -- 172.21.15.89:0/3186608530 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fe1d00893b0 con 0x7fe1dc0ceec0 2024-04-02T07:56:22.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.801+0000 7fe1e0e00700 1 --2- 172.21.15.89:0/3186608530 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe1bc06adb0 0x7fe1bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:22.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.801+0000 7fe1e0e00700 1 --2- 172.21.15.89:0/3186608530 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe1bc06adb0 0x7fe1bc06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc15b000 tx=0x7fe1c8008040 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:22.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:22.805+0000 7fe1d9600700 1 -- 172.21.15.89:0/3186608530 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe1d00542f0 con 0x7fe1dc0ceec0 2024-04-02T07:56:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.052+0000 7fe1e4200700 1 -- 172.21.15.89:0/3186608530 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1dc0b5c70 con 0x7fe1dc0ceec0 2024-04-02T07:56:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.053+0000 7fe1d9600700 1 -- 172.21.15.89:0/3186608530 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fe1d0057910 con 0x7fe1dc0ceec0 2024-04-02T07:56:23.054 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:23.055 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7445,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/4112144997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":4112144997},{"type":"v1","addr":"172.21.15.113:6834","nonce":4112144997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":8416,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2047642384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2047642384},{"type":"v1","addr":"172.21.15.113:6835","nonce":2047642384}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":9030,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2201184833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2201184833},{"type":"v1","addr":"172.21.15.89:6835","nonce":2201184833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:56:17.192894+0000","modified":"2024-04-02T07:56:20.200137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9033},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9033":{"gid":9033,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3861363328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3861363328},{"type":"v1","addr":"172.21.15.89:6837","nonce":3861363328}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-02T07:56:23.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.055+0000 7fe1e4200700 1 -- 172.21.15.89:0/3186608530 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe1bc06adb0 msgr2=0x7fe1bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.055+0000 7fe1e4200700 1 --2- 172.21.15.89:0/3186608530 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe1bc06adb0 0x7fe1bc06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc15b000 tx=0x7fe1c8008040 comp rx=0 tx=0).stop 2024-04-02T07:56:23.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.055+0000 7fe1e4200700 1 -- 172.21.15.89:0/3186608530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc0ceec0 msgr2=0x7fe1dc14c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.055+0000 7fe1e4200700 1 --2- 172.21.15.89:0/3186608530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc0ceec0 0x7fe1dc14c910 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00011e0 tx=0x7fe1d000cae0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.056+0000 7fe1e4200700 1 -- 172.21.15.89:0/3186608530 shutdown_connections 2024-04-02T07:56:23.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.056+0000 7fe1e4200700 1 --2- 172.21.15.89:0/3186608530 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe1bc06adb0 0x7fe1bc06d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.056+0000 7fe1e4200700 1 --2- 172.21.15.89:0/3186608530 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1dc161800 0x7fe1dc1f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.056+0000 7fe1e4200700 1 --2- 172.21.15.89:0/3186608530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1dc0ceec0 0x7fe1dc14c910 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.056+0000 7fe1e4200700 1 --2- 172.21.15.89:0/3186608530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1dc15d640 0x7fe1dc1f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.056+0000 7fe1e4200700 1 -- 172.21.15.89:0/3186608530 >> 172.21.15.89:0/3186608530 conn(0x7fe1dc08aff0 msgr2=0x7fe1dc15b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:23.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.056+0000 7fe1e4200700 1 -- 172.21.15.89:0/3186608530 shutdown_connections 2024-04-02T07:56:23.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.056+0000 7fe1e4200700 1 -- 172.21.15.89:0/3186608530 wait complete. 2024-04-02T07:56:23.058 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 73 2024-04-02T07:56:23.478 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T07:56:17.192894+0000', 'modified': '2024-04-02T07:56:20.200137+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9033}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9033': {'gid': 9033, 'name': 'c', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6837/3861363328', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6836', 'nonce': 3861363328}, {'type': 'v1', 'addr': '172.21.15.89:6837', 'nonce': 3861363328}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T07:56:23.478 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T07:56:23.478 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-02T07:56:23.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.670+0000 7f7143a00700 1 Processor -- start 2024-04-02T07:56:23.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.670+0000 7f7143a00700 1 -- start start 2024-04-02T07:56:23.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.671+0000 7f7143a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f713c15d9d0 0x7f713c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.671+0000 7f7143a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f713c0d2540 0x7f713c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.672+0000 7f7143a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c161420 con 0x7f713c0d16a0 2024-04-02T07:56:23.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.672+0000 7f7143a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f713c15edc0 con 0x7f713c15d9d0 2024-04-02T07:56:23.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.672+0000 7f7143a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f713c161700 con 0x7f713c0d2540 2024-04-02T07:56:23.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.672+0000 7f7141000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f713c15d9d0 0x7f713c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.672+0000 7f7141000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f713c15d9d0 0x7f713c0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42370/0 (socket says 172.21.15.89:42370) 2024-04-02T07:56:23.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.672+0000 7f7141000700 1 -- 172.21.15.89:0/309820549 learned_addr learned my addr 172.21.15.89:0/309820549 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:23.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.672+0000 7f7141a00700 1 --2- 172.21.15.89:0/309820549 >> v2:172.21.15.113:3300/0 conn(0x7f713c0d2540 0x7f713c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.672+0000 7f7141000700 1 -- 172.21.15.89:0/309820549 >> v2:172.21.15.113:3300/0 conn(0x7f713c0d2540 msgr2=0x7f713c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:23.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.672+0000 7f7141000700 1 --2- 172.21.15.89:0/309820549 >> v2:172.21.15.113:3300/0 conn(0x7f713c0d2540 0x7f713c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.672+0000 7f7141000700 1 -- 172.21.15.89:0/309820549 >> v1:172.21.15.113:6789/0 conn(0x7f713c0d16a0 legacy=0x7f713c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:23.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.672+0000 7f7141000700 1 -- 172.21.15.89:0/309820549 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c1619e0 con 0x7f713c15d9d0 2024-04-02T07:56:23.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.672+0000 7f7141a00700 1 --2- 172.21.15.89:0/309820549 >> v2:172.21.15.113:3300/0 conn(0x7f713c0d2540 0x7f713c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:23.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.673+0000 7f7141000700 1 --2- 172.21.15.89:0/309820549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f713c15d9d0 0x7f713c0d3c30 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f713c08ca10 tx=0x7f7130009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d0fe8e4eadda30 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:23.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.673+0000 7f713b400700 1 -- 172.21.15.89:0/309820549 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7130004230 con 0x7f713c15d9d0 2024-04-02T07:56:23.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.673+0000 7f713b400700 1 -- 172.21.15.89:0/309820549 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7130004390 con 0x7f713c15d9d0 2024-04-02T07:56:23.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.673+0000 7f713b400700 1 -- 172.21.15.89:0/309820549 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71300036a0 con 0x7f713c15d9d0 2024-04-02T07:56:23.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.673+0000 7f7143a00700 1 -- 172.21.15.89:0/309820549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f713c15d9d0 msgr2=0x7f713c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.673+0000 7f7143a00700 1 --2- 172.21.15.89:0/309820549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f713c15d9d0 0x7f713c0d3c30 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f713c08ca10 tx=0x7f7130009bd0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.673+0000 7f7143a00700 1 -- 172.21.15.89:0/309820549 shutdown_connections 2024-04-02T07:56:23.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.673+0000 7f7143a00700 1 --2- 172.21.15.89:0/309820549 >> v2:172.21.15.113:3300/0 conn(0x7f713c0d2540 0x7f713c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.673+0000 7f7143a00700 1 --2- 172.21.15.89:0/309820549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f713c15d9d0 0x7f713c0d3c30 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.673+0000 7f7143a00700 1 -- 172.21.15.89:0/309820549 >> 172.21.15.89:0/309820549 conn(0x7f713c08aff0 msgr2=0x7f713c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:23.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.673+0000 7f7143a00700 1 -- 172.21.15.89:0/309820549 shutdown_connections 2024-04-02T07:56:23.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.674+0000 7f7143a00700 1 -- 172.21.15.89:0/309820549 wait complete. 2024-04-02T07:56:23.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.676+0000 7f7143a00700 1 Processor -- start 2024-04-02T07:56:23.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.676+0000 7f7143a00700 1 -- start start 2024-04-02T07:56:23.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.677+0000 7f7143a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f713c0d16a0 0x7f713c0b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.677+0000 7f7143a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f713c0d2540 0x7f713c0b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.677+0000 7f7143a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f713c15d9d0 0x7f713c1e8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.677+0000 7f7143a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713c0cb150 con 0x7f713c0d2540 2024-04-02T07:56:23.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.677+0000 7f7143a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713c0b7fc0 con 0x7f713c15d9d0 2024-04-02T07:56:23.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.677+0000 7f7143a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f713c1e7840 con 0x7f713c0d16a0 2024-04-02T07:56:23.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.677+0000 7f713be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f713c0d2540 0x7f713c0b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.677+0000 7f713be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f713c0d2540 0x7f713c0b07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42386/0 (socket says 172.21.15.89:42386) 2024-04-02T07:56:23.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.677+0000 7f713be00700 1 -- 172.21.15.89:0/757246181 learned_addr learned my addr 172.21.15.89:0/757246181 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:23.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.678+0000 7f7141a00700 1 --2- 172.21.15.89:0/757246181 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f713c15d9d0 0x7f713c1e8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.678+0000 7f7141000700 1 --2- 172.21.15.89:0/757246181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f713c0d16a0 0x7f713c0b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.678+0000 7f713be00700 1 -- 172.21.15.89:0/757246181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f713c0d16a0 msgr2=0x7f713c0b7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.678+0000 7f713be00700 1 --2- 172.21.15.89:0/757246181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f713c0d16a0 0x7f713c0b7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.678+0000 7f713be00700 1 -- 172.21.15.89:0/757246181 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f713c15d9d0 msgr2=0x7f713c1e8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.678+0000 7f713be00700 1 --2- 172.21.15.89:0/757246181 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f713c15d9d0 0x7f713c1e8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.678+0000 7f713be00700 1 -- 172.21.15.89:0/757246181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7130009550 con 0x7f713c0d2540 2024-04-02T07:56:23.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.678+0000 7f713be00700 1 --2- 172.21.15.89:0/757246181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f713c0d2540 0x7f713c0b07d0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f712800c480 tx=0x7f712800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:23.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.678+0000 7f7138c00700 1 -- 172.21.15.89:0/757246181 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7128011790 con 0x7f713c0d2540 2024-04-02T07:56:23.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.678+0000 7f7138c00700 1 -- 172.21.15.89:0/757246181 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712800ce90 con 0x7f713c0d2540 2024-04-02T07:56:23.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.678+0000 7f7138c00700 1 -- 172.21.15.89:0/757246181 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7128011b50 con 0x7f713c0d2540 2024-04-02T07:56:23.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.678+0000 7f7143a00700 1 -- 172.21.15.89:0/757246181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713c1e7b20 con 0x7f713c0d2540 2024-04-02T07:56:23.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.679+0000 7f7143a00700 1 -- 172.21.15.89:0/757246181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f713c1e80e0 con 0x7f713c0d2540 2024-04-02T07:56:23.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.681+0000 7f7138c00700 1 -- 172.21.15.89:0/757246181 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7128011cb0 con 0x7f713c0d2540 2024-04-02T07:56:23.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.681+0000 7f7138c00700 1 --2- 172.21.15.89:0/757246181 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f711c06ade0 0x7f711c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.682+0000 7f7138c00700 1 -- 172.21.15.89:0/757246181 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f7128089380 con 0x7f713c0d2540 2024-04-02T07:56:23.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.682+0000 7f7141000700 1 --2- 172.21.15.89:0/757246181 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f711c06ade0 0x7f711c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.682+0000 7f7141000700 1 --2- 172.21.15.89:0/757246181 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f711c06ade0 0x7f711c06d2a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7130009130 tx=0x7f7130008030 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:23.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.683+0000 7f71575cbb80 1 Processor -- start 2024-04-02T07:56:23.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.684+0000 7f71575cbb80 1 -- start start 2024-04-02T07:56:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f71575cbb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22233830 0x555a22233c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f71575cbb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22206be0 0x555a22209a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f71575cbb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 0x555a22230a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f71575cbb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a2217e650 con 0x555a22233830 2024-04-02T07:56:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f71575cbb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a2215fc30 con 0x555a22204b60 2024-04-02T07:56:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f71575cbb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a2215d430 con 0x555a22206be0 2024-04-02T07:56:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f7141000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22233830 0x555a22233c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f7141000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22233830 0x555a22233c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42388/0 (socket says 172.21.15.89:42388) 2024-04-02T07:56:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f7141000700 1 -- 172.21.15.89:0/3946181616 learned_addr learned my addr 172.21.15.89:0/3946181616 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f7141a00700 1 --2- 172.21.15.89:0/3946181616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 0x555a22230a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f713be00700 1 --2- 172.21.15.89:0/3946181616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22206be0 0x555a22209a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f7141000700 1 -- 172.21.15.89:0/3946181616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22206be0 msgr2=0x555a22209a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f7141000700 1 --2- 172.21.15.89:0/3946181616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22206be0 0x555a22209a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f7141000700 1 -- 172.21.15.89:0/3946181616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 msgr2=0x555a22230a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f7141000700 1 --2- 172.21.15.89:0/3946181616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 0x555a22230a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f7141000700 1 -- 172.21.15.89:0/3946181616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a21f4c790 con 0x555a22233830 2024-04-02T07:56:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f7141a00700 1 --2- 172.21.15.89:0/3946181616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 0x555a22230a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.685+0000 7f713be00700 1 --2- 172.21.15.89:0/3946181616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22206be0 0x555a22209a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.686+0000 7f7141000700 1 --2- 172.21.15.89:0/3946181616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22233830 0x555a22233c10 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f7130009fd0 tx=0x7f7130010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d7b6633cb05ed1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:23.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.686+0000 7f712d600700 1 -- 172.21.15.89:0/3946181616 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7130012640 con 0x555a22233830 2024-04-02T07:56:23.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.686+0000 7f712d600700 1 -- 172.21.15.89:0/3946181616 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71300127a0 con 0x555a22233830 2024-04-02T07:56:23.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.686+0000 7f712d600700 1 -- 172.21.15.89:0/3946181616 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7130012b10 con 0x555a22233830 2024-04-02T07:56:23.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.686+0000 7f71575cbb80 1 -- 172.21.15.89:0/3946181616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22233830 msgr2=0x555a22233c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.686+0000 7f71575cbb80 1 --2- 172.21.15.89:0/3946181616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22233830 0x555a22233c10 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f7130009fd0 tx=0x7f7130010000 comp rx=0 tx=0).stop 2024-04-02T07:56:23.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.686+0000 7f71575cbb80 1 -- 172.21.15.89:0/3946181616 shutdown_connections 2024-04-02T07:56:23.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.686+0000 7f71575cbb80 1 --2- 172.21.15.89:0/3946181616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 0x555a22230a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.686+0000 7f71575cbb80 1 --2- 172.21.15.89:0/3946181616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22233830 0x555a22233c10 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.686+0000 7f71575cbb80 1 --2- 172.21.15.89:0/3946181616 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22206be0 0x555a22209a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.686+0000 7f71575cbb80 1 -- 172.21.15.89:0/3946181616 >> 172.21.15.89:0/3946181616 conn(0x555a221676a0 msgr2=0x555a2224e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:23.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.687+0000 7f71575cbb80 1 -- 172.21.15.89:0/3946181616 shutdown_connections 2024-04-02T07:56:23.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.687+0000 7f71575cbb80 1 -- 172.21.15.89:0/3946181616 wait complete. 2024-04-02T07:56:23.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.687+0000 7f71575cbb80 1 -- start start 2024-04-02T07:56:23.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.687+0000 7f71575cbb80 1 Processor -- start 2024-04-02T07:56:23.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.688+0000 7f71575cbb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 0x555a2224ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.688+0000 7f71575cbb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22206be0 0x555a22251ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.688+0000 7f71575cbb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22233830 0x555a22252020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.688+0000 7f71575cbb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a22158d20 con 0x555a22206be0 2024-04-02T07:56:23.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.688+0000 7f71575cbb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a220f4770 con 0x555a22204b60 2024-04-02T07:56:23.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.688+0000 7f71575cbb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a221176f0 con 0x555a22233830 2024-04-02T07:56:23.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.688+0000 7f71575cbb80 10 client.0 _ll_register_callbacks cb 0x555a220fe5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:56:23.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.689+0000 7f713be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22206be0 0x555a22251ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.689+0000 7f713be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22206be0 0x555a22251ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42400/0 (socket says 172.21.15.89:42400) 2024-04-02T07:56:23.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.689+0000 7f713be00700 1 -- 172.21.15.89:0/2332464345 learned_addr learned my addr 172.21.15.89:0/2332464345 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:23.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.689+0000 7f713be00700 1 -- 172.21.15.89:0/2332464345 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22233830 msgr2=0x555a22252020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:23.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.689+0000 7f713be00700 1 --2- 172.21.15.89:0/2332464345 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22233830 0x555a22252020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.689+0000 7f713be00700 1 -- 172.21.15.89:0/2332464345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 msgr2=0x555a2224ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:23.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.689+0000 7f713be00700 1 --2- 172.21.15.89:0/2332464345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 0x555a2224ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.689+0000 7f713be00700 1 -- 172.21.15.89:0/2332464345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f713c1619e0 con 0x555a22206be0 2024-04-02T07:56:23.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.689+0000 7f713be00700 1 --2- 172.21.15.89:0/2332464345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22206be0 0x555a22251ae0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f7128011fd0 tx=0x7f7128058690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1d4095ee821ce2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:23.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.689+0000 7f7122000700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:56:23.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.689+0000 7f7122000700 1 -- 172.21.15.89:0/2332464345 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71280582c0 con 0x555a22206be0 2024-04-02T07:56:23.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.689+0000 7f7122000700 1 -- 172.21.15.89:0/2332464345 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f71280602e0 con 0x555a22206be0 2024-04-02T07:56:23.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.689+0000 7f71575cbb80 1 -- 172.21.15.89:0/2332464345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555a221fb920 con 0x555a22206be0 2024-04-02T07:56:23.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.690+0000 7f7122000700 1 -- 172.21.15.89:0/2332464345 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7128087250 con 0x555a22206be0 2024-04-02T07:56:23.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.690+0000 7f71575cbb80 10 client.9168 fetch_fsmap learned FSMap version 73 2024-04-02T07:56:23.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.690+0000 7f71575cbb80 1 -- 172.21.15.89:0/2332464345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x555a220390a0 con 0x555a22206be0 2024-04-02T07:56:23.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.690+0000 7f7122000700 1 -- 172.21.15.89:0/2332464345 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555a221fb920 con 0x555a22206be0 2024-04-02T07:56:23.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.690+0000 7f7122000700 1 -- 172.21.15.89:0/2332464345 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 73) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7128057350 con 0x555a22206be0 2024-04-02T07:56:23.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.690+0000 7f71575cbb80 10 client.9168 fetch_fsmap finished waiting for FSMap version 73 2024-04-02T07:56:23.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.690+0000 7f71575cbb80 10 client.9168 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T07:56:23.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.690+0000 7f71575cbb80 20 client.9168 populate_metadata read hostname 'smithi089' 2024-04-02T07:56:23.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.690+0000 7f71575cbb80 1 --2- 172.21.15.89:0/2332464345 >> [v2:172.21.15.89:6836/3861363328,v1:172.21.15.89:6837/3861363328] conn(0x555a22235f10 0x555a222362f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.690+0000 7f71575cbb80 4 client.9168 mds_command: new command op to 9033 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T07:56:23.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.690+0000 7f71575cbb80 1 -- 172.21.15.89:0/2332464345 --> [v2:172.21.15.89:6836/3861363328,v1:172.21.15.89:6837/3861363328] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a221d86f0 con 0x555a22235f10 2024-04-02T07:56:23.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.690+0000 7f7141000700 1 --2- 172.21.15.89:0/2332464345 >> [v2:172.21.15.89:6836/3861363328,v1:172.21.15.89:6837/3861363328] conn(0x555a22235f10 0x555a222362f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.691+0000 7f7122000700 10 client.9168 ms_handle_connect on v2:172.21.15.89:6836/3861363328 2024-04-02T07:56:23.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.691+0000 7f7141000700 1 --2- 172.21.15.89:0/2332464345 >> [v2:172.21.15.89:6836/3861363328,v1:172.21.15.89:6837/3861363328] conn(0x555a22235f10 0x555a222362f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26ad1c70c39874fd server_cookie=52be4574488c0ca6 in_seq=0 out_seq=0 2024-04-02T07:56:23.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.691+0000 7f7122000700 1 -- 172.21.15.89:0/2332464345 <== mds.0 v2:172.21.15.89:6836/3861363328 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555a221d86f0 con 0x555a22235f10 2024-04-02T07:56:23.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.691+0000 7f7122000700 10 client.9168 handle_command_reply: tid=0 2024-04-02T07:56:23.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.692+0000 7f71575cbb80 1 client.9168 shutdown 2024-04-02T07:56:23.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.692+0000 7f71575cbb80 1 -- 172.21.15.89:0/2332464345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22206be0 msgr2=0x555a22251ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.692+0000 7f71575cbb80 1 --2- 172.21.15.89:0/2332464345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22206be0 0x555a22251ae0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f7128011fd0 tx=0x7f7128058690 comp rx=0 tx=0).stop 2024-04-02T07:56:23.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.692+0000 7f71575cbb80 1 -- 172.21.15.89:0/2332464345 shutdown_connections 2024-04-02T07:56:23.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.692+0000 7f71575cbb80 1 --2- 172.21.15.89:0/2332464345 >> [v2:172.21.15.89:6836/3861363328,v1:172.21.15.89:6837/3861363328] conn(0x555a22235f10 0x555a222362f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.692+0000 7f71575cbb80 1 --2- 172.21.15.89:0/2332464345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 0x555a2224ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.692+0000 7f71575cbb80 1 --2- 172.21.15.89:0/2332464345 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22233830 0x555a22252020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.692+0000 7f71575cbb80 1 --2- 172.21.15.89:0/2332464345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22206be0 0x555a22251ae0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.692+0000 7f71575cbb80 1 -- 172.21.15.89:0/2332464345 >> 172.21.15.89:0/2332464345 conn(0x555a221676a0 msgr2=0x555a2224c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:23.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.692+0000 7f7122000700 0 client.9168 ms_handle_reset on v2:172.21.15.89:6836/3861363328 2024-04-02T07:56:23.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.692+0000 7f71575cbb80 1 -- 172.21.15.89:0/2332464345 shutdown_connections 2024-04-02T07:56:23.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.692+0000 7f71575cbb80 1 -- 172.21.15.89:0/2332464345 wait complete. 2024-04-02T07:56:23.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.692+0000 7f71575cbb80 20 client.9168 trim_cache size 0 max 16384 2024-04-02T07:56:23.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.710+0000 7f71575cbb80 1 Processor -- start 2024-04-02T07:56:23.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.710+0000 7f71575cbb80 1 -- start start 2024-04-02T07:56:23.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.711+0000 7f71575cbb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 0x555a22223b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.711+0000 7f71575cbb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22206be0 0x555a22251ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.711+0000 7f71575cbb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22233830 0x555a22252010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.711+0000 7f71575cbb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a21f4d390 con 0x555a22233830 2024-04-02T07:56:23.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.711+0000 7f71575cbb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a2220a9c0 con 0x555a22204b60 2024-04-02T07:56:23.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.711+0000 7f71575cbb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a22224190 con 0x555a22206be0 2024-04-02T07:56:23.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.711+0000 7f7141a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22233830 0x555a22252010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.711+0000 7f7141a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22233830 0x555a22252010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42414/0 (socket says 172.21.15.89:42414) 2024-04-02T07:56:23.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.711+0000 7f7141a00700 1 -- 172.21.15.89:0/2864050911 learned_addr learned my addr 172.21.15.89:0/2864050911 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:23.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.711+0000 7f7141000700 1 --2- 172.21.15.89:0/2864050911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 0x555a22223b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.711+0000 7f713be00700 1 --2- 172.21.15.89:0/2864050911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22206be0 0x555a22251ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.712+0000 7f7141a00700 1 -- 172.21.15.89:0/2864050911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22206be0 msgr2=0x555a22251ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:23.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.712+0000 7f7141a00700 1 --2- 172.21.15.89:0/2864050911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22206be0 0x555a22251ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.712+0000 7f7141a00700 1 -- 172.21.15.89:0/2864050911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 msgr2=0x555a22223b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:23.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.712+0000 7f7141a00700 1 --2- 172.21.15.89:0/2864050911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 0x555a22223b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.712+0000 7f7141a00700 1 -- 172.21.15.89:0/2864050911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a2211f010 con 0x555a22233830 2024-04-02T07:56:23.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.712+0000 7f7141a00700 1 --2- 172.21.15.89:0/2864050911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22233830 0x555a22252010 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f713400e540 tx=0x7f713400e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738eb71b6ef5bf3a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:23.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.712+0000 7f712e000700 1 -- 172.21.15.89:0/2864050911 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7134013070 con 0x555a22233830 2024-04-02T07:56:23.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.712+0000 7f712e000700 1 -- 172.21.15.89:0/2864050911 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713400ec80 con 0x555a22233830 2024-04-02T07:56:23.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.712+0000 7f712e000700 1 -- 172.21.15.89:0/2864050911 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713400c420 con 0x555a22233830 2024-04-02T07:56:23.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.713+0000 7f71575cbb80 1 -- 172.21.15.89:0/2864050911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22233830 msgr2=0x555a22252010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.713+0000 7f71575cbb80 1 --2- 172.21.15.89:0/2864050911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22233830 0x555a22252010 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f713400e540 tx=0x7f713400e570 comp rx=0 tx=0).stop 2024-04-02T07:56:23.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.713+0000 7f71575cbb80 1 -- 172.21.15.89:0/2864050911 shutdown_connections 2024-04-02T07:56:23.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.713+0000 7f71575cbb80 1 --2- 172.21.15.89:0/2864050911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22204b60 0x555a22223b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.713+0000 7f71575cbb80 1 --2- 172.21.15.89:0/2864050911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22233830 0x555a22252010 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.713+0000 7f71575cbb80 1 --2- 172.21.15.89:0/2864050911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22206be0 0x555a22251ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.713+0000 7f71575cbb80 1 -- 172.21.15.89:0/2864050911 >> 172.21.15.89:0/2864050911 conn(0x555a221676a0 msgr2=0x555a2224ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:23.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.713+0000 7f71575cbb80 1 -- 172.21.15.89:0/2864050911 shutdown_connections 2024-04-02T07:56:23.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.713+0000 7f71575cbb80 1 -- 172.21.15.89:0/2864050911 wait complete. 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.714+0000 7f71575cbb80 1 -- start start 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.714+0000 7f71575cbb80 1 Processor -- start 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f71575cbb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22204b60 0x555a22252070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f71575cbb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22206be0 0x555a222b3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f71575cbb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22233830 0x555a22234600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f71575cbb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a222b5da0 con 0x555a22206be0 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f71575cbb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a22208ef0 con 0x555a22233830 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f71575cbb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a222b6bc0 con 0x555a22204b60 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f71575cbb80 10 client.0 _ll_register_callbacks cb 0x7f71280578b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f713be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22206be0 0x555a222b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f713be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22206be0 0x555a222b3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42424/0 (socket says 172.21.15.89:42424) 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f713be00700 1 -- 172.21.15.89:0/118226718 learned_addr learned my addr 172.21.15.89:0/118226718 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f713be00700 1 -- 172.21.15.89:0/118226718 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22204b60 msgr2=0x555a22252070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f713be00700 1 --2- 172.21.15.89:0/118226718 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22204b60 0x555a22252070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f713be00700 1 -- 172.21.15.89:0/118226718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22233830 msgr2=0x555a22234600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:23.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f713be00700 1 --2- 172.21.15.89:0/118226718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22233830 0x555a22234600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f713be00700 1 -- 172.21.15.89:0/118226718 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7134007d60 con 0x555a22206be0 2024-04-02T07:56:23.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f713be00700 1 --2- 172.21.15.89:0/118226718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22206be0 0x555a222b3220 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f712800cd50 tx=0x7f712801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce2231581525af5e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:23.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.715+0000 7f7122000700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T07:56:23.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f7122000700 1 -- 172.21.15.89:0/118226718 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71280582c0 con 0x555a22206be0 2024-04-02T07:56:23.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f7122000700 1 -- 172.21.15.89:0/118226718 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f712805fe50 con 0x555a22206be0 2024-04-02T07:56:23.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f71575cbb80 1 -- 172.21.15.89:0/118226718 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f713c1e96e0 con 0x555a22206be0 2024-04-02T07:56:23.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f7122000700 1 -- 172.21.15.89:0/118226718 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7128060010 con 0x555a22206be0 2024-04-02T07:56:23.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f71575cbb80 10 client.9174 fetch_fsmap learned FSMap version 73 2024-04-02T07:56:23.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f71575cbb80 1 -- 172.21.15.89:0/118226718 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x555a21fad1d0 con 0x555a22206be0 2024-04-02T07:56:23.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f7122000700 1 -- 172.21.15.89:0/118226718 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c1e96e0 con 0x555a22206be0 2024-04-02T07:56:23.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f7122000700 1 -- 172.21.15.89:0/118226718 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 73) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7128081080 con 0x555a22206be0 2024-04-02T07:56:23.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f71575cbb80 10 client.9174 fetch_fsmap finished waiting for FSMap version 73 2024-04-02T07:56:23.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f71575cbb80 10 client.9174 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T07:56:23.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f71575cbb80 20 client.9174 populate_metadata read hostname 'smithi089' 2024-04-02T07:56:23.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f71575cbb80 1 --2- 172.21.15.89:0/118226718 >> [v2:172.21.15.89:6836/3861363328,v1:172.21.15.89:6837/3861363328] conn(0x555a222b96d0 0x555a222b9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f71575cbb80 4 client.9174 mds_command: new command op to 9033 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T07:56:23.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f71575cbb80 1 -- 172.21.15.89:0/118226718 --> [v2:172.21.15.89:6836/3861363328,v1:172.21.15.89:6837/3861363328] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f71300127a0 con 0x555a222b96d0 2024-04-02T07:56:23.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.716+0000 7f7141000700 1 --2- 172.21.15.89:0/118226718 >> [v2:172.21.15.89:6836/3861363328,v1:172.21.15.89:6837/3861363328] conn(0x555a222b96d0 0x555a222b9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f7122000700 10 client.9174 ms_handle_connect on v2:172.21.15.89:6836/3861363328 2024-04-02T07:56:23.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f7141000700 1 --2- 172.21.15.89:0/118226718 >> [v2:172.21.15.89:6836/3861363328,v1:172.21.15.89:6837/3861363328] conn(0x555a222b96d0 0x555a222b9ab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e97d019c7613f9d server_cookie=102f6f4e73abaa72 in_seq=0 out_seq=0 2024-04-02T07:56:23.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f7122000700 1 -- 172.21.15.89:0/118226718 <== mds.0 v2:172.21.15.89:6836/3861363328 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f71300127a0 con 0x555a222b96d0 2024-04-02T07:56:23.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f7122000700 10 client.9174 handle_command_reply: tid=0 2024-04-02T07:56:23.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f71575cbb80 1 client.9174 shutdown 2024-04-02T07:56:23.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f71575cbb80 1 -- 172.21.15.89:0/118226718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22206be0 msgr2=0x555a222b3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f71575cbb80 1 --2- 172.21.15.89:0/118226718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22206be0 0x555a222b3220 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f712800cd50 tx=0x7f712801c000 comp rx=0 tx=0).stop 2024-04-02T07:56:23.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f71575cbb80 1 -- 172.21.15.89:0/118226718 shutdown_connections 2024-04-02T07:56:23.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f71575cbb80 1 --2- 172.21.15.89:0/118226718 >> [v2:172.21.15.89:6836/3861363328,v1:172.21.15.89:6837/3861363328] conn(0x555a222b96d0 0x555a222b9ab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f71575cbb80 1 --2- 172.21.15.89:0/118226718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555a22233830 0x555a22234600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f71575cbb80 1 --2- 172.21.15.89:0/118226718 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x555a22204b60 0x555a22252070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f71575cbb80 1 --2- 172.21.15.89:0/118226718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a22206be0 0x555a222b3220 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f7122000700 0 client.9174 ms_handle_reset on v2:172.21.15.89:6836/3861363328 2024-04-02T07:56:23.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f71575cbb80 1 -- 172.21.15.89:0/118226718 >> 172.21.15.89:0/118226718 conn(0x555a221676a0 msgr2=0x555a2224d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:23.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.717+0000 7f71575cbb80 1 -- 172.21.15.89:0/118226718 shutdown_connections 2024-04-02T07:56:23.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.718+0000 7f71575cbb80 1 -- 172.21.15.89:0/118226718 wait complete. 2024-04-02T07:56:23.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.718+0000 7f71575cbb80 20 client.9174 trim_cache size 0 max 16384 2024-04-02T07:56:23.723 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T07:56:23.724 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T07:56:23.724 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T07:56:23.724 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 9033, 2024-04-02T07:56:23.724 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T07:56:23.724 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T07:56:23.724 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T07:56:23.724 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 6.5235171379999999, 2024-04-02T07:56:23.724 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 73, 2024-04-02T07:56:23.724 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 154, 2024-04-02T07:56:23.724 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 154, 2024-04-02T07:56:23.724 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 11.982805008 2024-04-02T07:56:23.724 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T07:56:23.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.718+0000 7f7143a00700 1 -- 172.21.15.89:0/757246181 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f711c06ade0 msgr2=0x7f711c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.718+0000 7f7143a00700 1 --2- 172.21.15.89:0/757246181 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f711c06ade0 0x7f711c06d2a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7130009130 tx=0x7f7130008030 comp rx=0 tx=0).stop 2024-04-02T07:56:23.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.718+0000 7f7143a00700 1 -- 172.21.15.89:0/757246181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f713c0d2540 msgr2=0x7f713c0b07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.718+0000 7f7143a00700 1 --2- 172.21.15.89:0/757246181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f713c0d2540 0x7f713c0b07d0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f712800c480 tx=0x7f712800c790 comp rx=0 tx=0).stop 2024-04-02T07:56:23.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.718+0000 7f7143a00700 1 -- 172.21.15.89:0/757246181 shutdown_connections 2024-04-02T07:56:23.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.718+0000 7f7143a00700 1 --2- 172.21.15.89:0/757246181 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f711c06ade0 0x7f711c06d2a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.718+0000 7f7143a00700 1 --2- 172.21.15.89:0/757246181 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f713c15d9d0 0x7f713c1e8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.718+0000 7f7143a00700 1 --2- 172.21.15.89:0/757246181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f713c0d16a0 0x7f713c0b7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.718+0000 7f7143a00700 1 --2- 172.21.15.89:0/757246181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f713c0d2540 0x7f713c0b07d0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.718+0000 7f7143a00700 1 -- 172.21.15.89:0/757246181 >> 172.21.15.89:0/757246181 conn(0x7f713c08aff0 msgr2=0x7f713c0b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:23.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.718+0000 7f7143a00700 1 -- 172.21.15.89:0/757246181 shutdown_connections 2024-04-02T07:56:23.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.718+0000 7f7143a00700 1 -- 172.21.15.89:0/757246181 wait complete. 2024-04-02T07:56:23.737 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T07:56:23.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.927+0000 7f1b91800700 1 Processor -- start 2024-04-02T07:56:23.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.927+0000 7f1b91800700 1 -- start start 2024-04-02T07:56:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.928+0000 7f1b91800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1b8c160be0 0x7f1b8c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.928+0000 7f1b91800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1b8c0ceca0 0x7f1b8c0b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.928+0000 7f1b91800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c160fc0 con 0x7f1b8c0ce0e0 2024-04-02T07:56:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.928+0000 7f1b91800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8c0b65d0 con 0x7f1b8c0ceca0 2024-04-02T07:56:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.928+0000 7f1b91800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8c0d2a80 con 0x7f1b8c160be0 2024-04-02T07:56:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.928+0000 7f1b8b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1b8c0ceca0 0x7f1b8c0b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.928+0000 7f1b8b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1b8c0ceca0 0x7f1b8c0b6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42426/0 (socket says 172.21.15.89:42426) 2024-04-02T07:56:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.928+0000 7f1b8b400700 1 -- 172.21.15.89:0/2781970881 learned_addr learned my addr 172.21.15.89:0/2781970881 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.929+0000 7f1b8aa00700 1 --2- 172.21.15.89:0/2781970881 >> v2:172.21.15.113:3300/0 conn(0x7f1b8c160be0 0x7f1b8c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.929+0000 7f1b8b400700 1 -- 172.21.15.89:0/2781970881 >> v2:172.21.15.113:3300/0 conn(0x7f1b8c160be0 msgr2=0x7f1b8c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.929+0000 7f1b8b400700 1 --2- 172.21.15.89:0/2781970881 >> v2:172.21.15.113:3300/0 conn(0x7f1b8c160be0 0x7f1b8c0cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.929+0000 7f1b8b400700 1 -- 172.21.15.89:0/2781970881 >> v1:172.21.15.113:6789/0 conn(0x7f1b8c0ce0e0 legacy=0x7f1b8c0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.929+0000 7f1b8b400700 1 -- 172.21.15.89:0/2781970881 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c0d21e0 con 0x7f1b8c0ceca0 2024-04-02T07:56:23.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.929+0000 7f1b8aa00700 1 --2- 172.21.15.89:0/2781970881 >> v2:172.21.15.113:3300/0 conn(0x7f1b8c160be0 0x7f1b8c0cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:23.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.929+0000 7f1b8b400700 1 --2- 172.21.15.89:0/2781970881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b8c0ceca0 0x7f1b8c0b6a20 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c08b590 tx=0x7f1b8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf5ea620c061ba07 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:23.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.929+0000 7f1b89600700 1 -- 172.21.15.89:0/2781970881 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8400e070 con 0x7f1b8c0ceca0 2024-04-02T07:56:23.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.929+0000 7f1b89600700 1 -- 172.21.15.89:0/2781970881 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8400b9c0 con 0x7f1b8c0ceca0 2024-04-02T07:56:23.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.929+0000 7f1b89600700 1 -- 172.21.15.89:0/2781970881 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8400bd70 con 0x7f1b8c0ceca0 2024-04-02T07:56:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.930+0000 7f1b91800700 1 -- 172.21.15.89:0/2781970881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b8c0ceca0 msgr2=0x7f1b8c0b6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.930+0000 7f1b91800700 1 --2- 172.21.15.89:0/2781970881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b8c0ceca0 0x7f1b8c0b6a20 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c08b590 tx=0x7f1b8400b330 comp rx=0 tx=0).stop 2024-04-02T07:56:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.930+0000 7f1b91800700 1 -- 172.21.15.89:0/2781970881 shutdown_connections 2024-04-02T07:56:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.930+0000 7f1b91800700 1 --2- 172.21.15.89:0/2781970881 >> v2:172.21.15.113:3300/0 conn(0x7f1b8c160be0 0x7f1b8c0cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.930+0000 7f1b91800700 1 --2- 172.21.15.89:0/2781970881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b8c0ceca0 0x7f1b8c0b6a20 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.930+0000 7f1b91800700 1 -- 172.21.15.89:0/2781970881 >> 172.21.15.89:0/2781970881 conn(0x7f1b8c08aff0 msgr2=0x7f1b8c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.930+0000 7f1b91800700 1 -- 172.21.15.89:0/2781970881 shutdown_connections 2024-04-02T07:56:23.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.930+0000 7f1b91800700 1 -- 172.21.15.89:0/2781970881 wait complete. 2024-04-02T07:56:23.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.932+0000 7f1b91800700 1 Processor -- start 2024-04-02T07:56:23.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.932+0000 7f1b91800700 1 -- start start 2024-04-02T07:56:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b91800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b8c0ce0e0 0x7f1b8c1f3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b91800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1b8c0ceca0 0x7f1b8c0b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b91800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b8c160be0 0x7f1b8c09d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b91800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c1f37e0 con 0x7f1b8c160be0 2024-04-02T07:56:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b91800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c1f3ac0 con 0x7f1b8c0ce0e0 2024-04-02T07:56:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b91800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c1f3da0 con 0x7f1b8c0ceca0 2024-04-02T07:56:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b8b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b8c160be0 0x7f1b8c09d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b8b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b8c160be0 0x7f1b8c09d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42442/0 (socket says 172.21.15.89:42442) 2024-04-02T07:56:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b8b400700 1 -- 172.21.15.89:0/3357691402 learned_addr learned my addr 172.21.15.89:0/3357691402 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b8aa00700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b8c0ce0e0 0x7f1b8c1f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b8a000700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1b8c0ceca0 0x7f1b8c0b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b8b400700 1 -- 172.21.15.89:0/3357691402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1b8c0ceca0 msgr2=0x7f1b8c0b77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b8b400700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1b8c0ceca0 0x7f1b8c0b77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b8b400700 1 -- 172.21.15.89:0/3357691402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b8c0ce0e0 msgr2=0x7f1b8c1f3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b8b400700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b8c0ce0e0 0x7f1b8c1f3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b8b400700 1 -- 172.21.15.89:0/3357691402 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b84009b30 con 0x7f1b8c160be0 2024-04-02T07:56:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.933+0000 7f1b8a000700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1b8c0ceca0 0x7f1b8c0b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.934+0000 7f1b8b400700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b8c160be0 0x7f1b8c09d230 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f1b84012000 tx=0x7f1b84004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.934+0000 7f1b82a00700 1 -- 172.21.15.89:0/3357691402 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b840099b0 con 0x7f1b8c160be0 2024-04-02T07:56:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.934+0000 7f1b82a00700 1 -- 172.21.15.89:0/3357691402 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b840092e0 con 0x7f1b8c160be0 2024-04-02T07:56:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.934+0000 7f1b91800700 1 -- 172.21.15.89:0/3357691402 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8c1f4080 con 0x7f1b8c160be0 2024-04-02T07:56:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.934+0000 7f1b82a00700 1 -- 172.21.15.89:0/3357691402 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8401c640 con 0x7f1b8c160be0 2024-04-02T07:56:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.934+0000 7f1b91800700 1 -- 172.21.15.89:0/3357691402 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8c0cd920 con 0x7f1b8c160be0 2024-04-02T07:56:23.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.935+0000 7f1b91800700 1 -- 172.21.15.89:0/3357691402 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8c1f6a60 con 0x7f1b8c160be0 2024-04-02T07:56:23.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.936+0000 7f1b82a00700 1 -- 172.21.15.89:0/3357691402 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1b8401c7a0 con 0x7f1b8c160be0 2024-04-02T07:56:23.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.937+0000 7f1b82a00700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1b6c06adb0 0x7f1b6c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:23.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.937+0000 7f1b82a00700 1 -- 172.21.15.89:0/3357691402 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f1b8408b050 con 0x7f1b8c160be0 2024-04-02T07:56:23.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.937+0000 7f1b8aa00700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1b6c06adb0 0x7f1b6c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:23.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.938+0000 7f1b8aa00700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1b6c06adb0 0x7f1b6c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c15f2e0 tx=0x7f1b7c008040 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:23.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:23.942+0000 7f1b82a00700 1 -- 172.21.15.89:0/3357691402 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1b8c1f6a60 con 0x7f1b8c160be0 2024-04-02T07:56:24.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:24.191+0000 7f1b91800700 1 -- 172.21.15.89:0/3357691402 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1b8c1f6a60 con 0x7f1b8c160be0 2024-04-02T07:56:24.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:24.191+0000 7f1b82a00700 1 -- 172.21.15.89:0/3357691402 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v73) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1b8c1f6a60 con 0x7f1b8c160be0 2024-04-02T07:56:24.193 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T07:56:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:24.193+0000 7f1b91800700 1 -- 172.21.15.89:0/3357691402 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1b6c06adb0 msgr2=0x7f1b6c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:24.193+0000 7f1b91800700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1b6c06adb0 0x7f1b6c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c15f2e0 tx=0x7f1b7c008040 comp rx=0 tx=0).stop 2024-04-02T07:56:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:24.194+0000 7f1b91800700 1 -- 172.21.15.89:0/3357691402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b8c160be0 msgr2=0x7f1b8c09d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:24.194+0000 7f1b91800700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b8c160be0 0x7f1b8c09d230 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f1b84012000 tx=0x7f1b84004ad0 comp rx=0 tx=0).stop 2024-04-02T07:56:24.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:24.194+0000 7f1b91800700 1 -- 172.21.15.89:0/3357691402 shutdown_connections 2024-04-02T07:56:24.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:24.194+0000 7f1b91800700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1b6c06adb0 0x7f1b6c06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:24.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:24.194+0000 7f1b91800700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b8c0ce0e0 0x7f1b8c1f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:24.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:24.194+0000 7f1b91800700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b8c160be0 0x7f1b8c09d230 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:24.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:24.194+0000 7f1b91800700 1 --2- 172.21.15.89:0/3357691402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1b8c0ceca0 0x7f1b8c0b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:24.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:24.194+0000 7f1b91800700 1 -- 172.21.15.89:0/3357691402 >> 172.21.15.89:0/3357691402 conn(0x7f1b8c08aff0 msgr2=0x7f1b8c0b8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:24.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:24.194+0000 7f1b91800700 1 -- 172.21.15.89:0/3357691402 shutdown_connections 2024-04-02T07:56:24.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:24.194+0000 7f1b91800700 1 -- 172.21.15.89:0/3357691402 wait complete. 2024-04-02T07:56:24.213 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T07:56:24.213 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T07:56:24.213 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T07:56:24.213 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T07:56:24.213 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T07:56:24.214 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T07:56:24.214 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T07:56:24.214 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T07:56:24.214 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:56:24.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:56:24.214 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T07:56:24.259 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T07:56:24.259 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T07:56:24.259 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T07:56:24.259 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:56:24.259 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T07:56:24.259 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:56:24.259 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:56:24.259 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:56:24.259 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T07:56:24.259 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:56:24.259 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:56:24.260 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T07:56:24.260 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T07:56:24.260 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:56:24.260 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T07:56:24.260 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:56:24.260 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T07:56:24.260 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 41244sec preferred_lft 41244sec 2024-04-02T07:56:24.260 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T07:56:24.260 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:56:24.260 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T07:56:24.260 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T07:56:24.260 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T07:56:24.260 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:56:24.260 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T07:56:24.261 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:56:24.261 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:56:24.261 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T07:56:24.261 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T07:56:24.261 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:56:24.261 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T07:56:24.261 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T07:56:24.261 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T07:56:24.261 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T07:56:24.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:56:24.262 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T07:56:24.366 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T07:56:24.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:56:24.369 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T07:56:24.416 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T07:56:24.416 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T07:56:24.416 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T07:56:24.416 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T07:56:24.416 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T07:56:24.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:56:24.417 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T07:56:24.417 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T07:56:24.417 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T07:56:24.417 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T07:56:24.417 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T07:56:24.417 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T07:56:24.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:56:24.576 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T07:56:24.620 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T07:56:24.620 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T07:56:24.621 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:56:24.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:56:24.621 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T07:56:24.729 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:56:24.730 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T07:56:24.775 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T07:56:24.775 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T07:56:24.776 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:56:24.859 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T07:56:24.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:56:24.859 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:56:24.999 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:56:24.999 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:56:25.026 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T07:56:25.026 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T07:56:25.026 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T07:56:25.026 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T07:56:25.026 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T07:56:25.032 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T07:56:25.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:56:25.032 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T07:56:25.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:56:25.098 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T07:56:25.197 INFO:teuthology.orchestra.run.smithi089.stdout:client8527 2024-04-02T07:56:25.198 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T07:56:25.198 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T07:56:25.244 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T07:56:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.475+0000 7f6b84e00700 1 Processor -- start 2024-04-02T07:56:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.475+0000 7f6b84e00700 1 -- start start 2024-04-02T07:56:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b84e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6b800cbb30 0x7f6b800cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b84e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6b800cd020 0x7f6b800cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b84e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6b800b2070 0x7f6b800b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b84e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b800cc450 con 0x7f6b800cd020 2024-04-02T07:56:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b84e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b800cdae0 con 0x7f6b800cbb30 2024-04-02T07:56:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b84e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800cddc0 con 0x7f6b800b2070 2024-04-02T07:56:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b7d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6b800cd020 0x7f6b800cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b7d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6b800cd020 0x7f6b800cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42452/0 (socket says 172.21.15.89:42452) 2024-04-02T07:56:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b7d400700 1 -- 172.21.15.89:0/1263063197 learned_addr learned my addr 172.21.15.89:0/1263063197 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b7de00700 1 --2- 172.21.15.89:0/1263063197 >> v2:172.21.15.113:3300/0 conn(0x7f6b800cbb30 0x7f6b800cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b7e800700 1 --2- 172.21.15.89:0/1263063197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6b800b2070 0x7f6b800b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b7d400700 1 -- 172.21.15.89:0/1263063197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6b800b2070 msgr2=0x7f6b800b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b7d400700 1 --2- 172.21.15.89:0/1263063197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6b800b2070 0x7f6b800b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b7d400700 1 -- 172.21.15.89:0/1263063197 >> v2:172.21.15.113:3300/0 conn(0x7f6b800cbb30 msgr2=0x7f6b800cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b7d400700 1 --2- 172.21.15.89:0/1263063197 >> v2:172.21.15.113:3300/0 conn(0x7f6b800cbb30 0x7f6b800cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.477+0000 7f6b7d400700 1 -- 172.21.15.89:0/1263063197 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b800ce0a0 con 0x7f6b800cd020 2024-04-02T07:56:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.478+0000 7f6b7d400700 1 --2- 172.21.15.89:0/1263063197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b800cd020 0x7f6b800cd480 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f6b8006fd00 tx=0x7f6b6c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b931d58e08bb6b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.478+0000 7f6b7ca00700 1 -- 172.21.15.89:0/1263063197 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c00b870 con 0x7f6b800cd020 2024-04-02T07:56:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.478+0000 7f6b7ca00700 1 -- 172.21.15.89:0/1263063197 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6c00b9d0 con 0x7f6b800cd020 2024-04-02T07:56:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.478+0000 7f6b7ca00700 1 -- 172.21.15.89:0/1263063197 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c00bd60 con 0x7f6b800cd020 2024-04-02T07:56:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.478+0000 7f6b84e00700 1 -- 172.21.15.89:0/1263063197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b800cd020 msgr2=0x7f6b800cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.478+0000 7f6b84e00700 1 --2- 172.21.15.89:0/1263063197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b800cd020 0x7f6b800cd480 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f6b8006fd00 tx=0x7f6b6c009bd0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.478+0000 7f6b84e00700 1 -- 172.21.15.89:0/1263063197 shutdown_connections 2024-04-02T07:56:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.478+0000 7f6b84e00700 1 --2- 172.21.15.89:0/1263063197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6b800b2070 0x7f6b800b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.478+0000 7f6b84e00700 1 --2- 172.21.15.89:0/1263063197 >> v2:172.21.15.113:3300/0 conn(0x7f6b800cbb30 0x7f6b800cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.478+0000 7f6b84e00700 1 --2- 172.21.15.89:0/1263063197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b800cd020 0x7f6b800cd480 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.478+0000 7f6b84e00700 1 -- 172.21.15.89:0/1263063197 >> 172.21.15.89:0/1263063197 conn(0x7f6b800b0d70 msgr2=0x7f6b800b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.478+0000 7f6b84e00700 1 -- 172.21.15.89:0/1263063197 shutdown_connections 2024-04-02T07:56:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.479+0000 7f6b84e00700 1 -- 172.21.15.89:0/1263063197 wait complete. 2024-04-02T07:56:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.480+0000 7f6b84e00700 1 Processor -- start 2024-04-02T07:56:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.480+0000 7f6b84e00700 1 -- start start 2024-04-02T07:56:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.481+0000 7f6b84e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b800b2070 0x7f6b80151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.481+0000 7f6b84e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b800cbb30 0x7f6b800ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.481+0000 7f6b84e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6b800cd020 0x7f6b800b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.481+0000 7f6b84e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b80152970 con 0x7f6b800cbb30 2024-04-02T07:56:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.481+0000 7f6b84e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b80070a60 con 0x7f6b800b2070 2024-04-02T07:56:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.481+0000 7f6b84e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b80070d40 con 0x7f6b800cd020 2024-04-02T07:56:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b7d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b800cbb30 0x7f6b800ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b7d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b800cbb30 0x7f6b800ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42458/0 (socket says 172.21.15.89:42458) 2024-04-02T07:56:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b7d400700 1 -- 172.21.15.89:0/3506060136 learned_addr learned my addr 172.21.15.89:0/3506060136 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b7de00700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b800b2070 0x7f6b80151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b7e800700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6b800cd020 0x7f6b800b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b7d400700 1 -- 172.21.15.89:0/3506060136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6b800cd020 msgr2=0x7f6b800b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b7d400700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6b800cd020 0x7f6b800b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b7d400700 1 -- 172.21.15.89:0/3506060136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b800b2070 msgr2=0x7f6b80151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b7d400700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b800b2070 0x7f6b80151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b7d400700 1 -- 172.21.15.89:0/3506060136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c009550 con 0x7f6b800cbb30 2024-04-02T07:56:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b7de00700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b800b2070 0x7f6b80151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b7d400700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b800cbb30 0x7f6b800ad1d0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009c10 tx=0x7f6b6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b76000700 1 -- 172.21.15.89:0/3506060136 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c0036a0 con 0x7f6b800cbb30 2024-04-02T07:56:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b76000700 1 -- 172.21.15.89:0/3506060136 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6c003800 con 0x7f6b800cbb30 2024-04-02T07:56:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b76000700 1 -- 172.21.15.89:0/3506060136 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c003bb0 con 0x7f6b800cbb30 2024-04-02T07:56:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.482+0000 7f6b84e00700 1 -- 172.21.15.89:0/3506060136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b80071020 con 0x7f6b800cbb30 2024-04-02T07:56:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.483+0000 7f6b84e00700 1 -- 172.21.15.89:0/3506060136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b800715e0 con 0x7f6b800cbb30 2024-04-02T07:56:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.484+0000 7f6b84e00700 1 -- 172.21.15.89:0/3506060136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b800717c0 con 0x7f6b800cbb30 2024-04-02T07:56:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.484+0000 7f6b76000700 1 -- 172.21.15.89:0/3506060136 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6b6c003d10 con 0x7f6b800cbb30 2024-04-02T07:56:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.485+0000 7f6b76000700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b6006adb0 0x7f6b6006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.485+0000 7f6b76000700 1 -- 172.21.15.89:0/3506060136 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f6b6c08ad40 con 0x7f6b800cbb30 2024-04-02T07:56:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.486+0000 7f6b7de00700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b6006adb0 0x7f6b6006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.486+0000 7f6b7de00700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b6006adb0 0x7f6b6006d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6b700079d0 tx=0x7f6b70008040 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.490+0000 7f6b76000700 1 -- 172.21.15.89:0/3506060136 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6b6c055da0 con 0x7f6b800cbb30 2024-04-02T07:56:25.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.718+0000 7f6b84e00700 1 -- 172.21.15.89:0/3506060136 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b801594a0 con 0x7f6b6006adb0 2024-04-02T07:56:25.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.720+0000 7f6b76000700 1 -- 172.21.15.89:0/3506060136 <== mgr.8790 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6b70007e40 con 0x7f6b6006adb0 2024-04-02T07:56:25.721 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T07:56:25.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T07:56:25.721 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T07:56:25.722 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T07:56:25.722 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T07:56:25.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.722+0000 7f6b84e00700 1 -- 172.21.15.89:0/3506060136 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b6006adb0 msgr2=0x7f6b6006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:25.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.722+0000 7f6b84e00700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b6006adb0 0x7f6b6006d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6b700079d0 tx=0x7f6b70008040 comp rx=0 tx=0).stop 2024-04-02T07:56:25.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.722+0000 7f6b84e00700 1 -- 172.21.15.89:0/3506060136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b800cbb30 msgr2=0x7f6b800ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:25.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.722+0000 7f6b84e00700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b800cbb30 0x7f6b800ad1d0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009c10 tx=0x7f6b6c004750 comp rx=0 tx=0).stop 2024-04-02T07:56:25.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.722+0000 7f6b84e00700 1 -- 172.21.15.89:0/3506060136 shutdown_connections 2024-04-02T07:56:25.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.722+0000 7f6b84e00700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6b6006adb0 0x7f6b6006d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.722+0000 7f6b84e00700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b800b2070 0x7f6b80151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.722+0000 7f6b84e00700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6b800cd020 0x7f6b800b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.722+0000 7f6b84e00700 1 --2- 172.21.15.89:0/3506060136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b800cbb30 0x7f6b800ad1d0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.722+0000 7f6b84e00700 1 -- 172.21.15.89:0/3506060136 >> 172.21.15.89:0/3506060136 conn(0x7f6b800b0d70 msgr2=0x7f6b800b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:25.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.723+0000 7f6b84e00700 1 -- 172.21.15.89:0/3506060136 shutdown_connections 2024-04-02T07:56:25.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.723+0000 7f6b84e00700 1 -- 172.21.15.89:0/3506060136 wait complete. 2024-04-02T07:56:25.742 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T07:56:25.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.934+0000 7f9abfa00700 1 Processor -- start 2024-04-02T07:56:25.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.934+0000 7f9abfa00700 1 -- start start 2024-04-02T07:56:25.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.936+0000 7f9abfa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab80d21f0 0x7f9ab80ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:25.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.936+0000 7f9abfa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ab80cec40 0x7f9ab8161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:25.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.936+0000 7f9abfa00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab80ce270 con 0x7f9ab81611b0 2024-04-02T07:56:25.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.936+0000 7f9abfa00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab815b600 con 0x7f9ab80cec40 2024-04-02T07:56:25.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.936+0000 7f9abfa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab80cf0b0 con 0x7f9ab80d21f0 2024-04-02T07:56:25.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.936+0000 7f9abda00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ab80cec40 0x7f9ab8161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:25.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.936+0000 7f9abda00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ab80cec40 0x7f9ab8161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42472/0 (socket says 172.21.15.89:42472) 2024-04-02T07:56:25.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.936+0000 7f9abda00700 1 -- 172.21.15.89:0/1136745611 learned_addr learned my addr 172.21.15.89:0/1136745611 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:25.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.936+0000 7f9ab7e00700 1 --2- 172.21.15.89:0/1136745611 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab80d21f0 0x7f9ab80ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:25.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.936+0000 7f9abda00700 1 -- 172.21.15.89:0/1136745611 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab80d21f0 msgr2=0x7f9ab80ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:25.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.936+0000 7f9abda00700 1 --2- 172.21.15.89:0/1136745611 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab80d21f0 0x7f9ab80ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.936+0000 7f9abda00700 1 -- 172.21.15.89:0/1136745611 >> v1:172.21.15.113:6789/0 conn(0x7f9ab81611b0 legacy=0x7f9ab815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:25.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.936+0000 7f9abda00700 1 -- 172.21.15.89:0/1136745611 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab80d2760 con 0x7f9ab80cec40 2024-04-02T07:56:25.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.937+0000 7f9abda00700 1 --2- 172.21.15.89:0/1136745611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab80cec40 0x7f9ab8161590 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009e50 tx=0x7f9ab000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e07d8a09229104 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.937+0000 7f9ab7400700 1 -- 172.21.15.89:0/1136745611 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab000e070 con 0x7f9ab80cec40 2024-04-02T07:56:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.937+0000 7f9ab7400700 1 -- 172.21.15.89:0/1136745611 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab000b910 con 0x7f9ab80cec40 2024-04-02T07:56:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.937+0000 7f9ab7400700 1 -- 172.21.15.89:0/1136745611 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab000bcc0 con 0x7f9ab80cec40 2024-04-02T07:56:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.937+0000 7f9abfa00700 1 -- 172.21.15.89:0/1136745611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab80cec40 msgr2=0x7f9ab8161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.937+0000 7f9abfa00700 1 --2- 172.21.15.89:0/1136745611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab80cec40 0x7f9ab8161590 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009e50 tx=0x7f9ab000b330 comp rx=0 tx=0).stop 2024-04-02T07:56:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.937+0000 7f9abfa00700 1 -- 172.21.15.89:0/1136745611 shutdown_connections 2024-04-02T07:56:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.937+0000 7f9abfa00700 1 --2- 172.21.15.89:0/1136745611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab80cec40 0x7f9ab8161590 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.937+0000 7f9abfa00700 1 --2- 172.21.15.89:0/1136745611 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab80d21f0 0x7f9ab80ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.937+0000 7f9abfa00700 1 -- 172.21.15.89:0/1136745611 >> 172.21.15.89:0/1136745611 conn(0x7f9ab808aff0 msgr2=0x7f9ab80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.938+0000 7f9abfa00700 1 -- 172.21.15.89:0/1136745611 shutdown_connections 2024-04-02T07:56:25.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.938+0000 7f9abfa00700 1 -- 172.21.15.89:0/1136745611 wait complete. 2024-04-02T07:56:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.939+0000 7f9abfa00700 1 Processor -- start 2024-04-02T07:56:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.939+0000 7f9abfa00700 1 -- start start 2024-04-02T07:56:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.940+0000 7f9abfa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ab80cec40 0x7f9ab80ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.940+0000 7f9abfa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab80d21f0 0x7f9ab81f26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.940+0000 7f9abfa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab81611b0 0x7f9ab8158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.940+0000 7f9abfa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab81577c0 con 0x7f9ab80d21f0 2024-04-02T07:56:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.940+0000 7f9abfa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab8155240 con 0x7f9ab80cec40 2024-04-02T07:56:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.940+0000 7f9abfa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab8155490 con 0x7f9ab81611b0 2024-04-02T07:56:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.940+0000 7f9ab7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab80d21f0 0x7f9ab81f26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9ab7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab80d21f0 0x7f9ab81f26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42474/0 (socket says 172.21.15.89:42474) 2024-04-02T07:56:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9ab7e00700 1 -- 172.21.15.89:0/2977762981 learned_addr learned my addr 172.21.15.89:0/2977762981 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9abda00700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab81611b0 0x7f9ab8158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9abd000700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ab80cec40 0x7f9ab80ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9ab7e00700 1 -- 172.21.15.89:0/2977762981 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab81611b0 msgr2=0x7f9ab8158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9ab7e00700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab81611b0 0x7f9ab8158340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9ab7e00700 1 -- 172.21.15.89:0/2977762981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ab80cec40 msgr2=0x7f9ab80ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9ab7e00700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ab80cec40 0x7f9ab80ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9ab7e00700 1 -- 172.21.15.89:0/2977762981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab0009b30 con 0x7f9ab80d21f0 2024-04-02T07:56:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9abd000700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ab80cec40 0x7f9ab80ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9ab7e00700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab80d21f0 0x7f9ab81f26d0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400e9b0 tx=0x7f9aa400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9ab4c00700 1 -- 172.21.15.89:0/2977762981 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4009710 con 0x7f9ab80d21f0 2024-04-02T07:56:25.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9ab4c00700 1 -- 172.21.15.89:0/2977762981 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa400c850 con 0x7f9ab80d21f0 2024-04-02T07:56:25.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9ab4c00700 1 -- 172.21.15.89:0/2977762981 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa400cbc0 con 0x7f9ab80d21f0 2024-04-02T07:56:25.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.941+0000 7f9abfa00700 1 -- 172.21.15.89:0/2977762981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab8155770 con 0x7f9ab80d21f0 2024-04-02T07:56:25.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.942+0000 7f9abfa00700 1 -- 172.21.15.89:0/2977762981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab8155d30 con 0x7f9ab80d21f0 2024-04-02T07:56:25.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.943+0000 7f9abfa00700 1 -- 172.21.15.89:0/2977762981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab8155f50 con 0x7f9ab80d21f0 2024-04-02T07:56:25.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.944+0000 7f9ab4c00700 1 -- 172.21.15.89:0/2977762981 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 90) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9aa400cd20 con 0x7f9ab80d21f0 2024-04-02T07:56:25.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.944+0000 7f9ab4c00700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a9806adb0 0x7f9a9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:25.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.944+0000 7f9ab4c00700 1 -- 172.21.15.89:0/2977762981 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f9aa4089700 con 0x7f9ab80d21f0 2024-04-02T07:56:25.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.945+0000 7f9abd000700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a9806adb0 0x7f9a9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:25.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.945+0000 7f9abd000700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a9806adb0 0x7f9a9806d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8156e40 tx=0x7f9aac009040 comp rx=0 tx=0).ready entity=mgr.8790 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:25.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:25.949+0000 7f9ab4c00700 1 -- 172.21.15.89:0/2977762981 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9aa4054640 con 0x7f9ab80d21f0 2024-04-02T07:56:26.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:26.207+0000 7f9abfa00700 1 -- 172.21.15.89:0/2977762981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f9ab8067720 con 0x7f9ab80d21f0 2024-04-02T07:56:27.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.069+0000 7f9ab4c00700 1 -- 172.21.15.89:0/2977762981 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 91) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f9aa4054a10 con 0x7f9ab80d21f0 2024-04-02T07:56:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.070+0000 7f9ab4c00700 1 -- 172.21.15.89:0/2977762981 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v91) v1 ==== 92+0+0 (secure 0 0 0) 0x7f9aa4087630 con 0x7f9ab80d21f0 2024-04-02T07:56:27.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.074+0000 7f9abfa00700 1 -- 172.21.15.89:0/2977762981 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a9806adb0 msgr2=0x7f9a9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:27.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.074+0000 7f9abfa00700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a9806adb0 0x7f9a9806d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8156e40 tx=0x7f9aac009040 comp rx=0 tx=0).stop 2024-04-02T07:56:27.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.074+0000 7f9abfa00700 1 -- 172.21.15.89:0/2977762981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab80d21f0 msgr2=0x7f9ab81f26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:27.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.074+0000 7f9abfa00700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab80d21f0 0x7f9ab81f26d0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400e9b0 tx=0x7f9aa400ecc0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.074+0000 7f9abfa00700 1 -- 172.21.15.89:0/2977762981 shutdown_connections 2024-04-02T07:56:27.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.074+0000 7f9abfa00700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9a9806adb0 0x7f9a9806d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.074+0000 7f9abfa00700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ab80cec40 0x7f9ab80ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.074+0000 7f9abfa00700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ab81611b0 0x7f9ab8158340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.074+0000 7f9abfa00700 1 --2- 172.21.15.89:0/2977762981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ab80d21f0 0x7f9ab81f26d0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.074+0000 7f9abfa00700 1 -- 172.21.15.89:0/2977762981 >> 172.21.15.89:0/2977762981 conn(0x7f9ab808aff0 msgr2=0x7f9ab81598c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:27.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.074+0000 7f9abfa00700 1 -- 172.21.15.89:0/2977762981 shutdown_connections 2024-04-02T07:56:27.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.074+0000 7f9abfa00700 1 -- 172.21.15.89:0/2977762981 wait complete. 2024-04-02T07:56:27.105 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T07:56:27.287 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:27.287+0000 7f9463e78240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:56:27.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.372+0000 7fd6cb200700 1 Processor -- start 2024-04-02T07:56:27.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.372+0000 7fd6cb200700 1 -- start start 2024-04-02T07:56:27.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.373+0000 7fd6cb200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd6c40cbb30 0x7fd6c40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:27.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.373+0000 7fd6cb200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6c40cd020 0x7fd6c40cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:27.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.373+0000 7fd6cb200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6c40b2070 0x7fd6c40b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:27.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.373+0000 7fd6cb200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6c40cc450 con 0x7fd6c40b2070 2024-04-02T07:56:27.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.373+0000 7fd6cb200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6c40cdae0 con 0x7fd6c40cbb30 2024-04-02T07:56:27.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.373+0000 7fd6cb200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c40cddc0 con 0x7fd6c40cd020 2024-04-02T07:56:27.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.374+0000 7fd6c9200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6c40b2070 0x7fd6c40b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:27.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.374+0000 7fd6c3400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6c40cd020 0x7fd6c40cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:27.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.374+0000 7fd6c9200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6c40b2070 0x7fd6c40b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42512/0 (socket says 172.21.15.89:42512) 2024-04-02T07:56:27.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.374+0000 7fd6c9200700 1 -- 172.21.15.89:0/918885186 learned_addr learned my addr 172.21.15.89:0/918885186 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:27.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.374+0000 7fd6c3e00700 1 --2- 172.21.15.89:0/918885186 >> v2:172.21.15.113:3300/0 conn(0x7fd6c40cbb30 0x7fd6c40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:27.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.374+0000 7fd6c9200700 1 -- 172.21.15.89:0/918885186 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6c40cd020 msgr2=0x7fd6c40cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:27.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.374+0000 7fd6c9200700 1 --2- 172.21.15.89:0/918885186 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6c40cd020 0x7fd6c40cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.374+0000 7fd6c9200700 1 -- 172.21.15.89:0/918885186 >> v2:172.21.15.113:3300/0 conn(0x7fd6c40cbb30 msgr2=0x7fd6c40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:27.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.374+0000 7fd6c9200700 1 --2- 172.21.15.89:0/918885186 >> v2:172.21.15.113:3300/0 conn(0x7fd6c40cbb30 0x7fd6c40cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.374+0000 7fd6c9200700 1 -- 172.21.15.89:0/918885186 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c40ce0a0 con 0x7fd6c40b2070 2024-04-02T07:56:27.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.374+0000 7fd6c3e00700 1 --2- 172.21.15.89:0/918885186 >> v2:172.21.15.113:3300/0 conn(0x7fd6c40cbb30 0x7fd6c40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T07:56:27.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.374+0000 7fd6c9200700 1 --2- 172.21.15.89:0/918885186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c40b2070 0x7fd6c40b2500 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fd6c406fd00 tx=0x7fd6bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53294aa62de19 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:27.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.375+0000 7fd6c2a00700 1 -- 172.21.15.89:0/918885186 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc00e070 con 0x7fd6c40b2070 2024-04-02T07:56:27.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.375+0000 7fd6c2a00700 1 -- 172.21.15.89:0/918885186 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc00b910 con 0x7fd6c40b2070 2024-04-02T07:56:27.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.375+0000 7fd6c2a00700 1 -- 172.21.15.89:0/918885186 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc00bcc0 con 0x7fd6c40b2070 2024-04-02T07:56:27.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.375+0000 7fd6cb200700 1 -- 172.21.15.89:0/918885186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c40b2070 msgr2=0x7fd6c40b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:27.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.375+0000 7fd6cb200700 1 --2- 172.21.15.89:0/918885186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c40b2070 0x7fd6c40b2500 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fd6c406fd00 tx=0x7fd6bc00b330 comp rx=0 tx=0).stop 2024-04-02T07:56:27.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.375+0000 7fd6cb200700 1 -- 172.21.15.89:0/918885186 shutdown_connections 2024-04-02T07:56:27.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.375+0000 7fd6cb200700 1 --2- 172.21.15.89:0/918885186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c40b2070 0x7fd6c40b2500 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.375+0000 7fd6cb200700 1 --2- 172.21.15.89:0/918885186 >> v2:172.21.15.113:3300/0 conn(0x7fd6c40cbb30 0x7fd6c40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.375+0000 7fd6cb200700 1 --2- 172.21.15.89:0/918885186 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6c40cd020 0x7fd6c40cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.375+0000 7fd6cb200700 1 -- 172.21.15.89:0/918885186 >> 172.21.15.89:0/918885186 conn(0x7fd6c40b0d70 msgr2=0x7fd6c40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:27.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.375+0000 7fd6cb200700 1 -- 172.21.15.89:0/918885186 shutdown_connections 2024-04-02T07:56:27.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.375+0000 7fd6cb200700 1 -- 172.21.15.89:0/918885186 wait complete. 2024-04-02T07:56:27.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.377+0000 7fd6cb200700 1 Processor -- start 2024-04-02T07:56:27.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.377+0000 7fd6cb200700 1 -- start start 2024-04-02T07:56:27.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.378+0000 7fd6cb200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6c40b2070 0x7fd6c4157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:27.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.378+0000 7fd6cb200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c40cbb30 0x7fd6c406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:27.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.378+0000 7fd6cb200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6c40cd020 0x7fd6c406e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:27.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.378+0000 7fd6cb200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c41f5460 con 0x7fd6c40cbb30 2024-04-02T07:56:27.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.378+0000 7fd6cb200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c41f47e0 con 0x7fd6c40cd020 2024-04-02T07:56:27.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.378+0000 7fd6cb200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c41f4ac0 con 0x7fd6c40b2070 2024-04-02T07:56:27.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.378+0000 7fd6c3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c40cbb30 0x7fd6c406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:27.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.379+0000 7fd6c3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c40cbb30 0x7fd6c406e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42528/0 (socket says 172.21.15.89:42528) 2024-04-02T07:56:27.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.379+0000 7fd6c3400700 1 -- 172.21.15.89:0/2587511322 learned_addr learned my addr 172.21.15.89:0/2587511322 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:27.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.379+0000 7fd6c3e00700 1 --2- 172.21.15.89:0/2587511322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6c40b2070 0x7fd6c4157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:27.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.379+0000 7fd6c9200700 1 --2- 172.21.15.89:0/2587511322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6c40cd020 0x7fd6c406e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:27.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.379+0000 7fd6c3400700 1 -- 172.21.15.89:0/2587511322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6c40b2070 msgr2=0x7fd6c4157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:27.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.379+0000 7fd6c3400700 1 --2- 172.21.15.89:0/2587511322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6c40b2070 0x7fd6c4157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.379+0000 7fd6c3400700 1 -- 172.21.15.89:0/2587511322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6c40cd020 msgr2=0x7fd6c406e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:27.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.379+0000 7fd6c3400700 1 --2- 172.21.15.89:0/2587511322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6c40cd020 0x7fd6c406e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.379+0000 7fd6c3400700 1 -- 172.21.15.89:0/2587511322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc009b30 con 0x7fd6c40cbb30 2024-04-02T07:56:27.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.379+0000 7fd6c3400700 1 --2- 172.21.15.89:0/2587511322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c40cbb30 0x7fd6c406e320 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000e9b0 tx=0x7fd6b000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:27.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.379+0000 7fd6bbe00700 1 -- 172.21.15.89:0/2587511322 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0009710 con 0x7fd6c40cbb30 2024-04-02T07:56:27.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.379+0000 7fd6bbe00700 1 -- 172.21.15.89:0/2587511322 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b000ee50 con 0x7fd6c40cbb30 2024-04-02T07:56:27.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.379+0000 7fd6bbe00700 1 -- 172.21.15.89:0/2587511322 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b000ca50 con 0x7fd6c40cbb30 2024-04-02T07:56:27.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.379+0000 7fd6cb200700 1 -- 172.21.15.89:0/2587511322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c41f4da0 con 0x7fd6c40cbb30 2024-04-02T07:56:27.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.380+0000 7fd6cb200700 1 -- 172.21.15.89:0/2587511322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c41578a0 con 0x7fd6c40cbb30 2024-04-02T07:56:27.383 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:27.380+0000 7fdc4c071240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T07:56:27.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.381+0000 7fd6cb200700 1 -- 172.21.15.89:0/2587511322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c4067720 con 0x7fd6c40cbb30 2024-04-02T07:56:27.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.382+0000 7fd6bbe00700 1 -- 172.21.15.89:0/2587511322 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 91) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fd6b000cbb0 con 0x7fd6c40cbb30 2024-04-02T07:56:27.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.383+0000 7fd6bbe00700 1 --2- 172.21.15.89:0/2587511322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd6a406ae00 0x7fd6a406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:27.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.383+0000 7fd6bbe00700 1 -- 172.21.15.89:0/2587511322 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fd6b00895d0 con 0x7fd6c40cbb30 2024-04-02T07:56:27.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.383+0000 7fd6c3e00700 1 -- 172.21.15.89:0/2587511322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd6a406ae00 msgr2=0x7fd6a406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:27.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.383+0000 7fd6c3e00700 1 --2- 172.21.15.89:0/2587511322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd6a406ae00 0x7fd6a406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:56:27.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.387+0000 7fd6bbe00700 1 -- 172.21.15.89:0/2587511322 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd6b0054510 con 0x7fd6c40cbb30 2024-04-02T07:56:27.472 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:27.472+0000 7f9463e78240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:56:27.570 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:27.568+0000 7fdc4c071240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T07:56:27.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.584+0000 7fd6c3e00700 1 -- 172.21.15.89:0/2587511322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd6a406ae00 msgr2=0x7fd6a406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:27.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.584+0000 7fd6c3e00700 1 --2- 172.21.15.89:0/2587511322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd6a406ae00 0x7fd6a406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:56:27.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.632+0000 7fd6cb200700 1 -- 172.21.15.89:0/2587511322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fd6c415bda0 con 0x7fd6c40cbb30 2024-04-02T07:56:27.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.636+0000 7fd6bbe00700 1 -- 172.21.15.89:0/2587511322 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd6b000ce60 con 0x7fd6c40cbb30 2024-04-02T07:56:27.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.638+0000 7fd6cb200700 1 -- 172.21.15.89:0/2587511322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd6a406ae00 msgr2=0x7fd6a406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:56:27.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.638+0000 7fd6cb200700 1 --2- 172.21.15.89:0/2587511322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd6a406ae00 0x7fd6a406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.638+0000 7fd6cb200700 1 -- 172.21.15.89:0/2587511322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c40cbb30 msgr2=0x7fd6c406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:27.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.638+0000 7fd6cb200700 1 --2- 172.21.15.89:0/2587511322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c40cbb30 0x7fd6c406e320 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000e9b0 tx=0x7fd6b000ecc0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.638+0000 7fd6cb200700 1 -- 172.21.15.89:0/2587511322 shutdown_connections 2024-04-02T07:56:27.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.638+0000 7fd6cb200700 1 --2- 172.21.15.89:0/2587511322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd6a406ae00 0x7fd6a406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.638+0000 7fd6cb200700 1 --2- 172.21.15.89:0/2587511322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6c40cd020 0x7fd6c406e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.638+0000 7fd6cb200700 1 --2- 172.21.15.89:0/2587511322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6c40b2070 0x7fd6c4157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.638+0000 7fd6cb200700 1 --2- 172.21.15.89:0/2587511322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c40cbb30 0x7fd6c406e320 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.638+0000 7fd6cb200700 1 -- 172.21.15.89:0/2587511322 >> 172.21.15.89:0/2587511322 conn(0x7fd6c40b0d70 msgr2=0x7fd6c415e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:27.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.639+0000 7fd6cb200700 1 -- 172.21.15.89:0/2587511322 shutdown_connections 2024-04-02T07:56:27.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.639+0000 7fd6cb200700 1 -- 172.21.15.89:0/2587511322 wait complete. 2024-04-02T07:56:27.659 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T07:56:27.788 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:27.788+0000 7f9463e78240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:56:27.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.857+0000 7f030e800700 1 Processor -- start 2024-04-02T07:56:27.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.857+0000 7f030e800700 1 -- start start 2024-04-02T07:56:27.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.858+0000 7f030e800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f030815d8a0 0x7f03080b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:27.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.858+0000 7f030e800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f03080d3eb0 0x7f030806b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:27.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.858+0000 7f030e800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03080ced20 con 0x7f030806bdd0 2024-04-02T07:56:27.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.858+0000 7f030e800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03080d4340 con 0x7f03080d3eb0 2024-04-02T07:56:27.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.858+0000 7f030e800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03080ce6d0 con 0x7f030815d8a0 2024-04-02T07:56:27.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.858+0000 7f0306a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f03080d3eb0 0x7f030806b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:27.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.859+0000 7f0307400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f030815d8a0 0x7f03080b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:27.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.859+0000 7f0306a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f03080d3eb0 0x7f030806b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42532/0 (socket says 172.21.15.89:42532) 2024-04-02T07:56:27.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.859+0000 7f0306a00700 1 -- 172.21.15.89:0/3426119547 learned_addr learned my addr 172.21.15.89:0/3426119547 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:27.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.859+0000 7f0306a00700 1 -- 172.21.15.89:0/3426119547 >> v2:172.21.15.113:3300/0 conn(0x7f030815d8a0 msgr2=0x7f03080b2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:27.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.859+0000 7f0306a00700 1 --2- 172.21.15.89:0/3426119547 >> v2:172.21.15.113:3300/0 conn(0x7f030815d8a0 0x7f03080b2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.859+0000 7f0306a00700 1 -- 172.21.15.89:0/3426119547 >> v1:172.21.15.113:6789/0 conn(0x7f030806bdd0 legacy=0x7f03080cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:27.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.859+0000 7f0306000700 1 -- 172.21.15.89:0/3426119547 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903148250 0 0) 0x7f03080ced20 con 0x7f030806bdd0 2024-04-02T07:56:27.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.859+0000 7f0306a00700 1 -- 172.21.15.89:0/3426119547 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03080ce9b0 con 0x7f03080d3eb0 2024-04-02T07:56:27.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.859+0000 7f0306a00700 1 --2- 172.21.15.89:0/3426119547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03080d3eb0 0x7f030806b890 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f030808bb90 tx=0x7f02f4009bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972372be53f90d24 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.859+0000 7f0306000700 1 -- 172.21.15.89:0/3426119547 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4011830 con 0x7f03080d3eb0 2024-04-02T07:56:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.860+0000 7f0306000700 1 -- 172.21.15.89:0/3426119547 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f4011990 con 0x7f03080d3eb0 2024-04-02T07:56:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.860+0000 7f0306000700 1 -- 172.21.15.89:0/3426119547 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4011d40 con 0x7f03080d3eb0 2024-04-02T07:56:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.860+0000 7f030e800700 1 -- 172.21.15.89:0/3426119547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03080d3eb0 msgr2=0x7f030806b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.860+0000 7f030e800700 1 --2- 172.21.15.89:0/3426119547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03080d3eb0 0x7f030806b890 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f030808bb90 tx=0x7f02f4009bc0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.860+0000 7f030e800700 1 -- 172.21.15.89:0/3426119547 shutdown_connections 2024-04-02T07:56:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.860+0000 7f030e800700 1 --2- 172.21.15.89:0/3426119547 >> v2:172.21.15.113:3300/0 conn(0x7f030815d8a0 0x7f03080b2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.860+0000 7f030e800700 1 --2- 172.21.15.89:0/3426119547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03080d3eb0 0x7f030806b890 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.860+0000 7f030e800700 1 -- 172.21.15.89:0/3426119547 >> 172.21.15.89:0/3426119547 conn(0x7f030808aff0 msgr2=0x7f03080d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.860+0000 7f030e800700 1 -- 172.21.15.89:0/3426119547 shutdown_connections 2024-04-02T07:56:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.860+0000 7f030e800700 1 -- 172.21.15.89:0/3426119547 wait complete. 2024-04-02T07:56:27.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.862+0000 7f030e800700 1 Processor -- start 2024-04-02T07:56:27.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.862+0000 7f030e800700 1 -- start start 2024-04-02T07:56:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.863+0000 7f030e800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f030806bdd0 0x7f03080b8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.863+0000 7f030e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030815d8a0 0x7f03081e88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.863+0000 7f030e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f03081e8e10 0x7f03080c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.863+0000 7f030e800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0308149dc0 con 0x7f030815d8a0 2024-04-02T07:56:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.863+0000 7f030e800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03080b9280 con 0x7f030806bdd0 2024-04-02T07:56:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.863+0000 7f030e800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03081e9290 con 0x7f03081e8e10 2024-04-02T07:56:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.863+0000 7f0306a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030815d8a0 0x7f03081e88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.863+0000 7f0307400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f030806bdd0 0x7f03080b8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.863+0000 7f0306a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030815d8a0 0x7f03081e88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42534/0 (socket says 172.21.15.89:42534) 2024-04-02T07:56:27.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.863+0000 7f0306a00700 1 -- 172.21.15.89:0/3556242283 learned_addr learned my addr 172.21.15.89:0/3556242283 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.863+0000 7f0307e00700 1 --2- 172.21.15.89:0/3556242283 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f03081e8e10 0x7f03080c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.864+0000 7f0306a00700 1 -- 172.21.15.89:0/3556242283 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f03081e8e10 msgr2=0x7f03080c8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.864+0000 7f0306a00700 1 --2- 172.21.15.89:0/3556242283 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f03081e8e10 0x7f03080c8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.864+0000 7f0306a00700 1 -- 172.21.15.89:0/3556242283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f030806bdd0 msgr2=0x7f03080b8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.864+0000 7f0306a00700 1 --2- 172.21.15.89:0/3556242283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f030806bdd0 0x7f03080b8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.864+0000 7f0306a00700 1 -- 172.21.15.89:0/3556242283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f40094f0 con 0x7f030815d8a0 2024-04-02T07:56:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.864+0000 7f0307400700 1 --2- 172.21.15.89:0/3556242283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f030806bdd0 0x7f03080b8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:56:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.864+0000 7f0306a00700 1 --2- 172.21.15.89:0/3556242283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030815d8a0 0x7f03081e88d0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f030808bef0 tx=0x7f02f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.864+0000 7f02ff400700 1 -- 172.21.15.89:0/3556242283 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4004440 con 0x7f030815d8a0 2024-04-02T07:56:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.864+0000 7f02ff400700 1 -- 172.21.15.89:0/3556242283 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f40045a0 con 0x7f030815d8a0 2024-04-02T07:56:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.864+0000 7f030e800700 1 -- 172.21.15.89:0/3556242283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03080c9140 con 0x7f030815d8a0 2024-04-02T07:56:27.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.864+0000 7f02ff400700 1 -- 172.21.15.89:0/3556242283 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f40038a0 con 0x7f030815d8a0 2024-04-02T07:56:27.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.865+0000 7f030e800700 1 -- 172.21.15.89:0/3556242283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03080c8e60 con 0x7f030815d8a0 2024-04-02T07:56:27.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.866+0000 7f030e800700 1 -- 172.21.15.89:0/3556242283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0308149f70 con 0x7f030815d8a0 2024-04-02T07:56:27.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.866+0000 7f02ff400700 1 -- 172.21.15.89:0/3556242283 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 91) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f02f4003a00 con 0x7f030815d8a0 2024-04-02T07:56:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.867+0000 7f02ff400700 1 --2- 172.21.15.89:0/3556242283 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f02e806ae00 0x7f02e806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.867+0000 7f02ff400700 1 -- 172.21.15.89:0/3556242283 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f02f4021030 con 0x7f030815d8a0 2024-04-02T07:56:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.867+0000 7f0307400700 1 -- 172.21.15.89:0/3556242283 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f02e806ae00 msgr2=0x7f02e806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.868+0000 7f0307400700 1 --2- 172.21.15.89:0/3556242283 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f02e806ae00 0x7f02e806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:56:27.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:27.876+0000 7f02ff400700 1 -- 172.21.15.89:0/3556242283 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0308149f70 con 0x7f030815d8a0 2024-04-02T07:56:27.915 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:27.914+0000 7fdc4c071240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T07:56:28.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.069+0000 7f0307400700 1 -- 172.21.15.89:0/3556242283 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f02e806ae00 msgr2=0x7f02e806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:28.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.069+0000 7f0307400700 1 --2- 172.21.15.89:0/3556242283 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f02e806ae00 0x7f02e806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:56:28.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.088+0000 7f030e800700 1 -- 172.21.15.89:0/3556242283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f0308149f70 con 0x7f030815d8a0 2024-04-02T07:56:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.094+0000 7f02ff400700 1 -- 172.21.15.89:0/3556242283 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0308149f70 con 0x7f030815d8a0 2024-04-02T07:56:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.096+0000 7f030e800700 1 -- 172.21.15.89:0/3556242283 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f02e806ae00 msgr2=0x7f02e806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:56:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.096+0000 7f030e800700 1 --2- 172.21.15.89:0/3556242283 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f02e806ae00 0x7f02e806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.096+0000 7f030e800700 1 -- 172.21.15.89:0/3556242283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030815d8a0 msgr2=0x7f03081e88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.096+0000 7f030e800700 1 --2- 172.21.15.89:0/3556242283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030815d8a0 0x7f03081e88d0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f030808bef0 tx=0x7f02f4004750 comp rx=0 tx=0).stop 2024-04-02T07:56:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.096+0000 7f030e800700 1 -- 172.21.15.89:0/3556242283 shutdown_connections 2024-04-02T07:56:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.096+0000 7f030e800700 1 --2- 172.21.15.89:0/3556242283 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f02e806ae00 0x7f02e806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.096+0000 7f030e800700 1 --2- 172.21.15.89:0/3556242283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f030806bdd0 0x7f03080b8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:28.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.096+0000 7f030e800700 1 --2- 172.21.15.89:0/3556242283 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f03081e8e10 0x7f03080c8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:28.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.096+0000 7f030e800700 1 --2- 172.21.15.89:0/3556242283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f030815d8a0 0x7f03081e88d0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:28.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.096+0000 7f030e800700 1 -- 172.21.15.89:0/3556242283 >> 172.21.15.89:0/3556242283 conn(0x7f030808aff0 msgr2=0x7f030815e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:28.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.096+0000 7f030e800700 1 -- 172.21.15.89:0/3556242283 shutdown_connections 2024-04-02T07:56:28.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.096+0000 7f030e800700 1 -- 172.21.15.89:0/3556242283 wait complete. 2024-04-02T07:56:28.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T07:56:28.116 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-02T07:56:28.165 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2024-04-02T07:56:28.319 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:28.319+0000 7f9463e78240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:56:28.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.364+0000 7f2974000700 1 Processor -- start 2024-04-02T07:56:28.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.364+0000 7f2974000700 1 -- start start 2024-04-02T07:56:28.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.366+0000 7f2974000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f296c0b41d0 0x7f296c06d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:28.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.366+0000 7f2974000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f296c15dd90 0x7f296c0ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:28.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.366+0000 7f2974000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296c06de80 con 0x7f296c0cc880 2024-04-02T07:56:28.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.366+0000 7f2974000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f296c06e160 con 0x7f296c0b41d0 2024-04-02T07:56:28.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.366+0000 7f2974000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296c06e440 con 0x7f296c15dd90 2024-04-02T07:56:28.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.366+0000 7f2971600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f296c0b41d0 0x7f296c06d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:28.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.366+0000 7f2972000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f296c15dd90 0x7f296c0ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:28.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.366+0000 7f2971600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f296c0b41d0 0x7f296c06d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:57872/0 (socket says 172.21.15.89:57872) 2024-04-02T07:56:28.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.366+0000 7f2971600700 1 -- 172.21.15.89:0/2815506433 learned_addr learned my addr 172.21.15.89:0/2815506433 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:28.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.367+0000 7f2971600700 1 -- 172.21.15.89:0/2815506433 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f296c15dd90 msgr2=0x7f296c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:28.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.367+0000 7f2971600700 1 --2- 172.21.15.89:0/2815506433 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f296c15dd90 0x7f296c0ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:28.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.367+0000 7f2971600700 1 -- 172.21.15.89:0/2815506433 >> v1:172.21.15.113:6789/0 conn(0x7f296c0cc880 legacy=0x7f296c15d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:28.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.367+0000 7f2971600700 1 -- 172.21.15.89:0/2815506433 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296c06e720 con 0x7f296c0b41d0 2024-04-02T07:56:28.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.367+0000 7f296be00700 1 -- 172.21.15.89:0/2815506433 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401626416 0 0) 0x7f296c06de80 con 0x7f296c0cc880 2024-04-02T07:56:28.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.367+0000 7f2971600700 1 --2- 172.21.15.89:0/2815506433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f296c0b41d0 0x7f296c06d580 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f296000bb90 tx=0x7f296000bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28e3946646b9c4b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:28.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.367+0000 7f296be00700 1 -- 172.21.15.89:0/2815506433 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296000dd30 con 0x7f296c0b41d0 2024-04-02T07:56:28.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.367+0000 7f296be00700 1 -- 172.21.15.89:0/2815506433 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296000de90 con 0x7f296c0b41d0 2024-04-02T07:56:28.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.367+0000 7f296be00700 1 -- 172.21.15.89:0/2815506433 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2960010690 con 0x7f296c0b41d0 2024-04-02T07:56:28.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.368+0000 7f2974000700 1 -- 172.21.15.89:0/2815506433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f296c0b41d0 msgr2=0x7f296c06d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:28.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.368+0000 7f2974000700 1 --2- 172.21.15.89:0/2815506433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f296c0b41d0 0x7f296c06d580 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f296000bb90 tx=0x7f296000bea0 comp rx=0 tx=0).stop 2024-04-02T07:56:28.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.368+0000 7f2974000700 1 -- 172.21.15.89:0/2815506433 shutdown_connections 2024-04-02T07:56:28.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.368+0000 7f2974000700 1 --2- 172.21.15.89:0/2815506433 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f296c15dd90 0x7f296c0ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:28.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.368+0000 7f2974000700 1 --2- 172.21.15.89:0/2815506433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f296c0b41d0 0x7f296c06d580 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:28.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.368+0000 7f2974000700 1 -- 172.21.15.89:0/2815506433 >> 172.21.15.89:0/2815506433 conn(0x7f296c071fb0 msgr2=0x7f296c06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:28.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.368+0000 7f2974000700 1 -- 172.21.15.89:0/2815506433 shutdown_connections 2024-04-02T07:56:28.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.368+0000 7f2974000700 1 -- 172.21.15.89:0/2815506433 wait complete. 2024-04-02T07:56:28.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.370+0000 7f2974000700 1 Processor -- start 2024-04-02T07:56:28.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.370+0000 7f2974000700 1 -- start start 2024-04-02T07:56:28.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2974000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f296c0b41d0 0x7f296c158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:28.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2974000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f296c0cc880 0x7f296c1510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:28.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2974000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f296c15dd90 0x7f296c151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:28.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2974000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c151b50 con 0x7f296c0cc880 2024-04-02T07:56:28.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2974000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c1e6100 con 0x7f296c15dd90 2024-04-02T07:56:28.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2974000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296c153d40 con 0x7f296c0b41d0 2024-04-02T07:56:28.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2970c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f296c0cc880 0x7f296c1510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:28.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2970c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f296c0cc880 0x7f296c1510d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42538/0 (socket says 172.21.15.89:42538) 2024-04-02T07:56:28.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2970c00700 1 -- 172.21.15.89:0/343418028 learned_addr learned my addr 172.21.15.89:0/343418028 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:28.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2971600700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f296c0b41d0 0x7f296c158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:28.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2972000700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f296c15dd90 0x7f296c151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:28.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2970c00700 1 -- 172.21.15.89:0/343418028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f296c0b41d0 msgr2=0x7f296c158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:28.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2970c00700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f296c0b41d0 0x7f296c158dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:28.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2970c00700 1 -- 172.21.15.89:0/343418028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f296c15dd90 msgr2=0x7f296c151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:28.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2970c00700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f296c15dd90 0x7f296c151610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:28.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.371+0000 7f2970c00700 1 -- 172.21.15.89:0/343418028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296000b7f0 con 0x7f296c0cc880 2024-04-02T07:56:28.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.372+0000 7f2970c00700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f296c0cc880 0x7f296c1510d0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f295800c750 tx=0x7f295800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:28.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.372+0000 7f2969600700 1 -- 172.21.15.89:0/343418028 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29580054e0 con 0x7f296c0cc880 2024-04-02T07:56:28.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.372+0000 7f2969600700 1 -- 172.21.15.89:0/343418028 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2958003680 con 0x7f296c0cc880 2024-04-02T07:56:28.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.372+0000 7f2969600700 1 -- 172.21.15.89:0/343418028 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2958003a30 con 0x7f296c0cc880 2024-04-02T07:56:28.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.372+0000 7f2974000700 1 -- 172.21.15.89:0/343418028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296c154020 con 0x7f296c0cc880 2024-04-02T07:56:28.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.372+0000 7f2974000700 1 -- 172.21.15.89:0/343418028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296c1e6630 con 0x7f296c0cc880 2024-04-02T07:56:28.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.373+0000 7f2974000700 1 -- 172.21.15.89:0/343418028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296c067720 con 0x7f296c0cc880 2024-04-02T07:56:28.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.374+0000 7f2969600700 1 -- 172.21.15.89:0/343418028 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 91) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f2958003b90 con 0x7f296c0cc880 2024-04-02T07:56:28.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.375+0000 7f2969600700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 0x7f294c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:28.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.375+0000 7f2969600700 1 -- 172.21.15.89:0/343418028 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f2958088a50 con 0x7f296c0cc880 2024-04-02T07:56:28.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.375+0000 7f2971600700 1 -- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 msgr2=0x7f294c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:28.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.375+0000 7f2971600700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 0x7f294c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T07:56:28.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.380+0000 7f2969600700 1 -- 172.21.15.89:0/343418028 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2958053a60 con 0x7f296c0cc880 2024-04-02T07:56:28.474 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:28.472+0000 7fdc4c071240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T07:56:28.572 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:28.573+0000 7f9463e78240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:56:28.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.576+0000 7f2971600700 1 -- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 msgr2=0x7f294c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:28.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.576+0000 7f2971600700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 0x7f294c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T07:56:28.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.616+0000 7f2974000700 1 -- 172.21.15.89:0/343418028 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f296c0b45b0 con 0x7f294c06ae00 2024-04-02T07:56:28.733 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:28.732+0000 7fdc4c071240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T07:56:28.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.977+0000 7f2971600700 1 -- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 msgr2=0x7f294c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:28.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:28.977+0000 7f2971600700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 0x7f294c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T07:56:29.030 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:29.030+0000 7f9463e78240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:56:29.189 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:29.188+0000 7fdc4c071240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T07:56:29.738 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:29.739+0000 7f9463e78240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:56:29.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:29.778+0000 7f2971600700 1 -- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 msgr2=0x7f294c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:29.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:29.778+0000 7f2971600700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 0x7f294c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T07:56:29.829 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:29.829+0000 7f9463e78240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:56:29.895 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:29.894+0000 7fdc4c071240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T07:56:29.987 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:29.986+0000 7fdc4c071240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T07:56:29.988 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:29.989+0000 7f9463e78240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:56:30.149 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:30.148+0000 7fdc4c071240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T07:56:30.202 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:30.202+0000 7f9463e78240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:56:30.294 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:30.294+0000 7f9463e78240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:56:30.364 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:30.363+0000 7fdc4c071240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T07:56:30.393 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:30.393+0000 7f9463e78240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:56:30.457 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:30.456+0000 7fdc4c071240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T07:56:30.490 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:30.491+0000 7f9463e78240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:56:30.555 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:30.554+0000 7fdc4c071240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T07:56:30.590 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:30.590+0000 7f9463e78240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:56:30.654 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:30.653+0000 7fdc4c071240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T07:56:30.753 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:30.752+0000 7fdc4c071240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T07:56:30.802 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:30.802+0000 7f9463e78240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:56:30.967 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:30.966+0000 7fdc4c071240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T07:56:31.056 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:31.056+0000 7f9463e78240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:56:31.223 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:31.222+0000 7fdc4c071240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T07:56:31.246 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:31.247+0000 7f9463e78240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:56:31.339 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:31.339+0000 7f9463e78240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:56:31.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:31.380+0000 7f2971600700 1 -- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 msgr2=0x7f294c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T07:56:31.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:31.380+0000 7f2971600700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 0x7f294c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T07:56:31.417 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:31.415+0000 7fdc4c071240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T07:56:31.510 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:31.509+0000 7fdc4c071240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T07:56:32.065 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:32.065+0000 7f9463e78240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:56:32.241 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:32.239+0000 7fdc4c071240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T07:56:32.275 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:32.276+0000 7f9463e78240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:56:32.453 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:32.452+0000 7fdc4c071240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T07:56:32.480 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:32.480+0000 7f9463e78240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:56:32.660 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:32.659+0000 7fdc4c071240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:56:32.953 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:32.954+0000 7f9463e78240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:56:33.137 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:33.136+0000 7fdc4c071240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T07:56:33.161 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:33.160+0000 7f9463e78240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:56:33.350 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:33.348+0000 7fdc4c071240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T07:56:33.603 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:33.603+0000 7f9463e78240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:56:33.715 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T07:56:33.714+0000 7f9463e78240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:56:33.789 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:33.789+0000 7fdc4c071240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T07:56:33.902 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:33.901+0000 7fdc4c071240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T07:56:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:33.911+0000 7f2969600700 1 -- 172.21.15.89:0/343418028 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 92) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f2958086df0 con 0x7f296c0cc880 2024-04-02T07:56:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:33.911+0000 7f2969600700 1 -- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 msgr2=0x7f294c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T07:56:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:33.911+0000 7f2969600700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 0x7f294c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:34.079 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:34.077+0000 7fdbb8400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:34.079 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:34.077+0000 7fdbb8400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:34.079 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:34.077+0000 7fdbb8400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:34.080 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:34.077+0000 7fdbb8400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:34.080 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:34.077+0000 7fdbb8400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:34.080 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:34.078+0000 7fdbb4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:34.080 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:34.078+0000 7fdbb4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:34.080 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:34.078+0000 7fdbb4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:34.080 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:34.078+0000 7fdbb4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:34.080 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:56:34.078+0000 7fdbb4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:56:34.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.919+0000 7f2969600700 1 -- 172.21.15.89:0/343418028 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 93) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f2958053e30 con 0x7f296c0cc880 2024-04-02T07:56:34.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.920+0000 7f2969600700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 0x7f294c06d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:34.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.920+0000 7f2969600700 1 -- 172.21.15.89:0/343418028 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f294c0038f0 con 0x7f294c06ae00 2024-04-02T07:56:34.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.921+0000 7f2971600700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 0x7f294c06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:34.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.924+0000 7f2971600700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 0x7f294c06d240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f296000bee0 tx=0x7f2960013000 comp rx=0 tx=0).ready entity=mgr.7544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:34.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.992+0000 7f2969600700 1 -- 172.21.15.89:0/343418028 <== mgr.7544 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f296000dc50 con 0x7f294c06ae00 2024-04-02T07:56:35.107 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path /snap_test_dir1 2024-04-02T07:56:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.995+0000 7f2974000700 1 -- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 msgr2=0x7f294c06d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.995+0000 7f2974000700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 0x7f294c06d240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f296000bee0 tx=0x7f2960013000 comp rx=0 tx=0).stop 2024-04-02T07:56:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.995+0000 7f2974000700 1 -- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f296c0cc880 msgr2=0x7f296c1510d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.995+0000 7f2974000700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f296c0cc880 0x7f296c1510d0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f295800c750 tx=0x7f295800ca60 comp rx=0 tx=0).stop 2024-04-02T07:56:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.995+0000 7f2974000700 1 -- 172.21.15.89:0/343418028 shutdown_connections 2024-04-02T07:56:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.995+0000 7f2974000700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f294c06ae00 0x7f294c06d240 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.995+0000 7f2974000700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f296c15dd90 0x7f296c151610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.995+0000 7f2974000700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f296c0b41d0 0x7f296c158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.995+0000 7f2974000700 1 --2- 172.21.15.89:0/343418028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f296c0cc880 0x7f296c1510d0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.995+0000 7f2974000700 1 -- 172.21.15.89:0/343418028 >> 172.21.15.89:0/343418028 conn(0x7f296c071fb0 msgr2=0x7f296c0736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:35.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.995+0000 7f2974000700 1 -- 172.21.15.89:0/343418028 shutdown_connections 2024-04-02T07:56:35.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:34.995+0000 7f2974000700 1 -- 172.21.15.89:0/343418028 wait complete. 2024-04-02T07:56:35.110 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2024-04-02T07:56:35.110 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2024-04-02T07:56:35.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.306+0000 7f177f800700 1 Processor -- start 2024-04-02T07:56:35.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.306+0000 7f177f800700 1 -- start start 2024-04-02T07:56:35.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.307+0000 7f177f800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f177815c700 0x7f17780adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:35.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.307+0000 7f177f800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f177806dd00 con 0x7f1778155ac0 2024-04-02T07:56:35.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.307+0000 7f177f800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f177806d0f0 con 0x7f17780ae4a0 2024-04-02T07:56:35.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.307+0000 7f177f800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f177806d3d0 con 0x7f177815c700 2024-04-02T07:56:35.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.307+0000 7f177ce00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1778155ac0 0x7f177806c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58588/0 (socket says 172.21.15.89:58588) 2024-04-02T07:56:35.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.307+0000 7f177ce00700 1 -- 172.21.15.89:0/1139388623 learned_addr learned my addr 172.21.15.89:0/1139388623 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:35.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777e00700 1 --2- 172.21.15.89:0/1139388623 >> v2:172.21.15.113:3300/0 conn(0x7f177815c700 0x7f17780adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 -- 172.21.15.89:0/1139388623 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921462345 0 0) 0x7f177806dd00 con 0x7f1778155ac0 2024-04-02T07:56:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 -- 172.21.15.89:0/1139388623 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17600035b0 con 0x7f1778155ac0 2024-04-02T07:56:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 -- 172.21.15.89:0/1139388623 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4083493282 0 0) 0x7f17600035b0 con 0x7f1778155ac0 2024-04-02T07:56:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 -- 172.21.15.89:0/1139388623 >> v2:172.21.15.113:3300/0 conn(0x7f177815c700 msgr2=0x7f17780adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 --2- 172.21.15.89:0/1139388623 >> v2:172.21.15.113:3300/0 conn(0x7f177815c700 0x7f17780adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 -- 172.21.15.89:0/1139388623 >> v1:172.21.15.113:6789/0 conn(0x7f17780ae4a0 legacy=0x7f177815ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 -- 172.21.15.89:0/1139388623 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f177806d6b0 con 0x7f1778155ac0 2024-04-02T07:56:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 -- 172.21.15.89:0/1139388623 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f176c001670 con 0x7f1778155ac0 2024-04-02T07:56:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 -- 172.21.15.89:0/1139388623 >> v1:172.21.15.89:6789/0 conn(0x7f1778155ac0 legacy=0x7f177806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 --2- 172.21.15.89:0/1139388623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1760003a00 0x7f1760003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 --2- 172.21.15.89:0/1139388623 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1760004850 0x7f1760004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 --2- 172.21.15.89:0/1139388623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1760005210 0x7f176000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:35.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 -- 172.21.15.89:0/1139388623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177806d3d0 con 0x7f1760003a00 2024-04-02T07:56:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 -- 172.21.15.89:0/1139388623 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176000c060 con 0x7f1760005210 2024-04-02T07:56:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.308+0000 7f1777400700 1 -- 172.21.15.89:0/1139388623 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176000c220 con 0x7f1760004850 2024-04-02T07:56:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.309+0000 7f177ce00700 1 --2- 172.21.15.89:0/1139388623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1760003a00 0x7f1760003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.309+0000 7f1777e00700 1 --2- 172.21.15.89:0/1139388623 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1760004850 0x7f1760004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.309+0000 7f177d800700 1 --2- 172.21.15.89:0/1139388623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1760005210 0x7f176000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.309+0000 7f177ce00700 1 -- 172.21.15.89:0/1139388623 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1760004850 msgr2=0x7f1760004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.309+0000 7f177ce00700 1 --2- 172.21.15.89:0/1139388623 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1760004850 0x7f1760004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.309+0000 7f177ce00700 1 -- 172.21.15.89:0/1139388623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1760005210 msgr2=0x7f176000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.309+0000 7f177ce00700 1 --2- 172.21.15.89:0/1139388623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1760005210 0x7f176000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.309+0000 7f177ce00700 1 -- 172.21.15.89:0/1139388623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17600035b0 con 0x7f1760003a00 2024-04-02T07:56:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.309+0000 7f177d800700 1 -- 172.21.15.89:0/1139388623 reap_dead start 2024-04-02T07:56:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.309+0000 7f177ce00700 1 --2- 172.21.15.89:0/1139388623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1760003a00 0x7f1760003de0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f176c00c210 tx=0x7f1760005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e618e5654098bc5a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:35.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.310+0000 7f1777400700 1 -- 172.21.15.89:0/1139388623 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c007b70 con 0x7f1760003a00 2024-04-02T07:56:35.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.310+0000 7f1777400700 1 -- 172.21.15.89:0/1139388623 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176c00cdb0 con 0x7f1760003a00 2024-04-02T07:56:35.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.310+0000 7f1777400700 1 -- 172.21.15.89:0/1139388623 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c005ba0 con 0x7f1760003a00 2024-04-02T07:56:35.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.310+0000 7f177f800700 1 -- 172.21.15.89:0/1139388623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1760003a00 msgr2=0x7f1760003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:35.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.310+0000 7f177f800700 1 --2- 172.21.15.89:0/1139388623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1760003a00 0x7f1760003de0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f176c00c210 tx=0x7f1760005880 comp rx=0 tx=0).stop 2024-04-02T07:56:35.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.310+0000 7f177f800700 1 -- 172.21.15.89:0/1139388623 shutdown_connections 2024-04-02T07:56:35.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.310+0000 7f177f800700 1 --2- 172.21.15.89:0/1139388623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1760003a00 0x7f1760003de0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.310+0000 7f177f800700 1 -- 172.21.15.89:0/1139388623 >> 172.21.15.89:0/1139388623 conn(0x7f17780cf210 msgr2=0x7f17780cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:35.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.310+0000 7f177f800700 1 -- 172.21.15.89:0/1139388623 shutdown_connections 2024-04-02T07:56:35.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.310+0000 7f177f800700 1 -- 172.21.15.89:0/1139388623 wait complete. 2024-04-02T07:56:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.312+0000 7f177f800700 1 Processor -- start 2024-04-02T07:56:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.312+0000 7f177f800700 1 -- start start 2024-04-02T07:56:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.313+0000 7f177f800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1760003a00 0x7f17781f15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.313+0000 7f177f800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17781f1b00 0x7f177814fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.313+0000 7f177f800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1778150130 0x7f177814dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.313+0000 7f177f800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1778070c90 con 0x7f17781f1b00 2024-04-02T07:56:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.313+0000 7f177f800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1778150660 con 0x7f1778150130 2024-04-02T07:56:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.313+0000 7f177f800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1778150940 con 0x7f1760003a00 2024-04-02T07:56:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.313+0000 7f1777e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17781f1b00 0x7f177814fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.313+0000 7f1777e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17781f1b00 0x7f177814fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39824/0 (socket says 172.21.15.89:39824) 2024-04-02T07:56:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.313+0000 7f1777e00700 1 -- 172.21.15.89:0/1689688137 learned_addr learned my addr 172.21.15.89:0/1689688137 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.313+0000 7f177ce00700 1 --2- 172.21.15.89:0/1689688137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1760003a00 0x7f17781f15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.314+0000 7f177d800700 1 --2- 172.21.15.89:0/1689688137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1778150130 0x7f177814dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.314+0000 7f1777e00700 1 -- 172.21.15.89:0/1689688137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1760003a00 msgr2=0x7f17781f15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.314+0000 7f1777e00700 1 --2- 172.21.15.89:0/1689688137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1760003a00 0x7f17781f15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.314+0000 7f1777e00700 1 -- 172.21.15.89:0/1689688137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1778150130 msgr2=0x7f177814dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.314+0000 7f1777e00700 1 --2- 172.21.15.89:0/1689688137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1778150130 0x7f177814dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.314+0000 7f1777e00700 1 -- 172.21.15.89:0/1689688137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1778156020 con 0x7f17781f1b00 2024-04-02T07:56:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.314+0000 7f1777e00700 1 --2- 172.21.15.89:0/1689688137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17781f1b00 0x7f177814fbf0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f1764011aa0 tx=0x7f1764011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.314+0000 7f1774c00700 1 -- 172.21.15.89:0/1689688137 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1764011ee0 con 0x7f17781f1b00 2024-04-02T07:56:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.314+0000 7f1774c00700 1 -- 172.21.15.89:0/1689688137 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176400f940 con 0x7f17781f1b00 2024-04-02T07:56:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.314+0000 7f1774c00700 1 -- 172.21.15.89:0/1689688137 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176400fd10 con 0x7f17781f1b00 2024-04-02T07:56:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.314+0000 7f177f800700 1 -- 172.21.15.89:0/1689688137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1778150c20 con 0x7f17781f1b00 2024-04-02T07:56:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.315+0000 7f177f800700 1 -- 172.21.15.89:0/1689688137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f177814e260 con 0x7f17781f1b00 2024-04-02T07:56:35.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.315+0000 7f177f800700 1 -- 172.21.15.89:0/1689688137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f177814b6e0 con 0x7f17781f1b00 2024-04-02T07:56:35.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.316+0000 7f1774c00700 1 -- 172.21.15.89:0/1689688137 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 93) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f1764014440 con 0x7f17781f1b00 2024-04-02T07:56:35.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.317+0000 7f1774c00700 1 --2- 172.21.15.89:0/1689688137 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f175806ab30 0x7f175806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:35.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.317+0000 7f1774c00700 1 -- 172.21.15.89:0/1689688137 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f1764089870 con 0x7f17781f1b00 2024-04-02T07:56:35.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.317+0000 7f177ce00700 1 --2- 172.21.15.89:0/1689688137 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f175806ab30 0x7f175806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:35.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.318+0000 7f177ce00700 1 --2- 172.21.15.89:0/1689688137 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f175806ab30 0x7f175806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f176c015040 tx=0x7f176c007000 comp rx=0 tx=0).ready entity=mgr.7544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:35.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.322+0000 7f1774c00700 1 -- 172.21.15.89:0/1689688137 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1764054ab0 con 0x7f17781f1b00 2024-04-02T07:56:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.547+0000 7f177f800700 1 -- 172.21.15.89:0/1689688137 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7f1778155ea0 con 0x7f175806ab30 2024-04-02T07:56:35.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.604+0000 7f1774c00700 1 -- 172.21.15.89:0/1689688137 <== mgr.7544 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7f176c023070 con 0x7f175806ab30 2024-04-02T07:56:35.605 INFO:teuthology.orchestra.run.smithi089.stdout:Retention added to path /snap_test_dir1 2024-04-02T07:56:35.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.606+0000 7f177f800700 1 -- 172.21.15.89:0/1689688137 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f175806ab30 msgr2=0x7f175806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:35.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.606+0000 7f177f800700 1 --2- 172.21.15.89:0/1689688137 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f175806ab30 0x7f175806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f176c015040 tx=0x7f176c007000 comp rx=0 tx=0).stop 2024-04-02T07:56:35.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.606+0000 7f177f800700 1 -- 172.21.15.89:0/1689688137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17781f1b00 msgr2=0x7f177814fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:35.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.606+0000 7f177f800700 1 --2- 172.21.15.89:0/1689688137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17781f1b00 0x7f177814fbf0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f1764011aa0 tx=0x7f1764011db0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.606+0000 7f177f800700 1 -- 172.21.15.89:0/1689688137 shutdown_connections 2024-04-02T07:56:35.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.606+0000 7f177f800700 1 --2- 172.21.15.89:0/1689688137 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f175806ab30 0x7f175806cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.606+0000 7f177f800700 1 --2- 172.21.15.89:0/1689688137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1778150130 0x7f177814dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.606+0000 7f177f800700 1 --2- 172.21.15.89:0/1689688137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1760003a00 0x7f17781f15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.606+0000 7f177f800700 1 --2- 172.21.15.89:0/1689688137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17781f1b00 0x7f177814fbf0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.606+0000 7f177f800700 1 -- 172.21.15.89:0/1689688137 >> 172.21.15.89:0/1689688137 conn(0x7f17780cf210 msgr2=0x7f177815e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:35.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.607+0000 7f177f800700 1 -- 172.21.15.89:0/1689688137 shutdown_connections 2024-04-02T07:56:35.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.607+0000 7f177f800700 1 -- 172.21.15.89:0/1689688137 wait complete. 2024-04-02T07:56:35.624 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2024-04-02T07:56:35.624 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-04-02T07:56:35.625 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2024-04-02T07:56:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.819+0000 7f550ce00700 1 Processor -- start 2024-04-02T07:56:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.819+0000 7f550ce00700 1 -- start start 2024-04-02T07:56:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.821+0000 7f550ce00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55080d16a0 0x7f55080d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.821+0000 7f550ce00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f55080d2370 0x7f55080ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.821+0000 7f550ce00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5508161420 con 0x7f550815d9d0 2024-04-02T07:56:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.821+0000 7f550ce00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55080d2820 con 0x7f55080d2370 2024-04-02T07:56:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.821+0000 7f550ce00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55080d4340 con 0x7f55080d16a0 2024-04-02T07:56:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.821+0000 7f5505400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55080d16a0 0x7f55080d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.821+0000 7f5505e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f550815d9d0 0x7f55080d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:60494/0 (socket says 172.21.15.89:60494) 2024-04-02T07:56:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.821+0000 7f5505e00700 1 -- 172.21.15.89:0/1254403233 learned_addr learned my addr 172.21.15.89:0/1254403233 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T07:56:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.821+0000 7f5506800700 1 --2- 172.21.15.89:0/1254403233 >> v2:172.21.15.113:3300/0 conn(0x7f55080d2370 0x7f55080ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.822+0000 7f5504a00700 1 -- 172.21.15.89:0/1254403233 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536529154 0 0) 0x7f5508161420 con 0x7f550815d9d0 2024-04-02T07:56:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.822+0000 7f5504a00700 1 -- 172.21.15.89:0/1254403233 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ec0035b0 con 0x7f550815d9d0 2024-04-02T07:56:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.822+0000 7f5506800700 1 -- 172.21.15.89:0/1254403233 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55080d16a0 msgr2=0x7f55080d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T07:56:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.822+0000 7f5506800700 1 --2- 172.21.15.89:0/1254403233 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55080d16a0 0x7f55080d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.822+0000 7f5506800700 1 -- 172.21.15.89:0/1254403233 >> v1:172.21.15.113:6789/0 conn(0x7f550815d9d0 legacy=0x7f55080d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.822+0000 7f5506800700 1 -- 172.21.15.89:0/1254403233 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5508161700 con 0x7f55080d2370 2024-04-02T07:56:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.822+0000 7f5505400700 1 --2- 172.21.15.89:0/1254403233 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55080d16a0 0x7f55080d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T07:56:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.822+0000 7f5506800700 1 --2- 172.21.15.89:0/1254403233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55080d2370 0x7f55080ce360 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5500009dc0 tx=0x7f550000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3a6fab9c09521c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.822+0000 7f5504a00700 1 -- 172.21.15.89:0/1254403233 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550000e070 con 0x7f55080d2370 2024-04-02T07:56:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.822+0000 7f5504a00700 1 -- 172.21.15.89:0/1254403233 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550000bae0 con 0x7f55080d2370 2024-04-02T07:56:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.822+0000 7f5504a00700 1 -- 172.21.15.89:0/1254403233 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550000be90 con 0x7f55080d2370 2024-04-02T07:56:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.823+0000 7f550ce00700 1 -- 172.21.15.89:0/1254403233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55080d2370 msgr2=0x7f55080ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.823+0000 7f550ce00700 1 --2- 172.21.15.89:0/1254403233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55080d2370 0x7f55080ce360 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5500009dc0 tx=0x7f550000b330 comp rx=0 tx=0).stop 2024-04-02T07:56:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.823+0000 7f550ce00700 1 -- 172.21.15.89:0/1254403233 shutdown_connections 2024-04-02T07:56:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.823+0000 7f550ce00700 1 --2- 172.21.15.89:0/1254403233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55080d2370 0x7f55080ce360 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.823+0000 7f550ce00700 1 --2- 172.21.15.89:0/1254403233 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55080d16a0 0x7f55080d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.823+0000 7f550ce00700 1 -- 172.21.15.89:0/1254403233 >> 172.21.15.89:0/1254403233 conn(0x7f550808aff0 msgr2=0x7f55080d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.823+0000 7f550ce00700 1 -- 172.21.15.89:0/1254403233 shutdown_connections 2024-04-02T07:56:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.823+0000 7f550ce00700 1 -- 172.21.15.89:0/1254403233 wait complete. 2024-04-02T07:56:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.825+0000 7f550ce00700 1 Processor -- start 2024-04-02T07:56:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.825+0000 7f550ce00700 1 -- start start 2024-04-02T07:56:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f550ce00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55080d16a0 0x7f550815eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f550ce00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55080d2370 0x7f55080ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:35.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f550ce00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f550815d9d0 0x7f55080cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:35.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f550ce00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550815e6c0 con 0x7f550815d9d0 2024-04-02T07:56:35.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f550ce00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550815ddb0 con 0x7f55080d2370 2024-04-02T07:56:35.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f550ce00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55080b7f10 con 0x7f55080d16a0 2024-04-02T07:56:35.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f5506800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f550815d9d0 0x7f55080cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:35.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f5506800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f550815d9d0 0x7f55080cd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39828/0 (socket says 172.21.15.89:39828) 2024-04-02T07:56:35.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f5506800700 1 -- 172.21.15.89:0/2106185712 learned_addr learned my addr 172.21.15.89:0/2106185712 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T07:56:35.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f5505e00700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55080d16a0 0x7f550815eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:35.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f5505400700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55080d2370 0x7f55080ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:35.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f5506800700 1 -- 172.21.15.89:0/2106185712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55080d16a0 msgr2=0x7f550815eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:35.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f5506800700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55080d16a0 0x7f550815eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f5506800700 1 -- 172.21.15.89:0/2106185712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55080d2370 msgr2=0x7f55080ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:35.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f5506800700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55080d2370 0x7f55080ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f5506800700 1 -- 172.21.15.89:0/2106185712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f4009430 con 0x7f550815d9d0 2024-04-02T07:56:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.827+0000 7f5505400700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55080d2370 0x7f55080ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T07:56:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.828+0000 7f5506800700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f550815d9d0 0x7f55080cd420 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f550808c230 tx=0x7f5500004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.828+0000 7f54fe000700 1 -- 172.21.15.89:0/2106185712 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5500004680 con 0x7f550815d9d0 2024-04-02T07:56:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.828+0000 7f54fe000700 1 -- 172.21.15.89:0/2106185712 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5500020070 con 0x7f550815d9d0 2024-04-02T07:56:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.828+0000 7f54fe000700 1 -- 172.21.15.89:0/2106185712 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5500027480 con 0x7f550815d9d0 2024-04-02T07:56:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.828+0000 7f550ce00700 1 -- 172.21.15.89:0/2106185712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5500009a70 con 0x7f550815d9d0 2024-04-02T07:56:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.828+0000 7f550ce00700 1 -- 172.21.15.89:0/2106185712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f550815e2e0 con 0x7f550815d9d0 2024-04-02T07:56:35.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.829+0000 7f550ce00700 1 -- 172.21.15.89:0/2106185712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55080d1ca0 con 0x7f550815d9d0 2024-04-02T07:56:35.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.830+0000 7f54fe000700 1 -- 172.21.15.89:0/2106185712 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 93) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f5500012430 con 0x7f550815d9d0 2024-04-02T07:56:35.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.831+0000 7f54fe000700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f54e806ab30 0x7f54e806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T07:56:35.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.831+0000 7f54fe000700 1 -- 172.21.15.89:0/2106185712 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f5500022070 con 0x7f550815d9d0 2024-04-02T07:56:35.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.831+0000 7f5505e00700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f54e806ab30 0x7f54e806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T07:56:35.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.832+0000 7f5505e00700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f54e806ab30 0x7f54e806cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f54f80011e0 tx=0x7f54f8009040 comp rx=0 tx=0).ready entity=mgr.7544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T07:56:35.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.835+0000 7f54fe000700 1 -- 172.21.15.89:0/2106185712 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5500059c80 con 0x7f550815d9d0 2024-04-02T07:56:35.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:35.927+0000 7f54fe000700 1 -- 172.21.15.89:0/2106185712 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 94) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f55000565c0 con 0x7f550815d9d0 2024-04-02T07:56:36.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:36.074+0000 7f550ce00700 1 -- 172.21.15.89:0/2106185712 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5508161fa0 con 0x7f54e806ab30 2024-04-02T07:56:36.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:36.081+0000 7f54fe000700 1 -- 172.21.15.89:0/2106185712 <== mgr.7544 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7f54f80013b0 con 0x7f54e806ab30 2024-04-02T07:56:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T07:56:36.082 INFO:teuthology.orchestra.run.smithi089.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-04-02T07:56:36.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:36.084+0000 7f550ce00700 1 -- 172.21.15.89:0/2106185712 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f54e806ab30 msgr2=0x7f54e806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:36.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:36.084+0000 7f550ce00700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f54e806ab30 0x7f54e806cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f54f80011e0 tx=0x7f54f8009040 comp rx=0 tx=0).stop 2024-04-02T07:56:36.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:36.084+0000 7f550ce00700 1 -- 172.21.15.89:0/2106185712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f550815d9d0 msgr2=0x7f55080cd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T07:56:36.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:36.084+0000 7f550ce00700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f550815d9d0 0x7f55080cd420 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f550808c230 tx=0x7f5500004ad0 comp rx=0 tx=0).stop 2024-04-02T07:56:36.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:36.084+0000 7f550ce00700 1 -- 172.21.15.89:0/2106185712 shutdown_connections 2024-04-02T07:56:36.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:36.084+0000 7f550ce00700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f54e806ab30 0x7f54e806cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:36.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:36.084+0000 7f550ce00700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f550815d9d0 0x7f55080cd420 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:36.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:36.084+0000 7f550ce00700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55080d16a0 0x7f550815eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:36.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:36.084+0000 7f550ce00700 1 --2- 172.21.15.89:0/2106185712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55080d2370 0x7f55080ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T07:56:36.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:36.084+0000 7f550ce00700 1 -- 172.21.15.89:0/2106185712 >> 172.21.15.89:0/2106185712 conn(0x7f550808aff0 msgr2=0x7f55080b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T07:56:36.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:36.084+0000 7f550ce00700 1 -- 172.21.15.89:0/2106185712 shutdown_connections 2024-04-02T07:56:36.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T07:56:36.084+0000 7f550ce00700 1 -- 172.21.15.89:0/2106185712 wait complete. 2024-04-02T07:56:36.102 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-04-02T07:56:36.102 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2024-04-02T07:56:47.832 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:56:47.837 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:57:00.617 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:57:00.617+0000 7fdba3000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:57:00.617 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:57:00.617+0000 7fdba3000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:57:00.617 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:57:00.617+0000 7fdba3000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:57:00.617 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:57:00.617+0000 7fdba3000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:57:00.617 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T07:57:00.617+0000 7fdba3000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T07:57:17.943 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:57:17.949 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:57:48.053 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:57:48.059 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:58:18.167 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:58:18.173 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:58:48.281 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:58:48.289 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:59:18.399 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:59:18.404 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:59:48.510 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T07:59:48.515 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:00:18.780 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:00:18.786 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:00:48.925 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:00:48.930 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:01:19.036 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:01:19.042 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:01:49.299 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:01:49.304 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:02:19.428 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:02:19.433 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:02:49.547 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:02:49.552 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:03:19.658 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:03:19.664 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:03:49.788 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:03:49.794 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:04:20.149 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:04:20.155 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:04:50.255 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:04:50.260 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:05:20.372 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:05:20.378 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:05:50.484 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:05:50.489 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:06:20.601 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:06:20.607 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:06:50.941 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:06:50.947 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:07:21.052 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:07:21.057 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:07:51.173 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:07:51.178 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:08:21.277 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:08:21.282 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:08:51.392 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:08:51.397 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:09:21.505 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:09:21.511 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:09:51.619 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:09:51.624 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:10:21.733 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:10:21.739 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:10:52.093 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:10:52.098 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:11:22.203 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:11:22.209 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:11:52.507 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:11:52.513 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:12:22.661 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:12:22.666 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:12:52.774 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:12:52.825 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:13:22.936 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:13:22.941 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:13:53.048 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:13:53.053 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:14:23.162 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:14:23.167 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:14:53.284 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:14:53.290 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:15:23.391 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:15:23.396 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:15:53.503 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:15:53.508 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:16:23.617 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:16:23.622 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:16:53.735 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:16:53.741 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:17:23.847 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:17:23.852 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:17:53.959 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:17:53.964 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:18:24.070 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:18:24.075 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:18:54.182 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:18:54.188 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:19:24.302 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:19:24.308 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:19:54.672 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:19:54.678 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:20:24.934 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:20:24.939 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:20:55.070 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:20:55.102 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:21:25.293 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:21:25.299 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:21:55.427 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:21:55.433 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:22:25.697 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:22:25.703 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:22:55.809 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:22:55.815 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:23:25.929 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:23:25.935 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:23:56.042 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:23:56.047 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:24:26.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:24:26.157 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:24:56.264 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:24:56.269 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:25:26.392 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:25:26.397 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:25:56.503 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:25:56.509 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:26:26.616 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:26:26.621 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:26:56.976 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:26:56.981 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:27:27.406 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:27:27.412 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:27:57.526 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:27:57.531 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:28:27.641 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:28:27.647 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:28:57.758 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:28:57.764 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:29:27.870 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:29:27.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:29:57.974 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:29:57.979 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:30:28.152 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:30:28.158 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:30:58.539 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:30:58.545 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:31:28.641 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:31:28.646 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:31:58.754 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:31:58.759 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:32:28.868 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:32:28.874 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:32:58.985 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:32:58.990 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:33:29.104 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:33:29.110 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:33:59.234 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:33:59.239 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:34:29.513 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:34:29.518 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:34:59.850 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:34:59.856 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:35:29.985 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:35:29.991 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:36:00.139 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:36:00.145 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:36:30.443 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:36:30.449 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:37:00.604 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:37:00.609 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:37:30.790 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:37:30.795 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:38:00.919 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:38:00.923 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:38:31.306 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:38:31.311 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:39:01.769 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:39:01.774 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:39:31.895 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:39:31.901 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:40:02.032 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:40:02.037 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:40:32.202 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:40:32.207 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:41:02.556 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:41:02.562 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:41:32.955 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:41:32.960 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:42:03.070 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:42:03.075 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:42:33.182 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:42:33.188 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:43:03.303 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:43:03.309 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:43:33.415 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:43:33.420 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:44:03.551 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:44:03.556 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:44:33.663 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:44:33.668 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:45:03.783 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:45:03.789 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:45:33.927 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:45:33.933 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:46:04.059 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:46:04.064 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:46:34.268 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:46:34.273 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:47:04.421 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:47:04.426 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:47:34.533 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:47:34.539 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:48:04.647 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:48:04.652 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:48:34.797 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:48:34.802 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:49:04.926 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:49:04.932 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:49:35.053 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:49:35.059 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:50:05.174 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:50:05.180 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:50:35.294 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:50:35.299 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:51:05.644 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:51:05.649 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:51:35.762 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:51:35.767 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:52:05.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:52:05.888 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:52:16.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:16.103 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T08:52:16.159 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_58_00_UTC 2024-04-02T08:52:16.159 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_59_00_UTC 2024-04-02T08:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_00_00_UTC 2024-04-02T08:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_01_00_UTC 2024-04-02T08:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_02_01_UTC 2024-04-02T08:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_03_00_UTC 2024-04-02T08:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_04_00_UTC 2024-04-02T08:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_05_00_UTC 2024-04-02T08:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_06_00_UTC 2024-04-02T08:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_07_00_UTC 2024-04-02T08:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_08_00_UTC 2024-04-02T08:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_09_00_UTC 2024-04-02T08:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_10_00_UTC 2024-04-02T08:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_11_00_UTC 2024-04-02T08:52:16.160 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_12_00_UTC 2024-04-02T08:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_13_00_UTC 2024-04-02T08:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_14_00_UTC 2024-04-02T08:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_15_00_UTC 2024-04-02T08:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_16_00_UTC 2024-04-02T08:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_17_00_UTC 2024-04-02T08:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_18_00_UTC 2024-04-02T08:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_19_00_UTC 2024-04-02T08:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_20_00_UTC 2024-04-02T08:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_21_00_UTC 2024-04-02T08:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_22_00_UTC 2024-04-02T08:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_23_00_UTC 2024-04-02T08:52:16.161 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_24_00_UTC 2024-04-02T08:52:16.162 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_25_00_UTC 2024-04-02T08:52:16.162 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_26_00_UTC 2024-04-02T08:52:16.162 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_27_00_UTC 2024-04-02T08:52:16.162 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_28_00_UTC 2024-04-02T08:52:16.162 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_29_00_UTC 2024-04-02T08:52:16.162 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_30_00_UTC 2024-04-02T08:52:16.162 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_31_00_UTC 2024-04-02T08:52:16.162 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_32_00_UTC 2024-04-02T08:52:16.162 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_33_00_UTC 2024-04-02T08:52:16.162 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_34_00_UTC 2024-04-02T08:52:16.162 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_35_00_UTC 2024-04-02T08:52:16.162 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_36_00_UTC 2024-04-02T08:52:16.162 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_37_00_UTC 2024-04-02T08:52:16.163 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_38_00_UTC 2024-04-02T08:52:16.163 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_39_00_UTC 2024-04-02T08:52:16.163 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_40_00_UTC 2024-04-02T08:52:16.163 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_41_00_UTC 2024-04-02T08:52:16.163 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_42_00_UTC 2024-04-02T08:52:16.163 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_43_00_UTC 2024-04-02T08:52:16.163 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_44_00_UTC 2024-04-02T08:52:16.163 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_45_00_UTC 2024-04-02T08:52:16.163 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_46_00_UTC 2024-04-02T08:52:16.163 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_47_00_UTC 2024-04-02T08:52:16.163 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_48_00_UTC 2024-04-02T08:52:16.163 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_49_00_UTC 2024-04-02T08:52:16.163 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_50_00_UTC 2024-04-02T08:52:16.163 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_51_00_UTC 2024-04-02T08:52:16.164 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_52_00_UTC 2024-04-02T08:52:16.165 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-02-07_58_00_UTC', 'scheduled-2024-04-02-07_59_00_UTC', 'scheduled-2024-04-02-08_00_00_UTC', 'scheduled-2024-04-02-08_01_00_UTC', 'scheduled-2024-04-02-08_02_01_UTC', 'scheduled-2024-04-02-08_03_00_UTC', 'scheduled-2024-04-02-08_04_00_UTC', 'scheduled-2024-04-02-08_05_00_UTC', 'scheduled-2024-04-02-08_06_00_UTC', 'scheduled-2024-04-02-08_07_00_UTC', 'scheduled-2024-04-02-08_08_00_UTC', 'scheduled-2024-04-02-08_09_00_UTC', 'scheduled-2024-04-02-08_10_00_UTC', 'scheduled-2024-04-02-08_11_00_UTC', 'scheduled-2024-04-02-08_12_00_UTC', 'scheduled-2024-04-02-08_13_00_UTC', 'scheduled-2024-04-02-08_14_00_UTC', 'scheduled-2024-04-02-08_15_00_UTC', 'scheduled-2024-04-02-08_16_00_UTC', 'scheduled-2024-04-02-08_17_00_UTC', 'scheduled-2024-04-02-08_18_00_UTC', 'scheduled-2024-04-02-08_19_00_UTC', 'scheduled-2024-04-02-08_20_00_UTC', 'scheduled-2024-04-02-08_21_00_UTC', 'scheduled-2024-04-02-08_22_00_UTC', 'scheduled-2024-04-02-08_23_00_UTC', 'scheduled-2024-04-02-08_24_00_UTC', 'scheduled-2024-04-02-08_25_00_UTC', 'scheduled-2024-04-02-08_26_00_UTC', 'scheduled-2024-04-02-08_27_00_UTC', 'scheduled-2024-04-02-08_28_00_UTC', 'scheduled-2024-04-02-08_29_00_UTC', 'scheduled-2024-04-02-08_30_00_UTC', 'scheduled-2024-04-02-08_31_00_UTC', 'scheduled-2024-04-02-08_32_00_UTC', 'scheduled-2024-04-02-08_33_00_UTC', 'scheduled-2024-04-02-08_34_00_UTC', 'scheduled-2024-04-02-08_35_00_UTC', 'scheduled-2024-04-02-08_36_00_UTC', 'scheduled-2024-04-02-08_37_00_UTC', 'scheduled-2024-04-02-08_38_00_UTC', 'scheduled-2024-04-02-08_39_00_UTC', 'scheduled-2024-04-02-08_40_00_UTC', 'scheduled-2024-04-02-08_41_00_UTC', 'scheduled-2024-04-02-08_42_00_UTC', 'scheduled-2024-04-02-08_43_00_UTC', 'scheduled-2024-04-02-08_44_00_UTC', 'scheduled-2024-04-02-08_45_00_UTC', 'scheduled-2024-04-02-08_46_00_UTC', 'scheduled-2024-04-02-08_47_00_UTC', 'scheduled-2024-04-02-08_48_00_UTC', 'scheduled-2024-04-02-08_49_00_UTC', 'scheduled-2024-04-02-08_50_00_UTC', 'scheduled-2024-04-02-08_51_00_UTC', 'scheduled-2024-04-02-08_52_00_UTC'] 2024-04-02T08:52:16.165 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2024-04-02T08:52:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.362+0000 7fc516000700 1 Processor -- start 2024-04-02T08:52:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.363+0000 7fc516000700 1 -- start start 2024-04-02T08:52:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.364+0000 7fc516000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc510099a60 0x7fc5100d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.364+0000 7fc516000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc51009a750 0x7fc51006c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.364+0000 7fc516000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5100d0970 con 0x7fc51015d990 2024-04-02T08:52:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.364+0000 7fc516000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc51006c020 con 0x7fc51009a750 2024-04-02T08:52:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.364+0000 7fc516000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100c81a0 con 0x7fc510099a60 2024-04-02T08:52:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.364+0000 7fc50fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc51009a750 0x7fc51006c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.364+0000 7fc50ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc51015d990 0x7fc510099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33204/0 (socket says 172.21.15.89:33204) 2024-04-02T08:52:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.364+0000 7fc50ea00700 1 -- 172.21.15.89:0/115344938 learned_addr learned my addr 172.21.15.89:0/115344938 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.364+0000 7fc50f400700 1 --2- 172.21.15.89:0/115344938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc510099a60 0x7fc5100d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.364+0000 7fc50fe00700 1 -- 172.21.15.89:0/115344938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc510099a60 msgr2=0x7fc5100d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:16.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.364+0000 7fc50fe00700 1 --2- 172.21.15.89:0/115344938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc510099a60 0x7fc5100d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:16.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.364+0000 7fc50fe00700 1 -- 172.21.15.89:0/115344938 >> v1:172.21.15.89:6789/0 conn(0x7fc51015d990 legacy=0x7fc510099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:16.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.364+0000 7fc50fe00700 1 -- 172.21.15.89:0/115344938 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5100d0690 con 0x7fc51009a750 2024-04-02T08:52:16.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.364+0000 7fc50e000700 1 -- 172.21.15.89:0/115344938 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479427847 0 0) 0x7fc5100d0970 con 0x7fc51015d990 2024-04-02T08:52:16.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.365+0000 7fc50fe00700 1 --2- 172.21.15.89:0/115344938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc51009a750 0x7fc51006c4b0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fc508009d90 tx=0x7fc50800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71373e597ee18ebf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:16.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.365+0000 7fc50e000700 1 -- 172.21.15.89:0/115344938 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50800e070 con 0x7fc51009a750 2024-04-02T08:52:16.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.365+0000 7fc50e000700 1 -- 172.21.15.89:0/115344938 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50800bbd0 con 0x7fc51009a750 2024-04-02T08:52:16.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.365+0000 7fc50e000700 1 -- 172.21.15.89:0/115344938 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc508012430 con 0x7fc51009a750 2024-04-02T08:52:16.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.365+0000 7fc516000700 1 -- 172.21.15.89:0/115344938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc51009a750 msgr2=0x7fc51006c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:16.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.365+0000 7fc516000700 1 --2- 172.21.15.89:0/115344938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc51009a750 0x7fc51006c4b0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fc508009d90 tx=0x7fc50800b330 comp rx=0 tx=0).stop 2024-04-02T08:52:16.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.365+0000 7fc516000700 1 -- 172.21.15.89:0/115344938 shutdown_connections 2024-04-02T08:52:16.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.365+0000 7fc516000700 1 --2- 172.21.15.89:0/115344938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc51009a750 0x7fc51006c4b0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:16.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.365+0000 7fc516000700 1 --2- 172.21.15.89:0/115344938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc510099a60 0x7fc5100d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:16.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.366+0000 7fc516000700 1 -- 172.21.15.89:0/115344938 >> 172.21.15.89:0/115344938 conn(0x7fc5100b2e20 msgr2=0x7fc5100b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:16.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.366+0000 7fc516000700 1 -- 172.21.15.89:0/115344938 shutdown_connections 2024-04-02T08:52:16.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.366+0000 7fc516000700 1 -- 172.21.15.89:0/115344938 wait complete. 2024-04-02T08:52:16.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.367+0000 7fc516000700 1 Processor -- start 2024-04-02T08:52:16.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.368+0000 7fc516000700 1 -- start start 2024-04-02T08:52:16.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.368+0000 7fc516000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc510099a60 0x7fc51015e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:16.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc516000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc51009a750 0x7fc51015faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:16.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc516000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc51015d990 0x7fc510160030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:16.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc516000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100c62d0 con 0x7fc51009a750 2024-04-02T08:52:16.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc516000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51015ecf0 con 0x7fc51015d990 2024-04-02T08:52:16.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc516000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc51015ef70 con 0x7fc510099a60 2024-04-02T08:52:16.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc50ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc51009a750 0x7fc51015faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:16.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc50ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc51009a750 0x7fc51015faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55338/0 (socket says 172.21.15.89:55338) 2024-04-02T08:52:16.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc50ea00700 1 -- 172.21.15.89:0/1662668107 learned_addr learned my addr 172.21.15.89:0/1662668107 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:16.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc50f400700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc510099a60 0x7fc51015e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:16.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc50fe00700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc51015d990 0x7fc510160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:16.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc50ea00700 1 -- 172.21.15.89:0/1662668107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc510099a60 msgr2=0x7fc51015e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:16.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc50ea00700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc510099a60 0x7fc51015e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:16.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc50ea00700 1 -- 172.21.15.89:0/1662668107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc51015d990 msgr2=0x7fc510160030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:16.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc50ea00700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc51015d990 0x7fc510160030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:16.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc50ea00700 1 -- 172.21.15.89:0/1662668107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc508009a70 con 0x7fc51009a750 2024-04-02T08:52:16.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc50fe00700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc51015d990 0x7fc510160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:52:16.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.369+0000 7fc50ea00700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc51009a750 0x7fc51015faf0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00c550 tx=0x7fc4fc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:16.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.370+0000 7fc507400700 1 -- 172.21.15.89:0/1662668107 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc004c80 con 0x7fc51009a750 2024-04-02T08:52:16.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.370+0000 7fc507400700 1 -- 172.21.15.89:0/1662668107 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4fc004de0 con 0x7fc51009a750 2024-04-02T08:52:16.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.370+0000 7fc507400700 1 -- 172.21.15.89:0/1662668107 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc0072d0 con 0x7fc51009a750 2024-04-02T08:52:16.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.370+0000 7fc516000700 1 -- 172.21.15.89:0/1662668107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51015f250 con 0x7fc51009a750 2024-04-02T08:52:16.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.370+0000 7fc516000700 1 -- 172.21.15.89:0/1662668107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5100c8f80 con 0x7fc51009a750 2024-04-02T08:52:16.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.371+0000 7fc516000700 1 -- 172.21.15.89:0/1662668107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5100c6480 con 0x7fc51009a750 2024-04-02T08:52:16.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.372+0000 7fc507400700 1 -- 172.21.15.89:0/1662668107 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 97) v1 ==== 89268+0+0 (secure 0 0 0) 0x7fc4fc007430 con 0x7fc51009a750 2024-04-02T08:52:16.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.372+0000 7fc507400700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4f006ae00 0x7fc4f006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:16.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.372+0000 7fc507400700 1 -- 172.21.15.89:0/1662668107 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fc4fc0895c0 con 0x7fc51009a750 2024-04-02T08:52:16.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.373+0000 7fc50f400700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4f006ae00 0x7fc4f006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:16.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.373+0000 7fc50f400700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4f006ae00 0x7fc4f006d2c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc50000b9b0 tx=0x7fc500008040 comp rx=0 tx=0).ready entity=mgr.7544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:16.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.377+0000 7fc507400700 1 -- 172.21.15.89:0/1662668107 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc5100c6480 con 0x7fc51009a750 2024-04-02T08:52:16.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.604+0000 7fc516000700 1 -- 172.21.15.89:0/1662668107 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5100c5de0 con 0x7fc4f006ae00 2024-04-02T08:52:16.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.612+0000 7fc507400700 1 -- 172.21.15.89:0/1662668107 <== mgr.7544 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+373 (secure 0 0 0) 0x7fc50000be80 con 0x7fc4f006ae00 2024-04-02T08:52:16.614 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:16.615 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:56:34", "first": "2024-04-02T07:57:00", "last": "2024-04-02T08:52:00", "last_pruned": "2024-04-02T08:52:01", "created_count": 56, "pruned_count": 1, "active": true}] 2024-04-02T08:52:16.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.615+0000 7fc516000700 1 -- 172.21.15.89:0/1662668107 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4f006ae00 msgr2=0x7fc4f006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.615+0000 7fc516000700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4f006ae00 0x7fc4f006d2c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc50000b9b0 tx=0x7fc500008040 comp rx=0 tx=0).stop 2024-04-02T08:52:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.615+0000 7fc516000700 1 -- 172.21.15.89:0/1662668107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc51009a750 msgr2=0x7fc51015faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.615+0000 7fc516000700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc51009a750 0x7fc51015faf0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00c550 tx=0x7fc4fc00c910 comp rx=0 tx=0).stop 2024-04-02T08:52:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.615+0000 7fc516000700 1 -- 172.21.15.89:0/1662668107 shutdown_connections 2024-04-02T08:52:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.615+0000 7fc516000700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4f006ae00 0x7fc4f006d2c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.615+0000 7fc516000700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc51015d990 0x7fc510160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.615+0000 7fc516000700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc510099a60 0x7fc51015e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:16.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.615+0000 7fc516000700 1 --2- 172.21.15.89:0/1662668107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc51009a750 0x7fc51015faf0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:16.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.615+0000 7fc516000700 1 -- 172.21.15.89:0/1662668107 >> 172.21.15.89:0/1662668107 conn(0x7fc5100b2e20 msgr2=0x7fc5100cd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:16.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.615+0000 7fc516000700 1 -- 172.21.15.89:0/1662668107 shutdown_connections 2024-04-02T08:52:16.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.615+0000 7fc516000700 1 -- 172.21.15.89:0/1662668107 wait complete. 2024-04-02T08:52:16.633 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T07:56:34", "first": "2024-04-02T07:57:00", "last": "2024-04-02T08:52:00", "last_pruned": "2024-04-02T08:52:01", "created_count": 56, "pruned_count": 1, "active": true}] 2024-04-02T08:52:16.633 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2024-04-02T00:00:00', 'created': '2024-04-02T07:56:34', 'first': '2024-04-02T07:57:00', 'last': '2024-04-02T08:52:00', 'last_pruned': '2024-04-02T08:52:01', 'created_count': 56, 'pruned_count': 1, 'active': True} 2024-04-02T08:52:16.633 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2024-04-02T08:52:16.633 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2024-04-02T08:52:16.634 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2024-04-02T08:52:16.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.828+0000 7fac9e600700 1 Processor -- start 2024-04-02T08:52:16.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.828+0000 7fac9e600700 1 -- start start 2024-04-02T08:52:16.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.829+0000 7fac9e600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fac980d2540 0x7fac981687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:16.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.830+0000 7fac9e600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac98161420 con 0x7fac980d16a0 2024-04-02T08:52:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.830+0000 7fac9e600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac98161700 con 0x7fac9815d9d0 2024-04-02T08:52:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.830+0000 7fac9e600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac980d4340 con 0x7fac980d2540 2024-04-02T08:52:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.830+0000 7fac97e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fac980d2540 0x7fac981687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.830+0000 7fac97e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fac980d2540 0x7fac981687b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55354/0 (socket says 172.21.15.89:55354) 2024-04-02T08:52:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.830+0000 7fac97e00700 1 -- 172.21.15.89:0/2491890287 learned_addr learned my addr 172.21.15.89:0/2491890287 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.830+0000 7fac8fe00700 1 --1- 172.21.15.89:0/2491890287 >> v1:172.21.15.89:6789/0 conn(0x7fac980d16a0 0x7fac980d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33220/0 (socket says 172.21.15.89:33220) 2024-04-02T08:52:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.830+0000 7fac97e00700 1 -- 172.21.15.89:0/2491890287 >> v1:172.21.15.113:6789/0 conn(0x7fac9815d9d0 legacy=0x7fac980d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.830+0000 7fac97e00700 1 -- 172.21.15.89:0/2491890287 >> v1:172.21.15.89:6789/0 conn(0x7fac980d16a0 legacy=0x7fac980d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.830+0000 7fac97e00700 1 -- 172.21.15.89:0/2491890287 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac981619e0 con 0x7fac980d2540 2024-04-02T08:52:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.830+0000 7fac96a00700 1 -- 172.21.15.89:0/2491890287 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518405263 0 0) 0x7fac98161420 con 0x7fac980d16a0 2024-04-02T08:52:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.831+0000 7fac97e00700 1 --2- 172.21.15.89:0/2491890287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac980d2540 0x7fac981687b0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fac90009e50 tx=0x7fac9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2881c2721c0337cd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.831+0000 7fac96a00700 1 -- 172.21.15.89:0/2491890287 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9000e070 con 0x7fac980d2540 2024-04-02T08:52:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.831+0000 7fac96a00700 1 -- 172.21.15.89:0/2491890287 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9000b9c0 con 0x7fac980d2540 2024-04-02T08:52:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.831+0000 7fac96a00700 1 -- 172.21.15.89:0/2491890287 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9000bd70 con 0x7fac980d2540 2024-04-02T08:52:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.831+0000 7fac9e600700 1 -- 172.21.15.89:0/2491890287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac980d2540 msgr2=0x7fac981687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.831+0000 7fac9e600700 1 --2- 172.21.15.89:0/2491890287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac980d2540 0x7fac981687b0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fac90009e50 tx=0x7fac9000b330 comp rx=0 tx=0).stop 2024-04-02T08:52:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.831+0000 7fac9e600700 1 -- 172.21.15.89:0/2491890287 shutdown_connections 2024-04-02T08:52:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.831+0000 7fac9e600700 1 --2- 172.21.15.89:0/2491890287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac980d2540 0x7fac981687b0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.831+0000 7fac9e600700 1 -- 172.21.15.89:0/2491890287 >> 172.21.15.89:0/2491890287 conn(0x7fac9808aff0 msgr2=0x7fac980d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.831+0000 7fac9e600700 1 -- 172.21.15.89:0/2491890287 shutdown_connections 2024-04-02T08:52:16.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.832+0000 7fac9e600700 1 -- 172.21.15.89:0/2491890287 wait complete. 2024-04-02T08:52:16.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.833+0000 7fac9e600700 1 Processor -- start 2024-04-02T08:52:16.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.833+0000 7fac9e600700 1 -- start start 2024-04-02T08:52:16.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.834+0000 7fac9e600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac980d16a0 0x7fac980cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:16.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.834+0000 7fac9e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac980d2540 0x7fac980b89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:16.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.834+0000 7fac9e600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac9815d9d0 0x7fac981f6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:16.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.834+0000 7fac9e600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac980cca20 con 0x7fac9815d9d0 2024-04-02T08:52:16.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.834+0000 7fac9e600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac980cbee0 con 0x7fac980d16a0 2024-04-02T08:52:16.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.834+0000 7fac9e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac980b8ef0 con 0x7fac980d2540 2024-04-02T08:52:16.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.834+0000 7fac97e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac9815d9d0 0x7fac981f6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:16.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.835+0000 7fac97e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac9815d9d0 0x7fac981f6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55368/0 (socket says 172.21.15.89:55368) 2024-04-02T08:52:16.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.835+0000 7fac97e00700 1 -- 172.21.15.89:0/3971189010 learned_addr learned my addr 172.21.15.89:0/3971189010 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:16.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.835+0000 7fac8fe00700 1 --2- 172.21.15.89:0/3971189010 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac980d2540 0x7fac980b89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:16.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.835+0000 7fac97400700 1 --2- 172.21.15.89:0/3971189010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac980d16a0 0x7fac980cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:16.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.835+0000 7fac97e00700 1 -- 172.21.15.89:0/3971189010 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac980d2540 msgr2=0x7fac980b89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:16.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.835+0000 7fac97e00700 1 --2- 172.21.15.89:0/3971189010 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac980d2540 0x7fac980b89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:16.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.835+0000 7fac97e00700 1 -- 172.21.15.89:0/3971189010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac980d16a0 msgr2=0x7fac980cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:16.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.835+0000 7fac97e00700 1 --2- 172.21.15.89:0/3971189010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac980d16a0 0x7fac980cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:16.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.835+0000 7fac97e00700 1 -- 172.21.15.89:0/3971189010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac90009b30 con 0x7fac9815d9d0 2024-04-02T08:52:16.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.835+0000 7fac97e00700 1 --2- 172.21.15.89:0/3971189010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac9815d9d0 0x7fac981f6950 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fac9808c230 tx=0x7fac90004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:16.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.835+0000 7fac8f400700 1 -- 172.21.15.89:0/3971189010 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9001c640 con 0x7fac9815d9d0 2024-04-02T08:52:16.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.835+0000 7fac8f400700 1 -- 172.21.15.89:0/3971189010 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9001c7c0 con 0x7fac9815d9d0 2024-04-02T08:52:16.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.835+0000 7fac8f400700 1 -- 172.21.15.89:0/3971189010 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9001cb70 con 0x7fac9815d9d0 2024-04-02T08:52:16.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.835+0000 7fac9e600700 1 -- 172.21.15.89:0/3971189010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac980cc180 con 0x7fac9815d9d0 2024-04-02T08:52:16.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.836+0000 7fac9e600700 1 -- 172.21.15.89:0/3971189010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac980cc740 con 0x7fac9815d9d0 2024-04-02T08:52:16.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.837+0000 7fac9e600700 1 -- 172.21.15.89:0/3971189010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac980d1ca0 con 0x7fac9815d9d0 2024-04-02T08:52:16.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.837+0000 7fac8f400700 1 -- 172.21.15.89:0/3971189010 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 97) v1 ==== 89268+0+0 (secure 0 0 0) 0x7fac9001ccd0 con 0x7fac9815d9d0 2024-04-02T08:52:16.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.839+0000 7fac8f400700 1 --2- 172.21.15.89:0/3971189010 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac7406ae00 0x7fac7406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:16.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.839+0000 7fac8f400700 1 -- 172.21.15.89:0/3971189010 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fac9008b3b0 con 0x7fac9815d9d0 2024-04-02T08:52:16.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.839+0000 7fac97400700 1 --2- 172.21.15.89:0/3971189010 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac7406ae00 0x7fac7406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:16.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.840+0000 7fac97400700 1 --2- 172.21.15.89:0/3971189010 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac7406ae00 0x7fac7406d2c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fac88000f60 tx=0x7fac88009040 comp rx=0 tx=0).ready entity=mgr.7544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:16.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:16.843+0000 7fac8f400700 1 -- 172.21.15.89:0/3971189010 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac90056060 con 0x7fac9815d9d0 2024-04-02T08:52:17.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:17.073+0000 7fac9e600700 1 -- 172.21.15.89:0/3971189010 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fac98161fa0 con 0x7fac7406ae00 2024-04-02T08:52:17.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:17.146+0000 7fac8f400700 1 -- 172.21.15.89:0/3971189010 <== mgr.7544 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7fac880050d0 con 0x7fac7406ae00 2024-04-02T08:52:17.146 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path /snap_test_dir1 2024-04-02T08:52:17.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:17.148+0000 7fac9e600700 1 -- 172.21.15.89:0/3971189010 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac7406ae00 msgr2=0x7fac7406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:17.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:17.148+0000 7fac9e600700 1 --2- 172.21.15.89:0/3971189010 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac7406ae00 0x7fac7406d2c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fac88000f60 tx=0x7fac88009040 comp rx=0 tx=0).stop 2024-04-02T08:52:17.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:17.148+0000 7fac9e600700 1 -- 172.21.15.89:0/3971189010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac9815d9d0 msgr2=0x7fac981f6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:17.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:17.148+0000 7fac9e600700 1 --2- 172.21.15.89:0/3971189010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac9815d9d0 0x7fac981f6950 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fac9808c230 tx=0x7fac90004ad0 comp rx=0 tx=0).stop 2024-04-02T08:52:17.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:17.148+0000 7fac9e600700 1 -- 172.21.15.89:0/3971189010 shutdown_connections 2024-04-02T08:52:17.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:17.148+0000 7fac9e600700 1 --2- 172.21.15.89:0/3971189010 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fac7406ae00 0x7fac7406d2c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:17.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:17.148+0000 7fac9e600700 1 --2- 172.21.15.89:0/3971189010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac980d16a0 0x7fac980cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:17.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:17.148+0000 7fac9e600700 1 --2- 172.21.15.89:0/3971189010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac9815d9d0 0x7fac981f6950 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:17.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:17.148+0000 7fac9e600700 1 --2- 172.21.15.89:0/3971189010 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac980d2540 0x7fac980b89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:17.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:17.148+0000 7fac9e600700 1 -- 172.21.15.89:0/3971189010 >> 172.21.15.89:0/3971189010 conn(0x7fac9808aff0 msgr2=0x7fac9815ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:17.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:17.149+0000 7fac9e600700 1 -- 172.21.15.89:0/3971189010 shutdown_connections 2024-04-02T08:52:17.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:17.149+0000 7fac9e600700 1 -- 172.21.15.89:0/3971189010 wait complete. 2024-04-02T08:52:17.166 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2024-04-02T08:52:17.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.166 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T08:52:17.214 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_58_00_UTC 2024-04-02T08:52:17.214 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-07_59_00_UTC 2024-04-02T08:52:17.215 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_00_00_UTC 2024-04-02T08:52:17.215 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_01_00_UTC 2024-04-02T08:52:17.215 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_02_01_UTC 2024-04-02T08:52:17.215 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_03_00_UTC 2024-04-02T08:52:17.215 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_04_00_UTC 2024-04-02T08:52:17.215 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_05_00_UTC 2024-04-02T08:52:17.215 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_06_00_UTC 2024-04-02T08:52:17.215 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_07_00_UTC 2024-04-02T08:52:17.215 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_08_00_UTC 2024-04-02T08:52:17.215 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_09_00_UTC 2024-04-02T08:52:17.215 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_10_00_UTC 2024-04-02T08:52:17.215 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_11_00_UTC 2024-04-02T08:52:17.215 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_12_00_UTC 2024-04-02T08:52:17.216 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_13_00_UTC 2024-04-02T08:52:17.216 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_14_00_UTC 2024-04-02T08:52:17.216 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_15_00_UTC 2024-04-02T08:52:17.216 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_16_00_UTC 2024-04-02T08:52:17.216 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_17_00_UTC 2024-04-02T08:52:17.216 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_18_00_UTC 2024-04-02T08:52:17.216 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_19_00_UTC 2024-04-02T08:52:17.216 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_20_00_UTC 2024-04-02T08:52:17.216 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_21_00_UTC 2024-04-02T08:52:17.216 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_22_00_UTC 2024-04-02T08:52:17.216 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_23_00_UTC 2024-04-02T08:52:17.216 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_24_00_UTC 2024-04-02T08:52:17.216 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_25_00_UTC 2024-04-02T08:52:17.216 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_26_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_27_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_28_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_29_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_30_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_31_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_32_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_33_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_34_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_35_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_36_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_37_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_38_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_39_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_40_00_UTC 2024-04-02T08:52:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_41_00_UTC 2024-04-02T08:52:17.218 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_42_00_UTC 2024-04-02T08:52:17.218 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_43_00_UTC 2024-04-02T08:52:17.218 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_44_00_UTC 2024-04-02T08:52:17.218 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_45_00_UTC 2024-04-02T08:52:17.218 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_46_00_UTC 2024-04-02T08:52:17.218 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_47_00_UTC 2024-04-02T08:52:17.218 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_48_00_UTC 2024-04-02T08:52:17.218 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_49_00_UTC 2024-04-02T08:52:17.218 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_50_00_UTC 2024-04-02T08:52:17.218 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_51_00_UTC 2024-04-02T08:52:17.218 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_52_00_UTC 2024-04-02T08:52:17.219 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-07_58_00_UTC 2024-04-02T08:52:17.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.219 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-07_58_00_UTC) 2024-04-02T08:52:17.270 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-07_59_00_UTC 2024-04-02T08:52:17.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.270 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-07_59_00_UTC) 2024-04-02T08:52:17.321 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_00_00_UTC 2024-04-02T08:52:17.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.321 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_00_00_UTC) 2024-04-02T08:52:17.373 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_01_00_UTC 2024-04-02T08:52:17.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.373 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_01_00_UTC) 2024-04-02T08:52:17.425 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_02_01_UTC 2024-04-02T08:52:17.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.425 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_02_01_UTC) 2024-04-02T08:52:17.475 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_03_00_UTC 2024-04-02T08:52:17.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.475 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_03_00_UTC) 2024-04-02T08:52:17.526 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_04_00_UTC 2024-04-02T08:52:17.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.526 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_04_00_UTC) 2024-04-02T08:52:17.577 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_05_00_UTC 2024-04-02T08:52:17.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.577 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_05_00_UTC) 2024-04-02T08:52:17.628 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_06_00_UTC 2024-04-02T08:52:17.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.629 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_06_00_UTC) 2024-04-02T08:52:17.680 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_07_00_UTC 2024-04-02T08:52:17.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.680 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_07_00_UTC) 2024-04-02T08:52:17.731 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_08_00_UTC 2024-04-02T08:52:17.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.732 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_08_00_UTC) 2024-04-02T08:52:17.783 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_09_00_UTC 2024-04-02T08:52:17.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.784 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_09_00_UTC) 2024-04-02T08:52:17.834 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_10_00_UTC 2024-04-02T08:52:17.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.835 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_10_00_UTC) 2024-04-02T08:52:17.885 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_11_00_UTC 2024-04-02T08:52:17.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.885 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_11_00_UTC) 2024-04-02T08:52:17.935 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_12_00_UTC 2024-04-02T08:52:17.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.935 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_12_00_UTC) 2024-04-02T08:52:17.986 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_13_00_UTC 2024-04-02T08:52:17.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:17.986 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_13_00_UTC) 2024-04-02T08:52:18.037 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_14_00_UTC 2024-04-02T08:52:18.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.037 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_14_00_UTC) 2024-04-02T08:52:18.087 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_15_00_UTC 2024-04-02T08:52:18.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.087 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_15_00_UTC) 2024-04-02T08:52:18.137 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_16_00_UTC 2024-04-02T08:52:18.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.137 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_16_00_UTC) 2024-04-02T08:52:18.188 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_17_00_UTC 2024-04-02T08:52:18.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.188 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_17_00_UTC) 2024-04-02T08:52:18.238 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_18_00_UTC 2024-04-02T08:52:18.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.238 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_18_00_UTC) 2024-04-02T08:52:18.288 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_19_00_UTC 2024-04-02T08:52:18.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.289 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_19_00_UTC) 2024-04-02T08:52:18.338 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_20_00_UTC 2024-04-02T08:52:18.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.338 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_20_00_UTC) 2024-04-02T08:52:18.389 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_21_00_UTC 2024-04-02T08:52:18.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.389 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_21_00_UTC) 2024-04-02T08:52:18.440 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_22_00_UTC 2024-04-02T08:52:18.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.440 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_22_00_UTC) 2024-04-02T08:52:18.490 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_23_00_UTC 2024-04-02T08:52:18.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.491 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_23_00_UTC) 2024-04-02T08:52:18.541 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_24_00_UTC 2024-04-02T08:52:18.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.542 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_24_00_UTC) 2024-04-02T08:52:18.591 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_25_00_UTC 2024-04-02T08:52:18.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.592 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_25_00_UTC) 2024-04-02T08:52:18.643 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_26_00_UTC 2024-04-02T08:52:18.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.643 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_26_00_UTC) 2024-04-02T08:52:18.693 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_27_00_UTC 2024-04-02T08:52:18.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.693 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_27_00_UTC) 2024-04-02T08:52:18.744 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_28_00_UTC 2024-04-02T08:52:18.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.744 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_28_00_UTC) 2024-04-02T08:52:18.794 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_29_00_UTC 2024-04-02T08:52:18.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.795 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_29_00_UTC) 2024-04-02T08:52:18.845 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_30_00_UTC 2024-04-02T08:52:18.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.846 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_30_00_UTC) 2024-04-02T08:52:18.896 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_31_00_UTC 2024-04-02T08:52:18.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.897 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_31_00_UTC) 2024-04-02T08:52:18.948 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_32_00_UTC 2024-04-02T08:52:18.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.948 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_32_00_UTC) 2024-04-02T08:52:18.998 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_33_00_UTC 2024-04-02T08:52:18.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:18.999 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_33_00_UTC) 2024-04-02T08:52:19.048 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_34_00_UTC 2024-04-02T08:52:19.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.049 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_34_00_UTC) 2024-04-02T08:52:19.099 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_35_00_UTC 2024-04-02T08:52:19.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.099 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_35_00_UTC) 2024-04-02T08:52:19.149 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_36_00_UTC 2024-04-02T08:52:19.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.150 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_36_00_UTC) 2024-04-02T08:52:19.200 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_37_00_UTC 2024-04-02T08:52:19.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.200 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_37_00_UTC) 2024-04-02T08:52:19.254 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_38_00_UTC 2024-04-02T08:52:19.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.254 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_38_00_UTC) 2024-04-02T08:52:19.305 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_39_00_UTC 2024-04-02T08:52:19.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.305 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_39_00_UTC) 2024-04-02T08:52:19.356 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_40_00_UTC 2024-04-02T08:52:19.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.356 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_40_00_UTC) 2024-04-02T08:52:19.407 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_41_00_UTC 2024-04-02T08:52:19.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.407 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_41_00_UTC) 2024-04-02T08:52:19.458 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_42_00_UTC 2024-04-02T08:52:19.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.458 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_42_00_UTC) 2024-04-02T08:52:19.509 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_43_00_UTC 2024-04-02T08:52:19.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.509 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_43_00_UTC) 2024-04-02T08:52:19.560 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_44_00_UTC 2024-04-02T08:52:19.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.560 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_44_00_UTC) 2024-04-02T08:52:19.611 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_45_00_UTC 2024-04-02T08:52:19.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.611 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_45_00_UTC) 2024-04-02T08:52:19.662 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_46_00_UTC 2024-04-02T08:52:19.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.662 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_46_00_UTC) 2024-04-02T08:52:19.713 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_47_00_UTC 2024-04-02T08:52:19.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.714 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_47_00_UTC) 2024-04-02T08:52:19.765 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_48_00_UTC 2024-04-02T08:52:19.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.765 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_48_00_UTC) 2024-04-02T08:52:19.816 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_49_00_UTC 2024-04-02T08:52:19.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.816 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_49_00_UTC) 2024-04-02T08:52:19.867 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_50_00_UTC 2024-04-02T08:52:19.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.867 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_50_00_UTC) 2024-04-02T08:52:19.917 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_51_00_UTC 2024-04-02T08:52:19.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.918 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_51_00_UTC) 2024-04-02T08:52:19.968 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-08_52_00_UTC 2024-04-02T08:52:19.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:19.969 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-08_52_00_UTC) 2024-04-02T08:52:20.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:20.020 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-02T08:52:20.067 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T08:52:20.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.261+0000 7fae94400700 1 Processor -- start 2024-04-02T08:52:20.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.261+0000 7fae94400700 1 -- start start 2024-04-02T08:52:20.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.262+0000 7fae94400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fae8c15d800 0x7fae8c0725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:20.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.263+0000 7fae94400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fae8c0ce230 0x7fae8c0ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:20.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.263+0000 7fae94400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c0709c0 con 0x7fae8c0b2070 2024-04-02T08:52:20.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.263+0000 7fae94400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8c0b40f0 con 0x7fae8c15d800 2024-04-02T08:52:20.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.263+0000 7fae94400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8c070c80 con 0x7fae8c0ce230 2024-04-02T08:52:20.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.263+0000 7fae91a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fae8c15d800 0x7fae8c0725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:20.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.263+0000 7fae91a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fae8c15d800 0x7fae8c0725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39102/0 (socket says 172.21.15.89:39102) 2024-04-02T08:52:20.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.263+0000 7fae91a00700 1 -- 172.21.15.89:0/2954261584 learned_addr learned my addr 172.21.15.89:0/2954261584 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:20.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.263+0000 7fae91000700 1 --2- 172.21.15.89:0/2954261584 >> v2:172.21.15.113:3300/0 conn(0x7fae8c0ce230 0x7fae8c0ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:20.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.263+0000 7fae91a00700 1 -- 172.21.15.89:0/2954261584 >> v2:172.21.15.113:3300/0 conn(0x7fae8c0ce230 msgr2=0x7fae8c0ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:20.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.263+0000 7fae91a00700 1 --2- 172.21.15.89:0/2954261584 >> v2:172.21.15.113:3300/0 conn(0x7fae8c0ce230 0x7fae8c0ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:20.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.263+0000 7fae91a00700 1 -- 172.21.15.89:0/2954261584 >> v1:172.21.15.113:6789/0 conn(0x7fae8c0b2070 legacy=0x7fae8c15dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:20.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.263+0000 7fae91a00700 1 -- 172.21.15.89:0/2954261584 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c070f60 con 0x7fae8c15d800 2024-04-02T08:52:20.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.264+0000 7fae91a00700 1 --2- 172.21.15.89:0/2954261584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8c15d800 0x7fae8c0725c0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0cce70 tx=0x7fae80009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e5829cba313445 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:20.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.264+0000 7fae8be00700 1 -- 172.21.15.89:0/2954261584 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80004230 con 0x7fae8c15d800 2024-04-02T08:52:20.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.264+0000 7fae8be00700 1 -- 172.21.15.89:0/2954261584 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae80004390 con 0x7fae8c15d800 2024-04-02T08:52:20.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.264+0000 7fae8be00700 1 -- 172.21.15.89:0/2954261584 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae800036a0 con 0x7fae8c15d800 2024-04-02T08:52:20.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.264+0000 7fae94400700 1 -- 172.21.15.89:0/2954261584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8c15d800 msgr2=0x7fae8c0725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:20.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.264+0000 7fae94400700 1 --2- 172.21.15.89:0/2954261584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8c15d800 0x7fae8c0725c0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0cce70 tx=0x7fae80009bd0 comp rx=0 tx=0).stop 2024-04-02T08:52:20.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.264+0000 7fae94400700 1 -- 172.21.15.89:0/2954261584 shutdown_connections 2024-04-02T08:52:20.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.264+0000 7fae94400700 1 --2- 172.21.15.89:0/2954261584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8c15d800 0x7fae8c0725c0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:20.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.264+0000 7fae94400700 1 --2- 172.21.15.89:0/2954261584 >> v2:172.21.15.113:3300/0 conn(0x7fae8c0ce230 0x7fae8c0ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:20.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.264+0000 7fae94400700 1 -- 172.21.15.89:0/2954261584 >> 172.21.15.89:0/2954261584 conn(0x7fae8c0b0d70 msgr2=0x7fae8c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:20.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.265+0000 7fae94400700 1 -- 172.21.15.89:0/2954261584 shutdown_connections 2024-04-02T08:52:20.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.265+0000 7fae94400700 1 -- 172.21.15.89:0/2954261584 wait complete. 2024-04-02T08:52:20.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.266+0000 7fae94400700 1 Processor -- start 2024-04-02T08:52:20.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.266+0000 7fae94400700 1 -- start start 2024-04-02T08:52:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.267+0000 7fae94400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8c0b2070 0x7fae8c153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.267+0000 7fae94400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae8c0ce230 0x7fae8c156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.267+0000 7fae94400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae8c15d800 0x7fae8c1569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.267+0000 7fae94400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c157d30 con 0x7fae8c0b2070 2024-04-02T08:52:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.267+0000 7fae94400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c156f00 con 0x7fae8c15d800 2024-04-02T08:52:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.267+0000 7fae94400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c154fd0 con 0x7fae8c0ce230 2024-04-02T08:52:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.267+0000 7fae91a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8c0b2070 0x7fae8c153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.268+0000 7fae91a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8c0b2070 0x7fae8c153e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39112/0 (socket says 172.21.15.89:39112) 2024-04-02T08:52:20.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.268+0000 7fae91a00700 1 -- 172.21.15.89:0/2060436168 learned_addr learned my addr 172.21.15.89:0/2060436168 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:20.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.268+0000 7fae92400700 1 --2- 172.21.15.89:0/2060436168 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae8c15d800 0x7fae8c1569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:20.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.268+0000 7fae91000700 1 --2- 172.21.15.89:0/2060436168 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae8c0ce230 0x7fae8c156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:20.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.268+0000 7fae91a00700 1 -- 172.21.15.89:0/2060436168 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae8c0ce230 msgr2=0x7fae8c156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:20.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.268+0000 7fae91a00700 1 --2- 172.21.15.89:0/2060436168 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae8c0ce230 0x7fae8c156480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:20.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.268+0000 7fae91a00700 1 -- 172.21.15.89:0/2060436168 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae8c15d800 msgr2=0x7fae8c1569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:20.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.268+0000 7fae91a00700 1 --2- 172.21.15.89:0/2060436168 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae8c15d800 0x7fae8c1569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:20.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.268+0000 7fae91a00700 1 -- 172.21.15.89:0/2060436168 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae80009550 con 0x7fae8c0b2070 2024-04-02T08:52:20.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.268+0000 7fae91a00700 1 --2- 172.21.15.89:0/2060436168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8c0b2070 0x7fae8c153e90 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fae80009130 tx=0x7fae80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:20.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.268+0000 7fae89600700 1 -- 172.21.15.89:0/2060436168 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80011630 con 0x7fae8c0b2070 2024-04-02T08:52:20.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.268+0000 7fae89600700 1 -- 172.21.15.89:0/2060436168 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae80011790 con 0x7fae8c0b2070 2024-04-02T08:52:20.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.268+0000 7fae94400700 1 -- 172.21.15.89:0/2060436168 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8c154cf0 con 0x7fae8c0b2070 2024-04-02T08:52:20.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.268+0000 7fae89600700 1 -- 172.21.15.89:0/2060436168 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80011b40 con 0x7fae8c0b2070 2024-04-02T08:52:20.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.269+0000 7fae94400700 1 -- 172.21.15.89:0/2060436168 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c0adaf0 con 0x7fae8c0b2070 2024-04-02T08:52:20.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.270+0000 7fae94400700 1 -- 172.21.15.89:0/2060436168 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae8c0ad670 con 0x7fae8c0b2070 2024-04-02T08:52:20.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.270+0000 7fae89600700 1 -- 172.21.15.89:0/2060436168 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 97) v1 ==== 89268+0+0 (secure 0 0 0) 0x7fae80011ca0 con 0x7fae8c0b2070 2024-04-02T08:52:20.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.271+0000 7fae89600700 1 --2- 172.21.15.89:0/2060436168 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae6c06ae00 0x7fae6c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:20.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.271+0000 7fae89600700 1 -- 172.21.15.89:0/2060436168 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fae8008ade0 con 0x7fae8c0b2070 2024-04-02T08:52:20.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.271+0000 7fae91000700 1 --2- 172.21.15.89:0/2060436168 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae6c06ae00 0x7fae6c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:20.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.272+0000 7fae91000700 1 --2- 172.21.15.89:0/2060436168 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae6c06ae00 0x7fae6c06d2c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fae8c1573e0 tx=0x7fae78008040 comp rx=0 tx=0).ready entity=mgr.7544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:20.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.276+0000 7fae89600700 1 -- 172.21.15.89:0/2060436168 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae8c0ad670 con 0x7fae8c0b2070 2024-04-02T08:52:20.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.535+0000 7fae94400700 1 -- 172.21.15.89:0/2060436168 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fae8c1f4820 con 0x7fae8c0b2070 2024-04-02T08:52:20.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.882+0000 7fae89600700 1 -- 172.21.15.89:0/2060436168 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 98) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fae800560c0 con 0x7fae8c0b2070 2024-04-02T08:52:20.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.882+0000 7fae89600700 1 -- 172.21.15.89:0/2060436168 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v98) v1 ==== 93+0+0 (secure 0 0 0) 0x7fae80052bc0 con 0x7fae8c0b2070 2024-04-02T08:52:20.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.885+0000 7fae94400700 1 -- 172.21.15.89:0/2060436168 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae6c06ae00 msgr2=0x7fae6c06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:20.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.885+0000 7fae94400700 1 --2- 172.21.15.89:0/2060436168 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae6c06ae00 0x7fae6c06d2c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fae8c1573e0 tx=0x7fae78008040 comp rx=0 tx=0).stop 2024-04-02T08:52:20.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.885+0000 7fae94400700 1 -- 172.21.15.89:0/2060436168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8c0b2070 msgr2=0x7fae8c153e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:20.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.885+0000 7fae94400700 1 --2- 172.21.15.89:0/2060436168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8c0b2070 0x7fae8c153e90 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fae80009130 tx=0x7fae80004750 comp rx=0 tx=0).stop 2024-04-02T08:52:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.886+0000 7fae94400700 1 -- 172.21.15.89:0/2060436168 shutdown_connections 2024-04-02T08:52:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.886+0000 7fae94400700 1 --2- 172.21.15.89:0/2060436168 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fae6c06ae00 0x7fae6c06d2c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.886+0000 7fae94400700 1 --2- 172.21.15.89:0/2060436168 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae8c15d800 0x7fae8c1569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.886+0000 7fae94400700 1 --2- 172.21.15.89:0/2060436168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae8c0b2070 0x7fae8c153e90 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.886+0000 7fae94400700 1 --2- 172.21.15.89:0/2060436168 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae8c0ce230 0x7fae8c156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.886+0000 7fae94400700 1 -- 172.21.15.89:0/2060436168 >> 172.21.15.89:0/2060436168 conn(0x7fae8c0b0d70 msgr2=0x7fae8c0b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.886+0000 7fae94400700 1 -- 172.21.15.89:0/2060436168 shutdown_connections 2024-04-02T08:52:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:20.886+0000 7fae94400700 1 -- 172.21.15.89:0/2060436168 wait complete. 2024-04-02T08:52:20.907 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T08:52:20.907 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T08:52:21.005 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T08:52:21.094 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T08:52:21.094 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T08:52:21.094 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:52:21.107 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:21.108+0000 7fee12653240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T08:52:21.115 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:21.116+0000 7f53e59b5240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T08:52:21.139 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T08:52:21.139 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:52:21.224 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:52:21.291 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:21.292+0000 7fee12653240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T08:52:21.308 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:21.309+0000 7f53e59b5240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T08:52:21.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.481+0000 7f3a46200700 1 Processor -- start 2024-04-02T08:52:21.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.481+0000 7f3a46200700 1 -- start start 2024-04-02T08:52:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.482+0000 7f3a46200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3a400b3af0 0x7f3a4015a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a46200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3a40072580 0x7f3a400b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a46200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3a400b2e50 0x7f3a400709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a46200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a400b2180 con 0x7f3a40072580 2024-04-02T08:52:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a46200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a4015b100 con 0x7f3a400b2e50 2024-04-02T08:52:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a46200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4015b3e0 con 0x7f3a400b3af0 2024-04-02T08:52:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a3ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3a40072580 0x7f3a400b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a3ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3a40072580 0x7f3a400b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39144/0 (socket says 172.21.15.89:39144) 2024-04-02T08:52:21.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a3ea00700 1 -- 172.21.15.89:0/163845029 learned_addr learned my addr 172.21.15.89:0/163845029 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:21.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a3fe00700 1 --2- 172.21.15.89:0/163845029 >> v2:172.21.15.113:3300/0 conn(0x7f3a400b2e50 0x7f3a400709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:21.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a3f400700 1 --2- 172.21.15.89:0/163845029 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3a400b3af0 0x7f3a4015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:21.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a3ea00700 1 -- 172.21.15.89:0/163845029 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3a400b3af0 msgr2=0x7f3a4015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:21.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a3ea00700 1 --2- 172.21.15.89:0/163845029 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3a400b3af0 0x7f3a4015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:21.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a3ea00700 1 -- 172.21.15.89:0/163845029 >> v2:172.21.15.113:3300/0 conn(0x7f3a400b2e50 msgr2=0x7f3a400709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:21.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a3ea00700 1 --2- 172.21.15.89:0/163845029 >> v2:172.21.15.113:3300/0 conn(0x7f3a400b2e50 0x7f3a400709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:21.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.483+0000 7f3a3ea00700 1 -- 172.21.15.89:0/163845029 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a400b23f0 con 0x7f3a40072580 2024-04-02T08:52:21.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.484+0000 7f3a3ea00700 1 --2- 172.21.15.89:0/163845029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a40072580 0x7f3a400b2910 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f3a4015dd90 tx=0x7f3a2c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7bbd73c9b72e63b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:21.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.484+0000 7f3a3e000700 1 -- 172.21.15.89:0/163845029 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c011830 con 0x7f3a40072580 2024-04-02T08:52:21.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.484+0000 7f3a3e000700 1 -- 172.21.15.89:0/163845029 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2c011990 con 0x7f3a40072580 2024-04-02T08:52:21.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.484+0000 7f3a3e000700 1 -- 172.21.15.89:0/163845029 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c011d20 con 0x7f3a40072580 2024-04-02T08:52:21.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.485+0000 7f3a46200700 1 -- 172.21.15.89:0/163845029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a40072580 msgr2=0x7f3a400b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:21.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.485+0000 7f3a46200700 1 --2- 172.21.15.89:0/163845029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a40072580 0x7f3a400b2910 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f3a4015dd90 tx=0x7f3a2c009bd0 comp rx=0 tx=0).stop 2024-04-02T08:52:21.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.485+0000 7f3a46200700 1 -- 172.21.15.89:0/163845029 shutdown_connections 2024-04-02T08:52:21.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.485+0000 7f3a46200700 1 --2- 172.21.15.89:0/163845029 >> v2:172.21.15.113:3300/0 conn(0x7f3a400b2e50 0x7f3a400709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:21.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.485+0000 7f3a46200700 1 --2- 172.21.15.89:0/163845029 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3a400b3af0 0x7f3a4015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:21.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.485+0000 7f3a46200700 1 --2- 172.21.15.89:0/163845029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a40072580 0x7f3a400b2910 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:21.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.485+0000 7f3a46200700 1 -- 172.21.15.89:0/163845029 >> 172.21.15.89:0/163845029 conn(0x7f3a400cf190 msgr2=0x7f3a400cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:21.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.485+0000 7f3a46200700 1 -- 172.21.15.89:0/163845029 shutdown_connections 2024-04-02T08:52:21.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.485+0000 7f3a46200700 1 -- 172.21.15.89:0/163845029 wait complete. 2024-04-02T08:52:21.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.487+0000 7f3a46200700 1 Processor -- start 2024-04-02T08:52:21.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.487+0000 7f3a46200700 1 -- start start 2024-04-02T08:52:21.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a46200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a40072580 0x7f3a401f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:21.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a46200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3a400b2e50 0x7f3a400ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:21.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a46200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a400b3af0 0x7f3a400709c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:21.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a46200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a40147c00 con 0x7f3a400b3af0 2024-04-02T08:52:21.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a46200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400cd040 con 0x7f3a40072580 2024-04-02T08:52:21.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a46200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400cc670 con 0x7f3a400b2e50 2024-04-02T08:52:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a3f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a40072580 0x7f3a401f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a3fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a400b3af0 0x7f3a400709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a3ea00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3a400b2e50 0x7f3a400ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a3f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a40072580 0x7f3a401f3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:40554/0 (socket says 172.21.15.89:40554) 2024-04-02T08:52:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a3f400700 1 -- 172.21.15.89:0/749740353 learned_addr learned my addr 172.21.15.89:0/749740353 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a3ea00700 1 -- 172.21.15.89:0/749740353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a40072580 msgr2=0x7f3a401f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a3ea00700 1 --2- 172.21.15.89:0/749740353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a40072580 0x7f3a401f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a3ea00700 1 -- 172.21.15.89:0/749740353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a400b3af0 msgr2=0x7f3a400709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a3ea00700 1 --2- 172.21.15.89:0/749740353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a400b3af0 0x7f3a400709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:21.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.489+0000 7f3a3ea00700 1 -- 172.21.15.89:0/749740353 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c009550 con 0x7f3a400b2e50 2024-04-02T08:52:21.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.490+0000 7f3a3ea00700 1 --2- 172.21.15.89:0/749740353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3a400b2e50 0x7f3a400ccb00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c016000 tx=0x7f3a2c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:21.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.490+0000 7f3a37400700 1 -- 172.21.15.89:0/749740353 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c004450 con 0x7f3a400b2e50 2024-04-02T08:52:21.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.490+0000 7f3a37400700 1 -- 172.21.15.89:0/749740353 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2c01e070 con 0x7f3a400b2e50 2024-04-02T08:52:21.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.490+0000 7f3a46200700 1 -- 172.21.15.89:0/749740353 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a400cbd40 con 0x7f3a400b2e50 2024-04-02T08:52:21.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.490+0000 7f3a37400700 1 -- 172.21.15.89:0/749740353 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c0038e0 con 0x7f3a400b2e50 2024-04-02T08:52:21.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.490+0000 7f3a46200700 1 -- 172.21.15.89:0/749740353 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a400cc300 con 0x7f3a400b2e50 2024-04-02T08:52:21.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.491+0000 7f3a46200700 1 -- 172.21.15.89:0/749740353 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a40067720 con 0x7f3a400b2e50 2024-04-02T08:52:21.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.492+0000 7f3a37400700 1 -- 172.21.15.89:0/749740353 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 98) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f3a2c003a40 con 0x7f3a400b2e50 2024-04-02T08:52:21.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.493+0000 7f3a37400700 1 --2- 172.21.15.89:0/749740353 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3a2006adb0 0x7f3a2006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:21.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.493+0000 7f3a37400700 1 -- 172.21.15.89:0/749740353 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f3a2c08a380 con 0x7f3a400b2e50 2024-04-02T08:52:21.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.493+0000 7f3a3f400700 1 -- 172.21.15.89:0/749740353 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3a2006adb0 msgr2=0x7f3a2006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:21.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.493+0000 7f3a3f400700 1 --2- 172.21.15.89:0/749740353 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3a2006adb0 0x7f3a2006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:21.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.498+0000 7f3a37400700 1 -- 172.21.15.89:0/749740353 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3a2c0551e0 con 0x7f3a400b2e50 2024-04-02T08:52:21.610 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:21.610+0000 7fee12653240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T08:52:21.646 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:21.647+0000 7f53e59b5240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T08:52:21.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.694+0000 7f3a3f400700 1 -- 172.21.15.89:0/749740353 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3a2006adb0 msgr2=0x7f3a2006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:21.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.694+0000 7f3a3f400700 1 --2- 172.21.15.89:0/749740353 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3a2006adb0 0x7f3a2006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:21.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.712+0000 7f3a46200700 1 -- 172.21.15.89:0/749740353 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a400cc7b0 con 0x7f3a400b2e50 2024-04-02T08:52:21.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.714+0000 7f3a37400700 1 -- 172.21.15.89:0/749740353 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+17208 (secure 0 0 0) 0x7f3a2c058800 con 0x7f3a400b2e50 2024-04-02T08:52:21.714 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:21.714 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":157,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:52:13.887623+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:56:15.040512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-02T07:56:16.179443+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":58,"snap_epoch":156,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/23967222":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/3508000969":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/75408226":"2024-04-03T07:56:33.905850+0000","172.21.15.89:6833/43357":"2024-04-03T07:56:33.905850+0000","172.21.15.89:6832/43357":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/2379953994":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/65229891":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/1849807649":"2024-04-03T07:56:33.905850+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":19,"snaps":[{"begin":2,"length":1},{"begin":58,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:52:21.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.717+0000 7f3a46200700 1 -- 172.21.15.89:0/749740353 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3a2006adb0 msgr2=0x7f3a2006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:21.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.717+0000 7f3a46200700 1 --2- 172.21.15.89:0/749740353 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3a2006adb0 0x7f3a2006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:21.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.717+0000 7f3a46200700 1 -- 172.21.15.89:0/749740353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3a400b2e50 msgr2=0x7f3a400ccb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:21.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.717+0000 7f3a46200700 1 --2- 172.21.15.89:0/749740353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3a400b2e50 0x7f3a400ccb00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c016000 tx=0x7f3a2c004750 comp rx=0 tx=0).stop 2024-04-02T08:52:21.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.717+0000 7f3a46200700 1 -- 172.21.15.89:0/749740353 shutdown_connections 2024-04-02T08:52:21.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.717+0000 7f3a46200700 1 --2- 172.21.15.89:0/749740353 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3a2006adb0 0x7f3a2006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:21.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.717+0000 7f3a46200700 1 --2- 172.21.15.89:0/749740353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a40072580 0x7f3a401f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:21.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.717+0000 7f3a46200700 1 --2- 172.21.15.89:0/749740353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a400b3af0 0x7f3a400709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:21.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.717+0000 7f3a46200700 1 --2- 172.21.15.89:0/749740353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3a400b2e50 0x7f3a400ccb00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:21.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.717+0000 7f3a46200700 1 -- 172.21.15.89:0/749740353 >> 172.21.15.89:0/749740353 conn(0x7f3a400cf190 msgr2=0x7f3a4006d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:21.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.717+0000 7f3a46200700 1 -- 172.21.15.89:0/749740353 shutdown_connections 2024-04-02T08:52:21.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:21.717+0000 7f3a46200700 1 -- 172.21.15.89:0/749740353 wait complete. 2024-04-02T08:52:21.736 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:56:15.040512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:56:16.179443+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '156', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 58, 'snap_epoch': 156, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T08:52:21.736 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T08:52:22.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.049+0000 7f8c6d200700 1 Processor -- start 2024-04-02T08:52:22.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.050+0000 7f8c6d200700 1 -- start start 2024-04-02T08:52:22.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.052+0000 7f8c6d200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8c6815dad0 0x7f8c680ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:22.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.052+0000 7f8c6d200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6815f820 con 0x7f8c680d21d0 2024-04-02T08:52:22.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.052+0000 7f8c6d200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c680cf420 con 0x7f8c680d1630 2024-04-02T08:52:22.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.052+0000 7f8c6d200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c680cfcc0 con 0x7f8c6815dad0 2024-04-02T08:52:22.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.053+0000 7f8c65e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8c6815dad0 0x7f8c680ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:22.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.053+0000 7f8c65e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8c6815dad0 0x7f8c680ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39158/0 (socket says 172.21.15.89:39158) 2024-04-02T08:52:22.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.053+0000 7f8c65e00700 1 -- 172.21.15.89:0/400607042 learned_addr learned my addr 172.21.15.89:0/400607042 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:22.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.054+0000 7f8c65e00700 1 -- 172.21.15.89:0/400607042 >> v1:172.21.15.113:6789/0 conn(0x7f8c680d1630 legacy=0x7f8c680d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:22.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.054+0000 7f8c65e00700 1 -- 172.21.15.89:0/400607042 >> v1:172.21.15.89:6789/0 conn(0x7f8c680d21d0 legacy=0x7f8c6815fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:22.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.054+0000 7f8c65e00700 1 -- 172.21.15.89:0/400607042 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c680cf700 con 0x7f8c6815dad0 2024-04-02T08:52:22.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.054+0000 7f8c65e00700 1 --2- 172.21.15.89:0/400607042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c6815dad0 0x7f8c680ced80 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f8c580098f0 tx=0x7f8c58009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78d542ce0f0257a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:22.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.055+0000 7f8c64a00700 1 -- 172.21.15.89:0/400607042 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c58004230 con 0x7f8c6815dad0 2024-04-02T08:52:22.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.055+0000 7f8c64a00700 1 -- 172.21.15.89:0/400607042 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c58004390 con 0x7f8c6815dad0 2024-04-02T08:52:22.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.055+0000 7f8c64a00700 1 -- 172.21.15.89:0/400607042 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c580036a0 con 0x7f8c6815dad0 2024-04-02T08:52:22.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.057+0000 7f8c6d200700 1 -- 172.21.15.89:0/400607042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c6815dad0 msgr2=0x7f8c680ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:22.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.057+0000 7f8c6d200700 1 --2- 172.21.15.89:0/400607042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c6815dad0 0x7f8c680ced80 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f8c580098f0 tx=0x7f8c58009c00 comp rx=0 tx=0).stop 2024-04-02T08:52:22.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.057+0000 7f8c6d200700 1 -- 172.21.15.89:0/400607042 shutdown_connections 2024-04-02T08:52:22.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.057+0000 7f8c6d200700 1 --2- 172.21.15.89:0/400607042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c6815dad0 0x7f8c680ced80 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:22.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.057+0000 7f8c6d200700 1 -- 172.21.15.89:0/400607042 >> 172.21.15.89:0/400607042 conn(0x7f8c6808aff0 msgr2=0x7f8c680d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:22.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.058+0000 7f8c6d200700 1 -- 172.21.15.89:0/400607042 shutdown_connections 2024-04-02T08:52:22.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.058+0000 7f8c6d200700 1 -- 172.21.15.89:0/400607042 wait complete. 2024-04-02T08:52:22.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.062+0000 7f8c6d200700 1 Processor -- start 2024-04-02T08:52:22.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.062+0000 7f8c6d200700 1 -- start start 2024-04-02T08:52:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.064+0000 7f8c6d200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c680d1630 0x7f8c68158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.064+0000 7f8c6d200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c680d21d0 0x7f8c6815c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.064+0000 7f8c6d200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c6806ca10 0x7f8c6815c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.064+0000 7f8c6d200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c68159370 con 0x7f8c680d1630 2024-04-02T08:52:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.064+0000 7f8c6d200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6809e670 con 0x7f8c6806ca10 2024-04-02T08:52:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.064+0000 7f8c6d200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c680b8520 con 0x7f8c680d21d0 2024-04-02T08:52:22.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.065+0000 7f8c65e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c680d1630 0x7f8c68158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:22.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.065+0000 7f8c65e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c680d1630 0x7f8c68158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39164/0 (socket says 172.21.15.89:39164) 2024-04-02T08:52:22.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.065+0000 7f8c65e00700 1 -- 172.21.15.89:0/1047877502 learned_addr learned my addr 172.21.15.89:0/1047877502 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:22.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.066+0000 7f8c65400700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c680d21d0 0x7f8c6815c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:22.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.066+0000 7f8c66800700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c6806ca10 0x7f8c6815c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:22.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.066+0000 7f8c65e00700 1 -- 172.21.15.89:0/1047877502 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c680d21d0 msgr2=0x7f8c6815c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:22.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.066+0000 7f8c65e00700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c680d21d0 0x7f8c6815c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:22.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.066+0000 7f8c65e00700 1 -- 172.21.15.89:0/1047877502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c6806ca10 msgr2=0x7f8c6815c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:22.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.066+0000 7f8c65e00700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c6806ca10 0x7f8c6815c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:22.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.066+0000 7f8c65e00700 1 -- 172.21.15.89:0/1047877502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c58009550 con 0x7f8c680d1630 2024-04-02T08:52:22.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.067+0000 7f8c66800700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c6806ca10 0x7f8c6815c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:22.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.067+0000 7f8c65e00700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c680d1630 0x7f8c68158e30 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f8c58004000 tx=0x7f8c58004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:22.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.068+0000 7f8c5e000700 1 -- 172.21.15.89:0/1047877502 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5800b660 con 0x7f8c680d1630 2024-04-02T08:52:22.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.069+0000 7f8c5e000700 1 -- 172.21.15.89:0/1047877502 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5800b7e0 con 0x7f8c680d1630 2024-04-02T08:52:22.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.069+0000 7f8c5e000700 1 -- 172.21.15.89:0/1047877502 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5800bb90 con 0x7f8c680d1630 2024-04-02T08:52:22.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.069+0000 7f8c6d200700 1 -- 172.21.15.89:0/1047877502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c680b7c80 con 0x7f8c680d1630 2024-04-02T08:52:22.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.069+0000 7f8c6d200700 1 -- 172.21.15.89:0/1047877502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c680b8240 con 0x7f8c680d1630 2024-04-02T08:52:22.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.071+0000 7f8c6d200700 1 -- 172.21.15.89:0/1047877502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c68067720 con 0x7f8c680d1630 2024-04-02T08:52:22.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.074+0000 7f8c5e000700 1 -- 172.21.15.89:0/1047877502 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 98) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f8c58022070 con 0x7f8c680d1630 2024-04-02T08:52:22.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.077+0000 7f8c5e000700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8c4806ae80 0x7f8c4806d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:22.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.077+0000 7f8c65400700 1 -- 172.21.15.89:0/1047877502 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8c4806ae80 msgr2=0x7f8c4806d340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:22.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.078+0000 7f8c5e000700 1 -- 172.21.15.89:0/1047877502 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f8c5808b300 con 0x7f8c680d1630 2024-04-02T08:52:22.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.078+0000 7f8c65400700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8c4806ae80 0x7f8c4806d340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:22.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.085+0000 7f8c5e000700 1 -- 172.21.15.89:0/1047877502 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8c58056140 con 0x7f8c680d1630 2024-04-02T08:52:22.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.279+0000 7f8c65400700 1 -- 172.21.15.89:0/1047877502 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8c4806ae80 msgr2=0x7f8c4806d340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:22.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.279+0000 7f8c65400700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8c4806ae80 0x7f8c4806d340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:22.327 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:22.328+0000 7fee12653240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T08:52:22.432 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:22.433+0000 7f53e59b5240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T08:52:22.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.511+0000 7f8c6d200700 1 -- 172.21.15.89:0/1047877502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8c6815cfe0 con 0x7f8c680d1630 2024-04-02T08:52:22.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.512+0000 7f8c5e000700 1 -- 172.21.15.89:0/1047877502 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v158) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8c58059760 con 0x7f8c680d1630 2024-04-02T08:52:22.515 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T08:52:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.517+0000 7f8c6d200700 1 -- 172.21.15.89:0/1047877502 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8c4806ae80 msgr2=0x7f8c4806d340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.517+0000 7f8c6d200700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8c4806ae80 0x7f8c4806d340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.517+0000 7f8c6d200700 1 -- 172.21.15.89:0/1047877502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c680d1630 msgr2=0x7f8c68158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:22.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.517+0000 7f8c6d200700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c680d1630 0x7f8c68158e30 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f8c58004000 tx=0x7f8c58004750 comp rx=0 tx=0).stop 2024-04-02T08:52:22.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.517+0000 7f8c6d200700 1 -- 172.21.15.89:0/1047877502 shutdown_connections 2024-04-02T08:52:22.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.517+0000 7f8c6d200700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8c4806ae80 0x7f8c4806d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:22.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.517+0000 7f8c6d200700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c6806ca10 0x7f8c6815c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:22.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.517+0000 7f8c6d200700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c680d1630 0x7f8c68158e30 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:22.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.517+0000 7f8c6d200700 1 --2- 172.21.15.89:0/1047877502 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c680d21d0 0x7f8c6815c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:22.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.517+0000 7f8c6d200700 1 -- 172.21.15.89:0/1047877502 >> 172.21.15.89:0/1047877502 conn(0x7f8c6808aff0 msgr2=0x7f8c6806d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:22.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.517+0000 7f8c6d200700 1 -- 172.21.15.89:0/1047877502 shutdown_connections 2024-04-02T08:52:22.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.518+0000 7f8c6d200700 1 -- 172.21.15.89:0/1047877502 wait complete. 2024-04-02T08:52:22.550 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T08:52:22.660 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:22.661+0000 7fee12653240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T08:52:22.825 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:22.825+0000 7f53e59b5240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T08:52:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.847+0000 7f3ebd200700 1 Processor -- start 2024-04-02T08:52:22.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.847+0000 7f3ebd200700 1 -- start start 2024-04-02T08:52:22.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.848+0000 7f3ebd200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3eb8160d70 0x7f3eb80b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:22.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.848+0000 7f3ebd200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb815ef70 0x7f3eb815e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:22.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.848+0000 7f3ebd200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb8161150 con 0x7f3eb806a790 2024-04-02T08:52:22.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.848+0000 7f3ebd200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb80b4cd0 con 0x7f3eb8160d70 2024-04-02T08:52:22.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.848+0000 7f3ebd200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb806dba0 con 0x7f3eb815ef70 2024-04-02T08:52:22.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.849+0000 7f3eb5e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3eb8160d70 0x7f3eb80b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:22.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.849+0000 7f3eb5400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb815ef70 0x7f3eb815e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:22.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.849+0000 7f3eb5e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3eb8160d70 0x7f3eb80b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:40570/0 (socket says 172.21.15.89:40570) 2024-04-02T08:52:22.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.849+0000 7f3eb5e00700 1 -- 172.21.15.89:0/2534502251 learned_addr learned my addr 172.21.15.89:0/2534502251 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:22.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.849+0000 7f3eb5e00700 1 -- 172.21.15.89:0/2534502251 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb815ef70 msgr2=0x7f3eb815e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:22.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.849+0000 7f3eb5e00700 1 --2- 172.21.15.89:0/2534502251 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb815ef70 0x7f3eb815e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:22.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.849+0000 7f3eb5e00700 1 -- 172.21.15.89:0/2534502251 >> v1:172.21.15.113:6789/0 conn(0x7f3eb806a790 legacy=0x7f3eb81613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:22.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.849+0000 7f3eb5e00700 1 -- 172.21.15.89:0/2534502251 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb815ed30 con 0x7f3eb8160d70 2024-04-02T08:52:22.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.849+0000 7f3eb5e00700 1 --2- 172.21.15.89:0/2534502251 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb8160d70 0x7f3eb80b3200 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80098f0 tx=0x7f3ea8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe52739134fcb391 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:22.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.850+0000 7f3eb4a00700 1 -- 172.21.15.89:0/2534502251 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8011830 con 0x7f3eb8160d70 2024-04-02T08:52:22.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.850+0000 7f3eb4a00700 1 -- 172.21.15.89:0/2534502251 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea8011990 con 0x7f3eb8160d70 2024-04-02T08:52:22.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.850+0000 7f3eb4a00700 1 -- 172.21.15.89:0/2534502251 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8011d20 con 0x7f3eb8160d70 2024-04-02T08:52:22.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.850+0000 7f3ebd200700 1 -- 172.21.15.89:0/2534502251 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb8160d70 msgr2=0x7f3eb80b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:22.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.850+0000 7f3ebd200700 1 --2- 172.21.15.89:0/2534502251 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb8160d70 0x7f3eb80b3200 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80098f0 tx=0x7f3ea8009c00 comp rx=0 tx=0).stop 2024-04-02T08:52:22.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.850+0000 7f3ebd200700 1 -- 172.21.15.89:0/2534502251 shutdown_connections 2024-04-02T08:52:22.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.850+0000 7f3ebd200700 1 --2- 172.21.15.89:0/2534502251 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb8160d70 0x7f3eb80b3200 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:22.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.850+0000 7f3ebd200700 1 --2- 172.21.15.89:0/2534502251 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb815ef70 0x7f3eb815e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:22.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.850+0000 7f3ebd200700 1 -- 172.21.15.89:0/2534502251 >> 172.21.15.89:0/2534502251 conn(0x7f3eb80b5720 msgr2=0x7f3eb80b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:22.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.850+0000 7f3ebd200700 1 -- 172.21.15.89:0/2534502251 shutdown_connections 2024-04-02T08:52:22.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.850+0000 7f3ebd200700 1 -- 172.21.15.89:0/2534502251 wait complete. 2024-04-02T08:52:22.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.852+0000 7f3ebd200700 1 Processor -- start 2024-04-02T08:52:22.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.852+0000 7f3ebd200700 1 -- start start 2024-04-02T08:52:22.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.853+0000 7f3ebd200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb806a790 0x7f3eb815d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:22.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.853+0000 7f3ebd200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb815ef70 0x7f3eb80b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:22.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.853+0000 7f3ebd200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb8160d70 0x7f3eb809c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:22.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.853+0000 7f3ebd200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80b1d90 con 0x7f3eb806a790 2024-04-02T08:52:22.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.853+0000 7f3ebd200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80b0ec0 con 0x7f3eb8160d70 2024-04-02T08:52:22.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.853+0000 7f3ebd200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb815d7d0 con 0x7f3eb815ef70 2024-04-02T08:52:22.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.853+0000 7f3eb5400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb815ef70 0x7f3eb80b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:22.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.853+0000 7f3eb5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb806a790 0x7f3eb815d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:22.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.853+0000 7f3eb5400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb815ef70 0x7f3eb80b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:45810/0 (socket says 172.21.15.89:45810) 2024-04-02T08:52:22.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.853+0000 7f3eb5400700 1 -- 172.21.15.89:0/3086492452 learned_addr learned my addr 172.21.15.89:0/3086492452 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:22.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.853+0000 7f3eb6800700 1 --2- 172.21.15.89:0/3086492452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb8160d70 0x7f3eb809c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:22.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.854+0000 7f3eb5e00700 1 -- 172.21.15.89:0/3086492452 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb815ef70 msgr2=0x7f3eb80b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:22.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.854+0000 7f3eb5e00700 1 --2- 172.21.15.89:0/3086492452 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb815ef70 0x7f3eb80b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:22.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.854+0000 7f3eb5e00700 1 -- 172.21.15.89:0/3086492452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb8160d70 msgr2=0x7f3eb809c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:22.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.854+0000 7f3eb5e00700 1 --2- 172.21.15.89:0/3086492452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb8160d70 0x7f3eb809c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:22.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.854+0000 7f3eb5e00700 1 -- 172.21.15.89:0/3086492452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea8009550 con 0x7f3eb806a790 2024-04-02T08:52:22.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.854+0000 7f3eb5400700 1 --2- 172.21.15.89:0/3086492452 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb815ef70 0x7f3eb80b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:22.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.854+0000 7f3eb5e00700 1 --2- 172.21.15.89:0/3086492452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb806a790 0x7f3eb815d290 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8016000 tx=0x7f3ea8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:22.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.854+0000 7f3eae000700 1 -- 172.21.15.89:0/3086492452 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8004510 con 0x7f3eb806a790 2024-04-02T08:52:22.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.854+0000 7f3eae000700 1 -- 172.21.15.89:0/3086492452 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea801e070 con 0x7f3eb806a790 2024-04-02T08:52:22.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.854+0000 7f3ebd200700 1 -- 172.21.15.89:0/3086492452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb80b1110 con 0x7f3eb806a790 2024-04-02T08:52:22.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.855+0000 7f3ebd200700 1 -- 172.21.15.89:0/3086492452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb80b16d0 con 0x7f3eb806a790 2024-04-02T08:52:22.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.855+0000 7f3eae000700 1 -- 172.21.15.89:0/3086492452 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80038a0 con 0x7f3eb806a790 2024-04-02T08:52:22.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.856+0000 7f3ebd200700 1 -- 172.21.15.89:0/3086492452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb8067720 con 0x7f3eb806a790 2024-04-02T08:52:22.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.857+0000 7f3eae000700 1 -- 172.21.15.89:0/3086492452 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 98) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f3ea8003a00 con 0x7f3eb806a790 2024-04-02T08:52:22.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.858+0000 7f3eae000700 1 --2- 172.21.15.89:0/3086492452 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e9806adb0 0x7f3e9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:22.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.858+0000 7f3eae000700 1 -- 172.21.15.89:0/3086492452 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f3ea808a770 con 0x7f3eb806a790 2024-04-02T08:52:22.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.858+0000 7f3eb5400700 1 -- 172.21.15.89:0/3086492452 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e9806adb0 msgr2=0x7f3e9806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:22.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.858+0000 7f3eb5400700 1 --2- 172.21.15.89:0/3086492452 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e9806adb0 0x7f3e9806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:22.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:22.866+0000 7f3eae000700 1 -- 172.21.15.89:0/3086492452 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ea8055480 con 0x7f3eb806a790 2024-04-02T08:52:23.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.059+0000 7f3eb5400700 1 -- 172.21.15.89:0/3086492452 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e9806adb0 msgr2=0x7f3e9806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:23.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.059+0000 7f3eb5400700 1 --2- 172.21.15.89:0/3086492452 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e9806adb0 0x7f3e9806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:23.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.087+0000 7f3ebd200700 1 -- 172.21.15.89:0/3086492452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3eb80ae100 con 0x7f3eb806a790 2024-04-02T08:52:23.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.087+0000 7f3eae000700 1 -- 172.21.15.89:0/3086492452 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v159) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ea8003cb0 con 0x7f3eb806a790 2024-04-02T08:52:23.087 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T08:52:23.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.090+0000 7f3ebd200700 1 -- 172.21.15.89:0/3086492452 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e9806adb0 msgr2=0x7f3e9806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:23.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.090+0000 7f3ebd200700 1 --2- 172.21.15.89:0/3086492452 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e9806adb0 0x7f3e9806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.090+0000 7f3ebd200700 1 -- 172.21.15.89:0/3086492452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb806a790 msgr2=0x7f3eb815d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:23.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.090+0000 7f3ebd200700 1 --2- 172.21.15.89:0/3086492452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb806a790 0x7f3eb815d290 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8016000 tx=0x7f3ea8004750 comp rx=0 tx=0).stop 2024-04-02T08:52:23.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.090+0000 7f3ebd200700 1 -- 172.21.15.89:0/3086492452 shutdown_connections 2024-04-02T08:52:23.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.090+0000 7f3ebd200700 1 --2- 172.21.15.89:0/3086492452 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e9806adb0 0x7f3e9806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.090+0000 7f3ebd200700 1 --2- 172.21.15.89:0/3086492452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3eb8160d70 0x7f3eb809c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.090+0000 7f3ebd200700 1 --2- 172.21.15.89:0/3086492452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb806a790 0x7f3eb815d290 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.090+0000 7f3ebd200700 1 --2- 172.21.15.89:0/3086492452 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3eb815ef70 0x7f3eb80b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.090+0000 7f3ebd200700 1 -- 172.21.15.89:0/3086492452 >> 172.21.15.89:0/3086492452 conn(0x7f3eb80b5720 msgr2=0x7f3eb80ad650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:23.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.090+0000 7f3ebd200700 1 -- 172.21.15.89:0/3086492452 shutdown_connections 2024-04-02T08:52:23.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.090+0000 7f3ebd200700 1 -- 172.21.15.89:0/3086492452 wait complete. 2024-04-02T08:52:23.108 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T08:52:23.152 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:23.153+0000 7fee12653240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T08:52:23.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.310+0000 7fd92ec00700 1 Processor -- start 2024-04-02T08:52:23.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.310+0000 7fd92ec00700 1 -- start start 2024-04-02T08:52:23.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.311+0000 7fd92ec00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd928160d70 0x7fd9280b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:23.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.311+0000 7fd92ec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd92806a790 0x7fd92806ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:23.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.311+0000 7fd92ec00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd928161150 con 0x7fd92815ef70 2024-04-02T08:52:23.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.311+0000 7fd92ec00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd92806b180 con 0x7fd928160d70 2024-04-02T08:52:23.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.311+0000 7fd92ec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9280b4cd0 con 0x7fd92806a790 2024-04-02T08:52:23.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.311+0000 7fd927e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd928160d70 0x7fd9280b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.312+0000 7fd927400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd92815ef70 0x7fd92815e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51874/0 (socket says 172.21.15.89:51874) 2024-04-02T08:52:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.312+0000 7fd927400700 1 -- 172.21.15.89:0/4170332186 learned_addr learned my addr 172.21.15.89:0/4170332186 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:23.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.312+0000 7fd92cc00700 1 --2- 172.21.15.89:0/4170332186 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd92806a790 0x7fd92806ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:23.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.312+0000 7fd926a00700 1 -- 172.21.15.89:0/4170332186 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696774247 0 0) 0x7fd928161150 con 0x7fd92815ef70 2024-04-02T08:52:23.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.312+0000 7fd927e00700 1 -- 172.21.15.89:0/4170332186 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd92806a790 msgr2=0x7fd92806ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:23.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.312+0000 7fd927e00700 1 --2- 172.21.15.89:0/4170332186 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd92806a790 0x7fd92806ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.312+0000 7fd927e00700 1 -- 172.21.15.89:0/4170332186 >> v1:172.21.15.89:6789/0 conn(0x7fd92815ef70 legacy=0x7fd92815e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:23.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.312+0000 7fd927e00700 1 -- 172.21.15.89:0/4170332186 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92806dba0 con 0x7fd928160d70 2024-04-02T08:52:23.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.312+0000 7fd927e00700 1 --2- 172.21.15.89:0/4170332186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd928160d70 0x7fd9280b3200 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fd918009840 tx=0x7fd918009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d8f25b7cbe50e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:23.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.313+0000 7fd926a00700 1 -- 172.21.15.89:0/4170332186 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd918011830 con 0x7fd928160d70 2024-04-02T08:52:23.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.313+0000 7fd926a00700 1 -- 172.21.15.89:0/4170332186 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd918011990 con 0x7fd928160d70 2024-04-02T08:52:23.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.313+0000 7fd926a00700 1 -- 172.21.15.89:0/4170332186 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd918011d20 con 0x7fd928160d70 2024-04-02T08:52:23.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.313+0000 7fd92ec00700 1 -- 172.21.15.89:0/4170332186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd928160d70 msgr2=0x7fd9280b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:23.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.313+0000 7fd92ec00700 1 --2- 172.21.15.89:0/4170332186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd928160d70 0x7fd9280b3200 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fd918009840 tx=0x7fd918009b50 comp rx=0 tx=0).stop 2024-04-02T08:52:23.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.313+0000 7fd92ec00700 1 -- 172.21.15.89:0/4170332186 shutdown_connections 2024-04-02T08:52:23.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.313+0000 7fd92ec00700 1 --2- 172.21.15.89:0/4170332186 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd92806a790 0x7fd92806ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.313+0000 7fd92ec00700 1 --2- 172.21.15.89:0/4170332186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd928160d70 0x7fd9280b3200 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.313+0000 7fd92ec00700 1 -- 172.21.15.89:0/4170332186 >> 172.21.15.89:0/4170332186 conn(0x7fd9280b5720 msgr2=0x7fd9280b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:23.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.313+0000 7fd92ec00700 1 -- 172.21.15.89:0/4170332186 shutdown_connections 2024-04-02T08:52:23.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.313+0000 7fd92ec00700 1 -- 172.21.15.89:0/4170332186 wait complete. 2024-04-02T08:52:23.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.315+0000 7fd92ec00700 1 Processor -- start 2024-04-02T08:52:23.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.315+0000 7fd92ec00700 1 -- start start 2024-04-02T08:52:23.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.316+0000 7fd92ec00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd92806a790 0x7fd9280c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:23.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.316+0000 7fd92ec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd92815ef70 0x7fd92814f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:23.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.316+0000 7fd92ec00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd928160d70 0x7fd9280ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:23.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.316+0000 7fd92ec00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9280c9ac0 con 0x7fd928160d70 2024-04-02T08:52:23.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.316+0000 7fd92ec00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9280aff50 con 0x7fd92806a790 2024-04-02T08:52:23.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.316+0000 7fd92ec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9280b0230 con 0x7fd92815ef70 2024-04-02T08:52:23.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.317+0000 7fd92cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd928160d70 0x7fd9280ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:23.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.317+0000 7fd92cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd928160d70 0x7fd9280ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39190/0 (socket says 172.21.15.89:39190) 2024-04-02T08:52:23.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.317+0000 7fd927400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd92815ef70 0x7fd92814f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:23.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.317+0000 7fd92cc00700 1 -- 172.21.15.89:0/3760649090 learned_addr learned my addr 172.21.15.89:0/3760649090 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:23.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.317+0000 7fd927e00700 1 --2- 172.21.15.89:0/3760649090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd92806a790 0x7fd9280c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:23.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.317+0000 7fd92cc00700 1 -- 172.21.15.89:0/3760649090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd92815ef70 msgr2=0x7fd92814f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:23.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.317+0000 7fd92cc00700 1 --2- 172.21.15.89:0/3760649090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd92815ef70 0x7fd92814f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.317+0000 7fd92cc00700 1 -- 172.21.15.89:0/3760649090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd92806a790 msgr2=0x7fd9280c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:23.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.317+0000 7fd92cc00700 1 --2- 172.21.15.89:0/3760649090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd92806a790 0x7fd9280c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.317+0000 7fd92cc00700 1 -- 172.21.15.89:0/3760649090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9180094a0 con 0x7fd928160d70 2024-04-02T08:52:23.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.317+0000 7fd92cc00700 1 --2- 172.21.15.89:0/3760649090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd928160d70 0x7fd9280ca1e0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fd92000efc0 tx=0x7fd92000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:23.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.317+0000 7fd91fe00700 1 -- 172.21.15.89:0/3760649090 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd920009ea0 con 0x7fd928160d70 2024-04-02T08:52:23.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.317+0000 7fd91fe00700 1 -- 172.21.15.89:0/3760649090 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92000f040 con 0x7fd928160d70 2024-04-02T08:52:23.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.318+0000 7fd91fe00700 1 -- 172.21.15.89:0/3760649090 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92000ced0 con 0x7fd928160d70 2024-04-02T08:52:23.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.318+0000 7fd92ec00700 1 -- 172.21.15.89:0/3760649090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9280b0510 con 0x7fd928160d70 2024-04-02T08:52:23.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.318+0000 7fd92ec00700 1 -- 172.21.15.89:0/3760649090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9280b0ad0 con 0x7fd928160d70 2024-04-02T08:52:23.320 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:23.318+0000 7f53e59b5240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T08:52:23.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.319+0000 7fd92ec00700 1 -- 172.21.15.89:0/3760649090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd928072660 con 0x7fd928160d70 2024-04-02T08:52:23.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.320+0000 7fd91fe00700 1 -- 172.21.15.89:0/3760649090 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 98) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fd920014480 con 0x7fd928160d70 2024-04-02T08:52:23.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.321+0000 7fd91fe00700 1 --2- 172.21.15.89:0/3760649090 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd90806adb0 0x7fd90806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:23.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.321+0000 7fd91fe00700 1 -- 172.21.15.89:0/3760649090 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fd92008a380 con 0x7fd928160d70 2024-04-02T08:52:23.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.321+0000 7fd927e00700 1 -- 172.21.15.89:0/3760649090 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd90806adb0 msgr2=0x7fd90806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:23.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.321+0000 7fd927e00700 1 --2- 172.21.15.89:0/3760649090 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd90806adb0 0x7fd90806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:23.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.325+0000 7fd91fe00700 1 -- 172.21.15.89:0/3760649090 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd920055000 con 0x7fd928160d70 2024-04-02T08:52:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.522+0000 7fd927e00700 1 -- 172.21.15.89:0/3760649090 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd90806adb0 msgr2=0x7fd90806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.522+0000 7fd927e00700 1 --2- 172.21.15.89:0/3760649090 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd90806adb0 0x7fd90806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:23.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.535+0000 7fd92ec00700 1 -- 172.21.15.89:0/3760649090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd92815e0f0 con 0x7fd928160d70 2024-04-02T08:52:23.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.535+0000 7fd91fe00700 1 -- 172.21.15.89:0/3760649090 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v159) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd920058620 con 0x7fd928160d70 2024-04-02T08:52:23.535 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T08:52:23.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.537+0000 7fd92ec00700 1 -- 172.21.15.89:0/3760649090 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd90806adb0 msgr2=0x7fd90806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:23.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.537+0000 7fd92ec00700 1 --2- 172.21.15.89:0/3760649090 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd90806adb0 0x7fd90806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.537+0000 7fd92ec00700 1 -- 172.21.15.89:0/3760649090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd928160d70 msgr2=0x7fd9280ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:23.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.537+0000 7fd92ec00700 1 --2- 172.21.15.89:0/3760649090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd928160d70 0x7fd9280ca1e0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fd92000efc0 tx=0x7fd92000c3a0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.537+0000 7fd92ec00700 1 -- 172.21.15.89:0/3760649090 shutdown_connections 2024-04-02T08:52:23.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.537+0000 7fd92ec00700 1 --2- 172.21.15.89:0/3760649090 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd90806adb0 0x7fd90806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.537+0000 7fd92ec00700 1 --2- 172.21.15.89:0/3760649090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd92806a790 0x7fd9280c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.537+0000 7fd92ec00700 1 --2- 172.21.15.89:0/3760649090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd928160d70 0x7fd9280ca1e0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.537+0000 7fd92ec00700 1 --2- 172.21.15.89:0/3760649090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd92815ef70 0x7fd92814f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.537+0000 7fd92ec00700 1 -- 172.21.15.89:0/3760649090 >> 172.21.15.89:0/3760649090 conn(0x7fd9280b5720 msgr2=0x7fd92809bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:23.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.538+0000 7fd92ec00700 1 -- 172.21.15.89:0/3760649090 shutdown_connections 2024-04-02T08:52:23.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.538+0000 7fd92ec00700 1 -- 172.21.15.89:0/3760649090 wait complete. 2024-04-02T08:52:23.556 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T08:52:23.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.754+0000 7fec48800700 1 Processor -- start 2024-04-02T08:52:23.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.754+0000 7fec48800700 1 -- start start 2024-04-02T08:52:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.755+0000 7fec48800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec40099a60 0x7fec400d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.755+0000 7fec48800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fec4015d990 0x7fec40099e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.755+0000 7fec48800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec400d0970 con 0x7fec4009a750 2024-04-02T08:52:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.755+0000 7fec48800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec4006c020 con 0x7fec4015d990 2024-04-02T08:52:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.755+0000 7fec48800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4009a380 con 0x7fec40099a60 2024-04-02T08:52:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.756+0000 7fec46800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fec4009a750 0x7fec4006c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51882/0 (socket says 172.21.15.89:51882) 2024-04-02T08:52:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.756+0000 7fec46800700 1 -- 172.21.15.89:0/2220647965 learned_addr learned my addr 172.21.15.89:0/2220647965 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:23.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.756+0000 7fec45e00700 1 --2- 172.21.15.89:0/2220647965 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec40099a60 0x7fec400d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:23.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.756+0000 7fec45400700 1 --2- 172.21.15.89:0/2220647965 >> v2:172.21.15.113:3300/0 conn(0x7fec4015d990 0x7fec40099e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:23.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.756+0000 7fec44a00700 1 -- 172.21.15.89:0/2220647965 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741521365 0 0) 0x7fec400d0970 con 0x7fec4009a750 2024-04-02T08:52:23.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.756+0000 7fec44a00700 1 -- 172.21.15.89:0/2220647965 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec280035b0 con 0x7fec4009a750 2024-04-02T08:52:23.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.756+0000 7fec45e00700 1 -- 172.21.15.89:0/2220647965 >> v2:172.21.15.113:3300/0 conn(0x7fec4015d990 msgr2=0x7fec40099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.756+0000 7fec45e00700 1 --2- 172.21.15.89:0/2220647965 >> v2:172.21.15.113:3300/0 conn(0x7fec4015d990 0x7fec40099e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.756+0000 7fec45e00700 1 -- 172.21.15.89:0/2220647965 >> v1:172.21.15.89:6789/0 conn(0x7fec4009a750 legacy=0x7fec4006c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.756+0000 7fec45e00700 1 -- 172.21.15.89:0/2220647965 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec400c81a0 con 0x7fec40099a60 2024-04-02T08:52:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.756+0000 7fec45400700 1 --2- 172.21.15.89:0/2220647965 >> v2:172.21.15.113:3300/0 conn(0x7fec4015d990 0x7fec40099e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.757+0000 7fec45e00700 1 --2- 172.21.15.89:0/2220647965 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec40099a60 0x7fec400d0060 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fec34009840 tx=0x7fec34009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb9e5051bedffb58 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.757+0000 7fec44a00700 1 -- 172.21.15.89:0/2220647965 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34011830 con 0x7fec40099a60 2024-04-02T08:52:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.757+0000 7fec44a00700 1 -- 172.21.15.89:0/2220647965 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec34011990 con 0x7fec40099a60 2024-04-02T08:52:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.757+0000 7fec44a00700 1 -- 172.21.15.89:0/2220647965 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34011d20 con 0x7fec40099a60 2024-04-02T08:52:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.757+0000 7fec48800700 1 -- 172.21.15.89:0/2220647965 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec40099a60 msgr2=0x7fec400d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.757+0000 7fec48800700 1 --2- 172.21.15.89:0/2220647965 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec40099a60 0x7fec400d0060 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fec34009840 tx=0x7fec34009b50 comp rx=0 tx=0).stop 2024-04-02T08:52:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.757+0000 7fec48800700 1 -- 172.21.15.89:0/2220647965 shutdown_connections 2024-04-02T08:52:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.757+0000 7fec48800700 1 --2- 172.21.15.89:0/2220647965 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec40099a60 0x7fec400d0060 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.757+0000 7fec48800700 1 --2- 172.21.15.89:0/2220647965 >> v2:172.21.15.113:3300/0 conn(0x7fec4015d990 0x7fec40099e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.757+0000 7fec48800700 1 -- 172.21.15.89:0/2220647965 >> 172.21.15.89:0/2220647965 conn(0x7fec400b2e20 msgr2=0x7fec400b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:23.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.758+0000 7fec48800700 1 -- 172.21.15.89:0/2220647965 shutdown_connections 2024-04-02T08:52:23.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.758+0000 7fec48800700 1 -- 172.21.15.89:0/2220647965 wait complete. 2024-04-02T08:52:23.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.760+0000 7fec48800700 1 Processor -- start 2024-04-02T08:52:23.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.760+0000 7fec48800700 1 -- start start 2024-04-02T08:52:23.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.761+0000 7fec48800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec40099a60 0x7fec40159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:23.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.761+0000 7fec48800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec4009a750 0x7fec4015b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:23.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.761+0000 7fec48800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec4015d990 0x7fec4015bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:23.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.761+0000 7fec48800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4015f2e0 con 0x7fec4009a750 2024-04-02T08:52:23.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.761+0000 7fec48800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec400b2770 con 0x7fec40099a60 2024-04-02T08:52:23.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.761+0000 7fec48800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec400c5970 con 0x7fec4015d990 2024-04-02T08:52:23.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.762+0000 7fec45400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec4009a750 0x7fec4015b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.762+0000 7fec45e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec40099a60 0x7fec40159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.762+0000 7fec45400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec4009a750 0x7fec4015b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39198/0 (socket says 172.21.15.89:39198) 2024-04-02T08:52:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.762+0000 7fec45e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec40099a60 0x7fec40159a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:40610/0 (socket says 172.21.15.89:40610) 2024-04-02T08:52:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.762+0000 7fec45400700 1 -- 172.21.15.89:0/3739821390 learned_addr learned my addr 172.21.15.89:0/3739821390 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.762+0000 7fec46800700 1 --2- 172.21.15.89:0/3739821390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec4015d990 0x7fec4015bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.762+0000 7fec45400700 1 -- 172.21.15.89:0/3739821390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec4015d990 msgr2=0x7fec4015bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.762+0000 7fec45400700 1 --2- 172.21.15.89:0/3739821390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec4015d990 0x7fec4015bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.762+0000 7fec45400700 1 -- 172.21.15.89:0/3739821390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec40099a60 msgr2=0x7fec40159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.762+0000 7fec45400700 1 --2- 172.21.15.89:0/3739821390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec40099a60 0x7fec40159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.762+0000 7fec45400700 1 -- 172.21.15.89:0/3739821390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec300094a0 con 0x7fec4009a750 2024-04-02T08:52:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.762+0000 7fec45400700 1 --2- 172.21.15.89:0/3739821390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec4009a750 0x7fec4015b7f0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fec3000e9e0 tx=0x7fec3000ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.763+0000 7fec3a000700 1 -- 172.21.15.89:0/3739821390 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3000ee80 con 0x7fec4009a750 2024-04-02T08:52:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.763+0000 7fec3a000700 1 -- 172.21.15.89:0/3739821390 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3000c850 con 0x7fec4009a750 2024-04-02T08:52:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.763+0000 7fec48800700 1 -- 172.21.15.89:0/3739821390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec340094a0 con 0x7fec4009a750 2024-04-02T08:52:23.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.763+0000 7fec3a000700 1 -- 172.21.15.89:0/3739821390 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3000cc00 con 0x7fec4009a750 2024-04-02T08:52:23.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.763+0000 7fec48800700 1 -- 172.21.15.89:0/3739821390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4015ef70 con 0x7fec4009a750 2024-04-02T08:52:23.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.764+0000 7fec48800700 1 -- 172.21.15.89:0/3739821390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec400c5e90 con 0x7fec4009a750 2024-04-02T08:52:23.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.765+0000 7fec3a000700 1 -- 172.21.15.89:0/3739821390 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 98) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fec3000cd60 con 0x7fec4009a750 2024-04-02T08:52:23.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.766+0000 7fec3a000700 1 --2- 172.21.15.89:0/3739821390 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fec2406adb0 0x7fec2406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:23.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.766+0000 7fec3a000700 1 -- 172.21.15.89:0/3739821390 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fec30089530 con 0x7fec4009a750 2024-04-02T08:52:23.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.766+0000 7fec45e00700 1 -- 172.21.15.89:0/3739821390 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fec2406adb0 msgr2=0x7fec2406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:23.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.766+0000 7fec45e00700 1 --2- 172.21.15.89:0/3739821390 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fec2406adb0 0x7fec2406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:23.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.770+0000 7fec3a000700 1 -- 172.21.15.89:0/3739821390 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fec400c5e90 con 0x7fec4009a750 2024-04-02T08:52:23.868 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:23.869+0000 7fee12653240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T08:52:23.963 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:23.964+0000 7fee12653240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T08:52:23.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.967+0000 7fec45e00700 1 -- 172.21.15.89:0/3739821390 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fec2406adb0 msgr2=0x7fec2406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:23.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:23.967+0000 7fec45e00700 1 --2- 172.21.15.89:0/3739821390 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fec2406adb0 0x7fec2406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:24.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.015+0000 7fec48800700 1 -- 172.21.15.89:0/3739821390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fec4015f490 con 0x7fec4009a750 2024-04-02T08:52:24.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.015+0000 7fec3a000700 1 -- 172.21.15.89:0/3739821390 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v159) v1 ==== 80+0+454 (secure 0 0 0) 0x7fec4015f490 con 0x7fec4009a750 2024-04-02T08:52:24.015 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/23967222 2024-04-03T07:56:33.905850+0000 2024-04-02T08:52:24.015 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3508000969 2024-04-03T07:56:33.905850+0000 2024-04-02T08:52:24.015 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/75408226 2024-04-03T07:56:33.905850+0000 2024-04-02T08:52:24.015 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/43357 2024-04-03T07:56:33.905850+0000 2024-04-02T08:52:24.015 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/43357 2024-04-03T07:56:33.905850+0000 2024-04-02T08:52:24.015 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2379953994 2024-04-03T07:56:33.905850+0000 2024-04-02T08:52:24.015 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/65229891 2024-04-03T07:56:33.905850+0000 2024-04-02T08:52:24.015 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1849807649 2024-04-03T07:56:33.905850+0000 2024-04-02T08:52:24.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.017+0000 7fec48800700 1 -- 172.21.15.89:0/3739821390 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fec2406adb0 msgr2=0x7fec2406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:24.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.017+0000 7fec48800700 1 --2- 172.21.15.89:0/3739821390 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fec2406adb0 0x7fec2406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.018+0000 7fec48800700 1 -- 172.21.15.89:0/3739821390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec4009a750 msgr2=0x7fec4015b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:24.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.018+0000 7fec48800700 1 --2- 172.21.15.89:0/3739821390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec4009a750 0x7fec4015b7f0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fec3000e9e0 tx=0x7fec3000ecf0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.018+0000 7fec48800700 1 -- 172.21.15.89:0/3739821390 shutdown_connections 2024-04-02T08:52:24.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.018+0000 7fec48800700 1 --2- 172.21.15.89:0/3739821390 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fec2406adb0 0x7fec2406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.018+0000 7fec48800700 1 --2- 172.21.15.89:0/3739821390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec40099a60 0x7fec40159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.018+0000 7fec48800700 1 --2- 172.21.15.89:0/3739821390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec4015d990 0x7fec4015bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.018+0000 7fec48800700 1 --2- 172.21.15.89:0/3739821390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec4009a750 0x7fec4015b7f0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.018+0000 7fec48800700 1 -- 172.21.15.89:0/3739821390 >> 172.21.15.89:0/3739821390 conn(0x7fec400b2e20 msgr2=0x7fec400d0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:24.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.018+0000 7fec48800700 1 -- 172.21.15.89:0/3739821390 shutdown_connections 2024-04-02T08:52:24.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.018+0000 7fec48800700 1 -- 172.21.15.89:0/3739821390 wait complete. 2024-04-02T08:52:24.019 INFO:teuthology.orchestra.run.smithi089.stderr:listed 8 entries 2024-04-02T08:52:24.037 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T08:52:24.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:24.037 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T08:52:24.085 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:24.085+0000 7f53e59b5240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T08:52:24.117 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T08:52:24.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:24.118 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T08:52:24.124 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:24.125+0000 7fee12653240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T08:52:24.163 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T08:52:24.163 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:52:24.186 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:24.187+0000 7f53e59b5240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T08:52:24.207 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T08:52:24.207 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T08:52:24.208 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:52:24.252 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T08:52:24.253 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T08:52:24.253 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:52:24.298 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T08:52:24.298 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T08:52:24.345 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:24.346+0000 7fee12653240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T08:52:24.353 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:24.354+0000 7f53e59b5240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T08:52:24.435 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:24.435+0000 7fee12653240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T08:52:24.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.499+0000 7fb033000700 1 Processor -- start 2024-04-02T08:52:24.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.499+0000 7fb033000700 1 -- start start 2024-04-02T08:52:24.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.500+0000 7fb033000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb02c15da50 0x7fb02c0ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:24.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.500+0000 7fb033000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb02c0cec70 0x7fb02c0d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:24.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.500+0000 7fb033000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb02c0d0b20 0x7fb02c0b52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:24.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.500+0000 7fb033000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02c0b95d0 con 0x7fb02c15da50 2024-04-02T08:52:24.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.500+0000 7fb033000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02c15de30 con 0x7fb02c0d0b20 2024-04-02T08:52:24.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.500+0000 7fb033000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0d0fa0 con 0x7fb02c0cec70 2024-04-02T08:52:24.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.501+0000 7fb02be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb02c15da50 0x7fb02c0ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:24.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.501+0000 7fb02be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb02c15da50 0x7fb02c0ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39214/0 (socket says 172.21.15.89:39214) 2024-04-02T08:52:24.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.501+0000 7fb02be00700 1 -- 172.21.15.89:0/3964493999 learned_addr learned my addr 172.21.15.89:0/3964493999 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:24.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.501+0000 7fb02b400700 1 --2- 172.21.15.89:0/3964493999 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb02c0cec70 0x7fb02c0d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:24.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.501+0000 7fb031000700 1 --2- 172.21.15.89:0/3964493999 >> v2:172.21.15.113:3300/0 conn(0x7fb02c0d0b20 0x7fb02c0b52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:24.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.501+0000 7fb02be00700 1 -- 172.21.15.89:0/3964493999 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb02c0cec70 msgr2=0x7fb02c0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:24.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.501+0000 7fb02be00700 1 --2- 172.21.15.89:0/3964493999 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb02c0cec70 0x7fb02c0d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.501+0000 7fb02be00700 1 -- 172.21.15.89:0/3964493999 >> v2:172.21.15.113:3300/0 conn(0x7fb02c0d0b20 msgr2=0x7fb02c0b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:24.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.501+0000 7fb02be00700 1 --2- 172.21.15.89:0/3964493999 >> v2:172.21.15.113:3300/0 conn(0x7fb02c0d0b20 0x7fb02c0b52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.501+0000 7fb02be00700 1 -- 172.21.15.89:0/3964493999 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c0d24e0 con 0x7fb02c15da50 2024-04-02T08:52:24.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.501+0000 7fb02be00700 1 --2- 172.21.15.89:0/3964493999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb02c15da50 0x7fb02c0ce730 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fb02c08c190 tx=0x7fb01c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c42628bfb9e51f8b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:24.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.502+0000 7fb02aa00700 1 -- 172.21.15.89:0/3964493999 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c011830 con 0x7fb02c15da50 2024-04-02T08:52:24.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.502+0000 7fb02aa00700 1 -- 172.21.15.89:0/3964493999 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c011990 con 0x7fb02c15da50 2024-04-02T08:52:24.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.502+0000 7fb02aa00700 1 -- 172.21.15.89:0/3964493999 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c011d20 con 0x7fb02c15da50 2024-04-02T08:52:24.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.502+0000 7fb033000700 1 -- 172.21.15.89:0/3964493999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb02c15da50 msgr2=0x7fb02c0ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:24.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.502+0000 7fb033000700 1 --2- 172.21.15.89:0/3964493999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb02c15da50 0x7fb02c0ce730 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fb02c08c190 tx=0x7fb01c009bd0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.502+0000 7fb033000700 1 -- 172.21.15.89:0/3964493999 shutdown_connections 2024-04-02T08:52:24.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.502+0000 7fb033000700 1 --2- 172.21.15.89:0/3964493999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb02c15da50 0x7fb02c0ce730 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.502+0000 7fb033000700 1 --2- 172.21.15.89:0/3964493999 >> v2:172.21.15.113:3300/0 conn(0x7fb02c0d0b20 0x7fb02c0b52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.502+0000 7fb033000700 1 --2- 172.21.15.89:0/3964493999 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb02c0cec70 0x7fb02c0d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.502+0000 7fb033000700 1 -- 172.21.15.89:0/3964493999 >> 172.21.15.89:0/3964493999 conn(0x7fb02c0666a0 msgr2=0x7fb02c066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:24.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.502+0000 7fb033000700 1 -- 172.21.15.89:0/3964493999 shutdown_connections 2024-04-02T08:52:24.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.503+0000 7fb033000700 1 -- 172.21.15.89:0/3964493999 wait complete. 2024-04-02T08:52:24.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.504+0000 7fb033000700 1 Processor -- start 2024-04-02T08:52:24.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.504+0000 7fb033000700 1 -- start start 2024-04-02T08:52:24.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.505+0000 7fb033000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb02c0cec70 0x7fb02c158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:24.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.505+0000 7fb033000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb02c0d0b20 0x7fb02c158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:24.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.505+0000 7fb033000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb02c15da50 0x7fb02c0b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:24.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.505+0000 7fb033000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c1f4c10 con 0x7fb02c0d0b20 2024-04-02T08:52:24.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.505+0000 7fb033000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c1f4000 con 0x7fb02c0cec70 2024-04-02T08:52:24.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.505+0000 7fb033000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c1f42e0 con 0x7fb02c15da50 2024-04-02T08:52:24.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.506+0000 7fb02b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb02c0d0b20 0x7fb02c158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:24.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.506+0000 7fb02be00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb02c0cec70 0x7fb02c158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:24.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.506+0000 7fb02b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb02c0d0b20 0x7fb02c158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39218/0 (socket says 172.21.15.89:39218) 2024-04-02T08:52:24.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.506+0000 7fb02b400700 1 -- 172.21.15.89:0/403415054 learned_addr learned my addr 172.21.15.89:0/403415054 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:24.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.506+0000 7fb02be00700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb02c0cec70 0x7fb02c158210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:40628/0 (socket says 172.21.15.89:40628) 2024-04-02T08:52:24.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.506+0000 7fb031000700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb02c15da50 0x7fb02c0b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:24.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.506+0000 7fb02b400700 1 -- 172.21.15.89:0/403415054 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb02c15da50 msgr2=0x7fb02c0b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:24.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.506+0000 7fb02b400700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb02c15da50 0x7fb02c0b5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.506+0000 7fb02b400700 1 -- 172.21.15.89:0/403415054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb02c0cec70 msgr2=0x7fb02c158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:24.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.506+0000 7fb02b400700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb02c0cec70 0x7fb02c158210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.506+0000 7fb02b400700 1 -- 172.21.15.89:0/403415054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c009550 con 0x7fb02c0d0b20 2024-04-02T08:52:24.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.506+0000 7fb02be00700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb02c0cec70 0x7fb02c158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:24.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.506+0000 7fb02b400700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb02c0d0b20 0x7fb02c158750 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fb01800e980 tx=0x7fb01800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:24.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.506+0000 7fb023e00700 1 -- 172.21.15.89:0/403415054 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb018009710 con 0x7fb02c0d0b20 2024-04-02T08:52:24.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.507+0000 7fb023e00700 1 -- 172.21.15.89:0/403415054 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01800c690 con 0x7fb02c0d0b20 2024-04-02T08:52:24.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.507+0000 7fb033000700 1 -- 172.21.15.89:0/403415054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02c1f45c0 con 0x7fb02c0d0b20 2024-04-02T08:52:24.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.507+0000 7fb023e00700 1 -- 172.21.15.89:0/403415054 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01800ca20 con 0x7fb02c0d0b20 2024-04-02T08:52:24.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.507+0000 7fb033000700 1 -- 172.21.15.89:0/403415054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02c1557f0 con 0x7fb02c0d0b20 2024-04-02T08:52:24.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.508+0000 7fb033000700 1 -- 172.21.15.89:0/403415054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02c1f4dc0 con 0x7fb02c0d0b20 2024-04-02T08:52:24.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.510+0000 7fb023e00700 1 -- 172.21.15.89:0/403415054 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 98) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fb01800cb80 con 0x7fb02c0d0b20 2024-04-02T08:52:24.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.511+0000 7fb023e00700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb00c06adb0 0x7fb00c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:24.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.511+0000 7fb023e00700 1 -- 172.21.15.89:0/403415054 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fb018089c70 con 0x7fb02c0d0b20 2024-04-02T08:52:24.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.511+0000 7fb02be00700 1 -- 172.21.15.89:0/403415054 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb00c06adb0 msgr2=0x7fb00c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:24.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.511+0000 7fb02be00700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb00c06adb0 0x7fb00c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:24.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.514+0000 7fb023e00700 1 -- 172.21.15.89:0/403415054 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb018054b80 con 0x7fb02c0d0b20 2024-04-02T08:52:24.533 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:24.534+0000 7fee12653240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T08:52:24.575 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:24.575+0000 7f53e59b5240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T08:52:24.630 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:24.631+0000 7fee12653240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T08:52:24.667 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:24.667+0000 7f53e59b5240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T08:52:24.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.712+0000 7fb02be00700 1 -- 172.21.15.89:0/403415054 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb00c06adb0 msgr2=0x7fb00c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:24.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.712+0000 7fb02be00700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb00c06adb0 0x7fb00c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:24.727 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:24.728+0000 7fee12653240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T08:52:24.765 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:24.766+0000 7f53e59b5240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T08:52:24.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.769+0000 7fb033000700 1 -- 172.21.15.89:0/403415054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb02c067720 con 0x7fb02c0d0b20 2024-04-02T08:52:24.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.769+0000 7fb023e00700 1 -- 172.21.15.89:0/403415054 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fb0180581a0 con 0x7fb02c0d0b20 2024-04-02T08:52:24.769 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:24.769 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7445,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/4112144997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":4112144997},{"type":"v1","addr":"172.21.15.113:6834","nonce":4112144997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":8416,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2047642384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2047642384},{"type":"v1","addr":"172.21.15.113:6835","nonce":2047642384}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":9030,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2201184833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2201184833},{"type":"v1","addr":"172.21.15.89:6835","nonce":2201184833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:56:17.192894+0000","modified":"2024-04-02T07:56:20.200137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9033},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9033":{"gid":9033,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3861363328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3861363328},{"type":"v1","addr":"172.21.15.89:6837","nonce":3861363328}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-02T08:52:24.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.772+0000 7fb033000700 1 -- 172.21.15.89:0/403415054 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb00c06adb0 msgr2=0x7fb00c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:24.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.772+0000 7fb033000700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb00c06adb0 0x7fb00c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.772+0000 7fb033000700 1 -- 172.21.15.89:0/403415054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb02c0d0b20 msgr2=0x7fb02c158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:24.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.772+0000 7fb033000700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb02c0d0b20 0x7fb02c158750 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fb01800e980 tx=0x7fb01800ec90 comp rx=0 tx=0).stop 2024-04-02T08:52:24.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.772+0000 7fb033000700 1 -- 172.21.15.89:0/403415054 shutdown_connections 2024-04-02T08:52:24.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.772+0000 7fb033000700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb00c06adb0 0x7fb00c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.772+0000 7fb033000700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb02c0cec70 0x7fb02c158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.772+0000 7fb033000700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb02c15da50 0x7fb02c0b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.772+0000 7fb033000700 1 --2- 172.21.15.89:0/403415054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb02c0d0b20 0x7fb02c158750 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.772+0000 7fb033000700 1 -- 172.21.15.89:0/403415054 >> 172.21.15.89:0/403415054 conn(0x7fb02c0666a0 msgr2=0x7fb02c15cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:24.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.772+0000 7fb033000700 1 -- 172.21.15.89:0/403415054 shutdown_connections 2024-04-02T08:52:24.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.772+0000 7fb033000700 1 -- 172.21.15.89:0/403415054 wait complete. 2024-04-02T08:52:24.772 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 73 2024-04-02T08:52:24.790 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:52:24.878 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:24.879+0000 7f53e59b5240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T08:52:24.941 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:24.942+0000 7fee12653240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T08:52:24.982 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:24.982+0000 7f53e59b5240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T08:52:24.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.995+0000 7fdd58a00700 1 Processor -- start 2024-04-02T08:52:24.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.995+0000 7fdd58a00700 1 -- start start 2024-04-02T08:52:24.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.996+0000 7fdd58a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 0x7fdd540d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:24.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.996+0000 7fdd58a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd540ceda0 con 0x7fdd540cf230 2024-04-02T08:52:24.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.996+0000 7fdd58a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd541612f0 con 0x7fdd54161590 2024-04-02T08:52:24.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.996+0000 7fdd58a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd540ce470 con 0x7fdd54160f10 2024-04-02T08:52:24.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.996+0000 7fdd4be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdd540cf230 0x7fdd540d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51886/0 (socket says 172.21.15.89:51886) 2024-04-02T08:52:24.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.996+0000 7fdd4be00700 1 -- 172.21.15.89:0/212090956 learned_addr learned my addr 172.21.15.89:0/212090956 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:24.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.997+0000 7fdd51e00700 1 --2- 172.21.15.89:0/212090956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 0x7fdd540d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:24.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.997+0000 7fdd51400700 1 -- 172.21.15.89:0/212090956 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784740647 0 0) 0x7fdd541612f0 con 0x7fdd54161590 2024-04-02T08:52:24.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.997+0000 7fdd51400700 1 -- 172.21.15.89:0/212090956 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd380035b0 con 0x7fdd54161590 2024-04-02T08:52:24.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.997+0000 7fdd51400700 1 -- 172.21.15.89:0/212090956 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842450989 0 0) 0x7fdd540ceda0 con 0x7fdd540cf230 2024-04-02T08:52:24.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.997+0000 7fdd51400700 1 -- 172.21.15.89:0/212090956 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd541612f0 con 0x7fdd540cf230 2024-04-02T08:52:24.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.997+0000 7fdd51e00700 1 -- 172.21.15.89:0/212090956 >> v1:172.21.15.113:6789/0 conn(0x7fdd54161590 legacy=0x7fdd540b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:24.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.997+0000 7fdd51e00700 1 -- 172.21.15.89:0/212090956 >> v1:172.21.15.89:6789/0 conn(0x7fdd540cf230 legacy=0x7fdd540d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:24.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.997+0000 7fdd51e00700 1 -- 172.21.15.89:0/212090956 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd540ce750 con 0x7fdd54160f10 2024-04-02T08:52:24.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.997+0000 7fdd51e00700 1 --2- 172.21.15.89:0/212090956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 0x7fdd540d18b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdd440098a0 tx=0x7fdd44009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a3665cd69d3ffa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:24.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.998+0000 7fdd51400700 1 -- 172.21.15.89:0/212090956 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44011830 con 0x7fdd54160f10 2024-04-02T08:52:24.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.998+0000 7fdd51400700 1 -- 172.21.15.89:0/212090956 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd44011990 con 0x7fdd54160f10 2024-04-02T08:52:24.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.998+0000 7fdd51400700 1 -- 172.21.15.89:0/212090956 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44011d20 con 0x7fdd54160f10 2024-04-02T08:52:24.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.998+0000 7fdd58a00700 1 -- 172.21.15.89:0/212090956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 msgr2=0x7fdd540d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:24.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.998+0000 7fdd58a00700 1 --2- 172.21.15.89:0/212090956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 0x7fdd540d18b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdd440098a0 tx=0x7fdd44009bb0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.998+0000 7fdd58a00700 1 -- 172.21.15.89:0/212090956 shutdown_connections 2024-04-02T08:52:24.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.998+0000 7fdd58a00700 1 --2- 172.21.15.89:0/212090956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd54160f10 0x7fdd540d18b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:24.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.998+0000 7fdd58a00700 1 -- 172.21.15.89:0/212090956 >> 172.21.15.89:0/212090956 conn(0x7fdd5408aff0 msgr2=0x7fdd540d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:24.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.998+0000 7fdd58a00700 1 -- 172.21.15.89:0/212090956 shutdown_connections 2024-04-02T08:52:24.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:24.998+0000 7fdd58a00700 1 -- 172.21.15.89:0/212090956 wait complete. 2024-04-02T08:52:25.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.001+0000 7fdd58a00700 1 Processor -- start 2024-04-02T08:52:25.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.001+0000 7fdd58a00700 1 -- start start 2024-04-02T08:52:25.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.002+0000 7fdd58a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd540cf230 0x7fdd541f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.002+0000 7fdd58a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd54160f10 0x7fdd5415ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.002+0000 7fdd58a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd54161590 0x7fdd5415f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.002+0000 7fdd58a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd540b6c60 con 0x7fdd54161590 2024-04-02T08:52:25.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.002+0000 7fdd58a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd540b6f40 con 0x7fdd54160f10 2024-04-02T08:52:25.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.002+0000 7fdd58a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd540b7220 con 0x7fdd540cf230 2024-04-02T08:52:25.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.002+0000 7fdd51e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd540cf230 0x7fdd541f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:25.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.002+0000 7fdd4be00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd54160f10 0x7fdd5415ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:25.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.002+0000 7fdd52800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd54161590 0x7fdd5415f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:25.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.002+0000 7fdd51e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd540cf230 0x7fdd541f6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:45878/0 (socket says 172.21.15.89:45878) 2024-04-02T08:52:25.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.002+0000 7fdd51e00700 1 -- 172.21.15.89:0/535764140 learned_addr learned my addr 172.21.15.89:0/535764140 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:25.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.003+0000 7fdd51e00700 1 -- 172.21.15.89:0/535764140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd54160f10 msgr2=0x7fdd5415ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:25.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.003+0000 7fdd51e00700 1 --2- 172.21.15.89:0/535764140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd54160f10 0x7fdd5415ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.003+0000 7fdd51e00700 1 -- 172.21.15.89:0/535764140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd54161590 msgr2=0x7fdd5415f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:25.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.003+0000 7fdd51e00700 1 --2- 172.21.15.89:0/535764140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd54161590 0x7fdd5415f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.003+0000 7fdd51e00700 1 -- 172.21.15.89:0/535764140 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd440094a0 con 0x7fdd540cf230 2024-04-02T08:52:25.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.003+0000 7fdd52800700 1 --2- 172.21.15.89:0/535764140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd54161590 0x7fdd5415f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:25.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.003+0000 7fdd51e00700 1 --2- 172.21.15.89:0/535764140 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd540cf230 0x7fdd541f6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdd44000c00 tx=0x7fdd44004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:25.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.003+0000 7fdd4a000700 1 -- 172.21.15.89:0/535764140 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4401a6a0 con 0x7fdd540cf230 2024-04-02T08:52:25.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.003+0000 7fdd4a000700 1 -- 172.21.15.89:0/535764140 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4401a800 con 0x7fdd540cf230 2024-04-02T08:52:25.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.003+0000 7fdd4a000700 1 -- 172.21.15.89:0/535764140 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4401abb0 con 0x7fdd540cf230 2024-04-02T08:52:25.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.003+0000 7fdd58a00700 1 -- 172.21.15.89:0/535764140 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd540b7500 con 0x7fdd540cf230 2024-04-02T08:52:25.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.004+0000 7fdd58a00700 1 -- 172.21.15.89:0/535764140 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd540b8670 con 0x7fdd540cf230 2024-04-02T08:52:25.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.005+0000 7fdd58a00700 1 -- 172.21.15.89:0/535764140 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd540b7a20 con 0x7fdd540cf230 2024-04-02T08:52:25.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.006+0000 7fdd4a000700 1 -- 172.21.15.89:0/535764140 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 98) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fdd44022070 con 0x7fdd540cf230 2024-04-02T08:52:25.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.006+0000 7fdd4a000700 1 --2- 172.21.15.89:0/535764140 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd3006adb0 0x7fdd3006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.007+0000 7fdd4a000700 1 -- 172.21.15.89:0/535764140 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fdd4408a4a0 con 0x7fdd540cf230 2024-04-02T08:52:25.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.007+0000 7fdd4be00700 1 -- 172.21.15.89:0/535764140 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd3006adb0 msgr2=0x7fdd3006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:25.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.007+0000 7fdd4be00700 1 --2- 172.21.15.89:0/535764140 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd3006adb0 0x7fdd3006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:25.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.011+0000 7fdd4a000700 1 -- 172.21.15.89:0/535764140 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdd540b7a20 con 0x7fdd540cf230 2024-04-02T08:52:25.196 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:25.197+0000 7fee12653240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T08:52:25.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.208+0000 7fdd4be00700 1 -- 172.21.15.89:0/535764140 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd3006adb0 msgr2=0x7fdd3006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:25.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.208+0000 7fdd4be00700 1 --2- 172.21.15.89:0/535764140 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd3006adb0 0x7fdd3006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:25.242 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:25.243+0000 7f53e59b5240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T08:52:25.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.275+0000 7fdd58a00700 1 -- 172.21.15.89:0/535764140 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd540b7a20 con 0x7fdd540cf230 2024-04-02T08:52:25.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.277+0000 7fdd4a000700 1 -- 172.21.15.89:0/535764140 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v159) v1 ==== 74+0+17282 (secure 0 0 0) 0x7fdd540b7a20 con 0x7fdd540cf230 2024-04-02T08:52:25.277 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:25.278 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":159,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:52:22.879866+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:56:15.040512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-02T07:56:16.179443+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":159,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/23967222":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/3508000969":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/75408226":"2024-04-03T07:56:33.905850+0000","172.21.15.89:6833/43357":"2024-04-03T07:56:33.905850+0000","172.21.15.89:6832/43357":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/2379953994":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/65229891":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/1849807649":"2024-04-03T07:56:33.905850+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:52:25.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.280+0000 7fdd58a00700 1 -- 172.21.15.89:0/535764140 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd3006adb0 msgr2=0x7fdd3006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:25.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.280+0000 7fdd58a00700 1 --2- 172.21.15.89:0/535764140 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd3006adb0 0x7fdd3006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.280+0000 7fdd58a00700 1 -- 172.21.15.89:0/535764140 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd540cf230 msgr2=0x7fdd541f6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:25.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.280+0000 7fdd58a00700 1 --2- 172.21.15.89:0/535764140 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd540cf230 0x7fdd541f6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdd44000c00 tx=0x7fdd44004750 comp rx=0 tx=0).stop 2024-04-02T08:52:25.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.280+0000 7fdd58a00700 1 -- 172.21.15.89:0/535764140 shutdown_connections 2024-04-02T08:52:25.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.280+0000 7fdd58a00700 1 --2- 172.21.15.89:0/535764140 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fdd3006adb0 0x7fdd3006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.280+0000 7fdd58a00700 1 --2- 172.21.15.89:0/535764140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd54161590 0x7fdd5415f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.280+0000 7fdd58a00700 1 --2- 172.21.15.89:0/535764140 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd540cf230 0x7fdd541f6210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.280+0000 7fdd58a00700 1 --2- 172.21.15.89:0/535764140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd54160f10 0x7fdd5415ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.280+0000 7fdd58a00700 1 -- 172.21.15.89:0/535764140 >> 172.21.15.89:0/535764140 conn(0x7fdd5408aff0 msgr2=0x7fdd540d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:25.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.282+0000 7fdd58a00700 1 -- 172.21.15.89:0/535764140 shutdown_connections 2024-04-02T08:52:25.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.282+0000 7fdd58a00700 1 -- 172.21.15.89:0/535764140 wait complete. 2024-04-02T08:52:25.298 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T07:56:15.040512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T07:56:16.179443+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '159', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 113, 'snap_epoch': 159, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T08:52:25.299 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T08:52:25.387 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:25.388+0000 7fee12653240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T08:52:25.478 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:25.479+0000 7fee12653240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T08:52:25.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.505+0000 7f8fa4c00700 1 Processor -- start 2024-04-02T08:52:25.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.505+0000 7f8fa4c00700 1 -- start start 2024-04-02T08:52:25.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.506+0000 7f8fa4c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8fa015a5c0 0x7f8fa00709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.506+0000 7f8fa4c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa015dfe0 0x7f8fa00ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.506+0000 7f8fa4c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8fa00ce670 0x7f8fa00b2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.506+0000 7f8fa4c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa00ceb00 con 0x7f8fa00ce670 2024-04-02T08:52:25.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.507+0000 7f8fa4c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa00cb310 con 0x7f8fa015a5c0 2024-04-02T08:52:25.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.507+0000 7f8fa4c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa0070f00 con 0x7f8fa015dfe0 2024-04-02T08:52:25.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.507+0000 7f8f9e800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8fa00ce670 0x7f8fa00b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:25.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.507+0000 7f8f97e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa015dfe0 0x7f8fa00ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:25.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.507+0000 7f8f9de00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8fa015a5c0 0x7f8fa00709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:25.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.507+0000 7f8f9e800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8fa00ce670 0x7f8fa00b2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39230/0 (socket says 172.21.15.89:39230) 2024-04-02T08:52:25.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.507+0000 7f8f9e800700 1 -- 172.21.15.89:0/4183327309 learned_addr learned my addr 172.21.15.89:0/4183327309 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:25.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.507+0000 7f8f9e800700 1 -- 172.21.15.89:0/4183327309 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa015dfe0 msgr2=0x7f8fa00ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:25.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.507+0000 7f8f9e800700 1 --2- 172.21.15.89:0/4183327309 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa015dfe0 0x7f8fa00ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.507+0000 7f8f9e800700 1 -- 172.21.15.89:0/4183327309 >> v2:172.21.15.113:3300/0 conn(0x7f8fa015a5c0 msgr2=0x7f8fa00709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:25.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.507+0000 7f8f9e800700 1 --2- 172.21.15.89:0/4183327309 >> v2:172.21.15.113:3300/0 conn(0x7f8fa015a5c0 0x7f8fa00709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.507+0000 7f8f9e800700 1 -- 172.21.15.89:0/4183327309 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa00711a0 con 0x7f8fa00ce670 2024-04-02T08:52:25.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.508+0000 7f8f9e800700 1 --2- 172.21.15.89:0/4183327309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00ce670 0x7f8fa00b2640 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0072e70 tx=0x7f8f9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db445e06dde4346d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:25.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.508+0000 7f8f9d400700 1 -- 172.21.15.89:0/4183327309 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9800e070 con 0x7f8fa00ce670 2024-04-02T08:52:25.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.508+0000 7f8f9d400700 1 -- 172.21.15.89:0/4183327309 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9800b910 con 0x7f8fa00ce670 2024-04-02T08:52:25.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.508+0000 7f8f9d400700 1 -- 172.21.15.89:0/4183327309 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9800bcc0 con 0x7f8fa00ce670 2024-04-02T08:52:25.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.509+0000 7f8fa4c00700 1 -- 172.21.15.89:0/4183327309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00ce670 msgr2=0x7f8fa00b2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:25.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.509+0000 7f8fa4c00700 1 --2- 172.21.15.89:0/4183327309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00ce670 0x7f8fa00b2640 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0072e70 tx=0x7f8f9800b330 comp rx=0 tx=0).stop 2024-04-02T08:52:25.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.509+0000 7f8fa4c00700 1 -- 172.21.15.89:0/4183327309 shutdown_connections 2024-04-02T08:52:25.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.509+0000 7f8fa4c00700 1 --2- 172.21.15.89:0/4183327309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00ce670 0x7f8fa00b2640 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.509+0000 7f8fa4c00700 1 --2- 172.21.15.89:0/4183327309 >> v2:172.21.15.113:3300/0 conn(0x7f8fa015a5c0 0x7f8fa00709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.509+0000 7f8fa4c00700 1 --2- 172.21.15.89:0/4183327309 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa015dfe0 0x7f8fa00ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.509+0000 7f8fa4c00700 1 -- 172.21.15.89:0/4183327309 >> 172.21.15.89:0/4183327309 conn(0x7f8fa00b0d70 msgr2=0x7f8fa00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:25.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.509+0000 7f8fa4c00700 1 -- 172.21.15.89:0/4183327309 shutdown_connections 2024-04-02T08:52:25.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.509+0000 7f8fa4c00700 1 -- 172.21.15.89:0/4183327309 wait complete. 2024-04-02T08:52:25.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.510+0000 7f8fa4c00700 1 Processor -- start 2024-04-02T08:52:25.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.511+0000 7f8fa4c00700 1 -- start start 2024-04-02T08:52:25.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.511+0000 7f8fa4c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fa00ce670 0x7f8fa00cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.511+0000 7f8fa4c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa015a5c0 0x7f8fa01456b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8fa4c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa015dfe0 0x7f8fa0146e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8fa4c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa0159800 con 0x7f8fa015dfe0 2024-04-02T08:52:25.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8fa4c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa0145bf0 con 0x7f8fa00ce670 2024-04-02T08:52:25.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8fa4c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa0158bf0 con 0x7f8fa015a5c0 2024-04-02T08:52:25.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8f9e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa015dfe0 0x7f8fa0146e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:25.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8f9e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa015dfe0 0x7f8fa0146e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39232/0 (socket says 172.21.15.89:39232) 2024-04-02T08:52:25.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8f9e800700 1 -- 172.21.15.89:0/2649980300 learned_addr learned my addr 172.21.15.89:0/2649980300 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:25.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8f9de00700 1 --2- 172.21.15.89:0/2649980300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fa00ce670 0x7f8fa00cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:25.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8f97e00700 1 --2- 172.21.15.89:0/2649980300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa015a5c0 0x7f8fa01456b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:25.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8f9e800700 1 -- 172.21.15.89:0/2649980300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa015a5c0 msgr2=0x7f8fa01456b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:25.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8f9e800700 1 --2- 172.21.15.89:0/2649980300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa015a5c0 0x7f8fa01456b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8f9e800700 1 -- 172.21.15.89:0/2649980300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fa00ce670 msgr2=0x7f8fa00cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:25.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8f9e800700 1 --2- 172.21.15.89:0/2649980300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fa00ce670 0x7f8fa00cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8f9e800700 1 -- 172.21.15.89:0/2649980300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f98009b30 con 0x7f8fa015dfe0 2024-04-02T08:52:25.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.512+0000 7f8f9e800700 1 --2- 172.21.15.89:0/2649980300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa015dfe0 0x7f8fa0146e20 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f8fa006c130 tx=0x7f8f98004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:25.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.513+0000 7f8f96000700 1 -- 172.21.15.89:0/2649980300 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f980099b0 con 0x7f8fa015dfe0 2024-04-02T08:52:25.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.513+0000 7f8f96000700 1 -- 172.21.15.89:0/2649980300 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9800ce80 con 0x7f8fa015dfe0 2024-04-02T08:52:25.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.513+0000 7f8fa4c00700 1 -- 172.21.15.89:0/2649980300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa0158ed0 con 0x7f8fa015dfe0 2024-04-02T08:52:25.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.513+0000 7f8f96000700 1 -- 172.21.15.89:0/2649980300 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98004490 con 0x7f8fa015dfe0 2024-04-02T08:52:25.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.513+0000 7f8fa4c00700 1 -- 172.21.15.89:0/2649980300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa0159490 con 0x7f8fa015dfe0 2024-04-02T08:52:25.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.514+0000 7f8fa4c00700 1 -- 172.21.15.89:0/2649980300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa0159650 con 0x7f8fa015dfe0 2024-04-02T08:52:25.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.516+0000 7f8f96000700 1 -- 172.21.15.89:0/2649980300 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 98) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f8f980045f0 con 0x7f8fa015dfe0 2024-04-02T08:52:25.514 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:25.516+0000 7f53e59b5240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T08:52:25.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.516+0000 7f8f96000700 1 --2- 172.21.15.89:0/2649980300 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8f7c06adb0 0x7f8f7c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.517+0000 7f8f96000700 1 -- 172.21.15.89:0/2649980300 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f8f9808a910 con 0x7f8fa015dfe0 2024-04-02T08:52:25.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.517+0000 7f8f9de00700 1 -- 172.21.15.89:0/2649980300 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8f7c06adb0 msgr2=0x7f8f7c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:25.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.517+0000 7f8f9de00700 1 --2- 172.21.15.89:0/2649980300 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8f7c06adb0 0x7f8f7c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:25.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.521+0000 7f8f96000700 1 -- 172.21.15.89:0/2649980300 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8f980556e0 con 0x7f8fa015dfe0 2024-04-02T08:52:25.713 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:25.715+0000 7f53e59b5240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T08:52:25.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.718+0000 7f8f9de00700 1 -- 172.21.15.89:0/2649980300 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8f7c06adb0 msgr2=0x7f8f7c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:25.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.718+0000 7f8f9de00700 1 --2- 172.21.15.89:0/2649980300 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8f7c06adb0 0x7f8f7c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:25.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.730+0000 7f8fa4c00700 1 -- 172.21.15.89:0/2649980300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8fa0067720 con 0x7f8fa015dfe0 2024-04-02T08:52:25.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.730+0000 7f8f96000700 1 -- 172.21.15.89:0/2649980300 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v159) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8f9801f0a0 con 0x7f8fa015dfe0 2024-04-02T08:52:25.729 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T08:52:25.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.733+0000 7f8fa4c00700 1 -- 172.21.15.89:0/2649980300 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8f7c06adb0 msgr2=0x7f8f7c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:25.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.733+0000 7f8fa4c00700 1 --2- 172.21.15.89:0/2649980300 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8f7c06adb0 0x7f8f7c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.733+0000 7f8fa4c00700 1 -- 172.21.15.89:0/2649980300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa015dfe0 msgr2=0x7f8fa0146e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:25.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.733+0000 7f8fa4c00700 1 --2- 172.21.15.89:0/2649980300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa015dfe0 0x7f8fa0146e20 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f8fa006c130 tx=0x7f8f98004ad0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.733+0000 7f8fa4c00700 1 -- 172.21.15.89:0/2649980300 shutdown_connections 2024-04-02T08:52:25.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.733+0000 7f8fa4c00700 1 --2- 172.21.15.89:0/2649980300 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8f7c06adb0 0x7f8f7c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.733+0000 7f8fa4c00700 1 --2- 172.21.15.89:0/2649980300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fa00ce670 0x7f8fa00cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.733+0000 7f8fa4c00700 1 --2- 172.21.15.89:0/2649980300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa015dfe0 0x7f8fa0146e20 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.733+0000 7f8fa4c00700 1 --2- 172.21.15.89:0/2649980300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa015a5c0 0x7f8fa01456b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.733+0000 7f8fa4c00700 1 -- 172.21.15.89:0/2649980300 >> 172.21.15.89:0/2649980300 conn(0x7f8fa00b0d70 msgr2=0x7f8fa0148220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:25.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.733+0000 7f8fa4c00700 1 -- 172.21.15.89:0/2649980300 shutdown_connections 2024-04-02T08:52:25.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.733+0000 7f8fa4c00700 1 -- 172.21.15.89:0/2649980300 wait complete. 2024-04-02T08:52:25.749 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T08:52:25.816 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:25.817+0000 7f53e59b5240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T08:52:25.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.953+0000 7fe47b200700 1 Processor -- start 2024-04-02T08:52:25.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.953+0000 7fe47b200700 1 -- start start 2024-04-02T08:52:25.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.954+0000 7fe47b200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe4740cbb30 0x7fe4740cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.955+0000 7fe47b200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4740cc450 con 0x7fe4740cd480 2024-04-02T08:52:25.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.955+0000 7fe47b200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4740cdae0 con 0x7fe4740cd020 2024-04-02T08:52:25.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.955+0000 7fe47b200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4740cd8f0 con 0x7fe4740cbb30 2024-04-02T08:52:25.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.955+0000 7fe479200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe4740cd480 0x7fe47406e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51888/0 (socket says 172.21.15.89:51888) 2024-04-02T08:52:25.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.955+0000 7fe479200700 1 -- 172.21.15.89:0/770480029 learned_addr learned my addr 172.21.15.89:0/770480029 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:25.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.955+0000 7fe473e00700 1 --2- 172.21.15.89:0/770480029 >> v2:172.21.15.113:3300/0 conn(0x7fe4740cbb30 0x7fe4740cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:25.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.955+0000 7fe473400700 1 -- 172.21.15.89:0/770480029 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154379343 0 0) 0x7fe4740cc450 con 0x7fe4740cd480 2024-04-02T08:52:25.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.955+0000 7fe473400700 1 -- 172.21.15.89:0/770480029 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4580035b0 con 0x7fe4740cd480 2024-04-02T08:52:25.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.955+0000 7fe473e00700 1 -- 172.21.15.89:0/770480029 >> v1:172.21.15.113:6789/0 conn(0x7fe4740cd020 legacy=0x7fe4740b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:25.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.955+0000 7fe473e00700 1 -- 172.21.15.89:0/770480029 >> v1:172.21.15.89:6789/0 conn(0x7fe4740cd480 legacy=0x7fe47406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:25.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.956+0000 7fe473e00700 1 -- 172.21.15.89:0/770480029 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4740cddc0 con 0x7fe4740cbb30 2024-04-02T08:52:25.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.956+0000 7fe473e00700 1 --2- 172.21.15.89:0/770480029 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4740cbb30 0x7fe4740cbf10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe464009840 tx=0x7fe464009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5cd798c329acfca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:25.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.956+0000 7fe473400700 1 -- 172.21.15.89:0/770480029 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464011830 con 0x7fe4740cbb30 2024-04-02T08:52:25.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.956+0000 7fe473400700 1 -- 172.21.15.89:0/770480029 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe464011990 con 0x7fe4740cbb30 2024-04-02T08:52:25.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.956+0000 7fe473400700 1 -- 172.21.15.89:0/770480029 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464011d20 con 0x7fe4740cbb30 2024-04-02T08:52:25.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.957+0000 7fe47b200700 1 -- 172.21.15.89:0/770480029 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4740cbb30 msgr2=0x7fe4740cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:25.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.957+0000 7fe47b200700 1 --2- 172.21.15.89:0/770480029 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4740cbb30 0x7fe4740cbf10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe464009840 tx=0x7fe464009b50 comp rx=0 tx=0).stop 2024-04-02T08:52:25.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.957+0000 7fe47b200700 1 -- 172.21.15.89:0/770480029 shutdown_connections 2024-04-02T08:52:25.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.957+0000 7fe47b200700 1 --2- 172.21.15.89:0/770480029 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4740cbb30 0x7fe4740cbf10 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.957+0000 7fe47b200700 1 -- 172.21.15.89:0/770480029 >> 172.21.15.89:0/770480029 conn(0x7fe4740b0d70 msgr2=0x7fe4740b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:25.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.957+0000 7fe47b200700 1 -- 172.21.15.89:0/770480029 shutdown_connections 2024-04-02T08:52:25.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.957+0000 7fe47b200700 1 -- 172.21.15.89:0/770480029 wait complete. 2024-04-02T08:52:25.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.959+0000 7fe47b200700 1 Processor -- start 2024-04-02T08:52:25.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.959+0000 7fe47b200700 1 -- start start 2024-04-02T08:52:25.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.960+0000 7fe47b200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4740cbb30 0x7fe474152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.960+0000 7fe47b200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4740cd020 0x7fe4740af560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.960+0000 7fe47b200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4740cd480 0x7fe4740afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.960+0000 7fe47b200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe474150e60 con 0x7fe4740cd020 2024-04-02T08:52:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.960+0000 7fe47b200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe474150110 con 0x7fe4740cbb30 2024-04-02T08:52:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.960+0000 7fe47b200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4740b30f0 con 0x7fe4740cd480 2024-04-02T08:52:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.961+0000 7fe46be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4740cd020 0x7fe4740af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.961+0000 7fe473e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4740cbb30 0x7fe474152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.961+0000 7fe473e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4740cbb30 0x7fe474152290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:40662/0 (socket says 172.21.15.89:40662) 2024-04-02T08:52:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.961+0000 7fe473e00700 1 -- 172.21.15.89:0/2675174073 learned_addr learned my addr 172.21.15.89:0/2675174073 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.961+0000 7fe46be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4740cd020 0x7fe4740af560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39240/0 (socket says 172.21.15.89:39240) 2024-04-02T08:52:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.961+0000 7fe479200700 1 --2- 172.21.15.89:0/2675174073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4740cd480 0x7fe4740afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.961+0000 7fe46be00700 1 -- 172.21.15.89:0/2675174073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4740cd480 msgr2=0x7fe4740afaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.961+0000 7fe46be00700 1 --2- 172.21.15.89:0/2675174073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4740cd480 0x7fe4740afaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.961+0000 7fe46be00700 1 -- 172.21.15.89:0/2675174073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4740cbb30 msgr2=0x7fe474152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.961+0000 7fe46be00700 1 --2- 172.21.15.89:0/2675174073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4740cbb30 0x7fe474152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:25.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.961+0000 7fe46be00700 1 -- 172.21.15.89:0/2675174073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640094a0 con 0x7fe4740cd020 2024-04-02T08:52:25.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.961+0000 7fe473e00700 1 --2- 172.21.15.89:0/2675174073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4740cbb30 0x7fe474152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:25.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.962+0000 7fe46be00700 1 --2- 172.21.15.89:0/2675174073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4740cd020 0x7fe4740af560 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fe45c000f60 tx=0x7fe45c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:25.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.962+0000 7fe470c00700 1 -- 172.21.15.89:0/2675174073 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c013070 con 0x7fe4740cd020 2024-04-02T08:52:25.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.962+0000 7fe470c00700 1 -- 172.21.15.89:0/2675174073 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45c00e2e0 con 0x7fe4740cd020 2024-04-02T08:52:25.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.962+0000 7fe47b200700 1 -- 172.21.15.89:0/2675174073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4741527d0 con 0x7fe4740cd020 2024-04-02T08:52:25.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.962+0000 7fe470c00700 1 -- 172.21.15.89:0/2675174073 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c0052d0 con 0x7fe4740cd020 2024-04-02T08:52:25.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.962+0000 7fe47b200700 1 -- 172.21.15.89:0/2675174073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe474150660 con 0x7fe4740cd020 2024-04-02T08:52:25.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.963+0000 7fe47b200700 1 -- 172.21.15.89:0/2675174073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe474067720 con 0x7fe4740cd020 2024-04-02T08:52:25.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.964+0000 7fe470c00700 1 -- 172.21.15.89:0/2675174073 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 98) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fe45c005430 con 0x7fe4740cd020 2024-04-02T08:52:25.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.965+0000 7fe470c00700 1 --2- 172.21.15.89:0/2675174073 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe45006adb0 0x7fe45006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:25.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.965+0000 7fe473e00700 1 -- 172.21.15.89:0/2675174073 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe45006adb0 msgr2=0x7fe45006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:25.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.965+0000 7fe470c00700 1 -- 172.21.15.89:0/2675174073 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fe45c08cef0 con 0x7fe4740cd020 2024-04-02T08:52:25.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.965+0000 7fe473e00700 1 --2- 172.21.15.89:0/2675174073 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe45006adb0 0x7fe45006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:25.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:25.974+0000 7fe470c00700 1 -- 172.21.15.89:0/2675174073 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe45c057b90 con 0x7fe4740cd020 2024-04-02T08:52:26.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.166+0000 7fe473e00700 1 -- 172.21.15.89:0/2675174073 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe45006adb0 msgr2=0x7fe45006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:26.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.166+0000 7fe473e00700 1 --2- 172.21.15.89:0/2675174073 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe45006adb0 0x7fe45006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:26.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.187+0000 7fe47b200700 1 -- 172.21.15.89:0/2675174073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe474155d90 con 0x7fe4740cd020 2024-04-02T08:52:26.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.187+0000 7fe470c00700 1 -- 172.21.15.89:0/2675174073 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v159) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe45c05b1b0 con 0x7fe4740cd020 2024-04-02T08:52:26.186 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T08:52:26.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.190+0000 7fe47b200700 1 -- 172.21.15.89:0/2675174073 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe45006adb0 msgr2=0x7fe45006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:26.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.190+0000 7fe47b200700 1 --2- 172.21.15.89:0/2675174073 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe45006adb0 0x7fe45006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.190+0000 7fe47b200700 1 -- 172.21.15.89:0/2675174073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4740cd020 msgr2=0x7fe4740af560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:26.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.190+0000 7fe47b200700 1 --2- 172.21.15.89:0/2675174073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4740cd020 0x7fe4740af560 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fe45c000f60 tx=0x7fe45c001a40 comp rx=0 tx=0).stop 2024-04-02T08:52:26.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.190+0000 7fe47b200700 1 -- 172.21.15.89:0/2675174073 shutdown_connections 2024-04-02T08:52:26.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.190+0000 7fe47b200700 1 --2- 172.21.15.89:0/2675174073 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe45006adb0 0x7fe45006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.190+0000 7fe47b200700 1 --2- 172.21.15.89:0/2675174073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4740cbb30 0x7fe474152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.190+0000 7fe47b200700 1 --2- 172.21.15.89:0/2675174073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4740cd480 0x7fe4740afaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.190+0000 7fe47b200700 1 --2- 172.21.15.89:0/2675174073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4740cd020 0x7fe4740af560 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.190+0000 7fe47b200700 1 -- 172.21.15.89:0/2675174073 >> 172.21.15.89:0/2675174073 conn(0x7fe4740b0d70 msgr2=0x7fe47406e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:26.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.190+0000 7fe47b200700 1 -- 172.21.15.89:0/2675174073 shutdown_connections 2024-04-02T08:52:26.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.190+0000 7fe47b200700 1 -- 172.21.15.89:0/2675174073 wait complete. 2024-04-02T08:52:26.202 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:26.203+0000 7fee12653240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T08:52:26.206 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T08:52:26.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.406+0000 7f0a5bc00700 1 Processor -- start 2024-04-02T08:52:26.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.407+0000 7f0a5bc00700 1 -- start start 2024-04-02T08:52:26.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.408+0000 7f0a5bc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0a540cbb30 0x7f0a540cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:26.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.408+0000 7f0a5bc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0a540cd020 0x7f0a540cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:26.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.408+0000 7f0a5bc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a540cc450 con 0x7f0a540b2070 2024-04-02T08:52:26.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.408+0000 7f0a5bc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a540cdae0 con 0x7f0a540cd020 2024-04-02T08:52:26.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.408+0000 7f0a5bc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a540cddc0 con 0x7f0a540cbb30 2024-04-02T08:52:26.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.408+0000 7f0a53e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0a540cd020 0x7f0a540cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.408+0000 7f0a53e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0a540cd020 0x7f0a540cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39252/0 (socket says 172.21.15.89:39252) 2024-04-02T08:52:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.408+0000 7f0a53e00700 1 -- 172.21.15.89:0/4116580419 learned_addr learned my addr 172.21.15.89:0/4116580419 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.408+0000 7f0a59200700 1 --2- 172.21.15.89:0/4116580419 >> v2:172.21.15.113:3300/0 conn(0x7f0a540cbb30 0x7f0a540cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.409+0000 7f0a53e00700 1 -- 172.21.15.89:0/4116580419 >> v2:172.21.15.113:3300/0 conn(0x7f0a540cbb30 msgr2=0x7f0a540cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.409+0000 7f0a53e00700 1 --2- 172.21.15.89:0/4116580419 >> v2:172.21.15.113:3300/0 conn(0x7f0a540cbb30 0x7f0a540cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.409+0000 7f0a53e00700 1 -- 172.21.15.89:0/4116580419 >> v1:172.21.15.113:6789/0 conn(0x7f0a540b2070 legacy=0x7f0a5406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.409+0000 7f0a53e00700 1 -- 172.21.15.89:0/4116580419 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a540ce0a0 con 0x7f0a540cd020 2024-04-02T08:52:26.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.409+0000 7f0a53e00700 1 --2- 172.21.15.89:0/4116580419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a540cd020 0x7f0a540cd480 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f0a5406fd00 tx=0x7f0a40009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=861a12699af7a418 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:26.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.409+0000 7f0a53400700 1 -- 172.21.15.89:0/4116580419 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40011830 con 0x7f0a540cd020 2024-04-02T08:52:26.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.409+0000 7f0a53400700 1 -- 172.21.15.89:0/4116580419 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a40011990 con 0x7f0a540cd020 2024-04-02T08:52:26.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.409+0000 7f0a53400700 1 -- 172.21.15.89:0/4116580419 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40011d20 con 0x7f0a540cd020 2024-04-02T08:52:26.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.410+0000 7f0a5bc00700 1 -- 172.21.15.89:0/4116580419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a540cd020 msgr2=0x7f0a540cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:26.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.410+0000 7f0a5bc00700 1 --2- 172.21.15.89:0/4116580419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a540cd020 0x7f0a540cd480 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f0a5406fd00 tx=0x7f0a40009bd0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.410+0000 7f0a5bc00700 1 -- 172.21.15.89:0/4116580419 shutdown_connections 2024-04-02T08:52:26.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.410+0000 7f0a5bc00700 1 --2- 172.21.15.89:0/4116580419 >> v2:172.21.15.113:3300/0 conn(0x7f0a540cbb30 0x7f0a540cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.410+0000 7f0a5bc00700 1 --2- 172.21.15.89:0/4116580419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a540cd020 0x7f0a540cd480 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.410+0000 7f0a5bc00700 1 -- 172.21.15.89:0/4116580419 >> 172.21.15.89:0/4116580419 conn(0x7f0a540b0d70 msgr2=0x7f0a540b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:26.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.410+0000 7f0a5bc00700 1 -- 172.21.15.89:0/4116580419 shutdown_connections 2024-04-02T08:52:26.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.410+0000 7f0a5bc00700 1 -- 172.21.15.89:0/4116580419 wait complete. 2024-04-02T08:52:26.411 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:26.412+0000 7fee12653240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T08:52:26.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.412+0000 7f0a5bc00700 1 Processor -- start 2024-04-02T08:52:26.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.412+0000 7f0a5bc00700 1 -- start start 2024-04-02T08:52:26.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a5bc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a540b2070 0x7f0a5415db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:26.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a5bc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a540cbb30 0x7f0a5409caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:26.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a5bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a540cd020 0x7f0a541f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:26.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a5bc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5415e0c0 con 0x7f0a540cbb30 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a5bc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a541f6cc0 con 0x7f0a540b2070 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a5bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a541f6990 con 0x7f0a540cd020 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a53e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a540cbb30 0x7f0a5409caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a53e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a540cbb30 0x7f0a5409caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39258/0 (socket says 172.21.15.89:39258) 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a53e00700 1 -- 172.21.15.89:0/1183312082 learned_addr learned my addr 172.21.15.89:0/1183312082 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a59200700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a540b2070 0x7f0a5415db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a59c00700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a540cd020 0x7f0a541f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a53e00700 1 -- 172.21.15.89:0/1183312082 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a540cd020 msgr2=0x7f0a541f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a53e00700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a540cd020 0x7f0a541f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a53e00700 1 -- 172.21.15.89:0/1183312082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a540b2070 msgr2=0x7f0a5415db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a53e00700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a540b2070 0x7f0a5415db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.414+0000 7f0a53e00700 1 -- 172.21.15.89:0/1183312082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a40009550 con 0x7f0a540cbb30 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.415+0000 7f0a59200700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a540b2070 0x7f0a5415db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.415+0000 7f0a53e00700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a540cbb30 0x7f0a5409caf0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f0a40000c00 tx=0x7f0a40004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.415+0000 7f0a50c00700 1 -- 172.21.15.89:0/1183312082 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40004380 con 0x7f0a540cbb30 2024-04-02T08:52:26.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.415+0000 7f0a50c00700 1 -- 172.21.15.89:0/1183312082 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4001f070 con 0x7f0a540cbb30 2024-04-02T08:52:26.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.415+0000 7f0a50c00700 1 -- 172.21.15.89:0/1183312082 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a400036a0 con 0x7f0a540cbb30 2024-04-02T08:52:26.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.415+0000 7f0a5bc00700 1 -- 172.21.15.89:0/1183312082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a540c8570 con 0x7f0a540cbb30 2024-04-02T08:52:26.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.415+0000 7f0a5bc00700 1 -- 172.21.15.89:0/1183312082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a540c8b30 con 0x7f0a540cbb30 2024-04-02T08:52:26.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.416+0000 7f0a5bc00700 1 -- 172.21.15.89:0/1183312082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a541f6e50 con 0x7f0a540cbb30 2024-04-02T08:52:26.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.417+0000 7f0a50c00700 1 -- 172.21.15.89:0/1183312082 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 98) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f0a40021030 con 0x7f0a540cbb30 2024-04-02T08:52:26.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.418+0000 7f0a50c00700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0a3406adb0 0x7f0a3406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:26.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.418+0000 7f0a50c00700 1 -- 172.21.15.89:0/1183312082 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f0a4008ad30 con 0x7f0a540cbb30 2024-04-02T08:52:26.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.418+0000 7f0a59200700 1 -- 172.21.15.89:0/1183312082 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0a3406adb0 msgr2=0x7f0a3406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:26.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.418+0000 7f0a59200700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0a3406adb0 0x7f0a3406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:26.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.422+0000 7f0a50c00700 1 -- 172.21.15.89:0/1183312082 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0a40055b60 con 0x7f0a540cbb30 2024-04-02T08:52:26.587 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:26.588+0000 7f53e59b5240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T08:52:26.615 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:26.616+0000 7fee12653240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:52:26.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.619+0000 7f0a59200700 1 -- 172.21.15.89:0/1183312082 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0a3406adb0 msgr2=0x7f0a3406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:26.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.619+0000 7f0a59200700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0a3406adb0 0x7f0a3406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:26.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.634+0000 7f0a5bc00700 1 -- 172.21.15.89:0/1183312082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0a5415bda0 con 0x7f0a540cbb30 2024-04-02T08:52:26.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.635+0000 7f0a50c00700 1 -- 172.21.15.89:0/1183312082 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v159) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0a40015070 con 0x7f0a540cbb30 2024-04-02T08:52:26.634 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T08:52:26.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.637+0000 7f0a5bc00700 1 -- 172.21.15.89:0/1183312082 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0a3406adb0 msgr2=0x7f0a3406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:26.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.637+0000 7f0a5bc00700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0a3406adb0 0x7f0a3406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.638+0000 7f0a5bc00700 1 -- 172.21.15.89:0/1183312082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a540cbb30 msgr2=0x7f0a5409caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:26.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.638+0000 7f0a5bc00700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a540cbb30 0x7f0a5409caf0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f0a40000c00 tx=0x7f0a40004750 comp rx=0 tx=0).stop 2024-04-02T08:52:26.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.638+0000 7f0a5bc00700 1 -- 172.21.15.89:0/1183312082 shutdown_connections 2024-04-02T08:52:26.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.638+0000 7f0a5bc00700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0a3406adb0 0x7f0a3406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.638+0000 7f0a5bc00700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a540b2070 0x7f0a5415db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.638+0000 7f0a5bc00700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a540cd020 0x7f0a541f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.638+0000 7f0a5bc00700 1 --2- 172.21.15.89:0/1183312082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a540cbb30 0x7f0a5409caf0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.638+0000 7f0a5bc00700 1 -- 172.21.15.89:0/1183312082 >> 172.21.15.89:0/1183312082 conn(0x7f0a540b0d70 msgr2=0x7f0a540ce540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:26.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.638+0000 7f0a5bc00700 1 -- 172.21.15.89:0/1183312082 shutdown_connections 2024-04-02T08:52:26.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.638+0000 7f0a5bc00700 1 -- 172.21.15.89:0/1183312082 wait complete. 2024-04-02T08:52:26.654 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T08:52:26.808 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:26.810+0000 7f53e59b5240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T08:52:26.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.851+0000 7fa2bf800700 1 Processor -- start 2024-04-02T08:52:26.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.852+0000 7fa2bf800700 1 -- start start 2024-04-02T08:52:26.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.853+0000 7fa2bf800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2b815d920 0x7fa2b806c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:26.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.853+0000 7fa2bf800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa2b8099a60 0x7fa2b8099eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:26.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.853+0000 7fa2bf800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b809a3f0 con 0x7fa2b809a910 2024-04-02T08:52:26.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.853+0000 7fa2bf800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b80d08d0 con 0x7fa2b8099a60 2024-04-02T08:52:26.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.853+0000 7fa2bf800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b806c300 con 0x7fa2b815d920 2024-04-02T08:52:26.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.853+0000 7fa2bce00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2b815d920 0x7fa2b806c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:26.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.853+0000 7fa2bd800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa2b8099a60 0x7fa2b8099eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:26.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.853+0000 7fa2bd800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa2b8099a60 0x7fa2b8099eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:40676/0 (socket says 172.21.15.89:40676) 2024-04-02T08:52:26.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.853+0000 7fa2bd800700 1 -- 172.21.15.89:0/2790874450 learned_addr learned my addr 172.21.15.89:0/2790874450 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:26.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.854+0000 7fa2bce00700 1 -- 172.21.15.89:0/2790874450 >> v2:172.21.15.113:3300/0 conn(0x7fa2b8099a60 msgr2=0x7fa2b8099eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:26.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.854+0000 7fa2bce00700 1 --2- 172.21.15.89:0/2790874450 >> v2:172.21.15.113:3300/0 conn(0x7fa2b8099a60 0x7fa2b8099eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.854+0000 7fa2bce00700 1 -- 172.21.15.89:0/2790874450 >> v1:172.21.15.113:6789/0 conn(0x7fa2b809a910 legacy=0x7fa2b80cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:26.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.854+0000 7fa2bce00700 1 -- 172.21.15.89:0/2790874450 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b806b9d0 con 0x7fa2b815d920 2024-04-02T08:52:26.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.854+0000 7fa2bd800700 1 --2- 172.21.15.89:0/2790874450 >> v2:172.21.15.113:3300/0 conn(0x7fa2b8099a60 0x7fa2b8099eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:26.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.854+0000 7fa2bce00700 1 --2- 172.21.15.89:0/2790874450 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2b815d920 0x7fa2b806c790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009840 tx=0x7fa2ac009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7eba73028ac1921b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:26.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.854+0000 7fa2b7400700 1 -- 172.21.15.89:0/2790874450 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac011830 con 0x7fa2b815d920 2024-04-02T08:52:26.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.854+0000 7fa2b7400700 1 -- 172.21.15.89:0/2790874450 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac011990 con 0x7fa2b815d920 2024-04-02T08:52:26.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.854+0000 7fa2b7400700 1 -- 172.21.15.89:0/2790874450 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac011d20 con 0x7fa2b815d920 2024-04-02T08:52:26.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.855+0000 7fa2bf800700 1 -- 172.21.15.89:0/2790874450 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2b815d920 msgr2=0x7fa2b806c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:26.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.855+0000 7fa2bf800700 1 --2- 172.21.15.89:0/2790874450 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2b815d920 0x7fa2b806c790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009840 tx=0x7fa2ac009b50 comp rx=0 tx=0).stop 2024-04-02T08:52:26.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.855+0000 7fa2bf800700 1 -- 172.21.15.89:0/2790874450 shutdown_connections 2024-04-02T08:52:26.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.855+0000 7fa2bf800700 1 --2- 172.21.15.89:0/2790874450 >> v2:172.21.15.113:3300/0 conn(0x7fa2b8099a60 0x7fa2b8099eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.855+0000 7fa2bf800700 1 --2- 172.21.15.89:0/2790874450 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2b815d920 0x7fa2b806c790 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.855+0000 7fa2bf800700 1 -- 172.21.15.89:0/2790874450 >> 172.21.15.89:0/2790874450 conn(0x7fa2b80d1350 msgr2=0x7fa2b80d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:26.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.855+0000 7fa2bf800700 1 -- 172.21.15.89:0/2790874450 shutdown_connections 2024-04-02T08:52:26.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.855+0000 7fa2bf800700 1 -- 172.21.15.89:0/2790874450 wait complete. 2024-04-02T08:52:26.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.857+0000 7fa2bf800700 1 Processor -- start 2024-04-02T08:52:26.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.857+0000 7fa2bf800700 1 -- start start 2024-04-02T08:52:26.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bf800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2b8099a60 0x7fa2b81e7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:26.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bf800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2b809a910 0x7fa2b80b1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:26.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bf800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2b815d920 0x7fa2b81545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:26.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bf800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b81e8720 con 0x7fa2b8099a60 2024-04-02T08:52:26.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bf800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b81e8a00 con 0x7fa2b815d920 2024-04-02T08:52:26.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bf800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b81e8ce0 con 0x7fa2b809a910 2024-04-02T08:52:26.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bce00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2b8099a60 0x7fa2b81e7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:26.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2b7e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2b809a910 0x7fa2b80b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:26.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bce00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2b8099a60 0x7fa2b81e7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39260/0 (socket says 172.21.15.89:39260) 2024-04-02T08:52:26.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bce00700 1 -- 172.21.15.89:0/4211938151 learned_addr learned my addr 172.21.15.89:0/4211938151 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:26.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bd800700 1 --2- 172.21.15.89:0/4211938151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2b815d920 0x7fa2b81545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:26.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bce00700 1 -- 172.21.15.89:0/4211938151 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2b809a910 msgr2=0x7fa2b80b1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:26.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bce00700 1 --2- 172.21.15.89:0/4211938151 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2b809a910 0x7fa2b80b1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bce00700 1 -- 172.21.15.89:0/4211938151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2b815d920 msgr2=0x7fa2b81545b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:26.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bce00700 1 --2- 172.21.15.89:0/4211938151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2b815d920 0x7fa2b81545b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:26.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2bce00700 1 -- 172.21.15.89:0/4211938151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b0009b30 con 0x7fa2b8099a60 2024-04-02T08:52:26.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.858+0000 7fa2b7e00700 1 --2- 172.21.15.89:0/4211938151 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2b809a910 0x7fa2b80b1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:26.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.859+0000 7fa2bce00700 1 --2- 172.21.15.89:0/4211938151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2b8099a60 0x7fa2b81e7e60 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009fd0 tx=0x7fa2ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:26.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.859+0000 7fa2b4c00700 1 -- 172.21.15.89:0/4211938151 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac0036a0 con 0x7fa2b8099a60 2024-04-02T08:52:26.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.859+0000 7fa2b4c00700 1 -- 172.21.15.89:0/4211938151 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac01e070 con 0x7fa2b8099a60 2024-04-02T08:52:26.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.859+0000 7fa2bf800700 1 -- 172.21.15.89:0/4211938151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ac0094a0 con 0x7fa2b8099a60 2024-04-02T08:52:26.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.859+0000 7fa2b4c00700 1 -- 172.21.15.89:0/4211938151 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac003a40 con 0x7fa2b8099a60 2024-04-02T08:52:26.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.859+0000 7fa2bf800700 1 -- 172.21.15.89:0/4211938151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b81e83a0 con 0x7fa2b8099a60 2024-04-02T08:52:26.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.860+0000 7fa2bf800700 1 -- 172.21.15.89:0/4211938151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b8067720 con 0x7fa2b8099a60 2024-04-02T08:52:26.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.861+0000 7fa2b4c00700 1 -- 172.21.15.89:0/4211938151 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 98) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fa2ac003ba0 con 0x7fa2b8099a60 2024-04-02T08:52:26.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.862+0000 7fa2b4c00700 1 --2- 172.21.15.89:0/4211938151 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa29806adb0 0x7fa29806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:26.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.863+0000 7fa2b7e00700 1 -- 172.21.15.89:0/4211938151 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa29806adb0 msgr2=0x7fa29806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:26.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.863+0000 7fa2b4c00700 1 -- 172.21.15.89:0/4211938151 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fa2ac08a700 con 0x7fa2b8099a60 2024-04-02T08:52:26.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.863+0000 7fa2b7e00700 1 --2- 172.21.15.89:0/4211938151 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa29806adb0 0x7fa29806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:26.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:26.867+0000 7fa2b4c00700 1 -- 172.21.15.89:0/4211938151 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa2ac055490 con 0x7fa2b8099a60 2024-04-02T08:52:27.030 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:27.031+0000 7f53e59b5240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:52:27.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.064+0000 7fa2b7e00700 1 -- 172.21.15.89:0/4211938151 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa29806adb0 msgr2=0x7fa29806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:27.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.064+0000 7fa2b7e00700 1 --2- 172.21.15.89:0/4211938151 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa29806adb0 0x7fa29806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:27.084 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:27.085+0000 7fee12653240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T08:52:27.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.123+0000 7fa2bf800700 1 -- 172.21.15.89:0/4211938151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2b81e8e20 con 0x7fa2b8099a60 2024-04-02T08:52:27.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.124+0000 7fa2b4c00700 1 -- 172.21.15.89:0/4211938151 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fa2b81e8e20 con 0x7fa2b8099a60 2024-04-02T08:52:27.124 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:27.125 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7445,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/4112144997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":4112144997},{"type":"v1","addr":"172.21.15.113:6834","nonce":4112144997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":8416,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2047642384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2047642384},{"type":"v1","addr":"172.21.15.113:6835","nonce":2047642384}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":9030,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2201184833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2201184833},{"type":"v1","addr":"172.21.15.89:6835","nonce":2201184833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:56:17.192894+0000","modified":"2024-04-02T07:56:20.200137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9033},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9033":{"gid":9033,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3861363328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3861363328},{"type":"v1","addr":"172.21.15.89:6837","nonce":3861363328}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-02T08:52:27.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.126+0000 7fa2bf800700 1 -- 172.21.15.89:0/4211938151 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa29806adb0 msgr2=0x7fa29806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:27.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.126+0000 7fa2bf800700 1 --2- 172.21.15.89:0/4211938151 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa29806adb0 0x7fa29806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.127+0000 7fa2bf800700 1 -- 172.21.15.89:0/4211938151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2b8099a60 msgr2=0x7fa2b81e7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:27.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.127+0000 7fa2bf800700 1 --2- 172.21.15.89:0/4211938151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2b8099a60 0x7fa2b81e7e60 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009fd0 tx=0x7fa2ac004750 comp rx=0 tx=0).stop 2024-04-02T08:52:27.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.127+0000 7fa2bf800700 1 -- 172.21.15.89:0/4211938151 shutdown_connections 2024-04-02T08:52:27.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.127+0000 7fa2bf800700 1 --2- 172.21.15.89:0/4211938151 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa29806adb0 0x7fa29806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.127+0000 7fa2bf800700 1 --2- 172.21.15.89:0/4211938151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2b815d920 0x7fa2b81545b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.127+0000 7fa2bf800700 1 --2- 172.21.15.89:0/4211938151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2b8099a60 0x7fa2b81e7e60 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.127+0000 7fa2bf800700 1 --2- 172.21.15.89:0/4211938151 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa2b809a910 0x7fa2b80b1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.127+0000 7fa2bf800700 1 -- 172.21.15.89:0/4211938151 >> 172.21.15.89:0/4211938151 conn(0x7fa2b80d1350 msgr2=0x7fa2b806ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:27.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.127+0000 7fa2bf800700 1 -- 172.21.15.89:0/4211938151 shutdown_connections 2024-04-02T08:52:27.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.127+0000 7fa2bf800700 1 -- 172.21.15.89:0/4211938151 wait complete. 2024-04-02T08:52:27.128 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 73 2024-04-02T08:52:27.144 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:52:27.287 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:27.288+0000 7fee12653240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:52:27.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.340+0000 7f6d36000700 1 Processor -- start 2024-04-02T08:52:27.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.341+0000 7f6d36000700 1 -- start start 2024-04-02T08:52:27.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.342+0000 7f6d36000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6d30160f10 0x7f6d300d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:27.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.342+0000 7f6d36000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6d300cf230 0x7f6d300d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:27.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.342+0000 7f6d36000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d300ceda0 con 0x7f6d30161590 2024-04-02T08:52:27.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.342+0000 7f6d36000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d300d23d0 con 0x7f6d300cf230 2024-04-02T08:52:27.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.342+0000 7f6d36000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d301612f0 con 0x7f6d30160f10 2024-04-02T08:52:27.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.342+0000 7f6d2ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6d300cf230 0x7f6d300d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:27.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.342+0000 7f6d2ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6d300cf230 0x7f6d300d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39264/0 (socket says 172.21.15.89:39264) 2024-04-02T08:52:27.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.342+0000 7f6d2ea00700 1 -- 172.21.15.89:0/1046734499 learned_addr learned my addr 172.21.15.89:0/1046734499 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:27.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.342+0000 7f6d2f400700 1 --2- 172.21.15.89:0/1046734499 >> v2:172.21.15.113:3300/0 conn(0x7f6d30160f10 0x7f6d300d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:27.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.342+0000 7f6d2ea00700 1 -- 172.21.15.89:0/1046734499 >> v2:172.21.15.113:3300/0 conn(0x7f6d30160f10 msgr2=0x7f6d300d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:27.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.342+0000 7f6d2ea00700 1 --2- 172.21.15.89:0/1046734499 >> v2:172.21.15.113:3300/0 conn(0x7f6d30160f10 0x7f6d300d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.342+0000 7f6d2ea00700 1 -- 172.21.15.89:0/1046734499 >> v1:172.21.15.113:6789/0 conn(0x7f6d30161590 legacy=0x7f6d300b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:27.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.343+0000 7f6d2ea00700 1 -- 172.21.15.89:0/1046734499 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d300ce470 con 0x7f6d300cf230 2024-04-02T08:52:27.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.343+0000 7f6d2f400700 1 --2- 172.21.15.89:0/1046734499 >> v2:172.21.15.113:3300/0 conn(0x7f6d30160f10 0x7f6d300d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:27.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.343+0000 7f6d2ea00700 1 --2- 172.21.15.89:0/1046734499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d300cf230 0x7f6d300d1e90 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f6d300d1080 tx=0x7f6d1c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fede7de286bf6430 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:27.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.343+0000 7f6d2e000700 1 -- 172.21.15.89:0/1046734499 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c00b870 con 0x7f6d300cf230 2024-04-02T08:52:27.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.343+0000 7f6d2e000700 1 -- 172.21.15.89:0/1046734499 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1c00b9d0 con 0x7f6d300cf230 2024-04-02T08:52:27.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.343+0000 7f6d2e000700 1 -- 172.21.15.89:0/1046734499 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c00bd60 con 0x7f6d300cf230 2024-04-02T08:52:27.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.344+0000 7f6d36000700 1 -- 172.21.15.89:0/1046734499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d300cf230 msgr2=0x7f6d300d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:27.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.344+0000 7f6d36000700 1 --2- 172.21.15.89:0/1046734499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d300cf230 0x7f6d300d1e90 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f6d300d1080 tx=0x7f6d1c009bd0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.344+0000 7f6d36000700 1 -- 172.21.15.89:0/1046734499 shutdown_connections 2024-04-02T08:52:27.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.344+0000 7f6d36000700 1 --2- 172.21.15.89:0/1046734499 >> v2:172.21.15.113:3300/0 conn(0x7f6d30160f10 0x7f6d300d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.344+0000 7f6d36000700 1 --2- 172.21.15.89:0/1046734499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d300cf230 0x7f6d300d1e90 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.344+0000 7f6d36000700 1 -- 172.21.15.89:0/1046734499 >> 172.21.15.89:0/1046734499 conn(0x7f6d3008aff0 msgr2=0x7f6d300d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:27.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.344+0000 7f6d36000700 1 -- 172.21.15.89:0/1046734499 shutdown_connections 2024-04-02T08:52:27.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.344+0000 7f6d36000700 1 -- 172.21.15.89:0/1046734499 wait complete. 2024-04-02T08:52:27.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.346+0000 7f6d36000700 1 Processor -- start 2024-04-02T08:52:27.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.346+0000 7f6d36000700 1 -- start start 2024-04-02T08:52:27.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d36000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d300cf230 0x7f6d300b84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:27.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d36000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d30160f10 0x7f6d300b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:27.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d36000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d30161590 0x7f6d30154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:27.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d36000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d30152b00 con 0x7f6d30161590 2024-04-02T08:52:27.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d36000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d300b9070 con 0x7f6d30160f10 2024-04-02T08:52:27.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d36000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d300b9350 con 0x7f6d300cf230 2024-04-02T08:52:27.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d2f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d300cf230 0x7f6d300b84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:27.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d2fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d30161590 0x7f6d30154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:27.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d2f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d300cf230 0x7f6d300b84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:45924/0 (socket says 172.21.15.89:45924) 2024-04-02T08:52:27.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d2f400700 1 -- 172.21.15.89:0/3846308958 learned_addr learned my addr 172.21.15.89:0/3846308958 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:27.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d2ea00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d30160f10 0x7f6d300b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:27.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d2f400700 1 -- 172.21.15.89:0/3846308958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d30160f10 msgr2=0x7f6d300b8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:27.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d2f400700 1 --2- 172.21.15.89:0/3846308958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d30160f10 0x7f6d300b8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d2f400700 1 -- 172.21.15.89:0/3846308958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d30161590 msgr2=0x7f6d30154a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:27.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d2f400700 1 --2- 172.21.15.89:0/3846308958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d30161590 0x7f6d30154a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d2f400700 1 -- 172.21.15.89:0/3846308958 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c009550 con 0x7f6d300cf230 2024-04-02T08:52:27.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d2fe00700 1 --2- 172.21.15.89:0/3846308958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d30161590 0x7f6d30154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:27.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.347+0000 7f6d2ea00700 1 --2- 172.21.15.89:0/3846308958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d30160f10 0x7f6d300b8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:27.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.348+0000 7f6d2f400700 1 --2- 172.21.15.89:0/3846308958 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d300cf230 0x7f6d300b84d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000e760 tx=0x7f6d2000ea70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:27.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.348+0000 7f6d27400700 1 -- 172.21.15.89:0/3846308958 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2000c070 con 0x7f6d300cf230 2024-04-02T08:52:27.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.348+0000 7f6d27400700 1 -- 172.21.15.89:0/3846308958 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d20013610 con 0x7f6d300cf230 2024-04-02T08:52:27.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.348+0000 7f6d27400700 1 -- 172.21.15.89:0/3846308958 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d200139c0 con 0x7f6d300cf230 2024-04-02T08:52:27.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.348+0000 7f6d36000700 1 -- 172.21.15.89:0/3846308958 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d300b6d60 con 0x7f6d300cf230 2024-04-02T08:52:27.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.348+0000 7f6d36000700 1 -- 172.21.15.89:0/3846308958 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d300b7320 con 0x7f6d300cf230 2024-04-02T08:52:27.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.349+0000 7f6d36000700 1 -- 172.21.15.89:0/3846308958 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d30067720 con 0x7f6d300cf230 2024-04-02T08:52:27.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.350+0000 7f6d27400700 1 -- 172.21.15.89:0/3846308958 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 98) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f6d20013b20 con 0x7f6d300cf230 2024-04-02T08:52:27.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.351+0000 7f6d27400700 1 --2- 172.21.15.89:0/3846308958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6d1006adb0 0x7f6d1006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:27.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.351+0000 7f6d27400700 1 -- 172.21.15.89:0/3846308958 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f6d200895f0 con 0x7f6d300cf230 2024-04-02T08:52:27.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.351+0000 7f6d2ea00700 1 -- 172.21.15.89:0/3846308958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6d1006adb0 msgr2=0x7f6d1006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:27.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.351+0000 7f6d2ea00700 1 --2- 172.21.15.89:0/3846308958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6d1006adb0 0x7f6d1006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:27.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.355+0000 7f6d27400700 1 -- 172.21.15.89:0/3846308958 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6d200541e0 con 0x7f6d300cf230 2024-04-02T08:52:27.535 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:27.536+0000 7f53e59b5240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T08:52:27.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.552+0000 7f6d2ea00700 1 -- 172.21.15.89:0/3846308958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6d1006adb0 msgr2=0x7f6d1006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:27.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.552+0000 7f6d2ea00700 1 --2- 172.21.15.89:0/3846308958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6d1006adb0 0x7f6d1006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:27.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.570+0000 7f6d36000700 1 -- 172.21.15.89:0/3846308958 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d300d3f20 con 0x7f6d300cf230 2024-04-02T08:52:27.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.572+0000 7f6d27400700 1 -- 172.21.15.89:0/3846308958 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v159) v1 ==== 74+0+17282 (secure 0 0 0) 0x7f6d20057800 con 0x7f6d300cf230 2024-04-02T08:52:27.571 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:27.571 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":159,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:52:22.879866+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:56:15.040512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-02T07:56:16.179443+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":159,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/23967222":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/3508000969":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/75408226":"2024-04-03T07:56:33.905850+0000","172.21.15.89:6833/43357":"2024-04-03T07:56:33.905850+0000","172.21.15.89:6832/43357":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/2379953994":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/65229891":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/1849807649":"2024-04-03T07:56:33.905850+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:52:27.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.574+0000 7f6d36000700 1 -- 172.21.15.89:0/3846308958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6d1006adb0 msgr2=0x7f6d1006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:27.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.574+0000 7f6d36000700 1 --2- 172.21.15.89:0/3846308958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6d1006adb0 0x7f6d1006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.574+0000 7f6d36000700 1 -- 172.21.15.89:0/3846308958 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d300cf230 msgr2=0x7f6d300b84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:27.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.574+0000 7f6d36000700 1 --2- 172.21.15.89:0/3846308958 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d300cf230 0x7f6d300b84d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000e760 tx=0x7f6d2000ea70 comp rx=0 tx=0).stop 2024-04-02T08:52:27.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.575+0000 7f6d36000700 1 -- 172.21.15.89:0/3846308958 shutdown_connections 2024-04-02T08:52:27.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.575+0000 7f6d36000700 1 --2- 172.21.15.89:0/3846308958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6d1006adb0 0x7f6d1006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.575+0000 7f6d36000700 1 --2- 172.21.15.89:0/3846308958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d30161590 0x7f6d30154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.575+0000 7f6d36000700 1 --2- 172.21.15.89:0/3846308958 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6d300cf230 0x7f6d300b84d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.575+0000 7f6d36000700 1 --2- 172.21.15.89:0/3846308958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d30160f10 0x7f6d300b8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.575+0000 7f6d36000700 1 -- 172.21.15.89:0/3846308958 >> 172.21.15.89:0/3846308958 conn(0x7f6d3008aff0 msgr2=0x7f6d30160210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:27.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.575+0000 7f6d36000700 1 -- 172.21.15.89:0/3846308958 shutdown_connections 2024-04-02T08:52:27.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.575+0000 7f6d36000700 1 -- 172.21.15.89:0/3846308958 wait complete. 2024-04-02T08:52:27.591 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T08:52:27.591 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T08:52:27.723 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:27.724+0000 7fee12653240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T08:52:27.753 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:27.755+0000 7f53e59b5240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:52:27.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.789+0000 7f814d600700 1 Processor -- start 2024-04-02T08:52:27.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.789+0000 7f814d600700 1 -- start start 2024-04-02T08:52:27.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.790+0000 7f814d600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8148160f10 0x7f81480d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:27.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.790+0000 7f814d600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81480cf230 0x7f81480d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:27.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.790+0000 7f814d600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81480ceda0 con 0x7f8148161590 2024-04-02T08:52:27.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.790+0000 7f814d600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81480d23d0 con 0x7f8148160f10 2024-04-02T08:52:27.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.790+0000 7f814d600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81481612f0 con 0x7f81480cf230 2024-04-02T08:52:27.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.790+0000 7f8147400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8148161590 0x7f81480b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51898/0 (socket says 172.21.15.89:51898) 2024-04-02T08:52:27.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.790+0000 7f8147400700 1 -- 172.21.15.89:0/2465269718 learned_addr learned my addr 172.21.15.89:0/2465269718 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:27.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.790+0000 7f8146000700 1 --2- 172.21.15.89:0/2465269718 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81480cf230 0x7f81480d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:27.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.790+0000 7f8146a00700 1 --2- 172.21.15.89:0/2465269718 >> v2:172.21.15.113:3300/0 conn(0x7f8148160f10 0x7f81480d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:27.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.791+0000 7f8145600700 1 -- 172.21.15.89:0/2465269718 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247388440 0 0) 0x7f81480ceda0 con 0x7f8148161590 2024-04-02T08:52:27.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.791+0000 7f8145600700 1 -- 172.21.15.89:0/2465269718 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f812c0035b0 con 0x7f8148161590 2024-04-02T08:52:27.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.791+0000 7f8146a00700 1 -- 172.21.15.89:0/2465269718 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81480cf230 msgr2=0x7f81480d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:27.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.791+0000 7f8146a00700 1 --2- 172.21.15.89:0/2465269718 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81480cf230 0x7f81480d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.791+0000 7f8146a00700 1 -- 172.21.15.89:0/2465269718 >> v1:172.21.15.89:6789/0 conn(0x7f8148161590 legacy=0x7f81480b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:27.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.791+0000 7f8146a00700 1 -- 172.21.15.89:0/2465269718 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81480ce470 con 0x7f8148160f10 2024-04-02T08:52:27.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.791+0000 7f8146000700 1 --2- 172.21.15.89:0/2465269718 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81480cf230 0x7f81480d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:27.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.791+0000 7f8146a00700 1 --2- 172.21.15.89:0/2465269718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8148160f10 0x7f81480d18b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8138009840 tx=0x7f8138009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b715c9c112948f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:27.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.792+0000 7f8145600700 1 -- 172.21.15.89:0/2465269718 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8138011830 con 0x7f8148160f10 2024-04-02T08:52:27.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.792+0000 7f8145600700 1 -- 172.21.15.89:0/2465269718 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8138011990 con 0x7f8148160f10 2024-04-02T08:52:27.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.792+0000 7f8145600700 1 -- 172.21.15.89:0/2465269718 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8138011d20 con 0x7f8148160f10 2024-04-02T08:52:27.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.792+0000 7f814d600700 1 -- 172.21.15.89:0/2465269718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8148160f10 msgr2=0x7f81480d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:27.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.792+0000 7f814d600700 1 --2- 172.21.15.89:0/2465269718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8148160f10 0x7f81480d18b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8138009840 tx=0x7f8138009b50 comp rx=0 tx=0).stop 2024-04-02T08:52:27.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.792+0000 7f814d600700 1 -- 172.21.15.89:0/2465269718 shutdown_connections 2024-04-02T08:52:27.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.792+0000 7f814d600700 1 --2- 172.21.15.89:0/2465269718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8148160f10 0x7f81480d18b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.792+0000 7f814d600700 1 --2- 172.21.15.89:0/2465269718 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81480cf230 0x7f81480d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.792+0000 7f814d600700 1 -- 172.21.15.89:0/2465269718 >> 172.21.15.89:0/2465269718 conn(0x7f814808aff0 msgr2=0x7f81480d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:27.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.792+0000 7f814d600700 1 -- 172.21.15.89:0/2465269718 shutdown_connections 2024-04-02T08:52:27.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.792+0000 7f814d600700 1 -- 172.21.15.89:0/2465269718 wait complete. 2024-04-02T08:52:27.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.794+0000 7f814d600700 1 Processor -- start 2024-04-02T08:52:27.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.794+0000 7f814d600700 1 -- start start 2024-04-02T08:52:27.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.795+0000 7f814d600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81480cf230 0x7f81480b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:27.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.795+0000 7f814d600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8148160f10 0x7f814809efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:27.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.795+0000 7f814d600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8148161590 0x7f81481f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:27.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.795+0000 7f814d600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814814c250 con 0x7f8148161590 2024-04-02T08:52:27.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.795+0000 7f814d600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81481f3580 con 0x7f81480cf230 2024-04-02T08:52:27.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.795+0000 7f814d600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f814809f5f0 con 0x7f8148160f10 2024-04-02T08:52:27.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.795+0000 7f8147400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8148161590 0x7f81481f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:27.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.795+0000 7f8146a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81480cf230 0x7f81480b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:27.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.795+0000 7f8147400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8148161590 0x7f81481f3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39276/0 (socket says 172.21.15.89:39276) 2024-04-02T08:52:27.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.795+0000 7f8146000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8148160f10 0x7f814809efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:27.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.795+0000 7f8147400700 1 -- 172.21.15.89:0/119911332 learned_addr learned my addr 172.21.15.89:0/119911332 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:27.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.795+0000 7f8146a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81480cf230 0x7f81480b14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:40724/0 (socket says 172.21.15.89:40724) 2024-04-02T08:52:27.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.796+0000 7f8147400700 1 -- 172.21.15.89:0/119911332 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8148160f10 msgr2=0x7f814809efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:27.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.796+0000 7f8147400700 1 --2- 172.21.15.89:0/119911332 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8148160f10 0x7f814809efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.796+0000 7f8147400700 1 -- 172.21.15.89:0/119911332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81480cf230 msgr2=0x7f81480b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:27.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.796+0000 7f8147400700 1 --2- 172.21.15.89:0/119911332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81480cf230 0x7f81480b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:27.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.796+0000 7f8147400700 1 -- 172.21.15.89:0/119911332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81340094a0 con 0x7f8148161590 2024-04-02T08:52:27.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.796+0000 7f8146a00700 1 --2- 172.21.15.89:0/119911332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81480cf230 0x7f81480b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:27.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.796+0000 7f8147400700 1 --2- 172.21.15.89:0/119911332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8148161590 0x7f81481f3040 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f814000caa0 tx=0x7f814000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:27.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.796+0000 7f813ea00700 1 -- 172.21.15.89:0/119911332 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814000a9e0 con 0x7f8148161590 2024-04-02T08:52:27.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.796+0000 7f813ea00700 1 -- 172.21.15.89:0/119911332 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f814000ab40 con 0x7f8148161590 2024-04-02T08:52:27.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.796+0000 7f814d600700 1 -- 172.21.15.89:0/119911332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81380094a0 con 0x7f8148161590 2024-04-02T08:52:27.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.796+0000 7f813ea00700 1 -- 172.21.15.89:0/119911332 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8140012400 con 0x7f8148161590 2024-04-02T08:52:27.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.797+0000 7f814d600700 1 -- 172.21.15.89:0/119911332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f814809fbb0 con 0x7f8148161590 2024-04-02T08:52:27.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.798+0000 7f814d600700 1 -- 172.21.15.89:0/119911332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8148067720 con 0x7f8148161590 2024-04-02T08:52:27.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.799+0000 7f813ea00700 1 -- 172.21.15.89:0/119911332 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 98) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f814001b3b0 con 0x7f8148161590 2024-04-02T08:52:27.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.799+0000 7f813ea00700 1 --2- 172.21.15.89:0/119911332 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f812806adb0 0x7f812806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:27.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.800+0000 7f813ea00700 1 -- 172.21.15.89:0/119911332 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f8140089480 con 0x7f8148161590 2024-04-02T08:52:27.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.800+0000 7f8146a00700 1 -- 172.21.15.89:0/119911332 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f812806adb0 msgr2=0x7f812806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:27.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.800+0000 7f8146a00700 1 --2- 172.21.15.89:0/119911332 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f812806adb0 0x7f812806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:27.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.804+0000 7f813ea00700 1 -- 172.21.15.89:0/119911332 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8140054240 con 0x7f8148161590 2024-04-02T08:52:27.832 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:27.833+0000 7fee12653240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T08:52:27.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:27.913+0000 7f813ea00700 1 -- 172.21.15.89:0/119911332 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 99) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f8140054610 con 0x7f8148161590 2024-04-02T08:52:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.001+0000 7f8146a00700 1 -- 172.21.15.89:0/119911332 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f812806adb0 msgr2=0x7f812806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:28.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.001+0000 7f8146a00700 1 --2- 172.21.15.89:0/119911332 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f812806adb0 0x7f812806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:28.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.067+0000 7f814d600700 1 -- 172.21.15.89:0/119911332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81480c9ee0 con 0x7f8148161590 2024-04-02T08:52:28.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.068+0000 7f813ea00700 1 -- 172.21.15.89:0/119911332 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f81400764e0 con 0x7f8148161590 2024-04-02T08:52:28.066 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:28.066 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7445,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/4112144997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":4112144997},{"type":"v1","addr":"172.21.15.113:6834","nonce":4112144997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":8416,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2047642384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2047642384},{"type":"v1","addr":"172.21.15.113:6835","nonce":2047642384}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":9030,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2201184833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2201184833},{"type":"v1","addr":"172.21.15.89:6835","nonce":2201184833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:56:17.192894+0000","modified":"2024-04-02T07:56:20.200137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9033},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9033":{"gid":9033,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3861363328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3861363328},{"type":"v1","addr":"172.21.15.89:6837","nonce":3861363328}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-02T08:52:28.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.070+0000 7f814d600700 1 -- 172.21.15.89:0/119911332 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f812806adb0 msgr2=0x7f812806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:28.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.070+0000 7f814d600700 1 --2- 172.21.15.89:0/119911332 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f812806adb0 0x7f812806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.070+0000 7f814d600700 1 -- 172.21.15.89:0/119911332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8148161590 msgr2=0x7f81481f3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:28.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.070+0000 7f814d600700 1 --2- 172.21.15.89:0/119911332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8148161590 0x7f81481f3040 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f814000caa0 tx=0x7f814000cdb0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.070+0000 7f814d600700 1 -- 172.21.15.89:0/119911332 shutdown_connections 2024-04-02T08:52:28.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.070+0000 7f814d600700 1 --2- 172.21.15.89:0/119911332 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f812806adb0 0x7f812806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.070+0000 7f814d600700 1 --2- 172.21.15.89:0/119911332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81480cf230 0x7f81480b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.070+0000 7f814d600700 1 --2- 172.21.15.89:0/119911332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8148161590 0x7f81481f3040 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.070+0000 7f814d600700 1 --2- 172.21.15.89:0/119911332 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8148160f10 0x7f814809efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.070+0000 7f814d600700 1 -- 172.21.15.89:0/119911332 >> 172.21.15.89:0/119911332 conn(0x7f814808aff0 msgr2=0x7f814815b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:28.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.071+0000 7f814d600700 1 -- 172.21.15.89:0/119911332 shutdown_connections 2024-04-02T08:52:28.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.071+0000 7f814d600700 1 -- 172.21.15.89:0/119911332 wait complete. 2024-04-02T08:52:28.070 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 73 2024-04-02T08:52:28.087 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T08:52:28.224 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:28.225+0000 7f53e59b5240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T08:52:28.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.284+0000 7f7c6e600700 1 Processor -- start 2024-04-02T08:52:28.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.284+0000 7f7c6e600700 1 -- start start 2024-04-02T08:52:28.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.285+0000 7f7c6e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c68160f10 0x7f7c680d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:28.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.285+0000 7f7c6e600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7c68161590 0x7f7c68161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:28.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.285+0000 7f7c6e600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c680ceda0 con 0x7f7c680cf230 2024-04-02T08:52:28.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.285+0000 7f7c6e600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c68161f80 con 0x7f7c68161590 2024-04-02T08:52:28.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.285+0000 7f7c6e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c681612f0 con 0x7f7c68160f10 2024-04-02T08:52:28.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.285+0000 7f7c66a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7c680cf230 0x7f7c680d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51906/0 (socket says 172.21.15.89:51906) 2024-04-02T08:52:28.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.285+0000 7f7c66a00700 1 -- 172.21.15.89:0/3702674050 learned_addr learned my addr 172.21.15.89:0/3702674050 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:28.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.285+0000 7f7c67e00700 1 --2- 172.21.15.89:0/3702674050 >> v2:172.21.15.113:3300/0 conn(0x7f7c68161590 0x7f7c68161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:28.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.285+0000 7f7c67400700 1 --2- 172.21.15.89:0/3702674050 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c68160f10 0x7f7c680d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:28.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.286+0000 7f7c66000700 1 -- 172.21.15.89:0/3702674050 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33920650 0 0) 0x7f7c680ceda0 con 0x7f7c680cf230 2024-04-02T08:52:28.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.286+0000 7f7c66000700 1 -- 172.21.15.89:0/3702674050 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c4c0035b0 con 0x7f7c680cf230 2024-04-02T08:52:28.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.286+0000 7f7c67400700 1 -- 172.21.15.89:0/3702674050 >> v2:172.21.15.113:3300/0 conn(0x7f7c68161590 msgr2=0x7f7c68161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:28.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.286+0000 7f7c67400700 1 --2- 172.21.15.89:0/3702674050 >> v2:172.21.15.113:3300/0 conn(0x7f7c68161590 0x7f7c68161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.286+0000 7f7c67400700 1 -- 172.21.15.89:0/3702674050 >> v1:172.21.15.89:6789/0 conn(0x7f7c680cf230 legacy=0x7f7c680d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:28.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.286+0000 7f7c67400700 1 -- 172.21.15.89:0/3702674050 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c680ce470 con 0x7f7c68160f10 2024-04-02T08:52:28.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.286+0000 7f7c67400700 1 --2- 172.21.15.89:0/3702674050 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c68160f10 0x7f7c680d18b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7c58009870 tx=0x7f7c58009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b43cbf072f5523a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:28.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.287+0000 7f7c66000700 1 -- 172.21.15.89:0/3702674050 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c58011830 con 0x7f7c68160f10 2024-04-02T08:52:28.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.287+0000 7f7c66000700 1 -- 172.21.15.89:0/3702674050 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c58011990 con 0x7f7c68160f10 2024-04-02T08:52:28.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.287+0000 7f7c66000700 1 -- 172.21.15.89:0/3702674050 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c58011d00 con 0x7f7c68160f10 2024-04-02T08:52:28.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.287+0000 7f7c6e600700 1 -- 172.21.15.89:0/3702674050 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c68160f10 msgr2=0x7f7c680d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:28.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.287+0000 7f7c6e600700 1 --2- 172.21.15.89:0/3702674050 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c68160f10 0x7f7c680d18b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7c58009870 tx=0x7f7c58009b80 comp rx=0 tx=0).stop 2024-04-02T08:52:28.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.287+0000 7f7c6e600700 1 -- 172.21.15.89:0/3702674050 shutdown_connections 2024-04-02T08:52:28.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.287+0000 7f7c6e600700 1 --2- 172.21.15.89:0/3702674050 >> v2:172.21.15.113:3300/0 conn(0x7f7c68161590 0x7f7c68161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.287+0000 7f7c6e600700 1 --2- 172.21.15.89:0/3702674050 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c68160f10 0x7f7c680d18b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.287+0000 7f7c6e600700 1 -- 172.21.15.89:0/3702674050 >> 172.21.15.89:0/3702674050 conn(0x7f7c6808aff0 msgr2=0x7f7c680d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:28.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.287+0000 7f7c6e600700 1 -- 172.21.15.89:0/3702674050 shutdown_connections 2024-04-02T08:52:28.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.287+0000 7f7c6e600700 1 -- 172.21.15.89:0/3702674050 wait complete. 2024-04-02T08:52:28.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.289+0000 7f7c6e600700 1 Processor -- start 2024-04-02T08:52:28.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.289+0000 7f7c6e600700 1 -- start start 2024-04-02T08:52:28.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.290+0000 7f7c6e600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c680cf230 0x7f7c681f6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:28.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.290+0000 7f7c6e600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c68160f10 0x7f7c6809f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:28.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.290+0000 7f7c6e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c68161590 0x7f7c6809f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:28.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.290+0000 7f7c6e600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680a0ad0 con 0x7f7c680cf230 2024-04-02T08:52:28.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.290+0000 7f7c6e600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6809fec0 con 0x7f7c68160f10 2024-04-02T08:52:28.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.290+0000 7f7c6e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680a01a0 con 0x7f7c68161590 2024-04-02T08:52:28.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.290+0000 7f7c67400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c680cf230 0x7f7c681f6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:28.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.290+0000 7f7c67400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c680cf230 0x7f7c681f6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39284/0 (socket says 172.21.15.89:39284) 2024-04-02T08:52:28.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.290+0000 7f7c67400700 1 -- 172.21.15.89:0/2936295882 learned_addr learned my addr 172.21.15.89:0/2936295882 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:28.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.290+0000 7f7c66a00700 1 --2- 172.21.15.89:0/2936295882 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c68160f10 0x7f7c6809f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:28.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.290+0000 7f7c67e00700 1 --2- 172.21.15.89:0/2936295882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c68161590 0x7f7c6809f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:28.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.291+0000 7f7c67400700 1 -- 172.21.15.89:0/2936295882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c68161590 msgr2=0x7f7c6809f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:28.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.291+0000 7f7c67400700 1 --2- 172.21.15.89:0/2936295882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c68161590 0x7f7c6809f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.291+0000 7f7c67400700 1 -- 172.21.15.89:0/2936295882 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c68160f10 msgr2=0x7f7c6809f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:28.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.291+0000 7f7c67400700 1 --2- 172.21.15.89:0/2936295882 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c68160f10 0x7f7c6809f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.291+0000 7f7c67400700 1 -- 172.21.15.89:0/2936295882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c580094a0 con 0x7f7c680cf230 2024-04-02T08:52:28.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.291+0000 7f7c67400700 1 --2- 172.21.15.89:0/2936295882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c680cf230 0x7f7c681f6060 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f7c58014040 tx=0x7f7c58004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:28.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.291+0000 7f7c5f400700 1 -- 172.21.15.89:0/2936295882 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c580044d0 con 0x7f7c680cf230 2024-04-02T08:52:28.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.291+0000 7f7c5f400700 1 -- 172.21.15.89:0/2936295882 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5801e070 con 0x7f7c680cf230 2024-04-02T08:52:28.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.291+0000 7f7c5f400700 1 -- 172.21.15.89:0/2936295882 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c580038e0 con 0x7f7c680cf230 2024-04-02T08:52:28.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.292+0000 7f7c6e600700 1 -- 172.21.15.89:0/2936295882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c680a0480 con 0x7f7c680cf230 2024-04-02T08:52:28.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.292+0000 7f7c6e600700 1 -- 172.21.15.89:0/2936295882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6809fbe0 con 0x7f7c680cf230 2024-04-02T08:52:28.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.293+0000 7f7c6e600700 1 -- 172.21.15.89:0/2936295882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c68067720 con 0x7f7c680cf230 2024-04-02T08:52:28.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.294+0000 7f7c5f400700 1 -- 172.21.15.89:0/2936295882 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 99) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f7c58003a40 con 0x7f7c680cf230 2024-04-02T08:52:28.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.295+0000 7f7c5f400700 1 --2- 172.21.15.89:0/2936295882 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c4806adb0 0x7f7c4806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:28.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.295+0000 7f7c5f400700 1 -- 172.21.15.89:0/2936295882 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f7c5808a620 con 0x7f7c680cf230 2024-04-02T08:52:28.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.295+0000 7f7c66a00700 1 -- 172.21.15.89:0/2936295882 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c4806adb0 msgr2=0x7f7c4806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:28.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.295+0000 7f7c66a00700 1 --2- 172.21.15.89:0/2936295882 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c4806adb0 0x7f7c4806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:28.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.300+0000 7f7c5f400700 1 -- 172.21.15.89:0/2936295882 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7c580553b0 con 0x7f7c680cf230 2024-04-02T08:52:28.342 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:28.344+0000 7f53e59b5240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T08:52:28.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.353+0000 7f7c5f400700 1 -- 172.21.15.89:0/2936295882 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 100) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f7c58003cf0 con 0x7f7c680cf230 2024-04-02T08:52:28.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.354+0000 7f7c5f400700 1 -- 172.21.15.89:0/2936295882 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c4806adb0 msgr2=0x7f7c4806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:28.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.354+0000 7f7c5f400700 1 --2- 172.21.15.89:0/2936295882 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c4806adb0 0x7f7c4806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.422 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:28.423+0000 7f5358c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:28.423 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:28.423+0000 7f5358c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:28.423 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:28.423+0000 7f5353200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:28.423 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:28.423+0000 7f5353200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:28.423 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:28.423+0000 7f5353200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:28.423 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:28.423+0000 7f5353200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:28.423 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:28.423+0000 7f5353200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:28.423 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:28.423+0000 7f5358c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:28.423 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:28.423+0000 7f5358c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:28.423 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:28.423+0000 7f5358c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:28.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.569+0000 7f7c6e600700 1 -- 172.21.15.89:0/2936295882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c68067f50 con 0x7f7c680cf230 2024-04-02T08:52:28.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.570+0000 7f7c5f400700 1 -- 172.21.15.89:0/2936295882 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f7c58020090 con 0x7f7c680cf230 2024-04-02T08:52:28.569 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:28.569 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7445,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/4112144997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":4112144997},{"type":"v1","addr":"172.21.15.113:6834","nonce":4112144997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":8416,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2047642384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2047642384},{"type":"v1","addr":"172.21.15.113:6835","nonce":2047642384}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":9030,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2201184833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2201184833},{"type":"v1","addr":"172.21.15.89:6835","nonce":2201184833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T07:56:17.192894+0000","modified":"2024-04-02T07:56:20.200137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9033},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9033":{"gid":9033,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3861363328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3861363328},{"type":"v1","addr":"172.21.15.89:6837","nonce":3861363328}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-02T08:52:28.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.572+0000 7f7c6e600700 1 -- 172.21.15.89:0/2936295882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c680cf230 msgr2=0x7f7c681f6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:28.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.572+0000 7f7c6e600700 1 --2- 172.21.15.89:0/2936295882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c680cf230 0x7f7c681f6060 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f7c58014040 tx=0x7f7c58004750 comp rx=0 tx=0).stop 2024-04-02T08:52:28.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.573+0000 7f7c6e600700 1 -- 172.21.15.89:0/2936295882 shutdown_connections 2024-04-02T08:52:28.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.573+0000 7f7c6e600700 1 --2- 172.21.15.89:0/2936295882 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f7c4806adb0 0x7f7c4806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.573+0000 7f7c6e600700 1 --2- 172.21.15.89:0/2936295882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c68161590 0x7f7c6809f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.573+0000 7f7c6e600700 1 --2- 172.21.15.89:0/2936295882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c680cf230 0x7f7c681f6060 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.573+0000 7f7c6e600700 1 --2- 172.21.15.89:0/2936295882 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c68160f10 0x7f7c6809f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.573+0000 7f7c6e600700 1 -- 172.21.15.89:0/2936295882 >> 172.21.15.89:0/2936295882 conn(0x7f7c6808aff0 msgr2=0x7f7c680cab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:28.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.573+0000 7f7c6e600700 1 -- 172.21.15.89:0/2936295882 shutdown_connections 2024-04-02T08:52:28.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.573+0000 7f7c6e600700 1 -- 172.21.15.89:0/2936295882 wait complete. 2024-04-02T08:52:28.572 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 73 2024-04-02T08:52:28.589 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:52:28.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.781+0000 7fcbf0c00700 1 Processor -- start 2024-04-02T08:52:28.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.781+0000 7fcbf0c00700 1 -- start start 2024-04-02T08:52:28.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.782+0000 7fcbf0c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcbec15d640 0x7fcbec0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:28.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.782+0000 7fcbf0c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbec0ceec0 0x7fcbec1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:28.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.782+0000 7fcbf0c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcbec161800 0x7fcbec0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:28.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.782+0000 7fcbf0c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbec161c90 con 0x7fcbec161800 2024-04-02T08:52:28.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.782+0000 7fcbf0c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbec0d30d0 con 0x7fcbec15d640 2024-04-02T08:52:28.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.782+0000 7fcbf0c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec0b6840 con 0x7fcbec0ceec0 2024-04-02T08:52:28.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.782+0000 7fcbea800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcbec161800 0x7fcbec0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:28.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.783+0000 7fcbea800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcbec161800 0x7fcbec0cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36892/0 (socket says 172.21.15.89:36892) 2024-04-02T08:52:28.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.783+0000 7fcbea800700 1 -- 172.21.15.89:0/3784795758 learned_addr learned my addr 172.21.15.89:0/3784795758 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:28.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.783+0000 7fcbe9400700 1 --2- 172.21.15.89:0/3784795758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbec0ceec0 0x7fcbec1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:28.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.783+0000 7fcbe9e00700 1 --2- 172.21.15.89:0/3784795758 >> v2:172.21.15.113:3300/0 conn(0x7fcbec15d640 0x7fcbec0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:28.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.783+0000 7fcbea800700 1 -- 172.21.15.89:0/3784795758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbec0ceec0 msgr2=0x7fcbec1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:28.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.783+0000 7fcbea800700 1 --2- 172.21.15.89:0/3784795758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbec0ceec0 0x7fcbec1612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.783+0000 7fcbea800700 1 -- 172.21.15.89:0/3784795758 >> v2:172.21.15.113:3300/0 conn(0x7fcbec15d640 msgr2=0x7fcbec0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:28.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.783+0000 7fcbea800700 1 --2- 172.21.15.89:0/3784795758 >> v2:172.21.15.113:3300/0 conn(0x7fcbec15d640 0x7fcbec0d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.783+0000 7fcbea800700 1 -- 172.21.15.89:0/3784795758 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec0d2880 con 0x7fcbec161800 2024-04-02T08:52:28.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.783+0000 7fcbea800700 1 --2- 172.21.15.89:0/3784795758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbec161800 0x7fcbec0cdba0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0b5c20 tx=0x7fcbe400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e78bd27b7c71ad server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:28.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.783+0000 7fcbe8a00700 1 -- 172.21.15.89:0/3784795758 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe400e070 con 0x7fcbec161800 2024-04-02T08:52:28.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.784+0000 7fcbe8a00700 1 -- 172.21.15.89:0/3784795758 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe400b910 con 0x7fcbec161800 2024-04-02T08:52:28.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.784+0000 7fcbe8a00700 1 -- 172.21.15.89:0/3784795758 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe400bcc0 con 0x7fcbec161800 2024-04-02T08:52:28.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.784+0000 7fcbf0c00700 1 -- 172.21.15.89:0/3784795758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbec161800 msgr2=0x7fcbec0cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:28.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.784+0000 7fcbf0c00700 1 --2- 172.21.15.89:0/3784795758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbec161800 0x7fcbec0cdba0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0b5c20 tx=0x7fcbe400b330 comp rx=0 tx=0).stop 2024-04-02T08:52:28.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.784+0000 7fcbf0c00700 1 -- 172.21.15.89:0/3784795758 shutdown_connections 2024-04-02T08:52:28.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.784+0000 7fcbf0c00700 1 --2- 172.21.15.89:0/3784795758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbec161800 0x7fcbec0cdba0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.784+0000 7fcbf0c00700 1 --2- 172.21.15.89:0/3784795758 >> v2:172.21.15.113:3300/0 conn(0x7fcbec15d640 0x7fcbec0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.784+0000 7fcbf0c00700 1 --2- 172.21.15.89:0/3784795758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbec0ceec0 0x7fcbec1612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.784+0000 7fcbf0c00700 1 -- 172.21.15.89:0/3784795758 >> 172.21.15.89:0/3784795758 conn(0x7fcbec08aff0 msgr2=0x7fcbec0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:28.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.784+0000 7fcbf0c00700 1 -- 172.21.15.89:0/3784795758 shutdown_connections 2024-04-02T08:52:28.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.784+0000 7fcbf0c00700 1 -- 172.21.15.89:0/3784795758 wait complete. 2024-04-02T08:52:28.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.786+0000 7fcbf0c00700 1 Processor -- start 2024-04-02T08:52:28.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.786+0000 7fcbf0c00700 1 -- start start 2024-04-02T08:52:28.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.787+0000 7fcbf0c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbec0ceec0 0x7fcbec155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:28.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.787+0000 7fcbf0c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbec15d640 0x7fcbec1f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.787+0000 7fcbf0c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbec161800 0x7fcbec1f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.787+0000 7fcbf0c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec160760 con 0x7fcbec15d640 2024-04-02T08:52:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.787+0000 7fcbf0c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec15fae0 con 0x7fcbec161800 2024-04-02T08:52:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.787+0000 7fcbf0c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec15fdc0 con 0x7fcbec0ceec0 2024-04-02T08:52:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.787+0000 7fcbe9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbec15d640 0x7fcbec1f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.787+0000 7fcbe9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbec15d640 0x7fcbec1f2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36908/0 (socket says 172.21.15.89:36908) 2024-04-02T08:52:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.787+0000 7fcbe9400700 1 -- 172.21.15.89:0/2455784386 learned_addr learned my addr 172.21.15.89:0/2455784386 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.787+0000 7fcbe9e00700 1 --2- 172.21.15.89:0/2455784386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbec0ceec0 0x7fcbec155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.787+0000 7fcbea800700 1 --2- 172.21.15.89:0/2455784386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbec161800 0x7fcbec1f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.788+0000 7fcbe9400700 1 -- 172.21.15.89:0/2455784386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbec0ceec0 msgr2=0x7fcbec155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.788+0000 7fcbe9400700 1 --2- 172.21.15.89:0/2455784386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbec0ceec0 0x7fcbec155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.788+0000 7fcbe9400700 1 -- 172.21.15.89:0/2455784386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbec161800 msgr2=0x7fcbec1f2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:28.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.788+0000 7fcbe9400700 1 --2- 172.21.15.89:0/2455784386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbec161800 0x7fcbec1f2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:28.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.788+0000 7fcbe9400700 1 -- 172.21.15.89:0/2455784386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe4009b30 con 0x7fcbec15d640 2024-04-02T08:52:28.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.788+0000 7fcbe9400700 1 --2- 172.21.15.89:0/2455784386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbec15d640 0x7fcbec1f2510 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fcbd800e870 tx=0x7fcbd800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:28.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.788+0000 7fcbe2000700 1 -- 172.21.15.89:0/2455784386 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8009590 con 0x7fcbec15d640 2024-04-02T08:52:28.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.788+0000 7fcbe2000700 1 -- 172.21.15.89:0/2455784386 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd8013680 con 0x7fcbec15d640 2024-04-02T08:52:28.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.788+0000 7fcbf0c00700 1 -- 172.21.15.89:0/2455784386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbec1600a0 con 0x7fcbec15d640 2024-04-02T08:52:28.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.788+0000 7fcbe2000700 1 -- 172.21.15.89:0/2455784386 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd80139f0 con 0x7fcbec15d640 2024-04-02T08:52:28.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.788+0000 7fcbf0c00700 1 -- 172.21.15.89:0/2455784386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbec156250 con 0x7fcbec15d640 2024-04-02T08:52:28.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.789+0000 7fcbf0c00700 1 -- 172.21.15.89:0/2455784386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbec067720 con 0x7fcbec15d640 2024-04-02T08:52:28.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.790+0000 7fcbe2000700 1 -- 172.21.15.89:0/2455784386 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 100) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fcbd8013b50 con 0x7fcbec15d640 2024-04-02T08:52:28.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.791+0000 7fcbe2000700 1 -- 172.21.15.89:0/2455784386 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6731+0+0 (secure 0 0 0) 0x7fcbd8088fa0 con 0x7fcbec15d640 2024-04-02T08:52:28.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:28.795+0000 7fcbe2000700 1 -- 172.21.15.89:0/2455784386 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcbd80541d0 con 0x7fcbec15d640 2024-04-02T08:52:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.001+0000 7fcbf0c00700 1 -- 172.21.15.89:0/2455784386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbec0b5c70 con 0x7fcbec15d640 2024-04-02T08:52:29.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.003+0000 7fcbe2000700 1 -- 172.21.15.89:0/2455784386 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+17581 (secure 0 0 0) 0x7fcbd80577f0 con 0x7fcbec15d640 2024-04-02T08:52:29.001 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:29.002 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":160,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:52:28.346685+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-02T07:56:15.040512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-02T07:56:16.179443+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":159,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/3091491570":"2024-04-03T08:52:28.346631+0000","172.21.15.89:0/502328971":"2024-04-03T08:52:28.346631+0000","172.21.15.89:0/3246028660":"2024-04-03T08:52:28.346631+0000","172.21.15.89:0/3156282459":"2024-04-03T08:52:28.346631+0000","172.21.15.89:0/23967222":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/3508000969":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/75408226":"2024-04-03T07:56:33.905850+0000","172.21.15.89:6833/43357":"2024-04-03T07:56:33.905850+0000","172.21.15.89:6832/43357":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/2379953994":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/30895032":"2024-04-03T08:52:28.346631+0000","172.21.15.89:0/65229891":"2024-04-03T07:56:33.905850+0000","172.21.15.89:0/3938975418":"2024-04-03T08:52:28.346631+0000","172.21.15.89:0/1849807649":"2024-04-03T07:56:33.905850+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:52:29.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.006+0000 7fcbf0c00700 1 -- 172.21.15.89:0/2455784386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbec15d640 msgr2=0x7fcbec1f2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.006+0000 7fcbf0c00700 1 --2- 172.21.15.89:0/2455784386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbec15d640 0x7fcbec1f2510 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fcbd800e870 tx=0x7fcbd800eb80 comp rx=0 tx=0).stop 2024-04-02T08:52:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.006+0000 7fcbf0c00700 1 -- 172.21.15.89:0/2455784386 shutdown_connections 2024-04-02T08:52:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.006+0000 7fcbf0c00700 1 --2- 172.21.15.89:0/2455784386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbec161800 0x7fcbec1f2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.006+0000 7fcbf0c00700 1 --2- 172.21.15.89:0/2455784386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbec0ceec0 0x7fcbec155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.006+0000 7fcbf0c00700 1 --2- 172.21.15.89:0/2455784386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbec15d640 0x7fcbec1f2510 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.006+0000 7fcbf0c00700 1 -- 172.21.15.89:0/2455784386 >> 172.21.15.89:0/2455784386 conn(0x7fcbec08aff0 msgr2=0x7fcbec0d0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.006+0000 7fcbf0c00700 1 -- 172.21.15.89:0/2455784386 shutdown_connections 2024-04-02T08:52:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.006+0000 7fcbf0c00700 1 -- 172.21.15.89:0/2455784386 wait complete. 2024-04-02T08:52:29.022 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T08:52:29.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.215+0000 7f0c44000700 1 Processor -- start 2024-04-02T08:52:29.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.215+0000 7f0c44000700 1 -- start start 2024-04-02T08:52:29.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.216+0000 7f0c44000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c3c0d16a0 0x7f0c3c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:29.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.216+0000 7f0c44000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0c3c0d2370 0x7f0c3c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:29.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.216+0000 7f0c44000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c3c161420 con 0x7f0c3c15d9d0 2024-04-02T08:52:29.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.216+0000 7f0c44000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c3c0d2820 con 0x7f0c3c0d2370 2024-04-02T08:52:29.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.216+0000 7f0c44000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c0d4340 con 0x7f0c3c0d16a0 2024-04-02T08:52:29.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.217+0000 7f0c40c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c3c0d16a0 0x7f0c3c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:29.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.217+0000 7f0c42000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0c3c0d2370 0x7f0c3c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:29.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.217+0000 7f0c40c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c3c0d16a0 0x7f0c3c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:36060/0 (socket says 172.21.15.89:36060) 2024-04-02T08:52:29.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.217+0000 7f0c41600700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f0c3c15d9d0 0x7f0c3c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:47200/0 (socket says 172.21.15.89:47200) 2024-04-02T08:52:29.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.217+0000 7f0c40c00700 1 -- 172.21.15.89:0/2300792704 learned_addr learned my addr 172.21.15.89:0/2300792704 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:29.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.217+0000 7f0c40c00700 1 -- 172.21.15.89:0/2300792704 >> v2:172.21.15.113:3300/0 conn(0x7f0c3c0d2370 msgr2=0x7f0c3c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:29.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.217+0000 7f0c40c00700 1 --2- 172.21.15.89:0/2300792704 >> v2:172.21.15.113:3300/0 conn(0x7f0c3c0d2370 0x7f0c3c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.217+0000 7f0c40c00700 1 -- 172.21.15.89:0/2300792704 >> v1:172.21.15.113:6789/0 conn(0x7f0c3c15d9d0 legacy=0x7f0c3c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:29.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.217+0000 7f0c40c00700 1 -- 172.21.15.89:0/2300792704 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3c161700 con 0x7f0c3c0d16a0 2024-04-02T08:52:29.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.218+0000 7f0c40c00700 1 --2- 172.21.15.89:0/2300792704 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c3c0d16a0 0x7f0c3c0d1e30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0c28006fd0 tx=0x7f0c28009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efc5d8ff0d3efecb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:29.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.218+0000 7f0c3be00700 1 -- 172.21.15.89:0/2300792704 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28005e30 con 0x7f0c3c0d16a0 2024-04-02T08:52:29.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.218+0000 7f0c3be00700 1 -- 172.21.15.89:0/2300792704 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c28004030 con 0x7f0c3c0d16a0 2024-04-02T08:52:29.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.218+0000 7f0c3be00700 1 -- 172.21.15.89:0/2300792704 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c280043e0 con 0x7f0c3c0d16a0 2024-04-02T08:52:29.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.218+0000 7f0c44000700 1 -- 172.21.15.89:0/2300792704 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c3c0d16a0 msgr2=0x7f0c3c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:29.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.218+0000 7f0c44000700 1 --2- 172.21.15.89:0/2300792704 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c3c0d16a0 0x7f0c3c0d1e30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0c28006fd0 tx=0x7f0c28009bd0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.219+0000 7f0c44000700 1 -- 172.21.15.89:0/2300792704 shutdown_connections 2024-04-02T08:52:29.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.219+0000 7f0c44000700 1 --2- 172.21.15.89:0/2300792704 >> v2:172.21.15.113:3300/0 conn(0x7f0c3c0d2370 0x7f0c3c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.219+0000 7f0c44000700 1 --2- 172.21.15.89:0/2300792704 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c3c0d16a0 0x7f0c3c0d1e30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.219+0000 7f0c44000700 1 -- 172.21.15.89:0/2300792704 >> 172.21.15.89:0/2300792704 conn(0x7f0c3c08aff0 msgr2=0x7f0c3c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:29.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.219+0000 7f0c44000700 1 -- 172.21.15.89:0/2300792704 shutdown_connections 2024-04-02T08:52:29.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.219+0000 7f0c44000700 1 -- 172.21.15.89:0/2300792704 wait complete. 2024-04-02T08:52:29.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.220+0000 7f0c44000700 1 Processor -- start 2024-04-02T08:52:29.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.220+0000 7f0c44000700 1 -- start start 2024-04-02T08:52:29.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.221+0000 7f0c44000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c3c0d16a0 0x7f0c3c1f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:29.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.221+0000 7f0c44000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c3c0d2370 0x7f0c3c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:29.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.221+0000 7f0c44000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c3c15d9d0 0x7f0c3c155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:29.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.221+0000 7f0c44000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c0b6020 con 0x7f0c3c0d2370 2024-04-02T08:52:29.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.221+0000 7f0c44000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c0b65e0 con 0x7f0c3c0d16a0 2024-04-02T08:52:29.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.221+0000 7f0c44000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c1f7870 con 0x7f0c3c15d9d0 2024-04-02T08:52:29.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.222+0000 7f0c40c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c3c0d2370 0x7f0c3c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:29.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.222+0000 7f0c40c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c3c0d2370 0x7f0c3c0b4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36918/0 (socket says 172.21.15.89:36918) 2024-04-02T08:52:29.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.222+0000 7f0c40c00700 1 -- 172.21.15.89:0/1833625680 learned_addr learned my addr 172.21.15.89:0/1833625680 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:29.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.222+0000 7f0c42000700 1 --2- 172.21.15.89:0/1833625680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c3c15d9d0 0x7f0c3c155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:29.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.222+0000 7f0c40c00700 1 -- 172.21.15.89:0/1833625680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c3c15d9d0 msgr2=0x7f0c3c155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:29.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.222+0000 7f0c41600700 1 --2- 172.21.15.89:0/1833625680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c3c0d16a0 0x7f0c3c1f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:29.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.222+0000 7f0c40c00700 1 --2- 172.21.15.89:0/1833625680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c3c15d9d0 0x7f0c3c155540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.222+0000 7f0c40c00700 1 -- 172.21.15.89:0/1833625680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c3c0d16a0 msgr2=0x7f0c3c1f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:29.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.222+0000 7f0c40c00700 1 --2- 172.21.15.89:0/1833625680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c3c0d16a0 0x7f0c3c1f7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.222+0000 7f0c40c00700 1 -- 172.21.15.89:0/1833625680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c28009500 con 0x7f0c3c0d2370 2024-04-02T08:52:29.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.222+0000 7f0c40c00700 1 --2- 172.21.15.89:0/1833625680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c3c0d2370 0x7f0c3c0b4e60 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0d0ff0 tx=0x7f0c28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:29.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.223+0000 7f0c39600700 1 -- 172.21.15.89:0/1833625680 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28003d90 con 0x7f0c3c0d2370 2024-04-02T08:52:29.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.223+0000 7f0c39600700 1 -- 172.21.15.89:0/1833625680 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2801d430 con 0x7f0c3c0d2370 2024-04-02T08:52:29.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.223+0000 7f0c39600700 1 -- 172.21.15.89:0/1833625680 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2801d7e0 con 0x7f0c3c0d2370 2024-04-02T08:52:29.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.223+0000 7f0c44000700 1 -- 172.21.15.89:0/1833625680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c3c0cc670 con 0x7f0c3c0d2370 2024-04-02T08:52:29.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.223+0000 7f0c44000700 1 -- 172.21.15.89:0/1833625680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3c0b6270 con 0x7f0c3c0d2370 2024-04-02T08:52:29.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.224+0000 7f0c44000700 1 -- 172.21.15.89:0/1833625680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c3c0d1ca0 con 0x7f0c3c0d2370 2024-04-02T08:52:29.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.224+0000 7f0c39600700 1 -- 172.21.15.89:0/1833625680 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 100) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f0c2801d940 con 0x7f0c3c0d2370 2024-04-02T08:52:29.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.225+0000 7f0c39600700 1 -- 172.21.15.89:0/1833625680 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6731+0+0 (secure 0 0 0) 0x7f0c2808a690 con 0x7f0c3c0d2370 2024-04-02T08:52:29.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.230+0000 7f0c39600700 1 -- 172.21.15.89:0/1833625680 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0c280557d0 con 0x7f0c3c0d2370 2024-04-02T08:52:29.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.361+0000 7f0c39600700 1 -- 172.21.15.89:0/1833625680 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 101) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f0c28055280 con 0x7f0c3c0d2370 2024-04-02T08:52:29.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.362+0000 7f0c39600700 1 --2- 172.21.15.89:0/1833625680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c1c036150 0x7f0c1c072710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:29.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.363+0000 7f0c41600700 1 --2- 172.21.15.89:0/1833625680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c1c036150 0x7f0c1c072710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:29.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.366+0000 7f0c41600700 1 --2- 172.21.15.89:0/1833625680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c1c036150 0x7f0c1c072710 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0c30000f60 tx=0x7f0c30009040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:29.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.485+0000 7f0c44000700 1 -- 172.21.15.89:0/1833625680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0c3c067720 con 0x7f0c3c0d2370 2024-04-02T08:52:29.484 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T08:52:29.486+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T08:52:29.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.498+0000 7f0c39600700 1 -- 172.21.15.89:0/1833625680 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0c2805cd50 con 0x7f0c3c0d2370 2024-04-02T08:52:29.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.501+0000 7f0c44000700 1 -- 172.21.15.89:0/1833625680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c1c036150 msgr2=0x7f0c1c072710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.501+0000 7f0c44000700 1 --2- 172.21.15.89:0/1833625680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c1c036150 0x7f0c1c072710 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0c30000f60 tx=0x7f0c30009040 comp rx=0 tx=0).stop 2024-04-02T08:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.501+0000 7f0c44000700 1 -- 172.21.15.89:0/1833625680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c3c0d2370 msgr2=0x7f0c3c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.501+0000 7f0c44000700 1 --2- 172.21.15.89:0/1833625680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c3c0d2370 0x7f0c3c0b4e60 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0d0ff0 tx=0x7f0c28004750 comp rx=0 tx=0).stop 2024-04-02T08:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.501+0000 7f0c44000700 1 -- 172.21.15.89:0/1833625680 shutdown_connections 2024-04-02T08:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.501+0000 7f0c44000700 1 --2- 172.21.15.89:0/1833625680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c1c036150 0x7f0c1c072710 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.501+0000 7f0c44000700 1 --2- 172.21.15.89:0/1833625680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c3c0d16a0 0x7f0c3c1f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.501+0000 7f0c44000700 1 --2- 172.21.15.89:0/1833625680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c3c15d9d0 0x7f0c3c155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.501+0000 7f0c44000700 1 --2- 172.21.15.89:0/1833625680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c3c0d2370 0x7f0c3c0b4e60 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.501+0000 7f0c44000700 1 -- 172.21.15.89:0/1833625680 >> 172.21.15.89:0/1833625680 conn(0x7f0c3c08aff0 msgr2=0x7f0c3c0b16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.501+0000 7f0c44000700 1 -- 172.21.15.89:0/1833625680 shutdown_connections 2024-04-02T08:52:29.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.501+0000 7f0c44000700 1 -- 172.21.15.89:0/1833625680 wait complete. 2024-04-02T08:52:29.501 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T08:52:29.518 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T08:52:29.565 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T08:52:29.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.716+0000 7fd588400700 1 Processor -- start 2024-04-02T08:52:29.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.717+0000 7fd588400700 1 -- start start 2024-04-02T08:52:29.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.718+0000 7fd588400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd5800cd020 0x7fd5800cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:29.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.718+0000 7fd588400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5800cdae0 con 0x7fd5800b2070 2024-04-02T08:52:29.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.718+0000 7fd588400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5800cddc0 con 0x7fd5800cbb30 2024-04-02T08:52:29.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.718+0000 7fd588400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5800ce0a0 con 0x7fd5800cd020 2024-04-02T08:52:29.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.718+0000 7fd585000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd5800cd020 0x7fd5800cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:29.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.718+0000 7fd586400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd5800b2070 0x7fd58006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44668/0 (socket says 172.21.15.89:44668) 2024-04-02T08:52:29.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.718+0000 7fd586400700 1 -- 172.21.15.89:0/4275259604 learned_addr learned my addr 172.21.15.89:0/4275259604 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:29.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.719+0000 7fd585000700 1 -- 172.21.15.89:0/4275259604 >> v1:172.21.15.113:6789/0 conn(0x7fd5800cbb30 legacy=0x7fd5800cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:29.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.719+0000 7fd57fe00700 1 -- 172.21.15.89:0/4275259604 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399655817 0 0) 0x7fd5800cddc0 con 0x7fd5800cbb30 2024-04-02T08:52:29.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.719+0000 7fd585000700 1 -- 172.21.15.89:0/4275259604 >> v1:172.21.15.89:6789/0 conn(0x7fd5800b2070 legacy=0x7fd58006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:29.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.719+0000 7fd585000700 1 -- 172.21.15.89:0/4275259604 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58006dc20 con 0x7fd5800cd020 2024-04-02T08:52:29.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.719+0000 7fd585000700 1 --2- 172.21.15.89:0/4275259604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5800cd020 0x7fd5800cd480 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fd56c009840 tx=0x7fd56c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c186401ba4ee16 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:29.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.719+0000 7fd57fe00700 1 -- 172.21.15.89:0/4275259604 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c004230 con 0x7fd5800cd020 2024-04-02T08:52:29.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.719+0000 7fd57fe00700 1 -- 172.21.15.89:0/4275259604 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56c004390 con 0x7fd5800cd020 2024-04-02T08:52:29.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.719+0000 7fd57fe00700 1 -- 172.21.15.89:0/4275259604 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c0036a0 con 0x7fd5800cd020 2024-04-02T08:52:29.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.720+0000 7fd588400700 1 -- 172.21.15.89:0/4275259604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5800cd020 msgr2=0x7fd5800cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:29.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.720+0000 7fd588400700 1 --2- 172.21.15.89:0/4275259604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5800cd020 0x7fd5800cd480 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fd56c009840 tx=0x7fd56c009b50 comp rx=0 tx=0).stop 2024-04-02T08:52:29.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.720+0000 7fd588400700 1 -- 172.21.15.89:0/4275259604 shutdown_connections 2024-04-02T08:52:29.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.720+0000 7fd588400700 1 --2- 172.21.15.89:0/4275259604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5800cd020 0x7fd5800cd480 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.720+0000 7fd588400700 1 -- 172.21.15.89:0/4275259604 >> 172.21.15.89:0/4275259604 conn(0x7fd5800b0d70 msgr2=0x7fd5800b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:29.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.720+0000 7fd588400700 1 -- 172.21.15.89:0/4275259604 shutdown_connections 2024-04-02T08:52:29.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.720+0000 7fd588400700 1 -- 172.21.15.89:0/4275259604 wait complete. 2024-04-02T08:52:29.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.721+0000 7fd588400700 1 Processor -- start 2024-04-02T08:52:29.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.722+0000 7fd588400700 1 -- start start 2024-04-02T08:52:29.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.722+0000 7fd588400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5800b2070 0x7fd5800adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:29.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd588400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5800cbb30 0x7fd580071c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:29.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd588400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5800cd020 0x7fd5800c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:29.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd588400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5801584b0 con 0x7fd5800cbb30 2024-04-02T08:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd588400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5800c8880 con 0x7fd5800b2070 2024-04-02T08:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd588400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5800c8b60 con 0x7fd5800cd020 2024-04-02T08:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd585000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5800cbb30 0x7fd580071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd585000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5800cbb30 0x7fd580071c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36942/0 (socket says 172.21.15.89:36942) 2024-04-02T08:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd585000700 1 -- 172.21.15.89:0/3179491927 learned_addr learned my addr 172.21.15.89:0/3179491927 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd586400700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5800cd020 0x7fd5800c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:29.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd585a00700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5800b2070 0x7fd5800adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:29.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd585000700 1 -- 172.21.15.89:0/3179491927 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5800cd020 msgr2=0x7fd5800c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:29.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd585000700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5800cd020 0x7fd5800c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd585000700 1 -- 172.21.15.89:0/3179491927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5800b2070 msgr2=0x7fd5800adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:29.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd585000700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5800b2070 0x7fd5800adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:29.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd585000700 1 -- 172.21.15.89:0/3179491927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c0094a0 con 0x7fd5800cbb30 2024-04-02T08:52:29.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd586400700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5800cd020 0x7fd5800c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:52:29.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.723+0000 7fd585000700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5800cbb30 0x7fd580071c80 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fd56c000c00 tx=0x7fd56c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:29.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.724+0000 7fd57d600700 1 -- 172.21.15.89:0/3179491927 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c011630 con 0x7fd5800cbb30 2024-04-02T08:52:29.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.724+0000 7fd57d600700 1 -- 172.21.15.89:0/3179491927 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56c01f070 con 0x7fd5800cbb30 2024-04-02T08:52:29.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.724+0000 7fd57d600700 1 -- 172.21.15.89:0/3179491927 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c0119b0 con 0x7fd5800cbb30 2024-04-02T08:52:29.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.724+0000 7fd588400700 1 -- 172.21.15.89:0/3179491927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5800c8e40 con 0x7fd5800cbb30 2024-04-02T08:52:29.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.724+0000 7fd588400700 1 -- 172.21.15.89:0/3179491927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5800c9400 con 0x7fd5800cbb30 2024-04-02T08:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.725+0000 7fd588400700 1 -- 172.21.15.89:0/3179491927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd580067720 con 0x7fd5800cbb30 2024-04-02T08:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.726+0000 7fd57d600700 1 -- 172.21.15.89:0/3179491927 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 101) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fd56c011b10 con 0x7fd5800cbb30 2024-04-02T08:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.726+0000 7fd57d600700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd56006aae0 0x7fd56006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.726+0000 7fd57d600700 1 -- 172.21.15.89:0/3179491927 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6817+0+0 (secure 0 0 0) 0x7fd56c021070 con 0x7fd5800cbb30 2024-04-02T08:52:29.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.727+0000 7fd585a00700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd56006aae0 0x7fd56006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:29.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.727+0000 7fd585a00700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd56006aae0 0x7fd56006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd574000f60 tx=0x7fd574009040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:29.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.731+0000 7fd57d600700 1 -- 172.21.15.89:0/3179491927 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd56c056090 con 0x7fd5800cbb30 2024-04-02T08:52:29.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:29.977+0000 7fd588400700 1 -- 172.21.15.89:0/3179491927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd5800ce2e0 con 0x7fd5800cbb30 2024-04-02T08:52:30.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.501+0000 7fd57d600700 1 -- 172.21.15.89:0/3179491927 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd56c02b0a0 con 0x7fd5800cbb30 2024-04-02T08:52:30.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.503+0000 7fd588400700 1 -- 172.21.15.89:0/3179491927 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd56006aae0 msgr2=0x7fd56006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:30.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.503+0000 7fd588400700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd56006aae0 0x7fd56006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd574000f60 tx=0x7fd574009040 comp rx=0 tx=0).stop 2024-04-02T08:52:30.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.503+0000 7fd588400700 1 -- 172.21.15.89:0/3179491927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5800cbb30 msgr2=0x7fd580071c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:30.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.503+0000 7fd588400700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5800cbb30 0x7fd580071c80 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fd56c000c00 tx=0x7fd56c004750 comp rx=0 tx=0).stop 2024-04-02T08:52:30.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.503+0000 7fd588400700 1 -- 172.21.15.89:0/3179491927 shutdown_connections 2024-04-02T08:52:30.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.503+0000 7fd588400700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd56006aae0 0x7fd56006cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:30.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.503+0000 7fd588400700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5800b2070 0x7fd5800adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:30.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.503+0000 7fd588400700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5800cd020 0x7fd5800c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:30.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.503+0000 7fd588400700 1 --2- 172.21.15.89:0/3179491927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5800cbb30 0x7fd580071c80 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:30.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.503+0000 7fd588400700 1 -- 172.21.15.89:0/3179491927 >> 172.21.15.89:0/3179491927 conn(0x7fd5800b0d70 msgr2=0x7fd5800af440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:30.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.503+0000 7fd588400700 1 -- 172.21.15.89:0/3179491927 shutdown_connections 2024-04-02T08:52:30.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.504+0000 7fd588400700 1 -- 172.21.15.89:0/3179491927 wait complete. 2024-04-02T08:52:30.521 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T08:52:30.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.718+0000 7f4ac3800700 1 Processor -- start 2024-04-02T08:52:30.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.718+0000 7f4ac3800700 1 -- start start 2024-04-02T08:52:30.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.719+0000 7f4ac3800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4abc15ddf0 0x7f4abc0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:30.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.719+0000 7f4ac3800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4abc15e1d0 0x7f4abc0d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:30.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.719+0000 7f4ac3800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4abc0ce8a0 con 0x7f4abc0d2710 2024-04-02T08:52:30.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.719+0000 7f4ac3800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4abc0d2b80 con 0x7f4abc15ddf0 2024-04-02T08:52:30.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.719+0000 7f4ac3800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc15ec60 con 0x7f4abc15e1d0 2024-04-02T08:52:30.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.719+0000 7f4ac0e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4abc15ddf0 0x7f4abc0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:30.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.720+0000 7f4abbe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4abc15e1d0 0x7f4abc0d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:30.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.720+0000 7f4ac0e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4abc15ddf0 0x7f4abc0ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:46900/0 (socket says 172.21.15.89:46900) 2024-04-02T08:52:30.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.720+0000 7f4ac0e00700 1 -- 172.21.15.89:0/1900787565 learned_addr learned my addr 172.21.15.89:0/1900787565 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:30.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.720+0000 7f4ac0e00700 1 -- 172.21.15.89:0/1900787565 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4abc15e1d0 msgr2=0x7f4abc0d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:30.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.720+0000 7f4ac0e00700 1 --2- 172.21.15.89:0/1900787565 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4abc15e1d0 0x7f4abc0d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:30.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.720+0000 7f4ac0e00700 1 -- 172.21.15.89:0/1900787565 >> v1:172.21.15.113:6789/0 conn(0x7f4abc0d2710 legacy=0x7f4abc1612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:30.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.720+0000 7f4ac0e00700 1 -- 172.21.15.89:0/1900787565 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc0cdc90 con 0x7f4abc15ddf0 2024-04-02T08:52:30.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.720+0000 7f4abb400700 1 -- 172.21.15.89:0/1900787565 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069304826 0 0) 0x7f4abc0ce8a0 con 0x7f4abc0d2710 2024-04-02T08:52:30.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.720+0000 7f4ac0e00700 1 --2- 172.21.15.89:0/1900787565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4abc15ddf0 0x7f4abc0ced30 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f4ab00098f0 tx=0x7f4ab0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ba18055615ca046 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:30.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.721+0000 7f4abb400700 1 -- 172.21.15.89:0/1900787565 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0011830 con 0x7f4abc15ddf0 2024-04-02T08:52:30.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.721+0000 7f4abb400700 1 -- 172.21.15.89:0/1900787565 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab0011990 con 0x7f4abc15ddf0 2024-04-02T08:52:30.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.721+0000 7f4abb400700 1 -- 172.21.15.89:0/1900787565 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0011d20 con 0x7f4abc15ddf0 2024-04-02T08:52:30.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.721+0000 7f4ac3800700 1 -- 172.21.15.89:0/1900787565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4abc15ddf0 msgr2=0x7f4abc0ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:30.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.721+0000 7f4ac3800700 1 --2- 172.21.15.89:0/1900787565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4abc15ddf0 0x7f4abc0ced30 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f4ab00098f0 tx=0x7f4ab0009c00 comp rx=0 tx=0).stop 2024-04-02T08:52:30.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.721+0000 7f4ac3800700 1 -- 172.21.15.89:0/1900787565 shutdown_connections 2024-04-02T08:52:30.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.721+0000 7f4ac3800700 1 --2- 172.21.15.89:0/1900787565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4abc15ddf0 0x7f4abc0ced30 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:30.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.721+0000 7f4ac3800700 1 --2- 172.21.15.89:0/1900787565 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4abc15e1d0 0x7f4abc0d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:30.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.721+0000 7f4ac3800700 1 -- 172.21.15.89:0/1900787565 >> 172.21.15.89:0/1900787565 conn(0x7f4abc08aff0 msgr2=0x7f4abc0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:30.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.721+0000 7f4ac3800700 1 -- 172.21.15.89:0/1900787565 shutdown_connections 2024-04-02T08:52:30.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.721+0000 7f4ac3800700 1 -- 172.21.15.89:0/1900787565 wait complete. 2024-04-02T08:52:30.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.724+0000 7f4ac3800700 1 Processor -- start 2024-04-02T08:52:30.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.724+0000 7f4ac3800700 1 -- start start 2024-04-02T08:52:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.725+0000 7f4ac3800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4abc0d2710 0x7f4abc152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.725+0000 7f4ac3800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4abc15ddf0 0x7f4abc0a0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.725+0000 7f4ac3800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4abc15e1d0 0x7f4abc0c8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.725+0000 7f4ac3800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc0a1220 con 0x7f4abc0d2710 2024-04-02T08:52:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.725+0000 7f4ac3800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc152f70 con 0x7f4abc15e1d0 2024-04-02T08:52:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.725+0000 7f4ac3800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc0a0980 con 0x7f4abc15ddf0 2024-04-02T08:52:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.725+0000 7f4ac0e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4abc0d2710 0x7f4abc152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.725+0000 7f4ac0e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4abc0d2710 0x7f4abc152a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36952/0 (socket says 172.21.15.89:36952) 2024-04-02T08:52:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.725+0000 7f4ac0e00700 1 -- 172.21.15.89:0/3966229558 learned_addr learned my addr 172.21.15.89:0/3966229558 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4abbe00700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4abc15ddf0 0x7f4abc0a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4ac0e00700 1 -- 172.21.15.89:0/3966229558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4abc15ddf0 msgr2=0x7f4abc0a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4ac1800700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4abc15e1d0 0x7f4abc0c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4ac0e00700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4abc15ddf0 0x7f4abc0a0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4ac0e00700 1 -- 172.21.15.89:0/3966229558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4abc15e1d0 msgr2=0x7f4abc0c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4ac0e00700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4abc15e1d0 0x7f4abc0c8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4ac0e00700 1 -- 172.21.15.89:0/3966229558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab0009550 con 0x7f4abc0d2710 2024-04-02T08:52:30.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4ac1800700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4abc15e1d0 0x7f4abc0c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:52:30.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4abbe00700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4abc15ddf0 0x7f4abc0a0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:52:30.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4ac0e00700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4abc0d2710 0x7f4abc152a30 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f4abc08ce40 tx=0x7f4ab0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:30.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4ab8c00700 1 -- 172.21.15.89:0/3966229558 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0004470 con 0x7f4abc0d2710 2024-04-02T08:52:30.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4ab8c00700 1 -- 172.21.15.89:0/3966229558 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab001f070 con 0x7f4abc0d2710 2024-04-02T08:52:30.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4ac3800700 1 -- 172.21.15.89:0/3966229558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4abc0a0c60 con 0x7f4abc0d2710 2024-04-02T08:52:30.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4ab8c00700 1 -- 172.21.15.89:0/3966229558 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab00038e0 con 0x7f4abc0d2710 2024-04-02T08:52:30.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.726+0000 7f4ac3800700 1 -- 172.21.15.89:0/3966229558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4abc160690 con 0x7f4abc0d2710 2024-04-02T08:52:30.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.727+0000 7f4ac3800700 1 -- 172.21.15.89:0/3966229558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4abc1604f0 con 0x7f4abc0d2710 2024-04-02T08:52:30.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.728+0000 7f4ab8c00700 1 -- 172.21.15.89:0/3966229558 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 101) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f4ab0021030 con 0x7f4abc0d2710 2024-04-02T08:52:30.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.729+0000 7f4ab8c00700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a9c06aae0 0x7f4a9c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:30.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.729+0000 7f4ab8c00700 1 -- 172.21.15.89:0/3966229558 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f4ab008a8f0 con 0x7f4abc0d2710 2024-04-02T08:52:30.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.729+0000 7f4abbe00700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a9c06aae0 0x7f4a9c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:30.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.730+0000 7f4abbe00700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a9c06aae0 0x7f4a9c06cfa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4abc153240 tx=0x7f4aa8008040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:30.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.734+0000 7f4ab8c00700 1 -- 172.21.15.89:0/3966229558 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4abc1604f0 con 0x7f4abc0d2710 2024-04-02T08:52:30.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.920+0000 7f4ab8c00700 1 -- 172.21.15.89:0/3966229558 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 102) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f4ab0055cb0 con 0x7f4abc0d2710 2024-04-02T08:52:30.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:30.940+0000 7f4ac3800700 1 -- 172.21.15.89:0/3966229558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4abc1604f0 con 0x7f4abc0d2710 2024-04-02T08:52:31.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:31.930+0000 7f4ab8c00700 1 -- 172.21.15.89:0/3966229558 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v162) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4abc1604f0 con 0x7f4abc0d2710 2024-04-02T08:52:31.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:31.933+0000 7f4ac3800700 1 -- 172.21.15.89:0/3966229558 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a9c06aae0 msgr2=0x7f4a9c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:31.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:31.933+0000 7f4ac3800700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a9c06aae0 0x7f4a9c06cfa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4abc153240 tx=0x7f4aa8008040 comp rx=0 tx=0).stop 2024-04-02T08:52:31.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:31.933+0000 7f4ac3800700 1 -- 172.21.15.89:0/3966229558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4abc0d2710 msgr2=0x7f4abc152a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:31.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:31.933+0000 7f4ac3800700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4abc0d2710 0x7f4abc152a30 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f4abc08ce40 tx=0x7f4ab0004750 comp rx=0 tx=0).stop 2024-04-02T08:52:31.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:31.933+0000 7f4ac3800700 1 -- 172.21.15.89:0/3966229558 shutdown_connections 2024-04-02T08:52:31.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:31.933+0000 7f4ac3800700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4a9c06aae0 0x7f4a9c06cfa0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:31.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:31.933+0000 7f4ac3800700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4abc15e1d0 0x7f4abc0c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:31.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:31.933+0000 7f4ac3800700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4abc0d2710 0x7f4abc152a30 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:31.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:31.933+0000 7f4ac3800700 1 --2- 172.21.15.89:0/3966229558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4abc15ddf0 0x7f4abc0a0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:31.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:31.933+0000 7f4ac3800700 1 -- 172.21.15.89:0/3966229558 >> 172.21.15.89:0/3966229558 conn(0x7f4abc08aff0 msgr2=0x7f4abc0b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:31.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:31.933+0000 7f4ac3800700 1 -- 172.21.15.89:0/3966229558 shutdown_connections 2024-04-02T08:52:31.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:31.933+0000 7f4ac3800700 1 -- 172.21.15.89:0/3966229558 wait complete. 2024-04-02T08:52:31.933 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T08:52:31.953 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T08:52:32.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.155+0000 7f398ec00700 1 Processor -- start 2024-04-02T08:52:32.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.155+0000 7f398ec00700 1 -- start start 2024-04-02T08:52:32.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.156+0000 7f398ec00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f398815d7f0 0x7f39880d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.156+0000 7f398ec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39880b56e0 0x7f398806ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.156+0000 7f398ec00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39880ceb70 con 0x7f39880d1770 2024-04-02T08:52:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.156+0000 7f398ec00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f398815dbd0 con 0x7f398815d7f0 2024-04-02T08:52:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.156+0000 7f398ec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39880cf000 con 0x7f39880b56e0 2024-04-02T08:52:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.156+0000 7f3987e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f398815d7f0 0x7f39880d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.156+0000 7f3987e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f398815d7f0 0x7f39880d25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36954/0 (socket says 172.21.15.89:36954) 2024-04-02T08:52:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.156+0000 7f3987e00700 1 -- 172.21.15.89:0/3474507772 learned_addr learned my addr 172.21.15.89:0/3474507772 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.157+0000 7f398cc00700 1 --2- 172.21.15.89:0/3474507772 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39880b56e0 0x7f398806ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.157+0000 7f3987e00700 1 -- 172.21.15.89:0/3474507772 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39880b56e0 msgr2=0x7f398806ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.157+0000 7f3987e00700 1 --2- 172.21.15.89:0/3474507772 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39880b56e0 0x7f398806ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.157+0000 7f3987e00700 1 -- 172.21.15.89:0/3474507772 >> v1:172.21.15.113:6789/0 conn(0x7f39880d1770 legacy=0x7f39880b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.157+0000 7f3987e00700 1 -- 172.21.15.89:0/3474507772 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39880cdf60 con 0x7f398815d7f0 2024-04-02T08:52:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.157+0000 7f398cc00700 1 --2- 172.21.15.89:0/3474507772 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39880b56e0 0x7f398806ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:52:32.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.157+0000 7f3987e00700 1 --2- 172.21.15.89:0/3474507772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398815d7f0 0x7f39880d25d0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f39780098f0 tx=0x7f3978009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33cdfcbd61e0bfb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:32.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.157+0000 7f3986a00700 1 -- 172.21.15.89:0/3474507772 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3978004230 con 0x7f398815d7f0 2024-04-02T08:52:32.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.157+0000 7f3986a00700 1 -- 172.21.15.89:0/3474507772 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3978004390 con 0x7f398815d7f0 2024-04-02T08:52:32.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.157+0000 7f3986a00700 1 -- 172.21.15.89:0/3474507772 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39780036a0 con 0x7f398815d7f0 2024-04-02T08:52:32.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.158+0000 7f398ec00700 1 -- 172.21.15.89:0/3474507772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398815d7f0 msgr2=0x7f39880d25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:32.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.158+0000 7f398ec00700 1 --2- 172.21.15.89:0/3474507772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398815d7f0 0x7f39880d25d0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f39780098f0 tx=0x7f3978009c00 comp rx=0 tx=0).stop 2024-04-02T08:52:32.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.158+0000 7f398ec00700 1 -- 172.21.15.89:0/3474507772 shutdown_connections 2024-04-02T08:52:32.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.158+0000 7f398ec00700 1 --2- 172.21.15.89:0/3474507772 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39880b56e0 0x7f398806ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:32.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.158+0000 7f398ec00700 1 --2- 172.21.15.89:0/3474507772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398815d7f0 0x7f39880d25d0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:32.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.158+0000 7f398ec00700 1 -- 172.21.15.89:0/3474507772 >> 172.21.15.89:0/3474507772 conn(0x7f398808aff0 msgr2=0x7f39880d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:32.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.158+0000 7f398ec00700 1 -- 172.21.15.89:0/3474507772 shutdown_connections 2024-04-02T08:52:32.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.158+0000 7f398ec00700 1 -- 172.21.15.89:0/3474507772 wait complete. 2024-04-02T08:52:32.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.160+0000 7f398ec00700 1 Processor -- start 2024-04-02T08:52:32.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.160+0000 7f398ec00700 1 -- start start 2024-04-02T08:52:32.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398ec00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39880b56e0 0x7f39880b6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:32.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398ec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39880d1770 0x7f3988152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:32.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398ec00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398815d7f0 0x7f39880b8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:32.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398ec00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398815ee30 con 0x7f398815d7f0 2024-04-02T08:52:32.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398ec00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39880cc6d0 con 0x7f39880b56e0 2024-04-02T08:52:32.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398ec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39881532b0 con 0x7f39880d1770 2024-04-02T08:52:32.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398815d7f0 0x7f39880b8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398815d7f0 0x7f39880b8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36956/0 (socket says 172.21.15.89:36956) 2024-04-02T08:52:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398cc00700 1 -- 172.21.15.89:0/1965538977 learned_addr learned my addr 172.21.15.89:0/1965538977 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f3987e00700 1 --2- 172.21.15.89:0/1965538977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39880b56e0 0x7f39880b6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f3987400700 1 --2- 172.21.15.89:0/1965538977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39880d1770 0x7f3988152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398cc00700 1 -- 172.21.15.89:0/1965538977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39880d1770 msgr2=0x7f3988152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398cc00700 1 --2- 172.21.15.89:0/1965538977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39880d1770 0x7f3988152d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398cc00700 1 -- 172.21.15.89:0/1965538977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39880b56e0 msgr2=0x7f39880b6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398cc00700 1 --2- 172.21.15.89:0/1965538977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39880b56e0 0x7f39880b6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398cc00700 1 -- 172.21.15.89:0/1965538977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3978009550 con 0x7f398815d7f0 2024-04-02T08:52:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.161+0000 7f398cc00700 1 --2- 172.21.15.89:0/1965538977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398815d7f0 0x7f39880b8470 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f398000ce70 tx=0x7f398000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.162+0000 7f397fe00700 1 -- 172.21.15.89:0/1965538977 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000ae10 con 0x7f398815d7f0 2024-04-02T08:52:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.162+0000 7f397fe00700 1 -- 172.21.15.89:0/1965538977 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3980003a20 con 0x7f398815d7f0 2024-04-02T08:52:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.162+0000 7f398ec00700 1 -- 172.21.15.89:0/1965538977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3988153cc0 con 0x7f398815d7f0 2024-04-02T08:52:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.162+0000 7f397fe00700 1 -- 172.21.15.89:0/1965538977 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980003dd0 con 0x7f398815d7f0 2024-04-02T08:52:32.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.162+0000 7f398ec00700 1 -- 172.21.15.89:0/1965538977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39880cc3f0 con 0x7f398815d7f0 2024-04-02T08:52:32.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.163+0000 7f398ec00700 1 -- 172.21.15.89:0/1965538977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398815ef70 con 0x7f398815d7f0 2024-04-02T08:52:32.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.164+0000 7f397fe00700 1 -- 172.21.15.89:0/1965538977 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 102) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f3980003f30 con 0x7f398815d7f0 2024-04-02T08:52:32.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.165+0000 7f397fe00700 1 --2- 172.21.15.89:0/1965538977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f396806ad20 0x7f396806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:32.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.165+0000 7f397fe00700 1 -- 172.21.15.89:0/1965538977 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6379+0+0 (secure 0 0 0) 0x7f3980089a10 con 0x7f398815d7f0 2024-04-02T08:52:32.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.165+0000 7f3987e00700 1 --2- 172.21.15.89:0/1965538977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f396806ad20 0x7f396806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:32.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.166+0000 7f3987e00700 1 --2- 172.21.15.89:0/1965538977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f396806ad20 0x7f396806d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3978009130 tx=0x7f3978008030 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:32.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.170+0000 7f397fe00700 1 -- 172.21.15.89:0/1965538977 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f398815ef70 con 0x7f398815d7f0 2024-04-02T08:52:32.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.383+0000 7f398ec00700 1 -- 172.21.15.89:0/1965538977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f398815ef70 con 0x7f398815d7f0 2024-04-02T08:52:32.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.933+0000 7f397fe00700 1 -- 172.21.15.89:0/1965538977 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v163) v1 ==== 169+0+0 (secure 0 0 0) 0x7f398815ef70 con 0x7f398815d7f0 2024-04-02T08:52:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.937+0000 7f398ec00700 1 -- 172.21.15.89:0/1965538977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f396806ad20 msgr2=0x7f396806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.937+0000 7f398ec00700 1 --2- 172.21.15.89:0/1965538977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f396806ad20 0x7f396806d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3978009130 tx=0x7f3978008030 comp rx=0 tx=0).stop 2024-04-02T08:52:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.937+0000 7f398ec00700 1 -- 172.21.15.89:0/1965538977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398815d7f0 msgr2=0x7f39880b8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.937+0000 7f398ec00700 1 --2- 172.21.15.89:0/1965538977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398815d7f0 0x7f39880b8470 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f398000ce70 tx=0x7f398000cf50 comp rx=0 tx=0).stop 2024-04-02T08:52:32.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.937+0000 7f398ec00700 1 -- 172.21.15.89:0/1965538977 shutdown_connections 2024-04-02T08:52:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.937+0000 7f398ec00700 1 --2- 172.21.15.89:0/1965538977 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f396806ad20 0x7f396806d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.937+0000 7f398ec00700 1 --2- 172.21.15.89:0/1965538977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39880b56e0 0x7f39880b6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.937+0000 7f398ec00700 1 --2- 172.21.15.89:0/1965538977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f398815d7f0 0x7f39880b8470 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.937+0000 7f398ec00700 1 --2- 172.21.15.89:0/1965538977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39880d1770 0x7f3988152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.937+0000 7f398ec00700 1 -- 172.21.15.89:0/1965538977 >> 172.21.15.89:0/1965538977 conn(0x7f398808aff0 msgr2=0x7f39880d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.937+0000 7f398ec00700 1 -- 172.21.15.89:0/1965538977 shutdown_connections 2024-04-02T08:52:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:32.937+0000 7f398ec00700 1 -- 172.21.15.89:0/1965538977 wait complete. 2024-04-02T08:52:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T08:52:32.958 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T08:52:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.153+0000 7f706d600700 1 Processor -- start 2024-04-02T08:52:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.154+0000 7f706d600700 1 -- start start 2024-04-02T08:52:33.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.155+0000 7f706d600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f70680d16a0 0x7f70680d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:33.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.155+0000 7f706d600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f70680d2370 0x7f70680ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:33.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.155+0000 7f706d600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7068161420 con 0x7f706815d9d0 2024-04-02T08:52:33.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.155+0000 7f706d600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70680d2820 con 0x7f70680d16a0 2024-04-02T08:52:33.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.155+0000 7f706d600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70680d4340 con 0x7f70680d2370 2024-04-02T08:52:33.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.155+0000 7f705fe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f70680d16a0 0x7f70680d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:33.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.155+0000 7f7066a00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f706815d9d0 0x7f70680d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:47240/0 (socket says 172.21.15.89:47240) 2024-04-02T08:52:33.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.155+0000 7f7066a00700 1 -- 172.21.15.89:0/2092123235 learned_addr learned my addr 172.21.15.89:0/2092123235 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:33.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.155+0000 7f7067400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f70680d2370 0x7f70680ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:33.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.156+0000 7f7067400700 1 -- 172.21.15.89:0/2092123235 >> v2:172.21.15.113:3300/0 conn(0x7f70680d16a0 msgr2=0x7f70680d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:33.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.156+0000 7f7067400700 1 --2- 172.21.15.89:0/2092123235 >> v2:172.21.15.113:3300/0 conn(0x7f70680d16a0 0x7f70680d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.156+0000 7f7067400700 1 -- 172.21.15.89:0/2092123235 >> v1:172.21.15.113:6789/0 conn(0x7f706815d9d0 legacy=0x7f70680d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:33.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.156+0000 7f7067400700 1 -- 172.21.15.89:0/2092123235 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7068161700 con 0x7f70680d2370 2024-04-02T08:52:33.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.156+0000 7f705fe00700 1 --2- 172.21.15.89:0/2092123235 >> v2:172.21.15.113:3300/0 conn(0x7f70680d16a0 0x7f70680d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:33.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.156+0000 7f7066000700 1 -- 172.21.15.89:0/2092123235 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729632536 0 0) 0x7f7068161420 con 0x7f706815d9d0 2024-04-02T08:52:33.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.156+0000 7f7067400700 1 --2- 172.21.15.89:0/2092123235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f70680d2370 0x7f70680ce360 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7060009d90 tx=0x7f706000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bb8c850f1edb547 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:33.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.157+0000 7f7066000700 1 -- 172.21.15.89:0/2092123235 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706000e070 con 0x7f70680d2370 2024-04-02T08:52:33.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.157+0000 7f7066000700 1 -- 172.21.15.89:0/2092123235 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706000bb20 con 0x7f70680d2370 2024-04-02T08:52:33.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.157+0000 7f7066000700 1 -- 172.21.15.89:0/2092123235 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7060012430 con 0x7f70680d2370 2024-04-02T08:52:33.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.157+0000 7f706d600700 1 -- 172.21.15.89:0/2092123235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f70680d2370 msgr2=0x7f70680ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:33.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.157+0000 7f706d600700 1 --2- 172.21.15.89:0/2092123235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f70680d2370 0x7f70680ce360 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7060009d90 tx=0x7f706000b330 comp rx=0 tx=0).stop 2024-04-02T08:52:33.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.157+0000 7f706d600700 1 -- 172.21.15.89:0/2092123235 shutdown_connections 2024-04-02T08:52:33.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.157+0000 7f706d600700 1 --2- 172.21.15.89:0/2092123235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f70680d2370 0x7f70680ce360 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.157+0000 7f706d600700 1 --2- 172.21.15.89:0/2092123235 >> v2:172.21.15.113:3300/0 conn(0x7f70680d16a0 0x7f70680d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.157+0000 7f706d600700 1 -- 172.21.15.89:0/2092123235 >> 172.21.15.89:0/2092123235 conn(0x7f706808aff0 msgr2=0x7f70680d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:33.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.157+0000 7f706d600700 1 -- 172.21.15.89:0/2092123235 shutdown_connections 2024-04-02T08:52:33.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.157+0000 7f706d600700 1 -- 172.21.15.89:0/2092123235 wait complete. 2024-04-02T08:52:33.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.159+0000 7f706d600700 1 Processor -- start 2024-04-02T08:52:33.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.159+0000 7f706d600700 1 -- start start 2024-04-02T08:52:33.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.160+0000 7f706d600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70680d16a0 0x7f70680cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:33.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.160+0000 7f706d600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70680d2370 0x7f7068156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:33.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.160+0000 7f706d600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f706815d9d0 0x7f7068160210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:33.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.160+0000 7f706d600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70680b89c0 con 0x7f70680d2370 2024-04-02T08:52:33.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.160+0000 7f706d600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706815df50 con 0x7f70680d16a0 2024-04-02T08:52:33.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.160+0000 7f706d600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706815e230 con 0x7f706815d9d0 2024-04-02T08:52:33.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.160+0000 7f705fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70680d2370 0x7f7068156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:33.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.160+0000 7f705fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70680d2370 0x7f7068156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36964/0 (socket says 172.21.15.89:36964) 2024-04-02T08:52:33.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.160+0000 7f705fe00700 1 -- 172.21.15.89:0/3789563391 learned_addr learned my addr 172.21.15.89:0/3789563391 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:33.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.160+0000 7f7066a00700 1 --2- 172.21.15.89:0/3789563391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70680d16a0 0x7f70680cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:33.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.160+0000 7f7067400700 1 --2- 172.21.15.89:0/3789563391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f706815d9d0 0x7f7068160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:33.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.160+0000 7f705fe00700 1 -- 172.21.15.89:0/3789563391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f706815d9d0 msgr2=0x7f7068160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:33.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.161+0000 7f705fe00700 1 --2- 172.21.15.89:0/3789563391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f706815d9d0 0x7f7068160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.161+0000 7f705fe00700 1 -- 172.21.15.89:0/3789563391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70680d16a0 msgr2=0x7f70680cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:33.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.161+0000 7f705fe00700 1 --2- 172.21.15.89:0/3789563391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70680d16a0 0x7f70680cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.161+0000 7f705fe00700 1 -- 172.21.15.89:0/3789563391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7050009430 con 0x7f70680d2370 2024-04-02T08:52:33.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.161+0000 7f705fe00700 1 --2- 172.21.15.89:0/3789563391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70680d2370 0x7f7068156940 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f705000ebe0 tx=0x7f7050009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:33.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.161+0000 7f705ea00700 1 -- 172.21.15.89:0/3789563391 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7050005a20 con 0x7f70680d2370 2024-04-02T08:52:33.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.161+0000 7f705ea00700 1 -- 172.21.15.89:0/3789563391 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7050005b80 con 0x7f70680d2370 2024-04-02T08:52:33.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.161+0000 7f706d600700 1 -- 172.21.15.89:0/3789563391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7060009a70 con 0x7f70680d2370 2024-04-02T08:52:33.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.161+0000 7f705ea00700 1 -- 172.21.15.89:0/3789563391 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705000c8b0 con 0x7f70680d2370 2024-04-02T08:52:33.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.161+0000 7f706d600700 1 -- 172.21.15.89:0/3789563391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706815e7f0 con 0x7f70680d2370 2024-04-02T08:52:33.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.162+0000 7f706d600700 1 -- 172.21.15.89:0/3789563391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70680d1ca0 con 0x7f70680d2370 2024-04-02T08:52:33.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.163+0000 7f705ea00700 1 -- 172.21.15.89:0/3789563391 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f705000ca10 con 0x7f70680d2370 2024-04-02T08:52:33.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.164+0000 7f705ea00700 1 --2- 172.21.15.89:0/3789563391 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f704406adb0 0x7f704406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:33.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.164+0000 7f705ea00700 1 -- 172.21.15.89:0/3789563391 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5965+0+0 (secure 0 0 0) 0x7f70500899b0 con 0x7f70680d2370 2024-04-02T08:52:33.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.164+0000 7f7066a00700 1 --2- 172.21.15.89:0/3789563391 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f704406adb0 0x7f704406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:33.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.165+0000 7f7066a00700 1 --2- 172.21.15.89:0/3789563391 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f704406adb0 0x7f704406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7058000f60 tx=0x7f7058007c90 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:33.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.168+0000 7f705ea00700 1 -- 172.21.15.89:0/3789563391 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f70500548e0 con 0x7f70680d2370 2024-04-02T08:52:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.375+0000 7f706d600700 1 -- 172.21.15.89:0/3789563391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f7068067720 con 0x7f70680d2370 2024-04-02T08:52:33.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.380+0000 7f705ea00700 1 -- 172.21.15.89:0/3789563391 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v55) v1 ==== 115+0+0 (secure 0 0 0) 0x7f70500200b0 con 0x7f70680d2370 2024-04-02T08:52:33.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.385+0000 7f706d600700 1 -- 172.21.15.89:0/3789563391 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f704406adb0 msgr2=0x7f704406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:33.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.385+0000 7f706d600700 1 --2- 172.21.15.89:0/3789563391 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f704406adb0 0x7f704406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7058000f60 tx=0x7f7058007c90 comp rx=0 tx=0).stop 2024-04-02T08:52:33.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.385+0000 7f706d600700 1 -- 172.21.15.89:0/3789563391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70680d2370 msgr2=0x7f7068156940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:33.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.385+0000 7f706d600700 1 --2- 172.21.15.89:0/3789563391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70680d2370 0x7f7068156940 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f705000ebe0 tx=0x7f7050009e40 comp rx=0 tx=0).stop 2024-04-02T08:52:33.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.385+0000 7f706d600700 1 -- 172.21.15.89:0/3789563391 shutdown_connections 2024-04-02T08:52:33.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.385+0000 7f706d600700 1 --2- 172.21.15.89:0/3789563391 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f704406adb0 0x7f704406d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.385+0000 7f706d600700 1 --2- 172.21.15.89:0/3789563391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70680d16a0 0x7f70680cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.385+0000 7f706d600700 1 --2- 172.21.15.89:0/3789563391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f706815d9d0 0x7f7068160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.385+0000 7f706d600700 1 --2- 172.21.15.89:0/3789563391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70680d2370 0x7f7068156940 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.385+0000 7f706d600700 1 -- 172.21.15.89:0/3789563391 >> 172.21.15.89:0/3789563391 conn(0x7f706808aff0 msgr2=0x7f70680b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:33.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.386+0000 7f706d600700 1 -- 172.21.15.89:0/3789563391 shutdown_connections 2024-04-02T08:52:33.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.386+0000 7f706d600700 1 -- 172.21.15.89:0/3789563391 wait complete. 2024-04-02T08:52:33.407 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T08:52:33.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.601+0000 7f6036c00700 1 Processor -- start 2024-04-02T08:52:33.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.601+0000 7f6036c00700 1 -- start start 2024-04-02T08:52:33.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.602+0000 7f6036c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60300d16a0 0x7f60300d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:33.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.602+0000 7f6036c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f60300d2370 0x7f60300ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:33.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.603+0000 7f6036c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6030161420 con 0x7f603015d9d0 2024-04-02T08:52:33.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.603+0000 7f6036c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60300d2820 con 0x7f60300d2370 2024-04-02T08:52:33.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.603+0000 7f6036c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60300d4340 con 0x7f60300d16a0 2024-04-02T08:52:33.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.603+0000 7f602f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60300d16a0 0x7f60300d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:33.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.603+0000 7f6034c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f60300d2370 0x7f60300ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:33.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.603+0000 7f6034c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f60300d2370 0x7f60300ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:46940/0 (socket says 172.21.15.89:46940) 2024-04-02T08:52:33.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.603+0000 7f6034c00700 1 -- 172.21.15.89:0/654071503 learned_addr learned my addr 172.21.15.89:0/654071503 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:33.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.603+0000 7f602fe00700 1 --1- 172.21.15.89:0/654071503 >> v1:172.21.15.113:6789/0 conn(0x7f603015d9d0 0x7f60300d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:47252/0 (socket says 172.21.15.89:47252) 2024-04-02T08:52:33.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.604+0000 7f6034c00700 1 -- 172.21.15.89:0/654071503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60300d16a0 msgr2=0x7f60300d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:33.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.604+0000 7f6034c00700 1 --2- 172.21.15.89:0/654071503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60300d16a0 0x7f60300d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.604+0000 7f6034c00700 1 -- 172.21.15.89:0/654071503 >> v1:172.21.15.113:6789/0 conn(0x7f603015d9d0 legacy=0x7f60300d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:33.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.604+0000 7f6034c00700 1 -- 172.21.15.89:0/654071503 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6030161700 con 0x7f60300d2370 2024-04-02T08:52:33.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.604+0000 7f602f400700 1 --2- 172.21.15.89:0/654071503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60300d16a0 0x7f60300d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:33.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.604+0000 7f602ea00700 1 -- 172.21.15.89:0/654071503 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567619147 0 0) 0x7f6030161420 con 0x7f603015d9d0 2024-04-02T08:52:33.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.604+0000 7f6034c00700 1 --2- 172.21.15.89:0/654071503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60300d2370 0x7f60300ce360 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6028009e80 tx=0x7f602800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8847104041ddc9d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:33.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.604+0000 7f602ea00700 1 -- 172.21.15.89:0/654071503 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602800e070 con 0x7f60300d2370 2024-04-02T08:52:33.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.604+0000 7f602ea00700 1 -- 172.21.15.89:0/654071503 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602800b8d0 con 0x7f60300d2370 2024-04-02T08:52:33.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.604+0000 7f602ea00700 1 -- 172.21.15.89:0/654071503 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602800bc80 con 0x7f60300d2370 2024-04-02T08:52:33.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.605+0000 7f6036c00700 1 -- 172.21.15.89:0/654071503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60300d2370 msgr2=0x7f60300ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:33.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.605+0000 7f6036c00700 1 --2- 172.21.15.89:0/654071503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60300d2370 0x7f60300ce360 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6028009e80 tx=0x7f602800b330 comp rx=0 tx=0).stop 2024-04-02T08:52:33.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.605+0000 7f6036c00700 1 -- 172.21.15.89:0/654071503 shutdown_connections 2024-04-02T08:52:33.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.605+0000 7f6036c00700 1 --2- 172.21.15.89:0/654071503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60300d2370 0x7f60300ce360 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.605+0000 7f6036c00700 1 --2- 172.21.15.89:0/654071503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60300d16a0 0x7f60300d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.605+0000 7f6036c00700 1 -- 172.21.15.89:0/654071503 >> 172.21.15.89:0/654071503 conn(0x7f603008aff0 msgr2=0x7f60300d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:33.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.605+0000 7f6036c00700 1 -- 172.21.15.89:0/654071503 shutdown_connections 2024-04-02T08:52:33.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.605+0000 7f6036c00700 1 -- 172.21.15.89:0/654071503 wait complete. 2024-04-02T08:52:33.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.606+0000 7f6036c00700 1 Processor -- start 2024-04-02T08:52:33.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.607+0000 7f6036c00700 1 -- start start 2024-04-02T08:52:33.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.607+0000 7f6036c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60300d16a0 0x7f60301f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:33.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.607+0000 7f6036c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60300d2370 0x7f603015f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:33.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.607+0000 7f6036c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f603015d9d0 0x7f60300b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:33.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.607+0000 7f6036c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6030160a80 con 0x7f603015d9d0 2024-04-02T08:52:33.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.607+0000 7f6036c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60301f64e0 con 0x7f60300d16a0 2024-04-02T08:52:33.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.607+0000 7f6036c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603015fe00 con 0x7f60300d2370 2024-04-02T08:52:33.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.608+0000 7f6034c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f603015d9d0 0x7f60300b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:33.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.608+0000 7f6034c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f603015d9d0 0x7f60300b45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36970/0 (socket says 172.21.15.89:36970) 2024-04-02T08:52:33.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.608+0000 7f6034c00700 1 -- 172.21.15.89:0/2163158831 learned_addr learned my addr 172.21.15.89:0/2163158831 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:33.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.608+0000 7f602f400700 1 --2- 172.21.15.89:0/2163158831 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60300d2370 0x7f603015f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:33.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.608+0000 7f602fe00700 1 --2- 172.21.15.89:0/2163158831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60300d16a0 0x7f60301f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:33.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.608+0000 7f6034c00700 1 -- 172.21.15.89:0/2163158831 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60300d2370 msgr2=0x7f603015f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:33.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.608+0000 7f6034c00700 1 --2- 172.21.15.89:0/2163158831 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60300d2370 0x7f603015f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.608+0000 7f6034c00700 1 -- 172.21.15.89:0/2163158831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60300d16a0 msgr2=0x7f60301f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:33.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.608+0000 7f6034c00700 1 --2- 172.21.15.89:0/2163158831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60300d16a0 0x7f60301f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.608+0000 7f6034c00700 1 -- 172.21.15.89:0/2163158831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601c009430 con 0x7f603015d9d0 2024-04-02T08:52:33.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.608+0000 7f6034c00700 1 --2- 172.21.15.89:0/2163158831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f603015d9d0 0x7f60300b45b0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f602800b680 tx=0x7f6028004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:33.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.609+0000 7f6027e00700 1 -- 172.21.15.89:0/2163158831 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602801c640 con 0x7f603015d9d0 2024-04-02T08:52:33.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.609+0000 7f6027e00700 1 -- 172.21.15.89:0/2163158831 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602801c7c0 con 0x7f603015d9d0 2024-04-02T08:52:33.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.609+0000 7f6027e00700 1 -- 172.21.15.89:0/2163158831 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602801cb70 con 0x7f603015d9d0 2024-04-02T08:52:33.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.609+0000 7f6036c00700 1 -- 172.21.15.89:0/2163158831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6028009b30 con 0x7f603015d9d0 2024-04-02T08:52:33.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.609+0000 7f6036c00700 1 -- 172.21.15.89:0/2163158831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60301603c0 con 0x7f603015d9d0 2024-04-02T08:52:33.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.610+0000 7f6036c00700 1 -- 172.21.15.89:0/2163158831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60300d1ca0 con 0x7f603015d9d0 2024-04-02T08:52:33.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.611+0000 7f6027e00700 1 -- 172.21.15.89:0/2163158831 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f602801ccd0 con 0x7f603015d9d0 2024-04-02T08:52:33.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.612+0000 7f6027e00700 1 --2- 172.21.15.89:0/2163158831 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f601006adb0 0x7f601006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:33.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.612+0000 7f6027e00700 1 -- 172.21.15.89:0/2163158831 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5965+0+0 (secure 0 0 0) 0x7f602808aff0 con 0x7f603015d9d0 2024-04-02T08:52:33.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.612+0000 7f602fe00700 1 --2- 172.21.15.89:0/2163158831 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f601006adb0 0x7f601006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:33.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.612+0000 7f602fe00700 1 --2- 172.21.15.89:0/2163158831 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f601006adb0 0x7f601006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6020000f60 tx=0x7f6020007e70 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:33.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.616+0000 7f6027e00700 1 -- 172.21.15.89:0/2163158831 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6028055f40 con 0x7f603015d9d0 2024-04-02T08:52:33.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.815+0000 7f6036c00700 1 -- 172.21.15.89:0/2163158831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f60300ce870 con 0x7f603015d9d0 2024-04-02T08:52:33.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.819+0000 7f6027e00700 1 -- 172.21.15.89:0/2163158831 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v56) v1 ==== 120+0+0 (secure 0 0 0) 0x7f6028059560 con 0x7f603015d9d0 2024-04-02T08:52:33.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.822+0000 7f6036c00700 1 -- 172.21.15.89:0/2163158831 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f601006adb0 msgr2=0x7f601006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:33.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.822+0000 7f6036c00700 1 --2- 172.21.15.89:0/2163158831 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f601006adb0 0x7f601006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6020000f60 tx=0x7f6020007e70 comp rx=0 tx=0).stop 2024-04-02T08:52:33.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.822+0000 7f6036c00700 1 -- 172.21.15.89:0/2163158831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f603015d9d0 msgr2=0x7f60300b45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:33.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.822+0000 7f6036c00700 1 --2- 172.21.15.89:0/2163158831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f603015d9d0 0x7f60300b45b0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f602800b680 tx=0x7f6028004ad0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.822+0000 7f6036c00700 1 -- 172.21.15.89:0/2163158831 shutdown_connections 2024-04-02T08:52:33.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.822+0000 7f6036c00700 1 --2- 172.21.15.89:0/2163158831 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f601006adb0 0x7f601006d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.822+0000 7f6036c00700 1 --2- 172.21.15.89:0/2163158831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60300d16a0 0x7f60301f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.822+0000 7f6036c00700 1 --2- 172.21.15.89:0/2163158831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f603015d9d0 0x7f60300b45b0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.822+0000 7f6036c00700 1 --2- 172.21.15.89:0/2163158831 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60300d2370 0x7f603015f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:33.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.822+0000 7f6036c00700 1 -- 172.21.15.89:0/2163158831 >> 172.21.15.89:0/2163158831 conn(0x7f603008aff0 msgr2=0x7f603015ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:33.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.822+0000 7f6036c00700 1 -- 172.21.15.89:0/2163158831 shutdown_connections 2024-04-02T08:52:33.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:33.822+0000 7f6036c00700 1 -- 172.21.15.89:0/2163158831 wait complete. 2024-04-02T08:52:33.840 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T08:52:34.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.038+0000 7f3f6da00700 1 Processor -- start 2024-04-02T08:52:34.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.038+0000 7f3f6da00700 1 -- start start 2024-04-02T08:52:34.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.039+0000 7f3f6da00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3f6815dfe0 0x7f3f680ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.040+0000 7f3f6da00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3f680ce670 0x7f3f680cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.040+0000 7f3f6da00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f680cb310 con 0x7f3f6815a5c0 2024-04-02T08:52:34.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.040+0000 7f3f6da00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f680ceb00 con 0x7f3f680ce670 2024-04-02T08:52:34.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.040+0000 7f3f6da00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f680711a0 con 0x7f3f6815dfe0 2024-04-02T08:52:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.040+0000 7f3f66a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3f6815a5c0 0x7f3f680709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44676/0 (socket says 172.21.15.89:44676) 2024-04-02T08:52:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.040+0000 7f3f66a00700 1 -- 172.21.15.89:0/2671568997 learned_addr learned my addr 172.21.15.89:0/2671568997 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.040+0000 7f3f67400700 1 --2- 172.21.15.89:0/2671568997 >> v2:172.21.15.89:3300/0 conn(0x7f3f680ce670 0x7f3f680cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.040+0000 7f3f66000700 1 --2- 172.21.15.89:0/2671568997 >> v2:172.21.15.113:3300/0 conn(0x7f3f6815dfe0 0x7f3f680ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.040+0000 7f3f67400700 1 -- 172.21.15.89:0/2671568997 >> v2:172.21.15.113:3300/0 conn(0x7f3f6815dfe0 msgr2=0x7f3f680ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.040+0000 7f3f67400700 1 --2- 172.21.15.89:0/2671568997 >> v2:172.21.15.113:3300/0 conn(0x7f3f6815dfe0 0x7f3f680ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.040+0000 7f3f67400700 1 -- 172.21.15.89:0/2671568997 >> v1:172.21.15.89:6789/0 conn(0x7f3f6815a5c0 legacy=0x7f3f680709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.040+0000 7f3f67400700 1 -- 172.21.15.89:0/2671568997 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f68071480 con 0x7f3f680ce670 2024-04-02T08:52:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.041+0000 7f3f67400700 1 --2- 172.21.15.89:0/2671568997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f680ce670 0x7f3f680cba80 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f3f680cb2c0 tx=0x7f3f6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b2511baf0eda28 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.041+0000 7f3f65600700 1 -- 172.21.15.89:0/2671568997 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6000e070 con 0x7f3f680ce670 2024-04-02T08:52:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.041+0000 7f3f65600700 1 -- 172.21.15.89:0/2671568997 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6000bb20 con 0x7f3f680ce670 2024-04-02T08:52:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.041+0000 7f3f65600700 1 -- 172.21.15.89:0/2671568997 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f60012430 con 0x7f3f680ce670 2024-04-02T08:52:34.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.041+0000 7f3f6da00700 1 -- 172.21.15.89:0/2671568997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f680ce670 msgr2=0x7f3f680cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:34.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.041+0000 7f3f6da00700 1 --2- 172.21.15.89:0/2671568997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f680ce670 0x7f3f680cba80 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f3f680cb2c0 tx=0x7f3f6000b330 comp rx=0 tx=0).stop 2024-04-02T08:52:34.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.041+0000 7f3f6da00700 1 -- 172.21.15.89:0/2671568997 shutdown_connections 2024-04-02T08:52:34.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.042+0000 7f3f6da00700 1 --2- 172.21.15.89:0/2671568997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f680ce670 0x7f3f680cba80 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.042+0000 7f3f6da00700 1 --2- 172.21.15.89:0/2671568997 >> v2:172.21.15.113:3300/0 conn(0x7f3f6815dfe0 0x7f3f680ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.042+0000 7f3f6da00700 1 -- 172.21.15.89:0/2671568997 >> 172.21.15.89:0/2671568997 conn(0x7f3f680b0d70 msgr2=0x7f3f680b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:34.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.042+0000 7f3f6da00700 1 -- 172.21.15.89:0/2671568997 shutdown_connections 2024-04-02T08:52:34.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.042+0000 7f3f6da00700 1 -- 172.21.15.89:0/2671568997 wait complete. 2024-04-02T08:52:34.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.043+0000 7f3f6da00700 1 Processor -- start 2024-04-02T08:52:34.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.043+0000 7f3f6da00700 1 -- start start 2024-04-02T08:52:34.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.044+0000 7f3f6da00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f680ce670 0x7f3f6814b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.044+0000 7f3f6da00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f6815a5c0 0x7f3f6814e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.044+0000 7f3f6da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f6815dfe0 0x7f3f6814e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.044+0000 7f3f6da00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f680ac050 con 0x7f3f6815dfe0 2024-04-02T08:52:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.044+0000 7f3f6da00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f680ac5f0 con 0x7f3f6815a5c0 2024-04-02T08:52:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.044+0000 7f3f6da00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6814edf0 con 0x7f3f680ce670 2024-04-02T08:52:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f67400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f6815dfe0 0x7f3f6814e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f67400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f6815dfe0 0x7f3f6814e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36988/0 (socket says 172.21.15.89:36988) 2024-04-02T08:52:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f67400700 1 -- 172.21.15.89:0/2600205041 learned_addr learned my addr 172.21.15.89:0/2600205041 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f66000700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f6815a5c0 0x7f3f6814e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f66a00700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f680ce670 0x7f3f6814b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f67400700 1 -- 172.21.15.89:0/2600205041 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f680ce670 msgr2=0x7f3f6814b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f67400700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f680ce670 0x7f3f6814b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f67400700 1 -- 172.21.15.89:0/2600205041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f6815a5c0 msgr2=0x7f3f6814e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f67400700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f6815a5c0 0x7f3f6814e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f67400700 1 -- 172.21.15.89:0/2600205041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f60009a70 con 0x7f3f6815dfe0 2024-04-02T08:52:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f66a00700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f680ce670 0x7f3f6814b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:34.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f67400700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f6815dfe0 0x7f3f6814e770 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f3f60014040 tx=0x7f3f60004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:34.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f5ea00700 1 -- 172.21.15.89:0/2600205041 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f60012430 con 0x7f3f6815dfe0 2024-04-02T08:52:34.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f5ea00700 1 -- 172.21.15.89:0/2600205041 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f60020070 con 0x7f3f6815dfe0 2024-04-02T08:52:34.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f6da00700 1 -- 172.21.15.89:0/2600205041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6814f0d0 con 0x7f3f6815dfe0 2024-04-02T08:52:34.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.045+0000 7f3f5ea00700 1 -- 172.21.15.89:0/2600205041 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f60004280 con 0x7f3f6815dfe0 2024-04-02T08:52:34.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.046+0000 7f3f6da00700 1 -- 172.21.15.89:0/2600205041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6814c1c0 con 0x7f3f6815dfe0 2024-04-02T08:52:34.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.047+0000 7f3f6da00700 1 -- 172.21.15.89:0/2600205041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f680ac7a0 con 0x7f3f6815dfe0 2024-04-02T08:52:34.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.047+0000 7f3f5ea00700 1 -- 172.21.15.89:0/2600205041 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3f600043e0 con 0x7f3f6815dfe0 2024-04-02T08:52:34.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.048+0000 7f3f5ea00700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f4806ade0 0x7f3f4806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.048+0000 7f3f5ea00700 1 -- 172.21.15.89:0/2600205041 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5965+0+0 (secure 0 0 0) 0x7f3f60022070 con 0x7f3f6815dfe0 2024-04-02T08:52:34.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.049+0000 7f3f66a00700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f4806ade0 0x7f3f4806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.049+0000 7f3f66a00700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f4806ade0 0x7f3f4806d2a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3f58007fd0 tx=0x7f3f58009040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:34.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.053+0000 7f3f5ea00700 1 -- 172.21.15.89:0/2600205041 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3f680ac7a0 con 0x7f3f6815dfe0 2024-04-02T08:52:34.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.255+0000 7f3f6da00700 1 -- 172.21.15.89:0/2600205041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3f680ac760 con 0x7f3f6815dfe0 2024-04-02T08:52:34.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.259+0000 7f3f5ea00700 1 -- 172.21.15.89:0/2600205041 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v57) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3f600562a0 con 0x7f3f6815dfe0 2024-04-02T08:52:34.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.261+0000 7f3f6da00700 1 -- 172.21.15.89:0/2600205041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f4806ade0 msgr2=0x7f3f4806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:34.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.261+0000 7f3f6da00700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f4806ade0 0x7f3f4806d2a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3f58007fd0 tx=0x7f3f58009040 comp rx=0 tx=0).stop 2024-04-02T08:52:34.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.261+0000 7f3f6da00700 1 -- 172.21.15.89:0/2600205041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f6815dfe0 msgr2=0x7f3f6814e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:34.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.261+0000 7f3f6da00700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f6815dfe0 0x7f3f6814e770 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f3f60014040 tx=0x7f3f60004ad0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.261+0000 7f3f6da00700 1 -- 172.21.15.89:0/2600205041 shutdown_connections 2024-04-02T08:52:34.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.261+0000 7f3f6da00700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3f4806ade0 0x7f3f4806d2a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.261+0000 7f3f6da00700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f6815dfe0 0x7f3f6814e770 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.261+0000 7f3f6da00700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f680ce670 0x7f3f6814b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.261+0000 7f3f6da00700 1 --2- 172.21.15.89:0/2600205041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f6815a5c0 0x7f3f6814e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.261+0000 7f3f6da00700 1 -- 172.21.15.89:0/2600205041 >> 172.21.15.89:0/2600205041 conn(0x7f3f680b0d70 msgr2=0x7f3f680b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:34.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.262+0000 7f3f6da00700 1 -- 172.21.15.89:0/2600205041 shutdown_connections 2024-04-02T08:52:34.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.262+0000 7f3f6da00700 1 -- 172.21.15.89:0/2600205041 wait complete. 2024-04-02T08:52:34.278 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-04-02T08:52:34.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.471+0000 7f117b800700 1 Processor -- start 2024-04-02T08:52:34.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.471+0000 7f117b800700 1 -- start start 2024-04-02T08:52:34.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.473+0000 7f117b800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f11740ceec0 0x7f11741612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.473+0000 7f117b800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1174161800 0x7f11740b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.473+0000 7f117b800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11740d30d0 con 0x7f117415d640 2024-04-02T08:52:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.473+0000 7f117b800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1174161c90 con 0x7f11740ceec0 2024-04-02T08:52:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.473+0000 7f117b800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11740b6840 con 0x7f1174161800 2024-04-02T08:52:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.473+0000 7f1178e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f117415d640 0x7f11740d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44680/0 (socket says 172.21.15.89:44680) 2024-04-02T08:52:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.473+0000 7f1178e00700 1 -- 172.21.15.89:0/58345138 learned_addr learned my addr 172.21.15.89:0/58345138 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.473+0000 7f1173e00700 1 --2- 172.21.15.89:0/58345138 >> v2:172.21.15.113:3300/0 conn(0x7f11740ceec0 0x7f11741612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.473+0000 7f1179800700 1 --2- 172.21.15.89:0/58345138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1174161800 0x7f11740b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.473+0000 7f1173400700 1 -- 172.21.15.89:0/58345138 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679275296 0 0) 0x7f11740d30d0 con 0x7f117415d640 2024-04-02T08:52:34.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.473+0000 7f1173400700 1 -- 172.21.15.89:0/58345138 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f115c0035b0 con 0x7f117415d640 2024-04-02T08:52:34.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173400700 1 -- 172.21.15.89:0/58345138 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4272179542 0 0) 0x7f115c0035b0 con 0x7f117415d640 2024-04-02T08:52:34.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173400700 1 -- 172.21.15.89:0/58345138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1174161800 msgr2=0x7f11740b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:34.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173400700 1 --2- 172.21.15.89:0/58345138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1174161800 0x7f11740b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173400700 1 -- 172.21.15.89:0/58345138 >> v2:172.21.15.113:3300/0 conn(0x7f11740ceec0 msgr2=0x7f11741612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:34.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173400700 1 --2- 172.21.15.89:0/58345138 >> v2:172.21.15.113:3300/0 conn(0x7f11740ceec0 0x7f11741612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173400700 1 -- 172.21.15.89:0/58345138 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11740d2b30 con 0x7f117415d640 2024-04-02T08:52:34.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173400700 1 -- 172.21.15.89:0/58345138 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f1168001670 con 0x7f117415d640 2024-04-02T08:52:34.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173400700 1 -- 172.21.15.89:0/58345138 >> v1:172.21.15.89:6789/0 conn(0x7f117415d640 legacy=0x7f11740d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:34.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173400700 1 --2- 172.21.15.89:0/58345138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1174161800 0x7f115c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173400700 1 --2- 172.21.15.89:0/58345138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c003f80 0x7f115c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173400700 1 --2- 172.21.15.89:0/58345138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f115c004d90 0x7f115c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173400700 1 -- 172.21.15.89:0/58345138 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1174161c90 con 0x7f115c003f80 2024-04-02T08:52:34.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173400700 1 -- 172.21.15.89:0/58345138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11740b6840 con 0x7f115c004d90 2024-04-02T08:52:34.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173400700 1 -- 172.21.15.89:0/58345138 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115c00ba60 con 0x7f1174161800 2024-04-02T08:52:34.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173e00700 1 --2- 172.21.15.89:0/58345138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c003f80 0x7f115c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1178e00700 1 --2- 172.21.15.89:0/58345138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1174161800 0x7f115c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1179800700 1 --2- 172.21.15.89:0/58345138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f115c004d90 0x7f115c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173e00700 1 -- 172.21.15.89:0/58345138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1174161800 msgr2=0x7f115c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:34.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173e00700 1 --2- 172.21.15.89:0/58345138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1174161800 0x7f115c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173e00700 1 -- 172.21.15.89:0/58345138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f115c004d90 msgr2=0x7f115c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:34.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173e00700 1 --2- 172.21.15.89:0/58345138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f115c004d90 0x7f115c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.474+0000 7f1173e00700 1 -- 172.21.15.89:0/58345138 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115c0035b0 con 0x7f115c003f80 2024-04-02T08:52:34.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.475+0000 7f1173e00700 1 --2- 172.21.15.89:0/58345138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c003f80 0x7f115c004850 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f1160009b90 tx=0x7f115c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=515c4feb4195f6a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:34.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.475+0000 7f1173400700 1 -- 172.21.15.89:0/58345138 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116000e6c0 con 0x7f115c003f80 2024-04-02T08:52:34.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.475+0000 7f1173400700 1 -- 172.21.15.89:0/58345138 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116000e820 con 0x7f115c003f80 2024-04-02T08:52:34.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.475+0000 7f1173400700 1 -- 172.21.15.89:0/58345138 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116000eb90 con 0x7f115c003f80 2024-04-02T08:52:34.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.475+0000 7f117b800700 1 -- 172.21.15.89:0/58345138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c003f80 msgr2=0x7f115c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.475+0000 7f117b800700 1 --2- 172.21.15.89:0/58345138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c003f80 0x7f115c004850 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f1160009b90 tx=0x7f115c005880 comp rx=0 tx=0).stop 2024-04-02T08:52:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.475+0000 7f1179800700 1 -- 172.21.15.89:0/58345138 reap_dead start 2024-04-02T08:52:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.475+0000 7f117b800700 1 -- 172.21.15.89:0/58345138 shutdown_connections 2024-04-02T08:52:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.475+0000 7f117b800700 1 -- 172.21.15.89:0/58345138 >> 172.21.15.89:0/58345138 conn(0x7f117408aff0 msgr2=0x7f11740d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.476+0000 7f117b800700 1 -- 172.21.15.89:0/58345138 shutdown_connections 2024-04-02T08:52:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.476+0000 7f117b800700 1 -- 172.21.15.89:0/58345138 wait complete. 2024-04-02T08:52:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.477+0000 7f117b800700 1 Processor -- start 2024-04-02T08:52:34.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.477+0000 7f117b800700 1 -- start start 2024-04-02T08:52:34.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.478+0000 7f117b800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f115c003f80 0x7f117415a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.478+0000 7f117b800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11740c99b0 0x7f117409f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.478+0000 7f117b800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f117409f6b0 0x7f117409fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.478+0000 7f117b800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11740a0070 con 0x7f117409f6b0 2024-04-02T08:52:34.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.478+0000 7f117b800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11740c89b0 con 0x7f11740c99b0 2024-04-02T08:52:34.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.478+0000 7f117b800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11740c7e30 con 0x7f115c003f80 2024-04-02T08:52:34.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.479+0000 7f1179800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f117409f6b0 0x7f117409fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.479+0000 7f1179800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f117409f6b0 0x7f117409fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37016/0 (socket says 172.21.15.89:37016) 2024-04-02T08:52:34.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.479+0000 7f1179800700 1 -- 172.21.15.89:0/4129948859 learned_addr learned my addr 172.21.15.89:0/4129948859 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:34.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.479+0000 7f1173e00700 1 --2- 172.21.15.89:0/4129948859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11740c99b0 0x7f117409f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.479+0000 7f1178e00700 1 --2- 172.21.15.89:0/4129948859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f115c003f80 0x7f117415a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.479+0000 7f1179800700 1 -- 172.21.15.89:0/4129948859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f115c003f80 msgr2=0x7f117415a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:34.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.479+0000 7f1179800700 1 --2- 172.21.15.89:0/4129948859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f115c003f80 0x7f117415a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.479+0000 7f1179800700 1 -- 172.21.15.89:0/4129948859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11740c99b0 msgr2=0x7f117409f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:34.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.479+0000 7f1179800700 1 --2- 172.21.15.89:0/4129948859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11740c99b0 0x7f117409f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.479+0000 7f1179800700 1 -- 172.21.15.89:0/4129948859 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11740c8110 con 0x7f117409f6b0 2024-04-02T08:52:34.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.479+0000 7f1179800700 1 --2- 172.21.15.89:0/4129948859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f117409f6b0 0x7f117409fb30 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f11740b6980 tx=0x7f116c011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:34.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.479+0000 7f1170c00700 1 -- 172.21.15.89:0/4129948859 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c00e040 con 0x7f117409f6b0 2024-04-02T08:52:34.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.480+0000 7f1170c00700 1 -- 172.21.15.89:0/4129948859 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116c00f040 con 0x7f117409f6b0 2024-04-02T08:52:34.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.480+0000 7f117b800700 1 -- 172.21.15.89:0/4129948859 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11740c83f0 con 0x7f117409f6b0 2024-04-02T08:52:34.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.480+0000 7f1170c00700 1 -- 172.21.15.89:0/4129948859 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c00fc40 con 0x7f117409f6b0 2024-04-02T08:52:34.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.480+0000 7f117b800700 1 -- 172.21.15.89:0/4129948859 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117415a6b0 con 0x7f117409f6b0 2024-04-02T08:52:34.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.481+0000 7f117b800700 1 -- 172.21.15.89:0/4129948859 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1174067720 con 0x7f117409f6b0 2024-04-02T08:52:34.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.482+0000 7f1170c00700 1 -- 172.21.15.89:0/4129948859 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f116c022460 con 0x7f117409f6b0 2024-04-02T08:52:34.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.482+0000 7f1170c00700 1 --2- 172.21.15.89:0/4129948859 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f115406adb0 0x7f115406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:34.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.483+0000 7f1170c00700 1 -- 172.21.15.89:0/4129948859 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5965+0+0 (secure 0 0 0) 0x7f116c089b30 con 0x7f117409f6b0 2024-04-02T08:52:34.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.483+0000 7f1178e00700 1 --2- 172.21.15.89:0/4129948859 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f115406adb0 0x7f115406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:34.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.483+0000 7f1178e00700 1 --2- 172.21.15.89:0/4129948859 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f115406adb0 0x7f115406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1168000c00 tx=0x7f1168007980 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:34.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.487+0000 7f1170c00700 1 -- 172.21.15.89:0/4129948859 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f116c054950 con 0x7f117409f6b0 2024-04-02T08:52:34.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.765+0000 7f117b800700 1 -- 172.21.15.89:0/4129948859 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f11740b5c70 con 0x7f117409f6b0 2024-04-02T08:52:34.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.943+0000 7f1170c00700 1 -- 172.21.15.89:0/4129948859 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2486) v1 ==== 178+0+0 (secure 0 0 0) 0x7f116c057f70 con 0x7f117409f6b0 2024-04-02T08:52:34.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.945+0000 7f117b800700 1 -- 172.21.15.89:0/4129948859 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f115406adb0 msgr2=0x7f115406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:34.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.945+0000 7f117b800700 1 --2- 172.21.15.89:0/4129948859 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f115406adb0 0x7f115406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1168000c00 tx=0x7f1168007980 comp rx=0 tx=0).stop 2024-04-02T08:52:34.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.945+0000 7f117b800700 1 -- 172.21.15.89:0/4129948859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f117409f6b0 msgr2=0x7f117409fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:34.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.945+0000 7f117b800700 1 --2- 172.21.15.89:0/4129948859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f117409f6b0 0x7f117409fb30 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f11740b6980 tx=0x7f116c011ed0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.946+0000 7f117b800700 1 -- 172.21.15.89:0/4129948859 shutdown_connections 2024-04-02T08:52:34.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.946+0000 7f117b800700 1 --2- 172.21.15.89:0/4129948859 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f115406adb0 0x7f115406d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.946+0000 7f117b800700 1 --2- 172.21.15.89:0/4129948859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f117409f6b0 0x7f117409fb30 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.946+0000 7f117b800700 1 --2- 172.21.15.89:0/4129948859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f115c003f80 0x7f117415a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.946+0000 7f117b800700 1 --2- 172.21.15.89:0/4129948859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11740c99b0 0x7f117409f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:34.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.946+0000 7f117b800700 1 -- 172.21.15.89:0/4129948859 >> 172.21.15.89:0/4129948859 conn(0x7f117408aff0 msgr2=0x7f117415b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:34.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.946+0000 7f117b800700 1 -- 172.21.15.89:0/4129948859 shutdown_connections 2024-04-02T08:52:34.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:34.946+0000 7f117b800700 1 -- 172.21.15.89:0/4129948859 wait complete. 2024-04-02T08:52:34.963 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2024-04-02T08:52:34.963 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2024-04-02T08:52:34.963 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-02T08:52:34.963 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-04-02T08:52:35.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.155+0000 7fc902000700 1 Processor -- start 2024-04-02T08:52:35.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.155+0000 7fc902000700 1 -- start start 2024-04-02T08:52:35.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.157+0000 7fc902000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8fc15da50 0x7fc8fc0ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:35.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.157+0000 7fc902000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc8fc0d0cf0 0x7fc8fc0b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:35.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.157+0000 7fc902000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8fc0d24e0 con 0x7fc8fc0cec70 2024-04-02T08:52:35.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.157+0000 7fc902000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8fc0b95d0 con 0x7fc8fc15da50 2024-04-02T08:52:35.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.157+0000 7fc902000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8fc15de30 con 0x7fc8fc0d0cf0 2024-04-02T08:52:35.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.157+0000 7fc8fb400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8fc15da50 0x7fc8fc0ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:35.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.157+0000 7fc8fb400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8fc15da50 0x7fc8fc0ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37022/0 (socket says 172.21.15.89:37022) 2024-04-02T08:52:35.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.157+0000 7fc8fb400700 1 -- 172.21.15.89:0/1415614217 learned_addr learned my addr 172.21.15.89:0/1415614217 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:35.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.157+0000 7fc8fbe00700 1 --2- 172.21.15.89:0/1415614217 >> v2:172.21.15.113:3300/0 conn(0x7fc8fc0d0cf0 0x7fc8fc0b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:35.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.157+0000 7fc8fb400700 1 -- 172.21.15.89:0/1415614217 >> v2:172.21.15.113:3300/0 conn(0x7fc8fc0d0cf0 msgr2=0x7fc8fc0b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:35.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.157+0000 7fc8fb400700 1 --2- 172.21.15.89:0/1415614217 >> v2:172.21.15.113:3300/0 conn(0x7fc8fc0d0cf0 0x7fc8fc0b6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:35.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.157+0000 7fc8fb400700 1 -- 172.21.15.89:0/1415614217 >> v1:172.21.15.113:6789/0 conn(0x7fc8fc0cec70 legacy=0x7fc8fc0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:35.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.157+0000 7fc8fb400700 1 -- 172.21.15.89:0/1415614217 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8fc0d27c0 con 0x7fc8fc15da50 2024-04-02T08:52:35.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.157+0000 7fc8fbe00700 1 --2- 172.21.15.89:0/1415614217 >> v2:172.21.15.113:3300/0 conn(0x7fc8fc0d0cf0 0x7fc8fc0b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:52:35.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.158+0000 7fc8fb400700 1 --2- 172.21.15.89:0/1415614217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc15da50 0x7fc8fc0ce730 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc0d1660 tx=0x7fc8ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e48235e72f3c864d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:35.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.158+0000 7fc8fa000700 1 -- 172.21.15.89:0/1415614217 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec004230 con 0x7fc8fc15da50 2024-04-02T08:52:35.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.158+0000 7fc8fa000700 1 -- 172.21.15.89:0/1415614217 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ec004390 con 0x7fc8fc15da50 2024-04-02T08:52:35.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.158+0000 7fc8fa000700 1 -- 172.21.15.89:0/1415614217 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec0036a0 con 0x7fc8fc15da50 2024-04-02T08:52:35.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.158+0000 7fc902000700 1 -- 172.21.15.89:0/1415614217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc15da50 msgr2=0x7fc8fc0ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:35.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.158+0000 7fc902000700 1 --2- 172.21.15.89:0/1415614217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc15da50 0x7fc8fc0ce730 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc0d1660 tx=0x7fc8ec009bd0 comp rx=0 tx=0).stop 2024-04-02T08:52:35.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.158+0000 7fc902000700 1 -- 172.21.15.89:0/1415614217 shutdown_connections 2024-04-02T08:52:35.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.158+0000 7fc902000700 1 --2- 172.21.15.89:0/1415614217 >> v2:172.21.15.113:3300/0 conn(0x7fc8fc0d0cf0 0x7fc8fc0b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:35.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.158+0000 7fc902000700 1 --2- 172.21.15.89:0/1415614217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc15da50 0x7fc8fc0ce730 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:35.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.158+0000 7fc902000700 1 -- 172.21.15.89:0/1415614217 >> 172.21.15.89:0/1415614217 conn(0x7fc8fc0666a0 msgr2=0x7fc8fc066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:35.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.159+0000 7fc902000700 1 -- 172.21.15.89:0/1415614217 shutdown_connections 2024-04-02T08:52:35.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.159+0000 7fc902000700 1 -- 172.21.15.89:0/1415614217 wait complete. 2024-04-02T08:52:35.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.160+0000 7fc902000700 1 Processor -- start 2024-04-02T08:52:35.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.160+0000 7fc902000700 1 -- start start 2024-04-02T08:52:35.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.161+0000 7fc902000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8fc0cec70 0x7fc8fc160bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:35.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.161+0000 7fc902000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8fc0d0cf0 0x7fc8fc15e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:35.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.161+0000 7fc902000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc15da50 0x7fc8fc15eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:35.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.161+0000 7fc902000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc155bb0 con 0x7fc8fc15da50 2024-04-02T08:52:35.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.161+0000 7fc902000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc15fa10 con 0x7fc8fc0cec70 2024-04-02T08:52:35.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.161+0000 7fc902000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc1555f0 con 0x7fc8fc0d0cf0 2024-04-02T08:52:35.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.161+0000 7fc8fbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc15da50 0x7fc8fc15eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:35.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.162+0000 7fc8fbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc15da50 0x7fc8fc15eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37028/0 (socket says 172.21.15.89:37028) 2024-04-02T08:52:35.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.162+0000 7fc8fbe00700 1 -- 172.21.15.89:0/14007513 learned_addr learned my addr 172.21.15.89:0/14007513 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:35.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.162+0000 7fc8faa00700 1 --2- 172.21.15.89:0/14007513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8fc0d0cf0 0x7fc8fc15e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:35.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.162+0000 7fc8fb400700 1 --2- 172.21.15.89:0/14007513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8fc0cec70 0x7fc8fc160bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:35.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.162+0000 7fc8fbe00700 1 -- 172.21.15.89:0/14007513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8fc0d0cf0 msgr2=0x7fc8fc15e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:35.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.162+0000 7fc8fbe00700 1 --2- 172.21.15.89:0/14007513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8fc0d0cf0 0x7fc8fc15e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:35.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.162+0000 7fc8fbe00700 1 -- 172.21.15.89:0/14007513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8fc0cec70 msgr2=0x7fc8fc160bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:35.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.162+0000 7fc8fbe00700 1 --2- 172.21.15.89:0/14007513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8fc0cec70 0x7fc8fc160bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:35.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.162+0000 7fc8fbe00700 1 -- 172.21.15.89:0/14007513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec009550 con 0x7fc8fc15da50 2024-04-02T08:52:35.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.162+0000 7fc8fbe00700 1 --2- 172.21.15.89:0/14007513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc15da50 0x7fc8fc15eaa0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400ce00 tx=0x7fc8f400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:35.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.162+0000 7fc8f3400700 1 -- 172.21.15.89:0/14007513 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f400ac40 con 0x7fc8fc15da50 2024-04-02T08:52:35.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.162+0000 7fc8f3400700 1 -- 172.21.15.89:0/14007513 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f400adc0 con 0x7fc8fc15da50 2024-04-02T08:52:35.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.162+0000 7fc902000700 1 -- 172.21.15.89:0/14007513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8fc1558d0 con 0x7fc8fc15da50 2024-04-02T08:52:35.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.162+0000 7fc8f3400700 1 -- 172.21.15.89:0/14007513 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4003c60 con 0x7fc8fc15da50 2024-04-02T08:52:35.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.163+0000 7fc902000700 1 -- 172.21.15.89:0/14007513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8fc161130 con 0x7fc8fc15da50 2024-04-02T08:52:35.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.163+0000 7fc902000700 1 -- 172.21.15.89:0/14007513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8fc15fda0 con 0x7fc8fc15da50 2024-04-02T08:52:35.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.164+0000 7fc8f3400700 1 -- 172.21.15.89:0/14007513 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc8f4015440 con 0x7fc8fc15da50 2024-04-02T08:52:35.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.165+0000 7fc8f3400700 1 --2- 172.21.15.89:0/14007513 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc8dc06adb0 0x7fc8dc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:35.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.165+0000 7fc8f3400700 1 -- 172.21.15.89:0/14007513 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5965+0+0 (secure 0 0 0) 0x7fc8f4089840 con 0x7fc8fc15da50 2024-04-02T08:52:35.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.166+0000 7fc8fb400700 1 --2- 172.21.15.89:0/14007513 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc8dc06adb0 0x7fc8dc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:35.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.166+0000 7fc8fb400700 1 --2- 172.21.15.89:0/14007513 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc8dc06adb0 0x7fc8dc06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009130 tx=0x7fc8ec008030 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:35.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.170+0000 7fc8f3400700 1 -- 172.21.15.89:0/14007513 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc8fc15fda0 con 0x7fc8fc15da50 2024-04-02T08:52:35.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.452+0000 7fc902000700 1 -- 172.21.15.89:0/14007513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7fc8fc15fda0 con 0x7fc8fc15da50 2024-04-02T08:52:35.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.945+0000 7fc8f3400700 1 -- 172.21.15.89:0/14007513 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2487) v1 ==== 177+0+0 (secure 0 0 0) 0x7fc8fc15fda0 con 0x7fc8fc15da50 2024-04-02T08:52:35.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.948+0000 7fc902000700 1 -- 172.21.15.89:0/14007513 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc8dc06adb0 msgr2=0x7fc8dc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:35.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.948+0000 7fc902000700 1 --2- 172.21.15.89:0/14007513 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc8dc06adb0 0x7fc8dc06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009130 tx=0x7fc8ec008030 comp rx=0 tx=0).stop 2024-04-02T08:52:35.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.948+0000 7fc902000700 1 -- 172.21.15.89:0/14007513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc15da50 msgr2=0x7fc8fc15eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:35.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.948+0000 7fc902000700 1 --2- 172.21.15.89:0/14007513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc15da50 0x7fc8fc15eaa0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400ce00 tx=0x7fc8f400ce30 comp rx=0 tx=0).stop 2024-04-02T08:52:35.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.948+0000 7fc902000700 1 -- 172.21.15.89:0/14007513 shutdown_connections 2024-04-02T08:52:35.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.948+0000 7fc902000700 1 --2- 172.21.15.89:0/14007513 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc8dc06adb0 0x7fc8dc06d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:35.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.948+0000 7fc902000700 1 --2- 172.21.15.89:0/14007513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8fc0cec70 0x7fc8fc160bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:35.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.948+0000 7fc902000700 1 --2- 172.21.15.89:0/14007513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8fc15da50 0x7fc8fc15eaa0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:35.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.948+0000 7fc902000700 1 --2- 172.21.15.89:0/14007513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8fc0d0cf0 0x7fc8fc15e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:35.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.948+0000 7fc902000700 1 -- 172.21.15.89:0/14007513 >> 172.21.15.89:0/14007513 conn(0x7fc8fc0666a0 msgr2=0x7fc8fc0b8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:35.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.949+0000 7fc902000700 1 -- 172.21.15.89:0/14007513 shutdown_connections 2024-04-02T08:52:35.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:35.949+0000 7fc902000700 1 -- 172.21.15.89:0/14007513 wait complete. 2024-04-02T08:52:35.969 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T08:52:36.015 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:52:36.020 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:52:36.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.175+0000 7f3e56a00700 1 Processor -- start 2024-04-02T08:52:36.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.175+0000 7f3e56a00700 1 -- start start 2024-04-02T08:52:36.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.176+0000 7f3e56a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e500cbb30 0x7f3e500cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:36.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.176+0000 7f3e56a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e500cc450 con 0x7f3e500cd4a0 2024-04-02T08:52:36.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.176+0000 7f3e56a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e500cdae0 con 0x7f3e500cd020 2024-04-02T08:52:36.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.176+0000 7f3e56a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500cddc0 con 0x7f3e500cbb30 2024-04-02T08:52:36.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.176+0000 7f3e54a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3e500cd4a0 0x7f3e5006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44694/0 (socket says 172.21.15.89:44694) 2024-04-02T08:52:36.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.176+0000 7f3e54a00700 1 -- 172.21.15.89:0/4132064037 learned_addr learned my addr 172.21.15.89:0/4132064037 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:36.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.176+0000 7f3e4fe00700 1 --2- 172.21.15.89:0/4132064037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e500cbb30 0x7f3e500cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:36.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324038612 0 0) 0x7f3e500cc450 con 0x7f3e500cd4a0 2024-04-02T08:52:36.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e340035b0 con 0x7f3e500cd4a0 2024-04-02T08:52:36.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565611741 0 0) 0x7f3e500cdae0 con 0x7f3e500cd020 2024-04-02T08:52:36.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e500cc450 con 0x7f3e500cd020 2024-04-02T08:52:36.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1804592298 0 0) 0x7f3e340035b0 con 0x7f3e500cd4a0 2024-04-02T08:52:36.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e500cbb30 msgr2=0x7f3e500cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:36.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 --2- 172.21.15.89:0/4132064037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e500cbb30 0x7f3e500cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:36.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 >> v1:172.21.15.113:6789/0 conn(0x7f3e500cd020 legacy=0x7f3e500b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:36.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e500ce0a0 con 0x7f3e500cd4a0 2024-04-02T08:52:36.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f3e480019f0 con 0x7f3e500cd4a0 2024-04-02T08:52:36.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4fe00700 1 --2- 172.21.15.89:0/4132064037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e500cbb30 0x7f3e500cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:36.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 >> v1:172.21.15.89:6789/0 conn(0x7f3e500cd4a0 legacy=0x7f3e5006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:36.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 --2- 172.21.15.89:0/4132064037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e500cbb30 0x7f3e34003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:36.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 --2- 172.21.15.89:0/4132064037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34004150 0x7f3e34004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:36.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 --2- 172.21.15.89:0/4132064037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e34004d90 0x7f3e34005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:36.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500cddc0 con 0x7f3e34004150 2024-04-02T08:52:36.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3400bb20 con 0x7f3e34004d90 2024-04-02T08:52:36.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3400bce0 con 0x7f3e500cbb30 2024-04-02T08:52:36.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e47e00700 1 --2- 172.21.15.89:0/4132064037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34004150 0x7f3e34004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:36.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e4fe00700 1 --2- 172.21.15.89:0/4132064037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e500cbb30 0x7f3e34003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:36.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.177+0000 7f3e54a00700 1 --2- 172.21.15.89:0/4132064037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e34004d90 0x7f3e34005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:36.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.178+0000 7f3e47e00700 1 -- 172.21.15.89:0/4132064037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e500cbb30 msgr2=0x7f3e34003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:36.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.178+0000 7f3e47e00700 1 --2- 172.21.15.89:0/4132064037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e500cbb30 0x7f3e34003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:36.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.178+0000 7f3e47e00700 1 -- 172.21.15.89:0/4132064037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e34004d90 msgr2=0x7f3e34005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:36.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.178+0000 7f3e47e00700 1 --2- 172.21.15.89:0/4132064037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e34004d90 0x7f3e34005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:36.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.178+0000 7f3e47e00700 1 -- 172.21.15.89:0/4132064037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e400094a0 con 0x7f3e34004150 2024-04-02T08:52:36.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.178+0000 7f3e47e00700 1 --2- 172.21.15.89:0/4132064037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34004150 0x7f3e34004850 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f3e38007fd0 tx=0x7f3e34005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f8dcc7b4ed40a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:36.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.178+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e38001430 con 0x7f3e34004150 2024-04-02T08:52:36.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.178+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e380052d0 con 0x7f3e34004150 2024-04-02T08:52:36.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.178+0000 7f3e4f400700 1 -- 172.21.15.89:0/4132064037 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e38001d40 con 0x7f3e34004150 2024-04-02T08:52:36.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.179+0000 7f3e56a00700 1 -- 172.21.15.89:0/4132064037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34004150 msgr2=0x7f3e34004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:36.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.179+0000 7f3e56a00700 1 --2- 172.21.15.89:0/4132064037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34004150 0x7f3e34004850 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f3e38007fd0 tx=0x7f3e34005880 comp rx=0 tx=0).stop 2024-04-02T08:52:36.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.179+0000 7f3e54a00700 1 -- 172.21.15.89:0/4132064037 reap_dead start 2024-04-02T08:52:36.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.179+0000 7f3e56a00700 1 -- 172.21.15.89:0/4132064037 shutdown_connections 2024-04-02T08:52:36.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.179+0000 7f3e56a00700 1 -- 172.21.15.89:0/4132064037 >> 172.21.15.89:0/4132064037 conn(0x7f3e500b0d70 msgr2=0x7f3e500b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:36.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.179+0000 7f3e56a00700 1 -- 172.21.15.89:0/4132064037 shutdown_connections 2024-04-02T08:52:36.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.179+0000 7f3e56a00700 1 -- 172.21.15.89:0/4132064037 wait complete. 2024-04-02T08:52:36.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.181+0000 7f3e56a00700 1 Processor -- start 2024-04-02T08:52:36.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.181+0000 7f3e56a00700 1 -- start start 2024-04-02T08:52:36.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e56a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34004150 0x7f3e5014fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:36.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e56a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e50150440 0x7f3e5015dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:36.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e56a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e5015e500 0x7f3e50156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:36.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e56a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e50156730 con 0x7f3e34004150 2024-04-02T08:52:36.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e56a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500b0990 con 0x7f3e50150440 2024-04-02T08:52:36.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e56a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500b03d0 con 0x7f3e5015e500 2024-04-02T08:52:36.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e4fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34004150 0x7f3e5014fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:36.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e4fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34004150 0x7f3e5014fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37040/0 (socket says 172.21.15.89:37040) 2024-04-02T08:52:36.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e4fe00700 1 -- 172.21.15.89:0/3594490059 learned_addr learned my addr 172.21.15.89:0/3594490059 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:36.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e54a00700 1 --2- 172.21.15.89:0/3594490059 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e5015e500 0x7f3e50156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:36.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e47e00700 1 --2- 172.21.15.89:0/3594490059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e50150440 0x7f3e5015dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:36.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e4fe00700 1 -- 172.21.15.89:0/3594490059 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e5015e500 msgr2=0x7f3e50156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:36.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e4fe00700 1 --2- 172.21.15.89:0/3594490059 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e5015e500 0x7f3e50156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:36.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e4fe00700 1 -- 172.21.15.89:0/3594490059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e50150440 msgr2=0x7f3e5015dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:36.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e4fe00700 1 --2- 172.21.15.89:0/3594490059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e50150440 0x7f3e5015dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:36.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.182+0000 7f3e4fe00700 1 -- 172.21.15.89:0/3594490059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e500b06b0 con 0x7f3e34004150 2024-04-02T08:52:36.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.183+0000 7f3e4fe00700 1 --2- 172.21.15.89:0/3594490059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34004150 0x7f3e5014fa80 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f3e40011cd0 tx=0x7f3e4000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:36.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.183+0000 7f3e4cc00700 1 -- 172.21.15.89:0/3594490059 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e40011e70 con 0x7f3e34004150 2024-04-02T08:52:36.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.183+0000 7f3e4cc00700 1 -- 172.21.15.89:0/3594490059 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4000fd40 con 0x7f3e34004150 2024-04-02T08:52:36.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.183+0000 7f3e4cc00700 1 -- 172.21.15.89:0/3594490059 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e40014680 con 0x7f3e34004150 2024-04-02T08:52:36.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.183+0000 7f3e56a00700 1 -- 172.21.15.89:0/3594490059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e50156070 con 0x7f3e34004150 2024-04-02T08:52:36.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.183+0000 7f3e56a00700 1 -- 172.21.15.89:0/3594490059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5015e990 con 0x7f3e34004150 2024-04-02T08:52:36.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.184+0000 7f3e56a00700 1 -- 172.21.15.89:0/3594490059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e50156870 con 0x7f3e34004150 2024-04-02T08:52:36.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.185+0000 7f3e4cc00700 1 -- 172.21.15.89:0/3594490059 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3e400147e0 con 0x7f3e34004150 2024-04-02T08:52:36.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.186+0000 7f3e4cc00700 1 --2- 172.21.15.89:0/3594490059 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e2c06adb0 0x7f3e2c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:36.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.186+0000 7f3e4cc00700 1 -- 172.21.15.89:0/3594490059 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5965+0+0 (secure 0 0 0) 0x7f3e40089ec0 con 0x7f3e34004150 2024-04-02T08:52:36.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.186+0000 7f3e47e00700 1 --2- 172.21.15.89:0/3594490059 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e2c06adb0 0x7f3e2c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:36.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.186+0000 7f3e47e00700 1 --2- 172.21.15.89:0/3594490059 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e2c06adb0 0x7f3e2c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3e38015040 tx=0x7f3e3800a000 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:36.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.190+0000 7f3e4cc00700 1 -- 172.21.15.89:0/3594490059 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3e50156870 con 0x7f3e34004150 2024-04-02T08:52:36.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.391+0000 7f3e56a00700 1 -- 172.21.15.89:0/3594490059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3e50156870 con 0x7f3e34004150 2024-04-02T08:52:36.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.395+0000 7f3e4cc00700 1 -- 172.21.15.89:0/3594490059 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3e40054df0 con 0x7f3e34004150 2024-04-02T08:52:36.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.397+0000 7f3e56a00700 1 -- 172.21.15.89:0/3594490059 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e2c06adb0 msgr2=0x7f3e2c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:36.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.397+0000 7f3e56a00700 1 --2- 172.21.15.89:0/3594490059 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e2c06adb0 0x7f3e2c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3e38015040 tx=0x7f3e3800a000 comp rx=0 tx=0).stop 2024-04-02T08:52:36.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.397+0000 7f3e56a00700 1 -- 172.21.15.89:0/3594490059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34004150 msgr2=0x7f3e5014fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:36.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.397+0000 7f3e56a00700 1 --2- 172.21.15.89:0/3594490059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34004150 0x7f3e5014fa80 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f3e40011cd0 tx=0x7f3e4000f520 comp rx=0 tx=0).stop 2024-04-02T08:52:36.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.398+0000 7f3e56a00700 1 -- 172.21.15.89:0/3594490059 shutdown_connections 2024-04-02T08:52:36.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.398+0000 7f3e56a00700 1 --2- 172.21.15.89:0/3594490059 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3e2c06adb0 0x7f3e2c06d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:36.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.398+0000 7f3e56a00700 1 --2- 172.21.15.89:0/3594490059 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e5015e500 0x7f3e50156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:36.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.398+0000 7f3e56a00700 1 --2- 172.21.15.89:0/3594490059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34004150 0x7f3e5014fa80 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:36.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.398+0000 7f3e56a00700 1 --2- 172.21.15.89:0/3594490059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e50150440 0x7f3e5015dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:36.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.398+0000 7f3e56a00700 1 -- 172.21.15.89:0/3594490059 >> 172.21.15.89:0/3594490059 conn(0x7f3e500b0d70 msgr2=0x7f3e5015a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:36.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.398+0000 7f3e56a00700 1 -- 172.21.15.89:0/3594490059 shutdown_connections 2024-04-02T08:52:36.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.398+0000 7f3e56a00700 1 -- 172.21.15.89:0/3594490059 wait complete. 2024-04-02T08:52:36.415 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T08:52:36.415 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T08:52:36.506 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T08:52:36.596 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T08:52:36.597 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T08:52:36.597 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:52:36.641 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T08:52:36.642 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T08:52:36.642 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:52:36.687 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T08:52:36.687 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T08:52:36.687 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:52:36.731 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T08:52:36.732 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T08:52:36.732 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:52:36.776 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T08:52:36.776 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T08:52:36.776 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T08:52:36.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.969+0000 7f37e2400700 1 Processor -- start 2024-04-02T08:52:36.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.970+0000 7f37e2400700 1 -- start start 2024-04-02T08:52:36.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.971+0000 7f37e2400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f37dc1661e0 0x7f37dc1685d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:36.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.971+0000 7f37e2400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37dc0ae7b0 con 0x7f37dc0acda0 2024-04-02T08:52:36.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.971+0000 7f37e2400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37dc0aea90 con 0x7f37dc0b5160 2024-04-02T08:52:36.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.971+0000 7f37e2400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37dc0b55e0 con 0x7f37dc1661e0 2024-04-02T08:52:36.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.971+0000 7f37dbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f37dc1661e0 0x7f37dc1685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:36.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.971+0000 7f37dbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f37dc1661e0 0x7f37dc1685d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37044/0 (socket says 172.21.15.89:37044) 2024-04-02T08:52:36.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.971+0000 7f37dbe00700 1 -- 172.21.15.89:0/2023779288 learned_addr learned my addr 172.21.15.89:0/2023779288 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:36.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.971+0000 7f37db400700 1 --1- 172.21.15.89:0/2023779288 >> v1:172.21.15.89:6789/0 conn(0x7f37dc0acda0 0x7f37dc15e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44698/0 (socket says 172.21.15.89:44698) 2024-04-02T08:52:36.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.972+0000 7f37dbe00700 1 -- 172.21.15.89:0/2023779288 >> v1:172.21.15.113:6789/0 conn(0x7f37dc0b5160 legacy=0x7f37dc0ad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:36.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.972+0000 7f37dbe00700 1 -- 172.21.15.89:0/2023779288 >> v1:172.21.15.89:6789/0 conn(0x7f37dc0acda0 legacy=0x7f37dc15e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:36.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.972+0000 7f37dbe00700 1 -- 172.21.15.89:0/2023779288 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc0aed70 con 0x7f37dc1661e0 2024-04-02T08:52:36.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.972+0000 7f37dbe00700 1 --2- 172.21.15.89:0/2023779288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37dc1661e0 0x7f37dc1685d0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f37d4009e80 tx=0x7f37d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a8d13dbddfec29 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:36.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.972+0000 7f37da000700 1 -- 172.21.15.89:0/2023779288 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d400e070 con 0x7f37dc1661e0 2024-04-02T08:52:36.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.972+0000 7f37da000700 1 -- 172.21.15.89:0/2023779288 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d400b980 con 0x7f37dc1661e0 2024-04-02T08:52:36.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.972+0000 7f37da000700 1 -- 172.21.15.89:0/2023779288 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d400bd30 con 0x7f37dc1661e0 2024-04-02T08:52:36.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.973+0000 7f37e2400700 1 -- 172.21.15.89:0/2023779288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37dc1661e0 msgr2=0x7f37dc1685d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:36.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.973+0000 7f37e2400700 1 --2- 172.21.15.89:0/2023779288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37dc1661e0 0x7f37dc1685d0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f37d4009e80 tx=0x7f37d400b330 comp rx=0 tx=0).stop 2024-04-02T08:52:36.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.973+0000 7f37e2400700 1 -- 172.21.15.89:0/2023779288 shutdown_connections 2024-04-02T08:52:36.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.973+0000 7f37e2400700 1 --2- 172.21.15.89:0/2023779288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37dc1661e0 0x7f37dc1685d0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:36.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.973+0000 7f37e2400700 1 -- 172.21.15.89:0/2023779288 >> 172.21.15.89:0/2023779288 conn(0x7f37dc072170 msgr2=0x7f37dc06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:36.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.973+0000 7f37e2400700 1 -- 172.21.15.89:0/2023779288 shutdown_connections 2024-04-02T08:52:36.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.973+0000 7f37e2400700 1 -- 172.21.15.89:0/2023779288 wait complete. 2024-04-02T08:52:36.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.974+0000 7f37e2400700 1 Processor -- start 2024-04-02T08:52:36.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.975+0000 7f37e2400700 1 -- start start 2024-04-02T08:52:36.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.975+0000 7f37e2400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37dc0acda0 0x7f37dc0ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:36.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.975+0000 7f37e2400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f37dc0b5160 0x7f37dc0ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:36.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37e2400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37dc1661e0 0x7f37dc1511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:36.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37e2400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37dc0cd840 con 0x7f37dc1661e0 2024-04-02T08:52:36.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37e2400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37dc1581e0 con 0x7f37dc0acda0 2024-04-02T08:52:36.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37e2400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37dc0cc410 con 0x7f37dc0b5160 2024-04-02T08:52:36.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37dbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37dc1661e0 0x7f37dc1511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:36.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37dbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37dc1661e0 0x7f37dc1511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37050/0 (socket says 172.21.15.89:37050) 2024-04-02T08:52:36.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37dbe00700 1 -- 172.21.15.89:0/1394181608 learned_addr learned my addr 172.21.15.89:0/1394181608 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:36.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37db400700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37dc0acda0 0x7f37dc0ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:36.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37daa00700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f37dc0b5160 0x7f37dc0ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:36.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37dbe00700 1 -- 172.21.15.89:0/1394181608 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f37dc0b5160 msgr2=0x7f37dc0ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:36.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37dbe00700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f37dc0b5160 0x7f37dc0ce9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:36.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37dbe00700 1 -- 172.21.15.89:0/1394181608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37dc0acda0 msgr2=0x7f37dc0ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:36.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37dbe00700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37dc0acda0 0x7f37dc0ccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:36.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37dbe00700 1 -- 172.21.15.89:0/1394181608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d4009b30 con 0x7f37dc1661e0 2024-04-02T08:52:36.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37daa00700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f37dc0b5160 0x7f37dc0ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:36.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.976+0000 7f37dbe00700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37dc1661e0 0x7f37dc1511c0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f37d4012000 tx=0x7f37d4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:36.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.977+0000 7f37d3400700 1 -- 172.21.15.89:0/1394181608 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d40048b0 con 0x7f37dc1661e0 2024-04-02T08:52:36.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.977+0000 7f37d3400700 1 -- 172.21.15.89:0/1394181608 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d40099b0 con 0x7f37dc1661e0 2024-04-02T08:52:36.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.977+0000 7f37e2400700 1 -- 172.21.15.89:0/1394181608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37dc0cc6f0 con 0x7f37dc1661e0 2024-04-02T08:52:36.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.977+0000 7f37d3400700 1 -- 172.21.15.89:0/1394181608 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d401b670 con 0x7f37dc1661e0 2024-04-02T08:52:36.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.977+0000 7f37e2400700 1 -- 172.21.15.89:0/1394181608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37dc15f010 con 0x7f37dc1661e0 2024-04-02T08:52:36.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.978+0000 7f37e2400700 1 -- 172.21.15.89:0/1394181608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37dc0cc930 con 0x7f37dc1661e0 2024-04-02T08:52:36.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.979+0000 7f37d3400700 1 -- 172.21.15.89:0/1394181608 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f37d401b7d0 con 0x7f37dc1661e0 2024-04-02T08:52:36.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.979+0000 7f37d3400700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f37bc06adb0 0x7f37bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:36.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.980+0000 7f37d3400700 1 -- 172.21.15.89:0/1394181608 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5965+0+0 (secure 0 0 0) 0x7f37d408ab70 con 0x7f37dc1661e0 2024-04-02T08:52:36.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.980+0000 7f37db400700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f37bc06adb0 0x7f37bc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:36.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.980+0000 7f37db400700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f37bc06adb0 0x7f37bc06d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f37cc000f60 tx=0x7f37cc007ed0 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:36.984+0000 7f37d3400700 1 -- 172.21.15.89:0/1394181608 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f37dc0cc930 con 0x7f37dc1661e0 2024-04-02T08:52:37.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.229+0000 7f37e2400700 1 -- 172.21.15.89:0/1394181608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37dc067720 con 0x7f37dc1661e0 2024-04-02T08:52:37.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.229+0000 7f37d3400700 1 -- 172.21.15.89:0/1394181608 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f37d401fde0 con 0x7f37dc1661e0 2024-04-02T08:52:37.229 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:37.229 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":75,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7445,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/4112144997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":4112144997},{"type":"v1","addr":"172.21.15.113:6834","nonce":4112144997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":8416,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2047642384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2047642384},{"type":"v1","addr":"172.21.15.113:6835","nonce":2047642384}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":9030,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2201184833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2201184833},{"type":"v1","addr":"172.21.15.89:6835","nonce":2201184833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":69},{"gid":9444,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3212932914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3212932914},{"type":"v1","addr":"172.21.15.89:6837","nonce":3212932914}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[]} 2024-04-02T08:52:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.232+0000 7f37e2400700 1 -- 172.21.15.89:0/1394181608 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f37bc06adb0 msgr2=0x7f37bc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.232+0000 7f37e2400700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f37bc06adb0 0x7f37bc06d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f37cc000f60 tx=0x7f37cc007ed0 comp rx=0 tx=0).stop 2024-04-02T08:52:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.232+0000 7f37e2400700 1 -- 172.21.15.89:0/1394181608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37dc1661e0 msgr2=0x7f37dc1511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.232+0000 7f37e2400700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37dc1661e0 0x7f37dc1511c0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f37d4012000 tx=0x7f37d4004ad0 comp rx=0 tx=0).stop 2024-04-02T08:52:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.232+0000 7f37e2400700 1 -- 172.21.15.89:0/1394181608 shutdown_connections 2024-04-02T08:52:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.232+0000 7f37e2400700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f37bc06adb0 0x7f37bc06d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.232+0000 7f37e2400700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37dc0acda0 0x7f37dc0ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.232+0000 7f37e2400700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37dc1661e0 0x7f37dc1511c0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.232+0000 7f37e2400700 1 --2- 172.21.15.89:0/1394181608 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f37dc0b5160 0x7f37dc0ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.232+0000 7f37e2400700 1 -- 172.21.15.89:0/1394181608 >> 172.21.15.89:0/1394181608 conn(0x7f37dc072170 msgr2=0x7f37dc06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.232+0000 7f37e2400700 1 -- 172.21.15.89:0/1394181608 shutdown_connections 2024-04-02T08:52:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.232+0000 7f37e2400700 1 -- 172.21.15.89:0/1394181608 wait complete. 2024-04-02T08:52:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 75 2024-04-02T08:52:37.249 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T08:52:37.249 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T08:52:37.249 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T08:52:37.249 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T08:52:37.249 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T08:52:37.249 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T08:52:37.250 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T08:52:37.250 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T08:52:37.250 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T08:52:37.250 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T08:52:37.250 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T08:52:37.250 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T08:52:37.250 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T08:52:37.250 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T08:52:37.251 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T08:52:37.251 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T08:52:37.263 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T08:52:37.263 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T08:52:37.264 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T08:52:37.265 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T08:52:37.267 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T08:52:37.267 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T08:52:37.268 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T08:52:37.268 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T08:52:37.313 INFO:tasks.ceph.mds.c:Started 2024-04-02T08:52:37.314 DEBUG:teuthology.parallel:result is None 2024-04-02T08:52:37.316 INFO:tasks.ceph.mds.d:Started 2024-04-02T08:52:37.316 DEBUG:teuthology.parallel:result is None 2024-04-02T08:52:37.318 INFO:tasks.ceph.mds.a:Started 2024-04-02T08:52:37.318 DEBUG:teuthology.parallel:result is None 2024-04-02T08:52:37.319 INFO:tasks.ceph.mds.b:Started 2024-04-02T08:52:37.319 DEBUG:teuthology.parallel:result is None 2024-04-02T08:52:37.319 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T08:52:37.453 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T08:52:37.459 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T08:52:37.459 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T08:52:37.466 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T08:52:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.527+0000 7f2312e00700 1 Processor -- start 2024-04-02T08:52:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.527+0000 7f2312e00700 1 -- start start 2024-04-02T08:52:37.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.528+0000 7f2312e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f230c0cbb30 0x7f230c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:37.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.528+0000 7f2312e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f230c0cd020 0x7f230c0cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:37.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.528+0000 7f2312e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230c0cc450 con 0x7f230c0b2070 2024-04-02T08:52:37.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.528+0000 7f2312e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f230c0cdae0 con 0x7f230c0cd020 2024-04-02T08:52:37.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.528+0000 7f2312e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230c0cddc0 con 0x7f230c0cbb30 2024-04-02T08:52:37.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.528+0000 7f2303e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f230c0cd020 0x7f230c0cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:37.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.528+0000 7f2310e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f230c0b2070 0x7f230c06e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44714/0 (socket says 172.21.15.89:44714) 2024-04-02T08:52:37.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.528+0000 7f2310e00700 1 -- 172.21.15.89:0/3921299973 learned_addr learned my addr 172.21.15.89:0/3921299973 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:37.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.528+0000 7f2303e00700 1 --2- 172.21.15.89:0/3921299973 >> v2:172.21.15.89:3300/0 conn(0x7f230c0cd020 0x7f230c0cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37074/0 (socket says 172.21.15.89:37074) 2024-04-02T08:52:37.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.528+0000 7f230be00700 1 --2- 172.21.15.89:0/3921299973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f230c0cbb30 0x7f230c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:37.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.529+0000 7f2303e00700 1 -- 172.21.15.89:0/3921299973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f230c0cbb30 msgr2=0x7f230c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:37.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.529+0000 7f2303e00700 1 --2- 172.21.15.89:0/3921299973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f230c0cbb30 0x7f230c0cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:37.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.529+0000 7f2303e00700 1 -- 172.21.15.89:0/3921299973 >> v1:172.21.15.89:6789/0 conn(0x7f230c0b2070 legacy=0x7f230c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:37.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.529+0000 7f2303e00700 1 -- 172.21.15.89:0/3921299973 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c0ce0a0 con 0x7f230c0cd020 2024-04-02T08:52:37.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.529+0000 7f230be00700 1 --2- 172.21.15.89:0/3921299973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f230c0cbb30 0x7f230c0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:37.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.529+0000 7f230b400700 1 -- 172.21.15.89:0/3921299973 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468509069 0 0) 0x7f230c0cc450 con 0x7f230c0b2070 2024-04-02T08:52:37.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.529+0000 7f2303e00700 1 --2- 172.21.15.89:0/3921299973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f230c0cd020 0x7f230c0cd4a0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009840 tx=0x7f22f4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8cbd2f5b815f83 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:37.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.529+0000 7f230b400700 1 -- 172.21.15.89:0/3921299973 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4011830 con 0x7f230c0cd020 2024-04-02T08:52:37.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.529+0000 7f230b400700 1 -- 172.21.15.89:0/3921299973 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f4011990 con 0x7f230c0cd020 2024-04-02T08:52:37.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.529+0000 7f230b400700 1 -- 172.21.15.89:0/3921299973 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4011d20 con 0x7f230c0cd020 2024-04-02T08:52:37.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.530+0000 7f2312e00700 1 -- 172.21.15.89:0/3921299973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f230c0cd020 msgr2=0x7f230c0cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:37.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.530+0000 7f2312e00700 1 --2- 172.21.15.89:0/3921299973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f230c0cd020 0x7f230c0cd4a0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009840 tx=0x7f22f4009b50 comp rx=0 tx=0).stop 2024-04-02T08:52:37.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.530+0000 7f2312e00700 1 -- 172.21.15.89:0/3921299973 shutdown_connections 2024-04-02T08:52:37.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.530+0000 7f2312e00700 1 --2- 172.21.15.89:0/3921299973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f230c0cbb30 0x7f230c0cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:37.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.530+0000 7f2312e00700 1 --2- 172.21.15.89:0/3921299973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f230c0cd020 0x7f230c0cd4a0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:37.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.530+0000 7f2312e00700 1 -- 172.21.15.89:0/3921299973 >> 172.21.15.89:0/3921299973 conn(0x7f230c0b0d70 msgr2=0x7f230c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:37.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.530+0000 7f2312e00700 1 -- 172.21.15.89:0/3921299973 shutdown_connections 2024-04-02T08:52:37.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.530+0000 7f2312e00700 1 -- 172.21.15.89:0/3921299973 wait complete. 2024-04-02T08:52:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.532+0000 7f2312e00700 1 Processor -- start 2024-04-02T08:52:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.532+0000 7f2312e00700 1 -- start start 2024-04-02T08:52:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f2312e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f230c0b2070 0x7f230c14dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f2312e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f230c0cbb30 0x7f230c15dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f2312e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f230c0cd020 0x7f230c15e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f2312e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c0c8d00 con 0x7f230c0b2070 2024-04-02T08:52:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f2312e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c0c83f0 con 0x7f230c0cd020 2024-04-02T08:52:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f2312e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230c071860 con 0x7f230c0cbb30 2024-04-02T08:52:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f230be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f230c0b2070 0x7f230c14dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f230be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f230c0b2070 0x7f230c14dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37084/0 (socket says 172.21.15.89:37084) 2024-04-02T08:52:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f230be00700 1 -- 172.21.15.89:0/615075636 learned_addr learned my addr 172.21.15.89:0/615075636 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f2303e00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f230c0cbb30 0x7f230c15dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f2310e00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f230c0cd020 0x7f230c15e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f230be00700 1 -- 172.21.15.89:0/615075636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f230c0cbb30 msgr2=0x7f230c15dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f230be00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f230c0cbb30 0x7f230c15dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f230be00700 1 -- 172.21.15.89:0/615075636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f230c0cd020 msgr2=0x7f230c15e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f230be00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f230c0cd020 0x7f230c15e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f230be00700 1 -- 172.21.15.89:0/615075636 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f40094a0 con 0x7f230c0b2070 2024-04-02T08:52:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f2310e00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f230c0cd020 0x7f230c15e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.533+0000 7f230be00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f230c0b2070 0x7f230c14dff0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00e760 tx=0x7f22fc00ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.534+0000 7f2308c00700 1 -- 172.21.15.89:0/615075636 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc013610 con 0x7f230c0b2070 2024-04-02T08:52:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.534+0000 7f2308c00700 1 -- 172.21.15.89:0/615075636 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22fc013770 con 0x7f230c0b2070 2024-04-02T08:52:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.534+0000 7f2312e00700 1 -- 172.21.15.89:0/615075636 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230c0c8640 con 0x7f230c0b2070 2024-04-02T08:52:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.534+0000 7f2308c00700 1 -- 172.21.15.89:0/615075636 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc013b20 con 0x7f230c0b2070 2024-04-02T08:52:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.534+0000 7f2312e00700 1 -- 172.21.15.89:0/615075636 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230c0cabd0 con 0x7f230c0b2070 2024-04-02T08:52:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.535+0000 7f2312e00700 1 -- 172.21.15.89:0/615075636 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f230c067720 con 0x7f230c0b2070 2024-04-02T08:52:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.536+0000 7f2308c00700 1 -- 172.21.15.89:0/615075636 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f22fc013c80 con 0x7f230c0b2070 2024-04-02T08:52:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.537+0000 7f2308c00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22e806adb0 0x7f22e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.537+0000 7f2308c00700 1 -- 172.21.15.89:0/615075636 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5965+0+0 (secure 0 0 0) 0x7f22fc0891c0 con 0x7f230c0b2070 2024-04-02T08:52:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.537+0000 7f2303e00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22e806adb0 0x7f22e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.537+0000 7f2303e00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22e806adb0 0x7f22e806d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f22f40090b0 tx=0x7f22f401c040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:37.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.541+0000 7f2308c00700 1 -- 172.21.15.89:0/615075636 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f22fc054070 con 0x7f230c0b2070 2024-04-02T08:52:37.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:37.758+0000 7f2312e00700 1 -- 172.21.15.89:0/615075636 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f230c15bda0 con 0x7f230c0b2070 2024-04-02T08:52:38.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.506+0000 7f2308c00700 1 -- 172.21.15.89:0/615075636 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v168) v1 ==== 97+0+0 (secure 0 0 0) 0x7f22fc057690 con 0x7f230c0b2070 2024-04-02T08:52:38.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.509+0000 7f2312e00700 1 -- 172.21.15.89:0/615075636 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22e806adb0 msgr2=0x7f22e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:38.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.509+0000 7f2312e00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22e806adb0 0x7f22e806d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f22f40090b0 tx=0x7f22f401c040 comp rx=0 tx=0).stop 2024-04-02T08:52:38.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.509+0000 7f2312e00700 1 -- 172.21.15.89:0/615075636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f230c0b2070 msgr2=0x7f230c14dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:38.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.509+0000 7f2312e00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f230c0b2070 0x7f230c14dff0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00e760 tx=0x7f22fc00ea70 comp rx=0 tx=0).stop 2024-04-02T08:52:38.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.509+0000 7f2312e00700 1 -- 172.21.15.89:0/615075636 shutdown_connections 2024-04-02T08:52:38.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.509+0000 7f2312e00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f22e806adb0 0x7f22e806d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:38.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.509+0000 7f2312e00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f230c0cd020 0x7f230c15e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:38.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.509+0000 7f2312e00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f230c0b2070 0x7f230c14dff0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:38.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.509+0000 7f2312e00700 1 --2- 172.21.15.89:0/615075636 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f230c0cbb30 0x7f230c15dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:38.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.509+0000 7f2312e00700 1 -- 172.21.15.89:0/615075636 >> 172.21.15.89:0/615075636 conn(0x7f230c0b0d70 msgr2=0x7f230c0aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:38.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.509+0000 7f2312e00700 1 -- 172.21.15.89:0/615075636 shutdown_connections 2024-04-02T08:52:38.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.509+0000 7f2312e00700 1 -- 172.21.15.89:0/615075636 wait complete. 2024-04-02T08:52:38.509 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T08:52:38.530 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T08:52:38.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.727+0000 7f3841a00700 1 Processor -- start 2024-04-02T08:52:38.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.727+0000 7f3841a00700 1 -- start start 2024-04-02T08:52:38.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.728+0000 7f3841a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f383c0ceec0 0x7f383c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:38.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.729+0000 7f3841a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f383c161800 0x7f383c0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:38.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.729+0000 7f3841a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c0d30d0 con 0x7f383c15d640 2024-04-02T08:52:38.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.729+0000 7f3841a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383c161c90 con 0x7f383c0ceec0 2024-04-02T08:52:38.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.729+0000 7f3841a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0b6840 con 0x7f383c161800 2024-04-02T08:52:38.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.729+0000 7f383a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f383c0ceec0 0x7f383c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:38.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.729+0000 7f383a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f383c0ceec0 0x7f383c1612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37086/0 (socket says 172.21.15.89:37086) 2024-04-02T08:52:38.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.729+0000 7f383a000700 1 -- 172.21.15.89:0/2405119780 learned_addr learned my addr 172.21.15.89:0/2405119780 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:38.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.729+0000 7f383b400700 1 --2- 172.21.15.89:0/2405119780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f383c161800 0x7f383c0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:38.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.729+0000 7f383a000700 1 -- 172.21.15.89:0/2405119780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f383c161800 msgr2=0x7f383c0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:38.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.729+0000 7f383a000700 1 --2- 172.21.15.89:0/2405119780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f383c161800 0x7f383c0b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:38.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.729+0000 7f383a000700 1 -- 172.21.15.89:0/2405119780 >> v1:172.21.15.113:6789/0 conn(0x7f383c15d640 legacy=0x7f383c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:38.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.729+0000 7f383a000700 1 -- 172.21.15.89:0/2405119780 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c0d2b30 con 0x7f383c0ceec0 2024-04-02T08:52:38.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.729+0000 7f383b400700 1 --2- 172.21.15.89:0/2405119780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f383c161800 0x7f383c0b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:38.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.730+0000 7f383a000700 1 --2- 172.21.15.89:0/2405119780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f383c0ceec0 0x7f383c1612c0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f38280098f0 tx=0x7f3828009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce1315d955cc50b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:38.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.730+0000 7f3839600700 1 -- 172.21.15.89:0/2405119780 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382800b8a0 con 0x7f383c0ceec0 2024-04-02T08:52:38.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.730+0000 7f3839600700 1 -- 172.21.15.89:0/2405119780 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382800ba00 con 0x7f383c0ceec0 2024-04-02T08:52:38.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.730+0000 7f3839600700 1 -- 172.21.15.89:0/2405119780 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382800bd90 con 0x7f383c0ceec0 2024-04-02T08:52:38.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.730+0000 7f3841a00700 1 -- 172.21.15.89:0/2405119780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f383c0ceec0 msgr2=0x7f383c1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:38.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.730+0000 7f3841a00700 1 --2- 172.21.15.89:0/2405119780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f383c0ceec0 0x7f383c1612c0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f38280098f0 tx=0x7f3828009c00 comp rx=0 tx=0).stop 2024-04-02T08:52:38.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.730+0000 7f3841a00700 1 -- 172.21.15.89:0/2405119780 shutdown_connections 2024-04-02T08:52:38.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.731+0000 7f3841a00700 1 --2- 172.21.15.89:0/2405119780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f383c161800 0x7f383c0b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:38.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.731+0000 7f3841a00700 1 --2- 172.21.15.89:0/2405119780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f383c0ceec0 0x7f383c1612c0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:38.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.731+0000 7f3841a00700 1 -- 172.21.15.89:0/2405119780 >> 172.21.15.89:0/2405119780 conn(0x7f383c08aff0 msgr2=0x7f383c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:38.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.731+0000 7f3841a00700 1 -- 172.21.15.89:0/2405119780 shutdown_connections 2024-04-02T08:52:38.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.731+0000 7f3841a00700 1 -- 172.21.15.89:0/2405119780 wait complete. 2024-04-02T08:52:38.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.732+0000 7f3841a00700 1 Processor -- start 2024-04-02T08:52:38.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.732+0000 7f3841a00700 1 -- start start 2024-04-02T08:52:38.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.733+0000 7f3841a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f383c0ceec0 0x7f383c14b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:38.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.733+0000 7f3841a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f383c15d640 0x7f383c0ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:38.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.733+0000 7f3841a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f383c161800 0x7f383c0cab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:38.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.733+0000 7f3841a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c14baf0 con 0x7f383c161800 2024-04-02T08:52:38.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.733+0000 7f3841a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c14f260 con 0x7f383c0ceec0 2024-04-02T08:52:38.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.733+0000 7f3841a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c09e080 con 0x7f383c15d640 2024-04-02T08:52:38.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.734+0000 7f383b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f383c161800 0x7f383c0cab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:38.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.734+0000 7f383b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f383c161800 0x7f383c0cab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37090/0 (socket says 172.21.15.89:37090) 2024-04-02T08:52:38.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.734+0000 7f383a000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f383c15d640 0x7f383c0ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:38.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.734+0000 7f383aa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f383c0ceec0 0x7f383c14b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:38.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.734+0000 7f383b400700 1 -- 172.21.15.89:0/2185561691 learned_addr learned my addr 172.21.15.89:0/2185561691 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:38.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.734+0000 7f383b400700 1 -- 172.21.15.89:0/2185561691 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f383c15d640 msgr2=0x7f383c0ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:38.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.734+0000 7f383b400700 1 --2- 172.21.15.89:0/2185561691 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f383c15d640 0x7f383c0ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:38.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.734+0000 7f383b400700 1 -- 172.21.15.89:0/2185561691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f383c0ceec0 msgr2=0x7f383c14b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:38.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.734+0000 7f383b400700 1 --2- 172.21.15.89:0/2185561691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f383c0ceec0 0x7f383c14b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:38.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.734+0000 7f383b400700 1 -- 172.21.15.89:0/2185561691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3828009550 con 0x7f383c161800 2024-04-02T08:52:38.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.734+0000 7f383b400700 1 --2- 172.21.15.89:0/2185561691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f383c161800 0x7f383c0cab60 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f3834009fd0 tx=0x7f383400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:38.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.735+0000 7f3832a00700 1 -- 172.21.15.89:0/2185561691 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383400dde0 con 0x7f383c161800 2024-04-02T08:52:38.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.735+0000 7f3832a00700 1 -- 172.21.15.89:0/2185561691 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383400f040 con 0x7f383c161800 2024-04-02T08:52:38.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.735+0000 7f3841a00700 1 -- 172.21.15.89:0/2185561691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383c14bf10 con 0x7f383c161800 2024-04-02T08:52:38.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.735+0000 7f3832a00700 1 -- 172.21.15.89:0/2185561691 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834003c60 con 0x7f383c161800 2024-04-02T08:52:38.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.735+0000 7f3841a00700 1 -- 172.21.15.89:0/2185561691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383c14c4d0 con 0x7f383c161800 2024-04-02T08:52:38.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.736+0000 7f3841a00700 1 -- 172.21.15.89:0/2185561691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383c14bd70 con 0x7f383c161800 2024-04-02T08:52:38.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.737+0000 7f3832a00700 1 -- 172.21.15.89:0/2185561691 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3834004030 con 0x7f383c161800 2024-04-02T08:52:38.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.738+0000 7f3832a00700 1 --2- 172.21.15.89:0/2185561691 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f381c06adb0 0x7f381c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:38.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.738+0000 7f3832a00700 1 -- 172.21.15.89:0/2185561691 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f3834089490 con 0x7f383c161800 2024-04-02T08:52:38.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.738+0000 7f383aa00700 1 --2- 172.21.15.89:0/2185561691 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f381c06adb0 0x7f381c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:38.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.738+0000 7f383aa00700 1 --2- 172.21.15.89:0/2185561691 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f381c06adb0 0x7f381c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f382c007fd0 tx=0x7f382c009040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:38.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:38.742+0000 7f3832a00700 1 -- 172.21.15.89:0/2185561691 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f383c14bd70 con 0x7f383c161800 2024-04-02T08:52:39.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.035+0000 7f3841a00700 1 -- 172.21.15.89:0/2185561691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f383c14bd70 con 0x7f383c161800 2024-04-02T08:52:39.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.036+0000 7f3832a00700 1 -- 172.21.15.89:0/2185561691 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v44) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f383c14bd70 con 0x7f383c161800 2024-04-02T08:52:39.035 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T08:52:39.035 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T08:52:39.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.035 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T08:52:39.035 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T08:52:39.035 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.035 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:52:39.035 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T08:52:39.035 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T08:52:39.036 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.036 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.036 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T08:52:39.036 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T08:52:39.036 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.036 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:52:39.036 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T08:52:39.036 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T08:52:39.036 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.036 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:52:39.037 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T08:52:39.038 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T08:52:39.038 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.038 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T08:52:39.038 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T08:52:39.038 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.038 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:52:39.038 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:52:39.038 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.039 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T08:52:39.039 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T08:52:39.039 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.039 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:52:39.039 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:52:39.039 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.039 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.039 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T08:52:39.039 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T08:52:39.039 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.039 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:52:39.039 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:52:39.040 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.040 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.040 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T08:52:39.040 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T08:52:39.040 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.040 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:52:39.040 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:52:39.040 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.040 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.041 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.042 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T08:52:39.043 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T08:52:39.043 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.043 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:52:39.043 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:52:39.043 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.043 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.043 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T08:52:39.043 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T08:52:39.043 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.043 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:52:39.043 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2024-04-02T08:52:39.043 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T08:52:39.044 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T08:52:39.045 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T08:52:39.046 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T08:52:39.047 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.047 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T08:52:39.047 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T08:52:39.047 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:52:39.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:39.047 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T08:52:39.047 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T08:52:39.047 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:52:39.047 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T08:52:39.047 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T08:52:39.047 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:52:39.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.048 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:39.048 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T08:52:39.048 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T08:52:39.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.039+0000 7f3841a00700 1 -- 172.21.15.89:0/2185561691 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f381c06adb0 msgr2=0x7f381c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.039+0000 7f3841a00700 1 --2- 172.21.15.89:0/2185561691 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f381c06adb0 0x7f381c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f382c007fd0 tx=0x7f382c009040 comp rx=0 tx=0).stop 2024-04-02T08:52:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.039+0000 7f3841a00700 1 -- 172.21.15.89:0/2185561691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f383c161800 msgr2=0x7f383c0cab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.039+0000 7f3841a00700 1 --2- 172.21.15.89:0/2185561691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f383c161800 0x7f383c0cab60 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f3834009fd0 tx=0x7f383400d3b0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.039+0000 7f3841a00700 1 -- 172.21.15.89:0/2185561691 shutdown_connections 2024-04-02T08:52:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.039+0000 7f3841a00700 1 --2- 172.21.15.89:0/2185561691 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f381c06adb0 0x7f381c06d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.039+0000 7f3841a00700 1 --2- 172.21.15.89:0/2185561691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f383c0ceec0 0x7f383c14b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.039+0000 7f3841a00700 1 --2- 172.21.15.89:0/2185561691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f383c161800 0x7f383c0cab60 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.039+0000 7f3841a00700 1 --2- 172.21.15.89:0/2185561691 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f383c15d640 0x7f383c0ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.039+0000 7f3841a00700 1 -- 172.21.15.89:0/2185561691 >> 172.21.15.89:0/2185561691 conn(0x7f383c08aff0 msgr2=0x7f383c0d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.039+0000 7f3841a00700 1 -- 172.21.15.89:0/2185561691 shutdown_connections 2024-04-02T08:52:39.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.039+0000 7f3841a00700 1 -- 172.21.15.89:0/2185561691 wait complete. 2024-04-02T08:52:39.057 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:52:39.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.248+0000 7fcbb1200700 1 Processor -- start 2024-04-02T08:52:39.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.248+0000 7fcbb1200700 1 -- start start 2024-04-02T08:52:39.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.249+0000 7fcbb1200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbac15d5d0 0x7fcbac0d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:39.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.250+0000 7fcbb1200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcbac0d2870 0x7fcbac0d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:39.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.250+0000 7fcbb1200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbac0d2dc0 con 0x7fcbac0d0830 2024-04-02T08:52:39.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.250+0000 7fcbb1200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbac0d0ce0 con 0x7fcbac0d2870 2024-04-02T08:52:39.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.250+0000 7fcbb1200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac15d9b0 con 0x7fcbac15d5d0 2024-04-02T08:52:39.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.250+0000 7fcbaaa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcbac0d2870 0x7fcbac0d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:39.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.250+0000 7fcbaaa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcbac0d2870 0x7fcbac0d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52504/0 (socket says 172.21.15.89:52504) 2024-04-02T08:52:39.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.250+0000 7fcbaaa00700 1 -- 172.21.15.89:0/144484289 learned_addr learned my addr 172.21.15.89:0/144484289 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:39.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.250+0000 7fcbab400700 1 --2- 172.21.15.89:0/144484289 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbac15d5d0 0x7fcbac0d3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:39.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.250+0000 7fcbaaa00700 1 -- 172.21.15.89:0/144484289 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbac15d5d0 msgr2=0x7fcbac0d3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:39.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.250+0000 7fcbaaa00700 1 --2- 172.21.15.89:0/144484289 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbac15d5d0 0x7fcbac0d3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.250+0000 7fcbaaa00700 1 -- 172.21.15.89:0/144484289 >> v1:172.21.15.113:6789/0 conn(0x7fcbac0d0830 legacy=0x7fcbac0b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:39.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.250+0000 7fcbaaa00700 1 -- 172.21.15.89:0/144484289 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac0d30a0 con 0x7fcbac0d2870 2024-04-02T08:52:39.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.251+0000 7fcbaaa00700 1 --2- 172.21.15.89:0/144484289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbac0d2870 0x7fcbac0d02f0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fcb980098f0 tx=0x7fcb98009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46176e58403e3bee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:39.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.251+0000 7fcbaa000700 1 -- 172.21.15.89:0/144484289 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9800b8a0 con 0x7fcbac0d2870 2024-04-02T08:52:39.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.251+0000 7fcbaa000700 1 -- 172.21.15.89:0/144484289 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9800ba00 con 0x7fcbac0d2870 2024-04-02T08:52:39.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.251+0000 7fcbaa000700 1 -- 172.21.15.89:0/144484289 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9800bd90 con 0x7fcbac0d2870 2024-04-02T08:52:39.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.251+0000 7fcbb1200700 1 -- 172.21.15.89:0/144484289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbac0d2870 msgr2=0x7fcbac0d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:39.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.251+0000 7fcbb1200700 1 --2- 172.21.15.89:0/144484289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbac0d2870 0x7fcbac0d02f0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fcb980098f0 tx=0x7fcb98009c00 comp rx=0 tx=0).stop 2024-04-02T08:52:39.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.251+0000 7fcbb1200700 1 -- 172.21.15.89:0/144484289 shutdown_connections 2024-04-02T08:52:39.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.251+0000 7fcbb1200700 1 --2- 172.21.15.89:0/144484289 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbac15d5d0 0x7fcbac0d3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.251+0000 7fcbb1200700 1 --2- 172.21.15.89:0/144484289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbac0d2870 0x7fcbac0d02f0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.251+0000 7fcbb1200700 1 -- 172.21.15.89:0/144484289 >> 172.21.15.89:0/144484289 conn(0x7fcbac0668e0 msgr2=0x7fcbac066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:39.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.251+0000 7fcbb1200700 1 -- 172.21.15.89:0/144484289 shutdown_connections 2024-04-02T08:52:39.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.252+0000 7fcbb1200700 1 -- 172.21.15.89:0/144484289 wait complete. 2024-04-02T08:52:39.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.253+0000 7fcbb1200700 1 Processor -- start 2024-04-02T08:52:39.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.253+0000 7fcbb1200700 1 -- start start 2024-04-02T08:52:39.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.254+0000 7fcbb1200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbac0d0830 0x7fcbac1f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:39.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.254+0000 7fcbb1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbac0d2870 0x7fcbac0cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:39.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.254+0000 7fcbb1200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbac15d5d0 0x7fcbac0aa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:39.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.254+0000 7fcbb1200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac0ccc40 con 0x7fcbac0d2870 2024-04-02T08:52:39.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.254+0000 7fcbb1200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac1f2a90 con 0x7fcbac0d0830 2024-04-02T08:52:39.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.254+0000 7fcbb1200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac0cc070 con 0x7fcbac15d5d0 2024-04-02T08:52:39.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.255+0000 7fcbaaa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbac0d2870 0x7fcbac0cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:39.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.255+0000 7fcbaaa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbac0d2870 0x7fcbac0cd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52510/0 (socket says 172.21.15.89:52510) 2024-04-02T08:52:39.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.255+0000 7fcbaaa00700 1 -- 172.21.15.89:0/1232692953 learned_addr learned my addr 172.21.15.89:0/1232692953 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:39.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.255+0000 7fcbab400700 1 --2- 172.21.15.89:0/1232692953 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbac0d0830 0x7fcbac1f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:39.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.255+0000 7fcbabe00700 1 --2- 172.21.15.89:0/1232692953 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbac15d5d0 0x7fcbac0aa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:39.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.255+0000 7fcbaaa00700 1 -- 172.21.15.89:0/1232692953 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbac15d5d0 msgr2=0x7fcbac0aa6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:39.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.255+0000 7fcbaaa00700 1 --2- 172.21.15.89:0/1232692953 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbac15d5d0 0x7fcbac0aa6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.255+0000 7fcbaaa00700 1 -- 172.21.15.89:0/1232692953 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbac0d0830 msgr2=0x7fcbac1f2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:39.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.255+0000 7fcbaaa00700 1 --2- 172.21.15.89:0/1232692953 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbac0d0830 0x7fcbac1f2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.255+0000 7fcbaaa00700 1 -- 172.21.15.89:0/1232692953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb98009550 con 0x7fcbac0d2870 2024-04-02T08:52:39.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.255+0000 7fcbaaa00700 1 --2- 172.21.15.89:0/1232692953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbac0d2870 0x7fcbac0cd0d0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fcb98009c40 tx=0x7fcb98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:39.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.255+0000 7fcba3400700 1 -- 172.21.15.89:0/1232692953 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb98004380 con 0x7fcbac0d2870 2024-04-02T08:52:39.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.256+0000 7fcba3400700 1 -- 172.21.15.89:0/1232692953 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9801f070 con 0x7fcbac0d2870 2024-04-02T08:52:39.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.256+0000 7fcba3400700 1 -- 172.21.15.89:0/1232692953 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb980036a0 con 0x7fcbac0d2870 2024-04-02T08:52:39.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.256+0000 7fcbb1200700 1 -- 172.21.15.89:0/1232692953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbac0cc310 con 0x7fcbac0d2870 2024-04-02T08:52:39.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.256+0000 7fcbb1200700 1 -- 172.21.15.89:0/1232692953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbac0cc8d0 con 0x7fcbac0d2870 2024-04-02T08:52:39.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.257+0000 7fcbb1200700 1 -- 172.21.15.89:0/1232692953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbac0b2860 con 0x7fcbac0d2870 2024-04-02T08:52:39.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.258+0000 7fcba3400700 1 -- 172.21.15.89:0/1232692953 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcb98021030 con 0x7fcbac0d2870 2024-04-02T08:52:39.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.258+0000 7fcba3400700 1 --2- 172.21.15.89:0/1232692953 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb8c06adb0 0x7fcb8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:39.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.259+0000 7fcba3400700 1 -- 172.21.15.89:0/1232692953 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5277+0+0 (secure 0 0 0) 0x7fcb9808a620 con 0x7fcbac0d2870 2024-04-02T08:52:39.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.259+0000 7fcbab400700 1 --2- 172.21.15.89:0/1232692953 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb8c06adb0 0x7fcb8c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:39.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.259+0000 7fcbab400700 1 --2- 172.21.15.89:0/1232692953 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb8c06adb0 0x7fcb8c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c006b20 tx=0x7fcb9c008040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:39.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.263+0000 7fcba3400700 1 -- 172.21.15.89:0/1232692953 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcb98055930 con 0x7fcbac0d2870 2024-04-02T08:52:39.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.466+0000 7fcbb1200700 1 -- 172.21.15.89:0/1232692953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbac04ea80 con 0x7fcbac0d2870 2024-04-02T08:52:39.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.467+0000 7fcba3400700 1 -- 172.21.15.89:0/1232692953 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+13042 (secure 0 0 0) 0x7fcb98058f50 con 0x7fcbac0d2870 2024-04-02T08:52:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:39.466 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":168,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:52:38.502181+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:52:39.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.469+0000 7fcbb1200700 1 -- 172.21.15.89:0/1232692953 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb8c06adb0 msgr2=0x7fcb8c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:39.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.469+0000 7fcbb1200700 1 --2- 172.21.15.89:0/1232692953 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb8c06adb0 0x7fcb8c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c006b20 tx=0x7fcb9c008040 comp rx=0 tx=0).stop 2024-04-02T08:52:39.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.469+0000 7fcbb1200700 1 -- 172.21.15.89:0/1232692953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbac0d2870 msgr2=0x7fcbac0cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:39.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.469+0000 7fcbb1200700 1 --2- 172.21.15.89:0/1232692953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbac0d2870 0x7fcbac0cd0d0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fcb98009c40 tx=0x7fcb98004750 comp rx=0 tx=0).stop 2024-04-02T08:52:39.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.469+0000 7fcbb1200700 1 -- 172.21.15.89:0/1232692953 shutdown_connections 2024-04-02T08:52:39.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.469+0000 7fcbb1200700 1 --2- 172.21.15.89:0/1232692953 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb8c06adb0 0x7fcb8c06d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.469+0000 7fcbb1200700 1 --2- 172.21.15.89:0/1232692953 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbac0d0830 0x7fcbac1f2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.469+0000 7fcbb1200700 1 --2- 172.21.15.89:0/1232692953 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbac15d5d0 0x7fcbac0aa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.469+0000 7fcbb1200700 1 --2- 172.21.15.89:0/1232692953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbac0d2870 0x7fcbac0cd0d0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.469+0000 7fcbb1200700 1 -- 172.21.15.89:0/1232692953 >> 172.21.15.89:0/1232692953 conn(0x7fcbac0668e0 msgr2=0x7fcbac0b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:39.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.469+0000 7fcbb1200700 1 -- 172.21.15.89:0/1232692953 shutdown_connections 2024-04-02T08:52:39.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.469+0000 7fcbb1200700 1 -- 172.21.15.89:0/1232692953 wait complete. 2024-04-02T08:52:39.487 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T08:52:39.487 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T08:52:39.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.683+0000 7ffb99000700 1 Processor -- start 2024-04-02T08:52:39.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.683+0000 7ffb99000700 1 -- start start 2024-04-02T08:52:39.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.684+0000 7ffb99000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb9415d8a0 0x7ffb940b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:39.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.684+0000 7ffb99000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ffb940d3eb0 0x7ffb9406b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:39.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.684+0000 7ffb99000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb9406bdd0 0x7ffb940cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:39.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.684+0000 7ffb99000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb940ced20 con 0x7ffb9415d8a0 2024-04-02T08:52:39.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.684+0000 7ffb99000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb940d4340 con 0x7ffb940d3eb0 2024-04-02T08:52:39.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.684+0000 7ffb99000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940ce6d0 con 0x7ffb9406bdd0 2024-04-02T08:52:39.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.684+0000 7ffb91e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb9415d8a0 0x7ffb940b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:39.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.684+0000 7ffb91e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb9415d8a0 0x7ffb940b2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52512/0 (socket says 172.21.15.89:52512) 2024-04-02T08:52:39.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.684+0000 7ffb91e00700 1 -- 172.21.15.89:0/3178938997 learned_addr learned my addr 172.21.15.89:0/3178938997 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:39.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.684+0000 7ffb91400700 1 --2- 172.21.15.89:0/3178938997 >> v2:172.21.15.113:3300/0 conn(0x7ffb940d3eb0 0x7ffb9406b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:39.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.684+0000 7ffb92800700 1 --2- 172.21.15.89:0/3178938997 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb9406bdd0 0x7ffb940cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:39.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.685+0000 7ffb91e00700 1 -- 172.21.15.89:0/3178938997 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb9406bdd0 msgr2=0x7ffb940cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:39.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.685+0000 7ffb91e00700 1 --2- 172.21.15.89:0/3178938997 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb9406bdd0 0x7ffb940cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.685+0000 7ffb91e00700 1 -- 172.21.15.89:0/3178938997 >> v2:172.21.15.113:3300/0 conn(0x7ffb940d3eb0 msgr2=0x7ffb9406b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:39.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.685+0000 7ffb91e00700 1 --2- 172.21.15.89:0/3178938997 >> v2:172.21.15.113:3300/0 conn(0x7ffb940d3eb0 0x7ffb9406b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.685+0000 7ffb91e00700 1 -- 172.21.15.89:0/3178938997 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb940ce9b0 con 0x7ffb9415d8a0 2024-04-02T08:52:39.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.685+0000 7ffb91e00700 1 --2- 172.21.15.89:0/3178938997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb9415d8a0 0x7ffb940b2990 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7ffb9408d6c0 tx=0x7ffb84009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3219edc04028f1d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:39.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.685+0000 7ffb90a00700 1 -- 172.21.15.89:0/3178938997 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8400b870 con 0x7ffb9415d8a0 2024-04-02T08:52:39.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.685+0000 7ffb90a00700 1 -- 172.21.15.89:0/3178938997 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb8400b9d0 con 0x7ffb9415d8a0 2024-04-02T08:52:39.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.685+0000 7ffb90a00700 1 -- 172.21.15.89:0/3178938997 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8400bd60 con 0x7ffb9415d8a0 2024-04-02T08:52:39.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.686+0000 7ffb99000700 1 -- 172.21.15.89:0/3178938997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb9415d8a0 msgr2=0x7ffb940b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:39.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.686+0000 7ffb99000700 1 --2- 172.21.15.89:0/3178938997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb9415d8a0 0x7ffb940b2990 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7ffb9408d6c0 tx=0x7ffb84009bd0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.686+0000 7ffb99000700 1 -- 172.21.15.89:0/3178938997 shutdown_connections 2024-04-02T08:52:39.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.686+0000 7ffb99000700 1 --2- 172.21.15.89:0/3178938997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb9415d8a0 0x7ffb940b2990 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.686+0000 7ffb99000700 1 --2- 172.21.15.89:0/3178938997 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb9406bdd0 0x7ffb940cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.686+0000 7ffb99000700 1 --2- 172.21.15.89:0/3178938997 >> v2:172.21.15.113:3300/0 conn(0x7ffb940d3eb0 0x7ffb9406b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.686+0000 7ffb99000700 1 -- 172.21.15.89:0/3178938997 >> 172.21.15.89:0/3178938997 conn(0x7ffb9408aff0 msgr2=0x7ffb940d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:39.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.686+0000 7ffb99000700 1 -- 172.21.15.89:0/3178938997 shutdown_connections 2024-04-02T08:52:39.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.686+0000 7ffb99000700 1 -- 172.21.15.89:0/3178938997 wait complete. 2024-04-02T08:52:39.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.687+0000 7ffb99000700 1 Processor -- start 2024-04-02T08:52:39.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.688+0000 7ffb99000700 1 -- start start 2024-04-02T08:52:39.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.688+0000 7ffb99000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb9406bdd0 0x7ffb940c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:39.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.688+0000 7ffb99000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb9415d8a0 0x7ffb940c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:39.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.688+0000 7ffb99000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb94144750 0x7ffb9415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:39.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.688+0000 7ffb99000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940cd790 con 0x7ffb94144750 2024-04-02T08:52:39.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.688+0000 7ffb99000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940cd4b0 con 0x7ffb9415d8a0 2024-04-02T08:52:39.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.688+0000 7ffb99000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb94159ed0 con 0x7ffb9406bdd0 2024-04-02T08:52:39.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.689+0000 7ffb92800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb94144750 0x7ffb9415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:39.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.689+0000 7ffb92800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb94144750 0x7ffb9415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52514/0 (socket says 172.21.15.89:52514) 2024-04-02T08:52:39.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.689+0000 7ffb92800700 1 -- 172.21.15.89:0/1572947890 learned_addr learned my addr 172.21.15.89:0/1572947890 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:39.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.689+0000 7ffb91400700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb9415d8a0 0x7ffb940c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.689+0000 7ffb91e00700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb9406bdd0 0x7ffb940c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.689+0000 7ffb92800700 1 -- 172.21.15.89:0/1572947890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb9406bdd0 msgr2=0x7ffb940c8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.689+0000 7ffb92800700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb9406bdd0 0x7ffb940c8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.689+0000 7ffb92800700 1 -- 172.21.15.89:0/1572947890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb9415d8a0 msgr2=0x7ffb940c88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.689+0000 7ffb92800700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb9415d8a0 0x7ffb940c88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.689+0000 7ffb92800700 1 -- 172.21.15.89:0/1572947890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb84009550 con 0x7ffb94144750 2024-04-02T08:52:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.689+0000 7ffb91e00700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb9406bdd0 0x7ffb940c8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:52:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.689+0000 7ffb91400700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb9415d8a0 0x7ffb940c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:52:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.689+0000 7ffb92800700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb94144750 0x7ffb9415a9b0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c007e30 tx=0x7ffb8c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.690+0000 7ffb8a000700 1 -- 172.21.15.89:0/1572947890 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c00eea0 con 0x7ffb94144750 2024-04-02T08:52:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.690+0000 7ffb8a000700 1 -- 172.21.15.89:0/1572947890 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb8c014420 con 0x7ffb94144750 2024-04-02T08:52:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.690+0000 7ffb8a000700 1 -- 172.21.15.89:0/1572947890 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c0147d0 con 0x7ffb94144750 2024-04-02T08:52:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.690+0000 7ffb99000700 1 -- 172.21.15.89:0/1572947890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb9415a1b0 con 0x7ffb94144750 2024-04-02T08:52:39.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.690+0000 7ffb99000700 1 -- 172.21.15.89:0/1572947890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb940cd170 con 0x7ffb94144750 2024-04-02T08:52:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.691+0000 7ffb99000700 1 -- 172.21.15.89:0/1572947890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb94159d30 con 0x7ffb94144750 2024-04-02T08:52:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.692+0000 7ffb8a000700 1 -- 172.21.15.89:0/1572947890 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ffb8c014930 con 0x7ffb94144750 2024-04-02T08:52:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.692+0000 7ffb8a000700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb7406adb0 0x7ffb7406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.693+0000 7ffb8a000700 1 -- 172.21.15.89:0/1572947890 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5277+0+0 (secure 0 0 0) 0x7ffb8c089b30 con 0x7ffb94144750 2024-04-02T08:52:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.693+0000 7ffb91e00700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb7406adb0 0x7ffb7406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.693+0000 7ffb91e00700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb7406adb0 0x7ffb7406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffb84009130 tx=0x7ffb84008030 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:39.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.697+0000 7ffb8a000700 1 -- 172.21.15.89:0/1572947890 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffb94159d30 con 0x7ffb94144750 2024-04-02T08:52:39.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.898+0000 7ffb99000700 1 -- 172.21.15.89:0/1572947890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffb9415a6d0 con 0x7ffb94144750 2024-04-02T08:52:39.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.898+0000 7ffb8a000700 1 -- 172.21.15.89:0/1572947890 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v168) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffb9415a6d0 con 0x7ffb94144750 2024-04-02T08:52:39.897 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T08:52:39.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.900+0000 7ffb99000700 1 -- 172.21.15.89:0/1572947890 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb7406adb0 msgr2=0x7ffb7406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:39.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.900+0000 7ffb99000700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb7406adb0 0x7ffb7406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffb84009130 tx=0x7ffb84008030 comp rx=0 tx=0).stop 2024-04-02T08:52:39.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.900+0000 7ffb99000700 1 -- 172.21.15.89:0/1572947890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb94144750 msgr2=0x7ffb9415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:39.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.900+0000 7ffb99000700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb94144750 0x7ffb9415a9b0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c007e30 tx=0x7ffb8c00e3c0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.901+0000 7ffb99000700 1 -- 172.21.15.89:0/1572947890 shutdown_connections 2024-04-02T08:52:39.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.901+0000 7ffb99000700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ffb7406adb0 0x7ffb7406d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.901+0000 7ffb99000700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb94144750 0x7ffb9415a9b0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.901+0000 7ffb99000700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb9406bdd0 0x7ffb940c8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.901+0000 7ffb99000700 1 --2- 172.21.15.89:0/1572947890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb9415d8a0 0x7ffb940c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:39.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.901+0000 7ffb99000700 1 -- 172.21.15.89:0/1572947890 >> 172.21.15.89:0/1572947890 conn(0x7ffb9408aff0 msgr2=0x7ffb940b6090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:39.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.901+0000 7ffb99000700 1 -- 172.21.15.89:0/1572947890 shutdown_connections 2024-04-02T08:52:39.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:39.901+0000 7ffb99000700 1 -- 172.21.15.89:0/1572947890 wait complete. 2024-04-02T08:52:39.918 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T08:52:40.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.111+0000 7f6ff1400700 1 Processor -- start 2024-04-02T08:52:40.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.112+0000 7f6ff1400700 1 -- start start 2024-04-02T08:52:40.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.113+0000 7f6ff1400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6fec15d640 0x7f6fec0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:40.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.113+0000 7f6ff1400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6fec0ceec0 0x7f6fec1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:40.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.113+0000 7f6ff1400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec0d30d0 con 0x7f6fec161800 2024-04-02T08:52:40.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.113+0000 7f6ff1400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fec0b6840 con 0x7f6fec0ceec0 2024-04-02T08:52:40.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.113+0000 7f6ff1400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fec161c20 con 0x7f6fec15d640 2024-04-02T08:52:40.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.113+0000 7f6fe9e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6fec0ceec0 0x7f6fec1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:40.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.113+0000 7f6fe9e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6fec0ceec0 0x7f6fec1612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52526/0 (socket says 172.21.15.89:52526) 2024-04-02T08:52:40.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.113+0000 7f6fe9e00700 1 -- 172.21.15.89:0/2110253339 learned_addr learned my addr 172.21.15.89:0/2110253339 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:40.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.113+0000 7f6fea800700 1 --2- 172.21.15.89:0/2110253339 >> v2:172.21.15.113:3300/0 conn(0x7f6fec15d640 0x7f6fec0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:40.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.113+0000 7f6fe9e00700 1 -- 172.21.15.89:0/2110253339 >> v2:172.21.15.113:3300/0 conn(0x7f6fec15d640 msgr2=0x7f6fec0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:40.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.113+0000 7f6fe9e00700 1 --2- 172.21.15.89:0/2110253339 >> v2:172.21.15.113:3300/0 conn(0x7f6fec15d640 0x7f6fec0d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.113+0000 7f6fe9e00700 1 -- 172.21.15.89:0/2110253339 >> v1:172.21.15.113:6789/0 conn(0x7f6fec161800 legacy=0x7f6fec0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:40.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.113+0000 7f6fe9e00700 1 -- 172.21.15.89:0/2110253339 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec0d2880 con 0x7f6fec0ceec0 2024-04-02T08:52:40.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.114+0000 7f6fe9e00700 1 --2- 172.21.15.89:0/2110253339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec0ceec0 0x7f6fec1612c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0b5c20 tx=0x7f6fd8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96b14fbe4b4d6d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:40.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.114+0000 7f6fe9400700 1 -- 172.21.15.89:0/2110253339 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8011830 con 0x7f6fec0ceec0 2024-04-02T08:52:40.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.114+0000 7f6fe9400700 1 -- 172.21.15.89:0/2110253339 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd8011990 con 0x7f6fec0ceec0 2024-04-02T08:52:40.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.114+0000 7f6fe9400700 1 -- 172.21.15.89:0/2110253339 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8011d20 con 0x7f6fec0ceec0 2024-04-02T08:52:40.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.114+0000 7f6ff1400700 1 -- 172.21.15.89:0/2110253339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec0ceec0 msgr2=0x7f6fec1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:40.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.114+0000 7f6ff1400700 1 --2- 172.21.15.89:0/2110253339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec0ceec0 0x7f6fec1612c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0b5c20 tx=0x7f6fd8009bd0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.115+0000 7f6ff1400700 1 -- 172.21.15.89:0/2110253339 shutdown_connections 2024-04-02T08:52:40.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.115+0000 7f6ff1400700 1 --2- 172.21.15.89:0/2110253339 >> v2:172.21.15.113:3300/0 conn(0x7f6fec15d640 0x7f6fec0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.115+0000 7f6ff1400700 1 --2- 172.21.15.89:0/2110253339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec0ceec0 0x7f6fec1612c0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.115+0000 7f6ff1400700 1 -- 172.21.15.89:0/2110253339 >> 172.21.15.89:0/2110253339 conn(0x7f6fec08aff0 msgr2=0x7f6fec0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:40.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.115+0000 7f6ff1400700 1 -- 172.21.15.89:0/2110253339 shutdown_connections 2024-04-02T08:52:40.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.115+0000 7f6ff1400700 1 -- 172.21.15.89:0/2110253339 wait complete. 2024-04-02T08:52:40.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.116+0000 7f6ff1400700 1 Processor -- start 2024-04-02T08:52:40.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.116+0000 7f6ff1400700 1 -- start start 2024-04-02T08:52:40.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.117+0000 7f6ff1400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fec0ceec0 0x7f6fec158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:40.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.117+0000 7f6ff1400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fec15d640 0x7f6fec09e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:40.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.117+0000 7f6ff1400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec161800 0x7f6fec15f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.117+0000 7f6ff1400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec0a0be0 con 0x7f6fec161800 2024-04-02T08:52:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.117+0000 7f6ff1400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec158f80 con 0x7f6fec15d640 2024-04-02T08:52:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.117+0000 7f6ff1400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec0a02b0 con 0x7f6fec0ceec0 2024-04-02T08:52:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.118+0000 7f6feb200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec161800 0x7f6fec15f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.118+0000 7f6feb200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec161800 0x7f6fec15f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52532/0 (socket says 172.21.15.89:52532) 2024-04-02T08:52:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.118+0000 7f6feb200700 1 -- 172.21.15.89:0/1325059764 learned_addr learned my addr 172.21.15.89:0/1325059764 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.118+0000 7f6fe9e00700 1 --2- 172.21.15.89:0/1325059764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fec15d640 0x7f6fec09e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.118+0000 7f6fea800700 1 --2- 172.21.15.89:0/1325059764 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fec0ceec0 0x7f6fec158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.118+0000 7f6feb200700 1 -- 172.21.15.89:0/1325059764 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fec0ceec0 msgr2=0x7f6fec158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.118+0000 7f6feb200700 1 --2- 172.21.15.89:0/1325059764 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fec0ceec0 0x7f6fec158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.118+0000 7f6feb200700 1 -- 172.21.15.89:0/1325059764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fec15d640 msgr2=0x7f6fec09e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.118+0000 7f6feb200700 1 --2- 172.21.15.89:0/1325059764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fec15d640 0x7f6fec09e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.118+0000 7f6feb200700 1 -- 172.21.15.89:0/1325059764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd8009550 con 0x7f6fec161800 2024-04-02T08:52:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.118+0000 7f6feb200700 1 --2- 172.21.15.89:0/1325059764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec161800 0x7f6fec15f370 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400cc50 tx=0x7f6fe400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:40.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.119+0000 7f6fe2a00700 1 -- 172.21.15.89:0/1325059764 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4003c00 con 0x7f6fec161800 2024-04-02T08:52:40.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.119+0000 7f6fe2a00700 1 -- 172.21.15.89:0/1325059764 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe4003d60 con 0x7f6fec161800 2024-04-02T08:52:40.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.119+0000 7f6fe2a00700 1 -- 172.21.15.89:0/1325059764 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe400aab0 con 0x7f6fec161800 2024-04-02T08:52:40.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.119+0000 7f6ff1400700 1 -- 172.21.15.89:0/1325059764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fec0a0590 con 0x7f6fec161800 2024-04-02T08:52:40.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.119+0000 7f6ff1400700 1 -- 172.21.15.89:0/1325059764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fec0a1070 con 0x7f6fec161800 2024-04-02T08:52:40.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.120+0000 7f6ff1400700 1 -- 172.21.15.89:0/1325059764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fec067720 con 0x7f6fec161800 2024-04-02T08:52:40.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.121+0000 7f6fe2a00700 1 -- 172.21.15.89:0/1325059764 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6fe400ac10 con 0x7f6fec161800 2024-04-02T08:52:40.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.121+0000 7f6fe2a00700 1 --2- 172.21.15.89:0/1325059764 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6fcc06adb0 0x7f6fcc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:40.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.122+0000 7f6fe2a00700 1 -- 172.21.15.89:0/1325059764 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f6fe40892d0 con 0x7f6fec161800 2024-04-02T08:52:40.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.122+0000 7f6fea800700 1 --2- 172.21.15.89:0/1325059764 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6fcc06adb0 0x7f6fcc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:40.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.122+0000 7f6fea800700 1 --2- 172.21.15.89:0/1325059764 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6fcc06adb0 0x7f6fcc06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6fec1f0e80 tx=0x7f6fdc009210 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:40.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.126+0000 7f6fe2a00700 1 -- 172.21.15.89:0/1325059764 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6fe4054540 con 0x7f6fec161800 2024-04-02T08:52:40.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.329+0000 7f6ff1400700 1 -- 172.21.15.89:0/1325059764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6fec0ceb00 con 0x7f6fec161800 2024-04-02T08:52:40.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.329+0000 7f6fe2a00700 1 -- 172.21.15.89:0/1325059764 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v168) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6fe4057b60 con 0x7f6fec161800 2024-04-02T08:52:40.328 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:40.328 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T08:52:40.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:40.328 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T08:52:40.328 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T08:52:40.328 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:40.328 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T08:52:40.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.331+0000 7f6ff1400700 1 -- 172.21.15.89:0/1325059764 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6fcc06adb0 msgr2=0x7f6fcc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:40.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.331+0000 7f6ff1400700 1 --2- 172.21.15.89:0/1325059764 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6fcc06adb0 0x7f6fcc06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6fec1f0e80 tx=0x7f6fdc009210 comp rx=0 tx=0).stop 2024-04-02T08:52:40.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.331+0000 7f6ff1400700 1 -- 172.21.15.89:0/1325059764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec161800 msgr2=0x7f6fec15f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:40.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.331+0000 7f6ff1400700 1 --2- 172.21.15.89:0/1325059764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec161800 0x7f6fec15f370 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400cc50 tx=0x7f6fe400cf60 comp rx=0 tx=0).stop 2024-04-02T08:52:40.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.332+0000 7f6ff1400700 1 -- 172.21.15.89:0/1325059764 shutdown_connections 2024-04-02T08:52:40.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.332+0000 7f6ff1400700 1 --2- 172.21.15.89:0/1325059764 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f6fcc06adb0 0x7f6fcc06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.332+0000 7f6ff1400700 1 --2- 172.21.15.89:0/1325059764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec161800 0x7f6fec15f370 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.332+0000 7f6ff1400700 1 --2- 172.21.15.89:0/1325059764 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fec0ceec0 0x7f6fec158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.332+0000 7f6ff1400700 1 --2- 172.21.15.89:0/1325059764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fec15d640 0x7f6fec09e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.332+0000 7f6ff1400700 1 -- 172.21.15.89:0/1325059764 >> 172.21.15.89:0/1325059764 conn(0x7f6fec08aff0 msgr2=0x7f6fec15b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:40.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.332+0000 7f6ff1400700 1 -- 172.21.15.89:0/1325059764 shutdown_connections 2024-04-02T08:52:40.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.332+0000 7f6ff1400700 1 -- 172.21.15.89:0/1325059764 wait complete. 2024-04-02T08:52:40.349 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T08:52:40.349 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T08:52:40.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.546+0000 7f4b40800700 1 Processor -- start 2024-04-02T08:52:40.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.547+0000 7f4b40800700 1 -- start start 2024-04-02T08:52:40.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b40800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4b380ced30 0x7f4b380d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:40.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b40800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4b380d2710 0x7f4b3815f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:40.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b40800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b380ce8a0 con 0x7f4b3815ddf0 2024-04-02T08:52:40.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b40800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3815ec60 con 0x7f4b380ced30 2024-04-02T08:52:40.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b40800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b380d2b60 con 0x7f4b380d2710 2024-04-02T08:52:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b3d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4b380ced30 0x7f4b380d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b3d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4b380ced30 0x7f4b380d21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52534/0 (socket says 172.21.15.89:52534) 2024-04-02T08:52:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b3d400700 1 -- 172.21.15.89:0/351435209 learned_addr learned my addr 172.21.15.89:0/351435209 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b3de00700 1 --1- 172.21.15.89:0/351435209 >> v1:172.21.15.89:6789/0 conn(0x7f4b3815ddf0 0x7f4b3815e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42744/0 (socket says 172.21.15.89:42744) 2024-04-02T08:52:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b3e800700 1 --2- 172.21.15.89:0/351435209 >> v2:172.21.15.113:3300/0 conn(0x7f4b380d2710 0x7f4b3815f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b3d400700 1 -- 172.21.15.89:0/351435209 >> v2:172.21.15.113:3300/0 conn(0x7f4b380d2710 msgr2=0x7f4b3815f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b3d400700 1 --2- 172.21.15.89:0/351435209 >> v2:172.21.15.113:3300/0 conn(0x7f4b380d2710 0x7f4b3815f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b3d400700 1 -- 172.21.15.89:0/351435209 >> v1:172.21.15.89:6789/0 conn(0x7f4b3815ddf0 legacy=0x7f4b3815e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b3d400700 1 -- 172.21.15.89:0/351435209 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b380cdc90 con 0x7f4b380ced30 2024-04-02T08:52:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b3ca00700 1 -- 172.21.15.89:0/351435209 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928761106 0 0) 0x7f4b380ce8a0 con 0x7f4b3815ddf0 2024-04-02T08:52:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.548+0000 7f4b3e800700 1 --2- 172.21.15.89:0/351435209 >> v2:172.21.15.113:3300/0 conn(0x7f4b380d2710 0x7f4b3815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:52:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.549+0000 7f4b3d400700 1 --2- 172.21.15.89:0/351435209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b380ced30 0x7f4b380d21d0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f4b380cf9f0 tx=0x7f4b28009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a9c34f56048dd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.549+0000 7f4b3ca00700 1 -- 172.21.15.89:0/351435209 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2800b870 con 0x7f4b380ced30 2024-04-02T08:52:40.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.549+0000 7f4b3ca00700 1 -- 172.21.15.89:0/351435209 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2800b9d0 con 0x7f4b380ced30 2024-04-02T08:52:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.549+0000 7f4b3ca00700 1 -- 172.21.15.89:0/351435209 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2800bd60 con 0x7f4b380ced30 2024-04-02T08:52:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.549+0000 7f4b40800700 1 -- 172.21.15.89:0/351435209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b380ced30 msgr2=0x7f4b380d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.549+0000 7f4b40800700 1 --2- 172.21.15.89:0/351435209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b380ced30 0x7f4b380d21d0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f4b380cf9f0 tx=0x7f4b28009bd0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.549+0000 7f4b40800700 1 -- 172.21.15.89:0/351435209 shutdown_connections 2024-04-02T08:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.549+0000 7f4b40800700 1 --2- 172.21.15.89:0/351435209 >> v2:172.21.15.113:3300/0 conn(0x7f4b380d2710 0x7f4b3815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.549+0000 7f4b40800700 1 --2- 172.21.15.89:0/351435209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b380ced30 0x7f4b380d21d0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.549+0000 7f4b40800700 1 -- 172.21.15.89:0/351435209 >> 172.21.15.89:0/351435209 conn(0x7f4b3808aff0 msgr2=0x7f4b380d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.550+0000 7f4b40800700 1 -- 172.21.15.89:0/351435209 shutdown_connections 2024-04-02T08:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.550+0000 7f4b40800700 1 -- 172.21.15.89:0/351435209 wait complete. 2024-04-02T08:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.551+0000 7f4b40800700 1 Processor -- start 2024-04-02T08:52:40.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.551+0000 7f4b40800700 1 -- start start 2024-04-02T08:52:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.552+0000 7f4b40800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b380ced30 0x7f4b38156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.552+0000 7f4b40800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b380d2710 0x7f4b381e7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.552+0000 7f4b40800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b3815ddf0 0x7f4b380cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.552+0000 7f4b40800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b381610d0 con 0x7f4b380d2710 2024-04-02T08:52:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.552+0000 7f4b40800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b381613b0 con 0x7f4b380ced30 2024-04-02T08:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.552+0000 7f4b40800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b38161690 con 0x7f4b3815ddf0 2024-04-02T08:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.552+0000 7f4b3d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b380d2710 0x7f4b381e7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.552+0000 7f4b3d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b380d2710 0x7f4b381e7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52546/0 (socket says 172.21.15.89:52546) 2024-04-02T08:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.552+0000 7f4b3d400700 1 -- 172.21.15.89:0/497560982 learned_addr learned my addr 172.21.15.89:0/497560982 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.553+0000 7f4b3de00700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b380ced30 0x7f4b38156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.553+0000 7f4b3e800700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b3815ddf0 0x7f4b380cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.553+0000 7f4b3d400700 1 -- 172.21.15.89:0/497560982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b3815ddf0 msgr2=0x7f4b380cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.553+0000 7f4b3d400700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b3815ddf0 0x7f4b380cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.553+0000 7f4b3d400700 1 -- 172.21.15.89:0/497560982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b380ced30 msgr2=0x7f4b38156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.553+0000 7f4b3d400700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b380ced30 0x7f4b38156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.553+0000 7f4b3d400700 1 -- 172.21.15.89:0/497560982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b28009550 con 0x7f4b380d2710 2024-04-02T08:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.553+0000 7f4b3de00700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b380ced30 0x7f4b38156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.553+0000 7f4b3d400700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b380d2710 0x7f4b381e7f70 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f4b3808ce40 tx=0x7f4b28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:40.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.553+0000 7f4b32000700 1 -- 172.21.15.89:0/497560982 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b28011760 con 0x7f4b380d2710 2024-04-02T08:52:40.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.553+0000 7f4b32000700 1 -- 172.21.15.89:0/497560982 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2801f070 con 0x7f4b380d2710 2024-04-02T08:52:40.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.553+0000 7f4b40800700 1 -- 172.21.15.89:0/497560982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b38161970 con 0x7f4b380d2710 2024-04-02T08:52:40.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.553+0000 7f4b32000700 1 -- 172.21.15.89:0/497560982 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b28011ac0 con 0x7f4b380d2710 2024-04-02T08:52:40.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.554+0000 7f4b40800700 1 -- 172.21.15.89:0/497560982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b38160cf0 con 0x7f4b380d2710 2024-04-02T08:52:40.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.555+0000 7f4b40800700 1 -- 172.21.15.89:0/497560982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b38160e80 con 0x7f4b380d2710 2024-04-02T08:52:40.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.555+0000 7f4b32000700 1 -- 172.21.15.89:0/497560982 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4b28021030 con 0x7f4b380d2710 2024-04-02T08:52:40.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.556+0000 7f4b32000700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4b1c06adb0 0x7f4b1c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:40.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.557+0000 7f4b32000700 1 -- 172.21.15.89:0/497560982 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f4b2808a880 con 0x7f4b380d2710 2024-04-02T08:52:40.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.557+0000 7f4b3de00700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4b1c06adb0 0x7f4b1c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:40.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.557+0000 7f4b3de00700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4b1c06adb0 0x7f4b1c06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c000f60 tx=0x7f4b2c007d30 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:40.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.560+0000 7f4b32000700 1 -- 172.21.15.89:0/497560982 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4b28055af0 con 0x7f4b380d2710 2024-04-02T08:52:40.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.763+0000 7f4b40800700 1 -- 172.21.15.89:0/497560982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4b38067720 con 0x7f4b380d2710 2024-04-02T08:52:40.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.978+0000 7f4b32000700 1 -- 172.21.15.89:0/497560982 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v169) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4b28059110 con 0x7f4b380d2710 2024-04-02T08:52:40.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.981+0000 7f4b40800700 1 -- 172.21.15.89:0/497560982 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4b1c06adb0 msgr2=0x7f4b1c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:40.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.981+0000 7f4b40800700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4b1c06adb0 0x7f4b1c06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c000f60 tx=0x7f4b2c007d30 comp rx=0 tx=0).stop 2024-04-02T08:52:40.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.981+0000 7f4b40800700 1 -- 172.21.15.89:0/497560982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b380d2710 msgr2=0x7f4b381e7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:40.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.981+0000 7f4b40800700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b380d2710 0x7f4b381e7f70 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f4b3808ce40 tx=0x7f4b28004750 comp rx=0 tx=0).stop 2024-04-02T08:52:40.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.982+0000 7f4b40800700 1 -- 172.21.15.89:0/497560982 shutdown_connections 2024-04-02T08:52:40.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.982+0000 7f4b40800700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4b1c06adb0 0x7f4b1c06d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.982+0000 7f4b40800700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b380ced30 0x7f4b38156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.982+0000 7f4b40800700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b3815ddf0 0x7f4b380cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.982+0000 7f4b40800700 1 --2- 172.21.15.89:0/497560982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b380d2710 0x7f4b381e7f70 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:40.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.982+0000 7f4b40800700 1 -- 172.21.15.89:0/497560982 >> 172.21.15.89:0/497560982 conn(0x7f4b3808aff0 msgr2=0x7f4b380b8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:40.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.982+0000 7f4b40800700 1 -- 172.21.15.89:0/497560982 shutdown_connections 2024-04-02T08:52:40.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:40.982+0000 7f4b40800700 1 -- 172.21.15.89:0/497560982 wait complete. 2024-04-02T08:52:40.984 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T08:52:41.001 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T08:52:41.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.201+0000 7f4c57800700 1 Processor -- start 2024-04-02T08:52:41.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.201+0000 7f4c57800700 1 -- start start 2024-04-02T08:52:41.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.202+0000 7f4c57800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4c500d16a0 0x7f4c500d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:41.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.202+0000 7f4c57800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c500d2370 0x7f4c500ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:41.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.202+0000 7f4c57800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c50161420 con 0x7f4c5015d9d0 2024-04-02T08:52:41.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.202+0000 7f4c57800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c500d2820 con 0x7f4c500d16a0 2024-04-02T08:52:41.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.203+0000 7f4c57800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c500d4340 con 0x7f4c500d2370 2024-04-02T08:52:41.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.203+0000 7f4c4fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4c500d16a0 0x7f4c500d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:41.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.203+0000 7f4c4fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4c500d16a0 0x7f4c500d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52560/0 (socket says 172.21.15.89:52560) 2024-04-02T08:52:41.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.203+0000 7f4c4fe00700 1 -- 172.21.15.89:0/3360451225 learned_addr learned my addr 172.21.15.89:0/3360451225 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:41.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.203+0000 7f4c55800700 1 --2- 172.21.15.89:0/3360451225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c500d2370 0x7f4c500ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:41.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.203+0000 7f4c4fe00700 1 -- 172.21.15.89:0/3360451225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c500d2370 msgr2=0x7f4c500ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:41.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.203+0000 7f4c4fe00700 1 --2- 172.21.15.89:0/3360451225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c500d2370 0x7f4c500ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:41.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.203+0000 7f4c4fe00700 1 -- 172.21.15.89:0/3360451225 >> v1:172.21.15.89:6789/0 conn(0x7f4c5015d9d0 legacy=0x7f4c500d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:41.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.203+0000 7f4c4fe00700 1 -- 172.21.15.89:0/3360451225 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c50161700 con 0x7f4c500d16a0 2024-04-02T08:52:41.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.204+0000 7f4c4fe00700 1 --2- 172.21.15.89:0/3360451225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c500d16a0 0x7f4c500d1e30 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c006fd0 tx=0x7f4c3c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a8e9a0b2651c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:41.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.204+0000 7f4c4f400700 1 -- 172.21.15.89:0/3360451225 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c005e30 con 0x7f4c500d16a0 2024-04-02T08:52:41.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.204+0000 7f4c4f400700 1 -- 172.21.15.89:0/3360451225 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3c004030 con 0x7f4c500d16a0 2024-04-02T08:52:41.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.204+0000 7f4c4f400700 1 -- 172.21.15.89:0/3360451225 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c0043e0 con 0x7f4c500d16a0 2024-04-02T08:52:41.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.204+0000 7f4c57800700 1 -- 172.21.15.89:0/3360451225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c500d16a0 msgr2=0x7f4c500d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:41.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.204+0000 7f4c57800700 1 --2- 172.21.15.89:0/3360451225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c500d16a0 0x7f4c500d1e30 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c006fd0 tx=0x7f4c3c009bd0 comp rx=0 tx=0).stop 2024-04-02T08:52:41.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.204+0000 7f4c57800700 1 -- 172.21.15.89:0/3360451225 shutdown_connections 2024-04-02T08:52:41.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.204+0000 7f4c57800700 1 --2- 172.21.15.89:0/3360451225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c500d2370 0x7f4c500ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:41.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.204+0000 7f4c57800700 1 --2- 172.21.15.89:0/3360451225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c500d16a0 0x7f4c500d1e30 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:41.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.204+0000 7f4c57800700 1 -- 172.21.15.89:0/3360451225 >> 172.21.15.89:0/3360451225 conn(0x7f4c5008aff0 msgr2=0x7f4c500d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:41.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.204+0000 7f4c57800700 1 -- 172.21.15.89:0/3360451225 shutdown_connections 2024-04-02T08:52:41.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.205+0000 7f4c57800700 1 -- 172.21.15.89:0/3360451225 wait complete. 2024-04-02T08:52:41.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.206+0000 7f4c57800700 1 Processor -- start 2024-04-02T08:52:41.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.206+0000 7f4c57800700 1 -- start start 2024-04-02T08:52:41.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.207+0000 7f4c57800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c500d16a0 0x7f4c50154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:41.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.207+0000 7f4c57800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c500d2370 0x7f4c500cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:41.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.207+0000 7f4c57800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c5015d9d0 0x7f4c500cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:41.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.207+0000 7f4c57800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c500cd5c0 con 0x7f4c500d16a0 2024-04-02T08:52:41.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.207+0000 7f4c57800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c500cd2e0 con 0x7f4c5015d9d0 2024-04-02T08:52:41.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.207+0000 7f4c57800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5015ef60 con 0x7f4c500d2370 2024-04-02T08:52:41.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.207+0000 7f4c54e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c500d16a0 0x7f4c50154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:41.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.207+0000 7f4c54e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c500d16a0 0x7f4c50154bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52570/0 (socket says 172.21.15.89:52570) 2024-04-02T08:52:41.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.207+0000 7f4c54e00700 1 -- 172.21.15.89:0/3359518717 learned_addr learned my addr 172.21.15.89:0/3359518717 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:41.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.208+0000 7f4c55800700 1 --2- 172.21.15.89:0/3359518717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c5015d9d0 0x7f4c500cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:41.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.208+0000 7f4c54e00700 1 -- 172.21.15.89:0/3359518717 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c500d2370 msgr2=0x7f4c500cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:41.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.208+0000 7f4c54e00700 1 --2- 172.21.15.89:0/3359518717 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c500d2370 0x7f4c500cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:41.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.208+0000 7f4c54e00700 1 -- 172.21.15.89:0/3359518717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c5015d9d0 msgr2=0x7f4c500cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:41.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.208+0000 7f4c54e00700 1 --2- 172.21.15.89:0/3359518717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c5015d9d0 0x7f4c500cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:41.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.208+0000 7f4c54e00700 1 -- 172.21.15.89:0/3359518717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c009500 con 0x7f4c500d16a0 2024-04-02T08:52:41.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.208+0000 7f4c54e00700 1 --2- 172.21.15.89:0/3359518717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c500d16a0 0x7f4c50154bd0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400c450 tx=0x7f4c4400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:41.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.208+0000 7f4c4cc00700 1 -- 172.21.15.89:0/3359518717 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c440052d0 con 0x7f4c500d16a0 2024-04-02T08:52:41.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.208+0000 7f4c4cc00700 1 -- 172.21.15.89:0/3359518717 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c44005430 con 0x7f4c500d16a0 2024-04-02T08:52:41.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.208+0000 7f4c57800700 1 -- 172.21.15.89:0/3359518717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5015f240 con 0x7f4c500d16a0 2024-04-02T08:52:41.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.208+0000 7f4c4cc00700 1 -- 172.21.15.89:0/3359518717 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c440038c0 con 0x7f4c500d16a0 2024-04-02T08:52:41.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.209+0000 7f4c57800700 1 -- 172.21.15.89:0/3359518717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c50155110 con 0x7f4c500d16a0 2024-04-02T08:52:41.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.210+0000 7f4c57800700 1 -- 172.21.15.89:0/3359518717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c500d1ca0 con 0x7f4c500d16a0 2024-04-02T08:52:41.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.211+0000 7f4c4cc00700 1 -- 172.21.15.89:0/3359518717 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4c44003a20 con 0x7f4c500d16a0 2024-04-02T08:52:41.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.212+0000 7f4c4cc00700 1 --2- 172.21.15.89:0/3359518717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4c3006adb0 0x7f4c3006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:41.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.212+0000 7f4c4cc00700 1 -- 172.21.15.89:0/3359518717 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5647+0+0 (secure 0 0 0) 0x7f4c440889b0 con 0x7f4c500d16a0 2024-04-02T08:52:41.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.212+0000 7f4c4fe00700 1 --2- 172.21.15.89:0/3359518717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4c3006adb0 0x7f4c3006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:41.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.213+0000 7f4c4fe00700 1 --2- 172.21.15.89:0/3359518717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4c3006adb0 0x7f4c3006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4c500d1ae0 tx=0x7f4c3c010000 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:41.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.216+0000 7f4c4cc00700 1 -- 172.21.15.89:0/3359518717 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c44053aa0 con 0x7f4c500d16a0 2024-04-02T08:52:41.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.419+0000 7f4c57800700 1 -- 172.21.15.89:0/3359518717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4c5015f760 con 0x7f4c500d16a0 2024-04-02T08:52:41.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.990+0000 7f4c4cc00700 1 -- 172.21.15.89:0/3359518717 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v170) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4c5015f760 con 0x7f4c500d16a0 2024-04-02T08:52:41.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.995+0000 7f4c57800700 1 -- 172.21.15.89:0/3359518717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4c3006adb0 msgr2=0x7f4c3006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:41.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.995+0000 7f4c57800700 1 --2- 172.21.15.89:0/3359518717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4c3006adb0 0x7f4c3006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4c500d1ae0 tx=0x7f4c3c010000 comp rx=0 tx=0).stop 2024-04-02T08:52:41.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.995+0000 7f4c57800700 1 -- 172.21.15.89:0/3359518717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c500d16a0 msgr2=0x7f4c50154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:41.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.995+0000 7f4c57800700 1 --2- 172.21.15.89:0/3359518717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c500d16a0 0x7f4c50154bd0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400c450 tx=0x7f4c4400c810 comp rx=0 tx=0).stop 2024-04-02T08:52:41.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.996+0000 7f4c57800700 1 -- 172.21.15.89:0/3359518717 shutdown_connections 2024-04-02T08:52:41.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.996+0000 7f4c57800700 1 --2- 172.21.15.89:0/3359518717 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f4c3006adb0 0x7f4c3006d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:41.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.996+0000 7f4c57800700 1 --2- 172.21.15.89:0/3359518717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c5015d9d0 0x7f4c500cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:41.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.996+0000 7f4c57800700 1 --2- 172.21.15.89:0/3359518717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c500d16a0 0x7f4c50154bd0 secure :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400c450 tx=0x7f4c4400c810 comp rx=0 tx=0).stop 2024-04-02T08:52:41.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.996+0000 7f4c57800700 1 --2- 172.21.15.89:0/3359518717 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c500d2370 0x7f4c500cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:41.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.996+0000 7f4c57800700 1 -- 172.21.15.89:0/3359518717 >> 172.21.15.89:0/3359518717 conn(0x7f4c5008aff0 msgr2=0x7f4c5015beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:41.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.997+0000 7f4c57800700 1 -- 172.21.15.89:0/3359518717 shutdown_connections 2024-04-02T08:52:41.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:41.998+0000 7f4c57800700 1 -- 172.21.15.89:0/3359518717 wait complete. 2024-04-02T08:52:42.002 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T08:52:42.026 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T08:52:42.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.222+0000 7f77b2600700 1 Processor -- start 2024-04-02T08:52:42.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.222+0000 7f77b2600700 1 -- start start 2024-04-02T08:52:42.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.223+0000 7f77b2600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f77ac0d2540 0x7f77ac1687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:42.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.223+0000 7f77b2600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ac161420 con 0x7f77ac0d16a0 2024-04-02T08:52:42.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.223+0000 7f77b2600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ac161700 con 0x7f77ac15d9d0 2024-04-02T08:52:42.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.223+0000 7f77b2600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ac0d4340 con 0x7f77ac0d2540 2024-04-02T08:52:42.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.223+0000 7f77abe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f77ac0d2540 0x7f77ac1687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:42.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.223+0000 7f77abe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f77ac0d2540 0x7f77ac1687b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52572/0 (socket says 172.21.15.89:52572) 2024-04-02T08:52:42.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.223+0000 7f77abe00700 1 -- 172.21.15.89:0/3666675330 learned_addr learned my addr 172.21.15.89:0/3666675330 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:42.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.224+0000 7f77abe00700 1 -- 172.21.15.89:0/3666675330 >> v1:172.21.15.113:6789/0 conn(0x7f77ac15d9d0 legacy=0x7f77ac0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:42.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.224+0000 7f77abe00700 1 -- 172.21.15.89:0/3666675330 >> v1:172.21.15.89:6789/0 conn(0x7f77ac0d16a0 legacy=0x7f77ac0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:42.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.224+0000 7f77abe00700 1 -- 172.21.15.89:0/3666675330 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ac1619e0 con 0x7f77ac0d2540 2024-04-02T08:52:42.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.224+0000 7f77aaa00700 1 -- 172.21.15.89:0/3666675330 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115616769 0 0) 0x7f77ac161420 con 0x7f77ac0d16a0 2024-04-02T08:52:42.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.224+0000 7f77abe00700 1 --2- 172.21.15.89:0/3666675330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0d2540 0x7f77ac1687b0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f77a4009e50 tx=0x7f77a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe9cb503ca21ae1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:42.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.224+0000 7f77aaa00700 1 -- 172.21.15.89:0/3666675330 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a400e070 con 0x7f77ac0d2540 2024-04-02T08:52:42.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.224+0000 7f77aaa00700 1 -- 172.21.15.89:0/3666675330 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a400b9c0 con 0x7f77ac0d2540 2024-04-02T08:52:42.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.224+0000 7f77aaa00700 1 -- 172.21.15.89:0/3666675330 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a400bd70 con 0x7f77ac0d2540 2024-04-02T08:52:42.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.225+0000 7f77b2600700 1 -- 172.21.15.89:0/3666675330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0d2540 msgr2=0x7f77ac1687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:42.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.225+0000 7f77b2600700 1 --2- 172.21.15.89:0/3666675330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0d2540 0x7f77ac1687b0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f77a4009e50 tx=0x7f77a400b330 comp rx=0 tx=0).stop 2024-04-02T08:52:42.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.225+0000 7f77b2600700 1 -- 172.21.15.89:0/3666675330 shutdown_connections 2024-04-02T08:52:42.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.225+0000 7f77b2600700 1 --2- 172.21.15.89:0/3666675330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0d2540 0x7f77ac1687b0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.225+0000 7f77b2600700 1 -- 172.21.15.89:0/3666675330 >> 172.21.15.89:0/3666675330 conn(0x7f77ac08aff0 msgr2=0x7f77ac0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:42.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.225+0000 7f77b2600700 1 -- 172.21.15.89:0/3666675330 shutdown_connections 2024-04-02T08:52:42.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.225+0000 7f77b2600700 1 -- 172.21.15.89:0/3666675330 wait complete. 2024-04-02T08:52:42.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.227+0000 7f77b2600700 1 Processor -- start 2024-04-02T08:52:42.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.227+0000 7f77b2600700 1 -- start start 2024-04-02T08:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77b2600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77ac0d16a0 0x7f77ac0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77b2600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0d2540 0x7f77ac0b89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77b2600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77ac15d9d0 0x7f77ac1f6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77b2600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac0cca20 con 0x7f77ac0d2540 2024-04-02T08:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77b2600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac0cbee0 con 0x7f77ac0d16a0 2024-04-02T08:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77b2600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac0b8ef0 con 0x7f77ac15d9d0 2024-04-02T08:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77a3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0d2540 0x7f77ac0b89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77a3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0d2540 0x7f77ac0b89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52576/0 (socket says 172.21.15.89:52576) 2024-04-02T08:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77a3e00700 1 -- 172.21.15.89:0/90121120 learned_addr learned my addr 172.21.15.89:0/90121120 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77ab400700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77ac0d16a0 0x7f77ac0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77abe00700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77ac15d9d0 0x7f77ac1f6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77a3e00700 1 -- 172.21.15.89:0/90121120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77ac15d9d0 msgr2=0x7f77ac1f6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77a3e00700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77ac15d9d0 0x7f77ac1f6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77a3e00700 1 -- 172.21.15.89:0/90121120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77ac0d16a0 msgr2=0x7f77ac0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:42.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77a3e00700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77ac0d16a0 0x7f77ac0cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77a3e00700 1 -- 172.21.15.89:0/90121120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a4009b30 con 0x7f77ac0d2540 2024-04-02T08:52:42.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.228+0000 7f77abe00700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77ac15d9d0 0x7f77ac1f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:42.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.229+0000 7f77a3e00700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0d2540 0x7f77ac0b89b0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f779400c550 tx=0x7f779400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:42.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.229+0000 7f77a3400700 1 -- 172.21.15.89:0/90121120 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77940054e0 con 0x7f77ac0d2540 2024-04-02T08:52:42.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.229+0000 7f77a3400700 1 -- 172.21.15.89:0/90121120 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7794003680 con 0x7f77ac0d2540 2024-04-02T08:52:42.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.229+0000 7f77a3400700 1 -- 172.21.15.89:0/90121120 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794003a30 con 0x7f77ac0d2540 2024-04-02T08:52:42.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.229+0000 7f77b2600700 1 -- 172.21.15.89:0/90121120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ac0cc180 con 0x7f77ac0d2540 2024-04-02T08:52:42.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.229+0000 7f77b2600700 1 -- 172.21.15.89:0/90121120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ac0cc740 con 0x7f77ac0d2540 2024-04-02T08:52:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.230+0000 7f77b2600700 1 -- 172.21.15.89:0/90121120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ac0d1ca0 con 0x7f77ac0d2540 2024-04-02T08:52:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.231+0000 7f77a3400700 1 -- 172.21.15.89:0/90121120 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7794003b90 con 0x7f77ac0d2540 2024-04-02T08:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.232+0000 7f77a3400700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f778806adb0 0x7f778806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.232+0000 7f77a3400700 1 -- 172.21.15.89:0/90121120 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7794088a40 con 0x7f77ac0d2540 2024-04-02T08:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.233+0000 7f77ab400700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f778806adb0 0x7f778806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.233+0000 7f77ab400700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f778806adb0 0x7f778806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f779c007fd0 tx=0x7f779c009040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:42.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.237+0000 7f77a3400700 1 -- 172.21.15.89:0/90121120 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f77940539c0 con 0x7f77ac0d2540 2024-04-02T08:52:42.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.483+0000 7f77b2600700 1 -- 172.21.15.89:0/90121120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f77ac067720 con 0x7f77ac0d2540 2024-04-02T08:52:42.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.489+0000 7f77a3400700 1 -- 172.21.15.89:0/90121120 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v77) v1 ==== 174+0+0 (secure 0 0 0) 0x7f7794017020 con 0x7f77ac0d2540 2024-04-02T08:52:42.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.494+0000 7f77b2600700 1 -- 172.21.15.89:0/90121120 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f778806adb0 msgr2=0x7f778806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:42.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.494+0000 7f77b2600700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f778806adb0 0x7f778806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f779c007fd0 tx=0x7f779c009040 comp rx=0 tx=0).stop 2024-04-02T08:52:42.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.494+0000 7f77b2600700 1 -- 172.21.15.89:0/90121120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0d2540 msgr2=0x7f77ac0b89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:42.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.494+0000 7f77b2600700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0d2540 0x7f77ac0b89b0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f779400c550 tx=0x7f779400c860 comp rx=0 tx=0).stop 2024-04-02T08:52:42.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.494+0000 7f77b2600700 1 -- 172.21.15.89:0/90121120 shutdown_connections 2024-04-02T08:52:42.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.494+0000 7f77b2600700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f778806adb0 0x7f778806d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.494+0000 7f77b2600700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77ac0d16a0 0x7f77ac0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.494+0000 7f77b2600700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f77ac15d9d0 0x7f77ac1f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.494+0000 7f77b2600700 1 --2- 172.21.15.89:0/90121120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77ac0d2540 0x7f77ac0b89b0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.494+0000 7f77b2600700 1 -- 172.21.15.89:0/90121120 >> 172.21.15.89:0/90121120 conn(0x7f77ac08aff0 msgr2=0x7f77ac15ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:42.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.494+0000 7f77b2600700 1 -- 172.21.15.89:0/90121120 shutdown_connections 2024-04-02T08:52:42.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.495+0000 7f77b2600700 1 -- 172.21.15.89:0/90121120 wait complete. 2024-04-02T08:52:42.502 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 20 and data pool 21 2024-04-02T08:52:42.530 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:52:42.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.722+0000 7f4582a00700 1 Processor -- start 2024-04-02T08:52:42.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.722+0000 7f4582a00700 1 -- start start 2024-04-02T08:52:42.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.723+0000 7f4582a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f457c15d640 0x7f457c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:42.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.723+0000 7f4582a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f457c0ceec0 0x7f457c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:42.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.723+0000 7f4582a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c0d30d0 con 0x7f457c161800 2024-04-02T08:52:42.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.723+0000 7f4582a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f457c161c90 con 0x7f457c15d640 2024-04-02T08:52:42.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.723+0000 7f4582a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457c0b6840 con 0x7f457c0ceec0 2024-04-02T08:52:42.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.724+0000 7f4580a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f457c161800 0x7f457c0cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42768/0 (socket says 172.21.15.89:42768) 2024-04-02T08:52:42.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.724+0000 7f4580a00700 1 -- 172.21.15.89:0/1734524044 learned_addr learned my addr 172.21.15.89:0/1734524044 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:42.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.724+0000 7f457b400700 1 --2- 172.21.15.89:0/1734524044 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f457c0ceec0 0x7f457c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:42.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.724+0000 7f457be00700 1 --2- 172.21.15.89:0/1734524044 >> v2:172.21.15.113:3300/0 conn(0x7f457c15d640 0x7f457c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:42.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.724+0000 7f457aa00700 1 -- 172.21.15.89:0/1734524044 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615550700 0 0) 0x7f457c0d30d0 con 0x7f457c161800 2024-04-02T08:52:42.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.724+0000 7f457aa00700 1 -- 172.21.15.89:0/1734524044 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45600035b0 con 0x7f457c161800 2024-04-02T08:52:42.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.724+0000 7f457b400700 1 -- 172.21.15.89:0/1734524044 >> v2:172.21.15.113:3300/0 conn(0x7f457c15d640 msgr2=0x7f457c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:42.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.724+0000 7f457b400700 1 --2- 172.21.15.89:0/1734524044 >> v2:172.21.15.113:3300/0 conn(0x7f457c15d640 0x7f457c0d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.724+0000 7f457aa00700 1 -- 172.21.15.89:0/1734524044 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 155759192 0 0) 0x7f45600035b0 con 0x7f457c161800 2024-04-02T08:52:42.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.724+0000 7f457b400700 1 -- 172.21.15.89:0/1734524044 >> v1:172.21.15.89:6789/0 conn(0x7f457c161800 legacy=0x7f457c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:42.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.724+0000 7f457b400700 1 -- 172.21.15.89:0/1734524044 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c0d2880 con 0x7f457c0ceec0 2024-04-02T08:52:42.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.724+0000 7f457be00700 1 --2- 172.21.15.89:0/1734524044 >> v2:172.21.15.113:3300/0 conn(0x7f457c15d640 0x7f457c0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:42.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.725+0000 7f457b400700 1 --2- 172.21.15.89:0/1734524044 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f457c0ceec0 0x7f457c1612c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4568009840 tx=0x7f4568009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bc50edd1d926523 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:42.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.725+0000 7f457aa00700 1 -- 172.21.15.89:0/1734524044 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45740019f0 con 0x7f457c0ceec0 2024-04-02T08:52:42.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.725+0000 7f457aa00700 1 -- 172.21.15.89:0/1734524044 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4568004030 con 0x7f457c0ceec0 2024-04-02T08:52:42.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.725+0000 7f457aa00700 1 -- 172.21.15.89:0/1734524044 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45680043c0 con 0x7f457c0ceec0 2024-04-02T08:52:42.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.726+0000 7f4582a00700 1 -- 172.21.15.89:0/1734524044 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f457c0ceec0 msgr2=0x7f457c1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:42.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.726+0000 7f4582a00700 1 --2- 172.21.15.89:0/1734524044 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f457c0ceec0 0x7f457c1612c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4568009840 tx=0x7f4568009b50 comp rx=0 tx=0).stop 2024-04-02T08:52:42.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.726+0000 7f4582a00700 1 -- 172.21.15.89:0/1734524044 shutdown_connections 2024-04-02T08:52:42.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.726+0000 7f4582a00700 1 --2- 172.21.15.89:0/1734524044 >> v2:172.21.15.113:3300/0 conn(0x7f457c15d640 0x7f457c0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.726+0000 7f4582a00700 1 --2- 172.21.15.89:0/1734524044 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f457c0ceec0 0x7f457c1612c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.726+0000 7f4582a00700 1 -- 172.21.15.89:0/1734524044 >> 172.21.15.89:0/1734524044 conn(0x7f457c08aff0 msgr2=0x7f457c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:42.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.726+0000 7f4582a00700 1 -- 172.21.15.89:0/1734524044 shutdown_connections 2024-04-02T08:52:42.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.726+0000 7f4582a00700 1 -- 172.21.15.89:0/1734524044 wait complete. 2024-04-02T08:52:42.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.727+0000 7f4582a00700 1 Processor -- start 2024-04-02T08:52:42.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.727+0000 7f4582a00700 1 -- start start 2024-04-02T08:52:42.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.728+0000 7f4582a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f457c0ceec0 0x7f457c0c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:42.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.728+0000 7f4582a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f457c15d640 0x7f457c0cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:42.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.728+0000 7f4582a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f457c161800 0x7f457c0cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:42.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.728+0000 7f4582a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c0c8440 con 0x7f457c15d640 2024-04-02T08:52:42.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.728+0000 7f4582a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c0c8720 con 0x7f457c0ceec0 2024-04-02T08:52:42.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.728+0000 7f4582a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457c0c8a00 con 0x7f457c161800 2024-04-02T08:52:42.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.729+0000 7f457b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f457c15d640 0x7f457c0cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:42.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.729+0000 7f457b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f457c15d640 0x7f457c0cc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52578/0 (socket says 172.21.15.89:52578) 2024-04-02T08:52:42.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.729+0000 7f457b400700 1 -- 172.21.15.89:0/3362097759 learned_addr learned my addr 172.21.15.89:0/3362097759 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:42.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.729+0000 7f4580a00700 1 --2- 172.21.15.89:0/3362097759 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f457c161800 0x7f457c0cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:42.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.729+0000 7f457be00700 1 --2- 172.21.15.89:0/3362097759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f457c0ceec0 0x7f457c0c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:42.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.729+0000 7f457b400700 1 -- 172.21.15.89:0/3362097759 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f457c161800 msgr2=0x7f457c0cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:42.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.729+0000 7f457b400700 1 --2- 172.21.15.89:0/3362097759 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f457c161800 0x7f457c0cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.729+0000 7f457b400700 1 -- 172.21.15.89:0/3362097759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f457c0ceec0 msgr2=0x7f457c0c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:42.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.729+0000 7f457b400700 1 --2- 172.21.15.89:0/3362097759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f457c0ceec0 0x7f457c0c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.729+0000 7f457b400700 1 -- 172.21.15.89:0/3362097759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456c0094a0 con 0x7f457c15d640 2024-04-02T08:52:42.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.729+0000 7f457b400700 1 --2- 172.21.15.89:0/3362097759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f457c15d640 0x7f457c0cc8e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f456800a010 tx=0x7f4568004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:42.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.729+0000 7f4573e00700 1 -- 172.21.15.89:0/3362097759 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568003c00 con 0x7f457c15d640 2024-04-02T08:52:42.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.730+0000 7f4573e00700 1 -- 172.21.15.89:0/3362097759 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456801f070 con 0x7f457c15d640 2024-04-02T08:52:42.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.730+0000 7f4582a00700 1 -- 172.21.15.89:0/3362097759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45680094a0 con 0x7f457c15d640 2024-04-02T08:52:42.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.730+0000 7f4573e00700 1 -- 172.21.15.89:0/3362097759 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568011630 con 0x7f457c15d640 2024-04-02T08:52:42.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.730+0000 7f4582a00700 1 -- 172.21.15.89:0/3362097759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457c0c8fc0 con 0x7f457c15d640 2024-04-02T08:52:42.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.731+0000 7f4582a00700 1 -- 172.21.15.89:0/3362097759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457c0c91b0 con 0x7f457c15d640 2024-04-02T08:52:42.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.732+0000 7f4573e00700 1 -- 172.21.15.89:0/3362097759 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4568021030 con 0x7f457c15d640 2024-04-02T08:52:42.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.732+0000 7f4573e00700 1 --2- 172.21.15.89:0/3362097759 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f455c06adb0 0x7f455c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:42.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.732+0000 7f4573e00700 1 -- 172.21.15.89:0/3362097759 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f456808a860 con 0x7f457c15d640 2024-04-02T08:52:42.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.733+0000 7f457be00700 1 --2- 172.21.15.89:0/3362097759 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f455c06adb0 0x7f455c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:42.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.733+0000 7f457be00700 1 --2- 172.21.15.89:0/3362097759 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f455c06adb0 0x7f455c06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f456c009d00 tx=0x7f456c008040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:42.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.737+0000 7f4573e00700 1 -- 172.21.15.89:0/3362097759 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4568055930 con 0x7f457c15d640 2024-04-02T08:52:42.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.940+0000 7f4582a00700 1 -- 172.21.15.89:0/3362097759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f457c067720 con 0x7f457c15d640 2024-04-02T08:52:42.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.941+0000 7f4573e00700 1 -- 172.21.15.89:0/3362097759 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+16767 (secure 0 0 0) 0x7f4568058f50 con 0x7f457c15d640 2024-04-02T08:52:42.940 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:42.940 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":171,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:52:42.484573+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-02T08:52:40.806770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-02T08:52:41.463828+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:52:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.943+0000 7f4582a00700 1 -- 172.21.15.89:0/3362097759 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f455c06adb0 msgr2=0x7f455c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.943+0000 7f4582a00700 1 --2- 172.21.15.89:0/3362097759 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f455c06adb0 0x7f455c06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f456c009d00 tx=0x7f456c008040 comp rx=0 tx=0).stop 2024-04-02T08:52:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.943+0000 7f4582a00700 1 -- 172.21.15.89:0/3362097759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f457c15d640 msgr2=0x7f457c0cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.943+0000 7f4582a00700 1 --2- 172.21.15.89:0/3362097759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f457c15d640 0x7f457c0cc8e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f456800a010 tx=0x7f4568004750 comp rx=0 tx=0).stop 2024-04-02T08:52:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.944+0000 7f4582a00700 1 -- 172.21.15.89:0/3362097759 shutdown_connections 2024-04-02T08:52:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.944+0000 7f4582a00700 1 --2- 172.21.15.89:0/3362097759 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f455c06adb0 0x7f455c06d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.944+0000 7f4582a00700 1 --2- 172.21.15.89:0/3362097759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f457c0ceec0 0x7f457c0c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.944+0000 7f4582a00700 1 --2- 172.21.15.89:0/3362097759 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f457c161800 0x7f457c0cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.944+0000 7f4582a00700 1 --2- 172.21.15.89:0/3362097759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f457c15d640 0x7f457c0cc8e0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:42.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.944+0000 7f4582a00700 1 -- 172.21.15.89:0/3362097759 >> 172.21.15.89:0/3362097759 conn(0x7f457c08aff0 msgr2=0x7f457c0b7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:42.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.944+0000 7f4582a00700 1 -- 172.21.15.89:0/3362097759 shutdown_connections 2024-04-02T08:52:42.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:42.944+0000 7f4582a00700 1 -- 172.21.15.89:0/3362097759 wait complete. 2024-04-02T08:52:42.960 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:52:43.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.163+0000 7f11ba200700 1 Processor -- start 2024-04-02T08:52:43.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.163+0000 7f11ba200700 1 -- start start 2024-04-02T08:52:43.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.164+0000 7f11ba200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f11b415d640 0x7f11b40d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:43.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.164+0000 7f11ba200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f11b40ceec0 0x7f11b41612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:43.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.164+0000 7f11ba200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b40d30d0 con 0x7f11b4161800 2024-04-02T08:52:43.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.165+0000 7f11ba200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b40b6840 con 0x7f11b40ceec0 2024-04-02T08:52:43.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.165+0000 7f11ba200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b4161c20 con 0x7f11b415d640 2024-04-02T08:52:43.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.165+0000 7f11b2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f11b40ceec0 0x7f11b41612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:43.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.165+0000 7f11b2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f11b40ceec0 0x7f11b41612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52584/0 (socket says 172.21.15.89:52584) 2024-04-02T08:52:43.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.165+0000 7f11b2a00700 1 -- 172.21.15.89:0/455110054 learned_addr learned my addr 172.21.15.89:0/455110054 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:43.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.165+0000 7f11b3400700 1 --2- 172.21.15.89:0/455110054 >> v2:172.21.15.113:3300/0 conn(0x7f11b415d640 0x7f11b40d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:43.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.165+0000 7f11b2a00700 1 -- 172.21.15.89:0/455110054 >> v2:172.21.15.113:3300/0 conn(0x7f11b415d640 msgr2=0x7f11b40d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:43.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.165+0000 7f11b2a00700 1 --2- 172.21.15.89:0/455110054 >> v2:172.21.15.113:3300/0 conn(0x7f11b415d640 0x7f11b40d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.165+0000 7f11b2a00700 1 -- 172.21.15.89:0/455110054 >> v1:172.21.15.113:6789/0 conn(0x7f11b4161800 legacy=0x7f11b40cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:43.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.165+0000 7f11b2a00700 1 -- 172.21.15.89:0/455110054 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b40d2880 con 0x7f11b40ceec0 2024-04-02T08:52:43.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.165+0000 7f11b2a00700 1 --2- 172.21.15.89:0/455110054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11b40ceec0 0x7f11b41612c0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f11b40b5c20 tx=0x7f11a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd1de98ef961532 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:43.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.166+0000 7f11b2000700 1 -- 172.21.15.89:0/455110054 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a000b870 con 0x7f11b40ceec0 2024-04-02T08:52:43.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.166+0000 7f11b2000700 1 -- 172.21.15.89:0/455110054 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a000b9d0 con 0x7f11b40ceec0 2024-04-02T08:52:43.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.166+0000 7f11b2000700 1 -- 172.21.15.89:0/455110054 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a000bd60 con 0x7f11b40ceec0 2024-04-02T08:52:43.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.166+0000 7f11ba200700 1 -- 172.21.15.89:0/455110054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11b40ceec0 msgr2=0x7f11b41612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:43.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.166+0000 7f11ba200700 1 --2- 172.21.15.89:0/455110054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11b40ceec0 0x7f11b41612c0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f11b40b5c20 tx=0x7f11a0009bd0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.166+0000 7f11ba200700 1 -- 172.21.15.89:0/455110054 shutdown_connections 2024-04-02T08:52:43.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.166+0000 7f11ba200700 1 --2- 172.21.15.89:0/455110054 >> v2:172.21.15.113:3300/0 conn(0x7f11b415d640 0x7f11b40d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.166+0000 7f11ba200700 1 --2- 172.21.15.89:0/455110054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11b40ceec0 0x7f11b41612c0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.166+0000 7f11ba200700 1 -- 172.21.15.89:0/455110054 >> 172.21.15.89:0/455110054 conn(0x7f11b408aff0 msgr2=0x7f11b40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:43.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.166+0000 7f11ba200700 1 -- 172.21.15.89:0/455110054 shutdown_connections 2024-04-02T08:52:43.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.166+0000 7f11ba200700 1 -- 172.21.15.89:0/455110054 wait complete. 2024-04-02T08:52:43.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.168+0000 7f11ba200700 1 Processor -- start 2024-04-02T08:52:43.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.168+0000 7f11ba200700 1 -- start start 2024-04-02T08:52:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.169+0000 7f11ba200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11b40ceec0 0x7f11b40c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.169+0000 7f11ba200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11b415d640 0x7f11b41597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.169+0000 7f11ba200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11b4161800 0x7f11b4159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.169+0000 7f11ba200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b41578d0 con 0x7f11b415d640 2024-04-02T08:52:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.169+0000 7f11ba200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b4157bb0 con 0x7f11b40ceec0 2024-04-02T08:52:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.169+0000 7f11ba200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b4157e90 con 0x7f11b4161800 2024-04-02T08:52:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.169+0000 7f11b2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11b415d640 0x7f11b41597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:43.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.169+0000 7f11b2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11b415d640 0x7f11b41597d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52596/0 (socket says 172.21.15.89:52596) 2024-04-02T08:52:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.169+0000 7f11b3400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11b40ceec0 0x7f11b40c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.169+0000 7f11b2a00700 1 -- 172.21.15.89:0/496634908 learned_addr learned my addr 172.21.15.89:0/496634908 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.170+0000 7f11b3e00700 1 --2- 172.21.15.89:0/496634908 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11b4161800 0x7f11b4159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.170+0000 7f11b2a00700 1 -- 172.21.15.89:0/496634908 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11b4161800 msgr2=0x7f11b4159d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.170+0000 7f11b2a00700 1 --2- 172.21.15.89:0/496634908 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11b4161800 0x7f11b4159d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.170+0000 7f11b2a00700 1 -- 172.21.15.89:0/496634908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11b40ceec0 msgr2=0x7f11b40c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.170+0000 7f11b2a00700 1 --2- 172.21.15.89:0/496634908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11b40ceec0 0x7f11b40c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.170+0000 7f11b2a00700 1 -- 172.21.15.89:0/496634908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a0009550 con 0x7f11b415d640 2024-04-02T08:52:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.170+0000 7f11b2a00700 1 --2- 172.21.15.89:0/496634908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11b415d640 0x7f11b41597d0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f11a0000c00 tx=0x7f11a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.170+0000 7f11ab400700 1 -- 172.21.15.89:0/496634908 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a0004520 con 0x7f11b415d640 2024-04-02T08:52:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.170+0000 7f11ab400700 1 -- 172.21.15.89:0/496634908 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a001e070 con 0x7f11b415d640 2024-04-02T08:52:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.170+0000 7f11ab400700 1 -- 172.21.15.89:0/496634908 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a00038a0 con 0x7f11b415d640 2024-04-02T08:52:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.170+0000 7f11ba200700 1 -- 172.21.15.89:0/496634908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b4158170 con 0x7f11b415d640 2024-04-02T08:52:43.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.171+0000 7f11ba200700 1 -- 172.21.15.89:0/496634908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b4155ca0 con 0x7f11b415d640 2024-04-02T08:52:43.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.172+0000 7f11ba200700 1 -- 172.21.15.89:0/496634908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b4158690 con 0x7f11b415d640 2024-04-02T08:52:43.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.172+0000 7f11ab400700 1 -- 172.21.15.89:0/496634908 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f11a0003a00 con 0x7f11b415d640 2024-04-02T08:52:43.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.173+0000 7f11ab400700 1 --2- 172.21.15.89:0/496634908 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f119406adb0 0x7f119406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:43.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.173+0000 7f11ab400700 1 -- 172.21.15.89:0/496634908 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f11a0020070 con 0x7f11b415d640 2024-04-02T08:52:43.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.173+0000 7f11b3400700 1 --2- 172.21.15.89:0/496634908 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f119406adb0 0x7f119406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:43.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.174+0000 7f11b3400700 1 --2- 172.21.15.89:0/496634908 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f119406adb0 0x7f119406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f11a4006b20 tx=0x7f11a4008040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:43.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.178+0000 7f11ab400700 1 -- 172.21.15.89:0/496634908 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f11b4158690 con 0x7f11b415d640 2024-04-02T08:52:43.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.380+0000 7f11ba200700 1 -- 172.21.15.89:0/496634908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11b4158690 con 0x7f11b415d640 2024-04-02T08:52:43.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.381+0000 7f11ab400700 1 -- 172.21.15.89:0/496634908 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+16767 (secure 0 0 0) 0x7f11b4158690 con 0x7f11b415d640 2024-04-02T08:52:43.380 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:43.380 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":171,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:52:42.484573+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-02T08:52:40.806770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-02T08:52:41.463828+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:52:43.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.384+0000 7f11ba200700 1 -- 172.21.15.89:0/496634908 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f119406adb0 msgr2=0x7f119406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:43.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.384+0000 7f11ba200700 1 --2- 172.21.15.89:0/496634908 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f119406adb0 0x7f119406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f11a4006b20 tx=0x7f11a4008040 comp rx=0 tx=0).stop 2024-04-02T08:52:43.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.384+0000 7f11ba200700 1 -- 172.21.15.89:0/496634908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11b415d640 msgr2=0x7f11b41597d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:43.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.384+0000 7f11ba200700 1 --2- 172.21.15.89:0/496634908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11b415d640 0x7f11b41597d0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f11a0000c00 tx=0x7f11a0004750 comp rx=0 tx=0).stop 2024-04-02T08:52:43.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.384+0000 7f11ba200700 1 -- 172.21.15.89:0/496634908 shutdown_connections 2024-04-02T08:52:43.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.384+0000 7f11ba200700 1 --2- 172.21.15.89:0/496634908 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f119406adb0 0x7f119406d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.384+0000 7f11ba200700 1 --2- 172.21.15.89:0/496634908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11b40ceec0 0x7f11b40c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.384+0000 7f11ba200700 1 --2- 172.21.15.89:0/496634908 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11b4161800 0x7f11b4159d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.384+0000 7f11ba200700 1 --2- 172.21.15.89:0/496634908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11b415d640 0x7f11b41597d0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.384+0000 7f11ba200700 1 -- 172.21.15.89:0/496634908 >> 172.21.15.89:0/496634908 conn(0x7f11b408aff0 msgr2=0x7f11b40b72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:43.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.384+0000 7f11ba200700 1 -- 172.21.15.89:0/496634908 shutdown_connections 2024-04-02T08:52:43.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.384+0000 7f11ba200700 1 -- 172.21.15.89:0/496634908 wait complete. 2024-04-02T08:52:43.401 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T08:52:43.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.596+0000 7fd6b4600700 1 Processor -- start 2024-04-02T08:52:43.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.597+0000 7fd6b4600700 1 -- start start 2024-04-02T08:52:43.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.598+0000 7fd6b4600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6ac0cd020 0x7fd6ac0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:43.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.598+0000 7fd6b4600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ac0cdae0 con 0x7fd6ac0b2070 2024-04-02T08:52:43.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.598+0000 7fd6b4600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ac0cddc0 con 0x7fd6ac0cbb30 2024-04-02T08:52:43.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.598+0000 7fd6b4600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac0ce0a0 con 0x7fd6ac0cd020 2024-04-02T08:52:43.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.598+0000 7fd6b2600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd6ac0b2070 0x7fd6ac06e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42776/0 (socket says 172.21.15.89:42776) 2024-04-02T08:52:43.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.598+0000 7fd6b2600700 1 -- 172.21.15.89:0/2622687382 learned_addr learned my addr 172.21.15.89:0/2622687382 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:43.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.598+0000 7fd6b1200700 1 --2- 172.21.15.89:0/2622687382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6ac0cd020 0x7fd6ac0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:43.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.598+0000 7fd6abe00700 1 -- 172.21.15.89:0/2622687382 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835637798 0 0) 0x7fd6ac0cdae0 con 0x7fd6ac0b2070 2024-04-02T08:52:43.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.598+0000 7fd6abe00700 1 -- 172.21.15.89:0/2622687382 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6940035b0 con 0x7fd6ac0b2070 2024-04-02T08:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6abe00700 1 -- 172.21.15.89:0/2622687382 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3064042702 0 0) 0x7fd6940035b0 con 0x7fd6ac0b2070 2024-04-02T08:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6abe00700 1 -- 172.21.15.89:0/2622687382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6ac0cd020 msgr2=0x7fd6ac0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6abe00700 1 --2- 172.21.15.89:0/2622687382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6ac0cd020 0x7fd6ac0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6abe00700 1 -- 172.21.15.89:0/2622687382 >> v1:172.21.15.113:6789/0 conn(0x7fd6ac0cbb30 legacy=0x7fd6ac0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6abe00700 1 -- 172.21.15.89:0/2622687382 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ac06dc20 con 0x7fd6ac0b2070 2024-04-02T08:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6abe00700 1 -- 172.21.15.89:0/2622687382 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fd6a40019f0 con 0x7fd6ac0b2070 2024-04-02T08:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6abe00700 1 -- 172.21.15.89:0/2622687382 >> v1:172.21.15.89:6789/0 conn(0x7fd6ac0b2070 legacy=0x7fd6ac06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6abe00700 1 --2- 172.21.15.89:0/2622687382 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd694003a30 0x7fd694003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6abe00700 1 --2- 172.21.15.89:0/2622687382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd694004850 0x7fd694004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:43.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6abe00700 1 --2- 172.21.15.89:0/2622687382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6ac0cd020 0x7fd694005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6abe00700 1 -- 172.21.15.89:0/2622687382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac0ce0a0 con 0x7fd694004850 2024-04-02T08:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6abe00700 1 -- 172.21.15.89:0/2622687382 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69400baf0 con 0x7fd694003a30 2024-04-02T08:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6abe00700 1 -- 172.21.15.89:0/2622687382 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69400bcb0 con 0x7fd6ac0cd020 2024-04-02T08:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6b1200700 1 --2- 172.21.15.89:0/2622687382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd694004850 0x7fd694004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6b1c00700 1 --2- 172.21.15.89:0/2622687382 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd694003a30 0x7fd694003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.599+0000 7fd6b2600700 1 --2- 172.21.15.89:0/2622687382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6ac0cd020 0x7fd694005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.600+0000 7fd6b1200700 1 -- 172.21.15.89:0/2622687382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6ac0cd020 msgr2=0x7fd694005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.600+0000 7fd6b1200700 1 --2- 172.21.15.89:0/2622687382 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6ac0cd020 0x7fd694005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.600+0000 7fd6b1200700 1 -- 172.21.15.89:0/2622687382 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd694003a30 msgr2=0x7fd694003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.600+0000 7fd6b1200700 1 --2- 172.21.15.89:0/2622687382 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd694003a30 0x7fd694003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.600+0000 7fd6b1200700 1 -- 172.21.15.89:0/2622687382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6940035b0 con 0x7fd694004850 2024-04-02T08:52:43.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.600+0000 7fd6b1200700 1 --2- 172.21.15.89:0/2622687382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd694004850 0x7fd694004cd0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fd698009b30 tx=0x7fd694005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60dbbc0a70bf247c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.600+0000 7fd6abe00700 1 -- 172.21.15.89:0/2622687382 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69800e6c0 con 0x7fd694004850 2024-04-02T08:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.600+0000 7fd6abe00700 1 -- 172.21.15.89:0/2622687382 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69800e820 con 0x7fd694004850 2024-04-02T08:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.600+0000 7fd6abe00700 1 -- 172.21.15.89:0/2622687382 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69800eb90 con 0x7fd694004850 2024-04-02T08:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.600+0000 7fd6b4600700 1 -- 172.21.15.89:0/2622687382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd694004850 msgr2=0x7fd694004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.600+0000 7fd6b4600700 1 --2- 172.21.15.89:0/2622687382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd694004850 0x7fd694004cd0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fd698009b30 tx=0x7fd694005880 comp rx=0 tx=0).stop 2024-04-02T08:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.600+0000 7fd6b2600700 1 -- 172.21.15.89:0/2622687382 reap_dead start 2024-04-02T08:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.601+0000 7fd6b4600700 1 -- 172.21.15.89:0/2622687382 shutdown_connections 2024-04-02T08:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.601+0000 7fd6b4600700 1 -- 172.21.15.89:0/2622687382 >> 172.21.15.89:0/2622687382 conn(0x7fd6ac0b0d70 msgr2=0x7fd6ac0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.601+0000 7fd6b4600700 1 -- 172.21.15.89:0/2622687382 shutdown_connections 2024-04-02T08:52:43.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.601+0000 7fd6b4600700 1 -- 172.21.15.89:0/2622687382 wait complete. 2024-04-02T08:52:43.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.602+0000 7fd6b4600700 1 Processor -- start 2024-04-02T08:52:43.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.602+0000 7fd6b4600700 1 -- start start 2024-04-02T08:52:43.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.603+0000 7fd6b4600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd694004850 0x7fd6ac0ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:43.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.603+0000 7fd6b4600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac0b0270 0x7fd6ac0aef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:43.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.603+0000 7fd6b4600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6ac1f0c80 0x7fd6ac15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:43.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.603+0000 7fd6b4600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac1f1bb0 con 0x7fd6ac0b0270 2024-04-02T08:52:43.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.603+0000 7fd6b4600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac0aea90 con 0x7fd694004850 2024-04-02T08:52:43.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.603+0000 7fd6b4600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac1f15f0 con 0x7fd6ac1f0c80 2024-04-02T08:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.604+0000 7fd6b1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac0b0270 0x7fd6ac0aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.604+0000 7fd6b1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac0b0270 0x7fd6ac0aef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52612/0 (socket says 172.21.15.89:52612) 2024-04-02T08:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.604+0000 7fd6b1200700 1 -- 172.21.15.89:0/532824222 learned_addr learned my addr 172.21.15.89:0/532824222 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.604+0000 7fd6b1c00700 1 --2- 172.21.15.89:0/532824222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd694004850 0x7fd6ac0ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.604+0000 7fd6b2600700 1 --2- 172.21.15.89:0/532824222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6ac1f0c80 0x7fd6ac15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.604+0000 7fd6b1200700 1 -- 172.21.15.89:0/532824222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6ac1f0c80 msgr2=0x7fd6ac15aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.604+0000 7fd6b1200700 1 --2- 172.21.15.89:0/532824222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6ac1f0c80 0x7fd6ac15aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.604+0000 7fd6b1200700 1 -- 172.21.15.89:0/532824222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd694004850 msgr2=0x7fd6ac0ad9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.604+0000 7fd6b1200700 1 --2- 172.21.15.89:0/532824222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd694004850 0x7fd6ac0ad9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.604+0000 7fd6b1200700 1 -- 172.21.15.89:0/532824222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ac1f18d0 con 0x7fd6ac0b0270 2024-04-02T08:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.604+0000 7fd6b1200700 1 --2- 172.21.15.89:0/532824222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac0b0270 0x7fd6ac0aef20 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fd69400bac0 tx=0x7fd6980047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.604+0000 7fd6a9600700 1 -- 172.21.15.89:0/532824222 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd698021070 con 0x7fd6ac0b0270 2024-04-02T08:52:43.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.605+0000 7fd6a9600700 1 -- 172.21.15.89:0/532824222 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd698004520 con 0x7fd6ac0b0270 2024-04-02T08:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.605+0000 7fd6a9600700 1 -- 172.21.15.89:0/532824222 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69800e6c0 con 0x7fd6ac0b0270 2024-04-02T08:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.605+0000 7fd6b4600700 1 -- 172.21.15.89:0/532824222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ac1f1bb0 con 0x7fd6ac0b0270 2024-04-02T08:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.605+0000 7fd6b4600700 1 -- 172.21.15.89:0/532824222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ac0af460 con 0x7fd6ac0b0270 2024-04-02T08:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.606+0000 7fd6b4600700 1 -- 172.21.15.89:0/532824222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ac0aec40 con 0x7fd6ac0b0270 2024-04-02T08:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.606+0000 7fd6a9600700 1 -- 172.21.15.89:0/532824222 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd698025070 con 0x7fd6ac0b0270 2024-04-02T08:52:43.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.607+0000 7fd6a9600700 1 --2- 172.21.15.89:0/532824222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd68c06adb0 0x7fd68c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:43.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.607+0000 7fd6a9600700 1 -- 172.21.15.89:0/532824222 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fd69808a670 con 0x7fd6ac0b0270 2024-04-02T08:52:43.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.608+0000 7fd6b1c00700 1 --2- 172.21.15.89:0/532824222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd68c06adb0 0x7fd68c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:43.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.608+0000 7fd6b1c00700 1 --2- 172.21.15.89:0/532824222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd68c06adb0 0x7fd68c06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0000c00 tx=0x7fd6a0009040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:43.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.612+0000 7fd6a9600700 1 -- 172.21.15.89:0/532824222 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd6ac0aec40 con 0x7fd6ac0b0270 2024-04-02T08:52:43.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:43.856+0000 7fd6b4600700 1 -- 172.21.15.89:0/532824222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd6ac0aec00 con 0x7fd6ac0b0270 2024-04-02T08:52:44.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.499+0000 7fd6a9600700 1 -- 172.21.15.89:0/532824222 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v80) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd698055620 con 0x7fd6ac0b0270 2024-04-02T08:52:44.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.502+0000 7fd6b4600700 1 -- 172.21.15.89:0/532824222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd68c06adb0 msgr2=0x7fd68c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:44.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.502+0000 7fd6b4600700 1 --2- 172.21.15.89:0/532824222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd68c06adb0 0x7fd68c06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0000c00 tx=0x7fd6a0009040 comp rx=0 tx=0).stop 2024-04-02T08:52:44.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.502+0000 7fd6b4600700 1 -- 172.21.15.89:0/532824222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac0b0270 msgr2=0x7fd6ac0aef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:44.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.502+0000 7fd6b4600700 1 --2- 172.21.15.89:0/532824222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac0b0270 0x7fd6ac0aef20 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fd69400bac0 tx=0x7fd6980047c0 comp rx=0 tx=0).stop 2024-04-02T08:52:44.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.502+0000 7fd6b4600700 1 -- 172.21.15.89:0/532824222 shutdown_connections 2024-04-02T08:52:44.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.502+0000 7fd6b4600700 1 --2- 172.21.15.89:0/532824222 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd68c06adb0 0x7fd68c06d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:44.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.502+0000 7fd6b4600700 1 --2- 172.21.15.89:0/532824222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd694004850 0x7fd6ac0ad9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:44.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.502+0000 7fd6b4600700 1 --2- 172.21.15.89:0/532824222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6ac1f0c80 0x7fd6ac15aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:44.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.502+0000 7fd6b4600700 1 --2- 172.21.15.89:0/532824222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6ac0b0270 0x7fd6ac0aef20 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:44.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.502+0000 7fd6b4600700 1 -- 172.21.15.89:0/532824222 >> 172.21.15.89:0/532824222 conn(0x7fd6ac0b0d70 msgr2=0x7fd6ac0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:44.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.502+0000 7fd6b4600700 1 -- 172.21.15.89:0/532824222 shutdown_connections 2024-04-02T08:52:44.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.502+0000 7fd6b4600700 1 -- 172.21.15.89:0/532824222 wait complete. 2024-04-02T08:52:44.521 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T08:52:44.521 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T08:52:44.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.718+0000 7f5ce0000700 1 Processor -- start 2024-04-02T08:52:44.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.718+0000 7f5ce0000700 1 -- start start 2024-04-02T08:52:44.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.719+0000 7f5ce0000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5cd81611b0 0x7f5cd815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:44.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.719+0000 7f5ce0000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cd815f4d0 0x7f5cd80ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:44.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.719+0000 7f5ce0000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd80ce270 con 0x7f5cd80d21f0 2024-04-02T08:52:44.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.719+0000 7f5ce0000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd815b600 con 0x7f5cd81611b0 2024-04-02T08:52:44.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.719+0000 7f5ce0000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80d2760 con 0x7f5cd815f4d0 2024-04-02T08:52:44.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.719+0000 7f5cdd600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5cd81611b0 0x7f5cd815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:44.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.719+0000 7f5cde000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5cd80d21f0 0x7f5cd8161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42778/0 (socket says 172.21.15.89:42778) 2024-04-02T08:52:44.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.719+0000 7f5cde000700 1 -- 172.21.15.89:0/928970435 learned_addr learned my addr 172.21.15.89:0/928970435 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:44.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.719+0000 7f5cdcc00700 1 --2- 172.21.15.89:0/928970435 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cd815f4d0 0x7f5cd80ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:44.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.720+0000 7f5cd7e00700 1 -- 172.21.15.89:0/928970435 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489070901 0 0) 0x7f5cd80ce270 con 0x7f5cd80d21f0 2024-04-02T08:52:44.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.720+0000 7f5cd7e00700 1 -- 172.21.15.89:0/928970435 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cc00035b0 con 0x7f5cd80d21f0 2024-04-02T08:52:44.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.720+0000 7f5cdd600700 1 -- 172.21.15.89:0/928970435 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cd815f4d0 msgr2=0x7f5cd80ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:44.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.720+0000 7f5cdd600700 1 --2- 172.21.15.89:0/928970435 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cd815f4d0 0x7f5cd80ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:44.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.720+0000 7f5cdd600700 1 -- 172.21.15.89:0/928970435 >> v1:172.21.15.89:6789/0 conn(0x7f5cd80d21f0 legacy=0x7f5cd8161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:44.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.720+0000 7f5cdd600700 1 -- 172.21.15.89:0/928970435 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd80d2a40 con 0x7f5cd81611b0 2024-04-02T08:52:44.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.720+0000 7f5cdcc00700 1 --2- 172.21.15.89:0/928970435 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cd815f4d0 0x7f5cd80ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:44.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.720+0000 7f5cdd600700 1 --2- 172.21.15.89:0/928970435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd81611b0 0x7f5cd815ef90 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc009840 tx=0x7f5ccc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5880f5b1933cf4b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:44.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.720+0000 7f5cd7e00700 1 -- 172.21.15.89:0/928970435 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc011830 con 0x7f5cd81611b0 2024-04-02T08:52:44.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.720+0000 7f5cd7e00700 1 -- 172.21.15.89:0/928970435 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc011990 con 0x7f5cd81611b0 2024-04-02T08:52:44.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.720+0000 7f5cd7e00700 1 -- 172.21.15.89:0/928970435 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc011d20 con 0x7f5cd81611b0 2024-04-02T08:52:44.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.721+0000 7f5ce0000700 1 -- 172.21.15.89:0/928970435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd81611b0 msgr2=0x7f5cd815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:44.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.721+0000 7f5ce0000700 1 --2- 172.21.15.89:0/928970435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd81611b0 0x7f5cd815ef90 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc009840 tx=0x7f5ccc009b50 comp rx=0 tx=0).stop 2024-04-02T08:52:44.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.721+0000 7f5ce0000700 1 -- 172.21.15.89:0/928970435 shutdown_connections 2024-04-02T08:52:44.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.721+0000 7f5ce0000700 1 --2- 172.21.15.89:0/928970435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd81611b0 0x7f5cd815ef90 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:44.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.721+0000 7f5ce0000700 1 --2- 172.21.15.89:0/928970435 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cd815f4d0 0x7f5cd80ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:44.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.721+0000 7f5ce0000700 1 -- 172.21.15.89:0/928970435 >> 172.21.15.89:0/928970435 conn(0x7f5cd808aff0 msgr2=0x7f5cd80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:44.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.721+0000 7f5ce0000700 1 -- 172.21.15.89:0/928970435 shutdown_connections 2024-04-02T08:52:44.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.721+0000 7f5ce0000700 1 -- 172.21.15.89:0/928970435 wait complete. 2024-04-02T08:52:44.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.723+0000 7f5ce0000700 1 Processor -- start 2024-04-02T08:52:44.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.723+0000 7f5ce0000700 1 -- start start 2024-04-02T08:52:44.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5ce0000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cd80d21f0 0x7f5cd8156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:44.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5ce0000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd815f4d0 0x7f5cd80b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:44.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5ce0000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cd81611b0 0x7f5cd815c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:44.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5ce0000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd8156b80 con 0x7f5cd815f4d0 2024-04-02T08:52:44.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5ce0000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80cbbc0 con 0x7f5cd80d21f0 2024-04-02T08:52:44.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5ce0000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80b7420 con 0x7f5cd81611b0 2024-04-02T08:52:44.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5cdcc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd815f4d0 0x7f5cd80b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:44.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5cdcc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd815f4d0 0x7f5cd80b6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52630/0 (socket says 172.21.15.89:52630) 2024-04-02T08:52:44.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5cdcc00700 1 -- 172.21.15.89:0/2297396393 learned_addr learned my addr 172.21.15.89:0/2297396393 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:44.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5cdd600700 1 --2- 172.21.15.89:0/2297396393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cd80d21f0 0x7f5cd8156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:44.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5cde000700 1 --2- 172.21.15.89:0/2297396393 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cd81611b0 0x7f5cd815c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:44.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5cdcc00700 1 -- 172.21.15.89:0/2297396393 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cd81611b0 msgr2=0x7f5cd815c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:44.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5cdcc00700 1 --2- 172.21.15.89:0/2297396393 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cd81611b0 0x7f5cd815c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:44.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5cdcc00700 1 -- 172.21.15.89:0/2297396393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cd80d21f0 msgr2=0x7f5cd8156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:44.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5cdcc00700 1 --2- 172.21.15.89:0/2297396393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cd80d21f0 0x7f5cd8156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:44.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.724+0000 7f5cdcc00700 1 -- 172.21.15.89:0/2297396393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc0094a0 con 0x7f5cd815f4d0 2024-04-02T08:52:44.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.725+0000 7f5cdcc00700 1 --2- 172.21.15.89:0/2297396393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd815f4d0 0x7f5cd80b6a40 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400e7e0 tx=0x7f5cc400eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:44.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.725+0000 7f5cd5600700 1 -- 172.21.15.89:0/2297396393 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4013610 con 0x7f5cd815f4d0 2024-04-02T08:52:44.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.725+0000 7f5cd5600700 1 -- 172.21.15.89:0/2297396393 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc4013770 con 0x7f5cd815f4d0 2024-04-02T08:52:44.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.725+0000 7f5cd5600700 1 -- 172.21.15.89:0/2297396393 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4013b00 con 0x7f5cd815f4d0 2024-04-02T08:52:44.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.725+0000 7f5ce0000700 1 -- 172.21.15.89:0/2297396393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd80b7700 con 0x7f5cd815f4d0 2024-04-02T08:52:44.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.725+0000 7f5ce0000700 1 -- 172.21.15.89:0/2297396393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd80b79e0 con 0x7f5cd815f4d0 2024-04-02T08:52:44.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.726+0000 7f5ce0000700 1 -- 172.21.15.89:0/2297396393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd80cc0a0 con 0x7f5cd815f4d0 2024-04-02T08:52:44.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.727+0000 7f5cd5600700 1 -- 172.21.15.89:0/2297396393 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5cc4013c60 con 0x7f5cd815f4d0 2024-04-02T08:52:44.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.728+0000 7f5cd5600700 1 --2- 172.21.15.89:0/2297396393 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5cb806adb0 0x7f5cb806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:44.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.728+0000 7f5cd5600700 1 -- 172.21.15.89:0/2297396393 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f5cc40893d0 con 0x7f5cd815f4d0 2024-04-02T08:52:44.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.728+0000 7f5cdd600700 1 --2- 172.21.15.89:0/2297396393 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5cb806adb0 0x7f5cb806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:44.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.728+0000 7f5cdd600700 1 --2- 172.21.15.89:0/2297396393 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5cb806adb0 0x7f5cb806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc000c00 tx=0x7f5ccc01c040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:44.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.732+0000 7f5cd5600700 1 -- 172.21.15.89:0/2297396393 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5cd80cc0a0 con 0x7f5cd815f4d0 2024-04-02T08:52:44.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:44.981+0000 7f5ce0000700 1 -- 172.21.15.89:0/2297396393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5cd80cc0a0 con 0x7f5cd815f4d0 2024-04-02T08:52:45.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.504+0000 7f5cd5600700 1 -- 172.21.15.89:0/2297396393 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v81) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5cd80cc0a0 con 0x7f5cd815f4d0 2024-04-02T08:52:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.506+0000 7f5ce0000700 1 -- 172.21.15.89:0/2297396393 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5cb806adb0 msgr2=0x7f5cb806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.506+0000 7f5ce0000700 1 --2- 172.21.15.89:0/2297396393 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5cb806adb0 0x7f5cb806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc000c00 tx=0x7f5ccc01c040 comp rx=0 tx=0).stop 2024-04-02T08:52:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.506+0000 7f5ce0000700 1 -- 172.21.15.89:0/2297396393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd815f4d0 msgr2=0x7f5cd80b6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.506+0000 7f5ce0000700 1 --2- 172.21.15.89:0/2297396393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd815f4d0 0x7f5cd80b6a40 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400e7e0 tx=0x7f5cc400eaf0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.506+0000 7f5ce0000700 1 -- 172.21.15.89:0/2297396393 shutdown_connections 2024-04-02T08:52:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.506+0000 7f5ce0000700 1 --2- 172.21.15.89:0/2297396393 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5cb806adb0 0x7f5cb806d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.506+0000 7f5ce0000700 1 --2- 172.21.15.89:0/2297396393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cd80d21f0 0x7f5cd8156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.506+0000 7f5ce0000700 1 --2- 172.21.15.89:0/2297396393 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cd81611b0 0x7f5cd815c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.506+0000 7f5ce0000700 1 --2- 172.21.15.89:0/2297396393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd815f4d0 0x7f5cd80b6a40 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.506+0000 7f5ce0000700 1 -- 172.21.15.89:0/2297396393 >> 172.21.15.89:0/2297396393 conn(0x7f5cd808aff0 msgr2=0x7f5cd815fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.507+0000 7f5ce0000700 1 -- 172.21.15.89:0/2297396393 shutdown_connections 2024-04-02T08:52:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.507+0000 7f5ce0000700 1 -- 172.21.15.89:0/2297396393 wait complete. 2024-04-02T08:52:45.525 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T08:52:45.525 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T08:52:45.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.717+0000 7fd913c00700 1 Processor -- start 2024-04-02T08:52:45.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.718+0000 7fd913c00700 1 -- start start 2024-04-02T08:52:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.719+0000 7fd913c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd90c0d18b0 0x7fd90c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.719+0000 7fd913c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd90c161590 0x7fd90c161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.719+0000 7fd913c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c0ceda0 con 0x7fd90c160f10 2024-04-02T08:52:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.719+0000 7fd913c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90c0d1d30 con 0x7fd90c0d18b0 2024-04-02T08:52:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.719+0000 7fd913c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90c161f60 con 0x7fd90c161590 2024-04-02T08:52:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.720+0000 7fd90be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd90c0d18b0 0x7fd90c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.720+0000 7fd90be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd90c0d18b0 0x7fd90c0cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52632/0 (socket says 172.21.15.89:52632) 2024-04-02T08:52:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.720+0000 7fd90be00700 1 -- 172.21.15.89:0/998015843 learned_addr learned my addr 172.21.15.89:0/998015843 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.720+0000 7fd911c00700 1 --2- 172.21.15.89:0/998015843 >> v2:172.21.15.113:3300/0 conn(0x7fd90c161590 0x7fd90c161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.720+0000 7fd90be00700 1 -- 172.21.15.89:0/998015843 >> v2:172.21.15.113:3300/0 conn(0x7fd90c161590 msgr2=0x7fd90c161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.720+0000 7fd90be00700 1 --2- 172.21.15.89:0/998015843 >> v2:172.21.15.113:3300/0 conn(0x7fd90c161590 0x7fd90c161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.720+0000 7fd90be00700 1 -- 172.21.15.89:0/998015843 >> v1:172.21.15.113:6789/0 conn(0x7fd90c160f10 legacy=0x7fd90c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:45.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.720+0000 7fd90be00700 1 -- 172.21.15.89:0/998015843 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c1612f0 con 0x7fd90c0d18b0 2024-04-02T08:52:45.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.720+0000 7fd90be00700 1 --2- 172.21.15.89:0/998015843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0d18b0 0x7fd90c0cf230 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0d1030 tx=0x7fd8f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cb3894ccb298d5b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.721+0000 7fd90b400700 1 -- 172.21.15.89:0/998015843 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f800b870 con 0x7fd90c0d18b0 2024-04-02T08:52:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.721+0000 7fd90b400700 1 -- 172.21.15.89:0/998015843 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f800b9d0 con 0x7fd90c0d18b0 2024-04-02T08:52:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.721+0000 7fd90b400700 1 -- 172.21.15.89:0/998015843 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f800bd60 con 0x7fd90c0d18b0 2024-04-02T08:52:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.721+0000 7fd913c00700 1 -- 172.21.15.89:0/998015843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0d18b0 msgr2=0x7fd90c0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.721+0000 7fd913c00700 1 --2- 172.21.15.89:0/998015843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0d18b0 0x7fd90c0cf230 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0d1030 tx=0x7fd8f8009bd0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.721+0000 7fd913c00700 1 -- 172.21.15.89:0/998015843 shutdown_connections 2024-04-02T08:52:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.721+0000 7fd913c00700 1 --2- 172.21.15.89:0/998015843 >> v2:172.21.15.113:3300/0 conn(0x7fd90c161590 0x7fd90c161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.721+0000 7fd913c00700 1 --2- 172.21.15.89:0/998015843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c0d18b0 0x7fd90c0cf230 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.721+0000 7fd913c00700 1 -- 172.21.15.89:0/998015843 >> 172.21.15.89:0/998015843 conn(0x7fd90c08aff0 msgr2=0x7fd90c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.721+0000 7fd913c00700 1 -- 172.21.15.89:0/998015843 shutdown_connections 2024-04-02T08:52:45.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.721+0000 7fd913c00700 1 -- 172.21.15.89:0/998015843 wait complete. 2024-04-02T08:52:45.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.723+0000 7fd913c00700 1 Processor -- start 2024-04-02T08:52:45.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.723+0000 7fd913c00700 1 -- start start 2024-04-02T08:52:45.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.724+0000 7fd913c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd90c0d18b0 0x7fd90c09f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:45.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.724+0000 7fd913c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c160f10 0x7fd90c15bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:45.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.724+0000 7fd913c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c161590 0x7fd90c160130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:45.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.724+0000 7fd913c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0a0790 con 0x7fd90c161590 2024-04-02T08:52:45.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.724+0000 7fd913c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c15c070 con 0x7fd90c0d18b0 2024-04-02T08:52:45.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.724+0000 7fd913c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c09fb10 con 0x7fd90c160f10 2024-04-02T08:52:45.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.724+0000 7fd911c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c161590 0x7fd90c160130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:45.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.724+0000 7fd911c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c161590 0x7fd90c160130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52646/0 (socket says 172.21.15.89:52646) 2024-04-02T08:52:45.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.724+0000 7fd911c00700 1 -- 172.21.15.89:0/2665059061 learned_addr learned my addr 172.21.15.89:0/2665059061 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.724+0000 7fd90be00700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c160f10 0x7fd90c15bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.724+0000 7fd911200700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd90c0d18b0 0x7fd90c09f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.725+0000 7fd911c00700 1 -- 172.21.15.89:0/2665059061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c160f10 msgr2=0x7fd90c15bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.725+0000 7fd911c00700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c160f10 0x7fd90c15bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.725+0000 7fd911c00700 1 -- 172.21.15.89:0/2665059061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd90c0d18b0 msgr2=0x7fd90c09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.725+0000 7fd911c00700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd90c0d18b0 0x7fd90c09f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.725+0000 7fd911c00700 1 -- 172.21.15.89:0/2665059061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f8009550 con 0x7fd90c161590 2024-04-02T08:52:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.725+0000 7fd911200700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd90c0d18b0 0x7fd90c09f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.725+0000 7fd911c00700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c161590 0x7fd90c160130 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fd904007e60 tx=0x7fd90400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.725+0000 7fd908c00700 1 -- 172.21.15.89:0/2665059061 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90400ede0 con 0x7fd90c161590 2024-04-02T08:52:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.725+0000 7fd908c00700 1 -- 172.21.15.89:0/2665059061 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd904014420 con 0x7fd90c161590 2024-04-02T08:52:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.725+0000 7fd908c00700 1 -- 172.21.15.89:0/2665059061 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9040147d0 con 0x7fd90c161590 2024-04-02T08:52:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.725+0000 7fd913c00700 1 -- 172.21.15.89:0/2665059061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90c09fdf0 con 0x7fd90c161590 2024-04-02T08:52:45.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.725+0000 7fd913c00700 1 -- 172.21.15.89:0/2665059061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c0a03b0 con 0x7fd90c161590 2024-04-02T08:52:45.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.726+0000 7fd913c00700 1 -- 172.21.15.89:0/2665059061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90c0a08d0 con 0x7fd90c161590 2024-04-02T08:52:45.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.727+0000 7fd908c00700 1 -- 172.21.15.89:0/2665059061 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd904014930 con 0x7fd90c161590 2024-04-02T08:52:45.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.728+0000 7fd908c00700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd8ec06adb0 0x7fd8ec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:45.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.728+0000 7fd908c00700 1 -- 172.21.15.89:0/2665059061 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fd904089dd0 con 0x7fd90c161590 2024-04-02T08:52:45.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.728+0000 7fd911200700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd8ec06adb0 0x7fd8ec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:45.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.729+0000 7fd911200700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd8ec06adb0 0x7fd8ec06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0ca330 tx=0x7fd900009040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:45.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.732+0000 7fd908c00700 1 -- 172.21.15.89:0/2665059061 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd90c0a08d0 con 0x7fd90c161590 2024-04-02T08:52:45.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.977+0000 7fd913c00700 1 -- 172.21.15.89:0/2665059061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd90c0a08d0 con 0x7fd90c161590 2024-04-02T08:52:45.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.978+0000 7fd908c00700 1 -- 172.21.15.89:0/2665059061 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4738 (secure 0 0 0) 0x7fd90c0a08d0 con 0x7fd90c161590 2024-04-02T08:52:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:45.977 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8881,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/547244866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":547244866},{"type":"v1","addr":"172.21.15.113:6834","nonce":547244866}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":8884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2822200553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2822200553},{"type":"v1","addr":"172.21.15.113:6835","nonce":2822200553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/342426898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":342426898},{"type":"v1","addr":"172.21.15.89:6835","nonce":342426898}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T08:52:42.484905+0000","modified":"2024-04-02T08:52:45.498450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9534},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9534":{"gid":9534,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2812511535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2812511535},{"type":"v1","addr":"172.21.15.89:6837","nonce":2812511535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-02T08:52:45.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.980+0000 7fd913c00700 1 -- 172.21.15.89:0/2665059061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd8ec06adb0 msgr2=0x7fd8ec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:45.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.980+0000 7fd913c00700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd8ec06adb0 0x7fd8ec06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0ca330 tx=0x7fd900009040 comp rx=0 tx=0).stop 2024-04-02T08:52:45.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.980+0000 7fd913c00700 1 -- 172.21.15.89:0/2665059061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c161590 msgr2=0x7fd90c160130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:45.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.980+0000 7fd913c00700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c161590 0x7fd90c160130 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fd904007e60 tx=0x7fd90400e3c0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.981+0000 7fd913c00700 1 -- 172.21.15.89:0/2665059061 shutdown_connections 2024-04-02T08:52:45.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.981+0000 7fd913c00700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd8ec06adb0 0x7fd8ec06d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.981+0000 7fd913c00700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd90c0d18b0 0x7fd90c09f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.981+0000 7fd913c00700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd90c161590 0x7fd90c160130 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.981+0000 7fd913c00700 1 --2- 172.21.15.89:0/2665059061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c160f10 0x7fd90c15bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:45.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.981+0000 7fd913c00700 1 -- 172.21.15.89:0/2665059061 >> 172.21.15.89:0/2665059061 conn(0x7fd90c08aff0 msgr2=0x7fd90c0cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:45.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.981+0000 7fd913c00700 1 -- 172.21.15.89:0/2665059061 shutdown_connections 2024-04-02T08:52:45.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:45.981+0000 7fd913c00700 1 -- 172.21.15.89:0/2665059061 wait complete. 2024-04-02T08:52:45.980 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 81 2024-04-02T08:52:45.998 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:52:46.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.193+0000 7f2b97c00700 1 Processor -- start 2024-04-02T08:52:46.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.193+0000 7f2b97c00700 1 -- start start 2024-04-02T08:52:46.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.194+0000 7f2b97c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2b9015d640 0x7f2b900d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:46.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.194+0000 7f2b97c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2b901619d0 0x7f2b900b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:46.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.194+0000 7f2b97c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b900d30d0 con 0x7f2b900ceec0 2024-04-02T08:52:46.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.194+0000 7f2b97c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b900b6840 con 0x7f2b901619d0 2024-04-02T08:52:46.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.194+0000 7f2b97c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b900d2880 con 0x7f2b9015d640 2024-04-02T08:52:46.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.194+0000 7f2b8fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2b900ceec0 0x7f2b901612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42792/0 (socket says 172.21.15.89:42792) 2024-04-02T08:52:46.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.194+0000 7f2b8fe00700 1 -- 172.21.15.89:0/1029182858 learned_addr learned my addr 172.21.15.89:0/1029182858 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:46.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.194+0000 7f2b95c00700 1 --2- 172.21.15.89:0/1029182858 >> v2:172.21.15.89:3300/0 conn(0x7f2b901619d0 0x7f2b900b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:46.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.195+0000 7f2b95200700 1 --2- 172.21.15.89:0/1029182858 >> v2:172.21.15.113:3300/0 conn(0x7f2b9015d640 0x7f2b900d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:46.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.195+0000 7f2b8f400700 1 -- 172.21.15.89:0/1029182858 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641934290 0 0) 0x7f2b900d30d0 con 0x7f2b900ceec0 2024-04-02T08:52:46.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.195+0000 7f2b8f400700 1 -- 172.21.15.89:0/1029182858 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b780035b0 con 0x7f2b900ceec0 2024-04-02T08:52:46.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.195+0000 7f2b95c00700 1 -- 172.21.15.89:0/1029182858 >> v2:172.21.15.113:3300/0 conn(0x7f2b9015d640 msgr2=0x7f2b900d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:46.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.195+0000 7f2b95c00700 1 --2- 172.21.15.89:0/1029182858 >> v2:172.21.15.113:3300/0 conn(0x7f2b9015d640 0x7f2b900d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.195+0000 7f2b95c00700 1 -- 172.21.15.89:0/1029182858 >> v1:172.21.15.89:6789/0 conn(0x7f2b900ceec0 legacy=0x7f2b901612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:46.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.195+0000 7f2b95c00700 1 -- 172.21.15.89:0/1029182858 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b900d2b30 con 0x7f2b901619d0 2024-04-02T08:52:46.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.195+0000 7f2b95c00700 1 --2- 172.21.15.89:0/1029182858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b901619d0 0x7f2b900b6cd0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f2b900b5c20 tx=0x7f2b8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b7c1ca8f4a6507 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:46.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.195+0000 7f2b8f400700 1 -- 172.21.15.89:0/1029182858 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8800e070 con 0x7f2b901619d0 2024-04-02T08:52:46.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.195+0000 7f2b8f400700 1 -- 172.21.15.89:0/1029182858 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8800bb20 con 0x7f2b901619d0 2024-04-02T08:52:46.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.195+0000 7f2b8f400700 1 -- 172.21.15.89:0/1029182858 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b88012430 con 0x7f2b901619d0 2024-04-02T08:52:46.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.196+0000 7f2b97c00700 1 -- 172.21.15.89:0/1029182858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b901619d0 msgr2=0x7f2b900b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:46.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.196+0000 7f2b97c00700 1 --2- 172.21.15.89:0/1029182858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b901619d0 0x7f2b900b6cd0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f2b900b5c20 tx=0x7f2b8800b330 comp rx=0 tx=0).stop 2024-04-02T08:52:46.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.196+0000 7f2b97c00700 1 -- 172.21.15.89:0/1029182858 shutdown_connections 2024-04-02T08:52:46.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.196+0000 7f2b97c00700 1 --2- 172.21.15.89:0/1029182858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b901619d0 0x7f2b900b6cd0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.196+0000 7f2b97c00700 1 --2- 172.21.15.89:0/1029182858 >> v2:172.21.15.113:3300/0 conn(0x7f2b9015d640 0x7f2b900d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.196+0000 7f2b97c00700 1 -- 172.21.15.89:0/1029182858 >> 172.21.15.89:0/1029182858 conn(0x7f2b9008aff0 msgr2=0x7f2b900d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:46.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.196+0000 7f2b97c00700 1 -- 172.21.15.89:0/1029182858 shutdown_connections 2024-04-02T08:52:46.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.196+0000 7f2b97c00700 1 -- 172.21.15.89:0/1029182858 wait complete. 2024-04-02T08:52:46.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.198+0000 7f2b97c00700 1 Processor -- start 2024-04-02T08:52:46.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.198+0000 7f2b97c00700 1 -- start start 2024-04-02T08:52:46.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b97c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b900ceec0 0x7f2b90152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:46.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b97c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b9015d640 0x7f2b901600d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:46.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b97c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b901619d0 0x7f2b90160610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:46.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b97c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b90155700 con 0x7f2b900ceec0 2024-04-02T08:52:46.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b97c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b90152c00 con 0x7f2b901619d0 2024-04-02T08:52:46.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b97c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b90154d60 con 0x7f2b9015d640 2024-04-02T08:52:46.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b95200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b900ceec0 0x7f2b90152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:46.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b95200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b900ceec0 0x7f2b90152540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52664/0 (socket says 172.21.15.89:52664) 2024-04-02T08:52:46.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b95200700 1 -- 172.21.15.89:0/3757836891 learned_addr learned my addr 172.21.15.89:0/3757836891 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:46.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b8fe00700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b9015d640 0x7f2b901600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:46.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b95c00700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b901619d0 0x7f2b90160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:46.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b95200700 1 -- 172.21.15.89:0/3757836891 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b9015d640 msgr2=0x7f2b901600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:46.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b95200700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b9015d640 0x7f2b901600d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b95200700 1 -- 172.21.15.89:0/3757836891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b901619d0 msgr2=0x7f2b90160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:46.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b95200700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b901619d0 0x7f2b90160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b95200700 1 -- 172.21.15.89:0/3757836891 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b88009a70 con 0x7f2b900ceec0 2024-04-02T08:52:46.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.199+0000 7f2b95c00700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b901619d0 0x7f2b90160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:52:46.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.200+0000 7f2b95200700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b900ceec0 0x7f2b90152540 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400d800 tx=0x7f2b8400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:46.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.200+0000 7f2b8cc00700 1 -- 172.21.15.89:0/3757836891 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b840125f0 con 0x7f2b900ceec0 2024-04-02T08:52:46.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.200+0000 7f2b8cc00700 1 -- 172.21.15.89:0/3757836891 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b84012750 con 0x7f2b900ceec0 2024-04-02T08:52:46.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.200+0000 7f2b8cc00700 1 -- 172.21.15.89:0/3757836891 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b84012b00 con 0x7f2b900ceec0 2024-04-02T08:52:46.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.200+0000 7f2b97c00700 1 -- 172.21.15.89:0/3757836891 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b90155040 con 0x7f2b900ceec0 2024-04-02T08:52:46.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.200+0000 7f2b97c00700 1 -- 172.21.15.89:0/3757836891 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b90160b50 con 0x7f2b900ceec0 2024-04-02T08:52:46.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.201+0000 7f2b97c00700 1 -- 172.21.15.89:0/3757836891 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b90067720 con 0x7f2b900ceec0 2024-04-02T08:52:46.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.202+0000 7f2b8cc00700 1 -- 172.21.15.89:0/3757836891 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2b84012c60 con 0x7f2b900ceec0 2024-04-02T08:52:46.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.203+0000 7f2b8cc00700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2b7006adb0 0x7f2b7006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:46.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.203+0000 7f2b8cc00700 1 -- 172.21.15.89:0/3757836891 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f2b84089940 con 0x7f2b900ceec0 2024-04-02T08:52:46.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.203+0000 7f2b8fe00700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2b7006adb0 0x7f2b7006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:46.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.204+0000 7f2b8fe00700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2b7006adb0 0x7f2b7006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0011e0 tx=0x7f2b7c009040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:46.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.207+0000 7f2b8cc00700 1 -- 172.21.15.89:0/3757836891 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2b840548f0 con 0x7f2b900ceec0 2024-04-02T08:52:46.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.412+0000 7f2b97c00700 1 -- 172.21.15.89:0/3757836891 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b90160ce0 con 0x7f2b900ceec0 2024-04-02T08:52:46.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.413+0000 7f2b8cc00700 1 -- 172.21.15.89:0/3757836891 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+16754 (secure 0 0 0) 0x7f2b90160ce0 con 0x7f2b900ceec0 2024-04-02T08:52:46.412 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:46.412 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":172,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:52:43.487697+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-02T08:52:40.806770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-02T08:52:41.463828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:52:46.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.416+0000 7f2b97c00700 1 -- 172.21.15.89:0/3757836891 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2b7006adb0 msgr2=0x7f2b7006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:46.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.416+0000 7f2b97c00700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2b7006adb0 0x7f2b7006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0011e0 tx=0x7f2b7c009040 comp rx=0 tx=0).stop 2024-04-02T08:52:46.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.416+0000 7f2b97c00700 1 -- 172.21.15.89:0/3757836891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b900ceec0 msgr2=0x7f2b90152540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:46.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.416+0000 7f2b97c00700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b900ceec0 0x7f2b90152540 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400d800 tx=0x7f2b8400dbc0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.416+0000 7f2b97c00700 1 -- 172.21.15.89:0/3757836891 shutdown_connections 2024-04-02T08:52:46.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.416+0000 7f2b97c00700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f2b7006adb0 0x7f2b7006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.416+0000 7f2b97c00700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b901619d0 0x7f2b90160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.416+0000 7f2b97c00700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b900ceec0 0x7f2b90152540 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.416+0000 7f2b97c00700 1 --2- 172.21.15.89:0/3757836891 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b9015d640 0x7f2b901600d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.416+0000 7f2b97c00700 1 -- 172.21.15.89:0/3757836891 >> 172.21.15.89:0/3757836891 conn(0x7f2b9008aff0 msgr2=0x7f2b900d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:46.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.416+0000 7f2b97c00700 1 -- 172.21.15.89:0/3757836891 shutdown_connections 2024-04-02T08:52:46.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.416+0000 7f2b97c00700 1 -- 172.21.15.89:0/3757836891 wait complete. 2024-04-02T08:52:46.433 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T08:52:46.433 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T08:52:46.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.626+0000 7efec6c00700 1 Processor -- start 2024-04-02T08:52:46.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.627+0000 7efec6c00700 1 -- start start 2024-04-02T08:52:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.628+0000 7efec6c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efec00d05e0 0x7efec00d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.628+0000 7efec6c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efec00b6990 0x7efec00b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.628+0000 7efec6c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec00d0f80 con 0x7efec015da50 2024-04-02T08:52:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.628+0000 7efec6c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec00b95d0 con 0x7efec00b6990 2024-04-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.628+0000 7efec6c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec015de30 con 0x7efec00d05e0 2024-04-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.628+0000 7efec4c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efec00b6990 0x7efec00b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.628+0000 7efec4c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efec00b6990 0x7efec00b6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52674/0 (socket says 172.21.15.89:52674) 2024-04-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.628+0000 7efec4c00700 1 -- 172.21.15.89:0/2818372411 learned_addr learned my addr 172.21.15.89:0/2818372411 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.628+0000 7efebf400700 1 --2- 172.21.15.89:0/2818372411 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efec00d05e0 0x7efec00d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.628+0000 7efec4c00700 1 -- 172.21.15.89:0/2818372411 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efec00d05e0 msgr2=0x7efec00d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.628+0000 7efec4c00700 1 --2- 172.21.15.89:0/2818372411 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efec00d05e0 0x7efec00d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.628+0000 7efec4c00700 1 -- 172.21.15.89:0/2818372411 >> v1:172.21.15.113:6789/0 conn(0x7efec015da50 legacy=0x7efec00ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.628+0000 7efec4c00700 1 -- 172.21.15.89:0/2818372411 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec00d24e0 con 0x7efec00b6990 2024-04-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.629+0000 7efec4c00700 1 --2- 172.21.15.89:0/2818372411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efec00b6990 0x7efec00b6e10 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7efeb8009e50 tx=0x7efeb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd82b8cfec78716 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.629+0000 7efebea00700 1 -- 172.21.15.89:0/2818372411 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb800e070 con 0x7efec00b6990 2024-04-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.629+0000 7efebea00700 1 -- 172.21.15.89:0/2818372411 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb800b9c0 con 0x7efec00b6990 2024-04-02T08:52:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.629+0000 7efebea00700 1 -- 172.21.15.89:0/2818372411 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb800bd70 con 0x7efec00b6990 2024-04-02T08:52:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.629+0000 7efec6c00700 1 -- 172.21.15.89:0/2818372411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efec00b6990 msgr2=0x7efec00b6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.629+0000 7efec6c00700 1 --2- 172.21.15.89:0/2818372411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efec00b6990 0x7efec00b6e10 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7efeb8009e50 tx=0x7efeb800b330 comp rx=0 tx=0).stop 2024-04-02T08:52:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.629+0000 7efec6c00700 1 -- 172.21.15.89:0/2818372411 shutdown_connections 2024-04-02T08:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.629+0000 7efec6c00700 1 --2- 172.21.15.89:0/2818372411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efec00b6990 0x7efec00b6e10 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.629+0000 7efec6c00700 1 --2- 172.21.15.89:0/2818372411 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efec00d05e0 0x7efec00d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.629+0000 7efec6c00700 1 -- 172.21.15.89:0/2818372411 >> 172.21.15.89:0/2818372411 conn(0x7efec00666a0 msgr2=0x7efec0066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.630+0000 7efec6c00700 1 -- 172.21.15.89:0/2818372411 shutdown_connections 2024-04-02T08:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.630+0000 7efec6c00700 1 -- 172.21.15.89:0/2818372411 wait complete. 2024-04-02T08:52:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.631+0000 7efec6c00700 1 Processor -- start 2024-04-02T08:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.631+0000 7efec6c00700 1 -- start start 2024-04-02T08:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.632+0000 7efec6c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efec00b6990 0x7efec01f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.632+0000 7efec6c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efec00d05e0 0x7efec00b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.632+0000 7efec6c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efec015da50 0x7efec0156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.632+0000 7efec6c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec01565a0 con 0x7efec015da50 2024-04-02T08:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.632+0000 7efec6c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec0156840 con 0x7efec00b6990 2024-04-02T08:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.632+0000 7efec6c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec01f31f0 con 0x7efec00d05e0 2024-04-02T08:52:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.633+0000 7efec4c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efec015da50 0x7efec0156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.633+0000 7efec4c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efec015da50 0x7efec0156cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52688/0 (socket says 172.21.15.89:52688) 2024-04-02T08:52:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.633+0000 7efec4c00700 1 -- 172.21.15.89:0/264557933 learned_addr learned my addr 172.21.15.89:0/264557933 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.633+0000 7efebfe00700 1 --2- 172.21.15.89:0/264557933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efec00b6990 0x7efec01f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.633+0000 7efebf400700 1 --2- 172.21.15.89:0/264557933 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efec00d05e0 0x7efec00b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.633+0000 7efec4c00700 1 -- 172.21.15.89:0/264557933 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efec00d05e0 msgr2=0x7efec00b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.633+0000 7efec4c00700 1 --2- 172.21.15.89:0/264557933 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efec00d05e0 0x7efec00b4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.633+0000 7efec4c00700 1 -- 172.21.15.89:0/264557933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efec00b6990 msgr2=0x7efec01f29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.633+0000 7efec4c00700 1 --2- 172.21.15.89:0/264557933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efec00b6990 0x7efec01f29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.633+0000 7efec4c00700 1 -- 172.21.15.89:0/264557933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb8009b30 con 0x7efec015da50 2024-04-02T08:52:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.633+0000 7efec4c00700 1 --2- 172.21.15.89:0/264557933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efec015da50 0x7efec0156cd0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7efec00741d0 tx=0x7efeb8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.634+0000 7efeb7e00700 1 -- 172.21.15.89:0/264557933 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb8004800 con 0x7efec015da50 2024-04-02T08:52:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.634+0000 7efeb7e00700 1 -- 172.21.15.89:0/264557933 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb8004960 con 0x7efec015da50 2024-04-02T08:52:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.634+0000 7efeb7e00700 1 -- 172.21.15.89:0/264557933 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb801c680 con 0x7efec015da50 2024-04-02T08:52:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.634+0000 7efec6c00700 1 -- 172.21.15.89:0/264557933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec01f34d0 con 0x7efec015da50 2024-04-02T08:52:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.634+0000 7efec6c00700 1 -- 172.21.15.89:0/264557933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec01f2ee0 con 0x7efec015da50 2024-04-02T08:52:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.635+0000 7efec6c00700 1 -- 172.21.15.89:0/264557933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec0067720 con 0x7efec015da50 2024-04-02T08:52:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.636+0000 7efeb7e00700 1 -- 172.21.15.89:0/264557933 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7efeb801c7e0 con 0x7efec015da50 2024-04-02T08:52:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.636+0000 7efeb7e00700 1 --2- 172.21.15.89:0/264557933 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efea006adb0 0x7efea006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.637+0000 7efeb7e00700 1 -- 172.21.15.89:0/264557933 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7efeb808b1c0 con 0x7efec015da50 2024-04-02T08:52:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.637+0000 7efebfe00700 1 --2- 172.21.15.89:0/264557933 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efea006adb0 0x7efea006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:46.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.637+0000 7efebfe00700 1 --2- 172.21.15.89:0/264557933 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efea006adb0 0x7efea006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efeb0007fd0 tx=0x7efeb0009040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:46.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.641+0000 7efeb7e00700 1 -- 172.21.15.89:0/264557933 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efeb8056060 con 0x7efec015da50 2024-04-02T08:52:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.840+0000 7efec6c00700 1 -- 172.21.15.89:0/264557933 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efec01f39f0 con 0x7efea006adb0 2024-04-02T08:52:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.851+0000 7efeb7e00700 1 -- 172.21.15.89:0/264557933 <== mgr.9318 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289409 (secure 0 0 0) 0x7efec01f39f0 con 0x7efea006adb0 2024-04-02T08:52:46.853 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.855+0000 7efec6c00700 1 -- 172.21.15.89:0/264557933 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efea006adb0 msgr2=0x7efea006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.855+0000 7efec6c00700 1 --2- 172.21.15.89:0/264557933 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efea006adb0 0x7efea006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efeb0007fd0 tx=0x7efeb0009040 comp rx=0 tx=0).stop 2024-04-02T08:52:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.855+0000 7efec6c00700 1 -- 172.21.15.89:0/264557933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efec015da50 msgr2=0x7efec0156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.855+0000 7efec6c00700 1 --2- 172.21.15.89:0/264557933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efec015da50 0x7efec0156cd0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7efec00741d0 tx=0x7efeb8004ad0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.855+0000 7efec6c00700 1 -- 172.21.15.89:0/264557933 shutdown_connections 2024-04-02T08:52:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.855+0000 7efec6c00700 1 --2- 172.21.15.89:0/264557933 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efea006adb0 0x7efea006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.855+0000 7efec6c00700 1 --2- 172.21.15.89:0/264557933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efec00b6990 0x7efec01f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.855+0000 7efec6c00700 1 --2- 172.21.15.89:0/264557933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efec015da50 0x7efec0156cd0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.855+0000 7efec6c00700 1 --2- 172.21.15.89:0/264557933 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efec00d05e0 0x7efec00b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.855+0000 7efec6c00700 1 -- 172.21.15.89:0/264557933 >> 172.21.15.89:0/264557933 conn(0x7efec00666a0 msgr2=0x7efec015a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.855+0000 7efec6c00700 1 -- 172.21.15.89:0/264557933 shutdown_connections 2024-04-02T08:52:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:46.855+0000 7efec6c00700 1 -- 172.21.15.89:0/264557933 wait complete. 2024-04-02T08:52:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T08:52:46.873 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T08:52:46.367399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":403868,"kb_used_data":17924,"kb_used_omap":12,"kb_used_meta":385779,"kb_avail":754570852,"statfs":{"total":773094113280,"available":772680552448,"internally_reserved":0,"allocated":18354176,"data_stored":9739046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":395038264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.390562"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.505423+0000","last_change":"2024-04-02T08:52:42.523267+0000","last_active":"2024-04-02T08:52:43.505423+0000","last_peered":"2024-04-02T08:52:43.505423+0000","last_clean":"2024-04-02T08:52:43.505423+0000","last_became_active":"2024-04-02T08:52:42.522160+0000","last_became_peered":"2024-04-02T08:52:42.522160+0000","last_unstale":"2024-04-02T08:52:43.505423+0000","last_undegraded":"2024-04-02T08:52:43.505423+0000","last_fullsized":"2024-04-02T08:52:43.505423+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:17:44.210549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504072+0000","last_change":"2024-04-02T08:52:42.510935+0000","last_active":"2024-04-02T08:52:43.504072+0000","last_peered":"2024-04-02T08:52:43.504072+0000","last_clean":"2024-04-02T08:52:43.504072+0000","last_became_active":"2024-04-02T08:52:42.510488+0000","last_became_peered":"2024-04-02T08:52:42.510488+0000","last_unstale":"2024-04-02T08:52:43.504072+0000","last_undegraded":"2024-04-02T08:52:43.504072+0000","last_fullsized":"2024-04-02T08:52:43.504072+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:30:49.616210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524246+0000","last_change":"2024-04-02T08:52:42.521836+0000","last_active":"2024-04-02T08:52:42.524246+0000","last_peered":"2024-04-02T08:52:42.524246+0000","last_clean":"2024-04-02T08:52:42.524246+0000","last_became_active":"2024-04-02T08:52:42.521074+0000","last_became_peered":"2024-04-02T08:52:42.521074+0000","last_unstale":"2024-04-02T08:52:42.524246+0000","last_undegraded":"2024-04-02T08:52:42.524246+0000","last_fullsized":"2024-04-02T08:52:42.524246+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:16:55.801303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.526929+0000","last_change":"2024-04-02T08:52:42.521870+0000","last_active":"2024-04-02T08:52:42.526929+0000","last_peered":"2024-04-02T08:52:42.526929+0000","last_clean":"2024-04-02T08:52:42.526929+0000","last_became_active":"2024-04-02T08:52:42.521562+0000","last_became_peered":"2024-04-02T08:52:42.521562+0000","last_unstale":"2024-04-02T08:52:42.526929+0000","last_undegraded":"2024-04-02T08:52:42.526929+0000","last_fullsized":"2024-04-02T08:52:42.526929+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:45:26.025962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.523110+0000","last_change":"2024-04-02T08:52:42.521162+0000","last_active":"2024-04-02T08:52:42.523110+0000","last_peered":"2024-04-02T08:52:42.523110+0000","last_clean":"2024-04-02T08:52:42.523110+0000","last_became_active":"2024-04-02T08:52:42.520895+0000","last_became_peered":"2024-04-02T08:52:42.520895+0000","last_unstale":"2024-04-02T08:52:42.523110+0000","last_undegraded":"2024-04-02T08:52:42.523110+0000","last_fullsized":"2024-04-02T08:52:42.523110+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:33:34.561166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986097+0000","last_change":"2024-04-02T08:52:42.525666+0000","last_active":"2024-04-02T08:52:43.986097+0000","last_peered":"2024-04-02T08:52:43.986097+0000","last_clean":"2024-04-02T08:52:43.986097+0000","last_became_active":"2024-04-02T08:52:42.525382+0000","last_became_peered":"2024-04-02T08:52:42.525382+0000","last_unstale":"2024-04-02T08:52:43.986097+0000","last_undegraded":"2024-04-02T08:52:43.986097+0000","last_fullsized":"2024-04-02T08:52:43.986097+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:31.170482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.502872+0000","last_change":"2024-04-02T08:52:42.514048+0000","last_active":"2024-04-02T08:52:43.502872+0000","last_peered":"2024-04-02T08:52:43.502872+0000","last_clean":"2024-04-02T08:52:43.502872+0000","last_became_active":"2024-04-02T08:52:42.513772+0000","last_became_peered":"2024-04-02T08:52:42.513772+0000","last_unstale":"2024-04-02T08:52:43.502872+0000","last_undegraded":"2024-04-02T08:52:43.502872+0000","last_fullsized":"2024-04-02T08:52:43.502872+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:07.399026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504649+0000","last_change":"2024-04-02T08:52:42.515605+0000","last_active":"2024-04-02T08:52:43.504649+0000","last_peered":"2024-04-02T08:52:43.504649+0000","last_clean":"2024-04-02T08:52:43.504649+0000","last_became_active":"2024-04-02T08:52:42.515316+0000","last_became_peered":"2024-04-02T08:52:42.515316+0000","last_unstale":"2024-04-02T08:52:43.504649+0000","last_undegraded":"2024-04-02T08:52:43.504649+0000","last_fullsized":"2024-04-02T08:52:43.504649+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:02:40.043066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986293+0000","last_change":"2024-04-02T08:52:42.515813+0000","last_active":"2024-04-02T08:52:43.986293+0000","last_peered":"2024-04-02T08:52:43.986293+0000","last_clean":"2024-04-02T08:52:43.986293+0000","last_became_active":"2024-04-02T08:52:42.515542+0000","last_became_peered":"2024-04-02T08:52:42.515542+0000","last_unstale":"2024-04-02T08:52:43.986293+0000","last_undegraded":"2024-04-02T08:52:43.986293+0000","last_fullsized":"2024-04-02T08:52:43.986293+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:32:09.597113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.519687+0000","last_change":"2024-04-02T08:52:42.510156+0000","last_active":"2024-04-02T08:52:42.519687+0000","last_peered":"2024-04-02T08:52:42.519687+0000","last_clean":"2024-04-02T08:52:42.519687+0000","last_became_active":"2024-04-02T08:52:42.504894+0000","last_became_peered":"2024-04-02T08:52:42.504894+0000","last_unstale":"2024-04-02T08:52:42.519687+0000","last_undegraded":"2024-04-02T08:52:42.519687+0000","last_fullsized":"2024-04-02T08:52:42.519687+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:15:32.537347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986186+0000","last_change":"2024-04-02T08:52:42.515909+0000","last_active":"2024-04-02T08:52:43.986186+0000","last_peered":"2024-04-02T08:52:43.986186+0000","last_clean":"2024-04-02T08:52:43.986186+0000","last_became_active":"2024-04-02T08:52:42.513069+0000","last_became_peered":"2024-04-02T08:52:42.513069+0000","last_unstale":"2024-04-02T08:52:43.986186+0000","last_undegraded":"2024-04-02T08:52:43.986186+0000","last_fullsized":"2024-04-02T08:52:43.986186+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:39:21.005383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.515332+0000","last_change":"2024-04-02T08:52:42.512377+0000","last_active":"2024-04-02T08:52:42.515332+0000","last_peered":"2024-04-02T08:52:42.515332+0000","last_clean":"2024-04-02T08:52:42.515332+0000","last_became_active":"2024-04-02T08:52:42.510307+0000","last_became_peered":"2024-04-02T08:52:42.510307+0000","last_unstale":"2024-04-02T08:52:42.515332+0000","last_undegraded":"2024-04-02T08:52:42.515332+0000","last_fullsized":"2024-04-02T08:52:42.515332+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:39:38.170946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.523129+0000","last_change":"2024-04-02T08:52:42.522105+0000","last_active":"2024-04-02T08:52:42.523129+0000","last_peered":"2024-04-02T08:52:42.523129+0000","last_clean":"2024-04-02T08:52:42.523129+0000","last_became_active":"2024-04-02T08:52:42.519471+0000","last_became_peered":"2024-04-02T08:52:42.519471+0000","last_unstale":"2024-04-02T08:52:42.523129+0000","last_undegraded":"2024-04-02T08:52:42.523129+0000","last_fullsized":"2024-04-02T08:52:42.523129+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:15.923579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.527693+0000","last_change":"2024-04-02T08:52:42.526972+0000","last_active":"2024-04-02T08:52:42.527693+0000","last_peered":"2024-04-02T08:52:42.527693+0000","last_clean":"2024-04-02T08:52:42.527693+0000","last_became_active":"2024-04-02T08:52:42.526258+0000","last_became_peered":"2024-04-02T08:52:42.526258+0000","last_unstale":"2024-04-02T08:52:42.527693+0000","last_undegraded":"2024-04-02T08:52:42.527693+0000","last_fullsized":"2024-04-02T08:52:42.527693+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:15:06.853630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.506226+0000","last_change":"2024-04-02T08:52:42.503486+0000","last_active":"2024-04-02T08:52:42.506226+0000","last_peered":"2024-04-02T08:52:42.506226+0000","last_clean":"2024-04-02T08:52:42.506226+0000","last_became_active":"2024-04-02T08:52:42.503249+0000","last_became_peered":"2024-04-02T08:52:42.503249+0000","last_unstale":"2024-04-02T08:52:42.506226+0000","last_undegraded":"2024-04-02T08:52:42.506226+0000","last_fullsized":"2024-04-02T08:52:42.506226+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:32:04.541950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.502832+0000","last_change":"2024-04-02T08:52:42.497825+0000","last_active":"2024-04-02T08:52:43.502832+0000","last_peered":"2024-04-02T08:52:43.502832+0000","last_clean":"2024-04-02T08:52:43.502832+0000","last_became_active":"2024-04-02T08:52:42.497517+0000","last_became_peered":"2024-04-02T08:52:42.497517+0000","last_unstale":"2024-04-02T08:52:43.502832+0000","last_undegraded":"2024-04-02T08:52:43.502832+0000","last_fullsized":"2024-04-02T08:52:43.502832+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:10:35.665594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.522854+0000","last_change":"2024-04-02T08:52:42.511203+0000","last_active":"2024-04-02T08:52:42.522854+0000","last_peered":"2024-04-02T08:52:42.522854+0000","last_clean":"2024-04-02T08:52:42.522854+0000","last_became_active":"2024-04-02T08:52:42.510938+0000","last_became_peered":"2024-04-02T08:52:42.510938+0000","last_unstale":"2024-04-02T08:52:42.522854+0000","last_undegraded":"2024-04-02T08:52:42.522854+0000","last_fullsized":"2024-04-02T08:52:42.522854+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:39.813595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.521678+0000","last_change":"2024-04-02T08:52:42.517643+0000","last_active":"2024-04-02T08:52:42.521678+0000","last_peered":"2024-04-02T08:52:42.521678+0000","last_clean":"2024-04-02T08:52:42.521678+0000","last_became_active":"2024-04-02T08:52:42.517396+0000","last_became_peered":"2024-04-02T08:52:42.517396+0000","last_unstale":"2024-04-02T08:52:42.521678+0000","last_undegraded":"2024-04-02T08:52:42.521678+0000","last_fullsized":"2024-04-02T08:52:42.521678+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:36:26.482902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.505050+0000","last_change":"2024-04-02T08:52:42.519398+0000","last_active":"2024-04-02T08:52:43.505050+0000","last_peered":"2024-04-02T08:52:43.505050+0000","last_clean":"2024-04-02T08:52:43.505050+0000","last_became_active":"2024-04-02T08:52:42.518839+0000","last_became_peered":"2024-04-02T08:52:42.518839+0000","last_unstale":"2024-04-02T08:52:43.505050+0000","last_undegraded":"2024-04-02T08:52:43.505050+0000","last_fullsized":"2024-04-02T08:52:43.505050+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:10:35.665594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.505006+0000","last_change":"2024-04-02T08:52:42.504263+0000","last_active":"2024-04-02T08:52:43.505006+0000","last_peered":"2024-04-02T08:52:43.505006+0000","last_clean":"2024-04-02T08:52:43.505006+0000","last_became_active":"2024-04-02T08:52:42.503847+0000","last_became_peered":"2024-04-02T08:52:42.503847+0000","last_unstale":"2024-04-02T08:52:43.505006+0000","last_undegraded":"2024-04-02T08:52:43.505006+0000","last_fullsized":"2024-04-02T08:52:43.505006+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:07.399026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524322+0000","last_change":"2024-04-02T08:52:42.515542+0000","last_active":"2024-04-02T08:52:42.524322+0000","last_peered":"2024-04-02T08:52:42.524322+0000","last_clean":"2024-04-02T08:52:42.524322+0000","last_became_active":"2024-04-02T08:52:42.510200+0000","last_became_peered":"2024-04-02T08:52:42.510200+0000","last_unstale":"2024-04-02T08:52:42.524322+0000","last_undegraded":"2024-04-02T08:52:42.524322+0000","last_fullsized":"2024-04-02T08:52:42.524322+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:56:27.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.506341+0000","last_change":"2024-04-02T08:52:42.504094+0000","last_active":"2024-04-02T08:52:42.506341+0000","last_peered":"2024-04-02T08:52:42.506341+0000","last_clean":"2024-04-02T08:52:42.506341+0000","last_became_active":"2024-04-02T08:52:42.503556+0000","last_became_peered":"2024-04-02T08:52:42.503556+0000","last_unstale":"2024-04-02T08:52:42.506341+0000","last_undegraded":"2024-04-02T08:52:42.506341+0000","last_fullsized":"2024-04-02T08:52:42.506341+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:55:39.440345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.527280+0000","last_change":"2024-04-02T08:52:42.525035+0000","last_active":"2024-04-02T08:52:42.527280+0000","last_peered":"2024-04-02T08:52:42.527280+0000","last_clean":"2024-04-02T08:52:42.527280+0000","last_became_active":"2024-04-02T08:52:42.520037+0000","last_became_peered":"2024-04-02T08:52:42.520037+0000","last_unstale":"2024-04-02T08:52:42.527280+0000","last_undegraded":"2024-04-02T08:52:42.527280+0000","last_fullsized":"2024-04-02T08:52:42.527280+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:39:21.005383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.519234+0000","last_change":"2024-04-02T08:52:42.516592+0000","last_active":"2024-04-02T08:52:42.519234+0000","last_peered":"2024-04-02T08:52:42.519234+0000","last_clean":"2024-04-02T08:52:42.519234+0000","last_became_active":"2024-04-02T08:52:42.515697+0000","last_became_peered":"2024-04-02T08:52:42.515697+0000","last_unstale":"2024-04-02T08:52:42.519234+0000","last_undegraded":"2024-04-02T08:52:42.519234+0000","last_fullsized":"2024-04-02T08:52:42.519234+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:59:01.321730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504078+0000","last_change":"2024-04-02T08:52:42.507409+0000","last_active":"2024-04-02T08:52:43.504078+0000","last_peered":"2024-04-02T08:52:43.504078+0000","last_clean":"2024-04-02T08:52:43.504078+0000","last_became_active":"2024-04-02T08:52:42.507123+0000","last_became_peered":"2024-04-02T08:52:42.507123+0000","last_unstale":"2024-04-02T08:52:43.504078+0000","last_undegraded":"2024-04-02T08:52:43.504078+0000","last_fullsized":"2024-04-02T08:52:43.504078+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:39:21.005383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.505237+0000","last_change":"2024-04-02T08:52:42.506905+0000","last_active":"2024-04-02T08:52:43.505237+0000","last_peered":"2024-04-02T08:52:43.505237+0000","last_clean":"2024-04-02T08:52:43.505237+0000","last_became_active":"2024-04-02T08:52:42.505098+0000","last_became_peered":"2024-04-02T08:52:42.505098+0000","last_unstale":"2024-04-02T08:52:43.505237+0000","last_undegraded":"2024-04-02T08:52:43.505237+0000","last_fullsized":"2024-04-02T08:52:43.505237+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:17:18.662540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503947+0000","last_change":"2024-04-02T08:52:42.503146+0000","last_active":"2024-04-02T08:52:43.503947+0000","last_peered":"2024-04-02T08:52:43.503947+0000","last_clean":"2024-04-02T08:52:43.503947+0000","last_became_active":"2024-04-02T08:52:42.502862+0000","last_became_peered":"2024-04-02T08:52:42.502862+0000","last_unstale":"2024-04-02T08:52:43.503947+0000","last_undegraded":"2024-04-02T08:52:43.503947+0000","last_fullsized":"2024-04-02T08:52:43.503947+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:08:18.767645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503507+0000","last_change":"2024-04-02T08:52:42.509837+0000","last_active":"2024-04-02T08:52:43.503507+0000","last_peered":"2024-04-02T08:52:43.503507+0000","last_clean":"2024-04-02T08:52:43.503507+0000","last_became_active":"2024-04-02T08:52:42.509587+0000","last_became_peered":"2024-04-02T08:52:42.509587+0000","last_unstale":"2024-04-02T08:52:43.503507+0000","last_undegraded":"2024-04-02T08:52:43.503507+0000","last_fullsized":"2024-04-02T08:52:43.503507+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:06.680663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504864+0000","last_change":"2024-04-02T08:52:42.518262+0000","last_active":"2024-04-02T08:52:43.504864+0000","last_peered":"2024-04-02T08:52:43.504864+0000","last_clean":"2024-04-02T08:52:43.504864+0000","last_became_active":"2024-04-02T08:52:42.517816+0000","last_became_peered":"2024-04-02T08:52:42.517816+0000","last_unstale":"2024-04-02T08:52:43.504864+0000","last_undegraded":"2024-04-02T08:52:43.504864+0000","last_fullsized":"2024-04-02T08:52:43.504864+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:27:40.539739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.506366+0000","last_change":"2024-04-02T08:52:42.504235+0000","last_active":"2024-04-02T08:52:42.506366+0000","last_peered":"2024-04-02T08:52:42.506366+0000","last_clean":"2024-04-02T08:52:42.506366+0000","last_became_active":"2024-04-02T08:52:42.503854+0000","last_became_peered":"2024-04-02T08:52:42.503854+0000","last_unstale":"2024-04-02T08:52:42.506366+0000","last_undegraded":"2024-04-02T08:52:42.506366+0000","last_fullsized":"2024-04-02T08:52:42.506366+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:50.168508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503534+0000","last_change":"2024-04-02T08:52:42.509956+0000","last_active":"2024-04-02T08:52:43.503534+0000","last_peered":"2024-04-02T08:52:43.503534+0000","last_clean":"2024-04-02T08:52:43.503534+0000","last_became_active":"2024-04-02T08:52:42.504622+0000","last_became_peered":"2024-04-02T08:52:42.504622+0000","last_unstale":"2024-04-02T08:52:43.503534+0000","last_undegraded":"2024-04-02T08:52:43.503534+0000","last_fullsized":"2024-04-02T08:52:43.503534+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:18:36.365111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503102+0000","last_change":"2024-04-02T08:52:42.515515+0000","last_active":"2024-04-02T08:52:43.503102+0000","last_peered":"2024-04-02T08:52:43.503102+0000","last_clean":"2024-04-02T08:52:43.503102+0000","last_became_active":"2024-04-02T08:52:42.515000+0000","last_became_peered":"2024-04-02T08:52:42.515000+0000","last_unstale":"2024-04-02T08:52:43.503102+0000","last_undegraded":"2024-04-02T08:52:43.503102+0000","last_fullsized":"2024-04-02T08:52:43.503102+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:15:39.710604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524555+0000","last_change":"2024-04-02T08:52:42.521825+0000","last_active":"2024-04-02T08:52:42.524555+0000","last_peered":"2024-04-02T08:52:42.524555+0000","last_clean":"2024-04-02T08:52:42.524555+0000","last_became_active":"2024-04-02T08:52:42.519988+0000","last_became_peered":"2024-04-02T08:52:42.519988+0000","last_unstale":"2024-04-02T08:52:42.524555+0000","last_undegraded":"2024-04-02T08:52:42.524555+0000","last_fullsized":"2024-04-02T08:52:42.524555+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:40:28.725332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"171'4","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524343+0000","last_change":"2024-04-02T08:52:42.003517+0000","last_active":"2024-04-02T08:52:42.524343+0000","last_peered":"2024-04-02T08:52:42.524343+0000","last_clean":"2024-04-02T08:52:42.524343+0000","last_became_active":"2024-04-02T08:52:42.003267+0000","last_became_peered":"2024-04-02T08:52:42.003267+0000","last_unstale":"2024-04-02T08:52:42.524343+0000","last_undegraded":"2024-04-02T08:52:42.524343+0000","last_fullsized":"2024-04-02T08:52:42.524343+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:42.599747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503707+0000","last_change":"2024-04-02T08:52:42.512866+0000","last_active":"2024-04-02T08:52:43.503707+0000","last_peered":"2024-04-02T08:52:43.503707+0000","last_clean":"2024-04-02T08:52:43.503707+0000","last_became_active":"2024-04-02T08:52:42.512546+0000","last_became_peered":"2024-04-02T08:52:42.512546+0000","last_unstale":"2024-04-02T08:52:43.503707+0000","last_undegraded":"2024-04-02T08:52:43.503707+0000","last_fullsized":"2024-04-02T08:52:43.503707+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:17:18.662540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.1e","version":"171'1","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986265+0000","last_change":"2024-04-02T08:52:42.011712+0000","last_active":"2024-04-02T08:52:43.986265+0000","last_peered":"2024-04-02T08:52:43.986265+0000","last_clean":"2024-04-02T08:52:43.986265+0000","last_became_active":"2024-04-02T08:52:42.011324+0000","last_became_peered":"2024-04-02T08:52:42.011324+0000","last_unstale":"2024-04-02T08:52:43.986265+0000","last_undegraded":"2024-04-02T08:52:43.986265+0000","last_fullsized":"2024-04-02T08:52:43.986265+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:05.673571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986103+0000","last_change":"2024-04-02T08:52:42.520857+0000","last_active":"2024-04-02T08:52:43.986103+0000","last_peered":"2024-04-02T08:52:43.986103+0000","last_clean":"2024-04-02T08:52:43.986103+0000","last_became_active":"2024-04-02T08:52:42.520559+0000","last_became_peered":"2024-04-02T08:52:42.520559+0000","last_unstale":"2024-04-02T08:52:43.986103+0000","last_undegraded":"2024-04-02T08:52:43.986103+0000","last_fullsized":"2024-04-02T08:52:43.986103+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:16:45.977261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1d","version":"171'2","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.517048+0000","last_change":"2024-04-02T08:52:42.008308+0000","last_active":"2024-04-02T08:52:42.517048+0000","last_peered":"2024-04-02T08:52:42.517048+0000","last_clean":"2024-04-02T08:52:42.517048+0000","last_became_active":"2024-04-02T08:52:42.008053+0000","last_became_peered":"2024-04-02T08:52:42.008053+0000","last_unstale":"2024-04-02T08:52:42.517048+0000","last_undegraded":"2024-04-02T08:52:42.517048+0000","last_fullsized":"2024-04-02T08:52:42.517048+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:18:06.535318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.520000+0000","last_change":"2024-04-02T08:52:42.511289+0000","last_active":"2024-04-02T08:52:42.520000+0000","last_peered":"2024-04-02T08:52:42.520000+0000","last_clean":"2024-04-02T08:52:42.520000+0000","last_became_active":"2024-04-02T08:52:42.511026+0000","last_became_peered":"2024-04-02T08:52:42.511026+0000","last_unstale":"2024-04-02T08:52:42.520000+0000","last_undegraded":"2024-04-02T08:52:42.520000+0000","last_fullsized":"2024-04-02T08:52:42.520000+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:56:32.295359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1c","version":"171'2","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524007+0000","last_change":"2024-04-02T08:52:42.010268+0000","last_active":"2024-04-02T08:52:42.524007+0000","last_peered":"2024-04-02T08:52:42.524007+0000","last_clean":"2024-04-02T08:52:42.524007+0000","last_became_active":"2024-04-02T08:52:42.009384+0000","last_became_peered":"2024-04-02T08:52:42.009384+0000","last_unstale":"2024-04-02T08:52:42.524007+0000","last_undegraded":"2024-04-02T08:52:42.524007+0000","last_fullsized":"2024-04-02T08:52:42.524007+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:14:17.826438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503508+0000","last_change":"2024-04-02T08:52:42.522266+0000","last_active":"2024-04-02T08:52:43.503508+0000","last_peered":"2024-04-02T08:52:43.503508+0000","last_clean":"2024-04-02T08:52:43.503508+0000","last_became_active":"2024-04-02T08:52:42.521874+0000","last_became_peered":"2024-04-02T08:52:42.521874+0000","last_unstale":"2024-04-02T08:52:43.503508+0000","last_undegraded":"2024-04-02T08:52:43.503508+0000","last_fullsized":"2024-04-02T08:52:43.503508+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:00:21.809746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"171'1","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504982+0000","last_change":"2024-04-02T08:52:42.000929+0000","last_active":"2024-04-02T08:52:43.504982+0000","last_peered":"2024-04-02T08:52:43.504982+0000","last_clean":"2024-04-02T08:52:43.504982+0000","last_became_active":"2024-04-02T08:52:42.000418+0000","last_became_peered":"2024-04-02T08:52:42.000418+0000","last_unstale":"2024-04-02T08:52:43.504982+0000","last_undegraded":"2024-04-02T08:52:43.504982+0000","last_fullsized":"2024-04-02T08:52:43.504982+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:42:14.230762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.520239+0000","last_change":"2024-04-02T08:52:42.514952+0000","last_active":"2024-04-02T08:52:42.520239+0000","last_peered":"2024-04-02T08:52:42.520239+0000","last_clean":"2024-04-02T08:52:42.520239+0000","last_became_active":"2024-04-02T08:52:42.514700+0000","last_became_peered":"2024-04-02T08:52:42.514700+0000","last_unstale":"2024-04-02T08:52:42.520239+0000","last_undegraded":"2024-04-02T08:52:42.520239+0000","last_fullsized":"2024-04-02T08:52:42.520239+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:18:36.365111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1a","version":"171'1","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524946+0000","last_change":"2024-04-02T08:52:42.001735+0000","last_active":"2024-04-02T08:52:42.524946+0000","last_peered":"2024-04-02T08:52:42.524946+0000","last_clean":"2024-04-02T08:52:42.524946+0000","last_became_active":"2024-04-02T08:52:42.000896+0000","last_became_peered":"2024-04-02T08:52:42.000896+0000","last_unstale":"2024-04-02T08:52:42.524946+0000","last_undegraded":"2024-04-02T08:52:42.524946+0000","last_fullsized":"2024-04-02T08:52:42.524946+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:06.781538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503218+0000","last_change":"2024-04-02T08:52:42.516017+0000","last_active":"2024-04-02T08:52:43.503218+0000","last_peered":"2024-04-02T08:52:43.503218+0000","last_clean":"2024-04-02T08:52:43.503218+0000","last_became_active":"2024-04-02T08:52:42.515288+0000","last_became_peered":"2024-04-02T08:52:42.515288+0000","last_unstale":"2024-04-02T08:52:43.503218+0000","last_undegraded":"2024-04-02T08:52:43.503218+0000","last_fullsized":"2024-04-02T08:52:43.503218+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:18:05.763436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986283+0000","last_change":"2024-04-02T08:52:42.009019+0000","last_active":"2024-04-02T08:52:43.986283+0000","last_peered":"2024-04-02T08:52:43.986283+0000","last_clean":"2024-04-02T08:52:43.986283+0000","last_became_active":"2024-04-02T08:52:42.007461+0000","last_became_peered":"2024-04-02T08:52:42.007461+0000","last_unstale":"2024-04-02T08:52:43.986283+0000","last_undegraded":"2024-04-02T08:52:43.986283+0000","last_fullsized":"2024-04-02T08:52:43.986283+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:15:38.703511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.520904+0000","last_change":"2024-04-02T08:52:42.508371+0000","last_active":"2024-04-02T08:52:42.520904+0000","last_peered":"2024-04-02T08:52:42.520904+0000","last_clean":"2024-04-02T08:52:42.520904+0000","last_became_active":"2024-04-02T08:52:42.507952+0000","last_became_peered":"2024-04-02T08:52:42.507952+0000","last_unstale":"2024-04-02T08:52:42.520904+0000","last_undegraded":"2024-04-02T08:52:42.520904+0000","last_fullsized":"2024-04-02T08:52:42.520904+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:08:18.767645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.492931+0000","last_change":"2024-04-02T08:52:42.009648+0000","last_active":"2024-04-02T08:52:42.492931+0000","last_peered":"2024-04-02T08:52:42.492931+0000","last_clean":"2024-04-02T08:52:42.492931+0000","last_became_active":"2024-04-02T08:52:42.008796+0000","last_became_peered":"2024-04-02T08:52:42.008796+0000","last_unstale":"2024-04-02T08:52:42.492931+0000","last_undegraded":"2024-04-02T08:52:42.492931+0000","last_fullsized":"2024-04-02T08:52:42.492931+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:49:17.177132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.523923+0000","last_change":"2024-04-02T08:52:42.518633+0000","last_active":"2024-04-02T08:52:42.523923+0000","last_peered":"2024-04-02T08:52:42.523923+0000","last_clean":"2024-04-02T08:52:42.523923+0000","last_became_active":"2024-04-02T08:52:42.518291+0000","last_became_peered":"2024-04-02T08:52:42.518291+0000","last_unstale":"2024-04-02T08:52:42.523923+0000","last_undegraded":"2024-04-02T08:52:42.523923+0000","last_fullsized":"2024-04-02T08:52:42.523923+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:36.498912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986293+0000","last_change":"2024-04-02T08:52:42.008236+0000","last_active":"2024-04-02T08:52:43.986293+0000","last_peered":"2024-04-02T08:52:43.986293+0000","last_clean":"2024-04-02T08:52:43.986293+0000","last_became_active":"2024-04-02T08:52:42.007963+0000","last_became_peered":"2024-04-02T08:52:42.007963+0000","last_unstale":"2024-04-02T08:52:43.986293+0000","last_undegraded":"2024-04-02T08:52:43.986293+0000","last_fullsized":"2024-04-02T08:52:43.986293+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:30:48.609117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.515922+0000","last_change":"2024-04-02T08:52:42.513612+0000","last_active":"2024-04-02T08:52:42.515922+0000","last_peered":"2024-04-02T08:52:42.515922+0000","last_clean":"2024-04-02T08:52:42.515922+0000","last_became_active":"2024-04-02T08:52:42.513220+0000","last_became_peered":"2024-04-02T08:52:42.513220+0000","last_unstale":"2024-04-02T08:52:42.515922+0000","last_undegraded":"2024-04-02T08:52:42.515922+0000","last_fullsized":"2024-04-02T08:52:42.515922+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:17:29.804325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.500130+0000","last_change":"2024-04-02T08:52:42.008640+0000","last_active":"2024-04-02T08:52:42.500130+0000","last_peered":"2024-04-02T08:52:42.500130+0000","last_clean":"2024-04-02T08:52:42.500130+0000","last_became_active":"2024-04-02T08:52:42.007386+0000","last_became_peered":"2024-04-02T08:52:42.007386+0000","last_unstale":"2024-04-02T08:52:42.500130+0000","last_undegraded":"2024-04-02T08:52:42.500130+0000","last_fullsized":"2024-04-02T08:52:42.500130+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:31.353853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503448+0000","last_change":"2024-04-02T08:52:42.498531+0000","last_active":"2024-04-02T08:52:43.503448+0000","last_peered":"2024-04-02T08:52:43.503448+0000","last_clean":"2024-04-02T08:52:43.503448+0000","last_became_active":"2024-04-02T08:52:42.497676+0000","last_became_peered":"2024-04-02T08:52:42.497676+0000","last_unstale":"2024-04-02T08:52:43.503448+0000","last_undegraded":"2024-04-02T08:52:43.503448+0000","last_fullsized":"2024-04-02T08:52:43.503448+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:27:40.539739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.14","version":"171'6","reported_seq":17,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503320+0000","last_change":"2024-04-02T08:52:42.005636+0000","last_active":"2024-04-02T08:52:43.503320+0000","last_peered":"2024-04-02T08:52:43.503320+0000","last_clean":"2024-04-02T08:52:43.503320+0000","last_became_active":"2024-04-02T08:52:42.005368+0000","last_became_peered":"2024-04-02T08:52:42.005368+0000","last_unstale":"2024-04-02T08:52:43.503320+0000","last_undegraded":"2024-04-02T08:52:43.503320+0000","last_fullsized":"2024-04-02T08:52:43.503320+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:19:07.924940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.520816+0000","last_change":"2024-04-02T08:52:42.508364+0000","last_active":"2024-04-02T08:52:42.520816+0000","last_peered":"2024-04-02T08:52:42.520816+0000","last_clean":"2024-04-02T08:52:42.520816+0000","last_became_active":"2024-04-02T08:52:42.507935+0000","last_became_peered":"2024-04-02T08:52:42.507935+0000","last_unstale":"2024-04-02T08:52:42.520816+0000","last_undegraded":"2024-04-02T08:52:42.520816+0000","last_fullsized":"2024-04-02T08:52:42.520816+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:30:49.616210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.13","version":"171'2","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524368+0000","last_change":"2024-04-02T08:52:42.009185+0000","last_active":"2024-04-02T08:52:42.524368+0000","last_peered":"2024-04-02T08:52:42.524368+0000","last_clean":"2024-04-02T08:52:42.524368+0000","last_became_active":"2024-04-02T08:52:42.008916+0000","last_became_peered":"2024-04-02T08:52:42.008916+0000","last_unstale":"2024-04-02T08:52:42.524368+0000","last_undegraded":"2024-04-02T08:52:42.524368+0000","last_fullsized":"2024-04-02T08:52:42.524368+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:29.849209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504909+0000","last_change":"2024-04-02T08:52:42.518098+0000","last_active":"2024-04-02T08:52:43.504909+0000","last_peered":"2024-04-02T08:52:43.504909+0000","last_clean":"2024-04-02T08:52:43.504909+0000","last_became_active":"2024-04-02T08:52:42.517269+0000","last_became_peered":"2024-04-02T08:52:42.517269+0000","last_unstale":"2024-04-02T08:52:43.504909+0000","last_undegraded":"2024-04-02T08:52:43.504909+0000","last_fullsized":"2024-04-02T08:52:43.504909+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:32:09.794631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.501773+0000","last_change":"2024-04-02T08:52:42.011331+0000","last_active":"2024-04-02T08:52:42.501773+0000","last_peered":"2024-04-02T08:52:42.501773+0000","last_clean":"2024-04-02T08:52:42.501773+0000","last_became_active":"2024-04-02T08:52:42.011038+0000","last_became_peered":"2024-04-02T08:52:42.011038+0000","last_unstale":"2024-04-02T08:52:42.501773+0000","last_undegraded":"2024-04-02T08:52:42.501773+0000","last_fullsized":"2024-04-02T08:52:42.501773+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:02:48.920433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.526572+0000","last_change":"2024-04-02T08:52:42.520472+0000","last_active":"2024-04-02T08:52:42.526572+0000","last_peered":"2024-04-02T08:52:42.526572+0000","last_clean":"2024-04-02T08:52:42.526572+0000","last_became_active":"2024-04-02T08:52:42.520166+0000","last_became_peered":"2024-04-02T08:52:42.520166+0000","last_unstale":"2024-04-02T08:52:42.526572+0000","last_undegraded":"2024-04-02T08:52:42.526572+0000","last_fullsized":"2024-04-02T08:52:42.526572+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:27.050671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503625+0000","last_change":"2024-04-02T08:52:42.010289+0000","last_active":"2024-04-02T08:52:43.503625+0000","last_peered":"2024-04-02T08:52:43.503625+0000","last_clean":"2024-04-02T08:52:43.503625+0000","last_became_active":"2024-04-02T08:52:42.009720+0000","last_became_peered":"2024-04-02T08:52:42.009720+0000","last_unstale":"2024-04-02T08:52:43.503625+0000","last_undegraded":"2024-04-02T08:52:43.503625+0000","last_fullsized":"2024-04-02T08:52:43.503625+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:34:58.203723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.527413+0000","last_change":"2024-04-02T08:52:42.525284+0000","last_active":"2024-04-02T08:52:42.527413+0000","last_peered":"2024-04-02T08:52:42.527413+0000","last_clean":"2024-04-02T08:52:42.527413+0000","last_became_active":"2024-04-02T08:52:42.520143+0000","last_became_peered":"2024-04-02T08:52:42.520143+0000","last_unstale":"2024-04-02T08:52:42.527413+0000","last_undegraded":"2024-04-02T08:52:42.527413+0000","last_fullsized":"2024-04-02T08:52:42.527413+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:00:21.809746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.10","version":"171'8","reported_seq":19,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504745+0000","last_change":"2024-04-02T08:52:42.007328+0000","last_active":"2024-04-02T08:52:43.504745+0000","last_peered":"2024-04-02T08:52:43.504745+0000","last_clean":"2024-04-02T08:52:43.504745+0000","last_became_active":"2024-04-02T08:52:42.007044+0000","last_became_peered":"2024-04-02T08:52:42.007044+0000","last_unstale":"2024-04-02T08:52:43.504745+0000","last_undegraded":"2024-04-02T08:52:43.504745+0000","last_fullsized":"2024-04-02T08:52:43.504745+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:12.170720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.521653+0000","last_change":"2024-04-02T08:52:42.515821+0000","last_active":"2024-04-02T08:52:42.521653+0000","last_peered":"2024-04-02T08:52:42.521653+0000","last_clean":"2024-04-02T08:52:42.521653+0000","last_became_active":"2024-04-02T08:52:42.515548+0000","last_became_peered":"2024-04-02T08:52:42.515548+0000","last_unstale":"2024-04-02T08:52:42.521653+0000","last_undegraded":"2024-04-02T08:52:42.521653+0000","last_fullsized":"2024-04-02T08:52:42.521653+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:52.449340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.f","version":"171'1","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524596+0000","last_change":"2024-04-02T08:52:42.006635+0000","last_active":"2024-04-02T08:52:42.524596+0000","last_peered":"2024-04-02T08:52:42.524596+0000","last_clean":"2024-04-02T08:52:42.524596+0000","last_became_active":"2024-04-02T08:52:42.003755+0000","last_became_peered":"2024-04-02T08:52:42.003755+0000","last_unstale":"2024-04-02T08:52:42.524596+0000","last_undegraded":"2024-04-02T08:52:42.524596+0000","last_fullsized":"2024-04-02T08:52:42.524596+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:41:26.162559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504139+0000","last_change":"2024-04-02T08:52:42.504557+0000","last_active":"2024-04-02T08:52:43.504139+0000","last_peered":"2024-04-02T08:52:43.504139+0000","last_clean":"2024-04-02T08:52:43.504139+0000","last_became_active":"2024-04-02T08:52:42.504250+0000","last_became_peered":"2024-04-02T08:52:42.504250+0000","last_unstale":"2024-04-02T08:52:43.504139+0000","last_undegraded":"2024-04-02T08:52:43.504139+0000","last_fullsized":"2024-04-02T08:52:43.504139+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:36.498912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.e","version":"171'2","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.505306+0000","last_change":"2024-04-02T08:52:42.005337+0000","last_active":"2024-04-02T08:52:43.505306+0000","last_peered":"2024-04-02T08:52:43.505306+0000","last_clean":"2024-04-02T08:52:43.505306+0000","last_became_active":"2024-04-02T08:52:42.003327+0000","last_became_peered":"2024-04-02T08:52:42.003327+0000","last_unstale":"2024-04-02T08:52:43.505306+0000","last_undegraded":"2024-04-02T08:52:43.505306+0000","last_fullsized":"2024-04-02T08:52:43.505306+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:46.403723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.512227+0000","last_change":"2024-04-02T08:52:42.508038+0000","last_active":"2024-04-02T08:52:42.512227+0000","last_peered":"2024-04-02T08:52:42.512227+0000","last_clean":"2024-04-02T08:52:42.512227+0000","last_became_active":"2024-04-02T08:52:42.507789+0000","last_became_peered":"2024-04-02T08:52:42.507789+0000","last_unstale":"2024-04-02T08:52:42.512227+0000","last_undegraded":"2024-04-02T08:52:42.512227+0000","last_fullsized":"2024-04-02T08:52:42.512227+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:34:27.421047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.d","version":"171'1","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503622+0000","last_change":"2024-04-02T08:52:42.007277+0000","last_active":"2024-04-02T08:52:43.503622+0000","last_peered":"2024-04-02T08:52:43.503622+0000","last_clean":"2024-04-02T08:52:43.503622+0000","last_became_active":"2024-04-02T08:52:42.005700+0000","last_became_peered":"2024-04-02T08:52:42.005700+0000","last_unstale":"2024-04-02T08:52:43.503622+0000","last_undegraded":"2024-04-02T08:52:43.503622+0000","last_fullsized":"2024-04-02T08:52:43.503622+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:19:06.944801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986186+0000","last_change":"2024-04-02T08:52:42.520926+0000","last_active":"2024-04-02T08:52:43.986186+0000","last_peered":"2024-04-02T08:52:43.986186+0000","last_clean":"2024-04-02T08:52:43.986186+0000","last_became_active":"2024-04-02T08:52:42.520640+0000","last_became_peered":"2024-04-02T08:52:42.520640+0000","last_unstale":"2024-04-02T08:52:43.986186+0000","last_undegraded":"2024-04-02T08:52:43.986186+0000","last_fullsized":"2024-04-02T08:52:43.986186+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:02:38.305601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503776+0000","last_change":"2024-04-02T08:52:42.009748+0000","last_active":"2024-04-02T08:52:43.503776+0000","last_peered":"2024-04-02T08:52:43.503776+0000","last_clean":"2024-04-02T08:52:43.503776+0000","last_became_active":"2024-04-02T08:52:42.009199+0000","last_became_peered":"2024-04-02T08:52:42.009199+0000","last_unstale":"2024-04-02T08:52:43.503776+0000","last_undegraded":"2024-04-02T08:52:43.503776+0000","last_fullsized":"2024-04-02T08:52:43.503776+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:09:55.394419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.515136+0000","last_change":"2024-04-02T08:52:42.508085+0000","last_active":"2024-04-02T08:52:42.515136+0000","last_peered":"2024-04-02T08:52:42.515136+0000","last_clean":"2024-04-02T08:52:42.515136+0000","last_became_active":"2024-04-02T08:52:42.507847+0000","last_became_peered":"2024-04-02T08:52:42.507847+0000","last_unstale":"2024-04-02T08:52:42.515136+0000","last_undegraded":"2024-04-02T08:52:42.515136+0000","last_fullsized":"2024-04-02T08:52:42.515136+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:03.828822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.b","version":"171'2","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986009+0000","last_change":"2024-04-02T08:52:42.011571+0000","last_active":"2024-04-02T08:52:43.986009+0000","last_peered":"2024-04-02T08:52:43.986009+0000","last_clean":"2024-04-02T08:52:43.986009+0000","last_became_active":"2024-04-02T08:52:42.011109+0000","last_became_peered":"2024-04-02T08:52:42.011109+0000","last_unstale":"2024-04-02T08:52:43.986009+0000","last_undegraded":"2024-04-02T08:52:43.986009+0000","last_fullsized":"2024-04-02T08:52:43.986009+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:26.043578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503710+0000","last_change":"2024-04-02T08:52:42.504471+0000","last_active":"2024-04-02T08:52:43.503710+0000","last_peered":"2024-04-02T08:52:43.503710+0000","last_clean":"2024-04-02T08:52:43.503710+0000","last_became_active":"2024-04-02T08:52:42.504150+0000","last_became_peered":"2024-04-02T08:52:42.504150+0000","last_unstale":"2024-04-02T08:52:43.503710+0000","last_undegraded":"2024-04-02T08:52:43.503710+0000","last_fullsized":"2024-04-02T08:52:43.503710+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:56:32.295359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.497903+0000","last_change":"2024-04-02T08:52:42.002985+0000","last_active":"2024-04-02T08:52:42.497903+0000","last_peered":"2024-04-02T08:52:42.497903+0000","last_clean":"2024-04-02T08:52:42.497903+0000","last_became_active":"2024-04-02T08:52:42.002727+0000","last_became_peered":"2024-04-02T08:52:42.002727+0000","last_unstale":"2024-04-02T08:52:42.497903+0000","last_undegraded":"2024-04-02T08:52:42.497903+0000","last_fullsized":"2024-04-02T08:52:42.497903+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:24:28.756704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.515969+0000","last_change":"2024-04-02T08:52:42.513627+0000","last_active":"2024-04-02T08:52:42.515969+0000","last_peered":"2024-04-02T08:52:42.515969+0000","last_clean":"2024-04-02T08:52:42.515969+0000","last_became_active":"2024-04-02T08:52:42.513174+0000","last_became_peered":"2024-04-02T08:52:42.513174+0000","last_unstale":"2024-04-02T08:52:42.515969+0000","last_undegraded":"2024-04-02T08:52:42.515969+0000","last_fullsized":"2024-04-02T08:52:42.515969+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:22.165775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.497665+0000","last_change":"2024-04-02T08:52:42.004657+0000","last_active":"2024-04-02T08:52:42.497665+0000","last_peered":"2024-04-02T08:52:42.497665+0000","last_clean":"2024-04-02T08:52:42.497665+0000","last_became_active":"2024-04-02T08:52:42.003264+0000","last_became_peered":"2024-04-02T08:52:42.003264+0000","last_unstale":"2024-04-02T08:52:42.497665+0000","last_undegraded":"2024-04-02T08:52:42.497665+0000","last_fullsized":"2024-04-02T08:52:42.497665+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:15:58.269557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503592+0000","last_change":"2024-04-02T08:52:42.505564+0000","last_active":"2024-04-02T08:52:43.503592+0000","last_peered":"2024-04-02T08:52:43.503592+0000","last_clean":"2024-04-02T08:52:43.503592+0000","last_became_active":"2024-04-02T08:52:42.502911+0000","last_became_peered":"2024-04-02T08:52:42.502911+0000","last_unstale":"2024-04-02T08:52:43.503592+0000","last_undegraded":"2024-04-02T08:52:43.503592+0000","last_fullsized":"2024-04-02T08:52:43.503592+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:45:26.025962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.503849+0000","last_change":"2024-04-02T08:52:42.002807+0000","last_active":"2024-04-02T08:52:42.503849+0000","last_peered":"2024-04-02T08:52:42.503849+0000","last_clean":"2024-04-02T08:52:42.503849+0000","last_became_active":"2024-04-02T08:52:42.002474+0000","last_became_peered":"2024-04-02T08:52:42.002474+0000","last_unstale":"2024-04-02T08:52:42.503849+0000","last_undegraded":"2024-04-02T08:52:42.503849+0000","last_fullsized":"2024-04-02T08:52:42.503849+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:01:08.178914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.520055+0000","last_change":"2024-04-02T08:52:42.511855+0000","last_active":"2024-04-02T08:52:42.520055+0000","last_peered":"2024-04-02T08:52:42.520055+0000","last_clean":"2024-04-02T08:52:42.520055+0000","last_became_active":"2024-04-02T08:52:42.511568+0000","last_became_peered":"2024-04-02T08:52:42.511568+0000","last_unstale":"2024-04-02T08:52:42.520055+0000","last_undegraded":"2024-04-02T08:52:42.520055+0000","last_fullsized":"2024-04-02T08:52:42.520055+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:06.680663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.7","version":"171'2","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.525290+0000","last_change":"2024-04-02T08:52:42.007221+0000","last_active":"2024-04-02T08:52:42.525290+0000","last_peered":"2024-04-02T08:52:42.525290+0000","last_clean":"2024-04-02T08:52:42.525290+0000","last_became_active":"2024-04-02T08:52:42.006357+0000","last_became_peered":"2024-04-02T08:52:42.006357+0000","last_unstale":"2024-04-02T08:52:42.525290+0000","last_undegraded":"2024-04-02T08:52:42.525290+0000","last_fullsized":"2024-04-02T08:52:42.525290+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:36:25.475809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.519664+0000","last_change":"2024-04-02T08:52:42.511983+0000","last_active":"2024-04-02T08:52:42.519664+0000","last_peered":"2024-04-02T08:52:42.519664+0000","last_clean":"2024-04-02T08:52:42.519664+0000","last_became_active":"2024-04-02T08:52:42.511729+0000","last_became_peered":"2024-04-02T08:52:42.511729+0000","last_unstale":"2024-04-02T08:52:42.519664+0000","last_undegraded":"2024-04-02T08:52:42.519664+0000","last_fullsized":"2024-04-02T08:52:42.519664+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:06:30.703910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504054+0000","last_change":"2024-04-02T08:52:42.012534+0000","last_active":"2024-04-02T08:52:43.504054+0000","last_peered":"2024-04-02T08:52:43.504054+0000","last_clean":"2024-04-02T08:52:43.504054+0000","last_became_active":"2024-04-02T08:52:42.012223+0000","last_became_peered":"2024-04-02T08:52:42.012223+0000","last_unstale":"2024-04-02T08:52:43.504054+0000","last_undegraded":"2024-04-02T08:52:43.504054+0000","last_fullsized":"2024-04-02T08:52:43.504054+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:41:26.162559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.505135+0000","last_change":"2024-04-02T08:52:42.509214+0000","last_active":"2024-04-02T08:52:43.505135+0000","last_peered":"2024-04-02T08:52:43.505135+0000","last_clean":"2024-04-02T08:52:43.505135+0000","last_became_active":"2024-04-02T08:52:42.508650+0000","last_became_peered":"2024-04-02T08:52:42.508650+0000","last_unstale":"2024-04-02T08:52:43.505135+0000","last_undegraded":"2024-04-02T08:52:43.505135+0000","last_fullsized":"2024-04-02T08:52:43.505135+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:15:29.237137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503777+0000","last_change":"2024-04-02T08:52:42.000283+0000","last_active":"2024-04-02T08:52:43.503777+0000","last_peered":"2024-04-02T08:52:43.503777+0000","last_clean":"2024-04-02T08:52:43.503777+0000","last_became_active":"2024-04-02T08:52:41.999538+0000","last_became_peered":"2024-04-02T08:52:41.999538+0000","last_unstale":"2024-04-02T08:52:43.503777+0000","last_undegraded":"2024-04-02T08:52:43.503777+0000","last_fullsized":"2024-04-02T08:52:43.503777+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:25.602471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.505197+0000","last_change":"2024-04-02T08:52:42.522912+0000","last_active":"2024-04-02T08:52:43.505197+0000","last_peered":"2024-04-02T08:52:43.505197+0000","last_clean":"2024-04-02T08:52:43.505197+0000","last_became_active":"2024-04-02T08:52:42.522432+0000","last_became_peered":"2024-04-02T08:52:42.522432+0000","last_unstale":"2024-04-02T08:52:43.505197+0000","last_undegraded":"2024-04-02T08:52:43.505197+0000","last_fullsized":"2024-04-02T08:52:43.505197+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:38:36.042236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503417+0000","last_change":"2024-04-02T08:52:42.005085+0000","last_active":"2024-04-02T08:52:43.503417+0000","last_peered":"2024-04-02T08:52:43.503417+0000","last_clean":"2024-04-02T08:52:43.503417+0000","last_became_active":"2024-04-02T08:52:42.003345+0000","last_became_peered":"2024-04-02T08:52:42.003345+0000","last_unstale":"2024-04-02T08:52:43.503417+0000","last_undegraded":"2024-04-02T08:52:43.503417+0000","last_fullsized":"2024-04-02T08:52:43.503417+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:09.652023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503040+0000","last_change":"2024-04-02T08:52:42.508364+0000","last_active":"2024-04-02T08:52:43.503040+0000","last_peered":"2024-04-02T08:52:43.503040+0000","last_clean":"2024-04-02T08:52:43.503040+0000","last_became_active":"2024-04-02T08:52:42.508065+0000","last_became_peered":"2024-04-02T08:52:42.508065+0000","last_unstale":"2024-04-02T08:52:43.503040+0000","last_undegraded":"2024-04-02T08:52:43.503040+0000","last_fullsized":"2024-04-02T08:52:43.503040+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:15:29.237137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503800+0000","last_change":"2024-04-02T08:52:42.004725+0000","last_active":"2024-04-02T08:52:43.503800+0000","last_peered":"2024-04-02T08:52:43.503800+0000","last_clean":"2024-04-02T08:52:43.503800+0000","last_became_active":"2024-04-02T08:52:42.004441+0000","last_became_peered":"2024-04-02T08:52:42.004441+0000","last_unstale":"2024-04-02T08:52:43.503800+0000","last_undegraded":"2024-04-02T08:52:43.503800+0000","last_fullsized":"2024-04-02T08:52:43.503800+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:32:22.762908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.519554+0000","last_change":"2024-04-02T08:52:42.508983+0000","last_active":"2024-04-02T08:52:42.519554+0000","last_peered":"2024-04-02T08:52:42.519554+0000","last_clean":"2024-04-02T08:52:42.519554+0000","last_became_active":"2024-04-02T08:52:42.508685+0000","last_became_peered":"2024-04-02T08:52:42.508685+0000","last_unstale":"2024-04-02T08:52:42.519554+0000","last_undegraded":"2024-04-02T08:52:42.519554+0000","last_fullsized":"2024-04-02T08:52:42.519554+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:47:48.659307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.494289+0000","last_change":"2024-04-02T08:52:42.004729+0000","last_active":"2024-04-02T08:52:42.494289+0000","last_peered":"2024-04-02T08:52:42.494289+0000","last_clean":"2024-04-02T08:52:42.494289+0000","last_became_active":"2024-04-02T08:52:42.004398+0000","last_became_peered":"2024-04-02T08:52:42.004398+0000","last_unstale":"2024-04-02T08:52:42.494289+0000","last_undegraded":"2024-04-02T08:52:42.494289+0000","last_fullsized":"2024-04-02T08:52:42.494289+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:30.565732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.520554+0000","last_change":"2024-04-02T08:52:42.507360+0000","last_active":"2024-04-02T08:52:42.520554+0000","last_peered":"2024-04-02T08:52:42.520554+0000","last_clean":"2024-04-02T08:52:42.520554+0000","last_became_active":"2024-04-02T08:52:42.507111+0000","last_became_peered":"2024-04-02T08:52:42.507111+0000","last_unstale":"2024-04-02T08:52:42.520554+0000","last_undegraded":"2024-04-02T08:52:42.520554+0000","last_fullsized":"2024-04-02T08:52:42.520554+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:15:40.194442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.495267+0000","last_change":"2024-04-02T08:52:42.010060+0000","last_active":"2024-04-02T08:52:42.495267+0000","last_peered":"2024-04-02T08:52:42.495267+0000","last_clean":"2024-04-02T08:52:42.495267+0000","last_became_active":"2024-04-02T08:52:42.009795+0000","last_became_peered":"2024-04-02T08:52:42.009795+0000","last_unstale":"2024-04-02T08:52:42.495267+0000","last_undegraded":"2024-04-02T08:52:42.495267+0000","last_fullsized":"2024-04-02T08:52:42.495267+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:10:54.483338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.509124+0000","last_change":"2024-04-02T08:52:42.501713+0000","last_active":"2024-04-02T08:52:42.509124+0000","last_peered":"2024-04-02T08:52:42.509124+0000","last_clean":"2024-04-02T08:52:42.509124+0000","last_became_active":"2024-04-02T08:52:42.501414+0000","last_became_peered":"2024-04-02T08:52:42.501414+0000","last_unstale":"2024-04-02T08:52:42.509124+0000","last_undegraded":"2024-04-02T08:52:42.509124+0000","last_fullsized":"2024-04-02T08:52:42.509124+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:46:24.377267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.985830+0000","last_change":"2024-04-02T08:52:42.011570+0000","last_active":"2024-04-02T08:52:43.985830+0000","last_peered":"2024-04-02T08:52:43.985830+0000","last_clean":"2024-04-02T08:52:43.985830+0000","last_became_active":"2024-04-02T08:52:42.011071+0000","last_became_peered":"2024-04-02T08:52:42.011071+0000","last_unstale":"2024-04-02T08:52:43.985830+0000","last_undegraded":"2024-04-02T08:52:43.985830+0000","last_fullsized":"2024-04-02T08:52:43.985830+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:18:04.756343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503221+0000","last_change":"2024-04-02T08:52:42.498110+0000","last_active":"2024-04-02T08:52:43.503221+0000","last_peered":"2024-04-02T08:52:43.503221+0000","last_clean":"2024-04-02T08:52:43.503221+0000","last_became_active":"2024-04-02T08:52:42.497098+0000","last_became_peered":"2024-04-02T08:52:42.497098+0000","last_unstale":"2024-04-02T08:52:43.503221+0000","last_undegraded":"2024-04-02T08:52:43.503221+0000","last_fullsized":"2024-04-02T08:52:43.503221+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:32:09.794631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.15","version":"171'3","reported_seq":13,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.522954+0000","last_change":"2024-04-02T08:52:42.011270+0000","last_active":"2024-04-02T08:52:42.522954+0000","last_peered":"2024-04-02T08:52:42.522954+0000","last_clean":"2024-04-02T08:52:42.522954+0000","last_became_active":"2024-04-02T08:52:42.010962+0000","last_became_peered":"2024-04-02T08:52:42.010962+0000","last_unstale":"2024-04-02T08:52:42.522954+0000","last_undegraded":"2024-04-02T08:52:42.522954+0000","last_fullsized":"2024-04-02T08:52:42.522954+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:40:37.133822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"161'271","reported_seq":881,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-04-02T08:52:41.995462+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T08:52:41.995462+0000","last_peered":"2024-04-02T08:52:41.995462+0000","last_clean":"2024-04-02T08:52:41.995462+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T08:52:41.995462+0000","last_undegraded":"2024-04-02T08:52:41.995462+0000","last_fullsized":"2024-04-02T08:52:41.995462+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575759,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55364,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316476,"statfs":{"total":96636764160,"available":96580071424,"internally_reserved":0,"allocated":2146304,"data_stored":1073653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.72599999999999998},"min":{"1min":0.56699999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":0.85799999999999998,"5min":1.069,"15min":1.069},"last":0.375},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.59799999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85899999999999999,"5min":1.0129999999999999,"15min":1.0169999999999999},"last":0.41499999999999998}]},{"osd":1,"last update":"Tue Apr 2 08:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.73599999999999999,"15min":0.73999999999999999},"min":{"1min":0.59299999999999997,"5min":0.51500000000000001,"15min":0.48499999999999999},"max":{"1min":0.93200000000000005,"5min":1.0089999999999999,"15min":1.028},"last":0.496},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72599999999999998,"15min":0.71699999999999997},"min":{"1min":0.56100000000000005,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.90200000000000002,"5min":1.016,"15min":1.016},"last":0.48099999999999998}]},{"osd":2,"last update":"Tue Apr 2 08:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.76000000000000001,"15min":0.76600000000000001},"min":{"1min":0.61799999999999999,"5min":0.46100000000000002,"15min":0.44400000000000001},"max":{"1min":0.88500000000000001,"5min":1.105,"15min":1.105},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81999999999999995,"15min":0.80900000000000005},"min":{"1min":0.65700000000000003,"5min":0.56999999999999995,"15min":0.497},"max":{"1min":0.93200000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.65800000000000003}]},{"osd":3,"last update":"Tue Apr 2 08:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.75700000000000001,"15min":0.746},"min":{"1min":0.52700000000000002,"5min":0.52500000000000002,"15min":0.48099999999999998},"max":{"1min":0.89500000000000002,"5min":1.073,"15min":1.073},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.76100000000000001,"15min":0.76400000000000001},"min":{"1min":0.58999999999999997,"5min":0.53200000000000003,"15min":0.49099999999999999},"max":{"1min":0.89200000000000002,"5min":1.048,"15min":1.056},"last":0.60899999999999999}]},{"osd":4,"last update":"Tue Apr 2 08:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.625,"15min":0.60099999999999998},"min":{"1min":0.48599999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81799999999999995,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.70999999999999996},"min":{"1min":0.47899999999999998,"5min":0.39700000000000002,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":0.997,"15min":1.014},"last":0.74099999999999999}]},{"osd":5,"last update":"Tue Apr 2 08:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68400000000000005,"15min":0.66600000000000004},"min":{"1min":0.47699999999999998,"5min":0.441,"15min":0.38700000000000001},"max":{"1min":0.95799999999999996,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72399999999999998,"15min":0.70899999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.41999999999999998},"max":{"1min":0.89700000000000002,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.78200000000000003}]},{"osd":6,"last update":"Tue Apr 2 08:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.66700000000000004},"min":{"1min":0.52800000000000002,"5min":0.47699999999999998,"15min":0.46400000000000002},"max":{"1min":0.82699999999999996,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.41299999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92800000000000005,"5min":0.98699999999999999,"15min":1.0149999999999999},"last":0.69399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575757,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37304,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334536,"statfs":{"total":96636764160,"available":96598564864,"internally_reserved":0,"allocated":2134016,"data_stored":1066933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.74099999999999999,"15min":0.73399999999999999},"min":{"1min":0.45100000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.96199999999999997,"5min":1.847,"15min":1.847},"last":1.1990000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.745,"15min":0.74199999999999999},"min":{"1min":0.52700000000000002,"5min":0.46600000000000003,"15min":0.40400000000000003},"max":{"1min":0.99399999999999999,"5min":1.641,"15min":1.641},"last":0.98799999999999999}]},{"osd":1,"last update":"Tue Apr 2 08:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.72799999999999998,"15min":0.72699999999999998},"min":{"1min":0.57099999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":1.117},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.76300000000000001,"15min":0.751},"min":{"1min":0.52700000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.93899999999999995,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.97399999999999998}]},{"osd":2,"last update":"Tue Apr 2 08:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.749,"15min":0.751},"min":{"1min":0.52400000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.91200000000000003,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":1.141},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.60399999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.95499999999999996,"5min":0.96799999999999997,"15min":1.0289999999999999},"last":1.006}]},{"osd":3,"last update":"Tue Apr 2 08:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.75800000000000001,"15min":0.754},"min":{"1min":0.57399999999999995,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0009999999999999,"5min":1.722,"15min":1.722},"last":1.238},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.77800000000000002,"15min":0.77600000000000002},"min":{"1min":0.57099999999999995,"5min":0.53200000000000003,"15min":0.52900000000000003},"max":{"1min":0.97899999999999998,"5min":1.329,"15min":1.329},"last":1.0249999999999999}]},{"osd":4,"last update":"Tue Apr 2 08:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.63800000000000001,"15min":0.623},"min":{"1min":0.45400000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85799999999999998,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67200000000000004,"15min":0.64900000000000002},"min":{"1min":0.45200000000000001,"5min":0.41499999999999998,"15min":0.41299999999999998},"max":{"1min":0.91400000000000003,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":1.069}]},{"osd":5,"last update":"Tue Apr 2 08:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.66600000000000004,"15min":0.63900000000000001},"min":{"1min":0.496,"5min":0.439,"15min":0.40100000000000002},"max":{"1min":0.92900000000000005,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72599999999999998,"15min":0.69899999999999995},"min":{"1min":0.56899999999999995,"5min":0.45400000000000001,"15min":0.43099999999999999},"max":{"1min":0.90900000000000003,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":1.0960000000000001}]},{"osd":7,"last update":"Tue Apr 2 08:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73499999999999999,"15min":0.70799999999999996},"min":{"1min":0.51000000000000001,"5min":0.41999999999999998,"15min":0.40999999999999998},"max":{"1min":0.96399999999999997,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.73699999999999999,"15min":0.71599999999999997},"min":{"1min":0.55000000000000004,"5min":0.48899999999999999,"15min":0.46300000000000002},"max":{"1min":0.94899999999999995,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":1.1819999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47744,"kb_used_data":2668,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324096,"statfs":{"total":96636764160,"available":96587874304,"internally_reserved":0,"allocated":2732032,"data_stored":1657371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.47399999999999998,"15min":0.46899999999999997},"min":{"1min":0.32700000000000001,"5min":0.313,"15min":0.27300000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.81100000000000005},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34899999999999998,"5min":0.34399999999999997,"15min":0.34100000000000003},"max":{"1min":0.66600000000000004,"5min":0.876,"15min":0.94799999999999995},"last":0.505}]},{"osd":2,"last update":"Tue Apr 2 08:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55200000000000005,"15min":0.53400000000000003},"min":{"1min":0.40200000000000002,"5min":0.35399999999999998,"15min":0.34000000000000002},"max":{"1min":0.64700000000000002,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.64600000000000002,"15min":0.61899999999999999},"min":{"1min":0.44900000000000001,"5min":0.44700000000000001,"15min":0.39200000000000002},"max":{"1min":0.73499999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.73899999999999999}]},{"osd":3,"last update":"Tue Apr 2 08:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.61899999999999999,"15min":0.60099999999999998},"min":{"1min":0.44800000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.68300000000000005,"5min":1.137,"15min":1.226},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58599999999999997,"15min":0.56999999999999995},"min":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.36399999999999999},"max":{"1min":0.72399999999999998,"5min":0.93200000000000005,"15min":1.0700000000000001},"last":0.46400000000000002}]},{"osd":4,"last update":"Tue Apr 2 08:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.60899999999999999,"5min":0.58999999999999997,"15min":0.54100000000000004},"max":{"1min":0.95099999999999996,"5min":1.1140000000000001,"15min":1.321},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.81699999999999995,"15min":0.81100000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.99299999999999999,"5min":1.381,"15min":1.381},"last":0.71299999999999997}]},{"osd":5,"last update":"Tue Apr 2 08:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.81699999999999995,"15min":0.80800000000000005},"min":{"1min":0.58599999999999997,"5min":0.57999999999999996,"15min":0.52100000000000002},"max":{"1min":0.94999999999999996,"5min":1.131,"15min":1.131},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.83699999999999997,"15min":0.82299999999999995},"min":{"1min":0.64000000000000001,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.046,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.77900000000000003}]},{"osd":6,"last update":"Tue Apr 2 08:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.84599999999999997,"15min":0.83299999999999996},"min":{"1min":0.627,"5min":0.627,"15min":0.57299999999999995},"max":{"1min":0.96699999999999997,"5min":1.2809999999999999,"15min":1.3759999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.83999999999999997,"15min":0.83199999999999996},"min":{"1min":0.69299999999999995,"5min":0.61299999999999999,"15min":0.53700000000000003},"max":{"1min":1.012,"5min":1.1839999999999999,"15min":1.242},"last":0.84199999999999997}]},{"osd":7,"last update":"Tue Apr 2 08:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.85899999999999999,"15min":0.84599999999999997},"min":{"1min":0.624,"5min":0.624,"15min":0.56299999999999994},"max":{"1min":1.071,"5min":1.347,"15min":1.347},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.85599999999999998,"15min":0.84699999999999998},"min":{"1min":0.68799999999999994,"5min":0.623,"15min":0.621},"max":{"1min":0.97399999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.75700000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575758,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47160,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324680,"statfs":{"total":96636764160,"available":96588472320,"internally_reserved":0,"allocated":2134016,"data_stored":1066004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.45400000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60099999999999998,"5min":0.64100000000000001,"15min":0.75700000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.5,"15min":0.498},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73499999999999999,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.44}]},{"osd":2,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.50700000000000001,"15min":0.50800000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.76200000000000001,"5min":0.80700000000000005,"15min":0.83299999999999996},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.54500000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73999999999999999,"5min":0.79700000000000004,"15min":0.80900000000000005},"last":0.55400000000000005}]},{"osd":3,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55700000000000005,"15min":0.56699999999999995},"min":{"1min":0.40799999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.86799999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63,"15min":0.64300000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81999999999999995,"5min":0.88100000000000001,"15min":0.91900000000000004},"last":0.46500000000000002}]},{"osd":4,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.83199999999999996,"15min":0.81899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.999,"5min":1.0980000000000001,"15min":1.236},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.83899999999999997,"15min":0.82899999999999996},"min":{"1min":0.61899999999999999,"5min":0.60199999999999998,"15min":0.56200000000000006},"max":{"1min":1.056,"5min":1.056,"15min":1.2929999999999999},"last":0.63300000000000001}]},{"osd":5,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.83199999999999996,"15min":0.82699999999999996},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.57799999999999996},"max":{"1min":0.95599999999999996,"5min":1.0820000000000001,"15min":1.369},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.53700000000000003},"max":{"1min":0.98899999999999999,"5min":1.042,"15min":1.27},"last":0.66000000000000003}]},{"osd":6,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.57999999999999996,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.052,"5min":1.052,"15min":1.1259999999999999},"last":0.622},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.80200000000000005,"15min":0.79600000000000004},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.99099999999999999,"5min":1.089,"15min":1.262},"last":0.80600000000000005}]},{"osd":7,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.85999999999999999,"15min":0.84999999999999998},"min":{"1min":0.57099999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0680000000000001,"5min":1.125,"15min":1.321},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.83299999999999996,"15min":0.84199999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.083,"5min":1.1120000000000001,"15min":1.2989999999999999},"last":0.66800000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575757,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45496,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94326344,"statfs":{"total":96636764160,"available":96590176256,"internally_reserved":0,"allocated":2138112,"data_stored":1066955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47599999999999998,"15min":0.45600000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.31},"max":{"1min":0.70099999999999996,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.50800000000000001,"15min":0.50700000000000001},"min":{"1min":0.40000000000000002,"5min":0.35099999999999998,"15min":0.316},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":0.81000000000000005},"last":0.53000000000000003}]},{"osd":1,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54100000000000004,"15min":0.51700000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.313},"max":{"1min":0.751,"5min":1.262,"15min":1.262},"last":0.433},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.39500000000000002,"5min":0.38,"15min":0.35599999999999998},"max":{"1min":0.72399999999999998,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.56100000000000005}]},{"osd":3,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.39400000000000002,"5min":0.38500000000000001,"15min":0.30599999999999999},"max":{"1min":2.3039999999999998,"5min":2.3039999999999998,"15min":2.3039999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.65700000000000003,"15min":0.64000000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.45000000000000001},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.51000000000000001}]},{"osd":4,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.81200000000000006,"15min":0.80500000000000005},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.443},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80300000000000005,"15min":0.80600000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.42399999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.73299999999999998}]},{"osd":5,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.82099999999999995,"15min":0.80600000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.48399999999999999},"max":{"1min":2.177,"5min":2.177,"15min":2.177},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.83399999999999996,"15min":0.81799999999999995},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.47899999999999998},"max":{"1min":1.319,"5min":1.319,"15min":1.3600000000000001},"last":0.74199999999999999}]},{"osd":6,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.47499999999999998},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.84299999999999997,"15min":0.83199999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.50700000000000001},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":0.84299999999999997}]},{"osd":7,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.84699999999999998,"15min":0.83799999999999997},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.502},"max":{"1min":1.528,"5min":1.528,"15min":1.528},"last":0.87},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.86199999999999999,"15min":0.84599999999999997},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.47399999999999998},"max":{"1min":2.258,"5min":2.258,"15min":2.258},"last":0.79600000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575757,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47164,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324676,"statfs":{"total":96636764160,"available":96588468224,"internally_reserved":0,"allocated":2142208,"data_stored":1067549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.45600000000000002,"15min":0.45200000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.28699999999999998},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.92700000000000005},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.49199999999999999,"15min":0.49299999999999999},"min":{"1min":0.35999999999999999,"5min":0.33200000000000002,"15min":0.30599999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.87},"last":0.36599999999999999}]},{"osd":1,"last update":"Tue Apr 2 08:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.39700000000000002,"5min":0.34699999999999998,"15min":0.309},"max":{"1min":0.90300000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59299999999999997,"15min":0.58099999999999996},"min":{"1min":0.45200000000000001,"5min":0.43099999999999999,"15min":0.40100000000000002},"max":{"1min":0.78200000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.44800000000000001}]},{"osd":2,"last update":"Tue Apr 2 08:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.63800000000000001,"15min":0.63600000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.39000000000000001},"max":{"1min":0.79500000000000004,"5min":0.83599999999999997,"15min":0.86299999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.629},"min":{"1min":0.54900000000000004,"5min":0.49399999999999999,"15min":0.40200000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.45900000000000002}]},{"osd":4,"last update":"Tue Apr 2 08:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.80900000000000005,"15min":0.78500000000000003},"min":{"1min":0.68100000000000005,"5min":0.56100000000000005,"15min":0.51500000000000001},"max":{"1min":1.0740000000000001,"5min":1.1000000000000001,"15min":1.167},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.81200000000000006,"15min":0.80100000000000005},"min":{"1min":0.67600000000000005,"5min":0.57999999999999996,"15min":0.54800000000000004},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.63900000000000001}]},{"osd":5,"last update":"Tue Apr 2 08:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.82199999999999995,"15min":0.80000000000000004},"min":{"1min":0.73699999999999999,"5min":0.61699999999999999,"15min":0.54600000000000004},"max":{"1min":1.018,"5min":1.115,"15min":1.149},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.81699999999999995,"15min":0.80000000000000004},"min":{"1min":0.63600000000000001,"5min":0.58799999999999997,"15min":0.52700000000000002},"max":{"1min":0.97799999999999998,"5min":1.0760000000000001,"15min":1.1020000000000001},"last":0.624}]},{"osd":6,"last update":"Tue Apr 2 08:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.82299999999999995,"15min":0.80500000000000005},"min":{"1min":0.622,"5min":0.622,"15min":0.56799999999999995},"max":{"1min":1.133,"5min":1.153,"15min":1.153},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86399999999999999,"15min":0.82899999999999996},"min":{"1min":0.64500000000000002,"5min":0.63700000000000001,"15min":0.55600000000000005},"max":{"1min":1.0680000000000001,"5min":1.143,"15min":1.1870000000000001},"last":0.745}]},{"osd":7,"last update":"Tue Apr 2 08:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.85499999999999998,"15min":0.82599999999999996},"min":{"1min":0.67100000000000004,"5min":0.624,"15min":0.57699999999999996},"max":{"1min":1.073,"5min":1.125,"15min":1.2070000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.871,"5min":0.86799999999999999,"15min":0.83199999999999996},"min":{"1min":0.67800000000000005,"5min":0.626,"15min":0.56499999999999995},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.73599999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575757,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59480,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312360,"statfs":{"total":96636764160,"available":96575856640,"internally_reserved":0,"allocated":2166784,"data_stored":1075821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.71499999999999997,"15min":0.70499999999999996},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.46999999999999997},"max":{"1min":0.92400000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.50900000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.85199999999999998,"5min":0.95999999999999996,"15min":1.0429999999999999},"last":0.629}]},{"osd":1,"last update":"Tue Apr 2 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.76100000000000001,"15min":0.75600000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.95999999999999996,"5min":1.095,"15min":1.121},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78000000000000003},"min":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.49099999999999999},"max":{"1min":0.97199999999999998,"5min":1.087,"15min":1.1200000000000001},"last":0.74199999999999999}]},{"osd":2,"last update":"Tue Apr 2 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.51000000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92700000000000005,"5min":0.97899999999999998,"15min":1.1879999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72199999999999998,"15min":0.73299999999999998},"min":{"1min":0.52700000000000002,"5min":0.47599999999999998,"15min":0.46300000000000002},"max":{"1min":0.91700000000000004,"5min":1.0209999999999999,"15min":1.1259999999999999},"last":0.68899999999999995}]},{"osd":3,"last update":"Tue Apr 2 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.753,"15min":0.76300000000000001},"min":{"1min":0.52800000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98599999999999999,"5min":0.997,"15min":1.2589999999999999},"last":0.755},{"interface":"front","average":{"1min":0.75,"5min":0.74099999999999999,"15min":0.74399999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.90200000000000002,"5min":1.0389999999999999,"15min":1.077},"last":0.54600000000000004}]},{"osd":5,"last update":"Tue Apr 2 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59399999999999997,"15min":0.60499999999999998},"min":{"1min":0.434,"5min":0.40400000000000003,"15min":0.38400000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":1.0940000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.60499999999999998,"15min":0.60399999999999998},"min":{"1min":0.438,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":1.073},"last":0.67700000000000005}]},{"osd":6,"last update":"Tue Apr 2 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.63400000000000001,"15min":0.63100000000000001},"min":{"1min":0.54200000000000004,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68999999999999995,"15min":0.69099999999999995},"min":{"1min":0.50700000000000001,"5min":0.46899999999999997,"15min":0.439},"max":{"1min":0.92900000000000005,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.85699999999999998}]},{"osd":7,"last update":"Tue Apr 2 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.68000000000000005,"15min":0.69099999999999995},"min":{"1min":0.497,"5min":0.46100000000000002,"15min":0.442},"max":{"1min":1.048,"5min":1.048,"15min":1.129},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.77000000000000002,"15min":0.77400000000000002},"min":{"1min":0.56699999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.3440000000000001},"last":0.78200000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575760,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64156,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307684,"statfs":{"total":96636764160,"available":96571068416,"internally_reserved":0,"allocated":2760704,"data_stored":1664760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.72899999999999998,"15min":0.72199999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.377},"max":{"1min":0.878,"5min":1.502,"15min":1.502},"last":0.433},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.749,"15min":0.73599999999999999},"min":{"1min":0.58799999999999997,"5min":0.51000000000000001,"15min":0.41999999999999998},"max":{"1min":0.84899999999999998,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.44800000000000001}]},{"osd":1,"last update":"Tue Apr 2 08:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.72499999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.433},"max":{"1min":0.88900000000000001,"5min":0.999,"15min":0.999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.73499999999999999,"15min":0.73699999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.92100000000000004,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.48899999999999999}]},{"osd":2,"last update":"Tue Apr 2 08:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.76500000000000001,"15min":0.746},"min":{"1min":0.55900000000000005,"5min":0.51700000000000002,"15min":0.47599999999999998},"max":{"1min":0.90900000000000003,"5min":1.232,"15min":1.232},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.747,"15min":0.747},"min":{"1min":0.57599999999999996,"5min":0.53400000000000003,"15min":0.51000000000000001},"max":{"1min":0.88500000000000001,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.747}]},{"osd":3,"last update":"Tue Apr 2 08:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.80500000000000005,"15min":0.79300000000000004},"min":{"1min":0.65300000000000002,"5min":0.64400000000000002,"15min":0.46800000000000003},"max":{"1min":0.93799999999999994,"5min":1.556,"15min":1.556},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80200000000000005,"15min":0.80000000000000004},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.95399999999999996,"5min":1.0189999999999999,"15min":1.069},"last":0.71999999999999997}]},{"osd":4,"last update":"Tue Apr 2 08:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.626,"15min":0.61599999999999999},"min":{"1min":0.437,"5min":0.40200000000000002,"15min":0.32700000000000001},"max":{"1min":0.89500000000000002,"5min":0.97899999999999998,"15min":1.0189999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.435},"max":{"1min":0.94099999999999995,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.63400000000000001}]},{"osd":6,"last update":"Tue Apr 2 08:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.65900000000000003,"15min":0.66800000000000004},"min":{"1min":0.51300000000000001,"5min":0.47399999999999998,"15min":0.44700000000000001},"max":{"1min":0.89800000000000002,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68600000000000005,"15min":0.69499999999999995},"min":{"1min":0.503,"5min":0.46999999999999997,"15min":0.377},"max":{"1min":0.97199999999999998,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.58299999999999996}]},{"osd":7,"last update":"Tue Apr 2 08:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.47299999999999998,"5min":0.45700000000000002,"15min":0.41599999999999998},"max":{"1min":0.995,"5min":1.423,"15min":1.423},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.73599999999999999,"15min":0.748},"min":{"1min":0.58699999999999997,"5min":0.51500000000000001,"15min":0.44500000000000001},"max":{"1min":1.004,"5min":1.038,"15min":1.038},"last":0.77900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T08:52:46.878 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T08:52:47.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.107+0000 7f519ea00700 1 Processor -- start 2024-04-02T08:52:47.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.107+0000 7f519ea00700 1 -- start start 2024-04-02T08:52:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.108+0000 7f519ea00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f519815b5d0 0x7f51980ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.108+0000 7f519ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51980ac7d0 0x7f519806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.109+0000 7f519ea00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519815fd50 con 0x7f51980cb3f0 2024-04-02T08:52:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.109+0000 7f519ea00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f519815b9b0 con 0x7f51980ac7d0 2024-04-02T08:52:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.109+0000 7f519ea00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f519815f140 con 0x7f519815b5d0 2024-04-02T08:52:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.109+0000 7f519ca00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51980ac7d0 0x7f519806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.109+0000 7f5197400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f51980cb3f0 0x7f51980ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42808/0 (socket says 172.21.15.89:42808) 2024-04-02T08:52:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.109+0000 7f5197400700 1 -- 172.21.15.89:0/1584086297 learned_addr learned my addr 172.21.15.89:0/1584086297 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.109+0000 7f5197e00700 1 --2- 172.21.15.89:0/1584086297 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f519815b5d0 0x7f51980ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.109+0000 7f519ca00700 1 -- 172.21.15.89:0/1584086297 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f519815b5d0 msgr2=0x7f51980ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.109+0000 7f519ca00700 1 --2- 172.21.15.89:0/1584086297 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f519815b5d0 0x7f51980ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.109+0000 7f519ca00700 1 -- 172.21.15.89:0/1584086297 >> v1:172.21.15.89:6789/0 conn(0x7f51980cb3f0 legacy=0x7f51980ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.109+0000 7f519ca00700 1 -- 172.21.15.89:0/1584086297 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519815f420 con 0x7f51980ac7d0 2024-04-02T08:52:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.109+0000 7f5196a00700 1 -- 172.21.15.89:0/1584086297 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123637182 0 0) 0x7f519815fd50 con 0x7f51980cb3f0 2024-04-02T08:52:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.110+0000 7f519ca00700 1 --2- 172.21.15.89:0/1584086297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51980ac7d0 0x7f519806c9d0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f5190009d90 tx=0x7f519000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11d0cc26fefa887 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.110+0000 7f5196a00700 1 -- 172.21.15.89:0/1584086297 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519000e070 con 0x7f51980ac7d0 2024-04-02T08:52:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.110+0000 7f5196a00700 1 -- 172.21.15.89:0/1584086297 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519000bb20 con 0x7f51980ac7d0 2024-04-02T08:52:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.110+0000 7f5196a00700 1 -- 172.21.15.89:0/1584086297 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5190012430 con 0x7f51980ac7d0 2024-04-02T08:52:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.110+0000 7f519ea00700 1 -- 172.21.15.89:0/1584086297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51980ac7d0 msgr2=0x7f519806c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.110+0000 7f519ea00700 1 --2- 172.21.15.89:0/1584086297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51980ac7d0 0x7f519806c9d0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f5190009d90 tx=0x7f519000b330 comp rx=0 tx=0).stop 2024-04-02T08:52:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.110+0000 7f519ea00700 1 -- 172.21.15.89:0/1584086297 shutdown_connections 2024-04-02T08:52:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.110+0000 7f519ea00700 1 --2- 172.21.15.89:0/1584086297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51980ac7d0 0x7f519806c9d0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.110+0000 7f519ea00700 1 --2- 172.21.15.89:0/1584086297 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f519815b5d0 0x7f51980ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.110+0000 7f519ea00700 1 -- 172.21.15.89:0/1584086297 >> 172.21.15.89:0/1584086297 conn(0x7f51980cf190 msgr2=0x7f51980cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.110+0000 7f519ea00700 1 -- 172.21.15.89:0/1584086297 shutdown_connections 2024-04-02T08:52:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.111+0000 7f519ea00700 1 -- 172.21.15.89:0/1584086297 wait complete. 2024-04-02T08:52:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.113+0000 7f519ea00700 1 Processor -- start 2024-04-02T08:52:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.113+0000 7f519ea00700 1 -- start start 2024-04-02T08:52:47.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.114+0000 7f519ea00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51980ac7d0 0x7f51981f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:47.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.114+0000 7f519ea00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f519815b5d0 0x7f5198154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:47.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.114+0000 7f519ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5198154d50 0x7f51980cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:47.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.114+0000 7f519ea00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51980c8460 con 0x7f5198154d50 2024-04-02T08:52:47.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.114+0000 7f519ea00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51981551e0 con 0x7f519815b5d0 2024-04-02T08:52:47.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.114+0000 7f519ea00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51980cded0 con 0x7f51980ac7d0 2024-04-02T08:52:47.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.114+0000 7f519ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5198154d50 0x7f51980cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:47.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.114+0000 7f519ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5198154d50 0x7f51980cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52700/0 (socket says 172.21.15.89:52700) 2024-04-02T08:52:47.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.114+0000 7f519ca00700 1 -- 172.21.15.89:0/2782688188 learned_addr learned my addr 172.21.15.89:0/2782688188 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:47.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.114+0000 7f5197e00700 1 --2- 172.21.15.89:0/2782688188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51980ac7d0 0x7f51981f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:47.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.115+0000 7f5197400700 1 --2- 172.21.15.89:0/2782688188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f519815b5d0 0x7f5198154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:47.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.115+0000 7f519ca00700 1 -- 172.21.15.89:0/2782688188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51980ac7d0 msgr2=0x7f51981f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:47.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.115+0000 7f519ca00700 1 --2- 172.21.15.89:0/2782688188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51980ac7d0 0x7f51981f2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.115+0000 7f519ca00700 1 -- 172.21.15.89:0/2782688188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f519815b5d0 msgr2=0x7f5198154810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:47.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.115+0000 7f519ca00700 1 --2- 172.21.15.89:0/2782688188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f519815b5d0 0x7f5198154810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.115+0000 7f519ca00700 1 -- 172.21.15.89:0/2782688188 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5190009a70 con 0x7f5198154d50 2024-04-02T08:52:47.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.115+0000 7f519ca00700 1 --2- 172.21.15.89:0/2782688188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5198154d50 0x7f51980cd1e0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f5190013000 tx=0x7f5190004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:47.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.115+0000 7f518fe00700 1 -- 172.21.15.89:0/2782688188 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5190012430 con 0x7f5198154d50 2024-04-02T08:52:47.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.115+0000 7f518fe00700 1 -- 172.21.15.89:0/2782688188 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5190020070 con 0x7f5198154d50 2024-04-02T08:52:47.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.115+0000 7f519ea00700 1 -- 172.21.15.89:0/2782688188 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51980709c0 con 0x7f5198154d50 2024-04-02T08:52:47.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.115+0000 7f518fe00700 1 -- 172.21.15.89:0/2782688188 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5190004280 con 0x7f5198154d50 2024-04-02T08:52:47.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.116+0000 7f519ea00700 1 -- 172.21.15.89:0/2782688188 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5198070f80 con 0x7f5198154d50 2024-04-02T08:52:47.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.117+0000 7f519ea00700 1 -- 172.21.15.89:0/2782688188 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519806f2d0 con 0x7f5198154d50 2024-04-02T08:52:47.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.117+0000 7f518fe00700 1 -- 172.21.15.89:0/2782688188 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f51900043e0 con 0x7f5198154d50 2024-04-02T08:52:47.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.118+0000 7f518fe00700 1 --2- 172.21.15.89:0/2782688188 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f517806adb0 0x7f517806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:47.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.118+0000 7f518fe00700 1 -- 172.21.15.89:0/2782688188 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f5190022070 con 0x7f5198154d50 2024-04-02T08:52:47.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.119+0000 7f5197e00700 1 --2- 172.21.15.89:0/2782688188 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f517806adb0 0x7f517806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:47.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.119+0000 7f5197e00700 1 --2- 172.21.15.89:0/2782688188 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f517806adb0 0x7f517806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5188009b40 tx=0x7f5188009210 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:47.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.123+0000 7f518fe00700 1 -- 172.21.15.89:0/2782688188 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f51900561f0 con 0x7f5198154d50 2024-04-02T08:52:47.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.318+0000 7f519ea00700 1 -- 172.21.15.89:0/2782688188 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f519815f9e0 con 0x7f517806adb0 2024-04-02T08:52:47.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.326+0000 7f518fe00700 1 -- 172.21.15.89:0/2782688188 <== mgr.9318 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289409 (secure 0 0 0) 0x7f5188011810 con 0x7f517806adb0 2024-04-02T08:52:47.329 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:47.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.331+0000 7f519ea00700 1 -- 172.21.15.89:0/2782688188 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f517806adb0 msgr2=0x7f517806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:47.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.331+0000 7f519ea00700 1 --2- 172.21.15.89:0/2782688188 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f517806adb0 0x7f517806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5188009b40 tx=0x7f5188009210 comp rx=0 tx=0).stop 2024-04-02T08:52:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.331+0000 7f519ea00700 1 -- 172.21.15.89:0/2782688188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5198154d50 msgr2=0x7f51980cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.331+0000 7f519ea00700 1 --2- 172.21.15.89:0/2782688188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5198154d50 0x7f51980cd1e0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f5190013000 tx=0x7f5190004ad0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.331+0000 7f519ea00700 1 -- 172.21.15.89:0/2782688188 shutdown_connections 2024-04-02T08:52:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.331+0000 7f519ea00700 1 --2- 172.21.15.89:0/2782688188 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f517806adb0 0x7f517806d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.331+0000 7f519ea00700 1 --2- 172.21.15.89:0/2782688188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5198154d50 0x7f51980cd1e0 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.331+0000 7f519ea00700 1 --2- 172.21.15.89:0/2782688188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51980ac7d0 0x7f51981f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.331+0000 7f519ea00700 1 --2- 172.21.15.89:0/2782688188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f519815b5d0 0x7f5198154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.331+0000 7f519ea00700 1 -- 172.21.15.89:0/2782688188 >> 172.21.15.89:0/2782688188 conn(0x7f51980cf190 msgr2=0x7f51980aed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.331+0000 7f519ea00700 1 -- 172.21.15.89:0/2782688188 shutdown_connections 2024-04-02T08:52:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.331+0000 7f519ea00700 1 -- 172.21.15.89:0/2782688188 wait complete. 2024-04-02T08:52:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T08:52:47.348 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T08:52:46.367399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":403868,"kb_used_data":17924,"kb_used_omap":12,"kb_used_meta":385779,"kb_avail":754570852,"statfs":{"total":773094113280,"available":772680552448,"internally_reserved":0,"allocated":18354176,"data_stored":9739046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":395038264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.390562"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.505423+0000","last_change":"2024-04-02T08:52:42.523267+0000","last_active":"2024-04-02T08:52:43.505423+0000","last_peered":"2024-04-02T08:52:43.505423+0000","last_clean":"2024-04-02T08:52:43.505423+0000","last_became_active":"2024-04-02T08:52:42.522160+0000","last_became_peered":"2024-04-02T08:52:42.522160+0000","last_unstale":"2024-04-02T08:52:43.505423+0000","last_undegraded":"2024-04-02T08:52:43.505423+0000","last_fullsized":"2024-04-02T08:52:43.505423+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:17:44.210549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504072+0000","last_change":"2024-04-02T08:52:42.510935+0000","last_active":"2024-04-02T08:52:43.504072+0000","last_peered":"2024-04-02T08:52:43.504072+0000","last_clean":"2024-04-02T08:52:43.504072+0000","last_became_active":"2024-04-02T08:52:42.510488+0000","last_became_peered":"2024-04-02T08:52:42.510488+0000","last_unstale":"2024-04-02T08:52:43.504072+0000","last_undegraded":"2024-04-02T08:52:43.504072+0000","last_fullsized":"2024-04-02T08:52:43.504072+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:30:49.616210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524246+0000","last_change":"2024-04-02T08:52:42.521836+0000","last_active":"2024-04-02T08:52:42.524246+0000","last_peered":"2024-04-02T08:52:42.524246+0000","last_clean":"2024-04-02T08:52:42.524246+0000","last_became_active":"2024-04-02T08:52:42.521074+0000","last_became_peered":"2024-04-02T08:52:42.521074+0000","last_unstale":"2024-04-02T08:52:42.524246+0000","last_undegraded":"2024-04-02T08:52:42.524246+0000","last_fullsized":"2024-04-02T08:52:42.524246+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:16:55.801303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.526929+0000","last_change":"2024-04-02T08:52:42.521870+0000","last_active":"2024-04-02T08:52:42.526929+0000","last_peered":"2024-04-02T08:52:42.526929+0000","last_clean":"2024-04-02T08:52:42.526929+0000","last_became_active":"2024-04-02T08:52:42.521562+0000","last_became_peered":"2024-04-02T08:52:42.521562+0000","last_unstale":"2024-04-02T08:52:42.526929+0000","last_undegraded":"2024-04-02T08:52:42.526929+0000","last_fullsized":"2024-04-02T08:52:42.526929+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:45:26.025962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.523110+0000","last_change":"2024-04-02T08:52:42.521162+0000","last_active":"2024-04-02T08:52:42.523110+0000","last_peered":"2024-04-02T08:52:42.523110+0000","last_clean":"2024-04-02T08:52:42.523110+0000","last_became_active":"2024-04-02T08:52:42.520895+0000","last_became_peered":"2024-04-02T08:52:42.520895+0000","last_unstale":"2024-04-02T08:52:42.523110+0000","last_undegraded":"2024-04-02T08:52:42.523110+0000","last_fullsized":"2024-04-02T08:52:42.523110+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:33:34.561166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986097+0000","last_change":"2024-04-02T08:52:42.525666+0000","last_active":"2024-04-02T08:52:43.986097+0000","last_peered":"2024-04-02T08:52:43.986097+0000","last_clean":"2024-04-02T08:52:43.986097+0000","last_became_active":"2024-04-02T08:52:42.525382+0000","last_became_peered":"2024-04-02T08:52:42.525382+0000","last_unstale":"2024-04-02T08:52:43.986097+0000","last_undegraded":"2024-04-02T08:52:43.986097+0000","last_fullsized":"2024-04-02T08:52:43.986097+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:31.170482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.502872+0000","last_change":"2024-04-02T08:52:42.514048+0000","last_active":"2024-04-02T08:52:43.502872+0000","last_peered":"2024-04-02T08:52:43.502872+0000","last_clean":"2024-04-02T08:52:43.502872+0000","last_became_active":"2024-04-02T08:52:42.513772+0000","last_became_peered":"2024-04-02T08:52:42.513772+0000","last_unstale":"2024-04-02T08:52:43.502872+0000","last_undegraded":"2024-04-02T08:52:43.502872+0000","last_fullsized":"2024-04-02T08:52:43.502872+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:07.399026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504649+0000","last_change":"2024-04-02T08:52:42.515605+0000","last_active":"2024-04-02T08:52:43.504649+0000","last_peered":"2024-04-02T08:52:43.504649+0000","last_clean":"2024-04-02T08:52:43.504649+0000","last_became_active":"2024-04-02T08:52:42.515316+0000","last_became_peered":"2024-04-02T08:52:42.515316+0000","last_unstale":"2024-04-02T08:52:43.504649+0000","last_undegraded":"2024-04-02T08:52:43.504649+0000","last_fullsized":"2024-04-02T08:52:43.504649+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:02:40.043066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986293+0000","last_change":"2024-04-02T08:52:42.515813+0000","last_active":"2024-04-02T08:52:43.986293+0000","last_peered":"2024-04-02T08:52:43.986293+0000","last_clean":"2024-04-02T08:52:43.986293+0000","last_became_active":"2024-04-02T08:52:42.515542+0000","last_became_peered":"2024-04-02T08:52:42.515542+0000","last_unstale":"2024-04-02T08:52:43.986293+0000","last_undegraded":"2024-04-02T08:52:43.986293+0000","last_fullsized":"2024-04-02T08:52:43.986293+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:32:09.597113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.519687+0000","last_change":"2024-04-02T08:52:42.510156+0000","last_active":"2024-04-02T08:52:42.519687+0000","last_peered":"2024-04-02T08:52:42.519687+0000","last_clean":"2024-04-02T08:52:42.519687+0000","last_became_active":"2024-04-02T08:52:42.504894+0000","last_became_peered":"2024-04-02T08:52:42.504894+0000","last_unstale":"2024-04-02T08:52:42.519687+0000","last_undegraded":"2024-04-02T08:52:42.519687+0000","last_fullsized":"2024-04-02T08:52:42.519687+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:15:32.537347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986186+0000","last_change":"2024-04-02T08:52:42.515909+0000","last_active":"2024-04-02T08:52:43.986186+0000","last_peered":"2024-04-02T08:52:43.986186+0000","last_clean":"2024-04-02T08:52:43.986186+0000","last_became_active":"2024-04-02T08:52:42.513069+0000","last_became_peered":"2024-04-02T08:52:42.513069+0000","last_unstale":"2024-04-02T08:52:43.986186+0000","last_undegraded":"2024-04-02T08:52:43.986186+0000","last_fullsized":"2024-04-02T08:52:43.986186+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:39:21.005383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.515332+0000","last_change":"2024-04-02T08:52:42.512377+0000","last_active":"2024-04-02T08:52:42.515332+0000","last_peered":"2024-04-02T08:52:42.515332+0000","last_clean":"2024-04-02T08:52:42.515332+0000","last_became_active":"2024-04-02T08:52:42.510307+0000","last_became_peered":"2024-04-02T08:52:42.510307+0000","last_unstale":"2024-04-02T08:52:42.515332+0000","last_undegraded":"2024-04-02T08:52:42.515332+0000","last_fullsized":"2024-04-02T08:52:42.515332+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:39:38.170946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.523129+0000","last_change":"2024-04-02T08:52:42.522105+0000","last_active":"2024-04-02T08:52:42.523129+0000","last_peered":"2024-04-02T08:52:42.523129+0000","last_clean":"2024-04-02T08:52:42.523129+0000","last_became_active":"2024-04-02T08:52:42.519471+0000","last_became_peered":"2024-04-02T08:52:42.519471+0000","last_unstale":"2024-04-02T08:52:42.523129+0000","last_undegraded":"2024-04-02T08:52:42.523129+0000","last_fullsized":"2024-04-02T08:52:42.523129+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:15.923579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.527693+0000","last_change":"2024-04-02T08:52:42.526972+0000","last_active":"2024-04-02T08:52:42.527693+0000","last_peered":"2024-04-02T08:52:42.527693+0000","last_clean":"2024-04-02T08:52:42.527693+0000","last_became_active":"2024-04-02T08:52:42.526258+0000","last_became_peered":"2024-04-02T08:52:42.526258+0000","last_unstale":"2024-04-02T08:52:42.527693+0000","last_undegraded":"2024-04-02T08:52:42.527693+0000","last_fullsized":"2024-04-02T08:52:42.527693+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:15:06.853630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.506226+0000","last_change":"2024-04-02T08:52:42.503486+0000","last_active":"2024-04-02T08:52:42.506226+0000","last_peered":"2024-04-02T08:52:42.506226+0000","last_clean":"2024-04-02T08:52:42.506226+0000","last_became_active":"2024-04-02T08:52:42.503249+0000","last_became_peered":"2024-04-02T08:52:42.503249+0000","last_unstale":"2024-04-02T08:52:42.506226+0000","last_undegraded":"2024-04-02T08:52:42.506226+0000","last_fullsized":"2024-04-02T08:52:42.506226+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:32:04.541950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.502832+0000","last_change":"2024-04-02T08:52:42.497825+0000","last_active":"2024-04-02T08:52:43.502832+0000","last_peered":"2024-04-02T08:52:43.502832+0000","last_clean":"2024-04-02T08:52:43.502832+0000","last_became_active":"2024-04-02T08:52:42.497517+0000","last_became_peered":"2024-04-02T08:52:42.497517+0000","last_unstale":"2024-04-02T08:52:43.502832+0000","last_undegraded":"2024-04-02T08:52:43.502832+0000","last_fullsized":"2024-04-02T08:52:43.502832+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:10:35.665594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.522854+0000","last_change":"2024-04-02T08:52:42.511203+0000","last_active":"2024-04-02T08:52:42.522854+0000","last_peered":"2024-04-02T08:52:42.522854+0000","last_clean":"2024-04-02T08:52:42.522854+0000","last_became_active":"2024-04-02T08:52:42.510938+0000","last_became_peered":"2024-04-02T08:52:42.510938+0000","last_unstale":"2024-04-02T08:52:42.522854+0000","last_undegraded":"2024-04-02T08:52:42.522854+0000","last_fullsized":"2024-04-02T08:52:42.522854+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:39.813595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.521678+0000","last_change":"2024-04-02T08:52:42.517643+0000","last_active":"2024-04-02T08:52:42.521678+0000","last_peered":"2024-04-02T08:52:42.521678+0000","last_clean":"2024-04-02T08:52:42.521678+0000","last_became_active":"2024-04-02T08:52:42.517396+0000","last_became_peered":"2024-04-02T08:52:42.517396+0000","last_unstale":"2024-04-02T08:52:42.521678+0000","last_undegraded":"2024-04-02T08:52:42.521678+0000","last_fullsized":"2024-04-02T08:52:42.521678+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:36:26.482902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.505050+0000","last_change":"2024-04-02T08:52:42.519398+0000","last_active":"2024-04-02T08:52:43.505050+0000","last_peered":"2024-04-02T08:52:43.505050+0000","last_clean":"2024-04-02T08:52:43.505050+0000","last_became_active":"2024-04-02T08:52:42.518839+0000","last_became_peered":"2024-04-02T08:52:42.518839+0000","last_unstale":"2024-04-02T08:52:43.505050+0000","last_undegraded":"2024-04-02T08:52:43.505050+0000","last_fullsized":"2024-04-02T08:52:43.505050+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:10:35.665594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.505006+0000","last_change":"2024-04-02T08:52:42.504263+0000","last_active":"2024-04-02T08:52:43.505006+0000","last_peered":"2024-04-02T08:52:43.505006+0000","last_clean":"2024-04-02T08:52:43.505006+0000","last_became_active":"2024-04-02T08:52:42.503847+0000","last_became_peered":"2024-04-02T08:52:42.503847+0000","last_unstale":"2024-04-02T08:52:43.505006+0000","last_undegraded":"2024-04-02T08:52:43.505006+0000","last_fullsized":"2024-04-02T08:52:43.505006+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:07.399026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524322+0000","last_change":"2024-04-02T08:52:42.515542+0000","last_active":"2024-04-02T08:52:42.524322+0000","last_peered":"2024-04-02T08:52:42.524322+0000","last_clean":"2024-04-02T08:52:42.524322+0000","last_became_active":"2024-04-02T08:52:42.510200+0000","last_became_peered":"2024-04-02T08:52:42.510200+0000","last_unstale":"2024-04-02T08:52:42.524322+0000","last_undegraded":"2024-04-02T08:52:42.524322+0000","last_fullsized":"2024-04-02T08:52:42.524322+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:56:27.893975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.506341+0000","last_change":"2024-04-02T08:52:42.504094+0000","last_active":"2024-04-02T08:52:42.506341+0000","last_peered":"2024-04-02T08:52:42.506341+0000","last_clean":"2024-04-02T08:52:42.506341+0000","last_became_active":"2024-04-02T08:52:42.503556+0000","last_became_peered":"2024-04-02T08:52:42.503556+0000","last_unstale":"2024-04-02T08:52:42.506341+0000","last_undegraded":"2024-04-02T08:52:42.506341+0000","last_fullsized":"2024-04-02T08:52:42.506341+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:55:39.440345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.527280+0000","last_change":"2024-04-02T08:52:42.525035+0000","last_active":"2024-04-02T08:52:42.527280+0000","last_peered":"2024-04-02T08:52:42.527280+0000","last_clean":"2024-04-02T08:52:42.527280+0000","last_became_active":"2024-04-02T08:52:42.520037+0000","last_became_peered":"2024-04-02T08:52:42.520037+0000","last_unstale":"2024-04-02T08:52:42.527280+0000","last_undegraded":"2024-04-02T08:52:42.527280+0000","last_fullsized":"2024-04-02T08:52:42.527280+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:39:21.005383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.519234+0000","last_change":"2024-04-02T08:52:42.516592+0000","last_active":"2024-04-02T08:52:42.519234+0000","last_peered":"2024-04-02T08:52:42.519234+0000","last_clean":"2024-04-02T08:52:42.519234+0000","last_became_active":"2024-04-02T08:52:42.515697+0000","last_became_peered":"2024-04-02T08:52:42.515697+0000","last_unstale":"2024-04-02T08:52:42.519234+0000","last_undegraded":"2024-04-02T08:52:42.519234+0000","last_fullsized":"2024-04-02T08:52:42.519234+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:59:01.321730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504078+0000","last_change":"2024-04-02T08:52:42.507409+0000","last_active":"2024-04-02T08:52:43.504078+0000","last_peered":"2024-04-02T08:52:43.504078+0000","last_clean":"2024-04-02T08:52:43.504078+0000","last_became_active":"2024-04-02T08:52:42.507123+0000","last_became_peered":"2024-04-02T08:52:42.507123+0000","last_unstale":"2024-04-02T08:52:43.504078+0000","last_undegraded":"2024-04-02T08:52:43.504078+0000","last_fullsized":"2024-04-02T08:52:43.504078+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:39:21.005383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.505237+0000","last_change":"2024-04-02T08:52:42.506905+0000","last_active":"2024-04-02T08:52:43.505237+0000","last_peered":"2024-04-02T08:52:43.505237+0000","last_clean":"2024-04-02T08:52:43.505237+0000","last_became_active":"2024-04-02T08:52:42.505098+0000","last_became_peered":"2024-04-02T08:52:42.505098+0000","last_unstale":"2024-04-02T08:52:43.505237+0000","last_undegraded":"2024-04-02T08:52:43.505237+0000","last_fullsized":"2024-04-02T08:52:43.505237+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:17:18.662540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503947+0000","last_change":"2024-04-02T08:52:42.503146+0000","last_active":"2024-04-02T08:52:43.503947+0000","last_peered":"2024-04-02T08:52:43.503947+0000","last_clean":"2024-04-02T08:52:43.503947+0000","last_became_active":"2024-04-02T08:52:42.502862+0000","last_became_peered":"2024-04-02T08:52:42.502862+0000","last_unstale":"2024-04-02T08:52:43.503947+0000","last_undegraded":"2024-04-02T08:52:43.503947+0000","last_fullsized":"2024-04-02T08:52:43.503947+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:08:18.767645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503507+0000","last_change":"2024-04-02T08:52:42.509837+0000","last_active":"2024-04-02T08:52:43.503507+0000","last_peered":"2024-04-02T08:52:43.503507+0000","last_clean":"2024-04-02T08:52:43.503507+0000","last_became_active":"2024-04-02T08:52:42.509587+0000","last_became_peered":"2024-04-02T08:52:42.509587+0000","last_unstale":"2024-04-02T08:52:43.503507+0000","last_undegraded":"2024-04-02T08:52:43.503507+0000","last_fullsized":"2024-04-02T08:52:43.503507+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:06.680663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504864+0000","last_change":"2024-04-02T08:52:42.518262+0000","last_active":"2024-04-02T08:52:43.504864+0000","last_peered":"2024-04-02T08:52:43.504864+0000","last_clean":"2024-04-02T08:52:43.504864+0000","last_became_active":"2024-04-02T08:52:42.517816+0000","last_became_peered":"2024-04-02T08:52:42.517816+0000","last_unstale":"2024-04-02T08:52:43.504864+0000","last_undegraded":"2024-04-02T08:52:43.504864+0000","last_fullsized":"2024-04-02T08:52:43.504864+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:27:40.539739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.506366+0000","last_change":"2024-04-02T08:52:42.504235+0000","last_active":"2024-04-02T08:52:42.506366+0000","last_peered":"2024-04-02T08:52:42.506366+0000","last_clean":"2024-04-02T08:52:42.506366+0000","last_became_active":"2024-04-02T08:52:42.503854+0000","last_became_peered":"2024-04-02T08:52:42.503854+0000","last_unstale":"2024-04-02T08:52:42.506366+0000","last_undegraded":"2024-04-02T08:52:42.506366+0000","last_fullsized":"2024-04-02T08:52:42.506366+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:50.168508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503534+0000","last_change":"2024-04-02T08:52:42.509956+0000","last_active":"2024-04-02T08:52:43.503534+0000","last_peered":"2024-04-02T08:52:43.503534+0000","last_clean":"2024-04-02T08:52:43.503534+0000","last_became_active":"2024-04-02T08:52:42.504622+0000","last_became_peered":"2024-04-02T08:52:42.504622+0000","last_unstale":"2024-04-02T08:52:43.503534+0000","last_undegraded":"2024-04-02T08:52:43.503534+0000","last_fullsized":"2024-04-02T08:52:43.503534+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:18:36.365111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503102+0000","last_change":"2024-04-02T08:52:42.515515+0000","last_active":"2024-04-02T08:52:43.503102+0000","last_peered":"2024-04-02T08:52:43.503102+0000","last_clean":"2024-04-02T08:52:43.503102+0000","last_became_active":"2024-04-02T08:52:42.515000+0000","last_became_peered":"2024-04-02T08:52:42.515000+0000","last_unstale":"2024-04-02T08:52:43.503102+0000","last_undegraded":"2024-04-02T08:52:43.503102+0000","last_fullsized":"2024-04-02T08:52:43.503102+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:15:39.710604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524555+0000","last_change":"2024-04-02T08:52:42.521825+0000","last_active":"2024-04-02T08:52:42.524555+0000","last_peered":"2024-04-02T08:52:42.524555+0000","last_clean":"2024-04-02T08:52:42.524555+0000","last_became_active":"2024-04-02T08:52:42.519988+0000","last_became_peered":"2024-04-02T08:52:42.519988+0000","last_unstale":"2024-04-02T08:52:42.524555+0000","last_undegraded":"2024-04-02T08:52:42.524555+0000","last_fullsized":"2024-04-02T08:52:42.524555+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:40:28.725332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"171'4","reported_seq":14,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524343+0000","last_change":"2024-04-02T08:52:42.003517+0000","last_active":"2024-04-02T08:52:42.524343+0000","last_peered":"2024-04-02T08:52:42.524343+0000","last_clean":"2024-04-02T08:52:42.524343+0000","last_became_active":"2024-04-02T08:52:42.003267+0000","last_became_peered":"2024-04-02T08:52:42.003267+0000","last_unstale":"2024-04-02T08:52:42.524343+0000","last_undegraded":"2024-04-02T08:52:42.524343+0000","last_fullsized":"2024-04-02T08:52:42.524343+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:42.599747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503707+0000","last_change":"2024-04-02T08:52:42.512866+0000","last_active":"2024-04-02T08:52:43.503707+0000","last_peered":"2024-04-02T08:52:43.503707+0000","last_clean":"2024-04-02T08:52:43.503707+0000","last_became_active":"2024-04-02T08:52:42.512546+0000","last_became_peered":"2024-04-02T08:52:42.512546+0000","last_unstale":"2024-04-02T08:52:43.503707+0000","last_undegraded":"2024-04-02T08:52:43.503707+0000","last_fullsized":"2024-04-02T08:52:43.503707+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:17:18.662540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.1e","version":"171'1","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986265+0000","last_change":"2024-04-02T08:52:42.011712+0000","last_active":"2024-04-02T08:52:43.986265+0000","last_peered":"2024-04-02T08:52:43.986265+0000","last_clean":"2024-04-02T08:52:43.986265+0000","last_became_active":"2024-04-02T08:52:42.011324+0000","last_became_peered":"2024-04-02T08:52:42.011324+0000","last_unstale":"2024-04-02T08:52:43.986265+0000","last_undegraded":"2024-04-02T08:52:43.986265+0000","last_fullsized":"2024-04-02T08:52:43.986265+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:05.673571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986103+0000","last_change":"2024-04-02T08:52:42.520857+0000","last_active":"2024-04-02T08:52:43.986103+0000","last_peered":"2024-04-02T08:52:43.986103+0000","last_clean":"2024-04-02T08:52:43.986103+0000","last_became_active":"2024-04-02T08:52:42.520559+0000","last_became_peered":"2024-04-02T08:52:42.520559+0000","last_unstale":"2024-04-02T08:52:43.986103+0000","last_undegraded":"2024-04-02T08:52:43.986103+0000","last_fullsized":"2024-04-02T08:52:43.986103+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:16:45.977261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.1d","version":"171'2","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.517048+0000","last_change":"2024-04-02T08:52:42.008308+0000","last_active":"2024-04-02T08:52:42.517048+0000","last_peered":"2024-04-02T08:52:42.517048+0000","last_clean":"2024-04-02T08:52:42.517048+0000","last_became_active":"2024-04-02T08:52:42.008053+0000","last_became_peered":"2024-04-02T08:52:42.008053+0000","last_unstale":"2024-04-02T08:52:42.517048+0000","last_undegraded":"2024-04-02T08:52:42.517048+0000","last_fullsized":"2024-04-02T08:52:42.517048+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:18:06.535318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.520000+0000","last_change":"2024-04-02T08:52:42.511289+0000","last_active":"2024-04-02T08:52:42.520000+0000","last_peered":"2024-04-02T08:52:42.520000+0000","last_clean":"2024-04-02T08:52:42.520000+0000","last_became_active":"2024-04-02T08:52:42.511026+0000","last_became_peered":"2024-04-02T08:52:42.511026+0000","last_unstale":"2024-04-02T08:52:42.520000+0000","last_undegraded":"2024-04-02T08:52:42.520000+0000","last_fullsized":"2024-04-02T08:52:42.520000+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:56:32.295359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1c","version":"171'2","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524007+0000","last_change":"2024-04-02T08:52:42.010268+0000","last_active":"2024-04-02T08:52:42.524007+0000","last_peered":"2024-04-02T08:52:42.524007+0000","last_clean":"2024-04-02T08:52:42.524007+0000","last_became_active":"2024-04-02T08:52:42.009384+0000","last_became_peered":"2024-04-02T08:52:42.009384+0000","last_unstale":"2024-04-02T08:52:42.524007+0000","last_undegraded":"2024-04-02T08:52:42.524007+0000","last_fullsized":"2024-04-02T08:52:42.524007+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:14:17.826438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503508+0000","last_change":"2024-04-02T08:52:42.522266+0000","last_active":"2024-04-02T08:52:43.503508+0000","last_peered":"2024-04-02T08:52:43.503508+0000","last_clean":"2024-04-02T08:52:43.503508+0000","last_became_active":"2024-04-02T08:52:42.521874+0000","last_became_peered":"2024-04-02T08:52:42.521874+0000","last_unstale":"2024-04-02T08:52:43.503508+0000","last_undegraded":"2024-04-02T08:52:43.503508+0000","last_fullsized":"2024-04-02T08:52:43.503508+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:00:21.809746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"171'1","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504982+0000","last_change":"2024-04-02T08:52:42.000929+0000","last_active":"2024-04-02T08:52:43.504982+0000","last_peered":"2024-04-02T08:52:43.504982+0000","last_clean":"2024-04-02T08:52:43.504982+0000","last_became_active":"2024-04-02T08:52:42.000418+0000","last_became_peered":"2024-04-02T08:52:42.000418+0000","last_unstale":"2024-04-02T08:52:43.504982+0000","last_undegraded":"2024-04-02T08:52:43.504982+0000","last_fullsized":"2024-04-02T08:52:43.504982+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:42:14.230762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.520239+0000","last_change":"2024-04-02T08:52:42.514952+0000","last_active":"2024-04-02T08:52:42.520239+0000","last_peered":"2024-04-02T08:52:42.520239+0000","last_clean":"2024-04-02T08:52:42.520239+0000","last_became_active":"2024-04-02T08:52:42.514700+0000","last_became_peered":"2024-04-02T08:52:42.514700+0000","last_unstale":"2024-04-02T08:52:42.520239+0000","last_undegraded":"2024-04-02T08:52:42.520239+0000","last_fullsized":"2024-04-02T08:52:42.520239+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:18:36.365111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1a","version":"171'1","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524946+0000","last_change":"2024-04-02T08:52:42.001735+0000","last_active":"2024-04-02T08:52:42.524946+0000","last_peered":"2024-04-02T08:52:42.524946+0000","last_clean":"2024-04-02T08:52:42.524946+0000","last_became_active":"2024-04-02T08:52:42.000896+0000","last_became_peered":"2024-04-02T08:52:42.000896+0000","last_unstale":"2024-04-02T08:52:42.524946+0000","last_undegraded":"2024-04-02T08:52:42.524946+0000","last_fullsized":"2024-04-02T08:52:42.524946+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:06.781538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503218+0000","last_change":"2024-04-02T08:52:42.516017+0000","last_active":"2024-04-02T08:52:43.503218+0000","last_peered":"2024-04-02T08:52:43.503218+0000","last_clean":"2024-04-02T08:52:43.503218+0000","last_became_active":"2024-04-02T08:52:42.515288+0000","last_became_peered":"2024-04-02T08:52:42.515288+0000","last_unstale":"2024-04-02T08:52:43.503218+0000","last_undegraded":"2024-04-02T08:52:43.503218+0000","last_fullsized":"2024-04-02T08:52:43.503218+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:18:05.763436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986283+0000","last_change":"2024-04-02T08:52:42.009019+0000","last_active":"2024-04-02T08:52:43.986283+0000","last_peered":"2024-04-02T08:52:43.986283+0000","last_clean":"2024-04-02T08:52:43.986283+0000","last_became_active":"2024-04-02T08:52:42.007461+0000","last_became_peered":"2024-04-02T08:52:42.007461+0000","last_unstale":"2024-04-02T08:52:43.986283+0000","last_undegraded":"2024-04-02T08:52:43.986283+0000","last_fullsized":"2024-04-02T08:52:43.986283+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:15:38.703511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.520904+0000","last_change":"2024-04-02T08:52:42.508371+0000","last_active":"2024-04-02T08:52:42.520904+0000","last_peered":"2024-04-02T08:52:42.520904+0000","last_clean":"2024-04-02T08:52:42.520904+0000","last_became_active":"2024-04-02T08:52:42.507952+0000","last_became_peered":"2024-04-02T08:52:42.507952+0000","last_unstale":"2024-04-02T08:52:42.520904+0000","last_undegraded":"2024-04-02T08:52:42.520904+0000","last_fullsized":"2024-04-02T08:52:42.520904+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:08:18.767645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.492931+0000","last_change":"2024-04-02T08:52:42.009648+0000","last_active":"2024-04-02T08:52:42.492931+0000","last_peered":"2024-04-02T08:52:42.492931+0000","last_clean":"2024-04-02T08:52:42.492931+0000","last_became_active":"2024-04-02T08:52:42.008796+0000","last_became_peered":"2024-04-02T08:52:42.008796+0000","last_unstale":"2024-04-02T08:52:42.492931+0000","last_undegraded":"2024-04-02T08:52:42.492931+0000","last_fullsized":"2024-04-02T08:52:42.492931+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:49:17.177132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.523923+0000","last_change":"2024-04-02T08:52:42.518633+0000","last_active":"2024-04-02T08:52:42.523923+0000","last_peered":"2024-04-02T08:52:42.523923+0000","last_clean":"2024-04-02T08:52:42.523923+0000","last_became_active":"2024-04-02T08:52:42.518291+0000","last_became_peered":"2024-04-02T08:52:42.518291+0000","last_unstale":"2024-04-02T08:52:42.523923+0000","last_undegraded":"2024-04-02T08:52:42.523923+0000","last_fullsized":"2024-04-02T08:52:42.523923+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:36.498912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986293+0000","last_change":"2024-04-02T08:52:42.008236+0000","last_active":"2024-04-02T08:52:43.986293+0000","last_peered":"2024-04-02T08:52:43.986293+0000","last_clean":"2024-04-02T08:52:43.986293+0000","last_became_active":"2024-04-02T08:52:42.007963+0000","last_became_peered":"2024-04-02T08:52:42.007963+0000","last_unstale":"2024-04-02T08:52:43.986293+0000","last_undegraded":"2024-04-02T08:52:43.986293+0000","last_fullsized":"2024-04-02T08:52:43.986293+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:30:48.609117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.515922+0000","last_change":"2024-04-02T08:52:42.513612+0000","last_active":"2024-04-02T08:52:42.515922+0000","last_peered":"2024-04-02T08:52:42.515922+0000","last_clean":"2024-04-02T08:52:42.515922+0000","last_became_active":"2024-04-02T08:52:42.513220+0000","last_became_peered":"2024-04-02T08:52:42.513220+0000","last_unstale":"2024-04-02T08:52:42.515922+0000","last_undegraded":"2024-04-02T08:52:42.515922+0000","last_fullsized":"2024-04-02T08:52:42.515922+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:17:29.804325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.500130+0000","last_change":"2024-04-02T08:52:42.008640+0000","last_active":"2024-04-02T08:52:42.500130+0000","last_peered":"2024-04-02T08:52:42.500130+0000","last_clean":"2024-04-02T08:52:42.500130+0000","last_became_active":"2024-04-02T08:52:42.007386+0000","last_became_peered":"2024-04-02T08:52:42.007386+0000","last_unstale":"2024-04-02T08:52:42.500130+0000","last_undegraded":"2024-04-02T08:52:42.500130+0000","last_fullsized":"2024-04-02T08:52:42.500130+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:31.353853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503448+0000","last_change":"2024-04-02T08:52:42.498531+0000","last_active":"2024-04-02T08:52:43.503448+0000","last_peered":"2024-04-02T08:52:43.503448+0000","last_clean":"2024-04-02T08:52:43.503448+0000","last_became_active":"2024-04-02T08:52:42.497676+0000","last_became_peered":"2024-04-02T08:52:42.497676+0000","last_unstale":"2024-04-02T08:52:43.503448+0000","last_undegraded":"2024-04-02T08:52:43.503448+0000","last_fullsized":"2024-04-02T08:52:43.503448+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:27:40.539739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.14","version":"171'6","reported_seq":17,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503320+0000","last_change":"2024-04-02T08:52:42.005636+0000","last_active":"2024-04-02T08:52:43.503320+0000","last_peered":"2024-04-02T08:52:43.503320+0000","last_clean":"2024-04-02T08:52:43.503320+0000","last_became_active":"2024-04-02T08:52:42.005368+0000","last_became_peered":"2024-04-02T08:52:42.005368+0000","last_unstale":"2024-04-02T08:52:43.503320+0000","last_undegraded":"2024-04-02T08:52:43.503320+0000","last_fullsized":"2024-04-02T08:52:43.503320+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:19:07.924940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.520816+0000","last_change":"2024-04-02T08:52:42.508364+0000","last_active":"2024-04-02T08:52:42.520816+0000","last_peered":"2024-04-02T08:52:42.520816+0000","last_clean":"2024-04-02T08:52:42.520816+0000","last_became_active":"2024-04-02T08:52:42.507935+0000","last_became_peered":"2024-04-02T08:52:42.507935+0000","last_unstale":"2024-04-02T08:52:42.520816+0000","last_undegraded":"2024-04-02T08:52:42.520816+0000","last_fullsized":"2024-04-02T08:52:42.520816+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:30:49.616210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.13","version":"171'2","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524368+0000","last_change":"2024-04-02T08:52:42.009185+0000","last_active":"2024-04-02T08:52:42.524368+0000","last_peered":"2024-04-02T08:52:42.524368+0000","last_clean":"2024-04-02T08:52:42.524368+0000","last_became_active":"2024-04-02T08:52:42.008916+0000","last_became_peered":"2024-04-02T08:52:42.008916+0000","last_unstale":"2024-04-02T08:52:42.524368+0000","last_undegraded":"2024-04-02T08:52:42.524368+0000","last_fullsized":"2024-04-02T08:52:42.524368+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:29.849209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504909+0000","last_change":"2024-04-02T08:52:42.518098+0000","last_active":"2024-04-02T08:52:43.504909+0000","last_peered":"2024-04-02T08:52:43.504909+0000","last_clean":"2024-04-02T08:52:43.504909+0000","last_became_active":"2024-04-02T08:52:42.517269+0000","last_became_peered":"2024-04-02T08:52:42.517269+0000","last_unstale":"2024-04-02T08:52:43.504909+0000","last_undegraded":"2024-04-02T08:52:43.504909+0000","last_fullsized":"2024-04-02T08:52:43.504909+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:32:09.794631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.501773+0000","last_change":"2024-04-02T08:52:42.011331+0000","last_active":"2024-04-02T08:52:42.501773+0000","last_peered":"2024-04-02T08:52:42.501773+0000","last_clean":"2024-04-02T08:52:42.501773+0000","last_became_active":"2024-04-02T08:52:42.011038+0000","last_became_peered":"2024-04-02T08:52:42.011038+0000","last_unstale":"2024-04-02T08:52:42.501773+0000","last_undegraded":"2024-04-02T08:52:42.501773+0000","last_fullsized":"2024-04-02T08:52:42.501773+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:02:48.920433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.526572+0000","last_change":"2024-04-02T08:52:42.520472+0000","last_active":"2024-04-02T08:52:42.526572+0000","last_peered":"2024-04-02T08:52:42.526572+0000","last_clean":"2024-04-02T08:52:42.526572+0000","last_became_active":"2024-04-02T08:52:42.520166+0000","last_became_peered":"2024-04-02T08:52:42.520166+0000","last_unstale":"2024-04-02T08:52:42.526572+0000","last_undegraded":"2024-04-02T08:52:42.526572+0000","last_fullsized":"2024-04-02T08:52:42.526572+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:27.050671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503625+0000","last_change":"2024-04-02T08:52:42.010289+0000","last_active":"2024-04-02T08:52:43.503625+0000","last_peered":"2024-04-02T08:52:43.503625+0000","last_clean":"2024-04-02T08:52:43.503625+0000","last_became_active":"2024-04-02T08:52:42.009720+0000","last_became_peered":"2024-04-02T08:52:42.009720+0000","last_unstale":"2024-04-02T08:52:43.503625+0000","last_undegraded":"2024-04-02T08:52:43.503625+0000","last_fullsized":"2024-04-02T08:52:43.503625+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:34:58.203723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.527413+0000","last_change":"2024-04-02T08:52:42.525284+0000","last_active":"2024-04-02T08:52:42.527413+0000","last_peered":"2024-04-02T08:52:42.527413+0000","last_clean":"2024-04-02T08:52:42.527413+0000","last_became_active":"2024-04-02T08:52:42.520143+0000","last_became_peered":"2024-04-02T08:52:42.520143+0000","last_unstale":"2024-04-02T08:52:42.527413+0000","last_undegraded":"2024-04-02T08:52:42.527413+0000","last_fullsized":"2024-04-02T08:52:42.527413+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:00:21.809746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.10","version":"171'8","reported_seq":19,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504745+0000","last_change":"2024-04-02T08:52:42.007328+0000","last_active":"2024-04-02T08:52:43.504745+0000","last_peered":"2024-04-02T08:52:43.504745+0000","last_clean":"2024-04-02T08:52:43.504745+0000","last_became_active":"2024-04-02T08:52:42.007044+0000","last_became_peered":"2024-04-02T08:52:42.007044+0000","last_unstale":"2024-04-02T08:52:43.504745+0000","last_undegraded":"2024-04-02T08:52:43.504745+0000","last_fullsized":"2024-04-02T08:52:43.504745+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:12.170720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.521653+0000","last_change":"2024-04-02T08:52:42.515821+0000","last_active":"2024-04-02T08:52:42.521653+0000","last_peered":"2024-04-02T08:52:42.521653+0000","last_clean":"2024-04-02T08:52:42.521653+0000","last_became_active":"2024-04-02T08:52:42.515548+0000","last_became_peered":"2024-04-02T08:52:42.515548+0000","last_unstale":"2024-04-02T08:52:42.521653+0000","last_undegraded":"2024-04-02T08:52:42.521653+0000","last_fullsized":"2024-04-02T08:52:42.521653+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:02:52.449340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.f","version":"171'1","reported_seq":11,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.524596+0000","last_change":"2024-04-02T08:52:42.006635+0000","last_active":"2024-04-02T08:52:42.524596+0000","last_peered":"2024-04-02T08:52:42.524596+0000","last_clean":"2024-04-02T08:52:42.524596+0000","last_became_active":"2024-04-02T08:52:42.003755+0000","last_became_peered":"2024-04-02T08:52:42.003755+0000","last_unstale":"2024-04-02T08:52:42.524596+0000","last_undegraded":"2024-04-02T08:52:42.524596+0000","last_fullsized":"2024-04-02T08:52:42.524596+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:41:26.162559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504139+0000","last_change":"2024-04-02T08:52:42.504557+0000","last_active":"2024-04-02T08:52:43.504139+0000","last_peered":"2024-04-02T08:52:43.504139+0000","last_clean":"2024-04-02T08:52:43.504139+0000","last_became_active":"2024-04-02T08:52:42.504250+0000","last_became_peered":"2024-04-02T08:52:42.504250+0000","last_unstale":"2024-04-02T08:52:43.504139+0000","last_undegraded":"2024-04-02T08:52:43.504139+0000","last_fullsized":"2024-04-02T08:52:43.504139+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:36.498912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.e","version":"171'2","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.505306+0000","last_change":"2024-04-02T08:52:42.005337+0000","last_active":"2024-04-02T08:52:43.505306+0000","last_peered":"2024-04-02T08:52:43.505306+0000","last_clean":"2024-04-02T08:52:43.505306+0000","last_became_active":"2024-04-02T08:52:42.003327+0000","last_became_peered":"2024-04-02T08:52:42.003327+0000","last_unstale":"2024-04-02T08:52:43.505306+0000","last_undegraded":"2024-04-02T08:52:43.505306+0000","last_fullsized":"2024-04-02T08:52:43.505306+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:46.403723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.512227+0000","last_change":"2024-04-02T08:52:42.508038+0000","last_active":"2024-04-02T08:52:42.512227+0000","last_peered":"2024-04-02T08:52:42.512227+0000","last_clean":"2024-04-02T08:52:42.512227+0000","last_became_active":"2024-04-02T08:52:42.507789+0000","last_became_peered":"2024-04-02T08:52:42.507789+0000","last_unstale":"2024-04-02T08:52:42.512227+0000","last_undegraded":"2024-04-02T08:52:42.512227+0000","last_fullsized":"2024-04-02T08:52:42.512227+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:34:27.421047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.d","version":"171'1","reported_seq":12,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503622+0000","last_change":"2024-04-02T08:52:42.007277+0000","last_active":"2024-04-02T08:52:43.503622+0000","last_peered":"2024-04-02T08:52:43.503622+0000","last_clean":"2024-04-02T08:52:43.503622+0000","last_became_active":"2024-04-02T08:52:42.005700+0000","last_became_peered":"2024-04-02T08:52:42.005700+0000","last_unstale":"2024-04-02T08:52:43.503622+0000","last_undegraded":"2024-04-02T08:52:43.503622+0000","last_fullsized":"2024-04-02T08:52:43.503622+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:19:06.944801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986186+0000","last_change":"2024-04-02T08:52:42.520926+0000","last_active":"2024-04-02T08:52:43.986186+0000","last_peered":"2024-04-02T08:52:43.986186+0000","last_clean":"2024-04-02T08:52:43.986186+0000","last_became_active":"2024-04-02T08:52:42.520640+0000","last_became_peered":"2024-04-02T08:52:42.520640+0000","last_unstale":"2024-04-02T08:52:43.986186+0000","last_undegraded":"2024-04-02T08:52:43.986186+0000","last_fullsized":"2024-04-02T08:52:43.986186+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:02:38.305601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503776+0000","last_change":"2024-04-02T08:52:42.009748+0000","last_active":"2024-04-02T08:52:43.503776+0000","last_peered":"2024-04-02T08:52:43.503776+0000","last_clean":"2024-04-02T08:52:43.503776+0000","last_became_active":"2024-04-02T08:52:42.009199+0000","last_became_peered":"2024-04-02T08:52:42.009199+0000","last_unstale":"2024-04-02T08:52:43.503776+0000","last_undegraded":"2024-04-02T08:52:43.503776+0000","last_fullsized":"2024-04-02T08:52:43.503776+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:09:55.394419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.515136+0000","last_change":"2024-04-02T08:52:42.508085+0000","last_active":"2024-04-02T08:52:42.515136+0000","last_peered":"2024-04-02T08:52:42.515136+0000","last_clean":"2024-04-02T08:52:42.515136+0000","last_became_active":"2024-04-02T08:52:42.507847+0000","last_became_peered":"2024-04-02T08:52:42.507847+0000","last_unstale":"2024-04-02T08:52:42.515136+0000","last_undegraded":"2024-04-02T08:52:42.515136+0000","last_fullsized":"2024-04-02T08:52:42.515136+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:03.828822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.b","version":"171'2","reported_seq":13,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.986009+0000","last_change":"2024-04-02T08:52:42.011571+0000","last_active":"2024-04-02T08:52:43.986009+0000","last_peered":"2024-04-02T08:52:43.986009+0000","last_clean":"2024-04-02T08:52:43.986009+0000","last_became_active":"2024-04-02T08:52:42.011109+0000","last_became_peered":"2024-04-02T08:52:42.011109+0000","last_unstale":"2024-04-02T08:52:43.986009+0000","last_undegraded":"2024-04-02T08:52:43.986009+0000","last_fullsized":"2024-04-02T08:52:43.986009+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:26.043578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503710+0000","last_change":"2024-04-02T08:52:42.504471+0000","last_active":"2024-04-02T08:52:43.503710+0000","last_peered":"2024-04-02T08:52:43.503710+0000","last_clean":"2024-04-02T08:52:43.503710+0000","last_became_active":"2024-04-02T08:52:42.504150+0000","last_became_peered":"2024-04-02T08:52:42.504150+0000","last_unstale":"2024-04-02T08:52:43.503710+0000","last_undegraded":"2024-04-02T08:52:43.503710+0000","last_fullsized":"2024-04-02T08:52:43.503710+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:56:32.295359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.497903+0000","last_change":"2024-04-02T08:52:42.002985+0000","last_active":"2024-04-02T08:52:42.497903+0000","last_peered":"2024-04-02T08:52:42.497903+0000","last_clean":"2024-04-02T08:52:42.497903+0000","last_became_active":"2024-04-02T08:52:42.002727+0000","last_became_peered":"2024-04-02T08:52:42.002727+0000","last_unstale":"2024-04-02T08:52:42.497903+0000","last_undegraded":"2024-04-02T08:52:42.497903+0000","last_fullsized":"2024-04-02T08:52:42.497903+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:24:28.756704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.515969+0000","last_change":"2024-04-02T08:52:42.513627+0000","last_active":"2024-04-02T08:52:42.515969+0000","last_peered":"2024-04-02T08:52:42.515969+0000","last_clean":"2024-04-02T08:52:42.515969+0000","last_became_active":"2024-04-02T08:52:42.513174+0000","last_became_peered":"2024-04-02T08:52:42.513174+0000","last_unstale":"2024-04-02T08:52:42.515969+0000","last_undegraded":"2024-04-02T08:52:42.515969+0000","last_fullsized":"2024-04-02T08:52:42.515969+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:22.165775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.497665+0000","last_change":"2024-04-02T08:52:42.004657+0000","last_active":"2024-04-02T08:52:42.497665+0000","last_peered":"2024-04-02T08:52:42.497665+0000","last_clean":"2024-04-02T08:52:42.497665+0000","last_became_active":"2024-04-02T08:52:42.003264+0000","last_became_peered":"2024-04-02T08:52:42.003264+0000","last_unstale":"2024-04-02T08:52:42.497665+0000","last_undegraded":"2024-04-02T08:52:42.497665+0000","last_fullsized":"2024-04-02T08:52:42.497665+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:15:58.269557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503592+0000","last_change":"2024-04-02T08:52:42.505564+0000","last_active":"2024-04-02T08:52:43.503592+0000","last_peered":"2024-04-02T08:52:43.503592+0000","last_clean":"2024-04-02T08:52:43.503592+0000","last_became_active":"2024-04-02T08:52:42.502911+0000","last_became_peered":"2024-04-02T08:52:42.502911+0000","last_unstale":"2024-04-02T08:52:43.503592+0000","last_undegraded":"2024-04-02T08:52:43.503592+0000","last_fullsized":"2024-04-02T08:52:43.503592+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:45:26.025962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.503849+0000","last_change":"2024-04-02T08:52:42.002807+0000","last_active":"2024-04-02T08:52:42.503849+0000","last_peered":"2024-04-02T08:52:42.503849+0000","last_clean":"2024-04-02T08:52:42.503849+0000","last_became_active":"2024-04-02T08:52:42.002474+0000","last_became_peered":"2024-04-02T08:52:42.002474+0000","last_unstale":"2024-04-02T08:52:42.503849+0000","last_undegraded":"2024-04-02T08:52:42.503849+0000","last_fullsized":"2024-04-02T08:52:42.503849+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:01:08.178914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.520055+0000","last_change":"2024-04-02T08:52:42.511855+0000","last_active":"2024-04-02T08:52:42.520055+0000","last_peered":"2024-04-02T08:52:42.520055+0000","last_clean":"2024-04-02T08:52:42.520055+0000","last_became_active":"2024-04-02T08:52:42.511568+0000","last_became_peered":"2024-04-02T08:52:42.511568+0000","last_unstale":"2024-04-02T08:52:42.520055+0000","last_undegraded":"2024-04-02T08:52:42.520055+0000","last_fullsized":"2024-04-02T08:52:42.520055+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:06.680663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.7","version":"171'2","reported_seq":12,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.525290+0000","last_change":"2024-04-02T08:52:42.007221+0000","last_active":"2024-04-02T08:52:42.525290+0000","last_peered":"2024-04-02T08:52:42.525290+0000","last_clean":"2024-04-02T08:52:42.525290+0000","last_became_active":"2024-04-02T08:52:42.006357+0000","last_became_peered":"2024-04-02T08:52:42.006357+0000","last_unstale":"2024-04-02T08:52:42.525290+0000","last_undegraded":"2024-04-02T08:52:42.525290+0000","last_fullsized":"2024-04-02T08:52:42.525290+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:36:25.475809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.519664+0000","last_change":"2024-04-02T08:52:42.511983+0000","last_active":"2024-04-02T08:52:42.519664+0000","last_peered":"2024-04-02T08:52:42.519664+0000","last_clean":"2024-04-02T08:52:42.519664+0000","last_became_active":"2024-04-02T08:52:42.511729+0000","last_became_peered":"2024-04-02T08:52:42.511729+0000","last_unstale":"2024-04-02T08:52:42.519664+0000","last_undegraded":"2024-04-02T08:52:42.519664+0000","last_fullsized":"2024-04-02T08:52:42.519664+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:06:30.703910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.504054+0000","last_change":"2024-04-02T08:52:42.012534+0000","last_active":"2024-04-02T08:52:43.504054+0000","last_peered":"2024-04-02T08:52:43.504054+0000","last_clean":"2024-04-02T08:52:43.504054+0000","last_became_active":"2024-04-02T08:52:42.012223+0000","last_became_peered":"2024-04-02T08:52:42.012223+0000","last_unstale":"2024-04-02T08:52:43.504054+0000","last_undegraded":"2024-04-02T08:52:43.504054+0000","last_fullsized":"2024-04-02T08:52:43.504054+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:41:26.162559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.505135+0000","last_change":"2024-04-02T08:52:42.509214+0000","last_active":"2024-04-02T08:52:43.505135+0000","last_peered":"2024-04-02T08:52:43.505135+0000","last_clean":"2024-04-02T08:52:43.505135+0000","last_became_active":"2024-04-02T08:52:42.508650+0000","last_became_peered":"2024-04-02T08:52:42.508650+0000","last_unstale":"2024-04-02T08:52:43.505135+0000","last_undegraded":"2024-04-02T08:52:43.505135+0000","last_fullsized":"2024-04-02T08:52:43.505135+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:15:29.237137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503777+0000","last_change":"2024-04-02T08:52:42.000283+0000","last_active":"2024-04-02T08:52:43.503777+0000","last_peered":"2024-04-02T08:52:43.503777+0000","last_clean":"2024-04-02T08:52:43.503777+0000","last_became_active":"2024-04-02T08:52:41.999538+0000","last_became_peered":"2024-04-02T08:52:41.999538+0000","last_unstale":"2024-04-02T08:52:43.503777+0000","last_undegraded":"2024-04-02T08:52:43.503777+0000","last_fullsized":"2024-04-02T08:52:43.503777+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:35:25.602471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.505197+0000","last_change":"2024-04-02T08:52:42.522912+0000","last_active":"2024-04-02T08:52:43.505197+0000","last_peered":"2024-04-02T08:52:43.505197+0000","last_clean":"2024-04-02T08:52:43.505197+0000","last_became_active":"2024-04-02T08:52:42.522432+0000","last_became_peered":"2024-04-02T08:52:42.522432+0000","last_unstale":"2024-04-02T08:52:43.505197+0000","last_undegraded":"2024-04-02T08:52:43.505197+0000","last_fullsized":"2024-04-02T08:52:43.505197+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:38:36.042236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503417+0000","last_change":"2024-04-02T08:52:42.005085+0000","last_active":"2024-04-02T08:52:43.503417+0000","last_peered":"2024-04-02T08:52:43.503417+0000","last_clean":"2024-04-02T08:52:43.503417+0000","last_became_active":"2024-04-02T08:52:42.003345+0000","last_became_peered":"2024-04-02T08:52:42.003345+0000","last_unstale":"2024-04-02T08:52:43.503417+0000","last_undegraded":"2024-04-02T08:52:43.503417+0000","last_fullsized":"2024-04-02T08:52:43.503417+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:09.652023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503040+0000","last_change":"2024-04-02T08:52:42.508364+0000","last_active":"2024-04-02T08:52:43.503040+0000","last_peered":"2024-04-02T08:52:43.503040+0000","last_clean":"2024-04-02T08:52:43.503040+0000","last_became_active":"2024-04-02T08:52:42.508065+0000","last_became_peered":"2024-04-02T08:52:42.508065+0000","last_unstale":"2024-04-02T08:52:43.503040+0000","last_undegraded":"2024-04-02T08:52:43.503040+0000","last_fullsized":"2024-04-02T08:52:43.503040+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:15:29.237137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503800+0000","last_change":"2024-04-02T08:52:42.004725+0000","last_active":"2024-04-02T08:52:43.503800+0000","last_peered":"2024-04-02T08:52:43.503800+0000","last_clean":"2024-04-02T08:52:43.503800+0000","last_became_active":"2024-04-02T08:52:42.004441+0000","last_became_peered":"2024-04-02T08:52:42.004441+0000","last_unstale":"2024-04-02T08:52:43.503800+0000","last_undegraded":"2024-04-02T08:52:43.503800+0000","last_fullsized":"2024-04-02T08:52:43.503800+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:32:22.762908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.519554+0000","last_change":"2024-04-02T08:52:42.508983+0000","last_active":"2024-04-02T08:52:42.519554+0000","last_peered":"2024-04-02T08:52:42.519554+0000","last_clean":"2024-04-02T08:52:42.519554+0000","last_became_active":"2024-04-02T08:52:42.508685+0000","last_became_peered":"2024-04-02T08:52:42.508685+0000","last_unstale":"2024-04-02T08:52:42.519554+0000","last_undegraded":"2024-04-02T08:52:42.519554+0000","last_fullsized":"2024-04-02T08:52:42.519554+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:47:48.659307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.494289+0000","last_change":"2024-04-02T08:52:42.004729+0000","last_active":"2024-04-02T08:52:42.494289+0000","last_peered":"2024-04-02T08:52:42.494289+0000","last_clean":"2024-04-02T08:52:42.494289+0000","last_became_active":"2024-04-02T08:52:42.004398+0000","last_became_peered":"2024-04-02T08:52:42.004398+0000","last_unstale":"2024-04-02T08:52:42.494289+0000","last_undegraded":"2024-04-02T08:52:42.494289+0000","last_fullsized":"2024-04-02T08:52:42.494289+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:30.565732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.520554+0000","last_change":"2024-04-02T08:52:42.507360+0000","last_active":"2024-04-02T08:52:42.520554+0000","last_peered":"2024-04-02T08:52:42.520554+0000","last_clean":"2024-04-02T08:52:42.520554+0000","last_became_active":"2024-04-02T08:52:42.507111+0000","last_became_peered":"2024-04-02T08:52:42.507111+0000","last_unstale":"2024-04-02T08:52:42.520554+0000","last_undegraded":"2024-04-02T08:52:42.520554+0000","last_fullsized":"2024-04-02T08:52:42.520554+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:15:40.194442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":10,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.495267+0000","last_change":"2024-04-02T08:52:42.010060+0000","last_active":"2024-04-02T08:52:42.495267+0000","last_peered":"2024-04-02T08:52:42.495267+0000","last_clean":"2024-04-02T08:52:42.495267+0000","last_became_active":"2024-04-02T08:52:42.009795+0000","last_became_peered":"2024-04-02T08:52:42.009795+0000","last_unstale":"2024-04-02T08:52:42.495267+0000","last_undegraded":"2024-04-02T08:52:42.495267+0000","last_fullsized":"2024-04-02T08:52:42.495267+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:10:54.483338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.509124+0000","last_change":"2024-04-02T08:52:42.501713+0000","last_active":"2024-04-02T08:52:42.509124+0000","last_peered":"2024-04-02T08:52:42.509124+0000","last_clean":"2024-04-02T08:52:42.509124+0000","last_became_active":"2024-04-02T08:52:42.501414+0000","last_became_peered":"2024-04-02T08:52:42.501414+0000","last_unstale":"2024-04-02T08:52:42.509124+0000","last_undegraded":"2024-04-02T08:52:42.509124+0000","last_fullsized":"2024-04-02T08:52:42.509124+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:46:24.377267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.985830+0000","last_change":"2024-04-02T08:52:42.011570+0000","last_active":"2024-04-02T08:52:43.985830+0000","last_peered":"2024-04-02T08:52:43.985830+0000","last_clean":"2024-04-02T08:52:43.985830+0000","last_became_active":"2024-04-02T08:52:42.011071+0000","last_became_peered":"2024-04-02T08:52:42.011071+0000","last_unstale":"2024-04-02T08:52:43.985830+0000","last_undegraded":"2024-04-02T08:52:43.985830+0000","last_fullsized":"2024-04-02T08:52:43.985830+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:18:04.756343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-02T08:52:43.503221+0000","last_change":"2024-04-02T08:52:42.498110+0000","last_active":"2024-04-02T08:52:43.503221+0000","last_peered":"2024-04-02T08:52:43.503221+0000","last_clean":"2024-04-02T08:52:43.503221+0000","last_became_active":"2024-04-02T08:52:42.497098+0000","last_became_peered":"2024-04-02T08:52:42.497098+0000","last_unstale":"2024-04-02T08:52:43.503221+0000","last_undegraded":"2024-04-02T08:52:43.503221+0000","last_fullsized":"2024-04-02T08:52:43.503221+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":170,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:41.976411+0000","last_clean_scrub_stamp":"2024-04-02T08:52:41.976411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:32:09.794631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.15","version":"171'3","reported_seq":13,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-02T08:52:42.522954+0000","last_change":"2024-04-02T08:52:42.011270+0000","last_active":"2024-04-02T08:52:42.522954+0000","last_peered":"2024-04-02T08:52:42.522954+0000","last_clean":"2024-04-02T08:52:42.522954+0000","last_became_active":"2024-04-02T08:52:42.010962+0000","last_became_peered":"2024-04-02T08:52:42.010962+0000","last_unstale":"2024-04-02T08:52:42.522954+0000","last_undegraded":"2024-04-02T08:52:42.522954+0000","last_fullsized":"2024-04-02T08:52:42.522954+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":169,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:52:40.969318+0000","last_clean_scrub_stamp":"2024-04-02T08:52:40.969318+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:40:37.133822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"161'271","reported_seq":881,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-04-02T08:52:41.995462+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T08:52:41.995462+0000","last_peered":"2024-04-02T08:52:41.995462+0000","last_clean":"2024-04-02T08:52:41.995462+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T08:52:41.995462+0000","last_undegraded":"2024-04-02T08:52:41.995462+0000","last_fullsized":"2024-04-02T08:52:41.995462+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575759,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55364,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316476,"statfs":{"total":96636764160,"available":96580071424,"internally_reserved":0,"allocated":2146304,"data_stored":1073653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.72599999999999998},"min":{"1min":0.56699999999999995,"5min":0.47199999999999998,"15min":0.41099999999999998},"max":{"1min":0.85799999999999998,"5min":1.069,"15min":1.069},"last":0.375},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.59799999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85899999999999999,"5min":1.0129999999999999,"15min":1.0169999999999999},"last":0.41499999999999998}]},{"osd":1,"last update":"Tue Apr 2 08:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.73599999999999999,"15min":0.73999999999999999},"min":{"1min":0.59299999999999997,"5min":0.51500000000000001,"15min":0.48499999999999999},"max":{"1min":0.93200000000000005,"5min":1.0089999999999999,"15min":1.028},"last":0.496},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72599999999999998,"15min":0.71699999999999997},"min":{"1min":0.56100000000000005,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.90200000000000002,"5min":1.016,"15min":1.016},"last":0.48099999999999998}]},{"osd":2,"last update":"Tue Apr 2 08:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.76000000000000001,"15min":0.76600000000000001},"min":{"1min":0.61799999999999999,"5min":0.46100000000000002,"15min":0.44400000000000001},"max":{"1min":0.88500000000000001,"5min":1.105,"15min":1.105},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81999999999999995,"15min":0.80900000000000005},"min":{"1min":0.65700000000000003,"5min":0.56999999999999995,"15min":0.497},"max":{"1min":0.93200000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.65800000000000003}]},{"osd":3,"last update":"Tue Apr 2 08:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.75700000000000001,"15min":0.746},"min":{"1min":0.52700000000000002,"5min":0.52500000000000002,"15min":0.48099999999999998},"max":{"1min":0.89500000000000002,"5min":1.073,"15min":1.073},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.76100000000000001,"15min":0.76400000000000001},"min":{"1min":0.58999999999999997,"5min":0.53200000000000003,"15min":0.49099999999999999},"max":{"1min":0.89200000000000002,"5min":1.048,"15min":1.056},"last":0.60899999999999999}]},{"osd":4,"last update":"Tue Apr 2 08:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.625,"15min":0.60099999999999998},"min":{"1min":0.48599999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81799999999999995,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.70999999999999996},"min":{"1min":0.47899999999999998,"5min":0.39700000000000002,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":0.997,"15min":1.014},"last":0.74099999999999999}]},{"osd":5,"last update":"Tue Apr 2 08:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68400000000000005,"15min":0.66600000000000004},"min":{"1min":0.47699999999999998,"5min":0.441,"15min":0.38700000000000001},"max":{"1min":0.95799999999999996,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72399999999999998,"15min":0.70899999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.41999999999999998},"max":{"1min":0.89700000000000002,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.78200000000000003}]},{"osd":6,"last update":"Tue Apr 2 08:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.66700000000000004},"min":{"1min":0.52800000000000002,"5min":0.47699999999999998,"15min":0.46400000000000002},"max":{"1min":0.82699999999999996,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.41299999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92800000000000005,"5min":0.98699999999999999,"15min":1.0149999999999999},"last":0.69399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575757,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37304,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334536,"statfs":{"total":96636764160,"available":96598564864,"internally_reserved":0,"allocated":2134016,"data_stored":1066933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.74099999999999999,"15min":0.73399999999999999},"min":{"1min":0.45100000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.96199999999999997,"5min":1.847,"15min":1.847},"last":1.1990000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.745,"15min":0.74199999999999999},"min":{"1min":0.52700000000000002,"5min":0.46600000000000003,"15min":0.40400000000000003},"max":{"1min":0.99399999999999999,"5min":1.641,"15min":1.641},"last":0.98799999999999999}]},{"osd":1,"last update":"Tue Apr 2 08:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.72799999999999998,"15min":0.72699999999999998},"min":{"1min":0.57099999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":1.117},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.76300000000000001,"15min":0.751},"min":{"1min":0.52700000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.93899999999999995,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.97399999999999998}]},{"osd":2,"last update":"Tue Apr 2 08:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.749,"15min":0.751},"min":{"1min":0.52400000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.91200000000000003,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":1.141},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.60399999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.95499999999999996,"5min":0.96799999999999997,"15min":1.0289999999999999},"last":1.006}]},{"osd":3,"last update":"Tue Apr 2 08:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.75800000000000001,"15min":0.754},"min":{"1min":0.57399999999999995,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0009999999999999,"5min":1.722,"15min":1.722},"last":1.238},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.77800000000000002,"15min":0.77600000000000002},"min":{"1min":0.57099999999999995,"5min":0.53200000000000003,"15min":0.52900000000000003},"max":{"1min":0.97899999999999998,"5min":1.329,"15min":1.329},"last":1.0249999999999999}]},{"osd":4,"last update":"Tue Apr 2 08:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.63800000000000001,"15min":0.623},"min":{"1min":0.45400000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85799999999999998,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67200000000000004,"15min":0.64900000000000002},"min":{"1min":0.45200000000000001,"5min":0.41499999999999998,"15min":0.41299999999999998},"max":{"1min":0.91400000000000003,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":1.069}]},{"osd":5,"last update":"Tue Apr 2 08:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.66600000000000004,"15min":0.63900000000000001},"min":{"1min":0.496,"5min":0.439,"15min":0.40100000000000002},"max":{"1min":0.92900000000000005,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72599999999999998,"15min":0.69899999999999995},"min":{"1min":0.56899999999999995,"5min":0.45400000000000001,"15min":0.43099999999999999},"max":{"1min":0.90900000000000003,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":1.0960000000000001}]},{"osd":7,"last update":"Tue Apr 2 08:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73499999999999999,"15min":0.70799999999999996},"min":{"1min":0.51000000000000001,"5min":0.41999999999999998,"15min":0.40999999999999998},"max":{"1min":0.96399999999999997,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.73699999999999999,"15min":0.71599999999999997},"min":{"1min":0.55000000000000004,"5min":0.48899999999999999,"15min":0.46300000000000002},"max":{"1min":0.94899999999999995,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":1.1819999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575758,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47744,"kb_used_data":2668,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324096,"statfs":{"total":96636764160,"available":96587874304,"internally_reserved":0,"allocated":2732032,"data_stored":1657371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.47399999999999998,"15min":0.46899999999999997},"min":{"1min":0.32700000000000001,"5min":0.313,"15min":0.27300000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.81100000000000005},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34899999999999998,"5min":0.34399999999999997,"15min":0.34100000000000003},"max":{"1min":0.66600000000000004,"5min":0.876,"15min":0.94799999999999995},"last":0.505}]},{"osd":2,"last update":"Tue Apr 2 08:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55200000000000005,"15min":0.53400000000000003},"min":{"1min":0.40200000000000002,"5min":0.35399999999999998,"15min":0.34000000000000002},"max":{"1min":0.64700000000000002,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.64600000000000002,"15min":0.61899999999999999},"min":{"1min":0.44900000000000001,"5min":0.44700000000000001,"15min":0.39200000000000002},"max":{"1min":0.73499999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.73899999999999999}]},{"osd":3,"last update":"Tue Apr 2 08:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.61899999999999999,"15min":0.60099999999999998},"min":{"1min":0.44800000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.68300000000000005,"5min":1.137,"15min":1.226},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58599999999999997,"15min":0.56999999999999995},"min":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.36399999999999999},"max":{"1min":0.72399999999999998,"5min":0.93200000000000005,"15min":1.0700000000000001},"last":0.46400000000000002}]},{"osd":4,"last update":"Tue Apr 2 08:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.60899999999999999,"5min":0.58999999999999997,"15min":0.54100000000000004},"max":{"1min":0.95099999999999996,"5min":1.1140000000000001,"15min":1.321},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.81699999999999995,"15min":0.81100000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.99299999999999999,"5min":1.381,"15min":1.381},"last":0.71299999999999997}]},{"osd":5,"last update":"Tue Apr 2 08:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.81699999999999995,"15min":0.80800000000000005},"min":{"1min":0.58599999999999997,"5min":0.57999999999999996,"15min":0.52100000000000002},"max":{"1min":0.94999999999999996,"5min":1.131,"15min":1.131},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.83699999999999997,"15min":0.82299999999999995},"min":{"1min":0.64000000000000001,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.046,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.77900000000000003}]},{"osd":6,"last update":"Tue Apr 2 08:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.84599999999999997,"15min":0.83299999999999996},"min":{"1min":0.627,"5min":0.627,"15min":0.57299999999999995},"max":{"1min":0.96699999999999997,"5min":1.2809999999999999,"15min":1.3759999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.83999999999999997,"15min":0.83199999999999996},"min":{"1min":0.69299999999999995,"5min":0.61299999999999999,"15min":0.53700000000000003},"max":{"1min":1.012,"5min":1.1839999999999999,"15min":1.242},"last":0.84199999999999997}]},{"osd":7,"last update":"Tue Apr 2 08:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.85899999999999999,"15min":0.84599999999999997},"min":{"1min":0.624,"5min":0.624,"15min":0.56299999999999994},"max":{"1min":1.071,"5min":1.347,"15min":1.347},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.85599999999999998,"15min":0.84699999999999998},"min":{"1min":0.68799999999999994,"5min":0.623,"15min":0.621},"max":{"1min":0.97399999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.75700000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575758,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47160,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324680,"statfs":{"total":96636764160,"available":96588472320,"internally_reserved":0,"allocated":2134016,"data_stored":1066004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.45400000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60099999999999998,"5min":0.64100000000000001,"15min":0.75700000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.5,"15min":0.498},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73499999999999999,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.44}]},{"osd":2,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.50700000000000001,"15min":0.50800000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.76200000000000001,"5min":0.80700000000000005,"15min":0.83299999999999996},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.54500000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73999999999999999,"5min":0.79700000000000004,"15min":0.80900000000000005},"last":0.55400000000000005}]},{"osd":3,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55700000000000005,"15min":0.56699999999999995},"min":{"1min":0.40799999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.86799999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63,"15min":0.64300000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81999999999999995,"5min":0.88100000000000001,"15min":0.91900000000000004},"last":0.46500000000000002}]},{"osd":4,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.83199999999999996,"15min":0.81899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.999,"5min":1.0980000000000001,"15min":1.236},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.83899999999999997,"15min":0.82899999999999996},"min":{"1min":0.61899999999999999,"5min":0.60199999999999998,"15min":0.56200000000000006},"max":{"1min":1.056,"5min":1.056,"15min":1.2929999999999999},"last":0.63300000000000001}]},{"osd":5,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.83199999999999996,"15min":0.82699999999999996},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.57799999999999996},"max":{"1min":0.95599999999999996,"5min":1.0820000000000001,"15min":1.369},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.53700000000000003},"max":{"1min":0.98899999999999999,"5min":1.042,"15min":1.27},"last":0.66000000000000003}]},{"osd":6,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.57999999999999996,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.052,"5min":1.052,"15min":1.1259999999999999},"last":0.622},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.80200000000000005,"15min":0.79600000000000004},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.99099999999999999,"5min":1.089,"15min":1.262},"last":0.80600000000000005}]},{"osd":7,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.85999999999999999,"15min":0.84999999999999998},"min":{"1min":0.57099999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0680000000000001,"5min":1.125,"15min":1.321},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.83299999999999996,"15min":0.84199999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.083,"5min":1.1120000000000001,"15min":1.2989999999999999},"last":0.66800000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575757,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45496,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94326344,"statfs":{"total":96636764160,"available":96590176256,"internally_reserved":0,"allocated":2138112,"data_stored":1066955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47599999999999998,"15min":0.45600000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.31},"max":{"1min":0.70099999999999996,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.50800000000000001,"15min":0.50700000000000001},"min":{"1min":0.40000000000000002,"5min":0.35099999999999998,"15min":0.316},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":0.81000000000000005},"last":0.53000000000000003}]},{"osd":1,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54100000000000004,"15min":0.51700000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.313},"max":{"1min":0.751,"5min":1.262,"15min":1.262},"last":0.433},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.39500000000000002,"5min":0.38,"15min":0.35599999999999998},"max":{"1min":0.72399999999999998,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.56100000000000005}]},{"osd":3,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.39400000000000002,"5min":0.38500000000000001,"15min":0.30599999999999999},"max":{"1min":2.3039999999999998,"5min":2.3039999999999998,"15min":2.3039999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.65700000000000003,"15min":0.64000000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.45000000000000001},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.51000000000000001}]},{"osd":4,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.81200000000000006,"15min":0.80500000000000005},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.443},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80300000000000005,"15min":0.80600000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.42399999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.73299999999999998}]},{"osd":5,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.82099999999999995,"15min":0.80600000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.48399999999999999},"max":{"1min":2.177,"5min":2.177,"15min":2.177},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.83399999999999996,"15min":0.81799999999999995},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.47899999999999998},"max":{"1min":1.319,"5min":1.319,"15min":1.3600000000000001},"last":0.74199999999999999}]},{"osd":6,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.47499999999999998},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.84299999999999997,"15min":0.83199999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.50700000000000001},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":0.84299999999999997}]},{"osd":7,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.84699999999999998,"15min":0.83799999999999997},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.502},"max":{"1min":1.528,"5min":1.528,"15min":1.528},"last":0.87},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.86199999999999999,"15min":0.84599999999999997},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.47399999999999998},"max":{"1min":2.258,"5min":2.258,"15min":2.258},"last":0.79600000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575757,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47164,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324676,"statfs":{"total":96636764160,"available":96588468224,"internally_reserved":0,"allocated":2142208,"data_stored":1067549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.45600000000000002,"15min":0.45200000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.28699999999999998},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.92700000000000005},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.49199999999999999,"15min":0.49299999999999999},"min":{"1min":0.35999999999999999,"5min":0.33200000000000002,"15min":0.30599999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.87},"last":0.36599999999999999}]},{"osd":1,"last update":"Tue Apr 2 08:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.39700000000000002,"5min":0.34699999999999998,"15min":0.309},"max":{"1min":0.90300000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59299999999999997,"15min":0.58099999999999996},"min":{"1min":0.45200000000000001,"5min":0.43099999999999999,"15min":0.40100000000000002},"max":{"1min":0.78200000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.44800000000000001}]},{"osd":2,"last update":"Tue Apr 2 08:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.63800000000000001,"15min":0.63600000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.39000000000000001},"max":{"1min":0.79500000000000004,"5min":0.83599999999999997,"15min":0.86299999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.629},"min":{"1min":0.54900000000000004,"5min":0.49399999999999999,"15min":0.40200000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.45900000000000002}]},{"osd":4,"last update":"Tue Apr 2 08:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.80900000000000005,"15min":0.78500000000000003},"min":{"1min":0.68100000000000005,"5min":0.56100000000000005,"15min":0.51500000000000001},"max":{"1min":1.0740000000000001,"5min":1.1000000000000001,"15min":1.167},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.81200000000000006,"15min":0.80100000000000005},"min":{"1min":0.67600000000000005,"5min":0.57999999999999996,"15min":0.54800000000000004},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.63900000000000001}]},{"osd":5,"last update":"Tue Apr 2 08:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.82199999999999995,"15min":0.80000000000000004},"min":{"1min":0.73699999999999999,"5min":0.61699999999999999,"15min":0.54600000000000004},"max":{"1min":1.018,"5min":1.115,"15min":1.149},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.81699999999999995,"15min":0.80000000000000004},"min":{"1min":0.63600000000000001,"5min":0.58799999999999997,"15min":0.52700000000000002},"max":{"1min":0.97799999999999998,"5min":1.0760000000000001,"15min":1.1020000000000001},"last":0.624}]},{"osd":6,"last update":"Tue Apr 2 08:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.82299999999999995,"15min":0.80500000000000005},"min":{"1min":0.622,"5min":0.622,"15min":0.56799999999999995},"max":{"1min":1.133,"5min":1.153,"15min":1.153},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86399999999999999,"15min":0.82899999999999996},"min":{"1min":0.64500000000000002,"5min":0.63700000000000001,"15min":0.55600000000000005},"max":{"1min":1.0680000000000001,"5min":1.143,"15min":1.1870000000000001},"last":0.745}]},{"osd":7,"last update":"Tue Apr 2 08:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.85499999999999998,"15min":0.82599999999999996},"min":{"1min":0.67100000000000004,"5min":0.624,"15min":0.57699999999999996},"max":{"1min":1.073,"5min":1.125,"15min":1.2070000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.871,"5min":0.86799999999999999,"15min":0.83199999999999996},"min":{"1min":0.67800000000000005,"5min":0.626,"15min":0.56499999999999995},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.73599999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575757,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59480,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312360,"statfs":{"total":96636764160,"available":96575856640,"internally_reserved":0,"allocated":2166784,"data_stored":1075821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.71499999999999997,"15min":0.70499999999999996},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.46999999999999997},"max":{"1min":0.92400000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69099999999999995,"15min":0.69299999999999995},"min":{"1min":0.50900000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.85199999999999998,"5min":0.95999999999999996,"15min":1.0429999999999999},"last":0.629}]},{"osd":1,"last update":"Tue Apr 2 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.76100000000000001,"15min":0.75600000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.95999999999999996,"5min":1.095,"15min":1.121},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78000000000000003},"min":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.49099999999999999},"max":{"1min":0.97199999999999998,"5min":1.087,"15min":1.1200000000000001},"last":0.74199999999999999}]},{"osd":2,"last update":"Tue Apr 2 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.51000000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92700000000000005,"5min":0.97899999999999998,"15min":1.1879999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72199999999999998,"15min":0.73299999999999998},"min":{"1min":0.52700000000000002,"5min":0.47599999999999998,"15min":0.46300000000000002},"max":{"1min":0.91700000000000004,"5min":1.0209999999999999,"15min":1.1259999999999999},"last":0.68899999999999995}]},{"osd":3,"last update":"Tue Apr 2 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.753,"15min":0.76300000000000001},"min":{"1min":0.52800000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98599999999999999,"5min":0.997,"15min":1.2589999999999999},"last":0.755},{"interface":"front","average":{"1min":0.75,"5min":0.74099999999999999,"15min":0.74399999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.90200000000000002,"5min":1.0389999999999999,"15min":1.077},"last":0.54600000000000004}]},{"osd":5,"last update":"Tue Apr 2 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59399999999999997,"15min":0.60499999999999998},"min":{"1min":0.434,"5min":0.40400000000000003,"15min":0.38400000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":1.0940000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.60499999999999998,"15min":0.60399999999999998},"min":{"1min":0.438,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":1.073},"last":0.67700000000000005}]},{"osd":6,"last update":"Tue Apr 2 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.63400000000000001,"15min":0.63100000000000001},"min":{"1min":0.54200000000000004,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68999999999999995,"15min":0.69099999999999995},"min":{"1min":0.50700000000000001,"5min":0.46899999999999997,"15min":0.439},"max":{"1min":0.92900000000000005,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.85699999999999998}]},{"osd":7,"last update":"Tue Apr 2 08:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.68000000000000005,"15min":0.69099999999999995},"min":{"1min":0.497,"5min":0.46100000000000002,"15min":0.442},"max":{"1min":1.048,"5min":1.048,"15min":1.129},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.77000000000000002,"15min":0.77400000000000002},"min":{"1min":0.56699999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.3440000000000001},"last":0.78200000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575760,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64156,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307684,"statfs":{"total":96636764160,"available":96571068416,"internally_reserved":0,"allocated":2760704,"data_stored":1664760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.72899999999999998,"15min":0.72199999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.377},"max":{"1min":0.878,"5min":1.502,"15min":1.502},"last":0.433},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.749,"15min":0.73599999999999999},"min":{"1min":0.58799999999999997,"5min":0.51000000000000001,"15min":0.41999999999999998},"max":{"1min":0.84899999999999998,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.44800000000000001}]},{"osd":1,"last update":"Tue Apr 2 08:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.72499999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.433},"max":{"1min":0.88900000000000001,"5min":0.999,"15min":0.999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.73499999999999999,"15min":0.73699999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.92100000000000004,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.48899999999999999}]},{"osd":2,"last update":"Tue Apr 2 08:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.76500000000000001,"15min":0.746},"min":{"1min":0.55900000000000005,"5min":0.51700000000000002,"15min":0.47599999999999998},"max":{"1min":0.90900000000000003,"5min":1.232,"15min":1.232},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.747,"15min":0.747},"min":{"1min":0.57599999999999996,"5min":0.53400000000000003,"15min":0.51000000000000001},"max":{"1min":0.88500000000000001,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.747}]},{"osd":3,"last update":"Tue Apr 2 08:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.80500000000000005,"15min":0.79300000000000004},"min":{"1min":0.65300000000000002,"5min":0.64400000000000002,"15min":0.46800000000000003},"max":{"1min":0.93799999999999994,"5min":1.556,"15min":1.556},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80200000000000005,"15min":0.80000000000000004},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.95399999999999996,"5min":1.0189999999999999,"15min":1.069},"last":0.71999999999999997}]},{"osd":4,"last update":"Tue Apr 2 08:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.626,"15min":0.61599999999999999},"min":{"1min":0.437,"5min":0.40200000000000002,"15min":0.32700000000000001},"max":{"1min":0.89500000000000002,"5min":0.97899999999999998,"15min":1.0189999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.435},"max":{"1min":0.94099999999999995,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.63400000000000001}]},{"osd":6,"last update":"Tue Apr 2 08:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.65900000000000003,"15min":0.66800000000000004},"min":{"1min":0.51300000000000001,"5min":0.47399999999999998,"15min":0.44700000000000001},"max":{"1min":0.89800000000000002,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68600000000000005,"15min":0.69499999999999995},"min":{"1min":0.503,"5min":0.46999999999999997,"15min":0.377},"max":{"1min":0.97199999999999998,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.58299999999999996}]},{"osd":7,"last update":"Tue Apr 2 08:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.47299999999999998,"5min":0.45700000000000002,"15min":0.41599999999999998},"max":{"1min":0.995,"5min":1.423,"15min":1.423},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.73599999999999999,"15min":0.748},"min":{"1min":0.58699999999999997,"5min":0.51500000000000001,"15min":0.44500000000000001},"max":{"1min":1.004,"5min":1.038,"15min":1.038},"last":0.77900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T08:52:47.355 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T08:52:47.355 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T08:52:47.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.589+0000 7f8e02400700 1 Processor -- start 2024-04-02T08:52:47.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.589+0000 7f8e02400700 1 -- start start 2024-04-02T08:52:47.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.591+0000 7f8e02400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8dfc0cd020 0x7f8dfc0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:47.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.591+0000 7f8e02400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dfc0cdae0 con 0x7f8dfc0cbb30 2024-04-02T08:52:47.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.591+0000 7f8e02400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dfc0cddc0 con 0x7f8dfc0b2070 2024-04-02T08:52:47.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.591+0000 7f8e02400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dfc0ce0a0 con 0x7f8dfc0cd020 2024-04-02T08:52:47.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.591+0000 7f8dfb400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8dfc0cbb30 0x7f8dfc0cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42824/0 (socket says 172.21.15.89:42824) 2024-04-02T08:52:47.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.591+0000 7f8dfb400700 1 -- 172.21.15.89:0/1468107936 learned_addr learned my addr 172.21.15.89:0/1468107936 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:47.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.591+0000 7f8dfa000700 1 -- 172.21.15.89:0/1468107936 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84822443 0 0) 0x7f8dfc0cdae0 con 0x7f8dfc0cbb30 2024-04-02T08:52:47.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.591+0000 7f8dfa000700 1 -- 172.21.15.89:0/1468107936 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de00035b0 con 0x7f8dfc0cbb30 2024-04-02T08:52:47.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.591+0000 7f8dfaa00700 1 --2- 172.21.15.89:0/1468107936 >> v2:172.21.15.113:3300/0 conn(0x7f8dfc0cd020 0x7f8dfc0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:47.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfa000700 1 -- 172.21.15.89:0/1468107936 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1695384261 0 0) 0x7f8de00035b0 con 0x7f8dfc0cbb30 2024-04-02T08:52:47.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfa000700 1 -- 172.21.15.89:0/1468107936 >> v2:172.21.15.113:3300/0 conn(0x7f8dfc0cd020 msgr2=0x7f8dfc0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:47.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfa000700 1 --2- 172.21.15.89:0/1468107936 >> v2:172.21.15.113:3300/0 conn(0x7f8dfc0cd020 0x7f8dfc0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfa000700 1 -- 172.21.15.89:0/1468107936 >> v1:172.21.15.113:6789/0 conn(0x7f8dfc0b2070 legacy=0x7f8dfc06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:47.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfa000700 1 -- 172.21.15.89:0/1468107936 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc06dc20 con 0x7f8dfc0cbb30 2024-04-02T08:52:47.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfa000700 1 -- 172.21.15.89:0/1468107936 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f8dec001670 con 0x7f8dfc0cbb30 2024-04-02T08:52:47.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfa000700 1 -- 172.21.15.89:0/1468107936 >> v1:172.21.15.89:6789/0 conn(0x7f8dfc0cbb30 legacy=0x7f8dfc0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:47.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfa000700 1 --2- 172.21.15.89:0/1468107936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8de0003a00 0x7f8de0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:47.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfa000700 1 --2- 172.21.15.89:0/1468107936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de0004850 0x7f8de0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:47.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfa000700 1 --2- 172.21.15.89:0/1468107936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8de0005210 0x7f8de000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:47.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfa000700 1 -- 172.21.15.89:0/1468107936 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0ce0a0 con 0x7f8de0005210 2024-04-02T08:52:47.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfa000700 1 -- 172.21.15.89:0/1468107936 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de000c0b0 con 0x7f8de0004850 2024-04-02T08:52:47.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfa000700 1 -- 172.21.15.89:0/1468107936 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de000c270 con 0x7f8de0003a00 2024-04-02T08:52:47.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfbe00700 1 --2- 172.21.15.89:0/1468107936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8de0005210 0x7f8de000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfaa00700 1 --2- 172.21.15.89:0/1468107936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de0004850 0x7f8de0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfb400700 1 --2- 172.21.15.89:0/1468107936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8de0003a00 0x7f8de0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfbe00700 1 -- 172.21.15.89:0/1468107936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8de0003a00 msgr2=0x7f8de0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfbe00700 1 --2- 172.21.15.89:0/1468107936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8de0003a00 0x7f8de0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfbe00700 1 -- 172.21.15.89:0/1468107936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de0004850 msgr2=0x7f8de0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.592+0000 7f8dfbe00700 1 --2- 172.21.15.89:0/1468107936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de0004850 0x7f8de0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.593+0000 7f8dfbe00700 1 -- 172.21.15.89:0/1468107936 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de00035b0 con 0x7f8de0005210 2024-04-02T08:52:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.593+0000 7f8dfbe00700 1 -- 172.21.15.89:0/1468107936 reap_dead start 2024-04-02T08:52:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.593+0000 7f8dfbe00700 1 --2- 172.21.15.89:0/1468107936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8de0005210 0x7f8de000ba10 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f8df400c190 tx=0x7f8de0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b3c48d3b8b9da0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.593+0000 7f8dfa000700 1 -- 172.21.15.89:0/1468107936 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df4001db0 con 0x7f8de0005210 2024-04-02T08:52:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.593+0000 7f8dfa000700 1 -- 172.21.15.89:0/1468107936 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df400ccf0 con 0x7f8de0005210 2024-04-02T08:52:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.593+0000 7f8dfa000700 1 -- 172.21.15.89:0/1468107936 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df4003c40 con 0x7f8de0005210 2024-04-02T08:52:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.593+0000 7f8e02400700 1 -- 172.21.15.89:0/1468107936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8de0005210 msgr2=0x7f8de000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.593+0000 7f8e02400700 1 --2- 172.21.15.89:0/1468107936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8de0005210 0x7f8de000ba10 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f8df400c190 tx=0x7f8de0005880 comp rx=0 tx=0).stop 2024-04-02T08:52:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.593+0000 7f8e02400700 1 -- 172.21.15.89:0/1468107936 shutdown_connections 2024-04-02T08:52:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.594+0000 7f8e02400700 1 --2- 172.21.15.89:0/1468107936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8de0005210 0x7f8de000ba10 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.594+0000 7f8e02400700 1 -- 172.21.15.89:0/1468107936 >> 172.21.15.89:0/1468107936 conn(0x7f8dfc0b0d70 msgr2=0x7f8dfc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.594+0000 7f8e02400700 1 -- 172.21.15.89:0/1468107936 shutdown_connections 2024-04-02T08:52:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.594+0000 7f8e02400700 1 -- 172.21.15.89:0/1468107936 wait complete. 2024-04-02T08:52:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.595+0000 7f8e02400700 1 Processor -- start 2024-04-02T08:52:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.596+0000 7f8e02400700 1 -- start start 2024-04-02T08:52:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.596+0000 7f8e02400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de0005210 0x7f8dfc070a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.596+0000 7f8e02400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8dfc156420 0x7f8dfc155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.596+0000 7f8e02400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dfc0aeef0 0x7f8dfc0af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8e02400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0af8e0 con 0x7f8dfc0aeef0 2024-04-02T08:52:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8e02400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc070f70 con 0x7f8de0005210 2024-04-02T08:52:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8e02400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc155cd0 con 0x7f8dfc156420 2024-04-02T08:52:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8dfbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dfc0aeef0 0x7f8dfc0af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8dfbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dfc0aeef0 0x7f8dfc0af3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52712/0 (socket says 172.21.15.89:52712) 2024-04-02T08:52:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8dfbe00700 1 -- 172.21.15.89:0/901708447 learned_addr learned my addr 172.21.15.89:0/901708447 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8dfaa00700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8dfc156420 0x7f8dfc155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:47.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8dfb400700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de0005210 0x7f8dfc070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:47.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8dfbe00700 1 -- 172.21.15.89:0/901708447 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8dfc156420 msgr2=0x7f8dfc155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:47.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8dfbe00700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8dfc156420 0x7f8dfc155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8dfbe00700 1 -- 172.21.15.89:0/901708447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de0005210 msgr2=0x7f8dfc070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:47.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8dfbe00700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de0005210 0x7f8dfc070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8dfbe00700 1 -- 172.21.15.89:0/901708447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc156bb0 con 0x7f8dfc0aeef0 2024-04-02T08:52:47.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8dfb400700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de0005210 0x7f8dfc070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:47.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.597+0000 7f8dfbe00700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dfc0aeef0 0x7f8dfc0af3a0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc0b0d00 tx=0x7f8df40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:47.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.598+0000 7f8df3400700 1 -- 172.21.15.89:0/901708447 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df4003e90 con 0x7f8dfc0aeef0 2024-04-02T08:52:47.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.598+0000 7f8df3400700 1 -- 172.21.15.89:0/901708447 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df40050d0 con 0x7f8dfc0aeef0 2024-04-02T08:52:47.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.598+0000 7f8df3400700 1 -- 172.21.15.89:0/901708447 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df4005460 con 0x7f8dfc0aeef0 2024-04-02T08:52:47.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.598+0000 7f8e02400700 1 -- 172.21.15.89:0/901708447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dfc156e90 con 0x7f8dfc0aeef0 2024-04-02T08:52:47.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.598+0000 7f8e02400700 1 -- 172.21.15.89:0/901708447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dfc157450 con 0x7f8dfc0aeef0 2024-04-02T08:52:47.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.599+0000 7f8e02400700 1 -- 172.21.15.89:0/901708447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dfc0cd8b0 con 0x7f8dfc0aeef0 2024-04-02T08:52:47.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.600+0000 7f8df3400700 1 -- 172.21.15.89:0/901708447 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8df40055c0 con 0x7f8dfc0aeef0 2024-04-02T08:52:47.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.600+0000 7f8df3400700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8ddc06adb0 0x7f8ddc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:47.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.601+0000 7f8df3400700 1 -- 172.21.15.89:0/901708447 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f8df408ad90 con 0x7f8dfc0aeef0 2024-04-02T08:52:47.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.601+0000 7f8dfb400700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8ddc06adb0 0x7f8ddc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:47.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.601+0000 7f8dfb400700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8ddc06adb0 0x7f8ddc06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8dec000c00 tx=0x7f8dec009040 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:47.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.605+0000 7f8df3400700 1 -- 172.21.15.89:0/901708447 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8df4055d50 con 0x7f8dfc0aeef0 2024-04-02T08:52:47.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.885+0000 7f8e02400700 1 -- 172.21.15.89:0/901708447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f8dfc159c30 con 0x7f8dfc0aeef0 2024-04-02T08:52:47.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.889+0000 7f8df3400700 1 -- 172.21.15.89:0/901708447 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v45) v1 ==== 190+0+0 (secure 0 0 0) 0x7f8dfc159c30 con 0x7f8dfc0aeef0 2024-04-02T08:52:47.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.892+0000 7f8e02400700 1 -- 172.21.15.89:0/901708447 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8ddc06adb0 msgr2=0x7f8ddc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:47.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.892+0000 7f8e02400700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8ddc06adb0 0x7f8ddc06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8dec000c00 tx=0x7f8dec009040 comp rx=0 tx=0).stop 2024-04-02T08:52:47.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.892+0000 7f8e02400700 1 -- 172.21.15.89:0/901708447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dfc0aeef0 msgr2=0x7f8dfc0af3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:47.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.892+0000 7f8e02400700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dfc0aeef0 0x7f8dfc0af3a0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc0b0d00 tx=0x7f8df40014a0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.892+0000 7f8e02400700 1 -- 172.21.15.89:0/901708447 shutdown_connections 2024-04-02T08:52:47.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.892+0000 7f8e02400700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8ddc06adb0 0x7f8ddc06d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.892+0000 7f8e02400700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de0005210 0x7f8dfc070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.892+0000 7f8e02400700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dfc0aeef0 0x7f8dfc0af3a0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.892+0000 7f8e02400700 1 --2- 172.21.15.89:0/901708447 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8dfc156420 0x7f8dfc155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:47.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.892+0000 7f8e02400700 1 -- 172.21.15.89:0/901708447 >> 172.21.15.89:0/901708447 conn(0x7f8dfc0b0d70 msgr2=0x7f8dfc0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:47.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.892+0000 7f8e02400700 1 -- 172.21.15.89:0/901708447 shutdown_connections 2024-04-02T08:52:47.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:47.892+0000 7f8e02400700 1 -- 172.21.15.89:0/901708447 wait complete. 2024-04-02T08:52:47.893 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T08:52:47.910 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T08:52:47.960 INFO:tasks.ceph.mds.a.smithi089.stderr:2024-04-02T08:52:47.962+0000 7f36b4a00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T08:52:48.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.103+0000 7fabc3000700 1 Processor -- start 2024-04-02T08:52:48.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.103+0000 7fabc3000700 1 -- start start 2024-04-02T08:52:48.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.104+0000 7fabc3000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabbc160be0 0x7fabbc0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:48.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.104+0000 7fabc3000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fabbc0ceca0 0x7fabbc0b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:48.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.104+0000 7fabc3000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabbc160fc0 con 0x7fabbc0ce0e0 2024-04-02T08:52:48.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.104+0000 7fabc3000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabbc0b65d0 con 0x7fabbc0ceca0 2024-04-02T08:52:48.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.104+0000 7fabc3000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc0d2a80 con 0x7fabbc160be0 2024-04-02T08:52:48.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.104+0000 7fabbb400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fabbc0ce0e0 0x7fabbc0ce590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42840/0 (socket says 172.21.15.89:42840) 2024-04-02T08:52:48.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.104+0000 7fabbb400700 1 -- 172.21.15.89:0/2046891693 learned_addr learned my addr 172.21.15.89:0/2046891693 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:48.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.104+0000 7fabc1000700 1 --2- 172.21.15.89:0/2046891693 >> v2:172.21.15.113:3300/0 conn(0x7fabbc0ceca0 0x7fabbc0b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:48.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.104+0000 7fabbbe00700 1 --2- 172.21.15.89:0/2046891693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabbc160be0 0x7fabbc0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:48.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 -- 172.21.15.89:0/2046891693 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159152533 0 0) 0x7fabbc160fc0 con 0x7fabbc0ce0e0 2024-04-02T08:52:48.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 -- 172.21.15.89:0/2046891693 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faba00035b0 con 0x7fabbc0ce0e0 2024-04-02T08:52:48.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 -- 172.21.15.89:0/2046891693 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3696869057 0 0) 0x7faba00035b0 con 0x7fabbc0ce0e0 2024-04-02T08:52:48.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 -- 172.21.15.89:0/2046891693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabbc160be0 msgr2=0x7fabbc0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:48.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 --2- 172.21.15.89:0/2046891693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabbc160be0 0x7fabbc0cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 -- 172.21.15.89:0/2046891693 >> v2:172.21.15.113:3300/0 conn(0x7fabbc0ceca0 msgr2=0x7fabbc0b6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:48.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 --2- 172.21.15.89:0/2046891693 >> v2:172.21.15.113:3300/0 conn(0x7fabbc0ceca0 0x7fabbc0b6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 -- 172.21.15.89:0/2046891693 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabbc0d21e0 con 0x7fabbc0ce0e0 2024-04-02T08:52:48.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 -- 172.21.15.89:0/2046891693 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7faba8001670 con 0x7fabbc0ce0e0 2024-04-02T08:52:48.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 -- 172.21.15.89:0/2046891693 >> v1:172.21.15.89:6789/0 conn(0x7fabbc0ce0e0 legacy=0x7fabbc0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:48.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 --2- 172.21.15.89:0/2046891693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba0003a00 0x7faba0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:48.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 --2- 172.21.15.89:0/2046891693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabbc160be0 0x7faba0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:48.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 --2- 172.21.15.89:0/2046891693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faba0004d90 0x7faba0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:48.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 -- 172.21.15.89:0/2046891693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc0b65d0 con 0x7faba0003a00 2024-04-02T08:52:48.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 -- 172.21.15.89:0/2046891693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc0d2a80 con 0x7faba0004d90 2024-04-02T08:52:48.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbaa00700 1 -- 172.21.15.89:0/2046891693 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba000ba10 con 0x7fabbc160be0 2024-04-02T08:52:48.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbbe00700 1 --2- 172.21.15.89:0/2046891693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba0003a00 0x7faba0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:48.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabc1000700 1 --2- 172.21.15.89:0/2046891693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faba0004d90 0x7faba0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:48.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.105+0000 7fabbb400700 1 --2- 172.21.15.89:0/2046891693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabbc160be0 0x7faba0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:48.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.106+0000 7fabbbe00700 1 -- 172.21.15.89:0/2046891693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabbc160be0 msgr2=0x7faba0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:48.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.106+0000 7fabbbe00700 1 --2- 172.21.15.89:0/2046891693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabbc160be0 0x7faba0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.106+0000 7fabbbe00700 1 -- 172.21.15.89:0/2046891693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faba0004d90 msgr2=0x7faba0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:48.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.106+0000 7fabbbe00700 1 --2- 172.21.15.89:0/2046891693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faba0004d90 0x7faba0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.106+0000 7fabbbe00700 1 -- 172.21.15.89:0/2046891693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba00035b0 con 0x7faba0003a00 2024-04-02T08:52:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.106+0000 7fabbbe00700 1 --2- 172.21.15.89:0/2046891693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba0003a00 0x7faba0003e20 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fabac009b60 tx=0x7faba0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f33cd0a870a705 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.106+0000 7fabbaa00700 1 -- 172.21.15.89:0/2046891693 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac00e6c0 con 0x7faba0003a00 2024-04-02T08:52:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.106+0000 7fabbaa00700 1 -- 172.21.15.89:0/2046891693 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabac00e820 con 0x7faba0003a00 2024-04-02T08:52:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.106+0000 7fabbaa00700 1 -- 172.21.15.89:0/2046891693 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac00eb90 con 0x7faba0003a00 2024-04-02T08:52:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.107+0000 7fabc3000700 1 -- 172.21.15.89:0/2046891693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba0003a00 msgr2=0x7faba0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:48.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.107+0000 7fabc3000700 1 --2- 172.21.15.89:0/2046891693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba0003a00 0x7faba0003e20 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fabac009b60 tx=0x7faba0005880 comp rx=0 tx=0).stop 2024-04-02T08:52:48.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.107+0000 7fabc1000700 1 -- 172.21.15.89:0/2046891693 reap_dead start 2024-04-02T08:52:48.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.107+0000 7fabc3000700 1 -- 172.21.15.89:0/2046891693 shutdown_connections 2024-04-02T08:52:48.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.107+0000 7fabc3000700 1 -- 172.21.15.89:0/2046891693 >> 172.21.15.89:0/2046891693 conn(0x7fabbc08aff0 msgr2=0x7fabbc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:48.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.107+0000 7fabc3000700 1 -- 172.21.15.89:0/2046891693 shutdown_connections 2024-04-02T08:52:48.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.107+0000 7fabc3000700 1 -- 172.21.15.89:0/2046891693 wait complete. 2024-04-02T08:52:48.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.108+0000 7fabc3000700 1 Processor -- start 2024-04-02T08:52:48.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.109+0000 7fabc3000700 1 -- start start 2024-04-02T08:52:48.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.109+0000 7fabc3000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faba0003a00 0x7fabbc0b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:48.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.110+0000 7fabc3000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabbc0b4b10 0x7fabbc06cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:48.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.110+0000 7fabc3000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabbc157950 0x7fabbc157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:48.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.110+0000 7fabc3000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc158340 con 0x7fabbc157950 2024-04-02T08:52:48.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.110+0000 7fabc3000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc1f6930 con 0x7faba0003a00 2024-04-02T08:52:48.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.110+0000 7fabc3000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc1f6c10 con 0x7fabbc0b4b10 2024-04-02T08:52:48.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.110+0000 7fabc1000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabbc157950 0x7fabbc157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:48.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.110+0000 7fabc1000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabbc157950 0x7fabbc157e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52726/0 (socket says 172.21.15.89:52726) 2024-04-02T08:52:48.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.110+0000 7fabc1000700 1 -- 172.21.15.89:0/3600023109 learned_addr learned my addr 172.21.15.89:0/3600023109 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:48.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.110+0000 7fabc1000700 1 -- 172.21.15.89:0/3600023109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabbc0b4b10 msgr2=0x7fabbc06cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:48.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.110+0000 7fabc1000700 1 --2- 172.21.15.89:0/3600023109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabbc0b4b10 0x7fabbc06cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.110+0000 7fabc1000700 1 -- 172.21.15.89:0/3600023109 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faba0003a00 msgr2=0x7fabbc0b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:48.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.110+0000 7fabbbe00700 1 --2- 172.21.15.89:0/3600023109 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faba0003a00 0x7fabbc0b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:48.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.110+0000 7fabc1000700 1 --2- 172.21.15.89:0/3600023109 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faba0003a00 0x7fabbc0b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.110+0000 7fabc1000700 1 -- 172.21.15.89:0/3600023109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabbc1f6ef0 con 0x7fabbc157950 2024-04-02T08:52:48.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.111+0000 7fabc1000700 1 --2- 172.21.15.89:0/3600023109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabbc157950 0x7fabbc157e00 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fabbc0b6540 tx=0x7fabb4011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:48.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.111+0000 7fabb3e00700 1 -- 172.21.15.89:0/3600023109 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb400e040 con 0x7fabbc157950 2024-04-02T08:52:48.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.111+0000 7fabb3e00700 1 -- 172.21.15.89:0/3600023109 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb400f040 con 0x7fabbc157950 2024-04-02T08:52:48.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.111+0000 7fabc3000700 1 -- 172.21.15.89:0/3600023109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabbc1f71d0 con 0x7fabbc157950 2024-04-02T08:52:48.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.111+0000 7fabb3e00700 1 -- 172.21.15.89:0/3600023109 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb400fdb0 con 0x7fabbc157950 2024-04-02T08:52:48.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.111+0000 7fabc3000700 1 -- 172.21.15.89:0/3600023109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabbc1f7790 con 0x7fabbc157950 2024-04-02T08:52:48.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.112+0000 7fabc3000700 1 -- 172.21.15.89:0/3600023109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabbc067720 con 0x7fabbc157950 2024-04-02T08:52:48.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.113+0000 7fabb3e00700 1 -- 172.21.15.89:0/3600023109 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fabb4022460 con 0x7fabbc157950 2024-04-02T08:52:48.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.114+0000 7fabb3e00700 1 --2- 172.21.15.89:0/3600023109 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fab9c06ade0 0x7fab9c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:48.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.114+0000 7fabb3e00700 1 -- 172.21.15.89:0/3600023109 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fabb4089c60 con 0x7fabbc157950 2024-04-02T08:52:48.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.114+0000 7fabbbe00700 1 --2- 172.21.15.89:0/3600023109 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fab9c06ade0 0x7fab9c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:48.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.115+0000 7fabbbe00700 1 --2- 172.21.15.89:0/3600023109 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fab9c06ade0 0x7fab9c06d2a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fabac000c00 tx=0x7fabac008030 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:48.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.119+0000 7fabb3e00700 1 -- 172.21.15.89:0/3600023109 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fabb40549d0 con 0x7fabbc157950 2024-04-02T08:52:48.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.367+0000 7fabc3000700 1 -- 172.21.15.89:0/3600023109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabbc06d770 con 0x7fabbc157950 2024-04-02T08:52:48.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.368+0000 7fabb3e00700 1 -- 172.21.15.89:0/3600023109 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4738 (secure 0 0 0) 0x7fabb4057ff0 con 0x7fabbc157950 2024-04-02T08:52:48.367 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:52:48.367 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8881,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/547244866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":547244866},{"type":"v1","addr":"172.21.15.113:6834","nonce":547244866}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":8884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2822200553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2822200553},{"type":"v1","addr":"172.21.15.113:6835","nonce":2822200553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/342426898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":342426898},{"type":"v1","addr":"172.21.15.89:6835","nonce":342426898}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T08:52:42.484905+0000","modified":"2024-04-02T08:52:45.498450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9534},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9534":{"gid":9534,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2812511535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2812511535},{"type":"v1","addr":"172.21.15.89:6837","nonce":2812511535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-02T08:52:48.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.371+0000 7fabc3000700 1 -- 172.21.15.89:0/3600023109 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fab9c06ade0 msgr2=0x7fab9c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:48.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.371+0000 7fabc3000700 1 --2- 172.21.15.89:0/3600023109 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fab9c06ade0 0x7fab9c06d2a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fabac000c00 tx=0x7fabac008030 comp rx=0 tx=0).stop 2024-04-02T08:52:48.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.371+0000 7fabc3000700 1 -- 172.21.15.89:0/3600023109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabbc157950 msgr2=0x7fabbc157e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:48.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.371+0000 7fabc3000700 1 --2- 172.21.15.89:0/3600023109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabbc157950 0x7fabbc157e00 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fabbc0b6540 tx=0x7fabb4011f80 comp rx=0 tx=0).stop 2024-04-02T08:52:48.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.371+0000 7fabc3000700 1 -- 172.21.15.89:0/3600023109 shutdown_connections 2024-04-02T08:52:48.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.371+0000 7fabc3000700 1 --2- 172.21.15.89:0/3600023109 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fab9c06ade0 0x7fab9c06d2a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.371+0000 7fabc3000700 1 --2- 172.21.15.89:0/3600023109 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faba0003a00 0x7fabbc0b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.371+0000 7fabc3000700 1 --2- 172.21.15.89:0/3600023109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabbc157950 0x7fabbc157e00 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.371+0000 7fabc3000700 1 --2- 172.21.15.89:0/3600023109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabbc0b4b10 0x7fabbc06cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.371+0000 7fabc3000700 1 -- 172.21.15.89:0/3600023109 >> 172.21.15.89:0/3600023109 conn(0x7fabbc08aff0 msgr2=0x7fabbc0b7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:48.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.371+0000 7fabc3000700 1 -- 172.21.15.89:0/3600023109 shutdown_connections 2024-04-02T08:52:48.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.371+0000 7fabc3000700 1 -- 172.21.15.89:0/3600023109 wait complete. 2024-04-02T08:52:48.371 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 81 2024-04-02T08:52:48.792 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T08:52:42.484905+0000', 'modified': '2024-04-02T08:52:45.498450+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9534}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9534': {'gid': 9534, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6837/2812511535', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6836', 'nonce': 2812511535}, {'type': 'v1', 'addr': '172.21.15.89:6837', 'nonce': 2812511535}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T08:52:48.792 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T08:52:48.792 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T08:52:48.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.989+0000 7f79ba200700 1 Processor -- start 2024-04-02T08:52:48.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.990+0000 7f79ba200700 1 -- start start 2024-04-02T08:52:48.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.991+0000 7f79ba200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79b415d9d0 0x7f79b40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:48.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.991+0000 7f79ba200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f79b40d16a0 0x7f79b40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:48.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.991+0000 7f79ba200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b4161420 con 0x7f79b40d2370 2024-04-02T08:52:48.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.991+0000 7f79ba200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b40d2820 con 0x7f79b40d16a0 2024-04-02T08:52:48.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.991+0000 7f79ba200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b4161700 con 0x7f79b415d9d0 2024-04-02T08:52:48.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.991+0000 7f79b3400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79b415d9d0 0x7f79b40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:48.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.991+0000 7f79b2a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f79b40d16a0 0x7f79b40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:48.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.991+0000 7f79b3e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f79b40d2370 0x7f79b415ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:34774/0 (socket says 172.21.15.89:34774) 2024-04-02T08:52:48.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.991+0000 7f79b3e00700 1 -- 172.21.15.89:0/1405043974 learned_addr learned my addr 172.21.15.89:0/1405043974 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:48.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.992+0000 7f79b2000700 1 -- 172.21.15.89:0/1405043974 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993419546 0 0) 0x7f79b4161420 con 0x7f79b40d2370 2024-04-02T08:52:48.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.992+0000 7f79b2000700 1 -- 172.21.15.89:0/1405043974 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79980035b0 con 0x7f79b40d2370 2024-04-02T08:52:48.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.992+0000 7f79b2a00700 1 -- 172.21.15.89:0/1405043974 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79b415d9d0 msgr2=0x7f79b40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:48.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.992+0000 7f79b2a00700 1 --2- 172.21.15.89:0/1405043974 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79b415d9d0 0x7f79b40d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.992+0000 7f79b2a00700 1 -- 172.21.15.89:0/1405043974 >> v1:172.21.15.113:6789/0 conn(0x7f79b40d2370 legacy=0x7f79b415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:48.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.992+0000 7f79b2a00700 1 -- 172.21.15.89:0/1405043974 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b41619e0 con 0x7f79b40d16a0 2024-04-02T08:52:48.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.992+0000 7f79b2a00700 1 --2- 172.21.15.89:0/1405043974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b40d16a0 0x7f79b40d1e30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f79a0009820 tx=0x7f79a0009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ee449253f6bb6ae server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:48.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.993+0000 7f79b2000700 1 -- 172.21.15.89:0/1405043974 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0005e30 con 0x7f79b40d16a0 2024-04-02T08:52:48.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.993+0000 7f79b2000700 1 -- 172.21.15.89:0/1405043974 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a0006030 con 0x7f79b40d16a0 2024-04-02T08:52:48.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.993+0000 7f79b2000700 1 -- 172.21.15.89:0/1405043974 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a00063c0 con 0x7f79b40d16a0 2024-04-02T08:52:48.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.993+0000 7f79ba200700 1 -- 172.21.15.89:0/1405043974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b40d16a0 msgr2=0x7f79b40d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:48.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.993+0000 7f79ba200700 1 --2- 172.21.15.89:0/1405043974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b40d16a0 0x7f79b40d1e30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f79a0009820 tx=0x7f79a0009b30 comp rx=0 tx=0).stop 2024-04-02T08:52:48.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.993+0000 7f79ba200700 1 -- 172.21.15.89:0/1405043974 shutdown_connections 2024-04-02T08:52:48.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.993+0000 7f79ba200700 1 --2- 172.21.15.89:0/1405043974 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79b415d9d0 0x7f79b40d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.993+0000 7f79ba200700 1 --2- 172.21.15.89:0/1405043974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b40d16a0 0x7f79b40d1e30 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.993+0000 7f79ba200700 1 -- 172.21.15.89:0/1405043974 >> 172.21.15.89:0/1405043974 conn(0x7f79b408aff0 msgr2=0x7f79b40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:48.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.993+0000 7f79ba200700 1 -- 172.21.15.89:0/1405043974 shutdown_connections 2024-04-02T08:52:48.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.993+0000 7f79ba200700 1 -- 172.21.15.89:0/1405043974 wait complete. 2024-04-02T08:52:48.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.995+0000 7f79ba200700 1 Processor -- start 2024-04-02T08:52:48.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.995+0000 7f79ba200700 1 -- start start 2024-04-02T08:52:48.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.996+0000 7f79ba200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79b40d16a0 0x7f79b4152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.996+0000 7f79ba200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b40d2370 0x7f79b4151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.996+0000 7f79ba200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79b415d9d0 0x7f79b40c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.996+0000 7f79ba200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b4153210 con 0x7f79b415d9d0 2024-04-02T08:52:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.996+0000 7f79ba200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b4152190 con 0x7f79b40d2370 2024-04-02T08:52:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.996+0000 7f79ba200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b40b61c0 con 0x7f79b40d16a0 2024-04-02T08:52:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.997+0000 7f79b3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79b415d9d0 0x7f79b40c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.997+0000 7f79b3400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79b40d16a0 0x7f79b4152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.997+0000 7f79b3400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79b40d16a0 0x7f79b4152cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:44004/0 (socket says 172.21.15.89:44004) 2024-04-02T08:52:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.997+0000 7f79b3400700 1 -- 172.21.15.89:0/2922298649 learned_addr learned my addr 172.21.15.89:0/2922298649 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.997+0000 7f79b2a00700 1 --2- 172.21.15.89:0/2922298649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b40d2370 0x7f79b4151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.997+0000 7f79b3e00700 1 -- 172.21.15.89:0/2922298649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79b40d16a0 msgr2=0x7f79b4152cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.997+0000 7f79b3e00700 1 --2- 172.21.15.89:0/2922298649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79b40d16a0 0x7f79b4152cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.997+0000 7f79b3e00700 1 -- 172.21.15.89:0/2922298649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b40d2370 msgr2=0x7f79b4151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:48.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.997+0000 7f79b3e00700 1 --2- 172.21.15.89:0/2922298649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b40d2370 0x7f79b4151c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:48.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.997+0000 7f79b3e00700 1 -- 172.21.15.89:0/2922298649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a0009430 con 0x7f79b415d9d0 2024-04-02T08:52:48.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.997+0000 7f79b3400700 1 --2- 172.21.15.89:0/2922298649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79b40d16a0 0x7f79b4152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:48.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.997+0000 7f79b3e00700 1 --2- 172.21.15.89:0/2922298649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79b415d9d0 0x7f79b40c99d0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f79ac00cab0 tx=0x7f79ac00cdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:48.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.998+0000 7f79ab400700 1 -- 172.21.15.89:0/2922298649 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ac00a9a0 con 0x7f79b415d9d0 2024-04-02T08:52:48.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.998+0000 7f79ab400700 1 -- 172.21.15.89:0/2922298649 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ac00ab00 con 0x7f79b415d9d0 2024-04-02T08:52:48.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.998+0000 7f79ba200700 1 -- 172.21.15.89:0/2922298649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b40b64a0 con 0x7f79b415d9d0 2024-04-02T08:52:48.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.998+0000 7f79ab400700 1 -- 172.21.15.89:0/2922298649 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ac00ae70 con 0x7f79b415d9d0 2024-04-02T08:52:48.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:48.998+0000 7f79ba200700 1 -- 172.21.15.89:0/2922298649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b40b6a60 con 0x7f79b415d9d0 2024-04-02T08:52:48.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.000+0000 7f79ab400700 1 -- 172.21.15.89:0/2922298649 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f79ac01b3b0 con 0x7f79b415d9d0 2024-04-02T08:52:49.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.001+0000 7f79ab400700 1 --2- 172.21.15.89:0/2922298649 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f799406adb0 0x7f799406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.001+0000 7f79ab400700 1 -- 172.21.15.89:0/2922298649 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f79ac0890f0 con 0x7f79b415d9d0 2024-04-02T08:52:49.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.001+0000 7f79b3400700 1 --2- 172.21.15.89:0/2922298649 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f799406adb0 0x7f799406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.002+0000 7f79b3400700 1 --2- 172.21.15.89:0/2922298649 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f799406adb0 0x7f799406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f79a40112c0 tx=0x7f79a4009210 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:49.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.003+0000 7f79cdc36b80 1 Processor -- start 2024-04-02T08:52:49.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.003+0000 7f79cdc36b80 1 -- start start 2024-04-02T08:52:49.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.004+0000 7f79cdc36b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102f5830 0x557b102f5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.004+0000 7f79cdc36b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102c8be0 0x557b102cba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.004+0000 7f79cdc36b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c6b60 0x557b102f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.004+0000 7f79cdc36b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b10240650 con 0x557b102c8be0 2024-04-02T08:52:49.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.004+0000 7f79cdc36b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b10221c30 con 0x557b102c6b60 2024-04-02T08:52:49.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.004+0000 7f79cdc36b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x557b1021f430 con 0x557b102f5830 2024-04-02T08:52:49.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.004+0000 7f79b2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102c8be0 0x557b102cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.004+0000 7f79b3400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102f5830 0x557b102f5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.004+0000 7f79b2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102c8be0 0x557b102cba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45954/0 (socket says 172.21.15.89:45954) 2024-04-02T08:52:49.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.004+0000 7f79b2a00700 1 -- 172.21.15.89:0/599112583 learned_addr learned my addr 172.21.15.89:0/599112583 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:49.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.004+0000 7f79b3e00700 1 --2- 172.21.15.89:0/599112583 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c6b60 0x557b102f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.005+0000 7f79b2a00700 1 -- 172.21.15.89:0/599112583 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102f5830 msgr2=0x557b102f5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:49.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.005+0000 7f79b2a00700 1 --2- 172.21.15.89:0/599112583 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102f5830 0x557b102f5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.005+0000 7f79b2a00700 1 -- 172.21.15.89:0/599112583 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c6b60 msgr2=0x557b102f2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:49.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.005+0000 7f79b2a00700 1 --2- 172.21.15.89:0/599112583 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c6b60 0x557b102f2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.005+0000 7f79b2a00700 1 -- 172.21.15.89:0/599112583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b1000e790 con 0x557b102c8be0 2024-04-02T08:52:49.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.005+0000 7f79b3400700 1 --2- 172.21.15.89:0/599112583 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102f5830 0x557b102f5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:49.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.005+0000 7f79b2a00700 1 --2- 172.21.15.89:0/599112583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102c8be0 0x557b102cba10 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f79a0013040 tx=0x7f79a0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a762fafa0923c44 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:49.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.005+0000 7f799f400700 1 -- 172.21.15.89:0/599112583 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0021070 con 0x557b102c8be0 2024-04-02T08:52:49.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.005+0000 7f799f400700 1 -- 172.21.15.89:0/599112583 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a0005cb0 con 0x557b102c8be0 2024-04-02T08:52:49.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.005+0000 7f799f400700 1 -- 172.21.15.89:0/599112583 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0004040 con 0x557b102c8be0 2024-04-02T08:52:49.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.006+0000 7f79cdc36b80 1 -- 172.21.15.89:0/599112583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102c8be0 msgr2=0x557b102cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:49.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.006+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/599112583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102c8be0 0x557b102cba10 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f79a0013040 tx=0x7f79a0010000 comp rx=0 tx=0).stop 2024-04-02T08:52:49.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.006+0000 7f79cdc36b80 1 -- 172.21.15.89:0/599112583 shutdown_connections 2024-04-02T08:52:49.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.006+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/599112583 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c6b60 0x557b102f2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.006+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/599112583 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102f5830 0x557b102f5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.006+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/599112583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102c8be0 0x557b102cba10 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.006+0000 7f79cdc36b80 1 -- 172.21.15.89:0/599112583 >> 172.21.15.89:0/599112583 conn(0x557b102296a0 msgr2=0x557b10310640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:49.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.006+0000 7f79cdc36b80 1 -- 172.21.15.89:0/599112583 shutdown_connections 2024-04-02T08:52:49.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.006+0000 7f79cdc36b80 1 -- 172.21.15.89:0/599112583 wait complete. 2024-04-02T08:52:49.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.007+0000 7f79cdc36b80 1 -- start start 2024-04-02T08:52:49.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.007+0000 7f79cdc36b80 1 Processor -- start 2024-04-02T08:52:49.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.008+0000 7f79cdc36b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102c6b60 0x557b102f13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.008+0000 7f79cdc36b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c8be0 0x557b102f85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.008+0000 7f79cdc36b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102f5830 0x557b102f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.008+0000 7f79cdc36b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b1021ad20 con 0x557b102c6b60 2024-04-02T08:52:49.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.008+0000 7f79cdc36b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b101b6770 con 0x557b102f5830 2024-04-02T08:52:49.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.008+0000 7f79cdc36b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x557b101d96f0 con 0x557b102c8be0 2024-04-02T08:52:49.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.008+0000 7f79cdc36b80 10 client.0 _ll_register_callbacks cb 0x557b101c05e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T08:52:49.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.008+0000 7f79b3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102c6b60 0x557b102f13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.008+0000 7f79b3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102c6b60 0x557b102f13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45964/0 (socket says 172.21.15.89:45964) 2024-04-02T08:52:49.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.008+0000 7f79b3400700 1 -- 172.21.15.89:0/3239190657 learned_addr learned my addr 172.21.15.89:0/3239190657 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:49.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.009+0000 7f79b2a00700 1 --2- 172.21.15.89:0/3239190657 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c8be0 0x557b102f85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.009+0000 7f79b3e00700 1 --2- 172.21.15.89:0/3239190657 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102f5830 0x557b102f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.009+0000 7f79b3400700 1 -- 172.21.15.89:0/3239190657 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c8be0 msgr2=0x557b102f85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:49.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.009+0000 7f79b3400700 1 --2- 172.21.15.89:0/3239190657 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c8be0 0x557b102f85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.009+0000 7f79b3400700 1 -- 172.21.15.89:0/3239190657 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102f5830 msgr2=0x557b102f27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.009+0000 7f79b3400700 1 --2- 172.21.15.89:0/3239190657 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102f5830 0x557b102f27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.009+0000 7f79b3400700 1 -- 172.21.15.89:0/3239190657 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a4011da0 con 0x557b102c6b60 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.009+0000 7f79b3400700 1 --2- 172.21.15.89:0/3239190657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102c6b60 0x557b102f13f0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f79a4003a20 tx=0x7f79a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcfdd31204650ed8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.009+0000 7f7993e00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.009+0000 7f7993e00700 1 -- 172.21.15.89:0/3239190657 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a40148d0 con 0x557b102c6b60 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.009+0000 7f7993e00700 1 -- 172.21.15.89:0/3239190657 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f79a4014a30 con 0x557b102c6b60 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.009+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3239190657 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557b102bd920 con 0x557b102c6b60 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.010+0000 7f7993e00700 1 -- 172.21.15.89:0/3239190657 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79a4014bf0 con 0x557b102c6b60 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.010+0000 7f79cdc36b80 10 client.9672 fetch_fsmap learned FSMap version 81 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.010+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3239190657 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) v3 -- 0x7f79b41619e0 con 0x557b102c6b60 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.010+0000 7f7993e00700 1 -- 172.21.15.89:0/3239190657 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557b102bd920 con 0x557b102c6b60 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.010+0000 7f7993e00700 1 -- 172.21.15.89:0/3239190657 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 81) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f79a40233f0 con 0x557b102c6b60 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.010+0000 7f79cdc36b80 10 client.9672 fetch_fsmap finished waiting for FSMap version 81 2024-04-02T08:52:49.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.010+0000 7f79cdc36b80 10 client.9672 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T08:52:49.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.010+0000 7f79cdc36b80 20 client.9672 populate_metadata read hostname 'smithi089' 2024-04-02T08:52:49.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.010+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/3239190657 >> [v2:172.21.15.89:6836/2812511535,v1:172.21.15.89:6837/2812511535] conn(0x557b102ef9c0 0x557b102efda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.010+0000 7f79cdc36b80 4 client.9672 mds_command: new command op to 9534 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T08:52:49.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.010+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3239190657 --> [v2:172.21.15.89:6836/2812511535,v1:172.21.15.89:6837/2812511535] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b1029a6f0 con 0x557b102ef9c0 2024-04-02T08:52:49.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.010+0000 7f79b2a00700 1 --2- 172.21.15.89:0/3239190657 >> [v2:172.21.15.89:6836/2812511535,v1:172.21.15.89:6837/2812511535] conn(0x557b102ef9c0 0x557b102efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.011+0000 7f79b2a00700 1 --2- 172.21.15.89:0/3239190657 >> [v2:172.21.15.89:6836/2812511535,v1:172.21.15.89:6837/2812511535] conn(0x557b102ef9c0 0x557b102efda0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c93db286be3722c server_cookie=852964ce2c8ee1ae in_seq=0 out_seq=0 2024-04-02T08:52:49.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.011+0000 7f7993e00700 10 client.9672 ms_handle_connect on v2:172.21.15.89:6836/2812511535 2024-04-02T08:52:49.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.011+0000 7f7993e00700 1 -- 172.21.15.89:0/3239190657 <== mds.0 v2:172.21.15.89:6836/2812511535 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557b1029a6f0 con 0x557b102ef9c0 2024-04-02T08:52:49.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.011+0000 7f7993e00700 10 client.9672 handle_command_reply: tid=0 2024-04-02T08:52:49.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.011+0000 7f79cdc36b80 1 client.9672 shutdown 2024-04-02T08:52:49.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.012+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3239190657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102c6b60 msgr2=0x557b102f13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:49.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.012+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/3239190657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102c6b60 0x557b102f13f0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f79a4003a20 tx=0x7f79a4004750 comp rx=0 tx=0).stop 2024-04-02T08:52:49.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.012+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3239190657 shutdown_connections 2024-04-02T08:52:49.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.012+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/3239190657 >> [v2:172.21.15.89:6836/2812511535,v1:172.21.15.89:6837/2812511535] conn(0x557b102ef9c0 0x557b102efda0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.012+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/3239190657 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102f5830 0x557b102f27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.012+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/3239190657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102c6b60 0x557b102f13f0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.012+0000 7f7993e00700 0 client.9672 ms_handle_reset on v2:172.21.15.89:6836/2812511535 2024-04-02T08:52:49.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.012+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/3239190657 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c8be0 0x557b102f85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.012+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3239190657 >> 172.21.15.89:0/3239190657 conn(0x557b102296a0 msgr2=0x557b103132b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:49.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.012+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3239190657 shutdown_connections 2024-04-02T08:52:49.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.012+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3239190657 wait complete. 2024-04-02T08:52:49.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.012+0000 7f79cdc36b80 20 client.9672 trim_cache size 0 max 16384 2024-04-02T08:52:49.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.030+0000 7f79cdc36b80 1 Processor -- start 2024-04-02T08:52:49.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.030+0000 7f79cdc36b80 1 -- start start 2024-04-02T08:52:49.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.031+0000 7f79cdc36b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c6b60 0x557b102c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.031+0000 7f79cdc36b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c8be0 0x557b102c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.031+0000 7f79cdc36b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102f5830 0x557b102c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.031+0000 7f79cdc36b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b1000f390 con 0x557b102f5830 2024-04-02T08:52:49.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.031+0000 7f79cdc36b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b103105f0 con 0x557b102c8be0 2024-04-02T08:52:49.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.031+0000 7f79cdc36b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x557b102f5f00 con 0x557b102c6b60 2024-04-02T08:52:49.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.031+0000 7f79b3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102f5830 0x557b102c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.031+0000 7f79b3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102f5830 0x557b102c8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45970/0 (socket says 172.21.15.89:45970) 2024-04-02T08:52:49.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.031+0000 7f79b3e00700 1 -- 172.21.15.89:0/4124800233 learned_addr learned my addr 172.21.15.89:0/4124800233 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:49.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.032+0000 7f79b2a00700 1 --2- 172.21.15.89:0/4124800233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c8be0 0x557b102c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.032+0000 7f79b3400700 1 --2- 172.21.15.89:0/4124800233 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c6b60 0x557b102c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.032+0000 7f79b3e00700 1 -- 172.21.15.89:0/4124800233 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c6b60 msgr2=0x557b102c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:49.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.032+0000 7f79b3e00700 1 --2- 172.21.15.89:0/4124800233 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c6b60 0x557b102c55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.032+0000 7f79b3e00700 1 -- 172.21.15.89:0/4124800233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c8be0 msgr2=0x557b102c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:49.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.032+0000 7f79b3e00700 1 --2- 172.21.15.89:0/4124800233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c8be0 0x557b102c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.032+0000 7f79b3e00700 1 -- 172.21.15.89:0/4124800233 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b102f9820 con 0x557b102f5830 2024-04-02T08:52:49.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.032+0000 7f79b2a00700 1 --2- 172.21.15.89:0/4124800233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c8be0 0x557b102c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:52:49.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.032+0000 7f79b3400700 1 --2- 172.21.15.89:0/4124800233 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c6b60 0x557b102c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:52:49.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.032+0000 7f79b3e00700 1 --2- 172.21.15.89:0/4124800233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102f5830 0x557b102c8fc0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f79ac0541f0 tx=0x7f79ac05ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0adca4f00c6b0d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:49.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.032+0000 7f799fe00700 1 -- 172.21.15.89:0/4124800233 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ac058150 con 0x557b102f5830 2024-04-02T08:52:49.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.032+0000 7f799fe00700 1 -- 172.21.15.89:0/4124800233 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ac055ef0 con 0x557b102f5830 2024-04-02T08:52:49.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.032+0000 7f799fe00700 1 -- 172.21.15.89:0/4124800233 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ac0573e0 con 0x557b102f5830 2024-04-02T08:52:49.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.033+0000 7f79cdc36b80 1 -- 172.21.15.89:0/4124800233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102f5830 msgr2=0x557b102c8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:49.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.033+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/4124800233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102f5830 0x557b102c8fc0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f79ac0541f0 tx=0x7f79ac05ba70 comp rx=0 tx=0).stop 2024-04-02T08:52:49.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.033+0000 7f79cdc36b80 1 -- 172.21.15.89:0/4124800233 shutdown_connections 2024-04-02T08:52:49.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.033+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/4124800233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102f5830 0x557b102c8fc0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.033+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/4124800233 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c6b60 0x557b102c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.033+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/4124800233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c8be0 0x557b102c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.033+0000 7f79cdc36b80 1 -- 172.21.15.89:0/4124800233 >> 172.21.15.89:0/4124800233 conn(0x557b102296a0 msgr2=0x557b102cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:49.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.033+0000 7f79cdc36b80 1 -- 172.21.15.89:0/4124800233 shutdown_connections 2024-04-02T08:52:49.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.033+0000 7f79cdc36b80 1 -- 172.21.15.89:0/4124800233 wait complete. 2024-04-02T08:52:49.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.033+0000 7f79cdc36b80 1 -- start start 2024-04-02T08:52:49.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.034+0000 7f79cdc36b80 1 Processor -- start 2024-04-02T08:52:49.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79cdc36b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c6b60 0x557b102c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79cdc36b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c8be0 0x557b102c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79cdc36b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102f5830 0x557b102efda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79cdc36b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b102f0b80 con 0x557b102f5830 2024-04-02T08:52:49.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79cdc36b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b10378250 con 0x557b102c8be0 2024-04-02T08:52:49.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79cdc36b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x557b10378780 con 0x557b102c6b60 2024-04-02T08:52:49.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79cdc36b80 10 client.0 _ll_register_callbacks cb 0x7f79a4004430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T08:52:49.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79b3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102f5830 0x557b102efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79b3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102f5830 0x557b102efda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45972/0 (socket says 172.21.15.89:45972) 2024-04-02T08:52:49.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79b3e00700 1 -- 172.21.15.89:0/3821011562 learned_addr learned my addr 172.21.15.89:0/3821011562 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:49.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79b3400700 1 --2- 172.21.15.89:0/3821011562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c6b60 0x557b102c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79b2a00700 1 --2- 172.21.15.89:0/3821011562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c8be0 0x557b102c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79b3e00700 1 -- 172.21.15.89:0/3821011562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c6b60 msgr2=0x557b102c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:49.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79b3e00700 1 --2- 172.21.15.89:0/3821011562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c6b60 0x557b102c55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79b3e00700 1 -- 172.21.15.89:0/3821011562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c8be0 msgr2=0x557b102c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:49.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79b3e00700 1 --2- 172.21.15.89:0/3821011562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c8be0 0x557b102c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79b3e00700 1 -- 172.21.15.89:0/3821011562 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b40b6a60 con 0x557b102f5830 2024-04-02T08:52:49.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f79b3e00700 1 --2- 172.21.15.89:0/3821011562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102f5830 0x557b102efda0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f79ac0579f0 tx=0x7f79ac00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec85a5f7c29364ca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:49.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.035+0000 7f7993e00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T08:52:49.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f7993e00700 1 -- 172.21.15.89:0/3821011562 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ac0870b0 con 0x557b102f5830 2024-04-02T08:52:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f7993e00700 1 -- 172.21.15.89:0/3821011562 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f79ac053ae0 con 0x557b102f5830 2024-04-02T08:52:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3821011562 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79a0005cb0 con 0x557b102f5830 2024-04-02T08:52:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f7993e00700 1 -- 172.21.15.89:0/3821011562 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79ac055ef0 con 0x557b102f5830 2024-04-02T08:52:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f79cdc36b80 10 client.9678 fetch_fsmap learned FSMap version 81 2024-04-02T08:52:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3821011562 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) v3 -- 0x557b100fb0a0 con 0x557b102f5830 2024-04-02T08:52:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f7993e00700 1 -- 172.21.15.89:0/3821011562 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a0005cb0 con 0x557b102f5830 2024-04-02T08:52:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f7993e00700 1 -- 172.21.15.89:0/3821011562 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 81) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f79ac053ca0 con 0x557b102f5830 2024-04-02T08:52:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f79cdc36b80 10 client.9678 fetch_fsmap finished waiting for FSMap version 81 2024-04-02T08:52:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f79cdc36b80 10 client.9678 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T08:52:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f79cdc36b80 20 client.9678 populate_metadata read hostname 'smithi089' 2024-04-02T08:52:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/3821011562 >> [v2:172.21.15.89:6836/2812511535,v1:172.21.15.89:6837/2812511535] conn(0x557b1037b990 0x557b102f02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f79cdc36b80 4 client.9678 mds_command: new command op to 9534 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T08:52:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3821011562 --> [v2:172.21.15.89:6836/2812511535,v1:172.21.15.89:6837/2812511535] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557b10054040 con 0x557b1037b990 2024-04-02T08:52:49.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.036+0000 7f79b3400700 1 --2- 172.21.15.89:0/3821011562 >> [v2:172.21.15.89:6836/2812511535,v1:172.21.15.89:6837/2812511535] conn(0x557b1037b990 0x557b102f02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f79b3400700 1 --2- 172.21.15.89:0/3821011562 >> [v2:172.21.15.89:6836/2812511535,v1:172.21.15.89:6837/2812511535] conn(0x557b1037b990 0x557b102f02e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7139a6c33b395f8 server_cookie=f633d82b329e8dc1 in_seq=0 out_seq=0 2024-04-02T08:52:49.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f7993e00700 10 client.9678 ms_handle_connect on v2:172.21.15.89:6836/2812511535 2024-04-02T08:52:49.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f7993e00700 1 -- 172.21.15.89:0/3821011562 <== mds.0 v2:172.21.15.89:6836/2812511535 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x557b10054040 con 0x557b1037b990 2024-04-02T08:52:49.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f7993e00700 10 client.9678 handle_command_reply: tid=0 2024-04-02T08:52:49.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f79cdc36b80 1 client.9678 shutdown 2024-04-02T08:52:49.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3821011562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102f5830 msgr2=0x557b102efda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:49.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/3821011562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102f5830 0x557b102efda0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f79ac0579f0 tx=0x7f79ac00cf50 comp rx=0 tx=0).stop 2024-04-02T08:52:49.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3821011562 shutdown_connections 2024-04-02T08:52:49.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/3821011562 >> [v2:172.21.15.89:6836/2812511535,v1:172.21.15.89:6837/2812511535] conn(0x557b1037b990 0x557b102f02e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/3821011562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b102f5830 0x557b102efda0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/3821011562 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x557b102c6b60 0x557b102c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f79cdc36b80 1 --2- 172.21.15.89:0/3821011562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557b102c8be0 0x557b102c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3821011562 >> 172.21.15.89:0/3821011562 conn(0x557b102296a0 msgr2=0x557b102f7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:49.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f7993e00700 0 client.9678 ms_handle_reset on v2:172.21.15.89:6836/2812511535 2024-04-02T08:52:49.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.037+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3821011562 shutdown_connections 2024-04-02T08:52:49.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.038+0000 7f79cdc36b80 1 -- 172.21.15.89:0/3821011562 wait complete. 2024-04-02T08:52:49.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.038+0000 7f79cdc36b80 20 client.9678 trim_cache size 0 max 16384 2024-04-02T08:52:49.041 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T08:52:49.041 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T08:52:49.041 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T08:52:49.041 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 9534, 2024-04-02T08:52:49.041 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T08:52:49.041 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T08:52:49.042 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T08:52:49.042 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 6.5501077270000003, 2024-04-02T08:52:49.042 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 81, 2024-04-02T08:52:49.042 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 172, 2024-04-02T08:52:49.042 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 172, 2024-04-02T08:52:49.042 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 11.566249439 2024-04-02T08:52:49.042 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T08:52:49.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.038+0000 7f79ba200700 1 -- 172.21.15.89:0/2922298649 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f799406adb0 msgr2=0x7f799406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:49.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.038+0000 7f79ba200700 1 --2- 172.21.15.89:0/2922298649 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f799406adb0 0x7f799406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f79a40112c0 tx=0x7f79a4009210 comp rx=0 tx=0).stop 2024-04-02T08:52:49.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.038+0000 7f79ba200700 1 -- 172.21.15.89:0/2922298649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79b415d9d0 msgr2=0x7f79b40c99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:49.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.038+0000 7f79ba200700 1 --2- 172.21.15.89:0/2922298649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79b415d9d0 0x7f79b40c99d0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f79ac00cab0 tx=0x7f79ac00cdc0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.038+0000 7f79ba200700 1 -- 172.21.15.89:0/2922298649 shutdown_connections 2024-04-02T08:52:49.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.038+0000 7f79ba200700 1 --2- 172.21.15.89:0/2922298649 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f799406adb0 0x7f799406d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.038+0000 7f79ba200700 1 --2- 172.21.15.89:0/2922298649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79b415d9d0 0x7f79b40c99d0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.038+0000 7f79ba200700 1 --2- 172.21.15.89:0/2922298649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f79b40d16a0 0x7f79b4152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.038+0000 7f79ba200700 1 --2- 172.21.15.89:0/2922298649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b40d2370 0x7f79b4151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.038+0000 7f79ba200700 1 -- 172.21.15.89:0/2922298649 >> 172.21.15.89:0/2922298649 conn(0x7f79b408aff0 msgr2=0x7f79b40b1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:49.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.038+0000 7f79ba200700 1 -- 172.21.15.89:0/2922298649 shutdown_connections 2024-04-02T08:52:49.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.038+0000 7f79ba200700 1 -- 172.21.15.89:0/2922298649 wait complete. 2024-04-02T08:52:49.055 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T08:52:49.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.246+0000 7f1ff6c00700 1 Processor -- start 2024-04-02T08:52:49.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.247+0000 7f1ff6c00700 1 -- start start 2024-04-02T08:52:49.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.248+0000 7f1ff6c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1ff01619d0 0x7f1ff01685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.248+0000 7f1ff6c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff00d30d0 con 0x7f1ff015d640 2024-04-02T08:52:49.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.248+0000 7f1ff6c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff00b6840 con 0x7f1ff00ceec0 2024-04-02T08:52:49.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.248+0000 7f1ff6c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff00d2b30 con 0x7f1ff01619d0 2024-04-02T08:52:49.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.248+0000 7f1fefe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1ff015d640 0x7f1ff00d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38670/0 (socket says 172.21.15.89:38670) 2024-04-02T08:52:49.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.248+0000 7f1fefe00700 1 -- 172.21.15.89:0/3903888595 learned_addr learned my addr 172.21.15.89:0/3903888595 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:49.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.248+0000 7f1ff4c00700 1 --2- 172.21.15.89:0/3903888595 >> v2:172.21.15.113:3300/0 conn(0x7f1ff01619d0 0x7f1ff01685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.248+0000 7f1feea00700 1 -- 172.21.15.89:0/3903888595 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019045411 0 0) 0x7f1ff00d30d0 con 0x7f1ff015d640 2024-04-02T08:52:49.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.248+0000 7f1feea00700 1 -- 172.21.15.89:0/3903888595 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fd40035b0 con 0x7f1ff015d640 2024-04-02T08:52:49.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.249+0000 7f1feea00700 1 -- 172.21.15.89:0/3903888595 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947416144 0 0) 0x7f1ff00b6840 con 0x7f1ff00ceec0 2024-04-02T08:52:49.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.249+0000 7f1feea00700 1 -- 172.21.15.89:0/3903888595 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff00d30d0 con 0x7f1ff00ceec0 2024-04-02T08:52:49.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.249+0000 7f1ff4c00700 1 -- 172.21.15.89:0/3903888595 >> v1:172.21.15.113:6789/0 conn(0x7f1ff00ceec0 legacy=0x7f1ff01612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:49.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.249+0000 7f1ff4c00700 1 -- 172.21.15.89:0/3903888595 >> v1:172.21.15.89:6789/0 conn(0x7f1ff015d640 legacy=0x7f1ff00d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:49.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.249+0000 7f1ff4c00700 1 -- 172.21.15.89:0/3903888595 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff00d2df0 con 0x7f1ff01619d0 2024-04-02T08:52:49.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.249+0000 7f1ff4c00700 1 --2- 172.21.15.89:0/3903888595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ff01619d0 0x7f1ff01685f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8009d90 tx=0x7f1fe800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51b250ffdcffc466 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:49.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.249+0000 7f1feea00700 1 -- 172.21.15.89:0/3903888595 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe800e070 con 0x7f1ff01619d0 2024-04-02T08:52:49.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.249+0000 7f1feea00700 1 -- 172.21.15.89:0/3903888595 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe800bb20 con 0x7f1ff01619d0 2024-04-02T08:52:49.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.249+0000 7f1feea00700 1 -- 172.21.15.89:0/3903888595 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8012430 con 0x7f1ff01619d0 2024-04-02T08:52:49.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.250+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3903888595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ff01619d0 msgr2=0x7f1ff01685f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:49.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.250+0000 7f1ff6c00700 1 --2- 172.21.15.89:0/3903888595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ff01619d0 0x7f1ff01685f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8009d90 tx=0x7f1fe800b330 comp rx=0 tx=0).stop 2024-04-02T08:52:49.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.250+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3903888595 shutdown_connections 2024-04-02T08:52:49.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.250+0000 7f1ff6c00700 1 --2- 172.21.15.89:0/3903888595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ff01619d0 0x7f1ff01685f0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.250+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3903888595 >> 172.21.15.89:0/3903888595 conn(0x7f1ff008aff0 msgr2=0x7f1ff00d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:49.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.250+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3903888595 shutdown_connections 2024-04-02T08:52:49.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.250+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3903888595 wait complete. 2024-04-02T08:52:49.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.252+0000 7f1ff6c00700 1 Processor -- start 2024-04-02T08:52:49.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.252+0000 7f1ff6c00700 1 -- start start 2024-04-02T08:52:49.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.253+0000 7f1ff6c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ff00ceec0 0x7f1ff009e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.253+0000 7f1ff6c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff015d640 0x7f1ff01e6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.253+0000 7f1ff6c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ff01619d0 0x7f1ff01e9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.253+0000 7f1ff6c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff01e7180 con 0x7f1ff015d640 2024-04-02T08:52:49.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.253+0000 7f1ff6c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff01e7460 con 0x7f1ff01619d0 2024-04-02T08:52:49.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.253+0000 7f1ff6c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff01e7740 con 0x7f1ff00ceec0 2024-04-02T08:52:49.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.253+0000 7f1fef400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff015d640 0x7f1ff01e6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.253+0000 7f1fef400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff015d640 0x7f1ff01e6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45974/0 (socket says 172.21.15.89:45974) 2024-04-02T08:52:49.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.253+0000 7f1fef400700 1 -- 172.21.15.89:0/3422333037 learned_addr learned my addr 172.21.15.89:0/3422333037 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:49.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.253+0000 7f1fef400700 1 -- 172.21.15.89:0/3422333037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ff00ceec0 msgr2=0x7f1ff009e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:49.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.253+0000 7f1fef400700 1 --2- 172.21.15.89:0/3422333037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ff00ceec0 0x7f1ff009e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.253+0000 7f1fef400700 1 -- 172.21.15.89:0/3422333037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ff01619d0 msgr2=0x7f1ff01e9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:49.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.253+0000 7f1fef400700 1 --2- 172.21.15.89:0/3422333037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ff01619d0 0x7f1ff01e9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.253+0000 7f1fef400700 1 -- 172.21.15.89:0/3422333037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe8009a70 con 0x7f1ff015d640 2024-04-02T08:52:49.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.254+0000 7f1fef400700 1 --2- 172.21.15.89:0/3422333037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff015d640 0x7f1ff01e6ac0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00c520 tx=0x7f1fdc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:49.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.254+0000 7f1fe7e00700 1 -- 172.21.15.89:0/3422333037 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0052d0 con 0x7f1ff015d640 2024-04-02T08:52:49.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.254+0000 7f1fe7e00700 1 -- 172.21.15.89:0/3422333037 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc005430 con 0x7f1ff015d640 2024-04-02T08:52:49.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.254+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3422333037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff01e7a20 con 0x7f1ff015d640 2024-04-02T08:52:49.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.254+0000 7f1fe7e00700 1 -- 172.21.15.89:0/3422333037 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0038c0 con 0x7f1ff015d640 2024-04-02T08:52:49.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.254+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3422333037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff01e9860 con 0x7f1ff015d640 2024-04-02T08:52:49.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.255+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3422333037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff01e7f40 con 0x7f1ff015d640 2024-04-02T08:52:49.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.256+0000 7f1fe7e00700 1 -- 172.21.15.89:0/3422333037 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1fdc003a20 con 0x7f1ff015d640 2024-04-02T08:52:49.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.257+0000 7f1fe7e00700 1 --2- 172.21.15.89:0/3422333037 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1fd006adb0 0x7f1fd006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:49.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.257+0000 7f1fe7e00700 1 -- 172.21.15.89:0/3422333037 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f1fdc088d10 con 0x7f1ff015d640 2024-04-02T08:52:49.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.257+0000 7f1fefe00700 1 --2- 172.21.15.89:0/3422333037 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1fd006adb0 0x7f1fd006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:49.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.258+0000 7f1fefe00700 1 --2- 172.21.15.89:0/3422333037 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1fd006adb0 0x7f1fd006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00016b0 tx=0x7f1fe00079c0 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:49.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.262+0000 7f1fe7e00700 1 -- 172.21.15.89:0/3422333037 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ff01e7f40 con 0x7f1ff015d640 2024-04-02T08:52:49.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.507+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3422333037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1ff01e7f40 con 0x7f1ff015d640 2024-04-02T08:52:49.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.508+0000 7f1fe7e00700 1 -- 172.21.15.89:0/3422333037 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v81) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1ff01e7f40 con 0x7f1ff015d640 2024-04-02T08:52:49.506 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T08:52:49.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.510+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3422333037 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1fd006adb0 msgr2=0x7f1fd006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:49.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.510+0000 7f1ff6c00700 1 --2- 172.21.15.89:0/3422333037 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1fd006adb0 0x7f1fd006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00016b0 tx=0x7f1fe00079c0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.510+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3422333037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff015d640 msgr2=0x7f1ff01e6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.510+0000 7f1ff6c00700 1 --2- 172.21.15.89:0/3422333037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff015d640 0x7f1ff01e6ac0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00c520 tx=0x7f1fdc00c8e0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.510+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3422333037 shutdown_connections 2024-04-02T08:52:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.510+0000 7f1ff6c00700 1 --2- 172.21.15.89:0/3422333037 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1fd006adb0 0x7f1fd006d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.510+0000 7f1ff6c00700 1 --2- 172.21.15.89:0/3422333037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ff01619d0 0x7f1ff01e9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.510+0000 7f1ff6c00700 1 --2- 172.21.15.89:0/3422333037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ff00ceec0 0x7f1ff009e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.510+0000 7f1ff6c00700 1 --2- 172.21.15.89:0/3422333037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff015d640 0x7f1ff01e6ac0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.510+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3422333037 >> 172.21.15.89:0/3422333037 conn(0x7f1ff008aff0 msgr2=0x7f1ff015e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.510+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3422333037 shutdown_connections 2024-04-02T08:52:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:49.510+0000 7f1ff6c00700 1 -- 172.21.15.89:0/3422333037 wait complete. 2024-04-02T08:52:49.527 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T08:52:49.528 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T08:52:49.528 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T08:52:49.528 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T08:52:49.528 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T08:52:49.528 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T08:52:49.528 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T08:52:49.528 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T08:52:49.528 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T08:52:49.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:49.528 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T08:52:49.573 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T08:52:49.573 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T08:52:49.573 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T08:52:49.573 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T08:52:49.573 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T08:52:49.573 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T08:52:49.573 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T08:52:49.574 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T08:52:49.574 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T08:52:49.574 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T08:52:49.574 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T08:52:49.574 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T08:52:49.574 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T08:52:49.574 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T08:52:49.574 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T08:52:49.574 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T08:52:49.574 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T08:52:49.574 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 37859sec preferred_lft 37859sec 2024-04-02T08:52:49.574 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T08:52:49.574 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T08:52:49.574 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T08:52:49.575 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T08:52:49.575 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T08:52:49.575 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T08:52:49.575 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T08:52:49.575 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T08:52:49.575 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T08:52:49.575 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T08:52:49.575 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T08:52:49.575 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T08:52:49.575 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T08:52:49.575 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T08:52:49.575 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T08:52:49.575 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T08:52:49.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:49.576 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T08:52:49.681 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T08:52:49.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:49.684 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T08:52:49.732 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T08:52:49.732 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T08:52:49.733 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T08:52:49.733 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T08:52:49.733 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T08:52:49.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:49.733 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T08:52:49.733 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T08:52:49.733 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T08:52:49.733 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T08:52:49.733 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T08:52:49.734 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T08:52:49.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:49.895 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T08:52:49.940 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T08:52:49.940 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T08:52:49.941 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T08:52:49.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:49.941 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T08:52:50.049 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T08:52:50.049 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T08:52:50.093 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T08:52:50.093 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T08:52:50.093 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T08:52:50.177 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T08:52:50.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:50.177 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T08:52:50.317 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T08:52:50.318 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T08:52:50.345 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T08:52:50.346 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T08:52:50.346 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T08:52:50.346 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T08:52:50.346 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T08:52:50.350 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T08:52:50.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:50.351 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T08:52:50.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:50.418 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T08:52:50.519 INFO:teuthology.orchestra.run.smithi089.stdout:client7916 2024-04-02T08:52:50.520 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T08:52:50.520 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:52:50.565 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T08:52:50.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.798+0000 7f3c77000700 1 Processor -- start 2024-04-02T08:52:50.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.798+0000 7f3c77000700 1 -- start start 2024-04-02T08:52:50.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.799+0000 7f3c77000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3c701611b0 0x7f3c7015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:50.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.799+0000 7f3c77000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3c700cee10 0x7f3c70161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:50.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.799+0000 7f3c77000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c700ce270 con 0x7f3c7015f4d0 2024-04-02T08:52:50.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.799+0000 7f3c77000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7015b600 con 0x7f3c701611b0 2024-04-02T08:52:50.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.799+0000 7f3c77000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c700d2760 con 0x7f3c700cee10 2024-04-02T08:52:50.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.799+0000 7f3c6fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3c701611b0 0x7f3c7015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:50.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.799+0000 7f3c6f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3c7015f4d0 0x7f3c700ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38676/0 (socket says 172.21.15.89:38676) 2024-04-02T08:52:50.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.800+0000 7f3c6f400700 1 -- 172.21.15.89:0/1730455444 learned_addr learned my addr 172.21.15.89:0/1730455444 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:50.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.800+0000 7f3c75000700 1 --2- 172.21.15.89:0/1730455444 >> v2:172.21.15.113:3300/0 conn(0x7f3c700cee10 0x7f3c70161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:50.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.800+0000 7f3c6fe00700 1 -- 172.21.15.89:0/1730455444 >> v2:172.21.15.113:3300/0 conn(0x7f3c700cee10 msgr2=0x7f3c70161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:50.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.800+0000 7f3c6fe00700 1 --2- 172.21.15.89:0/1730455444 >> v2:172.21.15.113:3300/0 conn(0x7f3c700cee10 0x7f3c70161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:50.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.800+0000 7f3c6fe00700 1 -- 172.21.15.89:0/1730455444 >> v1:172.21.15.89:6789/0 conn(0x7f3c7015f4d0 legacy=0x7f3c700ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:50.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.800+0000 7f3c6fe00700 1 -- 172.21.15.89:0/1730455444 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c700d2a40 con 0x7f3c701611b0 2024-04-02T08:52:50.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.800+0000 7f3c6ea00700 1 -- 172.21.15.89:0/1730455444 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645857638 0 0) 0x7f3c700ce270 con 0x7f3c7015f4d0 2024-04-02T08:52:50.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.800+0000 7f3c6fe00700 1 --2- 172.21.15.89:0/1730455444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c701611b0 0x7f3c7015ef90 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f3c700d2f60 tx=0x7f3c60009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7123c48f01edbaa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:50.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.800+0000 7f3c6ea00700 1 -- 172.21.15.89:0/1730455444 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c60011830 con 0x7f3c701611b0 2024-04-02T08:52:50.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.801+0000 7f3c6ea00700 1 -- 172.21.15.89:0/1730455444 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c60011990 con 0x7f3c701611b0 2024-04-02T08:52:50.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.801+0000 7f3c6ea00700 1 -- 172.21.15.89:0/1730455444 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c60011d20 con 0x7f3c701611b0 2024-04-02T08:52:50.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.801+0000 7f3c77000700 1 -- 172.21.15.89:0/1730455444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c701611b0 msgr2=0x7f3c7015ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:50.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.801+0000 7f3c77000700 1 --2- 172.21.15.89:0/1730455444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c701611b0 0x7f3c7015ef90 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f3c700d2f60 tx=0x7f3c60009b20 comp rx=0 tx=0).stop 2024-04-02T08:52:50.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.801+0000 7f3c77000700 1 -- 172.21.15.89:0/1730455444 shutdown_connections 2024-04-02T08:52:50.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.801+0000 7f3c77000700 1 --2- 172.21.15.89:0/1730455444 >> v2:172.21.15.113:3300/0 conn(0x7f3c700cee10 0x7f3c70161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:50.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.801+0000 7f3c77000700 1 --2- 172.21.15.89:0/1730455444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c701611b0 0x7f3c7015ef90 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:50.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.801+0000 7f3c77000700 1 -- 172.21.15.89:0/1730455444 >> 172.21.15.89:0/1730455444 conn(0x7f3c7008aff0 msgr2=0x7f3c700d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:50.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.801+0000 7f3c77000700 1 -- 172.21.15.89:0/1730455444 shutdown_connections 2024-04-02T08:52:50.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.801+0000 7f3c77000700 1 -- 172.21.15.89:0/1730455444 wait complete. 2024-04-02T08:52:50.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.803+0000 7f3c77000700 1 Processor -- start 2024-04-02T08:52:50.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.803+0000 7f3c77000700 1 -- start start 2024-04-02T08:52:50.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.804+0000 7f3c77000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c700cee10 0x7f3c700b7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:50.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.804+0000 7f3c77000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c7015f4d0 0x7f3c701603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:50.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.804+0000 7f3c77000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c701611b0 0x7f3c700b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:50.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.804+0000 7f3c77000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c700b78b0 con 0x7f3c700cee10 2024-04-02T08:52:50.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.804+0000 7f3c77000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c700b86f0 con 0x7f3c701611b0 2024-04-02T08:52:50.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.804+0000 7f3c77000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c700b7ae0 con 0x7f3c7015f4d0 2024-04-02T08:52:50.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.804+0000 7f3c6fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c700cee10 0x7f3c700b7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:50.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.804+0000 7f3c6fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c700cee10 0x7f3c700b7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45992/0 (socket says 172.21.15.89:45992) 2024-04-02T08:52:50.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.804+0000 7f3c6fe00700 1 -- 172.21.15.89:0/2789998366 learned_addr learned my addr 172.21.15.89:0/2789998366 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:50.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.805+0000 7f3c6f400700 1 --2- 172.21.15.89:0/2789998366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c7015f4d0 0x7f3c701603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:50.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.805+0000 7f3c75000700 1 --2- 172.21.15.89:0/2789998366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c701611b0 0x7f3c700b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:50.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.805+0000 7f3c6fe00700 1 -- 172.21.15.89:0/2789998366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c7015f4d0 msgr2=0x7f3c701603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:50.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.805+0000 7f3c6fe00700 1 --2- 172.21.15.89:0/2789998366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c7015f4d0 0x7f3c701603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:50.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.805+0000 7f3c6fe00700 1 -- 172.21.15.89:0/2789998366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c701611b0 msgr2=0x7f3c700b6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:50.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.805+0000 7f3c6fe00700 1 --2- 172.21.15.89:0/2789998366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c701611b0 0x7f3c700b6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:50.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.805+0000 7f3c6fe00700 1 -- 172.21.15.89:0/2789998366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c600094a0 con 0x7f3c700cee10 2024-04-02T08:52:50.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.805+0000 7f3c6fe00700 1 --2- 172.21.15.89:0/2789998366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c700cee10 0x7f3c700b7370 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f3c60000c00 tx=0x7f3c60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:50.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.805+0000 7f3c67e00700 1 -- 172.21.15.89:0/2789998366 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c600036a0 con 0x7f3c700cee10 2024-04-02T08:52:50.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.805+0000 7f3c67e00700 1 -- 172.21.15.89:0/2789998366 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6001e070 con 0x7f3c700cee10 2024-04-02T08:52:50.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.805+0000 7f3c77000700 1 -- 172.21.15.89:0/2789998366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c700b7dc0 con 0x7f3c700cee10 2024-04-02T08:52:50.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.805+0000 7f3c67e00700 1 -- 172.21.15.89:0/2789998366 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c60003a60 con 0x7f3c700cee10 2024-04-02T08:52:50.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.806+0000 7f3c77000700 1 -- 172.21.15.89:0/2789998366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c700b8380 con 0x7f3c700cee10 2024-04-02T08:52:50.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.806+0000 7f3c77000700 1 -- 172.21.15.89:0/2789998366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c70067720 con 0x7f3c700cee10 2024-04-02T08:52:50.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.807+0000 7f3c67e00700 1 -- 172.21.15.89:0/2789998366 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3c60003bc0 con 0x7f3c700cee10 2024-04-02T08:52:50.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.808+0000 7f3c67e00700 1 --2- 172.21.15.89:0/2789998366 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3c5006ade0 0x7f3c5006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:50.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.808+0000 7f3c67e00700 1 -- 172.21.15.89:0/2789998366 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f3c6008a770 con 0x7f3c700cee10 2024-04-02T08:52:50.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.808+0000 7f3c6f400700 1 --2- 172.21.15.89:0/2789998366 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3c5006ade0 0x7f3c5006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:50.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.809+0000 7f3c6f400700 1 --2- 172.21.15.89:0/2789998366 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3c5006ade0 0x7f3c5006d2a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3c70072340 tx=0x7f3c5c007be0 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:50.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:50.812+0000 7f3c67e00700 1 -- 172.21.15.89:0/2789998366 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3c600555f0 con 0x7f3c700cee10 2024-04-02T08:52:51.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.040+0000 7f3c77000700 1 -- 172.21.15.89:0/2789998366 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c700cdf00 con 0x7f3c5006ade0 2024-04-02T08:52:51.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.042+0000 7f3c67e00700 1 -- 172.21.15.89:0/2789998366 <== mgr.9318 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3c5c005aa0 con 0x7f3c5006ade0 2024-04-02T08:52:51.040 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T08:52:51.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:52:51.040 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T08:52:51.041 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:52:51.041 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T08:52:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.044+0000 7f3c77000700 1 -- 172.21.15.89:0/2789998366 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3c5006ade0 msgr2=0x7f3c5006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.044+0000 7f3c77000700 1 --2- 172.21.15.89:0/2789998366 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3c5006ade0 0x7f3c5006d2a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3c70072340 tx=0x7f3c5c007be0 comp rx=0 tx=0).stop 2024-04-02T08:52:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.044+0000 7f3c77000700 1 -- 172.21.15.89:0/2789998366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c700cee10 msgr2=0x7f3c700b7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.044+0000 7f3c77000700 1 --2- 172.21.15.89:0/2789998366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c700cee10 0x7f3c700b7370 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f3c60000c00 tx=0x7f3c60004750 comp rx=0 tx=0).stop 2024-04-02T08:52:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.044+0000 7f3c77000700 1 -- 172.21.15.89:0/2789998366 shutdown_connections 2024-04-02T08:52:51.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.044+0000 7f3c77000700 1 --2- 172.21.15.89:0/2789998366 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f3c5006ade0 0x7f3c5006d2a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.044+0000 7f3c77000700 1 --2- 172.21.15.89:0/2789998366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c701611b0 0x7f3c700b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.044+0000 7f3c77000700 1 --2- 172.21.15.89:0/2789998366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c700cee10 0x7f3c700b7370 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.044+0000 7f3c77000700 1 --2- 172.21.15.89:0/2789998366 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c7015f4d0 0x7f3c701603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.044+0000 7f3c77000700 1 -- 172.21.15.89:0/2789998366 >> 172.21.15.89:0/2789998366 conn(0x7f3c7008aff0 msgr2=0x7f3c70161a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.044+0000 7f3c77000700 1 -- 172.21.15.89:0/2789998366 shutdown_connections 2024-04-02T08:52:51.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.044+0000 7f3c77000700 1 -- 172.21.15.89:0/2789998366 wait complete. 2024-04-02T08:52:51.062 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T08:52:51.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.256+0000 7fd0cf000700 1 Processor -- start 2024-04-02T08:52:51.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.256+0000 7fd0cf000700 1 -- start start 2024-04-02T08:52:51.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.257+0000 7fd0cf000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0c815d800 0x7fd0c80725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:51.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.257+0000 7fd0cf000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0c80ce6a0 0x7fd0c815dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:51.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.257+0000 7fd0cf000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c80709c0 con 0x7fd0c80ce230 2024-04-02T08:52:51.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.257+0000 7fd0cf000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c80b40f0 con 0x7fd0c815d800 2024-04-02T08:52:51.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.257+0000 7fd0cf000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c8070c80 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.257+0000 7fd0c7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0c815d800 0x7fd0c80725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:51.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.257+0000 7fd0c7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0c815d800 0x7fd0c80725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45998/0 (socket says 172.21.15.89:45998) 2024-04-02T08:52:51.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.257+0000 7fd0c7e00700 1 -- 172.21.15.89:0/547871736 learned_addr learned my addr 172.21.15.89:0/547871736 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:51.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.258+0000 7fd0c7e00700 1 -- 172.21.15.89:0/547871736 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0c80ce6a0 msgr2=0x7fd0c815dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:51.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.258+0000 7fd0c7e00700 1 --2- 172.21.15.89:0/547871736 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0c80ce6a0 0x7fd0c815dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:51.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.258+0000 7fd0c7e00700 1 -- 172.21.15.89:0/547871736 >> v1:172.21.15.113:6789/0 conn(0x7fd0c80ce230 legacy=0x7fd0c80b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:51.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.258+0000 7fd0c7e00700 1 -- 172.21.15.89:0/547871736 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c8070f60 con 0x7fd0c815d800 2024-04-02T08:52:51.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.258+0000 7fd0c7e00700 1 --2- 172.21.15.89:0/547871736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c815d800 0x7fd0c80725c0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fd0b80098f0 tx=0x7fd0b8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f9ca683aca1a50 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:51.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.258+0000 7fd0c6a00700 1 -- 172.21.15.89:0/547871736 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b8004230 con 0x7fd0c815d800 2024-04-02T08:52:51.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.258+0000 7fd0c6a00700 1 -- 172.21.15.89:0/547871736 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b8004390 con 0x7fd0c815d800 2024-04-02T08:52:51.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.258+0000 7fd0c6a00700 1 -- 172.21.15.89:0/547871736 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b80036a0 con 0x7fd0c815d800 2024-04-02T08:52:51.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.259+0000 7fd0cf000700 1 -- 172.21.15.89:0/547871736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c815d800 msgr2=0x7fd0c80725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:51.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.259+0000 7fd0cf000700 1 --2- 172.21.15.89:0/547871736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c815d800 0x7fd0c80725c0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fd0b80098f0 tx=0x7fd0b8009c00 comp rx=0 tx=0).stop 2024-04-02T08:52:51.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.259+0000 7fd0cf000700 1 -- 172.21.15.89:0/547871736 shutdown_connections 2024-04-02T08:52:51.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.259+0000 7fd0cf000700 1 --2- 172.21.15.89:0/547871736 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0c80ce6a0 0x7fd0c815dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:51.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.259+0000 7fd0cf000700 1 --2- 172.21.15.89:0/547871736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c815d800 0x7fd0c80725c0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:51.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.259+0000 7fd0cf000700 1 -- 172.21.15.89:0/547871736 >> 172.21.15.89:0/547871736 conn(0x7fd0c80b0d70 msgr2=0x7fd0c80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:51.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.259+0000 7fd0cf000700 1 -- 172.21.15.89:0/547871736 shutdown_connections 2024-04-02T08:52:51.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.259+0000 7fd0cf000700 1 -- 172.21.15.89:0/547871736 wait complete. 2024-04-02T08:52:51.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.260+0000 7fd0cf000700 1 Processor -- start 2024-04-02T08:52:51.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.261+0000 7fd0cf000700 1 -- start start 2024-04-02T08:52:51.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.261+0000 7fd0cf000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0c80ce230 0x7fd0c8155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:51.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0cf000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c80ce6a0 0x7fd0c80b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:51.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0cf000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0c815d800 0x7fd0c80b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:51.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0cf000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c80c90b0 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0cf000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c80c9390 con 0x7fd0c80ce230 2024-04-02T08:52:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0cf000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c80c9670 con 0x7fd0c815d800 2024-04-02T08:52:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0c7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c80ce6a0 0x7fd0c80b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0c7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c80ce6a0 0x7fd0c80b0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46000/0 (socket says 172.21.15.89:46000) 2024-04-02T08:52:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0c7400700 1 -- 172.21.15.89:0/1817258359 learned_addr learned my addr 172.21.15.89:0/1817258359 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0c7e00700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0c80ce230 0x7fd0c8155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0cd000700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0c815d800 0x7fd0c80b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0c7400700 1 -- 172.21.15.89:0/1817258359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0c815d800 msgr2=0x7fd0c80b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0c7400700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0c815d800 0x7fd0c80b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0c7400700 1 -- 172.21.15.89:0/1817258359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0c80ce230 msgr2=0x7fd0c8155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0c7400700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0c80ce230 0x7fd0c8155050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0c7400700 1 -- 172.21.15.89:0/1817258359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b8009550 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0c7e00700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0c80ce230 0x7fd0c8155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:51.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0cd000700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0c815d800 0x7fd0c80b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:51.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.262+0000 7fd0c7400700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c80ce6a0 0x7fd0c80b0030 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400b4a0 tx=0x7fd0b400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:51.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.263+0000 7fd0bfe00700 1 -- 172.21.15.89:0/1817258359 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4011830 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.263+0000 7fd0bfe00700 1 -- 172.21.15.89:0/1817258359 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b4011990 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.263+0000 7fd0bfe00700 1 -- 172.21.15.89:0/1817258359 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4011d40 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.263+0000 7fd0cf000700 1 -- 172.21.15.89:0/1817258359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c80c9950 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.263+0000 7fd0cf000700 1 -- 172.21.15.89:0/1817258359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c8155590 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.264+0000 7fd0cf000700 1 -- 172.21.15.89:0/1817258359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c8067720 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.265+0000 7fd0bfe00700 1 -- 172.21.15.89:0/1817258359 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 103) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd0b401a400 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.266+0000 7fd0bfe00700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd0a806adb0 0x7fd0a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:51.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.266+0000 7fd0bfe00700 1 -- 172.21.15.89:0/1817258359 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fd0b4089340 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.266+0000 7fd0c7e00700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd0a806adb0 0x7fd0a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:51.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.266+0000 7fd0c7e00700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd0a806adb0 0x7fd0a806d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8009130 tx=0x7fd0b8008030 comp rx=0 tx=0).ready entity=mgr.9318 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:51.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.270+0000 7fd0bfe00700 1 -- 172.21.15.89:0/1817258359 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd0b4054300 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.534+0000 7fd0cf000700 1 -- 172.21.15.89:0/1817258359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fd0c8158fb0 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.921+0000 7fd0bfe00700 1 -- 172.21.15.89:0/1817258359 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 104) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fd0b40546d0 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.923+0000 7fd0bfe00700 1 -- 172.21.15.89:0/1817258359 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v104) v1 ==== 92+0+0 (secure 0 0 0) 0x7fd0b4071650 con 0x7fd0c80ce6a0 2024-04-02T08:52:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.926+0000 7fd0cf000700 1 -- 172.21.15.89:0/1817258359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd0a806adb0 msgr2=0x7fd0a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.926+0000 7fd0cf000700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd0a806adb0 0x7fd0a806d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8009130 tx=0x7fd0b8008030 comp rx=0 tx=0).stop 2024-04-02T08:52:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.926+0000 7fd0cf000700 1 -- 172.21.15.89:0/1817258359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c80ce6a0 msgr2=0x7fd0c80b0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.926+0000 7fd0cf000700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c80ce6a0 0x7fd0c80b0030 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400b4a0 tx=0x7fd0b400b860 comp rx=0 tx=0).stop 2024-04-02T08:52:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.926+0000 7fd0cf000700 1 -- 172.21.15.89:0/1817258359 shutdown_connections 2024-04-02T08:52:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.926+0000 7fd0cf000700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd0a806adb0 0x7fd0a806d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.926+0000 7fd0cf000700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0c80ce230 0x7fd0c8155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.926+0000 7fd0cf000700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0c815d800 0x7fd0c80b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.926+0000 7fd0cf000700 1 --2- 172.21.15.89:0/1817258359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c80ce6a0 0x7fd0c80b0030 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.926+0000 7fd0cf000700 1 -- 172.21.15.89:0/1817258359 >> 172.21.15.89:0/1817258359 conn(0x7fd0c80b0d70 msgr2=0x7fd0c8156a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.926+0000 7fd0cf000700 1 -- 172.21.15.89:0/1817258359 shutdown_connections 2024-04-02T08:52:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:51.926+0000 7fd0cf000700 1 -- 172.21.15.89:0/1817258359 wait complete. 2024-04-02T08:52:51.944 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T08:52:52.138 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:52.138+0000 7fb4a483a240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T08:52:52.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.145+0000 7f547b800700 1 Processor -- start 2024-04-02T08:52:52.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.145+0000 7f547b800700 1 -- start start 2024-04-02T08:52:52.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.146+0000 7f547b800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54740ced80 0x7f54740d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:52.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.146+0000 7f547b800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f547415fcb0 0x7f5474160140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:52.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.146+0000 7f547b800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54740cf1e0 con 0x7f547415dad0 2024-04-02T08:52:52.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.147+0000 7f547b800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54740cfcc0 con 0x7f547415fcb0 2024-04-02T08:52:52.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.147+0000 7f547b800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54740cd9f0 con 0x7f54740ced80 2024-04-02T08:52:52.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.147+0000 7f5479800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f547415fcb0 0x7f5474160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:52.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.147+0000 7f5479800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f547415fcb0 0x7f5474160140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46014/0 (socket says 172.21.15.89:46014) 2024-04-02T08:52:52.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.147+0000 7f5479800700 1 -- 172.21.15.89:0/87399973 learned_addr learned my addr 172.21.15.89:0/87399973 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:52.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.147+0000 7f5473e00700 1 --2- 172.21.15.89:0/87399973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54740ced80 0x7f54740d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:52.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.147+0000 7f5479800700 1 -- 172.21.15.89:0/87399973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54740ced80 msgr2=0x7f54740d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:52.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.147+0000 7f5479800700 1 --2- 172.21.15.89:0/87399973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54740ced80 0x7f54740d1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.147+0000 7f5479800700 1 -- 172.21.15.89:0/87399973 >> v1:172.21.15.113:6789/0 conn(0x7f547415dad0 legacy=0x7f54740d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:52.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.147+0000 7f5479800700 1 -- 172.21.15.89:0/87399973 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547415f820 con 0x7f547415fcb0 2024-04-02T08:52:52.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.147+0000 7f5473400700 1 -- 172.21.15.89:0/87399973 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481031037 0 0) 0x7f5468001110 con 0x7f547415dad0 2024-04-02T08:52:52.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.148+0000 7f5479800700 1 --2- 172.21.15.89:0/87399973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f547415fcb0 0x7f5474160140 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f546c009e50 tx=0x7f546c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df45f12354a533a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:52.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.148+0000 7f5473400700 1 -- 172.21.15.89:0/87399973 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c00e070 con 0x7f547415fcb0 2024-04-02T08:52:52.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.148+0000 7f5473400700 1 -- 172.21.15.89:0/87399973 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c00b910 con 0x7f547415fcb0 2024-04-02T08:52:52.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.148+0000 7f5473400700 1 -- 172.21.15.89:0/87399973 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c00bcc0 con 0x7f547415fcb0 2024-04-02T08:52:52.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.148+0000 7f547b800700 1 -- 172.21.15.89:0/87399973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f547415fcb0 msgr2=0x7f5474160140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:52.148 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:52.148+0000 7f80395b7240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T08:52:52.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.148+0000 7f547b800700 1 --2- 172.21.15.89:0/87399973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f547415fcb0 0x7f5474160140 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f546c009e50 tx=0x7f546c00b330 comp rx=0 tx=0).stop 2024-04-02T08:52:52.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.149+0000 7f547b800700 1 -- 172.21.15.89:0/87399973 shutdown_connections 2024-04-02T08:52:52.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.149+0000 7f547b800700 1 --2- 172.21.15.89:0/87399973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f547415fcb0 0x7f5474160140 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.149+0000 7f547b800700 1 --2- 172.21.15.89:0/87399973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54740ced80 0x7f54740d1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.149+0000 7f547b800700 1 -- 172.21.15.89:0/87399973 >> 172.21.15.89:0/87399973 conn(0x7f547408aff0 msgr2=0x7f54740d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:52.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.149+0000 7f547b800700 1 -- 172.21.15.89:0/87399973 shutdown_connections 2024-04-02T08:52:52.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.149+0000 7f547b800700 1 -- 172.21.15.89:0/87399973 wait complete. 2024-04-02T08:52:52.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.150+0000 7f547b800700 1 Processor -- start 2024-04-02T08:52:52.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.150+0000 7f547b800700 1 -- start start 2024-04-02T08:52:52.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.151+0000 7f547b800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54740ced80 0x7f54740b7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:52.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.151+0000 7f547b800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f547415dad0 0x7f54741548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:52.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.151+0000 7f547b800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f547415fcb0 0x7f54740c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:52.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.151+0000 7f547b800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5474150c30 con 0x7f54740ced80 2024-04-02T08:52:52.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.151+0000 7f547b800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54740b8040 con 0x7f547415dad0 2024-04-02T08:52:52.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.151+0000 7f547b800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54740b6a60 con 0x7f547415fcb0 2024-04-02T08:52:52.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.152+0000 7f5478e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54740ced80 0x7f54740b7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:52.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.152+0000 7f5478e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54740ced80 0x7f54740b7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46020/0 (socket says 172.21.15.89:46020) 2024-04-02T08:52:52.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.152+0000 7f5478e00700 1 -- 172.21.15.89:0/2009256608 learned_addr learned my addr 172.21.15.89:0/2009256608 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:52.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.152+0000 7f5479800700 1 --2- 172.21.15.89:0/2009256608 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f547415fcb0 0x7f54740c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:52.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.152+0000 7f5473e00700 1 --2- 172.21.15.89:0/2009256608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f547415dad0 0x7f54741548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:52.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.152+0000 7f5478e00700 1 -- 172.21.15.89:0/2009256608 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f547415fcb0 msgr2=0x7f54740c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:52.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.152+0000 7f5478e00700 1 --2- 172.21.15.89:0/2009256608 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f547415fcb0 0x7f54740c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.152+0000 7f5478e00700 1 -- 172.21.15.89:0/2009256608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f547415dad0 msgr2=0x7f54741548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:52.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.152+0000 7f5478e00700 1 --2- 172.21.15.89:0/2009256608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f547415dad0 0x7f54741548c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.152+0000 7f5478e00700 1 -- 172.21.15.89:0/2009256608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c009b30 con 0x7f54740ced80 2024-04-02T08:52:52.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.152+0000 7f5478e00700 1 --2- 172.21.15.89:0/2009256608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54740ced80 0x7f54740b7b00 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f546800c750 tx=0x7f546800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:52.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.153+0000 7f5470c00700 1 -- 172.21.15.89:0/2009256608 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54680054e0 con 0x7f54740ced80 2024-04-02T08:52:52.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.153+0000 7f5470c00700 1 -- 172.21.15.89:0/2009256608 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5468003680 con 0x7f54740ced80 2024-04-02T08:52:52.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.153+0000 7f5470c00700 1 -- 172.21.15.89:0/2009256608 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5468003a30 con 0x7f54740ced80 2024-04-02T08:52:52.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.153+0000 7f547b800700 1 -- 172.21.15.89:0/2009256608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54740b6d40 con 0x7f54740ced80 2024-04-02T08:52:52.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.153+0000 7f547b800700 1 -- 172.21.15.89:0/2009256608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54740b7300 con 0x7f54740ced80 2024-04-02T08:52:52.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.154+0000 7f547b800700 1 -- 172.21.15.89:0/2009256608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5474067720 con 0x7f54740ced80 2024-04-02T08:52:52.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.155+0000 7f5470c00700 1 -- 172.21.15.89:0/2009256608 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 104) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f5468003b90 con 0x7f54740ced80 2024-04-02T08:52:52.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.156+0000 7f5470c00700 1 --2- 172.21.15.89:0/2009256608 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f545406ae00 0x7f545406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:52.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.156+0000 7f5470c00700 1 -- 172.21.15.89:0/2009256608 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f5468088930 con 0x7f54740ced80 2024-04-02T08:52:52.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.156+0000 7f5473e00700 1 -- 172.21.15.89:0/2009256608 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f545406ae00 msgr2=0x7f545406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:52.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.156+0000 7f5473e00700 1 --2- 172.21.15.89:0/2009256608 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f545406ae00 0x7f545406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:52.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.160+0000 7f5470c00700 1 -- 172.21.15.89:0/2009256608 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f54680572e0 con 0x7f54740ced80 2024-04-02T08:52:52.326 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:52.327+0000 7fb4a483a240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T08:52:52.336 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:52.337+0000 7f80395b7240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T08:52:52.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.357+0000 7f5473e00700 1 -- 172.21.15.89:0/2009256608 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f545406ae00 msgr2=0x7f545406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:52.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.357+0000 7f5473e00700 1 --2- 172.21.15.89:0/2009256608 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f545406ae00 0x7f545406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:52.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.406+0000 7f547b800700 1 -- 172.21.15.89:0/2009256608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f54740cf1e0 con 0x7f54740ced80 2024-04-02T08:52:52.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.410+0000 7f5470c00700 1 -- 172.21.15.89:0/2009256608 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7f54740cf1e0 con 0x7f54740ced80 2024-04-02T08:52:52.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.414+0000 7f547b800700 1 -- 172.21.15.89:0/2009256608 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f545406ae00 msgr2=0x7f545406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.414+0000 7f547b800700 1 --2- 172.21.15.89:0/2009256608 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f545406ae00 0x7f545406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.414+0000 7f547b800700 1 -- 172.21.15.89:0/2009256608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54740ced80 msgr2=0x7f54740b7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.414+0000 7f547b800700 1 --2- 172.21.15.89:0/2009256608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54740ced80 0x7f54740b7b00 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f546800c750 tx=0x7f546800ca60 comp rx=0 tx=0).stop 2024-04-02T08:52:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.414+0000 7f547b800700 1 -- 172.21.15.89:0/2009256608 shutdown_connections 2024-04-02T08:52:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.414+0000 7f547b800700 1 --2- 172.21.15.89:0/2009256608 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f545406ae00 0x7f545406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.414+0000 7f547b800700 1 --2- 172.21.15.89:0/2009256608 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f547415fcb0 0x7f54740c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.414+0000 7f547b800700 1 --2- 172.21.15.89:0/2009256608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54740ced80 0x7f54740b7b00 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.414+0000 7f547b800700 1 --2- 172.21.15.89:0/2009256608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f547415dad0 0x7f54741548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.414+0000 7f547b800700 1 -- 172.21.15.89:0/2009256608 >> 172.21.15.89:0/2009256608 conn(0x7f547408aff0 msgr2=0x7f54740d0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.414+0000 7f547b800700 1 -- 172.21.15.89:0/2009256608 shutdown_connections 2024-04-02T08:52:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.414+0000 7f547b800700 1 -- 172.21.15.89:0/2009256608 wait complete. 2024-04-02T08:52:52.439 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T08:52:52.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.637+0000 7fb5a7e00700 1 Processor -- start 2024-04-02T08:52:52.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.637+0000 7fb5a7e00700 1 -- start start 2024-04-02T08:52:52.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.639+0000 7fb5a7e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5a00ce230 0x7fb5a00ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:52.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.639+0000 7fb5a7e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb5a00b2070 0x7fb5a00b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:52.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.639+0000 7fb5a7e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a00709c0 con 0x7fb5a015d800 2024-04-02T08:52:52.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.639+0000 7fb5a7e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a00b40f0 con 0x7fb5a00b2070 2024-04-02T08:52:52.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.639+0000 7fb5a7e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a0070c80 con 0x7fb5a00ce230 2024-04-02T08:52:52.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.639+0000 7fb5a5400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb5a015d800 0x7fb5a00725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38688/0 (socket says 172.21.15.89:38688) 2024-04-02T08:52:52.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.639+0000 7fb5a5400700 1 -- 172.21.15.89:0/179318745 learned_addr learned my addr 172.21.15.89:0/179318745 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:52.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.639+0000 7fb5a4a00700 1 --2- 172.21.15.89:0/179318745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5a00ce230 0x7fb5a00ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:52.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.639+0000 7fb5a5e00700 1 --2- 172.21.15.89:0/179318745 >> v2:172.21.15.113:3300/0 conn(0x7fb5a00b2070 0x7fb5a00b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:52.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.640+0000 7fb59fe00700 1 -- 172.21.15.89:0/179318745 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769437425 0 0) 0x7fb5a00709c0 con 0x7fb5a015d800 2024-04-02T08:52:52.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.640+0000 7fb59fe00700 1 -- 172.21.15.89:0/179318745 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5880035b0 con 0x7fb5a015d800 2024-04-02T08:52:52.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.640+0000 7fb5a4a00700 1 -- 172.21.15.89:0/179318745 >> v2:172.21.15.113:3300/0 conn(0x7fb5a00b2070 msgr2=0x7fb5a00b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:52.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.640+0000 7fb5a4a00700 1 --2- 172.21.15.89:0/179318745 >> v2:172.21.15.113:3300/0 conn(0x7fb5a00b2070 0x7fb5a00b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.640+0000 7fb5a4a00700 1 -- 172.21.15.89:0/179318745 >> v1:172.21.15.89:6789/0 conn(0x7fb5a015d800 legacy=0x7fb5a00725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:52.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.640+0000 7fb5a4a00700 1 -- 172.21.15.89:0/179318745 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a0070f60 con 0x7fb5a00ce230 2024-04-02T08:52:52.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.640+0000 7fb5a5e00700 1 --2- 172.21.15.89:0/179318745 >> v2:172.21.15.113:3300/0 conn(0x7fb5a00b2070 0x7fb5a00b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:52.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.640+0000 7fb5a4a00700 1 --2- 172.21.15.89:0/179318745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5a00ce230 0x7fb5a00ce670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb58c009840 tx=0x7fb58c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c48aef14e883f384 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:52.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.640+0000 7fb59fe00700 1 -- 172.21.15.89:0/179318745 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c011830 con 0x7fb5a00ce230 2024-04-02T08:52:52.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.640+0000 7fb59fe00700 1 -- 172.21.15.89:0/179318745 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58c011990 con 0x7fb5a00ce230 2024-04-02T08:52:52.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.640+0000 7fb59fe00700 1 -- 172.21.15.89:0/179318745 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c011d20 con 0x7fb5a00ce230 2024-04-02T08:52:52.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.641+0000 7fb5a7e00700 1 -- 172.21.15.89:0/179318745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5a00ce230 msgr2=0x7fb5a00ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:52.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.641+0000 7fb5a7e00700 1 --2- 172.21.15.89:0/179318745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5a00ce230 0x7fb5a00ce670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb58c009840 tx=0x7fb58c009b50 comp rx=0 tx=0).stop 2024-04-02T08:52:52.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.641+0000 7fb5a7e00700 1 -- 172.21.15.89:0/179318745 shutdown_connections 2024-04-02T08:52:52.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.641+0000 7fb5a7e00700 1 --2- 172.21.15.89:0/179318745 >> v2:172.21.15.113:3300/0 conn(0x7fb5a00b2070 0x7fb5a00b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.641+0000 7fb5a7e00700 1 --2- 172.21.15.89:0/179318745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5a00ce230 0x7fb5a00ce670 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.641+0000 7fb5a7e00700 1 -- 172.21.15.89:0/179318745 >> 172.21.15.89:0/179318745 conn(0x7fb5a00b0d70 msgr2=0x7fb5a00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:52.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.641+0000 7fb5a7e00700 1 -- 172.21.15.89:0/179318745 shutdown_connections 2024-04-02T08:52:52.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.641+0000 7fb5a7e00700 1 -- 172.21.15.89:0/179318745 wait complete. 2024-04-02T08:52:52.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.643+0000 7fb5a7e00700 1 Processor -- start 2024-04-02T08:52:52.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.643+0000 7fb5a7e00700 1 -- start start 2024-04-02T08:52:52.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.644+0000 7fb5a7e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5a00b2070 0x7fb5a00984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:52.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.644+0000 7fb5a7e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5a00ce230 0x7fb5a015cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:52.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.644+0000 7fb5a7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5a015d800 0x7fb5a014f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:52.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.644+0000 7fb5a7e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a014caa0 con 0x7fb5a015d800 2024-04-02T08:52:52.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.644+0000 7fb5a7e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a0098a10 con 0x7fb5a00ce230 2024-04-02T08:52:52.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.644+0000 7fb5a7e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a014be20 con 0x7fb5a00b2070 2024-04-02T08:52:52.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.644+0000 7fb5a5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5a015d800 0x7fb5a014f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:52.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.644+0000 7fb5a5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5a015d800 0x7fb5a014f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46026/0 (socket says 172.21.15.89:46026) 2024-04-02T08:52:52.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.644+0000 7fb5a5e00700 1 -- 172.21.15.89:0/624673161 learned_addr learned my addr 172.21.15.89:0/624673161 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:52.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.644+0000 7fb5a5400700 1 --2- 172.21.15.89:0/624673161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5a00b2070 0x7fb5a00984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:52.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.644+0000 7fb5a4a00700 1 --2- 172.21.15.89:0/624673161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5a00ce230 0x7fb5a015cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:52.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.645+0000 7fb5a5e00700 1 -- 172.21.15.89:0/624673161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5a00b2070 msgr2=0x7fb5a00984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:52.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.645+0000 7fb5a5e00700 1 --2- 172.21.15.89:0/624673161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5a00b2070 0x7fb5a00984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.645+0000 7fb5a5e00700 1 -- 172.21.15.89:0/624673161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5a00ce230 msgr2=0x7fb5a015cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:52.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.645+0000 7fb5a5e00700 1 --2- 172.21.15.89:0/624673161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5a00ce230 0x7fb5a015cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.645+0000 7fb5a5e00700 1 -- 172.21.15.89:0/624673161 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb598009a70 con 0x7fb5a015d800 2024-04-02T08:52:52.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.645+0000 7fb5a5e00700 1 --2- 172.21.15.89:0/624673161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5a015d800 0x7fb5a014f260 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fb59800efc0 tx=0x7fb59800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:52.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.645+0000 7fb59d600700 1 -- 172.21.15.89:0/624673161 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59800ca90 con 0x7fb5a015d800 2024-04-02T08:52:52.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.645+0000 7fb59d600700 1 -- 172.21.15.89:0/624673161 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59800f040 con 0x7fb5a015d800 2024-04-02T08:52:52.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.645+0000 7fb5a7e00700 1 -- 172.21.15.89:0/624673161 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb58c0094a0 con 0x7fb5a015d800 2024-04-02T08:52:52.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.645+0000 7fb59d600700 1 -- 172.21.15.89:0/624673161 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59800ce30 con 0x7fb5a015d800 2024-04-02T08:52:52.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.645+0000 7fb5a7e00700 1 -- 172.21.15.89:0/624673161 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a014c3e0 con 0x7fb5a015d800 2024-04-02T08:52:52.647 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:52.646+0000 7fb4a483a240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T08:52:52.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.646+0000 7fb5a7e00700 1 -- 172.21.15.89:0/624673161 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a014cbe0 con 0x7fb5a015d800 2024-04-02T08:52:52.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.647+0000 7fb59d600700 1 -- 172.21.15.89:0/624673161 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 104) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fb598014480 con 0x7fb5a015d800 2024-04-02T08:52:52.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.648+0000 7fb59d600700 1 --2- 172.21.15.89:0/624673161 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb58006ae00 0x7fb58006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:52.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.649+0000 7fb5a5400700 1 -- 172.21.15.89:0/624673161 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb58006ae00 msgr2=0x7fb58006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:52.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.649+0000 7fb59d600700 1 -- 172.21.15.89:0/624673161 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fb598089f10 con 0x7fb5a015d800 2024-04-02T08:52:52.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.649+0000 7fb5a5400700 1 --2- 172.21.15.89:0/624673161 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb58006ae00 0x7fb58006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:52.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.652+0000 7fb59d600700 1 -- 172.21.15.89:0/624673161 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb5a014cbe0 con 0x7fb5a015d800 2024-04-02T08:52:52.678 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:52.679+0000 7f80395b7240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T08:52:52.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.850+0000 7fb5a5400700 1 -- 172.21.15.89:0/624673161 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb58006ae00 msgr2=0x7fb58006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:52.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.850+0000 7fb5a5400700 1 --2- 172.21.15.89:0/624673161 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb58006ae00 0x7fb58006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:52.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.861+0000 7fb5a7e00700 1 -- 172.21.15.89:0/624673161 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fb5a014cbe0 con 0x7fb5a015d800 2024-04-02T08:52:52.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.865+0000 7fb59d600700 1 -- 172.21.15.89:0/624673161 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb5a014cbe0 con 0x7fb5a015d800 2024-04-02T08:52:52.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.867+0000 7fb5a7e00700 1 -- 172.21.15.89:0/624673161 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb58006ae00 msgr2=0x7fb58006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:52.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.867+0000 7fb5a7e00700 1 --2- 172.21.15.89:0/624673161 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb58006ae00 0x7fb58006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.867+0000 7fb5a7e00700 1 -- 172.21.15.89:0/624673161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5a015d800 msgr2=0x7fb5a014f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:52.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.867+0000 7fb5a7e00700 1 --2- 172.21.15.89:0/624673161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5a015d800 0x7fb5a014f260 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fb59800efc0 tx=0x7fb59800c3a0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.867+0000 7fb5a7e00700 1 -- 172.21.15.89:0/624673161 shutdown_connections 2024-04-02T08:52:52.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.867+0000 7fb5a7e00700 1 --2- 172.21.15.89:0/624673161 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb58006ae00 0x7fb58006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.867+0000 7fb5a7e00700 1 --2- 172.21.15.89:0/624673161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5a015d800 0x7fb5a014f260 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.867+0000 7fb5a7e00700 1 --2- 172.21.15.89:0/624673161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5a00b2070 0x7fb5a00984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.867+0000 7fb5a7e00700 1 --2- 172.21.15.89:0/624673161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5a00ce230 0x7fb5a015cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:52.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.867+0000 7fb5a7e00700 1 -- 172.21.15.89:0/624673161 >> 172.21.15.89:0/624673161 conn(0x7fb5a00b0d70 msgr2=0x7fb5a00cbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:52.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.868+0000 7fb5a7e00700 1 -- 172.21.15.89:0/624673161 shutdown_connections 2024-04-02T08:52:52.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:52.868+0000 7fb5a7e00700 1 -- 172.21.15.89:0/624673161 wait complete. 2024-04-02T08:52:52.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:52:52.886 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-02T08:52:52.935 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2024-04-02T08:52:53.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.139+0000 7fd322e00700 1 Processor -- start 2024-04-02T08:52:53.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.139+0000 7fd322e00700 1 -- start start 2024-04-02T08:52:53.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.140+0000 7fd322e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd31c15d9d0 0x7fd31c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:53.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.140+0000 7fd322e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd31c0d2540 0x7fd31c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:53.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.140+0000 7fd322e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31c161420 con 0x7fd31c0d16a0 2024-04-02T08:52:53.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.140+0000 7fd322e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31c15edc0 con 0x7fd31c15d9d0 2024-04-02T08:52:53.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.140+0000 7fd322e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c161700 con 0x7fd31c0d2540 2024-04-02T08:52:53.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.140+0000 7fd31be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd31c15d9d0 0x7fd31c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:53.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.140+0000 7fd31be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd31c15d9d0 0x7fd31c0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46040/0 (socket says 172.21.15.89:46040) 2024-04-02T08:52:53.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.140+0000 7fd31be00700 1 -- 172.21.15.89:0/1868667633 learned_addr learned my addr 172.21.15.89:0/1868667633 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:53.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.140+0000 7fd320e00700 1 --2- 172.21.15.89:0/1868667633 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd31c0d2540 0x7fd31c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:53.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.141+0000 7fd31be00700 1 -- 172.21.15.89:0/1868667633 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd31c0d2540 msgr2=0x7fd31c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:52:53.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.141+0000 7fd31be00700 1 --2- 172.21.15.89:0/1868667633 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd31c0d2540 0x7fd31c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:53.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.141+0000 7fd31be00700 1 -- 172.21.15.89:0/1868667633 >> v1:172.21.15.113:6789/0 conn(0x7fd31c0d16a0 legacy=0x7fd31c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:53.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.141+0000 7fd31be00700 1 -- 172.21.15.89:0/1868667633 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31c1619e0 con 0x7fd31c15d9d0 2024-04-02T08:52:53.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.141+0000 7fd31be00700 1 --2- 172.21.15.89:0/1868667633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31c15d9d0 0x7fd31c0d3c30 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fd30c0098f0 tx=0x7fd30c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca5d686c4b72d348 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:53.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.141+0000 7fd31aa00700 1 -- 172.21.15.89:0/1868667633 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c011830 con 0x7fd31c15d9d0 2024-04-02T08:52:53.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.141+0000 7fd31aa00700 1 -- 172.21.15.89:0/1868667633 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c011990 con 0x7fd31c15d9d0 2024-04-02T08:52:53.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.141+0000 7fd31aa00700 1 -- 172.21.15.89:0/1868667633 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c011d20 con 0x7fd31c15d9d0 2024-04-02T08:52:53.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.142+0000 7fd322e00700 1 -- 172.21.15.89:0/1868667633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31c15d9d0 msgr2=0x7fd31c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:53.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.142+0000 7fd322e00700 1 --2- 172.21.15.89:0/1868667633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31c15d9d0 0x7fd31c0d3c30 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fd30c0098f0 tx=0x7fd30c009c00 comp rx=0 tx=0).stop 2024-04-02T08:52:53.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.142+0000 7fd322e00700 1 -- 172.21.15.89:0/1868667633 shutdown_connections 2024-04-02T08:52:53.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.142+0000 7fd322e00700 1 --2- 172.21.15.89:0/1868667633 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd31c0d2540 0x7fd31c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:53.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.142+0000 7fd322e00700 1 --2- 172.21.15.89:0/1868667633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31c15d9d0 0x7fd31c0d3c30 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:53.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.142+0000 7fd322e00700 1 -- 172.21.15.89:0/1868667633 >> 172.21.15.89:0/1868667633 conn(0x7fd31c08aff0 msgr2=0x7fd31c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:53.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.142+0000 7fd322e00700 1 -- 172.21.15.89:0/1868667633 shutdown_connections 2024-04-02T08:52:53.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.142+0000 7fd322e00700 1 -- 172.21.15.89:0/1868667633 wait complete. 2024-04-02T08:52:53.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.144+0000 7fd322e00700 1 Processor -- start 2024-04-02T08:52:53.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.144+0000 7fd322e00700 1 -- start start 2024-04-02T08:52:53.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.145+0000 7fd322e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31c0d16a0 0x7fd31c1e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:53.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.145+0000 7fd322e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd31c0d2540 0x7fd31c14f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:53.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.145+0000 7fd322e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd31c15d9d0 0x7fd31c0b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:53.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.145+0000 7fd322e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c0cb5a0 con 0x7fd31c0d16a0 2024-04-02T08:52:53.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.145+0000 7fd322e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c1e82e0 con 0x7fd31c0d2540 2024-04-02T08:52:53.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.145+0000 7fd322e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c1e85c0 con 0x7fd31c15d9d0 2024-04-02T08:52:53.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.145+0000 7fd31be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31c0d16a0 0x7fd31c1e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:53.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.145+0000 7fd320e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd31c15d9d0 0x7fd31c0b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:53.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.145+0000 7fd31b400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd31c0d2540 0x7fd31c14f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:53.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.146+0000 7fd31be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31c0d16a0 0x7fd31c1e7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46044/0 (socket says 172.21.15.89:46044) 2024-04-02T08:52:53.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.146+0000 7fd320e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd31c15d9d0 0x7fd31c0b7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:44138/0 (socket says 172.21.15.89:44138) 2024-04-02T08:52:53.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.146+0000 7fd31be00700 1 -- 172.21.15.89:0/3477115012 learned_addr learned my addr 172.21.15.89:0/3477115012 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:53.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.146+0000 7fd31be00700 1 -- 172.21.15.89:0/3477115012 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd31c15d9d0 msgr2=0x7fd31c0b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:53.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.146+0000 7fd31be00700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd31c15d9d0 0x7fd31c0b7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:53.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.146+0000 7fd31be00700 1 -- 172.21.15.89:0/3477115012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd31c0d2540 msgr2=0x7fd31c14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:53.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.146+0000 7fd31be00700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd31c0d2540 0x7fd31c14f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:53.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.146+0000 7fd31be00700 1 -- 172.21.15.89:0/3477115012 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c009550 con 0x7fd31c0d16a0 2024-04-02T08:52:53.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.146+0000 7fd320e00700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd31c15d9d0 0x7fd31c0b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:52:53.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.146+0000 7fd31be00700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31c0d16a0 0x7fd31c1e7c90 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009c40 tx=0x7fd30c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:53.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.146+0000 7fd313e00700 1 -- 172.21.15.89:0/3477115012 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c004510 con 0x7fd31c0d16a0 2024-04-02T08:52:53.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.147+0000 7fd313e00700 1 -- 172.21.15.89:0/3477115012 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c01e070 con 0x7fd31c0d16a0 2024-04-02T08:52:53.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.147+0000 7fd322e00700 1 -- 172.21.15.89:0/3477115012 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd31c1e88a0 con 0x7fd31c0d16a0 2024-04-02T08:52:53.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.147+0000 7fd313e00700 1 -- 172.21.15.89:0/3477115012 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c0038e0 con 0x7fd31c0d16a0 2024-04-02T08:52:53.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.147+0000 7fd322e00700 1 -- 172.21.15.89:0/3477115012 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31c14fd90 con 0x7fd31c0d16a0 2024-04-02T08:52:53.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.148+0000 7fd322e00700 1 -- 172.21.15.89:0/3477115012 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd31c0d1ca0 con 0x7fd31c0d16a0 2024-04-02T08:52:53.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.149+0000 7fd313e00700 1 -- 172.21.15.89:0/3477115012 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 104) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fd30c003a40 con 0x7fd31c0d16a0 2024-04-02T08:52:53.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.150+0000 7fd313e00700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 0x7fd2fc06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:53.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.150+0000 7fd313e00700 1 -- 172.21.15.89:0/3477115012 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fd30c08a530 con 0x7fd31c0d16a0 2024-04-02T08:52:53.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.150+0000 7fd31b400700 1 -- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 msgr2=0x7fd2fc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:53.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.150+0000 7fd31b400700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 0x7fd2fc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:52:53.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.154+0000 7fd313e00700 1 -- 172.21.15.89:0/3477115012 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd30c055460 con 0x7fd31c0d16a0 2024-04-02T08:52:53.176 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:53.177+0000 7fb4a483a240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T08:52:53.229 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:53.231+0000 7f80395b7240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T08:52:53.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.351+0000 7fd31b400700 1 -- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 msgr2=0x7fd2fc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:53.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.351+0000 7fd31b400700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 0x7fd2fc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:52:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.384+0000 7fd322e00700 1 -- 172.21.15.89:0/3477115012 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7fd31c161fa0 con 0x7fd2fc06ae00 2024-04-02T08:52:53.430 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:53.431+0000 7fb4a483a240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T08:52:53.488 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:53.489+0000 7f80395b7240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T08:52:53.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.752+0000 7fd31b400700 1 -- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 msgr2=0x7fd2fc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:53.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:53.752+0000 7fd31b400700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 0x7fd2fc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T08:52:53.885 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:53.886+0000 7fb4a483a240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T08:52:53.946 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:53.948+0000 7f80395b7240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T08:52:54.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:54.553+0000 7fd31b400700 1 -- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 msgr2=0x7fd2fc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:54.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:54.553+0000 7fd31b400700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 0x7fd2fc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T08:52:54.587 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:54.588+0000 7fb4a483a240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T08:52:54.650 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:54.651+0000 7f80395b7240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T08:52:54.677 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:54.678+0000 7fb4a483a240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T08:52:54.742 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:54.743+0000 7f80395b7240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T08:52:54.835 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:54.836+0000 7fb4a483a240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T08:52:54.899 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:54.900+0000 7f80395b7240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T08:52:55.048 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:55.049+0000 7fb4a483a240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T08:52:55.112 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:55.113+0000 7f80395b7240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T08:52:55.138 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:55.139+0000 7fb4a483a240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T08:52:55.203 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:55.204+0000 7f80395b7240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T08:52:55.236 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:55.237+0000 7fb4a483a240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T08:52:55.301 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:55.301+0000 7f80395b7240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T08:52:55.334 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:55.334+0000 7fb4a483a240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T08:52:55.398 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:55.398+0000 7f80395b7240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T08:52:55.431 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:55.432+0000 7fb4a483a240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T08:52:55.498 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:55.497+0000 7f80395b7240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T08:52:55.643 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:55.644+0000 7fb4a483a240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T08:52:55.708 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:55.708+0000 7f80395b7240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T08:52:55.896 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:55.897+0000 7fb4a483a240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T08:52:55.963 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:55.963+0000 7f80395b7240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T08:52:56.088 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:56.089+0000 7fb4a483a240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T08:52:56.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:56.153+0000 7fd31b400700 1 -- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 msgr2=0x7fd2fc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:52:56.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:56.154+0000 7fd31b400700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 0x7fd2fc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T08:52:56.154 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:56.155+0000 7f80395b7240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T08:52:56.180 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:56.181+0000 7fb4a483a240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T08:52:56.246 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:56.247+0000 7f80395b7240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T08:52:56.905 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:56.906+0000 7fb4a483a240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T08:52:56.972 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:56.973+0000 7f80395b7240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T08:52:57.116 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:57.117+0000 7fb4a483a240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T08:52:57.182 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:57.183+0000 7f80395b7240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T08:52:57.323 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:57.324+0000 7fb4a483a240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:52:57.387 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:57.387+0000 7f80395b7240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:52:57.792 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:57.793+0000 7fb4a483a240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T08:52:57.859 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:57.859+0000 7f80395b7240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T08:52:57.996 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:57.997+0000 7fb4a483a240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:52:58.067 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:58.067+0000 7f80395b7240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:52:58.434 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:58.435+0000 7fb4a483a240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T08:52:58.511 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:58.509+0000 7f80395b7240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T08:52:58.546 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:52:58.547+0000 7fb4a483a240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T08:52:58.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:58.607+0000 7fd313e00700 1 -- 172.21.15.89:0/3477115012 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 105) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fd30c003cf0 con 0x7fd31c0d16a0 2024-04-02T08:52:58.625 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:58.626+0000 7f80395b7240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T08:52:58.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:58.636+0000 7fd313e00700 1 -- 172.21.15.89:0/3477115012 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 106) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fd30c055830 con 0x7fd31c0d16a0 2024-04-02T08:52:58.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:58.636+0000 7fd313e00700 1 -- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 msgr2=0x7fd2fc06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:52:58.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:58.636+0000 7fd313e00700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 0x7fd2fc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:58.807 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:58.807+0000 7f7fa1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:58.807 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:58.807+0000 7f7fa1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:58.807 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:58.807+0000 7f7fa1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:58.808 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:58.807+0000 7f7fa1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:58.808 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:58.807+0000 7f7fa1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:58.808 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:58.808+0000 7f7fa5a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:58.808 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:58.808+0000 7f7fa5a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:58.808 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:58.808+0000 7f7fa5a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:58.808 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:58.808+0000 7f7fa5a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:58.808 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:52:58.808+0000 7f7fa5a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:52:59.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.641+0000 7fd313e00700 1 -- 172.21.15.89:0/3477115012 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mgrmap(e 107) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fd30c056100 con 0x7fd31c0d16a0 2024-04-02T08:52:59.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.642+0000 7fd313e00700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 0x7fd2fc06d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:59.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.642+0000 7fd313e00700 1 -- 172.21.15.89:0/3477115012 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2fc01d200 con 0x7fd2fc06ae00 2024-04-02T08:52:59.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.645+0000 7fd31b400700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 0x7fd2fc06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:59.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.648+0000 7fd31b400700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 0x7fd2fc06d240 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd31c15c930 tx=0x7fd308007eb0 comp rx=0 tx=0).ready entity=mgr.9714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:59.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.719+0000 7fd313e00700 1 -- 172.21.15.89:0/3477115012 <== mgr.9714 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd2fc01d200 con 0x7fd2fc06ae00 2024-04-02T08:52:59.719 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path snap_test_dir1 2024-04-02T08:52:59.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.722+0000 7fd322e00700 1 -- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 msgr2=0x7fd2fc06d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:59.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.722+0000 7fd322e00700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 0x7fd2fc06d240 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd31c15c930 tx=0x7fd308007eb0 comp rx=0 tx=0).stop 2024-04-02T08:52:59.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.722+0000 7fd322e00700 1 -- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31c0d16a0 msgr2=0x7fd31c1e7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:59.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.722+0000 7fd322e00700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31c0d16a0 0x7fd31c1e7c90 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009c40 tx=0x7fd30c004750 comp rx=0 tx=0).stop 2024-04-02T08:52:59.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.722+0000 7fd322e00700 1 -- 172.21.15.89:0/3477115012 shutdown_connections 2024-04-02T08:52:59.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.722+0000 7fd322e00700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd2fc06ae00 0x7fd2fc06d240 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:59.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.722+0000 7fd322e00700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd31c15d9d0 0x7fd31c0b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:59.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.722+0000 7fd322e00700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd31c0d16a0 0x7fd31c1e7c90 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:59.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.722+0000 7fd322e00700 1 --2- 172.21.15.89:0/3477115012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd31c0d2540 0x7fd31c14f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:59.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.722+0000 7fd322e00700 1 -- 172.21.15.89:0/3477115012 >> 172.21.15.89:0/3477115012 conn(0x7fd31c08aff0 msgr2=0x7fd31c0b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:59.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.722+0000 7fd322e00700 1 -- 172.21.15.89:0/3477115012 shutdown_connections 2024-04-02T08:52:59.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.722+0000 7fd322e00700 1 -- 172.21.15.89:0/3477115012 wait complete. 2024-04-02T08:52:59.741 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-02T08:52:59.742 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-02T08:52:59.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.936+0000 7fc010c00700 1 Processor -- start 2024-04-02T08:52:59.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.936+0000 7fc010c00700 1 -- start start 2024-04-02T08:52:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.937+0000 7fc010c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc00c15dad0 0x7fc00c0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.937+0000 7fc010c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00c15f820 con 0x7fc00c0d21d0 2024-04-02T08:52:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.937+0000 7fc010c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00c0cf420 con 0x7fc00c0d1630 2024-04-02T08:52:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.937+0000 7fc010c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc00c0cfcc0 con 0x7fc00c15dad0 2024-04-02T08:52:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.937+0000 7fc00a800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc00c0d21d0 0x7fc00c15fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45442/0 (socket says 172.21.15.89:45442) 2024-04-02T08:52:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.937+0000 7fc00a800700 1 -- 172.21.15.89:0/1810247834 learned_addr learned my addr 172.21.15.89:0/1810247834 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:52:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.937+0000 7fc009e00700 1 --2- 172.21.15.89:0/1810247834 >> v2:172.21.15.113:3300/0 conn(0x7fc00c15dad0 0x7fc00c0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.938+0000 7fc009400700 1 -- 172.21.15.89:0/1810247834 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476161467 0 0) 0x7fc00c15f820 con 0x7fc00c0d21d0 2024-04-02T08:52:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.938+0000 7fc009400700 1 -- 172.21.15.89:0/1810247834 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbff00035b0 con 0x7fc00c0d21d0 2024-04-02T08:52:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.938+0000 7fc009400700 1 -- 172.21.15.89:0/1810247834 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316286835 0 0) 0x7fc00c0cf420 con 0x7fc00c0d1630 2024-04-02T08:52:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.938+0000 7fc009400700 1 -- 172.21.15.89:0/1810247834 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc00c15f820 con 0x7fc00c0d1630 2024-04-02T08:52:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.938+0000 7fc009e00700 1 -- 172.21.15.89:0/1810247834 >> v1:172.21.15.113:6789/0 conn(0x7fc00c0d1630 legacy=0x7fc00c0d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:59.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.938+0000 7fc009e00700 1 -- 172.21.15.89:0/1810247834 >> v1:172.21.15.89:6789/0 conn(0x7fc00c0d21d0 legacy=0x7fc00c15fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.938+0000 7fc009e00700 1 -- 172.21.15.89:0/1810247834 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c0cf700 con 0x7fc00c15dad0 2024-04-02T08:52:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.938+0000 7fc009e00700 1 --2- 172.21.15.89:0/1810247834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc00c15dad0 0x7fc00c0ced80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009840 tx=0x7fbffc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c72bb654f117773 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.938+0000 7fc009400700 1 -- 172.21.15.89:0/1810247834 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc011830 con 0x7fc00c15dad0 2024-04-02T08:52:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.938+0000 7fc009400700 1 -- 172.21.15.89:0/1810247834 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbffc011990 con 0x7fc00c15dad0 2024-04-02T08:52:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.939+0000 7fc009400700 1 -- 172.21.15.89:0/1810247834 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc011d20 con 0x7fc00c15dad0 2024-04-02T08:52:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.939+0000 7fc010c00700 1 -- 172.21.15.89:0/1810247834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc00c15dad0 msgr2=0x7fc00c0ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.939+0000 7fc010c00700 1 --2- 172.21.15.89:0/1810247834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc00c15dad0 0x7fc00c0ced80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009840 tx=0x7fbffc009b50 comp rx=0 tx=0).stop 2024-04-02T08:52:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.939+0000 7fc010c00700 1 -- 172.21.15.89:0/1810247834 shutdown_connections 2024-04-02T08:52:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.939+0000 7fc010c00700 1 --2- 172.21.15.89:0/1810247834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc00c15dad0 0x7fc00c0ced80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.939+0000 7fc010c00700 1 -- 172.21.15.89:0/1810247834 >> 172.21.15.89:0/1810247834 conn(0x7fc00c08aff0 msgr2=0x7fc00c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:52:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.939+0000 7fc010c00700 1 -- 172.21.15.89:0/1810247834 shutdown_connections 2024-04-02T08:52:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.939+0000 7fc010c00700 1 -- 172.21.15.89:0/1810247834 wait complete. 2024-04-02T08:52:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.941+0000 7fc010c00700 1 Processor -- start 2024-04-02T08:52:59.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.941+0000 7fc010c00700 1 -- start start 2024-04-02T08:52:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc010c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc00c0d1630 0x7fc00c0c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc010c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc00c0d21d0 0x7fc00c06b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc010c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc00c15dad0 0x7fc00c06be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc010c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c14ab60 con 0x7fc00c0d1630 2024-04-02T08:52:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc010c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c0ca1e0 con 0x7fc00c15dad0 2024-04-02T08:52:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc010c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c0b82f0 con 0x7fc00c0d21d0 2024-04-02T08:52:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc009e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc00c0d1630 0x7fc00c0c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc009e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc00c0d1630 0x7fc00c0c9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38406/0 (socket says 172.21.15.89:38406) 2024-04-02T08:52:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc009e00700 1 -- 172.21.15.89:0/3913085513 learned_addr learned my addr 172.21.15.89:0/3913085513 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:52:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc003e00700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc00c0d21d0 0x7fc00c06b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc00a800700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc00c15dad0 0x7fc00c06be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc009e00700 1 -- 172.21.15.89:0/3913085513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc00c0d21d0 msgr2=0x7fc00c06b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc009e00700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc00c0d21d0 0x7fc00c06b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc009e00700 1 -- 172.21.15.89:0/3913085513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc00c15dad0 msgr2=0x7fc00c06be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:52:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc009e00700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc00c15dad0 0x7fc00c06be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:52:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc009e00700 1 -- 172.21.15.89:0/3913085513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc0094a0 con 0x7fc00c0d1630 2024-04-02T08:52:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.942+0000 7fc003e00700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc00c0d21d0 0x7fc00c06b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:52:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.943+0000 7fc009e00700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc00c0d1630 0x7fc00c0c9ca0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0090b0 tx=0x7fbffc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.943+0000 7fc002000700 1 -- 172.21.15.89:0/3913085513 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc01aa70 con 0x7fc00c0d1630 2024-04-02T08:52:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.943+0000 7fc002000700 1 -- 172.21.15.89:0/3913085513 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbffc01abd0 con 0x7fc00c0d1630 2024-04-02T08:52:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.943+0000 7fc010c00700 1 -- 172.21.15.89:0/3913085513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc00c0b85d0 con 0x7fc00c0d1630 2024-04-02T08:52:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.943+0000 7fc002000700 1 -- 172.21.15.89:0/3913085513 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc0243f0 con 0x7fc00c0d1630 2024-04-02T08:52:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.943+0000 7fc010c00700 1 -- 172.21.15.89:0/3913085513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc00c14a780 con 0x7fc00c0d1630 2024-04-02T08:52:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.944+0000 7fc010c00700 1 -- 172.21.15.89:0/3913085513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00c067720 con 0x7fc00c0d1630 2024-04-02T08:52:59.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.945+0000 7fc002000700 1 -- 172.21.15.89:0/3913085513 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 107) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fbffc02d450 con 0x7fc00c0d1630 2024-04-02T08:52:59.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.946+0000 7fc002000700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbfe806ab30 0x7fbfe806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:52:59.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.946+0000 7fc002000700 1 -- 172.21.15.89:0/3913085513 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fbffc08a650 con 0x7fc00c0d1630 2024-04-02T08:52:59.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.946+0000 7fc003e00700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbfe806ab30 0x7fbfe806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:52:59.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.947+0000 7fc003e00700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbfe806ab30 0x7fbfe806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbff40011e0 tx=0x7fbff4009040 comp rx=0 tx=0).ready entity=mgr.9714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:52:59.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:52:59.951+0000 7fc002000700 1 -- 172.21.15.89:0/3913085513 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbffc055990 con 0x7fc00c0d1630 2024-04-02T08:53:00.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.179+0000 7fc010c00700 1 -- 172.21.15.89:0/3913085513 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc00c1603c0 con 0x7fbfe806ab30 2024-04-02T08:53:00.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.260+0000 7fc002000700 1 -- 172.21.15.89:0/3913085513 <== mgr.9714 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fbff4001cf0 con 0x7fbfe806ab30 2024-04-02T08:53:00.260 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path snap_test_dir1 2024-04-02T08:53:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.263+0000 7fc010c00700 1 -- 172.21.15.89:0/3913085513 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbfe806ab30 msgr2=0x7fbfe806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.263+0000 7fc010c00700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbfe806ab30 0x7fbfe806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbff40011e0 tx=0x7fbff4009040 comp rx=0 tx=0).stop 2024-04-02T08:53:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.263+0000 7fc010c00700 1 -- 172.21.15.89:0/3913085513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc00c0d1630 msgr2=0x7fc00c0c9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.263+0000 7fc010c00700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc00c0d1630 0x7fc00c0c9ca0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0090b0 tx=0x7fbffc004750 comp rx=0 tx=0).stop 2024-04-02T08:53:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.263+0000 7fc010c00700 1 -- 172.21.15.89:0/3913085513 shutdown_connections 2024-04-02T08:53:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.263+0000 7fc010c00700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbfe806ab30 0x7fbfe806cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.263+0000 7fc010c00700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc00c15dad0 0x7fc00c06be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.263+0000 7fc010c00700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc00c0d1630 0x7fc00c0c9ca0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.263+0000 7fc010c00700 1 --2- 172.21.15.89:0/3913085513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc00c0d21d0 0x7fc00c06b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:00.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.263+0000 7fc010c00700 1 -- 172.21.15.89:0/3913085513 >> 172.21.15.89:0/3913085513 conn(0x7fc00c08aff0 msgr2=0x7fc00c15b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:00.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.263+0000 7fc010c00700 1 -- 172.21.15.89:0/3913085513 shutdown_connections 2024-04-02T08:53:00.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.263+0000 7fc010c00700 1 -- 172.21.15.89:0/3913085513 wait complete. 2024-04-02T08:53:00.280 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-02T08:53:00.280 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-04-02T08:53:00.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.473+0000 7fe487200700 1 Processor -- start 2024-04-02T08:53:00.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.473+0000 7fe487200700 1 -- start start 2024-04-02T08:53:00.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.475+0000 7fe487200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4801611b0 0x7fe48015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:00.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.475+0000 7fe487200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe4800cee10 0x7fe480161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:00.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.475+0000 7fe487200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4800ce270 con 0x7fe48015f4d0 2024-04-02T08:53:00.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.475+0000 7fe487200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48015b600 con 0x7fe4800cee10 2024-04-02T08:53:00.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.475+0000 7fe487200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4800d2760 con 0x7fe4801611b0 2024-04-02T08:53:00.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.475+0000 7fe485200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe4800cee10 0x7fe480161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:00.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.475+0000 7fe47fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4801611b0 0x7fe48015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:00.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.475+0000 7fe485200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe4800cee10 0x7fe480161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:34086/0 (socket says 172.21.15.89:34086) 2024-04-02T08:53:00.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.475+0000 7fe485200700 1 -- 172.21.15.89:0/4129369315 learned_addr learned my addr 172.21.15.89:0/4129369315 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:00.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.475+0000 7fe47f400700 1 --1- 172.21.15.89:0/4129369315 >> v1:172.21.15.113:6789/0 conn(0x7fe48015f4d0 0x7fe4800ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:51920/0 (socket says 172.21.15.89:51920) 2024-04-02T08:53:00.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.476+0000 7fe485200700 1 -- 172.21.15.89:0/4129369315 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4801611b0 msgr2=0x7fe48015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:00.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.476+0000 7fe485200700 1 --2- 172.21.15.89:0/4129369315 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4801611b0 0x7fe48015ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:00.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.476+0000 7fe485200700 1 -- 172.21.15.89:0/4129369315 >> v1:172.21.15.113:6789/0 conn(0x7fe48015f4d0 legacy=0x7fe4800ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:00.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.476+0000 7fe485200700 1 -- 172.21.15.89:0/4129369315 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4800d2a40 con 0x7fe4800cee10 2024-04-02T08:53:00.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.476+0000 7fe485200700 1 --2- 172.21.15.89:0/4129369315 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4800cee10 0x7fe480161590 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe478009e50 tx=0x7fe47800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1efc7894a6af7b82 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:00.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.476+0000 7fe47ea00700 1 -- 172.21.15.89:0/4129369315 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47800e070 con 0x7fe4800cee10 2024-04-02T08:53:00.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.476+0000 7fe47ea00700 1 -- 172.21.15.89:0/4129369315 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47800b910 con 0x7fe4800cee10 2024-04-02T08:53:00.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.476+0000 7fe47ea00700 1 -- 172.21.15.89:0/4129369315 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47800bcc0 con 0x7fe4800cee10 2024-04-02T08:53:00.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.477+0000 7fe487200700 1 -- 172.21.15.89:0/4129369315 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4800cee10 msgr2=0x7fe480161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:00.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.477+0000 7fe487200700 1 --2- 172.21.15.89:0/4129369315 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4800cee10 0x7fe480161590 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe478009e50 tx=0x7fe47800b330 comp rx=0 tx=0).stop 2024-04-02T08:53:00.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.477+0000 7fe487200700 1 -- 172.21.15.89:0/4129369315 shutdown_connections 2024-04-02T08:53:00.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.477+0000 7fe487200700 1 --2- 172.21.15.89:0/4129369315 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4800cee10 0x7fe480161590 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:00.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.477+0000 7fe487200700 1 --2- 172.21.15.89:0/4129369315 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4801611b0 0x7fe48015ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:00.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.477+0000 7fe487200700 1 -- 172.21.15.89:0/4129369315 >> 172.21.15.89:0/4129369315 conn(0x7fe48008aff0 msgr2=0x7fe4800d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:00.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.477+0000 7fe487200700 1 -- 172.21.15.89:0/4129369315 shutdown_connections 2024-04-02T08:53:00.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.477+0000 7fe487200700 1 -- 172.21.15.89:0/4129369315 wait complete. 2024-04-02T08:53:00.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.479+0000 7fe487200700 1 Processor -- start 2024-04-02T08:53:00.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.479+0000 7fe487200700 1 -- start start 2024-04-02T08:53:00.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe487200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4800cee10 0x7fe48006c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:00.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe487200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe48015f4d0 0x7fe4800a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:00.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe487200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4801611b0 0x7fe48015a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:00.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe487200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4801f3e50 con 0x7fe4800cee10 2024-04-02T08:53:00.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe487200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4801f32d0 con 0x7fe4801611b0 2024-04-02T08:53:00.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe487200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4801f35b0 con 0x7fe48015f4d0 2024-04-02T08:53:00.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe47fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4800cee10 0x7fe48006c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:00.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe47fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4800cee10 0x7fe48006c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38410/0 (socket says 172.21.15.89:38410) 2024-04-02T08:53:00.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe47fe00700 1 -- 172.21.15.89:0/834896162 learned_addr learned my addr 172.21.15.89:0/834896162 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:00.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe47f400700 1 --2- 172.21.15.89:0/834896162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe48015f4d0 0x7fe4800a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:00.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe485200700 1 --2- 172.21.15.89:0/834896162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4801611b0 0x7fe48015a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:00.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe47fe00700 1 -- 172.21.15.89:0/834896162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe48015f4d0 msgr2=0x7fe4800a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:00.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe47fe00700 1 --2- 172.21.15.89:0/834896162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe48015f4d0 0x7fe4800a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:00.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe47fe00700 1 -- 172.21.15.89:0/834896162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4801611b0 msgr2=0x7fe48015a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:00.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe47fe00700 1 --2- 172.21.15.89:0/834896162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4801611b0 0x7fe48015a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:00.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.480+0000 7fe47fe00700 1 -- 172.21.15.89:0/834896162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe478009b30 con 0x7fe4800cee10 2024-04-02T08:53:00.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.481+0000 7fe47fe00700 1 --2- 172.21.15.89:0/834896162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4800cee10 0x7fe48006c900 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fe47000e870 tx=0x7fe47000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:00.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.481+0000 7fe477e00700 1 -- 172.21.15.89:0/834896162 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470009590 con 0x7fe4800cee10 2024-04-02T08:53:00.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.481+0000 7fe477e00700 1 -- 172.21.15.89:0/834896162 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe470013680 con 0x7fe4800cee10 2024-04-02T08:53:00.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.481+0000 7fe487200700 1 -- 172.21.15.89:0/834896162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4801f3890 con 0x7fe4800cee10 2024-04-02T08:53:00.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.481+0000 7fe477e00700 1 -- 172.21.15.89:0/834896162 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4700139f0 con 0x7fe4800cee10 2024-04-02T08:53:00.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.481+0000 7fe487200700 1 -- 172.21.15.89:0/834896162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48006ce40 con 0x7fe4800cee10 2024-04-02T08:53:00.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.482+0000 7fe487200700 1 -- 172.21.15.89:0/834896162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe480067720 con 0x7fe4800cee10 2024-04-02T08:53:00.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.483+0000 7fe477e00700 1 -- 172.21.15.89:0/834896162 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 107) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fe470013b50 con 0x7fe4800cee10 2024-04-02T08:53:00.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.484+0000 7fe477e00700 1 --2- 172.21.15.89:0/834896162 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe46006ab30 0x7fe46006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:00.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.484+0000 7fe477e00700 1 -- 172.21.15.89:0/834896162 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fe470089230 con 0x7fe4800cee10 2024-04-02T08:53:00.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.484+0000 7fe47f400700 1 --2- 172.21.15.89:0/834896162 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe46006ab30 0x7fe46006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:00.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.484+0000 7fe47f400700 1 --2- 172.21.15.89:0/834896162 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe46006ab30 0x7fe46006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe4800cede0 tx=0x7fe46c007ed0 comp rx=0 tx=0).ready entity=mgr.9714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:00.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.488+0000 7fe477e00700 1 -- 172.21.15.89:0/834896162 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe470054380 con 0x7fe4800cee10 2024-04-02T08:53:00.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.647+0000 7fe477e00700 1 -- 172.21.15.89:0/834896162 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 108) v1 ==== 89257+0+0 (secure 0 0 0) 0x7fe470054750 con 0x7fe4800cee10 2024-04-02T08:53:00.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.724+0000 7fe487200700 1 -- 172.21.15.89:0/834896162 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4800cdf00 con 0x7fe46006ab30 2024-04-02T08:53:00.732 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:00.732+0000 7f7fae600700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-04-02T08:53:00.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:00.732+0000 7fe477e00700 1 -- 172.21.15.89:0/834896162 <== mgr.9714 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7fe46c004af0 con 0x7fe46006ab30 2024-04-02T08:53:00.733 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-04-02T08:53:00.741 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-02T08:53:00.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:53:00.742 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-02T08:53:00.788 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T08:53:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.021+0000 7fbb38000700 1 Processor -- start 2024-04-02T08:53:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.022+0000 7fbb38000700 1 -- start start 2024-04-02T08:53:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.023+0000 7fbb38000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbb3015dad0 0x7fbb300ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.023+0000 7fbb38000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb300d1630 0x7fbb300d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.023+0000 7fbb38000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbb300d1fd0 0x7fbb3015deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.023+0000 7fbb38000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb300cfcc0 con 0x7fbb300d1fd0 2024-04-02T08:53:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.023+0000 7fbb38000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb300cd9f0 con 0x7fbb3015dad0 2024-04-02T08:53:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.023+0000 7fbb38000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3015f820 con 0x7fbb300d1630 2024-04-02T08:53:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.023+0000 7fbb36000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbb300d1fd0 0x7fbb3015deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.023+0000 7fbb36000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbb300d1fd0 0x7fbb3015deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38436/0 (socket says 172.21.15.89:38436) 2024-04-02T08:53:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.023+0000 7fbb36000700 1 -- 172.21.15.89:0/1113564595 learned_addr learned my addr 172.21.15.89:0/1113564595 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.023+0000 7fbb35600700 1 --2- 172.21.15.89:0/1113564595 >> v2:172.21.15.113:3300/0 conn(0x7fbb3015dad0 0x7fbb300ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.023+0000 7fbb34c00700 1 --2- 172.21.15.89:0/1113564595 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb300d1630 0x7fbb300d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.024+0000 7fbb36000700 1 -- 172.21.15.89:0/1113564595 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb300d1630 msgr2=0x7fbb300d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.024+0000 7fbb36000700 1 --2- 172.21.15.89:0/1113564595 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb300d1630 0x7fbb300d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.024+0000 7fbb36000700 1 -- 172.21.15.89:0/1113564595 >> v2:172.21.15.113:3300/0 conn(0x7fbb3015dad0 msgr2=0x7fbb300ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.024+0000 7fbb36000700 1 --2- 172.21.15.89:0/1113564595 >> v2:172.21.15.113:3300/0 conn(0x7fbb3015dad0 0x7fbb300ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.024+0000 7fbb36000700 1 -- 172.21.15.89:0/1113564595 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb300d2460 con 0x7fbb300d1fd0 2024-04-02T08:53:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.024+0000 7fbb35600700 1 --2- 172.21.15.89:0/1113564595 >> v2:172.21.15.113:3300/0 conn(0x7fbb3015dad0 0x7fbb300ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.024+0000 7fbb36000700 1 --2- 172.21.15.89:0/1113564595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb300d1fd0 0x7fbb3015deb0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fbb3008d380 tx=0x7fbb2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467b04bfd2feacbf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.024+0000 7fbb2fe00700 1 -- 172.21.15.89:0/1113564595 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800e070 con 0x7fbb300d1fd0 2024-04-02T08:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.024+0000 7fbb2fe00700 1 -- 172.21.15.89:0/1113564595 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2800b910 con 0x7fbb300d1fd0 2024-04-02T08:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.024+0000 7fbb2fe00700 1 -- 172.21.15.89:0/1113564595 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800bcc0 con 0x7fbb300d1fd0 2024-04-02T08:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.025+0000 7fbb38000700 1 -- 172.21.15.89:0/1113564595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb300d1fd0 msgr2=0x7fbb3015deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.025+0000 7fbb38000700 1 --2- 172.21.15.89:0/1113564595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb300d1fd0 0x7fbb3015deb0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fbb3008d380 tx=0x7fbb2800b330 comp rx=0 tx=0).stop 2024-04-02T08:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.025+0000 7fbb38000700 1 -- 172.21.15.89:0/1113564595 shutdown_connections 2024-04-02T08:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.025+0000 7fbb38000700 1 --2- 172.21.15.89:0/1113564595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb300d1fd0 0x7fbb3015deb0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.025+0000 7fbb38000700 1 --2- 172.21.15.89:0/1113564595 >> v2:172.21.15.113:3300/0 conn(0x7fbb3015dad0 0x7fbb300ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.025+0000 7fbb38000700 1 --2- 172.21.15.89:0/1113564595 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb300d1630 0x7fbb300d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.025+0000 7fbb38000700 1 -- 172.21.15.89:0/1113564595 >> 172.21.15.89:0/1113564595 conn(0x7fbb3008aff0 msgr2=0x7fbb300d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.025+0000 7fbb38000700 1 -- 172.21.15.89:0/1113564595 shutdown_connections 2024-04-02T08:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.025+0000 7fbb38000700 1 -- 172.21.15.89:0/1113564595 wait complete. 2024-04-02T08:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.027+0000 7fbb38000700 1 Processor -- start 2024-04-02T08:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.027+0000 7fbb38000700 1 -- start start 2024-04-02T08:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.028+0000 7fbb38000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb300d1630 0x7fbb30153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.028+0000 7fbb38000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb300d1fd0 0x7fbb30150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.028+0000 7fbb38000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb3015dad0 0x7fbb30156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.028+0000 7fbb38000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb30153880 con 0x7fbb3015dad0 2024-04-02T08:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.028+0000 7fbb38000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300b6480 con 0x7fbb300d1630 2024-04-02T08:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.028+0000 7fbb38000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300b6760 con 0x7fbb300d1fd0 2024-04-02T08:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.028+0000 7fbb36000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb3015dad0 0x7fbb30156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.028+0000 7fbb36000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb3015dad0 0x7fbb30156520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38440/0 (socket says 172.21.15.89:38440) 2024-04-02T08:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.028+0000 7fbb36000700 1 -- 172.21.15.89:0/2220290789 learned_addr learned my addr 172.21.15.89:0/2220290789 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.028+0000 7fbb34c00700 1 --2- 172.21.15.89:0/2220290789 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb300d1fd0 0x7fbb30150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.028+0000 7fbb35600700 1 --2- 172.21.15.89:0/2220290789 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb300d1630 0x7fbb30153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.029+0000 7fbb36000700 1 -- 172.21.15.89:0/2220290789 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb300d1fd0 msgr2=0x7fbb30150b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.029+0000 7fbb36000700 1 --2- 172.21.15.89:0/2220290789 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb300d1fd0 0x7fbb30150b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.029+0000 7fbb36000700 1 -- 172.21.15.89:0/2220290789 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb300d1630 msgr2=0x7fbb30153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.029+0000 7fbb36000700 1 --2- 172.21.15.89:0/2220290789 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb300d1630 0x7fbb30153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.029+0000 7fbb36000700 1 -- 172.21.15.89:0/2220290789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb240094a0 con 0x7fbb3015dad0 2024-04-02T08:53:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.029+0000 7fbb36000700 1 --2- 172.21.15.89:0/2220290789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb3015dad0 0x7fbb30156520 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fbb3008ce40 tx=0x7fbb28004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.029+0000 7fbb2d600700 1 -- 172.21.15.89:0/2220290789 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb280092e0 con 0x7fbb3015dad0 2024-04-02T08:53:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.029+0000 7fbb2d600700 1 -- 172.21.15.89:0/2220290789 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2800ce80 con 0x7fbb3015dad0 2024-04-02T08:53:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.029+0000 7fbb2d600700 1 -- 172.21.15.89:0/2220290789 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb280044b0 con 0x7fbb3015dad0 2024-04-02T08:53:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.029+0000 7fbb38000700 1 -- 172.21.15.89:0/2220290789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb28009b30 con 0x7fbb3015dad0 2024-04-02T08:53:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.029+0000 7fbb38000700 1 -- 172.21.15.89:0/2220290789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb300b6d20 con 0x7fbb3015dad0 2024-04-02T08:53:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.030+0000 7fbb38000700 1 -- 172.21.15.89:0/2220290789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb30067720 con 0x7fbb3015dad0 2024-04-02T08:53:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.031+0000 7fbb2d600700 1 -- 172.21.15.89:0/2220290789 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 108) v1 ==== 89257+0+0 (secure 0 0 0) 0x7fbb28004610 con 0x7fbb3015dad0 2024-04-02T08:53:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.032+0000 7fbb2d600700 1 --2- 172.21.15.89:0/2220290789 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbb1006ae00 0x7fbb1006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:01.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.032+0000 7fbb2d600700 1 -- 172.21.15.89:0/2220290789 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fbb2808aea0 con 0x7fbb3015dad0 2024-04-02T08:53:01.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.032+0000 7fbb35600700 1 --2- 172.21.15.89:0/2220290789 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbb1006ae00 0x7fbb1006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:01.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.033+0000 7fbb35600700 1 --2- 172.21.15.89:0/2220290789 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbb1006ae00 0x7fbb1006d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009cd0 tx=0x7fbb24009210 comp rx=0 tx=0).ready entity=mgr.9714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:01.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.037+0000 7fbb2d600700 1 -- 172.21.15.89:0/2220290789 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbb28059680 con 0x7fbb3015dad0 2024-04-02T08:53:01.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.296+0000 7fbb38000700 1 -- 172.21.15.89:0/2220290789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fbb3006a840 con 0x7fbb3015dad0 2024-04-02T08:53:01.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.655+0000 7fbb2d600700 1 -- 172.21.15.89:0/2220290789 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fbb280890c0 con 0x7fbb3015dad0 2024-04-02T08:53:01.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.657+0000 7fbb2d600700 1 -- 172.21.15.89:0/2220290789 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v109) v1 ==== 93+0+0 (secure 0 0 0) 0x7fbb2806f330 con 0x7fbb3015dad0 2024-04-02T08:53:01.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.661+0000 7fbb38000700 1 -- 172.21.15.89:0/2220290789 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbb1006ae00 msgr2=0x7fbb1006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:01.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.661+0000 7fbb38000700 1 --2- 172.21.15.89:0/2220290789 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbb1006ae00 0x7fbb1006d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009cd0 tx=0x7fbb24009210 comp rx=0 tx=0).stop 2024-04-02T08:53:01.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.661+0000 7fbb38000700 1 -- 172.21.15.89:0/2220290789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb3015dad0 msgr2=0x7fbb30156520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:01.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.661+0000 7fbb38000700 1 --2- 172.21.15.89:0/2220290789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb3015dad0 0x7fbb30156520 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fbb3008ce40 tx=0x7fbb28004ad0 comp rx=0 tx=0).stop 2024-04-02T08:53:01.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.661+0000 7fbb38000700 1 -- 172.21.15.89:0/2220290789 shutdown_connections 2024-04-02T08:53:01.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.661+0000 7fbb38000700 1 --2- 172.21.15.89:0/2220290789 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fbb1006ae00 0x7fbb1006d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:01.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.661+0000 7fbb38000700 1 --2- 172.21.15.89:0/2220290789 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb300d1630 0x7fbb30153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:01.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.661+0000 7fbb38000700 1 --2- 172.21.15.89:0/2220290789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb3015dad0 0x7fbb30156520 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:01.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.661+0000 7fbb38000700 1 --2- 172.21.15.89:0/2220290789 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb300d1fd0 0x7fbb30150b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:01.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.661+0000 7fbb38000700 1 -- 172.21.15.89:0/2220290789 >> 172.21.15.89:0/2220290789 conn(0x7fbb3008aff0 msgr2=0x7fbb300ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:01.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.661+0000 7fbb38000700 1 -- 172.21.15.89:0/2220290789 shutdown_connections 2024-04-02T08:53:01.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:01.661+0000 7fbb38000700 1 -- 172.21.15.89:0/2220290789 wait complete. 2024-04-02T08:53:01.687 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T08:53:01.687 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T08:53:01.779 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T08:53:01.869 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T08:53:01.870 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T08:53:01.870 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:53:01.882 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:01.884+0000 7f4b2ce59240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T08:53:01.894 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:01.895+0000 7fb0bdd5f240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T08:53:01.915 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T08:53:01.915 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:53:02.005 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:53:02.068 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:02.069+0000 7f4b2ce59240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T08:53:02.093 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:02.094+0000 7fb0bdd5f240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T08:53:02.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.235+0000 7f0a1d000700 1 Processor -- start 2024-04-02T08:53:02.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.235+0000 7f0a1d000700 1 -- start start 2024-04-02T08:53:02.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.236+0000 7f0a1d000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0a1815da50 0x7f0a180ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:02.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.236+0000 7f0a1d000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a180d24e0 con 0x7f0a180cec70 2024-04-02T08:53:02.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.236+0000 7f0a1d000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a180d27c0 con 0x7f0a180d0cf0 2024-04-02T08:53:02.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.236+0000 7f0a1d000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a180b95d0 con 0x7f0a1815da50 2024-04-02T08:53:02.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.236+0000 7f0a15400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0a180cec70 0x7f0a180d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45454/0 (socket says 172.21.15.89:45454) 2024-04-02T08:53:02.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.236+0000 7f0a15400700 1 -- 172.21.15.89:0/417393584 learned_addr learned my addr 172.21.15.89:0/417393584 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:02.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.236+0000 7f0a15e00700 1 --2- 172.21.15.89:0/417393584 >> v2:172.21.15.113:3300/0 conn(0x7f0a1815da50 0x7f0a180ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:02.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.237+0000 7f0a14a00700 1 -- 172.21.15.89:0/417393584 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876064201 0 0) 0x7f0a180d24e0 con 0x7f0a180cec70 2024-04-02T08:53:02.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.237+0000 7f0a14a00700 1 -- 172.21.15.89:0/417393584 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09fc0035b0 con 0x7f0a180cec70 2024-04-02T08:53:02.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.237+0000 7f0a14a00700 1 -- 172.21.15.89:0/417393584 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258015206 0 0) 0x7f0a180d27c0 con 0x7f0a180d0cf0 2024-04-02T08:53:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.237+0000 7f0a14a00700 1 -- 172.21.15.89:0/417393584 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a180d24e0 con 0x7f0a180d0cf0 2024-04-02T08:53:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.237+0000 7f0a15e00700 1 -- 172.21.15.89:0/417393584 >> v1:172.21.15.113:6789/0 conn(0x7f0a180d0cf0 legacy=0x7f0a180b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.237+0000 7f0a15e00700 1 -- 172.21.15.89:0/417393584 >> v1:172.21.15.89:6789/0 conn(0x7f0a180cec70 legacy=0x7f0a180d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.237+0000 7f0a15e00700 1 -- 172.21.15.89:0/417393584 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a180d2aa0 con 0x7f0a1815da50 2024-04-02T08:53:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.237+0000 7f0a15e00700 1 --2- 172.21.15.89:0/417393584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a1815da50 0x7f0a180ce730 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0a08009840 tx=0x7f0a08009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94b12eac261be5c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.237+0000 7f0a14a00700 1 -- 172.21.15.89:0/417393584 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08004230 con 0x7f0a1815da50 2024-04-02T08:53:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.237+0000 7f0a14a00700 1 -- 172.21.15.89:0/417393584 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a08004390 con 0x7f0a1815da50 2024-04-02T08:53:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.237+0000 7f0a14a00700 1 -- 172.21.15.89:0/417393584 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a080036a0 con 0x7f0a1815da50 2024-04-02T08:53:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.238+0000 7f0a1d000700 1 -- 172.21.15.89:0/417393584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a1815da50 msgr2=0x7f0a180ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.238+0000 7f0a1d000700 1 --2- 172.21.15.89:0/417393584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a1815da50 0x7f0a180ce730 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0a08009840 tx=0x7f0a08009b50 comp rx=0 tx=0).stop 2024-04-02T08:53:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.238+0000 7f0a1d000700 1 -- 172.21.15.89:0/417393584 shutdown_connections 2024-04-02T08:53:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.238+0000 7f0a1d000700 1 --2- 172.21.15.89:0/417393584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a1815da50 0x7f0a180ce730 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.238+0000 7f0a1d000700 1 -- 172.21.15.89:0/417393584 >> 172.21.15.89:0/417393584 conn(0x7f0a180666a0 msgr2=0x7f0a18066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:02.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.238+0000 7f0a1d000700 1 -- 172.21.15.89:0/417393584 shutdown_connections 2024-04-02T08:53:02.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.238+0000 7f0a1d000700 1 -- 172.21.15.89:0/417393584 wait complete. 2024-04-02T08:53:02.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.240+0000 7f0a1d000700 1 Processor -- start 2024-04-02T08:53:02.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.240+0000 7f0a1d000700 1 -- start start 2024-04-02T08:53:02.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.241+0000 7f0a1d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a180cec70 0x7f0a181f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:02.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.241+0000 7f0a1d000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a180d0cf0 0x7f0a181556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:02.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.241+0000 7f0a1d000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a1815da50 0x7f0a180d1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:02.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.241+0000 7f0a1d000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a18156060 con 0x7f0a180cec70 2024-04-02T08:53:02.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.241+0000 7f0a1d000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a181f3430 con 0x7f0a180d0cf0 2024-04-02T08:53:02.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.241+0000 7f0a1d000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a181f3710 con 0x7f0a1815da50 2024-04-02T08:53:02.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.241+0000 7f0a15e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a180cec70 0x7f0a181f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:02.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.241+0000 7f0a15e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a180cec70 0x7f0a181f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38470/0 (socket says 172.21.15.89:38470) 2024-04-02T08:53:02.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.241+0000 7f0a15e00700 1 -- 172.21.15.89:0/1594362858 learned_addr learned my addr 172.21.15.89:0/1594362858 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:02.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.241+0000 7f0a15400700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a180d0cf0 0x7f0a181556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:02.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.241+0000 7f0a16800700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a1815da50 0x7f0a180d1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.242+0000 7f0a15e00700 1 -- 172.21.15.89:0/1594362858 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a1815da50 msgr2=0x7f0a180d1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.242+0000 7f0a15e00700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a1815da50 0x7f0a180d1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.242+0000 7f0a15e00700 1 -- 172.21.15.89:0/1594362858 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a180d0cf0 msgr2=0x7f0a181556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.242+0000 7f0a15e00700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a180d0cf0 0x7f0a181556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.242+0000 7f0a15e00700 1 -- 172.21.15.89:0/1594362858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a080094a0 con 0x7f0a180cec70 2024-04-02T08:53:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.242+0000 7f0a16800700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a1815da50 0x7f0a180d1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.242+0000 7f0a15e00700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a180cec70 0x7f0a181f2b80 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f0a08014040 tx=0x7f0a08004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.242+0000 7f0a0e000700 1 -- 172.21.15.89:0/1594362858 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08003d30 con 0x7f0a180cec70 2024-04-02T08:53:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.242+0000 7f0a0e000700 1 -- 172.21.15.89:0/1594362858 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a08003e90 con 0x7f0a180cec70 2024-04-02T08:53:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.242+0000 7f0a1d000700 1 -- 172.21.15.89:0/1594362858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a181f39f0 con 0x7f0a180cec70 2024-04-02T08:53:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.242+0000 7f0a0e000700 1 -- 172.21.15.89:0/1594362858 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08011830 con 0x7f0a180cec70 2024-04-02T08:53:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.242+0000 7f0a1d000700 1 -- 172.21.15.89:0/1594362858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a18155c20 con 0x7f0a180cec70 2024-04-02T08:53:02.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.244+0000 7f0a1d000700 1 -- 172.21.15.89:0/1594362858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a181f32a0 con 0x7f0a180cec70 2024-04-02T08:53:02.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.245+0000 7f0a0e000700 1 -- 172.21.15.89:0/1594362858 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f0a08018040 con 0x7f0a180cec70 2024-04-02T08:53:02.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.246+0000 7f0a0e000700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09f806adb0 0x7f09f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:02.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.246+0000 7f0a15400700 1 -- 172.21.15.89:0/1594362858 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09f806adb0 msgr2=0x7f09f806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:02.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.246+0000 7f0a0e000700 1 -- 172.21.15.89:0/1594362858 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f0a08089fc0 con 0x7f0a180cec70 2024-04-02T08:53:02.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.246+0000 7f0a15400700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09f806adb0 0x7f09f806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:02.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.250+0000 7f0a0e000700 1 -- 172.21.15.89:0/1594362858 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0a08054d80 con 0x7f0a180cec70 2024-04-02T08:53:02.385 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:02.386+0000 7f4b2ce59240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T08:53:02.427 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:02.428+0000 7fb0bdd5f240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T08:53:02.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.447+0000 7f0a15400700 1 -- 172.21.15.89:0/1594362858 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09f806adb0 msgr2=0x7f09f806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:02.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.447+0000 7f0a15400700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09f806adb0 0x7f09f806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:02.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.461+0000 7f0a1d000700 1 -- 172.21.15.89:0/1594362858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a18067720 con 0x7f0a180cec70 2024-04-02T08:53:02.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.462+0000 7f0a0e000700 1 -- 172.21.15.89:0/1594362858 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) v1 ==== 74+0+17243 (secure 0 0 0) 0x7f0a0802b090 con 0x7f0a180cec70 2024-04-02T08:53:02.462 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:02.462 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":173,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:52:58.629629+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-02T08:52:40.806770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-02T08:52:41.463828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/3680638162":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/263272397":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/1641401996":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/419225467":"2024-04-03T08:52:58.629596+0000","172.21.15.89:6833/43357":"2024-04-03T08:52:58.629596+0000","172.21.15.89:6832/43357":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/2329039936":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/3986748726":"2024-04-03T08:52:58.629596+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:53:02.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.465+0000 7f0a1d000700 1 -- 172.21.15.89:0/1594362858 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09f806adb0 msgr2=0x7f09f806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:02.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.465+0000 7f0a1d000700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09f806adb0 0x7f09f806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.465+0000 7f0a1d000700 1 -- 172.21.15.89:0/1594362858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a180cec70 msgr2=0x7f0a181f2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:02.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.465+0000 7f0a1d000700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a180cec70 0x7f0a181f2b80 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f0a08014040 tx=0x7f0a08004750 comp rx=0 tx=0).stop 2024-04-02T08:53:02.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.465+0000 7f0a1d000700 1 -- 172.21.15.89:0/1594362858 shutdown_connections 2024-04-02T08:53:02.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.465+0000 7f0a1d000700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f09f806adb0 0x7f09f806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.465+0000 7f0a1d000700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a1815da50 0x7f0a180d1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.465+0000 7f0a1d000700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a180cec70 0x7f0a181f2b80 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.465+0000 7f0a1d000700 1 --2- 172.21.15.89:0/1594362858 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a180d0cf0 0x7f0a181556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.465+0000 7f0a1d000700 1 -- 172.21.15.89:0/1594362858 >> 172.21.15.89:0/1594362858 conn(0x7f0a180666a0 msgr2=0x7f0a1815b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:02.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.465+0000 7f0a1d000700 1 -- 172.21.15.89:0/1594362858 shutdown_connections 2024-04-02T08:53:02.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.465+0000 7f0a1d000700 1 -- 172.21.15.89:0/1594362858 wait complete. 2024-04-02T08:53:02.484 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T08:52:40.806770+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T08:52:41.463828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T08:53:02.484 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T08:53:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.682+0000 7f8197e00700 1 Processor -- start 2024-04-02T08:53:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.682+0000 7f8197e00700 1 -- start start 2024-04-02T08:53:02.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.683+0000 7f8197e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81900b2b30 0x7f81900b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:02.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.683+0000 7f8197e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f81900b3960 0x7f81900cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:02.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.683+0000 7f8197e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81900b3db0 con 0x7f8190157f90 2024-04-02T08:53:02.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.683+0000 7f8197e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8190158370 con 0x7f81900b3960 2024-04-02T08:53:02.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.683+0000 7f8197e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81900b2710 con 0x7f81900b2b30 2024-04-02T08:53:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.684+0000 7f8195400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f8190157f90 0x7f81901613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:51940/0 (socket says 172.21.15.89:51940) 2024-04-02T08:53:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.684+0000 7f8195400700 1 -- 172.21.15.89:0/4243258947 learned_addr learned my addr 172.21.15.89:0/4243258947 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.684+0000 7f8195e00700 1 --2- 172.21.15.89:0/4243258947 >> v2:172.21.15.113:3300/0 conn(0x7f81900b3960 0x7f81900cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.684+0000 7f8194a00700 1 --2- 172.21.15.89:0/4243258947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81900b2b30 0x7f81900b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.684+0000 7f818fe00700 1 -- 172.21.15.89:0/4243258947 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155949265 0 0) 0x7f8184001110 con 0x7f8190157f90 2024-04-02T08:53:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.684+0000 7f818fe00700 1 -- 172.21.15.89:0/4243258947 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81780035b0 con 0x7f8190157f90 2024-04-02T08:53:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.684+0000 7f8194a00700 1 -- 172.21.15.89:0/4243258947 >> v2:172.21.15.113:3300/0 conn(0x7f81900b3960 msgr2=0x7f81900cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.684+0000 7f8194a00700 1 --2- 172.21.15.89:0/4243258947 >> v2:172.21.15.113:3300/0 conn(0x7f81900b3960 0x7f81900cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.684+0000 7f8194a00700 1 -- 172.21.15.89:0/4243258947 >> v1:172.21.15.113:6789/0 conn(0x7f8190157f90 legacy=0x7f81901613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.684+0000 7f8194a00700 1 -- 172.21.15.89:0/4243258947 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81900b2070 con 0x7f81900b2b30 2024-04-02T08:53:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.684+0000 7f8195e00700 1 --2- 172.21.15.89:0/4243258947 >> v2:172.21.15.113:3300/0 conn(0x7f81900b3960 0x7f81900cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.685+0000 7f8194a00700 1 --2- 172.21.15.89:0/4243258947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81900b2b30 0x7f81900b3420 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f817c009840 tx=0x7f817c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4406beb19d89402f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.685+0000 7f818fe00700 1 -- 172.21.15.89:0/4243258947 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c011830 con 0x7f81900b2b30 2024-04-02T08:53:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.685+0000 7f818fe00700 1 -- 172.21.15.89:0/4243258947 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817c011990 con 0x7f81900b2b30 2024-04-02T08:53:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.685+0000 7f818fe00700 1 -- 172.21.15.89:0/4243258947 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c011d20 con 0x7f81900b2b30 2024-04-02T08:53:02.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.686+0000 7f8197e00700 1 -- 172.21.15.89:0/4243258947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81900b2b30 msgr2=0x7f81900b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:02.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.686+0000 7f8197e00700 1 --2- 172.21.15.89:0/4243258947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81900b2b30 0x7f81900b3420 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f817c009840 tx=0x7f817c009b50 comp rx=0 tx=0).stop 2024-04-02T08:53:02.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.686+0000 7f8197e00700 1 -- 172.21.15.89:0/4243258947 shutdown_connections 2024-04-02T08:53:02.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.686+0000 7f8197e00700 1 --2- 172.21.15.89:0/4243258947 >> v2:172.21.15.113:3300/0 conn(0x7f81900b3960 0x7f81900cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.686+0000 7f8197e00700 1 --2- 172.21.15.89:0/4243258947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81900b2b30 0x7f81900b3420 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.686+0000 7f8197e00700 1 -- 172.21.15.89:0/4243258947 >> 172.21.15.89:0/4243258947 conn(0x7f81900b0d70 msgr2=0x7f81900b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:02.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.686+0000 7f8197e00700 1 -- 172.21.15.89:0/4243258947 shutdown_connections 2024-04-02T08:53:02.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.686+0000 7f8197e00700 1 -- 172.21.15.89:0/4243258947 wait complete. 2024-04-02T08:53:02.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.687+0000 7f8197e00700 1 Processor -- start 2024-04-02T08:53:02.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.688+0000 7f8197e00700 1 -- start start 2024-04-02T08:53:02.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.688+0000 7f8197e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81900b2b30 0x7f81900c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:02.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.688+0000 7f8197e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81900b3960 0x7f81900aca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:02.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.688+0000 7f8197e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8190157f90 0x7f81901495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:02.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.688+0000 7f8197e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81900ae4b0 con 0x7f81900b3960 2024-04-02T08:53:02.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.688+0000 7f8197e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81900acfa0 con 0x7f81900b2b30 2024-04-02T08:53:02.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.688+0000 7f8197e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8190149b10 con 0x7f8190157f90 2024-04-02T08:53:02.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.689+0000 7f8194a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81900b3960 0x7f81900aca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:02.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.689+0000 7f8195400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81900b2b30 0x7f81900c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:02.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.689+0000 7f8195400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81900b2b30 0x7f81900c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:34184/0 (socket says 172.21.15.89:34184) 2024-04-02T08:53:02.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.689+0000 7f8195400700 1 -- 172.21.15.89:0/1155510387 learned_addr learned my addr 172.21.15.89:0/1155510387 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:02.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.689+0000 7f8195e00700 1 --2- 172.21.15.89:0/1155510387 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8190157f90 0x7f81901495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:02.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.689+0000 7f8195400700 1 -- 172.21.15.89:0/1155510387 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8190157f90 msgr2=0x7f81901495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:02.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.689+0000 7f8195400700 1 --2- 172.21.15.89:0/1155510387 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8190157f90 0x7f81901495d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.689+0000 7f8195400700 1 -- 172.21.15.89:0/1155510387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81900b3960 msgr2=0x7f81900aca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:02.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.689+0000 7f8195400700 1 --2- 172.21.15.89:0/1155510387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81900b3960 0x7f81900aca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.689+0000 7f8195400700 1 -- 172.21.15.89:0/1155510387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8188009a70 con 0x7f81900b2b30 2024-04-02T08:53:02.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.689+0000 7f8195e00700 1 --2- 172.21.15.89:0/1155510387 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8190157f90 0x7f81901495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:02.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.690+0000 7f8195400700 1 --2- 172.21.15.89:0/1155510387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81900b2b30 0x7f81900c83f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8184007fd0 tx=0x7f818400c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:02.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.690+0000 7f818d600700 1 -- 172.21.15.89:0/1155510387 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8184005d70 con 0x7f81900b2b30 2024-04-02T08:53:02.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.690+0000 7f818d600700 1 -- 172.21.15.89:0/1155510387 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81840070d0 con 0x7f81900b2b30 2024-04-02T08:53:02.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.690+0000 7f8197e00700 1 -- 172.21.15.89:0/1155510387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817c0094a0 con 0x7f81900b2b30 2024-04-02T08:53:02.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.690+0000 7f818d600700 1 -- 172.21.15.89:0/1155510387 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8184007460 con 0x7f81900b2b30 2024-04-02T08:53:02.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.690+0000 7f8197e00700 1 -- 172.21.15.89:0/1155510387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81900ad500 con 0x7f81900b2b30 2024-04-02T08:53:02.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.692+0000 7f8197e00700 1 -- 172.21.15.89:0/1155510387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81900b2fb0 con 0x7f81900b2b30 2024-04-02T08:53:02.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.692+0000 7f818d600700 1 -- 172.21.15.89:0/1155510387 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f81840075c0 con 0x7f81900b2b30 2024-04-02T08:53:02.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.693+0000 7f818d600700 1 --2- 172.21.15.89:0/1155510387 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f817006adb0 0x7f817006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:02.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.693+0000 7f8194a00700 1 -- 172.21.15.89:0/1155510387 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f817006adb0 msgr2=0x7f817006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:02.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.693+0000 7f818d600700 1 -- 172.21.15.89:0/1155510387 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f81840889a0 con 0x7f81900b2b30 2024-04-02T08:53:02.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.693+0000 7f8194a00700 1 --2- 172.21.15.89:0/1155510387 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f817006adb0 0x7f817006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:02.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.697+0000 7f818d600700 1 -- 172.21.15.89:0/1155510387 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8184053800 con 0x7f81900b2b30 2024-04-02T08:53:02.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.894+0000 7f8194a00700 1 -- 172.21.15.89:0/1155510387 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f817006adb0 msgr2=0x7f817006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:02.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.894+0000 7f8194a00700 1 --2- 172.21.15.89:0/1155510387 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f817006adb0 0x7f817006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:02.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.904+0000 7f8197e00700 1 -- 172.21.15.89:0/1155510387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f81900b3db0 con 0x7f81900b2b30 2024-04-02T08:53:02.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.905+0000 7f818d600700 1 -- 172.21.15.89:0/1155510387 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v173) v1 ==== 93+0+10 (secure 0 0 0) 0x7f81900b3db0 con 0x7f81900b2b30 2024-04-02T08:53:02.905 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T08:53:02.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.908+0000 7f8197e00700 1 -- 172.21.15.89:0/1155510387 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f817006adb0 msgr2=0x7f817006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.908+0000 7f8197e00700 1 --2- 172.21.15.89:0/1155510387 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f817006adb0 0x7f817006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.908+0000 7f8197e00700 1 -- 172.21.15.89:0/1155510387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81900b2b30 msgr2=0x7f81900c83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.908+0000 7f8197e00700 1 --2- 172.21.15.89:0/1155510387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81900b2b30 0x7f81900c83f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8184007fd0 tx=0x7f818400c9e0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.908+0000 7f8197e00700 1 -- 172.21.15.89:0/1155510387 shutdown_connections 2024-04-02T08:53:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.908+0000 7f8197e00700 1 --2- 172.21.15.89:0/1155510387 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f817006adb0 0x7f817006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.908+0000 7f8197e00700 1 --2- 172.21.15.89:0/1155510387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81900b2b30 0x7f81900c83f0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.908+0000 7f8197e00700 1 --2- 172.21.15.89:0/1155510387 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8190157f90 0x7f81901495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.908+0000 7f8197e00700 1 --2- 172.21.15.89:0/1155510387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81900b3960 0x7f81900aca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.908+0000 7f8197e00700 1 -- 172.21.15.89:0/1155510387 >> 172.21.15.89:0/1155510387 conn(0x7f81900b0d70 msgr2=0x7f819006e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.908+0000 7f8197e00700 1 -- 172.21.15.89:0/1155510387 shutdown_connections 2024-04-02T08:53:02.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:02.908+0000 7f8197e00700 1 -- 172.21.15.89:0/1155510387 wait complete. 2024-04-02T08:53:02.913 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:02.914+0000 7f4b2ce59240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T08:53:02.926 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T08:53:02.983 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:02.984+0000 7fb0bdd5f240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T08:53:03.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.125+0000 7fe435200700 1 Processor -- start 2024-04-02T08:53:03.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.125+0000 7fe435200700 1 -- start start 2024-04-02T08:53:03.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.127+0000 7fe435200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4300d2540 0x7fe4301687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:03.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.127+0000 7fe435200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe430161420 con 0x7fe43015d9d0 2024-04-02T08:53:03.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.127+0000 7fe435200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe430161700 con 0x7fe4300d16a0 2024-04-02T08:53:03.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.127+0000 7fe435200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300d4340 con 0x7fe4300d2540 2024-04-02T08:53:03.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.127+0000 7fe42de00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe43015d9d0 0x7fe4300d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45466/0 (socket says 172.21.15.89:45466) 2024-04-02T08:53:03.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.127+0000 7fe42de00700 1 -- 172.21.15.89:0/998678146 learned_addr learned my addr 172.21.15.89:0/998678146 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:03.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.127+0000 7fe42e800700 1 --2- 172.21.15.89:0/998678146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4300d2540 0x7fe4301687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:03.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.127+0000 7fe42ca00700 1 -- 172.21.15.89:0/998678146 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294009930 0 0) 0x7fe430161420 con 0x7fe43015d9d0 2024-04-02T08:53:03.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.127+0000 7fe42ca00700 1 -- 172.21.15.89:0/998678146 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4140035b0 con 0x7fe43015d9d0 2024-04-02T08:53:03.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.127+0000 7fe42e800700 1 -- 172.21.15.89:0/998678146 >> v1:172.21.15.113:6789/0 conn(0x7fe4300d16a0 legacy=0x7fe4300d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:03.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.127+0000 7fe42ca00700 1 -- 172.21.15.89:0/998678146 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190383393 0 0) 0x7fe430161700 con 0x7fe4300d16a0 2024-04-02T08:53:03.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.127+0000 7fe42e800700 1 -- 172.21.15.89:0/998678146 >> v1:172.21.15.89:6789/0 conn(0x7fe43015d9d0 legacy=0x7fe4300d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:03.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.128+0000 7fe42e800700 1 -- 172.21.15.89:0/998678146 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4301619e0 con 0x7fe4300d2540 2024-04-02T08:53:03.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.128+0000 7fe42e800700 1 --2- 172.21.15.89:0/998678146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4300d2540 0x7fe4301687b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe428009dc0 tx=0x7fe42800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51dbbdeb1bb70b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:03.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.128+0000 7fe42ca00700 1 -- 172.21.15.89:0/998678146 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42800e070 con 0x7fe4300d2540 2024-04-02T08:53:03.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.128+0000 7fe42ca00700 1 -- 172.21.15.89:0/998678146 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42800bb20 con 0x7fe4300d2540 2024-04-02T08:53:03.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.128+0000 7fe42ca00700 1 -- 172.21.15.89:0/998678146 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42800be90 con 0x7fe4300d2540 2024-04-02T08:53:03.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.129+0000 7fe435200700 1 -- 172.21.15.89:0/998678146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4300d2540 msgr2=0x7fe4301687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:03.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.129+0000 7fe435200700 1 --2- 172.21.15.89:0/998678146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4300d2540 0x7fe4301687b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe428009dc0 tx=0x7fe42800b330 comp rx=0 tx=0).stop 2024-04-02T08:53:03.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.129+0000 7fe435200700 1 -- 172.21.15.89:0/998678146 shutdown_connections 2024-04-02T08:53:03.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.129+0000 7fe435200700 1 --2- 172.21.15.89:0/998678146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4300d2540 0x7fe4301687b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.129+0000 7fe435200700 1 -- 172.21.15.89:0/998678146 >> 172.21.15.89:0/998678146 conn(0x7fe43008aff0 msgr2=0x7fe4300d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:03.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.129+0000 7fe435200700 1 -- 172.21.15.89:0/998678146 shutdown_connections 2024-04-02T08:53:03.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.129+0000 7fe435200700 1 -- 172.21.15.89:0/998678146 wait complete. 2024-04-02T08:53:03.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.131+0000 7fe435200700 1 Processor -- start 2024-04-02T08:53:03.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.131+0000 7fe435200700 1 -- start start 2024-04-02T08:53:03.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.132+0000 7fe435200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4300d16a0 0x7fe4300cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:03.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.132+0000 7fe435200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4300d2540 0x7fe4300b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:03.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.132+0000 7fe435200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe43015d9d0 0x7fe4300cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:03.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.132+0000 7fe435200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300cb7b0 con 0x7fe4300d16a0 2024-04-02T08:53:03.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.132+0000 7fe435200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300ccb70 con 0x7fe4300d2540 2024-04-02T08:53:03.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.132+0000 7fe435200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300cce50 con 0x7fe43015d9d0 2024-04-02T08:53:03.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.132+0000 7fe42de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4300d16a0 0x7fe4300cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:03.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.132+0000 7fe42de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4300d16a0 0x7fe4300cb270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38478/0 (socket says 172.21.15.89:38478) 2024-04-02T08:53:03.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.132+0000 7fe42de00700 1 -- 172.21.15.89:0/3648561680 learned_addr learned my addr 172.21.15.89:0/3648561680 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:03.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.132+0000 7fe42e800700 1 --2- 172.21.15.89:0/3648561680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe43015d9d0 0x7fe4300cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:03.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.132+0000 7fe42d400700 1 --2- 172.21.15.89:0/3648561680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4300d2540 0x7fe4300b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:03.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.133+0000 7fe42de00700 1 -- 172.21.15.89:0/3648561680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe43015d9d0 msgr2=0x7fe4300cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:03.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.133+0000 7fe42de00700 1 --2- 172.21.15.89:0/3648561680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe43015d9d0 0x7fe4300cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.133+0000 7fe42de00700 1 -- 172.21.15.89:0/3648561680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4300d2540 msgr2=0x7fe4300b13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:03.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.133+0000 7fe42de00700 1 --2- 172.21.15.89:0/3648561680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4300d2540 0x7fe4300b13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.133+0000 7fe42de00700 1 -- 172.21.15.89:0/3648561680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe428009a70 con 0x7fe4300d16a0 2024-04-02T08:53:03.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.133+0000 7fe42de00700 1 --2- 172.21.15.89:0/3648561680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4300d16a0 0x7fe4300cb270 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fe42000c420 tx=0x7fe42000c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:03.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.133+0000 7fe426000700 1 -- 172.21.15.89:0/3648561680 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe420005d70 con 0x7fe4300d16a0 2024-04-02T08:53:03.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.133+0000 7fe426000700 1 -- 172.21.15.89:0/3648561680 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42000ce30 con 0x7fe4300d16a0 2024-04-02T08:53:03.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.133+0000 7fe435200700 1 -- 172.21.15.89:0/3648561680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4300ca790 con 0x7fe4300d16a0 2024-04-02T08:53:03.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.133+0000 7fe426000700 1 -- 172.21.15.89:0/3648561680 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4200072d0 con 0x7fe4300d16a0 2024-04-02T08:53:03.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.133+0000 7fe435200700 1 -- 172.21.15.89:0/3648561680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4300cad50 con 0x7fe4300d16a0 2024-04-02T08:53:03.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.134+0000 7fe435200700 1 -- 172.21.15.89:0/3648561680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4300d1ca0 con 0x7fe4300d16a0 2024-04-02T08:53:03.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.135+0000 7fe426000700 1 -- 172.21.15.89:0/3648561680 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fe420007430 con 0x7fe4300d16a0 2024-04-02T08:53:03.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.137+0000 7fe426000700 1 --2- 172.21.15.89:0/3648561680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe41006adb0 0x7fe41006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:03.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.137+0000 7fe42d400700 1 -- 172.21.15.89:0/3648561680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe41006adb0 msgr2=0x7fe41006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:03.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.137+0000 7fe426000700 1 -- 172.21.15.89:0/3648561680 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fe420088910 con 0x7fe4300d16a0 2024-04-02T08:53:03.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.137+0000 7fe42d400700 1 --2- 172.21.15.89:0/3648561680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe41006adb0 0x7fe41006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.141+0000 7fe426000700 1 -- 172.21.15.89:0/3648561680 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe420053820 con 0x7fe4300d16a0 2024-04-02T08:53:03.168 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:03.169+0000 7f4b2ce59240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T08:53:03.254 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:03.255+0000 7fb0bdd5f240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T08:53:03.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.338+0000 7fe42d400700 1 -- 172.21.15.89:0/3648561680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe41006adb0 msgr2=0x7fe41006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:03.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.338+0000 7fe42d400700 1 --2- 172.21.15.89:0/3648561680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe41006adb0 0x7fe41006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:03.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.351+0000 7fe435200700 1 -- 172.21.15.89:0/3648561680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe4300ca5f0 con 0x7fe4300d16a0 2024-04-02T08:53:03.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.351+0000 7fe426000700 1 -- 172.21.15.89:0/3648561680 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v173) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe420017020 con 0x7fe4300d16a0 2024-04-02T08:53:03.351 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T08:53:03.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.354+0000 7fe435200700 1 -- 172.21.15.89:0/3648561680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe41006adb0 msgr2=0x7fe41006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:03.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.354+0000 7fe435200700 1 --2- 172.21.15.89:0/3648561680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe41006adb0 0x7fe41006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.354+0000 7fe435200700 1 -- 172.21.15.89:0/3648561680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4300d16a0 msgr2=0x7fe4300cb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:03.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.354+0000 7fe435200700 1 --2- 172.21.15.89:0/3648561680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4300d16a0 0x7fe4300cb270 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fe42000c420 tx=0x7fe42000c730 comp rx=0 tx=0).stop 2024-04-02T08:53:03.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.354+0000 7fe435200700 1 -- 172.21.15.89:0/3648561680 shutdown_connections 2024-04-02T08:53:03.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.354+0000 7fe435200700 1 --2- 172.21.15.89:0/3648561680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fe41006adb0 0x7fe41006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.354+0000 7fe435200700 1 --2- 172.21.15.89:0/3648561680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe43015d9d0 0x7fe4300cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.354+0000 7fe435200700 1 --2- 172.21.15.89:0/3648561680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4300d16a0 0x7fe4300cb270 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.354+0000 7fe435200700 1 --2- 172.21.15.89:0/3648561680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4300d2540 0x7fe4300b13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.354+0000 7fe435200700 1 -- 172.21.15.89:0/3648561680 >> 172.21.15.89:0/3648561680 conn(0x7fe43008aff0 msgr2=0x7fe4300b0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:03.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.354+0000 7fe435200700 1 -- 172.21.15.89:0/3648561680 shutdown_connections 2024-04-02T08:53:03.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.354+0000 7fe435200700 1 -- 172.21.15.89:0/3648561680 wait complete. 2024-04-02T08:53:03.372 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T08:53:03.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.570+0000 7f8b6fa00700 1 Processor -- start 2024-04-02T08:53:03.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.571+0000 7f8b6fa00700 1 -- start start 2024-04-02T08:53:03.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.572+0000 7f8b6fa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8b680d32f0 0x7f8b6815dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:03.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.572+0000 7f8b6fa00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8b680d0240 0x7f8b680d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:03.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.572+0000 7f8b6fa00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b680d0bf0 con 0x7f8b680ceb60 2024-04-02T08:53:03.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.572+0000 7f8b6fa00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b680cef40 con 0x7f8b680d0240 2024-04-02T08:53:03.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.572+0000 7f8b6fa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b680d3060 con 0x7f8b680d32f0 2024-04-02T08:53:03.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.572+0000 7f8b6d000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8b680ceb60 0x7f8b6815eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45468/0 (socket says 172.21.15.89:45468) 2024-04-02T08:53:03.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.572+0000 7f8b6d000700 1 -- 172.21.15.89:0/416210278 learned_addr learned my addr 172.21.15.89:0/416210278 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:03.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.572+0000 7f8b6da00700 1 --2- 172.21.15.89:0/416210278 >> v2:172.21.15.113:3300/0 conn(0x7f8b680d0240 0x7f8b680d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:03.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.572+0000 7f8b67e00700 1 --2- 172.21.15.89:0/416210278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8b680d32f0 0x7f8b6815dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:03.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 -- 172.21.15.89:0/416210278 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368745446 0 0) 0x7f8b5c001110 con 0x7f8b680ceb60 2024-04-02T08:53:03.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 -- 172.21.15.89:0/416210278 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b500035b0 con 0x7f8b680ceb60 2024-04-02T08:53:03.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 -- 172.21.15.89:0/416210278 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2907176216 0 0) 0x7f8b500035b0 con 0x7f8b680ceb60 2024-04-02T08:53:03.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 -- 172.21.15.89:0/416210278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8b680d32f0 msgr2=0x7f8b6815dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:03.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 --2- 172.21.15.89:0/416210278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8b680d32f0 0x7f8b6815dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 -- 172.21.15.89:0/416210278 >> v2:172.21.15.113:3300/0 conn(0x7f8b680d0240 msgr2=0x7f8b680d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:03.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 --2- 172.21.15.89:0/416210278 >> v2:172.21.15.113:3300/0 conn(0x7f8b680d0240 0x7f8b680d06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 -- 172.21.15.89:0/416210278 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6815e210 con 0x7f8b680ceb60 2024-04-02T08:53:03.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 -- 172.21.15.89:0/416210278 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f8b5c001800 con 0x7f8b680ceb60 2024-04-02T08:53:03.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 -- 172.21.15.89:0/416210278 >> v1:172.21.15.89:6789/0 conn(0x7f8b680ceb60 legacy=0x7f8b6815eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:03.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 --2- 172.21.15.89:0/416210278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b50003a30 0x7f8b50003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:03.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 --2- 172.21.15.89:0/416210278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b50004850 0x7f8b50004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:03.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 --2- 172.21.15.89:0/416210278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8b680d32f0 0x7f8b50005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:03.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 -- 172.21.15.89:0/416210278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b680d3060 con 0x7f8b50004850 2024-04-02T08:53:03.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 -- 172.21.15.89:0/416210278 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5000ba10 con 0x7f8b50003a30 2024-04-02T08:53:03.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67400700 1 -- 172.21.15.89:0/416210278 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5000bb80 con 0x7f8b680d32f0 2024-04-02T08:53:03.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b67e00700 1 --2- 172.21.15.89:0/416210278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b50004850 0x7f8b50004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:03.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b6da00700 1 --2- 172.21.15.89:0/416210278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8b680d32f0 0x7f8b50005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:03.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.573+0000 7f8b6d000700 1 --2- 172.21.15.89:0/416210278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b50003a30 0x7f8b50003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:03.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.574+0000 7f8b6da00700 1 -- 172.21.15.89:0/416210278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b50003a30 msgr2=0x7f8b50003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:03.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.574+0000 7f8b6da00700 1 --2- 172.21.15.89:0/416210278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b50003a30 0x7f8b50003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.574+0000 7f8b6da00700 1 -- 172.21.15.89:0/416210278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b50004850 msgr2=0x7f8b50004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:03.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.574+0000 7f8b6da00700 1 --2- 172.21.15.89:0/416210278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b50004850 0x7f8b50004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.574+0000 7f8b6da00700 1 -- 172.21.15.89:0/416210278 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b500035b0 con 0x7f8b680d32f0 2024-04-02T08:53:03.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.574+0000 7f8b67e00700 1 --2- 172.21.15.89:0/416210278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b50004850 0x7f8b50004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:03.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.574+0000 7f8b6da00700 1 --2- 172.21.15.89:0/416210278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8b680d32f0 0x7f8b50005210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8b60009fc0 tx=0x7f8b50005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bcf795fd515e5fd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:03.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.574+0000 7f8b67400700 1 -- 172.21.15.89:0/416210278 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6000ec30 con 0x7f8b680d32f0 2024-04-02T08:53:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.574+0000 7f8b67400700 1 -- 172.21.15.89:0/416210278 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b6000f040 con 0x7f8b680d32f0 2024-04-02T08:53:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.574+0000 7f8b67400700 1 -- 172.21.15.89:0/416210278 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b60014480 con 0x7f8b680d32f0 2024-04-02T08:53:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.575+0000 7f8b6fa00700 1 -- 172.21.15.89:0/416210278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8b680d32f0 msgr2=0x7f8b50005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.575+0000 7f8b6fa00700 1 --2- 172.21.15.89:0/416210278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8b680d32f0 0x7f8b50005210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8b60009fc0 tx=0x7f8b50005880 comp rx=0 tx=0).stop 2024-04-02T08:53:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.575+0000 7f8b6da00700 1 -- 172.21.15.89:0/416210278 reap_dead start 2024-04-02T08:53:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.575+0000 7f8b6fa00700 1 -- 172.21.15.89:0/416210278 shutdown_connections 2024-04-02T08:53:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.575+0000 7f8b6fa00700 1 -- 172.21.15.89:0/416210278 >> 172.21.15.89:0/416210278 conn(0x7f8b680664c0 msgr2=0x7f8b680668d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.575+0000 7f8b6fa00700 1 -- 172.21.15.89:0/416210278 shutdown_connections 2024-04-02T08:53:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.575+0000 7f8b6fa00700 1 -- 172.21.15.89:0/416210278 wait complete. 2024-04-02T08:53:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.577+0000 7f8b6fa00700 1 Processor -- start 2024-04-02T08:53:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.577+0000 7f8b6fa00700 1 -- start start 2024-04-02T08:53:03.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.578+0000 7f8b6fa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b680d32f0 0x7f8b680b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:03.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.578+0000 7f8b6fa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8b68159f10 0x7f8b680cb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.578+0000 7f8b6fa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b681f2bc0 0x7f8b681f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.578+0000 7f8b6fa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6815a390 con 0x7f8b681f2bc0 2024-04-02T08:53:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.578+0000 7f8b6fa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b680cb9a0 con 0x7f8b680d32f0 2024-04-02T08:53:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.578+0000 7f8b6fa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b680b3950 con 0x7f8b68159f10 2024-04-02T08:53:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.578+0000 7f8b6da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b681f2bc0 0x7f8b681f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.578+0000 7f8b6da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b681f2bc0 0x7f8b681f3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38502/0 (socket says 172.21.15.89:38502) 2024-04-02T08:53:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.578+0000 7f8b6da00700 1 -- 172.21.15.89:0/4122818781 learned_addr learned my addr 172.21.15.89:0/4122818781 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.578+0000 7f8b6d000700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b680d32f0 0x7f8b680b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.578+0000 7f8b67e00700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8b68159f10 0x7f8b680cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.579+0000 7f8b6d000700 1 -- 172.21.15.89:0/4122818781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8b68159f10 msgr2=0x7f8b680cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.579+0000 7f8b6d000700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8b68159f10 0x7f8b680cb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.579+0000 7f8b6d000700 1 -- 172.21.15.89:0/4122818781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b681f2bc0 msgr2=0x7f8b681f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:03.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.579+0000 7f8b6d000700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b681f2bc0 0x7f8b681f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.579+0000 7f8b6d000700 1 -- 172.21.15.89:0/4122818781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b680b3c30 con 0x7f8b680d32f0 2024-04-02T08:53:03.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.579+0000 7f8b6da00700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b681f2bc0 0x7f8b681f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:03.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.579+0000 7f8b6d000700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b680d32f0 0x7f8b680b32e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c007310 tx=0x7f8b5c007620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:03.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.579+0000 7f8b64c00700 1 -- 172.21.15.89:0/4122818781 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c001200 con 0x7f8b680d32f0 2024-04-02T08:53:03.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.579+0000 7f8b64c00700 1 -- 172.21.15.89:0/4122818781 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5c00f6c0 con 0x7f8b680d32f0 2024-04-02T08:53:03.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.579+0000 7f8b6fa00700 1 -- 172.21.15.89:0/4122818781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b680b3f10 con 0x7f8b680d32f0 2024-04-02T08:53:03.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.579+0000 7f8b64c00700 1 -- 172.21.15.89:0/4122818781 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c00fa70 con 0x7f8b680d32f0 2024-04-02T08:53:03.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.579+0000 7f8b6fa00700 1 -- 172.21.15.89:0/4122818781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b680b44d0 con 0x7f8b680d32f0 2024-04-02T08:53:03.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.580+0000 7f8b6fa00700 1 -- 172.21.15.89:0/4122818781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b68067720 con 0x7f8b680d32f0 2024-04-02T08:53:03.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.581+0000 7f8b64c00700 1 -- 172.21.15.89:0/4122818781 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f8b5c00fbd0 con 0x7f8b680d32f0 2024-04-02T08:53:03.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.582+0000 7f8b64c00700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8b4806adb0 0x7f8b4806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:03.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.582+0000 7f8b67e00700 1 -- 172.21.15.89:0/4122818781 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8b4806adb0 msgr2=0x7f8b4806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:03.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.582+0000 7f8b64c00700 1 -- 172.21.15.89:0/4122818781 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f8b5c089400 con 0x7f8b680d32f0 2024-04-02T08:53:03.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.582+0000 7f8b67e00700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8b4806adb0 0x7f8b4806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.587+0000 7f8b64c00700 1 -- 172.21.15.89:0/4122818781 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8b5c054260 con 0x7f8b680d32f0 2024-04-02T08:53:03.624 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:03.625+0000 7f4b2ce59240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T08:53:03.739 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:03.739+0000 7fb0bdd5f240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T08:53:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.783+0000 7f8b67e00700 1 -- 172.21.15.89:0/4122818781 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8b4806adb0 msgr2=0x7f8b4806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.783+0000 7f8b67e00700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8b4806adb0 0x7f8b4806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:03.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.799+0000 7f8b6fa00700 1 -- 172.21.15.89:0/4122818781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8b680d0bf0 con 0x7f8b680d32f0 2024-04-02T08:53:03.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.799+0000 7f8b64c00700 1 -- 172.21.15.89:0/4122818781 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v173) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8b680d0bf0 con 0x7f8b680d32f0 2024-04-02T08:53:03.799 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T08:53:03.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.802+0000 7f8b6fa00700 1 -- 172.21.15.89:0/4122818781 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8b4806adb0 msgr2=0x7f8b4806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:03.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.802+0000 7f8b6fa00700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8b4806adb0 0x7f8b4806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.802+0000 7f8b6fa00700 1 -- 172.21.15.89:0/4122818781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b680d32f0 msgr2=0x7f8b680b32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:03.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.802+0000 7f8b6fa00700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b680d32f0 0x7f8b680b32e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c007310 tx=0x7f8b5c007620 comp rx=0 tx=0).stop 2024-04-02T08:53:03.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.802+0000 7f8b6fa00700 1 -- 172.21.15.89:0/4122818781 shutdown_connections 2024-04-02T08:53:03.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.802+0000 7f8b6fa00700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8b4806adb0 0x7f8b4806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.802+0000 7f8b6fa00700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b680d32f0 0x7f8b680b32e0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.802+0000 7f8b6fa00700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b681f2bc0 0x7f8b681f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.802+0000 7f8b6fa00700 1 --2- 172.21.15.89:0/4122818781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8b68159f10 0x7f8b680cb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:03.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.802+0000 7f8b6fa00700 1 -- 172.21.15.89:0/4122818781 >> 172.21.15.89:0/4122818781 conn(0x7f8b680664c0 msgr2=0x7f8b6815b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:03.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.802+0000 7f8b6fa00700 1 -- 172.21.15.89:0/4122818781 shutdown_connections 2024-04-02T08:53:03.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:03.802+0000 7f8b6fa00700 1 -- 172.21.15.89:0/4122818781 wait complete. 2024-04-02T08:53:03.823 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T08:53:04.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.021+0000 7fc1d1a00700 1 Processor -- start 2024-04-02T08:53:04.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.022+0000 7fc1d1a00700 1 -- start start 2024-04-02T08:53:04.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.023+0000 7fc1d1a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc1cc0d16a0 0x7fc1cc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.023+0000 7fc1d1a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc1cc0d2370 0x7fc1cc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.023+0000 7fc1d1a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1cc161420 con 0x7fc1cc15d9d0 2024-04-02T08:53:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.023+0000 7fc1d1a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1cc0d2820 con 0x7fc1cc0d16a0 2024-04-02T08:53:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.023+0000 7fc1d1a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0d4340 con 0x7fc1cc0d2370 2024-04-02T08:53:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.024+0000 7fc1c3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc1cc0d16a0 0x7fc1cc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.024+0000 7fc1caa00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc1cc15d9d0 0x7fc1cc0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45478/0 (socket says 172.21.15.89:45478) 2024-04-02T08:53:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.024+0000 7fc1caa00700 1 -- 172.21.15.89:0/3968756046 learned_addr learned my addr 172.21.15.89:0/3968756046 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.024+0000 7fc1cb400700 1 --2- 172.21.15.89:0/3968756046 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc1cc0d2370 0x7fc1cc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.024+0000 7fc1c3e00700 1 -- 172.21.15.89:0/3968756046 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc1cc0d2370 msgr2=0x7fc1cc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.024+0000 7fc1c3e00700 1 --2- 172.21.15.89:0/3968756046 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc1cc0d2370 0x7fc1cc0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:04.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.024+0000 7fc1c3e00700 1 -- 172.21.15.89:0/3968756046 >> v1:172.21.15.89:6789/0 conn(0x7fc1cc15d9d0 legacy=0x7fc1cc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:04.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.024+0000 7fc1c3e00700 1 -- 172.21.15.89:0/3968756046 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc161700 con 0x7fc1cc0d16a0 2024-04-02T08:53:04.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.025+0000 7fc1c3e00700 1 --2- 172.21.15.89:0/3968756046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc0d16a0 0x7fc1cc0d1e30 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009820 tx=0x7fc1b4009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ad624547685181 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:04.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.025+0000 7fc1ca000700 1 -- 172.21.15.89:0/3968756046 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4005e30 con 0x7fc1cc0d16a0 2024-04-02T08:53:04.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.025+0000 7fc1ca000700 1 -- 172.21.15.89:0/3968756046 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b4006030 con 0x7fc1cc0d16a0 2024-04-02T08:53:04.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.025+0000 7fc1ca000700 1 -- 172.21.15.89:0/3968756046 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b40063c0 con 0x7fc1cc0d16a0 2024-04-02T08:53:04.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.025+0000 7fc1d1a00700 1 -- 172.21.15.89:0/3968756046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc0d16a0 msgr2=0x7fc1cc0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.025+0000 7fc1d1a00700 1 --2- 172.21.15.89:0/3968756046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc0d16a0 0x7fc1cc0d1e30 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009820 tx=0x7fc1b4009b30 comp rx=0 tx=0).stop 2024-04-02T08:53:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.026+0000 7fc1d1a00700 1 -- 172.21.15.89:0/3968756046 shutdown_connections 2024-04-02T08:53:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.026+0000 7fc1d1a00700 1 --2- 172.21.15.89:0/3968756046 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc1cc0d2370 0x7fc1cc0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.026+0000 7fc1d1a00700 1 --2- 172.21.15.89:0/3968756046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc0d16a0 0x7fc1cc0d1e30 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.026+0000 7fc1d1a00700 1 -- 172.21.15.89:0/3968756046 >> 172.21.15.89:0/3968756046 conn(0x7fc1cc08aff0 msgr2=0x7fc1cc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.026+0000 7fc1d1a00700 1 -- 172.21.15.89:0/3968756046 shutdown_connections 2024-04-02T08:53:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.026+0000 7fc1d1a00700 1 -- 172.21.15.89:0/3968756046 wait complete. 2024-04-02T08:53:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.027+0000 7fc1d1a00700 1 Processor -- start 2024-04-02T08:53:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.027+0000 7fc1d1a00700 1 -- start start 2024-04-02T08:53:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.028+0000 7fc1d1a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc1cc0d16a0 0x7fc1cc1e92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.028+0000 7fc1d1a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1cc0d2370 0x7fc1cc0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.028+0000 7fc1d1a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc15d9d0 0x7fc1cc14f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.028+0000 7fc1d1a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0cc970 con 0x7fc1cc15d9d0 2024-04-02T08:53:04.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.028+0000 7fc1d1a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0cb420 con 0x7fc1cc0d2370 2024-04-02T08:53:04.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.028+0000 7fc1d1a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0cae60 con 0x7fc1cc0d16a0 2024-04-02T08:53:04.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.029+0000 7fc1cb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc15d9d0 0x7fc1cc14f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:04.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.029+0000 7fc1cb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc15d9d0 0x7fc1cc14f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38518/0 (socket says 172.21.15.89:38518) 2024-04-02T08:53:04.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.029+0000 7fc1cb400700 1 -- 172.21.15.89:0/1110756461 learned_addr learned my addr 172.21.15.89:0/1110756461 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:04.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.029+0000 7fc1caa00700 1 --2- 172.21.15.89:0/1110756461 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc1cc0d16a0 0x7fc1cc1e92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:04.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.029+0000 7fc1c3e00700 1 --2- 172.21.15.89:0/1110756461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1cc0d2370 0x7fc1cc0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:04.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.029+0000 7fc1cb400700 1 -- 172.21.15.89:0/1110756461 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc1cc0d16a0 msgr2=0x7fc1cc1e92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:04.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.029+0000 7fc1cb400700 1 --2- 172.21.15.89:0/1110756461 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc1cc0d16a0 0x7fc1cc1e92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:04.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.029+0000 7fc1cb400700 1 -- 172.21.15.89:0/1110756461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1cc0d2370 msgr2=0x7fc1cc0ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:04.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.029+0000 7fc1cb400700 1 --2- 172.21.15.89:0/1110756461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1cc0d2370 0x7fc1cc0ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:04.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.029+0000 7fc1cb400700 1 -- 172.21.15.89:0/1110756461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b4009430 con 0x7fc1cc15d9d0 2024-04-02T08:53:04.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.029+0000 7fc1cb400700 1 --2- 172.21.15.89:0/1110756461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc15d9d0 0x7fc1cc14f6a0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400efc0 tx=0x7fc1c400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:04.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.030+0000 7fc1c2a00700 1 -- 172.21.15.89:0/1110756461 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4009ea0 con 0x7fc1cc15d9d0 2024-04-02T08:53:04.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.030+0000 7fc1c2a00700 1 -- 172.21.15.89:0/1110756461 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c400f040 con 0x7fc1cc15d9d0 2024-04-02T08:53:04.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.030+0000 7fc1c2a00700 1 -- 172.21.15.89:0/1110756461 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c400ced0 con 0x7fc1cc15d9d0 2024-04-02T08:53:04.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.030+0000 7fc1d1a00700 1 -- 172.21.15.89:0/1110756461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1cc0cb140 con 0x7fc1cc15d9d0 2024-04-02T08:53:04.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.030+0000 7fc1d1a00700 1 -- 172.21.15.89:0/1110756461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc0cab30 con 0x7fc1cc15d9d0 2024-04-02T08:53:04.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.031+0000 7fc1d1a00700 1 -- 172.21.15.89:0/1110756461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1cc0d1ca0 con 0x7fc1cc15d9d0 2024-04-02T08:53:04.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.032+0000 7fc1c2a00700 1 -- 172.21.15.89:0/1110756461 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fc1c4014480 con 0x7fc1cc15d9d0 2024-04-02T08:53:04.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.033+0000 7fc1c2a00700 1 --2- 172.21.15.89:0/1110756461 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1a806adb0 0x7fc1a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:04.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.033+0000 7fc1c2a00700 1 -- 172.21.15.89:0/1110756461 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fc1c408a1b0 con 0x7fc1cc15d9d0 2024-04-02T08:53:04.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.033+0000 7fc1caa00700 1 -- 172.21.15.89:0/1110756461 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1a806adb0 msgr2=0x7fc1a806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:04.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.033+0000 7fc1caa00700 1 --2- 172.21.15.89:0/1110756461 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1a806adb0 0x7fc1a806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:04.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.037+0000 7fc1c2a00700 1 -- 172.21.15.89:0/1110756461 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc1c4054de0 con 0x7fc1cc15d9d0 2024-04-02T08:53:04.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.234+0000 7fc1caa00700 1 -- 172.21.15.89:0/1110756461 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1a806adb0 msgr2=0x7fc1a806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:04.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.234+0000 7fc1caa00700 1 --2- 172.21.15.89:0/1110756461 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1a806adb0 0x7fc1a806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:04.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.262+0000 7fc1d1a00700 1 -- 172.21.15.89:0/1110756461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc1cc067720 con 0x7fc1cc15d9d0 2024-04-02T08:53:04.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.263+0000 7fc1c2a00700 1 -- 172.21.15.89:0/1110756461 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v173) v1 ==== 80+0+458 (secure 0 0 0) 0x7fc1c40181e0 con 0x7fc1cc15d9d0 2024-04-02T08:53:04.262 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3680638162 2024-04-03T08:52:58.629596+0000 2024-04-02T08:53:04.262 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/263272397 2024-04-03T08:52:58.629596+0000 2024-04-02T08:53:04.262 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1641401996 2024-04-03T08:52:58.629596+0000 2024-04-02T08:53:04.262 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/419225467 2024-04-03T08:52:58.629596+0000 2024-04-02T08:53:04.263 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/43357 2024-04-03T08:52:58.629596+0000 2024-04-02T08:53:04.263 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/43357 2024-04-03T08:52:58.629596+0000 2024-04-02T08:53:04.263 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2329039936 2024-04-03T08:52:58.629596+0000 2024-04-02T08:53:04.263 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3986748726 2024-04-03T08:52:58.629596+0000 2024-04-02T08:53:04.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.265+0000 7fc1d1a00700 1 -- 172.21.15.89:0/1110756461 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1a806adb0 msgr2=0x7fc1a806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:04.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.265+0000 7fc1d1a00700 1 --2- 172.21.15.89:0/1110756461 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1a806adb0 0x7fc1a806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:04.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.265+0000 7fc1d1a00700 1 -- 172.21.15.89:0/1110756461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc15d9d0 msgr2=0x7fc1cc14f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:04.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.265+0000 7fc1d1a00700 1 --2- 172.21.15.89:0/1110756461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc15d9d0 0x7fc1cc14f6a0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400efc0 tx=0x7fc1c400c3a0 comp rx=0 tx=0).stop 2024-04-02T08:53:04.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.265+0000 7fc1d1a00700 1 -- 172.21.15.89:0/1110756461 shutdown_connections 2024-04-02T08:53:04.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.265+0000 7fc1d1a00700 1 --2- 172.21.15.89:0/1110756461 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc1a806adb0 0x7fc1a806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:04.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.265+0000 7fc1d1a00700 1 --2- 172.21.15.89:0/1110756461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc15d9d0 0x7fc1cc14f6a0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:04.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.265+0000 7fc1d1a00700 1 --2- 172.21.15.89:0/1110756461 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc1cc0d16a0 0x7fc1cc1e92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:04.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.265+0000 7fc1d1a00700 1 --2- 172.21.15.89:0/1110756461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1cc0d2370 0x7fc1cc0ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:04.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.265+0000 7fc1d1a00700 1 -- 172.21.15.89:0/1110756461 >> 172.21.15.89:0/1110756461 conn(0x7fc1cc08aff0 msgr2=0x7fc1cc0b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:04.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.266+0000 7fc1d1a00700 1 -- 172.21.15.89:0/1110756461 shutdown_connections 2024-04-02T08:53:04.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.266+0000 7fc1d1a00700 1 -- 172.21.15.89:0/1110756461 wait complete. 2024-04-02T08:53:04.266 INFO:teuthology.orchestra.run.smithi089.stderr:listed 8 entries 2024-04-02T08:53:04.285 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T08:53:04.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:53:04.285 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T08:53:04.338 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:04.339+0000 7f4b2ce59240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T08:53:04.358 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T08:53:04.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:53:04.358 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T08:53:04.403 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T08:53:04.403 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:53:04.434 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:04.434+0000 7f4b2ce59240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T08:53:04.448 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T08:53:04.448 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T08:53:04.448 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:53:04.492 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T08:53:04.492 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T08:53:04.492 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:53:04.494 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:04.494+0000 7fb0bdd5f240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T08:53:04.537 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T08:53:04.537 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T08:53:04.594 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:04.595+0000 7fb0bdd5f240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T08:53:04.600 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:04.602+0000 7f4b2ce59240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T08:53:04.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.738+0000 7fa56f400700 1 Processor -- start 2024-04-02T08:53:04.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.738+0000 7fa56f400700 1 -- start start 2024-04-02T08:53:04.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.739+0000 7fa56f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa5680ac7d0 0x7fa568166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:04.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.739+0000 7fa56f400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56815fd50 con 0x7fa56815b5d0 2024-04-02T08:53:04.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.739+0000 7fa56f400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56815f140 con 0x7fa5680cb3f0 2024-04-02T08:53:04.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.739+0000 7fa56f400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa56815b9b0 con 0x7fa5680ac7d0 2024-04-02T08:53:04.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.740+0000 7fa56d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa5680ac7d0 0x7fa568166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:04.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.740+0000 7fa567e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fa5680cb3f0 0x7fa5680ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:51956/0 (socket says 172.21.15.89:51956) 2024-04-02T08:53:04.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.740+0000 7fa567e00700 1 -- 172.21.15.89:0/4269955228 learned_addr learned my addr 172.21.15.89:0/4269955228 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:04.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.740+0000 7fa567400700 1 -- 172.21.15.89:0/4269955228 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560145911 0 0) 0x7fa56815f140 con 0x7fa5680cb3f0 2024-04-02T08:53:04.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.740+0000 7fa567400700 1 -- 172.21.15.89:0/4269955228 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5500035b0 con 0x7fa5680cb3f0 2024-04-02T08:53:04.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.740+0000 7fa567400700 1 -- 172.21.15.89:0/4269955228 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513298833 0 0) 0x7fa56815fd50 con 0x7fa56815b5d0 2024-04-02T08:53:04.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.740+0000 7fa56d400700 1 -- 172.21.15.89:0/4269955228 >> v1:172.21.15.113:6789/0 conn(0x7fa5680cb3f0 legacy=0x7fa5680ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:04.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.740+0000 7fa56d400700 1 -- 172.21.15.89:0/4269955228 >> v1:172.21.15.89:6789/0 conn(0x7fa56815b5d0 legacy=0x7fa5680ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:04.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.740+0000 7fa56d400700 1 -- 172.21.15.89:0/4269955228 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56815f420 con 0x7fa5680ac7d0 2024-04-02T08:53:04.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.741+0000 7fa56d400700 1 --2- 172.21.15.89:0/4269955228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5680ac7d0 0x7fa568166430 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fa560009d90 tx=0x7fa56000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab5119ee05ddf32 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:04.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.741+0000 7fa567400700 1 -- 172.21.15.89:0/4269955228 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56000e070 con 0x7fa5680ac7d0 2024-04-02T08:53:04.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.741+0000 7fa567400700 1 -- 172.21.15.89:0/4269955228 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56000bb20 con 0x7fa5680ac7d0 2024-04-02T08:53:04.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.741+0000 7fa567400700 1 -- 172.21.15.89:0/4269955228 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56000bed0 con 0x7fa5680ac7d0 2024-04-02T08:53:04.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.741+0000 7fa56f400700 1 -- 172.21.15.89:0/4269955228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5680ac7d0 msgr2=0x7fa568166430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:04.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.741+0000 7fa56f400700 1 --2- 172.21.15.89:0/4269955228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5680ac7d0 0x7fa568166430 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fa560009d90 tx=0x7fa56000b330 comp rx=0 tx=0).stop 2024-04-02T08:53:04.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.742+0000 7fa56f400700 1 -- 172.21.15.89:0/4269955228 shutdown_connections 2024-04-02T08:53:04.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.742+0000 7fa56f400700 1 --2- 172.21.15.89:0/4269955228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5680ac7d0 0x7fa568166430 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:04.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.742+0000 7fa56f400700 1 -- 172.21.15.89:0/4269955228 >> 172.21.15.89:0/4269955228 conn(0x7fa5680cf190 msgr2=0x7fa5680cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:04.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.742+0000 7fa56f400700 1 -- 172.21.15.89:0/4269955228 shutdown_connections 2024-04-02T08:53:04.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.742+0000 7fa56f400700 1 -- 172.21.15.89:0/4269955228 wait complete. 2024-04-02T08:53:04.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.744+0000 7fa56f400700 1 Processor -- start 2024-04-02T08:53:04.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.744+0000 7fa56f400700 1 -- start start 2024-04-02T08:53:04.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.745+0000 7fa56f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5680ac7d0 0x7fa5680b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:04.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.745+0000 7fa56f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56815b5d0 0x7fa568098b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:04.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.745+0000 7fa56f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5680cbba0 0x7fa5680c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:04.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.745+0000 7fa56f400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5680b3500 con 0x7fa56815b5d0 2024-04-02T08:53:04.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.745+0000 7fa56f400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5680b37e0 con 0x7fa5680ac7d0 2024-04-02T08:53:04.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.745+0000 7fa56f400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5680b3ac0 con 0x7fa5680cbba0 2024-04-02T08:53:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.746+0000 7fa567e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56815b5d0 0x7fa568098b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.746+0000 7fa567e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56815b5d0 0x7fa568098b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38542/0 (socket says 172.21.15.89:38542) 2024-04-02T08:53:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.746+0000 7fa567e00700 1 -- 172.21.15.89:0/1072985958 learned_addr learned my addr 172.21.15.89:0/1072985958 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.746+0000 7fa56ca00700 1 --2- 172.21.15.89:0/1072985958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5680ac7d0 0x7fa5680b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.746+0000 7fa56d400700 1 --2- 172.21.15.89:0/1072985958 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5680cbba0 0x7fa5680c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.746+0000 7fa567e00700 1 -- 172.21.15.89:0/1072985958 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5680cbba0 msgr2=0x7fa5680c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.746+0000 7fa567e00700 1 --2- 172.21.15.89:0/1072985958 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5680cbba0 0x7fa5680c9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.746+0000 7fa567e00700 1 -- 172.21.15.89:0/1072985958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5680ac7d0 msgr2=0x7fa5680b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.746+0000 7fa567e00700 1 --2- 172.21.15.89:0/1072985958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5680ac7d0 0x7fa5680b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.746+0000 7fa567e00700 1 -- 172.21.15.89:0/1072985958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa560009a70 con 0x7fa56815b5d0 2024-04-02T08:53:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.747+0000 7fa567e00700 1 --2- 172.21.15.89:0/1072985958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56815b5d0 0x7fa568098b40 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fa55400c700 tx=0x7fa55400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.747+0000 7fa564c00700 1 -- 172.21.15.89:0/1072985958 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa554005d70 con 0x7fa56815b5d0 2024-04-02T08:53:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.747+0000 7fa564c00700 1 -- 172.21.15.89:0/1072985958 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5540070d0 con 0x7fa56815b5d0 2024-04-02T08:53:04.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.747+0000 7fa564c00700 1 -- 172.21.15.89:0/1072985958 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa554007460 con 0x7fa56815b5d0 2024-04-02T08:53:04.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.747+0000 7fa56f400700 1 -- 172.21.15.89:0/1072985958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5680b3da0 con 0x7fa56815b5d0 2024-04-02T08:53:04.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.747+0000 7fa56f400700 1 -- 172.21.15.89:0/1072985958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa568158b20 con 0x7fa56815b5d0 2024-04-02T08:53:04.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.748+0000 7fa56f400700 1 -- 172.21.15.89:0/1072985958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa568066ff0 con 0x7fa56815b5d0 2024-04-02T08:53:04.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.750+0000 7fa564c00700 1 -- 172.21.15.89:0/1072985958 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fa5540075c0 con 0x7fa56815b5d0 2024-04-02T08:53:04.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.751+0000 7fa564c00700 1 --2- 172.21.15.89:0/1072985958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa54806adb0 0x7fa54806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:04.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.751+0000 7fa564c00700 1 -- 172.21.15.89:0/1072985958 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fa554088c20 con 0x7fa56815b5d0 2024-04-02T08:53:04.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.751+0000 7fa56ca00700 1 -- 172.21.15.89:0/1072985958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa54806adb0 msgr2=0x7fa54806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:04.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.751+0000 7fa56ca00700 1 --2- 172.21.15.89:0/1072985958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa54806adb0 0x7fa54806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:04.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.755+0000 7fa564c00700 1 -- 172.21.15.89:0/1072985958 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa5540539e0 con 0x7fa56815b5d0 2024-04-02T08:53:04.767 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:04.767+0000 7fb0bdd5f240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T08:53:04.813 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:04.814+0000 7f4b2ce59240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T08:53:04.904 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:04.905+0000 7f4b2ce59240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T08:53:04.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.952+0000 7fa56ca00700 1 -- 172.21.15.89:0/1072985958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa54806adb0 msgr2=0x7fa54806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:04.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:04.952+0000 7fa56ca00700 1 --2- 172.21.15.89:0/1072985958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa54806adb0 0x7fa54806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:04.986 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:04.987+0000 7fb0bdd5f240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T08:53:05.002 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:05.003+0000 7f4b2ce59240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T08:53:05.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.014+0000 7fa56f400700 1 -- 172.21.15.89:0/1072985958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa568158d70 con 0x7fa56815b5d0 2024-04-02T08:53:05.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.015+0000 7fa564c00700 1 -- 172.21.15.89:0/1072985958 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4738 (secure 0 0 0) 0x7fa554017020 con 0x7fa56815b5d0 2024-04-02T08:53:05.014 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:05.014 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8881,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/547244866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":547244866},{"type":"v1","addr":"172.21.15.113:6834","nonce":547244866}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":8884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2822200553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2822200553},{"type":"v1","addr":"172.21.15.113:6835","nonce":2822200553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/342426898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":342426898},{"type":"v1","addr":"172.21.15.89:6835","nonce":342426898}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T08:52:42.484905+0000","modified":"2024-04-02T08:52:45.498450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9534},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9534":{"gid":9534,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2812511535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2812511535},{"type":"v1","addr":"172.21.15.89:6837","nonce":2812511535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-02T08:53:05.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.017+0000 7fa56f400700 1 -- 172.21.15.89:0/1072985958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa54806adb0 msgr2=0x7fa54806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:05.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.017+0000 7fa56f400700 1 --2- 172.21.15.89:0/1072985958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa54806adb0 0x7fa54806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.017+0000 7fa56f400700 1 -- 172.21.15.89:0/1072985958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56815b5d0 msgr2=0x7fa568098b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:05.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.017+0000 7fa56f400700 1 --2- 172.21.15.89:0/1072985958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56815b5d0 0x7fa568098b40 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fa55400c700 tx=0x7fa55400ca10 comp rx=0 tx=0).stop 2024-04-02T08:53:05.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.017+0000 7fa56f400700 1 -- 172.21.15.89:0/1072985958 shutdown_connections 2024-04-02T08:53:05.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.017+0000 7fa56f400700 1 --2- 172.21.15.89:0/1072985958 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa54806adb0 0x7fa54806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.017+0000 7fa56f400700 1 --2- 172.21.15.89:0/1072985958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5680ac7d0 0x7fa5680b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.017+0000 7fa56f400700 1 --2- 172.21.15.89:0/1072985958 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5680cbba0 0x7fa5680c9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.017+0000 7fa56f400700 1 --2- 172.21.15.89:0/1072985958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56815b5d0 0x7fa568098b40 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.017+0000 7fa56f400700 1 -- 172.21.15.89:0/1072985958 >> 172.21.15.89:0/1072985958 conn(0x7fa5680cf190 msgr2=0x7fa56815e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:05.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.017+0000 7fa56f400700 1 -- 172.21.15.89:0/1072985958 shutdown_connections 2024-04-02T08:53:05.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.017+0000 7fa56f400700 1 -- 172.21.15.89:0/1072985958 wait complete. 2024-04-02T08:53:05.018 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 81 2024-04-02T08:53:05.037 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:53:05.089 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:05.090+0000 7fb0bdd5f240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T08:53:05.099 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:05.100+0000 7f4b2ce59240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T08:53:05.194 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:05.195+0000 7fb0bdd5f240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T08:53:05.195 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:05.197+0000 7f4b2ce59240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T08:53:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.235+0000 7fd77f200700 1 Processor -- start 2024-04-02T08:53:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.235+0000 7fd77f200700 1 -- start start 2024-04-02T08:53:05.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.237+0000 7fd77f200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd7780b3af0 0x7fd77815a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:05.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.237+0000 7fd77f200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd7780709c0 0x7fd778070e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:05.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.237+0000 7fd77f200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7780b2180 con 0x7fd778072580 2024-04-02T08:53:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.237+0000 7fd77f200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd77815b100 con 0x7fd7780709c0 2024-04-02T08:53:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.237+0000 7fd77f200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77815b3e0 con 0x7fd7780b3af0 2024-04-02T08:53:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.238+0000 7fd777400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd778072580 0x7fd7780b2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45490/0 (socket says 172.21.15.89:45490) 2024-04-02T08:53:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.238+0000 7fd777400700 1 -- 172.21.15.89:0/40711687 learned_addr learned my addr 172.21.15.89:0/40711687 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.238+0000 7fd777e00700 1 --2- 172.21.15.89:0/40711687 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd7780b3af0 0x7fd77815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.238+0000 7fd77d200700 1 --2- 172.21.15.89:0/40711687 >> v2:172.21.15.113:3300/0 conn(0x7fd7780709c0 0x7fd778070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.238+0000 7fd776a00700 1 -- 172.21.15.89:0/40711687 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189376950 0 0) 0x7fd7780b2180 con 0x7fd778072580 2024-04-02T08:53:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.238+0000 7fd776a00700 1 -- 172.21.15.89:0/40711687 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd75c0035b0 con 0x7fd778072580 2024-04-02T08:53:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.238+0000 7fd777e00700 1 -- 172.21.15.89:0/40711687 >> v2:172.21.15.113:3300/0 conn(0x7fd7780709c0 msgr2=0x7fd778070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.238+0000 7fd777e00700 1 --2- 172.21.15.89:0/40711687 >> v2:172.21.15.113:3300/0 conn(0x7fd7780709c0 0x7fd778070e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.238+0000 7fd777e00700 1 -- 172.21.15.89:0/40711687 >> v1:172.21.15.89:6789/0 conn(0x7fd778072580 legacy=0x7fd7780b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.238+0000 7fd777e00700 1 -- 172.21.15.89:0/40711687 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7780b23f0 con 0x7fd7780b3af0 2024-04-02T08:53:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.239+0000 7fd777e00700 1 --2- 172.21.15.89:0/40711687 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd7780b3af0 0x7fd77815a860 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd768009840 tx=0x7fd768009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b20cc7ac81e1c130 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.239+0000 7fd776a00700 1 -- 172.21.15.89:0/40711687 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768011830 con 0x7fd7780b3af0 2024-04-02T08:53:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.239+0000 7fd776a00700 1 -- 172.21.15.89:0/40711687 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd768011990 con 0x7fd7780b3af0 2024-04-02T08:53:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.239+0000 7fd776a00700 1 -- 172.21.15.89:0/40711687 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768011d20 con 0x7fd7780b3af0 2024-04-02T08:53:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.240+0000 7fd77f200700 1 -- 172.21.15.89:0/40711687 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd7780b3af0 msgr2=0x7fd77815a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.240+0000 7fd77f200700 1 --2- 172.21.15.89:0/40711687 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd7780b3af0 0x7fd77815a860 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd768009840 tx=0x7fd768009b50 comp rx=0 tx=0).stop 2024-04-02T08:53:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.240+0000 7fd77f200700 1 -- 172.21.15.89:0/40711687 shutdown_connections 2024-04-02T08:53:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.240+0000 7fd77f200700 1 --2- 172.21.15.89:0/40711687 >> v2:172.21.15.113:3300/0 conn(0x7fd7780709c0 0x7fd778070e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.240+0000 7fd77f200700 1 --2- 172.21.15.89:0/40711687 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd7780b3af0 0x7fd77815a860 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.240+0000 7fd77f200700 1 -- 172.21.15.89:0/40711687 >> 172.21.15.89:0/40711687 conn(0x7fd7780cf190 msgr2=0x7fd7780cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.240+0000 7fd77f200700 1 -- 172.21.15.89:0/40711687 shutdown_connections 2024-04-02T08:53:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.240+0000 7fd77f200700 1 -- 172.21.15.89:0/40711687 wait complete. 2024-04-02T08:53:05.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.241+0000 7fd77f200700 1 Processor -- start 2024-04-02T08:53:05.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.242+0000 7fd77f200700 1 -- start start 2024-04-02T08:53:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.242+0000 7fd77f200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7780709c0 0x7fd778151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd77f200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd778072580 0x7fd7781f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd77f200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd7780b3af0 0x7fd7780adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd77f200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7781535a0 con 0x7fd778072580 2024-04-02T08:53:05.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd77f200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd778152cb0 con 0x7fd7780709c0 2024-04-02T08:53:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd77f200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd778152f50 con 0x7fd7780b3af0 2024-04-02T08:53:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd777400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd778072580 0x7fd7781f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd777400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd778072580 0x7fd7781f40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38548/0 (socket says 172.21.15.89:38548) 2024-04-02T08:53:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd777400700 1 -- 172.21.15.89:0/3853152248 learned_addr learned my addr 172.21.15.89:0/3853152248 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd77d200700 1 --2- 172.21.15.89:0/3853152248 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd7780b3af0 0x7fd7780adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd777e00700 1 --2- 172.21.15.89:0/3853152248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7780709c0 0x7fd778151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd777400700 1 -- 172.21.15.89:0/3853152248 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd7780b3af0 msgr2=0x7fd7780adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd777400700 1 --2- 172.21.15.89:0/3853152248 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd7780b3af0 0x7fd7780adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd777400700 1 -- 172.21.15.89:0/3853152248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7780709c0 msgr2=0x7fd778151490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd777400700 1 --2- 172.21.15.89:0/3853152248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7780709c0 0x7fd778151490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.243+0000 7fd777400700 1 -- 172.21.15.89:0/3853152248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7680094a0 con 0x7fd778072580 2024-04-02T08:53:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.244+0000 7fd777400700 1 --2- 172.21.15.89:0/3853152248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd778072580 0x7fd7781f40f0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fd76400c700 tx=0x7fd76400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.244+0000 7fd76fe00700 1 -- 172.21.15.89:0/3853152248 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd764005d70 con 0x7fd778072580 2024-04-02T08:53:05.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.244+0000 7fd76fe00700 1 -- 172.21.15.89:0/3853152248 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7640070d0 con 0x7fd778072580 2024-04-02T08:53:05.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.244+0000 7fd77f200700 1 -- 172.21.15.89:0/3853152248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd778153230 con 0x7fd778072580 2024-04-02T08:53:05.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.244+0000 7fd76fe00700 1 -- 172.21.15.89:0/3853152248 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd764007460 con 0x7fd778072580 2024-04-02T08:53:05.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.244+0000 7fd77f200700 1 -- 172.21.15.89:0/3853152248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7781f4630 con 0x7fd778072580 2024-04-02T08:53:05.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.245+0000 7fd77f200700 1 -- 172.21.15.89:0/3853152248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7781f48b0 con 0x7fd778072580 2024-04-02T08:53:05.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.246+0000 7fd76fe00700 1 -- 172.21.15.89:0/3853152248 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fd7640075c0 con 0x7fd778072580 2024-04-02T08:53:05.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.247+0000 7fd76fe00700 1 --2- 172.21.15.89:0/3853152248 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd75806adb0 0x7fd75806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:05.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.247+0000 7fd76fe00700 1 -- 172.21.15.89:0/3853152248 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fd764088c70 con 0x7fd778072580 2024-04-02T08:53:05.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.247+0000 7fd777e00700 1 -- 172.21.15.89:0/3853152248 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd75806adb0 msgr2=0x7fd75806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:05.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.247+0000 7fd777e00700 1 --2- 172.21.15.89:0/3853152248 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd75806adb0 0x7fd75806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:05.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.251+0000 7fd76fe00700 1 -- 172.21.15.89:0/3853152248 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd7640539a0 con 0x7fd778072580 2024-04-02T08:53:05.300 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:05.300+0000 7fb0bdd5f240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T08:53:05.400 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:05.400+0000 7fb0bdd5f240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T08:53:05.405 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:05.407+0000 7f4b2ce59240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T08:53:05.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.448+0000 7fd777e00700 1 -- 172.21.15.89:0/3853152248 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd75806adb0 msgr2=0x7fd75806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:05.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.448+0000 7fd777e00700 1 --2- 172.21.15.89:0/3853152248 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd75806adb0 0x7fd75806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:05.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.458+0000 7fd77f200700 1 -- 172.21.15.89:0/3853152248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd778067720 con 0x7fd778072580 2024-04-02T08:53:05.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.459+0000 7fd76fe00700 1 -- 172.21.15.89:0/3853152248 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) v1 ==== 74+0+17243 (secure 0 0 0) 0x7fd764017020 con 0x7fd778072580 2024-04-02T08:53:05.459 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:05.459 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":173,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:52:58.629629+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-02T08:52:40.806770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-02T08:52:41.463828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/3680638162":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/263272397":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/1641401996":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/419225467":"2024-04-03T08:52:58.629596+0000","172.21.15.89:6833/43357":"2024-04-03T08:52:58.629596+0000","172.21.15.89:6832/43357":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/2329039936":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/3986748726":"2024-04-03T08:52:58.629596+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:53:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.462+0000 7fd77f200700 1 -- 172.21.15.89:0/3853152248 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd75806adb0 msgr2=0x7fd75806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.462+0000 7fd77f200700 1 --2- 172.21.15.89:0/3853152248 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd75806adb0 0x7fd75806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.462+0000 7fd77f200700 1 -- 172.21.15.89:0/3853152248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd778072580 msgr2=0x7fd7781f40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.462+0000 7fd77f200700 1 --2- 172.21.15.89:0/3853152248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd778072580 0x7fd7781f40f0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fd76400c700 tx=0x7fd76400ca10 comp rx=0 tx=0).stop 2024-04-02T08:53:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.462+0000 7fd77f200700 1 -- 172.21.15.89:0/3853152248 shutdown_connections 2024-04-02T08:53:05.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.462+0000 7fd77f200700 1 --2- 172.21.15.89:0/3853152248 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd75806adb0 0x7fd75806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.462+0000 7fd77f200700 1 --2- 172.21.15.89:0/3853152248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7780709c0 0x7fd778151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.462+0000 7fd77f200700 1 --2- 172.21.15.89:0/3853152248 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd7780b3af0 0x7fd7780adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.462+0000 7fd77f200700 1 --2- 172.21.15.89:0/3853152248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd778072580 0x7fd7781f40f0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.462+0000 7fd77f200700 1 -- 172.21.15.89:0/3853152248 >> 172.21.15.89:0/3853152248 conn(0x7fd7780cf190 msgr2=0x7fd7780cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.462+0000 7fd77f200700 1 -- 172.21.15.89:0/3853152248 shutdown_connections 2024-04-02T08:53:05.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.462+0000 7fd77f200700 1 -- 172.21.15.89:0/3853152248 wait complete. 2024-04-02T08:53:05.482 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T08:52:40.806770+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T08:52:41.463828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T08:53:05.483 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T08:53:05.626 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:05.626+0000 7fb0bdd5f240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T08:53:05.660 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:05.661+0000 7f4b2ce59240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T08:53:05.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.681+0000 7f5bda000700 1 Processor -- start 2024-04-02T08:53:05.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.681+0000 7f5bda000700 1 -- start start 2024-04-02T08:53:05.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.682+0000 7f5bda000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd4099a60 0x7f5bd40d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:05.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.683+0000 7f5bda000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40d0970 con 0x7f5bd409a750 2024-04-02T08:53:05.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.683+0000 7f5bda000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd406c020 con 0x7f5bd415d990 2024-04-02T08:53:05.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.683+0000 7f5bda000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd40c81a0 con 0x7f5bd4099a60 2024-04-02T08:53:05.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.683+0000 7f5bd3e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5bd409a750 0x7f5bd406c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45492/0 (socket says 172.21.15.89:45492) 2024-04-02T08:53:05.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.683+0000 7f5bd3e00700 1 -- 172.21.15.89:0/264810954 learned_addr learned my addr 172.21.15.89:0/264810954 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:05.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.683+0000 7f5bd3400700 1 --2- 172.21.15.89:0/264810954 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd4099a60 0x7f5bd40d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.683+0000 7f5bd2000700 1 -- 172.21.15.89:0/264810954 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080760250 0 0) 0x7f5bd40d0970 con 0x7f5bd409a750 2024-04-02T08:53:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.683+0000 7f5bd2000700 1 -- 172.21.15.89:0/264810954 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb80035b0 con 0x7f5bd409a750 2024-04-02T08:53:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.683+0000 7f5bd2000700 1 -- 172.21.15.89:0/264810954 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588175800 0 0) 0x7f5bd406c020 con 0x7f5bd415d990 2024-04-02T08:53:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.683+0000 7f5bd3400700 1 -- 172.21.15.89:0/264810954 >> v1:172.21.15.113:6789/0 conn(0x7f5bd415d990 legacy=0x7f5bd4099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.683+0000 7f5bd3400700 1 -- 172.21.15.89:0/264810954 >> v1:172.21.15.89:6789/0 conn(0x7f5bd409a750 legacy=0x7f5bd406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.683+0000 7f5bd3400700 1 -- 172.21.15.89:0/264810954 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd40d0690 con 0x7f5bd4099a60 2024-04-02T08:53:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.684+0000 7f5bd3400700 1 --2- 172.21.15.89:0/264810954 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd4099a60 0x7f5bd40d0060 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4009870 tx=0x7f5bc4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49aa49eeef0ffe3f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.684+0000 7f5bd2000700 1 -- 172.21.15.89:0/264810954 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4011830 con 0x7f5bd4099a60 2024-04-02T08:53:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.684+0000 7f5bd2000700 1 -- 172.21.15.89:0/264810954 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc4011990 con 0x7f5bd4099a60 2024-04-02T08:53:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.684+0000 7f5bd2000700 1 -- 172.21.15.89:0/264810954 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4011d20 con 0x7f5bd4099a60 2024-04-02T08:53:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.685+0000 7f5bda000700 1 -- 172.21.15.89:0/264810954 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd4099a60 msgr2=0x7f5bd40d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.685+0000 7f5bda000700 1 --2- 172.21.15.89:0/264810954 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd4099a60 0x7f5bd40d0060 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4009870 tx=0x7f5bc4009b80 comp rx=0 tx=0).stop 2024-04-02T08:53:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.685+0000 7f5bda000700 1 -- 172.21.15.89:0/264810954 shutdown_connections 2024-04-02T08:53:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.685+0000 7f5bda000700 1 --2- 172.21.15.89:0/264810954 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd4099a60 0x7f5bd40d0060 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.685+0000 7f5bda000700 1 -- 172.21.15.89:0/264810954 >> 172.21.15.89:0/264810954 conn(0x7f5bd40b2e20 msgr2=0x7f5bd40b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.685+0000 7f5bda000700 1 -- 172.21.15.89:0/264810954 shutdown_connections 2024-04-02T08:53:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.685+0000 7f5bda000700 1 -- 172.21.15.89:0/264810954 wait complete. 2024-04-02T08:53:05.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.686+0000 7f5bda000700 1 Processor -- start 2024-04-02T08:53:05.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.687+0000 7f5bda000700 1 -- start start 2024-04-02T08:53:05.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.687+0000 7f5bda000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bd4099a60 0x7f5bd41f33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:05.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.687+0000 7f5bda000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd409a750 0x7f5bd40cf780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:05.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.687+0000 7f5bda000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd415d990 0x7f5bd41587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.687+0000 7f5bda000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd41f38f0 con 0x7f5bd4099a60 2024-04-02T08:53:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.687+0000 7f5bda000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd40c8770 con 0x7f5bd415d990 2024-04-02T08:53:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.687+0000 7f5bda000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd40c8a50 con 0x7f5bd409a750 2024-04-02T08:53:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.688+0000 7f5bd3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bd4099a60 0x7f5bd41f33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.688+0000 7f5bd3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bd4099a60 0x7f5bd41f33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38552/0 (socket says 172.21.15.89:38552) 2024-04-02T08:53:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.688+0000 7f5bd3400700 1 -- 172.21.15.89:0/992865020 learned_addr learned my addr 172.21.15.89:0/992865020 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.688+0000 7f5bd2a00700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd409a750 0x7f5bd40cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.688+0000 7f5bd3e00700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd415d990 0x7f5bd41587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.688+0000 7f5bd3400700 1 -- 172.21.15.89:0/992865020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd409a750 msgr2=0x7f5bd40cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.688+0000 7f5bd3400700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd409a750 0x7f5bd40cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.688+0000 7f5bd3400700 1 -- 172.21.15.89:0/992865020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd415d990 msgr2=0x7f5bd41587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.688+0000 7f5bd3400700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd415d990 0x7f5bd41587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.688+0000 7f5bd3400700 1 -- 172.21.15.89:0/992865020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc40094a0 con 0x7f5bd4099a60 2024-04-02T08:53:05.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.688+0000 7f5bd2a00700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd409a750 0x7f5bd40cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:05.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.689+0000 7f5bd3400700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bd4099a60 0x7f5bd41f33b0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40090b0 tx=0x7f5bc4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:05.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.689+0000 7f5bcb400700 1 -- 172.21.15.89:0/992865020 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc401a7e0 con 0x7f5bd4099a60 2024-04-02T08:53:05.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.689+0000 7f5bcb400700 1 -- 172.21.15.89:0/992865020 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc401a940 con 0x7f5bd4099a60 2024-04-02T08:53:05.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.689+0000 7f5bda000700 1 -- 172.21.15.89:0/992865020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd40c8d30 con 0x7f5bd4099a60 2024-04-02T08:53:05.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.689+0000 7f5bcb400700 1 -- 172.21.15.89:0/992865020 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc401acd0 con 0x7f5bd4099a60 2024-04-02T08:53:05.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.689+0000 7f5bda000700 1 -- 172.21.15.89:0/992865020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd40c92f0 con 0x7f5bd4099a60 2024-04-02T08:53:05.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.690+0000 7f5bda000700 1 -- 172.21.15.89:0/992865020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd40c84c0 con 0x7f5bd4099a60 2024-04-02T08:53:05.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.691+0000 7f5bcb400700 1 -- 172.21.15.89:0/992865020 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f5bc4022070 con 0x7f5bd4099a60 2024-04-02T08:53:05.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.692+0000 7f5bcb400700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5bb406adb0 0x7f5bb406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:05.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.692+0000 7f5bd2a00700 1 -- 172.21.15.89:0/992865020 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5bb406adb0 msgr2=0x7f5bb406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:05.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.692+0000 7f5bcb400700 1 -- 172.21.15.89:0/992865020 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f5bc408a740 con 0x7f5bd4099a60 2024-04-02T08:53:05.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.692+0000 7f5bd2a00700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5bb406adb0 0x7f5bb406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:05.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.696+0000 7f5bcb400700 1 -- 172.21.15.89:0/992865020 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5bc4055500 con 0x7f5bd4099a60 2024-04-02T08:53:05.850 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:05.851+0000 7f4b2ce59240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T08:53:05.889 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:05.889+0000 7fb0bdd5f240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T08:53:05.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.893+0000 7f5bd2a00700 1 -- 172.21.15.89:0/992865020 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5bb406adb0 msgr2=0x7f5bb406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:05.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.893+0000 7f5bd2a00700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5bb406adb0 0x7f5bb406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:05.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.903+0000 7f5bda000700 1 -- 172.21.15.89:0/992865020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5bd4067720 con 0x7f5bd4099a60 2024-04-02T08:53:05.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.903+0000 7f5bcb400700 1 -- 172.21.15.89:0/992865020 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v173) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5bc4058b20 con 0x7f5bd4099a60 2024-04-02T08:53:05.903 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T08:53:05.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.906+0000 7f5bda000700 1 -- 172.21.15.89:0/992865020 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5bb406adb0 msgr2=0x7f5bb406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:05.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.906+0000 7f5bda000700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5bb406adb0 0x7f5bb406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.906+0000 7f5bda000700 1 -- 172.21.15.89:0/992865020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bd4099a60 msgr2=0x7f5bd41f33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:05.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.906+0000 7f5bda000700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bd4099a60 0x7f5bd41f33b0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40090b0 tx=0x7f5bc4004750 comp rx=0 tx=0).stop 2024-04-02T08:53:05.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.906+0000 7f5bda000700 1 -- 172.21.15.89:0/992865020 shutdown_connections 2024-04-02T08:53:05.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.906+0000 7f5bda000700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f5bb406adb0 0x7f5bb406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.906+0000 7f5bda000700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd415d990 0x7f5bd41587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.906+0000 7f5bda000700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5bd4099a60 0x7f5bd41f33b0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.906+0000 7f5bda000700 1 --2- 172.21.15.89:0/992865020 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5bd409a750 0x7f5bd40cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:05.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.906+0000 7f5bda000700 1 -- 172.21.15.89:0/992865020 >> 172.21.15.89:0/992865020 conn(0x7f5bd40b2e20 msgr2=0x7f5bd40b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:05.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.906+0000 7f5bda000700 1 -- 172.21.15.89:0/992865020 shutdown_connections 2024-04-02T08:53:05.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:05.906+0000 7f5bda000700 1 -- 172.21.15.89:0/992865020 wait complete. 2024-04-02T08:53:05.924 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T08:53:05.941 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:05.942+0000 7f4b2ce59240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T08:53:06.099 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:06.100+0000 7fb0bdd5f240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T08:53:06.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.123+0000 7fc502c00700 1 Processor -- start 2024-04-02T08:53:06.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.123+0000 7fc502c00700 1 -- start start 2024-04-02T08:53:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.124+0000 7fc502c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4fc15d9d0 0x7fc4fc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.124+0000 7fc502c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc4fc0d16a0 0x7fc4fc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.124+0000 7fc502c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4fc0d2370 0x7fc4fc15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.124+0000 7fc502c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4fc0d2820 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.124+0000 7fc502c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4fc161420 con 0x7fc4fc0d16a0 2024-04-02T08:53:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.124+0000 7fc502c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc161700 con 0x7fc4fc0d2370 2024-04-02T08:53:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.124+0000 7fc4fbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4fc15d9d0 0x7fc4fc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.124+0000 7fc4fbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4fc15d9d0 0x7fc4fc0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38568/0 (socket says 172.21.15.89:38568) 2024-04-02T08:53:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.124+0000 7fc4fbe00700 1 -- 172.21.15.89:0/4031720895 learned_addr learned my addr 172.21.15.89:0/4031720895 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.124+0000 7fc4fb400700 1 --2- 172.21.15.89:0/4031720895 >> v2:172.21.15.113:3300/0 conn(0x7fc4fc0d16a0 0x7fc4fc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.124+0000 7fc500c00700 1 --2- 172.21.15.89:0/4031720895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4fc0d2370 0x7fc4fc15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.125+0000 7fc4fbe00700 1 -- 172.21.15.89:0/4031720895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4fc0d2370 msgr2=0x7fc4fc15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.125+0000 7fc4fbe00700 1 --2- 172.21.15.89:0/4031720895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4fc0d2370 0x7fc4fc15ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.125+0000 7fc4fbe00700 1 -- 172.21.15.89:0/4031720895 >> v2:172.21.15.113:3300/0 conn(0x7fc4fc0d16a0 msgr2=0x7fc4fc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.125+0000 7fc4fbe00700 1 --2- 172.21.15.89:0/4031720895 >> v2:172.21.15.113:3300/0 conn(0x7fc4fc0d16a0 0x7fc4fc0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.125+0000 7fc4fbe00700 1 -- 172.21.15.89:0/4031720895 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4fc1619e0 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.125+0000 7fc4fbe00700 1 --2- 172.21.15.89:0/4031720895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4fc15d9d0 0x7fc4fc0d3c30 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0d12f0 tx=0x7fc4ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4763a0c49e125620 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.125+0000 7fc4faa00700 1 -- 172.21.15.89:0/4031720895 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec011830 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.125+0000 7fc4faa00700 1 -- 172.21.15.89:0/4031720895 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ec011990 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.125+0000 7fc4faa00700 1 -- 172.21.15.89:0/4031720895 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec011d20 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.126+0000 7fc502c00700 1 -- 172.21.15.89:0/4031720895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4fc15d9d0 msgr2=0x7fc4fc0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.126+0000 7fc502c00700 1 --2- 172.21.15.89:0/4031720895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4fc15d9d0 0x7fc4fc0d3c30 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0d12f0 tx=0x7fc4ec009bd0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.126+0000 7fc502c00700 1 -- 172.21.15.89:0/4031720895 shutdown_connections 2024-04-02T08:53:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.126+0000 7fc502c00700 1 --2- 172.21.15.89:0/4031720895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4fc15d9d0 0x7fc4fc0d3c30 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.126+0000 7fc502c00700 1 --2- 172.21.15.89:0/4031720895 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4fc0d2370 0x7fc4fc15ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.126+0000 7fc502c00700 1 --2- 172.21.15.89:0/4031720895 >> v2:172.21.15.113:3300/0 conn(0x7fc4fc0d16a0 0x7fc4fc0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.126+0000 7fc502c00700 1 -- 172.21.15.89:0/4031720895 >> 172.21.15.89:0/4031720895 conn(0x7fc4fc08aff0 msgr2=0x7fc4fc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.126+0000 7fc502c00700 1 -- 172.21.15.89:0/4031720895 shutdown_connections 2024-04-02T08:53:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.126+0000 7fc502c00700 1 -- 172.21.15.89:0/4031720895 wait complete. 2024-04-02T08:53:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.128+0000 7fc502c00700 1 Processor -- start 2024-04-02T08:53:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.128+0000 7fc502c00700 1 -- start start 2024-04-02T08:53:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.129+0000 7fc502c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4fc0d16a0 0x7fc4fc0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc502c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4fc0d2370 0x7fc4fc14a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc502c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4fc15d9d0 0x7fc4fc14ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc502c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc0cac80 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc502c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc0cd390 con 0x7fc4fc0d2370 2024-04-02T08:53:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc502c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc0cd670 con 0x7fc4fc0d16a0 2024-04-02T08:53:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc500c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4fc15d9d0 0x7fc4fc14ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc4fb400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4fc0d2370 0x7fc4fc14a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc4fbe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4fc0d16a0 0x7fc4fc0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc500c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4fc15d9d0 0x7fc4fc14ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38580/0 (socket says 172.21.15.89:38580) 2024-04-02T08:53:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc500c00700 1 -- 172.21.15.89:0/3978989832 learned_addr learned my addr 172.21.15.89:0/3978989832 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc4fb400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4fc0d2370 0x7fc4fc14a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:34298/0 (socket says 172.21.15.89:34298) 2024-04-02T08:53:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc500c00700 1 -- 172.21.15.89:0/3978989832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4fc0d16a0 msgr2=0x7fc4fc0ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc500c00700 1 --2- 172.21.15.89:0/3978989832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4fc0d16a0 0x7fc4fc0ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc500c00700 1 -- 172.21.15.89:0/3978989832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4fc0d2370 msgr2=0x7fc4fc14a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:06.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc500c00700 1 --2- 172.21.15.89:0/3978989832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4fc0d2370 0x7fc4fc14a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.130+0000 7fc500c00700 1 -- 172.21.15.89:0/3978989832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec009550 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.131+0000 7fc500c00700 1 --2- 172.21.15.89:0/3978989832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4fc15d9d0 0x7fc4fc14ac00 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fc4f400efc0 tx=0x7fc4f400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:06.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.131+0000 7fc4f3e00700 1 -- 172.21.15.89:0/3978989832 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4009ea0 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.131+0000 7fc4f3e00700 1 -- 172.21.15.89:0/3978989832 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f400f040 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.131+0000 7fc4f3e00700 1 -- 172.21.15.89:0/3978989832 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f400ced0 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.131+0000 7fc502c00700 1 -- 172.21.15.89:0/3978989832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4fc0cd950 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.131+0000 7fc502c00700 1 -- 172.21.15.89:0/3978989832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4fc14a7f0 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.132+0000 7fc502c00700 1 -- 172.21.15.89:0/3978989832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4fc0d1ca0 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.133+0000 7fc4f3e00700 1 -- 172.21.15.89:0/3978989832 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fc4f4014480 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.134+0000 7fc4f3e00700 1 --2- 172.21.15.89:0/3978989832 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4dc06adb0 0x7fc4dc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:06.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.134+0000 7fc4f3e00700 1 -- 172.21.15.89:0/3978989832 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fc4f408a160 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.134+0000 7fc4fbe00700 1 -- 172.21.15.89:0/3978989832 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4dc06adb0 msgr2=0x7fc4dc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:06.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.134+0000 7fc4fbe00700 1 --2- 172.21.15.89:0/3978989832 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4dc06adb0 0x7fc4dc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:06.135 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:53:06.140 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:53:06.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.139+0000 7fc4f3e00700 1 -- 172.21.15.89:0/3978989832 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc4f4054d80 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.203 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:06.204+0000 7fb0bdd5f240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T08:53:06.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.335+0000 7fc4fbe00700 1 -- 172.21.15.89:0/3978989832 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4dc06adb0 msgr2=0x7fc4dc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.335+0000 7fc4fbe00700 1 --2- 172.21.15.89:0/3978989832 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4dc06adb0 0x7fc4dc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:06.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.353+0000 7fc502c00700 1 -- 172.21.15.89:0/3978989832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc4fc0b7d60 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.354+0000 7fc4f3e00700 1 -- 172.21.15.89:0/3978989832 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v173) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc4f40583a0 con 0x7fc4fc15d9d0 2024-04-02T08:53:06.353 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T08:53:06.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.356+0000 7fc502c00700 1 -- 172.21.15.89:0/3978989832 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4dc06adb0 msgr2=0x7fc4dc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:06.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.356+0000 7fc502c00700 1 --2- 172.21.15.89:0/3978989832 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4dc06adb0 0x7fc4dc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.356+0000 7fc502c00700 1 -- 172.21.15.89:0/3978989832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4fc15d9d0 msgr2=0x7fc4fc14ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:06.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.356+0000 7fc502c00700 1 --2- 172.21.15.89:0/3978989832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4fc15d9d0 0x7fc4fc14ac00 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fc4f400efc0 tx=0x7fc4f400c3a0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.356+0000 7fc502c00700 1 -- 172.21.15.89:0/3978989832 shutdown_connections 2024-04-02T08:53:06.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.356+0000 7fc502c00700 1 --2- 172.21.15.89:0/3978989832 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc4dc06adb0 0x7fc4dc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.356+0000 7fc502c00700 1 --2- 172.21.15.89:0/3978989832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4fc15d9d0 0x7fc4fc14ac00 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.356+0000 7fc502c00700 1 --2- 172.21.15.89:0/3978989832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4fc0d16a0 0x7fc4fc0ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.356+0000 7fc502c00700 1 --2- 172.21.15.89:0/3978989832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4fc0d2370 0x7fc4fc14a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.356+0000 7fc502c00700 1 -- 172.21.15.89:0/3978989832 >> 172.21.15.89:0/3978989832 conn(0x7fc4fc08aff0 msgr2=0x7fc4fc0ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:06.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.357+0000 7fc502c00700 1 -- 172.21.15.89:0/3978989832 shutdown_connections 2024-04-02T08:53:06.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.357+0000 7fc502c00700 1 -- 172.21.15.89:0/3978989832 wait complete. 2024-04-02T08:53:06.375 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T08:53:06.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.575+0000 7f39c0200700 1 Processor -- start 2024-04-02T08:53:06.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.575+0000 7f39c0200700 1 -- start start 2024-04-02T08:53:06.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.576+0000 7f39c0200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f39b8157f90 0x7f39b80b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:06.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.576+0000 7f39c0200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f39b80b3b30 0x7f39b8161890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:06.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.576+0000 7f39c0200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b80b3070 con 0x7f39b81613e0 2024-04-02T08:53:06.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.576+0000 7f39c0200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b8158370 con 0x7f39b8157f90 2024-04-02T08:53:06.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.576+0000 7f39c0200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b80b2710 con 0x7f39b80b3b30 2024-04-02T08:53:06.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.576+0000 7f39bd800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f39b8157f90 0x7f39b80b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:06.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.576+0000 7f39bd800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f39b8157f90 0x7f39b80b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38584/0 (socket says 172.21.15.89:38584) 2024-04-02T08:53:06.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.576+0000 7f39bd800700 1 -- 172.21.15.89:0/2839820743 learned_addr learned my addr 172.21.15.89:0/2839820743 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:06.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.577+0000 7f39be200700 1 --2- 172.21.15.89:0/2839820743 >> v2:172.21.15.113:3300/0 conn(0x7f39b80b3b30 0x7f39b8161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:06.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.577+0000 7f39bd800700 1 -- 172.21.15.89:0/2839820743 >> v2:172.21.15.113:3300/0 conn(0x7f39b80b3b30 msgr2=0x7f39b8161890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:06.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.577+0000 7f39bd800700 1 --2- 172.21.15.89:0/2839820743 >> v2:172.21.15.113:3300/0 conn(0x7f39b80b3b30 0x7f39b8161890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.577+0000 7f39bd800700 1 -- 172.21.15.89:0/2839820743 >> v1:172.21.15.113:6789/0 conn(0x7f39b81613e0 legacy=0x7f39b80b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:06.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.577+0000 7f39bd800700 1 -- 172.21.15.89:0/2839820743 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80b2070 con 0x7f39b8157f90 2024-04-02T08:53:06.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.577+0000 7f39bd800700 1 --2- 172.21.15.89:0/2839820743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b8157f90 0x7f39b80b2b30 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f39b806eeb0 tx=0x7f39ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434b0bb45aafc9a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:06.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.577+0000 7f39b7e00700 1 -- 172.21.15.89:0/2839820743 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac00b870 con 0x7f39b8157f90 2024-04-02T08:53:06.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.577+0000 7f39b7e00700 1 -- 172.21.15.89:0/2839820743 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ac00b9d0 con 0x7f39b8157f90 2024-04-02T08:53:06.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.577+0000 7f39b7e00700 1 -- 172.21.15.89:0/2839820743 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac00bd60 con 0x7f39b8157f90 2024-04-02T08:53:06.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.578+0000 7f39c0200700 1 -- 172.21.15.89:0/2839820743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b8157f90 msgr2=0x7f39b80b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:06.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.578+0000 7f39c0200700 1 --2- 172.21.15.89:0/2839820743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b8157f90 0x7f39b80b2b30 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f39b806eeb0 tx=0x7f39ac009bd0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.578+0000 7f39c0200700 1 -- 172.21.15.89:0/2839820743 shutdown_connections 2024-04-02T08:53:06.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.578+0000 7f39c0200700 1 --2- 172.21.15.89:0/2839820743 >> v2:172.21.15.113:3300/0 conn(0x7f39b80b3b30 0x7f39b8161890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.578+0000 7f39c0200700 1 --2- 172.21.15.89:0/2839820743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b8157f90 0x7f39b80b2b30 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.578+0000 7f39c0200700 1 -- 172.21.15.89:0/2839820743 >> 172.21.15.89:0/2839820743 conn(0x7f39b80b0d70 msgr2=0x7f39b80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:06.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.578+0000 7f39c0200700 1 -- 172.21.15.89:0/2839820743 shutdown_connections 2024-04-02T08:53:06.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.578+0000 7f39c0200700 1 -- 172.21.15.89:0/2839820743 wait complete. 2024-04-02T08:53:06.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.581+0000 7f39c0200700 1 Processor -- start 2024-04-02T08:53:06.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.581+0000 7f39c0200700 1 -- start start 2024-04-02T08:53:06.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.582+0000 7f39c0200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80b3b30 0x7f39b8156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:06.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39c0200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39b8157f90 0x7f39b806e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39c0200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39b81613e0 0x7f39b814d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39c0200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b806ec90 con 0x7f39b80b3b30 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39c0200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8151ea0 con 0x7f39b8157f90 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39c0200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8152180 con 0x7f39b81613e0 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39bd800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80b3b30 0x7f39b8156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39bd800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80b3b30 0x7f39b8156b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38590/0 (socket says 172.21.15.89:38590) 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39bd800700 1 -- 172.21.15.89:0/471711926 learned_addr learned my addr 172.21.15.89:0/471711926 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39be200700 1 --2- 172.21.15.89:0/471711926 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39b81613e0 0x7f39b814d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39bce00700 1 --2- 172.21.15.89:0/471711926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39b8157f90 0x7f39b806e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39bd800700 1 -- 172.21.15.89:0/471711926 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39b81613e0 msgr2=0x7f39b814d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39bd800700 1 --2- 172.21.15.89:0/471711926 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39b81613e0 0x7f39b814d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39bd800700 1 -- 172.21.15.89:0/471711926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39b8157f90 msgr2=0x7f39b806e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39bd800700 1 --2- 172.21.15.89:0/471711926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39b8157f90 0x7f39b806e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.583+0000 7f39bd800700 1 -- 172.21.15.89:0/471711926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac009550 con 0x7f39b80b3b30 2024-04-02T08:53:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.584+0000 7f39bd800700 1 --2- 172.21.15.89:0/471711926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80b3b30 0x7f39b8156b30 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f39ac000c00 tx=0x7f39ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:06.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.584+0000 7f39b5600700 1 -- 172.21.15.89:0/471711926 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac004520 con 0x7f39b80b3b30 2024-04-02T08:53:06.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.584+0000 7f39b5600700 1 -- 172.21.15.89:0/471711926 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ac01e070 con 0x7f39b80b3b30 2024-04-02T08:53:06.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.584+0000 7f39c0200700 1 -- 172.21.15.89:0/471711926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b8152460 con 0x7f39b80b3b30 2024-04-02T08:53:06.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.584+0000 7f39b5600700 1 -- 172.21.15.89:0/471711926 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac0038a0 con 0x7f39b80b3b30 2024-04-02T08:53:06.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.584+0000 7f39c0200700 1 -- 172.21.15.89:0/471711926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b8152c60 con 0x7f39b80b3b30 2024-04-02T08:53:06.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.585+0000 7f39c0200700 1 -- 172.21.15.89:0/471711926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b8152920 con 0x7f39b80b3b30 2024-04-02T08:53:06.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.586+0000 7f39b5600700 1 -- 172.21.15.89:0/471711926 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f39ac003a00 con 0x7f39b80b3b30 2024-04-02T08:53:06.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.587+0000 7f39b5600700 1 --2- 172.21.15.89:0/471711926 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 0x7f399806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:06.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.587+0000 7f39b5600700 1 -- 172.21.15.89:0/471711926 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f39ac020070 con 0x7f39b80b3b30 2024-04-02T08:53:06.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.587+0000 7f39bce00700 1 -- 172.21.15.89:0/471711926 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 msgr2=0x7f399806d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:06.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.587+0000 7f39bce00700 1 --2- 172.21.15.89:0/471711926 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 0x7f399806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:06.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.591+0000 7f39b5600700 1 -- 172.21.15.89:0/471711926 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f39ac055b90 con 0x7f39b80b3b30 2024-04-02T08:53:06.690 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:06.688+0000 7f4b2ce59240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T08:53:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.788+0000 7f39bce00700 1 -- 172.21.15.89:0/471711926 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 msgr2=0x7f399806d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.788+0000 7f39bce00700 1 --2- 172.21.15.89:0/471711926 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 0x7f399806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:06.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.797+0000 7f39c0200700 1 -- 172.21.15.89:0/471711926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f39b8067720 con 0x7f39b80b3b30 2024-04-02T08:53:06.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.798+0000 7f39b5600700 1 -- 172.21.15.89:0/471711926 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v173) v1 ==== 100+0+11 (secure 0 0 0) 0x7f39ac015020 con 0x7f39b80b3b30 2024-04-02T08:53:06.797 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T08:53:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.800+0000 7f39c0200700 1 -- 172.21.15.89:0/471711926 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 msgr2=0x7f399806d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.800+0000 7f39c0200700 1 --2- 172.21.15.89:0/471711926 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 0x7f399806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.800+0000 7f39c0200700 1 -- 172.21.15.89:0/471711926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80b3b30 msgr2=0x7f39b8156b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.800+0000 7f39c0200700 1 --2- 172.21.15.89:0/471711926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80b3b30 0x7f39b8156b30 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f39ac000c00 tx=0x7f39ac004750 comp rx=0 tx=0).stop 2024-04-02T08:53:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.800+0000 7f39c0200700 1 -- 172.21.15.89:0/471711926 shutdown_connections 2024-04-02T08:53:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.800+0000 7f39c0200700 1 --2- 172.21.15.89:0/471711926 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f399806ade0 0x7f399806d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.800+0000 7f39c0200700 1 --2- 172.21.15.89:0/471711926 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f39b81613e0 0x7f39b814d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.800+0000 7f39c0200700 1 --2- 172.21.15.89:0/471711926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39b80b3b30 0x7f39b8156b30 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.800+0000 7f39c0200700 1 --2- 172.21.15.89:0/471711926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39b8157f90 0x7f39b806e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.800+0000 7f39c0200700 1 -- 172.21.15.89:0/471711926 >> 172.21.15.89:0/471711926 conn(0x7f39b80b0d70 msgr2=0x7f39b80b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.801+0000 7f39c0200700 1 -- 172.21.15.89:0/471711926 shutdown_connections 2024-04-02T08:53:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:06.801+0000 7f39c0200700 1 -- 172.21.15.89:0/471711926 wait complete. 2024-04-02T08:53:06.821 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T08:53:06.896 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:06.897+0000 7f4b2ce59240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T08:53:06.975 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:06.975+0000 7fb0bdd5f240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T08:53:07.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.018+0000 7fb089a00700 1 Processor -- start 2024-04-02T08:53:07.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.019+0000 7fb089a00700 1 -- start start 2024-04-02T08:53:07.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.020+0000 7fb089a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb08415da50 0x7fb0840ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.020+0000 7fb089a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0840d0cf0 0x7fb0840b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.020+0000 7fb089a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0840d24e0 con 0x7fb0840cec70 2024-04-02T08:53:07.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.020+0000 7fb089a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0840b95d0 con 0x7fb08415da50 2024-04-02T08:53:07.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.020+0000 7fb089a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb08415de30 con 0x7fb0840d0cf0 2024-04-02T08:53:07.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.020+0000 7fb082a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb08415da50 0x7fb0840ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.020+0000 7fb082000700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb0840cec70 0x7fb0840d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:51978/0 (socket says 172.21.15.89:51978) 2024-04-02T08:53:07.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.020+0000 7fb082000700 1 -- 172.21.15.89:0/658997188 learned_addr learned my addr 172.21.15.89:0/658997188 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:07.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.020+0000 7fb083400700 1 --2- 172.21.15.89:0/658997188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0840d0cf0 0x7fb0840b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.020+0000 7fb082a00700 1 -- 172.21.15.89:0/658997188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0840d0cf0 msgr2=0x7fb0840b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:07.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.020+0000 7fb082a00700 1 --2- 172.21.15.89:0/658997188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0840d0cf0 0x7fb0840b6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.020+0000 7fb082a00700 1 -- 172.21.15.89:0/658997188 >> v1:172.21.15.113:6789/0 conn(0x7fb0840cec70 legacy=0x7fb0840d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:07.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.021+0000 7fb082a00700 1 -- 172.21.15.89:0/658997188 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0840d27c0 con 0x7fb08415da50 2024-04-02T08:53:07.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.021+0000 7fb081600700 1 -- 172.21.15.89:0/658997188 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453685082 0 0) 0x7fb0840d24e0 con 0x7fb0840cec70 2024-04-02T08:53:07.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.021+0000 7fb082a00700 1 --2- 172.21.15.89:0/658997188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08415da50 0x7fb0840ce730 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb074009840 tx=0x7fb074009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7585f713862a26ee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:07.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.021+0000 7fb081600700 1 -- 172.21.15.89:0/658997188 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb074011830 con 0x7fb08415da50 2024-04-02T08:53:07.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.021+0000 7fb081600700 1 -- 172.21.15.89:0/658997188 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb074011990 con 0x7fb08415da50 2024-04-02T08:53:07.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.021+0000 7fb081600700 1 -- 172.21.15.89:0/658997188 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb074011d20 con 0x7fb08415da50 2024-04-02T08:53:07.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.021+0000 7fb089a00700 1 -- 172.21.15.89:0/658997188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08415da50 msgr2=0x7fb0840ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:07.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.022+0000 7fb089a00700 1 --2- 172.21.15.89:0/658997188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08415da50 0x7fb0840ce730 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb074009840 tx=0x7fb074009b50 comp rx=0 tx=0).stop 2024-04-02T08:53:07.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.022+0000 7fb089a00700 1 -- 172.21.15.89:0/658997188 shutdown_connections 2024-04-02T08:53:07.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.022+0000 7fb089a00700 1 --2- 172.21.15.89:0/658997188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0840d0cf0 0x7fb0840b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.022+0000 7fb089a00700 1 --2- 172.21.15.89:0/658997188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08415da50 0x7fb0840ce730 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.022+0000 7fb089a00700 1 -- 172.21.15.89:0/658997188 >> 172.21.15.89:0/658997188 conn(0x7fb0840666a0 msgr2=0x7fb084066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:07.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.022+0000 7fb089a00700 1 -- 172.21.15.89:0/658997188 shutdown_connections 2024-04-02T08:53:07.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.022+0000 7fb089a00700 1 -- 172.21.15.89:0/658997188 wait complete. 2024-04-02T08:53:07.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.023+0000 7fb089a00700 1 Processor -- start 2024-04-02T08:53:07.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.024+0000 7fb089a00700 1 -- start start 2024-04-02T08:53:07.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.024+0000 7fb089a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0840cec70 0x7fb0840cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.024+0000 7fb089a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0840d0cf0 0x7fb0841603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.024+0000 7fb089a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb08415da50 0x7fb084152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.024+0000 7fb089a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0840cb370 con 0x7fb08415da50 2024-04-02T08:53:07.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.024+0000 7fb089a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0840cda00 con 0x7fb0840cec70 2024-04-02T08:53:07.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.024+0000 7fb089a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0840ca9d0 con 0x7fb0840d0cf0 2024-04-02T08:53:07.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.025+0000 7fb082a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0840cec70 0x7fb0840cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.025+0000 7fb082a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0840cec70 0x7fb0840cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:34348/0 (socket says 172.21.15.89:34348) 2024-04-02T08:53:07.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.025+0000 7fb082a00700 1 -- 172.21.15.89:0/1507096144 learned_addr learned my addr 172.21.15.89:0/1507096144 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:07.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.025+0000 7fb083400700 1 --2- 172.21.15.89:0/1507096144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb08415da50 0x7fb084152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.025+0000 7fb082000700 1 --2- 172.21.15.89:0/1507096144 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0840d0cf0 0x7fb0841603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.025+0000 7fb082a00700 1 -- 172.21.15.89:0/1507096144 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0840d0cf0 msgr2=0x7fb0841603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:07.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.025+0000 7fb082a00700 1 --2- 172.21.15.89:0/1507096144 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0840d0cf0 0x7fb0841603b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.025+0000 7fb082a00700 1 -- 172.21.15.89:0/1507096144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb08415da50 msgr2=0x7fb084152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:07.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.025+0000 7fb082a00700 1 --2- 172.21.15.89:0/1507096144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb08415da50 0x7fb084152750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.025+0000 7fb082a00700 1 -- 172.21.15.89:0/1507096144 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0740094a0 con 0x7fb0840cec70 2024-04-02T08:53:07.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.025+0000 7fb082a00700 1 --2- 172.21.15.89:0/1507096144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0840cec70 0x7fb0840cd140 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fb074009b90 tx=0x7fb074004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:07.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.026+0000 7fb07aa00700 1 -- 172.21.15.89:0/1507096144 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0740036a0 con 0x7fb0840cec70 2024-04-02T08:53:07.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.026+0000 7fb07aa00700 1 -- 172.21.15.89:0/1507096144 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07401e070 con 0x7fb0840cec70 2024-04-02T08:53:07.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.026+0000 7fb089a00700 1 -- 172.21.15.89:0/1507096144 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0840cacb0 con 0x7fb0840cec70 2024-04-02T08:53:07.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.026+0000 7fb07aa00700 1 -- 172.21.15.89:0/1507096144 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb074003a60 con 0x7fb0840cec70 2024-04-02T08:53:07.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.026+0000 7fb089a00700 1 -- 172.21.15.89:0/1507096144 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0840cd680 con 0x7fb0840cec70 2024-04-02T08:53:07.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.027+0000 7fb089a00700 1 -- 172.21.15.89:0/1507096144 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0840cb4b0 con 0x7fb0840cec70 2024-04-02T08:53:07.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.028+0000 7fb07aa00700 1 -- 172.21.15.89:0/1507096144 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fb074003bc0 con 0x7fb0840cec70 2024-04-02T08:53:07.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.029+0000 7fb07aa00700 1 --2- 172.21.15.89:0/1507096144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb06406adb0 0x7fb06406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.029+0000 7fb07aa00700 1 -- 172.21.15.89:0/1507096144 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fb07408a900 con 0x7fb0840cec70 2024-04-02T08:53:07.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.029+0000 7fb082000700 1 -- 172.21.15.89:0/1507096144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb06406adb0 msgr2=0x7fb06406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:07.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.029+0000 7fb082000700 1 --2- 172.21.15.89:0/1507096144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb06406adb0 0x7fb06406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:07.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.032+0000 7fb07aa00700 1 -- 172.21.15.89:0/1507096144 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb074055740 con 0x7fb0840cec70 2024-04-02T08:53:07.103 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:07.103+0000 7f4b2ce59240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:53:07.196 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:07.197+0000 7fb0bdd5f240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T08:53:07.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.230+0000 7fb082000700 1 -- 172.21.15.89:0/1507096144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb06406adb0 msgr2=0x7fb06406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:07.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.230+0000 7fb082000700 1 --2- 172.21.15.89:0/1507096144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb06406adb0 0x7fb06406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:07.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.286+0000 7fb089a00700 1 -- 172.21.15.89:0/1507096144 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0840cb4b0 con 0x7fb0840cec70 2024-04-02T08:53:07.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.287+0000 7fb07aa00700 1 -- 172.21.15.89:0/1507096144 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4738 (secure 0 0 0) 0x7fb0840cb4b0 con 0x7fb0840cec70 2024-04-02T08:53:07.286 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:07.286 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8881,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/547244866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":547244866},{"type":"v1","addr":"172.21.15.113:6834","nonce":547244866}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":8884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2822200553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2822200553},{"type":"v1","addr":"172.21.15.113:6835","nonce":2822200553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/342426898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":342426898},{"type":"v1","addr":"172.21.15.89:6835","nonce":342426898}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T08:52:42.484905+0000","modified":"2024-04-02T08:52:45.498450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9534},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9534":{"gid":9534,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2812511535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2812511535},{"type":"v1","addr":"172.21.15.89:6837","nonce":2812511535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-02T08:53:07.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.289+0000 7fb089a00700 1 -- 172.21.15.89:0/1507096144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb06406adb0 msgr2=0x7fb06406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:07.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.289+0000 7fb089a00700 1 --2- 172.21.15.89:0/1507096144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb06406adb0 0x7fb06406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.289+0000 7fb089a00700 1 -- 172.21.15.89:0/1507096144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0840cec70 msgr2=0x7fb0840cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:07.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.289+0000 7fb089a00700 1 --2- 172.21.15.89:0/1507096144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0840cec70 0x7fb0840cd140 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fb074009b90 tx=0x7fb074004750 comp rx=0 tx=0).stop 2024-04-02T08:53:07.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.289+0000 7fb089a00700 1 -- 172.21.15.89:0/1507096144 shutdown_connections 2024-04-02T08:53:07.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.289+0000 7fb089a00700 1 --2- 172.21.15.89:0/1507096144 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fb06406adb0 0x7fb06406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.289+0000 7fb089a00700 1 --2- 172.21.15.89:0/1507096144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0840cec70 0x7fb0840cd140 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.289+0000 7fb089a00700 1 --2- 172.21.15.89:0/1507096144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb08415da50 0x7fb084152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.289+0000 7fb089a00700 1 --2- 172.21.15.89:0/1507096144 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0840d0cf0 0x7fb0841603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.289+0000 7fb089a00700 1 -- 172.21.15.89:0/1507096144 >> 172.21.15.89:0/1507096144 conn(0x7fb0840666a0 msgr2=0x7fb084066a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:07.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.289+0000 7fb089a00700 1 -- 172.21.15.89:0/1507096144 shutdown_connections 2024-04-02T08:53:07.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.289+0000 7fb089a00700 1 -- 172.21.15.89:0/1507096144 wait complete. 2024-04-02T08:53:07.290 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 81 2024-04-02T08:53:07.307 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:53:07.415 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:07.416+0000 7fb0bdd5f240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:53:07.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.504+0000 7f32e7800700 1 Processor -- start 2024-04-02T08:53:07.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.504+0000 7f32e7800700 1 -- start start 2024-04-02T08:53:07.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.505+0000 7f32e7800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32e0099b40 0x7f32e006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.505+0000 7f32e7800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f32e015f200 0x7f32e015f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.506+0000 7f32e7800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e00b39d0 con 0x7f32e015b7e0 2024-04-02T08:53:07.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.506+0000 7f32e7800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e015fbd0 con 0x7f32e015f200 2024-04-02T08:53:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.506+0000 7f32e7800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00b3410 con 0x7f32e0099b40 2024-04-02T08:53:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.506+0000 7f32e5800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f32e015f200 0x7f32e015f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.506+0000 7f32e4e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f32e015b7e0 0x7f32e00b2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45508/0 (socket says 172.21.15.89:45508) 2024-04-02T08:53:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.506+0000 7f32e4e00700 1 -- 172.21.15.89:0/3021115421 learned_addr learned my addr 172.21.15.89:0/3021115421 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.506+0000 7f32dfe00700 1 --2- 172.21.15.89:0/3021115421 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32e0099b40 0x7f32e006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.506+0000 7f32e5800700 1 -- 172.21.15.89:0/3021115421 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32e0099b40 msgr2=0x7f32e006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.506+0000 7f32e5800700 1 --2- 172.21.15.89:0/3021115421 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32e0099b40 0x7f32e006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.506+0000 7f32e5800700 1 -- 172.21.15.89:0/3021115421 >> v1:172.21.15.89:6789/0 conn(0x7f32e015b7e0 legacy=0x7f32e00b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.506+0000 7f32df400700 1 -- 172.21.15.89:0/3021115421 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206150524 0 0) 0x7f32e00b39d0 con 0x7f32e015b7e0 2024-04-02T08:53:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.506+0000 7f32e5800700 1 -- 172.21.15.89:0/3021115421 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e00b36f0 con 0x7f32e015f200 2024-04-02T08:53:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.507+0000 7f32e5800700 1 --2- 172.21.15.89:0/3021115421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e015f200 0x7f32e015f690 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009d90 tx=0x7f32d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e997fc4bcc0d5211 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.507+0000 7f32df400700 1 -- 172.21.15.89:0/3021115421 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d800e070 con 0x7f32e015f200 2024-04-02T08:53:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.507+0000 7f32df400700 1 -- 172.21.15.89:0/3021115421 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d800bb20 con 0x7f32e015f200 2024-04-02T08:53:07.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.507+0000 7f32df400700 1 -- 172.21.15.89:0/3021115421 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8012430 con 0x7f32e015f200 2024-04-02T08:53:07.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.508+0000 7f32e7800700 1 -- 172.21.15.89:0/3021115421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e015f200 msgr2=0x7f32e015f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:07.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.508+0000 7f32e7800700 1 --2- 172.21.15.89:0/3021115421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e015f200 0x7f32e015f690 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009d90 tx=0x7f32d800b330 comp rx=0 tx=0).stop 2024-04-02T08:53:07.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.508+0000 7f32e7800700 1 -- 172.21.15.89:0/3021115421 shutdown_connections 2024-04-02T08:53:07.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.508+0000 7f32e7800700 1 --2- 172.21.15.89:0/3021115421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e015f200 0x7f32e015f690 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.508+0000 7f32e7800700 1 --2- 172.21.15.89:0/3021115421 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32e0099b40 0x7f32e006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.508+0000 7f32e7800700 1 -- 172.21.15.89:0/3021115421 >> 172.21.15.89:0/3021115421 conn(0x7f32e00b5ab0 msgr2=0x7f32e00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:07.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.508+0000 7f32e7800700 1 -- 172.21.15.89:0/3021115421 shutdown_connections 2024-04-02T08:53:07.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.508+0000 7f32e7800700 1 -- 172.21.15.89:0/3021115421 wait complete. 2024-04-02T08:53:07.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.511+0000 7f32e7800700 1 Processor -- start 2024-04-02T08:53:07.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.511+0000 7f32e7800700 1 -- start start 2024-04-02T08:53:07.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.512+0000 7f32e7800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32e0099b40 0x7f32e00c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.512+0000 7f32e7800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e015b7e0 0x7f32e00ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.512+0000 7f32e7800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e015f200 0x7f32e00ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.513+0000 7f32e7800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00ca650 con 0x7f32e015f200 2024-04-02T08:53:07.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.513+0000 7f32e7800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00cd2c0 con 0x7f32e015b7e0 2024-04-02T08:53:07.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.513+0000 7f32e7800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00ca890 con 0x7f32e0099b40 2024-04-02T08:53:07.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.513+0000 7f32e5800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e015f200 0x7f32e00ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.513+0000 7f32e4e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32e0099b40 0x7f32e00c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.513+0000 7f32dfe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e015b7e0 0x7f32e00ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.513+0000 7f32e5800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e015f200 0x7f32e00ccd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38618/0 (socket says 172.21.15.89:38618) 2024-04-02T08:53:07.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.513+0000 7f32e5800700 1 -- 172.21.15.89:0/1683307630 learned_addr learned my addr 172.21.15.89:0/1683307630 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:07.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.513+0000 7f32e5800700 1 -- 172.21.15.89:0/1683307630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32e0099b40 msgr2=0x7f32e00c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:07.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.513+0000 7f32e5800700 1 --2- 172.21.15.89:0/1683307630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32e0099b40 0x7f32e00c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.513+0000 7f32e5800700 1 -- 172.21.15.89:0/1683307630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e015b7e0 msgr2=0x7f32e00ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:07.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.513+0000 7f32e5800700 1 --2- 172.21.15.89:0/1683307630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e015b7e0 0x7f32e00ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.513+0000 7f32e5800700 1 -- 172.21.15.89:0/1683307630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d8009a70 con 0x7f32e015f200 2024-04-02T08:53:07.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.514+0000 7f32e5800700 1 --2- 172.21.15.89:0/1683307630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e015f200 0x7f32e00ccd80 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f32e0070560 tx=0x7f32d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:07.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.514+0000 7f32dcc00700 1 -- 172.21.15.89:0/1683307630 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8012430 con 0x7f32e015f200 2024-04-02T08:53:07.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.514+0000 7f32dcc00700 1 -- 172.21.15.89:0/1683307630 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d8020070 con 0x7f32e015f200 2024-04-02T08:53:07.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.514+0000 7f32dcc00700 1 -- 172.21.15.89:0/1683307630 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8004280 con 0x7f32e015f200 2024-04-02T08:53:07.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.514+0000 7f32e7800700 1 -- 172.21.15.89:0/1683307630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e00cab70 con 0x7f32e015f200 2024-04-02T08:53:07.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.514+0000 7f32e7800700 1 -- 172.21.15.89:0/1683307630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e00cc860 con 0x7f32e015f200 2024-04-02T08:53:07.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.515+0000 7f32e7800700 1 -- 172.21.15.89:0/1683307630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e0067720 con 0x7f32e015f200 2024-04-02T08:53:07.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.516+0000 7f32dcc00700 1 -- 172.21.15.89:0/1683307630 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f32d80043e0 con 0x7f32e015f200 2024-04-02T08:53:07.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.517+0000 7f32dcc00700 1 --2- 172.21.15.89:0/1683307630 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32c006adb0 0x7f32c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.517+0000 7f32e4e00700 1 -- 172.21.15.89:0/1683307630 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32c006adb0 msgr2=0x7f32c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:07.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.517+0000 7f32dcc00700 1 -- 172.21.15.89:0/1683307630 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f32d8022070 con 0x7f32e015f200 2024-04-02T08:53:07.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.517+0000 7f32e4e00700 1 --2- 172.21.15.89:0/1683307630 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32c006adb0 0x7f32c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:07.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.521+0000 7f32dcc00700 1 -- 172.21.15.89:0/1683307630 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f32d8056220 con 0x7f32e015f200 2024-04-02T08:53:07.574 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:07.575+0000 7f4b2ce59240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T08:53:07.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.718+0000 7f32e4e00700 1 -- 172.21.15.89:0/1683307630 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32c006adb0 msgr2=0x7f32c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:07.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.718+0000 7f32e4e00700 1 --2- 172.21.15.89:0/1683307630 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32c006adb0 0x7f32c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:07.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.729+0000 7f32e7800700 1 -- 172.21.15.89:0/1683307630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32e0072750 con 0x7f32e015f200 2024-04-02T08:53:07.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.731+0000 7f32dcc00700 1 -- 172.21.15.89:0/1683307630 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) v1 ==== 74+0+17243 (secure 0 0 0) 0x7f32d8059840 con 0x7f32e015f200 2024-04-02T08:53:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:07.731 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":173,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:52:58.629629+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-02T08:52:40.806770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-02T08:52:41.463828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/3680638162":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/263272397":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/1641401996":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/419225467":"2024-04-03T08:52:58.629596+0000","172.21.15.89:6833/43357":"2024-04-03T08:52:58.629596+0000","172.21.15.89:6832/43357":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/2329039936":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/3986748726":"2024-04-03T08:52:58.629596+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:53:07.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.733+0000 7f32e7800700 1 -- 172.21.15.89:0/1683307630 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32c006adb0 msgr2=0x7f32c006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:07.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.733+0000 7f32e7800700 1 --2- 172.21.15.89:0/1683307630 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32c006adb0 0x7f32c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.733+0000 7f32e7800700 1 -- 172.21.15.89:0/1683307630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e015f200 msgr2=0x7f32e00ccd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:07.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.733+0000 7f32e7800700 1 --2- 172.21.15.89:0/1683307630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e015f200 0x7f32e00ccd80 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f32e0070560 tx=0x7f32d8004ad0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.733+0000 7f32e7800700 1 -- 172.21.15.89:0/1683307630 shutdown_connections 2024-04-02T08:53:07.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.733+0000 7f32e7800700 1 --2- 172.21.15.89:0/1683307630 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f32c006adb0 0x7f32c006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.733+0000 7f32e7800700 1 --2- 172.21.15.89:0/1683307630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e015f200 0x7f32e00ccd80 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.733+0000 7f32e7800700 1 --2- 172.21.15.89:0/1683307630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32e0099b40 0x7f32e00c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.733+0000 7f32e7800700 1 --2- 172.21.15.89:0/1683307630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e015b7e0 0x7f32e00ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.733+0000 7f32e7800700 1 -- 172.21.15.89:0/1683307630 >> 172.21.15.89:0/1683307630 conn(0x7f32e00b5ab0 msgr2=0x7f32e015d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:07.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.733+0000 7f32e7800700 1 -- 172.21.15.89:0/1683307630 shutdown_connections 2024-04-02T08:53:07.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.733+0000 7f32e7800700 1 -- 172.21.15.89:0/1683307630 wait complete. 2024-04-02T08:53:07.751 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T08:53:07.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T08:53:07.780 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:07.781+0000 7f4b2ce59240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:53:07.918 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:07.919+0000 7fb0bdd5f240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T08:53:07.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.948+0000 7f811e600700 1 Processor -- start 2024-04-02T08:53:07.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.948+0000 7f811e600700 1 -- start start 2024-04-02T08:53:07.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.949+0000 7f811e600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8118160be0 0x7f81180cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.949+0000 7f811e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81180ce0e0 0x7f81180ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.949+0000 7f811e600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8118160fc0 con 0x7f81180cead0 2024-04-02T08:53:07.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.949+0000 7f811e600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81180cef60 con 0x7f8118160be0 2024-04-02T08:53:07.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.949+0000 7f811e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81180b65d0 con 0x7f81180ce0e0 2024-04-02T08:53:07.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.949+0000 7f8117e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f81180cead0 0x7f8118161240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45516/0 (socket says 172.21.15.89:45516) 2024-04-02T08:53:07.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.949+0000 7f8117e00700 1 -- 172.21.15.89:0/1434775500 learned_addr learned my addr 172.21.15.89:0/1434775500 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:07.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.949+0000 7f8117400700 1 --2- 172.21.15.89:0/1434775500 >> v2:172.21.15.113:3300/0 conn(0x7f8118160be0 0x7f81180cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.949+0000 7f8116a00700 1 --2- 172.21.15.89:0/1434775500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81180ce0e0 0x7f81180ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.950+0000 7f8116000700 1 -- 172.21.15.89:0/1434775500 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111397095 0 0) 0x7f8118160fc0 con 0x7f81180cead0 2024-04-02T08:53:07.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.950+0000 7f8116000700 1 -- 172.21.15.89:0/1434775500 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80fc0035b0 con 0x7f81180cead0 2024-04-02T08:53:07.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.950+0000 7f8117400700 1 -- 172.21.15.89:0/1434775500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81180ce0e0 msgr2=0x7f81180ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:07.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.950+0000 7f8117400700 1 --2- 172.21.15.89:0/1434775500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81180ce0e0 0x7f81180ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.950+0000 7f8117400700 1 -- 172.21.15.89:0/1434775500 >> v1:172.21.15.89:6789/0 conn(0x7f81180cead0 legacy=0x7f8118161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:07.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.950+0000 7f8117400700 1 -- 172.21.15.89:0/1434775500 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81180d21e0 con 0x7f8118160be0 2024-04-02T08:53:07.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.950+0000 7f8116a00700 1 --2- 172.21.15.89:0/1434775500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81180ce0e0 0x7f81180ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:07.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.950+0000 7f8117400700 1 --2- 172.21.15.89:0/1434775500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8118160be0 0x7f81180cdba0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8108009840 tx=0x7f8108009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25c1dad71d05f12d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:07.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.950+0000 7f8116000700 1 -- 172.21.15.89:0/1434775500 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108011830 con 0x7f8118160be0 2024-04-02T08:53:07.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.950+0000 7f8116000700 1 -- 172.21.15.89:0/1434775500 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8108011990 con 0x7f8118160be0 2024-04-02T08:53:07.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.950+0000 7f8116000700 1 -- 172.21.15.89:0/1434775500 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108011d20 con 0x7f8118160be0 2024-04-02T08:53:07.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.951+0000 7f811e600700 1 -- 172.21.15.89:0/1434775500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8118160be0 msgr2=0x7f81180cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:07.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.951+0000 7f811e600700 1 --2- 172.21.15.89:0/1434775500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8118160be0 0x7f81180cdba0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8108009840 tx=0x7f8108009b50 comp rx=0 tx=0).stop 2024-04-02T08:53:07.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.951+0000 7f811e600700 1 -- 172.21.15.89:0/1434775500 shutdown_connections 2024-04-02T08:53:07.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.951+0000 7f811e600700 1 --2- 172.21.15.89:0/1434775500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8118160be0 0x7f81180cdba0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.951+0000 7f811e600700 1 --2- 172.21.15.89:0/1434775500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81180ce0e0 0x7f81180ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.951+0000 7f811e600700 1 -- 172.21.15.89:0/1434775500 >> 172.21.15.89:0/1434775500 conn(0x7f811808aff0 msgr2=0x7f81180d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:07.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.951+0000 7f811e600700 1 -- 172.21.15.89:0/1434775500 shutdown_connections 2024-04-02T08:53:07.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.951+0000 7f811e600700 1 -- 172.21.15.89:0/1434775500 wait complete. 2024-04-02T08:53:07.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.953+0000 7f811e600700 1 Processor -- start 2024-04-02T08:53:07.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.953+0000 7f811e600700 1 -- start start 2024-04-02T08:53:07.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.954+0000 7f811e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81180ce0e0 0x7f811815d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.954+0000 7f811e600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81180cead0 0x7f81181588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.954+0000 7f811e600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8118160be0 0x7f81180cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.954+0000 7f811e600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81180cd550 con 0x7f81180cead0 2024-04-02T08:53:07.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.954+0000 7f811e600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811815d9e0 con 0x7f8118160be0 2024-04-02T08:53:07.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.954+0000 7f811e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81180cccb0 con 0x7f81180ce0e0 2024-04-02T08:53:07.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.954+0000 7f8116a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81180cead0 0x7f81181588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.954+0000 7f8116a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81180cead0 0x7f81181588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38626/0 (socket says 172.21.15.89:38626) 2024-04-02T08:53:07.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.954+0000 7f8116a00700 1 -- 172.21.15.89:0/3004301016 learned_addr learned my addr 172.21.15.89:0/3004301016 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:07.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.954+0000 7f8117e00700 1 --2- 172.21.15.89:0/3004301016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8118160be0 0x7f81180cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.954+0000 7f8117400700 1 --2- 172.21.15.89:0/3004301016 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81180ce0e0 0x7f811815d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:07.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.955+0000 7f8116a00700 1 -- 172.21.15.89:0/3004301016 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81180ce0e0 msgr2=0x7f811815d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:07.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.955+0000 7f8116a00700 1 --2- 172.21.15.89:0/3004301016 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81180ce0e0 0x7f811815d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.955+0000 7f8116a00700 1 -- 172.21.15.89:0/3004301016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8118160be0 msgr2=0x7f81180cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:07.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.955+0000 7f8116a00700 1 --2- 172.21.15.89:0/3004301016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8118160be0 0x7f81180cc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:07.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.955+0000 7f8116a00700 1 -- 172.21.15.89:0/3004301016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81040094a0 con 0x7f81180cead0 2024-04-02T08:53:07.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.955+0000 7f8116a00700 1 --2- 172.21.15.89:0/3004301016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81180cead0 0x7f81181588f0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f810400ebe0 tx=0x7f810400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:07.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.955+0000 7f810f400700 1 -- 172.21.15.89:0/3004301016 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810400c850 con 0x7f81180cead0 2024-04-02T08:53:07.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.955+0000 7f810f400700 1 -- 172.21.15.89:0/3004301016 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810400c9b0 con 0x7f81180cead0 2024-04-02T08:53:07.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.955+0000 7f811e600700 1 -- 172.21.15.89:0/3004301016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81080094a0 con 0x7f81180cead0 2024-04-02T08:53:07.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.955+0000 7f810f400700 1 -- 172.21.15.89:0/3004301016 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810400cd40 con 0x7f81180cead0 2024-04-02T08:53:07.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.955+0000 7f811e600700 1 -- 172.21.15.89:0/3004301016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81180cd270 con 0x7f81180cead0 2024-04-02T08:53:07.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.956+0000 7f811e600700 1 -- 172.21.15.89:0/3004301016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8118067720 con 0x7f81180cead0 2024-04-02T08:53:07.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.957+0000 7f810f400700 1 -- 172.21.15.89:0/3004301016 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f8104013470 con 0x7f81180cead0 2024-04-02T08:53:07.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.958+0000 7f810f400700 1 --2- 172.21.15.89:0/3004301016 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f80f806adb0 0x7f80f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:07.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.958+0000 7f8117400700 1 -- 172.21.15.89:0/3004301016 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f80f806adb0 msgr2=0x7f80f806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:07.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.958+0000 7f810f400700 1 -- 172.21.15.89:0/3004301016 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f81040898c0 con 0x7f81180cead0 2024-04-02T08:53:07.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.958+0000 7f8117400700 1 --2- 172.21.15.89:0/3004301016 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f80f806adb0 0x7f80f806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:07.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:07.967+0000 7f810f400700 1 -- 172.21.15.89:0/3004301016 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f81040545f0 con 0x7f81180cead0 2024-04-02T08:53:08.138 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:08.139+0000 7fb0bdd5f240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:53:08.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.159+0000 7f8117400700 1 -- 172.21.15.89:0/3004301016 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f80f806adb0 msgr2=0x7f80f806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:08.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.159+0000 7f8117400700 1 --2- 172.21.15.89:0/3004301016 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f80f806adb0 0x7f80f806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:08.216 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:08.217+0000 7f4b2ce59240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T08:53:08.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.220+0000 7f811e600700 1 -- 172.21.15.89:0/3004301016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81180b5cb0 con 0x7f81180cead0 2024-04-02T08:53:08.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.221+0000 7f810f400700 1 -- 172.21.15.89:0/3004301016 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f810400cea0 con 0x7f81180cead0 2024-04-02T08:53:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:08.221 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8881,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/547244866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":547244866},{"type":"v1","addr":"172.21.15.113:6834","nonce":547244866}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":8884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2822200553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2822200553},{"type":"v1","addr":"172.21.15.113:6835","nonce":2822200553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/342426898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":342426898},{"type":"v1","addr":"172.21.15.89:6835","nonce":342426898}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T08:52:42.484905+0000","modified":"2024-04-02T08:52:45.498450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9534},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9534":{"gid":9534,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2812511535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2812511535},{"type":"v1","addr":"172.21.15.89:6837","nonce":2812511535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-02T08:53:08.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.224+0000 7f811e600700 1 -- 172.21.15.89:0/3004301016 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f80f806adb0 msgr2=0x7f80f806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:08.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.224+0000 7f811e600700 1 --2- 172.21.15.89:0/3004301016 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f80f806adb0 0x7f80f806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.224+0000 7f811e600700 1 -- 172.21.15.89:0/3004301016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81180cead0 msgr2=0x7f81181588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:08.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.224+0000 7f811e600700 1 --2- 172.21.15.89:0/3004301016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81180cead0 0x7f81181588f0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f810400ebe0 tx=0x7f810400eef0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.224+0000 7f811e600700 1 -- 172.21.15.89:0/3004301016 shutdown_connections 2024-04-02T08:53:08.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.224+0000 7f811e600700 1 --2- 172.21.15.89:0/3004301016 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f80f806adb0 0x7f80f806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.224+0000 7f811e600700 1 --2- 172.21.15.89:0/3004301016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8118160be0 0x7f81180cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.224+0000 7f811e600700 1 --2- 172.21.15.89:0/3004301016 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81180ce0e0 0x7f811815d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.224+0000 7f811e600700 1 --2- 172.21.15.89:0/3004301016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81180cead0 0x7f81181588f0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.224+0000 7f811e600700 1 -- 172.21.15.89:0/3004301016 >> 172.21.15.89:0/3004301016 conn(0x7f811808aff0 msgr2=0x7f81180b7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:08.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.224+0000 7f811e600700 1 -- 172.21.15.89:0/3004301016 shutdown_connections 2024-04-02T08:53:08.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.224+0000 7f811e600700 1 -- 172.21.15.89:0/3004301016 wait complete. 2024-04-02T08:53:08.225 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 81 2024-04-02T08:53:08.243 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T08:53:08.325 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:08.326+0000 7f4b2ce59240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T08:53:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.442+0000 7f9cb5a00700 1 Processor -- start 2024-04-02T08:53:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.442+0000 7f9cb5a00700 1 -- start start 2024-04-02T08:53:08.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.443+0000 7f9cb5a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9cb015b5d0 0x7f9cb00ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.443+0000 7f9cb5a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cb00cb3f0 0x7f9cb00ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.443+0000 7f9cb5a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb015fd50 con 0x7f9cb00ae930 2024-04-02T08:53:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.443+0000 7f9cb5a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb015b9b0 con 0x7f9cb015b5d0 2024-04-02T08:53:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.443+0000 7f9cb5a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb015f140 con 0x7f9cb00cb3f0 2024-04-02T08:53:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.444+0000 7f9caea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9cb015b5d0 0x7f9cb00ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.444+0000 7f9caea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9cb015b5d0 0x7f9cb00ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38636/0 (socket says 172.21.15.89:38636) 2024-04-02T08:53:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.444+0000 7f9caea00700 1 -- 172.21.15.89:0/3149895308 learned_addr learned my addr 172.21.15.89:0/3149895308 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.444+0000 7f9cae000700 1 --2- 172.21.15.89:0/3149895308 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cb00cb3f0 0x7f9cb00ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.444+0000 7f9caea00700 1 -- 172.21.15.89:0/3149895308 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cb00cb3f0 msgr2=0x7f9cb00ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.444+0000 7f9caea00700 1 --2- 172.21.15.89:0/3149895308 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cb00cb3f0 0x7f9cb00ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.444+0000 7f9caea00700 1 -- 172.21.15.89:0/3149895308 >> v1:172.21.15.113:6789/0 conn(0x7f9cb00ae930 legacy=0x7f9cb0140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.444+0000 7f9caea00700 1 -- 172.21.15.89:0/3149895308 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb015f420 con 0x7f9cb015b5d0 2024-04-02T08:53:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.445+0000 7f9caea00700 1 --2- 172.21.15.89:0/3149895308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cb015b5d0 0x7f9cb00ae3f0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00098f0 tx=0x7f9ca0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277b7d479ace78f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.445+0000 7f9cad600700 1 -- 172.21.15.89:0/3149895308 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca000b8a0 con 0x7f9cb015b5d0 2024-04-02T08:53:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.445+0000 7f9cad600700 1 -- 172.21.15.89:0/3149895308 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca000ba00 con 0x7f9cb015b5d0 2024-04-02T08:53:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.445+0000 7f9cad600700 1 -- 172.21.15.89:0/3149895308 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca000bd90 con 0x7f9cb015b5d0 2024-04-02T08:53:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.445+0000 7f9cb5a00700 1 -- 172.21.15.89:0/3149895308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cb015b5d0 msgr2=0x7f9cb00ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.445+0000 7f9cb5a00700 1 --2- 172.21.15.89:0/3149895308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cb015b5d0 0x7f9cb00ae3f0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00098f0 tx=0x7f9ca0009c00 comp rx=0 tx=0).stop 2024-04-02T08:53:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.446+0000 7f9cb5a00700 1 -- 172.21.15.89:0/3149895308 shutdown_connections 2024-04-02T08:53:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.446+0000 7f9cb5a00700 1 --2- 172.21.15.89:0/3149895308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cb015b5d0 0x7f9cb00ae3f0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.446+0000 7f9cb5a00700 1 --2- 172.21.15.89:0/3149895308 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cb00cb3f0 0x7f9cb00ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.446+0000 7f9cb5a00700 1 -- 172.21.15.89:0/3149895308 >> 172.21.15.89:0/3149895308 conn(0x7f9cb00cf190 msgr2=0x7f9cb00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.446+0000 7f9cb5a00700 1 -- 172.21.15.89:0/3149895308 shutdown_connections 2024-04-02T08:53:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.446+0000 7f9cb5a00700 1 -- 172.21.15.89:0/3149895308 wait complete. 2024-04-02T08:53:08.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.448+0000 7f9cb5a00700 1 Processor -- start 2024-04-02T08:53:08.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.448+0000 7f9cb5a00700 1 -- start start 2024-04-02T08:53:08.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.449+0000 7f9cb5a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cb00ae930 0x7f9cb0158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:08.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.449+0000 7f9cb5a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cb015b5d0 0x7f9cb0159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:08.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.449+0000 7f9cb5a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cb01f6740 0x7f9cb01f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:08.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.449+0000 7f9cb5a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb00ca230 con 0x7f9cb01f6740 2024-04-02T08:53:08.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.449+0000 7f9cb5a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb01f7100 con 0x7f9cb015b5d0 2024-04-02T08:53:08.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.449+0000 7f9cb5a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb0159820 con 0x7f9cb00ae930 2024-04-02T08:53:08.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.449+0000 7f9caf400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cb01f6740 0x7f9cb01f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:08.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.449+0000 7f9caf400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cb01f6740 0x7f9cb01f6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38652/0 (socket says 172.21.15.89:38652) 2024-04-02T08:53:08.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.449+0000 7f9caf400700 1 -- 172.21.15.89:0/1606067794 learned_addr learned my addr 172.21.15.89:0/1606067794 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:08.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.449+0000 7f9cae000700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cb015b5d0 0x7f9cb0159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:08.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.449+0000 7f9caea00700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cb00ae930 0x7f9cb0158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:08.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.450+0000 7f9cae000700 1 -- 172.21.15.89:0/1606067794 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cb00ae930 msgr2=0x7f9cb0158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:08.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.450+0000 7f9cae000700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cb00ae930 0x7f9cb0158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.450+0000 7f9cae000700 1 -- 172.21.15.89:0/1606067794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cb01f6740 msgr2=0x7f9cb01f6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:08.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.450+0000 7f9cae000700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cb01f6740 0x7f9cb01f6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.450+0000 7f9cae000700 1 -- 172.21.15.89:0/1606067794 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca0009550 con 0x7f9cb015b5d0 2024-04-02T08:53:08.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.450+0000 7f9caf400700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cb01f6740 0x7f9cb01f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:08.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.450+0000 7f9caea00700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cb00ae930 0x7f9cb0158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:08.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.450+0000 7f9cae000700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cb015b5d0 0x7f9cb0159210 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800e810 tx=0x7f9c9800eb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:08.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.450+0000 7f9ca6a00700 1 -- 172.21.15.89:0/1606067794 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9800c070 con 0x7f9cb015b5d0 2024-04-02T08:53:08.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.450+0000 7f9ca6a00700 1 -- 172.21.15.89:0/1606067794 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c98004020 con 0x7f9cb015b5d0 2024-04-02T08:53:08.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.450+0000 7f9ca6a00700 1 -- 172.21.15.89:0/1606067794 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c980043d0 con 0x7f9cb015b5d0 2024-04-02T08:53:08.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.450+0000 7f9cb5a00700 1 -- 172.21.15.89:0/1606067794 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb0159b00 con 0x7f9cb015b5d0 2024-04-02T08:53:08.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.450+0000 7f9cb5a00700 1 -- 172.21.15.89:0/1606067794 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb015a0c0 con 0x7f9cb015b5d0 2024-04-02T08:53:08.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.451+0000 7f9cb5a00700 1 -- 172.21.15.89:0/1606067794 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb0067720 con 0x7f9cb015b5d0 2024-04-02T08:53:08.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.452+0000 7f9ca6a00700 1 -- 172.21.15.89:0/1606067794 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 109) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f9c980036a0 con 0x7f9cb015b5d0 2024-04-02T08:53:08.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.453+0000 7f9ca6a00700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c8c06adb0 0x7f9c8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:08.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.453+0000 7f9caea00700 1 -- 172.21.15.89:0/1606067794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c8c06adb0 msgr2=0x7f9c8c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:08.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.453+0000 7f9ca6a00700 1 -- 172.21.15.89:0/1606067794 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f9c98089270 con 0x7f9cb015b5d0 2024-04-02T08:53:08.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.453+0000 7f9caea00700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c8c06adb0 0x7f9c8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:08.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.458+0000 7f9ca6a00700 1 -- 172.21.15.89:0/1606067794 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9c98053f30 con 0x7f9cb015b5d0 2024-04-02T08:53:08.595 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:08.595+0000 7fb0bdd5f240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T08:53:08.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.654+0000 7f9caea00700 1 -- 172.21.15.89:0/1606067794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c8c06adb0 msgr2=0x7f9c8c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:08.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.654+0000 7f9caea00700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c8c06adb0 0x7f9c8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.670+0000 7f9ca6a00700 1 -- 172.21.15.89:0/1606067794 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mgrmap(e 110) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f9c98087490 con 0x7f9cb015b5d0 2024-04-02T08:53:08.710 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:08.710+0000 7fb0bdd5f240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T08:53:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.712+0000 7f9cb5a00700 1 -- 172.21.15.89:0/1606067794 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cb015e590 con 0x7f9cb015b5d0 2024-04-02T08:53:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.712+0000 7f9ca6a00700 1 -- 172.21.15.89:0/1606067794 <== mon.1 v2:172.21.15.113:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f9c980849d0 con 0x7f9cb015b5d0 2024-04-02T08:53:08.712 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:08.712 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8881,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/547244866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":547244866},{"type":"v1","addr":"172.21.15.113:6834","nonce":547244866}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":8884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2822200553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2822200553},{"type":"v1","addr":"172.21.15.113:6835","nonce":2822200553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/342426898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":342426898},{"type":"v1","addr":"172.21.15.89:6835","nonce":342426898}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T08:52:42.484905+0000","modified":"2024-04-02T08:52:45.498450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9534},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9534":{"gid":9534,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2812511535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2812511535},{"type":"v1","addr":"172.21.15.89:6837","nonce":2812511535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-02T08:53:08.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.715+0000 7f9cb5a00700 1 -- 172.21.15.89:0/1606067794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c8c06adb0 msgr2=0x7f9c8c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:08.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.715+0000 7f9cb5a00700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c8c06adb0 0x7f9c8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.715+0000 7f9cb5a00700 1 -- 172.21.15.89:0/1606067794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cb015b5d0 msgr2=0x7f9cb0159210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:08.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.715+0000 7f9cb5a00700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cb015b5d0 0x7f9cb0159210 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800e810 tx=0x7f9c9800eb20 comp rx=0 tx=0).stop 2024-04-02T08:53:08.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.715+0000 7f9cb5a00700 1 -- 172.21.15.89:0/1606067794 shutdown_connections 2024-04-02T08:53:08.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.715+0000 7f9cb5a00700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f9c8c06adb0 0x7f9c8c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.715+0000 7f9cb5a00700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cb01f6740 0x7f9cb01f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.715+0000 7f9cb5a00700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cb00ae930 0x7f9cb0158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.715+0000 7f9cb5a00700 1 --2- 172.21.15.89:0/1606067794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cb015b5d0 0x7f9cb0159210 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.715+0000 7f9cb5a00700 1 -- 172.21.15.89:0/1606067794 >> 172.21.15.89:0/1606067794 conn(0x7f9cb00cf190 msgr2=0x7f9cb00aedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:08.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.715+0000 7f9cb5a00700 1 -- 172.21.15.89:0/1606067794 shutdown_connections 2024-04-02T08:53:08.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.715+0000 7f9cb5a00700 1 -- 172.21.15.89:0/1606067794 wait complete. 2024-04-02T08:53:08.716 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 81 2024-04-02T08:53:08.739 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:53:08.794 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:08.794+0000 7fb02d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:08.795 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:08.794+0000 7fb02d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:08.796 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:08.794+0000 7fb02d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:08.796 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:08.794+0000 7fb02d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:08.796 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:08.794+0000 7fb02d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:08.796 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:08.795+0000 7fb030e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:08.796 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:08.795+0000 7fb030e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:08.796 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:08.795+0000 7fb030e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:08.796 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:08.795+0000 7fb030e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:08.796 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:08.795+0000 7fb030e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:08.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.941+0000 7fb591600700 1 Processor -- start 2024-04-02T08:53:08.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.941+0000 7fb591600700 1 -- start start 2024-04-02T08:53:08.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb591600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb58c15d640 0x7fb58c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:08.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb591600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb58c0ceec0 0x7fb58c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:08.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb591600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb58c0d30d0 con 0x7fb58c161800 2024-04-02T08:53:08.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb591600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb58c0b6840 con 0x7fb58c15d640 2024-04-02T08:53:08.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb591600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb58c161c20 con 0x7fb58c0ceec0 2024-04-02T08:53:08.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb58aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb58c15d640 0x7fb58c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:08.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb58aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb58c15d640 0x7fb58c0d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53306/0 (socket says 172.21.15.89:53306) 2024-04-02T08:53:08.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb58aa00700 1 -- 172.21.15.89:0/2945477523 learned_addr learned my addr 172.21.15.89:0/2945477523 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:08.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb58aa00700 1 -- 172.21.15.89:0/2945477523 >> v2:172.21.15.113:3300/0 conn(0x7fb58c0ceec0 msgr2=0x7fb58c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:08.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb58a000700 1 --2- 172.21.15.89:0/2945477523 >> v2:172.21.15.113:3300/0 conn(0x7fb58c0ceec0 0x7fb58c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:08.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb58aa00700 1 --2- 172.21.15.89:0/2945477523 >> v2:172.21.15.113:3300/0 conn(0x7fb58c0ceec0 0x7fb58c1612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb58aa00700 1 -- 172.21.15.89:0/2945477523 >> v1:172.21.15.113:6789/0 conn(0x7fb58c161800 legacy=0x7fb58c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:08.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb58aa00700 1 -- 172.21.15.89:0/2945477523 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58c0d2880 con 0x7fb58c15d640 2024-04-02T08:53:08.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb58a000700 1 --2- 172.21.15.89:0/2945477523 >> v2:172.21.15.113:3300/0 conn(0x7fb58c0ceec0 0x7fb58c1612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:53:08.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb58aa00700 1 --2- 172.21.15.89:0/2945477523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb58c15d640 0x7fb58c0d2340 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fb58c0cfbf0 tx=0x7fb57c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c80b89b5bad10ca6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:08.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb589600700 1 -- 172.21.15.89:0/2945477523 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c004230 con 0x7fb58c15d640 2024-04-02T08:53:08.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.943+0000 7fb589600700 1 -- 172.21.15.89:0/2945477523 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57c004390 con 0x7fb58c15d640 2024-04-02T08:53:08.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.945+0000 7fb589600700 1 -- 172.21.15.89:0/2945477523 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c0036a0 con 0x7fb58c15d640 2024-04-02T08:53:08.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.945+0000 7fb591600700 1 -- 172.21.15.89:0/2945477523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb58c15d640 msgr2=0x7fb58c0d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:08.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.945+0000 7fb591600700 1 --2- 172.21.15.89:0/2945477523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb58c15d640 0x7fb58c0d2340 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fb58c0cfbf0 tx=0x7fb57c009bd0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.945+0000 7fb591600700 1 -- 172.21.15.89:0/2945477523 shutdown_connections 2024-04-02T08:53:08.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.945+0000 7fb591600700 1 --2- 172.21.15.89:0/2945477523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb58c15d640 0x7fb58c0d2340 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.945+0000 7fb591600700 1 --2- 172.21.15.89:0/2945477523 >> v2:172.21.15.113:3300/0 conn(0x7fb58c0ceec0 0x7fb58c1612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.945+0000 7fb591600700 1 -- 172.21.15.89:0/2945477523 >> 172.21.15.89:0/2945477523 conn(0x7fb58c08aff0 msgr2=0x7fb58c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:08.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.945+0000 7fb591600700 1 -- 172.21.15.89:0/2945477523 shutdown_connections 2024-04-02T08:53:08.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.945+0000 7fb591600700 1 -- 172.21.15.89:0/2945477523 wait complete. 2024-04-02T08:53:08.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.947+0000 7fb591600700 1 Processor -- start 2024-04-02T08:53:08.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.948+0000 7fb591600700 1 -- start start 2024-04-02T08:53:08.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.949+0000 7fb591600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb58c0ceec0 0x7fb58c1f6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:08.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.949+0000 7fb591600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb58c15d640 0x7fb58c154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:08.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.949+0000 7fb591600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb58c161800 0x7fb58c154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:08.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.949+0000 7fb591600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58c0c8300 con 0x7fb58c161800 2024-04-02T08:53:08.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.949+0000 7fb591600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58c1f7450 con 0x7fb58c0ceec0 2024-04-02T08:53:08.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.949+0000 7fb591600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb58c155aa0 con 0x7fb58c15d640 2024-04-02T08:53:08.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.949+0000 7fb58b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb58c161800 0x7fb58c154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:08.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.949+0000 7fb58aa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb58c0ceec0 0x7fb58c1f6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:08.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.949+0000 7fb58aa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb58c0ceec0 0x7fb58c1f6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33546/0 (socket says 172.21.15.89:33546) 2024-04-02T08:53:08.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.949+0000 7fb58b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb58c161800 0x7fb58c154b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53318/0 (socket says 172.21.15.89:53318) 2024-04-02T08:53:08.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.949+0000 7fb58aa00700 1 -- 172.21.15.89:0/3163210882 learned_addr learned my addr 172.21.15.89:0/3163210882 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:08.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.949+0000 7fb58a000700 1 --2- 172.21.15.89:0/3163210882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb58c15d640 0x7fb58c154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:08.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.950+0000 7fb58b400700 1 -- 172.21.15.89:0/3163210882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb58c15d640 msgr2=0x7fb58c154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:08.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.950+0000 7fb58b400700 1 --2- 172.21.15.89:0/3163210882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb58c15d640 0x7fb58c154620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.950+0000 7fb58b400700 1 -- 172.21.15.89:0/3163210882 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb58c0ceec0 msgr2=0x7fb58c1f6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:08.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.950+0000 7fb58b400700 1 --2- 172.21.15.89:0/3163210882 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb58c0ceec0 0x7fb58c1f6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:08.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.950+0000 7fb58b400700 1 -- 172.21.15.89:0/3163210882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c009550 con 0x7fb58c161800 2024-04-02T08:53:08.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.950+0000 7fb58b400700 1 --2- 172.21.15.89:0/3163210882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb58c161800 0x7fb58c154b60 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fb58400cb40 tx=0x7fb58400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:08.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.950+0000 7fb582a00700 1 -- 172.21.15.89:0/3163210882 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58400a9a0 con 0x7fb58c161800 2024-04-02T08:53:08.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.950+0000 7fb582a00700 1 -- 172.21.15.89:0/3163210882 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58400ab00 con 0x7fb58c161800 2024-04-02T08:53:08.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.950+0000 7fb582a00700 1 -- 172.21.15.89:0/3163210882 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb584012420 con 0x7fb58c161800 2024-04-02T08:53:08.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.950+0000 7fb591600700 1 -- 172.21.15.89:0/3163210882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb58c155200 con 0x7fb58c161800 2024-04-02T08:53:08.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.950+0000 7fb591600700 1 -- 172.21.15.89:0/3163210882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58c1557c0 con 0x7fb58c161800 2024-04-02T08:53:08.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.951+0000 7fb591600700 1 -- 172.21.15.89:0/3163210882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb58c0c8440 con 0x7fb58c161800 2024-04-02T08:53:08.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.952+0000 7fb582a00700 1 -- 172.21.15.89:0/3163210882 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 111) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fb584012580 con 0x7fb58c161800 2024-04-02T08:53:08.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.952+0000 7fb582a00700 1 -- 172.21.15.89:0/3163210882 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6731+0+0 (secure 0 0 0) 0x7fb584089740 con 0x7fb58c161800 2024-04-02T08:53:08.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:08.958+0000 7fb582a00700 1 -- 172.21.15.89:0/3163210882 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb58c0c8440 con 0x7fb58c161800 2024-04-02T08:53:09.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.160+0000 7fb591600700 1 -- 172.21.15.89:0/3163210882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb58c0c8440 con 0x7fb58c161800 2024-04-02T08:53:09.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.162+0000 7fb582a00700 1 -- 172.21.15.89:0/3163210882 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+17612 (secure 0 0 0) 0x7fb58c0c8440 con 0x7fb58c161800 2024-04-02T08:53:09.160 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:09.161 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":174,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:53:08.714271+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-02T08:52:40.806770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-02T08:52:41.463828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/4076742655":"2024-04-03T08:53:08.714235+0000","172.21.15.89:0/2959654621":"2024-04-03T08:53:08.714235+0000","172.21.15.89:0/954151624":"2024-04-03T08:53:08.714235+0000","172.21.15.89:0/3459982277":"2024-04-03T08:53:08.714235+0000","172.21.15.89:0/970708539":"2024-04-03T08:53:08.714235+0000","172.21.15.89:0/3680638162":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/263272397":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/404616533":"2024-04-03T08:53:08.714235+0000","172.21.15.89:0/1641401996":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/419225467":"2024-04-03T08:52:58.629596+0000","172.21.15.89:6833/43357":"2024-04-03T08:52:58.629596+0000","172.21.15.89:6832/43357":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/2329039936":"2024-04-03T08:52:58.629596+0000","172.21.15.89:0/3986748726":"2024-04-03T08:52:58.629596+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:53:09.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.164+0000 7fb591600700 1 -- 172.21.15.89:0/3163210882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb58c161800 msgr2=0x7fb58c154b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:09.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.164+0000 7fb591600700 1 --2- 172.21.15.89:0/3163210882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb58c161800 0x7fb58c154b60 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fb58400cb40 tx=0x7fb58400ce50 comp rx=0 tx=0).stop 2024-04-02T08:53:09.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.164+0000 7fb591600700 1 -- 172.21.15.89:0/3163210882 shutdown_connections 2024-04-02T08:53:09.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.164+0000 7fb591600700 1 --2- 172.21.15.89:0/3163210882 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb58c0ceec0 0x7fb58c1f6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.164+0000 7fb591600700 1 --2- 172.21.15.89:0/3163210882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb58c161800 0x7fb58c154b60 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.164+0000 7fb591600700 1 --2- 172.21.15.89:0/3163210882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb58c15d640 0x7fb58c154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.164+0000 7fb591600700 1 -- 172.21.15.89:0/3163210882 >> 172.21.15.89:0/3163210882 conn(0x7fb58c08aff0 msgr2=0x7fb58c0b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:09.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.164+0000 7fb591600700 1 -- 172.21.15.89:0/3163210882 shutdown_connections 2024-04-02T08:53:09.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.164+0000 7fb591600700 1 -- 172.21.15.89:0/3163210882 wait complete. 2024-04-02T08:53:09.180 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T08:53:09.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.375+0000 7fda5e800700 1 Processor -- start 2024-04-02T08:53:09.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.376+0000 7fda5e800700 1 -- start start 2024-04-02T08:53:09.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.377+0000 7fda5e800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fda580b3f80 0x7fda580ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:09.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.377+0000 7fda5e800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda58161230 con 0x7fda58160e50 2024-04-02T08:53:09.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.377+0000 7fda5e800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda580d2a80 con 0x7fda580d3f20 2024-04-02T08:53:09.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.377+0000 7fda5e800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda580d43b0 con 0x7fda580b3f80 2024-04-02T08:53:09.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.377+0000 7fda57400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fda58160e50 0x7fda580b4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58124/0 (socket says 172.21.15.89:58124) 2024-04-02T08:53:09.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.377+0000 7fda57400700 1 -- 172.21.15.89:0/385881034 learned_addr learned my addr 172.21.15.89:0/385881034 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:09.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.377+0000 7fda57e00700 1 --2- 172.21.15.89:0/385881034 >> v2:172.21.15.89:3300/0 conn(0x7fda580b3f80 0x7fda580ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:09.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.377+0000 7fda56000700 1 -- 172.21.15.89:0/385881034 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069514126 0 0) 0x7fda58161230 con 0x7fda58160e50 2024-04-02T08:53:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.377+0000 7fda56000700 1 -- 172.21.15.89:0/385881034 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda3c0035b0 con 0x7fda58160e50 2024-04-02T08:53:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.377+0000 7fda57e00700 1 -- 172.21.15.89:0/385881034 >> v1:172.21.15.113:6789/0 conn(0x7fda580d3f20 legacy=0x7fda580cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.377+0000 7fda57e00700 1 -- 172.21.15.89:0/385881034 >> v1:172.21.15.89:6789/0 conn(0x7fda58160e50 legacy=0x7fda580b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.378+0000 7fda57e00700 1 -- 172.21.15.89:0/385881034 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda580d21e0 con 0x7fda580b3f80 2024-04-02T08:53:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.378+0000 7fda57e00700 1 --2- 172.21.15.89:0/385881034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda580b3f80 0x7fda580ce600 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fda50009d90 tx=0x7fda5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c204c9f108bc0a2b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.378+0000 7fda56000700 1 -- 172.21.15.89:0/385881034 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5000e070 con 0x7fda580b3f80 2024-04-02T08:53:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.378+0000 7fda56000700 1 -- 172.21.15.89:0/385881034 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5000bb20 con 0x7fda580b3f80 2024-04-02T08:53:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.378+0000 7fda56000700 1 -- 172.21.15.89:0/385881034 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5000be90 con 0x7fda580b3f80 2024-04-02T08:53:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.378+0000 7fda5e800700 1 -- 172.21.15.89:0/385881034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda580b3f80 msgr2=0x7fda580ce600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.379+0000 7fda5e800700 1 --2- 172.21.15.89:0/385881034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda580b3f80 0x7fda580ce600 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fda50009d90 tx=0x7fda5000b330 comp rx=0 tx=0).stop 2024-04-02T08:53:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.379+0000 7fda5e800700 1 -- 172.21.15.89:0/385881034 shutdown_connections 2024-04-02T08:53:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.379+0000 7fda5e800700 1 --2- 172.21.15.89:0/385881034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda580b3f80 0x7fda580ce600 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.379+0000 7fda5e800700 1 -- 172.21.15.89:0/385881034 >> 172.21.15.89:0/385881034 conn(0x7fda5808aff0 msgr2=0x7fda580d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:09.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.379+0000 7fda5e800700 1 -- 172.21.15.89:0/385881034 shutdown_connections 2024-04-02T08:53:09.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.379+0000 7fda5e800700 1 -- 172.21.15.89:0/385881034 wait complete. 2024-04-02T08:53:09.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.380+0000 7fda5e800700 1 Processor -- start 2024-04-02T08:53:09.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.381+0000 7fda5e800700 1 -- start start 2024-04-02T08:53:09.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.381+0000 7fda5e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda580b3f80 0x7fda58159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.381+0000 7fda5e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda58160e50 0x7fda581f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.381+0000 7fda5e800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda5815bf40 0x7fda580b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda5e800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda581f40e0 con 0x7fda58160e50 2024-04-02T08:53:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda5e800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda580b6280 con 0x7fda5815bf40 2024-04-02T08:53:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda5e800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda581f3840 con 0x7fda580b3f80 2024-04-02T08:53:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda56a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda58160e50 0x7fda581f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda56a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda58160e50 0x7fda581f2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53322/0 (socket says 172.21.15.89:53322) 2024-04-02T08:53:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda56a00700 1 -- 172.21.15.89:0/1054179629 learned_addr learned my addr 172.21.15.89:0/1054179629 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda57400700 1 --2- 172.21.15.89:0/1054179629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda580b3f80 0x7fda58159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda57e00700 1 --2- 172.21.15.89:0/1054179629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda5815bf40 0x7fda580b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda56a00700 1 -- 172.21.15.89:0/1054179629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda580b3f80 msgr2=0x7fda58159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda56a00700 1 --2- 172.21.15.89:0/1054179629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda580b3f80 0x7fda58159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda56a00700 1 -- 172.21.15.89:0/1054179629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda5815bf40 msgr2=0x7fda580b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda56a00700 1 --2- 172.21.15.89:0/1054179629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda5815bf40 0x7fda580b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda56a00700 1 -- 172.21.15.89:0/1054179629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda50009a70 con 0x7fda58160e50 2024-04-02T08:53:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda57400700 1 --2- 172.21.15.89:0/1054179629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda580b3f80 0x7fda58159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.382+0000 7fda56a00700 1 --2- 172.21.15.89:0/1054179629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda58160e50 0x7fda581f2ec0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fda4400c4e0 tx=0x7fda4400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.383+0000 7fda4f400700 1 -- 172.21.15.89:0/1054179629 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda440052d0 con 0x7fda58160e50 2024-04-02T08:53:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.383+0000 7fda4f400700 1 -- 172.21.15.89:0/1054179629 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda44005430 con 0x7fda58160e50 2024-04-02T08:53:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.383+0000 7fda4f400700 1 -- 172.21.15.89:0/1054179629 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda440038c0 con 0x7fda58160e50 2024-04-02T08:53:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.383+0000 7fda5e800700 1 -- 172.21.15.89:0/1054179629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda581f3b20 con 0x7fda58160e50 2024-04-02T08:53:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.383+0000 7fda5e800700 1 -- 172.21.15.89:0/1054179629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda581f3400 con 0x7fda58160e50 2024-04-02T08:53:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.384+0000 7fda5e800700 1 -- 172.21.15.89:0/1054179629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda580b6430 con 0x7fda58160e50 2024-04-02T08:53:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.385+0000 7fda4f400700 1 -- 172.21.15.89:0/1054179629 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 111) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fda44003a20 con 0x7fda58160e50 2024-04-02T08:53:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.386+0000 7fda4f400700 1 -- 172.21.15.89:0/1054179629 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6731+0+0 (secure 0 0 0) 0x7fda44088770 con 0x7fda58160e50 2024-04-02T08:53:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.390+0000 7fda4f400700 1 -- 172.21.15.89:0/1054179629 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fda440538b0 con 0x7fda58160e50 2024-04-02T08:53:09.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.637+0000 7fda5e800700 1 -- 172.21.15.89:0/1054179629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fda58067720 con 0x7fda58160e50 2024-04-02T08:53:09.636 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T08:53:09.638+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T08:53:09.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.651+0000 7fda4f400700 1 -- 172.21.15.89:0/1054179629 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v82) v1 ==== 161+0+0 (secure 0 0 0) 0x7fda44056ed0 con 0x7fda58160e50 2024-04-02T08:53:09.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.653+0000 7fda5e800700 1 -- 172.21.15.89:0/1054179629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda58160e50 msgr2=0x7fda581f2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:09.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.653+0000 7fda5e800700 1 --2- 172.21.15.89:0/1054179629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda58160e50 0x7fda581f2ec0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fda4400c4e0 tx=0x7fda4400c8a0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.653+0000 7fda5e800700 1 -- 172.21.15.89:0/1054179629 shutdown_connections 2024-04-02T08:53:09.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.653+0000 7fda5e800700 1 --2- 172.21.15.89:0/1054179629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda5815bf40 0x7fda580b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.653+0000 7fda5e800700 1 --2- 172.21.15.89:0/1054179629 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda580b3f80 0x7fda58159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.653+0000 7fda5e800700 1 --2- 172.21.15.89:0/1054179629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda58160e50 0x7fda581f2ec0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.653+0000 7fda5e800700 1 -- 172.21.15.89:0/1054179629 >> 172.21.15.89:0/1054179629 conn(0x7fda5808aff0 msgr2=0x7fda580b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:09.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.653+0000 7fda5e800700 1 -- 172.21.15.89:0/1054179629 shutdown_connections 2024-04-02T08:53:09.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.653+0000 7fda5e800700 1 -- 172.21.15.89:0/1054179629 wait complete. 2024-04-02T08:53:09.653 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T08:53:09.659 INFO:tasks.ceph.mds.a.smithi089.stderr: -4959> 2024-04-02T08:52:47.962+0000 7f36b4a00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T08:53:09.672 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T08:53:09.713 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T08:53:09.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.880+0000 7f2072e00700 1 Processor -- start 2024-04-02T08:53:09.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.880+0000 7f2072e00700 1 -- start start 2024-04-02T08:53:09.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.881+0000 7f2072e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f206c15d6d0 0x7f206c161430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:09.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.881+0000 7f2072e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f206c0d40f0 0x7f206c161970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:09.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.881+0000 7f2072e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c15dab0 0x7f206c0d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:09.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.882+0000 7f2072e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f206c0b2d20 con 0x7f206c15d6d0 2024-04-02T08:53:09.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.882+0000 7f2072e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f206c1623e0 con 0x7f206c0d40f0 2024-04-02T08:53:09.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.882+0000 7f2072e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206c161eb0 con 0x7f206c15dab0 2024-04-02T08:53:09.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.882+0000 7f206be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f206c15d6d0 0x7f206c161430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:09.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.882+0000 7f206be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f206c15d6d0 0x7f206c161430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53360/0 (socket says 172.21.15.89:53360) 2024-04-02T08:53:09.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.882+0000 7f206be00700 1 -- 172.21.15.89:0/2133644448 learned_addr learned my addr 172.21.15.89:0/2133644448 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:09.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.882+0000 7f206b400700 1 --2- 172.21.15.89:0/2133644448 >> v2:172.21.15.113:3300/0 conn(0x7f206c0d40f0 0x7f206c161970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:09.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.882+0000 7f2070e00700 1 --2- 172.21.15.89:0/2133644448 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c15dab0 0x7f206c0d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:09.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.882+0000 7f206be00700 1 -- 172.21.15.89:0/2133644448 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c15dab0 msgr2=0x7f206c0d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:09.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.882+0000 7f206be00700 1 --2- 172.21.15.89:0/2133644448 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c15dab0 0x7f206c0d1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.882+0000 7f206be00700 1 -- 172.21.15.89:0/2133644448 >> v2:172.21.15.113:3300/0 conn(0x7f206c0d40f0 msgr2=0x7f206c161970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:09.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.882+0000 7f206be00700 1 --2- 172.21.15.89:0/2133644448 >> v2:172.21.15.113:3300/0 conn(0x7f206c0d40f0 0x7f206c161970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.882+0000 7f206be00700 1 -- 172.21.15.89:0/2133644448 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206c162100 con 0x7f206c15d6d0 2024-04-02T08:53:09.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.883+0000 7f206be00700 1 --2- 172.21.15.89:0/2133644448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c15d6d0 0x7f206c161430 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f206c08b980 tx=0x7f205c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51dfcadb2f800ee2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:09.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.883+0000 7f206aa00700 1 -- 172.21.15.89:0/2133644448 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c00b870 con 0x7f206c15d6d0 2024-04-02T08:53:09.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.883+0000 7f206aa00700 1 -- 172.21.15.89:0/2133644448 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205c00b9d0 con 0x7f206c15d6d0 2024-04-02T08:53:09.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.883+0000 7f206aa00700 1 -- 172.21.15.89:0/2133644448 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c00bd60 con 0x7f206c15d6d0 2024-04-02T08:53:09.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.883+0000 7f2072e00700 1 -- 172.21.15.89:0/2133644448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c15d6d0 msgr2=0x7f206c161430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:09.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.883+0000 7f2072e00700 1 --2- 172.21.15.89:0/2133644448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c15d6d0 0x7f206c161430 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f206c08b980 tx=0x7f205c009bd0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.883+0000 7f2072e00700 1 -- 172.21.15.89:0/2133644448 shutdown_connections 2024-04-02T08:53:09.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.883+0000 7f2072e00700 1 --2- 172.21.15.89:0/2133644448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c15d6d0 0x7f206c161430 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.883+0000 7f2072e00700 1 --2- 172.21.15.89:0/2133644448 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c15dab0 0x7f206c0d1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.883+0000 7f2072e00700 1 --2- 172.21.15.89:0/2133644448 >> v2:172.21.15.113:3300/0 conn(0x7f206c0d40f0 0x7f206c161970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.883+0000 7f2072e00700 1 -- 172.21.15.89:0/2133644448 >> 172.21.15.89:0/2133644448 conn(0x7f206c08aff0 msgr2=0x7f206c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:09.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.883+0000 7f2072e00700 1 -- 172.21.15.89:0/2133644448 shutdown_connections 2024-04-02T08:53:09.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.884+0000 7f2072e00700 1 -- 172.21.15.89:0/2133644448 wait complete. 2024-04-02T08:53:09.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.885+0000 7f2072e00700 1 Processor -- start 2024-04-02T08:53:09.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.885+0000 7f2072e00700 1 -- start start 2024-04-02T08:53:09.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.886+0000 7f2072e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c0d40f0 0x7f206c0cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:09.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.886+0000 7f2072e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f206c15d6d0 0x7f206c0b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:09.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.886+0000 7f2072e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c15dab0 0x7f206c0b15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:09.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.886+0000 7f2072e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c14b290 con 0x7f206c0d40f0 2024-04-02T08:53:09.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.886+0000 7f2072e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c14b570 con 0x7f206c15d6d0 2024-04-02T08:53:09.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.886+0000 7f2072e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206c14b850 con 0x7f206c15dab0 2024-04-02T08:53:09.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.886+0000 7f206be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c0d40f0 0x7f206c0cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:09.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.886+0000 7f206be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c0d40f0 0x7f206c0cbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53364/0 (socket says 172.21.15.89:53364) 2024-04-02T08:53:09.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.886+0000 7f206be00700 1 -- 172.21.15.89:0/2773360381 learned_addr learned my addr 172.21.15.89:0/2773360381 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:09.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.887+0000 7f206b400700 1 --2- 172.21.15.89:0/2773360381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f206c15d6d0 0x7f206c0b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:09.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.887+0000 7f2070e00700 1 --2- 172.21.15.89:0/2773360381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c15dab0 0x7f206c0b15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:09.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.887+0000 7f206be00700 1 -- 172.21.15.89:0/2773360381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c15dab0 msgr2=0x7f206c0b15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:09.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.887+0000 7f206be00700 1 --2- 172.21.15.89:0/2773360381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c15dab0 0x7f206c0b15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.887+0000 7f206be00700 1 -- 172.21.15.89:0/2773360381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f206c15d6d0 msgr2=0x7f206c0b10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:09.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.887+0000 7f206be00700 1 --2- 172.21.15.89:0/2773360381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f206c15d6d0 0x7f206c0b10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:09.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.887+0000 7f206be00700 1 -- 172.21.15.89:0/2773360381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205c009550 con 0x7f206c0d40f0 2024-04-02T08:53:09.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.887+0000 7f206be00700 1 --2- 172.21.15.89:0/2773360381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c0d40f0 0x7f206c0cbb60 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f205c000c00 tx=0x7f205c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:09.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.887+0000 7f2063e00700 1 -- 172.21.15.89:0/2773360381 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c004540 con 0x7f206c0d40f0 2024-04-02T08:53:09.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.887+0000 7f2063e00700 1 -- 172.21.15.89:0/2773360381 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205c0036a0 con 0x7f206c0d40f0 2024-04-02T08:53:09.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.887+0000 7f2063e00700 1 -- 172.21.15.89:0/2773360381 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c003a50 con 0x7f206c0d40f0 2024-04-02T08:53:09.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.887+0000 7f2072e00700 1 -- 172.21.15.89:0/2773360381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206c14bb30 con 0x7f206c0d40f0 2024-04-02T08:53:09.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.887+0000 7f2072e00700 1 -- 172.21.15.89:0/2773360381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f206c14c050 con 0x7f206c0d40f0 2024-04-02T08:53:09.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.889+0000 7f2072e00700 1 -- 172.21.15.89:0/2773360381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f206c14bd70 con 0x7f206c0d40f0 2024-04-02T08:53:09.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.889+0000 7f2063e00700 1 -- 172.21.15.89:0/2773360381 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 112) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f205c021070 con 0x7f206c0d40f0 2024-04-02T08:53:09.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.890+0000 7f2063e00700 1 --2- 172.21.15.89:0/2773360381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f204c06aae0 0x7f204c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:09.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.890+0000 7f2063e00700 1 -- 172.21.15.89:0/2773360381 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f205c08ab80 con 0x7f206c0d40f0 2024-04-02T08:53:09.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.890+0000 7f206b400700 1 --2- 172.21.15.89:0/2773360381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f204c06aae0 0x7f204c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:09.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.891+0000 7f206b400700 1 --2- 172.21.15.89:0/2773360381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f204c06aae0 0x7f204c06cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f206c0a1470 tx=0x7f2058008040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:09.895+0000 7f2063e00700 1 -- 172.21.15.89:0/2773360381 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f206c14bd70 con 0x7f206c0d40f0 2024-04-02T08:53:10.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.143+0000 7f2072e00700 1 -- 172.21.15.89:0/2773360381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f206c14bd70 con 0x7f206c0d40f0 2024-04-02T08:53:10.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.673+0000 7f2063e00700 1 -- 172.21.15.89:0/2773360381 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v83) v1 ==== 104+0+0 (secure 0 0 0) 0x7f206c14bd70 con 0x7f206c0d40f0 2024-04-02T08:53:10.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.676+0000 7f2072e00700 1 -- 172.21.15.89:0/2773360381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f204c06aae0 msgr2=0x7f204c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:10.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.676+0000 7f2072e00700 1 --2- 172.21.15.89:0/2773360381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f204c06aae0 0x7f204c06cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f206c0a1470 tx=0x7f2058008040 comp rx=0 tx=0).stop 2024-04-02T08:53:10.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.676+0000 7f2072e00700 1 -- 172.21.15.89:0/2773360381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c0d40f0 msgr2=0x7f206c0cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:10.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.676+0000 7f2072e00700 1 --2- 172.21.15.89:0/2773360381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c0d40f0 0x7f206c0cbb60 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f205c000c00 tx=0x7f205c004750 comp rx=0 tx=0).stop 2024-04-02T08:53:10.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.676+0000 7f2072e00700 1 -- 172.21.15.89:0/2773360381 shutdown_connections 2024-04-02T08:53:10.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.676+0000 7f2072e00700 1 --2- 172.21.15.89:0/2773360381 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f204c06aae0 0x7f204c06cfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:10.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.676+0000 7f2072e00700 1 --2- 172.21.15.89:0/2773360381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f206c15dab0 0x7f206c0b15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:10.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.676+0000 7f2072e00700 1 --2- 172.21.15.89:0/2773360381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f206c0d40f0 0x7f206c0cbb60 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:10.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.676+0000 7f2072e00700 1 --2- 172.21.15.89:0/2773360381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f206c15d6d0 0x7f206c0b10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:10.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.676+0000 7f2072e00700 1 -- 172.21.15.89:0/2773360381 >> 172.21.15.89:0/2773360381 conn(0x7f206c08aff0 msgr2=0x7f206c15abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:10.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.676+0000 7f2072e00700 1 -- 172.21.15.89:0/2773360381 shutdown_connections 2024-04-02T08:53:10.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.676+0000 7f2072e00700 1 -- 172.21.15.89:0/2773360381 wait complete. 2024-04-02T08:53:10.696 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T08:53:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.944+0000 7f19b5600700 1 Processor -- start 2024-04-02T08:53:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.945+0000 7f19b5600700 1 -- start start 2024-04-02T08:53:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19b5600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f19b015d930 0x7f19b00d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19b5600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19b0162030 0x7f19b00d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19b5600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f19b00d20d0 0x7f19b00cfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19b5600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19b015ec40 con 0x7f19b00d20d0 2024-04-02T08:53:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19b5600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19b00d2560 con 0x7f19b015d930 2024-04-02T08:53:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19b5600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b015dd10 con 0x7f19b0162030 2024-04-02T08:53:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19af400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f19b00d20d0 0x7f19b00cfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19af400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f19b00d20d0 0x7f19b00cfc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53388/0 (socket says 172.21.15.89:53388) 2024-04-02T08:53:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19af400700 1 -- 172.21.15.89:0/990184156 learned_addr learned my addr 172.21.15.89:0/990184156 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19aea00700 1 --2- 172.21.15.89:0/990184156 >> v2:172.21.15.113:3300/0 conn(0x7f19b015d930 0x7f19b00d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19ae000700 1 --2- 172.21.15.89:0/990184156 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19b0162030 0x7f19b00d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19af400700 1 -- 172.21.15.89:0/990184156 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19b0162030 msgr2=0x7f19b00d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19af400700 1 --2- 172.21.15.89:0/990184156 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19b0162030 0x7f19b00d1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19af400700 1 -- 172.21.15.89:0/990184156 >> v2:172.21.15.113:3300/0 conn(0x7f19b015d930 msgr2=0x7f19b00d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.946+0000 7f19af400700 1 --2- 172.21.15.89:0/990184156 >> v2:172.21.15.113:3300/0 conn(0x7f19b015d930 0x7f19b00d3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.947+0000 7f19af400700 1 -- 172.21.15.89:0/990184156 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b015dfc0 con 0x7f19b00d20d0 2024-04-02T08:53:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.947+0000 7f19af400700 1 --2- 172.21.15.89:0/990184156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b00d20d0 0x7f19b00cfc00 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f19b00a1430 tx=0x7f19a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e1e036d3086507 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.947+0000 7f19ad600700 1 -- 172.21.15.89:0/990184156 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a800e070 con 0x7f19b00d20d0 2024-04-02T08:53:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.947+0000 7f19ad600700 1 -- 172.21.15.89:0/990184156 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a800b9c0 con 0x7f19b00d20d0 2024-04-02T08:53:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.947+0000 7f19ad600700 1 -- 172.21.15.89:0/990184156 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a800bd70 con 0x7f19b00d20d0 2024-04-02T08:53:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.947+0000 7f19b5600700 1 -- 172.21.15.89:0/990184156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b00d20d0 msgr2=0x7f19b00cfc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.947+0000 7f19b5600700 1 --2- 172.21.15.89:0/990184156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b00d20d0 0x7f19b00cfc00 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f19b00a1430 tx=0x7f19a800b330 comp rx=0 tx=0).stop 2024-04-02T08:53:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.948+0000 7f19b5600700 1 -- 172.21.15.89:0/990184156 shutdown_connections 2024-04-02T08:53:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.948+0000 7f19b5600700 1 --2- 172.21.15.89:0/990184156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b00d20d0 0x7f19b00cfc00 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.948+0000 7f19b5600700 1 --2- 172.21.15.89:0/990184156 >> v2:172.21.15.113:3300/0 conn(0x7f19b015d930 0x7f19b00d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.948+0000 7f19b5600700 1 --2- 172.21.15.89:0/990184156 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19b0162030 0x7f19b00d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.948+0000 7f19b5600700 1 -- 172.21.15.89:0/990184156 >> 172.21.15.89:0/990184156 conn(0x7f19b008aff0 msgr2=0x7f19b00d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.948+0000 7f19b5600700 1 -- 172.21.15.89:0/990184156 shutdown_connections 2024-04-02T08:53:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.948+0000 7f19b5600700 1 -- 172.21.15.89:0/990184156 wait complete. 2024-04-02T08:53:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.949+0000 7f19b5600700 1 Processor -- start 2024-04-02T08:53:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.950+0000 7f19b5600700 1 -- start start 2024-04-02T08:53:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.950+0000 7f19b5600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19b00d20d0 0x7f19b0157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.950+0000 7f19b5600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b015d930 0x7f19b01f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.951+0000 7f19b5600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19b0162030 0x7f19b00ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.951+0000 7f19b5600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b00caff0 con 0x7f19b015d930 2024-04-02T08:53:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.951+0000 7f19b5600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b01f3090 con 0x7f19b0162030 2024-04-02T08:53:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.951+0000 7f19b5600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b01f3370 con 0x7f19b00d20d0 2024-04-02T08:53:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.951+0000 7f19ae000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b015d930 0x7f19b01f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.951+0000 7f19ae000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b015d930 0x7f19b01f3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53396/0 (socket says 172.21.15.89:53396) 2024-04-02T08:53:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.951+0000 7f19ae000700 1 -- 172.21.15.89:0/2013394322 learned_addr learned my addr 172.21.15.89:0/2013394322 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.951+0000 7f19ae000700 1 -- 172.21.15.89:0/2013394322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19b00d20d0 msgr2=0x7f19b0157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.951+0000 7f19ae000700 1 --2- 172.21.15.89:0/2013394322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19b00d20d0 0x7f19b0157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.951+0000 7f19ae000700 1 -- 172.21.15.89:0/2013394322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19b0162030 msgr2=0x7f19b00ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.951+0000 7f19af400700 1 --2- 172.21.15.89:0/2013394322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19b0162030 0x7f19b00ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.951+0000 7f19ae000700 1 --2- 172.21.15.89:0/2013394322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19b0162030 0x7f19b00ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.951+0000 7f19ae000700 1 -- 172.21.15.89:0/2013394322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a8009b30 con 0x7f19b015d930 2024-04-02T08:53:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.951+0000 7f19ae000700 1 --2- 172.21.15.89:0/2013394322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b015d930 0x7f19b01f3b70 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f199c00d800 tx=0x7f199c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.952+0000 7f19a6a00700 1 -- 172.21.15.89:0/2013394322 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c0125f0 con 0x7f19b015d930 2024-04-02T08:53:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.952+0000 7f19a6a00700 1 -- 172.21.15.89:0/2013394322 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199c012750 con 0x7f19b015d930 2024-04-02T08:53:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.952+0000 7f19b5600700 1 -- 172.21.15.89:0/2013394322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b01f3650 con 0x7f19b015d930 2024-04-02T08:53:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.952+0000 7f19a6a00700 1 -- 172.21.15.89:0/2013394322 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c012b00 con 0x7f19b015d930 2024-04-02T08:53:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.952+0000 7f19b5600700 1 -- 172.21.15.89:0/2013394322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b00cabc0 con 0x7f19b015d930 2024-04-02T08:53:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.953+0000 7f19b5600700 1 -- 172.21.15.89:0/2013394322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b0067720 con 0x7f19b015d930 2024-04-02T08:53:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.954+0000 7f19a6a00700 1 -- 172.21.15.89:0/2013394322 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 112) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f199c012c60 con 0x7f19b015d930 2024-04-02T08:53:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.954+0000 7f19a6a00700 1 --2- 172.21.15.89:0/2013394322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f199006aae0 0x7f199006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.955+0000 7f19a6a00700 1 -- 172.21.15.89:0/2013394322 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f199c089790 con 0x7f19b015d930 2024-04-02T08:53:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.955+0000 7f19aea00700 1 --2- 172.21.15.89:0/2013394322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f199006aae0 0x7f199006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:10.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.955+0000 7f19aea00700 1 --2- 172.21.15.89:0/2013394322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f199006aae0 0x7f199006cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f19a0009770 tx=0x7f19a0008040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:10.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:10.959+0000 7f19a6a00700 1 -- 172.21.15.89:0/2013394322 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f199c054850 con 0x7f19b015d930 2024-04-02T08:53:11.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.159+0000 7f19b5600700 1 -- 172.21.15.89:0/2013394322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f19b00ced80 con 0x7f19b015d930 2024-04-02T08:53:11.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.679+0000 7f19a6a00700 1 -- 172.21.15.89:0/2013394322 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 113) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f199c054c20 con 0x7f19b015d930 2024-04-02T08:53:11.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.751+0000 7f19a6a00700 1 -- 172.21.15.89:0/2013394322 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v176) v1 ==== 181+0+0 (secure 0 0 0) 0x7f199c0582e0 con 0x7f19b015d930 2024-04-02T08:53:11.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.755+0000 7f19b5600700 1 -- 172.21.15.89:0/2013394322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f199006aae0 msgr2=0x7f199006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:11.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.755+0000 7f19b5600700 1 --2- 172.21.15.89:0/2013394322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f199006aae0 0x7f199006cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f19a0009770 tx=0x7f19a0008040 comp rx=0 tx=0).stop 2024-04-02T08:53:11.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.755+0000 7f19b5600700 1 -- 172.21.15.89:0/2013394322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b015d930 msgr2=0x7f19b01f3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:11.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.755+0000 7f19b5600700 1 --2- 172.21.15.89:0/2013394322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b015d930 0x7f19b01f3b70 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f199c00d800 tx=0x7f199c00dbc0 comp rx=0 tx=0).stop 2024-04-02T08:53:11.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.755+0000 7f19b5600700 1 -- 172.21.15.89:0/2013394322 shutdown_connections 2024-04-02T08:53:11.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.755+0000 7f19b5600700 1 --2- 172.21.15.89:0/2013394322 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f199006aae0 0x7f199006cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:11.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.755+0000 7f19b5600700 1 --2- 172.21.15.89:0/2013394322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19b0162030 0x7f19b00ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:11.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.755+0000 7f19b5600700 1 --2- 172.21.15.89:0/2013394322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19b00d20d0 0x7f19b0157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:11.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.755+0000 7f19b5600700 1 --2- 172.21.15.89:0/2013394322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19b015d930 0x7f19b01f3b70 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:11.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.755+0000 7f19b5600700 1 -- 172.21.15.89:0/2013394322 >> 172.21.15.89:0/2013394322 conn(0x7f19b008aff0 msgr2=0x7f19b00b00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:11.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.755+0000 7f19b5600700 1 -- 172.21.15.89:0/2013394322 shutdown_connections 2024-04-02T08:53:11.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.755+0000 7f19b5600700 1 -- 172.21.15.89:0/2013394322 wait complete. 2024-04-02T08:53:11.755 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T08:53:11.775 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T08:53:11.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.977+0000 7fc490000700 1 Processor -- start 2024-04-02T08:53:11.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.977+0000 7fc490000700 1 -- start start 2024-04-02T08:53:11.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.978+0000 7fc490000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc48806aea0 0x7fc48806dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:11.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.978+0000 7fc490000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48815de10 con 0x7fc4880d0bd0 2024-04-02T08:53:11.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.978+0000 7fc490000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4880b4fd0 con 0x7fc48815e8d0 2024-04-02T08:53:11.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.978+0000 7fc490000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4880b52b0 con 0x7fc48806aea0 2024-04-02T08:53:11.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.979+0000 7fc48d600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc4880d0bd0 0x7fc4880b3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58138/0 (socket says 172.21.15.89:58138) 2024-04-02T08:53:11.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.979+0000 7fc48d600700 1 -- 172.21.15.89:0/3213823086 learned_addr learned my addr 172.21.15.89:0/3213823086 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:11.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.979+0000 7fc48e000700 1 --2- 172.21.15.89:0/3213823086 >> v2:172.21.15.113:3300/0 conn(0x7fc48806aea0 0x7fc48806dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.979+0000 7fc487e00700 1 -- 172.21.15.89:0/3213823086 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676079932 0 0) 0x7fc48815de10 con 0x7fc4880d0bd0 2024-04-02T08:53:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.979+0000 7fc487e00700 1 -- 172.21.15.89:0/3213823086 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4700035b0 con 0x7fc4880d0bd0 2024-04-02T08:53:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.979+0000 7fc48e000700 1 -- 172.21.15.89:0/3213823086 >> v1:172.21.15.113:6789/0 conn(0x7fc48815e8d0 legacy=0x7fc48806a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.979+0000 7fc48e000700 1 -- 172.21.15.89:0/3213823086 >> v1:172.21.15.89:6789/0 conn(0x7fc4880d0bd0 legacy=0x7fc4880b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.979+0000 7fc48e000700 1 -- 172.21.15.89:0/3213823086 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48815e0d0 con 0x7fc48806aea0 2024-04-02T08:53:11.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.980+0000 7fc48e000700 1 --2- 172.21.15.89:0/3213823086 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc48806aea0 0x7fc48806dae0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc480009d90 tx=0x7fc48000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58b2dd92ab17a713 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:11.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.980+0000 7fc487e00700 1 -- 172.21.15.89:0/3213823086 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48000e070 con 0x7fc48806aea0 2024-04-02T08:53:11.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.980+0000 7fc487e00700 1 -- 172.21.15.89:0/3213823086 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48000bb20 con 0x7fc48806aea0 2024-04-02T08:53:11.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.980+0000 7fc487e00700 1 -- 172.21.15.89:0/3213823086 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480012430 con 0x7fc48806aea0 2024-04-02T08:53:11.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.980+0000 7fc490000700 1 -- 172.21.15.89:0/3213823086 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc48806aea0 msgr2=0x7fc48806dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:11.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.980+0000 7fc490000700 1 --2- 172.21.15.89:0/3213823086 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc48806aea0 0x7fc48806dae0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc480009d90 tx=0x7fc48000b330 comp rx=0 tx=0).stop 2024-04-02T08:53:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.980+0000 7fc490000700 1 -- 172.21.15.89:0/3213823086 shutdown_connections 2024-04-02T08:53:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.980+0000 7fc490000700 1 --2- 172.21.15.89:0/3213823086 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc48806aea0 0x7fc48806dae0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.980+0000 7fc490000700 1 -- 172.21.15.89:0/3213823086 >> 172.21.15.89:0/3213823086 conn(0x7fc4880b5640 msgr2=0x7fc4880b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.981+0000 7fc490000700 1 -- 172.21.15.89:0/3213823086 shutdown_connections 2024-04-02T08:53:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.981+0000 7fc490000700 1 -- 172.21.15.89:0/3213823086 wait complete. 2024-04-02T08:53:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.982+0000 7fc490000700 1 Processor -- start 2024-04-02T08:53:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.982+0000 7fc490000700 1 -- start start 2024-04-02T08:53:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.983+0000 7fc490000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48806aea0 0x7fc488155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.983+0000 7fc490000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4880d0bd0 0x7fc48814ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.983+0000 7fc490000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc48815e8d0 0x7fc48815caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.983+0000 7fc490000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4881f3ea0 con 0x7fc4880d0bd0 2024-04-02T08:53:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.983+0000 7fc490000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4881548f0 con 0x7fc48815e8d0 2024-04-02T08:53:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.983+0000 7fc490000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc488154bd0 con 0x7fc48806aea0 2024-04-02T08:53:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.984+0000 7fc48cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4880d0bd0 0x7fc48814ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.984+0000 7fc48cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4880d0bd0 0x7fc48814ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53402/0 (socket says 172.21.15.89:53402) 2024-04-02T08:53:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.984+0000 7fc48cc00700 1 -- 172.21.15.89:0/1149755328 learned_addr learned my addr 172.21.15.89:0/1149755328 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.984+0000 7fc48e000700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc48815e8d0 0x7fc48815caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.984+0000 7fc48d600700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48806aea0 0x7fc488155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.984+0000 7fc48cc00700 1 -- 172.21.15.89:0/1149755328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48806aea0 msgr2=0x7fc488155990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.984+0000 7fc48cc00700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48806aea0 0x7fc488155990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.984+0000 7fc48cc00700 1 -- 172.21.15.89:0/1149755328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc48815e8d0 msgr2=0x7fc48815caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.984+0000 7fc48cc00700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc48815e8d0 0x7fc48815caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.984+0000 7fc48cc00700 1 -- 172.21.15.89:0/1149755328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc480009a70 con 0x7fc4880d0bd0 2024-04-02T08:53:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.984+0000 7fc48d600700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48806aea0 0x7fc488155990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:53:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.984+0000 7fc48cc00700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4880d0bd0 0x7fc48814ed30 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fc47400c4e0 tx=0x7fc47400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.984+0000 7fc485600700 1 -- 172.21.15.89:0/1149755328 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4740052d0 con 0x7fc4880d0bd0 2024-04-02T08:53:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.985+0000 7fc485600700 1 -- 172.21.15.89:0/1149755328 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc474005430 con 0x7fc4880d0bd0 2024-04-02T08:53:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.985+0000 7fc490000700 1 -- 172.21.15.89:0/1149755328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc488154eb0 con 0x7fc4880d0bd0 2024-04-02T08:53:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.985+0000 7fc485600700 1 -- 172.21.15.89:0/1149755328 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4740038c0 con 0x7fc4880d0bd0 2024-04-02T08:53:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.985+0000 7fc490000700 1 -- 172.21.15.89:0/1149755328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc488155470 con 0x7fc4880d0bd0 2024-04-02T08:53:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.986+0000 7fc490000700 1 -- 172.21.15.89:0/1149755328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc488067720 con 0x7fc4880d0bd0 2024-04-02T08:53:11.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.987+0000 7fc485600700 1 -- 172.21.15.89:0/1149755328 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 113) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fc474003a20 con 0x7fc4880d0bd0 2024-04-02T08:53:11.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.987+0000 7fc485600700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc46806ad20 0x7fc46806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:11.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.988+0000 7fc485600700 1 -- 172.21.15.89:0/1149755328 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6379+0+0 (secure 0 0 0) 0x7fc474088ba0 con 0x7fc4880d0bd0 2024-04-02T08:53:11.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.988+0000 7fc48d600700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc46806ad20 0x7fc46806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:11.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.988+0000 7fc48d600700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc46806ad20 0x7fc46806d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc47c0011e0 tx=0x7fc47c009040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:11.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:11.992+0000 7fc485600700 1 -- 172.21.15.89:0/1149755328 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc4740539f0 con 0x7fc4880d0bd0 2024-04-02T08:53:12.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.197+0000 7fc490000700 1 -- 172.21.15.89:0/1149755328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc48806f420 con 0x7fc4880d0bd0 2024-04-02T08:53:12.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.763+0000 7fc485600700 1 -- 172.21.15.89:0/1149755328 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc474053dc0 con 0x7fc4880d0bd0 2024-04-02T08:53:12.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.763+0000 7fc485600700 1 -- 172.21.15.89:0/1149755328 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v177) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc4740514b0 con 0x7fc4880d0bd0 2024-04-02T08:53:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.767+0000 7fc490000700 1 -- 172.21.15.89:0/1149755328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc46806ad20 msgr2=0x7fc46806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.767+0000 7fc490000700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc46806ad20 0x7fc46806d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc47c0011e0 tx=0x7fc47c009040 comp rx=0 tx=0).stop 2024-04-02T08:53:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.767+0000 7fc490000700 1 -- 172.21.15.89:0/1149755328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4880d0bd0 msgr2=0x7fc48814ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.767+0000 7fc490000700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4880d0bd0 0x7fc48814ed30 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fc47400c4e0 tx=0x7fc47400c8a0 comp rx=0 tx=0).stop 2024-04-02T08:53:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.767+0000 7fc490000700 1 -- 172.21.15.89:0/1149755328 shutdown_connections 2024-04-02T08:53:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.767+0000 7fc490000700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc46806ad20 0x7fc46806d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.767+0000 7fc490000700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc48815e8d0 0x7fc48815caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.767+0000 7fc490000700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc48806aea0 0x7fc488155990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.767+0000 7fc490000700 1 --2- 172.21.15.89:0/1149755328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4880d0bd0 0x7fc48814ed30 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:12.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.767+0000 7fc490000700 1 -- 172.21.15.89:0/1149755328 >> 172.21.15.89:0/1149755328 conn(0x7fc4880b5640 msgr2=0x7fc4880b2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:12.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.767+0000 7fc490000700 1 -- 172.21.15.89:0/1149755328 shutdown_connections 2024-04-02T08:53:12.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.767+0000 7fc490000700 1 -- 172.21.15.89:0/1149755328 wait complete. 2024-04-02T08:53:12.767 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T08:53:12.785 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T08:53:12.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.981+0000 7f4994400700 1 Processor -- start 2024-04-02T08:53:12.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.981+0000 7f4994400700 1 -- start start 2024-04-02T08:53:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.982+0000 7f4994400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f498c0d16a0 0x7f498c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.983+0000 7f4994400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498c161420 con 0x7f498c15d9d0 2024-04-02T08:53:12.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.983+0000 7f4994400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498c161700 con 0x7f498c0d2370 2024-04-02T08:53:12.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.983+0000 7f4994400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f498c0d4340 con 0x7f498c0d16a0 2024-04-02T08:53:12.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.983+0000 7f4991a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f498c15d9d0 0x7f498c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58154/0 (socket says 172.21.15.89:58154) 2024-04-02T08:53:12.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.983+0000 7f4991000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f498c0d16a0 0x7f498c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:12.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.983+0000 7f4991a00700 1 -- 172.21.15.89:0/2531586728 learned_addr learned my addr 172.21.15.89:0/2531586728 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:12.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.983+0000 7f4991000700 1 -- 172.21.15.89:0/2531586728 >> v1:172.21.15.113:6789/0 conn(0x7f498c0d2370 legacy=0x7f498c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:12.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.983+0000 7f4991000700 1 -- 172.21.15.89:0/2531586728 >> v1:172.21.15.89:6789/0 conn(0x7f498c15d9d0 legacy=0x7f498c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:12.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.983+0000 7f4991000700 1 -- 172.21.15.89:0/2531586728 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c1619e0 con 0x7f498c0d16a0 2024-04-02T08:53:12.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.983+0000 7f498be00700 1 -- 172.21.15.89:0/2531586728 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780195302 0 0) 0x7f498c161420 con 0x7f498c15d9d0 2024-04-02T08:53:12.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.984+0000 7f4991000700 1 --2- 172.21.15.89:0/2531586728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c0d16a0 0x7f498c0d1e30 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f4978009820 tx=0x7f4978009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97548642ff257f23 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:12.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.984+0000 7f498be00700 1 -- 172.21.15.89:0/2531586728 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978005e30 con 0x7f498c0d16a0 2024-04-02T08:53:12.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.984+0000 7f498be00700 1 -- 172.21.15.89:0/2531586728 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4978006030 con 0x7f498c0d16a0 2024-04-02T08:53:12.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.984+0000 7f498be00700 1 -- 172.21.15.89:0/2531586728 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49780063e0 con 0x7f498c0d16a0 2024-04-02T08:53:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.984+0000 7f4994400700 1 -- 172.21.15.89:0/2531586728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c0d16a0 msgr2=0x7f498c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.984+0000 7f4994400700 1 --2- 172.21.15.89:0/2531586728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c0d16a0 0x7f498c0d1e30 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f4978009820 tx=0x7f4978009b30 comp rx=0 tx=0).stop 2024-04-02T08:53:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.984+0000 7f4994400700 1 -- 172.21.15.89:0/2531586728 shutdown_connections 2024-04-02T08:53:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.984+0000 7f4994400700 1 --2- 172.21.15.89:0/2531586728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c0d16a0 0x7f498c0d1e30 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.984+0000 7f4994400700 1 -- 172.21.15.89:0/2531586728 >> 172.21.15.89:0/2531586728 conn(0x7f498c08aff0 msgr2=0x7f498c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.985+0000 7f4994400700 1 -- 172.21.15.89:0/2531586728 shutdown_connections 2024-04-02T08:53:12.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.985+0000 7f4994400700 1 -- 172.21.15.89:0/2531586728 wait complete. 2024-04-02T08:53:12.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.986+0000 7f4994400700 1 Processor -- start 2024-04-02T08:53:12.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.986+0000 7f4994400700 1 -- start start 2024-04-02T08:53:12.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.987+0000 7f4994400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c0d16a0 0x7f498c0cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:12.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.987+0000 7f4994400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0d2370 0x7f498c0cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:12.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.987+0000 7f4994400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c15d9d0 0x7f498c1550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:12.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.987+0000 7f4994400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c0cd4d0 con 0x7f498c15d9d0 2024-04-02T08:53:12.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.987+0000 7f4994400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c0cc5f0 con 0x7f498c0d2370 2024-04-02T08:53:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.987+0000 7f4994400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498c0cc8c0 con 0x7f498c0d16a0 2024-04-02T08:53:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.987+0000 7f4992400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c15d9d0 0x7f498c1550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.987+0000 7f4991a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c0d16a0 0x7f498c0cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.988+0000 7f4992400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c15d9d0 0x7f498c1550f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53424/0 (socket says 172.21.15.89:53424) 2024-04-02T08:53:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.988+0000 7f4992400700 1 -- 172.21.15.89:0/2908474766 learned_addr learned my addr 172.21.15.89:0/2908474766 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.988+0000 7f4991000700 1 --2- 172.21.15.89:0/2908474766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0d2370 0x7f498c0cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.988+0000 7f4992400700 1 -- 172.21.15.89:0/2908474766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c0d16a0 msgr2=0x7f498c0cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.988+0000 7f4992400700 1 --2- 172.21.15.89:0/2908474766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c0d16a0 0x7f498c0cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.988+0000 7f4992400700 1 -- 172.21.15.89:0/2908474766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0d2370 msgr2=0x7f498c0cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.988+0000 7f4992400700 1 --2- 172.21.15.89:0/2908474766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0d2370 0x7f498c0cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.988+0000 7f4992400700 1 -- 172.21.15.89:0/2908474766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4978009430 con 0x7f498c15d9d0 2024-04-02T08:53:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.988+0000 7f4992400700 1 --2- 172.21.15.89:0/2908474766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c15d9d0 0x7f498c1550f0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f498400cc50 tx=0x7f498400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.988+0000 7f4989600700 1 -- 172.21.15.89:0/2908474766 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4984003c00 con 0x7f498c15d9d0 2024-04-02T08:53:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.989+0000 7f4989600700 1 -- 172.21.15.89:0/2908474766 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4984003d60 con 0x7f498c15d9d0 2024-04-02T08:53:12.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.989+0000 7f4994400700 1 -- 172.21.15.89:0/2908474766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498c0ccba0 con 0x7f498c15d9d0 2024-04-02T08:53:12.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.989+0000 7f4989600700 1 -- 172.21.15.89:0/2908474766 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498400aab0 con 0x7f498c15d9d0 2024-04-02T08:53:12.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.989+0000 7f4994400700 1 -- 172.21.15.89:0/2908474766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498c0cd160 con 0x7f498c15d9d0 2024-04-02T08:53:12.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.990+0000 7f4994400700 1 -- 172.21.15.89:0/2908474766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498c0d1ca0 con 0x7f498c15d9d0 2024-04-02T08:53:12.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.990+0000 7f4989600700 1 -- 172.21.15.89:0/2908474766 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f498400ac10 con 0x7f498c15d9d0 2024-04-02T08:53:12.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.991+0000 7f4989600700 1 --2- 172.21.15.89:0/2908474766 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f496c06adb0 0x7f496c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:12.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.991+0000 7f4989600700 1 -- 172.21.15.89:0/2908474766 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5965+0+0 (secure 0 0 0) 0x7f4984089580 con 0x7f498c15d9d0 2024-04-02T08:53:12.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.992+0000 7f4991a00700 1 --2- 172.21.15.89:0/2908474766 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f496c06adb0 0x7f496c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:12.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.992+0000 7f4991a00700 1 --2- 172.21.15.89:0/2908474766 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f496c06adb0 0x7f496c06d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4980000f60 tx=0x7f4980004a50 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:12.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:12.996+0000 7f4989600700 1 -- 172.21.15.89:0/2908474766 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4984054540 con 0x7f498c15d9d0 2024-04-02T08:53:13.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.195+0000 7f4994400700 1 -- 172.21.15.89:0/2908474766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f498c067720 con 0x7f498c15d9d0 2024-04-02T08:53:13.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.199+0000 7f4989600700 1 -- 172.21.15.89:0/2908474766 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v61) v1 ==== 115+0+0 (secure 0 0 0) 0x7f4984057b60 con 0x7f498c15d9d0 2024-04-02T08:53:13.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.202+0000 7f4994400700 1 -- 172.21.15.89:0/2908474766 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f496c06adb0 msgr2=0x7f496c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:13.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.202+0000 7f4994400700 1 --2- 172.21.15.89:0/2908474766 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f496c06adb0 0x7f496c06d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4980000f60 tx=0x7f4980004a50 comp rx=0 tx=0).stop 2024-04-02T08:53:13.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.202+0000 7f4994400700 1 -- 172.21.15.89:0/2908474766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c15d9d0 msgr2=0x7f498c1550f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:13.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.202+0000 7f4994400700 1 --2- 172.21.15.89:0/2908474766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c15d9d0 0x7f498c1550f0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f498400cc50 tx=0x7f498400cf60 comp rx=0 tx=0).stop 2024-04-02T08:53:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.202+0000 7f4994400700 1 -- 172.21.15.89:0/2908474766 shutdown_connections 2024-04-02T08:53:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.202+0000 7f4994400700 1 --2- 172.21.15.89:0/2908474766 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f496c06adb0 0x7f496c06d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.202+0000 7f4994400700 1 --2- 172.21.15.89:0/2908474766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c15d9d0 0x7f498c1550f0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.202+0000 7f4994400700 1 --2- 172.21.15.89:0/2908474766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c0d16a0 0x7f498c0cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.202+0000 7f4994400700 1 --2- 172.21.15.89:0/2908474766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0d2370 0x7f498c0cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.202+0000 7f4994400700 1 -- 172.21.15.89:0/2908474766 >> 172.21.15.89:0/2908474766 conn(0x7f498c08aff0 msgr2=0x7f498c0b1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:13.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.202+0000 7f4994400700 1 -- 172.21.15.89:0/2908474766 shutdown_connections 2024-04-02T08:53:13.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.202+0000 7f4994400700 1 -- 172.21.15.89:0/2908474766 wait complete. 2024-04-02T08:53:13.221 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T08:53:13.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.415+0000 7fd8b3c00700 1 Processor -- start 2024-04-02T08:53:13.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.415+0000 7fd8b3c00700 1 -- start start 2024-04-02T08:53:13.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.416+0000 7fd8b3c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd8ac0d16a0 0x7fd8ac0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:13.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.416+0000 7fd8b3c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ac161420 con 0x7fd8ac0d2370 2024-04-02T08:53:13.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.416+0000 7fd8b3c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ac161700 con 0x7fd8ac15d9d0 2024-04-02T08:53:13.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.416+0000 7fd8b3c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ac0d4340 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.416+0000 7fd8b1c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd8ac0d2370 0x7fd8ac0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58162/0 (socket says 172.21.15.89:58162) 2024-04-02T08:53:13.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.416+0000 7fd8abe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd8ac0d16a0 0x7fd8ac0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:13.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.416+0000 7fd8b1c00700 1 -- 172.21.15.89:0/61562026 learned_addr learned my addr 172.21.15.89:0/61562026 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:13.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.417+0000 7fd8abe00700 1 -- 172.21.15.89:0/61562026 >> v1:172.21.15.113:6789/0 conn(0x7fd8ac15d9d0 legacy=0x7fd8ac0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:13.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.417+0000 7fd8abe00700 1 -- 172.21.15.89:0/61562026 >> v1:172.21.15.89:6789/0 conn(0x7fd8ac0d2370 legacy=0x7fd8ac0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:13.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.417+0000 7fd8ab400700 1 -- 172.21.15.89:0/61562026 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245209249 0 0) 0x7fd8ac161420 con 0x7fd8ac0d2370 2024-04-02T08:53:13.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.417+0000 7fd8abe00700 1 -- 172.21.15.89:0/61562026 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ac1619e0 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.417+0000 7fd8abe00700 1 --2- 172.21.15.89:0/61562026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8ac0d16a0 0x7fd8ac0d1e30 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fd898009820 tx=0x7fd898009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4708a33af407af84 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:13.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.417+0000 7fd8ab400700 1 -- 172.21.15.89:0/61562026 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898005e30 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.417+0000 7fd8ab400700 1 -- 172.21.15.89:0/61562026 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd898006030 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.417+0000 7fd8ab400700 1 -- 172.21.15.89:0/61562026 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8980063e0 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.418+0000 7fd8b3c00700 1 -- 172.21.15.89:0/61562026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8ac0d16a0 msgr2=0x7fd8ac0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:13.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.418+0000 7fd8b3c00700 1 --2- 172.21.15.89:0/61562026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8ac0d16a0 0x7fd8ac0d1e30 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fd898009820 tx=0x7fd898009b30 comp rx=0 tx=0).stop 2024-04-02T08:53:13.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.418+0000 7fd8b3c00700 1 -- 172.21.15.89:0/61562026 shutdown_connections 2024-04-02T08:53:13.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.418+0000 7fd8b3c00700 1 --2- 172.21.15.89:0/61562026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8ac0d16a0 0x7fd8ac0d1e30 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.418+0000 7fd8b3c00700 1 -- 172.21.15.89:0/61562026 >> 172.21.15.89:0/61562026 conn(0x7fd8ac08aff0 msgr2=0x7fd8ac0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:13.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.418+0000 7fd8b3c00700 1 -- 172.21.15.89:0/61562026 shutdown_connections 2024-04-02T08:53:13.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.418+0000 7fd8b3c00700 1 -- 172.21.15.89:0/61562026 wait complete. 2024-04-02T08:53:13.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.420+0000 7fd8b3c00700 1 Processor -- start 2024-04-02T08:53:13.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.420+0000 7fd8b3c00700 1 -- start start 2024-04-02T08:53:13.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8b3c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8ac0d16a0 0x7fd8ac14e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:13.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8b3c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8ac0d2370 0x7fd8ac0cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:13.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8b3c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ac15d9d0 0x7fd8ac1606d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:13.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8b3c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac14ef20 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8b3c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac0cc2f0 con 0x7fd8ac15d9d0 2024-04-02T08:53:13.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8b3c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac0cba70 con 0x7fd8ac0d2370 2024-04-02T08:53:13.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8b1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8ac0d16a0 0x7fd8ac14e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:13.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8b1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8ac0d16a0 0x7fd8ac14e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53450/0 (socket says 172.21.15.89:53450) 2024-04-02T08:53:13.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8b1200700 1 -- 172.21.15.89:0/3629808113 learned_addr learned my addr 172.21.15.89:0/3629808113 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:13.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8abe00700 1 --2- 172.21.15.89:0/3629808113 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8ac0d2370 0x7fd8ac0cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8b1200700 1 -- 172.21.15.89:0/3629808113 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8ac0d2370 msgr2=0x7fd8ac0cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8b1200700 1 --2- 172.21.15.89:0/3629808113 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8ac0d2370 0x7fd8ac0cbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8b1200700 1 -- 172.21.15.89:0/3629808113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ac15d9d0 msgr2=0x7fd8ac1606d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8b1200700 1 --2- 172.21.15.89:0/3629808113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ac15d9d0 0x7fd8ac1606d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.421+0000 7fd8b1200700 1 -- 172.21.15.89:0/3629808113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd898009430 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.422+0000 7fd8b1200700 1 --2- 172.21.15.89:0/3629808113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8ac0d16a0 0x7fd8ac14e9e0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000c450 tx=0x7fd8a000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.422+0000 7fd8a8c00700 1 -- 172.21.15.89:0/3629808113 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a00052d0 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.422+0000 7fd8a8c00700 1 -- 172.21.15.89:0/3629808113 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a0005430 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.422+0000 7fd8b3c00700 1 -- 172.21.15.89:0/3629808113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ac1f39d0 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.422+0000 7fd8a8c00700 1 -- 172.21.15.89:0/3629808113 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a00038c0 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.422+0000 7fd8b3c00700 1 -- 172.21.15.89:0/3629808113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ac1f3f90 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.423+0000 7fd8b3c00700 1 -- 172.21.15.89:0/3629808113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ac0d1ca0 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.424+0000 7fd8a8c00700 1 -- 172.21.15.89:0/3629808113 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd8a0003a20 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.425+0000 7fd8a8c00700 1 --2- 172.21.15.89:0/3629808113 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd88c06adb0 0x7fd88c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.425+0000 7fd8a8c00700 1 -- 172.21.15.89:0/3629808113 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5965+0+0 (secure 0 0 0) 0x7fd8a00889e0 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.425+0000 7fd8abe00700 1 --2- 172.21.15.89:0/3629808113 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd88c06adb0 0x7fd88c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.425+0000 7fd8abe00700 1 --2- 172.21.15.89:0/3629808113 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd88c06adb0 0x7fd88c06d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac0d1ae0 tx=0x7fd898008070 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.429+0000 7fd8a8c00700 1 -- 172.21.15.89:0/3629808113 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd8a0053910 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.634+0000 7fd8b3c00700 1 -- 172.21.15.89:0/3629808113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fd8ac067720 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.638+0000 7fd8a8c00700 1 -- 172.21.15.89:0/3629808113 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v62) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd8a0056f30 con 0x7fd8ac0d16a0 2024-04-02T08:53:13.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.640+0000 7fd8b3c00700 1 -- 172.21.15.89:0/3629808113 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd88c06adb0 msgr2=0x7fd88c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:13.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.640+0000 7fd8b3c00700 1 --2- 172.21.15.89:0/3629808113 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd88c06adb0 0x7fd88c06d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac0d1ae0 tx=0x7fd898008070 comp rx=0 tx=0).stop 2024-04-02T08:53:13.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.641+0000 7fd8b3c00700 1 -- 172.21.15.89:0/3629808113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8ac0d16a0 msgr2=0x7fd8ac14e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:13.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.641+0000 7fd8b3c00700 1 --2- 172.21.15.89:0/3629808113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8ac0d16a0 0x7fd8ac14e9e0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000c450 tx=0x7fd8a000c810 comp rx=0 tx=0).stop 2024-04-02T08:53:13.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.641+0000 7fd8b3c00700 1 -- 172.21.15.89:0/3629808113 shutdown_connections 2024-04-02T08:53:13.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.641+0000 7fd8b3c00700 1 --2- 172.21.15.89:0/3629808113 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd88c06adb0 0x7fd88c06d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.641+0000 7fd8b3c00700 1 --2- 172.21.15.89:0/3629808113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ac15d9d0 0x7fd8ac1606d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.641+0000 7fd8b3c00700 1 --2- 172.21.15.89:0/3629808113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8ac0d16a0 0x7fd8ac14e9e0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.641+0000 7fd8b3c00700 1 --2- 172.21.15.89:0/3629808113 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd8ac0d2370 0x7fd8ac0cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.641+0000 7fd8b3c00700 1 -- 172.21.15.89:0/3629808113 >> 172.21.15.89:0/3629808113 conn(0x7fd8ac08aff0 msgr2=0x7fd8ac0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:13.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.641+0000 7fd8b3c00700 1 -- 172.21.15.89:0/3629808113 shutdown_connections 2024-04-02T08:53:13.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.641+0000 7fd8b3c00700 1 -- 172.21.15.89:0/3629808113 wait complete. 2024-04-02T08:53:13.658 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T08:53:13.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.856+0000 7f78ff600700 1 Processor -- start 2024-04-02T08:53:13.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.856+0000 7f78ff600700 1 -- start start 2024-04-02T08:53:13.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.857+0000 7f78ff600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f78f815d9d0 0x7f78f80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:13.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.857+0000 7f78ff600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78f80d16a0 0x7f78f80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:13.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.857+0000 7f78ff600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78f80d2370 0x7f78f815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:13.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.857+0000 7f78ff600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f80d2820 con 0x7f78f80d2370 2024-04-02T08:53:13.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.857+0000 7f78ff600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f8161420 con 0x7f78f815d9d0 2024-04-02T08:53:13.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.857+0000 7f78ff600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f8161700 con 0x7f78f80d16a0 2024-04-02T08:53:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.857+0000 7f78fd600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78f80d2370 0x7f78f815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.857+0000 7f78fd600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78f80d2370 0x7f78f815ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53462/0 (socket says 172.21.15.89:53462) 2024-04-02T08:53:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.857+0000 7f78fd600700 1 -- 172.21.15.89:0/38455084 learned_addr learned my addr 172.21.15.89:0/38455084 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.857+0000 7f78fcc00700 1 --2- 172.21.15.89:0/38455084 >> v2:172.21.15.113:3300/0 conn(0x7f78f815d9d0 0x7f78f80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.857+0000 7f78f7e00700 1 --2- 172.21.15.89:0/38455084 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78f80d16a0 0x7f78f80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.858+0000 7f78fd600700 1 -- 172.21.15.89:0/38455084 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78f80d16a0 msgr2=0x7f78f80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.858+0000 7f78fd600700 1 --2- 172.21.15.89:0/38455084 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78f80d16a0 0x7f78f80d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.858+0000 7f78fd600700 1 -- 172.21.15.89:0/38455084 >> v2:172.21.15.113:3300/0 conn(0x7f78f815d9d0 msgr2=0x7f78f80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.858+0000 7f78fd600700 1 --2- 172.21.15.89:0/38455084 >> v2:172.21.15.113:3300/0 conn(0x7f78f815d9d0 0x7f78f80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.858+0000 7f78fd600700 1 -- 172.21.15.89:0/38455084 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f81619e0 con 0x7f78f80d2370 2024-04-02T08:53:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.858+0000 7f78f7e00700 1 --2- 172.21.15.89:0/38455084 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78f80d16a0 0x7f78f80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:13.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.858+0000 7f78fd600700 1 --2- 172.21.15.89:0/38455084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78f80d2370 0x7f78f815ddb0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f78f80d0ff0 tx=0x7f78f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd5e65274b221c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:13.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.858+0000 7f78f7400700 1 -- 172.21.15.89:0/38455084 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f000e070 con 0x7f78f80d2370 2024-04-02T08:53:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.858+0000 7f78f7400700 1 -- 172.21.15.89:0/38455084 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f000b910 con 0x7f78f80d2370 2024-04-02T08:53:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.858+0000 7f78f7400700 1 -- 172.21.15.89:0/38455084 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f000bcc0 con 0x7f78f80d2370 2024-04-02T08:53:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.859+0000 7f78ff600700 1 -- 172.21.15.89:0/38455084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78f80d2370 msgr2=0x7f78f815ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.859+0000 7f78ff600700 1 --2- 172.21.15.89:0/38455084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78f80d2370 0x7f78f815ddb0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f78f80d0ff0 tx=0x7f78f000b330 comp rx=0 tx=0).stop 2024-04-02T08:53:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.859+0000 7f78ff600700 1 -- 172.21.15.89:0/38455084 shutdown_connections 2024-04-02T08:53:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.859+0000 7f78ff600700 1 --2- 172.21.15.89:0/38455084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78f80d2370 0x7f78f815ddb0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.859+0000 7f78ff600700 1 --2- 172.21.15.89:0/38455084 >> v2:172.21.15.113:3300/0 conn(0x7f78f815d9d0 0x7f78f80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.859+0000 7f78ff600700 1 --2- 172.21.15.89:0/38455084 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78f80d16a0 0x7f78f80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.859+0000 7f78ff600700 1 -- 172.21.15.89:0/38455084 >> 172.21.15.89:0/38455084 conn(0x7f78f808aff0 msgr2=0x7f78f80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.859+0000 7f78ff600700 1 -- 172.21.15.89:0/38455084 shutdown_connections 2024-04-02T08:53:13.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.859+0000 7f78ff600700 1 -- 172.21.15.89:0/38455084 wait complete. 2024-04-02T08:53:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.861+0000 7f78ff600700 1 Processor -- start 2024-04-02T08:53:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.861+0000 7f78ff600700 1 -- start start 2024-04-02T08:53:13.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.861+0000 7f78ff600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78f80d16a0 0x7f78f81f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:13.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78ff600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78f80d2370 0x7f78f815cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:13.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78ff600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78f815d9d0 0x7f78f80b8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:13.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78ff600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f80b7b90 con 0x7f78f80d2370 2024-04-02T08:53:13.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78ff600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f81f42a0 con 0x7f78f80d16a0 2024-04-02T08:53:13.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78ff600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f815d000 con 0x7f78f815d9d0 2024-04-02T08:53:13.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78f7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78f80d2370 0x7f78f815cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:13.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78f7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78f80d2370 0x7f78f815cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53474/0 (socket says 172.21.15.89:53474) 2024-04-02T08:53:13.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78f7e00700 1 -- 172.21.15.89:0/1341417983 learned_addr learned my addr 172.21.15.89:0/1341417983 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:13.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78fd600700 1 --2- 172.21.15.89:0/1341417983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78f815d9d0 0x7f78f80b8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:13.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78fcc00700 1 --2- 172.21.15.89:0/1341417983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78f80d16a0 0x7f78f81f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:13.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78f7e00700 1 -- 172.21.15.89:0/1341417983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78f815d9d0 msgr2=0x7f78f80b8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:13.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78f7e00700 1 --2- 172.21.15.89:0/1341417983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78f815d9d0 0x7f78f80b8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78f7e00700 1 -- 172.21.15.89:0/1341417983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78f80d16a0 msgr2=0x7f78f81f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:13.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78f7e00700 1 --2- 172.21.15.89:0/1341417983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78f80d16a0 0x7f78f81f3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.862+0000 7f78f7e00700 1 -- 172.21.15.89:0/1341417983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f0009b30 con 0x7f78f80d2370 2024-04-02T08:53:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.863+0000 7f78f7e00700 1 --2- 172.21.15.89:0/1341417983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78f80d2370 0x7f78f815cac0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f78e400e7e0 tx=0x7f78e400eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.863+0000 7f78f4c00700 1 -- 172.21.15.89:0/1341417983 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e400c070 con 0x7f78f80d2370 2024-04-02T08:53:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.863+0000 7f78f4c00700 1 -- 172.21.15.89:0/1341417983 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e4005c20 con 0x7f78f80d2370 2024-04-02T08:53:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.863+0000 7f78ff600700 1 -- 172.21.15.89:0/1341417983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f80ccbc0 con 0x7f78f80d2370 2024-04-02T08:53:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.863+0000 7f78f4c00700 1 -- 172.21.15.89:0/1341417983 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4004030 con 0x7f78f80d2370 2024-04-02T08:53:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.863+0000 7f78ff600700 1 -- 172.21.15.89:0/1341417983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f80b7540 con 0x7f78f80d2370 2024-04-02T08:53:13.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.864+0000 7f78ff600700 1 -- 172.21.15.89:0/1341417983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f80d1ca0 con 0x7f78f80d2370 2024-04-02T08:53:13.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.865+0000 7f78f4c00700 1 -- 172.21.15.89:0/1341417983 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f78e4004190 con 0x7f78f80d2370 2024-04-02T08:53:13.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.866+0000 7f78f4c00700 1 --2- 172.21.15.89:0/1341417983 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f78d806adb0 0x7f78d806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:13.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.866+0000 7f78f4c00700 1 -- 172.21.15.89:0/1341417983 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5965+0+0 (secure 0 0 0) 0x7f78e4088dd0 con 0x7f78f80d2370 2024-04-02T08:53:13.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.866+0000 7f78fcc00700 1 --2- 172.21.15.89:0/1341417983 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f78d806adb0 0x7f78d806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:13.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.867+0000 7f78fcc00700 1 --2- 172.21.15.89:0/1341417983 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f78d806adb0 0x7f78d806d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009b40 tx=0x7f78ec009210 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:13.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:13.870+0000 7f78f4c00700 1 -- 172.21.15.89:0/1341417983 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f78e4053c00 con 0x7f78f80d2370 2024-04-02T08:53:14.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.073+0000 7f78ff600700 1 -- 172.21.15.89:0/1341417983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f78f8067720 con 0x7f78f80d2370 2024-04-02T08:53:14.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.077+0000 7f78f4c00700 1 -- 172.21.15.89:0/1341417983 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v63) v1 ==== 104+0+0 (secure 0 0 0) 0x7f78e4057220 con 0x7f78f80d2370 2024-04-02T08:53:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.080+0000 7f78ff600700 1 -- 172.21.15.89:0/1341417983 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f78d806adb0 msgr2=0x7f78d806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.080+0000 7f78ff600700 1 --2- 172.21.15.89:0/1341417983 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f78d806adb0 0x7f78d806d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009b40 tx=0x7f78ec009210 comp rx=0 tx=0).stop 2024-04-02T08:53:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.080+0000 7f78ff600700 1 -- 172.21.15.89:0/1341417983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78f80d2370 msgr2=0x7f78f815cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.080+0000 7f78ff600700 1 --2- 172.21.15.89:0/1341417983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78f80d2370 0x7f78f815cac0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f78e400e7e0 tx=0x7f78e400eaf0 comp rx=0 tx=0).stop 2024-04-02T08:53:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.080+0000 7f78ff600700 1 -- 172.21.15.89:0/1341417983 shutdown_connections 2024-04-02T08:53:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.080+0000 7f78ff600700 1 --2- 172.21.15.89:0/1341417983 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f78d806adb0 0x7f78d806d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.080+0000 7f78ff600700 1 --2- 172.21.15.89:0/1341417983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78f80d16a0 0x7f78f81f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.080+0000 7f78ff600700 1 --2- 172.21.15.89:0/1341417983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78f815d9d0 0x7f78f80b8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.080+0000 7f78ff600700 1 --2- 172.21.15.89:0/1341417983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78f80d2370 0x7f78f815cac0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.080+0000 7f78ff600700 1 -- 172.21.15.89:0/1341417983 >> 172.21.15.89:0/1341417983 conn(0x7f78f808aff0 msgr2=0x7f78f80b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.080+0000 7f78ff600700 1 -- 172.21.15.89:0/1341417983 shutdown_connections 2024-04-02T08:53:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.080+0000 7f78ff600700 1 -- 172.21.15.89:0/1341417983 wait complete. 2024-04-02T08:53:14.097 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-04-02T08:53:14.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.289+0000 7fa573400700 1 Processor -- start 2024-04-02T08:53:14.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.289+0000 7fa573400700 1 -- start start 2024-04-02T08:53:14.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.290+0000 7fa573400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa56c1619d0 0x7fa56c1685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:14.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.290+0000 7fa573400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c0d30d0 con 0x7fa56c15d640 2024-04-02T08:53:14.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.290+0000 7fa573400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c0b6840 con 0x7fa56c0ceec0 2024-04-02T08:53:14.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.290+0000 7fa573400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa56c0d2b30 con 0x7fa56c1619d0 2024-04-02T08:53:14.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.291+0000 7fa571400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa56c1619d0 0x7fa56c1685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:14.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.291+0000 7fa570a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa56c15d640 0x7fa56c0d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58166/0 (socket says 172.21.15.89:58166) 2024-04-02T08:53:14.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.291+0000 7fa570a00700 1 -- 172.21.15.89:0/881833285 learned_addr learned my addr 172.21.15.89:0/881833285 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:14.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.291+0000 7fa571400700 1 -- 172.21.15.89:0/881833285 >> v1:172.21.15.113:6789/0 conn(0x7fa56c0ceec0 legacy=0x7fa56c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:14.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.291+0000 7fa571400700 1 -- 172.21.15.89:0/881833285 >> v1:172.21.15.89:6789/0 conn(0x7fa56c15d640 legacy=0x7fa56c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:14.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.291+0000 7fa571400700 1 -- 172.21.15.89:0/881833285 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c0d2df0 con 0x7fa56c1619d0 2024-04-02T08:53:14.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.291+0000 7fa571400700 1 --2- 172.21.15.89:0/881833285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56c1619d0 0x7fa56c1685f0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fa564009d90 tx=0x7fa56400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a5098a9b96c3052 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:14.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.291+0000 7fa56b400700 1 -- 172.21.15.89:0/881833285 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56400e070 con 0x7fa56c1619d0 2024-04-02T08:53:14.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.292+0000 7fa56b400700 1 -- 172.21.15.89:0/881833285 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56400bb90 con 0x7fa56c1619d0 2024-04-02T08:53:14.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.292+0000 7fa56b400700 1 -- 172.21.15.89:0/881833285 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa564003a20 con 0x7fa56c1619d0 2024-04-02T08:53:14.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.292+0000 7fa573400700 1 -- 172.21.15.89:0/881833285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56c1619d0 msgr2=0x7fa56c1685f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:14.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.292+0000 7fa573400700 1 --2- 172.21.15.89:0/881833285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56c1619d0 0x7fa56c1685f0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fa564009d90 tx=0x7fa56400b330 comp rx=0 tx=0).stop 2024-04-02T08:53:14.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.292+0000 7fa573400700 1 -- 172.21.15.89:0/881833285 shutdown_connections 2024-04-02T08:53:14.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.292+0000 7fa573400700 1 --2- 172.21.15.89:0/881833285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56c1619d0 0x7fa56c1685f0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:14.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.292+0000 7fa573400700 1 -- 172.21.15.89:0/881833285 >> 172.21.15.89:0/881833285 conn(0x7fa56c08aff0 msgr2=0x7fa56c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:14.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.292+0000 7fa573400700 1 -- 172.21.15.89:0/881833285 shutdown_connections 2024-04-02T08:53:14.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.292+0000 7fa573400700 1 -- 172.21.15.89:0/881833285 wait complete. 2024-04-02T08:53:14.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.294+0000 7fa573400700 1 Processor -- start 2024-04-02T08:53:14.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.294+0000 7fa573400700 1 -- start start 2024-04-02T08:53:14.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.295+0000 7fa573400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56c0ceec0 0x7fa56c15be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:14.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.295+0000 7fa573400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa56c15d640 0x7fa56c0c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:14.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.295+0000 7fa573400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa56c1619d0 0x7fa56c1f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:14.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.295+0000 7fa573400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0c7a90 con 0x7fa56c0ceec0 2024-04-02T08:53:14.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.295+0000 7fa573400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c15c360 con 0x7fa56c15d640 2024-04-02T08:53:14.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.295+0000 7fa573400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0c9bc0 con 0x7fa56c1619d0 2024-04-02T08:53:14.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.295+0000 7fa570a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56c0ceec0 0x7fa56c15be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:14.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.295+0000 7fa570a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56c0ceec0 0x7fa56c15be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53486/0 (socket says 172.21.15.89:53486) 2024-04-02T08:53:14.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.295+0000 7fa570a00700 1 -- 172.21.15.89:0/1983241777 learned_addr learned my addr 172.21.15.89:0/1983241777 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:14.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.295+0000 7fa56be00700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa56c15d640 0x7fa56c0c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:14.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.295+0000 7fa571400700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa56c1619d0 0x7fa56c1f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:14.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.296+0000 7fa570a00700 1 -- 172.21.15.89:0/1983241777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa56c1619d0 msgr2=0x7fa56c1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:14.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.296+0000 7fa570a00700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa56c1619d0 0x7fa56c1f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:14.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.296+0000 7fa570a00700 1 -- 172.21.15.89:0/1983241777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa56c15d640 msgr2=0x7fa56c0c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:14.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.296+0000 7fa570a00700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa56c15d640 0x7fa56c0c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:14.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.296+0000 7fa570a00700 1 -- 172.21.15.89:0/1983241777 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa564009a70 con 0x7fa56c0ceec0 2024-04-02T08:53:14.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.296+0000 7fa56be00700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa56c15d640 0x7fa56c0c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:14.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.296+0000 7fa570a00700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56c0ceec0 0x7fa56c15be20 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fa56000c4c0 tx=0x7fa56000c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:14.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.296+0000 7fa568c00700 1 -- 172.21.15.89:0/1983241777 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa560005d70 con 0x7fa56c0ceec0 2024-04-02T08:53:14.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.296+0000 7fa568c00700 1 -- 172.21.15.89:0/1983241777 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5600070d0 con 0x7fa56c0ceec0 2024-04-02T08:53:14.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.296+0000 7fa568c00700 1 -- 172.21.15.89:0/1983241777 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa560007460 con 0x7fa56c0ceec0 2024-04-02T08:53:14.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.296+0000 7fa573400700 1 -- 172.21.15.89:0/1983241777 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56c0c9ea0 con 0x7fa56c0ceec0 2024-04-02T08:53:14.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.296+0000 7fa573400700 1 -- 172.21.15.89:0/1983241777 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56c0c7fc0 con 0x7fa56c0ceec0 2024-04-02T08:53:14.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.297+0000 7fa573400700 1 -- 172.21.15.89:0/1983241777 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56c0c81b0 con 0x7fa56c0ceec0 2024-04-02T08:53:14.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.298+0000 7fa568c00700 1 -- 172.21.15.89:0/1983241777 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa5600075c0 con 0x7fa56c0ceec0 2024-04-02T08:53:14.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.299+0000 7fa568c00700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa54c06adb0 0x7fa54c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:14.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.299+0000 7fa568c00700 1 -- 172.21.15.89:0/1983241777 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5965+0+0 (secure 0 0 0) 0x7fa5600889f0 con 0x7fa56c0ceec0 2024-04-02T08:53:14.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.299+0000 7fa56be00700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa54c06adb0 0x7fa54c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:14.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.300+0000 7fa56be00700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa54c06adb0 0x7fa54c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa56c161780 tx=0x7fa558009040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:14.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.304+0000 7fa568c00700 1 -- 172.21.15.89:0/1983241777 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa560053a70 con 0x7fa56c0ceec0 2024-04-02T08:53:14.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.596+0000 7fa573400700 1 -- 172.21.15.89:0/1983241777 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7fa56c067720 con 0x7fa56c0ceec0 2024-04-02T08:53:14.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.917+0000 7fa568c00700 1 -- 172.21.15.89:0/1983241777 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2519) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa560017020 con 0x7fa56c0ceec0 2024-04-02T08:53:14.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.919+0000 7fa573400700 1 -- 172.21.15.89:0/1983241777 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa54c06adb0 msgr2=0x7fa54c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:14.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.919+0000 7fa573400700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa54c06adb0 0x7fa54c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa56c161780 tx=0x7fa558009040 comp rx=0 tx=0).stop 2024-04-02T08:53:14.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.919+0000 7fa573400700 1 -- 172.21.15.89:0/1983241777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56c0ceec0 msgr2=0x7fa56c15be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:14.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.919+0000 7fa573400700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56c0ceec0 0x7fa56c15be20 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fa56000c4c0 tx=0x7fa56000c7d0 comp rx=0 tx=0).stop 2024-04-02T08:53:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.920+0000 7fa573400700 1 -- 172.21.15.89:0/1983241777 shutdown_connections 2024-04-02T08:53:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.920+0000 7fa573400700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fa54c06adb0 0x7fa54c06d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.920+0000 7fa573400700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa56c1619d0 0x7fa56c1f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.920+0000 7fa573400700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa56c0ceec0 0x7fa56c15be20 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.920+0000 7fa573400700 1 --2- 172.21.15.89:0/1983241777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa56c15d640 0x7fa56c0c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.920+0000 7fa573400700 1 -- 172.21.15.89:0/1983241777 >> 172.21.15.89:0/1983241777 conn(0x7fa56c08aff0 msgr2=0x7fa56c0b8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.920+0000 7fa573400700 1 -- 172.21.15.89:0/1983241777 shutdown_connections 2024-04-02T08:53:14.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:14.920+0000 7fa573400700 1 -- 172.21.15.89:0/1983241777 wait complete. 2024-04-02T08:53:14.938 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2024-04-02T08:53:14.938 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2024-04-02T08:53:14.938 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-02T08:53:14.939 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-04-02T08:53:15.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.134+0000 7f90b1600700 1 Processor -- start 2024-04-02T08:53:15.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.134+0000 7f90b1600700 1 -- start start 2024-04-02T08:53:15.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.135+0000 7f90b1600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f90ac160f10 0x7f90ac0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:15.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.135+0000 7f90b1600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f90ac161590 0x7f90ac161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:15.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.135+0000 7f90b1600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ac0ceda0 con 0x7f90ac0cf230 2024-04-02T08:53:15.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.135+0000 7f90b1600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ac161f80 con 0x7f90ac161590 2024-04-02T08:53:15.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.135+0000 7f90b1600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ac1612f0 con 0x7f90ac160f10 2024-04-02T08:53:15.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.135+0000 7f90ab400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f90ac161590 0x7f90ac161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:15.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.135+0000 7f90aa000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f90ac0cf230 0x7f90ac0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58174/0 (socket says 172.21.15.89:58174) 2024-04-02T08:53:15.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.135+0000 7f90aa000700 1 -- 172.21.15.89:0/1066755869 learned_addr learned my addr 172.21.15.89:0/1066755869 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:15.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.136+0000 7f90aaa00700 1 --2- 172.21.15.89:0/1066755869 >> v2:172.21.15.113:3300/0 conn(0x7f90ac160f10 0x7f90ac0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:15.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.136+0000 7f90ab400700 1 -- 172.21.15.89:0/1066755869 >> v2:172.21.15.113:3300/0 conn(0x7f90ac160f10 msgr2=0x7f90ac0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:15.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.136+0000 7f90ab400700 1 --2- 172.21.15.89:0/1066755869 >> v2:172.21.15.113:3300/0 conn(0x7f90ac160f10 0x7f90ac0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:15.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.136+0000 7f90ab400700 1 -- 172.21.15.89:0/1066755869 >> v1:172.21.15.89:6789/0 conn(0x7f90ac0cf230 legacy=0x7f90ac0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:15.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.136+0000 7f90ab400700 1 -- 172.21.15.89:0/1066755869 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac0ce470 con 0x7f90ac161590 2024-04-02T08:53:15.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.136+0000 7f90aaa00700 1 --2- 172.21.15.89:0/1066755869 >> v2:172.21.15.113:3300/0 conn(0x7f90ac160f10 0x7f90ac0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:53:15.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.136+0000 7f90a9600700 1 -- 172.21.15.89:0/1066755869 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122220803 0 0) 0x7f90ac0ceda0 con 0x7f90ac0cf230 2024-04-02T08:53:15.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.136+0000 7f90ab400700 1 --2- 172.21.15.89:0/1066755869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90ac161590 0x7f90ac161a40 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0d1080 tx=0x7f90a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cea91e16476fc2f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:15.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.136+0000 7f90a9600700 1 -- 172.21.15.89:0/1066755869 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a400e070 con 0x7f90ac161590 2024-04-02T08:53:15.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.136+0000 7f90a9600700 1 -- 172.21.15.89:0/1066755869 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a400bbd0 con 0x7f90ac161590 2024-04-02T08:53:15.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.136+0000 7f90a9600700 1 -- 172.21.15.89:0/1066755869 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a4012430 con 0x7f90ac161590 2024-04-02T08:53:15.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.137+0000 7f90b1600700 1 -- 172.21.15.89:0/1066755869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90ac161590 msgr2=0x7f90ac161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:15.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.137+0000 7f90b1600700 1 --2- 172.21.15.89:0/1066755869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90ac161590 0x7f90ac161a40 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0d1080 tx=0x7f90a400b330 comp rx=0 tx=0).stop 2024-04-02T08:53:15.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.137+0000 7f90b1600700 1 -- 172.21.15.89:0/1066755869 shutdown_connections 2024-04-02T08:53:15.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.137+0000 7f90b1600700 1 --2- 172.21.15.89:0/1066755869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90ac161590 0x7f90ac161a40 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:15.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.137+0000 7f90b1600700 1 --2- 172.21.15.89:0/1066755869 >> v2:172.21.15.113:3300/0 conn(0x7f90ac160f10 0x7f90ac0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:15.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.137+0000 7f90b1600700 1 -- 172.21.15.89:0/1066755869 >> 172.21.15.89:0/1066755869 conn(0x7f90ac08aff0 msgr2=0x7f90ac0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:15.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.137+0000 7f90b1600700 1 -- 172.21.15.89:0/1066755869 shutdown_connections 2024-04-02T08:53:15.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.137+0000 7f90b1600700 1 -- 172.21.15.89:0/1066755869 wait complete. 2024-04-02T08:53:15.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.139+0000 7f90b1600700 1 Processor -- start 2024-04-02T08:53:15.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.139+0000 7f90b1600700 1 -- start start 2024-04-02T08:53:15.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90b1600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90ac0cf230 0x7f90ac15f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:15.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90b1600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90ac160f10 0x7f90ac15d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:15.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90b1600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90ac161590 0x7f90ac15d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:15.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90b1600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac0b6120 con 0x7f90ac160f10 2024-04-02T08:53:15.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90b1600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac0b6400 con 0x7f90ac161590 2024-04-02T08:53:15.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90b1600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac15dd90 con 0x7f90ac0cf230 2024-04-02T08:53:15.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90aa000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90ac160f10 0x7f90ac15d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:15.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90aa000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90ac160f10 0x7f90ac15d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53504/0 (socket says 172.21.15.89:53504) 2024-04-02T08:53:15.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90aa000700 1 -- 172.21.15.89:0/1214521680 learned_addr learned my addr 172.21.15.89:0/1214521680 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:15.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90aaa00700 1 --2- 172.21.15.89:0/1214521680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90ac0cf230 0x7f90ac15f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:15.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90ab400700 1 --2- 172.21.15.89:0/1214521680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90ac161590 0x7f90ac15d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:15.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90aa000700 1 -- 172.21.15.89:0/1214521680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90ac0cf230 msgr2=0x7f90ac15f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:15.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90aa000700 1 --2- 172.21.15.89:0/1214521680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90ac0cf230 0x7f90ac15f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:15.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90aa000700 1 -- 172.21.15.89:0/1214521680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90ac161590 msgr2=0x7f90ac15d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:15.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90aa000700 1 --2- 172.21.15.89:0/1214521680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90ac161590 0x7f90ac15d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:15.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90aa000700 1 -- 172.21.15.89:0/1214521680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a4009a70 con 0x7f90ac160f10 2024-04-02T08:53:15.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.140+0000 7f90aa000700 1 --2- 172.21.15.89:0/1214521680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90ac160f10 0x7f90ac15d310 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f909800c550 tx=0x7f909800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:15.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.141+0000 7f90a2a00700 1 -- 172.21.15.89:0/1214521680 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9098004c80 con 0x7f90ac160f10 2024-04-02T08:53:15.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.141+0000 7f90a2a00700 1 -- 172.21.15.89:0/1214521680 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9098004de0 con 0x7f90ac160f10 2024-04-02T08:53:15.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.141+0000 7f90a2a00700 1 -- 172.21.15.89:0/1214521680 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90980072d0 con 0x7f90ac160f10 2024-04-02T08:53:15.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.141+0000 7f90b1600700 1 -- 172.21.15.89:0/1214521680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ac15e060 con 0x7f90ac160f10 2024-04-02T08:53:15.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.141+0000 7f90b1600700 1 -- 172.21.15.89:0/1214521680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ac15e620 con 0x7f90ac160f10 2024-04-02T08:53:15.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.142+0000 7f90b1600700 1 -- 172.21.15.89:0/1214521680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ac067720 con 0x7f90ac160f10 2024-04-02T08:53:15.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.143+0000 7f90a2a00700 1 -- 172.21.15.89:0/1214521680 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9098007430 con 0x7f90ac160f10 2024-04-02T08:53:15.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.144+0000 7f90a2a00700 1 --2- 172.21.15.89:0/1214521680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f908c06adb0 0x7f908c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:15.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.144+0000 7f90a2a00700 1 -- 172.21.15.89:0/1214521680 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5965+0+0 (secure 0 0 0) 0x7f9098089370 con 0x7f90ac160f10 2024-04-02T08:53:15.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.144+0000 7f90aaa00700 1 --2- 172.21.15.89:0/1214521680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f908c06adb0 0x7f908c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:15.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.144+0000 7f90aaa00700 1 --2- 172.21.15.89:0/1214521680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f908c06adb0 0x7f908c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f909c006fd0 tx=0x7f909c008040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:15.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.148+0000 7f90a2a00700 1 -- 172.21.15.89:0/1214521680 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f90980542a0 con 0x7f90ac160f10 2024-04-02T08:53:15.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.426+0000 7f90b1600700 1 -- 172.21.15.89:0/1214521680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f90ac0b6540 con 0x7f90ac160f10 2024-04-02T08:53:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.921+0000 7f90a2a00700 1 -- 172.21.15.89:0/1214521680 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2520) v1 ==== 168+0+0 (secure 0 0 0) 0x7f90980578c0 con 0x7f90ac160f10 2024-04-02T08:53:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.923+0000 7f90b1600700 1 -- 172.21.15.89:0/1214521680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f908c06adb0 msgr2=0x7f908c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.923+0000 7f90b1600700 1 --2- 172.21.15.89:0/1214521680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f908c06adb0 0x7f908c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f909c006fd0 tx=0x7f909c008040 comp rx=0 tx=0).stop 2024-04-02T08:53:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.923+0000 7f90b1600700 1 -- 172.21.15.89:0/1214521680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90ac160f10 msgr2=0x7f90ac15d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.923+0000 7f90b1600700 1 --2- 172.21.15.89:0/1214521680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90ac160f10 0x7f90ac15d310 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f909800c550 tx=0x7f909800c910 comp rx=0 tx=0).stop 2024-04-02T08:53:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.923+0000 7f90b1600700 1 -- 172.21.15.89:0/1214521680 shutdown_connections 2024-04-02T08:53:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.923+0000 7f90b1600700 1 --2- 172.21.15.89:0/1214521680 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f908c06adb0 0x7f908c06d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.923+0000 7f90b1600700 1 --2- 172.21.15.89:0/1214521680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90ac161590 0x7f90ac15d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.923+0000 7f90b1600700 1 --2- 172.21.15.89:0/1214521680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90ac0cf230 0x7f90ac15f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.923+0000 7f90b1600700 1 --2- 172.21.15.89:0/1214521680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90ac160f10 0x7f90ac15d310 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.923+0000 7f90b1600700 1 -- 172.21.15.89:0/1214521680 >> 172.21.15.89:0/1214521680 conn(0x7f90ac08aff0 msgr2=0x7f90ac0ca8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.924+0000 7f90b1600700 1 -- 172.21.15.89:0/1214521680 shutdown_connections 2024-04-02T08:53:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:15.924+0000 7f90b1600700 1 -- 172.21.15.89:0/1214521680 wait complete. 2024-04-02T08:53:15.942 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T08:53:16.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.140+0000 7f86cc200700 1 Processor -- start 2024-04-02T08:53:16.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.140+0000 7f86cc200700 1 -- start start 2024-04-02T08:53:16.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.141+0000 7f86cc200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f86c415a5c0 0x7f86c40709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:16.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.141+0000 7f86cc200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f86c40ce840 0x7f86c40cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:16.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.141+0000 7f86cc200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c40cb310 con 0x7f86c415dfe0 2024-04-02T08:53:16.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.141+0000 7f86cc200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c4070f00 con 0x7f86c415a5c0 2024-04-02T08:53:16.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.141+0000 7f86cc200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c40711a0 con 0x7f86c40ce840 2024-04-02T08:53:16.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.141+0000 7f86c9800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f86c415a5c0 0x7f86c40709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:16.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.141+0000 7f86c9800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f86c415a5c0 0x7f86c40709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53516/0 (socket says 172.21.15.89:53516) 2024-04-02T08:53:16.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.141+0000 7f86c9800700 1 -- 172.21.15.89:0/1494783190 learned_addr learned my addr 172.21.15.89:0/1494783190 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:16.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.142+0000 7f86c9800700 1 -- 172.21.15.89:0/1494783190 >> v2:172.21.15.113:3300/0 conn(0x7f86c40ce840 msgr2=0x7f86c40cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:16.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.142+0000 7f86c9800700 1 --2- 172.21.15.89:0/1494783190 >> v2:172.21.15.113:3300/0 conn(0x7f86c40ce840 0x7f86c40cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.142+0000 7f86c9800700 1 -- 172.21.15.89:0/1494783190 >> v1:172.21.15.113:6789/0 conn(0x7f86c415dfe0 legacy=0x7f86c40ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:16.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.142+0000 7f86c9800700 1 -- 172.21.15.89:0/1494783190 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c4071480 con 0x7f86c415a5c0 2024-04-02T08:53:16.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.142+0000 7f86c9800700 1 --2- 172.21.15.89:0/1494783190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c415a5c0 0x7f86c40709c0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f86c406b630 tx=0x7f86b800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4b14daa5618bef server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:16.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.142+0000 7f86c3e00700 1 -- 172.21.15.89:0/1494783190 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b800dcf0 con 0x7f86c415a5c0 2024-04-02T08:53:16.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.142+0000 7f86c3e00700 1 -- 172.21.15.89:0/1494783190 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b800de50 con 0x7f86c415a5c0 2024-04-02T08:53:16.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.142+0000 7f86c3e00700 1 -- 172.21.15.89:0/1494783190 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8004250 con 0x7f86c415a5c0 2024-04-02T08:53:16.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.143+0000 7f86cc200700 1 -- 172.21.15.89:0/1494783190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c415a5c0 msgr2=0x7f86c40709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:16.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.143+0000 7f86cc200700 1 --2- 172.21.15.89:0/1494783190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c415a5c0 0x7f86c40709c0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f86c406b630 tx=0x7f86b800be70 comp rx=0 tx=0).stop 2024-04-02T08:53:16.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.143+0000 7f86cc200700 1 -- 172.21.15.89:0/1494783190 shutdown_connections 2024-04-02T08:53:16.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.143+0000 7f86cc200700 1 --2- 172.21.15.89:0/1494783190 >> v2:172.21.15.113:3300/0 conn(0x7f86c40ce840 0x7f86c40cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.143+0000 7f86cc200700 1 --2- 172.21.15.89:0/1494783190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c415a5c0 0x7f86c40709c0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.143+0000 7f86cc200700 1 -- 172.21.15.89:0/1494783190 >> 172.21.15.89:0/1494783190 conn(0x7f86c40b0d70 msgr2=0x7f86c40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:16.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.143+0000 7f86cc200700 1 -- 172.21.15.89:0/1494783190 shutdown_connections 2024-04-02T08:53:16.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.143+0000 7f86cc200700 1 -- 172.21.15.89:0/1494783190 wait complete. 2024-04-02T08:53:16.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.145+0000 7f86cc200700 1 Processor -- start 2024-04-02T08:53:16.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.145+0000 7f86cc200700 1 -- start start 2024-04-02T08:53:16.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86cc200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40ce840 0x7f86c414c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:16.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86cc200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c415a5c0 0x7f86c414f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:16.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86cc200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c415dfe0 0x7f86c414f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:16.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86cc200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c41f43b0 con 0x7f86c415dfe0 2024-04-02T08:53:16.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86cc200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c415abd0 con 0x7f86c415a5c0 2024-04-02T08:53:16.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86cc200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c415aeb0 con 0x7f86c40ce840 2024-04-02T08:53:16.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86ca200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c415dfe0 0x7f86c414f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:16.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86ca200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c415dfe0 0x7f86c414f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53518/0 (socket says 172.21.15.89:53518) 2024-04-02T08:53:16.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86ca200700 1 -- 172.21.15.89:0/4159845463 learned_addr learned my addr 172.21.15.89:0/4159845463 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:16.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86c9800700 1 --2- 172.21.15.89:0/4159845463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40ce840 0x7f86c414c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:16.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86c8e00700 1 --2- 172.21.15.89:0/4159845463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c415a5c0 0x7f86c414f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:16.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86ca200700 1 -- 172.21.15.89:0/4159845463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40ce840 msgr2=0x7f86c414c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:16.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86ca200700 1 --2- 172.21.15.89:0/4159845463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40ce840 0x7f86c414c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86ca200700 1 -- 172.21.15.89:0/4159845463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c415a5c0 msgr2=0x7f86c414f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:16.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86ca200700 1 --2- 172.21.15.89:0/4159845463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c415a5c0 0x7f86c414f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.146+0000 7f86ca200700 1 -- 172.21.15.89:0/4159845463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b800b7f0 con 0x7f86c415dfe0 2024-04-02T08:53:16.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.147+0000 7f86ca200700 1 --2- 172.21.15.89:0/4159845463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c415dfe0 0x7f86c414f9a0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f86bc00b9e0 tx=0x7f86bc00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:16.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.147+0000 7f86c1600700 1 -- 172.21.15.89:0/4159845463 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86bc00c9b0 con 0x7f86c415dfe0 2024-04-02T08:53:16.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.147+0000 7f86c1600700 1 -- 172.21.15.89:0/4159845463 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86bc00cb10 con 0x7f86c415dfe0 2024-04-02T08:53:16.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.147+0000 7f86cc200700 1 -- 172.21.15.89:0/4159845463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c415b190 con 0x7f86c415dfe0 2024-04-02T08:53:16.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.147+0000 7f86c1600700 1 -- 172.21.15.89:0/4159845463 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86bc014440 con 0x7f86c415dfe0 2024-04-02T08:53:16.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.147+0000 7f86cc200700 1 -- 172.21.15.89:0/4159845463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c414c5f0 con 0x7f86c415dfe0 2024-04-02T08:53:16.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.148+0000 7f86cc200700 1 -- 172.21.15.89:0/4159845463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c415b6b0 con 0x7f86c415dfe0 2024-04-02T08:53:16.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.149+0000 7f86c1600700 1 -- 172.21.15.89:0/4159845463 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f86bc01d3b0 con 0x7f86c415dfe0 2024-04-02T08:53:16.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.150+0000 7f86c1600700 1 --2- 172.21.15.89:0/4159845463 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f86a406adb0 0x7f86a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:16.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.150+0000 7f86c1600700 1 -- 172.21.15.89:0/4159845463 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5965+0+0 (secure 0 0 0) 0x7f86bc006e40 con 0x7f86c415dfe0 2024-04-02T08:53:16.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.150+0000 7f86c9800700 1 --2- 172.21.15.89:0/4159845463 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f86a406adb0 0x7f86a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:16.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.151+0000 7f86c9800700 1 --2- 172.21.15.89:0/4159845463 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f86a406adb0 0x7f86a406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f86c40ad780 tx=0x7f86b801b040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:16.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.154+0000 7f86c1600700 1 -- 172.21.15.89:0/4159845463 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f86c415b6b0 con 0x7f86c415dfe0 2024-04-02T08:53:16.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.355+0000 7f86cc200700 1 -- 172.21.15.89:0/4159845463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f86c415b6b0 con 0x7f86c415dfe0 2024-04-02T08:53:16.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.360+0000 7f86c1600700 1 -- 172.21.15.89:0/4159845463 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f86c415b6b0 con 0x7f86c415dfe0 2024-04-02T08:53:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.362+0000 7f86cc200700 1 -- 172.21.15.89:0/4159845463 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f86a406adb0 msgr2=0x7f86a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.362+0000 7f86cc200700 1 --2- 172.21.15.89:0/4159845463 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f86a406adb0 0x7f86a406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f86c40ad780 tx=0x7f86b801b040 comp rx=0 tx=0).stop 2024-04-02T08:53:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.362+0000 7f86cc200700 1 -- 172.21.15.89:0/4159845463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c415dfe0 msgr2=0x7f86c414f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.362+0000 7f86cc200700 1 --2- 172.21.15.89:0/4159845463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c415dfe0 0x7f86c414f9a0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f86bc00b9e0 tx=0x7f86bc00bda0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.362+0000 7f86cc200700 1 -- 172.21.15.89:0/4159845463 shutdown_connections 2024-04-02T08:53:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.362+0000 7f86cc200700 1 --2- 172.21.15.89:0/4159845463 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f86a406adb0 0x7f86a406d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.362+0000 7f86cc200700 1 --2- 172.21.15.89:0/4159845463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86c415dfe0 0x7f86c414f9a0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.362+0000 7f86cc200700 1 --2- 172.21.15.89:0/4159845463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86c40ce840 0x7f86c414c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.362+0000 7f86cc200700 1 --2- 172.21.15.89:0/4159845463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86c415a5c0 0x7f86c414f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.362+0000 7f86cc200700 1 -- 172.21.15.89:0/4159845463 >> 172.21.15.89:0/4159845463 conn(0x7f86c40b0d70 msgr2=0x7f86c40cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.362+0000 7f86cc200700 1 -- 172.21.15.89:0/4159845463 shutdown_connections 2024-04-02T08:53:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.362+0000 7f86cc200700 1 -- 172.21.15.89:0/4159845463 wait complete. 2024-04-02T08:53:16.379 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T08:53:16.379 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T08:53:16.461 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T08:53:16.548 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T08:53:16.549 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T08:53:16.549 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:53:16.593 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T08:53:16.594 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T08:53:16.594 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:53:16.639 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T08:53:16.639 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T08:53:16.640 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:53:16.683 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T08:53:16.683 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T08:53:16.684 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:53:16.728 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T08:53:16.728 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T08:53:16.729 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T08:53:16.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.922+0000 7fcb7be00700 1 Processor -- start 2024-04-02T08:53:16.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.922+0000 7fcb7be00700 1 -- start start 2024-04-02T08:53:16.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.923+0000 7fcb7be00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcb740ac580 0x7fcb740b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:16.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.923+0000 7fcb7be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb740b3000 0x7fcb7406dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:16.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.923+0000 7fcb7be00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb740b3890 con 0x7fcb7415d670 2024-04-02T08:53:16.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.923+0000 7fcb7be00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb74159f50 con 0x7fcb740ac580 2024-04-02T08:53:16.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.923+0000 7fcb7be00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb740b35b0 con 0x7fcb740b3000 2024-04-02T08:53:16.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.923+0000 7fcb78a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcb740ac580 0x7fcb740b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:16.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.923+0000 7fcb79e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb740b3000 0x7fcb7406dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:16.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.924+0000 7fcb79e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb740b3000 0x7fcb7406dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:58016/0 (socket says 172.21.15.89:58016) 2024-04-02T08:53:16.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.924+0000 7fcb78a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcb740ac580 0x7fcb740b2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:33740/0 (socket says 172.21.15.89:33740) 2024-04-02T08:53:16.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.924+0000 7fcb79e00700 1 -- 172.21.15.89:0/3283460688 learned_addr learned my addr 172.21.15.89:0/3283460688 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:16.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.923+0000 7fcb79400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fcb7415d670 0x7fcb7415da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:56882/0 (socket says 172.21.15.89:56882) 2024-04-02T08:53:16.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.924+0000 7fcb79e00700 1 -- 172.21.15.89:0/3283460688 >> v2:172.21.15.113:3300/0 conn(0x7fcb740ac580 msgr2=0x7fcb740b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:16.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.924+0000 7fcb79e00700 1 --2- 172.21.15.89:0/3283460688 >> v2:172.21.15.113:3300/0 conn(0x7fcb740ac580 0x7fcb740b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.924+0000 7fcb73e00700 1 -- 172.21.15.89:0/3283460688 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473460536 0 0) 0x7fcb740b3890 con 0x7fcb7415d670 2024-04-02T08:53:16.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.924+0000 7fcb79e00700 1 -- 172.21.15.89:0/3283460688 >> v1:172.21.15.113:6789/0 conn(0x7fcb7415d670 legacy=0x7fcb7415da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:16.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.924+0000 7fcb79e00700 1 -- 172.21.15.89:0/3283460688 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7406d400 con 0x7fcb740b3000 2024-04-02T08:53:16.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.924+0000 7fcb78a00700 1 --2- 172.21.15.89:0/3283460688 >> v2:172.21.15.113:3300/0 conn(0x7fcb740ac580 0x7fcb740b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:16.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.924+0000 7fcb79e00700 1 --2- 172.21.15.89:0/3283460688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb740b3000 0x7fcb7406dee0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009e50 tx=0x7fcb6c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f66c336a99c01f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:16.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.924+0000 7fcb73e00700 1 -- 172.21.15.89:0/3283460688 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c00e070 con 0x7fcb740b3000 2024-04-02T08:53:16.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.924+0000 7fcb73e00700 1 -- 172.21.15.89:0/3283460688 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6c00b910 con 0x7fcb740b3000 2024-04-02T08:53:16.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.924+0000 7fcb73e00700 1 -- 172.21.15.89:0/3283460688 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c00bcc0 con 0x7fcb740b3000 2024-04-02T08:53:16.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.925+0000 7fcb7be00700 1 -- 172.21.15.89:0/3283460688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb740b3000 msgr2=0x7fcb7406dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:16.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.925+0000 7fcb7be00700 1 --2- 172.21.15.89:0/3283460688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb740b3000 0x7fcb7406dee0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009e50 tx=0x7fcb6c00b330 comp rx=0 tx=0).stop 2024-04-02T08:53:16.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.925+0000 7fcb7be00700 1 -- 172.21.15.89:0/3283460688 shutdown_connections 2024-04-02T08:53:16.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.925+0000 7fcb7be00700 1 --2- 172.21.15.89:0/3283460688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb740b3000 0x7fcb7406dee0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.925+0000 7fcb7be00700 1 --2- 172.21.15.89:0/3283460688 >> v2:172.21.15.113:3300/0 conn(0x7fcb740ac580 0x7fcb740b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.925+0000 7fcb7be00700 1 -- 172.21.15.89:0/3283460688 >> 172.21.15.89:0/3283460688 conn(0x7fcb740cf190 msgr2=0x7fcb740cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:16.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.925+0000 7fcb7be00700 1 -- 172.21.15.89:0/3283460688 shutdown_connections 2024-04-02T08:53:16.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.925+0000 7fcb7be00700 1 -- 172.21.15.89:0/3283460688 wait complete. 2024-04-02T08:53:16.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.927+0000 7fcb7be00700 1 Processor -- start 2024-04-02T08:53:16.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.927+0000 7fcb7be00700 1 -- start start 2024-04-02T08:53:16.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.928+0000 7fcb7be00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb740ac580 0x7fcb7415cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:16.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.928+0000 7fcb7be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb740b3000 0x7fcb740ae5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:16.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.928+0000 7fcb7be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb7415d670 0x7fcb740aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:16.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.928+0000 7fcb7be00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb740caba0 con 0x7fcb740b3000 2024-04-02T08:53:16.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.928+0000 7fcb7be00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7415d020 con 0x7fcb740ac580 2024-04-02T08:53:16.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.928+0000 7fcb7be00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb740cb030 con 0x7fcb7415d670 2024-04-02T08:53:16.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.928+0000 7fcb78a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb740b3000 0x7fcb740ae5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:16.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.928+0000 7fcb79e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb7415d670 0x7fcb740aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:16.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.928+0000 7fcb79400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb740ac580 0x7fcb7415cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:16.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.928+0000 7fcb78a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb740b3000 0x7fcb740ae5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53530/0 (socket says 172.21.15.89:53530) 2024-04-02T08:53:16.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.928+0000 7fcb79e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb7415d670 0x7fcb740aeaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:58020/0 (socket says 172.21.15.89:58020) 2024-04-02T08:53:16.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.928+0000 7fcb78a00700 1 -- 172.21.15.89:0/2155432796 learned_addr learned my addr 172.21.15.89:0/2155432796 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:16.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.929+0000 7fcb79400700 1 -- 172.21.15.89:0/2155432796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb7415d670 msgr2=0x7fcb740aeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:16.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.929+0000 7fcb79400700 1 --2- 172.21.15.89:0/2155432796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb7415d670 0x7fcb740aeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.929+0000 7fcb79400700 1 -- 172.21.15.89:0/2155432796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb740b3000 msgr2=0x7fcb740ae5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:16.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.929+0000 7fcb79400700 1 --2- 172.21.15.89:0/2155432796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb740b3000 0x7fcb740ae5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:16.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.929+0000 7fcb79400700 1 -- 172.21.15.89:0/2155432796 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6000b740 con 0x7fcb740ac580 2024-04-02T08:53:16.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.929+0000 7fcb79e00700 1 --2- 172.21.15.89:0/2155432796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb7415d670 0x7fcb740aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:16.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.929+0000 7fcb79400700 1 --2- 172.21.15.89:0/2155432796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb740ac580 0x7fcb7415cae0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fcb6800c410 tx=0x7fcb6800c720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:16.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.929+0000 7fcb71600700 1 -- 172.21.15.89:0/2155432796 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb68004c80 con 0x7fcb740ac580 2024-04-02T08:53:16.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.929+0000 7fcb71600700 1 -- 172.21.15.89:0/2155432796 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb68004de0 con 0x7fcb740ac580 2024-04-02T08:53:16.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.929+0000 7fcb71600700 1 -- 172.21.15.89:0/2155432796 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb68007310 con 0x7fcb740ac580 2024-04-02T08:53:16.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.929+0000 7fcb7be00700 1 -- 172.21.15.89:0/2155432796 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c009b30 con 0x7fcb740ac580 2024-04-02T08:53:16.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.930+0000 7fcb7be00700 1 -- 172.21.15.89:0/2155432796 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb740ca270 con 0x7fcb740ac580 2024-04-02T08:53:16.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.930+0000 7fcb7be00700 1 -- 172.21.15.89:0/2155432796 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb740cae80 con 0x7fcb740ac580 2024-04-02T08:53:16.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.931+0000 7fcb71600700 1 -- 172.21.15.89:0/2155432796 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcb68007470 con 0x7fcb740ac580 2024-04-02T08:53:16.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.932+0000 7fcb71600700 1 --2- 172.21.15.89:0/2155432796 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb5406adb0 0x7fcb5406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:16.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.932+0000 7fcb71600700 1 -- 172.21.15.89:0/2155432796 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5965+0+0 (secure 0 0 0) 0x7fcb68088ff0 con 0x7fcb740ac580 2024-04-02T08:53:16.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.932+0000 7fcb78a00700 1 --2- 172.21.15.89:0/2155432796 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb5406adb0 0x7fcb5406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:16.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.933+0000 7fcb78a00700 1 --2- 172.21.15.89:0/2155432796 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb5406adb0 0x7fcb5406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000b240 tx=0x7fcb6000b420 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:16.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:16.937+0000 7fcb71600700 1 -- 172.21.15.89:0/2155432796 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcb68054030 con 0x7fcb740ac580 2024-04-02T08:53:17.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.186+0000 7fcb7be00700 1 -- 172.21.15.89:0/2155432796 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb74066ff0 con 0x7fcb740ac580 2024-04-02T08:53:17.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.187+0000 7fcb71600700 1 -- 172.21.15.89:0/2155432796 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3529 (secure 0 0 0) 0x7fcb68057650 con 0x7fcb740ac580 2024-04-02T08:53:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:17.187 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":83,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8881,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/547244866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":547244866},{"type":"v1","addr":"172.21.15.113:6834","nonce":547244866}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":8884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2822200553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2822200553},{"type":"v1","addr":"172.21.15.113:6835","nonce":2822200553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/342426898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":342426898},{"type":"v1","addr":"172.21.15.89:6835","nonce":342426898}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9942,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3127269702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3127269702},{"type":"v1","addr":"172.21.15.89:6837","nonce":3127269702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83}],"filesystems":[]} 2024-04-02T08:53:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.190+0000 7fcb7be00700 1 -- 172.21.15.89:0/2155432796 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb5406adb0 msgr2=0x7fcb5406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.190+0000 7fcb7be00700 1 --2- 172.21.15.89:0/2155432796 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb5406adb0 0x7fcb5406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000b240 tx=0x7fcb6000b420 comp rx=0 tx=0).stop 2024-04-02T08:53:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.190+0000 7fcb7be00700 1 -- 172.21.15.89:0/2155432796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb740ac580 msgr2=0x7fcb7415cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.190+0000 7fcb7be00700 1 --2- 172.21.15.89:0/2155432796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb740ac580 0x7fcb7415cae0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fcb6800c410 tx=0x7fcb6800c720 comp rx=0 tx=0).stop 2024-04-02T08:53:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.190+0000 7fcb7be00700 1 -- 172.21.15.89:0/2155432796 shutdown_connections 2024-04-02T08:53:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.190+0000 7fcb7be00700 1 --2- 172.21.15.89:0/2155432796 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcb5406adb0 0x7fcb5406d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.190+0000 7fcb7be00700 1 --2- 172.21.15.89:0/2155432796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb740ac580 0x7fcb7415cae0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.190+0000 7fcb7be00700 1 --2- 172.21.15.89:0/2155432796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb7415d670 0x7fcb740aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.190+0000 7fcb7be00700 1 --2- 172.21.15.89:0/2155432796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb740b3000 0x7fcb740ae5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.190+0000 7fcb7be00700 1 -- 172.21.15.89:0/2155432796 >> 172.21.15.89:0/2155432796 conn(0x7fcb740cf190 msgr2=0x7fcb74071ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.190+0000 7fcb7be00700 1 -- 172.21.15.89:0/2155432796 shutdown_connections 2024-04-02T08:53:17.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.190+0000 7fcb7be00700 1 -- 172.21.15.89:0/2155432796 wait complete. 2024-04-02T08:53:17.190 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 83 2024-04-02T08:53:17.207 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T08:53:17.208 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T08:53:17.208 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T08:53:17.208 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T08:53:17.208 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T08:53:17.208 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T08:53:17.208 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T08:53:17.208 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T08:53:17.209 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T08:53:17.209 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T08:53:17.209 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T08:53:17.209 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T08:53:17.209 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T08:53:17.209 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T08:53:17.209 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T08:53:17.210 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T08:53:17.222 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T08:53:17.223 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T08:53:17.224 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T08:53:17.224 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T08:53:17.225 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T08:53:17.225 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T08:53:17.226 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T08:53:17.226 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T08:53:17.268 INFO:tasks.ceph.mds.a:Started 2024-04-02T08:53:17.269 DEBUG:teuthology.parallel:result is None 2024-04-02T08:53:17.269 INFO:tasks.ceph.mds.b:Started 2024-04-02T08:53:17.269 DEBUG:teuthology.parallel:result is None 2024-04-02T08:53:17.271 INFO:tasks.ceph.mds.d:Started 2024-04-02T08:53:17.272 INFO:tasks.ceph.mds.c:Started 2024-04-02T08:53:17.272 DEBUG:teuthology.parallel:result is None 2024-04-02T08:53:17.272 DEBUG:teuthology.parallel:result is None 2024-04-02T08:53:17.272 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T08:53:17.412 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T08:53:17.412 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T08:53:17.413 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T08:53:17.413 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T08:53:17.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.478+0000 7f8c12000700 1 Processor -- start 2024-04-02T08:53:17.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.478+0000 7f8c12000700 1 -- start start 2024-04-02T08:53:17.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.480+0000 7f8c12000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c0c160d70 0x7f8c0c0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:17.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.480+0000 7f8c12000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8c0c15ef70 0x7f8c0c15e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.480+0000 7f8c12000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8c0c06a790 0x7f8c0c06ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.480+0000 7f8c12000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c0c06b180 con 0x7f8c0c06a790 2024-04-02T08:53:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.480+0000 7f8c12000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c0c0b4cd0 con 0x7f8c0c15ef70 2024-04-02T08:53:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.480+0000 7f8c12000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c161150 con 0x7f8c0c160d70 2024-04-02T08:53:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.480+0000 7f8c0be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8c0c06a790 0x7f8c0c06ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.481+0000 7f8c0be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8c0c06a790 0x7f8c0c06ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53572/0 (socket says 172.21.15.89:53572) 2024-04-02T08:53:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.481+0000 7f8c0be00700 1 -- 172.21.15.89:0/2441885212 learned_addr learned my addr 172.21.15.89:0/2441885212 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.481+0000 7f8c0b400700 1 --2- 172.21.15.89:0/2441885212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c0c160d70 0x7f8c0c0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.481+0000 7f8c0aa00700 1 --2- 172.21.15.89:0/2441885212 >> v2:172.21.15.113:3300/0 conn(0x7f8c0c15ef70 0x7f8c0c15e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.481+0000 7f8c0be00700 1 -- 172.21.15.89:0/2441885212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c0c160d70 msgr2=0x7f8c0c0b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.481+0000 7f8c0be00700 1 --2- 172.21.15.89:0/2441885212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c0c160d70 0x7f8c0c0b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.481+0000 7f8c0be00700 1 -- 172.21.15.89:0/2441885212 >> v2:172.21.15.113:3300/0 conn(0x7f8c0c15ef70 msgr2=0x7f8c0c15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.481+0000 7f8c0be00700 1 --2- 172.21.15.89:0/2441885212 >> v2:172.21.15.113:3300/0 conn(0x7f8c0c15ef70 0x7f8c0c15e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:17.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.481+0000 7f8c0be00700 1 -- 172.21.15.89:0/2441885212 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c06dba0 con 0x7f8c0c06a790 2024-04-02T08:53:17.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.481+0000 7f8c0be00700 1 --2- 172.21.15.89:0/2441885212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c0c06a790 0x7f8c0c06ac40 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0713b0 tx=0x7f8c0400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44eebff719402fb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:17.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.481+0000 7f8c0a000700 1 -- 172.21.15.89:0/2441885212 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0400e070 con 0x7f8c0c06a790 2024-04-02T08:53:17.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.482+0000 7f8c0a000700 1 -- 172.21.15.89:0/2441885212 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0400b910 con 0x7f8c0c06a790 2024-04-02T08:53:17.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.482+0000 7f8c0a000700 1 -- 172.21.15.89:0/2441885212 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0400bcc0 con 0x7f8c0c06a790 2024-04-02T08:53:17.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.482+0000 7f8c12000700 1 -- 172.21.15.89:0/2441885212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c0c06a790 msgr2=0x7f8c0c06ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:17.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.482+0000 7f8c12000700 1 --2- 172.21.15.89:0/2441885212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c0c06a790 0x7f8c0c06ac40 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0713b0 tx=0x7f8c0400b330 comp rx=0 tx=0).stop 2024-04-02T08:53:17.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.482+0000 7f8c12000700 1 -- 172.21.15.89:0/2441885212 shutdown_connections 2024-04-02T08:53:17.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.482+0000 7f8c12000700 1 --2- 172.21.15.89:0/2441885212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c0c06a790 0x7f8c0c06ac40 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:17.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.482+0000 7f8c12000700 1 --2- 172.21.15.89:0/2441885212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c0c160d70 0x7f8c0c0b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:17.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.482+0000 7f8c12000700 1 --2- 172.21.15.89:0/2441885212 >> v2:172.21.15.113:3300/0 conn(0x7f8c0c15ef70 0x7f8c0c15e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:17.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.482+0000 7f8c12000700 1 -- 172.21.15.89:0/2441885212 >> 172.21.15.89:0/2441885212 conn(0x7f8c0c0b5720 msgr2=0x7f8c0c0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:17.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.482+0000 7f8c12000700 1 -- 172.21.15.89:0/2441885212 shutdown_connections 2024-04-02T08:53:17.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.482+0000 7f8c12000700 1 -- 172.21.15.89:0/2441885212 wait complete. 2024-04-02T08:53:17.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.484+0000 7f8c12000700 1 Processor -- start 2024-04-02T08:53:17.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.484+0000 7f8c12000700 1 -- start start 2024-04-02T08:53:17.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.485+0000 7f8c12000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c0c06a790 0x7f8c0c134900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:17.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.485+0000 7f8c12000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c0c15ef70 0x7f8c0c15d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:17.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.485+0000 7f8c12000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c0c160d70 0x7f8c0c1f64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:17.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.485+0000 7f8c12000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c1f2da0 con 0x7f8c0c15ef70 2024-04-02T08:53:17.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.485+0000 7f8c12000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c131410 con 0x7f8c0c160d70 2024-04-02T08:53:17.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.485+0000 7f8c12000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c1316f0 con 0x7f8c0c06a790 2024-04-02T08:53:17.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.485+0000 7f8c0aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c0c15ef70 0x7f8c0c15d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:17.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.485+0000 7f8c0aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c0c15ef70 0x7f8c0c15d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53576/0 (socket says 172.21.15.89:53576) 2024-04-02T08:53:17.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.485+0000 7f8c0aa00700 1 -- 172.21.15.89:0/1858295720 learned_addr learned my addr 172.21.15.89:0/1858295720 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:17.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.485+0000 7f8c0b400700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c0c06a790 0x7f8c0c134900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:17.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.485+0000 7f8c0be00700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c0c160d70 0x7f8c0c1f64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:17.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.486+0000 7f8c0aa00700 1 -- 172.21.15.89:0/1858295720 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c0c06a790 msgr2=0x7f8c0c134900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:17.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.486+0000 7f8c0aa00700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c0c06a790 0x7f8c0c134900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:17.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.486+0000 7f8c0aa00700 1 -- 172.21.15.89:0/1858295720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c0c160d70 msgr2=0x7f8c0c1f64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:17.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.486+0000 7f8c0aa00700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c0c160d70 0x7f8c0c1f64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:17.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.486+0000 7f8c0aa00700 1 -- 172.21.15.89:0/1858295720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c04009b30 con 0x7f8c0c15ef70 2024-04-02T08:53:17.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.486+0000 7f8c0b400700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c0c06a790 0x7f8c0c134900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:17.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.486+0000 7f8c0aa00700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c0c15ef70 0x7f8c0c15d820 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f8bf800e900 tx=0x7f8bf800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:17.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.486+0000 7f8c03400700 1 -- 172.21.15.89:0/1858295720 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8013680 con 0x7f8c0c15ef70 2024-04-02T08:53:17.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.486+0000 7f8c03400700 1 -- 172.21.15.89:0/1858295720 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf80137e0 con 0x7f8c0c15ef70 2024-04-02T08:53:17.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.486+0000 7f8c03400700 1 -- 172.21.15.89:0/1858295720 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8013b70 con 0x7f8c0c15ef70 2024-04-02T08:53:17.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.486+0000 7f8c12000700 1 -- 172.21.15.89:0/1858295720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0c1319d0 con 0x7f8c0c15ef70 2024-04-02T08:53:17.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.486+0000 7f8c12000700 1 -- 172.21.15.89:0/1858295720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0c131f90 con 0x7f8c0c15ef70 2024-04-02T08:53:17.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.487+0000 7f8c12000700 1 -- 172.21.15.89:0/1858295720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0c132150 con 0x7f8c0c15ef70 2024-04-02T08:53:17.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.488+0000 7f8c03400700 1 -- 172.21.15.89:0/1858295720 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8bf8013cd0 con 0x7f8c0c15ef70 2024-04-02T08:53:17.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.489+0000 7f8c03400700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8bec06adb0 0x7f8bec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:17.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.489+0000 7f8c03400700 1 -- 172.21.15.89:0/1858295720 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5965+0+0 (secure 0 0 0) 0x7f8bf8089520 con 0x7f8c0c15ef70 2024-04-02T08:53:17.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.489+0000 7f8c0b400700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8bec06adb0 0x7f8bec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:17.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.490+0000 7f8c0b400700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8bec06adb0 0x7f8bec06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc009fd0 tx=0x7f8bfc009210 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:17.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.493+0000 7f8c03400700 1 -- 172.21.15.89:0/1858295720 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8bf8054450 con 0x7f8c0c15ef70 2024-04-02T08:53:17.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:17.709+0000 7f8c12000700 1 -- 172.21.15.89:0/1858295720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8c0c06f160 con 0x7f8c0c15ef70 2024-04-02T08:53:18.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.450+0000 7f8c03400700 1 -- 172.21.15.89:0/1858295720 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v182) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8bf805ce60 con 0x7f8c0c15ef70 2024-04-02T08:53:18.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.453+0000 7f8c12000700 1 -- 172.21.15.89:0/1858295720 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8bec06adb0 msgr2=0x7f8bec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:18.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.453+0000 7f8c12000700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8bec06adb0 0x7f8bec06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc009fd0 tx=0x7f8bfc009210 comp rx=0 tx=0).stop 2024-04-02T08:53:18.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.453+0000 7f8c12000700 1 -- 172.21.15.89:0/1858295720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c0c15ef70 msgr2=0x7f8c0c15d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:18.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.453+0000 7f8c12000700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c0c15ef70 0x7f8c0c15d820 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f8bf800e900 tx=0x7f8bf800ec10 comp rx=0 tx=0).stop 2024-04-02T08:53:18.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.453+0000 7f8c12000700 1 -- 172.21.15.89:0/1858295720 shutdown_connections 2024-04-02T08:53:18.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.453+0000 7f8c12000700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f8bec06adb0 0x7f8bec06d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:18.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.453+0000 7f8c12000700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c0c160d70 0x7f8c0c1f64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:18.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.453+0000 7f8c12000700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8c0c06a790 0x7f8c0c134900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:18.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.453+0000 7f8c12000700 1 --2- 172.21.15.89:0/1858295720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c0c15ef70 0x7f8c0c15d820 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:18.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.453+0000 7f8c12000700 1 -- 172.21.15.89:0/1858295720 >> 172.21.15.89:0/1858295720 conn(0x7f8c0c0b5720 msgr2=0x7f8c0c06df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:18.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.454+0000 7f8c12000700 1 -- 172.21.15.89:0/1858295720 shutdown_connections 2024-04-02T08:53:18.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.454+0000 7f8c12000700 1 -- 172.21.15.89:0/1858295720 wait complete. 2024-04-02T08:53:18.454 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T08:53:18.472 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T08:53:18.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.666+0000 7fd772c00700 1 Processor -- start 2024-04-02T08:53:18.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.666+0000 7fd772c00700 1 -- start start 2024-04-02T08:53:18.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.667+0000 7fd772c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd76c15d640 0x7fd76c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:18.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.667+0000 7fd772c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76c0d30d0 con 0x7fd76c0ceec0 2024-04-02T08:53:18.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.667+0000 7fd772c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76c0b6840 con 0x7fd76c1619d0 2024-04-02T08:53:18.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.667+0000 7fd772c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c0d2880 con 0x7fd76c15d640 2024-04-02T08:53:18.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.667+0000 7fd76b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd76c0ceec0 0x7fd76c1612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58178/0 (socket says 172.21.15.89:58178) 2024-04-02T08:53:18.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.667+0000 7fd76b400700 1 -- 172.21.15.89:0/651065856 learned_addr learned my addr 172.21.15.89:0/651065856 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:18.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76be00700 1 --2- 172.21.15.89:0/651065856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd76c15d640 0x7fd76c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:18.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 -- 172.21.15.89:0/651065856 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560517160 0 0) 0x7fd76c0d30d0 con 0x7fd76c0ceec0 2024-04-02T08:53:18.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 -- 172.21.15.89:0/651065856 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7500035b0 con 0x7fd76c0ceec0 2024-04-02T08:53:18.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 -- 172.21.15.89:0/651065856 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1151571506 0 0) 0x7fd7500035b0 con 0x7fd76c0ceec0 2024-04-02T08:53:18.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 -- 172.21.15.89:0/651065856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd76c15d640 msgr2=0x7fd76c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:18.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 --2- 172.21.15.89:0/651065856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd76c15d640 0x7fd76c0d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:18.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 -- 172.21.15.89:0/651065856 >> v1:172.21.15.113:6789/0 conn(0x7fd76c1619d0 legacy=0x7fd76c0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:18.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 -- 172.21.15.89:0/651065856 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c0d2b30 con 0x7fd76c0ceec0 2024-04-02T08:53:18.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 -- 172.21.15.89:0/651065856 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fd758001670 con 0x7fd76c0ceec0 2024-04-02T08:53:18.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 -- 172.21.15.89:0/651065856 >> v1:172.21.15.89:6789/0 conn(0x7fd76c0ceec0 legacy=0x7fd76c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:18.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 --2- 172.21.15.89:0/651065856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd750003a30 0x7fd750003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:18.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 --2- 172.21.15.89:0/651065856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7500043e0 0x7fd7500048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:18.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 --2- 172.21.15.89:0/651065856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd76c15d640 0x7fd7500051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:18.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 -- 172.21.15.89:0/651065856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c0d2880 con 0x7fd750003a30 2024-04-02T08:53:18.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 -- 172.21.15.89:0/651065856 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75000bac0 con 0x7fd7500043e0 2024-04-02T08:53:18.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76aa00700 1 -- 172.21.15.89:0/651065856 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75000bc80 con 0x7fd76c15d640 2024-04-02T08:53:18.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.668+0000 7fd76be00700 1 --2- 172.21.15.89:0/651065856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd750003a30 0x7fd750003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:18.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.669+0000 7fd76b400700 1 --2- 172.21.15.89:0/651065856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7500043e0 0x7fd7500048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:18.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.669+0000 7fd770c00700 1 --2- 172.21.15.89:0/651065856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd76c15d640 0x7fd7500051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:18.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.669+0000 7fd76be00700 1 -- 172.21.15.89:0/651065856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd76c15d640 msgr2=0x7fd7500051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:18.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.669+0000 7fd76be00700 1 --2- 172.21.15.89:0/651065856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd76c15d640 0x7fd7500051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:18.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.669+0000 7fd76be00700 1 -- 172.21.15.89:0/651065856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7500043e0 msgr2=0x7fd7500048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:18.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.669+0000 7fd76be00700 1 --2- 172.21.15.89:0/651065856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7500043e0 0x7fd7500048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:18.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.669+0000 7fd76be00700 1 -- 172.21.15.89:0/651065856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7500035b0 con 0x7fd750003a30 2024-04-02T08:53:18.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.669+0000 7fd76be00700 1 --2- 172.21.15.89:0/651065856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd750003a30 0x7fd750003e10 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009b60 tx=0x7fd750005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2069a3af6d4c8221 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:18.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.669+0000 7fd76aa00700 1 -- 172.21.15.89:0/651065856 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c00e6c0 con 0x7fd750003a30 2024-04-02T08:53:18.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.669+0000 7fd76aa00700 1 -- 172.21.15.89:0/651065856 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75c00e820 con 0x7fd750003a30 2024-04-02T08:53:18.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.669+0000 7fd76aa00700 1 -- 172.21.15.89:0/651065856 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c00eb90 con 0x7fd750003a30 2024-04-02T08:53:18.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.670+0000 7fd772c00700 1 -- 172.21.15.89:0/651065856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd750003a30 msgr2=0x7fd750003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:18.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.670+0000 7fd772c00700 1 --2- 172.21.15.89:0/651065856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd750003a30 0x7fd750003e10 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009b60 tx=0x7fd750005880 comp rx=0 tx=0).stop 2024-04-02T08:53:18.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.670+0000 7fd770c00700 1 -- 172.21.15.89:0/651065856 reap_dead start 2024-04-02T08:53:18.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.670+0000 7fd772c00700 1 -- 172.21.15.89:0/651065856 shutdown_connections 2024-04-02T08:53:18.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.670+0000 7fd772c00700 1 -- 172.21.15.89:0/651065856 >> 172.21.15.89:0/651065856 conn(0x7fd76c08aff0 msgr2=0x7fd76c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:18.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.670+0000 7fd772c00700 1 -- 172.21.15.89:0/651065856 shutdown_connections 2024-04-02T08:53:18.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.670+0000 7fd772c00700 1 -- 172.21.15.89:0/651065856 wait complete. 2024-04-02T08:53:18.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.672+0000 7fd772c00700 1 Processor -- start 2024-04-02T08:53:18.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.672+0000 7fd772c00700 1 -- start start 2024-04-02T08:53:18.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd772c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd750003a30 0x7fd76c15bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:18.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd772c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd76c157f20 0x7fd76c15c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:18.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd772c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd76c1f2920 0x7fd76c1f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:18.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd772c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c0c74c0 con 0x7fd750003a30 2024-04-02T08:53:18.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd772c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c1f3e90 con 0x7fd76c157f20 2024-04-02T08:53:18.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd772c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c1f38d0 con 0x7fd76c1f2920 2024-04-02T08:53:18.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd76be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd750003a30 0x7fd76c15bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:18.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd76be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd750003a30 0x7fd76c15bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53602/0 (socket says 172.21.15.89:53602) 2024-04-02T08:53:18.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd76be00700 1 -- 172.21.15.89:0/596798075 learned_addr learned my addr 172.21.15.89:0/596798075 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:18.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd76b400700 1 --2- 172.21.15.89:0/596798075 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd76c157f20 0x7fd76c15c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:18.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd770c00700 1 --2- 172.21.15.89:0/596798075 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd76c1f2920 0x7fd76c1f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:18.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd76be00700 1 -- 172.21.15.89:0/596798075 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd76c1f2920 msgr2=0x7fd76c1f2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:18.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd76be00700 1 --2- 172.21.15.89:0/596798075 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd76c1f2920 0x7fd76c1f2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:18.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd76be00700 1 -- 172.21.15.89:0/596798075 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd76c157f20 msgr2=0x7fd76c15c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:18.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd76be00700 1 --2- 172.21.15.89:0/596798075 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd76c157f20 0x7fd76c15c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:18.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.673+0000 7fd76be00700 1 -- 172.21.15.89:0/596798075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c1f3bb0 con 0x7fd750003a30 2024-04-02T08:53:18.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.674+0000 7fd76be00700 1 --2- 172.21.15.89:0/596798075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd750003a30 0x7fd76c15bcb0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fd750005810 tx=0x7fd75c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:18.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.674+0000 7fd763e00700 1 -- 172.21.15.89:0/596798075 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c021070 con 0x7fd750003a30 2024-04-02T08:53:18.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.674+0000 7fd763e00700 1 -- 172.21.15.89:0/596798075 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75c0045d0 con 0x7fd750003a30 2024-04-02T08:53:18.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.674+0000 7fd772c00700 1 -- 172.21.15.89:0/596798075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76c0c70e0 con 0x7fd750003a30 2024-04-02T08:53:18.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.674+0000 7fd763e00700 1 -- 172.21.15.89:0/596798075 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c00e6c0 con 0x7fd750003a30 2024-04-02T08:53:18.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.674+0000 7fd772c00700 1 -- 172.21.15.89:0/596798075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76c1f32f0 con 0x7fd750003a30 2024-04-02T08:53:18.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.675+0000 7fd772c00700 1 -- 172.21.15.89:0/596798075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76c067720 con 0x7fd750003a30 2024-04-02T08:53:18.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.676+0000 7fd763e00700 1 -- 172.21.15.89:0/596798075 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd75c025070 con 0x7fd750003a30 2024-04-02T08:53:18.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.677+0000 7fd763e00700 1 --2- 172.21.15.89:0/596798075 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd74c06adb0 0x7fd74c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:18.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.677+0000 7fd763e00700 1 -- 172.21.15.89:0/596798075 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5277+0+0 (secure 0 0 0) 0x7fd75c08a480 con 0x7fd750003a30 2024-04-02T08:53:18.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.677+0000 7fd76b400700 1 --2- 172.21.15.89:0/596798075 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd74c06adb0 0x7fd74c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:18.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.677+0000 7fd76b400700 1 --2- 172.21.15.89:0/596798075 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd74c06adb0 0x7fd74c06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd76c158340 tx=0x7fd758009040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:18.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.681+0000 7fd763e00700 1 -- 172.21.15.89:0/596798075 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd75c0558b0 con 0x7fd750003a30 2024-04-02T08:53:18.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.964+0000 7fd772c00700 1 -- 172.21.15.89:0/596798075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd76c0b5c70 con 0x7fd750003a30 2024-04-02T08:53:18.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.965+0000 7fd763e00700 1 -- 172.21.15.89:0/596798075 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v45) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd75c058ed0 con 0x7fd750003a30 2024-04-02T08:53:18.964 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:18.964 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T08:53:18.964 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T08:53:18.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.964 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T08:53:18.964 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T08:53:18.964 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.964 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:53:18.964 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T08:53:18.964 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T08:53:18.965 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.965 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T08:53:18.965 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T08:53:18.965 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.965 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:53:18.965 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T08:53:18.965 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T08:53:18.965 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.966 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T08:53:18.966 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T08:53:18.966 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.966 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:53:18.966 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T08:53:18.966 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T08:53:18.966 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.966 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.966 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T08:53:18.966 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T08:53:18.966 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.968 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T08:53:18.968 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T08:53:18.968 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.968 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:53:18.968 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:53:18.968 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.968 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.968 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T08:53:18.968 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T08:53:18.968 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.968 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.969 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.970 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.971 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T08:53:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T08:53:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T08:53:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T08:53:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:53:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2024-04-02T08:53:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T08:53:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:53:18.973 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T08:53:18.974 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T08:53:18.974 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T08:53:18.974 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.974 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T08:53:18.974 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T08:53:18.974 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.974 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T08:53:18.974 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T08:53:18.974 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T08:53:18.974 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T08:53:18.975 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T08:53:18.976 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.976 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T08:53:18.976 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T08:53:18.976 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.976 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T08:53:18.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:18.976 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T08:53:18.976 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T08:53:18.976 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T08:53:18.976 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T08:53:18.976 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T08:53:18.976 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T08:53:18.976 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.977 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:18.977 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T08:53:18.977 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T08:53:18.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.967+0000 7fd772c00700 1 -- 172.21.15.89:0/596798075 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd74c06adb0 msgr2=0x7fd74c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:18.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.967+0000 7fd772c00700 1 --2- 172.21.15.89:0/596798075 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd74c06adb0 0x7fd74c06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd76c158340 tx=0x7fd758009040 comp rx=0 tx=0).stop 2024-04-02T08:53:18.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.967+0000 7fd772c00700 1 -- 172.21.15.89:0/596798075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd750003a30 msgr2=0x7fd76c15bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:18.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.967+0000 7fd772c00700 1 --2- 172.21.15.89:0/596798075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd750003a30 0x7fd76c15bcb0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fd750005810 tx=0x7fd75c0047c0 comp rx=0 tx=0).stop 2024-04-02T08:53:18.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.967+0000 7fd772c00700 1 -- 172.21.15.89:0/596798075 shutdown_connections 2024-04-02T08:53:18.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.967+0000 7fd772c00700 1 --2- 172.21.15.89:0/596798075 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd74c06adb0 0x7fd74c06d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:18.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.967+0000 7fd772c00700 1 --2- 172.21.15.89:0/596798075 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd76c1f2920 0x7fd76c1f2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:18.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.967+0000 7fd772c00700 1 --2- 172.21.15.89:0/596798075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd750003a30 0x7fd76c15bcb0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:18.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.967+0000 7fd772c00700 1 --2- 172.21.15.89:0/596798075 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd76c157f20 0x7fd76c15c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:18.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.967+0000 7fd772c00700 1 -- 172.21.15.89:0/596798075 >> 172.21.15.89:0/596798075 conn(0x7fd76c08aff0 msgr2=0x7fd76c0b7460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:18.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.968+0000 7fd772c00700 1 -- 172.21.15.89:0/596798075 shutdown_connections 2024-04-02T08:53:18.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:18.968+0000 7fd772c00700 1 -- 172.21.15.89:0/596798075 wait complete. 2024-04-02T08:53:18.985 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:53:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.179+0000 7f274dc00700 1 Processor -- start 2024-04-02T08:53:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.179+0000 7f274dc00700 1 -- start start 2024-04-02T08:53:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.180+0000 7f274dc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2748160f10 0x7f27480d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.180+0000 7f274dc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f27480cf230 0x7f27480d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.180+0000 7f274dc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27480ceda0 con 0x7f2748161590 2024-04-02T08:53:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.180+0000 7f274dc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27480d23d0 con 0x7f27480cf230 2024-04-02T08:53:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.180+0000 7f274dc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27481612f0 con 0x7f2748160f10 2024-04-02T08:53:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.181+0000 7f2746000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f27480cf230 0x7f27480d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.181+0000 7f2747400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2748161590 0x7f27480b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46330/0 (socket says 172.21.15.89:46330) 2024-04-02T08:53:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.181+0000 7f2747400700 1 -- 172.21.15.89:0/2284793496 learned_addr learned my addr 172.21.15.89:0/2284793496 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.181+0000 7f2746a00700 1 --2- 172.21.15.89:0/2284793496 >> v2:172.21.15.113:3300/0 conn(0x7f2748160f10 0x7f27480d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.181+0000 7f2746000700 1 -- 172.21.15.89:0/2284793496 >> v2:172.21.15.113:3300/0 conn(0x7f2748160f10 msgr2=0x7f27480d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.181+0000 7f2746000700 1 --2- 172.21.15.89:0/2284793496 >> v2:172.21.15.113:3300/0 conn(0x7f2748160f10 0x7f27480d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.181+0000 7f2746000700 1 -- 172.21.15.89:0/2284793496 >> v1:172.21.15.89:6789/0 conn(0x7f2748161590 legacy=0x7f27480b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.181+0000 7f2746000700 1 -- 172.21.15.89:0/2284793496 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27480ce470 con 0x7f27480cf230 2024-04-02T08:53:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.181+0000 7f2746a00700 1 --2- 172.21.15.89:0/2284793496 >> v2:172.21.15.113:3300/0 conn(0x7f2748160f10 0x7f27480d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.181+0000 7f2746000700 1 --2- 172.21.15.89:0/2284793496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27480cf230 0x7f27480d1e90 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f27480d1080 tx=0x7f2734009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55da6bdf9657ebc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.182+0000 7f2745600700 1 -- 172.21.15.89:0/2284793496 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2734011830 con 0x7f27480cf230 2024-04-02T08:53:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.182+0000 7f2745600700 1 -- 172.21.15.89:0/2284793496 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2734011990 con 0x7f27480cf230 2024-04-02T08:53:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.182+0000 7f2745600700 1 -- 172.21.15.89:0/2284793496 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2734011d20 con 0x7f27480cf230 2024-04-02T08:53:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.182+0000 7f274dc00700 1 -- 172.21.15.89:0/2284793496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27480cf230 msgr2=0x7f27480d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.182+0000 7f274dc00700 1 --2- 172.21.15.89:0/2284793496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27480cf230 0x7f27480d1e90 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f27480d1080 tx=0x7f2734009b20 comp rx=0 tx=0).stop 2024-04-02T08:53:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.182+0000 7f274dc00700 1 -- 172.21.15.89:0/2284793496 shutdown_connections 2024-04-02T08:53:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.182+0000 7f274dc00700 1 --2- 172.21.15.89:0/2284793496 >> v2:172.21.15.113:3300/0 conn(0x7f2748160f10 0x7f27480d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.182+0000 7f274dc00700 1 --2- 172.21.15.89:0/2284793496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27480cf230 0x7f27480d1e90 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.182+0000 7f274dc00700 1 -- 172.21.15.89:0/2284793496 >> 172.21.15.89:0/2284793496 conn(0x7f274808aff0 msgr2=0x7f27480d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.182+0000 7f274dc00700 1 -- 172.21.15.89:0/2284793496 shutdown_connections 2024-04-02T08:53:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.182+0000 7f274dc00700 1 -- 172.21.15.89:0/2284793496 wait complete. 2024-04-02T08:53:19.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.184+0000 7f274dc00700 1 Processor -- start 2024-04-02T08:53:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.184+0000 7f274dc00700 1 -- start start 2024-04-02T08:53:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.185+0000 7f274dc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27480cf230 0x7f2748156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.185+0000 7f274dc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2748160f10 0x7f27481484c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.185+0000 7f274dc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2748161590 0x7f27481583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.185+0000 7f274dc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2748159200 con 0x7f27480cf230 2024-04-02T08:53:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.185+0000 7f274dc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27481594e0 con 0x7f2748160f10 2024-04-02T08:53:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.185+0000 7f274dc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27481597c0 con 0x7f2748161590 2024-04-02T08:53:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.185+0000 7f2746a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27480cf230 0x7f2748156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.185+0000 7f2746a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27480cf230 0x7f2748156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53926/0 (socket says 172.21.15.89:53926) 2024-04-02T08:53:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.185+0000 7f2746a00700 1 -- 172.21.15.89:0/130507940 learned_addr learned my addr 172.21.15.89:0/130507940 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.185+0000 7f2746a00700 1 -- 172.21.15.89:0/130507940 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2748161590 msgr2=0x7f27481583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.185+0000 7f2747400700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2748161590 0x7f27481583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.186+0000 7f2746000700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2748160f10 0x7f27481484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:19.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.186+0000 7f2746a00700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2748161590 0x7f27481583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.186+0000 7f2746a00700 1 -- 172.21.15.89:0/130507940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2748160f10 msgr2=0x7f27481484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:19.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.186+0000 7f2746a00700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2748160f10 0x7f27481484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.186+0000 7f2746a00700 1 -- 172.21.15.89:0/130507940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27340094a0 con 0x7f27480cf230 2024-04-02T08:53:19.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.186+0000 7f2747400700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2748161590 0x7f27481583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:53:19.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.186+0000 7f2746000700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2748160f10 0x7f27481484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:53:19.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.186+0000 7f2746a00700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27480cf230 0x7f2748156940 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f273800e760 tx=0x7f273800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:19.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.186+0000 7f273ea00700 1 -- 172.21.15.89:0/130507940 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738013610 con 0x7f27480cf230 2024-04-02T08:53:19.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.186+0000 7f273ea00700 1 -- 172.21.15.89:0/130507940 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2738013770 con 0x7f27480cf230 2024-04-02T08:53:19.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.186+0000 7f273ea00700 1 -- 172.21.15.89:0/130507940 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738013b20 con 0x7f27480cf230 2024-04-02T08:53:19.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.186+0000 7f274dc00700 1 -- 172.21.15.89:0/130507940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2748159aa0 con 0x7f27480cf230 2024-04-02T08:53:19.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.186+0000 7f274dc00700 1 -- 172.21.15.89:0/130507940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2748148a00 con 0x7f27480cf230 2024-04-02T08:53:19.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.187+0000 7f274dc00700 1 -- 172.21.15.89:0/130507940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2748067720 con 0x7f27480cf230 2024-04-02T08:53:19.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.188+0000 7f273ea00700 1 -- 172.21.15.89:0/130507940 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2738013c80 con 0x7f27480cf230 2024-04-02T08:53:19.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.189+0000 7f273ea00700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f272806adb0 0x7f272806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:19.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.189+0000 7f273ea00700 1 -- 172.21.15.89:0/130507940 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f2738089070 con 0x7f27480cf230 2024-04-02T08:53:19.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.189+0000 7f2746000700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f272806adb0 0x7f272806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:19.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.190+0000 7f2746000700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f272806adb0 0x7f272806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f27480cfad0 tx=0x7f2734008030 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:19.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.194+0000 7f273ea00700 1 -- 172.21.15.89:0/130507940 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f27380541d0 con 0x7f27480cf230 2024-04-02T08:53:19.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.396+0000 7f274dc00700 1 -- 172.21.15.89:0/130507940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27480cb410 con 0x7f27480cf230 2024-04-02T08:53:19.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.396+0000 7f273ea00700 1 -- 172.21.15.89:0/130507940 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v182) v1 ==== 74+0+13042 (secure 0 0 0) 0x7f27380577f0 con 0x7f27480cf230 2024-04-02T08:53:19.395 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:19.395 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":182,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:53:18.443014+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:53:19.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.399+0000 7f274dc00700 1 -- 172.21.15.89:0/130507940 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f272806adb0 msgr2=0x7f272806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:19.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.399+0000 7f274dc00700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f272806adb0 0x7f272806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f27480cfad0 tx=0x7f2734008030 comp rx=0 tx=0).stop 2024-04-02T08:53:19.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.399+0000 7f274dc00700 1 -- 172.21.15.89:0/130507940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27480cf230 msgr2=0x7f2748156940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:19.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.399+0000 7f274dc00700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27480cf230 0x7f2748156940 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f273800e760 tx=0x7f273800eb20 comp rx=0 tx=0).stop 2024-04-02T08:53:19.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.399+0000 7f274dc00700 1 -- 172.21.15.89:0/130507940 shutdown_connections 2024-04-02T08:53:19.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.399+0000 7f274dc00700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f272806adb0 0x7f272806d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.399+0000 7f274dc00700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2748161590 0x7f27481583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.399+0000 7f274dc00700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27480cf230 0x7f2748156940 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.399+0000 7f274dc00700 1 --2- 172.21.15.89:0/130507940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2748160f10 0x7f27481484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.399+0000 7f274dc00700 1 -- 172.21.15.89:0/130507940 >> 172.21.15.89:0/130507940 conn(0x7f274808aff0 msgr2=0x7f27480cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:19.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.399+0000 7f274dc00700 1 -- 172.21.15.89:0/130507940 shutdown_connections 2024-04-02T08:53:19.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.399+0000 7f274dc00700 1 -- 172.21.15.89:0/130507940 wait complete. 2024-04-02T08:53:19.417 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T08:53:19.417 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T08:53:19.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.613+0000 7f1189c00700 1 Processor -- start 2024-04-02T08:53:19.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.613+0000 7f1189c00700 1 -- start start 2024-04-02T08:53:19.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.614+0000 7f1189c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f118415dad0 0x7f11840ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:19.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.614+0000 7f1189c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f11840d1630 0x7f11840d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:19.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.614+0000 7f1189c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f11840d1fd0 0x7f118415deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:19.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.614+0000 7f1189c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11840cfcc0 con 0x7f11840d1fd0 2024-04-02T08:53:19.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.614+0000 7f1189c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11840cd9f0 con 0x7f11840d1630 2024-04-02T08:53:19.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.614+0000 7f1189c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118415f820 con 0x7f118415dad0 2024-04-02T08:53:19.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.614+0000 7f1183400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f11840d1fd0 0x7f118415deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:19.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.614+0000 7f1183400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f11840d1fd0 0x7f118415deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53942/0 (socket says 172.21.15.89:53942) 2024-04-02T08:53:19.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.614+0000 7f1183400700 1 -- 172.21.15.89:0/2668282657 learned_addr learned my addr 172.21.15.89:0/2668282657 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:19.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.615+0000 7f1182a00700 1 --2- 172.21.15.89:0/2668282657 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f118415dad0 0x7f11840ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:19.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.615+0000 7f1182000700 1 --2- 172.21.15.89:0/2668282657 >> v2:172.21.15.113:3300/0 conn(0x7f11840d1630 0x7f11840d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:19.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.615+0000 7f1183400700 1 -- 172.21.15.89:0/2668282657 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f118415dad0 msgr2=0x7f11840ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:19.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.615+0000 7f1183400700 1 --2- 172.21.15.89:0/2668282657 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f118415dad0 0x7f11840ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.615+0000 7f1183400700 1 -- 172.21.15.89:0/2668282657 >> v2:172.21.15.113:3300/0 conn(0x7f11840d1630 msgr2=0x7f11840d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:19.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.615+0000 7f1183400700 1 --2- 172.21.15.89:0/2668282657 >> v2:172.21.15.113:3300/0 conn(0x7f11840d1630 0x7f11840d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.615+0000 7f1183400700 1 -- 172.21.15.89:0/2668282657 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11840d2460 con 0x7f11840d1fd0 2024-04-02T08:53:19.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.615+0000 7f1182000700 1 --2- 172.21.15.89:0/2668282657 >> v2:172.21.15.113:3300/0 conn(0x7f11840d1630 0x7f11840d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:53:19.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.615+0000 7f1183400700 1 --2- 172.21.15.89:0/2668282657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11840d1fd0 0x7f118415deb0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f118408c330 tx=0x7f117c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb92a441627460bd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:19.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.615+0000 7f1181600700 1 -- 172.21.15.89:0/2668282657 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c00e070 con 0x7f11840d1fd0 2024-04-02T08:53:19.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.615+0000 7f1181600700 1 -- 172.21.15.89:0/2668282657 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117c00b9c0 con 0x7f11840d1fd0 2024-04-02T08:53:19.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.615+0000 7f1181600700 1 -- 172.21.15.89:0/2668282657 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c00bd70 con 0x7f11840d1fd0 2024-04-02T08:53:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.616+0000 7f1189c00700 1 -- 172.21.15.89:0/2668282657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11840d1fd0 msgr2=0x7f118415deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.616+0000 7f1189c00700 1 --2- 172.21.15.89:0/2668282657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11840d1fd0 0x7f118415deb0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f118408c330 tx=0x7f117c00b330 comp rx=0 tx=0).stop 2024-04-02T08:53:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.616+0000 7f1189c00700 1 -- 172.21.15.89:0/2668282657 shutdown_connections 2024-04-02T08:53:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.616+0000 7f1189c00700 1 --2- 172.21.15.89:0/2668282657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11840d1fd0 0x7f118415deb0 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.616+0000 7f1189c00700 1 --2- 172.21.15.89:0/2668282657 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f118415dad0 0x7f11840ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.616+0000 7f1189c00700 1 --2- 172.21.15.89:0/2668282657 >> v2:172.21.15.113:3300/0 conn(0x7f11840d1630 0x7f11840d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.616+0000 7f1189c00700 1 -- 172.21.15.89:0/2668282657 >> 172.21.15.89:0/2668282657 conn(0x7f118408aff0 msgr2=0x7f11840d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:19.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.616+0000 7f1189c00700 1 -- 172.21.15.89:0/2668282657 shutdown_connections 2024-04-02T08:53:19.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.616+0000 7f1189c00700 1 -- 172.21.15.89:0/2668282657 wait complete. 2024-04-02T08:53:19.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.617+0000 7f1189c00700 1 Processor -- start 2024-04-02T08:53:19.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.618+0000 7f1189c00700 1 -- start start 2024-04-02T08:53:19.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.618+0000 7f1189c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11840d1630 0x7f11840c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:19.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.618+0000 7f1189c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11840d1fd0 0x7f11840c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:19.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1189c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f118415dad0 0x7f11841f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:19.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1189c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11841520e0 con 0x7f11840d1630 2024-04-02T08:53:19.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1189c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11840c84c0 con 0x7f118415dad0 2024-04-02T08:53:19.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1189c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11841f2e60 con 0x7f11840d1fd0 2024-04-02T08:53:19.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1182a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11840d1630 0x7f11840c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:19.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1182a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11840d1630 0x7f11840c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53954/0 (socket says 172.21.15.89:53954) 2024-04-02T08:53:19.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1182a00700 1 -- 172.21.15.89:0/3184908026 learned_addr learned my addr 172.21.15.89:0/3184908026 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:19.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1182000700 1 --2- 172.21.15.89:0/3184908026 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11840d1fd0 0x7f11840c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:19.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1183400700 1 --2- 172.21.15.89:0/3184908026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f118415dad0 0x7f11841f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:19.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1182a00700 1 -- 172.21.15.89:0/3184908026 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11840d1fd0 msgr2=0x7f11840c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:19.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1182a00700 1 --2- 172.21.15.89:0/3184908026 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11840d1fd0 0x7f11840c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1182a00700 1 -- 172.21.15.89:0/3184908026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f118415dad0 msgr2=0x7f11841f2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:19.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1182a00700 1 --2- 172.21.15.89:0/3184908026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f118415dad0 0x7f11841f2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1182a00700 1 -- 172.21.15.89:0/3184908026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c009b30 con 0x7f11840d1630 2024-04-02T08:53:19.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.619+0000 7f1182a00700 1 --2- 172.21.15.89:0/3184908026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11840d1630 0x7f11840c9830 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f117400d800 tx=0x7f117400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:19.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.620+0000 7f117aa00700 1 -- 172.21.15.89:0/3184908026 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11740125f0 con 0x7f11840d1630 2024-04-02T08:53:19.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.620+0000 7f117aa00700 1 -- 172.21.15.89:0/3184908026 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1174012750 con 0x7f11840d1630 2024-04-02T08:53:19.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.620+0000 7f117aa00700 1 -- 172.21.15.89:0/3184908026 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174012b00 con 0x7f11840d1630 2024-04-02T08:53:19.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.620+0000 7f1189c00700 1 -- 172.21.15.89:0/3184908026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11840c9e80 con 0x7f11840d1630 2024-04-02T08:53:19.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.620+0000 7f1189c00700 1 -- 172.21.15.89:0/3184908026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11840c7e70 con 0x7f11840d1630 2024-04-02T08:53:19.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.621+0000 7f1189c00700 1 -- 172.21.15.89:0/3184908026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11840c87a0 con 0x7f11840d1630 2024-04-02T08:53:19.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.622+0000 7f117aa00700 1 -- 172.21.15.89:0/3184908026 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1174012c60 con 0x7f11840d1630 2024-04-02T08:53:19.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.622+0000 7f117aa00700 1 --2- 172.21.15.89:0/3184908026 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f116406adb0 0x7f116406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:19.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.623+0000 7f117aa00700 1 -- 172.21.15.89:0/3184908026 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f11740893b0 con 0x7f11840d1630 2024-04-02T08:53:19.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.623+0000 7f1182000700 1 --2- 172.21.15.89:0/3184908026 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f116406adb0 0x7f116406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:19.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.623+0000 7f1182000700 1 --2- 172.21.15.89:0/3184908026 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f116406adb0 0x7f116406d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f118408d650 tx=0x7f1170008040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:19.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.627+0000 7f117aa00700 1 -- 172.21.15.89:0/3184908026 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1174054640 con 0x7f11840d1630 2024-04-02T08:53:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.829+0000 7f1189c00700 1 -- 172.21.15.89:0/3184908026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1184067720 con 0x7f11840d1630 2024-04-02T08:53:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.829+0000 7f117aa00700 1 -- 172.21.15.89:0/3184908026 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v182) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1174057c60 con 0x7f11840d1630 2024-04-02T08:53:19.828 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T08:53:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.831+0000 7f1189c00700 1 -- 172.21.15.89:0/3184908026 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f116406adb0 msgr2=0x7f116406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.831+0000 7f1189c00700 1 --2- 172.21.15.89:0/3184908026 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f116406adb0 0x7f116406d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f118408d650 tx=0x7f1170008040 comp rx=0 tx=0).stop 2024-04-02T08:53:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.831+0000 7f1189c00700 1 -- 172.21.15.89:0/3184908026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11840d1630 msgr2=0x7f11840c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.831+0000 7f1189c00700 1 --2- 172.21.15.89:0/3184908026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11840d1630 0x7f11840c9830 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f117400d800 tx=0x7f117400db10 comp rx=0 tx=0).stop 2024-04-02T08:53:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.831+0000 7f1189c00700 1 -- 172.21.15.89:0/3184908026 shutdown_connections 2024-04-02T08:53:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.831+0000 7f1189c00700 1 --2- 172.21.15.89:0/3184908026 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f116406adb0 0x7f116406d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.831+0000 7f1189c00700 1 --2- 172.21.15.89:0/3184908026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f118415dad0 0x7f11841f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.831+0000 7f1189c00700 1 --2- 172.21.15.89:0/3184908026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11840d1630 0x7f11840c9830 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.831+0000 7f1189c00700 1 --2- 172.21.15.89:0/3184908026 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11840d1fd0 0x7f11840c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.831+0000 7f1189c00700 1 -- 172.21.15.89:0/3184908026 >> 172.21.15.89:0/3184908026 conn(0x7f118408aff0 msgr2=0x7f11840ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.832+0000 7f1189c00700 1 -- 172.21.15.89:0/3184908026 shutdown_connections 2024-04-02T08:53:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:19.832+0000 7f1189c00700 1 -- 172.21.15.89:0/3184908026 wait complete. 2024-04-02T08:53:19.848 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T08:53:20.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.039+0000 7fc0c0200700 1 Processor -- start 2024-04-02T08:53:20.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.039+0000 7fc0c0200700 1 -- start start 2024-04-02T08:53:20.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.040+0000 7fc0c0200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc0b80b7a30 0x7fc0b81685d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:20.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.040+0000 7fc0c0200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b8160fc0 con 0x7fc0b80ce590 2024-04-02T08:53:20.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.040+0000 7fc0c0200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b80d21e0 con 0x7fc0b8160be0 2024-04-02T08:53:20.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.040+0000 7fc0c0200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b80b65d0 con 0x7fc0b80b7a30 2024-04-02T08:53:20.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0bce00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc0b80ce590 0x7fc0b80ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46344/0 (socket says 172.21.15.89:46344) 2024-04-02T08:53:20.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0bce00700 1 -- 172.21.15.89:0/99627185 learned_addr learned my addr 172.21.15.89:0/99627185 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0be200700 1 --2- 172.21.15.89:0/99627185 >> v2:172.21.15.113:3300/0 conn(0x7fc0b80b7a30 0x7fc0b81685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0b7e00700 1 -- 172.21.15.89:0/99627185 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492047890 0 0) 0x7fc0b8160fc0 con 0x7fc0b80ce590 2024-04-02T08:53:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0b7e00700 1 -- 172.21.15.89:0/99627185 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a00035b0 con 0x7fc0b80ce590 2024-04-02T08:53:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0b7e00700 1 -- 172.21.15.89:0/99627185 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3655209072 0 0) 0x7fc0a00035b0 con 0x7fc0b80ce590 2024-04-02T08:53:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0b7e00700 1 -- 172.21.15.89:0/99627185 >> v2:172.21.15.113:3300/0 conn(0x7fc0b80b7a30 msgr2=0x7fc0b81685d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0b7e00700 1 --2- 172.21.15.89:0/99627185 >> v2:172.21.15.113:3300/0 conn(0x7fc0b80b7a30 0x7fc0b81685d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0b7e00700 1 -- 172.21.15.89:0/99627185 >> v1:172.21.15.113:6789/0 conn(0x7fc0b8160be0 legacy=0x7fc0b80cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0b7e00700 1 -- 172.21.15.89:0/99627185 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b80d24c0 con 0x7fc0b80ce590 2024-04-02T08:53:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0b7e00700 1 -- 172.21.15.89:0/99627185 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fc0a4001670 con 0x7fc0b80ce590 2024-04-02T08:53:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0b7e00700 1 -- 172.21.15.89:0/99627185 >> v1:172.21.15.89:6789/0 conn(0x7fc0b80ce590 legacy=0x7fc0b80ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0b7e00700 1 --2- 172.21.15.89:0/99627185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0a0003a00 0x7fc0a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0b7e00700 1 --2- 172.21.15.89:0/99627185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0a0004850 0x7fc0a0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.041+0000 7fc0b7e00700 1 --2- 172.21.15.89:0/99627185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0a0005210 0x7fc0a000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.042+0000 7fc0b7e00700 1 -- 172.21.15.89:0/99627185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b80b65d0 con 0x7fc0a0003a00 2024-04-02T08:53:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.042+0000 7fc0b7e00700 1 -- 172.21.15.89:0/99627185 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a000c0b0 con 0x7fc0a0004850 2024-04-02T08:53:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.042+0000 7fc0b7e00700 1 -- 172.21.15.89:0/99627185 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a000c270 con 0x7fc0a0005210 2024-04-02T08:53:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.042+0000 7fc0bd800700 1 --2- 172.21.15.89:0/99627185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0a0003a00 0x7fc0a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.042+0000 7fc0bce00700 1 --2- 172.21.15.89:0/99627185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0a0004850 0x7fc0a0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.042+0000 7fc0be200700 1 --2- 172.21.15.89:0/99627185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0a0005210 0x7fc0a000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.042+0000 7fc0bd800700 1 -- 172.21.15.89:0/99627185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0a0005210 msgr2=0x7fc0a000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.042+0000 7fc0bd800700 1 --2- 172.21.15.89:0/99627185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0a0005210 0x7fc0a000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.042+0000 7fc0bd800700 1 -- 172.21.15.89:0/99627185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0a0004850 msgr2=0x7fc0a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.042+0000 7fc0bd800700 1 --2- 172.21.15.89:0/99627185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0a0004850 0x7fc0a0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.042+0000 7fc0bd800700 1 -- 172.21.15.89:0/99627185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a00035b0 con 0x7fc0a0003a00 2024-04-02T08:53:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.042+0000 7fc0be200700 1 -- 172.21.15.89:0/99627185 reap_dead start 2024-04-02T08:53:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.042+0000 7fc0bd800700 1 --2- 172.21.15.89:0/99627185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0a0003a00 0x7fc0a0003e20 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac0017c0 tx=0x7fc0a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c81e321d4c6bec8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.043+0000 7fc0b7e00700 1 -- 172.21.15.89:0/99627185 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac0052d0 con 0x7fc0a0003a00 2024-04-02T08:53:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.043+0000 7fc0b7e00700 1 -- 172.21.15.89:0/99627185 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ac005430 con 0x7fc0a0003a00 2024-04-02T08:53:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.043+0000 7fc0b7e00700 1 -- 172.21.15.89:0/99627185 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac0038c0 con 0x7fc0a0003a00 2024-04-02T08:53:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.043+0000 7fc0c0200700 1 -- 172.21.15.89:0/99627185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0a0003a00 msgr2=0x7fc0a0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.043+0000 7fc0c0200700 1 --2- 172.21.15.89:0/99627185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0a0003a00 0x7fc0a0003e20 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac0017c0 tx=0x7fc0a0005880 comp rx=0 tx=0).stop 2024-04-02T08:53:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.044+0000 7fc0c0200700 1 -- 172.21.15.89:0/99627185 shutdown_connections 2024-04-02T08:53:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.044+0000 7fc0c0200700 1 --2- 172.21.15.89:0/99627185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0a0003a00 0x7fc0a0003e20 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.044+0000 7fc0c0200700 1 -- 172.21.15.89:0/99627185 >> 172.21.15.89:0/99627185 conn(0x7fc0b808aff0 msgr2=0x7fc0b80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.044+0000 7fc0c0200700 1 -- 172.21.15.89:0/99627185 shutdown_connections 2024-04-02T08:53:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.044+0000 7fc0c0200700 1 -- 172.21.15.89:0/99627185 wait complete. 2024-04-02T08:53:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.046+0000 7fc0c0200700 1 Processor -- start 2024-04-02T08:53:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.046+0000 7fc0c0200700 1 -- start start 2024-04-02T08:53:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0c0200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0a0003a00 0x7fc0b8146d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0c0200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0b8145a00 0x7fc0b8144110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0c0200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0b806c3c0 0x7fc0b81f2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0c0200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b8145570 con 0x7fc0a0003a00 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0c0200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b8144960 con 0x7fc0b8145a00 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0c0200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b8144c40 con 0x7fc0b806c3c0 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0bd800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0a0003a00 0x7fc0b8146d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0bd800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0a0003a00 0x7fc0b8146d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53966/0 (socket says 172.21.15.89:53966) 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0bd800700 1 -- 172.21.15.89:0/3624259307 learned_addr learned my addr 172.21.15.89:0/3624259307 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0bce00700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0b8145a00 0x7fc0b8144110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0be200700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0b806c3c0 0x7fc0b81f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0bd800700 1 -- 172.21.15.89:0/3624259307 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0b806c3c0 msgr2=0x7fc0b81f2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0bd800700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0b806c3c0 0x7fc0b81f2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0bd800700 1 -- 172.21.15.89:0/3624259307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0b8145a00 msgr2=0x7fc0b8144110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0bd800700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0b8145a00 0x7fc0b8144110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0bd800700 1 -- 172.21.15.89:0/3624259307 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b8144f20 con 0x7fc0a0003a00 2024-04-02T08:53:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0bce00700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0b8145a00 0x7fc0b8144110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:20.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.047+0000 7fc0bd800700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0a0003a00 0x7fc0b8146d70 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fc0a000c080 tx=0x7fc0ac001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:20.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.048+0000 7fc0b5600700 1 -- 172.21.15.89:0/3624259307 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac001290 con 0x7fc0a0003a00 2024-04-02T08:53:20.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.048+0000 7fc0b5600700 1 -- 172.21.15.89:0/3624259307 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ac026460 con 0x7fc0a0003a00 2024-04-02T08:53:20.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.048+0000 7fc0c0200700 1 -- 172.21.15.89:0/3624259307 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b8145200 con 0x7fc0a0003a00 2024-04-02T08:53:20.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.048+0000 7fc0b5600700 1 -- 172.21.15.89:0/3624259307 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac026810 con 0x7fc0a0003a00 2024-04-02T08:53:20.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.048+0000 7fc0c0200700 1 -- 172.21.15.89:0/3624259307 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b8145e80 con 0x7fc0a0003a00 2024-04-02T08:53:20.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.049+0000 7fc0c0200700 1 -- 172.21.15.89:0/3624259307 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b8067720 con 0x7fc0a0003a00 2024-04-02T08:53:20.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.050+0000 7fc0b5600700 1 -- 172.21.15.89:0/3624259307 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc0ac019070 con 0x7fc0a0003a00 2024-04-02T08:53:20.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.051+0000 7fc0b5600700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc09806adb0 0x7fc09806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:20.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.051+0000 7fc0b5600700 1 -- 172.21.15.89:0/3624259307 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5277+0+0 (secure 0 0 0) 0x7fc0ac08a260 con 0x7fc0a0003a00 2024-04-02T08:53:20.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.051+0000 7fc0bce00700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc09806adb0 0x7fc09806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:20.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.051+0000 7fc0bce00700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc09806adb0 0x7fc09806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80cd1b0 tx=0x7fc0a4009040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:20.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.055+0000 7fc0b5600700 1 -- 172.21.15.89:0/3624259307 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc0ac0555f0 con 0x7fc0a0003a00 2024-04-02T08:53:20.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.258+0000 7fc0c0200700 1 -- 172.21.15.89:0/3624259307 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc0b80b5cb0 con 0x7fc0a0003a00 2024-04-02T08:53:20.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.258+0000 7fc0b5600700 1 -- 172.21.15.89:0/3624259307 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v182) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc0ac058c10 con 0x7fc0a0003a00 2024-04-02T08:53:20.257 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:20.258 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T08:53:20.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:20.258 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T08:53:20.258 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T08:53:20.258 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:20.258 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T08:53:20.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.261+0000 7fc0c0200700 1 -- 172.21.15.89:0/3624259307 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc09806adb0 msgr2=0x7fc09806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:20.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.261+0000 7fc0c0200700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc09806adb0 0x7fc09806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80cd1b0 tx=0x7fc0a4009040 comp rx=0 tx=0).stop 2024-04-02T08:53:20.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.261+0000 7fc0c0200700 1 -- 172.21.15.89:0/3624259307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0a0003a00 msgr2=0x7fc0b8146d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:20.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.261+0000 7fc0c0200700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0a0003a00 0x7fc0b8146d70 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fc0a000c080 tx=0x7fc0ac001120 comp rx=0 tx=0).stop 2024-04-02T08:53:20.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.261+0000 7fc0c0200700 1 -- 172.21.15.89:0/3624259307 shutdown_connections 2024-04-02T08:53:20.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.261+0000 7fc0c0200700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fc09806adb0 0x7fc09806d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.261+0000 7fc0c0200700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0b806c3c0 0x7fc0b81f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.261+0000 7fc0c0200700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0a0003a00 0x7fc0b8146d70 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.261+0000 7fc0c0200700 1 --2- 172.21.15.89:0/3624259307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0b8145a00 0x7fc0b8144110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.261+0000 7fc0c0200700 1 -- 172.21.15.89:0/3624259307 >> 172.21.15.89:0/3624259307 conn(0x7fc0b808aff0 msgr2=0x7fc0b80b5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:20.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.261+0000 7fc0c0200700 1 -- 172.21.15.89:0/3624259307 shutdown_connections 2024-04-02T08:53:20.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.261+0000 7fc0c0200700 1 -- 172.21.15.89:0/3624259307 wait complete. 2024-04-02T08:53:20.277 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T08:53:20.277 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T08:53:20.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.475+0000 7f1162c00700 1 Processor -- start 2024-04-02T08:53:20.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.475+0000 7f1162c00700 1 -- start start 2024-04-02T08:53:20.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.476+0000 7f1162c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f115c15d930 0x7f115c0d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:20.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.476+0000 7f1162c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f115c0d22a0 0x7f115c0ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:20.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.476+0000 7f1162c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115c15dd10 con 0x7f115c162030 2024-04-02T08:53:20.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.476+0000 7f1162c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f115c15ec40 con 0x7f115c15d930 2024-04-02T08:53:20.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.476+0000 7f1162c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f115c15dfc0 con 0x7f115c0d22a0 2024-04-02T08:53:20.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.476+0000 7f115be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f115c15d930 0x7f115c0d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:20.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.476+0000 7f115b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f115c162030 0x7f115c0d1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46356/0 (socket says 172.21.15.89:46356) 2024-04-02T08:53:20.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.476+0000 7f115b400700 1 -- 172.21.15.89:0/2738975529 learned_addr learned my addr 172.21.15.89:0/2738975529 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:20.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.477+0000 7f115be00700 1 -- 172.21.15.89:0/2738975529 >> v2:172.21.15.113:3300/0 conn(0x7f115c0d22a0 msgr2=0x7f115c0ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:20.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.477+0000 7f1160c00700 1 --2- 172.21.15.89:0/2738975529 >> v2:172.21.15.113:3300/0 conn(0x7f115c0d22a0 0x7f115c0ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:20.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.477+0000 7f115be00700 1 --2- 172.21.15.89:0/2738975529 >> v2:172.21.15.113:3300/0 conn(0x7f115c0d22a0 0x7f115c0ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.477+0000 7f115be00700 1 -- 172.21.15.89:0/2738975529 >> v1:172.21.15.89:6789/0 conn(0x7f115c162030 legacy=0x7f115c0d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:20.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.477+0000 7f115aa00700 1 -- 172.21.15.89:0/2738975529 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412341691 0 0) 0x7f115c15dd10 con 0x7f115c162030 2024-04-02T08:53:20.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.477+0000 7f115be00700 1 -- 172.21.15.89:0/2738975529 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115c15e2a0 con 0x7f115c15d930 2024-04-02T08:53:20.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.477+0000 7f1160c00700 1 --2- 172.21.15.89:0/2738975529 >> v2:172.21.15.113:3300/0 conn(0x7f115c0d22a0 0x7f115c0ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:53:20.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.477+0000 7f115be00700 1 --2- 172.21.15.89:0/2738975529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c15d930 0x7f115c0d3ed0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f115c0d2800 tx=0x7f114c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53368ad0f6f7fa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:20.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.477+0000 7f115aa00700 1 -- 172.21.15.89:0/2738975529 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c011830 con 0x7f115c15d930 2024-04-02T08:53:20.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.477+0000 7f115aa00700 1 -- 172.21.15.89:0/2738975529 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114c011990 con 0x7f115c15d930 2024-04-02T08:53:20.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.477+0000 7f115aa00700 1 -- 172.21.15.89:0/2738975529 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c011d20 con 0x7f115c15d930 2024-04-02T08:53:20.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.478+0000 7f1162c00700 1 -- 172.21.15.89:0/2738975529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c15d930 msgr2=0x7f115c0d3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:20.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.478+0000 7f1162c00700 1 --2- 172.21.15.89:0/2738975529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c15d930 0x7f115c0d3ed0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f115c0d2800 tx=0x7f114c009b20 comp rx=0 tx=0).stop 2024-04-02T08:53:20.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.478+0000 7f1162c00700 1 -- 172.21.15.89:0/2738975529 shutdown_connections 2024-04-02T08:53:20.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.478+0000 7f1162c00700 1 --2- 172.21.15.89:0/2738975529 >> v2:172.21.15.113:3300/0 conn(0x7f115c0d22a0 0x7f115c0ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.478+0000 7f1162c00700 1 --2- 172.21.15.89:0/2738975529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c15d930 0x7f115c0d3ed0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.478+0000 7f1162c00700 1 -- 172.21.15.89:0/2738975529 >> 172.21.15.89:0/2738975529 conn(0x7f115c08aff0 msgr2=0x7f115c0d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:20.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.478+0000 7f1162c00700 1 -- 172.21.15.89:0/2738975529 shutdown_connections 2024-04-02T08:53:20.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.478+0000 7f1162c00700 1 -- 172.21.15.89:0/2738975529 wait complete. 2024-04-02T08:53:20.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.479+0000 7f1162c00700 1 Processor -- start 2024-04-02T08:53:20.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.480+0000 7f1162c00700 1 -- start start 2024-04-02T08:53:20.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.480+0000 7f1162c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c0d22a0 0x7f115c15cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:20.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.480+0000 7f1162c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f115c15d930 0x7f115c0c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:20.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.481+0000 7f1162c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f115c162030 0x7f115c0b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:20.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.481+0000 7f1162c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c0cb0c0 con 0x7f115c0d22a0 2024-04-02T08:53:20.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.481+0000 7f1162c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c15d090 con 0x7f115c15d930 2024-04-02T08:53:20.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.481+0000 7f1162c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115c0c7ae0 con 0x7f115c162030 2024-04-02T08:53:20.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.481+0000 7f115be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c0d22a0 0x7f115c15cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:20.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.481+0000 7f115be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c0d22a0 0x7f115c15cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53986/0 (socket says 172.21.15.89:53986) 2024-04-02T08:53:20.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.481+0000 7f115be00700 1 -- 172.21.15.89:0/1734180265 learned_addr learned my addr 172.21.15.89:0/1734180265 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:20.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.481+0000 7f115be00700 1 -- 172.21.15.89:0/1734180265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f115c162030 msgr2=0x7f115c0b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:20.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.481+0000 7f115be00700 1 --2- 172.21.15.89:0/1734180265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f115c162030 0x7f115c0b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.481+0000 7f115be00700 1 -- 172.21.15.89:0/1734180265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f115c15d930 msgr2=0x7f115c0c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:20.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.481+0000 7f115be00700 1 --2- 172.21.15.89:0/1734180265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f115c15d930 0x7f115c0c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:20.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.481+0000 7f115be00700 1 -- 172.21.15.89:0/1734180265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c0094a0 con 0x7f115c0d22a0 2024-04-02T08:53:20.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.481+0000 7f115be00700 1 --2- 172.21.15.89:0/1734180265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c0d22a0 0x7f115c15cb50 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f114c000c00 tx=0x7f114c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:20.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.482+0000 7f1153e00700 1 -- 172.21.15.89:0/1734180265 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c0038a0 con 0x7f115c0d22a0 2024-04-02T08:53:20.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.482+0000 7f1153e00700 1 -- 172.21.15.89:0/1734180265 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114c01f070 con 0x7f115c0d22a0 2024-04-02T08:53:20.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.482+0000 7f1153e00700 1 -- 172.21.15.89:0/1734180265 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c003c40 con 0x7f115c0d22a0 2024-04-02T08:53:20.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.482+0000 7f1162c00700 1 -- 172.21.15.89:0/1734180265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f115c0ca720 con 0x7f115c0d22a0 2024-04-02T08:53:20.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.482+0000 7f1162c00700 1 -- 172.21.15.89:0/1734180265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f115c0cace0 con 0x7f115c0d22a0 2024-04-02T08:53:20.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.483+0000 7f1162c00700 1 -- 172.21.15.89:0/1734180265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f115c067720 con 0x7f115c0d22a0 2024-04-02T08:53:20.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.484+0000 7f1153e00700 1 -- 172.21.15.89:0/1734180265 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f114c026480 con 0x7f115c0d22a0 2024-04-02T08:53:20.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.484+0000 7f1153e00700 1 --2- 172.21.15.89:0/1734180265 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f113c06adb0 0x7f113c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:20.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.485+0000 7f1153e00700 1 -- 172.21.15.89:0/1734180265 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f114c021070 con 0x7f115c0d22a0 2024-04-02T08:53:20.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.485+0000 7f115b400700 1 --2- 172.21.15.89:0/1734180265 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f113c06adb0 0x7f113c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:20.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.485+0000 7f115b400700 1 --2- 172.21.15.89:0/1734180265 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f113c06adb0 0x7f113c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f115c0b3110 tx=0x7f11480079f0 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:20.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.489+0000 7f1153e00700 1 -- 172.21.15.89:0/1734180265 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f114c0560f0 con 0x7f115c0d22a0 2024-04-02T08:53:20.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:20.691+0000 7f1162c00700 1 -- 172.21.15.89:0/1734180265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f115c0cae70 con 0x7f115c0d22a0 2024-04-02T08:53:21.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.374+0000 7f1153e00700 1 -- 172.21.15.89:0/1734180265 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v183) v1 ==== 138+0+0 (secure 0 0 0) 0x7f114c02b020 con 0x7f115c0d22a0 2024-04-02T08:53:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.378+0000 7f1162c00700 1 -- 172.21.15.89:0/1734180265 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f113c06adb0 msgr2=0x7f113c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.378+0000 7f1162c00700 1 --2- 172.21.15.89:0/1734180265 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f113c06adb0 0x7f113c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f115c0b3110 tx=0x7f11480079f0 comp rx=0 tx=0).stop 2024-04-02T08:53:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.379+0000 7f1162c00700 1 -- 172.21.15.89:0/1734180265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c0d22a0 msgr2=0x7f115c15cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.379+0000 7f1162c00700 1 --2- 172.21.15.89:0/1734180265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c0d22a0 0x7f115c15cb50 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f114c000c00 tx=0x7f114c004750 comp rx=0 tx=0).stop 2024-04-02T08:53:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.379+0000 7f1162c00700 1 -- 172.21.15.89:0/1734180265 shutdown_connections 2024-04-02T08:53:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.379+0000 7f1162c00700 1 --2- 172.21.15.89:0/1734180265 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f113c06adb0 0x7f113c06d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.379+0000 7f1162c00700 1 --2- 172.21.15.89:0/1734180265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f115c162030 0x7f115c0b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.379+0000 7f1162c00700 1 --2- 172.21.15.89:0/1734180265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115c0d22a0 0x7f115c15cb50 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.379+0000 7f1162c00700 1 --2- 172.21.15.89:0/1734180265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f115c15d930 0x7f115c0c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.379+0000 7f1162c00700 1 -- 172.21.15.89:0/1734180265 >> 172.21.15.89:0/1734180265 conn(0x7f115c08aff0 msgr2=0x7f115c06cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.379+0000 7f1162c00700 1 -- 172.21.15.89:0/1734180265 shutdown_connections 2024-04-02T08:53:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.380+0000 7f1162c00700 1 -- 172.21.15.89:0/1734180265 wait complete. 2024-04-02T08:53:21.380 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T08:53:21.400 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T08:53:21.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.601+0000 7f4994800700 1 Processor -- start 2024-04-02T08:53:21.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.602+0000 7f4994800700 1 -- start start 2024-04-02T08:53:21.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4994800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f498c160d70 0x7f498c0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:21.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4994800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c15ef70 0x7f498c15e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:21.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4994800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f498c06a790 0x7f498c06ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:21.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4994800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f498c06b180 con 0x7f498c06a790 2024-04-02T08:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4994800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f498c0b4cd0 con 0x7f498c160d70 2024-04-02T08:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4994800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498c161150 con 0x7f498c15ef70 2024-04-02T08:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4992800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f498c06a790 0x7f498c06ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4992800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f498c06a790 0x7f498c06ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54002/0 (socket says 172.21.15.89:54002) 2024-04-02T08:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4992800700 1 -- 172.21.15.89:0/2218794440 learned_addr learned my addr 172.21.15.89:0/2218794440 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4991e00700 1 --2- 172.21.15.89:0/2218794440 >> v2:172.21.15.113:3300/0 conn(0x7f498c160d70 0x7f498c0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4991400700 1 --2- 172.21.15.89:0/2218794440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c15ef70 0x7f498c15e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4992800700 1 -- 172.21.15.89:0/2218794440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c15ef70 msgr2=0x7f498c15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4992800700 1 --2- 172.21.15.89:0/2218794440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c15ef70 0x7f498c15e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4992800700 1 -- 172.21.15.89:0/2218794440 >> v2:172.21.15.113:3300/0 conn(0x7f498c160d70 msgr2=0x7f498c0b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.603+0000 7f4992800700 1 --2- 172.21.15.89:0/2218794440 >> v2:172.21.15.113:3300/0 conn(0x7f498c160d70 0x7f498c0b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.604+0000 7f4992800700 1 -- 172.21.15.89:0/2218794440 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c06dba0 con 0x7f498c06a790 2024-04-02T08:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.604+0000 7f4992800700 1 --2- 172.21.15.89:0/2218794440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c06a790 0x7f498c06ac40 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f498c06c880 tx=0x7f498800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d74f4f31d40c98 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.604+0000 7f4990a00700 1 -- 172.21.15.89:0/2218794440 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498800e070 con 0x7f498c06a790 2024-04-02T08:53:21.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.604+0000 7f4990a00700 1 -- 172.21.15.89:0/2218794440 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498800b9c0 con 0x7f498c06a790 2024-04-02T08:53:21.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.604+0000 7f4990a00700 1 -- 172.21.15.89:0/2218794440 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498800bd70 con 0x7f498c06a790 2024-04-02T08:53:21.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.604+0000 7f4994800700 1 -- 172.21.15.89:0/2218794440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c06a790 msgr2=0x7f498c06ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:21.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.604+0000 7f4994800700 1 --2- 172.21.15.89:0/2218794440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c06a790 0x7f498c06ac40 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f498c06c880 tx=0x7f498800b330 comp rx=0 tx=0).stop 2024-04-02T08:53:21.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.605+0000 7f4994800700 1 -- 172.21.15.89:0/2218794440 shutdown_connections 2024-04-02T08:53:21.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.605+0000 7f4994800700 1 --2- 172.21.15.89:0/2218794440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c06a790 0x7f498c06ac40 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:21.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.605+0000 7f4994800700 1 --2- 172.21.15.89:0/2218794440 >> v2:172.21.15.113:3300/0 conn(0x7f498c160d70 0x7f498c0b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:21.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.605+0000 7f4994800700 1 --2- 172.21.15.89:0/2218794440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c15ef70 0x7f498c15e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:21.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.605+0000 7f4994800700 1 -- 172.21.15.89:0/2218794440 >> 172.21.15.89:0/2218794440 conn(0x7f498c0b5720 msgr2=0x7f498c0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:21.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.605+0000 7f4994800700 1 -- 172.21.15.89:0/2218794440 shutdown_connections 2024-04-02T08:53:21.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.605+0000 7f4994800700 1 -- 172.21.15.89:0/2218794440 wait complete. 2024-04-02T08:53:21.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.606+0000 7f4994800700 1 Processor -- start 2024-04-02T08:53:21.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.606+0000 7f4994800700 1 -- start start 2024-04-02T08:53:21.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.607+0000 7f4994800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c06a790 0x7f498c1f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:21.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.607+0000 7f4994800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c15ef70 0x7f498c06df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:21.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.607+0000 7f4994800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c160d70 0x7f498c06e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:21.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.607+0000 7f4994800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c14bd00 con 0x7f498c15ef70 2024-04-02T08:53:21.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.607+0000 7f4994800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c1488f0 con 0x7f498c160d70 2024-04-02T08:53:21.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.607+0000 7f4994800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498c148610 con 0x7f498c06a790 2024-04-02T08:53:21.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.608+0000 7f4991400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c15ef70 0x7f498c06df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:21.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.608+0000 7f4991400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c15ef70 0x7f498c06df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54014/0 (socket says 172.21.15.89:54014) 2024-04-02T08:53:21.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.608+0000 7f4991400700 1 -- 172.21.15.89:0/2388953359 learned_addr learned my addr 172.21.15.89:0/2388953359 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:21.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.608+0000 7f4991e00700 1 --2- 172.21.15.89:0/2388953359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c06a790 0x7f498c1f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:21.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.608+0000 7f4992800700 1 --2- 172.21.15.89:0/2388953359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c160d70 0x7f498c06e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:21.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.608+0000 7f4991400700 1 -- 172.21.15.89:0/2388953359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c06a790 msgr2=0x7f498c1f2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.608+0000 7f4991400700 1 --2- 172.21.15.89:0/2388953359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c06a790 0x7f498c1f2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.608+0000 7f4991400700 1 -- 172.21.15.89:0/2388953359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c160d70 msgr2=0x7f498c06e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.608+0000 7f4991400700 1 --2- 172.21.15.89:0/2388953359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c160d70 0x7f498c06e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.608+0000 7f4991400700 1 -- 172.21.15.89:0/2388953359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4988009b30 con 0x7f498c15ef70 2024-04-02T08:53:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.608+0000 7f4991400700 1 --2- 172.21.15.89:0/2388953359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c15ef70 0x7f498c06df80 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f497c00e810 tx=0x7f497c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.608+0000 7f4986000700 1 -- 172.21.15.89:0/2388953359 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c00c070 con 0x7f498c15ef70 2024-04-02T08:53:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.608+0000 7f4986000700 1 -- 172.21.15.89:0/2388953359 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497c013610 con 0x7f498c15ef70 2024-04-02T08:53:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.609+0000 7f4994800700 1 -- 172.21.15.89:0/2388953359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498c1f67b0 con 0x7f498c15ef70 2024-04-02T08:53:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.609+0000 7f4986000700 1 -- 172.21.15.89:0/2388953359 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c0139a0 con 0x7f498c15ef70 2024-04-02T08:53:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.609+0000 7f4994800700 1 -- 172.21.15.89:0/2388953359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498c1f6d70 con 0x7f498c15ef70 2024-04-02T08:53:21.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.610+0000 7f4994800700 1 -- 172.21.15.89:0/2388953359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498c06c6a0 con 0x7f498c15ef70 2024-04-02T08:53:21.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.611+0000 7f4986000700 1 -- 172.21.15.89:0/2388953359 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f497c013b00 con 0x7f498c15ef70 2024-04-02T08:53:21.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.612+0000 7f4986000700 1 --2- 172.21.15.89:0/2388953359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f497006adb0 0x7f497006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:21.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.612+0000 7f4986000700 1 -- 172.21.15.89:0/2388953359 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5647+0+0 (secure 0 0 0) 0x7f497c0891e0 con 0x7f498c15ef70 2024-04-02T08:53:21.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.612+0000 7f4991e00700 1 --2- 172.21.15.89:0/2388953359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f497006adb0 0x7f497006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:21.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.613+0000 7f4991e00700 1 --2- 172.21.15.89:0/2388953359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f497006adb0 0x7f497006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4980006b20 tx=0x7f4980008040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:21.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.617+0000 7f4986000700 1 -- 172.21.15.89:0/2388953359 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f497c0541d0 con 0x7f498c15ef70 2024-04-02T08:53:21.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:21.817+0000 7f4994800700 1 -- 172.21.15.89:0/2388953359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f498c15e480 con 0x7f498c15ef70 2024-04-02T08:53:22.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.382+0000 7f4986000700 1 -- 172.21.15.89:0/2388953359 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v184) v1 ==== 170+0+0 (secure 0 0 0) 0x7f497c013db0 con 0x7f498c15ef70 2024-04-02T08:53:22.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.389+0000 7f4994800700 1 -- 172.21.15.89:0/2388953359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f497006adb0 msgr2=0x7f497006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:22.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.389+0000 7f4994800700 1 --2- 172.21.15.89:0/2388953359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f497006adb0 0x7f497006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4980006b20 tx=0x7f4980008040 comp rx=0 tx=0).stop 2024-04-02T08:53:22.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.389+0000 7f4994800700 1 -- 172.21.15.89:0/2388953359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c15ef70 msgr2=0x7f498c06df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:22.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.389+0000 7f4994800700 1 --2- 172.21.15.89:0/2388953359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c15ef70 0x7f498c06df80 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f497c00e810 tx=0x7f497c00eb20 comp rx=0 tx=0).stop 2024-04-02T08:53:22.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.389+0000 7f4994800700 1 -- 172.21.15.89:0/2388953359 shutdown_connections 2024-04-02T08:53:22.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.389+0000 7f4994800700 1 --2- 172.21.15.89:0/2388953359 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f497006adb0 0x7f497006d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:22.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.389+0000 7f4994800700 1 --2- 172.21.15.89:0/2388953359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c160d70 0x7f498c06e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:22.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.389+0000 7f4994800700 1 --2- 172.21.15.89:0/2388953359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498c06a790 0x7f498c1f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:22.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.389+0000 7f4994800700 1 --2- 172.21.15.89:0/2388953359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f498c15ef70 0x7f498c06df80 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:22.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.389+0000 7f4994800700 1 -- 172.21.15.89:0/2388953359 >> 172.21.15.89:0/2388953359 conn(0x7f498c0b5720 msgr2=0x7f498c15d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:22.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.391+0000 7f4994800700 1 -- 172.21.15.89:0/2388953359 shutdown_connections 2024-04-02T08:53:22.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.391+0000 7f4994800700 1 -- 172.21.15.89:0/2388953359 wait complete. 2024-04-02T08:53:22.399 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T08:53:22.420 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T08:53:22.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.616+0000 7f715a600700 1 Processor -- start 2024-04-02T08:53:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.616+0000 7f715a600700 1 -- start start 2024-04-02T08:53:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.617+0000 7f715a600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71540d16a0 0x7f71540d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.617+0000 7f715a600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f71540d2370 0x7f71540ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.617+0000 7f715a600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7154161420 con 0x7f715415d9d0 2024-04-02T08:53:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.617+0000 7f715a600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71540d2820 con 0x7f71540d2370 2024-04-02T08:53:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.617+0000 7f715a600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71540d4340 con 0x7f71540d16a0 2024-04-02T08:53:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.617+0000 7f7153e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f71540d2370 0x7f71540ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.618+0000 7f7153e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f71540d2370 0x7f71540ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54020/0 (socket says 172.21.15.89:54020) 2024-04-02T08:53:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.618+0000 7f7153e00700 1 -- 172.21.15.89:0/1271709912 learned_addr learned my addr 172.21.15.89:0/1271709912 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.618+0000 7f7152a00700 1 --2- 172.21.15.89:0/1271709912 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71540d16a0 0x7f71540d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.618+0000 7f7153e00700 1 -- 172.21.15.89:0/1271709912 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71540d16a0 msgr2=0x7f71540d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.618+0000 7f7153e00700 1 --2- 172.21.15.89:0/1271709912 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71540d16a0 0x7f71540d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:22.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.618+0000 7f7153e00700 1 -- 172.21.15.89:0/1271709912 >> v1:172.21.15.113:6789/0 conn(0x7f715415d9d0 legacy=0x7f71540d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:22.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.618+0000 7f7153e00700 1 -- 172.21.15.89:0/1271709912 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7154161700 con 0x7f71540d2370 2024-04-02T08:53:22.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.618+0000 7f7153e00700 1 --2- 172.21.15.89:0/1271709912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71540d2370 0x7f71540ce360 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f714c009e80 tx=0x7f714c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c99566d438d1948b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:22.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.618+0000 7f7152000700 1 -- 172.21.15.89:0/1271709912 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c00e070 con 0x7f71540d2370 2024-04-02T08:53:22.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.619+0000 7f7152000700 1 -- 172.21.15.89:0/1271709912 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c00b8d0 con 0x7f71540d2370 2024-04-02T08:53:22.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.619+0000 7f7152000700 1 -- 172.21.15.89:0/1271709912 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c00bc80 con 0x7f71540d2370 2024-04-02T08:53:22.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.619+0000 7f715a600700 1 -- 172.21.15.89:0/1271709912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71540d2370 msgr2=0x7f71540ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:22.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.619+0000 7f715a600700 1 --2- 172.21.15.89:0/1271709912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71540d2370 0x7f71540ce360 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f714c009e80 tx=0x7f714c00b330 comp rx=0 tx=0).stop 2024-04-02T08:53:22.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.619+0000 7f715a600700 1 -- 172.21.15.89:0/1271709912 shutdown_connections 2024-04-02T08:53:22.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.619+0000 7f715a600700 1 --2- 172.21.15.89:0/1271709912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71540d2370 0x7f71540ce360 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:22.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.619+0000 7f715a600700 1 --2- 172.21.15.89:0/1271709912 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71540d16a0 0x7f71540d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:22.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.619+0000 7f715a600700 1 -- 172.21.15.89:0/1271709912 >> 172.21.15.89:0/1271709912 conn(0x7f715408aff0 msgr2=0x7f71540d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:22.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.619+0000 7f715a600700 1 -- 172.21.15.89:0/1271709912 shutdown_connections 2024-04-02T08:53:22.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.619+0000 7f715a600700 1 -- 172.21.15.89:0/1271709912 wait complete. 2024-04-02T08:53:22.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.621+0000 7f715a600700 1 Processor -- start 2024-04-02T08:53:22.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.622+0000 7f715a600700 1 -- start start 2024-04-02T08:53:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f715a600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71540d16a0 0x7f71540cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f715a600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71540d2370 0x7f7154159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f715a600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f715415d9d0 0x7f71540cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f715a600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71540b6020 con 0x7f715415d9d0 2024-04-02T08:53:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f715a600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715415a440 con 0x7f71540d2370 2024-04-02T08:53:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f715a600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71540cb2e0 con 0x7f71540d16a0 2024-04-02T08:53:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f7153e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f715415d9d0 0x7f71540cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f7153400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71540d16a0 0x7f71540cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f7153e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f715415d9d0 0x7f71540cc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54022/0 (socket says 172.21.15.89:54022) 2024-04-02T08:53:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f7153e00700 1 -- 172.21.15.89:0/3547876409 learned_addr learned my addr 172.21.15.89:0/3547876409 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f7152a00700 1 --2- 172.21.15.89:0/3547876409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71540d2370 0x7f7154159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f7153e00700 1 -- 172.21.15.89:0/3547876409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71540d16a0 msgr2=0x7f71540cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f7153e00700 1 --2- 172.21.15.89:0/3547876409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71540d16a0 0x7f71540cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f7153e00700 1 -- 172.21.15.89:0/3547876409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71540d2370 msgr2=0x7f7154159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f7153e00700 1 --2- 172.21.15.89:0/3547876409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71540d2370 0x7f7154159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.623+0000 7f7153e00700 1 -- 172.21.15.89:0/3547876409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c009b30 con 0x7f715415d9d0 2024-04-02T08:53:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.624+0000 7f7153e00700 1 --2- 172.21.15.89:0/3547876409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f715415d9d0 0x7f71540cc9e0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f715408c230 tx=0x7f714c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.624+0000 7f714b400700 1 -- 172.21.15.89:0/3547876409 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c004800 con 0x7f715415d9d0 2024-04-02T08:53:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.624+0000 7f714b400700 1 -- 172.21.15.89:0/3547876409 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c004960 con 0x7f715415d9d0 2024-04-02T08:53:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.624+0000 7f714b400700 1 -- 172.21.15.89:0/3547876409 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c01c680 con 0x7f715415d9d0 2024-04-02T08:53:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.624+0000 7f715a600700 1 -- 172.21.15.89:0/3547876409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71540cb5c0 con 0x7f715415d9d0 2024-04-02T08:53:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.624+0000 7f715a600700 1 -- 172.21.15.89:0/3547876409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715415f490 con 0x7f715415d9d0 2024-04-02T08:53:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.625+0000 7f715a600700 1 -- 172.21.15.89:0/3547876409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71540d1ca0 con 0x7f715415d9d0 2024-04-02T08:53:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.626+0000 7f714b400700 1 -- 172.21.15.89:0/3547876409 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f714c01c7e0 con 0x7f715415d9d0 2024-04-02T08:53:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.627+0000 7f714b400700 1 --2- 172.21.15.89:0/3547876409 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f713406ade0 0x7f713406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.627+0000 7f714b400700 1 -- 172.21.15.89:0/3547876409 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f714c08aff0 con 0x7f715415d9d0 2024-04-02T08:53:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.627+0000 7f7153400700 1 --2- 172.21.15.89:0/3547876409 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f713406ade0 0x7f713406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.628+0000 7f7153400700 1 --2- 172.21.15.89:0/3547876409 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f713406ade0 0x7f713406d2a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7144007fd0 tx=0x7f7144009040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:22.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.632+0000 7f714b400700 1 -- 172.21.15.89:0/3547876409 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f714c055e80 con 0x7f715415d9d0 2024-04-02T08:53:22.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.880+0000 7f715a600700 1 -- 172.21.15.89:0/3547876409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7154067720 con 0x7f715415d9d0 2024-04-02T08:53:22.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.890+0000 7f714b400700 1 -- 172.21.15.89:0/3547876409 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v85) v1 ==== 174+0+0 (secure 0 0 0) 0x7f714c0594a0 con 0x7f715415d9d0 2024-04-02T08:53:22.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.895+0000 7f715a600700 1 -- 172.21.15.89:0/3547876409 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f713406ade0 msgr2=0x7f713406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:22.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.895+0000 7f715a600700 1 --2- 172.21.15.89:0/3547876409 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f713406ade0 0x7f713406d2a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7144007fd0 tx=0x7f7144009040 comp rx=0 tx=0).stop 2024-04-02T08:53:22.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.895+0000 7f715a600700 1 -- 172.21.15.89:0/3547876409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f715415d9d0 msgr2=0x7f71540cc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:22.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.895+0000 7f715a600700 1 --2- 172.21.15.89:0/3547876409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f715415d9d0 0x7f71540cc9e0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f715408c230 tx=0x7f714c004ad0 comp rx=0 tx=0).stop 2024-04-02T08:53:22.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.896+0000 7f715a600700 1 -- 172.21.15.89:0/3547876409 shutdown_connections 2024-04-02T08:53:22.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.896+0000 7f715a600700 1 --2- 172.21.15.89:0/3547876409 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f713406ade0 0x7f713406d2a0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7144007fd0 tx=0x7f7144009040 comp rx=0 tx=0).stop 2024-04-02T08:53:22.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.896+0000 7f715a600700 1 --2- 172.21.15.89:0/3547876409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f715415d9d0 0x7f71540cc9e0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:22.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.896+0000 7f715a600700 1 --2- 172.21.15.89:0/3547876409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71540d16a0 0x7f71540cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:22.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.896+0000 7f715a600700 1 --2- 172.21.15.89:0/3547876409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71540d2370 0x7f7154159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:22.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.896+0000 7f715a600700 1 -- 172.21.15.89:0/3547876409 >> 172.21.15.89:0/3547876409 conn(0x7f715408aff0 msgr2=0x7f71540b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:22.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.896+0000 7f715a600700 1 -- 172.21.15.89:0/3547876409 shutdown_connections 2024-04-02T08:53:22.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:22.897+0000 7f715a600700 1 -- 172.21.15.89:0/3547876409 wait complete. 2024-04-02T08:53:22.904 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 22 and data pool 23 2024-04-02T08:53:22.928 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:53:23.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.120+0000 7fed84200700 1 Processor -- start 2024-04-02T08:53:23.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.120+0000 7fed84200700 1 -- start start 2024-04-02T08:53:23.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.121+0000 7fed84200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed7c15b5d0 0x7fed7c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.121+0000 7fed84200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed7c0cb3f0 0x7fed7c0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.121+0000 7fed84200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7c15fd50 con 0x7fed7c0ae930 2024-04-02T08:53:23.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.121+0000 7fed84200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed7c15b9b0 con 0x7fed7c15b5d0 2024-04-02T08:53:23.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.121+0000 7fed84200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c15f140 con 0x7fed7c0cb3f0 2024-04-02T08:53:23.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.121+0000 7fed81800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed7c15b5d0 0x7fed7c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.121+0000 7fed81800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed7c15b5d0 0x7fed7c0ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54032/0 (socket says 172.21.15.89:54032) 2024-04-02T08:53:23.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.121+0000 7fed81800700 1 -- 172.21.15.89:0/1809875005 learned_addr learned my addr 172.21.15.89:0/1809875005 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.122+0000 7fed81800700 1 -- 172.21.15.89:0/1809875005 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed7c0cb3f0 msgr2=0x7fed7c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.122+0000 7fed81800700 1 --2- 172.21.15.89:0/1809875005 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed7c0cb3f0 0x7fed7c0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.122+0000 7fed81800700 1 -- 172.21.15.89:0/1809875005 >> v1:172.21.15.113:6789/0 conn(0x7fed7c0ae930 legacy=0x7fed7c140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.122+0000 7fed81800700 1 -- 172.21.15.89:0/1809875005 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c15f420 con 0x7fed7c15b5d0 2024-04-02T08:53:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.122+0000 7fed81800700 1 --2- 172.21.15.89:0/1809875005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c15b5d0 0x7fed7c0ae3f0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fed700098f0 tx=0x7fed70009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=643f2f88cf984607 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.122+0000 7fed7be00700 1 -- 172.21.15.89:0/1809875005 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed70004230 con 0x7fed7c15b5d0 2024-04-02T08:53:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.122+0000 7fed7be00700 1 -- 172.21.15.89:0/1809875005 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed70004390 con 0x7fed7c15b5d0 2024-04-02T08:53:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.122+0000 7fed7be00700 1 -- 172.21.15.89:0/1809875005 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed700036a0 con 0x7fed7c15b5d0 2024-04-02T08:53:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.123+0000 7fed84200700 1 -- 172.21.15.89:0/1809875005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c15b5d0 msgr2=0x7fed7c0ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.123+0000 7fed84200700 1 --2- 172.21.15.89:0/1809875005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c15b5d0 0x7fed7c0ae3f0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fed700098f0 tx=0x7fed70009c00 comp rx=0 tx=0).stop 2024-04-02T08:53:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.123+0000 7fed84200700 1 -- 172.21.15.89:0/1809875005 shutdown_connections 2024-04-02T08:53:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.123+0000 7fed84200700 1 --2- 172.21.15.89:0/1809875005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c15b5d0 0x7fed7c0ae3f0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.123+0000 7fed84200700 1 --2- 172.21.15.89:0/1809875005 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed7c0cb3f0 0x7fed7c0ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.123+0000 7fed84200700 1 -- 172.21.15.89:0/1809875005 >> 172.21.15.89:0/1809875005 conn(0x7fed7c0cf190 msgr2=0x7fed7c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:23.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.123+0000 7fed84200700 1 -- 172.21.15.89:0/1809875005 shutdown_connections 2024-04-02T08:53:23.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.123+0000 7fed84200700 1 -- 172.21.15.89:0/1809875005 wait complete. 2024-04-02T08:53:23.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.125+0000 7fed84200700 1 Processor -- start 2024-04-02T08:53:23.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.125+0000 7fed84200700 1 -- start start 2024-04-02T08:53:23.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed84200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed7c0ae930 0x7fed7c1f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed84200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed7c15b5d0 0x7fed7c0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed84200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c09cb60 0x7fed7c158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed84200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c0cca90 con 0x7fed7c09cb60 2024-04-02T08:53:23.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed84200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c0ce9e0 con 0x7fed7c15b5d0 2024-04-02T08:53:23.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed84200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c0cbe80 con 0x7fed7c0ae930 2024-04-02T08:53:23.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed82200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c09cb60 0x7fed7c158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed82200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c09cb60 0x7fed7c158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54036/0 (socket says 172.21.15.89:54036) 2024-04-02T08:53:23.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed82200700 1 -- 172.21.15.89:0/3666372780 learned_addr learned my addr 172.21.15.89:0/3666372780 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:23.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed81800700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed7c0ae930 0x7fed7c1f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed80e00700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed7c15b5d0 0x7fed7c0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed82200700 1 -- 172.21.15.89:0/3666372780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed7c0ae930 msgr2=0x7fed7c1f3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed82200700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed7c0ae930 0x7fed7c1f3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed82200700 1 -- 172.21.15.89:0/3666372780 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed7c15b5d0 msgr2=0x7fed7c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed82200700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed7c15b5d0 0x7fed7c0ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed82200700 1 -- 172.21.15.89:0/3666372780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed70009550 con 0x7fed7c09cb60 2024-04-02T08:53:23.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed81800700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed7c0ae930 0x7fed7c1f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:23.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.126+0000 7fed80e00700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed7c15b5d0 0x7fed7c0ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:23.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.127+0000 7fed82200700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c09cb60 0x7fed7c158f40 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fed7400c780 tx=0x7fed7400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:23.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.127+0000 7fed79600700 1 -- 172.21.15.89:0/3666372780 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74003e10 con 0x7fed7c09cb60 2024-04-02T08:53:23.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.127+0000 7fed79600700 1 -- 172.21.15.89:0/3666372780 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed74005070 con 0x7fed7c09cb60 2024-04-02T08:53:23.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.127+0000 7fed84200700 1 -- 172.21.15.89:0/3666372780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7c0cc160 con 0x7fed7c09cb60 2024-04-02T08:53:23.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.127+0000 7fed79600700 1 -- 172.21.15.89:0/3666372780 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74005420 con 0x7fed7c09cb60 2024-04-02T08:53:23.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.127+0000 7fed84200700 1 -- 172.21.15.89:0/3666372780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7c0cc720 con 0x7fed7c09cb60 2024-04-02T08:53:23.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.128+0000 7fed84200700 1 -- 172.21.15.89:0/3666372780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7c067720 con 0x7fed7c09cb60 2024-04-02T08:53:23.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.129+0000 7fed79600700 1 -- 172.21.15.89:0/3666372780 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fed74006ba0 con 0x7fed7c09cb60 2024-04-02T08:53:23.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.130+0000 7fed79600700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fed5c06adb0 0x7fed5c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.130+0000 7fed79600700 1 -- 172.21.15.89:0/3666372780 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fed74088930 con 0x7fed7c09cb60 2024-04-02T08:53:23.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.130+0000 7fed81800700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fed5c06adb0 0x7fed5c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.131+0000 7fed81800700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fed5c06adb0 0x7fed5c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fed70009130 tx=0x7fed70008030 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:23.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.134+0000 7fed79600700 1 -- 172.21.15.89:0/3666372780 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fed74053850 con 0x7fed7c09cb60 2024-04-02T08:53:23.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.334+0000 7fed84200700 1 -- 172.21.15.89:0/3666372780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed7c0ccbd0 con 0x7fed7c09cb60 2024-04-02T08:53:23.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.335+0000 7fed79600700 1 -- 172.21.15.89:0/3666372780 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+16767 (secure 0 0 0) 0x7fed74017020 con 0x7fed7c09cb60 2024-04-02T08:53:23.334 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:23.334 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":185,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:53:22.881246+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-02T08:53:20.736248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-02T08:53:21.860376+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:53:23.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.338+0000 7fed84200700 1 -- 172.21.15.89:0/3666372780 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fed5c06adb0 msgr2=0x7fed5c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.338+0000 7fed84200700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fed5c06adb0 0x7fed5c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fed70009130 tx=0x7fed70008030 comp rx=0 tx=0).stop 2024-04-02T08:53:23.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.338+0000 7fed84200700 1 -- 172.21.15.89:0/3666372780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c09cb60 msgr2=0x7fed7c158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.338+0000 7fed84200700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c09cb60 0x7fed7c158f40 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fed7400c780 tx=0x7fed7400cb40 comp rx=0 tx=0).stop 2024-04-02T08:53:23.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.338+0000 7fed84200700 1 -- 172.21.15.89:0/3666372780 shutdown_connections 2024-04-02T08:53:23.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.338+0000 7fed84200700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fed5c06adb0 0x7fed5c06d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.338+0000 7fed84200700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c09cb60 0x7fed7c158f40 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.338+0000 7fed84200700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed7c0ae930 0x7fed7c1f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.338+0000 7fed84200700 1 --2- 172.21.15.89:0/3666372780 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed7c15b5d0 0x7fed7c0ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.338+0000 7fed84200700 1 -- 172.21.15.89:0/3666372780 >> 172.21.15.89:0/3666372780 conn(0x7fed7c0cf190 msgr2=0x7fed7c15e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:23.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.338+0000 7fed84200700 1 -- 172.21.15.89:0/3666372780 shutdown_connections 2024-04-02T08:53:23.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.338+0000 7fed84200700 1 -- 172.21.15.89:0/3666372780 wait complete. 2024-04-02T08:53:23.355 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:53:23.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.547+0000 7f1bcac00700 1 Processor -- start 2024-04-02T08:53:23.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.548+0000 7f1bcac00700 1 -- start start 2024-04-02T08:53:23.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.549+0000 7f1bcac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bc415d640 0x7f1bc40d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.549+0000 7f1bcac00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1bc40ceec0 0x7f1bc41612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.549+0000 7f1bcac00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc40d30d0 con 0x7f1bc4161800 2024-04-02T08:53:23.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.549+0000 7f1bcac00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bc4161c90 con 0x7f1bc40ceec0 2024-04-02T08:53:23.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.549+0000 7f1bcac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40b6840 con 0x7f1bc415d640 2024-04-02T08:53:23.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.549+0000 7f1bc3400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1bc40ceec0 0x7f1bc41612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.549+0000 7f1bc3e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bc415d640 0x7f1bc40d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.549+0000 7f1bc8c00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f1bc4161800 0x7f1bc40cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:46950/0 (socket says 172.21.15.89:46950) 2024-04-02T08:53:23.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.549+0000 7f1bc8c00700 1 -- 172.21.15.89:0/1478810740 learned_addr learned my addr 172.21.15.89:0/1478810740 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:23.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.550+0000 7f1bc2a00700 1 -- 172.21.15.89:0/1478810740 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318325682 0 0) 0x7f1bc40d30d0 con 0x7f1bc4161800 2024-04-02T08:53:23.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.550+0000 7f1bc2a00700 1 -- 172.21.15.89:0/1478810740 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba80035b0 con 0x7f1bc4161800 2024-04-02T08:53:23.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.550+0000 7f1bc3400700 1 -- 172.21.15.89:0/1478810740 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bc415d640 msgr2=0x7f1bc40d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:23.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.550+0000 7f1bc3400700 1 --2- 172.21.15.89:0/1478810740 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bc415d640 0x7f1bc40d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.550+0000 7f1bc3400700 1 -- 172.21.15.89:0/1478810740 >> v1:172.21.15.113:6789/0 conn(0x7f1bc4161800 legacy=0x7f1bc40cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.550+0000 7f1bc3400700 1 -- 172.21.15.89:0/1478810740 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc40d2880 con 0x7f1bc40ceec0 2024-04-02T08:53:23.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.550+0000 7f1bc3e00700 1 --2- 172.21.15.89:0/1478810740 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bc415d640 0x7f1bc40d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:23.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.550+0000 7f1bc3400700 1 --2- 172.21.15.89:0/1478810740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bc40ceec0 0x7f1bc41612c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0009840 tx=0x7f1bb0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afe38658c5872256 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:23.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.550+0000 7f1bc2a00700 1 -- 172.21.15.89:0/1478810740 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0011830 con 0x7f1bc40ceec0 2024-04-02T08:53:23.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.550+0000 7f1bc2a00700 1 -- 172.21.15.89:0/1478810740 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb0011990 con 0x7f1bc40ceec0 2024-04-02T08:53:23.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.550+0000 7f1bc2a00700 1 -- 172.21.15.89:0/1478810740 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0011d20 con 0x7f1bc40ceec0 2024-04-02T08:53:23.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.551+0000 7f1bcac00700 1 -- 172.21.15.89:0/1478810740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bc40ceec0 msgr2=0x7f1bc41612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.551+0000 7f1bcac00700 1 --2- 172.21.15.89:0/1478810740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bc40ceec0 0x7f1bc41612c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0009840 tx=0x7f1bb0009b50 comp rx=0 tx=0).stop 2024-04-02T08:53:23.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.551+0000 7f1bcac00700 1 -- 172.21.15.89:0/1478810740 shutdown_connections 2024-04-02T08:53:23.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.551+0000 7f1bcac00700 1 --2- 172.21.15.89:0/1478810740 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bc415d640 0x7f1bc40d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.551+0000 7f1bcac00700 1 --2- 172.21.15.89:0/1478810740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bc40ceec0 0x7f1bc41612c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.551+0000 7f1bcac00700 1 -- 172.21.15.89:0/1478810740 >> 172.21.15.89:0/1478810740 conn(0x7f1bc408aff0 msgr2=0x7f1bc40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.551+0000 7f1bcac00700 1 -- 172.21.15.89:0/1478810740 shutdown_connections 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.551+0000 7f1bcac00700 1 -- 172.21.15.89:0/1478810740 wait complete. 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.553+0000 7f1bcac00700 1 Processor -- start 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.553+0000 7f1bcac00700 1 -- start start 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.554+0000 7f1bcac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bc40ceec0 0x7f1bc4152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.554+0000 7f1bcac00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc415d640 0x7f1bc41f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.554+0000 7f1bcac00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bc4161800 0x7f1bc4143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.554+0000 7f1bcac00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc415f920 con 0x7f1bc415d640 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.554+0000 7f1bcac00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc415ed30 con 0x7f1bc4161800 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.554+0000 7f1bcac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc415fd40 con 0x7f1bc40ceec0 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.554+0000 7f1bc3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc415d640 0x7f1bc41f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.554+0000 7f1bc3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc415d640 0x7f1bc41f2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54042/0 (socket says 172.21.15.89:54042) 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.554+0000 7f1bc3400700 1 -- 172.21.15.89:0/181894023 learned_addr learned my addr 172.21.15.89:0/181894023 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.554+0000 7f1bc8c00700 1 --2- 172.21.15.89:0/181894023 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bc4161800 0x7f1bc4143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.554+0000 7f1bc3e00700 1 --2- 172.21.15.89:0/181894023 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bc40ceec0 0x7f1bc4152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.555+0000 7f1bc3400700 1 -- 172.21.15.89:0/181894023 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bc40ceec0 msgr2=0x7f1bc4152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.555+0000 7f1bc3400700 1 --2- 172.21.15.89:0/181894023 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bc40ceec0 0x7f1bc4152730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.555+0000 7f1bc3400700 1 -- 172.21.15.89:0/181894023 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bc4161800 msgr2=0x7f1bc4143070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.555+0000 7f1bc3400700 1 --2- 172.21.15.89:0/181894023 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bc4161800 0x7f1bc4143070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.555+0000 7f1bc3400700 1 -- 172.21.15.89:0/181894023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb40094a0 con 0x7f1bc415d640 2024-04-02T08:53:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.555+0000 7f1bc3400700 1 --2- 172.21.15.89:0/181894023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc415d640 0x7f1bc41f2710 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0009fd0 tx=0x7f1bb0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.555+0000 7f1bbbe00700 1 -- 172.21.15.89:0/181894023 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0004590 con 0x7f1bc415d640 2024-04-02T08:53:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.555+0000 7f1bbbe00700 1 -- 172.21.15.89:0/181894023 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb001e070 con 0x7f1bc415d640 2024-04-02T08:53:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.555+0000 7f1bbbe00700 1 -- 172.21.15.89:0/181894023 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb00038e0 con 0x7f1bc415d640 2024-04-02T08:53:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.555+0000 7f1bcac00700 1 -- 172.21.15.89:0/181894023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb00094a0 con 0x7f1bc415d640 2024-04-02T08:53:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.555+0000 7f1bcac00700 1 -- 172.21.15.89:0/181894023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc415f260 con 0x7f1bc415d640 2024-04-02T08:53:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.556+0000 7f1bcac00700 1 -- 172.21.15.89:0/181894023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc415fa60 con 0x7f1bc415d640 2024-04-02T08:53:23.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.557+0000 7f1bbbe00700 1 -- 172.21.15.89:0/181894023 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1bb0003a40 con 0x7f1bc415d640 2024-04-02T08:53:23.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.559+0000 7f1bbbe00700 1 --2- 172.21.15.89:0/181894023 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1ba406adb0 0x7f1ba406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.559+0000 7f1bbbe00700 1 -- 172.21.15.89:0/181894023 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f1bb0020070 con 0x7f1bc415d640 2024-04-02T08:53:23.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.559+0000 7f1bc3e00700 1 --2- 172.21.15.89:0/181894023 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1ba406adb0 0x7f1ba406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.559+0000 7f1bc3e00700 1 --2- 172.21.15.89:0/181894023 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1ba406adb0 0x7f1ba406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4009d70 tx=0x7f1bb4009210 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:23.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.563+0000 7f1bbbe00700 1 -- 172.21.15.89:0/181894023 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1bc415fa60 con 0x7f1bc415d640 2024-04-02T08:53:23.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.763+0000 7f1bcac00700 1 -- 172.21.15.89:0/181894023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bc415fa60 con 0x7f1bc415d640 2024-04-02T08:53:23.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.765+0000 7f1bbbe00700 1 -- 172.21.15.89:0/181894023 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+16767 (secure 0 0 0) 0x7f1bc415fa60 con 0x7f1bc415d640 2024-04-02T08:53:23.763 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:23.764 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":185,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:53:22.881246+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-02T08:53:20.736248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-02T08:53:21.860376+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:53:23.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.767+0000 7f1bcac00700 1 -- 172.21.15.89:0/181894023 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1ba406adb0 msgr2=0x7f1ba406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.767+0000 7f1bcac00700 1 --2- 172.21.15.89:0/181894023 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1ba406adb0 0x7f1ba406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4009d70 tx=0x7f1bb4009210 comp rx=0 tx=0).stop 2024-04-02T08:53:23.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.767+0000 7f1bcac00700 1 -- 172.21.15.89:0/181894023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc415d640 msgr2=0x7f1bc41f2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.767+0000 7f1bcac00700 1 --2- 172.21.15.89:0/181894023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc415d640 0x7f1bc41f2710 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0009fd0 tx=0x7f1bb0004750 comp rx=0 tx=0).stop 2024-04-02T08:53:23.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.767+0000 7f1bcac00700 1 -- 172.21.15.89:0/181894023 shutdown_connections 2024-04-02T08:53:23.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.767+0000 7f1bcac00700 1 --2- 172.21.15.89:0/181894023 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f1ba406adb0 0x7f1ba406d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.767+0000 7f1bcac00700 1 --2- 172.21.15.89:0/181894023 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bc4161800 0x7f1bc4143070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.767+0000 7f1bcac00700 1 --2- 172.21.15.89:0/181894023 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bc40ceec0 0x7f1bc4152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.767+0000 7f1bcac00700 1 --2- 172.21.15.89:0/181894023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc415d640 0x7f1bc41f2710 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.767+0000 7f1bcac00700 1 -- 172.21.15.89:0/181894023 >> 172.21.15.89:0/181894023 conn(0x7f1bc408aff0 msgr2=0x7f1bc40b5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:23.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.767+0000 7f1bcac00700 1 -- 172.21.15.89:0/181894023 shutdown_connections 2024-04-02T08:53:23.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.767+0000 7f1bcac00700 1 -- 172.21.15.89:0/181894023 wait complete. 2024-04-02T08:53:23.784 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T08:53:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.983+0000 7f89ea000700 1 Processor -- start 2024-04-02T08:53:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.983+0000 7f89ea000700 1 -- start start 2024-04-02T08:53:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.984+0000 7f89ea000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f89e40d3eb0 0x7f89e40b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.984+0000 7f89ea000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f89e40cf1b0 0x7f89e40cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.984+0000 7f89ea000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e40ced20 con 0x7f89e415d8a0 2024-04-02T08:53:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.984+0000 7f89ea000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e40d4340 con 0x7f89e40d3eb0 2024-04-02T08:53:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.984+0000 7f89ea000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e40ce6d0 con 0x7f89e40cf1b0 2024-04-02T08:53:23.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.984+0000 7f89e2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f89e40d3eb0 0x7f89e40b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.984+0000 7f89e3400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f89e415d8a0 0x7f89e406b890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46362/0 (socket says 172.21.15.89:46362) 2024-04-02T08:53:23.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.984+0000 7f89e3400700 1 -- 172.21.15.89:0/328231349 learned_addr learned my addr 172.21.15.89:0/328231349 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:23.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.984+0000 7f89e3e00700 1 --2- 172.21.15.89:0/328231349 >> v2:172.21.15.113:3300/0 conn(0x7f89e40cf1b0 0x7f89e40cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.985+0000 7f89e2a00700 1 -- 172.21.15.89:0/328231349 >> v2:172.21.15.113:3300/0 conn(0x7f89e40cf1b0 msgr2=0x7f89e40cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:23.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.985+0000 7f89e2a00700 1 --2- 172.21.15.89:0/328231349 >> v2:172.21.15.113:3300/0 conn(0x7f89e40cf1b0 0x7f89e40cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.985+0000 7f89e2000700 1 -- 172.21.15.89:0/328231349 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177369920 0 0) 0x7f89e40ced20 con 0x7f89e415d8a0 2024-04-02T08:53:23.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.985+0000 7f89e2a00700 1 -- 172.21.15.89:0/328231349 >> v1:172.21.15.89:6789/0 conn(0x7f89e415d8a0 legacy=0x7f89e406b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.985+0000 7f89e2a00700 1 -- 172.21.15.89:0/328231349 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e40ce9b0 con 0x7f89e40d3eb0 2024-04-02T08:53:23.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.985+0000 7f89e2a00700 1 --2- 172.21.15.89:0/328231349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89e40d3eb0 0x7f89e40b2990 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f89e40b2940 tx=0x7f89d0009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c9e42b121eaee0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:23.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.985+0000 7f89e2000700 1 -- 172.21.15.89:0/328231349 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d0005e30 con 0x7f89e40d3eb0 2024-04-02T08:53:23.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.985+0000 7f89e2000700 1 -- 172.21.15.89:0/328231349 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d0006030 con 0x7f89e40d3eb0 2024-04-02T08:53:23.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.985+0000 7f89e2000700 1 -- 172.21.15.89:0/328231349 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d00063c0 con 0x7f89e40d3eb0 2024-04-02T08:53:23.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.986+0000 7f89ea000700 1 -- 172.21.15.89:0/328231349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89e40d3eb0 msgr2=0x7f89e40b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.986+0000 7f89ea000700 1 --2- 172.21.15.89:0/328231349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89e40d3eb0 0x7f89e40b2990 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f89e40b2940 tx=0x7f89d0009b00 comp rx=0 tx=0).stop 2024-04-02T08:53:23.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.986+0000 7f89ea000700 1 -- 172.21.15.89:0/328231349 shutdown_connections 2024-04-02T08:53:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.986+0000 7f89ea000700 1 --2- 172.21.15.89:0/328231349 >> v2:172.21.15.113:3300/0 conn(0x7f89e40cf1b0 0x7f89e40cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.986+0000 7f89ea000700 1 --2- 172.21.15.89:0/328231349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89e40d3eb0 0x7f89e40b2990 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.986+0000 7f89ea000700 1 -- 172.21.15.89:0/328231349 >> 172.21.15.89:0/328231349 conn(0x7f89e408aff0 msgr2=0x7f89e40d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.986+0000 7f89ea000700 1 -- 172.21.15.89:0/328231349 shutdown_connections 2024-04-02T08:53:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.986+0000 7f89ea000700 1 -- 172.21.15.89:0/328231349 wait complete. 2024-04-02T08:53:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.988+0000 7f89ea000700 1 Processor -- start 2024-04-02T08:53:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.988+0000 7f89ea000700 1 -- start start 2024-04-02T08:53:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89ea000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89e40cf1b0 0x7f89e415ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89ea000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89e415d8a0 0x7f89e41f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89ea000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89e40b4060 0x7f89e40c8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89ea000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40cb2e0 con 0x7f89e40cf1b0 2024-04-02T08:53:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89ea000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40cb700 con 0x7f89e415d8a0 2024-04-02T08:53:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89ea000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e415bf80 con 0x7f89e40b4060 2024-04-02T08:53:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89e3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89e40cf1b0 0x7f89e415ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89e3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89e40cf1b0 0x7f89e415ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54048/0 (socket says 172.21.15.89:54048) 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89e3400700 1 -- 172.21.15.89:0/3087284316 learned_addr learned my addr 172.21.15.89:0/3087284316 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89e3e00700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89e40b4060 0x7f89e40c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89e2a00700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89e415d8a0 0x7f89e41f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89e3400700 1 -- 172.21.15.89:0/3087284316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89e40b4060 msgr2=0x7f89e40c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89e3400700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89e40b4060 0x7f89e40c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89e3400700 1 -- 172.21.15.89:0/3087284316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89e415d8a0 msgr2=0x7f89e41f37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89e3400700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89e415d8a0 0x7f89e41f37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89e3400700 1 -- 172.21.15.89:0/3087284316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d0009430 con 0x7f89e40cf1b0 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.989+0000 7f89e2a00700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89e415d8a0 0x7f89e41f37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.990+0000 7f89e3400700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89e40cf1b0 0x7f89e415ba40 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f89d400c550 tx=0x7f89d400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.990+0000 7f89db400700 1 -- 172.21.15.89:0/3087284316 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d4004c80 con 0x7f89e40cf1b0 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.990+0000 7f89db400700 1 -- 172.21.15.89:0/3087284316 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d4004e00 con 0x7f89e40cf1b0 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.990+0000 7f89ea000700 1 -- 172.21.15.89:0/3087284316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e40ca660 con 0x7f89e40cf1b0 2024-04-02T08:53:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.990+0000 7f89db400700 1 -- 172.21.15.89:0/3087284316 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d4007310 con 0x7f89e40cf1b0 2024-04-02T08:53:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.990+0000 7f89ea000700 1 -- 172.21.15.89:0/3087284316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e40cac20 con 0x7f89e40cf1b0 2024-04-02T08:53:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.991+0000 7f89ea000700 1 -- 172.21.15.89:0/3087284316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e40cb550 con 0x7f89e40cf1b0 2024-04-02T08:53:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.992+0000 7f89db400700 1 -- 172.21.15.89:0/3087284316 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f89d4007470 con 0x7f89e40cf1b0 2024-04-02T08:53:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.993+0000 7f89db400700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f89c406adb0 0x7f89c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.993+0000 7f89db400700 1 -- 172.21.15.89:0/3087284316 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f89d4089470 con 0x7f89e40cf1b0 2024-04-02T08:53:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.993+0000 7f89e2a00700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f89c406adb0 0x7f89c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.993+0000 7f89e2a00700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f89c406adb0 0x7f89c406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f89e415dfe0 tx=0x7f89d0008070 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:23.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:23.998+0000 7f89db400700 1 -- 172.21.15.89:0/3087284316 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f89d40542f0 con 0x7f89e40cf1b0 2024-04-02T08:53:24.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:24.246+0000 7f89ea000700 1 -- 172.21.15.89:0/3087284316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f89e4067720 con 0x7f89e40cf1b0 2024-04-02T08:53:24.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:24.919+0000 7f89db400700 1 -- 172.21.15.89:0/3087284316 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v88) v1 ==== 118+0+0 (secure 0 0 0) 0x7f89d4057910 con 0x7f89e40cf1b0 2024-04-02T08:53:24.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:24.922+0000 7f89ea000700 1 -- 172.21.15.89:0/3087284316 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f89c406adb0 msgr2=0x7f89c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:24.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:24.922+0000 7f89ea000700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f89c406adb0 0x7f89c406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f89e415dfe0 tx=0x7f89d0008070 comp rx=0 tx=0).stop 2024-04-02T08:53:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:24.922+0000 7f89ea000700 1 -- 172.21.15.89:0/3087284316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89e40cf1b0 msgr2=0x7f89e415ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:24.922+0000 7f89ea000700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89e40cf1b0 0x7f89e415ba40 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f89d400c550 tx=0x7f89d400c910 comp rx=0 tx=0).stop 2024-04-02T08:53:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:24.922+0000 7f89ea000700 1 -- 172.21.15.89:0/3087284316 shutdown_connections 2024-04-02T08:53:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:24.922+0000 7f89ea000700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f89c406adb0 0x7f89c406d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:24.922+0000 7f89ea000700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f89e40b4060 0x7f89e40c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:24.922+0000 7f89ea000700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89e40cf1b0 0x7f89e415ba40 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:24.922+0000 7f89ea000700 1 --2- 172.21.15.89:0/3087284316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89e415d8a0 0x7f89e41f37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:24.922+0000 7f89ea000700 1 -- 172.21.15.89:0/3087284316 >> 172.21.15.89:0/3087284316 conn(0x7f89e408aff0 msgr2=0x7f89e40b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:24.923+0000 7f89ea000700 1 -- 172.21.15.89:0/3087284316 shutdown_connections 2024-04-02T08:53:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:24.923+0000 7f89ea000700 1 -- 172.21.15.89:0/3087284316 wait complete. 2024-04-02T08:53:24.942 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T08:53:24.942 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T08:53:25.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.142+0000 7f785da00700 1 Processor -- start 2024-04-02T08:53:25.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.142+0000 7f785da00700 1 -- start start 2024-04-02T08:53:25.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.143+0000 7f785da00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f78580cee10 0x7f7858161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:25.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.143+0000 7f785da00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78580ce270 con 0x7f78580d21f0 2024-04-02T08:53:25.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.144+0000 7f785da00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785815b600 con 0x7f78581611b0 2024-04-02T08:53:25.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.144+0000 7f785da00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78580d2760 con 0x7f78580cee10 2024-04-02T08:53:25.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.144+0000 7f7856000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f78580d21f0 0x7f78580ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46370/0 (socket says 172.21.15.89:46370) 2024-04-02T08:53:25.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.144+0000 7f7856000700 1 -- 172.21.15.89:0/1003533641 learned_addr learned my addr 172.21.15.89:0/1003533641 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:25.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.144+0000 7f7857400700 1 --2- 172.21.15.89:0/1003533641 >> v2:172.21.15.113:3300/0 conn(0x7f78580cee10 0x7f7858161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:25.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.144+0000 7f7855600700 1 -- 172.21.15.89:0/1003533641 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873024125 0 0) 0x7f78580ce270 con 0x7f78580d21f0 2024-04-02T08:53:25.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.144+0000 7f7855600700 1 -- 172.21.15.89:0/1003533641 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f783c0035b0 con 0x7f78580d21f0 2024-04-02T08:53:25.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.145+0000 7f7855600700 1 -- 172.21.15.89:0/1003533641 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3515658196 0 0) 0x7f783c0035b0 con 0x7f78580d21f0 2024-04-02T08:53:25.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.145+0000 7f7857400700 1 -- 172.21.15.89:0/1003533641 >> v1:172.21.15.113:6789/0 conn(0x7f78581611b0 legacy=0x7f785815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:25.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.145+0000 7f7857400700 1 -- 172.21.15.89:0/1003533641 >> v1:172.21.15.89:6789/0 conn(0x7f78580d21f0 legacy=0x7f78580ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:25.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.145+0000 7f7857400700 1 -- 172.21.15.89:0/1003533641 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78580d2a40 con 0x7f78580cee10 2024-04-02T08:53:25.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.145+0000 7f7857400700 1 --2- 172.21.15.89:0/1003533641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78580cee10 0x7f7858161590 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f7850009d90 tx=0x7f785000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dded1104f63fee93 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:25.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.145+0000 7f7855600700 1 -- 172.21.15.89:0/1003533641 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785000e070 con 0x7f78580cee10 2024-04-02T08:53:25.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.145+0000 7f7855600700 1 -- 172.21.15.89:0/1003533641 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7850003a20 con 0x7f78580cee10 2024-04-02T08:53:25.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.145+0000 7f7855600700 1 -- 172.21.15.89:0/1003533641 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7844001670 con 0x7f78580cee10 2024-04-02T08:53:25.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.146+0000 7f785da00700 1 -- 172.21.15.89:0/1003533641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78580cee10 msgr2=0x7f7858161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:25.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.146+0000 7f785da00700 1 --2- 172.21.15.89:0/1003533641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78580cee10 0x7f7858161590 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f7850009d90 tx=0x7f785000b330 comp rx=0 tx=0).stop 2024-04-02T08:53:25.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.146+0000 7f785da00700 1 -- 172.21.15.89:0/1003533641 shutdown_connections 2024-04-02T08:53:25.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.146+0000 7f785da00700 1 --2- 172.21.15.89:0/1003533641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78580cee10 0x7f7858161590 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:25.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.146+0000 7f785da00700 1 -- 172.21.15.89:0/1003533641 >> 172.21.15.89:0/1003533641 conn(0x7f785808aff0 msgr2=0x7f78580d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:25.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.146+0000 7f785da00700 1 -- 172.21.15.89:0/1003533641 shutdown_connections 2024-04-02T08:53:25.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.146+0000 7f785da00700 1 -- 172.21.15.89:0/1003533641 wait complete. 2024-04-02T08:53:25.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.147+0000 7f785da00700 1 Processor -- start 2024-04-02T08:53:25.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.148+0000 7f785da00700 1 -- start start 2024-04-02T08:53:25.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.148+0000 7f785da00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78580cee10 0x7f785815aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:25.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.148+0000 7f785da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78580d21f0 0x7f78580b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:25.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.148+0000 7f785da00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78581611b0 0x7f78580b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:25.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f785da00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785815af80 con 0x7f78580d21f0 2024-04-02T08:53:25.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f785da00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78580b7080 con 0x7f78580cee10 2024-04-02T08:53:25.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f785da00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78580b7360 con 0x7f78581611b0 2024-04-02T08:53:25.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f7856000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78580d21f0 0x7f78580b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:25.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f7856000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78580d21f0 0x7f78580b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54056/0 (socket says 172.21.15.89:54056) 2024-04-02T08:53:25.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f7856000700 1 -- 172.21.15.89:0/1733620545 learned_addr learned my addr 172.21.15.89:0/1733620545 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:25.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f7857400700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78581611b0 0x7f78580b8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:25.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f7856a00700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78580cee10 0x7f785815aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:25.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f7856000700 1 -- 172.21.15.89:0/1733620545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78581611b0 msgr2=0x7f78580b8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:25.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f7856000700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78581611b0 0x7f78580b8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:25.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f7856000700 1 -- 172.21.15.89:0/1733620545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78580cee10 msgr2=0x7f785815aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:25.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f7856000700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78580cee10 0x7f785815aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:25.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f7856000700 1 -- 172.21.15.89:0/1733620545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7850009a70 con 0x7f78580d21f0 2024-04-02T08:53:25.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f7856a00700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78580cee10 0x7f785815aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:25.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.149+0000 7f7856000700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78580d21f0 0x7f78580b84f0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f78440011e0 tx=0x7f784400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:25.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.150+0000 7f784ea00700 1 -- 172.21.15.89:0/1733620545 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7844007560 con 0x7f78580d21f0 2024-04-02T08:53:25.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.150+0000 7f784ea00700 1 -- 172.21.15.89:0/1733620545 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78440076c0 con 0x7f78580d21f0 2024-04-02T08:53:25.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.150+0000 7f785da00700 1 -- 172.21.15.89:0/1733620545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78580b7640 con 0x7f78580d21f0 2024-04-02T08:53:25.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.150+0000 7f784ea00700 1 -- 172.21.15.89:0/1733620545 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7844004a90 con 0x7f78580d21f0 2024-04-02T08:53:25.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.150+0000 7f785da00700 1 -- 172.21.15.89:0/1733620545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78580b6310 con 0x7f78580d21f0 2024-04-02T08:53:25.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.151+0000 7f785da00700 1 -- 172.21.15.89:0/1733620545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7858067720 con 0x7f78580d21f0 2024-04-02T08:53:25.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.152+0000 7f784ea00700 1 -- 172.21.15.89:0/1733620545 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7844004bf0 con 0x7f78580d21f0 2024-04-02T08:53:25.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.153+0000 7f784ea00700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f783806adb0 0x7f783806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:25.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.153+0000 7f784ea00700 1 -- 172.21.15.89:0/1733620545 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f7844088b00 con 0x7f78580d21f0 2024-04-02T08:53:25.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.153+0000 7f7856a00700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f783806adb0 0x7f783806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:25.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.154+0000 7f7856a00700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f783806adb0 0x7f783806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7848000f60 tx=0x7f7848009040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:25.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.157+0000 7f784ea00700 1 -- 172.21.15.89:0/1733620545 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f784408ae60 con 0x7f78580d21f0 2024-04-02T08:53:25.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:25.406+0000 7f785da00700 1 -- 172.21.15.89:0/1733620545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f78580b5d00 con 0x7f78580d21f0 2024-04-02T08:53:26.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.397+0000 7f784ea00700 1 -- 172.21.15.89:0/1733620545 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v89) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7844018020 con 0x7f78580d21f0 2024-04-02T08:53:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.400+0000 7f785da00700 1 -- 172.21.15.89:0/1733620545 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f783806adb0 msgr2=0x7f783806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.400+0000 7f785da00700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f783806adb0 0x7f783806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7848000f60 tx=0x7f7848009040 comp rx=0 tx=0).stop 2024-04-02T08:53:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.400+0000 7f785da00700 1 -- 172.21.15.89:0/1733620545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78580d21f0 msgr2=0x7f78580b84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.400+0000 7f785da00700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78580d21f0 0x7f78580b84f0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f78440011e0 tx=0x7f784400cb00 comp rx=0 tx=0).stop 2024-04-02T08:53:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.400+0000 7f785da00700 1 -- 172.21.15.89:0/1733620545 shutdown_connections 2024-04-02T08:53:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.400+0000 7f785da00700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f783806adb0 0x7f783806d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.400+0000 7f785da00700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78580cee10 0x7f785815aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:26.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.400+0000 7f785da00700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f78581611b0 0x7f78580b8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:26.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.400+0000 7f785da00700 1 --2- 172.21.15.89:0/1733620545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78580d21f0 0x7f78580b84f0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:26.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.400+0000 7f785da00700 1 -- 172.21.15.89:0/1733620545 >> 172.21.15.89:0/1733620545 conn(0x7f785808aff0 msgr2=0x7f7858160250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:26.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.401+0000 7f785da00700 1 -- 172.21.15.89:0/1733620545 shutdown_connections 2024-04-02T08:53:26.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.401+0000 7f785da00700 1 -- 172.21.15.89:0/1733620545 wait complete. 2024-04-02T08:53:26.420 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T08:53:26.421 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T08:53:26.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.616+0000 7fcfa5200700 1 Processor -- start 2024-04-02T08:53:26.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.616+0000 7fcfa5200700 1 -- start start 2024-04-02T08:53:26.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.617+0000 7fcfa5200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcfa00d18b0 0x7fcfa00cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:26.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.617+0000 7fcfa5200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa00ceda0 con 0x7fcfa0160f10 2024-04-02T08:53:26.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.617+0000 7fcfa5200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa01612f0 con 0x7fcfa0161590 2024-04-02T08:53:26.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.617+0000 7fcfa5200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa00d1d30 con 0x7fcfa00d18b0 2024-04-02T08:53:26.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.617+0000 7fcf9d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcfa00d18b0 0x7fcfa00cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:26.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.617+0000 7fcf9de00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcfa0160f10 0x7fcfa00d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46374/0 (socket says 172.21.15.89:46374) 2024-04-02T08:53:26.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.617+0000 7fcf9de00700 1 -- 172.21.15.89:0/4217931255 learned_addr learned my addr 172.21.15.89:0/4217931255 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:26.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.618+0000 7fcf9d400700 1 -- 172.21.15.89:0/4217931255 >> v1:172.21.15.113:6789/0 conn(0x7fcfa0161590 legacy=0x7fcfa00b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:26.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.618+0000 7fcf9d400700 1 -- 172.21.15.89:0/4217931255 >> v1:172.21.15.89:6789/0 conn(0x7fcfa0160f10 legacy=0x7fcfa00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:26.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.618+0000 7fcf9ca00700 1 -- 172.21.15.89:0/4217931255 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843152253 0 0) 0x7fcfa00ceda0 con 0x7fcfa0160f10 2024-04-02T08:53:26.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.618+0000 7fcf9d400700 1 -- 172.21.15.89:0/4217931255 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa00ce470 con 0x7fcfa00d18b0 2024-04-02T08:53:26.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.618+0000 7fcf9d400700 1 --2- 172.21.15.89:0/4217931255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa00d18b0 0x7fcfa00cf230 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009840 tx=0x7fcf8c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad344068ce131835 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:26.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.618+0000 7fcf9ca00700 1 -- 172.21.15.89:0/4217931255 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c011830 con 0x7fcfa00d18b0 2024-04-02T08:53:26.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.618+0000 7fcf9ca00700 1 -- 172.21.15.89:0/4217931255 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c011990 con 0x7fcfa00d18b0 2024-04-02T08:53:26.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.618+0000 7fcf9ca00700 1 -- 172.21.15.89:0/4217931255 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c011d20 con 0x7fcfa00d18b0 2024-04-02T08:53:26.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.619+0000 7fcfa5200700 1 -- 172.21.15.89:0/4217931255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa00d18b0 msgr2=0x7fcfa00cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:26.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.619+0000 7fcfa5200700 1 --2- 172.21.15.89:0/4217931255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa00d18b0 0x7fcfa00cf230 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009840 tx=0x7fcf8c009b50 comp rx=0 tx=0).stop 2024-04-02T08:53:26.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.619+0000 7fcfa5200700 1 -- 172.21.15.89:0/4217931255 shutdown_connections 2024-04-02T08:53:26.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.619+0000 7fcfa5200700 1 --2- 172.21.15.89:0/4217931255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa00d18b0 0x7fcfa00cf230 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:26.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.619+0000 7fcfa5200700 1 -- 172.21.15.89:0/4217931255 >> 172.21.15.89:0/4217931255 conn(0x7fcfa008aff0 msgr2=0x7fcfa00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:26.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.619+0000 7fcfa5200700 1 -- 172.21.15.89:0/4217931255 shutdown_connections 2024-04-02T08:53:26.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.619+0000 7fcfa5200700 1 -- 172.21.15.89:0/4217931255 wait complete. 2024-04-02T08:53:26.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.621+0000 7fcfa5200700 1 Processor -- start 2024-04-02T08:53:26.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.621+0000 7fcfa5200700 1 -- start start 2024-04-02T08:53:26.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcfa5200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa00d18b0 0x7fcfa015ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:26.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcfa5200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa0160f10 0x7fcfa015e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:26.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcfa5200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa0161590 0x7fcfa01f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:26.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcfa5200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa015fe90 con 0x7fcfa0161590 2024-04-02T08:53:26.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcfa5200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa015e760 con 0x7fcfa0160f10 2024-04-02T08:53:26.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcfa5200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa015f280 con 0x7fcfa00d18b0 2024-04-02T08:53:26.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcf9e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa0161590 0x7fcfa01f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:26.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcf9e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa0161590 0x7fcfa01f6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54074/0 (socket says 172.21.15.89:54074) 2024-04-02T08:53:26.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcf9e800700 1 -- 172.21.15.89:0/4001861524 learned_addr learned my addr 172.21.15.89:0/4001861524 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:26.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcf9d400700 1 --2- 172.21.15.89:0/4001861524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa0160f10 0x7fcfa015e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:26.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcf9de00700 1 --2- 172.21.15.89:0/4001861524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa00d18b0 0x7fcfa015ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:26.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcf9e800700 1 -- 172.21.15.89:0/4001861524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa00d18b0 msgr2=0x7fcfa015ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:26.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcf9e800700 1 --2- 172.21.15.89:0/4001861524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa00d18b0 0x7fcfa015ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:26.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcf9e800700 1 -- 172.21.15.89:0/4001861524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa0160f10 msgr2=0x7fcfa015e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:26.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcf9e800700 1 --2- 172.21.15.89:0/4001861524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa0160f10 0x7fcfa015e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:26.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.623+0000 7fcf9e800700 1 -- 172.21.15.89:0/4001861524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8c0094a0 con 0x7fcfa0161590 2024-04-02T08:53:26.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.624+0000 7fcf9e800700 1 --2- 172.21.15.89:0/4001861524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa0161590 0x7fcfa01f6a10 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800cb50 tx=0x7fcf9800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:26.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.624+0000 7fcf96000700 1 -- 172.21.15.89:0/4001861524 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98003c00 con 0x7fcfa0161590 2024-04-02T08:53:26.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.624+0000 7fcf96000700 1 -- 172.21.15.89:0/4001861524 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf98003d60 con 0x7fcfa0161590 2024-04-02T08:53:26.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.624+0000 7fcf96000700 1 -- 172.21.15.89:0/4001861524 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9800ab70 con 0x7fcfa0161590 2024-04-02T08:53:26.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.624+0000 7fcfa5200700 1 -- 172.21.15.89:0/4001861524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa015f560 con 0x7fcfa0161590 2024-04-02T08:53:26.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.624+0000 7fcfa5200700 1 -- 172.21.15.89:0/4001861524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa015fb20 con 0x7fcfa0161590 2024-04-02T08:53:26.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.625+0000 7fcfa5200700 1 -- 172.21.15.89:0/4001861524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa0160000 con 0x7fcfa0161590 2024-04-02T08:53:26.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.626+0000 7fcf96000700 1 -- 172.21.15.89:0/4001861524 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcf9800acd0 con 0x7fcfa0161590 2024-04-02T08:53:26.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.627+0000 7fcf96000700 1 --2- 172.21.15.89:0/4001861524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8006adb0 0x7fcf8006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:26.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.627+0000 7fcf96000700 1 -- 172.21.15.89:0/4001861524 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fcf980896a0 con 0x7fcfa0161590 2024-04-02T08:53:26.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.627+0000 7fcf9de00700 1 --2- 172.21.15.89:0/4001861524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8006adb0 0x7fcf8006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:26.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.628+0000 7fcf9de00700 1 --2- 172.21.15.89:0/4001861524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8006adb0 0x7fcf8006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcf90000f60 tx=0x7fcf90007d90 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:26.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.631+0000 7fcf96000700 1 -- 172.21.15.89:0/4001861524 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf980545b0 con 0x7fcfa0161590 2024-04-02T08:53:26.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.875+0000 7fcfa5200700 1 -- 172.21.15.89:0/4001861524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfa0067720 con 0x7fcfa0161590 2024-04-02T08:53:26.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.876+0000 7fcf96000700 1 -- 172.21.15.89:0/4001861524 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4745 (secure 0 0 0) 0x7fcf98057bd0 con 0x7fcfa0161590 2024-04-02T08:53:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:26.875 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1296569976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1296569976},{"type":"v1","addr":"172.21.15.113:6835","nonce":1296569976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9331,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1328950634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1328950634},{"type":"v1","addr":"172.21.15.89:6837","nonce":1328950634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":10032,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1167691520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1167691520},{"type":"v1","addr":"172.21.15.113:6834","nonce":1167691520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T08:53:22.881600+0000","modified":"2024-04-02T08:53:26.390144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10038":{"gid":10038,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/583021497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":583021497},{"type":"v1","addr":"172.21.15.89:6836","nonce":583021497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-04-02T08:53:26.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.878+0000 7fcfa5200700 1 -- 172.21.15.89:0/4001861524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8006adb0 msgr2=0x7fcf8006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:26.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.878+0000 7fcfa5200700 1 --2- 172.21.15.89:0/4001861524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8006adb0 0x7fcf8006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcf90000f60 tx=0x7fcf90007d90 comp rx=0 tx=0).stop 2024-04-02T08:53:26.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.878+0000 7fcfa5200700 1 -- 172.21.15.89:0/4001861524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa0161590 msgr2=0x7fcfa01f6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:26.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.878+0000 7fcfa5200700 1 --2- 172.21.15.89:0/4001861524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa0161590 0x7fcfa01f6a10 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800cb50 tx=0x7fcf9800cf10 comp rx=0 tx=0).stop 2024-04-02T08:53:26.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.878+0000 7fcfa5200700 1 -- 172.21.15.89:0/4001861524 shutdown_connections 2024-04-02T08:53:26.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.878+0000 7fcfa5200700 1 --2- 172.21.15.89:0/4001861524 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcf8006adb0 0x7fcf8006d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:26.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.878+0000 7fcfa5200700 1 --2- 172.21.15.89:0/4001861524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfa0161590 0x7fcfa01f6a10 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:26.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.878+0000 7fcfa5200700 1 --2- 172.21.15.89:0/4001861524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfa00d18b0 0x7fcfa015ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:26.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.878+0000 7fcfa5200700 1 --2- 172.21.15.89:0/4001861524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfa0160f10 0x7fcfa015e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:26.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.878+0000 7fcfa5200700 1 -- 172.21.15.89:0/4001861524 >> 172.21.15.89:0/4001861524 conn(0x7fcfa008aff0 msgr2=0x7fcfa00cb020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:26.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.878+0000 7fcfa5200700 1 -- 172.21.15.89:0/4001861524 shutdown_connections 2024-04-02T08:53:26.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:26.878+0000 7fcfa5200700 1 -- 172.21.15.89:0/4001861524 wait complete. 2024-04-02T08:53:26.879 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 89 2024-04-02T08:53:26.895 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T08:53:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.087+0000 7fcccde00700 1 Processor -- start 2024-04-02T08:53:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.087+0000 7fcccde00700 1 -- start start 2024-04-02T08:53:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.088+0000 7fcccde00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fccc80d18b0 0x7fccc80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.088+0000 7fcccde00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc8161590 0x7fccc8161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.088+0000 7fcccde00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc80ceda0 con 0x7fccc8160f10 2024-04-02T08:53:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.089+0000 7fcccde00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc80d1d30 con 0x7fccc80d18b0 2024-04-02T08:53:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.089+0000 7fcccde00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc8161f60 con 0x7fccc8161590 2024-04-02T08:53:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.089+0000 7fccc6a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fccc8160f10 0x7fccc80d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46378/0 (socket says 172.21.15.89:46378) 2024-04-02T08:53:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.089+0000 7fccc6000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fccc80d18b0 0x7fccc80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.089+0000 7fccc6a00700 1 -- 172.21.15.89:0/1915255938 learned_addr learned my addr 172.21.15.89:0/1915255938 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:27.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.089+0000 7fccc7400700 1 --2- 172.21.15.89:0/1915255938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc8161590 0x7fccc8161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:27.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.089+0000 7fccc5600700 1 -- 172.21.15.89:0/1915255938 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082081543 0 0) 0x7fccc80ceda0 con 0x7fccc8160f10 2024-04-02T08:53:27.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.089+0000 7fccc6000700 1 -- 172.21.15.89:0/1915255938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc8161590 msgr2=0x7fccc8161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:27.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.089+0000 7fccc6000700 1 --2- 172.21.15.89:0/1915255938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc8161590 0x7fccc8161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.089+0000 7fccc6000700 1 -- 172.21.15.89:0/1915255938 >> v1:172.21.15.89:6789/0 conn(0x7fccc8160f10 legacy=0x7fccc80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:27.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.089+0000 7fccc6000700 1 -- 172.21.15.89:0/1915255938 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc81612f0 con 0x7fccc80d18b0 2024-04-02T08:53:27.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.090+0000 7fccc6000700 1 --2- 172.21.15.89:0/1915255938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc80d18b0 0x7fccc80cf230 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009870 tx=0x7fccb4009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c12a6182862eb08 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:27.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.090+0000 7fccc5600700 1 -- 172.21.15.89:0/1915255938 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4011830 con 0x7fccc80d18b0 2024-04-02T08:53:27.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.090+0000 7fccc5600700 1 -- 172.21.15.89:0/1915255938 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb4011990 con 0x7fccc80d18b0 2024-04-02T08:53:27.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.090+0000 7fccc5600700 1 -- 172.21.15.89:0/1915255938 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4011d20 con 0x7fccc80d18b0 2024-04-02T08:53:27.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.090+0000 7fcccde00700 1 -- 172.21.15.89:0/1915255938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc80d18b0 msgr2=0x7fccc80cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:27.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.090+0000 7fcccde00700 1 --2- 172.21.15.89:0/1915255938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc80d18b0 0x7fccc80cf230 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009870 tx=0x7fccb4009b80 comp rx=0 tx=0).stop 2024-04-02T08:53:27.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.090+0000 7fcccde00700 1 -- 172.21.15.89:0/1915255938 shutdown_connections 2024-04-02T08:53:27.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.090+0000 7fcccde00700 1 --2- 172.21.15.89:0/1915255938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc8161590 0x7fccc8161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.090+0000 7fcccde00700 1 --2- 172.21.15.89:0/1915255938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc80d18b0 0x7fccc80cf230 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.090+0000 7fcccde00700 1 -- 172.21.15.89:0/1915255938 >> 172.21.15.89:0/1915255938 conn(0x7fccc808aff0 msgr2=0x7fccc80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:27.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.090+0000 7fcccde00700 1 -- 172.21.15.89:0/1915255938 shutdown_connections 2024-04-02T08:53:27.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.090+0000 7fcccde00700 1 -- 172.21.15.89:0/1915255938 wait complete. 2024-04-02T08:53:27.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.092+0000 7fcccde00700 1 Processor -- start 2024-04-02T08:53:27.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.092+0000 7fcccde00700 1 -- start start 2024-04-02T08:53:27.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.093+0000 7fcccde00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc80d18b0 0x7fccc815dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.093+0000 7fcccde00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc8160f10 0x7fccc815fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.093+0000 7fcccde00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc8161590 0x7fccc814b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.093+0000 7fcccde00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc814c2b0 con 0x7fccc8161590 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.093+0000 7fcccde00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc81493c0 con 0x7fccc80d18b0 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.093+0000 7fcccde00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc815e450 con 0x7fccc8160f10 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.093+0000 7fccc7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc8161590 0x7fccc814b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.093+0000 7fccc7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc8161590 0x7fccc814b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54098/0 (socket says 172.21.15.89:54098) 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.093+0000 7fccc7400700 1 -- 172.21.15.89:0/1207717675 learned_addr learned my addr 172.21.15.89:0/1207717675 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.094+0000 7fccc6a00700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc80d18b0 0x7fccc815dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.094+0000 7fccc6000700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc8160f10 0x7fccc815fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.094+0000 7fccc7400700 1 -- 172.21.15.89:0/1207717675 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc8160f10 msgr2=0x7fccc815fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.094+0000 7fccc7400700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc8160f10 0x7fccc815fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.094+0000 7fccc7400700 1 -- 172.21.15.89:0/1207717675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc80d18b0 msgr2=0x7fccc815dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.094+0000 7fccc7400700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc80d18b0 0x7fccc815dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.094+0000 7fccc7400700 1 -- 172.21.15.89:0/1207717675 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb40094a0 con 0x7fccc8161590 2024-04-02T08:53:27.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.094+0000 7fccc6a00700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc80d18b0 0x7fccc815dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:27.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.094+0000 7fccc7400700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc8161590 0x7fccc814b450 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fccc0007e90 tx=0x7fccc000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:27.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.094+0000 7fccbf400700 1 -- 172.21.15.89:0/1207717675 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc000ede0 con 0x7fccc8161590 2024-04-02T08:53:27.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.094+0000 7fccbf400700 1 -- 172.21.15.89:0/1207717675 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc0014420 con 0x7fccc8161590 2024-04-02T08:53:27.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.094+0000 7fccbf400700 1 -- 172.21.15.89:0/1207717675 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc00147d0 con 0x7fccc8161590 2024-04-02T08:53:27.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.094+0000 7fcccde00700 1 -- 172.21.15.89:0/1207717675 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc815e230 con 0x7fccc8161590 2024-04-02T08:53:27.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.094+0000 7fcccde00700 1 -- 172.21.15.89:0/1207717675 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc8149610 con 0x7fccc8161590 2024-04-02T08:53:27.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.095+0000 7fcccde00700 1 -- 172.21.15.89:0/1207717675 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc808b3d0 con 0x7fccc8161590 2024-04-02T08:53:27.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.097+0000 7fccbf400700 1 -- 172.21.15.89:0/1207717675 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fccc0014930 con 0x7fccc8161590 2024-04-02T08:53:27.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.097+0000 7fccbf400700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcca806adb0 0x7fcca806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:27.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.098+0000 7fccbf400700 1 -- 172.21.15.89:0/1207717675 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fccc0089de0 con 0x7fccc8161590 2024-04-02T08:53:27.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.098+0000 7fccc6a00700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcca806adb0 0x7fcca806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:27.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.098+0000 7fccc6a00700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcca806adb0 0x7fcca806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fccb8000f60 tx=0x7fccb8007d50 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:27.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.102+0000 7fccbf400700 1 -- 172.21.15.89:0/1207717675 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fccc0054c60 con 0x7fccc8161590 2024-04-02T08:53:27.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.305+0000 7fcccde00700 1 -- 172.21.15.89:0/1207717675 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccc8067720 con 0x7fccc8161590 2024-04-02T08:53:27.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.307+0000 7fccbf400700 1 -- 172.21.15.89:0/1207717675 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) v1 ==== 74+0+16754 (secure 0 0 0) 0x7fccc0058280 con 0x7fccc8161590 2024-04-02T08:53:27.306 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:27.306 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":186,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T08:53:23.884074+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-02T08:53:20.736248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-02T08:53:21.860376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T08:53:27.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.309+0000 7fcccde00700 1 -- 172.21.15.89:0/1207717675 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcca806adb0 msgr2=0x7fcca806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:27.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.309+0000 7fcccde00700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcca806adb0 0x7fcca806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fccb8000f60 tx=0x7fccb8007d50 comp rx=0 tx=0).stop 2024-04-02T08:53:27.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.309+0000 7fcccde00700 1 -- 172.21.15.89:0/1207717675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc8161590 msgr2=0x7fccc814b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:27.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.309+0000 7fcccde00700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc8161590 0x7fccc814b450 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fccc0007e90 tx=0x7fccc000e3c0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.309+0000 7fcccde00700 1 -- 172.21.15.89:0/1207717675 shutdown_connections 2024-04-02T08:53:27.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.309+0000 7fcccde00700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fcca806adb0 0x7fcca806d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.309+0000 7fcccde00700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc80d18b0 0x7fccc815dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.309+0000 7fcccde00700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccc8161590 0x7fccc814b450 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.309+0000 7fcccde00700 1 --2- 172.21.15.89:0/1207717675 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccc8160f10 0x7fccc815fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.309+0000 7fcccde00700 1 -- 172.21.15.89:0/1207717675 >> 172.21.15.89:0/1207717675 conn(0x7fccc808aff0 msgr2=0x7fccc80caae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:27.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.309+0000 7fcccde00700 1 -- 172.21.15.89:0/1207717675 shutdown_connections 2024-04-02T08:53:27.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.309+0000 7fcccde00700 1 -- 172.21.15.89:0/1207717675 wait complete. 2024-04-02T08:53:27.327 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T08:53:27.327 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T08:53:27.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.521+0000 7fd5d2c00700 1 Processor -- start 2024-04-02T08:53:27.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.521+0000 7fd5d2c00700 1 -- start start 2024-04-02T08:53:27.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.522+0000 7fd5d2c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5cc15d640 0x7fd5cc0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:27.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.522+0000 7fd5d2c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd5cc1619d0 0x7fd5cc0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:27.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.522+0000 7fd5d2c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5cc0d30d0 con 0x7fd5cc0ceec0 2024-04-02T08:53:27.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.522+0000 7fd5d2c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5cc0b6840 con 0x7fd5cc1619d0 2024-04-02T08:53:27.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.522+0000 7fd5d2c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc0d2880 con 0x7fd5cc15d640 2024-04-02T08:53:27.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.522+0000 7fd5cb400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd5cc0ceec0 0x7fd5cc1612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46386/0 (socket says 172.21.15.89:46386) 2024-04-02T08:53:27.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.522+0000 7fd5cb400700 1 -- 172.21.15.89:0/2019009752 learned_addr learned my addr 172.21.15.89:0/2019009752 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:27.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.522+0000 7fd5d0c00700 1 --2- 172.21.15.89:0/2019009752 >> v2:172.21.15.89:3300/0 conn(0x7fd5cc1619d0 0x7fd5cc0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:27.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.523+0000 7fd5cbe00700 1 --2- 172.21.15.89:0/2019009752 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5cc15d640 0x7fd5cc0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:27.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.523+0000 7fd5caa00700 1 -- 172.21.15.89:0/2019009752 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104607096 0 0) 0x7fd5cc0d30d0 con 0x7fd5cc0ceec0 2024-04-02T08:53:27.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.523+0000 7fd5caa00700 1 -- 172.21.15.89:0/2019009752 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b00035b0 con 0x7fd5cc0ceec0 2024-04-02T08:53:27.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.523+0000 7fd5d0c00700 1 -- 172.21.15.89:0/2019009752 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5cc15d640 msgr2=0x7fd5cc0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:27.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.523+0000 7fd5d0c00700 1 --2- 172.21.15.89:0/2019009752 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5cc15d640 0x7fd5cc0d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.523+0000 7fd5d0c00700 1 -- 172.21.15.89:0/2019009752 >> v1:172.21.15.89:6789/0 conn(0x7fd5cc0ceec0 legacy=0x7fd5cc1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:27.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.523+0000 7fd5d0c00700 1 -- 172.21.15.89:0/2019009752 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5cc0d2b30 con 0x7fd5cc1619d0 2024-04-02T08:53:27.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.523+0000 7fd5cbe00700 1 --2- 172.21.15.89:0/2019009752 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5cc15d640 0x7fd5cc0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:27.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.523+0000 7fd5d0c00700 1 --2- 172.21.15.89:0/2019009752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5cc1619d0 0x7fd5cc0b6cd0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009d90 tx=0x7fd5c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64dd69f4f53aaf87 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:27.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.523+0000 7fd5caa00700 1 -- 172.21.15.89:0/2019009752 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c400e070 con 0x7fd5cc1619d0 2024-04-02T08:53:27.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.523+0000 7fd5caa00700 1 -- 172.21.15.89:0/2019009752 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c400bb20 con 0x7fd5cc1619d0 2024-04-02T08:53:27.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.523+0000 7fd5caa00700 1 -- 172.21.15.89:0/2019009752 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4012430 con 0x7fd5cc1619d0 2024-04-02T08:53:27.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.524+0000 7fd5d2c00700 1 -- 172.21.15.89:0/2019009752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5cc1619d0 msgr2=0x7fd5cc0b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:27.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.524+0000 7fd5d2c00700 1 --2- 172.21.15.89:0/2019009752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5cc1619d0 0x7fd5cc0b6cd0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009d90 tx=0x7fd5c400b330 comp rx=0 tx=0).stop 2024-04-02T08:53:27.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.524+0000 7fd5d2c00700 1 -- 172.21.15.89:0/2019009752 shutdown_connections 2024-04-02T08:53:27.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.524+0000 7fd5d2c00700 1 --2- 172.21.15.89:0/2019009752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5cc1619d0 0x7fd5cc0b6cd0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.524+0000 7fd5d2c00700 1 --2- 172.21.15.89:0/2019009752 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5cc15d640 0x7fd5cc0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.524+0000 7fd5d2c00700 1 -- 172.21.15.89:0/2019009752 >> 172.21.15.89:0/2019009752 conn(0x7fd5cc08aff0 msgr2=0x7fd5cc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:27.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.524+0000 7fd5d2c00700 1 -- 172.21.15.89:0/2019009752 shutdown_connections 2024-04-02T08:53:27.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.524+0000 7fd5d2c00700 1 -- 172.21.15.89:0/2019009752 wait complete. 2024-04-02T08:53:27.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.526+0000 7fd5d2c00700 1 Processor -- start 2024-04-02T08:53:27.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.526+0000 7fd5d2c00700 1 -- start start 2024-04-02T08:53:27.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.527+0000 7fd5d2c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5cc0ceec0 0x7fd5cc1556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:27.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.527+0000 7fd5d2c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5cc15d640 0x7fd5cc15efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:27.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.527+0000 7fd5d2c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5cc1619d0 0x7fd5cc15f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:27.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.527+0000 7fd5d2c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc0c8e50 con 0x7fd5cc1619d0 2024-04-02T08:53:27.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.527+0000 7fd5d2c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc0c9130 con 0x7fd5cc15d640 2024-04-02T08:53:27.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.527+0000 7fd5d2c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc156760 con 0x7fd5cc0ceec0 2024-04-02T08:53:27.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.527+0000 7fd5d0c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5cc1619d0 0x7fd5cc15f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:27.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.527+0000 7fd5d0c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5cc1619d0 0x7fd5cc15f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54126/0 (socket says 172.21.15.89:54126) 2024-04-02T08:53:27.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.527+0000 7fd5d0c00700 1 -- 172.21.15.89:0/1328907794 learned_addr learned my addr 172.21.15.89:0/1328907794 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:27.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.528+0000 7fd5cb400700 1 --2- 172.21.15.89:0/1328907794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5cc15d640 0x7fd5cc15efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:27.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.528+0000 7fd5cbe00700 1 --2- 172.21.15.89:0/1328907794 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5cc0ceec0 0x7fd5cc1556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:27.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.528+0000 7fd5d0c00700 1 -- 172.21.15.89:0/1328907794 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5cc0ceec0 msgr2=0x7fd5cc1556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:27.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.528+0000 7fd5d0c00700 1 --2- 172.21.15.89:0/1328907794 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5cc0ceec0 0x7fd5cc1556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.528+0000 7fd5d0c00700 1 -- 172.21.15.89:0/1328907794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5cc15d640 msgr2=0x7fd5cc15efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:27.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.528+0000 7fd5d0c00700 1 --2- 172.21.15.89:0/1328907794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5cc15d640 0x7fd5cc15efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.528+0000 7fd5d0c00700 1 -- 172.21.15.89:0/1328907794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c4009a70 con 0x7fd5cc1619d0 2024-04-02T08:53:27.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.528+0000 7fd5d0c00700 1 --2- 172.21.15.89:0/1328907794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5cc1619d0 0x7fd5cc15f4e0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4014040 tx=0x7fd5c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:27.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.528+0000 7fd5c3e00700 1 -- 172.21.15.89:0/1328907794 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4012430 con 0x7fd5cc1619d0 2024-04-02T08:53:27.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.528+0000 7fd5c3e00700 1 -- 172.21.15.89:0/1328907794 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c4020070 con 0x7fd5cc1619d0 2024-04-02T08:53:27.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.528+0000 7fd5c3e00700 1 -- 172.21.15.89:0/1328907794 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4003ea0 con 0x7fd5cc1619d0 2024-04-02T08:53:27.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.528+0000 7fd5d2c00700 1 -- 172.21.15.89:0/1328907794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5cc156a40 con 0x7fd5cc1619d0 2024-04-02T08:53:27.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.528+0000 7fd5d2c00700 1 -- 172.21.15.89:0/1328907794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5cc1563a0 con 0x7fd5cc1619d0 2024-04-02T08:53:27.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.529+0000 7fd5d2c00700 1 -- 172.21.15.89:0/1328907794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5cc156590 con 0x7fd5cc1619d0 2024-04-02T08:53:27.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.530+0000 7fd5c3e00700 1 -- 172.21.15.89:0/1328907794 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd5c4004280 con 0x7fd5cc1619d0 2024-04-02T08:53:27.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.531+0000 7fd5c3e00700 1 --2- 172.21.15.89:0/1328907794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd5ac06adb0 0x7fd5ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:27.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.531+0000 7fd5c3e00700 1 -- 172.21.15.89:0/1328907794 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fd5c4022070 con 0x7fd5cc1619d0 2024-04-02T08:53:27.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.531+0000 7fd5cbe00700 1 --2- 172.21.15.89:0/1328907794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd5ac06adb0 0x7fd5ac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:27.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.532+0000 7fd5cbe00700 1 --2- 172.21.15.89:0/1328907794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd5ac06adb0 0x7fd5ac06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc158c10 tx=0x7fd5bc008040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:27.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.535+0000 7fd5c3e00700 1 -- 172.21.15.89:0/1328907794 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd5c4056390 con 0x7fd5cc1619d0 2024-04-02T08:53:27.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.732+0000 7fd5d2c00700 1 -- 172.21.15.89:0/1328907794 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd5cc0b64d0 con 0x7fd5ac06adb0 2024-04-02T08:53:27.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.742+0000 7fd5c3e00700 1 -- 172.21.15.89:0/1328907794 <== mgr.9807 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288920 (secure 0 0 0) 0x7fd5bc00c050 con 0x7fd5ac06adb0 2024-04-02T08:53:27.746 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:27.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.747+0000 7fd5d2c00700 1 -- 172.21.15.89:0/1328907794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd5ac06adb0 msgr2=0x7fd5ac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:27.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.747+0000 7fd5d2c00700 1 --2- 172.21.15.89:0/1328907794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd5ac06adb0 0x7fd5ac06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc158c10 tx=0x7fd5bc008040 comp rx=0 tx=0).stop 2024-04-02T08:53:27.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.747+0000 7fd5d2c00700 1 -- 172.21.15.89:0/1328907794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5cc1619d0 msgr2=0x7fd5cc15f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:27.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.747+0000 7fd5d2c00700 1 --2- 172.21.15.89:0/1328907794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5cc1619d0 0x7fd5cc15f4e0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4014040 tx=0x7fd5c4004ad0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.747+0000 7fd5d2c00700 1 -- 172.21.15.89:0/1328907794 shutdown_connections 2024-04-02T08:53:27.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.747+0000 7fd5d2c00700 1 --2- 172.21.15.89:0/1328907794 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7fd5ac06adb0 0x7fd5ac06d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.747+0000 7fd5d2c00700 1 --2- 172.21.15.89:0/1328907794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5cc1619d0 0x7fd5cc15f4e0 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.747+0000 7fd5d2c00700 1 --2- 172.21.15.89:0/1328907794 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd5cc0ceec0 0x7fd5cc1556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.747+0000 7fd5d2c00700 1 --2- 172.21.15.89:0/1328907794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5cc15d640 0x7fd5cc15efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.747+0000 7fd5d2c00700 1 -- 172.21.15.89:0/1328907794 >> 172.21.15.89:0/1328907794 conn(0x7fd5cc08aff0 msgr2=0x7fd5cc160000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:27.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.747+0000 7fd5d2c00700 1 -- 172.21.15.89:0/1328907794 shutdown_connections 2024-04-02T08:53:27.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.747+0000 7fd5d2c00700 1 -- 172.21.15.89:0/1328907794 wait complete. 2024-04-02T08:53:27.750 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T08:53:27.764 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T08:53:26.738532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":394624,"kb_used_data":19332,"kb_used_omap":12,"kb_used_meta":375155,"kb_avail":754580096,"statfs":{"total":773094113280,"available":772690018304,"internally_reserved":0,"allocated":19795968,"data_stored":10479454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":384159288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.365739"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.917437+0000","last_change":"2024-04-02T08:53:22.911486+0000","last_active":"2024-04-02T08:53:22.917437+0000","last_peered":"2024-04-02T08:53:22.917437+0000","last_clean":"2024-04-02T08:53:22.917437+0000","last_became_active":"2024-04-02T08:53:22.909502+0000","last_became_peered":"2024-04-02T08:53:22.909502+0000","last_unstale":"2024-04-02T08:53:22.917437+0000","last_undegraded":"2024-04-02T08:53:22.917437+0000","last_fullsized":"2024-04-02T08:53:22.917437+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:29:45.566973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.913553+0000","last_change":"2024-04-02T08:53:22.910212+0000","last_active":"2024-04-02T08:53:22.913553+0000","last_peered":"2024-04-02T08:53:22.913553+0000","last_clean":"2024-04-02T08:53:22.913553+0000","last_became_active":"2024-04-02T08:53:22.908146+0000","last_became_peered":"2024-04-02T08:53:22.908146+0000","last_unstale":"2024-04-02T08:53:22.913553+0000","last_undegraded":"2024-04-02T08:53:22.913553+0000","last_fullsized":"2024-04-02T08:53:22.913553+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:57:08.290485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.908450+0000","last_change":"2024-04-02T08:53:22.899012+0000","last_active":"2024-04-02T08:53:22.908450+0000","last_peered":"2024-04-02T08:53:22.908450+0000","last_clean":"2024-04-02T08:53:22.908450+0000","last_became_active":"2024-04-02T08:53:22.898750+0000","last_became_peered":"2024-04-02T08:53:22.898750+0000","last_unstale":"2024-04-02T08:53:22.908450+0000","last_undegraded":"2024-04-02T08:53:22.908450+0000","last_fullsized":"2024-04-02T08:53:22.908450+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:30.142896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896089+0000","last_change":"2024-04-02T08:53:22.899906+0000","last_active":"2024-04-02T08:53:23.896089+0000","last_peered":"2024-04-02T08:53:23.896089+0000","last_clean":"2024-04-02T08:53:23.896089+0000","last_became_active":"2024-04-02T08:53:22.899274+0000","last_became_peered":"2024-04-02T08:53:22.899274+0000","last_unstale":"2024-04-02T08:53:23.896089+0000","last_undegraded":"2024-04-02T08:53:23.896089+0000","last_fullsized":"2024-04-02T08:53:23.896089+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:35:49.453211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.919529+0000","last_change":"2024-04-02T08:53:22.912546+0000","last_active":"2024-04-02T08:53:22.919529+0000","last_peered":"2024-04-02T08:53:22.919529+0000","last_clean":"2024-04-02T08:53:22.919529+0000","last_became_active":"2024-04-02T08:53:22.912084+0000","last_became_peered":"2024-04-02T08:53:22.912084+0000","last_unstale":"2024-04-02T08:53:22.919529+0000","last_undegraded":"2024-04-02T08:53:22.919529+0000","last_fullsized":"2024-04-02T08:53:22.919529+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:18:42.804000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895591+0000","last_change":"2024-04-02T08:53:22.909227+0000","last_active":"2024-04-02T08:53:23.895591+0000","last_peered":"2024-04-02T08:53:23.895591+0000","last_clean":"2024-04-02T08:53:23.895591+0000","last_became_active":"2024-04-02T08:53:22.908942+0000","last_became_peered":"2024-04-02T08:53:22.908942+0000","last_unstale":"2024-04-02T08:53:23.895591+0000","last_undegraded":"2024-04-02T08:53:23.895591+0000","last_fullsized":"2024-04-02T08:53:23.895591+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:57.619946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.916391+0000","last_change":"2024-04-02T08:53:22.913548+0000","last_active":"2024-04-02T08:53:22.916391+0000","last_peered":"2024-04-02T08:53:22.916391+0000","last_clean":"2024-04-02T08:53:22.916391+0000","last_became_active":"2024-04-02T08:53:22.913269+0000","last_became_peered":"2024-04-02T08:53:22.913269+0000","last_unstale":"2024-04-02T08:53:22.916391+0000","last_undegraded":"2024-04-02T08:53:22.916391+0000","last_fullsized":"2024-04-02T08:53:22.916391+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:20:20.210105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.909222+0000","last_change":"2024-04-02T08:53:22.906726+0000","last_active":"2024-04-02T08:53:22.909222+0000","last_peered":"2024-04-02T08:53:22.909222+0000","last_clean":"2024-04-02T08:53:22.909222+0000","last_became_active":"2024-04-02T08:53:22.906129+0000","last_became_peered":"2024-04-02T08:53:22.906129+0000","last_unstale":"2024-04-02T08:53:22.909222+0000","last_undegraded":"2024-04-02T08:53:22.909222+0000","last_fullsized":"2024-04-02T08:53:22.909222+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:15.249827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.910619+0000","last_change":"2024-04-02T08:53:22.909792+0000","last_active":"2024-04-02T08:53:22.910619+0000","last_peered":"2024-04-02T08:53:22.910619+0000","last_clean":"2024-04-02T08:53:22.910619+0000","last_became_active":"2024-04-02T08:53:22.908655+0000","last_became_peered":"2024-04-02T08:53:22.908655+0000","last_unstale":"2024-04-02T08:53:22.910619+0000","last_undegraded":"2024-04-02T08:53:22.910619+0000","last_fullsized":"2024-04-02T08:53:22.910619+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:33:25.298095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.919914+0000","last_change":"2024-04-02T08:53:22.914731+0000","last_active":"2024-04-02T08:53:22.919914+0000","last_peered":"2024-04-02T08:53:22.919914+0000","last_clean":"2024-04-02T08:53:22.919914+0000","last_became_active":"2024-04-02T08:53:22.913815+0000","last_became_peered":"2024-04-02T08:53:22.913815+0000","last_unstale":"2024-04-02T08:53:22.919914+0000","last_undegraded":"2024-04-02T08:53:22.919914+0000","last_fullsized":"2024-04-02T08:53:22.919914+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:49:10.627711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915715+0000","last_change":"2024-04-02T08:53:22.904769+0000","last_active":"2024-04-02T08:53:22.915715+0000","last_peered":"2024-04-02T08:53:22.915715+0000","last_clean":"2024-04-02T08:53:22.915715+0000","last_became_active":"2024-04-02T08:53:22.898177+0000","last_became_peered":"2024-04-02T08:53:22.898177+0000","last_unstale":"2024-04-02T08:53:22.915715+0000","last_undegraded":"2024-04-02T08:53:22.915715+0000","last_fullsized":"2024-04-02T08:53:22.915715+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:42:55.438534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.891377+0000","last_change":"2024-04-02T08:53:22.906882+0000","last_active":"2024-04-02T08:53:23.891377+0000","last_peered":"2024-04-02T08:53:23.891377+0000","last_clean":"2024-04-02T08:53:23.891377+0000","last_became_active":"2024-04-02T08:53:22.906600+0000","last_became_peered":"2024-04-02T08:53:22.906600+0000","last_unstale":"2024-04-02T08:53:23.891377+0000","last_undegraded":"2024-04-02T08:53:23.891377+0000","last_fullsized":"2024-04-02T08:53:23.891377+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:37:49.959337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.891265+0000","last_change":"2024-04-02T08:53:22.900339+0000","last_active":"2024-04-02T08:53:23.891265+0000","last_peered":"2024-04-02T08:53:23.891265+0000","last_clean":"2024-04-02T08:53:23.891265+0000","last_became_active":"2024-04-02T08:53:22.900073+0000","last_became_peered":"2024-04-02T08:53:22.900073+0000","last_unstale":"2024-04-02T08:53:23.891265+0000","last_undegraded":"2024-04-02T08:53:23.891265+0000","last_fullsized":"2024-04-02T08:53:23.891265+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:07:06.537235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.919736+0000","last_change":"2024-04-02T08:53:22.911050+0000","last_active":"2024-04-02T08:53:22.919736+0000","last_peered":"2024-04-02T08:53:22.919736+0000","last_clean":"2024-04-02T08:53:22.919736+0000","last_became_active":"2024-04-02T08:53:22.908720+0000","last_became_peered":"2024-04-02T08:53:22.908720+0000","last_unstale":"2024-04-02T08:53:22.919736+0000","last_undegraded":"2024-04-02T08:53:22.919736+0000","last_fullsized":"2024-04-02T08:53:22.919736+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:45:44.350132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.920415+0000","last_change":"2024-04-02T08:53:22.916457+0000","last_active":"2024-04-02T08:53:22.920415+0000","last_peered":"2024-04-02T08:53:22.920415+0000","last_clean":"2024-04-02T08:53:22.920415+0000","last_became_active":"2024-04-02T08:53:22.916022+0000","last_became_peered":"2024-04-02T08:53:22.916022+0000","last_unstale":"2024-04-02T08:53:22.920415+0000","last_undegraded":"2024-04-02T08:53:22.920415+0000","last_fullsized":"2024-04-02T08:53:22.920415+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:54.588880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915598+0000","last_change":"2024-04-02T08:53:22.910047+0000","last_active":"2024-04-02T08:53:22.915598+0000","last_peered":"2024-04-02T08:53:22.915598+0000","last_clean":"2024-04-02T08:53:22.915598+0000","last_became_active":"2024-04-02T08:53:22.909738+0000","last_became_peered":"2024-04-02T08:53:22.909738+0000","last_unstale":"2024-04-02T08:53:22.915598+0000","last_undegraded":"2024-04-02T08:53:22.915598+0000","last_fullsized":"2024-04-02T08:53:22.915598+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:57:08.290485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896304+0000","last_change":"2024-04-02T08:53:22.901205+0000","last_active":"2024-04-02T08:53:23.896304+0000","last_peered":"2024-04-02T08:53:23.896304+0000","last_clean":"2024-04-02T08:53:23.896304+0000","last_became_active":"2024-04-02T08:53:22.900823+0000","last_became_peered":"2024-04-02T08:53:22.900823+0000","last_unstale":"2024-04-02T08:53:23.896304+0000","last_undegraded":"2024-04-02T08:53:23.896304+0000","last_fullsized":"2024-04-02T08:53:23.896304+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:17:36.197813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896085+0000","last_change":"2024-04-02T08:53:22.906959+0000","last_active":"2024-04-02T08:53:23.896085+0000","last_peered":"2024-04-02T08:53:23.896085+0000","last_clean":"2024-04-02T08:53:23.896085+0000","last_became_active":"2024-04-02T08:53:22.906636+0000","last_became_peered":"2024-04-02T08:53:22.906636+0000","last_unstale":"2024-04-02T08:53:23.896085+0000","last_undegraded":"2024-04-02T08:53:23.896085+0000","last_fullsized":"2024-04-02T08:53:23.896085+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:51.246631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.916081+0000","last_change":"2024-04-02T08:53:22.913253+0000","last_active":"2024-04-02T08:53:22.916081+0000","last_peered":"2024-04-02T08:53:22.916081+0000","last_clean":"2024-04-02T08:53:22.916081+0000","last_became_active":"2024-04-02T08:53:22.912885+0000","last_became_peered":"2024-04-02T08:53:22.912885+0000","last_unstale":"2024-04-02T08:53:22.916081+0000","last_undegraded":"2024-04-02T08:53:22.916081+0000","last_fullsized":"2024-04-02T08:53:22.916081+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:07:20.689753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.891507+0000","last_change":"2024-04-02T08:53:22.898307+0000","last_active":"2024-04-02T08:53:23.891507+0000","last_peered":"2024-04-02T08:53:23.891507+0000","last_clean":"2024-04-02T08:53:23.891507+0000","last_became_active":"2024-04-02T08:53:22.897981+0000","last_became_peered":"2024-04-02T08:53:22.897981+0000","last_unstale":"2024-04-02T08:53:23.891507+0000","last_undegraded":"2024-04-02T08:53:23.891507+0000","last_fullsized":"2024-04-02T08:53:23.891507+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:16:56.465320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895543+0000","last_change":"2024-04-02T08:53:22.910525+0000","last_active":"2024-04-02T08:53:23.895543+0000","last_peered":"2024-04-02T08:53:23.895543+0000","last_clean":"2024-04-02T08:53:23.895543+0000","last_became_active":"2024-04-02T08:53:22.910243+0000","last_became_peered":"2024-04-02T08:53:22.910243+0000","last_unstale":"2024-04-02T08:53:23.895543+0000","last_undegraded":"2024-04-02T08:53:23.895543+0000","last_fullsized":"2024-04-02T08:53:23.895543+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:34.518513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915834+0000","last_change":"2024-04-02T08:53:22.904389+0000","last_active":"2024-04-02T08:53:22.915834+0000","last_peered":"2024-04-02T08:53:22.915834+0000","last_clean":"2024-04-02T08:53:22.915834+0000","last_became_active":"2024-04-02T08:53:22.904123+0000","last_became_peered":"2024-04-02T08:53:22.904123+0000","last_unstale":"2024-04-02T08:53:22.915834+0000","last_undegraded":"2024-04-02T08:53:22.915834+0000","last_fullsized":"2024-04-02T08:53:22.915834+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:41:18.537425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915985+0000","last_change":"2024-04-02T08:53:22.907284+0000","last_active":"2024-04-02T08:53:22.915985+0000","last_peered":"2024-04-02T08:53:22.915985+0000","last_clean":"2024-04-02T08:53:22.915985+0000","last_became_active":"2024-04-02T08:53:22.900183+0000","last_became_peered":"2024-04-02T08:53:22.900183+0000","last_unstale":"2024-04-02T08:53:22.915985+0000","last_undegraded":"2024-04-02T08:53:22.915985+0000","last_fullsized":"2024-04-02T08:53:22.915985+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:08:17.507337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.919811+0000","last_change":"2024-04-02T08:53:22.911453+0000","last_active":"2024-04-02T08:53:22.919811+0000","last_peered":"2024-04-02T08:53:22.919811+0000","last_clean":"2024-04-02T08:53:22.919811+0000","last_became_active":"2024-04-02T08:53:22.910973+0000","last_became_peered":"2024-04-02T08:53:22.910973+0000","last_unstale":"2024-04-02T08:53:22.919811+0000","last_undegraded":"2024-04-02T08:53:22.919811+0000","last_fullsized":"2024-04-02T08:53:22.919811+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:35.886941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.898051+0000","last_change":"2024-04-02T08:53:22.896319+0000","last_active":"2024-04-02T08:53:22.898051+0000","last_peered":"2024-04-02T08:53:22.898051+0000","last_clean":"2024-04-02T08:53:22.898051+0000","last_became_active":"2024-04-02T08:53:22.896066+0000","last_became_peered":"2024-04-02T08:53:22.896066+0000","last_unstale":"2024-04-02T08:53:22.898051+0000","last_undegraded":"2024-04-02T08:53:22.898051+0000","last_fullsized":"2024-04-02T08:53:22.898051+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:03:30.324036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.911129+0000","last_change":"2024-04-02T08:53:22.902522+0000","last_active":"2024-04-02T08:53:22.911129+0000","last_peered":"2024-04-02T08:53:22.911129+0000","last_clean":"2024-04-02T08:53:22.911129+0000","last_became_active":"2024-04-02T08:53:22.901620+0000","last_became_peered":"2024-04-02T08:53:22.901620+0000","last_unstale":"2024-04-02T08:53:22.911129+0000","last_undegraded":"2024-04-02T08:53:22.911129+0000","last_fullsized":"2024-04-02T08:53:22.911129+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:27.613450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.919307+0000","last_change":"2024-04-02T08:53:22.911275+0000","last_active":"2024-04-02T08:53:22.919307+0000","last_peered":"2024-04-02T08:53:22.919307+0000","last_clean":"2024-04-02T08:53:22.919307+0000","last_became_active":"2024-04-02T08:53:22.910684+0000","last_became_peered":"2024-04-02T08:53:22.910684+0000","last_unstale":"2024-04-02T08:53:22.919307+0000","last_undegraded":"2024-04-02T08:53:22.919307+0000","last_fullsized":"2024-04-02T08:53:22.919307+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:18:00.950788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.916461+0000","last_change":"2024-04-02T08:53:22.914295+0000","last_active":"2024-04-02T08:53:22.916461+0000","last_peered":"2024-04-02T08:53:22.916461+0000","last_clean":"2024-04-02T08:53:22.916461+0000","last_became_active":"2024-04-02T08:53:22.913973+0000","last_became_peered":"2024-04-02T08:53:22.913973+0000","last_unstale":"2024-04-02T08:53:22.916461+0000","last_undegraded":"2024-04-02T08:53:22.916461+0000","last_fullsized":"2024-04-02T08:53:22.916461+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:30.142896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.900556+0000","last_change":"2024-04-02T08:53:22.896360+0000","last_active":"2024-04-02T08:53:22.900556+0000","last_peered":"2024-04-02T08:53:22.900556+0000","last_clean":"2024-04-02T08:53:22.900556+0000","last_became_active":"2024-04-02T08:53:22.896076+0000","last_became_peered":"2024-04-02T08:53:22.896076+0000","last_unstale":"2024-04-02T08:53:22.900556+0000","last_undegraded":"2024-04-02T08:53:22.900556+0000","last_fullsized":"2024-04-02T08:53:22.900556+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:16:12.933857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.919757+0000","last_change":"2024-04-02T08:53:22.913110+0000","last_active":"2024-04-02T08:53:22.919757+0000","last_peered":"2024-04-02T08:53:22.919757+0000","last_clean":"2024-04-02T08:53:22.919757+0000","last_became_active":"2024-04-02T08:53:22.909606+0000","last_became_peered":"2024-04-02T08:53:22.909606+0000","last_unstale":"2024-04-02T08:53:22.919757+0000","last_undegraded":"2024-04-02T08:53:22.919757+0000","last_fullsized":"2024-04-02T08:53:22.919757+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:00:23.503152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.898087+0000","last_change":"2024-04-02T08:53:22.896929+0000","last_active":"2024-04-02T08:53:22.898087+0000","last_peered":"2024-04-02T08:53:22.898087+0000","last_clean":"2024-04-02T08:53:22.898087+0000","last_became_active":"2024-04-02T08:53:22.896632+0000","last_became_peered":"2024-04-02T08:53:22.896632+0000","last_unstale":"2024-04-02T08:53:22.898087+0000","last_undegraded":"2024-04-02T08:53:22.898087+0000","last_fullsized":"2024-04-02T08:53:22.898087+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:07:20.689753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896482+0000","last_change":"2024-04-02T08:53:22.905257+0000","last_active":"2024-04-02T08:53:23.896482+0000","last_peered":"2024-04-02T08:53:23.896482+0000","last_clean":"2024-04-02T08:53:23.896482+0000","last_became_active":"2024-04-02T08:53:22.904963+0000","last_became_peered":"2024-04-02T08:53:22.904963+0000","last_unstale":"2024-04-02T08:53:23.896482+0000","last_undegraded":"2024-04-02T08:53:23.896482+0000","last_fullsized":"2024-04-02T08:53:23.896482+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:16:12.933857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896214+0000","last_change":"2024-04-02T08:53:22.902994+0000","last_active":"2024-04-02T08:53:23.896214+0000","last_peered":"2024-04-02T08:53:23.896214+0000","last_clean":"2024-04-02T08:53:23.896214+0000","last_became_active":"2024-04-02T08:53:22.902493+0000","last_became_peered":"2024-04-02T08:53:22.902493+0000","last_unstale":"2024-04-02T08:53:23.896214+0000","last_undegraded":"2024-04-02T08:53:23.896214+0000","last_fullsized":"2024-04-02T08:53:23.896214+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:27.613450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1f","version":"185'4","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.921373+0000","last_change":"2024-04-02T08:53:22.400085+0000","last_active":"2024-04-02T08:53:22.921373+0000","last_peered":"2024-04-02T08:53:22.921373+0000","last_clean":"2024-04-02T08:53:22.921373+0000","last_became_active":"2024-04-02T08:53:22.399817+0000","last_became_peered":"2024-04-02T08:53:22.399817+0000","last_unstale":"2024-04-02T08:53:22.921373+0000","last_undegraded":"2024-04-02T08:53:22.921373+0000","last_fullsized":"2024-04-02T08:53:22.921373+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:12.895858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.909660+0000","last_change":"2024-04-02T08:53:22.905495+0000","last_active":"2024-04-02T08:53:22.909660+0000","last_peered":"2024-04-02T08:53:22.909660+0000","last_clean":"2024-04-02T08:53:22.909660+0000","last_became_active":"2024-04-02T08:53:22.905253+0000","last_became_peered":"2024-04-02T08:53:22.905253+0000","last_unstale":"2024-04-02T08:53:22.909660+0000","last_undegraded":"2024-04-02T08:53:22.909660+0000","last_fullsized":"2024-04-02T08:53:22.909660+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:20:17.970790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"185'1","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.908631+0000","last_change":"2024-04-02T08:53:22.393126+0000","last_active":"2024-04-02T08:53:22.908631+0000","last_peered":"2024-04-02T08:53:22.908631+0000","last_clean":"2024-04-02T08:53:22.908631+0000","last_became_active":"2024-04-02T08:53:22.392818+0000","last_became_peered":"2024-04-02T08:53:22.392818+0000","last_unstale":"2024-04-02T08:53:22.908631+0000","last_undegraded":"2024-04-02T08:53:22.908631+0000","last_fullsized":"2024-04-02T08:53:22.908631+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:37:19.868360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895861+0000","last_change":"2024-04-02T08:53:22.911103+0000","last_active":"2024-04-02T08:53:23.895861+0000","last_peered":"2024-04-02T08:53:23.895861+0000","last_clean":"2024-04-02T08:53:23.895861+0000","last_became_active":"2024-04-02T08:53:22.910837+0000","last_became_peered":"2024-04-02T08:53:22.910837+0000","last_unstale":"2024-04-02T08:53:23.895861+0000","last_undegraded":"2024-04-02T08:53:23.895861+0000","last_fullsized":"2024-04-02T08:53:23.895861+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:57:08.290485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1d","version":"185'2","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895594+0000","last_change":"2024-04-02T08:53:22.399283+0000","last_active":"2024-04-02T08:53:23.895594+0000","last_peered":"2024-04-02T08:53:23.895594+0000","last_clean":"2024-04-02T08:53:23.895594+0000","last_became_active":"2024-04-02T08:53:22.398989+0000","last_became_peered":"2024-04-02T08:53:22.398989+0000","last_unstale":"2024-04-02T08:53:23.895594+0000","last_undegraded":"2024-04-02T08:53:23.895594+0000","last_fullsized":"2024-04-02T08:53:23.895594+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:54:09.448505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.920120+0000","last_change":"2024-04-02T08:53:22.915179+0000","last_active":"2024-04-02T08:53:22.920120+0000","last_peered":"2024-04-02T08:53:22.920120+0000","last_clean":"2024-04-02T08:53:22.920120+0000","last_became_active":"2024-04-02T08:53:22.914930+0000","last_became_peered":"2024-04-02T08:53:22.914930+0000","last_unstale":"2024-04-02T08:53:22.920120+0000","last_undegraded":"2024-04-02T08:53:22.920120+0000","last_fullsized":"2024-04-02T08:53:22.920120+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:02:46.249414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1c","version":"185'2","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.920362+0000","last_change":"2024-04-02T08:53:22.394260+0000","last_active":"2024-04-02T08:53:22.920362+0000","last_peered":"2024-04-02T08:53:22.920362+0000","last_clean":"2024-04-02T08:53:22.920362+0000","last_became_active":"2024-04-02T08:53:22.393832+0000","last_became_peered":"2024-04-02T08:53:22.393832+0000","last_unstale":"2024-04-02T08:53:22.920362+0000","last_undegraded":"2024-04-02T08:53:22.920362+0000","last_fullsized":"2024-04-02T08:53:22.920362+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:09.753474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895786+0000","last_change":"2024-04-02T08:53:22.900232+0000","last_active":"2024-04-02T08:53:23.895786+0000","last_peered":"2024-04-02T08:53:23.895786+0000","last_clean":"2024-04-02T08:53:23.895786+0000","last_became_active":"2024-04-02T08:53:22.899365+0000","last_became_peered":"2024-04-02T08:53:22.899365+0000","last_unstale":"2024-04-02T08:53:23.895786+0000","last_undegraded":"2024-04-02T08:53:23.895786+0000","last_fullsized":"2024-04-02T08:53:23.895786+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:31.589774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1b","version":"185'1","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.912683+0000","last_change":"2024-04-02T08:53:22.403201+0000","last_active":"2024-04-02T08:53:22.912683+0000","last_peered":"2024-04-02T08:53:22.912683+0000","last_clean":"2024-04-02T08:53:22.912683+0000","last_became_active":"2024-04-02T08:53:22.402941+0000","last_became_peered":"2024-04-02T08:53:22.402941+0000","last_unstale":"2024-04-02T08:53:22.912683+0000","last_undegraded":"2024-04-02T08:53:22.912683+0000","last_fullsized":"2024-04-02T08:53:22.912683+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:10.962513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895654+0000","last_change":"2024-04-02T08:53:22.907452+0000","last_active":"2024-04-02T08:53:23.895654+0000","last_peered":"2024-04-02T08:53:23.895654+0000","last_clean":"2024-04-02T08:53:23.895654+0000","last_became_active":"2024-04-02T08:53:22.904802+0000","last_became_peered":"2024-04-02T08:53:22.904802+0000","last_unstale":"2024-04-02T08:53:23.895654+0000","last_undegraded":"2024-04-02T08:53:23.895654+0000","last_fullsized":"2024-04-02T08:53:23.895654+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:47:04.773777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1a","version":"185'1","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895887+0000","last_change":"2024-04-02T08:53:22.402079+0000","last_active":"2024-04-02T08:53:23.895887+0000","last_peered":"2024-04-02T08:53:23.895887+0000","last_clean":"2024-04-02T08:53:23.895887+0000","last_became_active":"2024-04-02T08:53:22.401822+0000","last_became_peered":"2024-04-02T08:53:22.401822+0000","last_unstale":"2024-04-02T08:53:23.895887+0000","last_undegraded":"2024-04-02T08:53:23.895887+0000","last_fullsized":"2024-04-02T08:53:23.895887+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:57:11.838951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.907648+0000","last_change":"2024-04-02T08:53:22.899291+0000","last_active":"2024-04-02T08:53:22.907648+0000","last_peered":"2024-04-02T08:53:22.907648+0000","last_clean":"2024-04-02T08:53:22.907648+0000","last_became_active":"2024-04-02T08:53:22.898374+0000","last_became_peered":"2024-04-02T08:53:22.898374+0000","last_unstale":"2024-04-02T08:53:22.907648+0000","last_undegraded":"2024-04-02T08:53:22.907648+0000","last_fullsized":"2024-04-02T08:53:22.907648+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:45:29.735767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.887167+0000","last_change":"2024-04-02T08:53:22.395845+0000","last_active":"2024-04-02T08:53:22.887167+0000","last_peered":"2024-04-02T08:53:22.887167+0000","last_clean":"2024-04-02T08:53:22.887167+0000","last_became_active":"2024-04-02T08:53:22.395542+0000","last_became_peered":"2024-04-02T08:53:22.395542+0000","last_unstale":"2024-04-02T08:53:22.887167+0000","last_undegraded":"2024-04-02T08:53:22.887167+0000","last_fullsized":"2024-04-02T08:53:22.887167+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:19:11.682642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895344+0000","last_change":"2024-04-02T08:53:22.897914+0000","last_active":"2024-04-02T08:53:23.895344+0000","last_peered":"2024-04-02T08:53:23.895344+0000","last_clean":"2024-04-02T08:53:23.895344+0000","last_became_active":"2024-04-02T08:53:22.897643+0000","last_became_peered":"2024-04-02T08:53:22.897643+0000","last_unstale":"2024-04-02T08:53:23.895344+0000","last_undegraded":"2024-04-02T08:53:23.895344+0000","last_fullsized":"2024-04-02T08:53:23.895344+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:45:29.735767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.897699+0000","last_change":"2024-04-02T08:53:22.402944+0000","last_active":"2024-04-02T08:53:22.897699+0000","last_peered":"2024-04-02T08:53:22.897699+0000","last_clean":"2024-04-02T08:53:22.897699+0000","last_became_active":"2024-04-02T08:53:22.401902+0000","last_became_peered":"2024-04-02T08:53:22.401902+0000","last_unstale":"2024-04-02T08:53:22.897699+0000","last_undegraded":"2024-04-02T08:53:22.897699+0000","last_fullsized":"2024-04-02T08:53:22.897699+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:57:11.838951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915809+0000","last_change":"2024-04-02T08:53:22.911485+0000","last_active":"2024-04-02T08:53:22.915809+0000","last_peered":"2024-04-02T08:53:22.915809+0000","last_clean":"2024-04-02T08:53:22.915809+0000","last_became_active":"2024-04-02T08:53:22.911208+0000","last_became_peered":"2024-04-02T08:53:22.911208+0000","last_unstale":"2024-04-02T08:53:22.915809+0000","last_undegraded":"2024-04-02T08:53:22.915809+0000","last_fullsized":"2024-04-02T08:53:22.915809+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:21.004514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":11,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896217+0000","last_change":"2024-04-02T08:53:22.408387+0000","last_active":"2024-04-02T08:53:23.896217+0000","last_peered":"2024-04-02T08:53:23.896217+0000","last_clean":"2024-04-02T08:53:23.896217+0000","last_became_active":"2024-04-02T08:53:22.408002+0000","last_became_peered":"2024-04-02T08:53:22.408002+0000","last_unstale":"2024-04-02T08:53:23.896217+0000","last_undegraded":"2024-04-02T08:53:23.896217+0000","last_fullsized":"2024-04-02T08:53:23.896217+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:10:29.755694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.916291+0000","last_change":"2024-04-02T08:53:22.908670+0000","last_active":"2024-04-02T08:53:22.916291+0000","last_peered":"2024-04-02T08:53:22.916291+0000","last_clean":"2024-04-02T08:53:22.916291+0000","last_became_active":"2024-04-02T08:53:22.907830+0000","last_became_peered":"2024-04-02T08:53:22.907830+0000","last_unstale":"2024-04-02T08:53:22.916291+0000","last_undegraded":"2024-04-02T08:53:22.916291+0000","last_fullsized":"2024-04-02T08:53:22.916291+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:20:24.495239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.15","version":"185'3","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.921029+0000","last_change":"2024-04-02T08:53:22.394298+0000","last_active":"2024-04-02T08:53:22.921029+0000","last_peered":"2024-04-02T08:53:22.921029+0000","last_clean":"2024-04-02T08:53:22.921029+0000","last_became_active":"2024-04-02T08:53:22.393753+0000","last_became_peered":"2024-04-02T08:53:22.393753+0000","last_unstale":"2024-04-02T08:53:22.921029+0000","last_undegraded":"2024-04-02T08:53:22.921029+0000","last_fullsized":"2024-04-02T08:53:22.921029+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:49:51.216455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.908432+0000","last_change":"2024-04-02T08:53:22.899132+0000","last_active":"2024-04-02T08:53:22.908432+0000","last_peered":"2024-04-02T08:53:22.908432+0000","last_clean":"2024-04-02T08:53:22.908432+0000","last_became_active":"2024-04-02T08:53:22.898859+0000","last_became_peered":"2024-04-02T08:53:22.898859+0000","last_unstale":"2024-04-02T08:53:22.908432+0000","last_undegraded":"2024-04-02T08:53:22.908432+0000","last_fullsized":"2024-04-02T08:53:22.908432+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:33.443826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.14","version":"185'6","reported_seq":16,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.922204+0000","last_change":"2024-04-02T08:53:22.402901+0000","last_active":"2024-04-02T08:53:22.922204+0000","last_peered":"2024-04-02T08:53:22.922204+0000","last_clean":"2024-04-02T08:53:22.922204+0000","last_became_active":"2024-04-02T08:53:22.402573+0000","last_became_peered":"2024-04-02T08:53:22.402573+0000","last_unstale":"2024-04-02T08:53:22.922204+0000","last_undegraded":"2024-04-02T08:53:22.922204+0000","last_fullsized":"2024-04-02T08:53:22.922204+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:18:23.600237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.911452+0000","last_change":"2024-04-02T08:53:22.906555+0000","last_active":"2024-04-02T08:53:22.911452+0000","last_peered":"2024-04-02T08:53:22.911452+0000","last_clean":"2024-04-02T08:53:22.911452+0000","last_became_active":"2024-04-02T08:53:22.906230+0000","last_became_peered":"2024-04-02T08:53:22.906230+0000","last_unstale":"2024-04-02T08:53:22.911452+0000","last_undegraded":"2024-04-02T08:53:22.911452+0000","last_fullsized":"2024-04-02T08:53:22.911452+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:33.443826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"185'2","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896122+0000","last_change":"2024-04-02T08:53:22.406651+0000","last_active":"2024-04-02T08:53:23.896122+0000","last_peered":"2024-04-02T08:53:23.896122+0000","last_clean":"2024-04-02T08:53:23.896122+0000","last_became_active":"2024-04-02T08:53:22.406085+0000","last_became_peered":"2024-04-02T08:53:22.406085+0000","last_unstale":"2024-04-02T08:53:23.896122+0000","last_undegraded":"2024-04-02T08:53:23.896122+0000","last_fullsized":"2024-04-02T08:53:23.896122+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:09.753474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.917576+0000","last_change":"2024-04-02T08:53:22.911150+0000","last_active":"2024-04-02T08:53:22.917576+0000","last_peered":"2024-04-02T08:53:22.917576+0000","last_clean":"2024-04-02T08:53:22.917576+0000","last_became_active":"2024-04-02T08:53:22.904223+0000","last_became_peered":"2024-04-02T08:53:22.904223+0000","last_unstale":"2024-04-02T08:53:22.917576+0000","last_undegraded":"2024-04-02T08:53:22.917576+0000","last_fullsized":"2024-04-02T08:53:22.917576+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:35:49.453211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":11,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895898+0000","last_change":"2024-04-02T08:53:22.401280+0000","last_active":"2024-04-02T08:53:23.895898+0000","last_peered":"2024-04-02T08:53:23.895898+0000","last_clean":"2024-04-02T08:53:23.895898+0000","last_became_active":"2024-04-02T08:53:22.400999+0000","last_became_peered":"2024-04-02T08:53:22.400999+0000","last_unstale":"2024-04-02T08:53:23.895898+0000","last_undegraded":"2024-04-02T08:53:23.895898+0000","last_fullsized":"2024-04-02T08:53:23.895898+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:12.895858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896520+0000","last_change":"2024-04-02T08:53:22.910543+0000","last_active":"2024-04-02T08:53:23.896520+0000","last_peered":"2024-04-02T08:53:23.896520+0000","last_clean":"2024-04-02T08:53:23.896520+0000","last_became_active":"2024-04-02T08:53:22.909429+0000","last_became_peered":"2024-04-02T08:53:22.909429+0000","last_unstale":"2024-04-02T08:53:23.896520+0000","last_undegraded":"2024-04-02T08:53:23.896520+0000","last_fullsized":"2024-04-02T08:53:23.896520+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:08:21.301397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.903502+0000","last_change":"2024-04-02T08:53:22.405873+0000","last_active":"2024-04-02T08:53:22.903502+0000","last_peered":"2024-04-02T08:53:22.903502+0000","last_clean":"2024-04-02T08:53:22.903502+0000","last_became_active":"2024-04-02T08:53:22.405303+0000","last_became_peered":"2024-04-02T08:53:22.405303+0000","last_unstale":"2024-04-02T08:53:22.903502+0000","last_undegraded":"2024-04-02T08:53:22.903502+0000","last_fullsized":"2024-04-02T08:53:22.903502+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:12.895858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.910718+0000","last_change":"2024-04-02T08:53:22.906928+0000","last_active":"2024-04-02T08:53:22.910718+0000","last_peered":"2024-04-02T08:53:22.910718+0000","last_clean":"2024-04-02T08:53:22.910718+0000","last_became_active":"2024-04-02T08:53:22.906668+0000","last_became_peered":"2024-04-02T08:53:22.906668+0000","last_unstale":"2024-04-02T08:53:22.910718+0000","last_undegraded":"2024-04-02T08:53:22.910718+0000","last_fullsized":"2024-04-02T08:53:22.910718+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:47:39.254268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"185'8","reported_seq":18,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.920515+0000","last_change":"2024-04-02T08:53:22.398497+0000","last_active":"2024-04-02T08:53:22.920515+0000","last_peered":"2024-04-02T08:53:22.920515+0000","last_clean":"2024-04-02T08:53:22.920515+0000","last_became_active":"2024-04-02T08:53:22.398238+0000","last_became_peered":"2024-04-02T08:53:22.398238+0000","last_unstale":"2024-04-02T08:53:22.920515+0000","last_undegraded":"2024-04-02T08:53:22.920515+0000","last_fullsized":"2024-04-02T08:53:22.920515+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:19.432754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.920803+0000","last_change":"2024-04-02T08:53:22.916024+0000","last_active":"2024-04-02T08:53:22.920803+0000","last_peered":"2024-04-02T08:53:22.920803+0000","last_clean":"2024-04-02T08:53:22.920803+0000","last_became_active":"2024-04-02T08:53:22.915538+0000","last_became_peered":"2024-04-02T08:53:22.915538+0000","last_unstale":"2024-04-02T08:53:22.920803+0000","last_undegraded":"2024-04-02T08:53:22.920803+0000","last_fullsized":"2024-04-02T08:53:22.920803+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:16:12.933857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.f","version":"185'1","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915574+0000","last_change":"2024-04-02T08:53:22.404220+0000","last_active":"2024-04-02T08:53:22.915574+0000","last_peered":"2024-04-02T08:53:22.915574+0000","last_clean":"2024-04-02T08:53:22.915574+0000","last_became_active":"2024-04-02T08:53:22.403960+0000","last_became_peered":"2024-04-02T08:53:22.403960+0000","last_unstale":"2024-04-02T08:53:22.915574+0000","last_undegraded":"2024-04-02T08:53:22.915574+0000","last_fullsized":"2024-04-02T08:53:22.915574+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:53:21.563617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896503+0000","last_change":"2024-04-02T08:53:22.900299+0000","last_active":"2024-04-02T08:53:23.896503+0000","last_peered":"2024-04-02T08:53:23.896503+0000","last_clean":"2024-04-02T08:53:23.896503+0000","last_became_active":"2024-04-02T08:53:22.899946+0000","last_became_peered":"2024-04-02T08:53:22.899946+0000","last_unstale":"2024-04-02T08:53:23.896503+0000","last_undegraded":"2024-04-02T08:53:23.896503+0000","last_fullsized":"2024-04-02T08:53:23.896503+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:37:20.875182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"185'2","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915044+0000","last_change":"2024-04-02T08:53:22.396923+0000","last_active":"2024-04-02T08:53:22.915044+0000","last_peered":"2024-04-02T08:53:22.915044+0000","last_clean":"2024-04-02T08:53:22.915044+0000","last_became_active":"2024-04-02T08:53:22.396312+0000","last_became_peered":"2024-04-02T08:53:22.396312+0000","last_unstale":"2024-04-02T08:53:22.915044+0000","last_undegraded":"2024-04-02T08:53:22.915044+0000","last_fullsized":"2024-04-02T08:53:22.915044+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:41:39.706658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895731+0000","last_change":"2024-04-02T08:53:22.905459+0000","last_active":"2024-04-02T08:53:23.895731+0000","last_peered":"2024-04-02T08:53:23.895731+0000","last_clean":"2024-04-02T08:53:23.895731+0000","last_became_active":"2024-04-02T08:53:22.904516+0000","last_became_peered":"2024-04-02T08:53:22.904516+0000","last_unstale":"2024-04-02T08:53:23.895731+0000","last_undegraded":"2024-04-02T08:53:23.895731+0000","last_fullsized":"2024-04-02T08:53:23.895731+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:59:28.052580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.d","version":"185'1","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.920209+0000","last_change":"2024-04-02T08:53:22.400578+0000","last_active":"2024-04-02T08:53:22.920209+0000","last_peered":"2024-04-02T08:53:22.920209+0000","last_clean":"2024-04-02T08:53:22.920209+0000","last_became_active":"2024-04-02T08:53:22.400120+0000","last_became_peered":"2024-04-02T08:53:22.400120+0000","last_unstale":"2024-04-02T08:53:22.920209+0000","last_undegraded":"2024-04-02T08:53:22.920209+0000","last_fullsized":"2024-04-02T08:53:22.920209+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:41.730302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.912426+0000","last_change":"2024-04-02T08:53:22.901564+0000","last_active":"2024-04-02T08:53:22.912426+0000","last_peered":"2024-04-02T08:53:22.912426+0000","last_clean":"2024-04-02T08:53:22.912426+0000","last_became_active":"2024-04-02T08:53:22.901281+0000","last_became_peered":"2024-04-02T08:53:22.901281+0000","last_unstale":"2024-04-02T08:53:22.912426+0000","last_undegraded":"2024-04-02T08:53:22.912426+0000","last_fullsized":"2024-04-02T08:53:22.912426+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:10:25.277292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":11,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.891487+0000","last_change":"2024-04-02T08:53:22.405963+0000","last_active":"2024-04-02T08:53:23.891487+0000","last_peered":"2024-04-02T08:53:23.891487+0000","last_clean":"2024-04-02T08:53:23.891487+0000","last_became_active":"2024-04-02T08:53:22.405028+0000","last_became_peered":"2024-04-02T08:53:22.405028+0000","last_unstale":"2024-04-02T08:53:23.891487+0000","last_undegraded":"2024-04-02T08:53:23.891487+0000","last_fullsized":"2024-04-02T08:53:23.891487+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:33:24.291273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.908222+0000","last_change":"2024-04-02T08:53:22.900003+0000","last_active":"2024-04-02T08:53:22.908222+0000","last_peered":"2024-04-02T08:53:22.908222+0000","last_clean":"2024-04-02T08:53:22.908222+0000","last_became_active":"2024-04-02T08:53:22.897858+0000","last_became_peered":"2024-04-02T08:53:22.897858+0000","last_unstale":"2024-04-02T08:53:22.908222+0000","last_undegraded":"2024-04-02T08:53:22.908222+0000","last_fullsized":"2024-04-02T08:53:22.908222+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:20:24.495239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.b","version":"185'2","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.919532+0000","last_change":"2024-04-02T08:53:22.398735+0000","last_active":"2024-04-02T08:53:22.919532+0000","last_peered":"2024-04-02T08:53:22.919532+0000","last_clean":"2024-04-02T08:53:22.919532+0000","last_became_active":"2024-04-02T08:53:22.398441+0000","last_became_peered":"2024-04-02T08:53:22.398441+0000","last_unstale":"2024-04-02T08:53:22.919532+0000","last_undegraded":"2024-04-02T08:53:22.919532+0000","last_fullsized":"2024-04-02T08:53:22.919532+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:28.073575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.918072+0000","last_change":"2024-04-02T08:53:22.914016+0000","last_active":"2024-04-02T08:53:22.918072+0000","last_peered":"2024-04-02T08:53:22.918072+0000","last_clean":"2024-04-02T08:53:22.918072+0000","last_became_active":"2024-04-02T08:53:22.913758+0000","last_became_peered":"2024-04-02T08:53:22.913758+0000","last_unstale":"2024-04-02T08:53:22.918072+0000","last_undegraded":"2024-04-02T08:53:22.918072+0000","last_fullsized":"2024-04-02T08:53:22.918072+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:02:38.382306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":11,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.891132+0000","last_change":"2024-04-02T08:53:22.397314+0000","last_active":"2024-04-02T08:53:23.891132+0000","last_peered":"2024-04-02T08:53:23.891132+0000","last_clean":"2024-04-02T08:53:23.891132+0000","last_became_active":"2024-04-02T08:53:22.395250+0000","last_became_peered":"2024-04-02T08:53:22.395250+0000","last_unstale":"2024-04-02T08:53:23.891132+0000","last_undegraded":"2024-04-02T08:53:23.891132+0000","last_fullsized":"2024-04-02T08:53:23.891132+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:47:38.247446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.908129+0000","last_change":"2024-04-02T08:53:22.900240+0000","last_active":"2024-04-02T08:53:22.908129+0000","last_peered":"2024-04-02T08:53:22.908129+0000","last_clean":"2024-04-02T08:53:22.908129+0000","last_became_active":"2024-04-02T08:53:22.899975+0000","last_became_peered":"2024-04-02T08:53:22.899975+0000","last_unstale":"2024-04-02T08:53:22.908129+0000","last_undegraded":"2024-04-02T08:53:22.908129+0000","last_fullsized":"2024-04-02T08:53:22.908129+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:20:20.210105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":11,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895402+0000","last_change":"2024-04-02T08:53:22.405750+0000","last_active":"2024-04-02T08:53:23.895402+0000","last_peered":"2024-04-02T08:53:23.895402+0000","last_clean":"2024-04-02T08:53:23.895402+0000","last_became_active":"2024-04-02T08:53:22.405394+0000","last_became_peered":"2024-04-02T08:53:22.405394+0000","last_unstale":"2024-04-02T08:53:23.895402+0000","last_undegraded":"2024-04-02T08:53:23.895402+0000","last_fullsized":"2024-04-02T08:53:23.895402+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:28.073575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.920856+0000","last_change":"2024-04-02T08:53:22.917070+0000","last_active":"2024-04-02T08:53:22.920856+0000","last_peered":"2024-04-02T08:53:22.920856+0000","last_clean":"2024-04-02T08:53:22.920856+0000","last_became_active":"2024-04-02T08:53:22.916562+0000","last_became_peered":"2024-04-02T08:53:22.916562+0000","last_unstale":"2024-04-02T08:53:22.920856+0000","last_undegraded":"2024-04-02T08:53:22.920856+0000","last_fullsized":"2024-04-02T08:53:22.920856+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:08:21.301397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":11,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.891140+0000","last_change":"2024-04-02T08:53:22.391813+0000","last_active":"2024-04-02T08:53:23.891140+0000","last_peered":"2024-04-02T08:53:23.891140+0000","last_clean":"2024-04-02T08:53:23.891140+0000","last_became_active":"2024-04-02T08:53:22.391483+0000","last_became_peered":"2024-04-02T08:53:22.391483+0000","last_unstale":"2024-04-02T08:53:23.891140+0000","last_undegraded":"2024-04-02T08:53:23.891140+0000","last_fullsized":"2024-04-02T08:53:23.891140+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:45:43.343310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.911149+0000","last_change":"2024-04-02T08:53:22.904782+0000","last_active":"2024-04-02T08:53:22.911149+0000","last_peered":"2024-04-02T08:53:22.911149+0000","last_clean":"2024-04-02T08:53:22.911149+0000","last_became_active":"2024-04-02T08:53:22.901209+0000","last_became_peered":"2024-04-02T08:53:22.901209+0000","last_unstale":"2024-04-02T08:53:22.911149+0000","last_undegraded":"2024-04-02T08:53:22.911149+0000","last_fullsized":"2024-04-02T08:53:22.911149+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:31.589774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.7","version":"185'2","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896349+0000","last_change":"2024-04-02T08:53:22.403624+0000","last_active":"2024-04-02T08:53:23.896349+0000","last_peered":"2024-04-02T08:53:23.896349+0000","last_clean":"2024-04-02T08:53:23.896349+0000","last_became_active":"2024-04-02T08:53:22.401306+0000","last_became_peered":"2024-04-02T08:53:22.401306+0000","last_unstale":"2024-04-02T08:53:23.896349+0000","last_undegraded":"2024-04-02T08:53:23.896349+0000","last_fullsized":"2024-04-02T08:53:23.896349+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:19:11.682642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896408+0000","last_change":"2024-04-02T08:53:22.904129+0000","last_active":"2024-04-02T08:53:23.896408+0000","last_peered":"2024-04-02T08:53:23.896408+0000","last_clean":"2024-04-02T08:53:23.896408+0000","last_became_active":"2024-04-02T08:53:22.901928+0000","last_became_peered":"2024-04-02T08:53:22.901928+0000","last_unstale":"2024-04-02T08:53:23.896408+0000","last_undegraded":"2024-04-02T08:53:23.896408+0000","last_fullsized":"2024-04-02T08:53:23.896408+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:35:40.545122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":11,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.891323+0000","last_change":"2024-04-02T08:53:22.405805+0000","last_active":"2024-04-02T08:53:23.891323+0000","last_peered":"2024-04-02T08:53:23.891323+0000","last_clean":"2024-04-02T08:53:23.891323+0000","last_became_active":"2024-04-02T08:53:22.405500+0000","last_became_peered":"2024-04-02T08:53:22.405500+0000","last_unstale":"2024-04-02T08:53:23.891323+0000","last_undegraded":"2024-04-02T08:53:23.891323+0000","last_fullsized":"2024-04-02T08:53:23.891323+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:47.690639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895991+0000","last_change":"2024-04-02T08:53:22.893882+0000","last_active":"2024-04-02T08:53:23.895991+0000","last_peered":"2024-04-02T08:53:23.895991+0000","last_clean":"2024-04-02T08:53:23.895991+0000","last_became_active":"2024-04-02T08:53:22.893112+0000","last_became_peered":"2024-04-02T08:53:22.893112+0000","last_unstale":"2024-04-02T08:53:23.895991+0000","last_undegraded":"2024-04-02T08:53:23.895991+0000","last_fullsized":"2024-04-02T08:53:23.895991+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:10:25.277292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.888349+0000","last_change":"2024-04-02T08:53:22.406825+0000","last_active":"2024-04-02T08:53:22.888349+0000","last_peered":"2024-04-02T08:53:22.888349+0000","last_clean":"2024-04-02T08:53:22.888349+0000","last_became_active":"2024-04-02T08:53:22.406525+0000","last_became_peered":"2024-04-02T08:53:22.406525+0000","last_unstale":"2024-04-02T08:53:22.888349+0000","last_undegraded":"2024-04-02T08:53:22.888349+0000","last_fullsized":"2024-04-02T08:53:22.888349+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:49:57.573914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896254+0000","last_change":"2024-04-02T08:53:22.910421+0000","last_active":"2024-04-02T08:53:23.896254+0000","last_peered":"2024-04-02T08:53:23.896254+0000","last_clean":"2024-04-02T08:53:23.896254+0000","last_became_active":"2024-04-02T08:53:22.910122+0000","last_became_peered":"2024-04-02T08:53:22.910122+0000","last_unstale":"2024-04-02T08:53:23.896254+0000","last_undegraded":"2024-04-02T08:53:23.896254+0000","last_fullsized":"2024-04-02T08:53:23.896254+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:41:40.713480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.899968+0000","last_change":"2024-04-02T08:53:22.395003+0000","last_active":"2024-04-02T08:53:22.899968+0000","last_peered":"2024-04-02T08:53:22.899968+0000","last_clean":"2024-04-02T08:53:22.899968+0000","last_became_active":"2024-04-02T08:53:22.394674+0000","last_became_peered":"2024-04-02T08:53:22.394674+0000","last_unstale":"2024-04-02T08:53:22.899968+0000","last_undegraded":"2024-04-02T08:53:22.899968+0000","last_fullsized":"2024-04-02T08:53:22.899968+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:00:11.383903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895700+0000","last_change":"2024-04-02T08:53:22.908654+0000","last_active":"2024-04-02T08:53:23.895700+0000","last_peered":"2024-04-02T08:53:23.895700+0000","last_clean":"2024-04-02T08:53:23.895700+0000","last_became_active":"2024-04-02T08:53:22.904691+0000","last_became_peered":"2024-04-02T08:53:22.904691+0000","last_unstale":"2024-04-02T08:53:23.895700+0000","last_undegraded":"2024-04-02T08:53:23.895700+0000","last_fullsized":"2024-04-02T08:53:23.895700+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:20:20.210105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.889255+0000","last_change":"2024-04-02T08:53:22.401575+0000","last_active":"2024-04-02T08:53:22.889255+0000","last_peered":"2024-04-02T08:53:22.889255+0000","last_clean":"2024-04-02T08:53:22.889255+0000","last_became_active":"2024-04-02T08:53:22.401137+0000","last_became_peered":"2024-04-02T08:53:22.401137+0000","last_unstale":"2024-04-02T08:53:22.889255+0000","last_undegraded":"2024-04-02T08:53:22.889255+0000","last_fullsized":"2024-04-02T08:53:22.889255+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:44.571285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.913207+0000","last_change":"2024-04-02T08:53:22.907747+0000","last_active":"2024-04-02T08:53:22.913207+0000","last_peered":"2024-04-02T08:53:22.913207+0000","last_clean":"2024-04-02T08:53:22.913207+0000","last_became_active":"2024-04-02T08:53:22.907496+0000","last_became_peered":"2024-04-02T08:53:22.907496+0000","last_unstale":"2024-04-02T08:53:22.913207+0000","last_undegraded":"2024-04-02T08:53:22.913207+0000","last_fullsized":"2024-04-02T08:53:22.913207+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:59:13.633190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.903595+0000","last_change":"2024-04-02T08:53:22.408202+0000","last_active":"2024-04-02T08:53:22.903595+0000","last_peered":"2024-04-02T08:53:22.903595+0000","last_clean":"2024-04-02T08:53:22.903595+0000","last_became_active":"2024-04-02T08:53:22.407929+0000","last_became_peered":"2024-04-02T08:53:22.407929+0000","last_unstale":"2024-04-02T08:53:22.903595+0000","last_undegraded":"2024-04-02T08:53:22.903595+0000","last_fullsized":"2024-04-02T08:53:22.903595+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:10:29.755694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915578+0000","last_change":"2024-04-02T08:53:22.909704+0000","last_active":"2024-04-02T08:53:22.915578+0000","last_peered":"2024-04-02T08:53:22.915578+0000","last_clean":"2024-04-02T08:53:22.915578+0000","last_became_active":"2024-04-02T08:53:22.909119+0000","last_became_peered":"2024-04-02T08:53:22.909119+0000","last_unstale":"2024-04-02T08:53:22.915578+0000","last_undegraded":"2024-04-02T08:53:22.915578+0000","last_fullsized":"2024-04-02T08:53:22.915578+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:32:52.919204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.887259+0000","last_change":"2024-04-02T08:53:22.407011+0000","last_active":"2024-04-02T08:53:22.887259+0000","last_peered":"2024-04-02T08:53:22.887259+0000","last_clean":"2024-04-02T08:53:22.887259+0000","last_became_active":"2024-04-02T08:53:22.406722+0000","last_became_peered":"2024-04-02T08:53:22.406722+0000","last_unstale":"2024-04-02T08:53:22.887259+0000","last_undegraded":"2024-04-02T08:53:22.887259+0000","last_fullsized":"2024-04-02T08:53:22.887259+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:10:29.755694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.908252+0000","last_change":"2024-04-02T08:53:22.898910+0000","last_active":"2024-04-02T08:53:22.908252+0000","last_peered":"2024-04-02T08:53:22.908252+0000","last_clean":"2024-04-02T08:53:22.908252+0000","last_became_active":"2024-04-02T08:53:22.898627+0000","last_became_peered":"2024-04-02T08:53:22.898627+0000","last_unstale":"2024-04-02T08:53:22.908252+0000","last_undegraded":"2024-04-02T08:53:22.908252+0000","last_fullsized":"2024-04-02T08:53:22.908252+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:47:04.773777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.886684+0000","last_change":"2024-04-02T08:53:22.405009+0000","last_active":"2024-04-02T08:53:22.886684+0000","last_peered":"2024-04-02T08:53:22.886684+0000","last_clean":"2024-04-02T08:53:22.886684+0000","last_became_active":"2024-04-02T08:53:22.404071+0000","last_became_peered":"2024-04-02T08:53:22.404071+0000","last_unstale":"2024-04-02T08:53:22.886684+0000","last_undegraded":"2024-04-02T08:53:22.886684+0000","last_fullsized":"2024-04-02T08:53:22.886684+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:15:46.243319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.911310+0000","last_change":"2024-04-02T08:53:22.907997+0000","last_active":"2024-04-02T08:53:22.911310+0000","last_peered":"2024-04-02T08:53:22.911310+0000","last_clean":"2024-04-02T08:53:22.911310+0000","last_became_active":"2024-04-02T08:53:22.905426+0000","last_became_peered":"2024-04-02T08:53:22.905426+0000","last_unstale":"2024-04-02T08:53:22.911310+0000","last_undegraded":"2024-04-02T08:53:22.911310+0000","last_fullsized":"2024-04-02T08:53:22.911310+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:47:04.773777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.890347+0000","last_change":"2024-04-02T08:53:22.406009+0000","last_active":"2024-04-02T08:53:22.890347+0000","last_peered":"2024-04-02T08:53:22.890347+0000","last_clean":"2024-04-02T08:53:22.890347+0000","last_became_active":"2024-04-02T08:53:22.405740+0000","last_became_peered":"2024-04-02T08:53:22.405740+0000","last_unstale":"2024-04-02T08:53:22.890347+0000","last_undegraded":"2024-04-02T08:53:22.890347+0000","last_fullsized":"2024-04-02T08:53:22.890347+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:39:15.431924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"175'293","reported_seq":972,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.392303+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T08:53:22.392303+0000","last_peered":"2024-04-02T08:53:22.392303+0000","last_clean":"2024-04-02T08:53:22.392303+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T08:53:22.392303+0000","last_undegraded":"2024-04-02T08:53:22.392303+0000","last_fullsized":"2024-04-02T08:53:22.392303+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575765,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55532,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316308,"statfs":{"total":96636764160,"available":96579899392,"internally_reserved":0,"allocated":2326528,"data_stored":1166204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:53:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.70699999999999996,"15min":0.71599999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.86199999999999999,"5min":1.069,"15min":1.069},"last":0.432},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.69899999999999995,"15min":0.70899999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.85399999999999998,"5min":1.0129999999999999,"15min":1.0169999999999999},"last":0.67600000000000005}]},{"osd":1,"last update":"Tue Apr 2 08:53:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.70899999999999996,"15min":0.72899999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.028},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.70299999999999996,"15min":0.70899999999999996},"min":{"1min":0.46000000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82799999999999996,"5min":1.016,"15min":1.016},"last":0.53000000000000003}]},{"osd":2,"last update":"Tue Apr 2 08:53:17 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.73399999999999999,"15min":0.752},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.82099999999999995,"5min":1.105,"15min":1.105},"last":0.498},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.79600000000000004,"15min":0.80000000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.497},"max":{"1min":0.93400000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.60199999999999998}]},{"osd":3,"last update":"Tue Apr 2 08:53:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.72499999999999998,"15min":0.73399999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":1.073,"15min":1.073},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.745,"15min":0.75600000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.97699999999999998,"5min":1.048,"15min":1.056},"last":0.51900000000000002}]},{"osd":4,"last update":"Tue Apr 2 08:53:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.35199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.84299999999999997,"5min":0.90700000000000003,"15min":0.93200000000000005},"last":0.622},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71299999999999997,"15min":0.70699999999999996},"min":{"1min":0.55200000000000005,"5min":0.44600000000000001,"15min":0.376},"max":{"1min":0.89200000000000002,"5min":0.997,"15min":1.014},"last":0.63100000000000001}]},{"osd":5,"last update":"Tue Apr 2 08:53:17 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.68500000000000005,"15min":0.66800000000000004},"min":{"1min":0.435,"5min":0.435,"15min":0.38700000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.70599999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.873,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.58099999999999996}]},{"osd":6,"last update":"Tue Apr 2 08:53:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.66600000000000004,"15min":0.66300000000000003},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.46500000000000002},"max":{"1min":0.81599999999999995,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91800000000000004,"5min":0.98699999999999999,"15min":1.0149999999999999},"last":0.69299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575763,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41572,"kb_used_data":2264,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330268,"statfs":{"total":96636764160,"available":96594194432,"internally_reserved":0,"allocated":2318336,"data_stored":1159518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.72299999999999998,"15min":0.72499999999999998},"min":{"1min":0.38100000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.1990000000000001,"5min":1.847,"15min":1.847},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.113,"5min":0.82799999999999996,"15min":0.76600000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":10.914,"5min":10.914,"15min":10.914},"last":10.914}]},{"osd":1,"last update":"Tue Apr 2 08:53:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":0.80500000000000005,"15min":0.753},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":10.522,"5min":10.522,"15min":10.522},"last":10.522},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.753,"15min":0.748},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.4139999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.4139999999999999}]},{"osd":2,"last update":"Tue Apr 2 08:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.72899999999999998,"15min":0.746},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.141,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.107,"5min":0.81000000000000005,"15min":0.755},"min":{"1min":0.42999999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":10.779,"5min":10.779,"15min":10.779},"last":10.779}]},{"osd":3,"last update":"Tue Apr 2 08:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.749,"15min":0.754},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.238,"5min":1.722,"15min":1.722},"last":1.119},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77900000000000003,"15min":0.77500000000000002},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":2.5209999999999999}]},{"osd":4,"last update":"Tue Apr 2 08:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63500000000000001,"15min":0.625},"min":{"1min":0.40500000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.278,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.278},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67800000000000005,"15min":0.65400000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.069,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.70999999999999996}]},{"osd":5,"last update":"Tue Apr 2 08:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.67300000000000004,"15min":0.64400000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.40100000000000002},"max":{"1min":1.3280000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.3280000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.73399999999999999,"15min":0.70599999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.43099999999999999},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.4790000000000001}]},{"osd":7,"last update":"Tue Apr 2 08:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.748,"15min":0.71599999999999997},"min":{"1min":0.58899999999999997,"5min":0.41999999999999998,"15min":0.40999999999999998},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":1.46},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.74199999999999999,"15min":0.71999999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":1.1910000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47936,"kb_used_data":2864,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323904,"statfs":{"total":96636764160,"available":96587677696,"internally_reserved":0,"allocated":2932736,"data_stored":1751598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:53:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.46000000000000002,"15min":0.46400000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.59099999999999997,"5min":0.80100000000000005,"15min":0.81100000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.51400000000000001,"15min":0.51800000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34100000000000003},"max":{"1min":0.68300000000000005,"5min":0.876,"15min":0.94799999999999995},"last":0.438}]},{"osd":2,"last update":"Tue Apr 2 08:53:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.54100000000000004,"15min":0.53000000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.65400000000000003,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.63200000000000001,"15min":0.61399999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73899999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.53200000000000003}]},{"osd":3,"last update":"Tue Apr 2 08:53:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60899999999999999,"15min":0.59799999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.69099999999999995,"5min":1.137,"15min":1.226},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69299999999999995,"5min":0.93200000000000005,"15min":1.0700000000000001},"last":0.495}]},{"osd":4,"last update":"Tue Apr 2 08:53:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.05,"5min":1.1140000000000001,"15min":1.321},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.81999999999999995,"15min":0.80500000000000005},"min":{"1min":0.55800000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2310000000000001,"5min":1.381,"15min":1.381},"last":0.66900000000000004}]},{"osd":5,"last update":"Tue Apr 2 08:53:08 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.80900000000000005,"15min":0.80200000000000005},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0580000000000001,"5min":1.131,"15min":1.131},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.751,"5min":0.83399999999999996,"15min":0.81699999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53600000000000003},"max":{"1min":1.081,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.78300000000000003}]},{"osd":6,"last update":"Tue Apr 2 08:53:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.83499999999999996,"15min":0.82999999999999996},"min":{"1min":0.64600000000000002,"5min":0.627,"15min":0.57299999999999995},"max":{"1min":1.147,"5min":1.2809999999999999,"15min":1.3759999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.83599999999999997,"15min":0.82899999999999996},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.53700000000000003},"max":{"1min":1.1220000000000001,"5min":1.1839999999999999,"15min":1.242},"last":0.63}]},{"osd":7,"last update":"Tue Apr 2 08:53:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.85699999999999998,"15min":0.84399999999999997},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.56299999999999994},"max":{"1min":1.1850000000000001,"5min":1.347,"15min":1.347},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.84699999999999998,"15min":0.84399999999999997},"min":{"1min":0.627,"5min":0.623,"15min":0.621},"max":{"1min":1.2090000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.60499999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47328,"kb_used_data":2260,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324512,"statfs":{"total":96636764160,"available":96588300288,"internally_reserved":0,"allocated":2314240,"data_stored":1158535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.45400000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60099999999999998,"5min":0.64100000000000001,"15min":0.75700000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.5,"15min":0.498},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73499999999999999,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.41799999999999998}]},{"osd":2,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.50700000000000001,"15min":0.50800000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.76200000000000001,"5min":0.80700000000000005,"15min":0.83299999999999996},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.54500000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73999999999999999,"5min":0.79700000000000004,"15min":0.80900000000000005},"last":0.45700000000000002}]},{"osd":3,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55700000000000005,"15min":0.56699999999999995},"min":{"1min":0.40799999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.86799999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63,"15min":0.64300000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81999999999999995,"5min":0.88100000000000001,"15min":0.91900000000000004},"last":0.64900000000000002}]},{"osd":4,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.83199999999999996,"15min":0.81899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.999,"5min":1.0980000000000001,"15min":1.236},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.83899999999999997,"15min":0.82899999999999996},"min":{"1min":0.61899999999999999,"5min":0.60199999999999998,"15min":0.56200000000000006},"max":{"1min":1.056,"5min":1.056,"15min":1.2929999999999999},"last":0.69699999999999995}]},{"osd":5,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.83199999999999996,"15min":0.82699999999999996},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.57799999999999996},"max":{"1min":0.95599999999999996,"5min":1.0820000000000001,"15min":1.369},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.53700000000000003},"max":{"1min":0.98899999999999999,"5min":1.042,"15min":1.27},"last":0.504}]},{"osd":6,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.57999999999999996,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.052,"5min":1.052,"15min":1.1259999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.80200000000000005,"15min":0.79600000000000004},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.99099999999999999,"5min":1.089,"15min":1.262},"last":0.68100000000000005}]},{"osd":7,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.85999999999999999,"15min":0.84999999999999998},"min":{"1min":0.57099999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0680000000000001,"5min":1.125,"15min":1.321},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.83299999999999996,"15min":0.84199999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.083,"5min":1.1120000000000001,"15min":1.2989999999999999},"last":0.77400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575763,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47340,"kb_used_data":2264,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324500,"statfs":{"total":96636764160,"available":96588288000,"internally_reserved":0,"allocated":2318336,"data_stored":1160066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47599999999999998,"15min":0.45600000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.31},"max":{"1min":0.70099999999999996,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.309},{"interface":"front","average":{"1min":0.499,"5min":0.50800000000000001,"15min":0.50700000000000001},"min":{"1min":0.40000000000000002,"5min":0.35099999999999998,"15min":0.316},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":0.81000000000000005},"last":0.45000000000000001}]},{"osd":1,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54100000000000004,"15min":0.51700000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.313},"max":{"1min":0.751,"5min":1.262,"15min":1.262},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.39500000000000002,"5min":0.38,"15min":0.35599999999999998},"max":{"1min":0.72399999999999998,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.501}]},{"osd":3,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.39400000000000002,"5min":0.38500000000000001,"15min":0.30599999999999999},"max":{"1min":2.3039999999999998,"5min":2.3039999999999998,"15min":2.3039999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.65700000000000003,"15min":0.64000000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.45000000000000001},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.47299999999999998}]},{"osd":4,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.81200000000000006,"15min":0.80500000000000005},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.443},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80300000000000005,"15min":0.80600000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.42399999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.83799999999999997}]},{"osd":5,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.82099999999999995,"15min":0.80600000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.48399999999999999},"max":{"1min":2.177,"5min":2.177,"15min":2.177},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.83399999999999996,"15min":0.81799999999999995},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.47899999999999998},"max":{"1min":1.319,"5min":1.319,"15min":1.3600000000000001},"last":0.76300000000000001}]},{"osd":6,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.47499999999999998},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.84299999999999997,"15min":0.83199999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.50700000000000001},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":0.71599999999999997}]},{"osd":7,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.84699999999999998,"15min":0.83799999999999997},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.502},"max":{"1min":1.528,"5min":1.528,"15min":1.528},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.86199999999999999,"15min":0.84599999999999997},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.47399999999999998},"max":{"1min":2.258,"5min":2.258,"15min":2.258},"last":0.74299999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575763,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47368,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324472,"statfs":{"total":96636764160,"available":96588259328,"internally_reserved":0,"allocated":2342912,"data_stored":1167790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.44600000000000001,"15min":0.45100000000000001},"min":{"1min":0.315,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.627,"5min":0.64800000000000002,"15min":0.92700000000000005},"last":0.371},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.48499999999999999,"15min":0.49099999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.30599999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.87},"last":0.51000000000000001}]},{"osd":1,"last update":"Tue Apr 2 08:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.50600000000000001,"15min":0.50700000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.66600000000000004,"5min":0.90300000000000002,"15min":0.91400000000000003},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40100000000000002},"max":{"1min":0.69899999999999995,"5min":0.82999999999999996,"15min":0.92000000000000004},"last":0.47799999999999998}]},{"osd":2,"last update":"Tue Apr 2 08:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.623,"15min":0.63300000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78300000000000003,"5min":0.83599999999999997,"15min":0.86299999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.745,"5min":1.022,"15min":1.022},"last":0.52000000000000002}]},{"osd":4,"last update":"Tue Apr 2 08:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.79100000000000004,"15min":0.78300000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.95099999999999996,"5min":1.1000000000000001,"15min":1.167},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.80300000000000005,"15min":0.79600000000000004},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.94399999999999995,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.66500000000000004}]},{"osd":5,"last update":"Tue Apr 2 08:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.80900000000000005,"15min":0.79800000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.54600000000000004},"max":{"1min":0.97699999999999998,"5min":1.115,"15min":1.149},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.79900000000000004,"15min":0.79800000000000004},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.05,"5min":1.0760000000000001,"15min":1.1020000000000001},"last":0.74199999999999999}]},{"osd":6,"last update":"Tue Apr 2 08:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.80400000000000005,"15min":0.80100000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.99299999999999999,"5min":1.153,"15min":1.153},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.747,"5min":0.84299999999999997,"15min":0.82599999999999996},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.55600000000000005},"max":{"1min":1.0289999999999999,"5min":1.143,"15min":1.1870000000000001},"last":0.72499999999999998}]},{"osd":7,"last update":"Tue Apr 2 08:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.84799999999999998,"15min":0.82299999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97499999999999998,"5min":1.125,"15min":1.2070000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.752,"5min":0.85099999999999998,"15min":0.82999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0309999999999999,"5min":1.194,"15min":1.194},"last":0.69399999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575763,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59632,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312208,"statfs":{"total":96636764160,"available":96575700992,"internally_reserved":0,"allocated":2330624,"data_stored":1166238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.69199999999999995,"15min":0.70099999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89600000000000002,"5min":0.92400000000000004,"15min":1.0600000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.68500000000000005,"15min":0.69199999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.0429999999999999},"last":0.56000000000000005}]},{"osd":1,"last update":"Tue Apr 2 08:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.747,"15min":0.752},"min":{"1min":0.48199999999999998,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":1.121},"last":0.495},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78200000000000003,"15min":0.78000000000000003},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.49099999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1200000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Tue Apr 2 08:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.69999999999999996,"15min":0.71099999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.46600000000000003},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.1879999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71299999999999997,"15min":0.73099999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.1259999999999999},"last":0.53000000000000003}]},{"osd":3,"last update":"Tue Apr 2 08:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.75700000000000001,"15min":0.76500000000000001},"min":{"1min":0.60399999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.2589999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.74099999999999999,"15min":0.74399999999999999},"min":{"1min":0.54600000000000004,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.077},"last":0.69999999999999996}]},{"osd":5,"last update":"Tue Apr 2 08:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.59699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41599999999999998,"5min":0.40400000000000003,"15min":0.38400000000000001},"max":{"1min":0.877,"5min":0.90600000000000003,"15min":1.0940000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.90700000000000003,"5min":0.93899999999999995,"15min":1.073},"last":0.63200000000000001}]},{"osd":6,"last update":"Tue Apr 2 08:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.64000000000000001,"15min":0.63700000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.98199999999999998,"5min":1.022,"15min":1.022},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.68400000000000005,"15min":0.69399999999999995},"min":{"1min":0.48999999999999999,"5min":0.46899999999999997,"15min":0.439},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.1120000000000001},"last":0.72999999999999998}]},{"osd":7,"last update":"Tue Apr 2 08:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.67800000000000005,"15min":0.69699999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98599999999999999,"5min":1.048,"15min":1.129},"last":0.749},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.64500000000000002,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.3440000000000001},"last":0.76100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575766,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47916,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323924,"statfs":{"total":96636764160,"available":96587698176,"internally_reserved":0,"allocated":2912256,"data_stored":1749505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.70399999999999996,"15min":0.71499999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.377},"max":{"1min":0.84999999999999998,"5min":1.502,"15min":1.502},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.72899999999999998,"15min":0.72999999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.89800000000000002,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.752}]},{"osd":1,"last update":"Tue Apr 2 08:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71799999999999997,"15min":0.72399999999999998},"min":{"1min":0.53500000000000003,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.71999999999999997,"15min":0.73399999999999999},"min":{"1min":0.48099999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.91500000000000004,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.78400000000000003}]},{"osd":2,"last update":"Tue Apr 2 08:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.749,"15min":0.74099999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.49199999999999999},"max":{"1min":0.94399999999999995,"5min":1.232,"15min":1.232},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.73799999999999999,"15min":0.74199999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92500000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.624}]},{"osd":3,"last update":"Tue Apr 2 08:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.79300000000000004,"15min":0.78900000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.46800000000000003},"max":{"1min":0.93799999999999994,"5min":1.556,"15min":1.556},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.79200000000000004,"15min":0.79500000000000004},"min":{"1min":0.53400000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0029999999999999,"5min":1.0189999999999999,"15min":1.069},"last":0.76600000000000001}]},{"osd":4,"last update":"Tue Apr 2 08:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63,"15min":0.61599999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.32700000000000001},"max":{"1min":0.91700000000000004,"5min":0.97899999999999998,"15min":1.0189999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.66100000000000003},"min":{"1min":0.46300000000000002,"5min":0.44600000000000001,"15min":0.435},"max":{"1min":0.97899999999999998,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.82999999999999996}]},{"osd":6,"last update":"Tue Apr 2 08:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.66600000000000004,"15min":0.66800000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.44700000000000001},"max":{"1min":0.91600000000000004,"5min":0.99199999999999999,"15min":1.0640000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.68700000000000006,"15min":0.69499999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.377},"max":{"1min":0.879,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.72199999999999998}]},{"osd":7,"last update":"Tue Apr 2 08:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.72399999999999998,"15min":0.71799999999999997},"min":{"1min":0.441,"5min":0.441,"15min":0.41599999999999998},"max":{"1min":1.0049999999999999,"5min":1.423,"15min":1.423},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73799999999999999,"15min":0.745},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.44500000000000001},"max":{"1min":0.92600000000000005,"5min":1.038,"15min":1.038},"last":0.68400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T08:53:27.770 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T08:53:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.996+0000 7eff04200700 1 Processor -- start 2024-04-02T08:53:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.997+0000 7eff04200700 1 -- start start 2024-04-02T08:53:27.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.998+0000 7eff04200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efefc0d18b0 0x7efefc0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:27.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.998+0000 7eff04200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efefc0ceda0 con 0x7efefc160f10 2024-04-02T08:53:27.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.998+0000 7eff04200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efefc1612f0 con 0x7efefc161590 2024-04-02T08:53:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.998+0000 7eff04200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efefc0d1d30 con 0x7efefc0d18b0 2024-04-02T08:53:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.998+0000 7eff00e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efefc0d18b0 0x7efefc0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.998+0000 7eff01800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efefc160f10 0x7efefc0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46388/0 (socket says 172.21.15.89:46388) 2024-04-02T08:53:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.998+0000 7eff01800700 1 -- 172.21.15.89:0/3809555394 learned_addr learned my addr 172.21.15.89:0/3809555394 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.998+0000 7eff00e00700 1 -- 172.21.15.89:0/3809555394 >> v1:172.21.15.113:6789/0 conn(0x7efefc161590 legacy=0x7efefc0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.998+0000 7eff00e00700 1 -- 172.21.15.89:0/3809555394 >> v1:172.21.15.89:6789/0 conn(0x7efefc160f10 legacy=0x7efefc0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.998+0000 7eff00e00700 1 -- 172.21.15.89:0/3809555394 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc0ce470 con 0x7efefc0d18b0 2024-04-02T08:53:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.999+0000 7eff00e00700 1 --2- 172.21.15.89:0/3809555394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efefc0d18b0 0x7efefc0cf230 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7efee8009840 tx=0x7efee8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82566f49e7f118fb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:27.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.999+0000 7efefbe00700 1 -- 172.21.15.89:0/3809555394 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8004230 con 0x7efefc0d18b0 2024-04-02T08:53:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.999+0000 7efefbe00700 1 -- 172.21.15.89:0/3809555394 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee8004390 con 0x7efefc0d18b0 2024-04-02T08:53:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.999+0000 7efefbe00700 1 -- 172.21.15.89:0/3809555394 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee80036a0 con 0x7efefc0d18b0 2024-04-02T08:53:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.999+0000 7eff04200700 1 -- 172.21.15.89:0/3809555394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efefc0d18b0 msgr2=0x7efefc0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.999+0000 7eff04200700 1 --2- 172.21.15.89:0/3809555394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efefc0d18b0 0x7efefc0cf230 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7efee8009840 tx=0x7efee8009b50 comp rx=0 tx=0).stop 2024-04-02T08:53:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.999+0000 7eff04200700 1 -- 172.21.15.89:0/3809555394 shutdown_connections 2024-04-02T08:53:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:27.999+0000 7eff04200700 1 --2- 172.21.15.89:0/3809555394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efefc0d18b0 0x7efefc0cf230 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.000+0000 7eff04200700 1 -- 172.21.15.89:0/3809555394 >> 172.21.15.89:0/3809555394 conn(0x7efefc08aff0 msgr2=0x7efefc0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.000+0000 7eff04200700 1 -- 172.21.15.89:0/3809555394 shutdown_connections 2024-04-02T08:53:27.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.000+0000 7eff04200700 1 -- 172.21.15.89:0/3809555394 wait complete. 2024-04-02T08:53:28.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.001+0000 7eff04200700 1 Processor -- start 2024-04-02T08:53:28.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.001+0000 7eff04200700 1 -- start start 2024-04-02T08:53:28.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.002+0000 7eff04200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efefc0d18b0 0x7efefc1f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:28.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.002+0000 7eff04200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efefc160f10 0x7efefc145fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:28.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.002+0000 7eff04200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efefc161590 0x7efefc1569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:28.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.002+0000 7eff04200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc0b94b0 con 0x7efefc0d18b0 2024-04-02T08:53:28.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.002+0000 7eff04200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc159720 con 0x7efefc161590 2024-04-02T08:53:28.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.002+0000 7eff04200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc158b10 con 0x7efefc160f10 2024-04-02T08:53:28.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.002+0000 7eff01800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efefc0d18b0 0x7efefc1f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:28.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.002+0000 7eff01800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efefc0d18b0 0x7efefc1f1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54156/0 (socket says 172.21.15.89:54156) 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.002+0000 7eff01800700 1 -- 172.21.15.89:0/277547597 learned_addr learned my addr 172.21.15.89:0/277547597 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7eff02200700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efefc161590 0x7efefc1569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7eff00e00700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efefc160f10 0x7efefc145fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7eff01800700 1 -- 172.21.15.89:0/277547597 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efefc160f10 msgr2=0x7efefc145fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7eff01800700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efefc160f10 0x7efefc145fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7eff01800700 1 -- 172.21.15.89:0/277547597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efefc161590 msgr2=0x7efefc1569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7eff01800700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efefc161590 0x7efefc1569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7eff01800700 1 -- 172.21.15.89:0/277547597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee80094a0 con 0x7efefc0d18b0 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7eff02200700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efefc161590 0x7efefc1569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7eff00e00700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efefc160f10 0x7efefc145fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7eff01800700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efefc0d18b0 0x7efefc1f1060 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7efef000c430 tx=0x7efef000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7efef9600700 1 -- 172.21.15.89:0/277547597 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef0005d70 con 0x7efefc0d18b0 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7efef9600700 1 -- 172.21.15.89:0/277547597 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef00070d0 con 0x7efefc0d18b0 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7eff04200700 1 -- 172.21.15.89:0/277547597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efefc158df0 con 0x7efefc0d18b0 2024-04-02T08:53:28.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7efef9600700 1 -- 172.21.15.89:0/277547597 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef0007460 con 0x7efefc0d18b0 2024-04-02T08:53:28.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.003+0000 7eff04200700 1 -- 172.21.15.89:0/277547597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efefc1593b0 con 0x7efefc0d18b0 2024-04-02T08:53:28.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.004+0000 7eff04200700 1 -- 172.21.15.89:0/277547597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efefc067720 con 0x7efefc0d18b0 2024-04-02T08:53:28.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.005+0000 7efef9600700 1 -- 172.21.15.89:0/277547597 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7efef00075c0 con 0x7efefc0d18b0 2024-04-02T08:53:28.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.006+0000 7efef9600700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efedc06adb0 0x7efedc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:28.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.006+0000 7efef9600700 1 -- 172.21.15.89:0/277547597 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7efef0088b40 con 0x7efefc0d18b0 2024-04-02T08:53:28.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.006+0000 7eff00e00700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efedc06adb0 0x7efedc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:28.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.007+0000 7eff00e00700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efedc06adb0 0x7efedc06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efee80090b0 tx=0x7efee8008030 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:28.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.011+0000 7efef9600700 1 -- 172.21.15.89:0/277547597 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efef00539c0 con 0x7efefc0d18b0 2024-04-02T08:53:28.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.208+0000 7eff04200700 1 -- 172.21.15.89:0/277547597 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efefc0cea30 con 0x7efedc06adb0 2024-04-02T08:53:28.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.216+0000 7efef9600700 1 -- 172.21.15.89:0/277547597 <== mgr.9807 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288920 (secure 0 0 0) 0x7efee8003850 con 0x7efedc06adb0 2024-04-02T08:53:28.219 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:28.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.221+0000 7eff04200700 1 -- 172.21.15.89:0/277547597 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efedc06adb0 msgr2=0x7efedc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:28.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.221+0000 7eff04200700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efedc06adb0 0x7efedc06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efee80090b0 tx=0x7efee8008030 comp rx=0 tx=0).stop 2024-04-02T08:53:28.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.221+0000 7eff04200700 1 -- 172.21.15.89:0/277547597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efefc0d18b0 msgr2=0x7efefc1f1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:28.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.221+0000 7eff04200700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efefc0d18b0 0x7efefc1f1060 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7efef000c430 tx=0x7efef000c7f0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.221+0000 7eff04200700 1 -- 172.21.15.89:0/277547597 shutdown_connections 2024-04-02T08:53:28.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.221+0000 7eff04200700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7efedc06adb0 0x7efedc06d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.221+0000 7eff04200700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efefc161590 0x7efefc1569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.221+0000 7eff04200700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efefc0d18b0 0x7efefc1f1060 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.221+0000 7eff04200700 1 --2- 172.21.15.89:0/277547597 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efefc160f10 0x7efefc145fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.221+0000 7eff04200700 1 -- 172.21.15.89:0/277547597 >> 172.21.15.89:0/277547597 conn(0x7efefc08aff0 msgr2=0x7efefc0cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:28.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.221+0000 7eff04200700 1 -- 172.21.15.89:0/277547597 shutdown_connections 2024-04-02T08:53:28.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.221+0000 7eff04200700 1 -- 172.21.15.89:0/277547597 wait complete. 2024-04-02T08:53:28.224 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T08:53:28.238 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T08:53:26.738532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":394624,"kb_used_data":19332,"kb_used_omap":12,"kb_used_meta":375155,"kb_avail":754580096,"statfs":{"total":773094113280,"available":772690018304,"internally_reserved":0,"allocated":19795968,"data_stored":10479454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":384159288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.365739"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.917437+0000","last_change":"2024-04-02T08:53:22.911486+0000","last_active":"2024-04-02T08:53:22.917437+0000","last_peered":"2024-04-02T08:53:22.917437+0000","last_clean":"2024-04-02T08:53:22.917437+0000","last_became_active":"2024-04-02T08:53:22.909502+0000","last_became_peered":"2024-04-02T08:53:22.909502+0000","last_unstale":"2024-04-02T08:53:22.917437+0000","last_undegraded":"2024-04-02T08:53:22.917437+0000","last_fullsized":"2024-04-02T08:53:22.917437+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:29:45.566973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.913553+0000","last_change":"2024-04-02T08:53:22.910212+0000","last_active":"2024-04-02T08:53:22.913553+0000","last_peered":"2024-04-02T08:53:22.913553+0000","last_clean":"2024-04-02T08:53:22.913553+0000","last_became_active":"2024-04-02T08:53:22.908146+0000","last_became_peered":"2024-04-02T08:53:22.908146+0000","last_unstale":"2024-04-02T08:53:22.913553+0000","last_undegraded":"2024-04-02T08:53:22.913553+0000","last_fullsized":"2024-04-02T08:53:22.913553+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:57:08.290485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.908450+0000","last_change":"2024-04-02T08:53:22.899012+0000","last_active":"2024-04-02T08:53:22.908450+0000","last_peered":"2024-04-02T08:53:22.908450+0000","last_clean":"2024-04-02T08:53:22.908450+0000","last_became_active":"2024-04-02T08:53:22.898750+0000","last_became_peered":"2024-04-02T08:53:22.898750+0000","last_unstale":"2024-04-02T08:53:22.908450+0000","last_undegraded":"2024-04-02T08:53:22.908450+0000","last_fullsized":"2024-04-02T08:53:22.908450+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:30.142896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896089+0000","last_change":"2024-04-02T08:53:22.899906+0000","last_active":"2024-04-02T08:53:23.896089+0000","last_peered":"2024-04-02T08:53:23.896089+0000","last_clean":"2024-04-02T08:53:23.896089+0000","last_became_active":"2024-04-02T08:53:22.899274+0000","last_became_peered":"2024-04-02T08:53:22.899274+0000","last_unstale":"2024-04-02T08:53:23.896089+0000","last_undegraded":"2024-04-02T08:53:23.896089+0000","last_fullsized":"2024-04-02T08:53:23.896089+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:35:49.453211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.919529+0000","last_change":"2024-04-02T08:53:22.912546+0000","last_active":"2024-04-02T08:53:22.919529+0000","last_peered":"2024-04-02T08:53:22.919529+0000","last_clean":"2024-04-02T08:53:22.919529+0000","last_became_active":"2024-04-02T08:53:22.912084+0000","last_became_peered":"2024-04-02T08:53:22.912084+0000","last_unstale":"2024-04-02T08:53:22.919529+0000","last_undegraded":"2024-04-02T08:53:22.919529+0000","last_fullsized":"2024-04-02T08:53:22.919529+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:18:42.804000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895591+0000","last_change":"2024-04-02T08:53:22.909227+0000","last_active":"2024-04-02T08:53:23.895591+0000","last_peered":"2024-04-02T08:53:23.895591+0000","last_clean":"2024-04-02T08:53:23.895591+0000","last_became_active":"2024-04-02T08:53:22.908942+0000","last_became_peered":"2024-04-02T08:53:22.908942+0000","last_unstale":"2024-04-02T08:53:23.895591+0000","last_undegraded":"2024-04-02T08:53:23.895591+0000","last_fullsized":"2024-04-02T08:53:23.895591+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:57.619946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.916391+0000","last_change":"2024-04-02T08:53:22.913548+0000","last_active":"2024-04-02T08:53:22.916391+0000","last_peered":"2024-04-02T08:53:22.916391+0000","last_clean":"2024-04-02T08:53:22.916391+0000","last_became_active":"2024-04-02T08:53:22.913269+0000","last_became_peered":"2024-04-02T08:53:22.913269+0000","last_unstale":"2024-04-02T08:53:22.916391+0000","last_undegraded":"2024-04-02T08:53:22.916391+0000","last_fullsized":"2024-04-02T08:53:22.916391+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:20:20.210105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.909222+0000","last_change":"2024-04-02T08:53:22.906726+0000","last_active":"2024-04-02T08:53:22.909222+0000","last_peered":"2024-04-02T08:53:22.909222+0000","last_clean":"2024-04-02T08:53:22.909222+0000","last_became_active":"2024-04-02T08:53:22.906129+0000","last_became_peered":"2024-04-02T08:53:22.906129+0000","last_unstale":"2024-04-02T08:53:22.909222+0000","last_undegraded":"2024-04-02T08:53:22.909222+0000","last_fullsized":"2024-04-02T08:53:22.909222+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:15.249827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.910619+0000","last_change":"2024-04-02T08:53:22.909792+0000","last_active":"2024-04-02T08:53:22.910619+0000","last_peered":"2024-04-02T08:53:22.910619+0000","last_clean":"2024-04-02T08:53:22.910619+0000","last_became_active":"2024-04-02T08:53:22.908655+0000","last_became_peered":"2024-04-02T08:53:22.908655+0000","last_unstale":"2024-04-02T08:53:22.910619+0000","last_undegraded":"2024-04-02T08:53:22.910619+0000","last_fullsized":"2024-04-02T08:53:22.910619+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:33:25.298095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.919914+0000","last_change":"2024-04-02T08:53:22.914731+0000","last_active":"2024-04-02T08:53:22.919914+0000","last_peered":"2024-04-02T08:53:22.919914+0000","last_clean":"2024-04-02T08:53:22.919914+0000","last_became_active":"2024-04-02T08:53:22.913815+0000","last_became_peered":"2024-04-02T08:53:22.913815+0000","last_unstale":"2024-04-02T08:53:22.919914+0000","last_undegraded":"2024-04-02T08:53:22.919914+0000","last_fullsized":"2024-04-02T08:53:22.919914+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:49:10.627711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915715+0000","last_change":"2024-04-02T08:53:22.904769+0000","last_active":"2024-04-02T08:53:22.915715+0000","last_peered":"2024-04-02T08:53:22.915715+0000","last_clean":"2024-04-02T08:53:22.915715+0000","last_became_active":"2024-04-02T08:53:22.898177+0000","last_became_peered":"2024-04-02T08:53:22.898177+0000","last_unstale":"2024-04-02T08:53:22.915715+0000","last_undegraded":"2024-04-02T08:53:22.915715+0000","last_fullsized":"2024-04-02T08:53:22.915715+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:42:55.438534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.891377+0000","last_change":"2024-04-02T08:53:22.906882+0000","last_active":"2024-04-02T08:53:23.891377+0000","last_peered":"2024-04-02T08:53:23.891377+0000","last_clean":"2024-04-02T08:53:23.891377+0000","last_became_active":"2024-04-02T08:53:22.906600+0000","last_became_peered":"2024-04-02T08:53:22.906600+0000","last_unstale":"2024-04-02T08:53:23.891377+0000","last_undegraded":"2024-04-02T08:53:23.891377+0000","last_fullsized":"2024-04-02T08:53:23.891377+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:37:49.959337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.891265+0000","last_change":"2024-04-02T08:53:22.900339+0000","last_active":"2024-04-02T08:53:23.891265+0000","last_peered":"2024-04-02T08:53:23.891265+0000","last_clean":"2024-04-02T08:53:23.891265+0000","last_became_active":"2024-04-02T08:53:22.900073+0000","last_became_peered":"2024-04-02T08:53:22.900073+0000","last_unstale":"2024-04-02T08:53:23.891265+0000","last_undegraded":"2024-04-02T08:53:23.891265+0000","last_fullsized":"2024-04-02T08:53:23.891265+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:07:06.537235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.919736+0000","last_change":"2024-04-02T08:53:22.911050+0000","last_active":"2024-04-02T08:53:22.919736+0000","last_peered":"2024-04-02T08:53:22.919736+0000","last_clean":"2024-04-02T08:53:22.919736+0000","last_became_active":"2024-04-02T08:53:22.908720+0000","last_became_peered":"2024-04-02T08:53:22.908720+0000","last_unstale":"2024-04-02T08:53:22.919736+0000","last_undegraded":"2024-04-02T08:53:22.919736+0000","last_fullsized":"2024-04-02T08:53:22.919736+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:45:44.350132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.920415+0000","last_change":"2024-04-02T08:53:22.916457+0000","last_active":"2024-04-02T08:53:22.920415+0000","last_peered":"2024-04-02T08:53:22.920415+0000","last_clean":"2024-04-02T08:53:22.920415+0000","last_became_active":"2024-04-02T08:53:22.916022+0000","last_became_peered":"2024-04-02T08:53:22.916022+0000","last_unstale":"2024-04-02T08:53:22.920415+0000","last_undegraded":"2024-04-02T08:53:22.920415+0000","last_fullsized":"2024-04-02T08:53:22.920415+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:54.588880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915598+0000","last_change":"2024-04-02T08:53:22.910047+0000","last_active":"2024-04-02T08:53:22.915598+0000","last_peered":"2024-04-02T08:53:22.915598+0000","last_clean":"2024-04-02T08:53:22.915598+0000","last_became_active":"2024-04-02T08:53:22.909738+0000","last_became_peered":"2024-04-02T08:53:22.909738+0000","last_unstale":"2024-04-02T08:53:22.915598+0000","last_undegraded":"2024-04-02T08:53:22.915598+0000","last_fullsized":"2024-04-02T08:53:22.915598+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:57:08.290485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896304+0000","last_change":"2024-04-02T08:53:22.901205+0000","last_active":"2024-04-02T08:53:23.896304+0000","last_peered":"2024-04-02T08:53:23.896304+0000","last_clean":"2024-04-02T08:53:23.896304+0000","last_became_active":"2024-04-02T08:53:22.900823+0000","last_became_peered":"2024-04-02T08:53:22.900823+0000","last_unstale":"2024-04-02T08:53:23.896304+0000","last_undegraded":"2024-04-02T08:53:23.896304+0000","last_fullsized":"2024-04-02T08:53:23.896304+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:17:36.197813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896085+0000","last_change":"2024-04-02T08:53:22.906959+0000","last_active":"2024-04-02T08:53:23.896085+0000","last_peered":"2024-04-02T08:53:23.896085+0000","last_clean":"2024-04-02T08:53:23.896085+0000","last_became_active":"2024-04-02T08:53:22.906636+0000","last_became_peered":"2024-04-02T08:53:22.906636+0000","last_unstale":"2024-04-02T08:53:23.896085+0000","last_undegraded":"2024-04-02T08:53:23.896085+0000","last_fullsized":"2024-04-02T08:53:23.896085+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:51.246631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.916081+0000","last_change":"2024-04-02T08:53:22.913253+0000","last_active":"2024-04-02T08:53:22.916081+0000","last_peered":"2024-04-02T08:53:22.916081+0000","last_clean":"2024-04-02T08:53:22.916081+0000","last_became_active":"2024-04-02T08:53:22.912885+0000","last_became_peered":"2024-04-02T08:53:22.912885+0000","last_unstale":"2024-04-02T08:53:22.916081+0000","last_undegraded":"2024-04-02T08:53:22.916081+0000","last_fullsized":"2024-04-02T08:53:22.916081+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:07:20.689753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.891507+0000","last_change":"2024-04-02T08:53:22.898307+0000","last_active":"2024-04-02T08:53:23.891507+0000","last_peered":"2024-04-02T08:53:23.891507+0000","last_clean":"2024-04-02T08:53:23.891507+0000","last_became_active":"2024-04-02T08:53:22.897981+0000","last_became_peered":"2024-04-02T08:53:22.897981+0000","last_unstale":"2024-04-02T08:53:23.891507+0000","last_undegraded":"2024-04-02T08:53:23.891507+0000","last_fullsized":"2024-04-02T08:53:23.891507+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:16:56.465320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895543+0000","last_change":"2024-04-02T08:53:22.910525+0000","last_active":"2024-04-02T08:53:23.895543+0000","last_peered":"2024-04-02T08:53:23.895543+0000","last_clean":"2024-04-02T08:53:23.895543+0000","last_became_active":"2024-04-02T08:53:22.910243+0000","last_became_peered":"2024-04-02T08:53:22.910243+0000","last_unstale":"2024-04-02T08:53:23.895543+0000","last_undegraded":"2024-04-02T08:53:23.895543+0000","last_fullsized":"2024-04-02T08:53:23.895543+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:34.518513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915834+0000","last_change":"2024-04-02T08:53:22.904389+0000","last_active":"2024-04-02T08:53:22.915834+0000","last_peered":"2024-04-02T08:53:22.915834+0000","last_clean":"2024-04-02T08:53:22.915834+0000","last_became_active":"2024-04-02T08:53:22.904123+0000","last_became_peered":"2024-04-02T08:53:22.904123+0000","last_unstale":"2024-04-02T08:53:22.915834+0000","last_undegraded":"2024-04-02T08:53:22.915834+0000","last_fullsized":"2024-04-02T08:53:22.915834+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:41:18.537425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915985+0000","last_change":"2024-04-02T08:53:22.907284+0000","last_active":"2024-04-02T08:53:22.915985+0000","last_peered":"2024-04-02T08:53:22.915985+0000","last_clean":"2024-04-02T08:53:22.915985+0000","last_became_active":"2024-04-02T08:53:22.900183+0000","last_became_peered":"2024-04-02T08:53:22.900183+0000","last_unstale":"2024-04-02T08:53:22.915985+0000","last_undegraded":"2024-04-02T08:53:22.915985+0000","last_fullsized":"2024-04-02T08:53:22.915985+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:08:17.507337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.919811+0000","last_change":"2024-04-02T08:53:22.911453+0000","last_active":"2024-04-02T08:53:22.919811+0000","last_peered":"2024-04-02T08:53:22.919811+0000","last_clean":"2024-04-02T08:53:22.919811+0000","last_became_active":"2024-04-02T08:53:22.910973+0000","last_became_peered":"2024-04-02T08:53:22.910973+0000","last_unstale":"2024-04-02T08:53:22.919811+0000","last_undegraded":"2024-04-02T08:53:22.919811+0000","last_fullsized":"2024-04-02T08:53:22.919811+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:35.886941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.898051+0000","last_change":"2024-04-02T08:53:22.896319+0000","last_active":"2024-04-02T08:53:22.898051+0000","last_peered":"2024-04-02T08:53:22.898051+0000","last_clean":"2024-04-02T08:53:22.898051+0000","last_became_active":"2024-04-02T08:53:22.896066+0000","last_became_peered":"2024-04-02T08:53:22.896066+0000","last_unstale":"2024-04-02T08:53:22.898051+0000","last_undegraded":"2024-04-02T08:53:22.898051+0000","last_fullsized":"2024-04-02T08:53:22.898051+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:03:30.324036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.911129+0000","last_change":"2024-04-02T08:53:22.902522+0000","last_active":"2024-04-02T08:53:22.911129+0000","last_peered":"2024-04-02T08:53:22.911129+0000","last_clean":"2024-04-02T08:53:22.911129+0000","last_became_active":"2024-04-02T08:53:22.901620+0000","last_became_peered":"2024-04-02T08:53:22.901620+0000","last_unstale":"2024-04-02T08:53:22.911129+0000","last_undegraded":"2024-04-02T08:53:22.911129+0000","last_fullsized":"2024-04-02T08:53:22.911129+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:27.613450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.919307+0000","last_change":"2024-04-02T08:53:22.911275+0000","last_active":"2024-04-02T08:53:22.919307+0000","last_peered":"2024-04-02T08:53:22.919307+0000","last_clean":"2024-04-02T08:53:22.919307+0000","last_became_active":"2024-04-02T08:53:22.910684+0000","last_became_peered":"2024-04-02T08:53:22.910684+0000","last_unstale":"2024-04-02T08:53:22.919307+0000","last_undegraded":"2024-04-02T08:53:22.919307+0000","last_fullsized":"2024-04-02T08:53:22.919307+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:18:00.950788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.916461+0000","last_change":"2024-04-02T08:53:22.914295+0000","last_active":"2024-04-02T08:53:22.916461+0000","last_peered":"2024-04-02T08:53:22.916461+0000","last_clean":"2024-04-02T08:53:22.916461+0000","last_became_active":"2024-04-02T08:53:22.913973+0000","last_became_peered":"2024-04-02T08:53:22.913973+0000","last_unstale":"2024-04-02T08:53:22.916461+0000","last_undegraded":"2024-04-02T08:53:22.916461+0000","last_fullsized":"2024-04-02T08:53:22.916461+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:30.142896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.900556+0000","last_change":"2024-04-02T08:53:22.896360+0000","last_active":"2024-04-02T08:53:22.900556+0000","last_peered":"2024-04-02T08:53:22.900556+0000","last_clean":"2024-04-02T08:53:22.900556+0000","last_became_active":"2024-04-02T08:53:22.896076+0000","last_became_peered":"2024-04-02T08:53:22.896076+0000","last_unstale":"2024-04-02T08:53:22.900556+0000","last_undegraded":"2024-04-02T08:53:22.900556+0000","last_fullsized":"2024-04-02T08:53:22.900556+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:16:12.933857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.919757+0000","last_change":"2024-04-02T08:53:22.913110+0000","last_active":"2024-04-02T08:53:22.919757+0000","last_peered":"2024-04-02T08:53:22.919757+0000","last_clean":"2024-04-02T08:53:22.919757+0000","last_became_active":"2024-04-02T08:53:22.909606+0000","last_became_peered":"2024-04-02T08:53:22.909606+0000","last_unstale":"2024-04-02T08:53:22.919757+0000","last_undegraded":"2024-04-02T08:53:22.919757+0000","last_fullsized":"2024-04-02T08:53:22.919757+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:00:23.503152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.898087+0000","last_change":"2024-04-02T08:53:22.896929+0000","last_active":"2024-04-02T08:53:22.898087+0000","last_peered":"2024-04-02T08:53:22.898087+0000","last_clean":"2024-04-02T08:53:22.898087+0000","last_became_active":"2024-04-02T08:53:22.896632+0000","last_became_peered":"2024-04-02T08:53:22.896632+0000","last_unstale":"2024-04-02T08:53:22.898087+0000","last_undegraded":"2024-04-02T08:53:22.898087+0000","last_fullsized":"2024-04-02T08:53:22.898087+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:07:20.689753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896482+0000","last_change":"2024-04-02T08:53:22.905257+0000","last_active":"2024-04-02T08:53:23.896482+0000","last_peered":"2024-04-02T08:53:23.896482+0000","last_clean":"2024-04-02T08:53:23.896482+0000","last_became_active":"2024-04-02T08:53:22.904963+0000","last_became_peered":"2024-04-02T08:53:22.904963+0000","last_unstale":"2024-04-02T08:53:23.896482+0000","last_undegraded":"2024-04-02T08:53:23.896482+0000","last_fullsized":"2024-04-02T08:53:23.896482+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:16:12.933857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896214+0000","last_change":"2024-04-02T08:53:22.902994+0000","last_active":"2024-04-02T08:53:23.896214+0000","last_peered":"2024-04-02T08:53:23.896214+0000","last_clean":"2024-04-02T08:53:23.896214+0000","last_became_active":"2024-04-02T08:53:22.902493+0000","last_became_peered":"2024-04-02T08:53:22.902493+0000","last_unstale":"2024-04-02T08:53:23.896214+0000","last_undegraded":"2024-04-02T08:53:23.896214+0000","last_fullsized":"2024-04-02T08:53:23.896214+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:27.613450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1f","version":"185'4","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.921373+0000","last_change":"2024-04-02T08:53:22.400085+0000","last_active":"2024-04-02T08:53:22.921373+0000","last_peered":"2024-04-02T08:53:22.921373+0000","last_clean":"2024-04-02T08:53:22.921373+0000","last_became_active":"2024-04-02T08:53:22.399817+0000","last_became_peered":"2024-04-02T08:53:22.399817+0000","last_unstale":"2024-04-02T08:53:22.921373+0000","last_undegraded":"2024-04-02T08:53:22.921373+0000","last_fullsized":"2024-04-02T08:53:22.921373+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:12.895858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.909660+0000","last_change":"2024-04-02T08:53:22.905495+0000","last_active":"2024-04-02T08:53:22.909660+0000","last_peered":"2024-04-02T08:53:22.909660+0000","last_clean":"2024-04-02T08:53:22.909660+0000","last_became_active":"2024-04-02T08:53:22.905253+0000","last_became_peered":"2024-04-02T08:53:22.905253+0000","last_unstale":"2024-04-02T08:53:22.909660+0000","last_undegraded":"2024-04-02T08:53:22.909660+0000","last_fullsized":"2024-04-02T08:53:22.909660+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:20:17.970790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"185'1","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.908631+0000","last_change":"2024-04-02T08:53:22.393126+0000","last_active":"2024-04-02T08:53:22.908631+0000","last_peered":"2024-04-02T08:53:22.908631+0000","last_clean":"2024-04-02T08:53:22.908631+0000","last_became_active":"2024-04-02T08:53:22.392818+0000","last_became_peered":"2024-04-02T08:53:22.392818+0000","last_unstale":"2024-04-02T08:53:22.908631+0000","last_undegraded":"2024-04-02T08:53:22.908631+0000","last_fullsized":"2024-04-02T08:53:22.908631+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:37:19.868360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895861+0000","last_change":"2024-04-02T08:53:22.911103+0000","last_active":"2024-04-02T08:53:23.895861+0000","last_peered":"2024-04-02T08:53:23.895861+0000","last_clean":"2024-04-02T08:53:23.895861+0000","last_became_active":"2024-04-02T08:53:22.910837+0000","last_became_peered":"2024-04-02T08:53:22.910837+0000","last_unstale":"2024-04-02T08:53:23.895861+0000","last_undegraded":"2024-04-02T08:53:23.895861+0000","last_fullsized":"2024-04-02T08:53:23.895861+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:57:08.290485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1d","version":"185'2","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895594+0000","last_change":"2024-04-02T08:53:22.399283+0000","last_active":"2024-04-02T08:53:23.895594+0000","last_peered":"2024-04-02T08:53:23.895594+0000","last_clean":"2024-04-02T08:53:23.895594+0000","last_became_active":"2024-04-02T08:53:22.398989+0000","last_became_peered":"2024-04-02T08:53:22.398989+0000","last_unstale":"2024-04-02T08:53:23.895594+0000","last_undegraded":"2024-04-02T08:53:23.895594+0000","last_fullsized":"2024-04-02T08:53:23.895594+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:54:09.448505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.920120+0000","last_change":"2024-04-02T08:53:22.915179+0000","last_active":"2024-04-02T08:53:22.920120+0000","last_peered":"2024-04-02T08:53:22.920120+0000","last_clean":"2024-04-02T08:53:22.920120+0000","last_became_active":"2024-04-02T08:53:22.914930+0000","last_became_peered":"2024-04-02T08:53:22.914930+0000","last_unstale":"2024-04-02T08:53:22.920120+0000","last_undegraded":"2024-04-02T08:53:22.920120+0000","last_fullsized":"2024-04-02T08:53:22.920120+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:02:46.249414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1c","version":"185'2","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.920362+0000","last_change":"2024-04-02T08:53:22.394260+0000","last_active":"2024-04-02T08:53:22.920362+0000","last_peered":"2024-04-02T08:53:22.920362+0000","last_clean":"2024-04-02T08:53:22.920362+0000","last_became_active":"2024-04-02T08:53:22.393832+0000","last_became_peered":"2024-04-02T08:53:22.393832+0000","last_unstale":"2024-04-02T08:53:22.920362+0000","last_undegraded":"2024-04-02T08:53:22.920362+0000","last_fullsized":"2024-04-02T08:53:22.920362+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:09.753474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895786+0000","last_change":"2024-04-02T08:53:22.900232+0000","last_active":"2024-04-02T08:53:23.895786+0000","last_peered":"2024-04-02T08:53:23.895786+0000","last_clean":"2024-04-02T08:53:23.895786+0000","last_became_active":"2024-04-02T08:53:22.899365+0000","last_became_peered":"2024-04-02T08:53:22.899365+0000","last_unstale":"2024-04-02T08:53:23.895786+0000","last_undegraded":"2024-04-02T08:53:23.895786+0000","last_fullsized":"2024-04-02T08:53:23.895786+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:31.589774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1b","version":"185'1","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.912683+0000","last_change":"2024-04-02T08:53:22.403201+0000","last_active":"2024-04-02T08:53:22.912683+0000","last_peered":"2024-04-02T08:53:22.912683+0000","last_clean":"2024-04-02T08:53:22.912683+0000","last_became_active":"2024-04-02T08:53:22.402941+0000","last_became_peered":"2024-04-02T08:53:22.402941+0000","last_unstale":"2024-04-02T08:53:22.912683+0000","last_undegraded":"2024-04-02T08:53:22.912683+0000","last_fullsized":"2024-04-02T08:53:22.912683+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:10.962513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895654+0000","last_change":"2024-04-02T08:53:22.907452+0000","last_active":"2024-04-02T08:53:23.895654+0000","last_peered":"2024-04-02T08:53:23.895654+0000","last_clean":"2024-04-02T08:53:23.895654+0000","last_became_active":"2024-04-02T08:53:22.904802+0000","last_became_peered":"2024-04-02T08:53:22.904802+0000","last_unstale":"2024-04-02T08:53:23.895654+0000","last_undegraded":"2024-04-02T08:53:23.895654+0000","last_fullsized":"2024-04-02T08:53:23.895654+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:47:04.773777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1a","version":"185'1","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895887+0000","last_change":"2024-04-02T08:53:22.402079+0000","last_active":"2024-04-02T08:53:23.895887+0000","last_peered":"2024-04-02T08:53:23.895887+0000","last_clean":"2024-04-02T08:53:23.895887+0000","last_became_active":"2024-04-02T08:53:22.401822+0000","last_became_peered":"2024-04-02T08:53:22.401822+0000","last_unstale":"2024-04-02T08:53:23.895887+0000","last_undegraded":"2024-04-02T08:53:23.895887+0000","last_fullsized":"2024-04-02T08:53:23.895887+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:57:11.838951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.907648+0000","last_change":"2024-04-02T08:53:22.899291+0000","last_active":"2024-04-02T08:53:22.907648+0000","last_peered":"2024-04-02T08:53:22.907648+0000","last_clean":"2024-04-02T08:53:22.907648+0000","last_became_active":"2024-04-02T08:53:22.898374+0000","last_became_peered":"2024-04-02T08:53:22.898374+0000","last_unstale":"2024-04-02T08:53:22.907648+0000","last_undegraded":"2024-04-02T08:53:22.907648+0000","last_fullsized":"2024-04-02T08:53:22.907648+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:45:29.735767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.887167+0000","last_change":"2024-04-02T08:53:22.395845+0000","last_active":"2024-04-02T08:53:22.887167+0000","last_peered":"2024-04-02T08:53:22.887167+0000","last_clean":"2024-04-02T08:53:22.887167+0000","last_became_active":"2024-04-02T08:53:22.395542+0000","last_became_peered":"2024-04-02T08:53:22.395542+0000","last_unstale":"2024-04-02T08:53:22.887167+0000","last_undegraded":"2024-04-02T08:53:22.887167+0000","last_fullsized":"2024-04-02T08:53:22.887167+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:19:11.682642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895344+0000","last_change":"2024-04-02T08:53:22.897914+0000","last_active":"2024-04-02T08:53:23.895344+0000","last_peered":"2024-04-02T08:53:23.895344+0000","last_clean":"2024-04-02T08:53:23.895344+0000","last_became_active":"2024-04-02T08:53:22.897643+0000","last_became_peered":"2024-04-02T08:53:22.897643+0000","last_unstale":"2024-04-02T08:53:23.895344+0000","last_undegraded":"2024-04-02T08:53:23.895344+0000","last_fullsized":"2024-04-02T08:53:23.895344+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:45:29.735767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.897699+0000","last_change":"2024-04-02T08:53:22.402944+0000","last_active":"2024-04-02T08:53:22.897699+0000","last_peered":"2024-04-02T08:53:22.897699+0000","last_clean":"2024-04-02T08:53:22.897699+0000","last_became_active":"2024-04-02T08:53:22.401902+0000","last_became_peered":"2024-04-02T08:53:22.401902+0000","last_unstale":"2024-04-02T08:53:22.897699+0000","last_undegraded":"2024-04-02T08:53:22.897699+0000","last_fullsized":"2024-04-02T08:53:22.897699+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:57:11.838951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915809+0000","last_change":"2024-04-02T08:53:22.911485+0000","last_active":"2024-04-02T08:53:22.915809+0000","last_peered":"2024-04-02T08:53:22.915809+0000","last_clean":"2024-04-02T08:53:22.915809+0000","last_became_active":"2024-04-02T08:53:22.911208+0000","last_became_peered":"2024-04-02T08:53:22.911208+0000","last_unstale":"2024-04-02T08:53:22.915809+0000","last_undegraded":"2024-04-02T08:53:22.915809+0000","last_fullsized":"2024-04-02T08:53:22.915809+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:21.004514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":11,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896217+0000","last_change":"2024-04-02T08:53:22.408387+0000","last_active":"2024-04-02T08:53:23.896217+0000","last_peered":"2024-04-02T08:53:23.896217+0000","last_clean":"2024-04-02T08:53:23.896217+0000","last_became_active":"2024-04-02T08:53:22.408002+0000","last_became_peered":"2024-04-02T08:53:22.408002+0000","last_unstale":"2024-04-02T08:53:23.896217+0000","last_undegraded":"2024-04-02T08:53:23.896217+0000","last_fullsized":"2024-04-02T08:53:23.896217+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:10:29.755694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.916291+0000","last_change":"2024-04-02T08:53:22.908670+0000","last_active":"2024-04-02T08:53:22.916291+0000","last_peered":"2024-04-02T08:53:22.916291+0000","last_clean":"2024-04-02T08:53:22.916291+0000","last_became_active":"2024-04-02T08:53:22.907830+0000","last_became_peered":"2024-04-02T08:53:22.907830+0000","last_unstale":"2024-04-02T08:53:22.916291+0000","last_undegraded":"2024-04-02T08:53:22.916291+0000","last_fullsized":"2024-04-02T08:53:22.916291+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:20:24.495239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.15","version":"185'3","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.921029+0000","last_change":"2024-04-02T08:53:22.394298+0000","last_active":"2024-04-02T08:53:22.921029+0000","last_peered":"2024-04-02T08:53:22.921029+0000","last_clean":"2024-04-02T08:53:22.921029+0000","last_became_active":"2024-04-02T08:53:22.393753+0000","last_became_peered":"2024-04-02T08:53:22.393753+0000","last_unstale":"2024-04-02T08:53:22.921029+0000","last_undegraded":"2024-04-02T08:53:22.921029+0000","last_fullsized":"2024-04-02T08:53:22.921029+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:49:51.216455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.908432+0000","last_change":"2024-04-02T08:53:22.899132+0000","last_active":"2024-04-02T08:53:22.908432+0000","last_peered":"2024-04-02T08:53:22.908432+0000","last_clean":"2024-04-02T08:53:22.908432+0000","last_became_active":"2024-04-02T08:53:22.898859+0000","last_became_peered":"2024-04-02T08:53:22.898859+0000","last_unstale":"2024-04-02T08:53:22.908432+0000","last_undegraded":"2024-04-02T08:53:22.908432+0000","last_fullsized":"2024-04-02T08:53:22.908432+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:33.443826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.14","version":"185'6","reported_seq":16,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.922204+0000","last_change":"2024-04-02T08:53:22.402901+0000","last_active":"2024-04-02T08:53:22.922204+0000","last_peered":"2024-04-02T08:53:22.922204+0000","last_clean":"2024-04-02T08:53:22.922204+0000","last_became_active":"2024-04-02T08:53:22.402573+0000","last_became_peered":"2024-04-02T08:53:22.402573+0000","last_unstale":"2024-04-02T08:53:22.922204+0000","last_undegraded":"2024-04-02T08:53:22.922204+0000","last_fullsized":"2024-04-02T08:53:22.922204+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:18:23.600237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.911452+0000","last_change":"2024-04-02T08:53:22.906555+0000","last_active":"2024-04-02T08:53:22.911452+0000","last_peered":"2024-04-02T08:53:22.911452+0000","last_clean":"2024-04-02T08:53:22.911452+0000","last_became_active":"2024-04-02T08:53:22.906230+0000","last_became_peered":"2024-04-02T08:53:22.906230+0000","last_unstale":"2024-04-02T08:53:22.911452+0000","last_undegraded":"2024-04-02T08:53:22.911452+0000","last_fullsized":"2024-04-02T08:53:22.911452+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:33.443826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"185'2","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896122+0000","last_change":"2024-04-02T08:53:22.406651+0000","last_active":"2024-04-02T08:53:23.896122+0000","last_peered":"2024-04-02T08:53:23.896122+0000","last_clean":"2024-04-02T08:53:23.896122+0000","last_became_active":"2024-04-02T08:53:22.406085+0000","last_became_peered":"2024-04-02T08:53:22.406085+0000","last_unstale":"2024-04-02T08:53:23.896122+0000","last_undegraded":"2024-04-02T08:53:23.896122+0000","last_fullsized":"2024-04-02T08:53:23.896122+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:09.753474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.917576+0000","last_change":"2024-04-02T08:53:22.911150+0000","last_active":"2024-04-02T08:53:22.917576+0000","last_peered":"2024-04-02T08:53:22.917576+0000","last_clean":"2024-04-02T08:53:22.917576+0000","last_became_active":"2024-04-02T08:53:22.904223+0000","last_became_peered":"2024-04-02T08:53:22.904223+0000","last_unstale":"2024-04-02T08:53:22.917576+0000","last_undegraded":"2024-04-02T08:53:22.917576+0000","last_fullsized":"2024-04-02T08:53:22.917576+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:35:49.453211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":11,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895898+0000","last_change":"2024-04-02T08:53:22.401280+0000","last_active":"2024-04-02T08:53:23.895898+0000","last_peered":"2024-04-02T08:53:23.895898+0000","last_clean":"2024-04-02T08:53:23.895898+0000","last_became_active":"2024-04-02T08:53:22.400999+0000","last_became_peered":"2024-04-02T08:53:22.400999+0000","last_unstale":"2024-04-02T08:53:23.895898+0000","last_undegraded":"2024-04-02T08:53:23.895898+0000","last_fullsized":"2024-04-02T08:53:23.895898+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:12.895858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896520+0000","last_change":"2024-04-02T08:53:22.910543+0000","last_active":"2024-04-02T08:53:23.896520+0000","last_peered":"2024-04-02T08:53:23.896520+0000","last_clean":"2024-04-02T08:53:23.896520+0000","last_became_active":"2024-04-02T08:53:22.909429+0000","last_became_peered":"2024-04-02T08:53:22.909429+0000","last_unstale":"2024-04-02T08:53:23.896520+0000","last_undegraded":"2024-04-02T08:53:23.896520+0000","last_fullsized":"2024-04-02T08:53:23.896520+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:08:21.301397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.903502+0000","last_change":"2024-04-02T08:53:22.405873+0000","last_active":"2024-04-02T08:53:22.903502+0000","last_peered":"2024-04-02T08:53:22.903502+0000","last_clean":"2024-04-02T08:53:22.903502+0000","last_became_active":"2024-04-02T08:53:22.405303+0000","last_became_peered":"2024-04-02T08:53:22.405303+0000","last_unstale":"2024-04-02T08:53:22.903502+0000","last_undegraded":"2024-04-02T08:53:22.903502+0000","last_fullsized":"2024-04-02T08:53:22.903502+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:12.895858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.910718+0000","last_change":"2024-04-02T08:53:22.906928+0000","last_active":"2024-04-02T08:53:22.910718+0000","last_peered":"2024-04-02T08:53:22.910718+0000","last_clean":"2024-04-02T08:53:22.910718+0000","last_became_active":"2024-04-02T08:53:22.906668+0000","last_became_peered":"2024-04-02T08:53:22.906668+0000","last_unstale":"2024-04-02T08:53:22.910718+0000","last_undegraded":"2024-04-02T08:53:22.910718+0000","last_fullsized":"2024-04-02T08:53:22.910718+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:47:39.254268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"185'8","reported_seq":18,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.920515+0000","last_change":"2024-04-02T08:53:22.398497+0000","last_active":"2024-04-02T08:53:22.920515+0000","last_peered":"2024-04-02T08:53:22.920515+0000","last_clean":"2024-04-02T08:53:22.920515+0000","last_became_active":"2024-04-02T08:53:22.398238+0000","last_became_peered":"2024-04-02T08:53:22.398238+0000","last_unstale":"2024-04-02T08:53:22.920515+0000","last_undegraded":"2024-04-02T08:53:22.920515+0000","last_fullsized":"2024-04-02T08:53:22.920515+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:19.432754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.920803+0000","last_change":"2024-04-02T08:53:22.916024+0000","last_active":"2024-04-02T08:53:22.920803+0000","last_peered":"2024-04-02T08:53:22.920803+0000","last_clean":"2024-04-02T08:53:22.920803+0000","last_became_active":"2024-04-02T08:53:22.915538+0000","last_became_peered":"2024-04-02T08:53:22.915538+0000","last_unstale":"2024-04-02T08:53:22.920803+0000","last_undegraded":"2024-04-02T08:53:22.920803+0000","last_fullsized":"2024-04-02T08:53:22.920803+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:16:12.933857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.f","version":"185'1","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915574+0000","last_change":"2024-04-02T08:53:22.404220+0000","last_active":"2024-04-02T08:53:22.915574+0000","last_peered":"2024-04-02T08:53:22.915574+0000","last_clean":"2024-04-02T08:53:22.915574+0000","last_became_active":"2024-04-02T08:53:22.403960+0000","last_became_peered":"2024-04-02T08:53:22.403960+0000","last_unstale":"2024-04-02T08:53:22.915574+0000","last_undegraded":"2024-04-02T08:53:22.915574+0000","last_fullsized":"2024-04-02T08:53:22.915574+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:53:21.563617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896503+0000","last_change":"2024-04-02T08:53:22.900299+0000","last_active":"2024-04-02T08:53:23.896503+0000","last_peered":"2024-04-02T08:53:23.896503+0000","last_clean":"2024-04-02T08:53:23.896503+0000","last_became_active":"2024-04-02T08:53:22.899946+0000","last_became_peered":"2024-04-02T08:53:22.899946+0000","last_unstale":"2024-04-02T08:53:23.896503+0000","last_undegraded":"2024-04-02T08:53:23.896503+0000","last_fullsized":"2024-04-02T08:53:23.896503+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:37:20.875182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"185'2","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915044+0000","last_change":"2024-04-02T08:53:22.396923+0000","last_active":"2024-04-02T08:53:22.915044+0000","last_peered":"2024-04-02T08:53:22.915044+0000","last_clean":"2024-04-02T08:53:22.915044+0000","last_became_active":"2024-04-02T08:53:22.396312+0000","last_became_peered":"2024-04-02T08:53:22.396312+0000","last_unstale":"2024-04-02T08:53:22.915044+0000","last_undegraded":"2024-04-02T08:53:22.915044+0000","last_fullsized":"2024-04-02T08:53:22.915044+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:41:39.706658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895731+0000","last_change":"2024-04-02T08:53:22.905459+0000","last_active":"2024-04-02T08:53:23.895731+0000","last_peered":"2024-04-02T08:53:23.895731+0000","last_clean":"2024-04-02T08:53:23.895731+0000","last_became_active":"2024-04-02T08:53:22.904516+0000","last_became_peered":"2024-04-02T08:53:22.904516+0000","last_unstale":"2024-04-02T08:53:23.895731+0000","last_undegraded":"2024-04-02T08:53:23.895731+0000","last_fullsized":"2024-04-02T08:53:23.895731+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:59:28.052580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.d","version":"185'1","reported_seq":11,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.920209+0000","last_change":"2024-04-02T08:53:22.400578+0000","last_active":"2024-04-02T08:53:22.920209+0000","last_peered":"2024-04-02T08:53:22.920209+0000","last_clean":"2024-04-02T08:53:22.920209+0000","last_became_active":"2024-04-02T08:53:22.400120+0000","last_became_peered":"2024-04-02T08:53:22.400120+0000","last_unstale":"2024-04-02T08:53:22.920209+0000","last_undegraded":"2024-04-02T08:53:22.920209+0000","last_fullsized":"2024-04-02T08:53:22.920209+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:41.730302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.912426+0000","last_change":"2024-04-02T08:53:22.901564+0000","last_active":"2024-04-02T08:53:22.912426+0000","last_peered":"2024-04-02T08:53:22.912426+0000","last_clean":"2024-04-02T08:53:22.912426+0000","last_became_active":"2024-04-02T08:53:22.901281+0000","last_became_peered":"2024-04-02T08:53:22.901281+0000","last_unstale":"2024-04-02T08:53:22.912426+0000","last_undegraded":"2024-04-02T08:53:22.912426+0000","last_fullsized":"2024-04-02T08:53:22.912426+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:10:25.277292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":11,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.891487+0000","last_change":"2024-04-02T08:53:22.405963+0000","last_active":"2024-04-02T08:53:23.891487+0000","last_peered":"2024-04-02T08:53:23.891487+0000","last_clean":"2024-04-02T08:53:23.891487+0000","last_became_active":"2024-04-02T08:53:22.405028+0000","last_became_peered":"2024-04-02T08:53:22.405028+0000","last_unstale":"2024-04-02T08:53:23.891487+0000","last_undegraded":"2024-04-02T08:53:23.891487+0000","last_fullsized":"2024-04-02T08:53:23.891487+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:33:24.291273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.908222+0000","last_change":"2024-04-02T08:53:22.900003+0000","last_active":"2024-04-02T08:53:22.908222+0000","last_peered":"2024-04-02T08:53:22.908222+0000","last_clean":"2024-04-02T08:53:22.908222+0000","last_became_active":"2024-04-02T08:53:22.897858+0000","last_became_peered":"2024-04-02T08:53:22.897858+0000","last_unstale":"2024-04-02T08:53:22.908222+0000","last_undegraded":"2024-04-02T08:53:22.908222+0000","last_fullsized":"2024-04-02T08:53:22.908222+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:20:24.495239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.b","version":"185'2","reported_seq":12,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.919532+0000","last_change":"2024-04-02T08:53:22.398735+0000","last_active":"2024-04-02T08:53:22.919532+0000","last_peered":"2024-04-02T08:53:22.919532+0000","last_clean":"2024-04-02T08:53:22.919532+0000","last_became_active":"2024-04-02T08:53:22.398441+0000","last_became_peered":"2024-04-02T08:53:22.398441+0000","last_unstale":"2024-04-02T08:53:22.919532+0000","last_undegraded":"2024-04-02T08:53:22.919532+0000","last_fullsized":"2024-04-02T08:53:22.919532+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:28.073575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.918072+0000","last_change":"2024-04-02T08:53:22.914016+0000","last_active":"2024-04-02T08:53:22.918072+0000","last_peered":"2024-04-02T08:53:22.918072+0000","last_clean":"2024-04-02T08:53:22.918072+0000","last_became_active":"2024-04-02T08:53:22.913758+0000","last_became_peered":"2024-04-02T08:53:22.913758+0000","last_unstale":"2024-04-02T08:53:22.918072+0000","last_undegraded":"2024-04-02T08:53:22.918072+0000","last_fullsized":"2024-04-02T08:53:22.918072+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:02:38.382306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":11,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.891132+0000","last_change":"2024-04-02T08:53:22.397314+0000","last_active":"2024-04-02T08:53:23.891132+0000","last_peered":"2024-04-02T08:53:23.891132+0000","last_clean":"2024-04-02T08:53:23.891132+0000","last_became_active":"2024-04-02T08:53:22.395250+0000","last_became_peered":"2024-04-02T08:53:22.395250+0000","last_unstale":"2024-04-02T08:53:23.891132+0000","last_undegraded":"2024-04-02T08:53:23.891132+0000","last_fullsized":"2024-04-02T08:53:23.891132+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:47:38.247446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.908129+0000","last_change":"2024-04-02T08:53:22.900240+0000","last_active":"2024-04-02T08:53:22.908129+0000","last_peered":"2024-04-02T08:53:22.908129+0000","last_clean":"2024-04-02T08:53:22.908129+0000","last_became_active":"2024-04-02T08:53:22.899975+0000","last_became_peered":"2024-04-02T08:53:22.899975+0000","last_unstale":"2024-04-02T08:53:22.908129+0000","last_undegraded":"2024-04-02T08:53:22.908129+0000","last_fullsized":"2024-04-02T08:53:22.908129+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:20:20.210105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":11,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895402+0000","last_change":"2024-04-02T08:53:22.405750+0000","last_active":"2024-04-02T08:53:23.895402+0000","last_peered":"2024-04-02T08:53:23.895402+0000","last_clean":"2024-04-02T08:53:23.895402+0000","last_became_active":"2024-04-02T08:53:22.405394+0000","last_became_peered":"2024-04-02T08:53:22.405394+0000","last_unstale":"2024-04-02T08:53:23.895402+0000","last_undegraded":"2024-04-02T08:53:23.895402+0000","last_fullsized":"2024-04-02T08:53:23.895402+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:28.073575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.920856+0000","last_change":"2024-04-02T08:53:22.917070+0000","last_active":"2024-04-02T08:53:22.920856+0000","last_peered":"2024-04-02T08:53:22.920856+0000","last_clean":"2024-04-02T08:53:22.920856+0000","last_became_active":"2024-04-02T08:53:22.916562+0000","last_became_peered":"2024-04-02T08:53:22.916562+0000","last_unstale":"2024-04-02T08:53:22.920856+0000","last_undegraded":"2024-04-02T08:53:22.920856+0000","last_fullsized":"2024-04-02T08:53:22.920856+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:08:21.301397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":11,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.891140+0000","last_change":"2024-04-02T08:53:22.391813+0000","last_active":"2024-04-02T08:53:23.891140+0000","last_peered":"2024-04-02T08:53:23.891140+0000","last_clean":"2024-04-02T08:53:23.891140+0000","last_became_active":"2024-04-02T08:53:22.391483+0000","last_became_peered":"2024-04-02T08:53:22.391483+0000","last_unstale":"2024-04-02T08:53:23.891140+0000","last_undegraded":"2024-04-02T08:53:23.891140+0000","last_fullsized":"2024-04-02T08:53:23.891140+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:45:43.343310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.911149+0000","last_change":"2024-04-02T08:53:22.904782+0000","last_active":"2024-04-02T08:53:22.911149+0000","last_peered":"2024-04-02T08:53:22.911149+0000","last_clean":"2024-04-02T08:53:22.911149+0000","last_became_active":"2024-04-02T08:53:22.901209+0000","last_became_peered":"2024-04-02T08:53:22.901209+0000","last_unstale":"2024-04-02T08:53:22.911149+0000","last_undegraded":"2024-04-02T08:53:22.911149+0000","last_fullsized":"2024-04-02T08:53:22.911149+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:31.589774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.7","version":"185'2","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896349+0000","last_change":"2024-04-02T08:53:22.403624+0000","last_active":"2024-04-02T08:53:23.896349+0000","last_peered":"2024-04-02T08:53:23.896349+0000","last_clean":"2024-04-02T08:53:23.896349+0000","last_became_active":"2024-04-02T08:53:22.401306+0000","last_became_peered":"2024-04-02T08:53:22.401306+0000","last_unstale":"2024-04-02T08:53:23.896349+0000","last_undegraded":"2024-04-02T08:53:23.896349+0000","last_fullsized":"2024-04-02T08:53:23.896349+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:19:11.682642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896408+0000","last_change":"2024-04-02T08:53:22.904129+0000","last_active":"2024-04-02T08:53:23.896408+0000","last_peered":"2024-04-02T08:53:23.896408+0000","last_clean":"2024-04-02T08:53:23.896408+0000","last_became_active":"2024-04-02T08:53:22.901928+0000","last_became_peered":"2024-04-02T08:53:22.901928+0000","last_unstale":"2024-04-02T08:53:23.896408+0000","last_undegraded":"2024-04-02T08:53:23.896408+0000","last_fullsized":"2024-04-02T08:53:23.896408+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:35:40.545122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":11,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.891323+0000","last_change":"2024-04-02T08:53:22.405805+0000","last_active":"2024-04-02T08:53:23.891323+0000","last_peered":"2024-04-02T08:53:23.891323+0000","last_clean":"2024-04-02T08:53:23.891323+0000","last_became_active":"2024-04-02T08:53:22.405500+0000","last_became_peered":"2024-04-02T08:53:22.405500+0000","last_unstale":"2024-04-02T08:53:23.891323+0000","last_undegraded":"2024-04-02T08:53:23.891323+0000","last_fullsized":"2024-04-02T08:53:23.891323+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:03:47.690639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895991+0000","last_change":"2024-04-02T08:53:22.893882+0000","last_active":"2024-04-02T08:53:23.895991+0000","last_peered":"2024-04-02T08:53:23.895991+0000","last_clean":"2024-04-02T08:53:23.895991+0000","last_became_active":"2024-04-02T08:53:22.893112+0000","last_became_peered":"2024-04-02T08:53:22.893112+0000","last_unstale":"2024-04-02T08:53:23.895991+0000","last_undegraded":"2024-04-02T08:53:23.895991+0000","last_fullsized":"2024-04-02T08:53:23.895991+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:10:25.277292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.888349+0000","last_change":"2024-04-02T08:53:22.406825+0000","last_active":"2024-04-02T08:53:22.888349+0000","last_peered":"2024-04-02T08:53:22.888349+0000","last_clean":"2024-04-02T08:53:22.888349+0000","last_became_active":"2024-04-02T08:53:22.406525+0000","last_became_peered":"2024-04-02T08:53:22.406525+0000","last_unstale":"2024-04-02T08:53:22.888349+0000","last_undegraded":"2024-04-02T08:53:22.888349+0000","last_fullsized":"2024-04-02T08:53:22.888349+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:49:57.573914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.896254+0000","last_change":"2024-04-02T08:53:22.910421+0000","last_active":"2024-04-02T08:53:23.896254+0000","last_peered":"2024-04-02T08:53:23.896254+0000","last_clean":"2024-04-02T08:53:23.896254+0000","last_became_active":"2024-04-02T08:53:22.910122+0000","last_became_peered":"2024-04-02T08:53:22.910122+0000","last_unstale":"2024-04-02T08:53:23.896254+0000","last_undegraded":"2024-04-02T08:53:23.896254+0000","last_fullsized":"2024-04-02T08:53:23.896254+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:41:40.713480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.899968+0000","last_change":"2024-04-02T08:53:22.395003+0000","last_active":"2024-04-02T08:53:22.899968+0000","last_peered":"2024-04-02T08:53:22.899968+0000","last_clean":"2024-04-02T08:53:22.899968+0000","last_became_active":"2024-04-02T08:53:22.394674+0000","last_became_peered":"2024-04-02T08:53:22.394674+0000","last_unstale":"2024-04-02T08:53:22.899968+0000","last_undegraded":"2024-04-02T08:53:22.899968+0000","last_fullsized":"2024-04-02T08:53:22.899968+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:00:11.383903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":10,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-02T08:53:23.895700+0000","last_change":"2024-04-02T08:53:22.908654+0000","last_active":"2024-04-02T08:53:23.895700+0000","last_peered":"2024-04-02T08:53:23.895700+0000","last_clean":"2024-04-02T08:53:23.895700+0000","last_became_active":"2024-04-02T08:53:22.904691+0000","last_became_peered":"2024-04-02T08:53:22.904691+0000","last_unstale":"2024-04-02T08:53:23.895700+0000","last_undegraded":"2024-04-02T08:53:23.895700+0000","last_fullsized":"2024-04-02T08:53:23.895700+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:20:20.210105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.889255+0000","last_change":"2024-04-02T08:53:22.401575+0000","last_active":"2024-04-02T08:53:22.889255+0000","last_peered":"2024-04-02T08:53:22.889255+0000","last_clean":"2024-04-02T08:53:22.889255+0000","last_became_active":"2024-04-02T08:53:22.401137+0000","last_became_peered":"2024-04-02T08:53:22.401137+0000","last_unstale":"2024-04-02T08:53:22.889255+0000","last_undegraded":"2024-04-02T08:53:22.889255+0000","last_fullsized":"2024-04-02T08:53:22.889255+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:31:44.571285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.913207+0000","last_change":"2024-04-02T08:53:22.907747+0000","last_active":"2024-04-02T08:53:22.913207+0000","last_peered":"2024-04-02T08:53:22.913207+0000","last_clean":"2024-04-02T08:53:22.913207+0000","last_became_active":"2024-04-02T08:53:22.907496+0000","last_became_peered":"2024-04-02T08:53:22.907496+0000","last_unstale":"2024-04-02T08:53:22.913207+0000","last_undegraded":"2024-04-02T08:53:22.913207+0000","last_fullsized":"2024-04-02T08:53:22.913207+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:59:13.633190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.903595+0000","last_change":"2024-04-02T08:53:22.408202+0000","last_active":"2024-04-02T08:53:22.903595+0000","last_peered":"2024-04-02T08:53:22.903595+0000","last_clean":"2024-04-02T08:53:22.903595+0000","last_became_active":"2024-04-02T08:53:22.407929+0000","last_became_peered":"2024-04-02T08:53:22.407929+0000","last_unstale":"2024-04-02T08:53:22.903595+0000","last_undegraded":"2024-04-02T08:53:22.903595+0000","last_fullsized":"2024-04-02T08:53:22.903595+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:10:29.755694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.915578+0000","last_change":"2024-04-02T08:53:22.909704+0000","last_active":"2024-04-02T08:53:22.915578+0000","last_peered":"2024-04-02T08:53:22.915578+0000","last_clean":"2024-04-02T08:53:22.915578+0000","last_became_active":"2024-04-02T08:53:22.909119+0000","last_became_peered":"2024-04-02T08:53:22.909119+0000","last_unstale":"2024-04-02T08:53:22.915578+0000","last_undegraded":"2024-04-02T08:53:22.915578+0000","last_fullsized":"2024-04-02T08:53:22.915578+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:32:52.919204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.887259+0000","last_change":"2024-04-02T08:53:22.407011+0000","last_active":"2024-04-02T08:53:22.887259+0000","last_peered":"2024-04-02T08:53:22.887259+0000","last_clean":"2024-04-02T08:53:22.887259+0000","last_became_active":"2024-04-02T08:53:22.406722+0000","last_became_peered":"2024-04-02T08:53:22.406722+0000","last_unstale":"2024-04-02T08:53:22.887259+0000","last_undegraded":"2024-04-02T08:53:22.887259+0000","last_fullsized":"2024-04-02T08:53:22.887259+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:10:29.755694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.908252+0000","last_change":"2024-04-02T08:53:22.898910+0000","last_active":"2024-04-02T08:53:22.908252+0000","last_peered":"2024-04-02T08:53:22.908252+0000","last_clean":"2024-04-02T08:53:22.908252+0000","last_became_active":"2024-04-02T08:53:22.898627+0000","last_became_peered":"2024-04-02T08:53:22.898627+0000","last_unstale":"2024-04-02T08:53:22.908252+0000","last_undegraded":"2024-04-02T08:53:22.908252+0000","last_fullsized":"2024-04-02T08:53:22.908252+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:47:04.773777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.886684+0000","last_change":"2024-04-02T08:53:22.405009+0000","last_active":"2024-04-02T08:53:22.886684+0000","last_peered":"2024-04-02T08:53:22.886684+0000","last_clean":"2024-04-02T08:53:22.886684+0000","last_became_active":"2024-04-02T08:53:22.404071+0000","last_became_peered":"2024-04-02T08:53:22.404071+0000","last_unstale":"2024-04-02T08:53:22.886684+0000","last_undegraded":"2024-04-02T08:53:22.886684+0000","last_fullsized":"2024-04-02T08:53:22.886684+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:15:46.243319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":9,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.911310+0000","last_change":"2024-04-02T08:53:22.907997+0000","last_active":"2024-04-02T08:53:22.911310+0000","last_peered":"2024-04-02T08:53:22.911310+0000","last_clean":"2024-04-02T08:53:22.911310+0000","last_became_active":"2024-04-02T08:53:22.905426+0000","last_became_peered":"2024-04-02T08:53:22.905426+0000","last_unstale":"2024-04-02T08:53:22.911310+0000","last_undegraded":"2024-04-02T08:53:22.911310+0000","last_fullsized":"2024-04-02T08:53:22.911310+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:22.372921+0000","last_clean_scrub_stamp":"2024-04-02T08:53:22.372921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:47:04.773777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.890347+0000","last_change":"2024-04-02T08:53:22.406009+0000","last_active":"2024-04-02T08:53:22.890347+0000","last_peered":"2024-04-02T08:53:22.890347+0000","last_clean":"2024-04-02T08:53:22.890347+0000","last_became_active":"2024-04-02T08:53:22.405740+0000","last_became_peered":"2024-04-02T08:53:22.405740+0000","last_unstale":"2024-04-02T08:53:22.890347+0000","last_undegraded":"2024-04-02T08:53:22.890347+0000","last_fullsized":"2024-04-02T08:53:22.890347+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T08:53:21.366099+0000","last_clean_scrub_stamp":"2024-04-02T08:53:21.366099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:39:15.431924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"175'293","reported_seq":972,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-02T08:53:22.392303+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T08:53:22.392303+0000","last_peered":"2024-04-02T08:53:22.392303+0000","last_clean":"2024-04-02T08:53:22.392303+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T08:53:22.392303+0000","last_undegraded":"2024-04-02T08:53:22.392303+0000","last_fullsized":"2024-04-02T08:53:22.392303+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575765,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55532,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316308,"statfs":{"total":96636764160,"available":96579899392,"internally_reserved":0,"allocated":2326528,"data_stored":1166204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:53:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.70699999999999996,"15min":0.71599999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.86199999999999999,"5min":1.069,"15min":1.069},"last":0.432},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.69899999999999995,"15min":0.70899999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.85399999999999998,"5min":1.0129999999999999,"15min":1.0169999999999999},"last":0.67600000000000005}]},{"osd":1,"last update":"Tue Apr 2 08:53:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.70899999999999996,"15min":0.72899999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.028},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.70299999999999996,"15min":0.70899999999999996},"min":{"1min":0.46000000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82799999999999996,"5min":1.016,"15min":1.016},"last":0.53000000000000003}]},{"osd":2,"last update":"Tue Apr 2 08:53:17 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.73399999999999999,"15min":0.752},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.82099999999999995,"5min":1.105,"15min":1.105},"last":0.498},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.79600000000000004,"15min":0.80000000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.497},"max":{"1min":0.93400000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.60199999999999998}]},{"osd":3,"last update":"Tue Apr 2 08:53:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.72499999999999998,"15min":0.73399999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":1.073,"15min":1.073},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.745,"15min":0.75600000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.97699999999999998,"5min":1.048,"15min":1.056},"last":0.51900000000000002}]},{"osd":4,"last update":"Tue Apr 2 08:53:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.35199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.84299999999999997,"5min":0.90700000000000003,"15min":0.93200000000000005},"last":0.622},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71299999999999997,"15min":0.70699999999999996},"min":{"1min":0.55200000000000005,"5min":0.44600000000000001,"15min":0.376},"max":{"1min":0.89200000000000002,"5min":0.997,"15min":1.014},"last":0.63100000000000001}]},{"osd":5,"last update":"Tue Apr 2 08:53:17 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.68500000000000005,"15min":0.66800000000000004},"min":{"1min":0.435,"5min":0.435,"15min":0.38700000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.70599999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.873,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.58099999999999996}]},{"osd":6,"last update":"Tue Apr 2 08:53:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.66600000000000004,"15min":0.66300000000000003},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.46500000000000002},"max":{"1min":0.81599999999999995,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91800000000000004,"5min":0.98699999999999999,"15min":1.0149999999999999},"last":0.69299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575763,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41572,"kb_used_data":2264,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330268,"statfs":{"total":96636764160,"available":96594194432,"internally_reserved":0,"allocated":2318336,"data_stored":1159518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.72299999999999998,"15min":0.72499999999999998},"min":{"1min":0.38100000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.1990000000000001,"5min":1.847,"15min":1.847},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.113,"5min":0.82799999999999996,"15min":0.76600000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":10.914,"5min":10.914,"15min":10.914},"last":10.914}]},{"osd":1,"last update":"Tue Apr 2 08:53:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":0.80500000000000005,"15min":0.753},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":10.522,"5min":10.522,"15min":10.522},"last":10.522},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.753,"15min":0.748},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.4139999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.4139999999999999}]},{"osd":2,"last update":"Tue Apr 2 08:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.72899999999999998,"15min":0.746},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.141,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.107,"5min":0.81000000000000005,"15min":0.755},"min":{"1min":0.42999999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":10.779,"5min":10.779,"15min":10.779},"last":10.779}]},{"osd":3,"last update":"Tue Apr 2 08:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.749,"15min":0.754},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.238,"5min":1.722,"15min":1.722},"last":1.119},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77900000000000003,"15min":0.77500000000000002},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":2.5209999999999999}]},{"osd":4,"last update":"Tue Apr 2 08:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63500000000000001,"15min":0.625},"min":{"1min":0.40500000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.278,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.278},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67800000000000005,"15min":0.65400000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.069,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.70999999999999996}]},{"osd":5,"last update":"Tue Apr 2 08:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.67300000000000004,"15min":0.64400000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.40100000000000002},"max":{"1min":1.3280000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.3280000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.73399999999999999,"15min":0.70599999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.43099999999999999},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.4790000000000001}]},{"osd":7,"last update":"Tue Apr 2 08:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.748,"15min":0.71599999999999997},"min":{"1min":0.58899999999999997,"5min":0.41999999999999998,"15min":0.40999999999999998},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":1.46},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.74199999999999999,"15min":0.71999999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":1.1910000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575764,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47936,"kb_used_data":2864,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323904,"statfs":{"total":96636764160,"available":96587677696,"internally_reserved":0,"allocated":2932736,"data_stored":1751598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:53:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.46000000000000002,"15min":0.46400000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.59099999999999997,"5min":0.80100000000000005,"15min":0.81100000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.51400000000000001,"15min":0.51800000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34100000000000003},"max":{"1min":0.68300000000000005,"5min":0.876,"15min":0.94799999999999995},"last":0.438}]},{"osd":2,"last update":"Tue Apr 2 08:53:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.54100000000000004,"15min":0.53000000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.65400000000000003,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.63200000000000001,"15min":0.61399999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73899999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.53200000000000003}]},{"osd":3,"last update":"Tue Apr 2 08:53:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60899999999999999,"15min":0.59799999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.69099999999999995,"5min":1.137,"15min":1.226},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69299999999999995,"5min":0.93200000000000005,"15min":1.0700000000000001},"last":0.495}]},{"osd":4,"last update":"Tue Apr 2 08:53:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.05,"5min":1.1140000000000001,"15min":1.321},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.81999999999999995,"15min":0.80500000000000005},"min":{"1min":0.55800000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.2310000000000001,"5min":1.381,"15min":1.381},"last":0.66900000000000004}]},{"osd":5,"last update":"Tue Apr 2 08:53:08 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.80900000000000005,"15min":0.80200000000000005},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0580000000000001,"5min":1.131,"15min":1.131},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.751,"5min":0.83399999999999996,"15min":0.81699999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53600000000000003},"max":{"1min":1.081,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.78300000000000003}]},{"osd":6,"last update":"Tue Apr 2 08:53:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.83499999999999996,"15min":0.82999999999999996},"min":{"1min":0.64600000000000002,"5min":0.627,"15min":0.57299999999999995},"max":{"1min":1.147,"5min":1.2809999999999999,"15min":1.3759999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.83599999999999997,"15min":0.82899999999999996},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.53700000000000003},"max":{"1min":1.1220000000000001,"5min":1.1839999999999999,"15min":1.242},"last":0.63}]},{"osd":7,"last update":"Tue Apr 2 08:53:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.85699999999999998,"15min":0.84399999999999997},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.56299999999999994},"max":{"1min":1.1850000000000001,"5min":1.347,"15min":1.347},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.84699999999999998,"15min":0.84399999999999997},"min":{"1min":0.627,"5min":0.623,"15min":0.621},"max":{"1min":1.2090000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.60499999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47328,"kb_used_data":2260,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324512,"statfs":{"total":96636764160,"available":96588300288,"internally_reserved":0,"allocated":2314240,"data_stored":1158535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.45400000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60099999999999998,"5min":0.64100000000000001,"15min":0.75700000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.5,"15min":0.498},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73499999999999999,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.41799999999999998}]},{"osd":2,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.50700000000000001,"15min":0.50800000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.76200000000000001,"5min":0.80700000000000005,"15min":0.83299999999999996},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.54500000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73999999999999999,"5min":0.79700000000000004,"15min":0.80900000000000005},"last":0.45700000000000002}]},{"osd":3,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55700000000000005,"15min":0.56699999999999995},"min":{"1min":0.40799999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.86799999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63,"15min":0.64300000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81999999999999995,"5min":0.88100000000000001,"15min":0.91900000000000004},"last":0.64900000000000002}]},{"osd":4,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.83199999999999996,"15min":0.81899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.999,"5min":1.0980000000000001,"15min":1.236},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.83899999999999997,"15min":0.82899999999999996},"min":{"1min":0.61899999999999999,"5min":0.60199999999999998,"15min":0.56200000000000006},"max":{"1min":1.056,"5min":1.056,"15min":1.2929999999999999},"last":0.69699999999999995}]},{"osd":5,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.83199999999999996,"15min":0.82699999999999996},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.57799999999999996},"max":{"1min":0.95599999999999996,"5min":1.0820000000000001,"15min":1.369},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.53700000000000003},"max":{"1min":0.98899999999999999,"5min":1.042,"15min":1.27},"last":0.504}]},{"osd":6,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.57999999999999996,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.052,"5min":1.052,"15min":1.1259999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.80200000000000005,"15min":0.79600000000000004},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.99099999999999999,"5min":1.089,"15min":1.262},"last":0.68100000000000005}]},{"osd":7,"last update":"Tue Apr 2 08:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.85999999999999999,"15min":0.84999999999999998},"min":{"1min":0.57099999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0680000000000001,"5min":1.125,"15min":1.321},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.83299999999999996,"15min":0.84199999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.083,"5min":1.1120000000000001,"15min":1.2989999999999999},"last":0.77400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575763,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47340,"kb_used_data":2264,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324500,"statfs":{"total":96636764160,"available":96588288000,"internally_reserved":0,"allocated":2318336,"data_stored":1160066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47599999999999998,"15min":0.45600000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.31},"max":{"1min":0.70099999999999996,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.309},{"interface":"front","average":{"1min":0.499,"5min":0.50800000000000001,"15min":0.50700000000000001},"min":{"1min":0.40000000000000002,"5min":0.35099999999999998,"15min":0.316},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":0.81000000000000005},"last":0.45000000000000001}]},{"osd":1,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54100000000000004,"15min":0.51700000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.313},"max":{"1min":0.751,"5min":1.262,"15min":1.262},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.39500000000000002,"5min":0.38,"15min":0.35599999999999998},"max":{"1min":0.72399999999999998,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.501}]},{"osd":3,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.39400000000000002,"5min":0.38500000000000001,"15min":0.30599999999999999},"max":{"1min":2.3039999999999998,"5min":2.3039999999999998,"15min":2.3039999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.65700000000000003,"15min":0.64000000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.45000000000000001},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.47299999999999998}]},{"osd":4,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.81200000000000006,"15min":0.80500000000000005},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.443},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80300000000000005,"15min":0.80600000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.42399999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.83799999999999997}]},{"osd":5,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.82099999999999995,"15min":0.80600000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.48399999999999999},"max":{"1min":2.177,"5min":2.177,"15min":2.177},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.83399999999999996,"15min":0.81799999999999995},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.47899999999999998},"max":{"1min":1.319,"5min":1.319,"15min":1.3600000000000001},"last":0.76300000000000001}]},{"osd":6,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.47499999999999998},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.84299999999999997,"15min":0.83199999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.50700000000000001},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":0.71599999999999997}]},{"osd":7,"last update":"Tue Apr 2 08:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.84699999999999998,"15min":0.83799999999999997},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.502},"max":{"1min":1.528,"5min":1.528,"15min":1.528},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.86199999999999999,"15min":0.84599999999999997},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.47399999999999998},"max":{"1min":2.258,"5min":2.258,"15min":2.258},"last":0.74299999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575763,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47368,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324472,"statfs":{"total":96636764160,"available":96588259328,"internally_reserved":0,"allocated":2342912,"data_stored":1167790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.44600000000000001,"15min":0.45100000000000001},"min":{"1min":0.315,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.627,"5min":0.64800000000000002,"15min":0.92700000000000005},"last":0.371},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.48499999999999999,"15min":0.49099999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.30599999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.87},"last":0.51000000000000001}]},{"osd":1,"last update":"Tue Apr 2 08:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.50600000000000001,"15min":0.50700000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.66600000000000004,"5min":0.90300000000000002,"15min":0.91400000000000003},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40100000000000002},"max":{"1min":0.69899999999999995,"5min":0.82999999999999996,"15min":0.92000000000000004},"last":0.47799999999999998}]},{"osd":2,"last update":"Tue Apr 2 08:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.623,"15min":0.63300000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78300000000000003,"5min":0.83599999999999997,"15min":0.86299999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.745,"5min":1.022,"15min":1.022},"last":0.52000000000000002}]},{"osd":4,"last update":"Tue Apr 2 08:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.79100000000000004,"15min":0.78300000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.95099999999999996,"5min":1.1000000000000001,"15min":1.167},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.80300000000000005,"15min":0.79600000000000004},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.94399999999999995,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.66500000000000004}]},{"osd":5,"last update":"Tue Apr 2 08:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.80900000000000005,"15min":0.79800000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.54600000000000004},"max":{"1min":0.97699999999999998,"5min":1.115,"15min":1.149},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.79900000000000004,"15min":0.79800000000000004},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.05,"5min":1.0760000000000001,"15min":1.1020000000000001},"last":0.74199999999999999}]},{"osd":6,"last update":"Tue Apr 2 08:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.80400000000000005,"15min":0.80100000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.99299999999999999,"5min":1.153,"15min":1.153},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.747,"5min":0.84299999999999997,"15min":0.82599999999999996},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.55600000000000005},"max":{"1min":1.0289999999999999,"5min":1.143,"15min":1.1870000000000001},"last":0.72499999999999998}]},{"osd":7,"last update":"Tue Apr 2 08:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.84799999999999998,"15min":0.82299999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97499999999999998,"5min":1.125,"15min":1.2070000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.752,"5min":0.85099999999999998,"15min":0.82999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0309999999999999,"5min":1.194,"15min":1.194},"last":0.69399999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575763,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59632,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312208,"statfs":{"total":96636764160,"available":96575700992,"internally_reserved":0,"allocated":2330624,"data_stored":1166238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.69199999999999995,"15min":0.70099999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89600000000000002,"5min":0.92400000000000004,"15min":1.0600000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.68500000000000005,"15min":0.69199999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.0429999999999999},"last":0.56000000000000005}]},{"osd":1,"last update":"Tue Apr 2 08:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.747,"15min":0.752},"min":{"1min":0.48199999999999998,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":1.121},"last":0.495},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78200000000000003,"15min":0.78000000000000003},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.49099999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1200000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Tue Apr 2 08:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.69999999999999996,"15min":0.71099999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.46600000000000003},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.1879999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71299999999999997,"15min":0.73099999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.1259999999999999},"last":0.53000000000000003}]},{"osd":3,"last update":"Tue Apr 2 08:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.75700000000000001,"15min":0.76500000000000001},"min":{"1min":0.60399999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.2589999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.74099999999999999,"15min":0.74399999999999999},"min":{"1min":0.54600000000000004,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.077},"last":0.69999999999999996}]},{"osd":5,"last update":"Tue Apr 2 08:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.59699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41599999999999998,"5min":0.40400000000000003,"15min":0.38400000000000001},"max":{"1min":0.877,"5min":0.90600000000000003,"15min":1.0940000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.90700000000000003,"5min":0.93899999999999995,"15min":1.073},"last":0.63200000000000001}]},{"osd":6,"last update":"Tue Apr 2 08:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.64000000000000001,"15min":0.63700000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.98199999999999998,"5min":1.022,"15min":1.022},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.68400000000000005,"15min":0.69399999999999995},"min":{"1min":0.48999999999999999,"5min":0.46899999999999997,"15min":0.439},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.1120000000000001},"last":0.72999999999999998}]},{"osd":7,"last update":"Tue Apr 2 08:52:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.67800000000000005,"15min":0.69699999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98599999999999999,"5min":1.048,"15min":1.129},"last":0.749},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.64500000000000002,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.3440000000000001},"last":0.76100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575766,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47916,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323924,"statfs":{"total":96636764160,"available":96587698176,"internally_reserved":0,"allocated":2912256,"data_stored":1749505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 08:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.70399999999999996,"15min":0.71499999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.377},"max":{"1min":0.84999999999999998,"5min":1.502,"15min":1.502},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.72899999999999998,"15min":0.72999999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.89800000000000002,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.752}]},{"osd":1,"last update":"Tue Apr 2 08:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71799999999999997,"15min":0.72399999999999998},"min":{"1min":0.53500000000000003,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.71999999999999997,"15min":0.73399999999999999},"min":{"1min":0.48099999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.91500000000000004,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.78400000000000003}]},{"osd":2,"last update":"Tue Apr 2 08:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.749,"15min":0.74099999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.49199999999999999},"max":{"1min":0.94399999999999995,"5min":1.232,"15min":1.232},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.73799999999999999,"15min":0.74199999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92500000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.624}]},{"osd":3,"last update":"Tue Apr 2 08:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.79300000000000004,"15min":0.78900000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.46800000000000003},"max":{"1min":0.93799999999999994,"5min":1.556,"15min":1.556},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.79200000000000004,"15min":0.79500000000000004},"min":{"1min":0.53400000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0029999999999999,"5min":1.0189999999999999,"15min":1.069},"last":0.76600000000000001}]},{"osd":4,"last update":"Tue Apr 2 08:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63,"15min":0.61599999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.32700000000000001},"max":{"1min":0.91700000000000004,"5min":0.97899999999999998,"15min":1.0189999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.66100000000000003},"min":{"1min":0.46300000000000002,"5min":0.44600000000000001,"15min":0.435},"max":{"1min":0.97899999999999998,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.82999999999999996}]},{"osd":6,"last update":"Tue Apr 2 08:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.66600000000000004,"15min":0.66800000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.44700000000000001},"max":{"1min":0.91600000000000004,"5min":0.99199999999999999,"15min":1.0640000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.68700000000000006,"15min":0.69499999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.377},"max":{"1min":0.879,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.72199999999999998}]},{"osd":7,"last update":"Tue Apr 2 08:53:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.72399999999999998,"15min":0.71799999999999997},"min":{"1min":0.441,"5min":0.441,"15min":0.41599999999999998},"max":{"1min":1.0049999999999999,"5min":1.423,"15min":1.423},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73799999999999999,"15min":0.745},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.44500000000000001},"max":{"1min":0.92600000000000005,"5min":1.038,"15min":1.038},"last":0.68400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T08:53:28.279 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T08:53:28.279 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T08:53:28.368 INFO:tasks.ceph.mds.c.smithi089.stderr:2024-04-02T08:53:28.369+0000 7f9f5da00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T08:53:28.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.481+0000 7f0c38000700 1 Processor -- start 2024-04-02T08:53:28.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.482+0000 7f0c38000700 1 -- start start 2024-04-02T08:53:28.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.483+0000 7f0c38000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0c3015a5c0 0x7f0c300709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:28.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.483+0000 7f0c38000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c300cb310 con 0x7f0c300ce840 2024-04-02T08:53:28.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.483+0000 7f0c38000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c30070f00 con 0x7f0c3015dfe0 2024-04-02T08:53:28.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.483+0000 7f0c38000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c300711a0 con 0x7f0c3015a5c0 2024-04-02T08:53:28.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.483+0000 7f0c36000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0c300ce840 0x7f0c300cba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46396/0 (socket says 172.21.15.89:46396) 2024-04-02T08:53:28.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.483+0000 7f0c36000700 1 -- 172.21.15.89:0/1366527327 learned_addr learned my addr 172.21.15.89:0/1366527327 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.483+0000 7f0c35600700 1 --2- 172.21.15.89:0/1366527327 >> v2:172.21.15.113:3300/0 conn(0x7f0c3015a5c0 0x7f0c300709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.483+0000 7f0c2fe00700 1 -- 172.21.15.89:0/1366527327 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879881423 0 0) 0x7f0c300cb310 con 0x7f0c300ce840 2024-04-02T08:53:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.483+0000 7f0c2fe00700 1 -- 172.21.15.89:0/1366527327 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c180035b0 con 0x7f0c300ce840 2024-04-02T08:53:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c2fe00700 1 -- 172.21.15.89:0/1366527327 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1283192495 0 0) 0x7f0c180035b0 con 0x7f0c300ce840 2024-04-02T08:53:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c2fe00700 1 -- 172.21.15.89:0/1366527327 >> v2:172.21.15.113:3300/0 conn(0x7f0c3015a5c0 msgr2=0x7f0c300709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c2fe00700 1 --2- 172.21.15.89:0/1366527327 >> v2:172.21.15.113:3300/0 conn(0x7f0c3015a5c0 0x7f0c300709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c2fe00700 1 -- 172.21.15.89:0/1366527327 >> v1:172.21.15.113:6789/0 conn(0x7f0c3015dfe0 legacy=0x7f0c300ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c2fe00700 1 -- 172.21.15.89:0/1366527327 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c30071480 con 0x7f0c300ce840 2024-04-02T08:53:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c2fe00700 1 -- 172.21.15.89:0/1366527327 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f0c280019f0 con 0x7f0c300ce840 2024-04-02T08:53:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c2fe00700 1 -- 172.21.15.89:0/1366527327 >> v1:172.21.15.89:6789/0 conn(0x7f0c300ce840 legacy=0x7f0c300cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c2fe00700 1 --2- 172.21.15.89:0/1366527327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c18003a00 0x7f0c18003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c2fe00700 1 --2- 172.21.15.89:0/1366527327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c18004850 0x7f0c18004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c2fe00700 1 --2- 172.21.15.89:0/1366527327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c18005210 0x7f0c1800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c2fe00700 1 -- 172.21.15.89:0/1366527327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c300711a0 con 0x7f0c18004850 2024-04-02T08:53:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c2fe00700 1 -- 172.21.15.89:0/1366527327 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1800c0b0 con 0x7f0c18005210 2024-04-02T08:53:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c2fe00700 1 -- 172.21.15.89:0/1366527327 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1800c270 con 0x7f0c18003a00 2024-04-02T08:53:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c34c00700 1 --2- 172.21.15.89:0/1366527327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c18004850 0x7f0c18004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:28.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c35600700 1 --2- 172.21.15.89:0/1366527327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c18003a00 0x7f0c18003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:28.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.484+0000 7f0c36000700 1 --2- 172.21.15.89:0/1366527327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c18005210 0x7f0c1800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:28.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.485+0000 7f0c34c00700 1 -- 172.21.15.89:0/1366527327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c18003a00 msgr2=0x7f0c18003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:28.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.485+0000 7f0c34c00700 1 --2- 172.21.15.89:0/1366527327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c18003a00 0x7f0c18003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.485+0000 7f0c34c00700 1 -- 172.21.15.89:0/1366527327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c18005210 msgr2=0x7f0c1800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:28.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.485+0000 7f0c34c00700 1 --2- 172.21.15.89:0/1366527327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c18005210 0x7f0c1800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.485+0000 7f0c34c00700 1 -- 172.21.15.89:0/1366527327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c180035b0 con 0x7f0c18004850 2024-04-02T08:53:28.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.485+0000 7f0c36000700 1 -- 172.21.15.89:0/1366527327 reap_dead start 2024-04-02T08:53:28.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.485+0000 7f0c34c00700 1 --2- 172.21.15.89:0/1366527327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c18004850 0x7f0c18004cd0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0017c0 tx=0x7f0c18005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a885d42b324fdb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:28.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.485+0000 7f0c2fe00700 1 -- 172.21.15.89:0/1366527327 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c0052d0 con 0x7f0c18004850 2024-04-02T08:53:28.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.485+0000 7f0c2fe00700 1 -- 172.21.15.89:0/1366527327 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1c005450 con 0x7f0c18004850 2024-04-02T08:53:28.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.485+0000 7f0c2fe00700 1 -- 172.21.15.89:0/1366527327 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c0038c0 con 0x7f0c18004850 2024-04-02T08:53:28.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.486+0000 7f0c38000700 1 -- 172.21.15.89:0/1366527327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c18004850 msgr2=0x7f0c18004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:28.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.486+0000 7f0c38000700 1 --2- 172.21.15.89:0/1366527327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c18004850 0x7f0c18004cd0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0017c0 tx=0x7f0c18005880 comp rx=0 tx=0).stop 2024-04-02T08:53:28.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.486+0000 7f0c38000700 1 -- 172.21.15.89:0/1366527327 shutdown_connections 2024-04-02T08:53:28.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.486+0000 7f0c38000700 1 --2- 172.21.15.89:0/1366527327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c18004850 0x7f0c18004cd0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.486+0000 7f0c38000700 1 -- 172.21.15.89:0/1366527327 >> 172.21.15.89:0/1366527327 conn(0x7f0c300b0d70 msgr2=0x7f0c300b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:28.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.486+0000 7f0c38000700 1 -- 172.21.15.89:0/1366527327 shutdown_connections 2024-04-02T08:53:28.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.486+0000 7f0c38000700 1 -- 172.21.15.89:0/1366527327 wait complete. 2024-04-02T08:53:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.487+0000 7f0c38000700 1 Processor -- start 2024-04-02T08:53:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.488+0000 7f0c38000700 1 -- start start 2024-04-02T08:53:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c38000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c18004850 0x7f0c30155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c38000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c300cc620 0x7f0c300c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c38000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c300c9980 0x7f0c30157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c38000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c301592f0 con 0x7f0c18004850 2024-04-02T08:53:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c38000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c30155c40 con 0x7f0c300c9980 2024-04-02T08:53:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c38000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c300ca630 con 0x7f0c300cc620 2024-04-02T08:53:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c35600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c18004850 0x7f0c30155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c35600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c18004850 0x7f0c30155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54170/0 (socket says 172.21.15.89:54170) 2024-04-02T08:53:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c35600700 1 -- 172.21.15.89:0/1396077633 learned_addr learned my addr 172.21.15.89:0/1396077633 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c34c00700 1 --2- 172.21.15.89:0/1396077633 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c300cc620 0x7f0c300c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c36000700 1 --2- 172.21.15.89:0/1396077633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c300c9980 0x7f0c30157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c35600700 1 -- 172.21.15.89:0/1396077633 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c300cc620 msgr2=0x7f0c300c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c35600700 1 --2- 172.21.15.89:0/1396077633 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c300cc620 0x7f0c300c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c35600700 1 -- 172.21.15.89:0/1396077633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c300c9980 msgr2=0x7f0c30157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c35600700 1 --2- 172.21.15.89:0/1396077633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c300c9980 0x7f0c30157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c35600700 1 -- 172.21.15.89:0/1396077633 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3015b570 con 0x7f0c18004850 2024-04-02T08:53:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.489+0000 7f0c35600700 1 --2- 172.21.15.89:0/1396077633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c18004850 0x7f0c30155700 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f0c24011aa0 tx=0x7f0c24011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.490+0000 7f0c2d600700 1 -- 172.21.15.89:0/1396077633 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2400f9e0 con 0x7f0c18004850 2024-04-02T08:53:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.490+0000 7f0c2d600700 1 -- 172.21.15.89:0/1396077633 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2400fb40 con 0x7f0c18004850 2024-04-02T08:53:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.490+0000 7f0c2d600700 1 -- 172.21.15.89:0/1396077633 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c24014440 con 0x7f0c18004850 2024-04-02T08:53:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.490+0000 7f0c38000700 1 -- 172.21.15.89:0/1396077633 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c300ca350 con 0x7f0c18004850 2024-04-02T08:53:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.490+0000 7f0c38000700 1 -- 172.21.15.89:0/1396077633 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3015baf0 con 0x7f0c18004850 2024-04-02T08:53:28.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.491+0000 7f0c38000700 1 -- 172.21.15.89:0/1396077633 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c30067720 con 0x7f0c18004850 2024-04-02T08:53:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.492+0000 7f0c2d600700 1 -- 172.21.15.89:0/1396077633 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0c240145a0 con 0x7f0c18004850 2024-04-02T08:53:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.492+0000 7f0c2d600700 1 --2- 172.21.15.89:0/1396077633 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c1006adb0 0x7f0c1006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.493+0000 7f0c2d600700 1 -- 172.21.15.89:0/1396077633 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f0c24089e20 con 0x7f0c18004850 2024-04-02T08:53:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.493+0000 7f0c34c00700 1 --2- 172.21.15.89:0/1396077633 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c1006adb0 0x7f0c1006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.493+0000 7f0c34c00700 1 --2- 172.21.15.89:0/1396077633 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c1006adb0 0x7f0c1006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c000c00 tx=0x7f0c1c009030 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:28.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.497+0000 7f0c2d600700 1 -- 172.21.15.89:0/1396077633 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0c24054d30 con 0x7f0c18004850 2024-04-02T08:53:28.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.780+0000 7f0c38000700 1 -- 172.21.15.89:0/1396077633 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0c300b1ad0 con 0x7f0c18004850 2024-04-02T08:53:28.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.784+0000 7f0c2d600700 1 -- 172.21.15.89:0/1396077633 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v46) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0c24058350 con 0x7f0c18004850 2024-04-02T08:53:28.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.787+0000 7f0c38000700 1 -- 172.21.15.89:0/1396077633 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c1006adb0 msgr2=0x7f0c1006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:28.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.787+0000 7f0c38000700 1 --2- 172.21.15.89:0/1396077633 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c1006adb0 0x7f0c1006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c000c00 tx=0x7f0c1c009030 comp rx=0 tx=0).stop 2024-04-02T08:53:28.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.787+0000 7f0c38000700 1 -- 172.21.15.89:0/1396077633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c18004850 msgr2=0x7f0c30155700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:28.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.787+0000 7f0c38000700 1 --2- 172.21.15.89:0/1396077633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c18004850 0x7f0c30155700 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f0c24011aa0 tx=0x7f0c24011e60 comp rx=0 tx=0).stop 2024-04-02T08:53:28.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.787+0000 7f0c38000700 1 -- 172.21.15.89:0/1396077633 shutdown_connections 2024-04-02T08:53:28.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.787+0000 7f0c38000700 1 --2- 172.21.15.89:0/1396077633 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f0c1006adb0 0x7f0c1006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.787+0000 7f0c38000700 1 --2- 172.21.15.89:0/1396077633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c300c9980 0x7f0c30157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.787+0000 7f0c38000700 1 --2- 172.21.15.89:0/1396077633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c18004850 0x7f0c30155700 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.787+0000 7f0c38000700 1 --2- 172.21.15.89:0/1396077633 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c300cc620 0x7f0c300c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:28.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.787+0000 7f0c38000700 1 -- 172.21.15.89:0/1396077633 >> 172.21.15.89:0/1396077633 conn(0x7f0c300b0d70 msgr2=0x7f0c3006dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:28.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.787+0000 7f0c38000700 1 -- 172.21.15.89:0/1396077633 shutdown_connections 2024-04-02T08:53:28.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.787+0000 7f0c38000700 1 -- 172.21.15.89:0/1396077633 wait complete. 2024-04-02T08:53:28.787 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T08:53:28.804 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T08:53:28.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.998+0000 7f9857600700 1 Processor -- start 2024-04-02T08:53:28.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.998+0000 7f9857600700 1 -- start start 2024-04-02T08:53:28.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.999+0000 7f9857600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f985015d640 0x7f98500d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:28.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.999+0000 7f9857600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98500ceec0 0x7f98501612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:28.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.999+0000 7f9857600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98500d30d0 con 0x7f9850161800 2024-04-02T08:53:28.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.999+0000 7f9857600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9850161c90 con 0x7f985015d640 2024-04-02T08:53:28.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:28.999+0000 7f9857600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98500b6840 con 0x7f98500ceec0 2024-04-02T08:53:28.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.000+0000 7f984fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98500ceec0 0x7f98501612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.000+0000 7f9854c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f985015d640 0x7f98500d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.000+0000 7f9854c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f985015d640 0x7f98500d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:55020/0 (socket says 172.21.15.89:55020) 2024-04-02T08:53:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.000+0000 7f9854c00700 1 -- 172.21.15.89:0/778939029 learned_addr learned my addr 172.21.15.89:0/778939029 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.000+0000 7f984fe00700 1 --2- 172.21.15.89:0/778939029 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98500ceec0 0x7f98501612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:56210/0 (socket says 172.21.15.89:56210) 2024-04-02T08:53:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.000+0000 7f9854c00700 1 -- 172.21.15.89:0/778939029 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98500ceec0 msgr2=0x7f98501612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.001+0000 7f9854c00700 1 --2- 172.21.15.89:0/778939029 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98500ceec0 0x7f98501612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.001+0000 7f9854c00700 1 -- 172.21.15.89:0/778939029 >> v1:172.21.15.113:6789/0 conn(0x7f9850161800 legacy=0x7f98500cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.001+0000 7f9854c00700 1 -- 172.21.15.89:0/778939029 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98500d2880 con 0x7f985015d640 2024-04-02T08:53:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.001+0000 7f984fe00700 1 --2- 172.21.15.89:0/778939029 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98500ceec0 0x7f98501612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.001+0000 7f9854c00700 1 --2- 172.21.15.89:0/778939029 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985015d640 0x7f98500d2340 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f98440098f0 tx=0x7f9844009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0940ffd2efecea1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.001+0000 7f984f400700 1 -- 172.21.15.89:0/778939029 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844011830 con 0x7f985015d640 2024-04-02T08:53:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.001+0000 7f984f400700 1 -- 172.21.15.89:0/778939029 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9844011990 con 0x7f985015d640 2024-04-02T08:53:29.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.001+0000 7f984f400700 1 -- 172.21.15.89:0/778939029 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844011d20 con 0x7f985015d640 2024-04-02T08:53:29.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.002+0000 7f9857600700 1 -- 172.21.15.89:0/778939029 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985015d640 msgr2=0x7f98500d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.002+0000 7f9857600700 1 --2- 172.21.15.89:0/778939029 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985015d640 0x7f98500d2340 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f98440098f0 tx=0x7f9844009c00 comp rx=0 tx=0).stop 2024-04-02T08:53:29.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.002+0000 7f9857600700 1 -- 172.21.15.89:0/778939029 shutdown_connections 2024-04-02T08:53:29.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.002+0000 7f9857600700 1 --2- 172.21.15.89:0/778939029 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985015d640 0x7f98500d2340 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.002+0000 7f9857600700 1 --2- 172.21.15.89:0/778939029 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98500ceec0 0x7f98501612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.002+0000 7f9857600700 1 -- 172.21.15.89:0/778939029 >> 172.21.15.89:0/778939029 conn(0x7f985008aff0 msgr2=0x7f98500d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:29.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.002+0000 7f9857600700 1 -- 172.21.15.89:0/778939029 shutdown_connections 2024-04-02T08:53:29.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.002+0000 7f9857600700 1 -- 172.21.15.89:0/778939029 wait complete. 2024-04-02T08:53:29.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.003+0000 7f9857600700 1 Processor -- start 2024-04-02T08:53:29.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.004+0000 7f9857600700 1 -- start start 2024-04-02T08:53:29.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.004+0000 7f9857600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500ceec0 0x7f9850158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.004+0000 7f9857600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985015d640 0x7f98500c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f9857600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9850161800 0x7f985015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f9857600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9850159230 con 0x7f98500ceec0 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f9857600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98500c8b00 con 0x7f985015d640 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f9857600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98500c8de0 con 0x7f9850161800 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f9854c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500ceec0 0x7f9850158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f9854c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500ceec0 0x7f9850158cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57832/0 (socket says 172.21.15.89:57832) 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f9854c00700 1 -- 172.21.15.89:0/3579583549 learned_addr learned my addr 172.21.15.89:0/3579583549 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f984fe00700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985015d640 0x7f98500c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f9855600700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9850161800 0x7f985015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f9854c00700 1 -- 172.21.15.89:0/3579583549 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9850161800 msgr2=0x7f985015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f9854c00700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9850161800 0x7f985015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f9854c00700 1 -- 172.21.15.89:0/3579583549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985015d640 msgr2=0x7f98500c8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f9854c00700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985015d640 0x7f98500c8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f9854c00700 1 -- 172.21.15.89:0/3579583549 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983c0094a0 con 0x7f98500ceec0 2024-04-02T08:53:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.005+0000 7f9855600700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9850161800 0x7f985015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:29.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.006+0000 7f9854c00700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500ceec0 0x7f9850158cf0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f9844000c00 tx=0x7f9844004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:29.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.006+0000 7f984cc00700 1 -- 172.21.15.89:0/3579583549 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844004480 con 0x7f98500ceec0 2024-04-02T08:53:29.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.006+0000 7f984cc00700 1 -- 172.21.15.89:0/3579583549 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984401f070 con 0x7f98500ceec0 2024-04-02T08:53:29.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.006+0000 7f984cc00700 1 -- 172.21.15.89:0/3579583549 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98440038c0 con 0x7f98500ceec0 2024-04-02T08:53:29.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.006+0000 7f9857600700 1 -- 172.21.15.89:0/3579583549 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9844009550 con 0x7f98500ceec0 2024-04-02T08:53:29.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.006+0000 7f9857600700 1 -- 172.21.15.89:0/3579583549 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98500c93a0 con 0x7f98500ceec0 2024-04-02T08:53:29.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.007+0000 7f9857600700 1 -- 172.21.15.89:0/3579583549 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98500c95c0 con 0x7f98500ceec0 2024-04-02T08:53:29.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.008+0000 7f984cc00700 1 -- 172.21.15.89:0/3579583549 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9844021030 con 0x7f98500ceec0 2024-04-02T08:53:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.009+0000 7f984cc00700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f983006adb0 0x7f983006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.009+0000 7f984cc00700 1 -- 172.21.15.89:0/3579583549 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f984408aa10 con 0x7f98500ceec0 2024-04-02T08:53:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.009+0000 7f984fe00700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f983006adb0 0x7f983006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.010+0000 7f984fe00700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f983006adb0 0x7f983006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f983c009cd0 tx=0x7f983c009210 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:29.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.013+0000 7f984cc00700 1 -- 172.21.15.89:0/3579583549 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9844055a60 con 0x7f98500ceec0 2024-04-02T08:53:29.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.257+0000 7f9857600700 1 -- 172.21.15.89:0/3579583549 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9850067720 con 0x7f98500ceec0 2024-04-02T08:53:29.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.258+0000 7f984cc00700 1 -- 172.21.15.89:0/3579583549 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f9844015020 con 0x7f98500ceec0 2024-04-02T08:53:29.256 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T08:53:29.257 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1296569976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1296569976},{"type":"v1","addr":"172.21.15.113:6835","nonce":1296569976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9331,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1328950634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1328950634},{"type":"v1","addr":"172.21.15.89:6837","nonce":1328950634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":10032,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1167691520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1167691520},{"type":"v1","addr":"172.21.15.113:6834","nonce":1167691520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T08:53:22.881600+0000","modified":"2024-04-02T08:53:26.390144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10038":{"gid":10038,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/583021497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":583021497},{"type":"v1","addr":"172.21.15.89:6836","nonce":583021497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-04-02T08:53:29.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.260+0000 7f9857600700 1 -- 172.21.15.89:0/3579583549 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f983006adb0 msgr2=0x7f983006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.260+0000 7f9857600700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f983006adb0 0x7f983006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f983c009cd0 tx=0x7f983c009210 comp rx=0 tx=0).stop 2024-04-02T08:53:29.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.260+0000 7f9857600700 1 -- 172.21.15.89:0/3579583549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500ceec0 msgr2=0x7f9850158cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.260+0000 7f9857600700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500ceec0 0x7f9850158cf0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f9844000c00 tx=0x7f9844004750 comp rx=0 tx=0).stop 2024-04-02T08:53:29.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.260+0000 7f9857600700 1 -- 172.21.15.89:0/3579583549 shutdown_connections 2024-04-02T08:53:29.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.260+0000 7f9857600700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f983006adb0 0x7f983006d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.260+0000 7f9857600700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9850161800 0x7f985015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.260+0000 7f9857600700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500ceec0 0x7f9850158cf0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.260+0000 7f9857600700 1 --2- 172.21.15.89:0/3579583549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985015d640 0x7f98500c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.260+0000 7f9857600700 1 -- 172.21.15.89:0/3579583549 >> 172.21.15.89:0/3579583549 conn(0x7f985008aff0 msgr2=0x7f98500cfca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:29.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.260+0000 7f9857600700 1 -- 172.21.15.89:0/3579583549 shutdown_connections 2024-04-02T08:53:29.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.260+0000 7f9857600700 1 -- 172.21.15.89:0/3579583549 wait complete. 2024-04-02T08:53:29.260 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 89 2024-04-02T08:53:29.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T08:53:22.881600+0000', 'modified': '2024-04-02T08:53:26.390144+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10038}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10038': {'gid': 10038, 'name': 'c', 'rank': 0, 'incarnation': 85, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6836/583021497', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6834', 'nonce': 583021497}, {'type': 'v1', 'addr': '172.21.15.89:6836', 'nonce': 583021497}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T08:53:29.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T08:53:29.680 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-02T08:53:29.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.875+0000 7fab00400700 1 Processor -- start 2024-04-02T08:53:29.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.876+0000 7fab00400700 1 -- start start 2024-04-02T08:53:29.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.877+0000 7fab00400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faaf815d9d0 0x7faaf80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.877+0000 7fab00400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faaf80d2540 0x7faaf80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.877+0000 7fab00400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf8161420 con 0x7faaf80d16a0 2024-04-02T08:53:29.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.877+0000 7fab00400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf815edc0 con 0x7faaf80d2540 2024-04-02T08:53:29.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.877+0000 7fab00400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8161700 con 0x7faaf815d9d0 2024-04-02T08:53:29.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.877+0000 7faafd000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7faaf80d16a0 0x7faaf80d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57376/0 (socket says 172.21.15.89:57376) 2024-04-02T08:53:29.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.877+0000 7faafd000700 1 -- 172.21.15.89:0/2104308216 learned_addr learned my addr 172.21.15.89:0/2104308216 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:29.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.877+0000 7faafe400700 1 --2- 172.21.15.89:0/2104308216 >> v2:172.21.15.89:3300/0 conn(0x7faaf80d2540 0x7faaf80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.877+0000 7faafda00700 1 --2- 172.21.15.89:0/2104308216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faaf815d9d0 0x7faaf80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.877+0000 7faafe400700 1 -- 172.21.15.89:0/2104308216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faaf815d9d0 msgr2=0x7faaf80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.878+0000 7faafe400700 1 --2- 172.21.15.89:0/2104308216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faaf815d9d0 0x7faaf80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.878+0000 7faafe400700 1 -- 172.21.15.89:0/2104308216 >> v1:172.21.15.89:6789/0 conn(0x7faaf80d16a0 legacy=0x7faaf80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.878+0000 7faafe400700 1 -- 172.21.15.89:0/2104308216 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf81619e0 con 0x7faaf80d2540 2024-04-02T08:53:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.878+0000 7faaf7e00700 1 -- 172.21.15.89:0/2104308216 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127192642 0 0) 0x7faaf8161420 con 0x7faaf80d16a0 2024-04-02T08:53:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.878+0000 7faafda00700 1 --2- 172.21.15.89:0/2104308216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faaf815d9d0 0x7faaf80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:53:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.878+0000 7faafe400700 1 --2- 172.21.15.89:0/2104308216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf80d2540 0x7faaf80ce360 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009dc0 tx=0x7faaf000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e85c07e8447173 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.878+0000 7faaf7e00700 1 -- 172.21.15.89:0/2104308216 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf000e070 con 0x7faaf80d2540 2024-04-02T08:53:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.878+0000 7faaf7e00700 1 -- 172.21.15.89:0/2104308216 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf000bb90 con 0x7faaf80d2540 2024-04-02T08:53:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.878+0000 7faaf7e00700 1 -- 172.21.15.89:0/2104308216 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0012430 con 0x7faaf80d2540 2024-04-02T08:53:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.879+0000 7fab00400700 1 -- 172.21.15.89:0/2104308216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf80d2540 msgr2=0x7faaf80ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.879+0000 7fab00400700 1 --2- 172.21.15.89:0/2104308216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf80d2540 0x7faaf80ce360 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009dc0 tx=0x7faaf000b330 comp rx=0 tx=0).stop 2024-04-02T08:53:29.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.879+0000 7fab00400700 1 -- 172.21.15.89:0/2104308216 shutdown_connections 2024-04-02T08:53:29.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.879+0000 7fab00400700 1 --2- 172.21.15.89:0/2104308216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf80d2540 0x7faaf80ce360 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.879+0000 7fab00400700 1 --2- 172.21.15.89:0/2104308216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faaf815d9d0 0x7faaf80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.879+0000 7fab00400700 1 -- 172.21.15.89:0/2104308216 >> 172.21.15.89:0/2104308216 conn(0x7faaf808aff0 msgr2=0x7faaf80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:29.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.879+0000 7fab00400700 1 -- 172.21.15.89:0/2104308216 shutdown_connections 2024-04-02T08:53:29.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.879+0000 7fab00400700 1 -- 172.21.15.89:0/2104308216 wait complete. 2024-04-02T08:53:29.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.880+0000 7fab00400700 1 Processor -- start 2024-04-02T08:53:29.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.881+0000 7fab00400700 1 -- start start 2024-04-02T08:53:29.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.881+0000 7fab00400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faaf80d16a0 0x7faaf814fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.881+0000 7fab00400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80d2540 0x7faaf815be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7fab00400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf815d9d0 0x7faaf80b1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7fab00400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf80ca990 con 0x7faaf815d9d0 2024-04-02T08:53:29.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7fab00400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf80cac70 con 0x7faaf80d2540 2024-04-02T08:53:29.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7fab00400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf80caf50 con 0x7faaf80d16a0 2024-04-02T08:53:29.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7faafe400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf815d9d0 0x7faaf80b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7faafe400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf815d9d0 0x7faaf80b1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57848/0 (socket says 172.21.15.89:57848) 2024-04-02T08:53:29.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7faafe400700 1 -- 172.21.15.89:0/3662419225 learned_addr learned my addr 172.21.15.89:0/3662419225 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:29.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7faafda00700 1 --2- 172.21.15.89:0/3662419225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faaf80d16a0 0x7faaf814fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7faafd000700 1 --2- 172.21.15.89:0/3662419225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80d2540 0x7faaf815be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7faafe400700 1 -- 172.21.15.89:0/3662419225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faaf80d16a0 msgr2=0x7faaf814fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7faafe400700 1 --2- 172.21.15.89:0/3662419225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faaf80d16a0 0x7faaf814fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7faafe400700 1 -- 172.21.15.89:0/3662419225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80d2540 msgr2=0x7faaf815be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7faafe400700 1 --2- 172.21.15.89:0/3662419225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80d2540 0x7faaf815be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7faafe400700 1 -- 172.21.15.89:0/3662419225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf0009a70 con 0x7faaf815d9d0 2024-04-02T08:53:29.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.882+0000 7faafe400700 1 --2- 172.21.15.89:0/3662419225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf815d9d0 0x7faaf80b1480 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7faaf0014040 tx=0x7faaf0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:29.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.883+0000 7faaf5600700 1 -- 172.21.15.89:0/3662419225 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf001b8a0 con 0x7faaf815d9d0 2024-04-02T08:53:29.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.883+0000 7faaf5600700 1 -- 172.21.15.89:0/3662419225 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf0020070 con 0x7faaf815d9d0 2024-04-02T08:53:29.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.883+0000 7fab00400700 1 -- 172.21.15.89:0/3662419225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf80cb230 con 0x7faaf815d9d0 2024-04-02T08:53:29.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.883+0000 7faaf5600700 1 -- 172.21.15.89:0/3662419225 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf001bc60 con 0x7faaf815d9d0 2024-04-02T08:53:29.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.883+0000 7fab00400700 1 -- 172.21.15.89:0/3662419225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf81502e0 con 0x7faaf815d9d0 2024-04-02T08:53:29.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.885+0000 7faaf5600700 1 -- 172.21.15.89:0/3662419225 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7faaf002f440 con 0x7faaf815d9d0 2024-04-02T08:53:29.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.886+0000 7faaf5600700 1 --2- 172.21.15.89:0/3662419225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faad806adb0 0x7faad806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.886+0000 7faaf5600700 1 -- 172.21.15.89:0/3662419225 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7faaf008b300 con 0x7faaf815d9d0 2024-04-02T08:53:29.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.886+0000 7faafda00700 1 --2- 172.21.15.89:0/3662419225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faad806adb0 0x7faad806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.886+0000 7faafda00700 1 --2- 172.21.15.89:0/3662419225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faad806adb0 0x7faad806d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faaec006fd0 tx=0x7faaec008040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:29.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.888+0000 7fab13f0db80 1 Processor -- start 2024-04-02T08:53:29.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.888+0000 7fab13f0db80 1 -- start start 2024-04-02T08:53:29.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7fab13f0db80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3036830 0x5556a3036c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7fab13f0db80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3009be0 0x5556a300ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7fab13f0db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3033a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7fab13f0db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2f81650 con 0x5556a3007b60 2024-04-02T08:53:29.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7fab13f0db80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2f62c30 con 0x5556a3036830 2024-04-02T08:53:29.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7fab13f0db80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x5556a2f60430 con 0x5556a3009be0 2024-04-02T08:53:29.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7faafe400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3033a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7faafe400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3033a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57864/0 (socket says 172.21.15.89:57864) 2024-04-02T08:53:29.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7faafe400700 1 -- 172.21.15.89:0/3892753614 learned_addr learned my addr 172.21.15.89:0/3892753614 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:29.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7faafda00700 1 --2- 172.21.15.89:0/3892753614 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3036830 0x5556a3036c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7faafd000700 1 --2- 172.21.15.89:0/3892753614 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3009be0 0x5556a300ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7faafe400700 1 -- 172.21.15.89:0/3892753614 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3009be0 msgr2=0x5556a300ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:29.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7faafe400700 1 --2- 172.21.15.89:0/3892753614 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3009be0 0x5556a300ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7faafe400700 1 -- 172.21.15.89:0/3892753614 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3036830 msgr2=0x5556a3036c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:29.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7faafe400700 1 --2- 172.21.15.89:0/3892753614 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3036830 0x5556a3036c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.889+0000 7faafe400700 1 -- 172.21.15.89:0/3892753614 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556a2d4f790 con 0x5556a3007b60 2024-04-02T08:53:29.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.890+0000 7faafe400700 1 --2- 172.21.15.89:0/3892753614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3033a20 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7faaf000b370 tx=0x7faaf005a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7269d7fe3c2168ac server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:29.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.890+0000 7faaea000700 1 -- 172.21.15.89:0/3892753614 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0059ab0 con 0x5556a3007b60 2024-04-02T08:53:29.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.890+0000 7faaea000700 1 -- 172.21.15.89:0/3892753614 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf005dda0 con 0x5556a3007b60 2024-04-02T08:53:29.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.890+0000 7faaea000700 1 -- 172.21.15.89:0/3892753614 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0062360 con 0x5556a3007b60 2024-04-02T08:53:29.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.890+0000 7fab13f0db80 1 -- 172.21.15.89:0/3892753614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 msgr2=0x5556a3033a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.890+0000 7fab13f0db80 1 --2- 172.21.15.89:0/3892753614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3033a20 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7faaf000b370 tx=0x7faaf005a6f0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.890+0000 7fab13f0db80 1 -- 172.21.15.89:0/3892753614 shutdown_connections 2024-04-02T08:53:29.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.890+0000 7fab13f0db80 1 --2- 172.21.15.89:0/3892753614 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3036830 0x5556a3036c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.890+0000 7fab13f0db80 1 --2- 172.21.15.89:0/3892753614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3033a20 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.890+0000 7fab13f0db80 1 --2- 172.21.15.89:0/3892753614 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3009be0 0x5556a300ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.891+0000 7fab13f0db80 1 -- 172.21.15.89:0/3892753614 >> 172.21.15.89:0/3892753614 conn(0x5556a2f6a6a0 msgr2=0x5556a3051640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:29.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.891+0000 7fab13f0db80 1 -- 172.21.15.89:0/3892753614 shutdown_connections 2024-04-02T08:53:29.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.891+0000 7fab13f0db80 1 -- 172.21.15.89:0/3892753614 wait complete. 2024-04-02T08:53:29.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.891+0000 7fab13f0db80 1 -- start start 2024-04-02T08:53:29.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.892+0000 7fab13f0db80 1 Processor -- start 2024-04-02T08:53:29.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7fab13f0db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3055690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7fab13f0db80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 0x5556a3038740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7fab13f0db80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3036830 0x5556a3052ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7fab13f0db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2f5bd20 con 0x5556a3007b60 2024-04-02T08:53:29.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7fab13f0db80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2ef7770 con 0x5556a3009be0 2024-04-02T08:53:29.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7fab13f0db80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x5556a2f1a6f0 con 0x5556a3036830 2024-04-02T08:53:29.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7fab13f0db80 10 client.0 _ll_register_callbacks cb 0x5556a2f015e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T08:53:29.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7faafda00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3055690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7faafda00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3055690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57866/0 (socket says 172.21.15.89:57866) 2024-04-02T08:53:29.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7faafda00700 1 -- 172.21.15.89:0/2140135243 learned_addr learned my addr 172.21.15.89:0/2140135243 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:29.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7faafe400700 1 --2- 172.21.15.89:0/2140135243 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3036830 0x5556a3052ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7faafd000700 1 --2- 172.21.15.89:0/2140135243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 0x5556a3038740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7faafda00700 1 -- 172.21.15.89:0/2140135243 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3036830 msgr2=0x5556a3052ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:29.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7faafda00700 1 --2- 172.21.15.89:0/2140135243 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3036830 0x5556a3052ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7faafda00700 1 -- 172.21.15.89:0/2140135243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 msgr2=0x5556a3038740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:29.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7faafda00700 1 --2- 172.21.15.89:0/2140135243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 0x5556a3038740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7faafda00700 1 -- 172.21.15.89:0/2140135243 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf81502e0 con 0x5556a3007b60 2024-04-02T08:53:29.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7faafd000700 1 --2- 172.21.15.89:0/2140135243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 0x5556a3038740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:29.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7faafda00700 1 --2- 172.21.15.89:0/2140135243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3055690 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7faaec00ff90 tx=0x7faaec0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f1019d7980d2b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:29.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.893+0000 7faadea00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T08:53:29.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.894+0000 7faadea00700 1 -- 172.21.15.89:0/2140135243 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaec021620 con 0x5556a3007b60 2024-04-02T08:53:29.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.894+0000 7faadea00700 1 -- 172.21.15.89:0/2140135243 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7faaec021780 con 0x5556a3007b60 2024-04-02T08:53:29.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.894+0000 7fab13f0db80 1 -- 172.21.15.89:0/2140135243 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5556a2ffe920 con 0x5556a3007b60 2024-04-02T08:53:29.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.894+0000 7faadea00700 1 -- 172.21.15.89:0/2140135243 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7faaec021940 con 0x5556a3007b60 2024-04-02T08:53:29.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.894+0000 7fab13f0db80 10 client.10176 fetch_fsmap learned FSMap version 89 2024-04-02T08:53:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.894+0000 7fab13f0db80 1 -- 172.21.15.89:0/2140135243 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) v3 -- 0x5556a2e3c0a0 con 0x5556a3007b60 2024-04-02T08:53:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.894+0000 7faadea00700 1 -- 172.21.15.89:0/2140135243 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5556a2ffe920 con 0x5556a3007b60 2024-04-02T08:53:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.894+0000 7faadea00700 1 -- 172.21.15.89:0/2140135243 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 89) v1 ==== 3114+0+0 (secure 0 0 0) 0x7faaec021aa0 con 0x5556a3007b60 2024-04-02T08:53:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.894+0000 7fab13f0db80 10 client.10176 fetch_fsmap finished waiting for FSMap version 89 2024-04-02T08:53:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.894+0000 7fab13f0db80 10 client.10176 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T08:53:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.894+0000 7fab13f0db80 20 client.10176 populate_metadata read hostname 'smithi089' 2024-04-02T08:53:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.894+0000 7fab13f0db80 1 --2- 172.21.15.89:0/2140135243 >> [v2:172.21.15.89:6834/583021497,v1:172.21.15.89:6836/583021497] conn(0x5556a30399c0 0x5556a306e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.894+0000 7fab13f0db80 4 client.10176 mds_command: new command op to 10038 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T08:53:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.894+0000 7fab13f0db80 1 -- 172.21.15.89:0/2140135243 --> [v2:172.21.15.89:6834/583021497,v1:172.21.15.89:6836/583021497] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5556a2fdb6f0 con 0x5556a30399c0 2024-04-02T08:53:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.895+0000 7faafd000700 1 --2- 172.21.15.89:0/2140135243 >> [v2:172.21.15.89:6834/583021497,v1:172.21.15.89:6836/583021497] conn(0x5556a30399c0 0x5556a306e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.895+0000 7faafd000700 1 --2- 172.21.15.89:0/2140135243 >> [v2:172.21.15.89:6834/583021497,v1:172.21.15.89:6836/583021497] conn(0x5556a30399c0 0x5556a306e230 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9e6fe7bb10273d9 server_cookie=63dbb5264b363c1 in_seq=0 out_seq=0 2024-04-02T08:53:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.895+0000 7faadea00700 10 client.10176 ms_handle_connect on v2:172.21.15.89:6834/583021497 2024-04-02T08:53:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7faadea00700 1 -- 172.21.15.89:0/2140135243 <== mds.0 v2:172.21.15.89:6834/583021497 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5556a2fdb6f0 con 0x5556a30399c0 2024-04-02T08:53:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7faadea00700 10 client.10176 handle_command_reply: tid=0 2024-04-02T08:53:29.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7fab13f0db80 1 client.10176 shutdown 2024-04-02T08:53:29.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7fab13f0db80 1 -- 172.21.15.89:0/2140135243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 msgr2=0x5556a3055690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7fab13f0db80 1 --2- 172.21.15.89:0/2140135243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3055690 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7faaec00ff90 tx=0x7faaec0047c0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7fab13f0db80 1 -- 172.21.15.89:0/2140135243 shutdown_connections 2024-04-02T08:53:29.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7fab13f0db80 1 --2- 172.21.15.89:0/2140135243 >> [v2:172.21.15.89:6834/583021497,v1:172.21.15.89:6836/583021497] conn(0x5556a30399c0 0x5556a306e230 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7fab13f0db80 1 --2- 172.21.15.89:0/2140135243 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3036830 0x5556a3052ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7fab13f0db80 1 --2- 172.21.15.89:0/2140135243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3055690 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7faadea00700 0 client.10176 ms_handle_reset on v2:172.21.15.89:6834/583021497 2024-04-02T08:53:29.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7fab13f0db80 1 --2- 172.21.15.89:0/2140135243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 0x5556a3038740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7fab13f0db80 1 -- 172.21.15.89:0/2140135243 >> 172.21.15.89:0/2140135243 conn(0x5556a2f6a6a0 msgr2=0x5556a300bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:29.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7fab13f0db80 1 -- 172.21.15.89:0/2140135243 shutdown_connections 2024-04-02T08:53:29.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7fab13f0db80 1 -- 172.21.15.89:0/2140135243 wait complete. 2024-04-02T08:53:29.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.896+0000 7fab13f0db80 20 client.10176 trim_cache size 0 max 16384 2024-04-02T08:53:29.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.914+0000 7fab13f0db80 1 Processor -- start 2024-04-02T08:53:29.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.914+0000 7fab13f0db80 1 -- start start 2024-04-02T08:53:29.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.915+0000 7fab13f0db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3038500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.915+0000 7fab13f0db80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 0x5556a30055a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.915+0000 7fab13f0db80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3036830 0x5556a3005ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.915+0000 7fab13f0db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a2d50390 con 0x5556a3007b60 2024-04-02T08:53:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.915+0000 7fab13f0db80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a3053850 con 0x5556a3009be0 2024-04-02T08:53:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.915+0000 7fab13f0db80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x5556a300cc40 con 0x5556a3036830 2024-04-02T08:53:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.915+0000 7faafda00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3038500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.915+0000 7faafda00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3038500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57874/0 (socket says 172.21.15.89:57874) 2024-04-02T08:53:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.915+0000 7faafda00700 1 -- 172.21.15.89:0/3304383507 learned_addr learned my addr 172.21.15.89:0/3304383507 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.916+0000 7faafe400700 1 --2- 172.21.15.89:0/3304383507 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3036830 0x5556a3005ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.916+0000 7faafd000700 1 --2- 172.21.15.89:0/3304383507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 0x5556a30055a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.916+0000 7faafda00700 1 -- 172.21.15.89:0/3304383507 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3036830 msgr2=0x5556a3005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.916+0000 7faafda00700 1 --2- 172.21.15.89:0/3304383507 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3036830 0x5556a3005ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.916+0000 7faafda00700 1 -- 172.21.15.89:0/3304383507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 msgr2=0x5556a30055a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:29.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.916+0000 7faafda00700 1 --2- 172.21.15.89:0/3304383507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 0x5556a30055a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.916+0000 7faafda00700 1 -- 172.21.15.89:0/3304383507 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556a2f22010 con 0x5556a3007b60 2024-04-02T08:53:29.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.916+0000 7faafe400700 1 --2- 172.21.15.89:0/3304383507 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3036830 0x5556a3005ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:53:29.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.916+0000 7faafd000700 1 --2- 172.21.15.89:0/3304383507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 0x5556a30055a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T08:53:29.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.916+0000 7faafda00700 1 --2- 172.21.15.89:0/3304383507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3038500 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7faaec00b010 tx=0x7faaec020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c455cfccac0032 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:29.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.916+0000 7faaeaa00700 1 -- 172.21.15.89:0/3304383507 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaec02a5f0 con 0x5556a3007b60 2024-04-02T08:53:29.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.916+0000 7faaeaa00700 1 -- 172.21.15.89:0/3304383507 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaec02a750 con 0x5556a3007b60 2024-04-02T08:53:29.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.916+0000 7faaeaa00700 1 -- 172.21.15.89:0/3304383507 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaec021420 con 0x5556a3007b60 2024-04-02T08:53:29.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.917+0000 7fab13f0db80 1 -- 172.21.15.89:0/3304383507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 msgr2=0x5556a3038500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.917+0000 7fab13f0db80 1 --2- 172.21.15.89:0/3304383507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3038500 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7faaec00b010 tx=0x7faaec020000 comp rx=0 tx=0).stop 2024-04-02T08:53:29.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.917+0000 7fab13f0db80 1 -- 172.21.15.89:0/3304383507 shutdown_connections 2024-04-02T08:53:29.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.917+0000 7fab13f0db80 1 --2- 172.21.15.89:0/3304383507 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3036830 0x5556a3005ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.917+0000 7fab13f0db80 1 --2- 172.21.15.89:0/3304383507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3007b60 0x5556a3038500 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.917+0000 7fab13f0db80 1 --2- 172.21.15.89:0/3304383507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 0x5556a30055a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.917+0000 7fab13f0db80 1 -- 172.21.15.89:0/3304383507 >> 172.21.15.89:0/3304383507 conn(0x5556a2f6a6a0 msgr2=0x5556a3054ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.917+0000 7fab13f0db80 1 -- 172.21.15.89:0/3304383507 shutdown_connections 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.917+0000 7fab13f0db80 1 -- 172.21.15.89:0/3304383507 wait complete. 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.917+0000 7fab13f0db80 1 -- start start 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.918+0000 7fab13f0db80 1 Processor -- start 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.918+0000 7fab13f0db80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3007b60 0x5556a30520d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7fab13f0db80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 0x5556a3047fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7fab13f0db80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3036830 0x5556a3038740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7fab13f0db80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a3048520 con 0x5556a3036830 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7fab13f0db80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a300a590 con 0x5556a3009be0 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7fab13f0db80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x5556a3035900 con 0x5556a3007b60 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7fab13f0db80 10 client.0 _ll_register_callbacks cb 0x7faaec00fb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T08:53:29.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7faafe400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3036830 0x5556a3038740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7faafe400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3036830 0x5556a3038740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57888/0 (socket says 172.21.15.89:57888) 2024-04-02T08:53:29.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7faafe400700 1 -- 172.21.15.89:0/4094468812 learned_addr learned my addr 172.21.15.89:0/4094468812 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:29.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7faafda00700 1 --2- 172.21.15.89:0/4094468812 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3007b60 0x5556a30520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7faafd000700 1 --2- 172.21.15.89:0/4094468812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 0x5556a3047fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7faafe400700 1 -- 172.21.15.89:0/4094468812 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3007b60 msgr2=0x5556a30520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:29.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7faafe400700 1 --2- 172.21.15.89:0/4094468812 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3007b60 0x5556a30520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7faafe400700 1 -- 172.21.15.89:0/4094468812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 msgr2=0x5556a3047fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:29.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7faafe400700 1 --2- 172.21.15.89:0/4094468812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 0x5556a3047fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.919+0000 7faafe400700 1 -- 172.21.15.89:0/4094468812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556a2e3c0a0 con 0x5556a3036830 2024-04-02T08:53:29.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7faafe400700 1 --2- 172.21.15.89:0/4094468812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3036830 0x5556a3038740 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7faaf000b370 tx=0x7faaf008b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b4e288de37ef66 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:29.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7faadea00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T08:53:29.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7faadea00700 1 -- 172.21.15.89:0/4094468812 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf00892f0 con 0x5556a3036830 2024-04-02T08:53:29.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7faadea00700 1 -- 172.21.15.89:0/4094468812 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7faaf0062360 con 0x5556a3036830 2024-04-02T08:53:29.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7fab13f0db80 1 -- 172.21.15.89:0/4094468812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faaf80b8020 con 0x5556a3036830 2024-04-02T08:53:29.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7faadea00700 1 -- 172.21.15.89:0/4094468812 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7faaf0058310 con 0x5556a3036830 2024-04-02T08:53:29.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7fab13f0db80 10 client.10182 fetch_fsmap learned FSMap version 89 2024-04-02T08:53:29.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7fab13f0db80 1 -- 172.21.15.89:0/4094468812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) v3 -- 0x5556a2db01d0 con 0x5556a3036830 2024-04-02T08:53:29.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7faadea00700 1 -- 172.21.15.89:0/4094468812 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf007c020 con 0x5556a3036830 2024-04-02T08:53:29.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7faadea00700 1 -- 172.21.15.89:0/4094468812 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 89) v1 ==== 3114+0+0 (secure 0 0 0) 0x7faaf005a1d0 con 0x5556a3036830 2024-04-02T08:53:29.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7fab13f0db80 10 client.10182 fetch_fsmap finished waiting for FSMap version 89 2024-04-02T08:53:29.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7fab13f0db80 10 client.10182 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T08:53:29.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7fab13f0db80 20 client.10182 populate_metadata read hostname 'smithi089' 2024-04-02T08:53:29.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7fab13f0db80 1 --2- 172.21.15.89:0/4094468812 >> [v2:172.21.15.89:6834/583021497,v1:172.21.15.89:6836/583021497] conn(0x5556a300a150 0x5556a304f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:29.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7fab13f0db80 4 client.10182 mds_command: new command op to 10038 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T08:53:29.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7fab13f0db80 1 -- 172.21.15.89:0/4094468812 --> [v2:172.21.15.89:6834/583021497,v1:172.21.15.89:6836/583021497] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7faaf005dda0 con 0x5556a300a150 2024-04-02T08:53:29.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.920+0000 7faafda00700 1 --2- 172.21.15.89:0/4094468812 >> [v2:172.21.15.89:6834/583021497,v1:172.21.15.89:6836/583021497] conn(0x5556a300a150 0x5556a304f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:29.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.921+0000 7faadea00700 10 client.10182 ms_handle_connect on v2:172.21.15.89:6834/583021497 2024-04-02T08:53:29.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.921+0000 7faafda00700 1 --2- 172.21.15.89:0/4094468812 >> [v2:172.21.15.89:6834/583021497,v1:172.21.15.89:6836/583021497] conn(0x5556a300a150 0x5556a304f1f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dc1d1a2284edbc7 server_cookie=a0dff8169cb3ec36 in_seq=0 out_seq=0 2024-04-02T08:53:29.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.921+0000 7faadea00700 1 -- 172.21.15.89:0/4094468812 <== mds.0 v2:172.21.15.89:6834/583021497 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x7faaf005dda0 con 0x5556a300a150 2024-04-02T08:53:29.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.921+0000 7faadea00700 10 client.10182 handle_command_reply: tid=0 2024-04-02T08:53:29.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.921+0000 7fab13f0db80 1 client.10182 shutdown 2024-04-02T08:53:29.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.921+0000 7fab13f0db80 1 -- 172.21.15.89:0/4094468812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3036830 msgr2=0x5556a3038740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.921+0000 7fab13f0db80 1 --2- 172.21.15.89:0/4094468812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3036830 0x5556a3038740 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7faaf000b370 tx=0x7faaf008b040 comp rx=0 tx=0).stop 2024-04-02T08:53:29.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.921+0000 7fab13f0db80 1 -- 172.21.15.89:0/4094468812 shutdown_connections 2024-04-02T08:53:29.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.921+0000 7fab13f0db80 1 --2- 172.21.15.89:0/4094468812 >> [v2:172.21.15.89:6834/583021497,v1:172.21.15.89:6836/583021497] conn(0x5556a300a150 0x5556a304f1f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.921+0000 7fab13f0db80 1 --2- 172.21.15.89:0/4094468812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556a3036830 0x5556a3038740 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.921+0000 7fab13f0db80 1 --2- 172.21.15.89:0/4094468812 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5556a3007b60 0x5556a30520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.921+0000 7faadea00700 0 client.10182 ms_handle_reset on v2:172.21.15.89:6834/583021497 2024-04-02T08:53:29.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.921+0000 7fab13f0db80 1 --2- 172.21.15.89:0/4094468812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a3009be0 0x5556a3047fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.921+0000 7fab13f0db80 1 -- 172.21.15.89:0/4094468812 >> 172.21.15.89:0/4094468812 conn(0x5556a2f6a6a0 msgr2=0x5556a3058b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:29.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.922+0000 7fab13f0db80 1 -- 172.21.15.89:0/4094468812 shutdown_connections 2024-04-02T08:53:29.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.922+0000 7fab13f0db80 1 -- 172.21.15.89:0/4094468812 wait complete. 2024-04-02T08:53:29.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.922+0000 7fab13f0db80 20 client.10182 trim_cache size 0 max 16384 2024-04-02T08:53:29.925 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T08:53:29.925 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T08:53:29.925 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T08:53:29.925 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 10038, 2024-04-02T08:53:29.925 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T08:53:29.925 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T08:53:29.925 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T08:53:29.925 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 7.0377415230000002, 2024-04-02T08:53:29.925 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 89, 2024-04-02T08:53:29.926 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 186, 2024-04-02T08:53:29.926 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 186, 2024-04-02T08:53:29.926 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 12.502839559 2024-04-02T08:53:29.926 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T08:53:29.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.922+0000 7fab00400700 1 -- 172.21.15.89:0/3662419225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faad806adb0 msgr2=0x7faad806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.922+0000 7fab00400700 1 --2- 172.21.15.89:0/3662419225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faad806adb0 0x7faad806d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faaec006fd0 tx=0x7faaec008040 comp rx=0 tx=0).stop 2024-04-02T08:53:29.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.922+0000 7fab00400700 1 -- 172.21.15.89:0/3662419225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf815d9d0 msgr2=0x7faaf80b1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:29.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.922+0000 7fab00400700 1 --2- 172.21.15.89:0/3662419225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf815d9d0 0x7faaf80b1480 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7faaf0014040 tx=0x7faaf0004ad0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.922+0000 7fab00400700 1 -- 172.21.15.89:0/3662419225 shutdown_connections 2024-04-02T08:53:29.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.922+0000 7fab00400700 1 --2- 172.21.15.89:0/3662419225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faad806adb0 0x7faad806d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.922+0000 7fab00400700 1 --2- 172.21.15.89:0/3662419225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf815d9d0 0x7faaf80b1480 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.922+0000 7fab00400700 1 --2- 172.21.15.89:0/3662419225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faaf80d16a0 0x7faaf814fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.922+0000 7fab00400700 1 --2- 172.21.15.89:0/3662419225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80d2540 0x7faaf815be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:29.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.922+0000 7fab00400700 1 -- 172.21.15.89:0/3662419225 >> 172.21.15.89:0/3662419225 conn(0x7faaf808aff0 msgr2=0x7faaf80b0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:29.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.923+0000 7fab00400700 1 -- 172.21.15.89:0/3662419225 shutdown_connections 2024-04-02T08:53:29.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:29.923+0000 7fab00400700 1 -- 172.21.15.89:0/3662419225 wait complete. 2024-04-02T08:53:29.939 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T08:53:30.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.131+0000 7f431b000700 1 Processor -- start 2024-04-02T08:53:30.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.132+0000 7f431b000700 1 -- start start 2024-04-02T08:53:30.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.133+0000 7f431b000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43140ce590 0x7f43140ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:30.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.133+0000 7f431b000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f43140b6a20 0x7f43140b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:30.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.133+0000 7f431b000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43140ceef0 con 0x7f4314160be0 2024-04-02T08:53:30.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.133+0000 7f431b000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43140b65d0 con 0x7f43140b6a20 2024-04-02T08:53:30.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.133+0000 7f431b000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43140d2a80 con 0x7f43140ce590 2024-04-02T08:53:30.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.133+0000 7f4319000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f43140b6a20 0x7f43140b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:30.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.133+0000 7f4319000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f43140b6a20 0x7f43140b6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57900/0 (socket says 172.21.15.89:57900) 2024-04-02T08:53:30.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.133+0000 7f4319000700 1 -- 172.21.15.89:0/2794307383 learned_addr learned my addr 172.21.15.89:0/2794307383 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:30.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.133+0000 7f430be00700 1 --2- 172.21.15.89:0/2794307383 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43140ce590 0x7f43140ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:30.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.134+0000 7f4319000700 1 -- 172.21.15.89:0/2794307383 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43140ce590 msgr2=0x7f43140ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:30.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.134+0000 7f4319000700 1 --2- 172.21.15.89:0/2794307383 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43140ce590 0x7f43140ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:30.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.134+0000 7f4319000700 1 -- 172.21.15.89:0/2794307383 >> v1:172.21.15.113:6789/0 conn(0x7f4314160be0 legacy=0x7f43140cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:30.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.134+0000 7f4319000700 1 -- 172.21.15.89:0/2794307383 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4314160fc0 con 0x7f43140b6a20 2024-04-02T08:53:30.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.134+0000 7f4313400700 1 -- 172.21.15.89:0/2794307383 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793947107 0 0) 0x7f43140ceef0 con 0x7f4314160be0 2024-04-02T08:53:30.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.134+0000 7f4319000700 1 --2- 172.21.15.89:0/2794307383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43140b6a20 0x7f43140b6eb0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f430c009e50 tx=0x7f430c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d88d5b18f019ea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:30.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.134+0000 7f4313400700 1 -- 172.21.15.89:0/2794307383 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430c00e070 con 0x7f43140b6a20 2024-04-02T08:53:30.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.134+0000 7f4313400700 1 -- 172.21.15.89:0/2794307383 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430c00b910 con 0x7f43140b6a20 2024-04-02T08:53:30.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.134+0000 7f4313400700 1 -- 172.21.15.89:0/2794307383 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430c00bcc0 con 0x7f43140b6a20 2024-04-02T08:53:30.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.135+0000 7f431b000700 1 -- 172.21.15.89:0/2794307383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43140b6a20 msgr2=0x7f43140b6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:30.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.135+0000 7f431b000700 1 --2- 172.21.15.89:0/2794307383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43140b6a20 0x7f43140b6eb0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f430c009e50 tx=0x7f430c00b330 comp rx=0 tx=0).stop 2024-04-02T08:53:30.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.135+0000 7f431b000700 1 -- 172.21.15.89:0/2794307383 shutdown_connections 2024-04-02T08:53:30.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.135+0000 7f431b000700 1 --2- 172.21.15.89:0/2794307383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43140b6a20 0x7f43140b6eb0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:30.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.135+0000 7f431b000700 1 --2- 172.21.15.89:0/2794307383 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43140ce590 0x7f43140ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:30.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.135+0000 7f431b000700 1 -- 172.21.15.89:0/2794307383 >> 172.21.15.89:0/2794307383 conn(0x7f431408aff0 msgr2=0x7f43140d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:30.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.135+0000 7f431b000700 1 -- 172.21.15.89:0/2794307383 shutdown_connections 2024-04-02T08:53:30.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.135+0000 7f431b000700 1 -- 172.21.15.89:0/2794307383 wait complete. 2024-04-02T08:53:30.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.137+0000 7f431b000700 1 Processor -- start 2024-04-02T08:53:30.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.137+0000 7f431b000700 1 -- start start 2024-04-02T08:53:30.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f431b000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43140b6a20 0x7f431414ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:30.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f431b000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43140ce590 0x7f431415d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:30.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f431b000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4314160be0 0x7f431415d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:30.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f431b000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431409e1d0 con 0x7f4314160be0 2024-04-02T08:53:30.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f431b000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43140ca620 con 0x7f43140ce590 2024-04-02T08:53:30.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f431b000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43140cb480 con 0x7f43140b6a20 2024-04-02T08:53:30.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f4319000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4314160be0 0x7f431415d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:30.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f4319000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4314160be0 0x7f431415d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57910/0 (socket says 172.21.15.89:57910) 2024-04-02T08:53:30.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f4319000700 1 -- 172.21.15.89:0/2479136423 learned_addr learned my addr 172.21.15.89:0/2479136423 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:30.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f4313e00700 1 --2- 172.21.15.89:0/2479136423 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43140b6a20 0x7f431414ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:30.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f430be00700 1 --2- 172.21.15.89:0/2479136423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43140ce590 0x7f431415d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:30.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f4319000700 1 -- 172.21.15.89:0/2479136423 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43140b6a20 msgr2=0x7f431414ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:30.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f4319000700 1 --2- 172.21.15.89:0/2479136423 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43140b6a20 0x7f431414ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:30.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f4319000700 1 -- 172.21.15.89:0/2479136423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43140ce590 msgr2=0x7f431415d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:30.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f4319000700 1 --2- 172.21.15.89:0/2479136423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43140ce590 0x7f431415d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:30.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.138+0000 7f4319000700 1 -- 172.21.15.89:0/2479136423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c009b30 con 0x7f4314160be0 2024-04-02T08:53:30.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.139+0000 7f4319000700 1 --2- 172.21.15.89:0/2479136423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4314160be0 0x7f431415d9b0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f430c00b680 tx=0x7f430c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:30.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.139+0000 7f4310c00700 1 -- 172.21.15.89:0/2479136423 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430c0099b0 con 0x7f4314160be0 2024-04-02T08:53:30.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.139+0000 7f4310c00700 1 -- 172.21.15.89:0/2479136423 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430c0092e0 con 0x7f4314160be0 2024-04-02T08:53:30.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.139+0000 7f431b000700 1 -- 172.21.15.89:0/2479136423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431414b0c0 con 0x7f4314160be0 2024-04-02T08:53:30.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.139+0000 7f4310c00700 1 -- 172.21.15.89:0/2479136423 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430c01b650 con 0x7f4314160be0 2024-04-02T08:53:30.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.139+0000 7f431b000700 1 -- 172.21.15.89:0/2479136423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43140cab50 con 0x7f4314160be0 2024-04-02T08:53:30.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.140+0000 7f431b000700 1 -- 172.21.15.89:0/2479136423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4314067720 con 0x7f4314160be0 2024-04-02T08:53:30.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.141+0000 7f4310c00700 1 -- 172.21.15.89:0/2479136423 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f430c01b7b0 con 0x7f4314160be0 2024-04-02T08:53:30.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.142+0000 7f4310c00700 1 --2- 172.21.15.89:0/2479136423 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f42f006adb0 0x7f42f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:30.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.142+0000 7f4310c00700 1 -- 172.21.15.89:0/2479136423 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f430c08ad70 con 0x7f4314160be0 2024-04-02T08:53:30.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.142+0000 7f4313e00700 1 --2- 172.21.15.89:0/2479136423 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f42f006adb0 0x7f42f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:30.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.143+0000 7f4313e00700 1 --2- 172.21.15.89:0/2479136423 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f42f006adb0 0x7f42f006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4304000f60 tx=0x7f4304009040 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:30.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.146+0000 7f4310c00700 1 -- 172.21.15.89:0/2479136423 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f430c055b70 con 0x7f4314160be0 2024-04-02T08:53:30.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.392+0000 7f431b000700 1 -- 172.21.15.89:0/2479136423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f43140cb5c0 con 0x7f4314160be0 2024-04-02T08:53:30.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.392+0000 7f4310c00700 1 -- 172.21.15.89:0/2479136423 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v89) v1 ==== 53+0+73 (secure 0 0 0) 0x7f430c01f060 con 0x7f4314160be0 2024-04-02T08:53:30.391 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T08:53:30.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.395+0000 7f431b000700 1 -- 172.21.15.89:0/2479136423 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f42f006adb0 msgr2=0x7f42f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:30.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.395+0000 7f431b000700 1 --2- 172.21.15.89:0/2479136423 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f42f006adb0 0x7f42f006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4304000f60 tx=0x7f4304009040 comp rx=0 tx=0).stop 2024-04-02T08:53:30.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.395+0000 7f431b000700 1 -- 172.21.15.89:0/2479136423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4314160be0 msgr2=0x7f431415d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:30.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.395+0000 7f431b000700 1 --2- 172.21.15.89:0/2479136423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4314160be0 0x7f431415d9b0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f430c00b680 tx=0x7f430c004ad0 comp rx=0 tx=0).stop 2024-04-02T08:53:30.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.395+0000 7f431b000700 1 -- 172.21.15.89:0/2479136423 shutdown_connections 2024-04-02T08:53:30.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.395+0000 7f431b000700 1 --2- 172.21.15.89:0/2479136423 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f42f006adb0 0x7f42f006d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:30.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.395+0000 7f431b000700 1 --2- 172.21.15.89:0/2479136423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4314160be0 0x7f431415d9b0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:30.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.395+0000 7f431b000700 1 --2- 172.21.15.89:0/2479136423 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43140b6a20 0x7f431414ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:30.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.395+0000 7f431b000700 1 --2- 172.21.15.89:0/2479136423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43140ce590 0x7f431415d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:30.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.395+0000 7f431b000700 1 -- 172.21.15.89:0/2479136423 >> 172.21.15.89:0/2479136423 conn(0x7f431408aff0 msgr2=0x7f43140d0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:30.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.395+0000 7f431b000700 1 -- 172.21.15.89:0/2479136423 shutdown_connections 2024-04-02T08:53:30.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:30.395+0000 7f431b000700 1 -- 172.21.15.89:0/2479136423 wait complete. 2024-04-02T08:53:30.413 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T08:53:30.413 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T08:53:30.413 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T08:53:30.413 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T08:53:30.413 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T08:53:30.413 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T08:53:30.414 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T08:53:30.414 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T08:53:30.414 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T08:53:30.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:53:30.414 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T08:53:30.459 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T08:53:30.459 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T08:53:30.459 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T08:53:30.459 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T08:53:30.459 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T08:53:30.460 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T08:53:30.460 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T08:53:30.460 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T08:53:30.460 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T08:53:30.460 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T08:53:30.460 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T08:53:30.460 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T08:53:30.460 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T08:53:30.460 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T08:53:30.460 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T08:53:30.460 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T08:53:30.460 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T08:53:30.460 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 37818sec preferred_lft 37818sec 2024-04-02T08:53:30.460 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T08:53:30.461 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T08:53:30.461 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T08:53:30.461 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T08:53:30.461 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T08:53:30.461 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T08:53:30.461 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T08:53:30.461 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T08:53:30.461 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T08:53:30.461 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T08:53:30.461 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T08:53:30.461 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T08:53:30.461 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T08:53:30.461 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T08:53:30.462 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T08:53:30.462 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T08:53:30.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:53:30.462 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T08:53:30.563 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T08:53:30.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:53:30.567 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T08:53:30.613 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T08:53:30.613 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T08:53:30.613 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T08:53:30.613 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T08:53:30.613 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T08:53:30.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:53:30.614 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T08:53:30.615 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T08:53:30.615 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T08:53:30.615 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T08:53:30.615 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T08:53:30.615 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T08:53:30.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:53:30.777 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T08:53:30.822 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T08:53:30.822 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T08:53:30.822 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T08:53:30.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:53:30.823 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T08:53:30.932 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T08:53:30.932 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T08:53:30.976 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T08:53:30.976 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T08:53:30.976 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T08:53:31.060 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T08:53:31.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:53:31.061 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T08:53:31.200 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T08:53:31.201 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T08:53:31.226 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T08:53:31.227 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T08:53:31.227 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T08:53:31.227 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T08:53:31.227 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T08:53:31.232 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T08:53:31.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:53:31.232 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T08:53:31.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:53:31.299 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T08:53:31.397 INFO:teuthology.orchestra.run.smithi089.stdout:client10191 2024-04-02T08:53:31.398 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T08:53:31.398 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T08:53:31.442 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T08:53:31.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.677+0000 7f5612000700 1 Processor -- start 2024-04-02T08:53:31.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.677+0000 7f5612000700 1 -- start start 2024-04-02T08:53:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.678+0000 7f5612000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f560c15d990 0x7f560c0d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.679+0000 7f5612000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f560c09a750 0x7f560c06c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.679+0000 7f5612000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c0d0970 con 0x7f560c099a60 2024-04-02T08:53:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.679+0000 7f5612000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f560c06c020 con 0x7f560c15d990 2024-04-02T08:53:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.679+0000 7f5612000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560c0c81a0 con 0x7f560c09a750 2024-04-02T08:53:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.679+0000 7f560b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f560c099a60 0x7f560c099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57392/0 (socket says 172.21.15.89:57392) 2024-04-02T08:53:31.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.679+0000 7f560b400700 1 -- 172.21.15.89:0/1340740111 learned_addr learned my addr 172.21.15.89:0/1340740111 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:31.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.679+0000 7f560be00700 1 --2- 172.21.15.89:0/1340740111 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f560c09a750 0x7f560c06c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:31.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.679+0000 7f560aa00700 1 --2- 172.21.15.89:0/1340740111 >> v2:172.21.15.113:3300/0 conn(0x7f560c15d990 0x7f560c0d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:31.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.679+0000 7f560a000700 1 -- 172.21.15.89:0/1340740111 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608380808 0 0) 0x7f560c0d0970 con 0x7f560c099a60 2024-04-02T08:53:31.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.679+0000 7f560a000700 1 -- 172.21.15.89:0/1340740111 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55f00035b0 con 0x7f560c099a60 2024-04-02T08:53:31.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560a000700 1 -- 172.21.15.89:0/1340740111 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3591354339 0 0) 0x7f55f00035b0 con 0x7f560c099a60 2024-04-02T08:53:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560a000700 1 -- 172.21.15.89:0/1340740111 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f560c09a750 msgr2=0x7f560c06c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560a000700 1 --2- 172.21.15.89:0/1340740111 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f560c09a750 0x7f560c06c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560a000700 1 -- 172.21.15.89:0/1340740111 >> v2:172.21.15.113:3300/0 conn(0x7f560c15d990 msgr2=0x7f560c0d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560a000700 1 --2- 172.21.15.89:0/1340740111 >> v2:172.21.15.113:3300/0 conn(0x7f560c15d990 0x7f560c0d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560a000700 1 -- 172.21.15.89:0/1340740111 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c0d0690 con 0x7f560c099a60 2024-04-02T08:53:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560a000700 1 -- 172.21.15.89:0/1340740111 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f55fc001670 con 0x7f560c099a60 2024-04-02T08:53:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560a000700 1 -- 172.21.15.89:0/1340740111 >> v1:172.21.15.89:6789/0 conn(0x7f560c099a60 legacy=0x7f560c099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560a000700 1 --2- 172.21.15.89:0/1340740111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f0003a00 0x7f55f0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560a000700 1 --2- 172.21.15.89:0/1340740111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55f0004850 0x7f55f0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560a000700 1 --2- 172.21.15.89:0/1340740111 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f560c09a750 0x7f55f0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560a000700 1 -- 172.21.15.89:0/1340740111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c06c020 con 0x7f55f0004850 2024-04-02T08:53:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560a000700 1 -- 172.21.15.89:0/1340740111 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c0c81a0 con 0x7f55f0003a00 2024-04-02T08:53:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560a000700 1 -- 172.21.15.89:0/1340740111 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f000baf0 con 0x7f560c09a750 2024-04-02T08:53:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560aa00700 1 --2- 172.21.15.89:0/1340740111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55f0004850 0x7f55f0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560b400700 1 --2- 172.21.15.89:0/1340740111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f0003a00 0x7f55f0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560be00700 1 --2- 172.21.15.89:0/1340740111 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f560c09a750 0x7f55f0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560aa00700 1 -- 172.21.15.89:0/1340740111 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f560c09a750 msgr2=0x7f55f0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560aa00700 1 --2- 172.21.15.89:0/1340740111 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f560c09a750 0x7f55f0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560aa00700 1 -- 172.21.15.89:0/1340740111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f0003a00 msgr2=0x7f55f0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560aa00700 1 --2- 172.21.15.89:0/1340740111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f0003a00 0x7f55f0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.680+0000 7f560aa00700 1 -- 172.21.15.89:0/1340740111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f00035b0 con 0x7f55f0004850 2024-04-02T08:53:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.681+0000 7f560aa00700 1 --2- 172.21.15.89:0/1340740111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55f0004850 0x7f55f0004cd0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009b30 tx=0x7f55f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3c0486ffaff41 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.681+0000 7f560a000700 1 -- 172.21.15.89:0/1340740111 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f800e6c0 con 0x7f55f0004850 2024-04-02T08:53:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.681+0000 7f560a000700 1 -- 172.21.15.89:0/1340740111 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f800e820 con 0x7f55f0004850 2024-04-02T08:53:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.681+0000 7f560a000700 1 -- 172.21.15.89:0/1340740111 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f800eb90 con 0x7f55f0004850 2024-04-02T08:53:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.681+0000 7f5612000700 1 -- 172.21.15.89:0/1340740111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55f0004850 msgr2=0x7f55f0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.681+0000 7f5612000700 1 --2- 172.21.15.89:0/1340740111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55f0004850 0x7f55f0004cd0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009b30 tx=0x7f55f0005880 comp rx=0 tx=0).stop 2024-04-02T08:53:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.681+0000 7f560be00700 1 -- 172.21.15.89:0/1340740111 reap_dead start 2024-04-02T08:53:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.682+0000 7f5612000700 1 -- 172.21.15.89:0/1340740111 shutdown_connections 2024-04-02T08:53:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.682+0000 7f5612000700 1 -- 172.21.15.89:0/1340740111 >> 172.21.15.89:0/1340740111 conn(0x7f560c0b2e20 msgr2=0x7f560c0b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.682+0000 7f5612000700 1 -- 172.21.15.89:0/1340740111 shutdown_connections 2024-04-02T08:53:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.682+0000 7f5612000700 1 -- 172.21.15.89:0/1340740111 wait complete. 2024-04-02T08:53:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.684+0000 7f5612000700 1 Processor -- start 2024-04-02T08:53:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.684+0000 7f5612000700 1 -- start start 2024-04-02T08:53:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.685+0000 7f5612000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f0004850 0x7f560c0cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.685+0000 7f5612000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f560c1605f0 0x7f560c15dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.685+0000 7f5612000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f560c15e2e0 0x7f560c15e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.685+0000 7f5612000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c0c87c0 con 0x7f560c1605f0 2024-04-02T08:53:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.685+0000 7f5612000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c0c8aa0 con 0x7f55f0004850 2024-04-02T08:53:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.685+0000 7f5612000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560c0c8d80 con 0x7f560c15e2e0 2024-04-02T08:53:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f560aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f560c1605f0 0x7f560c15dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f560aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f560c1605f0 0x7f560c15dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57922/0 (socket says 172.21.15.89:57922) 2024-04-02T08:53:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f560aa00700 1 -- 172.21.15.89:0/39358394 learned_addr learned my addr 172.21.15.89:0/39358394 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f560b400700 1 --2- 172.21.15.89:0/39358394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f0004850 0x7f560c0cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f560be00700 1 --2- 172.21.15.89:0/39358394 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f560c15e2e0 0x7f560c15e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f560aa00700 1 -- 172.21.15.89:0/39358394 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f560c15e2e0 msgr2=0x7f560c15e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f560aa00700 1 --2- 172.21.15.89:0/39358394 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f560c15e2e0 0x7f560c15e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f560aa00700 1 -- 172.21.15.89:0/39358394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f0004850 msgr2=0x7f560c0cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f560aa00700 1 --2- 172.21.15.89:0/39358394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f0004850 0x7f560c0cbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f560aa00700 1 -- 172.21.15.89:0/39358394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c0c9060 con 0x7f560c1605f0 2024-04-02T08:53:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f560aa00700 1 --2- 172.21.15.89:0/39358394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f560c1605f0 0x7f560c15dda0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f55f0004780 tx=0x7f55f80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f5603400700 1 -- 172.21.15.89:0/39358394 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f800e6c0 con 0x7f560c1605f0 2024-04-02T08:53:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f5603400700 1 -- 172.21.15.89:0/39358394 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f80038a0 con 0x7f560c1605f0 2024-04-02T08:53:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f5612000700 1 -- 172.21.15.89:0/39358394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f560c0c9340 con 0x7f560c1605f0 2024-04-02T08:53:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.686+0000 7f5603400700 1 -- 172.21.15.89:0/39358394 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8003c50 con 0x7f560c1605f0 2024-04-02T08:53:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.687+0000 7f5612000700 1 -- 172.21.15.89:0/39358394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f560c160a70 con 0x7f560c1605f0 2024-04-02T08:53:31.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.688+0000 7f5612000700 1 -- 172.21.15.89:0/39358394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f560c067720 con 0x7f560c1605f0 2024-04-02T08:53:31.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.689+0000 7f5603400700 1 -- 172.21.15.89:0/39358394 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f55f8013470 con 0x7f560c1605f0 2024-04-02T08:53:31.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.689+0000 7f5603400700 1 --2- 172.21.15.89:0/39358394 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f55ec06adb0 0x7f55ec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:31.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.690+0000 7f5603400700 1 -- 172.21.15.89:0/39358394 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f55f808af20 con 0x7f560c1605f0 2024-04-02T08:53:31.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.690+0000 7f560b400700 1 --2- 172.21.15.89:0/39358394 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f55ec06adb0 0x7f55ec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:31.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.690+0000 7f560b400700 1 --2- 172.21.15.89:0/39358394 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f55ec06adb0 0x7f55ec06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f55fc000fb0 tx=0x7f55fc001e00 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:31.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.694+0000 7f5603400700 1 -- 172.21.15.89:0/39358394 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f55f8059a80 con 0x7f560c1605f0 2024-04-02T08:53:31.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.928+0000 7f5612000700 1 -- 172.21.15.89:0/39358394 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f560c0c8c10 con 0x7f55ec06adb0 2024-04-02T08:53:31.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.929+0000 7f5603400700 1 -- 172.21.15.89:0/39358394 <== mgr.9807 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f560c0c8c10 con 0x7f55ec06adb0 2024-04-02T08:53:31.928 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T08:53:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T08:53:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T08:53:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T08:53:31.929 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T08:53:31.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.932+0000 7f5612000700 1 -- 172.21.15.89:0/39358394 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f55ec06adb0 msgr2=0x7f55ec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:31.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.932+0000 7f5612000700 1 --2- 172.21.15.89:0/39358394 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f55ec06adb0 0x7f55ec06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f55fc000fb0 tx=0x7f55fc001e00 comp rx=0 tx=0).stop 2024-04-02T08:53:31.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.932+0000 7f5612000700 1 -- 172.21.15.89:0/39358394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f560c1605f0 msgr2=0x7f560c15dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:31.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.932+0000 7f5612000700 1 --2- 172.21.15.89:0/39358394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f560c1605f0 0x7f560c15dda0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f55f0004780 tx=0x7f55f80047c0 comp rx=0 tx=0).stop 2024-04-02T08:53:31.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.932+0000 7f5612000700 1 -- 172.21.15.89:0/39358394 shutdown_connections 2024-04-02T08:53:31.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.932+0000 7f5612000700 1 --2- 172.21.15.89:0/39358394 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f55ec06adb0 0x7f55ec06d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:31.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.932+0000 7f5612000700 1 --2- 172.21.15.89:0/39358394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f0004850 0x7f560c0cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:31.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.932+0000 7f5612000700 1 --2- 172.21.15.89:0/39358394 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f560c15e2e0 0x7f560c15e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:31.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.932+0000 7f5612000700 1 --2- 172.21.15.89:0/39358394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f560c1605f0 0x7f560c15dda0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:31.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.932+0000 7f5612000700 1 -- 172.21.15.89:0/39358394 >> 172.21.15.89:0/39358394 conn(0x7f560c0b2e20 msgr2=0x7f560c0b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:31.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.932+0000 7f5612000700 1 -- 172.21.15.89:0/39358394 shutdown_connections 2024-04-02T08:53:31.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:31.932+0000 7f5612000700 1 -- 172.21.15.89:0/39358394 wait complete. 2024-04-02T08:53:31.950 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T08:53:32.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.143+0000 7f45b9c00700 1 Processor -- start 2024-04-02T08:53:32.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.143+0000 7f45b9c00700 1 -- start start 2024-04-02T08:53:32.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.144+0000 7f45b9c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f45b40d21f0 0x7f45b40ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:32.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.144+0000 7f45b9c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b40ce270 con 0x7f45b41611b0 2024-04-02T08:53:32.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.144+0000 7f45b9c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b415b600 con 0x7f45b40cec40 2024-04-02T08:53:32.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.144+0000 7f45b9c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b40d2760 con 0x7f45b40d21f0 2024-04-02T08:53:32.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.144+0000 7f45b2000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f45b40d21f0 0x7f45b40ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:32.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.145+0000 7f45b2000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f45b40d21f0 0x7f45b40ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57934/0 (socket says 172.21.15.89:57934) 2024-04-02T08:53:32.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.145+0000 7f45b2000700 1 -- 172.21.15.89:0/697661078 learned_addr learned my addr 172.21.15.89:0/697661078 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:32.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.145+0000 7f45b2a00700 1 --1- 172.21.15.89:0/697661078 >> v1:172.21.15.89:6789/0 conn(0x7f45b41611b0 0x7f45b415ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57408/0 (socket says 172.21.15.89:57408) 2024-04-02T08:53:32.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.145+0000 7f45b2000700 1 -- 172.21.15.89:0/697661078 >> v1:172.21.15.113:6789/0 conn(0x7f45b40cec40 legacy=0x7f45b4161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:32.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.145+0000 7f45b2000700 1 -- 172.21.15.89:0/697661078 >> v1:172.21.15.89:6789/0 conn(0x7f45b41611b0 legacy=0x7f45b415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:32.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.145+0000 7f45b2000700 1 -- 172.21.15.89:0/697661078 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b40d2a40 con 0x7f45b40d21f0 2024-04-02T08:53:32.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.145+0000 7f45b1600700 1 -- 172.21.15.89:0/697661078 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811016909 0 0) 0x7f45b40ce270 con 0x7f45b41611b0 2024-04-02T08:53:32.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.145+0000 7f45b2000700 1 --2- 172.21.15.89:0/697661078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b40d21f0 0x7f45b40ce700 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f45a00098f0 tx=0x7f45a0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eefa8a9b49630f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:32.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.145+0000 7f45b1600700 1 -- 172.21.15.89:0/697661078 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a000b8a0 con 0x7f45b40d21f0 2024-04-02T08:53:32.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.146+0000 7f45b1600700 1 -- 172.21.15.89:0/697661078 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a000ba00 con 0x7f45b40d21f0 2024-04-02T08:53:32.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.146+0000 7f45b1600700 1 -- 172.21.15.89:0/697661078 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a000bd90 con 0x7f45b40d21f0 2024-04-02T08:53:32.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.146+0000 7f45b9c00700 1 -- 172.21.15.89:0/697661078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b40d21f0 msgr2=0x7f45b40ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:32.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.146+0000 7f45b9c00700 1 --2- 172.21.15.89:0/697661078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b40d21f0 0x7f45b40ce700 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f45a00098f0 tx=0x7f45a0009c00 comp rx=0 tx=0).stop 2024-04-02T08:53:32.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.146+0000 7f45b9c00700 1 -- 172.21.15.89:0/697661078 shutdown_connections 2024-04-02T08:53:32.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.146+0000 7f45b9c00700 1 --2- 172.21.15.89:0/697661078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b40d21f0 0x7f45b40ce700 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.146+0000 7f45b9c00700 1 -- 172.21.15.89:0/697661078 >> 172.21.15.89:0/697661078 conn(0x7f45b408aff0 msgr2=0x7f45b40d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:32.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.146+0000 7f45b9c00700 1 -- 172.21.15.89:0/697661078 shutdown_connections 2024-04-02T08:53:32.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.146+0000 7f45b9c00700 1 -- 172.21.15.89:0/697661078 wait complete. 2024-04-02T08:53:32.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.148+0000 7f45b9c00700 1 Processor -- start 2024-04-02T08:53:32.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.148+0000 7f45b9c00700 1 -- start start 2024-04-02T08:53:32.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b9c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b40cec40 0x7f45b4153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:32.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b9c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b40d21f0 0x7f45b4152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:32.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b9c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b41611b0 0x7f45b40ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:32.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b9c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b40c79a0 con 0x7f45b40cec40 2024-04-02T08:53:32.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b9c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b41e8c60 con 0x7f45b41611b0 2024-04-02T08:53:32.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b9c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b41e8330 con 0x7f45b40d21f0 2024-04-02T08:53:32.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b40cec40 0x7f45b4153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:32.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b40cec40 0x7f45b4153300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57944/0 (socket says 172.21.15.89:57944) 2024-04-02T08:53:32.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b2a00700 1 -- 172.21.15.89:0/3969373417 learned_addr learned my addr 172.21.15.89:0/3969373417 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:32.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b2000700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b40d21f0 0x7f45b4152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:32.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b3400700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b41611b0 0x7f45b40ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:32.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b2a00700 1 -- 172.21.15.89:0/3969373417 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b40d21f0 msgr2=0x7f45b4152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:32.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b2a00700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b40d21f0 0x7f45b4152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b2a00700 1 -- 172.21.15.89:0/3969373417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b41611b0 msgr2=0x7f45b40ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:32.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b2a00700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b41611b0 0x7f45b40ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b2a00700 1 -- 172.21.15.89:0/3969373417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a0009550 con 0x7f45b40cec40 2024-04-02T08:53:32.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.149+0000 7f45b3400700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b41611b0 0x7f45b40ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T08:53:32.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.150+0000 7f45b2a00700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b40cec40 0x7f45b4153300 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f45a400c520 tx=0x7f45a400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:32.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.150+0000 7f45aaa00700 1 -- 172.21.15.89:0/3969373417 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4004c80 con 0x7f45b40cec40 2024-04-02T08:53:32.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.150+0000 7f45aaa00700 1 -- 172.21.15.89:0/3969373417 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a4004e00 con 0x7f45b40cec40 2024-04-02T08:53:32.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.150+0000 7f45aaa00700 1 -- 172.21.15.89:0/3969373417 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4007310 con 0x7f45b40cec40 2024-04-02T08:53:32.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.150+0000 7f45b9c00700 1 -- 172.21.15.89:0/3969373417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b41e8610 con 0x7f45b40cec40 2024-04-02T08:53:32.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.150+0000 7f45b9c00700 1 -- 172.21.15.89:0/3969373417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b40c75c0 con 0x7f45b40cec40 2024-04-02T08:53:32.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.151+0000 7f45b9c00700 1 -- 172.21.15.89:0/3969373417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b4067720 con 0x7f45b40cec40 2024-04-02T08:53:32.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.152+0000 7f45aaa00700 1 -- 172.21.15.89:0/3969373417 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 114) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f45a4007470 con 0x7f45b40cec40 2024-04-02T08:53:32.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.153+0000 7f45aaa00700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f459406adb0 0x7f459406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:32.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.153+0000 7f45aaa00700 1 -- 172.21.15.89:0/3969373417 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f45a4089390 con 0x7f45b40cec40 2024-04-02T08:53:32.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.153+0000 7f45b2000700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f459406adb0 0x7f459406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:32.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.154+0000 7f45b2000700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f459406adb0 0x7f459406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f45a0000c00 tx=0x7f45a0008030 comp rx=0 tx=0).ready entity=mgr.9807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.158+0000 7f45aaa00700 1 -- 172.21.15.89:0/3969373417 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f45a40542b0 con 0x7f45b40cec40 2024-04-02T08:53:32.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.412+0000 7f45b9c00700 1 -- 172.21.15.89:0/3969373417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f45b40b5d00 con 0x7f45b40cec40 2024-04-02T08:53:32.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.431+0000 7f45aaa00700 1 -- 172.21.15.89:0/3969373417 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 115) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f45a4087700 con 0x7f45b40cec40 2024-04-02T08:53:32.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.432+0000 7f45aaa00700 1 -- 172.21.15.89:0/3969373417 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v115) v1 ==== 92+0+0 (secure 0 0 0) 0x7f45a40745d0 con 0x7f45b40cec40 2024-04-02T08:53:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.435+0000 7f45b9c00700 1 -- 172.21.15.89:0/3969373417 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f459406adb0 msgr2=0x7f459406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.435+0000 7f45b9c00700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f459406adb0 0x7f459406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f45a0000c00 tx=0x7f45a0008030 comp rx=0 tx=0).stop 2024-04-02T08:53:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.435+0000 7f45b9c00700 1 -- 172.21.15.89:0/3969373417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b40cec40 msgr2=0x7f45b4153300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.435+0000 7f45b9c00700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b40cec40 0x7f45b4153300 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f45a400c520 tx=0x7f45a400c8e0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.435+0000 7f45b9c00700 1 -- 172.21.15.89:0/3969373417 shutdown_connections 2024-04-02T08:53:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.435+0000 7f45b9c00700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f459406adb0 0x7f459406d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.435+0000 7f45b9c00700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45b41611b0 0x7f45b40ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.435+0000 7f45b9c00700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45b40cec40 0x7f45b4153300 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.435+0000 7f45b9c00700 1 --2- 172.21.15.89:0/3969373417 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f45b40d21f0 0x7f45b4152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.435+0000 7f45b9c00700 1 -- 172.21.15.89:0/3969373417 >> 172.21.15.89:0/3969373417 conn(0x7f45b408aff0 msgr2=0x7f45b4160960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:32.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.435+0000 7f45b9c00700 1 -- 172.21.15.89:0/3969373417 shutdown_connections 2024-04-02T08:53:32.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.435+0000 7f45b9c00700 1 -- 172.21.15.89:0/3969373417 wait complete. 2024-04-02T08:53:32.454 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T08:53:32.647 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:32.648+0000 7f5564ee0240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T08:53:32.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.656+0000 7ff28fe00700 1 Processor -- start 2024-04-02T08:53:32.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.656+0000 7ff28fe00700 1 -- start start 2024-04-02T08:53:32.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.657+0000 7ff28fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815a5c0 0x7ff2880709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:32.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.657+0000 7ff28fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff28815dfe0 0x7ff2880ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:32.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.657+0000 7ff28fe00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2880cb310 con 0x7ff2880ce670 2024-04-02T08:53:32.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.657+0000 7ff28fe00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2880ceb00 con 0x7ff28815dfe0 2024-04-02T08:53:32.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.657+0000 7ff28fe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff288070f00 con 0x7ff28815a5c0 2024-04-02T08:53:32.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.658+0000 7ff28ca00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff28815dfe0 0x7ff2880ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:32.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.658+0000 7ff28ca00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff28815dfe0 0x7ff2880ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57962/0 (socket says 172.21.15.89:57962) 2024-04-02T08:53:32.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.658+0000 7ff28ca00700 1 -- 172.21.15.89:0/1546706882 learned_addr learned my addr 172.21.15.89:0/1546706882 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:32.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.658+0000 7ff28d400700 1 --2- 172.21.15.89:0/1546706882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815a5c0 0x7ff2880709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:32.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.658+0000 7ff28ca00700 1 -- 172.21.15.89:0/1546706882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815a5c0 msgr2=0x7ff2880709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:32.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.658+0000 7ff28ca00700 1 --2- 172.21.15.89:0/1546706882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815a5c0 0x7ff2880709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.658+0000 7ff28ca00700 1 -- 172.21.15.89:0/1546706882 >> v1:172.21.15.113:6789/0 conn(0x7ff2880ce670 legacy=0x7ff2880b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:32.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.658+0000 7ff28ca00700 1 -- 172.21.15.89:0/1546706882 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2880cca90 con 0x7ff28815dfe0 2024-04-02T08:53:32.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.658+0000 7ff28ca00700 1 --2- 172.21.15.89:0/1546706882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff28815dfe0 0x7ff2880ce130 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7ff27400bb90 tx=0x7ff27400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed9984223d22726 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.659+0000 7ff287e00700 1 -- 172.21.15.89:0/1546706882 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27400dd30 con 0x7ff28815dfe0 2024-04-02T08:53:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.659+0000 7ff287e00700 1 -- 172.21.15.89:0/1546706882 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27400de90 con 0x7ff28815dfe0 2024-04-02T08:53:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.659+0000 7ff287e00700 1 -- 172.21.15.89:0/1546706882 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff274010690 con 0x7ff28815dfe0 2024-04-02T08:53:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.659+0000 7ff28fe00700 1 -- 172.21.15.89:0/1546706882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff28815dfe0 msgr2=0x7ff2880ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.659+0000 7ff28fe00700 1 --2- 172.21.15.89:0/1546706882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff28815dfe0 0x7ff2880ce130 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7ff27400bb90 tx=0x7ff27400bea0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.659+0000 7ff28fe00700 1 -- 172.21.15.89:0/1546706882 shutdown_connections 2024-04-02T08:53:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.659+0000 7ff28fe00700 1 --2- 172.21.15.89:0/1546706882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815a5c0 0x7ff2880709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.659+0000 7ff28fe00700 1 --2- 172.21.15.89:0/1546706882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff28815dfe0 0x7ff2880ce130 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.659+0000 7ff28fe00700 1 -- 172.21.15.89:0/1546706882 >> 172.21.15.89:0/1546706882 conn(0x7ff2880b0d70 msgr2=0x7ff2880b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.659+0000 7ff28fe00700 1 -- 172.21.15.89:0/1546706882 shutdown_connections 2024-04-02T08:53:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.659+0000 7ff28fe00700 1 -- 172.21.15.89:0/1546706882 wait complete. 2024-04-02T08:53:32.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.661+0000 7ff28fe00700 1 Processor -- start 2024-04-02T08:53:32.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.661+0000 7ff28fe00700 1 -- start start 2024-04-02T08:53:32.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.662+0000 7ff28fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2880ce670 0x7ff288155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:32.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.662+0000 7ff28fe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28815a5c0 0x7ff2881f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:32.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.662+0000 7ff28fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815dfe0 0x7ff28806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:32.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.662+0000 7ff28fe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff288144530 con 0x7ff2880ce670 2024-04-02T08:53:32.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.662+0000 7ff28fe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff288144810 con 0x7ff28815a5c0 2024-04-02T08:53:32.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.662+0000 7ff28fe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff288144af0 con 0x7ff28815dfe0 2024-04-02T08:53:32.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.662+0000 7ff28d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2880ce670 0x7ff288155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:32.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.662+0000 7ff28ca00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28815a5c0 0x7ff2881f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:32.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.663+0000 7ff28d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2880ce670 0x7ff288155030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57970/0 (socket says 172.21.15.89:57970) 2024-04-02T08:53:32.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.663+0000 7ff28d400700 1 -- 172.21.15.89:0/2966218889 learned_addr learned my addr 172.21.15.89:0/2966218889 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:32.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.663+0000 7ff28de00700 1 --2- 172.21.15.89:0/2966218889 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815dfe0 0x7ff28806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:32.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.663+0000 7ff28ca00700 1 --2- 172.21.15.89:0/2966218889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28815a5c0 0x7ff2881f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:55136/0 (socket says 172.21.15.89:55136) 2024-04-02T08:53:32.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.663+0000 7ff28d400700 1 -- 172.21.15.89:0/2966218889 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815dfe0 msgr2=0x7ff28806d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:32.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.663+0000 7ff28d400700 1 --2- 172.21.15.89:0/2966218889 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815dfe0 0x7ff28806d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.663+0000 7ff28d400700 1 -- 172.21.15.89:0/2966218889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28815a5c0 msgr2=0x7ff2881f2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:32.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.663+0000 7ff28d400700 1 --2- 172.21.15.89:0/2966218889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28815a5c0 0x7ff2881f2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.663+0000 7ff28d400700 1 -- 172.21.15.89:0/2966218889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27400b7f0 con 0x7ff2880ce670 2024-04-02T08:53:32.663 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:32.663+0000 7ff3ebbaf240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T08:53:32.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.663+0000 7ff28d400700 1 --2- 172.21.15.89:0/2966218889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2880ce670 0x7ff288155030 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00e9b0 tx=0x7ff27c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:32.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.663+0000 7ff285600700 1 -- 172.21.15.89:0/2966218889 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27c009710 con 0x7ff2880ce670 2024-04-02T08:53:32.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.663+0000 7ff285600700 1 -- 172.21.15.89:0/2966218889 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27c00ee50 con 0x7ff2880ce670 2024-04-02T08:53:32.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.663+0000 7ff285600700 1 -- 172.21.15.89:0/2966218889 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27c00ca50 con 0x7ff2880ce670 2024-04-02T08:53:32.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.664+0000 7ff28fe00700 1 -- 172.21.15.89:0/2966218889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff288144dd0 con 0x7ff2880ce670 2024-04-02T08:53:32.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.664+0000 7ff28fe00700 1 -- 172.21.15.89:0/2966218889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff288155740 con 0x7ff2880ce670 2024-04-02T08:53:32.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.665+0000 7ff28fe00700 1 -- 172.21.15.89:0/2966218889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff288067720 con 0x7ff2880ce670 2024-04-02T08:53:32.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.666+0000 7ff285600700 1 -- 172.21.15.89:0/2966218889 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 115) v1 ==== 89256+0+0 (secure 0 0 0) 0x7ff27c00cbb0 con 0x7ff2880ce670 2024-04-02T08:53:32.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.666+0000 7ff285600700 1 --2- 172.21.15.89:0/2966218889 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff26806ae00 0x7ff26806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:32.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.667+0000 7ff285600700 1 -- 172.21.15.89:0/2966218889 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7ff27c089650 con 0x7ff2880ce670 2024-04-02T08:53:32.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.667+0000 7ff28ca00700 1 -- 172.21.15.89:0/2966218889 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff26806ae00 msgr2=0x7ff26806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:32.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.667+0000 7ff28ca00700 1 --2- 172.21.15.89:0/2966218889 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff26806ae00 0x7ff26806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:32.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.671+0000 7ff285600700 1 -- 172.21.15.89:0/2966218889 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff27c054570 con 0x7ff2880ce670 2024-04-02T08:53:32.833 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:32.834+0000 7f5564ee0240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T08:53:32.852 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:32.854+0000 7ff3ebbaf240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T08:53:32.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.868+0000 7ff28ca00700 1 -- 172.21.15.89:0/2966218889 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff26806ae00 msgr2=0x7ff26806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:32.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.868+0000 7ff28ca00700 1 --2- 172.21.15.89:0/2966218889 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff26806ae00 0x7ff26806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:32.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.918+0000 7ff28fe00700 1 -- 172.21.15.89:0/2966218889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff2880b1ad0 con 0x7ff2880ce670 2024-04-02T08:53:32.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.924+0000 7ff285600700 1 -- 172.21.15.89:0/2966218889 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff27c00ce60 con 0x7ff2880ce670 2024-04-02T08:53:32.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.927+0000 7ff28fe00700 1 -- 172.21.15.89:0/2966218889 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff26806ae00 msgr2=0x7ff26806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:32.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.927+0000 7ff28fe00700 1 --2- 172.21.15.89:0/2966218889 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff26806ae00 0x7ff26806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.927+0000 7ff28fe00700 1 -- 172.21.15.89:0/2966218889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2880ce670 msgr2=0x7ff288155030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:32.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.927+0000 7ff28fe00700 1 --2- 172.21.15.89:0/2966218889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2880ce670 0x7ff288155030 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00e9b0 tx=0x7ff27c00ecc0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.927+0000 7ff28fe00700 1 -- 172.21.15.89:0/2966218889 shutdown_connections 2024-04-02T08:53:32.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.927+0000 7ff28fe00700 1 --2- 172.21.15.89:0/2966218889 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7ff26806ae00 0x7ff26806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.927+0000 7ff28fe00700 1 --2- 172.21.15.89:0/2966218889 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815dfe0 0x7ff28806d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.927+0000 7ff28fe00700 1 --2- 172.21.15.89:0/2966218889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2880ce670 0x7ff288155030 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.927+0000 7ff28fe00700 1 --2- 172.21.15.89:0/2966218889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28815a5c0 0x7ff2881f2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:32.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.927+0000 7ff28fe00700 1 -- 172.21.15.89:0/2966218889 >> 172.21.15.89:0/2966218889 conn(0x7ff2880b0d70 msgr2=0x7ff2880cdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:32.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.927+0000 7ff28fe00700 1 -- 172.21.15.89:0/2966218889 shutdown_connections 2024-04-02T08:53:32.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:32.927+0000 7ff28fe00700 1 -- 172.21.15.89:0/2966218889 wait complete. 2024-04-02T08:53:32.945 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T08:53:33.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.144+0000 7f011b600700 1 Processor -- start 2024-04-02T08:53:33.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.145+0000 7f011b600700 1 -- start start 2024-04-02T08:53:33.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.146+0000 7f011b600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f011415dad0 0x7f01140ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:33.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.146+0000 7f011b600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f01140d21d0 0x7f011415fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:33.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.146+0000 7f011b600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011415f820 con 0x7f01140d1630 2024-04-02T08:53:33.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.146+0000 7f011b600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01140cfcc0 con 0x7f01140d21d0 2024-04-02T08:53:33.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.146+0000 7f011b600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01140cd9f0 con 0x7f011415dad0 2024-04-02T08:53:33.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.146+0000 7f0119600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f01140d21d0 0x7f011415fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:33.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.146+0000 7f0119600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f01140d21d0 0x7f011415fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57976/0 (socket says 172.21.15.89:57976) 2024-04-02T08:53:33.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.146+0000 7f0119600700 1 -- 172.21.15.89:0/2994866478 learned_addr learned my addr 172.21.15.89:0/2994866478 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:33.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.146+0000 7f0118c00700 1 --2- 172.21.15.89:0/2994866478 >> v2:172.21.15.113:3300/0 conn(0x7f011415dad0 0x7f01140ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:33.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.146+0000 7f0113e00700 1 --1- 172.21.15.89:0/2994866478 >> v1:172.21.15.113:6789/0 conn(0x7f01140d1630 0x7f01140d1ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:54030/0 (socket says 172.21.15.89:54030) 2024-04-02T08:53:33.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.146+0000 7f0119600700 1 -- 172.21.15.89:0/2994866478 >> v2:172.21.15.113:3300/0 conn(0x7f011415dad0 msgr2=0x7f01140ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:33.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.146+0000 7f0119600700 1 --2- 172.21.15.89:0/2994866478 >> v2:172.21.15.113:3300/0 conn(0x7f011415dad0 0x7f01140ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.147+0000 7f0119600700 1 -- 172.21.15.89:0/2994866478 >> v1:172.21.15.113:6789/0 conn(0x7f01140d1630 legacy=0x7f01140d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:33.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.146+0000 7f0113400700 1 -- 172.21.15.89:0/2994866478 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311368936 0 0) 0x7f011415f820 con 0x7f01140d1630 2024-04-02T08:53:33.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.147+0000 7f0119600700 1 -- 172.21.15.89:0/2994866478 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01140cf420 con 0x7f01140d21d0 2024-04-02T08:53:33.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.147+0000 7f0119600700 1 --2- 172.21.15.89:0/2994866478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01140d21d0 0x7f011415fcb0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f011408d380 tx=0x7f010c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e62347b093fc59d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:33.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.147+0000 7f0113400700 1 -- 172.21.15.89:0/2994866478 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c00e070 con 0x7f01140d21d0 2024-04-02T08:53:33.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.147+0000 7f0113400700 1 -- 172.21.15.89:0/2994866478 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010c00b910 con 0x7f01140d21d0 2024-04-02T08:53:33.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.147+0000 7f0113400700 1 -- 172.21.15.89:0/2994866478 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c00bcc0 con 0x7f01140d21d0 2024-04-02T08:53:33.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.148+0000 7f011b600700 1 -- 172.21.15.89:0/2994866478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01140d21d0 msgr2=0x7f011415fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:33.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.148+0000 7f011b600700 1 --2- 172.21.15.89:0/2994866478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01140d21d0 0x7f011415fcb0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f011408d380 tx=0x7f010c00b330 comp rx=0 tx=0).stop 2024-04-02T08:53:33.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.148+0000 7f011b600700 1 -- 172.21.15.89:0/2994866478 shutdown_connections 2024-04-02T08:53:33.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.148+0000 7f011b600700 1 --2- 172.21.15.89:0/2994866478 >> v2:172.21.15.113:3300/0 conn(0x7f011415dad0 0x7f01140ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.148+0000 7f011b600700 1 --2- 172.21.15.89:0/2994866478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01140d21d0 0x7f011415fcb0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.148+0000 7f011b600700 1 -- 172.21.15.89:0/2994866478 >> 172.21.15.89:0/2994866478 conn(0x7f011408aff0 msgr2=0x7f01140d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:33.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.148+0000 7f011b600700 1 -- 172.21.15.89:0/2994866478 shutdown_connections 2024-04-02T08:53:33.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.148+0000 7f011b600700 1 -- 172.21.15.89:0/2994866478 wait complete. 2024-04-02T08:53:33.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.150+0000 7f011b600700 1 Processor -- start 2024-04-02T08:53:33.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.150+0000 7f011b600700 1 -- start start 2024-04-02T08:53:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f011b600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01140d1630 0x7f01141515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f011b600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01140d21d0 0x7f01140b8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f011b600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f011415dad0 0x7f01140c9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f011b600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0114154980 con 0x7f011415dad0 2024-04-02T08:53:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f011b600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0114154e10 con 0x7f01140d21d0 2024-04-02T08:53:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f011b600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0114154370 con 0x7f01140d1630 2024-04-02T08:53:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f0119600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f011415dad0 0x7f01140c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f0119600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f011415dad0 0x7f01140c9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57980/0 (socket says 172.21.15.89:57980) 2024-04-02T08:53:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f0119600700 1 -- 172.21.15.89:0/3760366577 learned_addr learned my addr 172.21.15.89:0/3760366577 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f0118c00700 1 --2- 172.21.15.89:0/3760366577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01140d1630 0x7f01141515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f0113e00700 1 --2- 172.21.15.89:0/3760366577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01140d21d0 0x7f01140b8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f0119600700 1 -- 172.21.15.89:0/3760366577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01140d1630 msgr2=0x7f01141515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f0119600700 1 --2- 172.21.15.89:0/3760366577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01140d1630 0x7f01141515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f0119600700 1 -- 172.21.15.89:0/3760366577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01140d21d0 msgr2=0x7f01140b8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f0119600700 1 --2- 172.21.15.89:0/3760366577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01140d21d0 0x7f01140b8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.151+0000 7f0119600700 1 -- 172.21.15.89:0/3760366577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010c009b30 con 0x7f011415dad0 2024-04-02T08:53:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.152+0000 7f0119600700 1 --2- 172.21.15.89:0/3760366577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f011415dad0 0x7f01140c9100 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f011408ce40 tx=0x7f010c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.152+0000 7f0110c00700 1 -- 172.21.15.89:0/3760366577 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c004840 con 0x7f011415dad0 2024-04-02T08:53:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.152+0000 7f0110c00700 1 -- 172.21.15.89:0/3760366577 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010c0092e0 con 0x7f011415dad0 2024-04-02T08:53:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.152+0000 7f011b600700 1 -- 172.21.15.89:0/3760366577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0114155670 con 0x7f011415dad0 2024-04-02T08:53:33.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.152+0000 7f0110c00700 1 -- 172.21.15.89:0/3760366577 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c01c660 con 0x7f011415dad0 2024-04-02T08:53:33.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.152+0000 7f011b600700 1 -- 172.21.15.89:0/3760366577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0114155950 con 0x7f011415dad0 2024-04-02T08:53:33.153 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:33.153+0000 7f5564ee0240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T08:53:33.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.153+0000 7f011b600700 1 -- 172.21.15.89:0/3760366577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0114154af0 con 0x7f011415dad0 2024-04-02T08:53:33.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.154+0000 7f0110c00700 1 -- 172.21.15.89:0/3760366577 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 115) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f010c01c7c0 con 0x7f011415dad0 2024-04-02T08:53:33.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.155+0000 7f0110c00700 1 --2- 172.21.15.89:0/3760366577 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f00f406ae00 0x7f00f406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:33.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.155+0000 7f0110c00700 1 -- 172.21.15.89:0/3760366577 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f010c08b090 con 0x7f011415dad0 2024-04-02T08:53:33.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.155+0000 7f0118c00700 1 -- 172.21.15.89:0/3760366577 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f00f406ae00 msgr2=0x7f00f406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:33.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.155+0000 7f0118c00700 1 --2- 172.21.15.89:0/3760366577 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f00f406ae00 0x7f00f406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:33.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.162+0000 7f0110c00700 1 -- 172.21.15.89:0/3760366577 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f010c055e90 con 0x7f011415dad0 2024-04-02T08:53:33.193 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:33.195+0000 7ff3ebbaf240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T08:53:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.356+0000 7f0118c00700 1 -- 172.21.15.89:0/3760366577 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f00f406ae00 msgr2=0x7f00f406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.356+0000 7f0118c00700 1 --2- 172.21.15.89:0/3760366577 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f00f406ae00 0x7f00f406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:33.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.369+0000 7f011b600700 1 -- 172.21.15.89:0/3760366577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f0114067720 con 0x7f011415dad0 2024-04-02T08:53:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.373+0000 7f0110c00700 1 -- 172.21.15.89:0/3760366577 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7f010c0594b0 con 0x7f011415dad0 2024-04-02T08:53:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.376+0000 7f011b600700 1 -- 172.21.15.89:0/3760366577 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f00f406ae00 msgr2=0x7f00f406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.376+0000 7f011b600700 1 --2- 172.21.15.89:0/3760366577 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f00f406ae00 0x7f00f406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.376+0000 7f011b600700 1 -- 172.21.15.89:0/3760366577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f011415dad0 msgr2=0x7f01140c9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.376+0000 7f011b600700 1 --2- 172.21.15.89:0/3760366577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f011415dad0 0x7f01140c9100 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f011408ce40 tx=0x7f010c004ad0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.376+0000 7f011b600700 1 -- 172.21.15.89:0/3760366577 shutdown_connections 2024-04-02T08:53:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.376+0000 7f011b600700 1 --2- 172.21.15.89:0/3760366577 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f00f406ae00 0x7f00f406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.376+0000 7f011b600700 1 --2- 172.21.15.89:0/3760366577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f011415dad0 0x7f01140c9100 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.376+0000 7f011b600700 1 --2- 172.21.15.89:0/3760366577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01140d1630 0x7f01141515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.376+0000 7f011b600700 1 --2- 172.21.15.89:0/3760366577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01140d21d0 0x7f01140b8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.376+0000 7f011b600700 1 -- 172.21.15.89:0/3760366577 >> 172.21.15.89:0/3760366577 conn(0x7f011408aff0 msgr2=0x7f01140ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:33.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.376+0000 7f011b600700 1 -- 172.21.15.89:0/3760366577 shutdown_connections 2024-04-02T08:53:33.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.376+0000 7f011b600700 1 -- 172.21.15.89:0/3760366577 wait complete. 2024-04-02T08:53:33.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:53:33.394 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-02T08:53:33.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T08:53:33.443 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2024-04-02T08:53:33.488 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2024-04-02T08:53:33.681 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:33.682+0000 7f5564ee0240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T08:53:33.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.691+0000 7f432e600700 1 Processor -- start 2024-04-02T08:53:33.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.691+0000 7f432e600700 1 -- start start 2024-04-02T08:53:33.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.692+0000 7f432e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43280ced80 0x7f43280d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:33.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.692+0000 7f432e600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f432815fcb0 0x7f4328160140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:33.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.692+0000 7f432e600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43280cf1e0 con 0x7f432815dad0 2024-04-02T08:53:33.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.692+0000 7f432e600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43280cfcc0 con 0x7f432815fcb0 2024-04-02T08:53:33.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.692+0000 7f432e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43280cd9f0 con 0x7f43280ced80 2024-04-02T08:53:33.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.692+0000 7f4327400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f432815dad0 0x7f43280d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57430/0 (socket says 172.21.15.89:57430) 2024-04-02T08:53:33.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.692+0000 7f4327400700 1 -- 172.21.15.89:0/259499131 learned_addr learned my addr 172.21.15.89:0/259499131 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T08:53:33.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.692+0000 7f4326a00700 1 --2- 172.21.15.89:0/259499131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43280ced80 0x7f43280d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:33.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.692+0000 7f4327e00700 1 --2- 172.21.15.89:0/259499131 >> v2:172.21.15.113:3300/0 conn(0x7f432815fcb0 0x7f4328160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:33.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.693+0000 7f4326000700 1 -- 172.21.15.89:0/259499131 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178201845 0 0) 0x7f4318001110 con 0x7f432815dad0 2024-04-02T08:53:33.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.693+0000 7f4326000700 1 -- 172.21.15.89:0/259499131 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f430c0035b0 con 0x7f432815dad0 2024-04-02T08:53:33.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.693+0000 7f4326a00700 1 -- 172.21.15.89:0/259499131 >> v2:172.21.15.113:3300/0 conn(0x7f432815fcb0 msgr2=0x7f4328160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:33.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.693+0000 7f4326a00700 1 --2- 172.21.15.89:0/259499131 >> v2:172.21.15.113:3300/0 conn(0x7f432815fcb0 0x7f4328160140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.693+0000 7f4326a00700 1 -- 172.21.15.89:0/259499131 >> v1:172.21.15.89:6789/0 conn(0x7f432815dad0 legacy=0x7f43280d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.693+0000 7f4326a00700 1 -- 172.21.15.89:0/259499131 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432815f820 con 0x7f43280ced80 2024-04-02T08:53:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.693+0000 7f4327e00700 1 --2- 172.21.15.89:0/259499131 >> v2:172.21.15.113:3300/0 conn(0x7f432815fcb0 0x7f4328160140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T08:53:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.693+0000 7f4326a00700 1 --2- 172.21.15.89:0/259499131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43280ced80 0x7f43280d1d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4314009840 tx=0x7f4314009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8ad64091c525039 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.693+0000 7f4326000700 1 -- 172.21.15.89:0/259499131 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314011830 con 0x7f43280ced80 2024-04-02T08:53:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.693+0000 7f4326000700 1 -- 172.21.15.89:0/259499131 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4314011990 con 0x7f43280ced80 2024-04-02T08:53:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.693+0000 7f4326000700 1 -- 172.21.15.89:0/259499131 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314011d00 con 0x7f43280ced80 2024-04-02T08:53:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.694+0000 7f432e600700 1 -- 172.21.15.89:0/259499131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43280ced80 msgr2=0x7f43280d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.694+0000 7f432e600700 1 --2- 172.21.15.89:0/259499131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43280ced80 0x7f43280d1d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4314009840 tx=0x7f4314009b50 comp rx=0 tx=0).stop 2024-04-02T08:53:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.694+0000 7f432e600700 1 -- 172.21.15.89:0/259499131 shutdown_connections 2024-04-02T08:53:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.694+0000 7f432e600700 1 --2- 172.21.15.89:0/259499131 >> v2:172.21.15.113:3300/0 conn(0x7f432815fcb0 0x7f4328160140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.694+0000 7f432e600700 1 --2- 172.21.15.89:0/259499131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43280ced80 0x7f43280d1d40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.694+0000 7f432e600700 1 -- 172.21.15.89:0/259499131 >> 172.21.15.89:0/259499131 conn(0x7f432808aff0 msgr2=0x7f43280d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.694+0000 7f432e600700 1 -- 172.21.15.89:0/259499131 shutdown_connections 2024-04-02T08:53:33.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.694+0000 7f432e600700 1 -- 172.21.15.89:0/259499131 wait complete. 2024-04-02T08:53:33.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.697+0000 7f432e600700 1 Processor -- start 2024-04-02T08:53:33.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.698+0000 7f432e600700 1 -- start start 2024-04-02T08:53:33.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.699+0000 7f432e600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43280ced80 0x7f43280c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:33.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.699+0000 7f432e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f432815dad0 0x7f43280c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:33.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.699+0000 7f432e600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f432815fcb0 0x7f4328148450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:33.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.699+0000 7f432e600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43280c9970 con 0x7f432815fcb0 2024-04-02T08:53:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.699+0000 7f432e600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43280c8cf0 con 0x7f43280ced80 2024-04-02T08:53:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.699+0000 7f432e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43280c8fd0 con 0x7f432815dad0 2024-04-02T08:53:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.699+0000 7f4327e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f432815fcb0 0x7f4328148450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.699+0000 7f4327400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43280ced80 0x7f43280c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.700+0000 7f4327400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43280ced80 0x7f43280c80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:55170/0 (socket says 172.21.15.89:55170) 2024-04-02T08:53:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.700+0000 7f4327e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f432815fcb0 0x7f4328148450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57982/0 (socket says 172.21.15.89:57982) 2024-04-02T08:53:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.700+0000 7f4327400700 1 -- 172.21.15.89:0/1009086278 learned_addr learned my addr 172.21.15.89:0/1009086278 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.700+0000 7f4326a00700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f432815dad0 0x7f43280c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.700+0000 7f4327400700 1 -- 172.21.15.89:0/1009086278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f432815dad0 msgr2=0x7f43280c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.700+0000 7f4327400700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f432815dad0 0x7f43280c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.700+0000 7f4327400700 1 -- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f432815fcb0 msgr2=0x7f4328148450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:33.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.700+0000 7f4327400700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f432815fcb0 0x7f4328148450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:33.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.700+0000 7f4327400700 1 -- 172.21.15.89:0/1009086278 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4320009a70 con 0x7f43280ced80 2024-04-02T08:53:33.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.700+0000 7f4327400700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43280ced80 0x7f43280c80a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f431800c4d0 tx=0x7f431800c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:33.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.700+0000 7f431f400700 1 -- 172.21.15.89:0/1009086278 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4318005d70 con 0x7f43280ced80 2024-04-02T08:53:33.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.700+0000 7f431f400700 1 -- 172.21.15.89:0/1009086278 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43180070d0 con 0x7f43280ced80 2024-04-02T08:53:33.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.700+0000 7f431f400700 1 -- 172.21.15.89:0/1009086278 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4318007480 con 0x7f43280ced80 2024-04-02T08:53:33.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.701+0000 7f432e600700 1 -- 172.21.15.89:0/1009086278 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43140094a0 con 0x7f43280ced80 2024-04-02T08:53:33.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.701+0000 7f432e600700 1 -- 172.21.15.89:0/1009086278 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43280c9590 con 0x7f43280ced80 2024-04-02T08:53:33.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.702+0000 7f432e600700 1 -- 172.21.15.89:0/1009086278 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43281463c0 con 0x7f43280ced80 2024-04-02T08:53:33.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.703+0000 7f431f400700 1 -- 172.21.15.89:0/1009086278 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 115) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f43180075e0 con 0x7f43280ced80 2024-04-02T08:53:33.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.703+0000 7f431f400700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 0x7f430806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:33.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.703+0000 7f4326a00700 1 -- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 msgr2=0x7f430806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:33.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.703+0000 7f431f400700 1 -- 172.21.15.89:0/1009086278 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f4318088980 con 0x7f43280ced80 2024-04-02T08:53:33.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.703+0000 7f4326a00700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 0x7f430806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T08:53:33.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.708+0000 7f431f400700 1 -- 172.21.15.89:0/1009086278 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4318053890 con 0x7f43280ced80 2024-04-02T08:53:33.746 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:33.748+0000 7ff3ebbaf240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T08:53:33.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.904+0000 7f4326a00700 1 -- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 msgr2=0x7f430806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:33.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.904+0000 7f4326a00700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 0x7f430806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T08:53:33.938 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:33.939+0000 7f5564ee0240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T08:53:33.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:33.944+0000 7f432e600700 1 -- 172.21.15.89:0/1009086278 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f43280cf9e0 con 0x7f430806ae00 2024-04-02T08:53:34.005 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:34.006+0000 7ff3ebbaf240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T08:53:34.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:34.305+0000 7f4326a00700 1 -- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 msgr2=0x7f430806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:34.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:34.305+0000 7f4326a00700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 0x7f430806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T08:53:34.395 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:34.396+0000 7f5564ee0240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T08:53:34.464 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:34.465+0000 7ff3ebbaf240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T08:53:35.101 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:35.102+0000 7f5564ee0240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T08:53:35.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:35.106+0000 7f4326a00700 1 -- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 msgr2=0x7f430806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:35.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:35.106+0000 7f4326a00700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 0x7f430806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T08:53:35.173 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:35.175+0000 7ff3ebbaf240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T08:53:35.195 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:35.196+0000 7f5564ee0240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T08:53:35.265 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:35.267+0000 7ff3ebbaf240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T08:53:35.352 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:35.353+0000 7f5564ee0240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T08:53:35.423 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:35.424+0000 7ff3ebbaf240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T08:53:35.563 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:35.564+0000 7f5564ee0240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T08:53:35.638 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:35.639+0000 7ff3ebbaf240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T08:53:35.653 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:35.654+0000 7f5564ee0240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T08:53:35.730 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:35.731+0000 7ff3ebbaf240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T08:53:35.751 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:35.752+0000 7f5564ee0240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T08:53:35.829 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:35.830+0000 7ff3ebbaf240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T08:53:35.849 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:35.850+0000 7f5564ee0240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T08:53:35.927 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:35.928+0000 7ff3ebbaf240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T08:53:35.946 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:35.947+0000 7f5564ee0240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T08:53:36.025 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:36.027+0000 7ff3ebbaf240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T08:53:36.158 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:36.159+0000 7f5564ee0240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T08:53:36.239 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:36.240+0000 7ff3ebbaf240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T08:53:36.242 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:53:36.246 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:53:36.419 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:36.420+0000 7f5564ee0240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T08:53:36.502 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:36.503+0000 7ff3ebbaf240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T08:53:36.610 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:36.611+0000 7f5564ee0240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T08:53:36.693 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:36.695+0000 7ff3ebbaf240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T08:53:36.702 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:36.703+0000 7f5564ee0240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T08:53:36.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:36.708+0000 7f4326a00700 1 -- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 msgr2=0x7f430806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/43357 2024-04-02T08:53:36.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:36.708+0000 7f4326a00700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 0x7f430806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T08:53:36.786 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:36.788+0000 7ff3ebbaf240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T08:53:37.432 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:37.433+0000 7f5564ee0240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T08:53:37.517 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:37.518+0000 7ff3ebbaf240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T08:53:37.643 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:37.644+0000 7f5564ee0240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T08:53:37.729 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:37.730+0000 7ff3ebbaf240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T08:53:37.851 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:37.852+0000 7f5564ee0240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:53:37.937 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:37.939+0000 7ff3ebbaf240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:53:38.322 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:38.323+0000 7f5564ee0240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T08:53:38.414 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:38.414+0000 7ff3ebbaf240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T08:53:38.529 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:38.530+0000 7f5564ee0240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:53:38.622 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:38.622+0000 7ff3ebbaf240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T08:53:38.968 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:38.969+0000 7f5564ee0240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T08:53:39.066 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:39.066+0000 7ff3ebbaf240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T08:53:39.078 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T08:53:39.080+0000 7f5564ee0240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T08:53:39.178 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:39.178+0000 7ff3ebbaf240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T08:53:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:39.190+0000 7f431f400700 1 -- 172.21.15.89:0/1009086278 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mgrmap(e 116) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f4318053c60 con 0x7f43280ced80 2024-04-02T08:53:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:39.190+0000 7f431f400700 1 -- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 msgr2=0x7f430806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T08:53:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:39.190+0000 7f431f400700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 0x7f430806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:39.350 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:39.350+0000 7ff358000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:39.350 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:39.350+0000 7ff358000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:39.350 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:39.350+0000 7ff358000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:39.350 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:39.350+0000 7ff358000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:39.350 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:39.350+0000 7ff358000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:39.350 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:39.350+0000 7ff353a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:39.350 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:39.350+0000 7ff353a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:39.350 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:39.350+0000 7ff353a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:39.351 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:39.350+0000 7ff353a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:39.351 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:53:39.350+0000 7ff353a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:53:40.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.195+0000 7f431f400700 1 -- 172.21.15.89:0/1009086278 <== mon.1 v2:172.21.15.113:3300/0 8 ==== mgrmap(e 117) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f4318086c50 con 0x7f43280ced80 2024-04-02T08:53:40.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.196+0000 7f431f400700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 0x7f430806d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:40.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.196+0000 7f431f400700 1 -- 172.21.15.89:0/1009086278 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f430801d540 con 0x7f430806ae00 2024-04-02T08:53:40.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.197+0000 7f4326a00700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 0x7f430806d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:40.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.199+0000 7f4326a00700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 0x7f430806d240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4314009fd0 tx=0x7f4314008030 comp rx=0 tx=0).ready entity=mgr.10221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:40.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.267+0000 7f431f400700 1 -- 172.21.15.89:0/1009086278 <== mgr.10221 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f430801d540 con 0x7f430806ae00 2024-04-02T08:53:40.267 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path /snap_test_dir1/test_restart 2024-04-02T08:53:40.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.269+0000 7f432e600700 1 -- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 msgr2=0x7f430806d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:40.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.269+0000 7f432e600700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 0x7f430806d240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4314009fd0 tx=0x7f4314008030 comp rx=0 tx=0).stop 2024-04-02T08:53:40.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.269+0000 7f432e600700 1 -- 172.21.15.89:0/1009086278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43280ced80 msgr2=0x7f43280c80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:40.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.269+0000 7f432e600700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43280ced80 0x7f43280c80a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f431800c4d0 tx=0x7f431800c7e0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.269+0000 7f432e600700 1 -- 172.21.15.89:0/1009086278 shutdown_connections 2024-04-02T08:53:40.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.269+0000 7f432e600700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f430806ae00 0x7f430806d240 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.269+0000 7f432e600700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43280ced80 0x7f43280c80a0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.269+0000 7f432e600700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f432815fcb0 0x7f4328148450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.269+0000 7f432e600700 1 --2- 172.21.15.89:0/1009086278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f432815dad0 0x7f43280c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.269+0000 7f432e600700 1 -- 172.21.15.89:0/1009086278 >> 172.21.15.89:0/1009086278 conn(0x7f432808aff0 msgr2=0x7f43280ce270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:40.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.270+0000 7f432e600700 1 -- 172.21.15.89:0/1009086278 shutdown_connections 2024-04-02T08:53:40.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.270+0000 7f432e600700 1 -- 172.21.15.89:0/1009086278 wait complete. 2024-04-02T08:53:40.287 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2024-04-02T08:53:40.288 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2024-04-02T08:53:40.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.483+0000 7f2192200700 1 Processor -- start 2024-04-02T08:53:40.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.483+0000 7f2192200700 1 -- start start 2024-04-02T08:53:40.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.484+0000 7f2192200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f218c0d16a0 0x7f218c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:40.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.484+0000 7f2192200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f218c0d2370 0x7f218c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:40.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.484+0000 7f2192200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f218c161420 con 0x7f218c15d9d0 2024-04-02T08:53:40.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.484+0000 7f2192200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f218c0d2820 con 0x7f218c0d16a0 2024-04-02T08:53:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.484+0000 7f2192200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0d4340 con 0x7f218c0d2370 2024-04-02T08:53:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.484+0000 7f218aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f218c0d16a0 0x7f218c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.484+0000 7f218aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f218c0d16a0 0x7f218c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60876/0 (socket says 172.21.15.89:60876) 2024-04-02T08:53:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.484+0000 7f218aa00700 1 -- 172.21.15.89:0/1544141000 learned_addr learned my addr 172.21.15.89:0/1544141000 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.485+0000 7f218be00700 1 --2- 172.21.15.89:0/1544141000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f218c0d2370 0x7f218c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.485+0000 7f218aa00700 1 -- 172.21.15.89:0/1544141000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f218c0d2370 msgr2=0x7f218c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T08:53:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.485+0000 7f218aa00700 1 --2- 172.21.15.89:0/1544141000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f218c0d2370 0x7f218c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.485+0000 7f218aa00700 1 -- 172.21.15.89:0/1544141000 >> v1:172.21.15.113:6789/0 conn(0x7f218c15d9d0 legacy=0x7f218c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.485+0000 7f218aa00700 1 -- 172.21.15.89:0/1544141000 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c161700 con 0x7f218c0d16a0 2024-04-02T08:53:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.485+0000 7f218aa00700 1 --2- 172.21.15.89:0/1544141000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f218c0d16a0 0x7f218c0d1e30 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f2178006fd0 tx=0x7f2178009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7d214fc2384523 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.485+0000 7f218a000700 1 -- 172.21.15.89:0/1544141000 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2178005e30 con 0x7f218c0d16a0 2024-04-02T08:53:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.485+0000 7f218a000700 1 -- 172.21.15.89:0/1544141000 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2178004030 con 0x7f218c0d16a0 2024-04-02T08:53:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.485+0000 7f218a000700 1 -- 172.21.15.89:0/1544141000 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21780043e0 con 0x7f218c0d16a0 2024-04-02T08:53:40.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.486+0000 7f2192200700 1 -- 172.21.15.89:0/1544141000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f218c0d16a0 msgr2=0x7f218c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.486+0000 7f2192200700 1 --2- 172.21.15.89:0/1544141000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f218c0d16a0 0x7f218c0d1e30 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f2178006fd0 tx=0x7f2178009bd0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.486+0000 7f2192200700 1 -- 172.21.15.89:0/1544141000 shutdown_connections 2024-04-02T08:53:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.486+0000 7f2192200700 1 --2- 172.21.15.89:0/1544141000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f218c0d2370 0x7f218c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.486+0000 7f2192200700 1 --2- 172.21.15.89:0/1544141000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f218c0d16a0 0x7f218c0d1e30 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.486+0000 7f2192200700 1 -- 172.21.15.89:0/1544141000 >> 172.21.15.89:0/1544141000 conn(0x7f218c08aff0 msgr2=0x7f218c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.486+0000 7f2192200700 1 -- 172.21.15.89:0/1544141000 shutdown_connections 2024-04-02T08:53:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.486+0000 7f2192200700 1 -- 172.21.15.89:0/1544141000 wait complete. 2024-04-02T08:53:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.488+0000 7f2192200700 1 Processor -- start 2024-04-02T08:53:40.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.488+0000 7f2192200700 1 -- start start 2024-04-02T08:53:40.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f2192200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f218c0d16a0 0x7f218c0ca860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:40.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f2192200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f218c0d2370 0x7f218c0b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:40.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f2192200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f218c15d9d0 0x7f218c160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:40.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f2192200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0c9d10 con 0x7f218c0d2370 2024-04-02T08:53:40.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f2192200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c1475b0 con 0x7f218c15d9d0 2024-04-02T08:53:40.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f2192200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218c149a50 con 0x7f218c0d16a0 2024-04-02T08:53:40.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f218aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f218c0d2370 0x7f218c0b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:40.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f218aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f218c0d2370 0x7f218c0b7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60880/0 (socket says 172.21.15.89:60880) 2024-04-02T08:53:40.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f218aa00700 1 -- 172.21.15.89:0/565878225 learned_addr learned my addr 172.21.15.89:0/565878225 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T08:53:40.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f218b400700 1 --2- 172.21.15.89:0/565878225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f218c0d16a0 0x7f218c0ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:40.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f218be00700 1 --2- 172.21.15.89:0/565878225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f218c15d9d0 0x7f218c160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:40.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f218aa00700 1 -- 172.21.15.89:0/565878225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f218c0d16a0 msgr2=0x7f218c0ca860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:40.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f218aa00700 1 --2- 172.21.15.89:0/565878225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f218c0d16a0 0x7f218c0ca860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f218aa00700 1 -- 172.21.15.89:0/565878225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f218c15d9d0 msgr2=0x7f218c160350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:40.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f218aa00700 1 --2- 172.21.15.89:0/565878225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f218c15d9d0 0x7f218c160350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.489+0000 7f218aa00700 1 -- 172.21.15.89:0/565878225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2178009500 con 0x7f218c0d2370 2024-04-02T08:53:40.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.490+0000 7f218aa00700 1 --2- 172.21.15.89:0/565878225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f218c0d2370 0x7f218c0b7a80 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f2178004000 tx=0x7f2178004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:40.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.490+0000 7f2183400700 1 -- 172.21.15.89:0/565878225 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2178003d90 con 0x7f218c0d2370 2024-04-02T08:53:40.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.490+0000 7f2183400700 1 -- 172.21.15.89:0/565878225 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217801c420 con 0x7f218c0d2370 2024-04-02T08:53:40.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.490+0000 7f2192200700 1 -- 172.21.15.89:0/565878225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f218c0b89b0 con 0x7f218c0d2370 2024-04-02T08:53:40.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.490+0000 7f2183400700 1 -- 172.21.15.89:0/565878225 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217801c7b0 con 0x7f218c0d2370 2024-04-02T08:53:40.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.490+0000 7f2192200700 1 -- 172.21.15.89:0/565878225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f218c0b86d0 con 0x7f218c0d2370 2024-04-02T08:53:40.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.491+0000 7f2192200700 1 -- 172.21.15.89:0/565878225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f218c0d1ca0 con 0x7f218c0d2370 2024-04-02T08:53:40.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.492+0000 7f2183400700 1 -- 172.21.15.89:0/565878225 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 117) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f217801c910 con 0x7f218c0d2370 2024-04-02T08:53:40.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.493+0000 7f2183400700 1 --2- 172.21.15.89:0/565878225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f216c06ab30 0x7f216c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T08:53:40.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.493+0000 7f2183400700 1 -- 172.21.15.89:0/565878225 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f217808a400 con 0x7f218c0d2370 2024-04-02T08:53:40.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.493+0000 7f218b400700 1 --2- 172.21.15.89:0/565878225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f216c06ab30 0x7f216c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T08:53:40.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.494+0000 7f218b400700 1 --2- 172.21.15.89:0/565878225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f216c06ab30 0x7f216c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f217c000f60 tx=0x7f217c009040 comp rx=0 tx=0).ready entity=mgr.10221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T08:53:40.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.498+0000 7f2183400700 1 -- 172.21.15.89:0/565878225 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2178055550 con 0x7f218c0d2370 2024-04-02T08:53:40.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.730+0000 7f2192200700 1 -- 172.21.15.89:0/565878225 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f218c161fa0 con 0x7f216c06ab30 2024-04-02T08:53:40.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.782+0000 7f2183400700 1 -- 172.21.15.89:0/565878225 <== mgr.10221 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f217c0050d0 con 0x7f216c06ab30 2024-04-02T08:53:40.781 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule activated for path /snap_test_dir1/test_restart 2024-04-02T08:53:40.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.784+0000 7f2192200700 1 -- 172.21.15.89:0/565878225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f216c06ab30 msgr2=0x7f216c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:40.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.784+0000 7f2192200700 1 --2- 172.21.15.89:0/565878225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f216c06ab30 0x7f216c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f217c000f60 tx=0x7f217c009040 comp rx=0 tx=0).stop 2024-04-02T08:53:40.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.784+0000 7f2192200700 1 -- 172.21.15.89:0/565878225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f218c0d2370 msgr2=0x7f218c0b7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T08:53:40.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.784+0000 7f2192200700 1 --2- 172.21.15.89:0/565878225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f218c0d2370 0x7f218c0b7a80 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f2178004000 tx=0x7f2178004750 comp rx=0 tx=0).stop 2024-04-02T08:53:40.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.784+0000 7f2192200700 1 -- 172.21.15.89:0/565878225 shutdown_connections 2024-04-02T08:53:40.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.784+0000 7f2192200700 1 --2- 172.21.15.89:0/565878225 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f216c06ab30 0x7f216c06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.784+0000 7f2192200700 1 --2- 172.21.15.89:0/565878225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f218c15d9d0 0x7f218c160350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.784+0000 7f2192200700 1 --2- 172.21.15.89:0/565878225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f218c0d16a0 0x7f218c0ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.784+0000 7f2192200700 1 --2- 172.21.15.89:0/565878225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f218c0d2370 0x7f218c0b7a80 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T08:53:40.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.784+0000 7f2192200700 1 -- 172.21.15.89:0/565878225 >> 172.21.15.89:0/565878225 conn(0x7f218c08aff0 msgr2=0x7f218c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T08:53:40.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.784+0000 7f2192200700 1 -- 172.21.15.89:0/565878225 shutdown_connections 2024-04-02T08:53:40.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T08:53:40.784+0000 7f2192200700 1 -- 172.21.15.89:0/565878225 wait complete. 2024-04-02T08:53:40.802 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2024-04-02T08:54:00.794 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:54:00.794+0000 7ff342c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:54:00.794 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:54:00.794+0000 7ff342c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:54:00.795 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:54:00.795+0000 7ff342c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:54:00.795 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:54:00.795+0000 7ff342c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:54:00.795 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T08:54:00.795+0000 7ff342c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T08:54:06.370 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:54:06.375 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:54:36.479 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:54:36.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:55:06.933 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:55:06.938 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:55:37.053 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:55:37.058 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:56:07.168 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:56:07.173 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:56:37.315 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:56:37.321 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:57:07.438 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:57:07.444 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:57:37.551 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:57:37.557 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:58:07.664 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:58:07.670 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:58:37.779 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:58:37.785 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:59:07.945 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:59:07.950 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:59:38.082 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T08:59:38.087 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:00:08.285 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:00:08.290 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:00:38.614 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:00:38.620 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:01:08.942 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:01:08.947 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:01:39.071 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:01:39.077 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:02:09.256 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:02:09.261 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:02:39.621 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:02:39.626 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:03:09.723 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:03:09.729 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:03:39.834 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:03:39.840 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:04:09.954 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:04:09.960 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:04:15.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:04:15.804 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-04-02T09:04:15.856 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_54_00_UTC 2024-04-02T09:04:15.856 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_55_00_UTC 2024-04-02T09:04:15.856 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_56_00_UTC 2024-04-02T09:04:15.856 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_57_00_UTC 2024-04-02T09:04:15.856 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_58_00_UTC 2024-04-02T09:04:15.856 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_59_00_UTC 2024-04-02T09:04:15.856 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_00_00_UTC 2024-04-02T09:04:15.856 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_01_00_UTC 2024-04-02T09:04:15.856 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_02_00_UTC 2024-04-02T09:04:15.857 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_03_00_UTC 2024-04-02T09:04:15.857 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_04_00_UTC 2024-04-02T09:04:15.858 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-02-08_54_00_UTC', 'scheduled-2024-04-02-08_55_00_UTC', 'scheduled-2024-04-02-08_56_00_UTC', 'scheduled-2024-04-02-08_57_00_UTC', 'scheduled-2024-04-02-08_58_00_UTC', 'scheduled-2024-04-02-08_59_00_UTC', 'scheduled-2024-04-02-09_00_00_UTC', 'scheduled-2024-04-02-09_01_00_UTC', 'scheduled-2024-04-02-09_02_00_UTC', 'scheduled-2024-04-02-09_03_00_UTC', 'scheduled-2024-04-02-09_04_00_UTC'] 2024-04-02T09:04:15.858 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-04-02T09:04:16.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.055+0000 7f0166c00700 1 Processor -- start 2024-04-02T09:04:16.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.055+0000 7f0166c00700 1 -- start start 2024-04-02T09:04:16.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.057+0000 7f0166c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01601611b0 0x7f016015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:04:16.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.057+0000 7f0166c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f016015f4d0 0x7f01600ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:04:16.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.057+0000 7f0166c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01600ce270 con 0x7f01600d21f0 2024-04-02T09:04:16.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.057+0000 7f0166c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f016015b600 con 0x7f016015f4d0 2024-04-02T09:04:16.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.057+0000 7f0166c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01600d2760 con 0x7f01601611b0 2024-04-02T09:04:16.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.057+0000 7f015f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f016015f4d0 0x7f01600ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:04:16.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.057+0000 7f015f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f016015f4d0 0x7f01600ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58176/0 (socket says 172.21.15.89:58176) 2024-04-02T09:04:16.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.057+0000 7f015f400700 1 -- 172.21.15.89:0/508704866 learned_addr learned my addr 172.21.15.89:0/508704866 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:04:16.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.057+0000 7f015fe00700 1 --2- 172.21.15.89:0/508704866 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01601611b0 0x7f016015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:04:16.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.057+0000 7f015f400700 1 -- 172.21.15.89:0/508704866 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01601611b0 msgr2=0x7f016015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:04:16.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.057+0000 7f015f400700 1 --2- 172.21.15.89:0/508704866 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01601611b0 0x7f016015ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.057+0000 7f015f400700 1 -- 172.21.15.89:0/508704866 >> v1:172.21.15.113:6789/0 conn(0x7f01600d21f0 legacy=0x7f0160161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:04:16.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.057+0000 7f015f400700 1 -- 172.21.15.89:0/508704866 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01600d2a40 con 0x7f016015f4d0 2024-04-02T09:04:16.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.058+0000 7f015f400700 1 --2- 172.21.15.89:0/508704866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f016015f4d0 0x7f01600ce700 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f014c0098f0 tx=0x7f014c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b771c666403147 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:04:16.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.058+0000 7f015ea00700 1 -- 172.21.15.89:0/508704866 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c00b8a0 con 0x7f016015f4d0 2024-04-02T09:04:16.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.058+0000 7f015ea00700 1 -- 172.21.15.89:0/508704866 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c00ba00 con 0x7f016015f4d0 2024-04-02T09:04:16.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.058+0000 7f015ea00700 1 -- 172.21.15.89:0/508704866 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c00bd90 con 0x7f016015f4d0 2024-04-02T09:04:16.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.058+0000 7f0166c00700 1 -- 172.21.15.89:0/508704866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f016015f4d0 msgr2=0x7f01600ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:04:16.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.058+0000 7f0166c00700 1 --2- 172.21.15.89:0/508704866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f016015f4d0 0x7f01600ce700 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f014c0098f0 tx=0x7f014c009c00 comp rx=0 tx=0).stop 2024-04-02T09:04:16.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.058+0000 7f0166c00700 1 -- 172.21.15.89:0/508704866 shutdown_connections 2024-04-02T09:04:16.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.058+0000 7f0166c00700 1 --2- 172.21.15.89:0/508704866 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01601611b0 0x7f016015ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.058+0000 7f0166c00700 1 --2- 172.21.15.89:0/508704866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f016015f4d0 0x7f01600ce700 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.058+0000 7f0166c00700 1 -- 172.21.15.89:0/508704866 >> 172.21.15.89:0/508704866 conn(0x7f016008aff0 msgr2=0x7f01600d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:04:16.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.059+0000 7f0166c00700 1 -- 172.21.15.89:0/508704866 shutdown_connections 2024-04-02T09:04:16.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.059+0000 7f0166c00700 1 -- 172.21.15.89:0/508704866 wait complete. 2024-04-02T09:04:16.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.061+0000 7f0166c00700 1 Processor -- start 2024-04-02T09:04:16.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.061+0000 7f0166c00700 1 -- start start 2024-04-02T09:04:16.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.062+0000 7f0166c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01600d21f0 0x7f0160156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:04:16.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.062+0000 7f0166c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f016015f4d0 0x7f016006be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:04:16.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.062+0000 7f0166c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01601611b0 0x7f016006c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:04:16.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.062+0000 7f0166c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01600ca6a0 con 0x7f01601611b0 2024-04-02T09:04:16.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.062+0000 7f0166c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01600ca980 con 0x7f016015f4d0 2024-04-02T09:04:16.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.062+0000 7f0166c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01600cac60 con 0x7f01600d21f0 2024-04-02T09:04:16.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.063+0000 7f0164c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01601611b0 0x7f016006c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:04:16.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.063+0000 7f0164c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01601611b0 0x7f016006c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58186/0 (socket says 172.21.15.89:58186) 2024-04-02T09:04:16.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.063+0000 7f0164c00700 1 -- 172.21.15.89:0/1904547312 learned_addr learned my addr 172.21.15.89:0/1904547312 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:04:16.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.063+0000 7f015f400700 1 --2- 172.21.15.89:0/1904547312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f016015f4d0 0x7f016006be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:04:16.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.063+0000 7f015fe00700 1 --2- 172.21.15.89:0/1904547312 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01600d21f0 0x7f0160156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:04:16.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.063+0000 7f0164c00700 1 -- 172.21.15.89:0/1904547312 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01600d21f0 msgr2=0x7f0160156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:04:16.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.063+0000 7f0164c00700 1 --2- 172.21.15.89:0/1904547312 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01600d21f0 0x7f0160156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.063+0000 7f0164c00700 1 -- 172.21.15.89:0/1904547312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f016015f4d0 msgr2=0x7f016006be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:04:16.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.063+0000 7f0164c00700 1 --2- 172.21.15.89:0/1904547312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f016015f4d0 0x7f016006be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.063+0000 7f0164c00700 1 -- 172.21.15.89:0/1904547312 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c009550 con 0x7f01601611b0 2024-04-02T09:04:16.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.063+0000 7f0164c00700 1 --2- 172.21.15.89:0/1904547312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01601611b0 0x7f016006c3c0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f015800cc50 tx=0x7f015800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:04:16.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.063+0000 7f0157e00700 1 -- 172.21.15.89:0/1904547312 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015800ad80 con 0x7f01601611b0 2024-04-02T09:04:16.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.064+0000 7f0157e00700 1 -- 172.21.15.89:0/1904547312 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015800ce50 con 0x7f01601611b0 2024-04-02T09:04:16.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.064+0000 7f0157e00700 1 -- 172.21.15.89:0/1904547312 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0158003c40 con 0x7f01601611b0 2024-04-02T09:04:16.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.064+0000 7f0166c00700 1 -- 172.21.15.89:0/1904547312 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01600caf40 con 0x7f01601611b0 2024-04-02T09:04:16.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.064+0000 7f0166c00700 1 -- 172.21.15.89:0/1904547312 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01600b7a70 con 0x7f01601611b0 2024-04-02T09:04:16.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.065+0000 7f0166c00700 1 -- 172.21.15.89:0/1904547312 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01600cb130 con 0x7f01601611b0 2024-04-02T09:04:16.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.066+0000 7f0157e00700 1 -- 172.21.15.89:0/1904547312 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 121) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f01580050d0 con 0x7f01601611b0 2024-04-02T09:04:16.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.067+0000 7f0157e00700 1 --2- 172.21.15.89:0/1904547312 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f014006ae00 0x7f014006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:04:16.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.067+0000 7f0157e00700 1 -- 172.21.15.89:0/1904547312 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6531+0+0 (secure 0 0 0) 0x7f0158089a10 con 0x7f01601611b0 2024-04-02T09:04:16.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.067+0000 7f015fe00700 1 --2- 172.21.15.89:0/1904547312 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f014006ae00 0x7f014006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:04:16.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.068+0000 7f015fe00700 1 --2- 172.21.15.89:0/1904547312 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f014006ae00 0x7f014006d2c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f015000b9b0 tx=0x7f0150008040 comp rx=0 tx=0).ready entity=mgr.10221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:04:16.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.072+0000 7f0157e00700 1 -- 172.21.15.89:0/1904547312 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0158054700 con 0x7f01601611b0 2024-04-02T09:04:16.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.295+0000 7f0166c00700 1 -- 172.21.15.89:0/1904547312 --> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01600caaf0 con 0x7f014006ae00 2024-04-02T09:04:16.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.302+0000 7f0157e00700 1 -- 172.21.15.89:0/1904547312 <== mgr.10221 v2:172.21.15.89:6832/43357 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+375 (secure 0 0 0) 0x7f01600caaf0 con 0x7f014006ae00 2024-04-02T09:04:16.301 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:04:16.301 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T08:53:40", "first": "2024-04-02T08:54:00", "last": "2024-04-02T09:04:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-04-02T09:04:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.304+0000 7f0166c00700 1 -- 172.21.15.89:0/1904547312 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f014006ae00 msgr2=0x7f014006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:04:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.304+0000 7f0166c00700 1 --2- 172.21.15.89:0/1904547312 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f014006ae00 0x7f014006d2c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f015000b9b0 tx=0x7f0150008040 comp rx=0 tx=0).stop 2024-04-02T09:04:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.304+0000 7f0166c00700 1 -- 172.21.15.89:0/1904547312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01601611b0 msgr2=0x7f016006c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:04:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.304+0000 7f0166c00700 1 --2- 172.21.15.89:0/1904547312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01601611b0 0x7f016006c3c0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f015800cc50 tx=0x7f015800a340 comp rx=0 tx=0).stop 2024-04-02T09:04:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.304+0000 7f0166c00700 1 -- 172.21.15.89:0/1904547312 shutdown_connections 2024-04-02T09:04:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.304+0000 7f0166c00700 1 --2- 172.21.15.89:0/1904547312 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7f014006ae00 0x7f014006d2c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.304+0000 7f0166c00700 1 --2- 172.21.15.89:0/1904547312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01601611b0 0x7f016006c3c0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.304+0000 7f0166c00700 1 --2- 172.21.15.89:0/1904547312 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01600d21f0 0x7f0160156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.304+0000 7f0166c00700 1 --2- 172.21.15.89:0/1904547312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f016015f4d0 0x7f016006be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.304+0000 7f0166c00700 1 -- 172.21.15.89:0/1904547312 >> 172.21.15.89:0/1904547312 conn(0x7f016008aff0 msgr2=0x7f01600d1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:04:16.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.305+0000 7f0166c00700 1 -- 172.21.15.89:0/1904547312 shutdown_connections 2024-04-02T09:04:16.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.305+0000 7f0166c00700 1 -- 172.21.15.89:0/1904547312 wait complete. 2024-04-02T09:04:16.321 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T08:53:40", "first": "2024-04-02T08:54:00", "last": "2024-04-02T09:04:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-04-02T09:04:16.321 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-04-02T00:00:00', 'created': '2024-04-02T08:53:40', 'first': '2024-04-02T08:54:00', 'last': '2024-04-02T09:04:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2024-04-02T09:04:16.321 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-02T09:04:16.321 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-02T09:04:16.322 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-02T09:04:16.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.516+0000 7faa55e00700 1 Processor -- start 2024-04-02T09:04:16.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.516+0000 7faa55e00700 1 -- start start 2024-04-02T09:04:16.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.517+0000 7faa55e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faa50160c90 0x7faa500b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:04:16.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.518+0000 7faa55e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa500718e0 0x7faa500b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:04:16.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.518+0000 7faa55e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa500ac4d0 con 0x7faa500709c0 2024-04-02T09:04:16.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.518+0000 7faa55e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa50071e50 con 0x7faa50160c90 2024-04-02T09:04:16.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.518+0000 7faa55e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa50072130 con 0x7faa500718e0 2024-04-02T09:04:16.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.518+0000 7faa4ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faa50160c90 0x7faa500b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:04:16.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.518+0000 7faa4ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faa50160c90 0x7faa500b20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58200/0 (socket says 172.21.15.89:58200) 2024-04-02T09:04:16.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.518+0000 7faa4ea00700 1 -- 172.21.15.89:0/176579410 learned_addr learned my addr 172.21.15.89:0/176579410 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:04:16.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.518+0000 7faa4e000700 1 --2- 172.21.15.89:0/176579410 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa500718e0 0x7faa500b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:04:16.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.518+0000 7faa4ea00700 1 -- 172.21.15.89:0/176579410 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa500718e0 msgr2=0x7faa500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:04:16.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.518+0000 7faa4ea00700 1 --2- 172.21.15.89:0/176579410 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa500718e0 0x7faa500b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.518+0000 7faa4ea00700 1 -- 172.21.15.89:0/176579410 >> v1:172.21.15.113:6789/0 conn(0x7faa500709c0 legacy=0x7faa5006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:04:16.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.518+0000 7faa4ea00700 1 -- 172.21.15.89:0/176579410 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa50072410 con 0x7faa50160c90 2024-04-02T09:04:16.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.519+0000 7faa4ea00700 1 --2- 172.21.15.89:0/176579410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa50160c90 0x7faa500b20e0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7faa400098f0 tx=0x7faa40009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae9cab3924ea69c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:04:16.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.519+0000 7faa4d600700 1 -- 172.21.15.89:0/176579410 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4000b8a0 con 0x7faa50160c90 2024-04-02T09:04:16.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.519+0000 7faa4d600700 1 -- 172.21.15.89:0/176579410 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4000ba00 con 0x7faa50160c90 2024-04-02T09:04:16.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.519+0000 7faa4d600700 1 -- 172.21.15.89:0/176579410 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4000bd90 con 0x7faa50160c90 2024-04-02T09:04:16.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.519+0000 7faa55e00700 1 -- 172.21.15.89:0/176579410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa50160c90 msgr2=0x7faa500b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:04:16.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.519+0000 7faa55e00700 1 --2- 172.21.15.89:0/176579410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa50160c90 0x7faa500b20e0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7faa400098f0 tx=0x7faa40009c00 comp rx=0 tx=0).stop 2024-04-02T09:04:16.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.519+0000 7faa55e00700 1 -- 172.21.15.89:0/176579410 shutdown_connections 2024-04-02T09:04:16.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.519+0000 7faa55e00700 1 --2- 172.21.15.89:0/176579410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa50160c90 0x7faa500b20e0 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.519+0000 7faa55e00700 1 --2- 172.21.15.89:0/176579410 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa500718e0 0x7faa500b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.519+0000 7faa55e00700 1 -- 172.21.15.89:0/176579410 >> 172.21.15.89:0/176579410 conn(0x7faa500cf190 msgr2=0x7faa500cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:04:16.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.519+0000 7faa55e00700 1 -- 172.21.15.89:0/176579410 shutdown_connections 2024-04-02T09:04:16.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.520+0000 7faa55e00700 1 -- 172.21.15.89:0/176579410 wait complete. 2024-04-02T09:04:16.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.521+0000 7faa55e00700 1 Processor -- start 2024-04-02T09:04:16.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.521+0000 7faa55e00700 1 -- start start 2024-04-02T09:04:16.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.522+0000 7faa55e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa500709c0 0x7faa5014f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:04:16.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.522+0000 7faa55e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa500718e0 0x7faa500c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:04:16.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.522+0000 7faa55e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa50160c90 0x7faa500c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:04:16.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.522+0000 7faa55e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa501f3d60 con 0x7faa500709c0 2024-04-02T09:04:16.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.522+0000 7faa55e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa501f4040 con 0x7faa50160c90 2024-04-02T09:04:16.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.522+0000 7faa55e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa501f4320 con 0x7faa500718e0 2024-04-02T09:04:16.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.522+0000 7faa4ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa500709c0 0x7faa5014f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:04:16.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.522+0000 7faa4ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa500709c0 0x7faa5014f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58216/0 (socket says 172.21.15.89:58216) 2024-04-02T09:04:16.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.522+0000 7faa4ea00700 1 -- 172.21.15.89:0/971579462 learned_addr learned my addr 172.21.15.89:0/971579462 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:04:16.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.523+0000 7faa4e000700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa500718e0 0x7faa500c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:04:16.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.523+0000 7faa4f400700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa50160c90 0x7faa500c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:04:16.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.523+0000 7faa4ea00700 1 -- 172.21.15.89:0/971579462 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa500718e0 msgr2=0x7faa500c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:04:16.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.523+0000 7faa4ea00700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa500718e0 0x7faa500c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.523+0000 7faa4ea00700 1 -- 172.21.15.89:0/971579462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa50160c90 msgr2=0x7faa500c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:04:16.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.523+0000 7faa4ea00700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa50160c90 0x7faa500c89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.523+0000 7faa4ea00700 1 -- 172.21.15.89:0/971579462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa40009550 con 0x7faa500709c0 2024-04-02T09:04:16.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.523+0000 7faa4f400700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa50160c90 0x7faa500c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:04:16.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.523+0000 7faa4ea00700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa500709c0 0x7faa5014f8c0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7faa40009c40 tx=0x7faa40004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:04:16.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.523+0000 7faa47400700 1 -- 172.21.15.89:0/971579462 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa40004590 con 0x7faa500709c0 2024-04-02T09:04:16.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.523+0000 7faa47400700 1 -- 172.21.15.89:0/971579462 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4001e070 con 0x7faa500709c0 2024-04-02T09:04:16.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.523+0000 7faa47400700 1 -- 172.21.15.89:0/971579462 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa400038e0 con 0x7faa500709c0 2024-04-02T09:04:16.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.523+0000 7faa55e00700 1 -- 172.21.15.89:0/971579462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa501f4600 con 0x7faa500709c0 2024-04-02T09:04:16.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.524+0000 7faa55e00700 1 -- 172.21.15.89:0/971579462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa5014fe00 con 0x7faa500709c0 2024-04-02T09:04:16.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.525+0000 7faa55e00700 1 -- 172.21.15.89:0/971579462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa5014ff90 con 0x7faa500709c0 2024-04-02T09:04:16.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.525+0000 7faa47400700 1 -- 172.21.15.89:0/971579462 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 121) v1 ==== 89268+0+0 (secure 0 0 0) 0x7faa40003a40 con 0x7faa500709c0 2024-04-02T09:04:16.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.526+0000 7faa47400700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faa3006ae00 0x7faa3006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:04:16.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.526+0000 7faa47400700 1 -- 172.21.15.89:0/971579462 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6531+0+0 (secure 0 0 0) 0x7faa40020070 con 0x7faa500709c0 2024-04-02T09:04:16.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.527+0000 7faa4e000700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faa3006ae00 0x7faa3006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:04:16.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.527+0000 7faa4e000700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faa3006ae00 0x7faa3006d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faa3c006b20 tx=0x7faa3c008040 comp rx=0 tx=0).ready entity=mgr.10221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:04:16.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.531+0000 7faa47400700 1 -- 172.21.15.89:0/971579462 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faa40055a30 con 0x7faa500709c0 2024-04-02T09:04:16.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.787+0000 7faa55e00700 1 -- 172.21.15.89:0/971579462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7faa501f4b20 con 0x7faa500709c0 2024-04-02T09:04:16.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.790+0000 7faa47400700 1 -- 172.21.15.89:0/971579462 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v121) v1 ==== 74+0+171136 (secure 0 0 0) 0x7faa40015090 con 0x7faa500709c0 2024-04-02T09:04:16.793 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:04:16.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.794+0000 7faa55e00700 1 -- 172.21.15.89:0/971579462 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faa3006ae00 msgr2=0x7faa3006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:04:16.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.794+0000 7faa55e00700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faa3006ae00 0x7faa3006d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faa3c006b20 tx=0x7faa3c008040 comp rx=0 tx=0).stop 2024-04-02T09:04:16.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.794+0000 7faa55e00700 1 -- 172.21.15.89:0/971579462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa500709c0 msgr2=0x7faa5014f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:04:16.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.794+0000 7faa55e00700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa500709c0 0x7faa5014f8c0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7faa40009c40 tx=0x7faa40004750 comp rx=0 tx=0).stop 2024-04-02T09:04:16.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.794+0000 7faa55e00700 1 -- 172.21.15.89:0/971579462 shutdown_connections 2024-04-02T09:04:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.794+0000 7faa55e00700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.89:6832/43357,v1:172.21.15.89:6833/43357] conn(0x7faa3006ae00 0x7faa3006d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.794+0000 7faa55e00700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa50160c90 0x7faa500c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.794+0000 7faa55e00700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa500709c0 0x7faa5014f8c0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.794+0000 7faa55e00700 1 --2- 172.21.15.89:0/971579462 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa500718e0 0x7faa500c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:04:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.794+0000 7faa55e00700 1 -- 172.21.15.89:0/971579462 >> 172.21.15.89:0/971579462 conn(0x7faa500cf190 msgr2=0x7faa50099570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:04:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.794+0000 7faa55e00700 1 -- 172.21.15.89:0/971579462 shutdown_connections 2024-04-02T09:04:16.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:04:16.794+0000 7faa55e00700 1 -- 172.21.15.89:0/971579462 wait complete. 2024-04-02T09:04:16.810 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":121,"active_gid":10221,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":43357},{"type":"v1","addr":"172.21.15.89:6833","nonce":43357}]},"active_addr":"172.21.15.89:6833/43357","active_change":"2024-04-02T08:53:39.182404+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":9454,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":187,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":3070765417}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":4284766432}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":3350462335}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":2003321918}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":4154199909}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":3938746027}]}]} 2024-04-02T09:04:16.815 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: y 2024-04-02T09:04:16.815 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-02T09:04:16.815 INFO:tasks.ceph.mgr.y:Stopping old one... 2024-04-02T09:04:16.815 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-02T09:04:16.815 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:04:16.892 INFO:tasks.ceph.mgr.y:Stopped 2024-04-02T09:04:16.893 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-02T09:04:16.897 INFO:tasks.ceph.mgr.y:Started 2024-04-02T09:04:17.158 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:17.160+0000 7f1dc249b240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:04:17.342 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:17.344+0000 7f1dc249b240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:04:17.659 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:17.660+0000 7f1dc249b240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:04:18.180 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:18.182+0000 7f1dc249b240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:04:18.438 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:18.440+0000 7f1dc249b240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:04:18.894 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:18.896+0000 7f1dc249b240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:04:19.605 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:19.606+0000 7f1dc249b240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:04:19.696 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:19.698+0000 7f1dc249b240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:04:19.853 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:19.855+0000 7f1dc249b240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:04:20.066 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:20.068+0000 7f1dc249b240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:04:20.159 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:20.161+0000 7f1dc249b240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:04:20.258 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:20.259+0000 7f1dc249b240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:04:20.356 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:20.357+0000 7f1dc249b240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:04:20.455 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:20.456+0000 7f1dc249b240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:04:20.668 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:20.669+0000 7f1dc249b240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:04:20.924 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:20.926+0000 7f1dc249b240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:04:21.116 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:21.117+0000 7f1dc249b240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:04:21.208 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:21.210+0000 7f1dc249b240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:04:21.938 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:21.940+0000 7f1dc249b240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:04:22.154 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:22.156+0000 7f1dc249b240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:04:22.366 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:22.367+0000 7f1dc249b240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:04:22.844 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:22.846+0000 7f1dc249b240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:04:23.054 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:23.056+0000 7f1dc249b240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:04:23.497 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:23.499+0000 7f1dc249b240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:04:23.610 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:23.612+0000 7f1dc249b240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:04:23.791 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:23.793+0000 7f1d2de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:04:23.792 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:23.793+0000 7f1d2de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:04:23.792 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:23.793+0000 7f1d2de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:04:23.792 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:23.793+0000 7f1d2de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:04:23.792 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:23.793+0000 7f1d2de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:04:23.792 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:23.793+0000 7f1d2a200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:04:23.792 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:23.793+0000 7f1d2a200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:04:23.792 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:23.793+0000 7f1d2a200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:04:23.792 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:23.793+0000 7f1d2a200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:04:23.792 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:04:23.793+0000 7f1d2a200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:04:40.065 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:04:40.072 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:05:00.787 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:05:00.788+0000 7f1d37e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:05:00.787 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:05:00.788+0000 7f1d37e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:05:00.788 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:05:00.788+0000 7f1d37e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:05:00.788 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:05:00.788+0000 7f1d37e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:05:00.788 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:05:00.788+0000 7f1d37e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:05:10.194 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:05:10.199 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:05:40.389 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:05:40.394 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:06:10.545 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:06:10.551 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:06:40.663 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:06:40.668 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:07:10.783 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:07:10.789 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:07:40.894 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:07:40.900 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:08:11.022 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:08:11.027 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:08:41.141 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:08:41.148 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:09:11.261 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:09:11.267 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:09:16.899 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2024-04-02T09:09:17.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.105+0000 7f1d2f600700 1 Processor -- start 2024-04-02T09:09:17.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.106+0000 7f1d2f600700 1 -- start start 2024-04-02T09:09:17.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.107+0000 7f1d2f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d28160cc0 0x7f1d28168730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:17.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.107+0000 7f1d2f600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2815f820 con 0x7f1d280ced80 2024-04-02T09:09:17.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.107+0000 7f1d2f600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d280d2450 con 0x7f1d2815dad0 2024-04-02T09:09:17.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.107+0000 7f1d2f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d280cfcc0 con 0x7f1d28160cc0 2024-04-02T09:09:17.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.107+0000 7f1d27e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1d280ced80 0x7f1d280d1d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33570/0 (socket says 172.21.15.89:33570) 2024-04-02T09:09:17.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.107+0000 7f1d27e00700 1 -- 172.21.15.89:0/679890983 learned_addr learned my addr 172.21.15.89:0/679890983 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:17.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.107+0000 7f1d2d600700 1 --2- 172.21.15.89:0/679890983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d28160cc0 0x7f1d28168730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:17.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.107+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450245593 0 0) 0x7f1d2815f820 con 0x7f1d280ced80 2024-04-02T09:09:17.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.107+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d100035b0 con 0x7f1d280ced80 2024-04-02T09:09:17.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777404652 0 0) 0x7f1d280d2450 con 0x7f1d2815dad0 2024-04-02T09:09:17.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d2815f820 con 0x7f1d2815dad0 2024-04-02T09:09:17.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4194140967 0 0) 0x7f1d100035b0 con 0x7f1d280ced80 2024-04-02T09:09:17.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d28160cc0 msgr2=0x7f1d28168730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:17.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 --2- 172.21.15.89:0/679890983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d28160cc0 0x7f1d28168730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 >> v1:172.21.15.113:6789/0 conn(0x7f1d2815dad0 legacy=0x7f1d280d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:17.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d280cf420 con 0x7f1d280ced80 2024-04-02T09:09:17.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f1d140015f0 con 0x7f1d280ced80 2024-04-02T09:09:17.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 >> v1:172.21.15.89:6789/0 conn(0x7f1d280ced80 legacy=0x7f1d280d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:17.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 --2- 172.21.15.89:0/679890983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d10003bc0 0x7f1d10003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:17.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 --2- 172.21.15.89:0/679890983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d10004850 0x7f1d10004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:17.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 --2- 172.21.15.89:0/679890983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d28160cc0 0x7f1d10005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:17.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d280cfcc0 con 0x7f1d10004850 2024-04-02T09:09:17.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1000bac0 con 0x7f1d10003bc0 2024-04-02T09:09:17.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1000bc80 con 0x7f1d28160cc0 2024-04-02T09:09:17.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d27e00700 1 --2- 172.21.15.89:0/679890983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d10004850 0x7f1d10004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:17.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d2cc00700 1 --2- 172.21.15.89:0/679890983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d10003bc0 0x7f1d10003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:17.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.108+0000 7f1d2d600700 1 --2- 172.21.15.89:0/679890983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d28160cc0 0x7f1d10005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:17.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.109+0000 7f1d27e00700 1 -- 172.21.15.89:0/679890983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d28160cc0 msgr2=0x7f1d10005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:17.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.109+0000 7f1d27e00700 1 --2- 172.21.15.89:0/679890983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d28160cc0 0x7f1d10005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.109+0000 7f1d27e00700 1 -- 172.21.15.89:0/679890983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d10003bc0 msgr2=0x7f1d10003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:17.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.109+0000 7f1d27e00700 1 --2- 172.21.15.89:0/679890983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d10003bc0 0x7f1d10003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.109+0000 7f1d27e00700 1 -- 172.21.15.89:0/679890983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d100035b0 con 0x7f1d10004850 2024-04-02T09:09:17.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.109+0000 7f1d2d600700 1 --2- 172.21.15.89:0/679890983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d28160cc0 0x7f1d10005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:17.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.109+0000 7f1d27e00700 1 --2- 172.21.15.89:0/679890983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d10004850 0x7f1d10004cd0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f1d14007fd0 tx=0x7f1d10005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4178924f9fe4934 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:17.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.109+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1400cd10 con 0x7f1d10004850 2024-04-02T09:09:17.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.109+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d140019b0 con 0x7f1d10004850 2024-04-02T09:09:17.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.109+0000 7f1d27400700 1 -- 172.21.15.89:0/679890983 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d14005ba0 con 0x7f1d10004850 2024-04-02T09:09:17.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.110+0000 7f1d2f600700 1 -- 172.21.15.89:0/679890983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d10004850 msgr2=0x7f1d10004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:17.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.110+0000 7f1d2f600700 1 --2- 172.21.15.89:0/679890983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d10004850 0x7f1d10004cd0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f1d14007fd0 tx=0x7f1d10005880 comp rx=0 tx=0).stop 2024-04-02T09:09:17.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.110+0000 7f1d2d600700 1 -- 172.21.15.89:0/679890983 reap_dead start 2024-04-02T09:09:17.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.110+0000 7f1d2f600700 1 -- 172.21.15.89:0/679890983 shutdown_connections 2024-04-02T09:09:17.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.110+0000 7f1d2f600700 1 -- 172.21.15.89:0/679890983 >> 172.21.15.89:0/679890983 conn(0x7f1d2808aff0 msgr2=0x7f1d280d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:17.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.110+0000 7f1d2f600700 1 -- 172.21.15.89:0/679890983 shutdown_connections 2024-04-02T09:09:17.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.110+0000 7f1d2f600700 1 -- 172.21.15.89:0/679890983 wait complete. 2024-04-02T09:09:17.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.112+0000 7f1d2f600700 1 Processor -- start 2024-04-02T09:09:17.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.112+0000 7f1d2f600700 1 -- start start 2024-04-02T09:09:17.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2f600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d10004850 0x7f1d280b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:17.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d28153e20 0x7f1d281542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:17.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2f600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d280c95c0 0x7f1d280c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:17.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2f600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d281547e0 con 0x7f1d280c95c0 2024-04-02T09:09:17.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2f600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d280b74d0 con 0x7f1d10004850 2024-04-02T09:09:17.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d280cb640 con 0x7f1d28153e20 2024-04-02T09:09:17.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2d600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d280c95c0 0x7f1d280c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:17.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2d600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d280c95c0 0x7f1d280c9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40442/0 (socket says 172.21.15.89:40442) 2024-04-02T09:09:17.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2d600700 1 -- 172.21.15.89:0/1479054353 learned_addr learned my addr 172.21.15.89:0/1479054353 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:17.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2cc00700 1 --2- 172.21.15.89:0/1479054353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d10004850 0x7f1d280b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:17.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d27e00700 1 --2- 172.21.15.89:0/1479054353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d28153e20 0x7f1d281542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:17.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2d600700 1 -- 172.21.15.89:0/1479054353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d28153e20 msgr2=0x7f1d281542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:17.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2d600700 1 --2- 172.21.15.89:0/1479054353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d28153e20 0x7f1d281542a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2d600700 1 -- 172.21.15.89:0/1479054353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d10004850 msgr2=0x7f1d280b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:17.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2d600700 1 --2- 172.21.15.89:0/1479054353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d10004850 0x7f1d280b6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.113+0000 7f1d2d600700 1 -- 172.21.15.89:0/1479054353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d280c9f80 con 0x7f1d280c95c0 2024-04-02T09:09:17.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.114+0000 7f1d2d600700 1 --2- 172.21.15.89:0/1479054353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d280c95c0 0x7f1d280c9a40 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f1d2808d4f0 tx=0x7f1d200104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:17.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.114+0000 7f1d24c00700 1 -- 172.21.15.89:0/1479054353 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2000e050 con 0x7f1d280c95c0 2024-04-02T09:09:17.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.114+0000 7f1d24c00700 1 -- 172.21.15.89:0/1479054353 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2000f040 con 0x7f1d280c95c0 2024-04-02T09:09:17.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.114+0000 7f1d24c00700 1 -- 172.21.15.89:0/1479054353 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d20010e90 con 0x7f1d280c95c0 2024-04-02T09:09:17.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.114+0000 7f1d2f600700 1 -- 172.21.15.89:0/1479054353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d280ca230 con 0x7f1d280c95c0 2024-04-02T09:09:17.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.114+0000 7f1d2f600700 1 -- 172.21.15.89:0/1479054353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d280cbbc0 con 0x7f1d280c95c0 2024-04-02T09:09:17.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.115+0000 7f1d2f600700 1 -- 172.21.15.89:0/1479054353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d280d0ca0 con 0x7f1d280c95c0 2024-04-02T09:09:17.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.116+0000 7f1d24c00700 1 -- 172.21.15.89:0/1479054353 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 127) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f1d20015440 con 0x7f1d280c95c0 2024-04-02T09:09:17.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.117+0000 7f1d24c00700 1 --2- 172.21.15.89:0/1479054353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1d0806ae00 0x7f1d0806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:17.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.117+0000 7f1d24c00700 1 -- 172.21.15.89:0/1479054353 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f1d20089c80 con 0x7f1d280c95c0 2024-04-02T09:09:17.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.117+0000 7f1d2cc00700 1 --2- 172.21.15.89:0/1479054353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1d0806ae00 0x7f1d0806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:17.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.118+0000 7f1d2cc00700 1 --2- 172.21.15.89:0/1479054353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1d0806ae00 0x7f1d0806d2c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c007fd0 tx=0x7f1d1c009040 comp rx=0 tx=0).ready entity=mgr.10311 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:17.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.121+0000 7f1d24c00700 1 -- 172.21.15.89:0/1479054353 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1d200547f0 con 0x7f1d280c95c0 2024-04-02T09:09:17.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.351+0000 7f1d2f600700 1 -- 172.21.15.89:0/1479054353 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d28068290 con 0x7f1d0806ae00 2024-04-02T09:09:17.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.412+0000 7f1d24c00700 1 -- 172.21.15.89:0/1479054353 <== mgr.10311 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f1d28068290 con 0x7f1d0806ae00 2024-04-02T09:09:17.411 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2024-04-02T09:09:17.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.414+0000 7f1d2f600700 1 -- 172.21.15.89:0/1479054353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1d0806ae00 msgr2=0x7f1d0806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:17.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.414+0000 7f1d2f600700 1 --2- 172.21.15.89:0/1479054353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1d0806ae00 0x7f1d0806d2c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c007fd0 tx=0x7f1d1c009040 comp rx=0 tx=0).stop 2024-04-02T09:09:17.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.414+0000 7f1d2f600700 1 -- 172.21.15.89:0/1479054353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d280c95c0 msgr2=0x7f1d280c9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:17.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.414+0000 7f1d2f600700 1 --2- 172.21.15.89:0/1479054353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d280c95c0 0x7f1d280c9a40 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f1d2808d4f0 tx=0x7f1d200104a0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.414+0000 7f1d2f600700 1 -- 172.21.15.89:0/1479054353 shutdown_connections 2024-04-02T09:09:17.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.414+0000 7f1d2f600700 1 --2- 172.21.15.89:0/1479054353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1d0806ae00 0x7f1d0806d2c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.414+0000 7f1d2f600700 1 --2- 172.21.15.89:0/1479054353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d10004850 0x7f1d280b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.414+0000 7f1d2f600700 1 --2- 172.21.15.89:0/1479054353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d280c95c0 0x7f1d280c9a40 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.414+0000 7f1d2f600700 1 --2- 172.21.15.89:0/1479054353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d28153e20 0x7f1d281542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.414+0000 7f1d2f600700 1 -- 172.21.15.89:0/1479054353 >> 172.21.15.89:0/1479054353 conn(0x7f1d2808aff0 msgr2=0x7f1d2815b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:17.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.415+0000 7f1d2f600700 1 -- 172.21.15.89:0/1479054353 shutdown_connections 2024-04-02T09:09:17.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.415+0000 7f1d2f600700 1 -- 172.21.15.89:0/1479054353 wait complete. 2024-04-02T09:09:17.431 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2024-04-02T09:09:17.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:17.432 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-04-02T09:09:17.476 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_54_00_UTC 2024-04-02T09:09:17.476 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_55_00_UTC 2024-04-02T09:09:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_56_00_UTC 2024-04-02T09:09:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_57_00_UTC 2024-04-02T09:09:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_58_00_UTC 2024-04-02T09:09:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_59_00_UTC 2024-04-02T09:09:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_00_00_UTC 2024-04-02T09:09:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_01_00_UTC 2024-04-02T09:09:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_02_00_UTC 2024-04-02T09:09:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_03_00_UTC 2024-04-02T09:09:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_04_00_UTC 2024-04-02T09:09:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_05_00_UTC 2024-04-02T09:09:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_06_00_UTC 2024-04-02T09:09:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_07_00_UTC 2024-04-02T09:09:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_08_00_UTC 2024-04-02T09:09:17.477 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_09_00_UTC 2024-04-02T09:09:17.478 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-02-08_54_00_UTC', 'scheduled-2024-04-02-08_55_00_UTC', 'scheduled-2024-04-02-08_56_00_UTC', 'scheduled-2024-04-02-08_57_00_UTC', 'scheduled-2024-04-02-08_58_00_UTC', 'scheduled-2024-04-02-08_59_00_UTC', 'scheduled-2024-04-02-09_00_00_UTC', 'scheduled-2024-04-02-09_01_00_UTC', 'scheduled-2024-04-02-09_02_00_UTC', 'scheduled-2024-04-02-09_03_00_UTC', 'scheduled-2024-04-02-09_04_00_UTC', 'scheduled-2024-04-02-09_05_00_UTC', 'scheduled-2024-04-02-09_06_00_UTC', 'scheduled-2024-04-02-09_07_00_UTC', 'scheduled-2024-04-02-09_08_00_UTC', 'scheduled-2024-04-02-09_09_00_UTC'] 2024-04-02T09:09:17.479 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-04-02T09:09:17.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.714+0000 7fb20b000700 1 Processor -- start 2024-04-02T09:09:17.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.715+0000 7fb20b000700 1 -- start start 2024-04-02T09:09:17.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.716+0000 7fb20b000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb2040cd5c0 0x7fb20415e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:17.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.716+0000 7fb20b000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20415eda0 con 0x7fb20406f310 2024-04-02T09:09:17.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.716+0000 7fb20b000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb204099dd0 con 0x7fb20415cfe0 2024-04-02T09:09:17.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.716+0000 7fb20b000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb20409a0b0 con 0x7fb2040cd5c0 2024-04-02T09:09:17.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.716+0000 7fb203e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb2040cd5c0 0x7fb20415e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:17.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.716+0000 7fb203e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb2040cd5c0 0x7fb20415e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40458/0 (socket says 172.21.15.89:40458) 2024-04-02T09:09:17.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.716+0000 7fb203e00700 1 -- 172.21.15.89:0/3817047746 learned_addr learned my addr 172.21.15.89:0/3817047746 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.716+0000 7fb203e00700 1 -- 172.21.15.89:0/3817047746 >> v1:172.21.15.113:6789/0 conn(0x7fb20415cfe0 legacy=0x7fb20415d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.716+0000 7fb203e00700 1 -- 172.21.15.89:0/3817047746 >> v1:172.21.15.89:6789/0 conn(0x7fb20406f310 legacy=0x7fb20406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.716+0000 7fb203e00700 1 -- 172.21.15.89:0/3817047746 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20409a390 con 0x7fb2040cd5c0 2024-04-02T09:09:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.717+0000 7fb203e00700 1 --2- 172.21.15.89:0/3817047746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2040cd5c0 0x7fb20415e860 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009920 tx=0x7fb1f4009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23181fcf7d4107c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.717+0000 7fb202a00700 1 -- 172.21.15.89:0/3817047746 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4004230 con 0x7fb2040cd5c0 2024-04-02T09:09:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.717+0000 7fb202a00700 1 -- 172.21.15.89:0/3817047746 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f4004390 con 0x7fb2040cd5c0 2024-04-02T09:09:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.717+0000 7fb202a00700 1 -- 172.21.15.89:0/3817047746 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4009e70 con 0x7fb2040cd5c0 2024-04-02T09:09:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.717+0000 7fb20b000700 1 -- 172.21.15.89:0/3817047746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2040cd5c0 msgr2=0x7fb20415e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:17.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.717+0000 7fb20b000700 1 --2- 172.21.15.89:0/3817047746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2040cd5c0 0x7fb20415e860 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009920 tx=0x7fb1f4009c30 comp rx=0 tx=0).stop 2024-04-02T09:09:17.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.718+0000 7fb20b000700 1 -- 172.21.15.89:0/3817047746 shutdown_connections 2024-04-02T09:09:17.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.718+0000 7fb20b000700 1 --2- 172.21.15.89:0/3817047746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2040cd5c0 0x7fb20415e860 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.718+0000 7fb20b000700 1 -- 172.21.15.89:0/3817047746 >> 172.21.15.89:0/3817047746 conn(0x7fb2040d1370 msgr2=0x7fb2040d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:17.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.718+0000 7fb20b000700 1 -- 172.21.15.89:0/3817047746 shutdown_connections 2024-04-02T09:09:17.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.718+0000 7fb20b000700 1 -- 172.21.15.89:0/3817047746 wait complete. 2024-04-02T09:09:17.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.719+0000 7fb20b000700 1 Processor -- start 2024-04-02T09:09:17.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.720+0000 7fb20b000700 1 -- start start 2024-04-02T09:09:17.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.720+0000 7fb20b000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb20406f310 0x7fb2040b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:17.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.720+0000 7fb20b000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2040cd5c0 0x7fb2040d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:17.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.720+0000 7fb20b000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb20415cfe0 0x7fb2040d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:17.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.720+0000 7fb20b000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20415b600 con 0x7fb20406f310 2024-04-02T09:09:17.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.720+0000 7fb20b000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2040b70a0 con 0x7fb20415cfe0 2024-04-02T09:09:17.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.720+0000 7fb20b000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2041f45c0 con 0x7fb2040cd5c0 2024-04-02T09:09:17.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.721+0000 7fb203e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb20406f310 0x7fb2040b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:17.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.721+0000 7fb203e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb20406f310 0x7fb2040b6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40464/0 (socket says 172.21.15.89:40464) 2024-04-02T09:09:17.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.721+0000 7fb203e00700 1 -- 172.21.15.89:0/3566794464 learned_addr learned my addr 172.21.15.89:0/3566794464 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:17.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.721+0000 7fb203400700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2040cd5c0 0x7fb2040d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.721+0000 7fb209000700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb20415cfe0 0x7fb2040d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.721+0000 7fb203e00700 1 -- 172.21.15.89:0/3566794464 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2040cd5c0 msgr2=0x7fb2040d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.721+0000 7fb203e00700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2040cd5c0 0x7fb2040d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.721+0000 7fb203e00700 1 -- 172.21.15.89:0/3566794464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb20415cfe0 msgr2=0x7fb2040d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.721+0000 7fb203e00700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb20415cfe0 0x7fb2040d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.721+0000 7fb203e00700 1 -- 172.21.15.89:0/3566794464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f4009550 con 0x7fb20406f310 2024-04-02T09:09:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.721+0000 7fb203400700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2040cd5c0 0x7fb2040d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.721+0000 7fb209000700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb20415cfe0 0x7fb2040d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.721+0000 7fb203e00700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb20406f310 0x7fb2040b6b60 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fb204072450 tx=0x7fb1f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.722+0000 7fb1fbe00700 1 -- 172.21.15.89:0/3566794464 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f400bda0 con 0x7fb20406f310 2024-04-02T09:09:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.722+0000 7fb1fbe00700 1 -- 172.21.15.89:0/3566794464 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f401f070 con 0x7fb20406f310 2024-04-02T09:09:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.722+0000 7fb20b000700 1 -- 172.21.15.89:0/3566794464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2041f48a0 con 0x7fb20406f310 2024-04-02T09:09:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.722+0000 7fb1fbe00700 1 -- 172.21.15.89:0/3566794464 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f40038e0 con 0x7fb20406f310 2024-04-02T09:09:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.722+0000 7fb20b000700 1 -- 172.21.15.89:0/3566794464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2040b6360 con 0x7fb20406f310 2024-04-02T09:09:17.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.723+0000 7fb20b000700 1 -- 172.21.15.89:0/3566794464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20406d820 con 0x7fb20406f310 2024-04-02T09:09:17.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.724+0000 7fb1fbe00700 1 -- 172.21.15.89:0/3566794464 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 127) v1 ==== 89268+0+0 (secure 0 0 0) 0x7fb1f4021030 con 0x7fb20406f310 2024-04-02T09:09:17.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.724+0000 7fb1fbe00700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb1e406ae00 0x7fb1e406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:17.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.725+0000 7fb1fbe00700 1 -- 172.21.15.89:0/3566794464 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6789+0+0 (secure 0 0 0) 0x7fb1f408b1d0 con 0x7fb20406f310 2024-04-02T09:09:17.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.725+0000 7fb203400700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb1e406ae00 0x7fb1e406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:17.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.725+0000 7fb203400700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb1e406ae00 0x7fb1e406d2c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb2040ccc60 tx=0x7fb1f0009040 comp rx=0 tx=0).ready entity=mgr.10311 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:17.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.729+0000 7fb1fbe00700 1 -- 172.21.15.89:0/3566794464 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb1f4055dc0 con 0x7fb20406f310 2024-04-02T09:09:17.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.955+0000 7fb20b000700 1 -- 172.21.15.89:0/3566794464 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb204099a60 con 0x7fb1e406ae00 2024-04-02T09:09:17.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.962+0000 7fb1fbe00700 1 -- 172.21.15.89:0/3566794464 <== mgr.10311 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+376 (secure 0 0 0) 0x7fb1f00013d0 con 0x7fb1e406ae00 2024-04-02T09:09:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:17.960 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T08:53:40", "first": "2024-04-02T08:54:00", "last": "2024-04-02T09:09:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-04-02T09:09:17.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.964+0000 7fb20b000700 1 -- 172.21.15.89:0/3566794464 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb1e406ae00 msgr2=0x7fb1e406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:17.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.964+0000 7fb20b000700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb1e406ae00 0x7fb1e406d2c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb2040ccc60 tx=0x7fb1f0009040 comp rx=0 tx=0).stop 2024-04-02T09:09:17.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.964+0000 7fb20b000700 1 -- 172.21.15.89:0/3566794464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb20406f310 msgr2=0x7fb2040b6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:17.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.964+0000 7fb20b000700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb20406f310 0x7fb2040b6b60 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fb204072450 tx=0x7fb1f4004750 comp rx=0 tx=0).stop 2024-04-02T09:09:17.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.964+0000 7fb20b000700 1 -- 172.21.15.89:0/3566794464 shutdown_connections 2024-04-02T09:09:17.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.964+0000 7fb20b000700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb1e406ae00 0x7fb1e406d2c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.964+0000 7fb20b000700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb20415cfe0 0x7fb2040d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.964+0000 7fb20b000700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb20406f310 0x7fb2040b6b60 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.964+0000 7fb20b000700 1 --2- 172.21.15.89:0/3566794464 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2040cd5c0 0x7fb2040d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:17.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.964+0000 7fb20b000700 1 -- 172.21.15.89:0/3566794464 >> 172.21.15.89:0/3566794464 conn(0x7fb2040d1370 msgr2=0x7fb20415c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:17.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.965+0000 7fb20b000700 1 -- 172.21.15.89:0/3566794464 shutdown_connections 2024-04-02T09:09:17.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:17.965+0000 7fb20b000700 1 -- 172.21.15.89:0/3566794464 wait complete. 2024-04-02T09:09:17.981 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T08:53:40", "first": "2024-04-02T08:54:00", "last": "2024-04-02T09:09:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-04-02T09:09:17.981 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-04-02T00:00:00', 'created': '2024-04-02T08:53:40', 'first': '2024-04-02T08:54:00', 'last': '2024-04-02T09:09:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2024-04-02T09:09:17.982 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2024-04-02T09:09:17.982 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2024-04-02T09:09:17.986 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2024-04-02T09:09:18.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.182+0000 7f49ae800700 1 Processor -- start 2024-04-02T09:09:18.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.183+0000 7f49ae800700 1 -- start start 2024-04-02T09:09:18.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49ae800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f49a815d800 0x7f49a80725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:18.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49ae800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49a80ce230 0x7f49a80ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:18.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49ae800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49a80b2070 0x7f49a80b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:18.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49ae800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a80b40f0 con 0x7f49a80b2070 2024-04-02T09:09:18.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49ae800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a80709c0 con 0x7f49a815d800 2024-04-02T09:09:18.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49ae800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a8070c80 con 0x7f49a80ce230 2024-04-02T09:09:18.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49a7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49a80b2070 0x7f49a80b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:18.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49a7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49a80b2070 0x7f49a80b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40478/0 (socket says 172.21.15.89:40478) 2024-04-02T09:09:18.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49a7e00700 1 -- 172.21.15.89:0/1750370212 learned_addr learned my addr 172.21.15.89:0/1750370212 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49a6a00700 1 --2- 172.21.15.89:0/1750370212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49a80ce230 0x7f49a80ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49a7400700 1 --2- 172.21.15.89:0/1750370212 >> v2:172.21.15.113:3300/0 conn(0x7f49a815d800 0x7f49a80725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49a7e00700 1 -- 172.21.15.89:0/1750370212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49a80ce230 msgr2=0x7f49a80ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49a7e00700 1 --2- 172.21.15.89:0/1750370212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49a80ce230 0x7f49a80ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49a7e00700 1 -- 172.21.15.89:0/1750370212 >> v2:172.21.15.113:3300/0 conn(0x7f49a815d800 msgr2=0x7f49a80725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49a7e00700 1 --2- 172.21.15.89:0/1750370212 >> v2:172.21.15.113:3300/0 conn(0x7f49a815d800 0x7f49a80725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.184+0000 7f49a7e00700 1 -- 172.21.15.89:0/1750370212 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a8070f60 con 0x7f49a80b2070 2024-04-02T09:09:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.185+0000 7f49a7e00700 1 --2- 172.21.15.89:0/1750370212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a80b2070 0x7f49a80b2500 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f49a806c100 tx=0x7f49a000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b9abdb769ebee6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.185+0000 7f49a6000700 1 -- 172.21.15.89:0/1750370212 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a0010070 con 0x7f49a80b2070 2024-04-02T09:09:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.185+0000 7f49a6000700 1 -- 172.21.15.89:0/1750370212 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a000dd90 con 0x7f49a80b2070 2024-04-02T09:09:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.185+0000 7f49a6000700 1 -- 172.21.15.89:0/1750370212 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a000b310 con 0x7f49a80b2070 2024-04-02T09:09:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.185+0000 7f49ae800700 1 -- 172.21.15.89:0/1750370212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a80b2070 msgr2=0x7f49a80b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.185+0000 7f49ae800700 1 --2- 172.21.15.89:0/1750370212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a80b2070 0x7f49a80b2500 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f49a806c100 tx=0x7f49a000d430 comp rx=0 tx=0).stop 2024-04-02T09:09:18.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.185+0000 7f49ae800700 1 -- 172.21.15.89:0/1750370212 shutdown_connections 2024-04-02T09:09:18.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.186+0000 7f49ae800700 1 --2- 172.21.15.89:0/1750370212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a80b2070 0x7f49a80b2500 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:18.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.186+0000 7f49ae800700 1 --2- 172.21.15.89:0/1750370212 >> v2:172.21.15.113:3300/0 conn(0x7f49a815d800 0x7f49a80725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:18.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.186+0000 7f49ae800700 1 --2- 172.21.15.89:0/1750370212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49a80ce230 0x7f49a80ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:18.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.186+0000 7f49ae800700 1 -- 172.21.15.89:0/1750370212 >> 172.21.15.89:0/1750370212 conn(0x7f49a80b0d70 msgr2=0x7f49a80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:18.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.186+0000 7f49ae800700 1 -- 172.21.15.89:0/1750370212 shutdown_connections 2024-04-02T09:09:18.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.186+0000 7f49ae800700 1 -- 172.21.15.89:0/1750370212 wait complete. 2024-04-02T09:09:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.187+0000 7f49ae800700 1 Processor -- start 2024-04-02T09:09:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.187+0000 7f49ae800700 1 -- start start 2024-04-02T09:09:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.188+0000 7f49ae800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49a80b2070 0x7f49a8098a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.188+0000 7f49ae800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a80ce230 0x7f49a8154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.188+0000 7f49ae800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49a815d800 0x7f49a815c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.188+0000 7f49ae800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a8098f60 con 0x7f49a80ce230 2024-04-02T09:09:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.188+0000 7f49ae800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a8156690 con 0x7f49a80b2070 2024-04-02T09:09:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.188+0000 7f49ae800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a8155a80 con 0x7f49a815d800 2024-04-02T09:09:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.189+0000 7f49a6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a80ce230 0x7f49a8154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.189+0000 7f49a6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a80ce230 0x7f49a8154fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40484/0 (socket says 172.21.15.89:40484) 2024-04-02T09:09:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.189+0000 7f49a6a00700 1 -- 172.21.15.89:0/1523878793 learned_addr learned my addr 172.21.15.89:0/1523878793 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.189+0000 7f49a7400700 1 --2- 172.21.15.89:0/1523878793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49a80b2070 0x7f49a8098a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.189+0000 7f49a7e00700 1 --2- 172.21.15.89:0/1523878793 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49a815d800 0x7f49a815c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.189+0000 7f49a6a00700 1 -- 172.21.15.89:0/1523878793 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49a815d800 msgr2=0x7f49a815c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.189+0000 7f49a6a00700 1 --2- 172.21.15.89:0/1523878793 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49a815d800 0x7f49a815c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.189+0000 7f49a6a00700 1 -- 172.21.15.89:0/1523878793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49a80b2070 msgr2=0x7f49a8098a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.189+0000 7f49a6a00700 1 --2- 172.21.15.89:0/1523878793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49a80b2070 0x7f49a8098a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.189+0000 7f49a6a00700 1 -- 172.21.15.89:0/1523878793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a000bd40 con 0x7f49a80ce230 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.189+0000 7f49a6a00700 1 --2- 172.21.15.89:0/1523878793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a80ce230 0x7f49a8154fc0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f499400e810 tx=0x7f499400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.189+0000 7f499f400700 1 -- 172.21.15.89:0/1523878793 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499400c070 con 0x7f49a80ce230 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.190+0000 7f499f400700 1 -- 172.21.15.89:0/1523878793 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4994013610 con 0x7f49a80ce230 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.190+0000 7f499f400700 1 -- 172.21.15.89:0/1523878793 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49940139a0 con 0x7f49a80ce230 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.190+0000 7f49ae800700 1 -- 172.21.15.89:0/1523878793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a8155d60 con 0x7f49a80ce230 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.190+0000 7f49ae800700 1 -- 172.21.15.89:0/1523878793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a8156320 con 0x7f49a80ce230 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.191+0000 7f49ae800700 1 -- 172.21.15.89:0/1523878793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a81564e0 con 0x7f49a80ce230 2024-04-02T09:09:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.192+0000 7f499f400700 1 -- 172.21.15.89:0/1523878793 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 127) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f4994013b00 con 0x7f49a80ce230 2024-04-02T09:09:18.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.192+0000 7f499f400700 1 --2- 172.21.15.89:0/1523878793 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f498806ae30 0x7f498806d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:18.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.193+0000 7f499f400700 1 -- 172.21.15.89:0/1523878793 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f4994089840 con 0x7f49a80ce230 2024-04-02T09:09:18.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.193+0000 7f49a7400700 1 --2- 172.21.15.89:0/1523878793 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f498806ae30 0x7f498806d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:18.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.193+0000 7f49a7400700 1 --2- 172.21.15.89:0/1523878793 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f498806ae30 0x7f498806d2f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4998006b20 tx=0x7f4998008040 comp rx=0 tx=0).ready entity=mgr.10311 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:18.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.197+0000 7f499f400700 1 -- 172.21.15.89:0/1523878793 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4994054290 con 0x7f49a80ce230 2024-04-02T09:09:18.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.424+0000 7f49ae800700 1 -- 172.21.15.89:0/1523878793 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f49a8072b00 con 0x7f498806ae30 2024-04-02T09:09:18.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.492+0000 7f499f400700 1 -- 172.21.15.89:0/1523878793 <== mgr.10311 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f4998009e40 con 0x7f498806ae30 2024-04-02T09:09:18.491 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path /snap_test_dir1/test_restart 2024-04-02T09:09:18.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.494+0000 7f49ae800700 1 -- 172.21.15.89:0/1523878793 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f498806ae30 msgr2=0x7f498806d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:18.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.494+0000 7f49ae800700 1 --2- 172.21.15.89:0/1523878793 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f498806ae30 0x7f498806d2f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4998006b20 tx=0x7f4998008040 comp rx=0 tx=0).stop 2024-04-02T09:09:18.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.494+0000 7f49ae800700 1 -- 172.21.15.89:0/1523878793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a80ce230 msgr2=0x7f49a8154fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:18.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.494+0000 7f49ae800700 1 --2- 172.21.15.89:0/1523878793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a80ce230 0x7f49a8154fc0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f499400e810 tx=0x7f499400eb20 comp rx=0 tx=0).stop 2024-04-02T09:09:18.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.494+0000 7f49ae800700 1 -- 172.21.15.89:0/1523878793 shutdown_connections 2024-04-02T09:09:18.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.494+0000 7f49ae800700 1 --2- 172.21.15.89:0/1523878793 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f498806ae30 0x7f498806d2f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:18.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.494+0000 7f49ae800700 1 --2- 172.21.15.89:0/1523878793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49a80b2070 0x7f49a8098a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:18.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.494+0000 7f49ae800700 1 --2- 172.21.15.89:0/1523878793 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49a815d800 0x7f49a815c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:18.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.494+0000 7f49ae800700 1 --2- 172.21.15.89:0/1523878793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49a80ce230 0x7f49a8154fc0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:18.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.494+0000 7f49ae800700 1 -- 172.21.15.89:0/1523878793 >> 172.21.15.89:0/1523878793 conn(0x7f49a80b0d70 msgr2=0x7f49a80af660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:18.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.495+0000 7f49ae800700 1 -- 172.21.15.89:0/1523878793 shutdown_connections 2024-04-02T09:09:18.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:18.495+0000 7f49ae800700 1 -- 172.21.15.89:0/1523878793 wait complete. 2024-04-02T09:09:18.511 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2024-04-02T09:09:18.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:18.511 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-04-02T09:09:18.556 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_54_00_UTC 2024-04-02T09:09:18.556 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_55_00_UTC 2024-04-02T09:09:18.557 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_56_00_UTC 2024-04-02T09:09:18.557 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_57_00_UTC 2024-04-02T09:09:18.557 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_58_00_UTC 2024-04-02T09:09:18.557 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-08_59_00_UTC 2024-04-02T09:09:18.557 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_00_00_UTC 2024-04-02T09:09:18.557 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_01_00_UTC 2024-04-02T09:09:18.557 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_02_00_UTC 2024-04-02T09:09:18.557 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_03_00_UTC 2024-04-02T09:09:18.557 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_04_00_UTC 2024-04-02T09:09:18.558 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_05_00_UTC 2024-04-02T09:09:18.558 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_06_00_UTC 2024-04-02T09:09:18.558 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_07_00_UTC 2024-04-02T09:09:18.558 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_08_00_UTC 2024-04-02T09:09:18.558 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_09_00_UTC 2024-04-02T09:09:18.559 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-08_54_00_UTC 2024-04-02T09:09:18.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:18.559 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-08_54_00_UTC) 2024-04-02T09:09:18.649 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-08_55_00_UTC 2024-04-02T09:09:18.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:18.650 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-08_55_00_UTC) 2024-04-02T09:09:18.699 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-08_56_00_UTC 2024-04-02T09:09:18.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:18.699 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-08_56_00_UTC) 2024-04-02T09:09:18.749 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-08_57_00_UTC 2024-04-02T09:09:18.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:18.749 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-08_57_00_UTC) 2024-04-02T09:09:18.802 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-08_58_00_UTC 2024-04-02T09:09:18.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:18.802 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-08_58_00_UTC) 2024-04-02T09:09:18.852 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-08_59_00_UTC 2024-04-02T09:09:18.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:18.852 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-08_59_00_UTC) 2024-04-02T09:09:18.902 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_00_00_UTC 2024-04-02T09:09:18.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:18.903 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_00_00_UTC) 2024-04-02T09:09:18.952 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_01_00_UTC 2024-04-02T09:09:18.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:18.953 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_01_00_UTC) 2024-04-02T09:09:19.006 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_02_00_UTC 2024-04-02T09:09:19.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:19.007 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_02_00_UTC) 2024-04-02T09:09:19.056 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_03_00_UTC 2024-04-02T09:09:19.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:19.056 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_03_00_UTC) 2024-04-02T09:09:19.105 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_04_00_UTC 2024-04-02T09:09:19.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:19.106 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_04_00_UTC) 2024-04-02T09:09:19.155 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_05_00_UTC 2024-04-02T09:09:19.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:19.156 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_05_00_UTC) 2024-04-02T09:09:19.206 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_06_00_UTC 2024-04-02T09:09:19.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:19.206 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_06_00_UTC) 2024-04-02T09:09:19.259 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_07_00_UTC 2024-04-02T09:09:19.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:19.260 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_07_00_UTC) 2024-04-02T09:09:19.310 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_08_00_UTC 2024-04-02T09:09:19.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:19.310 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_08_00_UTC) 2024-04-02T09:09:19.360 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_09_00_UTC 2024-04-02T09:09:19.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:19.360 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-02-09_09_00_UTC) 2024-04-02T09:09:19.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:19.410 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2024-04-02T09:09:19.460 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T09:09:19.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.655+0000 7f05e3000700 1 Processor -- start 2024-04-02T09:09:19.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.655+0000 7f05e3000700 1 -- start start 2024-04-02T09:09:19.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.657+0000 7f05e3000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f05dc0d16a0 0x7f05dc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:19.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.657+0000 7f05e3000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05dc0d2370 0x7f05dc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:19.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.657+0000 7f05e3000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05dc161420 con 0x7f05dc15d9d0 2024-04-02T09:09:19.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.657+0000 7f05e3000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05dc0d2820 con 0x7f05dc0d16a0 2024-04-02T09:09:19.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.657+0000 7f05e3000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc0d4340 con 0x7f05dc0d2370 2024-04-02T09:09:19.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.657+0000 7f05db400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f05dc0d16a0 0x7f05dc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:19.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.657+0000 7f05db400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f05dc0d16a0 0x7f05dc0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51884/0 (socket says 172.21.15.89:51884) 2024-04-02T09:09:19.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.657+0000 7f05db400700 1 -- 172.21.15.89:0/2954823332 learned_addr learned my addr 172.21.15.89:0/2954823332 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:19.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.657+0000 7f05db400700 1 -- 172.21.15.89:0/2954823332 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05dc0d2370 msgr2=0x7f05dc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:19.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.657+0000 7f05e1000700 1 --2- 172.21.15.89:0/2954823332 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05dc0d2370 0x7f05dc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:19.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.657+0000 7f05db400700 1 --2- 172.21.15.89:0/2954823332 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05dc0d2370 0x7f05dc0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:19.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.657+0000 7f05db400700 1 -- 172.21.15.89:0/2954823332 >> v1:172.21.15.113:6789/0 conn(0x7f05dc15d9d0 legacy=0x7f05dc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:19.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.657+0000 7f05db400700 1 -- 172.21.15.89:0/2954823332 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05dc161700 con 0x7f05dc0d16a0 2024-04-02T09:09:19.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.657+0000 7f05e1000700 1 --2- 172.21.15.89:0/2954823332 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05dc0d2370 0x7f05dc0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:19.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.658+0000 7f05db400700 1 --2- 172.21.15.89:0/2954823332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05dc0d16a0 0x7f05dc0d1e30 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f05c80098e0 tx=0x7f05c8009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8025c5045ef5f3e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:19.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.658+0000 7f05daa00700 1 -- 172.21.15.89:0/2954823332 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c8004230 con 0x7f05dc0d16a0 2024-04-02T09:09:19.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.658+0000 7f05daa00700 1 -- 172.21.15.89:0/2954823332 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c8004390 con 0x7f05dc0d16a0 2024-04-02T09:09:19.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.658+0000 7f05daa00700 1 -- 172.21.15.89:0/2954823332 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c80036a0 con 0x7f05dc0d16a0 2024-04-02T09:09:19.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.658+0000 7f05e3000700 1 -- 172.21.15.89:0/2954823332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05dc0d16a0 msgr2=0x7f05dc0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:19.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.658+0000 7f05e3000700 1 --2- 172.21.15.89:0/2954823332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05dc0d16a0 0x7f05dc0d1e30 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f05c80098e0 tx=0x7f05c8009bf0 comp rx=0 tx=0).stop 2024-04-02T09:09:19.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.659+0000 7f05e3000700 1 -- 172.21.15.89:0/2954823332 shutdown_connections 2024-04-02T09:09:19.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.659+0000 7f05e3000700 1 --2- 172.21.15.89:0/2954823332 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05dc0d2370 0x7f05dc0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:19.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.659+0000 7f05e3000700 1 --2- 172.21.15.89:0/2954823332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05dc0d16a0 0x7f05dc0d1e30 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:19.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.659+0000 7f05e3000700 1 -- 172.21.15.89:0/2954823332 >> 172.21.15.89:0/2954823332 conn(0x7f05dc08aff0 msgr2=0x7f05dc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:19.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.659+0000 7f05e3000700 1 -- 172.21.15.89:0/2954823332 shutdown_connections 2024-04-02T09:09:19.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.659+0000 7f05e3000700 1 -- 172.21.15.89:0/2954823332 wait complete. 2024-04-02T09:09:19.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.661+0000 7f05e3000700 1 Processor -- start 2024-04-02T09:09:19.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.661+0000 7f05e3000700 1 -- start start 2024-04-02T09:09:19.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.662+0000 7f05e3000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05dc0d16a0 0x7f05dc15e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:19.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.662+0000 7f05e3000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05dc0d2370 0x7f05dc15b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:19.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.662+0000 7f05e3000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05dc15d9d0 0x7f05dc0cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:19.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.662+0000 7f05e3000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc15e7b0 con 0x7f05dc15d9d0 2024-04-02T09:09:19.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.662+0000 7f05e3000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc0b8030 con 0x7f05dc0d2370 2024-04-02T09:09:19.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.662+0000 7f05e3000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc0b8310 con 0x7f05dc0d16a0 2024-04-02T09:09:19.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.663+0000 7f05e1000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05dc15d9d0 0x7f05dc0cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:19.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.663+0000 7f05e1000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05dc15d9d0 0x7f05dc0cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51886/0 (socket says 172.21.15.89:51886) 2024-04-02T09:09:19.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.663+0000 7f05e1000700 1 -- 172.21.15.89:0/1011855215 learned_addr learned my addr 172.21.15.89:0/1011855215 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:19.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.663+0000 7f05dbe00700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05dc0d16a0 0x7f05dc15e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:19.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.663+0000 7f05db400700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05dc0d2370 0x7f05dc15b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:19.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.663+0000 7f05e1000700 1 -- 172.21.15.89:0/1011855215 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05dc0d16a0 msgr2=0x7f05dc15e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:19.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.663+0000 7f05e1000700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05dc0d16a0 0x7f05dc15e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:19.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.663+0000 7f05e1000700 1 -- 172.21.15.89:0/1011855215 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05dc0d2370 msgr2=0x7f05dc15b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:19.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.663+0000 7f05e1000700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05dc0d2370 0x7f05dc15b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:19.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.663+0000 7f05e1000700 1 -- 172.21.15.89:0/1011855215 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c80094f0 con 0x7f05dc15d9d0 2024-04-02T09:09:19.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.663+0000 7f05db400700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05dc0d2370 0x7f05dc15b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:19.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.663+0000 7f05e1000700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05dc15d9d0 0x7f05dc0cbf50 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f05d400cef0 tx=0x7f05d400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:19.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.663+0000 7f05d3e00700 1 -- 172.21.15.89:0/1011855215 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d400adb0 con 0x7f05dc15d9d0 2024-04-02T09:09:19.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.664+0000 7f05d3e00700 1 -- 172.21.15.89:0/1011855215 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d4015440 con 0x7f05dc15d9d0 2024-04-02T09:09:19.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.664+0000 7f05e3000700 1 -- 172.21.15.89:0/1011855215 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05dc0b85f0 con 0x7f05dc15d9d0 2024-04-02T09:09:19.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.664+0000 7f05d3e00700 1 -- 172.21.15.89:0/1011855215 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d40157f0 con 0x7f05dc15d9d0 2024-04-02T09:09:19.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.664+0000 7f05e3000700 1 -- 172.21.15.89:0/1011855215 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05dc0b7cc0 con 0x7f05dc15d9d0 2024-04-02T09:09:19.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.665+0000 7f05e3000700 1 -- 172.21.15.89:0/1011855215 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05dc0d1ca0 con 0x7f05dc15d9d0 2024-04-02T09:09:19.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.666+0000 7f05d3e00700 1 -- 172.21.15.89:0/1011855215 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 127) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f05d400d430 con 0x7f05dc15d9d0 2024-04-02T09:09:19.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.666+0000 7f05d3e00700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f05bc06ae00 0x7f05bc06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:19.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.667+0000 7f05d3e00700 1 -- 172.21.15.89:0/1011855215 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f05d4089f50 con 0x7f05dc15d9d0 2024-04-02T09:09:19.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.667+0000 7f05dbe00700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f05bc06ae00 0x7f05bc06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:19.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.667+0000 7f05dbe00700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f05bc06ae00 0x7f05bc06d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f05dc1e9b10 tx=0x7f05cc0090e0 comp rx=0 tx=0).ready entity=mgr.10311 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:19.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.672+0000 7f05d3e00700 1 -- 172.21.15.89:0/1011855215 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f05d4054b40 con 0x7f05dc15d9d0 2024-04-02T09:09:19.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:19.936+0000 7f05e3000700 1 -- 172.21.15.89:0/1011855215 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f05dc0ca5f0 con 0x7f05dc15d9d0 2024-04-02T09:09:20.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:20.795+0000 7f05d3e00700 1 -- 172.21.15.89:0/1011855215 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 128) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f05d4054f10 con 0x7f05dc15d9d0 2024-04-02T09:09:20.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:20.796+0000 7f05d3e00700 1 -- 172.21.15.89:0/1011855215 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v128) v1 ==== 93+0+0 (secure 0 0 0) 0x7f05d4071ac0 con 0x7f05dc15d9d0 2024-04-02T09:09:20.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:20.799+0000 7f05e3000700 1 -- 172.21.15.89:0/1011855215 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f05bc06ae00 msgr2=0x7f05bc06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:20.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:20.799+0000 7f05e3000700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f05bc06ae00 0x7f05bc06d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f05dc1e9b10 tx=0x7f05cc0090e0 comp rx=0 tx=0).stop 2024-04-02T09:09:20.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:20.799+0000 7f05e3000700 1 -- 172.21.15.89:0/1011855215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05dc15d9d0 msgr2=0x7f05dc0cbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:20.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:20.799+0000 7f05e3000700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05dc15d9d0 0x7f05dc0cbf50 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f05d400cef0 tx=0x7f05d400cf20 comp rx=0 tx=0).stop 2024-04-02T09:09:20.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:20.799+0000 7f05e3000700 1 -- 172.21.15.89:0/1011855215 shutdown_connections 2024-04-02T09:09:20.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:20.799+0000 7f05e3000700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05dc15d9d0 0x7f05dc0cbf50 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:20.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:20.800+0000 7f05e3000700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f05dc0d16a0 0x7f05dc15e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:20.800+0000 7f05e3000700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f05bc06ae00 0x7f05bc06d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:20.800+0000 7f05e3000700 1 --2- 172.21.15.89:0/1011855215 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05dc0d2370 0x7f05dc15b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:20.800+0000 7f05e3000700 1 -- 172.21.15.89:0/1011855215 >> 172.21.15.89:0/1011855215 conn(0x7f05dc08aff0 msgr2=0x7f05dc0b5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:20.800+0000 7f05e3000700 1 -- 172.21.15.89:0/1011855215 shutdown_connections 2024-04-02T09:09:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:20.800+0000 7f05e3000700 1 -- 172.21.15.89:0/1011855215 wait complete. 2024-04-02T09:09:20.820 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T09:09:20.820 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:09:20.905 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:09:20.993 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T09:09:20.994 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:09:20.994 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:09:21.013 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:21.015+0000 7ff0714f6240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:09:21.030 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:21.032+0000 7f83a0cb5240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:09:21.039 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:09:21.040 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:09:21.124 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:09:21.198 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:21.200+0000 7ff0714f6240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:09:21.220 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:21.222+0000 7f83a0cb5240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:09:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.386+0000 7f1065800700 1 Processor -- start 2024-04-02T09:09:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.386+0000 7f1065800700 1 -- start start 2024-04-02T09:09:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.388+0000 7f1065800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f106015da50 0x7f10600ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.388+0000 7f1065800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f10600cec70 0x7f10600d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.388+0000 7f1065800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10600d0b20 0x7f10600b52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.388+0000 7f1065800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10600b95d0 con 0x7f10600d0b20 2024-04-02T09:09:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.388+0000 7f1065800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106015de30 con 0x7f10600cec70 2024-04-02T09:09:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.388+0000 7f1065800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10600d0f80 con 0x7f106015da50 2024-04-02T09:09:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.389+0000 7f105f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10600d0b20 0x7f10600b52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.389+0000 7f105f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10600d0b20 0x7f10600b52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51908/0 (socket says 172.21.15.89:51908) 2024-04-02T09:09:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.389+0000 7f105f400700 1 -- 172.21.15.89:0/1624718310 learned_addr learned my addr 172.21.15.89:0/1624718310 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.389+0000 7f105e000700 1 --2- 172.21.15.89:0/1624718310 >> v2:172.21.15.113:3300/0 conn(0x7f10600cec70 0x7f10600d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.389+0000 7f105ea00700 1 --2- 172.21.15.89:0/1624718310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f106015da50 0x7f10600ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.389+0000 7f105f400700 1 -- 172.21.15.89:0/1624718310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f106015da50 msgr2=0x7f10600ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.389+0000 7f105f400700 1 --2- 172.21.15.89:0/1624718310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f106015da50 0x7f10600ce730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.389+0000 7f105f400700 1 -- 172.21.15.89:0/1624718310 >> v2:172.21.15.113:3300/0 conn(0x7f10600cec70 msgr2=0x7f10600d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.389+0000 7f105f400700 1 --2- 172.21.15.89:0/1624718310 >> v2:172.21.15.113:3300/0 conn(0x7f10600cec70 0x7f10600d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.389+0000 7f105f400700 1 -- 172.21.15.89:0/1624718310 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10600d24e0 con 0x7f10600d0b20 2024-04-02T09:09:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.389+0000 7f105ea00700 1 --2- 172.21.15.89:0/1624718310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f106015da50 0x7f10600ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.389+0000 7f105f400700 1 --2- 172.21.15.89:0/1624718310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10600d0b20 0x7f10600b52e0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f10600742a0 tx=0x7f105800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=329e37eb185a2226 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.390+0000 7f105d600700 1 -- 172.21.15.89:0/1624718310 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800e070 con 0x7f10600d0b20 2024-04-02T09:09:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.390+0000 7f105d600700 1 -- 172.21.15.89:0/1624718310 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105800b910 con 0x7f10600d0b20 2024-04-02T09:09:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.390+0000 7f105d600700 1 -- 172.21.15.89:0/1624718310 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800bcc0 con 0x7f10600d0b20 2024-04-02T09:09:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.391+0000 7f1065800700 1 -- 172.21.15.89:0/1624718310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10600d0b20 msgr2=0x7f10600b52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.391+0000 7f1065800700 1 --2- 172.21.15.89:0/1624718310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10600d0b20 0x7f10600b52e0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f10600742a0 tx=0x7f105800b330 comp rx=0 tx=0).stop 2024-04-02T09:09:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.391+0000 7f1065800700 1 -- 172.21.15.89:0/1624718310 shutdown_connections 2024-04-02T09:09:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.391+0000 7f1065800700 1 --2- 172.21.15.89:0/1624718310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10600d0b20 0x7f10600b52e0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.391+0000 7f1065800700 1 --2- 172.21.15.89:0/1624718310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f106015da50 0x7f10600ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.391+0000 7f1065800700 1 --2- 172.21.15.89:0/1624718310 >> v2:172.21.15.113:3300/0 conn(0x7f10600cec70 0x7f10600d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.391+0000 7f1065800700 1 -- 172.21.15.89:0/1624718310 >> 172.21.15.89:0/1624718310 conn(0x7f10600666a0 msgr2=0x7f1060066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.391+0000 7f1065800700 1 -- 172.21.15.89:0/1624718310 shutdown_connections 2024-04-02T09:09:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.391+0000 7f1065800700 1 -- 172.21.15.89:0/1624718310 wait complete. 2024-04-02T09:09:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.394+0000 7f1065800700 1 Processor -- start 2024-04-02T09:09:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.394+0000 7f1065800700 1 -- start start 2024-04-02T09:09:21.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.395+0000 7f1065800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10600cec70 0x7f106014d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:21.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.395+0000 7f1065800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10600d0b20 0x7f10600c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:21.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.395+0000 7f1065800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f106015da50 0x7f10601f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:21.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.395+0000 7f1065800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10600b4e10 con 0x7f106015da50 2024-04-02T09:09:21.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.396+0000 7f1065800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10600c7890 con 0x7f10600d0b20 2024-04-02T09:09:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.396+0000 7f1065800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10600c7b30 con 0x7f10600cec70 2024-04-02T09:09:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.396+0000 7f105f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f106015da50 0x7f10601f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.396+0000 7f105ea00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10600cec70 0x7f106014d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.396+0000 7f105e000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10600d0b20 0x7f10600c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.396+0000 7f105f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f106015da50 0x7f10601f2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51920/0 (socket says 172.21.15.89:51920) 2024-04-02T09:09:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.396+0000 7f105f400700 1 -- 172.21.15.89:0/3275439485 learned_addr learned my addr 172.21.15.89:0/3275439485 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.396+0000 7f105e000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10600d0b20 0x7f10600c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:60160/0 (socket says 172.21.15.89:60160) 2024-04-02T09:09:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.396+0000 7f105f400700 1 -- 172.21.15.89:0/3275439485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10600cec70 msgr2=0x7f106014d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.396+0000 7f105f400700 1 --2- 172.21.15.89:0/3275439485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10600cec70 0x7f106014d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.396+0000 7f105f400700 1 -- 172.21.15.89:0/3275439485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10600d0b20 msgr2=0x7f10600c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.396+0000 7f105f400700 1 --2- 172.21.15.89:0/3275439485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10600d0b20 0x7f10600c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.396+0000 7f105f400700 1 -- 172.21.15.89:0/3275439485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1058009b30 con 0x7f106015da50 2024-04-02T09:09:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.396+0000 7f105e000700 1 --2- 172.21.15.89:0/3275439485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10600d0b20 0x7f10600c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:09:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.397+0000 7f105f400700 1 --2- 172.21.15.89:0/3275439485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f106015da50 0x7f10601f2530 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f10600741d0 tx=0x7f1058004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:21.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.397+0000 7f1056a00700 1 -- 172.21.15.89:0/3275439485 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10580099b0 con 0x7f106015da50 2024-04-02T09:09:21.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.397+0000 7f1056a00700 1 -- 172.21.15.89:0/3275439485 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105800ce80 con 0x7f106015da50 2024-04-02T09:09:21.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.397+0000 7f1056a00700 1 -- 172.21.15.89:0/3275439485 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1058004490 con 0x7f106015da50 2024-04-02T09:09:21.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.397+0000 7f1065800700 1 -- 172.21.15.89:0/3275439485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10600c7e10 con 0x7f106015da50 2024-04-02T09:09:21.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.397+0000 7f1065800700 1 -- 172.21.15.89:0/3275439485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10600c83d0 con 0x7f106015da50 2024-04-02T09:09:21.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.398+0000 7f1065800700 1 -- 172.21.15.89:0/3275439485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10600b4f80 con 0x7f106015da50 2024-04-02T09:09:21.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.400+0000 7f1056a00700 1 -- 172.21.15.89:0/3275439485 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 128) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f10580045f0 con 0x7f106015da50 2024-04-02T09:09:21.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.400+0000 7f1056a00700 1 --2- 172.21.15.89:0/3275439485 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f104006adb0 0x7f104006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:21.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.400+0000 7f105ea00700 1 -- 172.21.15.89:0/3275439485 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f104006adb0 msgr2=0x7f104006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:21.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.400+0000 7f1056a00700 1 -- 172.21.15.89:0/3275439485 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f105808af90 con 0x7f106015da50 2024-04-02T09:09:21.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.401+0000 7f105ea00700 1 --2- 172.21.15.89:0/3275439485 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f104006adb0 0x7f104006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:21.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.408+0000 7f1056a00700 1 -- 172.21.15.89:0/3275439485 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1058055b80 con 0x7f106015da50 2024-04-02T09:09:21.516 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:21.518+0000 7ff0714f6240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:09:21.561 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:21.562+0000 7f83a0cb5240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:09:21.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.602+0000 7f105ea00700 1 -- 172.21.15.89:0/3275439485 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f104006adb0 msgr2=0x7f104006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:21.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.602+0000 7f105ea00700 1 --2- 172.21.15.89:0/3275439485 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f104006adb0 0x7f104006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:21.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.620+0000 7f1065800700 1 -- 172.21.15.89:0/3275439485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10600b4f50 con 0x7f106015da50 2024-04-02T09:09:21.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.622+0000 7f1056a00700 1 -- 172.21.15.89:0/3275439485 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v190) v1 ==== 74+0+17712 (secure 0 0 0) 0x7f105802a090 con 0x7f106015da50 2024-04-02T09:09:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:21.621 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":190,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:04:23.614793+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-02T08:53:20.736248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-02T08:53:21.860376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"23.b","mappings":[{"from":5,"to":3}]},{"pgid":"23.1d","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/4284766432":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/3938746027":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/3350462335":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/3070765417":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/4154199909":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/2003321918":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/2504651954":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/978973291":"2024-04-03T08:53:39.182240+0000","172.21.15.89:6833/43357":"2024-04-03T08:53:39.182240+0000","172.21.15.89:6832/43357":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/1859816274":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/477599566":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/3314137116":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/2444652307":"2024-04-03T08:53:39.182240+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:09:21.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.625+0000 7f1065800700 1 -- 172.21.15.89:0/3275439485 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f104006adb0 msgr2=0x7f104006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:21.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.625+0000 7f1065800700 1 --2- 172.21.15.89:0/3275439485 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f104006adb0 0x7f104006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.625+0000 7f1065800700 1 -- 172.21.15.89:0/3275439485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f106015da50 msgr2=0x7f10601f2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.625+0000 7f1065800700 1 --2- 172.21.15.89:0/3275439485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f106015da50 0x7f10601f2530 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f10600741d0 tx=0x7f1058004ad0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.625+0000 7f1065800700 1 -- 172.21.15.89:0/3275439485 shutdown_connections 2024-04-02T09:09:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.625+0000 7f1065800700 1 --2- 172.21.15.89:0/3275439485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f106015da50 0x7f10601f2530 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.625+0000 7f1065800700 1 --2- 172.21.15.89:0/3275439485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10600cec70 0x7f106014d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.625+0000 7f1065800700 1 --2- 172.21.15.89:0/3275439485 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f104006adb0 0x7f104006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.625+0000 7f1065800700 1 --2- 172.21.15.89:0/3275439485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10600d0b20 0x7f10600c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.625+0000 7f1065800700 1 -- 172.21.15.89:0/3275439485 >> 172.21.15.89:0/3275439485 conn(0x7f10600666a0 msgr2=0x7f106015aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.625+0000 7f1065800700 1 -- 172.21.15.89:0/3275439485 shutdown_connections 2024-04-02T09:09:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.625+0000 7f1065800700 1 -- 172.21.15.89:0/3275439485 wait complete. 2024-04-02T09:09:21.642 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T08:53:20.736248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T08:53:21.860376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '186', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:09:21.642 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:09:21.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.843+0000 7fb243600700 1 Processor -- start 2024-04-02T09:09:21.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.843+0000 7fb243600700 1 -- start start 2024-04-02T09:09:21.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.844+0000 7fb243600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb23c15d990 0x7fb23c0d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:21.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.845+0000 7fb243600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb23c09a750 0x7fb23c06c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:21.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.845+0000 7fb243600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb23c0d0970 con 0x7fb23c099a60 2024-04-02T09:09:21.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.845+0000 7fb243600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb23c06c020 con 0x7fb23c15d990 2024-04-02T09:09:21.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.845+0000 7fb243600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb23c0c81a0 con 0x7fb23c09a750 2024-04-02T09:09:21.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.845+0000 7fb23be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb23c15d990 0x7fb23c0d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:21.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.845+0000 7fb23be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb23c15d990 0x7fb23c0d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51930/0 (socket says 172.21.15.89:51930) 2024-04-02T09:09:21.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.845+0000 7fb23be00700 1 -- 172.21.15.89:0/2456688776 learned_addr learned my addr 172.21.15.89:0/2456688776 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:21.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.845+0000 7fb241600700 1 --2- 172.21.15.89:0/2456688776 >> v2:172.21.15.113:3300/0 conn(0x7fb23c09a750 0x7fb23c06c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:21.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.845+0000 7fb241600700 1 -- 172.21.15.89:0/2456688776 >> v2:172.21.15.89:3300/0 conn(0x7fb23c15d990 msgr2=0x7fb23c0d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:21.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.845+0000 7fb241600700 1 --2- 172.21.15.89:0/2456688776 >> v2:172.21.15.89:3300/0 conn(0x7fb23c15d990 0x7fb23c0d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.845+0000 7fb241600700 1 -- 172.21.15.89:0/2456688776 >> v1:172.21.15.113:6789/0 conn(0x7fb23c099a60 legacy=0x7fb23c099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:21.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.845+0000 7fb241600700 1 -- 172.21.15.89:0/2456688776 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c0d0690 con 0x7fb23c09a750 2024-04-02T09:09:21.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.845+0000 7fb23be00700 1 --2- 172.21.15.89:0/2456688776 >> v2:172.21.15.89:3300/0 conn(0x7fb23c15d990 0x7fb23c0d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:09:21.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.846+0000 7fb241600700 1 --2- 172.21.15.89:0/2456688776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb23c09a750 0x7fb23c06c4b0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb23c072af0 tx=0x7fb23400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=117ef29584d4f350 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:21.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.846+0000 7fb23b400700 1 -- 172.21.15.89:0/2456688776 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23400e070 con 0x7fb23c09a750 2024-04-02T09:09:21.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.846+0000 7fb23b400700 1 -- 172.21.15.89:0/2456688776 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23400bb20 con 0x7fb23c09a750 2024-04-02T09:09:21.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.846+0000 7fb23b400700 1 -- 172.21.15.89:0/2456688776 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb234012430 con 0x7fb23c09a750 2024-04-02T09:09:21.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.847+0000 7fb243600700 1 -- 172.21.15.89:0/2456688776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb23c09a750 msgr2=0x7fb23c06c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:21.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.847+0000 7fb243600700 1 --2- 172.21.15.89:0/2456688776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb23c09a750 0x7fb23c06c4b0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb23c072af0 tx=0x7fb23400b330 comp rx=0 tx=0).stop 2024-04-02T09:09:21.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.847+0000 7fb243600700 1 -- 172.21.15.89:0/2456688776 shutdown_connections 2024-04-02T09:09:21.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.847+0000 7fb243600700 1 --2- 172.21.15.89:0/2456688776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb23c09a750 0x7fb23c06c4b0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.847+0000 7fb243600700 1 --2- 172.21.15.89:0/2456688776 >> v2:172.21.15.89:3300/0 conn(0x7fb23c15d990 0x7fb23c0d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.847+0000 7fb243600700 1 -- 172.21.15.89:0/2456688776 >> 172.21.15.89:0/2456688776 conn(0x7fb23c0b2e20 msgr2=0x7fb23c0b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:21.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.847+0000 7fb243600700 1 -- 172.21.15.89:0/2456688776 shutdown_connections 2024-04-02T09:09:21.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.847+0000 7fb243600700 1 -- 172.21.15.89:0/2456688776 wait complete. 2024-04-02T09:09:21.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.849+0000 7fb243600700 1 Processor -- start 2024-04-02T09:09:21.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.849+0000 7fb243600700 1 -- start start 2024-04-02T09:09:21.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.850+0000 7fb243600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb23c099a60 0x7fb23c0c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:21.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb243600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23c09a750 0x7fb23c0c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb243600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb23c15d990 0x7fb23c0cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb243600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c0c66e0 con 0x7fb23c09a750 2024-04-02T09:09:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb243600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c0c69c0 con 0x7fb23c099a60 2024-04-02T09:09:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb243600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c0c6ca0 con 0x7fb23c15d990 2024-04-02T09:09:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb23be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23c09a750 0x7fb23c0c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb23be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23c09a750 0x7fb23c0c7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51936/0 (socket says 172.21.15.89:51936) 2024-04-02T09:09:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb23be00700 1 -- 172.21.15.89:0/3525379493 learned_addr learned my addr 172.21.15.89:0/3525379493 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb240c00700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb23c099a60 0x7fb23c0c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb241600700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb23c15d990 0x7fb23c0cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb23be00700 1 -- 172.21.15.89:0/3525379493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb23c15d990 msgr2=0x7fb23c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb23be00700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb23c15d990 0x7fb23c0cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb23be00700 1 -- 172.21.15.89:0/3525379493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb23c099a60 msgr2=0x7fb23c0c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb23be00700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb23c099a60 0x7fb23c0c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb23be00700 1 -- 172.21.15.89:0/3525379493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb228009550 con 0x7fb23c09a750 2024-04-02T09:09:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.851+0000 7fb240c00700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb23c099a60 0x7fb23c0c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:09:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.852+0000 7fb23be00700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23c09a750 0x7fb23c0c7760 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fb22800e9b0 tx=0x7fb22800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.852+0000 7fb238c00700 1 -- 172.21.15.89:0/3525379493 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22800ee50 con 0x7fb23c09a750 2024-04-02T09:09:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.852+0000 7fb238c00700 1 -- 172.21.15.89:0/3525379493 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22800c850 con 0x7fb23c09a750 2024-04-02T09:09:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.852+0000 7fb238c00700 1 -- 172.21.15.89:0/3525379493 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22800cbc0 con 0x7fb23c09a750 2024-04-02T09:09:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.852+0000 7fb243600700 1 -- 172.21.15.89:0/3525379493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb234009a70 con 0x7fb23c09a750 2024-04-02T09:09:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.852+0000 7fb243600700 1 -- 172.21.15.89:0/3525379493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23c0c7260 con 0x7fb23c09a750 2024-04-02T09:09:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.853+0000 7fb243600700 1 -- 172.21.15.89:0/3525379493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23c067720 con 0x7fb23c09a750 2024-04-02T09:09:21.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.855+0000 7fb238c00700 1 -- 172.21.15.89:0/3525379493 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 128) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fb22800cd20 con 0x7fb23c09a750 2024-04-02T09:09:21.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.856+0000 7fb238c00700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb21c06adb0 0x7fb21c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:21.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.856+0000 7fb238c00700 1 -- 172.21.15.89:0/3525379493 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6789+0+0 (secure 0 0 0) 0x7fb228089940 con 0x7fb23c09a750 2024-04-02T09:09:21.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.856+0000 7fb240c00700 1 -- 172.21.15.89:0/3525379493 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb21c06adb0 msgr2=0x7fb21c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:21.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.856+0000 7fb240c00700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb21c06adb0 0x7fb21c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:21.860+0000 7fb238c00700 1 -- 172.21.15.89:0/3525379493 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb228054530 con 0x7fb23c09a750 2024-04-02T09:09:22.045 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:22.046+0000 7ff0714f6240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:09:22.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.057+0000 7fb240c00700 1 -- 172.21.15.89:0/3525379493 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb21c06adb0 msgr2=0x7fb21c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:22.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.057+0000 7fb240c00700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb21c06adb0 0x7fb21c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:22.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.071+0000 7fb243600700 1 -- 172.21.15.89:0/3525379493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb23c0b3550 con 0x7fb23c09a750 2024-04-02T09:09:22.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.071+0000 7fb238c00700 1 -- 172.21.15.89:0/3525379493 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v190) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb228057b50 con 0x7fb23c09a750 2024-04-02T09:09:22.070 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:09:22.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.074+0000 7fb243600700 1 -- 172.21.15.89:0/3525379493 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb21c06adb0 msgr2=0x7fb21c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:22.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.074+0000 7fb243600700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb21c06adb0 0x7fb21c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.074+0000 7fb243600700 1 -- 172.21.15.89:0/3525379493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23c09a750 msgr2=0x7fb23c0c7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:22.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.074+0000 7fb243600700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23c09a750 0x7fb23c0c7760 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fb22800e9b0 tx=0x7fb22800ecc0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.074+0000 7fb243600700 1 -- 172.21.15.89:0/3525379493 shutdown_connections 2024-04-02T09:09:22.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.074+0000 7fb243600700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb21c06adb0 0x7fb21c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.074+0000 7fb243600700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb23c099a60 0x7fb23c0c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.074+0000 7fb243600700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb23c15d990 0x7fb23c0cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.074+0000 7fb243600700 1 --2- 172.21.15.89:0/3525379493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb23c09a750 0x7fb23c0c7760 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.074+0000 7fb243600700 1 -- 172.21.15.89:0/3525379493 >> 172.21.15.89:0/3525379493 conn(0x7fb23c0b2e20 msgr2=0x7fb23c15ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:22.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.074+0000 7fb243600700 1 -- 172.21.15.89:0/3525379493 shutdown_connections 2024-04-02T09:09:22.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.074+0000 7fb243600700 1 -- 172.21.15.89:0/3525379493 wait complete. 2024-04-02T09:09:22.091 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T09:09:22.117 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:22.119+0000 7f83a0cb5240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:09:22.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.291+0000 7f1de9200700 1 Processor -- start 2024-04-02T09:09:22.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.291+0000 7f1de9200700 1 -- start start 2024-04-02T09:09:22.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.293+0000 7f1de9200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1de415ef70 0x7f1de40b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:22.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.293+0000 7f1de9200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de4161150 con 0x7f1de406a790 2024-04-02T09:09:22.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.293+0000 7f1de9200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de406dba0 con 0x7f1de4160d70 2024-04-02T09:09:22.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.293+0000 7f1de9200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de40b4cd0 con 0x7f1de415ef70 2024-04-02T09:09:22.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.293+0000 7f1de1e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f1de4160d70 0x7f1de415e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:47686/0 (socket says 172.21.15.89:47686) 2024-04-02T09:09:22.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.293+0000 7f1de1e00700 1 -- 172.21.15.89:0/464091836 learned_addr learned my addr 172.21.15.89:0/464091836 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:22.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.293+0000 7f1de1400700 1 --2- 172.21.15.89:0/464091836 >> v2:172.21.15.89:3300/0 conn(0x7f1de415ef70 0x7f1de40b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:22.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.293+0000 7f1de0a00700 1 -- 172.21.15.89:0/464091836 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105679948 0 0) 0x7f1de406dba0 con 0x7f1de4160d70 2024-04-02T09:09:22.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.293+0000 7f1de1400700 1 -- 172.21.15.89:0/464091836 >> v1:172.21.15.113:6789/0 conn(0x7f1de4160d70 legacy=0x7f1de415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:22.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.293+0000 7f1de1400700 1 -- 172.21.15.89:0/464091836 >> v1:172.21.15.89:6789/0 conn(0x7f1de406a790 legacy=0x7f1de406e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:22.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.293+0000 7f1de1400700 1 -- 172.21.15.89:0/464091836 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de40b4ee0 con 0x7f1de415ef70 2024-04-02T09:09:22.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.294+0000 7f1de1400700 1 --2- 172.21.15.89:0/464091836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de415ef70 0x7f1de40b3200 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009840 tx=0x7f1dd0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b77ab1aad5f92a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:22.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.294+0000 7f1de0a00700 1 -- 172.21.15.89:0/464091836 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0011830 con 0x7f1de415ef70 2024-04-02T09:09:22.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.294+0000 7f1de0a00700 1 -- 172.21.15.89:0/464091836 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0011990 con 0x7f1de415ef70 2024-04-02T09:09:22.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.294+0000 7f1de0a00700 1 -- 172.21.15.89:0/464091836 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0011d20 con 0x7f1de415ef70 2024-04-02T09:09:22.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.295+0000 7f1de9200700 1 -- 172.21.15.89:0/464091836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de415ef70 msgr2=0x7f1de40b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:22.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.295+0000 7f1de9200700 1 --2- 172.21.15.89:0/464091836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de415ef70 0x7f1de40b3200 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009840 tx=0x7f1dd0009b50 comp rx=0 tx=0).stop 2024-04-02T09:09:22.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.295+0000 7f1de9200700 1 -- 172.21.15.89:0/464091836 shutdown_connections 2024-04-02T09:09:22.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.295+0000 7f1de9200700 1 --2- 172.21.15.89:0/464091836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de415ef70 0x7f1de40b3200 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.295+0000 7f1de9200700 1 -- 172.21.15.89:0/464091836 >> 172.21.15.89:0/464091836 conn(0x7f1de40b5720 msgr2=0x7f1de40b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:22.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.295+0000 7f1de9200700 1 -- 172.21.15.89:0/464091836 shutdown_connections 2024-04-02T09:09:22.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.295+0000 7f1de9200700 1 -- 172.21.15.89:0/464091836 wait complete. 2024-04-02T09:09:22.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.296+0000 7f1de9200700 1 Processor -- start 2024-04-02T09:09:22.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.297+0000 7f1de9200700 1 -- start start 2024-04-02T09:09:22.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.297+0000 7f1de9200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de406a790 0x7f1de41f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:22.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de9200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1de415ef70 0x7f1de40ad170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:22.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de9200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1de4160d70 0x7f1de40ad6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:22.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de9200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de414df20 con 0x7f1de406a790 2024-04-02T09:09:22.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de9200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de4148e00 con 0x7f1de415ef70 2024-04-02T09:09:22.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de9200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de40add50 con 0x7f1de4160d70 2024-04-02T09:09:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de406a790 0x7f1de41f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de406a790 0x7f1de41f2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51962/0 (socket says 172.21.15.89:51962) 2024-04-02T09:09:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de1e00700 1 -- 172.21.15.89:0/1530750622 learned_addr learned my addr 172.21.15.89:0/1530750622 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de1400700 1 --2- 172.21.15.89:0/1530750622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1de415ef70 0x7f1de40ad170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de2800700 1 --2- 172.21.15.89:0/1530750622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1de4160d70 0x7f1de40ad6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de1e00700 1 -- 172.21.15.89:0/1530750622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1de4160d70 msgr2=0x7f1de40ad6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de1e00700 1 --2- 172.21.15.89:0/1530750622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1de4160d70 0x7f1de40ad6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de1e00700 1 -- 172.21.15.89:0/1530750622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1de415ef70 msgr2=0x7f1de40ad170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de1e00700 1 --2- 172.21.15.89:0/1530750622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1de415ef70 0x7f1de40ad170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.298+0000 7f1de1e00700 1 -- 172.21.15.89:0/1530750622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd00094a0 con 0x7f1de406a790 2024-04-02T09:09:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.299+0000 7f1de1e00700 1 --2- 172.21.15.89:0/1530750622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de406a790 0x7f1de41f2750 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400c500 tx=0x7f1dd400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.299+0000 7f1dda000700 1 -- 172.21.15.89:0/1530750622 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4004c80 con 0x7f1de406a790 2024-04-02T09:09:22.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.299+0000 7f1dda000700 1 -- 172.21.15.89:0/1530750622 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd4004e00 con 0x7f1de406a790 2024-04-02T09:09:22.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.299+0000 7f1de9200700 1 -- 172.21.15.89:0/1530750622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de40ae030 con 0x7f1de406a790 2024-04-02T09:09:22.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.299+0000 7f1dda000700 1 -- 172.21.15.89:0/1530750622 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4007310 con 0x7f1de406a790 2024-04-02T09:09:22.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.299+0000 7f1de9200700 1 -- 172.21.15.89:0/1530750622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de40ae5f0 con 0x7f1de406a790 2024-04-02T09:09:22.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.300+0000 7f1de9200700 1 -- 172.21.15.89:0/1530750622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de406f410 con 0x7f1de406a790 2024-04-02T09:09:22.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.301+0000 7f1dda000700 1 -- 172.21.15.89:0/1530750622 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 128) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f1dd4007470 con 0x7f1de406a790 2024-04-02T09:09:22.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.303+0000 7f1dda000700 1 --2- 172.21.15.89:0/1530750622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1dc406adb0 0x7f1dc406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:22.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.303+0000 7f1de1400700 1 -- 172.21.15.89:0/1530750622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1dc406adb0 msgr2=0x7f1dc406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:22.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.303+0000 7f1dda000700 1 -- 172.21.15.89:0/1530750622 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f1dd40894e0 con 0x7f1de406a790 2024-04-02T09:09:22.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.303+0000 7f1de1400700 1 --2- 172.21.15.89:0/1530750622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1dc406adb0 0x7f1dc406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:22.302 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:22.303+0000 7ff0714f6240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:09:22.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.307+0000 7f1dda000700 1 -- 172.21.15.89:0/1530750622 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1dd40540d0 con 0x7f1de406a790 2024-04-02T09:09:22.385 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:22.387+0000 7f83a0cb5240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:09:22.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.504+0000 7f1de1400700 1 -- 172.21.15.89:0/1530750622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1dc406adb0 msgr2=0x7f1dc406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:22.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.504+0000 7f1de1400700 1 --2- 172.21.15.89:0/1530750622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1dc406adb0 0x7f1dc406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:22.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.518+0000 7f1de9200700 1 -- 172.21.15.89:0/1530750622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1de4067720 con 0x7f1de406a790 2024-04-02T09:09:22.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.519+0000 7f1dda000700 1 -- 172.21.15.89:0/1530750622 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v190) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1dd40576f0 con 0x7f1de406a790 2024-04-02T09:09:22.517 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:09:22.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.521+0000 7f1de9200700 1 -- 172.21.15.89:0/1530750622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1dc406adb0 msgr2=0x7f1dc406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:22.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.521+0000 7f1de9200700 1 --2- 172.21.15.89:0/1530750622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1dc406adb0 0x7f1dc406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.521+0000 7f1de9200700 1 -- 172.21.15.89:0/1530750622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de406a790 msgr2=0x7f1de41f2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:22.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.521+0000 7f1de9200700 1 --2- 172.21.15.89:0/1530750622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de406a790 0x7f1de41f2750 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400c500 tx=0x7f1dd400c810 comp rx=0 tx=0).stop 2024-04-02T09:09:22.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.521+0000 7f1de9200700 1 -- 172.21.15.89:0/1530750622 shutdown_connections 2024-04-02T09:09:22.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.521+0000 7f1de9200700 1 --2- 172.21.15.89:0/1530750622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1de4160d70 0x7f1de40ad6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.521+0000 7f1de9200700 1 --2- 172.21.15.89:0/1530750622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de406a790 0x7f1de41f2750 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.521+0000 7f1de9200700 1 --2- 172.21.15.89:0/1530750622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1dc406adb0 0x7f1dc406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.521+0000 7f1de9200700 1 --2- 172.21.15.89:0/1530750622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1de415ef70 0x7f1de40ad170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.521+0000 7f1de9200700 1 -- 172.21.15.89:0/1530750622 >> 172.21.15.89:0/1530750622 conn(0x7f1de40b5720 msgr2=0x7f1de409be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:22.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.522+0000 7f1de9200700 1 -- 172.21.15.89:0/1530750622 shutdown_connections 2024-04-02T09:09:22.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.522+0000 7f1de9200700 1 -- 172.21.15.89:0/1530750622 wait complete. 2024-04-02T09:09:22.539 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T09:09:22.761 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:22.762+0000 7ff0714f6240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:09:22.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.787+0000 7f8009600700 1 Processor -- start 2024-04-02T09:09:22.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.787+0000 7f8009600700 1 -- start start 2024-04-02T09:09:22.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.788+0000 7f8009600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80040cd020 0x7f80040cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:22.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.789+0000 7f8009600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f80040b2070 0x7f80040b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:22.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.789+0000 7f8009600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80040cdae0 con 0x7f80040cbb30 2024-04-02T09:09:22.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.789+0000 7f8009600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80040cddc0 con 0x7f80040b2070 2024-04-02T09:09:22.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.789+0000 7f8009600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80040ce0a0 con 0x7f80040cd020 2024-04-02T09:09:22.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.789+0000 7f7ffbe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80040cd020 0x7f80040cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:22.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.789+0000 7f7ffbe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80040cd020 0x7f80040cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:33004/0 (socket says 172.21.15.89:33004) 2024-04-02T09:09:22.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.789+0000 7f7ffbe00700 1 -- 172.21.15.89:0/2119447377 learned_addr learned my addr 172.21.15.89:0/2119447377 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:22.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.789+0000 7f8003400700 1 --2- 172.21.15.89:0/2119447377 >> v2:172.21.15.113:3300/0 conn(0x7f80040b2070 0x7f80040b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:22.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.789+0000 7f7ffbe00700 1 -- 172.21.15.89:0/2119447377 >> v2:172.21.15.113:3300/0 conn(0x7f80040b2070 msgr2=0x7f80040b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:22.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.789+0000 7f7ffbe00700 1 --2- 172.21.15.89:0/2119447377 >> v2:172.21.15.113:3300/0 conn(0x7f80040b2070 0x7f80040b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.790+0000 7f7ffbe00700 1 -- 172.21.15.89:0/2119447377 >> v1:172.21.15.113:6789/0 conn(0x7f80040cbb30 legacy=0x7f80040cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:22.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.790+0000 7f7ffbe00700 1 -- 172.21.15.89:0/2119447377 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800406dc20 con 0x7f80040cd020 2024-04-02T09:09:22.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.790+0000 7f7ffbe00700 1 --2- 172.21.15.89:0/2119447377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80040cd020 0x7f80040cd480 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0098f0 tx=0x7f7fec009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7ebb58d080d0fba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:22.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.792+0000 7f8002000700 1 -- 172.21.15.89:0/2119447377 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec011830 con 0x7f80040cd020 2024-04-02T09:09:22.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.793+0000 7f8002000700 1 -- 172.21.15.89:0/2119447377 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec011990 con 0x7f80040cd020 2024-04-02T09:09:22.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.793+0000 7f8002000700 1 -- 172.21.15.89:0/2119447377 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec011d20 con 0x7f80040cd020 2024-04-02T09:09:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.793+0000 7f8009600700 1 -- 172.21.15.89:0/2119447377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80040cd020 msgr2=0x7f80040cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.793+0000 7f8009600700 1 --2- 172.21.15.89:0/2119447377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80040cd020 0x7f80040cd480 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0098f0 tx=0x7f7fec009c00 comp rx=0 tx=0).stop 2024-04-02T09:09:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.793+0000 7f8009600700 1 -- 172.21.15.89:0/2119447377 shutdown_connections 2024-04-02T09:09:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.793+0000 7f8009600700 1 --2- 172.21.15.89:0/2119447377 >> v2:172.21.15.113:3300/0 conn(0x7f80040b2070 0x7f80040b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.793+0000 7f8009600700 1 --2- 172.21.15.89:0/2119447377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80040cd020 0x7f80040cd480 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.793+0000 7f8009600700 1 -- 172.21.15.89:0/2119447377 >> 172.21.15.89:0/2119447377 conn(0x7f80040b0d70 msgr2=0x7f80040b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.794+0000 7f8009600700 1 -- 172.21.15.89:0/2119447377 shutdown_connections 2024-04-02T09:09:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.794+0000 7f8009600700 1 -- 172.21.15.89:0/2119447377 wait complete. 2024-04-02T09:09:22.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.796+0000 7f8009600700 1 Processor -- start 2024-04-02T09:09:22.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.796+0000 7f8009600700 1 -- start start 2024-04-02T09:09:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.797+0000 7f8009600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80040b2070 0x7f80040ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.797+0000 7f8009600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80040cbb30 0x7f800415de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.797+0000 7f8009600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80040cd020 0x7f800414a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.797+0000 7f8009600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80040ae6c0 con 0x7f80040cbb30 2024-04-02T09:09:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.797+0000 7f8009600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80040ac550 con 0x7f80040cd020 2024-04-02T09:09:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.797+0000 7f8009600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80040ac830 con 0x7f80040b2070 2024-04-02T09:09:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.797+0000 7f7ffbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80040cbb30 0x7f800415de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.797+0000 7f7ffbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80040cbb30 0x7f800415de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51978/0 (socket says 172.21.15.89:51978) 2024-04-02T09:09:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.797+0000 7f7ffbe00700 1 -- 172.21.15.89:0/3998280649 learned_addr learned my addr 172.21.15.89:0/3998280649 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.797+0000 7f8002a00700 1 --2- 172.21.15.89:0/3998280649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80040b2070 0x7f80040ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.798+0000 7f8003400700 1 --2- 172.21.15.89:0/3998280649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80040cd020 0x7f800414a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.798+0000 7f7ffbe00700 1 -- 172.21.15.89:0/3998280649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80040b2070 msgr2=0x7f80040ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.798+0000 7f7ffbe00700 1 --2- 172.21.15.89:0/3998280649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80040b2070 0x7f80040ae180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.798+0000 7f7ffbe00700 1 -- 172.21.15.89:0/3998280649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80040cd020 msgr2=0x7f800414a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.798+0000 7f7ffbe00700 1 --2- 172.21.15.89:0/3998280649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80040cd020 0x7f800414a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.798+0000 7f7ffbe00700 1 -- 172.21.15.89:0/3998280649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec009550 con 0x7f80040cbb30 2024-04-02T09:09:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.799+0000 7f7ffbe00700 1 --2- 172.21.15.89:0/3998280649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80040cbb30 0x7f800415de70 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f7fec009c40 tx=0x7f7fec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:22.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.799+0000 7f7ffaa00700 1 -- 172.21.15.89:0/3998280649 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec0036a0 con 0x7f80040cbb30 2024-04-02T09:09:22.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.799+0000 7f7ffaa00700 1 -- 172.21.15.89:0/3998280649 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec01e070 con 0x7f80040cbb30 2024-04-02T09:09:22.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.799+0000 7f8009600700 1 -- 172.21.15.89:0/3998280649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80040acb10 con 0x7f80040cbb30 2024-04-02T09:09:22.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.800+0000 7f7ffaa00700 1 -- 172.21.15.89:0/3998280649 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec003a60 con 0x7f80040cbb30 2024-04-02T09:09:22.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.802+0000 7f8009600700 1 -- 172.21.15.89:0/3998280649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80040ad0d0 con 0x7f80040cbb30 2024-04-02T09:09:22.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.803+0000 7f8009600700 1 -- 172.21.15.89:0/3998280649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80040c9400 con 0x7f80040cbb30 2024-04-02T09:09:22.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.804+0000 7f7ffaa00700 1 -- 172.21.15.89:0/3998280649 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 128) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f7fec003bc0 con 0x7f80040cbb30 2024-04-02T09:09:22.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.805+0000 7f7ffaa00700 1 --2- 172.21.15.89:0/3998280649 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7fe006ae30 0x7f7fe006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:22.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.805+0000 7f7ffaa00700 1 -- 172.21.15.89:0/3998280649 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6877+0+0 (secure 0 0 0) 0x7f7fec08a3e0 con 0x7f80040cbb30 2024-04-02T09:09:22.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.806+0000 7f8002a00700 1 -- 172.21.15.89:0/3998280649 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7fe006ae30 msgr2=0x7f7fe006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:22.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.806+0000 7f8002a00700 1 --2- 172.21.15.89:0/3998280649 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7fe006ae30 0x7f7fe006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:22.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:22.815+0000 7f7ffaa00700 1 -- 172.21.15.89:0/3998280649 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f80040c9400 con 0x7f80040cbb30 2024-04-02T09:09:22.911 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:22.913+0000 7f83a0cb5240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:09:23.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.007+0000 7f8002a00700 1 -- 172.21.15.89:0/3998280649 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7fe006ae30 msgr2=0x7f7fe006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:23.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.007+0000 7f8002a00700 1 --2- 172.21.15.89:0/3998280649 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7fe006ae30 0x7f7fe006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:23.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.126+0000 7f8009600700 1 -- 172.21.15.89:0/3998280649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f80040c9400 con 0x7f80040cbb30 2024-04-02T09:09:23.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.127+0000 7f7ffaa00700 1 -- 172.21.15.89:0/3998280649 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v191) v1 ==== 100+0+11 (secure 0 0 0) 0x7f80040c9400 con 0x7f80040cbb30 2024-04-02T09:09:23.126 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:09:23.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.132+0000 7f8009600700 1 -- 172.21.15.89:0/3998280649 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7fe006ae30 msgr2=0x7f7fe006d2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:23.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.132+0000 7f8009600700 1 --2- 172.21.15.89:0/3998280649 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7fe006ae30 0x7f7fe006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.132+0000 7f8009600700 1 -- 172.21.15.89:0/3998280649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80040cbb30 msgr2=0x7f800415de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:23.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.132+0000 7f8009600700 1 --2- 172.21.15.89:0/3998280649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80040cbb30 0x7f800415de70 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f7fec009c40 tx=0x7f7fec004750 comp rx=0 tx=0).stop 2024-04-02T09:09:23.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.132+0000 7f8009600700 1 -- 172.21.15.89:0/3998280649 shutdown_connections 2024-04-02T09:09:23.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.132+0000 7f8009600700 1 --2- 172.21.15.89:0/3998280649 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7fe006ae30 0x7f7fe006d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.132+0000 7f8009600700 1 --2- 172.21.15.89:0/3998280649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80040cd020 0x7f800414a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.132+0000 7f8009600700 1 --2- 172.21.15.89:0/3998280649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f80040b2070 0x7f80040ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.132+0000 7f8009600700 1 --2- 172.21.15.89:0/3998280649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80040cbb30 0x7f800415de70 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.132+0000 7f8009600700 1 -- 172.21.15.89:0/3998280649 >> 172.21.15.89:0/3998280649 conn(0x7f80040b0d70 msgr2=0x7f80040cc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:23.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.132+0000 7f8009600700 1 -- 172.21.15.89:0/3998280649 shutdown_connections 2024-04-02T09:09:23.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.133+0000 7f8009600700 1 -- 172.21.15.89:0/3998280649 wait complete. 2024-04-02T09:09:23.162 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T09:09:23.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.376+0000 7fbf34e00700 1 Processor -- start 2024-04-02T09:09:23.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.376+0000 7fbf34e00700 1 -- start start 2024-04-02T09:09:23.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.377+0000 7fbf34e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbf300cd5c0 0x7fbf3015e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:23.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.377+0000 7fbf34e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbf3006f310 0x7fbf3006f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:23.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.377+0000 7fbf34e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf3015cfe0 0x7fbf3015d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:23.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.377+0000 7fbf34e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3015eda0 con 0x7fbf300cd5c0 2024-04-02T09:09:23.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.378+0000 7fbf34e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3006fce0 con 0x7fbf3006f310 2024-04-02T09:09:23.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.378+0000 7fbf34e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf30099dd0 con 0x7fbf3015cfe0 2024-04-02T09:09:23.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.378+0000 7fbf2de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbf300cd5c0 0x7fbf3015e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:23.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.378+0000 7fbf2de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbf300cd5c0 0x7fbf3015e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51992/0 (socket says 172.21.15.89:51992) 2024-04-02T09:09:23.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.378+0000 7fbf2de00700 1 -- 172.21.15.89:0/1684899226 learned_addr learned my addr 172.21.15.89:0/1684899226 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:23.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.378+0000 7fbf27e00700 1 --2- 172.21.15.89:0/1684899226 >> v2:172.21.15.113:3300/0 conn(0x7fbf3006f310 0x7fbf3006f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:23.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.378+0000 7fbf2e800700 1 --2- 172.21.15.89:0/1684899226 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf3015cfe0 0x7fbf3015d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.378+0000 7fbf2de00700 1 -- 172.21.15.89:0/1684899226 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf3015cfe0 msgr2=0x7fbf3015d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.378+0000 7fbf2de00700 1 --2- 172.21.15.89:0/1684899226 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf3015cfe0 0x7fbf3015d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.378+0000 7fbf2de00700 1 -- 172.21.15.89:0/1684899226 >> v2:172.21.15.113:3300/0 conn(0x7fbf3006f310 msgr2=0x7fbf3006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.378+0000 7fbf2de00700 1 --2- 172.21.15.89:0/1684899226 >> v2:172.21.15.113:3300/0 conn(0x7fbf3006f310 0x7fbf3006f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.378+0000 7fbf2de00700 1 -- 172.21.15.89:0/1684899226 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3009a0b0 con 0x7fbf300cd5c0 2024-04-02T09:09:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.378+0000 7fbf2e800700 1 --2- 172.21.15.89:0/1684899226 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf3015cfe0 0x7fbf3015d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.378+0000 7fbf27e00700 1 --2- 172.21.15.89:0/1684899226 >> v2:172.21.15.113:3300/0 conn(0x7fbf3006f310 0x7fbf3006f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.379+0000 7fbf2de00700 1 --2- 172.21.15.89:0/1684899226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf300cd5c0 0x7fbf3015e860 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fbf200098f0 tx=0x7fbf20009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=854f09a7facc16d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.379+0000 7fbf2d400700 1 -- 172.21.15.89:0/1684899226 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2000b860 con 0x7fbf300cd5c0 2024-04-02T09:09:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.379+0000 7fbf2d400700 1 -- 172.21.15.89:0/1684899226 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2000b9c0 con 0x7fbf300cd5c0 2024-04-02T09:09:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.379+0000 7fbf2d400700 1 -- 172.21.15.89:0/1684899226 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2000bd50 con 0x7fbf300cd5c0 2024-04-02T09:09:23.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.379+0000 7fbf34e00700 1 -- 172.21.15.89:0/1684899226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf300cd5c0 msgr2=0x7fbf3015e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:23.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.379+0000 7fbf34e00700 1 --2- 172.21.15.89:0/1684899226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf300cd5c0 0x7fbf3015e860 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fbf200098f0 tx=0x7fbf20009c00 comp rx=0 tx=0).stop 2024-04-02T09:09:23.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.379+0000 7fbf34e00700 1 -- 172.21.15.89:0/1684899226 shutdown_connections 2024-04-02T09:09:23.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.379+0000 7fbf34e00700 1 --2- 172.21.15.89:0/1684899226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf300cd5c0 0x7fbf3015e860 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.379+0000 7fbf34e00700 1 --2- 172.21.15.89:0/1684899226 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf3015cfe0 0x7fbf3015d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.379+0000 7fbf34e00700 1 --2- 172.21.15.89:0/1684899226 >> v2:172.21.15.113:3300/0 conn(0x7fbf3006f310 0x7fbf3006f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.379+0000 7fbf34e00700 1 -- 172.21.15.89:0/1684899226 >> 172.21.15.89:0/1684899226 conn(0x7fbf300d1370 msgr2=0x7fbf300d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:23.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.379+0000 7fbf34e00700 1 -- 172.21.15.89:0/1684899226 shutdown_connections 2024-04-02T09:09:23.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.380+0000 7fbf34e00700 1 -- 172.21.15.89:0/1684899226 wait complete. 2024-04-02T09:09:23.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.382+0000 7fbf34e00700 1 Processor -- start 2024-04-02T09:09:23.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.382+0000 7fbf34e00700 1 -- start start 2024-04-02T09:09:23.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.383+0000 7fbf34e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf3006f310 0x7fbf300c5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:23.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.383+0000 7fbf34e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf300cd5c0 0x7fbf301547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:23.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.383+0000 7fbf34e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3015cfe0 0x7fbf300aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:23.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.383+0000 7fbf34e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf300c5960 con 0x7fbf3006f310 2024-04-02T09:09:23.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.383+0000 7fbf34e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf300b45e0 con 0x7fbf3015cfe0 2024-04-02T09:09:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.383+0000 7fbf34e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf300b48c0 con 0x7fbf300cd5c0 2024-04-02T09:09:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.383+0000 7fbf2de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf3006f310 0x7fbf300c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.383+0000 7fbf2de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf3006f310 0x7fbf300c5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52008/0 (socket says 172.21.15.89:52008) 2024-04-02T09:09:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.383+0000 7fbf2de00700 1 -- 172.21.15.89:0/2183306819 learned_addr learned my addr 172.21.15.89:0/2183306819 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.384+0000 7fbf27e00700 1 --2- 172.21.15.89:0/2183306819 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf300cd5c0 0x7fbf301547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.384+0000 7fbf2e800700 1 --2- 172.21.15.89:0/2183306819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3015cfe0 0x7fbf300aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.384+0000 7fbf2de00700 1 -- 172.21.15.89:0/2183306819 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf300cd5c0 msgr2=0x7fbf301547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.384+0000 7fbf2de00700 1 --2- 172.21.15.89:0/2183306819 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf300cd5c0 0x7fbf301547d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.384+0000 7fbf2de00700 1 -- 172.21.15.89:0/2183306819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3015cfe0 msgr2=0x7fbf300aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.384+0000 7fbf2de00700 1 --2- 172.21.15.89:0/2183306819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3015cfe0 0x7fbf300aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.384+0000 7fbf2de00700 1 -- 172.21.15.89:0/2183306819 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf20009550 con 0x7fbf3006f310 2024-04-02T09:09:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.384+0000 7fbf2de00700 1 --2- 172.21.15.89:0/2183306819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf3006f310 0x7fbf300c5420 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fbf20000c00 tx=0x7fbf20004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:23.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.384+0000 7fbf26000700 1 -- 172.21.15.89:0/2183306819 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20004240 con 0x7fbf3006f310 2024-04-02T09:09:23.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.385+0000 7fbf26000700 1 -- 172.21.15.89:0/2183306819 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2001f070 con 0x7fbf3006f310 2024-04-02T09:09:23.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.385+0000 7fbf26000700 1 -- 172.21.15.89:0/2183306819 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf200036a0 con 0x7fbf3006f310 2024-04-02T09:09:23.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.385+0000 7fbf34e00700 1 -- 172.21.15.89:0/2183306819 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf300b4ba0 con 0x7fbf3006f310 2024-04-02T09:09:23.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.385+0000 7fbf34e00700 1 -- 172.21.15.89:0/2183306819 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf300b5160 con 0x7fbf3006f310 2024-04-02T09:09:23.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.386+0000 7fbf34e00700 1 -- 172.21.15.89:0/2183306819 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf30067720 con 0x7fbf3006f310 2024-04-02T09:09:23.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.387+0000 7fbf26000700 1 -- 172.21.15.89:0/2183306819 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 128) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fbf20021030 con 0x7fbf3006f310 2024-04-02T09:09:23.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.389+0000 7fbf26000700 1 --2- 172.21.15.89:0/2183306819 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbf0c06adb0 0x7fbf0c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:23.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.389+0000 7fbf27e00700 1 -- 172.21.15.89:0/2183306819 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbf0c06adb0 msgr2=0x7fbf0c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:23.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.389+0000 7fbf26000700 1 -- 172.21.15.89:0/2183306819 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6877+0+0 (secure 0 0 0) 0x7fbf2008abb0 con 0x7fbf3006f310 2024-04-02T09:09:23.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.389+0000 7fbf27e00700 1 --2- 172.21.15.89:0/2183306819 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbf0c06adb0 0x7fbf0c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:23.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.393+0000 7fbf26000700 1 -- 172.21.15.89:0/2183306819 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf20055910 con 0x7fbf3006f310 2024-04-02T09:09:23.559 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:23.560+0000 7ff0714f6240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:09:23.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.590+0000 7fbf27e00700 1 -- 172.21.15.89:0/2183306819 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbf0c06adb0 msgr2=0x7fbf0c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:23.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.590+0000 7fbf27e00700 1 --2- 172.21.15.89:0/2183306819 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbf0c06adb0 0x7fbf0c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:23.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.652+0000 7fbf34e00700 1 -- 172.21.15.89:0/2183306819 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbf300d0e20 con 0x7fbf3006f310 2024-04-02T09:09:23.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.653+0000 7fbf26000700 1 -- 172.21.15.89:0/2183306819 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 14 entries v191) v1 ==== 81+0+806 (secure 0 0 0) 0x7fbf20015020 con 0x7fbf3006f310 2024-04-02T09:09:23.652 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/4284766432 2024-04-03T09:04:23.614732+0000 2024-04-02T09:09:23.652 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3938746027 2024-04-03T09:04:23.614732+0000 2024-04-02T09:09:23.652 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3350462335 2024-04-03T09:04:23.614732+0000 2024-04-02T09:09:23.652 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3070765417 2024-04-03T09:04:23.614732+0000 2024-04-02T09:09:23.652 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/4154199909 2024-04-03T09:04:23.614732+0000 2024-04-02T09:09:23.652 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2003321918 2024-04-03T09:04:23.614732+0000 2024-04-02T09:09:23.652 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2504651954 2024-04-03T08:53:39.182240+0000 2024-04-02T09:09:23.653 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/978973291 2024-04-03T08:53:39.182240+0000 2024-04-02T09:09:23.653 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/43357 2024-04-03T08:53:39.182240+0000 2024-04-02T09:09:23.653 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/43357 2024-04-03T08:53:39.182240+0000 2024-04-02T09:09:23.653 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1859816274 2024-04-03T08:53:39.182240+0000 2024-04-02T09:09:23.653 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/477599566 2024-04-03T08:53:39.182240+0000 2024-04-02T09:09:23.653 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3314137116 2024-04-03T08:53:39.182240+0000 2024-04-02T09:09:23.653 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2444652307 2024-04-03T08:53:39.182240+0000 2024-04-02T09:09:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.655+0000 7fbf34e00700 1 -- 172.21.15.89:0/2183306819 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbf0c06adb0 msgr2=0x7fbf0c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.655+0000 7fbf34e00700 1 --2- 172.21.15.89:0/2183306819 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbf0c06adb0 0x7fbf0c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.655+0000 7fbf34e00700 1 -- 172.21.15.89:0/2183306819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf3006f310 msgr2=0x7fbf300c5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.655+0000 7fbf34e00700 1 --2- 172.21.15.89:0/2183306819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf3006f310 0x7fbf300c5420 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fbf20000c00 tx=0x7fbf20004750 comp rx=0 tx=0).stop 2024-04-02T09:09:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.655+0000 7fbf34e00700 1 -- 172.21.15.89:0/2183306819 shutdown_connections 2024-04-02T09:09:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.655+0000 7fbf34e00700 1 --2- 172.21.15.89:0/2183306819 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbf0c06adb0 0x7fbf0c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.655+0000 7fbf34e00700 1 --2- 172.21.15.89:0/2183306819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3015cfe0 0x7fbf300aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.655+0000 7fbf34e00700 1 --2- 172.21.15.89:0/2183306819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf3006f310 0x7fbf300c5420 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.655+0000 7fbf34e00700 1 --2- 172.21.15.89:0/2183306819 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf300cd5c0 0x7fbf301547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.655+0000 7fbf34e00700 1 -- 172.21.15.89:0/2183306819 >> 172.21.15.89:0/2183306819 conn(0x7fbf300d1370 msgr2=0x7fbf3006b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:23.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.655+0000 7fbf34e00700 1 -- 172.21.15.89:0/2183306819 shutdown_connections 2024-04-02T09:09:23.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:23.655+0000 7fbf34e00700 1 -- 172.21.15.89:0/2183306819 wait complete. 2024-04-02T09:09:23.656 INFO:teuthology.orchestra.run.smithi089.stderr:listed 14 entries 2024-04-02T09:09:23.663 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:23.665+0000 7ff0714f6240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:09:23.672 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T09:09:23.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:23.672 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T09:09:23.747 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T09:09:23.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:23.747 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T09:09:23.781 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:23.783+0000 7f83a0cb5240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:09:23.791 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:09:23.791 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:09:23.829 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:23.830+0000 7ff0714f6240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:09:23.841 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:09:23.842 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:09:23.842 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:09:23.881 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:23.883+0000 7f83a0cb5240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:09:23.887 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:09:23.887 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:09:23.887 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:09:23.935 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:09:23.935 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:09:24.051 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:24.052+0000 7ff0714f6240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:09:24.054 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:24.056+0000 7f83a0cb5240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:09:24.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.134+0000 7f7905400700 1 Processor -- start 2024-04-02T09:09:24.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.134+0000 7f7905400700 1 -- start start 2024-04-02T09:09:24.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.136+0000 7f7905400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f790015b5d0 0x7f79000ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:24.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.136+0000 7f7905400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f79000ac7d0 0x7f790006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:24.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.136+0000 7f7905400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790015fd50 con 0x7f79000cb3f0 2024-04-02T09:09:24.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.136+0000 7f7905400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f790015b9b0 con 0x7f79000ac7d0 2024-04-02T09:09:24.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.136+0000 7f7905400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f790015f140 con 0x7f790015b5d0 2024-04-02T09:09:24.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.136+0000 7f78ff200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f79000ac7d0 0x7f790006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:24.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.136+0000 7f78ff200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f79000ac7d0 0x7f790006c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52016/0 (socket says 172.21.15.89:52016) 2024-04-02T09:09:24.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.136+0000 7f78ff200700 1 -- 172.21.15.89:0/4147430306 learned_addr learned my addr 172.21.15.89:0/4147430306 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:24.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.136+0000 7f78fe800700 1 --2- 172.21.15.89:0/4147430306 >> v2:172.21.15.113:3300/0 conn(0x7f790015b5d0 0x7f79000ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:24.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.136+0000 7f78ff200700 1 -- 172.21.15.89:0/4147430306 >> v2:172.21.15.113:3300/0 conn(0x7f790015b5d0 msgr2=0x7f79000ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:24.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.136+0000 7f78ff200700 1 --2- 172.21.15.89:0/4147430306 >> v2:172.21.15.113:3300/0 conn(0x7f790015b5d0 0x7f79000ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.137+0000 7f78ff200700 1 -- 172.21.15.89:0/4147430306 >> v1:172.21.15.113:6789/0 conn(0x7f79000cb3f0 legacy=0x7f79000ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:24.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.137+0000 7f78ff200700 1 -- 172.21.15.89:0/4147430306 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790015f420 con 0x7f79000ac7d0 2024-04-02T09:09:24.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.137+0000 7f78fd400700 1 -- 172.21.15.89:0/4147430306 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687620691 0 0) 0x7f790015fd50 con 0x7f79000cb3f0 2024-04-02T09:09:24.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.137+0000 7f78ff200700 1 --2- 172.21.15.89:0/4147430306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79000ac7d0 0x7f790006c9d0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f790006b8b0 tx=0x7f78f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8752628bc640352c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:24.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.137+0000 7f78fd400700 1 -- 172.21.15.89:0/4147430306 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f800e070 con 0x7f79000ac7d0 2024-04-02T09:09:24.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.137+0000 7f78fd400700 1 -- 172.21.15.89:0/4147430306 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f800b910 con 0x7f79000ac7d0 2024-04-02T09:09:24.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.137+0000 7f78fd400700 1 -- 172.21.15.89:0/4147430306 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f800bcc0 con 0x7f79000ac7d0 2024-04-02T09:09:24.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.138+0000 7f7905400700 1 -- 172.21.15.89:0/4147430306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79000ac7d0 msgr2=0x7f790006c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:24.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.138+0000 7f7905400700 1 --2- 172.21.15.89:0/4147430306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79000ac7d0 0x7f790006c9d0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f790006b8b0 tx=0x7f78f800b330 comp rx=0 tx=0).stop 2024-04-02T09:09:24.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.138+0000 7f7905400700 1 -- 172.21.15.89:0/4147430306 shutdown_connections 2024-04-02T09:09:24.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.138+0000 7f7905400700 1 --2- 172.21.15.89:0/4147430306 >> v2:172.21.15.113:3300/0 conn(0x7f790015b5d0 0x7f79000ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.138+0000 7f7905400700 1 --2- 172.21.15.89:0/4147430306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79000ac7d0 0x7f790006c9d0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.138+0000 7f7905400700 1 -- 172.21.15.89:0/4147430306 >> 172.21.15.89:0/4147430306 conn(0x7f79000cf190 msgr2=0x7f79000cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:24.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.138+0000 7f7905400700 1 -- 172.21.15.89:0/4147430306 shutdown_connections 2024-04-02T09:09:24.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.138+0000 7f7905400700 1 -- 172.21.15.89:0/4147430306 wait complete. 2024-04-02T09:09:24.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.140+0000 7f7905400700 1 Processor -- start 2024-04-02T09:09:24.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.140+0000 7f7905400700 1 -- start start 2024-04-02T09:09:24.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.140+0000 7f7905400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79000ac7d0 0x7f7900156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f7905400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f790015b5d0 0x7f79000ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f7905400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7900071500 0x7f7900071980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f7905400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79000b3c40 con 0x7f7900071500 2024-04-02T09:09:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f7905400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7900071ec0 con 0x7f79000ac7d0 2024-04-02T09:09:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f7905400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7900157240 con 0x7f790015b5d0 2024-04-02T09:09:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f78ff200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7900071500 0x7f7900071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f78ff200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7900071500 0x7f7900071980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52020/0 (socket says 172.21.15.89:52020) 2024-04-02T09:09:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f78ff200700 1 -- 172.21.15.89:0/1292211074 learned_addr learned my addr 172.21.15.89:0/1292211074 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f78fe800700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79000ac7d0 0x7f7900156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f78fde00700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f790015b5d0 0x7f79000ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f78ff200700 1 -- 172.21.15.89:0/1292211074 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f790015b5d0 msgr2=0x7f79000ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f78ff200700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f790015b5d0 0x7f79000ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f78ff200700 1 -- 172.21.15.89:0/1292211074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79000ac7d0 msgr2=0x7f7900156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f78ff200700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79000ac7d0 0x7f7900156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f78ff200700 1 -- 172.21.15.89:0/1292211074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f8009b30 con 0x7f7900071500 2024-04-02T09:09:24.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.141+0000 7f78fe800700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79000ac7d0 0x7f7900156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:09:24.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.142+0000 7f78ff200700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7900071500 0x7f7900071980 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f790006b930 tx=0x7f78f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:24.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.142+0000 7f78f6a00700 1 -- 172.21.15.89:0/1292211074 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f8004840 con 0x7f7900071500 2024-04-02T09:09:24.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.142+0000 7f78f6a00700 1 -- 172.21.15.89:0/1292211074 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f80099b0 con 0x7f7900071500 2024-04-02T09:09:24.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.142+0000 7f7905400700 1 -- 172.21.15.89:0/1292211074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79000b2fc0 con 0x7f7900071500 2024-04-02T09:09:24.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.142+0000 7f78f6a00700 1 -- 172.21.15.89:0/1292211074 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f801c680 con 0x7f7900071500 2024-04-02T09:09:24.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.142+0000 7f7905400700 1 -- 172.21.15.89:0/1292211074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79000b3580 con 0x7f7900071500 2024-04-02T09:09:24.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.143+0000 7f7905400700 1 -- 172.21.15.89:0/1292211074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79000b3d80 con 0x7f7900071500 2024-04-02T09:09:24.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.145+0000 7f78f6a00700 1 -- 172.21.15.89:0/1292211074 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 128) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f78f801c7e0 con 0x7f7900071500 2024-04-02T09:09:24.143 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:24.144+0000 7ff0714f6240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:09:24.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.145+0000 7f78f6a00700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f78e006adb0 0x7f78e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:24.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.146+0000 7f78f6a00700 1 -- 172.21.15.89:0/1292211074 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6861+0+0 (secure 0 0 0) 0x7f78f808b620 con 0x7f7900071500 2024-04-02T09:09:24.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.146+0000 7f78fe800700 1 -- 172.21.15.89:0/1292211074 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f78e006adb0 msgr2=0x7f78e006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:24.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.146+0000 7f78fe800700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f78e006adb0 0x7f78e006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:24.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.150+0000 7f78f6a00700 1 -- 172.21.15.89:0/1292211074 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f79000b3d80 con 0x7f7900071500 2024-04-02T09:09:24.242 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:24.243+0000 7ff0714f6240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:09:24.279 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:24.281+0000 7f83a0cb5240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:09:24.340 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:24.342+0000 7ff0714f6240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:09:24.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.347+0000 7f78fe800700 1 -- 172.21.15.89:0/1292211074 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f78e006adb0 msgr2=0x7f78e006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:24.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.347+0000 7f78fe800700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f78e006adb0 0x7f78e006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:24.374 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:24.376+0000 7f83a0cb5240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:09:24.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.413+0000 7f7905400700 1 -- 172.21.15.89:0/1292211074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79000b3d80 con 0x7f7900071500 2024-04-02T09:09:24.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.414+0000 7f78f6a00700 1 -- 172.21.15.89:0/1292211074 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f79000b3d80 con 0x7f7900071500 2024-04-02T09:09:24.412 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:24.412 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1296569976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1296569976},{"type":"v1","addr":"172.21.15.113:6835","nonce":1296569976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9331,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1328950634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1328950634},{"type":"v1","addr":"172.21.15.89:6837","nonce":1328950634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":10032,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1167691520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1167691520},{"type":"v1","addr":"172.21.15.113:6834","nonce":1167691520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T08:53:22.881600+0000","modified":"2024-04-02T08:53:26.390144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10038":{"gid":10038,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/583021497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":583021497},{"type":"v1","addr":"172.21.15.89:6836","nonce":583021497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-04-02T09:09:24.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.416+0000 7f7905400700 1 -- 172.21.15.89:0/1292211074 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f78e006adb0 msgr2=0x7f78e006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:24.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.416+0000 7f7905400700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f78e006adb0 0x7f78e006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.416+0000 7f7905400700 1 -- 172.21.15.89:0/1292211074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7900071500 msgr2=0x7f7900071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:24.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.416+0000 7f7905400700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7900071500 0x7f7900071980 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f790006b930 tx=0x7f78f8004ad0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.417+0000 7f7905400700 1 -- 172.21.15.89:0/1292211074 shutdown_connections 2024-04-02T09:09:24.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.417+0000 7f7905400700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f78e006adb0 0x7f78e006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.417+0000 7f7905400700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79000ac7d0 0x7f7900156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.417+0000 7f7905400700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7900071500 0x7f7900071980 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.417+0000 7f7905400700 1 --2- 172.21.15.89:0/1292211074 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f790015b5d0 0x7f79000ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.417+0000 7f7905400700 1 -- 172.21.15.89:0/1292211074 >> 172.21.15.89:0/1292211074 conn(0x7f79000cf190 msgr2=0x7f79000aee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:24.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.417+0000 7f7905400700 1 -- 172.21.15.89:0/1292211074 shutdown_connections 2024-04-02T09:09:24.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.417+0000 7f7905400700 1 -- 172.21.15.89:0/1292211074 wait complete. 2024-04-02T09:09:24.416 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 89 2024-04-02T09:09:24.434 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:09:24.441 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:24.442+0000 7ff0714f6240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:09:24.487 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:24.489+0000 7f83a0cb5240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:09:24.593 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:24.595+0000 7f83a0cb5240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:09:24.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.632+0000 7fd20e400700 1 Processor -- start 2024-04-02T09:09:24.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.632+0000 7fd20e400700 1 -- start start 2024-04-02T09:09:24.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.633+0000 7fd20e400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd20815da50 0x7fd2080ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:24.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.633+0000 7fd20e400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd2080cec70 0x7fd2080d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:24.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.633+0000 7fd20e400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2080d0b20 0x7fd2080b52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:24.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.633+0000 7fd20e400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2080d0fd0 con 0x7fd20815da50 2024-04-02T09:09:24.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.633+0000 7fd20e400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2080b95d0 con 0x7fd2080cec70 2024-04-02T09:09:24.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.633+0000 7fd20e400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd20815de30 con 0x7fd2080d0b20 2024-04-02T09:09:24.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.633+0000 7fd207400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd20815da50 0x7fd2080ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:24.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.633+0000 7fd207400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd20815da50 0x7fd2080ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52032/0 (socket says 172.21.15.89:52032) 2024-04-02T09:09:24.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.633+0000 7fd207400700 1 -- 172.21.15.89:0/479383679 learned_addr learned my addr 172.21.15.89:0/479383679 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:24.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.633+0000 7fd207e00700 1 --2- 172.21.15.89:0/479383679 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2080d0b20 0x7fd2080b52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:24.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.634+0000 7fd206a00700 1 --2- 172.21.15.89:0/479383679 >> v2:172.21.15.113:3300/0 conn(0x7fd2080cec70 0x7fd2080d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:24.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.634+0000 7fd207400700 1 -- 172.21.15.89:0/479383679 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2080d0b20 msgr2=0x7fd2080b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:24.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.634+0000 7fd207400700 1 --2- 172.21.15.89:0/479383679 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2080d0b20 0x7fd2080b52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.634+0000 7fd207400700 1 -- 172.21.15.89:0/479383679 >> v2:172.21.15.113:3300/0 conn(0x7fd2080cec70 msgr2=0x7fd2080d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:24.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.634+0000 7fd207400700 1 --2- 172.21.15.89:0/479383679 >> v2:172.21.15.113:3300/0 conn(0x7fd2080cec70 0x7fd2080d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.634+0000 7fd207400700 1 -- 172.21.15.89:0/479383679 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2080d24e0 con 0x7fd20815da50 2024-04-02T09:09:24.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.634+0000 7fd207400700 1 --2- 172.21.15.89:0/479383679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd20815da50 0x7fd2080ce730 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fd2080740a0 tx=0x7fd1f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4edbb05f36c2eff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:24.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.634+0000 7fd206000700 1 -- 172.21.15.89:0/479383679 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8011830 con 0x7fd20815da50 2024-04-02T09:09:24.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.634+0000 7fd206000700 1 -- 172.21.15.89:0/479383679 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f8011990 con 0x7fd20815da50 2024-04-02T09:09:24.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.634+0000 7fd206000700 1 -- 172.21.15.89:0/479383679 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8011d20 con 0x7fd20815da50 2024-04-02T09:09:24.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.635+0000 7fd20e400700 1 -- 172.21.15.89:0/479383679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd20815da50 msgr2=0x7fd2080ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:24.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.635+0000 7fd20e400700 1 --2- 172.21.15.89:0/479383679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd20815da50 0x7fd2080ce730 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fd2080740a0 tx=0x7fd1f8009bd0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.635+0000 7fd20e400700 1 -- 172.21.15.89:0/479383679 shutdown_connections 2024-04-02T09:09:24.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.635+0000 7fd20e400700 1 --2- 172.21.15.89:0/479383679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd20815da50 0x7fd2080ce730 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.635+0000 7fd20e400700 1 --2- 172.21.15.89:0/479383679 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2080d0b20 0x7fd2080b52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.635+0000 7fd20e400700 1 --2- 172.21.15.89:0/479383679 >> v2:172.21.15.113:3300/0 conn(0x7fd2080cec70 0x7fd2080d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.635+0000 7fd20e400700 1 -- 172.21.15.89:0/479383679 >> 172.21.15.89:0/479383679 conn(0x7fd2080666a0 msgr2=0x7fd208066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:24.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.635+0000 7fd20e400700 1 -- 172.21.15.89:0/479383679 shutdown_connections 2024-04-02T09:09:24.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.635+0000 7fd20e400700 1 -- 172.21.15.89:0/479383679 wait complete. 2024-04-02T09:09:24.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.637+0000 7fd20e400700 1 Processor -- start 2024-04-02T09:09:24.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.637+0000 7fd20e400700 1 -- start start 2024-04-02T09:09:24.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd20e400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2080cec70 0x7fd2080cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:24.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd20e400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2080d0b20 0x7fd2081504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:24.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd20e400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd20815da50 0x7fd20815df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:24.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd20e400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20815f2d0 con 0x7fd2080cec70 2024-04-02T09:09:24.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd20e400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2080cd8a0 con 0x7fd2080d0b20 2024-04-02T09:09:24.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd20e400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2081509e0 con 0x7fd20815da50 2024-04-02T09:09:24.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd207400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2080cec70 0x7fd2080cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:24.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd207400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2080cec70 0x7fd2080cd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52038/0 (socket says 172.21.15.89:52038) 2024-04-02T09:09:24.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd207400700 1 -- 172.21.15.89:0/2696761969 learned_addr learned my addr 172.21.15.89:0/2696761969 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:24.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd207e00700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd20815da50 0x7fd20815df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:24.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd206a00700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2080d0b20 0x7fd2081504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:24.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd207400700 1 -- 172.21.15.89:0/2696761969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd20815da50 msgr2=0x7fd20815df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:24.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd207400700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd20815da50 0x7fd20815df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd207400700 1 -- 172.21.15.89:0/2696761969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2080d0b20 msgr2=0x7fd2081504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:24.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd207400700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2080d0b20 0x7fd2081504a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd207400700 1 -- 172.21.15.89:0/2696761969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f8009550 con 0x7fd2080cec70 2024-04-02T09:09:24.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.638+0000 7fd206a00700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2080d0b20 0x7fd2081504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:24.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.639+0000 7fd207400700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2080cec70 0x7fd2080cd2c0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009c10 tx=0x7fd1f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:24.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.639+0000 7fd1ff400700 1 -- 172.21.15.89:0/2696761969 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8004450 con 0x7fd2080cec70 2024-04-02T09:09:24.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.639+0000 7fd1ff400700 1 -- 172.21.15.89:0/2696761969 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f801e070 con 0x7fd2080cec70 2024-04-02T09:09:24.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.639+0000 7fd20e400700 1 -- 172.21.15.89:0/2696761969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2080cc230 con 0x7fd2080cec70 2024-04-02T09:09:24.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.639+0000 7fd1ff400700 1 -- 172.21.15.89:0/2696761969 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f80038e0 con 0x7fd2080cec70 2024-04-02T09:09:24.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.639+0000 7fd20e400700 1 -- 172.21.15.89:0/2696761969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2080cb750 con 0x7fd2080cec70 2024-04-02T09:09:24.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.640+0000 7fd20e400700 1 -- 172.21.15.89:0/2696761969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd208067720 con 0x7fd2080cec70 2024-04-02T09:09:24.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.641+0000 7fd1ff400700 1 -- 172.21.15.89:0/2696761969 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 128) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fd1f8003a40 con 0x7fd2080cec70 2024-04-02T09:09:24.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.642+0000 7fd1ff400700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd1e806adb0 0x7fd1e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:24.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.643+0000 7fd1ff400700 1 -- 172.21.15.89:0/2696761969 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6861+0+0 (secure 0 0 0) 0x7fd1f808a660 con 0x7fd2080cec70 2024-04-02T09:09:24.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.643+0000 7fd206a00700 1 -- 172.21.15.89:0/2696761969 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd1e806adb0 msgr2=0x7fd1e806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:24.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.643+0000 7fd206a00700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd1e806adb0 0x7fd1e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:24.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.647+0000 7fd1ff400700 1 -- 172.21.15.89:0/2696761969 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd1f8055340 con 0x7fd2080cec70 2024-04-02T09:09:24.655 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:24.656+0000 7ff0714f6240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:09:24.701 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:24.703+0000 7f83a0cb5240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:09:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.844+0000 7fd206a00700 1 -- 172.21.15.89:0/2696761969 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd1e806adb0 msgr2=0x7fd1e806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.844+0000 7fd206a00700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd1e806adb0 0x7fd1e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:24.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.861+0000 7fd20e400700 1 -- 172.21.15.89:0/2696761969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd208154f10 con 0x7fd2080cec70 2024-04-02T09:09:24.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.863+0000 7fd1ff400700 1 -- 172.21.15.89:0/2696761969 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+17851 (secure 0 0 0) 0x7fd1f8058960 con 0x7fd2080cec70 2024-04-02T09:09:24.861 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:24.862 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":192,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:09:23.795832+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-02T08:53:20.736248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-02T08:53:21.860376+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":192,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"23.b","mappings":[{"from":5,"to":3}]},{"pgid":"23.1d","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/4284766432":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/3938746027":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/3350462335":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/3070765417":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/4154199909":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/2003321918":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/2504651954":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/978973291":"2024-04-03T08:53:39.182240+0000","172.21.15.89:6833/43357":"2024-04-03T08:53:39.182240+0000","172.21.15.89:6832/43357":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/1859816274":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/477599566":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/3314137116":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/2444652307":"2024-04-03T08:53:39.182240+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:09:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.865+0000 7fd20e400700 1 -- 172.21.15.89:0/2696761969 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd1e806adb0 msgr2=0x7fd1e806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.865+0000 7fd20e400700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd1e806adb0 0x7fd1e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.865+0000 7fd20e400700 1 -- 172.21.15.89:0/2696761969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2080cec70 msgr2=0x7fd2080cd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.865+0000 7fd20e400700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2080cec70 0x7fd2080cd2c0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009c10 tx=0x7fd1f8004750 comp rx=0 tx=0).stop 2024-04-02T09:09:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.865+0000 7fd20e400700 1 -- 172.21.15.89:0/2696761969 shutdown_connections 2024-04-02T09:09:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.866+0000 7fd20e400700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd20815da50 0x7fd20815df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.866+0000 7fd20e400700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2080cec70 0x7fd2080cd2c0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.866+0000 7fd20e400700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd1e806adb0 0x7fd1e806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.866+0000 7fd20e400700 1 --2- 172.21.15.89:0/2696761969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2080d0b20 0x7fd2081504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:24.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.866+0000 7fd20e400700 1 -- 172.21.15.89:0/2696761969 >> 172.21.15.89:0/2696761969 conn(0x7fd2080666a0 msgr2=0x7fd2080b6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:24.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.866+0000 7fd20e400700 1 -- 172.21.15.89:0/2696761969 shutdown_connections 2024-04-02T09:09:24.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:24.866+0000 7fd20e400700 1 -- 172.21.15.89:0/2696761969 wait complete. 2024-04-02T09:09:24.883 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T08:53:20.736248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T08:53:21.860376+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '192', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 192, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:09:24.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:09:24.910 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:24.912+0000 7ff0714f6240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:09:24.927 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:24.929+0000 7f83a0cb5240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:09:25.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.089+0000 7f618d800700 1 Processor -- start 2024-04-02T09:09:25.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.089+0000 7f618d800700 1 -- start start 2024-04-02T09:09:25.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.090+0000 7f618d800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6188157f90 0x7f61880b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:25.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.090+0000 7f618d800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f61881613e0 0x7f6188161890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:25.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.090+0000 7f618d800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61880b3070 con 0x7f61880b3420 2024-04-02T09:09:25.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.090+0000 7f618d800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6188158370 con 0x7f61881613e0 2024-04-02T09:09:25.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.090+0000 7f618d800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61880b2710 con 0x7f6188157f90 2024-04-02T09:09:25.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.090+0000 7f6186000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f61881613e0 0x7f6188161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:25.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.090+0000 7f6186000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f61881613e0 0x7f6188161890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52044/0 (socket says 172.21.15.89:52044) 2024-04-02T09:09:25.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.090+0000 7f6186000700 1 -- 172.21.15.89:0/1087378548 learned_addr learned my addr 172.21.15.89:0/1087378548 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:25.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.091+0000 7f6186a00700 1 --2- 172.21.15.89:0/1087378548 >> v2:172.21.15.113:3300/0 conn(0x7f6188157f90 0x7f61880b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:25.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.091+0000 7f6186000700 1 -- 172.21.15.89:0/1087378548 >> v2:172.21.15.113:3300/0 conn(0x7f6188157f90 msgr2=0x7f61880b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:25.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.091+0000 7f6186000700 1 --2- 172.21.15.89:0/1087378548 >> v2:172.21.15.113:3300/0 conn(0x7f6188157f90 0x7f61880b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.091+0000 7f6186000700 1 -- 172.21.15.89:0/1087378548 >> v1:172.21.15.113:6789/0 conn(0x7f61880b3420 legacy=0x7f61880b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:25.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.091+0000 7f6186000700 1 -- 172.21.15.89:0/1087378548 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61880b2070 con 0x7f61881613e0 2024-04-02T09:09:25.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.091+0000 7f6186a00700 1 --2- 172.21.15.89:0/1087378548 >> v2:172.21.15.113:3300/0 conn(0x7f6188157f90 0x7f61880b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:25.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.091+0000 7f6186000700 1 --2- 172.21.15.89:0/1087378548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61881613e0 0x7f6188161890 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f618806fd80 tx=0x7f6174009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ff69779ace51dc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:25.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.091+0000 7f6185600700 1 -- 172.21.15.89:0/1087378548 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617400b870 con 0x7f61881613e0 2024-04-02T09:09:25.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.091+0000 7f6185600700 1 -- 172.21.15.89:0/1087378548 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f617400b9d0 con 0x7f61881613e0 2024-04-02T09:09:25.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.091+0000 7f6185600700 1 -- 172.21.15.89:0/1087378548 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617400bd60 con 0x7f61881613e0 2024-04-02T09:09:25.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.092+0000 7f618d800700 1 -- 172.21.15.89:0/1087378548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61881613e0 msgr2=0x7f6188161890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:25.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.092+0000 7f618d800700 1 --2- 172.21.15.89:0/1087378548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61881613e0 0x7f6188161890 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f618806fd80 tx=0x7f6174009bd0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.092+0000 7f618d800700 1 -- 172.21.15.89:0/1087378548 shutdown_connections 2024-04-02T09:09:25.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.092+0000 7f618d800700 1 --2- 172.21.15.89:0/1087378548 >> v2:172.21.15.113:3300/0 conn(0x7f6188157f90 0x7f61880b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.092+0000 7f618d800700 1 --2- 172.21.15.89:0/1087378548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61881613e0 0x7f6188161890 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.092+0000 7f618d800700 1 -- 172.21.15.89:0/1087378548 >> 172.21.15.89:0/1087378548 conn(0x7f61880b0d70 msgr2=0x7f61880b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:25.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.092+0000 7f618d800700 1 -- 172.21.15.89:0/1087378548 shutdown_connections 2024-04-02T09:09:25.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.092+0000 7f618d800700 1 -- 172.21.15.89:0/1087378548 wait complete. 2024-04-02T09:09:25.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.094+0000 7f618d800700 1 Processor -- start 2024-04-02T09:09:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.095+0000 7f618d800700 1 -- start start 2024-04-02T09:09:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.096+0000 7f618d800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61880b3420 0x7f6188153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.096+0000 7f618d800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6188157f90 0x7f618806d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.096+0000 7f618d800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61881613e0 0x7f618806db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.096+0000 7f618d800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6188099380 con 0x7f6188157f90 2024-04-02T09:09:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.096+0000 7f618d800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6188099660 con 0x7f61880b3420 2024-04-02T09:09:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.096+0000 7f618d800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6188099940 con 0x7f61881613e0 2024-04-02T09:09:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.096+0000 7f6186000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6188157f90 0x7f618806d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.096+0000 7f6186000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6188157f90 0x7f618806d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52058/0 (socket says 172.21.15.89:52058) 2024-04-02T09:09:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.096+0000 7f6186000700 1 -- 172.21.15.89:0/3394043200 learned_addr learned my addr 172.21.15.89:0/3394043200 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.096+0000 7f6186a00700 1 --2- 172.21.15.89:0/3394043200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61880b3420 0x7f6188153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.096+0000 7f6187400700 1 --2- 172.21.15.89:0/3394043200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61881613e0 0x7f618806db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.097+0000 7f6186000700 1 -- 172.21.15.89:0/3394043200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61881613e0 msgr2=0x7f618806db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:25.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.097+0000 7f6186000700 1 --2- 172.21.15.89:0/3394043200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61881613e0 0x7f618806db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.097+0000 7f6186000700 1 -- 172.21.15.89:0/3394043200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61880b3420 msgr2=0x7f6188153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:25.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.097+0000 7f6186000700 1 --2- 172.21.15.89:0/3394043200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61880b3420 0x7f6188153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.097+0000 7f6186000700 1 -- 172.21.15.89:0/3394043200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6174009550 con 0x7f6188157f90 2024-04-02T09:09:25.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.097+0000 7f6186000700 1 --2- 172.21.15.89:0/3394043200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6188157f90 0x7f618806d650 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f6174000c00 tx=0x7f6174004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:25.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.097+0000 7f617ea00700 1 -- 172.21.15.89:0/3394043200 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6174004480 con 0x7f6188157f90 2024-04-02T09:09:25.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.097+0000 7f617ea00700 1 -- 172.21.15.89:0/3394043200 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61740045e0 con 0x7f6188157f90 2024-04-02T09:09:25.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.097+0000 7f618d800700 1 -- 172.21.15.89:0/3394043200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6188099c20 con 0x7f6188157f90 2024-04-02T09:09:25.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.097+0000 7f617ea00700 1 -- 172.21.15.89:0/3394043200 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61740038e0 con 0x7f6188157f90 2024-04-02T09:09:25.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.097+0000 7f618d800700 1 -- 172.21.15.89:0/3394043200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6188099f00 con 0x7f6188157f90 2024-04-02T09:09:25.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.098+0000 7f618d800700 1 -- 172.21.15.89:0/3394043200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618809b460 con 0x7f6188157f90 2024-04-02T09:09:25.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.099+0000 7f617ea00700 1 -- 172.21.15.89:0/3394043200 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 128) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f6174021070 con 0x7f6188157f90 2024-04-02T09:09:25.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.100+0000 7f617ea00700 1 --2- 172.21.15.89:0/3394043200 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f616806adb0 0x7f616806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:25.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.100+0000 7f6186a00700 1 -- 172.21.15.89:0/3394043200 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f616806adb0 msgr2=0x7f616806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:25.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.100+0000 7f617ea00700 1 -- 172.21.15.89:0/3394043200 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6861+0+0 (secure 0 0 0) 0x7f617408b150 con 0x7f6188157f90 2024-04-02T09:09:25.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.100+0000 7f6186a00700 1 --2- 172.21.15.89:0/3394043200 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f616806adb0 0x7f616806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:25.105 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:25.102+0000 7ff0714f6240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:09:25.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.111+0000 7f617ea00700 1 -- 172.21.15.89:0/3394043200 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f618809b460 con 0x7f6188157f90 2024-04-02T09:09:25.194 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:25.195+0000 7ff0714f6240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:09:25.198 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:25.199+0000 7f83a0cb5240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:09:25.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.301+0000 7f6186a00700 1 -- 172.21.15.89:0/3394043200 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f616806adb0 msgr2=0x7f616806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:25.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.301+0000 7f6186a00700 1 --2- 172.21.15.89:0/3394043200 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f616806adb0 0x7f616806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:25.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.323+0000 7f618d800700 1 -- 172.21.15.89:0/3394043200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f618809b840 con 0x7f6188157f90 2024-04-02T09:09:25.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.324+0000 7f617ea00700 1 -- 172.21.15.89:0/3394043200 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v192) v1 ==== 93+0+10 (secure 0 0 0) 0x7f618809b840 con 0x7f6188157f90 2024-04-02T09:09:25.322 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:09:25.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.326+0000 7f618d800700 1 -- 172.21.15.89:0/3394043200 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f616806adb0 msgr2=0x7f616806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:25.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.326+0000 7f618d800700 1 --2- 172.21.15.89:0/3394043200 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f616806adb0 0x7f616806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.326+0000 7f618d800700 1 -- 172.21.15.89:0/3394043200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6188157f90 msgr2=0x7f618806d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:25.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.326+0000 7f618d800700 1 --2- 172.21.15.89:0/3394043200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6188157f90 0x7f618806d650 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f6174000c00 tx=0x7f6174004750 comp rx=0 tx=0).stop 2024-04-02T09:09:25.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.326+0000 7f618d800700 1 -- 172.21.15.89:0/3394043200 shutdown_connections 2024-04-02T09:09:25.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.326+0000 7f618d800700 1 --2- 172.21.15.89:0/3394043200 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f616806adb0 0x7f616806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.326+0000 7f618d800700 1 --2- 172.21.15.89:0/3394043200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61880b3420 0x7f6188153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.326+0000 7f618d800700 1 --2- 172.21.15.89:0/3394043200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61881613e0 0x7f618806db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.326+0000 7f618d800700 1 --2- 172.21.15.89:0/3394043200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6188157f90 0x7f618806d650 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.326+0000 7f618d800700 1 -- 172.21.15.89:0/3394043200 >> 172.21.15.89:0/3394043200 conn(0x7f61880b0d70 msgr2=0x7f61880afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:25.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.327+0000 7f618d800700 1 -- 172.21.15.89:0/3394043200 shutdown_connections 2024-04-02T09:09:25.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.327+0000 7f618d800700 1 -- 172.21.15.89:0/3394043200 wait complete. 2024-04-02T09:09:25.343 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T09:09:25.400 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:25.402+0000 7f83a0cb5240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:09:25.504 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:25.505+0000 7f83a0cb5240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:09:25.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.584+0000 7f976be00700 1 Processor -- start 2024-04-02T09:09:25.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.585+0000 7f976be00700 1 -- start start 2024-04-02T09:09:25.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.587+0000 7f976be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97640d2540 0x7f97641687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:25.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.587+0000 7f976be00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9764161420 con 0x7f97640d16a0 2024-04-02T09:09:25.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.587+0000 7f976be00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9764161700 con 0x7f976415d9d0 2024-04-02T09:09:25.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.587+0000 7f976be00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97640d4340 con 0x7f97640d2540 2024-04-02T09:09:25.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.587+0000 7f9768a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f97640d16a0 0x7f97640d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45428/0 (socket says 172.21.15.89:45428) 2024-04-02T09:09:25.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.587+0000 7f9768a00700 1 -- 172.21.15.89:0/558215338 learned_addr learned my addr 172.21.15.89:0/558215338 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:25.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.587+0000 7f9769e00700 1 --2- 172.21.15.89:0/558215338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97640d2540 0x7f97641687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:25.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.587+0000 7f9763e00700 1 -- 172.21.15.89:0/558215338 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015449484 0 0) 0x7f9764161420 con 0x7f97640d16a0 2024-04-02T09:09:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.588+0000 7f9763e00700 1 -- 172.21.15.89:0/558215338 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f974c0035b0 con 0x7f97640d16a0 2024-04-02T09:09:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.588+0000 7f9763e00700 1 -- 172.21.15.89:0/558215338 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646109056 0 0) 0x7f9764161700 con 0x7f976415d9d0 2024-04-02T09:09:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.588+0000 7f9763e00700 1 -- 172.21.15.89:0/558215338 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9764161420 con 0x7f976415d9d0 2024-04-02T09:09:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.588+0000 7f9769e00700 1 -- 172.21.15.89:0/558215338 >> v1:172.21.15.113:6789/0 conn(0x7f976415d9d0 legacy=0x7f97640d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.588+0000 7f9769e00700 1 -- 172.21.15.89:0/558215338 >> v1:172.21.15.89:6789/0 conn(0x7f97640d16a0 legacy=0x7f97640d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.588+0000 7f9769e00700 1 -- 172.21.15.89:0/558215338 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97641619e0 con 0x7f97640d2540 2024-04-02T09:09:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.588+0000 7f9769e00700 1 --2- 172.21.15.89:0/558215338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97640d2540 0x7f97641687b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f975c009dc0 tx=0x7f975c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caf9cf4a814c6bba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.588+0000 7f9763e00700 1 -- 172.21.15.89:0/558215338 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c00e070 con 0x7f97640d2540 2024-04-02T09:09:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.588+0000 7f9763e00700 1 -- 172.21.15.89:0/558215338 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975c00bb20 con 0x7f97640d2540 2024-04-02T09:09:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.588+0000 7f9763e00700 1 -- 172.21.15.89:0/558215338 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c012430 con 0x7f97640d2540 2024-04-02T09:09:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.589+0000 7f976be00700 1 -- 172.21.15.89:0/558215338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97640d2540 msgr2=0x7f97641687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.589+0000 7f976be00700 1 --2- 172.21.15.89:0/558215338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97640d2540 0x7f97641687b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f975c009dc0 tx=0x7f975c00b330 comp rx=0 tx=0).stop 2024-04-02T09:09:25.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.589+0000 7f976be00700 1 -- 172.21.15.89:0/558215338 shutdown_connections 2024-04-02T09:09:25.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.589+0000 7f976be00700 1 --2- 172.21.15.89:0/558215338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97640d2540 0x7f97641687b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.589+0000 7f976be00700 1 -- 172.21.15.89:0/558215338 >> 172.21.15.89:0/558215338 conn(0x7f976408aff0 msgr2=0x7f97640d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:25.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.589+0000 7f976be00700 1 -- 172.21.15.89:0/558215338 shutdown_connections 2024-04-02T09:09:25.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.589+0000 7f976be00700 1 -- 172.21.15.89:0/558215338 wait complete. 2024-04-02T09:09:25.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.591+0000 7f976be00700 1 Processor -- start 2024-04-02T09:09:25.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.592+0000 7f976be00700 1 -- start start 2024-04-02T09:09:25.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f976be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97640d16a0 0x7f976415c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:25.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f976be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97640d2540 0x7f976415c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:25.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f976be00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f976415d9d0 0x7f976415f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:25.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f976be00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976415c560 con 0x7f97640d16a0 2024-04-02T09:09:25.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f976be00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976415e370 con 0x7f976415d9d0 2024-04-02T09:09:25.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f976be00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976415e650 con 0x7f97640d2540 2024-04-02T09:09:25.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f9769400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97640d16a0 0x7f976415c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:25.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f9768a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97640d2540 0x7f976415c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:25.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f9769400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97640d16a0 0x7f976415c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52070/0 (socket says 172.21.15.89:52070) 2024-04-02T09:09:25.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f9769400700 1 -- 172.21.15.89:0/3844140573 learned_addr learned my addr 172.21.15.89:0/3844140573 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:25.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f9769e00700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f976415d9d0 0x7f976415f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:25.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f9769400700 1 -- 172.21.15.89:0/3844140573 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97640d2540 msgr2=0x7f976415c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:25.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f9769400700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97640d2540 0x7f976415c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f9769400700 1 -- 172.21.15.89:0/3844140573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f976415d9d0 msgr2=0x7f976415f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:25.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f9769400700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f976415d9d0 0x7f976415f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f9769400700 1 -- 172.21.15.89:0/3844140573 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c009a70 con 0x7f97640d16a0 2024-04-02T09:09:25.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f9768a00700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97640d2540 0x7f976415c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:09:25.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.593+0000 7f9769e00700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f976415d9d0 0x7f976415f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:09:25.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.594+0000 7f9769400700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97640d16a0 0x7f976415c740 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f975800c500 tx=0x7f975800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:25.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.594+0000 7f9761600700 1 -- 172.21.15.89:0/3844140573 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97580052d0 con 0x7f97640d16a0 2024-04-02T09:09:25.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.594+0000 7f9761600700 1 -- 172.21.15.89:0/3844140573 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9758005450 con 0x7f97640d16a0 2024-04-02T09:09:25.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.594+0000 7f976be00700 1 -- 172.21.15.89:0/3844140573 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976415e930 con 0x7f97640d16a0 2024-04-02T09:09:25.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.594+0000 7f9761600700 1 -- 172.21.15.89:0/3844140573 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97580038c0 con 0x7f97640d16a0 2024-04-02T09:09:25.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.594+0000 7f976be00700 1 -- 172.21.15.89:0/3844140573 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f976415eef0 con 0x7f97640d16a0 2024-04-02T09:09:25.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.595+0000 7f976be00700 1 -- 172.21.15.89:0/3844140573 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97640d1ca0 con 0x7f97640d16a0 2024-04-02T09:09:25.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.596+0000 7f9761600700 1 -- 172.21.15.89:0/3844140573 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 128) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f9758003a20 con 0x7f97640d16a0 2024-04-02T09:09:25.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.597+0000 7f9761600700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f974406adb0 0x7f974406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:25.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.597+0000 7f9768a00700 1 -- 172.21.15.89:0/3844140573 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f974406adb0 msgr2=0x7f974406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:25.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.597+0000 7f9761600700 1 -- 172.21.15.89:0/3844140573 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6861+0+0 (secure 0 0 0) 0x7f9758088c00 con 0x7f97640d16a0 2024-04-02T09:09:25.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.597+0000 7f9768a00700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f974406adb0 0x7f974406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:25.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.605+0000 7f9761600700 1 -- 172.21.15.89:0/3844140573 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f975808f080 con 0x7f97640d16a0 2024-04-02T09:09:25.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.798+0000 7f9768a00700 1 -- 172.21.15.89:0/3844140573 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f974406adb0 msgr2=0x7f974406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:25.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.798+0000 7f9768a00700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f974406adb0 0x7f974406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.816+0000 7f976be00700 1 -- 172.21.15.89:0/3844140573 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9764067720 con 0x7f97640d16a0 2024-04-02T09:09:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.817+0000 7f9761600700 1 -- 172.21.15.89:0/3844140573 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v192) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9758017020 con 0x7f97640d16a0 2024-04-02T09:09:25.815 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:09:25.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.819+0000 7f976be00700 1 -- 172.21.15.89:0/3844140573 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f974406adb0 msgr2=0x7f974406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:25.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.819+0000 7f976be00700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f974406adb0 0x7f974406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.819+0000 7f976be00700 1 -- 172.21.15.89:0/3844140573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97640d16a0 msgr2=0x7f976415c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:25.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.819+0000 7f976be00700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97640d16a0 0x7f976415c740 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f975800c500 tx=0x7f975800c810 comp rx=0 tx=0).stop 2024-04-02T09:09:25.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.819+0000 7f976be00700 1 -- 172.21.15.89:0/3844140573 shutdown_connections 2024-04-02T09:09:25.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.819+0000 7f976be00700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f974406adb0 0x7f974406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.819+0000 7f976be00700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f976415d9d0 0x7f976415f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.819+0000 7f976be00700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97640d16a0 0x7f976415c740 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.819+0000 7f976be00700 1 --2- 172.21.15.89:0/3844140573 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f97640d2540 0x7f976415c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:25.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.819+0000 7f976be00700 1 -- 172.21.15.89:0/3844140573 >> 172.21.15.89:0/3844140573 conn(0x7f976408aff0 msgr2=0x7f97640b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:25.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.820+0000 7f976be00700 1 -- 172.21.15.89:0/3844140573 shutdown_connections 2024-04-02T09:09:25.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:25.820+0000 7f976be00700 1 -- 172.21.15.89:0/3844140573 wait complete. 2024-04-02T09:09:25.837 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T09:09:25.922 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:25.923+0000 7ff0714f6240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:09:26.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.077+0000 7fdfdfc00700 1 Processor -- start 2024-04-02T09:09:26.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.078+0000 7fdfdfc00700 1 -- start start 2024-04-02T09:09:26.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.079+0000 7fdfdfc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdfd8099a60 0x7fdfd80d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:26.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.079+0000 7fdfdfc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdfd815d990 0x7fdfd8099e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:26.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.079+0000 7fdfdfc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd80d0970 con 0x7fdfd809a750 2024-04-02T09:09:26.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.079+0000 7fdfdfc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd806c020 con 0x7fdfd8099a60 2024-04-02T09:09:26.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.079+0000 7fdfdfc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd809a380 con 0x7fdfd815d990 2024-04-02T09:09:26.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.079+0000 7fdfdd200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdfd8099a60 0x7fdfd80d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:26.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.079+0000 7fdfdd200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdfd8099a60 0x7fdfd80d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52082/0 (socket says 172.21.15.89:52082) 2024-04-02T09:09:26.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.079+0000 7fdfdd200700 1 -- 172.21.15.89:0/263317226 learned_addr learned my addr 172.21.15.89:0/263317226 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:26.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.079+0000 7fdfd7e00700 1 --2- 172.21.15.89:0/263317226 >> v2:172.21.15.113:3300/0 conn(0x7fdfd815d990 0x7fdfd8099e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:26.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.079+0000 7fdfdd200700 1 -- 172.21.15.89:0/263317226 >> v2:172.21.15.113:3300/0 conn(0x7fdfd815d990 msgr2=0x7fdfd8099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:26.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.079+0000 7fdfdd200700 1 --2- 172.21.15.89:0/263317226 >> v2:172.21.15.113:3300/0 conn(0x7fdfd815d990 0x7fdfd8099e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.079+0000 7fdfdd200700 1 -- 172.21.15.89:0/263317226 >> v1:172.21.15.113:6789/0 conn(0x7fdfd809a750 legacy=0x7fdfd806c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:26.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.079+0000 7fdfdd200700 1 -- 172.21.15.89:0/263317226 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd80c81a0 con 0x7fdfd8099a60 2024-04-02T09:09:26.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.079+0000 7fdfd7e00700 1 --2- 172.21.15.89:0/263317226 >> v2:172.21.15.113:3300/0 conn(0x7fdfd815d990 0x7fdfd8099e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:26.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.080+0000 7fdfdd200700 1 --2- 172.21.15.89:0/263317226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd8099a60 0x7fdfd80d0060 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fdfd80b2df0 tx=0x7fdfcc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921855d8aef8fb7b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:26.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.080+0000 7fdfd7400700 1 -- 172.21.15.89:0/263317226 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc00b870 con 0x7fdfd8099a60 2024-04-02T09:09:26.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.080+0000 7fdfd7400700 1 -- 172.21.15.89:0/263317226 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfcc00b9d0 con 0x7fdfd8099a60 2024-04-02T09:09:26.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.080+0000 7fdfd7400700 1 -- 172.21.15.89:0/263317226 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc00bd60 con 0x7fdfd8099a60 2024-04-02T09:09:26.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.081+0000 7fdfdfc00700 1 -- 172.21.15.89:0/263317226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd8099a60 msgr2=0x7fdfd80d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:26.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.081+0000 7fdfdfc00700 1 --2- 172.21.15.89:0/263317226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd8099a60 0x7fdfd80d0060 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fdfd80b2df0 tx=0x7fdfcc009bd0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.081+0000 7fdfdfc00700 1 -- 172.21.15.89:0/263317226 shutdown_connections 2024-04-02T09:09:26.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.081+0000 7fdfdfc00700 1 --2- 172.21.15.89:0/263317226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd8099a60 0x7fdfd80d0060 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.081+0000 7fdfdfc00700 1 --2- 172.21.15.89:0/263317226 >> v2:172.21.15.113:3300/0 conn(0x7fdfd815d990 0x7fdfd8099e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.081+0000 7fdfdfc00700 1 -- 172.21.15.89:0/263317226 >> 172.21.15.89:0/263317226 conn(0x7fdfd80b2e20 msgr2=0x7fdfd80b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:26.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.081+0000 7fdfdfc00700 1 -- 172.21.15.89:0/263317226 shutdown_connections 2024-04-02T09:09:26.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.081+0000 7fdfdfc00700 1 -- 172.21.15.89:0/263317226 wait complete. 2024-04-02T09:09:26.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.083+0000 7fdfdfc00700 1 Processor -- start 2024-04-02T09:09:26.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.083+0000 7fdfdfc00700 1 -- start start 2024-04-02T09:09:26.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfdfc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdfd8099a60 0x7fdfd80c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:26.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfdfc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd809a750 0x7fdfd80ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:26.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfdfc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdfd815d990 0x7fdfd8148310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:26.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfdfc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd80c7bf0 con 0x7fdfd809a750 2024-04-02T09:09:26.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfdfc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd80c6fe0 con 0x7fdfd815d990 2024-04-02T09:09:26.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfdfc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd80c72c0 con 0x7fdfd8099a60 2024-04-02T09:09:26.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfd7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd809a750 0x7fdfd80ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:26.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfd7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd809a750 0x7fdfd80ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52084/0 (socket says 172.21.15.89:52084) 2024-04-02T09:09:26.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfd7e00700 1 -- 172.21.15.89:0/3416183661 learned_addr learned my addr 172.21.15.89:0/3416183661 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:26.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfddc00700 1 --2- 172.21.15.89:0/3416183661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdfd815d990 0x7fdfd8148310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:26.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfdd200700 1 --2- 172.21.15.89:0/3416183661 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdfd8099a60 0x7fdfd80c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:26.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfd7e00700 1 -- 172.21.15.89:0/3416183661 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdfd8099a60 msgr2=0x7fdfd80c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:26.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfd7e00700 1 --2- 172.21.15.89:0/3416183661 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdfd8099a60 0x7fdfd80c66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfd7e00700 1 -- 172.21.15.89:0/3416183661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdfd815d990 msgr2=0x7fdfd8148310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:26.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfd7e00700 1 --2- 172.21.15.89:0/3416183661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdfd815d990 0x7fdfd8148310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.084+0000 7fdfd7e00700 1 -- 172.21.15.89:0/3416183661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc009550 con 0x7fdfd809a750 2024-04-02T09:09:26.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.085+0000 7fdfd7e00700 1 --2- 172.21.15.89:0/3416183661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd809a750 0x7fdfd80ccc00 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400e760 tx=0x7fdfc400ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:26.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.085+0000 7fdfd4c00700 1 -- 172.21.15.89:0/3416183661 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc400c070 con 0x7fdfd809a750 2024-04-02T09:09:26.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.085+0000 7fdfd4c00700 1 -- 172.21.15.89:0/3416183661 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc4013610 con 0x7fdfd809a750 2024-04-02T09:09:26.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.085+0000 7fdfd4c00700 1 -- 172.21.15.89:0/3416183661 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc40139c0 con 0x7fdfd809a750 2024-04-02T09:09:26.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.085+0000 7fdfdfc00700 1 -- 172.21.15.89:0/3416183661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd80c75a0 con 0x7fdfd809a750 2024-04-02T09:09:26.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.085+0000 7fdfdfc00700 1 -- 172.21.15.89:0/3416183661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd80c6c20 con 0x7fdfd809a750 2024-04-02T09:09:26.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.086+0000 7fdfdfc00700 1 -- 172.21.15.89:0/3416183661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd8067720 con 0x7fdfd809a750 2024-04-02T09:09:26.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.087+0000 7fdfd4c00700 1 -- 172.21.15.89:0/3416183661 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 128) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fdfc4013b20 con 0x7fdfd809a750 2024-04-02T09:09:26.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.088+0000 7fdfd4c00700 1 --2- 172.21.15.89:0/3416183661 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdfb806adb0 0x7fdfb806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:26.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.088+0000 7fdfd4c00700 1 -- 172.21.15.89:0/3416183661 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6861+0+0 (secure 0 0 0) 0x7fdfc4089730 con 0x7fdfd809a750 2024-04-02T09:09:26.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.088+0000 7fdfdd200700 1 -- 172.21.15.89:0/3416183661 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdfb806adb0 msgr2=0x7fdfb806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:26.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.088+0000 7fdfdd200700 1 --2- 172.21.15.89:0/3416183661 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdfb806adb0 0x7fdfb806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:26.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.092+0000 7fdfd4c00700 1 -- 172.21.15.89:0/3416183661 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdfc40541d0 con 0x7fdfd809a750 2024-04-02T09:09:26.134 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:26.135+0000 7ff0714f6240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:09:26.281 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:26.283+0000 7f83a0cb5240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:09:26.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.289+0000 7fdfdd200700 1 -- 172.21.15.89:0/3416183661 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdfb806adb0 msgr2=0x7fdfb806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:26.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.289+0000 7fdfdd200700 1 --2- 172.21.15.89:0/3416183661 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdfb806adb0 0x7fdfb806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:26.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.301+0000 7fdfdfc00700 1 -- 172.21.15.89:0/3416183661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdfd80d0f80 con 0x7fdfd809a750 2024-04-02T09:09:26.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.302+0000 7fdfd4c00700 1 -- 172.21.15.89:0/3416183661 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v192) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdfc40577f0 con 0x7fdfd809a750 2024-04-02T09:09:26.300 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:09:26.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.304+0000 7fdfdfc00700 1 -- 172.21.15.89:0/3416183661 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdfb806adb0 msgr2=0x7fdfb806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:26.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.304+0000 7fdfdfc00700 1 --2- 172.21.15.89:0/3416183661 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdfb806adb0 0x7fdfb806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.304+0000 7fdfdfc00700 1 -- 172.21.15.89:0/3416183661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd809a750 msgr2=0x7fdfd80ccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:26.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.304+0000 7fdfdfc00700 1 --2- 172.21.15.89:0/3416183661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd809a750 0x7fdfd80ccc00 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400e760 tx=0x7fdfc400ea70 comp rx=0 tx=0).stop 2024-04-02T09:09:26.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.305+0000 7fdfdfc00700 1 -- 172.21.15.89:0/3416183661 shutdown_connections 2024-04-02T09:09:26.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.305+0000 7fdfdfc00700 1 --2- 172.21.15.89:0/3416183661 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdfb806adb0 0x7fdfb806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.305+0000 7fdfdfc00700 1 --2- 172.21.15.89:0/3416183661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdfd815d990 0x7fdfd8148310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.305+0000 7fdfdfc00700 1 --2- 172.21.15.89:0/3416183661 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdfd8099a60 0x7fdfd80c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.305+0000 7fdfdfc00700 1 --2- 172.21.15.89:0/3416183661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfd809a750 0x7fdfd80ccc00 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.305+0000 7fdfdfc00700 1 -- 172.21.15.89:0/3416183661 >> 172.21.15.89:0/3416183661 conn(0x7fdfd80b2e20 msgr2=0x7fdfd80ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:26.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.305+0000 7fdfdfc00700 1 -- 172.21.15.89:0/3416183661 shutdown_connections 2024-04-02T09:09:26.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.305+0000 7fdfdfc00700 1 -- 172.21.15.89:0/3416183661 wait complete. 2024-04-02T09:09:26.323 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:09:26.338 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:26.340+0000 7ff0714f6240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:09:26.509 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:26.510+0000 7f83a0cb5240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:09:26.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.520+0000 7fb31dc00700 1 Processor -- start 2024-04-02T09:09:26.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.520+0000 7fb31dc00700 1 -- start start 2024-04-02T09:09:26.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.521+0000 7fb31dc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb31815d760 0x7fb3180d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.521+0000 7fb31dc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb31815db40 0x7fb3180cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.521+0000 7fb31dc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3180cf1f0 0x7fb3180cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.521+0000 7fb31dc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3180d1fb0 con 0x7fb31815d760 2024-04-02T09:09:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.521+0000 7fb31dc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3180cf660 con 0x7fb31815db40 2024-04-02T09:09:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.521+0000 7fb31dc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180d1330 con 0x7fb3180cf1f0 2024-04-02T09:09:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.521+0000 7fb316a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb31815d760 0x7fb3180d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.521+0000 7fb316a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb31815d760 0x7fb3180d23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52096/0 (socket says 172.21.15.89:52096) 2024-04-02T09:09:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.521+0000 7fb316a00700 1 -- 172.21.15.89:0/4119120300 learned_addr learned my addr 172.21.15.89:0/4119120300 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.522+0000 7fb316000700 1 --2- 172.21.15.89:0/4119120300 >> v2:172.21.15.113:3300/0 conn(0x7fb31815db40 0x7fb3180cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.522+0000 7fb317400700 1 --2- 172.21.15.89:0/4119120300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3180cf1f0 0x7fb3180cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.522+0000 7fb316a00700 1 -- 172.21.15.89:0/4119120300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3180cf1f0 msgr2=0x7fb3180cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.522+0000 7fb316a00700 1 --2- 172.21.15.89:0/4119120300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3180cf1f0 0x7fb3180cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.522+0000 7fb316a00700 1 -- 172.21.15.89:0/4119120300 >> v2:172.21.15.113:3300/0 conn(0x7fb31815db40 msgr2=0x7fb3180cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.522+0000 7fb316a00700 1 --2- 172.21.15.89:0/4119120300 >> v2:172.21.15.113:3300/0 conn(0x7fb31815db40 0x7fb3180cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.522+0000 7fb316a00700 1 -- 172.21.15.89:0/4119120300 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3180d1610 con 0x7fb31815d760 2024-04-02T09:09:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.522+0000 7fb317400700 1 --2- 172.21.15.89:0/4119120300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3180cf1f0 0x7fb3180cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.522+0000 7fb316a00700 1 --2- 172.21.15.89:0/4119120300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31815d760 0x7fb3180d23d0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fb31808c480 tx=0x7fb308009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad7614e39111fa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.522+0000 7fb315600700 1 -- 172.21.15.89:0/4119120300 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308011830 con 0x7fb31815d760 2024-04-02T09:09:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.522+0000 7fb315600700 1 -- 172.21.15.89:0/4119120300 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb308011990 con 0x7fb31815d760 2024-04-02T09:09:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.522+0000 7fb315600700 1 -- 172.21.15.89:0/4119120300 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308011d20 con 0x7fb31815d760 2024-04-02T09:09:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.523+0000 7fb31dc00700 1 -- 172.21.15.89:0/4119120300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31815d760 msgr2=0x7fb3180d23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.523+0000 7fb31dc00700 1 --2- 172.21.15.89:0/4119120300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31815d760 0x7fb3180d23d0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fb31808c480 tx=0x7fb308009bd0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.523+0000 7fb31dc00700 1 -- 172.21.15.89:0/4119120300 shutdown_connections 2024-04-02T09:09:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.523+0000 7fb31dc00700 1 --2- 172.21.15.89:0/4119120300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb31815d760 0x7fb3180d23d0 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.523+0000 7fb31dc00700 1 --2- 172.21.15.89:0/4119120300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3180cf1f0 0x7fb3180cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.523+0000 7fb31dc00700 1 --2- 172.21.15.89:0/4119120300 >> v2:172.21.15.113:3300/0 conn(0x7fb31815db40 0x7fb3180cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.523+0000 7fb31dc00700 1 -- 172.21.15.89:0/4119120300 >> 172.21.15.89:0/4119120300 conn(0x7fb31808aff0 msgr2=0x7fb3180d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.523+0000 7fb31dc00700 1 -- 172.21.15.89:0/4119120300 shutdown_connections 2024-04-02T09:09:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.523+0000 7fb31dc00700 1 -- 172.21.15.89:0/4119120300 wait complete. 2024-04-02T09:09:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.525+0000 7fb31dc00700 1 Processor -- start 2024-04-02T09:09:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.525+0000 7fb31dc00700 1 -- start start 2024-04-02T09:09:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.526+0000 7fb31dc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3180cf1f0 0x7fb3181f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.526+0000 7fb31dc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb31815d760 0x7fb3180c8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.526+0000 7fb31dc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb31815db40 0x7fb3180b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.526+0000 7fb31dc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180b6200 con 0x7fb3180cf1f0 2024-04-02T09:09:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.526+0000 7fb31dc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180b64e0 con 0x7fb31815d760 2024-04-02T09:09:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.526+0000 7fb31dc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180c8fc0 con 0x7fb31815db40 2024-04-02T09:09:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.526+0000 7fb316a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3180cf1f0 0x7fb3181f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.526+0000 7fb316a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3180cf1f0 0x7fb3181f2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52104/0 (socket says 172.21.15.89:52104) 2024-04-02T09:09:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.526+0000 7fb316a00700 1 -- 172.21.15.89:0/1153730888 learned_addr learned my addr 172.21.15.89:0/1153730888 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.526+0000 7fb317400700 1 --2- 172.21.15.89:0/1153730888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb31815db40 0x7fb3180b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.526+0000 7fb316000700 1 --2- 172.21.15.89:0/1153730888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb31815d760 0x7fb3180c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.526+0000 7fb316a00700 1 -- 172.21.15.89:0/1153730888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb31815db40 msgr2=0x7fb3180b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.527+0000 7fb316a00700 1 --2- 172.21.15.89:0/1153730888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb31815db40 0x7fb3180b5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.527+0000 7fb316a00700 1 -- 172.21.15.89:0/1153730888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb31815d760 msgr2=0x7fb3180c8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.527+0000 7fb316a00700 1 --2- 172.21.15.89:0/1153730888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb31815d760 0x7fb3180c8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.527+0000 7fb316a00700 1 -- 172.21.15.89:0/1153730888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb308009550 con 0x7fb3180cf1f0 2024-04-02T09:09:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.527+0000 7fb316a00700 1 --2- 172.21.15.89:0/1153730888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3180cf1f0 0x7fb3181f2e40 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fb308000c00 tx=0x7fb308004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.527+0000 7fb30ea00700 1 -- 172.21.15.89:0/1153730888 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308004380 con 0x7fb3180cf1f0 2024-04-02T09:09:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.527+0000 7fb30ea00700 1 -- 172.21.15.89:0/1153730888 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30801f070 con 0x7fb3180cf1f0 2024-04-02T09:09:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.527+0000 7fb31dc00700 1 -- 172.21.15.89:0/1153730888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3180c92a0 con 0x7fb3180cf1f0 2024-04-02T09:09:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.527+0000 7fb30ea00700 1 -- 172.21.15.89:0/1153730888 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3080036a0 con 0x7fb3180cf1f0 2024-04-02T09:09:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.527+0000 7fb31dc00700 1 -- 172.21.15.89:0/1153730888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3181f3380 con 0x7fb3180cf1f0 2024-04-02T09:09:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.528+0000 7fb31dc00700 1 -- 172.21.15.89:0/1153730888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb318067720 con 0x7fb3180cf1f0 2024-04-02T09:09:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.529+0000 7fb30ea00700 1 -- 172.21.15.89:0/1153730888 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 128) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fb308021030 con 0x7fb3180cf1f0 2024-04-02T09:09:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.530+0000 7fb30ea00700 1 --2- 172.21.15.89:0/1153730888 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb2f806adb0 0x7fb2f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.531+0000 7fb30ea00700 1 -- 172.21.15.89:0/1153730888 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6861+0+0 (secure 0 0 0) 0x7fb30808ad70 con 0x7fb3180cf1f0 2024-04-02T09:09:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.531+0000 7fb316000700 1 -- 172.21.15.89:0/1153730888 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb2f806adb0 msgr2=0x7fb2f806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.531+0000 7fb316000700 1 --2- 172.21.15.89:0/1153730888 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb2f806adb0 0x7fb2f806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:26.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.535+0000 7fb30ea00700 1 -- 172.21.15.89:0/1153730888 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb308055b70 con 0x7fb3180cf1f0 2024-04-02T09:09:26.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.732+0000 7fb316000700 1 -- 172.21.15.89:0/1153730888 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb2f806adb0 msgr2=0x7fb2f806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:26.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.732+0000 7fb316000700 1 --2- 172.21.15.89:0/1153730888 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb2f806adb0 0x7fb2f806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:26.731 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:26.732+0000 7f83a0cb5240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:09:26.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.786+0000 7fb31dc00700 1 -- 172.21.15.89:0/1153730888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3180b2fb0 con 0x7fb3180cf1f0 2024-04-02T09:09:26.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.787+0000 7fb30ea00700 1 -- 172.21.15.89:0/1153730888 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4745 (secure 0 0 0) 0x7fb308015090 con 0x7fb3180cf1f0 2024-04-02T09:09:26.785 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:26.785 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1296569976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1296569976},{"type":"v1","addr":"172.21.15.113:6835","nonce":1296569976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9331,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1328950634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1328950634},{"type":"v1","addr":"172.21.15.89:6837","nonce":1328950634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":10032,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1167691520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1167691520},{"type":"v1","addr":"172.21.15.113:6834","nonce":1167691520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T08:53:22.881600+0000","modified":"2024-04-02T08:53:26.390144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10038":{"gid":10038,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/583021497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":583021497},{"type":"v1","addr":"172.21.15.89:6836","nonce":583021497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-04-02T09:09:26.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.789+0000 7fb31dc00700 1 -- 172.21.15.89:0/1153730888 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb2f806adb0 msgr2=0x7fb2f806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:26.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.789+0000 7fb31dc00700 1 --2- 172.21.15.89:0/1153730888 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb2f806adb0 0x7fb2f806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.789+0000 7fb31dc00700 1 -- 172.21.15.89:0/1153730888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3180cf1f0 msgr2=0x7fb3181f2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:26.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.789+0000 7fb31dc00700 1 --2- 172.21.15.89:0/1153730888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3180cf1f0 0x7fb3181f2e40 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fb308000c00 tx=0x7fb308004750 comp rx=0 tx=0).stop 2024-04-02T09:09:26.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.789+0000 7fb31dc00700 1 -- 172.21.15.89:0/1153730888 shutdown_connections 2024-04-02T09:09:26.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.790+0000 7fb31dc00700 1 --2- 172.21.15.89:0/1153730888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb31815db40 0x7fb3180b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.790+0000 7fb31dc00700 1 --2- 172.21.15.89:0/1153730888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3180cf1f0 0x7fb3181f2e40 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.790+0000 7fb31dc00700 1 --2- 172.21.15.89:0/1153730888 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb2f806adb0 0x7fb2f806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.790+0000 7fb31dc00700 1 --2- 172.21.15.89:0/1153730888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb31815d760 0x7fb3180c8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.790+0000 7fb31dc00700 1 -- 172.21.15.89:0/1153730888 >> 172.21.15.89:0/1153730888 conn(0x7fb31808aff0 msgr2=0x7fb3180b34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.790+0000 7fb31dc00700 1 -- 172.21.15.89:0/1153730888 shutdown_connections 2024-04-02T09:09:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:26.790+0000 7fb31dc00700 1 -- 172.21.15.89:0/1153730888 wait complete. 2024-04-02T09:09:26.789 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 89 2024-04-02T09:09:26.807 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:09:26.809 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:26.811+0000 7ff0714f6240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:09:27.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.003+0000 7f73b2c00700 1 Processor -- start 2024-04-02T09:09:27.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.003+0000 7f73b2c00700 1 -- start start 2024-04-02T09:09:27.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.004+0000 7f73b2c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f73ac0cb3f0 0x7f73ac0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.004+0000 7f73b2c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f73ac0ac600 0x7f73ac06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.004+0000 7f73b2c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac15fd50 con 0x7f73ac15b5d0 2024-04-02T09:09:27.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.004+0000 7f73b2c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ac15b9b0 con 0x7f73ac0cb3f0 2024-04-02T09:09:27.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.004+0000 7f73b2c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ac0aca10 con 0x7f73ac0ac600 2024-04-02T09:09:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.005+0000 7f73ab400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f73ac0cb3f0 0x7f73ac0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.005+0000 7f73abe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f73ac15b5d0 0x7f73ac0ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45440/0 (socket says 172.21.15.89:45440) 2024-04-02T09:09:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.005+0000 7f73abe00700 1 -- 172.21.15.89:0/52782 learned_addr learned my addr 172.21.15.89:0/52782 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.005+0000 7f73b0c00700 1 --2- 172.21.15.89:0/52782 >> v2:172.21.15.113:3300/0 conn(0x7f73ac0ac600 0x7f73ac06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.005+0000 7f73ab400700 1 -- 172.21.15.89:0/52782 >> v2:172.21.15.113:3300/0 conn(0x7f73ac0ac600 msgr2=0x7f73ac06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.005+0000 7f73ab400700 1 --2- 172.21.15.89:0/52782 >> v2:172.21.15.113:3300/0 conn(0x7f73ac0ac600 0x7f73ac06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.005+0000 7f73ab400700 1 -- 172.21.15.89:0/52782 >> v1:172.21.15.89:6789/0 conn(0x7f73ac15b5d0 legacy=0x7f73ac0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.005+0000 7f73ab400700 1 -- 172.21.15.89:0/52782 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac15f140 con 0x7f73ac0cb3f0 2024-04-02T09:09:27.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.005+0000 7f73aaa00700 1 -- 172.21.15.89:0/52782 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858340093 0 0) 0x7f73ac15fd50 con 0x7f73ac15b5d0 2024-04-02T09:09:27.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.005+0000 7f73ab400700 1 --2- 172.21.15.89:0/52782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73ac0cb3f0 0x7f73ac0ac0c0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0cfdc0 tx=0x7f7398009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac765b6deb0275e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:27.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.006+0000 7f73aaa00700 1 -- 172.21.15.89:0/52782 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7398011830 con 0x7f73ac0cb3f0 2024-04-02T09:09:27.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.006+0000 7f73aaa00700 1 -- 172.21.15.89:0/52782 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7398011990 con 0x7f73ac0cb3f0 2024-04-02T09:09:27.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.006+0000 7f73aaa00700 1 -- 172.21.15.89:0/52782 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7398011d20 con 0x7f73ac0cb3f0 2024-04-02T09:09:27.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.006+0000 7f73b2c00700 1 -- 172.21.15.89:0/52782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73ac0cb3f0 msgr2=0x7f73ac0ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.006+0000 7f73b2c00700 1 --2- 172.21.15.89:0/52782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73ac0cb3f0 0x7f73ac0ac0c0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0cfdc0 tx=0x7f7398009b20 comp rx=0 tx=0).stop 2024-04-02T09:09:27.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.006+0000 7f73b2c00700 1 -- 172.21.15.89:0/52782 shutdown_connections 2024-04-02T09:09:27.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.006+0000 7f73b2c00700 1 --2- 172.21.15.89:0/52782 >> v2:172.21.15.113:3300/0 conn(0x7f73ac0ac600 0x7f73ac06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.006+0000 7f73b2c00700 1 --2- 172.21.15.89:0/52782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73ac0cb3f0 0x7f73ac0ac0c0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.006+0000 7f73b2c00700 1 -- 172.21.15.89:0/52782 >> 172.21.15.89:0/52782 conn(0x7f73ac0cf190 msgr2=0x7f73ac0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:27.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.006+0000 7f73b2c00700 1 -- 172.21.15.89:0/52782 shutdown_connections 2024-04-02T09:09:27.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.006+0000 7f73b2c00700 1 -- 172.21.15.89:0/52782 wait complete. 2024-04-02T09:09:27.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.008+0000 7f73b2c00700 1 Processor -- start 2024-04-02T09:09:27.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.008+0000 7f73b2c00700 1 -- start start 2024-04-02T09:09:27.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.009+0000 7f73b2c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73ac0ac600 0x7f73ac09ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.009+0000 7f73b2c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73ac15b5d0 0x7f73ac157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.009+0000 7f73b2c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73ac0b3050 0x7f73ac0b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.009+0000 7f73b2c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac0b3a40 con 0x7f73ac0b3050 2024-04-02T09:09:27.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.009+0000 7f73b2c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac09cfc0 con 0x7f73ac15b5d0 2024-04-02T09:09:27.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.009+0000 7f73b2c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac15a2a0 con 0x7f73ac0ac600 2024-04-02T09:09:27.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73b0c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73ac0b3050 0x7f73ac0b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:27.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73b0c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73ac0b3050 0x7f73ac0b3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52122/0 (socket says 172.21.15.89:52122) 2024-04-02T09:09:27.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73b0c00700 1 -- 172.21.15.89:0/3826046181 learned_addr learned my addr 172.21.15.89:0/3826046181 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:27.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73ab400700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73ac15b5d0 0x7f73ac157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:27.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73abe00700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73ac0ac600 0x7f73ac09ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:27.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73b0c00700 1 -- 172.21.15.89:0/3826046181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73ac0ac600 msgr2=0x7f73ac09ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73b0c00700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73ac0ac600 0x7f73ac09ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73b0c00700 1 -- 172.21.15.89:0/3826046181 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73ac15b5d0 msgr2=0x7f73ac157470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73b0c00700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73ac15b5d0 0x7f73ac157470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73b0c00700 1 -- 172.21.15.89:0/3826046181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73980094a0 con 0x7f73ac0b3050 2024-04-02T09:09:27.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73abe00700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73ac0ac600 0x7f73ac09ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:27.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73b0c00700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73ac0b3050 0x7f73ac0b3500 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f73a400efc0 tx=0x7f73a400c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:27.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73a3e00700 1 -- 172.21.15.89:0/3826046181 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a400cb20 con 0x7f73ac0b3050 2024-04-02T09:09:27.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73a3e00700 1 -- 172.21.15.89:0/3826046181 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a400f040 con 0x7f73ac0b3050 2024-04-02T09:09:27.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.010+0000 7f73a3e00700 1 -- 172.21.15.89:0/3826046181 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a400cec0 con 0x7f73ac0b3050 2024-04-02T09:09:27.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.011+0000 7f73b2c00700 1 -- 172.21.15.89:0/3826046181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ac15a580 con 0x7f73ac0b3050 2024-04-02T09:09:27.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.011+0000 7f73b2c00700 1 -- 172.21.15.89:0/3826046181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ac15ab40 con 0x7f73ac0b3050 2024-04-02T09:09:27.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.012+0000 7f73b2c00700 1 -- 172.21.15.89:0/3826046181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73ac067720 con 0x7f73ac0b3050 2024-04-02T09:09:27.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.012+0000 7f73a3e00700 1 -- 172.21.15.89:0/3826046181 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 128) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f73a4014480 con 0x7f73ac0b3050 2024-04-02T09:09:27.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.013+0000 7f73a3e00700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f738c06adb0 0x7f738c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.013+0000 7f73a3e00700 1 -- 172.21.15.89:0/3826046181 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6861+0+0 (secure 0 0 0) 0x7f73a408a530 con 0x7f73ac0b3050 2024-04-02T09:09:27.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.013+0000 7f73abe00700 1 -- 172.21.15.89:0/3826046181 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f738c06adb0 msgr2=0x7f738c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:27.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.013+0000 7f73abe00700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f738c06adb0 0x7f738c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.018+0000 7f73a3e00700 1 -- 172.21.15.89:0/3826046181 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f73a4055060 con 0x7f73ac0b3050 2024-04-02T09:09:27.016 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:27.017+0000 7ff0714f6240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:09:27.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.214+0000 7f73abe00700 1 -- 172.21.15.89:0/3826046181 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f738c06adb0 msgr2=0x7f738c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:27.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.214+0000 7f73abe00700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f738c06adb0 0x7f738c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.225+0000 7f73b2c00700 1 -- 172.21.15.89:0/3826046181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73ac06ea60 con 0x7f73ac0b3050 2024-04-02T09:09:27.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.227+0000 7f73a3e00700 1 -- 172.21.15.89:0/3826046181 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+17851 (secure 0 0 0) 0x7f73a4058680 con 0x7f73ac0b3050 2024-04-02T09:09:27.226 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:27.226 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":192,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:09:23.795832+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-02T08:53:20.736248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-02T08:53:21.860376+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":192,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"23.b","mappings":[{"from":5,"to":3}]},{"pgid":"23.1d","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/4284766432":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/3938746027":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/3350462335":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/3070765417":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/4154199909":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/2003321918":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/2504651954":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/978973291":"2024-04-03T08:53:39.182240+0000","172.21.15.89:6833/43357":"2024-04-03T08:53:39.182240+0000","172.21.15.89:6832/43357":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/1859816274":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/477599566":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/3314137116":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/2444652307":"2024-04-03T08:53:39.182240+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:09:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.229+0000 7f73b2c00700 1 -- 172.21.15.89:0/3826046181 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f738c06adb0 msgr2=0x7f738c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.229+0000 7f73b2c00700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f738c06adb0 0x7f738c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.229+0000 7f73b2c00700 1 -- 172.21.15.89:0/3826046181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73ac0b3050 msgr2=0x7f73ac0b3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.229+0000 7f73b2c00700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73ac0b3050 0x7f73ac0b3500 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f73a400efc0 tx=0x7f73a400c0c0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.229+0000 7f73b2c00700 1 -- 172.21.15.89:0/3826046181 shutdown_connections 2024-04-02T09:09:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.229+0000 7f73b2c00700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73ac0b3050 0x7f73ac0b3500 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.229+0000 7f73b2c00700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73ac0ac600 0x7f73ac09ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.229+0000 7f73b2c00700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f738c06adb0 0x7f738c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.229+0000 7f73b2c00700 1 --2- 172.21.15.89:0/3826046181 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73ac15b5d0 0x7f73ac157470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.229+0000 7f73b2c00700 1 -- 172.21.15.89:0/3826046181 >> 172.21.15.89:0/3826046181 conn(0x7f73ac0cf190 msgr2=0x7f73ac0aed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:27.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.229+0000 7f73b2c00700 1 -- 172.21.15.89:0/3826046181 shutdown_connections 2024-04-02T09:09:27.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.230+0000 7f73b2c00700 1 -- 172.21.15.89:0/3826046181 wait complete. 2024-04-02T09:09:27.229 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:27.231+0000 7f83a0cb5240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:09:27.246 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T09:09:27.246 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:09:27.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.442+0000 7f1222400700 1 Processor -- start 2024-04-02T09:09:27.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.443+0000 7f1222400700 1 -- start start 2024-04-02T09:09:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.444+0000 7f1222400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f121c15b5d0 0x7f121c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.444+0000 7f1222400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121c15fd50 con 0x7f121c0ac7d0 2024-04-02T09:09:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.444+0000 7f1222400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121c15f140 con 0x7f121c0cb3f0 2024-04-02T09:09:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.444+0000 7f1222400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f121c15b9b0 con 0x7f121c15b5d0 2024-04-02T09:09:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.444+0000 7f121b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f121c15b5d0 0x7f121c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.444+0000 7f121b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f121c15b5d0 0x7f121c0ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52124/0 (socket says 172.21.15.89:52124) 2024-04-02T09:09:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.444+0000 7f121b400700 1 -- 172.21.15.89:0/3129031348 learned_addr learned my addr 172.21.15.89:0/3129031348 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.444+0000 7f121be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f121c0ac7d0 0x7f121c06c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45452/0 (socket says 172.21.15.89:45452) 2024-04-02T09:09:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.445+0000 7f121b400700 1 -- 172.21.15.89:0/3129031348 >> v1:172.21.15.113:6789/0 conn(0x7f121c0cb3f0 legacy=0x7f121c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.445+0000 7f121b400700 1 -- 172.21.15.89:0/3129031348 >> v1:172.21.15.89:6789/0 conn(0x7f121c0ac7d0 legacy=0x7f121c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.445+0000 7f121b400700 1 -- 172.21.15.89:0/3129031348 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c15f420 con 0x7f121c15b5d0 2024-04-02T09:09:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.445+0000 7f121a000700 1 -- 172.21.15.89:0/3129031348 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381598826 0 0) 0x7f121c15f140 con 0x7f121c0cb3f0 2024-04-02T09:09:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.445+0000 7f121b400700 1 --2- 172.21.15.89:0/3129031348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f121c15b5d0 0x7f121c0ae3f0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f120c0098f0 tx=0x7f120c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b48d503880fb155 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.445+0000 7f121a000700 1 -- 172.21.15.89:0/3129031348 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c004230 con 0x7f121c15b5d0 2024-04-02T09:09:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.445+0000 7f121a000700 1 -- 172.21.15.89:0/3129031348 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120c004390 con 0x7f121c15b5d0 2024-04-02T09:09:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.445+0000 7f121a000700 1 -- 172.21.15.89:0/3129031348 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c0036a0 con 0x7f121c15b5d0 2024-04-02T09:09:27.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.446+0000 7f1222400700 1 -- 172.21.15.89:0/3129031348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f121c15b5d0 msgr2=0x7f121c0ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.446+0000 7f1222400700 1 --2- 172.21.15.89:0/3129031348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f121c15b5d0 0x7f121c0ae3f0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f120c0098f0 tx=0x7f120c009c00 comp rx=0 tx=0).stop 2024-04-02T09:09:27.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.446+0000 7f1222400700 1 -- 172.21.15.89:0/3129031348 shutdown_connections 2024-04-02T09:09:27.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.446+0000 7f1222400700 1 --2- 172.21.15.89:0/3129031348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f121c15b5d0 0x7f121c0ae3f0 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.446+0000 7f1222400700 1 -- 172.21.15.89:0/3129031348 >> 172.21.15.89:0/3129031348 conn(0x7f121c0cf190 msgr2=0x7f121c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:27.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.446+0000 7f1222400700 1 -- 172.21.15.89:0/3129031348 shutdown_connections 2024-04-02T09:09:27.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.447+0000 7f1222400700 1 -- 172.21.15.89:0/3129031348 wait complete. 2024-04-02T09:09:27.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.449+0000 7f1222400700 1 Processor -- start 2024-04-02T09:09:27.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.450+0000 7f1222400700 1 -- start start 2024-04-02T09:09:27.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.450+0000 7f1222400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f121c0ac7d0 0x7f121c072630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.450+0000 7f1222400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f121c15b5d0 0x7f121c0c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.450+0000 7f1222400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f121c15e3d0 0x7f121c144b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.450+0000 7f1222400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c0c8e20 con 0x7f121c15e3d0 2024-04-02T09:09:27.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.450+0000 7f1222400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c15db00 con 0x7f121c15b5d0 2024-04-02T09:09:27.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.450+0000 7f1222400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f121c072100 con 0x7f121c0ac7d0 2024-04-02T09:09:27.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.451+0000 7f121be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f121c15e3d0 0x7f121c144b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:27.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.451+0000 7f121be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f121c15e3d0 0x7f121c144b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52136/0 (socket says 172.21.15.89:52136) 2024-04-02T09:09:27.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.451+0000 7f121be00700 1 -- 172.21.15.89:0/1282226360 learned_addr learned my addr 172.21.15.89:0/1282226360 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:27.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.451+0000 7f121b400700 1 --2- 172.21.15.89:0/1282226360 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f121c0ac7d0 0x7f121c072630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:27.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.451+0000 7f121aa00700 1 --2- 172.21.15.89:0/1282226360 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f121c15b5d0 0x7f121c0c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.451+0000 7f121be00700 1 -- 172.21.15.89:0/1282226360 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f121c0ac7d0 msgr2=0x7f121c072630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.451+0000 7f121be00700 1 --2- 172.21.15.89:0/1282226360 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f121c0ac7d0 0x7f121c072630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.451+0000 7f121be00700 1 -- 172.21.15.89:0/1282226360 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f121c15b5d0 msgr2=0x7f121c0c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.451+0000 7f121be00700 1 --2- 172.21.15.89:0/1282226360 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f121c15b5d0 0x7f121c0c9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.451+0000 7f121be00700 1 -- 172.21.15.89:0/1282226360 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c009550 con 0x7f121c15e3d0 2024-04-02T09:09:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.452+0000 7f121be00700 1 --2- 172.21.15.89:0/1282226360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f121c15e3d0 0x7f121c144b40 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f1214002000 tx=0x7f121400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.452+0000 7f1213400700 1 -- 172.21.15.89:0/1282226360 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121400ac90 con 0x7f121c15e3d0 2024-04-02T09:09:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.452+0000 7f1213400700 1 -- 172.21.15.89:0/1282226360 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121400adf0 con 0x7f121c15e3d0 2024-04-02T09:09:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.452+0000 7f1222400700 1 -- 172.21.15.89:0/1282226360 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f121c0c9a70 con 0x7f121c15e3d0 2024-04-02T09:09:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.452+0000 7f1213400700 1 -- 172.21.15.89:0/1282226360 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214007310 con 0x7f121c15e3d0 2024-04-02T09:09:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.452+0000 7f1222400700 1 -- 172.21.15.89:0/1282226360 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f121c071cb0 con 0x7f121c15e3d0 2024-04-02T09:09:27.452 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:27.452+0000 7ff0714f6240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:09:27.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.453+0000 7f1222400700 1 -- 172.21.15.89:0/1282226360 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f121c071ed0 con 0x7f121c15e3d0 2024-04-02T09:09:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.454+0000 7f1213400700 1 -- 172.21.15.89:0/1282226360 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 128) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f1214007470 con 0x7f121c15e3d0 2024-04-02T09:09:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.455+0000 7f1213400700 1 --2- 172.21.15.89:0/1282226360 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f11fc06adb0 0x7f11fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.455+0000 7f1213400700 1 -- 172.21.15.89:0/1282226360 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6861+0+0 (secure 0 0 0) 0x7f1214089710 con 0x7f121c15e3d0 2024-04-02T09:09:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.455+0000 7f121b400700 1 -- 172.21.15.89:0/1282226360 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f11fc06adb0 msgr2=0x7f11fc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.455+0000 7f121b400700 1 --2- 172.21.15.89:0/1282226360 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f11fc06adb0 0x7f11fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:27.455 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:27.457+0000 7f83a0cb5240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:09:27.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.459+0000 7f1213400700 1 -- 172.21.15.89:0/1282226360 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1214054360 con 0x7f121c15e3d0 2024-04-02T09:09:27.562 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:27.563+0000 7ff0714f6240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:09:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.656+0000 7f121b400700 1 -- 172.21.15.89:0/1282226360 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f11fc06adb0 msgr2=0x7f11fc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:27.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.656+0000 7f121b400700 1 --2- 172.21.15.89:0/1282226360 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f11fc06adb0 0x7f11fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:27.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.737+0000 7f1222400700 1 -- 172.21.15.89:0/1282226360 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f121c15f9a0 con 0x7f121c15e3d0 2024-04-02T09:09:27.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.738+0000 7f1213400700 1 -- 172.21.15.89:0/1282226360 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f1214057980 con 0x7f121c15e3d0 2024-04-02T09:09:27.736 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:27.736 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1296569976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1296569976},{"type":"v1","addr":"172.21.15.113:6835","nonce":1296569976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9331,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1328950634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1328950634},{"type":"v1","addr":"172.21.15.89:6837","nonce":1328950634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":10032,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1167691520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1167691520},{"type":"v1","addr":"172.21.15.113:6834","nonce":1167691520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T08:53:22.881600+0000","modified":"2024-04-02T08:53:26.390144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10038":{"gid":10038,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/583021497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":583021497},{"type":"v1","addr":"172.21.15.89:6836","nonce":583021497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-04-02T09:09:27.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.740+0000 7f1222400700 1 -- 172.21.15.89:0/1282226360 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f11fc06adb0 msgr2=0x7f11fc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:27.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.740+0000 7f1222400700 1 --2- 172.21.15.89:0/1282226360 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f11fc06adb0 0x7f11fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.740+0000 7f1222400700 1 -- 172.21.15.89:0/1282226360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f121c15e3d0 msgr2=0x7f121c144b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.740+0000 7f1222400700 1 --2- 172.21.15.89:0/1282226360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f121c15e3d0 0x7f121c144b40 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f1214002000 tx=0x7f121400a340 comp rx=0 tx=0).stop 2024-04-02T09:09:27.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.740+0000 7f1222400700 1 -- 172.21.15.89:0/1282226360 shutdown_connections 2024-04-02T09:09:27.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.740+0000 7f1222400700 1 --2- 172.21.15.89:0/1282226360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f121c15e3d0 0x7f121c144b40 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.740+0000 7f1222400700 1 --2- 172.21.15.89:0/1282226360 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f121c0ac7d0 0x7f121c072630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.741+0000 7f1222400700 1 --2- 172.21.15.89:0/1282226360 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f11fc06adb0 0x7f11fc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.741+0000 7f1222400700 1 --2- 172.21.15.89:0/1282226360 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f121c15b5d0 0x7f121c0c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.741+0000 7f1222400700 1 -- 172.21.15.89:0/1282226360 >> 172.21.15.89:0/1282226360 conn(0x7f121c0cf190 msgr2=0x7f121c15cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:27.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.741+0000 7f1222400700 1 -- 172.21.15.89:0/1282226360 shutdown_connections 2024-04-02T09:09:27.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.741+0000 7f1222400700 1 -- 172.21.15.89:0/1282226360 wait complete. 2024-04-02T09:09:27.740 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 89 2024-04-02T09:09:27.759 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:09:27.939 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:27.941+0000 7f83a0cb5240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:09:27.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.959+0000 7f927e000700 1 Processor -- start 2024-04-02T09:09:27.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.959+0000 7f927e000700 1 -- start start 2024-04-02T09:09:27.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.960+0000 7f927e000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9278160c90 0x7f92780b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.961+0000 7f927e000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92780ac4d0 con 0x7f92780718e0 2024-04-02T09:09:27.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.961+0000 7f927e000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927806ddd0 con 0x7f92780ad970 2024-04-02T09:09:27.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.961+0000 7f927e000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9278071e50 con 0x7f9278160c90 2024-04-02T09:09:27.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.961+0000 7f9277400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9278160c90 0x7f92780b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:27.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.961+0000 7f9277400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9278160c90 0x7f92780b20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52152/0 (socket says 172.21.15.89:52152) 2024-04-02T09:09:27.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.961+0000 7f9276a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f92780718e0 0x7f927806cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45462/0 (socket says 172.21.15.89:45462) 2024-04-02T09:09:27.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.961+0000 7f9277400700 1 -- 172.21.15.89:0/1284968257 learned_addr learned my addr 172.21.15.89:0/1284968257 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:27.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.961+0000 7f9277400700 1 -- 172.21.15.89:0/1284968257 >> v1:172.21.15.113:6789/0 conn(0x7f92780ad970 legacy=0x7f927806d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.961+0000 7f9277400700 1 -- 172.21.15.89:0/1284968257 >> v1:172.21.15.89:6789/0 conn(0x7f92780718e0 legacy=0x7f927806cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.961+0000 7f9277400700 1 -- 172.21.15.89:0/1284968257 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9278072130 con 0x7f9278160c90 2024-04-02T09:09:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.961+0000 7f9276000700 1 -- 172.21.15.89:0/1284968257 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318263810 0 0) 0x7f92780ac4d0 con 0x7f92780718e0 2024-04-02T09:09:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.962+0000 7f9277400700 1 --2- 172.21.15.89:0/1284968257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9278160c90 0x7f92780b20e0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f92680098f0 tx=0x7f9268009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c56d1602e3ceff38 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.962+0000 7f9276000700 1 -- 172.21.15.89:0/1284968257 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926800b8a0 con 0x7f9278160c90 2024-04-02T09:09:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.962+0000 7f9276000700 1 -- 172.21.15.89:0/1284968257 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926800ba00 con 0x7f9278160c90 2024-04-02T09:09:27.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.962+0000 7f9276000700 1 -- 172.21.15.89:0/1284968257 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926800bd90 con 0x7f9278160c90 2024-04-02T09:09:27.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.963+0000 7f927e000700 1 -- 172.21.15.89:0/1284968257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9278160c90 msgr2=0x7f92780b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.963+0000 7f927e000700 1 --2- 172.21.15.89:0/1284968257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9278160c90 0x7f92780b20e0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f92680098f0 tx=0x7f9268009c00 comp rx=0 tx=0).stop 2024-04-02T09:09:27.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.963+0000 7f927e000700 1 -- 172.21.15.89:0/1284968257 shutdown_connections 2024-04-02T09:09:27.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.963+0000 7f927e000700 1 --2- 172.21.15.89:0/1284968257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9278160c90 0x7f92780b20e0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.963+0000 7f927e000700 1 -- 172.21.15.89:0/1284968257 >> 172.21.15.89:0/1284968257 conn(0x7f92780cf190 msgr2=0x7f92780cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:27.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.963+0000 7f927e000700 1 -- 172.21.15.89:0/1284968257 shutdown_connections 2024-04-02T09:09:27.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.963+0000 7f927e000700 1 -- 172.21.15.89:0/1284968257 wait complete. 2024-04-02T09:09:27.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.966+0000 7f927e000700 1 Processor -- start 2024-04-02T09:09:27.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.966+0000 7f927e000700 1 -- start start 2024-04-02T09:09:27.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.967+0000 7f927e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92780718e0 0x7f927814cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.967+0000 7f927e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92780ad970 0x7f927814c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.968+0000 7f927e000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9278160c90 0x7f92780cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.968+0000 7f927e000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92780ce3c0 con 0x7f92780718e0 2024-04-02T09:09:27.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.968+0000 7f927e000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927809aeb0 con 0x7f9278160c90 2024-04-02T09:09:27.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.968+0000 7f927e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92780ce0e0 con 0x7f92780ad970 2024-04-02T09:09:27.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.968+0000 7f9277400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92780718e0 0x7f927814cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:27.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.968+0000 7f9277400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92780718e0 0x7f927814cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52158/0 (socket says 172.21.15.89:52158) 2024-04-02T09:09:27.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.968+0000 7f9277400700 1 -- 172.21.15.89:0/3332117247 learned_addr learned my addr 172.21.15.89:0/3332117247 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:27.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.968+0000 7f9276a00700 1 --2- 172.21.15.89:0/3332117247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92780ad970 0x7f927814c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:27.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.968+0000 7f9277e00700 1 --2- 172.21.15.89:0/3332117247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9278160c90 0x7f92780cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:27.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.968+0000 7f9277400700 1 -- 172.21.15.89:0/3332117247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92780ad970 msgr2=0x7f927814c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.968+0000 7f9277400700 1 --2- 172.21.15.89:0/3332117247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92780ad970 0x7f927814c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.968+0000 7f9277400700 1 -- 172.21.15.89:0/3332117247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9278160c90 msgr2=0x7f92780cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:27.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.968+0000 7f9277400700 1 --2- 172.21.15.89:0/3332117247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9278160c90 0x7f92780cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:27.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.968+0000 7f9277400700 1 -- 172.21.15.89:0/3332117247 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9268009550 con 0x7f92780718e0 2024-04-02T09:09:27.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.969+0000 7f9277400700 1 --2- 172.21.15.89:0/3332117247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92780718e0 0x7f927814cab0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f9268014040 tx=0x7f9268004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:27.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.969+0000 7f926f400700 1 -- 172.21.15.89:0/3332117247 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926801a8f0 con 0x7f92780718e0 2024-04-02T09:09:27.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.969+0000 7f926f400700 1 -- 172.21.15.89:0/3332117247 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926801e070 con 0x7f92780718e0 2024-04-02T09:09:27.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.969+0000 7f927e000700 1 -- 172.21.15.89:0/3332117247 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927809a860 con 0x7f92780718e0 2024-04-02T09:09:27.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.969+0000 7f926f400700 1 -- 172.21.15.89:0/3332117247 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926801ac50 con 0x7f92780718e0 2024-04-02T09:09:27.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.969+0000 7f927e000700 1 -- 172.21.15.89:0/3332117247 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927814c6f0 con 0x7f92780718e0 2024-04-02T09:09:27.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.970+0000 7f927e000700 1 -- 172.21.15.89:0/3332117247 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9278067720 con 0x7f92780718e0 2024-04-02T09:09:27.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.971+0000 7f926f400700 1 -- 172.21.15.89:0/3332117247 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 129) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f926802d450 con 0x7f92780718e0 2024-04-02T09:09:27.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.972+0000 7f926f400700 1 --2- 172.21.15.89:0/3332117247 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f925806adb0 0x7f925806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:27.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.973+0000 7f9276a00700 1 -- 172.21.15.89:0/3332117247 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f925806adb0 msgr2=0x7f925806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:27.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.973+0000 7f926f400700 1 -- 172.21.15.89:0/3332117247 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6861+0+0 (secure 0 0 0) 0x7f926808a8f0 con 0x7f92780718e0 2024-04-02T09:09:27.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.973+0000 7f9276a00700 1 --2- 172.21.15.89:0/3332117247 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f925806adb0 0x7f925806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:27.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:27.976+0000 7f926f400700 1 -- 172.21.15.89:0/3332117247 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9268055550 con 0x7f92780718e0 2024-04-02T09:09:28.076 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:28.078+0000 7f83a0cb5240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:09:28.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.088+0000 7f926f400700 1 -- 172.21.15.89:0/3332117247 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 130) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f9268055920 con 0x7f92780718e0 2024-04-02T09:09:28.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.089+0000 7f926f400700 1 -- 172.21.15.89:0/3332117247 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f925806adb0 msgr2=0x7f925806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:28.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.089+0000 7f926f400700 1 --2- 172.21.15.89:0/3332117247 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f925806adb0 0x7f925806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.151 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:28.153+0000 7f8310200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:28.151 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:28.153+0000 7f8310200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:28.151 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:28.153+0000 7f8310200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:28.152 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:28.153+0000 7f8310200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:28.152 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:28.153+0000 7f8310200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:28.152 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:28.153+0000 7f8313e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:28.152 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:28.153+0000 7f8313e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:28.152 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:28.153+0000 7f8313e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:28.152 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:28.153+0000 7f8313e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:28.152 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:28.153+0000 7f8313e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:28.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.267+0000 7f927e000700 1 -- 172.21.15.89:0/3332117247 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92780b0590 con 0x7f92780718e0 2024-04-02T09:09:28.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.268+0000 7f926f400700 1 -- 172.21.15.89:0/3332117247 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f926805caf0 con 0x7f92780718e0 2024-04-02T09:09:28.266 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:28.266 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1296569976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1296569976},{"type":"v1","addr":"172.21.15.113:6835","nonce":1296569976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9331,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1328950634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1328950634},{"type":"v1","addr":"172.21.15.89:6837","nonce":1328950634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":10032,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1167691520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1167691520},{"type":"v1","addr":"172.21.15.113:6834","nonce":1167691520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T08:53:22.881600+0000","modified":"2024-04-02T08:53:26.390144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10038":{"gid":10038,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/583021497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":583021497},{"type":"v1","addr":"172.21.15.89:6836","nonce":583021497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-04-02T09:09:28.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.270+0000 7f927e000700 1 -- 172.21.15.89:0/3332117247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92780718e0 msgr2=0x7f927814cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:28.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.270+0000 7f927e000700 1 --2- 172.21.15.89:0/3332117247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92780718e0 0x7f927814cab0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f9268014040 tx=0x7f9268004750 comp rx=0 tx=0).stop 2024-04-02T09:09:28.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.270+0000 7f927e000700 1 -- 172.21.15.89:0/3332117247 shutdown_connections 2024-04-02T09:09:28.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.270+0000 7f927e000700 1 --2- 172.21.15.89:0/3332117247 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f925806adb0 0x7f925806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.270+0000 7f927e000700 1 --2- 172.21.15.89:0/3332117247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9278160c90 0x7f92780cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.270+0000 7f927e000700 1 --2- 172.21.15.89:0/3332117247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92780718e0 0x7f927814cab0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.270+0000 7f927e000700 1 --2- 172.21.15.89:0/3332117247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92780ad970 0x7f927814c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.270+0000 7f927e000700 1 -- 172.21.15.89:0/3332117247 >> 172.21.15.89:0/3332117247 conn(0x7f92780cf190 msgr2=0x7f92780afe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:28.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.270+0000 7f927e000700 1 -- 172.21.15.89:0/3332117247 shutdown_connections 2024-04-02T09:09:28.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.270+0000 7f927e000700 1 -- 172.21.15.89:0/3332117247 wait complete. 2024-04-02T09:09:28.270 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 89 2024-04-02T09:09:28.287 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:09:28.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.480+0000 7f6f96e00700 1 Processor -- start 2024-04-02T09:09:28.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.481+0000 7f6f96e00700 1 -- start start 2024-04-02T09:09:28.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.482+0000 7f6f96e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f90160be0 0x7f6f900cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:28.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.482+0000 7f6f96e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6f900ceca0 0x7f6f900b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:28.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.482+0000 7f6f96e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f90160fc0 con 0x7f6f900ce0e0 2024-04-02T09:09:28.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.482+0000 7f6f96e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f900b65d0 con 0x7f6f900ceca0 2024-04-02T09:09:28.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.482+0000 7f6f96e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f900d2a80 con 0x7f6f90160be0 2024-04-02T09:09:28.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.482+0000 7f6f8fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f90160be0 0x7f6f900cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:28.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.482+0000 7f6f8f400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f6f900ce0e0 0x7f6f900ce590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:47750/0 (socket says 172.21.15.89:47750) 2024-04-02T09:09:28.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.482+0000 7f6f94e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6f900ceca0 0x7f6f900b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.482+0000 7f6f8fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f90160be0 0x7f6f900cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:33274/0 (socket says 172.21.15.89:33274) 2024-04-02T09:09:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.482+0000 7f6f8f400700 1 -- 172.21.15.89:0/38466166 learned_addr learned my addr 172.21.15.89:0/38466166 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.483+0000 7f6f8fe00700 1 -- 172.21.15.89:0/38466166 >> v2:172.21.15.113:3300/0 conn(0x7f6f900ceca0 msgr2=0x7f6f900b6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.483+0000 7f6f8fe00700 1 --2- 172.21.15.89:0/38466166 >> v2:172.21.15.113:3300/0 conn(0x7f6f900ceca0 0x7f6f900b6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.483+0000 7f6f8fe00700 1 -- 172.21.15.89:0/38466166 >> v1:172.21.15.113:6789/0 conn(0x7f6f900ce0e0 legacy=0x7f6f900ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.483+0000 7f6f8fe00700 1 -- 172.21.15.89:0/38466166 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f900d21e0 con 0x7f6f90160be0 2024-04-02T09:09:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.483+0000 7f6f94e00700 1 --2- 172.21.15.89:0/38466166 >> v2:172.21.15.113:3300/0 conn(0x7f6f900ceca0 0x7f6f900b6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:09:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.483+0000 7f6f8ea00700 1 -- 172.21.15.89:0/38466166 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121152779 0 0) 0x7f6f90160fc0 con 0x7f6f900ce0e0 2024-04-02T09:09:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.483+0000 7f6f8fe00700 1 --2- 172.21.15.89:0/38466166 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f90160be0 0x7f6f900cdba0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6f80009870 tx=0x7f6f80009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7e4883139c3ceb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.483+0000 7f6f8ea00700 1 -- 172.21.15.89:0/38466166 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80011830 con 0x7f6f90160be0 2024-04-02T09:09:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.483+0000 7f6f8ea00700 1 -- 172.21.15.89:0/38466166 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f80011990 con 0x7f6f90160be0 2024-04-02T09:09:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.483+0000 7f6f8ea00700 1 -- 172.21.15.89:0/38466166 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80011d20 con 0x7f6f90160be0 2024-04-02T09:09:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.484+0000 7f6f96e00700 1 -- 172.21.15.89:0/38466166 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f90160be0 msgr2=0x7f6f900cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.484+0000 7f6f96e00700 1 --2- 172.21.15.89:0/38466166 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f90160be0 0x7f6f900cdba0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6f80009870 tx=0x7f6f80009b80 comp rx=0 tx=0).stop 2024-04-02T09:09:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.484+0000 7f6f96e00700 1 -- 172.21.15.89:0/38466166 shutdown_connections 2024-04-02T09:09:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.484+0000 7f6f96e00700 1 --2- 172.21.15.89:0/38466166 >> v2:172.21.15.113:3300/0 conn(0x7f6f900ceca0 0x7f6f900b6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.484+0000 7f6f96e00700 1 --2- 172.21.15.89:0/38466166 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f90160be0 0x7f6f900cdba0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.484+0000 7f6f96e00700 1 -- 172.21.15.89:0/38466166 >> 172.21.15.89:0/38466166 conn(0x7f6f9008aff0 msgr2=0x7f6f900d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.484+0000 7f6f96e00700 1 -- 172.21.15.89:0/38466166 shutdown_connections 2024-04-02T09:09:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.484+0000 7f6f96e00700 1 -- 172.21.15.89:0/38466166 wait complete. 2024-04-02T09:09:28.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.486+0000 7f6f96e00700 1 Processor -- start 2024-04-02T09:09:28.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.487+0000 7f6f96e00700 1 -- start start 2024-04-02T09:09:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.488+0000 7f6f96e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f900ce0e0 0x7f6f901f3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.488+0000 7f6f96e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f900ceca0 0x7f6f9014db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.488+0000 7f6f96e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f90160be0 0x7f6f9006caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.488+0000 7f6f96e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f901f4e00 con 0x7f6f900ceca0 2024-04-02T09:09:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.488+0000 7f6f96e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f901f4180 con 0x7f6f90160be0 2024-04-02T09:09:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.488+0000 7f6f96e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f901f4460 con 0x7f6f900ce0e0 2024-04-02T09:09:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.488+0000 7f6f8f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f900ceca0 0x7f6f9014db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.488+0000 7f6f8f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f900ceca0 0x7f6f9014db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52254/0 (socket says 172.21.15.89:52254) 2024-04-02T09:09:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.488+0000 7f6f8fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f900ce0e0 0x7f6f901f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.488+0000 7f6f8f400700 1 -- 172.21.15.89:0/3577676714 learned_addr learned my addr 172.21.15.89:0/3577676714 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.488+0000 7f6f94e00700 1 --2- 172.21.15.89:0/3577676714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f90160be0 0x7f6f9006caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.489+0000 7f6f8f400700 1 -- 172.21.15.89:0/3577676714 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f900ce0e0 msgr2=0x7f6f901f3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.489+0000 7f6f8f400700 1 --2- 172.21.15.89:0/3577676714 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f900ce0e0 0x7f6f901f3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.489+0000 7f6f8f400700 1 -- 172.21.15.89:0/3577676714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f90160be0 msgr2=0x7f6f9006caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.489+0000 7f6f8f400700 1 --2- 172.21.15.89:0/3577676714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f90160be0 0x7f6f9006caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.489+0000 7f6f8f400700 1 -- 172.21.15.89:0/3577676714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f88009a70 con 0x7f6f900ceca0 2024-04-02T09:09:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.489+0000 7f6f94e00700 1 --2- 172.21.15.89:0/3577676714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f90160be0 0x7f6f9006caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:09:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.489+0000 7f6f8f400700 1 --2- 172.21.15.89:0/3577676714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f900ceca0 0x7f6f9014db20 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0011e0 tx=0x7f6f7c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.489+0000 7f6f87e00700 1 -- 172.21.15.89:0/3577676714 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c004c80 con 0x7f6f900ceca0 2024-04-02T09:09:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.489+0000 7f6f87e00700 1 -- 172.21.15.89:0/3577676714 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7c004de0 con 0x7f6f900ceca0 2024-04-02T09:09:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.489+0000 7f6f96e00700 1 -- 172.21.15.89:0/3577676714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f800094a0 con 0x7f6f900ceca0 2024-04-02T09:09:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.489+0000 7f6f87e00700 1 -- 172.21.15.89:0/3577676714 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c007310 con 0x7f6f900ceca0 2024-04-02T09:09:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.489+0000 7f6f96e00700 1 -- 172.21.15.89:0/3577676714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f901f4a20 con 0x7f6f900ceca0 2024-04-02T09:09:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.490+0000 7f6f96e00700 1 -- 172.21.15.89:0/3577676714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f901f4f40 con 0x7f6f900ceca0 2024-04-02T09:09:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.491+0000 7f6f87e00700 1 -- 172.21.15.89:0/3577676714 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 130) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f6f7c007470 con 0x7f6f900ceca0 2024-04-02T09:09:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.492+0000 7f6f87e00700 1 -- 172.21.15.89:0/3577676714 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 7161+0+0 (secure 0 0 0) 0x7f6f7c088ff0 con 0x7f6f900ceca0 2024-04-02T09:09:28.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.497+0000 7f6f87e00700 1 -- 172.21.15.89:0/3577676714 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6f7c0bc580 con 0x7f6f900ceca0 2024-04-02T09:09:28.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.698+0000 7f6f96e00700 1 -- 172.21.15.89:0/3577676714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f90067720 con 0x7f6f900ceca0 2024-04-02T09:09:28.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.700+0000 7f6f87e00700 1 -- 172.21.15.89:0/3577676714 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+18276 (secure 0 0 0) 0x7f6f901f4f40 con 0x7f6f900ceca0 2024-04-02T09:09:28.698 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:28.698 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":193,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:09:28.081011+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-02T08:53:20.736248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-02T08:53:21.860376+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":192,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"23.b","mappings":[{"from":5,"to":3}]},{"pgid":"23.1d","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/4040783770":"2024-04-03T09:09:28.080976+0000","172.21.15.89:0/3739614780":"2024-04-03T09:09:28.080976+0000","172.21.15.89:6833/108855":"2024-04-03T09:09:28.080976+0000","172.21.15.89:0/3293088546":"2024-04-03T09:09:28.080976+0000","172.21.15.89:0/3817839629":"2024-04-03T09:09:28.080976+0000","172.21.15.89:6832/108855":"2024-04-03T09:09:28.080976+0000","172.21.15.89:0/4284766432":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/3938746027":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/3350462335":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/3070765417":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/4154199909":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/2003321918":"2024-04-03T09:04:23.614732+0000","172.21.15.89:0/2504651954":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/978973291":"2024-04-03T08:53:39.182240+0000","172.21.15.89:6833/43357":"2024-04-03T08:53:39.182240+0000","172.21.15.89:6832/43357":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/1859816274":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/477599566":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/2861708222":"2024-04-03T09:09:28.080976+0000","172.21.15.89:0/2569329513":"2024-04-03T09:09:28.080976+0000","172.21.15.89:0/3314137116":"2024-04-03T08:53:39.182240+0000","172.21.15.89:0/2444652307":"2024-04-03T08:53:39.182240+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:09:28.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.702+0000 7f6f96e00700 1 -- 172.21.15.89:0/3577676714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f900ceca0 msgr2=0x7f6f9014db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:28.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.702+0000 7f6f96e00700 1 --2- 172.21.15.89:0/3577676714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f900ceca0 0x7f6f9014db20 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0011e0 tx=0x7f6f7c00ca30 comp rx=0 tx=0).stop 2024-04-02T09:09:28.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.702+0000 7f6f96e00700 1 -- 172.21.15.89:0/3577676714 shutdown_connections 2024-04-02T09:09:28.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.702+0000 7f6f96e00700 1 --2- 172.21.15.89:0/3577676714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f90160be0 0x7f6f9006caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.702+0000 7f6f96e00700 1 --2- 172.21.15.89:0/3577676714 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6f900ce0e0 0x7f6f901f3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.702+0000 7f6f96e00700 1 --2- 172.21.15.89:0/3577676714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f900ceca0 0x7f6f9014db20 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.702+0000 7f6f96e00700 1 -- 172.21.15.89:0/3577676714 >> 172.21.15.89:0/3577676714 conn(0x7f6f9008aff0 msgr2=0x7f6f900b8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:28.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.702+0000 7f6f96e00700 1 -- 172.21.15.89:0/3577676714 shutdown_connections 2024-04-02T09:09:28.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.702+0000 7f6f96e00700 1 -- 172.21.15.89:0/3577676714 wait complete. 2024-04-02T09:09:28.719 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T09:09:28.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.916+0000 7fa5a5000700 1 Processor -- start 2024-04-02T09:09:28.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.916+0000 7fa5a5000700 1 -- start start 2024-04-02T09:09:28.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.917+0000 7fa5a5000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa5a01611b0 0x7fa5a015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:28.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.917+0000 7fa5a5000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa5a00cee10 0x7fa5a0161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:28.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.917+0000 7fa5a5000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5a00ce270 con 0x7fa5a015f4d0 2024-04-02T09:09:28.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.917+0000 7fa5a5000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5a015b600 con 0x7fa5a01611b0 2024-04-02T09:09:28.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.917+0000 7fa5a5000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5a00d2760 con 0x7fa5a00cee10 2024-04-02T09:09:28.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.917+0000 7fa59de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa5a01611b0 0x7fa5a015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:28.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.917+0000 7fa59d400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa5a015f4d0 0x7fa5a00ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36308/0 (socket says 172.21.15.89:36308) 2024-04-02T09:09:28.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.917+0000 7fa59d400700 1 -- 172.21.15.89:0/3088560854 learned_addr learned my addr 172.21.15.89:0/3088560854 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:28.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.918+0000 7fa59de00700 1 -- 172.21.15.89:0/3088560854 >> v2:172.21.15.113:3300/0 conn(0x7fa5a00cee10 msgr2=0x7fa5a0161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:28.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.918+0000 7fa59e800700 1 --2- 172.21.15.89:0/3088560854 >> v2:172.21.15.113:3300/0 conn(0x7fa5a00cee10 0x7fa5a0161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:28.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.918+0000 7fa59de00700 1 --2- 172.21.15.89:0/3088560854 >> v2:172.21.15.113:3300/0 conn(0x7fa5a00cee10 0x7fa5a0161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.918+0000 7fa59de00700 1 -- 172.21.15.89:0/3088560854 >> v1:172.21.15.89:6789/0 conn(0x7fa5a015f4d0 legacy=0x7fa5a00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:28.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.918+0000 7fa59de00700 1 -- 172.21.15.89:0/3088560854 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a00d2a40 con 0x7fa5a01611b0 2024-04-02T09:09:28.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.918+0000 7fa59e800700 1 --2- 172.21.15.89:0/3088560854 >> v2:172.21.15.113:3300/0 conn(0x7fa5a00cee10 0x7fa5a0161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:28.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.918+0000 7fa59ca00700 1 -- 172.21.15.89:0/3088560854 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025132400 0 0) 0x7fa5a00ce270 con 0x7fa5a015f4d0 2024-04-02T09:09:28.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.918+0000 7fa59de00700 1 --2- 172.21.15.89:0/3088560854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5a01611b0 0x7fa5a015ef90 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fa5a00d2f60 tx=0x7fa590009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b4d0a03709ed08 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:28.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.918+0000 7fa59ca00700 1 -- 172.21.15.89:0/3088560854 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa590011830 con 0x7fa5a01611b0 2024-04-02T09:09:28.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.918+0000 7fa59ca00700 1 -- 172.21.15.89:0/3088560854 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa590011990 con 0x7fa5a01611b0 2024-04-02T09:09:28.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.918+0000 7fa59ca00700 1 -- 172.21.15.89:0/3088560854 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa590011d20 con 0x7fa5a01611b0 2024-04-02T09:09:28.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.919+0000 7fa5a5000700 1 -- 172.21.15.89:0/3088560854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5a01611b0 msgr2=0x7fa5a015ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:28.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.919+0000 7fa5a5000700 1 --2- 172.21.15.89:0/3088560854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5a01611b0 0x7fa5a015ef90 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fa5a00d2f60 tx=0x7fa590009b20 comp rx=0 tx=0).stop 2024-04-02T09:09:28.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.919+0000 7fa5a5000700 1 -- 172.21.15.89:0/3088560854 shutdown_connections 2024-04-02T09:09:28.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.919+0000 7fa5a5000700 1 --2- 172.21.15.89:0/3088560854 >> v2:172.21.15.113:3300/0 conn(0x7fa5a00cee10 0x7fa5a0161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.919+0000 7fa5a5000700 1 --2- 172.21.15.89:0/3088560854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5a01611b0 0x7fa5a015ef90 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.919+0000 7fa5a5000700 1 -- 172.21.15.89:0/3088560854 >> 172.21.15.89:0/3088560854 conn(0x7fa5a008aff0 msgr2=0x7fa5a00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:28.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.919+0000 7fa5a5000700 1 -- 172.21.15.89:0/3088560854 shutdown_connections 2024-04-02T09:09:28.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.919+0000 7fa5a5000700 1 -- 172.21.15.89:0/3088560854 wait complete. 2024-04-02T09:09:28.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.921+0000 7fa5a5000700 1 Processor -- start 2024-04-02T09:09:28.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.921+0000 7fa5a5000700 1 -- start start 2024-04-02T09:09:28.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.922+0000 7fa5a5000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5a00cee10 0x7fa5a00ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:28.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.922+0000 7fa5a5000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5a015f4d0 0x7fa5a00b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:28.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.922+0000 7fa5a5000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5a01611b0 0x7fa5a0156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:28.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.922+0000 7fa5a5000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a00b72a0 con 0x7fa5a01611b0 2024-04-02T09:09:28.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.922+0000 7fa5a5000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a00cac90 con 0x7fa5a015f4d0 2024-04-02T09:09:28.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.922+0000 7fa5a5000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a00caf70 con 0x7fa5a00cee10 2024-04-02T09:09:28.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.922+0000 7fa59e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5a01611b0 0x7fa5a0156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:28.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.922+0000 7fa59e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5a01611b0 0x7fa5a0156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47098/0 (socket says 172.21.15.89:47098) 2024-04-02T09:09:28.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.922+0000 7fa59e800700 1 -- 172.21.15.89:0/1967953427 learned_addr learned my addr 172.21.15.89:0/1967953427 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:28.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.922+0000 7fa59d400700 1 --2- 172.21.15.89:0/1967953427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5a015f4d0 0x7fa5a00b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:28.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.922+0000 7fa59de00700 1 --2- 172.21.15.89:0/1967953427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5a00cee10 0x7fa5a00ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:28.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.922+0000 7fa59e800700 1 -- 172.21.15.89:0/1967953427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5a00cee10 msgr2=0x7fa5a00ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:28.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.922+0000 7fa59e800700 1 --2- 172.21.15.89:0/1967953427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5a00cee10 0x7fa5a00ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.923+0000 7fa59e800700 1 -- 172.21.15.89:0/1967953427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5a015f4d0 msgr2=0x7fa5a00b6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:28.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.923+0000 7fa59e800700 1 --2- 172.21.15.89:0/1967953427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5a015f4d0 0x7fa5a00b6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:28.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.923+0000 7fa59e800700 1 -- 172.21.15.89:0/1967953427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5900094a0 con 0x7fa5a01611b0 2024-04-02T09:09:28.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.923+0000 7fa59e800700 1 --2- 172.21.15.89:0/1967953427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5a01611b0 0x7fa5a0156cb0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fa59800cef0 tx=0x7fa59800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:28.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.923+0000 7fa596000700 1 -- 172.21.15.89:0/1967953427 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59800ad00 con 0x7fa5a01611b0 2024-04-02T09:09:28.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.923+0000 7fa596000700 1 -- 172.21.15.89:0/1967953427 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa59800ae60 con 0x7fa5a01611b0 2024-04-02T09:09:28.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.923+0000 7fa5a5000700 1 -- 172.21.15.89:0/1967953427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a00cb250 con 0x7fa5a01611b0 2024-04-02T09:09:28.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.923+0000 7fa596000700 1 -- 172.21.15.89:0/1967953427 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa598015680 con 0x7fa5a01611b0 2024-04-02T09:09:28.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.923+0000 7fa5a5000700 1 -- 172.21.15.89:0/1967953427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a00cb810 con 0x7fa5a01611b0 2024-04-02T09:09:28.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.924+0000 7fa5a5000700 1 -- 172.21.15.89:0/1967953427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a00cba30 con 0x7fa5a01611b0 2024-04-02T09:09:28.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.925+0000 7fa596000700 1 -- 172.21.15.89:0/1967953427 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 130) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fa59800d430 con 0x7fa5a01611b0 2024-04-02T09:09:28.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.926+0000 7fa596000700 1 -- 172.21.15.89:0/1967953427 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 7161+0+0 (secure 0 0 0) 0x7fa5980898f0 con 0x7fa5a01611b0 2024-04-02T09:09:28.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:28.930+0000 7fa596000700 1 -- 172.21.15.89:0/1967953427 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa598054880 con 0x7fa5a01611b0 2024-04-02T09:09:29.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.092+0000 7fa596000700 1 -- 172.21.15.89:0/1967953427 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 131) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fa598054330 con 0x7fa5a01611b0 2024-04-02T09:09:29.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.092+0000 7fa596000700 1 --2- 172.21.15.89:0/1967953427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa580070b60 0x7fa580072f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:29.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.094+0000 7fa59de00700 1 --2- 172.21.15.89:0/1967953427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa580070b60 0x7fa580072f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:29.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.099+0000 7fa59de00700 1 --2- 172.21.15.89:0/1967953427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa580070b60 0x7fa580072f50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa5900090b0 tx=0x7fa59001c040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:29.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.186+0000 7fa5a5000700 1 -- 172.21.15.89:0/1967953427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa5a0160f40 con 0x7fa5a01611b0 2024-04-02T09:09:29.185 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T09:09:29.187+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T09:09:29.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.196+0000 7fa596000700 1 -- 172.21.15.89:0/1967953427 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v90) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa5980712a0 con 0x7fa5a01611b0 2024-04-02T09:09:29.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.198+0000 7fa5a5000700 1 -- 172.21.15.89:0/1967953427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa580070b60 msgr2=0x7fa580072f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:29.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.198+0000 7fa5a5000700 1 --2- 172.21.15.89:0/1967953427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa580070b60 0x7fa580072f50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa5900090b0 tx=0x7fa59001c040 comp rx=0 tx=0).stop 2024-04-02T09:09:29.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.198+0000 7fa5a5000700 1 -- 172.21.15.89:0/1967953427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5a01611b0 msgr2=0x7fa5a0156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:29.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.198+0000 7fa5a5000700 1 --2- 172.21.15.89:0/1967953427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5a01611b0 0x7fa5a0156cb0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fa59800cef0 tx=0x7fa59800cf20 comp rx=0 tx=0).stop 2024-04-02T09:09:29.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.199+0000 7fa5a5000700 1 -- 172.21.15.89:0/1967953427 shutdown_connections 2024-04-02T09:09:29.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.199+0000 7fa5a5000700 1 --2- 172.21.15.89:0/1967953427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5a01611b0 0x7fa5a0156cb0 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:29.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.199+0000 7fa5a5000700 1 --2- 172.21.15.89:0/1967953427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5a00cee10 0x7fa5a00ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:29.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.199+0000 7fa5a5000700 1 --2- 172.21.15.89:0/1967953427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa580070b60 0x7fa580072f50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:29.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.199+0000 7fa5a5000700 1 --2- 172.21.15.89:0/1967953427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5a015f4d0 0x7fa5a00b6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:29.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.199+0000 7fa5a5000700 1 -- 172.21.15.89:0/1967953427 >> 172.21.15.89:0/1967953427 conn(0x7fa5a008aff0 msgr2=0x7fa5a006bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:29.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.199+0000 7fa5a5000700 1 -- 172.21.15.89:0/1967953427 shutdown_connections 2024-04-02T09:09:29.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.199+0000 7fa5a5000700 1 -- 172.21.15.89:0/1967953427 wait complete. 2024-04-02T09:09:29.199 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T09:09:29.219 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T09:09:29.260 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T09:09:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.416+0000 7fbd2d400700 1 Processor -- start 2024-04-02T09:09:29.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.416+0000 7fbd2d400700 1 -- start start 2024-04-02T09:09:29.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.417+0000 7fbd2d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbd2815d230 0x7fbd280d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.417+0000 7fbd2d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbd280b8fe0 0x7fbd280b6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.417+0000 7fbd2d400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd280b6cf0 con 0x7fbd280b6870 2024-04-02T09:09:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.417+0000 7fbd2d400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd280b6fd0 con 0x7fbd280b8fe0 2024-04-02T09:09:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.417+0000 7fbd2d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd280b72b0 con 0x7fbd2815d230 2024-04-02T09:09:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.417+0000 7fbd27400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbd280b8fe0 0x7fbd280b6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.418+0000 7fbd27400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbd280b8fe0 0x7fbd280b6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47126/0 (socket says 172.21.15.89:47126) 2024-04-02T09:09:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.418+0000 7fbd27400700 1 -- 172.21.15.89:0/2486744601 learned_addr learned my addr 172.21.15.89:0/2486744601 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.418+0000 7fbd27e00700 1 --2- 172.21.15.89:0/2486744601 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbd2815d230 0x7fbd280d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.418+0000 7fbd27400700 1 -- 172.21.15.89:0/2486744601 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbd2815d230 msgr2=0x7fbd280d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.418+0000 7fbd27400700 1 --2- 172.21.15.89:0/2486744601 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbd2815d230 0x7fbd280d1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.418+0000 7fbd27400700 1 -- 172.21.15.89:0/2486744601 >> v1:172.21.15.113:6789/0 conn(0x7fbd280b6870 legacy=0x7fbd280d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.418+0000 7fbd27400700 1 -- 172.21.15.89:0/2486744601 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd280b7590 con 0x7fbd280b8fe0 2024-04-02T09:09:29.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.418+0000 7fbd27400700 1 --2- 172.21.15.89:0/2486744601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd280b8fe0 0x7fbd280b6330 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fbd140098f0 tx=0x7fbd14009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3b781a61a7ad64 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.418+0000 7fbd26a00700 1 -- 172.21.15.89:0/2486744601 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1400b8a0 con 0x7fbd280b8fe0 2024-04-02T09:09:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.418+0000 7fbd26a00700 1 -- 172.21.15.89:0/2486744601 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1400ba00 con 0x7fbd280b8fe0 2024-04-02T09:09:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.418+0000 7fbd26a00700 1 -- 172.21.15.89:0/2486744601 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1400bd90 con 0x7fbd280b8fe0 2024-04-02T09:09:29.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.419+0000 7fbd2d400700 1 -- 172.21.15.89:0/2486744601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd280b8fe0 msgr2=0x7fbd280b6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.419+0000 7fbd2d400700 1 --2- 172.21.15.89:0/2486744601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd280b8fe0 0x7fbd280b6330 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fbd140098f0 tx=0x7fbd14009c00 comp rx=0 tx=0).stop 2024-04-02T09:09:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.419+0000 7fbd2d400700 1 -- 172.21.15.89:0/2486744601 shutdown_connections 2024-04-02T09:09:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.419+0000 7fbd2d400700 1 --2- 172.21.15.89:0/2486744601 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbd2815d230 0x7fbd280d1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.419+0000 7fbd2d400700 1 --2- 172.21.15.89:0/2486744601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd280b8fe0 0x7fbd280b6330 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.419+0000 7fbd2d400700 1 -- 172.21.15.89:0/2486744601 >> 172.21.15.89:0/2486744601 conn(0x7fbd280668e0 msgr2=0x7fbd28066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.419+0000 7fbd2d400700 1 -- 172.21.15.89:0/2486744601 shutdown_connections 2024-04-02T09:09:29.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.419+0000 7fbd2d400700 1 -- 172.21.15.89:0/2486744601 wait complete. 2024-04-02T09:09:29.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.421+0000 7fbd2d400700 1 Processor -- start 2024-04-02T09:09:29.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.421+0000 7fbd2d400700 1 -- start start 2024-04-02T09:09:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.422+0000 7fbd2d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd280b6870 0x7fbd2815d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.422+0000 7fbd2d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbd280b8fe0 0x7fbd28156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.422+0000 7fbd2d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd2815d230 0x7fbd280cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.422+0000 7fbd2d400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2815e790 con 0x7fbd280b6870 2024-04-02T09:09:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.422+0000 7fbd2d400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2815e4b0 con 0x7fbd2815d230 2024-04-02T09:09:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.422+0000 7fbd2d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd281f66a0 con 0x7fbd280b8fe0 2024-04-02T09:09:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.422+0000 7fbd27e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd280b6870 0x7fbd2815d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:29.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.422+0000 7fbd27e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd280b6870 0x7fbd2815d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47128/0 (socket says 172.21.15.89:47128) 2024-04-02T09:09:29.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.422+0000 7fbd27e00700 1 -- 172.21.15.89:0/1979405300 learned_addr learned my addr 172.21.15.89:0/1979405300 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:29.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.422+0000 7fbd27400700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbd280b8fe0 0x7fbd28156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:29.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.422+0000 7fbd2ca00700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd2815d230 0x7fbd280cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:29.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.423+0000 7fbd27e00700 1 -- 172.21.15.89:0/1979405300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbd280b8fe0 msgr2=0x7fbd28156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:29.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.423+0000 7fbd27e00700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbd280b8fe0 0x7fbd28156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:29.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.423+0000 7fbd27e00700 1 -- 172.21.15.89:0/1979405300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd2815d230 msgr2=0x7fbd280cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:29.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.423+0000 7fbd27e00700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd2815d230 0x7fbd280cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:29.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.423+0000 7fbd27e00700 1 -- 172.21.15.89:0/1979405300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd14009550 con 0x7fbd280b6870 2024-04-02T09:09:29.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.423+0000 7fbd2ca00700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd2815d230 0x7fbd280cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:29.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.423+0000 7fbd27e00700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd280b6870 0x7fbd2815d680 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800d800 tx=0x7fbd1800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:29.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.423+0000 7fbd1fe00700 1 -- 172.21.15.89:0/1979405300 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd180125f0 con 0x7fbd280b6870 2024-04-02T09:09:29.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.423+0000 7fbd1fe00700 1 -- 172.21.15.89:0/1979405300 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd18012750 con 0x7fbd280b6870 2024-04-02T09:09:29.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.423+0000 7fbd2d400700 1 -- 172.21.15.89:0/1979405300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd281f6980 con 0x7fbd280b6870 2024-04-02T09:09:29.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.423+0000 7fbd1fe00700 1 -- 172.21.15.89:0/1979405300 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd18012b00 con 0x7fbd280b6870 2024-04-02T09:09:29.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.423+0000 7fbd2d400700 1 -- 172.21.15.89:0/1979405300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd2815dbc0 con 0x7fbd280b6870 2024-04-02T09:09:29.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.424+0000 7fbd2d400700 1 -- 172.21.15.89:0/1979405300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd281f6500 con 0x7fbd280b6870 2024-04-02T09:09:29.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.425+0000 7fbd1fe00700 1 -- 172.21.15.89:0/1979405300 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 131) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fbd18012c60 con 0x7fbd280b6870 2024-04-02T09:09:29.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.426+0000 7fbd1fe00700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbd0806aae0 0x7fbd0806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:29.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.427+0000 7fbd1fe00700 1 -- 172.21.15.89:0/1979405300 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 7247+0+0 (secure 0 0 0) 0x7fbd18089830 con 0x7fbd280b6870 2024-04-02T09:09:29.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.427+0000 7fbd27400700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbd0806aae0 0x7fbd0806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:29.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.427+0000 7fbd27400700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbd0806aae0 0x7fbd0806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009c40 tx=0x7fbd14008030 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:29.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.431+0000 7fbd1fe00700 1 -- 172.21.15.89:0/1979405300 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbd281f6500 con 0x7fbd280b6870 2024-04-02T09:09:29.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:29.677+0000 7fbd2d400700 1 -- 172.21.15.89:0/1979405300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbd281f6ea0 con 0x7fbd280b6870 2024-04-02T09:09:30.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.198+0000 7fbd1fe00700 1 -- 172.21.15.89:0/1979405300 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v91) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbd281f6ea0 con 0x7fbd280b6870 2024-04-02T09:09:30.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.200+0000 7fbd2d400700 1 -- 172.21.15.89:0/1979405300 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbd0806aae0 msgr2=0x7fbd0806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:30.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.200+0000 7fbd2d400700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbd0806aae0 0x7fbd0806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009c40 tx=0x7fbd14008030 comp rx=0 tx=0).stop 2024-04-02T09:09:30.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.200+0000 7fbd2d400700 1 -- 172.21.15.89:0/1979405300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd280b6870 msgr2=0x7fbd2815d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:30.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.200+0000 7fbd2d400700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd280b6870 0x7fbd2815d680 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800d800 tx=0x7fbd1800db10 comp rx=0 tx=0).stop 2024-04-02T09:09:30.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.201+0000 7fbd2d400700 1 -- 172.21.15.89:0/1979405300 shutdown_connections 2024-04-02T09:09:30.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.201+0000 7fbd2d400700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbd0806aae0 0x7fbd0806cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:30.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.201+0000 7fbd2d400700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd2815d230 0x7fbd280cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:30.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.201+0000 7fbd2d400700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd280b6870 0x7fbd2815d680 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:30.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.201+0000 7fbd2d400700 1 --2- 172.21.15.89:0/1979405300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbd280b8fe0 0x7fbd28156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:30.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.201+0000 7fbd2d400700 1 -- 172.21.15.89:0/1979405300 >> 172.21.15.89:0/1979405300 conn(0x7fbd280668e0 msgr2=0x7fbd280b4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:30.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.201+0000 7fbd2d400700 1 -- 172.21.15.89:0/1979405300 shutdown_connections 2024-04-02T09:09:30.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.201+0000 7fbd2d400700 1 -- 172.21.15.89:0/1979405300 wait complete. 2024-04-02T09:09:30.220 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T09:09:30.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.459+0000 7f081de00700 1 Processor -- start 2024-04-02T09:09:30.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.459+0000 7f081de00700 1 -- start start 2024-04-02T09:09:30.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.460+0000 7f081de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f081815f260 0x7f08180d1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:30.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.460+0000 7f081de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f081815f6c0 0x7f08180d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:30.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.460+0000 7f081de00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08180ce430 con 0x7f08180b5580 2024-04-02T09:09:30.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.460+0000 7f081de00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08180ce710 con 0x7f081815f260 2024-04-02T09:09:30.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.460+0000 7f081de00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08180ce9f0 con 0x7f081815f6c0 2024-04-02T09:09:30.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.461+0000 7f0816a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f08180b5580 0x7f081815e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36334/0 (socket says 172.21.15.89:36334) 2024-04-02T09:09:30.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.461+0000 7f0816a00700 1 -- 172.21.15.89:0/204191863 learned_addr learned my addr 172.21.15.89:0/204191863 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:30.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.461+0000 7f080fe00700 1 --2- 172.21.15.89:0/204191863 >> v2:172.21.15.89:3300/0 conn(0x7f081815f260 0x7f08180d1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:30.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.461+0000 7f0817400700 1 --2- 172.21.15.89:0/204191863 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f081815f6c0 0x7f08180d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:30.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.461+0000 7f0816000700 1 -- 172.21.15.89:0/204191863 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147539057 0 0) 0x7f08180ce430 con 0x7f08180b5580 2024-04-02T09:09:30.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.461+0000 7f0816000700 1 -- 172.21.15.89:0/204191863 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07fc0035b0 con 0x7f08180b5580 2024-04-02T09:09:30.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.461+0000 7f080fe00700 1 -- 172.21.15.89:0/204191863 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f081815f6c0 msgr2=0x7f08180d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:30.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.461+0000 7f080fe00700 1 --2- 172.21.15.89:0/204191863 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f081815f6c0 0x7f08180d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:30.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.461+0000 7f080fe00700 1 -- 172.21.15.89:0/204191863 >> v1:172.21.15.89:6789/0 conn(0x7f08180b5580 legacy=0x7f081815e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:30.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.461+0000 7f080fe00700 1 -- 172.21.15.89:0/204191863 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08180cecd0 con 0x7f081815f260 2024-04-02T09:09:30.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.462+0000 7f080fe00700 1 --2- 172.21.15.89:0/204191863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f081815f260 0x7f08180d1ae0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f0800009840 tx=0x7f0800009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb63f82632fed71 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:30.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.462+0000 7f0816000700 1 -- 172.21.15.89:0/204191863 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800011830 con 0x7f081815f260 2024-04-02T09:09:30.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.462+0000 7f0816000700 1 -- 172.21.15.89:0/204191863 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0800011990 con 0x7f081815f260 2024-04-02T09:09:30.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.462+0000 7f0816000700 1 -- 172.21.15.89:0/204191863 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800011d20 con 0x7f081815f260 2024-04-02T09:09:30.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.462+0000 7f081de00700 1 -- 172.21.15.89:0/204191863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f081815f260 msgr2=0x7f08180d1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:30.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.462+0000 7f081de00700 1 --2- 172.21.15.89:0/204191863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f081815f260 0x7f08180d1ae0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f0800009840 tx=0x7f0800009b50 comp rx=0 tx=0).stop 2024-04-02T09:09:30.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.462+0000 7f081de00700 1 -- 172.21.15.89:0/204191863 shutdown_connections 2024-04-02T09:09:30.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.462+0000 7f081de00700 1 --2- 172.21.15.89:0/204191863 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f081815f6c0 0x7f08180d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:30.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.462+0000 7f081de00700 1 --2- 172.21.15.89:0/204191863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f081815f260 0x7f08180d1ae0 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:30.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.462+0000 7f081de00700 1 -- 172.21.15.89:0/204191863 >> 172.21.15.89:0/204191863 conn(0x7f081808aff0 msgr2=0x7f08180d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:30.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.462+0000 7f081de00700 1 -- 172.21.15.89:0/204191863 shutdown_connections 2024-04-02T09:09:30.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.463+0000 7f081de00700 1 -- 172.21.15.89:0/204191863 wait complete. 2024-04-02T09:09:30.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.464+0000 7f081de00700 1 Processor -- start 2024-04-02T09:09:30.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.464+0000 7f081de00700 1 -- start start 2024-04-02T09:09:30.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.465+0000 7f081de00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08180b5580 0x7f08181f2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:30.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.465+0000 7f081de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f081815f260 0x7f08180b1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:30.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.465+0000 7f081de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f081815f6c0 0x7f08180c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:30.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.465+0000 7f081de00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081815bce0 con 0x7f081815f6c0 2024-04-02T09:09:30.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.465+0000 7f081de00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081815b0d0 con 0x7f08180b5580 2024-04-02T09:09:30.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.465+0000 7f081de00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081815b3b0 con 0x7f081815f260 2024-04-02T09:09:30.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.465+0000 7f0817400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f081815f6c0 0x7f08180c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:30.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.465+0000 7f0817400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f081815f6c0 0x7f08180c7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47168/0 (socket says 172.21.15.89:47168) 2024-04-02T09:09:30.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.465+0000 7f0817400700 1 -- 172.21.15.89:0/1976776835 learned_addr learned my addr 172.21.15.89:0/1976776835 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:30.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.466+0000 7f080fe00700 1 --2- 172.21.15.89:0/1976776835 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f081815f260 0x7f08180b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:30.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.466+0000 7f0816a00700 1 --2- 172.21.15.89:0/1976776835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08180b5580 0x7f08181f2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:30.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.466+0000 7f0817400700 1 -- 172.21.15.89:0/1976776835 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f081815f260 msgr2=0x7f08180b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:30.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.466+0000 7f0817400700 1 --2- 172.21.15.89:0/1976776835 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f081815f260 0x7f08180b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:30.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.466+0000 7f0817400700 1 -- 172.21.15.89:0/1976776835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08180b5580 msgr2=0x7f08181f2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:30.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.466+0000 7f0817400700 1 --2- 172.21.15.89:0/1976776835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08180b5580 0x7f08181f2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:30.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.466+0000 7f0817400700 1 -- 172.21.15.89:0/1976776835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08000094a0 con 0x7f081815f6c0 2024-04-02T09:09:30.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.466+0000 7f0817400700 1 --2- 172.21.15.89:0/1976776835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f081815f6c0 0x7f08180c7c90 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f0810007e60 tx=0x7f081000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:30.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.466+0000 7f080f400700 1 -- 172.21.15.89:0/1976776835 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081000ede0 con 0x7f081815f6c0 2024-04-02T09:09:30.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.466+0000 7f080f400700 1 -- 172.21.15.89:0/1976776835 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0810014420 con 0x7f081815f6c0 2024-04-02T09:09:30.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.466+0000 7f080f400700 1 -- 172.21.15.89:0/1976776835 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08100147d0 con 0x7f081815f6c0 2024-04-02T09:09:30.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.466+0000 7f081de00700 1 -- 172.21.15.89:0/1976776835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f081815b690 con 0x7f081815f6c0 2024-04-02T09:09:30.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.466+0000 7f081de00700 1 -- 172.21.15.89:0/1976776835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f081815acf0 con 0x7f081815f6c0 2024-04-02T09:09:30.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.467+0000 7f081de00700 1 -- 172.21.15.89:0/1976776835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f081815ae80 con 0x7f081815f6c0 2024-04-02T09:09:30.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.468+0000 7f080f400700 1 -- 172.21.15.89:0/1976776835 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 132) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f0810014930 con 0x7f081815f6c0 2024-04-02T09:09:30.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.469+0000 7f080f400700 1 --2- 172.21.15.89:0/1976776835 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f07f406ad20 0x7f07f406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:30.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.469+0000 7f080f400700 1 -- 172.21.15.89:0/1976776835 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 7247+0+0 (secure 0 0 0) 0x7f081008a1b0 con 0x7f081815f6c0 2024-04-02T09:09:30.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.469+0000 7f0816a00700 1 --2- 172.21.15.89:0/1976776835 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f07f406ad20 0x7f07f406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:30.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.470+0000 7f0816a00700 1 --2- 172.21.15.89:0/1976776835 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f07f406ad20 0x7f07f406d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f08080011e0 tx=0x7f0808009040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.474+0000 7f080f400700 1 -- 172.21.15.89:0/1976776835 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0810054c10 con 0x7f081815f6c0 2024-04-02T09:09:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:30.673+0000 7f081de00700 1 -- 172.21.15.89:0/1976776835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0818067720 con 0x7f081815f6c0 2024-04-02T09:09:31.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.204+0000 7f080f400700 1 -- 172.21.15.89:0/1976776835 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v195) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0810058230 con 0x7f081815f6c0 2024-04-02T09:09:31.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.207+0000 7f081de00700 1 -- 172.21.15.89:0/1976776835 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f07f406ad20 msgr2=0x7f07f406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:31.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.207+0000 7f081de00700 1 --2- 172.21.15.89:0/1976776835 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f07f406ad20 0x7f07f406d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f08080011e0 tx=0x7f0808009040 comp rx=0 tx=0).stop 2024-04-02T09:09:31.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.207+0000 7f081de00700 1 -- 172.21.15.89:0/1976776835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f081815f6c0 msgr2=0x7f08180c7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:31.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.207+0000 7f081de00700 1 --2- 172.21.15.89:0/1976776835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f081815f6c0 0x7f08180c7c90 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f0810007e60 tx=0x7f081000e3c0 comp rx=0 tx=0).stop 2024-04-02T09:09:31.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.207+0000 7f081de00700 1 -- 172.21.15.89:0/1976776835 shutdown_connections 2024-04-02T09:09:31.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.207+0000 7f081de00700 1 --2- 172.21.15.89:0/1976776835 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f07f406ad20 0x7f07f406d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:31.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.207+0000 7f081de00700 1 --2- 172.21.15.89:0/1976776835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08180b5580 0x7f08181f2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:31.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.207+0000 7f081de00700 1 --2- 172.21.15.89:0/1976776835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f081815f6c0 0x7f08180c7c90 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:31.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.207+0000 7f081de00700 1 --2- 172.21.15.89:0/1976776835 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f081815f260 0x7f08180b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:31.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.207+0000 7f081de00700 1 -- 172.21.15.89:0/1976776835 >> 172.21.15.89:0/1976776835 conn(0x7f081808aff0 msgr2=0x7f08180b3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:31.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.207+0000 7f081de00700 1 -- 172.21.15.89:0/1976776835 shutdown_connections 2024-04-02T09:09:31.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.207+0000 7f081de00700 1 -- 172.21.15.89:0/1976776835 wait complete. 2024-04-02T09:09:31.206 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T09:09:31.224 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T09:09:31.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.420+0000 7efd4b000700 1 Processor -- start 2024-04-02T09:09:31.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.421+0000 7efd4b000700 1 -- start start 2024-04-02T09:09:31.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.422+0000 7efd4b000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efd440d28e0 0x7efd44168680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:31.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.422+0000 7efd4b000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd440ce8a0 con 0x7efd440ced30 2024-04-02T09:09:31.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.422+0000 7efd4b000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd440cdc90 con 0x7efd4415ddf0 2024-04-02T09:09:31.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.422+0000 7efd4b000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd4415ec60 con 0x7efd440d28e0 2024-04-02T09:09:31.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.422+0000 7efd43400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efd440ced30 0x7efd440d21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36342/0 (socket says 172.21.15.89:36342) 2024-04-02T09:09:31.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.422+0000 7efd43400700 1 -- 172.21.15.89:0/3903074287 learned_addr learned my addr 172.21.15.89:0/3903074287 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:31.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.422+0000 7efd49000700 1 --2- 172.21.15.89:0/3903074287 >> v2:172.21.15.113:3300/0 conn(0x7efd440d28e0 0x7efd44168680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:31.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.422+0000 7efd42a00700 1 -- 172.21.15.89:0/3903074287 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512833297 0 0) 0x7efd440ce8a0 con 0x7efd440ced30 2024-04-02T09:09:31.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd42a00700 1 -- 172.21.15.89:0/3903074287 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd280035b0 con 0x7efd440ced30 2024-04-02T09:09:31.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd42a00700 1 -- 172.21.15.89:0/3903074287 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1183812839 0 0) 0x7efd280035b0 con 0x7efd440ced30 2024-04-02T09:09:31.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd42a00700 1 -- 172.21.15.89:0/3903074287 >> v2:172.21.15.113:3300/0 conn(0x7efd440d28e0 msgr2=0x7efd44168680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:31.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd42a00700 1 --2- 172.21.15.89:0/3903074287 >> v2:172.21.15.113:3300/0 conn(0x7efd440d28e0 0x7efd44168680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:31.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd42a00700 1 -- 172.21.15.89:0/3903074287 >> v1:172.21.15.113:6789/0 conn(0x7efd4415ddf0 legacy=0x7efd4415e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:31.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd42a00700 1 -- 172.21.15.89:0/3903074287 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd440cdf70 con 0x7efd440ced30 2024-04-02T09:09:31.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd42a00700 1 -- 172.21.15.89:0/3903074287 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7efd30001670 con 0x7efd440ced30 2024-04-02T09:09:31.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd49000700 1 --2- 172.21.15.89:0/3903074287 >> v2:172.21.15.113:3300/0 conn(0x7efd440d28e0 0x7efd44168680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:09:31.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd42a00700 1 -- 172.21.15.89:0/3903074287 >> v1:172.21.15.89:6789/0 conn(0x7efd440ced30 legacy=0x7efd440d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:31.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd42a00700 1 --2- 172.21.15.89:0/3903074287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd28003a00 0x7efd28003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:31.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd42a00700 1 --2- 172.21.15.89:0/3903074287 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd28004850 0x7efd28004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:31.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd42a00700 1 --2- 172.21.15.89:0/3903074287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd28005210 0x7efd2800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:31.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd42a00700 1 -- 172.21.15.89:0/3903074287 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4415ec60 con 0x7efd28003a00 2024-04-02T09:09:31.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd42a00700 1 -- 172.21.15.89:0/3903074287 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2800c1b0 con 0x7efd28005210 2024-04-02T09:09:31.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd42a00700 1 -- 172.21.15.89:0/3903074287 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd2800c370 con 0x7efd28004850 2024-04-02T09:09:31.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd43e00700 1 --2- 172.21.15.89:0/3903074287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd28003a00 0x7efd28003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:31.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd43400700 1 --2- 172.21.15.89:0/3903074287 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd28004850 0x7efd28004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:31.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.423+0000 7efd49000700 1 --2- 172.21.15.89:0/3903074287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd28005210 0x7efd2800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:31.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.424+0000 7efd43e00700 1 -- 172.21.15.89:0/3903074287 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd28004850 msgr2=0x7efd28004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:31.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.424+0000 7efd43e00700 1 --2- 172.21.15.89:0/3903074287 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd28004850 0x7efd28004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:31.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.424+0000 7efd43e00700 1 -- 172.21.15.89:0/3903074287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd28005210 msgr2=0x7efd2800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:31.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.424+0000 7efd43e00700 1 --2- 172.21.15.89:0/3903074287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd28005210 0x7efd2800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:31.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.424+0000 7efd43e00700 1 -- 172.21.15.89:0/3903074287 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd280035b0 con 0x7efd28003a00 2024-04-02T09:09:31.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.424+0000 7efd49000700 1 -- 172.21.15.89:0/3903074287 reap_dead start 2024-04-02T09:09:31.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.424+0000 7efd43e00700 1 --2- 172.21.15.89:0/3903074287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd28003a00 0x7efd28003e20 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7efd340017c0 tx=0x7efd28005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2453a4fbcd9b0bc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:31.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.424+0000 7efd42a00700 1 -- 172.21.15.89:0/3903074287 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340052d0 con 0x7efd28003a00 2024-04-02T09:09:31.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.424+0000 7efd42a00700 1 -- 172.21.15.89:0/3903074287 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd34005450 con 0x7efd28003a00 2024-04-02T09:09:31.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.424+0000 7efd42a00700 1 -- 172.21.15.89:0/3903074287 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340038c0 con 0x7efd28003a00 2024-04-02T09:09:31.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.425+0000 7efd4b000700 1 -- 172.21.15.89:0/3903074287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd28003a00 msgr2=0x7efd28003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:31.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.425+0000 7efd4b000700 1 --2- 172.21.15.89:0/3903074287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd28003a00 0x7efd28003e20 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7efd340017c0 tx=0x7efd28005880 comp rx=0 tx=0).stop 2024-04-02T09:09:31.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.425+0000 7efd4b000700 1 -- 172.21.15.89:0/3903074287 shutdown_connections 2024-04-02T09:09:31.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.425+0000 7efd4b000700 1 --2- 172.21.15.89:0/3903074287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd28003a00 0x7efd28003e20 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:31.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.425+0000 7efd4b000700 1 -- 172.21.15.89:0/3903074287 >> 172.21.15.89:0/3903074287 conn(0x7efd4408aff0 msgr2=0x7efd440d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:31.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.425+0000 7efd4b000700 1 -- 172.21.15.89:0/3903074287 shutdown_connections 2024-04-02T09:09:31.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.425+0000 7efd4b000700 1 -- 172.21.15.89:0/3903074287 wait complete. 2024-04-02T09:09:31.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.426+0000 7efd4b000700 1 Processor -- start 2024-04-02T09:09:31.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.427+0000 7efd4b000700 1 -- start start 2024-04-02T09:09:31.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.427+0000 7efd4b000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd28003a00 0x7efd440b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:31.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.427+0000 7efd4b000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd440b27d0 0x7efd440b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:31.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.427+0000 7efd4b000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd441f3570 0x7efd441f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:31.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.428+0000 7efd4b000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd440b3450 con 0x7efd440b27d0 2024-04-02T09:09:31.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.428+0000 7efd4b000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd440c7d60 con 0x7efd441f3570 2024-04-02T09:09:31.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.428+0000 7efd4b000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd440b3190 con 0x7efd28003a00 2024-04-02T09:09:31.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.428+0000 7efd43400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd440b27d0 0x7efd440b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:31.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.428+0000 7efd43400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd440b27d0 0x7efd440b2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47184/0 (socket says 172.21.15.89:47184) 2024-04-02T09:09:31.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.428+0000 7efd43400700 1 -- 172.21.15.89:0/2340970712 learned_addr learned my addr 172.21.15.89:0/2340970712 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:31.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.428+0000 7efd43e00700 1 --2- 172.21.15.89:0/2340970712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd28003a00 0x7efd440b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:31.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.428+0000 7efd49000700 1 --2- 172.21.15.89:0/2340970712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd441f3570 0x7efd441f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:31.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.428+0000 7efd43400700 1 -- 172.21.15.89:0/2340970712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd28003a00 msgr2=0x7efd440b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:31.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.428+0000 7efd43400700 1 --2- 172.21.15.89:0/2340970712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd28003a00 0x7efd440b1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:31.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.428+0000 7efd43400700 1 -- 172.21.15.89:0/2340970712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd441f3570 msgr2=0x7efd441f3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:31.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.428+0000 7efd43400700 1 --2- 172.21.15.89:0/2340970712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd441f3570 0x7efd441f3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:31.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.428+0000 7efd43400700 1 -- 172.21.15.89:0/2340970712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd440c6fa0 con 0x7efd440b27d0 2024-04-02T09:09:31.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.428+0000 7efd43400700 1 --2- 172.21.15.89:0/2340970712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd440b27d0 0x7efd440b2c50 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7efd3000e730 tx=0x7efd3000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:31.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.429+0000 7efd3be00700 1 -- 172.21.15.89:0/2340970712 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd30012590 con 0x7efd440b27d0 2024-04-02T09:09:31.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.429+0000 7efd3be00700 1 -- 172.21.15.89:0/2340970712 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd30012710 con 0x7efd440b27d0 2024-04-02T09:09:31.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.429+0000 7efd3be00700 1 -- 172.21.15.89:0/2340970712 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd30012ac0 con 0x7efd440b27d0 2024-04-02T09:09:31.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.429+0000 7efd4b000700 1 -- 172.21.15.89:0/2340970712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd440c7280 con 0x7efd440b27d0 2024-04-02T09:09:31.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.429+0000 7efd4b000700 1 -- 172.21.15.89:0/2340970712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd440c7840 con 0x7efd440b27d0 2024-04-02T09:09:31.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.430+0000 7efd4b000700 1 -- 172.21.15.89:0/2340970712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd44067720 con 0x7efd440b27d0 2024-04-02T09:09:31.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.431+0000 7efd3be00700 1 -- 172.21.15.89:0/2340970712 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 132) v1 ==== 89183+0+0 (secure 0 0 0) 0x7efd30012c20 con 0x7efd440b27d0 2024-04-02T09:09:31.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.432+0000 7efd3be00700 1 --2- 172.21.15.89:0/2340970712 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efd2406ad20 0x7efd2406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:31.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.432+0000 7efd3be00700 1 -- 172.21.15.89:0/2340970712 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6809+0+0 (secure 0 0 0) 0x7efd30089540 con 0x7efd440b27d0 2024-04-02T09:09:31.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.432+0000 7efd43e00700 1 --2- 172.21.15.89:0/2340970712 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efd2406ad20 0x7efd2406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:31.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.432+0000 7efd43e00700 1 --2- 172.21.15.89:0/2340970712 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efd2406ad20 0x7efd2406d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efd34000c00 tx=0x7efd34009030 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:31.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.436+0000 7efd3be00700 1 -- 172.21.15.89:0/2340970712 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efd30057c80 con 0x7efd440b27d0 2024-04-02T09:09:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:31.636+0000 7efd4b000700 1 -- 172.21.15.89:0/2340970712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efd440b5ce0 con 0x7efd440b27d0 2024-04-02T09:09:32.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.215+0000 7efd3be00700 1 -- 172.21.15.89:0/2340970712 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7efd300545c0 con 0x7efd440b27d0 2024-04-02T09:09:32.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.216+0000 7efd3be00700 1 -- 172.21.15.89:0/2340970712 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v196) v1 ==== 169+0+0 (secure 0 0 0) 0x7efd30074660 con 0x7efd440b27d0 2024-04-02T09:09:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.219+0000 7efd4b000700 1 -- 172.21.15.89:0/2340970712 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efd2406ad20 msgr2=0x7efd2406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.219+0000 7efd4b000700 1 --2- 172.21.15.89:0/2340970712 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efd2406ad20 0x7efd2406d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efd34000c00 tx=0x7efd34009030 comp rx=0 tx=0).stop 2024-04-02T09:09:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.219+0000 7efd4b000700 1 -- 172.21.15.89:0/2340970712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd440b27d0 msgr2=0x7efd440b2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.219+0000 7efd4b000700 1 --2- 172.21.15.89:0/2340970712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd440b27d0 0x7efd440b2c50 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7efd3000e730 tx=0x7efd3000ea40 comp rx=0 tx=0).stop 2024-04-02T09:09:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.219+0000 7efd4b000700 1 -- 172.21.15.89:0/2340970712 shutdown_connections 2024-04-02T09:09:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.219+0000 7efd4b000700 1 --2- 172.21.15.89:0/2340970712 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efd2406ad20 0x7efd2406d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.219+0000 7efd4b000700 1 --2- 172.21.15.89:0/2340970712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd441f3570 0x7efd441f3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.219+0000 7efd4b000700 1 --2- 172.21.15.89:0/2340970712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd28003a00 0x7efd440b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.219+0000 7efd4b000700 1 --2- 172.21.15.89:0/2340970712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd440b27d0 0x7efd440b2c50 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.219+0000 7efd4b000700 1 -- 172.21.15.89:0/2340970712 >> 172.21.15.89:0/2340970712 conn(0x7efd4408aff0 msgr2=0x7efd441606b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.219+0000 7efd4b000700 1 -- 172.21.15.89:0/2340970712 shutdown_connections 2024-04-02T09:09:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.219+0000 7efd4b000700 1 -- 172.21.15.89:0/2340970712 wait complete. 2024-04-02T09:09:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T09:09:32.237 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T09:09:32.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.433+0000 7fbfbc600700 1 Processor -- start 2024-04-02T09:09:32.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.434+0000 7fbfbc600700 1 -- start start 2024-04-02T09:09:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.435+0000 7fbfbc600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbfb40cbb30 0x7fbfb40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.435+0000 7fbfbc600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb40cd020 0x7fbfb40cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.435+0000 7fbfbc600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb40cc450 con 0x7fbfb40b2070 2024-04-02T09:09:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.435+0000 7fbfbc600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb40cdae0 con 0x7fbfb40cbb30 2024-04-02T09:09:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.435+0000 7fbfbc600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb40cddc0 con 0x7fbfb40cd020 2024-04-02T09:09:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.435+0000 7fbfb9c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbfb40cbb30 0x7fbfb40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.435+0000 7fbfb9c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbfb40cbb30 0x7fbfb40cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47198/0 (socket says 172.21.15.89:47198) 2024-04-02T09:09:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.435+0000 7fbfb9c00700 1 -- 172.21.15.89:0/1550944559 learned_addr learned my addr 172.21.15.89:0/1550944559 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.436+0000 7fbfb9c00700 1 -- 172.21.15.89:0/1550944559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb40cd020 msgr2=0x7fbfb40cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:32.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.436+0000 7fbfb9c00700 1 --2- 172.21.15.89:0/1550944559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb40cd020 0x7fbfb40cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.436+0000 7fbfb9c00700 1 -- 172.21.15.89:0/1550944559 >> v1:172.21.15.113:6789/0 conn(0x7fbfb40b2070 legacy=0x7fbfb406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:32.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.436+0000 7fbfb9c00700 1 -- 172.21.15.89:0/1550944559 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb40ce0a0 con 0x7fbfb40cbb30 2024-04-02T09:09:32.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.436+0000 7fbfb9c00700 1 --2- 172.21.15.89:0/1550944559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb40cbb30 0x7fbfb40cbf10 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fbfa80098f0 tx=0x7fbfa8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd159a1bf622f11 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:32.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.436+0000 7fbfb3e00700 1 -- 172.21.15.89:0/1550944559 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8004230 con 0x7fbfb40cbb30 2024-04-02T09:09:32.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.436+0000 7fbfb3e00700 1 -- 172.21.15.89:0/1550944559 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa8004390 con 0x7fbfb40cbb30 2024-04-02T09:09:32.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.436+0000 7fbfb3e00700 1 -- 172.21.15.89:0/1550944559 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa80036a0 con 0x7fbfb40cbb30 2024-04-02T09:09:32.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.437+0000 7fbfbc600700 1 -- 172.21.15.89:0/1550944559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb40cbb30 msgr2=0x7fbfb40cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:32.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.437+0000 7fbfbc600700 1 --2- 172.21.15.89:0/1550944559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb40cbb30 0x7fbfb40cbf10 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fbfa80098f0 tx=0x7fbfa8009c00 comp rx=0 tx=0).stop 2024-04-02T09:09:32.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.437+0000 7fbfbc600700 1 -- 172.21.15.89:0/1550944559 shutdown_connections 2024-04-02T09:09:32.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.437+0000 7fbfbc600700 1 --2- 172.21.15.89:0/1550944559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb40cbb30 0x7fbfb40cbf10 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.437+0000 7fbfbc600700 1 --2- 172.21.15.89:0/1550944559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb40cd020 0x7fbfb40cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.437+0000 7fbfbc600700 1 -- 172.21.15.89:0/1550944559 >> 172.21.15.89:0/1550944559 conn(0x7fbfb40b0d70 msgr2=0x7fbfb40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:32.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.437+0000 7fbfbc600700 1 -- 172.21.15.89:0/1550944559 shutdown_connections 2024-04-02T09:09:32.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.437+0000 7fbfbc600700 1 -- 172.21.15.89:0/1550944559 wait complete. 2024-04-02T09:09:32.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.438+0000 7fbfbc600700 1 Processor -- start 2024-04-02T09:09:32.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.439+0000 7fbfbc600700 1 -- start start 2024-04-02T09:09:32.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.439+0000 7fbfbc600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb40b2070 0x7fbfb415d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:32.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.439+0000 7fbfbc600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb40cbb30 0x7fbfb4149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:32.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.439+0000 7fbfbc600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfb40cd020 0x7fbfb415e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:32.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.439+0000 7fbfbc600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb40716c0 con 0x7fbfb40b2070 2024-04-02T09:09:32.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.439+0000 7fbfbc600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb40cb0a0 con 0x7fbfb40cd020 2024-04-02T09:09:32.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.439+0000 7fbfbc600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb415cd30 con 0x7fbfb40cbb30 2024-04-02T09:09:32.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.440+0000 7fbfb9c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb40b2070 0x7fbfb415d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:32.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.440+0000 7fbfb9c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb40b2070 0x7fbfb415d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47202/0 (socket says 172.21.15.89:47202) 2024-04-02T09:09:32.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.440+0000 7fbfb9c00700 1 -- 172.21.15.89:0/1403972197 learned_addr learned my addr 172.21.15.89:0/1403972197 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:32.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.440+0000 7fbfb9200700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb40cbb30 0x7fbfb4149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:32.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.440+0000 7fbfba600700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfb40cd020 0x7fbfb415e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:32.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.440+0000 7fbfb9c00700 1 -- 172.21.15.89:0/1403972197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb40cbb30 msgr2=0x7fbfb4149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:32.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.440+0000 7fbfb9c00700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb40cbb30 0x7fbfb4149c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.440+0000 7fbfb9c00700 1 -- 172.21.15.89:0/1403972197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfb40cd020 msgr2=0x7fbfb415e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:32.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.440+0000 7fbfb9c00700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfb40cd020 0x7fbfb415e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.440+0000 7fbfb9c00700 1 -- 172.21.15.89:0/1403972197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa8009550 con 0x7fbfb40b2070 2024-04-02T09:09:32.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.440+0000 7fbfba600700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfb40cd020 0x7fbfb415e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:32.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.440+0000 7fbfb9c00700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb40b2070 0x7fbfb415d810 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8009130 tx=0x7fbfa8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:32.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.441+0000 7fbfb1600700 1 -- 172.21.15.89:0/1403972197 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa800b860 con 0x7fbfb40b2070 2024-04-02T09:09:32.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.441+0000 7fbfb1600700 1 -- 172.21.15.89:0/1403972197 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa800b9c0 con 0x7fbfb40b2070 2024-04-02T09:09:32.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.441+0000 7fbfb1600700 1 -- 172.21.15.89:0/1403972197 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa800bd70 con 0x7fbfb40b2070 2024-04-02T09:09:32.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.441+0000 7fbfbc600700 1 -- 172.21.15.89:0/1403972197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb415d010 con 0x7fbfb40b2070 2024-04-02T09:09:32.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.441+0000 7fbfbc600700 1 -- 172.21.15.89:0/1403972197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb415ea20 con 0x7fbfb40b2070 2024-04-02T09:09:32.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.442+0000 7fbfbc600700 1 -- 172.21.15.89:0/1403972197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb415cb90 con 0x7fbfb40b2070 2024-04-02T09:09:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.443+0000 7fbfb1600700 1 -- 172.21.15.89:0/1403972197 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbfa801b440 con 0x7fbfb40b2070 2024-04-02T09:09:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.444+0000 7fbfb1600700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbf9406adb0 0x7fbf9406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.444+0000 7fbfb1600700 1 -- 172.21.15.89:0/1403972197 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fbfa808b1a0 con 0x7fbfb40b2070 2024-04-02T09:09:32.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.444+0000 7fbfb9200700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbf9406adb0 0x7fbf9406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:32.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.445+0000 7fbfb9200700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbf9406adb0 0x7fbf9406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0006fd0 tx=0x7fbfa0008040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:32.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.448+0000 7fbfb1600700 1 -- 172.21.15.89:0/1403972197 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbfb415cb90 con 0x7fbfb40b2070 2024-04-02T09:09:32.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.647+0000 7fbfbc600700 1 -- 172.21.15.89:0/1403972197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fbfb415d530 con 0x7fbfb40b2070 2024-04-02T09:09:32.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.651+0000 7fbfb1600700 1 -- 172.21.15.89:0/1403972197 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v67) v1 ==== 115+0+0 (secure 0 0 0) 0x7fbfb415d530 con 0x7fbfb40b2070 2024-04-02T09:09:32.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.654+0000 7fbfbc600700 1 -- 172.21.15.89:0/1403972197 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbf9406adb0 msgr2=0x7fbf9406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:32.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.654+0000 7fbfbc600700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbf9406adb0 0x7fbf9406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0006fd0 tx=0x7fbfa0008040 comp rx=0 tx=0).stop 2024-04-02T09:09:32.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.654+0000 7fbfbc600700 1 -- 172.21.15.89:0/1403972197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb40b2070 msgr2=0x7fbfb415d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:32.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.654+0000 7fbfbc600700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb40b2070 0x7fbfb415d810 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8009130 tx=0x7fbfa8004750 comp rx=0 tx=0).stop 2024-04-02T09:09:32.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.654+0000 7fbfbc600700 1 -- 172.21.15.89:0/1403972197 shutdown_connections 2024-04-02T09:09:32.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.654+0000 7fbfbc600700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbf9406adb0 0x7fbf9406d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.654+0000 7fbfbc600700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfb40cd020 0x7fbfb415e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.654+0000 7fbfbc600700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfb40b2070 0x7fbfb415d810 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.654+0000 7fbfbc600700 1 --2- 172.21.15.89:0/1403972197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb40cbb30 0x7fbfb4149c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.654+0000 7fbfbc600700 1 -- 172.21.15.89:0/1403972197 >> 172.21.15.89:0/1403972197 conn(0x7fbfb40b0d70 msgr2=0x7fbfb4154830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:32.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.654+0000 7fbfbc600700 1 -- 172.21.15.89:0/1403972197 shutdown_connections 2024-04-02T09:09:32.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.654+0000 7fbfbc600700 1 -- 172.21.15.89:0/1403972197 wait complete. 2024-04-02T09:09:32.670 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T09:09:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.866+0000 7f8fb9600700 1 Processor -- start 2024-04-02T09:09:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.866+0000 7f8fb9600700 1 -- start start 2024-04-02T09:09:32.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.867+0000 7f8fb9600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8fb415d9d0 0x7f8fb40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:32.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.867+0000 7f8fb9600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8fb40d16a0 0x7f8fb40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.867+0000 7f8fb9600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fb40d2370 0x7f8fb415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.867+0000 7f8fb9600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fb40d2820 con 0x7f8fb415d9d0 2024-04-02T09:09:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.867+0000 7f8fb9600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fb4161420 con 0x7f8fb40d16a0 2024-04-02T09:09:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.867+0000 7f8fb9600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb4161700 con 0x7f8fb40d2370 2024-04-02T09:09:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.867+0000 7f8fb2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8fb415d9d0 0x7f8fb40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.867+0000 7f8fb2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8fb415d9d0 0x7f8fb40d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47208/0 (socket says 172.21.15.89:47208) 2024-04-02T09:09:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.867+0000 7f8fb2a00700 1 -- 172.21.15.89:0/1691134454 learned_addr learned my addr 172.21.15.89:0/1691134454 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.868+0000 7f8fb2a00700 1 -- 172.21.15.89:0/1691134454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fb40d2370 msgr2=0x7f8fb415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.868+0000 7f8fb3400700 1 --2- 172.21.15.89:0/1691134454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fb40d2370 0x7f8fb415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.868+0000 7f8fb2000700 1 --2- 172.21.15.89:0/1691134454 >> v2:172.21.15.113:3300/0 conn(0x7f8fb40d16a0 0x7f8fb40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.868+0000 7f8fb2a00700 1 --2- 172.21.15.89:0/1691134454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fb40d2370 0x7f8fb415ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.868+0000 7f8fb2a00700 1 -- 172.21.15.89:0/1691134454 >> v2:172.21.15.113:3300/0 conn(0x7f8fb40d16a0 msgr2=0x7f8fb40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.868+0000 7f8fb2a00700 1 --2- 172.21.15.89:0/1691134454 >> v2:172.21.15.113:3300/0 conn(0x7f8fb40d16a0 0x7f8fb40d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.868+0000 7f8fb2a00700 1 -- 172.21.15.89:0/1691134454 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb41619e0 con 0x7f8fb415d9d0 2024-04-02T09:09:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.868+0000 7f8fb3400700 1 --2- 172.21.15.89:0/1691134454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fb40d2370 0x7f8fb415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.868+0000 7f8fb2000700 1 --2- 172.21.15.89:0/1691134454 >> v2:172.21.15.113:3300/0 conn(0x7f8fb40d16a0 0x7f8fb40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.868+0000 7f8fb2a00700 1 --2- 172.21.15.89:0/1691134454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb415d9d0 0x7f8fb40d3c30 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40d12f0 tx=0x7f8fa4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c436f610d926c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.868+0000 7f8fb1600700 1 -- 172.21.15.89:0/1691134454 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa400b870 con 0x7f8fb415d9d0 2024-04-02T09:09:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.868+0000 7f8fb1600700 1 -- 172.21.15.89:0/1691134454 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa400b9d0 con 0x7f8fb415d9d0 2024-04-02T09:09:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.868+0000 7f8fb1600700 1 -- 172.21.15.89:0/1691134454 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa400bd60 con 0x7f8fb415d9d0 2024-04-02T09:09:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.869+0000 7f8fb9600700 1 -- 172.21.15.89:0/1691134454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb415d9d0 msgr2=0x7f8fb40d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.869+0000 7f8fb9600700 1 --2- 172.21.15.89:0/1691134454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb415d9d0 0x7f8fb40d3c30 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40d12f0 tx=0x7f8fa4009bd0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.869+0000 7f8fb9600700 1 -- 172.21.15.89:0/1691134454 shutdown_connections 2024-04-02T09:09:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.869+0000 7f8fb9600700 1 --2- 172.21.15.89:0/1691134454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb415d9d0 0x7f8fb40d3c30 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.869+0000 7f8fb9600700 1 --2- 172.21.15.89:0/1691134454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fb40d2370 0x7f8fb415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.869+0000 7f8fb9600700 1 --2- 172.21.15.89:0/1691134454 >> v2:172.21.15.113:3300/0 conn(0x7f8fb40d16a0 0x7f8fb40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.869+0000 7f8fb9600700 1 -- 172.21.15.89:0/1691134454 >> 172.21.15.89:0/1691134454 conn(0x7f8fb408aff0 msgr2=0x7f8fb40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.869+0000 7f8fb9600700 1 -- 172.21.15.89:0/1691134454 shutdown_connections 2024-04-02T09:09:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.869+0000 7f8fb9600700 1 -- 172.21.15.89:0/1691134454 wait complete. 2024-04-02T09:09:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.871+0000 7f8fb9600700 1 Processor -- start 2024-04-02T09:09:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.871+0000 7f8fb9600700 1 -- start start 2024-04-02T09:09:32.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb9600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb40d16a0 0x7f8fb41517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:32.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb9600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fb40d2370 0x7f8fb415ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:32.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb9600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fb415d9d0 0x7f8fb40b7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:32.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb9600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb40b6f00 con 0x7f8fb40d16a0 2024-04-02T09:09:32.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb9600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb40b71e0 con 0x7f8fb40d2370 2024-04-02T09:09:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb9600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb40b74c0 con 0x7f8fb415d9d0 2024-04-02T09:09:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb40d16a0 0x7f8fb41517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb40d16a0 0x7f8fb41517e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47218/0 (socket says 172.21.15.89:47218) 2024-04-02T09:09:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb2a00700 1 -- 172.21.15.89:0/2602245496 learned_addr learned my addr 172.21.15.89:0/2602245496 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb2a00700 1 -- 172.21.15.89:0/2602245496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fb415d9d0 msgr2=0x7f8fb40b7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb3400700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fb415d9d0 0x7f8fb40b7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb2000700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fb40d2370 0x7f8fb415ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb2a00700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fb415d9d0 0x7f8fb40b7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb2a00700 1 -- 172.21.15.89:0/2602245496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fb40d2370 msgr2=0x7f8fb415ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb2a00700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fb40d2370 0x7f8fb415ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb2a00700 1 -- 172.21.15.89:0/2602245496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa4009550 con 0x7f8fb40d16a0 2024-04-02T09:09:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb3400700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fb415d9d0 0x7f8fb40b7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.872+0000 7f8fb2000700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fb40d2370 0x7f8fb415ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:32.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.873+0000 7f8fb2a00700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb40d16a0 0x7f8fb41517e0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4014040 tx=0x7f8fa4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:32.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.873+0000 7f8faaa00700 1 -- 172.21.15.89:0/2602245496 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4004520 con 0x7f8fb40d16a0 2024-04-02T09:09:32.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.873+0000 7f8faaa00700 1 -- 172.21.15.89:0/2602245496 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa401e070 con 0x7f8fb40d16a0 2024-04-02T09:09:32.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.873+0000 7f8faaa00700 1 -- 172.21.15.89:0/2602245496 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa40038e0 con 0x7f8fb40d16a0 2024-04-02T09:09:32.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.873+0000 7f8fb9600700 1 -- 172.21.15.89:0/2602245496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb40b77a0 con 0x7f8fb40d16a0 2024-04-02T09:09:32.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.873+0000 7f8fb9600700 1 -- 172.21.15.89:0/2602245496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb4151d20 con 0x7f8fb40d16a0 2024-04-02T09:09:32.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.874+0000 7f8fb9600700 1 -- 172.21.15.89:0/2602245496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fb40d1ca0 con 0x7f8fb40d16a0 2024-04-02T09:09:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.875+0000 7f8faaa00700 1 -- 172.21.15.89:0/2602245496 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8fa4003a40 con 0x7f8fb40d16a0 2024-04-02T09:09:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.876+0000 7f8faaa00700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f9406adb0 0x7f8f9406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.876+0000 7f8faaa00700 1 -- 172.21.15.89:0/2602245496 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f8fa4020070 con 0x7f8fb40d16a0 2024-04-02T09:09:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.876+0000 7f8fb2000700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f9406adb0 0x7f8f9406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.877+0000 7f8fb2000700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f9406adb0 0x7f8f9406d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40b5b20 tx=0x7f8fa0008040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:32.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:32.880+0000 7f8faaa00700 1 -- 172.21.15.89:0/2602245496 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8fa4055bc0 con 0x7f8fb40d16a0 2024-04-02T09:09:33.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.097+0000 7f8fb9600700 1 -- 172.21.15.89:0/2602245496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f8fb4067720 con 0x7f8fb40d16a0 2024-04-02T09:09:33.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.101+0000 7f8faaa00700 1 -- 172.21.15.89:0/2602245496 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v68) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8fa402a0b0 con 0x7f8fb40d16a0 2024-04-02T09:09:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.103+0000 7f8fb9600700 1 -- 172.21.15.89:0/2602245496 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f9406adb0 msgr2=0x7f8f9406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.103+0000 7f8fb9600700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f9406adb0 0x7f8f9406d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40b5b20 tx=0x7f8fa0008040 comp rx=0 tx=0).stop 2024-04-02T09:09:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.103+0000 7f8fb9600700 1 -- 172.21.15.89:0/2602245496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb40d16a0 msgr2=0x7f8fb41517e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.103+0000 7f8fb9600700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb40d16a0 0x7f8fb41517e0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4014040 tx=0x7f8fa4004750 comp rx=0 tx=0).stop 2024-04-02T09:09:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.103+0000 7f8fb9600700 1 -- 172.21.15.89:0/2602245496 shutdown_connections 2024-04-02T09:09:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.103+0000 7f8fb9600700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fb415d9d0 0x7f8fb40b7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.103+0000 7f8fb9600700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb40d16a0 0x7f8fb41517e0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.103+0000 7f8fb9600700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f9406adb0 0x7f8f9406d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.103+0000 7f8fb9600700 1 --2- 172.21.15.89:0/2602245496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fb40d2370 0x7f8fb415ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.103+0000 7f8fb9600700 1 -- 172.21.15.89:0/2602245496 >> 172.21.15.89:0/2602245496 conn(0x7f8fb408aff0 msgr2=0x7f8fb40b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:33.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.103+0000 7f8fb9600700 1 -- 172.21.15.89:0/2602245496 shutdown_connections 2024-04-02T09:09:33.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.103+0000 7f8fb9600700 1 -- 172.21.15.89:0/2602245496 wait complete. 2024-04-02T09:09:33.120 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T09:09:33.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.315+0000 7fb0da000700 1 Processor -- start 2024-04-02T09:09:33.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.315+0000 7fb0da000700 1 -- start start 2024-04-02T09:09:33.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.316+0000 7fb0da000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb0d415dfe0 0x7fb0d40ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:33.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.316+0000 7fb0da000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0d40ce670 0x7fb0d40cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:33.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.316+0000 7fb0da000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d40cb310 con 0x7fb0d415a5c0 2024-04-02T09:09:33.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.316+0000 7fb0da000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d40ceb00 con 0x7fb0d415dfe0 2024-04-02T09:09:33.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.316+0000 7fb0da000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40711a0 con 0x7fb0d40ce670 2024-04-02T09:09:33.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.316+0000 7fb0d2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb0d415dfe0 0x7fb0d40ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:33.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.316+0000 7fb0d3400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb0d415a5c0 0x7fb0d40709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36354/0 (socket says 172.21.15.89:36354) 2024-04-02T09:09:33.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.316+0000 7fb0d3400700 1 -- 172.21.15.89:0/2958633536 learned_addr learned my addr 172.21.15.89:0/2958633536 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:33.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.317+0000 7fb0d3e00700 1 --2- 172.21.15.89:0/2958633536 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0d40ce670 0x7fb0d40cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:33.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.317+0000 7fb0d2a00700 1 -- 172.21.15.89:0/2958633536 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0d40ce670 msgr2=0x7fb0d40cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:33.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.317+0000 7fb0d2a00700 1 --2- 172.21.15.89:0/2958633536 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0d40ce670 0x7fb0d40cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.317+0000 7fb0d2a00700 1 -- 172.21.15.89:0/2958633536 >> v1:172.21.15.89:6789/0 conn(0x7fb0d415a5c0 legacy=0x7fb0d40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:33.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.317+0000 7fb0d2a00700 1 -- 172.21.15.89:0/2958633536 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d4071480 con 0x7fb0d415dfe0 2024-04-02T09:09:33.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.317+0000 7fb0d2a00700 1 --2- 172.21.15.89:0/2958633536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d415dfe0 0x7fb0d40ce130 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009840 tx=0x7fb0c0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58aaff8522308ea4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:33.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.317+0000 7fb0d2000700 1 -- 172.21.15.89:0/2958633536 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0011830 con 0x7fb0d415dfe0 2024-04-02T09:09:33.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.317+0000 7fb0d2000700 1 -- 172.21.15.89:0/2958633536 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c0011990 con 0x7fb0d415dfe0 2024-04-02T09:09:33.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.317+0000 7fb0d2000700 1 -- 172.21.15.89:0/2958633536 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0011d20 con 0x7fb0d415dfe0 2024-04-02T09:09:33.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.318+0000 7fb0da000700 1 -- 172.21.15.89:0/2958633536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d415dfe0 msgr2=0x7fb0d40ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:33.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.318+0000 7fb0da000700 1 --2- 172.21.15.89:0/2958633536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d415dfe0 0x7fb0d40ce130 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009840 tx=0x7fb0c0009b50 comp rx=0 tx=0).stop 2024-04-02T09:09:33.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.318+0000 7fb0da000700 1 -- 172.21.15.89:0/2958633536 shutdown_connections 2024-04-02T09:09:33.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.318+0000 7fb0da000700 1 --2- 172.21.15.89:0/2958633536 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0d40ce670 0x7fb0d40cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.318+0000 7fb0da000700 1 --2- 172.21.15.89:0/2958633536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d415dfe0 0x7fb0d40ce130 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.318+0000 7fb0da000700 1 -- 172.21.15.89:0/2958633536 >> 172.21.15.89:0/2958633536 conn(0x7fb0d40b0d70 msgr2=0x7fb0d40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:33.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.318+0000 7fb0da000700 1 -- 172.21.15.89:0/2958633536 shutdown_connections 2024-04-02T09:09:33.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.318+0000 7fb0da000700 1 -- 172.21.15.89:0/2958633536 wait complete. 2024-04-02T09:09:33.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.320+0000 7fb0da000700 1 Processor -- start 2024-04-02T09:09:33.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.320+0000 7fb0da000700 1 -- start start 2024-04-02T09:09:33.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0da000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0d40ce670 0x7fb0d414fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:33.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0da000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d415a5c0 0x7fb0d4153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:33.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0da000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0d415dfe0 0x7fb0d414f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:33.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0da000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d4150660 con 0x7fb0d415a5c0 2024-04-02T09:09:33.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0da000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40ca200 con 0x7fb0d40ce670 2024-04-02T09:09:33.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0da000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40c9c40 con 0x7fb0d415dfe0 2024-04-02T09:09:33.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0d2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d415a5c0 0x7fb0d4153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:33.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0d2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d415a5c0 0x7fb0d4153ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47238/0 (socket says 172.21.15.89:47238) 2024-04-02T09:09:33.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0d2a00700 1 -- 172.21.15.89:0/3634999414 learned_addr learned my addr 172.21.15.89:0/3634999414 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:33.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0d3e00700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0d415dfe0 0x7fb0d414f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:33.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0d3400700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0d40ce670 0x7fb0d414fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:33.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0d2a00700 1 -- 172.21.15.89:0/3634999414 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0d415dfe0 msgr2=0x7fb0d414f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:33.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0d2a00700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0d415dfe0 0x7fb0d414f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0d2a00700 1 -- 172.21.15.89:0/3634999414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0d40ce670 msgr2=0x7fb0d414fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:33.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0d2a00700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0d40ce670 0x7fb0d414fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0d2a00700 1 -- 172.21.15.89:0/3634999414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c00094a0 con 0x7fb0d415a5c0 2024-04-02T09:09:33.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.322+0000 7fb0d3400700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0d40ce670 0x7fb0d414fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:33.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.323+0000 7fb0d2a00700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d415a5c0 0x7fb0d4153ae0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009fd0 tx=0x7fb0c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:33.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.323+0000 7fb0cb400700 1 -- 172.21.15.89:0/3634999414 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c00036a0 con 0x7fb0d415a5c0 2024-04-02T09:09:33.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.323+0000 7fb0cb400700 1 -- 172.21.15.89:0/3634999414 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c001e070 con 0x7fb0d415a5c0 2024-04-02T09:09:33.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.323+0000 7fb0cb400700 1 -- 172.21.15.89:0/3634999414 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0003a60 con 0x7fb0d415a5c0 2024-04-02T09:09:33.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.323+0000 7fb0da000700 1 -- 172.21.15.89:0/3634999414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d40c9f20 con 0x7fb0d415a5c0 2024-04-02T09:09:33.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.323+0000 7fb0da000700 1 -- 172.21.15.89:0/3634999414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d40afb00 con 0x7fb0d415a5c0 2024-04-02T09:09:33.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.324+0000 7fb0da000700 1 -- 172.21.15.89:0/3634999414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d4067720 con 0x7fb0d415a5c0 2024-04-02T09:09:33.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.325+0000 7fb0cb400700 1 -- 172.21.15.89:0/3634999414 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb0c0003bc0 con 0x7fb0d415a5c0 2024-04-02T09:09:33.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.326+0000 7fb0cb400700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb0b406adb0 0x7fb0b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:33.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.326+0000 7fb0cb400700 1 -- 172.21.15.89:0/3634999414 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fb0c008a780 con 0x7fb0d415a5c0 2024-04-02T09:09:33.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.326+0000 7fb0d3400700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb0b406adb0 0x7fb0b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:33.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.327+0000 7fb0d3400700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb0b406adb0 0x7fb0b406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4005910 tx=0x7fb0c4009040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:33.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.330+0000 7fb0cb400700 1 -- 172.21.15.89:0/3634999414 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb0c00555c0 con 0x7fb0d415a5c0 2024-04-02T09:09:33.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.533+0000 7fb0da000700 1 -- 172.21.15.89:0/3634999414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb0d40b1ad0 con 0x7fb0d415a5c0 2024-04-02T09:09:33.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.538+0000 7fb0cb400700 1 -- 172.21.15.89:0/3634999414 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v69) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb0c0058be0 con 0x7fb0d415a5c0 2024-04-02T09:09:33.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.540+0000 7fb0da000700 1 -- 172.21.15.89:0/3634999414 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb0b406adb0 msgr2=0x7fb0b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:33.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.540+0000 7fb0da000700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb0b406adb0 0x7fb0b406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4005910 tx=0x7fb0c4009040 comp rx=0 tx=0).stop 2024-04-02T09:09:33.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.540+0000 7fb0da000700 1 -- 172.21.15.89:0/3634999414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d415a5c0 msgr2=0x7fb0d4153ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:33.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.540+0000 7fb0da000700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d415a5c0 0x7fb0d4153ae0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009fd0 tx=0x7fb0c0004750 comp rx=0 tx=0).stop 2024-04-02T09:09:33.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.540+0000 7fb0da000700 1 -- 172.21.15.89:0/3634999414 shutdown_connections 2024-04-02T09:09:33.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.540+0000 7fb0da000700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb0b406adb0 0x7fb0b406d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.540+0000 7fb0da000700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0d40ce670 0x7fb0d414fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.540+0000 7fb0da000700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb0d415dfe0 0x7fb0d414f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.540+0000 7fb0da000700 1 --2- 172.21.15.89:0/3634999414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d415a5c0 0x7fb0d4153ae0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.540+0000 7fb0da000700 1 -- 172.21.15.89:0/3634999414 >> 172.21.15.89:0/3634999414 conn(0x7fb0d40b0d70 msgr2=0x7fb0d40b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:33.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.540+0000 7fb0da000700 1 -- 172.21.15.89:0/3634999414 shutdown_connections 2024-04-02T09:09:33.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.540+0000 7fb0da000700 1 -- 172.21.15.89:0/3634999414 wait complete. 2024-04-02T09:09:33.556 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-04-02T09:09:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.751+0000 7fc43f800700 1 Processor -- start 2024-04-02T09:09:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.751+0000 7fc43f800700 1 -- start start 2024-04-02T09:09:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.752+0000 7fc43f800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc43815da50 0x7fc4380ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.752+0000 7fc43f800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4380d24e0 con 0x7fc4380cec70 2024-04-02T09:09:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.752+0000 7fc43f800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4380d27c0 con 0x7fc4380d0cf0 2024-04-02T09:09:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.752+0000 7fc43f800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4380b95d0 con 0x7fc43815da50 2024-04-02T09:09:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.752+0000 7fc437e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc4380cec70 0x7fc4380d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36366/0 (socket says 172.21.15.89:36366) 2024-04-02T09:09:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.752+0000 7fc437e00700 1 -- 172.21.15.89:0/2166731221 learned_addr learned my addr 172.21.15.89:0/2166731221 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.752+0000 7fc43ce00700 1 --2- 172.21.15.89:0/2166731221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc43815da50 0x7fc4380ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.753+0000 7fc437400700 1 -- 172.21.15.89:0/2166731221 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957574097 0 0) 0x7fc4380d24e0 con 0x7fc4380cec70 2024-04-02T09:09:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.753+0000 7fc437400700 1 -- 172.21.15.89:0/2166731221 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4200035b0 con 0x7fc4380cec70 2024-04-02T09:09:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.753+0000 7fc437400700 1 -- 172.21.15.89:0/2166731221 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818083513 0 0) 0x7fc4380d27c0 con 0x7fc4380d0cf0 2024-04-02T09:09:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.753+0000 7fc437400700 1 -- 172.21.15.89:0/2166731221 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4380d24e0 con 0x7fc4380d0cf0 2024-04-02T09:09:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.753+0000 7fc43ce00700 1 -- 172.21.15.89:0/2166731221 >> v1:172.21.15.113:6789/0 conn(0x7fc4380d0cf0 legacy=0x7fc4380b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.753+0000 7fc43ce00700 1 -- 172.21.15.89:0/2166731221 >> v1:172.21.15.89:6789/0 conn(0x7fc4380cec70 legacy=0x7fc4380d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.753+0000 7fc43ce00700 1 -- 172.21.15.89:0/2166731221 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4380d2aa0 con 0x7fc43815da50 2024-04-02T09:09:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.753+0000 7fc43ce00700 1 --2- 172.21.15.89:0/2166731221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc43815da50 0x7fc4380ce730 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009870 tx=0x7fc42c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72f1b802eb0a7804 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.754+0000 7fc437400700 1 -- 172.21.15.89:0/2166731221 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c011830 con 0x7fc43815da50 2024-04-02T09:09:33.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.754+0000 7fc437400700 1 -- 172.21.15.89:0/2166731221 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42c011990 con 0x7fc43815da50 2024-04-02T09:09:33.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.754+0000 7fc437400700 1 -- 172.21.15.89:0/2166731221 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c011d20 con 0x7fc43815da50 2024-04-02T09:09:33.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.754+0000 7fc43f800700 1 -- 172.21.15.89:0/2166731221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc43815da50 msgr2=0x7fc4380ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:33.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.754+0000 7fc43f800700 1 --2- 172.21.15.89:0/2166731221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc43815da50 0x7fc4380ce730 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009870 tx=0x7fc42c009b80 comp rx=0 tx=0).stop 2024-04-02T09:09:33.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.754+0000 7fc43f800700 1 -- 172.21.15.89:0/2166731221 shutdown_connections 2024-04-02T09:09:33.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.754+0000 7fc43f800700 1 --2- 172.21.15.89:0/2166731221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc43815da50 0x7fc4380ce730 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.754+0000 7fc43f800700 1 -- 172.21.15.89:0/2166731221 >> 172.21.15.89:0/2166731221 conn(0x7fc4380666a0 msgr2=0x7fc438066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:33.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.754+0000 7fc43f800700 1 -- 172.21.15.89:0/2166731221 shutdown_connections 2024-04-02T09:09:33.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.754+0000 7fc43f800700 1 -- 172.21.15.89:0/2166731221 wait complete. 2024-04-02T09:09:33.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.756+0000 7fc43f800700 1 Processor -- start 2024-04-02T09:09:33.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.756+0000 7fc43f800700 1 -- start start 2024-04-02T09:09:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.757+0000 7fc43f800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4380cec70 0x7fc4380c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.757+0000 7fc43f800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4380d0cf0 0x7fc438155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.757+0000 7fc43f800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc43815da50 0x7fc4380ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.757+0000 7fc43f800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc438155a90 con 0x7fc4380cec70 2024-04-02T09:09:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.757+0000 7fc43f800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc438158a00 con 0x7fc43815da50 2024-04-02T09:09:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.757+0000 7fc43f800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc438155ca0 con 0x7fc4380d0cf0 2024-04-02T09:09:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.757+0000 7fc43ce00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4380cec70 0x7fc4380c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.757+0000 7fc43ce00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4380cec70 0x7fc4380c9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47250/0 (socket says 172.21.15.89:47250) 2024-04-02T09:09:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.757+0000 7fc43ce00700 1 -- 172.21.15.89:0/42225787 learned_addr learned my addr 172.21.15.89:0/42225787 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.757+0000 7fc437e00700 1 --2- 172.21.15.89:0/42225787 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4380d0cf0 0x7fc438155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.757+0000 7fc43d800700 1 --2- 172.21.15.89:0/42225787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc43815da50 0x7fc4380ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.758+0000 7fc43ce00700 1 -- 172.21.15.89:0/42225787 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4380d0cf0 msgr2=0x7fc438155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.758+0000 7fc43ce00700 1 --2- 172.21.15.89:0/42225787 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4380d0cf0 0x7fc438155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.758+0000 7fc43ce00700 1 -- 172.21.15.89:0/42225787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc43815da50 msgr2=0x7fc4380ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.758+0000 7fc43ce00700 1 --2- 172.21.15.89:0/42225787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc43815da50 0x7fc4380ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.758+0000 7fc43ce00700 1 -- 172.21.15.89:0/42225787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c0094a0 con 0x7fc4380cec70 2024-04-02T09:09:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.758+0000 7fc43ce00700 1 --2- 172.21.15.89:0/42225787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4380cec70 0x7fc4380c9330 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0090b0 tx=0x7fc42c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.758+0000 7fc434c00700 1 -- 172.21.15.89:0/42225787 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c01a8f0 con 0x7fc4380cec70 2024-04-02T09:09:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.758+0000 7fc434c00700 1 -- 172.21.15.89:0/42225787 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42c01aa50 con 0x7fc4380cec70 2024-04-02T09:09:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.758+0000 7fc43f800700 1 -- 172.21.15.89:0/42225787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc438155f80 con 0x7fc4380cec70 2024-04-02T09:09:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.758+0000 7fc434c00700 1 -- 172.21.15.89:0/42225787 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c01adc0 con 0x7fc4380cec70 2024-04-02T09:09:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.758+0000 7fc43f800700 1 -- 172.21.15.89:0/42225787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc438156540 con 0x7fc4380cec70 2024-04-02T09:09:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.759+0000 7fc43f800700 1 -- 172.21.15.89:0/42225787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc438158860 con 0x7fc4380cec70 2024-04-02T09:09:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.760+0000 7fc434c00700 1 -- 172.21.15.89:0/42225787 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc42c02d450 con 0x7fc4380cec70 2024-04-02T09:09:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.761+0000 7fc434c00700 1 --2- 172.21.15.89:0/42225787 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc41806adb0 0x7fc41806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:33.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.761+0000 7fc434c00700 1 -- 172.21.15.89:0/42225787 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fc42c08aa00 con 0x7fc4380cec70 2024-04-02T09:09:33.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.761+0000 7fc437e00700 1 --2- 172.21.15.89:0/42225787 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc41806adb0 0x7fc41806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:33.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.762+0000 7fc437e00700 1 --2- 172.21.15.89:0/42225787 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc41806adb0 0x7fc41806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc4380c8ce0 tx=0x7fc424007e00 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:33.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:33.765+0000 7fc434c00700 1 -- 172.21.15.89:0/42225787 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc438158860 con 0x7fc4380cec70 2024-04-02T09:09:34.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.053+0000 7fc43f800700 1 -- 172.21.15.89:0/42225787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7fc438158860 con 0x7fc4380cec70 2024-04-02T09:09:34.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.216+0000 7fc434c00700 1 -- 172.21.15.89:0/42225787 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3026) v1 ==== 165+0+0 (secure 0 0 0) 0x7fc438158860 con 0x7fc4380cec70 2024-04-02T09:09:34.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.219+0000 7fc43f800700 1 -- 172.21.15.89:0/42225787 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc41806adb0 msgr2=0x7fc41806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:34.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.219+0000 7fc43f800700 1 --2- 172.21.15.89:0/42225787 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc41806adb0 0x7fc41806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc4380c8ce0 tx=0x7fc424007e00 comp rx=0 tx=0).stop 2024-04-02T09:09:34.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.219+0000 7fc43f800700 1 -- 172.21.15.89:0/42225787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4380cec70 msgr2=0x7fc4380c9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:34.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.219+0000 7fc43f800700 1 --2- 172.21.15.89:0/42225787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4380cec70 0x7fc4380c9330 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0090b0 tx=0x7fc42c004750 comp rx=0 tx=0).stop 2024-04-02T09:09:34.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.219+0000 7fc43f800700 1 -- 172.21.15.89:0/42225787 shutdown_connections 2024-04-02T09:09:34.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.219+0000 7fc43f800700 1 --2- 172.21.15.89:0/42225787 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc41806adb0 0x7fc41806d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:34.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.219+0000 7fc43f800700 1 --2- 172.21.15.89:0/42225787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc43815da50 0x7fc4380ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:34.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.219+0000 7fc43f800700 1 --2- 172.21.15.89:0/42225787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4380cec70 0x7fc4380c9330 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:34.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.219+0000 7fc43f800700 1 --2- 172.21.15.89:0/42225787 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4380d0cf0 0x7fc438155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:34.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.219+0000 7fc43f800700 1 -- 172.21.15.89:0/42225787 >> 172.21.15.89:0/42225787 conn(0x7fc4380666a0 msgr2=0x7fc4380b52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:34.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.219+0000 7fc43f800700 1 -- 172.21.15.89:0/42225787 shutdown_connections 2024-04-02T09:09:34.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.219+0000 7fc43f800700 1 -- 172.21.15.89:0/42225787 wait complete. 2024-04-02T09:09:34.235 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2024-04-02T09:09:34.235 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2024-04-02T09:09:34.236 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-02T09:09:34.236 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-04-02T09:09:34.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.428+0000 7f6c35200700 1 Processor -- start 2024-04-02T09:09:34.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.428+0000 7f6c35200700 1 -- start start 2024-04-02T09:09:34.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.429+0000 7f6c35200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6c3015b5d0 0x7f6c300ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:34.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.429+0000 7f6c35200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c300cb3f0 0x7f6c300ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:34.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.429+0000 7f6c35200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3015fd50 con 0x7f6c300ae930 2024-04-02T09:09:34.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.429+0000 7f6c35200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3015b9b0 con 0x7f6c3015b5d0 2024-04-02T09:09:34.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.429+0000 7f6c35200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3015f140 con 0x7f6c300cb3f0 2024-04-02T09:09:34.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.429+0000 7f6c2e800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6c300ae930 0x7f6c30140dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36380/0 (socket says 172.21.15.89:36380) 2024-04-02T09:09:34.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.429+0000 7f6c2e800700 1 -- 172.21.15.89:0/328749793 learned_addr learned my addr 172.21.15.89:0/328749793 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:34.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.429+0000 7f6c2de00700 1 --2- 172.21.15.89:0/328749793 >> v2:172.21.15.89:3300/0 conn(0x7f6c3015b5d0 0x7f6c300ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:34.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.429+0000 7f6c27e00700 1 --2- 172.21.15.89:0/328749793 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c300cb3f0 0x7f6c300ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:34.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.430+0000 7f6c2d400700 1 -- 172.21.15.89:0/328749793 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138697719 0 0) 0x7f6c3015fd50 con 0x7f6c300ae930 2024-04-02T09:09:34.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.430+0000 7f6c2d400700 1 -- 172.21.15.89:0/328749793 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c140035b0 con 0x7f6c300ae930 2024-04-02T09:09:34.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.430+0000 7f6c2de00700 1 -- 172.21.15.89:0/328749793 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c300cb3f0 msgr2=0x7f6c300ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:34.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.430+0000 7f6c2de00700 1 --2- 172.21.15.89:0/328749793 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c300cb3f0 0x7f6c300ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:34.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.430+0000 7f6c2de00700 1 -- 172.21.15.89:0/328749793 >> v1:172.21.15.89:6789/0 conn(0x7f6c300ae930 legacy=0x7f6c30140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:34.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.430+0000 7f6c2de00700 1 -- 172.21.15.89:0/328749793 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3015f420 con 0x7f6c3015b5d0 2024-04-02T09:09:34.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.430+0000 7f6c2de00700 1 --2- 172.21.15.89:0/328749793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c3015b5d0 0x7f6c300ae3f0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009840 tx=0x7f6c20009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1822f87a5f42dd52 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:34.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.430+0000 7f6c2d400700 1 -- 172.21.15.89:0/328749793 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c20011830 con 0x7f6c3015b5d0 2024-04-02T09:09:34.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.430+0000 7f6c2d400700 1 -- 172.21.15.89:0/328749793 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c20011990 con 0x7f6c3015b5d0 2024-04-02T09:09:34.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.430+0000 7f6c2d400700 1 -- 172.21.15.89:0/328749793 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c20011d00 con 0x7f6c3015b5d0 2024-04-02T09:09:34.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.431+0000 7f6c35200700 1 -- 172.21.15.89:0/328749793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c3015b5d0 msgr2=0x7f6c300ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:34.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.431+0000 7f6c35200700 1 --2- 172.21.15.89:0/328749793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c3015b5d0 0x7f6c300ae3f0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009840 tx=0x7f6c20009b50 comp rx=0 tx=0).stop 2024-04-02T09:09:34.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.431+0000 7f6c35200700 1 -- 172.21.15.89:0/328749793 shutdown_connections 2024-04-02T09:09:34.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.431+0000 7f6c35200700 1 --2- 172.21.15.89:0/328749793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c3015b5d0 0x7f6c300ae3f0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:34.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.431+0000 7f6c35200700 1 --2- 172.21.15.89:0/328749793 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c300cb3f0 0x7f6c300ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:34.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.431+0000 7f6c35200700 1 -- 172.21.15.89:0/328749793 >> 172.21.15.89:0/328749793 conn(0x7f6c300cf190 msgr2=0x7f6c300cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:34.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.431+0000 7f6c35200700 1 -- 172.21.15.89:0/328749793 shutdown_connections 2024-04-02T09:09:34.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.431+0000 7f6c35200700 1 -- 172.21.15.89:0/328749793 wait complete. 2024-04-02T09:09:34.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.433+0000 7f6c35200700 1 Processor -- start 2024-04-02T09:09:34.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.433+0000 7f6c35200700 1 -- start start 2024-04-02T09:09:34.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c35200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c300ae930 0x7f6c300cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:34.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c35200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c3015b5d0 0x7f6c30155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:34.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c35200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c3015e800 0x7f6c30157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:34.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c35200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c300cd740 con 0x7f6c3015e800 2024-04-02T09:09:34.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c35200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c300cda20 con 0x7f6c300ae930 2024-04-02T09:09:34.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c35200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c300cbd20 con 0x7f6c3015b5d0 2024-04-02T09:09:34.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c2e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c3015e800 0x7f6c30157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:34.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c2e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c3015e800 0x7f6c30157b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47254/0 (socket says 172.21.15.89:47254) 2024-04-02T09:09:34.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c2e800700 1 -- 172.21.15.89:0/3613873475 learned_addr learned my addr 172.21.15.89:0/3613873475 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:34.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c27e00700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c3015b5d0 0x7f6c30155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:34.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c2de00700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c300ae930 0x7f6c300cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:34.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c2e800700 1 -- 172.21.15.89:0/3613873475 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c3015b5d0 msgr2=0x7f6c30155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:34.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c2e800700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c3015b5d0 0x7f6c30155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:34.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c2e800700 1 -- 172.21.15.89:0/3613873475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c300ae930 msgr2=0x7f6c300cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:34.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c2e800700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c300ae930 0x7f6c300cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:34.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c2e800700 1 -- 172.21.15.89:0/3613873475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c200094a0 con 0x7f6c3015e800 2024-04-02T09:09:34.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.435+0000 7f6c2de00700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c300ae930 0x7f6c300cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:34.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.436+0000 7f6c2e800700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c3015e800 0x7f6c30157b20 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f6c2800cae0 tx=0x7f6c2800cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:34.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.436+0000 7f6c26000700 1 -- 172.21.15.89:0/3613873475 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2800aaa0 con 0x7f6c3015e800 2024-04-02T09:09:34.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.436+0000 7f6c26000700 1 -- 172.21.15.89:0/3613873475 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2800ac00 con 0x7f6c3015e800 2024-04-02T09:09:34.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.436+0000 7f6c26000700 1 -- 172.21.15.89:0/3613873475 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c28012400 con 0x7f6c3015e800 2024-04-02T09:09:34.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.436+0000 7f6c35200700 1 -- 172.21.15.89:0/3613873475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c300cc000 con 0x7f6c3015e800 2024-04-02T09:09:34.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.436+0000 7f6c35200700 1 -- 172.21.15.89:0/3613873475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c30158060 con 0x7f6c3015e800 2024-04-02T09:09:34.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.437+0000 7f6c35200700 1 -- 172.21.15.89:0/3613873475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c300cc220 con 0x7f6c3015e800 2024-04-02T09:09:34.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.438+0000 7f6c26000700 1 -- 172.21.15.89:0/3613873475 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6c2801b3b0 con 0x7f6c3015e800 2024-04-02T09:09:34.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.439+0000 7f6c26000700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6c0c06adb0 0x7f6c0c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:34.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.439+0000 7f6c26000700 1 -- 172.21.15.89:0/3613873475 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f6c280894e0 con 0x7f6c3015e800 2024-04-02T09:09:34.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.439+0000 7f6c2de00700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6c0c06adb0 0x7f6c0c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:34.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.440+0000 7f6c2de00700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6c0c06adb0 0x7f6c0c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6c20000c00 tx=0x7f6c2001c040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:34.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.443+0000 7f6c26000700 1 -- 172.21.15.89:0/3613873475 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c28054290 con 0x7f6c3015e800 2024-04-02T09:09:34.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:34.727+0000 7f6c35200700 1 -- 172.21.15.89:0/3613873475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f6c300cbb20 con 0x7f6c3015e800 2024-04-02T09:09:35.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.221+0000 7f6c26000700 1 -- 172.21.15.89:0/3613873475 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3027) v1 ==== 166+0+0 (secure 0 0 0) 0x7f6c280578b0 con 0x7f6c3015e800 2024-04-02T09:09:35.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.224+0000 7f6c35200700 1 -- 172.21.15.89:0/3613873475 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6c0c06adb0 msgr2=0x7f6c0c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:35.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.224+0000 7f6c35200700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6c0c06adb0 0x7f6c0c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6c20000c00 tx=0x7f6c2001c040 comp rx=0 tx=0).stop 2024-04-02T09:09:35.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.224+0000 7f6c35200700 1 -- 172.21.15.89:0/3613873475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c3015e800 msgr2=0x7f6c30157b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:35.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.224+0000 7f6c35200700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c3015e800 0x7f6c30157b20 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f6c2800cae0 tx=0x7f6c2800cdf0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.224+0000 7f6c35200700 1 -- 172.21.15.89:0/3613873475 shutdown_connections 2024-04-02T09:09:35.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.224+0000 7f6c35200700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6c0c06adb0 0x7f6c0c06d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.224+0000 7f6c35200700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c300ae930 0x7f6c300cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.224+0000 7f6c35200700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c3015e800 0x7f6c30157b20 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.224+0000 7f6c35200700 1 --2- 172.21.15.89:0/3613873475 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c3015b5d0 0x7f6c30155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.224+0000 7f6c35200700 1 -- 172.21.15.89:0/3613873475 >> 172.21.15.89:0/3613873475 conn(0x7f6c300cf190 msgr2=0x7f6c300cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:35.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.224+0000 7f6c35200700 1 -- 172.21.15.89:0/3613873475 shutdown_connections 2024-04-02T09:09:35.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.224+0000 7f6c35200700 1 -- 172.21.15.89:0/3613873475 wait complete. 2024-04-02T09:09:35.241 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T09:09:35.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.438+0000 7f105ca00700 1 Processor -- start 2024-04-02T09:09:35.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.438+0000 7f105ca00700 1 -- start start 2024-04-02T09:09:35.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.439+0000 7f105ca00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10580d16a0 0x7f10580d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:35.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.440+0000 7f105ca00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f10580d2370 0x7f10580ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:35.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.440+0000 7f105ca00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1058161420 con 0x7f105815d9d0 2024-04-02T09:09:35.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.440+0000 7f105ca00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10580d4340 con 0x7f10580d16a0 2024-04-02T09:09:35.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.440+0000 7f105ca00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f105815edc0 con 0x7f10580d2370 2024-04-02T09:09:35.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.440+0000 7f104fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10580d16a0 0x7f10580d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:35.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.440+0000 7f104fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10580d16a0 0x7f10580d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47260/0 (socket says 172.21.15.89:47260) 2024-04-02T09:09:35.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.440+0000 7f104fe00700 1 -- 172.21.15.89:0/3111207153 learned_addr learned my addr 172.21.15.89:0/3111207153 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:35.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.440+0000 7f1056800700 1 --2- 172.21.15.89:0/3111207153 >> v2:172.21.15.113:3300/0 conn(0x7f10580d2370 0x7f10580ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:35.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.440+0000 7f104fe00700 1 -- 172.21.15.89:0/3111207153 >> v2:172.21.15.113:3300/0 conn(0x7f10580d2370 msgr2=0x7f10580ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:35.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.440+0000 7f104fe00700 1 --2- 172.21.15.89:0/3111207153 >> v2:172.21.15.113:3300/0 conn(0x7f10580d2370 0x7f10580ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.440+0000 7f104fe00700 1 -- 172.21.15.89:0/3111207153 >> v1:172.21.15.113:6789/0 conn(0x7f105815d9d0 legacy=0x7f10580d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:35.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.440+0000 7f104fe00700 1 -- 172.21.15.89:0/3111207153 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1058161700 con 0x7f10580d16a0 2024-04-02T09:09:35.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.441+0000 7f104fe00700 1 --2- 172.21.15.89:0/3111207153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10580d16a0 0x7f10580d1e30 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f105808ca10 tx=0x7f1040009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255e0a616c7f8e50 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:35.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.441+0000 7f1055400700 1 -- 172.21.15.89:0/3111207153 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1040005e30 con 0x7f10580d16a0 2024-04-02T09:09:35.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.441+0000 7f1055400700 1 -- 172.21.15.89:0/3111207153 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1040006030 con 0x7f10580d16a0 2024-04-02T09:09:35.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.441+0000 7f1055400700 1 -- 172.21.15.89:0/3111207153 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10400063e0 con 0x7f10580d16a0 2024-04-02T09:09:35.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.441+0000 7f105ca00700 1 -- 172.21.15.89:0/3111207153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10580d16a0 msgr2=0x7f10580d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:35.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.441+0000 7f105ca00700 1 --2- 172.21.15.89:0/3111207153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10580d16a0 0x7f10580d1e30 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f105808ca10 tx=0x7f1040009bd0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.441+0000 7f105ca00700 1 -- 172.21.15.89:0/3111207153 shutdown_connections 2024-04-02T09:09:35.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.441+0000 7f105ca00700 1 --2- 172.21.15.89:0/3111207153 >> v2:172.21.15.113:3300/0 conn(0x7f10580d2370 0x7f10580ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.441+0000 7f105ca00700 1 --2- 172.21.15.89:0/3111207153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10580d16a0 0x7f10580d1e30 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.441+0000 7f105ca00700 1 -- 172.21.15.89:0/3111207153 >> 172.21.15.89:0/3111207153 conn(0x7f105808aff0 msgr2=0x7f10580d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:35.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.441+0000 7f105ca00700 1 -- 172.21.15.89:0/3111207153 shutdown_connections 2024-04-02T09:09:35.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.441+0000 7f105ca00700 1 -- 172.21.15.89:0/3111207153 wait complete. 2024-04-02T09:09:35.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.443+0000 7f105ca00700 1 Processor -- start 2024-04-02T09:09:35.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.443+0000 7f105ca00700 1 -- start start 2024-04-02T09:09:35.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.444+0000 7f105ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10580d16a0 0x7f105815c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:35.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.444+0000 7f105ca00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10580d2370 0x7f105815bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:35.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.444+0000 7f105ca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f105815d9d0 0x7f10580b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:35.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.444+0000 7f105ca00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10581e8850 con 0x7f10580d16a0 2024-04-02T09:09:35.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.444+0000 7f105ca00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10580b7580 con 0x7f10580d2370 2024-04-02T09:09:35.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.444+0000 7f105ca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10581e8570 con 0x7f105815d9d0 2024-04-02T09:09:35.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.444+0000 7f1055e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10580d16a0 0x7f105815c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:35.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.445+0000 7f1055e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10580d16a0 0x7f105815c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47272/0 (socket says 172.21.15.89:47272) 2024-04-02T09:09:35.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.445+0000 7f1055e00700 1 -- 172.21.15.89:0/1507541728 learned_addr learned my addr 172.21.15.89:0/1507541728 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:35.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.445+0000 7f1056800700 1 --2- 172.21.15.89:0/1507541728 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f105815d9d0 0x7f10580b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:35.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.445+0000 7f104fe00700 1 --2- 172.21.15.89:0/1507541728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10580d2370 0x7f105815bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:35.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.445+0000 7f1055e00700 1 -- 172.21.15.89:0/1507541728 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f105815d9d0 msgr2=0x7f10580b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:35.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.445+0000 7f1055e00700 1 --2- 172.21.15.89:0/1507541728 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f105815d9d0 0x7f10580b89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.445+0000 7f1055e00700 1 -- 172.21.15.89:0/1507541728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10580d2370 msgr2=0x7f105815bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:35.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.445+0000 7f1055e00700 1 --2- 172.21.15.89:0/1507541728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10580d2370 0x7f105815bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.445+0000 7f1055e00700 1 -- 172.21.15.89:0/1507541728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1040009500 con 0x7f10580d16a0 2024-04-02T09:09:35.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.445+0000 7f1055e00700 1 --2- 172.21.15.89:0/1507541728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10580d16a0 0x7f105815c560 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f104800b4a0 tx=0x7f104800b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:35.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.445+0000 7f104e000700 1 -- 172.21.15.89:0/1507541728 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048011830 con 0x7f10580d16a0 2024-04-02T09:09:35.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.445+0000 7f104e000700 1 -- 172.21.15.89:0/1507541728 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1048011990 con 0x7f10580d16a0 2024-04-02T09:09:35.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.445+0000 7f105ca00700 1 -- 172.21.15.89:0/1507541728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10580b7210 con 0x7f10580d16a0 2024-04-02T09:09:35.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.445+0000 7f104e000700 1 -- 172.21.15.89:0/1507541728 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048011d40 con 0x7f10580d16a0 2024-04-02T09:09:35.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.446+0000 7f105ca00700 1 -- 172.21.15.89:0/1507541728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10580cac40 con 0x7f10580d16a0 2024-04-02T09:09:35.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.447+0000 7f105ca00700 1 -- 172.21.15.89:0/1507541728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10580d1ca0 con 0x7f10580d16a0 2024-04-02T09:09:35.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.448+0000 7f104e000700 1 -- 172.21.15.89:0/1507541728 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f104801a400 con 0x7f10580d16a0 2024-04-02T09:09:35.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.448+0000 7f104e000700 1 --2- 172.21.15.89:0/1507541728 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f103406adb0 0x7f103406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:35.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.448+0000 7f104e000700 1 -- 172.21.15.89:0/1507541728 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f1048089250 con 0x7f10580d16a0 2024-04-02T09:09:35.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.449+0000 7f104fe00700 1 --2- 172.21.15.89:0/1507541728 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f103406adb0 0x7f103406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:35.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.449+0000 7f104fe00700 1 --2- 172.21.15.89:0/1507541728 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f103406adb0 0x7f103406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f105808ca10 tx=0x7f1040008070 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:35.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.453+0000 7f104e000700 1 -- 172.21.15.89:0/1507541728 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1048054130 con 0x7f10580d16a0 2024-04-02T09:09:35.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.656+0000 7f105ca00700 1 -- 172.21.15.89:0/1507541728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f10580b7070 con 0x7f10580d16a0 2024-04-02T09:09:35.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.660+0000 7f104e000700 1 -- 172.21.15.89:0/1507541728 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7f10580b7070 con 0x7f10580d16a0 2024-04-02T09:09:35.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.663+0000 7f105ca00700 1 -- 172.21.15.89:0/1507541728 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f103406adb0 msgr2=0x7f103406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:35.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.663+0000 7f105ca00700 1 --2- 172.21.15.89:0/1507541728 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f103406adb0 0x7f103406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f105808ca10 tx=0x7f1040008070 comp rx=0 tx=0).stop 2024-04-02T09:09:35.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.663+0000 7f105ca00700 1 -- 172.21.15.89:0/1507541728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10580d16a0 msgr2=0x7f105815c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:35.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.663+0000 7f105ca00700 1 --2- 172.21.15.89:0/1507541728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10580d16a0 0x7f105815c560 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f104800b4a0 tx=0x7f104800b7b0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.663+0000 7f105ca00700 1 -- 172.21.15.89:0/1507541728 shutdown_connections 2024-04-02T09:09:35.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.663+0000 7f105ca00700 1 --2- 172.21.15.89:0/1507541728 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f105815d9d0 0x7f10580b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.663+0000 7f105ca00700 1 --2- 172.21.15.89:0/1507541728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10580d16a0 0x7f105815c560 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.663+0000 7f105ca00700 1 --2- 172.21.15.89:0/1507541728 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f103406adb0 0x7f103406d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.663+0000 7f105ca00700 1 --2- 172.21.15.89:0/1507541728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10580d2370 0x7f105815bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:35.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.663+0000 7f105ca00700 1 -- 172.21.15.89:0/1507541728 >> 172.21.15.89:0/1507541728 conn(0x7f105808aff0 msgr2=0x7f10580b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:35.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.663+0000 7f105ca00700 1 -- 172.21.15.89:0/1507541728 shutdown_connections 2024-04-02T09:09:35.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:35.663+0000 7f105ca00700 1 -- 172.21.15.89:0/1507541728 wait complete. 2024-04-02T09:09:35.680 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T09:09:35.681 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:09:35.775 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:09:35.872 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T09:09:35.872 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:09:35.872 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:09:35.932 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:09:35.932 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:09:35.932 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:09:35.981 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:09:35.981 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:09:35.981 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:09:36.030 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:09:36.031 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:09:36.031 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:09:36.092 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:09:36.092 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T09:09:36.093 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:09:36.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.285+0000 7f3929c00700 1 Processor -- start 2024-04-02T09:09:36.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.285+0000 7f3929c00700 1 -- start start 2024-04-02T09:09:36.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.286+0000 7f3929c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f392415d710 0x7f39240d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:36.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.286+0000 7f3929c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f392409aba0 0x7f39240d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:36.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.286+0000 7f3929c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f392406bb40 con 0x7f392406bf40 2024-04-02T09:09:36.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.286+0000 7f3929c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3924099b00 con 0x7f392415d710 2024-04-02T09:09:36.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.286+0000 7f3929c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3924099de0 con 0x7f392409aba0 2024-04-02T09:09:36.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.286+0000 7f3922a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f392415d710 0x7f39240d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:36.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.286+0000 7f3922a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f392415d710 0x7f39240d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47278/0 (socket says 172.21.15.89:47278) 2024-04-02T09:09:36.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.286+0000 7f3922a00700 1 -- 172.21.15.89:0/980444359 learned_addr learned my addr 172.21.15.89:0/980444359 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:36.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.287+0000 7f3922000700 1 --2- 172.21.15.89:0/980444359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f392409aba0 0x7f39240d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:36.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.287+0000 7f3922a00700 1 -- 172.21.15.89:0/980444359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f392409aba0 msgr2=0x7f39240d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:36.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.287+0000 7f3922a00700 1 --2- 172.21.15.89:0/980444359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f392409aba0 0x7f39240d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:36.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.287+0000 7f3922a00700 1 -- 172.21.15.89:0/980444359 >> v1:172.21.15.113:6789/0 conn(0x7f392406bf40 legacy=0x7f39240ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:36.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.287+0000 7f3922a00700 1 -- 172.21.15.89:0/980444359 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392409a0c0 con 0x7f392415d710 2024-04-02T09:09:36.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.287+0000 7f3921600700 1 -- 172.21.15.89:0/980444359 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950298963 0 0) 0x7f392406bb40 con 0x7f392406bf40 2024-04-02T09:09:36.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.287+0000 7f3922a00700 1 --2- 172.21.15.89:0/980444359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f392415d710 0x7f39240d0130 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f39140098f0 tx=0x7f3914009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64442e9edf924468 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:36.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.287+0000 7f3921600700 1 -- 172.21.15.89:0/980444359 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914011830 con 0x7f392415d710 2024-04-02T09:09:36.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.287+0000 7f3921600700 1 -- 172.21.15.89:0/980444359 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3914011990 con 0x7f392415d710 2024-04-02T09:09:36.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.287+0000 7f3921600700 1 -- 172.21.15.89:0/980444359 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914011d20 con 0x7f392415d710 2024-04-02T09:09:36.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.288+0000 7f3929c00700 1 -- 172.21.15.89:0/980444359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f392415d710 msgr2=0x7f39240d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:36.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.288+0000 7f3929c00700 1 --2- 172.21.15.89:0/980444359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f392415d710 0x7f39240d0130 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f39140098f0 tx=0x7f3914009c00 comp rx=0 tx=0).stop 2024-04-02T09:09:36.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.288+0000 7f3929c00700 1 -- 172.21.15.89:0/980444359 shutdown_connections 2024-04-02T09:09:36.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.288+0000 7f3929c00700 1 --2- 172.21.15.89:0/980444359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f392415d710 0x7f39240d0130 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:36.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.288+0000 7f3929c00700 1 --2- 172.21.15.89:0/980444359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f392409aba0 0x7f39240d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:36.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.288+0000 7f3929c00700 1 -- 172.21.15.89:0/980444359 >> 172.21.15.89:0/980444359 conn(0x7f39240d13e0 msgr2=0x7f39240d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:36.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.288+0000 7f3929c00700 1 -- 172.21.15.89:0/980444359 shutdown_connections 2024-04-02T09:09:36.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.288+0000 7f3929c00700 1 -- 172.21.15.89:0/980444359 wait complete. 2024-04-02T09:09:36.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.290+0000 7f3929c00700 1 Processor -- start 2024-04-02T09:09:36.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.290+0000 7f3929c00700 1 -- start start 2024-04-02T09:09:36.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3929c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f392406bf40 0x7f392414c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:36.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3929c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f392409aba0 0x7f39241f2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:36.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3929c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f392415d710 0x7f39241f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:36.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3929c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39240cb230 con 0x7f392409aba0 2024-04-02T09:09:36.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3929c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39240c84b0 con 0x7f392415d710 2024-04-02T09:09:36.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3929c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39240c8790 con 0x7f392406bf40 2024-04-02T09:09:36.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3922000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f392409aba0 0x7f39241f2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:36.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3922000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f392409aba0 0x7f39241f2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47292/0 (socket says 172.21.15.89:47292) 2024-04-02T09:09:36.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3922000700 1 -- 172.21.15.89:0/2620884843 learned_addr learned my addr 172.21.15.89:0/2620884843 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:36.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3922a00700 1 --2- 172.21.15.89:0/2620884843 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f392406bf40 0x7f392414c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:36.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3923400700 1 --2- 172.21.15.89:0/2620884843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f392415d710 0x7f39241f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:36.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3922000700 1 -- 172.21.15.89:0/2620884843 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f392406bf40 msgr2=0x7f392414c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:36.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3922000700 1 --2- 172.21.15.89:0/2620884843 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f392406bf40 0x7f392414c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:36.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3922000700 1 -- 172.21.15.89:0/2620884843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f392415d710 msgr2=0x7f39241f2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:36.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3922000700 1 --2- 172.21.15.89:0/2620884843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f392415d710 0x7f39241f2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:36.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.291+0000 7f3922000700 1 -- 172.21.15.89:0/2620884843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3914009550 con 0x7f392409aba0 2024-04-02T09:09:36.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.292+0000 7f3922000700 1 --2- 172.21.15.89:0/2620884843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f392409aba0 0x7f39241f2120 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f391000e980 tx=0x7f391000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:36.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.292+0000 7f391aa00700 1 -- 172.21.15.89:0/2620884843 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3910009710 con 0x7f392409aba0 2024-04-02T09:09:36.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.292+0000 7f391aa00700 1 -- 172.21.15.89:0/2620884843 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391000c690 con 0x7f392409aba0 2024-04-02T09:09:36.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.292+0000 7f3929c00700 1 -- 172.21.15.89:0/2620884843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39240cabe0 con 0x7f392409aba0 2024-04-02T09:09:36.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.292+0000 7f391aa00700 1 -- 172.21.15.89:0/2620884843 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391000ca40 con 0x7f392409aba0 2024-04-02T09:09:36.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.292+0000 7f3929c00700 1 -- 172.21.15.89:0/2620884843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39240c8a70 con 0x7f392409aba0 2024-04-02T09:09:36.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.293+0000 7f3929c00700 1 -- 172.21.15.89:0/2620884843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3924067720 con 0x7f392409aba0 2024-04-02T09:09:36.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.294+0000 7f391aa00700 1 -- 172.21.15.89:0/2620884843 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f391000cba0 con 0x7f392409aba0 2024-04-02T09:09:36.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.295+0000 7f391aa00700 1 --2- 172.21.15.89:0/2620884843 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f390406adb0 0x7f390406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:36.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.295+0000 7f391aa00700 1 -- 172.21.15.89:0/2620884843 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f39100896f0 con 0x7f392409aba0 2024-04-02T09:09:36.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.296+0000 7f3922a00700 1 --2- 172.21.15.89:0/2620884843 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f390406adb0 0x7f390406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:36.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.296+0000 7f3922a00700 1 --2- 172.21.15.89:0/2620884843 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f390406adb0 0x7f390406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3914014040 tx=0x7f3914008030 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:36.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.300+0000 7f391aa00700 1 -- 172.21.15.89:0/2620884843 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3910054530 con 0x7f392409aba0 2024-04-02T09:09:36.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.542+0000 7f3929c00700 1 -- 172.21.15.89:0/2620884843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39240d0f20 con 0x7f392409aba0 2024-04-02T09:09:36.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.543+0000 7f391aa00700 1 -- 172.21.15.89:0/2620884843 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+3537 (secure 0 0 0) 0x7f3910017df0 con 0x7f392409aba0 2024-04-02T09:09:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:36.542 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":91,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1296569976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1296569976},{"type":"v1","addr":"172.21.15.113:6835","nonce":1296569976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9331,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1328950634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1328950634},{"type":"v1","addr":"172.21.15.89:6837","nonce":1328950634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":10032,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1167691520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1167691520},{"type":"v1","addr":"172.21.15.113:6834","nonce":1167691520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":10539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3670166077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3670166077},{"type":"v1","addr":"172.21.15.89:6836","nonce":3670166077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91}],"filesystems":[]} 2024-04-02T09:09:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.545+0000 7f3929c00700 1 -- 172.21.15.89:0/2620884843 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f390406adb0 msgr2=0x7f390406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.545+0000 7f3929c00700 1 --2- 172.21.15.89:0/2620884843 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f390406adb0 0x7f390406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3914014040 tx=0x7f3914008030 comp rx=0 tx=0).stop 2024-04-02T09:09:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.545+0000 7f3929c00700 1 -- 172.21.15.89:0/2620884843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f392409aba0 msgr2=0x7f39241f2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.545+0000 7f3929c00700 1 --2- 172.21.15.89:0/2620884843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f392409aba0 0x7f39241f2120 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f391000e980 tx=0x7f391000ec90 comp rx=0 tx=0).stop 2024-04-02T09:09:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.545+0000 7f3929c00700 1 -- 172.21.15.89:0/2620884843 shutdown_connections 2024-04-02T09:09:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.545+0000 7f3929c00700 1 --2- 172.21.15.89:0/2620884843 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f390406adb0 0x7f390406d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.545+0000 7f3929c00700 1 --2- 172.21.15.89:0/2620884843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f392415d710 0x7f39241f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.545+0000 7f3929c00700 1 --2- 172.21.15.89:0/2620884843 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f392406bf40 0x7f392414c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.545+0000 7f3929c00700 1 --2- 172.21.15.89:0/2620884843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f392409aba0 0x7f39241f2120 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.546+0000 7f3929c00700 1 -- 172.21.15.89:0/2620884843 >> 172.21.15.89:0/2620884843 conn(0x7f39240d13e0 msgr2=0x7f39240b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.546+0000 7f3929c00700 1 -- 172.21.15.89:0/2620884843 shutdown_connections 2024-04-02T09:09:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.546+0000 7f3929c00700 1 -- 172.21.15.89:0/2620884843 wait complete. 2024-04-02T09:09:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 91 2024-04-02T09:09:36.565 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T09:09:36.565 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T09:09:36.566 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T09:09:36.566 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:09:36.566 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T09:09:36.566 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T09:09:36.566 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T09:09:36.566 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:09:36.567 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T09:09:36.567 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T09:09:36.567 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T09:09:36.567 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:09:36.568 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T09:09:36.568 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T09:09:36.568 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T09:09:36.568 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:09:36.581 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T09:09:36.581 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T09:09:36.582 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T09:09:36.583 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T09:09:36.584 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T09:09:36.585 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T09:09:36.585 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T09:09:36.586 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T09:09:36.628 INFO:tasks.ceph.mds.c:Started 2024-04-02T09:09:36.628 DEBUG:teuthology.parallel:result is None 2024-04-02T09:09:36.629 INFO:tasks.ceph.mds.d:Started 2024-04-02T09:09:36.629 DEBUG:teuthology.parallel:result is None 2024-04-02T09:09:36.631 INFO:tasks.ceph.mds.a:Started 2024-04-02T09:09:36.632 DEBUG:teuthology.parallel:result is None 2024-04-02T09:09:36.633 INFO:tasks.ceph.mds.b:Started 2024-04-02T09:09:36.633 DEBUG:teuthology.parallel:result is None 2024-04-02T09:09:36.633 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T09:09:36.773 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T09:09:36.773 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T09:09:36.774 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T09:09:36.774 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T09:09:36.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.844+0000 7f47ea200700 1 Processor -- start 2024-04-02T09:09:36.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.845+0000 7f47ea200700 1 -- start start 2024-04-02T09:09:36.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.846+0000 7f47ea200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f47e41611b0 0x7f47e415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:36.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.846+0000 7f47ea200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47e415f4d0 0x7f47e40ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:36.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.846+0000 7f47ea200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e40ce270 con 0x7f47e40d21f0 2024-04-02T09:09:36.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.846+0000 7f47ea200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47e415b600 con 0x7f47e41611b0 2024-04-02T09:09:36.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.846+0000 7f47ea200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e40d2760 con 0x7f47e415f4d0 2024-04-02T09:09:36.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.846+0000 7f47e3e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f47e40d21f0 0x7f47e4161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36408/0 (socket says 172.21.15.89:36408) 2024-04-02T09:09:36.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.846+0000 7f47e3e00700 1 -- 172.21.15.89:0/425332599 learned_addr learned my addr 172.21.15.89:0/425332599 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:36.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.846+0000 7f47e3400700 1 --2- 172.21.15.89:0/425332599 >> v2:172.21.15.113:3300/0 conn(0x7f47e41611b0 0x7f47e415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:36.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.846+0000 7f47dbe00700 1 --2- 172.21.15.89:0/425332599 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47e415f4d0 0x7f47e40ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:36.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.847+0000 7f47e2a00700 1 -- 172.21.15.89:0/425332599 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547057547 0 0) 0x7f47e40ce270 con 0x7f47e40d21f0 2024-04-02T09:09:36.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.847+0000 7f47e2a00700 1 -- 172.21.15.89:0/425332599 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c80035b0 con 0x7f47e40d21f0 2024-04-02T09:09:36.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.847+0000 7f47dbe00700 1 -- 172.21.15.89:0/425332599 >> v2:172.21.15.113:3300/0 conn(0x7f47e41611b0 msgr2=0x7f47e415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:36.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.847+0000 7f47dbe00700 1 --2- 172.21.15.89:0/425332599 >> v2:172.21.15.113:3300/0 conn(0x7f47e41611b0 0x7f47e415ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:36.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.847+0000 7f47dbe00700 1 -- 172.21.15.89:0/425332599 >> v1:172.21.15.89:6789/0 conn(0x7f47e40d21f0 legacy=0x7f47e4161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:36.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.847+0000 7f47dbe00700 1 -- 172.21.15.89:0/425332599 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e40d2a40 con 0x7f47e415f4d0 2024-04-02T09:09:36.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.847+0000 7f47e3400700 1 --2- 172.21.15.89:0/425332599 >> v2:172.21.15.113:3300/0 conn(0x7f47e41611b0 0x7f47e415ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:09:36.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.847+0000 7f47dbe00700 1 --2- 172.21.15.89:0/425332599 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47e415f4d0 0x7f47e40ce700 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009840 tx=0x7f47cc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a15a722becaac88 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:36.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.847+0000 7f47e2a00700 1 -- 172.21.15.89:0/425332599 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc011830 con 0x7f47e415f4d0 2024-04-02T09:09:36.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.847+0000 7f47e2a00700 1 -- 172.21.15.89:0/425332599 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47cc011990 con 0x7f47e415f4d0 2024-04-02T09:09:36.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.847+0000 7f47e2a00700 1 -- 172.21.15.89:0/425332599 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc011d20 con 0x7f47e415f4d0 2024-04-02T09:09:36.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.848+0000 7f47ea200700 1 -- 172.21.15.89:0/425332599 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47e415f4d0 msgr2=0x7f47e40ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:36.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.848+0000 7f47ea200700 1 --2- 172.21.15.89:0/425332599 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47e415f4d0 0x7f47e40ce700 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009840 tx=0x7f47cc009b50 comp rx=0 tx=0).stop 2024-04-02T09:09:36.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.848+0000 7f47ea200700 1 -- 172.21.15.89:0/425332599 shutdown_connections 2024-04-02T09:09:36.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.848+0000 7f47ea200700 1 --2- 172.21.15.89:0/425332599 >> v2:172.21.15.113:3300/0 conn(0x7f47e41611b0 0x7f47e415ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:36.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.848+0000 7f47ea200700 1 --2- 172.21.15.89:0/425332599 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47e415f4d0 0x7f47e40ce700 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:36.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.848+0000 7f47ea200700 1 -- 172.21.15.89:0/425332599 >> 172.21.15.89:0/425332599 conn(0x7f47e408aff0 msgr2=0x7f47e40d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:36.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.848+0000 7f47ea200700 1 -- 172.21.15.89:0/425332599 shutdown_connections 2024-04-02T09:09:36.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.848+0000 7f47ea200700 1 -- 172.21.15.89:0/425332599 wait complete. 2024-04-02T09:09:36.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.850+0000 7f47ea200700 1 Processor -- start 2024-04-02T09:09:36.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.850+0000 7f47ea200700 1 -- start start 2024-04-02T09:09:36.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47ea200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47e40d21f0 0x7f47e41f6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:36.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47ea200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e415f4d0 0x7f47e41530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:36.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47ea200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47e41611b0 0x7f47e4158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:36.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47ea200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e41f2fe0 con 0x7f47e415f4d0 2024-04-02T09:09:36.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47ea200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e41f6f40 con 0x7f47e41611b0 2024-04-02T09:09:36.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47ea200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e41f7220 con 0x7f47e40d21f0 2024-04-02T09:09:36.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47dbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e415f4d0 0x7f47e41530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:36.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47dbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e415f4d0 0x7f47e41530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47328/0 (socket says 172.21.15.89:47328) 2024-04-02T09:09:36.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47dbe00700 1 -- 172.21.15.89:0/3842655055 learned_addr learned my addr 172.21.15.89:0/3842655055 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:36.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47e3e00700 1 --2- 172.21.15.89:0/3842655055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47e41611b0 0x7f47e4158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:36.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47e3400700 1 --2- 172.21.15.89:0/3842655055 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47e40d21f0 0x7f47e41f6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:36.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47dbe00700 1 -- 172.21.15.89:0/3842655055 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47e40d21f0 msgr2=0x7f47e41f6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:36.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47dbe00700 1 --2- 172.21.15.89:0/3842655055 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47e40d21f0 0x7f47e41f6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:36.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47dbe00700 1 -- 172.21.15.89:0/3842655055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47e41611b0 msgr2=0x7f47e4158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:36.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47dbe00700 1 --2- 172.21.15.89:0/3842655055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47e41611b0 0x7f47e4158630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:36.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.851+0000 7f47dbe00700 1 -- 172.21.15.89:0/3842655055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d40094a0 con 0x7f47e415f4d0 2024-04-02T09:09:36.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.852+0000 7f47dbe00700 1 --2- 172.21.15.89:0/3842655055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e415f4d0 0x7f47e41530e0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0090b0 tx=0x7f47cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:36.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.852+0000 7f47db400700 1 -- 172.21.15.89:0/3842655055 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc0036a0 con 0x7f47e415f4d0 2024-04-02T09:09:36.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.852+0000 7f47db400700 1 -- 172.21.15.89:0/3842655055 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47cc01f070 con 0x7f47e415f4d0 2024-04-02T09:09:36.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.852+0000 7f47ea200700 1 -- 172.21.15.89:0/3842655055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47cc0094a0 con 0x7f47e415f4d0 2024-04-02T09:09:36.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.852+0000 7f47db400700 1 -- 172.21.15.89:0/3842655055 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc003a40 con 0x7f47e415f4d0 2024-04-02T09:09:36.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.852+0000 7f47ea200700 1 -- 172.21.15.89:0/3842655055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e40b7320 con 0x7f47e415f4d0 2024-04-02T09:09:36.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.853+0000 7f47ea200700 1 -- 172.21.15.89:0/3842655055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e4067720 con 0x7f47e415f4d0 2024-04-02T09:09:36.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.854+0000 7f47db400700 1 -- 172.21.15.89:0/3842655055 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f47cc003ba0 con 0x7f47e415f4d0 2024-04-02T09:09:36.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.855+0000 7f47db400700 1 --2- 172.21.15.89:0/3842655055 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f47c006adb0 0x7f47c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:36.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.855+0000 7f47db400700 1 -- 172.21.15.89:0/3842655055 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f47cc021070 con 0x7f47e415f4d0 2024-04-02T09:09:36.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.855+0000 7f47e3400700 1 --2- 172.21.15.89:0/3842655055 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f47c006adb0 0x7f47c006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:36.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.856+0000 7f47e3400700 1 --2- 172.21.15.89:0/3842655055 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f47c006adb0 0x7f47c006d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009cd0 tx=0x7f47d4009210 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:36.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:36.860+0000 7f47db400700 1 -- 172.21.15.89:0/3842655055 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f47cc055d00 con 0x7f47e415f4d0 2024-04-02T09:09:37.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:37.077+0000 7f47ea200700 1 -- 172.21.15.89:0/3842655055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f47e40ce990 con 0x7f47e415f4d0 2024-04-02T09:09:37.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:37.812+0000 7f47db400700 1 -- 172.21.15.89:0/3842655055 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v201) v1 ==== 97+0+0 (secure 0 0 0) 0x7f47cc059320 con 0x7f47e415f4d0 2024-04-02T09:09:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:37.815+0000 7f47ea200700 1 -- 172.21.15.89:0/3842655055 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f47c006adb0 msgr2=0x7f47c006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:37.815+0000 7f47ea200700 1 --2- 172.21.15.89:0/3842655055 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f47c006adb0 0x7f47c006d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009cd0 tx=0x7f47d4009210 comp rx=0 tx=0).stop 2024-04-02T09:09:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:37.815+0000 7f47ea200700 1 -- 172.21.15.89:0/3842655055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e415f4d0 msgr2=0x7f47e41530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:37.815+0000 7f47ea200700 1 --2- 172.21.15.89:0/3842655055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e415f4d0 0x7f47e41530e0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0090b0 tx=0x7f47cc004750 comp rx=0 tx=0).stop 2024-04-02T09:09:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:37.815+0000 7f47ea200700 1 -- 172.21.15.89:0/3842655055 shutdown_connections 2024-04-02T09:09:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:37.815+0000 7f47ea200700 1 --2- 172.21.15.89:0/3842655055 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f47c006adb0 0x7f47c006d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:37.815+0000 7f47ea200700 1 --2- 172.21.15.89:0/3842655055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47e41611b0 0x7f47e4158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:37.815+0000 7f47ea200700 1 --2- 172.21.15.89:0/3842655055 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f47e40d21f0 0x7f47e41f6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:37.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:37.815+0000 7f47ea200700 1 --2- 172.21.15.89:0/3842655055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47e415f4d0 0x7f47e41530e0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:37.815+0000 7f47ea200700 1 -- 172.21.15.89:0/3842655055 >> 172.21.15.89:0/3842655055 conn(0x7f47e408aff0 msgr2=0x7f47e415f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:37.816+0000 7f47ea200700 1 -- 172.21.15.89:0/3842655055 shutdown_connections 2024-04-02T09:09:37.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:37.816+0000 7f47ea200700 1 -- 172.21.15.89:0/3842655055 wait complete. 2024-04-02T09:09:37.815 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T09:09:37.833 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T09:09:38.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.027+0000 7f038cc00700 1 Processor -- start 2024-04-02T09:09:38.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.028+0000 7f038cc00700 1 -- start start 2024-04-02T09:09:38.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.029+0000 7f038cc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f038815b5d0 0x7f03880ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.029+0000 7f038cc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f03880cb3f0 0x7f03880ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.029+0000 7f038cc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038815fd50 con 0x7f03880ae930 2024-04-02T09:09:38.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.029+0000 7f038cc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f038815b9b0 con 0x7f038815b5d0 2024-04-02T09:09:38.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.029+0000 7f038cc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038815f140 con 0x7f03880cb3f0 2024-04-02T09:09:38.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.029+0000 7f0386800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f03880ae930 0x7f0388140dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36414/0 (socket says 172.21.15.89:36414) 2024-04-02T09:09:38.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.029+0000 7f0386800700 1 -- 172.21.15.89:0/1814391579 learned_addr learned my addr 172.21.15.89:0/1814391579 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.029+0000 7f0385400700 1 --2- 172.21.15.89:0/1814391579 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f03880cb3f0 0x7f03880ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.029+0000 7f0385e00700 1 --2- 172.21.15.89:0/1814391579 >> v2:172.21.15.113:3300/0 conn(0x7f038815b5d0 0x7f03880ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.029+0000 7f0384a00700 1 -- 172.21.15.89:0/1814391579 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569294696 0 0) 0x7f038815fd50 con 0x7f03880ae930 2024-04-02T09:09:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.029+0000 7f0384a00700 1 -- 172.21.15.89:0/1814391579 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f036c0035b0 con 0x7f03880ae930 2024-04-02T09:09:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0384a00700 1 -- 172.21.15.89:0/1814391579 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 279879387 0 0) 0x7f036c0035b0 con 0x7f03880ae930 2024-04-02T09:09:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0384a00700 1 -- 172.21.15.89:0/1814391579 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f03880cb3f0 msgr2=0x7f03880ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0384a00700 1 --2- 172.21.15.89:0/1814391579 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f03880cb3f0 0x7f03880ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0384a00700 1 -- 172.21.15.89:0/1814391579 >> v2:172.21.15.113:3300/0 conn(0x7f038815b5d0 msgr2=0x7f03880ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0384a00700 1 --2- 172.21.15.89:0/1814391579 >> v2:172.21.15.113:3300/0 conn(0x7f038815b5d0 0x7f03880ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0384a00700 1 -- 172.21.15.89:0/1814391579 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038815f420 con 0x7f03880ae930 2024-04-02T09:09:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0384a00700 1 -- 172.21.15.89:0/1814391579 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f03800019f0 con 0x7f03880ae930 2024-04-02T09:09:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0384a00700 1 -- 172.21.15.89:0/1814391579 >> v1:172.21.15.89:6789/0 conn(0x7f03880ae930 legacy=0x7f0388140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0384a00700 1 --2- 172.21.15.89:0/1814391579 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f036c003a00 0x7f036c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0384a00700 1 --2- 172.21.15.89:0/1814391579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f036c0043b0 0x7f036c004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0384a00700 1 --2- 172.21.15.89:0/1814391579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f036c004f00 0x7f036c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0384a00700 1 -- 172.21.15.89:0/1814391579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038815b9b0 con 0x7f036c0043b0 2024-04-02T09:09:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0384a00700 1 -- 172.21.15.89:0/1814391579 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038815f140 con 0x7f036c004f00 2024-04-02T09:09:38.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0384a00700 1 -- 172.21.15.89:0/1814391579 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036c00bf50 con 0x7f036c003a00 2024-04-02T09:09:38.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0385400700 1 --2- 172.21.15.89:0/1814391579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f036c0043b0 0x7f036c004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0386800700 1 --2- 172.21.15.89:0/1814391579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f036c004f00 0x7f036c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.030+0000 7f0385e00700 1 --2- 172.21.15.89:0/1814391579 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f036c003a00 0x7f036c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.031+0000 7f0385400700 1 -- 172.21.15.89:0/1814391579 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f036c003a00 msgr2=0x7f036c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:38.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.031+0000 7f0385400700 1 --2- 172.21.15.89:0/1814391579 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f036c003a00 0x7f036c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.031+0000 7f0385400700 1 -- 172.21.15.89:0/1814391579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f036c004f00 msgr2=0x7f036c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:38.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.031+0000 7f0385400700 1 --2- 172.21.15.89:0/1814391579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f036c004f00 0x7f036c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.031+0000 7f0385400700 1 -- 172.21.15.89:0/1814391579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c0035b0 con 0x7f036c0043b0 2024-04-02T09:09:38.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.031+0000 7f0385400700 1 --2- 172.21.15.89:0/1814391579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f036c0043b0 0x7f036c004890 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f0374009b60 tx=0x7f036c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c4212ae7aa91094 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:38.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.031+0000 7f0384a00700 1 -- 172.21.15.89:0/1814391579 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037400e6c0 con 0x7f036c0043b0 2024-04-02T09:09:38.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.031+0000 7f0384a00700 1 -- 172.21.15.89:0/1814391579 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037400e820 con 0x7f036c0043b0 2024-04-02T09:09:38.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.031+0000 7f0384a00700 1 -- 172.21.15.89:0/1814391579 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037400eb90 con 0x7f036c0043b0 2024-04-02T09:09:38.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.032+0000 7f038cc00700 1 -- 172.21.15.89:0/1814391579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f036c0043b0 msgr2=0x7f036c004890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.032+0000 7f038cc00700 1 --2- 172.21.15.89:0/1814391579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f036c0043b0 0x7f036c004890 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f0374009b60 tx=0x7f036c005880 comp rx=0 tx=0).stop 2024-04-02T09:09:38.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.032+0000 7f0386800700 1 -- 172.21.15.89:0/1814391579 reap_dead start 2024-04-02T09:09:38.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.032+0000 7f038cc00700 1 -- 172.21.15.89:0/1814391579 shutdown_connections 2024-04-02T09:09:38.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.032+0000 7f038cc00700 1 -- 172.21.15.89:0/1814391579 >> 172.21.15.89:0/1814391579 conn(0x7f03880cf190 msgr2=0x7f03880cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:38.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.032+0000 7f038cc00700 1 -- 172.21.15.89:0/1814391579 shutdown_connections 2024-04-02T09:09:38.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.032+0000 7f038cc00700 1 -- 172.21.15.89:0/1814391579 wait complete. 2024-04-02T09:09:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.033+0000 7f038cc00700 1 Processor -- start 2024-04-02T09:09:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.034+0000 7f038cc00700 1 -- start start 2024-04-02T09:09:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f038cc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f036c0043b0 0x7f038814fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f038cc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f03881446a0 0x7f0388152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f038cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0388152c90 0x7f038814c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f038cc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03880b3a60 con 0x7f0388152c90 2024-04-02T09:09:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f038cc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038814fce0 con 0x7f036c0043b0 2024-04-02T09:09:38.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f038cc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0388144b20 con 0x7f03881446a0 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f0386800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0388152c90 0x7f038814c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f0386800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0388152c90 0x7f038814c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47346/0 (socket says 172.21.15.89:47346) 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f0386800700 1 -- 172.21.15.89:0/2896982995 learned_addr learned my addr 172.21.15.89:0/2896982995 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f0385400700 1 --2- 172.21.15.89:0/2896982995 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f03881446a0 0x7f0388152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f0385e00700 1 --2- 172.21.15.89:0/2896982995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f036c0043b0 0x7f038814fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f0386800700 1 -- 172.21.15.89:0/2896982995 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f03881446a0 msgr2=0x7f0388152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f0386800700 1 --2- 172.21.15.89:0/2896982995 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f03881446a0 0x7f0388152750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f0386800700 1 -- 172.21.15.89:0/2896982995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f036c0043b0 msgr2=0x7f038814fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f0386800700 1 --2- 172.21.15.89:0/2896982995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f036c0043b0 0x7f038814fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.035+0000 7f0386800700 1 -- 172.21.15.89:0/2896982995 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03880b30c0 con 0x7f0388152c90 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.036+0000 7f0386800700 1 --2- 172.21.15.89:0/2896982995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0388152c90 0x7f038814c8f0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f03800013d0 tx=0x7f038000a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.036+0000 7f037e000700 1 -- 172.21.15.89:0/2896982995 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038000ae30 con 0x7f0388152c90 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.036+0000 7f037e000700 1 -- 172.21.15.89:0/2896982995 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03800133f0 con 0x7f0388152c90 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.036+0000 7f038cc00700 1 -- 172.21.15.89:0/2896982995 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03880b33a0 con 0x7f0388152c90 2024-04-02T09:09:38.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.036+0000 7f037e000700 1 -- 172.21.15.89:0/2896982995 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03800137c0 con 0x7f0388152c90 2024-04-02T09:09:38.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.036+0000 7f038cc00700 1 -- 172.21.15.89:0/2896982995 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0388153110 con 0x7f0388152c90 2024-04-02T09:09:38.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.037+0000 7f038cc00700 1 -- 172.21.15.89:0/2896982995 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f038815f9a0 con 0x7f0388152c90 2024-04-02T09:09:38.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.038+0000 7f037e000700 1 -- 172.21.15.89:0/2896982995 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0380013920 con 0x7f0388152c90 2024-04-02T09:09:38.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.040+0000 7f037e000700 1 --2- 172.21.15.89:0/2896982995 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f036806adb0 0x7f036806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.040+0000 7f037e000700 1 -- 172.21.15.89:0/2896982995 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f0380089d80 con 0x7f0388152c90 2024-04-02T09:09:38.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.040+0000 7f0385e00700 1 --2- 172.21.15.89:0/2896982995 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f036806adb0 0x7f036806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.040+0000 7f0385e00700 1 --2- 172.21.15.89:0/2896982995 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f036806adb0 0x7f036806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0378009ba0 tx=0x7f0378008040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:38.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.044+0000 7f037e000700 1 -- 172.21.15.89:0/2896982995 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0380054f40 con 0x7f0388152c90 2024-04-02T09:09:38.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.323+0000 7f038cc00700 1 -- 172.21.15.89:0/2896982995 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0388067720 con 0x7f0388152c90 2024-04-02T09:09:38.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.324+0000 7f037e000700 1 -- 172.21.15.89:0/2896982995 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v51) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0380058560 con 0x7f0388152c90 2024-04-02T09:09:38.322 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:38.323 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:09:38.323 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T09:09:38.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.323 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T09:09:38.323 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T09:09:38.323 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.323 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:09:38.323 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:09:38.323 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:09:38.323 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.323 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.323 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T09:09:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T09:09:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:09:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:09:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:09:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T09:09:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T09:09:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.324 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:09:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:09:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:09:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T09:09:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T09:09:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:09:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:09:38.325 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:09:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T09:09:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T09:09:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:09:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:09:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T09:09:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T09:09:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:09:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:09:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T09:09:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T09:09:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:09:38.327 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:09:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T09:09:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T09:09:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:09:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:09:38.328 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T09:09:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T09:09:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:09:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:09:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T09:09:38.329 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T09:09:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:09:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:09:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T09:09:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T09:09:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:09:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:09:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.330 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T09:09:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T09:09:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:09:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:09:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T09:09:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T09:09:38.331 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:09:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2024-04-02T09:09:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T09:09:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T09:09:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T09:09:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:09:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:09:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:09:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:09:38.332 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T09:09:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T09:09:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:09:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:09:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:09:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:09:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T09:09:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T09:09:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:09:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:09:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:09:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:09:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T09:09:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T09:09:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:09:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:09:38.334 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:09:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T09:09:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T09:09:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:09:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T09:09:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:09:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:38.335 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T09:09:38.336 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T09:09:38.336 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:09:38.336 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:09:38.336 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T09:09:38.336 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:09:38.336 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.336 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:38.336 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T09:09:38.336 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:09:38.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.326+0000 7f038cc00700 1 -- 172.21.15.89:0/2896982995 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f036806adb0 msgr2=0x7f036806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.326+0000 7f038cc00700 1 --2- 172.21.15.89:0/2896982995 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f036806adb0 0x7f036806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0378009ba0 tx=0x7f0378008040 comp rx=0 tx=0).stop 2024-04-02T09:09:38.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.326+0000 7f038cc00700 1 -- 172.21.15.89:0/2896982995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0388152c90 msgr2=0x7f038814c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.326+0000 7f038cc00700 1 --2- 172.21.15.89:0/2896982995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0388152c90 0x7f038814c8f0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f03800013d0 tx=0x7f038000a600 comp rx=0 tx=0).stop 2024-04-02T09:09:38.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.326+0000 7f038cc00700 1 -- 172.21.15.89:0/2896982995 shutdown_connections 2024-04-02T09:09:38.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.326+0000 7f038cc00700 1 --2- 172.21.15.89:0/2896982995 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f036806adb0 0x7f036806d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.326+0000 7f038cc00700 1 --2- 172.21.15.89:0/2896982995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f036c0043b0 0x7f038814fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.326+0000 7f038cc00700 1 --2- 172.21.15.89:0/2896982995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0388152c90 0x7f038814c8f0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.326+0000 7f038cc00700 1 --2- 172.21.15.89:0/2896982995 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f03881446a0 0x7f0388152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.326+0000 7f038cc00700 1 -- 172.21.15.89:0/2896982995 >> 172.21.15.89:0/2896982995 conn(0x7f03880cf190 msgr2=0x7f0388155840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:38.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.327+0000 7f038cc00700 1 -- 172.21.15.89:0/2896982995 shutdown_connections 2024-04-02T09:09:38.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.327+0000 7f038cc00700 1 -- 172.21.15.89:0/2896982995 wait complete. 2024-04-02T09:09:38.343 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:09:38.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.535+0000 7f4c6f000700 1 Processor -- start 2024-04-02T09:09:38.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.536+0000 7f4c6f000700 1 -- start start 2024-04-02T09:09:38.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.537+0000 7f4c6f000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4c6815b5d0 0x7f4c680ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.537+0000 7f4c6f000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c680ac7d0 0x7f4c6806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.537+0000 7f4c6f000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6815fd50 con 0x7f4c680cb3f0 2024-04-02T09:09:38.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.537+0000 7f4c6f000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c6815b9b0 con 0x7f4c6815b5d0 2024-04-02T09:09:38.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.537+0000 7f4c6f000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c6815f140 con 0x7f4c680ac7d0 2024-04-02T09:09:38.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.537+0000 7f4c67e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4c6815b5d0 0x7f4c680ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.537+0000 7f4c67e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4c6815b5d0 0x7f4c680ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47362/0 (socket says 172.21.15.89:47362) 2024-04-02T09:09:38.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.537+0000 7f4c67e00700 1 -- 172.21.15.89:0/3389086753 learned_addr learned my addr 172.21.15.89:0/3389086753 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:38.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.537+0000 7f4c6d000700 1 --2- 172.21.15.89:0/3389086753 >> v2:172.21.15.113:3300/0 conn(0x7f4c680ac7d0 0x7f4c6806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.538+0000 7f4c67e00700 1 -- 172.21.15.89:0/3389086753 >> v2:172.21.15.113:3300/0 conn(0x7f4c680ac7d0 msgr2=0x7f4c6806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:38.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.538+0000 7f4c67e00700 1 --2- 172.21.15.89:0/3389086753 >> v2:172.21.15.113:3300/0 conn(0x7f4c680ac7d0 0x7f4c6806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.538+0000 7f4c67e00700 1 -- 172.21.15.89:0/3389086753 >> v1:172.21.15.113:6789/0 conn(0x7f4c680cb3f0 legacy=0x7f4c680ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.538+0000 7f4c67e00700 1 -- 172.21.15.89:0/3389086753 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6815f420 con 0x7f4c6815b5d0 2024-04-02T09:09:38.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.538+0000 7f4c6d000700 1 --2- 172.21.15.89:0/3389086753 >> v2:172.21.15.113:3300/0 conn(0x7f4c680ac7d0 0x7f4c6806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:38.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.538+0000 7f4c67e00700 1 --2- 172.21.15.89:0/3389086753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c6815b5d0 0x7f4c680ae3f0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f4c680cfdc0 tx=0x7f4c58009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c2c4a094eb0a1cc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:38.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.538+0000 7f4c66a00700 1 -- 172.21.15.89:0/3389086753 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5800b870 con 0x7f4c6815b5d0 2024-04-02T09:09:38.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.538+0000 7f4c66a00700 1 -- 172.21.15.89:0/3389086753 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5800b9d0 con 0x7f4c6815b5d0 2024-04-02T09:09:38.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.538+0000 7f4c66a00700 1 -- 172.21.15.89:0/3389086753 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5800bd60 con 0x7f4c6815b5d0 2024-04-02T09:09:38.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.539+0000 7f4c6f000700 1 -- 172.21.15.89:0/3389086753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c6815b5d0 msgr2=0x7f4c680ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.539+0000 7f4c6f000700 1 --2- 172.21.15.89:0/3389086753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c6815b5d0 0x7f4c680ae3f0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f4c680cfdc0 tx=0x7f4c58009bd0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.539+0000 7f4c6f000700 1 -- 172.21.15.89:0/3389086753 shutdown_connections 2024-04-02T09:09:38.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.539+0000 7f4c6f000700 1 --2- 172.21.15.89:0/3389086753 >> v2:172.21.15.113:3300/0 conn(0x7f4c680ac7d0 0x7f4c6806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.539+0000 7f4c6f000700 1 --2- 172.21.15.89:0/3389086753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c6815b5d0 0x7f4c680ae3f0 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.539+0000 7f4c6f000700 1 -- 172.21.15.89:0/3389086753 >> 172.21.15.89:0/3389086753 conn(0x7f4c680cf190 msgr2=0x7f4c680cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:38.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.539+0000 7f4c6f000700 1 -- 172.21.15.89:0/3389086753 shutdown_connections 2024-04-02T09:09:38.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.539+0000 7f4c6f000700 1 -- 172.21.15.89:0/3389086753 wait complete. 2024-04-02T09:09:38.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.540+0000 7f4c6f000700 1 Processor -- start 2024-04-02T09:09:38.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.541+0000 7f4c6f000700 1 -- start start 2024-04-02T09:09:38.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.541+0000 7f4c6f000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c680ac7d0 0x7f4c680cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.541+0000 7f4c6f000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c6815b5d0 0x7f4c68159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.541+0000 7f4c6f000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c6814c700 0x7f4c680b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.541+0000 7f4c6f000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6814cbb0 con 0x7f4c6814c700 2024-04-02T09:09:38.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.541+0000 7f4c6f000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c680cc230 con 0x7f4c6815b5d0 2024-04-02T09:09:38.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.541+0000 7f4c6f000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c680cc510 con 0x7f4c680ac7d0 2024-04-02T09:09:38.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.542+0000 7f4c6d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c6814c700 0x7f4c680b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.542+0000 7f4c6d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c6814c700 0x7f4c680b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47376/0 (socket says 172.21.15.89:47376) 2024-04-02T09:09:38.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.542+0000 7f4c6d000700 1 -- 172.21.15.89:0/1938407688 learned_addr learned my addr 172.21.15.89:0/1938407688 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:38.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.542+0000 7f4c67e00700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c680ac7d0 0x7f4c680cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.542+0000 7f4c67400700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c6815b5d0 0x7f4c68159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.542+0000 7f4c6d000700 1 -- 172.21.15.89:0/1938407688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c680ac7d0 msgr2=0x7f4c680cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.542+0000 7f4c6d000700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c680ac7d0 0x7f4c680cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.542+0000 7f4c6d000700 1 -- 172.21.15.89:0/1938407688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c6815b5d0 msgr2=0x7f4c68159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.542+0000 7f4c6d000700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c6815b5d0 0x7f4c68159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.542+0000 7f4c6d000700 1 -- 172.21.15.89:0/1938407688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c58009550 con 0x7f4c6814c700 2024-04-02T09:09:38.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.542+0000 7f4c67400700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c6815b5d0 0x7f4c68159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:38.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.542+0000 7f4c6d000700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c6814c700 0x7f4c680b31d0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f4c60007ef0 tx=0x7f4c6000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:38.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.543+0000 7f4c5fe00700 1 -- 172.21.15.89:0/1938407688 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6000de30 con 0x7f4c6814c700 2024-04-02T09:09:38.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.543+0000 7f4c5fe00700 1 -- 172.21.15.89:0/1938407688 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6000f040 con 0x7f4c6814c700 2024-04-02T09:09:38.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.543+0000 7f4c6f000700 1 -- 172.21.15.89:0/1938407688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c680cc7f0 con 0x7f4c6814c700 2024-04-02T09:09:38.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.543+0000 7f4c5fe00700 1 -- 172.21.15.89:0/1938407688 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60003c60 con 0x7f4c6814c700 2024-04-02T09:09:38.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.543+0000 7f4c6f000700 1 -- 172.21.15.89:0/1938407688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c680cce10 con 0x7f4c6814c700 2024-04-02T09:09:38.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.544+0000 7f4c6f000700 1 -- 172.21.15.89:0/1938407688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c680cd030 con 0x7f4c6814c700 2024-04-02T09:09:38.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.545+0000 7f4c5fe00700 1 -- 172.21.15.89:0/1938407688 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4c60004030 con 0x7f4c6814c700 2024-04-02T09:09:38.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.545+0000 7f4c5fe00700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4c4806adb0 0x7f4c4806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.546+0000 7f4c5fe00700 1 -- 172.21.15.89:0/1938407688 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f4c60089560 con 0x7f4c6814c700 2024-04-02T09:09:38.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.546+0000 7f4c67e00700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4c4806adb0 0x7f4c4806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.546+0000 7f4c67e00700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4c4806adb0 0x7f4c4806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4c58014040 tx=0x7f4c58008030 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:38.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.550+0000 7f4c5fe00700 1 -- 172.21.15.89:0/1938407688 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c600546a0 con 0x7f4c6814c700 2024-04-02T09:09:38.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.756+0000 7f4c6f000700 1 -- 172.21.15.89:0/1938407688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c68067720 con 0x7f4c6814c700 2024-04-02T09:09:38.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.757+0000 7f4c5fe00700 1 -- 172.21.15.89:0/1938407688 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) v1 ==== 74+0+13088 (secure 0 0 0) 0x7f4c60057cc0 con 0x7f4c6814c700 2024-04-02T09:09:38.755 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:38.755 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":201,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:09:37.809223+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:09:38.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.759+0000 7f4c6f000700 1 -- 172.21.15.89:0/1938407688 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4c4806adb0 msgr2=0x7f4c4806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.759+0000 7f4c6f000700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4c4806adb0 0x7f4c4806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4c58014040 tx=0x7f4c58008030 comp rx=0 tx=0).stop 2024-04-02T09:09:38.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.759+0000 7f4c6f000700 1 -- 172.21.15.89:0/1938407688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c6814c700 msgr2=0x7f4c680b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.759+0000 7f4c6f000700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c6814c700 0x7f4c680b31d0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f4c60007ef0 tx=0x7f4c6000d3b0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.759+0000 7f4c6f000700 1 -- 172.21.15.89:0/1938407688 shutdown_connections 2024-04-02T09:09:38.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.759+0000 7f4c6f000700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c6814c700 0x7f4c680b31d0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.759+0000 7f4c6f000700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c680ac7d0 0x7f4c680cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.759+0000 7f4c6f000700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4c4806adb0 0x7f4c4806d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.759+0000 7f4c6f000700 1 --2- 172.21.15.89:0/1938407688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c6815b5d0 0x7f4c68159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.759+0000 7f4c6f000700 1 -- 172.21.15.89:0/1938407688 >> 172.21.15.89:0/1938407688 conn(0x7f4c680cf190 msgr2=0x7f4c6815dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:38.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.759+0000 7f4c6f000700 1 -- 172.21.15.89:0/1938407688 shutdown_connections 2024-04-02T09:09:38.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.759+0000 7f4c6f000700 1 -- 172.21.15.89:0/1938407688 wait complete. 2024-04-02T09:09:38.777 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:09:38.777 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:09:38.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.973+0000 7ff9a0200700 1 Processor -- start 2024-04-02T09:09:38.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.973+0000 7ff9a0200700 1 -- start start 2024-04-02T09:09:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.974+0000 7ff9a0200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff99815d9d0 0x7ff9980d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.975+0000 7ff9a0200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff998161420 con 0x7ff9980d16a0 2024-04-02T09:09:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.975+0000 7ff9a0200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff998161700 con 0x7ff9980d2540 2024-04-02T09:09:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.975+0000 7ff9a0200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff99815edc0 con 0x7ff99815d9d0 2024-04-02T09:09:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.975+0000 7ff99d800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff99815d9d0 0x7ff9980d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.975+0000 7ff99ce00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff9980d16a0 0x7ff9980d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36492/0 (socket says 172.21.15.89:36492) 2024-04-02T09:09:38.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.975+0000 7ff99ce00700 1 -- 172.21.15.89:0/4220448731 learned_addr learned my addr 172.21.15.89:0/4220448731 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:38.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.975+0000 7ff997e00700 1 -- 172.21.15.89:0/4220448731 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232319732 0 0) 0x7ff998161420 con 0x7ff9980d16a0 2024-04-02T09:09:38.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.975+0000 7ff997e00700 1 -- 172.21.15.89:0/4220448731 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9800035b0 con 0x7ff9980d16a0 2024-04-02T09:09:38.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.975+0000 7ff99d800700 1 -- 172.21.15.89:0/4220448731 >> v1:172.21.15.113:6789/0 conn(0x7ff9980d2540 legacy=0x7ff9980ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.975+0000 7ff99d800700 1 -- 172.21.15.89:0/4220448731 >> v1:172.21.15.89:6789/0 conn(0x7ff9980d16a0 legacy=0x7ff9980d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.975+0000 7ff99d800700 1 -- 172.21.15.89:0/4220448731 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9981619e0 con 0x7ff99815d9d0 2024-04-02T09:09:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.976+0000 7ff99d800700 1 --2- 172.21.15.89:0/4220448731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99815d9d0 0x7ff9980d3c30 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009840 tx=0x7ff98c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac91c9dafc722882 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.976+0000 7ff997e00700 1 -- 172.21.15.89:0/4220448731 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c004230 con 0x7ff99815d9d0 2024-04-02T09:09:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.976+0000 7ff997e00700 1 -- 172.21.15.89:0/4220448731 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c004390 con 0x7ff99815d9d0 2024-04-02T09:09:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.976+0000 7ff997e00700 1 -- 172.21.15.89:0/4220448731 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0036c0 con 0x7ff99815d9d0 2024-04-02T09:09:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.976+0000 7ff9a0200700 1 -- 172.21.15.89:0/4220448731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99815d9d0 msgr2=0x7ff9980d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.976+0000 7ff9a0200700 1 --2- 172.21.15.89:0/4220448731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99815d9d0 0x7ff9980d3c30 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009840 tx=0x7ff98c009b50 comp rx=0 tx=0).stop 2024-04-02T09:09:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.976+0000 7ff9a0200700 1 -- 172.21.15.89:0/4220448731 shutdown_connections 2024-04-02T09:09:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.976+0000 7ff9a0200700 1 --2- 172.21.15.89:0/4220448731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99815d9d0 0x7ff9980d3c30 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.976+0000 7ff9a0200700 1 -- 172.21.15.89:0/4220448731 >> 172.21.15.89:0/4220448731 conn(0x7ff99808aff0 msgr2=0x7ff9980d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.977+0000 7ff9a0200700 1 -- 172.21.15.89:0/4220448731 shutdown_connections 2024-04-02T09:09:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.977+0000 7ff9a0200700 1 -- 172.21.15.89:0/4220448731 wait complete. 2024-04-02T09:09:38.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.978+0000 7ff9a0200700 1 Processor -- start 2024-04-02T09:09:38.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.978+0000 7ff9a0200700 1 -- start start 2024-04-02T09:09:38.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.979+0000 7ff9a0200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff9980d16a0 0x7ff99815bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.979+0000 7ff9a0200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9980d2540 0x7ff9981e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.979+0000 7ff9a0200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99815d9d0 0x7ff9981e77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.979+0000 7ff9a0200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9980b6950 con 0x7ff99815d9d0 2024-04-02T09:09:38.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.979+0000 7ff9a0200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9981e93e0 con 0x7ff9980d2540 2024-04-02T09:09:38.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.979+0000 7ff9a0200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9980b6020 con 0x7ff9980d16a0 2024-04-02T09:09:38.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.980+0000 7ff99e200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99815d9d0 0x7ff9981e77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.980+0000 7ff99e200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99815d9d0 0x7ff9981e77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40292/0 (socket says 172.21.15.89:40292) 2024-04-02T09:09:38.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.980+0000 7ff99ce00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9980d2540 0x7ff9981e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.980+0000 7ff99e200700 1 -- 172.21.15.89:0/2517823288 learned_addr learned my addr 172.21.15.89:0/2517823288 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:38.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.980+0000 7ff99d800700 1 --2- 172.21.15.89:0/2517823288 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff9980d16a0 0x7ff99815bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.980+0000 7ff99e200700 1 -- 172.21.15.89:0/2517823288 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff9980d16a0 msgr2=0x7ff99815bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.980+0000 7ff99e200700 1 --2- 172.21.15.89:0/2517823288 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff9980d16a0 0x7ff99815bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.980+0000 7ff99e200700 1 -- 172.21.15.89:0/2517823288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9980d2540 msgr2=0x7ff9981e7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:38.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.980+0000 7ff99e200700 1 --2- 172.21.15.89:0/2517823288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9980d2540 0x7ff9981e7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:38.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.980+0000 7ff99e200700 1 -- 172.21.15.89:0/2517823288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c0094a0 con 0x7ff99815d9d0 2024-04-02T09:09:38.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.980+0000 7ff99e200700 1 --2- 172.21.15.89:0/2517823288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99815d9d0 0x7ff9981e77c0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7ff99000cba0 tx=0x7ff99000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:38.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.981+0000 7ff995600700 1 -- 172.21.15.89:0/2517823288 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99000aaa0 con 0x7ff99815d9d0 2024-04-02T09:09:38.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.981+0000 7ff995600700 1 -- 172.21.15.89:0/2517823288 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99000ac00 con 0x7ff99815d9d0 2024-04-02T09:09:38.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.981+0000 7ff9a0200700 1 -- 172.21.15.89:0/2517823288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9980b6300 con 0x7ff99815d9d0 2024-04-02T09:09:38.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.981+0000 7ff995600700 1 -- 172.21.15.89:0/2517823288 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff990012420 con 0x7ff99815d9d0 2024-04-02T09:09:38.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.981+0000 7ff9a0200700 1 -- 172.21.15.89:0/2517823288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99815ce50 con 0x7ff99815d9d0 2024-04-02T09:09:38.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.982+0000 7ff9a0200700 1 -- 172.21.15.89:0/2517823288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9980d1ca0 con 0x7ff99815d9d0 2024-04-02T09:09:38.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.983+0000 7ff995600700 1 -- 172.21.15.89:0/2517823288 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff990012580 con 0x7ff99815d9d0 2024-04-02T09:09:38.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.983+0000 7ff995600700 1 --2- 172.21.15.89:0/2517823288 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff97806adb0 0x7ff97806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:38.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.984+0000 7ff995600700 1 -- 172.21.15.89:0/2517823288 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5305+0+0 (secure 0 0 0) 0x7ff9900899b0 con 0x7ff99815d9d0 2024-04-02T09:09:38.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.984+0000 7ff99d800700 1 --2- 172.21.15.89:0/2517823288 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff97806adb0 0x7ff97806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:38.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.984+0000 7ff99d800700 1 --2- 172.21.15.89:0/2517823288 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff97806adb0 0x7ff97806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0090b0 tx=0x7ff98c010000 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:38.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:38.988+0000 7ff995600700 1 -- 172.21.15.89:0/2517823288 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff990054b70 con 0x7ff99815d9d0 2024-04-02T09:09:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.191+0000 7ff9a0200700 1 -- 172.21.15.89:0/2517823288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff998067720 con 0x7ff99815d9d0 2024-04-02T09:09:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.191+0000 7ff995600700 1 -- 172.21.15.89:0/2517823288 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v201) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff990016070 con 0x7ff99815d9d0 2024-04-02T09:09:39.190 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:09:39.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.193+0000 7ff9a0200700 1 -- 172.21.15.89:0/2517823288 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff97806adb0 msgr2=0x7ff97806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:39.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.193+0000 7ff9a0200700 1 --2- 172.21.15.89:0/2517823288 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff97806adb0 0x7ff97806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0090b0 tx=0x7ff98c010000 comp rx=0 tx=0).stop 2024-04-02T09:09:39.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.193+0000 7ff9a0200700 1 -- 172.21.15.89:0/2517823288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99815d9d0 msgr2=0x7ff9981e77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:39.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.193+0000 7ff9a0200700 1 --2- 172.21.15.89:0/2517823288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99815d9d0 0x7ff9981e77c0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7ff99000cba0 tx=0x7ff99000ceb0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.194+0000 7ff9a0200700 1 -- 172.21.15.89:0/2517823288 shutdown_connections 2024-04-02T09:09:39.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.194+0000 7ff9a0200700 1 --2- 172.21.15.89:0/2517823288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff99815d9d0 0x7ff9981e77c0 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.194+0000 7ff9a0200700 1 --2- 172.21.15.89:0/2517823288 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff9980d16a0 0x7ff99815bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.194+0000 7ff9a0200700 1 --2- 172.21.15.89:0/2517823288 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff97806adb0 0x7ff97806d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.194+0000 7ff9a0200700 1 --2- 172.21.15.89:0/2517823288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9980d2540 0x7ff9981e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.194+0000 7ff9a0200700 1 -- 172.21.15.89:0/2517823288 >> 172.21.15.89:0/2517823288 conn(0x7ff99808aff0 msgr2=0x7ff9980b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:39.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.194+0000 7ff9a0200700 1 -- 172.21.15.89:0/2517823288 shutdown_connections 2024-04-02T09:09:39.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.194+0000 7ff9a0200700 1 -- 172.21.15.89:0/2517823288 wait complete. 2024-04-02T09:09:39.210 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T09:09:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.403+0000 7fabf9c00700 1 Processor -- start 2024-04-02T09:09:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.403+0000 7fabf9c00700 1 -- start start 2024-04-02T09:09:39.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.404+0000 7fabf9c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf40b3af0 0x7fabf415a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:39.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.404+0000 7fabf9c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fabf40709c0 0x7fabf4070e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:39.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.404+0000 7fabf9c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf40b2180 con 0x7fabf4072580 2024-04-02T09:09:39.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.404+0000 7fabf9c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf415b100 con 0x7fabf40709c0 2024-04-02T09:09:39.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.404+0000 7fabf9c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf415b3e0 con 0x7fabf40b3af0 2024-04-02T09:09:39.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.405+0000 7fabf2a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf40b3af0 0x7fabf415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:39.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.405+0000 7fabf3400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fabf40709c0 0x7fabf4070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:39.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.405+0000 7fabf2a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf40b3af0 0x7fabf415a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:40098/0 (socket says 172.21.15.89:40098) 2024-04-02T09:09:39.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.405+0000 7fabf2a00700 1 -- 172.21.15.89:0/1290624453 learned_addr learned my addr 172.21.15.89:0/1290624453 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:39.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.405+0000 7fabf2000700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fabf4072580 0x7fabf40b2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:54412/0 (socket says 172.21.15.89:54412) 2024-04-02T09:09:39.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.405+0000 7fabf3400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fabf40709c0 0x7fabf4070e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:51994/0 (socket says 172.21.15.89:51994) 2024-04-02T09:09:39.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.405+0000 7fabf2a00700 1 -- 172.21.15.89:0/1290624453 >> v2:172.21.15.113:3300/0 conn(0x7fabf40709c0 msgr2=0x7fabf4070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:39.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.405+0000 7fabf2a00700 1 --2- 172.21.15.89:0/1290624453 >> v2:172.21.15.113:3300/0 conn(0x7fabf40709c0 0x7fabf4070e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.405+0000 7fabf2a00700 1 -- 172.21.15.89:0/1290624453 >> v1:172.21.15.113:6789/0 conn(0x7fabf4072580 legacy=0x7fabf40b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:39.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.405+0000 7fabf2a00700 1 -- 172.21.15.89:0/1290624453 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf40b23f0 con 0x7fabf40b3af0 2024-04-02T09:09:39.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.406+0000 7fabf2a00700 1 --2- 172.21.15.89:0/1290624453 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf40b3af0 0x7fabf415a860 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fabe400bb90 tx=0x7fabe400bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89889fd62b4b0e7a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:39.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.406+0000 7fabf1600700 1 -- 172.21.15.89:0/1290624453 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe400dd30 con 0x7fabf40b3af0 2024-04-02T09:09:39.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.406+0000 7fabf1600700 1 -- 172.21.15.89:0/1290624453 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe400de90 con 0x7fabf40b3af0 2024-04-02T09:09:39.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.406+0000 7fabf1600700 1 -- 172.21.15.89:0/1290624453 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe4010690 con 0x7fabf40b3af0 2024-04-02T09:09:39.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.407+0000 7fabf9c00700 1 -- 172.21.15.89:0/1290624453 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf40b3af0 msgr2=0x7fabf415a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:39.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.407+0000 7fabf9c00700 1 --2- 172.21.15.89:0/1290624453 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf40b3af0 0x7fabf415a860 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fabe400bb90 tx=0x7fabe400bea0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.407+0000 7fabf9c00700 1 -- 172.21.15.89:0/1290624453 shutdown_connections 2024-04-02T09:09:39.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.407+0000 7fabf9c00700 1 --2- 172.21.15.89:0/1290624453 >> v2:172.21.15.113:3300/0 conn(0x7fabf40709c0 0x7fabf4070e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.407+0000 7fabf9c00700 1 --2- 172.21.15.89:0/1290624453 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf40b3af0 0x7fabf415a860 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.407+0000 7fabf9c00700 1 -- 172.21.15.89:0/1290624453 >> 172.21.15.89:0/1290624453 conn(0x7fabf40cf190 msgr2=0x7fabf40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:39.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.407+0000 7fabf9c00700 1 -- 172.21.15.89:0/1290624453 shutdown_connections 2024-04-02T09:09:39.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.407+0000 7fabf9c00700 1 -- 172.21.15.89:0/1290624453 wait complete. 2024-04-02T09:09:39.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.408+0000 7fabf9c00700 1 Processor -- start 2024-04-02T09:09:39.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.409+0000 7fabf9c00700 1 -- start start 2024-04-02T09:09:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.409+0000 7fabf9c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf40709c0 0x7fabf414d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.409+0000 7fabf9c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf4072580 0x7fabf41f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.409+0000 7fabf9c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf40b3af0 0x7fabf415cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.409+0000 7fabf9c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf414dce0 con 0x7fabf40b3af0 2024-04-02T09:09:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.409+0000 7fabf9c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf41f4010 con 0x7fabf4072580 2024-04-02T09:09:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.409+0000 7fabf9c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf415c830 con 0x7fabf40709c0 2024-04-02T09:09:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.410+0000 7fabf3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf40b3af0 0x7fabf415cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.410+0000 7fabf3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf40b3af0 0x7fabf415cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40302/0 (socket says 172.21.15.89:40302) 2024-04-02T09:09:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.410+0000 7fabf3400700 1 -- 172.21.15.89:0/2941823849 learned_addr learned my addr 172.21.15.89:0/2941823849 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.410+0000 7fabf2a00700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf40709c0 0x7fabf414d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.410+0000 7fabf2000700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf4072580 0x7fabf41f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.410+0000 7fabf3400700 1 -- 172.21.15.89:0/2941823849 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf40709c0 msgr2=0x7fabf414d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.410+0000 7fabf3400700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf40709c0 0x7fabf414d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.410+0000 7fabf3400700 1 -- 172.21.15.89:0/2941823849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf4072580 msgr2=0x7fabf41f3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.410+0000 7fabf3400700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf4072580 0x7fabf41f3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.410+0000 7fabf3400700 1 -- 172.21.15.89:0/2941823849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe400b7f0 con 0x7fabf40b3af0 2024-04-02T09:09:39.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.410+0000 7fabf2000700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf4072580 0x7fabf41f3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:39.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.411+0000 7fabf3400700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf40b3af0 0x7fabf415cf20 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fabec00efc0 tx=0x7fabec00c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:39.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.411+0000 7fabeaa00700 1 -- 172.21.15.89:0/2941823849 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabec00cb80 con 0x7fabf40b3af0 2024-04-02T09:09:39.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.411+0000 7fabeaa00700 1 -- 172.21.15.89:0/2941823849 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabec00f040 con 0x7fabf40b3af0 2024-04-02T09:09:39.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.411+0000 7fabf9c00700 1 -- 172.21.15.89:0/2941823849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf415bcb0 con 0x7fabf40b3af0 2024-04-02T09:09:39.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.411+0000 7fabeaa00700 1 -- 172.21.15.89:0/2941823849 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabec014480 con 0x7fabf40b3af0 2024-04-02T09:09:39.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.411+0000 7fabf9c00700 1 -- 172.21.15.89:0/2941823849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf415c270 con 0x7fabf40b3af0 2024-04-02T09:09:39.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.412+0000 7fabf9c00700 1 -- 172.21.15.89:0/2941823849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf4067720 con 0x7fabf40b3af0 2024-04-02T09:09:39.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.413+0000 7fabeaa00700 1 -- 172.21.15.89:0/2941823849 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fabec0145e0 con 0x7fabf40b3af0 2024-04-02T09:09:39.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.414+0000 7fabeaa00700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fabd406adb0 0x7fabd406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:39.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.414+0000 7fabeaa00700 1 -- 172.21.15.89:0/2941823849 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fabec089f90 con 0x7fabf40b3af0 2024-04-02T09:09:39.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.414+0000 7fabf2a00700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fabd406adb0 0x7fabd406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:39.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.415+0000 7fabf2a00700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fabd406adb0 0x7fabd406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fabe400bee0 tx=0x7fabe401c040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:39.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.419+0000 7fabeaa00700 1 -- 172.21.15.89:0/2941823849 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fabec055040 con 0x7fabf40b3af0 2024-04-02T09:09:39.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.625+0000 7fabf9c00700 1 -- 172.21.15.89:0/2941823849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fabf40cb980 con 0x7fabf40b3af0 2024-04-02T09:09:39.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.626+0000 7fabeaa00700 1 -- 172.21.15.89:0/2941823849 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v201) v1 ==== 84+0+65 (secure 0 0 0) 0x7fabec018090 con 0x7fabf40b3af0 2024-04-02T09:09:39.624 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:39.625 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T09:09:39.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:39.625 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T09:09:39.625 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T09:09:39.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:39.625 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T09:09:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.628+0000 7fabf9c00700 1 -- 172.21.15.89:0/2941823849 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fabd406adb0 msgr2=0x7fabd406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.628+0000 7fabf9c00700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fabd406adb0 0x7fabd406d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fabe400bee0 tx=0x7fabe401c040 comp rx=0 tx=0).stop 2024-04-02T09:09:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.628+0000 7fabf9c00700 1 -- 172.21.15.89:0/2941823849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf40b3af0 msgr2=0x7fabf415cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.628+0000 7fabf9c00700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf40b3af0 0x7fabf415cf20 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fabec00efc0 tx=0x7fabec00c0c0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.628+0000 7fabf9c00700 1 -- 172.21.15.89:0/2941823849 shutdown_connections 2024-04-02T09:09:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.628+0000 7fabf9c00700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf40b3af0 0x7fabf415cf20 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.628+0000 7fabf9c00700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf40709c0 0x7fabf414d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.628+0000 7fabf9c00700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fabd406adb0 0x7fabd406d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.628+0000 7fabf9c00700 1 --2- 172.21.15.89:0/2941823849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf4072580 0x7fabf41f3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.629+0000 7fabf9c00700 1 -- 172.21.15.89:0/2941823849 >> 172.21.15.89:0/2941823849 conn(0x7fabf40cf190 msgr2=0x7fabf40ad4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:39.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.629+0000 7fabf9c00700 1 -- 172.21.15.89:0/2941823849 shutdown_connections 2024-04-02T09:09:39.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.629+0000 7fabf9c00700 1 -- 172.21.15.89:0/2941823849 wait complete. 2024-04-02T09:09:39.645 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T09:09:39.645 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T09:09:39.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.842+0000 7f9d35a00700 1 Processor -- start 2024-04-02T09:09:39.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.843+0000 7f9d35a00700 1 -- start start 2024-04-02T09:09:39.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.844+0000 7f9d35a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d3015ddf0 0x7f9d300ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:39.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.844+0000 7f9d35a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9d3015e1d0 0x7f9d300d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:39.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.844+0000 7f9d35a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d300ce8a0 con 0x7f9d300d2710 2024-04-02T09:09:39.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.844+0000 7f9d35a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d300d2b80 con 0x7f9d3015e1d0 2024-04-02T09:09:39.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.844+0000 7f9d35a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3015ec60 con 0x7f9d3015ddf0 2024-04-02T09:09:39.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.844+0000 7f9d2e000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9d3015e1d0 0x7f9d300d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:39.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.844+0000 7f9d2f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9d300d2710 0x7f9d301612f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36496/0 (socket says 172.21.15.89:36496) 2024-04-02T09:09:39.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.844+0000 7f9d2f400700 1 -- 172.21.15.89:0/2554593095 learned_addr learned my addr 172.21.15.89:0/2554593095 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:39.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.844+0000 7f9d2ea00700 1 --2- 172.21.15.89:0/2554593095 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d3015ddf0 0x7f9d300ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:39.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.844+0000 7f9d2e000700 1 -- 172.21.15.89:0/2554593095 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d3015ddf0 msgr2=0x7f9d300ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:39.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.844+0000 7f9d2e000700 1 --2- 172.21.15.89:0/2554593095 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d3015ddf0 0x7f9d300ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.844+0000 7f9d2e000700 1 -- 172.21.15.89:0/2554593095 >> v1:172.21.15.89:6789/0 conn(0x7f9d300d2710 legacy=0x7f9d301612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:39.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.844+0000 7f9d2e000700 1 -- 172.21.15.89:0/2554593095 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d300cdc90 con 0x7f9d3015e1d0 2024-04-02T09:09:39.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.845+0000 7f9d2e000700 1 --2- 172.21.15.89:0/2554593095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d3015e1d0 0x7f9d300d21d0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c009840 tx=0x7f9d1c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9faa385dc20ed47e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:39.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.845+0000 7f9d2d600700 1 -- 172.21.15.89:0/2554593095 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c011830 con 0x7f9d3015e1d0 2024-04-02T09:09:39.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.845+0000 7f9d2d600700 1 -- 172.21.15.89:0/2554593095 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1c011990 con 0x7f9d3015e1d0 2024-04-02T09:09:39.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.845+0000 7f9d2d600700 1 -- 172.21.15.89:0/2554593095 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c011d20 con 0x7f9d3015e1d0 2024-04-02T09:09:39.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.845+0000 7f9d35a00700 1 -- 172.21.15.89:0/2554593095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d3015e1d0 msgr2=0x7f9d300d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:39.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.845+0000 7f9d35a00700 1 --2- 172.21.15.89:0/2554593095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d3015e1d0 0x7f9d300d21d0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c009840 tx=0x7f9d1c009b50 comp rx=0 tx=0).stop 2024-04-02T09:09:39.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.846+0000 7f9d35a00700 1 -- 172.21.15.89:0/2554593095 shutdown_connections 2024-04-02T09:09:39.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.846+0000 7f9d35a00700 1 --2- 172.21.15.89:0/2554593095 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d3015ddf0 0x7f9d300ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.846+0000 7f9d35a00700 1 --2- 172.21.15.89:0/2554593095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d3015e1d0 0x7f9d300d21d0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.846+0000 7f9d35a00700 1 -- 172.21.15.89:0/2554593095 >> 172.21.15.89:0/2554593095 conn(0x7f9d3008aff0 msgr2=0x7f9d300d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:39.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.846+0000 7f9d35a00700 1 -- 172.21.15.89:0/2554593095 shutdown_connections 2024-04-02T09:09:39.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.846+0000 7f9d35a00700 1 -- 172.21.15.89:0/2554593095 wait complete. 2024-04-02T09:09:39.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.847+0000 7f9d35a00700 1 Processor -- start 2024-04-02T09:09:39.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.847+0000 7f9d35a00700 1 -- start start 2024-04-02T09:09:39.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.848+0000 7f9d35a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d300d2710 0x7f9d3009ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:39.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.848+0000 7f9d35a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d3015ddf0 0x7f9d30160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:39.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.848+0000 7f9d35a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d3015e1d0 0x7f9d301f6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:39.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.848+0000 7f9d35a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d301f3060 con 0x7f9d300d2710 2024-04-02T09:09:39.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.848+0000 7f9d35a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d300cbc20 con 0x7f9d3015e1d0 2024-04-02T09:09:39.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.848+0000 7f9d35a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3009f1c0 con 0x7f9d3015ddf0 2024-04-02T09:09:39.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.848+0000 7f9d2ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d300d2710 0x7f9d3009ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:39.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.848+0000 7f9d2ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d300d2710 0x7f9d3009ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40324/0 (socket says 172.21.15.89:40324) 2024-04-02T09:09:39.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.848+0000 7f9d2ea00700 1 -- 172.21.15.89:0/624063681 learned_addr learned my addr 172.21.15.89:0/624063681 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:39.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.849+0000 7f9d2ea00700 1 -- 172.21.15.89:0/624063681 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d3015ddf0 msgr2=0x7f9d30160fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:39.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.849+0000 7f9d2ea00700 1 --2- 172.21.15.89:0/624063681 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d3015ddf0 0x7f9d30160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.849+0000 7f9d2ea00700 1 -- 172.21.15.89:0/624063681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d3015e1d0 msgr2=0x7f9d301f6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:39.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.849+0000 7f9d2ea00700 1 --2- 172.21.15.89:0/624063681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d3015e1d0 0x7f9d301f6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:39.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.849+0000 7f9d2ea00700 1 -- 172.21.15.89:0/624063681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1c0094a0 con 0x7f9d300d2710 2024-04-02T09:09:39.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.849+0000 7f9d2ea00700 1 --2- 172.21.15.89:0/624063681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d300d2710 0x7f9d3009ec80 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000e840 tx=0x7f9d2000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:39.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.849+0000 7f9d26a00700 1 -- 172.21.15.89:0/624063681 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d20013610 con 0x7f9d300d2710 2024-04-02T09:09:39.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.849+0000 7f9d26a00700 1 -- 172.21.15.89:0/624063681 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d20013770 con 0x7f9d300d2710 2024-04-02T09:09:39.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.849+0000 7f9d35a00700 1 -- 172.21.15.89:0/624063681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d300cb010 con 0x7f9d300d2710 2024-04-02T09:09:39.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.849+0000 7f9d26a00700 1 -- 172.21.15.89:0/624063681 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d20013b20 con 0x7f9d300d2710 2024-04-02T09:09:39.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.849+0000 7f9d35a00700 1 -- 172.21.15.89:0/624063681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d300cb5d0 con 0x7f9d300d2710 2024-04-02T09:09:39.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.851+0000 7f9d35a00700 1 -- 172.21.15.89:0/624063681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d30067720 con 0x7f9d300d2710 2024-04-02T09:09:39.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.852+0000 7f9d26a00700 1 -- 172.21.15.89:0/624063681 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9d20013c80 con 0x7f9d300d2710 2024-04-02T09:09:39.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.853+0000 7f9d26a00700 1 --2- 172.21.15.89:0/624063681 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9d1006adb0 0x7f9d1006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:39.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.853+0000 7f9d26a00700 1 -- 172.21.15.89:0/624063681 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f9d20089100 con 0x7f9d300d2710 2024-04-02T09:09:39.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.853+0000 7f9d2e000700 1 --2- 172.21.15.89:0/624063681 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9d1006adb0 0x7f9d1006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:39.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.853+0000 7f9d2e000700 1 --2- 172.21.15.89:0/624063681 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9d1006adb0 0x7f9d1006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9d300cf910 tx=0x7f9d1c010000 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:39.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:39.857+0000 7f9d26a00700 1 -- 172.21.15.89:0/624063681 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9d20054240 con 0x7f9d300d2710 2024-04-02T09:09:40.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.058+0000 7f9d35a00700 1 -- 172.21.15.89:0/624063681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9d300d3050 con 0x7f9d300d2710 2024-04-02T09:09:40.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.677+0000 7f9d26a00700 1 -- 172.21.15.89:0/624063681 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v202) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9d20057860 con 0x7f9d300d2710 2024-04-02T09:09:40.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.681+0000 7f9d35a00700 1 -- 172.21.15.89:0/624063681 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9d1006adb0 msgr2=0x7f9d1006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:40.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.681+0000 7f9d35a00700 1 --2- 172.21.15.89:0/624063681 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9d1006adb0 0x7f9d1006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9d300cf910 tx=0x7f9d1c010000 comp rx=0 tx=0).stop 2024-04-02T09:09:40.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.681+0000 7f9d35a00700 1 -- 172.21.15.89:0/624063681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d300d2710 msgr2=0x7f9d3009ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:40.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.681+0000 7f9d35a00700 1 --2- 172.21.15.89:0/624063681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d300d2710 0x7f9d3009ec80 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000e840 tx=0x7f9d2000ec00 comp rx=0 tx=0).stop 2024-04-02T09:09:40.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.682+0000 7f9d35a00700 1 -- 172.21.15.89:0/624063681 shutdown_connections 2024-04-02T09:09:40.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.682+0000 7f9d35a00700 1 --2- 172.21.15.89:0/624063681 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9d1006adb0 0x7f9d1006d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:40.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.682+0000 7f9d35a00700 1 --2- 172.21.15.89:0/624063681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d3015e1d0 0x7f9d301f6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:40.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.682+0000 7f9d35a00700 1 --2- 172.21.15.89:0/624063681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d300d2710 0x7f9d3009ec80 secure :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000e840 tx=0x7f9d2000ec00 comp rx=0 tx=0).stop 2024-04-02T09:09:40.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.682+0000 7f9d35a00700 1 --2- 172.21.15.89:0/624063681 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d3015ddf0 0x7f9d30160fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:40.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.682+0000 7f9d35a00700 1 -- 172.21.15.89:0/624063681 >> 172.21.15.89:0/624063681 conn(0x7f9d3008aff0 msgr2=0x7f9d300d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:40.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.682+0000 7f9d35a00700 1 -- 172.21.15.89:0/624063681 shutdown_connections 2024-04-02T09:09:40.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.683+0000 7f9d35a00700 1 -- 172.21.15.89:0/624063681 wait complete. 2024-04-02T09:09:40.683 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T09:09:40.702 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T09:09:40.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.905+0000 7f5e01200700 1 Processor -- start 2024-04-02T09:09:40.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.905+0000 7f5e01200700 1 -- start start 2024-04-02T09:09:40.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.906+0000 7f5e01200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5dfc0cd5c0 0x7f5dfc15e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:40.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.906+0000 7f5e01200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dfc06f310 0x7f5dfc06f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:40.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.906+0000 7f5e01200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dfc15eda0 con 0x7f5dfc15cfe0 2024-04-02T09:09:40.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.906+0000 7f5e01200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dfc06fce0 con 0x7f5dfc0cd5c0 2024-04-02T09:09:40.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.906+0000 7f5e01200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc099dd0 con 0x7f5dfc06f310 2024-04-02T09:09:40.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.906+0000 7f5df9e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5dfc0cd5c0 0x7f5dfc15e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:40.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.906+0000 7f5df3e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dfc06f310 0x7f5dfc06f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:40.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.907+0000 7f5df9e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5dfc0cd5c0 0x7f5dfc15e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:52010/0 (socket says 172.21.15.89:52010) 2024-04-02T09:09:40.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.907+0000 7f5df3e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dfc06f310 0x7f5dfc06f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:40112/0 (socket says 172.21.15.89:40112) 2024-04-02T09:09:40.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.907+0000 7f5df9e00700 1 -- 172.21.15.89:0/933619724 learned_addr learned my addr 172.21.15.89:0/933619724 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:40.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.907+0000 7f5dfa800700 1 --1- 172.21.15.89:0/933619724 >> v1:172.21.15.113:6789/0 conn(0x7f5dfc15cfe0 0x7f5dfc15d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:54428/0 (socket says 172.21.15.89:54428) 2024-04-02T09:09:40.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.907+0000 7f5df9400700 1 -- 172.21.15.89:0/933619724 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179821595 0 0) 0x7f5df4001490 con 0x7f5dfc15cfe0 2024-04-02T09:09:40.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.907+0000 7f5df9400700 1 -- 172.21.15.89:0/933619724 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de00035b0 con 0x7f5dfc15cfe0 2024-04-02T09:09:40.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.907+0000 7f5df9e00700 1 -- 172.21.15.89:0/933619724 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dfc06f310 msgr2=0x7f5dfc06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:40.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.907+0000 7f5df9e00700 1 --2- 172.21.15.89:0/933619724 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dfc06f310 0x7f5dfc06f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:40.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.907+0000 7f5df9e00700 1 -- 172.21.15.89:0/933619724 >> v1:172.21.15.113:6789/0 conn(0x7f5dfc15cfe0 legacy=0x7f5dfc15d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:40.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.907+0000 7f5df9e00700 1 -- 172.21.15.89:0/933619724 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dfc09a0b0 con 0x7f5dfc0cd5c0 2024-04-02T09:09:40.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.907+0000 7f5df3e00700 1 --2- 172.21.15.89:0/933619724 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dfc06f310 0x7f5dfc06f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:09:40.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.907+0000 7f5df9e00700 1 --2- 172.21.15.89:0/933619724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dfc0cd5c0 0x7f5dfc15e860 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0098f0 tx=0x7f5dec009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f40e7d7376662a5d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:40.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.908+0000 7f5df9400700 1 -- 172.21.15.89:0/933619724 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec011830 con 0x7f5dfc0cd5c0 2024-04-02T09:09:40.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.908+0000 7f5df9400700 1 -- 172.21.15.89:0/933619724 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dec011990 con 0x7f5dfc0cd5c0 2024-04-02T09:09:40.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.908+0000 7f5df9400700 1 -- 172.21.15.89:0/933619724 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec011d20 con 0x7f5dfc0cd5c0 2024-04-02T09:09:40.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.908+0000 7f5e01200700 1 -- 172.21.15.89:0/933619724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dfc0cd5c0 msgr2=0x7f5dfc15e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:40.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.908+0000 7f5e01200700 1 --2- 172.21.15.89:0/933619724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dfc0cd5c0 0x7f5dfc15e860 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0098f0 tx=0x7f5dec009c00 comp rx=0 tx=0).stop 2024-04-02T09:09:40.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.908+0000 7f5e01200700 1 -- 172.21.15.89:0/933619724 shutdown_connections 2024-04-02T09:09:40.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.908+0000 7f5e01200700 1 --2- 172.21.15.89:0/933619724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dfc0cd5c0 0x7f5dfc15e860 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:40.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.908+0000 7f5e01200700 1 --2- 172.21.15.89:0/933619724 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dfc06f310 0x7f5dfc06f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:40.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.908+0000 7f5e01200700 1 -- 172.21.15.89:0/933619724 >> 172.21.15.89:0/933619724 conn(0x7f5dfc0d1370 msgr2=0x7f5dfc0d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:40.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.908+0000 7f5e01200700 1 -- 172.21.15.89:0/933619724 shutdown_connections 2024-04-02T09:09:40.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.908+0000 7f5e01200700 1 -- 172.21.15.89:0/933619724 wait complete. 2024-04-02T09:09:40.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.910+0000 7f5e01200700 1 Processor -- start 2024-04-02T09:09:40.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.910+0000 7f5e01200700 1 -- start start 2024-04-02T09:09:40.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.911+0000 7f5e01200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dfc06f310 0x7f5dfc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:40.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.911+0000 7f5e01200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dfc0cd5c0 0x7f5dfc1514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:40.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.911+0000 7f5e01200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dfc15cfe0 0x7f5dfc0c6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:40.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.911+0000 7f5e01200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc0b6660 con 0x7f5dfc15cfe0 2024-04-02T09:09:40.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.911+0000 7f5e01200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc0b5a50 con 0x7f5dfc0cd5c0 2024-04-02T09:09:40.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.911+0000 7f5e01200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc0b5d30 con 0x7f5dfc06f310 2024-04-02T09:09:40.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.911+0000 7f5dfa800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dfc15cfe0 0x7f5dfc0c6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:40.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.911+0000 7f5dfa800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dfc15cfe0 0x7f5dfc0c6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40330/0 (socket says 172.21.15.89:40330) 2024-04-02T09:09:40.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.911+0000 7f5dfa800700 1 -- 172.21.15.89:0/901129753 learned_addr learned my addr 172.21.15.89:0/901129753 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:40.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.911+0000 7f5df9e00700 1 --2- 172.21.15.89:0/901129753 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dfc06f310 0x7f5dfc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:40.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.911+0000 7f5df3e00700 1 --2- 172.21.15.89:0/901129753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dfc0cd5c0 0x7f5dfc1514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:40.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.912+0000 7f5dfa800700 1 -- 172.21.15.89:0/901129753 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dfc06f310 msgr2=0x7f5dfc06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:40.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.912+0000 7f5dfa800700 1 --2- 172.21.15.89:0/901129753 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dfc06f310 0x7f5dfc06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:40.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.912+0000 7f5dfa800700 1 -- 172.21.15.89:0/901129753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dfc0cd5c0 msgr2=0x7f5dfc1514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:40.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.912+0000 7f5dfa800700 1 --2- 172.21.15.89:0/901129753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dfc0cd5c0 0x7f5dfc1514e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:40.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.912+0000 7f5dfa800700 1 -- 172.21.15.89:0/901129753 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de40094a0 con 0x7f5dfc15cfe0 2024-04-02T09:09:40.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.912+0000 7f5dfa800700 1 --2- 172.21.15.89:0/901129753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dfc15cfe0 0x7f5dfc0c6590 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f5df400cbb0 tx=0x7f5df400cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:40.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.912+0000 7f5df2000700 1 -- 172.21.15.89:0/901129753 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df400aaa0 con 0x7f5dfc15cfe0 2024-04-02T09:09:40.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.912+0000 7f5df2000700 1 -- 172.21.15.89:0/901129753 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df400ac00 con 0x7f5dfc15cfe0 2024-04-02T09:09:40.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.912+0000 7f5df2000700 1 -- 172.21.15.89:0/901129753 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df4007070 con 0x7f5dfc15cfe0 2024-04-02T09:09:40.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.912+0000 7f5e01200700 1 -- 172.21.15.89:0/901129753 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dec009550 con 0x7f5dfc15cfe0 2024-04-02T09:09:40.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.912+0000 7f5e01200700 1 -- 172.21.15.89:0/901129753 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dfc0b62f0 con 0x7f5dfc15cfe0 2024-04-02T09:09:40.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.914+0000 7f5e01200700 1 -- 172.21.15.89:0/901129753 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dfc067720 con 0x7f5dfc15cfe0 2024-04-02T09:09:40.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.915+0000 7f5df2000700 1 -- 172.21.15.89:0/901129753 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5df40071d0 con 0x7f5dfc15cfe0 2024-04-02T09:09:40.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.916+0000 7f5df2000700 1 --2- 172.21.15.89:0/901129753 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dd806adb0 0x7f5dd806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:40.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.916+0000 7f5df2000700 1 -- 172.21.15.89:0/901129753 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f5df40893e0 con 0x7f5dfc15cfe0 2024-04-02T09:09:40.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.916+0000 7f5df9e00700 1 --2- 172.21.15.89:0/901129753 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dd806adb0 0x7f5dd806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:40.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.917+0000 7f5df9e00700 1 --2- 172.21.15.89:0/901129753 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dd806adb0 0x7f5dd806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5dec000c00 tx=0x7f5dec008030 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:40.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:40.920+0000 7f5df2000700 1 -- 172.21.15.89:0/901129753 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5df40545b0 con 0x7f5dfc15cfe0 2024-04-02T09:09:41.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.120+0000 7f5e01200700 1 -- 172.21.15.89:0/901129753 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5dfc0b67a0 con 0x7f5dfc15cfe0 2024-04-02T09:09:41.573 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:09:41.578 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:09:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.682+0000 7f5df2000700 1 -- 172.21.15.89:0/901129753 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v203) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5df4057bd0 con 0x7f5dfc15cfe0 2024-04-02T09:09:41.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.690+0000 7f5e01200700 1 -- 172.21.15.89:0/901129753 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dd806adb0 msgr2=0x7f5dd806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:41.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.690+0000 7f5e01200700 1 --2- 172.21.15.89:0/901129753 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dd806adb0 0x7f5dd806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5dec000c00 tx=0x7f5dec008030 comp rx=0 tx=0).stop 2024-04-02T09:09:41.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.690+0000 7f5e01200700 1 -- 172.21.15.89:0/901129753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dfc15cfe0 msgr2=0x7f5dfc0c6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:41.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.690+0000 7f5e01200700 1 --2- 172.21.15.89:0/901129753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dfc15cfe0 0x7f5dfc0c6590 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f5df400cbb0 tx=0x7f5df400cec0 comp rx=0 tx=0).stop 2024-04-02T09:09:41.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.691+0000 7f5e01200700 1 -- 172.21.15.89:0/901129753 shutdown_connections 2024-04-02T09:09:41.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.691+0000 7f5e01200700 1 --2- 172.21.15.89:0/901129753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dfc15cfe0 0x7f5dfc0c6590 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:41.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.691+0000 7f5e01200700 1 --2- 172.21.15.89:0/901129753 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dfc06f310 0x7f5dfc06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:41.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.691+0000 7f5e01200700 1 --2- 172.21.15.89:0/901129753 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dd806adb0 0x7f5dd806d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:41.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.691+0000 7f5e01200700 1 --2- 172.21.15.89:0/901129753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dfc0cd5c0 0x7f5dfc1514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:41.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.691+0000 7f5e01200700 1 -- 172.21.15.89:0/901129753 >> 172.21.15.89:0/901129753 conn(0x7f5dfc0d1370 msgr2=0x7f5dfc15c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:41.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.692+0000 7f5e01200700 1 -- 172.21.15.89:0/901129753 shutdown_connections 2024-04-02T09:09:41.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.692+0000 7f5e01200700 1 -- 172.21.15.89:0/901129753 wait complete. 2024-04-02T09:09:41.695 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T09:09:41.717 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T09:09:41.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.955+0000 7fc165a00700 1 Processor -- start 2024-04-02T09:09:41.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.955+0000 7fc165a00700 1 -- start start 2024-04-02T09:09:41.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.956+0000 7fc165a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc160160d70 0x7fc1600b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:41.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.956+0000 7fc165a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc16015ef70 0x7fc16015e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:41.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.956+0000 7fc165a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc160161150 con 0x7fc16006a790 2024-04-02T09:09:41.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.956+0000 7fc165a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1600b4cd0 con 0x7fc16015ef70 2024-04-02T09:09:41.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.956+0000 7fc165a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16006dba0 con 0x7fc160160d70 2024-04-02T09:09:41.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.956+0000 7fc157e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc16015ef70 0x7fc16015e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:41.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.956+0000 7fc157e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc16015ef70 0x7fc16015e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40340/0 (socket says 172.21.15.89:40340) 2024-04-02T09:09:41.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.956+0000 7fc157e00700 1 -- 172.21.15.89:0/4105963289 learned_addr learned my addr 172.21.15.89:0/4105963289 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:41.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.956+0000 7fc15ea00700 1 --2- 172.21.15.89:0/4105963289 >> v2:172.21.15.113:3300/0 conn(0x7fc160160d70 0x7fc1600b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:41.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.957+0000 7fc157e00700 1 -- 172.21.15.89:0/4105963289 >> v2:172.21.15.113:3300/0 conn(0x7fc160160d70 msgr2=0x7fc1600b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:41.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.957+0000 7fc157e00700 1 --2- 172.21.15.89:0/4105963289 >> v2:172.21.15.113:3300/0 conn(0x7fc160160d70 0x7fc1600b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:41.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.957+0000 7fc157e00700 1 -- 172.21.15.89:0/4105963289 >> v1:172.21.15.113:6789/0 conn(0x7fc16006a790 legacy=0x7fc1601613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:41.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.957+0000 7fc157e00700 1 -- 172.21.15.89:0/4105963289 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16015ed30 con 0x7fc16015ef70 2024-04-02T09:09:41.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.957+0000 7fc157e00700 1 --2- 172.21.15.89:0/4105963289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc16015ef70 0x7fc16015e7f0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7fc16006c880 tx=0x7fc148009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=633184848321c9cb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:41.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.957+0000 7fc15e000700 1 -- 172.21.15.89:0/4105963289 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc148011830 con 0x7fc16015ef70 2024-04-02T09:09:41.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.957+0000 7fc15e000700 1 -- 172.21.15.89:0/4105963289 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc148011990 con 0x7fc16015ef70 2024-04-02T09:09:41.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.957+0000 7fc15e000700 1 -- 172.21.15.89:0/4105963289 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc148011d20 con 0x7fc16015ef70 2024-04-02T09:09:41.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.958+0000 7fc165a00700 1 -- 172.21.15.89:0/4105963289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc16015ef70 msgr2=0x7fc16015e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:41.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.958+0000 7fc165a00700 1 --2- 172.21.15.89:0/4105963289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc16015ef70 0x7fc16015e7f0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7fc16006c880 tx=0x7fc148009bd0 comp rx=0 tx=0).stop 2024-04-02T09:09:41.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.958+0000 7fc165a00700 1 -- 172.21.15.89:0/4105963289 shutdown_connections 2024-04-02T09:09:41.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.958+0000 7fc165a00700 1 --2- 172.21.15.89:0/4105963289 >> v2:172.21.15.113:3300/0 conn(0x7fc160160d70 0x7fc1600b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:41.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.958+0000 7fc165a00700 1 --2- 172.21.15.89:0/4105963289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc16015ef70 0x7fc16015e7f0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:41.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.958+0000 7fc165a00700 1 -- 172.21.15.89:0/4105963289 >> 172.21.15.89:0/4105963289 conn(0x7fc1600b5720 msgr2=0x7fc1600b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:41.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.958+0000 7fc165a00700 1 -- 172.21.15.89:0/4105963289 shutdown_connections 2024-04-02T09:09:41.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.958+0000 7fc165a00700 1 -- 172.21.15.89:0/4105963289 wait complete. 2024-04-02T09:09:41.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.960+0000 7fc165a00700 1 Processor -- start 2024-04-02T09:09:41.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.960+0000 7fc165a00700 1 -- start start 2024-04-02T09:09:41.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.960+0000 7fc165a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc16006a790 0x7fc160146c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:41.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc165a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc16015ef70 0x7fc16015c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:41.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc165a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc160160d70 0x7fc16009d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:41.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc165a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc160155e20 con 0x7fc16015ef70 2024-04-02T09:09:41.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc165a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc160147170 con 0x7fc160160d70 2024-04-02T09:09:41.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc165a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1601551a0 con 0x7fc16006a790 2024-04-02T09:09:41.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc157e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc16015ef70 0x7fc16015c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:41.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc157e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc16015ef70 0x7fc16015c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40344/0 (socket says 172.21.15.89:40344) 2024-04-02T09:09:41.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc157e00700 1 -- 172.21.15.89:0/3375293462 learned_addr learned my addr 172.21.15.89:0/3375293462 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:41.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc15ea00700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc16006a790 0x7fc160146c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:41.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc15f400700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc160160d70 0x7fc16009d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:41.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc157e00700 1 -- 172.21.15.89:0/3375293462 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc16006a790 msgr2=0x7fc160146c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:41.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc157e00700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc16006a790 0x7fc160146c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:41.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc157e00700 1 -- 172.21.15.89:0/3375293462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc160160d70 msgr2=0x7fc16009d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:41.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc157e00700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc160160d70 0x7fc16009d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:41.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc157e00700 1 -- 172.21.15.89:0/3375293462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc148009550 con 0x7fc16015ef70 2024-04-02T09:09:41.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc15ea00700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc16006a790 0x7fc160146c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:41.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.961+0000 7fc15f400700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc160160d70 0x7fc16009d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:41.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.962+0000 7fc157e00700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc16015ef70 0x7fc16015c8c0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fc148000c00 tx=0x7fc148004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:41.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.962+0000 7fc156a00700 1 -- 172.21.15.89:0/3375293462 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc148004570 con 0x7fc16015ef70 2024-04-02T09:09:41.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.962+0000 7fc156a00700 1 -- 172.21.15.89:0/3375293462 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14801f070 con 0x7fc16015ef70 2024-04-02T09:09:41.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.962+0000 7fc165a00700 1 -- 172.21.15.89:0/3375293462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc160155480 con 0x7fc16015ef70 2024-04-02T09:09:41.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.962+0000 7fc156a00700 1 -- 172.21.15.89:0/3375293462 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1480038e0 con 0x7fc16015ef70 2024-04-02T09:09:41.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.962+0000 7fc165a00700 1 -- 172.21.15.89:0/3375293462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc160155a40 con 0x7fc16015ef70 2024-04-02T09:09:41.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.963+0000 7fc165a00700 1 -- 172.21.15.89:0/3375293462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc160155f60 con 0x7fc16015ef70 2024-04-02T09:09:41.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.965+0000 7fc156a00700 1 -- 172.21.15.89:0/3375293462 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc148021030 con 0x7fc16015ef70 2024-04-02T09:09:41.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.966+0000 7fc156a00700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc13c06adb0 0x7fc13c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:41.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.966+0000 7fc156a00700 1 -- 172.21.15.89:0/3375293462 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc14808ac50 con 0x7fc16015ef70 2024-04-02T09:09:41.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.966+0000 7fc15ea00700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc13c06adb0 0x7fc13c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:41.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.967+0000 7fc15ea00700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc13c06adb0 0x7fc13c06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc150009ac0 tx=0x7fc150008040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:41.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:41.970+0000 7fc156a00700 1 -- 172.21.15.89:0/3375293462 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc148055d30 con 0x7fc16015ef70 2024-04-02T09:09:42.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.218+0000 7fc165a00700 1 -- 172.21.15.89:0/3375293462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc160155f60 con 0x7fc16015ef70 2024-04-02T09:09:42.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.225+0000 7fc156a00700 1 -- 172.21.15.89:0/3375293462 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v93) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc160155f60 con 0x7fc16015ef70 2024-04-02T09:09:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.230+0000 7fc165a00700 1 -- 172.21.15.89:0/3375293462 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc13c06adb0 msgr2=0x7fc13c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.230+0000 7fc165a00700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc13c06adb0 0x7fc13c06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc150009ac0 tx=0x7fc150008040 comp rx=0 tx=0).stop 2024-04-02T09:09:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.230+0000 7fc165a00700 1 -- 172.21.15.89:0/3375293462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc16015ef70 msgr2=0x7fc16015c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:42.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.230+0000 7fc165a00700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc16015ef70 0x7fc16015c8c0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fc148000c00 tx=0x7fc148004750 comp rx=0 tx=0).stop 2024-04-02T09:09:42.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.231+0000 7fc165a00700 1 -- 172.21.15.89:0/3375293462 shutdown_connections 2024-04-02T09:09:42.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.231+0000 7fc165a00700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc13c06adb0 0x7fc13c06d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.231+0000 7fc165a00700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc160160d70 0x7fc16009d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.231+0000 7fc165a00700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc16006a790 0x7fc160146c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.231+0000 7fc165a00700 1 --2- 172.21.15.89:0/3375293462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc16015ef70 0x7fc16015c8c0 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.231+0000 7fc165a00700 1 -- 172.21.15.89:0/3375293462 >> 172.21.15.89:0/3375293462 conn(0x7fc1600b5720 msgr2=0x7fc160152e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:42.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.235+0000 7fc165a00700 1 -- 172.21.15.89:0/3375293462 shutdown_connections 2024-04-02T09:09:42.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.235+0000 7fc165a00700 1 -- 172.21.15.89:0/3375293462 wait complete. 2024-04-02T09:09:42.240 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 24 and data pool 25 2024-04-02T09:09:42.264 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:09:42.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.456+0000 7f956c200700 1 Processor -- start 2024-04-02T09:09:42.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.456+0000 7f956c200700 1 -- start start 2024-04-02T09:09:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.458+0000 7f956c200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f956406a790 0x7f956406ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.458+0000 7f956c200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f95640cebd0 0x7f95640cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.458+0000 7f956c200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956406f850 con 0x7f95640afbe0 2024-04-02T09:09:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.458+0000 7f956c200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f956406e3f0 con 0x7f95640cebd0 2024-04-02T09:09:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.458+0000 7f956c200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956406b0b0 con 0x7f956406a790 2024-04-02T09:09:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.458+0000 7f956a200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f95640cebd0 0x7f95640cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.458+0000 7f956a200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f95640cebd0 0x7f95640cf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40354/0 (socket says 172.21.15.89:40354) 2024-04-02T09:09:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.458+0000 7f956a200700 1 -- 172.21.15.89:0/3288473155 learned_addr learned my addr 172.21.15.89:0/3288473155 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.458+0000 7f9569800700 1 --2- 172.21.15.89:0/3288473155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f956406a790 0x7f956406ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.458+0000 7f956a200700 1 -- 172.21.15.89:0/3288473155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f956406a790 msgr2=0x7f956406ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.458+0000 7f956a200700 1 --2- 172.21.15.89:0/3288473155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f956406a790 0x7f956406ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.458+0000 7f956a200700 1 -- 172.21.15.89:0/3288473155 >> v1:172.21.15.113:6789/0 conn(0x7f95640afbe0 legacy=0x7f956406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:42.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.458+0000 7f956a200700 1 -- 172.21.15.89:0/3288473155 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956406fad0 con 0x7f95640cebd0 2024-04-02T09:09:42.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.459+0000 7f956a200700 1 --2- 172.21.15.89:0/3288473155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95640cebd0 0x7f95640cf040 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f955c009e50 tx=0x7f955c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2511fe986c7c4ae8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.459+0000 7f9563e00700 1 -- 172.21.15.89:0/3288473155 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c00e070 con 0x7f95640cebd0 2024-04-02T09:09:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.459+0000 7f9563e00700 1 -- 172.21.15.89:0/3288473155 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955c00b9c0 con 0x7f95640cebd0 2024-04-02T09:09:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.459+0000 7f9563e00700 1 -- 172.21.15.89:0/3288473155 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c00bd70 con 0x7f95640cebd0 2024-04-02T09:09:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.459+0000 7f956c200700 1 -- 172.21.15.89:0/3288473155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95640cebd0 msgr2=0x7f95640cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.459+0000 7f956c200700 1 --2- 172.21.15.89:0/3288473155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95640cebd0 0x7f95640cf040 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f955c009e50 tx=0x7f955c00b330 comp rx=0 tx=0).stop 2024-04-02T09:09:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.460+0000 7f956c200700 1 -- 172.21.15.89:0/3288473155 shutdown_connections 2024-04-02T09:09:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.460+0000 7f956c200700 1 --2- 172.21.15.89:0/3288473155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f956406a790 0x7f956406ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.460+0000 7f956c200700 1 --2- 172.21.15.89:0/3288473155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95640cebd0 0x7f95640cf040 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.460+0000 7f956c200700 1 -- 172.21.15.89:0/3288473155 >> 172.21.15.89:0/3288473155 conn(0x7f95640b5ab0 msgr2=0x7f95640b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.460+0000 7f956c200700 1 -- 172.21.15.89:0/3288473155 shutdown_connections 2024-04-02T09:09:42.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.460+0000 7f956c200700 1 -- 172.21.15.89:0/3288473155 wait complete. 2024-04-02T09:09:42.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.462+0000 7f956c200700 1 Processor -- start 2024-04-02T09:09:42.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.462+0000 7f956c200700 1 -- start start 2024-04-02T09:09:42.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.463+0000 7f956c200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f956406a790 0x7f956414fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.463+0000 7f956c200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95640afbe0 0x7f956409a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.463+0000 7f956c200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95640cebd0 0x7f95640cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.463+0000 7f956c200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956415c050 con 0x7f95640afbe0 2024-04-02T09:09:42.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.463+0000 7f956c200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956409a980 con 0x7f95640cebd0 2024-04-02T09:09:42.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.463+0000 7f956c200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956415b440 con 0x7f956406a790 2024-04-02T09:09:42.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.464+0000 7f9568e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95640afbe0 0x7f956409a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.464+0000 7f9568e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95640afbe0 0x7f956409a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40364/0 (socket says 172.21.15.89:40364) 2024-04-02T09:09:42.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.464+0000 7f9569800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f956406a790 0x7f956414fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.464+0000 7f9568e00700 1 -- 172.21.15.89:0/2066654959 learned_addr learned my addr 172.21.15.89:0/2066654959 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:42.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.464+0000 7f956a200700 1 --2- 172.21.15.89:0/2066654959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95640cebd0 0x7f95640cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.464+0000 7f9568e00700 1 -- 172.21.15.89:0/2066654959 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f956406a790 msgr2=0x7f956414fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:42.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.464+0000 7f9568e00700 1 --2- 172.21.15.89:0/2066654959 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f956406a790 0x7f956414fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.464+0000 7f9568e00700 1 -- 172.21.15.89:0/2066654959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95640cebd0 msgr2=0x7f95640cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:42.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.464+0000 7f9568e00700 1 --2- 172.21.15.89:0/2066654959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95640cebd0 0x7f95640cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.464+0000 7f9568e00700 1 -- 172.21.15.89:0/2066654959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c009b30 con 0x7f95640afbe0 2024-04-02T09:09:42.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.464+0000 7f9568e00700 1 --2- 172.21.15.89:0/2066654959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95640afbe0 0x7f956409a440 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f955000c450 tx=0x7f955000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:42.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.465+0000 7f9561600700 1 -- 172.21.15.89:0/2066654959 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95500052d0 con 0x7f95640afbe0 2024-04-02T09:09:42.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.465+0000 7f9561600700 1 -- 172.21.15.89:0/2066654959 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9550005430 con 0x7f95640afbe0 2024-04-02T09:09:42.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.465+0000 7f9561600700 1 -- 172.21.15.89:0/2066654959 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95500038c0 con 0x7f95640afbe0 2024-04-02T09:09:42.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.465+0000 7f956c200700 1 -- 172.21.15.89:0/2066654959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956415b720 con 0x7f95640afbe0 2024-04-02T09:09:42.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.465+0000 7f956c200700 1 -- 172.21.15.89:0/2066654959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956415bce0 con 0x7f95640afbe0 2024-04-02T09:09:42.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.466+0000 7f956c200700 1 -- 172.21.15.89:0/2066654959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95640cd960 con 0x7f95640afbe0 2024-04-02T09:09:42.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.467+0000 7f9561600700 1 -- 172.21.15.89:0/2066654959 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9550003a20 con 0x7f95640afbe0 2024-04-02T09:09:42.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.468+0000 7f9561600700 1 --2- 172.21.15.89:0/2066654959 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f954406adb0 0x7f954406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.468+0000 7f9561600700 1 -- 172.21.15.89:0/2066654959 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f95500888e0 con 0x7f95640afbe0 2024-04-02T09:09:42.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.468+0000 7f9569800700 1 --2- 172.21.15.89:0/2066654959 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f954406adb0 0x7f954406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.469+0000 7f9569800700 1 --2- 172.21.15.89:0/2066654959 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f954406adb0 0x7f954406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f95640680d0 tx=0x7f9558008040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:42.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.472+0000 7f9561600700 1 -- 172.21.15.89:0/2066654959 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9550053880 con 0x7f95640afbe0 2024-04-02T09:09:42.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.672+0000 7f956c200700 1 -- 172.21.15.89:0/2066654959 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9564067720 con 0x7f95640afbe0 2024-04-02T09:09:42.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.673+0000 7f9561600700 1 -- 172.21.15.89:0/2066654959 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) v1 ==== 74+0+16757 (secure 0 0 0) 0x7f9550017090 con 0x7f95640afbe0 2024-04-02T09:09:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:42.672 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":204,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:09:42.219865+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:09:40.103721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-04-02T09:09:41.165944+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:09:42.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.676+0000 7f956c200700 1 -- 172.21.15.89:0/2066654959 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f954406adb0 msgr2=0x7f954406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:42.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.676+0000 7f956c200700 1 --2- 172.21.15.89:0/2066654959 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f954406adb0 0x7f954406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f95640680d0 tx=0x7f9558008040 comp rx=0 tx=0).stop 2024-04-02T09:09:42.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.676+0000 7f956c200700 1 -- 172.21.15.89:0/2066654959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95640afbe0 msgr2=0x7f956409a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:42.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.676+0000 7f956c200700 1 --2- 172.21.15.89:0/2066654959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95640afbe0 0x7f956409a440 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f955000c450 tx=0x7f955000c760 comp rx=0 tx=0).stop 2024-04-02T09:09:42.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.676+0000 7f956c200700 1 -- 172.21.15.89:0/2066654959 shutdown_connections 2024-04-02T09:09:42.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.676+0000 7f956c200700 1 --2- 172.21.15.89:0/2066654959 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f954406adb0 0x7f954406d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.676+0000 7f956c200700 1 --2- 172.21.15.89:0/2066654959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95640cebd0 0x7f95640cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.676+0000 7f956c200700 1 --2- 172.21.15.89:0/2066654959 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f956406a790 0x7f956414fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.676+0000 7f956c200700 1 --2- 172.21.15.89:0/2066654959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95640afbe0 0x7f956409a440 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.676+0000 7f956c200700 1 -- 172.21.15.89:0/2066654959 >> 172.21.15.89:0/2066654959 conn(0x7f95640b5ab0 msgr2=0x7f95640b2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:42.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.676+0000 7f956c200700 1 -- 172.21.15.89:0/2066654959 shutdown_connections 2024-04-02T09:09:42.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.676+0000 7f956c200700 1 -- 172.21.15.89:0/2066654959 wait complete. 2024-04-02T09:09:42.694 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:09:42.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.885+0000 7fcbe4c00700 1 Processor -- start 2024-04-02T09:09:42.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.885+0000 7fcbe4c00700 1 -- start start 2024-04-02T09:09:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.887+0000 7fcbe4c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcbe00cb3f0 0x7fcbe00ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.887+0000 7fcbe4c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbe00ac600 0x7fcbe006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.887+0000 7fcbe4c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe015fd50 con 0x7fcbe015b5d0 2024-04-02T09:09:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.887+0000 7fcbe4c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe015b9b0 con 0x7fcbe00cb3f0 2024-04-02T09:09:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.887+0000 7fcbe4c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe00aca50 con 0x7fcbe00ac600 2024-04-02T09:09:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.887+0000 7fcbdde00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcbe015b5d0 0x7fcbe00ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36512/0 (socket says 172.21.15.89:36512) 2024-04-02T09:09:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.887+0000 7fcbdde00700 1 -- 172.21.15.89:0/1924968184 learned_addr learned my addr 172.21.15.89:0/1924968184 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.887+0000 7fcbdd400700 1 --2- 172.21.15.89:0/1924968184 >> v2:172.21.15.113:3300/0 conn(0x7fcbe00cb3f0 0x7fcbe00ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.887+0000 7fcbde800700 1 --2- 172.21.15.89:0/1924968184 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbe00ac600 0x7fcbe006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.887+0000 7fcbdca00700 1 -- 172.21.15.89:0/1924968184 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340737098 0 0) 0x7fcbe015fd50 con 0x7fcbe015b5d0 2024-04-02T09:09:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.887+0000 7fcbdca00700 1 -- 172.21.15.89:0/1924968184 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc40035b0 con 0x7fcbe015b5d0 2024-04-02T09:09:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdca00700 1 -- 172.21.15.89:0/1924968184 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3363356469 0 0) 0x7fcbc40035b0 con 0x7fcbe015b5d0 2024-04-02T09:09:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdca00700 1 -- 172.21.15.89:0/1924968184 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbe00ac600 msgr2=0x7fcbe006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdca00700 1 --2- 172.21.15.89:0/1924968184 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbe00ac600 0x7fcbe006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdca00700 1 -- 172.21.15.89:0/1924968184 >> v2:172.21.15.113:3300/0 conn(0x7fcbe00cb3f0 msgr2=0x7fcbe00ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdca00700 1 --2- 172.21.15.89:0/1924968184 >> v2:172.21.15.113:3300/0 conn(0x7fcbe00cb3f0 0x7fcbe00ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdca00700 1 -- 172.21.15.89:0/1924968184 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe015f140 con 0x7fcbe015b5d0 2024-04-02T09:09:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdca00700 1 -- 172.21.15.89:0/1924968184 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fcbd0001670 con 0x7fcbe015b5d0 2024-04-02T09:09:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdca00700 1 -- 172.21.15.89:0/1924968184 >> v1:172.21.15.89:6789/0 conn(0x7fcbe015b5d0 legacy=0x7fcbe00ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdca00700 1 --2- 172.21.15.89:0/1924968184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbc4003a00 0x7fcbc4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdca00700 1 --2- 172.21.15.89:0/1924968184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbc4004850 0x7fcbc4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdca00700 1 --2- 172.21.15.89:0/1924968184 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbe00ac600 0x7fcbc4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdca00700 1 -- 172.21.15.89:0/1924968184 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe015b9b0 con 0x7fcbc4003a00 2024-04-02T09:09:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdca00700 1 -- 172.21.15.89:0/1924968184 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe00aca50 con 0x7fcbc4004850 2024-04-02T09:09:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdca00700 1 -- 172.21.15.89:0/1924968184 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc400ba10 con 0x7fcbe00ac600 2024-04-02T09:09:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdde00700 1 --2- 172.21.15.89:0/1924968184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbc4003a00 0x7fcbc4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbde800700 1 --2- 172.21.15.89:0/1924968184 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbe00ac600 0x7fcbc4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdd400700 1 --2- 172.21.15.89:0/1924968184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbc4004850 0x7fcbc4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdde00700 1 -- 172.21.15.89:0/1924968184 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbe00ac600 msgr2=0x7fcbc4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdde00700 1 --2- 172.21.15.89:0/1924968184 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbe00ac600 0x7fcbc4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdde00700 1 -- 172.21.15.89:0/1924968184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbc4004850 msgr2=0x7fcbc4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdde00700 1 --2- 172.21.15.89:0/1924968184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbc4004850 0x7fcbc4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdde00700 1 -- 172.21.15.89:0/1924968184 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc40035b0 con 0x7fcbc4003a00 2024-04-02T09:09:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.888+0000 7fcbdd400700 1 --2- 172.21.15.89:0/1924968184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbc4004850 0x7fcbc4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.889+0000 7fcbdde00700 1 --2- 172.21.15.89:0/1924968184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbc4003a00 0x7fcbc4003e20 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000c350 tx=0x7fcbc4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28cc622e61897a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.889+0000 7fcbdca00700 1 -- 172.21.15.89:0/1924968184 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd000ce50 con 0x7fcbc4003a00 2024-04-02T09:09:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.889+0000 7fcbdca00700 1 -- 172.21.15.89:0/1924968184 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd0005960 con 0x7fcbc4003a00 2024-04-02T09:09:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.889+0000 7fcbdca00700 1 -- 172.21.15.89:0/1924968184 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd0005d10 con 0x7fcbc4003a00 2024-04-02T09:09:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.889+0000 7fcbe4c00700 1 -- 172.21.15.89:0/1924968184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbc4003a00 msgr2=0x7fcbc4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.889+0000 7fcbe4c00700 1 --2- 172.21.15.89:0/1924968184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbc4003a00 0x7fcbc4003e20 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000c350 tx=0x7fcbc4005880 comp rx=0 tx=0).stop 2024-04-02T09:09:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.889+0000 7fcbde800700 1 -- 172.21.15.89:0/1924968184 reap_dead start 2024-04-02T09:09:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.889+0000 7fcbe4c00700 1 -- 172.21.15.89:0/1924968184 shutdown_connections 2024-04-02T09:09:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.889+0000 7fcbe4c00700 1 -- 172.21.15.89:0/1924968184 >> 172.21.15.89:0/1924968184 conn(0x7fcbe00cf190 msgr2=0x7fcbe00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.890+0000 7fcbe4c00700 1 -- 172.21.15.89:0/1924968184 shutdown_connections 2024-04-02T09:09:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.890+0000 7fcbe4c00700 1 -- 172.21.15.89:0/1924968184 wait complete. 2024-04-02T09:09:42.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.891+0000 7fcbe4c00700 1 Processor -- start 2024-04-02T09:09:42.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.891+0000 7fcbe4c00700 1 -- start start 2024-04-02T09:09:42.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.892+0000 7fcbe4c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbc4003a00 0x7fcbe015a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.892+0000 7fcbe4c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbe0156250 0x7fcbe015a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.892+0000 7fcbe4c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbe0144810 0x7fcbe01f4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.892+0000 7fcbe4c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe00cc790 con 0x7fcbe0156250 2024-04-02T09:09:42.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.892+0000 7fcbe4c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe0153680 con 0x7fcbe0144810 2024-04-02T09:09:42.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.892+0000 7fcbe4c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe00cbba0 con 0x7fcbc4003a00 2024-04-02T09:09:42.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbdd400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbe0156250 0x7fcbe015a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbdd400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbe0156250 0x7fcbe015a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40378/0 (socket says 172.21.15.89:40378) 2024-04-02T09:09:42.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbdd400700 1 -- 172.21.15.89:0/2332296327 learned_addr learned my addr 172.21.15.89:0/2332296327 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:42.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbdde00700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbc4003a00 0x7fcbe015a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbde800700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbe0144810 0x7fcbe01f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbdd400700 1 -- 172.21.15.89:0/2332296327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbc4003a00 msgr2=0x7fcbe015a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:42.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbdd400700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbc4003a00 0x7fcbe015a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbdd400700 1 -- 172.21.15.89:0/2332296327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbe0144810 msgr2=0x7fcbe01f4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:42.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbdd400700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbe0144810 0x7fcbe01f4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:42.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbdd400700 1 -- 172.21.15.89:0/2332296327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe015ab90 con 0x7fcbe0156250 2024-04-02T09:09:42.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbde800700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbe0144810 0x7fcbe01f4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:42.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbdd400700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbe0156250 0x7fcbe015a650 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc010930 tx=0x7fcbcc010cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:42.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbd6000700 1 -- 172.21.15.89:0/2332296327 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc004020 con 0x7fcbe0156250 2024-04-02T09:09:42.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbd6000700 1 -- 172.21.15.89:0/2332296327 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc004180 con 0x7fcbe0156250 2024-04-02T09:09:42.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbe4c00700 1 -- 172.21.15.89:0/2332296327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe00cbdf0 con 0x7fcbe0156250 2024-04-02T09:09:42.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.893+0000 7fcbd6000700 1 -- 172.21.15.89:0/2332296327 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc004530 con 0x7fcbe0156250 2024-04-02T09:09:42.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.894+0000 7fcbe4c00700 1 -- 172.21.15.89:0/2332296327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe00cc3b0 con 0x7fcbe0156250 2024-04-02T09:09:42.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.895+0000 7fcbe4c00700 1 -- 172.21.15.89:0/2332296327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe00cc8d0 con 0x7fcbe0156250 2024-04-02T09:09:42.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.896+0000 7fcbd6000700 1 -- 172.21.15.89:0/2332296327 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcbcc00e870 con 0x7fcbe0156250 2024-04-02T09:09:42.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.897+0000 7fcbd6000700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcbc006adb0 0x7fcbc006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:42.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.897+0000 7fcbd6000700 1 -- 172.21.15.89:0/2332296327 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fcbcc0899e0 con 0x7fcbe0156250 2024-04-02T09:09:42.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.897+0000 7fcbdde00700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcbc006adb0 0x7fcbc006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:42.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.898+0000 7fcbdde00700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcbc006adb0 0x7fcbc006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0008fd0 tx=0x7fcbd000a000 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:42.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:42.901+0000 7fcbd6000700 1 -- 172.21.15.89:0/2332296327 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcbe00cc8d0 con 0x7fcbe0156250 2024-04-02T09:09:43.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.102+0000 7fcbe4c00700 1 -- 172.21.15.89:0/2332296327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbe00cc8d0 con 0x7fcbe0156250 2024-04-02T09:09:43.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.104+0000 7fcbd6000700 1 -- 172.21.15.89:0/2332296327 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) v1 ==== 74+0+16757 (secure 0 0 0) 0x7fcbe00cc8d0 con 0x7fcbe0156250 2024-04-02T09:09:43.104 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:43.104 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":204,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:09:42.219865+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:09:40.103721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-04-02T09:09:41.165944+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:09:43.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.106+0000 7fcbe4c00700 1 -- 172.21.15.89:0/2332296327 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcbc006adb0 msgr2=0x7fcbc006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:43.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.106+0000 7fcbe4c00700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcbc006adb0 0x7fcbc006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0008fd0 tx=0x7fcbd000a000 comp rx=0 tx=0).stop 2024-04-02T09:09:43.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.106+0000 7fcbe4c00700 1 -- 172.21.15.89:0/2332296327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbe0156250 msgr2=0x7fcbe015a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:43.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.106+0000 7fcbe4c00700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbe0156250 0x7fcbe015a650 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc010930 tx=0x7fcbcc010cf0 comp rx=0 tx=0).stop 2024-04-02T09:09:43.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.106+0000 7fcbe4c00700 1 -- 172.21.15.89:0/2332296327 shutdown_connections 2024-04-02T09:09:43.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.106+0000 7fcbe4c00700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcbc006adb0 0x7fcbc006d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:43.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.106+0000 7fcbe4c00700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbe0144810 0x7fcbe01f4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:43.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.106+0000 7fcbe4c00700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbc4003a00 0x7fcbe015a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:43.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.106+0000 7fcbe4c00700 1 --2- 172.21.15.89:0/2332296327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbe0156250 0x7fcbe015a650 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:43.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.106+0000 7fcbe4c00700 1 -- 172.21.15.89:0/2332296327 >> 172.21.15.89:0/2332296327 conn(0x7fcbe00cf190 msgr2=0x7fcbe015e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:43.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.106+0000 7fcbe4c00700 1 -- 172.21.15.89:0/2332296327 shutdown_connections 2024-04-02T09:09:43.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.106+0000 7fcbe4c00700 1 -- 172.21.15.89:0/2332296327 wait complete. 2024-04-02T09:09:43.124 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T09:09:43.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.323+0000 7facedc00700 1 Processor -- start 2024-04-02T09:09:43.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.323+0000 7facedc00700 1 -- start start 2024-04-02T09:09:43.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.325+0000 7facedc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7face80cc370 0x7face80b2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:43.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.325+0000 7facedc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face8163190 con 0x7face80ce500 2024-04-02T09:09:43.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.325+0000 7facedc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face8162580 con 0x7face80b3760 2024-04-02T09:09:43.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.325+0000 7facedc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face8162860 con 0x7face80cc370 2024-04-02T09:09:43.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.325+0000 7face6a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7face80ce500 0x7face80ce8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36514/0 (socket says 172.21.15.89:36514) 2024-04-02T09:09:43.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.325+0000 7face6a00700 1 -- 172.21.15.89:0/65777576 learned_addr learned my addr 172.21.15.89:0/65777576 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:43.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.325+0000 7facdfe00700 1 --2- 172.21.15.89:0/65777576 >> v2:172.21.15.113:3300/0 conn(0x7face80cc370 0x7face80b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:43.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.325+0000 7face6000700 1 -- 172.21.15.89:0/65777576 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560091768 0 0) 0x7face8163190 con 0x7face80ce500 2024-04-02T09:09:43.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.325+0000 7face6000700 1 -- 172.21.15.89:0/65777576 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faccc0035b0 con 0x7face80ce500 2024-04-02T09:09:43.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face6000700 1 -- 172.21.15.89:0/65777576 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1712128450 0 0) 0x7faccc0035b0 con 0x7face80ce500 2024-04-02T09:09:43.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face6000700 1 -- 172.21.15.89:0/65777576 >> v2:172.21.15.113:3300/0 conn(0x7face80cc370 msgr2=0x7face80b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:43.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face6000700 1 --2- 172.21.15.89:0/65777576 >> v2:172.21.15.113:3300/0 conn(0x7face80cc370 0x7face80b2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:43.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face6000700 1 -- 172.21.15.89:0/65777576 >> v1:172.21.15.113:6789/0 conn(0x7face80b3760 legacy=0x7face8165ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:43.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face6000700 1 -- 172.21.15.89:0/65777576 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face8162b40 con 0x7face80ce500 2024-04-02T09:09:43.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face6000700 1 -- 172.21.15.89:0/65777576 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7facd8001670 con 0x7face80ce500 2024-04-02T09:09:43.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face6000700 1 -- 172.21.15.89:0/65777576 >> v1:172.21.15.89:6789/0 conn(0x7face80ce500 legacy=0x7face80ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:43.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face6000700 1 --2- 172.21.15.89:0/65777576 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faccc003a00 0x7faccc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:43.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face6000700 1 --2- 172.21.15.89:0/65777576 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faccc004850 0x7faccc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:43.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face6000700 1 --2- 172.21.15.89:0/65777576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc005210 0x7faccc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:43.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face6000700 1 -- 172.21.15.89:0/65777576 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face8162860 con 0x7faccc005210 2024-04-02T09:09:43.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face6000700 1 -- 172.21.15.89:0/65777576 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc00c060 con 0x7faccc003a00 2024-04-02T09:09:43.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face6000700 1 -- 172.21.15.89:0/65777576 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faccc00c220 con 0x7faccc004850 2024-04-02T09:09:43.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face7400700 1 --2- 172.21.15.89:0/65777576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc005210 0x7faccc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7face6a00700 1 --2- 172.21.15.89:0/65777576 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faccc003a00 0x7faccc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.326+0000 7facdfe00700 1 --2- 172.21.15.89:0/65777576 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faccc004850 0x7faccc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.327+0000 7face7400700 1 -- 172.21.15.89:0/65777576 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faccc004850 msgr2=0x7faccc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.327+0000 7face7400700 1 --2- 172.21.15.89:0/65777576 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faccc004850 0x7faccc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.327+0000 7face7400700 1 -- 172.21.15.89:0/65777576 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faccc003a00 msgr2=0x7faccc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.327+0000 7face7400700 1 --2- 172.21.15.89:0/65777576 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faccc003a00 0x7faccc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.327+0000 7face7400700 1 -- 172.21.15.89:0/65777576 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc0035b0 con 0x7faccc005210 2024-04-02T09:09:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.327+0000 7face7400700 1 -- 172.21.15.89:0/65777576 reap_dead start 2024-04-02T09:09:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.327+0000 7face7400700 1 --2- 172.21.15.89:0/65777576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc005210 0x7faccc00ba10 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7face000c290 tx=0x7faccc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2131b39c07ec07 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.327+0000 7face6000700 1 -- 172.21.15.89:0/65777576 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face0001e70 con 0x7faccc005210 2024-04-02T09:09:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.327+0000 7face6000700 1 -- 172.21.15.89:0/65777576 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face000ce70 con 0x7faccc005210 2024-04-02T09:09:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.327+0000 7face6000700 1 -- 172.21.15.89:0/65777576 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face0003c20 con 0x7faccc005210 2024-04-02T09:09:43.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.328+0000 7facedc00700 1 -- 172.21.15.89:0/65777576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc005210 msgr2=0x7faccc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:43.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.328+0000 7facedc00700 1 --2- 172.21.15.89:0/65777576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc005210 0x7faccc00ba10 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7face000c290 tx=0x7faccc005880 comp rx=0 tx=0).stop 2024-04-02T09:09:43.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.328+0000 7facedc00700 1 -- 172.21.15.89:0/65777576 shutdown_connections 2024-04-02T09:09:43.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.328+0000 7facedc00700 1 --2- 172.21.15.89:0/65777576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faccc005210 0x7faccc00ba10 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:43.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.328+0000 7facedc00700 1 -- 172.21.15.89:0/65777576 >> 172.21.15.89:0/65777576 conn(0x7face80b0d70 msgr2=0x7face80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:43.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.328+0000 7facedc00700 1 -- 172.21.15.89:0/65777576 shutdown_connections 2024-04-02T09:09:43.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.328+0000 7facedc00700 1 -- 172.21.15.89:0/65777576 wait complete. 2024-04-02T09:09:43.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.330+0000 7facedc00700 1 Processor -- start 2024-04-02T09:09:43.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.330+0000 7facedc00700 1 -- start start 2024-04-02T09:09:43.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.331+0000 7facedc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faccc005210 0x7face809caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:43.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.331+0000 7facedc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face806dfb0 0x7face8164130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:43.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7facedc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face8164670 0x7face81f70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:43.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7facedc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face8164b00 con 0x7face8164670 2024-04-02T09:09:43.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7facedc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face80991d0 con 0x7faccc005210 2024-04-02T09:09:43.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7facedc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face80994b0 con 0x7face806dfb0 2024-04-02T09:09:43.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7face7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face8164670 0x7face81f70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:43.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7face7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face8164670 0x7face81f70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40398/0 (socket says 172.21.15.89:40398) 2024-04-02T09:09:43.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7face7400700 1 -- 172.21.15.89:0/1686532345 learned_addr learned my addr 172.21.15.89:0/1686532345 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:43.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7face6a00700 1 --2- 172.21.15.89:0/1686532345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faccc005210 0x7face809caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:43.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7facdfe00700 1 --2- 172.21.15.89:0/1686532345 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face806dfb0 0x7face8164130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:43.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7face7400700 1 -- 172.21.15.89:0/1686532345 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face806dfb0 msgr2=0x7face8164130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:43.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7face7400700 1 --2- 172.21.15.89:0/1686532345 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face806dfb0 0x7face8164130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:43.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7face7400700 1 -- 172.21.15.89:0/1686532345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faccc005210 msgr2=0x7face809caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:43.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7face7400700 1 --2- 172.21.15.89:0/1686532345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faccc005210 0x7face809caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:43.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7face7400700 1 -- 172.21.15.89:0/1686532345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face806e4b0 con 0x7face8164670 2024-04-02T09:09:43.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.332+0000 7face7400700 1 --2- 172.21.15.89:0/1686532345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face8164670 0x7face81f70d0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7face000c880 tx=0x7face00015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:43.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.333+0000 7facdea00700 1 -- 172.21.15.89:0/1686532345 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face0017070 con 0x7face8164670 2024-04-02T09:09:43.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.333+0000 7facdea00700 1 -- 172.21.15.89:0/1686532345 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face00052e0 con 0x7face8164670 2024-04-02T09:09:43.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.333+0000 7facedc00700 1 -- 172.21.15.89:0/1686532345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face806e790 con 0x7face8164670 2024-04-02T09:09:43.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.333+0000 7facdea00700 1 -- 172.21.15.89:0/1686532345 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face0004ea0 con 0x7face8164670 2024-04-02T09:09:43.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.333+0000 7facedc00700 1 -- 172.21.15.89:0/1686532345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face809d030 con 0x7face8164670 2024-04-02T09:09:43.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.334+0000 7facedc00700 1 -- 172.21.15.89:0/1686532345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face809d1f0 con 0x7face8164670 2024-04-02T09:09:43.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.335+0000 7facdea00700 1 -- 172.21.15.89:0/1686532345 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7face0005650 con 0x7face8164670 2024-04-02T09:09:43.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.336+0000 7facdea00700 1 --2- 172.21.15.89:0/1686532345 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7facc406adb0 0x7facc406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:43.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.336+0000 7facdea00700 1 -- 172.21.15.89:0/1686532345 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7face001a040 con 0x7face8164670 2024-04-02T09:09:43.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.336+0000 7face6a00700 1 --2- 172.21.15.89:0/1686532345 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7facc406adb0 0x7facc406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:43.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.337+0000 7face6a00700 1 --2- 172.21.15.89:0/1686532345 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7facc406adb0 0x7facc406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7facd8000c00 tx=0x7facd8000ed0 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:43.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.340+0000 7facdea00700 1 -- 172.21.15.89:0/1686532345 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7face0055070 con 0x7face8164670 2024-04-02T09:09:43.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:43.586+0000 7facedc00700 1 -- 172.21.15.89:0/1686532345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7face8067720 con 0x7face8164670 2024-04-02T09:09:44.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.232+0000 7facdea00700 1 -- 172.21.15.89:0/1686532345 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v96) v1 ==== 118+0+0 (secure 0 0 0) 0x7face00230b0 con 0x7face8164670 2024-04-02T09:09:44.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.235+0000 7facedc00700 1 -- 172.21.15.89:0/1686532345 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7facc406adb0 msgr2=0x7facc406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:44.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.235+0000 7facedc00700 1 --2- 172.21.15.89:0/1686532345 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7facc406adb0 0x7facc406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7facd8000c00 tx=0x7facd8000ed0 comp rx=0 tx=0).stop 2024-04-02T09:09:44.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.235+0000 7facedc00700 1 -- 172.21.15.89:0/1686532345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face8164670 msgr2=0x7face81f70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:44.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.235+0000 7facedc00700 1 --2- 172.21.15.89:0/1686532345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face8164670 0x7face81f70d0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7face000c880 tx=0x7face00015b0 comp rx=0 tx=0).stop 2024-04-02T09:09:44.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.235+0000 7facedc00700 1 -- 172.21.15.89:0/1686532345 shutdown_connections 2024-04-02T09:09:44.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.235+0000 7facedc00700 1 --2- 172.21.15.89:0/1686532345 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7facc406adb0 0x7facc406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:44.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.235+0000 7facedc00700 1 --2- 172.21.15.89:0/1686532345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faccc005210 0x7face809caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:44.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.235+0000 7facedc00700 1 --2- 172.21.15.89:0/1686532345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face8164670 0x7face81f70d0 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:44.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.235+0000 7facedc00700 1 --2- 172.21.15.89:0/1686532345 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face806dfb0 0x7face8164130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:44.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.235+0000 7facedc00700 1 -- 172.21.15.89:0/1686532345 >> 172.21.15.89:0/1686532345 conn(0x7face80b0d70 msgr2=0x7face80b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:44.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.235+0000 7facedc00700 1 -- 172.21.15.89:0/1686532345 shutdown_connections 2024-04-02T09:09:44.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.235+0000 7facedc00700 1 -- 172.21.15.89:0/1686532345 wait complete. 2024-04-02T09:09:44.254 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T09:09:44.254 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T09:09:44.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.451+0000 7fe5b1200700 1 Processor -- start 2024-04-02T09:09:44.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.451+0000 7fe5b1200700 1 -- start start 2024-04-02T09:09:44.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.453+0000 7fe5b1200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe5ac160d70 0x7fe5ac0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:44.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.453+0000 7fe5b1200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe5ac06a790 0x7fe5ac06ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:44.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.453+0000 7fe5b1200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ac161150 con 0x7fe5ac15ef70 2024-04-02T09:09:44.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.453+0000 7fe5b1200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5ac0b4cd0 con 0x7fe5ac06a790 2024-04-02T09:09:44.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.453+0000 7fe5b1200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5ac06b140 con 0x7fe5ac160d70 2024-04-02T09:09:44.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.453+0000 7fe5aa800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe5ac06a790 0x7fe5ac06ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:44.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.453+0000 7fe5a9400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe5ac15ef70 0x7fe5ac15e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36528/0 (socket says 172.21.15.89:36528) 2024-04-02T09:09:44.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.453+0000 7fe5a9400700 1 -- 172.21.15.89:0/1793037478 learned_addr learned my addr 172.21.15.89:0/1793037478 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:44.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.453+0000 7fe5a9e00700 1 --2- 172.21.15.89:0/1793037478 >> v2:172.21.15.113:3300/0 conn(0x7fe5ac160d70 0x7fe5ac0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:44.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.454+0000 7fe5aa800700 1 -- 172.21.15.89:0/1793037478 >> v2:172.21.15.113:3300/0 conn(0x7fe5ac160d70 msgr2=0x7fe5ac0b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:44.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.454+0000 7fe5aa800700 1 --2- 172.21.15.89:0/1793037478 >> v2:172.21.15.113:3300/0 conn(0x7fe5ac160d70 0x7fe5ac0b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:44.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.454+0000 7fe5aa800700 1 -- 172.21.15.89:0/1793037478 >> v1:172.21.15.89:6789/0 conn(0x7fe5ac15ef70 legacy=0x7fe5ac15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:44.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.454+0000 7fe5aa800700 1 -- 172.21.15.89:0/1793037478 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ac06dba0 con 0x7fe5ac06a790 2024-04-02T09:09:44.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.454+0000 7fe5a8a00700 1 -- 172.21.15.89:0/1793037478 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945460364 0 0) 0x7fe5ac161150 con 0x7fe5ac15ef70 2024-04-02T09:09:44.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.454+0000 7fe5aa800700 1 --2- 172.21.15.89:0/1793037478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5ac06a790 0x7fe5ac06ac00 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac06c880 tx=0x7fe5a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e52e63c6fc84637 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:44.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.454+0000 7fe5a8a00700 1 -- 172.21.15.89:0/1793037478 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a400e070 con 0x7fe5ac06a790 2024-04-02T09:09:44.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.454+0000 7fe5a8a00700 1 -- 172.21.15.89:0/1793037478 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a400bb20 con 0x7fe5ac06a790 2024-04-02T09:09:44.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.454+0000 7fe5a8a00700 1 -- 172.21.15.89:0/1793037478 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4012430 con 0x7fe5ac06a790 2024-04-02T09:09:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.455+0000 7fe5b1200700 1 -- 172.21.15.89:0/1793037478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5ac06a790 msgr2=0x7fe5ac06ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.455+0000 7fe5b1200700 1 --2- 172.21.15.89:0/1793037478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5ac06a790 0x7fe5ac06ac00 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac06c880 tx=0x7fe5a400b330 comp rx=0 tx=0).stop 2024-04-02T09:09:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.455+0000 7fe5b1200700 1 -- 172.21.15.89:0/1793037478 shutdown_connections 2024-04-02T09:09:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.455+0000 7fe5b1200700 1 --2- 172.21.15.89:0/1793037478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5ac06a790 0x7fe5ac06ac00 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.455+0000 7fe5b1200700 1 --2- 172.21.15.89:0/1793037478 >> v2:172.21.15.113:3300/0 conn(0x7fe5ac160d70 0x7fe5ac0b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.455+0000 7fe5b1200700 1 -- 172.21.15.89:0/1793037478 >> 172.21.15.89:0/1793037478 conn(0x7fe5ac0b5720 msgr2=0x7fe5ac0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.455+0000 7fe5b1200700 1 -- 172.21.15.89:0/1793037478 shutdown_connections 2024-04-02T09:09:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.455+0000 7fe5b1200700 1 -- 172.21.15.89:0/1793037478 wait complete. 2024-04-02T09:09:44.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.457+0000 7fe5b1200700 1 Processor -- start 2024-04-02T09:09:44.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.457+0000 7fe5b1200700 1 -- start start 2024-04-02T09:09:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5b1200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe5ac06a790 0x7fe5ac149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5b1200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5ac15ef70 0x7fe5ac09b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5b1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5ac160d70 0x7fe5ac09b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5b1200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac147090 con 0x7fe5ac160d70 2024-04-02T09:09:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5b1200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac149bc0 con 0x7fe5ac15ef70 2024-04-02T09:09:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5b1200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac149ea0 con 0x7fe5ac06a790 2024-04-02T09:09:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5aa800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5ac160d70 0x7fe5ac09b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5aa800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5ac160d70 0x7fe5ac09b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40412/0 (socket says 172.21.15.89:40412) 2024-04-02T09:09:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5aa800700 1 -- 172.21.15.89:0/4257864260 learned_addr learned my addr 172.21.15.89:0/4257864260 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5a9400700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5ac15ef70 0x7fe5ac09b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5a9e00700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe5ac06a790 0x7fe5ac149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5aa800700 1 -- 172.21.15.89:0/4257864260 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe5ac06a790 msgr2=0x7fe5ac149120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5aa800700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe5ac06a790 0x7fe5ac149120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5aa800700 1 -- 172.21.15.89:0/4257864260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5ac15ef70 msgr2=0x7fe5ac09b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5aa800700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5ac15ef70 0x7fe5ac09b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5aa800700 1 -- 172.21.15.89:0/4257864260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a4009a70 con 0x7fe5ac160d70 2024-04-02T09:09:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5a9e00700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe5ac06a790 0x7fe5ac149120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.459+0000 7fe5a9400700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5ac15ef70 0x7fe5ac09b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.460+0000 7fe5aa800700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5ac160d70 0x7fe5ac09b9c0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac070430 tx=0x7fe5a4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.460+0000 7fe5a2000700 1 -- 172.21.15.89:0/4257864260 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4012430 con 0x7fe5ac160d70 2024-04-02T09:09:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.460+0000 7fe5a2000700 1 -- 172.21.15.89:0/4257864260 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a4020070 con 0x7fe5ac160d70 2024-04-02T09:09:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.460+0000 7fe5b1200700 1 -- 172.21.15.89:0/4257864260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ac1472a0 con 0x7fe5ac160d70 2024-04-02T09:09:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.460+0000 7fe5a2000700 1 -- 172.21.15.89:0/4257864260 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4004280 con 0x7fe5ac160d70 2024-04-02T09:09:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.460+0000 7fe5b1200700 1 -- 172.21.15.89:0/4257864260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ac14d360 con 0x7fe5ac160d70 2024-04-02T09:09:44.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.461+0000 7fe5b1200700 1 -- 172.21.15.89:0/4257864260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5ac067720 con 0x7fe5ac160d70 2024-04-02T09:09:44.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.462+0000 7fe5a2000700 1 -- 172.21.15.89:0/4257864260 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe5a40043e0 con 0x7fe5ac160d70 2024-04-02T09:09:44.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.463+0000 7fe5a2000700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe58c06adb0 0x7fe58c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:44.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.463+0000 7fe5a2000700 1 -- 172.21.15.89:0/4257864260 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fe5a4022070 con 0x7fe5ac160d70 2024-04-02T09:09:44.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.464+0000 7fe5a9e00700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe58c06adb0 0x7fe58c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:44.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.464+0000 7fe5a9e00700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe58c06adb0 0x7fe58c06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe59c009b90 tx=0x7fe59c008040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:44.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.468+0000 7fe5a2000700 1 -- 172.21.15.89:0/4257864260 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe5a40562c0 con 0x7fe5ac160d70 2024-04-02T09:09:44.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:44.714+0000 7fe5b1200700 1 -- 172.21.15.89:0/4257864260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe5ac09afc0 con 0x7fe5ac160d70 2024-04-02T09:09:45.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:45.701+0000 7fe5a2000700 1 -- 172.21.15.89:0/4257864260 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v97) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe5a40598e0 con 0x7fe5ac160d70 2024-04-02T09:09:45.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:45.705+0000 7fe5b1200700 1 -- 172.21.15.89:0/4257864260 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe58c06adb0 msgr2=0x7fe58c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:45.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:45.705+0000 7fe5b1200700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe58c06adb0 0x7fe58c06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe59c009b90 tx=0x7fe59c008040 comp rx=0 tx=0).stop 2024-04-02T09:09:45.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:45.705+0000 7fe5b1200700 1 -- 172.21.15.89:0/4257864260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5ac160d70 msgr2=0x7fe5ac09b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:45.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:45.705+0000 7fe5b1200700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5ac160d70 0x7fe5ac09b9c0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac070430 tx=0x7fe5a4004ad0 comp rx=0 tx=0).stop 2024-04-02T09:09:45.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:45.705+0000 7fe5b1200700 1 -- 172.21.15.89:0/4257864260 shutdown_connections 2024-04-02T09:09:45.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:45.705+0000 7fe5b1200700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5ac160d70 0x7fe5ac09b9c0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:45.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:45.705+0000 7fe5b1200700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe5ac06a790 0x7fe5ac149120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:45.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:45.705+0000 7fe5b1200700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe58c06adb0 0x7fe58c06d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:45.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:45.705+0000 7fe5b1200700 1 --2- 172.21.15.89:0/4257864260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5ac15ef70 0x7fe5ac09b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:45.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:45.705+0000 7fe5b1200700 1 -- 172.21.15.89:0/4257864260 >> 172.21.15.89:0/4257864260 conn(0x7fe5ac0b5720 msgr2=0x7fe5ac06ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:45.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:45.705+0000 7fe5b1200700 1 -- 172.21.15.89:0/4257864260 shutdown_connections 2024-04-02T09:09:45.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:45.705+0000 7fe5b1200700 1 -- 172.21.15.89:0/4257864260 wait complete. 2024-04-02T09:09:45.883 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T09:09:45.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:09:46.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.081+0000 7f2b27200700 1 Processor -- start 2024-04-02T09:09:46.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.082+0000 7f2b27200700 1 -- start start 2024-04-02T09:09:46.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.083+0000 7f2b27200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2b20161ca0 0x7f2b200cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.083+0000 7f2b27200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b200ceda0 con 0x7f2b200d18b0 2024-04-02T09:09:46.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.083+0000 7f2b27200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b201612f0 con 0x7f2b20160f10 2024-04-02T09:09:46.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.083+0000 7f2b27200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b200d1d30 con 0x7f2b20161ca0 2024-04-02T09:09:46.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.083+0000 7f2b1f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2b200d18b0 0x7f2b20161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36530/0 (socket says 172.21.15.89:36530) 2024-04-02T09:09:46.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.083+0000 7f2b1f400700 1 -- 172.21.15.89:0/3151327559 learned_addr learned my addr 172.21.15.89:0/3151327559 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:46.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.083+0000 7f2b25200700 1 --2- 172.21.15.89:0/3151327559 >> v2:172.21.15.89:3300/0 conn(0x7f2b20161ca0 0x7f2b200cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.084+0000 7f2b1ea00700 1 -- 172.21.15.89:0/3151327559 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962269736 0 0) 0x7f2b200ceda0 con 0x7f2b200d18b0 2024-04-02T09:09:46.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.084+0000 7f2b1ea00700 1 -- 172.21.15.89:0/3151327559 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b040035b0 con 0x7f2b200d18b0 2024-04-02T09:09:46.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.084+0000 7f2b25200700 1 -- 172.21.15.89:0/3151327559 >> v1:172.21.15.113:6789/0 conn(0x7f2b20160f10 legacy=0x7f2b200d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.084+0000 7f2b1ea00700 1 -- 172.21.15.89:0/3151327559 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864062725 0 0) 0x7f2b201612f0 con 0x7f2b20160f10 2024-04-02T09:09:46.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.084+0000 7f2b25200700 1 -- 172.21.15.89:0/3151327559 >> v1:172.21.15.89:6789/0 conn(0x7f2b200d18b0 legacy=0x7f2b20161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.084+0000 7f2b25200700 1 -- 172.21.15.89:0/3151327559 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b200ce470 con 0x7f2b20161ca0 2024-04-02T09:09:46.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.084+0000 7f2b25200700 1 --2- 172.21.15.89:0/3151327559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b20161ca0 0x7f2b200cf230 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f2b18009d90 tx=0x7f2b1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=331618eb132639e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:46.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.084+0000 7f2b1ea00700 1 -- 172.21.15.89:0/3151327559 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1800e070 con 0x7f2b20161ca0 2024-04-02T09:09:46.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.084+0000 7f2b1ea00700 1 -- 172.21.15.89:0/3151327559 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1800bb20 con 0x7f2b20161ca0 2024-04-02T09:09:46.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.084+0000 7f2b1ea00700 1 -- 172.21.15.89:0/3151327559 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1800bed0 con 0x7f2b20161ca0 2024-04-02T09:09:46.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.085+0000 7f2b27200700 1 -- 172.21.15.89:0/3151327559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b20161ca0 msgr2=0x7f2b200cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.085+0000 7f2b27200700 1 --2- 172.21.15.89:0/3151327559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b20161ca0 0x7f2b200cf230 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f2b18009d90 tx=0x7f2b1800b330 comp rx=0 tx=0).stop 2024-04-02T09:09:46.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.085+0000 7f2b27200700 1 -- 172.21.15.89:0/3151327559 shutdown_connections 2024-04-02T09:09:46.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.085+0000 7f2b27200700 1 --2- 172.21.15.89:0/3151327559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b20161ca0 0x7f2b200cf230 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.085+0000 7f2b27200700 1 -- 172.21.15.89:0/3151327559 >> 172.21.15.89:0/3151327559 conn(0x7f2b2008aff0 msgr2=0x7f2b200d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:46.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.085+0000 7f2b27200700 1 -- 172.21.15.89:0/3151327559 shutdown_connections 2024-04-02T09:09:46.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.085+0000 7f2b27200700 1 -- 172.21.15.89:0/3151327559 wait complete. 2024-04-02T09:09:46.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.087+0000 7f2b27200700 1 Processor -- start 2024-04-02T09:09:46.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.087+0000 7f2b27200700 1 -- start start 2024-04-02T09:09:46.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b27200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b200d18b0 0x7f2b201f4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b27200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b20160f10 0x7f2b2014fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b27200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b20161ca0 0x7f2b2015f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b27200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2015cef0 con 0x7f2b20161ca0 2024-04-02T09:09:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b27200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b200b8a40 con 0x7f2b200d18b0 2024-04-02T09:09:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b27200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2015d0d0 con 0x7f2b20160f10 2024-04-02T09:09:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b25200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b20161ca0 0x7f2b2015f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b25200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b20161ca0 0x7f2b2015f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40434/0 (socket says 172.21.15.89:40434) 2024-04-02T09:09:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b25200700 1 -- 172.21.15.89:0/1155048345 learned_addr learned my addr 172.21.15.89:0/1155048345 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b1fe00700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b200d18b0 0x7f2b201f4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b25200700 1 -- 172.21.15.89:0/1155048345 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b20160f10 msgr2=0x7f2b2014fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b1f400700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b20160f10 0x7f2b2014fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b25200700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b20160f10 0x7f2b2014fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b25200700 1 -- 172.21.15.89:0/1155048345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b200d18b0 msgr2=0x7f2b201f4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b25200700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b200d18b0 0x7f2b201f4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b25200700 1 -- 172.21.15.89:0/1155048345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b18009a70 con 0x7f2b20161ca0 2024-04-02T09:09:46.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b1fe00700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b200d18b0 0x7f2b201f4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:46.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.088+0000 7f2b1f400700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b20160f10 0x7f2b2014fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:46.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.089+0000 7f2b25200700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b20161ca0 0x7f2b2015f2e0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f2b18014040 tx=0x7f2b18004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:46.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.089+0000 7f2b17e00700 1 -- 172.21.15.89:0/1155048345 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1801b810 con 0x7f2b20161ca0 2024-04-02T09:09:46.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.089+0000 7f2b17e00700 1 -- 172.21.15.89:0/1155048345 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b18020070 con 0x7f2b20161ca0 2024-04-02T09:09:46.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.089+0000 7f2b17e00700 1 -- 172.21.15.89:0/1155048345 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1801bbb0 con 0x7f2b20161ca0 2024-04-02T09:09:46.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.089+0000 7f2b27200700 1 -- 172.21.15.89:0/1155048345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b2015d3b0 con 0x7f2b20161ca0 2024-04-02T09:09:46.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.089+0000 7f2b27200700 1 -- 172.21.15.89:0/1155048345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2015d970 con 0x7f2b20161ca0 2024-04-02T09:09:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.090+0000 7f2b15600700 1 -- 172.21.15.89:0/1155048345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b20067720 con 0x7f2b20161ca0 2024-04-02T09:09:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.091+0000 7f2b17e00700 1 -- 172.21.15.89:0/1155048345 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2b18007230 con 0x7f2b20161ca0 2024-04-02T09:09:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.092+0000 7f2b17e00700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2b0006adb0 0x7f2b0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.092+0000 7f2b17e00700 1 -- 172.21.15.89:0/1155048345 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f2b18022070 con 0x7f2b20161ca0 2024-04-02T09:09:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.092+0000 7f2b1fe00700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2b0006adb0 0x7f2b0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.093+0000 7f2b1fe00700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2b0006adb0 0x7f2b0006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2b10000f60 tx=0x7f2b10009040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:46.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.096+0000 7f2b17e00700 1 -- 172.21.15.89:0/1155048345 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2b1805a150 con 0x7f2b20161ca0 2024-04-02T09:09:46.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.341+0000 7f2b15600700 1 -- 172.21.15.89:0/1155048345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b2008b760 con 0x7f2b20161ca0 2024-04-02T09:09:46.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.341+0000 7f2b17e00700 1 -- 172.21.15.89:0/1155048345 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4748 (secure 0 0 0) 0x7f2b18059ce0 con 0x7f2b20161ca0 2024-04-02T09:09:46.341 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:46.341 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1311594829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1311594829},{"type":"v1","addr":"172.21.15.113:6834","nonce":1311594829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1256547418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1256547418},{"type":"v1","addr":"172.21.15.89:6837","nonce":1256547418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1137044351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1137044351},{"type":"v1","addr":"172.21.15.113:6835","nonce":1137044351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:09:42.220191+0000","modified":"2024-04-02T09:09:45.699081+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10638":{"gid":10638,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/2257849268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2257849268},{"type":"v1","addr":"172.21.15.89:6836","nonce":2257849268}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-04-02T09:09:46.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.344+0000 7f2b15600700 1 -- 172.21.15.89:0/1155048345 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2b0006adb0 msgr2=0x7f2b0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.344+0000 7f2b15600700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2b0006adb0 0x7f2b0006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2b10000f60 tx=0x7f2b10009040 comp rx=0 tx=0).stop 2024-04-02T09:09:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.344+0000 7f2b15600700 1 -- 172.21.15.89:0/1155048345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b20161ca0 msgr2=0x7f2b2015f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.344+0000 7f2b15600700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b20161ca0 0x7f2b2015f2e0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f2b18014040 tx=0x7f2b18004ad0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.344+0000 7f2b15600700 1 -- 172.21.15.89:0/1155048345 shutdown_connections 2024-04-02T09:09:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.344+0000 7f2b15600700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2b0006adb0 0x7f2b0006d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.344+0000 7f2b15600700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b200d18b0 0x7f2b201f4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.344+0000 7f2b15600700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b20161ca0 0x7f2b2015f2e0 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.344+0000 7f2b15600700 1 --2- 172.21.15.89:0/1155048345 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2b20160f10 0x7f2b2014fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.344+0000 7f2b15600700 1 -- 172.21.15.89:0/1155048345 >> 172.21.15.89:0/1155048345 conn(0x7f2b2008aff0 msgr2=0x7f2b200ca650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.344+0000 7f2b15600700 1 -- 172.21.15.89:0/1155048345 shutdown_connections 2024-04-02T09:09:46.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.344+0000 7f2b15600700 1 -- 172.21.15.89:0/1155048345 wait complete. 2024-04-02T09:09:46.345 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 97 2024-04-02T09:09:46.361 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:09:46.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.553+0000 7f408b000700 1 Processor -- start 2024-04-02T09:09:46.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.554+0000 7f408b000700 1 -- start start 2024-04-02T09:09:46.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.555+0000 7f408b000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f40840cb3f0 0x7f40840ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.555+0000 7f408b000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40840ac600 0x7f408406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.555+0000 7f408b000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408415fd50 con 0x7f408415b5d0 2024-04-02T09:09:46.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.555+0000 7f408b000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f408415b9b0 con 0x7f40840cb3f0 2024-04-02T09:09:46.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.555+0000 7f408b000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40840aca50 con 0x7f40840ac600 2024-04-02T09:09:46.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.555+0000 7f4083400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f40840cb3f0 0x7f40840ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.555+0000 7f4083400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f40840cb3f0 0x7f40840ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40436/0 (socket says 172.21.15.89:40436) 2024-04-02T09:09:46.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.555+0000 7f4083400700 1 -- 172.21.15.89:0/2352935784 learned_addr learned my addr 172.21.15.89:0/2352935784 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:46.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.555+0000 7f4089000700 1 --2- 172.21.15.89:0/2352935784 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40840ac600 0x7f408406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.555+0000 7f4083400700 1 -- 172.21.15.89:0/2352935784 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40840ac600 msgr2=0x7f408406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.555+0000 7f4083400700 1 --2- 172.21.15.89:0/2352935784 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40840ac600 0x7f408406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.555+0000 7f4083400700 1 -- 172.21.15.89:0/2352935784 >> v1:172.21.15.113:6789/0 conn(0x7f408415b5d0 legacy=0x7f40840ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.555+0000 7f4083400700 1 -- 172.21.15.89:0/2352935784 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408415f140 con 0x7f40840cb3f0 2024-04-02T09:09:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.556+0000 7f4083400700 1 --2- 172.21.15.89:0/2352935784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40840cb3f0 0x7f40840ac0c0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f40700098f0 tx=0x7f4070009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800d4bb6d1989525 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.556+0000 7f4082a00700 1 -- 172.21.15.89:0/2352935784 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4070004230 con 0x7f40840cb3f0 2024-04-02T09:09:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.556+0000 7f4082a00700 1 -- 172.21.15.89:0/2352935784 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4070004390 con 0x7f40840cb3f0 2024-04-02T09:09:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.556+0000 7f4082a00700 1 -- 172.21.15.89:0/2352935784 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40700036a0 con 0x7f40840cb3f0 2024-04-02T09:09:46.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.556+0000 7f408b000700 1 -- 172.21.15.89:0/2352935784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40840cb3f0 msgr2=0x7f40840ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.556+0000 7f408b000700 1 --2- 172.21.15.89:0/2352935784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40840cb3f0 0x7f40840ac0c0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f40700098f0 tx=0x7f4070009c00 comp rx=0 tx=0).stop 2024-04-02T09:09:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.556+0000 7f408b000700 1 -- 172.21.15.89:0/2352935784 shutdown_connections 2024-04-02T09:09:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.556+0000 7f408b000700 1 --2- 172.21.15.89:0/2352935784 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f40840ac600 0x7f408406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.556+0000 7f408b000700 1 --2- 172.21.15.89:0/2352935784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40840cb3f0 0x7f40840ac0c0 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.556+0000 7f408b000700 1 -- 172.21.15.89:0/2352935784 >> 172.21.15.89:0/2352935784 conn(0x7f40840cf190 msgr2=0x7f40840cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.556+0000 7f408b000700 1 -- 172.21.15.89:0/2352935784 shutdown_connections 2024-04-02T09:09:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.557+0000 7f408b000700 1 -- 172.21.15.89:0/2352935784 wait complete. 2024-04-02T09:09:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.558+0000 7f408b000700 1 Processor -- start 2024-04-02T09:09:46.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.558+0000 7f408b000700 1 -- start start 2024-04-02T09:09:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.559+0000 7f408b000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40840ac600 0x7f408414e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.559+0000 7f408b000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f408415b5d0 0x7f40840ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.559+0000 7f408b000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4084148830 0x7f40840b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.559+0000 7f408b000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40840ce810 con 0x7f40840ac600 2024-04-02T09:09:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.559+0000 7f408b000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40841450f0 con 0x7f4084148830 2024-04-02T09:09:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.559+0000 7f408b000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40840ce9f0 con 0x7f408415b5d0 2024-04-02T09:09:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.559+0000 7f4083e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40840ac600 0x7f408414e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.559+0000 7f4083e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40840ac600 0x7f408414e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40450/0 (socket says 172.21.15.89:40450) 2024-04-02T09:09:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.559+0000 7f4083e00700 1 -- 172.21.15.89:0/984423598 learned_addr learned my addr 172.21.15.89:0/984423598 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.560+0000 7f4083400700 1 --2- 172.21.15.89:0/984423598 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f408415b5d0 0x7f40840ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.560+0000 7f4089000700 1 --2- 172.21.15.89:0/984423598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4084148830 0x7f40840b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.560+0000 7f4083e00700 1 -- 172.21.15.89:0/984423598 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f408415b5d0 msgr2=0x7f40840ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.560+0000 7f4083e00700 1 --2- 172.21.15.89:0/984423598 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f408415b5d0 0x7f40840ce2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.560+0000 7f4083e00700 1 -- 172.21.15.89:0/984423598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4084148830 msgr2=0x7f40840b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.560+0000 7f4083e00700 1 --2- 172.21.15.89:0/984423598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4084148830 0x7f40840b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.560+0000 7f4083e00700 1 -- 172.21.15.89:0/984423598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4070009550 con 0x7f40840ac600 2024-04-02T09:09:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.560+0000 7f4083e00700 1 --2- 172.21.15.89:0/984423598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40840ac600 0x7f408414e2c0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f407400b4d0 tx=0x7f407400b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.560+0000 7f407be00700 1 -- 172.21.15.89:0/984423598 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407400bea0 con 0x7f40840ac600 2024-04-02T09:09:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.560+0000 7f407be00700 1 -- 172.21.15.89:0/984423598 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4074011830 con 0x7f40840ac600 2024-04-02T09:09:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.560+0000 7f407be00700 1 -- 172.21.15.89:0/984423598 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4074011be0 con 0x7f40840ac600 2024-04-02T09:09:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.560+0000 7f408b000700 1 -- 172.21.15.89:0/984423598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40841578d0 con 0x7f40840ac600 2024-04-02T09:09:46.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.560+0000 7f408b000700 1 -- 172.21.15.89:0/984423598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40840b22e0 con 0x7f40840ac600 2024-04-02T09:09:46.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.561+0000 7f408b000700 1 -- 172.21.15.89:0/984423598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4084067720 con 0x7f40840ac600 2024-04-02T09:09:46.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.562+0000 7f407be00700 1 -- 172.21.15.89:0/984423598 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4074011d40 con 0x7f40840ac600 2024-04-02T09:09:46.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.563+0000 7f407be00700 1 --2- 172.21.15.89:0/984423598 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f406406adb0 0x7f406406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.563+0000 7f407be00700 1 -- 172.21.15.89:0/984423598 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f4074089180 con 0x7f40840ac600 2024-04-02T09:09:46.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.563+0000 7f4083400700 1 --2- 172.21.15.89:0/984423598 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f406406adb0 0x7f406406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.564+0000 7f4083400700 1 --2- 172.21.15.89:0/984423598 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f406406adb0 0x7f406406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4070009130 tx=0x7f4070008030 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:46.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.567+0000 7f407be00700 1 -- 172.21.15.89:0/984423598 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4074054120 con 0x7f40840ac600 2024-04-02T09:09:46.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.773+0000 7f408b000700 1 -- 172.21.15.89:0/984423598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f408406ea60 con 0x7f40840ac600 2024-04-02T09:09:46.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.774+0000 7f407be00700 1 -- 172.21.15.89:0/984423598 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v205) v1 ==== 74+0+16744 (secure 0 0 0) 0x7f4074057740 con 0x7f40840ac600 2024-04-02T09:09:46.773 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:46.773 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":205,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:09:43.222711+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:09:40.103721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-04-02T09:09:41.165944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:09:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.776+0000 7f408b000700 1 -- 172.21.15.89:0/984423598 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f406406adb0 msgr2=0x7f406406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.776+0000 7f408b000700 1 --2- 172.21.15.89:0/984423598 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f406406adb0 0x7f406406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4070009130 tx=0x7f4070008030 comp rx=0 tx=0).stop 2024-04-02T09:09:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.776+0000 7f408b000700 1 -- 172.21.15.89:0/984423598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40840ac600 msgr2=0x7f408414e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.776+0000 7f408b000700 1 --2- 172.21.15.89:0/984423598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40840ac600 0x7f408414e2c0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f407400b4d0 tx=0x7f407400b7e0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.776+0000 7f408b000700 1 -- 172.21.15.89:0/984423598 shutdown_connections 2024-04-02T09:09:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.776+0000 7f408b000700 1 --2- 172.21.15.89:0/984423598 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f406406adb0 0x7f406406d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.776+0000 7f408b000700 1 --2- 172.21.15.89:0/984423598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4084148830 0x7f40840b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.776+0000 7f408b000700 1 --2- 172.21.15.89:0/984423598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40840ac600 0x7f408414e2c0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.776+0000 7f408b000700 1 --2- 172.21.15.89:0/984423598 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f408415b5d0 0x7f40840ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.776+0000 7f408b000700 1 -- 172.21.15.89:0/984423598 >> 172.21.15.89:0/984423598 conn(0x7f40840cf190 msgr2=0x7f408415e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.776+0000 7f408b000700 1 -- 172.21.15.89:0/984423598 shutdown_connections 2024-04-02T09:09:46.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.776+0000 7f408b000700 1 -- 172.21.15.89:0/984423598 wait complete. 2024-04-02T09:09:46.794 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T09:09:46.794 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:09:46.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.987+0000 7fe40da00700 1 Processor -- start 2024-04-02T09:09:46.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.987+0000 7fe40da00700 1 -- start start 2024-04-02T09:09:46.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.988+0000 7fe40da00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe408160f10 0x7fe4080d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.988+0000 7fe40da00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe408161590 0x7fe408161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.988+0000 7fe40da00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4080ceda0 con 0x7fe4080cf230 2024-04-02T09:09:46.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.988+0000 7fe40da00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe408161f80 con 0x7fe408161590 2024-04-02T09:09:46.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.988+0000 7fe40da00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4081612f0 con 0x7fe408160f10 2024-04-02T09:09:46.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.989+0000 7fe406000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe4080cf230 0x7fe4080d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36546/0 (socket says 172.21.15.89:36546) 2024-04-02T09:09:46.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.989+0000 7fe406000700 1 -- 172.21.15.89:0/4240657479 learned_addr learned my addr 172.21.15.89:0/4240657479 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:46.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.989+0000 7fe407400700 1 --2- 172.21.15.89:0/4240657479 >> v2:172.21.15.89:3300/0 conn(0x7fe408161590 0x7fe408161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.989+0000 7fe406a00700 1 --2- 172.21.15.89:0/4240657479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe408160f10 0x7fe4080d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.989+0000 7fe405600700 1 -- 172.21.15.89:0/4240657479 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305931378 0 0) 0x7fe4080ceda0 con 0x7fe4080cf230 2024-04-02T09:09:46.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.989+0000 7fe405600700 1 -- 172.21.15.89:0/4240657479 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3ec0035b0 con 0x7fe4080cf230 2024-04-02T09:09:46.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.989+0000 7fe407400700 1 -- 172.21.15.89:0/4240657479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe408160f10 msgr2=0x7fe4080d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:46.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.989+0000 7fe407400700 1 --2- 172.21.15.89:0/4240657479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe408160f10 0x7fe4080d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.989+0000 7fe407400700 1 -- 172.21.15.89:0/4240657479 >> v1:172.21.15.89:6789/0 conn(0x7fe4080cf230 legacy=0x7fe4080d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.989+0000 7fe407400700 1 -- 172.21.15.89:0/4240657479 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4080ce470 con 0x7fe408161590 2024-04-02T09:09:46.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.989+0000 7fe407400700 1 --2- 172.21.15.89:0/4240657479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe408161590 0x7fe408161a40 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fe400009dc0 tx=0x7fe40000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e454dd9b6e17ce8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:46.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.989+0000 7fe405600700 1 -- 172.21.15.89:0/4240657479 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40000e070 con 0x7fe408161590 2024-04-02T09:09:46.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.990+0000 7fe405600700 1 -- 172.21.15.89:0/4240657479 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40000bae0 con 0x7fe408161590 2024-04-02T09:09:46.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.990+0000 7fe405600700 1 -- 172.21.15.89:0/4240657479 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40000be90 con 0x7fe408161590 2024-04-02T09:09:46.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.990+0000 7fe40da00700 1 -- 172.21.15.89:0/4240657479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe408161590 msgr2=0x7fe408161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.990+0000 7fe40da00700 1 --2- 172.21.15.89:0/4240657479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe408161590 0x7fe408161a40 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fe400009dc0 tx=0x7fe40000b330 comp rx=0 tx=0).stop 2024-04-02T09:09:46.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.990+0000 7fe40da00700 1 -- 172.21.15.89:0/4240657479 shutdown_connections 2024-04-02T09:09:46.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.990+0000 7fe40da00700 1 --2- 172.21.15.89:0/4240657479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe408161590 0x7fe408161a40 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.990+0000 7fe40da00700 1 --2- 172.21.15.89:0/4240657479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe408160f10 0x7fe4080d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.990+0000 7fe40da00700 1 -- 172.21.15.89:0/4240657479 >> 172.21.15.89:0/4240657479 conn(0x7fe40808aff0 msgr2=0x7fe4080d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:46.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.990+0000 7fe40da00700 1 -- 172.21.15.89:0/4240657479 shutdown_connections 2024-04-02T09:09:46.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.990+0000 7fe40da00700 1 -- 172.21.15.89:0/4240657479 wait complete. 2024-04-02T09:09:46.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.992+0000 7fe40da00700 1 Processor -- start 2024-04-02T09:09:46.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.992+0000 7fe40da00700 1 -- start start 2024-04-02T09:09:46.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.993+0000 7fe40da00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4080cf230 0x7fe40814b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.993+0000 7fe40da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe408160f10 0x7fe4081e7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.993+0000 7fe40da00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe408161590 0x7fe4081e75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.993+0000 7fe40da00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4081e87f0 con 0x7fe408160f10 2024-04-02T09:09:46.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.993+0000 7fe40da00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4081e7be0 con 0x7fe4080cf230 2024-04-02T09:09:46.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.993+0000 7fe40da00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4081e7ec0 con 0x7fe408161590 2024-04-02T09:09:46.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.993+0000 7fe406000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe408160f10 0x7fe4081e7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.993+0000 7fe406000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe408160f10 0x7fe4081e7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40462/0 (socket says 172.21.15.89:40462) 2024-04-02T09:09:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.993+0000 7fe406000700 1 -- 172.21.15.89:0/964939777 learned_addr learned my addr 172.21.15.89:0/964939777 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.993+0000 7fe406000700 1 -- 172.21.15.89:0/964939777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe408161590 msgr2=0x7fe4081e75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.993+0000 7fe407400700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe408161590 0x7fe4081e75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.993+0000 7fe406a00700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4080cf230 0x7fe40814b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.994+0000 7fe406000700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe408161590 0x7fe4081e75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.994+0000 7fe406000700 1 -- 172.21.15.89:0/964939777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4080cf230 msgr2=0x7fe40814b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.994+0000 7fe406000700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4080cf230 0x7fe40814b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.994+0000 7fe406000700 1 -- 172.21.15.89:0/964939777 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe400009a70 con 0x7fe408160f10 2024-04-02T09:09:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.994+0000 7fe407400700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe408161590 0x7fe4081e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.994+0000 7fe406a00700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4080cf230 0x7fe40814b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.994+0000 7fe406000700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe408160f10 0x7fe4081e7070 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8005910 tx=0x7fe3f800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.994+0000 7fe3fea00700 1 -- 172.21.15.89:0/964939777 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8005d70 con 0x7fe408160f10 2024-04-02T09:09:46.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.994+0000 7fe3fea00700 1 -- 172.21.15.89:0/964939777 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f80070d0 con 0x7fe408160f10 2024-04-02T09:09:46.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.994+0000 7fe3fea00700 1 -- 172.21.15.89:0/964939777 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8007440 con 0x7fe408160f10 2024-04-02T09:09:46.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.994+0000 7fe40da00700 1 -- 172.21.15.89:0/964939777 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4081e81a0 con 0x7fe408160f10 2024-04-02T09:09:46.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.994+0000 7fe40da00700 1 -- 172.21.15.89:0/964939777 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe408158e60 con 0x7fe408160f10 2024-04-02T09:09:46.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.995+0000 7fe40da00700 1 -- 172.21.15.89:0/964939777 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4081e8960 con 0x7fe408160f10 2024-04-02T09:09:46.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.996+0000 7fe3fea00700 1 -- 172.21.15.89:0/964939777 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe3f80075a0 con 0x7fe408160f10 2024-04-02T09:09:46.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.997+0000 7fe3fea00700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe3e406adb0 0x7fe3e406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:46.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.997+0000 7fe3fea00700 1 -- 172.21.15.89:0/964939777 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fe3f8088ac0 con 0x7fe408160f10 2024-04-02T09:09:46.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.997+0000 7fe406a00700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe3e406adb0 0x7fe3e406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:46.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:46.998+0000 7fe406a00700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe3e406adb0 0x7fe3e406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0009ac0 tx=0x7fe3f0008040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:47.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.001+0000 7fe3fea00700 1 -- 172.21.15.89:0/964939777 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe3f80539e0 con 0x7fe408160f10 2024-04-02T09:09:47.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.198+0000 7fe40da00700 1 -- 172.21.15.89:0/964939777 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe4080cf6b0 con 0x7fe3e406adb0 2024-04-02T09:09:47.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.208+0000 7fe3fea00700 1 -- 172.21.15.89:0/964939777 <== mgr.10398 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289760 (secure 0 0 0) 0x7fe3f00106e0 con 0x7fe3e406adb0 2024-04-02T09:09:47.211 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:47.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.212+0000 7fe40da00700 1 -- 172.21.15.89:0/964939777 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe3e406adb0 msgr2=0x7fe3e406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:47.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.212+0000 7fe40da00700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe3e406adb0 0x7fe3e406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0009ac0 tx=0x7fe3f0008040 comp rx=0 tx=0).stop 2024-04-02T09:09:47.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.212+0000 7fe40da00700 1 -- 172.21.15.89:0/964939777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe408160f10 msgr2=0x7fe4081e7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:47.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.212+0000 7fe40da00700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe408160f10 0x7fe4081e7070 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8005910 tx=0x7fe3f800c860 comp rx=0 tx=0).stop 2024-04-02T09:09:47.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.212+0000 7fe40da00700 1 -- 172.21.15.89:0/964939777 shutdown_connections 2024-04-02T09:09:47.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.212+0000 7fe40da00700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe3e406adb0 0x7fe3e406d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.212+0000 7fe40da00700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4080cf230 0x7fe40814b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.212+0000 7fe40da00700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe408161590 0x7fe4081e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.212+0000 7fe40da00700 1 --2- 172.21.15.89:0/964939777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe408160f10 0x7fe4081e7070 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.212+0000 7fe40da00700 1 -- 172.21.15.89:0/964939777 >> 172.21.15.89:0/964939777 conn(0x7fe40808aff0 msgr2=0x7fe4080cb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:47.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.213+0000 7fe40da00700 1 -- 172.21.15.89:0/964939777 shutdown_connections 2024-04-02T09:09:47.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.213+0000 7fe40da00700 1 -- 172.21.15.89:0/964939777 wait complete. 2024-04-02T09:09:47.215 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:09:47.231 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T09:09:46.103035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":400640,"kb_used_data":21232,"kb_used_omap":12,"kb_used_meta":379251,"kb_avail":754574080,"statfs":{"total":773094113280,"available":772683857920,"internally_reserved":0,"allocated":21741568,"data_stored":11542318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":388353592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.427265"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786087+0000","last_change":"2024-04-02T09:09:42.242472+0000","last_active":"2024-04-02T09:09:43.786087+0000","last_peered":"2024-04-02T09:09:43.786087+0000","last_clean":"2024-04-02T09:09:43.786087+0000","last_became_active":"2024-04-02T09:09:42.241872+0000","last_became_peered":"2024-04-02T09:09:42.241872+0000","last_unstale":"2024-04-02T09:09:43.786087+0000","last_undegraded":"2024-04-02T09:09:43.786087+0000","last_fullsized":"2024-04-02T09:09:43.786087+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:55.536352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235964+0000","last_change":"2024-04-02T09:09:42.250584+0000","last_active":"2024-04-02T09:09:43.235964+0000","last_peered":"2024-04-02T09:09:43.235964+0000","last_clean":"2024-04-02T09:09:43.235964+0000","last_became_active":"2024-04-02T09:09:42.250293+0000","last_became_peered":"2024-04-02T09:09:42.250293+0000","last_unstale":"2024-04-02T09:09:43.235964+0000","last_undegraded":"2024-04-02T09:09:43.235964+0000","last_fullsized":"2024-04-02T09:09:43.235964+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:37:41.368117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595689+0000","last_change":"2024-04-02T09:09:42.239005+0000","last_active":"2024-04-02T09:09:43.595689+0000","last_peered":"2024-04-02T09:09:43.595689+0000","last_clean":"2024-04-02T09:09:43.595689+0000","last_became_active":"2024-04-02T09:09:42.238737+0000","last_became_peered":"2024-04-02T09:09:42.238737+0000","last_unstale":"2024-04-02T09:09:43.595689+0000","last_undegraded":"2024-04-02T09:09:43.595689+0000","last_fullsized":"2024-04-02T09:09:43.595689+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:20:11.575610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235737+0000","last_change":"2024-04-02T09:09:42.253202+0000","last_active":"2024-04-02T09:09:43.235737+0000","last_peered":"2024-04-02T09:09:43.235737+0000","last_clean":"2024-04-02T09:09:43.235737+0000","last_became_active":"2024-04-02T09:09:42.251249+0000","last_became_peered":"2024-04-02T09:09:42.251249+0000","last_unstale":"2024-04-02T09:09:43.235737+0000","last_undegraded":"2024-04-02T09:09:43.235737+0000","last_fullsized":"2024-04-02T09:09:43.235737+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:03:36.858777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786940+0000","last_change":"2024-04-02T09:09:42.251147+0000","last_active":"2024-04-02T09:09:43.786940+0000","last_peered":"2024-04-02T09:09:43.786940+0000","last_clean":"2024-04-02T09:09:43.786940+0000","last_became_active":"2024-04-02T09:09:42.247444+0000","last_became_peered":"2024-04-02T09:09:42.247444+0000","last_unstale":"2024-04-02T09:09:43.786940+0000","last_undegraded":"2024-04-02T09:09:43.786940+0000","last_fullsized":"2024-04-02T09:09:43.786940+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:18.060077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235545+0000","last_change":"2024-04-02T09:09:42.237780+0000","last_active":"2024-04-02T09:09:43.235545+0000","last_peered":"2024-04-02T09:09:43.235545+0000","last_clean":"2024-04-02T09:09:43.235545+0000","last_became_active":"2024-04-02T09:09:42.235583+0000","last_became_peered":"2024-04-02T09:09:42.235583+0000","last_unstale":"2024-04-02T09:09:43.235545+0000","last_undegraded":"2024-04-02T09:09:43.235545+0000","last_fullsized":"2024-04-02T09:09:43.235545+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:35:08.812169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.233152+0000","last_change":"2024-04-02T09:09:42.244707+0000","last_active":"2024-04-02T09:09:43.233152+0000","last_peered":"2024-04-02T09:09:43.233152+0000","last_clean":"2024-04-02T09:09:43.233152+0000","last_became_active":"2024-04-02T09:09:42.244291+0000","last_became_peered":"2024-04-02T09:09:42.244291+0000","last_unstale":"2024-04-02T09:09:43.233152+0000","last_undegraded":"2024-04-02T09:09:43.233152+0000","last_fullsized":"2024-04-02T09:09:43.233152+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:51:52.730338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786231+0000","last_change":"2024-04-02T09:09:42.251079+0000","last_active":"2024-04-02T09:09:43.786231+0000","last_peered":"2024-04-02T09:09:43.786231+0000","last_clean":"2024-04-02T09:09:43.786231+0000","last_became_active":"2024-04-02T09:09:42.244304+0000","last_became_peered":"2024-04-02T09:09:42.244304+0000","last_unstale":"2024-04-02T09:09:43.786231+0000","last_undegraded":"2024-04-02T09:09:43.786231+0000","last_fullsized":"2024-04-02T09:09:43.786231+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:53:16.796340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235916+0000","last_change":"2024-04-02T09:09:42.253842+0000","last_active":"2024-04-02T09:09:43.235916+0000","last_peered":"2024-04-02T09:09:43.235916+0000","last_clean":"2024-04-02T09:09:43.235916+0000","last_became_active":"2024-04-02T09:09:42.253572+0000","last_became_peered":"2024-04-02T09:09:42.253572+0000","last_unstale":"2024-04-02T09:09:43.235916+0000","last_undegraded":"2024-04-02T09:09:43.235916+0000","last_fullsized":"2024-04-02T09:09:43.235916+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:41:17.862841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787108+0000","last_change":"2024-04-02T09:09:42.246345+0000","last_active":"2024-04-02T09:09:43.787108+0000","last_peered":"2024-04-02T09:09:43.787108+0000","last_clean":"2024-04-02T09:09:43.787108+0000","last_became_active":"2024-04-02T09:09:42.246054+0000","last_became_peered":"2024-04-02T09:09:42.246054+0000","last_unstale":"2024-04-02T09:09:43.787108+0000","last_undegraded":"2024-04-02T09:09:43.787108+0000","last_fullsized":"2024-04-02T09:09:43.787108+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:27.369585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785824+0000","last_change":"2024-04-02T09:09:42.254714+0000","last_active":"2024-04-02T09:09:43.785824+0000","last_peered":"2024-04-02T09:09:43.785824+0000","last_clean":"2024-04-02T09:09:43.785824+0000","last_became_active":"2024-04-02T09:09:42.254317+0000","last_became_peered":"2024-04-02T09:09:42.254317+0000","last_unstale":"2024-04-02T09:09:43.785824+0000","last_undegraded":"2024-04-02T09:09:43.785824+0000","last_fullsized":"2024-04-02T09:09:43.785824+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:30:27.478757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786341+0000","last_change":"2024-04-02T09:09:42.248010+0000","last_active":"2024-04-02T09:09:43.786341+0000","last_peered":"2024-04-02T09:09:43.786341+0000","last_clean":"2024-04-02T09:09:43.786341+0000","last_became_active":"2024-04-02T09:09:42.247251+0000","last_became_peered":"2024-04-02T09:09:42.247251+0000","last_unstale":"2024-04-02T09:09:43.786341+0000","last_undegraded":"2024-04-02T09:09:43.786341+0000","last_fullsized":"2024-04-02T09:09:43.786341+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:46.919953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785799+0000","last_change":"2024-04-02T09:09:42.240712+0000","last_active":"2024-04-02T09:09:43.785799+0000","last_peered":"2024-04-02T09:09:43.785799+0000","last_clean":"2024-04-02T09:09:43.785799+0000","last_became_active":"2024-04-02T09:09:42.240458+0000","last_became_peered":"2024-04-02T09:09:42.240458+0000","last_unstale":"2024-04-02T09:09:43.785799+0000","last_undegraded":"2024-04-02T09:09:43.785799+0000","last_fullsized":"2024-04-02T09:09:43.785799+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:15:14.241480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235547+0000","last_change":"2024-04-02T09:09:42.246629+0000","last_active":"2024-04-02T09:09:43.235547+0000","last_peered":"2024-04-02T09:09:43.235547+0000","last_clean":"2024-04-02T09:09:43.235547+0000","last_became_active":"2024-04-02T09:09:42.245713+0000","last_became_peered":"2024-04-02T09:09:42.245713+0000","last_unstale":"2024-04-02T09:09:43.235547+0000","last_undegraded":"2024-04-02T09:09:43.235547+0000","last_fullsized":"2024-04-02T09:09:43.235547+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:09:38.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785765+0000","last_change":"2024-04-02T09:09:42.252187+0000","last_active":"2024-04-02T09:09:43.785765+0000","last_peered":"2024-04-02T09:09:43.785765+0000","last_clean":"2024-04-02T09:09:43.785765+0000","last_became_active":"2024-04-02T09:09:42.250028+0000","last_became_peered":"2024-04-02T09:09:42.250028+0000","last_unstale":"2024-04-02T09:09:43.785765+0000","last_undegraded":"2024-04-02T09:09:43.785765+0000","last_fullsized":"2024-04-02T09:09:43.785765+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:41:37.653634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595437+0000","last_change":"2024-04-02T09:09:42.245600+0000","last_active":"2024-04-02T09:09:43.595437+0000","last_peered":"2024-04-02T09:09:43.595437+0000","last_clean":"2024-04-02T09:09:43.595437+0000","last_became_active":"2024-04-02T09:09:42.245321+0000","last_became_peered":"2024-04-02T09:09:42.245321+0000","last_unstale":"2024-04-02T09:09:43.595437+0000","last_undegraded":"2024-04-02T09:09:43.595437+0000","last_fullsized":"2024-04-02T09:09:43.595437+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:49:00.800523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235812+0000","last_change":"2024-04-02T09:09:42.242356+0000","last_active":"2024-04-02T09:09:43.235812+0000","last_peered":"2024-04-02T09:09:43.235812+0000","last_clean":"2024-04-02T09:09:43.235812+0000","last_became_active":"2024-04-02T09:09:42.242074+0000","last_became_peered":"2024-04-02T09:09:42.242074+0000","last_unstale":"2024-04-02T09:09:43.235812+0000","last_undegraded":"2024-04-02T09:09:43.235812+0000","last_fullsized":"2024-04-02T09:09:43.235812+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:02.167323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.233742+0000","last_change":"2024-04-02T09:09:42.258095+0000","last_active":"2024-04-02T09:09:43.233742+0000","last_peered":"2024-04-02T09:09:43.233742+0000","last_clean":"2024-04-02T09:09:43.233742+0000","last_became_active":"2024-04-02T09:09:42.257816+0000","last_became_peered":"2024-04-02T09:09:42.257816+0000","last_unstale":"2024-04-02T09:09:43.233742+0000","last_undegraded":"2024-04-02T09:09:43.233742+0000","last_fullsized":"2024-04-02T09:09:43.233742+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:56:51.798108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595888+0000","last_change":"2024-04-02T09:09:42.253363+0000","last_active":"2024-04-02T09:09:43.595888+0000","last_peered":"2024-04-02T09:09:43.595888+0000","last_clean":"2024-04-02T09:09:43.595888+0000","last_became_active":"2024-04-02T09:09:42.253071+0000","last_became_peered":"2024-04-02T09:09:42.253071+0000","last_unstale":"2024-04-02T09:09:43.595888+0000","last_undegraded":"2024-04-02T09:09:43.595888+0000","last_fullsized":"2024-04-02T09:09:43.595888+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:07:15.570245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786744+0000","last_change":"2024-04-02T09:09:42.238430+0000","last_active":"2024-04-02T09:09:43.786744+0000","last_peered":"2024-04-02T09:09:43.786744+0000","last_clean":"2024-04-02T09:09:43.786744+0000","last_became_active":"2024-04-02T09:09:42.238133+0000","last_became_peered":"2024-04-02T09:09:42.238133+0000","last_unstale":"2024-04-02T09:09:43.786744+0000","last_undegraded":"2024-04-02T09:09:43.786744+0000","last_fullsized":"2024-04-02T09:09:43.786744+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:30:07.695501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595547+0000","last_change":"2024-04-02T09:09:42.247622+0000","last_active":"2024-04-02T09:09:43.595547+0000","last_peered":"2024-04-02T09:09:43.595547+0000","last_clean":"2024-04-02T09:09:43.595547+0000","last_became_active":"2024-04-02T09:09:42.246824+0000","last_became_peered":"2024-04-02T09:09:42.246824+0000","last_unstale":"2024-04-02T09:09:43.595547+0000","last_undegraded":"2024-04-02T09:09:43.595547+0000","last_fullsized":"2024-04-02T09:09:43.595547+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:13:09.717628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786683+0000","last_change":"2024-04-02T09:09:42.247286+0000","last_active":"2024-04-02T09:09:43.786683+0000","last_peered":"2024-04-02T09:09:43.786683+0000","last_clean":"2024-04-02T09:09:43.786683+0000","last_became_active":"2024-04-02T09:09:42.245830+0000","last_became_peered":"2024-04-02T09:09:42.245830+0000","last_unstale":"2024-04-02T09:09:43.786683+0000","last_undegraded":"2024-04-02T09:09:43.786683+0000","last_fullsized":"2024-04-02T09:09:43.786683+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:54:22.001281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786323+0000","last_change":"2024-04-02T09:09:42.247444+0000","last_active":"2024-04-02T09:09:43.786323+0000","last_peered":"2024-04-02T09:09:43.786323+0000","last_clean":"2024-04-02T09:09:43.786323+0000","last_became_active":"2024-04-02T09:09:42.246972+0000","last_became_peered":"2024-04-02T09:09:42.246972+0000","last_unstale":"2024-04-02T09:09:43.786323+0000","last_undegraded":"2024-04-02T09:09:43.786323+0000","last_fullsized":"2024-04-02T09:09:43.786323+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:38:27.306368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786533+0000","last_change":"2024-04-02T09:09:42.231733+0000","last_active":"2024-04-02T09:09:43.786533+0000","last_peered":"2024-04-02T09:09:43.786533+0000","last_clean":"2024-04-02T09:09:43.786533+0000","last_became_active":"2024-04-02T09:09:42.231449+0000","last_became_peered":"2024-04-02T09:09:42.231449+0000","last_unstale":"2024-04-02T09:09:43.786533+0000","last_undegraded":"2024-04-02T09:09:43.786533+0000","last_fullsized":"2024-04-02T09:09:43.786533+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:11:05.741327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235990+0000","last_change":"2024-04-02T09:09:42.253498+0000","last_active":"2024-04-02T09:09:43.235990+0000","last_peered":"2024-04-02T09:09:43.235990+0000","last_clean":"2024-04-02T09:09:43.235990+0000","last_became_active":"2024-04-02T09:09:42.253165+0000","last_became_peered":"2024-04-02T09:09:42.253165+0000","last_unstale":"2024-04-02T09:09:43.235990+0000","last_undegraded":"2024-04-02T09:09:43.235990+0000","last_fullsized":"2024-04-02T09:09:43.235990+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:22.111661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786524+0000","last_change":"2024-04-02T09:09:42.242893+0000","last_active":"2024-04-02T09:09:43.786524+0000","last_peered":"2024-04-02T09:09:43.786524+0000","last_clean":"2024-04-02T09:09:43.786524+0000","last_became_active":"2024-04-02T09:09:42.242636+0000","last_became_peered":"2024-04-02T09:09:42.242636+0000","last_unstale":"2024-04-02T09:09:43.786524+0000","last_undegraded":"2024-04-02T09:09:43.786524+0000","last_fullsized":"2024-04-02T09:09:43.786524+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:33.746069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595704+0000","last_change":"2024-04-02T09:09:42.253299+0000","last_active":"2024-04-02T09:09:43.595704+0000","last_peered":"2024-04-02T09:09:43.595704+0000","last_clean":"2024-04-02T09:09:43.595704+0000","last_became_active":"2024-04-02T09:09:42.253029+0000","last_became_peered":"2024-04-02T09:09:42.253029+0000","last_unstale":"2024-04-02T09:09:43.595704+0000","last_undegraded":"2024-04-02T09:09:43.595704+0000","last_fullsized":"2024-04-02T09:09:43.595704+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:16:24.179056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786756+0000","last_change":"2024-04-02T09:09:42.243890+0000","last_active":"2024-04-02T09:09:43.786756+0000","last_peered":"2024-04-02T09:09:43.786756+0000","last_clean":"2024-04-02T09:09:43.786756+0000","last_became_active":"2024-04-02T09:09:42.243633+0000","last_became_peered":"2024-04-02T09:09:42.243633+0000","last_unstale":"2024-04-02T09:09:43.786756+0000","last_undegraded":"2024-04-02T09:09:43.786756+0000","last_fullsized":"2024-04-02T09:09:43.786756+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:25.782610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786277+0000","last_change":"2024-04-02T09:09:42.245543+0000","last_active":"2024-04-02T09:09:43.786277+0000","last_peered":"2024-04-02T09:09:43.786277+0000","last_clean":"2024-04-02T09:09:43.786277+0000","last_became_active":"2024-04-02T09:09:42.245268+0000","last_became_peered":"2024-04-02T09:09:42.245268+0000","last_unstale":"2024-04-02T09:09:43.786277+0000","last_undegraded":"2024-04-02T09:09:43.786277+0000","last_fullsized":"2024-04-02T09:09:43.786277+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:40:30.420225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785958+0000","last_change":"2024-04-02T09:09:42.243451+0000","last_active":"2024-04-02T09:09:43.785958+0000","last_peered":"2024-04-02T09:09:43.785958+0000","last_clean":"2024-04-02T09:09:43.785958+0000","last_became_active":"2024-04-02T09:09:42.243195+0000","last_became_peered":"2024-04-02T09:09:42.243195+0000","last_unstale":"2024-04-02T09:09:43.785958+0000","last_undegraded":"2024-04-02T09:09:43.785958+0000","last_fullsized":"2024-04-02T09:09:43.785958+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:04.400069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787245+0000","last_change":"2024-04-02T09:09:42.246514+0000","last_active":"2024-04-02T09:09:43.787245+0000","last_peered":"2024-04-02T09:09:43.787245+0000","last_clean":"2024-04-02T09:09:43.787245+0000","last_became_active":"2024-04-02T09:09:42.246180+0000","last_became_peered":"2024-04-02T09:09:42.246180+0000","last_unstale":"2024-04-02T09:09:43.787245+0000","last_undegraded":"2024-04-02T09:09:43.787245+0000","last_fullsized":"2024-04-02T09:09:43.787245+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:01:33.122142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786239+0000","last_change":"2024-04-02T09:09:42.245998+0000","last_active":"2024-04-02T09:09:43.786239+0000","last_peered":"2024-04-02T09:09:43.786239+0000","last_clean":"2024-04-02T09:09:43.786239+0000","last_became_active":"2024-04-02T09:09:42.243896+0000","last_became_peered":"2024-04-02T09:09:42.243896+0000","last_unstale":"2024-04-02T09:09:43.786239+0000","last_undegraded":"2024-04-02T09:09:43.786239+0000","last_fullsized":"2024-04-02T09:09:43.786239+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:47.798366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787198+0000","last_change":"2024-04-02T09:09:42.242627+0000","last_active":"2024-04-02T09:09:43.787198+0000","last_peered":"2024-04-02T09:09:43.787198+0000","last_clean":"2024-04-02T09:09:43.787198+0000","last_became_active":"2024-04-02T09:09:42.242045+0000","last_became_peered":"2024-04-02T09:09:42.242045+0000","last_unstale":"2024-04-02T09:09:43.787198+0000","last_undegraded":"2024-04-02T09:09:43.787198+0000","last_fullsized":"2024-04-02T09:09:43.787198+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:10:42.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1f","version":"204'4","reported_seq":15,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.596036+0000","last_change":"2024-04-02T09:09:41.701429+0000","last_active":"2024-04-02T09:09:43.596036+0000","last_peered":"2024-04-02T09:09:43.596036+0000","last_clean":"2024-04-02T09:09:43.596036+0000","last_became_active":"2024-04-02T09:09:41.700916+0000","last_became_peered":"2024-04-02T09:09:41.700916+0000","last_unstale":"2024-04-02T09:09:43.596036+0000","last_undegraded":"2024-04-02T09:09:43.596036+0000","last_fullsized":"2024-04-02T09:09:43.596036+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:35:07.805757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787321+0000","last_change":"2024-04-02T09:09:42.247954+0000","last_active":"2024-04-02T09:09:43.787321+0000","last_peered":"2024-04-02T09:09:43.787321+0000","last_clean":"2024-04-02T09:09:43.787321+0000","last_became_active":"2024-04-02T09:09:42.246592+0000","last_became_peered":"2024-04-02T09:09:42.246592+0000","last_unstale":"2024-04-02T09:09:43.787321+0000","last_undegraded":"2024-04-02T09:09:43.787321+0000","last_fullsized":"2024-04-02T09:09:43.787321+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:45:38.604503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1e","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786613+0000","last_change":"2024-04-02T09:09:41.702270+0000","last_active":"2024-04-02T09:09:43.786613+0000","last_peered":"2024-04-02T09:09:43.786613+0000","last_clean":"2024-04-02T09:09:43.786613+0000","last_became_active":"2024-04-02T09:09:41.701889+0000","last_became_peered":"2024-04-02T09:09:41.701889+0000","last_unstale":"2024-04-02T09:09:43.786613+0000","last_undegraded":"2024-04-02T09:09:43.786613+0000","last_fullsized":"2024-04-02T09:09:43.786613+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:24:53.906210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235798+0000","last_change":"2024-04-02T09:09:42.249222+0000","last_active":"2024-04-02T09:09:43.235798+0000","last_peered":"2024-04-02T09:09:43.235798+0000","last_clean":"2024-04-02T09:09:43.235798+0000","last_became_active":"2024-04-02T09:09:42.248961+0000","last_became_peered":"2024-04-02T09:09:42.248961+0000","last_unstale":"2024-04-02T09:09:43.235798+0000","last_undegraded":"2024-04-02T09:09:43.235798+0000","last_fullsized":"2024-04-02T09:09:43.235798+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:47:04.855474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1d","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786601+0000","last_change":"2024-04-02T09:09:41.702154+0000","last_active":"2024-04-02T09:09:43.786601+0000","last_peered":"2024-04-02T09:09:43.786601+0000","last_clean":"2024-04-02T09:09:43.786601+0000","last_became_active":"2024-04-02T09:09:41.700854+0000","last_became_peered":"2024-04-02T09:09:41.700854+0000","last_unstale":"2024-04-02T09:09:43.786601+0000","last_undegraded":"2024-04-02T09:09:43.786601+0000","last_fullsized":"2024-04-02T09:09:43.786601+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:26.363173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235861+0000","last_change":"2024-04-02T09:09:42.257248+0000","last_active":"2024-04-02T09:09:43.235861+0000","last_peered":"2024-04-02T09:09:43.235861+0000","last_clean":"2024-04-02T09:09:43.235861+0000","last_became_active":"2024-04-02T09:09:42.254749+0000","last_became_peered":"2024-04-02T09:09:42.254749+0000","last_unstale":"2024-04-02T09:09:43.235861+0000","last_undegraded":"2024-04-02T09:09:43.235861+0000","last_fullsized":"2024-04-02T09:09:43.235861+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:25:36.871271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1c","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786451+0000","last_change":"2024-04-02T09:09:41.703057+0000","last_active":"2024-04-02T09:09:43.786451+0000","last_peered":"2024-04-02T09:09:43.786451+0000","last_clean":"2024-04-02T09:09:43.786451+0000","last_became_active":"2024-04-02T09:09:41.702657+0000","last_became_peered":"2024-04-02T09:09:41.702657+0000","last_unstale":"2024-04-02T09:09:43.786451+0000","last_undegraded":"2024-04-02T09:09:43.786451+0000","last_fullsized":"2024-04-02T09:09:43.786451+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:12:57.221082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235923+0000","last_change":"2024-04-02T09:09:42.238974+0000","last_active":"2024-04-02T09:09:43.235923+0000","last_peered":"2024-04-02T09:09:43.235923+0000","last_clean":"2024-04-02T09:09:43.235923+0000","last_became_active":"2024-04-02T09:09:42.238143+0000","last_became_peered":"2024-04-02T09:09:42.238143+0000","last_unstale":"2024-04-02T09:09:43.235923+0000","last_undegraded":"2024-04-02T09:09:43.235923+0000","last_fullsized":"2024-04-02T09:09:43.235923+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:18.060077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786657+0000","last_change":"2024-04-02T09:09:41.697992+0000","last_active":"2024-04-02T09:09:43.786657+0000","last_peered":"2024-04-02T09:09:43.786657+0000","last_clean":"2024-04-02T09:09:43.786657+0000","last_became_active":"2024-04-02T09:09:41.696876+0000","last_became_peered":"2024-04-02T09:09:41.696876+0000","last_unstale":"2024-04-02T09:09:43.786657+0000","last_undegraded":"2024-04-02T09:09:43.786657+0000","last_fullsized":"2024-04-02T09:09:43.786657+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:06:25.992709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786475+0000","last_change":"2024-04-02T09:09:42.252164+0000","last_active":"2024-04-02T09:09:43.786475+0000","last_peered":"2024-04-02T09:09:43.786475+0000","last_clean":"2024-04-02T09:09:43.786475+0000","last_became_active":"2024-04-02T09:09:42.248376+0000","last_became_peered":"2024-04-02T09:09:42.248376+0000","last_unstale":"2024-04-02T09:09:43.786475+0000","last_undegraded":"2024-04-02T09:09:43.786475+0000","last_fullsized":"2024-04-02T09:09:43.786475+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:07:11.878538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1a","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786582+0000","last_change":"2024-04-02T09:09:41.703163+0000","last_active":"2024-04-02T09:09:43.786582+0000","last_peered":"2024-04-02T09:09:43.786582+0000","last_clean":"2024-04-02T09:09:43.786582+0000","last_became_active":"2024-04-02T09:09:41.702886+0000","last_became_peered":"2024-04-02T09:09:41.702886+0000","last_unstale":"2024-04-02T09:09:43.786582+0000","last_undegraded":"2024-04-02T09:09:43.786582+0000","last_fullsized":"2024-04-02T09:09:43.786582+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:59.773755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785813+0000","last_change":"2024-04-02T09:09:42.233652+0000","last_active":"2024-04-02T09:09:43.785813+0000","last_peered":"2024-04-02T09:09:43.785813+0000","last_clean":"2024-04-02T09:09:43.785813+0000","last_became_active":"2024-04-02T09:09:42.233275+0000","last_became_peered":"2024-04-02T09:09:42.233275+0000","last_unstale":"2024-04-02T09:09:43.785813+0000","last_undegraded":"2024-04-02T09:09:43.785813+0000","last_fullsized":"2024-04-02T09:09:43.785813+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:32:15.281056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.233142+0000","last_change":"2024-04-02T09:09:41.689454+0000","last_active":"2024-04-02T09:09:43.233142+0000","last_peered":"2024-04-02T09:09:43.233142+0000","last_clean":"2024-04-02T09:09:43.233142+0000","last_became_active":"2024-04-02T09:09:41.689187+0000","last_became_peered":"2024-04-02T09:09:41.689187+0000","last_unstale":"2024-04-02T09:09:43.233142+0000","last_undegraded":"2024-04-02T09:09:43.233142+0000","last_fullsized":"2024-04-02T09:09:43.233142+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:58.000014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595843+0000","last_change":"2024-04-02T09:09:42.244345+0000","last_active":"2024-04-02T09:09:43.595843+0000","last_peered":"2024-04-02T09:09:43.595843+0000","last_clean":"2024-04-02T09:09:43.595843+0000","last_became_active":"2024-04-02T09:09:42.244075+0000","last_became_peered":"2024-04-02T09:09:42.244075+0000","last_unstale":"2024-04-02T09:09:43.595843+0000","last_undegraded":"2024-04-02T09:09:43.595843+0000","last_fullsized":"2024-04-02T09:09:43.595843+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:35:14.920256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786052+0000","last_change":"2024-04-02T09:09:41.704076+0000","last_active":"2024-04-02T09:09:43.786052+0000","last_peered":"2024-04-02T09:09:43.786052+0000","last_clean":"2024-04-02T09:09:43.786052+0000","last_became_active":"2024-04-02T09:09:41.702808+0000","last_became_peered":"2024-04-02T09:09:41.702808+0000","last_unstale":"2024-04-02T09:09:43.786052+0000","last_undegraded":"2024-04-02T09:09:43.786052+0000","last_fullsized":"2024-04-02T09:09:43.786052+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:45.913542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787337+0000","last_change":"2024-04-02T09:09:42.247947+0000","last_active":"2024-04-02T09:09:43.787337+0000","last_peered":"2024-04-02T09:09:43.787337+0000","last_clean":"2024-04-02T09:09:43.787337+0000","last_became_active":"2024-04-02T09:09:42.247636+0000","last_became_peered":"2024-04-02T09:09:42.247636+0000","last_unstale":"2024-04-02T09:09:43.787337+0000","last_undegraded":"2024-04-02T09:09:43.787337+0000","last_fullsized":"2024-04-02T09:09:43.787337+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:16.873370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787326+0000","last_change":"2024-04-02T09:09:41.701973+0000","last_active":"2024-04-02T09:09:43.787326+0000","last_peered":"2024-04-02T09:09:43.787326+0000","last_clean":"2024-04-02T09:09:43.787326+0000","last_became_active":"2024-04-02T09:09:41.701605+0000","last_became_peered":"2024-04-02T09:09:41.701605+0000","last_unstale":"2024-04-02T09:09:43.787326+0000","last_undegraded":"2024-04-02T09:09:43.787326+0000","last_fullsized":"2024-04-02T09:09:43.787326+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:18:06.740716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786342+0000","last_change":"2024-04-02T09:09:42.247840+0000","last_active":"2024-04-02T09:09:43.786342+0000","last_peered":"2024-04-02T09:09:43.786342+0000","last_clean":"2024-04-02T09:09:43.786342+0000","last_became_active":"2024-04-02T09:09:42.246872+0000","last_became_peered":"2024-04-02T09:09:42.246872+0000","last_unstale":"2024-04-02T09:09:43.786342+0000","last_undegraded":"2024-04-02T09:09:43.786342+0000","last_fullsized":"2024-04-02T09:09:43.786342+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:44:23.519150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.15","version":"204'3","reported_seq":14,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786699+0000","last_change":"2024-04-02T09:09:41.706298+0000","last_active":"2024-04-02T09:09:43.786699+0000","last_peered":"2024-04-02T09:09:43.786699+0000","last_clean":"2024-04-02T09:09:43.786699+0000","last_became_active":"2024-04-02T09:09:41.705296+0000","last_became_peered":"2024-04-02T09:09:41.705296+0000","last_unstale":"2024-04-02T09:09:43.786699+0000","last_undegraded":"2024-04-02T09:09:43.786699+0000","last_fullsized":"2024-04-02T09:09:43.786699+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:00:21.176162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787106+0000","last_change":"2024-04-02T09:09:42.242612+0000","last_active":"2024-04-02T09:09:43.787106+0000","last_peered":"2024-04-02T09:09:43.787106+0000","last_clean":"2024-04-02T09:09:43.787106+0000","last_became_active":"2024-04-02T09:09:42.241681+0000","last_became_peered":"2024-04-02T09:09:42.241681+0000","last_unstale":"2024-04-02T09:09:43.787106+0000","last_undegraded":"2024-04-02T09:09:43.787106+0000","last_fullsized":"2024-04-02T09:09:43.787106+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:04:12.501107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.14","version":"204'6","reported_seq":17,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786850+0000","last_change":"2024-04-02T09:09:41.702571+0000","last_active":"2024-04-02T09:09:43.786850+0000","last_peered":"2024-04-02T09:09:43.786850+0000","last_clean":"2024-04-02T09:09:43.786850+0000","last_became_active":"2024-04-02T09:09:41.702287+0000","last_became_peered":"2024-04-02T09:09:41.702287+0000","last_unstale":"2024-04-02T09:09:43.786850+0000","last_undegraded":"2024-04-02T09:09:43.786850+0000","last_fullsized":"2024-04-02T09:09:43.786850+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:33:51.218638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786757+0000","last_change":"2024-04-02T09:09:42.245949+0000","last_active":"2024-04-02T09:09:43.786757+0000","last_peered":"2024-04-02T09:09:43.786757+0000","last_clean":"2024-04-02T09:09:43.786757+0000","last_became_active":"2024-04-02T09:09:42.245555+0000","last_became_peered":"2024-04-02T09:09:42.245555+0000","last_unstale":"2024-04-02T09:09:43.786757+0000","last_undegraded":"2024-04-02T09:09:43.786757+0000","last_fullsized":"2024-04-02T09:09:43.786757+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:32:36.042213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.13","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786307+0000","last_change":"2024-04-02T09:09:41.699908+0000","last_active":"2024-04-02T09:09:43.786307+0000","last_peered":"2024-04-02T09:09:43.786307+0000","last_clean":"2024-04-02T09:09:43.786307+0000","last_became_active":"2024-04-02T09:09:41.699215+0000","last_became_peered":"2024-04-02T09:09:41.699215+0000","last_unstale":"2024-04-02T09:09:43.786307+0000","last_undegraded":"2024-04-02T09:09:43.786307+0000","last_fullsized":"2024-04-02T09:09:43.786307+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:12.251299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.233449+0000","last_change":"2024-04-02T09:09:42.248728+0000","last_active":"2024-04-02T09:09:43.233449+0000","last_peered":"2024-04-02T09:09:43.233449+0000","last_clean":"2024-04-02T09:09:43.233449+0000","last_became_active":"2024-04-02T09:09:42.248479+0000","last_became_peered":"2024-04-02T09:09:42.248479+0000","last_unstale":"2024-04-02T09:09:43.233449+0000","last_undegraded":"2024-04-02T09:09:43.233449+0000","last_fullsized":"2024-04-02T09:09:43.233449+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:25.782610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786841+0000","last_change":"2024-04-02T09:09:41.703075+0000","last_active":"2024-04-02T09:09:43.786841+0000","last_peered":"2024-04-02T09:09:43.786841+0000","last_clean":"2024-04-02T09:09:43.786841+0000","last_became_active":"2024-04-02T09:09:41.702419+0000","last_became_peered":"2024-04-02T09:09:41.702419+0000","last_unstale":"2024-04-02T09:09:43.786841+0000","last_undegraded":"2024-04-02T09:09:43.786841+0000","last_fullsized":"2024-04-02T09:09:43.786841+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:47:26.806954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785983+0000","last_change":"2024-04-02T09:09:42.245777+0000","last_active":"2024-04-02T09:09:43.785983+0000","last_peered":"2024-04-02T09:09:43.785983+0000","last_clean":"2024-04-02T09:09:43.785983+0000","last_became_active":"2024-04-02T09:09:42.244519+0000","last_became_peered":"2024-04-02T09:09:42.244519+0000","last_unstale":"2024-04-02T09:09:43.785983+0000","last_undegraded":"2024-04-02T09:09:43.785983+0000","last_fullsized":"2024-04-02T09:09:43.785983+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:36:14.965927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787106+0000","last_change":"2024-04-02T09:09:41.698008+0000","last_active":"2024-04-02T09:09:43.787106+0000","last_peered":"2024-04-02T09:09:43.787106+0000","last_clean":"2024-04-02T09:09:43.787106+0000","last_became_active":"2024-04-02T09:09:41.697689+0000","last_became_peered":"2024-04-02T09:09:41.697689+0000","last_unstale":"2024-04-02T09:09:43.787106+0000","last_undegraded":"2024-04-02T09:09:43.787106+0000","last_fullsized":"2024-04-02T09:09:43.787106+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:55.340371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786910+0000","last_change":"2024-04-02T09:09:42.252918+0000","last_active":"2024-04-02T09:09:43.786910+0000","last_peered":"2024-04-02T09:09:43.786910+0000","last_clean":"2024-04-02T09:09:43.786910+0000","last_became_active":"2024-04-02T09:09:42.252622+0000","last_became_peered":"2024-04-02T09:09:42.252622+0000","last_unstale":"2024-04-02T09:09:43.786910+0000","last_undegraded":"2024-04-02T09:09:43.786910+0000","last_fullsized":"2024-04-02T09:09:43.786910+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:02.167323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.10","version":"204'8","reported_seq":19,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.233144+0000","last_change":"2024-04-02T09:09:41.696623+0000","last_active":"2024-04-02T09:09:43.233144+0000","last_peered":"2024-04-02T09:09:43.233144+0000","last_clean":"2024-04-02T09:09:43.233144+0000","last_became_active":"2024-04-02T09:09:41.696203+0000","last_became_peered":"2024-04-02T09:09:41.696203+0000","last_unstale":"2024-04-02T09:09:43.233144+0000","last_undegraded":"2024-04-02T09:09:43.233144+0000","last_fullsized":"2024-04-02T09:09:43.233144+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:04:11.494696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787205+0000","last_change":"2024-04-02T09:09:42.243606+0000","last_active":"2024-04-02T09:09:43.787205+0000","last_peered":"2024-04-02T09:09:43.787205+0000","last_clean":"2024-04-02T09:09:43.787205+0000","last_became_active":"2024-04-02T09:09:42.243330+0000","last_became_peered":"2024-04-02T09:09:42.243330+0000","last_unstale":"2024-04-02T09:09:43.787205+0000","last_undegraded":"2024-04-02T09:09:43.787205+0000","last_fullsized":"2024-04-02T09:09:43.787205+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:51:52.730338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.f","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787428+0000","last_change":"2024-04-02T09:09:41.701659+0000","last_active":"2024-04-02T09:09:43.787428+0000","last_peered":"2024-04-02T09:09:43.787428+0000","last_clean":"2024-04-02T09:09:43.787428+0000","last_became_active":"2024-04-02T09:09:41.701384+0000","last_became_peered":"2024-04-02T09:09:41.701384+0000","last_unstale":"2024-04-02T09:09:43.787428+0000","last_undegraded":"2024-04-02T09:09:43.787428+0000","last_fullsized":"2024-04-02T09:09:43.787428+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:51:51.723926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786612+0000","last_change":"2024-04-02T09:09:42.249444+0000","last_active":"2024-04-02T09:09:43.786612+0000","last_peered":"2024-04-02T09:09:43.786612+0000","last_clean":"2024-04-02T09:09:43.786612+0000","last_became_active":"2024-04-02T09:09:42.248608+0000","last_became_peered":"2024-04-02T09:09:42.248608+0000","last_unstale":"2024-04-02T09:09:43.786612+0000","last_undegraded":"2024-04-02T09:09:43.786612+0000","last_fullsized":"2024-04-02T09:09:43.786612+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:05:43.102187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.e","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.236009+0000","last_change":"2024-04-02T09:09:41.690750+0000","last_active":"2024-04-02T09:09:43.236009+0000","last_peered":"2024-04-02T09:09:43.236009+0000","last_clean":"2024-04-02T09:09:43.236009+0000","last_became_active":"2024-04-02T09:09:41.690461+0000","last_became_peered":"2024-04-02T09:09:41.690461+0000","last_unstale":"2024-04-02T09:09:43.236009+0000","last_undegraded":"2024-04-02T09:09:43.236009+0000","last_fullsized":"2024-04-02T09:09:43.236009+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:05.915049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235889+0000","last_change":"2024-04-02T09:09:42.252927+0000","last_active":"2024-04-02T09:09:43.235889+0000","last_peered":"2024-04-02T09:09:43.235889+0000","last_clean":"2024-04-02T09:09:43.235889+0000","last_became_active":"2024-04-02T09:09:42.251020+0000","last_became_peered":"2024-04-02T09:09:42.251020+0000","last_unstale":"2024-04-02T09:09:43.235889+0000","last_undegraded":"2024-04-02T09:09:43.235889+0000","last_fullsized":"2024-04-02T09:09:43.235889+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:43:11.717821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.d","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785928+0000","last_change":"2024-04-02T09:09:41.700719+0000","last_active":"2024-04-02T09:09:43.785928+0000","last_peered":"2024-04-02T09:09:43.785928+0000","last_clean":"2024-04-02T09:09:43.785928+0000","last_became_active":"2024-04-02T09:09:41.700442+0000","last_became_peered":"2024-04-02T09:09:41.700442+0000","last_unstale":"2024-04-02T09:09:43.785928+0000","last_undegraded":"2024-04-02T09:09:43.785928+0000","last_fullsized":"2024-04-02T09:09:43.785928+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:36:13.959516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786448+0000","last_change":"2024-04-02T09:09:42.240056+0000","last_active":"2024-04-02T09:09:43.786448+0000","last_peered":"2024-04-02T09:09:43.786448+0000","last_clean":"2024-04-02T09:09:43.786448+0000","last_became_active":"2024-04-02T09:09:42.239816+0000","last_became_peered":"2024-04-02T09:09:42.239816+0000","last_unstale":"2024-04-02T09:09:43.786448+0000","last_undegraded":"2024-04-02T09:09:43.786448+0000","last_fullsized":"2024-04-02T09:09:43.786448+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:39.357706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786083+0000","last_change":"2024-04-02T09:09:41.696017+0000","last_active":"2024-04-02T09:09:43.786083+0000","last_peered":"2024-04-02T09:09:43.786083+0000","last_clean":"2024-04-02T09:09:43.786083+0000","last_became_active":"2024-04-02T09:09:41.695114+0000","last_became_peered":"2024-04-02T09:09:41.695114+0000","last_unstale":"2024-04-02T09:09:43.786083+0000","last_undegraded":"2024-04-02T09:09:43.786083+0000","last_fullsized":"2024-04-02T09:09:43.786083+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:38.351295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235674+0000","last_change":"2024-04-02T09:09:42.235837+0000","last_active":"2024-04-02T09:09:43.235674+0000","last_peered":"2024-04-02T09:09:43.235674+0000","last_clean":"2024-04-02T09:09:43.235674+0000","last_became_active":"2024-04-02T09:09:42.235225+0000","last_became_peered":"2024-04-02T09:09:42.235225+0000","last_unstale":"2024-04-02T09:09:43.235674+0000","last_undegraded":"2024-04-02T09:09:43.235674+0000","last_fullsized":"2024-04-02T09:09:43.235674+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:07:11.878538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.b","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786794+0000","last_change":"2024-04-02T09:09:41.697089+0000","last_active":"2024-04-02T09:09:43.786794+0000","last_peered":"2024-04-02T09:09:43.786794+0000","last_clean":"2024-04-02T09:09:43.786794+0000","last_became_active":"2024-04-02T09:09:41.696796+0000","last_became_peered":"2024-04-02T09:09:41.696796+0000","last_unstale":"2024-04-02T09:09:43.786794+0000","last_undegraded":"2024-04-02T09:09:43.786794+0000","last_fullsized":"2024-04-02T09:09:43.786794+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:29:14.904075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.233417+0000","last_change":"2024-04-02T09:09:42.250501+0000","last_active":"2024-04-02T09:09:43.233417+0000","last_peered":"2024-04-02T09:09:43.233417+0000","last_clean":"2024-04-02T09:09:43.233417+0000","last_became_active":"2024-04-02T09:09:42.249931+0000","last_became_peered":"2024-04-02T09:09:42.249931+0000","last_unstale":"2024-04-02T09:09:43.233417+0000","last_undegraded":"2024-04-02T09:09:43.233417+0000","last_fullsized":"2024-04-02T09:09:43.233417+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:38:27.306368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235770+0000","last_change":"2024-04-02T09:09:41.697209+0000","last_active":"2024-04-02T09:09:43.235770+0000","last_peered":"2024-04-02T09:09:43.235770+0000","last_clean":"2024-04-02T09:09:43.235770+0000","last_became_active":"2024-04-02T09:09:41.695512+0000","last_became_peered":"2024-04-02T09:09:41.695512+0000","last_unstale":"2024-04-02T09:09:43.235770+0000","last_undegraded":"2024-04-02T09:09:43.235770+0000","last_fullsized":"2024-04-02T09:09:43.235770+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:36.674318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785775+0000","last_change":"2024-04-02T09:09:42.242910+0000","last_active":"2024-04-02T09:09:43.785775+0000","last_peered":"2024-04-02T09:09:43.785775+0000","last_clean":"2024-04-02T09:09:43.785775+0000","last_became_active":"2024-04-02T09:09:42.241582+0000","last_became_peered":"2024-04-02T09:09:42.241582+0000","last_unstale":"2024-04-02T09:09:43.785775+0000","last_undegraded":"2024-04-02T09:09:43.785775+0000","last_fullsized":"2024-04-02T09:09:43.785775+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:33:38.345884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235604+0000","last_change":"2024-04-02T09:09:41.704460+0000","last_active":"2024-04-02T09:09:43.235604+0000","last_peered":"2024-04-02T09:09:43.235604+0000","last_clean":"2024-04-02T09:09:43.235604+0000","last_became_active":"2024-04-02T09:09:41.703676+0000","last_became_peered":"2024-04-02T09:09:41.703676+0000","last_unstale":"2024-04-02T09:09:43.235604+0000","last_undegraded":"2024-04-02T09:09:43.235604+0000","last_fullsized":"2024-04-02T09:09:43.235604+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:55:54.788666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786461+0000","last_change":"2024-04-02T09:09:42.239798+0000","last_active":"2024-04-02T09:09:43.786461+0000","last_peered":"2024-04-02T09:09:43.786461+0000","last_clean":"2024-04-02T09:09:43.786461+0000","last_became_active":"2024-04-02T09:09:42.239548+0000","last_became_peered":"2024-04-02T09:09:42.239548+0000","last_unstale":"2024-04-02T09:09:43.786461+0000","last_undegraded":"2024-04-02T09:09:43.786461+0000","last_fullsized":"2024-04-02T09:09:43.786461+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:56:51.798108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785751+0000","last_change":"2024-04-02T09:09:41.701344+0000","last_active":"2024-04-02T09:09:43.785751+0000","last_peered":"2024-04-02T09:09:43.785751+0000","last_clean":"2024-04-02T09:09:43.785751+0000","last_became_active":"2024-04-02T09:09:41.700538+0000","last_became_peered":"2024-04-02T09:09:41.700538+0000","last_unstale":"2024-04-02T09:09:43.785751+0000","last_undegraded":"2024-04-02T09:09:43.785751+0000","last_fullsized":"2024-04-02T09:09:43.785751+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:40:29.413813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595902+0000","last_change":"2024-04-02T09:09:42.258348+0000","last_active":"2024-04-02T09:09:43.595902+0000","last_peered":"2024-04-02T09:09:43.595902+0000","last_clean":"2024-04-02T09:09:43.595902+0000","last_became_active":"2024-04-02T09:09:42.258064+0000","last_became_peered":"2024-04-02T09:09:42.258064+0000","last_unstale":"2024-04-02T09:09:43.595902+0000","last_undegraded":"2024-04-02T09:09:43.595902+0000","last_fullsized":"2024-04-02T09:09:43.595902+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:56:26.514942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.7","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786148+0000","last_change":"2024-04-02T09:09:41.690341+0000","last_active":"2024-04-02T09:09:43.786148+0000","last_peered":"2024-04-02T09:09:43.786148+0000","last_clean":"2024-04-02T09:09:43.786148+0000","last_became_active":"2024-04-02T09:09:41.690083+0000","last_became_peered":"2024-04-02T09:09:41.690083+0000","last_unstale":"2024-04-02T09:09:43.786148+0000","last_undegraded":"2024-04-02T09:09:43.786148+0000","last_fullsized":"2024-04-02T09:09:43.786148+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:05:42.095775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.596001+0000","last_change":"2024-04-02T09:09:42.237565+0000","last_active":"2024-04-02T09:09:43.596001+0000","last_peered":"2024-04-02T09:09:43.596001+0000","last_clean":"2024-04-02T09:09:43.596001+0000","last_became_active":"2024-04-02T09:09:42.237290+0000","last_became_peered":"2024-04-02T09:09:42.237290+0000","last_unstale":"2024-04-02T09:09:43.596001+0000","last_undegraded":"2024-04-02T09:09:43.596001+0000","last_fullsized":"2024-04-02T09:09:43.596001+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:36.821746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595969+0000","last_change":"2024-04-02T09:09:41.705910+0000","last_active":"2024-04-02T09:09:43.595969+0000","last_peered":"2024-04-02T09:09:43.595969+0000","last_clean":"2024-04-02T09:09:43.595969+0000","last_became_active":"2024-04-02T09:09:41.705355+0000","last_became_peered":"2024-04-02T09:09:41.705355+0000","last_unstale":"2024-04-02T09:09:43.595969+0000","last_undegraded":"2024-04-02T09:09:43.595969+0000","last_fullsized":"2024-04-02T09:09:43.595969+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:07:10.872126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595659+0000","last_change":"2024-04-02T09:09:42.239824+0000","last_active":"2024-04-02T09:09:43.595659+0000","last_peered":"2024-04-02T09:09:43.595659+0000","last_clean":"2024-04-02T09:09:43.595659+0000","last_became_active":"2024-04-02T09:09:42.239571+0000","last_became_peered":"2024-04-02T09:09:42.239571+0000","last_unstale":"2024-04-02T09:09:43.595659+0000","last_undegraded":"2024-04-02T09:09:43.595659+0000","last_fullsized":"2024-04-02T09:09:43.595659+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:15:04.716813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.596033+0000","last_change":"2024-04-02T09:09:41.705067+0000","last_active":"2024-04-02T09:09:43.596033+0000","last_peered":"2024-04-02T09:09:43.596033+0000","last_clean":"2024-04-02T09:09:43.596033+0000","last_became_active":"2024-04-02T09:09:41.704081+0000","last_became_peered":"2024-04-02T09:09:41.704081+0000","last_unstale":"2024-04-02T09:09:43.596033+0000","last_undegraded":"2024-04-02T09:09:43.596033+0000","last_fullsized":"2024-04-02T09:09:43.596033+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:17.053666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786491+0000","last_change":"2024-04-02T09:09:42.253453+0000","last_active":"2024-04-02T09:09:43.786491+0000","last_peered":"2024-04-02T09:09:43.786491+0000","last_clean":"2024-04-02T09:09:43.786491+0000","last_became_active":"2024-04-02T09:09:42.253128+0000","last_became_peered":"2024-04-02T09:09:42.253128+0000","last_unstale":"2024-04-02T09:09:43.786491+0000","last_undegraded":"2024-04-02T09:09:43.786491+0000","last_fullsized":"2024-04-02T09:09:43.786491+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:35:08.812169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786941+0000","last_change":"2024-04-02T09:09:41.704495+0000","last_active":"2024-04-02T09:09:43.786941+0000","last_peered":"2024-04-02T09:09:43.786941+0000","last_clean":"2024-04-02T09:09:43.786941+0000","last_became_active":"2024-04-02T09:09:41.704197+0000","last_became_peered":"2024-04-02T09:09:41.704197+0000","last_unstale":"2024-04-02T09:09:43.786941+0000","last_undegraded":"2024-04-02T09:09:43.786941+0000","last_fullsized":"2024-04-02T09:09:43.786941+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:45:23.586259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235600+0000","last_change":"2024-04-02T09:09:42.251210+0000","last_active":"2024-04-02T09:09:43.235600+0000","last_peered":"2024-04-02T09:09:43.235600+0000","last_clean":"2024-04-02T09:09:43.235600+0000","last_became_active":"2024-04-02T09:09:42.249687+0000","last_became_peered":"2024-04-02T09:09:42.249687+0000","last_unstale":"2024-04-02T09:09:43.235600+0000","last_undegraded":"2024-04-02T09:09:43.235600+0000","last_fullsized":"2024-04-02T09:09:43.235600+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:43.199123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235668+0000","last_change":"2024-04-02T09:09:41.695183+0000","last_active":"2024-04-02T09:09:43.235668+0000","last_peered":"2024-04-02T09:09:43.235668+0000","last_clean":"2024-04-02T09:09:43.235668+0000","last_became_active":"2024-04-02T09:09:41.694918+0000","last_became_peered":"2024-04-02T09:09:41.694918+0000","last_unstale":"2024-04-02T09:09:43.235668+0000","last_undegraded":"2024-04-02T09:09:43.235668+0000","last_fullsized":"2024-04-02T09:09:43.235668+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:32.358193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785895+0000","last_change":"2024-04-02T09:09:42.239879+0000","last_active":"2024-04-02T09:09:43.785895+0000","last_peered":"2024-04-02T09:09:43.785895+0000","last_clean":"2024-04-02T09:09:43.785895+0000","last_became_active":"2024-04-02T09:09:42.239502+0000","last_became_peered":"2024-04-02T09:09:42.239502+0000","last_unstale":"2024-04-02T09:09:43.785895+0000","last_undegraded":"2024-04-02T09:09:43.785895+0000","last_fullsized":"2024-04-02T09:09:43.785895+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:41:06.071269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235652+0000","last_change":"2024-04-02T09:09:41.695104+0000","last_active":"2024-04-02T09:09:43.235652+0000","last_peered":"2024-04-02T09:09:43.235652+0000","last_clean":"2024-04-02T09:09:43.235652+0000","last_became_active":"2024-04-02T09:09:41.694755+0000","last_became_peered":"2024-04-02T09:09:41.694755+0000","last_unstale":"2024-04-02T09:09:43.235652+0000","last_undegraded":"2024-04-02T09:09:43.235652+0000","last_fullsized":"2024-04-02T09:09:43.235652+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:22.618092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235432+0000","last_change":"2024-04-02T09:09:42.246880+0000","last_active":"2024-04-02T09:09:43.235432+0000","last_peered":"2024-04-02T09:09:43.235432+0000","last_clean":"2024-04-02T09:09:43.235432+0000","last_became_active":"2024-04-02T09:09:42.246426+0000","last_became_peered":"2024-04-02T09:09:42.246426+0000","last_unstale":"2024-04-02T09:09:43.235432+0000","last_undegraded":"2024-04-02T09:09:43.235432+0000","last_fullsized":"2024-04-02T09:09:43.235432+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:17.246822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235545+0000","last_change":"2024-04-02T09:09:41.696354+0000","last_active":"2024-04-02T09:09:43.235545+0000","last_peered":"2024-04-02T09:09:43.235545+0000","last_clean":"2024-04-02T09:09:43.235545+0000","last_became_active":"2024-04-02T09:09:41.696061+0000","last_became_peered":"2024-04-02T09:09:41.696061+0000","last_unstale":"2024-04-02T09:09:43.235545+0000","last_undegraded":"2024-04-02T09:09:43.235545+0000","last_fullsized":"2024-04-02T09:09:43.235545+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:45:21.309725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787016+0000","last_change":"2024-04-02T09:09:42.238260+0000","last_active":"2024-04-02T09:09:43.787016+0000","last_peered":"2024-04-02T09:09:43.787016+0000","last_clean":"2024-04-02T09:09:43.787016+0000","last_became_active":"2024-04-02T09:09:42.237968+0000","last_became_peered":"2024-04-02T09:09:42.237968+0000","last_unstale":"2024-04-02T09:09:43.787016+0000","last_undegraded":"2024-04-02T09:09:43.787016+0000","last_fullsized":"2024-04-02T09:09:43.787016+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:59:48.937092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235624+0000","last_change":"2024-04-02T09:09:41.700043+0000","last_active":"2024-04-02T09:09:43.235624+0000","last_peered":"2024-04-02T09:09:43.235624+0000","last_clean":"2024-04-02T09:09:43.235624+0000","last_became_active":"2024-04-02T09:09:41.699034+0000","last_became_peered":"2024-04-02T09:09:41.699034+0000","last_unstale":"2024-04-02T09:09:43.235624+0000","last_undegraded":"2024-04-02T09:09:43.235624+0000","last_fullsized":"2024-04-02T09:09:43.235624+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:18.870825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235634+0000","last_change":"2024-04-02T09:09:42.239645+0000","last_active":"2024-04-02T09:09:43.235634+0000","last_peered":"2024-04-02T09:09:43.235634+0000","last_clean":"2024-04-02T09:09:43.235634+0000","last_became_active":"2024-04-02T09:09:42.239149+0000","last_became_peered":"2024-04-02T09:09:42.239149+0000","last_unstale":"2024-04-02T09:09:43.235634+0000","last_undegraded":"2024-04-02T09:09:43.235634+0000","last_fullsized":"2024-04-02T09:09:43.235634+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:33:36.680512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786221+0000","last_change":"2024-04-02T09:09:41.697172+0000","last_active":"2024-04-02T09:09:43.786221+0000","last_peered":"2024-04-02T09:09:43.786221+0000","last_clean":"2024-04-02T09:09:43.786221+0000","last_became_active":"2024-04-02T09:09:41.696927+0000","last_became_peered":"2024-04-02T09:09:41.696927+0000","last_unstale":"2024-04-02T09:09:43.786221+0000","last_undegraded":"2024-04-02T09:09:43.786221+0000","last_fullsized":"2024-04-02T09:09:43.786221+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:52:39.629358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"194'326","reported_seq":1115,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-02T09:09:42.236996+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:09:42.236996+0000","last_peered":"2024-04-02T09:09:42.236996+0000","last_clean":"2024-04-02T09:09:42.236996+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:09:42.236996+0000","last_undegraded":"2024-04-02T09:09:42.236996+0000","last_fullsized":"2024-04-02T09:09:42.236996+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575958,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59888,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311952,"statfs":{"total":96636764160,"available":96575438848,"internally_reserved":0,"allocated":2588672,"data_stored":1299912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.71199999999999997,"15min":0.71999999999999997},"min":{"1min":0.41599999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.97999999999999998,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.72599999999999998,"15min":0.73499999999999999},"min":{"1min":0.501,"5min":0.46899999999999997,"15min":0.442},"max":{"1min":0.89400000000000002,"5min":1.052,"15min":2.0680000000000001},"last":0.53400000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.73699999999999999,"15min":0.73599999999999999},"min":{"1min":0.57699999999999996,"5min":0.48499999999999999,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.94299999999999995,"15min":1.125},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.70699999999999996,"15min":0.72499999999999998},"min":{"1min":0.53900000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.86299999999999999,"5min":1.0109999999999999,"15min":2.133},"last":0.63}]},{"osd":2,"last update":"Tue Apr 2 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.51400000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88100000000000001,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.81000000000000005,"15min":0.80900000000000005},"min":{"1min":0.64300000000000002,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.92700000000000005,"5min":1.0229999999999999,"15min":2.02},"last":0.82399999999999995}]},{"osd":3,"last update":"Tue Apr 2 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.748,"15min":0.753},"min":{"1min":0.51300000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92600000000000005,"5min":1.452,"15min":1.8859999999999999},"last":0.753},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.76200000000000001,"15min":0.76800000000000002},"min":{"1min":0.57899999999999996,"5min":0.52400000000000002,"15min":0.45700000000000002},"max":{"1min":0.88400000000000001,"5min":1.0389999999999999,"15min":1.855},"last":0.70399999999999996}]},{"osd":4,"last update":"Tue Apr 2 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38100000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":2.093},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.73299999999999998,"15min":0.71299999999999997},"min":{"1min":0.47399999999999998,"5min":0.42099999999999999,"15min":0.38300000000000001},"max":{"1min":0.89800000000000002,"5min":1.536,"15min":1.536},"last":1.008}]},{"osd":5,"last update":"Tue Apr 2 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68000000000000005,"15min":0.66900000000000004},"min":{"1min":0.48199999999999998,"5min":0.41299999999999998,"15min":0.41199999999999998},"max":{"1min":0.86299999999999999,"5min":0.88600000000000001,"15min":1.1850000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.40600000000000003},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":2.2480000000000002},"last":0.85799999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68200000000000005,"15min":0.68999999999999995},"min":{"1min":0.52400000000000002,"5min":0.48399999999999999,"15min":0.45800000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":2.9670000000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.70299999999999996,"15min":0.68400000000000005},"min":{"1min":0.54100000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":1.0489999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575956,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41828,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330012,"statfs":{"total":96636764160,"available":96593932288,"internally_reserved":0,"allocated":2572288,"data_stored":1298326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73499999999999999,"15min":0.73799999999999999},"min":{"1min":0.57799999999999996,"5min":0.42699999999999999,"15min":0.36799999999999999},"max":{"1min":0.90600000000000003,"5min":1,"15min":1.038},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.747,"15min":0.75700000000000001},"min":{"1min":0.64900000000000002,"5min":0.45800000000000002,"15min":0.441},"max":{"1min":0.89700000000000002,"5min":0.94499999999999995,"15min":1.077},"last":0.52000000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72499999999999998,"15min":0.72999999999999998},"min":{"1min":0.54000000000000004,"5min":0.48099999999999998,"15min":0.438},"max":{"1min":0.84599999999999997,"5min":0.95099999999999996,"15min":1.212},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.74199999999999999,"15min":0.76000000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.872,"5min":1.0109999999999999,"15min":1.1759999999999999},"last":0.57499999999999996}]},{"osd":2,"last update":"Tue Apr 2 09:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.75,"15min":0.76200000000000001},"min":{"1min":0.627,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":1.2789999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.74399999999999999,"15min":0.753},"min":{"1min":0.65800000000000003,"5min":0.53100000000000003,"15min":0.46700000000000003},"max":{"1min":0.95999999999999996,"5min":1.002,"15min":1.226},"last":0.55000000000000004}]},{"osd":3,"last update":"Tue Apr 2 09:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.753,"15min":0.77600000000000002},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.497},"max":{"1min":0.90800000000000003,"5min":0.96099999999999997,"15min":1.2549999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.77400000000000002,"15min":0.77600000000000002},"min":{"1min":0.60699999999999998,"5min":0.52700000000000002,"15min":0.42799999999999999},"max":{"1min":0.94199999999999995,"5min":1.0189999999999999,"15min":1.3200000000000001},"last":0.67600000000000005}]},{"osd":4,"last update":"Tue Apr 2 09:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.61299999999999999,"15min":0.628},"min":{"1min":0.44900000000000001,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":0.78200000000000003,"5min":0.88600000000000001,"15min":1.1259999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.64900000000000002,"15min":0.67000000000000004},"min":{"1min":0.49199999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88900000000000001,"5min":0.98299999999999998,"15min":1.1879999999999999},"last":0.78900000000000003}]},{"osd":5,"last update":"Tue Apr 2 09:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64600000000000002,"15min":0.66100000000000003},"min":{"1min":0.497,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.875,"5min":0.96699999999999997,"15min":1.2669999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.71399999999999997,"15min":0.72899999999999998},"min":{"1min":0.52200000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94699999999999995,"5min":1.054,"15min":1.204},"last":0.77700000000000002}]},{"osd":7,"last update":"Tue Apr 2 09:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.73099999999999998,"15min":0.74299999999999999},"min":{"1min":0.54100000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91800000000000004,"5min":1.0269999999999999,"15min":1.2370000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.58499999999999996,"5min":0.49399999999999999,"15min":0.48299999999999998},"max":{"1min":0.93100000000000005,"5min":1.02,"15min":1.1519999999999999},"last":0.75700000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575957,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52244,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319596,"statfs":{"total":96636764160,"available":96583266304,"internally_reserved":0,"allocated":3149824,"data_stored":1881856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.45100000000000001,"15min":0.45600000000000002},"min":{"1min":0.311,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51600000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32100000000000001},"max":{"1min":0.73699999999999999,"5min":0.78600000000000003,"15min":0.79800000000000004},"last":0.41799999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.376,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.86399999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.45200000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77500000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.443}]},{"osd":3,"last update":"Tue Apr 2 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.41699999999999998,"15min":0.37},"max":{"1min":0.80200000000000005,"5min":0.80500000000000005,"15min":0.88100000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.40400000000000003,"5min":0.378,"15min":0.375},"max":{"1min":0.86699999999999999,"5min":0.878,"15min":0.878},"last":0.498}]},{"osd":4,"last update":"Tue Apr 2 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.58999999999999997,"5min":0.433,"15min":0.433},"max":{"1min":1.077,"5min":1.077,"15min":1.1479999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76500000000000001,"15min":0.79300000000000004},"min":{"1min":0.57199999999999995,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.034,"5min":1.22,"15min":1.25},"last":0.72399999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78000000000000003,"15min":0.80100000000000005},"min":{"1min":0.57699999999999996,"5min":0.54500000000000004,"15min":0.52900000000000003},"max":{"1min":1.115,"5min":1.1220000000000001,"15min":1.321},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79800000000000004,"15min":0.81299999999999994},"min":{"1min":0.61799999999999999,"5min":0.55300000000000005,"15min":0.53500000000000003},"max":{"1min":1.05,"5min":1.0980000000000001,"15min":1.173},"last":0.85299999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.80600000000000005,"15min":0.82199999999999995},"min":{"1min":0.61199999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.159,"5min":1.159,"15min":1.2210000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.80300000000000005,"15min":0.81399999999999995},"min":{"1min":0.64000000000000001,"5min":0.57999999999999996,"15min":0.5},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.2529999999999999},"last":0.83199999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81100000000000005,"15min":0.83299999999999996},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.53700000000000003},"max":{"1min":1.1000000000000001,"5min":1.159,"15min":1.236},"last":0.755},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.83199999999999996,"15min":0.84199999999999997},"min":{"1min":0.62,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.048,"5min":1.1850000000000001,"15min":1.2789999999999999},"last":0.79300000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575957,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47576,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324264,"statfs":{"total":96636764160,"available":96588046336,"internally_reserved":0,"allocated":2560000,"data_stored":1291085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45100000000000001,"15min":0.46200000000000002},"min":{"1min":0.35299999999999998,"5min":0.311,"15min":0.28599999999999998},"max":{"1min":0.58099999999999996,"5min":0.78100000000000003,"15min":0.86699999999999999},"last":0.499},{"interface":"front","average":{"1min":0.503,"5min":0.496,"15min":0.51600000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64100000000000001,"5min":0.73299999999999998,"15min":0.79900000000000004},"last":0.45200000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.51800000000000002},"min":{"1min":0.41299999999999998,"5min":0.35599999999999998,"15min":0.33900000000000002},"max":{"1min":0.68000000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55200000000000005,"15min":0.55900000000000005},"min":{"1min":0.41899999999999998,"5min":0.376,"15min":0.36399999999999999},"max":{"1min":0.67400000000000004,"5min":0.752,"15min":0.84199999999999997},"last":0.52400000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.39800000000000002,"5min":0.374,"15min":0.34200000000000003},"max":{"1min":0.70799999999999996,"5min":0.85699999999999998,"15min":0.97899999999999998},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64500000000000002},"min":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.45000000000000001},"max":{"1min":0.76300000000000001,"5min":0.88400000000000001,"15min":0.96399999999999997},"last":0.627}]},{"osd":4,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.84499999999999997,"15min":0.83999999999999997},"min":{"1min":0.70299999999999996,"5min":0.58699999999999997,"15min":0.45600000000000002},"max":{"1min":1.129,"5min":1.129,"15min":1.3580000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.83599999999999997,"15min":0.83099999999999996},"min":{"1min":0.69799999999999995,"5min":0.55800000000000005,"15min":0.48199999999999998},"max":{"1min":1.0640000000000001,"5min":1.1439999999999999,"15min":1.3009999999999999},"last":0.88700000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.83699999999999997,"15min":0.83399999999999996},"min":{"1min":0.66100000000000003,"5min":0.60099999999999998,"15min":0.498},"max":{"1min":0.96599999999999997,"5min":1.163,"15min":1.232},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.83399999999999996,"15min":0.83999999999999997},"min":{"1min":0.68300000000000005,"5min":0.59999999999999998,"15min":0.55900000000000005},"max":{"1min":1.014,"5min":1.165,"15min":1.1719999999999999},"last":0.91900000000000004}]},{"osd":6,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80200000000000005,"15min":0.80400000000000005},"min":{"1min":0.58499999999999996,"5min":0.55600000000000005,"15min":0.51500000000000001},"max":{"1min":0.94599999999999995,"5min":1.196,"15min":1.196},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.59299999999999997,"5min":0.54900000000000004,"15min":0.499},"max":{"1min":1.085,"5min":1.179,"15min":1.234},"last":0.71699999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.86499999999999999,"15min":0.86399999999999999},"min":{"1min":0.67600000000000005,"5min":0.629,"15min":0.58899999999999997},"max":{"1min":1.0349999999999999,"5min":1.1499999999999999,"15min":1.2609999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84899999999999998,"15min":0.85199999999999998},"min":{"1min":0.60599999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.0189999999999999,"5min":1.1910000000000001,"15min":1.2589999999999999},"last":0.98199999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575956,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51680,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320160,"statfs":{"total":96636764160,"available":96583843840,"internally_reserved":0,"allocated":2572288,"data_stored":1298306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.45600000000000002,"15min":0.44500000000000001},"min":{"1min":0.33500000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66600000000000004,"5min":0.69399999999999995,"15min":0.78700000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52200000000000002,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79900000000000004,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.33700000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.51200000000000001},"min":{"1min":0.434,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.81799999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56299999999999994},"min":{"1min":0.44900000000000001,"5min":0.38900000000000001,"15min":0.378},"max":{"1min":0.71999999999999997,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.38200000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55300000000000005,"15min":0.56799999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.34599999999999997},"max":{"1min":0.73599999999999999,"5min":1.2629999999999999,"15min":1.2749999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64800000000000002,"15min":0.64300000000000002},"min":{"1min":0.505,"5min":0.48299999999999998,"15min":0.47899999999999998},"max":{"1min":0.83699999999999997,"5min":0.97699999999999998,"15min":1.0740000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.78600000000000003,"15min":0.79000000000000004},"min":{"1min":0.52400000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0700000000000001,"5min":1.087,"15min":1.206},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.68000000000000005,"5min":0.56999999999999995,"15min":0.56599999999999995},"max":{"1min":0.98599999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.996}]},{"osd":5,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.78400000000000003,"15min":0.79800000000000004},"min":{"1min":0.60899999999999999,"5min":0.55700000000000005,"15min":0.52600000000000002},"max":{"1min":1.004,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.81599999999999995,"15min":0.81399999999999995},"min":{"1min":0.64200000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.101,"5min":1.1259999999999999,"15min":1.2490000000000001},"last":0.96799999999999997}]},{"osd":6,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81799999999999995,"15min":0.82199999999999995},"min":{"1min":0.65500000000000003,"5min":0.54700000000000004,"15min":0.501},"max":{"1min":1.141,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.173},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81799999999999995,"15min":0.82299999999999995},"min":{"1min":0.66300000000000003,"5min":0.56599999999999995,"15min":0.55000000000000004},"max":{"1min":0.98899999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":1.0349999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.83599999999999997,"15min":0.84199999999999997},"min":{"1min":0.73599999999999999,"5min":0.55100000000000005,"15min":0.55000000000000004},"max":{"1min":1.095,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.83899999999999997,"15min":0.84099999999999997},"min":{"1min":0.70899999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.0860000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.82599999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575956,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51680,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320160,"statfs":{"total":96636764160,"available":96583843840,"internally_reserved":0,"allocated":2572288,"data_stored":1293122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45300000000000001,"15min":0.45400000000000001},"min":{"1min":0.318,"5min":0.30599999999999999,"15min":0.30299999999999999},"max":{"1min":0.57299999999999995,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.499,"15min":0.499},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.30599999999999999},"max":{"1min":0.65400000000000003,"5min":0.746,"15min":0.876},"last":0.438}]},{"osd":1,"last update":"Tue Apr 2 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.38700000000000001,"5min":0.32500000000000001,"15min":0.311},"max":{"1min":0.64900000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.34100000000000003},"max":{"1min":0.71799999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.68899999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.627,"15min":0.63700000000000001},"min":{"1min":0.47599999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83899999999999997,"5min":0.90600000000000003,"15min":1.2869999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.627,"15min":0.63500000000000001},"min":{"1min":0.46300000000000002,"5min":0.44600000000000001,"15min":0.34999999999999998},"max":{"1min":0.78000000000000003,"5min":0.91800000000000004,"15min":1.274},"last":0.53300000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.76200000000000001,"15min":0.76900000000000002},"min":{"1min":0.53900000000000003,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0369999999999999,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.748},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.58199999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.97199999999999998,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.72099999999999997}]},{"osd":5,"last update":"Tue Apr 2 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.78900000000000003,"15min":0.78400000000000003},"min":{"1min":0.57499999999999996,"5min":0.57299999999999995,"15min":0.55100000000000005},"max":{"1min":1.0469999999999999,"5min":1.099,"15min":1.157},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78800000000000003,"15min":0.79100000000000004},"min":{"1min":0.57799999999999996,"5min":0.55200000000000005,"15min":0.496},"max":{"1min":1.052,"5min":1.1120000000000001,"15min":1.129},"last":0.69799999999999995}]},{"osd":6,"last update":"Tue Apr 2 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.79300000000000004,"15min":0.79500000000000004},"min":{"1min":0.65600000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0089999999999999,"5min":1.22,"15min":1.22},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.82599999999999996,"15min":0.81999999999999995},"min":{"1min":0.64900000000000002,"5min":0.55700000000000005,"15min":0.50600000000000001},"max":{"1min":1.085,"5min":1.091,"15min":1.147},"last":0.875}]},{"osd":7,"last update":"Tue Apr 2 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.60399999999999998,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0189999999999999,"5min":1.181,"15min":1.2010000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.83599999999999997,"15min":0.83199999999999996},"min":{"1min":0.66400000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.006,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.76500000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575956,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47588,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324252,"statfs":{"total":96636764160,"available":96588034048,"internally_reserved":0,"allocated":2572288,"data_stored":1298270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68700000000000006,"15min":0.70899999999999996},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.29299999999999998},"max":{"1min":0.83899999999999997,"5min":0.91800000000000004,"15min":1.0389999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.68500000000000005,"15min":0.69799999999999995},"min":{"1min":0.54700000000000004,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86199999999999999,"5min":1.075,"15min":1.075},"last":0.623}]},{"osd":1,"last update":"Tue Apr 2 09:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.755,"15min":0.77300000000000002},"min":{"1min":0.58899999999999997,"5min":0.48699999999999999,"15min":0.434},"max":{"1min":0.88200000000000001,"5min":1.069,"15min":1.099},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.78500000000000003},"min":{"1min":0.57399999999999995,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89800000000000002,"5min":0.98499999999999999,"15min":1.0800000000000001},"last":1.03}]},{"osd":2,"last update":"Tue Apr 2 09:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71999999999999997,"15min":0.72599999999999998},"min":{"1min":0.502,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.93999999999999995,"5min":1.0129999999999999,"15min":1.089},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72899999999999998,"15min":0.74099999999999999},"min":{"1min":0.58999999999999997,"5min":0.501,"15min":0.41299999999999998},"max":{"1min":0.84399999999999997,"5min":0.95699999999999996,"15min":1.0329999999999999},"last":1.0740000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78300000000000003,"15min":0.78100000000000003},"min":{"1min":0.53600000000000003,"5min":0.50600000000000001,"15min":0.47399999999999998},"max":{"1min":0.998,"5min":1.018,"15min":1.0700000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.752,"15min":0.76800000000000002},"min":{"1min":0.59499999999999997,"5min":0.505,"15min":0.505},"max":{"1min":0.96099999999999997,"5min":1.028,"15min":1.038},"last":0.92700000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.60699999999999998,"15min":0.61099999999999999},"min":{"1min":0.44900000000000001,"5min":0.41499999999999998,"15min":0.34000000000000002},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":1.1140000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.46700000000000003,"5min":0.40999999999999998,"15min":0.35099999999999998},"max":{"1min":0.90700000000000003,"5min":0.90900000000000003,"15min":0.93100000000000005},"last":0.85099999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.65600000000000003,"15min":0.64000000000000001},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.42699999999999999},"max":{"1min":0.92000000000000004,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.70899999999999996,"15min":0.69299999999999995},"min":{"1min":0.48699999999999999,"5min":0.46999999999999997,"15min":0.38400000000000001},"max":{"1min":0.96999999999999997,"5min":1.0029999999999999,"15min":1.0469999999999999},"last":0.872}]},{"osd":7,"last update":"Tue Apr 2 09:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.72399999999999998,"15min":0.70799999999999996},"min":{"1min":0.58799999999999997,"5min":0.48299999999999998,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.0329999999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.79200000000000004,"15min":0.78600000000000003},"min":{"1min":0.61899999999999999,"5min":0.51000000000000001,"15min":0.501},"max":{"1min":1.008,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":1.0980000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575959,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48156,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323684,"statfs":{"total":96636764160,"available":96587452416,"internally_reserved":0,"allocated":3153920,"data_stored":1881441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69199999999999995,"15min":0.70799999999999996},"min":{"1min":0.47499999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86799999999999999,"5min":0.98999999999999999,"15min":1.004},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71699999999999997,"15min":0.72499999999999998},"min":{"1min":0.49399999999999999,"5min":0.432,"15min":0.432},"max":{"1min":0.97199999999999998,"5min":0.996,"15min":1.0600000000000001},"last":0.78800000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.70899999999999996,"15min":0.72699999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91300000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72199999999999998,"15min":0.73099999999999998},"min":{"1min":0.55500000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91600000000000004,"5min":0.99099999999999999,"15min":1.052},"last":0.69399999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73799999999999999,"15min":0.746},"min":{"1min":0.53400000000000003,"5min":0.433,"15min":0.433},"max":{"1min":0.92800000000000005,"5min":0.995,"15min":1.014},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.749,"15min":0.747},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.93000000000000005,"5min":1.026,"15min":1.0680000000000001},"last":0.64900000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.77000000000000002,"15min":0.79000000000000004},"min":{"1min":0.57399999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93500000000000005,"5min":1.036,"15min":1.083},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.80000000000000004,"15min":0.79200000000000004},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.1020000000000001},"last":0.71299999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.626,"15min":0.622},"min":{"1min":0.438,"5min":0.41599999999999998,"15min":0.373},"max":{"1min":0.90700000000000003,"5min":1.054,"15min":1.054},"last":0.62},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.48499999999999999,"5min":0.45400000000000001,"15min":0.39900000000000002},"max":{"1min":0.91800000000000004,"5min":0.95299999999999996,"15min":0.996},"last":0.73299999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.66600000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.41699999999999998},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71199999999999997,"15min":0.70499999999999996},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.38700000000000001},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":1.01},"last":0.76300000000000001}]},{"osd":7,"last update":"Tue Apr 2 09:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73299999999999998,"15min":0.72799999999999998},"min":{"1min":0.60399999999999998,"5min":0.49099999999999999,"15min":0.45200000000000001},"max":{"1min":0.93600000000000005,"5min":0.98799999999999999,"15min":1.0349999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.76200000000000001,"15min":0.75800000000000001},"min":{"1min":0.49199999999999999,"5min":0.48999999999999999,"15min":0.45700000000000002},"max":{"1min":1.018,"5min":1.079,"15min":1.079},"last":0.752}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:09:47.237 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:09:47.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.463+0000 7f3c94c00700 1 Processor -- start 2024-04-02T09:09:47.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.464+0000 7f3c94c00700 1 -- start start 2024-04-02T09:09:47.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.465+0000 7f3c94c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3c900d18b0 0x7f3c900cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:47.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.465+0000 7f3c94c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c900ceda0 con 0x7f3c90160f10 2024-04-02T09:09:47.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.465+0000 7f3c94c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c901612f0 con 0x7f3c90161590 2024-04-02T09:09:47.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.465+0000 7f3c94c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c900d1d30 con 0x7f3c900d18b0 2024-04-02T09:09:47.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.465+0000 7f3c8de00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3c90160f10 0x7f3c900d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36556/0 (socket says 172.21.15.89:36556) 2024-04-02T09:09:47.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.465+0000 7f3c8de00700 1 -- 172.21.15.89:0/2625053848 learned_addr learned my addr 172.21.15.89:0/2625053848 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:47.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.465+0000 7f3c8d400700 1 --2- 172.21.15.89:0/2625053848 >> v2:172.21.15.113:3300/0 conn(0x7f3c900d18b0 0x7f3c900cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:47.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.465+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921353924 0 0) 0x7f3c900ceda0 con 0x7f3c90160f10 2024-04-02T09:09:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.465+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c740035b0 con 0x7f3c90160f10 2024-04-02T09:09:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068814842 0 0) 0x7f3c901612f0 con 0x7f3c90161590 2024-04-02T09:09:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c900ceda0 con 0x7f3c90161590 2024-04-02T09:09:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3706241522 0 0) 0x7f3c740035b0 con 0x7f3c90160f10 2024-04-02T09:09:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 >> v2:172.21.15.113:3300/0 conn(0x7f3c900d18b0 msgr2=0x7f3c900cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:47.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 --2- 172.21.15.89:0/2625053848 >> v2:172.21.15.113:3300/0 conn(0x7f3c900d18b0 0x7f3c900cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 >> v1:172.21.15.113:6789/0 conn(0x7f3c90161590 legacy=0x7f3c900b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c900ce470 con 0x7f3c90160f10 2024-04-02T09:09:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f3c80001670 con 0x7f3c90160f10 2024-04-02T09:09:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8d400700 1 --2- 172.21.15.89:0/2625053848 >> v2:172.21.15.113:3300/0 conn(0x7f3c900d18b0 0x7f3c900cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:09:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 >> v1:172.21.15.89:6789/0 conn(0x7f3c90160f10 legacy=0x7f3c900d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 --2- 172.21.15.89:0/2625053848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74003b90 0x7f3c74003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 --2- 172.21.15.89:0/2625053848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c74004850 0x7f3c74004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 --2- 172.21.15.89:0/2625053848 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c74005210 0x7f3c7400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c900d1d30 con 0x7f3c74003b90 2024-04-02T09:09:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7400c140 con 0x7f3c74004850 2024-04-02T09:09:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7400c300 con 0x7f3c74005210 2024-04-02T09:09:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8de00700 1 --2- 172.21.15.89:0/2625053848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74003b90 0x7f3c74003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:47.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8d400700 1 --2- 172.21.15.89:0/2625053848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c74004850 0x7f3c74004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8e800700 1 --2- 172.21.15.89:0/2625053848 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c74005210 0x7f3c7400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8de00700 1 -- 172.21.15.89:0/2625053848 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c74005210 msgr2=0x7f3c7400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8de00700 1 --2- 172.21.15.89:0/2625053848 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c74005210 0x7f3c7400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8de00700 1 -- 172.21.15.89:0/2625053848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c74004850 msgr2=0x7f3c74004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8de00700 1 --2- 172.21.15.89:0/2625053848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c74004850 0x7f3c74004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8de00700 1 -- 172.21.15.89:0/2625053848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c740035b0 con 0x7f3c74003b90 2024-04-02T09:09:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.466+0000 7f3c8e800700 1 -- 172.21.15.89:0/2625053848 reap_dead start 2024-04-02T09:09:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.467+0000 7f3c8de00700 1 --2- 172.21.15.89:0/2625053848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74003b90 0x7f3c74003f90 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000c210 tx=0x7f3c74005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93ae92820571429b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.467+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8000cdb0 con 0x7f3c74003b90 2024-04-02T09:09:47.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.467+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c80001af0 con 0x7f3c74003b90 2024-04-02T09:09:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.467+0000 7f3c8ca00700 1 -- 172.21.15.89:0/2625053848 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c80005ba0 con 0x7f3c74003b90 2024-04-02T09:09:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.467+0000 7f3c94c00700 1 -- 172.21.15.89:0/2625053848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74003b90 msgr2=0x7f3c74003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.467+0000 7f3c94c00700 1 --2- 172.21.15.89:0/2625053848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74003b90 0x7f3c74003f90 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000c210 tx=0x7f3c74005880 comp rx=0 tx=0).stop 2024-04-02T09:09:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.467+0000 7f3c94c00700 1 -- 172.21.15.89:0/2625053848 shutdown_connections 2024-04-02T09:09:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.467+0000 7f3c94c00700 1 --2- 172.21.15.89:0/2625053848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74003b90 0x7f3c74003f90 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.467+0000 7f3c94c00700 1 -- 172.21.15.89:0/2625053848 >> 172.21.15.89:0/2625053848 conn(0x7f3c9008aff0 msgr2=0x7f3c900d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.467+0000 7f3c94c00700 1 -- 172.21.15.89:0/2625053848 shutdown_connections 2024-04-02T09:09:47.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.468+0000 7f3c94c00700 1 -- 172.21.15.89:0/2625053848 wait complete. 2024-04-02T09:09:47.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.469+0000 7f3c94c00700 1 Processor -- start 2024-04-02T09:09:47.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.469+0000 7f3c94c00700 1 -- start start 2024-04-02T09:09:47.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.470+0000 7f3c94c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c74003b90 0x7f3c901f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:47.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.470+0000 7f3c94c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c90155d90 0x7f3c90156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:47.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.470+0000 7f3c94c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c901f3370 0x7f3c901f3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:47.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.470+0000 7f3c94c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c901585b0 con 0x7f3c901f3370 2024-04-02T09:09:47.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.470+0000 7f3c94c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c901f3d40 con 0x7f3c74003b90 2024-04-02T09:09:47.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.470+0000 7f3c94c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c90157510 con 0x7f3c90155d90 2024-04-02T09:09:47.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c8e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c901f3370 0x7f3c901f3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:47.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c8e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c901f3370 0x7f3c901f3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40486/0 (socket says 172.21.15.89:40486) 2024-04-02T09:09:47.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c8e800700 1 -- 172.21.15.89:0/2236171630 learned_addr learned my addr 172.21.15.89:0/2236171630 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:47.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c8e800700 1 -- 172.21.15.89:0/2236171630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c90155d90 msgr2=0x7f3c90156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:47.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c8de00700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c74003b90 0x7f3c901f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:47.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c8d400700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c90155d90 0x7f3c90156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:47.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c8e800700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c90155d90 0x7f3c90156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c8e800700 1 -- 172.21.15.89:0/2236171630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c74003b90 msgr2=0x7f3c901f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:47.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c8e800700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c74003b90 0x7f3c901f62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c8e800700 1 -- 172.21.15.89:0/2236171630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c901577f0 con 0x7f3c901f3370 2024-04-02T09:09:47.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c8d400700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c90155d90 0x7f3c90156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:47.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c8de00700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c74003b90 0x7f3c901f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:09:47.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c8e800700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c901f3370 0x7f3c901f3800 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f3c9008d3f0 tx=0x7f3c8800c0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:47.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c86000700 1 -- 172.21.15.89:0/2236171630 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c88003c10 con 0x7f3c901f3370 2024-04-02T09:09:47.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c86000700 1 -- 172.21.15.89:0/2236171630 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c88003d90 con 0x7f3c901f3370 2024-04-02T09:09:47.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.471+0000 7f3c94c00700 1 -- 172.21.15.89:0/2236171630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c90157ad0 con 0x7f3c901f3370 2024-04-02T09:09:47.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.472+0000 7f3c86000700 1 -- 172.21.15.89:0/2236171630 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8800c940 con 0x7f3c901f3370 2024-04-02T09:09:47.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.472+0000 7f3c94c00700 1 -- 172.21.15.89:0/2236171630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c90158090 con 0x7f3c901f3370 2024-04-02T09:09:47.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.473+0000 7f3c94c00700 1 -- 172.21.15.89:0/2236171630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c90067720 con 0x7f3c901f3370 2024-04-02T09:09:47.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.473+0000 7f3c86000700 1 -- 172.21.15.89:0/2236171630 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3c8800caa0 con 0x7f3c901f3370 2024-04-02T09:09:47.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.475+0000 7f3c86000700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3c7006adb0 0x7f3c7006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:47.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.475+0000 7f3c86000700 1 -- 172.21.15.89:0/2236171630 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f3c880897a0 con 0x7f3c901f3370 2024-04-02T09:09:47.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.475+0000 7f3c8de00700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3c7006adb0 0x7f3c7006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:47.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.475+0000 7f3c8de00700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3c7006adb0 0x7f3c7006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3c80015040 tx=0x7f3c80006040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:47.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.479+0000 7f3c86000700 1 -- 172.21.15.89:0/2236171630 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3c88054610 con 0x7f3c901f3370 2024-04-02T09:09:47.670 INFO:tasks.ceph.mds.c.smithi089.stderr:2024-04-02T09:09:47.671+0000 7f0ff9000700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T09:09:47.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.675+0000 7f3c94c00700 1 -- 172.21.15.89:0/2236171630 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3c900ce120 con 0x7f3c7006adb0 2024-04-02T09:09:47.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.684+0000 7f3c86000700 1 -- 172.21.15.89:0/2236171630 <== mgr.10398 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289760 (secure 0 0 0) 0x7f3c80001430 con 0x7f3c7006adb0 2024-04-02T09:09:47.687 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:47.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.688+0000 7f3c94c00700 1 -- 172.21.15.89:0/2236171630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3c7006adb0 msgr2=0x7f3c7006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:47.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.688+0000 7f3c94c00700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3c7006adb0 0x7f3c7006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3c80015040 tx=0x7f3c80006040 comp rx=0 tx=0).stop 2024-04-02T09:09:47.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.688+0000 7f3c94c00700 1 -- 172.21.15.89:0/2236171630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c901f3370 msgr2=0x7f3c901f3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:47.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.688+0000 7f3c94c00700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c901f3370 0x7f3c901f3800 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f3c9008d3f0 tx=0x7f3c8800c0e0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.688+0000 7f3c94c00700 1 -- 172.21.15.89:0/2236171630 shutdown_connections 2024-04-02T09:09:47.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.688+0000 7f3c94c00700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3c7006adb0 0x7f3c7006d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.688+0000 7f3c94c00700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c74003b90 0x7f3c901f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.688+0000 7f3c94c00700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c901f3370 0x7f3c901f3800 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.688+0000 7f3c94c00700 1 --2- 172.21.15.89:0/2236171630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c90155d90 0x7f3c90156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.688+0000 7f3c94c00700 1 -- 172.21.15.89:0/2236171630 >> 172.21.15.89:0/2236171630 conn(0x7f3c9008aff0 msgr2=0x7f3c9015bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:47.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.688+0000 7f3c94c00700 1 -- 172.21.15.89:0/2236171630 shutdown_connections 2024-04-02T09:09:47.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.688+0000 7f3c94c00700 1 -- 172.21.15.89:0/2236171630 wait complete. 2024-04-02T09:09:47.691 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:09:47.705 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-02T09:09:46.103035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":400640,"kb_used_data":21232,"kb_used_omap":12,"kb_used_meta":379251,"kb_avail":754574080,"statfs":{"total":773094113280,"available":772683857920,"internally_reserved":0,"allocated":21741568,"data_stored":11542318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":388353592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.427265"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786087+0000","last_change":"2024-04-02T09:09:42.242472+0000","last_active":"2024-04-02T09:09:43.786087+0000","last_peered":"2024-04-02T09:09:43.786087+0000","last_clean":"2024-04-02T09:09:43.786087+0000","last_became_active":"2024-04-02T09:09:42.241872+0000","last_became_peered":"2024-04-02T09:09:42.241872+0000","last_unstale":"2024-04-02T09:09:43.786087+0000","last_undegraded":"2024-04-02T09:09:43.786087+0000","last_fullsized":"2024-04-02T09:09:43.786087+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:37:55.536352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235964+0000","last_change":"2024-04-02T09:09:42.250584+0000","last_active":"2024-04-02T09:09:43.235964+0000","last_peered":"2024-04-02T09:09:43.235964+0000","last_clean":"2024-04-02T09:09:43.235964+0000","last_became_active":"2024-04-02T09:09:42.250293+0000","last_became_peered":"2024-04-02T09:09:42.250293+0000","last_unstale":"2024-04-02T09:09:43.235964+0000","last_undegraded":"2024-04-02T09:09:43.235964+0000","last_fullsized":"2024-04-02T09:09:43.235964+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:37:41.368117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595689+0000","last_change":"2024-04-02T09:09:42.239005+0000","last_active":"2024-04-02T09:09:43.595689+0000","last_peered":"2024-04-02T09:09:43.595689+0000","last_clean":"2024-04-02T09:09:43.595689+0000","last_became_active":"2024-04-02T09:09:42.238737+0000","last_became_peered":"2024-04-02T09:09:42.238737+0000","last_unstale":"2024-04-02T09:09:43.595689+0000","last_undegraded":"2024-04-02T09:09:43.595689+0000","last_fullsized":"2024-04-02T09:09:43.595689+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:20:11.575610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235737+0000","last_change":"2024-04-02T09:09:42.253202+0000","last_active":"2024-04-02T09:09:43.235737+0000","last_peered":"2024-04-02T09:09:43.235737+0000","last_clean":"2024-04-02T09:09:43.235737+0000","last_became_active":"2024-04-02T09:09:42.251249+0000","last_became_peered":"2024-04-02T09:09:42.251249+0000","last_unstale":"2024-04-02T09:09:43.235737+0000","last_undegraded":"2024-04-02T09:09:43.235737+0000","last_fullsized":"2024-04-02T09:09:43.235737+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:03:36.858777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786940+0000","last_change":"2024-04-02T09:09:42.251147+0000","last_active":"2024-04-02T09:09:43.786940+0000","last_peered":"2024-04-02T09:09:43.786940+0000","last_clean":"2024-04-02T09:09:43.786940+0000","last_became_active":"2024-04-02T09:09:42.247444+0000","last_became_peered":"2024-04-02T09:09:42.247444+0000","last_unstale":"2024-04-02T09:09:43.786940+0000","last_undegraded":"2024-04-02T09:09:43.786940+0000","last_fullsized":"2024-04-02T09:09:43.786940+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:18.060077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235545+0000","last_change":"2024-04-02T09:09:42.237780+0000","last_active":"2024-04-02T09:09:43.235545+0000","last_peered":"2024-04-02T09:09:43.235545+0000","last_clean":"2024-04-02T09:09:43.235545+0000","last_became_active":"2024-04-02T09:09:42.235583+0000","last_became_peered":"2024-04-02T09:09:42.235583+0000","last_unstale":"2024-04-02T09:09:43.235545+0000","last_undegraded":"2024-04-02T09:09:43.235545+0000","last_fullsized":"2024-04-02T09:09:43.235545+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:35:08.812169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.233152+0000","last_change":"2024-04-02T09:09:42.244707+0000","last_active":"2024-04-02T09:09:43.233152+0000","last_peered":"2024-04-02T09:09:43.233152+0000","last_clean":"2024-04-02T09:09:43.233152+0000","last_became_active":"2024-04-02T09:09:42.244291+0000","last_became_peered":"2024-04-02T09:09:42.244291+0000","last_unstale":"2024-04-02T09:09:43.233152+0000","last_undegraded":"2024-04-02T09:09:43.233152+0000","last_fullsized":"2024-04-02T09:09:43.233152+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:51:52.730338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786231+0000","last_change":"2024-04-02T09:09:42.251079+0000","last_active":"2024-04-02T09:09:43.786231+0000","last_peered":"2024-04-02T09:09:43.786231+0000","last_clean":"2024-04-02T09:09:43.786231+0000","last_became_active":"2024-04-02T09:09:42.244304+0000","last_became_peered":"2024-04-02T09:09:42.244304+0000","last_unstale":"2024-04-02T09:09:43.786231+0000","last_undegraded":"2024-04-02T09:09:43.786231+0000","last_fullsized":"2024-04-02T09:09:43.786231+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:53:16.796340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235916+0000","last_change":"2024-04-02T09:09:42.253842+0000","last_active":"2024-04-02T09:09:43.235916+0000","last_peered":"2024-04-02T09:09:43.235916+0000","last_clean":"2024-04-02T09:09:43.235916+0000","last_became_active":"2024-04-02T09:09:42.253572+0000","last_became_peered":"2024-04-02T09:09:42.253572+0000","last_unstale":"2024-04-02T09:09:43.235916+0000","last_undegraded":"2024-04-02T09:09:43.235916+0000","last_fullsized":"2024-04-02T09:09:43.235916+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:41:17.862841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787108+0000","last_change":"2024-04-02T09:09:42.246345+0000","last_active":"2024-04-02T09:09:43.787108+0000","last_peered":"2024-04-02T09:09:43.787108+0000","last_clean":"2024-04-02T09:09:43.787108+0000","last_became_active":"2024-04-02T09:09:42.246054+0000","last_became_peered":"2024-04-02T09:09:42.246054+0000","last_unstale":"2024-04-02T09:09:43.787108+0000","last_undegraded":"2024-04-02T09:09:43.787108+0000","last_fullsized":"2024-04-02T09:09:43.787108+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:27.369585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785824+0000","last_change":"2024-04-02T09:09:42.254714+0000","last_active":"2024-04-02T09:09:43.785824+0000","last_peered":"2024-04-02T09:09:43.785824+0000","last_clean":"2024-04-02T09:09:43.785824+0000","last_became_active":"2024-04-02T09:09:42.254317+0000","last_became_peered":"2024-04-02T09:09:42.254317+0000","last_unstale":"2024-04-02T09:09:43.785824+0000","last_undegraded":"2024-04-02T09:09:43.785824+0000","last_fullsized":"2024-04-02T09:09:43.785824+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:30:27.478757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786341+0000","last_change":"2024-04-02T09:09:42.248010+0000","last_active":"2024-04-02T09:09:43.786341+0000","last_peered":"2024-04-02T09:09:43.786341+0000","last_clean":"2024-04-02T09:09:43.786341+0000","last_became_active":"2024-04-02T09:09:42.247251+0000","last_became_peered":"2024-04-02T09:09:42.247251+0000","last_unstale":"2024-04-02T09:09:43.786341+0000","last_undegraded":"2024-04-02T09:09:43.786341+0000","last_fullsized":"2024-04-02T09:09:43.786341+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:46.919953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785799+0000","last_change":"2024-04-02T09:09:42.240712+0000","last_active":"2024-04-02T09:09:43.785799+0000","last_peered":"2024-04-02T09:09:43.785799+0000","last_clean":"2024-04-02T09:09:43.785799+0000","last_became_active":"2024-04-02T09:09:42.240458+0000","last_became_peered":"2024-04-02T09:09:42.240458+0000","last_unstale":"2024-04-02T09:09:43.785799+0000","last_undegraded":"2024-04-02T09:09:43.785799+0000","last_fullsized":"2024-04-02T09:09:43.785799+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:15:14.241480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235547+0000","last_change":"2024-04-02T09:09:42.246629+0000","last_active":"2024-04-02T09:09:43.235547+0000","last_peered":"2024-04-02T09:09:43.235547+0000","last_clean":"2024-04-02T09:09:43.235547+0000","last_became_active":"2024-04-02T09:09:42.245713+0000","last_became_peered":"2024-04-02T09:09:42.245713+0000","last_unstale":"2024-04-02T09:09:43.235547+0000","last_undegraded":"2024-04-02T09:09:43.235547+0000","last_fullsized":"2024-04-02T09:09:43.235547+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:09:38.898450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785765+0000","last_change":"2024-04-02T09:09:42.252187+0000","last_active":"2024-04-02T09:09:43.785765+0000","last_peered":"2024-04-02T09:09:43.785765+0000","last_clean":"2024-04-02T09:09:43.785765+0000","last_became_active":"2024-04-02T09:09:42.250028+0000","last_became_peered":"2024-04-02T09:09:42.250028+0000","last_unstale":"2024-04-02T09:09:43.785765+0000","last_undegraded":"2024-04-02T09:09:43.785765+0000","last_fullsized":"2024-04-02T09:09:43.785765+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:41:37.653634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595437+0000","last_change":"2024-04-02T09:09:42.245600+0000","last_active":"2024-04-02T09:09:43.595437+0000","last_peered":"2024-04-02T09:09:43.595437+0000","last_clean":"2024-04-02T09:09:43.595437+0000","last_became_active":"2024-04-02T09:09:42.245321+0000","last_became_peered":"2024-04-02T09:09:42.245321+0000","last_unstale":"2024-04-02T09:09:43.595437+0000","last_undegraded":"2024-04-02T09:09:43.595437+0000","last_fullsized":"2024-04-02T09:09:43.595437+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:49:00.800523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235812+0000","last_change":"2024-04-02T09:09:42.242356+0000","last_active":"2024-04-02T09:09:43.235812+0000","last_peered":"2024-04-02T09:09:43.235812+0000","last_clean":"2024-04-02T09:09:43.235812+0000","last_became_active":"2024-04-02T09:09:42.242074+0000","last_became_peered":"2024-04-02T09:09:42.242074+0000","last_unstale":"2024-04-02T09:09:43.235812+0000","last_undegraded":"2024-04-02T09:09:43.235812+0000","last_fullsized":"2024-04-02T09:09:43.235812+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:02.167323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.233742+0000","last_change":"2024-04-02T09:09:42.258095+0000","last_active":"2024-04-02T09:09:43.233742+0000","last_peered":"2024-04-02T09:09:43.233742+0000","last_clean":"2024-04-02T09:09:43.233742+0000","last_became_active":"2024-04-02T09:09:42.257816+0000","last_became_peered":"2024-04-02T09:09:42.257816+0000","last_unstale":"2024-04-02T09:09:43.233742+0000","last_undegraded":"2024-04-02T09:09:43.233742+0000","last_fullsized":"2024-04-02T09:09:43.233742+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:56:51.798108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595888+0000","last_change":"2024-04-02T09:09:42.253363+0000","last_active":"2024-04-02T09:09:43.595888+0000","last_peered":"2024-04-02T09:09:43.595888+0000","last_clean":"2024-04-02T09:09:43.595888+0000","last_became_active":"2024-04-02T09:09:42.253071+0000","last_became_peered":"2024-04-02T09:09:42.253071+0000","last_unstale":"2024-04-02T09:09:43.595888+0000","last_undegraded":"2024-04-02T09:09:43.595888+0000","last_fullsized":"2024-04-02T09:09:43.595888+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:07:15.570245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786744+0000","last_change":"2024-04-02T09:09:42.238430+0000","last_active":"2024-04-02T09:09:43.786744+0000","last_peered":"2024-04-02T09:09:43.786744+0000","last_clean":"2024-04-02T09:09:43.786744+0000","last_became_active":"2024-04-02T09:09:42.238133+0000","last_became_peered":"2024-04-02T09:09:42.238133+0000","last_unstale":"2024-04-02T09:09:43.786744+0000","last_undegraded":"2024-04-02T09:09:43.786744+0000","last_fullsized":"2024-04-02T09:09:43.786744+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:30:07.695501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595547+0000","last_change":"2024-04-02T09:09:42.247622+0000","last_active":"2024-04-02T09:09:43.595547+0000","last_peered":"2024-04-02T09:09:43.595547+0000","last_clean":"2024-04-02T09:09:43.595547+0000","last_became_active":"2024-04-02T09:09:42.246824+0000","last_became_peered":"2024-04-02T09:09:42.246824+0000","last_unstale":"2024-04-02T09:09:43.595547+0000","last_undegraded":"2024-04-02T09:09:43.595547+0000","last_fullsized":"2024-04-02T09:09:43.595547+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:13:09.717628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786683+0000","last_change":"2024-04-02T09:09:42.247286+0000","last_active":"2024-04-02T09:09:43.786683+0000","last_peered":"2024-04-02T09:09:43.786683+0000","last_clean":"2024-04-02T09:09:43.786683+0000","last_became_active":"2024-04-02T09:09:42.245830+0000","last_became_peered":"2024-04-02T09:09:42.245830+0000","last_unstale":"2024-04-02T09:09:43.786683+0000","last_undegraded":"2024-04-02T09:09:43.786683+0000","last_fullsized":"2024-04-02T09:09:43.786683+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:54:22.001281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786323+0000","last_change":"2024-04-02T09:09:42.247444+0000","last_active":"2024-04-02T09:09:43.786323+0000","last_peered":"2024-04-02T09:09:43.786323+0000","last_clean":"2024-04-02T09:09:43.786323+0000","last_became_active":"2024-04-02T09:09:42.246972+0000","last_became_peered":"2024-04-02T09:09:42.246972+0000","last_unstale":"2024-04-02T09:09:43.786323+0000","last_undegraded":"2024-04-02T09:09:43.786323+0000","last_fullsized":"2024-04-02T09:09:43.786323+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:38:27.306368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786533+0000","last_change":"2024-04-02T09:09:42.231733+0000","last_active":"2024-04-02T09:09:43.786533+0000","last_peered":"2024-04-02T09:09:43.786533+0000","last_clean":"2024-04-02T09:09:43.786533+0000","last_became_active":"2024-04-02T09:09:42.231449+0000","last_became_peered":"2024-04-02T09:09:42.231449+0000","last_unstale":"2024-04-02T09:09:43.786533+0000","last_undegraded":"2024-04-02T09:09:43.786533+0000","last_fullsized":"2024-04-02T09:09:43.786533+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:11:05.741327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235990+0000","last_change":"2024-04-02T09:09:42.253498+0000","last_active":"2024-04-02T09:09:43.235990+0000","last_peered":"2024-04-02T09:09:43.235990+0000","last_clean":"2024-04-02T09:09:43.235990+0000","last_became_active":"2024-04-02T09:09:42.253165+0000","last_became_peered":"2024-04-02T09:09:42.253165+0000","last_unstale":"2024-04-02T09:09:43.235990+0000","last_undegraded":"2024-04-02T09:09:43.235990+0000","last_fullsized":"2024-04-02T09:09:43.235990+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:22.111661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786524+0000","last_change":"2024-04-02T09:09:42.242893+0000","last_active":"2024-04-02T09:09:43.786524+0000","last_peered":"2024-04-02T09:09:43.786524+0000","last_clean":"2024-04-02T09:09:43.786524+0000","last_became_active":"2024-04-02T09:09:42.242636+0000","last_became_peered":"2024-04-02T09:09:42.242636+0000","last_unstale":"2024-04-02T09:09:43.786524+0000","last_undegraded":"2024-04-02T09:09:43.786524+0000","last_fullsized":"2024-04-02T09:09:43.786524+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:33.746069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595704+0000","last_change":"2024-04-02T09:09:42.253299+0000","last_active":"2024-04-02T09:09:43.595704+0000","last_peered":"2024-04-02T09:09:43.595704+0000","last_clean":"2024-04-02T09:09:43.595704+0000","last_became_active":"2024-04-02T09:09:42.253029+0000","last_became_peered":"2024-04-02T09:09:42.253029+0000","last_unstale":"2024-04-02T09:09:43.595704+0000","last_undegraded":"2024-04-02T09:09:43.595704+0000","last_fullsized":"2024-04-02T09:09:43.595704+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:16:24.179056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786756+0000","last_change":"2024-04-02T09:09:42.243890+0000","last_active":"2024-04-02T09:09:43.786756+0000","last_peered":"2024-04-02T09:09:43.786756+0000","last_clean":"2024-04-02T09:09:43.786756+0000","last_became_active":"2024-04-02T09:09:42.243633+0000","last_became_peered":"2024-04-02T09:09:42.243633+0000","last_unstale":"2024-04-02T09:09:43.786756+0000","last_undegraded":"2024-04-02T09:09:43.786756+0000","last_fullsized":"2024-04-02T09:09:43.786756+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:25.782610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786277+0000","last_change":"2024-04-02T09:09:42.245543+0000","last_active":"2024-04-02T09:09:43.786277+0000","last_peered":"2024-04-02T09:09:43.786277+0000","last_clean":"2024-04-02T09:09:43.786277+0000","last_became_active":"2024-04-02T09:09:42.245268+0000","last_became_peered":"2024-04-02T09:09:42.245268+0000","last_unstale":"2024-04-02T09:09:43.786277+0000","last_undegraded":"2024-04-02T09:09:43.786277+0000","last_fullsized":"2024-04-02T09:09:43.786277+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:40:30.420225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785958+0000","last_change":"2024-04-02T09:09:42.243451+0000","last_active":"2024-04-02T09:09:43.785958+0000","last_peered":"2024-04-02T09:09:43.785958+0000","last_clean":"2024-04-02T09:09:43.785958+0000","last_became_active":"2024-04-02T09:09:42.243195+0000","last_became_peered":"2024-04-02T09:09:42.243195+0000","last_unstale":"2024-04-02T09:09:43.785958+0000","last_undegraded":"2024-04-02T09:09:43.785958+0000","last_fullsized":"2024-04-02T09:09:43.785958+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:04.400069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787245+0000","last_change":"2024-04-02T09:09:42.246514+0000","last_active":"2024-04-02T09:09:43.787245+0000","last_peered":"2024-04-02T09:09:43.787245+0000","last_clean":"2024-04-02T09:09:43.787245+0000","last_became_active":"2024-04-02T09:09:42.246180+0000","last_became_peered":"2024-04-02T09:09:42.246180+0000","last_unstale":"2024-04-02T09:09:43.787245+0000","last_undegraded":"2024-04-02T09:09:43.787245+0000","last_fullsized":"2024-04-02T09:09:43.787245+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:01:33.122142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786239+0000","last_change":"2024-04-02T09:09:42.245998+0000","last_active":"2024-04-02T09:09:43.786239+0000","last_peered":"2024-04-02T09:09:43.786239+0000","last_clean":"2024-04-02T09:09:43.786239+0000","last_became_active":"2024-04-02T09:09:42.243896+0000","last_became_peered":"2024-04-02T09:09:42.243896+0000","last_unstale":"2024-04-02T09:09:43.786239+0000","last_undegraded":"2024-04-02T09:09:43.786239+0000","last_fullsized":"2024-04-02T09:09:43.786239+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:47.798366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787198+0000","last_change":"2024-04-02T09:09:42.242627+0000","last_active":"2024-04-02T09:09:43.787198+0000","last_peered":"2024-04-02T09:09:43.787198+0000","last_clean":"2024-04-02T09:09:43.787198+0000","last_became_active":"2024-04-02T09:09:42.242045+0000","last_became_peered":"2024-04-02T09:09:42.242045+0000","last_unstale":"2024-04-02T09:09:43.787198+0000","last_undegraded":"2024-04-02T09:09:43.787198+0000","last_fullsized":"2024-04-02T09:09:43.787198+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:10:42.313663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1f","version":"204'4","reported_seq":15,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.596036+0000","last_change":"2024-04-02T09:09:41.701429+0000","last_active":"2024-04-02T09:09:43.596036+0000","last_peered":"2024-04-02T09:09:43.596036+0000","last_clean":"2024-04-02T09:09:43.596036+0000","last_became_active":"2024-04-02T09:09:41.700916+0000","last_became_peered":"2024-04-02T09:09:41.700916+0000","last_unstale":"2024-04-02T09:09:43.596036+0000","last_undegraded":"2024-04-02T09:09:43.596036+0000","last_fullsized":"2024-04-02T09:09:43.596036+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:35:07.805757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787321+0000","last_change":"2024-04-02T09:09:42.247954+0000","last_active":"2024-04-02T09:09:43.787321+0000","last_peered":"2024-04-02T09:09:43.787321+0000","last_clean":"2024-04-02T09:09:43.787321+0000","last_became_active":"2024-04-02T09:09:42.246592+0000","last_became_peered":"2024-04-02T09:09:42.246592+0000","last_unstale":"2024-04-02T09:09:43.787321+0000","last_undegraded":"2024-04-02T09:09:43.787321+0000","last_fullsized":"2024-04-02T09:09:43.787321+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:45:38.604503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1e","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786613+0000","last_change":"2024-04-02T09:09:41.702270+0000","last_active":"2024-04-02T09:09:43.786613+0000","last_peered":"2024-04-02T09:09:43.786613+0000","last_clean":"2024-04-02T09:09:43.786613+0000","last_became_active":"2024-04-02T09:09:41.701889+0000","last_became_peered":"2024-04-02T09:09:41.701889+0000","last_unstale":"2024-04-02T09:09:43.786613+0000","last_undegraded":"2024-04-02T09:09:43.786613+0000","last_fullsized":"2024-04-02T09:09:43.786613+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:24:53.906210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235798+0000","last_change":"2024-04-02T09:09:42.249222+0000","last_active":"2024-04-02T09:09:43.235798+0000","last_peered":"2024-04-02T09:09:43.235798+0000","last_clean":"2024-04-02T09:09:43.235798+0000","last_became_active":"2024-04-02T09:09:42.248961+0000","last_became_peered":"2024-04-02T09:09:42.248961+0000","last_unstale":"2024-04-02T09:09:43.235798+0000","last_undegraded":"2024-04-02T09:09:43.235798+0000","last_fullsized":"2024-04-02T09:09:43.235798+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:47:04.855474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1d","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786601+0000","last_change":"2024-04-02T09:09:41.702154+0000","last_active":"2024-04-02T09:09:43.786601+0000","last_peered":"2024-04-02T09:09:43.786601+0000","last_clean":"2024-04-02T09:09:43.786601+0000","last_became_active":"2024-04-02T09:09:41.700854+0000","last_became_peered":"2024-04-02T09:09:41.700854+0000","last_unstale":"2024-04-02T09:09:43.786601+0000","last_undegraded":"2024-04-02T09:09:43.786601+0000","last_fullsized":"2024-04-02T09:09:43.786601+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:26.363173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235861+0000","last_change":"2024-04-02T09:09:42.257248+0000","last_active":"2024-04-02T09:09:43.235861+0000","last_peered":"2024-04-02T09:09:43.235861+0000","last_clean":"2024-04-02T09:09:43.235861+0000","last_became_active":"2024-04-02T09:09:42.254749+0000","last_became_peered":"2024-04-02T09:09:42.254749+0000","last_unstale":"2024-04-02T09:09:43.235861+0000","last_undegraded":"2024-04-02T09:09:43.235861+0000","last_fullsized":"2024-04-02T09:09:43.235861+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:25:36.871271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1c","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786451+0000","last_change":"2024-04-02T09:09:41.703057+0000","last_active":"2024-04-02T09:09:43.786451+0000","last_peered":"2024-04-02T09:09:43.786451+0000","last_clean":"2024-04-02T09:09:43.786451+0000","last_became_active":"2024-04-02T09:09:41.702657+0000","last_became_peered":"2024-04-02T09:09:41.702657+0000","last_unstale":"2024-04-02T09:09:43.786451+0000","last_undegraded":"2024-04-02T09:09:43.786451+0000","last_fullsized":"2024-04-02T09:09:43.786451+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:12:57.221082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235923+0000","last_change":"2024-04-02T09:09:42.238974+0000","last_active":"2024-04-02T09:09:43.235923+0000","last_peered":"2024-04-02T09:09:43.235923+0000","last_clean":"2024-04-02T09:09:43.235923+0000","last_became_active":"2024-04-02T09:09:42.238143+0000","last_became_peered":"2024-04-02T09:09:42.238143+0000","last_unstale":"2024-04-02T09:09:43.235923+0000","last_undegraded":"2024-04-02T09:09:43.235923+0000","last_fullsized":"2024-04-02T09:09:43.235923+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:18.060077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786657+0000","last_change":"2024-04-02T09:09:41.697992+0000","last_active":"2024-04-02T09:09:43.786657+0000","last_peered":"2024-04-02T09:09:43.786657+0000","last_clean":"2024-04-02T09:09:43.786657+0000","last_became_active":"2024-04-02T09:09:41.696876+0000","last_became_peered":"2024-04-02T09:09:41.696876+0000","last_unstale":"2024-04-02T09:09:43.786657+0000","last_undegraded":"2024-04-02T09:09:43.786657+0000","last_fullsized":"2024-04-02T09:09:43.786657+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:06:25.992709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786475+0000","last_change":"2024-04-02T09:09:42.252164+0000","last_active":"2024-04-02T09:09:43.786475+0000","last_peered":"2024-04-02T09:09:43.786475+0000","last_clean":"2024-04-02T09:09:43.786475+0000","last_became_active":"2024-04-02T09:09:42.248376+0000","last_became_peered":"2024-04-02T09:09:42.248376+0000","last_unstale":"2024-04-02T09:09:43.786475+0000","last_undegraded":"2024-04-02T09:09:43.786475+0000","last_fullsized":"2024-04-02T09:09:43.786475+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:07:11.878538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.1a","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786582+0000","last_change":"2024-04-02T09:09:41.703163+0000","last_active":"2024-04-02T09:09:43.786582+0000","last_peered":"2024-04-02T09:09:43.786582+0000","last_clean":"2024-04-02T09:09:43.786582+0000","last_became_active":"2024-04-02T09:09:41.702886+0000","last_became_peered":"2024-04-02T09:09:41.702886+0000","last_unstale":"2024-04-02T09:09:43.786582+0000","last_undegraded":"2024-04-02T09:09:43.786582+0000","last_fullsized":"2024-04-02T09:09:43.786582+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:59.773755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785813+0000","last_change":"2024-04-02T09:09:42.233652+0000","last_active":"2024-04-02T09:09:43.785813+0000","last_peered":"2024-04-02T09:09:43.785813+0000","last_clean":"2024-04-02T09:09:43.785813+0000","last_became_active":"2024-04-02T09:09:42.233275+0000","last_became_peered":"2024-04-02T09:09:42.233275+0000","last_unstale":"2024-04-02T09:09:43.785813+0000","last_undegraded":"2024-04-02T09:09:43.785813+0000","last_fullsized":"2024-04-02T09:09:43.785813+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:32:15.281056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.233142+0000","last_change":"2024-04-02T09:09:41.689454+0000","last_active":"2024-04-02T09:09:43.233142+0000","last_peered":"2024-04-02T09:09:43.233142+0000","last_clean":"2024-04-02T09:09:43.233142+0000","last_became_active":"2024-04-02T09:09:41.689187+0000","last_became_peered":"2024-04-02T09:09:41.689187+0000","last_unstale":"2024-04-02T09:09:43.233142+0000","last_undegraded":"2024-04-02T09:09:43.233142+0000","last_fullsized":"2024-04-02T09:09:43.233142+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:58.000014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595843+0000","last_change":"2024-04-02T09:09:42.244345+0000","last_active":"2024-04-02T09:09:43.595843+0000","last_peered":"2024-04-02T09:09:43.595843+0000","last_clean":"2024-04-02T09:09:43.595843+0000","last_became_active":"2024-04-02T09:09:42.244075+0000","last_became_peered":"2024-04-02T09:09:42.244075+0000","last_unstale":"2024-04-02T09:09:43.595843+0000","last_undegraded":"2024-04-02T09:09:43.595843+0000","last_fullsized":"2024-04-02T09:09:43.595843+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:35:14.920256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786052+0000","last_change":"2024-04-02T09:09:41.704076+0000","last_active":"2024-04-02T09:09:43.786052+0000","last_peered":"2024-04-02T09:09:43.786052+0000","last_clean":"2024-04-02T09:09:43.786052+0000","last_became_active":"2024-04-02T09:09:41.702808+0000","last_became_peered":"2024-04-02T09:09:41.702808+0000","last_unstale":"2024-04-02T09:09:43.786052+0000","last_undegraded":"2024-04-02T09:09:43.786052+0000","last_fullsized":"2024-04-02T09:09:43.786052+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:45.913542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787337+0000","last_change":"2024-04-02T09:09:42.247947+0000","last_active":"2024-04-02T09:09:43.787337+0000","last_peered":"2024-04-02T09:09:43.787337+0000","last_clean":"2024-04-02T09:09:43.787337+0000","last_became_active":"2024-04-02T09:09:42.247636+0000","last_became_peered":"2024-04-02T09:09:42.247636+0000","last_unstale":"2024-04-02T09:09:43.787337+0000","last_undegraded":"2024-04-02T09:09:43.787337+0000","last_fullsized":"2024-04-02T09:09:43.787337+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:43:16.873370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787326+0000","last_change":"2024-04-02T09:09:41.701973+0000","last_active":"2024-04-02T09:09:43.787326+0000","last_peered":"2024-04-02T09:09:43.787326+0000","last_clean":"2024-04-02T09:09:43.787326+0000","last_became_active":"2024-04-02T09:09:41.701605+0000","last_became_peered":"2024-04-02T09:09:41.701605+0000","last_unstale":"2024-04-02T09:09:43.787326+0000","last_undegraded":"2024-04-02T09:09:43.787326+0000","last_fullsized":"2024-04-02T09:09:43.787326+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:18:06.740716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786342+0000","last_change":"2024-04-02T09:09:42.247840+0000","last_active":"2024-04-02T09:09:43.786342+0000","last_peered":"2024-04-02T09:09:43.786342+0000","last_clean":"2024-04-02T09:09:43.786342+0000","last_became_active":"2024-04-02T09:09:42.246872+0000","last_became_peered":"2024-04-02T09:09:42.246872+0000","last_unstale":"2024-04-02T09:09:43.786342+0000","last_undegraded":"2024-04-02T09:09:43.786342+0000","last_fullsized":"2024-04-02T09:09:43.786342+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:44:23.519150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.15","version":"204'3","reported_seq":14,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786699+0000","last_change":"2024-04-02T09:09:41.706298+0000","last_active":"2024-04-02T09:09:43.786699+0000","last_peered":"2024-04-02T09:09:43.786699+0000","last_clean":"2024-04-02T09:09:43.786699+0000","last_became_active":"2024-04-02T09:09:41.705296+0000","last_became_peered":"2024-04-02T09:09:41.705296+0000","last_unstale":"2024-04-02T09:09:43.786699+0000","last_undegraded":"2024-04-02T09:09:43.786699+0000","last_fullsized":"2024-04-02T09:09:43.786699+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:00:21.176162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787106+0000","last_change":"2024-04-02T09:09:42.242612+0000","last_active":"2024-04-02T09:09:43.787106+0000","last_peered":"2024-04-02T09:09:43.787106+0000","last_clean":"2024-04-02T09:09:43.787106+0000","last_became_active":"2024-04-02T09:09:42.241681+0000","last_became_peered":"2024-04-02T09:09:42.241681+0000","last_unstale":"2024-04-02T09:09:43.787106+0000","last_undegraded":"2024-04-02T09:09:43.787106+0000","last_fullsized":"2024-04-02T09:09:43.787106+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:04:12.501107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.14","version":"204'6","reported_seq":17,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786850+0000","last_change":"2024-04-02T09:09:41.702571+0000","last_active":"2024-04-02T09:09:43.786850+0000","last_peered":"2024-04-02T09:09:43.786850+0000","last_clean":"2024-04-02T09:09:43.786850+0000","last_became_active":"2024-04-02T09:09:41.702287+0000","last_became_peered":"2024-04-02T09:09:41.702287+0000","last_unstale":"2024-04-02T09:09:43.786850+0000","last_undegraded":"2024-04-02T09:09:43.786850+0000","last_fullsized":"2024-04-02T09:09:43.786850+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:33:51.218638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786757+0000","last_change":"2024-04-02T09:09:42.245949+0000","last_active":"2024-04-02T09:09:43.786757+0000","last_peered":"2024-04-02T09:09:43.786757+0000","last_clean":"2024-04-02T09:09:43.786757+0000","last_became_active":"2024-04-02T09:09:42.245555+0000","last_became_peered":"2024-04-02T09:09:42.245555+0000","last_unstale":"2024-04-02T09:09:43.786757+0000","last_undegraded":"2024-04-02T09:09:43.786757+0000","last_fullsized":"2024-04-02T09:09:43.786757+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:32:36.042213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.13","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786307+0000","last_change":"2024-04-02T09:09:41.699908+0000","last_active":"2024-04-02T09:09:43.786307+0000","last_peered":"2024-04-02T09:09:43.786307+0000","last_clean":"2024-04-02T09:09:43.786307+0000","last_became_active":"2024-04-02T09:09:41.699215+0000","last_became_peered":"2024-04-02T09:09:41.699215+0000","last_unstale":"2024-04-02T09:09:43.786307+0000","last_undegraded":"2024-04-02T09:09:43.786307+0000","last_fullsized":"2024-04-02T09:09:43.786307+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:12.251299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.233449+0000","last_change":"2024-04-02T09:09:42.248728+0000","last_active":"2024-04-02T09:09:43.233449+0000","last_peered":"2024-04-02T09:09:43.233449+0000","last_clean":"2024-04-02T09:09:43.233449+0000","last_became_active":"2024-04-02T09:09:42.248479+0000","last_became_peered":"2024-04-02T09:09:42.248479+0000","last_unstale":"2024-04-02T09:09:43.233449+0000","last_undegraded":"2024-04-02T09:09:43.233449+0000","last_fullsized":"2024-04-02T09:09:43.233449+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:13:25.782610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786841+0000","last_change":"2024-04-02T09:09:41.703075+0000","last_active":"2024-04-02T09:09:43.786841+0000","last_peered":"2024-04-02T09:09:43.786841+0000","last_clean":"2024-04-02T09:09:43.786841+0000","last_became_active":"2024-04-02T09:09:41.702419+0000","last_became_peered":"2024-04-02T09:09:41.702419+0000","last_unstale":"2024-04-02T09:09:43.786841+0000","last_undegraded":"2024-04-02T09:09:43.786841+0000","last_fullsized":"2024-04-02T09:09:43.786841+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:47:26.806954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785983+0000","last_change":"2024-04-02T09:09:42.245777+0000","last_active":"2024-04-02T09:09:43.785983+0000","last_peered":"2024-04-02T09:09:43.785983+0000","last_clean":"2024-04-02T09:09:43.785983+0000","last_became_active":"2024-04-02T09:09:42.244519+0000","last_became_peered":"2024-04-02T09:09:42.244519+0000","last_unstale":"2024-04-02T09:09:43.785983+0000","last_undegraded":"2024-04-02T09:09:43.785983+0000","last_fullsized":"2024-04-02T09:09:43.785983+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:36:14.965927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787106+0000","last_change":"2024-04-02T09:09:41.698008+0000","last_active":"2024-04-02T09:09:43.787106+0000","last_peered":"2024-04-02T09:09:43.787106+0000","last_clean":"2024-04-02T09:09:43.787106+0000","last_became_active":"2024-04-02T09:09:41.697689+0000","last_became_peered":"2024-04-02T09:09:41.697689+0000","last_unstale":"2024-04-02T09:09:43.787106+0000","last_undegraded":"2024-04-02T09:09:43.787106+0000","last_fullsized":"2024-04-02T09:09:43.787106+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:55.340371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786910+0000","last_change":"2024-04-02T09:09:42.252918+0000","last_active":"2024-04-02T09:09:43.786910+0000","last_peered":"2024-04-02T09:09:43.786910+0000","last_clean":"2024-04-02T09:09:43.786910+0000","last_became_active":"2024-04-02T09:09:42.252622+0000","last_became_peered":"2024-04-02T09:09:42.252622+0000","last_unstale":"2024-04-02T09:09:43.786910+0000","last_undegraded":"2024-04-02T09:09:43.786910+0000","last_fullsized":"2024-04-02T09:09:43.786910+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:02.167323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.10","version":"204'8","reported_seq":19,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.233144+0000","last_change":"2024-04-02T09:09:41.696623+0000","last_active":"2024-04-02T09:09:43.233144+0000","last_peered":"2024-04-02T09:09:43.233144+0000","last_clean":"2024-04-02T09:09:43.233144+0000","last_became_active":"2024-04-02T09:09:41.696203+0000","last_became_peered":"2024-04-02T09:09:41.696203+0000","last_unstale":"2024-04-02T09:09:43.233144+0000","last_undegraded":"2024-04-02T09:09:43.233144+0000","last_fullsized":"2024-04-02T09:09:43.233144+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:04:11.494696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787205+0000","last_change":"2024-04-02T09:09:42.243606+0000","last_active":"2024-04-02T09:09:43.787205+0000","last_peered":"2024-04-02T09:09:43.787205+0000","last_clean":"2024-04-02T09:09:43.787205+0000","last_became_active":"2024-04-02T09:09:42.243330+0000","last_became_peered":"2024-04-02T09:09:42.243330+0000","last_unstale":"2024-04-02T09:09:43.787205+0000","last_undegraded":"2024-04-02T09:09:43.787205+0000","last_fullsized":"2024-04-02T09:09:43.787205+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:51:52.730338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.f","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787428+0000","last_change":"2024-04-02T09:09:41.701659+0000","last_active":"2024-04-02T09:09:43.787428+0000","last_peered":"2024-04-02T09:09:43.787428+0000","last_clean":"2024-04-02T09:09:43.787428+0000","last_became_active":"2024-04-02T09:09:41.701384+0000","last_became_peered":"2024-04-02T09:09:41.701384+0000","last_unstale":"2024-04-02T09:09:43.787428+0000","last_undegraded":"2024-04-02T09:09:43.787428+0000","last_fullsized":"2024-04-02T09:09:43.787428+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:51:51.723926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786612+0000","last_change":"2024-04-02T09:09:42.249444+0000","last_active":"2024-04-02T09:09:43.786612+0000","last_peered":"2024-04-02T09:09:43.786612+0000","last_clean":"2024-04-02T09:09:43.786612+0000","last_became_active":"2024-04-02T09:09:42.248608+0000","last_became_peered":"2024-04-02T09:09:42.248608+0000","last_unstale":"2024-04-02T09:09:43.786612+0000","last_undegraded":"2024-04-02T09:09:43.786612+0000","last_fullsized":"2024-04-02T09:09:43.786612+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:05:43.102187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.e","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.236009+0000","last_change":"2024-04-02T09:09:41.690750+0000","last_active":"2024-04-02T09:09:43.236009+0000","last_peered":"2024-04-02T09:09:43.236009+0000","last_clean":"2024-04-02T09:09:43.236009+0000","last_became_active":"2024-04-02T09:09:41.690461+0000","last_became_peered":"2024-04-02T09:09:41.690461+0000","last_unstale":"2024-04-02T09:09:43.236009+0000","last_undegraded":"2024-04-02T09:09:43.236009+0000","last_fullsized":"2024-04-02T09:09:43.236009+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:05.915049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235889+0000","last_change":"2024-04-02T09:09:42.252927+0000","last_active":"2024-04-02T09:09:43.235889+0000","last_peered":"2024-04-02T09:09:43.235889+0000","last_clean":"2024-04-02T09:09:43.235889+0000","last_became_active":"2024-04-02T09:09:42.251020+0000","last_became_peered":"2024-04-02T09:09:42.251020+0000","last_unstale":"2024-04-02T09:09:43.235889+0000","last_undegraded":"2024-04-02T09:09:43.235889+0000","last_fullsized":"2024-04-02T09:09:43.235889+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:43:11.717821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.d","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785928+0000","last_change":"2024-04-02T09:09:41.700719+0000","last_active":"2024-04-02T09:09:43.785928+0000","last_peered":"2024-04-02T09:09:43.785928+0000","last_clean":"2024-04-02T09:09:43.785928+0000","last_became_active":"2024-04-02T09:09:41.700442+0000","last_became_peered":"2024-04-02T09:09:41.700442+0000","last_unstale":"2024-04-02T09:09:43.785928+0000","last_undegraded":"2024-04-02T09:09:43.785928+0000","last_fullsized":"2024-04-02T09:09:43.785928+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:36:13.959516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786448+0000","last_change":"2024-04-02T09:09:42.240056+0000","last_active":"2024-04-02T09:09:43.786448+0000","last_peered":"2024-04-02T09:09:43.786448+0000","last_clean":"2024-04-02T09:09:43.786448+0000","last_became_active":"2024-04-02T09:09:42.239816+0000","last_became_peered":"2024-04-02T09:09:42.239816+0000","last_unstale":"2024-04-02T09:09:43.786448+0000","last_undegraded":"2024-04-02T09:09:43.786448+0000","last_fullsized":"2024-04-02T09:09:43.786448+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:39.357706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786083+0000","last_change":"2024-04-02T09:09:41.696017+0000","last_active":"2024-04-02T09:09:43.786083+0000","last_peered":"2024-04-02T09:09:43.786083+0000","last_clean":"2024-04-02T09:09:43.786083+0000","last_became_active":"2024-04-02T09:09:41.695114+0000","last_became_peered":"2024-04-02T09:09:41.695114+0000","last_unstale":"2024-04-02T09:09:43.786083+0000","last_undegraded":"2024-04-02T09:09:43.786083+0000","last_fullsized":"2024-04-02T09:09:43.786083+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:38.351295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235674+0000","last_change":"2024-04-02T09:09:42.235837+0000","last_active":"2024-04-02T09:09:43.235674+0000","last_peered":"2024-04-02T09:09:43.235674+0000","last_clean":"2024-04-02T09:09:43.235674+0000","last_became_active":"2024-04-02T09:09:42.235225+0000","last_became_peered":"2024-04-02T09:09:42.235225+0000","last_unstale":"2024-04-02T09:09:43.235674+0000","last_undegraded":"2024-04-02T09:09:43.235674+0000","last_fullsized":"2024-04-02T09:09:43.235674+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:07:11.878538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.b","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786794+0000","last_change":"2024-04-02T09:09:41.697089+0000","last_active":"2024-04-02T09:09:43.786794+0000","last_peered":"2024-04-02T09:09:43.786794+0000","last_clean":"2024-04-02T09:09:43.786794+0000","last_became_active":"2024-04-02T09:09:41.696796+0000","last_became_peered":"2024-04-02T09:09:41.696796+0000","last_unstale":"2024-04-02T09:09:43.786794+0000","last_undegraded":"2024-04-02T09:09:43.786794+0000","last_fullsized":"2024-04-02T09:09:43.786794+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:29:14.904075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.233417+0000","last_change":"2024-04-02T09:09:42.250501+0000","last_active":"2024-04-02T09:09:43.233417+0000","last_peered":"2024-04-02T09:09:43.233417+0000","last_clean":"2024-04-02T09:09:43.233417+0000","last_became_active":"2024-04-02T09:09:42.249931+0000","last_became_peered":"2024-04-02T09:09:42.249931+0000","last_unstale":"2024-04-02T09:09:43.233417+0000","last_undegraded":"2024-04-02T09:09:43.233417+0000","last_fullsized":"2024-04-02T09:09:43.233417+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:38:27.306368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235770+0000","last_change":"2024-04-02T09:09:41.697209+0000","last_active":"2024-04-02T09:09:43.235770+0000","last_peered":"2024-04-02T09:09:43.235770+0000","last_clean":"2024-04-02T09:09:43.235770+0000","last_became_active":"2024-04-02T09:09:41.695512+0000","last_became_peered":"2024-04-02T09:09:41.695512+0000","last_unstale":"2024-04-02T09:09:43.235770+0000","last_undegraded":"2024-04-02T09:09:43.235770+0000","last_fullsized":"2024-04-02T09:09:43.235770+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:36.674318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785775+0000","last_change":"2024-04-02T09:09:42.242910+0000","last_active":"2024-04-02T09:09:43.785775+0000","last_peered":"2024-04-02T09:09:43.785775+0000","last_clean":"2024-04-02T09:09:43.785775+0000","last_became_active":"2024-04-02T09:09:42.241582+0000","last_became_peered":"2024-04-02T09:09:42.241582+0000","last_unstale":"2024-04-02T09:09:43.785775+0000","last_undegraded":"2024-04-02T09:09:43.785775+0000","last_fullsized":"2024-04-02T09:09:43.785775+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:33:38.345884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235604+0000","last_change":"2024-04-02T09:09:41.704460+0000","last_active":"2024-04-02T09:09:43.235604+0000","last_peered":"2024-04-02T09:09:43.235604+0000","last_clean":"2024-04-02T09:09:43.235604+0000","last_became_active":"2024-04-02T09:09:41.703676+0000","last_became_peered":"2024-04-02T09:09:41.703676+0000","last_unstale":"2024-04-02T09:09:43.235604+0000","last_undegraded":"2024-04-02T09:09:43.235604+0000","last_fullsized":"2024-04-02T09:09:43.235604+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:55:54.788666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786461+0000","last_change":"2024-04-02T09:09:42.239798+0000","last_active":"2024-04-02T09:09:43.786461+0000","last_peered":"2024-04-02T09:09:43.786461+0000","last_clean":"2024-04-02T09:09:43.786461+0000","last_became_active":"2024-04-02T09:09:42.239548+0000","last_became_peered":"2024-04-02T09:09:42.239548+0000","last_unstale":"2024-04-02T09:09:43.786461+0000","last_undegraded":"2024-04-02T09:09:43.786461+0000","last_fullsized":"2024-04-02T09:09:43.786461+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:56:51.798108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785751+0000","last_change":"2024-04-02T09:09:41.701344+0000","last_active":"2024-04-02T09:09:43.785751+0000","last_peered":"2024-04-02T09:09:43.785751+0000","last_clean":"2024-04-02T09:09:43.785751+0000","last_became_active":"2024-04-02T09:09:41.700538+0000","last_became_peered":"2024-04-02T09:09:41.700538+0000","last_unstale":"2024-04-02T09:09:43.785751+0000","last_undegraded":"2024-04-02T09:09:43.785751+0000","last_fullsized":"2024-04-02T09:09:43.785751+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:40:29.413813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595902+0000","last_change":"2024-04-02T09:09:42.258348+0000","last_active":"2024-04-02T09:09:43.595902+0000","last_peered":"2024-04-02T09:09:43.595902+0000","last_clean":"2024-04-02T09:09:43.595902+0000","last_became_active":"2024-04-02T09:09:42.258064+0000","last_became_peered":"2024-04-02T09:09:42.258064+0000","last_unstale":"2024-04-02T09:09:43.595902+0000","last_undegraded":"2024-04-02T09:09:43.595902+0000","last_fullsized":"2024-04-02T09:09:43.595902+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:56:26.514942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.7","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786148+0000","last_change":"2024-04-02T09:09:41.690341+0000","last_active":"2024-04-02T09:09:43.786148+0000","last_peered":"2024-04-02T09:09:43.786148+0000","last_clean":"2024-04-02T09:09:43.786148+0000","last_became_active":"2024-04-02T09:09:41.690083+0000","last_became_peered":"2024-04-02T09:09:41.690083+0000","last_unstale":"2024-04-02T09:09:43.786148+0000","last_undegraded":"2024-04-02T09:09:43.786148+0000","last_fullsized":"2024-04-02T09:09:43.786148+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:05:42.095775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.596001+0000","last_change":"2024-04-02T09:09:42.237565+0000","last_active":"2024-04-02T09:09:43.596001+0000","last_peered":"2024-04-02T09:09:43.596001+0000","last_clean":"2024-04-02T09:09:43.596001+0000","last_became_active":"2024-04-02T09:09:42.237290+0000","last_became_peered":"2024-04-02T09:09:42.237290+0000","last_unstale":"2024-04-02T09:09:43.596001+0000","last_undegraded":"2024-04-02T09:09:43.596001+0000","last_fullsized":"2024-04-02T09:09:43.596001+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:36.821746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595969+0000","last_change":"2024-04-02T09:09:41.705910+0000","last_active":"2024-04-02T09:09:43.595969+0000","last_peered":"2024-04-02T09:09:43.595969+0000","last_clean":"2024-04-02T09:09:43.595969+0000","last_became_active":"2024-04-02T09:09:41.705355+0000","last_became_peered":"2024-04-02T09:09:41.705355+0000","last_unstale":"2024-04-02T09:09:43.595969+0000","last_undegraded":"2024-04-02T09:09:43.595969+0000","last_fullsized":"2024-04-02T09:09:43.595969+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:07:10.872126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.595659+0000","last_change":"2024-04-02T09:09:42.239824+0000","last_active":"2024-04-02T09:09:43.595659+0000","last_peered":"2024-04-02T09:09:43.595659+0000","last_clean":"2024-04-02T09:09:43.595659+0000","last_became_active":"2024-04-02T09:09:42.239571+0000","last_became_peered":"2024-04-02T09:09:42.239571+0000","last_unstale":"2024-04-02T09:09:43.595659+0000","last_undegraded":"2024-04-02T09:09:43.595659+0000","last_fullsized":"2024-04-02T09:09:43.595659+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:15:04.716813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.596033+0000","last_change":"2024-04-02T09:09:41.705067+0000","last_active":"2024-04-02T09:09:43.596033+0000","last_peered":"2024-04-02T09:09:43.596033+0000","last_clean":"2024-04-02T09:09:43.596033+0000","last_became_active":"2024-04-02T09:09:41.704081+0000","last_became_peered":"2024-04-02T09:09:41.704081+0000","last_unstale":"2024-04-02T09:09:43.596033+0000","last_undegraded":"2024-04-02T09:09:43.596033+0000","last_fullsized":"2024-04-02T09:09:43.596033+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:17.053666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786491+0000","last_change":"2024-04-02T09:09:42.253453+0000","last_active":"2024-04-02T09:09:43.786491+0000","last_peered":"2024-04-02T09:09:43.786491+0000","last_clean":"2024-04-02T09:09:43.786491+0000","last_became_active":"2024-04-02T09:09:42.253128+0000","last_became_peered":"2024-04-02T09:09:42.253128+0000","last_unstale":"2024-04-02T09:09:43.786491+0000","last_undegraded":"2024-04-02T09:09:43.786491+0000","last_fullsized":"2024-04-02T09:09:43.786491+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:35:08.812169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786941+0000","last_change":"2024-04-02T09:09:41.704495+0000","last_active":"2024-04-02T09:09:43.786941+0000","last_peered":"2024-04-02T09:09:43.786941+0000","last_clean":"2024-04-02T09:09:43.786941+0000","last_became_active":"2024-04-02T09:09:41.704197+0000","last_became_peered":"2024-04-02T09:09:41.704197+0000","last_unstale":"2024-04-02T09:09:43.786941+0000","last_undegraded":"2024-04-02T09:09:43.786941+0000","last_fullsized":"2024-04-02T09:09:43.786941+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:45:23.586259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235600+0000","last_change":"2024-04-02T09:09:42.251210+0000","last_active":"2024-04-02T09:09:43.235600+0000","last_peered":"2024-04-02T09:09:43.235600+0000","last_clean":"2024-04-02T09:09:43.235600+0000","last_became_active":"2024-04-02T09:09:42.249687+0000","last_became_peered":"2024-04-02T09:09:42.249687+0000","last_unstale":"2024-04-02T09:09:43.235600+0000","last_undegraded":"2024-04-02T09:09:43.235600+0000","last_fullsized":"2024-04-02T09:09:43.235600+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:43.199123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235668+0000","last_change":"2024-04-02T09:09:41.695183+0000","last_active":"2024-04-02T09:09:43.235668+0000","last_peered":"2024-04-02T09:09:43.235668+0000","last_clean":"2024-04-02T09:09:43.235668+0000","last_became_active":"2024-04-02T09:09:41.694918+0000","last_became_peered":"2024-04-02T09:09:41.694918+0000","last_unstale":"2024-04-02T09:09:43.235668+0000","last_undegraded":"2024-04-02T09:09:43.235668+0000","last_fullsized":"2024-04-02T09:09:43.235668+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:32.358193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.785895+0000","last_change":"2024-04-02T09:09:42.239879+0000","last_active":"2024-04-02T09:09:43.785895+0000","last_peered":"2024-04-02T09:09:43.785895+0000","last_clean":"2024-04-02T09:09:43.785895+0000","last_became_active":"2024-04-02T09:09:42.239502+0000","last_became_peered":"2024-04-02T09:09:42.239502+0000","last_unstale":"2024-04-02T09:09:43.785895+0000","last_undegraded":"2024-04-02T09:09:43.785895+0000","last_fullsized":"2024-04-02T09:09:43.785895+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:41:06.071269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235652+0000","last_change":"2024-04-02T09:09:41.695104+0000","last_active":"2024-04-02T09:09:43.235652+0000","last_peered":"2024-04-02T09:09:43.235652+0000","last_clean":"2024-04-02T09:09:43.235652+0000","last_became_active":"2024-04-02T09:09:41.694755+0000","last_became_peered":"2024-04-02T09:09:41.694755+0000","last_unstale":"2024-04-02T09:09:43.235652+0000","last_undegraded":"2024-04-02T09:09:43.235652+0000","last_fullsized":"2024-04-02T09:09:43.235652+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:29:22.618092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235432+0000","last_change":"2024-04-02T09:09:42.246880+0000","last_active":"2024-04-02T09:09:43.235432+0000","last_peered":"2024-04-02T09:09:43.235432+0000","last_clean":"2024-04-02T09:09:43.235432+0000","last_became_active":"2024-04-02T09:09:42.246426+0000","last_became_peered":"2024-04-02T09:09:42.246426+0000","last_unstale":"2024-04-02T09:09:43.235432+0000","last_undegraded":"2024-04-02T09:09:43.235432+0000","last_fullsized":"2024-04-02T09:09:43.235432+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:17.246822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235545+0000","last_change":"2024-04-02T09:09:41.696354+0000","last_active":"2024-04-02T09:09:43.235545+0000","last_peered":"2024-04-02T09:09:43.235545+0000","last_clean":"2024-04-02T09:09:43.235545+0000","last_became_active":"2024-04-02T09:09:41.696061+0000","last_became_peered":"2024-04-02T09:09:41.696061+0000","last_unstale":"2024-04-02T09:09:43.235545+0000","last_undegraded":"2024-04-02T09:09:43.235545+0000","last_fullsized":"2024-04-02T09:09:43.235545+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:45:21.309725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.787016+0000","last_change":"2024-04-02T09:09:42.238260+0000","last_active":"2024-04-02T09:09:43.787016+0000","last_peered":"2024-04-02T09:09:43.787016+0000","last_clean":"2024-04-02T09:09:43.787016+0000","last_became_active":"2024-04-02T09:09:42.237968+0000","last_became_peered":"2024-04-02T09:09:42.237968+0000","last_unstale":"2024-04-02T09:09:43.787016+0000","last_undegraded":"2024-04-02T09:09:43.787016+0000","last_fullsized":"2024-04-02T09:09:43.787016+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:59:48.937092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235624+0000","last_change":"2024-04-02T09:09:41.700043+0000","last_active":"2024-04-02T09:09:43.235624+0000","last_peered":"2024-04-02T09:09:43.235624+0000","last_clean":"2024-04-02T09:09:43.235624+0000","last_became_active":"2024-04-02T09:09:41.699034+0000","last_became_peered":"2024-04-02T09:09:41.699034+0000","last_unstale":"2024-04-02T09:09:43.235624+0000","last_undegraded":"2024-04-02T09:09:43.235624+0000","last_fullsized":"2024-04-02T09:09:43.235624+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:18.870825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.235634+0000","last_change":"2024-04-02T09:09:42.239645+0000","last_active":"2024-04-02T09:09:43.235634+0000","last_peered":"2024-04-02T09:09:43.235634+0000","last_clean":"2024-04-02T09:09:43.235634+0000","last_became_active":"2024-04-02T09:09:42.239149+0000","last_became_peered":"2024-04-02T09:09:42.239149+0000","last_unstale":"2024-04-02T09:09:43.235634+0000","last_undegraded":"2024-04-02T09:09:43.235634+0000","last_fullsized":"2024-04-02T09:09:43.235634+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:41.675345+0000","last_clean_scrub_stamp":"2024-04-02T09:09:41.675345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:33:36.680512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-04-02T09:09:43.786221+0000","last_change":"2024-04-02T09:09:41.697172+0000","last_active":"2024-04-02T09:09:43.786221+0000","last_peered":"2024-04-02T09:09:43.786221+0000","last_clean":"2024-04-02T09:09:43.786221+0000","last_became_active":"2024-04-02T09:09:41.696927+0000","last_became_peered":"2024-04-02T09:09:41.696927+0000","last_unstale":"2024-04-02T09:09:43.786221+0000","last_undegraded":"2024-04-02T09:09:43.786221+0000","last_fullsized":"2024-04-02T09:09:43.786221+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:09:40.668933+0000","last_clean_scrub_stamp":"2024-04-02T09:09:40.668933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:52:39.629358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"194'326","reported_seq":1115,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-02T09:09:42.236996+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:09:42.236996+0000","last_peered":"2024-04-02T09:09:42.236996+0000","last_clean":"2024-04-02T09:09:42.236996+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:09:42.236996+0000","last_undegraded":"2024-04-02T09:09:42.236996+0000","last_fullsized":"2024-04-02T09:09:42.236996+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575958,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59888,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311952,"statfs":{"total":96636764160,"available":96575438848,"internally_reserved":0,"allocated":2588672,"data_stored":1299912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.71199999999999997,"15min":0.71999999999999997},"min":{"1min":0.41599999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.97999999999999998,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.72599999999999998,"15min":0.73499999999999999},"min":{"1min":0.501,"5min":0.46899999999999997,"15min":0.442},"max":{"1min":0.89400000000000002,"5min":1.052,"15min":2.0680000000000001},"last":0.53400000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.73699999999999999,"15min":0.73599999999999999},"min":{"1min":0.57699999999999996,"5min":0.48499999999999999,"15min":0.45500000000000002},"max":{"1min":0.879,"5min":0.94299999999999995,"15min":1.125},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.70699999999999996,"15min":0.72499999999999998},"min":{"1min":0.53900000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.86299999999999999,"5min":1.0109999999999999,"15min":2.133},"last":0.63}]},{"osd":2,"last update":"Tue Apr 2 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.51400000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88100000000000001,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.81000000000000005,"15min":0.80900000000000005},"min":{"1min":0.64300000000000002,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.92700000000000005,"5min":1.0229999999999999,"15min":2.02},"last":0.82399999999999995}]},{"osd":3,"last update":"Tue Apr 2 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.748,"15min":0.753},"min":{"1min":0.51300000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92600000000000005,"5min":1.452,"15min":1.8859999999999999},"last":0.753},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.76200000000000001,"15min":0.76800000000000002},"min":{"1min":0.57899999999999996,"5min":0.52400000000000002,"15min":0.45700000000000002},"max":{"1min":0.88400000000000001,"5min":1.0389999999999999,"15min":1.855},"last":0.70399999999999996}]},{"osd":4,"last update":"Tue Apr 2 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38100000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":2.093},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.73299999999999998,"15min":0.71299999999999997},"min":{"1min":0.47399999999999998,"5min":0.42099999999999999,"15min":0.38300000000000001},"max":{"1min":0.89800000000000002,"5min":1.536,"15min":1.536},"last":1.008}]},{"osd":5,"last update":"Tue Apr 2 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68000000000000005,"15min":0.66900000000000004},"min":{"1min":0.48199999999999998,"5min":0.41299999999999998,"15min":0.41199999999999998},"max":{"1min":0.86299999999999999,"5min":0.88600000000000001,"15min":1.1850000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.40600000000000003},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":2.2480000000000002},"last":0.85799999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68200000000000005,"15min":0.68999999999999995},"min":{"1min":0.52400000000000002,"5min":0.48399999999999999,"15min":0.45800000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":2.9670000000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.70299999999999996,"15min":0.68400000000000005},"min":{"1min":0.54100000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":1.0489999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575956,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41828,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330012,"statfs":{"total":96636764160,"available":96593932288,"internally_reserved":0,"allocated":2572288,"data_stored":1298326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73499999999999999,"15min":0.73799999999999999},"min":{"1min":0.57799999999999996,"5min":0.42699999999999999,"15min":0.36799999999999999},"max":{"1min":0.90600000000000003,"5min":1,"15min":1.038},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.747,"15min":0.75700000000000001},"min":{"1min":0.64900000000000002,"5min":0.45800000000000002,"15min":0.441},"max":{"1min":0.89700000000000002,"5min":0.94499999999999995,"15min":1.077},"last":0.52000000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72499999999999998,"15min":0.72999999999999998},"min":{"1min":0.54000000000000004,"5min":0.48099999999999998,"15min":0.438},"max":{"1min":0.84599999999999997,"5min":0.95099999999999996,"15min":1.212},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.74199999999999999,"15min":0.76000000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.872,"5min":1.0109999999999999,"15min":1.1759999999999999},"last":0.57499999999999996}]},{"osd":2,"last update":"Tue Apr 2 09:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.75,"15min":0.76200000000000001},"min":{"1min":0.627,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":1.2789999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.74399999999999999,"15min":0.753},"min":{"1min":0.65800000000000003,"5min":0.53100000000000003,"15min":0.46700000000000003},"max":{"1min":0.95999999999999996,"5min":1.002,"15min":1.226},"last":0.55000000000000004}]},{"osd":3,"last update":"Tue Apr 2 09:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.753,"15min":0.77600000000000002},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.497},"max":{"1min":0.90800000000000003,"5min":0.96099999999999997,"15min":1.2549999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.77400000000000002,"15min":0.77600000000000002},"min":{"1min":0.60699999999999998,"5min":0.52700000000000002,"15min":0.42799999999999999},"max":{"1min":0.94199999999999995,"5min":1.0189999999999999,"15min":1.3200000000000001},"last":0.67600000000000005}]},{"osd":4,"last update":"Tue Apr 2 09:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.61299999999999999,"15min":0.628},"min":{"1min":0.44900000000000001,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":0.78200000000000003,"5min":0.88600000000000001,"15min":1.1259999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.64900000000000002,"15min":0.67000000000000004},"min":{"1min":0.49199999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88900000000000001,"5min":0.98299999999999998,"15min":1.1879999999999999},"last":0.78900000000000003}]},{"osd":5,"last update":"Tue Apr 2 09:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64600000000000002,"15min":0.66100000000000003},"min":{"1min":0.497,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.875,"5min":0.96699999999999997,"15min":1.2669999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.71399999999999997,"15min":0.72899999999999998},"min":{"1min":0.52200000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94699999999999995,"5min":1.054,"15min":1.204},"last":0.77700000000000002}]},{"osd":7,"last update":"Tue Apr 2 09:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.73099999999999998,"15min":0.74299999999999999},"min":{"1min":0.54100000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91800000000000004,"5min":1.0269999999999999,"15min":1.2370000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.58499999999999996,"5min":0.49399999999999999,"15min":0.48299999999999998},"max":{"1min":0.93100000000000005,"5min":1.02,"15min":1.1519999999999999},"last":0.75700000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575957,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52244,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319596,"statfs":{"total":96636764160,"available":96583266304,"internally_reserved":0,"allocated":3149824,"data_stored":1881856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.45100000000000001,"15min":0.45600000000000002},"min":{"1min":0.311,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51600000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32100000000000001},"max":{"1min":0.73699999999999999,"5min":0.78600000000000003,"15min":0.79800000000000004},"last":0.41799999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.376,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.86399999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.45200000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77500000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.443}]},{"osd":3,"last update":"Tue Apr 2 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.57999999999999996},"min":{"1min":0.42299999999999999,"5min":0.41699999999999998,"15min":0.37},"max":{"1min":0.80200000000000005,"5min":0.80500000000000005,"15min":0.88100000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.40400000000000003,"5min":0.378,"15min":0.375},"max":{"1min":0.86699999999999999,"5min":0.878,"15min":0.878},"last":0.498}]},{"osd":4,"last update":"Tue Apr 2 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.58999999999999997,"5min":0.433,"15min":0.433},"max":{"1min":1.077,"5min":1.077,"15min":1.1479999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76500000000000001,"15min":0.79300000000000004},"min":{"1min":0.57199999999999995,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.034,"5min":1.22,"15min":1.25},"last":0.72399999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78000000000000003,"15min":0.80100000000000005},"min":{"1min":0.57699999999999996,"5min":0.54500000000000004,"15min":0.52900000000000003},"max":{"1min":1.115,"5min":1.1220000000000001,"15min":1.321},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79800000000000004,"15min":0.81299999999999994},"min":{"1min":0.61799999999999999,"5min":0.55300000000000005,"15min":0.53500000000000003},"max":{"1min":1.05,"5min":1.0980000000000001,"15min":1.173},"last":0.85299999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.80600000000000005,"15min":0.82199999999999995},"min":{"1min":0.61199999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.159,"5min":1.159,"15min":1.2210000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.80300000000000005,"15min":0.81399999999999995},"min":{"1min":0.64000000000000001,"5min":0.57999999999999996,"15min":0.5},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.2529999999999999},"last":0.83199999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:08:41 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81100000000000005,"15min":0.83299999999999996},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.53700000000000003},"max":{"1min":1.1000000000000001,"5min":1.159,"15min":1.236},"last":0.755},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.83199999999999996,"15min":0.84199999999999997},"min":{"1min":0.62,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.048,"5min":1.1850000000000001,"15min":1.2789999999999999},"last":0.79300000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575957,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47576,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324264,"statfs":{"total":96636764160,"available":96588046336,"internally_reserved":0,"allocated":2560000,"data_stored":1291085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45100000000000001,"15min":0.46200000000000002},"min":{"1min":0.35299999999999998,"5min":0.311,"15min":0.28599999999999998},"max":{"1min":0.58099999999999996,"5min":0.78100000000000003,"15min":0.86699999999999999},"last":0.499},{"interface":"front","average":{"1min":0.503,"5min":0.496,"15min":0.51600000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64100000000000001,"5min":0.73299999999999998,"15min":0.79900000000000004},"last":0.45200000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.51800000000000002},"min":{"1min":0.41299999999999998,"5min":0.35599999999999998,"15min":0.33900000000000002},"max":{"1min":0.68000000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55200000000000005,"15min":0.55900000000000005},"min":{"1min":0.41899999999999998,"5min":0.376,"15min":0.36399999999999999},"max":{"1min":0.67400000000000004,"5min":0.752,"15min":0.84199999999999997},"last":0.52400000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.56899999999999995},"min":{"1min":0.39800000000000002,"5min":0.374,"15min":0.34200000000000003},"max":{"1min":0.70799999999999996,"5min":0.85699999999999998,"15min":0.97899999999999998},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64500000000000002},"min":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.45000000000000001},"max":{"1min":0.76300000000000001,"5min":0.88400000000000001,"15min":0.96399999999999997},"last":0.627}]},{"osd":4,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.84499999999999997,"15min":0.83999999999999997},"min":{"1min":0.70299999999999996,"5min":0.58699999999999997,"15min":0.45600000000000002},"max":{"1min":1.129,"5min":1.129,"15min":1.3580000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.83599999999999997,"15min":0.83099999999999996},"min":{"1min":0.69799999999999995,"5min":0.55800000000000005,"15min":0.48199999999999998},"max":{"1min":1.0640000000000001,"5min":1.1439999999999999,"15min":1.3009999999999999},"last":0.88700000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.83699999999999997,"15min":0.83399999999999996},"min":{"1min":0.66100000000000003,"5min":0.60099999999999998,"15min":0.498},"max":{"1min":0.96599999999999997,"5min":1.163,"15min":1.232},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.83399999999999996,"15min":0.83999999999999997},"min":{"1min":0.68300000000000005,"5min":0.59999999999999998,"15min":0.55900000000000005},"max":{"1min":1.014,"5min":1.165,"15min":1.1719999999999999},"last":0.91900000000000004}]},{"osd":6,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80200000000000005,"15min":0.80400000000000005},"min":{"1min":0.58499999999999996,"5min":0.55600000000000005,"15min":0.51500000000000001},"max":{"1min":0.94599999999999995,"5min":1.196,"15min":1.196},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.59299999999999997,"5min":0.54900000000000004,"15min":0.499},"max":{"1min":1.085,"5min":1.179,"15min":1.234},"last":0.71699999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.86499999999999999,"15min":0.86399999999999999},"min":{"1min":0.67600000000000005,"5min":0.629,"15min":0.58899999999999997},"max":{"1min":1.0349999999999999,"5min":1.1499999999999999,"15min":1.2609999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84899999999999998,"15min":0.85199999999999998},"min":{"1min":0.60599999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.0189999999999999,"5min":1.1910000000000001,"15min":1.2589999999999999},"last":0.98199999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575956,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51680,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320160,"statfs":{"total":96636764160,"available":96583843840,"internally_reserved":0,"allocated":2572288,"data_stored":1298306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.45600000000000002,"15min":0.44500000000000001},"min":{"1min":0.33500000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66600000000000004,"5min":0.69399999999999995,"15min":0.78700000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52200000000000002,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79900000000000004,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.33700000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.51200000000000001},"min":{"1min":0.434,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.81799999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56299999999999994},"min":{"1min":0.44900000000000001,"5min":0.38900000000000001,"15min":0.378},"max":{"1min":0.71999999999999997,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.38200000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55300000000000005,"15min":0.56799999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.34599999999999997},"max":{"1min":0.73599999999999999,"5min":1.2629999999999999,"15min":1.2749999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64800000000000002,"15min":0.64300000000000002},"min":{"1min":0.505,"5min":0.48299999999999998,"15min":0.47899999999999998},"max":{"1min":0.83699999999999997,"5min":0.97699999999999998,"15min":1.0740000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.78600000000000003,"15min":0.79000000000000004},"min":{"1min":0.52400000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0700000000000001,"5min":1.087,"15min":1.206},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.68000000000000005,"5min":0.56999999999999995,"15min":0.56599999999999995},"max":{"1min":0.98599999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.996}]},{"osd":5,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.78400000000000003,"15min":0.79800000000000004},"min":{"1min":0.60899999999999999,"5min":0.55700000000000005,"15min":0.52600000000000002},"max":{"1min":1.004,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.81599999999999995,"15min":0.81399999999999995},"min":{"1min":0.64200000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.101,"5min":1.1259999999999999,"15min":1.2490000000000001},"last":0.96799999999999997}]},{"osd":6,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81799999999999995,"15min":0.82199999999999995},"min":{"1min":0.65500000000000003,"5min":0.54700000000000004,"15min":0.501},"max":{"1min":1.141,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.173},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81799999999999995,"15min":0.82299999999999995},"min":{"1min":0.66300000000000003,"5min":0.56599999999999995,"15min":0.55000000000000004},"max":{"1min":0.98899999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":1.0349999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.83599999999999997,"15min":0.84199999999999997},"min":{"1min":0.73599999999999999,"5min":0.55100000000000005,"15min":0.55000000000000004},"max":{"1min":1.095,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.83899999999999997,"15min":0.84099999999999997},"min":{"1min":0.70899999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.0860000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.82599999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575956,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51680,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320160,"statfs":{"total":96636764160,"available":96583843840,"internally_reserved":0,"allocated":2572288,"data_stored":1293122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45300000000000001,"15min":0.45400000000000001},"min":{"1min":0.318,"5min":0.30599999999999999,"15min":0.30299999999999999},"max":{"1min":0.57299999999999995,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.499,"15min":0.499},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.30599999999999999},"max":{"1min":0.65400000000000003,"5min":0.746,"15min":0.876},"last":0.438}]},{"osd":1,"last update":"Tue Apr 2 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.38700000000000001,"5min":0.32500000000000001,"15min":0.311},"max":{"1min":0.64900000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.34100000000000003},"max":{"1min":0.71799999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.68899999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.627,"15min":0.63700000000000001},"min":{"1min":0.47599999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83899999999999997,"5min":0.90600000000000003,"15min":1.2869999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.627,"15min":0.63500000000000001},"min":{"1min":0.46300000000000002,"5min":0.44600000000000001,"15min":0.34999999999999998},"max":{"1min":0.78000000000000003,"5min":0.91800000000000004,"15min":1.274},"last":0.53300000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.76200000000000001,"15min":0.76900000000000002},"min":{"1min":0.53900000000000003,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0369999999999999,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.748},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.58199999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.97199999999999998,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.72099999999999997}]},{"osd":5,"last update":"Tue Apr 2 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.78900000000000003,"15min":0.78400000000000003},"min":{"1min":0.57499999999999996,"5min":0.57299999999999995,"15min":0.55100000000000005},"max":{"1min":1.0469999999999999,"5min":1.099,"15min":1.157},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78800000000000003,"15min":0.79100000000000004},"min":{"1min":0.57799999999999996,"5min":0.55200000000000005,"15min":0.496},"max":{"1min":1.052,"5min":1.1120000000000001,"15min":1.129},"last":0.69799999999999995}]},{"osd":6,"last update":"Tue Apr 2 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.79300000000000004,"15min":0.79500000000000004},"min":{"1min":0.65600000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0089999999999999,"5min":1.22,"15min":1.22},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.82599999999999996,"15min":0.81999999999999995},"min":{"1min":0.64900000000000002,"5min":0.55700000000000005,"15min":0.50600000000000001},"max":{"1min":1.085,"5min":1.091,"15min":1.147},"last":0.875}]},{"osd":7,"last update":"Tue Apr 2 09:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.60399999999999998,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0189999999999999,"5min":1.181,"15min":1.2010000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.83599999999999997,"15min":0.83199999999999996},"min":{"1min":0.66400000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.006,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.76500000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575956,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47588,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324252,"statfs":{"total":96636764160,"available":96588034048,"internally_reserved":0,"allocated":2572288,"data_stored":1298270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68700000000000006,"15min":0.70899999999999996},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.29299999999999998},"max":{"1min":0.83899999999999997,"5min":0.91800000000000004,"15min":1.0389999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.68500000000000005,"15min":0.69799999999999995},"min":{"1min":0.54700000000000004,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86199999999999999,"5min":1.075,"15min":1.075},"last":0.623}]},{"osd":1,"last update":"Tue Apr 2 09:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.755,"15min":0.77300000000000002},"min":{"1min":0.58899999999999997,"5min":0.48699999999999999,"15min":0.434},"max":{"1min":0.88200000000000001,"5min":1.069,"15min":1.099},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.78500000000000003},"min":{"1min":0.57399999999999995,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89800000000000002,"5min":0.98499999999999999,"15min":1.0800000000000001},"last":1.03}]},{"osd":2,"last update":"Tue Apr 2 09:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71999999999999997,"15min":0.72599999999999998},"min":{"1min":0.502,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.93999999999999995,"5min":1.0129999999999999,"15min":1.089},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72899999999999998,"15min":0.74099999999999999},"min":{"1min":0.58999999999999997,"5min":0.501,"15min":0.41299999999999998},"max":{"1min":0.84399999999999997,"5min":0.95699999999999996,"15min":1.0329999999999999},"last":1.0740000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78300000000000003,"15min":0.78100000000000003},"min":{"1min":0.53600000000000003,"5min":0.50600000000000001,"15min":0.47399999999999998},"max":{"1min":0.998,"5min":1.018,"15min":1.0700000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.752,"15min":0.76800000000000002},"min":{"1min":0.59499999999999997,"5min":0.505,"15min":0.505},"max":{"1min":0.96099999999999997,"5min":1.028,"15min":1.038},"last":0.92700000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.60699999999999998,"15min":0.61099999999999999},"min":{"1min":0.44900000000000001,"5min":0.41499999999999998,"15min":0.34000000000000002},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":1.1140000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.46700000000000003,"5min":0.40999999999999998,"15min":0.35099999999999998},"max":{"1min":0.90700000000000003,"5min":0.90900000000000003,"15min":0.93100000000000005},"last":0.85099999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.65600000000000003,"15min":0.64000000000000001},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.42699999999999999},"max":{"1min":0.92000000000000004,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.70899999999999996,"15min":0.69299999999999995},"min":{"1min":0.48699999999999999,"5min":0.46999999999999997,"15min":0.38400000000000001},"max":{"1min":0.96999999999999997,"5min":1.0029999999999999,"15min":1.0469999999999999},"last":0.872}]},{"osd":7,"last update":"Tue Apr 2 09:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.72399999999999998,"15min":0.70799999999999996},"min":{"1min":0.58799999999999997,"5min":0.48299999999999998,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.0329999999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.79200000000000004,"15min":0.78600000000000003},"min":{"1min":0.61899999999999999,"5min":0.51000000000000001,"15min":0.501},"max":{"1min":1.008,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":1.0980000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575959,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48156,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323684,"statfs":{"total":96636764160,"available":96587452416,"internally_reserved":0,"allocated":3153920,"data_stored":1881441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69199999999999995,"15min":0.70799999999999996},"min":{"1min":0.47499999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86799999999999999,"5min":0.98999999999999999,"15min":1.004},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71699999999999997,"15min":0.72499999999999998},"min":{"1min":0.49399999999999999,"5min":0.432,"15min":0.432},"max":{"1min":0.97199999999999998,"5min":0.996,"15min":1.0600000000000001},"last":0.78800000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.70899999999999996,"15min":0.72699999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91300000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72199999999999998,"15min":0.73099999999999998},"min":{"1min":0.55500000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91600000000000004,"5min":0.99099999999999999,"15min":1.052},"last":0.69399999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73799999999999999,"15min":0.746},"min":{"1min":0.53400000000000003,"5min":0.433,"15min":0.433},"max":{"1min":0.92800000000000005,"5min":0.995,"15min":1.014},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.749,"15min":0.747},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.93000000000000005,"5min":1.026,"15min":1.0680000000000001},"last":0.64900000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.77000000000000002,"15min":0.79000000000000004},"min":{"1min":0.57399999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93500000000000005,"5min":1.036,"15min":1.083},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.80000000000000004,"15min":0.79200000000000004},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.1020000000000001},"last":0.71299999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.626,"15min":0.622},"min":{"1min":0.438,"5min":0.41599999999999998,"15min":0.373},"max":{"1min":0.90700000000000003,"5min":1.054,"15min":1.054},"last":0.62},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.48499999999999999,"5min":0.45400000000000001,"15min":0.39900000000000002},"max":{"1min":0.91800000000000004,"5min":0.95299999999999996,"15min":0.996},"last":0.73299999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.66600000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.41699999999999998},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71199999999999997,"15min":0.70499999999999996},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.38700000000000001},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":1.01},"last":0.76300000000000001}]},{"osd":7,"last update":"Tue Apr 2 09:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73299999999999998,"15min":0.72799999999999998},"min":{"1min":0.60399999999999998,"5min":0.49099999999999999,"15min":0.45200000000000001},"max":{"1min":0.93600000000000005,"5min":0.98799999999999999,"15min":1.0349999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.76200000000000001,"15min":0.75800000000000001},"min":{"1min":0.49199999999999999,"5min":0.48999999999999999,"15min":0.45700000000000002},"max":{"1min":1.018,"5min":1.079,"15min":1.079},"last":0.752}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:09:47.711 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T09:09:47.711 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T09:09:47.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.945+0000 7f32c9400700 1 Processor -- start 2024-04-02T09:09:47.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.945+0000 7f32c9400700 1 -- start start 2024-04-02T09:09:47.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.946+0000 7f32c9400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f32c415dfe0 0x7f32c40ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:47.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.946+0000 7f32c9400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c40cb310 con 0x7f32c40ce670 2024-04-02T09:09:47.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.946+0000 7f32c9400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c40711a0 con 0x7f32c415a5c0 2024-04-02T09:09:47.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.946+0000 7f32c9400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32c40ceb00 con 0x7f32c415dfe0 2024-04-02T09:09:47.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.947+0000 7f32c1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f32c415dfe0 0x7f32c40ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:47.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.947+0000 7f32c3200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f32c40ce670 0x7f32c40cba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36568/0 (socket says 172.21.15.89:36568) 2024-04-02T09:09:47.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.947+0000 7f32c3200700 1 -- 172.21.15.89:0/2171869967 learned_addr learned my addr 172.21.15.89:0/2171869967 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.947+0000 7f32c1e00700 1 -- 172.21.15.89:0/2171869967 >> v1:172.21.15.113:6789/0 conn(0x7f32c415a5c0 legacy=0x7f32c40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.947+0000 7f32c1e00700 1 -- 172.21.15.89:0/2171869967 >> v1:172.21.15.89:6789/0 conn(0x7f32c40ce670 legacy=0x7f32c40cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.947+0000 7f32c1e00700 1 -- 172.21.15.89:0/2171869967 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c4071480 con 0x7f32c415dfe0 2024-04-02T09:09:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.947+0000 7f32c1400700 1 -- 172.21.15.89:0/2171869967 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084733636 0 0) 0x7f32c40cb310 con 0x7f32c40ce670 2024-04-02T09:09:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.947+0000 7f32c1e00700 1 --2- 172.21.15.89:0/2171869967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c415dfe0 0x7f32c40ce130 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009840 tx=0x7f32b0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc6ec01dc460038 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.947+0000 7f32c1400700 1 -- 172.21.15.89:0/2171869967 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b0004230 con 0x7f32c415dfe0 2024-04-02T09:09:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.948+0000 7f32c1400700 1 -- 172.21.15.89:0/2171869967 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b0004390 con 0x7f32c415dfe0 2024-04-02T09:09:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.948+0000 7f32c1400700 1 -- 172.21.15.89:0/2171869967 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b00036a0 con 0x7f32c415dfe0 2024-04-02T09:09:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.948+0000 7f32c9400700 1 -- 172.21.15.89:0/2171869967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c415dfe0 msgr2=0x7f32c40ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.948+0000 7f32c9400700 1 --2- 172.21.15.89:0/2171869967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c415dfe0 0x7f32c40ce130 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009840 tx=0x7f32b0009b50 comp rx=0 tx=0).stop 2024-04-02T09:09:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.948+0000 7f32c9400700 1 -- 172.21.15.89:0/2171869967 shutdown_connections 2024-04-02T09:09:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.948+0000 7f32c9400700 1 --2- 172.21.15.89:0/2171869967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c415dfe0 0x7f32c40ce130 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.948+0000 7f32c9400700 1 -- 172.21.15.89:0/2171869967 >> 172.21.15.89:0/2171869967 conn(0x7f32c40b0d70 msgr2=0x7f32c40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:47.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.948+0000 7f32c9400700 1 -- 172.21.15.89:0/2171869967 shutdown_connections 2024-04-02T09:09:47.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.948+0000 7f32c9400700 1 -- 172.21.15.89:0/2171869967 wait complete. 2024-04-02T09:09:47.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.950+0000 7f32c9400700 1 Processor -- start 2024-04-02T09:09:47.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.950+0000 7f32c9400700 1 -- start start 2024-04-02T09:09:47.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.951+0000 7f32c9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c40ce670 0x7f32c4154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:47.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.951+0000 7f32c9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32c415a5c0 0x7f32c40acbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.951+0000 7f32c9400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c415dfe0 0x7f32c40adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.951+0000 7f32c9400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c40ae560 con 0x7f32c40ce670 2024-04-02T09:09:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.951+0000 7f32c9400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c4149e40 con 0x7f32c415dfe0 2024-04-02T09:09:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.951+0000 7f32c9400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c40aee10 con 0x7f32c415a5c0 2024-04-02T09:09:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.951+0000 7f32c2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c40ce670 0x7f32c4154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.951+0000 7f32c2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c40ce670 0x7f32c4154e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40512/0 (socket says 172.21.15.89:40512) 2024-04-02T09:09:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.951+0000 7f32c2800700 1 -- 172.21.15.89:0/1437827372 learned_addr learned my addr 172.21.15.89:0/1437827372 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.951+0000 7f32c3200700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c415dfe0 0x7f32c40adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.951+0000 7f32c1e00700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32c415a5c0 0x7f32c40acbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.952+0000 7f32c2800700 1 -- 172.21.15.89:0/1437827372 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32c415a5c0 msgr2=0x7f32c40acbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.952+0000 7f32c2800700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32c415a5c0 0x7f32c40acbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.952+0000 7f32c2800700 1 -- 172.21.15.89:0/1437827372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c415dfe0 msgr2=0x7f32c40adbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:47.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.952+0000 7f32c2800700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c415dfe0 0x7f32c40adbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.952+0000 7f32c2800700 1 -- 172.21.15.89:0/1437827372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b00094a0 con 0x7f32c40ce670 2024-04-02T09:09:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.952+0000 7f32c1e00700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32c415a5c0 0x7f32c40acbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.952+0000 7f32c3200700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c415dfe0 0x7f32c40adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.952+0000 7f32c2800700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c40ce670 0x7f32c4154e90 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f32b400c4d0 tx=0x7f32b400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:47.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.952+0000 7f32baa00700 1 -- 172.21.15.89:0/1437827372 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b4011790 con 0x7f32c40ce670 2024-04-02T09:09:47.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.952+0000 7f32baa00700 1 -- 172.21.15.89:0/1437827372 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b40118f0 con 0x7f32c40ce670 2024-04-02T09:09:47.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.952+0000 7f32c9400700 1 -- 172.21.15.89:0/1437827372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c40ae280 con 0x7f32c40ce670 2024-04-02T09:09:47.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.952+0000 7f32baa00700 1 -- 172.21.15.89:0/1437827372 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b4011ca0 con 0x7f32c40ce670 2024-04-02T09:09:47.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.952+0000 7f32c9400700 1 -- 172.21.15.89:0/1437827372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c40af390 con 0x7f32c40ce670 2024-04-02T09:09:47.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.953+0000 7f32c9400700 1 -- 172.21.15.89:0/1437827372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c4067720 con 0x7f32c40ce670 2024-04-02T09:09:47.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.954+0000 7f32baa00700 1 -- 172.21.15.89:0/1437827372 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f32b401c460 con 0x7f32c40ce670 2024-04-02T09:09:47.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.955+0000 7f32baa00700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f32a406adb0 0x7f32a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:47.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.955+0000 7f32baa00700 1 -- 172.21.15.89:0/1437827372 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f32b40893d0 con 0x7f32c40ce670 2024-04-02T09:09:47.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.955+0000 7f32c1e00700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f32a406adb0 0x7f32a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:47.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.956+0000 7f32c1e00700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f32a406adb0 0x7f32a406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32b00090b0 tx=0x7f32b0008030 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:47.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:47.959+0000 7f32baa00700 1 -- 172.21.15.89:0/1437827372 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f32b4054370 con 0x7f32c40ce670 2024-04-02T09:09:48.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.243+0000 7f32c9400700 1 -- 172.21.15.89:0/1437827372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f32c40ccc40 con 0x7f32c40ce670 2024-04-02T09:09:48.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.247+0000 7f32baa00700 1 -- 172.21.15.89:0/1437827372 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v52) v1 ==== 190+0+0 (secure 0 0 0) 0x7f32b4057990 con 0x7f32c40ce670 2024-04-02T09:09:48.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.250+0000 7f32c9400700 1 -- 172.21.15.89:0/1437827372 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f32a406adb0 msgr2=0x7f32a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:48.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.250+0000 7f32c9400700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f32a406adb0 0x7f32a406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32b00090b0 tx=0x7f32b0008030 comp rx=0 tx=0).stop 2024-04-02T09:09:48.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.250+0000 7f32c9400700 1 -- 172.21.15.89:0/1437827372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c40ce670 msgr2=0x7f32c4154e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:48.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.250+0000 7f32c9400700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c40ce670 0x7f32c4154e90 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f32b400c4d0 tx=0x7f32b400c890 comp rx=0 tx=0).stop 2024-04-02T09:09:48.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.250+0000 7f32c9400700 1 -- 172.21.15.89:0/1437827372 shutdown_connections 2024-04-02T09:09:48.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.250+0000 7f32c9400700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f32a406adb0 0x7f32a406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:48.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.250+0000 7f32c9400700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c415dfe0 0x7f32c40adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:48.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.250+0000 7f32c9400700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32c40ce670 0x7f32c4154e90 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:48.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.250+0000 7f32c9400700 1 --2- 172.21.15.89:0/1437827372 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32c415a5c0 0x7f32c40acbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:48.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.250+0000 7f32c9400700 1 -- 172.21.15.89:0/1437827372 >> 172.21.15.89:0/1437827372 conn(0x7f32c40b0d70 msgr2=0x7f32c40b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:48.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.250+0000 7f32c9400700 1 -- 172.21.15.89:0/1437827372 shutdown_connections 2024-04-02T09:09:48.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.250+0000 7f32c9400700 1 -- 172.21.15.89:0/1437827372 wait complete. 2024-04-02T09:09:48.251 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T09:09:48.267 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:09:48.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.460+0000 7f518bc00700 1 Processor -- start 2024-04-02T09:09:48.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.460+0000 7f518bc00700 1 -- start start 2024-04-02T09:09:48.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.461+0000 7f518bc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5184160f10 0x7f51840d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:48.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.461+0000 7f518bc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f51840cf230 0x7f51840d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:48.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.461+0000 7f518bc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51840ceda0 con 0x7f5184161590 2024-04-02T09:09:48.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.461+0000 7f518bc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51840d23d0 con 0x7f5184160f10 2024-04-02T09:09:48.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.461+0000 7f518bc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51841612f0 con 0x7f51840cf230 2024-04-02T09:09:48.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.462+0000 7f5189200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5184160f10 0x7f51840d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:48.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.462+0000 7f5189200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5184160f10 0x7f51840d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40514/0 (socket says 172.21.15.89:40514) 2024-04-02T09:09:48.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.462+0000 7f5189200700 1 -- 172.21.15.89:0/369749264 learned_addr learned my addr 172.21.15.89:0/369749264 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:48.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.462+0000 7f5183e00700 1 --2- 172.21.15.89:0/369749264 >> v2:172.21.15.113:3300/0 conn(0x7f51840cf230 0x7f51840d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:48.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.462+0000 7f5189200700 1 -- 172.21.15.89:0/369749264 >> v2:172.21.15.113:3300/0 conn(0x7f51840cf230 msgr2=0x7f51840d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:48.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.462+0000 7f5189200700 1 --2- 172.21.15.89:0/369749264 >> v2:172.21.15.113:3300/0 conn(0x7f51840cf230 0x7f51840d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:48.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.462+0000 7f5189200700 1 -- 172.21.15.89:0/369749264 >> v1:172.21.15.113:6789/0 conn(0x7f5184161590 legacy=0x7f51840b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:48.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.462+0000 7f5189200700 1 -- 172.21.15.89:0/369749264 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51840ce470 con 0x7f5184160f10 2024-04-02T09:09:48.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.462+0000 7f5189200700 1 --2- 172.21.15.89:0/369749264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5184160f10 0x7f51840d18b0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f518408cdb0 tx=0x7f5178009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d5571a633f8260 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:48.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.463+0000 7f5183400700 1 -- 172.21.15.89:0/369749264 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517800b870 con 0x7f5184160f10 2024-04-02T09:09:48.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.463+0000 7f5183400700 1 -- 172.21.15.89:0/369749264 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517800b9d0 con 0x7f5184160f10 2024-04-02T09:09:48.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.463+0000 7f5183400700 1 -- 172.21.15.89:0/369749264 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517800bd60 con 0x7f5184160f10 2024-04-02T09:09:48.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.463+0000 7f518bc00700 1 -- 172.21.15.89:0/369749264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5184160f10 msgr2=0x7f51840d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:48.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.463+0000 7f518bc00700 1 --2- 172.21.15.89:0/369749264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5184160f10 0x7f51840d18b0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f518408cdb0 tx=0x7f5178009bd0 comp rx=0 tx=0).stop 2024-04-02T09:09:48.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.463+0000 7f518bc00700 1 -- 172.21.15.89:0/369749264 shutdown_connections 2024-04-02T09:09:48.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.463+0000 7f518bc00700 1 --2- 172.21.15.89:0/369749264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5184160f10 0x7f51840d18b0 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:48.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.463+0000 7f518bc00700 1 --2- 172.21.15.89:0/369749264 >> v2:172.21.15.113:3300/0 conn(0x7f51840cf230 0x7f51840d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:48.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.463+0000 7f518bc00700 1 -- 172.21.15.89:0/369749264 >> 172.21.15.89:0/369749264 conn(0x7f518408aff0 msgr2=0x7f51840d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:48.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.463+0000 7f518bc00700 1 -- 172.21.15.89:0/369749264 shutdown_connections 2024-04-02T09:09:48.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.463+0000 7f518bc00700 1 -- 172.21.15.89:0/369749264 wait complete. 2024-04-02T09:09:48.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.465+0000 7f518bc00700 1 Processor -- start 2024-04-02T09:09:48.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.465+0000 7f518bc00700 1 -- start start 2024-04-02T09:09:48.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.466+0000 7f518bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51840cf230 0x7f5184148600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:48.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.466+0000 7f518bc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5184160f10 0x7f5184146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:48.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.466+0000 7f518bc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5184161590 0x7f51840b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:48.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.466+0000 7f518bc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51840b79d0 con 0x7f5184161590 2024-04-02T09:09:48.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.466+0000 7f518bc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5184148b40 con 0x7f5184160f10 2024-04-02T09:09:48.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.466+0000 7f518bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5184156f00 con 0x7f51840cf230 2024-04-02T09:09:48.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.466+0000 7f5189c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5184161590 0x7f51840b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:48.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.466+0000 7f5189c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5184161590 0x7f51840b7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40520/0 (socket says 172.21.15.89:40520) 2024-04-02T09:09:48.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.466+0000 7f5189c00700 1 -- 172.21.15.89:0/1054596 learned_addr learned my addr 172.21.15.89:0/1054596 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:48.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.466+0000 7f5189200700 1 --2- 172.21.15.89:0/1054596 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51840cf230 0x7f5184148600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:48.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.466+0000 7f5183e00700 1 --2- 172.21.15.89:0/1054596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5184160f10 0x7f5184146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:48.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.467+0000 7f5189c00700 1 -- 172.21.15.89:0/1054596 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51840cf230 msgr2=0x7f5184148600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:48.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.467+0000 7f5189c00700 1 --2- 172.21.15.89:0/1054596 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51840cf230 0x7f5184148600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:48.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.467+0000 7f5189c00700 1 -- 172.21.15.89:0/1054596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5184160f10 msgr2=0x7f5184146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:48.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.467+0000 7f5189c00700 1 --2- 172.21.15.89:0/1054596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5184160f10 0x7f5184146650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:48.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.467+0000 7f5189c00700 1 -- 172.21.15.89:0/1054596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5178009550 con 0x7f5184161590 2024-04-02T09:09:48.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.467+0000 7f5189c00700 1 --2- 172.21.15.89:0/1054596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5184161590 0x7f51840b7df0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f517c00cb50 tx=0x7f517c00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:48.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.467+0000 7f5180c00700 1 -- 172.21.15.89:0/1054596 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c003c00 con 0x7f5184161590 2024-04-02T09:09:48.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.467+0000 7f5180c00700 1 -- 172.21.15.89:0/1054596 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517c003d60 con 0x7f5184161590 2024-04-02T09:09:48.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.467+0000 7f5180c00700 1 -- 172.21.15.89:0/1054596 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c00ab70 con 0x7f5184161590 2024-04-02T09:09:48.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.467+0000 7f518bc00700 1 -- 172.21.15.89:0/1054596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5184159900 con 0x7f5184161590 2024-04-02T09:09:48.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.467+0000 7f518bc00700 1 -- 172.21.15.89:0/1054596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5184149470 con 0x7f5184161590 2024-04-02T09:09:48.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.468+0000 7f518bc00700 1 -- 172.21.15.89:0/1054596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51840b7b10 con 0x7f5184161590 2024-04-02T09:09:48.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.469+0000 7f5180c00700 1 -- 172.21.15.89:0/1054596 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f517c00acd0 con 0x7f5184161590 2024-04-02T09:09:48.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.470+0000 7f5180c00700 1 --2- 172.21.15.89:0/1054596 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f516406adb0 0x7f516406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:48.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.470+0000 7f5180c00700 1 -- 172.21.15.89:0/1054596 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f517c0896c0 con 0x7f5184161590 2024-04-02T09:09:48.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.470+0000 7f5189200700 1 --2- 172.21.15.89:0/1054596 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f516406adb0 0x7f516406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:48.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.471+0000 7f5189200700 1 --2- 172.21.15.89:0/1054596 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f516406adb0 0x7f516406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5178000c00 tx=0x7f5178008030 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:48.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.474+0000 7f5180c00700 1 -- 172.21.15.89:0/1054596 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f51840b7b10 con 0x7f5184161590 2024-04-02T09:09:48.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.721+0000 7f518bc00700 1 -- 172.21.15.89:0/1054596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51840b7b10 con 0x7f5184161590 2024-04-02T09:09:48.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.722+0000 7f5180c00700 1 -- 172.21.15.89:0/1054596 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4748 (secure 0 0 0) 0x7f51840b7b10 con 0x7f5184161590 2024-04-02T09:09:48.721 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:09:48.721 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1311594829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1311594829},{"type":"v1","addr":"172.21.15.113:6834","nonce":1311594829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1256547418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1256547418},{"type":"v1","addr":"172.21.15.89:6837","nonce":1256547418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1137044351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1137044351},{"type":"v1","addr":"172.21.15.113:6835","nonce":1137044351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:09:42.220191+0000","modified":"2024-04-02T09:09:45.699081+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10638":{"gid":10638,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/2257849268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2257849268},{"type":"v1","addr":"172.21.15.89:6836","nonce":2257849268}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-04-02T09:09:48.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.724+0000 7f518bc00700 1 -- 172.21.15.89:0/1054596 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f516406adb0 msgr2=0x7f516406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:48.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.724+0000 7f518bc00700 1 --2- 172.21.15.89:0/1054596 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f516406adb0 0x7f516406d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5178000c00 tx=0x7f5178008030 comp rx=0 tx=0).stop 2024-04-02T09:09:48.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.724+0000 7f518bc00700 1 -- 172.21.15.89:0/1054596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5184161590 msgr2=0x7f51840b7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:48.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.724+0000 7f518bc00700 1 --2- 172.21.15.89:0/1054596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5184161590 0x7f51840b7df0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f517c00cb50 tx=0x7f517c00cf10 comp rx=0 tx=0).stop 2024-04-02T09:09:48.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.724+0000 7f518bc00700 1 -- 172.21.15.89:0/1054596 shutdown_connections 2024-04-02T09:09:48.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.724+0000 7f518bc00700 1 --2- 172.21.15.89:0/1054596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5184161590 0x7f51840b7df0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:48.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.724+0000 7f518bc00700 1 --2- 172.21.15.89:0/1054596 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51840cf230 0x7f5184148600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:48.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.724+0000 7f518bc00700 1 --2- 172.21.15.89:0/1054596 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f516406adb0 0x7f516406d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:48.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.724+0000 7f518bc00700 1 --2- 172.21.15.89:0/1054596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5184160f10 0x7f5184146650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:48.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.724+0000 7f518bc00700 1 -- 172.21.15.89:0/1054596 >> 172.21.15.89:0/1054596 conn(0x7f518408aff0 msgr2=0x7f51840cad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:48.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.725+0000 7f518bc00700 1 -- 172.21.15.89:0/1054596 shutdown_connections 2024-04-02T09:09:48.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:48.725+0000 7f518bc00700 1 -- 172.21.15.89:0/1054596 wait complete. 2024-04-02T09:09:48.725 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 97 2024-04-02T09:09:49.145 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T09:09:42.220191+0000', 'modified': '2024-04-02T09:09:45.699081+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10638': {'gid': 10638, 'name': 'c', 'rank': 0, 'incarnation': 93, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6836/2257849268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6834', 'nonce': 2257849268}, {'type': 'v1', 'addr': '172.21.15.89:6836', 'nonce': 2257849268}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T09:09:49.145 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T09:09:49.145 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-02T09:09:49.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.342+0000 7f4229200700 1 Processor -- start 2024-04-02T09:09:49.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.342+0000 7f4229200700 1 -- start start 2024-04-02T09:09:49.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.343+0000 7f4229200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f422415dad0 0x7f42240ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.343+0000 7f4229200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f42240d1630 0x7f42240d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.343+0000 7f4229200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f42240d1fd0 0x7f422415deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.343+0000 7f4229200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42240cfcc0 con 0x7f42240d1630 2024-04-02T09:09:49.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.343+0000 7f4229200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42240cd9f0 con 0x7f42240d1fd0 2024-04-02T09:09:49.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.343+0000 7f4229200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422415f820 con 0x7f422415dad0 2024-04-02T09:09:49.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.344+0000 7f4221400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f42240d1630 0x7f42240d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.344+0000 7f4221400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f42240d1630 0x7f42240d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36552/0 (socket says 172.21.15.89:36552) 2024-04-02T09:09:49.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.344+0000 7f4221400700 1 -- 172.21.15.89:0/1434458735 learned_addr learned my addr 172.21.15.89:0/1434458735 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:49.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.344+0000 7f4221e00700 1 --2- 172.21.15.89:0/1434458735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f422415dad0 0x7f42240ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.344+0000 7f4222800700 1 --2- 172.21.15.89:0/1434458735 >> v2:172.21.15.113:3300/0 conn(0x7f42240d1fd0 0x7f422415deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.344+0000 7f4221400700 1 -- 172.21.15.89:0/1434458735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f422415dad0 msgr2=0x7f42240ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:49.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.344+0000 7f4221400700 1 --2- 172.21.15.89:0/1434458735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f422415dad0 0x7f42240ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.344+0000 7f4221400700 1 -- 172.21.15.89:0/1434458735 >> v2:172.21.15.113:3300/0 conn(0x7f42240d1fd0 msgr2=0x7f422415deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:49.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.344+0000 7f4221400700 1 --2- 172.21.15.89:0/1434458735 >> v2:172.21.15.113:3300/0 conn(0x7f42240d1fd0 0x7f422415deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.344+0000 7f4221400700 1 -- 172.21.15.89:0/1434458735 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42240d2460 con 0x7f42240d1630 2024-04-02T09:09:49.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.344+0000 7f4221400700 1 --2- 172.21.15.89:0/1434458735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d1630 0x7f42240d1a90 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f42240b95a0 tx=0x7f4210009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48fa6ed03f19c994 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.345+0000 7f4220a00700 1 -- 172.21.15.89:0/1434458735 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421000b870 con 0x7f42240d1630 2024-04-02T09:09:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.345+0000 7f4220a00700 1 -- 172.21.15.89:0/1434458735 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421000b9d0 con 0x7f42240d1630 2024-04-02T09:09:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.345+0000 7f4220a00700 1 -- 172.21.15.89:0/1434458735 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421000bd60 con 0x7f42240d1630 2024-04-02T09:09:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.345+0000 7f4229200700 1 -- 172.21.15.89:0/1434458735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d1630 msgr2=0x7f42240d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.345+0000 7f4229200700 1 --2- 172.21.15.89:0/1434458735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d1630 0x7f42240d1a90 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f42240b95a0 tx=0x7f4210009bd0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.345+0000 7f4229200700 1 -- 172.21.15.89:0/1434458735 shutdown_connections 2024-04-02T09:09:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.345+0000 7f4229200700 1 --2- 172.21.15.89:0/1434458735 >> v2:172.21.15.113:3300/0 conn(0x7f42240d1fd0 0x7f422415deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.345+0000 7f4229200700 1 --2- 172.21.15.89:0/1434458735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f422415dad0 0x7f42240ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.345+0000 7f4229200700 1 --2- 172.21.15.89:0/1434458735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d1630 0x7f42240d1a90 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.345+0000 7f4229200700 1 -- 172.21.15.89:0/1434458735 >> 172.21.15.89:0/1434458735 conn(0x7f422408aff0 msgr2=0x7f42240d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.345+0000 7f4229200700 1 -- 172.21.15.89:0/1434458735 shutdown_connections 2024-04-02T09:09:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.345+0000 7f4229200700 1 -- 172.21.15.89:0/1434458735 wait complete. 2024-04-02T09:09:49.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.347+0000 7f4229200700 1 Processor -- start 2024-04-02T09:09:49.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.348+0000 7f4229200700 1 -- start start 2024-04-02T09:09:49.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.349+0000 7f4229200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42240d1630 0x7f42240c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.349+0000 7f4229200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d1fd0 0x7f42240b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.349+0000 7f4229200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f422415dad0 0x7f42240b8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.349+0000 7f4229200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42240b8300 con 0x7f42240d1fd0 2024-04-02T09:09:49.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.349+0000 7f4229200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42240c8950 con 0x7f422415dad0 2024-04-02T09:09:49.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.349+0000 7f4229200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42240b8540 con 0x7f42240d1630 2024-04-02T09:09:49.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.349+0000 7f4221400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d1fd0 0x7f42240b7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.349+0000 7f4221e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42240d1630 0x7f42240c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.349+0000 7f4221e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42240d1630 0x7f42240c9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:48924/0 (socket says 172.21.15.89:48924) 2024-04-02T09:09:49.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.349+0000 7f4222800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f422415dad0 0x7f42240b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.349+0000 7f4221e00700 1 -- 172.21.15.89:0/694702903 learned_addr learned my addr 172.21.15.89:0/694702903 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:49.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.350+0000 7f4221400700 1 -- 172.21.15.89:0/694702903 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42240d1630 msgr2=0x7f42240c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.350+0000 7f4221400700 1 --2- 172.21.15.89:0/694702903 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42240d1630 0x7f42240c9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.350+0000 7f4221400700 1 -- 172.21.15.89:0/694702903 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f422415dad0 msgr2=0x7f42240b8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.350+0000 7f4221400700 1 --2- 172.21.15.89:0/694702903 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f422415dad0 0x7f42240b8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.350+0000 7f4221400700 1 -- 172.21.15.89:0/694702903 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4210009550 con 0x7f42240d1fd0 2024-04-02T09:09:49.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.350+0000 7f4221400700 1 --2- 172.21.15.89:0/694702903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d1fd0 0x7f42240b7dc0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f4210016000 tx=0x7f4210004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:49.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.350+0000 7f421a000700 1 -- 172.21.15.89:0/694702903 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4210004460 con 0x7f42240d1fd0 2024-04-02T09:09:49.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.350+0000 7f421a000700 1 -- 172.21.15.89:0/694702903 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42100045c0 con 0x7f42240d1fd0 2024-04-02T09:09:49.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.350+0000 7f4229200700 1 -- 172.21.15.89:0/694702903 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42240c7b90 con 0x7f42240d1fd0 2024-04-02T09:09:49.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.350+0000 7f421a000700 1 -- 172.21.15.89:0/694702903 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42100038c0 con 0x7f42240d1fd0 2024-04-02T09:09:49.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.350+0000 7f4229200700 1 -- 172.21.15.89:0/694702903 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42240c8150 con 0x7f42240d1fd0 2024-04-02T09:09:49.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.352+0000 7f421a000700 1 -- 172.21.15.89:0/694702903 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4210021070 con 0x7f42240d1fd0 2024-04-02T09:09:49.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.353+0000 7f421a000700 1 --2- 172.21.15.89:0/694702903 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f420406adb0 0x7f420406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.353+0000 7f421a000700 1 -- 172.21.15.89:0/694702903 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f421008abb0 con 0x7f42240d1fd0 2024-04-02T09:09:49.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.354+0000 7f4221e00700 1 --2- 172.21.15.89:0/694702903 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f420406adb0 0x7f420406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.354+0000 7f4221e00700 1 --2- 172.21.15.89:0/694702903 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f420406adb0 0x7f420406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4214007af0 tx=0x7f4214008040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:49.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.355+0000 7f423cdfdb80 1 Processor -- start 2024-04-02T09:09:49.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.355+0000 7f423cdfdb80 1 -- start start 2024-04-02T09:09:49.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.356+0000 7f423cdfdb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb756830 0x5604eb756c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.356+0000 7f423cdfdb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb729be0 0x5604eb72ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.356+0000 7f423cdfdb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb727b60 0x5604eb753a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.356+0000 7f423cdfdb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604eb6a1650 con 0x5604eb727b60 2024-04-02T09:09:49.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.357+0000 7f423cdfdb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604eb682c30 con 0x5604eb756830 2024-04-02T09:09:49.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.357+0000 7f423cdfdb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x5604eb680430 con 0x5604eb729be0 2024-04-02T09:09:49.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.357+0000 7f4222800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb727b60 0x5604eb753a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.357+0000 7f4222800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb727b60 0x5604eb753a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36572/0 (socket says 172.21.15.89:36572) 2024-04-02T09:09:49.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.357+0000 7f4222800700 1 -- 172.21.15.89:0/1818182550 learned_addr learned my addr 172.21.15.89:0/1818182550 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:49.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.357+0000 7f4221e00700 1 --2- 172.21.15.89:0/1818182550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb756830 0x5604eb756c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.357+0000 7f4221400700 1 --2- 172.21.15.89:0/1818182550 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb729be0 0x5604eb72ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.357+0000 7f4222800700 1 -- 172.21.15.89:0/1818182550 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb729be0 msgr2=0x5604eb72ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:49.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.357+0000 7f4222800700 1 --2- 172.21.15.89:0/1818182550 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb729be0 0x5604eb72ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.357+0000 7f4222800700 1 -- 172.21.15.89:0/1818182550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb756830 msgr2=0x5604eb756c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:49.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.357+0000 7f4222800700 1 --2- 172.21.15.89:0/1818182550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb756830 0x5604eb756c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.357+0000 7f4222800700 1 -- 172.21.15.89:0/1818182550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604eb46f790 con 0x5604eb727b60 2024-04-02T09:09:49.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.357+0000 7f4222800700 1 --2- 172.21.15.89:0/1818182550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb727b60 0x5604eb753a20 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f421c009f90 tx=0x7f421c003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef26214d2413ceb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:49.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.358+0000 7f420ea00700 1 -- 172.21.15.89:0/1818182550 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c003e90 con 0x5604eb727b60 2024-04-02T09:09:49.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.358+0000 7f420ea00700 1 -- 172.21.15.89:0/1818182550 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421c004740 con 0x5604eb727b60 2024-04-02T09:09:49.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.358+0000 7f420ea00700 1 -- 172.21.15.89:0/1818182550 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c015620 con 0x5604eb727b60 2024-04-02T09:09:49.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.358+0000 7f423cdfdb80 1 -- 172.21.15.89:0/1818182550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb727b60 msgr2=0x5604eb753a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.358+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/1818182550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb727b60 0x5604eb753a20 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f421c009f90 tx=0x7f421c003d00 comp rx=0 tx=0).stop 2024-04-02T09:09:49.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.358+0000 7f423cdfdb80 1 -- 172.21.15.89:0/1818182550 shutdown_connections 2024-04-02T09:09:49.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.358+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/1818182550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb756830 0x5604eb756c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.358+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/1818182550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb727b60 0x5604eb753a20 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.358+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/1818182550 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb729be0 0x5604eb72ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.358+0000 7f423cdfdb80 1 -- 172.21.15.89:0/1818182550 >> 172.21.15.89:0/1818182550 conn(0x5604eb68a6a0 msgr2=0x5604eb771640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:49.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.358+0000 7f423cdfdb80 1 -- 172.21.15.89:0/1818182550 shutdown_connections 2024-04-02T09:09:49.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.358+0000 7f423cdfdb80 1 -- 172.21.15.89:0/1818182550 wait complete. 2024-04-02T09:09:49.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.359+0000 7f423cdfdb80 1 -- start start 2024-04-02T09:09:49.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.359+0000 7f423cdfdb80 1 Processor -- start 2024-04-02T09:09:49.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.360+0000 7f423cdfdb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb727b60 0x5604eb7476d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.360+0000 7f423cdfdb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb729be0 0x5604eb754e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.360+0000 7f423cdfdb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb756830 0x5604eb751d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.360+0000 7f423cdfdb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604eb67bd20 con 0x5604eb727b60 2024-04-02T09:09:49.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.360+0000 7f423cdfdb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604eb617770 con 0x5604eb729be0 2024-04-02T09:09:49.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.360+0000 7f423cdfdb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x5604eb63a6f0 con 0x5604eb756830 2024-04-02T09:09:49.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.360+0000 7f423cdfdb80 10 client.0 _ll_register_callbacks cb 0x5604eb6215e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T09:09:49.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.360+0000 7f4221e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb727b60 0x5604eb7476d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.361+0000 7f4221e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb727b60 0x5604eb7476d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36588/0 (socket says 172.21.15.89:36588) 2024-04-02T09:09:49.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.361+0000 7f4221e00700 1 -- 172.21.15.89:0/1555015466 learned_addr learned my addr 172.21.15.89:0/1555015466 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:49.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.361+0000 7f4221400700 1 --2- 172.21.15.89:0/1555015466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb729be0 0x5604eb754e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.361+0000 7f4222800700 1 --2- 172.21.15.89:0/1555015466 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb756830 0x5604eb751d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.361+0000 7f4221e00700 1 -- 172.21.15.89:0/1555015466 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb756830 msgr2=0x5604eb751d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:49.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.361+0000 7f4221e00700 1 --2- 172.21.15.89:0/1555015466 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb756830 0x5604eb751d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.361+0000 7f4221e00700 1 -- 172.21.15.89:0/1555015466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb729be0 msgr2=0x5604eb754e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:49.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.361+0000 7f4221e00700 1 --2- 172.21.15.89:0/1555015466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb729be0 0x5604eb754e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.361+0000 7f4221e00700 1 -- 172.21.15.89:0/1555015466 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421c011040 con 0x5604eb727b60 2024-04-02T09:09:49.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.361+0000 7f4221e00700 1 --2- 172.21.15.89:0/1555015466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb727b60 0x5604eb7476d0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f4214016830 tx=0x7f4214004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bfc0a7970388829 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:49.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.361+0000 7f4203400700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T09:09:49.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f4203400700 1 -- 172.21.15.89:0/1555015466 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4214014700 con 0x5604eb727b60 2024-04-02T09:09:49.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f4203400700 1 -- 172.21.15.89:0/1555015466 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f4214014860 con 0x5604eb727b60 2024-04-02T09:09:49.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f423cdfdb80 1 -- 172.21.15.89:0/1555015466 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5604eb71e920 con 0x5604eb727b60 2024-04-02T09:09:49.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f4203400700 1 -- 172.21.15.89:0/1555015466 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4214014a20 con 0x5604eb727b60 2024-04-02T09:09:49.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f423cdfdb80 10 client.10773 fetch_fsmap learned FSMap version 97 2024-04-02T09:09:49.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f423cdfdb80 1 -- 172.21.15.89:0/1555015466 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) v3 -- 0x5604eb55c0a0 con 0x5604eb727b60 2024-04-02T09:09:49.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f4203400700 1 -- 172.21.15.89:0/1555015466 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5604eb71e920 con 0x5604eb727b60 2024-04-02T09:09:49.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f4203400700 1 -- 172.21.15.89:0/1555015466 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 97) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4214014b80 con 0x5604eb727b60 2024-04-02T09:09:49.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f423cdfdb80 10 client.10773 fetch_fsmap finished waiting for FSMap version 97 2024-04-02T09:09:49.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f423cdfdb80 10 client.10773 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T09:09:49.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f423cdfdb80 20 client.10773 populate_metadata read hostname 'smithi089' 2024-04-02T09:09:49.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/1555015466 >> [v2:172.21.15.89:6834/2257849268,v1:172.21.15.89:6836/2257849268] conn(0x5604eb74f020 0x5604eb78e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f423cdfdb80 4 client.10773 mds_command: new command op to 10638 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T09:09:49.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f423cdfdb80 1 -- 172.21.15.89:0/1555015466 --> [v2:172.21.15.89:6834/2257849268,v1:172.21.15.89:6836/2257849268] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5604eb6fb6f0 con 0x5604eb74f020 2024-04-02T09:09:49.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.362+0000 7f4221400700 1 --2- 172.21.15.89:0/1555015466 >> [v2:172.21.15.89:6834/2257849268,v1:172.21.15.89:6836/2257849268] conn(0x5604eb74f020 0x5604eb78e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.363+0000 7f4221400700 1 --2- 172.21.15.89:0/1555015466 >> [v2:172.21.15.89:6834/2257849268,v1:172.21.15.89:6836/2257849268] conn(0x5604eb74f020 0x5604eb78e020 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e615f0d2bc97127e server_cookie=415861031d70f4 in_seq=0 out_seq=0 2024-04-02T09:09:49.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.363+0000 7f4203400700 10 client.10773 ms_handle_connect on v2:172.21.15.89:6834/2257849268 2024-04-02T09:09:49.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f4203400700 1 -- 172.21.15.89:0/1555015466 <== mds.0 v2:172.21.15.89:6834/2257849268 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5604eb6fb6f0 con 0x5604eb74f020 2024-04-02T09:09:49.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f4203400700 10 client.10773 handle_command_reply: tid=0 2024-04-02T09:09:49.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f423cdfdb80 1 client.10773 shutdown 2024-04-02T09:09:49.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f423cdfdb80 1 -- 172.21.15.89:0/1555015466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb727b60 msgr2=0x5604eb7476d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/1555015466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb727b60 0x5604eb7476d0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f4214016830 tx=0x7f4214004750 comp rx=0 tx=0).stop 2024-04-02T09:09:49.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f423cdfdb80 1 -- 172.21.15.89:0/1555015466 shutdown_connections 2024-04-02T09:09:49.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/1555015466 >> [v2:172.21.15.89:6834/2257849268,v1:172.21.15.89:6836/2257849268] conn(0x5604eb74f020 0x5604eb78e020 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/1555015466 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb756830 0x5604eb751d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/1555015466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb727b60 0x5604eb7476d0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f4203400700 0 client.10773 ms_handle_reset on v2:172.21.15.89:6834/2257849268 2024-04-02T09:09:49.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/1555015466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb729be0 0x5604eb754e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f423cdfdb80 1 -- 172.21.15.89:0/1555015466 >> 172.21.15.89:0/1555015466 conn(0x5604eb68a6a0 msgr2=0x5604eb72a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:49.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f423cdfdb80 1 -- 172.21.15.89:0/1555015466 shutdown_connections 2024-04-02T09:09:49.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f423cdfdb80 1 -- 172.21.15.89:0/1555015466 wait complete. 2024-04-02T09:09:49.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.364+0000 7f423cdfdb80 20 client.10773 trim_cache size 0 max 16384 2024-04-02T09:09:49.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.382+0000 7f423cdfdb80 1 Processor -- start 2024-04-02T09:09:49.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.382+0000 7f423cdfdb80 1 -- start start 2024-04-02T09:09:49.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.383+0000 7f423cdfdb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb727b60 0x5604eb74c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.383+0000 7f423cdfdb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb729be0 0x5604eb792040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.383+0000 7f423cdfdb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb756830 0x5604eb792580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.383+0000 7f423cdfdb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604eb470390 con 0x5604eb729be0 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.383+0000 7f423cdfdb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604eb72d810 con 0x5604eb727b60 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.383+0000 7f423cdfdb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x5604eb67dc00 con 0x5604eb756830 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.383+0000 7f4221400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb729be0 0x5604eb792040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.383+0000 7f4221400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb729be0 0x5604eb792040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36596/0 (socket says 172.21.15.89:36596) 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.383+0000 7f4221400700 1 -- 172.21.15.89:0/3541493703 learned_addr learned my addr 172.21.15.89:0/3541493703 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.384+0000 7f4221e00700 1 --2- 172.21.15.89:0/3541493703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb727b60 0x5604eb74c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.384+0000 7f4222800700 1 --2- 172.21.15.89:0/3541493703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb756830 0x5604eb792580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.384+0000 7f4221400700 1 -- 172.21.15.89:0/3541493703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb756830 msgr2=0x5604eb792580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.384+0000 7f4221400700 1 --2- 172.21.15.89:0/3541493703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb756830 0x5604eb792580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.384+0000 7f4221400700 1 -- 172.21.15.89:0/3541493703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb727b60 msgr2=0x5604eb74c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.384+0000 7f4221400700 1 --2- 172.21.15.89:0/3541493703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb727b60 0x5604eb74c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.384+0000 7f4221400700 1 -- 172.21.15.89:0/3541493703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604eb642010 con 0x5604eb729be0 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.384+0000 7f4222800700 1 --2- 172.21.15.89:0/3541493703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb756830 0x5604eb792580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:49.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.384+0000 7f4221e00700 1 --2- 172.21.15.89:0/3541493703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb727b60 0x5604eb74c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:09:49.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.384+0000 7f4221400700 1 --2- 172.21.15.89:0/3541493703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb729be0 0x5604eb792040 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f4210009fd0 tx=0x7f421006f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca70153ddf71c92 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:49.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.384+0000 7f420f400700 1 -- 172.21.15.89:0/3541493703 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4210061c80 con 0x5604eb729be0 2024-04-02T09:09:49.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.384+0000 7f420f400700 1 -- 172.21.15.89:0/3541493703 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4210057c30 con 0x5604eb729be0 2024-04-02T09:09:49.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.384+0000 7f420f400700 1 -- 172.21.15.89:0/3541493703 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4210059af0 con 0x5604eb729be0 2024-04-02T09:09:49.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.385+0000 7f423cdfdb80 1 -- 172.21.15.89:0/3541493703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb729be0 msgr2=0x5604eb792040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.385+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/3541493703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb729be0 0x5604eb792040 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f4210009fd0 tx=0x7f421006f040 comp rx=0 tx=0).stop 2024-04-02T09:09:49.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.385+0000 7f423cdfdb80 1 -- 172.21.15.89:0/3541493703 shutdown_connections 2024-04-02T09:09:49.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.385+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/3541493703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb727b60 0x5604eb74c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.385+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/3541493703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb756830 0x5604eb792580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.385+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/3541493703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb729be0 0x5604eb792040 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.385+0000 7f423cdfdb80 1 -- 172.21.15.89:0/3541493703 >> 172.21.15.89:0/3541493703 conn(0x5604eb68a6a0 msgr2=0x5604eb726680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:49.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.385+0000 7f423cdfdb80 1 -- 172.21.15.89:0/3541493703 shutdown_connections 2024-04-02T09:09:49.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.385+0000 7f423cdfdb80 1 -- 172.21.15.89:0/3541493703 wait complete. 2024-04-02T09:09:49.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.386+0000 7f423cdfdb80 1 -- start start 2024-04-02T09:09:49.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.386+0000 7f423cdfdb80 1 Processor -- start 2024-04-02T09:09:49.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f423cdfdb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb727b60 0x5604eb7474e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f423cdfdb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb729be0 0x5604eb747a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f423cdfdb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb756830 0x5604eb766fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f423cdfdb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604eb767f60 con 0x5604eb756830 2024-04-02T09:09:49.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f423cdfdb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604eb7d91b0 con 0x5604eb729be0 2024-04-02T09:09:49.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f423cdfdb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x5604eb7d88a0 con 0x5604eb727b60 2024-04-02T09:09:49.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f423cdfdb80 10 client.0 _ll_register_callbacks cb 0x7f4214010cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T09:09:49.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f4222800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb756830 0x5604eb766fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f4222800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb756830 0x5604eb766fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36598/0 (socket says 172.21.15.89:36598) 2024-04-02T09:09:49.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f4222800700 1 -- 172.21.15.89:0/2090284097 learned_addr learned my addr 172.21.15.89:0/2090284097 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:49.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f4222800700 1 -- 172.21.15.89:0/2090284097 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb727b60 msgr2=0x5604eb7474e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:49.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f4222800700 1 --2- 172.21.15.89:0/2090284097 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb727b60 0x5604eb7474e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f4222800700 1 -- 172.21.15.89:0/2090284097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb729be0 msgr2=0x5604eb747a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:49.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f4222800700 1 --2- 172.21.15.89:0/2090284097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb729be0 0x5604eb747a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f4222800700 1 -- 172.21.15.89:0/2090284097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42240c8150 con 0x5604eb756830 2024-04-02T09:09:49.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f4222800700 1 --2- 172.21.15.89:0/2090284097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb756830 0x5604eb766fd0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f421c009f30 tx=0x7f421c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ba9562626204eb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:49.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.388+0000 7f4203400700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T09:09:49.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f4203400700 1 -- 172.21.15.89:0/2090284097 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c007040 con 0x5604eb756830 2024-04-02T09:09:49.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f4203400700 1 -- 172.21.15.89:0/2090284097 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f421c0217e0 con 0x5604eb756830 2024-04-02T09:09:49.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f423cdfdb80 1 -- 172.21.15.89:0/2090284097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4224159780 con 0x5604eb756830 2024-04-02T09:09:49.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f4203400700 1 -- 172.21.15.89:0/2090284097 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) v2 ==== 24+0+0 (secure 0 0 0) 0x7f421c0219c0 con 0x5604eb756830 2024-04-02T09:09:49.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f423cdfdb80 10 client.10779 fetch_fsmap learned FSMap version 97 2024-04-02T09:09:49.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f423cdfdb80 1 -- 172.21.15.89:0/2090284097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) v3 -- 0x5604eb4d01d0 con 0x5604eb756830 2024-04-02T09:09:49.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f4203400700 1 -- 172.21.15.89:0/2090284097 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4224159780 con 0x5604eb756830 2024-04-02T09:09:49.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f4203400700 1 -- 172.21.15.89:0/2090284097 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 97) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f421c021b20 con 0x5604eb756830 2024-04-02T09:09:49.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f423cdfdb80 10 client.10779 fetch_fsmap finished waiting for FSMap version 97 2024-04-02T09:09:49.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f423cdfdb80 10 client.10779 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T09:09:49.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f423cdfdb80 20 client.10779 populate_metadata read hostname 'smithi089' 2024-04-02T09:09:49.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/2090284097 >> [v2:172.21.15.89:6834/2257849268,v1:172.21.15.89:6836/2257849268] conn(0x5604eb767510 0x5604eb771440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f423cdfdb80 4 client.10779 mds_command: new command op to 10638 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T09:09:49.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f423cdfdb80 1 -- 172.21.15.89:0/2090284097 --> [v2:172.21.15.89:6834/2257849268,v1:172.21.15.89:6836/2257849268] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f421c004740 con 0x5604eb767510 2024-04-02T09:09:49.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.389+0000 7f4221e00700 1 --2- 172.21.15.89:0/2090284097 >> [v2:172.21.15.89:6834/2257849268,v1:172.21.15.89:6836/2257849268] conn(0x5604eb767510 0x5604eb771440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.390+0000 7f4221e00700 1 --2- 172.21.15.89:0/2090284097 >> [v2:172.21.15.89:6834/2257849268,v1:172.21.15.89:6836/2257849268] conn(0x5604eb767510 0x5604eb771440 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7626d78807bab57 server_cookie=1ca94cb26b2fe226 in_seq=0 out_seq=0 2024-04-02T09:09:49.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.390+0000 7f4203400700 10 client.10779 ms_handle_connect on v2:172.21.15.89:6834/2257849268 2024-04-02T09:09:49.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.390+0000 7f4203400700 1 -- 172.21.15.89:0/2090284097 <== mds.0 v2:172.21.15.89:6834/2257849268 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x7f421c004740 con 0x5604eb767510 2024-04-02T09:09:49.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.390+0000 7f4203400700 10 client.10779 handle_command_reply: tid=0 2024-04-02T09:09:49.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.390+0000 7f423cdfdb80 1 client.10779 shutdown 2024-04-02T09:09:49.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.390+0000 7f423cdfdb80 1 -- 172.21.15.89:0/2090284097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb756830 msgr2=0x5604eb766fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.390+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/2090284097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb756830 0x5604eb766fd0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f421c009f30 tx=0x7f421c004b40 comp rx=0 tx=0).stop 2024-04-02T09:09:49.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.390+0000 7f423cdfdb80 1 -- 172.21.15.89:0/2090284097 shutdown_connections 2024-04-02T09:09:49.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.390+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/2090284097 >> [v2:172.21.15.89:6834/2257849268,v1:172.21.15.89:6836/2257849268] conn(0x5604eb767510 0x5604eb771440 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.390+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/2090284097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604eb756830 0x5604eb766fd0 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.390+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/2090284097 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5604eb727b60 0x5604eb7474e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.390+0000 7f423cdfdb80 1 --2- 172.21.15.89:0/2090284097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604eb729be0 0x5604eb747a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.390+0000 7f4203400700 0 client.10779 ms_handle_reset on v2:172.21.15.89:6834/2257849268 2024-04-02T09:09:49.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.390+0000 7f423cdfdb80 1 -- 172.21.15.89:0/2090284097 >> 172.21.15.89:0/2090284097 conn(0x5604eb68a6a0 msgr2=0x5604eb728960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:49.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.391+0000 7f423cdfdb80 1 -- 172.21.15.89:0/2090284097 shutdown_connections 2024-04-02T09:09:49.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.391+0000 7f423cdfdb80 1 -- 172.21.15.89:0/2090284097 wait complete. 2024-04-02T09:09:49.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.391+0000 7f423cdfdb80 20 client.10779 trim_cache size 0 max 16384 2024-04-02T09:09:49.394 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:09:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T09:09:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T09:09:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 10638, 2024-04-02T09:09:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T09:09:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T09:09:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T09:09:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 7.1691419280000002, 2024-04-02T09:09:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 97, 2024-04-02T09:09:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 205, 2024-04-02T09:09:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 205, 2024-04-02T09:09:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 12.612343596000001 2024-04-02T09:09:49.395 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:09:49.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.391+0000 7f4229200700 1 -- 172.21.15.89:0/694702903 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f420406adb0 msgr2=0x7f420406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.391+0000 7f4229200700 1 --2- 172.21.15.89:0/694702903 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f420406adb0 0x7f420406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4214007af0 tx=0x7f4214008040 comp rx=0 tx=0).stop 2024-04-02T09:09:49.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.391+0000 7f4229200700 1 -- 172.21.15.89:0/694702903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d1fd0 msgr2=0x7f42240b7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.391+0000 7f4229200700 1 --2- 172.21.15.89:0/694702903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d1fd0 0x7f42240b7dc0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f4210016000 tx=0x7f4210004750 comp rx=0 tx=0).stop 2024-04-02T09:09:49.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.391+0000 7f4229200700 1 -- 172.21.15.89:0/694702903 shutdown_connections 2024-04-02T09:09:49.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.391+0000 7f4229200700 1 --2- 172.21.15.89:0/694702903 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f420406adb0 0x7f420406d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.391+0000 7f4229200700 1 --2- 172.21.15.89:0/694702903 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f422415dad0 0x7f42240b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.391+0000 7f4229200700 1 --2- 172.21.15.89:0/694702903 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42240d1630 0x7f42240c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.391+0000 7f4229200700 1 --2- 172.21.15.89:0/694702903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42240d1fd0 0x7f42240b7dc0 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.391+0000 7f4229200700 1 -- 172.21.15.89:0/694702903 >> 172.21.15.89:0/694702903 conn(0x7f422408aff0 msgr2=0x7f422406b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:49.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.392+0000 7f4229200700 1 -- 172.21.15.89:0/694702903 shutdown_connections 2024-04-02T09:09:49.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.392+0000 7f4229200700 1 -- 172.21.15.89:0/694702903 wait complete. 2024-04-02T09:09:49.408 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T09:09:49.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.600+0000 7ffb39400700 1 Processor -- start 2024-04-02T09:09:49.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.600+0000 7ffb39400700 1 -- start start 2024-04-02T09:09:49.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.601+0000 7ffb39400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb340b3af0 0x7ffb3415a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.601+0000 7ffb39400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb340709c0 0x7ffb34070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.601+0000 7ffb39400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb340b2180 con 0x7ffb34072580 2024-04-02T09:09:49.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.601+0000 7ffb39400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb3415b100 con 0x7ffb340b3af0 2024-04-02T09:09:49.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.601+0000 7ffb39400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3415b3e0 con 0x7ffb340709c0 2024-04-02T09:09:49.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.601+0000 7ffb32800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb340b3af0 0x7ffb3415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.601+0000 7ffb32800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb340b3af0 0x7ffb3415a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36614/0 (socket says 172.21.15.89:36614) 2024-04-02T09:09:49.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.601+0000 7ffb32800700 1 -- 172.21.15.89:0/4112761650 learned_addr learned my addr 172.21.15.89:0/4112761650 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:49.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.601+0000 7ffb33200700 1 --2- 172.21.15.89:0/4112761650 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb340709c0 0x7ffb34070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.602+0000 7ffb32800700 1 -- 172.21.15.89:0/4112761650 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb340709c0 msgr2=0x7ffb34070e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:49.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.602+0000 7ffb32800700 1 --2- 172.21.15.89:0/4112761650 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb340709c0 0x7ffb34070e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.602+0000 7ffb32800700 1 -- 172.21.15.89:0/4112761650 >> v1:172.21.15.113:6789/0 conn(0x7ffb34072580 legacy=0x7ffb340b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.602+0000 7ffb32800700 1 -- 172.21.15.89:0/4112761650 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb340b23f0 con 0x7ffb340b3af0 2024-04-02T09:09:49.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.602+0000 7ffb32800700 1 --2- 172.21.15.89:0/4112761650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb340b3af0 0x7ffb3415a860 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7ffb240098f0 tx=0x7ffb24009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cade37e8dadbc4ef server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:49.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.602+0000 7ffb31400700 1 -- 172.21.15.89:0/4112761650 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2400b8a0 con 0x7ffb340b3af0 2024-04-02T09:09:49.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.602+0000 7ffb31400700 1 -- 172.21.15.89:0/4112761650 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2400ba00 con 0x7ffb340b3af0 2024-04-02T09:09:49.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.602+0000 7ffb31400700 1 -- 172.21.15.89:0/4112761650 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2400bd90 con 0x7ffb340b3af0 2024-04-02T09:09:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.603+0000 7ffb39400700 1 -- 172.21.15.89:0/4112761650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb340b3af0 msgr2=0x7ffb3415a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.603+0000 7ffb39400700 1 --2- 172.21.15.89:0/4112761650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb340b3af0 0x7ffb3415a860 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7ffb240098f0 tx=0x7ffb24009c00 comp rx=0 tx=0).stop 2024-04-02T09:09:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.603+0000 7ffb39400700 1 -- 172.21.15.89:0/4112761650 shutdown_connections 2024-04-02T09:09:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.603+0000 7ffb39400700 1 --2- 172.21.15.89:0/4112761650 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb340709c0 0x7ffb34070e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.603+0000 7ffb39400700 1 --2- 172.21.15.89:0/4112761650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb340b3af0 0x7ffb3415a860 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.603+0000 7ffb39400700 1 -- 172.21.15.89:0/4112761650 >> 172.21.15.89:0/4112761650 conn(0x7ffb340cf190 msgr2=0x7ffb340cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.603+0000 7ffb39400700 1 -- 172.21.15.89:0/4112761650 shutdown_connections 2024-04-02T09:09:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.603+0000 7ffb39400700 1 -- 172.21.15.89:0/4112761650 wait complete. 2024-04-02T09:09:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.605+0000 7ffb39400700 1 Processor -- start 2024-04-02T09:09:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.605+0000 7ffb39400700 1 -- start start 2024-04-02T09:09:49.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb39400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb340709c0 0x7ffb34154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb39400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb34072580 0x7ffb34150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb39400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb340b3af0 0x7ffb341529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb39400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb34151d40 con 0x7ffb34072580 2024-04-02T09:09:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb39400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb34152230 con 0x7ffb340709c0 2024-04-02T09:09:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb39400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb341553b0 con 0x7ffb340b3af0 2024-04-02T09:09:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb31e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb34072580 0x7ffb34150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb31e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb34072580 0x7ffb34150db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36628/0 (socket says 172.21.15.89:36628) 2024-04-02T09:09:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb31e00700 1 -- 172.21.15.89:0/1320238378 learned_addr learned my addr 172.21.15.89:0/1320238378 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb32800700 1 --2- 172.21.15.89:0/1320238378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb340709c0 0x7ffb34154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb31e00700 1 -- 172.21.15.89:0/1320238378 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb340b3af0 msgr2=0x7ffb341529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb31e00700 1 --2- 172.21.15.89:0/1320238378 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb340b3af0 0x7ffb341529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb31e00700 1 -- 172.21.15.89:0/1320238378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb340709c0 msgr2=0x7ffb34154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb31e00700 1 --2- 172.21.15.89:0/1320238378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb340709c0 0x7ffb34154e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb31e00700 1 -- 172.21.15.89:0/1320238378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb24009550 con 0x7ffb34072580 2024-04-02T09:09:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.606+0000 7ffb31e00700 1 --2- 172.21.15.89:0/1320238378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb34072580 0x7ffb34150db0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000c4e0 tx=0x7ffb2000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:49.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.607+0000 7ffb2aa00700 1 -- 172.21.15.89:0/1320238378 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb200052d0 con 0x7ffb34072580 2024-04-02T09:09:49.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.607+0000 7ffb2aa00700 1 -- 172.21.15.89:0/1320238378 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb20005430 con 0x7ffb34072580 2024-04-02T09:09:49.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.607+0000 7ffb2aa00700 1 -- 172.21.15.89:0/1320238378 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb200038c0 con 0x7ffb34072580 2024-04-02T09:09:49.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.607+0000 7ffb39400700 1 -- 172.21.15.89:0/1320238378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb341512f0 con 0x7ffb34072580 2024-04-02T09:09:49.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.607+0000 7ffb39400700 1 -- 172.21.15.89:0/1320238378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb341502d0 con 0x7ffb34072580 2024-04-02T09:09:49.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.608+0000 7ffb39400700 1 -- 172.21.15.89:0/1320238378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb34067720 con 0x7ffb34072580 2024-04-02T09:09:49.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.609+0000 7ffb2aa00700 1 -- 172.21.15.89:0/1320238378 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ffb20003a20 con 0x7ffb34072580 2024-04-02T09:09:49.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.610+0000 7ffb2aa00700 1 --2- 172.21.15.89:0/1320238378 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffb1406adb0 0x7ffb1406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:49.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.610+0000 7ffb2aa00700 1 -- 172.21.15.89:0/1320238378 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7ffb20088a80 con 0x7ffb34072580 2024-04-02T09:09:49.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.610+0000 7ffb32800700 1 --2- 172.21.15.89:0/1320238378 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffb1406adb0 0x7ffb1406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:49.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.610+0000 7ffb32800700 1 --2- 172.21.15.89:0/1320238378 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffb1406adb0 0x7ffb1406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffb24009c40 tx=0x7ffb24008030 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:49.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.614+0000 7ffb2aa00700 1 -- 172.21.15.89:0/1320238378 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffb200538f0 con 0x7ffb34072580 2024-04-02T09:09:49.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.866+0000 7ffb39400700 1 -- 172.21.15.89:0/1320238378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ffb340cef70 con 0x7ffb34072580 2024-04-02T09:09:49.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.866+0000 7ffb2aa00700 1 -- 172.21.15.89:0/1320238378 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v97) v1 ==== 53+0+73 (secure 0 0 0) 0x7ffb20017070 con 0x7ffb34072580 2024-04-02T09:09:49.866 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T09:09:49.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.869+0000 7ffb39400700 1 -- 172.21.15.89:0/1320238378 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffb1406adb0 msgr2=0x7ffb1406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.869+0000 7ffb39400700 1 --2- 172.21.15.89:0/1320238378 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffb1406adb0 0x7ffb1406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffb24009c40 tx=0x7ffb24008030 comp rx=0 tx=0).stop 2024-04-02T09:09:49.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.869+0000 7ffb39400700 1 -- 172.21.15.89:0/1320238378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb34072580 msgr2=0x7ffb34150db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:49.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.869+0000 7ffb39400700 1 --2- 172.21.15.89:0/1320238378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb34072580 0x7ffb34150db0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000c4e0 tx=0x7ffb2000c8a0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.869+0000 7ffb39400700 1 -- 172.21.15.89:0/1320238378 shutdown_connections 2024-04-02T09:09:49.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.869+0000 7ffb39400700 1 --2- 172.21.15.89:0/1320238378 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffb1406adb0 0x7ffb1406d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.869+0000 7ffb39400700 1 --2- 172.21.15.89:0/1320238378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb340709c0 0x7ffb34154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.869+0000 7ffb39400700 1 --2- 172.21.15.89:0/1320238378 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb340b3af0 0x7ffb341529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.869+0000 7ffb39400700 1 --2- 172.21.15.89:0/1320238378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb34072580 0x7ffb34150db0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:49.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.869+0000 7ffb39400700 1 -- 172.21.15.89:0/1320238378 >> 172.21.15.89:0/1320238378 conn(0x7ffb340cf190 msgr2=0x7ffb34071420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:49.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.869+0000 7ffb39400700 1 -- 172.21.15.89:0/1320238378 shutdown_connections 2024-04-02T09:09:49.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:49.869+0000 7ffb39400700 1 -- 172.21.15.89:0/1320238378 wait complete. 2024-04-02T09:09:49.886 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T09:09:49.886 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T09:09:49.886 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T09:09:49.886 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T09:09:49.886 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T09:09:49.886 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T09:09:49.886 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T09:09:49.886 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T09:09:49.887 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:09:49.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:49.887 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T09:09:49.932 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T09:09:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T09:09:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T09:09:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:09:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T09:09:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:09:49.932 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:09:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:09:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T09:09:49.932 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:09:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:09:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 36839sec preferred_lft 36839sec 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:09:49.933 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T09:09:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T09:09:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T09:09:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:09:49.934 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T09:09:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T09:09:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T09:09:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:09:49.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:49.935 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T09:09:50.039 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T09:09:50.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:50.042 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T09:09:50.090 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T09:09:50.090 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T09:09:50.090 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T09:09:50.090 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T09:09:50.090 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T09:09:50.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:50.091 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T09:09:50.092 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T09:09:50.092 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T09:09:50.092 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T09:09:50.092 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T09:09:50.092 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T09:09:50.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:50.255 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T09:09:50.300 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T09:09:50.300 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T09:09:50.300 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:09:50.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:50.301 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T09:09:50.408 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T09:09:50.408 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T09:09:50.452 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:09:50.453 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T09:09:50.453 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T09:09:50.537 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T09:09:50.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:50.537 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:09:50.679 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:09:50.679 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T09:09:50.706 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:09:50.706 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T09:09:50.706 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T09:09:50.706 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T09:09:50.706 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T09:09:50.711 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T09:09:50.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:50.711 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T09:09:50.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:50.778 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T09:09:50.878 INFO:teuthology.orchestra.run.smithi089.stdout:client9952 2024-04-02T09:09:50.879 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:09:50.879 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:09:50.924 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T09:09:51.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.158+0000 7f0693800700 1 Processor -- start 2024-04-02T09:09:51.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.158+0000 7f0693800700 1 -- start start 2024-04-02T09:09:51.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.159+0000 7f0693800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f068c0d2540 0x7f068c1687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:51.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.159+0000 7f0693800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068c161420 con 0x7f068c15d9d0 2024-04-02T09:09:51.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.159+0000 7f0693800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068c161700 con 0x7f068c0d16a0 2024-04-02T09:09:51.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.159+0000 7f0693800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f068c0d4340 con 0x7f068c0d2540 2024-04-02T09:09:51.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.159+0000 7f0691800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f068c0d2540 0x7f068c1687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:51.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.159+0000 7f0690e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f068c15d9d0 0x7f068c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34358/0 (socket says 172.21.15.89:34358) 2024-04-02T09:09:51.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.159+0000 7f0690e00700 1 -- 172.21.15.89:0/757456469 learned_addr learned my addr 172.21.15.89:0/757456469 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:51.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.160+0000 7f0691800700 1 -- 172.21.15.89:0/757456469 >> v1:172.21.15.113:6789/0 conn(0x7f068c0d16a0 legacy=0x7f068c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:51.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.160+0000 7f0691800700 1 -- 172.21.15.89:0/757456469 >> v1:172.21.15.89:6789/0 conn(0x7f068c15d9d0 legacy=0x7f068c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:51.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.160+0000 7f0691800700 1 -- 172.21.15.89:0/757456469 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068c1619e0 con 0x7f068c0d2540 2024-04-02T09:09:51.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.160+0000 7f0691800700 1 --2- 172.21.15.89:0/757456469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068c0d2540 0x7f068c1687b0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f0684009d90 tx=0x7f068400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f79ad25bce3478 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:51.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.160+0000 7f068b400700 1 -- 172.21.15.89:0/757456469 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068400e070 con 0x7f068c0d2540 2024-04-02T09:09:51.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.160+0000 7f068b400700 1 -- 172.21.15.89:0/757456469 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068400bbd0 con 0x7f068c0d2540 2024-04-02T09:09:51.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.160+0000 7f068b400700 1 -- 172.21.15.89:0/757456469 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684012430 con 0x7f068c0d2540 2024-04-02T09:09:51.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.161+0000 7f0693800700 1 -- 172.21.15.89:0/757456469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068c0d2540 msgr2=0x7f068c1687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:51.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.161+0000 7f0693800700 1 --2- 172.21.15.89:0/757456469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068c0d2540 0x7f068c1687b0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f0684009d90 tx=0x7f068400b330 comp rx=0 tx=0).stop 2024-04-02T09:09:51.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.161+0000 7f0693800700 1 -- 172.21.15.89:0/757456469 shutdown_connections 2024-04-02T09:09:51.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.161+0000 7f0693800700 1 --2- 172.21.15.89:0/757456469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068c0d2540 0x7f068c1687b0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:51.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.161+0000 7f0693800700 1 -- 172.21.15.89:0/757456469 >> 172.21.15.89:0/757456469 conn(0x7f068c08aff0 msgr2=0x7f068c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:51.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.161+0000 7f0693800700 1 -- 172.21.15.89:0/757456469 shutdown_connections 2024-04-02T09:09:51.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.161+0000 7f0693800700 1 -- 172.21.15.89:0/757456469 wait complete. 2024-04-02T09:09:51.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.163+0000 7f0693800700 1 Processor -- start 2024-04-02T09:09:51.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.163+0000 7f0693800700 1 -- start start 2024-04-02T09:09:51.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0693800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f068c0d16a0 0x7f068c151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:51.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0693800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f068c0d2540 0x7f068c0b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:51.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0693800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068c15d9d0 0x7f068c0cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:51.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0693800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068c0b6870 con 0x7f068c15d9d0 2024-04-02T09:09:51.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0693800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068c0b6b50 con 0x7f068c0d2540 2024-04-02T09:09:51.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0693800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068c0b6e30 con 0x7f068c0d16a0 2024-04-02T09:09:51.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0691800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068c15d9d0 0x7f068c0cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:51.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0691800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068c15d9d0 0x7f068c0cafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36646/0 (socket says 172.21.15.89:36646) 2024-04-02T09:09:51.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0691800700 1 -- 172.21.15.89:0/1661777112 learned_addr learned my addr 172.21.15.89:0/1661777112 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:51.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0690e00700 1 --2- 172.21.15.89:0/1661777112 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f068c0d16a0 0x7f068c151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:51.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f068be00700 1 --2- 172.21.15.89:0/1661777112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f068c0d2540 0x7f068c0b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:51.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0691800700 1 -- 172.21.15.89:0/1661777112 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f068c0d16a0 msgr2=0x7f068c151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:51.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0691800700 1 --2- 172.21.15.89:0/1661777112 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f068c0d16a0 0x7f068c151b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:51.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0691800700 1 -- 172.21.15.89:0/1661777112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f068c0d2540 msgr2=0x7f068c0b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:51.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0691800700 1 --2- 172.21.15.89:0/1661777112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f068c0d2540 0x7f068c0b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:51.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.164+0000 7f0691800700 1 -- 172.21.15.89:0/1661777112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0684009a70 con 0x7f068c15d9d0 2024-04-02T09:09:51.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.165+0000 7f0691800700 1 --2- 172.21.15.89:0/1661777112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068c15d9d0 0x7f068c0cafb0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f0684014040 tx=0x7f0684004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:51.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.165+0000 7f0688c00700 1 -- 172.21.15.89:0/1661777112 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06840098f0 con 0x7f068c15d9d0 2024-04-02T09:09:51.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.165+0000 7f0688c00700 1 -- 172.21.15.89:0/1661777112 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0684020070 con 0x7f068c15d9d0 2024-04-02T09:09:51.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.165+0000 7f0693800700 1 -- 172.21.15.89:0/1661777112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f068c0b7110 con 0x7f068c15d9d0 2024-04-02T09:09:51.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.165+0000 7f0688c00700 1 -- 172.21.15.89:0/1661777112 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684004280 con 0x7f068c15d9d0 2024-04-02T09:09:51.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.165+0000 7f0693800700 1 -- 172.21.15.89:0/1661777112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f068c0b6560 con 0x7f068c15d9d0 2024-04-02T09:09:51.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.166+0000 7f0693800700 1 -- 172.21.15.89:0/1661777112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068c0d1ca0 con 0x7f068c15d9d0 2024-04-02T09:09:51.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.167+0000 7f0688c00700 1 -- 172.21.15.89:0/1661777112 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f06840043e0 con 0x7f068c15d9d0 2024-04-02T09:09:51.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.168+0000 7f0688c00700 1 --2- 172.21.15.89:0/1661777112 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f066c06adb0 0x7f066c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:51.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.168+0000 7f0688c00700 1 -- 172.21.15.89:0/1661777112 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f0684022070 con 0x7f068c15d9d0 2024-04-02T09:09:51.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.169+0000 7f0690e00700 1 --2- 172.21.15.89:0/1661777112 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f066c06adb0 0x7f066c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:51.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.169+0000 7f0690e00700 1 --2- 172.21.15.89:0/1661777112 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f066c06adb0 0x7f066c06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0680005910 tx=0x7f0680009040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:51.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.173+0000 7f0688c00700 1 -- 172.21.15.89:0/1661777112 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0684056420 con 0x7f068c15d9d0 2024-04-02T09:09:51.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.405+0000 7f0693800700 1 -- 172.21.15.89:0/1661777112 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f068c161fa0 con 0x7f066c06adb0 2024-04-02T09:09:51.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.407+0000 7f0688c00700 1 -- 172.21.15.89:0/1661777112 <== mgr.10398 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f06800013e0 con 0x7f066c06adb0 2024-04-02T09:09:51.406 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T09:09:51.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:09:51.407 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T09:09:51.407 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:09:51.407 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T09:09:51.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.409+0000 7f0693800700 1 -- 172.21.15.89:0/1661777112 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f066c06adb0 msgr2=0x7f066c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:51.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.409+0000 7f0693800700 1 --2- 172.21.15.89:0/1661777112 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f066c06adb0 0x7f066c06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0680005910 tx=0x7f0680009040 comp rx=0 tx=0).stop 2024-04-02T09:09:51.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.409+0000 7f0693800700 1 -- 172.21.15.89:0/1661777112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068c15d9d0 msgr2=0x7f068c0cafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:51.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.409+0000 7f0693800700 1 --2- 172.21.15.89:0/1661777112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068c15d9d0 0x7f068c0cafb0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f0684014040 tx=0x7f0684004ad0 comp rx=0 tx=0).stop 2024-04-02T09:09:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.409+0000 7f0693800700 1 -- 172.21.15.89:0/1661777112 shutdown_connections 2024-04-02T09:09:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.409+0000 7f0693800700 1 --2- 172.21.15.89:0/1661777112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f068c15d9d0 0x7f068c0cafb0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.409+0000 7f0693800700 1 --2- 172.21.15.89:0/1661777112 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f068c0d16a0 0x7f068c151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.409+0000 7f0693800700 1 --2- 172.21.15.89:0/1661777112 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f066c06adb0 0x7f066c06d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.409+0000 7f0693800700 1 --2- 172.21.15.89:0/1661777112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f068c0d2540 0x7f068c0b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.409+0000 7f0693800700 1 -- 172.21.15.89:0/1661777112 >> 172.21.15.89:0/1661777112 conn(0x7f068c08aff0 msgr2=0x7f068c15c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.410+0000 7f0693800700 1 -- 172.21.15.89:0/1661777112 shutdown_connections 2024-04-02T09:09:51.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.410+0000 7f0693800700 1 -- 172.21.15.89:0/1661777112 wait complete. 2024-04-02T09:09:51.427 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T09:09:51.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.623+0000 7fe48ce00700 1 Processor -- start 2024-04-02T09:09:51.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.623+0000 7fe48ce00700 1 -- start start 2024-04-02T09:09:51.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.624+0000 7fe48ce00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe4881611b0 0x7fe48815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:51.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.625+0000 7fe48ce00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe48815f4d0 0x7fe4880ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:51.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.625+0000 7fe48ce00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4880ce270 con 0x7fe4880d21f0 2024-04-02T09:09:51.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.625+0000 7fe48ce00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48815b600 con 0x7fe4881611b0 2024-04-02T09:09:51.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.625+0000 7fe48ce00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4880d2760 con 0x7fe48815f4d0 2024-04-02T09:09:51.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.625+0000 7fe486800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe4880d21f0 0x7fe488161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34366/0 (socket says 172.21.15.89:34366) 2024-04-02T09:09:51.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.625+0000 7fe485e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe4881611b0 0x7fe48815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:51.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.625+0000 7fe486800700 1 -- 172.21.15.89:0/3511445771 learned_addr learned my addr 172.21.15.89:0/3511445771 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:51.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.625+0000 7fe485400700 1 --2- 172.21.15.89:0/3511445771 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe48815f4d0 0x7fe4880ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:51.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.625+0000 7fe485e00700 1 -- 172.21.15.89:0/3511445771 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe48815f4d0 msgr2=0x7fe4880ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:51.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.625+0000 7fe485e00700 1 --2- 172.21.15.89:0/3511445771 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe48815f4d0 0x7fe4880ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:51.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.625+0000 7fe485e00700 1 -- 172.21.15.89:0/3511445771 >> v1:172.21.15.89:6789/0 conn(0x7fe4880d21f0 legacy=0x7fe488161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:51.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.625+0000 7fe485e00700 1 -- 172.21.15.89:0/3511445771 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4880d2a40 con 0x7fe4881611b0 2024-04-02T09:09:51.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.625+0000 7fe484a00700 1 -- 172.21.15.89:0/3511445771 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742785546 0 0) 0x7fe4880ce270 con 0x7fe4880d21f0 2024-04-02T09:09:51.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.626+0000 7fe485e00700 1 --2- 172.21.15.89:0/3511445771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4881611b0 0x7fe48815ef90 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fe478009840 tx=0x7fe478009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f7647629ebafc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:51.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.626+0000 7fe484a00700 1 -- 172.21.15.89:0/3511445771 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478011830 con 0x7fe4881611b0 2024-04-02T09:09:51.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.626+0000 7fe484a00700 1 -- 172.21.15.89:0/3511445771 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe478011990 con 0x7fe4881611b0 2024-04-02T09:09:51.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.626+0000 7fe484a00700 1 -- 172.21.15.89:0/3511445771 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478011d20 con 0x7fe4881611b0 2024-04-02T09:09:51.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.626+0000 7fe48ce00700 1 -- 172.21.15.89:0/3511445771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4881611b0 msgr2=0x7fe48815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:51.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.626+0000 7fe48ce00700 1 --2- 172.21.15.89:0/3511445771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4881611b0 0x7fe48815ef90 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fe478009840 tx=0x7fe478009b50 comp rx=0 tx=0).stop 2024-04-02T09:09:51.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.627+0000 7fe48ce00700 1 -- 172.21.15.89:0/3511445771 shutdown_connections 2024-04-02T09:09:51.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.627+0000 7fe48ce00700 1 --2- 172.21.15.89:0/3511445771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4881611b0 0x7fe48815ef90 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:51.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.627+0000 7fe48ce00700 1 --2- 172.21.15.89:0/3511445771 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe48815f4d0 0x7fe4880ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:51.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.627+0000 7fe48ce00700 1 -- 172.21.15.89:0/3511445771 >> 172.21.15.89:0/3511445771 conn(0x7fe48808aff0 msgr2=0x7fe4880d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:51.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.627+0000 7fe48ce00700 1 -- 172.21.15.89:0/3511445771 shutdown_connections 2024-04-02T09:09:51.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.627+0000 7fe48ce00700 1 -- 172.21.15.89:0/3511445771 wait complete. 2024-04-02T09:09:51.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.628+0000 7fe48ce00700 1 Processor -- start 2024-04-02T09:09:51.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.628+0000 7fe48ce00700 1 -- start start 2024-04-02T09:09:51.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.629+0000 7fe48ce00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4880d21f0 0x7fe4881f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:51.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.629+0000 7fe48ce00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe48815f4d0 0x7fe4880b67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:51.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.629+0000 7fe48ce00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4881611b0 0x7fe4880b8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:51.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.629+0000 7fe48ce00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4880c9c20 con 0x7fe48815f4d0 2024-04-02T09:09:51.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.629+0000 7fe48ce00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4880b64e0 con 0x7fe4880d21f0 2024-04-02T09:09:51.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.629+0000 7fe48ce00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4881f63c0 con 0x7fe4881611b0 2024-04-02T09:09:51.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.630+0000 7fe485400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe48815f4d0 0x7fe4880b67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:51.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.630+0000 7fe485400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe48815f4d0 0x7fe4880b67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36666/0 (socket says 172.21.15.89:36666) 2024-04-02T09:09:51.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.630+0000 7fe485400700 1 -- 172.21.15.89:0/261991480 learned_addr learned my addr 172.21.15.89:0/261991480 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:51.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.630+0000 7fe485e00700 1 --2- 172.21.15.89:0/261991480 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4880d21f0 0x7fe4881f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:51.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.630+0000 7fe486800700 1 --2- 172.21.15.89:0/261991480 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4881611b0 0x7fe4880b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:51.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.630+0000 7fe485400700 1 -- 172.21.15.89:0/261991480 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4881611b0 msgr2=0x7fe4880b8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:51.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.630+0000 7fe485400700 1 --2- 172.21.15.89:0/261991480 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4881611b0 0x7fe4880b8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:51.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.630+0000 7fe485400700 1 -- 172.21.15.89:0/261991480 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4880d21f0 msgr2=0x7fe4881f2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:51.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.630+0000 7fe485400700 1 --2- 172.21.15.89:0/261991480 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4880d21f0 0x7fe4881f2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:51.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.630+0000 7fe485400700 1 -- 172.21.15.89:0/261991480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4780094a0 con 0x7fe48815f4d0 2024-04-02T09:09:51.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.630+0000 7fe485400700 1 --2- 172.21.15.89:0/261991480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe48815f4d0 0x7fe4880b67a0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fe47400e840 tx=0x7fe47400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:51.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.630+0000 7fe47e000700 1 -- 172.21.15.89:0/261991480 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47400c070 con 0x7fe48815f4d0 2024-04-02T09:09:51.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.631+0000 7fe47e000700 1 -- 172.21.15.89:0/261991480 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe474013610 con 0x7fe48815f4d0 2024-04-02T09:09:51.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.631+0000 7fe48ce00700 1 -- 172.21.15.89:0/261991480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4880b6ce0 con 0x7fe48815f4d0 2024-04-02T09:09:51.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.631+0000 7fe47e000700 1 -- 172.21.15.89:0/261991480 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4740139c0 con 0x7fe48815f4d0 2024-04-02T09:09:51.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.631+0000 7fe48ce00700 1 -- 172.21.15.89:0/261991480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4881f35d0 con 0x7fe48815f4d0 2024-04-02T09:09:51.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.632+0000 7fe48ce00700 1 -- 172.21.15.89:0/261991480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4881f6550 con 0x7fe48815f4d0 2024-04-02T09:09:51.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.633+0000 7fe47e000700 1 -- 172.21.15.89:0/261991480 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 133) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe474013b20 con 0x7fe48815f4d0 2024-04-02T09:09:51.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.634+0000 7fe47e000700 1 --2- 172.21.15.89:0/261991480 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe46806adb0 0x7fe46806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:51.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.634+0000 7fe47e000700 1 -- 172.21.15.89:0/261991480 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fe474089390 con 0x7fe48815f4d0 2024-04-02T09:09:51.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.634+0000 7fe485e00700 1 --2- 172.21.15.89:0/261991480 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe46806adb0 0x7fe46806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:51.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.635+0000 7fe485e00700 1 --2- 172.21.15.89:0/261991480 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe46806adb0 0x7fe46806d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe4780090b0 tx=0x7fe47801c040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:51.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.638+0000 7fe47e000700 1 -- 172.21.15.89:0/261991480 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe474054220 con 0x7fe48815f4d0 2024-04-02T09:09:51.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:51.898+0000 7fe48ce00700 1 -- 172.21.15.89:0/261991480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe488067720 con 0x7fe48815f4d0 2024-04-02T09:09:52.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.736+0000 7fe47e000700 1 -- 172.21.15.89:0/261991480 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 134) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fe4740876f0 con 0x7fe48815f4d0 2024-04-02T09:09:52.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.738+0000 7fe47e000700 1 -- 172.21.15.89:0/261991480 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v134) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe474084d10 con 0x7fe48815f4d0 2024-04-02T09:09:52.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.741+0000 7fe48ce00700 1 -- 172.21.15.89:0/261991480 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe46806adb0 msgr2=0x7fe46806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:52.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.741+0000 7fe48ce00700 1 --2- 172.21.15.89:0/261991480 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe46806adb0 0x7fe46806d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe4780090b0 tx=0x7fe47801c040 comp rx=0 tx=0).stop 2024-04-02T09:09:52.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.741+0000 7fe48ce00700 1 -- 172.21.15.89:0/261991480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe48815f4d0 msgr2=0x7fe4880b67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:52.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.741+0000 7fe48ce00700 1 --2- 172.21.15.89:0/261991480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe48815f4d0 0x7fe4880b67a0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fe47400e840 tx=0x7fe47400eb50 comp rx=0 tx=0).stop 2024-04-02T09:09:52.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.741+0000 7fe48ce00700 1 -- 172.21.15.89:0/261991480 shutdown_connections 2024-04-02T09:09:52.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.741+0000 7fe48ce00700 1 --2- 172.21.15.89:0/261991480 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe46806adb0 0x7fe46806d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:52.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.741+0000 7fe48ce00700 1 --2- 172.21.15.89:0/261991480 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4880d21f0 0x7fe4881f2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:52.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.741+0000 7fe48ce00700 1 --2- 172.21.15.89:0/261991480 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe4881611b0 0x7fe4880b8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:52.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.741+0000 7fe48ce00700 1 --2- 172.21.15.89:0/261991480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe48815f4d0 0x7fe4880b67a0 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:52.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.741+0000 7fe48ce00700 1 -- 172.21.15.89:0/261991480 >> 172.21.15.89:0/261991480 conn(0x7fe48808aff0 msgr2=0x7fe48815f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:52.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.741+0000 7fe48ce00700 1 -- 172.21.15.89:0/261991480 shutdown_connections 2024-04-02T09:09:52.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.741+0000 7fe48ce00700 1 -- 172.21.15.89:0/261991480 wait complete. 2024-04-02T09:09:52.764 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T09:09:52.953 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:52.954+0000 7f6896b0a240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:09:52.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.977+0000 7f9e8e600700 1 Processor -- start 2024-04-02T09:09:52.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.977+0000 7f9e8e600700 1 -- start start 2024-04-02T09:09:52.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.978+0000 7f9e8e600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9e8815dad0 0x7f9e880ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:52.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.978+0000 7f9e8e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e880d1630 0x7f9e880d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:52.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.978+0000 7f9e8e600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e880d1fd0 0x7f9e8815deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:52.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.978+0000 7f9e8e600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e880cfcc0 con 0x7f9e880d1fd0 2024-04-02T09:09:52.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.978+0000 7f9e8e600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e880cd9f0 con 0x7f9e8815dad0 2024-04-02T09:09:52.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.978+0000 7f9e8e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8815f820 con 0x7f9e880d1630 2024-04-02T09:09:52.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.978+0000 7f9e87e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e880d1fd0 0x7f9e8815deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.978+0000 7f9e87400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9e8815dad0 0x7f9e880ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.978+0000 7f9e87e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e880d1fd0 0x7f9e8815deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36696/0 (socket says 172.21.15.89:36696) 2024-04-02T09:09:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.978+0000 7f9e86a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e880d1630 0x7f9e880d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.978+0000 7f9e87e00700 1 -- 172.21.15.89:0/1626286379 learned_addr learned my addr 172.21.15.89:0/1626286379 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.979+0000 7f9e87e00700 1 -- 172.21.15.89:0/1626286379 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e880d1630 msgr2=0x7f9e880d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.979+0000 7f9e87e00700 1 --2- 172.21.15.89:0/1626286379 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e880d1630 0x7f9e880d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.979+0000 7f9e87e00700 1 -- 172.21.15.89:0/1626286379 >> v2:172.21.15.113:3300/0 conn(0x7f9e8815dad0 msgr2=0x7f9e880ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.979+0000 7f9e87e00700 1 --2- 172.21.15.89:0/1626286379 >> v2:172.21.15.113:3300/0 conn(0x7f9e8815dad0 0x7f9e880ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.979+0000 7f9e87e00700 1 -- 172.21.15.89:0/1626286379 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e880d2460 con 0x7f9e880d1fd0 2024-04-02T09:09:52.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.979+0000 7f9e87400700 1 --2- 172.21.15.89:0/1626286379 >> v2:172.21.15.113:3300/0 conn(0x7f9e8815dad0 0x7f9e880ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:09:52.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.979+0000 7f9e87e00700 1 --2- 172.21.15.89:0/1626286379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e880d1fd0 0x7f9e8815deb0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f9e8808d380 tx=0x7f9e8000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8165fb9e98834c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:52.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.979+0000 7f9e86000700 1 -- 172.21.15.89:0/1626286379 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8000e070 con 0x7f9e880d1fd0 2024-04-02T09:09:52.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.979+0000 7f9e86000700 1 -- 172.21.15.89:0/1626286379 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8000b910 con 0x7f9e880d1fd0 2024-04-02T09:09:52.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.979+0000 7f9e86000700 1 -- 172.21.15.89:0/1626286379 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8000bcc0 con 0x7f9e880d1fd0 2024-04-02T09:09:52.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.980+0000 7f9e8e600700 1 -- 172.21.15.89:0/1626286379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e880d1fd0 msgr2=0x7f9e8815deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:52.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.980+0000 7f9e8e600700 1 --2- 172.21.15.89:0/1626286379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e880d1fd0 0x7f9e8815deb0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f9e8808d380 tx=0x7f9e8000b330 comp rx=0 tx=0).stop 2024-04-02T09:09:52.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.980+0000 7f9e8e600700 1 -- 172.21.15.89:0/1626286379 shutdown_connections 2024-04-02T09:09:52.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.980+0000 7f9e8e600700 1 --2- 172.21.15.89:0/1626286379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e880d1fd0 0x7f9e8815deb0 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:52.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.980+0000 7f9e8e600700 1 --2- 172.21.15.89:0/1626286379 >> v2:172.21.15.113:3300/0 conn(0x7f9e8815dad0 0x7f9e880ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:52.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.980+0000 7f9e8e600700 1 --2- 172.21.15.89:0/1626286379 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e880d1630 0x7f9e880d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:52.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.980+0000 7f9e8e600700 1 -- 172.21.15.89:0/1626286379 >> 172.21.15.89:0/1626286379 conn(0x7f9e8808aff0 msgr2=0x7f9e880d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:52.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.980+0000 7f9e8e600700 1 -- 172.21.15.89:0/1626286379 shutdown_connections 2024-04-02T09:09:52.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.980+0000 7f9e8e600700 1 -- 172.21.15.89:0/1626286379 wait complete. 2024-04-02T09:09:52.981 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:52.981+0000 7f8a38e84240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:09:52.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.983+0000 7f9e8e600700 1 Processor -- start 2024-04-02T09:09:52.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.983+0000 7f9e8e600700 1 -- start start 2024-04-02T09:09:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.984+0000 7f9e8e600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e880d1630 0x7f9e880b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.984+0000 7f9e8e600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e880d1fd0 0x7f9e881529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.984+0000 7f9e8e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e8815dad0 0x7f9e880c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.984+0000 7f9e8e600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e88153e50 con 0x7f9e880d1630 2024-04-02T09:09:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.984+0000 7f9e8e600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e880b8f60 con 0x7f9e880d1fd0 2024-04-02T09:09:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.984+0000 7f9e8e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e88152f30 con 0x7f9e8815dad0 2024-04-02T09:09:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.984+0000 7f9e87400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e880d1630 0x7f9e880b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.984+0000 7f9e87400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e880d1630 0x7f9e880b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36706/0 (socket says 172.21.15.89:36706) 2024-04-02T09:09:52.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.984+0000 7f9e87400700 1 -- 172.21.15.89:0/3880963399 learned_addr learned my addr 172.21.15.89:0/3880963399 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.984+0000 7f9e87e00700 1 --2- 172.21.15.89:0/3880963399 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e8815dad0 0x7f9e880c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.984+0000 7f9e86a00700 1 --2- 172.21.15.89:0/3880963399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e880d1fd0 0x7f9e881529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.985+0000 7f9e87400700 1 -- 172.21.15.89:0/3880963399 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e8815dad0 msgr2=0x7f9e880c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.985+0000 7f9e87400700 1 --2- 172.21.15.89:0/3880963399 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e8815dad0 0x7f9e880c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.985+0000 7f9e87400700 1 -- 172.21.15.89:0/3880963399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e880d1fd0 msgr2=0x7f9e881529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.985+0000 7f9e87400700 1 --2- 172.21.15.89:0/3880963399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e880d1fd0 0x7f9e881529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.985+0000 7f9e87400700 1 -- 172.21.15.89:0/3880963399 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e780094a0 con 0x7f9e880d1630 2024-04-02T09:09:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.985+0000 7f9e87400700 1 --2- 172.21.15.89:0/3880963399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e880d1630 0x7f9e880b76a0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800ebe0 tx=0x7f9e7800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:52.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.985+0000 7f9e7f400700 1 -- 172.21.15.89:0/3880963399 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e78009760 con 0x7f9e880d1630 2024-04-02T09:09:52.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.985+0000 7f9e7f400700 1 -- 172.21.15.89:0/3880963399 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7800c850 con 0x7f9e880d1630 2024-04-02T09:09:52.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.985+0000 7f9e7f400700 1 -- 172.21.15.89:0/3880963399 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7800cbe0 con 0x7f9e880d1630 2024-04-02T09:09:52.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.985+0000 7f9e8e600700 1 -- 172.21.15.89:0/3880963399 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e80009b30 con 0x7f9e880d1630 2024-04-02T09:09:52.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.985+0000 7f9e8e600700 1 -- 172.21.15.89:0/3880963399 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e881531d0 con 0x7f9e880d1630 2024-04-02T09:09:52.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.987+0000 7f9e8e600700 1 -- 172.21.15.89:0/3880963399 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e881540c0 con 0x7f9e880d1630 2024-04-02T09:09:52.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.988+0000 7f9e7f400700 1 -- 172.21.15.89:0/3880963399 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 134) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f9e7800cd40 con 0x7f9e880d1630 2024-04-02T09:09:52.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.988+0000 7f9e7f400700 1 --2- 172.21.15.89:0/3880963399 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e6806ae00 0x7f9e6806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:52.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.989+0000 7f9e7f400700 1 -- 172.21.15.89:0/3880963399 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f9e780897a0 con 0x7f9e880d1630 2024-04-02T09:09:52.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.989+0000 7f9e86a00700 1 -- 172.21.15.89:0/3880963399 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e6806ae00 msgr2=0x7f9e6806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:52.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.989+0000 7f9e86a00700 1 --2- 172.21.15.89:0/3880963399 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e6806ae00 0x7f9e6806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:52.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:52.993+0000 7f9e7f400700 1 -- 172.21.15.89:0/3880963399 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e78054610 con 0x7f9e880d1630 2024-04-02T09:09:53.138 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:53.139+0000 7f6896b0a240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:09:53.169 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:53.170+0000 7f8a38e84240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:09:53.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.190+0000 7f9e86a00700 1 -- 172.21.15.89:0/3880963399 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e6806ae00 msgr2=0x7f9e6806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:53.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.190+0000 7f9e86a00700 1 --2- 172.21.15.89:0/3880963399 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e6806ae00 0x7f9e6806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:53.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.240+0000 7f9e8e600700 1 -- 172.21.15.89:0/3880963399 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f9e88067720 con 0x7f9e880d1630 2024-04-02T09:09:53.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.245+0000 7f9e7f400700 1 -- 172.21.15.89:0/3880963399 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9e78057c30 con 0x7f9e880d1630 2024-04-02T09:09:53.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.247+0000 7f9e8e600700 1 -- 172.21.15.89:0/3880963399 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e6806ae00 msgr2=0x7f9e6806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:53.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.247+0000 7f9e8e600700 1 --2- 172.21.15.89:0/3880963399 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e6806ae00 0x7f9e6806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.247+0000 7f9e8e600700 1 -- 172.21.15.89:0/3880963399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e880d1630 msgr2=0x7f9e880b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:53.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.247+0000 7f9e8e600700 1 --2- 172.21.15.89:0/3880963399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e880d1630 0x7f9e880b76a0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800ebe0 tx=0x7f9e7800eef0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.247+0000 7f9e8e600700 1 -- 172.21.15.89:0/3880963399 shutdown_connections 2024-04-02T09:09:53.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.247+0000 7f9e8e600700 1 --2- 172.21.15.89:0/3880963399 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e8815dad0 0x7f9e880c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.247+0000 7f9e8e600700 1 --2- 172.21.15.89:0/3880963399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e880d1630 0x7f9e880b76a0 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.247+0000 7f9e8e600700 1 --2- 172.21.15.89:0/3880963399 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e6806ae00 0x7f9e6806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.247+0000 7f9e8e600700 1 --2- 172.21.15.89:0/3880963399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e880d1fd0 0x7f9e881529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.247+0000 7f9e8e600700 1 -- 172.21.15.89:0/3880963399 >> 172.21.15.89:0/3880963399 conn(0x7f9e8808aff0 msgr2=0x7f9e8806b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:53.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.247+0000 7f9e8e600700 1 -- 172.21.15.89:0/3880963399 shutdown_connections 2024-04-02T09:09:53.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.247+0000 7f9e8e600700 1 -- 172.21.15.89:0/3880963399 wait complete. 2024-04-02T09:09:53.265 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T09:09:53.455 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:53.456+0000 7f6896b0a240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:09:53.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.464+0000 7fd080000700 1 Processor -- start 2024-04-02T09:09:53.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.464+0000 7fd080000700 1 -- start start 2024-04-02T09:09:53.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.465+0000 7fd080000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780d21f0 0x7fd0780ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:53.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.465+0000 7fd080000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0780cec40 0x7fd078161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:53.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.465+0000 7fd080000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0780ce270 con 0x7fd0781611b0 2024-04-02T09:09:53.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.465+0000 7fd080000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07815b600 con 0x7fd0780cec40 2024-04-02T09:09:53.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.465+0000 7fd080000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0780cf0b0 con 0x7fd0780d21f0 2024-04-02T09:09:53.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.465+0000 7fd07d600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd0781611b0 0x7fd07815ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34392/0 (socket says 172.21.15.89:34392) 2024-04-02T09:09:53.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.465+0000 7fd07d600700 1 -- 172.21.15.89:0/3768357466 learned_addr learned my addr 172.21.15.89:0/3768357466 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:09:53.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.465+0000 7fd07e000700 1 --2- 172.21.15.89:0/3768357466 >> v2:172.21.15.89:3300/0 conn(0x7fd0780cec40 0x7fd078161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:53.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.465+0000 7fd07cc00700 1 --2- 172.21.15.89:0/3768357466 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780d21f0 0x7fd0780ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:53.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.466+0000 7fd077e00700 1 -- 172.21.15.89:0/3768357466 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176988023 0 0) 0x7fd0780ce270 con 0x7fd0781611b0 2024-04-02T09:09:53.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.466+0000 7fd077e00700 1 -- 172.21.15.89:0/3768357466 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0600035b0 con 0x7fd0781611b0 2024-04-02T09:09:53.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.466+0000 7fd07e000700 1 -- 172.21.15.89:0/3768357466 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780d21f0 msgr2=0x7fd0780ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:53.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.466+0000 7fd07e000700 1 --2- 172.21.15.89:0/3768357466 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780d21f0 0x7fd0780ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.466+0000 7fd07e000700 1 -- 172.21.15.89:0/3768357466 >> v1:172.21.15.89:6789/0 conn(0x7fd0781611b0 legacy=0x7fd07815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:53.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.466+0000 7fd07e000700 1 -- 172.21.15.89:0/3768357466 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0780d2760 con 0x7fd0780cec40 2024-04-02T09:09:53.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.466+0000 7fd07e000700 1 --2- 172.21.15.89:0/3768357466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0780cec40 0x7fd078161590 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fd070009d90 tx=0x7fd07000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15ebdc4987e7a1e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:53.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.466+0000 7fd077e00700 1 -- 172.21.15.89:0/3768357466 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07000e070 con 0x7fd0780cec40 2024-04-02T09:09:53.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.466+0000 7fd077e00700 1 -- 172.21.15.89:0/3768357466 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07000bb20 con 0x7fd0780cec40 2024-04-02T09:09:53.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.467+0000 7fd077e00700 1 -- 172.21.15.89:0/3768357466 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070012430 con 0x7fd0780cec40 2024-04-02T09:09:53.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.467+0000 7fd080000700 1 -- 172.21.15.89:0/3768357466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0780cec40 msgr2=0x7fd078161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:53.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.467+0000 7fd080000700 1 --2- 172.21.15.89:0/3768357466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0780cec40 0x7fd078161590 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fd070009d90 tx=0x7fd07000b330 comp rx=0 tx=0).stop 2024-04-02T09:09:53.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.467+0000 7fd080000700 1 -- 172.21.15.89:0/3768357466 shutdown_connections 2024-04-02T09:09:53.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.467+0000 7fd080000700 1 --2- 172.21.15.89:0/3768357466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0780cec40 0x7fd078161590 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.467+0000 7fd080000700 1 --2- 172.21.15.89:0/3768357466 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780d21f0 0x7fd0780ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.467+0000 7fd080000700 1 -- 172.21.15.89:0/3768357466 >> 172.21.15.89:0/3768357466 conn(0x7fd07808aff0 msgr2=0x7fd0780d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:53.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.467+0000 7fd080000700 1 -- 172.21.15.89:0/3768357466 shutdown_connections 2024-04-02T09:09:53.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.468+0000 7fd080000700 1 -- 172.21.15.89:0/3768357466 wait complete. 2024-04-02T09:09:53.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.470+0000 7fd080000700 1 Processor -- start 2024-04-02T09:09:53.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.470+0000 7fd080000700 1 -- start start 2024-04-02T09:09:53.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.471+0000 7fd080000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780cec40 0x7fd078154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:53.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.471+0000 7fd080000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0780d21f0 0x7fd0780b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:53.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.471+0000 7fd080000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0781611b0 0x7fd0780b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:53.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.471+0000 7fd080000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd078155410 con 0x7fd0781611b0 2024-04-02T09:09:53.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.471+0000 7fd080000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd078156250 con 0x7fd0780d21f0 2024-04-02T09:09:53.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.471+0000 7fd080000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd078155920 con 0x7fd0780cec40 2024-04-02T09:09:53.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.471+0000 7fd07e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0781611b0 0x7fd0780b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:53.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.472+0000 7fd07e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0781611b0 0x7fd0780b7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36736/0 (socket says 172.21.15.89:36736) 2024-04-02T09:09:53.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.472+0000 7fd07e000700 1 -- 172.21.15.89:0/2621149141 learned_addr learned my addr 172.21.15.89:0/2621149141 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:53.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.472+0000 7fd07cc00700 1 --2- 172.21.15.89:0/2621149141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0780d21f0 0x7fd0780b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:53.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.472+0000 7fd07d600700 1 --2- 172.21.15.89:0/2621149141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780cec40 0x7fd078154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:53.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.472+0000 7fd07e000700 1 -- 172.21.15.89:0/2621149141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780cec40 msgr2=0x7fd078154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:53.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.472+0000 7fd07e000700 1 --2- 172.21.15.89:0/2621149141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780cec40 0x7fd078154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.472+0000 7fd07e000700 1 -- 172.21.15.89:0/2621149141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0780d21f0 msgr2=0x7fd0780b6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:53.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.472+0000 7fd07e000700 1 --2- 172.21.15.89:0/2621149141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0780d21f0 0x7fd0780b6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.472+0000 7fd07e000700 1 -- 172.21.15.89:0/2621149141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd070009a70 con 0x7fd0781611b0 2024-04-02T09:09:53.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.472+0000 7fd07e000700 1 --2- 172.21.15.89:0/2621149141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0781611b0 0x7fd0780b7230 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fd070014040 tx=0x7fd070004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:53.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.473+0000 7fd075600700 1 -- 172.21.15.89:0/2621149141 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070012430 con 0x7fd0781611b0 2024-04-02T09:09:53.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.473+0000 7fd075600700 1 -- 172.21.15.89:0/2621149141 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd070020070 con 0x7fd0781611b0 2024-04-02T09:09:53.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.473+0000 7fd080000700 1 -- 172.21.15.89:0/2621149141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd078155c00 con 0x7fd0781611b0 2024-04-02T09:09:53.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.473+0000 7fd075600700 1 -- 172.21.15.89:0/2621149141 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0700098f0 con 0x7fd0781611b0 2024-04-02T09:09:53.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.473+0000 7fd080000700 1 -- 172.21.15.89:0/2621149141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0781566e0 con 0x7fd0781611b0 2024-04-02T09:09:53.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.474+0000 7fd080000700 1 -- 172.21.15.89:0/2621149141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd078156400 con 0x7fd0781611b0 2024-04-02T09:09:53.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.475+0000 7fd075600700 1 -- 172.21.15.89:0/2621149141 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 134) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fd070004040 con 0x7fd0781611b0 2024-04-02T09:09:53.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.476+0000 7fd075600700 1 --2- 172.21.15.89:0/2621149141 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd05806ae00 0x7fd05806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:53.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.476+0000 7fd075600700 1 -- 172.21.15.89:0/2621149141 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fd070022070 con 0x7fd0781611b0 2024-04-02T09:09:53.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.476+0000 7fd07d600700 1 -- 172.21.15.89:0/2621149141 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd05806ae00 msgr2=0x7fd05806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:53.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.476+0000 7fd07d600700 1 --2- 172.21.15.89:0/2621149141 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd05806ae00 0x7fd05806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:53.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.480+0000 7fd075600700 1 -- 172.21.15.89:0/2621149141 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd0700be580 con 0x7fd0781611b0 2024-04-02T09:09:53.523 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:53.523+0000 7f8a38e84240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:09:53.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.677+0000 7fd07d600700 1 -- 172.21.15.89:0/2621149141 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd05806ae00 msgr2=0x7fd05806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:53.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.677+0000 7fd07d600700 1 --2- 172.21.15.89:0/2621149141 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd05806ae00 0x7fd05806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:53.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.731+0000 7fd080000700 1 -- 172.21.15.89:0/2621149141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fd0781568a0 con 0x7fd0781611b0 2024-04-02T09:09:53.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.736+0000 7fd075600700 1 -- 172.21.15.89:0/2621149141 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd078156400 con 0x7fd0781611b0 2024-04-02T09:09:53.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.738+0000 7fd080000700 1 -- 172.21.15.89:0/2621149141 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd05806ae00 msgr2=0x7fd05806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:53.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.738+0000 7fd080000700 1 --2- 172.21.15.89:0/2621149141 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd05806ae00 0x7fd05806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.738+0000 7fd080000700 1 -- 172.21.15.89:0/2621149141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0781611b0 msgr2=0x7fd0780b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:53.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.738+0000 7fd080000700 1 --2- 172.21.15.89:0/2621149141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0781611b0 0x7fd0780b7230 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fd070014040 tx=0x7fd070004ad0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.739+0000 7fd080000700 1 -- 172.21.15.89:0/2621149141 shutdown_connections 2024-04-02T09:09:53.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.739+0000 7fd080000700 1 --2- 172.21.15.89:0/2621149141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0781611b0 0x7fd0780b7230 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.739+0000 7fd080000700 1 --2- 172.21.15.89:0/2621149141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd0780cec40 0x7fd078154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.739+0000 7fd080000700 1 --2- 172.21.15.89:0/2621149141 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd05806ae00 0x7fd05806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.739+0000 7fd080000700 1 --2- 172.21.15.89:0/2621149141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0780d21f0 0x7fd0780b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:53.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.739+0000 7fd080000700 1 -- 172.21.15.89:0/2621149141 >> 172.21.15.89:0/2621149141 conn(0x7fd07808aff0 msgr2=0x7fd07806d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:53.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.739+0000 7fd080000700 1 -- 172.21.15.89:0/2621149141 shutdown_connections 2024-04-02T09:09:53.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:53.739+0000 7fd080000700 1 -- 172.21.15.89:0/2621149141 wait complete. 2024-04-02T09:09:53.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:09:53.758 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-02T09:09:53.806 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-04-02T09:09:53.981 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:53.982+0000 7f6896b0a240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:09:54.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.006+0000 7f23ee600700 1 Processor -- start 2024-04-02T09:09:54.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.007+0000 7f23ee600700 1 -- start start 2024-04-02T09:09:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.008+0000 7f23ee600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23e815dad0 0x7f23e80ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.008+0000 7f23ee600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f23e80d1630 0x7f23e80d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.008+0000 7f23ee600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e815f820 con 0x7f23e80d1fd0 2024-04-02T09:09:54.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.008+0000 7f23ee600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e80cfcc0 con 0x7f23e80d1630 2024-04-02T09:09:54.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.008+0000 7f23ee600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80cd9f0 con 0x7f23e815dad0 2024-04-02T09:09:54.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.008+0000 7f23e6a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f23e80d1630 0x7f23e80d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:54.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.008+0000 7f23e7400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23e815dad0 0x7f23e80ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:54.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.008+0000 7f23e7400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23e815dad0 0x7f23e80ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:49086/0 (socket says 172.21.15.89:49086) 2024-04-02T09:09:54.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.008+0000 7f23e6a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f23e80d1630 0x7f23e80d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:47428/0 (socket says 172.21.15.89:47428) 2024-04-02T09:09:54.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.008+0000 7f23e7e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f23e80d1fd0 0x7f23e815deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:33134/0 (socket says 172.21.15.89:33134) 2024-04-02T09:09:54.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.008+0000 7f23e7400700 1 -- 172.21.15.89:0/3524598545 learned_addr learned my addr 172.21.15.89:0/3524598545 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:54.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.009+0000 7f23e6a00700 1 -- 172.21.15.89:0/3524598545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23e815dad0 msgr2=0x7f23e80ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:09:54.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.009+0000 7f23e6a00700 1 --2- 172.21.15.89:0/3524598545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23e815dad0 0x7f23e80ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:54.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.009+0000 7f23e6a00700 1 -- 172.21.15.89:0/3524598545 >> v1:172.21.15.113:6789/0 conn(0x7f23e80d1fd0 legacy=0x7f23e815deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:54.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.009+0000 7f23e6a00700 1 -- 172.21.15.89:0/3524598545 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e80d2460 con 0x7f23e80d1630 2024-04-02T09:09:54.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.009+0000 7f23e7400700 1 --2- 172.21.15.89:0/3524598545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23e815dad0 0x7f23e80ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:09:54.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.009+0000 7f23e6000700 1 -- 172.21.15.89:0/3524598545 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7243082 0 0) 0x7f23e815f820 con 0x7f23e80d1fd0 2024-04-02T09:09:54.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.009+0000 7f23e6a00700 1 --2- 172.21.15.89:0/3524598545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e80d1630 0x7f23e80d1a90 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009840 tx=0x7f23d4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61b2b41aafd4d59 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:54.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.009+0000 7f23e6000700 1 -- 172.21.15.89:0/3524598545 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d4011830 con 0x7f23e80d1630 2024-04-02T09:09:54.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.009+0000 7f23e6000700 1 -- 172.21.15.89:0/3524598545 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d4011990 con 0x7f23e80d1630 2024-04-02T09:09:54.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.009+0000 7f23e6000700 1 -- 172.21.15.89:0/3524598545 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d4011d20 con 0x7f23e80d1630 2024-04-02T09:09:54.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.010+0000 7f23ee600700 1 -- 172.21.15.89:0/3524598545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e80d1630 msgr2=0x7f23e80d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:54.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.010+0000 7f23ee600700 1 --2- 172.21.15.89:0/3524598545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e80d1630 0x7f23e80d1a90 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009840 tx=0x7f23d4009b50 comp rx=0 tx=0).stop 2024-04-02T09:09:54.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.010+0000 7f23ee600700 1 -- 172.21.15.89:0/3524598545 shutdown_connections 2024-04-02T09:09:54.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.010+0000 7f23ee600700 1 --2- 172.21.15.89:0/3524598545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23e815dad0 0x7f23e80ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:54.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.010+0000 7f23ee600700 1 --2- 172.21.15.89:0/3524598545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e80d1630 0x7f23e80d1a90 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:54.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.010+0000 7f23ee600700 1 -- 172.21.15.89:0/3524598545 >> 172.21.15.89:0/3524598545 conn(0x7f23e808aff0 msgr2=0x7f23e80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:09:54.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.010+0000 7f23ee600700 1 -- 172.21.15.89:0/3524598545 shutdown_connections 2024-04-02T09:09:54.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.010+0000 7f23ee600700 1 -- 172.21.15.89:0/3524598545 wait complete. 2024-04-02T09:09:54.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.012+0000 7f23ee600700 1 Processor -- start 2024-04-02T09:09:54.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.012+0000 7f23ee600700 1 -- start start 2024-04-02T09:09:54.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23ee600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23e80d1630 0x7f23e81f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:54.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23ee600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23e80d1fd0 0x7f23e80c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:54.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23ee600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e815dad0 0x7f23e80c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:54.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23ee600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80ca360 con 0x7f23e80d1fd0 2024-04-02T09:09:54.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23ee600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80c9ac0 con 0x7f23e815dad0 2024-04-02T09:09:54.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23ee600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80c9da0 con 0x7f23e80d1630 2024-04-02T09:09:54.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23e6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23e80d1fd0 0x7f23e80c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:54.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23e7400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23e80d1630 0x7f23e81f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:54.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23e7400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23e80d1630 0x7f23e81f2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:49090/0 (socket says 172.21.15.89:49090) 2024-04-02T09:09:54.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23e7400700 1 -- 172.21.15.89:0/3488363449 learned_addr learned my addr 172.21.15.89:0/3488363449 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:09:54.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23e7e00700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e815dad0 0x7f23e80c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:09:54.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23e7400700 1 -- 172.21.15.89:0/3488363449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e815dad0 msgr2=0x7f23e80c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:54.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23e7400700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e815dad0 0x7f23e80c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:54.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23e7400700 1 -- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23e80d1fd0 msgr2=0x7f23e80c8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:09:54.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23e7400700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23e80d1fd0 0x7f23e80c8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:54.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.013+0000 7f23e7400700 1 -- 172.21.15.89:0/3488363449 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d8009550 con 0x7f23e80d1630 2024-04-02T09:09:54.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.014+0000 7f23e7400700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23e80d1630 0x7f23e81f2800 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f23d800e9b0 tx=0x7f23d800ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:09:54.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.014+0000 7f23df400700 1 -- 172.21.15.89:0/3488363449 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d800ee50 con 0x7f23e80d1630 2024-04-02T09:09:54.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.014+0000 7f23df400700 1 -- 172.21.15.89:0/3488363449 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d800c850 con 0x7f23e80d1630 2024-04-02T09:09:54.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.014+0000 7f23df400700 1 -- 172.21.15.89:0/3488363449 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d800cbc0 con 0x7f23e80d1630 2024-04-02T09:09:54.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.014+0000 7f23ee600700 1 -- 172.21.15.89:0/3488363449 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d40094a0 con 0x7f23e80d1630 2024-04-02T09:09:54.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.014+0000 7f23ee600700 1 -- 172.21.15.89:0/3488363449 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e80cb750 con 0x7f23e80d1630 2024-04-02T09:09:54.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.015+0000 7f23ee600700 1 -- 172.21.15.89:0/3488363449 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e80cb940 con 0x7f23e80d1630 2024-04-02T09:09:54.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.017+0000 7f23df400700 1 -- 172.21.15.89:0/3488363449 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 134) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f23d800cd20 con 0x7f23e80d1630 2024-04-02T09:09:54.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.018+0000 7f23df400700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 0x7f23c806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:09:54.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.018+0000 7f23e6a00700 1 -- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 msgr2=0x7f23c806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:54.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.018+0000 7f23df400700 1 -- 172.21.15.89:0/3488363449 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f23d80896b0 con 0x7f23e80d1630 2024-04-02T09:09:54.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.018+0000 7f23e6a00700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 0x7f23c806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:09:54.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.023+0000 7f23df400700 1 -- 172.21.15.89:0/3488363449 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f23d80545c0 con 0x7f23e80d1630 2024-04-02T09:09:54.092 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:54.093+0000 7f8a38e84240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:09:54.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.219+0000 7f23e6a00700 1 -- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 msgr2=0x7f23c806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:54.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.219+0000 7f23e6a00700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 0x7f23c806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:09:54.237 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:54.239+0000 7f6896b0a240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:09:54.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.267+0000 7f23ee600700 1 -- 172.21.15.89:0/3488363449 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f23e80cf9e0 con 0x7f23c806ae00 2024-04-02T09:09:54.368 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:54.368+0000 7f8a38e84240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:09:54.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.620+0000 7f23e6a00700 1 -- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 msgr2=0x7f23c806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:54.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:54.620+0000 7f23e6a00700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 0x7f23c806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T09:09:54.691 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:54.692+0000 7f6896b0a240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:09:54.825 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:54.825+0000 7f8a38e84240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:09:55.395 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:55.396+0000 7f6896b0a240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:09:55.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:55.421+0000 7f23e6a00700 1 -- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 msgr2=0x7f23c806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:55.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:55.421+0000 7f23e6a00700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 0x7f23c806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T09:09:55.485 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:55.486+0000 7f6896b0a240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:09:55.528 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:55.529+0000 7f8a38e84240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:09:55.619 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:55.619+0000 7f8a38e84240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:09:55.642 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:55.643+0000 7f6896b0a240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:09:55.776 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:55.777+0000 7f8a38e84240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:09:55.853 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:55.854+0000 7f6896b0a240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:09:55.943 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:55.945+0000 7f6896b0a240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:09:55.989 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:55.989+0000 7f8a38e84240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:09:56.041 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:56.042+0000 7f6896b0a240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:09:56.079 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:56.080+0000 7f8a38e84240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:09:56.138 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:56.139+0000 7f6896b0a240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:09:56.177 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:56.178+0000 7f8a38e84240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:09:56.235 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:56.236+0000 7f6896b0a240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:09:56.274 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:56.275+0000 7f8a38e84240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:09:56.372 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:56.372+0000 7f8a38e84240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:09:56.446 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:56.448+0000 7f6896b0a240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:09:56.584 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:56.585+0000 7f8a38e84240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:09:56.699 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:56.700+0000 7f6896b0a240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:09:56.840 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:56.840+0000 7f8a38e84240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:09:56.889 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:56.891+0000 7f6896b0a240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:09:56.981 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:56.982+0000 7f6896b0a240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:09:57.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:57.022+0000 7f23e6a00700 1 -- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 msgr2=0x7f23c806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:09:57.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:57.022+0000 7f23e6a00700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 0x7f23c806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T09:09:57.031 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:57.032+0000 7f8a38e84240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:09:57.125 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:57.125+0000 7f8a38e84240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:09:57.709 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:57.710+0000 7f6896b0a240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:09:57.858 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:57.860+0000 7f8a38e84240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:09:57.920 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:57.922+0000 7f6896b0a240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:09:58.070 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:58.072+0000 7f8a38e84240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:09:58.127 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:58.130+0000 7f6896b0a240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:09:58.282 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:58.284+0000 7f8a38e84240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:09:58.604 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:58.606+0000 7f6896b0a240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:09:58.758 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:58.760+0000 7f8a38e84240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:09:58.809 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:58.811+0000 7f6896b0a240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:09:58.968 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:58.970+0000 7f8a38e84240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:09:59.246 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:59.248+0000 7f6896b0a240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:09:59.357 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:09:59.359+0000 7f6896b0a240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:09:59.413 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:59.415+0000 7f8a38e84240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:09:59.524 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:59.526+0000 7f8a38e84240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:09:59.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:59.536+0000 7f23df400700 1 -- 172.21.15.89:0/3488363449 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mgrmap(e 135) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f23d8087a10 con 0x7f23e80d1630 2024-04-02T09:09:59.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:59.536+0000 7f23df400700 1 -- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 msgr2=0x7f23c806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:09:59.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:09:59.536+0000 7f23df400700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 0x7f23c806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:09:59.707 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:59.708+0000 7f89a1600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:59.707 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:59.708+0000 7f89a1600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:59.707 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:59.708+0000 7f89a1600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:59.708 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:59.708+0000 7f89a1600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:59.708 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:59.708+0000 7f89a1600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:59.708 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:59.708+0000 7f89a4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:59.708 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:59.708+0000 7f89a4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:59.708 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:59.708+0000 7f89a4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:59.708 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:59.708+0000 7f89a4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:09:59.708 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:09:59.708+0000 7f89a4800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:10:00.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.542+0000 7f23df400700 1 -- 172.21.15.89:0/3488363449 <== mon.2 v2:172.21.15.113:3301/0 8 ==== mgrmap(e 136) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f23d8054990 con 0x7f23e80d1630 2024-04-02T09:10:00.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.542+0000 7f23df400700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 0x7f23c806d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:10:00.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.543+0000 7f23df400700 1 -- 172.21.15.89:0/3488363449 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f23c8035920 con 0x7f23c806ae00 2024-04-02T09:10:00.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.545+0000 7f23e6a00700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 0x7f23c806d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:10:00.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.548+0000 7f23e6a00700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 0x7f23c806d240 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f23d4000c00 tx=0x7f23d4008030 comp rx=0 tx=0).ready entity=mgr.10818 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:10:00.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.625+0000 7f23df400700 1 -- 172.21.15.89:0/3488363449 <== mgr.10818 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f23c8035920 con 0x7f23c806ae00 2024-04-02T09:10:00.623 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path snap_test_dir1 2024-04-02T09:10:00.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.627+0000 7f23ee600700 1 -- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 msgr2=0x7f23c806d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:10:00.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.627+0000 7f23ee600700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 0x7f23c806d240 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f23d4000c00 tx=0x7f23d4008030 comp rx=0 tx=0).stop 2024-04-02T09:10:00.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.627+0000 7f23ee600700 1 -- 172.21.15.89:0/3488363449 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23e80d1630 msgr2=0x7f23e81f2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:10:00.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.627+0000 7f23ee600700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23e80d1630 0x7f23e81f2800 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f23d800e9b0 tx=0x7f23d800ecc0 comp rx=0 tx=0).stop 2024-04-02T09:10:00.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.627+0000 7f23ee600700 1 -- 172.21.15.89:0/3488363449 shutdown_connections 2024-04-02T09:10:00.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.627+0000 7f23ee600700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23c806ae00 0x7f23c806d240 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:00.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.627+0000 7f23ee600700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e815dad0 0x7f23e80c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:00.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.627+0000 7f23ee600700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23e80d1630 0x7f23e81f2800 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:00.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.627+0000 7f23ee600700 1 --2- 172.21.15.89:0/3488363449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23e80d1fd0 0x7f23e80c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:00.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.627+0000 7f23ee600700 1 -- 172.21.15.89:0/3488363449 >> 172.21.15.89:0/3488363449 conn(0x7f23e808aff0 msgr2=0x7f23e815deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:10:00.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.628+0000 7f23ee600700 1 -- 172.21.15.89:0/3488363449 shutdown_connections 2024-04-02T09:10:00.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.628+0000 7f23ee600700 1 -- 172.21.15.89:0/3488363449 wait complete. 2024-04-02T09:10:00.645 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-02T09:10:00.645 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-04-02T09:10:00.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.848+0000 7fc539a00700 1 Processor -- start 2024-04-02T09:10:00.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.848+0000 7fc539a00700 1 -- start start 2024-04-02T09:10:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.849+0000 7fc539a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc534155ac0 0x7fc53415c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:10:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.849+0000 7fc539a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc53406c910 0x7fc5340adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:10:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.849+0000 7fc539a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc5340ae4a0 0x7fc53415f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:10:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.849+0000 7fc539a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc53406dd00 con 0x7fc5340ae4a0 2024-04-02T09:10:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.850+0000 7fc539a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc53406d0f0 con 0x7fc53406c910 2024-04-02T09:10:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.850+0000 7fc539a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc53406d3d0 con 0x7fc534155ac0 2024-04-02T09:10:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.850+0000 7fc533400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc5340ae4a0 0x7fc53415f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:10:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.850+0000 7fc533400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc5340ae4a0 0x7fc53415f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51778/0 (socket says 172.21.15.89:51778) 2024-04-02T09:10:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.850+0000 7fc533400700 1 -- 172.21.15.89:0/2271821983 learned_addr learned my addr 172.21.15.89:0/2271821983 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:10:00.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.850+0000 7fc52be00700 1 --2- 172.21.15.89:0/2271821983 >> v2:172.21.15.113:3300/0 conn(0x7fc53406c910 0x7fc5340adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:10:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.850+0000 7fc532a00700 1 --2- 172.21.15.89:0/2271821983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc534155ac0 0x7fc53415c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:10:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.850+0000 7fc533400700 1 -- 172.21.15.89:0/2271821983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc534155ac0 msgr2=0x7fc53415c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:10:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.850+0000 7fc533400700 1 --2- 172.21.15.89:0/2271821983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc534155ac0 0x7fc53415c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.850+0000 7fc533400700 1 -- 172.21.15.89:0/2271821983 >> v2:172.21.15.113:3300/0 conn(0x7fc53406c910 msgr2=0x7fc5340adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:10:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.850+0000 7fc533400700 1 --2- 172.21.15.89:0/2271821983 >> v2:172.21.15.113:3300/0 conn(0x7fc53406c910 0x7fc5340adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.850+0000 7fc533400700 1 -- 172.21.15.89:0/2271821983 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53406d6b0 con 0x7fc5340ae4a0 2024-04-02T09:10:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.850+0000 7fc52be00700 1 --2- 172.21.15.89:0/2271821983 >> v2:172.21.15.113:3300/0 conn(0x7fc53406c910 0x7fc5340adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:10:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.851+0000 7fc533400700 1 --2- 172.21.15.89:0/2271821983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5340ae4a0 0x7fc53415f0e0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fc53406c500 tx=0x7fc52c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbe2c571daa0a6b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:10:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.851+0000 7fc532000700 1 -- 172.21.15.89:0/2271821983 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c00e070 con 0x7fc5340ae4a0 2024-04-02T09:10:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.851+0000 7fc532000700 1 -- 172.21.15.89:0/2271821983 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52c00b910 con 0x7fc5340ae4a0 2024-04-02T09:10:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.851+0000 7fc532000700 1 -- 172.21.15.89:0/2271821983 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c00bcc0 con 0x7fc5340ae4a0 2024-04-02T09:10:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.851+0000 7fc539a00700 1 -- 172.21.15.89:0/2271821983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5340ae4a0 msgr2=0x7fc53415f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:10:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.851+0000 7fc539a00700 1 --2- 172.21.15.89:0/2271821983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5340ae4a0 0x7fc53415f0e0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fc53406c500 tx=0x7fc52c00b330 comp rx=0 tx=0).stop 2024-04-02T09:10:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.851+0000 7fc539a00700 1 -- 172.21.15.89:0/2271821983 shutdown_connections 2024-04-02T09:10:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.851+0000 7fc539a00700 1 --2- 172.21.15.89:0/2271821983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5340ae4a0 0x7fc53415f0e0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.851+0000 7fc539a00700 1 --2- 172.21.15.89:0/2271821983 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc534155ac0 0x7fc53415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.851+0000 7fc539a00700 1 --2- 172.21.15.89:0/2271821983 >> v2:172.21.15.113:3300/0 conn(0x7fc53406c910 0x7fc5340adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.851+0000 7fc539a00700 1 -- 172.21.15.89:0/2271821983 >> 172.21.15.89:0/2271821983 conn(0x7fc5340cf210 msgr2=0x7fc5340cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:10:00.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.852+0000 7fc539a00700 1 -- 172.21.15.89:0/2271821983 shutdown_connections 2024-04-02T09:10:00.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.852+0000 7fc539a00700 1 -- 172.21.15.89:0/2271821983 wait complete. 2024-04-02T09:10:00.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.854+0000 7fc539a00700 1 Processor -- start 2024-04-02T09:10:00.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.854+0000 7fc539a00700 1 -- start start 2024-04-02T09:10:00.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.855+0000 7fc539a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc53406c910 0x7fc534071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:10:00.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.855+0000 7fc539a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5340ae4a0 0x7fc53414d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:10:00.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.855+0000 7fc539a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc534155ac0 0x7fc5341f0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:10:00.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.855+0000 7fc539a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5341510e0 con 0x7fc534155ac0 2024-04-02T09:10:00.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.855+0000 7fc539a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53414aa80 con 0x7fc53406c910 2024-04-02T09:10:00.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.855+0000 7fc539a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc534071970 con 0x7fc5340ae4a0 2024-04-02T09:10:00.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.855+0000 7fc532a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc53406c910 0x7fc534071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:10:00.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.855+0000 7fc533400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc534155ac0 0x7fc5341f0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:10:00.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.855+0000 7fc52be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5340ae4a0 0x7fc53414d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:10:00.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.855+0000 7fc532a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc53406c910 0x7fc534071390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:47668/0 (socket says 172.21.15.89:47668) 2024-04-02T09:10:00.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.855+0000 7fc532a00700 1 -- 172.21.15.89:0/215595479 learned_addr learned my addr 172.21.15.89:0/215595479 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:10:00.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.856+0000 7fc533400700 1 -- 172.21.15.89:0/215595479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5340ae4a0 msgr2=0x7fc53414d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:10:00.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.856+0000 7fc533400700 1 --2- 172.21.15.89:0/215595479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5340ae4a0 0x7fc53414d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:00.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.856+0000 7fc533400700 1 -- 172.21.15.89:0/215595479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc53406c910 msgr2=0x7fc534071390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:10:00.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.856+0000 7fc533400700 1 --2- 172.21.15.89:0/215595479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc53406c910 0x7fc534071390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:00.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.856+0000 7fc533400700 1 -- 172.21.15.89:0/215595479 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c009b30 con 0x7fc534155ac0 2024-04-02T09:10:00.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.856+0000 7fc532a00700 1 --2- 172.21.15.89:0/215595479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc53406c910 0x7fc534071390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:10:00.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.856+0000 7fc533400700 1 --2- 172.21.15.89:0/215595479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc534155ac0 0x7fc5341f0c50 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fc53415cc40 tx=0x7fc52c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:10:00.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.856+0000 7fc52aa00700 1 -- 172.21.15.89:0/215595479 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c0099b0 con 0x7fc534155ac0 2024-04-02T09:10:00.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.856+0000 7fc52aa00700 1 -- 172.21.15.89:0/215595479 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52c0092e0 con 0x7fc534155ac0 2024-04-02T09:10:00.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.856+0000 7fc52aa00700 1 -- 172.21.15.89:0/215595479 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c004470 con 0x7fc534155ac0 2024-04-02T09:10:00.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.856+0000 7fc539a00700 1 -- 172.21.15.89:0/215595479 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc534071c50 con 0x7fc534155ac0 2024-04-02T09:10:00.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.857+0000 7fc539a00700 1 -- 172.21.15.89:0/215595479 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53414da70 con 0x7fc534155ac0 2024-04-02T09:10:00.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.858+0000 7fc539a00700 1 -- 172.21.15.89:0/215595479 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc53414ac50 con 0x7fc534155ac0 2024-04-02T09:10:00.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.858+0000 7fc52aa00700 1 -- 172.21.15.89:0/215595479 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 136) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fc52c0045d0 con 0x7fc534155ac0 2024-04-02T09:10:00.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.859+0000 7fc52aa00700 1 --2- 172.21.15.89:0/215595479 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc51006ab30 0x7fc51006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:10:00.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.859+0000 7fc52aa00700 1 -- 172.21.15.89:0/215595479 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fc52c08a690 con 0x7fc534155ac0 2024-04-02T09:10:00.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.859+0000 7fc532a00700 1 --2- 172.21.15.89:0/215595479 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc51006ab30 0x7fc51006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:10:00.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.860+0000 7fc532a00700 1 --2- 172.21.15.89:0/215595479 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc51006ab30 0x7fc51006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc5240112c0 tx=0x7fc524009210 comp rx=0 tx=0).ready entity=mgr.10818 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:10:00.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:00.864+0000 7fc52aa00700 1 -- 172.21.15.89:0/215595479 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc53414ac50 con 0x7fc534155ac0 2024-04-02T09:10:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.093+0000 7fc539a00700 1 -- 172.21.15.89:0/215595479 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc53406cd90 con 0x7fc51006ab30 2024-04-02T09:10:01.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.162+0000 7fc52aa00700 1 -- 172.21.15.89:0/215595479 <== mgr.10818 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7fc524011c80 con 0x7fc51006ab30 2024-04-02T09:10:01.161 INFO:teuthology.orchestra.run.smithi089.stdout:Retention added to path snap_test_dir1 2024-04-02T09:10:01.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.165+0000 7fc539a00700 1 -- 172.21.15.89:0/215595479 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc51006ab30 msgr2=0x7fc51006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:10:01.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.165+0000 7fc539a00700 1 --2- 172.21.15.89:0/215595479 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc51006ab30 0x7fc51006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc5240112c0 tx=0x7fc524009210 comp rx=0 tx=0).stop 2024-04-02T09:10:01.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.165+0000 7fc539a00700 1 -- 172.21.15.89:0/215595479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc534155ac0 msgr2=0x7fc5341f0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:10:01.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.165+0000 7fc539a00700 1 --2- 172.21.15.89:0/215595479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc534155ac0 0x7fc5341f0c50 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fc53415cc40 tx=0x7fc52c004ad0 comp rx=0 tx=0).stop 2024-04-02T09:10:01.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.165+0000 7fc539a00700 1 -- 172.21.15.89:0/215595479 shutdown_connections 2024-04-02T09:10:01.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.165+0000 7fc539a00700 1 --2- 172.21.15.89:0/215595479 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc51006ab30 0x7fc51006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:01.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.165+0000 7fc539a00700 1 --2- 172.21.15.89:0/215595479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc53406c910 0x7fc534071390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:01.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.165+0000 7fc539a00700 1 --2- 172.21.15.89:0/215595479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc534155ac0 0x7fc5341f0c50 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:01.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.165+0000 7fc539a00700 1 --2- 172.21.15.89:0/215595479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5340ae4a0 0x7fc53414d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:01.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.165+0000 7fc539a00700 1 -- 172.21.15.89:0/215595479 >> 172.21.15.89:0/215595479 conn(0x7fc5340cf210 msgr2=0x7fc53415cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:10:01.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.165+0000 7fc539a00700 1 -- 172.21.15.89:0/215595479 shutdown_connections 2024-04-02T09:10:01.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.165+0000 7fc539a00700 1 -- 172.21.15.89:0/215595479 wait complete. 2024-04-02T09:10:01.182 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-04-02T09:10:01.182 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-02-09_11_00 in ~59s... 2024-04-02T09:10:01.182 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-04-02T09:10:01.182 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-04-02T09:10:01.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.378+0000 7f6233600700 1 Processor -- start 2024-04-02T09:10:01.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.378+0000 7f6233600700 1 -- start start 2024-04-02T09:10:01.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.380+0000 7f6233600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f622c0ce6a0 0x7f622c15dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:10:01.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.380+0000 7f6233600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622c0709c0 con 0x7f622c15d800 2024-04-02T09:10:01.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.380+0000 7f6233600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622c070c80 con 0x7f622c0ce230 2024-04-02T09:10:01.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.380+0000 7f6233600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f622c0b40f0 con 0x7f622c0ce6a0 2024-04-02T09:10:01.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.380+0000 7f6231600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f622c0ce6a0 0x7f622c15dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:10:01.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.380+0000 7f6230c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f622c15d800 0x7f622c0725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58358/0 (socket says 172.21.15.89:58358) 2024-04-02T09:10:01.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.380+0000 7f6230c00700 1 -- 172.21.15.89:0/3493779102 learned_addr learned my addr 172.21.15.89:0/3493779102 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:10:01.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.380+0000 7f6231600700 1 -- 172.21.15.89:0/3493779102 >> v1:172.21.15.113:6789/0 conn(0x7f622c0ce230 legacy=0x7f622c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:10:01.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.380+0000 7f6231600700 1 -- 172.21.15.89:0/3493779102 >> v1:172.21.15.89:6789/0 conn(0x7f622c15d800 legacy=0x7f622c0725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:10:01.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.380+0000 7f6231600700 1 -- 172.21.15.89:0/3493779102 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c070f60 con 0x7f622c0ce6a0 2024-04-02T09:10:01.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.381+0000 7f6231600700 1 --2- 172.21.15.89:0/3493779102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f622c0ce6a0 0x7f622c15dbe0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f6224009d90 tx=0x7f622400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49a1a018fdd0f60 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:10:01.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.381+0000 7f622b400700 1 -- 172.21.15.89:0/3493779102 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622400e070 con 0x7f622c0ce6a0 2024-04-02T09:10:01.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.381+0000 7f622b400700 1 -- 172.21.15.89:0/3493779102 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622400bbd0 con 0x7f622c0ce6a0 2024-04-02T09:10:01.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.381+0000 7f622b400700 1 -- 172.21.15.89:0/3493779102 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6224012430 con 0x7f622c0ce6a0 2024-04-02T09:10:01.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.381+0000 7f6233600700 1 -- 172.21.15.89:0/3493779102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f622c0ce6a0 msgr2=0x7f622c15dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:10:01.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.381+0000 7f6233600700 1 --2- 172.21.15.89:0/3493779102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f622c0ce6a0 0x7f622c15dbe0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f6224009d90 tx=0x7f622400b330 comp rx=0 tx=0).stop 2024-04-02T09:10:01.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.381+0000 7f6233600700 1 -- 172.21.15.89:0/3493779102 shutdown_connections 2024-04-02T09:10:01.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.381+0000 7f6233600700 1 --2- 172.21.15.89:0/3493779102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f622c0ce6a0 0x7f622c15dbe0 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:01.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.381+0000 7f6233600700 1 -- 172.21.15.89:0/3493779102 >> 172.21.15.89:0/3493779102 conn(0x7f622c0b0d70 msgr2=0x7f622c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:10:01.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.381+0000 7f6233600700 1 -- 172.21.15.89:0/3493779102 shutdown_connections 2024-04-02T09:10:01.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.382+0000 7f6233600700 1 -- 172.21.15.89:0/3493779102 wait complete. 2024-04-02T09:10:01.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.383+0000 7f6233600700 1 Processor -- start 2024-04-02T09:10:01.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.383+0000 7f6233600700 1 -- start start 2024-04-02T09:10:01.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.384+0000 7f6233600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f622c0ce230 0x7f622c09c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:10:01.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.384+0000 7f6233600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f622c0ce6a0 0x7f622c0cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:10:01.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.384+0000 7f6233600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f622c15d800 0x7f622c149450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:10:01.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.384+0000 7f6233600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c0acb40 con 0x7f622c0ce6a0 2024-04-02T09:10:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.384+0000 7f6233600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c0ac050 con 0x7f622c15d800 2024-04-02T09:10:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.384+0000 7f6233600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f622c0ac2a0 con 0x7f622c0ce230 2024-04-02T09:10:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.384+0000 7f622be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f622c0ce6a0 0x7f622c0cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:10:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.384+0000 7f622be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f622c0ce6a0 0x7f622c0cb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51810/0 (socket says 172.21.15.89:51810) 2024-04-02T09:10:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.384+0000 7f622be00700 1 -- 172.21.15.89:0/3674138422 learned_addr learned my addr 172.21.15.89:0/3674138422 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:10:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.385+0000 7f6230c00700 1 --2- 172.21.15.89:0/3674138422 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f622c0ce230 0x7f622c09c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:10:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.385+0000 7f6231600700 1 --2- 172.21.15.89:0/3674138422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f622c15d800 0x7f622c149450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:10:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.385+0000 7f622be00700 1 -- 172.21.15.89:0/3674138422 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f622c0ce230 msgr2=0x7f622c09c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:10:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.385+0000 7f622be00700 1 --2- 172.21.15.89:0/3674138422 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f622c0ce230 0x7f622c09c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.385+0000 7f622be00700 1 -- 172.21.15.89:0/3674138422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f622c15d800 msgr2=0x7f622c149450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:10:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.385+0000 7f622be00700 1 --2- 172.21.15.89:0/3674138422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f622c15d800 0x7f622c149450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.385+0000 7f622be00700 1 -- 172.21.15.89:0/3674138422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6224009a70 con 0x7f622c0ce6a0 2024-04-02T09:10:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.385+0000 7f622be00700 1 --2- 172.21.15.89:0/3674138422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f622c0ce6a0 0x7f622c0cb810 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f621800c4e0 tx=0x7f621800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:10:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.385+0000 7f6228c00700 1 -- 172.21.15.89:0/3674138422 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62180052d0 con 0x7f622c0ce6a0 2024-04-02T09:10:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.385+0000 7f6228c00700 1 -- 172.21.15.89:0/3674138422 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6218005430 con 0x7f622c0ce6a0 2024-04-02T09:10:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.385+0000 7f6233600700 1 -- 172.21.15.89:0/3674138422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622c0ac580 con 0x7f622c0ce6a0 2024-04-02T09:10:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.385+0000 7f6228c00700 1 -- 172.21.15.89:0/3674138422 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62180038c0 con 0x7f622c0ce6a0 2024-04-02T09:10:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.385+0000 7f6233600700 1 -- 172.21.15.89:0/3674138422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622c09c8d0 con 0x7f622c0ce6a0 2024-04-02T09:10:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.386+0000 7f6233600700 1 -- 172.21.15.89:0/3674138422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f622c0b1150 con 0x7f622c0ce6a0 2024-04-02T09:10:01.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.387+0000 7f6228c00700 1 -- 172.21.15.89:0/3674138422 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 136) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f6218003a20 con 0x7f622c0ce6a0 2024-04-02T09:10:01.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.388+0000 7f6228c00700 1 --2- 172.21.15.89:0/3674138422 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f620c06ab30 0x7f620c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:10:01.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.388+0000 7f6228c00700 1 -- 172.21.15.89:0/3674138422 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f6218088830 con 0x7f622c0ce6a0 2024-04-02T09:10:01.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.388+0000 7f6230c00700 1 --2- 172.21.15.89:0/3674138422 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f620c06ab30 0x7f620c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:10:01.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.389+0000 7f6230c00700 1 --2- 172.21.15.89:0/3674138422 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f620c06ab30 0x7f620c06cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6220005910 tx=0x7f6220009040 comp rx=0 tx=0).ready entity=mgr.10818 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:10:01.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.392+0000 7f6228c00700 1 -- 172.21.15.89:0/3674138422 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6218053960 con 0x7f622c0ce6a0 2024-04-02T09:10:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.549+0000 7f6228c00700 1 -- 172.21.15.89:0/3674138422 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 137) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f6218053d30 con 0x7f622c0ce6a0 2024-04-02T09:10:01.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.633+0000 7f6233600700 1 -- 172.21.15.89:0/3674138422 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f622c0cd9e0 con 0x7f620c06ab30 2024-04-02T09:10:01.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.641+0000 7f6228c00700 1 -- 172.21.15.89:0/3674138422 <== mgr.10818 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f6220001340 con 0x7f620c06ab30 2024-04-02T09:10:01.639 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:10:01.639 INFO:teuthology.orchestra.run.smithi089.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-04-02T09:10:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.643+0000 7f6233600700 1 -- 172.21.15.89:0/3674138422 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f620c06ab30 msgr2=0x7f620c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:10:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.643+0000 7f6233600700 1 --2- 172.21.15.89:0/3674138422 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f620c06ab30 0x7f620c06cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6220005910 tx=0x7f6220009040 comp rx=0 tx=0).stop 2024-04-02T09:10:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.643+0000 7f6233600700 1 -- 172.21.15.89:0/3674138422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f622c0ce6a0 msgr2=0x7f622c0cb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:10:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.643+0000 7f6233600700 1 --2- 172.21.15.89:0/3674138422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f622c0ce6a0 0x7f622c0cb810 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f621800c4e0 tx=0x7f621800c8a0 comp rx=0 tx=0).stop 2024-04-02T09:10:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.643+0000 7f6233600700 1 -- 172.21.15.89:0/3674138422 shutdown_connections 2024-04-02T09:10:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.643+0000 7f6233600700 1 --2- 172.21.15.89:0/3674138422 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f620c06ab30 0x7f620c06cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.643+0000 7f6233600700 1 --2- 172.21.15.89:0/3674138422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f622c15d800 0x7f622c149450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.643+0000 7f6233600700 1 --2- 172.21.15.89:0/3674138422 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f622c0ce230 0x7f622c09c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.643+0000 7f6233600700 1 --2- 172.21.15.89:0/3674138422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f622c0ce6a0 0x7f622c0cb810 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:10:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.644+0000 7f6233600700 1 -- 172.21.15.89:0/3674138422 >> 172.21.15.89:0/3674138422 conn(0x7f622c0b0d70 msgr2=0x7f622c0cf640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:10:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.644+0000 7f6233600700 1 -- 172.21.15.89:0/3674138422 shutdown_connections 2024-04-02T09:10:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:10:01.644+0000 7f6233600700 1 -- 172.21.15.89:0/3674138422 wait complete. 2024-04-02T09:10:01.659 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-04-02T09:10:01.660 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-04-02T09:10:01.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:01.660 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:01.706 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:01.706 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:01.706 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:02.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:02.708 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:02.759 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:02.760 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:02.760 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:03.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:03.761 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:03.811 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:03.811 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:03.812 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:04.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:04.812 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:04.864 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:04.864 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:04.865 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:05.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:05.866 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:05.917 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:05.917 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:05.917 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:06.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:06.918 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:06.971 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:06.971 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:06.971 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:07.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:07.972 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:08.024 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:08.025 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:08.025 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:09.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:09.026 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:09.077 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:09.077 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:09.077 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:10.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:10.078 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:10.129 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:10.130 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:10.130 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:11.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:11.130 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:11.182 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:11.182 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:11.182 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:11.693 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:10:11.698 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:10:12.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:12.183 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:12.234 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:12.235 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:12.235 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:13.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:13.236 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:13.287 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:13.287 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:13.287 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:14.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:14.289 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:14.340 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:14.340 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:14.340 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:15.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:15.341 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:15.393 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:15.393 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:15.393 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:16.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:16.395 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:16.446 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:16.446 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:16.446 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:17.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:17.447 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:17.499 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:17.499 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:17.499 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:18.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:18.500 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:18.552 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:18.552 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:18.552 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:19.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:19.553 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:19.605 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:19.605 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:19.605 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:20.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:20.606 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:20.657 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:20.657 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:20.657 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:21.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:21.659 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:21.711 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:21.712 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:21.712 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:22.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:22.713 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:22.764 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:22.764 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:22.765 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:23.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:23.766 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:23.817 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:23.817 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:23.817 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:24.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:24.819 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:24.870 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:24.871 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:24.871 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:25.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:25.875 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:25.926 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:25.927 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:25.927 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:26.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:26.928 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:26.979 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:26.979 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:26.979 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:27.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:27.981 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:28.033 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:28.033 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:28.033 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:29.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:29.034 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:29.085 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:29.085 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:29.086 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:30.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:30.087 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:30.138 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:30.138 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:30.139 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:31.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:31.140 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:31.191 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:31.191 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:31.191 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:32.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:32.193 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:32.245 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:32.245 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:32.245 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:33.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:33.246 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:33.297 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:33.297 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:33.297 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:34.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:34.298 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:34.349 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:34.349 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:34.349 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:35.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:35.350 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:35.402 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:35.403 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:35.403 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:36.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:36.404 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:36.455 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:36.455 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:36.455 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:37.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:37.456 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:37.507 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:37.508 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:37.508 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:38.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:38.509 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:38.562 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:38.562 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:38.562 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:39.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:39.563 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:39.614 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:39.615 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:39.615 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:40.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:40.617 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:40.669 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:40.669 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:40.669 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:41.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:41.670 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:41.722 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:41.722 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:41.723 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:42.042 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:10:42.047 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:10:42.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:42.724 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:42.775 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:42.775 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:42.776 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:43.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:43.776 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:43.828 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:43.828 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:43.828 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:44.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:44.829 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:44.881 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:44.881 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:44.881 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:45.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:45.882 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:45.934 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:45.934 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:45.934 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:46.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:46.935 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:46.986 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:46.986 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:46.986 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:47.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:47.988 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:48.039 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:48.039 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:48.039 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:49.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:49.041 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:49.092 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:49.092 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:49.092 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:50.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:50.094 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:50.145 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:50.145 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:50.145 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:51.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:51.146 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:51.197 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:51.197 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:51.197 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:52.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:52.198 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:52.249 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:52.249 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:52.249 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:53.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:53.250 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:53.303 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:53.303 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:53.303 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:54.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:54.304 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:54.355 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:54.355 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:54.355 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:55.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:55.356 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:55.407 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:55.408 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:55.408 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:56.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:56.409 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:56.460 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:56.460 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:56.460 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:57.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:57.461 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:57.513 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:57.513 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:57.513 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:58.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:58.515 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:58.567 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:58.567 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:58.567 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:10:59.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:10:59.568 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:10:59.620 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-02T09:10:59.620 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:10:59.620 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:00.174 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:11:00.176+0000 7f898fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:11:00.175 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:11:00.176+0000 7f898fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:11:00.175 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:11:00.176+0000 7f898fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:11:00.175 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:11:00.176+0000 7f898fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:11:00.175 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:11:00.176+0000 7f898fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:11:00.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:00.621 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:00.671 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:00.672 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:00.672 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:00.672 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:00.673 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-04-02-09_11_00_UTC'] 2024-04-02T09:11:00.673 DEBUG:tasks.cephfs.test_snap_schedules:exec=1712049001.1824121, now = 1712049060.673123, timo = 59 2024-04-02T09:11:01.673 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2024-04-02-09_11_00 in ~119s... 2024-04-02T09:11:01.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:01.674 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:01.725 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:01.726 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:01.726 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:01.726 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:02.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:02.728 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:02.778 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:02.779 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:02.779 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:02.779 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:03.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:03.780 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:03.830 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:03.831 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:03.831 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:03.831 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:04.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:04.832 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:04.883 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:04.884 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:04.884 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:04.884 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:05.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:05.886 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:05.937 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:05.938 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:05.938 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:05.938 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:06.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:06.939 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:06.989 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:06.990 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:06.990 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:06.990 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:07.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:07.992 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:08.042 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:08.043 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:08.043 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:08.043 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:09.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:09.044 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:09.095 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:09.096 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:09.096 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:09.096 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:10.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:10.098 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:10.148 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:10.149 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:10.149 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:10.149 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:11.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:11.150 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:11.200 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:11.201 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:11.201 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:11.201 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:12.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:11:12.156 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:11:12.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:12.202 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:12.250 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:12.251 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:12.251 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:12.251 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:13.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:13.252 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:13.304 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:13.305 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:13.305 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:13.306 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:14.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:14.307 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:14.357 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:14.358 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:14.358 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:14.358 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:15.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:15.360 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:15.410 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:15.411 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:15.411 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:15.411 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:16.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:16.412 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:16.463 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:16.464 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:16.464 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:16.464 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:17.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:17.465 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:17.515 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:17.516 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:17.516 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:17.516 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:18.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:18.517 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:18.568 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:18.569 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:18.569 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:18.569 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:19.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:19.571 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:19.620 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:19.621 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:19.621 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:19.621 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:20.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:20.622 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:20.672 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:20.673 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:20.673 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:20.673 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:21.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:21.675 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:21.725 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:21.726 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:21.726 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:21.726 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:22.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:22.728 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:22.778 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:22.779 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:22.779 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:22.779 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:23.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:23.781 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:23.831 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:23.832 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:23.832 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:23.832 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:24.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:24.833 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:24.884 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:24.885 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:24.885 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:24.885 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:25.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:25.886 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:25.937 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:25.938 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:25.938 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:25.938 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:26.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:26.940 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:26.988 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:26.990 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:26.990 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:26.990 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:27.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:27.991 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:28.041 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:28.042 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:28.042 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:28.042 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:29.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:29.043 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:29.094 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:29.095 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:29.095 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:29.095 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:30.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:30.096 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:30.147 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:30.148 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:30.149 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:30.149 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:31.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:31.149 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:31.199 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:31.201 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:31.201 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:31.201 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:32.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:32.202 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:32.252 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:32.253 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:32.253 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:32.253 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:33.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:33.254 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:33.304 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:33.305 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:33.305 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:33.305 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:34.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:34.307 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:34.357 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:34.358 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:34.358 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:34.358 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:35.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:35.359 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:35.410 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:35.411 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:35.411 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:35.411 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:36.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:36.412 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:36.462 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:36.463 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:36.463 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:36.463 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:37.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:37.464 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:37.514 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:37.515 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:37.515 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:37.515 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:38.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:38.516 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:38.567 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:38.568 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:38.568 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:38.568 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:39.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:39.569 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:39.618 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:39.619 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:39.619 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:39.619 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:40.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:40.621 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:40.671 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:40.672 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:40.672 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:40.672 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:41.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:41.673 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:41.724 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:41.725 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:41.725 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:41.725 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:42.323 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:11:42.328 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:11:42.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:42.727 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:42.777 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:42.778 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:42.778 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:42.778 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:43.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:43.779 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:43.829 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:43.831 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:43.831 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:43.831 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:44.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:44.832 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:44.882 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:44.883 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:44.883 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:44.883 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:45.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:45.884 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:45.934 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:45.935 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:45.935 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:45.935 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:46.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:46.936 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:46.987 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:46.988 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:46.988 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:46.988 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:47.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:47.990 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:48.040 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:48.041 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:48.041 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:48.041 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:49.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:49.043 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:49.093 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:49.094 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:49.094 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:49.095 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:50.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:50.096 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:50.146 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:50.147 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:50.147 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:50.148 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:51.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:51.149 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:51.199 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:51.200 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:51.200 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:51.200 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:52.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:52.201 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:52.251 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:52.252 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:52.252 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:52.253 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:53.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:53.253 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:53.303 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:53.305 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:53.305 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:53.305 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:54.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:54.306 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:54.357 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:54.358 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:54.358 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:54.358 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:55.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:55.359 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:55.409 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:55.410 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:55.410 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:55.410 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:56.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:56.411 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:56.462 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:56.463 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:56.463 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:56.463 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:57.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:57.465 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:57.516 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:57.517 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:57.517 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:57.517 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:58.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:58.518 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:58.568 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:58.569 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:58.569 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:58.569 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:11:59.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:11:59.570 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:11:59.621 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_11_00_UTC 2024-04-02T09:11:59.622 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:11:59.622 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-02T09:11:59.622 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-02T09:12:00.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:00.623 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:12:00.674 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_12_00_UTC 2024-04-02T09:12:00.675 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-02-09_12_00_UTC'} 2024-04-02T09:12:00.675 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-02-09_12_00_UTC'} 2024-04-02T09:12:00.675 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2024-04-02-09_11_00_UTC'} 2024-04-02T09:12:00.675 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2024-04-02-09_11_00_UTC'] 2024-04-02T09:12:00.675 DEBUG:tasks.cephfs.test_snap_schedules:exec=1712049001.1824121, now = 1712049120.6758578, timo = 119 2024-04-02T09:12:01.676 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-02T09:12:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.881+0000 7f15e9400700 1 Processor -- start 2024-04-02T09:12:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.881+0000 7f15e9400700 1 -- start start 2024-04-02T09:12:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.883+0000 7f15e9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15e415a5c0 0x7f15e40709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.883+0000 7f15e9400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f15e415dfe0 0x7f15e40ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.883+0000 7f15e9400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e40cb310 con 0x7f15e40ce670 2024-04-02T09:12:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.883+0000 7f15e9400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e40ceb00 con 0x7f15e415dfe0 2024-04-02T09:12:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.883+0000 7f15e9400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e4070f00 con 0x7f15e415a5c0 2024-04-02T09:12:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.883+0000 7f15e3200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f15e40ce670 0x7f15e40b2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32834/0 (socket says 172.21.15.89:32834) 2024-04-02T09:12:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.883+0000 7f15e1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f15e415dfe0 0x7f15e40ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.883+0000 7f15e3200700 1 -- 172.21.15.89:0/1502694362 learned_addr learned my addr 172.21.15.89:0/1502694362 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.883+0000 7f15e2800700 1 --2- 172.21.15.89:0/1502694362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15e415a5c0 0x7f15e40709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:01.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.883+0000 7f15e1400700 1 -- 172.21.15.89:0/1502694362 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894772499 0 0) 0x7f15e40cb310 con 0x7f15e40ce670 2024-04-02T09:12:01.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.883+0000 7f15e1e00700 1 -- 172.21.15.89:0/1502694362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15e415a5c0 msgr2=0x7f15e40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:01.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.883+0000 7f15e1e00700 1 --2- 172.21.15.89:0/1502694362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15e415a5c0 0x7f15e40709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.883+0000 7f15e1e00700 1 -- 172.21.15.89:0/1502694362 >> v1:172.21.15.89:6789/0 conn(0x7f15e40ce670 legacy=0x7f15e40b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.883+0000 7f15e1e00700 1 -- 172.21.15.89:0/1502694362 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e40cca90 con 0x7f15e415dfe0 2024-04-02T09:12:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.884+0000 7f15e1e00700 1 --2- 172.21.15.89:0/1502694362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15e415dfe0 0x7f15e40ce130 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f15d0009840 tx=0x7f15d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ec82a0bf37e3ab server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.884+0000 7f15e1400700 1 -- 172.21.15.89:0/1502694362 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0011830 con 0x7f15e415dfe0 2024-04-02T09:12:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.884+0000 7f15e1400700 1 -- 172.21.15.89:0/1502694362 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d0011990 con 0x7f15e415dfe0 2024-04-02T09:12:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.884+0000 7f15e1400700 1 -- 172.21.15.89:0/1502694362 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0011d20 con 0x7f15e415dfe0 2024-04-02T09:12:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.884+0000 7f15e9400700 1 -- 172.21.15.89:0/1502694362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15e415dfe0 msgr2=0x7f15e40ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.884+0000 7f15e9400700 1 --2- 172.21.15.89:0/1502694362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15e415dfe0 0x7f15e40ce130 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f15d0009840 tx=0x7f15d0009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.884+0000 7f15e9400700 1 -- 172.21.15.89:0/1502694362 shutdown_connections 2024-04-02T09:12:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.884+0000 7f15e9400700 1 --2- 172.21.15.89:0/1502694362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15e415a5c0 0x7f15e40709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.884+0000 7f15e9400700 1 --2- 172.21.15.89:0/1502694362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15e415dfe0 0x7f15e40ce130 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.884+0000 7f15e9400700 1 -- 172.21.15.89:0/1502694362 >> 172.21.15.89:0/1502694362 conn(0x7f15e40b0d70 msgr2=0x7f15e40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.885+0000 7f15e9400700 1 -- 172.21.15.89:0/1502694362 shutdown_connections 2024-04-02T09:12:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.885+0000 7f15e9400700 1 -- 172.21.15.89:0/1502694362 wait complete. 2024-04-02T09:12:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.886+0000 7f15e9400700 1 Processor -- start 2024-04-02T09:12:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.886+0000 7f15e9400700 1 -- start start 2024-04-02T09:12:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.887+0000 7f15e9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15e40ce670 0x7f15e40cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.887+0000 7f15e9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15e415a5c0 0x7f15e41f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.887+0000 7f15e9400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15e415dfe0 0x7f15e415cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.887+0000 7f15e9400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e40cb110 con 0x7f15e40ce670 2024-04-02T09:12:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.887+0000 7f15e9400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e41f4090 con 0x7f15e415dfe0 2024-04-02T09:12:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.887+0000 7f15e9400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e41f6c40 con 0x7f15e415a5c0 2024-04-02T09:12:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.887+0000 7f15e2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15e40ce670 0x7f15e40cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.887+0000 7f15e2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15e40ce670 0x7f15e40cabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33924/0 (socket says 172.21.15.89:33924) 2024-04-02T09:12:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.887+0000 7f15e2800700 1 -- 172.21.15.89:0/484450979 learned_addr learned my addr 172.21.15.89:0/484450979 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.888+0000 7f15e2800700 1 -- 172.21.15.89:0/484450979 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15e415a5c0 msgr2=0x7f15e41f3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.888+0000 7f15e2800700 1 --2- 172.21.15.89:0/484450979 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15e415a5c0 0x7f15e41f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.888+0000 7f15e2800700 1 -- 172.21.15.89:0/484450979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15e415dfe0 msgr2=0x7f15e415cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.888+0000 7f15e2800700 1 --2- 172.21.15.89:0/484450979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15e415dfe0 0x7f15e415cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.888+0000 7f15e2800700 1 -- 172.21.15.89:0/484450979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d00094a0 con 0x7f15e40ce670 2024-04-02T09:12:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.888+0000 7f15e2800700 1 --2- 172.21.15.89:0/484450979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15e40ce670 0x7f15e40cabd0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f15d400e9e0 tx=0x7f15d400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.888+0000 7f15daa00700 1 -- 172.21.15.89:0/484450979 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d4009710 con 0x7f15e40ce670 2024-04-02T09:12:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.888+0000 7f15daa00700 1 -- 172.21.15.89:0/484450979 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d400c7f0 con 0x7f15e40ce670 2024-04-02T09:12:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.888+0000 7f15e9400700 1 -- 172.21.15.89:0/484450979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e41f6f20 con 0x7f15e40ce670 2024-04-02T09:12:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.888+0000 7f15daa00700 1 -- 172.21.15.89:0/484450979 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d400cb60 con 0x7f15e40ce670 2024-04-02T09:12:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.888+0000 7f15e9400700 1 -- 172.21.15.89:0/484450979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e41f74e0 con 0x7f15e40ce670 2024-04-02T09:12:01.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.889+0000 7f15e9400700 1 -- 172.21.15.89:0/484450979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e4067720 con 0x7f15e40ce670 2024-04-02T09:12:01.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.890+0000 7f15daa00700 1 -- 172.21.15.89:0/484450979 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 140) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f15d400ccc0 con 0x7f15e40ce670 2024-04-02T09:12:01.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.891+0000 7f15daa00700 1 --2- 172.21.15.89:0/484450979 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f15c406ae00 0x7f15c406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:01.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.891+0000 7f15daa00700 1 -- 172.21.15.89:0/484450979 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f15d4089860 con 0x7f15e40ce670 2024-04-02T09:12:01.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.891+0000 7f15e1e00700 1 --2- 172.21.15.89:0/484450979 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f15c406ae00 0x7f15c406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:01.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.892+0000 7f15e1e00700 1 --2- 172.21.15.89:0/484450979 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f15c406ae00 0x7f15c406d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f15e40c8b00 tx=0x7f15d0010000 comp rx=0 tx=0).ready entity=mgr.10818 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:01.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:01.896+0000 7f15daa00700 1 -- 172.21.15.89:0/484450979 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15d4054670 con 0x7f15e40ce670 2024-04-02T09:12:02.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.124+0000 7f15e9400700 1 -- 172.21.15.89:0/484450979 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f15e4071760 con 0x7f15c406ae00 2024-04-02T09:12:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.208+0000 7f15daa00700 1 -- 172.21.15.89:0/484450979 <== mgr.10818 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f15d001f070 con 0x7f15c406ae00 2024-04-02T09:12:02.210 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path snap_test_dir1 2024-04-02T09:12:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.210+0000 7f15e9400700 1 -- 172.21.15.89:0/484450979 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f15c406ae00 msgr2=0x7f15c406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.210+0000 7f15e9400700 1 --2- 172.21.15.89:0/484450979 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f15c406ae00 0x7f15c406d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f15e40c8b00 tx=0x7f15d0010000 comp rx=0 tx=0).stop 2024-04-02T09:12:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.210+0000 7f15e9400700 1 -- 172.21.15.89:0/484450979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15e40ce670 msgr2=0x7f15e40cabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.211+0000 7f15e9400700 1 --2- 172.21.15.89:0/484450979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15e40ce670 0x7f15e40cabd0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f15d400e9e0 tx=0x7f15d400eda0 comp rx=0 tx=0).stop 2024-04-02T09:12:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.211+0000 7f15e9400700 1 -- 172.21.15.89:0/484450979 shutdown_connections 2024-04-02T09:12:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.211+0000 7f15e9400700 1 --2- 172.21.15.89:0/484450979 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f15c406ae00 0x7f15c406d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.211+0000 7f15e9400700 1 --2- 172.21.15.89:0/484450979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15e415dfe0 0x7f15e415cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.211+0000 7f15e9400700 1 --2- 172.21.15.89:0/484450979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15e40ce670 0x7f15e40cabd0 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.211+0000 7f15e9400700 1 --2- 172.21.15.89:0/484450979 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15e415a5c0 0x7f15e41f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.211+0000 7f15e9400700 1 -- 172.21.15.89:0/484450979 >> 172.21.15.89:0/484450979 conn(0x7f15e40b0d70 msgr2=0x7f15e40aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.211+0000 7f15e9400700 1 -- 172.21.15.89:0/484450979 shutdown_connections 2024-04-02T09:12:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.211+0000 7f15e9400700 1 -- 172.21.15.89:0/484450979 wait complete. 2024-04-02T09:12:02.228 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-02T09:12:02.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:02.229 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:12:02.273 INFO:teuthology.orchestra.run.smithi089.stdout:scheduled-2024-04-02-09_12_00_UTC 2024-04-02T09:12:02.274 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-02-09_12_00_UTC 2024-04-02T09:12:02.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:02.274 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-02-09_12_00_UTC) 2024-04-02T09:12:02.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:02.365 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-02T09:12:02.413 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T09:12:02.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.606+0000 7fef8bc00700 1 Processor -- start 2024-04-02T09:12:02.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.606+0000 7fef8bc00700 1 -- start start 2024-04-02T09:12:02.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.607+0000 7fef8bc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef841611b0 0x7fef8415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:02.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef8bc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fef8415f4d0 0x7fef840ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:02.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef8bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef840d21f0 0x7fef840cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:02.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef8bc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef840cf180 con 0x7fef841611b0 2024-04-02T09:12:02.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef8bc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef840ce270 con 0x7fef8415f4d0 2024-04-02T09:12:02.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef8bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef8415b600 con 0x7fef840d21f0 2024-04-02T09:12:02.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef89200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef841611b0 0x7fef8415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:02.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef89200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef841611b0 0x7fef8415ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33936/0 (socket says 172.21.15.89:33936) 2024-04-02T09:12:02.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef89200700 1 -- 172.21.15.89:0/1723339961 learned_addr learned my addr 172.21.15.89:0/1723339961 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:02.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef83e00700 1 --2- 172.21.15.89:0/1723339961 >> v2:172.21.15.113:3300/0 conn(0x7fef8415f4d0 0x7fef840ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:02.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef89c00700 1 --2- 172.21.15.89:0/1723339961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef840d21f0 0x7fef840cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:02.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef89200700 1 -- 172.21.15.89:0/1723339961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef840d21f0 msgr2=0x7fef840cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:02.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef89200700 1 --2- 172.21.15.89:0/1723339961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef840d21f0 0x7fef840cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:02.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef89200700 1 -- 172.21.15.89:0/1723339961 >> v2:172.21.15.113:3300/0 conn(0x7fef8415f4d0 msgr2=0x7fef840ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:02.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef89200700 1 --2- 172.21.15.89:0/1723339961 >> v2:172.21.15.113:3300/0 conn(0x7fef8415f4d0 0x7fef840ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:02.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef89200700 1 -- 172.21.15.89:0/1723339961 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef840d2760 con 0x7fef841611b0 2024-04-02T09:12:02.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.608+0000 7fef83e00700 1 --2- 172.21.15.89:0/1723339961 >> v2:172.21.15.113:3300/0 conn(0x7fef8415f4d0 0x7fef840ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:02.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.609+0000 7fef89200700 1 --2- 172.21.15.89:0/1723339961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef841611b0 0x7fef8415ef90 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fef8408c030 tx=0x7fef78009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abffca999cb7b3d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:02.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.609+0000 7fef83400700 1 -- 172.21.15.89:0/1723339961 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7800b870 con 0x7fef841611b0 2024-04-02T09:12:02.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.609+0000 7fef83400700 1 -- 172.21.15.89:0/1723339961 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7800b9d0 con 0x7fef841611b0 2024-04-02T09:12:02.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.609+0000 7fef83400700 1 -- 172.21.15.89:0/1723339961 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7800bd60 con 0x7fef841611b0 2024-04-02T09:12:02.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.609+0000 7fef8bc00700 1 -- 172.21.15.89:0/1723339961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef841611b0 msgr2=0x7fef8415ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:02.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.609+0000 7fef8bc00700 1 --2- 172.21.15.89:0/1723339961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef841611b0 0x7fef8415ef90 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fef8408c030 tx=0x7fef78009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:02.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.609+0000 7fef8bc00700 1 -- 172.21.15.89:0/1723339961 shutdown_connections 2024-04-02T09:12:02.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.609+0000 7fef8bc00700 1 --2- 172.21.15.89:0/1723339961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef841611b0 0x7fef8415ef90 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:02.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.609+0000 7fef8bc00700 1 --2- 172.21.15.89:0/1723339961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef840d21f0 0x7fef840cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:02.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.609+0000 7fef8bc00700 1 --2- 172.21.15.89:0/1723339961 >> v2:172.21.15.113:3300/0 conn(0x7fef8415f4d0 0x7fef840ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:02.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.609+0000 7fef8bc00700 1 -- 172.21.15.89:0/1723339961 >> 172.21.15.89:0/1723339961 conn(0x7fef8408aff0 msgr2=0x7fef840d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:02.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.610+0000 7fef8bc00700 1 -- 172.21.15.89:0/1723339961 shutdown_connections 2024-04-02T09:12:02.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.610+0000 7fef8bc00700 1 -- 172.21.15.89:0/1723339961 wait complete. 2024-04-02T09:12:02.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.611+0000 7fef8bc00700 1 Processor -- start 2024-04-02T09:12:02.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.611+0000 7fef8bc00700 1 -- start start 2024-04-02T09:12:02.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.612+0000 7fef8bc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef840d21f0 0x7fef840a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:02.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.612+0000 7fef8bc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8415f4d0 0x7fef8415a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:02.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.612+0000 7fef8bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef841611b0 0x7fef8409ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:02.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.612+0000 7fef8bc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8415aef0 con 0x7fef8415f4d0 2024-04-02T09:12:02.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.612+0000 7fef8bc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8409fa10 con 0x7fef840d21f0 2024-04-02T09:12:02.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.612+0000 7fef8bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef8409fcf0 con 0x7fef841611b0 2024-04-02T09:12:02.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.612+0000 7fef83e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8415f4d0 0x7fef8415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:02.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.613+0000 7fef83e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8415f4d0 0x7fef8415a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33946/0 (socket says 172.21.15.89:33946) 2024-04-02T09:12:02.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.613+0000 7fef83e00700 1 -- 172.21.15.89:0/4181928107 learned_addr learned my addr 172.21.15.89:0/4181928107 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:02.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.613+0000 7fef89200700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef840d21f0 0x7fef840a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:02.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.613+0000 7fef83e00700 1 -- 172.21.15.89:0/4181928107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef841611b0 msgr2=0x7fef8409ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:02.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.613+0000 7fef89c00700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef841611b0 0x7fef8409ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:02.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.613+0000 7fef83e00700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef841611b0 0x7fef8409ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:02.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.613+0000 7fef83e00700 1 -- 172.21.15.89:0/4181928107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef840d21f0 msgr2=0x7fef840a0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:02.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.613+0000 7fef83e00700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef840d21f0 0x7fef840a0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:02.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.613+0000 7fef83e00700 1 -- 172.21.15.89:0/4181928107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef78009550 con 0x7fef8415f4d0 2024-04-02T09:12:02.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.613+0000 7fef89200700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef840d21f0 0x7fef840a0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:02.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.613+0000 7fef89c00700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef841611b0 0x7fef8409ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:02.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.613+0000 7fef83e00700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8415f4d0 0x7fef8415a380 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fef7000e7e0 tx=0x7fef7000eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:02.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.613+0000 7fef80c00700 1 -- 172.21.15.89:0/4181928107 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef70013610 con 0x7fef8415f4d0 2024-04-02T09:12:02.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.614+0000 7fef80c00700 1 -- 172.21.15.89:0/4181928107 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef70013770 con 0x7fef8415f4d0 2024-04-02T09:12:02.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.614+0000 7fef80c00700 1 -- 172.21.15.89:0/4181928107 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef70013b00 con 0x7fef8415f4d0 2024-04-02T09:12:02.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.614+0000 7fef8bc00700 1 -- 172.21.15.89:0/4181928107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef8409ffd0 con 0x7fef8415f4d0 2024-04-02T09:12:02.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.614+0000 7fef8bc00700 1 -- 172.21.15.89:0/4181928107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef840a0590 con 0x7fef8415f4d0 2024-04-02T09:12:02.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.615+0000 7fef8bc00700 1 -- 172.21.15.89:0/4181928107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef84067720 con 0x7fef8415f4d0 2024-04-02T09:12:02.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.616+0000 7fef80c00700 1 -- 172.21.15.89:0/4181928107 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 140) v1 ==== 89268+0+0 (secure 0 0 0) 0x7fef70013c60 con 0x7fef8415f4d0 2024-04-02T09:12:02.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.616+0000 7fef80c00700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef6406ae00 0x7fef6406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:02.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.617+0000 7fef80c00700 1 -- 172.21.15.89:0/4181928107 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6559+0+0 (secure 0 0 0) 0x7fef700895d0 con 0x7fef8415f4d0 2024-04-02T09:12:02.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.617+0000 7fef89200700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef6406ae00 0x7fef6406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:02.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.617+0000 7fef89200700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef6406ae00 0x7fef6406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fef78009130 tx=0x7fef78008030 comp rx=0 tx=0).ready entity=mgr.10818 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:02.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.621+0000 7fef80c00700 1 -- 172.21.15.89:0/4181928107 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fef70054240 con 0x7fef8415f4d0 2024-04-02T09:12:02.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:02.877+0000 7fef8bc00700 1 -- 172.21.15.89:0/4181928107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fef840b94b0 con 0x7fef8415f4d0 2024-04-02T09:12:03.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:03.646+0000 7fef80c00700 1 -- 172.21.15.89:0/4181928107 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fef700877a0 con 0x7fef8415f4d0 2024-04-02T09:12:03.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:03.647+0000 7fef80c00700 1 -- 172.21.15.89:0/4181928107 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v141) v1 ==== 93+0+0 (secure 0 0 0) 0x7fef70084910 con 0x7fef8415f4d0 2024-04-02T09:12:03.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:03.649+0000 7fef8bc00700 1 -- 172.21.15.89:0/4181928107 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef6406ae00 msgr2=0x7fef6406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:03.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:03.649+0000 7fef8bc00700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef6406ae00 0x7fef6406d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fef78009130 tx=0x7fef78008030 comp rx=0 tx=0).stop 2024-04-02T09:12:03.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:03.650+0000 7fef8bc00700 1 -- 172.21.15.89:0/4181928107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8415f4d0 msgr2=0x7fef8415a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:03.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:03.650+0000 7fef8bc00700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8415f4d0 0x7fef8415a380 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fef7000e7e0 tx=0x7fef7000eba0 comp rx=0 tx=0).stop 2024-04-02T09:12:03.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:03.650+0000 7fef8bc00700 1 -- 172.21.15.89:0/4181928107 shutdown_connections 2024-04-02T09:12:03.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:03.650+0000 7fef8bc00700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef6406ae00 0x7fef6406d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:03.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:03.650+0000 7fef8bc00700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef840d21f0 0x7fef840a0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:03.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:03.650+0000 7fef8bc00700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef841611b0 0x7fef8409ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:03.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:03.650+0000 7fef8bc00700 1 --2- 172.21.15.89:0/4181928107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8415f4d0 0x7fef8415a380 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:03.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:03.650+0000 7fef8bc00700 1 -- 172.21.15.89:0/4181928107 >> 172.21.15.89:0/4181928107 conn(0x7fef8408aff0 msgr2=0x7fef8415da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:03.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:03.650+0000 7fef8bc00700 1 -- 172.21.15.89:0/4181928107 shutdown_connections 2024-04-02T09:12:03.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:03.650+0000 7fef8bc00700 1 -- 172.21.15.89:0/4181928107 wait complete. 2024-04-02T09:12:03.668 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T09:12:03.669 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:12:03.755 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:12:03.840 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T09:12:03.840 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:12:03.840 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:12:03.867 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:03.869+0000 7f7821bc7240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:12:03.875 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:03.876+0000 7f0ca3caa240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:12:03.886 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:12:03.886 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:12:03.972 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:12:04.053 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:04.054+0000 7f7821bc7240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:12:04.064 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:04.065+0000 7f0ca3caa240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:12:04.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.234+0000 7f113d800700 1 Processor -- start 2024-04-02T09:12:04.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.235+0000 7f113d800700 1 -- start start 2024-04-02T09:12:04.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.236+0000 7f113d800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f113815d710 0x7f11380d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:04.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.237+0000 7f113d800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f113809aba0 0x7f11380d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:04.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.237+0000 7f113d800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113806bb40 con 0x7f113806bf40 2024-04-02T09:12:04.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.237+0000 7f113d800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1138099b00 con 0x7f113815d710 2024-04-02T09:12:04.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.237+0000 7f113d800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1138099de0 con 0x7f113809aba0 2024-04-02T09:12:04.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.237+0000 7f1136a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f113815d710 0x7f11380d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:04.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.237+0000 7f1136a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f113815d710 0x7f11380d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33964/0 (socket says 172.21.15.89:33964) 2024-04-02T09:12:04.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.237+0000 7f1136a00700 1 -- 172.21.15.89:0/2924131714 learned_addr learned my addr 172.21.15.89:0/2924131714 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:04.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.237+0000 7f112fe00700 1 --2- 172.21.15.89:0/2924131714 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f113809aba0 0x7f11380d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:04.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.237+0000 7f1136a00700 1 -- 172.21.15.89:0/2924131714 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f113809aba0 msgr2=0x7f11380d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:04.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.237+0000 7f1136a00700 1 --2- 172.21.15.89:0/2924131714 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f113809aba0 0x7f11380d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.237+0000 7f1136a00700 1 -- 172.21.15.89:0/2924131714 >> v1:172.21.15.113:6789/0 conn(0x7f113806bf40 legacy=0x7f11380ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:04.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.237+0000 7f1136a00700 1 -- 172.21.15.89:0/2924131714 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113809a0c0 con 0x7f113815d710 2024-04-02T09:12:04.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.238+0000 7f1136a00700 1 --2- 172.21.15.89:0/2924131714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f113815d710 0x7f11380d0130 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f11280098f0 tx=0x7f1128009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=148c1d31a3711b89 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:04.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.238+0000 7f1136000700 1 -- 172.21.15.89:0/2924131714 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1128011830 con 0x7f113815d710 2024-04-02T09:12:04.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.238+0000 7f1136000700 1 -- 172.21.15.89:0/2924131714 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1128011990 con 0x7f113815d710 2024-04-02T09:12:04.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.238+0000 7f1136000700 1 -- 172.21.15.89:0/2924131714 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1128011d20 con 0x7f113815d710 2024-04-02T09:12:04.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.239+0000 7f113d800700 1 -- 172.21.15.89:0/2924131714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f113815d710 msgr2=0x7f11380d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:04.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.239+0000 7f113d800700 1 --2- 172.21.15.89:0/2924131714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f113815d710 0x7f11380d0130 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f11280098f0 tx=0x7f1128009c00 comp rx=0 tx=0).stop 2024-04-02T09:12:04.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.239+0000 7f113d800700 1 -- 172.21.15.89:0/2924131714 shutdown_connections 2024-04-02T09:12:04.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.239+0000 7f113d800700 1 --2- 172.21.15.89:0/2924131714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f113815d710 0x7f11380d0130 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.239+0000 7f113d800700 1 --2- 172.21.15.89:0/2924131714 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f113809aba0 0x7f11380d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.239+0000 7f113d800700 1 -- 172.21.15.89:0/2924131714 >> 172.21.15.89:0/2924131714 conn(0x7f11380d13e0 msgr2=0x7f11380d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:04.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.239+0000 7f113d800700 1 -- 172.21.15.89:0/2924131714 shutdown_connections 2024-04-02T09:12:04.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.239+0000 7f113d800700 1 -- 172.21.15.89:0/2924131714 wait complete. 2024-04-02T09:12:04.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.241+0000 7f113d800700 1 Processor -- start 2024-04-02T09:12:04.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.242+0000 7f113d800700 1 -- start start 2024-04-02T09:12:04.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.243+0000 7f113d800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f113806bf40 0x7f11380caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:04.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.243+0000 7f113d800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f113809aba0 0x7f1138153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:04.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.243+0000 7f113d800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f113815d710 0x7f11380c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:04.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.243+0000 7f113d800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11380cb430 con 0x7f113809aba0 2024-04-02T09:12:04.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.243+0000 7f113d800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1138153dc0 con 0x7f113806bf40 2024-04-02T09:12:04.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.243+0000 7f113d800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11380c9e50 con 0x7f113815d710 2024-04-02T09:12:04.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.244+0000 7f112fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f113809aba0 0x7f1138153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:04.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.244+0000 7f112fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f113809aba0 0x7f1138153880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33980/0 (socket says 172.21.15.89:33980) 2024-04-02T09:12:04.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.244+0000 7f112fe00700 1 -- 172.21.15.89:0/2215251436 learned_addr learned my addr 172.21.15.89:0/2215251436 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:04.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.244+0000 7f1136a00700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f113806bf40 0x7f11380caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:04.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.244+0000 7f1137400700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f113815d710 0x7f11380c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:04.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.244+0000 7f112fe00700 1 -- 172.21.15.89:0/2215251436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f113815d710 msgr2=0x7f11380c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:04.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.244+0000 7f112fe00700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f113815d710 0x7f11380c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.244+0000 7f112fe00700 1 -- 172.21.15.89:0/2215251436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f113806bf40 msgr2=0x7f11380caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:04.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.244+0000 7f112fe00700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f113806bf40 0x7f11380caef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.244+0000 7f112fe00700 1 -- 172.21.15.89:0/2215251436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1128009550 con 0x7f113809aba0 2024-04-02T09:12:04.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.244+0000 7f1137400700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f113815d710 0x7f11380c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:12:04.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.244+0000 7f112fe00700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f113809aba0 0x7f1138153880 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f112000e980 tx=0x7f112000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:04.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.245+0000 7f112ea00700 1 -- 172.21.15.89:0/2215251436 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120009710 con 0x7f113809aba0 2024-04-02T09:12:04.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.245+0000 7f112ea00700 1 -- 172.21.15.89:0/2215251436 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112000c690 con 0x7f113809aba0 2024-04-02T09:12:04.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.245+0000 7f113d800700 1 -- 172.21.15.89:0/2215251436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11380ca130 con 0x7f113809aba0 2024-04-02T09:12:04.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.245+0000 7f112ea00700 1 -- 172.21.15.89:0/2215251436 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112000ca40 con 0x7f113809aba0 2024-04-02T09:12:04.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.245+0000 7f113d800700 1 -- 172.21.15.89:0/2215251436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11380ca6f0 con 0x7f113809aba0 2024-04-02T09:12:04.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.246+0000 7f113d800700 1 -- 172.21.15.89:0/2215251436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1138067720 con 0x7f113809aba0 2024-04-02T09:12:04.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.247+0000 7f112ea00700 1 -- 172.21.15.89:0/2215251436 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f112000cba0 con 0x7f113809aba0 2024-04-02T09:12:04.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.247+0000 7f112ea00700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f111406adb0 0x7f111406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:04.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.248+0000 7f112ea00700 1 -- 172.21.15.89:0/2215251436 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f1120089810 con 0x7f113809aba0 2024-04-02T09:12:04.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.248+0000 7f1136a00700 1 -- 172.21.15.89:0/2215251436 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f111406adb0 msgr2=0x7f111406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:04.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.248+0000 7f1136a00700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f111406adb0 0x7f111406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:04.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.252+0000 7f112ea00700 1 -- 172.21.15.89:0/2215251436 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f11200544f0 con 0x7f113809aba0 2024-04-02T09:12:04.375 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:04.376+0000 7f7821bc7240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:12:04.399 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:04.400+0000 7f0ca3caa240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:12:04.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.449+0000 7f1136a00700 1 -- 172.21.15.89:0/2215251436 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f111406adb0 msgr2=0x7f111406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:04.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.449+0000 7f1136a00700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f111406adb0 0x7f111406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:04.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.460+0000 7f113d800700 1 -- 172.21.15.89:0/2215251436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11380b28d0 con 0x7f113809aba0 2024-04-02T09:12:04.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.462+0000 7f112ea00700 1 -- 172.21.15.89:0/2215251436 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) v1 ==== 74+0+17337 (secure 0 0 0) 0x7f1120057b10 con 0x7f113809aba0 2024-04-02T09:12:04.461 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:04.462 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":208,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:11:01.188455+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:09:40.103721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-04-02T09:09:41.165944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.5","mappings":[{"from":7,"to":3}]},{"pgid":"25.3b","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/2552487672":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/1282451945":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/3725462960":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/1115849275":"2024-04-03T09:09:59.528553+0000","172.21.15.89:6833/108855":"2024-04-03T09:09:59.528553+0000","172.21.15.89:6832/108855":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/2251569695":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/248630547":"2024-04-03T09:09:59.528553+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:12:04.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.464+0000 7f113d800700 1 -- 172.21.15.89:0/2215251436 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f111406adb0 msgr2=0x7f111406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:04.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.464+0000 7f113d800700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f111406adb0 0x7f111406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.464+0000 7f113d800700 1 -- 172.21.15.89:0/2215251436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f113809aba0 msgr2=0x7f1138153880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:04.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.464+0000 7f113d800700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f113809aba0 0x7f1138153880 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f112000e980 tx=0x7f112000ec90 comp rx=0 tx=0).stop 2024-04-02T09:12:04.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.464+0000 7f113d800700 1 -- 172.21.15.89:0/2215251436 shutdown_connections 2024-04-02T09:12:04.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.464+0000 7f113d800700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f111406adb0 0x7f111406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.464+0000 7f113d800700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f113806bf40 0x7f11380caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.464+0000 7f113d800700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f113815d710 0x7f11380c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.464+0000 7f113d800700 1 --2- 172.21.15.89:0/2215251436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f113809aba0 0x7f1138153880 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.464+0000 7f113d800700 1 -- 172.21.15.89:0/2215251436 >> 172.21.15.89:0/2215251436 conn(0x7f11380d13e0 msgr2=0x7f11380cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:04.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.464+0000 7f113d800700 1 -- 172.21.15.89:0/2215251436 shutdown_connections 2024-04-02T09:12:04.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.465+0000 7f113d800700 1 -- 172.21.15.89:0/2215251436 wait complete. 2024-04-02T09:12:04.483 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T09:09:40.103721+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '204', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T09:09:41.165944+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '205', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:12:04.484 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:12:04.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.687+0000 7f85ea600700 1 Processor -- start 2024-04-02T09:12:04.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.688+0000 7f85ea600700 1 -- start start 2024-04-02T09:12:04.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.689+0000 7f85ea600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f85e409a750 0x7f85e40d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:04.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.689+0000 7f85ea600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e40d0970 con 0x7f85e4099a60 2024-04-02T09:12:04.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.689+0000 7f85ea600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e406c020 con 0x7f85e415d990 2024-04-02T09:12:04.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.689+0000 7f85ea600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e40c81a0 con 0x7f85e409a750 2024-04-02T09:12:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.689+0000 7f85e3400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f85e4099a60 0x7f85e4099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32848/0 (socket says 172.21.15.89:32848) 2024-04-02T09:12:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.689+0000 7f85e3400700 1 -- 172.21.15.89:0/741770411 learned_addr learned my addr 172.21.15.89:0/741770411 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.689+0000 7f85e3e00700 1 --2- 172.21.15.89:0/741770411 >> v2:172.21.15.113:3300/0 conn(0x7f85e409a750 0x7f85e40d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.690+0000 7f85e2000700 1 -- 172.21.15.89:0/741770411 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773389496 0 0) 0x7f85e40d0970 con 0x7f85e4099a60 2024-04-02T09:12:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.690+0000 7f85e2000700 1 -- 172.21.15.89:0/741770411 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c80035b0 con 0x7f85e4099a60 2024-04-02T09:12:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.690+0000 7f85e3e00700 1 -- 172.21.15.89:0/741770411 >> v1:172.21.15.113:6789/0 conn(0x7f85e415d990 legacy=0x7f85e406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.690+0000 7f85e2000700 1 -- 172.21.15.89:0/741770411 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209196400 0 0) 0x7f85e406c020 con 0x7f85e415d990 2024-04-02T09:12:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.690+0000 7f85e3e00700 1 -- 172.21.15.89:0/741770411 >> v1:172.21.15.89:6789/0 conn(0x7f85e4099a60 legacy=0x7f85e4099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.690+0000 7f85e3e00700 1 -- 172.21.15.89:0/741770411 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e40d0690 con 0x7f85e409a750 2024-04-02T09:12:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.690+0000 7f85e3e00700 1 --2- 172.21.15.89:0/741770411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85e409a750 0x7f85e40d0060 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f85dc009d90 tx=0x7f85dc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7797b6fab47f1879 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.690+0000 7f85e2000700 1 -- 172.21.15.89:0/741770411 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc00e070 con 0x7f85e409a750 2024-04-02T09:12:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.691+0000 7f85e2000700 1 -- 172.21.15.89:0/741770411 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85dc003a20 con 0x7f85e409a750 2024-04-02T09:12:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.691+0000 7f85e2000700 1 -- 172.21.15.89:0/741770411 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc003db0 con 0x7f85e409a750 2024-04-02T09:12:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.691+0000 7f85ea600700 1 -- 172.21.15.89:0/741770411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85e409a750 msgr2=0x7f85e40d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.691+0000 7f85ea600700 1 --2- 172.21.15.89:0/741770411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85e409a750 0x7f85e40d0060 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f85dc009d90 tx=0x7f85dc00b330 comp rx=0 tx=0).stop 2024-04-02T09:12:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.691+0000 7f85ea600700 1 -- 172.21.15.89:0/741770411 shutdown_connections 2024-04-02T09:12:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.691+0000 7f85ea600700 1 --2- 172.21.15.89:0/741770411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85e409a750 0x7f85e40d0060 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.691+0000 7f85ea600700 1 -- 172.21.15.89:0/741770411 >> 172.21.15.89:0/741770411 conn(0x7f85e40b2e20 msgr2=0x7f85e40b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.691+0000 7f85ea600700 1 -- 172.21.15.89:0/741770411 shutdown_connections 2024-04-02T09:12:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.691+0000 7f85ea600700 1 -- 172.21.15.89:0/741770411 wait complete. 2024-04-02T09:12:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.694+0000 7f85ea600700 1 Processor -- start 2024-04-02T09:12:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.694+0000 7f85ea600700 1 -- start start 2024-04-02T09:12:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.695+0000 7f85ea600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85e4099a60 0x7f85e4159830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.695+0000 7f85ea600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85e409a750 0x7f85e415dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.695+0000 7f85ea600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85e415d990 0x7f85e415e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.695+0000 7f85ea600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e40b26d0 con 0x7f85e415d990 2024-04-02T09:12:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.695+0000 7f85ea600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e415e8e0 con 0x7f85e409a750 2024-04-02T09:12:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.695+0000 7f85ea600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e415ebc0 con 0x7f85e4099a60 2024-04-02T09:12:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.695+0000 7f85e3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85e415d990 0x7f85e415e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.695+0000 7f85e3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85e415d990 0x7f85e415e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33986/0 (socket says 172.21.15.89:33986) 2024-04-02T09:12:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.695+0000 7f85e3e00700 1 -- 172.21.15.89:0/1332235751 learned_addr learned my addr 172.21.15.89:0/1332235751 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.695+0000 7f85e3400700 1 --2- 172.21.15.89:0/1332235751 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85e4099a60 0x7f85e4159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.695+0000 7f85e2a00700 1 --2- 172.21.15.89:0/1332235751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85e409a750 0x7f85e415dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.696+0000 7f85e3e00700 1 -- 172.21.15.89:0/1332235751 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85e4099a60 msgr2=0x7f85e4159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.696+0000 7f85e3e00700 1 --2- 172.21.15.89:0/1332235751 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85e4099a60 0x7f85e4159830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.696+0000 7f85e3e00700 1 -- 172.21.15.89:0/1332235751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85e409a750 msgr2=0x7f85e415dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.696+0000 7f85e3e00700 1 --2- 172.21.15.89:0/1332235751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85e409a750 0x7f85e415dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.696+0000 7f85e3e00700 1 -- 172.21.15.89:0/1332235751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc009a70 con 0x7f85e415d990 2024-04-02T09:12:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.696+0000 7f85e3e00700 1 --2- 172.21.15.89:0/1332235751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85e415d990 0x7f85e415e2b0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f85e40724f0 tx=0x7f85dc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.696+0000 7f85db400700 1 -- 172.21.15.89:0/1332235751 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc004770 con 0x7f85e415d990 2024-04-02T09:12:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.696+0000 7f85db400700 1 -- 172.21.15.89:0/1332235751 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85dc0048d0 con 0x7f85e415d990 2024-04-02T09:12:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.696+0000 7f85db400700 1 -- 172.21.15.89:0/1332235751 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc01d6b0 con 0x7f85e415d990 2024-04-02T09:12:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.696+0000 7f85ea600700 1 -- 172.21.15.89:0/1332235751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e40c4e30 con 0x7f85e415d990 2024-04-02T09:12:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.697+0000 7f85ea600700 1 -- 172.21.15.89:0/1332235751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e4159d70 con 0x7f85e415d990 2024-04-02T09:12:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.698+0000 7f85ea600700 1 -- 172.21.15.89:0/1332235751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e40ccd00 con 0x7f85e415d990 2024-04-02T09:12:04.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.698+0000 7f85db400700 1 -- 172.21.15.89:0/1332235751 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f85dc01d810 con 0x7f85e415d990 2024-04-02T09:12:04.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.699+0000 7f85db400700 1 --2- 172.21.15.89:0/1332235751 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f85c406adb0 0x7f85c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:04.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.699+0000 7f85e3400700 1 -- 172.21.15.89:0/1332235751 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f85c406adb0 msgr2=0x7f85c406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:04.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.699+0000 7f85db400700 1 -- 172.21.15.89:0/1332235751 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f85dc08b900 con 0x7f85e415d990 2024-04-02T09:12:04.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.699+0000 7f85e3400700 1 --2- 172.21.15.89:0/1332235751 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f85c406adb0 0x7f85c406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:04.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.704+0000 7f85db400700 1 -- 172.21.15.89:0/1332235751 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f85dc056560 con 0x7f85e415d990 2024-04-02T09:12:04.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.900+0000 7f85e3400700 1 -- 172.21.15.89:0/1332235751 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f85c406adb0 msgr2=0x7f85c406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:04.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.900+0000 7f85e3400700 1 --2- 172.21.15.89:0/1332235751 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f85c406adb0 0x7f85c406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:04.905 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:04.906+0000 7f7821bc7240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:12:04.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.913+0000 7f85ea600700 1 -- 172.21.15.89:0/1332235751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f85e4067720 con 0x7f85e415d990 2024-04-02T09:12:04.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.913+0000 7f85db400700 1 -- 172.21.15.89:0/1332235751 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v208) v1 ==== 93+0+10 (secure 0 0 0) 0x7f85dc089e60 con 0x7f85e415d990 2024-04-02T09:12:04.912 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:12:04.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.915+0000 7f85ea600700 1 -- 172.21.15.89:0/1332235751 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f85c406adb0 msgr2=0x7f85c406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:04.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.915+0000 7f85ea600700 1 --2- 172.21.15.89:0/1332235751 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f85c406adb0 0x7f85c406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.915+0000 7f85ea600700 1 -- 172.21.15.89:0/1332235751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85e415d990 msgr2=0x7f85e415e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:04.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.915+0000 7f85ea600700 1 --2- 172.21.15.89:0/1332235751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85e415d990 0x7f85e415e2b0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f85e40724f0 tx=0x7f85dc004ad0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.916+0000 7f85ea600700 1 -- 172.21.15.89:0/1332235751 shutdown_connections 2024-04-02T09:12:04.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.916+0000 7f85ea600700 1 --2- 172.21.15.89:0/1332235751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85e415d990 0x7f85e415e2b0 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.916+0000 7f85ea600700 1 --2- 172.21.15.89:0/1332235751 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85e4099a60 0x7f85e4159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.916+0000 7f85ea600700 1 --2- 172.21.15.89:0/1332235751 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f85c406adb0 0x7f85c406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.916+0000 7f85ea600700 1 --2- 172.21.15.89:0/1332235751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85e409a750 0x7f85e415dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:04.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.916+0000 7f85ea600700 1 -- 172.21.15.89:0/1332235751 >> 172.21.15.89:0/1332235751 conn(0x7f85e40b2e20 msgr2=0x7f85e40b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:04.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.916+0000 7f85ea600700 1 -- 172.21.15.89:0/1332235751 shutdown_connections 2024-04-02T09:12:04.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:04.916+0000 7f85ea600700 1 -- 172.21.15.89:0/1332235751 wait complete. 2024-04-02T09:12:04.933 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T09:12:04.966 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:04.967+0000 7f0ca3caa240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:12:05.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.136+0000 7fc678400700 1 Processor -- start 2024-04-02T09:12:05.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.137+0000 7fc678400700 1 -- start start 2024-04-02T09:12:05.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc678400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc6700cbb30 0x7fc6700cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:05.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc678400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc6700cd020 0x7fc6700cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:05.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc678400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6700b2070 0x7fc6700b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:05.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc678400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6700cc450 con 0x7fc6700cd020 2024-04-02T09:12:05.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc678400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6700cdae0 con 0x7fc6700cbb30 2024-04-02T09:12:05.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc678400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6700cddc0 con 0x7fc6700b2070 2024-04-02T09:12:05.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc675000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc6700cd020 0x7fc6700cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:05.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc675a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc6700cbb30 0x7fc6700cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:05.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc675000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc6700cd020 0x7fc6700cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33990/0 (socket says 172.21.15.89:33990) 2024-04-02T09:12:05.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc675000700 1 -- 172.21.15.89:0/3421168940 learned_addr learned my addr 172.21.15.89:0/3421168940 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:05.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc676400700 1 --2- 172.21.15.89:0/3421168940 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6700b2070 0x7fc6700b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:05.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc675000700 1 -- 172.21.15.89:0/3421168940 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6700b2070 msgr2=0x7fc6700b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:05.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc675000700 1 --2- 172.21.15.89:0/3421168940 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6700b2070 0x7fc6700b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc675000700 1 -- 172.21.15.89:0/3421168940 >> v2:172.21.15.113:3300/0 conn(0x7fc6700cbb30 msgr2=0x7fc6700cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:05.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc675000700 1 --2- 172.21.15.89:0/3421168940 >> v2:172.21.15.113:3300/0 conn(0x7fc6700cbb30 0x7fc6700cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.138+0000 7fc675000700 1 -- 172.21.15.89:0/3421168940 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6700ce0a0 con 0x7fc6700cd020 2024-04-02T09:12:05.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.139+0000 7fc675000700 1 --2- 172.21.15.89:0/3421168940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6700cd020 0x7fc6700cd480 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fc67006fd00 tx=0x7fc65c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d6c98fa13ec609 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:05.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.139+0000 7fc66fe00700 1 -- 172.21.15.89:0/3421168940 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c011830 con 0x7fc6700cd020 2024-04-02T09:12:05.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.139+0000 7fc66fe00700 1 -- 172.21.15.89:0/3421168940 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65c011990 con 0x7fc6700cd020 2024-04-02T09:12:05.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.139+0000 7fc66fe00700 1 -- 172.21.15.89:0/3421168940 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c011d20 con 0x7fc6700cd020 2024-04-02T09:12:05.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.139+0000 7fc678400700 1 -- 172.21.15.89:0/3421168940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6700cd020 msgr2=0x7fc6700cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:05.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.140+0000 7fc678400700 1 --2- 172.21.15.89:0/3421168940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6700cd020 0x7fc6700cd480 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fc67006fd00 tx=0x7fc65c009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.140+0000 7fc678400700 1 -- 172.21.15.89:0/3421168940 shutdown_connections 2024-04-02T09:12:05.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.140+0000 7fc678400700 1 --2- 172.21.15.89:0/3421168940 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6700b2070 0x7fc6700b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.140+0000 7fc678400700 1 --2- 172.21.15.89:0/3421168940 >> v2:172.21.15.113:3300/0 conn(0x7fc6700cbb30 0x7fc6700cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.140+0000 7fc678400700 1 --2- 172.21.15.89:0/3421168940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6700cd020 0x7fc6700cd480 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.140+0000 7fc678400700 1 -- 172.21.15.89:0/3421168940 >> 172.21.15.89:0/3421168940 conn(0x7fc6700b0d70 msgr2=0x7fc6700b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:05.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.140+0000 7fc678400700 1 -- 172.21.15.89:0/3421168940 shutdown_connections 2024-04-02T09:12:05.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.140+0000 7fc678400700 1 -- 172.21.15.89:0/3421168940 wait complete. 2024-04-02T09:12:05.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.141+0000 7fc678400700 1 Processor -- start 2024-04-02T09:12:05.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.142+0000 7fc678400700 1 -- start start 2024-04-02T09:12:05.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.142+0000 7fc678400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6700b2070 0x7fc670151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:05.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.142+0000 7fc678400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6700cbb30 0x7fc6700ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:05.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc678400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6700cd020 0x7fc6700b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:05.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc678400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc670152970 con 0x7fc6700b2070 2024-04-02T09:12:05.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc678400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc670070a60 con 0x7fc6700cbb30 2024-04-02T09:12:05.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc678400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc670070d40 con 0x7fc6700cd020 2024-04-02T09:12:05.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc675a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6700b2070 0x7fc670151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:05.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc675a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6700b2070 0x7fc670151c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34004/0 (socket says 172.21.15.89:34004) 2024-04-02T09:12:05.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc675a00700 1 -- 172.21.15.89:0/1939399028 learned_addr learned my addr 172.21.15.89:0/1939399028 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:05.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc675000700 1 --2- 172.21.15.89:0/1939399028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6700cbb30 0x7fc6700ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:05.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc676400700 1 --2- 172.21.15.89:0/1939399028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6700cd020 0x7fc6700b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:05.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc675a00700 1 -- 172.21.15.89:0/1939399028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6700cd020 msgr2=0x7fc6700b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:05.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc675a00700 1 --2- 172.21.15.89:0/1939399028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6700cd020 0x7fc6700b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc675a00700 1 -- 172.21.15.89:0/1939399028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6700cbb30 msgr2=0x7fc6700ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:05.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc675a00700 1 --2- 172.21.15.89:0/1939399028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6700cbb30 0x7fc6700ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc675a00700 1 -- 172.21.15.89:0/1939399028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c009550 con 0x7fc6700b2070 2024-04-02T09:12:05.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.143+0000 7fc675a00700 1 --2- 172.21.15.89:0/1939399028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6700b2070 0x7fc670151c30 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fc66400e870 tx=0x7fc66400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:05.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.144+0000 7fc66d600700 1 -- 172.21.15.89:0/1939399028 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664009590 con 0x7fc6700b2070 2024-04-02T09:12:05.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.144+0000 7fc66d600700 1 -- 172.21.15.89:0/1939399028 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc664013680 con 0x7fc6700b2070 2024-04-02T09:12:05.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.144+0000 7fc678400700 1 -- 172.21.15.89:0/1939399028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc670071020 con 0x7fc6700b2070 2024-04-02T09:12:05.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.144+0000 7fc66d600700 1 -- 172.21.15.89:0/1939399028 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6640139f0 con 0x7fc6700b2070 2024-04-02T09:12:05.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.144+0000 7fc678400700 1 -- 172.21.15.89:0/1939399028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6700715e0 con 0x7fc6700b2070 2024-04-02T09:12:05.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.145+0000 7fc678400700 1 -- 172.21.15.89:0/1939399028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc670071800 con 0x7fc6700b2070 2024-04-02T09:12:05.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.146+0000 7fc66d600700 1 -- 172.21.15.89:0/1939399028 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fc664013b50 con 0x7fc6700b2070 2024-04-02T09:12:05.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.147+0000 7fc66d600700 1 --2- 172.21.15.89:0/1939399028 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc65006adb0 0x7fc65006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:05.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.147+0000 7fc675000700 1 -- 172.21.15.89:0/1939399028 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc65006adb0 msgr2=0x7fc65006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:05.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.147+0000 7fc66d600700 1 -- 172.21.15.89:0/1939399028 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6559+0+0 (secure 0 0 0) 0x7fc664089670 con 0x7fc6700b2070 2024-04-02T09:12:05.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.147+0000 7fc675000700 1 --2- 172.21.15.89:0/1939399028 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc65006adb0 0x7fc65006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:05.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.155+0000 7fc66d600700 1 -- 172.21.15.89:0/1939399028 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc664054350 con 0x7fc6700b2070 2024-04-02T09:12:05.161 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:05.163+0000 7f7821bc7240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:12:05.239 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:05.240+0000 7f0ca3caa240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:12:05.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.348+0000 7fc675000700 1 -- 172.21.15.89:0/1939399028 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc65006adb0 msgr2=0x7fc65006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:05.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.348+0000 7fc675000700 1 --2- 172.21.15.89:0/1939399028 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc65006adb0 0x7fc65006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:05.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.364+0000 7fc678400700 1 -- 172.21.15.89:0/1939399028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc67015bda0 con 0x7fc6700b2070 2024-04-02T09:12:05.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.365+0000 7fc66d600700 1 -- 172.21.15.89:0/1939399028 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v208) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc664057970 con 0x7fc6700b2070 2024-04-02T09:12:05.364 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:12:05.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.367+0000 7fc678400700 1 -- 172.21.15.89:0/1939399028 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc65006adb0 msgr2=0x7fc65006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:05.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.367+0000 7fc678400700 1 --2- 172.21.15.89:0/1939399028 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc65006adb0 0x7fc65006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.367+0000 7fc678400700 1 -- 172.21.15.89:0/1939399028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6700b2070 msgr2=0x7fc670151c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:05.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.367+0000 7fc678400700 1 --2- 172.21.15.89:0/1939399028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6700b2070 0x7fc670151c30 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fc66400e870 tx=0x7fc66400eb80 comp rx=0 tx=0).stop 2024-04-02T09:12:05.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.368+0000 7fc678400700 1 -- 172.21.15.89:0/1939399028 shutdown_connections 2024-04-02T09:12:05.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.368+0000 7fc678400700 1 --2- 172.21.15.89:0/1939399028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6700cd020 0x7fc6700b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.368+0000 7fc678400700 1 --2- 172.21.15.89:0/1939399028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6700b2070 0x7fc670151c30 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.368+0000 7fc678400700 1 --2- 172.21.15.89:0/1939399028 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc65006adb0 0x7fc65006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.368+0000 7fc678400700 1 --2- 172.21.15.89:0/1939399028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6700cbb30 0x7fc6700ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.368+0000 7fc678400700 1 -- 172.21.15.89:0/1939399028 >> 172.21.15.89:0/1939399028 conn(0x7fc6700b0d70 msgr2=0x7fc6700b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:05.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.368+0000 7fc678400700 1 -- 172.21.15.89:0/1939399028 shutdown_connections 2024-04-02T09:12:05.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.368+0000 7fc678400700 1 -- 172.21.15.89:0/1939399028 wait complete. 2024-04-02T09:12:05.387 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T09:12:05.619 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:05.620+0000 7f7821bc7240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:12:05.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.627+0000 7f7a1fa00700 1 Processor -- start 2024-04-02T09:12:05.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.627+0000 7f7a1fa00700 1 -- start start 2024-04-02T09:12:05.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.628+0000 7f7a1fa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a180cd5c0 0x7f7a1815e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:05.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.628+0000 7f7a1fa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7a1806f310 0x7f7a1806f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:05.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.628+0000 7f7a1fa00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1815eda0 con 0x7f7a1815cfe0 2024-04-02T09:12:05.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.628+0000 7f7a1fa00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1806fcb0 con 0x7f7a1806f310 2024-04-02T09:12:05.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.628+0000 7f7a1fa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a18099dd0 con 0x7f7a180cd5c0 2024-04-02T09:12:05.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.628+0000 7f7a17e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7a1806f310 0x7f7a1806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:05.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.629+0000 7f7a1da00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7a1815cfe0 0x7f7a1815d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32858/0 (socket says 172.21.15.89:32858) 2024-04-02T09:12:05.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.629+0000 7f7a1da00700 1 -- 172.21.15.89:0/3098665589 learned_addr learned my addr 172.21.15.89:0/3098665589 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:05.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.629+0000 7f7a1d000700 1 --2- 172.21.15.89:0/3098665589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a180cd5c0 0x7f7a1815e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:05.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.629+0000 7f7a17e00700 1 -- 172.21.15.89:0/3098665589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a180cd5c0 msgr2=0x7f7a1815e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:05.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.629+0000 7f7a17e00700 1 --2- 172.21.15.89:0/3098665589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a180cd5c0 0x7f7a1815e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.629+0000 7f7a17e00700 1 -- 172.21.15.89:0/3098665589 >> v1:172.21.15.89:6789/0 conn(0x7f7a1815cfe0 legacy=0x7f7a1815d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:05.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.629+0000 7f7a17e00700 1 -- 172.21.15.89:0/3098665589 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1809a0b0 con 0x7f7a1806f310 2024-04-02T09:12:05.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.629+0000 7f7a17400700 1 -- 172.21.15.89:0/3098665589 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668444796 0 0) 0x7f7a10001490 con 0x7f7a1815cfe0 2024-04-02T09:12:05.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.629+0000 7f7a17e00700 1 --2- 172.21.15.89:0/3098665589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1806f310 0x7f7a1806f770 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f7a04009840 tx=0x7f7a04009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa73ac5396b764e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:05.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.630+0000 7f7a17400700 1 -- 172.21.15.89:0/3098665589 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04011830 con 0x7f7a1806f310 2024-04-02T09:12:05.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.630+0000 7f7a17400700 1 -- 172.21.15.89:0/3098665589 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a04011990 con 0x7f7a1806f310 2024-04-02T09:12:05.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.630+0000 7f7a17400700 1 -- 172.21.15.89:0/3098665589 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04011d20 con 0x7f7a1806f310 2024-04-02T09:12:05.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.630+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3098665589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1806f310 msgr2=0x7f7a1806f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:05.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.630+0000 7f7a1fa00700 1 --2- 172.21.15.89:0/3098665589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1806f310 0x7f7a1806f770 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f7a04009840 tx=0x7f7a04009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:05.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.630+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3098665589 shutdown_connections 2024-04-02T09:12:05.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.630+0000 7f7a1fa00700 1 --2- 172.21.15.89:0/3098665589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a180cd5c0 0x7f7a1815e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.630+0000 7f7a1fa00700 1 --2- 172.21.15.89:0/3098665589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1806f310 0x7f7a1806f770 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.630+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3098665589 >> 172.21.15.89:0/3098665589 conn(0x7f7a180d1370 msgr2=0x7f7a180d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:05.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.631+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3098665589 shutdown_connections 2024-04-02T09:12:05.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.631+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3098665589 wait complete. 2024-04-02T09:12:05.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.632+0000 7f7a1fa00700 1 Processor -- start 2024-04-02T09:12:05.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.633+0000 7f7a1fa00700 1 -- start start 2024-04-02T09:12:05.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.633+0000 7f7a1fa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a1806f310 0x7f7a1815bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:05.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.633+0000 7f7a1fa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a180cd5c0 0x7f7a180c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:05.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.633+0000 7f7a1fa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1815cfe0 0x7f7a180c6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:05.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.633+0000 7f7a1fa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1815c080 con 0x7f7a1815cfe0 2024-04-02T09:12:05.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.633+0000 7f7a1fa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a180b6570 con 0x7f7a1806f310 2024-04-02T09:12:05.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.633+0000 7f7a1fa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a180c8570 con 0x7f7a180cd5c0 2024-04-02T09:12:05.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.634+0000 7f7a1da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1815cfe0 0x7f7a180c6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:05.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.634+0000 7f7a1da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1815cfe0 0x7f7a180c6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34024/0 (socket says 172.21.15.89:34024) 2024-04-02T09:12:05.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.634+0000 7f7a1da00700 1 -- 172.21.15.89:0/3090978162 learned_addr learned my addr 172.21.15.89:0/3090978162 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:05.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.634+0000 7f7a1d000700 1 --2- 172.21.15.89:0/3090978162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a1806f310 0x7f7a1815bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:05.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.634+0000 7f7a17e00700 1 --2- 172.21.15.89:0/3090978162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a180cd5c0 0x7f7a180c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:05.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.634+0000 7f7a1da00700 1 -- 172.21.15.89:0/3090978162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a180cd5c0 msgr2=0x7f7a180c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:05.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.634+0000 7f7a1da00700 1 --2- 172.21.15.89:0/3090978162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a180cd5c0 0x7f7a180c92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.634+0000 7f7a1da00700 1 -- 172.21.15.89:0/3090978162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a1806f310 msgr2=0x7f7a1815bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:05.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.634+0000 7f7a1da00700 1 --2- 172.21.15.89:0/3090978162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a1806f310 0x7f7a1815bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.634+0000 7f7a1da00700 1 -- 172.21.15.89:0/3090978162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a040094a0 con 0x7f7a1815cfe0 2024-04-02T09:12:05.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.635+0000 7f7a1da00700 1 --2- 172.21.15.89:0/3090978162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1815cfe0 0x7f7a180c6f90 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000cba0 tx=0x7f7a1000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:05.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.635+0000 7f7a14c00700 1 -- 172.21.15.89:0/3090978162 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1000aaa0 con 0x7f7a1815cfe0 2024-04-02T09:12:05.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.635+0000 7f7a14c00700 1 -- 172.21.15.89:0/3090978162 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1000ac20 con 0x7f7a1815cfe0 2024-04-02T09:12:05.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.635+0000 7f7a14c00700 1 -- 172.21.15.89:0/3090978162 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a100070d0 con 0x7f7a1815cfe0 2024-04-02T09:12:05.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.635+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3090978162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a180c8810 con 0x7f7a1815cfe0 2024-04-02T09:12:05.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.635+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3090978162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a180c8dd0 con 0x7f7a1815cfe0 2024-04-02T09:12:05.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.636+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3090978162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a180c8fc0 con 0x7f7a1815cfe0 2024-04-02T09:12:05.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.638+0000 7f7a14c00700 1 -- 172.21.15.89:0/3090978162 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f7a10007230 con 0x7f7a1815cfe0 2024-04-02T09:12:05.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.638+0000 7f7a14c00700 1 --2- 172.21.15.89:0/3090978162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f79f806adb0 0x7f79f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:05.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.638+0000 7f7a14c00700 1 -- 172.21.15.89:0/3090978162 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f7a100890e0 con 0x7f7a1815cfe0 2024-04-02T09:12:05.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.639+0000 7f7a1d000700 1 -- 172.21.15.89:0/3090978162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f79f806adb0 msgr2=0x7f79f806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:05.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.639+0000 7f7a1d000700 1 --2- 172.21.15.89:0/3090978162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f79f806adb0 0x7f79f806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:05.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.643+0000 7f7a14c00700 1 -- 172.21.15.89:0/3090978162 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7a100bc580 con 0x7f7a1815cfe0 2024-04-02T09:12:05.724 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:05.725+0000 7f0ca3caa240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:12:05.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.840+0000 7f7a1d000700 1 -- 172.21.15.89:0/3090978162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f79f806adb0 msgr2=0x7f79f806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:05.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.840+0000 7f7a1d000700 1 --2- 172.21.15.89:0/3090978162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f79f806adb0 0x7f79f806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:05.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.854+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3090978162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7a1815eda0 con 0x7f7a1815cfe0 2024-04-02T09:12:05.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.855+0000 7f7a14c00700 1 -- 172.21.15.89:0/3090978162 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v208) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7a1815eda0 con 0x7f7a1815cfe0 2024-04-02T09:12:05.854 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:12:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.857+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3090978162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f79f806adb0 msgr2=0x7f79f806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.857+0000 7f7a1fa00700 1 --2- 172.21.15.89:0/3090978162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f79f806adb0 0x7f79f806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.857+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3090978162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1815cfe0 msgr2=0x7f7a180c6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.857+0000 7f7a1fa00700 1 --2- 172.21.15.89:0/3090978162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1815cfe0 0x7f7a180c6f90 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000cba0 tx=0x7f7a1000ceb0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.857+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3090978162 shutdown_connections 2024-04-02T09:12:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.857+0000 7f7a1fa00700 1 --2- 172.21.15.89:0/3090978162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f79f806adb0 0x7f79f806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.857+0000 7f7a1fa00700 1 --2- 172.21.15.89:0/3090978162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a1806f310 0x7f7a1815bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.857+0000 7f7a1fa00700 1 --2- 172.21.15.89:0/3090978162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a1815cfe0 0x7f7a180c6f90 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.857+0000 7f7a1fa00700 1 --2- 172.21.15.89:0/3090978162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7a180cd5c0 0x7f7a180c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:05.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.857+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3090978162 >> 172.21.15.89:0/3090978162 conn(0x7f7a180d1370 msgr2=0x7f7a1815e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:05.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.857+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3090978162 shutdown_connections 2024-04-02T09:12:05.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:05.857+0000 7f7a1fa00700 1 -- 172.21.15.89:0/3090978162 wait complete. 2024-04-02T09:12:05.875 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T09:12:06.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.072+0000 7fe122000700 1 Processor -- start 2024-04-02T09:12:06.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.072+0000 7fe122000700 1 -- start start 2024-04-02T09:12:06.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.073+0000 7fe122000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe11c157f90 0x7fe11c0b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:06.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.074+0000 7fe122000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c0b3070 con 0x7fe11c1613e0 2024-04-02T09:12:06.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.074+0000 7fe122000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c0b2070 con 0x7fe11c0b3b30 2024-04-02T09:12:06.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.074+0000 7fe122000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe11c158370 con 0x7fe11c157f90 2024-04-02T09:12:06.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.074+0000 7fe11aa00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe11c1613e0 0x7fe11c0b3420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32868/0 (socket says 172.21.15.89:32868) 2024-04-02T09:12:06.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.074+0000 7fe11aa00700 1 -- 172.21.15.89:0/1954057529 learned_addr learned my addr 172.21.15.89:0/1954057529 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:06.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.074+0000 7fe11b400700 1 --2- 172.21.15.89:0/1954057529 >> v2:172.21.15.113:3300/0 conn(0x7fe11c157f90 0x7fe11c0b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:06.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.074+0000 7fe11a000700 1 -- 172.21.15.89:0/1954057529 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612302525 0 0) 0x7fe108001110 con 0x7fe11c1613e0 2024-04-02T09:12:06.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.074+0000 7fe11a000700 1 -- 172.21.15.89:0/1954057529 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1000035b0 con 0x7fe11c1613e0 2024-04-02T09:12:06.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.074+0000 7fe11a000700 1 -- 172.21.15.89:0/1954057529 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853406044 0 0) 0x7fe11c0b2070 con 0x7fe11c0b3b30 2024-04-02T09:12:06.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.075+0000 7fe11a000700 1 -- 172.21.15.89:0/1954057529 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe108001110 con 0x7fe11c0b3b30 2024-04-02T09:12:06.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.075+0000 7fe11b400700 1 -- 172.21.15.89:0/1954057529 >> v1:172.21.15.113:6789/0 conn(0x7fe11c0b3b30 legacy=0x7fe11c0cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:06.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.075+0000 7fe11b400700 1 -- 172.21.15.89:0/1954057529 >> v1:172.21.15.89:6789/0 conn(0x7fe11c1613e0 legacy=0x7fe11c0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:06.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.075+0000 7fe11a000700 1 -- 172.21.15.89:0/1954057529 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2895298590 0 0) 0x7fe1000035b0 con 0x7fe11c1613e0 2024-04-02T09:12:06.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.075+0000 7fe11b400700 1 -- 172.21.15.89:0/1954057529 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c15e0f0 con 0x7fe11c157f90 2024-04-02T09:12:06.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.075+0000 7fe11b400700 1 --2- 172.21.15.89:0/1954057529 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c157f90 0x7fe11c0b2b30 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe10c009840 tx=0x7fe10c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d5961109882d635 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:06.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.075+0000 7fe11a000700 1 -- 172.21.15.89:0/1954057529 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c011830 con 0x7fe11c157f90 2024-04-02T09:12:06.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.075+0000 7fe11a000700 1 -- 172.21.15.89:0/1954057529 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10c011990 con 0x7fe11c157f90 2024-04-02T09:12:06.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.075+0000 7fe11a000700 1 -- 172.21.15.89:0/1954057529 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c011d20 con 0x7fe11c157f90 2024-04-02T09:12:06.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.076+0000 7fe122000700 1 -- 172.21.15.89:0/1954057529 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c157f90 msgr2=0x7fe11c0b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:06.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.076+0000 7fe122000700 1 --2- 172.21.15.89:0/1954057529 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c157f90 0x7fe11c0b2b30 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe10c009840 tx=0x7fe10c009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:06.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.076+0000 7fe122000700 1 -- 172.21.15.89:0/1954057529 shutdown_connections 2024-04-02T09:12:06.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.076+0000 7fe122000700 1 --2- 172.21.15.89:0/1954057529 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c157f90 0x7fe11c0b2b30 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.076+0000 7fe122000700 1 -- 172.21.15.89:0/1954057529 >> 172.21.15.89:0/1954057529 conn(0x7fe11c0b0d70 msgr2=0x7fe11c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:06.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.076+0000 7fe122000700 1 -- 172.21.15.89:0/1954057529 shutdown_connections 2024-04-02T09:12:06.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.076+0000 7fe122000700 1 -- 172.21.15.89:0/1954057529 wait complete. 2024-04-02T09:12:06.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.078+0000 7fe122000700 1 Processor -- start 2024-04-02T09:12:06.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.078+0000 7fe122000700 1 -- start start 2024-04-02T09:12:06.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe122000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c0b3b30 0x7fe11c0cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe122000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe11c157f90 0x7fe11c158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe122000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe11c1613e0 0x7fe11c1f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe122000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c0b2710 con 0x7fe11c157f90 2024-04-02T09:12:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe122000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c15bee0 con 0x7fe11c0b3b30 2024-04-02T09:12:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe122000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c15c130 con 0x7fe11c1613e0 2024-04-02T09:12:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe11aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe11c157f90 0x7fe11c158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe11aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe11c157f90 0x7fe11c158ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34040/0 (socket says 172.21.15.89:34040) 2024-04-02T09:12:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe11b400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c0b3b30 0x7fe11c0cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe11aa00700 1 -- 172.21.15.89:0/1118537310 learned_addr learned my addr 172.21.15.89:0/1118537310 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe11be00700 1 --2- 172.21.15.89:0/1118537310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe11c1613e0 0x7fe11c1f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe11aa00700 1 -- 172.21.15.89:0/1118537310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe11c1613e0 msgr2=0x7fe11c1f3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:06.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe11aa00700 1 --2- 172.21.15.89:0/1118537310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe11c1613e0 0x7fe11c1f3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe11aa00700 1 -- 172.21.15.89:0/1118537310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c0b3b30 msgr2=0x7fe11c0cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:06.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe11aa00700 1 --2- 172.21.15.89:0/1118537310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c0b3b30 0x7fe11c0cba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.080+0000 7fe11aa00700 1 -- 172.21.15.89:0/1118537310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c0094a0 con 0x7fe11c157f90 2024-04-02T09:12:06.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.081+0000 7fe11aa00700 1 --2- 172.21.15.89:0/1118537310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe11c157f90 0x7fe11c158ba0 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7fe10800c750 tx=0x7fe10800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:06.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.081+0000 7fe113400700 1 -- 172.21.15.89:0/1118537310 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe108005d70 con 0x7fe11c157f90 2024-04-02T09:12:06.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.081+0000 7fe113400700 1 -- 172.21.15.89:0/1118537310 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1080070d0 con 0x7fe11c157f90 2024-04-02T09:12:06.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.081+0000 7fe113400700 1 -- 172.21.15.89:0/1118537310 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe108007460 con 0x7fe11c157f90 2024-04-02T09:12:06.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.081+0000 7fe122000700 1 -- 172.21.15.89:0/1118537310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11c15c410 con 0x7fe11c157f90 2024-04-02T09:12:06.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.081+0000 7fe122000700 1 -- 172.21.15.89:0/1118537310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11c15c9d0 con 0x7fe11c157f90 2024-04-02T09:12:06.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.082+0000 7fe122000700 1 -- 172.21.15.89:0/1118537310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe11c067720 con 0x7fe11c157f90 2024-04-02T09:12:06.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.083+0000 7fe113400700 1 -- 172.21.15.89:0/1118537310 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fe1080075c0 con 0x7fe11c157f90 2024-04-02T09:12:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.084+0000 7fe113400700 1 --2- 172.21.15.89:0/1118537310 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe0fc06adb0 0x7fe0fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.084+0000 7fe11b400700 1 -- 172.21.15.89:0/1118537310 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe0fc06adb0 msgr2=0x7fe0fc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.084+0000 7fe113400700 1 -- 172.21.15.89:0/1118537310 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6559+0+0 (secure 0 0 0) 0x7fe108088bc0 con 0x7fe11c157f90 2024-04-02T09:12:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.084+0000 7fe11b400700 1 --2- 172.21.15.89:0/1118537310 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe0fc06adb0 0x7fe0fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.088+0000 7fe113400700 1 -- 172.21.15.89:0/1118537310 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe1080539d0 con 0x7fe11c157f90 2024-04-02T09:12:06.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.285+0000 7fe11b400700 1 -- 172.21.15.89:0/1118537310 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe0fc06adb0 msgr2=0x7fe0fc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:06.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.285+0000 7fe11b400700 1 --2- 172.21.15.89:0/1118537310 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe0fc06adb0 0x7fe0fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:06.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.317+0000 7fe122000700 1 -- 172.21.15.89:0/1118537310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe11c0b3070 con 0x7fe11c157f90 2024-04-02T09:12:06.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.318+0000 7fe113400700 1 -- 172.21.15.89:0/1118537310 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v208) v1 ==== 80+0+461 (secure 0 0 0) 0x7fe11c0b3070 con 0x7fe11c157f90 2024-04-02T09:12:06.317 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2552487672 2024-04-03T09:09:59.528553+0000 2024-04-02T09:12:06.317 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1282451945 2024-04-03T09:09:59.528553+0000 2024-04-02T09:12:06.317 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3725462960 2024-04-03T09:09:59.528553+0000 2024-04-02T09:12:06.317 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1115849275 2024-04-03T09:09:59.528553+0000 2024-04-02T09:12:06.317 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/108855 2024-04-03T09:09:59.528553+0000 2024-04-02T09:12:06.317 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/108855 2024-04-03T09:09:59.528553+0000 2024-04-02T09:12:06.317 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2251569695 2024-04-03T09:09:59.528553+0000 2024-04-02T09:12:06.317 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/248630547 2024-04-03T09:09:59.528553+0000 2024-04-02T09:12:06.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.320+0000 7fe122000700 1 -- 172.21.15.89:0/1118537310 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe0fc06adb0 msgr2=0x7fe0fc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:06.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.320+0000 7fe122000700 1 --2- 172.21.15.89:0/1118537310 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe0fc06adb0 0x7fe0fc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.320+0000 7fe122000700 1 -- 172.21.15.89:0/1118537310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe11c157f90 msgr2=0x7fe11c158ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:06.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.320+0000 7fe122000700 1 --2- 172.21.15.89:0/1118537310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe11c157f90 0x7fe11c158ba0 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7fe10800c750 tx=0x7fe10800ca60 comp rx=0 tx=0).stop 2024-04-02T09:12:06.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.320+0000 7fe122000700 1 -- 172.21.15.89:0/1118537310 shutdown_connections 2024-04-02T09:12:06.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.320+0000 7fe122000700 1 --2- 172.21.15.89:0/1118537310 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe0fc06adb0 0x7fe0fc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.320+0000 7fe122000700 1 --2- 172.21.15.89:0/1118537310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c0b3b30 0x7fe11c0cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.320+0000 7fe122000700 1 --2- 172.21.15.89:0/1118537310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe11c1613e0 0x7fe11c1f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.320+0000 7fe122000700 1 --2- 172.21.15.89:0/1118537310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe11c157f90 0x7fe11c158ba0 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.320+0000 7fe122000700 1 -- 172.21.15.89:0/1118537310 >> 172.21.15.89:0/1118537310 conn(0x7fe11c0b0d70 msgr2=0x7fe11c0afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:06.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.320+0000 7fe122000700 1 -- 172.21.15.89:0/1118537310 shutdown_connections 2024-04-02T09:12:06.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.320+0000 7fe122000700 1 -- 172.21.15.89:0/1118537310 wait complete. 2024-04-02T09:12:06.321 INFO:teuthology.orchestra.run.smithi089.stderr:listed 8 entries 2024-04-02T09:12:06.337 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:06.338+0000 7f7821bc7240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:12:06.338 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T09:12:06.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:06.339 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T09:12:06.416 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T09:12:06.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:06.416 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T09:12:06.429 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:06.431+0000 7f7821bc7240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:12:06.461 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:12:06.461 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:12:06.479 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:06.480+0000 7f0ca3caa240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:12:06.505 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:12:06.505 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:12:06.505 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:12:06.554 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:12:06.554 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:12:06.554 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:12:06.580 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:06.581+0000 7f0ca3caa240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:12:06.598 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:12:06.598 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:12:06.602 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:06.603+0000 7f7821bc7240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:12:06.758 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:06.759+0000 7f0ca3caa240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:12:06.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.804+0000 7fa6eac00700 1 Processor -- start 2024-04-02T09:12:06.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.804+0000 7fa6eac00700 1 -- start start 2024-04-02T09:12:06.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.805+0000 7fa6eac00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa6e415d710 0x7fa6e40d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:06.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.805+0000 7fa6eac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6e409aba0 0x7fa6e40d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:06.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.805+0000 7fa6eac00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa6e406bf40 0x7fa6e406c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:06.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.805+0000 7fa6eac00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e406c910 con 0x7fa6e415d710 2024-04-02T09:12:06.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.805+0000 7fa6eac00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e406bb40 con 0x7fa6e406bf40 2024-04-02T09:12:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.805+0000 7fa6eac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e4099b00 con 0x7fa6e409aba0 2024-04-02T09:12:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.805+0000 7fa6e3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa6e415d710 0x7fa6e40d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.806+0000 7fa6e3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa6e415d710 0x7fa6e40d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34052/0 (socket says 172.21.15.89:34052) 2024-04-02T09:12:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.806+0000 7fa6e3e00700 1 -- 172.21.15.89:0/3838056182 learned_addr learned my addr 172.21.15.89:0/3838056182 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.806+0000 7fa6e3400700 1 --2- 172.21.15.89:0/3838056182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6e409aba0 0x7fa6e40d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.806+0000 7fa6e8c00700 1 --2- 172.21.15.89:0/3838056182 >> v2:172.21.15.113:3300/0 conn(0x7fa6e406bf40 0x7fa6e406c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.806+0000 7fa6e3e00700 1 -- 172.21.15.89:0/3838056182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6e409aba0 msgr2=0x7fa6e40d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.806+0000 7fa6e3e00700 1 --2- 172.21.15.89:0/3838056182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6e409aba0 0x7fa6e40d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.806+0000 7fa6e3e00700 1 -- 172.21.15.89:0/3838056182 >> v2:172.21.15.113:3300/0 conn(0x7fa6e406bf40 msgr2=0x7fa6e406c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.806+0000 7fa6e3e00700 1 --2- 172.21.15.89:0/3838056182 >> v2:172.21.15.113:3300/0 conn(0x7fa6e406bf40 0x7fa6e406c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.806+0000 7fa6e3e00700 1 -- 172.21.15.89:0/3838056182 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e4099de0 con 0x7fa6e415d710 2024-04-02T09:12:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.806+0000 7fa6e3e00700 1 --2- 172.21.15.89:0/3838056182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6e415d710 0x7fa6e40d0130 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40cb9c0 tx=0x7fa6d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865a4b278a914546 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:06.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.806+0000 7fa6e2a00700 1 -- 172.21.15.89:0/3838056182 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4011830 con 0x7fa6e415d710 2024-04-02T09:12:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.807+0000 7fa6e2a00700 1 -- 172.21.15.89:0/3838056182 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d4011990 con 0x7fa6e415d710 2024-04-02T09:12:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.807+0000 7fa6e2a00700 1 -- 172.21.15.89:0/3838056182 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4011d20 con 0x7fa6e415d710 2024-04-02T09:12:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.807+0000 7fa6eac00700 1 -- 172.21.15.89:0/3838056182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6e415d710 msgr2=0x7fa6e40d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.807+0000 7fa6eac00700 1 --2- 172.21.15.89:0/3838056182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6e415d710 0x7fa6e40d0130 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40cb9c0 tx=0x7fa6d4009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.807+0000 7fa6eac00700 1 -- 172.21.15.89:0/3838056182 shutdown_connections 2024-04-02T09:12:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.807+0000 7fa6eac00700 1 --2- 172.21.15.89:0/3838056182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6e415d710 0x7fa6e40d0130 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.807+0000 7fa6eac00700 1 --2- 172.21.15.89:0/3838056182 >> v2:172.21.15.113:3300/0 conn(0x7fa6e406bf40 0x7fa6e406c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.807+0000 7fa6eac00700 1 --2- 172.21.15.89:0/3838056182 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6e409aba0 0x7fa6e40d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.807+0000 7fa6eac00700 1 -- 172.21.15.89:0/3838056182 >> 172.21.15.89:0/3838056182 conn(0x7fa6e40d13e0 msgr2=0x7fa6e40d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.808+0000 7fa6eac00700 1 -- 172.21.15.89:0/3838056182 shutdown_connections 2024-04-02T09:12:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.808+0000 7fa6eac00700 1 -- 172.21.15.89:0/3838056182 wait complete. 2024-04-02T09:12:06.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.810+0000 7fa6eac00700 1 Processor -- start 2024-04-02T09:12:06.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.811+0000 7fa6eac00700 1 -- start start 2024-04-02T09:12:06.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.812+0000 7fa6eac00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e406bf40 0x7fa6e41578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:06.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.812+0000 7fa6eac00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6e409aba0 0x7fa6e40c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:06.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.812+0000 7fa6eac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6e415d710 0x7fa6e415fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:06.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.812+0000 7fa6eac00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e415de50 con 0x7fa6e409aba0 2024-04-02T09:12:06.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.812+0000 7fa6eac00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e415e130 con 0x7fa6e406bf40 2024-04-02T09:12:06.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.812+0000 7fa6eac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e415e410 con 0x7fa6e415d710 2024-04-02T09:12:06.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.812+0000 7fa6e3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6e409aba0 0x7fa6e40c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:06.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.812+0000 7fa6e3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6e409aba0 0x7fa6e40c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34062/0 (socket says 172.21.15.89:34062) 2024-04-02T09:12:06.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.812+0000 7fa6e3400700 1 -- 172.21.15.89:0/2979562916 learned_addr learned my addr 172.21.15.89:0/2979562916 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:06.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.812+0000 7fa6e3e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e406bf40 0x7fa6e41578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:06.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.812+0000 7fa6e8c00700 1 --2- 172.21.15.89:0/2979562916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6e415d710 0x7fa6e415fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:06.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.813+0000 7fa6e3400700 1 -- 172.21.15.89:0/2979562916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6e415d710 msgr2=0x7fa6e415fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:06.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.813+0000 7fa6e3400700 1 --2- 172.21.15.89:0/2979562916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6e415d710 0x7fa6e415fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.813+0000 7fa6e3400700 1 -- 172.21.15.89:0/2979562916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e406bf40 msgr2=0x7fa6e41578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:06.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.813+0000 7fa6e3400700 1 --2- 172.21.15.89:0/2979562916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e406bf40 0x7fa6e41578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:06.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.813+0000 7fa6e3400700 1 -- 172.21.15.89:0/2979562916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d4009550 con 0x7fa6e409aba0 2024-04-02T09:12:06.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.813+0000 7fa6e3400700 1 --2- 172.21.15.89:0/2979562916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6e409aba0 0x7fa6e40c5850 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000e980 tx=0x7fa6d000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:06.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.813+0000 7fa6dbe00700 1 -- 172.21.15.89:0/2979562916 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0009710 con 0x7fa6e409aba0 2024-04-02T09:12:06.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.813+0000 7fa6dbe00700 1 -- 172.21.15.89:0/2979562916 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d000c690 con 0x7fa6e409aba0 2024-04-02T09:12:06.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.813+0000 7fa6eac00700 1 -- 172.21.15.89:0/2979562916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e415e6f0 con 0x7fa6e409aba0 2024-04-02T09:12:06.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.813+0000 7fa6dbe00700 1 -- 172.21.15.89:0/2979562916 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d000ca40 con 0x7fa6e409aba0 2024-04-02T09:12:06.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.813+0000 7fa6eac00700 1 -- 172.21.15.89:0/2979562916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e41f75a0 con 0x7fa6e409aba0 2024-04-02T09:12:06.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.815+0000 7fa6eac00700 1 -- 172.21.15.89:0/2979562916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e415ed40 con 0x7fa6e409aba0 2024-04-02T09:12:06.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.816+0000 7fa6dbe00700 1 -- 172.21.15.89:0/2979562916 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fa6d000cba0 con 0x7fa6e409aba0 2024-04-02T09:12:06.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.817+0000 7fa6dbe00700 1 --2- 172.21.15.89:0/2979562916 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa6c406adb0 0x7fa6c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:06.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.817+0000 7fa6e3e00700 1 -- 172.21.15.89:0/2979562916 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa6c406adb0 msgr2=0x7fa6c406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:06.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.817+0000 7fa6dbe00700 1 -- 172.21.15.89:0/2979562916 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6559+0+0 (secure 0 0 0) 0x7fa6d0089820 con 0x7fa6e409aba0 2024-04-02T09:12:06.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.817+0000 7fa6e3e00700 1 --2- 172.21.15.89:0/2979562916 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa6c406adb0 0x7fa6c406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:06.818 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:06.820+0000 7f7821bc7240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:12:06.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:06.821+0000 7fa6dbe00700 1 -- 172.21.15.89:0/2979562916 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa6d0054590 con 0x7fa6e409aba0 2024-04-02T09:12:06.910 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:06.911+0000 7f7821bc7240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:12:06.982 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:06.983+0000 7f0ca3caa240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:12:07.008 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:07.010+0000 7f7821bc7240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:12:07.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.018+0000 7fa6e3e00700 1 -- 172.21.15.89:0/2979562916 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa6c406adb0 msgr2=0x7fa6c406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:07.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.018+0000 7fa6e3e00700 1 --2- 172.21.15.89:0/2979562916 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa6c406adb0 0x7fa6c406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:07.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.075+0000 7fa6eac00700 1 -- 172.21.15.89:0/2979562916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6e4067720 con 0x7fa6e409aba0 2024-04-02T09:12:07.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.075+0000 7fa6dbe00700 1 -- 172.21.15.89:0/2979562916 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4748 (secure 0 0 0) 0x7fa6d0057bb0 con 0x7fa6e409aba0 2024-04-02T09:12:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:07.075 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1311594829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1311594829},{"type":"v1","addr":"172.21.15.113:6834","nonce":1311594829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1256547418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1256547418},{"type":"v1","addr":"172.21.15.89:6837","nonce":1256547418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1137044351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1137044351},{"type":"v1","addr":"172.21.15.113:6835","nonce":1137044351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:09:42.220191+0000","modified":"2024-04-02T09:09:45.699081+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10638":{"gid":10638,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/2257849268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2257849268},{"type":"v1","addr":"172.21.15.89:6836","nonce":2257849268}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-04-02T09:12:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.078+0000 7fa6eac00700 1 -- 172.21.15.89:0/2979562916 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa6c406adb0 msgr2=0x7fa6c406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.078+0000 7fa6eac00700 1 --2- 172.21.15.89:0/2979562916 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa6c406adb0 0x7fa6c406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.078+0000 7fa6eac00700 1 -- 172.21.15.89:0/2979562916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6e409aba0 msgr2=0x7fa6e40c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.078+0000 7fa6eac00700 1 --2- 172.21.15.89:0/2979562916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6e409aba0 0x7fa6e40c5850 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000e980 tx=0x7fa6d000ec90 comp rx=0 tx=0).stop 2024-04-02T09:12:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.078+0000 7fa6eac00700 1 -- 172.21.15.89:0/2979562916 shutdown_connections 2024-04-02T09:12:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.078+0000 7fa6eac00700 1 --2- 172.21.15.89:0/2979562916 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa6c406adb0 0x7fa6c406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.078+0000 7fa6eac00700 1 --2- 172.21.15.89:0/2979562916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e406bf40 0x7fa6e41578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.078+0000 7fa6eac00700 1 --2- 172.21.15.89:0/2979562916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6e415d710 0x7fa6e415fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.078+0000 7fa6eac00700 1 --2- 172.21.15.89:0/2979562916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6e409aba0 0x7fa6e40c5850 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.078+0000 7fa6eac00700 1 -- 172.21.15.89:0/2979562916 >> 172.21.15.89:0/2979562916 conn(0x7fa6e40d13e0 msgr2=0x7fa6e40736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.078+0000 7fa6eac00700 1 -- 172.21.15.89:0/2979562916 shutdown_connections 2024-04-02T09:12:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.078+0000 7fa6eac00700 1 -- 172.21.15.89:0/2979562916 wait complete. 2024-04-02T09:12:07.079 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:07.079+0000 7f0ca3caa240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:12:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 97 2024-04-02T09:12:07.096 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:12:07.106 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:07.108+0000 7f7821bc7240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:12:07.189 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:07.190+0000 7f0ca3caa240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:12:07.204 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:07.206+0000 7f7821bc7240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:12:07.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.293+0000 7f9c39600700 1 Processor -- start 2024-04-02T09:12:07.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.293+0000 7f9c39600700 1 -- start start 2024-04-02T09:12:07.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.294+0000 7f9c39600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9c340afbe0 0x7f9c3406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.294+0000 7f9c39600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3406f850 con 0x7f9c340cebd0 2024-04-02T09:12:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.294+0000 7f9c39600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3406fad0 con 0x7f9c3406a790 2024-04-02T09:12:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.294+0000 7f9c39600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c340cf710 con 0x7f9c340afbe0 2024-04-02T09:12:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.295+0000 7f9c33400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9c340cebd0 0x7f9c340cf000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32876/0 (socket says 172.21.15.89:32876) 2024-04-02T09:12:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.295+0000 7f9c33400700 1 -- 172.21.15.89:0/3788565174 learned_addr learned my addr 172.21.15.89:0/3788565174 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.295+0000 7f9c32000700 1 --2- 172.21.15.89:0/3788565174 >> v2:172.21.15.113:3300/0 conn(0x7f9c340afbe0 0x7f9c3406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.295+0000 7f9c31600700 1 -- 172.21.15.89:0/3788565174 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251586885 0 0) 0x7f9c3406f850 con 0x7f9c340cebd0 2024-04-02T09:12:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.295+0000 7f9c31600700 1 -- 172.21.15.89:0/3788565174 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c180035b0 con 0x7f9c340cebd0 2024-04-02T09:12:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.295+0000 7f9c32000700 1 -- 172.21.15.89:0/3788565174 >> v1:172.21.15.113:6789/0 conn(0x7f9c3406a790 legacy=0x7f9c3406ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.295+0000 7f9c32000700 1 -- 172.21.15.89:0/3788565174 >> v1:172.21.15.89:6789/0 conn(0x7f9c340cebd0 legacy=0x7f9c340cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.295+0000 7f9c32000700 1 -- 172.21.15.89:0/3788565174 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c340af970 con 0x7f9c340afbe0 2024-04-02T09:12:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.296+0000 7f9c32000700 1 --2- 172.21.15.89:0/3788565174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340afbe0 0x7f9c3406dce0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009840 tx=0x7f9c20009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16a2dc83e1ed6ea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.296+0000 7f9c31600700 1 -- 172.21.15.89:0/3788565174 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20011830 con 0x7f9c340afbe0 2024-04-02T09:12:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.296+0000 7f9c31600700 1 -- 172.21.15.89:0/3788565174 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c20011990 con 0x7f9c340afbe0 2024-04-02T09:12:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.296+0000 7f9c31600700 1 -- 172.21.15.89:0/3788565174 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20011d20 con 0x7f9c340afbe0 2024-04-02T09:12:07.298 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:07.296+0000 7f0ca3caa240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:12:07.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.296+0000 7f9c39600700 1 -- 172.21.15.89:0/3788565174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340afbe0 msgr2=0x7f9c3406dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:07.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.296+0000 7f9c39600700 1 --2- 172.21.15.89:0/3788565174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340afbe0 0x7f9c3406dce0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009840 tx=0x7f9c20009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:07.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.296+0000 7f9c39600700 1 -- 172.21.15.89:0/3788565174 shutdown_connections 2024-04-02T09:12:07.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.296+0000 7f9c39600700 1 --2- 172.21.15.89:0/3788565174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340afbe0 0x7f9c3406dce0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.296+0000 7f9c39600700 1 -- 172.21.15.89:0/3788565174 >> 172.21.15.89:0/3788565174 conn(0x7f9c340b5ab0 msgr2=0x7f9c340b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:07.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.297+0000 7f9c39600700 1 -- 172.21.15.89:0/3788565174 shutdown_connections 2024-04-02T09:12:07.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.297+0000 7f9c39600700 1 -- 172.21.15.89:0/3788565174 wait complete. 2024-04-02T09:12:07.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.298+0000 7f9c39600700 1 Processor -- start 2024-04-02T09:12:07.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.298+0000 7f9c39600700 1 -- start start 2024-04-02T09:12:07.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.299+0000 7f9c39600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c3406a790 0x7f9c34146580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:07.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.299+0000 7f9c39600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c340afbe0 0x7f9c34148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:07.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.299+0000 7f9c39600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340cebd0 0x7f9c34149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:07.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.299+0000 7f9c39600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c34146cb0 con 0x7f9c3406a790 2024-04-02T09:12:07.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.299+0000 7f9c39600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c34147550 con 0x7f9c340cebd0 2024-04-02T09:12:07.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.299+0000 7f9c39600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c34155840 con 0x7f9c340afbe0 2024-04-02T09:12:07.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.300+0000 7f9c32a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c3406a790 0x7f9c34146580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:07.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.300+0000 7f9c32a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c3406a790 0x7f9c34146580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34064/0 (socket says 172.21.15.89:34064) 2024-04-02T09:12:07.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.300+0000 7f9c32a00700 1 -- 172.21.15.89:0/3485521210 learned_addr learned my addr 172.21.15.89:0/3485521210 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:07.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.300+0000 7f9c32000700 1 --2- 172.21.15.89:0/3485521210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c340afbe0 0x7f9c34148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:07.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.300+0000 7f9c33400700 1 --2- 172.21.15.89:0/3485521210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340cebd0 0x7f9c34149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:07.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.300+0000 7f9c32a00700 1 -- 172.21.15.89:0/3485521210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c340afbe0 msgr2=0x7f9c34148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:07.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.300+0000 7f9c32a00700 1 --2- 172.21.15.89:0/3485521210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c340afbe0 0x7f9c34148d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.300+0000 7f9c32a00700 1 -- 172.21.15.89:0/3485521210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340cebd0 msgr2=0x7f9c34149280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:07.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.300+0000 7f9c32a00700 1 --2- 172.21.15.89:0/3485521210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340cebd0 0x7f9c34149280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.300+0000 7f9c32a00700 1 -- 172.21.15.89:0/3485521210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c200094a0 con 0x7f9c3406a790 2024-04-02T09:12:07.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.300+0000 7f9c32a00700 1 --2- 172.21.15.89:0/3485521210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c3406a790 0x7f9c34146580 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f9c2400c480 tx=0x7f9c2400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:07.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.300+0000 7f9c2aa00700 1 -- 172.21.15.89:0/3485521210 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c240052d0 con 0x7f9c3406a790 2024-04-02T09:12:07.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.301+0000 7f9c2aa00700 1 -- 172.21.15.89:0/3485521210 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c24005430 con 0x7f9c3406a790 2024-04-02T09:12:07.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.301+0000 7f9c2aa00700 1 -- 172.21.15.89:0/3485521210 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c240038c0 con 0x7f9c3406a790 2024-04-02T09:12:07.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.301+0000 7f9c39600700 1 -- 172.21.15.89:0/3485521210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c340b4650 con 0x7f9c3406a790 2024-04-02T09:12:07.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.301+0000 7f9c39600700 1 -- 172.21.15.89:0/3485521210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c341471e0 con 0x7f9c3406a790 2024-04-02T09:12:07.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.302+0000 7f9c39600700 1 -- 172.21.15.89:0/3485521210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c34067720 con 0x7f9c3406a790 2024-04-02T09:12:07.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.303+0000 7f9c2aa00700 1 -- 172.21.15.89:0/3485521210 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f9c24003a20 con 0x7f9c3406a790 2024-04-02T09:12:07.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.304+0000 7f9c2aa00700 1 --2- 172.21.15.89:0/3485521210 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9c1406adb0 0x7f9c1406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:07.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.304+0000 7f9c2aa00700 1 -- 172.21.15.89:0/3485521210 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f9c24088b00 con 0x7f9c3406a790 2024-04-02T09:12:07.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.304+0000 7f9c32000700 1 -- 172.21.15.89:0/3485521210 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9c1406adb0 msgr2=0x7f9c1406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:07.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.304+0000 7f9c32000700 1 --2- 172.21.15.89:0/3485521210 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9c1406adb0 0x7f9c1406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:07.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.308+0000 7f9c2aa00700 1 -- 172.21.15.89:0/3485521210 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9c24053880 con 0x7f9c3406a790 2024-04-02T09:12:07.401 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:07.402+0000 7f0ca3caa240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:12:07.417 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:07.419+0000 7f7821bc7240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:12:07.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.505+0000 7f9c32000700 1 -- 172.21.15.89:0/3485521210 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9c1406adb0 msgr2=0x7f9c1406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:07.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.505+0000 7f9c32000700 1 --2- 172.21.15.89:0/3485521210 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9c1406adb0 0x7f9c1406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:07.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.519+0000 7f9c39600700 1 -- 172.21.15.89:0/3485521210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c340af030 con 0x7f9c3406a790 2024-04-02T09:12:07.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.520+0000 7f9c2aa00700 1 -- 172.21.15.89:0/3485521210 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) v1 ==== 74+0+17337 (secure 0 0 0) 0x7f9c24017090 con 0x7f9c3406a790 2024-04-02T09:12:07.520 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:07.520 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":208,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:11:01.188455+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:09:40.103721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-04-02T09:09:41.165944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.5","mappings":[{"from":7,"to":3}]},{"pgid":"25.3b","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/2552487672":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/1282451945":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/3725462960":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/1115849275":"2024-04-03T09:09:59.528553+0000","172.21.15.89:6833/108855":"2024-04-03T09:09:59.528553+0000","172.21.15.89:6832/108855":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/2251569695":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/248630547":"2024-04-03T09:09:59.528553+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:12:07.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.523+0000 7f9c39600700 1 -- 172.21.15.89:0/3485521210 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9c1406adb0 msgr2=0x7f9c1406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:07.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.523+0000 7f9c39600700 1 --2- 172.21.15.89:0/3485521210 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9c1406adb0 0x7f9c1406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.523+0000 7f9c39600700 1 -- 172.21.15.89:0/3485521210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c3406a790 msgr2=0x7f9c34146580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:07.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.523+0000 7f9c39600700 1 --2- 172.21.15.89:0/3485521210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c3406a790 0x7f9c34146580 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f9c2400c480 tx=0x7f9c2400c790 comp rx=0 tx=0).stop 2024-04-02T09:12:07.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.523+0000 7f9c39600700 1 -- 172.21.15.89:0/3485521210 shutdown_connections 2024-04-02T09:12:07.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.523+0000 7f9c39600700 1 --2- 172.21.15.89:0/3485521210 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9c1406adb0 0x7f9c1406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.523+0000 7f9c39600700 1 --2- 172.21.15.89:0/3485521210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c340cebd0 0x7f9c34149280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.523+0000 7f9c39600700 1 --2- 172.21.15.89:0/3485521210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c3406a790 0x7f9c34146580 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.523+0000 7f9c39600700 1 --2- 172.21.15.89:0/3485521210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c340afbe0 0x7f9c34148d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.523+0000 7f9c39600700 1 -- 172.21.15.89:0/3485521210 >> 172.21.15.89:0/3485521210 conn(0x7f9c340b5ab0 msgr2=0x7f9c34159b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:07.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.523+0000 7f9c39600700 1 -- 172.21.15.89:0/3485521210 shutdown_connections 2024-04-02T09:12:07.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.523+0000 7f9c39600700 1 -- 172.21.15.89:0/3485521210 wait complete. 2024-04-02T09:12:07.542 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T09:09:40.103721+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '204', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T09:09:41.165944+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '205', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:12:07.543 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:12:07.629 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:07.630+0000 7f0ca3caa240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:12:07.678 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:07.679+0000 7f7821bc7240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:12:07.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.753+0000 7f5b09200700 1 Processor -- start 2024-04-02T09:12:07.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.753+0000 7f5b09200700 1 -- start start 2024-04-02T09:12:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.754+0000 7f5b09200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b04145050 0x7f5b040d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.754+0000 7f5b09200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5b040afa80 0x7f5b040b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.754+0000 7f5b09200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b040d1100 con 0x7f5b040af600 2024-04-02T09:12:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.754+0000 7f5b09200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b040aeb60 con 0x7f5b040afa80 2024-04-02T09:12:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.754+0000 7f5b09200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b040b5190 con 0x7f5b04145050 2024-04-02T09:12:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.755+0000 7f5b02800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5b040afa80 0x7f5b040b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.755+0000 7f5b01e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b04145050 0x7f5b040d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.755+0000 7f5b02800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5b040afa80 0x7f5b040b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:46624/0 (socket says 172.21.15.89:46624) 2024-04-02T09:12:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.755+0000 7f5b02800700 1 -- 172.21.15.89:0/37447597 learned_addr learned my addr 172.21.15.89:0/37447597 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:07.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.755+0000 7f5b01400700 1 --1- 172.21.15.89:0/37447597 >> v1:172.21.15.113:6789/0 conn(0x7f5b040af600 0x7f5b0406a790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:51492/0 (socket says 172.21.15.89:51492) 2024-04-02T09:12:07.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.755+0000 7f5b02800700 1 -- 172.21.15.89:0/37447597 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b04145050 msgr2=0x7f5b040d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:07.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.755+0000 7f5b02800700 1 --2- 172.21.15.89:0/37447597 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b04145050 0x7f5b040d0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.755+0000 7f5b02800700 1 -- 172.21.15.89:0/37447597 >> v1:172.21.15.113:6789/0 conn(0x7f5b040af600 legacy=0x7f5b0406a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:07.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.755+0000 7f5b02800700 1 -- 172.21.15.89:0/37447597 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b040aee00 con 0x7f5b040afa80 2024-04-02T09:12:07.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.755+0000 7f5b01e00700 1 --2- 172.21.15.89:0/37447597 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b04145050 0x7f5b040d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:12:07.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.755+0000 7f5b02800700 1 --2- 172.21.15.89:0/37447597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b040afa80 0x7f5b040b3200 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5afc009e50 tx=0x7f5afc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54aaa4be4ba27803 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:07.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.756+0000 7f5b00a00700 1 -- 172.21.15.89:0/37447597 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc00e070 con 0x7f5b040afa80 2024-04-02T09:12:07.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.756+0000 7f5b00a00700 1 -- 172.21.15.89:0/37447597 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5afc00b910 con 0x7f5b040afa80 2024-04-02T09:12:07.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.756+0000 7f5b00a00700 1 -- 172.21.15.89:0/37447597 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc00bcc0 con 0x7f5b040afa80 2024-04-02T09:12:07.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.756+0000 7f5b09200700 1 -- 172.21.15.89:0/37447597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b040afa80 msgr2=0x7f5b040b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:07.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.756+0000 7f5b09200700 1 --2- 172.21.15.89:0/37447597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b040afa80 0x7f5b040b3200 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5afc009e50 tx=0x7f5afc00b330 comp rx=0 tx=0).stop 2024-04-02T09:12:07.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.757+0000 7f5b09200700 1 -- 172.21.15.89:0/37447597 shutdown_connections 2024-04-02T09:12:07.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.757+0000 7f5b09200700 1 --2- 172.21.15.89:0/37447597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b040afa80 0x7f5b040b3200 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.757+0000 7f5b09200700 1 --2- 172.21.15.89:0/37447597 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b04145050 0x7f5b040d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.757+0000 7f5b09200700 1 -- 172.21.15.89:0/37447597 >> 172.21.15.89:0/37447597 conn(0x7f5b040b5720 msgr2=0x7f5b040b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:07.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.757+0000 7f5b09200700 1 -- 172.21.15.89:0/37447597 shutdown_connections 2024-04-02T09:12:07.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.757+0000 7f5b09200700 1 -- 172.21.15.89:0/37447597 wait complete. 2024-04-02T09:12:07.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.758+0000 7f5b09200700 1 Processor -- start 2024-04-02T09:12:07.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.759+0000 7f5b09200700 1 -- start start 2024-04-02T09:12:07.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.759+0000 7f5b09200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b040af600 0x7f5b041f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:07.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.759+0000 7f5b09200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b04145050 0x7f5b04140b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:07.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.759+0000 7f5b09200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b04135350 0x7f5b041357d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:07.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.759+0000 7f5b09200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b04139130 con 0x7f5b04135350 2024-04-02T09:12:07.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.759+0000 7f5b09200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b041385b0 con 0x7f5b04145050 2024-04-02T09:12:07.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.759+0000 7f5b09200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b04138890 con 0x7f5b040af600 2024-04-02T09:12:07.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.760+0000 7f5b02800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b04135350 0x7f5b041357d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:07.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.760+0000 7f5b01400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b04145050 0x7f5b04140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:07.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.760+0000 7f5b02800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b04135350 0x7f5b041357d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34080/0 (socket says 172.21.15.89:34080) 2024-04-02T09:12:07.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.760+0000 7f5b02800700 1 -- 172.21.15.89:0/2724398690 learned_addr learned my addr 172.21.15.89:0/2724398690 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:07.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.760+0000 7f5b01e00700 1 --2- 172.21.15.89:0/2724398690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b040af600 0x7f5b041f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:07.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.760+0000 7f5b01400700 1 -- 172.21.15.89:0/2724398690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b040af600 msgr2=0x7f5b041f6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:07.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.760+0000 7f5b01400700 1 --2- 172.21.15.89:0/2724398690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b040af600 0x7f5b041f6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.760+0000 7f5b01400700 1 -- 172.21.15.89:0/2724398690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b04135350 msgr2=0x7f5b041357d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:07.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.760+0000 7f5b01400700 1 --2- 172.21.15.89:0/2724398690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b04135350 0x7f5b041357d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.760+0000 7f5b01400700 1 -- 172.21.15.89:0/2724398690 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af40094a0 con 0x7f5b04145050 2024-04-02T09:12:07.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.760+0000 7f5b01400700 1 --2- 172.21.15.89:0/2724398690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b04145050 0x7f5b04140b50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f5af000c450 tx=0x7f5af000c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:07.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.761+0000 7f5afa000700 1 -- 172.21.15.89:0/2724398690 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af0005cb0 con 0x7f5b04145050 2024-04-02T09:12:07.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.761+0000 7f5afa000700 1 -- 172.21.15.89:0/2724398690 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af0005e10 con 0x7f5b04145050 2024-04-02T09:12:07.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.761+0000 7f5afa000700 1 -- 172.21.15.89:0/2724398690 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af00072d0 con 0x7f5b04145050 2024-04-02T09:12:07.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.761+0000 7f5b09200700 1 -- 172.21.15.89:0/2724398690 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc009b30 con 0x7f5b04145050 2024-04-02T09:12:07.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.761+0000 7f5b09200700 1 -- 172.21.15.89:0/2724398690 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b04138e50 con 0x7f5b04145050 2024-04-02T09:12:07.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.762+0000 7f5b09200700 1 -- 172.21.15.89:0/2724398690 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b04067720 con 0x7f5b04145050 2024-04-02T09:12:07.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.763+0000 7f5afa000700 1 -- 172.21.15.89:0/2724398690 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f5af0007430 con 0x7f5b04145050 2024-04-02T09:12:07.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.764+0000 7f5afa000700 1 --2- 172.21.15.89:0/2724398690 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ae406adb0 0x7f5ae406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:07.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.764+0000 7f5afa000700 1 -- 172.21.15.89:0/2724398690 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6615+0+0 (secure 0 0 0) 0x7f5af0088a40 con 0x7f5b04145050 2024-04-02T09:12:07.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.764+0000 7f5b01e00700 1 -- 172.21.15.89:0/2724398690 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ae406adb0 msgr2=0x7f5ae406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:07.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.764+0000 7f5b01e00700 1 --2- 172.21.15.89:0/2724398690 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ae406adb0 0x7f5ae406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:07.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.768+0000 7f5afa000700 1 -- 172.21.15.89:0/2724398690 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5af0053820 con 0x7f5b04145050 2024-04-02T09:12:07.879 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:07.880+0000 7f7821bc7240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:12:07.908 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:07.909+0000 7f0ca3caa240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:12:07.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.965+0000 7f5b01e00700 1 -- 172.21.15.89:0/2724398690 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ae406adb0 msgr2=0x7f5ae406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:07.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.965+0000 7f5b01e00700 1 --2- 172.21.15.89:0/2724398690 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ae406adb0 0x7f5ae406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:07.972 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:07.973+0000 7f7821bc7240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:12:07.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.990+0000 7f5b09200700 1 -- 172.21.15.89:0/2724398690 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5b040b4d40 con 0x7f5b04145050 2024-04-02T09:12:07.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.991+0000 7f5afa000700 1 -- 172.21.15.89:0/2724398690 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v209) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5af00170a0 con 0x7f5b04145050 2024-04-02T09:12:07.990 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:12:07.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.993+0000 7f5b09200700 1 -- 172.21.15.89:0/2724398690 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ae406adb0 msgr2=0x7f5ae406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:07.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.993+0000 7f5b09200700 1 --2- 172.21.15.89:0/2724398690 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ae406adb0 0x7f5ae406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.993+0000 7f5b09200700 1 -- 172.21.15.89:0/2724398690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b04145050 msgr2=0x7f5b04140b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:07.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.993+0000 7f5b09200700 1 --2- 172.21.15.89:0/2724398690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b04145050 0x7f5b04140b50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f5af000c450 tx=0x7f5af000c760 comp rx=0 tx=0).stop 2024-04-02T09:12:07.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.993+0000 7f5b09200700 1 -- 172.21.15.89:0/2724398690 shutdown_connections 2024-04-02T09:12:07.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.993+0000 7f5b09200700 1 --2- 172.21.15.89:0/2724398690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b04135350 0x7f5b041357d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.993+0000 7f5b09200700 1 --2- 172.21.15.89:0/2724398690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b040af600 0x7f5b041f6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.993+0000 7f5b09200700 1 --2- 172.21.15.89:0/2724398690 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ae406adb0 0x7f5ae406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.993+0000 7f5b09200700 1 --2- 172.21.15.89:0/2724398690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b04145050 0x7f5b04140b50 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:07.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.993+0000 7f5b09200700 1 -- 172.21.15.89:0/2724398690 >> 172.21.15.89:0/2724398690 conn(0x7f5b040b5720 msgr2=0x7f5b0406a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:07.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.993+0000 7f5b09200700 1 -- 172.21.15.89:0/2724398690 shutdown_connections 2024-04-02T09:12:07.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:07.993+0000 7f5b09200700 1 -- 172.21.15.89:0/2724398690 wait complete. 2024-04-02T09:12:08.010 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T09:12:08.120 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:08.121+0000 7f0ca3caa240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:12:08.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.209+0000 7fb40d400700 1 Processor -- start 2024-04-02T09:12:08.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.209+0000 7fb40d400700 1 -- start start 2024-04-02T09:12:08.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.211+0000 7fb40d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4080d2540 0x7fb4081687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:08.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.211+0000 7fb40d400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb408161420 con 0x7fb4080d16a0 2024-04-02T09:12:08.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.211+0000 7fb40d400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb408161700 con 0x7fb40815d9d0 2024-04-02T09:12:08.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.211+0000 7fb40d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4080d4340 con 0x7fb4080d2540 2024-04-02T09:12:08.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.211+0000 7fb405e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb4080d16a0 0x7fb4080d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32892/0 (socket says 172.21.15.89:32892) 2024-04-02T09:12:08.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.211+0000 7fb405e00700 1 -- 172.21.15.89:0/3371285322 learned_addr learned my addr 172.21.15.89:0/3371285322 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:08.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.211+0000 7fb407200700 1 --2- 172.21.15.89:0/3371285322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4080d2540 0x7fb4081687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:08.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.211+0000 7fb405400700 1 -- 172.21.15.89:0/3371285322 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336379336 0 0) 0x7fb408161420 con 0x7fb4080d16a0 2024-04-02T09:12:08.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.211+0000 7fb405400700 1 -- 172.21.15.89:0/3371285322 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3ec0035b0 con 0x7fb4080d16a0 2024-04-02T09:12:08.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.211+0000 7fb405400700 1 -- 172.21.15.89:0/3371285322 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809992034 0 0) 0x7fb408161700 con 0x7fb40815d9d0 2024-04-02T09:12:08.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.211+0000 7fb405400700 1 -- 172.21.15.89:0/3371285322 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb408161420 con 0x7fb40815d9d0 2024-04-02T09:12:08.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.211+0000 7fb407200700 1 -- 172.21.15.89:0/3371285322 >> v1:172.21.15.113:6789/0 conn(0x7fb40815d9d0 legacy=0x7fb4080d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:08.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.211+0000 7fb407200700 1 -- 172.21.15.89:0/3371285322 >> v1:172.21.15.89:6789/0 conn(0x7fb4080d16a0 legacy=0x7fb4080d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:08.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.211+0000 7fb407200700 1 -- 172.21.15.89:0/3371285322 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4081619e0 con 0x7fb4080d2540 2024-04-02T09:12:08.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.212+0000 7fb407200700 1 --2- 172.21.15.89:0/3371285322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4080d2540 0x7fb4081687b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb400009dc0 tx=0x7fb40000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=277fe4445cc77378 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:08.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.212+0000 7fb405400700 1 -- 172.21.15.89:0/3371285322 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40000e070 con 0x7fb4080d2540 2024-04-02T09:12:08.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.212+0000 7fb405400700 1 -- 172.21.15.89:0/3371285322 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40000bb20 con 0x7fb4080d2540 2024-04-02T09:12:08.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.212+0000 7fb405400700 1 -- 172.21.15.89:0/3371285322 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb400012430 con 0x7fb4080d2540 2024-04-02T09:12:08.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.212+0000 7fb40d400700 1 -- 172.21.15.89:0/3371285322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4080d2540 msgr2=0x7fb4081687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:08.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.212+0000 7fb40d400700 1 --2- 172.21.15.89:0/3371285322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4080d2540 0x7fb4081687b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb400009dc0 tx=0x7fb40000b330 comp rx=0 tx=0).stop 2024-04-02T09:12:08.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.213+0000 7fb40d400700 1 -- 172.21.15.89:0/3371285322 shutdown_connections 2024-04-02T09:12:08.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.213+0000 7fb40d400700 1 --2- 172.21.15.89:0/3371285322 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4080d2540 0x7fb4081687b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.213+0000 7fb40d400700 1 -- 172.21.15.89:0/3371285322 >> 172.21.15.89:0/3371285322 conn(0x7fb40808aff0 msgr2=0x7fb4080d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:08.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.213+0000 7fb40d400700 1 -- 172.21.15.89:0/3371285322 shutdown_connections 2024-04-02T09:12:08.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.213+0000 7fb40d400700 1 -- 172.21.15.89:0/3371285322 wait complete. 2024-04-02T09:12:08.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.214+0000 7fb40d400700 1 Processor -- start 2024-04-02T09:12:08.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.214+0000 7fb40d400700 1 -- start start 2024-04-02T09:12:08.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.215+0000 7fb40d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4080d16a0 0x7fb40815c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:08.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.215+0000 7fb40d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4080d2540 0x7fb40815c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:08.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.215+0000 7fb40d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb40815d9d0 0x7fb40815f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:08.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.215+0000 7fb40d400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40815c560 con 0x7fb4080d2540 2024-04-02T09:12:08.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.215+0000 7fb40d400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40815e370 con 0x7fb40815d9d0 2024-04-02T09:12:08.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.215+0000 7fb40d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40815e650 con 0x7fb4080d16a0 2024-04-02T09:12:08.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.216+0000 7fb405e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4080d2540 0x7fb40815c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:08.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.216+0000 7fb405e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4080d2540 0x7fb40815c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34090/0 (socket says 172.21.15.89:34090) 2024-04-02T09:12:08.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.216+0000 7fb405e00700 1 -- 172.21.15.89:0/1924905057 learned_addr learned my addr 172.21.15.89:0/1924905057 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:08.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.216+0000 7fb406800700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4080d16a0 0x7fb40815c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:08.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.216+0000 7fb407200700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb40815d9d0 0x7fb40815f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:08.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.216+0000 7fb405e00700 1 -- 172.21.15.89:0/1924905057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4080d16a0 msgr2=0x7fb40815c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:08.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.216+0000 7fb405e00700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4080d16a0 0x7fb40815c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.216+0000 7fb405e00700 1 -- 172.21.15.89:0/1924905057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb40815d9d0 msgr2=0x7fb40815f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:08.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.216+0000 7fb405e00700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb40815d9d0 0x7fb40815f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.216+0000 7fb405e00700 1 -- 172.21.15.89:0/1924905057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb400009a70 con 0x7fb4080d2540 2024-04-02T09:12:08.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.216+0000 7fb406800700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4080d16a0 0x7fb40815c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:12:08.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.216+0000 7fb405e00700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4080d2540 0x7fb40815c020 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4001260 tx=0x7fb3f400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:08.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.217+0000 7fb3fea00700 1 -- 172.21.15.89:0/1924905057 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f40074e0 con 0x7fb4080d2540 2024-04-02T09:12:08.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.217+0000 7fb3fea00700 1 -- 172.21.15.89:0/1924905057 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f4007640 con 0x7fb4080d2540 2024-04-02T09:12:08.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.217+0000 7fb40d400700 1 -- 172.21.15.89:0/1924905057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb40815e930 con 0x7fb4080d2540 2024-04-02T09:12:08.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.217+0000 7fb3fea00700 1 -- 172.21.15.89:0/1924905057 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4005a70 con 0x7fb4080d2540 2024-04-02T09:12:08.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.217+0000 7fb40d400700 1 -- 172.21.15.89:0/1924905057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40815eef0 con 0x7fb4080d2540 2024-04-02T09:12:08.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.218+0000 7fb40d400700 1 -- 172.21.15.89:0/1924905057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4080d1ca0 con 0x7fb4080d2540 2024-04-02T09:12:08.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.219+0000 7fb3fea00700 1 -- 172.21.15.89:0/1924905057 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fb3f4005bd0 con 0x7fb4080d2540 2024-04-02T09:12:08.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.220+0000 7fb3fea00700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb3e806adb0 0x7fb3e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:08.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.220+0000 7fb3fea00700 1 -- 172.21.15.89:0/1924905057 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6615+0+0 (secure 0 0 0) 0x7fb3f4088c90 con 0x7fb4080d2540 2024-04-02T09:12:08.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.220+0000 7fb406800700 1 -- 172.21.15.89:0/1924905057 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb3e806adb0 msgr2=0x7fb3e806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:08.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.220+0000 7fb406800700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb3e806adb0 0x7fb3e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:08.225 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:08.226+0000 7f0ca3caa240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:12:08.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.228+0000 7fb3fea00700 1 -- 172.21.15.89:0/1924905057 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb3f40539c0 con 0x7fb4080d2540 2024-04-02T09:12:08.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.421+0000 7fb406800700 1 -- 172.21.15.89:0/1924905057 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb3e806adb0 msgr2=0x7fb3e806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:08.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.421+0000 7fb406800700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb3e806adb0 0x7fb3e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:08.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.439+0000 7fb40d400700 1 -- 172.21.15.89:0/1924905057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb408067720 con 0x7fb4080d2540 2024-04-02T09:12:08.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.440+0000 7fb3fea00700 1 -- 172.21.15.89:0/1924905057 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v209) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb3f40170a0 con 0x7fb4080d2540 2024-04-02T09:12:08.439 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:12:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.443+0000 7fb40d400700 1 -- 172.21.15.89:0/1924905057 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb3e806adb0 msgr2=0x7fb3e806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.443+0000 7fb40d400700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb3e806adb0 0x7fb3e806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.443+0000 7fb40d400700 1 -- 172.21.15.89:0/1924905057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4080d2540 msgr2=0x7fb40815c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.443+0000 7fb40d400700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4080d2540 0x7fb40815c020 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4001260 tx=0x7fb3f400c9e0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.443+0000 7fb40d400700 1 -- 172.21.15.89:0/1924905057 shutdown_connections 2024-04-02T09:12:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.443+0000 7fb40d400700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb3e806adb0 0x7fb3e806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.443+0000 7fb40d400700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb40815d9d0 0x7fb40815f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.443+0000 7fb40d400700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4080d16a0 0x7fb40815c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.443+0000 7fb40d400700 1 --2- 172.21.15.89:0/1924905057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4080d2540 0x7fb40815c020 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.443+0000 7fb40d400700 1 -- 172.21.15.89:0/1924905057 >> 172.21.15.89:0/1924905057 conn(0x7fb40808aff0 msgr2=0x7fb4080b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:08.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.443+0000 7fb40d400700 1 -- 172.21.15.89:0/1924905057 shutdown_connections 2024-04-02T09:12:08.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.443+0000 7fb40d400700 1 -- 172.21.15.89:0/1924905057 wait complete. 2024-04-02T09:12:08.460 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T09:12:08.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.659+0000 7fbad8a00700 1 Processor -- start 2024-04-02T09:12:08.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.659+0000 7fbad8a00700 1 -- start start 2024-04-02T09:12:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.660+0000 7fbad8a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbad415d9d0 0x7fbad40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.660+0000 7fbad8a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbad40d16a0 0x7fbad40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.660+0000 7fbad8a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbad40d2370 0x7fbad415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.660+0000 7fbad8a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad40d2820 con 0x7fbad40d2370 2024-04-02T09:12:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.660+0000 7fbad8a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad4161420 con 0x7fbad40d16a0 2024-04-02T09:12:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.660+0000 7fbad8a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad4161700 con 0x7fbad415d9d0 2024-04-02T09:12:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.661+0000 7fbad1e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbad415d9d0 0x7fbad40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.661+0000 7fbad1e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbad415d9d0 0x7fbad40d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:49030/0 (socket says 172.21.15.89:49030) 2024-04-02T09:12:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.661+0000 7fbad1e00700 1 -- 172.21.15.89:0/3914779732 learned_addr learned my addr 172.21.15.89:0/3914779732 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.661+0000 7fbad1400700 1 --2- 172.21.15.89:0/3914779732 >> v2:172.21.15.113:3300/0 conn(0x7fbad40d16a0 0x7fbad40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.661+0000 7fbad2800700 1 --2- 172.21.15.89:0/3914779732 >> v2:172.21.15.89:3300/0 conn(0x7fbad40d2370 0x7fbad415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.661+0000 7fbad1e00700 1 -- 172.21.15.89:0/3914779732 >> v2:172.21.15.113:3300/0 conn(0x7fbad40d16a0 msgr2=0x7fbad40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.661+0000 7fbad1e00700 1 --2- 172.21.15.89:0/3914779732 >> v2:172.21.15.113:3300/0 conn(0x7fbad40d16a0 0x7fbad40d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.661+0000 7fbad1e00700 1 -- 172.21.15.89:0/3914779732 >> v2:172.21.15.89:3300/0 conn(0x7fbad40d2370 msgr2=0x7fbad415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.661+0000 7fbad1e00700 1 --2- 172.21.15.89:0/3914779732 >> v2:172.21.15.89:3300/0 conn(0x7fbad40d2370 0x7fbad415ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.661+0000 7fbad1e00700 1 -- 172.21.15.89:0/3914779732 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad41619e0 con 0x7fbad415d9d0 2024-04-02T09:12:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.661+0000 7fbad1400700 1 --2- 172.21.15.89:0/3914779732 >> v2:172.21.15.113:3300/0 conn(0x7fbad40d16a0 0x7fbad40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:12:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.661+0000 7fbad2800700 1 --2- 172.21.15.89:0/3914779732 >> v2:172.21.15.89:3300/0 conn(0x7fbad40d2370 0x7fbad415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.661+0000 7fbad1e00700 1 --2- 172.21.15.89:0/3914779732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbad415d9d0 0x7fbad40d3c30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbad406b5e0 tx=0x7fbac4009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a6ea48c62312b88 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.662+0000 7fbad0a00700 1 -- 172.21.15.89:0/3914779732 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4011830 con 0x7fbad415d9d0 2024-04-02T09:12:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.662+0000 7fbad0a00700 1 -- 172.21.15.89:0/3914779732 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac4011990 con 0x7fbad415d9d0 2024-04-02T09:12:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.662+0000 7fbad0a00700 1 -- 172.21.15.89:0/3914779732 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4011d20 con 0x7fbad415d9d0 2024-04-02T09:12:08.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.662+0000 7fbad8a00700 1 -- 172.21.15.89:0/3914779732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbad415d9d0 msgr2=0x7fbad40d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.662+0000 7fbad8a00700 1 --2- 172.21.15.89:0/3914779732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbad415d9d0 0x7fbad40d3c30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbad406b5e0 tx=0x7fbac4009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.662+0000 7fbad8a00700 1 -- 172.21.15.89:0/3914779732 shutdown_connections 2024-04-02T09:12:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.662+0000 7fbad8a00700 1 --2- 172.21.15.89:0/3914779732 >> v2:172.21.15.89:3300/0 conn(0x7fbad40d2370 0x7fbad415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.662+0000 7fbad8a00700 1 --2- 172.21.15.89:0/3914779732 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbad415d9d0 0x7fbad40d3c30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.662+0000 7fbad8a00700 1 --2- 172.21.15.89:0/3914779732 >> v2:172.21.15.113:3300/0 conn(0x7fbad40d16a0 0x7fbad40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.662+0000 7fbad8a00700 1 -- 172.21.15.89:0/3914779732 >> 172.21.15.89:0/3914779732 conn(0x7fbad408aff0 msgr2=0x7fbad40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.663+0000 7fbad8a00700 1 -- 172.21.15.89:0/3914779732 shutdown_connections 2024-04-02T09:12:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.663+0000 7fbad8a00700 1 -- 172.21.15.89:0/3914779732 wait complete. 2024-04-02T09:12:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.664+0000 7fbad8a00700 1 Processor -- start 2024-04-02T09:12:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.664+0000 7fbad8a00700 1 -- start start 2024-04-02T09:12:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.665+0000 7fbad8a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbad40d16a0 0x7fbad41f66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.665+0000 7fbad8a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbad40d2370 0x7fbad40b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.665+0000 7fbad8a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbad415d9d0 0x7fbad4155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.665+0000 7fbad8a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad41e7420 con 0x7fbad40d16a0 2024-04-02T09:12:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.665+0000 7fbad8a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad41f75c0 con 0x7fbad415d9d0 2024-04-02T09:12:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.665+0000 7fbad8a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad41f6d20 con 0x7fbad40d2370 2024-04-02T09:12:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.665+0000 7fbad1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbad40d16a0 0x7fbad41f66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.666+0000 7fbad1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbad40d16a0 0x7fbad41f66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34102/0 (socket says 172.21.15.89:34102) 2024-04-02T09:12:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.666+0000 7fbad1e00700 1 -- 172.21.15.89:0/974162714 learned_addr learned my addr 172.21.15.89:0/974162714 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.666+0000 7fbad1400700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbad40d2370 0x7fbad40b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.666+0000 7fbad2800700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbad415d9d0 0x7fbad4155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.666+0000 7fbad1e00700 1 -- 172.21.15.89:0/974162714 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbad40d2370 msgr2=0x7fbad40b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.666+0000 7fbad1e00700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbad40d2370 0x7fbad40b8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.666+0000 7fbad1e00700 1 -- 172.21.15.89:0/974162714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbad415d9d0 msgr2=0x7fbad4155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.666+0000 7fbad1e00700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbad415d9d0 0x7fbad4155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.666+0000 7fbad1e00700 1 -- 172.21.15.89:0/974162714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac0009430 con 0x7fbad40d16a0 2024-04-02T09:12:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.666+0000 7fbad2800700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbad415d9d0 0x7fbad4155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:08.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.666+0000 7fbad1e00700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbad40d16a0 0x7fbad41f66b0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fbac4009c10 tx=0x7fbac4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:08.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.666+0000 7fbaca000700 1 -- 172.21.15.89:0/974162714 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4004510 con 0x7fbad40d16a0 2024-04-02T09:12:08.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.667+0000 7fbaca000700 1 -- 172.21.15.89:0/974162714 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac401e070 con 0x7fbad40d16a0 2024-04-02T09:12:08.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.667+0000 7fbad8a00700 1 -- 172.21.15.89:0/974162714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac4009550 con 0x7fbad40d16a0 2024-04-02T09:12:08.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.667+0000 7fbaca000700 1 -- 172.21.15.89:0/974162714 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac40038e0 con 0x7fbad40d16a0 2024-04-02T09:12:08.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.667+0000 7fbad8a00700 1 -- 172.21.15.89:0/974162714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad41f72e0 con 0x7fbad40d16a0 2024-04-02T09:12:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.668+0000 7fbad8a00700 1 -- 172.21.15.89:0/974162714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad40d1ca0 con 0x7fbad40d16a0 2024-04-02T09:12:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.668+0000 7fbaca000700 1 -- 172.21.15.89:0/974162714 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fbac4003a40 con 0x7fbad40d16a0 2024-04-02T09:12:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.670+0000 7fbaca000700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbab406adb0 0x7fbab406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.670+0000 7fbaca000700 1 -- 172.21.15.89:0/974162714 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6615+0+0 (secure 0 0 0) 0x7fbac408a6e0 con 0x7fbad40d16a0 2024-04-02T09:12:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.670+0000 7fbad1400700 1 -- 172.21.15.89:0/974162714 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbab406adb0 msgr2=0x7fbab406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.670+0000 7fbad1400700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbab406adb0 0x7fbab406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:08.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.674+0000 7fbaca000700 1 -- 172.21.15.89:0/974162714 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbac4055390 con 0x7fbad40d16a0 2024-04-02T09:12:08.710 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:08.712+0000 7f7821bc7240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:12:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.871+0000 7fbad1400700 1 -- 172.21.15.89:0/974162714 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbab406adb0 msgr2=0x7fbab406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.871+0000 7fbad1400700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbab406adb0 0x7fbab406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:08.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.883+0000 7fbad8a00700 1 -- 172.21.15.89:0/974162714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbad41e7560 con 0x7fbad40d16a0 2024-04-02T09:12:08.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.883+0000 7fbaca000700 1 -- 172.21.15.89:0/974162714 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v209) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbad41e7560 con 0x7fbad40d16a0 2024-04-02T09:12:08.882 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:12:08.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.886+0000 7fbad8a00700 1 -- 172.21.15.89:0/974162714 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbab406adb0 msgr2=0x7fbab406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:08.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.886+0000 7fbad8a00700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbab406adb0 0x7fbab406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.886+0000 7fbad8a00700 1 -- 172.21.15.89:0/974162714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbad40d16a0 msgr2=0x7fbad41f66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:08.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.886+0000 7fbad8a00700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbad40d16a0 0x7fbad41f66b0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fbac4009c10 tx=0x7fbac4004750 comp rx=0 tx=0).stop 2024-04-02T09:12:08.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.886+0000 7fbad8a00700 1 -- 172.21.15.89:0/974162714 shutdown_connections 2024-04-02T09:12:08.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.886+0000 7fbad8a00700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbab406adb0 0x7fbab406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.886+0000 7fbad8a00700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbad415d9d0 0x7fbad4155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.886+0000 7fbad8a00700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbad40d16a0 0x7fbad41f66b0 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.886+0000 7fbad8a00700 1 --2- 172.21.15.89:0/974162714 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbad40d2370 0x7fbad40b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:08.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.886+0000 7fbad8a00700 1 -- 172.21.15.89:0/974162714 >> 172.21.15.89:0/974162714 conn(0x7fbad408aff0 msgr2=0x7fbad406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:08.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.886+0000 7fbad8a00700 1 -- 172.21.15.89:0/974162714 shutdown_connections 2024-04-02T09:12:08.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:08.886+0000 7fbad8a00700 1 -- 172.21.15.89:0/974162714 wait complete. 2024-04-02T09:12:08.904 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:12:08.923 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:08.924+0000 7f7821bc7240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:12:08.999 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:09.000+0000 7f0ca3caa240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:12:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.106+0000 7f0b92600700 1 Processor -- start 2024-04-02T09:12:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.106+0000 7f0b92600700 1 -- start start 2024-04-02T09:12:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.107+0000 7f0b92600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b8c09aba0 0x7f0b8c06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.107+0000 7f0b92600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0b8c06c480 0x7f0b8c0ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.107+0000 7f0b92600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c06c8d0 con 0x7f0b8c15d710 2024-04-02T09:12:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.108+0000 7f0b92600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b8c06bb40 con 0x7f0b8c09aba0 2024-04-02T09:12:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.108+0000 7f0b92600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b8c099b00 con 0x7f0b8c06c480 2024-04-02T09:12:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.108+0000 7f0b83e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b8c09aba0 0x7f0b8c06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.108+0000 7f0b8b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0b8c15d710 0x7f0b8c0d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57258/0 (socket says 172.21.15.89:57258) 2024-04-02T09:12:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.108+0000 7f0b8b400700 1 -- 172.21.15.89:0/3671232691 learned_addr learned my addr 172.21.15.89:0/3671232691 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.108+0000 7f0b83e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b8c09aba0 0x7f0b8c06bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59130/0 (socket says 172.21.15.89:59130) 2024-04-02T09:12:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.108+0000 7f0b8be00700 1 --2- 172.21.15.89:0/3671232691 >> v2:172.21.15.113:3300/0 conn(0x7f0b8c06c480 0x7f0b8c0ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.108+0000 7f0b83e00700 1 -- 172.21.15.89:0/3671232691 >> v2:172.21.15.113:3300/0 conn(0x7f0b8c06c480 msgr2=0x7f0b8c0ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.108+0000 7f0b83e00700 1 --2- 172.21.15.89:0/3671232691 >> v2:172.21.15.113:3300/0 conn(0x7f0b8c06c480 0x7f0b8c0ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.108+0000 7f0b83e00700 1 -- 172.21.15.89:0/3671232691 >> v1:172.21.15.89:6789/0 conn(0x7f0b8c15d710 legacy=0x7f0b8c0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.108+0000 7f0b83e00700 1 -- 172.21.15.89:0/3671232691 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c099de0 con 0x7f0b8c09aba0 2024-04-02T09:12:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.108+0000 7f0b8be00700 1 --2- 172.21.15.89:0/3671232691 >> v2:172.21.15.113:3300/0 conn(0x7f0b8c06c480 0x7f0b8c0ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.108+0000 7f0b8aa00700 1 -- 172.21.15.89:0/3671232691 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107350711 0 0) 0x7f0b8c06c8d0 con 0x7f0b8c15d710 2024-04-02T09:12:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.109+0000 7f0b83e00700 1 --2- 172.21.15.89:0/3671232691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b8c09aba0 0x7f0b8c06bf40 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c0722c0 tx=0x7f0b74009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e186210b53b1e7a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.109+0000 7f0b8aa00700 1 -- 172.21.15.89:0/3671232691 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b74011830 con 0x7f0b8c09aba0 2024-04-02T09:12:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.109+0000 7f0b8aa00700 1 -- 172.21.15.89:0/3671232691 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b74011990 con 0x7f0b8c09aba0 2024-04-02T09:12:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.109+0000 7f0b8aa00700 1 -- 172.21.15.89:0/3671232691 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b74011d20 con 0x7f0b8c09aba0 2024-04-02T09:12:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.109+0000 7f0b92600700 1 -- 172.21.15.89:0/3671232691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b8c09aba0 msgr2=0x7f0b8c06bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.109+0000 7f0b92600700 1 --2- 172.21.15.89:0/3671232691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b8c09aba0 0x7f0b8c06bf40 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c0722c0 tx=0x7f0b74009b20 comp rx=0 tx=0).stop 2024-04-02T09:12:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.109+0000 7f0b92600700 1 -- 172.21.15.89:0/3671232691 shutdown_connections 2024-04-02T09:12:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.109+0000 7f0b92600700 1 --2- 172.21.15.89:0/3671232691 >> v2:172.21.15.113:3300/0 conn(0x7f0b8c06c480 0x7f0b8c0ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.109+0000 7f0b92600700 1 --2- 172.21.15.89:0/3671232691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b8c09aba0 0x7f0b8c06bf40 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.109+0000 7f0b92600700 1 -- 172.21.15.89:0/3671232691 >> 172.21.15.89:0/3671232691 conn(0x7f0b8c0d13e0 msgr2=0x7f0b8c0d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.109+0000 7f0b92600700 1 -- 172.21.15.89:0/3671232691 shutdown_connections 2024-04-02T09:12:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.110+0000 7f0b92600700 1 -- 172.21.15.89:0/3671232691 wait complete. 2024-04-02T09:12:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.111+0000 7f0b92600700 1 Processor -- start 2024-04-02T09:12:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.111+0000 7f0b92600700 1 -- start start 2024-04-02T09:12:09.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.112+0000 7f0b92600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b8c06c480 0x7f0b8c1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:09.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.112+0000 7f0b92600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b8c09aba0 0x7f0b8c06f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:09.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.112+0000 7f0b92600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b8c15d710 0x7f0b8c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:09.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.112+0000 7f0b92600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c1f2e80 con 0x7f0b8c09aba0 2024-04-02T09:12:09.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.112+0000 7f0b92600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c15db90 con 0x7f0b8c15d710 2024-04-02T09:12:09.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.112+0000 7f0b92600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c15de70 con 0x7f0b8c06c480 2024-04-02T09:12:09.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.112+0000 7f0b83e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b8c09aba0 0x7f0b8c06f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:09.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.112+0000 7f0b83e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b8c09aba0 0x7f0b8c06f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59142/0 (socket says 172.21.15.89:59142) 2024-04-02T09:12:09.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.112+0000 7f0b83e00700 1 -- 172.21.15.89:0/2820875630 learned_addr learned my addr 172.21.15.89:0/2820875630 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:09.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.112+0000 7f0b8be00700 1 --2- 172.21.15.89:0/2820875630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b8c15d710 0x7f0b8c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:09.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.113+0000 7f0b8b400700 1 --2- 172.21.15.89:0/2820875630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b8c06c480 0x7f0b8c1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.113+0000 7f0b83e00700 1 -- 172.21.15.89:0/2820875630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b8c06c480 msgr2=0x7f0b8c1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.113+0000 7f0b83e00700 1 --2- 172.21.15.89:0/2820875630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b8c06c480 0x7f0b8c1551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.113+0000 7f0b83e00700 1 -- 172.21.15.89:0/2820875630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b8c15d710 msgr2=0x7f0b8c06fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.113+0000 7f0b83e00700 1 --2- 172.21.15.89:0/2820875630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b8c15d710 0x7f0b8c06fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.113+0000 7f0b83e00700 1 -- 172.21.15.89:0/2820875630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b740094a0 con 0x7f0b8c09aba0 2024-04-02T09:12:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.113+0000 7f0b83e00700 1 --2- 172.21.15.89:0/2820875630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b8c09aba0 0x7f0b8c06f790 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f0b740090b0 tx=0x7f0b74004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.113+0000 7f0b83400700 1 -- 172.21.15.89:0/2820875630 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b740036a0 con 0x7f0b8c09aba0 2024-04-02T09:12:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.113+0000 7f0b83400700 1 -- 172.21.15.89:0/2820875630 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7401f070 con 0x7f0b8c09aba0 2024-04-02T09:12:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.113+0000 7f0b83400700 1 -- 172.21.15.89:0/2820875630 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b74003a20 con 0x7f0b8c09aba0 2024-04-02T09:12:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.113+0000 7f0b92600700 1 -- 172.21.15.89:0/2820875630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b8c15e150 con 0x7f0b8c09aba0 2024-04-02T09:12:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.114+0000 7f0b92600700 1 -- 172.21.15.89:0/2820875630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b8c155b50 con 0x7f0b8c09aba0 2024-04-02T09:12:09.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.115+0000 7f0b92600700 1 -- 172.21.15.89:0/2820875630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b8c067720 con 0x7f0b8c09aba0 2024-04-02T09:12:09.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.115+0000 7f0b83400700 1 -- 172.21.15.89:0/2820875630 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f0b74003b80 con 0x7f0b8c09aba0 2024-04-02T09:12:09.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.116+0000 7f0b83400700 1 --2- 172.21.15.89:0/2820875630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b6806adb0 0x7f0b6806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:09.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.116+0000 7f0b83400700 1 -- 172.21.15.89:0/2820875630 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6615+0+0 (secure 0 0 0) 0x7f0b74021070 con 0x7f0b8c09aba0 2024-04-02T09:12:09.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.116+0000 7f0b8b400700 1 -- 172.21.15.89:0/2820875630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b6806adb0 msgr2=0x7f0b6806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:09.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.117+0000 7f0b8b400700 1 --2- 172.21.15.89:0/2820875630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b6806adb0 0x7f0b6806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:09.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.121+0000 7f0b83400700 1 -- 172.21.15.89:0/2820875630 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b74055da0 con 0x7f0b8c09aba0 2024-04-02T09:12:09.131 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:09.132+0000 7f7821bc7240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:12:09.223 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:09.225+0000 7f0ca3caa240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:12:09.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.318+0000 7f0b8b400700 1 -- 172.21.15.89:0/2820875630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b6806adb0 msgr2=0x7f0b6806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:09.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.318+0000 7f0b8b400700 1 --2- 172.21.15.89:0/2820875630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b6806adb0 0x7f0b6806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:09.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.378+0000 7f0b92600700 1 -- 172.21.15.89:0/2820875630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b8c0d0f20 con 0x7f0b8c09aba0 2024-04-02T09:12:09.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.378+0000 7f0b83400700 1 -- 172.21.15.89:0/2820875630 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4748 (secure 0 0 0) 0x7f0b7402b020 con 0x7f0b8c09aba0 2024-04-02T09:12:09.377 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:09.377 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1311594829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1311594829},{"type":"v1","addr":"172.21.15.113:6834","nonce":1311594829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1256547418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1256547418},{"type":"v1","addr":"172.21.15.89:6837","nonce":1256547418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1137044351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1137044351},{"type":"v1","addr":"172.21.15.113:6835","nonce":1137044351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:09:42.220191+0000","modified":"2024-04-02T09:09:45.699081+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10638":{"gid":10638,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/2257849268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2257849268},{"type":"v1","addr":"172.21.15.89:6836","nonce":2257849268}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-04-02T09:12:09.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.381+0000 7f0b92600700 1 -- 172.21.15.89:0/2820875630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b6806adb0 msgr2=0x7f0b6806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:09.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.381+0000 7f0b92600700 1 --2- 172.21.15.89:0/2820875630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b6806adb0 0x7f0b6806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.381+0000 7f0b92600700 1 -- 172.21.15.89:0/2820875630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b8c09aba0 msgr2=0x7f0b8c06f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:09.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.381+0000 7f0b92600700 1 --2- 172.21.15.89:0/2820875630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b8c09aba0 0x7f0b8c06f790 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f0b740090b0 tx=0x7f0b74004750 comp rx=0 tx=0).stop 2024-04-02T09:12:09.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.381+0000 7f0b92600700 1 -- 172.21.15.89:0/2820875630 shutdown_connections 2024-04-02T09:12:09.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.381+0000 7f0b92600700 1 --2- 172.21.15.89:0/2820875630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b6806adb0 0x7f0b6806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.381+0000 7f0b92600700 1 --2- 172.21.15.89:0/2820875630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b8c15d710 0x7f0b8c06fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.381+0000 7f0b92600700 1 --2- 172.21.15.89:0/2820875630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b8c06c480 0x7f0b8c1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.381+0000 7f0b92600700 1 --2- 172.21.15.89:0/2820875630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b8c09aba0 0x7f0b8c06f790 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.381+0000 7f0b92600700 1 -- 172.21.15.89:0/2820875630 >> 172.21.15.89:0/2820875630 conn(0x7f0b8c0d13e0 msgr2=0x7f0b8c0af460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:09.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.381+0000 7f0b92600700 1 -- 172.21.15.89:0/2820875630 shutdown_connections 2024-04-02T09:12:09.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.381+0000 7f0b92600700 1 -- 172.21.15.89:0/2820875630 wait complete. 2024-04-02T09:12:09.381 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 97 2024-04-02T09:12:09.398 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:12:09.445 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:09.447+0000 7f0ca3caa240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:12:09.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.595+0000 7f04e3c00700 1 Processor -- start 2024-04-02T09:12:09.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.595+0000 7f04e3c00700 1 -- start start 2024-04-02T09:12:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.596+0000 7f04e3c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f04dc160be0 0x7f04dc0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.596+0000 7f04e3c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc160fc0 con 0x7f04dc0ceca0 2024-04-02T09:12:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.596+0000 7f04e3c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc0d21e0 con 0x7f04dc0ce0e0 2024-04-02T09:12:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.596+0000 7f04e3c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04dc0b65d0 con 0x7f04dc160be0 2024-04-02T09:12:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.597+0000 7f04e1c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f04dc0ceca0 0x7f04dc0b6a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57268/0 (socket says 172.21.15.89:57268) 2024-04-02T09:12:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.597+0000 7f04e1c00700 1 -- 172.21.15.89:0/2774697494 learned_addr learned my addr 172.21.15.89:0/2774697494 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.597+0000 7f04e1200700 1 --2- 172.21.15.89:0/2774697494 >> v2:172.21.15.113:3300/0 conn(0x7f04dc160be0 0x7f04dc0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:09.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.597+0000 7f04db400700 1 -- 172.21.15.89:0/2774697494 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553396 0 0) 0x7f04dc160fc0 con 0x7f04dc0ceca0 2024-04-02T09:12:09.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.597+0000 7f04db400700 1 -- 172.21.15.89:0/2774697494 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04c40035b0 con 0x7f04dc0ceca0 2024-04-02T09:12:09.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.597+0000 7f04e1200700 1 -- 172.21.15.89:0/2774697494 >> v1:172.21.15.113:6789/0 conn(0x7f04dc0ce0e0 legacy=0x7f04dc0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:09.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.597+0000 7f04e1200700 1 -- 172.21.15.89:0/2774697494 >> v1:172.21.15.89:6789/0 conn(0x7f04dc0ceca0 legacy=0x7f04dc0b6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:09.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.597+0000 7f04e1200700 1 -- 172.21.15.89:0/2774697494 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc0d24c0 con 0x7f04dc160be0 2024-04-02T09:12:09.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.598+0000 7f04e1200700 1 --2- 172.21.15.89:0/2774697494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04dc160be0 0x7f04dc0cdba0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009840 tx=0x7f04d0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab3e36e4d615ca0a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:09.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.598+0000 7f04db400700 1 -- 172.21.15.89:0/2774697494 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0011830 con 0x7f04dc160be0 2024-04-02T09:12:09.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.598+0000 7f04db400700 1 -- 172.21.15.89:0/2774697494 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d0011990 con 0x7f04dc160be0 2024-04-02T09:12:09.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.598+0000 7f04db400700 1 -- 172.21.15.89:0/2774697494 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0011d00 con 0x7f04dc160be0 2024-04-02T09:12:09.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.598+0000 7f04e3c00700 1 -- 172.21.15.89:0/2774697494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04dc160be0 msgr2=0x7f04dc0cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:09.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.599+0000 7f04e3c00700 1 --2- 172.21.15.89:0/2774697494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04dc160be0 0x7f04dc0cdba0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009840 tx=0x7f04d0009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:09.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.599+0000 7f04e3c00700 1 -- 172.21.15.89:0/2774697494 shutdown_connections 2024-04-02T09:12:09.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.599+0000 7f04e3c00700 1 --2- 172.21.15.89:0/2774697494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04dc160be0 0x7f04dc0cdba0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.599+0000 7f04e3c00700 1 -- 172.21.15.89:0/2774697494 >> 172.21.15.89:0/2774697494 conn(0x7f04dc08aff0 msgr2=0x7f04dc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:09.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.599+0000 7f04e3c00700 1 -- 172.21.15.89:0/2774697494 shutdown_connections 2024-04-02T09:12:09.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.599+0000 7f04e3c00700 1 -- 172.21.15.89:0/2774697494 wait complete. 2024-04-02T09:12:09.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.600+0000 7f04e3c00700 1 Processor -- start 2024-04-02T09:12:09.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.601+0000 7f04e3c00700 1 -- start start 2024-04-02T09:12:09.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.601+0000 7f04e3c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04dc0ce0e0 0x7f04dc154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:09.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.601+0000 7f04e3c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04dc0ceca0 0x7f04dc155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:09.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.601+0000 7f04e3c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04dc160be0 0x7f04dc155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:09.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.602+0000 7f04e3c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc0ccf50 con 0x7f04dc0ce0e0 2024-04-02T09:12:09.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.602+0000 7f04e3c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc0cc030 con 0x7f04dc0ceca0 2024-04-02T09:12:09.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.602+0000 7f04e3c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc0cc2d0 con 0x7f04dc160be0 2024-04-02T09:12:09.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.602+0000 7f04e1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04dc0ce0e0 0x7f04dc154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:09.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.602+0000 7f04e1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04dc0ce0e0 0x7f04dc154c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59148/0 (socket says 172.21.15.89:59148) 2024-04-02T09:12:09.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.602+0000 7f04e1200700 1 -- 172.21.15.89:0/1365719575 learned_addr learned my addr 172.21.15.89:0/1365719575 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:09.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.602+0000 7f04e1c00700 1 --2- 172.21.15.89:0/1365719575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04dc160be0 0x7f04dc155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:09.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.602+0000 7f04dbe00700 1 --2- 172.21.15.89:0/1365719575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04dc0ceca0 0x7f04dc155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:09.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.602+0000 7f04e1200700 1 -- 172.21.15.89:0/1365719575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04dc160be0 msgr2=0x7f04dc155ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:09.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.602+0000 7f04e1200700 1 --2- 172.21.15.89:0/1365719575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04dc160be0 0x7f04dc155ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.602+0000 7f04e1200700 1 -- 172.21.15.89:0/1365719575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04dc0ceca0 msgr2=0x7f04dc155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:09.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.602+0000 7f04e1200700 1 --2- 172.21.15.89:0/1365719575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04dc0ceca0 0x7f04dc155440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.602+0000 7f04e1200700 1 -- 172.21.15.89:0/1365719575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d00094a0 con 0x7f04dc0ce0e0 2024-04-02T09:12:09.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.603+0000 7f04e1200700 1 --2- 172.21.15.89:0/1365719575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04dc0ce0e0 0x7f04dc154c20 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f04d00090b0 tx=0x7f04d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:09.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.603+0000 7f04d8c00700 1 -- 172.21.15.89:0/1365719575 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d00045d0 con 0x7f04dc0ce0e0 2024-04-02T09:12:09.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.603+0000 7f04d8c00700 1 -- 172.21.15.89:0/1365719575 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d001e070 con 0x7f04dc0ce0e0 2024-04-02T09:12:09.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.603+0000 7f04e3c00700 1 -- 172.21.15.89:0/1365719575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04dc0cc5b0 con 0x7f04dc0ce0e0 2024-04-02T09:12:09.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.603+0000 7f04d8c00700 1 -- 172.21.15.89:0/1365719575 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d00038e0 con 0x7f04dc0ce0e0 2024-04-02T09:12:09.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.603+0000 7f04e3c00700 1 -- 172.21.15.89:0/1365719575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04dc0ccb70 con 0x7f04dc0ce0e0 2024-04-02T09:12:09.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.604+0000 7f04e3c00700 1 -- 172.21.15.89:0/1365719575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04dc0cd090 con 0x7f04dc0ce0e0 2024-04-02T09:12:09.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.605+0000 7f04d8c00700 1 -- 172.21.15.89:0/1365719575 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f04d0003a40 con 0x7f04dc0ce0e0 2024-04-02T09:12:09.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.606+0000 7f04d8c00700 1 --2- 172.21.15.89:0/1365719575 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f04bc06adb0 0x7f04bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:09.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.606+0000 7f04d8c00700 1 -- 172.21.15.89:0/1365719575 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6615+0+0 (secure 0 0 0) 0x7f04d008a690 con 0x7f04dc0ce0e0 2024-04-02T09:12:09.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.606+0000 7f04dbe00700 1 -- 172.21.15.89:0/1365719575 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f04bc06adb0 msgr2=0x7f04bc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:09.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.606+0000 7f04dbe00700 1 --2- 172.21.15.89:0/1365719575 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f04bc06adb0 0x7f04bc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:09.606 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:09.607+0000 7f7821bc7240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:12:09.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.611+0000 7f04d8c00700 1 -- 172.21.15.89:0/1365719575 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f04dc0cd090 con 0x7f04dc0ce0e0 2024-04-02T09:12:09.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.807+0000 7f04dbe00700 1 -- 172.21.15.89:0/1365719575 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f04bc06adb0 msgr2=0x7f04bc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:09.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.807+0000 7f04dbe00700 1 --2- 172.21.15.89:0/1365719575 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f04bc06adb0 0x7f04bc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:09.814 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:09.815+0000 7f7821bc7240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:12:09.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.819+0000 7f04e3c00700 1 -- 172.21.15.89:0/1365719575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04dc0cd090 con 0x7f04dc0ce0e0 2024-04-02T09:12:09.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.820+0000 7f04d8c00700 1 -- 172.21.15.89:0/1365719575 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+17449 (secure 0 0 0) 0x7f04d0055440 con 0x7f04dc0ce0e0 2024-04-02T09:12:09.819 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:09.819 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":209,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:12:07.647296+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:09:40.103721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-04-02T09:09:41.165944+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":209,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.5","mappings":[{"from":7,"to":3}]},{"pgid":"25.3b","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/2552487672":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/1282451945":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/3725462960":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/1115849275":"2024-04-03T09:09:59.528553+0000","172.21.15.89:6833/108855":"2024-04-03T09:09:59.528553+0000","172.21.15.89:6832/108855":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/2251569695":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/248630547":"2024-04-03T09:09:59.528553+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:12:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.823+0000 7f04e3c00700 1 -- 172.21.15.89:0/1365719575 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f04bc06adb0 msgr2=0x7f04bc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.823+0000 7f04e3c00700 1 --2- 172.21.15.89:0/1365719575 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f04bc06adb0 0x7f04bc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.823+0000 7f04e3c00700 1 -- 172.21.15.89:0/1365719575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04dc0ce0e0 msgr2=0x7f04dc154c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.823+0000 7f04e3c00700 1 --2- 172.21.15.89:0/1365719575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04dc0ce0e0 0x7f04dc154c20 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f04d00090b0 tx=0x7f04d0004750 comp rx=0 tx=0).stop 2024-04-02T09:12:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.823+0000 7f04e3c00700 1 -- 172.21.15.89:0/1365719575 shutdown_connections 2024-04-02T09:12:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.823+0000 7f04e3c00700 1 --2- 172.21.15.89:0/1365719575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04dc160be0 0x7f04dc155ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.823+0000 7f04e3c00700 1 --2- 172.21.15.89:0/1365719575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04dc0ce0e0 0x7f04dc154c20 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.823+0000 7f04e3c00700 1 --2- 172.21.15.89:0/1365719575 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f04bc06adb0 0x7f04bc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.823+0000 7f04e3c00700 1 --2- 172.21.15.89:0/1365719575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04dc0ceca0 0x7f04dc155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:09.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.823+0000 7f04e3c00700 1 -- 172.21.15.89:0/1365719575 >> 172.21.15.89:0/1365719575 conn(0x7f04dc08aff0 msgr2=0x7f04dc0cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:09.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.824+0000 7f04e3c00700 1 -- 172.21.15.89:0/1365719575 shutdown_connections 2024-04-02T09:12:09.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:09.824+0000 7f04e3c00700 1 -- 172.21.15.89:0/1365719575 wait complete. 2024-04-02T09:12:09.846 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T09:12:09.847 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:12:09.957 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:09.959+0000 7f0ca3caa240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:12:10.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.044+0000 7fe6e7800700 1 Processor -- start 2024-04-02T09:12:10.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.044+0000 7fe6e7800700 1 -- start start 2024-04-02T09:12:10.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.045+0000 7fe6e7800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6e015b5d0 0x7fe6e00ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:10.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.045+0000 7fe6e7800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6e00cb3f0 0x7fe6e00ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:10.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.045+0000 7fe6e7800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e015fd50 con 0x7fe6e00ae930 2024-04-02T09:12:10.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.045+0000 7fe6e7800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e015b9b0 con 0x7fe6e015b5d0 2024-04-02T09:12:10.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.045+0000 7fe6e7800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e015f140 con 0x7fe6e00cb3f0 2024-04-02T09:12:10.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.045+0000 7fe6e4e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6e015b5d0 0x7fe6e00ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:10.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.045+0000 7fe6e5800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe6e00ae930 0x7fe6e0140dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57272/0 (socket says 172.21.15.89:57272) 2024-04-02T09:12:10.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.045+0000 7fe6e4e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe6e015b5d0 0x7fe6e00ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59164/0 (socket says 172.21.15.89:59164) 2024-04-02T09:12:10.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.045+0000 7fe6e5800700 1 -- 172.21.15.89:0/2778846655 learned_addr learned my addr 172.21.15.89:0/2778846655 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:10.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.045+0000 7fe6dfe00700 1 --2- 172.21.15.89:0/2778846655 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6e00cb3f0 0x7fe6e00ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:10.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.046+0000 7fe6e4e00700 1 -- 172.21.15.89:0/2778846655 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6e00cb3f0 msgr2=0x7fe6e00ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:10.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.046+0000 7fe6e4e00700 1 --2- 172.21.15.89:0/2778846655 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6e00cb3f0 0x7fe6e00ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.046+0000 7fe6e4e00700 1 -- 172.21.15.89:0/2778846655 >> v1:172.21.15.89:6789/0 conn(0x7fe6e00ae930 legacy=0x7fe6e0140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:10.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.046+0000 7fe6e4e00700 1 -- 172.21.15.89:0/2778846655 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e015f420 con 0x7fe6e015b5d0 2024-04-02T09:12:10.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.046+0000 7fe6e4e00700 1 --2- 172.21.15.89:0/2778846655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e015b5d0 0x7fe6e00ae3f0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4009840 tx=0x7fe6d4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24cb3782a04fc5b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:10.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.046+0000 7fe6df400700 1 -- 172.21.15.89:0/2778846655 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4011830 con 0x7fe6e015b5d0 2024-04-02T09:12:10.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.046+0000 7fe6df400700 1 -- 172.21.15.89:0/2778846655 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d4011990 con 0x7fe6e015b5d0 2024-04-02T09:12:10.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.046+0000 7fe6df400700 1 -- 172.21.15.89:0/2778846655 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4011d20 con 0x7fe6e015b5d0 2024-04-02T09:12:10.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.047+0000 7fe6e7800700 1 -- 172.21.15.89:0/2778846655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e015b5d0 msgr2=0x7fe6e00ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:10.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.047+0000 7fe6e7800700 1 --2- 172.21.15.89:0/2778846655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e015b5d0 0x7fe6e00ae3f0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4009840 tx=0x7fe6d4009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:10.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.047+0000 7fe6e7800700 1 -- 172.21.15.89:0/2778846655 shutdown_connections 2024-04-02T09:12:10.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.047+0000 7fe6e7800700 1 --2- 172.21.15.89:0/2778846655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e015b5d0 0x7fe6e00ae3f0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.047+0000 7fe6e7800700 1 --2- 172.21.15.89:0/2778846655 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6e00cb3f0 0x7fe6e00ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.047+0000 7fe6e7800700 1 -- 172.21.15.89:0/2778846655 >> 172.21.15.89:0/2778846655 conn(0x7fe6e00cf190 msgr2=0x7fe6e00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:10.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.047+0000 7fe6e7800700 1 -- 172.21.15.89:0/2778846655 shutdown_connections 2024-04-02T09:12:10.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.047+0000 7fe6e7800700 1 -- 172.21.15.89:0/2778846655 wait complete. 2024-04-02T09:12:10.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.049+0000 7fe6e7800700 1 Processor -- start 2024-04-02T09:12:10.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.049+0000 7fe6e7800700 1 -- start start 2024-04-02T09:12:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e7800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6e00ae930 0x7fe6e00ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e7800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6e015b5d0 0x7fe6e013ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e7800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e0098920 0x7fe6e013d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e7800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e00af250 con 0x7fe6e0098920 2024-04-02T09:12:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e7800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e00aed10 con 0x7fe6e00ae930 2024-04-02T09:12:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e7800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e00aefb0 con 0x7fe6e015b5d0 2024-04-02T09:12:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e5800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e0098920 0x7fe6e013d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e5800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e0098920 0x7fe6e013d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59172/0 (socket says 172.21.15.89:59172) 2024-04-02T09:12:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e5800700 1 -- 172.21.15.89:0/1129172929 learned_addr learned my addr 172.21.15.89:0/1129172929 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6dfe00700 1 --2- 172.21.15.89:0/1129172929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6e015b5d0 0x7fe6e013ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e4e00700 1 --2- 172.21.15.89:0/1129172929 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6e00ae930 0x7fe6e00ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e5800700 1 -- 172.21.15.89:0/1129172929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6e015b5d0 msgr2=0x7fe6e013ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e5800700 1 --2- 172.21.15.89:0/1129172929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6e015b5d0 0x7fe6e013ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e5800700 1 -- 172.21.15.89:0/1129172929 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6e00ae930 msgr2=0x7fe6e00ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e5800700 1 --2- 172.21.15.89:0/1129172929 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6e00ae930 0x7fe6e00ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.050+0000 7fe6e5800700 1 -- 172.21.15.89:0/1129172929 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d40094a0 con 0x7fe6e0098920 2024-04-02T09:12:10.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.051+0000 7fe6e5800700 1 --2- 172.21.15.89:0/1129172929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e0098920 0x7fe6e013d2f0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800ca20 tx=0x7fe6d800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:10.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.051+0000 7fe6dcc00700 1 -- 172.21.15.89:0/1129172929 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d800a9e0 con 0x7fe6e0098920 2024-04-02T09:12:10.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.051+0000 7fe6dcc00700 1 -- 172.21.15.89:0/1129172929 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d800ab40 con 0x7fe6e0098920 2024-04-02T09:12:10.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.051+0000 7fe6dcc00700 1 -- 172.21.15.89:0/1129172929 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8012400 con 0x7fe6e0098920 2024-04-02T09:12:10.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.051+0000 7fe6e7800700 1 -- 172.21.15.89:0/1129172929 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e0153b50 con 0x7fe6e0098920 2024-04-02T09:12:10.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.051+0000 7fe6e7800700 1 -- 172.21.15.89:0/1129172929 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e01540d0 con 0x7fe6e0098920 2024-04-02T09:12:10.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.052+0000 7fe6e7800700 1 -- 172.21.15.89:0/1129172929 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e01539b0 con 0x7fe6e0098920 2024-04-02T09:12:10.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.053+0000 7fe6dcc00700 1 -- 172.21.15.89:0/1129172929 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fe6d801b3b0 con 0x7fe6e0098920 2024-04-02T09:12:10.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.054+0000 7fe6dcc00700 1 --2- 172.21.15.89:0/1129172929 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe6c006adb0 0x7fe6c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:10.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.054+0000 7fe6dcc00700 1 -- 172.21.15.89:0/1129172929 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6615+0+0 (secure 0 0 0) 0x7fe6d8089230 con 0x7fe6e0098920 2024-04-02T09:12:10.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.054+0000 7fe6e4e00700 1 -- 172.21.15.89:0/1129172929 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe6c006adb0 msgr2=0x7fe6c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:10.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.054+0000 7fe6e4e00700 1 --2- 172.21.15.89:0/1129172929 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe6c006adb0 0x7fe6c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:10.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.058+0000 7fe6dcc00700 1 -- 172.21.15.89:0/1129172929 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6e01539b0 con 0x7fe6e0098920 2024-04-02T09:12:10.194 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:10.195+0000 7f0ca3caa240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:12:10.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.255+0000 7fe6e4e00700 1 -- 172.21.15.89:0/1129172929 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe6c006adb0 msgr2=0x7fe6c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:10.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.255+0000 7fe6e4e00700 1 --2- 172.21.15.89:0/1129172929 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe6c006adb0 0x7fe6c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:10.254 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:10.254+0000 7f7821bc7240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:12:10.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.329+0000 7fe6e7800700 1 -- 172.21.15.89:0/1129172929 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6e01539b0 con 0x7fe6e0098920 2024-04-02T09:12:10.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.330+0000 7fe6dcc00700 1 -- 172.21.15.89:0/1129172929 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4748 (secure 0 0 0) 0x7fe6e01539b0 con 0x7fe6e0098920 2024-04-02T09:12:10.328 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:10.328 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1311594829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1311594829},{"type":"v1","addr":"172.21.15.113:6834","nonce":1311594829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1256547418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1256547418},{"type":"v1","addr":"172.21.15.89:6837","nonce":1256547418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1137044351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1137044351},{"type":"v1","addr":"172.21.15.113:6835","nonce":1137044351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:09:42.220191+0000","modified":"2024-04-02T09:09:45.699081+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10638":{"gid":10638,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/2257849268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2257849268},{"type":"v1","addr":"172.21.15.89:6836","nonce":2257849268}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-04-02T09:12:10.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.332+0000 7fe6e7800700 1 -- 172.21.15.89:0/1129172929 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe6c006adb0 msgr2=0x7fe6c006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:10.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.332+0000 7fe6e7800700 1 --2- 172.21.15.89:0/1129172929 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe6c006adb0 0x7fe6c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.332+0000 7fe6e7800700 1 -- 172.21.15.89:0/1129172929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e0098920 msgr2=0x7fe6e013d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:10.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.332+0000 7fe6e7800700 1 --2- 172.21.15.89:0/1129172929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e0098920 0x7fe6e013d2f0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800ca20 tx=0x7fe6d800cd30 comp rx=0 tx=0).stop 2024-04-02T09:12:10.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.332+0000 7fe6e7800700 1 -- 172.21.15.89:0/1129172929 shutdown_connections 2024-04-02T09:12:10.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.332+0000 7fe6e7800700 1 --2- 172.21.15.89:0/1129172929 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe6c006adb0 0x7fe6c006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.332+0000 7fe6e7800700 1 --2- 172.21.15.89:0/1129172929 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6e00ae930 0x7fe6e00ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.332+0000 7fe6e7800700 1 --2- 172.21.15.89:0/1129172929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e0098920 0x7fe6e013d2f0 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.332+0000 7fe6e7800700 1 --2- 172.21.15.89:0/1129172929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6e015b5d0 0x7fe6e013ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.333+0000 7fe6e7800700 1 -- 172.21.15.89:0/1129172929 >> 172.21.15.89:0/1129172929 conn(0x7fe6e00cf190 msgr2=0x7fe6e006ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:10.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.333+0000 7fe6e7800700 1 -- 172.21.15.89:0/1129172929 shutdown_connections 2024-04-02T09:12:10.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.333+0000 7fe6e7800700 1 -- 172.21.15.89:0/1129172929 wait complete. 2024-04-02T09:12:10.333 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 97 2024-04-02T09:12:10.350 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:12:10.364 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:10.366+0000 7f7821bc7240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:12:10.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.553+0000 7f6478e00700 1 Processor -- start 2024-04-02T09:12:10.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.553+0000 7f6478e00700 1 -- start start 2024-04-02T09:12:10.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.554+0000 7f6478e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f647415b5d0 0x7f64740ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:10.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.554+0000 7f6478e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64740cb3f0 0x7f64740ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:10.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.554+0000 7f6478e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647415fd50 con 0x7f64740ae930 2024-04-02T09:12:10.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.554+0000 7f6478e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647415b9b0 con 0x7f647415b5d0 2024-04-02T09:12:10.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.554+0000 7f6478e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647415f140 con 0x7f64740cb3f0 2024-04-02T09:12:10.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.555+0000 7f6471e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f647415b5d0 0x7f64740ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:10.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.555+0000 7f6471e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f647415b5d0 0x7f64740ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59184/0 (socket says 172.21.15.89:59184) 2024-04-02T09:12:10.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.555+0000 7f6471e00700 1 -- 172.21.15.89:0/3759273817 learned_addr learned my addr 172.21.15.89:0/3759273817 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:10.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.555+0000 7f6471400700 1 --2- 172.21.15.89:0/3759273817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64740cb3f0 0x7f64740ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:10.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.555+0000 7f6471e00700 1 -- 172.21.15.89:0/3759273817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64740cb3f0 msgr2=0x7f64740ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:10.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.555+0000 7f6471e00700 1 --2- 172.21.15.89:0/3759273817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64740cb3f0 0x7f64740ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.555+0000 7f6471e00700 1 -- 172.21.15.89:0/3759273817 >> v1:172.21.15.113:6789/0 conn(0x7f64740ae930 legacy=0x7f6474140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:10.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.555+0000 7f6471e00700 1 -- 172.21.15.89:0/3759273817 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647415f420 con 0x7f647415b5d0 2024-04-02T09:12:10.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.555+0000 7f6470a00700 1 -- 172.21.15.89:0/3759273817 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912439718 0 0) 0x7f647415fd50 con 0x7f64740ae930 2024-04-02T09:12:10.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.556+0000 7f6471e00700 1 --2- 172.21.15.89:0/3759273817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f647415b5d0 0x7f64740ae3f0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f64640098f0 tx=0x7f6464009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9c00adaf714836 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:10.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.556+0000 7f6470a00700 1 -- 172.21.15.89:0/3759273817 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464011830 con 0x7f647415b5d0 2024-04-02T09:12:10.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.556+0000 7f6470a00700 1 -- 172.21.15.89:0/3759273817 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6464011990 con 0x7f647415b5d0 2024-04-02T09:12:10.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.556+0000 7f6470a00700 1 -- 172.21.15.89:0/3759273817 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464011d20 con 0x7f647415b5d0 2024-04-02T09:12:10.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.556+0000 7f6478e00700 1 -- 172.21.15.89:0/3759273817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f647415b5d0 msgr2=0x7f64740ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:10.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.556+0000 7f6478e00700 1 --2- 172.21.15.89:0/3759273817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f647415b5d0 0x7f64740ae3f0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f64640098f0 tx=0x7f6464009c00 comp rx=0 tx=0).stop 2024-04-02T09:12:10.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.556+0000 7f6478e00700 1 -- 172.21.15.89:0/3759273817 shutdown_connections 2024-04-02T09:12:10.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.556+0000 7f6478e00700 1 --2- 172.21.15.89:0/3759273817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f647415b5d0 0x7f64740ae3f0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.556+0000 7f6478e00700 1 --2- 172.21.15.89:0/3759273817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64740cb3f0 0x7f64740ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.556+0000 7f6478e00700 1 -- 172.21.15.89:0/3759273817 >> 172.21.15.89:0/3759273817 conn(0x7f64740cf190 msgr2=0x7f64740cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:10.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.557+0000 7f6478e00700 1 -- 172.21.15.89:0/3759273817 shutdown_connections 2024-04-02T09:12:10.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.557+0000 7f6478e00700 1 -- 172.21.15.89:0/3759273817 wait complete. 2024-04-02T09:12:10.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.559+0000 7f6478e00700 1 Processor -- start 2024-04-02T09:12:10.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.559+0000 7f6478e00700 1 -- start start 2024-04-02T09:12:10.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.560+0000 7f6478e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64740ae930 0x7f6474155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:10.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.560+0000 7f6478e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f647415b5d0 0x7f64740cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:10.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.560+0000 7f6478e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64740cd920 0x7f64740cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:10.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.560+0000 7f6478e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64740cddd0 con 0x7f64740ae930 2024-04-02T09:12:10.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.560+0000 7f6478e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64741f2ef0 con 0x7f647415b5d0 2024-04-02T09:12:10.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.560+0000 7f6478e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64740cc7b0 con 0x7f64740cd920 2024-04-02T09:12:10.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.560+0000 7f6471e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64740ae930 0x7f6474155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:10.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.561+0000 7f6471e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64740ae930 0x7f6474155d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59190/0 (socket says 172.21.15.89:59190) 2024-04-02T09:12:10.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.561+0000 7f6471e00700 1 -- 172.21.15.89:0/3631858899 learned_addr learned my addr 172.21.15.89:0/3631858899 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:10.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.561+0000 7f6471e00700 1 -- 172.21.15.89:0/3631858899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64740cd920 msgr2=0x7f64740cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:10.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.561+0000 7f6471400700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f647415b5d0 0x7f64740cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:10.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.561+0000 7f6472800700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64740cd920 0x7f64740cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:10.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.561+0000 7f6471e00700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64740cd920 0x7f64740cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.561+0000 7f6471e00700 1 -- 172.21.15.89:0/3631858899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f647415b5d0 msgr2=0x7f64740cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:10.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.561+0000 7f6471e00700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f647415b5d0 0x7f64740cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.561+0000 7f6471e00700 1 -- 172.21.15.89:0/3631858899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6464009550 con 0x7f64740ae930 2024-04-02T09:12:10.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.561+0000 7f6471400700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f647415b5d0 0x7f64740cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:10.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.561+0000 7f6472800700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64740cd920 0x7f64740cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:10.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.561+0000 7f6471e00700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64740ae930 0x7f6474155d70 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f6464009c40 tx=0x7f6464004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:10.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.561+0000 7f646a000700 1 -- 172.21.15.89:0/3631858899 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64640036a0 con 0x7f64740ae930 2024-04-02T09:12:10.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.562+0000 7f646a000700 1 -- 172.21.15.89:0/3631858899 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646401e070 con 0x7f64740ae930 2024-04-02T09:12:10.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.562+0000 7f6478e00700 1 -- 172.21.15.89:0/3631858899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64740cca90 con 0x7f64740ae930 2024-04-02T09:12:10.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.562+0000 7f646a000700 1 -- 172.21.15.89:0/3631858899 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464003a20 con 0x7f64740ae930 2024-04-02T09:12:10.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.562+0000 7f6478e00700 1 -- 172.21.15.89:0/3631858899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64740cd050 con 0x7f64740ae930 2024-04-02T09:12:10.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.563+0000 7f6478e00700 1 -- 172.21.15.89:0/3631858899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64740cd240 con 0x7f64740ae930 2024-04-02T09:12:10.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.564+0000 7f646a000700 1 -- 172.21.15.89:0/3631858899 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 141) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f6464003b80 con 0x7f64740ae930 2024-04-02T09:12:10.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.565+0000 7f646a000700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f645406adb0 0x7f645406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:10.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.565+0000 7f646a000700 1 -- 172.21.15.89:0/3631858899 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6615+0+0 (secure 0 0 0) 0x7f646408a7f0 con 0x7f64740ae930 2024-04-02T09:12:10.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.565+0000 7f6471400700 1 -- 172.21.15.89:0/3631858899 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f645406adb0 msgr2=0x7f645406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:10.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.565+0000 7f6471400700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f645406adb0 0x7f645406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:10.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.569+0000 7f646a000700 1 -- 172.21.15.89:0/3631858899 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f64740cd240 con 0x7f64740ae930 2024-04-02T09:12:10.654 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:10.656+0000 7f0ca3caa240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:12:10.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.670+0000 7f646a000700 1 -- 172.21.15.89:0/3631858899 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 142) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f64640558f0 con 0x7f64740ae930 2024-04-02T09:12:10.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.766+0000 7f6471400700 1 -- 172.21.15.89:0/3631858899 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f645406adb0 msgr2=0x7f645406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:10.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.766+0000 7f6471400700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f645406adb0 0x7f645406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:10.771 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:10.773+0000 7f0ca3caa240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:12:10.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.785+0000 7f646a000700 1 -- 172.21.15.89:0/3631858899 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 143) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f6464070ae0 con 0x7f64740ae930 2024-04-02T09:12:10.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.785+0000 7f646a000700 1 -- 172.21.15.89:0/3631858899 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f645406adb0 msgr2=0x7f645406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:10.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.785+0000 7f646a000700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f645406adb0 0x7f645406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.838+0000 7f6478e00700 1 -- 172.21.15.89:0/3631858899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64740cd240 con 0x7f64740ae930 2024-04-02T09:12:10.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.839+0000 7f646a000700 1 -- 172.21.15.89:0/3631858899 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4748 (secure 0 0 0) 0x7f64740cd240 con 0x7f64740ae930 2024-04-02T09:12:10.837 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:10.837 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1311594829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1311594829},{"type":"v1","addr":"172.21.15.113:6834","nonce":1311594829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1256547418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1256547418},{"type":"v1","addr":"172.21.15.89:6837","nonce":1256547418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1137044351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1137044351},{"type":"v1","addr":"172.21.15.113:6835","nonce":1137044351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:09:42.220191+0000","modified":"2024-04-02T09:09:45.699081+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10638":{"gid":10638,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6836/2257849268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2257849268},{"type":"v1","addr":"172.21.15.89:6836","nonce":2257849268}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-04-02T09:12:10.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.841+0000 7f6478e00700 1 -- 172.21.15.89:0/3631858899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64740ae930 msgr2=0x7f6474155d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:10.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.841+0000 7f6478e00700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64740ae930 0x7f6474155d70 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f6464009c40 tx=0x7f6464004750 comp rx=0 tx=0).stop 2024-04-02T09:12:10.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.841+0000 7f6478e00700 1 -- 172.21.15.89:0/3631858899 shutdown_connections 2024-04-02T09:12:10.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.841+0000 7f6478e00700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f64740cd920 0x7f64740cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.841+0000 7f6478e00700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64740ae930 0x7f6474155d70 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.841+0000 7f6478e00700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f645406adb0 0x7f645406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.841+0000 7f6478e00700 1 --2- 172.21.15.89:0/3631858899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f647415b5d0 0x7f64740cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:10.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.841+0000 7f6478e00700 1 -- 172.21.15.89:0/3631858899 >> 172.21.15.89:0/3631858899 conn(0x7f64740cf190 msgr2=0x7f64740cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:10.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.842+0000 7f6478e00700 1 -- 172.21.15.89:0/3631858899 shutdown_connections 2024-04-02T09:12:10.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:10.842+0000 7f6478e00700 1 -- 172.21.15.89:0/3631858899 wait complete. 2024-04-02T09:12:10.841 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 97 2024-04-02T09:12:10.856 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:10.858+0000 7f0c16e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:10.856 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:10.858+0000 7f0c16e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:10.856 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:10.858+0000 7f0c16e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:10.856 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:10.858+0000 7f0c16e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:10.857 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:10.858+0000 7f0c16e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:10.857 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:10.858+0000 7f0c13200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:10.857 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:10.858+0000 7f0c13200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:10.857 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:10.858+0000 7f0c13200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:10.857 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:10.858+0000 7f0c13200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:10.857 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:10.858+0000 7f0c13200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:10.862 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:12:11.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.054+0000 7f5ec9a00700 1 Processor -- start 2024-04-02T09:12:11.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.055+0000 7f5ec9a00700 1 -- start start 2024-04-02T09:12:11.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.056+0000 7f5ec9a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ec40d05e0 0x7f5ec40d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:11.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.056+0000 7f5ec9a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec40d0f80 con 0x7f5ec40b6990 2024-04-02T09:12:11.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.056+0000 7f5ec9a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec40d24e0 con 0x7f5ec415da50 2024-04-02T09:12:11.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.056+0000 7f5ec9a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec40b95d0 con 0x7f5ec40d05e0 2024-04-02T09:12:11.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.056+0000 7f5ec3400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5ec40b6990 0x7f5ec40b6e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57286/0 (socket says 172.21.15.89:57286) 2024-04-02T09:12:11.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.056+0000 7f5ec3400700 1 -- 172.21.15.89:0/3665773479 learned_addr learned my addr 172.21.15.89:0/3665773479 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:11.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.056+0000 7f5ec2000700 1 --2- 172.21.15.89:0/3665773479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ec40d05e0 0x7f5ec40d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:11.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.057+0000 7f5ec1600700 1 -- 172.21.15.89:0/3665773479 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012994367 0 0) 0x7f5ec40d0f80 con 0x7f5ec40b6990 2024-04-02T09:12:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.057+0000 7f5ec1600700 1 -- 172.21.15.89:0/3665773479 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea80035b0 con 0x7f5ec40b6990 2024-04-02T09:12:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.057+0000 7f5ec1600700 1 -- 172.21.15.89:0/3665773479 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913524814 0 0) 0x7f5ec40d24e0 con 0x7f5ec415da50 2024-04-02T09:12:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.057+0000 7f5ec1600700 1 -- 172.21.15.89:0/3665773479 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ec40d0f80 con 0x7f5ec415da50 2024-04-02T09:12:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.057+0000 7f5ec2000700 1 -- 172.21.15.89:0/3665773479 >> v1:172.21.15.113:6789/0 conn(0x7f5ec415da50 legacy=0x7f5ec40ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.057+0000 7f5ec2000700 1 -- 172.21.15.89:0/3665773479 >> v1:172.21.15.89:6789/0 conn(0x7f5ec40b6990 legacy=0x7f5ec40b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.057+0000 7f5ec2000700 1 -- 172.21.15.89:0/3665773479 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec40d27c0 con 0x7f5ec40d05e0 2024-04-02T09:12:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.057+0000 7f5ec2000700 1 --2- 172.21.15.89:0/3665773479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ec40d05e0 0x7f5ec40d0a40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0009870 tx=0x7f5eb0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a406d8516aaeafda server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.057+0000 7f5ec1600700 1 -- 172.21.15.89:0/3665773479 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0011830 con 0x7f5ec40d05e0 2024-04-02T09:12:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.057+0000 7f5ec1600700 1 -- 172.21.15.89:0/3665773479 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb0011990 con 0x7f5ec40d05e0 2024-04-02T09:12:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.057+0000 7f5ec1600700 1 -- 172.21.15.89:0/3665773479 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0011d20 con 0x7f5ec40d05e0 2024-04-02T09:12:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.058+0000 7f5ec9a00700 1 -- 172.21.15.89:0/3665773479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ec40d05e0 msgr2=0x7f5ec40d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.058+0000 7f5ec9a00700 1 --2- 172.21.15.89:0/3665773479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ec40d05e0 0x7f5ec40d0a40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0009870 tx=0x7f5eb0009b80 comp rx=0 tx=0).stop 2024-04-02T09:12:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.058+0000 7f5ec9a00700 1 -- 172.21.15.89:0/3665773479 shutdown_connections 2024-04-02T09:12:11.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.058+0000 7f5ec9a00700 1 --2- 172.21.15.89:0/3665773479 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ec40d05e0 0x7f5ec40d0a40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.058+0000 7f5ec9a00700 1 -- 172.21.15.89:0/3665773479 >> 172.21.15.89:0/3665773479 conn(0x7f5ec40666a0 msgr2=0x7f5ec4066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:11.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.058+0000 7f5ec9a00700 1 -- 172.21.15.89:0/3665773479 shutdown_connections 2024-04-02T09:12:11.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.058+0000 7f5ec9a00700 1 -- 172.21.15.89:0/3665773479 wait complete. 2024-04-02T09:12:11.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.060+0000 7f5ec9a00700 1 Processor -- start 2024-04-02T09:12:11.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.060+0000 7f5ec9a00700 1 -- start start 2024-04-02T09:12:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.061+0000 7f5ec9a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ec40b6990 0x7f5ec40c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.061+0000 7f5ec9a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ec40d05e0 0x7f5ec414cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.061+0000 7f5ec9a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ec415da50 0x7f5ec41f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.061+0000 7f5ec9a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec40c9700 con 0x7f5ec40d05e0 2024-04-02T09:12:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.061+0000 7f5ec9a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec414d260 con 0x7f5ec415da50 2024-04-02T09:12:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.061+0000 7f5ec9a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec41f3eb0 con 0x7f5ec40b6990 2024-04-02T09:12:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.061+0000 7f5ec2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ec40d05e0 0x7f5ec414cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.061+0000 7f5ec2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ec40d05e0 0x7f5ec414cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59298/0 (socket says 172.21.15.89:59298) 2024-04-02T09:12:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.061+0000 7f5ec2000700 1 -- 172.21.15.89:0/820648700 learned_addr learned my addr 172.21.15.89:0/820648700 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.061+0000 7f5ec2a00700 1 --2- 172.21.15.89:0/820648700 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ec40b6990 0x7f5ec40c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.061+0000 7f5ec3400700 1 --2- 172.21.15.89:0/820648700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ec415da50 0x7f5ec41f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.062+0000 7f5ec2000700 1 -- 172.21.15.89:0/820648700 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ec40b6990 msgr2=0x7f5ec40c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.062+0000 7f5ec2000700 1 --2- 172.21.15.89:0/820648700 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ec40b6990 0x7f5ec40c9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.062+0000 7f5ec2000700 1 -- 172.21.15.89:0/820648700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ec415da50 msgr2=0x7f5ec41f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.062+0000 7f5ec2000700 1 --2- 172.21.15.89:0/820648700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ec415da50 0x7f5ec41f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.062+0000 7f5ec2000700 1 -- 172.21.15.89:0/820648700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb00094a0 con 0x7f5ec40d05e0 2024-04-02T09:12:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.062+0000 7f5ec2000700 1 --2- 172.21.15.89:0/820648700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ec40d05e0 0x7f5ec414cd20 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00090b0 tx=0x7f5eb0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.062+0000 7f5ebaa00700 1 -- 172.21.15.89:0/820648700 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb001a6e0 con 0x7f5ec40d05e0 2024-04-02T09:12:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.062+0000 7f5ebaa00700 1 -- 172.21.15.89:0/820648700 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb001a840 con 0x7f5ec40d05e0 2024-04-02T09:12:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.062+0000 7f5ec9a00700 1 -- 172.21.15.89:0/820648700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec41f38f0 con 0x7f5ec40d05e0 2024-04-02T09:12:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.062+0000 7f5ebaa00700 1 -- 172.21.15.89:0/820648700 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb001abb0 con 0x7f5ec40d05e0 2024-04-02T09:12:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.062+0000 7f5ec9a00700 1 -- 172.21.15.89:0/820648700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec41f41c0 con 0x7f5ec40d05e0 2024-04-02T09:12:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.063+0000 7f5ec9a00700 1 -- 172.21.15.89:0/820648700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec4066a80 con 0x7f5ec40d05e0 2024-04-02T09:12:11.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.064+0000 7f5ebaa00700 1 -- 172.21.15.89:0/820648700 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 143) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f5eb0022070 con 0x7f5ec40d05e0 2024-04-02T09:12:11.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.065+0000 7f5ebaa00700 1 -- 172.21.15.89:0/820648700 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6845+0+0 (secure 0 0 0) 0x7f5eb0089c90 con 0x7f5ec40d05e0 2024-04-02T09:12:11.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.070+0000 7f5ebaa00700 1 -- 172.21.15.89:0/820648700 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5eb0054f00 con 0x7f5ec40d05e0 2024-04-02T09:12:11.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.272+0000 7f5ec9a00700 1 -- 172.21.15.89:0/820648700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ec4067720 con 0x7f5ec40d05e0 2024-04-02T09:12:11.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.273+0000 7f5ebaa00700 1 -- 172.21.15.89:0/820648700 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v210) v1 ==== 74+0+17777 (secure 0 0 0) 0x7f5eb0029020 con 0x7f5ec40d05e0 2024-04-02T09:12:11.272 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:11.272 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":210,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:12:10.775947+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:09:40.103721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-04-02T09:09:41.165944+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":209,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.5","mappings":[{"from":7,"to":3}]},{"pgid":"25.3b","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/2022668989":"2024-04-03T09:12:10.775906+0000","172.21.15.89:0/2945017765":"2024-04-03T09:12:10.775906+0000","172.21.15.89:0/2215575860":"2024-04-03T09:12:10.775906+0000","172.21.15.89:0/3196194838":"2024-04-03T09:12:10.775906+0000","172.21.15.89:0/2552487672":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/2965095958":"2024-04-03T09:12:10.775906+0000","172.21.15.89:0/1282451945":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/3725462960":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/3144849693":"2024-04-03T09:12:10.775906+0000","172.21.15.89:0/1115849275":"2024-04-03T09:09:59.528553+0000","172.21.15.89:6833/108855":"2024-04-03T09:09:59.528553+0000","172.21.15.89:6832/108855":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/2251569695":"2024-04-03T09:09:59.528553+0000","172.21.15.89:0/248630547":"2024-04-03T09:09:59.528553+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:12:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.276+0000 7f5ec9a00700 1 -- 172.21.15.89:0/820648700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ec40d05e0 msgr2=0x7f5ec414cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.276+0000 7f5ec9a00700 1 --2- 172.21.15.89:0/820648700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ec40d05e0 0x7f5ec414cd20 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00090b0 tx=0x7f5eb0004750 comp rx=0 tx=0).stop 2024-04-02T09:12:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.276+0000 7f5ec9a00700 1 -- 172.21.15.89:0/820648700 shutdown_connections 2024-04-02T09:12:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.276+0000 7f5ec9a00700 1 --2- 172.21.15.89:0/820648700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ec415da50 0x7f5ec41f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.276+0000 7f5ec9a00700 1 --2- 172.21.15.89:0/820648700 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ec40b6990 0x7f5ec40c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.276+0000 7f5ec9a00700 1 --2- 172.21.15.89:0/820648700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ec40d05e0 0x7f5ec414cd20 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.276+0000 7f5ec9a00700 1 -- 172.21.15.89:0/820648700 >> 172.21.15.89:0/820648700 conn(0x7f5ec40666a0 msgr2=0x7f5ec415a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.276+0000 7f5ec9a00700 1 -- 172.21.15.89:0/820648700 shutdown_connections 2024-04-02T09:12:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.276+0000 7f5ec9a00700 1 -- 172.21.15.89:0/820648700 wait complete. 2024-04-02T09:12:11.292 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T09:12:11.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.487+0000 7f66f7600700 1 Processor -- start 2024-04-02T09:12:11.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.487+0000 7f66f7600700 1 -- start start 2024-04-02T09:12:11.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.488+0000 7f66f7600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f015dfe0 0x7f66f00ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:11.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.488+0000 7f66f7600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f66f00ce670 0x7f66f00cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:11.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.488+0000 7f66f7600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f00cb310 con 0x7f66f015a5c0 2024-04-02T09:12:11.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.488+0000 7f66f7600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66f00ceb00 con 0x7f66f00ce670 2024-04-02T09:12:11.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.488+0000 7f66f7600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f00711a0 con 0x7f66f015dfe0 2024-04-02T09:12:11.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.488+0000 7f66f5600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f66f00ce670 0x7f66f00cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:11.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.488+0000 7f66efe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f015dfe0 0x7f66f00ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:11.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.488+0000 7f66f5600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f66f00ce670 0x7f66f00cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:45586/0 (socket says 172.21.15.89:45586) 2024-04-02T09:12:11.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.488+0000 7f66f5600700 1 -- 172.21.15.89:0/1179581210 learned_addr learned my addr 172.21.15.89:0/1179581210 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:11.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.488+0000 7f66f4c00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f66f015a5c0 0x7f66f00709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:52608/0 (socket says 172.21.15.89:52608) 2024-04-02T09:12:11.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.489+0000 7f66f5600700 1 -- 172.21.15.89:0/1179581210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f015dfe0 msgr2=0x7f66f00ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:11.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.489+0000 7f66f5600700 1 --2- 172.21.15.89:0/1179581210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f015dfe0 0x7f66f00ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.489+0000 7f66f5600700 1 -- 172.21.15.89:0/1179581210 >> v1:172.21.15.113:6789/0 conn(0x7f66f015a5c0 legacy=0x7f66f00709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.489+0000 7f66f5600700 1 -- 172.21.15.89:0/1179581210 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f0071480 con 0x7f66f00ce670 2024-04-02T09:12:11.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.489+0000 7f66efe00700 1 --2- 172.21.15.89:0/1179581210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f015dfe0 0x7f66f00ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:12:11.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.489+0000 7f66ef400700 1 -- 172.21.15.89:0/1179581210 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076391956 0 0) 0x7f66f00cb310 con 0x7f66f015a5c0 2024-04-02T09:12:11.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.489+0000 7f66f5600700 1 --2- 172.21.15.89:0/1179581210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f00ce670 0x7f66f00cba80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009e50 tx=0x7f66e800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=770357231c8fdb70 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:11.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.489+0000 7f66ef400700 1 -- 172.21.15.89:0/1179581210 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e800e070 con 0x7f66f00ce670 2024-04-02T09:12:11.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.489+0000 7f66ef400700 1 -- 172.21.15.89:0/1179581210 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e800b910 con 0x7f66f00ce670 2024-04-02T09:12:11.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.489+0000 7f66ef400700 1 -- 172.21.15.89:0/1179581210 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e800bcc0 con 0x7f66f00ce670 2024-04-02T09:12:11.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.490+0000 7f66f7600700 1 -- 172.21.15.89:0/1179581210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f00ce670 msgr2=0x7f66f00cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.490+0000 7f66f7600700 1 --2- 172.21.15.89:0/1179581210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f00ce670 0x7f66f00cba80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009e50 tx=0x7f66e800b330 comp rx=0 tx=0).stop 2024-04-02T09:12:11.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.490+0000 7f66f7600700 1 -- 172.21.15.89:0/1179581210 shutdown_connections 2024-04-02T09:12:11.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.490+0000 7f66f7600700 1 --2- 172.21.15.89:0/1179581210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f00ce670 0x7f66f00cba80 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.490+0000 7f66f7600700 1 --2- 172.21.15.89:0/1179581210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f015dfe0 0x7f66f00ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.490+0000 7f66f7600700 1 -- 172.21.15.89:0/1179581210 >> 172.21.15.89:0/1179581210 conn(0x7f66f00b0d70 msgr2=0x7f66f00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:11.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.490+0000 7f66f7600700 1 -- 172.21.15.89:0/1179581210 shutdown_connections 2024-04-02T09:12:11.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.490+0000 7f66f7600700 1 -- 172.21.15.89:0/1179581210 wait complete. 2024-04-02T09:12:11.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.492+0000 7f66f7600700 1 Processor -- start 2024-04-02T09:12:11.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.492+0000 7f66f7600700 1 -- start start 2024-04-02T09:12:11.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.493+0000 7f66f7600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f00ce670 0x7f66f00b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:11.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.493+0000 7f66f7600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f015a5c0 0x7f66f014dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:11.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.493+0000 7f66f7600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f015dfe0 0x7f66f01ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:11.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.493+0000 7f66f7600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f00b06b0 con 0x7f66f015a5c0 2024-04-02T09:12:11.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.493+0000 7f66f7600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f014e180 con 0x7f66f015dfe0 2024-04-02T09:12:11.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.493+0000 7f66f7600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f00b0b80 con 0x7f66f00ce670 2024-04-02T09:12:11.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.493+0000 7f66efe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f015a5c0 0x7f66f014dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:11.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.493+0000 7f66efe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f015a5c0 0x7f66f014dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59304/0 (socket says 172.21.15.89:59304) 2024-04-02T09:12:11.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.493+0000 7f66efe00700 1 -- 172.21.15.89:0/1414724955 learned_addr learned my addr 172.21.15.89:0/1414724955 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:11.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.494+0000 7f66f4c00700 1 --2- 172.21.15.89:0/1414724955 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f00ce670 0x7f66f00b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:11.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.494+0000 7f66f5600700 1 --2- 172.21.15.89:0/1414724955 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f015dfe0 0x7f66f01ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:11.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.494+0000 7f66efe00700 1 -- 172.21.15.89:0/1414724955 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f00ce670 msgr2=0x7f66f00b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.494+0000 7f66efe00700 1 --2- 172.21.15.89:0/1414724955 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f00ce670 0x7f66f00b0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.494+0000 7f66efe00700 1 -- 172.21.15.89:0/1414724955 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f015dfe0 msgr2=0x7f66f01ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.494+0000 7f66efe00700 1 --2- 172.21.15.89:0/1414724955 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f015dfe0 0x7f66f01ee3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.494+0000 7f66efe00700 1 -- 172.21.15.89:0/1414724955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc0094a0 con 0x7f66f015a5c0 2024-04-02T09:12:11.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.494+0000 7f66f4c00700 1 --2- 172.21.15.89:0/1414724955 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f00ce670 0x7f66f00b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:11.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.494+0000 7f66efe00700 1 --2- 172.21.15.89:0/1414724955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f015a5c0 0x7f66f014dc40 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00ebe0 tx=0x7f66dc009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:11.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.494+0000 7f66ecc00700 1 -- 172.21.15.89:0/1414724955 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc00c8b0 con 0x7f66f015a5c0 2024-04-02T09:12:11.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.494+0000 7f66ecc00700 1 -- 172.21.15.89:0/1414724955 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66dc00ca10 con 0x7f66f015a5c0 2024-04-02T09:12:11.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.494+0000 7f66f7600700 1 -- 172.21.15.89:0/1414724955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e8009b30 con 0x7f66f015a5c0 2024-04-02T09:12:11.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.494+0000 7f66ecc00700 1 -- 172.21.15.89:0/1414724955 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc00cda0 con 0x7f66f015a5c0 2024-04-02T09:12:11.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.494+0000 7f66f7600700 1 -- 172.21.15.89:0/1414724955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f00aee10 con 0x7f66f015a5c0 2024-04-02T09:12:11.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.495+0000 7f66f7600700 1 -- 172.21.15.89:0/1414724955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f0067720 con 0x7f66f015a5c0 2024-04-02T09:12:11.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.496+0000 7f66ecc00700 1 -- 172.21.15.89:0/1414724955 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 143) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f66dc013470 con 0x7f66f015a5c0 2024-04-02T09:12:11.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.497+0000 7f66ecc00700 1 -- 172.21.15.89:0/1414724955 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6845+0+0 (secure 0 0 0) 0x7f66dc089390 con 0x7f66f015a5c0 2024-04-02T09:12:11.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.502+0000 7f66ecc00700 1 -- 172.21.15.89:0/1414724955 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f66dc0544d0 con 0x7f66f015a5c0 2024-04-02T09:12:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.748+0000 7f66f7600700 1 -- 172.21.15.89:0/1414724955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f66f00b1ad0 con 0x7f66f015a5c0 2024-04-02T09:12:11.747 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T09:12:11.749+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T09:12:11.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.758+0000 7f66ecc00700 1 -- 172.21.15.89:0/1414724955 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v98) v1 ==== 161+0+0 (secure 0 0 0) 0x7f66dc057af0 con 0x7f66f015a5c0 2024-04-02T09:12:11.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.761+0000 7f66f7600700 1 -- 172.21.15.89:0/1414724955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f015a5c0 msgr2=0x7f66f014dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.761+0000 7f66f7600700 1 --2- 172.21.15.89:0/1414724955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f015a5c0 0x7f66f014dc40 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00ebe0 tx=0x7f66dc009e20 comp rx=0 tx=0).stop 2024-04-02T09:12:11.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.761+0000 7f66f7600700 1 -- 172.21.15.89:0/1414724955 shutdown_connections 2024-04-02T09:12:11.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.761+0000 7f66f7600700 1 --2- 172.21.15.89:0/1414724955 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66f015dfe0 0x7f66f01ee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.761+0000 7f66f7600700 1 --2- 172.21.15.89:0/1414724955 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f66f00ce670 0x7f66f00b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.761+0000 7f66f7600700 1 --2- 172.21.15.89:0/1414724955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f66f015a5c0 0x7f66f014dc40 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.761+0000 7f66f7600700 1 -- 172.21.15.89:0/1414724955 >> 172.21.15.89:0/1414724955 conn(0x7f66f00b0d70 msgr2=0x7f66f00b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:11.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.761+0000 7f66f7600700 1 -- 172.21.15.89:0/1414724955 shutdown_connections 2024-04-02T09:12:11.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.761+0000 7f66f7600700 1 -- 172.21.15.89:0/1414724955 wait complete. 2024-04-02T09:12:11.760 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T09:12:11.778 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T09:12:11.831 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T09:12:11.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.981+0000 7fdb32400700 1 Processor -- start 2024-04-02T09:12:11.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.981+0000 7fdb32400700 1 -- start start 2024-04-02T09:12:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.982+0000 7fdb32400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdb2c15d9d0 0x7fdb2c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.982+0000 7fdb32400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2c161420 con 0x7fdb2c0d2540 2024-04-02T09:12:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.982+0000 7fdb32400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2c161700 con 0x7fdb2c0d16a0 2024-04-02T09:12:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.982+0000 7fdb32400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb2c15edc0 con 0x7fdb2c15d9d0 2024-04-02T09:12:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.983+0000 7fdb2be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdb2c0d2540 0x7fdb2c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57300/0 (socket says 172.21.15.89:57300) 2024-04-02T09:12:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.983+0000 7fdb2be00700 1 -- 172.21.15.89:0/3840160110 learned_addr learned my addr 172.21.15.89:0/3840160110 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.983+0000 7fdb2b400700 1 --2- 172.21.15.89:0/3840160110 >> v2:172.21.15.89:3300/0 conn(0x7fdb2c15d9d0 0x7fdb2c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.983+0000 7fdb2a000700 1 -- 172.21.15.89:0/3840160110 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254512354 0 0) 0x7fdb2c161420 con 0x7fdb2c0d2540 2024-04-02T09:12:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.983+0000 7fdb2a000700 1 -- 172.21.15.89:0/3840160110 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb100035b0 con 0x7fdb2c0d2540 2024-04-02T09:12:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.983+0000 7fdb2b400700 1 -- 172.21.15.89:0/3840160110 >> v1:172.21.15.113:6789/0 conn(0x7fdb2c0d16a0 legacy=0x7fdb2c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.983+0000 7fdb2b400700 1 -- 172.21.15.89:0/3840160110 >> v1:172.21.15.89:6789/0 conn(0x7fdb2c0d2540 legacy=0x7fdb2c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.983+0000 7fdb2b400700 1 -- 172.21.15.89:0/3840160110 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2c1619e0 con 0x7fdb2c15d9d0 2024-04-02T09:12:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.984+0000 7fdb2b400700 1 --2- 172.21.15.89:0/3840160110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb2c15d9d0 0x7fdb2c0d3c30 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009840 tx=0x7fdb1c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd95a0e87e9ae013 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.984+0000 7fdb2a000700 1 -- 172.21.15.89:0/3840160110 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c011830 con 0x7fdb2c15d9d0 2024-04-02T09:12:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.984+0000 7fdb2a000700 1 -- 172.21.15.89:0/3840160110 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c011990 con 0x7fdb2c15d9d0 2024-04-02T09:12:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.984+0000 7fdb2a000700 1 -- 172.21.15.89:0/3840160110 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c011d20 con 0x7fdb2c15d9d0 2024-04-02T09:12:11.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.984+0000 7fdb32400700 1 -- 172.21.15.89:0/3840160110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb2c15d9d0 msgr2=0x7fdb2c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.984+0000 7fdb32400700 1 --2- 172.21.15.89:0/3840160110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb2c15d9d0 0x7fdb2c0d3c30 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009840 tx=0x7fdb1c009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:11.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.984+0000 7fdb32400700 1 -- 172.21.15.89:0/3840160110 shutdown_connections 2024-04-02T09:12:11.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.984+0000 7fdb32400700 1 --2- 172.21.15.89:0/3840160110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb2c15d9d0 0x7fdb2c0d3c30 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.984+0000 7fdb32400700 1 -- 172.21.15.89:0/3840160110 >> 172.21.15.89:0/3840160110 conn(0x7fdb2c08aff0 msgr2=0x7fdb2c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:11.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.984+0000 7fdb32400700 1 -- 172.21.15.89:0/3840160110 shutdown_connections 2024-04-02T09:12:11.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.985+0000 7fdb32400700 1 -- 172.21.15.89:0/3840160110 wait complete. 2024-04-02T09:12:11.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.986+0000 7fdb32400700 1 Processor -- start 2024-04-02T09:12:11.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.986+0000 7fdb32400700 1 -- start start 2024-04-02T09:12:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.987+0000 7fdb32400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb2c0d16a0 0x7fdb2c0b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.987+0000 7fdb32400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb2c0d2540 0x7fdb2c14f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.987+0000 7fdb32400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c15d9d0 0x7fdb2c14f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.987+0000 7fdb32400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c0cba80 con 0x7fdb2c0d16a0 2024-04-02T09:12:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.987+0000 7fdb32400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c14fdd0 con 0x7fdb2c15d9d0 2024-04-02T09:12:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.987+0000 7fdb32400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c1e8ca0 con 0x7fdb2c0d2540 2024-04-02T09:12:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.987+0000 7fdb2b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb2c0d16a0 0x7fdb2c0b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.987+0000 7fdb2b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb2c0d16a0 0x7fdb2c0b7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59314/0 (socket says 172.21.15.89:59314) 2024-04-02T09:12:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.987+0000 7fdb2b400700 1 -- 172.21.15.89:0/326893542 learned_addr learned my addr 172.21.15.89:0/326893542 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.987+0000 7fdb2aa00700 1 --2- 172.21.15.89:0/326893542 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb2c0d2540 0x7fdb2c14f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.988+0000 7fdb2be00700 1 --2- 172.21.15.89:0/326893542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c15d9d0 0x7fdb2c14f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.988+0000 7fdb2b400700 1 -- 172.21.15.89:0/326893542 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb2c0d2540 msgr2=0x7fdb2c14f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.988+0000 7fdb2b400700 1 --2- 172.21.15.89:0/326893542 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb2c0d2540 0x7fdb2c14f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.988+0000 7fdb2b400700 1 -- 172.21.15.89:0/326893542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c15d9d0 msgr2=0x7fdb2c14f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.988+0000 7fdb2b400700 1 --2- 172.21.15.89:0/326893542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c15d9d0 0x7fdb2c14f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.988+0000 7fdb2b400700 1 -- 172.21.15.89:0/326893542 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1c0094a0 con 0x7fdb2c0d16a0 2024-04-02T09:12:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.988+0000 7fdb2b400700 1 --2- 172.21.15.89:0/326893542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb2c0d16a0 0x7fdb2c0b7dd0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c000c00 tx=0x7fdb1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.988+0000 7fdb23400700 1 -- 172.21.15.89:0/326893542 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c0036a0 con 0x7fdb2c0d16a0 2024-04-02T09:12:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.988+0000 7fdb23400700 1 -- 172.21.15.89:0/326893542 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c01e070 con 0x7fdb2c0d16a0 2024-04-02T09:12:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.989+0000 7fdb23400700 1 -- 172.21.15.89:0/326893542 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c003a20 con 0x7fdb2c0d16a0 2024-04-02T09:12:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.989+0000 7fdb32400700 1 -- 172.21.15.89:0/326893542 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb2c0cb3c0 con 0x7fdb2c0d16a0 2024-04-02T09:12:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.989+0000 7fdb32400700 1 -- 172.21.15.89:0/326893542 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb2c0cb070 con 0x7fdb2c0d16a0 2024-04-02T09:12:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.990+0000 7fdb32400700 1 -- 172.21.15.89:0/326893542 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2c0d1ca0 con 0x7fdb2c0d16a0 2024-04-02T09:12:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.991+0000 7fdb23400700 1 -- 172.21.15.89:0/326893542 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 144) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fdb1c003b80 con 0x7fdb2c0d16a0 2024-04-02T09:12:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.991+0000 7fdb23400700 1 --2- 172.21.15.89:0/326893542 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb0c06aae0 0x7fdb0c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.992+0000 7fdb23400700 1 -- 172.21.15.89:0/326893542 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6931+0+0 (secure 0 0 0) 0x7fdb1c08a2f0 con 0x7fdb2c0d16a0 2024-04-02T09:12:11.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.992+0000 7fdb2aa00700 1 --2- 172.21.15.89:0/326893542 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb0c06aae0 0x7fdb0c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:11.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.992+0000 7fdb2aa00700 1 --2- 172.21.15.89:0/326893542 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb0c06aae0 0x7fdb0c06cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c15d1b0 tx=0x7fdb18009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:11.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:11.996+0000 7fdb23400700 1 -- 172.21.15.89:0/326893542 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdb1c055340 con 0x7fdb2c0d16a0 2024-04-02T09:12:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:12.242+0000 7fdb32400700 1 -- 172.21.15.89:0/326893542 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdb2c067720 con 0x7fdb2c0d16a0 2024-04-02T09:12:12.512 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:12:12.517 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:12:12.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:12.789+0000 7fdb23400700 1 -- 172.21.15.89:0/326893542 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v99) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdb1c058960 con 0x7fdb2c0d16a0 2024-04-02T09:12:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:12.791+0000 7fdb32400700 1 -- 172.21.15.89:0/326893542 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb0c06aae0 msgr2=0x7fdb0c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:12.791+0000 7fdb32400700 1 --2- 172.21.15.89:0/326893542 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb0c06aae0 0x7fdb0c06cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c15d1b0 tx=0x7fdb18009040 comp rx=0 tx=0).stop 2024-04-02T09:12:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:12.791+0000 7fdb32400700 1 -- 172.21.15.89:0/326893542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb2c0d16a0 msgr2=0x7fdb2c0b7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:12.791+0000 7fdb32400700 1 --2- 172.21.15.89:0/326893542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb2c0d16a0 0x7fdb2c0b7dd0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c000c00 tx=0x7fdb1c004750 comp rx=0 tx=0).stop 2024-04-02T09:12:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:12.791+0000 7fdb32400700 1 -- 172.21.15.89:0/326893542 shutdown_connections 2024-04-02T09:12:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:12.791+0000 7fdb32400700 1 --2- 172.21.15.89:0/326893542 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb0c06aae0 0x7fdb0c06cfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:12.791+0000 7fdb32400700 1 --2- 172.21.15.89:0/326893542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c15d9d0 0x7fdb2c14f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:12.791+0000 7fdb32400700 1 --2- 172.21.15.89:0/326893542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb2c0d16a0 0x7fdb2c0b7dd0 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:12.791+0000 7fdb32400700 1 --2- 172.21.15.89:0/326893542 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb2c0d2540 0x7fdb2c14f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:12.791+0000 7fdb32400700 1 -- 172.21.15.89:0/326893542 >> 172.21.15.89:0/326893542 conn(0x7fdb2c08aff0 msgr2=0x7fdb2c0ceca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:12.791+0000 7fdb32400700 1 -- 172.21.15.89:0/326893542 shutdown_connections 2024-04-02T09:12:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:12.791+0000 7fdb32400700 1 -- 172.21.15.89:0/326893542 wait complete. 2024-04-02T09:12:12.809 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T09:12:13.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.011+0000 7f3c04200700 1 Processor -- start 2024-04-02T09:12:13.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.011+0000 7f3c04200700 1 -- start start 2024-04-02T09:12:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.012+0000 7f3c04200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3bfc06aea0 0x7f3bfc06dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.012+0000 7f3c04200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc15de10 con 0x7f3bfc0d0bd0 2024-04-02T09:12:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.012+0000 7f3c04200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc0b4fd0 con 0x7f3bfc15e8d0 2024-04-02T09:12:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.012+0000 7f3c04200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bfc0b52b0 con 0x7f3bfc06aea0 2024-04-02T09:12:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.012+0000 7f3c02200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3bfc06aea0 0x7f3bfc06dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.012+0000 7f3c01800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3bfc0d0bd0 0x7f3bfc0b3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57314/0 (socket says 172.21.15.89:57314) 2024-04-02T09:12:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.012+0000 7f3c01800700 1 -- 172.21.15.89:0/4178273062 learned_addr learned my addr 172.21.15.89:0/4178273062 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:13.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.013+0000 7f3c02200700 1 -- 172.21.15.89:0/4178273062 >> v1:172.21.15.113:6789/0 conn(0x7f3bfc15e8d0 legacy=0x7f3bfc06a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:13.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.013+0000 7f3c02200700 1 -- 172.21.15.89:0/4178273062 >> v1:172.21.15.89:6789/0 conn(0x7f3bfc0d0bd0 legacy=0x7f3bfc0b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:13.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.013+0000 7f3c02200700 1 -- 172.21.15.89:0/4178273062 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc15e0d0 con 0x7f3bfc06aea0 2024-04-02T09:12:13.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.013+0000 7f3bfbe00700 1 -- 172.21.15.89:0/4178273062 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608616331 0 0) 0x7f3bfc15de10 con 0x7f3bfc0d0bd0 2024-04-02T09:12:13.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.013+0000 7f3c02200700 1 --2- 172.21.15.89:0/4178273062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bfc06aea0 0x7f3bfc06dae0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4009d90 tx=0x7f3bf400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e501c8061beb173 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:13.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.013+0000 7f3bfbe00700 1 -- 172.21.15.89:0/4178273062 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf400e070 con 0x7f3bfc06aea0 2024-04-02T09:12:13.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.013+0000 7f3bfbe00700 1 -- 172.21.15.89:0/4178273062 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf400bbd0 con 0x7f3bfc06aea0 2024-04-02T09:12:13.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.013+0000 7f3bfbe00700 1 -- 172.21.15.89:0/4178273062 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4012430 con 0x7f3bfc06aea0 2024-04-02T09:12:13.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.014+0000 7f3c04200700 1 -- 172.21.15.89:0/4178273062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bfc06aea0 msgr2=0x7f3bfc06dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:13.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.014+0000 7f3c04200700 1 --2- 172.21.15.89:0/4178273062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bfc06aea0 0x7f3bfc06dae0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4009d90 tx=0x7f3bf400b330 comp rx=0 tx=0).stop 2024-04-02T09:12:13.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.014+0000 7f3c04200700 1 -- 172.21.15.89:0/4178273062 shutdown_connections 2024-04-02T09:12:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.014+0000 7f3c04200700 1 --2- 172.21.15.89:0/4178273062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bfc06aea0 0x7f3bfc06dae0 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.014+0000 7f3c04200700 1 -- 172.21.15.89:0/4178273062 >> 172.21.15.89:0/4178273062 conn(0x7f3bfc0b5640 msgr2=0x7f3bfc0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.014+0000 7f3c04200700 1 -- 172.21.15.89:0/4178273062 shutdown_connections 2024-04-02T09:12:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.014+0000 7f3c04200700 1 -- 172.21.15.89:0/4178273062 wait complete. 2024-04-02T09:12:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.016+0000 7f3c04200700 1 Processor -- start 2024-04-02T09:12:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.017+0000 7f3c04200700 1 -- start start 2024-04-02T09:12:13.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c04200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bfc06aea0 0x7f3bfc159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:13.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c04200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bfc0d0bd0 0x7f3bfc157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:13.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c04200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bfc15e8d0 0x7f3bfc15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:13.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c04200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc1595c0 con 0x7f3bfc06aea0 2024-04-02T09:12:13.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c04200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc159130 con 0x7f3bfc15e8d0 2024-04-02T09:12:13.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c04200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc158520 con 0x7f3bfc0d0bd0 2024-04-02T09:12:13.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c01800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bfc06aea0 0x7f3bfc159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:13.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c01800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bfc06aea0 0x7f3bfc159f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59344/0 (socket says 172.21.15.89:59344) 2024-04-02T09:12:13.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c01800700 1 -- 172.21.15.89:0/2071295400 learned_addr learned my addr 172.21.15.89:0/2071295400 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:13.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c02200700 1 --2- 172.21.15.89:0/2071295400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bfc15e8d0 0x7f3bfc15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:13.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c00e00700 1 --2- 172.21.15.89:0/2071295400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bfc0d0bd0 0x7f3bfc157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:13.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c01800700 1 -- 172.21.15.89:0/2071295400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bfc0d0bd0 msgr2=0x7f3bfc157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:13.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c01800700 1 --2- 172.21.15.89:0/2071295400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bfc0d0bd0 0x7f3bfc157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:13.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c01800700 1 -- 172.21.15.89:0/2071295400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bfc15e8d0 msgr2=0x7f3bfc15c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:13.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c01800700 1 --2- 172.21.15.89:0/2071295400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bfc15e8d0 0x7f3bfc15c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:13.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.018+0000 7f3c01800700 1 -- 172.21.15.89:0/2071295400 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf4009a70 con 0x7f3bfc06aea0 2024-04-02T09:12:13.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.019+0000 7f3c01800700 1 --2- 172.21.15.89:0/2071295400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bfc06aea0 0x7f3bfc159f60 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c520 tx=0x7f3bf000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:13.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.019+0000 7f3bf9600700 1 -- 172.21.15.89:0/2071295400 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0004c80 con 0x7f3bfc06aea0 2024-04-02T09:12:13.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.019+0000 7f3bf9600700 1 -- 172.21.15.89:0/2071295400 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf0004e00 con 0x7f3bfc06aea0 2024-04-02T09:12:13.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.019+0000 7f3bf9600700 1 -- 172.21.15.89:0/2071295400 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0007310 con 0x7f3bfc06aea0 2024-04-02T09:12:13.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.019+0000 7f3c04200700 1 -- 172.21.15.89:0/2071295400 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bfc158800 con 0x7f3bfc06aea0 2024-04-02T09:12:13.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.019+0000 7f3c04200700 1 -- 172.21.15.89:0/2071295400 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bfc158dc0 con 0x7f3bfc06aea0 2024-04-02T09:12:13.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.020+0000 7f3c04200700 1 -- 172.21.15.89:0/2071295400 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bfc067720 con 0x7f3bfc06aea0 2024-04-02T09:12:13.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.021+0000 7f3bf9600700 1 -- 172.21.15.89:0/2071295400 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 144) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f3bf0007470 con 0x7f3bfc06aea0 2024-04-02T09:12:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.022+0000 7f3bf9600700 1 --2- 172.21.15.89:0/2071295400 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3bdc06aae0 0x7f3bdc06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.022+0000 7f3bf9600700 1 -- 172.21.15.89:0/2071295400 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6931+0+0 (secure 0 0 0) 0x7f3bf0089220 con 0x7f3bfc06aea0 2024-04-02T09:12:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.022+0000 7f3c00e00700 1 --2- 172.21.15.89:0/2071295400 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3bdc06aae0 0x7f3bdc06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.023+0000 7f3c00e00700 1 --2- 172.21.15.89:0/2071295400 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3bdc06aae0 0x7f3bdc06cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0d0930 tx=0x7f3be8009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:13.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.027+0000 7f3bf9600700 1 -- 172.21.15.89:0/2071295400 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3bf00541a0 con 0x7f3bfc06aea0 2024-04-02T09:12:13.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.229+0000 7f3c04200700 1 -- 172.21.15.89:0/2071295400 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3bfc06f420 con 0x7f3bfc06aea0 2024-04-02T09:12:13.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.677+0000 7f3bf9600700 1 -- 172.21.15.89:0/2071295400 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 145) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f3bf0054570 con 0x7f3bfc06aea0 2024-04-02T09:12:13.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.804+0000 7f3bf9600700 1 -- 172.21.15.89:0/2071295400 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v212) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3bf0073ed0 con 0x7f3bfc06aea0 2024-04-02T09:12:13.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.808+0000 7f3c04200700 1 -- 172.21.15.89:0/2071295400 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3bdc06aae0 msgr2=0x7f3bdc06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.808+0000 7f3c04200700 1 --2- 172.21.15.89:0/2071295400 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3bdc06aae0 0x7f3bdc06cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0d0930 tx=0x7f3be8009040 comp rx=0 tx=0).stop 2024-04-02T09:12:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.808+0000 7f3c04200700 1 -- 172.21.15.89:0/2071295400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bfc06aea0 msgr2=0x7f3bfc159f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.808+0000 7f3c04200700 1 --2- 172.21.15.89:0/2071295400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bfc06aea0 0x7f3bfc159f60 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c520 tx=0x7f3bf000c8e0 comp rx=0 tx=0).stop 2024-04-02T09:12:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.809+0000 7f3c04200700 1 -- 172.21.15.89:0/2071295400 shutdown_connections 2024-04-02T09:12:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.809+0000 7f3c04200700 1 --2- 172.21.15.89:0/2071295400 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3bdc06aae0 0x7f3bdc06cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.809+0000 7f3c04200700 1 --2- 172.21.15.89:0/2071295400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bfc15e8d0 0x7f3bfc15c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.809+0000 7f3c04200700 1 --2- 172.21.15.89:0/2071295400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bfc06aea0 0x7f3bfc159f60 secure :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c520 tx=0x7f3bf000c8e0 comp rx=0 tx=0).stop 2024-04-02T09:12:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.809+0000 7f3c04200700 1 --2- 172.21.15.89:0/2071295400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bfc0d0bd0 0x7f3bfc157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:13.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.809+0000 7f3c04200700 1 -- 172.21.15.89:0/2071295400 >> 172.21.15.89:0/2071295400 conn(0x7f3bfc0b5640 msgr2=0x7f3bfc0b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:13.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.809+0000 7f3c04200700 1 -- 172.21.15.89:0/2071295400 shutdown_connections 2024-04-02T09:12:13.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:13.809+0000 7f3c04200700 1 -- 172.21.15.89:0/2071295400 wait complete. 2024-04-02T09:12:13.809 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T09:12:13.827 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T09:12:14.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.024+0000 7f54a2a00700 1 Processor -- start 2024-04-02T09:12:14.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.024+0000 7f54a2a00700 1 -- start start 2024-04-02T09:12:14.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.026+0000 7f54a2a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f549c0ced30 0x7f549c0d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:14.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.026+0000 7f54a2a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f549c0ce8a0 con 0x7f549c0d2710 2024-04-02T09:12:14.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.026+0000 7f54a2a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f549c0cdc90 con 0x7f549c15ddf0 2024-04-02T09:12:14.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.026+0000 7f54a2a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549c15ec60 con 0x7f549c0ced30 2024-04-02T09:12:14.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.026+0000 7f54a0a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f549c0d2710 0x7f549c15f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57324/0 (socket says 172.21.15.89:57324) 2024-04-02T09:12:14.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.026+0000 7f54a0a00700 1 -- 172.21.15.89:0/164088213 learned_addr learned my addr 172.21.15.89:0/164088213 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:14.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.026+0000 7f549b400700 1 --2- 172.21.15.89:0/164088213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f549c0ced30 0x7f549c0d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:14.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.026+0000 7f549aa00700 1 -- 172.21.15.89:0/164088213 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390134394 0 0) 0x7f549c0ce8a0 con 0x7f549c0d2710 2024-04-02T09:12:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.026+0000 7f549aa00700 1 -- 172.21.15.89:0/164088213 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54800035b0 con 0x7f549c0d2710 2024-04-02T09:12:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.027+0000 7f549aa00700 1 -- 172.21.15.89:0/164088213 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444112379 0 0) 0x7f549c0cdc90 con 0x7f549c15ddf0 2024-04-02T09:12:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.027+0000 7f549b400700 1 -- 172.21.15.89:0/164088213 >> v1:172.21.15.113:6789/0 conn(0x7f549c15ddf0 legacy=0x7f549c15e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.027+0000 7f549b400700 1 -- 172.21.15.89:0/164088213 >> v1:172.21.15.89:6789/0 conn(0x7f549c0d2710 legacy=0x7f549c15f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.027+0000 7f549b400700 1 -- 172.21.15.89:0/164088213 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549c0cdf70 con 0x7f549c0ced30 2024-04-02T09:12:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.027+0000 7f549b400700 1 --2- 172.21.15.89:0/164088213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f549c0ced30 0x7f549c0d21d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5488009870 tx=0x7f5488009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb70d2d54a83b435 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.027+0000 7f549aa00700 1 -- 172.21.15.89:0/164088213 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5488011830 con 0x7f549c0ced30 2024-04-02T09:12:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.027+0000 7f549aa00700 1 -- 172.21.15.89:0/164088213 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5488011990 con 0x7f549c0ced30 2024-04-02T09:12:14.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.027+0000 7f549aa00700 1 -- 172.21.15.89:0/164088213 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5488011d00 con 0x7f549c0ced30 2024-04-02T09:12:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.028+0000 7f54a2a00700 1 -- 172.21.15.89:0/164088213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f549c0ced30 msgr2=0x7f549c0d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.028+0000 7f54a2a00700 1 --2- 172.21.15.89:0/164088213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f549c0ced30 0x7f549c0d21d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5488009870 tx=0x7f5488009b80 comp rx=0 tx=0).stop 2024-04-02T09:12:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.028+0000 7f54a2a00700 1 -- 172.21.15.89:0/164088213 shutdown_connections 2024-04-02T09:12:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.028+0000 7f54a2a00700 1 --2- 172.21.15.89:0/164088213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f549c0ced30 0x7f549c0d21d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.028+0000 7f54a2a00700 1 -- 172.21.15.89:0/164088213 >> 172.21.15.89:0/164088213 conn(0x7f549c08aff0 msgr2=0x7f549c0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.028+0000 7f54a2a00700 1 -- 172.21.15.89:0/164088213 shutdown_connections 2024-04-02T09:12:14.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.028+0000 7f54a2a00700 1 -- 172.21.15.89:0/164088213 wait complete. 2024-04-02T09:12:14.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.030+0000 7f54a2a00700 1 Processor -- start 2024-04-02T09:12:14.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.030+0000 7f54a2a00700 1 -- start start 2024-04-02T09:12:14.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f54a2a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0ced30 0x7f549c1563e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:14.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f54a2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f549c0d2710 0x7f549c1602a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:14.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f54a2a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f549c15ddf0 0x7f549c1607e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:14.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f54a2a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c160d20 con 0x7f549c0d2710 2024-04-02T09:12:14.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f54a2a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c156f60 con 0x7f549c0ced30 2024-04-02T09:12:14.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f54a2a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549c160f40 con 0x7f549c15ddf0 2024-04-02T09:12:14.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f549b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f549c0d2710 0x7f549c1602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:14.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f549b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f549c0d2710 0x7f549c1602a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59356/0 (socket says 172.21.15.89:59356) 2024-04-02T09:12:14.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f549b400700 1 -- 172.21.15.89:0/1065622235 learned_addr learned my addr 172.21.15.89:0/1065622235 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:14.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f549be00700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0ced30 0x7f549c1563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:14.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f549b400700 1 -- 172.21.15.89:0/1065622235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f549c15ddf0 msgr2=0x7f549c1607e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:14.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f54a0a00700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f549c15ddf0 0x7f549c1607e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:14.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f549b400700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f549c15ddf0 0x7f549c1607e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:14.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f549b400700 1 -- 172.21.15.89:0/1065622235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0ced30 msgr2=0x7f549c1563e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:14.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f549b400700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0ced30 0x7f549c1563e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:14.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f549b400700 1 -- 172.21.15.89:0/1065622235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54880094a0 con 0x7f549c0d2710 2024-04-02T09:12:14.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f549be00700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0ced30 0x7f549c1563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:14.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.031+0000 7f54a0a00700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f549c15ddf0 0x7f549c1607e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:14.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.032+0000 7f549b400700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f549c0d2710 0x7f549c1602a0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f54880097c0 tx=0x7f5488004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:14.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.032+0000 7f5493e00700 1 -- 172.21.15.89:0/1065622235 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54880036a0 con 0x7f549c0d2710 2024-04-02T09:12:14.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.032+0000 7f5493e00700 1 -- 172.21.15.89:0/1065622235 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548801f070 con 0x7f549c0d2710 2024-04-02T09:12:14.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.032+0000 7f54a2a00700 1 -- 172.21.15.89:0/1065622235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f549c161220 con 0x7f549c0d2710 2024-04-02T09:12:14.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.032+0000 7f5493e00700 1 -- 172.21.15.89:0/1065622235 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5488003a40 con 0x7f549c0d2710 2024-04-02T09:12:14.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.032+0000 7f54a2a00700 1 -- 172.21.15.89:0/1065622235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549c1615b0 con 0x7f549c0d2710 2024-04-02T09:12:14.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.033+0000 7f54a2a00700 1 -- 172.21.15.89:0/1065622235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549c067720 con 0x7f549c0d2710 2024-04-02T09:12:14.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.034+0000 7f5493e00700 1 -- 172.21.15.89:0/1065622235 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 145) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f5488003ba0 con 0x7f549c0d2710 2024-04-02T09:12:14.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.035+0000 7f5493e00700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f547c06ad20 0x7f547c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:14.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.035+0000 7f5493e00700 1 -- 172.21.15.89:0/1065622235 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6493+0+0 (secure 0 0 0) 0x7f5488021070 con 0x7f549c0d2710 2024-04-02T09:12:14.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.035+0000 7f549be00700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f547c06ad20 0x7f547c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:14.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.035+0000 7f549be00700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f547c06ad20 0x7f547c06d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f548c000f60 tx=0x7f548c009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:14.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.039+0000 7f5493e00700 1 -- 172.21.15.89:0/1065622235 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5488055ea0 con 0x7f549c0d2710 2024-04-02T09:12:14.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:14.245+0000 7f54a2a00700 1 -- 172.21.15.89:0/1065622235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f549c0ca0e0 con 0x7f549c0d2710 2024-04-02T09:12:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.212+0000 7f5493e00700 1 -- 172.21.15.89:0/1065622235 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v214) v1 ==== 169+0+0 (secure 0 0 0) 0x7f548802b0d0 con 0x7f549c0d2710 2024-04-02T09:12:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.216+0000 7f54a2a00700 1 -- 172.21.15.89:0/1065622235 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f547c06ad20 msgr2=0x7f547c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.216+0000 7f54a2a00700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f547c06ad20 0x7f547c06d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f548c000f60 tx=0x7f548c009040 comp rx=0 tx=0).stop 2024-04-02T09:12:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.216+0000 7f54a2a00700 1 -- 172.21.15.89:0/1065622235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f549c0d2710 msgr2=0x7f549c1602a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.216+0000 7f54a2a00700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f549c0d2710 0x7f549c1602a0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f54880097c0 tx=0x7f5488004750 comp rx=0 tx=0).stop 2024-04-02T09:12:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.217+0000 7f54a2a00700 1 -- 172.21.15.89:0/1065622235 shutdown_connections 2024-04-02T09:12:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.217+0000 7f54a2a00700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f547c06ad20 0x7f547c06d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.217+0000 7f54a2a00700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0ced30 0x7f549c1563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.217+0000 7f54a2a00700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f549c15ddf0 0x7f549c1607e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.217+0000 7f54a2a00700 1 --2- 172.21.15.89:0/1065622235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f549c0d2710 0x7f549c1602a0 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.217+0000 7f54a2a00700 1 -- 172.21.15.89:0/1065622235 >> 172.21.15.89:0/1065622235 conn(0x7f549c08aff0 msgr2=0x7f549c0b0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.217+0000 7f54a2a00700 1 -- 172.21.15.89:0/1065622235 shutdown_connections 2024-04-02T09:12:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.217+0000 7f54a2a00700 1 -- 172.21.15.89:0/1065622235 wait complete. 2024-04-02T09:12:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T09:12:15.236 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T09:12:15.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.434+0000 7f56adc00700 1 Processor -- start 2024-04-02T09:12:15.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.434+0000 7f56adc00700 1 -- start start 2024-04-02T09:12:15.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.435+0000 7f56adc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f56a8160e50 0x7f56a80b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:15.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.435+0000 7f56adc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f56a80b3f80 0x7f56a80b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:15.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.436+0000 7f56adc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a80cef00 con 0x7f56a80d3f20 2024-04-02T09:12:15.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.436+0000 7f56adc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a8161230 con 0x7f56a80b3f80 2024-04-02T09:12:15.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.436+0000 7f56adc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a80d2a80 con 0x7f56a8160e50 2024-04-02T09:12:15.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.436+0000 7f56a7400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f56a80b3f80 0x7f56a80b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:15.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.436+0000 7f56a6000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f56a80d3f20 0x7f56a80b2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57340/0 (socket says 172.21.15.89:57340) 2024-04-02T09:12:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.436+0000 7f56a7400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f56a80b3f80 0x7f56a80b53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59370/0 (socket says 172.21.15.89:59370) 2024-04-02T09:12:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.436+0000 7f56a6000700 1 -- 172.21.15.89:0/101821443 learned_addr learned my addr 172.21.15.89:0/101821443 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.436+0000 7f56a6a00700 1 --2- 172.21.15.89:0/101821443 >> v2:172.21.15.113:3300/0 conn(0x7f56a8160e50 0x7f56a80b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.436+0000 7f56a7400700 1 -- 172.21.15.89:0/101821443 >> v2:172.21.15.113:3300/0 conn(0x7f56a8160e50 msgr2=0x7f56a80b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.436+0000 7f56a7400700 1 --2- 172.21.15.89:0/101821443 >> v2:172.21.15.113:3300/0 conn(0x7f56a8160e50 0x7f56a80b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.436+0000 7f56a7400700 1 -- 172.21.15.89:0/101821443 >> v1:172.21.15.89:6789/0 conn(0x7f56a80d3f20 legacy=0x7f56a80b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.436+0000 7f56a7400700 1 -- 172.21.15.89:0/101821443 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a80d43b0 con 0x7f56a80b3f80 2024-04-02T09:12:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.436+0000 7f56a6a00700 1 --2- 172.21.15.89:0/101821443 >> v2:172.21.15.113:3300/0 conn(0x7f56a8160e50 0x7f56a80b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.436+0000 7f56a5600700 1 -- 172.21.15.89:0/101821443 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396658355 0 0) 0x7f56a80cef00 con 0x7f56a80d3f20 2024-04-02T09:12:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.437+0000 7f56a7400700 1 --2- 172.21.15.89:0/101821443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a80b3f80 0x7f56a80b53d0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f56a80cff10 tx=0x7f56a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220e37669e761d5c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.437+0000 7f56a5600700 1 -- 172.21.15.89:0/101821443 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a000e070 con 0x7f56a80b3f80 2024-04-02T09:12:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.437+0000 7f56a5600700 1 -- 172.21.15.89:0/101821443 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a000bbd0 con 0x7f56a80b3f80 2024-04-02T09:12:15.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.437+0000 7f56a5600700 1 -- 172.21.15.89:0/101821443 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a0012430 con 0x7f56a80b3f80 2024-04-02T09:12:15.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.437+0000 7f56adc00700 1 -- 172.21.15.89:0/101821443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a80b3f80 msgr2=0x7f56a80b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:15.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.437+0000 7f56adc00700 1 --2- 172.21.15.89:0/101821443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a80b3f80 0x7f56a80b53d0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f56a80cff10 tx=0x7f56a000b330 comp rx=0 tx=0).stop 2024-04-02T09:12:15.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.437+0000 7f56adc00700 1 -- 172.21.15.89:0/101821443 shutdown_connections 2024-04-02T09:12:15.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.437+0000 7f56adc00700 1 --2- 172.21.15.89:0/101821443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a80b3f80 0x7f56a80b53d0 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.437+0000 7f56adc00700 1 --2- 172.21.15.89:0/101821443 >> v2:172.21.15.113:3300/0 conn(0x7f56a8160e50 0x7f56a80b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.437+0000 7f56adc00700 1 -- 172.21.15.89:0/101821443 >> 172.21.15.89:0/101821443 conn(0x7f56a808aff0 msgr2=0x7f56a80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:15.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.437+0000 7f56adc00700 1 -- 172.21.15.89:0/101821443 shutdown_connections 2024-04-02T09:12:15.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.437+0000 7f56adc00700 1 -- 172.21.15.89:0/101821443 wait complete. 2024-04-02T09:12:15.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.440+0000 7f56adc00700 1 Processor -- start 2024-04-02T09:12:15.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.440+0000 7f56adc00700 1 -- start start 2024-04-02T09:12:15.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.441+0000 7f56adc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a80b3f80 0x7f56a815cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:15.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.441+0000 7f56adc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f56a8160e50 0x7f56a80c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:15.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.441+0000 7f56adc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56a80c9b60 0x7f56a80b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:15.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.441+0000 7f56adc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a80b9240 con 0x7f56a80b3f80 2024-04-02T09:12:15.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.441+0000 7f56adc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a80c9200 con 0x7f56a80c9b60 2024-04-02T09:12:15.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.441+0000 7f56adc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a80c85c0 con 0x7f56a8160e50 2024-04-02T09:12:15.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.441+0000 7f56a6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a80b3f80 0x7f56a815cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:15.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.441+0000 7f56a6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a80b3f80 0x7f56a815cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59372/0 (socket says 172.21.15.89:59372) 2024-04-02T09:12:15.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.441+0000 7f56a6a00700 1 -- 172.21.15.89:0/3148966085 learned_addr learned my addr 172.21.15.89:0/3148966085 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:15.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.442+0000 7f56a7400700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56a80c9b60 0x7f56a80b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:15.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.442+0000 7f56a6000700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f56a8160e50 0x7f56a80c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:15.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.442+0000 7f56a6a00700 1 -- 172.21.15.89:0/3148966085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f56a8160e50 msgr2=0x7f56a80c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:15.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.442+0000 7f56a6a00700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f56a8160e50 0x7f56a80c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.442+0000 7f56a6a00700 1 -- 172.21.15.89:0/3148966085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56a80c9b60 msgr2=0x7f56a80b6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:15.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.442+0000 7f56a6a00700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56a80c9b60 0x7f56a80b6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.442+0000 7f56a6a00700 1 -- 172.21.15.89:0/3148966085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a0009a70 con 0x7f56a80b3f80 2024-04-02T09:12:15.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.442+0000 7f56a6000700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f56a8160e50 0x7f56a80c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:15.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.442+0000 7f56a6a00700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a80b3f80 0x7f56a815cb60 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f569800c750 tx=0x7f569800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:15.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.442+0000 7f569ea00700 1 -- 172.21.15.89:0/3148966085 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5698011800 con 0x7f56a80b3f80 2024-04-02T09:12:15.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.442+0000 7f569ea00700 1 -- 172.21.15.89:0/3148966085 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5698011960 con 0x7f56a80b3f80 2024-04-02T09:12:15.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.442+0000 7f56adc00700 1 -- 172.21.15.89:0/3148966085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a80c8860 con 0x7f56a80b3f80 2024-04-02T09:12:15.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.442+0000 7f569ea00700 1 -- 172.21.15.89:0/3148966085 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5698011cd0 con 0x7f56a80b3f80 2024-04-02T09:12:15.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.442+0000 7f56adc00700 1 -- 172.21.15.89:0/3148966085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a80c8e20 con 0x7f56a80b3f80 2024-04-02T09:12:15.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.444+0000 7f56adc00700 1 -- 172.21.15.89:0/3148966085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a8067720 con 0x7f56a80b3f80 2024-04-02T09:12:15.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.444+0000 7f569ea00700 1 -- 172.21.15.89:0/3148966085 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 145) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f5698021350 con 0x7f56a80b3f80 2024-04-02T09:12:15.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.445+0000 7f569ea00700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f568806ad20 0x7f568806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:15.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.445+0000 7f569ea00700 1 -- 172.21.15.89:0/3148966085 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5993+0+0 (secure 0 0 0) 0x7f5698089500 con 0x7f56a80b3f80 2024-04-02T09:12:15.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.446+0000 7f56a6000700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f568806ad20 0x7f568806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:15.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.446+0000 7f56a6000700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f568806ad20 0x7f568806d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f56a81f6230 tx=0x7f5694009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:15.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.450+0000 7f569ea00700 1 -- 172.21.15.89:0/3148966085 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f56980544c0 con 0x7f56a80b3f80 2024-04-02T09:12:15.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.651+0000 7f56adc00700 1 -- 172.21.15.89:0/3148966085 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f56a80d2fc0 con 0x7f56a80b3f80 2024-04-02T09:12:15.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.656+0000 7f569ea00700 1 -- 172.21.15.89:0/3148966085 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v73) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5698057ae0 con 0x7f56a80b3f80 2024-04-02T09:12:15.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.658+0000 7f56adc00700 1 -- 172.21.15.89:0/3148966085 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f568806ad20 msgr2=0x7f568806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:15.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.658+0000 7f56adc00700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f568806ad20 0x7f568806d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f56a81f6230 tx=0x7f5694009040 comp rx=0 tx=0).stop 2024-04-02T09:12:15.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.658+0000 7f56adc00700 1 -- 172.21.15.89:0/3148966085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a80b3f80 msgr2=0x7f56a815cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:15.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.658+0000 7f56adc00700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a80b3f80 0x7f56a815cb60 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f569800c750 tx=0x7f569800cb10 comp rx=0 tx=0).stop 2024-04-02T09:12:15.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.658+0000 7f56adc00700 1 -- 172.21.15.89:0/3148966085 shutdown_connections 2024-04-02T09:12:15.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.658+0000 7f56adc00700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f568806ad20 0x7f568806d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.658+0000 7f56adc00700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56a80c9b60 0x7f56a80b6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.658+0000 7f56adc00700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a80b3f80 0x7f56a815cb60 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.658+0000 7f56adc00700 1 --2- 172.21.15.89:0/3148966085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f56a8160e50 0x7f56a80c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.658+0000 7f56adc00700 1 -- 172.21.15.89:0/3148966085 >> 172.21.15.89:0/3148966085 conn(0x7f56a808aff0 msgr2=0x7f56a815f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:15.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.659+0000 7f56adc00700 1 -- 172.21.15.89:0/3148966085 shutdown_connections 2024-04-02T09:12:15.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.659+0000 7f56adc00700 1 -- 172.21.15.89:0/3148966085 wait complete. 2024-04-02T09:12:15.675 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T09:12:15.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.880+0000 7fc01f000700 1 Processor -- start 2024-04-02T09:12:15.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.881+0000 7fc01f000700 1 -- start start 2024-04-02T09:12:15.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc01f000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc0181611b0 0x7fc01815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:15.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc01f000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc0180cee10 0x7fc018161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:15.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc01f000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0180ce270 con 0x7fc01815f4d0 2024-04-02T09:12:15.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc01f000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc01815b600 con 0x7fc0181611b0 2024-04-02T09:12:15.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc01f000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0180d2760 con 0x7fc0180cee10 2024-04-02T09:12:15.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc017e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc0181611b0 0x7fc01815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:15.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc017400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc01815f4d0 0x7fc0180ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57348/0 (socket says 172.21.15.89:57348) 2024-04-02T09:12:15.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc017e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc0181611b0 0x7fc01815ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59374/0 (socket says 172.21.15.89:59374) 2024-04-02T09:12:15.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc017400700 1 -- 172.21.15.89:0/3544855380 learned_addr learned my addr 172.21.15.89:0/3544855380 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:15.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc01d000700 1 --2- 172.21.15.89:0/3544855380 >> v2:172.21.15.113:3300/0 conn(0x7fc0180cee10 0x7fc018161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:15.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc017e00700 1 -- 172.21.15.89:0/3544855380 >> v2:172.21.15.113:3300/0 conn(0x7fc0180cee10 msgr2=0x7fc018161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:15.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc017e00700 1 --2- 172.21.15.89:0/3544855380 >> v2:172.21.15.113:3300/0 conn(0x7fc0180cee10 0x7fc018161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc017e00700 1 -- 172.21.15.89:0/3544855380 >> v1:172.21.15.89:6789/0 conn(0x7fc01815f4d0 legacy=0x7fc0180ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:15.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc017e00700 1 -- 172.21.15.89:0/3544855380 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0180d2a40 con 0x7fc0181611b0 2024-04-02T09:12:15.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc01d000700 1 --2- 172.21.15.89:0/3544855380 >> v2:172.21.15.113:3300/0 conn(0x7fc0180cee10 0x7fc018161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:15.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.882+0000 7fc016a00700 1 -- 172.21.15.89:0/3544855380 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291699173 0 0) 0x7fc0180ce270 con 0x7fc01815f4d0 2024-04-02T09:12:15.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.883+0000 7fc017e00700 1 --2- 172.21.15.89:0/3544855380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0181611b0 0x7fc01815ef90 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fc0180d2f60 tx=0x7fc008009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79657535825006bb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:15.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.883+0000 7fc016a00700 1 -- 172.21.15.89:0/3544855380 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc008011830 con 0x7fc0181611b0 2024-04-02T09:12:15.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.883+0000 7fc016a00700 1 -- 172.21.15.89:0/3544855380 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc008011990 con 0x7fc0181611b0 2024-04-02T09:12:15.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.883+0000 7fc016a00700 1 -- 172.21.15.89:0/3544855380 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc008011d20 con 0x7fc0181611b0 2024-04-02T09:12:15.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.883+0000 7fc01f000700 1 -- 172.21.15.89:0/3544855380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0181611b0 msgr2=0x7fc01815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:15.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.883+0000 7fc01f000700 1 --2- 172.21.15.89:0/3544855380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0181611b0 0x7fc01815ef90 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fc0180d2f60 tx=0x7fc008009b20 comp rx=0 tx=0).stop 2024-04-02T09:12:15.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.884+0000 7fc01f000700 1 -- 172.21.15.89:0/3544855380 shutdown_connections 2024-04-02T09:12:15.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.884+0000 7fc01f000700 1 --2- 172.21.15.89:0/3544855380 >> v2:172.21.15.113:3300/0 conn(0x7fc0180cee10 0x7fc018161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.884+0000 7fc01f000700 1 --2- 172.21.15.89:0/3544855380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0181611b0 0x7fc01815ef90 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.884+0000 7fc01f000700 1 -- 172.21.15.89:0/3544855380 >> 172.21.15.89:0/3544855380 conn(0x7fc01808aff0 msgr2=0x7fc0180d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:15.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.884+0000 7fc01f000700 1 -- 172.21.15.89:0/3544855380 shutdown_connections 2024-04-02T09:12:15.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.884+0000 7fc01f000700 1 -- 172.21.15.89:0/3544855380 wait complete. 2024-04-02T09:12:15.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.886+0000 7fc01f000700 1 Processor -- start 2024-04-02T09:12:15.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.886+0000 7fc01f000700 1 -- start start 2024-04-02T09:12:15.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.887+0000 7fc01f000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0180cee10 0x7fc0181531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:15.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.887+0000 7fc01f000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc01815f4d0 0x7fc0180c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:15.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.887+0000 7fc01f000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0181611b0 0x7fc0180b6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:15.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.887+0000 7fc01f000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180c94d0 con 0x7fc0181611b0 2024-04-02T09:12:15.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.887+0000 7fc01f000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc018153700 con 0x7fc01815f4d0 2024-04-02T09:12:15.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.887+0000 7fc01f000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180c88c0 con 0x7fc0180cee10 2024-04-02T09:12:15.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.888+0000 7fc01d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0181611b0 0x7fc0180b6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:15.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.888+0000 7fc01d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0181611b0 0x7fc0180b6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59384/0 (socket says 172.21.15.89:59384) 2024-04-02T09:12:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.888+0000 7fc01d000700 1 -- 172.21.15.89:0/537858100 learned_addr learned my addr 172.21.15.89:0/537858100 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.888+0000 7fc017400700 1 --2- 172.21.15.89:0/537858100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc01815f4d0 0x7fc0180c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.888+0000 7fc017e00700 1 --2- 172.21.15.89:0/537858100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0180cee10 0x7fc0181531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.888+0000 7fc01d000700 1 -- 172.21.15.89:0/537858100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0180cee10 msgr2=0x7fc0181531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.888+0000 7fc01d000700 1 --2- 172.21.15.89:0/537858100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0180cee10 0x7fc0181531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.888+0000 7fc01d000700 1 -- 172.21.15.89:0/537858100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc01815f4d0 msgr2=0x7fc0180c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.888+0000 7fc01d000700 1 --2- 172.21.15.89:0/537858100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc01815f4d0 0x7fc0180c9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.888+0000 7fc01d000700 1 -- 172.21.15.89:0/537858100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0080094a0 con 0x7fc0181611b0 2024-04-02T09:12:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.888+0000 7fc01d000700 1 --2- 172.21.15.89:0/537858100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0181611b0 0x7fc0180b6c40 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fc010007ef0 tx=0x7fc01000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.889+0000 7fc00fe00700 1 -- 172.21.15.89:0/537858100 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01000dd80 con 0x7fc0181611b0 2024-04-02T09:12:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.889+0000 7fc00fe00700 1 -- 172.21.15.89:0/537858100 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01000f040 con 0x7fc0181611b0 2024-04-02T09:12:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.889+0000 7fc00fe00700 1 -- 172.21.15.89:0/537858100 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc010003c60 con 0x7fc0181611b0 2024-04-02T09:12:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.889+0000 7fc01f000700 1 -- 172.21.15.89:0/537858100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0180c8ba0 con 0x7fc0181611b0 2024-04-02T09:12:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.889+0000 7fc01f000700 1 -- 172.21.15.89:0/537858100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0180c9160 con 0x7fc0181611b0 2024-04-02T09:12:15.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.890+0000 7fc01f000700 1 -- 172.21.15.89:0/537858100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0180c9640 con 0x7fc0181611b0 2024-04-02T09:12:15.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.891+0000 7fc00fe00700 1 -- 172.21.15.89:0/537858100 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc010004030 con 0x7fc0181611b0 2024-04-02T09:12:15.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.891+0000 7fc00fe00700 1 --2- 172.21.15.89:0/537858100 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbff806adb0 0x7fbff806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:15.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.891+0000 7fc00fe00700 1 -- 172.21.15.89:0/537858100 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5993+0+0 (secure 0 0 0) 0x7fc0100897f0 con 0x7fc0181611b0 2024-04-02T09:12:15.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.892+0000 7fc017e00700 1 --2- 172.21.15.89:0/537858100 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbff806adb0 0x7fbff806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:15.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.892+0000 7fc017e00700 1 --2- 172.21.15.89:0/537858100 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbff806adb0 0x7fbff806d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc0080090b0 tx=0x7fc00801c040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:15.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:15.896+0000 7fc00fe00700 1 -- 172.21.15.89:0/537858100 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc0100545f0 con 0x7fc0181611b0 2024-04-02T09:12:16.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.097+0000 7fc01f000700 1 -- 172.21.15.89:0/537858100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fc018067720 con 0x7fc0181611b0 2024-04-02T09:12:16.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.102+0000 7fc00fe00700 1 -- 172.21.15.89:0/537858100 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v74) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc010057c10 con 0x7fc0181611b0 2024-04-02T09:12:16.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.104+0000 7fc01f000700 1 -- 172.21.15.89:0/537858100 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbff806adb0 msgr2=0x7fbff806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:16.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.104+0000 7fc01f000700 1 --2- 172.21.15.89:0/537858100 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbff806adb0 0x7fbff806d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc0080090b0 tx=0x7fc00801c040 comp rx=0 tx=0).stop 2024-04-02T09:12:16.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.104+0000 7fc01f000700 1 -- 172.21.15.89:0/537858100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0181611b0 msgr2=0x7fc0180b6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:16.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.104+0000 7fc01f000700 1 --2- 172.21.15.89:0/537858100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0181611b0 0x7fc0180b6c40 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fc010007ef0 tx=0x7fc01000d3b0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.105+0000 7fc01f000700 1 -- 172.21.15.89:0/537858100 shutdown_connections 2024-04-02T09:12:16.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.105+0000 7fc01f000700 1 --2- 172.21.15.89:0/537858100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0181611b0 0x7fc0180b6c40 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.105+0000 7fc01f000700 1 --2- 172.21.15.89:0/537858100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0180cee10 0x7fc0181531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.105+0000 7fc01f000700 1 --2- 172.21.15.89:0/537858100 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbff806adb0 0x7fbff806d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.105+0000 7fc01f000700 1 --2- 172.21.15.89:0/537858100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc01815f4d0 0x7fc0180c9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.105+0000 7fc01f000700 1 -- 172.21.15.89:0/537858100 >> 172.21.15.89:0/537858100 conn(0x7fc01808aff0 msgr2=0x7fc01806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:16.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.105+0000 7fc01f000700 1 -- 172.21.15.89:0/537858100 shutdown_connections 2024-04-02T09:12:16.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.105+0000 7fc01f000700 1 -- 172.21.15.89:0/537858100 wait complete. 2024-04-02T09:12:16.121 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T09:12:16.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.316+0000 7fc8d6400700 1 Processor -- start 2024-04-02T09:12:16.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.316+0000 7fc8d6400700 1 -- start start 2024-04-02T09:12:16.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.317+0000 7fc8d6400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8d01611b0 0x7fc8d015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:16.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.317+0000 7fc8d6400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc8d015f4d0 0x7fc8d00ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:16.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.317+0000 7fc8d6400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d00ce270 con 0x7fc8d00d21f0 2024-04-02T09:12:16.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.317+0000 7fc8d6400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d015b600 con 0x7fc8d01611b0 2024-04-02T09:12:16.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.317+0000 7fc8d6400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d00d2760 con 0x7fc8d015f4d0 2024-04-02T09:12:16.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.317+0000 7fc8cf400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8d01611b0 0x7fc8d015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:16.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.318+0000 7fc8cf400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc8d01611b0 0x7fc8d015ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59388/0 (socket says 172.21.15.89:59388) 2024-04-02T09:12:16.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.318+0000 7fc8cf400700 1 -- 172.21.15.89:0/4021686843 learned_addr learned my addr 172.21.15.89:0/4021686843 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:16.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.318+0000 7fc8cea00700 1 --2- 172.21.15.89:0/4021686843 >> v2:172.21.15.113:3300/0 conn(0x7fc8d015f4d0 0x7fc8d00ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:16.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.318+0000 7fc8cf400700 1 -- 172.21.15.89:0/4021686843 >> v2:172.21.15.113:3300/0 conn(0x7fc8d015f4d0 msgr2=0x7fc8d00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:16.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.318+0000 7fc8cf400700 1 --2- 172.21.15.89:0/4021686843 >> v2:172.21.15.113:3300/0 conn(0x7fc8d015f4d0 0x7fc8d00ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.318+0000 7fc8cf400700 1 -- 172.21.15.89:0/4021686843 >> v1:172.21.15.113:6789/0 conn(0x7fc8d00d21f0 legacy=0x7fc8d0161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:16.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.318+0000 7fc8cf400700 1 -- 172.21.15.89:0/4021686843 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d00d2a40 con 0x7fc8d01611b0 2024-04-02T09:12:16.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.318+0000 7fc8cea00700 1 --2- 172.21.15.89:0/4021686843 >> v2:172.21.15.113:3300/0 conn(0x7fc8d015f4d0 0x7fc8d00ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:16.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.318+0000 7fc8cf400700 1 --2- 172.21.15.89:0/4021686843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d01611b0 0x7fc8d015ef90 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fc8d008c030 tx=0x7fc8c0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9764314a8831c5de server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:16.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.318+0000 7fc8ce000700 1 -- 172.21.15.89:0/4021686843 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0004230 con 0x7fc8d01611b0 2024-04-02T09:12:16.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.318+0000 7fc8ce000700 1 -- 172.21.15.89:0/4021686843 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c0004390 con 0x7fc8d01611b0 2024-04-02T09:12:16.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.318+0000 7fc8ce000700 1 -- 172.21.15.89:0/4021686843 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c00036a0 con 0x7fc8d01611b0 2024-04-02T09:12:16.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.319+0000 7fc8d6400700 1 -- 172.21.15.89:0/4021686843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d01611b0 msgr2=0x7fc8d015ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:16.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.319+0000 7fc8d6400700 1 --2- 172.21.15.89:0/4021686843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d01611b0 0x7fc8d015ef90 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fc8d008c030 tx=0x7fc8c0009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.319+0000 7fc8d6400700 1 -- 172.21.15.89:0/4021686843 shutdown_connections 2024-04-02T09:12:16.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.319+0000 7fc8d6400700 1 --2- 172.21.15.89:0/4021686843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d01611b0 0x7fc8d015ef90 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.319+0000 7fc8d6400700 1 --2- 172.21.15.89:0/4021686843 >> v2:172.21.15.113:3300/0 conn(0x7fc8d015f4d0 0x7fc8d00ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.319+0000 7fc8d6400700 1 -- 172.21.15.89:0/4021686843 >> 172.21.15.89:0/4021686843 conn(0x7fc8d008aff0 msgr2=0x7fc8d00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:16.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.319+0000 7fc8d6400700 1 -- 172.21.15.89:0/4021686843 shutdown_connections 2024-04-02T09:12:16.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.319+0000 7fc8d6400700 1 -- 172.21.15.89:0/4021686843 wait complete. 2024-04-02T09:12:16.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.321+0000 7fc8d6400700 1 Processor -- start 2024-04-02T09:12:16.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.321+0000 7fc8d6400700 1 -- start start 2024-04-02T09:12:16.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8d6400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8d00d21f0 0x7fc8d0156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:16.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8d6400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d015f4d0 0x7fc8d015c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:16.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8d6400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d01611b0 0x7fc8d015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:16.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8d6400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00b8f90 con 0x7fc8d015f4d0 2024-04-02T09:12:16.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8d6400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d01f7800 con 0x7fc8d00d21f0 2024-04-02T09:12:16.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8d6400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d01f7520 con 0x7fc8d01611b0 2024-04-02T09:12:16.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8cea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d015f4d0 0x7fc8d015c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:16.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8cea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d015f4d0 0x7fc8d015c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59398/0 (socket says 172.21.15.89:59398) 2024-04-02T09:12:16.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8cea00700 1 -- 172.21.15.89:0/513053337 learned_addr learned my addr 172.21.15.89:0/513053337 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:16.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8cf400700 1 --2- 172.21.15.89:0/513053337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8d00d21f0 0x7fc8d0156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:16.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8cfe00700 1 --2- 172.21.15.89:0/513053337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d01611b0 0x7fc8d015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:16.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8cea00700 1 -- 172.21.15.89:0/513053337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d01611b0 msgr2=0x7fc8d015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:16.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8cea00700 1 --2- 172.21.15.89:0/513053337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d01611b0 0x7fc8d015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8cea00700 1 -- 172.21.15.89:0/513053337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8d00d21f0 msgr2=0x7fc8d0156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:16.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8cea00700 1 --2- 172.21.15.89:0/513053337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8d00d21f0 0x7fc8d0156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.322+0000 7fc8cea00700 1 -- 172.21.15.89:0/513053337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c0009550 con 0x7fc8d015f4d0 2024-04-02T09:12:16.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.323+0000 7fc8cea00700 1 --2- 172.21.15.89:0/513053337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d015f4d0 0x7fc8d015c830 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc00c780 tx=0x7fc8bc00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:16.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.323+0000 7fc8c7400700 1 -- 172.21.15.89:0/513053337 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc011800 con 0x7fc8d015f4d0 2024-04-02T09:12:16.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.323+0000 7fc8c7400700 1 -- 172.21.15.89:0/513053337 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8bc011960 con 0x7fc8d015f4d0 2024-04-02T09:12:16.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.323+0000 7fc8d6400700 1 -- 172.21.15.89:0/513053337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d01e81e0 con 0x7fc8d015f4d0 2024-04-02T09:12:16.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.323+0000 7fc8c7400700 1 -- 172.21.15.89:0/513053337 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc011d10 con 0x7fc8d015f4d0 2024-04-02T09:12:16.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.323+0000 7fc8d6400700 1 -- 172.21.15.89:0/513053337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d01e87a0 con 0x7fc8d015f4d0 2024-04-02T09:12:16.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.324+0000 7fc8d6400700 1 -- 172.21.15.89:0/513053337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d01e8b10 con 0x7fc8d015f4d0 2024-04-02T09:12:16.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.325+0000 7fc8c7400700 1 -- 172.21.15.89:0/513053337 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc8bc018360 con 0x7fc8d015f4d0 2024-04-02T09:12:16.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.326+0000 7fc8c7400700 1 --2- 172.21.15.89:0/513053337 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc8b006adb0 0x7fc8b006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:16.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.326+0000 7fc8c7400700 1 -- 172.21.15.89:0/513053337 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5993+0+0 (secure 0 0 0) 0x7fc8bc089290 con 0x7fc8d015f4d0 2024-04-02T09:12:16.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.326+0000 7fc8cf400700 1 --2- 172.21.15.89:0/513053337 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc8b006adb0 0x7fc8b006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:16.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.327+0000 7fc8cf400700 1 --2- 172.21.15.89:0/513053337 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc8b006adb0 0x7fc8b006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0009130 tx=0x7fc8c0008030 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:16.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.330+0000 7fc8c7400700 1 -- 172.21.15.89:0/513053337 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc8bc054120 con 0x7fc8d015f4d0 2024-04-02T09:12:16.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.530+0000 7fc8d6400700 1 -- 172.21.15.89:0/513053337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc8d0067720 con 0x7fc8d015f4d0 2024-04-02T09:12:16.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.534+0000 7fc8c7400700 1 -- 172.21.15.89:0/513053337 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v75) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc8bc057740 con 0x7fc8d015f4d0 2024-04-02T09:12:16.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.537+0000 7fc8d6400700 1 -- 172.21.15.89:0/513053337 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc8b006adb0 msgr2=0x7fc8b006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:16.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.537+0000 7fc8d6400700 1 --2- 172.21.15.89:0/513053337 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc8b006adb0 0x7fc8b006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0009130 tx=0x7fc8c0008030 comp rx=0 tx=0).stop 2024-04-02T09:12:16.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.537+0000 7fc8d6400700 1 -- 172.21.15.89:0/513053337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d015f4d0 msgr2=0x7fc8d015c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:16.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.537+0000 7fc8d6400700 1 --2- 172.21.15.89:0/513053337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d015f4d0 0x7fc8d015c830 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc00c780 tx=0x7fc8bc00ca90 comp rx=0 tx=0).stop 2024-04-02T09:12:16.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.537+0000 7fc8d6400700 1 -- 172.21.15.89:0/513053337 shutdown_connections 2024-04-02T09:12:16.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.537+0000 7fc8d6400700 1 --2- 172.21.15.89:0/513053337 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc8b006adb0 0x7fc8b006d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.537+0000 7fc8d6400700 1 --2- 172.21.15.89:0/513053337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8d00d21f0 0x7fc8d0156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.537+0000 7fc8d6400700 1 --2- 172.21.15.89:0/513053337 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8d01611b0 0x7fc8d015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.537+0000 7fc8d6400700 1 --2- 172.21.15.89:0/513053337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8d015f4d0 0x7fc8d015c830 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.537+0000 7fc8d6400700 1 -- 172.21.15.89:0/513053337 >> 172.21.15.89:0/513053337 conn(0x7fc8d008aff0 msgr2=0x7fc8d0159950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:16.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.537+0000 7fc8d6400700 1 -- 172.21.15.89:0/513053337 shutdown_connections 2024-04-02T09:12:16.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.537+0000 7fc8d6400700 1 -- 172.21.15.89:0/513053337 wait complete. 2024-04-02T09:12:16.553 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-04-02T09:12:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.746+0000 7f2074c00700 1 Processor -- start 2024-04-02T09:12:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.746+0000 7f2074c00700 1 -- start start 2024-04-02T09:12:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.747+0000 7f2074c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f20700ceec0 0x7f20701612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.747+0000 7f2074c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2070161c50 con 0x7f207015d640 2024-04-02T09:12:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.747+0000 7f2074c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20700d30d0 con 0x7f2070161800 2024-04-02T09:12:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.747+0000 7f2074c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20700b6840 con 0x7f20700ceec0 2024-04-02T09:12:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.747+0000 7f206de00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f207015d640 0x7f20700d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57350/0 (socket says 172.21.15.89:57350) 2024-04-02T09:12:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.747+0000 7f206de00700 1 -- 172.21.15.89:0/1770930822 learned_addr learned my addr 172.21.15.89:0/1770930822 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206d400700 1 --2- 172.21.15.89:0/1770930822 >> v2:172.21.15.113:3300/0 conn(0x7f20700ceec0 0x7f20701612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 -- 172.21.15.89:0/1770930822 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605623493 0 0) 0x7f2060001110 con 0x7f207015d640 2024-04-02T09:12:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 -- 172.21.15.89:0/1770930822 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20540035b0 con 0x7f207015d640 2024-04-02T09:12:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 -- 172.21.15.89:0/1770930822 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4282029988 0 0) 0x7f20540035b0 con 0x7f207015d640 2024-04-02T09:12:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 -- 172.21.15.89:0/1770930822 >> v2:172.21.15.113:3300/0 conn(0x7f20700ceec0 msgr2=0x7f20701612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 --2- 172.21.15.89:0/1770930822 >> v2:172.21.15.113:3300/0 conn(0x7f20700ceec0 0x7f20701612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 -- 172.21.15.89:0/1770930822 >> v1:172.21.15.113:6789/0 conn(0x7f2070161800 legacy=0x7f20700b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 -- 172.21.15.89:0/1770930822 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20700d2b30 con 0x7f207015d640 2024-04-02T09:12:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 -- 172.21.15.89:0/1770930822 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f2060001800 con 0x7f207015d640 2024-04-02T09:12:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 -- 172.21.15.89:0/1770930822 >> v1:172.21.15.89:6789/0 conn(0x7f207015d640 legacy=0x7f20700d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 --2- 172.21.15.89:0/1770930822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2054003a00 0x7f2054003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 --2- 172.21.15.89:0/1770930822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2054004400 0x7f20540048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 --2- 172.21.15.89:0/1770930822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2054005170 0x7f205400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 -- 172.21.15.89:0/1770930822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20700b6840 con 0x7f2054004400 2024-04-02T09:12:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 -- 172.21.15.89:0/1770930822 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205400c0b0 con 0x7f2054005170 2024-04-02T09:12:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206ca00700 1 -- 172.21.15.89:0/1770930822 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205400c270 con 0x7f2054003a00 2024-04-02T09:12:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.748+0000 7f206d400700 1 --2- 172.21.15.89:0/1770930822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2054004400 0x7f20540048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.749+0000 7f206de00700 1 --2- 172.21.15.89:0/1770930822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2054003a00 0x7f2054003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.749+0000 7f206e800700 1 --2- 172.21.15.89:0/1770930822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2054005170 0x7f205400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.749+0000 7f206d400700 1 -- 172.21.15.89:0/1770930822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2054003a00 msgr2=0x7f2054003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.749+0000 7f206d400700 1 --2- 172.21.15.89:0/1770930822 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2054003a00 0x7f2054003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.749+0000 7f206d400700 1 -- 172.21.15.89:0/1770930822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2054005170 msgr2=0x7f205400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.749+0000 7f206d400700 1 --2- 172.21.15.89:0/1770930822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2054005170 0x7f205400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.749+0000 7f206d400700 1 -- 172.21.15.89:0/1770930822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20540035b0 con 0x7f2054004400 2024-04-02T09:12:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.749+0000 7f206e800700 1 --2- 172.21.15.89:0/1770930822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2054005170 0x7f205400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.749+0000 7f206e800700 1 -- 172.21.15.89:0/1770930822 reap_dead start 2024-04-02T09:12:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.749+0000 7f206d400700 1 --2- 172.21.15.89:0/1770930822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2054004400 0x7f20540048d0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f205c009b60 tx=0x7f2054005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c0dc8c3c9f1f32 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.749+0000 7f206ca00700 1 -- 172.21.15.89:0/1770930822 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c00e6c0 con 0x7f2054004400 2024-04-02T09:12:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.749+0000 7f206ca00700 1 -- 172.21.15.89:0/1770930822 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205c00e820 con 0x7f2054004400 2024-04-02T09:12:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.749+0000 7f206ca00700 1 -- 172.21.15.89:0/1770930822 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c00eb90 con 0x7f2054004400 2024-04-02T09:12:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.750+0000 7f2074c00700 1 -- 172.21.15.89:0/1770930822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2054004400 msgr2=0x7f20540048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.750+0000 7f2074c00700 1 --2- 172.21.15.89:0/1770930822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2054004400 0x7f20540048d0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f205c009b60 tx=0x7f2054005880 comp rx=0 tx=0).stop 2024-04-02T09:12:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.750+0000 7f2074c00700 1 -- 172.21.15.89:0/1770930822 shutdown_connections 2024-04-02T09:12:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.750+0000 7f2074c00700 1 --2- 172.21.15.89:0/1770930822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2054004400 0x7f20540048d0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.750+0000 7f2074c00700 1 -- 172.21.15.89:0/1770930822 >> 172.21.15.89:0/1770930822 conn(0x7f207008aff0 msgr2=0x7f20700d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.750+0000 7f2074c00700 1 -- 172.21.15.89:0/1770930822 shutdown_connections 2024-04-02T09:12:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.750+0000 7f2074c00700 1 -- 172.21.15.89:0/1770930822 wait complete. 2024-04-02T09:12:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.752+0000 7f2074c00700 1 Processor -- start 2024-04-02T09:12:16.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.752+0000 7f2074c00700 1 -- start start 2024-04-02T09:12:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.753+0000 7f2074c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2054004400 0x7f20701597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.753+0000 7f2074c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20700c8c50 0x7f20700a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.753+0000 7f2074c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f20700a0c40 0x7f20700c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.753+0000 7f2074c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207015a260 con 0x7f2054004400 2024-04-02T09:12:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.753+0000 7f2074c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2070159cf0 con 0x7f20700c8c50 2024-04-02T09:12:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.753+0000 7f2074c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f207009d2d0 con 0x7f20700a0c40 2024-04-02T09:12:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.753+0000 7f206de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2054004400 0x7f20701597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.753+0000 7f206de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2054004400 0x7f20701597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59414/0 (socket says 172.21.15.89:59414) 2024-04-02T09:12:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.753+0000 7f206de00700 1 -- 172.21.15.89:0/186187443 learned_addr learned my addr 172.21.15.89:0/186187443 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.753+0000 7f206d400700 1 --2- 172.21.15.89:0/186187443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20700c8c50 0x7f20700a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.753+0000 7f206e800700 1 --2- 172.21.15.89:0/186187443 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f20700a0c40 0x7f20700c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:16.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.754+0000 7f206de00700 1 -- 172.21.15.89:0/186187443 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f20700a0c40 msgr2=0x7f20700c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.754+0000 7f206de00700 1 --2- 172.21.15.89:0/186187443 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f20700a0c40 0x7f20700c7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.754+0000 7f206de00700 1 -- 172.21.15.89:0/186187443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20700c8c50 msgr2=0x7f20700a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.754+0000 7f206de00700 1 --2- 172.21.15.89:0/186187443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20700c8c50 0x7f20700a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.754+0000 7f206de00700 1 -- 172.21.15.89:0/186187443 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207015f9b0 con 0x7f2054004400 2024-04-02T09:12:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.754+0000 7f206de00700 1 --2- 172.21.15.89:0/186187443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2054004400 0x7f20701597b0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f20600072c0 tx=0x7f20600075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.754+0000 7f2066000700 1 -- 172.21.15.89:0/186187443 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2060012560 con 0x7f2054004400 2024-04-02T09:12:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.754+0000 7f2066000700 1 -- 172.21.15.89:0/186187443 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20600126e0 con 0x7f2054004400 2024-04-02T09:12:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.754+0000 7f2074c00700 1 -- 172.21.15.89:0/186187443 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f207015fc90 con 0x7f2054004400 2024-04-02T09:12:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.754+0000 7f2066000700 1 -- 172.21.15.89:0/186187443 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2060012a90 con 0x7f2054004400 2024-04-02T09:12:16.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.754+0000 7f2074c00700 1 -- 172.21.15.89:0/186187443 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20700a10c0 con 0x7f2054004400 2024-04-02T09:12:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.755+0000 7f2074c00700 1 -- 172.21.15.89:0/186187443 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f207009d7b0 con 0x7f2054004400 2024-04-02T09:12:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.756+0000 7f2066000700 1 -- 172.21.15.89:0/186187443 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2060012bf0 con 0x7f2054004400 2024-04-02T09:12:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.758+0000 7f2066000700 1 --2- 172.21.15.89:0/186187443 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f205006adb0 0x7f205006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.758+0000 7f2066000700 1 -- 172.21.15.89:0/186187443 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5993+0+0 (secure 0 0 0) 0x7f2060089400 con 0x7f2054004400 2024-04-02T09:12:16.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.758+0000 7f206d400700 1 --2- 172.21.15.89:0/186187443 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f205006adb0 0x7f205006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:16.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.759+0000 7f206d400700 1 --2- 172.21.15.89:0/186187443 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f205006adb0 0x7f205006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f205c0090b0 tx=0x7f205c01f040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:16.762+0000 7f2066000700 1 -- 172.21.15.89:0/186187443 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f207009d7b0 con 0x7f2054004400 2024-04-02T09:12:17.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.049+0000 7f2074c00700 1 -- 172.21.15.89:0/186187443 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f2070161c50 con 0x7f2054004400 2024-04-02T09:12:17.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.214+0000 7f2066000700 1 -- 172.21.15.89:0/186187443 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3125) v1 ==== 163+0+0 (secure 0 0 0) 0x7f2070161c50 con 0x7f2054004400 2024-04-02T09:12:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.217+0000 7f2074c00700 1 -- 172.21.15.89:0/186187443 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f205006adb0 msgr2=0x7f205006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.217+0000 7f2074c00700 1 --2- 172.21.15.89:0/186187443 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f205006adb0 0x7f205006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f205c0090b0 tx=0x7f205c01f040 comp rx=0 tx=0).stop 2024-04-02T09:12:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.217+0000 7f2074c00700 1 -- 172.21.15.89:0/186187443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2054004400 msgr2=0x7f20701597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.217+0000 7f2074c00700 1 --2- 172.21.15.89:0/186187443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2054004400 0x7f20701597b0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f20600072c0 tx=0x7f20600075d0 comp rx=0 tx=0).stop 2024-04-02T09:12:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.217+0000 7f2074c00700 1 -- 172.21.15.89:0/186187443 shutdown_connections 2024-04-02T09:12:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.217+0000 7f2074c00700 1 --2- 172.21.15.89:0/186187443 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f20700a0c40 0x7f20700c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.217+0000 7f2074c00700 1 --2- 172.21.15.89:0/186187443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2054004400 0x7f20701597b0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.217+0000 7f2074c00700 1 --2- 172.21.15.89:0/186187443 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f205006adb0 0x7f205006d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.217+0000 7f2074c00700 1 --2- 172.21.15.89:0/186187443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20700c8c50 0x7f20700a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.217+0000 7f2074c00700 1 -- 172.21.15.89:0/186187443 >> 172.21.15.89:0/186187443 conn(0x7f207008aff0 msgr2=0x7f20700d1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.217+0000 7f2074c00700 1 -- 172.21.15.89:0/186187443 shutdown_connections 2024-04-02T09:12:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.217+0000 7f2074c00700 1 -- 172.21.15.89:0/186187443 wait complete. 2024-04-02T09:12:17.233 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2024-04-02T09:12:17.233 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2024-04-02T09:12:17.233 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-04-02T09:12:17.233 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-04-02T09:12:17.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.426+0000 7f8f29400700 1 Processor -- start 2024-04-02T09:12:17.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.426+0000 7f8f29400700 1 -- start start 2024-04-02T09:12:17.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.427+0000 7f8f29400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8f240d05e0 0x7f8f240d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:17.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.427+0000 7f8f29400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8f240b6990 0x7f8f240b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:17.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.427+0000 7f8f29400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f240d0f80 con 0x7f8f2415da50 2024-04-02T09:12:17.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.427+0000 7f8f29400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f240b95d0 con 0x7f8f240b6990 2024-04-02T09:12:17.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.427+0000 7f8f29400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f2415de30 con 0x7f8f240d05e0 2024-04-02T09:12:17.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.427+0000 7f8f23200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8f240b6990 0x7f8f240b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:17.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.427+0000 7f8f23200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8f240b6990 0x7f8f240b6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59422/0 (socket says 172.21.15.89:59422) 2024-04-02T09:12:17.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.427+0000 7f8f23200700 1 -- 172.21.15.89:0/3018525565 learned_addr learned my addr 172.21.15.89:0/3018525565 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:17.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.428+0000 7f8f21e00700 1 --2- 172.21.15.89:0/3018525565 >> v2:172.21.15.113:3300/0 conn(0x7f8f240d05e0 0x7f8f240d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:17.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.428+0000 7f8f23200700 1 -- 172.21.15.89:0/3018525565 >> v2:172.21.15.113:3300/0 conn(0x7f8f240d05e0 msgr2=0x7f8f240d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:17.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.428+0000 7f8f23200700 1 --2- 172.21.15.89:0/3018525565 >> v2:172.21.15.113:3300/0 conn(0x7f8f240d05e0 0x7f8f240d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:17.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.428+0000 7f8f23200700 1 -- 172.21.15.89:0/3018525565 >> v1:172.21.15.113:6789/0 conn(0x7f8f2415da50 legacy=0x7f8f240ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:17.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.428+0000 7f8f23200700 1 -- 172.21.15.89:0/3018525565 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f240d24e0 con 0x7f8f240b6990 2024-04-02T09:12:17.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.428+0000 7f8f23200700 1 --2- 172.21.15.89:0/3018525565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f240b6990 0x7f8f240b6e40 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f8f240740a0 tx=0x7f8f1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5d2bb02c4a01419 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:17.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.428+0000 7f8f21400700 1 -- 172.21.15.89:0/3018525565 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c00e070 con 0x7f8f240b6990 2024-04-02T09:12:17.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.428+0000 7f8f21400700 1 -- 172.21.15.89:0/3018525565 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f1c003a20 con 0x7f8f240b6990 2024-04-02T09:12:17.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.428+0000 7f8f21400700 1 -- 172.21.15.89:0/3018525565 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c003dd0 con 0x7f8f240b6990 2024-04-02T09:12:17.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.429+0000 7f8f29400700 1 -- 172.21.15.89:0/3018525565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f240b6990 msgr2=0x7f8f240b6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:17.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.429+0000 7f8f29400700 1 --2- 172.21.15.89:0/3018525565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f240b6990 0x7f8f240b6e40 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f8f240740a0 tx=0x7f8f1c00b330 comp rx=0 tx=0).stop 2024-04-02T09:12:17.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.429+0000 7f8f29400700 1 -- 172.21.15.89:0/3018525565 shutdown_connections 2024-04-02T09:12:17.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.429+0000 7f8f29400700 1 --2- 172.21.15.89:0/3018525565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f240b6990 0x7f8f240b6e40 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:17.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.429+0000 7f8f29400700 1 --2- 172.21.15.89:0/3018525565 >> v2:172.21.15.113:3300/0 conn(0x7f8f240d05e0 0x7f8f240d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:17.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.429+0000 7f8f29400700 1 -- 172.21.15.89:0/3018525565 >> 172.21.15.89:0/3018525565 conn(0x7f8f240666a0 msgr2=0x7f8f24066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:17.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.429+0000 7f8f29400700 1 -- 172.21.15.89:0/3018525565 shutdown_connections 2024-04-02T09:12:17.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.429+0000 7f8f29400700 1 -- 172.21.15.89:0/3018525565 wait complete. 2024-04-02T09:12:17.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.431+0000 7f8f29400700 1 Processor -- start 2024-04-02T09:12:17.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.431+0000 7f8f29400700 1 -- start start 2024-04-02T09:12:17.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f29400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f240b6990 0x7f8f2414b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:17.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f29400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f240d05e0 0x7f8f2415f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:17.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f29400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f2415da50 0x7f8f2415f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:17.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f29400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f24156d80 con 0x7f8f240b6990 2024-04-02T09:12:17.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f29400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f24155f10 con 0x7f8f240d05e0 2024-04-02T09:12:17.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f29400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f241605c0 con 0x7f8f2415da50 2024-04-02T09:12:17.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f22800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f240b6990 0x7f8f2414b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f22800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f240b6990 0x7f8f2414b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59424/0 (socket says 172.21.15.89:59424) 2024-04-02T09:12:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f22800700 1 -- 172.21.15.89:0/3354665343 learned_addr learned my addr 172.21.15.89:0/3354665343 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f21e00700 1 --2- 172.21.15.89:0/3354665343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f240d05e0 0x7f8f2415f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f23200700 1 --2- 172.21.15.89:0/3354665343 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f2415da50 0x7f8f2415f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f22800700 1 -- 172.21.15.89:0/3354665343 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f2415da50 msgr2=0x7f8f2415f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f22800700 1 --2- 172.21.15.89:0/3354665343 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f2415da50 0x7f8f2415f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f22800700 1 -- 172.21.15.89:0/3354665343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f240d05e0 msgr2=0x7f8f2415f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f22800700 1 --2- 172.21.15.89:0/3354665343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f240d05e0 0x7f8f2415f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.432+0000 7f8f22800700 1 -- 172.21.15.89:0/3354665343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c009b30 con 0x7f8f240b6990 2024-04-02T09:12:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.433+0000 7f8f22800700 1 --2- 172.21.15.89:0/3354665343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f240b6990 0x7f8f2414b150 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f8f1400b4d0 tx=0x7f8f1400b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.433+0000 7f8f1aa00700 1 -- 172.21.15.89:0/3354665343 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1400bea0 con 0x7f8f240b6990 2024-04-02T09:12:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.433+0000 7f8f1aa00700 1 -- 172.21.15.89:0/3354665343 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f14011830 con 0x7f8f240b6990 2024-04-02T09:12:17.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.433+0000 7f8f29400700 1 -- 172.21.15.89:0/3354665343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f2414b690 con 0x7f8f240b6990 2024-04-02T09:12:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.433+0000 7f8f1aa00700 1 -- 172.21.15.89:0/3354665343 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f14011be0 con 0x7f8f240b6990 2024-04-02T09:12:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.433+0000 7f8f29400700 1 -- 172.21.15.89:0/3354665343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f241563e0 con 0x7f8f240b6990 2024-04-02T09:12:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.434+0000 7f8f29400700 1 -- 172.21.15.89:0/3354665343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f24156ec0 con 0x7f8f240b6990 2024-04-02T09:12:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.435+0000 7f8f1aa00700 1 -- 172.21.15.89:0/3354665343 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8f14011d40 con 0x7f8f240b6990 2024-04-02T09:12:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.436+0000 7f8f1aa00700 1 --2- 172.21.15.89:0/3354665343 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f0406adb0 0x7f8f0406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.436+0000 7f8f1aa00700 1 -- 172.21.15.89:0/3354665343 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5993+0+0 (secure 0 0 0) 0x7f8f14089170 con 0x7f8f240b6990 2024-04-02T09:12:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.436+0000 7f8f21e00700 1 --2- 172.21.15.89:0/3354665343 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f0406adb0 0x7f8f0406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.436+0000 7f8f21e00700 1 --2- 172.21.15.89:0/3354665343 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f0406adb0 0x7f8f0406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8f24155360 tx=0x7f8f10008040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.440+0000 7f8f1aa00700 1 -- 172.21.15.89:0/3354665343 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8f24156ec0 con 0x7f8f240b6990 2024-04-02T09:12:17.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:17.723+0000 7f8f29400700 1 -- 172.21.15.89:0/3354665343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f8f24156ec0 con 0x7f8f240b6990 2024-04-02T09:12:18.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.216+0000 7f8f1aa00700 1 -- 172.21.15.89:0/3354665343 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3126) v1 ==== 189+0+0 (secure 0 0 0) 0x7f8f24156ec0 con 0x7f8f240b6990 2024-04-02T09:12:18.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.219+0000 7f8f29400700 1 -- 172.21.15.89:0/3354665343 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f0406adb0 msgr2=0x7f8f0406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:18.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.219+0000 7f8f29400700 1 --2- 172.21.15.89:0/3354665343 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f0406adb0 0x7f8f0406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8f24155360 tx=0x7f8f10008040 comp rx=0 tx=0).stop 2024-04-02T09:12:18.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.219+0000 7f8f29400700 1 -- 172.21.15.89:0/3354665343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f240b6990 msgr2=0x7f8f2414b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:18.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.219+0000 7f8f29400700 1 --2- 172.21.15.89:0/3354665343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f240b6990 0x7f8f2414b150 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f8f1400b4d0 tx=0x7f8f1400b7e0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.219+0000 7f8f29400700 1 -- 172.21.15.89:0/3354665343 shutdown_connections 2024-04-02T09:12:18.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.219+0000 7f8f29400700 1 --2- 172.21.15.89:0/3354665343 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f2415da50 0x7f8f2415f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.219+0000 7f8f29400700 1 --2- 172.21.15.89:0/3354665343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f240b6990 0x7f8f2414b150 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.219+0000 7f8f29400700 1 --2- 172.21.15.89:0/3354665343 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f0406adb0 0x7f8f0406d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.219+0000 7f8f29400700 1 --2- 172.21.15.89:0/3354665343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f240d05e0 0x7f8f2415f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.219+0000 7f8f29400700 1 -- 172.21.15.89:0/3354665343 >> 172.21.15.89:0/3354665343 conn(0x7f8f240666a0 msgr2=0x7f8f2415ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:18.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.219+0000 7f8f29400700 1 -- 172.21.15.89:0/3354665343 shutdown_connections 2024-04-02T09:12:18.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.219+0000 7f8f29400700 1 -- 172.21.15.89:0/3354665343 wait complete. 2024-04-02T09:12:18.237 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T09:12:18.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.434+0000 7f3ab3c00700 1 Processor -- start 2024-04-02T09:12:18.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.434+0000 7f3ab3c00700 1 -- start start 2024-04-02T09:12:18.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.435+0000 7f3ab3c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aac15dad0 0x7f3aac0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:18.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.435+0000 7f3ab3c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3aac0d1630 0x7f3aac0d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:18.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.435+0000 7f3ab3c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3aac0d1fd0 0x7f3aac15deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:18.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.435+0000 7f3ab3c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aac0cfcc0 con 0x7f3aac0d1630 2024-04-02T09:12:18.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.435+0000 7f3ab3c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aac0cd9f0 con 0x7f3aac0d1fd0 2024-04-02T09:12:18.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.435+0000 7f3ab3c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac15f820 con 0x7f3aac15dad0 2024-04-02T09:12:18.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.435+0000 7f3aabe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3aac0d1630 0x7f3aac0d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:18.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.436+0000 7f3aabe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3aac0d1630 0x7f3aac0d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59440/0 (socket says 172.21.15.89:59440) 2024-04-02T09:12:18.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.436+0000 7f3aabe00700 1 -- 172.21.15.89:0/1918324524 learned_addr learned my addr 172.21.15.89:0/1918324524 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:18.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.436+0000 7f3ab1200700 1 --2- 172.21.15.89:0/1918324524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aac15dad0 0x7f3aac0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:18.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.436+0000 7f3ab1c00700 1 --2- 172.21.15.89:0/1918324524 >> v2:172.21.15.113:3300/0 conn(0x7f3aac0d1fd0 0x7f3aac15deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:18.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.436+0000 7f3aabe00700 1 -- 172.21.15.89:0/1918324524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aac15dad0 msgr2=0x7f3aac0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:18.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.436+0000 7f3aabe00700 1 --2- 172.21.15.89:0/1918324524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aac15dad0 0x7f3aac0ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.436+0000 7f3aabe00700 1 -- 172.21.15.89:0/1918324524 >> v2:172.21.15.113:3300/0 conn(0x7f3aac0d1fd0 msgr2=0x7f3aac15deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:18.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.436+0000 7f3aabe00700 1 --2- 172.21.15.89:0/1918324524 >> v2:172.21.15.113:3300/0 conn(0x7f3aac0d1fd0 0x7f3aac15deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.436+0000 7f3aabe00700 1 -- 172.21.15.89:0/1918324524 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac0d2460 con 0x7f3aac0d1630 2024-04-02T09:12:18.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.436+0000 7f3aabe00700 1 --2- 172.21.15.89:0/1918324524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aac0d1630 0x7f3aac0d1a90 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f3aac0b95a0 tx=0x7f3a98009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=284e1e8aa036602d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:18.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.436+0000 7f3aab400700 1 -- 172.21.15.89:0/1918324524 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9800b870 con 0x7f3aac0d1630 2024-04-02T09:12:18.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.436+0000 7f3aab400700 1 -- 172.21.15.89:0/1918324524 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9800b9d0 con 0x7f3aac0d1630 2024-04-02T09:12:18.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.436+0000 7f3aab400700 1 -- 172.21.15.89:0/1918324524 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9800bd60 con 0x7f3aac0d1630 2024-04-02T09:12:18.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.437+0000 7f3ab3c00700 1 -- 172.21.15.89:0/1918324524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aac0d1630 msgr2=0x7f3aac0d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:18.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.437+0000 7f3ab3c00700 1 --2- 172.21.15.89:0/1918324524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aac0d1630 0x7f3aac0d1a90 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f3aac0b95a0 tx=0x7f3a98009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.437+0000 7f3ab3c00700 1 -- 172.21.15.89:0/1918324524 shutdown_connections 2024-04-02T09:12:18.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.437+0000 7f3ab3c00700 1 --2- 172.21.15.89:0/1918324524 >> v2:172.21.15.113:3300/0 conn(0x7f3aac0d1fd0 0x7f3aac15deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.437+0000 7f3ab3c00700 1 --2- 172.21.15.89:0/1918324524 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aac15dad0 0x7f3aac0ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.437+0000 7f3ab3c00700 1 --2- 172.21.15.89:0/1918324524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aac0d1630 0x7f3aac0d1a90 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.437+0000 7f3ab3c00700 1 -- 172.21.15.89:0/1918324524 >> 172.21.15.89:0/1918324524 conn(0x7f3aac08aff0 msgr2=0x7f3aac0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:18.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.437+0000 7f3ab3c00700 1 -- 172.21.15.89:0/1918324524 shutdown_connections 2024-04-02T09:12:18.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.437+0000 7f3ab3c00700 1 -- 172.21.15.89:0/1918324524 wait complete. 2024-04-02T09:12:18.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.439+0000 7f3ab3c00700 1 Processor -- start 2024-04-02T09:12:18.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.439+0000 7f3ab3c00700 1 -- start start 2024-04-02T09:12:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.440+0000 7f3ab3c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aac0d1630 0x7f3aac154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.440+0000 7f3ab3c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aac0d1fd0 0x7f3aac155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.440+0000 7f3ab3c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3aac15dad0 0x7f3aac1f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.440+0000 7f3ab3c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac155df0 con 0x7f3aac0d1fd0 2024-04-02T09:12:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.440+0000 7f3ab3c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac155b10 con 0x7f3aac15dad0 2024-04-02T09:12:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.440+0000 7f3ab3c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac0b74d0 con 0x7f3aac0d1630 2024-04-02T09:12:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.440+0000 7f3aabe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aac0d1fd0 0x7f3aac155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.440+0000 7f3aabe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aac0d1fd0 0x7f3aac155490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59454/0 (socket says 172.21.15.89:59454) 2024-04-02T09:12:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.440+0000 7f3aabe00700 1 -- 172.21.15.89:0/746877973 learned_addr learned my addr 172.21.15.89:0/746877973 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.440+0000 7f3ab1200700 1 --2- 172.21.15.89:0/746877973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aac0d1630 0x7f3aac154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:18.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.440+0000 7f3ab1c00700 1 --2- 172.21.15.89:0/746877973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3aac15dad0 0x7f3aac1f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:18.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.441+0000 7f3aabe00700 1 -- 172.21.15.89:0/746877973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aac0d1630 msgr2=0x7f3aac154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:18.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.441+0000 7f3aabe00700 1 --2- 172.21.15.89:0/746877973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aac0d1630 0x7f3aac154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.441+0000 7f3aabe00700 1 -- 172.21.15.89:0/746877973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3aac15dad0 msgr2=0x7f3aac1f6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:18.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.441+0000 7f3aabe00700 1 --2- 172.21.15.89:0/746877973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3aac15dad0 0x7f3aac1f6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.441+0000 7f3aabe00700 1 -- 172.21.15.89:0/746877973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a98009550 con 0x7f3aac0d1fd0 2024-04-02T09:12:18.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.441+0000 7f3aabe00700 1 --2- 172.21.15.89:0/746877973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aac0d1fd0 0x7f3aac155490 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f3a98000c00 tx=0x7f3a98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:18.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.441+0000 7f3aa8c00700 1 -- 172.21.15.89:0/746877973 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a98004480 con 0x7f3aac0d1fd0 2024-04-02T09:12:18.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.441+0000 7f3aa8c00700 1 -- 172.21.15.89:0/746877973 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a980045e0 con 0x7f3aac0d1fd0 2024-04-02T09:12:18.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.441+0000 7f3aa8c00700 1 -- 172.21.15.89:0/746877973 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a980038e0 con 0x7f3aac0d1fd0 2024-04-02T09:12:18.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.441+0000 7f3ab3c00700 1 -- 172.21.15.89:0/746877973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aac0b77b0 con 0x7f3aac0d1fd0 2024-04-02T09:12:18.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.441+0000 7f3ab3c00700 1 -- 172.21.15.89:0/746877973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aac152650 con 0x7f3aac0d1fd0 2024-04-02T09:12:18.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.442+0000 7f3ab3c00700 1 -- 172.21.15.89:0/746877973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aac067720 con 0x7f3aac0d1fd0 2024-04-02T09:12:18.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.443+0000 7f3aa8c00700 1 -- 172.21.15.89:0/746877973 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3a98021070 con 0x7f3aac0d1fd0 2024-04-02T09:12:18.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.444+0000 7f3aa8c00700 1 --2- 172.21.15.89:0/746877973 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3a8c06adb0 0x7f3a8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:18.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.444+0000 7f3aa8c00700 1 -- 172.21.15.89:0/746877973 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5993+0+0 (secure 0 0 0) 0x7f3a9808ac20 con 0x7f3aac0d1fd0 2024-04-02T09:12:18.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.444+0000 7f3ab1200700 1 --2- 172.21.15.89:0/746877973 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3a8c06adb0 0x7f3a8c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:18.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.445+0000 7f3ab1200700 1 --2- 172.21.15.89:0/746877973 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3a8c06adb0 0x7f3a8c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0006b20 tx=0x7f3aa0008040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:18.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.448+0000 7f3aa8c00700 1 -- 172.21.15.89:0/746877973 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3a98055d80 con 0x7f3aac0d1fd0 2024-04-02T09:12:18.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.649+0000 7f3ab3c00700 1 -- 172.21.15.89:0/746877973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3aac0b6130 con 0x7f3aac0d1fd0 2024-04-02T09:12:18.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.653+0000 7f3aa8c00700 1 -- 172.21.15.89:0/746877973 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v76)=0 v76) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3a980290c0 con 0x7f3aac0d1fd0 2024-04-02T09:12:18.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.656+0000 7f3ab3c00700 1 -- 172.21.15.89:0/746877973 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3a8c06adb0 msgr2=0x7f3a8c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:18.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.656+0000 7f3ab3c00700 1 --2- 172.21.15.89:0/746877973 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3a8c06adb0 0x7f3a8c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0006b20 tx=0x7f3aa0008040 comp rx=0 tx=0).stop 2024-04-02T09:12:18.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.656+0000 7f3ab3c00700 1 -- 172.21.15.89:0/746877973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aac0d1fd0 msgr2=0x7f3aac155490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:18.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.656+0000 7f3ab3c00700 1 --2- 172.21.15.89:0/746877973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aac0d1fd0 0x7f3aac155490 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f3a98000c00 tx=0x7f3a98004750 comp rx=0 tx=0).stop 2024-04-02T09:12:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.656+0000 7f3ab3c00700 1 -- 172.21.15.89:0/746877973 shutdown_connections 2024-04-02T09:12:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.656+0000 7f3ab3c00700 1 --2- 172.21.15.89:0/746877973 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3a8c06adb0 0x7f3a8c06d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.656+0000 7f3ab3c00700 1 --2- 172.21.15.89:0/746877973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3aac15dad0 0x7f3aac1f6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.656+0000 7f3ab3c00700 1 --2- 172.21.15.89:0/746877973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aac0d1630 0x7f3aac154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.656+0000 7f3ab3c00700 1 --2- 172.21.15.89:0/746877973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aac0d1fd0 0x7f3aac155490 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.656+0000 7f3ab3c00700 1 -- 172.21.15.89:0/746877973 >> 172.21.15.89:0/746877973 conn(0x7f3aac08aff0 msgr2=0x7f3aac0ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.656+0000 7f3ab3c00700 1 -- 172.21.15.89:0/746877973 shutdown_connections 2024-04-02T09:12:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:18.656+0000 7f3ab3c00700 1 -- 172.21.15.89:0/746877973 wait complete. 2024-04-02T09:12:18.672 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T09:12:18.672 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:12:18.764 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:12:18.852 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T09:12:18.852 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:12:18.853 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:12:18.897 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:12:18.897 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:12:18.897 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:12:18.943 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:12:18.943 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:12:18.943 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:12:18.987 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:12:18.987 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:12:18.987 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:12:19.032 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:12:19.032 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T09:12:19.032 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:12:19.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.226+0000 7f22abe00700 1 Processor -- start 2024-04-02T09:12:19.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.227+0000 7f22abe00700 1 -- start start 2024-04-02T09:12:19.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.228+0000 7f22abe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f22a4160f10 0x7f22a40d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:19.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.228+0000 7f22abe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40cf230 0x7f22a40d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:19.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.228+0000 7f22abe00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a40ceda0 con 0x7f22a4161590 2024-04-02T09:12:19.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.228+0000 7f22abe00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a40d23d0 con 0x7f22a4160f10 2024-04-02T09:12:19.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.228+0000 7f22abe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a41612f0 con 0x7f22a40cf230 2024-04-02T09:12:19.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.228+0000 7f22a9400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f22a4160f10 0x7f22a40d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:19.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.228+0000 7f22a9e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f22a4161590 0x7f22a40b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49824/0 (socket says 172.21.15.89:49824) 2024-04-02T09:12:19.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.228+0000 7f22a9e00700 1 -- 172.21.15.89:0/1493089559 learned_addr learned my addr 172.21.15.89:0/1493089559 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:19.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.228+0000 7f22a8a00700 1 --2- 172.21.15.89:0/1493089559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40cf230 0x7f22a40d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:19.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.228+0000 7f22a3e00700 1 -- 172.21.15.89:0/1493089559 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663689929 0 0) 0x7f22a40ceda0 con 0x7f22a4161590 2024-04-02T09:12:19.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.228+0000 7f22a3e00700 1 -- 172.21.15.89:0/1493089559 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c0035b0 con 0x7f22a4161590 2024-04-02T09:12:19.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.229+0000 7f22a9400700 1 -- 172.21.15.89:0/1493089559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40cf230 msgr2=0x7f22a40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:19.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.229+0000 7f22a9400700 1 --2- 172.21.15.89:0/1493089559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40cf230 0x7f22a40d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:19.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.229+0000 7f22a9400700 1 -- 172.21.15.89:0/1493089559 >> v1:172.21.15.89:6789/0 conn(0x7f22a4161590 legacy=0x7f22a40b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:19.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.229+0000 7f22a9400700 1 -- 172.21.15.89:0/1493089559 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a40ce470 con 0x7f22a4160f10 2024-04-02T09:12:19.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.229+0000 7f22a9400700 1 --2- 172.21.15.89:0/1493089559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a4160f10 0x7f22a40d18b0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f2298009840 tx=0x7f2298009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4513d1f51c2ce0a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:19.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.229+0000 7f22a3e00700 1 -- 172.21.15.89:0/1493089559 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298011830 con 0x7f22a4160f10 2024-04-02T09:12:19.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.229+0000 7f22a3e00700 1 -- 172.21.15.89:0/1493089559 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2298011990 con 0x7f22a4160f10 2024-04-02T09:12:19.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.229+0000 7f22a3e00700 1 -- 172.21.15.89:0/1493089559 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298011d20 con 0x7f22a4160f10 2024-04-02T09:12:19.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.230+0000 7f22abe00700 1 -- 172.21.15.89:0/1493089559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a4160f10 msgr2=0x7f22a40d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:19.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.230+0000 7f22abe00700 1 --2- 172.21.15.89:0/1493089559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a4160f10 0x7f22a40d18b0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f2298009840 tx=0x7f2298009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:19.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.230+0000 7f22abe00700 1 -- 172.21.15.89:0/1493089559 shutdown_connections 2024-04-02T09:12:19.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.230+0000 7f22abe00700 1 --2- 172.21.15.89:0/1493089559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a4160f10 0x7f22a40d18b0 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:19.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.230+0000 7f22abe00700 1 --2- 172.21.15.89:0/1493089559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a40cf230 0x7f22a40d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:19.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.230+0000 7f22abe00700 1 -- 172.21.15.89:0/1493089559 >> 172.21.15.89:0/1493089559 conn(0x7f22a408aff0 msgr2=0x7f22a40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:19.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.230+0000 7f22abe00700 1 -- 172.21.15.89:0/1493089559 shutdown_connections 2024-04-02T09:12:19.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.230+0000 7f22abe00700 1 -- 172.21.15.89:0/1493089559 wait complete. 2024-04-02T09:12:19.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.231+0000 7f22abe00700 1 Processor -- start 2024-04-02T09:12:19.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.232+0000 7f22abe00700 1 -- start start 2024-04-02T09:12:19.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.232+0000 7f22abe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22a40cf230 0x7f22a40b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:19.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.232+0000 7f22abe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a4160f10 0x7f22a409efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.232+0000 7f22abe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a4161590 0x7f22a41f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.232+0000 7f22abe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a414c250 con 0x7f22a4160f10 2024-04-02T09:12:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.232+0000 7f22abe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a41f3580 con 0x7f22a40cf230 2024-04-02T09:12:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.232+0000 7f22abe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a409f5f0 con 0x7f22a4161590 2024-04-02T09:12:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.233+0000 7f22a8a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a4160f10 0x7f22a409efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.233+0000 7f22a8a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a4160f10 0x7f22a409efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52490/0 (socket says 172.21.15.89:52490) 2024-04-02T09:12:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.233+0000 7f22a8a00700 1 -- 172.21.15.89:0/58882093 learned_addr learned my addr 172.21.15.89:0/58882093 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.233+0000 7f22a9400700 1 --2- 172.21.15.89:0/58882093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22a40cf230 0x7f22a40b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.233+0000 7f22a9e00700 1 --2- 172.21.15.89:0/58882093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a4161590 0x7f22a41f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.233+0000 7f22a8a00700 1 -- 172.21.15.89:0/58882093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a4161590 msgr2=0x7f22a41f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.233+0000 7f22a8a00700 1 --2- 172.21.15.89:0/58882093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a4161590 0x7f22a41f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.233+0000 7f22a8a00700 1 -- 172.21.15.89:0/58882093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22a40cf230 msgr2=0x7f22a40b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.233+0000 7f22a8a00700 1 --2- 172.21.15.89:0/58882093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22a40cf230 0x7f22a40b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.233+0000 7f22a8a00700 1 -- 172.21.15.89:0/58882093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22980094a0 con 0x7f22a4160f10 2024-04-02T09:12:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.233+0000 7f22a8a00700 1 --2- 172.21.15.89:0/58882093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a4160f10 0x7f22a409efa0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f229000e870 tx=0x7f229000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.234+0000 7f22a1600700 1 -- 172.21.15.89:0/58882093 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290009590 con 0x7f22a4160f10 2024-04-02T09:12:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.234+0000 7f22a1600700 1 -- 172.21.15.89:0/58882093 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2290013680 con 0x7f22a4160f10 2024-04-02T09:12:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.234+0000 7f22abe00700 1 -- 172.21.15.89:0/58882093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a409f8d0 con 0x7f22a4160f10 2024-04-02T09:12:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.234+0000 7f22a1600700 1 -- 172.21.15.89:0/58882093 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22900139f0 con 0x7f22a4160f10 2024-04-02T09:12:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.234+0000 7f22abe00700 1 -- 172.21.15.89:0/58882093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a409fe90 con 0x7f22a4160f10 2024-04-02T09:12:19.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.235+0000 7f22abe00700 1 -- 172.21.15.89:0/58882093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a40a04e0 con 0x7f22a4160f10 2024-04-02T09:12:19.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.236+0000 7f22a1600700 1 -- 172.21.15.89:0/58882093 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2290013b50 con 0x7f22a4160f10 2024-04-02T09:12:19.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.237+0000 7f22a1600700 1 --2- 172.21.15.89:0/58882093 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f228406ae00 0x7f228406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:19.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.237+0000 7f22a1600700 1 -- 172.21.15.89:0/58882093 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5993+0+0 (secure 0 0 0) 0x7f2290089410 con 0x7f22a4160f10 2024-04-02T09:12:19.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.237+0000 7f22a9400700 1 --2- 172.21.15.89:0/58882093 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f228406ae00 0x7f228406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:19.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.237+0000 7f22a9400700 1 --2- 172.21.15.89:0/58882093 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f228406ae00 0x7f228406d2c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2298000c00 tx=0x7f229801c040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:19.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.241+0000 7f22a1600700 1 -- 172.21.15.89:0/58882093 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f22900543a0 con 0x7f22a4160f10 2024-04-02T09:12:19.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.487+0000 7f22abe00700 1 -- 172.21.15.89:0/58882093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22a4067720 con 0x7f22a4160f10 2024-04-02T09:12:19.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.488+0000 7f22a1600700 1 -- 172.21.15.89:0/58882093 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3536 (secure 0 0 0) 0x7f2290017de0 con 0x7f22a4160f10 2024-04-02T09:12:19.486 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:19.486 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":99,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1311594829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1311594829},{"type":"v1","addr":"172.21.15.113:6834","nonce":1311594829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":8912,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/2093635788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2093635788},{"type":"v1","addr":"172.21.15.89:6836","nonce":2093635788}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":99},{"gid":9838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1256547418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":1256547418},{"type":"v1","addr":"172.21.15.89:6837","nonce":1256547418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":10632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1137044351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1137044351},{"type":"v1","addr":"172.21.15.113:6835","nonce":1137044351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[]} 2024-04-02T09:12:19.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.490+0000 7f22abe00700 1 -- 172.21.15.89:0/58882093 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f228406ae00 msgr2=0x7f228406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:19.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.490+0000 7f22abe00700 1 --2- 172.21.15.89:0/58882093 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f228406ae00 0x7f228406d2c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2298000c00 tx=0x7f229801c040 comp rx=0 tx=0).stop 2024-04-02T09:12:19.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.490+0000 7f22abe00700 1 -- 172.21.15.89:0/58882093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a4160f10 msgr2=0x7f22a409efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:19.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.490+0000 7f22abe00700 1 --2- 172.21.15.89:0/58882093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a4160f10 0x7f22a409efa0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f229000e870 tx=0x7f229000eb80 comp rx=0 tx=0).stop 2024-04-02T09:12:19.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.490+0000 7f22abe00700 1 -- 172.21.15.89:0/58882093 shutdown_connections 2024-04-02T09:12:19.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.490+0000 7f22abe00700 1 --2- 172.21.15.89:0/58882093 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f228406ae00 0x7f228406d2c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:19.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.490+0000 7f22abe00700 1 --2- 172.21.15.89:0/58882093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22a40cf230 0x7f22a40b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:19.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.490+0000 7f22abe00700 1 --2- 172.21.15.89:0/58882093 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22a4161590 0x7f22a41f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:19.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.490+0000 7f22abe00700 1 --2- 172.21.15.89:0/58882093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a4160f10 0x7f22a409efa0 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.490+0000 7f22abe00700 1 -- 172.21.15.89:0/58882093 >> 172.21.15.89:0/58882093 conn(0x7f22a408aff0 msgr2=0x7f22a415b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.490+0000 7f22abe00700 1 -- 172.21.15.89:0/58882093 shutdown_connections 2024-04-02T09:12:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.491+0000 7f22abe00700 1 -- 172.21.15.89:0/58882093 wait complete. 2024-04-02T09:12:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 99 2024-04-02T09:12:19.506 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T09:12:19.507 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T09:12:19.507 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T09:12:19.507 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:12:19.507 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T09:12:19.507 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T09:12:19.507 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T09:12:19.507 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:12:19.508 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T09:12:19.508 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T09:12:19.508 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T09:12:19.508 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:12:19.508 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T09:12:19.508 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T09:12:19.508 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T09:12:19.509 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:12:19.522 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T09:12:19.522 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T09:12:19.523 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T09:12:19.523 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T09:12:19.524 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T09:12:19.524 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T09:12:19.525 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T09:12:19.525 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T09:12:19.566 INFO:tasks.ceph.mds.a:Started 2024-04-02T09:12:19.566 DEBUG:teuthology.parallel:result is None 2024-04-02T09:12:19.568 INFO:tasks.ceph.mds.d:Started 2024-04-02T09:12:19.569 DEBUG:teuthology.parallel:result is None 2024-04-02T09:12:19.569 INFO:tasks.ceph.mds.c:Started 2024-04-02T09:12:19.569 DEBUG:teuthology.parallel:result is None 2024-04-02T09:12:19.571 INFO:tasks.ceph.mds.b:Started 2024-04-02T09:12:19.571 DEBUG:teuthology.parallel:result is None 2024-04-02T09:12:19.571 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T09:12:19.714 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T09:12:19.714 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T09:12:19.715 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T09:12:19.715 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T09:12:19.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-04-02T09:12:19.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-04-02T09:12:19.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.780+0000 7f92eaa00700 1 Processor -- start 2024-04-02T09:12:19.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.780+0000 7f92eaa00700 1 -- start start 2024-04-02T09:12:19.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.781+0000 7f92eaa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f92e40cd5c0 0x7f92e415e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:19.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.781+0000 7f92eaa00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e415eda0 con 0x7f92e406f310 2024-04-02T09:12:19.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.781+0000 7f92eaa00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e4099dd0 con 0x7f92e415cfe0 2024-04-02T09:12:19.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.781+0000 7f92eaa00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92e409a0b0 con 0x7f92e40cd5c0 2024-04-02T09:12:19.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.781+0000 7f92e3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f92e40cd5c0 0x7f92e415e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:19.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.781+0000 7f92e3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f92e40cd5c0 0x7f92e415e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52526/0 (socket says 172.21.15.89:52526) 2024-04-02T09:12:19.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.781+0000 7f92e3e00700 1 -- 172.21.15.89:0/2461777277 learned_addr learned my addr 172.21.15.89:0/2461777277 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:19.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.782+0000 7f92e3e00700 1 -- 172.21.15.89:0/2461777277 >> v1:172.21.15.113:6789/0 conn(0x7f92e415cfe0 legacy=0x7f92e415d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:19.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.782+0000 7f92e3e00700 1 -- 172.21.15.89:0/2461777277 >> v1:172.21.15.89:6789/0 conn(0x7f92e406f310 legacy=0x7f92e406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:19.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.782+0000 7f92e3e00700 1 -- 172.21.15.89:0/2461777277 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e409a390 con 0x7f92e40cd5c0 2024-04-02T09:12:19.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.782+0000 7f92e3e00700 1 --2- 172.21.15.89:0/2461777277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e40cd5c0 0x7f92e415e860 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f92d4009920 tx=0x7f92d4009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd89459888658ee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:19.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.782+0000 7f92e3400700 1 -- 172.21.15.89:0/2461777277 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d4004230 con 0x7f92e40cd5c0 2024-04-02T09:12:19.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.782+0000 7f92e3400700 1 -- 172.21.15.89:0/2461777277 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d4004390 con 0x7f92e40cd5c0 2024-04-02T09:12:19.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.782+0000 7f92e3400700 1 -- 172.21.15.89:0/2461777277 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d4009e70 con 0x7f92e40cd5c0 2024-04-02T09:12:19.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.783+0000 7f92eaa00700 1 -- 172.21.15.89:0/2461777277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e40cd5c0 msgr2=0x7f92e415e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:19.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.783+0000 7f92eaa00700 1 --2- 172.21.15.89:0/2461777277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e40cd5c0 0x7f92e415e860 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f92d4009920 tx=0x7f92d4009c30 comp rx=0 tx=0).stop 2024-04-02T09:12:19.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.783+0000 7f92eaa00700 1 -- 172.21.15.89:0/2461777277 shutdown_connections 2024-04-02T09:12:19.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.783+0000 7f92eaa00700 1 --2- 172.21.15.89:0/2461777277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e40cd5c0 0x7f92e415e860 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:19.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.783+0000 7f92eaa00700 1 -- 172.21.15.89:0/2461777277 >> 172.21.15.89:0/2461777277 conn(0x7f92e40d1370 msgr2=0x7f92e40d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:19.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.783+0000 7f92eaa00700 1 -- 172.21.15.89:0/2461777277 shutdown_connections 2024-04-02T09:12:19.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.783+0000 7f92eaa00700 1 -- 172.21.15.89:0/2461777277 wait complete. 2024-04-02T09:12:19.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.785+0000 7f92eaa00700 1 Processor -- start 2024-04-02T09:12:19.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.785+0000 7f92eaa00700 1 -- start start 2024-04-02T09:12:19.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92eaa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92e406f310 0x7f92e40b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:19.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92eaa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e40cd5c0 0x7f92e40d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:19.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92eaa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92e415cfe0 0x7f92e40d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:19.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92eaa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e415b600 con 0x7f92e40cd5c0 2024-04-02T09:12:19.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92eaa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e40b70a0 con 0x7f92e406f310 2024-04-02T09:12:19.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92eaa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e41f45c0 con 0x7f92e415cfe0 2024-04-02T09:12:19.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92dbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e40cd5c0 0x7f92e40d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:19.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92dbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e40cd5c0 0x7f92e40d00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52540/0 (socket says 172.21.15.89:52540) 2024-04-02T09:12:19.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92dbe00700 1 -- 172.21.15.89:0/2777084185 learned_addr learned my addr 172.21.15.89:0/2777084185 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:19.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92e3e00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92e406f310 0x7f92e40b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:19.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92dbe00700 1 -- 172.21.15.89:0/2777084185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92e415cfe0 msgr2=0x7f92e40d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:19.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92e8a00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92e415cfe0 0x7f92e40d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:19.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92dbe00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92e415cfe0 0x7f92e40d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:19.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92dbe00700 1 -- 172.21.15.89:0/2777084185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92e406f310 msgr2=0x7f92e40b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:19.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92dbe00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92e406f310 0x7f92e40b6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:19.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92dbe00700 1 -- 172.21.15.89:0/2777084185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d4009550 con 0x7f92e40cd5c0 2024-04-02T09:12:19.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92e3e00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92e406f310 0x7f92e40b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:19.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.786+0000 7f92e8a00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92e415cfe0 0x7f92e40d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:19.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.787+0000 7f92dbe00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e40cd5c0 0x7f92e40d00a0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00c500 tx=0x7f92cc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:19.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.787+0000 7f92e0c00700 1 -- 172.21.15.89:0/2777084185 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc0052d0 con 0x7f92e40cd5c0 2024-04-02T09:12:19.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.787+0000 7f92e0c00700 1 -- 172.21.15.89:0/2777084185 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92cc005430 con 0x7f92e40cd5c0 2024-04-02T09:12:19.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.787+0000 7f92eaa00700 1 -- 172.21.15.89:0/2777084185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e41f48a0 con 0x7f92e40cd5c0 2024-04-02T09:12:19.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.787+0000 7f92e0c00700 1 -- 172.21.15.89:0/2777084185 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc006aa0 con 0x7f92e40cd5c0 2024-04-02T09:12:19.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.787+0000 7f92eaa00700 1 -- 172.21.15.89:0/2777084185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e40b6360 con 0x7f92e40cd5c0 2024-04-02T09:12:19.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.788+0000 7f92eaa00700 1 -- 172.21.15.89:0/2777084185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e406d820 con 0x7f92e40cd5c0 2024-04-02T09:12:19.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.789+0000 7f92e0c00700 1 -- 172.21.15.89:0/2777084185 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f92cc003680 con 0x7f92e40cd5c0 2024-04-02T09:12:19.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.790+0000 7f92e0c00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f92c006adb0 0x7f92c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:19.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.790+0000 7f92e0c00700 1 -- 172.21.15.89:0/2777084185 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 5993+0+0 (secure 0 0 0) 0x7f92cc088780 con 0x7f92e40cd5c0 2024-04-02T09:12:19.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.790+0000 7f92e3e00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f92c006adb0 0x7f92c006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:19.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.791+0000 7f92e3e00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f92c006adb0 0x7f92c006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f92d4009c70 tx=0x7f92d4008030 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:19.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:19.795+0000 7f92e0c00700 1 -- 172.21.15.89:0/2777084185 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f92cc053690 con 0x7f92e40cd5c0 2024-04-02T09:12:20.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.015+0000 7f92eaa00700 1 -- 172.21.15.89:0/2777084185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f92e415eda0 con 0x7f92e40cd5c0 2024-04-02T09:12:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.746+0000 7f92e0c00700 1 -- 172.21.15.89:0/2777084185 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v219) v1 ==== 97+0+0 (secure 0 0 0) 0x7f92e415eda0 con 0x7f92e40cd5c0 2024-04-02T09:12:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.750+0000 7f92eaa00700 1 -- 172.21.15.89:0/2777084185 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f92c006adb0 msgr2=0x7f92c006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.750+0000 7f92eaa00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f92c006adb0 0x7f92c006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f92d4009c70 tx=0x7f92d4008030 comp rx=0 tx=0).stop 2024-04-02T09:12:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.750+0000 7f92eaa00700 1 -- 172.21.15.89:0/2777084185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e40cd5c0 msgr2=0x7f92e40d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.750+0000 7f92eaa00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e40cd5c0 0x7f92e40d00a0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00c500 tx=0x7f92cc00c8c0 comp rx=0 tx=0).stop 2024-04-02T09:12:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.750+0000 7f92eaa00700 1 -- 172.21.15.89:0/2777084185 shutdown_connections 2024-04-02T09:12:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.750+0000 7f92eaa00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f92c006adb0 0x7f92c006d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.750+0000 7f92eaa00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92e406f310 0x7f92e40b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.750+0000 7f92eaa00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92e415cfe0 0x7f92e40d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.750+0000 7f92eaa00700 1 --2- 172.21.15.89:0/2777084185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92e40cd5c0 0x7f92e40d00a0 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.750+0000 7f92eaa00700 1 -- 172.21.15.89:0/2777084185 >> 172.21.15.89:0/2777084185 conn(0x7f92e40d1370 msgr2=0x7f92e415c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.750+0000 7f92eaa00700 1 -- 172.21.15.89:0/2777084185 shutdown_connections 2024-04-02T09:12:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.750+0000 7f92eaa00700 1 -- 172.21.15.89:0/2777084185 wait complete. 2024-04-02T09:12:20.749 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T09:12:20.769 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T09:12:20.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.965+0000 7fdefa600700 1 Processor -- start 2024-04-02T09:12:20.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.965+0000 7fdefa600700 1 -- start start 2024-04-02T09:12:20.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.966+0000 7fdefa600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdef4160c90 0x7fdef40b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:20.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.966+0000 7fdefa600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef40ac4d0 con 0x7fdef40718e0 2024-04-02T09:12:20.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.966+0000 7fdefa600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef406ddd0 con 0x7fdef40ad970 2024-04-02T09:12:20.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.966+0000 7fdefa600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef4071e50 con 0x7fdef4160c90 2024-04-02T09:12:20.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.967+0000 7fdef2a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdef40718e0 0x7fdef406cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49842/0 (socket says 172.21.15.89:49842) 2024-04-02T09:12:20.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.967+0000 7fdef2a00700 1 -- 172.21.15.89:0/3099086504 learned_addr learned my addr 172.21.15.89:0/3099086504 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:20.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.967+0000 7fdef3400700 1 --2- 172.21.15.89:0/3099086504 >> v2:172.21.15.113:3300/0 conn(0x7fdef4160c90 0x7fdef40b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:20.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.967+0000 7fdef2000700 1 -- 172.21.15.89:0/3099086504 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210283007 0 0) 0x7fdef40ac4d0 con 0x7fdef40718e0 2024-04-02T09:12:20.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.967+0000 7fdef2000700 1 -- 172.21.15.89:0/3099086504 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded80035b0 con 0x7fdef40718e0 2024-04-02T09:12:20.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.967+0000 7fdef2000700 1 -- 172.21.15.89:0/3099086504 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 939201785 0 0) 0x7fded80035b0 con 0x7fdef40718e0 2024-04-02T09:12:20.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.967+0000 7fdef2000700 1 -- 172.21.15.89:0/3099086504 >> v2:172.21.15.113:3300/0 conn(0x7fdef4160c90 msgr2=0x7fdef40b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:20.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.967+0000 7fdef2000700 1 --2- 172.21.15.89:0/3099086504 >> v2:172.21.15.113:3300/0 conn(0x7fdef4160c90 0x7fdef40b20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:20.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.967+0000 7fdef2000700 1 -- 172.21.15.89:0/3099086504 >> v1:172.21.15.113:6789/0 conn(0x7fdef40ad970 legacy=0x7fdef406d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:20.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.967+0000 7fdef2000700 1 -- 172.21.15.89:0/3099086504 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef4072130 con 0x7fdef40718e0 2024-04-02T09:12:20.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.967+0000 7fdef2000700 1 -- 172.21.15.89:0/3099086504 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fdee0001670 con 0x7fdef40718e0 2024-04-02T09:12:20.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef2000700 1 -- 172.21.15.89:0/3099086504 >> v1:172.21.15.89:6789/0 conn(0x7fdef40718e0 legacy=0x7fdef406cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:20.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef2000700 1 --2- 172.21.15.89:0/3099086504 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fded8003a50 0x7fded8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:20.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef2000700 1 --2- 172.21.15.89:0/3099086504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fded8004850 0x7fded8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:20.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef2000700 1 --2- 172.21.15.89:0/3099086504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fded8005210 0x7fded800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:20.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef2000700 1 -- 172.21.15.89:0/3099086504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef4071e50 con 0x7fded8004850 2024-04-02T09:12:20.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef2000700 1 -- 172.21.15.89:0/3099086504 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded800c110 con 0x7fded8005210 2024-04-02T09:12:20.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef2000700 1 -- 172.21.15.89:0/3099086504 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded800c2d0 con 0x7fded8003a50 2024-04-02T09:12:20.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef2a00700 1 --2- 172.21.15.89:0/3099086504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fded8004850 0x7fded8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:20.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef3400700 1 --2- 172.21.15.89:0/3099086504 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fded8003a50 0x7fded8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:20.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef3e00700 1 --2- 172.21.15.89:0/3099086504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fded8005210 0x7fded800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:20.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef2a00700 1 -- 172.21.15.89:0/3099086504 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fded8003a50 msgr2=0x7fded8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:20.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef2a00700 1 --2- 172.21.15.89:0/3099086504 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fded8003a50 0x7fded8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:20.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef2a00700 1 -- 172.21.15.89:0/3099086504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fded8005210 msgr2=0x7fded800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:20.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef2a00700 1 --2- 172.21.15.89:0/3099086504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fded8005210 0x7fded800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:20.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef2a00700 1 -- 172.21.15.89:0/3099086504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded80035b0 con 0x7fded8004850 2024-04-02T09:12:20.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef3e00700 1 --2- 172.21.15.89:0/3099086504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fded8005210 0x7fded800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:20.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.968+0000 7fdef3e00700 1 -- 172.21.15.89:0/3099086504 reap_dead start 2024-04-02T09:12:20.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.969+0000 7fdef2a00700 1 --2- 172.21.15.89:0/3099086504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fded8004850 0x7fded8004cd0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fdee000c160 tx=0x7fded8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd7983d333d50cb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:20.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.969+0000 7fdef2000700 1 -- 172.21.15.89:0/3099086504 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee0007b70 con 0x7fded8004850 2024-04-02T09:12:20.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.969+0000 7fdef2000700 1 -- 172.21.15.89:0/3099086504 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee000cd20 con 0x7fded8004850 2024-04-02T09:12:20.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.969+0000 7fdef2000700 1 -- 172.21.15.89:0/3099086504 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee0005ba0 con 0x7fded8004850 2024-04-02T09:12:20.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.969+0000 7fdefa600700 1 -- 172.21.15.89:0/3099086504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fded8004850 msgr2=0x7fded8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:20.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.969+0000 7fdefa600700 1 --2- 172.21.15.89:0/3099086504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fded8004850 0x7fded8004cd0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fdee000c160 tx=0x7fded8005880 comp rx=0 tx=0).stop 2024-04-02T09:12:20.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.969+0000 7fdefa600700 1 -- 172.21.15.89:0/3099086504 shutdown_connections 2024-04-02T09:12:20.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.969+0000 7fdefa600700 1 --2- 172.21.15.89:0/3099086504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fded8004850 0x7fded8004cd0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:20.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.969+0000 7fdefa600700 1 -- 172.21.15.89:0/3099086504 >> 172.21.15.89:0/3099086504 conn(0x7fdef40cf190 msgr2=0x7fdef40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:20.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.969+0000 7fdefa600700 1 -- 172.21.15.89:0/3099086504 shutdown_connections 2024-04-02T09:12:20.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.969+0000 7fdefa600700 1 -- 172.21.15.89:0/3099086504 wait complete. 2024-04-02T09:12:20.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.971+0000 7fdefa600700 1 Processor -- start 2024-04-02T09:12:20.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.971+0000 7fdefa600700 1 -- start start 2024-04-02T09:12:20.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.972+0000 7fdefa600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fded8004850 0x7fdef41f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.972+0000 7fdefa600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdef4155d00 0x7fdef409c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.972+0000 7fdefa600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdef40b0200 0x7fdef40b0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.972+0000 7fdefa600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef41f72c0 con 0x7fdef40b0200 2024-04-02T09:12:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.972+0000 7fdefa600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef40b0bc0 con 0x7fdef4155d00 2024-04-02T09:12:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.972+0000 7fdefa600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef4157a80 con 0x7fded8004850 2024-04-02T09:12:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.972+0000 7fdef3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdef40b0200 0x7fdef40b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.972+0000 7fdef3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdef40b0200 0x7fdef40b0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52562/0 (socket says 172.21.15.89:52562) 2024-04-02T09:12:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.972+0000 7fdef3e00700 1 -- 172.21.15.89:0/114977543 learned_addr learned my addr 172.21.15.89:0/114977543 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.973+0000 7fdef2a00700 1 --2- 172.21.15.89:0/114977543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdef4155d00 0x7fdef409c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.973+0000 7fdef3400700 1 --2- 172.21.15.89:0/114977543 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fded8004850 0x7fdef41f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.973+0000 7fdef3e00700 1 -- 172.21.15.89:0/114977543 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fded8004850 msgr2=0x7fdef41f6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.973+0000 7fdef3e00700 1 --2- 172.21.15.89:0/114977543 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fded8004850 0x7fdef41f6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.973+0000 7fdef3e00700 1 -- 172.21.15.89:0/114977543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdef4155d00 msgr2=0x7fdef409c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:20.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.973+0000 7fdef3e00700 1 --2- 172.21.15.89:0/114977543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdef4155d00 0x7fdef409c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:20.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.973+0000 7fdef3e00700 1 -- 172.21.15.89:0/114977543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef4158040 con 0x7fdef40b0200 2024-04-02T09:12:20.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.973+0000 7fdef3e00700 1 --2- 172.21.15.89:0/114977543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdef40b0200 0x7fdef40b0680 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7fdef4072e70 tx=0x7fdeec00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:20.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.973+0000 7fdeeb400700 1 -- 172.21.15.89:0/114977543 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec00ee70 con 0x7fdef40b0200 2024-04-02T09:12:20.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.973+0000 7fdeeb400700 1 -- 172.21.15.89:0/114977543 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec003c10 con 0x7fdef40b0200 2024-04-02T09:12:20.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.973+0000 7fdeeb400700 1 -- 172.21.15.89:0/114977543 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec0050d0 con 0x7fdef40b0200 2024-04-02T09:12:20.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.973+0000 7fdefa600700 1 -- 172.21.15.89:0/114977543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef41562c0 con 0x7fdef40b0200 2024-04-02T09:12:20.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.974+0000 7fdefa600700 1 -- 172.21.15.89:0/114977543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef4157cd0 con 0x7fdef40b0200 2024-04-02T09:12:20.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.975+0000 7fdefa600700 1 -- 172.21.15.89:0/114977543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef41581d0 con 0x7fdef40b0200 2024-04-02T09:12:20.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.976+0000 7fdeeb400700 1 -- 172.21.15.89:0/114977543 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdeec005230 con 0x7fdef40b0200 2024-04-02T09:12:20.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.977+0000 7fdeeb400700 1 --2- 172.21.15.89:0/114977543 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fded406adb0 0x7fded406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:20.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.977+0000 7fdeeb400700 1 -- 172.21.15.89:0/114977543 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fdeec089160 con 0x7fdef40b0200 2024-04-02T09:12:20.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.977+0000 7fdef3400700 1 --2- 172.21.15.89:0/114977543 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fded406adb0 0x7fded406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:20.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.978+0000 7fdef3400700 1 --2- 172.21.15.89:0/114977543 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fded406adb0 0x7fded406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009ce0 tx=0x7fdee4009210 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:20.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:20.981+0000 7fdeeb400700 1 -- 172.21.15.89:0/114977543 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdeec0543c0 con 0x7fdef40b0200 2024-04-02T09:12:21.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.263+0000 7fdefa600700 1 -- 172.21.15.89:0/114977543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fdef40709c0 con 0x7fdef40b0200 2024-04-02T09:12:21.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.263+0000 7fdeeb400700 1 -- 172.21.15.89:0/114977543 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v53) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fdeec0579e0 con 0x7fdef40b0200 2024-04-02T09:12:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:21.262 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:12:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T09:12:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T09:12:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T09:12:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:12:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:12:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:12:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T09:12:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T09:12:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:12:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:12:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:12:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T09:12:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T09:12:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:12:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:12:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:12:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T09:12:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T09:12:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:12:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:12:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:12:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T09:12:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T09:12:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:12:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:12:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T09:12:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T09:12:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:12:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:12:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T09:12:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T09:12:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:12:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:12:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T09:12:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T09:12:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:12:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:12:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T09:12:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T09:12:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:12:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:12:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T09:12:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T09:12:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:12:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:12:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T09:12:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T09:12:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:12:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:12:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T09:12:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T09:12:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:12:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:12:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T09:12:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T09:12:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:12:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2024-04-02T09:12:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T09:12:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T09:12:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T09:12:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:12:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:12:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:12:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:12:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T09:12:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T09:12:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:12:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:12:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:12:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:12:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T09:12:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T09:12:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:12:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:12:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:12:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:12:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T09:12:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T09:12:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:12:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:12:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:12:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T09:12:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T09:12:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:12:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T09:12:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T09:12:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T09:12:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:12:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:12:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T09:12:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:12:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T09:12:21.275 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:12:21.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.265+0000 7fdefa600700 1 -- 172.21.15.89:0/114977543 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fded406adb0 msgr2=0x7fded406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:21.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.265+0000 7fdefa600700 1 --2- 172.21.15.89:0/114977543 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fded406adb0 0x7fded406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009ce0 tx=0x7fdee4009210 comp rx=0 tx=0).stop 2024-04-02T09:12:21.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.265+0000 7fdefa600700 1 -- 172.21.15.89:0/114977543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdef40b0200 msgr2=0x7fdef40b0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:21.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.265+0000 7fdefa600700 1 --2- 172.21.15.89:0/114977543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdef40b0200 0x7fdef40b0680 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7fdef4072e70 tx=0x7fdeec00ed50 comp rx=0 tx=0).stop 2024-04-02T09:12:21.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.265+0000 7fdefa600700 1 -- 172.21.15.89:0/114977543 shutdown_connections 2024-04-02T09:12:21.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.265+0000 7fdefa600700 1 --2- 172.21.15.89:0/114977543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdef40b0200 0x7fdef40b0680 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.265+0000 7fdefa600700 1 --2- 172.21.15.89:0/114977543 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fded8004850 0x7fdef41f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.265+0000 7fdefa600700 1 --2- 172.21.15.89:0/114977543 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fded406adb0 0x7fded406d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.265+0000 7fdefa600700 1 --2- 172.21.15.89:0/114977543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdef4155d00 0x7fdef409c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.265+0000 7fdefa600700 1 -- 172.21.15.89:0/114977543 >> 172.21.15.89:0/114977543 conn(0x7fdef40cf190 msgr2=0x7fdef406cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:21.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.266+0000 7fdefa600700 1 -- 172.21.15.89:0/114977543 shutdown_connections 2024-04-02T09:12:21.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.266+0000 7fdefa600700 1 -- 172.21.15.89:0/114977543 wait complete. 2024-04-02T09:12:21.282 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:12:21.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.476+0000 7ff4c8000700 1 Processor -- start 2024-04-02T09:12:21.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.477+0000 7ff4c8000700 1 -- start start 2024-04-02T09:12:21.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.478+0000 7ff4c8000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff4c0160be0 0x7ff4c00cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:21.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.478+0000 7ff4c8000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff4c00ceca0 0x7ff4c00b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:21.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.478+0000 7ff4c8000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c0160fc0 con 0x7ff4c00ce0e0 2024-04-02T09:12:21.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.478+0000 7ff4c8000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c00b65d0 con 0x7ff4c00ceca0 2024-04-02T09:12:21.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.478+0000 7ff4c8000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c00d2a80 con 0x7ff4c0160be0 2024-04-02T09:12:21.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.478+0000 7ff4c6000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff4c00ceca0 0x7ff4c00b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:21.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.478+0000 7ff4c6000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff4c00ceca0 0x7ff4c00b6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52572/0 (socket says 172.21.15.89:52572) 2024-04-02T09:12:21.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.478+0000 7ff4c6000700 1 -- 172.21.15.89:0/3938537493 learned_addr learned my addr 172.21.15.89:0/3938537493 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:21.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.478+0000 7ff4c5600700 1 --2- 172.21.15.89:0/3938537493 >> v2:172.21.15.113:3300/0 conn(0x7ff4c0160be0 0x7ff4c00cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:21.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.478+0000 7ff4c6000700 1 -- 172.21.15.89:0/3938537493 >> v2:172.21.15.113:3300/0 conn(0x7ff4c0160be0 msgr2=0x7ff4c00cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:21.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.478+0000 7ff4c6000700 1 --2- 172.21.15.89:0/3938537493 >> v2:172.21.15.113:3300/0 conn(0x7ff4c0160be0 0x7ff4c00cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.479+0000 7ff4c6000700 1 -- 172.21.15.89:0/3938537493 >> v1:172.21.15.89:6789/0 conn(0x7ff4c00ce0e0 legacy=0x7ff4c00ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:21.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.479+0000 7ff4c6000700 1 -- 172.21.15.89:0/3938537493 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c00d21e0 con 0x7ff4c00ceca0 2024-04-02T09:12:21.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.479+0000 7ff4c6000700 1 --2- 172.21.15.89:0/3938537493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4c00ceca0 0x7ff4c00b6a20 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7ff4c008b590 tx=0x7ff4b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0fc1cc08b998377 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:21.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.479+0000 7ff4bfe00700 1 -- 172.21.15.89:0/3938537493 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b800e070 con 0x7ff4c00ceca0 2024-04-02T09:12:21.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.479+0000 7ff4bfe00700 1 -- 172.21.15.89:0/3938537493 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b800b910 con 0x7ff4c00ceca0 2024-04-02T09:12:21.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.479+0000 7ff4bfe00700 1 -- 172.21.15.89:0/3938537493 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b800bcc0 con 0x7ff4c00ceca0 2024-04-02T09:12:21.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.479+0000 7ff4c8000700 1 -- 172.21.15.89:0/3938537493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4c00ceca0 msgr2=0x7ff4c00b6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:21.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.479+0000 7ff4c8000700 1 --2- 172.21.15.89:0/3938537493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4c00ceca0 0x7ff4c00b6a20 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7ff4c008b590 tx=0x7ff4b800b330 comp rx=0 tx=0).stop 2024-04-02T09:12:21.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.480+0000 7ff4c8000700 1 -- 172.21.15.89:0/3938537493 shutdown_connections 2024-04-02T09:12:21.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.480+0000 7ff4c8000700 1 --2- 172.21.15.89:0/3938537493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4c00ceca0 0x7ff4c00b6a20 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.480+0000 7ff4c8000700 1 --2- 172.21.15.89:0/3938537493 >> v2:172.21.15.113:3300/0 conn(0x7ff4c0160be0 0x7ff4c00cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.480+0000 7ff4c8000700 1 -- 172.21.15.89:0/3938537493 >> 172.21.15.89:0/3938537493 conn(0x7ff4c008aff0 msgr2=0x7ff4c00d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:21.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.480+0000 7ff4c8000700 1 -- 172.21.15.89:0/3938537493 shutdown_connections 2024-04-02T09:12:21.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.480+0000 7ff4c8000700 1 -- 172.21.15.89:0/3938537493 wait complete. 2024-04-02T09:12:21.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.481+0000 7ff4c8000700 1 Processor -- start 2024-04-02T09:12:21.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.481+0000 7ff4c8000700 1 -- start start 2024-04-02T09:12:21.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.482+0000 7ff4c8000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4c00ce0e0 0x7ff4c01f4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:21.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.482+0000 7ff4c8000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff4c00ceca0 0x7ff4c0158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:21.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.482+0000 7ff4c8000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c0160be0 0x7ff4c015a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:21.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.482+0000 7ff4c8000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00c7ba0 con 0x7ff4c00ce0e0 2024-04-02T09:12:21.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.482+0000 7ff4c8000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00c7200 con 0x7ff4c0160be0 2024-04-02T09:12:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.482+0000 7ff4c8000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00c74e0 con 0x7ff4c00ceca0 2024-04-02T09:12:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4c5600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4c00ce0e0 0x7ff4c01f4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4c5600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4c00ce0e0 0x7ff4c01f4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52576/0 (socket says 172.21.15.89:52576) 2024-04-02T09:12:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4c5600700 1 -- 172.21.15.89:0/2236397196 learned_addr learned my addr 172.21.15.89:0/2236397196 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4c4c00700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff4c00ceca0 0x7ff4c0158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4c6000700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c0160be0 0x7ff4c015a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4c5600700 1 -- 172.21.15.89:0/2236397196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff4c00ceca0 msgr2=0x7ff4c0158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4c5600700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff4c00ceca0 0x7ff4c0158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4c5600700 1 -- 172.21.15.89:0/2236397196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c0160be0 msgr2=0x7ff4c015a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4c5600700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c0160be0 0x7ff4c015a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4c5600700 1 -- 172.21.15.89:0/2236397196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b8009b30 con 0x7ff4c00ce0e0 2024-04-02T09:12:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4c6000700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c0160be0 0x7ff4c015a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:21.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4c5600700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4c00ce0e0 0x7ff4c01f4220 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400e850 tx=0x7ff4b400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:21.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4bd600700 1 -- 172.21.15.89:0/2236397196 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b40095b0 con 0x7ff4c00ce0e0 2024-04-02T09:12:21.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4bd600700 1 -- 172.21.15.89:0/2236397196 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b400ee50 con 0x7ff4c00ce0e0 2024-04-02T09:12:21.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4c8000700 1 -- 172.21.15.89:0/2236397196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c00c77c0 con 0x7ff4c00ce0e0 2024-04-02T09:12:21.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.483+0000 7ff4bd600700 1 -- 172.21.15.89:0/2236397196 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b400ca70 con 0x7ff4c00ce0e0 2024-04-02T09:12:21.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.484+0000 7ff4c8000700 1 -- 172.21.15.89:0/2236397196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c01f4940 con 0x7ff4c00ce0e0 2024-04-02T09:12:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.485+0000 7ff4c8000700 1 -- 172.21.15.89:0/2236397196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c0067720 con 0x7ff4c00ce0e0 2024-04-02T09:12:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.486+0000 7ff4bd600700 1 -- 172.21.15.89:0/2236397196 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff4b400cbd0 con 0x7ff4c00ce0e0 2024-04-02T09:12:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.486+0000 7ff4bd600700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff4a006adb0 0x7ff4a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.486+0000 7ff4bd600700 1 -- 172.21.15.89:0/2236397196 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5305+0+0 (secure 0 0 0) 0x7ff4b4089220 con 0x7ff4c00ce0e0 2024-04-02T09:12:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.487+0000 7ff4c4c00700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff4a006adb0 0x7ff4a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.487+0000 7ff4c4c00700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff4a006adb0 0x7ff4a006d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff4c01f65b0 tx=0x7ff4ac009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:21.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.491+0000 7ff4bd600700 1 -- 172.21.15.89:0/2236397196 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff4b4054510 con 0x7ff4c00ce0e0 2024-04-02T09:12:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.693+0000 7ff4c8000700 1 -- 172.21.15.89:0/2236397196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4c00b5cb0 con 0x7ff4c00ce0e0 2024-04-02T09:12:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.694+0000 7ff4bd600700 1 -- 172.21.15.89:0/2236397196 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+13088 (secure 0 0 0) 0x7ff4b4057b30 con 0x7ff4c00ce0e0 2024-04-02T09:12:21.693 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:21.693 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":219,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:12:20.741527+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:12:21.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.696+0000 7ff4c8000700 1 -- 172.21.15.89:0/2236397196 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff4a006adb0 msgr2=0x7ff4a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:21.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.696+0000 7ff4c8000700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff4a006adb0 0x7ff4a006d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff4c01f65b0 tx=0x7ff4ac009040 comp rx=0 tx=0).stop 2024-04-02T09:12:21.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.696+0000 7ff4c8000700 1 -- 172.21.15.89:0/2236397196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4c00ce0e0 msgr2=0x7ff4c01f4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:21.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.696+0000 7ff4c8000700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4c00ce0e0 0x7ff4c01f4220 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400e850 tx=0x7ff4b400ec10 comp rx=0 tx=0).stop 2024-04-02T09:12:21.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.697+0000 7ff4c8000700 1 -- 172.21.15.89:0/2236397196 shutdown_connections 2024-04-02T09:12:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.697+0000 7ff4c8000700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff4a006adb0 0x7ff4a006d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.697+0000 7ff4c8000700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c0160be0 0x7ff4c015a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.697+0000 7ff4c8000700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4c00ce0e0 0x7ff4c01f4220 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.697+0000 7ff4c8000700 1 --2- 172.21.15.89:0/2236397196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff4c00ceca0 0x7ff4c0158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.697+0000 7ff4c8000700 1 -- 172.21.15.89:0/2236397196 >> 172.21.15.89:0/2236397196 conn(0x7ff4c008aff0 msgr2=0x7ff4c00d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.697+0000 7ff4c8000700 1 -- 172.21.15.89:0/2236397196 shutdown_connections 2024-04-02T09:12:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.697+0000 7ff4c8000700 1 -- 172.21.15.89:0/2236397196 wait complete. 2024-04-02T09:12:21.713 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:12:21.714 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:12:21.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.911+0000 7fae91c00700 1 Processor -- start 2024-04-02T09:12:21.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.911+0000 7fae91c00700 1 -- start start 2024-04-02T09:12:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.912+0000 7fae91c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fae8c15d9d0 0x7fae8c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.912+0000 7fae91c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c161420 con 0x7fae8c0d2540 2024-04-02T09:12:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.912+0000 7fae91c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c161700 con 0x7fae8c0d16a0 2024-04-02T09:12:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.912+0000 7fae91c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8c15edc0 con 0x7fae8c15d9d0 2024-04-02T09:12:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.913+0000 7fae8b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fae8c0d2540 0x7fae8c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49870/0 (socket says 172.21.15.89:49870) 2024-04-02T09:12:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.913+0000 7fae8b400700 1 -- 172.21.15.89:0/4035775640 learned_addr learned my addr 172.21.15.89:0/4035775640 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:21.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.913+0000 7fae8aa00700 1 --2- 172.21.15.89:0/4035775640 >> v2:172.21.15.113:3300/0 conn(0x7fae8c15d9d0 0x7fae8c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:21.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.913+0000 7fae89600700 1 -- 172.21.15.89:0/4035775640 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206977916 0 0) 0x7fae8c161420 con 0x7fae8c0d2540 2024-04-02T09:12:21.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.913+0000 7fae89600700 1 -- 172.21.15.89:0/4035775640 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae700035b0 con 0x7fae8c0d2540 2024-04-02T09:12:21.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.913+0000 7fae89600700 1 -- 172.21.15.89:0/4035775640 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3516932154 0 0) 0x7fae700035b0 con 0x7fae8c0d2540 2024-04-02T09:12:21.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.913+0000 7fae89600700 1 -- 172.21.15.89:0/4035775640 >> v2:172.21.15.113:3300/0 conn(0x7fae8c15d9d0 msgr2=0x7fae8c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:21.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.913+0000 7fae89600700 1 --2- 172.21.15.89:0/4035775640 >> v2:172.21.15.113:3300/0 conn(0x7fae8c15d9d0 0x7fae8c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.913+0000 7fae89600700 1 -- 172.21.15.89:0/4035775640 >> v1:172.21.15.113:6789/0 conn(0x7fae8c0d16a0 legacy=0x7fae8c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:21.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.913+0000 7fae89600700 1 -- 172.21.15.89:0/4035775640 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c1619e0 con 0x7fae8c0d2540 2024-04-02T09:12:21.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.913+0000 7fae89600700 1 -- 172.21.15.89:0/4035775640 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fae840019f0 con 0x7fae8c0d2540 2024-04-02T09:12:21.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae89600700 1 -- 172.21.15.89:0/4035775640 >> v1:172.21.15.89:6789/0 conn(0x7fae8c0d2540 legacy=0x7fae8c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:21.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae89600700 1 --2- 172.21.15.89:0/4035775640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae70003a00 0x7fae70003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:21.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae89600700 1 --2- 172.21.15.89:0/4035775640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae70004400 0x7fae700048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae89600700 1 --2- 172.21.15.89:0/4035775640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae70005170 0x7fae7000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae89600700 1 -- 172.21.15.89:0/4035775640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7000c080 con 0x7fae70003a00 2024-04-02T09:12:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae89600700 1 -- 172.21.15.89:0/4035775640 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7000c1f0 con 0x7fae70005170 2024-04-02T09:12:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae89600700 1 -- 172.21.15.89:0/4035775640 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7000c3b0 con 0x7fae70004400 2024-04-02T09:12:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae8aa00700 1 --2- 172.21.15.89:0/4035775640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae70003a00 0x7fae70003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae8b400700 1 --2- 172.21.15.89:0/4035775640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae70005170 0x7fae7000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae8a000700 1 --2- 172.21.15.89:0/4035775640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae70004400 0x7fae700048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae8aa00700 1 -- 172.21.15.89:0/4035775640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae70004400 msgr2=0x7fae700048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae8aa00700 1 --2- 172.21.15.89:0/4035775640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae70004400 0x7fae700048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae8aa00700 1 -- 172.21.15.89:0/4035775640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae70005170 msgr2=0x7fae7000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae8aa00700 1 --2- 172.21.15.89:0/4035775640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae70005170 0x7fae7000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae8aa00700 1 -- 172.21.15.89:0/4035775640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae700035b0 con 0x7fae70003a00 2024-04-02T09:12:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae8b400700 1 -- 172.21.15.89:0/4035775640 reap_dead start 2024-04-02T09:12:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.914+0000 7fae8aa00700 1 --2- 172.21.15.89:0/4035775640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae70003a00 0x7fae70003de0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fae7c009b60 tx=0x7fae70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb662aab7361273 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:21.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.915+0000 7fae89600700 1 -- 172.21.15.89:0/4035775640 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c00e6c0 con 0x7fae70003a00 2024-04-02T09:12:21.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.915+0000 7fae89600700 1 -- 172.21.15.89:0/4035775640 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7c00e820 con 0x7fae70003a00 2024-04-02T09:12:21.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.915+0000 7fae89600700 1 -- 172.21.15.89:0/4035775640 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c00eb90 con 0x7fae70003a00 2024-04-02T09:12:21.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.915+0000 7fae91c00700 1 -- 172.21.15.89:0/4035775640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae70003a00 msgr2=0x7fae70003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:21.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.915+0000 7fae91c00700 1 --2- 172.21.15.89:0/4035775640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae70003a00 0x7fae70003de0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fae7c009b60 tx=0x7fae70005880 comp rx=0 tx=0).stop 2024-04-02T09:12:21.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.915+0000 7fae91c00700 1 -- 172.21.15.89:0/4035775640 shutdown_connections 2024-04-02T09:12:21.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.915+0000 7fae91c00700 1 --2- 172.21.15.89:0/4035775640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae70003a00 0x7fae70003de0 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.915+0000 7fae91c00700 1 -- 172.21.15.89:0/4035775640 >> 172.21.15.89:0/4035775640 conn(0x7fae8c08aff0 msgr2=0x7fae8c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:21.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.915+0000 7fae91c00700 1 -- 172.21.15.89:0/4035775640 shutdown_connections 2024-04-02T09:12:21.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.915+0000 7fae91c00700 1 -- 172.21.15.89:0/4035775640 wait complete. 2024-04-02T09:12:21.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.917+0000 7fae91c00700 1 Processor -- start 2024-04-02T09:12:21.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.917+0000 7fae91c00700 1 -- start start 2024-04-02T09:12:21.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.918+0000 7fae91c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae70003a00 0x7fae8c15e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:21.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.918+0000 7fae91c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae8c0b6170 0x7fae8c160630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:21.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.918+0000 7fae91c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae8c0b4690 0x7fae8c14e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:21.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.918+0000 7fae91c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c15ed00 con 0x7fae70003a00 2024-04-02T09:12:21.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.918+0000 7fae91c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c15f340 con 0x7fae8c0b4690 2024-04-02T09:12:21.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.918+0000 7fae91c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c1601a0 con 0x7fae8c0b6170 2024-04-02T09:12:21.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.918+0000 7fae8aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae70003a00 0x7fae8c15e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:21.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.918+0000 7fae8aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae70003a00 0x7fae8c15e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52592/0 (socket says 172.21.15.89:52592) 2024-04-02T09:12:21.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.919+0000 7fae8aa00700 1 -- 172.21.15.89:0/3702742532 learned_addr learned my addr 172.21.15.89:0/3702742532 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:21.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.919+0000 7fae8a000700 1 --2- 172.21.15.89:0/3702742532 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae8c0b6170 0x7fae8c160630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:21.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.919+0000 7fae8b400700 1 --2- 172.21.15.89:0/3702742532 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae8c0b4690 0x7fae8c14e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:21.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.919+0000 7fae8aa00700 1 -- 172.21.15.89:0/3702742532 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae8c0b6170 msgr2=0x7fae8c160630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:21.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.919+0000 7fae8aa00700 1 --2- 172.21.15.89:0/3702742532 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae8c0b6170 0x7fae8c160630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.919+0000 7fae8aa00700 1 -- 172.21.15.89:0/3702742532 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae8c0b4690 msgr2=0x7fae8c14e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:21.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.919+0000 7fae8aa00700 1 --2- 172.21.15.89:0/3702742532 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae8c0b4690 0x7fae8c14e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:21.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.919+0000 7fae8aa00700 1 -- 172.21.15.89:0/3702742532 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c160b70 con 0x7fae70003a00 2024-04-02T09:12:21.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.919+0000 7fae8aa00700 1 --2- 172.21.15.89:0/3702742532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae70003a00 0x7fae8c15e7c0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00a010 tx=0x7fae7c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:21.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.919+0000 7fae82a00700 1 -- 172.21.15.89:0/3702742532 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c00e6c0 con 0x7fae70003a00 2024-04-02T09:12:21.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.919+0000 7fae82a00700 1 -- 172.21.15.89:0/3702742532 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7c003ad0 con 0x7fae70003a00 2024-04-02T09:12:21.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.919+0000 7fae82a00700 1 -- 172.21.15.89:0/3702742532 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c0045d0 con 0x7fae70003a00 2024-04-02T09:12:21.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.919+0000 7fae91c00700 1 -- 172.21.15.89:0/3702742532 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8c15f590 con 0x7fae70003a00 2024-04-02T09:12:21.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.920+0000 7fae91c00700 1 -- 172.21.15.89:0/3702742532 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c15fb50 con 0x7fae70003a00 2024-04-02T09:12:21.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.920+0000 7fae91c00700 1 -- 172.21.15.89:0/3702742532 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae8c160480 con 0x7fae70003a00 2024-04-02T09:12:21.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.921+0000 7fae82a00700 1 -- 172.21.15.89:0/3702742532 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fae7c013470 con 0x7fae70003a00 2024-04-02T09:12:21.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.922+0000 7fae82a00700 1 --2- 172.21.15.89:0/3702742532 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fae6c06adb0 0x7fae6c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:21.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.922+0000 7fae82a00700 1 -- 172.21.15.89:0/3702742532 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fae7c08aa40 con 0x7fae70003a00 2024-04-02T09:12:21.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.923+0000 7fae8a000700 1 --2- 172.21.15.89:0/3702742532 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fae6c06adb0 0x7fae6c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:21.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.923+0000 7fae8a000700 1 --2- 172.21.15.89:0/3702742532 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fae6c06adb0 0x7fae6c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0b5bb0 tx=0x7fae78009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:21.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:21.927+0000 7fae82a00700 1 -- 172.21.15.89:0/3702742532 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae7c08ce60 con 0x7fae70003a00 2024-04-02T09:12:22.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.127+0000 7fae91c00700 1 -- 172.21.15.89:0/3702742532 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fae8c067720 con 0x7fae70003a00 2024-04-02T09:12:22.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.127+0000 7fae82a00700 1 -- 172.21.15.89:0/3702742532 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v219) v1 ==== 93+0+10 (secure 0 0 0) 0x7fae7c0170a0 con 0x7fae70003a00 2024-04-02T09:12:22.126 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:12:22.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.130+0000 7fae91c00700 1 -- 172.21.15.89:0/3702742532 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fae6c06adb0 msgr2=0x7fae6c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:22.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.130+0000 7fae91c00700 1 --2- 172.21.15.89:0/3702742532 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fae6c06adb0 0x7fae6c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0b5bb0 tx=0x7fae78009040 comp rx=0 tx=0).stop 2024-04-02T09:12:22.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.130+0000 7fae91c00700 1 -- 172.21.15.89:0/3702742532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae70003a00 msgr2=0x7fae8c15e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:22.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.130+0000 7fae91c00700 1 --2- 172.21.15.89:0/3702742532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae70003a00 0x7fae8c15e7c0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00a010 tx=0x7fae7c0047c0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.130+0000 7fae91c00700 1 -- 172.21.15.89:0/3702742532 shutdown_connections 2024-04-02T09:12:22.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.130+0000 7fae91c00700 1 --2- 172.21.15.89:0/3702742532 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fae6c06adb0 0x7fae6c06d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.130+0000 7fae91c00700 1 --2- 172.21.15.89:0/3702742532 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae8c0b4690 0x7fae8c14e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.130+0000 7fae91c00700 1 --2- 172.21.15.89:0/3702742532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae70003a00 0x7fae8c15e7c0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.130+0000 7fae91c00700 1 --2- 172.21.15.89:0/3702742532 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae8c0b6170 0x7fae8c160630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.130+0000 7fae91c00700 1 -- 172.21.15.89:0/3702742532 >> 172.21.15.89:0/3702742532 conn(0x7fae8c08aff0 msgr2=0x7fae8c0b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:22.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.130+0000 7fae91c00700 1 -- 172.21.15.89:0/3702742532 shutdown_connections 2024-04-02T09:12:22.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.130+0000 7fae91c00700 1 -- 172.21.15.89:0/3702742532 wait complete. 2024-04-02T09:12:22.146 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T09:12:22.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.340+0000 7fa72e600700 1 Processor -- start 2024-04-02T09:12:22.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.340+0000 7fa72e600700 1 -- start start 2024-04-02T09:12:22.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.341+0000 7fa72e600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa728160be0 0x7fa7280cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:22.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.341+0000 7fa72e600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa728160fc0 con 0x7fa7280ceca0 2024-04-02T09:12:22.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.341+0000 7fa72e600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7280d21e0 con 0x7fa7280ce0e0 2024-04-02T09:12:22.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.341+0000 7fa72e600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7280b65d0 con 0x7fa728160be0 2024-04-02T09:12:22.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.341+0000 7fa727e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa7280ceca0 0x7fa7280b6a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49884/0 (socket says 172.21.15.89:49884) 2024-04-02T09:12:22.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.341+0000 7fa727e00700 1 -- 172.21.15.89:0/1008452152 learned_addr learned my addr 172.21.15.89:0/1008452152 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:22.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.342+0000 7fa727400700 1 --2- 172.21.15.89:0/1008452152 >> v2:172.21.15.113:3300/0 conn(0x7fa728160be0 0x7fa7280cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:22.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.342+0000 7fa726000700 1 -- 172.21.15.89:0/1008452152 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656998477 0 0) 0x7fa728160fc0 con 0x7fa7280ceca0 2024-04-02T09:12:22.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.342+0000 7fa726000700 1 -- 172.21.15.89:0/1008452152 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa70c0035b0 con 0x7fa7280ceca0 2024-04-02T09:12:22.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.342+0000 7fa727400700 1 -- 172.21.15.89:0/1008452152 >> v1:172.21.15.113:6789/0 conn(0x7fa7280ce0e0 legacy=0x7fa7280ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.342+0000 7fa726000700 1 -- 172.21.15.89:0/1008452152 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440885005 0 0) 0x7fa7280d21e0 con 0x7fa7280ce0e0 2024-04-02T09:12:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.342+0000 7fa727400700 1 -- 172.21.15.89:0/1008452152 >> v1:172.21.15.89:6789/0 conn(0x7fa7280ceca0 legacy=0x7fa7280b6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.342+0000 7fa727400700 1 -- 172.21.15.89:0/1008452152 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7280d24c0 con 0x7fa728160be0 2024-04-02T09:12:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.342+0000 7fa727400700 1 --2- 172.21.15.89:0/1008452152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa728160be0 0x7fa7280cdba0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa718009840 tx=0x7fa718009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f7d9cad617d7b33 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.343+0000 7fa726000700 1 -- 172.21.15.89:0/1008452152 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa718011830 con 0x7fa728160be0 2024-04-02T09:12:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.343+0000 7fa726000700 1 -- 172.21.15.89:0/1008452152 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa718011990 con 0x7fa728160be0 2024-04-02T09:12:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.343+0000 7fa726000700 1 -- 172.21.15.89:0/1008452152 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa718011d20 con 0x7fa728160be0 2024-04-02T09:12:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.343+0000 7fa72e600700 1 -- 172.21.15.89:0/1008452152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa728160be0 msgr2=0x7fa7280cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.343+0000 7fa72e600700 1 --2- 172.21.15.89:0/1008452152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa728160be0 0x7fa7280cdba0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa718009840 tx=0x7fa718009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.343+0000 7fa72e600700 1 -- 172.21.15.89:0/1008452152 shutdown_connections 2024-04-02T09:12:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.343+0000 7fa72e600700 1 --2- 172.21.15.89:0/1008452152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa728160be0 0x7fa7280cdba0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.343+0000 7fa72e600700 1 -- 172.21.15.89:0/1008452152 >> 172.21.15.89:0/1008452152 conn(0x7fa72808aff0 msgr2=0x7fa7280d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.343+0000 7fa72e600700 1 -- 172.21.15.89:0/1008452152 shutdown_connections 2024-04-02T09:12:22.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.343+0000 7fa72e600700 1 -- 172.21.15.89:0/1008452152 wait complete. 2024-04-02T09:12:22.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.345+0000 7fa72e600700 1 Processor -- start 2024-04-02T09:12:22.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.345+0000 7fa72e600700 1 -- start start 2024-04-02T09:12:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.346+0000 7fa72e600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7280ce0e0 0x7fa7281f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.346+0000 7fa72e600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7280ceca0 0x7fa728150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.346+0000 7fa72e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa728160be0 0x7fa728150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.346+0000 7fa72e600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7281f42b0 con 0x7fa7280ce0e0 2024-04-02T09:12:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.346+0000 7fa72e600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7280c8300 con 0x7fa7280ceca0 2024-04-02T09:12:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.346+0000 7fa72e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7281f3a10 con 0x7fa728160be0 2024-04-02T09:12:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.346+0000 7fa727400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7280ce0e0 0x7fa7281f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.346+0000 7fa727400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7280ce0e0 0x7fa7281f2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52602/0 (socket says 172.21.15.89:52602) 2024-04-02T09:12:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.346+0000 7fa727400700 1 -- 172.21.15.89:0/3691575883 learned_addr learned my addr 172.21.15.89:0/3691575883 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.346+0000 7fa727e00700 1 --2- 172.21.15.89:0/3691575883 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa728160be0 0x7fa728150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.346+0000 7fa726a00700 1 --2- 172.21.15.89:0/3691575883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7280ceca0 0x7fa728150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.347+0000 7fa727400700 1 -- 172.21.15.89:0/3691575883 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa728160be0 msgr2=0x7fa728150e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.347+0000 7fa727400700 1 --2- 172.21.15.89:0/3691575883 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa728160be0 0x7fa728150e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.347+0000 7fa727400700 1 -- 172.21.15.89:0/3691575883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7280ceca0 msgr2=0x7fa728150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.347+0000 7fa727400700 1 --2- 172.21.15.89:0/3691575883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7280ceca0 0x7fa728150950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.347+0000 7fa727400700 1 -- 172.21.15.89:0/3691575883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7180094a0 con 0x7fa7280ce0e0 2024-04-02T09:12:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.347+0000 7fa727400700 1 --2- 172.21.15.89:0/3691575883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7280ce0e0 0x7fa7281f2ff0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fa7180090b0 tx=0x7fa718004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.347+0000 7fa71f400700 1 -- 172.21.15.89:0/3691575883 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71801a910 con 0x7fa7280ce0e0 2024-04-02T09:12:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.347+0000 7fa71f400700 1 -- 172.21.15.89:0/3691575883 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa71801aa70 con 0x7fa7280ce0e0 2024-04-02T09:12:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.347+0000 7fa72e600700 1 -- 172.21.15.89:0/3691575883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7281f3cf0 con 0x7fa7280ce0e0 2024-04-02T09:12:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.347+0000 7fa71f400700 1 -- 172.21.15.89:0/3691575883 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71801ae00 con 0x7fa7280ce0e0 2024-04-02T09:12:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.347+0000 7fa72e600700 1 -- 172.21.15.89:0/3691575883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7281f3530 con 0x7fa7280ce0e0 2024-04-02T09:12:22.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.348+0000 7fa72e600700 1 -- 172.21.15.89:0/3691575883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7280c80d0 con 0x7fa7280ce0e0 2024-04-02T09:12:22.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.349+0000 7fa71f400700 1 -- 172.21.15.89:0/3691575883 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa718022070 con 0x7fa7280ce0e0 2024-04-02T09:12:22.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.350+0000 7fa71f400700 1 --2- 172.21.15.89:0/3691575883 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa70806adb0 0x7fa70806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:22.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.350+0000 7fa71f400700 1 -- 172.21.15.89:0/3691575883 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fa71808a050 con 0x7fa7280ce0e0 2024-04-02T09:12:22.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.350+0000 7fa726a00700 1 --2- 172.21.15.89:0/3691575883 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa70806adb0 0x7fa70806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:22.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.351+0000 7fa726a00700 1 --2- 172.21.15.89:0/3691575883 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa70806adb0 0x7fa70806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa72806b7a0 tx=0x7fa714009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:22.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.355+0000 7fa71f400700 1 -- 172.21.15.89:0/3691575883 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa7280c80d0 con 0x7fa7280ce0e0 2024-04-02T09:12:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.556+0000 7fa72e600700 1 -- 172.21.15.89:0/3691575883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa7280c80d0 con 0x7fa7280ce0e0 2024-04-02T09:12:22.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.556+0000 7fa71f400700 1 -- 172.21.15.89:0/3691575883 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v219) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa7280c80d0 con 0x7fa7280ce0e0 2024-04-02T09:12:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:22.555 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T09:12:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T09:12:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T09:12:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:22.555 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T09:12:22.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.559+0000 7fa72e600700 1 -- 172.21.15.89:0/3691575883 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa70806adb0 msgr2=0x7fa70806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:22.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.559+0000 7fa72e600700 1 --2- 172.21.15.89:0/3691575883 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa70806adb0 0x7fa70806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa72806b7a0 tx=0x7fa714009040 comp rx=0 tx=0).stop 2024-04-02T09:12:22.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.559+0000 7fa72e600700 1 -- 172.21.15.89:0/3691575883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7280ce0e0 msgr2=0x7fa7281f2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:22.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.559+0000 7fa72e600700 1 --2- 172.21.15.89:0/3691575883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7280ce0e0 0x7fa7281f2ff0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fa7180090b0 tx=0x7fa718004750 comp rx=0 tx=0).stop 2024-04-02T09:12:22.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.559+0000 7fa72e600700 1 -- 172.21.15.89:0/3691575883 shutdown_connections 2024-04-02T09:12:22.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.559+0000 7fa72e600700 1 --2- 172.21.15.89:0/3691575883 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa728160be0 0x7fa728150e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.559+0000 7fa72e600700 1 --2- 172.21.15.89:0/3691575883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7280ce0e0 0x7fa7281f2ff0 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.559+0000 7fa72e600700 1 --2- 172.21.15.89:0/3691575883 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa70806adb0 0x7fa70806d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.559+0000 7fa72e600700 1 --2- 172.21.15.89:0/3691575883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7280ceca0 0x7fa728150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.559+0000 7fa72e600700 1 -- 172.21.15.89:0/3691575883 >> 172.21.15.89:0/3691575883 conn(0x7fa72808aff0 msgr2=0x7fa7280b8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:22.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.559+0000 7fa72e600700 1 -- 172.21.15.89:0/3691575883 shutdown_connections 2024-04-02T09:12:22.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.559+0000 7fa72e600700 1 -- 172.21.15.89:0/3691575883 wait complete. 2024-04-02T09:12:22.577 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T09:12:22.577 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T09:12:22.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.774+0000 7fa6a0a00700 1 Processor -- start 2024-04-02T09:12:22.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.774+0000 7fa6a0a00700 1 -- start start 2024-04-02T09:12:22.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.775+0000 7fa6a0a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa69c1580c0 0x7fa69c06e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:22.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.775+0000 7fa6a0a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69c070980 con 0x7fa69c06e7f0 2024-04-02T09:12:22.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.775+0000 7fa6a0a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69c15b760 con 0x7fa69c15b330 2024-04-02T09:12:22.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.775+0000 7fa6a0a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c071570 con 0x7fa69c1580c0 2024-04-02T09:12:22.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.775+0000 7fa699400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa69c06e7f0 0x7fa69c15ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49898/0 (socket says 172.21.15.89:49898) 2024-04-02T09:12:22.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.775+0000 7fa699400700 1 -- 172.21.15.89:0/183574344 learned_addr learned my addr 172.21.15.89:0/183574344 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:22.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.776+0000 7fa699e00700 1 --2- 172.21.15.89:0/183574344 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa69c1580c0 0x7fa69c06e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.776+0000 7fa698a00700 1 -- 172.21.15.89:0/183574344 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070047787 0 0) 0x7fa69c070980 con 0x7fa69c06e7f0 2024-04-02T09:12:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.776+0000 7fa698a00700 1 -- 172.21.15.89:0/183574344 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6800035b0 con 0x7fa69c06e7f0 2024-04-02T09:12:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.776+0000 7fa698a00700 1 -- 172.21.15.89:0/183574344 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647284086 0 0) 0x7fa69c15b760 con 0x7fa69c15b330 2024-04-02T09:12:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.776+0000 7fa698a00700 1 -- 172.21.15.89:0/183574344 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa69c070980 con 0x7fa69c15b330 2024-04-02T09:12:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.776+0000 7fa699e00700 1 -- 172.21.15.89:0/183574344 >> v1:172.21.15.113:6789/0 conn(0x7fa69c15b330 legacy=0x7fa69c0b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.776+0000 7fa699e00700 1 -- 172.21.15.89:0/183574344 >> v1:172.21.15.89:6789/0 conn(0x7fa69c06e7f0 legacy=0x7fa69c15ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.776+0000 7fa699e00700 1 -- 172.21.15.89:0/183574344 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c070bd0 con 0x7fa69c1580c0 2024-04-02T09:12:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.776+0000 7fa699e00700 1 --2- 172.21.15.89:0/183574344 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa69c1580c0 0x7fa69c06e2b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009870 tx=0x7fa68c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e4303c2ea4d363e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.777+0000 7fa698a00700 1 -- 172.21.15.89:0/183574344 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c011830 con 0x7fa69c1580c0 2024-04-02T09:12:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.777+0000 7fa698a00700 1 -- 172.21.15.89:0/183574344 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68c011990 con 0x7fa69c1580c0 2024-04-02T09:12:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.777+0000 7fa698a00700 1 -- 172.21.15.89:0/183574344 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c011d20 con 0x7fa69c1580c0 2024-04-02T09:12:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.777+0000 7fa6a0a00700 1 -- 172.21.15.89:0/183574344 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa69c1580c0 msgr2=0x7fa69c06e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:22.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.777+0000 7fa6a0a00700 1 --2- 172.21.15.89:0/183574344 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa69c1580c0 0x7fa69c06e2b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009870 tx=0x7fa68c009b80 comp rx=0 tx=0).stop 2024-04-02T09:12:22.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.777+0000 7fa6a0a00700 1 -- 172.21.15.89:0/183574344 shutdown_connections 2024-04-02T09:12:22.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.777+0000 7fa6a0a00700 1 --2- 172.21.15.89:0/183574344 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa69c1580c0 0x7fa69c06e2b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.777+0000 7fa6a0a00700 1 -- 172.21.15.89:0/183574344 >> 172.21.15.89:0/183574344 conn(0x7fa69c0cf210 msgr2=0x7fa69c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:22.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.777+0000 7fa6a0a00700 1 -- 172.21.15.89:0/183574344 shutdown_connections 2024-04-02T09:12:22.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.777+0000 7fa6a0a00700 1 -- 172.21.15.89:0/183574344 wait complete. 2024-04-02T09:12:22.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.779+0000 7fa6a0a00700 1 Processor -- start 2024-04-02T09:12:22.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.779+0000 7fa6a0a00700 1 -- start start 2024-04-02T09:12:22.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.780+0000 7fa6a0a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa69c06e7f0 0x7fa69c099f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:22.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.780+0000 7fa6a0a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa69c1580c0 0x7fa69c09cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:22.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.780+0000 7fa6a0a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa69c15b330 0x7fa69c157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:22.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.780+0000 7fa6a0a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c071990 con 0x7fa69c06e7f0 2024-04-02T09:12:22.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.780+0000 7fa6a0a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c0ce2f0 con 0x7fa69c15b330 2024-04-02T09:12:22.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.780+0000 7fa6a0a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c0cdd30 con 0x7fa69c1580c0 2024-04-02T09:12:22.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.780+0000 7fa699e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa69c06e7f0 0x7fa69c099f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:22.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.780+0000 7fa699e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa69c06e7f0 0x7fa69c099f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52604/0 (socket says 172.21.15.89:52604) 2024-04-02T09:12:22.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.780+0000 7fa699e00700 1 -- 172.21.15.89:0/1169120088 learned_addr learned my addr 172.21.15.89:0/1169120088 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:22.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.781+0000 7fa699400700 1 --2- 172.21.15.89:0/1169120088 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa69c1580c0 0x7fa69c09cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:22.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.781+0000 7fa69a800700 1 --2- 172.21.15.89:0/1169120088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa69c15b330 0x7fa69c157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:22.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.781+0000 7fa699e00700 1 -- 172.21.15.89:0/1169120088 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa69c1580c0 msgr2=0x7fa69c09cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:22.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.781+0000 7fa699e00700 1 --2- 172.21.15.89:0/1169120088 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa69c1580c0 0x7fa69c09cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.781+0000 7fa699e00700 1 -- 172.21.15.89:0/1169120088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa69c15b330 msgr2=0x7fa69c157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:22.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.781+0000 7fa699e00700 1 --2- 172.21.15.89:0/1169120088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa69c15b330 0x7fa69c157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:22.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.781+0000 7fa699e00700 1 -- 172.21.15.89:0/1169120088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68c0094a0 con 0x7fa69c06e7f0 2024-04-02T09:12:22.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.781+0000 7fa699e00700 1 --2- 172.21.15.89:0/1169120088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa69c06e7f0 0x7fa69c099f10 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fa68c0090b0 tx=0x7fa68c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:22.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.781+0000 7fa692000700 1 -- 172.21.15.89:0/1169120088 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c01a860 con 0x7fa69c06e7f0 2024-04-02T09:12:22.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.781+0000 7fa692000700 1 -- 172.21.15.89:0/1169120088 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68c01a9c0 con 0x7fa69c06e7f0 2024-04-02T09:12:22.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.781+0000 7fa6a0a00700 1 -- 172.21.15.89:0/1169120088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa69c0ce010 con 0x7fa69c06e7f0 2024-04-02T09:12:22.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.781+0000 7fa692000700 1 -- 172.21.15.89:0/1169120088 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c01ad30 con 0x7fa69c06e7f0 2024-04-02T09:12:22.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.781+0000 7fa6a0a00700 1 -- 172.21.15.89:0/1169120088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa69c0cbd10 con 0x7fa69c06e7f0 2024-04-02T09:12:22.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.782+0000 7fa6a0a00700 1 -- 172.21.15.89:0/1169120088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69c067720 con 0x7fa69c06e7f0 2024-04-02T09:12:22.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.783+0000 7fa692000700 1 -- 172.21.15.89:0/1169120088 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa68c022070 con 0x7fa69c06e7f0 2024-04-02T09:12:22.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.784+0000 7fa692000700 1 --2- 172.21.15.89:0/1169120088 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa67c06adb0 0x7fa67c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:22.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.784+0000 7fa692000700 1 -- 172.21.15.89:0/1169120088 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fa68c08a4b0 con 0x7fa69c06e7f0 2024-04-02T09:12:22.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.785+0000 7fa699400700 1 --2- 172.21.15.89:0/1169120088 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa67c06adb0 0x7fa67c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:22.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.785+0000 7fa699400700 1 --2- 172.21.15.89:0/1169120088 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa67c06adb0 0x7fa67c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa69c1f2910 tx=0x7fa688009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:22.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.788+0000 7fa692000700 1 -- 172.21.15.89:0/1169120088 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa68c055700 con 0x7fa69c06e7f0 2024-04-02T09:12:22.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:22.990+0000 7fa6a0a00700 1 -- 172.21.15.89:0/1169120088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa69c071d00 con 0x7fa69c06e7f0 2024-04-02T09:12:23.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.670+0000 7fa692000700 1 -- 172.21.15.89:0/1169120088 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v220) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa68c0150b0 con 0x7fa69c06e7f0 2024-04-02T09:12:23.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.675+0000 7fa6a0a00700 1 -- 172.21.15.89:0/1169120088 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa67c06adb0 msgr2=0x7fa67c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:23.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.675+0000 7fa6a0a00700 1 --2- 172.21.15.89:0/1169120088 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa67c06adb0 0x7fa67c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa69c1f2910 tx=0x7fa688009040 comp rx=0 tx=0).stop 2024-04-02T09:12:23.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.675+0000 7fa6a0a00700 1 -- 172.21.15.89:0/1169120088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa69c06e7f0 msgr2=0x7fa69c099f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:23.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.675+0000 7fa6a0a00700 1 --2- 172.21.15.89:0/1169120088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa69c06e7f0 0x7fa69c099f10 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fa68c0090b0 tx=0x7fa68c004750 comp rx=0 tx=0).stop 2024-04-02T09:12:23.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.675+0000 7fa6a0a00700 1 -- 172.21.15.89:0/1169120088 shutdown_connections 2024-04-02T09:12:23.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.675+0000 7fa6a0a00700 1 --2- 172.21.15.89:0/1169120088 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa67c06adb0 0x7fa67c06d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:23.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.675+0000 7fa6a0a00700 1 --2- 172.21.15.89:0/1169120088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa69c15b330 0x7fa69c157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:23.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.675+0000 7fa6a0a00700 1 --2- 172.21.15.89:0/1169120088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa69c06e7f0 0x7fa69c099f10 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:23.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.675+0000 7fa6a0a00700 1 --2- 172.21.15.89:0/1169120088 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa69c1580c0 0x7fa69c09cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:23.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.675+0000 7fa6a0a00700 1 -- 172.21.15.89:0/1169120088 >> 172.21.15.89:0/1169120088 conn(0x7fa69c0cf210 msgr2=0x7fa69c0724e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:23.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.676+0000 7fa6a0a00700 1 -- 172.21.15.89:0/1169120088 shutdown_connections 2024-04-02T09:12:23.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.677+0000 7fa6a0a00700 1 -- 172.21.15.89:0/1169120088 wait complete. 2024-04-02T09:12:23.677 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T09:12:23.696 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T09:12:23.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.898+0000 7f5de2c00700 1 Processor -- start 2024-04-02T09:12:23.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.898+0000 7f5de2c00700 1 -- start start 2024-04-02T09:12:23.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.899+0000 7f5de2c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5ddc0ce230 0x7f5ddc0ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:23.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.899+0000 7f5de2c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ddc0b2070 0x7f5ddc0b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:23.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.899+0000 7f5de2c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ddc0709c0 con 0x7f5ddc15d800 2024-04-02T09:12:23.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.899+0000 7f5de2c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ddc0b40f0 con 0x7f5ddc0ce230 2024-04-02T09:12:23.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.899+0000 7f5de2c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc070c80 con 0x7f5ddc0b2070 2024-04-02T09:12:23.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.899+0000 7f5ddb400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5ddc0ce230 0x7f5ddc0ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.899+0000 7f5de0c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ddc0b2070 0x7f5ddc0b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.899+0000 7f5ddbe00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f5ddc15d800 0x7f5ddc0725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:47114/0 (socket says 172.21.15.89:47114) 2024-04-02T09:12:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.899+0000 7f5ddbe00700 1 -- 172.21.15.89:0/2470965877 learned_addr learned my addr 172.21.15.89:0/2470965877 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.900+0000 7f5de0c00700 1 -- 172.21.15.89:0/2470965877 >> v2:172.21.15.113:3300/0 conn(0x7f5ddc0ce230 msgr2=0x7f5ddc0ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.900+0000 7f5de0c00700 1 --2- 172.21.15.89:0/2470965877 >> v2:172.21.15.113:3300/0 conn(0x7f5ddc0ce230 0x7f5ddc0ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.900+0000 7f5de0c00700 1 -- 172.21.15.89:0/2470965877 >> v1:172.21.15.113:6789/0 conn(0x7f5ddc15d800 legacy=0x7f5ddc0725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.900+0000 7f5de0c00700 1 -- 172.21.15.89:0/2470965877 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ddc070f60 con 0x7f5ddc0b2070 2024-04-02T09:12:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.900+0000 7f5ddb400700 1 --2- 172.21.15.89:0/2470965877 >> v2:172.21.15.113:3300/0 conn(0x7f5ddc0ce230 0x7f5ddc0ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:12:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.900+0000 7f5de0c00700 1 --2- 172.21.15.89:0/2470965877 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ddc0b2070 0x7f5ddc0b2500 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400aea0 tx=0x7f5dd400c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89a33618cdde854e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.900+0000 7f5ddaa00700 1 -- 172.21.15.89:0/2470965877 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd400f070 con 0x7f5ddc0b2070 2024-04-02T09:12:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.900+0000 7f5ddaa00700 1 -- 172.21.15.89:0/2470965877 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd400a2e0 con 0x7f5ddc0b2070 2024-04-02T09:12:23.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.900+0000 7f5ddaa00700 1 -- 172.21.15.89:0/2470965877 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd400cb50 con 0x7f5ddc0b2070 2024-04-02T09:12:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.901+0000 7f5de2c00700 1 -- 172.21.15.89:0/2470965877 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ddc0b2070 msgr2=0x7f5ddc0b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.901+0000 7f5de2c00700 1 --2- 172.21.15.89:0/2470965877 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ddc0b2070 0x7f5ddc0b2500 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400aea0 tx=0x7f5dd400c330 comp rx=0 tx=0).stop 2024-04-02T09:12:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.901+0000 7f5de2c00700 1 -- 172.21.15.89:0/2470965877 shutdown_connections 2024-04-02T09:12:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.901+0000 7f5de2c00700 1 --2- 172.21.15.89:0/2470965877 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ddc0b2070 0x7f5ddc0b2500 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:23.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.901+0000 7f5de2c00700 1 --2- 172.21.15.89:0/2470965877 >> v2:172.21.15.113:3300/0 conn(0x7f5ddc0ce230 0x7f5ddc0ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:23.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.901+0000 7f5de2c00700 1 -- 172.21.15.89:0/2470965877 >> 172.21.15.89:0/2470965877 conn(0x7f5ddc0b0d70 msgr2=0x7f5ddc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:23.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.901+0000 7f5de2c00700 1 -- 172.21.15.89:0/2470965877 shutdown_connections 2024-04-02T09:12:23.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.901+0000 7f5de2c00700 1 -- 172.21.15.89:0/2470965877 wait complete. 2024-04-02T09:12:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.903+0000 7f5de2c00700 1 Processor -- start 2024-04-02T09:12:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.903+0000 7f5de2c00700 1 -- start start 2024-04-02T09:12:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.904+0000 7f5de2c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ddc0b2070 0x7f5ddc153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.904+0000 7f5de2c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ddc0ce230 0x7f5ddc0cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.904+0000 7f5de2c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ddc15d800 0x7f5ddc15c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.904+0000 7f5de2c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc153c40 con 0x7f5ddc0ce230 2024-04-02T09:12:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.904+0000 7f5de2c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc0c83f0 con 0x7f5ddc15d800 2024-04-02T09:12:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.904+0000 7f5de2c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc0c96e0 con 0x7f5ddc0b2070 2024-04-02T09:12:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.904+0000 7f5ddb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ddc0ce230 0x7f5ddc0cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:23.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.904+0000 7f5ddb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ddc0ce230 0x7f5ddc0cac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52620/0 (socket says 172.21.15.89:52620) 2024-04-02T09:12:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.904+0000 7f5ddb400700 1 -- 172.21.15.89:0/2388661534 learned_addr learned my addr 172.21.15.89:0/2388661534 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.904+0000 7f5ddbe00700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ddc0b2070 0x7f5ddc153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.904+0000 7f5de0c00700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ddc15d800 0x7f5ddc15c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.905+0000 7f5ddb400700 1 -- 172.21.15.89:0/2388661534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ddc0b2070 msgr2=0x7f5ddc153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.905+0000 7f5ddb400700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ddc0b2070 0x7f5ddc153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.905+0000 7f5ddb400700 1 -- 172.21.15.89:0/2388661534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ddc15d800 msgr2=0x7f5ddc15c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.905+0000 7f5ddb400700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ddc15d800 0x7f5ddc15c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.905+0000 7f5ddb400700 1 -- 172.21.15.89:0/2388661534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc80094a0 con 0x7f5ddc0ce230 2024-04-02T09:12:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.905+0000 7f5de0c00700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ddc15d800 0x7f5ddc15c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.905+0000 7f5ddbe00700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ddc0b2070 0x7f5ddc153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.905+0000 7f5ddb400700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ddc0ce230 0x7f5ddc0cac10 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f5dc800ebe0 tx=0x7f5dc8009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.905+0000 7f5dd3e00700 1 -- 172.21.15.89:0/2388661534 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc800c910 con 0x7f5ddc0ce230 2024-04-02T09:12:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.905+0000 7f5dd3e00700 1 -- 172.21.15.89:0/2388661534 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc800ca70 con 0x7f5ddc0ce230 2024-04-02T09:12:23.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.905+0000 7f5de2c00700 1 -- 172.21.15.89:0/2388661534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd400ab80 con 0x7f5ddc0ce230 2024-04-02T09:12:23.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.905+0000 7f5dd3e00700 1 -- 172.21.15.89:0/2388661534 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc800cde0 con 0x7f5ddc0ce230 2024-04-02T09:12:23.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.905+0000 7f5de2c00700 1 -- 172.21.15.89:0/2388661534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ddc0c8920 con 0x7f5ddc0ce230 2024-04-02T09:12:23.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.906+0000 7f5de2c00700 1 -- 172.21.15.89:0/2388661534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ddc0c9530 con 0x7f5ddc0ce230 2024-04-02T09:12:23.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.908+0000 7f5dd3e00700 1 -- 172.21.15.89:0/2388661534 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5dc8013470 con 0x7f5ddc0ce230 2024-04-02T09:12:23.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.909+0000 7f5dd3e00700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dbc06adb0 0x7f5dbc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:23.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.909+0000 7f5dd3e00700 1 -- 172.21.15.89:0/2388661534 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f5dc8089f30 con 0x7f5ddc0ce230 2024-04-02T09:12:23.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.909+0000 7f5ddbe00700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dbc06adb0 0x7f5dbc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:23.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.910+0000 7f5ddbe00700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dbc06adb0 0x7f5dbc06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc007fd0 tx=0x7f5dcc009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:23.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:23.913+0000 7f5dd3e00700 1 -- 172.21.15.89:0/2388661534 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5dc8054f90 con 0x7f5ddc0ce230 2024-04-02T09:12:24.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.113+0000 7f5de2c00700 1 -- 172.21.15.89:0/2388661534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5ddc067720 con 0x7f5ddc0ce230 2024-04-02T09:12:24.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.673+0000 7f5dd3e00700 1 -- 172.21.15.89:0/2388661534 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v221) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5dc80170d0 con 0x7f5ddc0ce230 2024-04-02T09:12:24.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.683+0000 7f5de2c00700 1 -- 172.21.15.89:0/2388661534 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dbc06adb0 msgr2=0x7f5dbc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:24.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.683+0000 7f5de2c00700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dbc06adb0 0x7f5dbc06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc007fd0 tx=0x7f5dcc009040 comp rx=0 tx=0).stop 2024-04-02T09:12:24.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.683+0000 7f5de2c00700 1 -- 172.21.15.89:0/2388661534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ddc0ce230 msgr2=0x7f5ddc0cac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:24.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.683+0000 7f5de2c00700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ddc0ce230 0x7f5ddc0cac10 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f5dc800ebe0 tx=0x7f5dc8009ec0 comp rx=0 tx=0).stop 2024-04-02T09:12:24.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.684+0000 7f5de2c00700 1 -- 172.21.15.89:0/2388661534 shutdown_connections 2024-04-02T09:12:24.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.684+0000 7f5de2c00700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dbc06adb0 0x7f5dbc06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:24.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.684+0000 7f5de2c00700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ddc15d800 0x7f5ddc15c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:24.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.684+0000 7f5de2c00700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ddc0b2070 0x7f5ddc153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:24.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.684+0000 7f5de2c00700 1 --2- 172.21.15.89:0/2388661534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ddc0ce230 0x7f5ddc0cac10 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:24.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.684+0000 7f5de2c00700 1 -- 172.21.15.89:0/2388661534 >> 172.21.15.89:0/2388661534 conn(0x7f5ddc0b0d70 msgr2=0x7f5ddc0cbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:24.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.684+0000 7f5de2c00700 1 -- 172.21.15.89:0/2388661534 shutdown_connections 2024-04-02T09:12:24.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.685+0000 7f5de2c00700 1 -- 172.21.15.89:0/2388661534 wait complete. 2024-04-02T09:12:24.686 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T09:12:24.711 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T09:12:24.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.907+0000 7f6400800700 1 Processor -- start 2024-04-02T09:12:24.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.908+0000 7f6400800700 1 -- start start 2024-04-02T09:12:24.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.909+0000 7f6400800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63f815d8a0 0x7f63f80b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:24.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.909+0000 7f6400800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f63f80cf1b0 0x7f63f80cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:24.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.909+0000 7f6400800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f80ced20 con 0x7f63f80d3eb0 2024-04-02T09:12:24.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.909+0000 7f6400800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f80d4340 con 0x7f63f80cf1b0 2024-04-02T09:12:24.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.909+0000 7f6400800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80ce6d0 con 0x7f63f815d8a0 2024-04-02T09:12:24.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.909+0000 7f63fd400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f63f80d3eb0 0x7f63f806b890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49906/0 (socket says 172.21.15.89:49906) 2024-04-02T09:12:24.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.909+0000 7f63fd400700 1 -- 172.21.15.89:0/2663880807 learned_addr learned my addr 172.21.15.89:0/2663880807 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:24.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.909+0000 7f63fe800700 1 --2- 172.21.15.89:0/2663880807 >> v2:172.21.15.113:3300/0 conn(0x7f63f80cf1b0 0x7f63f80cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:24.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.909+0000 7f63fde00700 1 --2- 172.21.15.89:0/2663880807 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63f815d8a0 0x7f63f80b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:24.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.909+0000 7f63fca00700 1 -- 172.21.15.89:0/2663880807 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561964607 0 0) 0x7f63f80ced20 con 0x7f63f80d3eb0 2024-04-02T09:12:24.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 -- 172.21.15.89:0/2663880807 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63e00035b0 con 0x7f63f80d3eb0 2024-04-02T09:12:24.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 -- 172.21.15.89:0/2663880807 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1492799727 0 0) 0x7f63e00035b0 con 0x7f63f80d3eb0 2024-04-02T09:12:24.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 -- 172.21.15.89:0/2663880807 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63f815d8a0 msgr2=0x7f63f80b2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:24.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 --2- 172.21.15.89:0/2663880807 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63f815d8a0 0x7f63f80b2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:24.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 -- 172.21.15.89:0/2663880807 >> v2:172.21.15.113:3300/0 conn(0x7f63f80cf1b0 msgr2=0x7f63f80cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:24.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 --2- 172.21.15.89:0/2663880807 >> v2:172.21.15.113:3300/0 conn(0x7f63f80cf1b0 0x7f63f80cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:24.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 -- 172.21.15.89:0/2663880807 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f80ce9b0 con 0x7f63f80d3eb0 2024-04-02T09:12:24.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 -- 172.21.15.89:0/2663880807 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f63e80017f0 con 0x7f63f80d3eb0 2024-04-02T09:12:24.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 -- 172.21.15.89:0/2663880807 >> v1:172.21.15.89:6789/0 conn(0x7f63f80d3eb0 legacy=0x7f63f806b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:24.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 --2- 172.21.15.89:0/2663880807 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63f815d8a0 0x7f63e0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:24.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 --2- 172.21.15.89:0/2663880807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e0003f80 0x7f63e0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:24.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 --2- 172.21.15.89:0/2663880807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63e0004d90 0x7f63e0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:24.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 -- 172.21.15.89:0/2663880807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80d4340 con 0x7f63e0004d90 2024-04-02T09:12:24.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 -- 172.21.15.89:0/2663880807 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80ce6d0 con 0x7f63e0003f80 2024-04-02T09:12:24.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fca00700 1 -- 172.21.15.89:0/2663880807 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ba10 con 0x7f63f815d8a0 2024-04-02T09:12:24.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fe800700 1 --2- 172.21.15.89:0/2663880807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63e0004d90 0x7f63e0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:24.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fde00700 1 --2- 172.21.15.89:0/2663880807 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63f815d8a0 0x7f63e0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:24.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.910+0000 7f63fd400700 1 --2- 172.21.15.89:0/2663880807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e0003f80 0x7f63e0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:24.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.911+0000 7f63fe800700 1 -- 172.21.15.89:0/2663880807 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63f815d8a0 msgr2=0x7f63e0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:24.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.911+0000 7f63fe800700 1 --2- 172.21.15.89:0/2663880807 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63f815d8a0 0x7f63e0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:24.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.911+0000 7f63fe800700 1 -- 172.21.15.89:0/2663880807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e0003f80 msgr2=0x7f63e0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:24.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.911+0000 7f63fe800700 1 --2- 172.21.15.89:0/2663880807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e0003f80 0x7f63e0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:24.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.911+0000 7f63fe800700 1 -- 172.21.15.89:0/2663880807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e00035b0 con 0x7f63e0004d90 2024-04-02T09:12:24.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.911+0000 7f63fe800700 1 --2- 172.21.15.89:0/2663880807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63e0004d90 0x7f63e0005210 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f63f4009fc0 tx=0x7f63e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1405fbf659aa4ef2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:24.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.911+0000 7f63fca00700 1 -- 172.21.15.89:0/2663880807 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400ebd0 con 0x7f63e0004d90 2024-04-02T09:12:24.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.911+0000 7f63fca00700 1 -- 172.21.15.89:0/2663880807 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400f040 con 0x7f63e0004d90 2024-04-02T09:12:24.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.911+0000 7f63fca00700 1 -- 172.21.15.89:0/2663880807 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4014480 con 0x7f63e0004d90 2024-04-02T09:12:24.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.912+0000 7f6400800700 1 -- 172.21.15.89:0/2663880807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63e0004d90 msgr2=0x7f63e0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:24.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.912+0000 7f6400800700 1 --2- 172.21.15.89:0/2663880807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63e0004d90 0x7f63e0005210 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f63f4009fc0 tx=0x7f63e0005880 comp rx=0 tx=0).stop 2024-04-02T09:12:24.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.912+0000 7f63fe800700 1 -- 172.21.15.89:0/2663880807 reap_dead start 2024-04-02T09:12:24.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.912+0000 7f6400800700 1 -- 172.21.15.89:0/2663880807 shutdown_connections 2024-04-02T09:12:24.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.912+0000 7f6400800700 1 -- 172.21.15.89:0/2663880807 >> 172.21.15.89:0/2663880807 conn(0x7f63f808aff0 msgr2=0x7f63f80d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:24.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.912+0000 7f6400800700 1 -- 172.21.15.89:0/2663880807 shutdown_connections 2024-04-02T09:12:24.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.912+0000 7f6400800700 1 -- 172.21.15.89:0/2663880807 wait complete. 2024-04-02T09:12:24.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.914+0000 7f6400800700 1 Processor -- start 2024-04-02T09:12:24.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.914+0000 7f6400800700 1 -- start start 2024-04-02T09:12:24.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.914+0000 7f6400800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e0004d90 0x7f63f80b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:24.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f6400800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80b7bb0 0x7f63f80b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:24.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f6400800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63f80b5100 0x7f63f80b5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:24.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f6400800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f815a170 con 0x7f63f80b7bb0 2024-04-02T09:12:24.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f6400800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80cb9c0 con 0x7f63e0004d90 2024-04-02T09:12:24.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f6400800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80b36a0 con 0x7f63f80b5100 2024-04-02T09:12:24.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f63fd400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80b7bb0 0x7f63f80b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:24.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f63fd400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80b7bb0 0x7f63f80b4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52624/0 (socket says 172.21.15.89:52624) 2024-04-02T09:12:24.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f63fd400700 1 -- 172.21.15.89:0/2175438656 learned_addr learned my addr 172.21.15.89:0/2175438656 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:24.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f63fde00700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e0004d90 0x7f63f80b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:24.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f63fe800700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63f80b5100 0x7f63f80b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:24.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f63fd400700 1 -- 172.21.15.89:0/2175438656 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63f80b5100 msgr2=0x7f63f80b5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:24.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f63fd400700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63f80b5100 0x7f63f80b5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:24.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f63fd400700 1 -- 172.21.15.89:0/2175438656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e0004d90 msgr2=0x7f63f80b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:24.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f63fd400700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e0004d90 0x7f63f80b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:24.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f63fd400700 1 -- 172.21.15.89:0/2175438656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f80b5ad0 con 0x7f63f80b7bb0 2024-04-02T09:12:24.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f63fde00700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e0004d90 0x7f63f80b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:24.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.915+0000 7f63fe800700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63f80b5100 0x7f63f80b5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:24.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.916+0000 7f63fd400700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80b7bb0 0x7f63f80b4bc0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f63e800fb40 tx=0x7f63e800ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:24.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.916+0000 7f63f2000700 1 -- 172.21.15.89:0/2175438656 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e80072e0 con 0x7f63f80b7bb0 2024-04-02T09:12:24.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.916+0000 7f63f2000700 1 -- 172.21.15.89:0/2175438656 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e8007440 con 0x7f63f80b7bb0 2024-04-02T09:12:24.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.916+0000 7f6400800700 1 -- 172.21.15.89:0/2175438656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f80b8170 con 0x7f63f80b7bb0 2024-04-02T09:12:24.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.916+0000 7f63f2000700 1 -- 172.21.15.89:0/2175438656 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8005830 con 0x7f63f80b7bb0 2024-04-02T09:12:24.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.916+0000 7f6400800700 1 -- 172.21.15.89:0/2175438656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f80b8730 con 0x7f63f80b7bb0 2024-04-02T09:12:24.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.917+0000 7f6400800700 1 -- 172.21.15.89:0/2175438656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f8067720 con 0x7f63f80b7bb0 2024-04-02T09:12:24.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.918+0000 7f63f2000700 1 -- 172.21.15.89:0/2175438656 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f63e8005990 con 0x7f63f80b7bb0 2024-04-02T09:12:24.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.919+0000 7f63f2000700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f63dc06adb0 0x7f63dc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:24.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.919+0000 7f63f2000700 1 -- 172.21.15.89:0/2175438656 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f63e8089710 con 0x7f63f80b7bb0 2024-04-02T09:12:24.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.920+0000 7f63fde00700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f63dc06adb0 0x7f63dc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:24.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.920+0000 7f63fde00700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f63dc06adb0 0x7f63dc06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f63ec009c60 tx=0x7f63ec008040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:24.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:24.924+0000 7f63f2000700 1 -- 172.21.15.89:0/2175438656 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f63e80545e0 con 0x7f63f80b7bb0 2024-04-02T09:12:25.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.169+0000 7f6400800700 1 -- 172.21.15.89:0/2175438656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f63f80cbb00 con 0x7f63f80b7bb0 2024-04-02T09:12:25.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.176+0000 7f63f2000700 1 -- 172.21.15.89:0/2175438656 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v101) v1 ==== 174+0+0 (secure 0 0 0) 0x7f63e8087e60 con 0x7f63f80b7bb0 2024-04-02T09:12:25.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.182+0000 7f6400800700 1 -- 172.21.15.89:0/2175438656 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f63dc06adb0 msgr2=0x7f63dc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:25.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.182+0000 7f6400800700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f63dc06adb0 0x7f63dc06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f63ec009c60 tx=0x7f63ec008040 comp rx=0 tx=0).stop 2024-04-02T09:12:25.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.182+0000 7f6400800700 1 -- 172.21.15.89:0/2175438656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80b7bb0 msgr2=0x7f63f80b4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:25.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.182+0000 7f6400800700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80b7bb0 0x7f63f80b4bc0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f63e800fb40 tx=0x7f63e800ff00 comp rx=0 tx=0).stop 2024-04-02T09:12:25.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.183+0000 7f6400800700 1 -- 172.21.15.89:0/2175438656 shutdown_connections 2024-04-02T09:12:25.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.183+0000 7f6400800700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f63dc06adb0 0x7f63dc06d270 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f63ec009c60 tx=0x7f63ec008040 comp rx=0 tx=0).stop 2024-04-02T09:12:25.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.183+0000 7f6400800700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e0004d90 0x7f63f80b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.183+0000 7f6400800700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f63f80b5100 0x7f63f80b5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.183+0000 7f6400800700 1 --2- 172.21.15.89:0/2175438656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63f80b7bb0 0x7f63f80b4bc0 secure :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f63e800fb40 tx=0x7f63e800ff00 comp rx=0 tx=0).stop 2024-04-02T09:12:25.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.183+0000 7f6400800700 1 -- 172.21.15.89:0/2175438656 >> 172.21.15.89:0/2175438656 conn(0x7f63f808aff0 msgr2=0x7f63f80b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:25.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.184+0000 7f6400800700 1 -- 172.21.15.89:0/2175438656 shutdown_connections 2024-04-02T09:12:25.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.186+0000 7f6400800700 1 -- 172.21.15.89:0/2175438656 wait complete. 2024-04-02T09:12:25.186 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 26 and data pool 27 2024-04-02T09:12:25.214 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:12:25.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.407+0000 7fa082000700 1 Processor -- start 2024-04-02T09:12:25.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.407+0000 7fa082000700 1 -- start start 2024-04-02T09:12:25.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.408+0000 7fa082000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa07c160f10 0x7fa07c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:25.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.408+0000 7fa082000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa07c0cf230 0x7fa07c0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:25.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.408+0000 7fa082000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c0ceda0 con 0x7fa07c161590 2024-04-02T09:12:25.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.408+0000 7fa082000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa07c0d23d0 con 0x7fa07c0cf230 2024-04-02T09:12:25.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.408+0000 7fa082000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa07c1612f0 con 0x7fa07c160f10 2024-04-02T09:12:25.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.408+0000 7fa07aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa07c0cf230 0x7fa07c0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:25.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.408+0000 7fa07aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa07c0cf230 0x7fa07c0d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52634/0 (socket says 172.21.15.89:52634) 2024-04-02T09:12:25.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.408+0000 7fa07aa00700 1 -- 172.21.15.89:0/2821564123 learned_addr learned my addr 172.21.15.89:0/2821564123 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:25.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.408+0000 7fa07be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa07c161590 0x7fa07c0b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49922/0 (socket says 172.21.15.89:49922) 2024-04-02T09:12:25.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.408+0000 7fa07b400700 1 --2- 172.21.15.89:0/2821564123 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa07c160f10 0x7fa07c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:25.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.409+0000 7fa07aa00700 1 -- 172.21.15.89:0/2821564123 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa07c160f10 msgr2=0x7fa07c0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:25.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.409+0000 7fa07aa00700 1 --2- 172.21.15.89:0/2821564123 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa07c160f10 0x7fa07c0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.409+0000 7fa07aa00700 1 -- 172.21.15.89:0/2821564123 >> v1:172.21.15.89:6789/0 conn(0x7fa07c161590 legacy=0x7fa07c0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:25.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.409+0000 7fa07aa00700 1 -- 172.21.15.89:0/2821564123 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07c0ce470 con 0x7fa07c0cf230 2024-04-02T09:12:25.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.409+0000 7fa07aa00700 1 --2- 172.21.15.89:0/2821564123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa07c0cf230 0x7fa07c0d1e90 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fa0680098f0 tx=0x7fa068009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c65e1ad77f0bc16 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:25.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.409+0000 7fa07a000700 1 -- 172.21.15.89:0/2821564123 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06800b8a0 con 0x7fa07c0cf230 2024-04-02T09:12:25.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.409+0000 7fa07a000700 1 -- 172.21.15.89:0/2821564123 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06800ba00 con 0x7fa07c0cf230 2024-04-02T09:12:25.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.409+0000 7fa07a000700 1 -- 172.21.15.89:0/2821564123 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06800bd90 con 0x7fa07c0cf230 2024-04-02T09:12:25.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.409+0000 7fa082000700 1 -- 172.21.15.89:0/2821564123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa07c0cf230 msgr2=0x7fa07c0d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:25.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.409+0000 7fa082000700 1 --2- 172.21.15.89:0/2821564123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa07c0cf230 0x7fa07c0d1e90 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fa0680098f0 tx=0x7fa068009c00 comp rx=0 tx=0).stop 2024-04-02T09:12:25.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.410+0000 7fa082000700 1 -- 172.21.15.89:0/2821564123 shutdown_connections 2024-04-02T09:12:25.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.410+0000 7fa082000700 1 --2- 172.21.15.89:0/2821564123 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa07c160f10 0x7fa07c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.410+0000 7fa082000700 1 --2- 172.21.15.89:0/2821564123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa07c0cf230 0x7fa07c0d1e90 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.410+0000 7fa082000700 1 -- 172.21.15.89:0/2821564123 >> 172.21.15.89:0/2821564123 conn(0x7fa07c08aff0 msgr2=0x7fa07c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:25.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.410+0000 7fa082000700 1 -- 172.21.15.89:0/2821564123 shutdown_connections 2024-04-02T09:12:25.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.410+0000 7fa082000700 1 -- 172.21.15.89:0/2821564123 wait complete. 2024-04-02T09:12:25.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.411+0000 7fa082000700 1 Processor -- start 2024-04-02T09:12:25.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.411+0000 7fa082000700 1 -- start start 2024-04-02T09:12:25.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.412+0000 7fa082000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa07c0cf230 0x7fa07c0b8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:25.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.412+0000 7fa082000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa07c160f10 0x7fa07c15dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:25.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.412+0000 7fa082000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa07c161590 0x7fa07c15f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:25.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.412+0000 7fa082000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07c0b8d40 con 0x7fa07c160f10 2024-04-02T09:12:25.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.412+0000 7fa082000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07c158bb0 con 0x7fa07c161590 2024-04-02T09:12:25.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.412+0000 7fa082000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa07c158e90 con 0x7fa07c0cf230 2024-04-02T09:12:25.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.413+0000 7fa07aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa07c160f10 0x7fa07c15dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:25.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.413+0000 7fa07aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa07c160f10 0x7fa07c15dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52642/0 (socket says 172.21.15.89:52642) 2024-04-02T09:12:25.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.413+0000 7fa07aa00700 1 -- 172.21.15.89:0/845460719 learned_addr learned my addr 172.21.15.89:0/845460719 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:25.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.413+0000 7fa07b400700 1 --2- 172.21.15.89:0/845460719 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa07c0cf230 0x7fa07c0b8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:25.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.413+0000 7fa07be00700 1 --2- 172.21.15.89:0/845460719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa07c161590 0x7fa07c15f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:25.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.413+0000 7fa07aa00700 1 -- 172.21.15.89:0/845460719 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa07c0cf230 msgr2=0x7fa07c0b8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:25.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.413+0000 7fa07aa00700 1 --2- 172.21.15.89:0/845460719 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa07c0cf230 0x7fa07c0b8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.413+0000 7fa07aa00700 1 -- 172.21.15.89:0/845460719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa07c161590 msgr2=0x7fa07c15f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:25.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.413+0000 7fa07aa00700 1 --2- 172.21.15.89:0/845460719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa07c161590 0x7fa07c15f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.413+0000 7fa07aa00700 1 -- 172.21.15.89:0/845460719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa068009550 con 0x7fa07c160f10 2024-04-02T09:12:25.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.413+0000 7fa07aa00700 1 --2- 172.21.15.89:0/845460719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa07c160f10 0x7fa07c15dd50 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fa068009c40 tx=0x7fa068004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:25.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.414+0000 7fa073400700 1 -- 172.21.15.89:0/845460719 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0680036a0 con 0x7fa07c160f10 2024-04-02T09:12:25.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.414+0000 7fa073400700 1 -- 172.21.15.89:0/845460719 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06801e070 con 0x7fa07c160f10 2024-04-02T09:12:25.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.414+0000 7fa082000700 1 -- 172.21.15.89:0/845460719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa07c159170 con 0x7fa07c160f10 2024-04-02T09:12:25.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.414+0000 7fa073400700 1 -- 172.21.15.89:0/845460719 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068003a40 con 0x7fa07c160f10 2024-04-02T09:12:25.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.414+0000 7fa082000700 1 -- 172.21.15.89:0/845460719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa07c159730 con 0x7fa07c160f10 2024-04-02T09:12:25.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.415+0000 7fa082000700 1 -- 172.21.15.89:0/845460719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07c0729f0 con 0x7fa07c160f10 2024-04-02T09:12:25.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.416+0000 7fa073400700 1 -- 172.21.15.89:0/845460719 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa068003ba0 con 0x7fa07c160f10 2024-04-02T09:12:25.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.417+0000 7fa073400700 1 --2- 172.21.15.89:0/845460719 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa05c06adb0 0x7fa05c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:25.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.417+0000 7fa073400700 1 -- 172.21.15.89:0/845460719 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fa068020070 con 0x7fa07c160f10 2024-04-02T09:12:25.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.417+0000 7fa07b400700 1 --2- 172.21.15.89:0/845460719 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa05c06adb0 0x7fa05c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:25.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.418+0000 7fa07b400700 1 --2- 172.21.15.89:0/845460719 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa05c06adb0 0x7fa05c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00b9b0 tx=0x7fa06c008040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:25.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.422+0000 7fa073400700 1 -- 172.21.15.89:0/845460719 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa068055ac0 con 0x7fa07c160f10 2024-04-02T09:12:25.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.622+0000 7fa082000700 1 -- 172.21.15.89:0/845460719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa07c159920 con 0x7fa07c160f10 2024-04-02T09:12:25.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.623+0000 7fa073400700 1 -- 172.21.15.89:0/845460719 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) v1 ==== 74+0+16749 (secure 0 0 0) 0x7fa0680590e0 con 0x7fa07c160f10 2024-04-02T09:12:25.622 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:25.622 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":222,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:12:25.170421+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:12:23.034869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-02T09:12:24.158560+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:12:25.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.626+0000 7fa082000700 1 -- 172.21.15.89:0/845460719 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa05c06adb0 msgr2=0x7fa05c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:25.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.626+0000 7fa082000700 1 --2- 172.21.15.89:0/845460719 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa05c06adb0 0x7fa05c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00b9b0 tx=0x7fa06c008040 comp rx=0 tx=0).stop 2024-04-02T09:12:25.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.626+0000 7fa082000700 1 -- 172.21.15.89:0/845460719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa07c160f10 msgr2=0x7fa07c15dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:25.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.626+0000 7fa082000700 1 --2- 172.21.15.89:0/845460719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa07c160f10 0x7fa07c15dd50 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fa068009c40 tx=0x7fa068004750 comp rx=0 tx=0).stop 2024-04-02T09:12:25.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.626+0000 7fa082000700 1 -- 172.21.15.89:0/845460719 shutdown_connections 2024-04-02T09:12:25.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.626+0000 7fa082000700 1 --2- 172.21.15.89:0/845460719 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa05c06adb0 0x7fa05c06d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.626+0000 7fa082000700 1 --2- 172.21.15.89:0/845460719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa07c161590 0x7fa07c15f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.626+0000 7fa082000700 1 --2- 172.21.15.89:0/845460719 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa07c0cf230 0x7fa07c0b8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.626+0000 7fa082000700 1 --2- 172.21.15.89:0/845460719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa07c160f10 0x7fa07c15dd50 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.626+0000 7fa082000700 1 -- 172.21.15.89:0/845460719 >> 172.21.15.89:0/845460719 conn(0x7fa07c08aff0 msgr2=0x7fa07c0d0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:25.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.626+0000 7fa082000700 1 -- 172.21.15.89:0/845460719 shutdown_connections 2024-04-02T09:12:25.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.626+0000 7fa082000700 1 -- 172.21.15.89:0/845460719 wait complete. 2024-04-02T09:12:25.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:12:25.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.838+0000 7f3aad000700 1 Processor -- start 2024-04-02T09:12:25.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.838+0000 7f3aad000700 1 -- start start 2024-04-02T09:12:25.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.839+0000 7f3aad000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3aa8160f10 0x7f3aa80d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:25.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.839+0000 7f3aad000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3aa80cf230 0x7f3aa80d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:25.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.839+0000 7f3aad000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa80ceda0 con 0x7f3aa8161590 2024-04-02T09:12:25.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.839+0000 7f3aad000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa80d23d0 con 0x7f3aa8160f10 2024-04-02T09:12:25.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.839+0000 7f3aad000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa81612f0 con 0x7f3aa80cf230 2024-04-02T09:12:25.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.839+0000 7f3aa5e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3aa8160f10 0x7f3aa80d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:25.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.839+0000 7f3aa5e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3aa8160f10 0x7f3aa80d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52646/0 (socket says 172.21.15.89:52646) 2024-04-02T09:12:25.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.839+0000 7f3aa5e00700 1 -- 172.21.15.89:0/2599523953 learned_addr learned my addr 172.21.15.89:0/2599523953 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:25.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.839+0000 7f3aa5400700 1 --2- 172.21.15.89:0/2599523953 >> v2:172.21.15.113:3300/0 conn(0x7f3aa80cf230 0x7f3aa80d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:25.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.840+0000 7f3aa5e00700 1 -- 172.21.15.89:0/2599523953 >> v2:172.21.15.113:3300/0 conn(0x7f3aa80cf230 msgr2=0x7f3aa80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:25.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.840+0000 7f3aa5e00700 1 --2- 172.21.15.89:0/2599523953 >> v2:172.21.15.113:3300/0 conn(0x7f3aa80cf230 0x7f3aa80d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.840+0000 7f3aa5e00700 1 -- 172.21.15.89:0/2599523953 >> v1:172.21.15.89:6789/0 conn(0x7f3aa8161590 legacy=0x7f3aa80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:25.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.840+0000 7f3aa5e00700 1 -- 172.21.15.89:0/2599523953 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa80ce470 con 0x7f3aa8160f10 2024-04-02T09:12:25.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.840+0000 7f3aa5e00700 1 --2- 172.21.15.89:0/2599523953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8160f10 0x7f3aa80d18b0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f3aa808cdb0 tx=0x7f3a98009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecfdf08a73b3695 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:25.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.840+0000 7f3aa4a00700 1 -- 172.21.15.89:0/2599523953 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a98011830 con 0x7f3aa8160f10 2024-04-02T09:12:25.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.840+0000 7f3aa4a00700 1 -- 172.21.15.89:0/2599523953 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a98011990 con 0x7f3aa8160f10 2024-04-02T09:12:25.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.840+0000 7f3aa4a00700 1 -- 172.21.15.89:0/2599523953 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a98011d20 con 0x7f3aa8160f10 2024-04-02T09:12:25.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.841+0000 7f3aad000700 1 -- 172.21.15.89:0/2599523953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8160f10 msgr2=0x7f3aa80d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:25.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.841+0000 7f3aad000700 1 --2- 172.21.15.89:0/2599523953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8160f10 0x7f3aa80d18b0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f3aa808cdb0 tx=0x7f3a98009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.841+0000 7f3aad000700 1 -- 172.21.15.89:0/2599523953 shutdown_connections 2024-04-02T09:12:25.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.841+0000 7f3aad000700 1 --2- 172.21.15.89:0/2599523953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8160f10 0x7f3aa80d18b0 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.841+0000 7f3aad000700 1 --2- 172.21.15.89:0/2599523953 >> v2:172.21.15.113:3300/0 conn(0x7f3aa80cf230 0x7f3aa80d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.841+0000 7f3aad000700 1 -- 172.21.15.89:0/2599523953 >> 172.21.15.89:0/2599523953 conn(0x7f3aa808aff0 msgr2=0x7f3aa80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:25.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.841+0000 7f3aad000700 1 -- 172.21.15.89:0/2599523953 shutdown_connections 2024-04-02T09:12:25.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.841+0000 7f3aad000700 1 -- 172.21.15.89:0/2599523953 wait complete. 2024-04-02T09:12:25.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.843+0000 7f3aad000700 1 Processor -- start 2024-04-02T09:12:25.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.843+0000 7f3aad000700 1 -- start start 2024-04-02T09:12:25.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aad000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aa80cf230 0x7f3aa81525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:25.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aad000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8160f10 0x7f3aa8158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:25.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aad000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3aa8161590 0x7f3aa8155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:25.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aad000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa81f74f0 con 0x7f3aa8160f10 2024-04-02T09:12:25.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aad000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa8157f00 con 0x7f3aa8161590 2024-04-02T09:12:25.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aad000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa81581e0 con 0x7f3aa80cf230 2024-04-02T09:12:25.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aa5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8160f10 0x7f3aa8158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:25.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aa5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8160f10 0x7f3aa8158fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52656/0 (socket says 172.21.15.89:52656) 2024-04-02T09:12:25.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aa5400700 1 -- 172.21.15.89:0/3755117544 learned_addr learned my addr 172.21.15.89:0/3755117544 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:25.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aa5e00700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aa80cf230 0x7f3aa81525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:25.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aa6800700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3aa8161590 0x7f3aa8155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:25.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aa5400700 1 -- 172.21.15.89:0/3755117544 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aa80cf230 msgr2=0x7f3aa81525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:25.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aa5400700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aa80cf230 0x7f3aa81525c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aa5400700 1 -- 172.21.15.89:0/3755117544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3aa8161590 msgr2=0x7f3aa8155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:25.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aa5400700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3aa8161590 0x7f3aa8155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:25.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aa5400700 1 -- 172.21.15.89:0/3755117544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a98009550 con 0x7f3aa8160f10 2024-04-02T09:12:25.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aa5e00700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aa80cf230 0x7f3aa81525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:25.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.844+0000 7f3aa6800700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3aa8161590 0x7f3aa8155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:25.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.845+0000 7f3aa5400700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8160f10 0x7f3aa8158fa0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f3a9400e870 tx=0x7f3a9400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:25.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.845+0000 7f3a9e000700 1 -- 172.21.15.89:0/3755117544 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a94009590 con 0x7f3aa8160f10 2024-04-02T09:12:25.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.845+0000 7f3a9e000700 1 -- 172.21.15.89:0/3755117544 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9400ee70 con 0x7f3aa8160f10 2024-04-02T09:12:25.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.845+0000 7f3aad000700 1 -- 172.21.15.89:0/3755117544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa81584c0 con 0x7f3aa8160f10 2024-04-02T09:12:25.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.845+0000 7f3a9e000700 1 -- 172.21.15.89:0/3755117544 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9400ca50 con 0x7f3aa8160f10 2024-04-02T09:12:25.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.845+0000 7f3aad000700 1 -- 172.21.15.89:0/3755117544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa8158a80 con 0x7f3aa8160f10 2024-04-02T09:12:25.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.846+0000 7f3aad000700 1 -- 172.21.15.89:0/3755117544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa8067720 con 0x7f3aa8160f10 2024-04-02T09:12:25.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.847+0000 7f3a9e000700 1 -- 172.21.15.89:0/3755117544 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3a9400cbb0 con 0x7f3aa8160f10 2024-04-02T09:12:25.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.849+0000 7f3a9e000700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3a8806adb0 0x7f3a8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:25.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.849+0000 7f3a9e000700 1 -- 172.21.15.89:0/3755117544 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f3a94089660 con 0x7f3aa8160f10 2024-04-02T09:12:25.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.849+0000 7f3aa5e00700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3a8806adb0 0x7f3a8806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:25.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.850+0000 7f3aa5e00700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3a8806adb0 0x7f3a8806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3a98000c00 tx=0x7f3a98008030 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:25.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:25.853+0000 7f3a9e000700 1 -- 172.21.15.89:0/3755117544 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3a94054570 con 0x7f3aa8160f10 2024-04-02T09:12:26.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.056+0000 7f3aad000700 1 -- 172.21.15.89:0/3755117544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3aa80d21b0 con 0x7f3aa8160f10 2024-04-02T09:12:26.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.057+0000 7f3a9e000700 1 -- 172.21.15.89:0/3755117544 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) v1 ==== 74+0+16749 (secure 0 0 0) 0x7f3a9400ce60 con 0x7f3aa8160f10 2024-04-02T09:12:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:26.056 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":222,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:12:25.170421+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:12:23.034869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-02T09:12:24.158560+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:12:26.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.060+0000 7f3aad000700 1 -- 172.21.15.89:0/3755117544 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3a8806adb0 msgr2=0x7f3a8806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:26.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.060+0000 7f3aad000700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3a8806adb0 0x7f3a8806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3a98000c00 tx=0x7f3a98008030 comp rx=0 tx=0).stop 2024-04-02T09:12:26.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.060+0000 7f3aad000700 1 -- 172.21.15.89:0/3755117544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8160f10 msgr2=0x7f3aa8158fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:26.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.060+0000 7f3aad000700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8160f10 0x7f3aa8158fa0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f3a9400e870 tx=0x7f3a9400ec30 comp rx=0 tx=0).stop 2024-04-02T09:12:26.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.060+0000 7f3aad000700 1 -- 172.21.15.89:0/3755117544 shutdown_connections 2024-04-02T09:12:26.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.060+0000 7f3aad000700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3a8806adb0 0x7f3a8806d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:26.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.060+0000 7f3aad000700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3aa8161590 0x7f3aa8155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:26.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.060+0000 7f3aad000700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3aa80cf230 0x7f3aa81525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:26.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.060+0000 7f3aad000700 1 --2- 172.21.15.89:0/3755117544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3aa8160f10 0x7f3aa8158fa0 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:26.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.060+0000 7f3aad000700 1 -- 172.21.15.89:0/3755117544 >> 172.21.15.89:0/3755117544 conn(0x7f3aa808aff0 msgr2=0x7f3aa80cad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:26.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.060+0000 7f3aad000700 1 -- 172.21.15.89:0/3755117544 shutdown_connections 2024-04-02T09:12:26.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.060+0000 7f3aad000700 1 -- 172.21.15.89:0/3755117544 wait complete. 2024-04-02T09:12:26.076 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T09:12:26.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.276+0000 7f59ed400700 1 Processor -- start 2024-04-02T09:12:26.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.277+0000 7f59ed400700 1 -- start start 2024-04-02T09:12:26.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.278+0000 7f59ed400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f59e815d9d0 0x7f59e80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:26.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.278+0000 7f59ed400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e8161420 con 0x7f59e80d2540 2024-04-02T09:12:26.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.278+0000 7f59ed400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e8161700 con 0x7f59e80d16a0 2024-04-02T09:12:26.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.278+0000 7f59ed400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e815edc0 con 0x7f59e815d9d0 2024-04-02T09:12:26.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.278+0000 7f59e7200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f59e80d2540 0x7f59e80ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49938/0 (socket says 172.21.15.89:49938) 2024-04-02T09:12:26.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.278+0000 7f59e7200700 1 -- 172.21.15.89:0/245028877 learned_addr learned my addr 172.21.15.89:0/245028877 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:26.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.278+0000 7f59e6800700 1 --2- 172.21.15.89:0/245028877 >> v2:172.21.15.113:3300/0 conn(0x7f59e815d9d0 0x7f59e80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:26.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.278+0000 7f59e5400700 1 -- 172.21.15.89:0/245028877 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008621267 0 0) 0x7f59e8161420 con 0x7f59e80d2540 2024-04-02T09:12:26.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.278+0000 7f59e5400700 1 -- 172.21.15.89:0/245028877 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59cc0035b0 con 0x7f59e80d2540 2024-04-02T09:12:26.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e5400700 1 -- 172.21.15.89:0/245028877 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 317429938 0 0) 0x7f59cc0035b0 con 0x7f59e80d2540 2024-04-02T09:12:26.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e5400700 1 -- 172.21.15.89:0/245028877 >> v2:172.21.15.113:3300/0 conn(0x7f59e815d9d0 msgr2=0x7f59e80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e5400700 1 --2- 172.21.15.89:0/245028877 >> v2:172.21.15.113:3300/0 conn(0x7f59e815d9d0 0x7f59e80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e5400700 1 -- 172.21.15.89:0/245028877 >> v1:172.21.15.113:6789/0 conn(0x7f59e80d16a0 legacy=0x7f59e80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e5400700 1 -- 172.21.15.89:0/245028877 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e81619e0 con 0x7f59e80d2540 2024-04-02T09:12:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e5400700 1 -- 172.21.15.89:0/245028877 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f59e00019f0 con 0x7f59e80d2540 2024-04-02T09:12:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e5400700 1 -- 172.21.15.89:0/245028877 >> v1:172.21.15.89:6789/0 conn(0x7f59e80d2540 legacy=0x7f59e80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e5400700 1 --2- 172.21.15.89:0/245028877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59cc003a50 0x7f59cc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e5400700 1 --2- 172.21.15.89:0/245028877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59cc004850 0x7f59cc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e5400700 1 --2- 172.21.15.89:0/245028877 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59cc005260 0x7f59cc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e5400700 1 -- 172.21.15.89:0/245028877 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc00c180 con 0x7f59cc004850 2024-04-02T09:12:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e5400700 1 -- 172.21.15.89:0/245028877 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc00c2f0 con 0x7f59cc003a50 2024-04-02T09:12:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e5400700 1 -- 172.21.15.89:0/245028877 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc00c4b0 con 0x7f59cc005260 2024-04-02T09:12:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e5e00700 1 --2- 172.21.15.89:0/245028877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59cc004850 0x7f59cc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e6800700 1 --2- 172.21.15.89:0/245028877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59cc003a50 0x7f59cc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.279+0000 7f59e7200700 1 --2- 172.21.15.89:0/245028877 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59cc005260 0x7f59cc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:26.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.280+0000 7f59e5e00700 1 -- 172.21.15.89:0/245028877 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59cc005260 msgr2=0x7f59cc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:26.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.280+0000 7f59e5e00700 1 --2- 172.21.15.89:0/245028877 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59cc005260 0x7f59cc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:26.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.280+0000 7f59e5e00700 1 -- 172.21.15.89:0/245028877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59cc003a50 msgr2=0x7f59cc003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:26.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.280+0000 7f59e5e00700 1 --2- 172.21.15.89:0/245028877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59cc003a50 0x7f59cc003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:26.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.280+0000 7f59e5e00700 1 -- 172.21.15.89:0/245028877 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc0035b0 con 0x7f59cc004850 2024-04-02T09:12:26.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.280+0000 7f59e7200700 1 -- 172.21.15.89:0/245028877 reap_dead start 2024-04-02T09:12:26.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.280+0000 7f59e6800700 1 --2- 172.21.15.89:0/245028877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59cc003a50 0x7f59cc003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:12:26.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.280+0000 7f59e5e00700 1 --2- 172.21.15.89:0/245028877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59cc004850 0x7f59cc004cd0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f59e80d1ae0 tx=0x7f59cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e91d7377f49d89f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:26.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.280+0000 7f59e5400700 1 -- 172.21.15.89:0/245028877 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d40012e0 con 0x7f59cc004850 2024-04-02T09:12:26.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.280+0000 7f59e5400700 1 -- 172.21.15.89:0/245028877 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d40052d0 con 0x7f59cc004850 2024-04-02T09:12:26.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.280+0000 7f59e5400700 1 -- 172.21.15.89:0/245028877 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d40036d0 con 0x7f59cc004850 2024-04-02T09:12:26.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.281+0000 7f59ed400700 1 -- 172.21.15.89:0/245028877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59cc004850 msgr2=0x7f59cc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:26.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.281+0000 7f59ed400700 1 --2- 172.21.15.89:0/245028877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59cc004850 0x7f59cc004cd0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f59e80d1ae0 tx=0x7f59cc005880 comp rx=0 tx=0).stop 2024-04-02T09:12:26.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.281+0000 7f59ed400700 1 -- 172.21.15.89:0/245028877 shutdown_connections 2024-04-02T09:12:26.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.281+0000 7f59ed400700 1 --2- 172.21.15.89:0/245028877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59cc004850 0x7f59cc004cd0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:26.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.281+0000 7f59ed400700 1 -- 172.21.15.89:0/245028877 >> 172.21.15.89:0/245028877 conn(0x7f59e808aff0 msgr2=0x7f59e80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:26.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.281+0000 7f59ed400700 1 -- 172.21.15.89:0/245028877 shutdown_connections 2024-04-02T09:12:26.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.281+0000 7f59ed400700 1 -- 172.21.15.89:0/245028877 wait complete. 2024-04-02T09:12:26.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.282+0000 7f59ed400700 1 Processor -- start 2024-04-02T09:12:26.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.283+0000 7f59ed400700 1 -- start start 2024-04-02T09:12:26.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.283+0000 7f59ed400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59cc004850 0x7f59e81589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:26.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.283+0000 7f59ed400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e815ddb0 0x7f59e815fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:26.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.283+0000 7f59ed400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59e80b4490 0x7f59e80b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:26.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.283+0000 7f59ed400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e80b4e50 con 0x7f59e815ddb0 2024-04-02T09:12:26.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.283+0000 7f59ed400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e81f3980 con 0x7f59e80b4490 2024-04-02T09:12:26.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.283+0000 7f59ed400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e815e320 con 0x7f59cc004850 2024-04-02T09:12:26.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.284+0000 7f59e5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e815ddb0 0x7f59e815fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:26.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.284+0000 7f59e5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e815ddb0 0x7f59e815fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52678/0 (socket says 172.21.15.89:52678) 2024-04-02T09:12:26.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.284+0000 7f59e5e00700 1 -- 172.21.15.89:0/2092073538 learned_addr learned my addr 172.21.15.89:0/2092073538 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:26.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.284+0000 7f59e6800700 1 --2- 172.21.15.89:0/2092073538 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59cc004850 0x7f59e81589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:26.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.284+0000 7f59e7200700 1 --2- 172.21.15.89:0/2092073538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59e80b4490 0x7f59e80b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:26.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.284+0000 7f59e5e00700 1 -- 172.21.15.89:0/2092073538 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59cc004850 msgr2=0x7f59e81589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:26.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.284+0000 7f59e5e00700 1 --2- 172.21.15.89:0/2092073538 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59cc004850 0x7f59e81589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:26.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.284+0000 7f59e5e00700 1 -- 172.21.15.89:0/2092073538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59e80b4490 msgr2=0x7f59e80b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:26.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.284+0000 7f59e5e00700 1 --2- 172.21.15.89:0/2092073538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59e80b4490 0x7f59e80b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:26.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.284+0000 7f59e5e00700 1 -- 172.21.15.89:0/2092073538 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e815e600 con 0x7f59e815ddb0 2024-04-02T09:12:26.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.284+0000 7f59e5e00700 1 --2- 172.21.15.89:0/2092073538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e815ddb0 0x7f59e815fdd0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f59d4001f00 tx=0x7f59d4001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:26.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.285+0000 7f59dea00700 1 -- 172.21.15.89:0/2092073538 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d40012c0 con 0x7f59e815ddb0 2024-04-02T09:12:26.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.285+0000 7f59dea00700 1 -- 172.21.15.89:0/2092073538 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d40050d0 con 0x7f59e815ddb0 2024-04-02T09:12:26.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.285+0000 7f59dea00700 1 -- 172.21.15.89:0/2092073538 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d401d420 con 0x7f59e815ddb0 2024-04-02T09:12:26.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.285+0000 7f59ed400700 1 -- 172.21.15.89:0/2092073538 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e815e8e0 con 0x7f59e815ddb0 2024-04-02T09:12:26.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.285+0000 7f59ed400700 1 -- 172.21.15.89:0/2092073538 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e8158f30 con 0x7f59e815ddb0 2024-04-02T09:12:26.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.286+0000 7f59ed400700 1 -- 172.21.15.89:0/2092073538 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e81f3cd0 con 0x7f59e815ddb0 2024-04-02T09:12:26.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.287+0000 7f59dea00700 1 -- 172.21.15.89:0/2092073538 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f59d401d580 con 0x7f59e815ddb0 2024-04-02T09:12:26.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.287+0000 7f59dea00700 1 --2- 172.21.15.89:0/2092073538 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f59c806adb0 0x7f59c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:26.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.288+0000 7f59dea00700 1 -- 172.21.15.89:0/2092073538 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f59d4089f70 con 0x7f59e815ddb0 2024-04-02T09:12:26.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.288+0000 7f59e6800700 1 --2- 172.21.15.89:0/2092073538 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f59c806adb0 0x7f59c806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:26.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.288+0000 7f59e6800700 1 --2- 172.21.15.89:0/2092073538 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f59c806adb0 0x7f59c806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f59e80684a0 tx=0x7f59d8008040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:26.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.292+0000 7f59dea00700 1 -- 172.21.15.89:0/2092073538 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59d40bd580 con 0x7f59e815ddb0 2024-04-02T09:12:26.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-04-02T09:12:26.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-04-02T09:12:26.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:26.540+0000 7f59ed400700 1 -- 172.21.15.89:0/2092073538 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f59e80b90c0 con 0x7f59e815ddb0 2024-04-02T09:12:27.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.186+0000 7f59dea00700 1 -- 172.21.15.89:0/2092073538 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v104) v1 ==== 118+0+0 (secure 0 0 0) 0x7f59d40210b0 con 0x7f59e815ddb0 2024-04-02T09:12:27.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.189+0000 7f59ed400700 1 -- 172.21.15.89:0/2092073538 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f59c806adb0 msgr2=0x7f59c806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:27.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.189+0000 7f59ed400700 1 --2- 172.21.15.89:0/2092073538 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f59c806adb0 0x7f59c806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f59e80684a0 tx=0x7f59d8008040 comp rx=0 tx=0).stop 2024-04-02T09:12:27.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.189+0000 7f59ed400700 1 -- 172.21.15.89:0/2092073538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e815ddb0 msgr2=0x7f59e815fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:27.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.189+0000 7f59ed400700 1 --2- 172.21.15.89:0/2092073538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e815ddb0 0x7f59e815fdd0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f59d4001f00 tx=0x7f59d4001170 comp rx=0 tx=0).stop 2024-04-02T09:12:27.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.189+0000 7f59ed400700 1 -- 172.21.15.89:0/2092073538 shutdown_connections 2024-04-02T09:12:27.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.189+0000 7f59ed400700 1 --2- 172.21.15.89:0/2092073538 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f59c806adb0 0x7f59c806d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:27.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.189+0000 7f59ed400700 1 --2- 172.21.15.89:0/2092073538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59e80b4490 0x7f59e80b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:27.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.189+0000 7f59ed400700 1 --2- 172.21.15.89:0/2092073538 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f59cc004850 0x7f59e81589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:27.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.189+0000 7f59ed400700 1 --2- 172.21.15.89:0/2092073538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59e815ddb0 0x7f59e815fdd0 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:27.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.189+0000 7f59ed400700 1 -- 172.21.15.89:0/2092073538 >> 172.21.15.89:0/2092073538 conn(0x7f59e808aff0 msgr2=0x7f59e80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:27.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.189+0000 7f59ed400700 1 -- 172.21.15.89:0/2092073538 shutdown_connections 2024-04-02T09:12:27.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.189+0000 7f59ed400700 1 -- 172.21.15.89:0/2092073538 wait complete. 2024-04-02T09:12:27.207 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T09:12:27.207 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T09:12:27.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.403+0000 7f9d7fe00700 1 Processor -- start 2024-04-02T09:12:27.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.403+0000 7f9d7fe00700 1 -- start start 2024-04-02T09:12:27.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.404+0000 7f9d7fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9d8015d230 0x7f9d800d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:27.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.404+0000 7f9d7fe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9d8015d610 0x7f9d800d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:27.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.404+0000 7f9d7fe00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d800b6a40 con 0x7f9d800b8fe0 2024-04-02T09:12:27.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.404+0000 7f9d7fe00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d800b6cf0 con 0x7f9d8015d230 2024-04-02T09:12:27.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.404+0000 7f9d7fe00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d800b6fd0 con 0x7f9d8015d610 2024-04-02T09:12:27.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.404+0000 7f9d7ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9d8015d230 0x7f9d800d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:27.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.404+0000 7f9d7ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9d8015d230 0x7f9d800d1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52684/0 (socket says 172.21.15.89:52684) 2024-04-02T09:12:27.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.405+0000 7f9d7ea00700 1 -- 172.21.15.89:0/1689707642 learned_addr learned my addr 172.21.15.89:0/1689707642 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:27.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.405+0000 7f9d7f400700 1 --2- 172.21.15.89:0/1689707642 >> v2:172.21.15.113:3300/0 conn(0x7f9d8015d610 0x7f9d800d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:27.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.405+0000 7f9d7ea00700 1 -- 172.21.15.89:0/1689707642 >> v2:172.21.15.113:3300/0 conn(0x7f9d8015d610 msgr2=0x7f9d800d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:27.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.405+0000 7f9d7ea00700 1 --2- 172.21.15.89:0/1689707642 >> v2:172.21.15.113:3300/0 conn(0x7f9d8015d610 0x7f9d800d02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:27.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.405+0000 7f9d7ea00700 1 -- 172.21.15.89:0/1689707642 >> v1:172.21.15.89:6789/0 conn(0x7f9d800b8fe0 legacy=0x7f9d800b6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:27.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.405+0000 7f9d7ea00700 1 -- 172.21.15.89:0/1689707642 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d800b72b0 con 0x7f9d8015d230 2024-04-02T09:12:27.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.405+0000 7f9d7f400700 1 --2- 172.21.15.89:0/1689707642 >> v2:172.21.15.113:3300/0 conn(0x7f9d8015d610 0x7f9d800d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:27.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.405+0000 7f9d7ea00700 1 --2- 172.21.15.89:0/1689707642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d8015d230 0x7f9d800d1390 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f9d700098f0 tx=0x7f9d70009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f92b6b8b3ebebb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:27.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.405+0000 7f9d7e000700 1 -- 172.21.15.89:0/1689707642 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7000b860 con 0x7f9d8015d230 2024-04-02T09:12:27.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.405+0000 7f9d7e000700 1 -- 172.21.15.89:0/1689707642 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7000b9c0 con 0x7f9d8015d230 2024-04-02T09:12:27.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.406+0000 7f9d7e000700 1 -- 172.21.15.89:0/1689707642 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7000bd50 con 0x7f9d8015d230 2024-04-02T09:12:27.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.406+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1689707642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d8015d230 msgr2=0x7f9d800d1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:27.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.406+0000 7f9d7fe00700 1 --2- 172.21.15.89:0/1689707642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d8015d230 0x7f9d800d1390 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f9d700098f0 tx=0x7f9d70009c00 comp rx=0 tx=0).stop 2024-04-02T09:12:27.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.406+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1689707642 shutdown_connections 2024-04-02T09:12:27.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.406+0000 7f9d7fe00700 1 --2- 172.21.15.89:0/1689707642 >> v2:172.21.15.113:3300/0 conn(0x7f9d8015d610 0x7f9d800d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:27.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.406+0000 7f9d7fe00700 1 --2- 172.21.15.89:0/1689707642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d8015d230 0x7f9d800d1390 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:27.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.406+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1689707642 >> 172.21.15.89:0/1689707642 conn(0x7f9d800668e0 msgr2=0x7f9d80066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:27.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.406+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1689707642 shutdown_connections 2024-04-02T09:12:27.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.406+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1689707642 wait complete. 2024-04-02T09:12:27.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.408+0000 7f9d7fe00700 1 Processor -- start 2024-04-02T09:12:27.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.408+0000 7f9d7fe00700 1 -- start start 2024-04-02T09:12:27.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d800b8fe0 0x7f9d80152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:27.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7fe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d8015d230 0x7f9d800c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:27.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d8015d610 0x7f9d80155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:27.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7fe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d800c9040 con 0x7f9d800b8fe0 2024-04-02T09:12:27.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7fe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d800c7b40 con 0x7f9d8015d230 2024-04-02T09:12:27.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7fe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d800c9b80 con 0x7f9d8015d610 2024-04-02T09:12:27.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d800b8fe0 0x7f9d80152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:27.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d800b8fe0 0x7f9d80152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52696/0 (socket says 172.21.15.89:52696) 2024-04-02T09:12:27.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7ea00700 1 -- 172.21.15.89:0/1584003728 learned_addr learned my addr 172.21.15.89:0/1584003728 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:27.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d77e00700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d8015d230 0x7f9d800c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:27.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7f400700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d8015d610 0x7f9d80155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:27.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7ea00700 1 -- 172.21.15.89:0/1584003728 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d8015d610 msgr2=0x7f9d80155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:27.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7ea00700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d8015d610 0x7f9d80155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:27.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7ea00700 1 -- 172.21.15.89:0/1584003728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d8015d230 msgr2=0x7f9d800c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:27.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7ea00700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d8015d230 0x7f9d800c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:27.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.409+0000 7f9d7ea00700 1 -- 172.21.15.89:0/1584003728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d70009550 con 0x7f9d800b8fe0 2024-04-02T09:12:27.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.410+0000 7f9d77e00700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d8015d230 0x7f9d800c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:27.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.410+0000 7f9d7ea00700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d800b8fe0 0x7f9d80152180 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f9d800d4470 tx=0x7f9d70004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:27.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.410+0000 7f9d76a00700 1 -- 172.21.15.89:0/1584003728 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70004400 con 0x7f9d800b8fe0 2024-04-02T09:12:27.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.410+0000 7f9d76a00700 1 -- 172.21.15.89:0/1584003728 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7001f070 con 0x7f9d800b8fe0 2024-04-02T09:12:27.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.410+0000 7f9d76a00700 1 -- 172.21.15.89:0/1584003728 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d700038c0 con 0x7f9d800b8fe0 2024-04-02T09:12:27.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.410+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1584003728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d800c9250 con 0x7f9d800b8fe0 2024-04-02T09:12:27.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.410+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1584003728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d800c9810 con 0x7f9d800b8fe0 2024-04-02T09:12:27.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.411+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1584003728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8004ea80 con 0x7f9d800b8fe0 2024-04-02T09:12:27.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.412+0000 7f9d76a00700 1 -- 172.21.15.89:0/1584003728 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9d70021030 con 0x7f9d800b8fe0 2024-04-02T09:12:27.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.413+0000 7f9d76a00700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9d5c06adb0 0x7f9d5c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:27.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.413+0000 7f9d76a00700 1 -- 172.21.15.89:0/1584003728 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f9d7008ab60 con 0x7f9d800b8fe0 2024-04-02T09:12:27.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.413+0000 7f9d77e00700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9d5c06adb0 0x7f9d5c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:27.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.414+0000 7f9d77e00700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9d5c06adb0 0x7f9d5c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9d801f6730 tx=0x7f9d68009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:27.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.418+0000 7f9d76a00700 1 -- 172.21.15.89:0/1584003728 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9d700559e0 con 0x7f9d800b8fe0 2024-04-02T09:12:27.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:27.666+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1584003728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9d800b2f00 con 0x7f9d800b8fe0 2024-04-02T09:12:28.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.196+0000 7f9d76a00700 1 -- 172.21.15.89:0/1584003728 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v105) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9d70059000 con 0x7f9d800b8fe0 2024-04-02T09:12:28.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.199+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1584003728 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9d5c06adb0 msgr2=0x7f9d5c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:28.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.199+0000 7f9d7fe00700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9d5c06adb0 0x7f9d5c06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9d801f6730 tx=0x7f9d68009040 comp rx=0 tx=0).stop 2024-04-02T09:12:28.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.199+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1584003728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d800b8fe0 msgr2=0x7f9d80152180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:28.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.199+0000 7f9d7fe00700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d800b8fe0 0x7f9d80152180 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f9d800d4470 tx=0x7f9d70004750 comp rx=0 tx=0).stop 2024-04-02T09:12:28.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.199+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1584003728 shutdown_connections 2024-04-02T09:12:28.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.199+0000 7f9d7fe00700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d8015d610 0x7f9d80155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.199+0000 7f9d7fe00700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d800b8fe0 0x7f9d80152180 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.199+0000 7f9d7fe00700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9d5c06adb0 0x7f9d5c06d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.199+0000 7f9d7fe00700 1 --2- 172.21.15.89:0/1584003728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d8015d230 0x7f9d800c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.199+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1584003728 >> 172.21.15.89:0/1584003728 conn(0x7f9d800668e0 msgr2=0x7f9d800d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:28.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.199+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1584003728 shutdown_connections 2024-04-02T09:12:28.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.199+0000 7f9d7fe00700 1 -- 172.21.15.89:0/1584003728 wait complete. 2024-04-02T09:12:28.216 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T09:12:28.216 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:12:28.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.411+0000 7f25df400700 1 Processor -- start 2024-04-02T09:12:28.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.411+0000 7f25df400700 1 -- start start 2024-04-02T09:12:28.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.412+0000 7f25df400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f25d80cb3f0 0x7f25d80ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.412+0000 7f25df400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f25d80ac600 0x7f25d806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.413+0000 7f25df400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d815fd50 con 0x7f25d815b5d0 2024-04-02T09:12:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.413+0000 7f25df400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d815b9b0 con 0x7f25d80ac600 2024-04-02T09:12:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.413+0000 7f25df400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d80aca10 con 0x7f25d80cb3f0 2024-04-02T09:12:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.413+0000 7f25dd400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f25d80ac600 0x7f25d806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.413+0000 7f25dd400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f25d80ac600 0x7f25d806c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52698/0 (socket says 172.21.15.89:52698) 2024-04-02T09:12:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.413+0000 7f25dd400700 1 -- 172.21.15.89:0/997690488 learned_addr learned my addr 172.21.15.89:0/997690488 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.413+0000 7f25d7e00700 1 --2- 172.21.15.89:0/997690488 >> v2:172.21.15.113:3300/0 conn(0x7f25d80cb3f0 0x7f25d80ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.413+0000 7f25dd400700 1 -- 172.21.15.89:0/997690488 >> v2:172.21.15.113:3300/0 conn(0x7f25d80cb3f0 msgr2=0x7f25d80ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.413+0000 7f25dd400700 1 --2- 172.21.15.89:0/997690488 >> v2:172.21.15.113:3300/0 conn(0x7f25d80cb3f0 0x7f25d80ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.413+0000 7f25dd400700 1 -- 172.21.15.89:0/997690488 >> v1:172.21.15.113:6789/0 conn(0x7f25d815b5d0 legacy=0x7f25d80ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:28.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.413+0000 7f25dd400700 1 -- 172.21.15.89:0/997690488 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d815f140 con 0x7f25d80ac600 2024-04-02T09:12:28.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.414+0000 7f25dd400700 1 --2- 172.21.15.89:0/997690488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25d80ac600 0x7f25d806c9d0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f25d806c6c0 tx=0x7f25d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c4543d4bdc8dc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.414+0000 7f25d7400700 1 -- 172.21.15.89:0/997690488 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d000e070 con 0x7f25d80ac600 2024-04-02T09:12:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.414+0000 7f25d7400700 1 -- 172.21.15.89:0/997690488 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d0003a20 con 0x7f25d80ac600 2024-04-02T09:12:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.414+0000 7f25d7400700 1 -- 172.21.15.89:0/997690488 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d0003dd0 con 0x7f25d80ac600 2024-04-02T09:12:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.414+0000 7f25df400700 1 -- 172.21.15.89:0/997690488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25d80ac600 msgr2=0x7f25d806c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.414+0000 7f25df400700 1 --2- 172.21.15.89:0/997690488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25d80ac600 0x7f25d806c9d0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f25d806c6c0 tx=0x7f25d000b330 comp rx=0 tx=0).stop 2024-04-02T09:12:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.414+0000 7f25df400700 1 -- 172.21.15.89:0/997690488 shutdown_connections 2024-04-02T09:12:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.414+0000 7f25df400700 1 --2- 172.21.15.89:0/997690488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25d80ac600 0x7f25d806c9d0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.414+0000 7f25df400700 1 --2- 172.21.15.89:0/997690488 >> v2:172.21.15.113:3300/0 conn(0x7f25d80cb3f0 0x7f25d80ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.414+0000 7f25df400700 1 -- 172.21.15.89:0/997690488 >> 172.21.15.89:0/997690488 conn(0x7f25d80cf190 msgr2=0x7f25d80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.414+0000 7f25df400700 1 -- 172.21.15.89:0/997690488 shutdown_connections 2024-04-02T09:12:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.414+0000 7f25df400700 1 -- 172.21.15.89:0/997690488 wait complete. 2024-04-02T09:12:28.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.417+0000 7f25df400700 1 Processor -- start 2024-04-02T09:12:28.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.417+0000 7f25df400700 1 -- start start 2024-04-02T09:12:28.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.419+0000 7f25df400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25d80ac600 0x7f25d81f2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:28.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.419+0000 7f25df400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d815b5d0 0x7f25d80cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.419+0000 7f25df400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25d80cc910 0x7f25d80ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.419+0000 7f25df400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d80724a0 con 0x7f25d80cc910 2024-04-02T09:12:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.419+0000 7f25df400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d80ca470 con 0x7f25d815b5d0 2024-04-02T09:12:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.419+0000 7f25df400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d8071820 con 0x7f25d80ac600 2024-04-02T09:12:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.419+0000 7f25dd400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25d80cc910 0x7f25d80ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.419+0000 7f25dd400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25d80cc910 0x7f25d80ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52702/0 (socket says 172.21.15.89:52702) 2024-04-02T09:12:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.419+0000 7f25dd400700 1 -- 172.21.15.89:0/1184104725 learned_addr learned my addr 172.21.15.89:0/1184104725 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.419+0000 7f25d7e00700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d815b5d0 0x7f25d80cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.419+0000 7f25dca00700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25d80ac600 0x7f25d81f2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.420+0000 7f25dd400700 1 -- 172.21.15.89:0/1184104725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25d80ac600 msgr2=0x7f25d81f2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.420+0000 7f25dd400700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25d80ac600 0x7f25d81f2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.420+0000 7f25dd400700 1 -- 172.21.15.89:0/1184104725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d815b5d0 msgr2=0x7f25d80cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.420+0000 7f25dd400700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d815b5d0 0x7f25d80cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.420+0000 7f25dd400700 1 -- 172.21.15.89:0/1184104725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d0009b30 con 0x7f25d80cc910 2024-04-02T09:12:28.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.420+0000 7f25dca00700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25d80ac600 0x7f25d81f2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:28.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.420+0000 7f25dd400700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25d80cc910 0x7f25d80ccd90 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f25d0015040 tx=0x7f25d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:28.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.420+0000 7f25d4c00700 1 -- 172.21.15.89:0/1184104725 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d0004440 con 0x7f25d80cc910 2024-04-02T09:12:28.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.420+0000 7f25d4c00700 1 -- 172.21.15.89:0/1184104725 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d00045c0 con 0x7f25d80cc910 2024-04-02T09:12:28.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.420+0000 7f25d4c00700 1 -- 172.21.15.89:0/1184104725 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d0004970 con 0x7f25d80cc910 2024-04-02T09:12:28.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.420+0000 7f25df400700 1 -- 172.21.15.89:0/1184104725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d8071b00 con 0x7f25d80cc910 2024-04-02T09:12:28.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.420+0000 7f25df400700 1 -- 172.21.15.89:0/1184104725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d80720c0 con 0x7f25d80cc910 2024-04-02T09:12:28.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.421+0000 7f25df400700 1 -- 172.21.15.89:0/1184104725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d8067720 con 0x7f25d80cc910 2024-04-02T09:12:28.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.422+0000 7f25d4c00700 1 -- 172.21.15.89:0/1184104725 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f25d001c440 con 0x7f25d80cc910 2024-04-02T09:12:28.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.424+0000 7f25d4c00700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f25b806adb0 0x7f25b806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:28.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.424+0000 7f25d4c00700 1 -- 172.21.15.89:0/1184104725 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f25d008aa30 con 0x7f25d80cc910 2024-04-02T09:12:28.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.424+0000 7f25dca00700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f25b806adb0 0x7f25b806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:28.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.424+0000 7f25dca00700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f25b806adb0 0x7f25b806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f25cc003fc0 tx=0x7f25cc009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:28.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.427+0000 7f25d4c00700 1 -- 172.21.15.89:0/1184104725 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f25d00592b0 con 0x7f25d80cc910 2024-04-02T09:12:28.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.669+0000 7f25df400700 1 -- 172.21.15.89:0/1184104725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25d806ea60 con 0x7f25d80cc910 2024-04-02T09:12:28.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.670+0000 7f25d4c00700 1 -- 172.21.15.89:0/1184104725 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+4755 (secure 0 0 0) 0x7f25d0058e40 con 0x7f25d80cc910 2024-04-02T09:12:28.668 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:28.669 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":105,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8969,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3650347444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3650347444},{"type":"v1","addr":"172.21.15.89:6836","nonce":3650347444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2152500685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2152500685},{"type":"v1","addr":"172.21.15.113:6834","nonce":2152500685}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2457040702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2457040702},{"type":"v1","addr":"172.21.15.113:6835","nonce":2457040702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:25.170849+0000","modified":"2024-04-02T09:12:28.185307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11157":{"gid":11157,"name":"a","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2615927215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2615927215},{"type":"v1","addr":"172.21.15.89:6837","nonce":2615927215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-04-02T09:12:28.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.673+0000 7f25df400700 1 -- 172.21.15.89:0/1184104725 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f25b806adb0 msgr2=0x7f25b806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:28.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.673+0000 7f25df400700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f25b806adb0 0x7f25b806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f25cc003fc0 tx=0x7f25cc009040 comp rx=0 tx=0).stop 2024-04-02T09:12:28.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.673+0000 7f25df400700 1 -- 172.21.15.89:0/1184104725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25d80cc910 msgr2=0x7f25d80ccd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:28.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.673+0000 7f25df400700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25d80cc910 0x7f25d80ccd90 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f25d0015040 tx=0x7f25d0004ad0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.673+0000 7f25df400700 1 -- 172.21.15.89:0/1184104725 shutdown_connections 2024-04-02T09:12:28.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.673+0000 7f25df400700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25d80cc910 0x7f25d80ccd90 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.673+0000 7f25df400700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f25d80ac600 0x7f25d81f2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.673+0000 7f25df400700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f25b806adb0 0x7f25b806d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.673+0000 7f25df400700 1 --2- 172.21.15.89:0/1184104725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d815b5d0 0x7f25d80cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.673+0000 7f25df400700 1 -- 172.21.15.89:0/1184104725 >> 172.21.15.89:0/1184104725 conn(0x7f25d80cf190 msgr2=0x7f25d80afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:28.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.673+0000 7f25df400700 1 -- 172.21.15.89:0/1184104725 shutdown_connections 2024-04-02T09:12:28.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.673+0000 7f25df400700 1 -- 172.21.15.89:0/1184104725 wait complete. 2024-04-02T09:12:28.672 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 105 2024-04-02T09:12:28.689 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:12:28.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.884+0000 7fc54fa00700 1 Processor -- start 2024-04-02T09:12:28.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.884+0000 7fc54fa00700 1 -- start start 2024-04-02T09:12:28.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.885+0000 7fc54fa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548161ca0 0x7fc5480cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:28.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.885+0000 7fc54fa00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5480ceda0 con 0x7fc548160f10 2024-04-02T09:12:28.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.885+0000 7fc54fa00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5481612f0 con 0x7fc5480d18b0 2024-04-02T09:12:28.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.885+0000 7fc54fa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5480d1d30 con 0x7fc548161ca0 2024-04-02T09:12:28.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.885+0000 7fc54d000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc548160f10 0x7fc5480d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52698/0 (socket says 172.21.15.89:52698) 2024-04-02T09:12:28.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.885+0000 7fc54d000700 1 -- 172.21.15.89:0/1616641681 learned_addr learned my addr 172.21.15.89:0/1616641681 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:28.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.885+0000 7fc54da00700 1 --2- 172.21.15.89:0/1616641681 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548161ca0 0x7fc5480cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:28.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.885+0000 7fc547400700 1 -- 172.21.15.89:0/1616641681 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159198693 0 0) 0x7fc5480ceda0 con 0x7fc548160f10 2024-04-02T09:12:28.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.885+0000 7fc547400700 1 -- 172.21.15.89:0/1616641681 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5300035b0 con 0x7fc548160f10 2024-04-02T09:12:28.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.886+0000 7fc547400700 1 -- 172.21.15.89:0/1616641681 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858923789 0 0) 0x7fc5481612f0 con 0x7fc5480d18b0 2024-04-02T09:12:28.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.886+0000 7fc547400700 1 -- 172.21.15.89:0/1616641681 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5480ceda0 con 0x7fc5480d18b0 2024-04-02T09:12:28.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.886+0000 7fc547400700 1 -- 172.21.15.89:0/1616641681 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3057415001 0 0) 0x7fc5300035b0 con 0x7fc548160f10 2024-04-02T09:12:28.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.886+0000 7fc54da00700 1 -- 172.21.15.89:0/1616641681 >> v1:172.21.15.113:6789/0 conn(0x7fc5480d18b0 legacy=0x7fc548161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:28.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.886+0000 7fc54da00700 1 -- 172.21.15.89:0/1616641681 >> v1:172.21.15.89:6789/0 conn(0x7fc548160f10 legacy=0x7fc5480d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:28.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.886+0000 7fc54da00700 1 -- 172.21.15.89:0/1616641681 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5480ce470 con 0x7fc548161ca0 2024-04-02T09:12:28.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.886+0000 7fc54da00700 1 --2- 172.21.15.89:0/1616641681 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548161ca0 0x7fc5480cf230 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc540009dc0 tx=0x7fc54000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=119caa32c2619f76 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:28.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.886+0000 7fc547400700 1 -- 172.21.15.89:0/1616641681 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54000e070 con 0x7fc548161ca0 2024-04-02T09:12:28.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.887+0000 7fc547400700 1 -- 172.21.15.89:0/1616641681 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc540003a20 con 0x7fc548161ca0 2024-04-02T09:12:28.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.887+0000 7fc547400700 1 -- 172.21.15.89:0/1616641681 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c001670 con 0x7fc548161ca0 2024-04-02T09:12:28.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.887+0000 7fc54fa00700 1 -- 172.21.15.89:0/1616641681 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548161ca0 msgr2=0x7fc5480cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:28.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.887+0000 7fc54fa00700 1 --2- 172.21.15.89:0/1616641681 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548161ca0 0x7fc5480cf230 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc540009dc0 tx=0x7fc54000b330 comp rx=0 tx=0).stop 2024-04-02T09:12:28.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.887+0000 7fc54fa00700 1 -- 172.21.15.89:0/1616641681 shutdown_connections 2024-04-02T09:12:28.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.887+0000 7fc54fa00700 1 --2- 172.21.15.89:0/1616641681 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548161ca0 0x7fc5480cf230 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.887+0000 7fc54fa00700 1 -- 172.21.15.89:0/1616641681 >> 172.21.15.89:0/1616641681 conn(0x7fc54808aff0 msgr2=0x7fc5480d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:28.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.887+0000 7fc54fa00700 1 -- 172.21.15.89:0/1616641681 shutdown_connections 2024-04-02T09:12:28.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.887+0000 7fc54fa00700 1 -- 172.21.15.89:0/1616641681 wait complete. 2024-04-02T09:12:28.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.889+0000 7fc54fa00700 1 Processor -- start 2024-04-02T09:12:28.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.889+0000 7fc54fa00700 1 -- start start 2024-04-02T09:12:28.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.890+0000 7fc54fa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5480d18b0 0x7fc5481f63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:28.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.890+0000 7fc54fa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548160f10 0x7fc5481f6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:28.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.890+0000 7fc54fa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc548161ca0 0x7fc5481f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:28.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.890+0000 7fc54fa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5481f68e0 con 0x7fc5480d18b0 2024-04-02T09:12:28.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.890+0000 7fc54fa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54815a220 con 0x7fc548161ca0 2024-04-02T09:12:28.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.890+0000 7fc54fa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54815a500 con 0x7fc548160f10 2024-04-02T09:12:28.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.890+0000 7fc54d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5480d18b0 0x7fc5481f63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:28.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.890+0000 7fc54d000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5480d18b0 0x7fc5481f63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52396/0 (socket says 172.21.15.89:52396) 2024-04-02T09:12:28.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.890+0000 7fc54d000700 1 -- 172.21.15.89:0/1481488644 learned_addr learned my addr 172.21.15.89:0/1481488644 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:28.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.890+0000 7fc54da00700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc548161ca0 0x7fc5481f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:28.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.890+0000 7fc547e00700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548160f10 0x7fc5481f6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:28.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.891+0000 7fc54d000700 1 -- 172.21.15.89:0/1481488644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548160f10 msgr2=0x7fc5481f6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:28.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.891+0000 7fc54d000700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548160f10 0x7fc5481f6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.891+0000 7fc54d000700 1 -- 172.21.15.89:0/1481488644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc548161ca0 msgr2=0x7fc5481f70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:28.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.891+0000 7fc54d000700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc548161ca0 0x7fc5481f70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:28.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.891+0000 7fc54d000700 1 -- 172.21.15.89:0/1481488644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc540009a70 con 0x7fc5480d18b0 2024-04-02T09:12:28.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.891+0000 7fc547e00700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548160f10 0x7fc5481f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:28.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.891+0000 7fc54d000700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5480d18b0 0x7fc5481f63a0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00c520 tx=0x7fc53c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:28.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.891+0000 7fc544c00700 1 -- 172.21.15.89:0/1481488644 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c004c80 con 0x7fc5480d18b0 2024-04-02T09:12:28.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.891+0000 7fc544c00700 1 -- 172.21.15.89:0/1481488644 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc53c004de0 con 0x7fc5480d18b0 2024-04-02T09:12:28.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.891+0000 7fc54fa00700 1 -- 172.21.15.89:0/1481488644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54815a7e0 con 0x7fc5480d18b0 2024-04-02T09:12:28.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.891+0000 7fc544c00700 1 -- 172.21.15.89:0/1481488644 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c007310 con 0x7fc5480d18b0 2024-04-02T09:12:28.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.891+0000 7fc54fa00700 1 -- 172.21.15.89:0/1481488644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54815ada0 con 0x7fc5480d18b0 2024-04-02T09:12:28.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.892+0000 7fc54fa00700 1 -- 172.21.15.89:0/1481488644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc54815afc0 con 0x7fc5480d18b0 2024-04-02T09:12:28.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.893+0000 7fc544c00700 1 -- 172.21.15.89:0/1481488644 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc53c007470 con 0x7fc5480d18b0 2024-04-02T09:12:28.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.894+0000 7fc544c00700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc52806adb0 0x7fc52806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:28.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.894+0000 7fc544c00700 1 -- 172.21.15.89:0/1481488644 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fc53c0892c0 con 0x7fc5480d18b0 2024-04-02T09:12:28.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.894+0000 7fc547e00700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc52806adb0 0x7fc52806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:28.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.895+0000 7fc547e00700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc52806adb0 0x7fc52806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc534007fd0 tx=0x7fc534009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:28.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:28.899+0000 7fc544c00700 1 -- 172.21.15.89:0/1481488644 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc53c054260 con 0x7fc5480d18b0 2024-04-02T09:12:29.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.098+0000 7fc54fa00700 1 -- 172.21.15.89:0/1481488644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc548067720 con 0x7fc5480d18b0 2024-04-02T09:12:29.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.099+0000 7fc544c00700 1 -- 172.21.15.89:0/1481488644 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+16736 (secure 0 0 0) 0x7fc53c057880 con 0x7fc5480d18b0 2024-04-02T09:12:29.097 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:29.098 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":223,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:12:26.173562+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:12:23.034869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-02T09:12:24.158560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:12:29.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.101+0000 7fc54fa00700 1 -- 172.21.15.89:0/1481488644 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc52806adb0 msgr2=0x7fc52806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:29.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.101+0000 7fc54fa00700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc52806adb0 0x7fc52806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc534007fd0 tx=0x7fc534009040 comp rx=0 tx=0).stop 2024-04-02T09:12:29.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.101+0000 7fc54fa00700 1 -- 172.21.15.89:0/1481488644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5480d18b0 msgr2=0x7fc5481f63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:29.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.101+0000 7fc54fa00700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5480d18b0 0x7fc5481f63a0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00c520 tx=0x7fc53c00c830 comp rx=0 tx=0).stop 2024-04-02T09:12:29.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.101+0000 7fc54fa00700 1 -- 172.21.15.89:0/1481488644 shutdown_connections 2024-04-02T09:12:29.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.101+0000 7fc54fa00700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc52806adb0 0x7fc52806d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.101+0000 7fc54fa00700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc548161ca0 0x7fc5481f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.101+0000 7fc54fa00700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5480d18b0 0x7fc5481f63a0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.101+0000 7fc54fa00700 1 --2- 172.21.15.89:0/1481488644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc548160f10 0x7fc5481f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.101+0000 7fc54fa00700 1 -- 172.21.15.89:0/1481488644 >> 172.21.15.89:0/1481488644 conn(0x7fc54808aff0 msgr2=0x7fc5480cadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:29.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.101+0000 7fc54fa00700 1 -- 172.21.15.89:0/1481488644 shutdown_connections 2024-04-02T09:12:29.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.101+0000 7fc54fa00700 1 -- 172.21.15.89:0/1481488644 wait complete. 2024-04-02T09:12:29.119 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T09:12:29.119 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:12:29.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.313+0000 7fa9fce00700 1 Processor -- start 2024-04-02T09:12:29.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.314+0000 7fa9fce00700 1 -- start start 2024-04-02T09:12:29.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.315+0000 7fa9fce00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa9f815da50 0x7fa9f80ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:29.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.315+0000 7fa9fce00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa9f80cec70 0x7fa9f80d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:29.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.315+0000 7fa9fce00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f80d24e0 con 0x7fa9f80d0b20 2024-04-02T09:12:29.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.315+0000 7fa9fce00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f80d0fd0 con 0x7fa9f815da50 2024-04-02T09:12:29.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.315+0000 7fa9fce00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f80b95d0 con 0x7fa9f80cec70 2024-04-02T09:12:29.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.316+0000 7fa9f5e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa9f815da50 0x7fa9f80ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:29.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.316+0000 7fa9f5e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa9f815da50 0x7fa9f80ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52398/0 (socket says 172.21.15.89:52398) 2024-04-02T09:12:29.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.316+0000 7fa9f5e00700 1 -- 172.21.15.89:0/539437802 learned_addr learned my addr 172.21.15.89:0/539437802 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:29.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.316+0000 7fa9f5400700 1 --2- 172.21.15.89:0/539437802 >> v2:172.21.15.113:3300/0 conn(0x7fa9f80cec70 0x7fa9f80d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:29.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.316+0000 7fa9f5e00700 1 -- 172.21.15.89:0/539437802 >> v2:172.21.15.113:3300/0 conn(0x7fa9f80cec70 msgr2=0x7fa9f80d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:29.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.316+0000 7fa9f5e00700 1 --2- 172.21.15.89:0/539437802 >> v2:172.21.15.113:3300/0 conn(0x7fa9f80cec70 0x7fa9f80d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.316+0000 7fa9f5e00700 1 -- 172.21.15.89:0/539437802 >> v1:172.21.15.89:6789/0 conn(0x7fa9f80d0b20 legacy=0x7fa9f80b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:29.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.316+0000 7fa9f5e00700 1 -- 172.21.15.89:0/539437802 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f80d27c0 con 0x7fa9f815da50 2024-04-02T09:12:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.316+0000 7fa9f5400700 1 --2- 172.21.15.89:0/539437802 >> v2:172.21.15.113:3300/0 conn(0x7fa9f80cec70 0x7fa9f80d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.317+0000 7fa9f5e00700 1 --2- 172.21.15.89:0/539437802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f815da50 0x7fa9f80ce730 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fa9f80740a0 tx=0x7fa9e8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d04534391dbda999 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.317+0000 7fa9f4a00700 1 -- 172.21.15.89:0/539437802 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e800b870 con 0x7fa9f815da50 2024-04-02T09:12:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.317+0000 7fa9f4a00700 1 -- 172.21.15.89:0/539437802 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e800b9d0 con 0x7fa9f815da50 2024-04-02T09:12:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.317+0000 7fa9f4a00700 1 -- 172.21.15.89:0/539437802 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e800bd60 con 0x7fa9f815da50 2024-04-02T09:12:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.317+0000 7fa9fce00700 1 -- 172.21.15.89:0/539437802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f815da50 msgr2=0x7fa9f80ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.317+0000 7fa9fce00700 1 --2- 172.21.15.89:0/539437802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f815da50 0x7fa9f80ce730 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fa9f80740a0 tx=0x7fa9e8009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.317+0000 7fa9fce00700 1 -- 172.21.15.89:0/539437802 shutdown_connections 2024-04-02T09:12:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.317+0000 7fa9fce00700 1 --2- 172.21.15.89:0/539437802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f815da50 0x7fa9f80ce730 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.317+0000 7fa9fce00700 1 --2- 172.21.15.89:0/539437802 >> v2:172.21.15.113:3300/0 conn(0x7fa9f80cec70 0x7fa9f80d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.317+0000 7fa9fce00700 1 -- 172.21.15.89:0/539437802 >> 172.21.15.89:0/539437802 conn(0x7fa9f80666a0 msgr2=0x7fa9f8066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.317+0000 7fa9fce00700 1 -- 172.21.15.89:0/539437802 shutdown_connections 2024-04-02T09:12:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.318+0000 7fa9fce00700 1 -- 172.21.15.89:0/539437802 wait complete. 2024-04-02T09:12:29.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.320+0000 7fa9fce00700 1 Processor -- start 2024-04-02T09:12:29.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.320+0000 7fa9fce00700 1 -- start start 2024-04-02T09:12:29.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.321+0000 7fa9fce00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80cec70 0x7fa9f80c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.321+0000 7fa9fce00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f80d0b20 0x7fa9f8151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.321+0000 7fa9fce00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f815da50 0x7fa9f80d3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.321+0000 7fa9fce00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f80cd8e0 con 0x7fa9f80d0b20 2024-04-02T09:12:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.321+0000 7fa9fce00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f80c7a60 con 0x7fa9f815da50 2024-04-02T09:12:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.321+0000 7fa9fce00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f815ed40 con 0x7fa9f80cec70 2024-04-02T09:12:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.321+0000 7fa9f5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f80d0b20 0x7fa9f8151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.321+0000 7fa9f5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f80d0b20 0x7fa9f8151e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52412/0 (socket says 172.21.15.89:52412) 2024-04-02T09:12:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.321+0000 7fa9f5400700 1 -- 172.21.15.89:0/3448678413 learned_addr learned my addr 172.21.15.89:0/3448678413 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9f5e00700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80cec70 0x7fa9f80c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9f6800700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f815da50 0x7fa9f80d3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9f5400700 1 -- 172.21.15.89:0/3448678413 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80cec70 msgr2=0x7fa9f80c7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9f5400700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80cec70 0x7fa9f80c7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9f5400700 1 -- 172.21.15.89:0/3448678413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f815da50 msgr2=0x7fa9f80d3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9f5400700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f815da50 0x7fa9f80d3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9f5400700 1 -- 172.21.15.89:0/3448678413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8009550 con 0x7fa9f80d0b20 2024-04-02T09:12:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9f5e00700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80cec70 0x7fa9f80c7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9f6800700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f815da50 0x7fa9f80d3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9f5400700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f80d0b20 0x7fa9f8151e70 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fa9e400e7e0 tx=0x7fa9e400eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9ee000700 1 -- 172.21.15.89:0/3448678413 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4013610 con 0x7fa9f80d0b20 2024-04-02T09:12:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9ee000700 1 -- 172.21.15.89:0/3448678413 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e4013770 con 0x7fa9f80d0b20 2024-04-02T09:12:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9fce00700 1 -- 172.21.15.89:0/3448678413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f815ea60 con 0x7fa9f80d0b20 2024-04-02T09:12:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9ee000700 1 -- 172.21.15.89:0/3448678413 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4013b20 con 0x7fa9f80d0b20 2024-04-02T09:12:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.322+0000 7fa9fce00700 1 -- 172.21.15.89:0/3448678413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f80c76f0 con 0x7fa9f80d0b20 2024-04-02T09:12:29.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.324+0000 7fa9fce00700 1 -- 172.21.15.89:0/3448678413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f80c8430 con 0x7fa9f80d0b20 2024-04-02T09:12:29.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.324+0000 7fa9ee000700 1 -- 172.21.15.89:0/3448678413 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa9e4013c80 con 0x7fa9f80d0b20 2024-04-02T09:12:29.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.326+0000 7fa9ee000700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa9d806adb0 0x7fa9d806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:29.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.326+0000 7fa9ee000700 1 -- 172.21.15.89:0/3448678413 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fa9e4089560 con 0x7fa9f80d0b20 2024-04-02T09:12:29.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.326+0000 7fa9f5e00700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa9d806adb0 0x7fa9d806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:29.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.327+0000 7fa9f5e00700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa9d806adb0 0x7fa9d806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8009130 tx=0x7fa9e8008030 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:29.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.330+0000 7fa9ee000700 1 -- 172.21.15.89:0/3448678413 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa9e4054230 con 0x7fa9f80d0b20 2024-04-02T09:12:29.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.526+0000 7fa9fce00700 1 -- 172.21.15.89:0/3448678413 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa9f80d3060 con 0x7fa9d806adb0 2024-04-02T09:12:29.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.537+0000 7fa9ee000700 1 -- 172.21.15.89:0/3448678413 <== mgr.10923 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289386 (secure 0 0 0) 0x7fa9e801f070 con 0x7fa9d806adb0 2024-04-02T09:12:29.539 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:29.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.541+0000 7fa9fce00700 1 -- 172.21.15.89:0/3448678413 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa9d806adb0 msgr2=0x7fa9d806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:29.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.541+0000 7fa9fce00700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa9d806adb0 0x7fa9d806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8009130 tx=0x7fa9e8008030 comp rx=0 tx=0).stop 2024-04-02T09:12:29.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.541+0000 7fa9fce00700 1 -- 172.21.15.89:0/3448678413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f80d0b20 msgr2=0x7fa9f8151e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:29.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.541+0000 7fa9fce00700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f80d0b20 0x7fa9f8151e70 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fa9e400e7e0 tx=0x7fa9e400eba0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.541+0000 7fa9fce00700 1 -- 172.21.15.89:0/3448678413 shutdown_connections 2024-04-02T09:12:29.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.541+0000 7fa9fce00700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa9d806adb0 0x7fa9d806d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.541+0000 7fa9fce00700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f815da50 0x7fa9f80d3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.541+0000 7fa9fce00700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80cec70 0x7fa9f80c7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.541+0000 7fa9fce00700 1 --2- 172.21.15.89:0/3448678413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f80d0b20 0x7fa9f8151e70 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.541+0000 7fa9fce00700 1 -- 172.21.15.89:0/3448678413 >> 172.21.15.89:0/3448678413 conn(0x7fa9f80666a0 msgr2=0x7fa9f80b7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:29.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.541+0000 7fa9fce00700 1 -- 172.21.15.89:0/3448678413 shutdown_connections 2024-04-02T09:12:29.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.541+0000 7fa9fce00700 1 -- 172.21.15.89:0/3448678413 wait complete. 2024-04-02T09:12:29.542 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:12:29.558 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-02T09:12:28.799428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":623,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":402416,"kb_used_data":23012,"kb_used_omap":12,"kb_used_meta":379251,"kb_avail":754572304,"statfs":{"total":773094113280,"available":772682039296,"internally_reserved":0,"allocated":23564288,"data_stored":12501762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":388353592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.129828"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424498+0000","last_change":"2024-04-02T09:12:25.187698+0000","last_active":"2024-04-02T09:12:26.424498+0000","last_peered":"2024-04-02T09:12:26.424498+0000","last_clean":"2024-04-02T09:12:26.424498+0000","last_became_active":"2024-04-02T09:12:25.187420+0000","last_became_peered":"2024-04-02T09:12:25.187420+0000","last_unstale":"2024-04-02T09:12:26.424498+0000","last_undegraded":"2024-04-02T09:12:26.424498+0000","last_fullsized":"2024-04-02T09:12:26.424498+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:45:21.225286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422997+0000","last_change":"2024-04-02T09:12:25.208060+0000","last_active":"2024-04-02T09:12:26.422997+0000","last_peered":"2024-04-02T09:12:26.422997+0000","last_clean":"2024-04-02T09:12:26.422997+0000","last_became_active":"2024-04-02T09:12:25.206932+0000","last_became_peered":"2024-04-02T09:12:25.206932+0000","last_unstale":"2024-04-02T09:12:26.422997+0000","last_undegraded":"2024-04-02T09:12:26.422997+0000","last_fullsized":"2024-04-02T09:12:26.422997+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:11.810703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423793+0000","last_change":"2024-04-02T09:12:25.194789+0000","last_active":"2024-04-02T09:12:26.423793+0000","last_peered":"2024-04-02T09:12:26.423793+0000","last_clean":"2024-04-02T09:12:26.423793+0000","last_became_active":"2024-04-02T09:12:25.194433+0000","last_became_peered":"2024-04-02T09:12:25.194433+0000","last_unstale":"2024-04-02T09:12:26.423793+0000","last_undegraded":"2024-04-02T09:12:26.423793+0000","last_fullsized":"2024-04-02T09:12:26.423793+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:18.798282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185951+0000","last_change":"2024-04-02T09:12:25.190996+0000","last_active":"2024-04-02T09:12:26.185951+0000","last_peered":"2024-04-02T09:12:26.185951+0000","last_clean":"2024-04-02T09:12:26.185951+0000","last_became_active":"2024-04-02T09:12:25.190655+0000","last_became_peered":"2024-04-02T09:12:25.190655+0000","last_unstale":"2024-04-02T09:12:26.185951+0000","last_undegraded":"2024-04-02T09:12:26.185951+0000","last_fullsized":"2024-04-02T09:12:26.185951+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:48:22.188323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423434+0000","last_change":"2024-04-02T09:12:25.191955+0000","last_active":"2024-04-02T09:12:26.423434+0000","last_peered":"2024-04-02T09:12:26.423434+0000","last_clean":"2024-04-02T09:12:26.423434+0000","last_became_active":"2024-04-02T09:12:25.191699+0000","last_became_peered":"2024-04-02T09:12:25.191699+0000","last_unstale":"2024-04-02T09:12:26.423434+0000","last_undegraded":"2024-04-02T09:12:26.423434+0000","last_fullsized":"2024-04-02T09:12:26.423434+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:22:42.917826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423946+0000","last_change":"2024-04-02T09:12:25.200889+0000","last_active":"2024-04-02T09:12:26.423946+0000","last_peered":"2024-04-02T09:12:26.423946+0000","last_clean":"2024-04-02T09:12:26.423946+0000","last_became_active":"2024-04-02T09:12:25.200635+0000","last_became_peered":"2024-04-02T09:12:25.200635+0000","last_unstale":"2024-04-02T09:12:26.423946+0000","last_undegraded":"2024-04-02T09:12:26.423946+0000","last_fullsized":"2024-04-02T09:12:26.423946+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:43:29.229527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422840+0000","last_change":"2024-04-02T09:12:25.201485+0000","last_active":"2024-04-02T09:12:26.422840+0000","last_peered":"2024-04-02T09:12:26.422840+0000","last_clean":"2024-04-02T09:12:26.422840+0000","last_became_active":"2024-04-02T09:12:25.200542+0000","last_became_peered":"2024-04-02T09:12:25.200542+0000","last_unstale":"2024-04-02T09:12:26.422840+0000","last_undegraded":"2024-04-02T09:12:26.422840+0000","last_fullsized":"2024-04-02T09:12:26.422840+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:19:30.245770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185880+0000","last_change":"2024-04-02T09:12:25.191970+0000","last_active":"2024-04-02T09:12:26.185880+0000","last_peered":"2024-04-02T09:12:26.185880+0000","last_clean":"2024-04-02T09:12:26.185880+0000","last_became_active":"2024-04-02T09:12:25.191572+0000","last_became_peered":"2024-04-02T09:12:25.191572+0000","last_unstale":"2024-04-02T09:12:26.185880+0000","last_undegraded":"2024-04-02T09:12:26.185880+0000","last_fullsized":"2024-04-02T09:12:26.185880+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:03.387546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422571+0000","last_change":"2024-04-02T09:12:25.201593+0000","last_active":"2024-04-02T09:12:26.422571+0000","last_peered":"2024-04-02T09:12:26.422571+0000","last_clean":"2024-04-02T09:12:26.422571+0000","last_became_active":"2024-04-02T09:12:25.200851+0000","last_became_peered":"2024-04-02T09:12:25.200851+0000","last_unstale":"2024-04-02T09:12:26.422571+0000","last_undegraded":"2024-04-02T09:12:26.422571+0000","last_fullsized":"2024-04-02T09:12:26.422571+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:26:33.117159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423365+0000","last_change":"2024-04-02T09:12:25.202349+0000","last_active":"2024-04-02T09:12:26.423365+0000","last_peered":"2024-04-02T09:12:26.423365+0000","last_clean":"2024-04-02T09:12:26.423365+0000","last_became_active":"2024-04-02T09:12:25.200134+0000","last_became_peered":"2024-04-02T09:12:25.200134+0000","last_unstale":"2024-04-02T09:12:26.423365+0000","last_undegraded":"2024-04-02T09:12:26.423365+0000","last_fullsized":"2024-04-02T09:12:26.423365+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:45:54.007817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424281+0000","last_change":"2024-04-02T09:12:25.203968+0000","last_active":"2024-04-02T09:12:26.424281+0000","last_peered":"2024-04-02T09:12:26.424281+0000","last_clean":"2024-04-02T09:12:26.424281+0000","last_became_active":"2024-04-02T09:12:25.203714+0000","last_became_peered":"2024-04-02T09:12:25.203714+0000","last_unstale":"2024-04-02T09:12:26.424281+0000","last_undegraded":"2024-04-02T09:12:26.424281+0000","last_fullsized":"2024-04-02T09:12:26.424281+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:44:28.685030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186694+0000","last_change":"2024-04-02T09:12:25.186623+0000","last_active":"2024-04-02T09:12:26.186694+0000","last_peered":"2024-04-02T09:12:26.186694+0000","last_clean":"2024-04-02T09:12:26.186694+0000","last_became_active":"2024-04-02T09:12:25.186297+0000","last_became_peered":"2024-04-02T09:12:25.186297+0000","last_unstale":"2024-04-02T09:12:26.186694+0000","last_undegraded":"2024-04-02T09:12:26.186694+0000","last_fullsized":"2024-04-02T09:12:26.186694+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:36:48.733839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.184901+0000","last_change":"2024-04-02T09:12:25.205465+0000","last_active":"2024-04-02T09:12:26.184901+0000","last_peered":"2024-04-02T09:12:26.184901+0000","last_clean":"2024-04-02T09:12:26.184901+0000","last_became_active":"2024-04-02T09:12:25.204248+0000","last_became_peered":"2024-04-02T09:12:25.204248+0000","last_unstale":"2024-04-02T09:12:26.184901+0000","last_undegraded":"2024-04-02T09:12:26.184901+0000","last_fullsized":"2024-04-02T09:12:26.184901+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:32:03.502011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422702+0000","last_change":"2024-04-02T09:12:25.198312+0000","last_active":"2024-04-02T09:12:26.422702+0000","last_peered":"2024-04-02T09:12:26.422702+0000","last_clean":"2024-04-02T09:12:26.422702+0000","last_became_active":"2024-04-02T09:12:25.198040+0000","last_became_peered":"2024-04-02T09:12:25.198040+0000","last_unstale":"2024-04-02T09:12:26.422702+0000","last_undegraded":"2024-04-02T09:12:26.422702+0000","last_fullsized":"2024-04-02T09:12:26.422702+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:02.893062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423721+0000","last_change":"2024-04-02T09:12:25.186757+0000","last_active":"2024-04-02T09:12:26.423721+0000","last_peered":"2024-04-02T09:12:26.423721+0000","last_clean":"2024-04-02T09:12:26.423721+0000","last_became_active":"2024-04-02T09:12:25.186436+0000","last_became_peered":"2024-04-02T09:12:25.186436+0000","last_unstale":"2024-04-02T09:12:26.423721+0000","last_undegraded":"2024-04-02T09:12:26.423721+0000","last_fullsized":"2024-04-02T09:12:26.423721+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:21.964569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422907+0000","last_change":"2024-04-02T09:12:25.188711+0000","last_active":"2024-04-02T09:12:26.422907+0000","last_peered":"2024-04-02T09:12:26.422907+0000","last_clean":"2024-04-02T09:12:26.422907+0000","last_became_active":"2024-04-02T09:12:25.188399+0000","last_became_peered":"2024-04-02T09:12:25.188399+0000","last_unstale":"2024-04-02T09:12:26.422907+0000","last_undegraded":"2024-04-02T09:12:26.422907+0000","last_fullsized":"2024-04-02T09:12:26.422907+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:20:21.844925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424498+0000","last_change":"2024-04-02T09:12:25.196993+0000","last_active":"2024-04-02T09:12:26.424498+0000","last_peered":"2024-04-02T09:12:26.424498+0000","last_clean":"2024-04-02T09:12:26.424498+0000","last_became_active":"2024-04-02T09:12:25.196316+0000","last_became_peered":"2024-04-02T09:12:25.196316+0000","last_unstale":"2024-04-02T09:12:26.424498+0000","last_undegraded":"2024-04-02T09:12:26.424498+0000","last_fullsized":"2024-04-02T09:12:26.424498+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:36:48.733839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423510+0000","last_change":"2024-04-02T09:12:25.207159+0000","last_active":"2024-04-02T09:12:26.423510+0000","last_peered":"2024-04-02T09:12:26.423510+0000","last_clean":"2024-04-02T09:12:26.423510+0000","last_became_active":"2024-04-02T09:12:25.206804+0000","last_became_peered":"2024-04-02T09:12:25.206804+0000","last_unstale":"2024-04-02T09:12:26.423510+0000","last_undegraded":"2024-04-02T09:12:26.423510+0000","last_fullsized":"2024-04-02T09:12:26.423510+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:19.646703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423916+0000","last_change":"2024-04-02T09:12:25.196715+0000","last_active":"2024-04-02T09:12:26.423916+0000","last_peered":"2024-04-02T09:12:26.423916+0000","last_clean":"2024-04-02T09:12:26.423916+0000","last_became_active":"2024-04-02T09:12:25.196433+0000","last_became_peered":"2024-04-02T09:12:25.196433+0000","last_unstale":"2024-04-02T09:12:26.423916+0000","last_undegraded":"2024-04-02T09:12:26.423916+0000","last_fullsized":"2024-04-02T09:12:26.423916+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:30.835437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423285+0000","last_change":"2024-04-02T09:12:25.196567+0000","last_active":"2024-04-02T09:12:26.423285+0000","last_peered":"2024-04-02T09:12:26.423285+0000","last_clean":"2024-04-02T09:12:26.423285+0000","last_became_active":"2024-04-02T09:12:25.196264+0000","last_became_peered":"2024-04-02T09:12:25.196264+0000","last_unstale":"2024-04-02T09:12:26.423285+0000","last_undegraded":"2024-04-02T09:12:26.423285+0000","last_fullsized":"2024-04-02T09:12:26.423285+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:03.387546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186592+0000","last_change":"2024-04-02T09:12:25.201010+0000","last_active":"2024-04-02T09:12:26.186592+0000","last_peered":"2024-04-02T09:12:26.186592+0000","last_clean":"2024-04-02T09:12:26.186592+0000","last_became_active":"2024-04-02T09:12:25.198423+0000","last_became_peered":"2024-04-02T09:12:25.198423+0000","last_unstale":"2024-04-02T09:12:26.186592+0000","last_undegraded":"2024-04-02T09:12:26.186592+0000","last_fullsized":"2024-04-02T09:12:26.186592+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:44:28.685030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422805+0000","last_change":"2024-04-02T09:12:25.197238+0000","last_active":"2024-04-02T09:12:26.422805+0000","last_peered":"2024-04-02T09:12:26.422805+0000","last_clean":"2024-04-02T09:12:26.422805+0000","last_became_active":"2024-04-02T09:12:25.196921+0000","last_became_peered":"2024-04-02T09:12:25.196921+0000","last_unstale":"2024-04-02T09:12:26.422805+0000","last_undegraded":"2024-04-02T09:12:26.422805+0000","last_fullsized":"2024-04-02T09:12:26.422805+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:40:34.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185295+0000","last_change":"2024-04-02T09:12:25.205832+0000","last_active":"2024-04-02T09:12:26.185295+0000","last_peered":"2024-04-02T09:12:26.185295+0000","last_clean":"2024-04-02T09:12:26.185295+0000","last_became_active":"2024-04-02T09:12:25.205206+0000","last_became_peered":"2024-04-02T09:12:25.205206+0000","last_unstale":"2024-04-02T09:12:26.185295+0000","last_undegraded":"2024-04-02T09:12:26.185295+0000","last_fullsized":"2024-04-02T09:12:26.185295+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:17.511411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422291+0000","last_change":"2024-04-02T09:12:25.185102+0000","last_active":"2024-04-02T09:12:26.422291+0000","last_peered":"2024-04-02T09:12:26.422291+0000","last_clean":"2024-04-02T09:12:26.422291+0000","last_became_active":"2024-04-02T09:12:25.184829+0000","last_became_peered":"2024-04-02T09:12:25.184829+0000","last_unstale":"2024-04-02T09:12:26.422291+0000","last_undegraded":"2024-04-02T09:12:26.422291+0000","last_fullsized":"2024-04-02T09:12:26.422291+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:28.343401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186057+0000","last_change":"2024-04-02T09:12:25.195404+0000","last_active":"2024-04-02T09:12:26.186057+0000","last_peered":"2024-04-02T09:12:26.186057+0000","last_clean":"2024-04-02T09:12:26.186057+0000","last_became_active":"2024-04-02T09:12:25.192051+0000","last_became_peered":"2024-04-02T09:12:25.192051+0000","last_unstale":"2024-04-02T09:12:26.186057+0000","last_undegraded":"2024-04-02T09:12:26.186057+0000","last_fullsized":"2024-04-02T09:12:26.186057+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:02.893062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186453+0000","last_change":"2024-04-02T09:12:25.200885+0000","last_active":"2024-04-02T09:12:26.186453+0000","last_peered":"2024-04-02T09:12:26.186453+0000","last_clean":"2024-04-02T09:12:26.186453+0000","last_became_active":"2024-04-02T09:12:25.200630+0000","last_became_peered":"2024-04-02T09:12:25.200630+0000","last_unstale":"2024-04-02T09:12:26.186453+0000","last_undegraded":"2024-04-02T09:12:26.186453+0000","last_fullsized":"2024-04-02T09:12:26.186453+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:18.798282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186550+0000","last_change":"2024-04-02T09:12:25.196804+0000","last_active":"2024-04-02T09:12:26.186550+0000","last_peered":"2024-04-02T09:12:26.186550+0000","last_clean":"2024-04-02T09:12:26.186550+0000","last_became_active":"2024-04-02T09:12:25.194411+0000","last_became_peered":"2024-04-02T09:12:25.194411+0000","last_unstale":"2024-04-02T09:12:26.186550+0000","last_undegraded":"2024-04-02T09:12:26.186550+0000","last_fullsized":"2024-04-02T09:12:26.186550+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:45:21.225286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186405+0000","last_change":"2024-04-02T09:12:25.192359+0000","last_active":"2024-04-02T09:12:26.186405+0000","last_peered":"2024-04-02T09:12:26.186405+0000","last_clean":"2024-04-02T09:12:26.186405+0000","last_became_active":"2024-04-02T09:12:25.192092+0000","last_became_peered":"2024-04-02T09:12:25.192092+0000","last_unstale":"2024-04-02T09:12:26.186405+0000","last_undegraded":"2024-04-02T09:12:26.186405+0000","last_fullsized":"2024-04-02T09:12:26.186405+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:01.618091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422831+0000","last_change":"2024-04-02T09:12:25.205454+0000","last_active":"2024-04-02T09:12:26.422831+0000","last_peered":"2024-04-02T09:12:26.422831+0000","last_clean":"2024-04-02T09:12:26.422831+0000","last_became_active":"2024-04-02T09:12:25.204157+0000","last_became_peered":"2024-04-02T09:12:25.204157+0000","last_unstale":"2024-04-02T09:12:26.422831+0000","last_undegraded":"2024-04-02T09:12:26.422831+0000","last_fullsized":"2024-04-02T09:12:26.422831+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:46:50.298119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.184933+0000","last_change":"2024-04-02T09:12:25.194990+0000","last_active":"2024-04-02T09:12:26.184933+0000","last_peered":"2024-04-02T09:12:26.184933+0000","last_clean":"2024-04-02T09:12:26.184933+0000","last_became_active":"2024-04-02T09:12:25.191623+0000","last_became_peered":"2024-04-02T09:12:25.191623+0000","last_unstale":"2024-04-02T09:12:26.184933+0000","last_undegraded":"2024-04-02T09:12:26.184933+0000","last_fullsized":"2024-04-02T09:12:26.184933+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:42:10.151562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422733+0000","last_change":"2024-04-02T09:12:25.203525+0000","last_active":"2024-04-02T09:12:26.422733+0000","last_peered":"2024-04-02T09:12:26.422733+0000","last_clean":"2024-04-02T09:12:26.422733+0000","last_became_active":"2024-04-02T09:12:25.202409+0000","last_became_peered":"2024-04-02T09:12:25.202409+0000","last_unstale":"2024-04-02T09:12:26.422733+0000","last_undegraded":"2024-04-02T09:12:26.422733+0000","last_fullsized":"2024-04-02T09:12:26.422733+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:38:54.918706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422701+0000","last_change":"2024-04-02T09:12:25.193316+0000","last_active":"2024-04-02T09:12:26.422701+0000","last_peered":"2024-04-02T09:12:26.422701+0000","last_clean":"2024-04-02T09:12:26.422701+0000","last_became_active":"2024-04-02T09:12:25.192672+0000","last_became_peered":"2024-04-02T09:12:25.192672+0000","last_unstale":"2024-04-02T09:12:26.422701+0000","last_undegraded":"2024-04-02T09:12:26.422701+0000","last_fullsized":"2024-04-02T09:12:26.422701+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:28.343401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186980+0000","last_change":"2024-04-02T09:12:25.195780+0000","last_active":"2024-04-02T09:12:26.186980+0000","last_peered":"2024-04-02T09:12:26.186980+0000","last_clean":"2024-04-02T09:12:26.186980+0000","last_became_active":"2024-04-02T09:12:25.194939+0000","last_became_peered":"2024-04-02T09:12:25.194939+0000","last_unstale":"2024-04-02T09:12:26.186980+0000","last_undegraded":"2024-04-02T09:12:26.186980+0000","last_fullsized":"2024-04-02T09:12:26.186980+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:20:21.844925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"222'4","reported_seq":15,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.187162+0000","last_change":"2024-04-02T09:12:24.695565+0000","last_active":"2024-04-02T09:12:26.187162+0000","last_peered":"2024-04-02T09:12:26.187162+0000","last_clean":"2024-04-02T09:12:26.187162+0000","last_became_active":"2024-04-02T09:12:24.695304+0000","last_became_peered":"2024-04-02T09:12:24.695304+0000","last_unstale":"2024-04-02T09:12:26.187162+0000","last_undegraded":"2024-04-02T09:12:26.187162+0000","last_fullsized":"2024-04-02T09:12:26.187162+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:56:16.018943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186201+0000","last_change":"2024-04-02T09:12:25.190209+0000","last_active":"2024-04-02T09:12:26.186201+0000","last_peered":"2024-04-02T09:12:26.186201+0000","last_clean":"2024-04-02T09:12:26.186201+0000","last_became_active":"2024-04-02T09:12:25.189908+0000","last_became_peered":"2024-04-02T09:12:25.189908+0000","last_unstale":"2024-04-02T09:12:26.186201+0000","last_undegraded":"2024-04-02T09:12:26.186201+0000","last_fullsized":"2024-04-02T09:12:26.186201+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:12.375324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1e","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423048+0000","last_change":"2024-04-02T09:12:24.684617+0000","last_active":"2024-04-02T09:12:26.423048+0000","last_peered":"2024-04-02T09:12:26.423048+0000","last_clean":"2024-04-02T09:12:26.423048+0000","last_became_active":"2024-04-02T09:12:24.684367+0000","last_became_peered":"2024-04-02T09:12:24.684367+0000","last_unstale":"2024-04-02T09:12:26.423048+0000","last_undegraded":"2024-04-02T09:12:26.423048+0000","last_fullsized":"2024-04-02T09:12:26.423048+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:48:45.999478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423420+0000","last_change":"2024-04-02T09:12:25.204722+0000","last_active":"2024-04-02T09:12:26.423420+0000","last_peered":"2024-04-02T09:12:26.423420+0000","last_clean":"2024-04-02T09:12:26.423420+0000","last_became_active":"2024-04-02T09:12:25.204031+0000","last_became_peered":"2024-04-02T09:12:25.204031+0000","last_unstale":"2024-04-02T09:12:26.423420+0000","last_undegraded":"2024-04-02T09:12:26.423420+0000","last_fullsized":"2024-04-02T09:12:26.423420+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:00:19.545624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186561+0000","last_change":"2024-04-02T09:12:24.691669+0000","last_active":"2024-04-02T09:12:26.186561+0000","last_peered":"2024-04-02T09:12:26.186561+0000","last_clean":"2024-04-02T09:12:26.186561+0000","last_became_active":"2024-04-02T09:12:24.689803+0000","last_became_peered":"2024-04-02T09:12:24.689803+0000","last_unstale":"2024-04-02T09:12:26.186561+0000","last_undegraded":"2024-04-02T09:12:26.186561+0000","last_fullsized":"2024-04-02T09:12:26.186561+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:25:15.021325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423945+0000","last_change":"2024-04-02T09:12:25.194989+0000","last_active":"2024-04-02T09:12:26.423945+0000","last_peered":"2024-04-02T09:12:26.423945+0000","last_clean":"2024-04-02T09:12:26.423945+0000","last_became_active":"2024-04-02T09:12:25.194466+0000","last_became_peered":"2024-04-02T09:12:25.194466+0000","last_unstale":"2024-04-02T09:12:26.423945+0000","last_undegraded":"2024-04-02T09:12:26.423945+0000","last_fullsized":"2024-04-02T09:12:26.423945+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:01.618091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1c","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186076+0000","last_change":"2024-04-02T09:12:24.692574+0000","last_active":"2024-04-02T09:12:26.186076+0000","last_peered":"2024-04-02T09:12:26.186076+0000","last_clean":"2024-04-02T09:12:26.186076+0000","last_became_active":"2024-04-02T09:12:24.692283+0000","last_became_peered":"2024-04-02T09:12:24.692283+0000","last_unstale":"2024-04-02T09:12:26.186076+0000","last_undegraded":"2024-04-02T09:12:26.186076+0000","last_fullsized":"2024-04-02T09:12:26.186076+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:56:59.029298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185775+0000","last_change":"2024-04-02T09:12:25.192982+0000","last_active":"2024-04-02T09:12:26.185775+0000","last_peered":"2024-04-02T09:12:26.185775+0000","last_clean":"2024-04-02T09:12:26.185775+0000","last_became_active":"2024-04-02T09:12:25.192737+0000","last_became_peered":"2024-04-02T09:12:25.192737+0000","last_unstale":"2024-04-02T09:12:26.185775+0000","last_undegraded":"2024-04-02T09:12:26.185775+0000","last_fullsized":"2024-04-02T09:12:26.185775+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:28.343401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424086+0000","last_change":"2024-04-02T09:12:24.690596+0000","last_active":"2024-04-02T09:12:26.424086+0000","last_peered":"2024-04-02T09:12:26.424086+0000","last_clean":"2024-04-02T09:12:26.424086+0000","last_became_active":"2024-04-02T09:12:24.690029+0000","last_became_peered":"2024-04-02T09:12:24.690029+0000","last_unstale":"2024-04-02T09:12:26.424086+0000","last_undegraded":"2024-04-02T09:12:26.424086+0000","last_fullsized":"2024-04-02T09:12:26.424086+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:00.608890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423300+0000","last_change":"2024-04-02T09:12:25.191145+0000","last_active":"2024-04-02T09:12:26.423300+0000","last_peered":"2024-04-02T09:12:26.423300+0000","last_clean":"2024-04-02T09:12:26.423300+0000","last_became_active":"2024-04-02T09:12:25.190868+0000","last_became_peered":"2024-04-02T09:12:25.190868+0000","last_unstale":"2024-04-02T09:12:26.423300+0000","last_undegraded":"2024-04-02T09:12:26.423300+0000","last_fullsized":"2024-04-02T09:12:26.423300+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:41.683998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186094+0000","last_change":"2024-04-02T09:12:24.698972+0000","last_active":"2024-04-02T09:12:26.186094+0000","last_peered":"2024-04-02T09:12:26.186094+0000","last_clean":"2024-04-02T09:12:26.186094+0000","last_became_active":"2024-04-02T09:12:24.698664+0000","last_became_peered":"2024-04-02T09:12:24.698664+0000","last_unstale":"2024-04-02T09:12:26.186094+0000","last_undegraded":"2024-04-02T09:12:26.186094+0000","last_fullsized":"2024-04-02T09:12:26.186094+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:11:06.573162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422499+0000","last_change":"2024-04-02T09:12:25.198098+0000","last_active":"2024-04-02T09:12:26.422499+0000","last_peered":"2024-04-02T09:12:26.422499+0000","last_clean":"2024-04-02T09:12:26.422499+0000","last_became_active":"2024-04-02T09:12:25.197848+0000","last_became_peered":"2024-04-02T09:12:25.197848+0000","last_unstale":"2024-04-02T09:12:26.422499+0000","last_undegraded":"2024-04-02T09:12:26.422499+0000","last_fullsized":"2024-04-02T09:12:26.422499+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:02.893062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.184920+0000","last_change":"2024-04-02T09:12:24.698699+0000","last_active":"2024-04-02T09:12:26.184920+0000","last_peered":"2024-04-02T09:12:26.184920+0000","last_clean":"2024-04-02T09:12:26.184920+0000","last_became_active":"2024-04-02T09:12:24.698347+0000","last_became_peered":"2024-04-02T09:12:24.698347+0000","last_unstale":"2024-04-02T09:12:26.184920+0000","last_undegraded":"2024-04-02T09:12:26.184920+0000","last_fullsized":"2024-04-02T09:12:26.184920+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:44:22.620155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186150+0000","last_change":"2024-04-02T09:12:25.192446+0000","last_active":"2024-04-02T09:12:26.186150+0000","last_peered":"2024-04-02T09:12:26.186150+0000","last_clean":"2024-04-02T09:12:26.186150+0000","last_became_active":"2024-04-02T09:12:25.192159+0000","last_became_peered":"2024-04-02T09:12:25.192159+0000","last_unstale":"2024-04-02T09:12:26.186150+0000","last_undegraded":"2024-04-02T09:12:26.186150+0000","last_fullsized":"2024-04-02T09:12:26.186150+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:38.903726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423659+0000","last_change":"2024-04-02T09:12:24.697017+0000","last_active":"2024-04-02T09:12:26.423659+0000","last_peered":"2024-04-02T09:12:26.423659+0000","last_clean":"2024-04-02T09:12:26.423659+0000","last_became_active":"2024-04-02T09:12:24.696537+0000","last_became_peered":"2024-04-02T09:12:24.696537+0000","last_unstale":"2024-04-02T09:12:26.423659+0000","last_undegraded":"2024-04-02T09:12:26.423659+0000","last_fullsized":"2024-04-02T09:12:26.423659+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:16.861790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423163+0000","last_change":"2024-04-02T09:12:25.198058+0000","last_active":"2024-04-02T09:12:26.423163+0000","last_peered":"2024-04-02T09:12:26.423163+0000","last_clean":"2024-04-02T09:12:26.423163+0000","last_became_active":"2024-04-02T09:12:25.197791+0000","last_became_peered":"2024-04-02T09:12:25.197791+0000","last_unstale":"2024-04-02T09:12:26.423163+0000","last_undegraded":"2024-04-02T09:12:26.423163+0000","last_fullsized":"2024-04-02T09:12:26.423163+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:38.903726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186231+0000","last_change":"2024-04-02T09:12:24.690012+0000","last_active":"2024-04-02T09:12:26.186231+0000","last_peered":"2024-04-02T09:12:26.186231+0000","last_clean":"2024-04-02T09:12:26.186231+0000","last_became_active":"2024-04-02T09:12:24.689551+0000","last_became_peered":"2024-04-02T09:12:24.689551+0000","last_unstale":"2024-04-02T09:12:26.186231+0000","last_undegraded":"2024-04-02T09:12:26.186231+0000","last_fullsized":"2024-04-02T09:12:26.186231+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:15.316332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422487+0000","last_change":"2024-04-02T09:12:25.202617+0000","last_active":"2024-04-02T09:12:26.422487+0000","last_peered":"2024-04-02T09:12:26.422487+0000","last_clean":"2024-04-02T09:12:26.422487+0000","last_became_active":"2024-04-02T09:12:25.201786+0000","last_became_peered":"2024-04-02T09:12:25.201786+0000","last_unstale":"2024-04-02T09:12:26.422487+0000","last_undegraded":"2024-04-02T09:12:26.422487+0000","last_fullsized":"2024-04-02T09:12:26.422487+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:32:03.502011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"222'3","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186588+0000","last_change":"2024-04-02T09:12:24.694400+0000","last_active":"2024-04-02T09:12:26.186588+0000","last_peered":"2024-04-02T09:12:26.186588+0000","last_clean":"2024-04-02T09:12:26.186588+0000","last_became_active":"2024-04-02T09:12:24.694118+0000","last_became_peered":"2024-04-02T09:12:24.694118+0000","last_unstale":"2024-04-02T09:12:26.186588+0000","last_undegraded":"2024-04-02T09:12:26.186588+0000","last_fullsized":"2024-04-02T09:12:26.186588+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:00.841741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186527+0000","last_change":"2024-04-02T09:12:25.199901+0000","last_active":"2024-04-02T09:12:26.186527+0000","last_peered":"2024-04-02T09:12:26.186527+0000","last_clean":"2024-04-02T09:12:26.186527+0000","last_became_active":"2024-04-02T09:12:25.198989+0000","last_became_peered":"2024-04-02T09:12:25.198989+0000","last_unstale":"2024-04-02T09:12:26.186527+0000","last_undegraded":"2024-04-02T09:12:26.186527+0000","last_fullsized":"2024-04-02T09:12:26.186527+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:30.835437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"222'6","reported_seq":17,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422587+0000","last_change":"2024-04-02T09:12:24.689123+0000","last_active":"2024-04-02T09:12:26.422587+0000","last_peered":"2024-04-02T09:12:26.422587+0000","last_clean":"2024-04-02T09:12:26.422587+0000","last_became_active":"2024-04-02T09:12:24.688848+0000","last_became_peered":"2024-04-02T09:12:24.688848+0000","last_unstale":"2024-04-02T09:12:26.422587+0000","last_undegraded":"2024-04-02T09:12:26.422587+0000","last_fullsized":"2024-04-02T09:12:26.422587+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:24.593956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423130+0000","last_change":"2024-04-02T09:12:25.192240+0000","last_active":"2024-04-02T09:12:26.423130+0000","last_peered":"2024-04-02T09:12:26.423130+0000","last_clean":"2024-04-02T09:12:26.423130+0000","last_became_active":"2024-04-02T09:12:25.191954+0000","last_became_peered":"2024-04-02T09:12:25.191954+0000","last_unstale":"2024-04-02T09:12:26.423130+0000","last_undegraded":"2024-04-02T09:12:26.423130+0000","last_fullsized":"2024-04-02T09:12:26.423130+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:26:33.117159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.13","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422301+0000","last_change":"2024-04-02T09:12:24.694118+0000","last_active":"2024-04-02T09:12:26.422301+0000","last_peered":"2024-04-02T09:12:26.422301+0000","last_clean":"2024-04-02T09:12:26.422301+0000","last_became_active":"2024-04-02T09:12:24.693829+0000","last_became_peered":"2024-04-02T09:12:24.693829+0000","last_unstale":"2024-04-02T09:12:26.422301+0000","last_undegraded":"2024-04-02T09:12:26.422301+0000","last_fullsized":"2024-04-02T09:12:26.422301+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:58:51.743784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423515+0000","last_change":"2024-04-02T09:12:25.205479+0000","last_active":"2024-04-02T09:12:26.423515+0000","last_peered":"2024-04-02T09:12:26.423515+0000","last_clean":"2024-04-02T09:12:26.423515+0000","last_became_active":"2024-04-02T09:12:25.205210+0000","last_became_peered":"2024-04-02T09:12:25.205210+0000","last_unstale":"2024-04-02T09:12:26.423515+0000","last_undegraded":"2024-04-02T09:12:26.423515+0000","last_fullsized":"2024-04-02T09:12:26.423515+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:03.985819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422807+0000","last_change":"2024-04-02T09:12:24.695778+0000","last_active":"2024-04-02T09:12:26.422807+0000","last_peered":"2024-04-02T09:12:26.422807+0000","last_clean":"2024-04-02T09:12:26.422807+0000","last_became_active":"2024-04-02T09:12:24.695349+0000","last_became_peered":"2024-04-02T09:12:24.695349+0000","last_unstale":"2024-04-02T09:12:26.422807+0000","last_undegraded":"2024-04-02T09:12:26.422807+0000","last_fullsized":"2024-04-02T09:12:26.422807+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:33:51.021792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422685+0000","last_change":"2024-04-02T09:12:25.203805+0000","last_active":"2024-04-02T09:12:26.422685+0000","last_peered":"2024-04-02T09:12:26.422685+0000","last_clean":"2024-04-02T09:12:26.422685+0000","last_became_active":"2024-04-02T09:12:25.203516+0000","last_became_peered":"2024-04-02T09:12:25.203516+0000","last_unstale":"2024-04-02T09:12:26.422685+0000","last_undegraded":"2024-04-02T09:12:26.422685+0000","last_fullsized":"2024-04-02T09:12:26.422685+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:00:15.805719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186210+0000","last_change":"2024-04-02T09:12:24.691834+0000","last_active":"2024-04-02T09:12:26.186210+0000","last_peered":"2024-04-02T09:12:26.186210+0000","last_clean":"2024-04-02T09:12:26.186210+0000","last_became_active":"2024-04-02T09:12:24.689829+0000","last_became_peered":"2024-04-02T09:12:24.689829+0000","last_unstale":"2024-04-02T09:12:26.186210+0000","last_undegraded":"2024-04-02T09:12:26.186210+0000","last_fullsized":"2024-04-02T09:12:26.186210+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:56.694970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423633+0000","last_change":"2024-04-02T09:12:25.201746+0000","last_active":"2024-04-02T09:12:26.423633+0000","last_peered":"2024-04-02T09:12:26.423633+0000","last_clean":"2024-04-02T09:12:26.423633+0000","last_became_active":"2024-04-02T09:12:25.201445+0000","last_became_peered":"2024-04-02T09:12:25.201445+0000","last_unstale":"2024-04-02T09:12:26.423633+0000","last_undegraded":"2024-04-02T09:12:26.423633+0000","last_fullsized":"2024-04-02T09:12:26.423633+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:06:08.648823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.10","version":"222'8","reported_seq":19,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186388+0000","last_change":"2024-04-02T09:12:24.692205+0000","last_active":"2024-04-02T09:12:26.186388+0000","last_peered":"2024-04-02T09:12:26.186388+0000","last_clean":"2024-04-02T09:12:26.186388+0000","last_became_active":"2024-04-02T09:12:24.690462+0000","last_became_peered":"2024-04-02T09:12:24.690462+0000","last_unstale":"2024-04-02T09:12:26.186388+0000","last_undegraded":"2024-04-02T09:12:26.186388+0000","last_fullsized":"2024-04-02T09:12:26.186388+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:00:14.796518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.187135+0000","last_change":"2024-04-02T09:12:25.196196+0000","last_active":"2024-04-02T09:12:26.187135+0000","last_peered":"2024-04-02T09:12:26.187135+0000","last_clean":"2024-04-02T09:12:26.187135+0000","last_became_active":"2024-04-02T09:12:25.195967+0000","last_became_peered":"2024-04-02T09:12:25.195967+0000","last_unstale":"2024-04-02T09:12:26.187135+0000","last_undegraded":"2024-04-02T09:12:26.187135+0000","last_fullsized":"2024-04-02T09:12:26.187135+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:41.683998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.f","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424609+0000","last_change":"2024-04-02T09:12:24.695567+0000","last_active":"2024-04-02T09:12:26.424609+0000","last_peered":"2024-04-02T09:12:26.424609+0000","last_clean":"2024-04-02T09:12:26.424609+0000","last_became_active":"2024-04-02T09:12:24.695267+0000","last_became_peered":"2024-04-02T09:12:24.695267+0000","last_unstale":"2024-04-02T09:12:26.424609+0000","last_undegraded":"2024-04-02T09:12:26.424609+0000","last_fullsized":"2024-04-02T09:12:26.424609+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:02.378346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186447+0000","last_change":"2024-04-02T09:12:25.196053+0000","last_active":"2024-04-02T09:12:26.186447+0000","last_peered":"2024-04-02T09:12:26.186447+0000","last_clean":"2024-04-02T09:12:26.186447+0000","last_became_active":"2024-04-02T09:12:25.195799+0000","last_became_peered":"2024-04-02T09:12:25.195799+0000","last_unstale":"2024-04-02T09:12:26.186447+0000","last_undegraded":"2024-04-02T09:12:26.186447+0000","last_fullsized":"2024-04-02T09:12:26.186447+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:26:33.117159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422476+0000","last_change":"2024-04-02T09:12:24.690213+0000","last_active":"2024-04-02T09:12:26.422476+0000","last_peered":"2024-04-02T09:12:26.422476+0000","last_clean":"2024-04-02T09:12:26.422476+0000","last_became_active":"2024-04-02T09:12:24.689813+0000","last_became_peered":"2024-04-02T09:12:24.689813+0000","last_unstale":"2024-04-02T09:12:26.422476+0000","last_undegraded":"2024-04-02T09:12:26.422476+0000","last_fullsized":"2024-04-02T09:12:26.422476+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:11.366124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423670+0000","last_change":"2024-04-02T09:12:25.193723+0000","last_active":"2024-04-02T09:12:26.423670+0000","last_peered":"2024-04-02T09:12:26.423670+0000","last_clean":"2024-04-02T09:12:26.423670+0000","last_became_active":"2024-04-02T09:12:25.193367+0000","last_became_peered":"2024-04-02T09:12:25.193367+0000","last_unstale":"2024-04-02T09:12:26.423670+0000","last_undegraded":"2024-04-02T09:12:26.423670+0000","last_fullsized":"2024-04-02T09:12:26.423670+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:48.218780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.d","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423498+0000","last_change":"2024-04-02T09:12:24.688637+0000","last_active":"2024-04-02T09:12:26.423498+0000","last_peered":"2024-04-02T09:12:26.423498+0000","last_clean":"2024-04-02T09:12:26.423498+0000","last_became_active":"2024-04-02T09:12:24.688321+0000","last_became_peered":"2024-04-02T09:12:24.688321+0000","last_unstale":"2024-04-02T09:12:26.423498+0000","last_undegraded":"2024-04-02T09:12:26.423498+0000","last_fullsized":"2024-04-02T09:12:26.423498+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:41:28.231249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424421+0000","last_change":"2024-04-02T09:12:25.190936+0000","last_active":"2024-04-02T09:12:26.424421+0000","last_peered":"2024-04-02T09:12:26.424421+0000","last_clean":"2024-04-02T09:12:26.424421+0000","last_became_active":"2024-04-02T09:12:25.189970+0000","last_became_peered":"2024-04-02T09:12:25.189970+0000","last_unstale":"2024-04-02T09:12:26.424421+0000","last_undegraded":"2024-04-02T09:12:26.424421+0000","last_fullsized":"2024-04-02T09:12:26.424421+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:57.372310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424062+0000","last_change":"2024-04-02T09:12:24.692444+0000","last_active":"2024-04-02T09:12:26.424062+0000","last_peered":"2024-04-02T09:12:26.424062+0000","last_clean":"2024-04-02T09:12:26.424062+0000","last_became_active":"2024-04-02T09:12:24.692165+0000","last_became_peered":"2024-04-02T09:12:24.692165+0000","last_unstale":"2024-04-02T09:12:26.424062+0000","last_undegraded":"2024-04-02T09:12:26.424062+0000","last_fullsized":"2024-04-02T09:12:26.424062+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:29.826236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423682+0000","last_change":"2024-04-02T09:12:25.195964+0000","last_active":"2024-04-02T09:12:26.423682+0000","last_peered":"2024-04-02T09:12:26.423682+0000","last_clean":"2024-04-02T09:12:26.423682+0000","last_became_active":"2024-04-02T09:12:25.195716+0000","last_became_peered":"2024-04-02T09:12:25.195716+0000","last_unstale":"2024-04-02T09:12:26.423682+0000","last_undegraded":"2024-04-02T09:12:26.423682+0000","last_fullsized":"2024-04-02T09:12:26.423682+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:25.408279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.b","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423122+0000","last_change":"2024-04-02T09:12:24.693402+0000","last_active":"2024-04-02T09:12:26.423122+0000","last_peered":"2024-04-02T09:12:26.423122+0000","last_clean":"2024-04-02T09:12:26.423122+0000","last_became_active":"2024-04-02T09:12:24.693047+0000","last_became_peered":"2024-04-02T09:12:24.693047+0000","last_unstale":"2024-04-02T09:12:26.423122+0000","last_undegraded":"2024-04-02T09:12:26.423122+0000","last_fullsized":"2024-04-02T09:12:26.423122+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:15:17.948098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424048+0000","last_change":"2024-04-02T09:12:25.195866+0000","last_active":"2024-04-02T09:12:26.424048+0000","last_peered":"2024-04-02T09:12:26.424048+0000","last_clean":"2024-04-02T09:12:26.424048+0000","last_became_active":"2024-04-02T09:12:25.195430+0000","last_became_peered":"2024-04-02T09:12:25.195430+0000","last_unstale":"2024-04-02T09:12:26.424048+0000","last_undegraded":"2024-04-02T09:12:26.424048+0000","last_fullsized":"2024-04-02T09:12:26.424048+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:42:28.562906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186235+0000","last_change":"2024-04-02T09:12:24.691812+0000","last_active":"2024-04-02T09:12:26.186235+0000","last_peered":"2024-04-02T09:12:26.186235+0000","last_clean":"2024-04-02T09:12:26.186235+0000","last_became_active":"2024-04-02T09:12:24.688935+0000","last_became_peered":"2024-04-02T09:12:24.688935+0000","last_unstale":"2024-04-02T09:12:26.186235+0000","last_undegraded":"2024-04-02T09:12:26.186235+0000","last_fullsized":"2024-04-02T09:12:26.186235+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:53.020585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185018+0000","last_change":"2024-04-02T09:12:25.189098+0000","last_active":"2024-04-02T09:12:26.185018+0000","last_peered":"2024-04-02T09:12:26.185018+0000","last_clean":"2024-04-02T09:12:26.185018+0000","last_became_active":"2024-04-02T09:12:25.187377+0000","last_became_peered":"2024-04-02T09:12:25.187377+0000","last_unstale":"2024-04-02T09:12:26.185018+0000","last_undegraded":"2024-04-02T09:12:26.185018+0000","last_fullsized":"2024-04-02T09:12:26.185018+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:58:52.752985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422824+0000","last_change":"2024-04-02T09:12:24.699586+0000","last_active":"2024-04-02T09:12:26.422824+0000","last_peered":"2024-04-02T09:12:26.422824+0000","last_clean":"2024-04-02T09:12:26.422824+0000","last_became_active":"2024-04-02T09:12:24.697592+0000","last_became_peered":"2024-04-02T09:12:24.697592+0000","last_unstale":"2024-04-02T09:12:26.422824+0000","last_undegraded":"2024-04-02T09:12:26.422824+0000","last_fullsized":"2024-04-02T09:12:26.422824+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:53.870135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185386+0000","last_change":"2024-04-02T09:12:25.205206+0000","last_active":"2024-04-02T09:12:26.185386+0000","last_peered":"2024-04-02T09:12:26.185386+0000","last_clean":"2024-04-02T09:12:26.185386+0000","last_became_active":"2024-04-02T09:12:25.204526+0000","last_became_peered":"2024-04-02T09:12:25.204526+0000","last_unstale":"2024-04-02T09:12:26.185386+0000","last_undegraded":"2024-04-02T09:12:26.185386+0000","last_fullsized":"2024-04-02T09:12:26.185386+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:33:41.191493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186038+0000","last_change":"2024-04-02T09:12:24.683516+0000","last_active":"2024-04-02T09:12:26.186038+0000","last_peered":"2024-04-02T09:12:26.186038+0000","last_clean":"2024-04-02T09:12:26.186038+0000","last_became_active":"2024-04-02T09:12:24.683207+0000","last_became_peered":"2024-04-02T09:12:24.683207+0000","last_unstale":"2024-04-02T09:12:26.186038+0000","last_undegraded":"2024-04-02T09:12:26.186038+0000","last_fullsized":"2024-04-02T09:12:26.186038+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:19:15.076490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.187152+0000","last_change":"2024-04-02T09:12:25.196365+0000","last_active":"2024-04-02T09:12:26.187152+0000","last_peered":"2024-04-02T09:12:26.187152+0000","last_clean":"2024-04-02T09:12:26.187152+0000","last_became_active":"2024-04-02T09:12:25.191868+0000","last_became_peered":"2024-04-02T09:12:25.191868+0000","last_unstale":"2024-04-02T09:12:26.187152+0000","last_undegraded":"2024-04-02T09:12:26.187152+0000","last_fullsized":"2024-04-02T09:12:26.187152+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:43:29.229527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186316+0000","last_change":"2024-04-02T09:12:24.696113+0000","last_active":"2024-04-02T09:12:26.186316+0000","last_peered":"2024-04-02T09:12:26.186316+0000","last_clean":"2024-04-02T09:12:26.186316+0000","last_became_active":"2024-04-02T09:12:24.695191+0000","last_became_peered":"2024-04-02T09:12:24.695191+0000","last_unstale":"2024-04-02T09:12:26.186316+0000","last_undegraded":"2024-04-02T09:12:26.186316+0000","last_fullsized":"2024-04-02T09:12:26.186316+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:41:21.090616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424613+0000","last_change":"2024-04-02T09:12:25.196988+0000","last_active":"2024-04-02T09:12:26.424613+0000","last_peered":"2024-04-02T09:12:26.424613+0000","last_clean":"2024-04-02T09:12:26.424613+0000","last_became_active":"2024-04-02T09:12:25.196031+0000","last_became_peered":"2024-04-02T09:12:25.196031+0000","last_unstale":"2024-04-02T09:12:26.424613+0000","last_undegraded":"2024-04-02T09:12:26.424613+0000","last_fullsized":"2024-04-02T09:12:26.424613+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:50.975576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423068+0000","last_change":"2024-04-02T09:12:24.686970+0000","last_active":"2024-04-02T09:12:26.423068+0000","last_peered":"2024-04-02T09:12:26.423068+0000","last_clean":"2024-04-02T09:12:26.423068+0000","last_became_active":"2024-04-02T09:12:24.686686+0000","last_became_peered":"2024-04-02T09:12:24.686686+0000","last_unstale":"2024-04-02T09:12:26.423068+0000","last_undegraded":"2024-04-02T09:12:26.423068+0000","last_fullsized":"2024-04-02T09:12:26.423068+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:17.201623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423095+0000","last_change":"2024-04-02T09:12:25.200491+0000","last_active":"2024-04-02T09:12:26.423095+0000","last_peered":"2024-04-02T09:12:26.423095+0000","last_clean":"2024-04-02T09:12:26.423095+0000","last_became_active":"2024-04-02T09:12:25.200208+0000","last_became_peered":"2024-04-02T09:12:25.200208+0000","last_unstale":"2024-04-02T09:12:26.423095+0000","last_undegraded":"2024-04-02T09:12:26.423095+0000","last_fullsized":"2024-04-02T09:12:26.423095+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:53:44.514037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424438+0000","last_change":"2024-04-02T09:12:24.691094+0000","last_active":"2024-04-02T09:12:26.424438+0000","last_peered":"2024-04-02T09:12:26.424438+0000","last_clean":"2024-04-02T09:12:26.424438+0000","last_became_active":"2024-04-02T09:12:24.690782+0000","last_became_peered":"2024-04-02T09:12:24.690782+0000","last_unstale":"2024-04-02T09:12:26.424438+0000","last_undegraded":"2024-04-02T09:12:26.424438+0000","last_fullsized":"2024-04-02T09:12:26.424438+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:17.789082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186855+0000","last_change":"2024-04-02T09:12:25.196931+0000","last_active":"2024-04-02T09:12:26.186855+0000","last_peered":"2024-04-02T09:12:26.186855+0000","last_clean":"2024-04-02T09:12:26.186855+0000","last_became_active":"2024-04-02T09:12:25.194947+0000","last_became_peered":"2024-04-02T09:12:25.194947+0000","last_unstale":"2024-04-02T09:12:26.186855+0000","last_undegraded":"2024-04-02T09:12:26.186855+0000","last_fullsized":"2024-04-02T09:12:26.186855+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:21.964569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424196+0000","last_change":"2024-04-02T09:12:24.693871+0000","last_active":"2024-04-02T09:12:26.424196+0000","last_peered":"2024-04-02T09:12:26.424196+0000","last_clean":"2024-04-02T09:12:26.424196+0000","last_became_active":"2024-04-02T09:12:24.693562+0000","last_became_peered":"2024-04-02T09:12:24.693562+0000","last_unstale":"2024-04-02T09:12:26.424196+0000","last_undegraded":"2024-04-02T09:12:26.424196+0000","last_fullsized":"2024-04-02T09:12:26.424196+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:48:24.676039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424040+0000","last_change":"2024-04-02T09:12:25.194789+0000","last_active":"2024-04-02T09:12:26.424040+0000","last_peered":"2024-04-02T09:12:26.424040+0000","last_clean":"2024-04-02T09:12:26.424040+0000","last_became_active":"2024-04-02T09:12:25.192429+0000","last_became_peered":"2024-04-02T09:12:25.192429+0000","last_unstale":"2024-04-02T09:12:26.424040+0000","last_undegraded":"2024-04-02T09:12:26.424040+0000","last_fullsized":"2024-04-02T09:12:26.424040+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:11:59.558881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185795+0000","last_change":"2024-04-02T09:12:24.693539+0000","last_active":"2024-04-02T09:12:26.185795+0000","last_peered":"2024-04-02T09:12:26.185795+0000","last_clean":"2024-04-02T09:12:26.185795+0000","last_became_active":"2024-04-02T09:12:24.693275+0000","last_became_peered":"2024-04-02T09:12:24.693275+0000","last_unstale":"2024-04-02T09:12:26.185795+0000","last_undegraded":"2024-04-02T09:12:26.185795+0000","last_fullsized":"2024-04-02T09:12:26.185795+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:22:34.495765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422860+0000","last_change":"2024-04-02T09:12:25.197899+0000","last_active":"2024-04-02T09:12:26.422860+0000","last_peered":"2024-04-02T09:12:26.422860+0000","last_clean":"2024-04-02T09:12:26.422860+0000","last_became_active":"2024-04-02T09:12:25.197631+0000","last_became_peered":"2024-04-02T09:12:25.197631+0000","last_unstale":"2024-04-02T09:12:26.422860+0000","last_undegraded":"2024-04-02T09:12:26.422860+0000","last_fullsized":"2024-04-02T09:12:26.422860+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:02.349107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423712+0000","last_change":"2024-04-02T09:12:24.694385+0000","last_active":"2024-04-02T09:12:26.423712+0000","last_peered":"2024-04-02T09:12:26.423712+0000","last_clean":"2024-04-02T09:12:26.423712+0000","last_became_active":"2024-04-02T09:12:24.693609+0000","last_became_peered":"2024-04-02T09:12:24.693609+0000","last_unstale":"2024-04-02T09:12:26.423712+0000","last_undegraded":"2024-04-02T09:12:26.423712+0000","last_fullsized":"2024-04-02T09:12:26.423712+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:46:08.099991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185973+0000","last_change":"2024-04-02T09:12:25.203957+0000","last_active":"2024-04-02T09:12:26.185973+0000","last_peered":"2024-04-02T09:12:26.185973+0000","last_clean":"2024-04-02T09:12:26.185973+0000","last_became_active":"2024-04-02T09:12:25.203680+0000","last_became_peered":"2024-04-02T09:12:25.203680+0000","last_unstale":"2024-04-02T09:12:26.185973+0000","last_undegraded":"2024-04-02T09:12:26.185973+0000","last_fullsized":"2024-04-02T09:12:26.185973+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:31.267333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186456+0000","last_change":"2024-04-02T09:12:24.695544+0000","last_active":"2024-04-02T09:12:26.186456+0000","last_peered":"2024-04-02T09:12:26.186456+0000","last_clean":"2024-04-02T09:12:26.186456+0000","last_became_active":"2024-04-02T09:12:24.695032+0000","last_became_peered":"2024-04-02T09:12:24.695032+0000","last_unstale":"2024-04-02T09:12:26.186456+0000","last_undegraded":"2024-04-02T09:12:26.186456+0000","last_fullsized":"2024-04-02T09:12:26.186456+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:41.705096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185482+0000","last_change":"2024-04-02T09:12:25.203302+0000","last_active":"2024-04-02T09:12:26.185482+0000","last_peered":"2024-04-02T09:12:26.185482+0000","last_clean":"2024-04-02T09:12:26.185482+0000","last_became_active":"2024-04-02T09:12:25.202579+0000","last_became_peered":"2024-04-02T09:12:25.202579+0000","last_unstale":"2024-04-02T09:12:26.185482+0000","last_undegraded":"2024-04-02T09:12:26.185482+0000","last_fullsized":"2024-04-02T09:12:26.185482+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:25.603157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185101+0000","last_change":"2024-04-02T09:12:24.698442+0000","last_active":"2024-04-02T09:12:26.185101+0000","last_peered":"2024-04-02T09:12:26.185101+0000","last_clean":"2024-04-02T09:12:26.185101+0000","last_became_active":"2024-04-02T09:12:24.698179+0000","last_became_peered":"2024-04-02T09:12:24.698179+0000","last_unstale":"2024-04-02T09:12:26.185101+0000","last_undegraded":"2024-04-02T09:12:26.185101+0000","last_fullsized":"2024-04-02T09:12:26.185101+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:48.278840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423599+0000","last_change":"2024-04-02T09:12:25.190434+0000","last_active":"2024-04-02T09:12:26.423599+0000","last_peered":"2024-04-02T09:12:26.423599+0000","last_clean":"2024-04-02T09:12:26.423599+0000","last_became_active":"2024-04-02T09:12:25.190187+0000","last_became_peered":"2024-04-02T09:12:25.190187+0000","last_unstale":"2024-04-02T09:12:26.423599+0000","last_undegraded":"2024-04-02T09:12:26.423599+0000","last_fullsized":"2024-04-02T09:12:26.423599+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:01.850942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422918+0000","last_change":"2024-04-02T09:12:24.687221+0000","last_active":"2024-04-02T09:12:26.422918+0000","last_peered":"2024-04-02T09:12:26.422918+0000","last_clean":"2024-04-02T09:12:26.422918+0000","last_became_active":"2024-04-02T09:12:24.686953+0000","last_became_peered":"2024-04-02T09:12:24.686953+0000","last_unstale":"2024-04-02T09:12:26.422918+0000","last_undegraded":"2024-04-02T09:12:26.422918+0000","last_fullsized":"2024-04-02T09:12:26.422918+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:39:31.379977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"211'348","reported_seq":1211,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-04-02T09:12:25.179741+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:12:25.179741+0000","last_peered":"2024-04-02T09:12:25.179741+0000","last_clean":"2024-04-02T09:12:25.179741+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:12:25.179741+0000","last_undegraded":"2024-04-02T09:12:25.179741+0000","last_fullsized":"2024-04-02T09:12:25.179741+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575988,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60084,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311756,"statfs":{"total":96636764160,"available":96575238144,"internally_reserved":0,"allocated":2789376,"data_stored":1412284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.68500000000000005,"15min":0.71199999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.93899999999999995,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":8.016},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68700000000000006,"15min":0.71999999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91800000000000004,"5min":0.94199999999999995,"15min":1.165},"last":5.1790000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.70599999999999996,"15min":0.73099999999999998},"min":{"1min":0.48899999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.95399999999999996,"5min":0.95699999999999996,"15min":1.125},"last":7.5819999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68100000000000005,"15min":0.71399999999999997},"min":{"1min":0.51700000000000002,"5min":0.497,"15min":0.42399999999999999},"max":{"1min":0.89100000000000001,"5min":0.92700000000000005,"15min":1.042},"last":4.2569999999999997}]},{"osd":2,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.749,"15min":0.76100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.873,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":7.9790000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.78200000000000003,"15min":0.80500000000000005},"min":{"1min":0.54900000000000004,"5min":0.504,"15min":0.504},"max":{"1min":0.98299999999999998,"5min":1.036,"15min":1.109},"last":4.8819999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.72399999999999998,"15min":0.74099999999999999},"min":{"1min":0.497,"5min":0.45800000000000002,"15min":0.41399999999999998},"max":{"1min":0.94399999999999995,"5min":1.452,"15min":1.452},"last":8.0530000000000008},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.748,"15min":0.76400000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.0740000000000001},"last":5.093}]},{"osd":4,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61699999999999999,"15min":0.60799999999999998},"min":{"1min":0.42799999999999999,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.95299999999999996,"15min":1.01},"last":5.4589999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.73699999999999999,"15min":0.71699999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.38300000000000001},"max":{"1min":1.006,"5min":1.536,"15min":1.536},"last":7.9409999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.68799999999999994,"15min":0.67500000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":0.93400000000000005,"5min":1.0169999999999999,"15min":1.1850000000000001},"last":5.1340000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.72499999999999998,"15min":0.71699999999999997},"min":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.40600000000000003},"max":{"1min":0.88500000000000001,"5min":1.097,"15min":1.1319999999999999},"last":3.492}]},{"osd":6,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45800000000000002},"max":{"1min":0.871,"5min":1.034,"15min":1.3919999999999999},"last":1.861},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.72299999999999998,"15min":0.69099999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.97299999999999998,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":7.8899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575986,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42040,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329800,"statfs":{"total":96636764160,"available":96593715200,"internally_reserved":0,"allocated":2789376,"data_stored":1412332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.73699999999999999},"min":{"1min":0.54900000000000004,"5min":0.50800000000000001,"15min":0.42699999999999999},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73199999999999998,"15min":0.747},"min":{"1min":0.48699999999999999,"5min":0.47899999999999998,"15min":0.45800000000000002},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":1.077},"last":0.55300000000000005}]},{"osd":1,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70399999999999996,"15min":0.72099999999999997},"min":{"1min":0.57199999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.212},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.73499999999999999,"15min":0.75800000000000001},"min":{"1min":0.55700000000000005,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.64000000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.73599999999999999,"15min":0.75800000000000001},"min":{"1min":0.53500000000000003,"5min":0.47699999999999998,"15min":0.47199999999999998},"max":{"1min":1.016,"5min":1.016,"15min":1.2789999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.73899999999999999,"15min":0.748},"min":{"1min":0.50900000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.93300000000000005,"5min":0.95999999999999996,"15min":1.226},"last":0.52000000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.74399999999999999,"15min":0.76900000000000002},"min":{"1min":0.50800000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.141,"5min":1.141,"15min":1.2549999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76900000000000002,"15min":0.77500000000000002},"min":{"1min":0.63100000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.94099999999999995,"5min":0.95399999999999996,"15min":1.3200000000000001},"last":0.61499999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60799999999999998,"15min":0.623},"min":{"1min":0.45300000000000001,"5min":0.42199999999999999,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.1259999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.65100000000000002,"15min":0.66700000000000004},"min":{"1min":0.51100000000000001,"5min":0.47399999999999998,"15min":0.41999999999999998},"max":{"1min":0.84399999999999997,"5min":0.92300000000000004,"15min":1.1879999999999999},"last":0.499}]},{"osd":5,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64500000000000002,"15min":0.65600000000000003},"min":{"1min":0.47899999999999998,"5min":0.42799999999999999,"15min":0.42199999999999999},"max":{"1min":0.86599999999999999,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70099999999999996,"15min":0.72599999999999998},"min":{"1min":0.505,"5min":0.48299999999999998,"15min":0.46700000000000003},"max":{"1min":0.92000000000000004,"5min":1.034,"15min":1.204},"last":0.69299999999999995}]},{"osd":7,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.2370000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.72199999999999998,"15min":0.73899999999999999},"min":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.48299999999999998},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.57899999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575987,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52488,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319352,"statfs":{"total":96636764160,"available":96583016448,"internally_reserved":0,"allocated":3391488,"data_stored":2009378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44400000000000001,"15min":0.45300000000000001},"min":{"1min":0.35499999999999998,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.67000000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.502,"15min":0.51200000000000001},"min":{"1min":0.371,"5min":0.312,"15min":0.312},"max":{"1min":0.76000000000000001,"5min":0.78600000000000003,"15min":0.79800000000000004},"last":0.48499999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.50900000000000001,"15min":0.505},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.753,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58299999999999996,"15min":0.59699999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74099999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.68100000000000005}]},{"osd":3,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.42699999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76200000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.505},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.40100000000000002,"5min":0.39000000000000001,"15min":0.378},"max":{"1min":0.73199999999999998,"5min":0.86699999999999999,"15min":0.878},"last":0.374}]},{"osd":4,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.79700000000000004,"15min":0.79200000000000004},"min":{"1min":0.61099999999999999,"5min":0.57599999999999996,"15min":0.433},"max":{"1min":0.98499999999999999,"5min":1.1120000000000001,"15min":1.1479999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.78000000000000003,"15min":0.78900000000000003},"min":{"1min":0.61299999999999999,"5min":0.57199999999999995,"15min":0.41799999999999998},"max":{"1min":1.05,"5min":1.05,"15min":1.25},"last":0.61299999999999999}]},{"osd":5,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.77700000000000002,"15min":0.79600000000000004},"min":{"1min":0.59799999999999998,"5min":0.54300000000000004,"15min":0.52900000000000003},"max":{"1min":1.01,"5min":1.115,"15min":1.321},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.65800000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.173},"last":0.754}]},{"osd":6,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.81200000000000006,"15min":0.81799999999999995},"min":{"1min":0.67300000000000004,"5min":0.53900000000000003,"15min":0.48099999999999998},"max":{"1min":1.0649999999999999,"5min":1.159,"15min":1.2210000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.82199999999999995,"15min":0.81299999999999994},"min":{"1min":0.61899999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.087,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.80900000000000005}]},{"osd":7,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.82099999999999995,"15min":0.82799999999999996},"min":{"1min":0.64200000000000002,"5min":0.58499999999999996,"15min":0.53700000000000003},"max":{"1min":1.0680000000000001,"5min":1.1080000000000001,"15min":1.236},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.83899999999999997,"15min":0.83999999999999997},"min":{"1min":0.63700000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0329999999999999,"5min":1.0720000000000001,"15min":1.2789999999999999},"last":0.66100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575987,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47804,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324036,"statfs":{"total":96636764160,"available":96587812864,"internally_reserved":0,"allocated":2797568,"data_stored":1418573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45400000000000001,"15min":0.45800000000000002},"min":{"1min":0.33700000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.69899999999999995,"5min":0.76500000000000001,"15min":0.86699999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.5,"15min":0.51000000000000001},"min":{"1min":0.32800000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.36899999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.42099999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66100000000000003,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53900000000000003,"15min":0.55400000000000005},"min":{"1min":0.36699999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72499999999999998,"5min":0.999,"15min":0.999},"last":0.495}]},{"osd":3,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.40100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.77800000000000002,"5min":0.80500000000000005,"15min":0.85699999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.63900000000000001,"15min":0.64300000000000002},"min":{"1min":0.5,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.82799999999999996,"5min":0.92000000000000004,"15min":0.96399999999999997},"last":0.63400000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.82499999999999996,"15min":0.83299999999999996},"min":{"1min":0.65700000000000003,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.2769999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81000000000000005,"15min":0.82399999999999995},"min":{"1min":0.621,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.83399999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.80100000000000005,"15min":0.82699999999999996},"min":{"1min":0.58899999999999997,"5min":0.497,"15min":0.497},"max":{"1min":1.1419999999999999,"5min":1.145,"15min":1.232},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.83499999999999996,"15min":0.83399999999999996},"min":{"1min":0.67700000000000005,"5min":0.57699999999999996,"15min":0.55900000000000005},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.84099999999999997}]},{"osd":6,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.58099999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1830000000000001,"5min":1.196,"15min":1.196},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.80500000000000005,"15min":0.81200000000000006},"min":{"1min":0.60499999999999998,"5min":0.52900000000000003,"15min":0.499},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.72599999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.84399999999999997,"15min":0.85399999999999998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.84099999999999997,"15min":0.85099999999999998},"min":{"1min":0.67700000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.1100000000000001,"5min":1.1719999999999999,"15min":1.2589999999999999},"last":0.76600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51908,"kb_used_data":2740,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319932,"statfs":{"total":96636764160,"available":96583610368,"internally_reserved":0,"allocated":2805760,"data_stored":1413930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44},"min":{"1min":0.32200000000000001,"5min":0.30499999999999999,"15min":0.29199999999999998},"max":{"1min":0.61499999999999999,"5min":0.66600000000000004,"15min":0.72999999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.502,"15min":0.503},"min":{"1min":0.375,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64800000000000002,"5min":0.79900000000000004,"15min":1.4930000000000001},"last":0.33200000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.40999999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65800000000000003,"5min":0.75800000000000001,"15min":0.80600000000000005},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.40100000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.68799999999999994,"5min":0.81999999999999995,"15min":1.4530000000000001},"last":0.39700000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.56000000000000005},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.80700000000000005,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.628,"15min":0.63500000000000001},"min":{"1min":0.50900000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77700000000000002,"5min":0.86799999999999999,"15min":1.0740000000000001},"last":0.48699999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.77700000000000002,"15min":0.77900000000000003},"min":{"1min":0.53700000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.008,"5min":1.121,"15min":1.121},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79300000000000004,"15min":0.79800000000000004},"min":{"1min":0.65200000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.1259999999999999,"5min":1.179,"15min":1.5329999999999999},"last":0.59699999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80000000000000004,"15min":0.79300000000000004},"min":{"1min":0.69899999999999995,"5min":0.53100000000000003,"15min":0.52600000000000002},"max":{"1min":1.054,"5min":1.99,"15min":1.99},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80500000000000005,"15min":0.80800000000000005},"min":{"1min":0.60499999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0660000000000001,"5min":1.101,"15min":1.1259999999999999},"last":0.73599999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.81599999999999995,"15min":0.81799999999999995},"min":{"1min":0.63500000000000001,"5min":0.54700000000000004,"15min":0.54600000000000004},"max":{"1min":0.96899999999999997,"5min":1.173,"15min":1.5940000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80400000000000005,"15min":0.81100000000000005},"min":{"1min":0.63200000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.99199999999999999,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.66600000000000004}]},{"osd":7,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.83199999999999996,"15min":0.83499999999999996},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55100000000000005},"max":{"1min":1.165,"5min":1.2110000000000001,"15min":1.5649999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.81999999999999995,"15min":0.83099999999999996},"min":{"1min":0.70499999999999996,"5min":0.497,"15min":0.497},"max":{"1min":1.0269999999999999,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.61499999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575986,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51900,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319940,"statfs":{"total":96636764160,"available":96583618560,"internally_reserved":0,"allocated":2797568,"data_stored":1413826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44400000000000001,"15min":0.45300000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.48699999999999999,"15min":0.496},"min":{"1min":0.36099999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.876},"last":0.55100000000000005}]},{"osd":1,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.49399999999999999,"15min":0.502},"min":{"1min":0.34300000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.64000000000000001,"5min":0.71499999999999997,"15min":0.90300000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.57699999999999996},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.34100000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.376}]},{"osd":2,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.622,"15min":0.63500000000000001},"min":{"1min":0.42599999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.61199999999999999,"15min":0.623},"min":{"1min":0.48099999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81699999999999995,"5min":0.82299999999999995,"15min":1.0840000000000001},"last":0.47999999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.74399999999999999,"15min":0.75900000000000001},"min":{"1min":0.503,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.071,"5min":1.071,"15min":1.0800000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.59099999999999997,"5min":0.57199999999999995,"15min":0.49199999999999999},"max":{"1min":1.018,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.75700000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.59599999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.054,"5min":1.1459999999999999,"15min":1.157},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.56599999999999995,"5min":0.505,"15min":0.505},"max":{"1min":0.99099999999999999,"5min":1.266,"15min":1.266},"last":0.83499999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.78500000000000003,"15min":0.79100000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.52600000000000002},"max":{"1min":1.0489999999999999,"5min":1.284,"15min":1.284},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.80400000000000005,"15min":0.81200000000000006},"min":{"1min":0.54900000000000004,"5min":0.495,"15min":0.495},"max":{"1min":1.04,"5min":1.2,"15min":1.2},"last":0.80700000000000005}]},{"osd":7,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.80400000000000005,"15min":0.81499999999999995},"min":{"1min":0.61399999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.024,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.57999999999999996,"5min":0.55700000000000005,"15min":0.53100000000000003},"max":{"1min":1.103,"5min":1.248,"15min":1.248},"last":0.79700000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575986,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47812,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324028,"statfs":{"total":96636764160,"available":96587804672,"internally_reserved":0,"allocated":2801664,"data_stored":1419100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68000000000000005,"15min":0.69899999999999995},"min":{"1min":0.55000000000000004,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.753},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66500000000000004,"15min":0.68899999999999995},"min":{"1min":0.40999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.83499999999999996,"5min":0.86199999999999999,"15min":1.075},"last":0.50900000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.73599999999999999,"15min":0.76100000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":1.0329999999999999,"5min":1.0549999999999999,"15min":1.069},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.75,"15min":0.77700000000000002},"min":{"1min":0.53900000000000003,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.70899999999999996}]},{"osd":2,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.69599999999999995,"15min":0.71799999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70499999999999996,"15min":0.73499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98999999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.623}]},{"osd":3,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75600000000000001,"15min":0.77000000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.47399999999999998},"max":{"1min":0.98099999999999998,"5min":0.998,"15min":1.0700000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.51800000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.002,"5min":1.002,"15min":1.028},"last":0.56899999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.60199999999999998,"15min":0.61099999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.91900000000000004,"5min":1.008,"15min":1.016},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60399999999999998,"15min":0.61699999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.63700000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.64100000000000001,"15min":0.64200000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67300000000000004,"15min":0.69099999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":0.98299999999999998,"5min":1.0269999999999999,"15min":1.0469999999999999},"last":0.78000000000000003}]},{"osd":7,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.0329999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.76600000000000001,"15min":0.78300000000000003},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0960000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.80100000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575989,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48380,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323460,"statfs":{"total":96636764160,"available":96587223040,"internally_reserved":0,"allocated":3391488,"data_stored":2002339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.68899999999999995,"15min":0.70799999999999996},"min":{"1min":0.57799999999999996,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.004},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.69199999999999995,"15min":0.72299999999999998},"min":{"1min":0.58199999999999996,"5min":0.432,"15min":0.432},"max":{"1min":0.92300000000000004,"5min":0.97199999999999998,"15min":1.0600000000000001},"last":0.63300000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.69699999999999995,"15min":0.72399999999999998},"min":{"1min":0.56000000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.747,"5min":0.70699999999999996,"15min":0.72899999999999998},"min":{"1min":0.53000000000000003,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":1.052},"last":0.71899999999999997}]},{"osd":2,"last update":"Tue Apr 2 09:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.73099999999999998,"15min":0.747},"min":{"1min":0.56399999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.014},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72699999999999998,"15min":0.74399999999999999},"min":{"1min":0.623,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.90200000000000002,"5min":0.98099999999999998,"15min":1.0680000000000001},"last":0.69499999999999995}]},{"osd":3,"last update":"Tue Apr 2 09:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.76300000000000001,"15min":0.78500000000000003},"min":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.52600000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":1.083},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.78500000000000003,"15min":0.79100000000000004},"min":{"1min":0.56100000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.95599999999999996,"5min":1.008,"15min":1.1020000000000001},"last":0.84299999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60299999999999998,"15min":0.621},"min":{"1min":0.42999999999999999,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.054},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65900000000000003,"15min":0.66600000000000004},"min":{"1min":0.47399999999999998,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.75700000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.67300000000000004,"15min":0.66400000000000003},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69299999999999995,"15min":0.70299999999999996},"min":{"1min":0.53600000000000003,"5min":0.44600000000000001,"15min":0.38700000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.86799999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71999999999999997,"15min":0.72599999999999998},"min":{"1min":0.499,"5min":0.441,"15min":0.441},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.754,"15min":0.75700000000000001},"min":{"1min":0.55900000000000005,"5min":0.48199999999999998,"15min":0.45700000000000002},"max":{"1min":1.0269999999999999,"5min":1.079,"15min":1.079},"last":0.88100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:12:29.563 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:12:29.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.793+0000 7f3505e00700 1 Processor -- start 2024-04-02T09:12:29.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.793+0000 7f3505e00700 1 -- start start 2024-04-02T09:12:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.794+0000 7f3505e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35000d18b0 0x7f35000cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.794+0000 7f3505e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3500161590 0x7f3500161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.794+0000 7f3505e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35000ceda0 con 0x7f3500160f10 2024-04-02T09:12:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.794+0000 7f3505e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35000d1d30 con 0x7f3500161590 2024-04-02T09:12:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.794+0000 7f3505e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3500161f60 con 0x7f35000d18b0 2024-04-02T09:12:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.794+0000 7f34fea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3500160f10 0x7f35000d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52720/0 (socket says 172.21.15.89:52720) 2024-04-02T09:12:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.794+0000 7f34fea00700 1 -- 172.21.15.89:0/2975911212 learned_addr learned my addr 172.21.15.89:0/2975911212 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.795+0000 7f34fe000700 1 --2- 172.21.15.89:0/2975911212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35000d18b0 0x7f35000cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.795+0000 7f34ff400700 1 --2- 172.21.15.89:0/2975911212 >> v2:172.21.15.113:3300/0 conn(0x7f3500161590 0x7f3500161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.795+0000 7f34fd600700 1 -- 172.21.15.89:0/2975911212 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436130058 0 0) 0x7f35000ceda0 con 0x7f3500160f10 2024-04-02T09:12:29.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.795+0000 7f34fd600700 1 -- 172.21.15.89:0/2975911212 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34e40035b0 con 0x7f3500160f10 2024-04-02T09:12:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.795+0000 7f34fd600700 1 -- 172.21.15.89:0/2975911212 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1310594868 0 0) 0x7f34e40035b0 con 0x7f3500160f10 2024-04-02T09:12:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.795+0000 7f34fe000700 1 -- 172.21.15.89:0/2975911212 >> v2:172.21.15.113:3300/0 conn(0x7f3500161590 msgr2=0x7f3500161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.795+0000 7f34fe000700 1 --2- 172.21.15.89:0/2975911212 >> v2:172.21.15.113:3300/0 conn(0x7f3500161590 0x7f3500161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.795+0000 7f34fe000700 1 -- 172.21.15.89:0/2975911212 >> v1:172.21.15.89:6789/0 conn(0x7f3500160f10 legacy=0x7f35000d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.795+0000 7f34fe000700 1 -- 172.21.15.89:0/2975911212 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35001612f0 con 0x7f35000d18b0 2024-04-02T09:12:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.795+0000 7f34ff400700 1 --2- 172.21.15.89:0/2975911212 >> v2:172.21.15.113:3300/0 conn(0x7f3500161590 0x7f3500161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:12:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.795+0000 7f34fe000700 1 --2- 172.21.15.89:0/2975911212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35000d18b0 0x7f35000cf230 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009870 tx=0x7f34ec009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e15cd972a8202be6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.796+0000 7f34fd600700 1 -- 172.21.15.89:0/2975911212 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f0001670 con 0x7f35000d18b0 2024-04-02T09:12:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.796+0000 7f34fd600700 1 -- 172.21.15.89:0/2975911212 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ec004030 con 0x7f35000d18b0 2024-04-02T09:12:29.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.796+0000 7f34fd600700 1 -- 172.21.15.89:0/2975911212 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec0043c0 con 0x7f35000d18b0 2024-04-02T09:12:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.796+0000 7f3505e00700 1 -- 172.21.15.89:0/2975911212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35000d18b0 msgr2=0x7f35000cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.796+0000 7f3505e00700 1 --2- 172.21.15.89:0/2975911212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35000d18b0 0x7f35000cf230 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009870 tx=0x7f34ec009b80 comp rx=0 tx=0).stop 2024-04-02T09:12:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.796+0000 7f3505e00700 1 -- 172.21.15.89:0/2975911212 shutdown_connections 2024-04-02T09:12:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.796+0000 7f3505e00700 1 --2- 172.21.15.89:0/2975911212 >> v2:172.21.15.113:3300/0 conn(0x7f3500161590 0x7f3500161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.796+0000 7f3505e00700 1 --2- 172.21.15.89:0/2975911212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35000d18b0 0x7f35000cf230 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.796+0000 7f3505e00700 1 -- 172.21.15.89:0/2975911212 >> 172.21.15.89:0/2975911212 conn(0x7f350008aff0 msgr2=0x7f35000d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.796+0000 7f3505e00700 1 -- 172.21.15.89:0/2975911212 shutdown_connections 2024-04-02T09:12:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.797+0000 7f3505e00700 1 -- 172.21.15.89:0/2975911212 wait complete. 2024-04-02T09:12:29.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.798+0000 7f3505e00700 1 Processor -- start 2024-04-02T09:12:29.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.798+0000 7f3505e00700 1 -- start start 2024-04-02T09:12:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.799+0000 7f3505e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35000d18b0 0x7f350015fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.799+0000 7f3505e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3500160f10 0x7f35001588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.799+0000 7f3505e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3500161590 0x7f35000d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.799+0000 7f3505e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350015ffb0 con 0x7f3500160f10 2024-04-02T09:12:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.799+0000 7f3505e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3500158df0 con 0x7f35000d18b0 2024-04-02T09:12:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.799+0000 7f3505e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350015e9d0 con 0x7f3500161590 2024-04-02T09:12:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.799+0000 7f34fe000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3500160f10 0x7f35001588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.800+0000 7f34fe000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3500160f10 0x7f35001588b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52414/0 (socket says 172.21.15.89:52414) 2024-04-02T09:12:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.800+0000 7f34fe000700 1 -- 172.21.15.89:0/1836986048 learned_addr learned my addr 172.21.15.89:0/1836986048 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.800+0000 7f34fea00700 1 --2- 172.21.15.89:0/1836986048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35000d18b0 0x7f350015fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:29.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.800+0000 7f34ff400700 1 --2- 172.21.15.89:0/1836986048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3500161590 0x7f35000d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.800+0000 7f34fe000700 1 -- 172.21.15.89:0/1836986048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3500161590 msgr2=0x7f35000d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.800+0000 7f34fe000700 1 --2- 172.21.15.89:0/1836986048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3500161590 0x7f35000d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.800+0000 7f34fe000700 1 -- 172.21.15.89:0/1836986048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35000d18b0 msgr2=0x7f350015fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.800+0000 7f34fe000700 1 --2- 172.21.15.89:0/1836986048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35000d18b0 0x7f350015fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.800+0000 7f34fe000700 1 -- 172.21.15.89:0/1836986048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f8009a70 con 0x7f3500160f10 2024-04-02T09:12:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.800+0000 7f34fe000700 1 --2- 172.21.15.89:0/1836986048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3500160f10 0x7f35001588b0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009840 tx=0x7f34ec0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.801+0000 7f34f7400700 1 -- 172.21.15.89:0/1836986048 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec003cb0 con 0x7f3500160f10 2024-04-02T09:12:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.801+0000 7f34f7400700 1 -- 172.21.15.89:0/1836986048 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ec003e10 con 0x7f3500160f10 2024-04-02T09:12:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.801+0000 7f3505e00700 1 -- 172.21.15.89:0/1836986048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ec0094a0 con 0x7f3500160f10 2024-04-02T09:12:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.801+0000 7f34f7400700 1 -- 172.21.15.89:0/1836986048 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec011870 con 0x7f3500160f10 2024-04-02T09:12:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.801+0000 7f3505e00700 1 -- 172.21.15.89:0/1836986048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350015ef90 con 0x7f3500160f10 2024-04-02T09:12:29.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.802+0000 7f3505e00700 1 -- 172.21.15.89:0/1836986048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3500067720 con 0x7f3500160f10 2024-04-02T09:12:29.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.803+0000 7f34f7400700 1 -- 172.21.15.89:0/1836986048 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f34ec0119d0 con 0x7f3500160f10 2024-04-02T09:12:29.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.803+0000 7f34f7400700 1 --2- 172.21.15.89:0/1836986048 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f34e006adb0 0x7f34e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:29.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.804+0000 7f34f7400700 1 -- 172.21.15.89:0/1836986048 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f34ec08ab70 con 0x7f3500160f10 2024-04-02T09:12:29.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.804+0000 7f34fea00700 1 --2- 172.21.15.89:0/1836986048 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f34e006adb0 0x7f34e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:29.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.804+0000 7f34fea00700 1 --2- 172.21.15.89:0/1836986048 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f34e006adb0 0x7f34e006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f34f0000f60 tx=0x7f34f0001e00 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:29.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:29.808+0000 7f34f7400700 1 -- 172.21.15.89:0/1836986048 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f34ec055c30 con 0x7f3500160f10 2024-04-02T09:12:30.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.009+0000 7f3505e00700 1 -- 172.21.15.89:0/1836986048 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f35000ce750 con 0x7f34e006adb0 2024-04-02T09:12:30.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.017+0000 7f34f7400700 1 -- 172.21.15.89:0/1836986048 <== mgr.10923 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289386 (secure 0 0 0) 0x7f34f000f610 con 0x7f34e006adb0 2024-04-02T09:12:30.019 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:30.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.021+0000 7f3505e00700 1 -- 172.21.15.89:0/1836986048 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f34e006adb0 msgr2=0x7f34e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:30.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.021+0000 7f3505e00700 1 --2- 172.21.15.89:0/1836986048 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f34e006adb0 0x7f34e006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f34f0000f60 tx=0x7f34f0001e00 comp rx=0 tx=0).stop 2024-04-02T09:12:30.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.021+0000 7f3505e00700 1 -- 172.21.15.89:0/1836986048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3500160f10 msgr2=0x7f35001588b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:30.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.021+0000 7f3505e00700 1 --2- 172.21.15.89:0/1836986048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3500160f10 0x7f35001588b0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009840 tx=0x7f34ec0047c0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.022+0000 7f3505e00700 1 -- 172.21.15.89:0/1836986048 shutdown_connections 2024-04-02T09:12:30.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.022+0000 7f3505e00700 1 --2- 172.21.15.89:0/1836986048 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f34e006adb0 0x7f34e006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.022+0000 7f3505e00700 1 --2- 172.21.15.89:0/1836986048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35000d18b0 0x7f350015fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.022+0000 7f3505e00700 1 --2- 172.21.15.89:0/1836986048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3500161590 0x7f35000d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.022+0000 7f3505e00700 1 --2- 172.21.15.89:0/1836986048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3500160f10 0x7f35001588b0 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.022+0000 7f3505e00700 1 -- 172.21.15.89:0/1836986048 >> 172.21.15.89:0/1836986048 conn(0x7f350008aff0 msgr2=0x7f35000b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:30.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.022+0000 7f3505e00700 1 -- 172.21.15.89:0/1836986048 shutdown_connections 2024-04-02T09:12:30.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.022+0000 7f3505e00700 1 -- 172.21.15.89:0/1836986048 wait complete. 2024-04-02T09:12:30.023 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:12:30.038 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-02T09:12:28.799428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":623,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":402416,"kb_used_data":23012,"kb_used_omap":12,"kb_used_meta":379251,"kb_avail":754572304,"statfs":{"total":773094113280,"available":772682039296,"internally_reserved":0,"allocated":23564288,"data_stored":12501762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":388353592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.129828"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424498+0000","last_change":"2024-04-02T09:12:25.187698+0000","last_active":"2024-04-02T09:12:26.424498+0000","last_peered":"2024-04-02T09:12:26.424498+0000","last_clean":"2024-04-02T09:12:26.424498+0000","last_became_active":"2024-04-02T09:12:25.187420+0000","last_became_peered":"2024-04-02T09:12:25.187420+0000","last_unstale":"2024-04-02T09:12:26.424498+0000","last_undegraded":"2024-04-02T09:12:26.424498+0000","last_fullsized":"2024-04-02T09:12:26.424498+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:45:21.225286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422997+0000","last_change":"2024-04-02T09:12:25.208060+0000","last_active":"2024-04-02T09:12:26.422997+0000","last_peered":"2024-04-02T09:12:26.422997+0000","last_clean":"2024-04-02T09:12:26.422997+0000","last_became_active":"2024-04-02T09:12:25.206932+0000","last_became_peered":"2024-04-02T09:12:25.206932+0000","last_unstale":"2024-04-02T09:12:26.422997+0000","last_undegraded":"2024-04-02T09:12:26.422997+0000","last_fullsized":"2024-04-02T09:12:26.422997+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:11.810703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423793+0000","last_change":"2024-04-02T09:12:25.194789+0000","last_active":"2024-04-02T09:12:26.423793+0000","last_peered":"2024-04-02T09:12:26.423793+0000","last_clean":"2024-04-02T09:12:26.423793+0000","last_became_active":"2024-04-02T09:12:25.194433+0000","last_became_peered":"2024-04-02T09:12:25.194433+0000","last_unstale":"2024-04-02T09:12:26.423793+0000","last_undegraded":"2024-04-02T09:12:26.423793+0000","last_fullsized":"2024-04-02T09:12:26.423793+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:18.798282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185951+0000","last_change":"2024-04-02T09:12:25.190996+0000","last_active":"2024-04-02T09:12:26.185951+0000","last_peered":"2024-04-02T09:12:26.185951+0000","last_clean":"2024-04-02T09:12:26.185951+0000","last_became_active":"2024-04-02T09:12:25.190655+0000","last_became_peered":"2024-04-02T09:12:25.190655+0000","last_unstale":"2024-04-02T09:12:26.185951+0000","last_undegraded":"2024-04-02T09:12:26.185951+0000","last_fullsized":"2024-04-02T09:12:26.185951+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:48:22.188323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423434+0000","last_change":"2024-04-02T09:12:25.191955+0000","last_active":"2024-04-02T09:12:26.423434+0000","last_peered":"2024-04-02T09:12:26.423434+0000","last_clean":"2024-04-02T09:12:26.423434+0000","last_became_active":"2024-04-02T09:12:25.191699+0000","last_became_peered":"2024-04-02T09:12:25.191699+0000","last_unstale":"2024-04-02T09:12:26.423434+0000","last_undegraded":"2024-04-02T09:12:26.423434+0000","last_fullsized":"2024-04-02T09:12:26.423434+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:22:42.917826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423946+0000","last_change":"2024-04-02T09:12:25.200889+0000","last_active":"2024-04-02T09:12:26.423946+0000","last_peered":"2024-04-02T09:12:26.423946+0000","last_clean":"2024-04-02T09:12:26.423946+0000","last_became_active":"2024-04-02T09:12:25.200635+0000","last_became_peered":"2024-04-02T09:12:25.200635+0000","last_unstale":"2024-04-02T09:12:26.423946+0000","last_undegraded":"2024-04-02T09:12:26.423946+0000","last_fullsized":"2024-04-02T09:12:26.423946+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:43:29.229527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422840+0000","last_change":"2024-04-02T09:12:25.201485+0000","last_active":"2024-04-02T09:12:26.422840+0000","last_peered":"2024-04-02T09:12:26.422840+0000","last_clean":"2024-04-02T09:12:26.422840+0000","last_became_active":"2024-04-02T09:12:25.200542+0000","last_became_peered":"2024-04-02T09:12:25.200542+0000","last_unstale":"2024-04-02T09:12:26.422840+0000","last_undegraded":"2024-04-02T09:12:26.422840+0000","last_fullsized":"2024-04-02T09:12:26.422840+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:19:30.245770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185880+0000","last_change":"2024-04-02T09:12:25.191970+0000","last_active":"2024-04-02T09:12:26.185880+0000","last_peered":"2024-04-02T09:12:26.185880+0000","last_clean":"2024-04-02T09:12:26.185880+0000","last_became_active":"2024-04-02T09:12:25.191572+0000","last_became_peered":"2024-04-02T09:12:25.191572+0000","last_unstale":"2024-04-02T09:12:26.185880+0000","last_undegraded":"2024-04-02T09:12:26.185880+0000","last_fullsized":"2024-04-02T09:12:26.185880+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:03.387546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422571+0000","last_change":"2024-04-02T09:12:25.201593+0000","last_active":"2024-04-02T09:12:26.422571+0000","last_peered":"2024-04-02T09:12:26.422571+0000","last_clean":"2024-04-02T09:12:26.422571+0000","last_became_active":"2024-04-02T09:12:25.200851+0000","last_became_peered":"2024-04-02T09:12:25.200851+0000","last_unstale":"2024-04-02T09:12:26.422571+0000","last_undegraded":"2024-04-02T09:12:26.422571+0000","last_fullsized":"2024-04-02T09:12:26.422571+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:26:33.117159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423365+0000","last_change":"2024-04-02T09:12:25.202349+0000","last_active":"2024-04-02T09:12:26.423365+0000","last_peered":"2024-04-02T09:12:26.423365+0000","last_clean":"2024-04-02T09:12:26.423365+0000","last_became_active":"2024-04-02T09:12:25.200134+0000","last_became_peered":"2024-04-02T09:12:25.200134+0000","last_unstale":"2024-04-02T09:12:26.423365+0000","last_undegraded":"2024-04-02T09:12:26.423365+0000","last_fullsized":"2024-04-02T09:12:26.423365+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:45:54.007817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424281+0000","last_change":"2024-04-02T09:12:25.203968+0000","last_active":"2024-04-02T09:12:26.424281+0000","last_peered":"2024-04-02T09:12:26.424281+0000","last_clean":"2024-04-02T09:12:26.424281+0000","last_became_active":"2024-04-02T09:12:25.203714+0000","last_became_peered":"2024-04-02T09:12:25.203714+0000","last_unstale":"2024-04-02T09:12:26.424281+0000","last_undegraded":"2024-04-02T09:12:26.424281+0000","last_fullsized":"2024-04-02T09:12:26.424281+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:44:28.685030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186694+0000","last_change":"2024-04-02T09:12:25.186623+0000","last_active":"2024-04-02T09:12:26.186694+0000","last_peered":"2024-04-02T09:12:26.186694+0000","last_clean":"2024-04-02T09:12:26.186694+0000","last_became_active":"2024-04-02T09:12:25.186297+0000","last_became_peered":"2024-04-02T09:12:25.186297+0000","last_unstale":"2024-04-02T09:12:26.186694+0000","last_undegraded":"2024-04-02T09:12:26.186694+0000","last_fullsized":"2024-04-02T09:12:26.186694+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:36:48.733839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.184901+0000","last_change":"2024-04-02T09:12:25.205465+0000","last_active":"2024-04-02T09:12:26.184901+0000","last_peered":"2024-04-02T09:12:26.184901+0000","last_clean":"2024-04-02T09:12:26.184901+0000","last_became_active":"2024-04-02T09:12:25.204248+0000","last_became_peered":"2024-04-02T09:12:25.204248+0000","last_unstale":"2024-04-02T09:12:26.184901+0000","last_undegraded":"2024-04-02T09:12:26.184901+0000","last_fullsized":"2024-04-02T09:12:26.184901+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:32:03.502011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422702+0000","last_change":"2024-04-02T09:12:25.198312+0000","last_active":"2024-04-02T09:12:26.422702+0000","last_peered":"2024-04-02T09:12:26.422702+0000","last_clean":"2024-04-02T09:12:26.422702+0000","last_became_active":"2024-04-02T09:12:25.198040+0000","last_became_peered":"2024-04-02T09:12:25.198040+0000","last_unstale":"2024-04-02T09:12:26.422702+0000","last_undegraded":"2024-04-02T09:12:26.422702+0000","last_fullsized":"2024-04-02T09:12:26.422702+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:02.893062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423721+0000","last_change":"2024-04-02T09:12:25.186757+0000","last_active":"2024-04-02T09:12:26.423721+0000","last_peered":"2024-04-02T09:12:26.423721+0000","last_clean":"2024-04-02T09:12:26.423721+0000","last_became_active":"2024-04-02T09:12:25.186436+0000","last_became_peered":"2024-04-02T09:12:25.186436+0000","last_unstale":"2024-04-02T09:12:26.423721+0000","last_undegraded":"2024-04-02T09:12:26.423721+0000","last_fullsized":"2024-04-02T09:12:26.423721+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:21.964569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422907+0000","last_change":"2024-04-02T09:12:25.188711+0000","last_active":"2024-04-02T09:12:26.422907+0000","last_peered":"2024-04-02T09:12:26.422907+0000","last_clean":"2024-04-02T09:12:26.422907+0000","last_became_active":"2024-04-02T09:12:25.188399+0000","last_became_peered":"2024-04-02T09:12:25.188399+0000","last_unstale":"2024-04-02T09:12:26.422907+0000","last_undegraded":"2024-04-02T09:12:26.422907+0000","last_fullsized":"2024-04-02T09:12:26.422907+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:20:21.844925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424498+0000","last_change":"2024-04-02T09:12:25.196993+0000","last_active":"2024-04-02T09:12:26.424498+0000","last_peered":"2024-04-02T09:12:26.424498+0000","last_clean":"2024-04-02T09:12:26.424498+0000","last_became_active":"2024-04-02T09:12:25.196316+0000","last_became_peered":"2024-04-02T09:12:25.196316+0000","last_unstale":"2024-04-02T09:12:26.424498+0000","last_undegraded":"2024-04-02T09:12:26.424498+0000","last_fullsized":"2024-04-02T09:12:26.424498+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:36:48.733839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423510+0000","last_change":"2024-04-02T09:12:25.207159+0000","last_active":"2024-04-02T09:12:26.423510+0000","last_peered":"2024-04-02T09:12:26.423510+0000","last_clean":"2024-04-02T09:12:26.423510+0000","last_became_active":"2024-04-02T09:12:25.206804+0000","last_became_peered":"2024-04-02T09:12:25.206804+0000","last_unstale":"2024-04-02T09:12:26.423510+0000","last_undegraded":"2024-04-02T09:12:26.423510+0000","last_fullsized":"2024-04-02T09:12:26.423510+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:19.646703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423916+0000","last_change":"2024-04-02T09:12:25.196715+0000","last_active":"2024-04-02T09:12:26.423916+0000","last_peered":"2024-04-02T09:12:26.423916+0000","last_clean":"2024-04-02T09:12:26.423916+0000","last_became_active":"2024-04-02T09:12:25.196433+0000","last_became_peered":"2024-04-02T09:12:25.196433+0000","last_unstale":"2024-04-02T09:12:26.423916+0000","last_undegraded":"2024-04-02T09:12:26.423916+0000","last_fullsized":"2024-04-02T09:12:26.423916+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:30.835437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423285+0000","last_change":"2024-04-02T09:12:25.196567+0000","last_active":"2024-04-02T09:12:26.423285+0000","last_peered":"2024-04-02T09:12:26.423285+0000","last_clean":"2024-04-02T09:12:26.423285+0000","last_became_active":"2024-04-02T09:12:25.196264+0000","last_became_peered":"2024-04-02T09:12:25.196264+0000","last_unstale":"2024-04-02T09:12:26.423285+0000","last_undegraded":"2024-04-02T09:12:26.423285+0000","last_fullsized":"2024-04-02T09:12:26.423285+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:03.387546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186592+0000","last_change":"2024-04-02T09:12:25.201010+0000","last_active":"2024-04-02T09:12:26.186592+0000","last_peered":"2024-04-02T09:12:26.186592+0000","last_clean":"2024-04-02T09:12:26.186592+0000","last_became_active":"2024-04-02T09:12:25.198423+0000","last_became_peered":"2024-04-02T09:12:25.198423+0000","last_unstale":"2024-04-02T09:12:26.186592+0000","last_undegraded":"2024-04-02T09:12:26.186592+0000","last_fullsized":"2024-04-02T09:12:26.186592+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:44:28.685030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422805+0000","last_change":"2024-04-02T09:12:25.197238+0000","last_active":"2024-04-02T09:12:26.422805+0000","last_peered":"2024-04-02T09:12:26.422805+0000","last_clean":"2024-04-02T09:12:26.422805+0000","last_became_active":"2024-04-02T09:12:25.196921+0000","last_became_peered":"2024-04-02T09:12:25.196921+0000","last_unstale":"2024-04-02T09:12:26.422805+0000","last_undegraded":"2024-04-02T09:12:26.422805+0000","last_fullsized":"2024-04-02T09:12:26.422805+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:40:34.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185295+0000","last_change":"2024-04-02T09:12:25.205832+0000","last_active":"2024-04-02T09:12:26.185295+0000","last_peered":"2024-04-02T09:12:26.185295+0000","last_clean":"2024-04-02T09:12:26.185295+0000","last_became_active":"2024-04-02T09:12:25.205206+0000","last_became_peered":"2024-04-02T09:12:25.205206+0000","last_unstale":"2024-04-02T09:12:26.185295+0000","last_undegraded":"2024-04-02T09:12:26.185295+0000","last_fullsized":"2024-04-02T09:12:26.185295+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:17.511411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422291+0000","last_change":"2024-04-02T09:12:25.185102+0000","last_active":"2024-04-02T09:12:26.422291+0000","last_peered":"2024-04-02T09:12:26.422291+0000","last_clean":"2024-04-02T09:12:26.422291+0000","last_became_active":"2024-04-02T09:12:25.184829+0000","last_became_peered":"2024-04-02T09:12:25.184829+0000","last_unstale":"2024-04-02T09:12:26.422291+0000","last_undegraded":"2024-04-02T09:12:26.422291+0000","last_fullsized":"2024-04-02T09:12:26.422291+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:28.343401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186057+0000","last_change":"2024-04-02T09:12:25.195404+0000","last_active":"2024-04-02T09:12:26.186057+0000","last_peered":"2024-04-02T09:12:26.186057+0000","last_clean":"2024-04-02T09:12:26.186057+0000","last_became_active":"2024-04-02T09:12:25.192051+0000","last_became_peered":"2024-04-02T09:12:25.192051+0000","last_unstale":"2024-04-02T09:12:26.186057+0000","last_undegraded":"2024-04-02T09:12:26.186057+0000","last_fullsized":"2024-04-02T09:12:26.186057+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:02.893062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186453+0000","last_change":"2024-04-02T09:12:25.200885+0000","last_active":"2024-04-02T09:12:26.186453+0000","last_peered":"2024-04-02T09:12:26.186453+0000","last_clean":"2024-04-02T09:12:26.186453+0000","last_became_active":"2024-04-02T09:12:25.200630+0000","last_became_peered":"2024-04-02T09:12:25.200630+0000","last_unstale":"2024-04-02T09:12:26.186453+0000","last_undegraded":"2024-04-02T09:12:26.186453+0000","last_fullsized":"2024-04-02T09:12:26.186453+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:18.798282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186550+0000","last_change":"2024-04-02T09:12:25.196804+0000","last_active":"2024-04-02T09:12:26.186550+0000","last_peered":"2024-04-02T09:12:26.186550+0000","last_clean":"2024-04-02T09:12:26.186550+0000","last_became_active":"2024-04-02T09:12:25.194411+0000","last_became_peered":"2024-04-02T09:12:25.194411+0000","last_unstale":"2024-04-02T09:12:26.186550+0000","last_undegraded":"2024-04-02T09:12:26.186550+0000","last_fullsized":"2024-04-02T09:12:26.186550+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:45:21.225286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186405+0000","last_change":"2024-04-02T09:12:25.192359+0000","last_active":"2024-04-02T09:12:26.186405+0000","last_peered":"2024-04-02T09:12:26.186405+0000","last_clean":"2024-04-02T09:12:26.186405+0000","last_became_active":"2024-04-02T09:12:25.192092+0000","last_became_peered":"2024-04-02T09:12:25.192092+0000","last_unstale":"2024-04-02T09:12:26.186405+0000","last_undegraded":"2024-04-02T09:12:26.186405+0000","last_fullsized":"2024-04-02T09:12:26.186405+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:01.618091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422831+0000","last_change":"2024-04-02T09:12:25.205454+0000","last_active":"2024-04-02T09:12:26.422831+0000","last_peered":"2024-04-02T09:12:26.422831+0000","last_clean":"2024-04-02T09:12:26.422831+0000","last_became_active":"2024-04-02T09:12:25.204157+0000","last_became_peered":"2024-04-02T09:12:25.204157+0000","last_unstale":"2024-04-02T09:12:26.422831+0000","last_undegraded":"2024-04-02T09:12:26.422831+0000","last_fullsized":"2024-04-02T09:12:26.422831+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:46:50.298119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.184933+0000","last_change":"2024-04-02T09:12:25.194990+0000","last_active":"2024-04-02T09:12:26.184933+0000","last_peered":"2024-04-02T09:12:26.184933+0000","last_clean":"2024-04-02T09:12:26.184933+0000","last_became_active":"2024-04-02T09:12:25.191623+0000","last_became_peered":"2024-04-02T09:12:25.191623+0000","last_unstale":"2024-04-02T09:12:26.184933+0000","last_undegraded":"2024-04-02T09:12:26.184933+0000","last_fullsized":"2024-04-02T09:12:26.184933+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:42:10.151562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422733+0000","last_change":"2024-04-02T09:12:25.203525+0000","last_active":"2024-04-02T09:12:26.422733+0000","last_peered":"2024-04-02T09:12:26.422733+0000","last_clean":"2024-04-02T09:12:26.422733+0000","last_became_active":"2024-04-02T09:12:25.202409+0000","last_became_peered":"2024-04-02T09:12:25.202409+0000","last_unstale":"2024-04-02T09:12:26.422733+0000","last_undegraded":"2024-04-02T09:12:26.422733+0000","last_fullsized":"2024-04-02T09:12:26.422733+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:38:54.918706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422701+0000","last_change":"2024-04-02T09:12:25.193316+0000","last_active":"2024-04-02T09:12:26.422701+0000","last_peered":"2024-04-02T09:12:26.422701+0000","last_clean":"2024-04-02T09:12:26.422701+0000","last_became_active":"2024-04-02T09:12:25.192672+0000","last_became_peered":"2024-04-02T09:12:25.192672+0000","last_unstale":"2024-04-02T09:12:26.422701+0000","last_undegraded":"2024-04-02T09:12:26.422701+0000","last_fullsized":"2024-04-02T09:12:26.422701+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:28.343401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186980+0000","last_change":"2024-04-02T09:12:25.195780+0000","last_active":"2024-04-02T09:12:26.186980+0000","last_peered":"2024-04-02T09:12:26.186980+0000","last_clean":"2024-04-02T09:12:26.186980+0000","last_became_active":"2024-04-02T09:12:25.194939+0000","last_became_peered":"2024-04-02T09:12:25.194939+0000","last_unstale":"2024-04-02T09:12:26.186980+0000","last_undegraded":"2024-04-02T09:12:26.186980+0000","last_fullsized":"2024-04-02T09:12:26.186980+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:20:21.844925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"222'4","reported_seq":15,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.187162+0000","last_change":"2024-04-02T09:12:24.695565+0000","last_active":"2024-04-02T09:12:26.187162+0000","last_peered":"2024-04-02T09:12:26.187162+0000","last_clean":"2024-04-02T09:12:26.187162+0000","last_became_active":"2024-04-02T09:12:24.695304+0000","last_became_peered":"2024-04-02T09:12:24.695304+0000","last_unstale":"2024-04-02T09:12:26.187162+0000","last_undegraded":"2024-04-02T09:12:26.187162+0000","last_fullsized":"2024-04-02T09:12:26.187162+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:56:16.018943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186201+0000","last_change":"2024-04-02T09:12:25.190209+0000","last_active":"2024-04-02T09:12:26.186201+0000","last_peered":"2024-04-02T09:12:26.186201+0000","last_clean":"2024-04-02T09:12:26.186201+0000","last_became_active":"2024-04-02T09:12:25.189908+0000","last_became_peered":"2024-04-02T09:12:25.189908+0000","last_unstale":"2024-04-02T09:12:26.186201+0000","last_undegraded":"2024-04-02T09:12:26.186201+0000","last_fullsized":"2024-04-02T09:12:26.186201+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:12.375324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1e","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423048+0000","last_change":"2024-04-02T09:12:24.684617+0000","last_active":"2024-04-02T09:12:26.423048+0000","last_peered":"2024-04-02T09:12:26.423048+0000","last_clean":"2024-04-02T09:12:26.423048+0000","last_became_active":"2024-04-02T09:12:24.684367+0000","last_became_peered":"2024-04-02T09:12:24.684367+0000","last_unstale":"2024-04-02T09:12:26.423048+0000","last_undegraded":"2024-04-02T09:12:26.423048+0000","last_fullsized":"2024-04-02T09:12:26.423048+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:48:45.999478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423420+0000","last_change":"2024-04-02T09:12:25.204722+0000","last_active":"2024-04-02T09:12:26.423420+0000","last_peered":"2024-04-02T09:12:26.423420+0000","last_clean":"2024-04-02T09:12:26.423420+0000","last_became_active":"2024-04-02T09:12:25.204031+0000","last_became_peered":"2024-04-02T09:12:25.204031+0000","last_unstale":"2024-04-02T09:12:26.423420+0000","last_undegraded":"2024-04-02T09:12:26.423420+0000","last_fullsized":"2024-04-02T09:12:26.423420+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:00:19.545624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186561+0000","last_change":"2024-04-02T09:12:24.691669+0000","last_active":"2024-04-02T09:12:26.186561+0000","last_peered":"2024-04-02T09:12:26.186561+0000","last_clean":"2024-04-02T09:12:26.186561+0000","last_became_active":"2024-04-02T09:12:24.689803+0000","last_became_peered":"2024-04-02T09:12:24.689803+0000","last_unstale":"2024-04-02T09:12:26.186561+0000","last_undegraded":"2024-04-02T09:12:26.186561+0000","last_fullsized":"2024-04-02T09:12:26.186561+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:25:15.021325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423945+0000","last_change":"2024-04-02T09:12:25.194989+0000","last_active":"2024-04-02T09:12:26.423945+0000","last_peered":"2024-04-02T09:12:26.423945+0000","last_clean":"2024-04-02T09:12:26.423945+0000","last_became_active":"2024-04-02T09:12:25.194466+0000","last_became_peered":"2024-04-02T09:12:25.194466+0000","last_unstale":"2024-04-02T09:12:26.423945+0000","last_undegraded":"2024-04-02T09:12:26.423945+0000","last_fullsized":"2024-04-02T09:12:26.423945+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:01.618091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1c","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186076+0000","last_change":"2024-04-02T09:12:24.692574+0000","last_active":"2024-04-02T09:12:26.186076+0000","last_peered":"2024-04-02T09:12:26.186076+0000","last_clean":"2024-04-02T09:12:26.186076+0000","last_became_active":"2024-04-02T09:12:24.692283+0000","last_became_peered":"2024-04-02T09:12:24.692283+0000","last_unstale":"2024-04-02T09:12:26.186076+0000","last_undegraded":"2024-04-02T09:12:26.186076+0000","last_fullsized":"2024-04-02T09:12:26.186076+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:56:59.029298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185775+0000","last_change":"2024-04-02T09:12:25.192982+0000","last_active":"2024-04-02T09:12:26.185775+0000","last_peered":"2024-04-02T09:12:26.185775+0000","last_clean":"2024-04-02T09:12:26.185775+0000","last_became_active":"2024-04-02T09:12:25.192737+0000","last_became_peered":"2024-04-02T09:12:25.192737+0000","last_unstale":"2024-04-02T09:12:26.185775+0000","last_undegraded":"2024-04-02T09:12:26.185775+0000","last_fullsized":"2024-04-02T09:12:26.185775+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:28.343401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424086+0000","last_change":"2024-04-02T09:12:24.690596+0000","last_active":"2024-04-02T09:12:26.424086+0000","last_peered":"2024-04-02T09:12:26.424086+0000","last_clean":"2024-04-02T09:12:26.424086+0000","last_became_active":"2024-04-02T09:12:24.690029+0000","last_became_peered":"2024-04-02T09:12:24.690029+0000","last_unstale":"2024-04-02T09:12:26.424086+0000","last_undegraded":"2024-04-02T09:12:26.424086+0000","last_fullsized":"2024-04-02T09:12:26.424086+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:00.608890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423300+0000","last_change":"2024-04-02T09:12:25.191145+0000","last_active":"2024-04-02T09:12:26.423300+0000","last_peered":"2024-04-02T09:12:26.423300+0000","last_clean":"2024-04-02T09:12:26.423300+0000","last_became_active":"2024-04-02T09:12:25.190868+0000","last_became_peered":"2024-04-02T09:12:25.190868+0000","last_unstale":"2024-04-02T09:12:26.423300+0000","last_undegraded":"2024-04-02T09:12:26.423300+0000","last_fullsized":"2024-04-02T09:12:26.423300+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:41.683998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186094+0000","last_change":"2024-04-02T09:12:24.698972+0000","last_active":"2024-04-02T09:12:26.186094+0000","last_peered":"2024-04-02T09:12:26.186094+0000","last_clean":"2024-04-02T09:12:26.186094+0000","last_became_active":"2024-04-02T09:12:24.698664+0000","last_became_peered":"2024-04-02T09:12:24.698664+0000","last_unstale":"2024-04-02T09:12:26.186094+0000","last_undegraded":"2024-04-02T09:12:26.186094+0000","last_fullsized":"2024-04-02T09:12:26.186094+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:11:06.573162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422499+0000","last_change":"2024-04-02T09:12:25.198098+0000","last_active":"2024-04-02T09:12:26.422499+0000","last_peered":"2024-04-02T09:12:26.422499+0000","last_clean":"2024-04-02T09:12:26.422499+0000","last_became_active":"2024-04-02T09:12:25.197848+0000","last_became_peered":"2024-04-02T09:12:25.197848+0000","last_unstale":"2024-04-02T09:12:26.422499+0000","last_undegraded":"2024-04-02T09:12:26.422499+0000","last_fullsized":"2024-04-02T09:12:26.422499+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:02.893062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.184920+0000","last_change":"2024-04-02T09:12:24.698699+0000","last_active":"2024-04-02T09:12:26.184920+0000","last_peered":"2024-04-02T09:12:26.184920+0000","last_clean":"2024-04-02T09:12:26.184920+0000","last_became_active":"2024-04-02T09:12:24.698347+0000","last_became_peered":"2024-04-02T09:12:24.698347+0000","last_unstale":"2024-04-02T09:12:26.184920+0000","last_undegraded":"2024-04-02T09:12:26.184920+0000","last_fullsized":"2024-04-02T09:12:26.184920+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:44:22.620155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186150+0000","last_change":"2024-04-02T09:12:25.192446+0000","last_active":"2024-04-02T09:12:26.186150+0000","last_peered":"2024-04-02T09:12:26.186150+0000","last_clean":"2024-04-02T09:12:26.186150+0000","last_became_active":"2024-04-02T09:12:25.192159+0000","last_became_peered":"2024-04-02T09:12:25.192159+0000","last_unstale":"2024-04-02T09:12:26.186150+0000","last_undegraded":"2024-04-02T09:12:26.186150+0000","last_fullsized":"2024-04-02T09:12:26.186150+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:38.903726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423659+0000","last_change":"2024-04-02T09:12:24.697017+0000","last_active":"2024-04-02T09:12:26.423659+0000","last_peered":"2024-04-02T09:12:26.423659+0000","last_clean":"2024-04-02T09:12:26.423659+0000","last_became_active":"2024-04-02T09:12:24.696537+0000","last_became_peered":"2024-04-02T09:12:24.696537+0000","last_unstale":"2024-04-02T09:12:26.423659+0000","last_undegraded":"2024-04-02T09:12:26.423659+0000","last_fullsized":"2024-04-02T09:12:26.423659+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:16.861790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423163+0000","last_change":"2024-04-02T09:12:25.198058+0000","last_active":"2024-04-02T09:12:26.423163+0000","last_peered":"2024-04-02T09:12:26.423163+0000","last_clean":"2024-04-02T09:12:26.423163+0000","last_became_active":"2024-04-02T09:12:25.197791+0000","last_became_peered":"2024-04-02T09:12:25.197791+0000","last_unstale":"2024-04-02T09:12:26.423163+0000","last_undegraded":"2024-04-02T09:12:26.423163+0000","last_fullsized":"2024-04-02T09:12:26.423163+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:38.903726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186231+0000","last_change":"2024-04-02T09:12:24.690012+0000","last_active":"2024-04-02T09:12:26.186231+0000","last_peered":"2024-04-02T09:12:26.186231+0000","last_clean":"2024-04-02T09:12:26.186231+0000","last_became_active":"2024-04-02T09:12:24.689551+0000","last_became_peered":"2024-04-02T09:12:24.689551+0000","last_unstale":"2024-04-02T09:12:26.186231+0000","last_undegraded":"2024-04-02T09:12:26.186231+0000","last_fullsized":"2024-04-02T09:12:26.186231+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:15.316332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422487+0000","last_change":"2024-04-02T09:12:25.202617+0000","last_active":"2024-04-02T09:12:26.422487+0000","last_peered":"2024-04-02T09:12:26.422487+0000","last_clean":"2024-04-02T09:12:26.422487+0000","last_became_active":"2024-04-02T09:12:25.201786+0000","last_became_peered":"2024-04-02T09:12:25.201786+0000","last_unstale":"2024-04-02T09:12:26.422487+0000","last_undegraded":"2024-04-02T09:12:26.422487+0000","last_fullsized":"2024-04-02T09:12:26.422487+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:32:03.502011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"222'3","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186588+0000","last_change":"2024-04-02T09:12:24.694400+0000","last_active":"2024-04-02T09:12:26.186588+0000","last_peered":"2024-04-02T09:12:26.186588+0000","last_clean":"2024-04-02T09:12:26.186588+0000","last_became_active":"2024-04-02T09:12:24.694118+0000","last_became_peered":"2024-04-02T09:12:24.694118+0000","last_unstale":"2024-04-02T09:12:26.186588+0000","last_undegraded":"2024-04-02T09:12:26.186588+0000","last_fullsized":"2024-04-02T09:12:26.186588+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:00.841741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186527+0000","last_change":"2024-04-02T09:12:25.199901+0000","last_active":"2024-04-02T09:12:26.186527+0000","last_peered":"2024-04-02T09:12:26.186527+0000","last_clean":"2024-04-02T09:12:26.186527+0000","last_became_active":"2024-04-02T09:12:25.198989+0000","last_became_peered":"2024-04-02T09:12:25.198989+0000","last_unstale":"2024-04-02T09:12:26.186527+0000","last_undegraded":"2024-04-02T09:12:26.186527+0000","last_fullsized":"2024-04-02T09:12:26.186527+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:30.835437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"222'6","reported_seq":17,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422587+0000","last_change":"2024-04-02T09:12:24.689123+0000","last_active":"2024-04-02T09:12:26.422587+0000","last_peered":"2024-04-02T09:12:26.422587+0000","last_clean":"2024-04-02T09:12:26.422587+0000","last_became_active":"2024-04-02T09:12:24.688848+0000","last_became_peered":"2024-04-02T09:12:24.688848+0000","last_unstale":"2024-04-02T09:12:26.422587+0000","last_undegraded":"2024-04-02T09:12:26.422587+0000","last_fullsized":"2024-04-02T09:12:26.422587+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:24.593956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423130+0000","last_change":"2024-04-02T09:12:25.192240+0000","last_active":"2024-04-02T09:12:26.423130+0000","last_peered":"2024-04-02T09:12:26.423130+0000","last_clean":"2024-04-02T09:12:26.423130+0000","last_became_active":"2024-04-02T09:12:25.191954+0000","last_became_peered":"2024-04-02T09:12:25.191954+0000","last_unstale":"2024-04-02T09:12:26.423130+0000","last_undegraded":"2024-04-02T09:12:26.423130+0000","last_fullsized":"2024-04-02T09:12:26.423130+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:26:33.117159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.13","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422301+0000","last_change":"2024-04-02T09:12:24.694118+0000","last_active":"2024-04-02T09:12:26.422301+0000","last_peered":"2024-04-02T09:12:26.422301+0000","last_clean":"2024-04-02T09:12:26.422301+0000","last_became_active":"2024-04-02T09:12:24.693829+0000","last_became_peered":"2024-04-02T09:12:24.693829+0000","last_unstale":"2024-04-02T09:12:26.422301+0000","last_undegraded":"2024-04-02T09:12:26.422301+0000","last_fullsized":"2024-04-02T09:12:26.422301+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:58:51.743784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423515+0000","last_change":"2024-04-02T09:12:25.205479+0000","last_active":"2024-04-02T09:12:26.423515+0000","last_peered":"2024-04-02T09:12:26.423515+0000","last_clean":"2024-04-02T09:12:26.423515+0000","last_became_active":"2024-04-02T09:12:25.205210+0000","last_became_peered":"2024-04-02T09:12:25.205210+0000","last_unstale":"2024-04-02T09:12:26.423515+0000","last_undegraded":"2024-04-02T09:12:26.423515+0000","last_fullsized":"2024-04-02T09:12:26.423515+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:03.985819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422807+0000","last_change":"2024-04-02T09:12:24.695778+0000","last_active":"2024-04-02T09:12:26.422807+0000","last_peered":"2024-04-02T09:12:26.422807+0000","last_clean":"2024-04-02T09:12:26.422807+0000","last_became_active":"2024-04-02T09:12:24.695349+0000","last_became_peered":"2024-04-02T09:12:24.695349+0000","last_unstale":"2024-04-02T09:12:26.422807+0000","last_undegraded":"2024-04-02T09:12:26.422807+0000","last_fullsized":"2024-04-02T09:12:26.422807+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:33:51.021792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422685+0000","last_change":"2024-04-02T09:12:25.203805+0000","last_active":"2024-04-02T09:12:26.422685+0000","last_peered":"2024-04-02T09:12:26.422685+0000","last_clean":"2024-04-02T09:12:26.422685+0000","last_became_active":"2024-04-02T09:12:25.203516+0000","last_became_peered":"2024-04-02T09:12:25.203516+0000","last_unstale":"2024-04-02T09:12:26.422685+0000","last_undegraded":"2024-04-02T09:12:26.422685+0000","last_fullsized":"2024-04-02T09:12:26.422685+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:00:15.805719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186210+0000","last_change":"2024-04-02T09:12:24.691834+0000","last_active":"2024-04-02T09:12:26.186210+0000","last_peered":"2024-04-02T09:12:26.186210+0000","last_clean":"2024-04-02T09:12:26.186210+0000","last_became_active":"2024-04-02T09:12:24.689829+0000","last_became_peered":"2024-04-02T09:12:24.689829+0000","last_unstale":"2024-04-02T09:12:26.186210+0000","last_undegraded":"2024-04-02T09:12:26.186210+0000","last_fullsized":"2024-04-02T09:12:26.186210+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:56.694970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423633+0000","last_change":"2024-04-02T09:12:25.201746+0000","last_active":"2024-04-02T09:12:26.423633+0000","last_peered":"2024-04-02T09:12:26.423633+0000","last_clean":"2024-04-02T09:12:26.423633+0000","last_became_active":"2024-04-02T09:12:25.201445+0000","last_became_peered":"2024-04-02T09:12:25.201445+0000","last_unstale":"2024-04-02T09:12:26.423633+0000","last_undegraded":"2024-04-02T09:12:26.423633+0000","last_fullsized":"2024-04-02T09:12:26.423633+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:06:08.648823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.10","version":"222'8","reported_seq":19,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186388+0000","last_change":"2024-04-02T09:12:24.692205+0000","last_active":"2024-04-02T09:12:26.186388+0000","last_peered":"2024-04-02T09:12:26.186388+0000","last_clean":"2024-04-02T09:12:26.186388+0000","last_became_active":"2024-04-02T09:12:24.690462+0000","last_became_peered":"2024-04-02T09:12:24.690462+0000","last_unstale":"2024-04-02T09:12:26.186388+0000","last_undegraded":"2024-04-02T09:12:26.186388+0000","last_fullsized":"2024-04-02T09:12:26.186388+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:00:14.796518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.187135+0000","last_change":"2024-04-02T09:12:25.196196+0000","last_active":"2024-04-02T09:12:26.187135+0000","last_peered":"2024-04-02T09:12:26.187135+0000","last_clean":"2024-04-02T09:12:26.187135+0000","last_became_active":"2024-04-02T09:12:25.195967+0000","last_became_peered":"2024-04-02T09:12:25.195967+0000","last_unstale":"2024-04-02T09:12:26.187135+0000","last_undegraded":"2024-04-02T09:12:26.187135+0000","last_fullsized":"2024-04-02T09:12:26.187135+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:41.683998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.f","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424609+0000","last_change":"2024-04-02T09:12:24.695567+0000","last_active":"2024-04-02T09:12:26.424609+0000","last_peered":"2024-04-02T09:12:26.424609+0000","last_clean":"2024-04-02T09:12:26.424609+0000","last_became_active":"2024-04-02T09:12:24.695267+0000","last_became_peered":"2024-04-02T09:12:24.695267+0000","last_unstale":"2024-04-02T09:12:26.424609+0000","last_undegraded":"2024-04-02T09:12:26.424609+0000","last_fullsized":"2024-04-02T09:12:26.424609+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:02.378346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186447+0000","last_change":"2024-04-02T09:12:25.196053+0000","last_active":"2024-04-02T09:12:26.186447+0000","last_peered":"2024-04-02T09:12:26.186447+0000","last_clean":"2024-04-02T09:12:26.186447+0000","last_became_active":"2024-04-02T09:12:25.195799+0000","last_became_peered":"2024-04-02T09:12:25.195799+0000","last_unstale":"2024-04-02T09:12:26.186447+0000","last_undegraded":"2024-04-02T09:12:26.186447+0000","last_fullsized":"2024-04-02T09:12:26.186447+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:26:33.117159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422476+0000","last_change":"2024-04-02T09:12:24.690213+0000","last_active":"2024-04-02T09:12:26.422476+0000","last_peered":"2024-04-02T09:12:26.422476+0000","last_clean":"2024-04-02T09:12:26.422476+0000","last_became_active":"2024-04-02T09:12:24.689813+0000","last_became_peered":"2024-04-02T09:12:24.689813+0000","last_unstale":"2024-04-02T09:12:26.422476+0000","last_undegraded":"2024-04-02T09:12:26.422476+0000","last_fullsized":"2024-04-02T09:12:26.422476+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:11.366124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423670+0000","last_change":"2024-04-02T09:12:25.193723+0000","last_active":"2024-04-02T09:12:26.423670+0000","last_peered":"2024-04-02T09:12:26.423670+0000","last_clean":"2024-04-02T09:12:26.423670+0000","last_became_active":"2024-04-02T09:12:25.193367+0000","last_became_peered":"2024-04-02T09:12:25.193367+0000","last_unstale":"2024-04-02T09:12:26.423670+0000","last_undegraded":"2024-04-02T09:12:26.423670+0000","last_fullsized":"2024-04-02T09:12:26.423670+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:48.218780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.d","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423498+0000","last_change":"2024-04-02T09:12:24.688637+0000","last_active":"2024-04-02T09:12:26.423498+0000","last_peered":"2024-04-02T09:12:26.423498+0000","last_clean":"2024-04-02T09:12:26.423498+0000","last_became_active":"2024-04-02T09:12:24.688321+0000","last_became_peered":"2024-04-02T09:12:24.688321+0000","last_unstale":"2024-04-02T09:12:26.423498+0000","last_undegraded":"2024-04-02T09:12:26.423498+0000","last_fullsized":"2024-04-02T09:12:26.423498+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:41:28.231249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424421+0000","last_change":"2024-04-02T09:12:25.190936+0000","last_active":"2024-04-02T09:12:26.424421+0000","last_peered":"2024-04-02T09:12:26.424421+0000","last_clean":"2024-04-02T09:12:26.424421+0000","last_became_active":"2024-04-02T09:12:25.189970+0000","last_became_peered":"2024-04-02T09:12:25.189970+0000","last_unstale":"2024-04-02T09:12:26.424421+0000","last_undegraded":"2024-04-02T09:12:26.424421+0000","last_fullsized":"2024-04-02T09:12:26.424421+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:57.372310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424062+0000","last_change":"2024-04-02T09:12:24.692444+0000","last_active":"2024-04-02T09:12:26.424062+0000","last_peered":"2024-04-02T09:12:26.424062+0000","last_clean":"2024-04-02T09:12:26.424062+0000","last_became_active":"2024-04-02T09:12:24.692165+0000","last_became_peered":"2024-04-02T09:12:24.692165+0000","last_unstale":"2024-04-02T09:12:26.424062+0000","last_undegraded":"2024-04-02T09:12:26.424062+0000","last_fullsized":"2024-04-02T09:12:26.424062+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:29.826236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423682+0000","last_change":"2024-04-02T09:12:25.195964+0000","last_active":"2024-04-02T09:12:26.423682+0000","last_peered":"2024-04-02T09:12:26.423682+0000","last_clean":"2024-04-02T09:12:26.423682+0000","last_became_active":"2024-04-02T09:12:25.195716+0000","last_became_peered":"2024-04-02T09:12:25.195716+0000","last_unstale":"2024-04-02T09:12:26.423682+0000","last_undegraded":"2024-04-02T09:12:26.423682+0000","last_fullsized":"2024-04-02T09:12:26.423682+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:25.408279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.b","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423122+0000","last_change":"2024-04-02T09:12:24.693402+0000","last_active":"2024-04-02T09:12:26.423122+0000","last_peered":"2024-04-02T09:12:26.423122+0000","last_clean":"2024-04-02T09:12:26.423122+0000","last_became_active":"2024-04-02T09:12:24.693047+0000","last_became_peered":"2024-04-02T09:12:24.693047+0000","last_unstale":"2024-04-02T09:12:26.423122+0000","last_undegraded":"2024-04-02T09:12:26.423122+0000","last_fullsized":"2024-04-02T09:12:26.423122+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:15:17.948098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424048+0000","last_change":"2024-04-02T09:12:25.195866+0000","last_active":"2024-04-02T09:12:26.424048+0000","last_peered":"2024-04-02T09:12:26.424048+0000","last_clean":"2024-04-02T09:12:26.424048+0000","last_became_active":"2024-04-02T09:12:25.195430+0000","last_became_peered":"2024-04-02T09:12:25.195430+0000","last_unstale":"2024-04-02T09:12:26.424048+0000","last_undegraded":"2024-04-02T09:12:26.424048+0000","last_fullsized":"2024-04-02T09:12:26.424048+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:42:28.562906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186235+0000","last_change":"2024-04-02T09:12:24.691812+0000","last_active":"2024-04-02T09:12:26.186235+0000","last_peered":"2024-04-02T09:12:26.186235+0000","last_clean":"2024-04-02T09:12:26.186235+0000","last_became_active":"2024-04-02T09:12:24.688935+0000","last_became_peered":"2024-04-02T09:12:24.688935+0000","last_unstale":"2024-04-02T09:12:26.186235+0000","last_undegraded":"2024-04-02T09:12:26.186235+0000","last_fullsized":"2024-04-02T09:12:26.186235+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:53.020585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185018+0000","last_change":"2024-04-02T09:12:25.189098+0000","last_active":"2024-04-02T09:12:26.185018+0000","last_peered":"2024-04-02T09:12:26.185018+0000","last_clean":"2024-04-02T09:12:26.185018+0000","last_became_active":"2024-04-02T09:12:25.187377+0000","last_became_peered":"2024-04-02T09:12:25.187377+0000","last_unstale":"2024-04-02T09:12:26.185018+0000","last_undegraded":"2024-04-02T09:12:26.185018+0000","last_fullsized":"2024-04-02T09:12:26.185018+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:58:52.752985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422824+0000","last_change":"2024-04-02T09:12:24.699586+0000","last_active":"2024-04-02T09:12:26.422824+0000","last_peered":"2024-04-02T09:12:26.422824+0000","last_clean":"2024-04-02T09:12:26.422824+0000","last_became_active":"2024-04-02T09:12:24.697592+0000","last_became_peered":"2024-04-02T09:12:24.697592+0000","last_unstale":"2024-04-02T09:12:26.422824+0000","last_undegraded":"2024-04-02T09:12:26.422824+0000","last_fullsized":"2024-04-02T09:12:26.422824+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:53.870135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185386+0000","last_change":"2024-04-02T09:12:25.205206+0000","last_active":"2024-04-02T09:12:26.185386+0000","last_peered":"2024-04-02T09:12:26.185386+0000","last_clean":"2024-04-02T09:12:26.185386+0000","last_became_active":"2024-04-02T09:12:25.204526+0000","last_became_peered":"2024-04-02T09:12:25.204526+0000","last_unstale":"2024-04-02T09:12:26.185386+0000","last_undegraded":"2024-04-02T09:12:26.185386+0000","last_fullsized":"2024-04-02T09:12:26.185386+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:33:41.191493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186038+0000","last_change":"2024-04-02T09:12:24.683516+0000","last_active":"2024-04-02T09:12:26.186038+0000","last_peered":"2024-04-02T09:12:26.186038+0000","last_clean":"2024-04-02T09:12:26.186038+0000","last_became_active":"2024-04-02T09:12:24.683207+0000","last_became_peered":"2024-04-02T09:12:24.683207+0000","last_unstale":"2024-04-02T09:12:26.186038+0000","last_undegraded":"2024-04-02T09:12:26.186038+0000","last_fullsized":"2024-04-02T09:12:26.186038+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:19:15.076490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.187152+0000","last_change":"2024-04-02T09:12:25.196365+0000","last_active":"2024-04-02T09:12:26.187152+0000","last_peered":"2024-04-02T09:12:26.187152+0000","last_clean":"2024-04-02T09:12:26.187152+0000","last_became_active":"2024-04-02T09:12:25.191868+0000","last_became_peered":"2024-04-02T09:12:25.191868+0000","last_unstale":"2024-04-02T09:12:26.187152+0000","last_undegraded":"2024-04-02T09:12:26.187152+0000","last_fullsized":"2024-04-02T09:12:26.187152+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:43:29.229527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"222'2","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186316+0000","last_change":"2024-04-02T09:12:24.696113+0000","last_active":"2024-04-02T09:12:26.186316+0000","last_peered":"2024-04-02T09:12:26.186316+0000","last_clean":"2024-04-02T09:12:26.186316+0000","last_became_active":"2024-04-02T09:12:24.695191+0000","last_became_peered":"2024-04-02T09:12:24.695191+0000","last_unstale":"2024-04-02T09:12:26.186316+0000","last_undegraded":"2024-04-02T09:12:26.186316+0000","last_fullsized":"2024-04-02T09:12:26.186316+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:41:21.090616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424613+0000","last_change":"2024-04-02T09:12:25.196988+0000","last_active":"2024-04-02T09:12:26.424613+0000","last_peered":"2024-04-02T09:12:26.424613+0000","last_clean":"2024-04-02T09:12:26.424613+0000","last_became_active":"2024-04-02T09:12:25.196031+0000","last_became_peered":"2024-04-02T09:12:25.196031+0000","last_unstale":"2024-04-02T09:12:26.424613+0000","last_undegraded":"2024-04-02T09:12:26.424613+0000","last_fullsized":"2024-04-02T09:12:26.424613+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:50.975576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423068+0000","last_change":"2024-04-02T09:12:24.686970+0000","last_active":"2024-04-02T09:12:26.423068+0000","last_peered":"2024-04-02T09:12:26.423068+0000","last_clean":"2024-04-02T09:12:26.423068+0000","last_became_active":"2024-04-02T09:12:24.686686+0000","last_became_peered":"2024-04-02T09:12:24.686686+0000","last_unstale":"2024-04-02T09:12:26.423068+0000","last_undegraded":"2024-04-02T09:12:26.423068+0000","last_fullsized":"2024-04-02T09:12:26.423068+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:17.201623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423095+0000","last_change":"2024-04-02T09:12:25.200491+0000","last_active":"2024-04-02T09:12:26.423095+0000","last_peered":"2024-04-02T09:12:26.423095+0000","last_clean":"2024-04-02T09:12:26.423095+0000","last_became_active":"2024-04-02T09:12:25.200208+0000","last_became_peered":"2024-04-02T09:12:25.200208+0000","last_unstale":"2024-04-02T09:12:26.423095+0000","last_undegraded":"2024-04-02T09:12:26.423095+0000","last_fullsized":"2024-04-02T09:12:26.423095+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:53:44.514037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424438+0000","last_change":"2024-04-02T09:12:24.691094+0000","last_active":"2024-04-02T09:12:26.424438+0000","last_peered":"2024-04-02T09:12:26.424438+0000","last_clean":"2024-04-02T09:12:26.424438+0000","last_became_active":"2024-04-02T09:12:24.690782+0000","last_became_peered":"2024-04-02T09:12:24.690782+0000","last_unstale":"2024-04-02T09:12:26.424438+0000","last_undegraded":"2024-04-02T09:12:26.424438+0000","last_fullsized":"2024-04-02T09:12:26.424438+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:17.789082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186855+0000","last_change":"2024-04-02T09:12:25.196931+0000","last_active":"2024-04-02T09:12:26.186855+0000","last_peered":"2024-04-02T09:12:26.186855+0000","last_clean":"2024-04-02T09:12:26.186855+0000","last_became_active":"2024-04-02T09:12:25.194947+0000","last_became_peered":"2024-04-02T09:12:25.194947+0000","last_unstale":"2024-04-02T09:12:26.186855+0000","last_undegraded":"2024-04-02T09:12:26.186855+0000","last_fullsized":"2024-04-02T09:12:26.186855+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:21.964569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424196+0000","last_change":"2024-04-02T09:12:24.693871+0000","last_active":"2024-04-02T09:12:26.424196+0000","last_peered":"2024-04-02T09:12:26.424196+0000","last_clean":"2024-04-02T09:12:26.424196+0000","last_became_active":"2024-04-02T09:12:24.693562+0000","last_became_peered":"2024-04-02T09:12:24.693562+0000","last_unstale":"2024-04-02T09:12:26.424196+0000","last_undegraded":"2024-04-02T09:12:26.424196+0000","last_fullsized":"2024-04-02T09:12:26.424196+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:48:24.676039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.424040+0000","last_change":"2024-04-02T09:12:25.194789+0000","last_active":"2024-04-02T09:12:26.424040+0000","last_peered":"2024-04-02T09:12:26.424040+0000","last_clean":"2024-04-02T09:12:26.424040+0000","last_became_active":"2024-04-02T09:12:25.192429+0000","last_became_peered":"2024-04-02T09:12:25.192429+0000","last_unstale":"2024-04-02T09:12:26.424040+0000","last_undegraded":"2024-04-02T09:12:26.424040+0000","last_fullsized":"2024-04-02T09:12:26.424040+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:11:59.558881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185795+0000","last_change":"2024-04-02T09:12:24.693539+0000","last_active":"2024-04-02T09:12:26.185795+0000","last_peered":"2024-04-02T09:12:26.185795+0000","last_clean":"2024-04-02T09:12:26.185795+0000","last_became_active":"2024-04-02T09:12:24.693275+0000","last_became_peered":"2024-04-02T09:12:24.693275+0000","last_unstale":"2024-04-02T09:12:26.185795+0000","last_undegraded":"2024-04-02T09:12:26.185795+0000","last_fullsized":"2024-04-02T09:12:26.185795+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:22:34.495765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422860+0000","last_change":"2024-04-02T09:12:25.197899+0000","last_active":"2024-04-02T09:12:26.422860+0000","last_peered":"2024-04-02T09:12:26.422860+0000","last_clean":"2024-04-02T09:12:26.422860+0000","last_became_active":"2024-04-02T09:12:25.197631+0000","last_became_peered":"2024-04-02T09:12:25.197631+0000","last_unstale":"2024-04-02T09:12:26.422860+0000","last_undegraded":"2024-04-02T09:12:26.422860+0000","last_fullsized":"2024-04-02T09:12:26.422860+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:02.349107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423712+0000","last_change":"2024-04-02T09:12:24.694385+0000","last_active":"2024-04-02T09:12:26.423712+0000","last_peered":"2024-04-02T09:12:26.423712+0000","last_clean":"2024-04-02T09:12:26.423712+0000","last_became_active":"2024-04-02T09:12:24.693609+0000","last_became_peered":"2024-04-02T09:12:24.693609+0000","last_unstale":"2024-04-02T09:12:26.423712+0000","last_undegraded":"2024-04-02T09:12:26.423712+0000","last_fullsized":"2024-04-02T09:12:26.423712+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:46:08.099991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185973+0000","last_change":"2024-04-02T09:12:25.203957+0000","last_active":"2024-04-02T09:12:26.185973+0000","last_peered":"2024-04-02T09:12:26.185973+0000","last_clean":"2024-04-02T09:12:26.185973+0000","last_became_active":"2024-04-02T09:12:25.203680+0000","last_became_peered":"2024-04-02T09:12:25.203680+0000","last_unstale":"2024-04-02T09:12:26.185973+0000","last_undegraded":"2024-04-02T09:12:26.185973+0000","last_fullsized":"2024-04-02T09:12:26.185973+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:31.267333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.186456+0000","last_change":"2024-04-02T09:12:24.695544+0000","last_active":"2024-04-02T09:12:26.186456+0000","last_peered":"2024-04-02T09:12:26.186456+0000","last_clean":"2024-04-02T09:12:26.186456+0000","last_became_active":"2024-04-02T09:12:24.695032+0000","last_became_peered":"2024-04-02T09:12:24.695032+0000","last_unstale":"2024-04-02T09:12:26.186456+0000","last_undegraded":"2024-04-02T09:12:26.186456+0000","last_fullsized":"2024-04-02T09:12:26.186456+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:41.705096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185482+0000","last_change":"2024-04-02T09:12:25.203302+0000","last_active":"2024-04-02T09:12:26.185482+0000","last_peered":"2024-04-02T09:12:26.185482+0000","last_clean":"2024-04-02T09:12:26.185482+0000","last_became_active":"2024-04-02T09:12:25.202579+0000","last_became_peered":"2024-04-02T09:12:25.202579+0000","last_unstale":"2024-04-02T09:12:26.185482+0000","last_undegraded":"2024-04-02T09:12:26.185482+0000","last_fullsized":"2024-04-02T09:12:26.185482+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:25.603157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.185101+0000","last_change":"2024-04-02T09:12:24.698442+0000","last_active":"2024-04-02T09:12:26.185101+0000","last_peered":"2024-04-02T09:12:26.185101+0000","last_clean":"2024-04-02T09:12:26.185101+0000","last_became_active":"2024-04-02T09:12:24.698179+0000","last_became_peered":"2024-04-02T09:12:24.698179+0000","last_unstale":"2024-04-02T09:12:26.185101+0000","last_undegraded":"2024-04-02T09:12:26.185101+0000","last_fullsized":"2024-04-02T09:12:26.185101+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:48.278840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":10,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.423599+0000","last_change":"2024-04-02T09:12:25.190434+0000","last_active":"2024-04-02T09:12:26.423599+0000","last_peered":"2024-04-02T09:12:26.423599+0000","last_clean":"2024-04-02T09:12:26.423599+0000","last_became_active":"2024-04-02T09:12:25.190187+0000","last_became_peered":"2024-04-02T09:12:25.190187+0000","last_unstale":"2024-04-02T09:12:26.423599+0000","last_undegraded":"2024-04-02T09:12:26.423599+0000","last_fullsized":"2024-04-02T09:12:26.423599+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:01.850942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"222'1","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-04-02T09:12:26.422918+0000","last_change":"2024-04-02T09:12:24.687221+0000","last_active":"2024-04-02T09:12:26.422918+0000","last_peered":"2024-04-02T09:12:26.422918+0000","last_clean":"2024-04-02T09:12:26.422918+0000","last_became_active":"2024-04-02T09:12:24.686953+0000","last_became_peered":"2024-04-02T09:12:24.686953+0000","last_unstale":"2024-04-02T09:12:26.422918+0000","last_undegraded":"2024-04-02T09:12:26.422918+0000","last_fullsized":"2024-04-02T09:12:26.422918+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:39:31.379977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"211'348","reported_seq":1211,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-04-02T09:12:25.179741+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:12:25.179741+0000","last_peered":"2024-04-02T09:12:25.179741+0000","last_clean":"2024-04-02T09:12:25.179741+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:12:25.179741+0000","last_undegraded":"2024-04-02T09:12:25.179741+0000","last_fullsized":"2024-04-02T09:12:25.179741+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575988,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60084,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311756,"statfs":{"total":96636764160,"available":96575238144,"internally_reserved":0,"allocated":2789376,"data_stored":1412284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.68500000000000005,"15min":0.71199999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.93899999999999995,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":8.016},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68700000000000006,"15min":0.71999999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91800000000000004,"5min":0.94199999999999995,"15min":1.165},"last":5.1790000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.70599999999999996,"15min":0.73099999999999998},"min":{"1min":0.48899999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.95399999999999996,"5min":0.95699999999999996,"15min":1.125},"last":7.5819999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68100000000000005,"15min":0.71399999999999997},"min":{"1min":0.51700000000000002,"5min":0.497,"15min":0.42399999999999999},"max":{"1min":0.89100000000000001,"5min":0.92700000000000005,"15min":1.042},"last":4.2569999999999997}]},{"osd":2,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.749,"15min":0.76100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.873,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":7.9790000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.78200000000000003,"15min":0.80500000000000005},"min":{"1min":0.54900000000000004,"5min":0.504,"15min":0.504},"max":{"1min":0.98299999999999998,"5min":1.036,"15min":1.109},"last":4.8819999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.72399999999999998,"15min":0.74099999999999999},"min":{"1min":0.497,"5min":0.45800000000000002,"15min":0.41399999999999998},"max":{"1min":0.94399999999999995,"5min":1.452,"15min":1.452},"last":8.0530000000000008},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.748,"15min":0.76400000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.0740000000000001},"last":5.093}]},{"osd":4,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61699999999999999,"15min":0.60799999999999998},"min":{"1min":0.42799999999999999,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.95299999999999996,"15min":1.01},"last":5.4589999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.73699999999999999,"15min":0.71699999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.38300000000000001},"max":{"1min":1.006,"5min":1.536,"15min":1.536},"last":7.9409999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.68799999999999994,"15min":0.67500000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":0.93400000000000005,"5min":1.0169999999999999,"15min":1.1850000000000001},"last":5.1340000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.72499999999999998,"15min":0.71699999999999997},"min":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.40600000000000003},"max":{"1min":0.88500000000000001,"5min":1.097,"15min":1.1319999999999999},"last":3.492}]},{"osd":6,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45800000000000002},"max":{"1min":0.871,"5min":1.034,"15min":1.3919999999999999},"last":1.861},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.72299999999999998,"15min":0.69099999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.97299999999999998,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":7.8899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575986,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42040,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329800,"statfs":{"total":96636764160,"available":96593715200,"internally_reserved":0,"allocated":2789376,"data_stored":1412332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.73699999999999999},"min":{"1min":0.54900000000000004,"5min":0.50800000000000001,"15min":0.42699999999999999},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73199999999999998,"15min":0.747},"min":{"1min":0.48699999999999999,"5min":0.47899999999999998,"15min":0.45800000000000002},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":1.077},"last":0.55300000000000005}]},{"osd":1,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70399999999999996,"15min":0.72099999999999997},"min":{"1min":0.57199999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.212},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.73499999999999999,"15min":0.75800000000000001},"min":{"1min":0.55700000000000005,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.64000000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.73599999999999999,"15min":0.75800000000000001},"min":{"1min":0.53500000000000003,"5min":0.47699999999999998,"15min":0.47199999999999998},"max":{"1min":1.016,"5min":1.016,"15min":1.2789999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.73899999999999999,"15min":0.748},"min":{"1min":0.50900000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.93300000000000005,"5min":0.95999999999999996,"15min":1.226},"last":0.52000000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.74399999999999999,"15min":0.76900000000000002},"min":{"1min":0.50800000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.141,"5min":1.141,"15min":1.2549999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76900000000000002,"15min":0.77500000000000002},"min":{"1min":0.63100000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.94099999999999995,"5min":0.95399999999999996,"15min":1.3200000000000001},"last":0.61499999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60799999999999998,"15min":0.623},"min":{"1min":0.45300000000000001,"5min":0.42199999999999999,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.1259999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.65100000000000002,"15min":0.66700000000000004},"min":{"1min":0.51100000000000001,"5min":0.47399999999999998,"15min":0.41999999999999998},"max":{"1min":0.84399999999999997,"5min":0.92300000000000004,"15min":1.1879999999999999},"last":0.499}]},{"osd":5,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64500000000000002,"15min":0.65600000000000003},"min":{"1min":0.47899999999999998,"5min":0.42799999999999999,"15min":0.42199999999999999},"max":{"1min":0.86599999999999999,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70099999999999996,"15min":0.72599999999999998},"min":{"1min":0.505,"5min":0.48299999999999998,"15min":0.46700000000000003},"max":{"1min":0.92000000000000004,"5min":1.034,"15min":1.204},"last":0.69299999999999995}]},{"osd":7,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.2370000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.72199999999999998,"15min":0.73899999999999999},"min":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.48299999999999998},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.57899999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575987,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52488,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319352,"statfs":{"total":96636764160,"available":96583016448,"internally_reserved":0,"allocated":3391488,"data_stored":2009378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44400000000000001,"15min":0.45300000000000001},"min":{"1min":0.35499999999999998,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.67000000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.502,"15min":0.51200000000000001},"min":{"1min":0.371,"5min":0.312,"15min":0.312},"max":{"1min":0.76000000000000001,"5min":0.78600000000000003,"15min":0.79800000000000004},"last":0.48499999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.50900000000000001,"15min":0.505},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.753,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58299999999999996,"15min":0.59699999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74099999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.68100000000000005}]},{"osd":3,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.42699999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76200000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.505},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.40100000000000002,"5min":0.39000000000000001,"15min":0.378},"max":{"1min":0.73199999999999998,"5min":0.86699999999999999,"15min":0.878},"last":0.374}]},{"osd":4,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.79700000000000004,"15min":0.79200000000000004},"min":{"1min":0.61099999999999999,"5min":0.57599999999999996,"15min":0.433},"max":{"1min":0.98499999999999999,"5min":1.1120000000000001,"15min":1.1479999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.78000000000000003,"15min":0.78900000000000003},"min":{"1min":0.61299999999999999,"5min":0.57199999999999995,"15min":0.41799999999999998},"max":{"1min":1.05,"5min":1.05,"15min":1.25},"last":0.61299999999999999}]},{"osd":5,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.77700000000000002,"15min":0.79600000000000004},"min":{"1min":0.59799999999999998,"5min":0.54300000000000004,"15min":0.52900000000000003},"max":{"1min":1.01,"5min":1.115,"15min":1.321},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.65800000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.173},"last":0.754}]},{"osd":6,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.81200000000000006,"15min":0.81799999999999995},"min":{"1min":0.67300000000000004,"5min":0.53900000000000003,"15min":0.48099999999999998},"max":{"1min":1.0649999999999999,"5min":1.159,"15min":1.2210000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.82199999999999995,"15min":0.81299999999999994},"min":{"1min":0.61899999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.087,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.80900000000000005}]},{"osd":7,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.82099999999999995,"15min":0.82799999999999996},"min":{"1min":0.64200000000000002,"5min":0.58499999999999996,"15min":0.53700000000000003},"max":{"1min":1.0680000000000001,"5min":1.1080000000000001,"15min":1.236},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.83899999999999997,"15min":0.83999999999999997},"min":{"1min":0.63700000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0329999999999999,"5min":1.0720000000000001,"15min":1.2789999999999999},"last":0.66100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575987,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47804,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324036,"statfs":{"total":96636764160,"available":96587812864,"internally_reserved":0,"allocated":2797568,"data_stored":1418573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45400000000000001,"15min":0.45800000000000002},"min":{"1min":0.33700000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.69899999999999995,"5min":0.76500000000000001,"15min":0.86699999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.5,"15min":0.51000000000000001},"min":{"1min":0.32800000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.36899999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.42099999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66100000000000003,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53900000000000003,"15min":0.55400000000000005},"min":{"1min":0.36699999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72499999999999998,"5min":0.999,"15min":0.999},"last":0.495}]},{"osd":3,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.40100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.77800000000000002,"5min":0.80500000000000005,"15min":0.85699999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.63900000000000001,"15min":0.64300000000000002},"min":{"1min":0.5,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.82799999999999996,"5min":0.92000000000000004,"15min":0.96399999999999997},"last":0.63400000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.82499999999999996,"15min":0.83299999999999996},"min":{"1min":0.65700000000000003,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.2769999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81000000000000005,"15min":0.82399999999999995},"min":{"1min":0.621,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.83399999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.80100000000000005,"15min":0.82699999999999996},"min":{"1min":0.58899999999999997,"5min":0.497,"15min":0.497},"max":{"1min":1.1419999999999999,"5min":1.145,"15min":1.232},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.83499999999999996,"15min":0.83399999999999996},"min":{"1min":0.67700000000000005,"5min":0.57699999999999996,"15min":0.55900000000000005},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.84099999999999997}]},{"osd":6,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.58099999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1830000000000001,"5min":1.196,"15min":1.196},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.80500000000000005,"15min":0.81200000000000006},"min":{"1min":0.60499999999999998,"5min":0.52900000000000003,"15min":0.499},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.72599999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.84399999999999997,"15min":0.85399999999999998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.84099999999999997,"15min":0.85099999999999998},"min":{"1min":0.67700000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.1100000000000001,"5min":1.1719999999999999,"15min":1.2589999999999999},"last":0.76600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51908,"kb_used_data":2740,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319932,"statfs":{"total":96636764160,"available":96583610368,"internally_reserved":0,"allocated":2805760,"data_stored":1413930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44},"min":{"1min":0.32200000000000001,"5min":0.30499999999999999,"15min":0.29199999999999998},"max":{"1min":0.61499999999999999,"5min":0.66600000000000004,"15min":0.72999999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.502,"15min":0.503},"min":{"1min":0.375,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64800000000000002,"5min":0.79900000000000004,"15min":1.4930000000000001},"last":0.33200000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.40999999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65800000000000003,"5min":0.75800000000000001,"15min":0.80600000000000005},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.40100000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.68799999999999994,"5min":0.81999999999999995,"15min":1.4530000000000001},"last":0.39700000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.56000000000000005},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.80700000000000005,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.628,"15min":0.63500000000000001},"min":{"1min":0.50900000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77700000000000002,"5min":0.86799999999999999,"15min":1.0740000000000001},"last":0.48699999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.77700000000000002,"15min":0.77900000000000003},"min":{"1min":0.53700000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.008,"5min":1.121,"15min":1.121},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79300000000000004,"15min":0.79800000000000004},"min":{"1min":0.65200000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.1259999999999999,"5min":1.179,"15min":1.5329999999999999},"last":0.59699999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80000000000000004,"15min":0.79300000000000004},"min":{"1min":0.69899999999999995,"5min":0.53100000000000003,"15min":0.52600000000000002},"max":{"1min":1.054,"5min":1.99,"15min":1.99},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80500000000000005,"15min":0.80800000000000005},"min":{"1min":0.60499999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0660000000000001,"5min":1.101,"15min":1.1259999999999999},"last":0.73599999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.81599999999999995,"15min":0.81799999999999995},"min":{"1min":0.63500000000000001,"5min":0.54700000000000004,"15min":0.54600000000000004},"max":{"1min":0.96899999999999997,"5min":1.173,"15min":1.5940000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80400000000000005,"15min":0.81100000000000005},"min":{"1min":0.63200000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.99199999999999999,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.66600000000000004}]},{"osd":7,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.83199999999999996,"15min":0.83499999999999996},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55100000000000005},"max":{"1min":1.165,"5min":1.2110000000000001,"15min":1.5649999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.81999999999999995,"15min":0.83099999999999996},"min":{"1min":0.70499999999999996,"5min":0.497,"15min":0.497},"max":{"1min":1.0269999999999999,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.61499999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575986,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51900,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319940,"statfs":{"total":96636764160,"available":96583618560,"internally_reserved":0,"allocated":2797568,"data_stored":1413826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44400000000000001,"15min":0.45300000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.48699999999999999,"15min":0.496},"min":{"1min":0.36099999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.876},"last":0.55100000000000005}]},{"osd":1,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.49399999999999999,"15min":0.502},"min":{"1min":0.34300000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.64000000000000001,"5min":0.71499999999999997,"15min":0.90300000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.57699999999999996},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.34100000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.376}]},{"osd":2,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.622,"15min":0.63500000000000001},"min":{"1min":0.42599999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.61199999999999999,"15min":0.623},"min":{"1min":0.48099999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81699999999999995,"5min":0.82299999999999995,"15min":1.0840000000000001},"last":0.47999999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.74399999999999999,"15min":0.75900000000000001},"min":{"1min":0.503,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.071,"5min":1.071,"15min":1.0800000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.59099999999999997,"5min":0.57199999999999995,"15min":0.49199999999999999},"max":{"1min":1.018,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.75700000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.59599999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.054,"5min":1.1459999999999999,"15min":1.157},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.56599999999999995,"5min":0.505,"15min":0.505},"max":{"1min":0.99099999999999999,"5min":1.266,"15min":1.266},"last":0.83499999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.78500000000000003,"15min":0.79100000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.52600000000000002},"max":{"1min":1.0489999999999999,"5min":1.284,"15min":1.284},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.80400000000000005,"15min":0.81200000000000006},"min":{"1min":0.54900000000000004,"5min":0.495,"15min":0.495},"max":{"1min":1.04,"5min":1.2,"15min":1.2},"last":0.80700000000000005}]},{"osd":7,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.80400000000000005,"15min":0.81499999999999995},"min":{"1min":0.61399999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.024,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.57999999999999996,"5min":0.55700000000000005,"15min":0.53100000000000003},"max":{"1min":1.103,"5min":1.248,"15min":1.248},"last":0.79700000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575986,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47812,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324028,"statfs":{"total":96636764160,"available":96587804672,"internally_reserved":0,"allocated":2801664,"data_stored":1419100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68000000000000005,"15min":0.69899999999999995},"min":{"1min":0.55000000000000004,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.753},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66500000000000004,"15min":0.68899999999999995},"min":{"1min":0.40999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.83499999999999996,"5min":0.86199999999999999,"15min":1.075},"last":0.50900000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.73599999999999999,"15min":0.76100000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":1.0329999999999999,"5min":1.0549999999999999,"15min":1.069},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.75,"15min":0.77700000000000002},"min":{"1min":0.53900000000000003,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.70899999999999996}]},{"osd":2,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.69599999999999995,"15min":0.71799999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70499999999999996,"15min":0.73499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98999999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.623}]},{"osd":3,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75600000000000001,"15min":0.77000000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.47399999999999998},"max":{"1min":0.98099999999999998,"5min":0.998,"15min":1.0700000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.51800000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.002,"5min":1.002,"15min":1.028},"last":0.56899999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.60199999999999998,"15min":0.61099999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.91900000000000004,"5min":1.008,"15min":1.016},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60399999999999998,"15min":0.61699999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.63700000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.64100000000000001,"15min":0.64200000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67300000000000004,"15min":0.69099999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":0.98299999999999998,"5min":1.0269999999999999,"15min":1.0469999999999999},"last":0.78000000000000003}]},{"osd":7,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.0329999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.76600000000000001,"15min":0.78300000000000003},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0960000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.80100000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575989,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48380,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323460,"statfs":{"total":96636764160,"available":96587223040,"internally_reserved":0,"allocated":3391488,"data_stored":2002339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.68899999999999995,"15min":0.70799999999999996},"min":{"1min":0.57799999999999996,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.004},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.69199999999999995,"15min":0.72299999999999998},"min":{"1min":0.58199999999999996,"5min":0.432,"15min":0.432},"max":{"1min":0.92300000000000004,"5min":0.97199999999999998,"15min":1.0600000000000001},"last":0.63300000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.69699999999999995,"15min":0.72399999999999998},"min":{"1min":0.56000000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.747,"5min":0.70699999999999996,"15min":0.72899999999999998},"min":{"1min":0.53000000000000003,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":1.052},"last":0.71899999999999997}]},{"osd":2,"last update":"Tue Apr 2 09:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.73099999999999998,"15min":0.747},"min":{"1min":0.56399999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.014},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72699999999999998,"15min":0.74399999999999999},"min":{"1min":0.623,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.90200000000000002,"5min":0.98099999999999998,"15min":1.0680000000000001},"last":0.69499999999999995}]},{"osd":3,"last update":"Tue Apr 2 09:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.76300000000000001,"15min":0.78500000000000003},"min":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.52600000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":1.083},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.78500000000000003,"15min":0.79100000000000004},"min":{"1min":0.56100000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.95599999999999996,"5min":1.008,"15min":1.1020000000000001},"last":0.84299999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60299999999999998,"15min":0.621},"min":{"1min":0.42999999999999999,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.054},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65900000000000003,"15min":0.66600000000000004},"min":{"1min":0.47399999999999998,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.75700000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.67300000000000004,"15min":0.66400000000000003},"min":{"1min":0.47199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69299999999999995,"15min":0.70299999999999996},"min":{"1min":0.53600000000000003,"5min":0.44600000000000001,"15min":0.38700000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.86799999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71999999999999997,"15min":0.72599999999999998},"min":{"1min":0.499,"5min":0.441,"15min":0.441},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.754,"15min":0.75700000000000001},"min":{"1min":0.55900000000000005,"5min":0.48199999999999998,"15min":0.45700000000000002},"max":{"1min":1.0269999999999999,"5min":1.079,"15min":1.079},"last":0.88100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:12:30.043 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T09:12:30.043 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T09:12:30.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.280+0000 7f4ef5200700 1 Processor -- start 2024-04-02T09:12:30.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.280+0000 7f4ef5200700 1 -- start start 2024-04-02T09:12:30.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.281+0000 7f4ef5200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4ef015d9d0 0x7f4ef00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:30.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.281+0000 7f4ef5200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4ef00d2540 0x7f4ef00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:30.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.281+0000 7f4ef5200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef0161420 con 0x7f4ef00d16a0 2024-04-02T09:12:30.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.281+0000 7f4ef5200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef015edc0 con 0x7f4ef015d9d0 2024-04-02T09:12:30.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.281+0000 7f4ef5200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef0161700 con 0x7f4ef00d2540 2024-04-02T09:12:30.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.281+0000 7f4eede00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4ef015d9d0 0x7f4ef00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:30.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.281+0000 7f4eede00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4ef015d9d0 0x7f4ef00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52428/0 (socket says 172.21.15.89:52428) 2024-04-02T09:12:30.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.281+0000 7f4eede00700 1 -- 172.21.15.89:0/1853106200 learned_addr learned my addr 172.21.15.89:0/1853106200 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:30.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.281+0000 7f4eee800700 1 --2- 172.21.15.89:0/1853106200 >> v2:172.21.15.113:3300/0 conn(0x7f4ef00d2540 0x7f4ef00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:30.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.282+0000 7f4eede00700 1 -- 172.21.15.89:0/1853106200 >> v2:172.21.15.113:3300/0 conn(0x7f4ef00d2540 msgr2=0x7f4ef00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:30.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.282+0000 7f4eede00700 1 --2- 172.21.15.89:0/1853106200 >> v2:172.21.15.113:3300/0 conn(0x7f4ef00d2540 0x7f4ef00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.282+0000 7f4eede00700 1 -- 172.21.15.89:0/1853106200 >> v1:172.21.15.89:6789/0 conn(0x7f4ef00d16a0 legacy=0x7f4ef00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:30.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.282+0000 7f4eede00700 1 -- 172.21.15.89:0/1853106200 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef01619e0 con 0x7f4ef015d9d0 2024-04-02T09:12:30.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.282+0000 7f4eee800700 1 --2- 172.21.15.89:0/1853106200 >> v2:172.21.15.113:3300/0 conn(0x7f4ef00d2540 0x7f4ef00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:30.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.282+0000 7f4eede00700 1 --2- 172.21.15.89:0/1853106200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef015d9d0 0x7f4ef00d3c30 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f4ef008ca10 tx=0x7f4ee0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1486982bcec6d76d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:30.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.282+0000 7f4eed400700 1 -- 172.21.15.89:0/1853106200 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee000b870 con 0x7f4ef015d9d0 2024-04-02T09:12:30.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.282+0000 7f4eed400700 1 -- 172.21.15.89:0/1853106200 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee000b9d0 con 0x7f4ef015d9d0 2024-04-02T09:12:30.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.282+0000 7f4eed400700 1 -- 172.21.15.89:0/1853106200 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee000bd60 con 0x7f4ef015d9d0 2024-04-02T09:12:30.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.282+0000 7f4ef5200700 1 -- 172.21.15.89:0/1853106200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef015d9d0 msgr2=0x7f4ef00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:30.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.282+0000 7f4ef5200700 1 --2- 172.21.15.89:0/1853106200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef015d9d0 0x7f4ef00d3c30 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f4ef008ca10 tx=0x7f4ee0009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.283+0000 7f4ef5200700 1 -- 172.21.15.89:0/1853106200 shutdown_connections 2024-04-02T09:12:30.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.283+0000 7f4ef5200700 1 --2- 172.21.15.89:0/1853106200 >> v2:172.21.15.113:3300/0 conn(0x7f4ef00d2540 0x7f4ef00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.283+0000 7f4ef5200700 1 --2- 172.21.15.89:0/1853106200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef015d9d0 0x7f4ef00d3c30 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.283+0000 7f4ef5200700 1 -- 172.21.15.89:0/1853106200 >> 172.21.15.89:0/1853106200 conn(0x7f4ef008aff0 msgr2=0x7f4ef00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:30.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.283+0000 7f4ef5200700 1 -- 172.21.15.89:0/1853106200 shutdown_connections 2024-04-02T09:12:30.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.283+0000 7f4ef5200700 1 -- 172.21.15.89:0/1853106200 wait complete. 2024-04-02T09:12:30.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.284+0000 7f4ef5200700 1 Processor -- start 2024-04-02T09:12:30.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.284+0000 7f4ef5200700 1 -- start start 2024-04-02T09:12:30.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.285+0000 7f4ef5200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef00d16a0 0x7f4ef015cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:30.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.285+0000 7f4ef5200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ef00d2540 0x7f4ef01f5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:30.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.285+0000 7f4ef5200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef015d9d0 0x7f4ef015bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:30.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.285+0000 7f4ef5200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef015ca30 con 0x7f4ef00d16a0 2024-04-02T09:12:30.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.285+0000 7f4ef5200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef01f68f0 con 0x7f4ef00d2540 2024-04-02T09:12:30.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.285+0000 7f4ef5200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef01f5ce0 con 0x7f4ef015d9d0 2024-04-02T09:12:30.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.286+0000 7f4eede00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef00d16a0 0x7f4ef015cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:30.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.286+0000 7f4eede00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef00d16a0 0x7f4ef015cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52430/0 (socket says 172.21.15.89:52430) 2024-04-02T09:12:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.286+0000 7f4eede00700 1 -- 172.21.15.89:0/3772184867 learned_addr learned my addr 172.21.15.89:0/3772184867 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.286+0000 7f4eede00700 1 -- 172.21.15.89:0/3772184867 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef015d9d0 msgr2=0x7f4ef015bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.286+0000 7f4ee7e00700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ef00d2540 0x7f4ef01f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.286+0000 7f4eee800700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef015d9d0 0x7f4ef015bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.286+0000 7f4eede00700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef015d9d0 0x7f4ef015bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.286+0000 7f4eede00700 1 -- 172.21.15.89:0/3772184867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ef00d2540 msgr2=0x7f4ef01f5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.286+0000 7f4eede00700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ef00d2540 0x7f4ef01f5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.286+0000 7f4eede00700 1 -- 172.21.15.89:0/3772184867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee0009550 con 0x7f4ef00d16a0 2024-04-02T09:12:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.286+0000 7f4ee7e00700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ef00d2540 0x7f4ef01f5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.286+0000 7f4eede00700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef00d16a0 0x7f4ef015cc50 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0014040 tx=0x7f4ee0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.286+0000 7f4ee6000700 1 -- 172.21.15.89:0/3772184867 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0004520 con 0x7f4ef00d16a0 2024-04-02T09:12:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.287+0000 7f4ee6000700 1 -- 172.21.15.89:0/3772184867 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee001e070 con 0x7f4ef00d16a0 2024-04-02T09:12:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.287+0000 7f4ee6000700 1 -- 172.21.15.89:0/3772184867 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee00038e0 con 0x7f4ef00d16a0 2024-04-02T09:12:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.287+0000 7f4ef5200700 1 -- 172.21.15.89:0/3772184867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef01f5fc0 con 0x7f4ef00d16a0 2024-04-02T09:12:30.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.287+0000 7f4ef5200700 1 -- 172.21.15.89:0/3772184867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef01f6580 con 0x7f4ef00d16a0 2024-04-02T09:12:30.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.288+0000 7f4ef5200700 1 -- 172.21.15.89:0/3772184867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef00d1ca0 con 0x7f4ef00d16a0 2024-04-02T09:12:30.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.289+0000 7f4ee6000700 1 -- 172.21.15.89:0/3772184867 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4ee0003a40 con 0x7f4ef00d16a0 2024-04-02T09:12:30.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.290+0000 7f4ee6000700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4ecc06adb0 0x7f4ecc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:30.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.290+0000 7f4ee6000700 1 -- 172.21.15.89:0/3772184867 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f4ee0020070 con 0x7f4ef00d16a0 2024-04-02T09:12:30.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.290+0000 7f4ee7e00700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4ecc06adb0 0x7f4ecc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:30.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.290+0000 7f4ee7e00700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4ecc06adb0 0x7f4ecc06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4ef008b710 tx=0x7f4ed8009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:30.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.294+0000 7f4ee6000700 1 -- 172.21.15.89:0/3772184867 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4ee0055b90 con 0x7f4ef00d16a0 2024-04-02T09:12:30.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.571+0000 7f4ef5200700 1 -- 172.21.15.89:0/3772184867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4ef0067720 con 0x7f4ef00d16a0 2024-04-02T09:12:30.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.576+0000 7f4ee6000700 1 -- 172.21.15.89:0/3772184867 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v54) v1 ==== 190+0+0 (secure 0 0 0) 0x7f4ee002a0e0 con 0x7f4ef00d16a0 2024-04-02T09:12:30.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.578+0000 7f4ef5200700 1 -- 172.21.15.89:0/3772184867 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4ecc06adb0 msgr2=0x7f4ecc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:30.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.578+0000 7f4ef5200700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4ecc06adb0 0x7f4ecc06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4ef008b710 tx=0x7f4ed8009040 comp rx=0 tx=0).stop 2024-04-02T09:12:30.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.578+0000 7f4ef5200700 1 -- 172.21.15.89:0/3772184867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef00d16a0 msgr2=0x7f4ef015cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:30.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.578+0000 7f4ef5200700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef00d16a0 0x7f4ef015cc50 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0014040 tx=0x7f4ee0004750 comp rx=0 tx=0).stop 2024-04-02T09:12:30.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.578+0000 7f4ef5200700 1 -- 172.21.15.89:0/3772184867 shutdown_connections 2024-04-02T09:12:30.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.578+0000 7f4ef5200700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef015d9d0 0x7f4ef015bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.578+0000 7f4ef5200700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ef00d16a0 0x7f4ef015cc50 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.578+0000 7f4ef5200700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4ecc06adb0 0x7f4ecc06d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.578+0000 7f4ef5200700 1 --2- 172.21.15.89:0/3772184867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ef00d2540 0x7f4ef01f5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.578+0000 7f4ef5200700 1 -- 172.21.15.89:0/3772184867 >> 172.21.15.89:0/3772184867 conn(0x7f4ef008aff0 msgr2=0x7f4ef00b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:30.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.578+0000 7f4ef5200700 1 -- 172.21.15.89:0/3772184867 shutdown_connections 2024-04-02T09:12:30.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.578+0000 7f4ef5200700 1 -- 172.21.15.89:0/3772184867 wait complete. 2024-04-02T09:12:30.578 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T09:12:30.595 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:12:30.662 INFO:tasks.ceph.mds.a.smithi089.stderr:2024-04-02T09:12:30.664+0000 7fef9e800700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T09:12:30.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.788+0000 7f12b2a00700 1 Processor -- start 2024-04-02T09:12:30.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.788+0000 7f12b2a00700 1 -- start start 2024-04-02T09:12:30.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12b2a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12ac160f10 0x7f12ac0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:30.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12b2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f12ac0cf230 0x7f12ac0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:30.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12b2a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f12ac161590 0x7f12ac161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:30.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12b2a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12ac161f60 con 0x7f12ac0cf230 2024-04-02T09:12:30.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12b2a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12ac0ceda0 con 0x7f12ac161590 2024-04-02T09:12:30.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12b2a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac0d23d0 con 0x7f12ac160f10 2024-04-02T09:12:30.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12ab400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f12ac0cf230 0x7f12ac0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:30.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12ab400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f12ac0cf230 0x7f12ac0d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52442/0 (socket says 172.21.15.89:52442) 2024-04-02T09:12:30.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12ab400700 1 -- 172.21.15.89:0/2783219138 learned_addr learned my addr 172.21.15.89:0/2783219138 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:30.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12b0a00700 1 --2- 172.21.15.89:0/2783219138 >> v2:172.21.15.113:3300/0 conn(0x7f12ac161590 0x7f12ac161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:30.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12abe00700 1 --2- 172.21.15.89:0/2783219138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12ac160f10 0x7f12ac0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:30.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12ab400700 1 -- 172.21.15.89:0/2783219138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12ac160f10 msgr2=0x7f12ac0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:30.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12ab400700 1 --2- 172.21.15.89:0/2783219138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12ac160f10 0x7f12ac0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12ab400700 1 -- 172.21.15.89:0/2783219138 >> v2:172.21.15.113:3300/0 conn(0x7f12ac161590 msgr2=0x7f12ac161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:30.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12ab400700 1 --2- 172.21.15.89:0/2783219138 >> v2:172.21.15.113:3300/0 conn(0x7f12ac161590 0x7f12ac161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12ab400700 1 -- 172.21.15.89:0/2783219138 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac1612f0 con 0x7f12ac0cf230 2024-04-02T09:12:30.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.790+0000 7f12b0a00700 1 --2- 172.21.15.89:0/2783219138 >> v2:172.21.15.113:3300/0 conn(0x7f12ac161590 0x7f12ac161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:30.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.791+0000 7f12ab400700 1 --2- 172.21.15.89:0/2783219138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12ac0cf230 0x7f12ac0d1e90 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0cfad0 tx=0x7f1298009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9342b343b3325d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:30.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.791+0000 7f12aaa00700 1 -- 172.21.15.89:0/2783219138 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129800b870 con 0x7f12ac0cf230 2024-04-02T09:12:30.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.791+0000 7f12aaa00700 1 -- 172.21.15.89:0/2783219138 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129800b9d0 con 0x7f12ac0cf230 2024-04-02T09:12:30.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.791+0000 7f12aaa00700 1 -- 172.21.15.89:0/2783219138 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129800bd60 con 0x7f12ac0cf230 2024-04-02T09:12:30.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.791+0000 7f12b2a00700 1 -- 172.21.15.89:0/2783219138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12ac0cf230 msgr2=0x7f12ac0d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:30.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.791+0000 7f12b2a00700 1 --2- 172.21.15.89:0/2783219138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12ac0cf230 0x7f12ac0d1e90 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0cfad0 tx=0x7f1298009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.791+0000 7f12b2a00700 1 -- 172.21.15.89:0/2783219138 shutdown_connections 2024-04-02T09:12:30.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.791+0000 7f12b2a00700 1 --2- 172.21.15.89:0/2783219138 >> v2:172.21.15.113:3300/0 conn(0x7f12ac161590 0x7f12ac161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.791+0000 7f12b2a00700 1 --2- 172.21.15.89:0/2783219138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12ac160f10 0x7f12ac0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.791+0000 7f12b2a00700 1 --2- 172.21.15.89:0/2783219138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12ac0cf230 0x7f12ac0d1e90 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.791+0000 7f12b2a00700 1 -- 172.21.15.89:0/2783219138 >> 172.21.15.89:0/2783219138 conn(0x7f12ac08aff0 msgr2=0x7f12ac0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:30.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.792+0000 7f12b2a00700 1 -- 172.21.15.89:0/2783219138 shutdown_connections 2024-04-02T09:12:30.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.792+0000 7f12b2a00700 1 -- 172.21.15.89:0/2783219138 wait complete. 2024-04-02T09:12:30.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.793+0000 7f12b2a00700 1 Processor -- start 2024-04-02T09:12:30.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.793+0000 7f12b2a00700 1 -- start start 2024-04-02T09:12:30.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.794+0000 7f12b2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12ac0cf230 0x7f12ac155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:30.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.794+0000 7f12b2a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ac160f10 0x7f12ac0b8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.794+0000 7f12b2a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12ac161590 0x7f12ac0b86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.794+0000 7f12b2a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac1e7b30 con 0x7f12ac0cf230 2024-04-02T09:12:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.794+0000 7f12b2a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac1e7e10 con 0x7f12ac160f10 2024-04-02T09:12:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.794+0000 7f12b2a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac1e80f0 con 0x7f12ac161590 2024-04-02T09:12:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.795+0000 7f12abe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12ac0cf230 0x7f12ac155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.795+0000 7f12abe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12ac0cf230 0x7f12ac155080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52454/0 (socket says 172.21.15.89:52454) 2024-04-02T09:12:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.795+0000 7f12abe00700 1 -- 172.21.15.89:0/517665284 learned_addr learned my addr 172.21.15.89:0/517665284 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.795+0000 7f12ab400700 1 --2- 172.21.15.89:0/517665284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ac160f10 0x7f12ac0b8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.795+0000 7f12b0a00700 1 --2- 172.21.15.89:0/517665284 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12ac161590 0x7f12ac0b86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.795+0000 7f12abe00700 1 -- 172.21.15.89:0/517665284 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12ac161590 msgr2=0x7f12ac0b86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.795+0000 7f12abe00700 1 --2- 172.21.15.89:0/517665284 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12ac161590 0x7f12ac0b86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.795+0000 7f12abe00700 1 -- 172.21.15.89:0/517665284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ac160f10 msgr2=0x7f12ac0b8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:30.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.795+0000 7f12abe00700 1 --2- 172.21.15.89:0/517665284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ac160f10 0x7f12ac0b8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:30.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.795+0000 7f12abe00700 1 -- 172.21.15.89:0/517665284 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1298009550 con 0x7f12ac0cf230 2024-04-02T09:12:30.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.795+0000 7f12abe00700 1 --2- 172.21.15.89:0/517665284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12ac0cf230 0x7f12ac155080 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f129c00d800 tx=0x7f129c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:30.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.795+0000 7f12a3e00700 1 -- 172.21.15.89:0/517665284 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c0125f0 con 0x7f12ac0cf230 2024-04-02T09:12:30.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.795+0000 7f12a3e00700 1 -- 172.21.15.89:0/517665284 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129c012750 con 0x7f12ac0cf230 2024-04-02T09:12:30.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.795+0000 7f12a3e00700 1 -- 172.21.15.89:0/517665284 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c012ae0 con 0x7f12ac0cf230 2024-04-02T09:12:30.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.796+0000 7f12b2a00700 1 -- 172.21.15.89:0/517665284 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ac1e83d0 con 0x7f12ac0cf230 2024-04-02T09:12:30.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.796+0000 7f12b2a00700 1 -- 172.21.15.89:0/517665284 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12ac1e7780 con 0x7f12ac0cf230 2024-04-02T09:12:30.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.797+0000 7f12b2a00700 1 -- 172.21.15.89:0/517665284 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12ac157180 con 0x7f12ac0cf230 2024-04-02T09:12:30.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.798+0000 7f12a3e00700 1 -- 172.21.15.89:0/517665284 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f129c012c40 con 0x7f12ac0cf230 2024-04-02T09:12:30.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.798+0000 7f12a3e00700 1 --2- 172.21.15.89:0/517665284 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f128c06adb0 0x7f128c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:30.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.799+0000 7f12a3e00700 1 -- 172.21.15.89:0/517665284 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f129c089740 con 0x7f12ac0cf230 2024-04-02T09:12:30.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.799+0000 7f12ab400700 1 --2- 172.21.15.89:0/517665284 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f128c06adb0 0x7f128c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:30.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.799+0000 7f12ab400700 1 --2- 172.21.15.89:0/517665284 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f128c06adb0 0x7f128c06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1298014040 tx=0x7f1298008030 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:30.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:30.803+0000 7f12a3e00700 1 -- 172.21.15.89:0/517665284 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f129c0545b0 con 0x7f12ac0cf230 2024-04-02T09:12:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.049+0000 7f12b2a00700 1 -- 172.21.15.89:0/517665284 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12ac1e7f50 con 0x7f12ac0cf230 2024-04-02T09:12:31.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.050+0000 7f12a3e00700 1 -- 172.21.15.89:0/517665284 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+4755 (secure 0 0 0) 0x7f12ac1e7f50 con 0x7f12ac0cf230 2024-04-02T09:12:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:31.048 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":105,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8969,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3650347444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3650347444},{"type":"v1","addr":"172.21.15.89:6836","nonce":3650347444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2152500685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2152500685},{"type":"v1","addr":"172.21.15.113:6834","nonce":2152500685}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2457040702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2457040702},{"type":"v1","addr":"172.21.15.113:6835","nonce":2457040702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:25.170849+0000","modified":"2024-04-02T09:12:28.185307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11157":{"gid":11157,"name":"a","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2615927215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2615927215},{"type":"v1","addr":"172.21.15.89:6837","nonce":2615927215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-04-02T09:12:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.052+0000 7f12b2a00700 1 -- 172.21.15.89:0/517665284 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f128c06adb0 msgr2=0x7f128c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.052+0000 7f12b2a00700 1 --2- 172.21.15.89:0/517665284 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f128c06adb0 0x7f128c06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1298014040 tx=0x7f1298008030 comp rx=0 tx=0).stop 2024-04-02T09:12:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.052+0000 7f12b2a00700 1 -- 172.21.15.89:0/517665284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12ac0cf230 msgr2=0x7f12ac155080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.052+0000 7f12b2a00700 1 --2- 172.21.15.89:0/517665284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12ac0cf230 0x7f12ac155080 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f129c00d800 tx=0x7f129c00db10 comp rx=0 tx=0).stop 2024-04-02T09:12:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.053+0000 7f12b2a00700 1 -- 172.21.15.89:0/517665284 shutdown_connections 2024-04-02T09:12:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.053+0000 7f12b2a00700 1 --2- 172.21.15.89:0/517665284 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12ac161590 0x7f12ac0b86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.053+0000 7f12b2a00700 1 --2- 172.21.15.89:0/517665284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12ac0cf230 0x7f12ac155080 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.053+0000 7f12b2a00700 1 --2- 172.21.15.89:0/517665284 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f128c06adb0 0x7f128c06d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.053+0000 7f12b2a00700 1 --2- 172.21.15.89:0/517665284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ac160f10 0x7f12ac0b8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.053+0000 7f12b2a00700 1 -- 172.21.15.89:0/517665284 >> 172.21.15.89:0/517665284 conn(0x7f12ac08aff0 msgr2=0x7f12ac0cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.053+0000 7f12b2a00700 1 -- 172.21.15.89:0/517665284 shutdown_connections 2024-04-02T09:12:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.053+0000 7f12b2a00700 1 -- 172.21.15.89:0/517665284 wait complete. 2024-04-02T09:12:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 105 2024-04-02T09:12:31.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T09:12:25.170849+0000', 'modified': '2024-04-02T09:12:28.185307+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11157}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11157': {'gid': 11157, 'name': 'a', 'rank': 0, 'incarnation': 101, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6837/2615927215', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6835', 'nonce': 2615927215}, {'type': 'v1', 'addr': '172.21.15.89:6837', 'nonce': 2615927215}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T09:12:31.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T09:12:31.472 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T09:12:31.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.669+0000 7f421e000700 1 Processor -- start 2024-04-02T09:12:31.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.669+0000 7f421e000700 1 -- start start 2024-04-02T09:12:31.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.670+0000 7f421e000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f421815dad0 0x7f42180ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.670+0000 7f421e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42180d1630 0x7f42180d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.671+0000 7f421e000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421815f820 con 0x7f42180d1fd0 2024-04-02T09:12:31.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.671+0000 7f421e000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42180cfcc0 con 0x7f421815dad0 2024-04-02T09:12:31.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.671+0000 7f421e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42180cd9f0 con 0x7f42180d1630 2024-04-02T09:12:31.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.671+0000 7f4217400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f421815dad0 0x7f42180ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.671+0000 7f4217400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f421815dad0 0x7f42180ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52470/0 (socket says 172.21.15.89:52470) 2024-04-02T09:12:31.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.671+0000 7f4217400700 1 -- 172.21.15.89:0/698500967 learned_addr learned my addr 172.21.15.89:0/698500967 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:31.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.671+0000 7f4217400700 1 -- 172.21.15.89:0/698500967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42180d1630 msgr2=0x7f42180d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:31.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.671+0000 7f4217400700 1 --2- 172.21.15.89:0/698500967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42180d1630 0x7f42180d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.671+0000 7f4217400700 1 -- 172.21.15.89:0/698500967 >> v1:172.21.15.89:6789/0 conn(0x7f42180d1fd0 legacy=0x7f421815deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.671+0000 7f4217400700 1 -- 172.21.15.89:0/698500967 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42180d2460 con 0x7f421815dad0 2024-04-02T09:12:31.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.671+0000 7f4217400700 1 --2- 172.21.15.89:0/698500967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f421815dad0 0x7f42180ced80 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f42080098f0 tx=0x7f4208009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c961c3713d6fab5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:31.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.672+0000 7f4216a00700 1 -- 172.21.15.89:0/698500967 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420800b8a0 con 0x7f421815dad0 2024-04-02T09:12:31.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.672+0000 7f4216a00700 1 -- 172.21.15.89:0/698500967 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420800ba00 con 0x7f421815dad0 2024-04-02T09:12:31.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.672+0000 7f4216a00700 1 -- 172.21.15.89:0/698500967 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420800bd90 con 0x7f421815dad0 2024-04-02T09:12:31.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.672+0000 7f421e000700 1 -- 172.21.15.89:0/698500967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f421815dad0 msgr2=0x7f42180ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.672+0000 7f421e000700 1 --2- 172.21.15.89:0/698500967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f421815dad0 0x7f42180ced80 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f42080098f0 tx=0x7f4208009c00 comp rx=0 tx=0).stop 2024-04-02T09:12:31.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.672+0000 7f421e000700 1 -- 172.21.15.89:0/698500967 shutdown_connections 2024-04-02T09:12:31.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.672+0000 7f421e000700 1 --2- 172.21.15.89:0/698500967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f421815dad0 0x7f42180ced80 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.672+0000 7f421e000700 1 --2- 172.21.15.89:0/698500967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42180d1630 0x7f42180d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.672+0000 7f421e000700 1 -- 172.21.15.89:0/698500967 >> 172.21.15.89:0/698500967 conn(0x7f421808aff0 msgr2=0x7f42180d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:31.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.672+0000 7f421e000700 1 -- 172.21.15.89:0/698500967 shutdown_connections 2024-04-02T09:12:31.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.672+0000 7f421e000700 1 -- 172.21.15.89:0/698500967 wait complete. 2024-04-02T09:12:31.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.674+0000 7f421e000700 1 Processor -- start 2024-04-02T09:12:31.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.674+0000 7f421e000700 1 -- start start 2024-04-02T09:12:31.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.675+0000 7f421e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42180d1630 0x7f421813ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.675+0000 7f421e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42180d1fd0 0x7f421814fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.675+0000 7f421e000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f421815dad0 0x7f421815e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.675+0000 7f421e000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421813f110 con 0x7f42180d1fd0 2024-04-02T09:12:31.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.675+0000 7f421e000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4218150b40 con 0x7f421815dad0 2024-04-02T09:12:31.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.675+0000 7f421e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42180c8d50 con 0x7f42180d1630 2024-04-02T09:12:31.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.675+0000 7f420fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42180d1fd0 0x7f421814fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.675+0000 7f420fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42180d1fd0 0x7f421814fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52476/0 (socket says 172.21.15.89:52476) 2024-04-02T09:12:31.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.675+0000 7f420fe00700 1 -- 172.21.15.89:0/1529795690 learned_addr learned my addr 172.21.15.89:0/1529795690 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:31.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.675+0000 7f4217400700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42180d1630 0x7f421813ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.675+0000 7f4217e00700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f421815dad0 0x7f421815e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.676+0000 7f420fe00700 1 -- 172.21.15.89:0/1529795690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42180d1630 msgr2=0x7f421813ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.676+0000 7f420fe00700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42180d1630 0x7f421813ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.676+0000 7f420fe00700 1 -- 172.21.15.89:0/1529795690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f421815dad0 msgr2=0x7f421815e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.676+0000 7f420fe00700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f421815dad0 0x7f421815e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.676+0000 7f420fe00700 1 -- 172.21.15.89:0/1529795690 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4208009550 con 0x7f42180d1fd0 2024-04-02T09:12:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.676+0000 7f4217e00700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f421815dad0 0x7f421815e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.676+0000 7f420fe00700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42180d1fd0 0x7f421814fae0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f420000b530 tx=0x7f420000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.676+0000 7f420f400700 1 -- 172.21.15.89:0/1529795690 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4200011830 con 0x7f42180d1fd0 2024-04-02T09:12:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.676+0000 7f420f400700 1 -- 172.21.15.89:0/1529795690 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4200011990 con 0x7f42180d1fd0 2024-04-02T09:12:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.676+0000 7f421e000700 1 -- 172.21.15.89:0/1529795690 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42180c7050 con 0x7f42180d1fd0 2024-04-02T09:12:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.676+0000 7f420f400700 1 -- 172.21.15.89:0/1529795690 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4200011d20 con 0x7f42180d1fd0 2024-04-02T09:12:31.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.676+0000 7f421e000700 1 -- 172.21.15.89:0/1529795690 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42180c8ff0 con 0x7f42180d1fd0 2024-04-02T09:12:31.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.678+0000 7f420f400700 1 -- 172.21.15.89:0/1529795690 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f420001a400 con 0x7f42180d1fd0 2024-04-02T09:12:31.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.679+0000 7f420f400700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f41f406adb0 0x7f41f406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.679+0000 7f420f400700 1 -- 172.21.15.89:0/1529795690 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f4200089730 con 0x7f42180d1fd0 2024-04-02T09:12:31.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.679+0000 7f4217400700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f41f406adb0 0x7f41f406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.680+0000 7f4217400700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f41f406adb0 0x7f41f406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4208014040 tx=0x7f4208008030 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.681+0000 7f4231a7bb80 1 Processor -- start 2024-04-02T09:12:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.681+0000 7f4231a7bb80 1 -- start start 2024-04-02T09:12:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.682+0000 7f4231a7bb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc10f830 0x55febc10fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.682+0000 7f4231a7bb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 0x55febc0e5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.682+0000 7f4231a7bb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 0x55febc10ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.682+0000 7f4231a7bb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55febc05a650 con 0x55febc10f830 2024-04-02T09:12:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.682+0000 7f4231a7bb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55febc03bc30 con 0x55febc0e0b60 2024-04-02T09:12:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.682+0000 7f4231a7bb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55febc039430 con 0x55febc0e2be0 2024-04-02T09:12:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.682+0000 7f4217400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc10f830 0x55febc10fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.682+0000 7f4217400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc10f830 0x55febc10fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52482/0 (socket says 172.21.15.89:52482) 2024-04-02T09:12:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.682+0000 7f4217400700 1 -- 172.21.15.89:0/4011470723 learned_addr learned my addr 172.21.15.89:0/4011470723 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.683+0000 7f4217400700 1 -- 172.21.15.89:0/4011470723 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 msgr2=0x55febc0e5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.683+0000 7f4217400700 1 --2- 172.21.15.89:0/4011470723 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 0x55febc0e5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.683+0000 7f4217400700 1 -- 172.21.15.89:0/4011470723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 msgr2=0x55febc10ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.683+0000 7f4217400700 1 --2- 172.21.15.89:0/4011470723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 0x55febc10ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.683+0000 7f4217400700 1 -- 172.21.15.89:0/4011470723 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55febbe28790 con 0x55febc10f830 2024-04-02T09:12:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.683+0000 7f4217400700 1 --2- 172.21.15.89:0/4011470723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc10f830 0x55febc10fc10 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f421808ce40 tx=0x7f4208010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02e64f88b19181 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.683+0000 7f4207400700 1 -- 172.21.15.89:0/4011470723 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420801f070 con 0x55febc10f830 2024-04-02T09:12:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.683+0000 7f4207400700 1 -- 172.21.15.89:0/4011470723 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4208003760 con 0x55febc10f830 2024-04-02T09:12:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.683+0000 7f4207400700 1 -- 172.21.15.89:0/4011470723 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208003ad0 con 0x55febc10f830 2024-04-02T09:12:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.684+0000 7f4231a7bb80 1 -- 172.21.15.89:0/4011470723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc10f830 msgr2=0x55febc10fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.684+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/4011470723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc10f830 0x55febc10fc10 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f421808ce40 tx=0x7f4208010000 comp rx=0 tx=0).stop 2024-04-02T09:12:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.684+0000 7f4231a7bb80 1 -- 172.21.15.89:0/4011470723 shutdown_connections 2024-04-02T09:12:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.684+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/4011470723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 0x55febc10ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.684+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/4011470723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc10f830 0x55febc10fc10 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.684+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/4011470723 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 0x55febc0e5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.684+0000 7f4231a7bb80 1 -- 172.21.15.89:0/4011470723 >> 172.21.15.89:0/4011470723 conn(0x55febc0436a0 msgr2=0x55febc12a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.684+0000 7f4231a7bb80 1 -- 172.21.15.89:0/4011470723 shutdown_connections 2024-04-02T09:12:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.684+0000 7f4231a7bb80 1 -- 172.21.15.89:0/4011470723 wait complete. 2024-04-02T09:12:31.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.685+0000 7f4231a7bb80 1 -- start start 2024-04-02T09:12:31.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.685+0000 7f4231a7bb80 1 Processor -- start 2024-04-02T09:12:31.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.686+0000 7f4231a7bb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 0x55febc13d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.686+0000 7f4231a7bb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc0e2be0 0x55febc12bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.686+0000 7f4231a7bb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc10f830 0x55febc111c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.686+0000 7f4231a7bb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55febc034d20 con 0x55febc0e2be0 2024-04-02T09:12:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.686+0000 7f4231a7bb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55febbfd0770 con 0x55febc0e0b60 2024-04-02T09:12:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.686+0000 7f4231a7bb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55febbff36f0 con 0x55febc10f830 2024-04-02T09:12:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.686+0000 7f4231a7bb80 10 client.0 _ll_register_callbacks cb 0x55febbfda5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T09:12:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.686+0000 7f420fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc0e2be0 0x55febc12bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.686+0000 7f420fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc0e2be0 0x55febc12bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52496/0 (socket says 172.21.15.89:52496) 2024-04-02T09:12:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.686+0000 7f420fe00700 1 -- 172.21.15.89:0/2278477048 learned_addr learned my addr 172.21.15.89:0/2278477048 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.686+0000 7f4217400700 1 --2- 172.21.15.89:0/2278477048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 0x55febc13d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.686+0000 7f4217e00700 1 --2- 172.21.15.89:0/2278477048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc10f830 0x55febc111c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f420fe00700 1 -- 172.21.15.89:0/2278477048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc10f830 msgr2=0x55febc111c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f420fe00700 1 --2- 172.21.15.89:0/2278477048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc10f830 0x55febc111c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f420fe00700 1 -- 172.21.15.89:0/2278477048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 msgr2=0x55febc13d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f420fe00700 1 --2- 172.21.15.89:0/2278477048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 0x55febc13d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f420fe00700 1 -- 172.21.15.89:0/2278477048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42180d2460 con 0x55febc0e2be0 2024-04-02T09:12:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f4217e00700 1 --2- 172.21.15.89:0/2278477048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc10f830 0x55febc111c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f420fe00700 1 --2- 172.21.15.89:0/2278477048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc0e2be0 0x55febc12bac0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f4200058000 tx=0x7f4200058300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f774863aa862ce05 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f41fbe00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T09:12:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f41fbe00700 1 -- 172.21.15.89:0/2278477048 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4200058760 con 0x55febc0e2be0 2024-04-02T09:12:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f41fbe00700 1 -- 172.21.15.89:0/2278477048 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f42000174d0 con 0x55febc0e2be0 2024-04-02T09:12:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f4231a7bb80 1 -- 172.21.15.89:0/2278477048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55febc0d7920 con 0x55febc0e2be0 2024-04-02T09:12:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f41fbe00700 1 -- 172.21.15.89:0/2278477048 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42000606f0 con 0x55febc0e2be0 2024-04-02T09:12:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f4231a7bb80 10 client.11292 fetch_fsmap learned FSMap version 105 2024-04-02T09:12:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f4231a7bb80 1 -- 172.21.15.89:0/2278477048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) v3 -- 0x55febbf150a0 con 0x55febc0e2be0 2024-04-02T09:12:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.687+0000 7f41fbe00700 1 -- 172.21.15.89:0/2278477048 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55febc0d7920 con 0x55febc0e2be0 2024-04-02T09:12:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.688+0000 7f41fbe00700 1 -- 172.21.15.89:0/2278477048 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 105) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f42000577f0 con 0x55febc0e2be0 2024-04-02T09:12:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.688+0000 7f4231a7bb80 10 client.11292 fetch_fsmap finished waiting for FSMap version 105 2024-04-02T09:12:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.688+0000 7f4231a7bb80 10 client.11292 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T09:12:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.688+0000 7f4231a7bb80 20 client.11292 populate_metadata read hostname 'smithi089' 2024-04-02T09:12:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.688+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/2278477048 >> [v2:172.21.15.89:6835/2615927215,v1:172.21.15.89:6837/2615927215] conn(0x55febc12df70 0x55febc1398c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.688+0000 7f4231a7bb80 4 client.11292 mds_command: new command op to 11157 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T09:12:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.688+0000 7f4231a7bb80 1 -- 172.21.15.89:0/2278477048 --> [v2:172.21.15.89:6835/2615927215,v1:172.21.15.89:6837/2615927215] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55febc0b46f0 con 0x55febc12df70 2024-04-02T09:12:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.688+0000 7f4217400700 1 --2- 172.21.15.89:0/2278477048 >> [v2:172.21.15.89:6835/2615927215,v1:172.21.15.89:6837/2615927215] conn(0x55febc12df70 0x55febc1398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.688+0000 7f4217400700 1 --2- 172.21.15.89:0/2278477048 >> [v2:172.21.15.89:6835/2615927215,v1:172.21.15.89:6837/2615927215] conn(0x55febc12df70 0x55febc1398c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=535570f44f812041 server_cookie=48dc4eddcc0bb79 in_seq=0 out_seq=0 2024-04-02T09:12:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.688+0000 7f41fbe00700 10 client.11292 ms_handle_connect on v2:172.21.15.89:6835/2615927215 2024-04-02T09:12:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.689+0000 7f41fbe00700 1 -- 172.21.15.89:0/2278477048 <== mds.0 v2:172.21.15.89:6835/2615927215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55febc0b46f0 con 0x55febc12df70 2024-04-02T09:12:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.689+0000 7f41fbe00700 10 client.11292 handle_command_reply: tid=0 2024-04-02T09:12:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.689+0000 7f4231a7bb80 1 client.11292 shutdown 2024-04-02T09:12:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.689+0000 7f4231a7bb80 1 -- 172.21.15.89:0/2278477048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc0e2be0 msgr2=0x55febc12bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.689+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/2278477048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc0e2be0 0x55febc12bac0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f4200058000 tx=0x7f4200058300 comp rx=0 tx=0).stop 2024-04-02T09:12:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.689+0000 7f4231a7bb80 1 -- 172.21.15.89:0/2278477048 shutdown_connections 2024-04-02T09:12:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.689+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/2278477048 >> [v2:172.21.15.89:6835/2615927215,v1:172.21.15.89:6837/2615927215] conn(0x55febc12df70 0x55febc1398c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.689+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/2278477048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 0x55febc13d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.689+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/2278477048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc10f830 0x55febc111c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.689+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/2278477048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc0e2be0 0x55febc12bac0 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.689+0000 7f41fbe00700 0 client.11292 ms_handle_reset on v2:172.21.15.89:6835/2615927215 2024-04-02T09:12:31.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.689+0000 7f4231a7bb80 1 -- 172.21.15.89:0/2278477048 >> 172.21.15.89:0/2278477048 conn(0x55febc0436a0 msgr2=0x55febc0e4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:31.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.690+0000 7f4231a7bb80 1 -- 172.21.15.89:0/2278477048 shutdown_connections 2024-04-02T09:12:31.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.690+0000 7f4231a7bb80 1 -- 172.21.15.89:0/2278477048 wait complete. 2024-04-02T09:12:31.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.690+0000 7f4231a7bb80 20 client.11292 trim_cache size 0 max 16384 2024-04-02T09:12:31.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.708+0000 7f4231a7bb80 1 Processor -- start 2024-04-02T09:12:31.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.708+0000 7f4231a7bb80 1 -- start start 2024-04-02T09:12:31.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4231a7bb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc0e0b60 0x55febc129620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4231a7bb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 0x55febc0e4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4231a7bb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc10f830 0x55febc0e4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4231a7bb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55febbe29390 con 0x55febc0e0b60 2024-04-02T09:12:31.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4231a7bb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55febc113390 con 0x55febc10f830 2024-04-02T09:12:31.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4231a7bb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55febc12e490 con 0x55febc0e2be0 2024-04-02T09:12:31.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4217400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc0e0b60 0x55febc129620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4217400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc0e0b60 0x55febc129620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52502/0 (socket says 172.21.15.89:52502) 2024-04-02T09:12:31.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4217400700 1 -- 172.21.15.89:0/3032193151 learned_addr learned my addr 172.21.15.89:0/3032193151 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:31.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4217e00700 1 --2- 172.21.15.89:0/3032193151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc10f830 0x55febc0e4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f420fe00700 1 --2- 172.21.15.89:0/3032193151 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 0x55febc0e4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4217400700 1 -- 172.21.15.89:0/3032193151 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 msgr2=0x55febc0e4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:31.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4217400700 1 --2- 172.21.15.89:0/3032193151 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 0x55febc0e4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4217400700 1 -- 172.21.15.89:0/3032193151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc10f830 msgr2=0x55febc0e4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:31.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4217400700 1 --2- 172.21.15.89:0/3032193151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc10f830 0x55febc0e4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4217400700 1 -- 172.21.15.89:0/3032193151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55febbffb010 con 0x55febc0e0b60 2024-04-02T09:12:31.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f420fe00700 1 --2- 172.21.15.89:0/3032193151 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 0x55febc0e4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:31.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.709+0000 7f4217e00700 1 --2- 172.21.15.89:0/3032193151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc10f830 0x55febc0e4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:31.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.710+0000 7f4217400700 1 --2- 172.21.15.89:0/3032193151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc0e0b60 0x55febc129620 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x55febc121690 tx=0x7f420801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf850b4bd17125c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.710+0000 7f4207e00700 1 -- 172.21.15.89:0/3032193151 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208003d70 con 0x55febc0e0b60 2024-04-02T09:12:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.710+0000 7f4207e00700 1 -- 172.21.15.89:0/3032193151 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4208003ed0 con 0x55febc0e0b60 2024-04-02T09:12:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.710+0000 7f4207e00700 1 -- 172.21.15.89:0/3032193151 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420803bc90 con 0x55febc0e0b60 2024-04-02T09:12:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.710+0000 7f4231a7bb80 1 -- 172.21.15.89:0/3032193151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc0e0b60 msgr2=0x55febc129620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.710+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/3032193151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc0e0b60 0x55febc129620 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x55febc121690 tx=0x7f420801f000 comp rx=0 tx=0).stop 2024-04-02T09:12:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.710+0000 7f4231a7bb80 1 -- 172.21.15.89:0/3032193151 shutdown_connections 2024-04-02T09:12:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.710+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/3032193151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc10f830 0x55febc0e4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.710+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/3032193151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc0e0b60 0x55febc129620 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.710+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/3032193151 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 0x55febc0e4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.710+0000 7f4231a7bb80 1 -- 172.21.15.89:0/3032193151 >> 172.21.15.89:0/3032193151 conn(0x55febc0436a0 msgr2=0x55febc10ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.711+0000 7f4231a7bb80 1 -- 172.21.15.89:0/3032193151 shutdown_connections 2024-04-02T09:12:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.711+0000 7f4231a7bb80 1 -- 172.21.15.89:0/3032193151 wait complete. 2024-04-02T09:12:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.711+0000 7f4231a7bb80 1 -- start start 2024-04-02T09:12:31.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.711+0000 7f4231a7bb80 1 Processor -- start 2024-04-02T09:12:31.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.712+0000 7f4231a7bb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 0x55febc0e4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.712+0000 7f4231a7bb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 0x55febc0de5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.712+0000 7f4231a7bb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc10f830 0x55febc0deae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.712+0000 7f4231a7bb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55febc13b5e0 con 0x55febc10f830 2024-04-02T09:12:31.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.712+0000 7f4231a7bb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55febc0dfa10 con 0x55febc0e0b60 2024-04-02T09:12:31.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.712+0000 7f4231a7bb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55febc0e5090 con 0x55febc0e2be0 2024-04-02T09:12:31.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.712+0000 7f4231a7bb80 10 client.0 _ll_register_callbacks cb 0x55febbfda5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T09:12:31.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.713+0000 7f4217e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc10f830 0x55febc0deae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.713+0000 7f4217e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc10f830 0x55febc0deae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52506/0 (socket says 172.21.15.89:52506) 2024-04-02T09:12:31.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.713+0000 7f4217e00700 1 -- 172.21.15.89:0/628860527 learned_addr learned my addr 172.21.15.89:0/628860527 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:31.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.713+0000 7f420fe00700 1 --2- 172.21.15.89:0/628860527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 0x55febc0de5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.713+0000 7f4217400700 1 --2- 172.21.15.89:0/628860527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 0x55febc0e4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.713+0000 7f4217e00700 1 -- 172.21.15.89:0/628860527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 msgr2=0x55febc0de5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:31.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.713+0000 7f4217e00700 1 --2- 172.21.15.89:0/628860527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 0x55febc0de5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.713+0000 7f4217e00700 1 -- 172.21.15.89:0/628860527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 msgr2=0x55febc0e4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:31.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.713+0000 7f4217e00700 1 --2- 172.21.15.89:0/628860527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 0x55febc0e4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.713+0000 7f4217e00700 1 -- 172.21.15.89:0/628860527 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55febbe28790 con 0x55febc10f830 2024-04-02T09:12:31.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.713+0000 7f4217400700 1 --2- 172.21.15.89:0/628860527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 0x55febc0e4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:31.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.713+0000 7f4217e00700 1 --2- 172.21.15.89:0/628860527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc10f830 0x55febc0deae0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x55febc112bd0 tx=0x7f42100053e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81daf22e933a1daf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:31.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.713+0000 7f41fbe00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T09:12:31.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.713+0000 7f41fbe00700 1 -- 172.21.15.89:0/628860527 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4210015610 con 0x55febc10f830 2024-04-02T09:12:31.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.714+0000 7f41fbe00700 1 -- 172.21.15.89:0/628860527 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f4210015770 con 0x55febc10f830 2024-04-02T09:12:31.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.714+0000 7f4231a7bb80 1 -- 172.21.15.89:0/628860527 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f42180c96a0 con 0x55febc10f830 2024-04-02T09:12:31.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.714+0000 7f41fbe00700 1 -- 172.21.15.89:0/628860527 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4210015930 con 0x55febc10f830 2024-04-02T09:12:31.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.714+0000 7f4231a7bb80 10 client.11298 fetch_fsmap learned FSMap version 105 2024-04-02T09:12:31.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.714+0000 7f4231a7bb80 1 -- 172.21.15.89:0/628860527 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) v3 -- 0x55febbe891d0 con 0x55febc10f830 2024-04-02T09:12:31.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.714+0000 7f41fbe00700 1 -- 172.21.15.89:0/628860527 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42180c96a0 con 0x55febc10f830 2024-04-02T09:12:31.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.714+0000 7f41fbe00700 1 -- 172.21.15.89:0/628860527 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 105) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4210015a90 con 0x55febc10f830 2024-04-02T09:12:31.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.714+0000 7f4231a7bb80 10 client.11298 fetch_fsmap finished waiting for FSMap version 105 2024-04-02T09:12:31.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.714+0000 7f4231a7bb80 10 client.11298 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T09:12:31.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.714+0000 7f4231a7bb80 20 client.11298 populate_metadata read hostname 'smithi089' 2024-04-02T09:12:31.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.714+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/628860527 >> [v2:172.21.15.89:6835/2615927215,v1:172.21.15.89:6837/2615927215] conn(0x55febc120c60 0x55febc121040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.714+0000 7f4231a7bb80 4 client.11298 mds_command: new command op to 11157 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T09:12:31.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.714+0000 7f4231a7bb80 1 -- 172.21.15.89:0/628860527 --> [v2:172.21.15.89:6835/2615927215,v1:172.21.15.89:6837/2615927215] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f4208003760 con 0x55febc120c60 2024-04-02T09:12:31.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.714+0000 7f4217400700 1 --2- 172.21.15.89:0/628860527 >> [v2:172.21.15.89:6835/2615927215,v1:172.21.15.89:6837/2615927215] conn(0x55febc120c60 0x55febc121040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.715+0000 7f4217400700 1 --2- 172.21.15.89:0/628860527 >> [v2:172.21.15.89:6835/2615927215,v1:172.21.15.89:6837/2615927215] conn(0x55febc120c60 0x55febc121040 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b10ce2ccc5e355 server_cookie=e60eb23f05798154 in_seq=0 out_seq=0 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.715+0000 7f41fbe00700 10 client.11298 ms_handle_connect on v2:172.21.15.89:6835/2615927215 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.715+0000 7f41fbe00700 1 -- 172.21.15.89:0/628860527 <== mds.0 v2:172.21.15.89:6835/2615927215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x7f4208003760 con 0x55febc120c60 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.715+0000 7f41fbe00700 10 client.11298 handle_command_reply: tid=0 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.715+0000 7f4231a7bb80 1 client.11298 shutdown 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.715+0000 7f4231a7bb80 1 -- 172.21.15.89:0/628860527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc10f830 msgr2=0x55febc0deae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.715+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/628860527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc10f830 0x55febc0deae0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x55febc112bd0 tx=0x7f42100053e0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.715+0000 7f4231a7bb80 1 -- 172.21.15.89:0/628860527 shutdown_connections 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.715+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/628860527 >> [v2:172.21.15.89:6835/2615927215,v1:172.21.15.89:6837/2615927215] conn(0x55febc120c60 0x55febc121040 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.715+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/628860527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55febc0e0b60 0x55febc0e4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.715+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/628860527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55febc10f830 0x55febc0deae0 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.715+0000 7f4231a7bb80 1 --2- 172.21.15.89:0/628860527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55febc0e2be0 0x55febc0de5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.715+0000 7f41fbe00700 0 client.11298 ms_handle_reset on v2:172.21.15.89:6835/2615927215 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.715+0000 7f4231a7bb80 1 -- 172.21.15.89:0/628860527 >> 172.21.15.89:0/628860527 conn(0x55febc0436a0 msgr2=0x55febc1216c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:31.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.716+0000 7f4231a7bb80 1 -- 172.21.15.89:0/628860527 shutdown_connections 2024-04-02T09:12:31.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.716+0000 7f4231a7bb80 1 -- 172.21.15.89:0/628860527 wait complete. 2024-04-02T09:12:31.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.716+0000 7f4231a7bb80 20 client.11298 trim_cache size 0 max 16384 2024-04-02T09:12:31.720 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:12:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T09:12:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T09:12:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 11157, 2024-04-02T09:12:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T09:12:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T09:12:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T09:12:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 6.5423331999999998, 2024-04-02T09:12:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 105, 2024-04-02T09:12:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 223, 2024-04-02T09:12:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 223, 2024-04-02T09:12:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 11.995003690000001 2024-04-02T09:12:31.721 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:12:31.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.716+0000 7f421e000700 1 -- 172.21.15.89:0/1529795690 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f41f406adb0 msgr2=0x7f41f406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.716+0000 7f421e000700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f41f406adb0 0x7f41f406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4208014040 tx=0x7f4208008030 comp rx=0 tx=0).stop 2024-04-02T09:12:31.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.716+0000 7f421e000700 1 -- 172.21.15.89:0/1529795690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42180d1fd0 msgr2=0x7f421814fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.716+0000 7f421e000700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42180d1fd0 0x7f421814fae0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f420000b530 tx=0x7f420000b840 comp rx=0 tx=0).stop 2024-04-02T09:12:31.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.716+0000 7f421e000700 1 -- 172.21.15.89:0/1529795690 shutdown_connections 2024-04-02T09:12:31.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.716+0000 7f421e000700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f41f406adb0 0x7f41f406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.716+0000 7f421e000700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f421815dad0 0x7f421815e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.716+0000 7f421e000700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f42180d1630 0x7f421813ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.716+0000 7f421e000700 1 --2- 172.21.15.89:0/1529795690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42180d1fd0 0x7f421814fae0 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.716+0000 7f421e000700 1 -- 172.21.15.89:0/1529795690 >> 172.21.15.89:0/1529795690 conn(0x7f421808aff0 msgr2=0x7f421815b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:31.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.716+0000 7f421e000700 1 -- 172.21.15.89:0/1529795690 shutdown_connections 2024-04-02T09:12:31.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.717+0000 7f421e000700 1 -- 172.21.15.89:0/1529795690 wait complete. 2024-04-02T09:12:31.732 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T09:12:31.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.926+0000 7f3dff200700 1 Processor -- start 2024-04-02T09:12:31.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.926+0000 7f3dff200700 1 -- start start 2024-04-02T09:12:31.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.928+0000 7f3dff200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3df80d2870 0x7f3df80d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.928+0000 7f3dff200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df80d2dc0 con 0x7f3df80d0a00 2024-04-02T09:12:31.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.928+0000 7f3dff200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df80d30a0 con 0x7f3df815d5d0 2024-04-02T09:12:31.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.928+0000 7f3dff200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df815d9b0 con 0x7f3df80d2870 2024-04-02T09:12:31.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.928+0000 7f3dfd400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3df80d0a00 0x7f3df80d0f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52754/0 (socket says 172.21.15.89:52754) 2024-04-02T09:12:31.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.928+0000 7f3dfd400700 1 -- 172.21.15.89:0/3592957725 learned_addr learned my addr 172.21.15.89:0/3592957725 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:31.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.928+0000 7f3dfca00700 1 -- 172.21.15.89:0/3592957725 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389074101 0 0) 0x7f3df80d2dc0 con 0x7f3df80d0a00 2024-04-02T09:12:31.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.928+0000 7f3dfca00700 1 -- 172.21.15.89:0/3592957725 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de00035b0 con 0x7f3df80d0a00 2024-04-02T09:12:31.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.928+0000 7f3dfe800700 1 --2- 172.21.15.89:0/3592957725 >> v2:172.21.15.113:3300/0 conn(0x7f3df80d2870 0x7f3df80d3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfca00700 1 -- 172.21.15.89:0/3592957725 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 909435759 0 0) 0x7f3de00035b0 con 0x7f3df80d0a00 2024-04-02T09:12:31.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfca00700 1 -- 172.21.15.89:0/3592957725 >> v2:172.21.15.113:3300/0 conn(0x7f3df80d2870 msgr2=0x7f3df80d3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:31.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfca00700 1 --2- 172.21.15.89:0/3592957725 >> v2:172.21.15.113:3300/0 conn(0x7f3df80d2870 0x7f3df80d3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfca00700 1 -- 172.21.15.89:0/3592957725 >> v1:172.21.15.113:6789/0 conn(0x7f3df815d5d0 legacy=0x7f3df80d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfca00700 1 -- 172.21.15.89:0/3592957725 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df80d3380 con 0x7f3df80d0a00 2024-04-02T09:12:31.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfca00700 1 -- 172.21.15.89:0/3592957725 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f3de8001670 con 0x7f3df80d0a00 2024-04-02T09:12:31.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfca00700 1 -- 172.21.15.89:0/3592957725 >> v1:172.21.15.89:6789/0 conn(0x7f3df80d0a00 legacy=0x7f3df80d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfca00700 1 --2- 172.21.15.89:0/3592957725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3de0003a50 0x7f3de0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfca00700 1 --2- 172.21.15.89:0/3592957725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3de0004850 0x7f3de0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfca00700 1 --2- 172.21.15.89:0/3592957725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3de0005210 0x7f3de000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfca00700 1 -- 172.21.15.89:0/3592957725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df815d9b0 con 0x7f3de0004850 2024-04-02T09:12:31.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfca00700 1 -- 172.21.15.89:0/3592957725 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de000c110 con 0x7f3de0005210 2024-04-02T09:12:31.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfca00700 1 -- 172.21.15.89:0/3592957725 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de000c2d0 con 0x7f3de0003a50 2024-04-02T09:12:31.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfd400700 1 --2- 172.21.15.89:0/3592957725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3de0004850 0x7f3de0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfe800700 1 --2- 172.21.15.89:0/3592957725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3de0005210 0x7f3de000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfde00700 1 --2- 172.21.15.89:0/3592957725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3de0003a50 0x7f3de0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfd400700 1 -- 172.21.15.89:0/3592957725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3de0003a50 msgr2=0x7f3de0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:31.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfd400700 1 --2- 172.21.15.89:0/3592957725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3de0003a50 0x7f3de0003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfd400700 1 -- 172.21.15.89:0/3592957725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3de0005210 msgr2=0x7f3de000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:31.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfd400700 1 --2- 172.21.15.89:0/3592957725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3de0005210 0x7f3de000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfd400700 1 -- 172.21.15.89:0/3592957725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de00035b0 con 0x7f3de0004850 2024-04-02T09:12:31.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfe800700 1 -- 172.21.15.89:0/3592957725 reap_dead start 2024-04-02T09:12:31.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.929+0000 7f3dfde00700 1 --2- 172.21.15.89:0/3592957725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3de0003a50 0x7f3de0003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:31.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.930+0000 7f3dfd400700 1 --2- 172.21.15.89:0/3592957725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3de0004850 0x7f3de0004cd0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f3de800c160 tx=0x7f3de0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb798499e0252c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:31.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.930+0000 7f3dfca00700 1 -- 172.21.15.89:0/3592957725 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8007b70 con 0x7f3de0004850 2024-04-02T09:12:31.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.930+0000 7f3dfca00700 1 -- 172.21.15.89:0/3592957725 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de800cca0 con 0x7f3de0004850 2024-04-02T09:12:31.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.930+0000 7f3dfca00700 1 -- 172.21.15.89:0/3592957725 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8005ba0 con 0x7f3de0004850 2024-04-02T09:12:31.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.930+0000 7f3dff200700 1 -- 172.21.15.89:0/3592957725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3de0004850 msgr2=0x7f3de0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.930+0000 7f3dff200700 1 --2- 172.21.15.89:0/3592957725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3de0004850 0x7f3de0004cd0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f3de800c160 tx=0x7f3de0005880 comp rx=0 tx=0).stop 2024-04-02T09:12:31.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.930+0000 7f3dff200700 1 -- 172.21.15.89:0/3592957725 shutdown_connections 2024-04-02T09:12:31.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.930+0000 7f3dff200700 1 --2- 172.21.15.89:0/3592957725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3de0004850 0x7f3de0004cd0 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.930+0000 7f3dff200700 1 -- 172.21.15.89:0/3592957725 >> 172.21.15.89:0/3592957725 conn(0x7f3df80668e0 msgr2=0x7f3df8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:31.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.931+0000 7f3dff200700 1 -- 172.21.15.89:0/3592957725 shutdown_connections 2024-04-02T09:12:31.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.931+0000 7f3dff200700 1 -- 172.21.15.89:0/3592957725 wait complete. 2024-04-02T09:12:31.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.933+0000 7f3dff200700 1 Processor -- start 2024-04-02T09:12:31.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.933+0000 7f3dff200700 1 -- start start 2024-04-02T09:12:31.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.934+0000 7f3dff200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3de0004850 0x7f3df8150a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.934+0000 7f3dff200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3df80b63a0 0x7f3df80b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.934+0000 7f3dff200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3df80c9220 0x7f3df80c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.934+0000 7f3dff200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df81f7090 con 0x7f3de0004850 2024-04-02T09:12:31.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.934+0000 7f3dff200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df80b6d60 con 0x7f3df80b63a0 2024-04-02T09:12:31.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.934+0000 7f3dff200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df81f6410 con 0x7f3df80c9220 2024-04-02T09:12:31.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.934+0000 7f3dfde00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3de0004850 0x7f3df8150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.935+0000 7f3dfde00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3de0004850 0x7f3df8150a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52518/0 (socket says 172.21.15.89:52518) 2024-04-02T09:12:31.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.935+0000 7f3dfde00700 1 -- 172.21.15.89:0/2501695240 learned_addr learned my addr 172.21.15.89:0/2501695240 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:31.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.935+0000 7f3dfe800700 1 --2- 172.21.15.89:0/2501695240 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3df80c9220 0x7f3df80c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.935+0000 7f3dfd400700 1 --2- 172.21.15.89:0/2501695240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3df80b63a0 0x7f3df80b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.935+0000 7f3dfde00700 1 -- 172.21.15.89:0/2501695240 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3df80c9220 msgr2=0x7f3df80c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.935+0000 7f3dfde00700 1 --2- 172.21.15.89:0/2501695240 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3df80c9220 0x7f3df80c96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.935+0000 7f3dfde00700 1 -- 172.21.15.89:0/2501695240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3df80b63a0 msgr2=0x7f3df80b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:31.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.935+0000 7f3dfde00700 1 --2- 172.21.15.89:0/2501695240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3df80b63a0 0x7f3df80b6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:31.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.935+0000 7f3dfde00700 1 -- 172.21.15.89:0/2501695240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df81f66f0 con 0x7f3de0004850 2024-04-02T09:12:31.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.935+0000 7f3dfde00700 1 --2- 172.21.15.89:0/2501695240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3de0004850 0x7f3df8150a50 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00e550 tx=0x7f3dec00e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:31.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.935+0000 7f3df2000700 1 -- 172.21.15.89:0/2501695240 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec0052e0 con 0x7f3de0004850 2024-04-02T09:12:31.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.935+0000 7f3df2000700 1 -- 172.21.15.89:0/2501695240 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dec005460 con 0x7f3de0004850 2024-04-02T09:12:31.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.935+0000 7f3df2000700 1 -- 172.21.15.89:0/2501695240 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec0038c0 con 0x7f3de0004850 2024-04-02T09:12:31.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.935+0000 7f3dff200700 1 -- 172.21.15.89:0/2501695240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df81f69d0 con 0x7f3de0004850 2024-04-02T09:12:31.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.936+0000 7f3dff200700 1 -- 172.21.15.89:0/2501695240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df8150f90 con 0x7f3de0004850 2024-04-02T09:12:31.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.937+0000 7f3dff200700 1 -- 172.21.15.89:0/2501695240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df80b2860 con 0x7f3de0004850 2024-04-02T09:12:31.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.937+0000 7f3df2000700 1 -- 172.21.15.89:0/2501695240 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3dec003a20 con 0x7f3de0004850 2024-04-02T09:12:31.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.938+0000 7f3df2000700 1 --2- 172.21.15.89:0/2501695240 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ddc06adb0 0x7f3ddc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:31.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.938+0000 7f3df2000700 1 -- 172.21.15.89:0/2501695240 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f3dec088c70 con 0x7f3de0004850 2024-04-02T09:12:31.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.939+0000 7f3dfd400700 1 --2- 172.21.15.89:0/2501695240 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ddc06adb0 0x7f3ddc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:31.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.939+0000 7f3dfd400700 1 --2- 172.21.15.89:0/2501695240 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ddc06adb0 0x7f3ddc06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3de8000c00 tx=0x7f3de8007000 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:31.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:31.943+0000 7f3df2000700 1 -- 172.21.15.89:0/2501695240 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3dec053c30 con 0x7f3de0004850 2024-04-02T09:12:32.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:32.190+0000 7f3dff200700 1 -- 172.21.15.89:0/2501695240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3df804ea80 con 0x7f3de0004850 2024-04-02T09:12:32.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:32.190+0000 7f3df2000700 1 -- 172.21.15.89:0/2501695240 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v105) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3dec019020 con 0x7f3de0004850 2024-04-02T09:12:32.189 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T09:12:32.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:32.192+0000 7f3dff200700 1 -- 172.21.15.89:0/2501695240 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ddc06adb0 msgr2=0x7f3ddc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:32.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:32.192+0000 7f3dff200700 1 --2- 172.21.15.89:0/2501695240 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ddc06adb0 0x7f3ddc06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3de8000c00 tx=0x7f3de8007000 comp rx=0 tx=0).stop 2024-04-02T09:12:32.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:32.192+0000 7f3dff200700 1 -- 172.21.15.89:0/2501695240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3de0004850 msgr2=0x7f3df8150a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:32.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:32.192+0000 7f3dff200700 1 --2- 172.21.15.89:0/2501695240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3de0004850 0x7f3df8150a50 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00e550 tx=0x7f3dec00e910 comp rx=0 tx=0).stop 2024-04-02T09:12:32.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:32.192+0000 7f3dff200700 1 -- 172.21.15.89:0/2501695240 shutdown_connections 2024-04-02T09:12:32.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:32.192+0000 7f3dff200700 1 --2- 172.21.15.89:0/2501695240 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3df80c9220 0x7f3df80c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:32.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:32.192+0000 7f3dff200700 1 --2- 172.21.15.89:0/2501695240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3de0004850 0x7f3df8150a50 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:32.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:32.192+0000 7f3dff200700 1 --2- 172.21.15.89:0/2501695240 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ddc06adb0 0x7f3ddc06d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:32.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:32.192+0000 7f3dff200700 1 --2- 172.21.15.89:0/2501695240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3df80b63a0 0x7f3df80b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:32.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:32.192+0000 7f3dff200700 1 -- 172.21.15.89:0/2501695240 >> 172.21.15.89:0/2501695240 conn(0x7f3df80668e0 msgr2=0x7f3df8066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:32.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:32.193+0000 7f3dff200700 1 -- 172.21.15.89:0/2501695240 shutdown_connections 2024-04-02T09:12:32.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:32.193+0000 7f3dff200700 1 -- 172.21.15.89:0/2501695240 wait complete. 2024-04-02T09:12:32.209 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T09:12:32.209 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T09:12:32.209 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T09:12:32.209 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T09:12:32.209 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T09:12:32.209 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T09:12:32.210 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T09:12:32.210 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T09:12:32.210 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:12:32.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:32.210 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T09:12:32.256 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T09:12:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T09:12:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T09:12:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:12:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T09:12:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:12:32.256 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:12:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:12:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T09:12:32.257 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:12:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:12:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T09:12:32.257 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:12:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:12:32.257 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T09:12:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:12:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T09:12:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 36676sec preferred_lft 36676sec 2024-04-02T09:12:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T09:12:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:12:32.259 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T09:12:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:12:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T09:12:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:12:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T09:12:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:12:32.260 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T09:12:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T09:12:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T09:12:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:12:32.261 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T09:12:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T09:12:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T09:12:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:12:32.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:32.262 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T09:12:32.360 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T09:12:32.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:32.363 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T09:12:32.411 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T09:12:32.411 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T09:12:32.411 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T09:12:32.411 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T09:12:32.411 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T09:12:32.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:32.412 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T09:12:32.413 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T09:12:32.413 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T09:12:32.413 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T09:12:32.413 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T09:12:32.413 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T09:12:32.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:32.573 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T09:12:32.619 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T09:12:32.619 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T09:12:32.619 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:12:32.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:32.620 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T09:12:32.728 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T09:12:32.728 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T09:12:32.773 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:12:32.773 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T09:12:32.774 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T09:12:32.859 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T09:12:32.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:32.859 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:12:33.001 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:12:33.001 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T09:12:33.028 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:12:33.029 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T09:12:33.029 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T09:12:33.029 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T09:12:33.029 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T09:12:33.034 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T09:12:33.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:33.035 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T09:12:33.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:33.102 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T09:12:33.201 INFO:teuthology.orchestra.run.smithi089.stdout:client9050 2024-04-02T09:12:33.202 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:12:33.202 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:12:33.247 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-04-02T09:12:33.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.486+0000 7f8f0ac00700 1 Processor -- start 2024-04-02T09:12:33.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.486+0000 7f8f0ac00700 1 -- start start 2024-04-02T09:12:33.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.487+0000 7f8f0ac00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8f0415ddf0 0x7f8f040ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:33.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.487+0000 7f8f0ac00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f040ce8a0 con 0x7f8f0415e1d0 2024-04-02T09:12:33.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.488+0000 7f8f0ac00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f040cdc90 con 0x7f8f040d28e0 2024-04-02T09:12:33.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.488+0000 7f8f0ac00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f0415ec60 con 0x7f8f0415ddf0 2024-04-02T09:12:33.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.488+0000 7f8f03400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8f0415e1d0 0x7f8f040d21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52766/0 (socket says 172.21.15.89:52766) 2024-04-02T09:12:33.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.488+0000 7f8f03400700 1 -- 172.21.15.89:0/2698389339 learned_addr learned my addr 172.21.15.89:0/2698389339 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:33.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.488+0000 7f8f03e00700 1 --2- 172.21.15.89:0/2698389339 >> v2:172.21.15.113:3300/0 conn(0x7f8f0415ddf0 0x7f8f040ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:33.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.488+0000 7f8f02a00700 1 -- 172.21.15.89:0/2698389339 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 884635452 0 0) 0x7f8f040ce8a0 con 0x7f8f0415e1d0 2024-04-02T09:12:33.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.488+0000 7f8f02a00700 1 -- 172.21.15.89:0/2698389339 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee80035b0 con 0x7f8f0415e1d0 2024-04-02T09:12:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.488+0000 7f8f02a00700 1 -- 172.21.15.89:0/2698389339 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1190423293 0 0) 0x7f8ee80035b0 con 0x7f8f0415e1d0 2024-04-02T09:12:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.488+0000 7f8f02a00700 1 -- 172.21.15.89:0/2698389339 >> v2:172.21.15.113:3300/0 conn(0x7f8f0415ddf0 msgr2=0x7f8f040ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.488+0000 7f8f02a00700 1 --2- 172.21.15.89:0/2698389339 >> v2:172.21.15.113:3300/0 conn(0x7f8f0415ddf0 0x7f8f040ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.488+0000 7f8f02a00700 1 -- 172.21.15.89:0/2698389339 >> v1:172.21.15.113:6789/0 conn(0x7f8f040d28e0 legacy=0x7f8f0415f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.488+0000 7f8f02a00700 1 -- 172.21.15.89:0/2698389339 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f040cdf70 con 0x7f8f0415e1d0 2024-04-02T09:12:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.488+0000 7f8f02a00700 1 -- 172.21.15.89:0/2698389339 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f8ef0001670 con 0x7f8f0415e1d0 2024-04-02T09:12:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f02a00700 1 -- 172.21.15.89:0/2698389339 >> v1:172.21.15.89:6789/0 conn(0x7f8f0415e1d0 legacy=0x7f8f040d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f02a00700 1 --2- 172.21.15.89:0/2698389339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ee8003a00 0x7f8ee8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f02a00700 1 --2- 172.21.15.89:0/2698389339 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8ee8004850 0x7f8ee8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f02a00700 1 --2- 172.21.15.89:0/2698389339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee8005210 0x7f8ee800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f02a00700 1 -- 172.21.15.89:0/2698389339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0415ec60 con 0x7f8ee8005210 2024-04-02T09:12:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f02a00700 1 -- 172.21.15.89:0/2698389339 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee800c1b0 con 0x7f8ee8003a00 2024-04-02T09:12:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f02a00700 1 -- 172.21.15.89:0/2698389339 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee800c370 con 0x7f8ee8004850 2024-04-02T09:12:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f08c00700 1 --2- 172.21.15.89:0/2698389339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee8005210 0x7f8ee800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f03e00700 1 --2- 172.21.15.89:0/2698389339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ee8003a00 0x7f8ee8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f03400700 1 --2- 172.21.15.89:0/2698389339 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8ee8004850 0x7f8ee8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f08c00700 1 -- 172.21.15.89:0/2698389339 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8ee8004850 msgr2=0x7f8ee8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f08c00700 1 --2- 172.21.15.89:0/2698389339 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8ee8004850 0x7f8ee8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:33.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f08c00700 1 -- 172.21.15.89:0/2698389339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ee8003a00 msgr2=0x7f8ee8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:33.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f08c00700 1 --2- 172.21.15.89:0/2698389339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ee8003a00 0x7f8ee8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:33.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f08c00700 1 -- 172.21.15.89:0/2698389339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee80035b0 con 0x7f8ee8005210 2024-04-02T09:12:33.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.489+0000 7f8f08c00700 1 -- 172.21.15.89:0/2698389339 reap_dead start 2024-04-02T09:12:33.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.490+0000 7f8f08c00700 1 --2- 172.21.15.89:0/2698389339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee8005210 0x7f8ee800ba10 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00c240 tx=0x7f8ee8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e7280bca507002 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:33.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.490+0000 7f8f02a00700 1 -- 172.21.15.89:0/2698389339 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc001df0 con 0x7f8ee8005210 2024-04-02T09:12:33.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.490+0000 7f8f02a00700 1 -- 172.21.15.89:0/2698389339 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8efc00cdc0 con 0x7f8ee8005210 2024-04-02T09:12:33.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.490+0000 7f8f02a00700 1 -- 172.21.15.89:0/2698389339 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc003c40 con 0x7f8ee8005210 2024-04-02T09:12:33.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.490+0000 7f8f0ac00700 1 -- 172.21.15.89:0/2698389339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee8005210 msgr2=0x7f8ee800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:33.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.490+0000 7f8f0ac00700 1 --2- 172.21.15.89:0/2698389339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee8005210 0x7f8ee800ba10 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00c240 tx=0x7f8ee8005880 comp rx=0 tx=0).stop 2024-04-02T09:12:33.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.490+0000 7f8f0ac00700 1 -- 172.21.15.89:0/2698389339 shutdown_connections 2024-04-02T09:12:33.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.490+0000 7f8f0ac00700 1 --2- 172.21.15.89:0/2698389339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee8005210 0x7f8ee800ba10 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:33.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.490+0000 7f8f0ac00700 1 -- 172.21.15.89:0/2698389339 >> 172.21.15.89:0/2698389339 conn(0x7f8f0408aff0 msgr2=0x7f8f040d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:33.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.490+0000 7f8f0ac00700 1 -- 172.21.15.89:0/2698389339 shutdown_connections 2024-04-02T09:12:33.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.490+0000 7f8f0ac00700 1 -- 172.21.15.89:0/2698389339 wait complete. 2024-04-02T09:12:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.492+0000 7f8f0ac00700 1 Processor -- start 2024-04-02T09:12:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.492+0000 7f8f0ac00700 1 -- start start 2024-04-02T09:12:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.493+0000 7f8f0ac00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee8005210 0x7f8f04161150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.493+0000 7f8f0ac00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f04161690 0x7f8f041494e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.493+0000 7f8f0ac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f04149a20 0x7f8f040c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.493+0000 7f8f0ac00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f04149eb0 con 0x7f8ee8005210 2024-04-02T09:12:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.493+0000 7f8f0ac00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f040c9e70 con 0x7f8f04161690 2024-04-02T09:12:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.493+0000 7f8f0ac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f040c92f0 con 0x7f8f04149a20 2024-04-02T09:12:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.493+0000 7f8f03e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee8005210 0x7f8f04161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.493+0000 7f8f03e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee8005210 0x7f8f04161150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52548/0 (socket says 172.21.15.89:52548) 2024-04-02T09:12:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.493+0000 7f8f03e00700 1 -- 172.21.15.89:0/1269332911 learned_addr learned my addr 172.21.15.89:0/1269332911 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:33.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.494+0000 7f8f08c00700 1 --2- 172.21.15.89:0/1269332911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f04149a20 0x7f8f040c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.494+0000 7f8f03400700 1 --2- 172.21.15.89:0/1269332911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f04161690 0x7f8f041494e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.494+0000 7f8f03e00700 1 -- 172.21.15.89:0/1269332911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f04149a20 msgr2=0x7f8f040c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.494+0000 7f8f03e00700 1 --2- 172.21.15.89:0/1269332911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f04149a20 0x7f8f040c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.494+0000 7f8f03e00700 1 -- 172.21.15.89:0/1269332911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f04161690 msgr2=0x7f8f041494e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.494+0000 7f8f03e00700 1 --2- 172.21.15.89:0/1269332911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f04161690 0x7f8f041494e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.494+0000 7f8f03e00700 1 -- 172.21.15.89:0/1269332911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f040c95d0 con 0x7f8ee8005210 2024-04-02T09:12:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.494+0000 7f8f03e00700 1 --2- 172.21.15.89:0/1269332911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee8005210 0x7f8f04161150 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40119a0 tx=0x7f8ef4011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.494+0000 7f8efbe00700 1 -- 172.21.15.89:0/1269332911 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef400f730 con 0x7f8ee8005210 2024-04-02T09:12:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.494+0000 7f8efbe00700 1 -- 172.21.15.89:0/1269332911 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef400f890 con 0x7f8ee8005210 2024-04-02T09:12:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.494+0000 7f8f0ac00700 1 -- 172.21.15.89:0/1269332911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f040c98b0 con 0x7f8ee8005210 2024-04-02T09:12:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.494+0000 7f8efbe00700 1 -- 172.21.15.89:0/1269332911 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef400fc20 con 0x7f8ee8005210 2024-04-02T09:12:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.495+0000 7f8f0ac00700 1 -- 172.21.15.89:0/1269332911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f040c8b20 con 0x7f8ee8005210 2024-04-02T09:12:33.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.496+0000 7f8f0ac00700 1 -- 172.21.15.89:0/1269332911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0406d790 con 0x7f8ee8005210 2024-04-02T09:12:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.496+0000 7f8efbe00700 1 -- 172.21.15.89:0/1269332911 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8ef400fd80 con 0x7f8ee8005210 2024-04-02T09:12:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.497+0000 7f8efbe00700 1 --2- 172.21.15.89:0/1269332911 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8ee406adb0 0x7f8ee406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.497+0000 7f8efbe00700 1 -- 172.21.15.89:0/1269332911 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f8ef4089b90 con 0x7f8ee8005210 2024-04-02T09:12:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.498+0000 7f8f03400700 1 --2- 172.21.15.89:0/1269332911 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8ee406adb0 0x7f8ee406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.498+0000 7f8f03400700 1 --2- 172.21.15.89:0/1269332911 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8ee406adb0 0x7f8ee406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8f04161a70 tx=0x7f8ef0001b20 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.502+0000 7f8efbe00700 1 -- 172.21.15.89:0/1269332911 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ef4054a90 con 0x7f8ee8005210 2024-04-02T09:12:33.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:33.748+0000 7f8f0ac00700 1 -- 172.21.15.89:0/1269332911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8f040ca670 con 0x7f8ee8005210 2024-04-02T09:12:34.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.592+0000 7f8efbe00700 1 -- 172.21.15.89:0/1269332911 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v106) v1 ==== 136+0+0 (secure 0 0 0) 0x7f8f040ca670 con 0x7f8ee8005210 2024-04-02T09:12:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.595+0000 7f8f0ac00700 1 -- 172.21.15.89:0/1269332911 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8ee406adb0 msgr2=0x7f8ee406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.595+0000 7f8f0ac00700 1 --2- 172.21.15.89:0/1269332911 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8ee406adb0 0x7f8ee406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8f04161a70 tx=0x7f8ef0001b20 comp rx=0 tx=0).stop 2024-04-02T09:12:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.595+0000 7f8f0ac00700 1 -- 172.21.15.89:0/1269332911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee8005210 msgr2=0x7f8f04161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.595+0000 7f8f0ac00700 1 --2- 172.21.15.89:0/1269332911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee8005210 0x7f8f04161150 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40119a0 tx=0x7f8ef4011cb0 comp rx=0 tx=0).stop 2024-04-02T09:12:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.595+0000 7f8f0ac00700 1 -- 172.21.15.89:0/1269332911 shutdown_connections 2024-04-02T09:12:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.595+0000 7f8f0ac00700 1 --2- 172.21.15.89:0/1269332911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f04149a20 0x7f8f040c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.595+0000 7f8f0ac00700 1 --2- 172.21.15.89:0/1269332911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee8005210 0x7f8f04161150 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.595+0000 7f8f0ac00700 1 --2- 172.21.15.89:0/1269332911 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8ee406adb0 0x7f8ee406d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.595+0000 7f8f0ac00700 1 --2- 172.21.15.89:0/1269332911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f04161690 0x7f8f041494e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.595+0000 7f8f0ac00700 1 -- 172.21.15.89:0/1269332911 >> 172.21.15.89:0/1269332911 conn(0x7f8f0408aff0 msgr2=0x7f8f040b8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.595+0000 7f8f0ac00700 1 -- 172.21.15.89:0/1269332911 shutdown_connections 2024-04-02T09:12:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.595+0000 7f8f0ac00700 1 -- 172.21.15.89:0/1269332911 wait complete. 2024-04-02T09:12:34.615 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:12:34.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.813+0000 7fdce2a00700 1 Processor -- start 2024-04-02T09:12:34.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.814+0000 7fdce2a00700 1 -- start start 2024-04-02T09:12:34.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.815+0000 7fdce2a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcdc15d710 0x7fdcdc0d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:34.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.815+0000 7fdce2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdcdc09aba0 0x7fdcdc0d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:34.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.815+0000 7fdce2a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc06bb40 con 0x7fdcdc06bf40 2024-04-02T09:12:34.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.815+0000 7fdce2a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcdc099b00 con 0x7fdcdc09aba0 2024-04-02T09:12:34.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.815+0000 7fdce2a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc099de0 con 0x7fdcdc15d710 2024-04-02T09:12:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.815+0000 7fdcdb400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdcdc09aba0 0x7fdcdc0d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.815+0000 7fdcdb400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdcdc09aba0 0x7fdcdc0d0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52556/0 (socket says 172.21.15.89:52556) 2024-04-02T09:12:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.815+0000 7fdcdb400700 1 -- 172.21.15.89:0/2138709508 learned_addr learned my addr 172.21.15.89:0/2138709508 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.815+0000 7fdcdbe00700 1 --2- 172.21.15.89:0/2138709508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcdc15d710 0x7fdcdc0d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.815+0000 7fdcdb400700 1 -- 172.21.15.89:0/2138709508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcdc15d710 msgr2=0x7fdcdc0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.815+0000 7fdcdb400700 1 --2- 172.21.15.89:0/2138709508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcdc15d710 0x7fdcdc0d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.815+0000 7fdcdb400700 1 -- 172.21.15.89:0/2138709508 >> v1:172.21.15.89:6789/0 conn(0x7fdcdc06bf40 legacy=0x7fdcdc0ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.815+0000 7fdcdb400700 1 -- 172.21.15.89:0/2138709508 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc09a0c0 con 0x7fdcdc09aba0 2024-04-02T09:12:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.815+0000 7fdcdaa00700 1 -- 172.21.15.89:0/2138709508 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897869227 0 0) 0x7fdcdc06bb40 con 0x7fdcdc06bf40 2024-04-02T09:12:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.816+0000 7fdcdb400700 1 --2- 172.21.15.89:0/2138709508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc09aba0 0x7fdcdc0d0670 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80098f0 tx=0x7fdcc8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b50b81393c1f54 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.816+0000 7fdcdaa00700 1 -- 172.21.15.89:0/2138709508 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc800b8a0 con 0x7fdcdc09aba0 2024-04-02T09:12:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.816+0000 7fdcdaa00700 1 -- 172.21.15.89:0/2138709508 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc800ba00 con 0x7fdcdc09aba0 2024-04-02T09:12:34.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.816+0000 7fdcdaa00700 1 -- 172.21.15.89:0/2138709508 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc800bd90 con 0x7fdcdc09aba0 2024-04-02T09:12:34.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.816+0000 7fdce2a00700 1 -- 172.21.15.89:0/2138709508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc09aba0 msgr2=0x7fdcdc0d0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:34.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.816+0000 7fdce2a00700 1 --2- 172.21.15.89:0/2138709508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc09aba0 0x7fdcdc0d0670 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80098f0 tx=0x7fdcc8009c00 comp rx=0 tx=0).stop 2024-04-02T09:12:34.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.817+0000 7fdce2a00700 1 -- 172.21.15.89:0/2138709508 shutdown_connections 2024-04-02T09:12:34.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.817+0000 7fdce2a00700 1 --2- 172.21.15.89:0/2138709508 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcdc15d710 0x7fdcdc0d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:34.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.817+0000 7fdce2a00700 1 --2- 172.21.15.89:0/2138709508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc09aba0 0x7fdcdc0d0670 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:34.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.817+0000 7fdce2a00700 1 -- 172.21.15.89:0/2138709508 >> 172.21.15.89:0/2138709508 conn(0x7fdcdc0d13e0 msgr2=0x7fdcdc0d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:34.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.817+0000 7fdce2a00700 1 -- 172.21.15.89:0/2138709508 shutdown_connections 2024-04-02T09:12:34.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.817+0000 7fdce2a00700 1 -- 172.21.15.89:0/2138709508 wait complete. 2024-04-02T09:12:34.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.819+0000 7fdce2a00700 1 Processor -- start 2024-04-02T09:12:34.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.819+0000 7fdce2a00700 1 -- start start 2024-04-02T09:12:34.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.820+0000 7fdce2a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcdc06bf40 0x7fdcdc1f7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:34.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.820+0000 7fdce2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc09aba0 0x7fdcdc14d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:34.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.820+0000 7fdce2a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcdc15d710 0x7fdcdc0b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:34.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.820+0000 7fdce2a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc153b80 con 0x7fdcdc09aba0 2024-04-02T09:12:34.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.820+0000 7fdce2a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc1f7ac0 con 0x7fdcdc06bf40 2024-04-02T09:12:34.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.820+0000 7fdce2a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc14dc40 con 0x7fdcdc15d710 2024-04-02T09:12:34.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.821+0000 7fdcdb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc09aba0 0x7fdcdc14d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:34.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.821+0000 7fdcdb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc09aba0 0x7fdcdc14d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52558/0 (socket says 172.21.15.89:52558) 2024-04-02T09:12:34.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.821+0000 7fdcdb400700 1 -- 172.21.15.89:0/2830099499 learned_addr learned my addr 172.21.15.89:0/2830099499 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:34.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.821+0000 7fdcdbe00700 1 --2- 172.21.15.89:0/2830099499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcdc06bf40 0x7fdcdc1f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:34.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.821+0000 7fdce0a00700 1 --2- 172.21.15.89:0/2830099499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcdc15d710 0x7fdcdc0b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:34.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.821+0000 7fdcdb400700 1 -- 172.21.15.89:0/2830099499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcdc15d710 msgr2=0x7fdcdc0b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:34.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.821+0000 7fdcdb400700 1 --2- 172.21.15.89:0/2830099499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcdc15d710 0x7fdcdc0b01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:34.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.821+0000 7fdcdb400700 1 -- 172.21.15.89:0/2830099499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcdc06bf40 msgr2=0x7fdcdc1f7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:34.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.821+0000 7fdcdb400700 1 --2- 172.21.15.89:0/2830099499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcdc06bf40 0x7fdcdc1f7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:34.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.821+0000 7fdcdb400700 1 -- 172.21.15.89:0/2830099499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc8009550 con 0x7fdcdc09aba0 2024-04-02T09:12:34.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.821+0000 7fdcdb400700 1 --2- 172.21.15.89:0/2830099499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc09aba0 0x7fdcdc14d3f0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8013000 tx=0x7fdcc8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:34.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.822+0000 7fdcd3e00700 1 -- 172.21.15.89:0/2830099499 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8004300 con 0x7fdcdc09aba0 2024-04-02T09:12:34.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.822+0000 7fdcd3e00700 1 -- 172.21.15.89:0/2830099499 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc801f070 con 0x7fdcdc09aba0 2024-04-02T09:12:34.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.822+0000 7fdce2a00700 1 -- 172.21.15.89:0/2830099499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcdc14d930 con 0x7fdcdc09aba0 2024-04-02T09:12:34.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.822+0000 7fdcd3e00700 1 -- 172.21.15.89:0/2830099499 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80036a0 con 0x7fdcdc09aba0 2024-04-02T09:12:34.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.822+0000 7fdce2a00700 1 -- 172.21.15.89:0/2830099499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcdc14e1c0 con 0x7fdcdc09aba0 2024-04-02T09:12:34.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.823+0000 7fdce2a00700 1 -- 172.21.15.89:0/2830099499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcdc067720 con 0x7fdcdc09aba0 2024-04-02T09:12:34.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.824+0000 7fdcd3e00700 1 -- 172.21.15.89:0/2830099499 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdcc8021030 con 0x7fdcdc09aba0 2024-04-02T09:12:34.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.825+0000 7fdcd3e00700 1 --2- 172.21.15.89:0/2830099499 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdcbc06adb0 0x7fdcbc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:34.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.825+0000 7fdcd3e00700 1 -- 172.21.15.89:0/2830099499 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fdcc808a950 con 0x7fdcdc09aba0 2024-04-02T09:12:34.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.825+0000 7fdcdbe00700 1 --2- 172.21.15.89:0/2830099499 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdcbc06adb0 0x7fdcbc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:34.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.826+0000 7fdcdbe00700 1 --2- 172.21.15.89:0/2830099499 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdcbc06adb0 0x7fdcbc06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdccc006b20 tx=0x7fdccc008040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:34.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:34.830+0000 7fdcd3e00700 1 -- 172.21.15.89:0/2830099499 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdcc8055850 con 0x7fdcdc09aba0 2024-04-02T09:12:35.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.034+0000 7fdce2a00700 1 -- 172.21.15.89:0/2830099499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcdc0b7360 con 0x7fdcdc09aba0 2024-04-02T09:12:35.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.035+0000 7fdcd3e00700 1 -- 172.21.15.89:0/2830099499 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+16736 (secure 0 0 0) 0x7fdcc8058e70 con 0x7fdcdc09aba0 2024-04-02T09:12:35.034 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:35.034 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":223,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:12:26.173562+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:12:23.034869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-02T09:12:24.158560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:12:35.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.038+0000 7fdce2a00700 1 -- 172.21.15.89:0/2830099499 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdcbc06adb0 msgr2=0x7fdcbc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.038+0000 7fdce2a00700 1 --2- 172.21.15.89:0/2830099499 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdcbc06adb0 0x7fdcbc06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdccc006b20 tx=0x7fdccc008040 comp rx=0 tx=0).stop 2024-04-02T09:12:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.038+0000 7fdce2a00700 1 -- 172.21.15.89:0/2830099499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc09aba0 msgr2=0x7fdcdc14d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.038+0000 7fdce2a00700 1 --2- 172.21.15.89:0/2830099499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc09aba0 0x7fdcdc14d3f0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8013000 tx=0x7fdcc8004750 comp rx=0 tx=0).stop 2024-04-02T09:12:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.038+0000 7fdce2a00700 1 -- 172.21.15.89:0/2830099499 shutdown_connections 2024-04-02T09:12:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.038+0000 7fdce2a00700 1 --2- 172.21.15.89:0/2830099499 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdcbc06adb0 0x7fdcbc06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.038+0000 7fdce2a00700 1 --2- 172.21.15.89:0/2830099499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcdc06bf40 0x7fdcdc1f7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.038+0000 7fdce2a00700 1 --2- 172.21.15.89:0/2830099499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcdc15d710 0x7fdcdc0b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.038+0000 7fdce2a00700 1 --2- 172.21.15.89:0/2830099499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc09aba0 0x7fdcdc14d3f0 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.038+0000 7fdce2a00700 1 -- 172.21.15.89:0/2830099499 >> 172.21.15.89:0/2830099499 conn(0x7fdcdc0d13e0 msgr2=0x7fdcdc06ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.038+0000 7fdce2a00700 1 -- 172.21.15.89:0/2830099499 shutdown_connections 2024-04-02T09:12:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.038+0000 7fdce2a00700 1 -- 172.21.15.89:0/2830099499 wait complete. 2024-04-02T09:12:35.055 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T09:12:23.034869+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T09:12:24.158560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '223', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:12:35.055 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:12:35.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.253+0000 7f2e72c00700 1 Processor -- start 2024-04-02T09:12:35.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.253+0000 7f2e72c00700 1 -- start start 2024-04-02T09:12:35.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.254+0000 7f2e72c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e6c0d16a0 0x7f2e6c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:35.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.254+0000 7f2e72c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6c161420 con 0x7f2e6c0d2370 2024-04-02T09:12:35.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.254+0000 7f2e72c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6c161700 con 0x7f2e6c15d9d0 2024-04-02T09:12:35.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.254+0000 7f2e72c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c0d2820 con 0x7f2e6c0d16a0 2024-04-02T09:12:35.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.255+0000 7f2e70c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2e6c0d2370 0x7f2e6c0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52774/0 (socket says 172.21.15.89:52774) 2024-04-02T09:12:35.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.255+0000 7f2e70c00700 1 -- 172.21.15.89:0/849470313 learned_addr learned my addr 172.21.15.89:0/849470313 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:35.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.255+0000 7f2e6b400700 1 --2- 172.21.15.89:0/849470313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e6c0d16a0 0x7f2e6c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:35.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.255+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087595809 0 0) 0x7f2e6c161420 con 0x7f2e6c0d2370 2024-04-02T09:12:35.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.255+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e500035b0 con 0x7f2e6c0d2370 2024-04-02T09:12:35.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.255+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585455431 0 0) 0x7f2e6c161700 con 0x7f2e6c15d9d0 2024-04-02T09:12:35.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.255+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e6c161420 con 0x7f2e6c15d9d0 2024-04-02T09:12:35.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.255+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1724360486 0 0) 0x7f2e500035b0 con 0x7f2e6c0d2370 2024-04-02T09:12:35.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.255+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e6c0d16a0 msgr2=0x7f2e6c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:35.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.255+0000 7f2e6aa00700 1 --2- 172.21.15.89:0/849470313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e6c0d16a0 0x7f2e6c0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.255+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 >> v1:172.21.15.113:6789/0 conn(0x7f2e6c15d9d0 legacy=0x7f2e6c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.255+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c1619e0 con 0x7f2e6c0d2370 2024-04-02T09:12:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.255+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f2e640019f0 con 0x7f2e6c0d2370 2024-04-02T09:12:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.255+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 >> v1:172.21.15.89:6789/0 conn(0x7f2e6c0d2370 legacy=0x7f2e6c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e6aa00700 1 --2- 172.21.15.89:0/849470313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e50003bc0 0x7f2e50003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e6aa00700 1 --2- 172.21.15.89:0/849470313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e50004850 0x7f2e50004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e6aa00700 1 --2- 172.21.15.89:0/849470313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e50005210 0x7f2e5000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c0d2820 con 0x7f2e50004850 2024-04-02T09:12:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5000c000 con 0x7f2e50003bc0 2024-04-02T09:12:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5000c1c0 con 0x7f2e50005210 2024-04-02T09:12:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e6b400700 1 --2- 172.21.15.89:0/849470313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e6c0d16a0 0x7f2e6c0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:12:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e6b400700 1 --2- 172.21.15.89:0/849470313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e50004850 0x7f2e50004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e6be00700 1 --2- 172.21.15.89:0/849470313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e50003bc0 0x7f2e50003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:35.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e70c00700 1 --2- 172.21.15.89:0/849470313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e50005210 0x7f2e5000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e6b400700 1 -- 172.21.15.89:0/849470313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e50005210 msgr2=0x7f2e5000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e6b400700 1 --2- 172.21.15.89:0/849470313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e50005210 0x7f2e5000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e6b400700 1 -- 172.21.15.89:0/849470313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e50003bc0 msgr2=0x7f2e50003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e6b400700 1 --2- 172.21.15.89:0/849470313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e50003bc0 0x7f2e50003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.256+0000 7f2e6b400700 1 -- 172.21.15.89:0/849470313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e500035b0 con 0x7f2e50004850 2024-04-02T09:12:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.257+0000 7f2e6b400700 1 --2- 172.21.15.89:0/849470313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e50004850 0x7f2e50004cd0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c0cfbc0 tx=0x7f2e50005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43962e60e3261551 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.257+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58004030 con 0x7f2e50004850 2024-04-02T09:12:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.257+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e58004190 con 0x7f2e50004850 2024-04-02T09:12:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.257+0000 7f2e6aa00700 1 -- 172.21.15.89:0/849470313 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58004540 con 0x7f2e50004850 2024-04-02T09:12:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.257+0000 7f2e72c00700 1 -- 172.21.15.89:0/849470313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e50004850 msgr2=0x7f2e50004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.257+0000 7f2e72c00700 1 --2- 172.21.15.89:0/849470313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e50004850 0x7f2e50004cd0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c0cfbc0 tx=0x7f2e50005880 comp rx=0 tx=0).stop 2024-04-02T09:12:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.257+0000 7f2e70c00700 1 -- 172.21.15.89:0/849470313 reap_dead start 2024-04-02T09:12:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.257+0000 7f2e72c00700 1 -- 172.21.15.89:0/849470313 shutdown_connections 2024-04-02T09:12:35.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.257+0000 7f2e72c00700 1 -- 172.21.15.89:0/849470313 >> 172.21.15.89:0/849470313 conn(0x7f2e6c08aff0 msgr2=0x7f2e6c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:35.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.257+0000 7f2e72c00700 1 -- 172.21.15.89:0/849470313 shutdown_connections 2024-04-02T09:12:35.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.257+0000 7f2e72c00700 1 -- 172.21.15.89:0/849470313 wait complete. 2024-04-02T09:12:35.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.259+0000 7f2e72c00700 1 Processor -- start 2024-04-02T09:12:35.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.259+0000 7f2e72c00700 1 -- start start 2024-04-02T09:12:35.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.260+0000 7f2e72c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e50004850 0x7f2e6c155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:35.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.260+0000 7f2e72c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c1e7560 0x7f2e6c0b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:35.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.260+0000 7f2e72c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e6c1e79e0 0x7f2e6c0b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:35.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.260+0000 7f2e72c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c0b7fe0 con 0x7f2e6c1e7560 2024-04-02T09:12:35.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.260+0000 7f2e72c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c0cbf80 con 0x7f2e50004850 2024-04-02T09:12:35.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.260+0000 7f2e72c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c0b6d10 con 0x7f2e6c1e79e0 2024-04-02T09:12:35.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.261+0000 7f2e6b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c1e7560 0x7f2e6c0b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:35.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.261+0000 7f2e6b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c1e7560 0x7f2e6c0b47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52574/0 (socket says 172.21.15.89:52574) 2024-04-02T09:12:35.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.261+0000 7f2e6b400700 1 -- 172.21.15.89:0/3852318099 learned_addr learned my addr 172.21.15.89:0/3852318099 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:35.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.261+0000 7f2e6be00700 1 --2- 172.21.15.89:0/3852318099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e50004850 0x7f2e6c155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:35.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.261+0000 7f2e6b400700 1 -- 172.21.15.89:0/3852318099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e6c1e79e0 msgr2=0x7f2e6c0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.261+0000 7f2e6b400700 1 --2- 172.21.15.89:0/3852318099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e6c1e79e0 0x7f2e6c0b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.261+0000 7f2e6b400700 1 -- 172.21.15.89:0/3852318099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e50004850 msgr2=0x7f2e6c155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.261+0000 7f2e6b400700 1 --2- 172.21.15.89:0/3852318099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e50004850 0x7f2e6c155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.261+0000 7f2e6b400700 1 -- 172.21.15.89:0/3852318099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c0b6ff0 con 0x7f2e6c1e7560 2024-04-02T09:12:35.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.261+0000 7f2e6be00700 1 --2- 172.21.15.89:0/3852318099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e50004850 0x7f2e6c155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:35.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.261+0000 7f2e6b400700 1 --2- 172.21.15.89:0/3852318099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c1e7560 0x7f2e6c0b47c0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f2e580095e0 tx=0x7f2e580047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:35.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.261+0000 7f2e63e00700 1 -- 172.21.15.89:0/3852318099 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58004030 con 0x7f2e6c1e7560 2024-04-02T09:12:35.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.261+0000 7f2e63e00700 1 -- 172.21.15.89:0/3852318099 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e5800e8a0 con 0x7f2e6c1e7560 2024-04-02T09:12:35.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.262+0000 7f2e72c00700 1 -- 172.21.15.89:0/3852318099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6c0cc1d0 con 0x7f2e6c1e7560 2024-04-02T09:12:35.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.262+0000 7f2e63e00700 1 -- 172.21.15.89:0/3852318099 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5800ec30 con 0x7f2e6c1e7560 2024-04-02T09:12:35.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.262+0000 7f2e72c00700 1 -- 172.21.15.89:0/3852318099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e6c1558a0 con 0x7f2e6c1e7560 2024-04-02T09:12:35.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.263+0000 7f2e72c00700 1 -- 172.21.15.89:0/3852318099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e6c067720 con 0x7f2e6c1e7560 2024-04-02T09:12:35.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.264+0000 7f2e63e00700 1 -- 172.21.15.89:0/3852318099 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2e58023070 con 0x7f2e6c1e7560 2024-04-02T09:12:35.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.265+0000 7f2e63e00700 1 --2- 172.21.15.89:0/3852318099 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2e4c06adb0 0x7f2e4c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:35.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.265+0000 7f2e63e00700 1 -- 172.21.15.89:0/3852318099 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f2e5808ad10 con 0x7f2e6c1e7560 2024-04-02T09:12:35.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.265+0000 7f2e6be00700 1 --2- 172.21.15.89:0/3852318099 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2e4c06adb0 0x7f2e4c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:35.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.266+0000 7f2e6be00700 1 --2- 172.21.15.89:0/3852318099 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2e4c06adb0 0x7f2e4c06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c0010e0 tx=0x7f2e5c009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:35.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.269+0000 7f2e63e00700 1 -- 172.21.15.89:0/3852318099 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e58090020 con 0x7f2e6c1e7560 2024-04-02T09:12:35.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.470+0000 7f2e72c00700 1 -- 172.21.15.89:0/3852318099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2e6c08d690 con 0x7f2e6c1e7560 2024-04-02T09:12:35.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.470+0000 7f2e63e00700 1 -- 172.21.15.89:0/3852318099 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v223) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2e5802b020 con 0x7f2e6c1e7560 2024-04-02T09:12:35.469 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:12:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.473+0000 7f2e72c00700 1 -- 172.21.15.89:0/3852318099 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2e4c06adb0 msgr2=0x7f2e4c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.473+0000 7f2e72c00700 1 --2- 172.21.15.89:0/3852318099 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2e4c06adb0 0x7f2e4c06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c0010e0 tx=0x7f2e5c009040 comp rx=0 tx=0).stop 2024-04-02T09:12:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.473+0000 7f2e72c00700 1 -- 172.21.15.89:0/3852318099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c1e7560 msgr2=0x7f2e6c0b47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.473+0000 7f2e72c00700 1 --2- 172.21.15.89:0/3852318099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c1e7560 0x7f2e6c0b47c0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f2e580095e0 tx=0x7f2e580047c0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.473+0000 7f2e72c00700 1 -- 172.21.15.89:0/3852318099 shutdown_connections 2024-04-02T09:12:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.473+0000 7f2e72c00700 1 --2- 172.21.15.89:0/3852318099 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2e4c06adb0 0x7f2e4c06d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.473+0000 7f2e72c00700 1 --2- 172.21.15.89:0/3852318099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e50004850 0x7f2e6c155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.473+0000 7f2e72c00700 1 --2- 172.21.15.89:0/3852318099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e6c1e79e0 0x7f2e6c0b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.473+0000 7f2e72c00700 1 --2- 172.21.15.89:0/3852318099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e6c1e7560 0x7f2e6c0b47c0 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.473+0000 7f2e72c00700 1 -- 172.21.15.89:0/3852318099 >> 172.21.15.89:0/3852318099 conn(0x7f2e6c08aff0 msgr2=0x7f2e6c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:35.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.473+0000 7f2e72c00700 1 -- 172.21.15.89:0/3852318099 shutdown_connections 2024-04-02T09:12:35.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.473+0000 7f2e72c00700 1 -- 172.21.15.89:0/3852318099 wait complete. 2024-04-02T09:12:35.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T09:12:35.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.686+0000 7f6842c00700 1 Processor -- start 2024-04-02T09:12:35.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.686+0000 7f6842c00700 1 -- start start 2024-04-02T09:12:35.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.687+0000 7f6842c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f683c1611b0 0x7f683c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:35.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.688+0000 7f6842c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f683c0cee10 0x7f683c161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:35.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.688+0000 7f6842c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f683c0ce270 con 0x7f683c15f4d0 2024-04-02T09:12:35.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.688+0000 7f6842c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f683c15b600 con 0x7f683c1611b0 2024-04-02T09:12:35.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.688+0000 7f6842c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f683c0d2760 con 0x7f683c0cee10 2024-04-02T09:12:35.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.688+0000 7f683be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f683c1611b0 0x7f683c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:35.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.688+0000 7f683be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f683c1611b0 0x7f683c15ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52582/0 (socket says 172.21.15.89:52582) 2024-04-02T09:12:35.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.688+0000 7f683be00700 1 -- 172.21.15.89:0/3092883071 learned_addr learned my addr 172.21.15.89:0/3092883071 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:35.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.688+0000 7f6840c00700 1 --2- 172.21.15.89:0/3092883071 >> v2:172.21.15.113:3300/0 conn(0x7f683c0cee10 0x7f683c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:35.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.688+0000 7f683be00700 1 -- 172.21.15.89:0/3092883071 >> v2:172.21.15.113:3300/0 conn(0x7f683c0cee10 msgr2=0x7f683c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:35.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.688+0000 7f683be00700 1 --2- 172.21.15.89:0/3092883071 >> v2:172.21.15.113:3300/0 conn(0x7f683c0cee10 0x7f683c161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.688+0000 7f683be00700 1 -- 172.21.15.89:0/3092883071 >> v1:172.21.15.113:6789/0 conn(0x7f683c15f4d0 legacy=0x7f683c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.688+0000 7f683be00700 1 -- 172.21.15.89:0/3092883071 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683c0d2a40 con 0x7f683c1611b0 2024-04-02T09:12:35.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.689+0000 7f683be00700 1 --2- 172.21.15.89:0/3092883071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f683c1611b0 0x7f683c15ef90 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f683c0d2f60 tx=0x7f682c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb83efa557e2c16 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:35.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.689+0000 7f683aa00700 1 -- 172.21.15.89:0/3092883071 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c00b870 con 0x7f683c1611b0 2024-04-02T09:12:35.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.689+0000 7f683aa00700 1 -- 172.21.15.89:0/3092883071 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682c00b9d0 con 0x7f683c1611b0 2024-04-02T09:12:35.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.689+0000 7f683aa00700 1 -- 172.21.15.89:0/3092883071 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c00bd60 con 0x7f683c1611b0 2024-04-02T09:12:35.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.689+0000 7f6842c00700 1 -- 172.21.15.89:0/3092883071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f683c1611b0 msgr2=0x7f683c15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.689+0000 7f6842c00700 1 --2- 172.21.15.89:0/3092883071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f683c1611b0 0x7f683c15ef90 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f683c0d2f60 tx=0x7f682c009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.689+0000 7f6842c00700 1 -- 172.21.15.89:0/3092883071 shutdown_connections 2024-04-02T09:12:35.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.689+0000 7f6842c00700 1 --2- 172.21.15.89:0/3092883071 >> v2:172.21.15.113:3300/0 conn(0x7f683c0cee10 0x7f683c161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.689+0000 7f6842c00700 1 --2- 172.21.15.89:0/3092883071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f683c1611b0 0x7f683c15ef90 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.689+0000 7f6842c00700 1 -- 172.21.15.89:0/3092883071 >> 172.21.15.89:0/3092883071 conn(0x7f683c08aff0 msgr2=0x7f683c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:35.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.690+0000 7f6842c00700 1 -- 172.21.15.89:0/3092883071 shutdown_connections 2024-04-02T09:12:35.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.690+0000 7f6842c00700 1 -- 172.21.15.89:0/3092883071 wait complete. 2024-04-02T09:12:35.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.691+0000 7f6842c00700 1 Processor -- start 2024-04-02T09:12:35.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.691+0000 7f6842c00700 1 -- start start 2024-04-02T09:12:35.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.692+0000 7f6842c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f683c0cee10 0x7f683c153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:35.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.692+0000 7f6842c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f683c15f4d0 0x7f683c0b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:35.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.692+0000 7f6842c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f683c1611b0 0x7f683c0cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:35.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.692+0000 7f6842c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c0b7460 con 0x7f683c0cee10 2024-04-02T09:12:35.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.692+0000 7f6842c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c0b7740 con 0x7f683c15f4d0 2024-04-02T09:12:35.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.692+0000 7f6842c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f683c0b7a20 con 0x7f683c1611b0 2024-04-02T09:12:35.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.692+0000 7f683be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f683c0cee10 0x7f683c153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.693+0000 7f683be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f683c0cee10 0x7f683c153df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52588/0 (socket says 172.21.15.89:52588) 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.693+0000 7f683be00700 1 -- 172.21.15.89:0/231290148 learned_addr learned my addr 172.21.15.89:0/231290148 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.693+0000 7f683b400700 1 --2- 172.21.15.89:0/231290148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f683c15f4d0 0x7f683c0b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.693+0000 7f6840c00700 1 --2- 172.21.15.89:0/231290148 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f683c1611b0 0x7f683c0cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.693+0000 7f683be00700 1 -- 172.21.15.89:0/231290148 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f683c1611b0 msgr2=0x7f683c0cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.693+0000 7f683be00700 1 --2- 172.21.15.89:0/231290148 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f683c1611b0 0x7f683c0cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.693+0000 7f683be00700 1 -- 172.21.15.89:0/231290148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f683c15f4d0 msgr2=0x7f683c0b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.693+0000 7f683be00700 1 --2- 172.21.15.89:0/231290148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f683c15f4d0 0x7f683c0b6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.693+0000 7f683be00700 1 -- 172.21.15.89:0/231290148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c009550 con 0x7f683c0cee10 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.693+0000 7f683be00700 1 --2- 172.21.15.89:0/231290148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f683c0cee10 0x7f683c153df0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f682c000c00 tx=0x7f682c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.693+0000 7f6833e00700 1 -- 172.21.15.89:0/231290148 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c004520 con 0x7f683c0cee10 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.693+0000 7f6833e00700 1 -- 172.21.15.89:0/231290148 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682c01e070 con 0x7f683c0cee10 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.693+0000 7f6842c00700 1 -- 172.21.15.89:0/231290148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683c0ca6a0 con 0x7f683c0cee10 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.693+0000 7f6833e00700 1 -- 172.21.15.89:0/231290148 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c0038a0 con 0x7f683c0cee10 2024-04-02T09:12:35.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.694+0000 7f6842c00700 1 -- 172.21.15.89:0/231290148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683c154330 con 0x7f683c0cee10 2024-04-02T09:12:35.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.694+0000 7f6842c00700 1 -- 172.21.15.89:0/231290148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f683c067720 con 0x7f683c0cee10 2024-04-02T09:12:35.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.695+0000 7f6833e00700 1 -- 172.21.15.89:0/231290148 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f682c003a00 con 0x7f683c0cee10 2024-04-02T09:12:35.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.696+0000 7f6833e00700 1 --2- 172.21.15.89:0/231290148 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f681c06adb0 0x7f681c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:35.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.696+0000 7f6833e00700 1 -- 172.21.15.89:0/231290148 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f682c020070 con 0x7f683c0cee10 2024-04-02T09:12:35.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.697+0000 7f683b400700 1 --2- 172.21.15.89:0/231290148 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f681c06adb0 0x7f681c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:35.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.697+0000 7f683b400700 1 --2- 172.21.15.89:0/231290148 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f681c06adb0 0x7f681c06d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f683c162100 tx=0x7f6828009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:35.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.701+0000 7f6833e00700 1 -- 172.21.15.89:0/231290148 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f682c055bd0 con 0x7f683c0cee10 2024-04-02T09:12:35.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.905+0000 7f6842c00700 1 -- 172.21.15.89:0/231290148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f683c0b5d00 con 0x7f683c0cee10 2024-04-02T09:12:35.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.905+0000 7f6833e00700 1 -- 172.21.15.89:0/231290148 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v223) v1 ==== 104+0+11 (secure 0 0 0) 0x7f682c0150a0 con 0x7f683c0cee10 2024-04-02T09:12:35.904 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:12:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.908+0000 7f6842c00700 1 -- 172.21.15.89:0/231290148 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f681c06adb0 msgr2=0x7f681c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.908+0000 7f6842c00700 1 --2- 172.21.15.89:0/231290148 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f681c06adb0 0x7f681c06d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f683c162100 tx=0x7f6828009040 comp rx=0 tx=0).stop 2024-04-02T09:12:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.908+0000 7f6842c00700 1 -- 172.21.15.89:0/231290148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f683c0cee10 msgr2=0x7f683c153df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.908+0000 7f6842c00700 1 --2- 172.21.15.89:0/231290148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f683c0cee10 0x7f683c153df0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f682c000c00 tx=0x7f682c004750 comp rx=0 tx=0).stop 2024-04-02T09:12:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.908+0000 7f6842c00700 1 -- 172.21.15.89:0/231290148 shutdown_connections 2024-04-02T09:12:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.908+0000 7f6842c00700 1 --2- 172.21.15.89:0/231290148 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f683c1611b0 0x7f683c0cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.908+0000 7f6842c00700 1 --2- 172.21.15.89:0/231290148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f683c0cee10 0x7f683c153df0 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.908+0000 7f6842c00700 1 --2- 172.21.15.89:0/231290148 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f681c06adb0 0x7f681c06d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.908+0000 7f6842c00700 1 --2- 172.21.15.89:0/231290148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f683c15f4d0 0x7f683c0b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.908+0000 7f6842c00700 1 -- 172.21.15.89:0/231290148 >> 172.21.15.89:0/231290148 conn(0x7f683c08aff0 msgr2=0x7f683c15c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.908+0000 7f6842c00700 1 -- 172.21.15.89:0/231290148 shutdown_connections 2024-04-02T09:12:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:35.908+0000 7f6842c00700 1 -- 172.21.15.89:0/231290148 wait complete. 2024-04-02T09:12:35.925 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T09:12:36.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.120+0000 7fa832800700 1 Processor -- start 2024-04-02T09:12:36.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.120+0000 7fa832800700 1 -- start start 2024-04-02T09:12:36.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.121+0000 7fa832800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa82c1611b0 0x7fa82c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.121+0000 7fa832800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82c0ce270 con 0x7fa82c15f4d0 2024-04-02T09:12:36.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.121+0000 7fa832800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82c15b600 con 0x7fa82c0cee10 2024-04-02T09:12:36.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.121+0000 7fa832800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0d2760 con 0x7fa82c1611b0 2024-04-02T09:12:36.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.122+0000 7fa82aa00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa82c15f4d0 0x7fa82c0ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52784/0 (socket says 172.21.15.89:52784) 2024-04-02T09:12:36.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.122+0000 7fa82aa00700 1 -- 172.21.15.89:0/3108867409 learned_addr learned my addr 172.21.15.89:0/3108867409 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:36.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.122+0000 7fa82b400700 1 --2- 172.21.15.89:0/3108867409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa82c1611b0 0x7fa82c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.122+0000 7fa82a000700 1 -- 172.21.15.89:0/3108867409 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225107388 0 0) 0x7fa82c0ce270 con 0x7fa82c15f4d0 2024-04-02T09:12:36.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.122+0000 7fa82a000700 1 -- 172.21.15.89:0/3108867409 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8100035b0 con 0x7fa82c15f4d0 2024-04-02T09:12:36.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.122+0000 7fa82a000700 1 -- 172.21.15.89:0/3108867409 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128237043 0 0) 0x7fa82c15b600 con 0x7fa82c0cee10 2024-04-02T09:12:36.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.122+0000 7fa82a000700 1 -- 172.21.15.89:0/3108867409 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa82c0ce270 con 0x7fa82c0cee10 2024-04-02T09:12:36.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.122+0000 7fa82b400700 1 -- 172.21.15.89:0/3108867409 >> v1:172.21.15.113:6789/0 conn(0x7fa82c0cee10 legacy=0x7fa82c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.122+0000 7fa82b400700 1 -- 172.21.15.89:0/3108867409 >> v1:172.21.15.89:6789/0 conn(0x7fa82c15f4d0 legacy=0x7fa82c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.122+0000 7fa82b400700 1 -- 172.21.15.89:0/3108867409 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c0d2a40 con 0x7fa82c1611b0 2024-04-02T09:12:36.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.123+0000 7fa82b400700 1 --2- 172.21.15.89:0/3108867409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa82c1611b0 0x7fa82c15ef90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa81c009870 tx=0x7fa81c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd38cb9b029e9f36 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:36.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.123+0000 7fa82a000700 1 -- 172.21.15.89:0/3108867409 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c011830 con 0x7fa82c1611b0 2024-04-02T09:12:36.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.123+0000 7fa82a000700 1 -- 172.21.15.89:0/3108867409 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81c011990 con 0x7fa82c1611b0 2024-04-02T09:12:36.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.123+0000 7fa82a000700 1 -- 172.21.15.89:0/3108867409 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c011d20 con 0x7fa82c1611b0 2024-04-02T09:12:36.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.123+0000 7fa832800700 1 -- 172.21.15.89:0/3108867409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa82c1611b0 msgr2=0x7fa82c15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.123+0000 7fa832800700 1 --2- 172.21.15.89:0/3108867409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa82c1611b0 0x7fa82c15ef90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa81c009870 tx=0x7fa81c009b80 comp rx=0 tx=0).stop 2024-04-02T09:12:36.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.124+0000 7fa832800700 1 -- 172.21.15.89:0/3108867409 shutdown_connections 2024-04-02T09:12:36.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.124+0000 7fa832800700 1 --2- 172.21.15.89:0/3108867409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa82c1611b0 0x7fa82c15ef90 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.124+0000 7fa832800700 1 -- 172.21.15.89:0/3108867409 >> 172.21.15.89:0/3108867409 conn(0x7fa82c08aff0 msgr2=0x7fa82c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:36.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.124+0000 7fa832800700 1 -- 172.21.15.89:0/3108867409 shutdown_connections 2024-04-02T09:12:36.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.124+0000 7fa832800700 1 -- 172.21.15.89:0/3108867409 wait complete. 2024-04-02T09:12:36.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.125+0000 7fa832800700 1 Processor -- start 2024-04-02T09:12:36.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.126+0000 7fa832800700 1 -- start start 2024-04-02T09:12:36.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.126+0000 7fa832800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa82c0cee10 0x7fa82c0b7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.126+0000 7fa832800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa82c15f4d0 0x7fa82c0cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.126+0000 7fa832800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa82c1611b0 0x7fa82c157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.126+0000 7fa832800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c1591b0 con 0x7fa82c15f4d0 2024-04-02T09:12:36.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.126+0000 7fa832800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0b8b50 con 0x7fa82c1611b0 2024-04-02T09:12:36.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.126+0000 7fa832800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0b82b0 con 0x7fa82c0cee10 2024-04-02T09:12:36.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.127+0000 7fa82aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa82c15f4d0 0x7fa82c0cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.127+0000 7fa82aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa82c15f4d0 0x7fa82c0cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52598/0 (socket says 172.21.15.89:52598) 2024-04-02T09:12:36.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.127+0000 7fa82aa00700 1 -- 172.21.15.89:0/368979090 learned_addr learned my addr 172.21.15.89:0/368979090 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.127+0000 7fa82b400700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa82c0cee10 0x7fa82c0b7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.127+0000 7fa82be00700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa82c1611b0 0x7fa82c157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.127+0000 7fa82aa00700 1 -- 172.21.15.89:0/368979090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa82c0cee10 msgr2=0x7fa82c0b7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.127+0000 7fa82aa00700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa82c0cee10 0x7fa82c0b7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.127+0000 7fa82aa00700 1 -- 172.21.15.89:0/368979090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa82c1611b0 msgr2=0x7fa82c157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.127+0000 7fa82aa00700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa82c1611b0 0x7fa82c157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.127+0000 7fa82aa00700 1 -- 172.21.15.89:0/368979090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c0094a0 con 0x7fa82c15f4d0 2024-04-02T09:12:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.127+0000 7fa82be00700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa82c1611b0 0x7fa82c157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.127+0000 7fa82aa00700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa82c15f4d0 0x7fa82c0cce00 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7fa818005910 tx=0x7fa81800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.128+0000 7fa823400700 1 -- 172.21.15.89:0/368979090 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818005d70 con 0x7fa82c15f4d0 2024-04-02T09:12:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.128+0000 7fa823400700 1 -- 172.21.15.89:0/368979090 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8180070d0 con 0x7fa82c15f4d0 2024-04-02T09:12:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.128+0000 7fa823400700 1 -- 172.21.15.89:0/368979090 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818007480 con 0x7fa82c15f4d0 2024-04-02T09:12:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.128+0000 7fa832800700 1 -- 172.21.15.89:0/368979090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82c0b8590 con 0x7fa82c15f4d0 2024-04-02T09:12:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.128+0000 7fa832800700 1 -- 172.21.15.89:0/368979090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82c0c6c80 con 0x7fa82c15f4d0 2024-04-02T09:12:36.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.129+0000 7fa832800700 1 -- 172.21.15.89:0/368979090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82c067720 con 0x7fa82c15f4d0 2024-04-02T09:12:36.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.130+0000 7fa823400700 1 -- 172.21.15.89:0/368979090 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa8180075e0 con 0x7fa82c15f4d0 2024-04-02T09:12:36.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.131+0000 7fa823400700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa80c06adb0 0x7fa80c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.131+0000 7fa823400700 1 -- 172.21.15.89:0/368979090 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fa818088bc0 con 0x7fa82c15f4d0 2024-04-02T09:12:36.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.131+0000 7fa82b400700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa80c06adb0 0x7fa80c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.132+0000 7fa82b400700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa80c06adb0 0x7fa80c06d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0097c0 tx=0x7fa81c01c040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:36.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.135+0000 7fa823400700 1 -- 172.21.15.89:0/368979090 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa818053a30 con 0x7fa82c15f4d0 2024-04-02T09:12:36.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.335+0000 7fa832800700 1 -- 172.21.15.89:0/368979090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa82c0b5d00 con 0x7fa82c15f4d0 2024-04-02T09:12:36.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.335+0000 7fa823400700 1 -- 172.21.15.89:0/368979090 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v223) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa818017020 con 0x7fa82c15f4d0 2024-04-02T09:12:36.333 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:12:36.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.337+0000 7fa832800700 1 -- 172.21.15.89:0/368979090 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa80c06adb0 msgr2=0x7fa80c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.337+0000 7fa832800700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa80c06adb0 0x7fa80c06d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0097c0 tx=0x7fa81c01c040 comp rx=0 tx=0).stop 2024-04-02T09:12:36.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.337+0000 7fa832800700 1 -- 172.21.15.89:0/368979090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa82c15f4d0 msgr2=0x7fa82c0cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.337+0000 7fa832800700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa82c15f4d0 0x7fa82c0cce00 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7fa818005910 tx=0x7fa81800c9e0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.338+0000 7fa832800700 1 -- 172.21.15.89:0/368979090 shutdown_connections 2024-04-02T09:12:36.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.338+0000 7fa832800700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa80c06adb0 0x7fa80c06d270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.338+0000 7fa832800700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa82c1611b0 0x7fa82c157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.338+0000 7fa832800700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa82c0cee10 0x7fa82c0b7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.338+0000 7fa832800700 1 --2- 172.21.15.89:0/368979090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa82c15f4d0 0x7fa82c0cce00 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.338+0000 7fa832800700 1 -- 172.21.15.89:0/368979090 >> 172.21.15.89:0/368979090 conn(0x7fa82c08aff0 msgr2=0x7fa82c0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:36.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.338+0000 7fa832800700 1 -- 172.21.15.89:0/368979090 shutdown_connections 2024-04-02T09:12:36.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.338+0000 7fa832800700 1 -- 172.21.15.89:0/368979090 wait complete. 2024-04-02T09:12:36.354 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T09:12:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.546+0000 7f4512c00700 1 Processor -- start 2024-04-02T09:12:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.546+0000 7f4512c00700 1 -- start start 2024-04-02T09:12:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.547+0000 7f4512c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f450c160f10 0x7f450c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.547+0000 7f4512c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f450c0cf230 0x7f450c0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.547+0000 7f4512c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f450c161590 0x7f450c161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.547+0000 7f4512c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f450c161f60 con 0x7f450c0cf230 2024-04-02T09:12:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.547+0000 7f4512c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f450c0ceda0 con 0x7f450c161590 2024-04-02T09:12:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.547+0000 7f4512c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450c0d23d0 con 0x7f450c160f10 2024-04-02T09:12:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.547+0000 7f450b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f450c0cf230 0x7f450c0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.547+0000 7f450b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f450c0cf230 0x7f450c0d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52606/0 (socket says 172.21.15.89:52606) 2024-04-02T09:12:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.547+0000 7f450b400700 1 -- 172.21.15.89:0/1062865673 learned_addr learned my addr 172.21.15.89:0/1062865673 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.548+0000 7f4510c00700 1 --2- 172.21.15.89:0/1062865673 >> v2:172.21.15.113:3300/0 conn(0x7f450c161590 0x7f450c161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.548+0000 7f450be00700 1 --2- 172.21.15.89:0/1062865673 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f450c160f10 0x7f450c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.548+0000 7f450b400700 1 -- 172.21.15.89:0/1062865673 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f450c160f10 msgr2=0x7f450c0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.548+0000 7f450b400700 1 --2- 172.21.15.89:0/1062865673 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f450c160f10 0x7f450c0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.548+0000 7f450b400700 1 -- 172.21.15.89:0/1062865673 >> v2:172.21.15.113:3300/0 conn(0x7f450c161590 msgr2=0x7f450c161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.548+0000 7f450b400700 1 --2- 172.21.15.89:0/1062865673 >> v2:172.21.15.113:3300/0 conn(0x7f450c161590 0x7f450c161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.548+0000 7f450b400700 1 -- 172.21.15.89:0/1062865673 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c1612f0 con 0x7f450c0cf230 2024-04-02T09:12:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.548+0000 7f450b400700 1 --2- 172.21.15.89:0/1062865673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f450c0cf230 0x7f450c0d1e90 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f450c0cfad0 tx=0x7f44f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e8c61ffeae5a92 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.548+0000 7f450aa00700 1 -- 172.21.15.89:0/1062865673 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f8011830 con 0x7f450c0cf230 2024-04-02T09:12:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.548+0000 7f450aa00700 1 -- 172.21.15.89:0/1062865673 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f8011990 con 0x7f450c0cf230 2024-04-02T09:12:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.548+0000 7f450aa00700 1 -- 172.21.15.89:0/1062865673 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f8011d20 con 0x7f450c0cf230 2024-04-02T09:12:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.549+0000 7f4512c00700 1 -- 172.21.15.89:0/1062865673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f450c0cf230 msgr2=0x7f450c0d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.549+0000 7f4512c00700 1 --2- 172.21.15.89:0/1062865673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f450c0cf230 0x7f450c0d1e90 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f450c0cfad0 tx=0x7f44f8009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.549+0000 7f4512c00700 1 -- 172.21.15.89:0/1062865673 shutdown_connections 2024-04-02T09:12:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.549+0000 7f4512c00700 1 --2- 172.21.15.89:0/1062865673 >> v2:172.21.15.113:3300/0 conn(0x7f450c161590 0x7f450c161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.549+0000 7f4512c00700 1 --2- 172.21.15.89:0/1062865673 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f450c160f10 0x7f450c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.549+0000 7f4512c00700 1 --2- 172.21.15.89:0/1062865673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f450c0cf230 0x7f450c0d1e90 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.549+0000 7f4512c00700 1 -- 172.21.15.89:0/1062865673 >> 172.21.15.89:0/1062865673 conn(0x7f450c08aff0 msgr2=0x7f450c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.549+0000 7f4512c00700 1 -- 172.21.15.89:0/1062865673 shutdown_connections 2024-04-02T09:12:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.549+0000 7f4512c00700 1 -- 172.21.15.89:0/1062865673 wait complete. 2024-04-02T09:12:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.551+0000 7f4512c00700 1 Processor -- start 2024-04-02T09:12:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.551+0000 7f4512c00700 1 -- start start 2024-04-02T09:12:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.552+0000 7f4512c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f450c0cf230 0x7f450c1e9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.552+0000 7f4512c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f450c160f10 0x7f450c1e77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.552+0000 7f4512c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f450c161590 0x7f450c1f62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.552+0000 7f4512c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c1e86f0 con 0x7f450c0cf230 2024-04-02T09:12:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.552+0000 7f4512c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c1e8130 con 0x7f450c161590 2024-04-02T09:12:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.552+0000 7f4512c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450c1e8410 con 0x7f450c160f10 2024-04-02T09:12:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.552+0000 7f450be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f450c0cf230 0x7f450c1e9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.552+0000 7f450be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f450c0cf230 0x7f450c1e9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52626/0 (socket says 172.21.15.89:52626) 2024-04-02T09:12:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.552+0000 7f450be00700 1 -- 172.21.15.89:0/1543463762 learned_addr learned my addr 172.21.15.89:0/1543463762 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.552+0000 7f450b400700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f450c160f10 0x7f450c1e77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.552+0000 7f4510c00700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f450c161590 0x7f450c1f62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.553+0000 7f450be00700 1 -- 172.21.15.89:0/1543463762 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f450c160f10 msgr2=0x7f450c1e77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.553+0000 7f450be00700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f450c160f10 0x7f450c1e77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.553+0000 7f450be00700 1 -- 172.21.15.89:0/1543463762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f450c161590 msgr2=0x7f450c1f62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.553+0000 7f450be00700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f450c161590 0x7f450c1f62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.553+0000 7f450be00700 1 -- 172.21.15.89:0/1543463762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f8009550 con 0x7f450c0cf230 2024-04-02T09:12:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.553+0000 7f4510c00700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f450c161590 0x7f450c1f62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.553+0000 7f450be00700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f450c0cf230 0x7f450c1e9030 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f44fc00e870 tx=0x7f44fc00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.553+0000 7f4503e00700 1 -- 172.21.15.89:0/1543463762 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc009590 con 0x7f450c0cf230 2024-04-02T09:12:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.553+0000 7f4503e00700 1 -- 172.21.15.89:0/1543463762 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44fc013680 con 0x7f450c0cf230 2024-04-02T09:12:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.553+0000 7f4512c00700 1 -- 172.21.15.89:0/1543463762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c1454b0 con 0x7f450c0cf230 2024-04-02T09:12:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.553+0000 7f4503e00700 1 -- 172.21.15.89:0/1543463762 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc0139f0 con 0x7f450c0cf230 2024-04-02T09:12:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.553+0000 7f4512c00700 1 -- 172.21.15.89:0/1543463762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c1e9570 con 0x7f450c0cf230 2024-04-02T09:12:36.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.554+0000 7f4512c00700 1 -- 172.21.15.89:0/1543463762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c067720 con 0x7f450c0cf230 2024-04-02T09:12:36.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.555+0000 7f4503e00700 1 -- 172.21.15.89:0/1543463762 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f44fc013b50 con 0x7f450c0cf230 2024-04-02T09:12:36.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.556+0000 7f4503e00700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f44ec06adb0 0x7f44ec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.556+0000 7f4503e00700 1 -- 172.21.15.89:0/1543463762 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f44fc089590 con 0x7f450c0cf230 2024-04-02T09:12:36.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.556+0000 7f450b400700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f44ec06adb0 0x7f44ec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.557+0000 7f450b400700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f44ec06adb0 0x7f44ec06d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f44f8014040 tx=0x7f44f8008030 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.560+0000 7f4503e00700 1 -- 172.21.15.89:0/1543463762 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f44fc054490 con 0x7f450c0cf230 2024-04-02T09:12:36.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.763+0000 7f4512c00700 1 -- 172.21.15.89:0/1543463762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f450c06a920 con 0x7f450c0cf230 2024-04-02T09:12:36.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.763+0000 7f4503e00700 1 -- 172.21.15.89:0/1543463762 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v223) v1 ==== 84+0+209 (secure 0 0 0) 0x7f44fc057ab0 con 0x7f450c0cf230 2024-04-02T09:12:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:36.762 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T09:12:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T09:12:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T09:12:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 26, 2024-04-02T09:12:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": "cephfs_metadata" 2024-04-02T09:12:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 27, 2024-04-02T09:12:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": "cephfs_data" 2024-04-02T09:12:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:36.763 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T09:12:36.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.765+0000 7f4512c00700 1 -- 172.21.15.89:0/1543463762 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f44ec06adb0 msgr2=0x7f44ec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.765+0000 7f4512c00700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f44ec06adb0 0x7f44ec06d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f44f8014040 tx=0x7f44f8008030 comp rx=0 tx=0).stop 2024-04-02T09:12:36.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.765+0000 7f4512c00700 1 -- 172.21.15.89:0/1543463762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f450c0cf230 msgr2=0x7f450c1e9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.765+0000 7f4512c00700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f450c0cf230 0x7f450c1e9030 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f44fc00e870 tx=0x7f44fc00eb80 comp rx=0 tx=0).stop 2024-04-02T09:12:36.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.766+0000 7f4512c00700 1 -- 172.21.15.89:0/1543463762 shutdown_connections 2024-04-02T09:12:36.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.766+0000 7f4512c00700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f44ec06adb0 0x7f44ec06d270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.766+0000 7f4512c00700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f450c161590 0x7f450c1f62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.766+0000 7f4512c00700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f450c0cf230 0x7f450c1e9030 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.766+0000 7f4512c00700 1 --2- 172.21.15.89:0/1543463762 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f450c160f10 0x7f450c1e77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.766+0000 7f4512c00700 1 -- 172.21.15.89:0/1543463762 >> 172.21.15.89:0/1543463762 conn(0x7f450c08aff0 msgr2=0x7f450c15ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:36.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.766+0000 7f4512c00700 1 -- 172.21.15.89:0/1543463762 shutdown_connections 2024-04-02T09:12:36.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.766+0000 7f4512c00700 1 -- 172.21.15.89:0/1543463762 wait complete. 2024-04-02T09:12:36.782 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-04-02T09:12:36.782 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-04-02T09:12:36.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.978+0000 7fa154200700 1 Processor -- start 2024-04-02T09:12:36.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.979+0000 7fa154200700 1 -- start start 2024-04-02T09:12:36.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.980+0000 7fa154200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c160e10 0x7fa14c099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.980+0000 7fa154200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa14c0cbc20 0x7fa14c0cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.980+0000 7fa154200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c09ada0 con 0x7fa14c0b2380 2024-04-02T09:12:36.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.980+0000 7fa154200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa14c1611f0 con 0x7fa14c0cbc20 2024-04-02T09:12:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.980+0000 7fa154200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c06cc80 con 0x7fa14c160e10 2024-04-02T09:12:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.980+0000 7fa150e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa14c0b2380 0x7fa14c099fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52792/0 (socket says 172.21.15.89:52792) 2024-04-02T09:12:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.980+0000 7fa150e00700 1 -- 172.21.15.89:0/3542945497 learned_addr learned my addr 172.21.15.89:0/3542945497 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.980+0000 7fa151800700 1 --2- 172.21.15.89:0/3542945497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c160e10 0x7fa14c099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.980+0000 7fa152200700 1 --2- 172.21.15.89:0/3542945497 >> v2:172.21.15.113:3300/0 conn(0x7fa14c0cbc20 0x7fa14c0cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.980+0000 7fa14be00700 1 -- 172.21.15.89:0/3542945497 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222070759 0 0) 0x7fa14c09ada0 con 0x7fa14c0b2380 2024-04-02T09:12:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 -- 172.21.15.89:0/3542945497 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1340035b0 con 0x7fa14c0b2380 2024-04-02T09:12:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 -- 172.21.15.89:0/3542945497 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3511702110 0 0) 0x7fa1340035b0 con 0x7fa14c0b2380 2024-04-02T09:12:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 -- 172.21.15.89:0/3542945497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c160e10 msgr2=0x7fa14c099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 --2- 172.21.15.89:0/3542945497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c160e10 0x7fa14c099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 -- 172.21.15.89:0/3542945497 >> v2:172.21.15.113:3300/0 conn(0x7fa14c0cbc20 msgr2=0x7fa14c0cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 --2- 172.21.15.89:0/3542945497 >> v2:172.21.15.113:3300/0 conn(0x7fa14c0cbc20 0x7fa14c0cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 -- 172.21.15.89:0/3542945497 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c0d0840 con 0x7fa14c0b2380 2024-04-02T09:12:36.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 -- 172.21.15.89:0/3542945497 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fa138001670 con 0x7fa14c0b2380 2024-04-02T09:12:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 -- 172.21.15.89:0/3542945497 >> v1:172.21.15.89:6789/0 conn(0x7fa14c0b2380 legacy=0x7fa14c099fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 --2- 172.21.15.89:0/3542945497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa134003a00 0x7fa134003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 --2- 172.21.15.89:0/3542945497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa134004850 0x7fa134004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 --2- 172.21.15.89:0/3542945497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c160e10 0x7fa134005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 -- 172.21.15.89:0/3542945497 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c1611f0 con 0x7fa134003a00 2024-04-02T09:12:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 -- 172.21.15.89:0/3542945497 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c06cc80 con 0x7fa134004850 2024-04-02T09:12:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa14be00700 1 -- 172.21.15.89:0/3542945497 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13400ba10 con 0x7fa14c160e10 2024-04-02T09:12:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa151800700 1 --2- 172.21.15.89:0/3542945497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa134003a00 0x7fa134003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.981+0000 7fa150e00700 1 --2- 172.21.15.89:0/3542945497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa134004850 0x7fa134004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.982+0000 7fa151800700 1 -- 172.21.15.89:0/3542945497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c160e10 msgr2=0x7fa134005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.982+0000 7fa152200700 1 --2- 172.21.15.89:0/3542945497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c160e10 0x7fa134005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.982+0000 7fa151800700 1 --2- 172.21.15.89:0/3542945497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c160e10 0x7fa134005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.982+0000 7fa151800700 1 -- 172.21.15.89:0/3542945497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa134004850 msgr2=0x7fa134004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:36.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.982+0000 7fa151800700 1 --2- 172.21.15.89:0/3542945497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa134004850 0x7fa134004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.982+0000 7fa151800700 1 -- 172.21.15.89:0/3542945497 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1340035b0 con 0x7fa134003a00 2024-04-02T09:12:36.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.982+0000 7fa152200700 1 --2- 172.21.15.89:0/3542945497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c160e10 0x7fa134005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:36.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.982+0000 7fa151800700 1 --2- 172.21.15.89:0/3542945497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa134003a00 0x7fa134003e20 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7fa140009b90 tx=0x7fa134005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178e7950b743de3b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:36.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.982+0000 7fa14be00700 1 -- 172.21.15.89:0/3542945497 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14000e6c0 con 0x7fa134003a00 2024-04-02T09:12:36.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.982+0000 7fa14be00700 1 -- 172.21.15.89:0/3542945497 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14000e820 con 0x7fa134003a00 2024-04-02T09:12:36.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.982+0000 7fa14be00700 1 -- 172.21.15.89:0/3542945497 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14000eb90 con 0x7fa134003a00 2024-04-02T09:12:36.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.983+0000 7fa154200700 1 -- 172.21.15.89:0/3542945497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa134003a00 msgr2=0x7fa134003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.983+0000 7fa154200700 1 --2- 172.21.15.89:0/3542945497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa134003a00 0x7fa134003e20 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7fa140009b90 tx=0x7fa134005880 comp rx=0 tx=0).stop 2024-04-02T09:12:36.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.983+0000 7fa152200700 1 -- 172.21.15.89:0/3542945497 reap_dead start 2024-04-02T09:12:36.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.983+0000 7fa154200700 1 -- 172.21.15.89:0/3542945497 shutdown_connections 2024-04-02T09:12:36.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.983+0000 7fa154200700 1 -- 172.21.15.89:0/3542945497 >> 172.21.15.89:0/3542945497 conn(0x7fa14c0b2e20 msgr2=0x7fa14c0b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:36.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.983+0000 7fa154200700 1 -- 172.21.15.89:0/3542945497 shutdown_connections 2024-04-02T09:12:36.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.983+0000 7fa154200700 1 -- 172.21.15.89:0/3542945497 wait complete. 2024-04-02T09:12:36.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.985+0000 7fa154200700 1 Processor -- start 2024-04-02T09:12:36.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.985+0000 7fa154200700 1 -- start start 2024-04-02T09:12:36.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa154200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa134003a00 0x7fa14c099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa154200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa14c09a180 0x7fa14c0c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa154200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c1f6d80 0x7fa14c15f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa154200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c0c79d0 con 0x7fa14c09a180 2024-04-02T09:12:36.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa154200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c1f62a0 con 0x7fa134003a00 2024-04-02T09:12:36.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa154200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c1f6580 con 0x7fa14c1f6d80 2024-04-02T09:12:36.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa150e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa14c09a180 0x7fa14c0c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa150e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa14c09a180 0x7fa14c0c6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52644/0 (socket says 172.21.15.89:52644) 2024-04-02T09:12:36.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa150e00700 1 -- 172.21.15.89:0/1530862200 learned_addr learned my addr 172.21.15.89:0/1530862200 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:36.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa151800700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa134003a00 0x7fa14c099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa152200700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c1f6d80 0x7fa14c15f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa150e00700 1 -- 172.21.15.89:0/1530862200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c1f6d80 msgr2=0x7fa14c15f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa150e00700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c1f6d80 0x7fa14c15f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa150e00700 1 -- 172.21.15.89:0/1530862200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa134003a00 msgr2=0x7fa14c099c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa150e00700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa134003a00 0x7fa14c099c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa150e00700 1 -- 172.21.15.89:0/1530862200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c1f6860 con 0x7fa14c09a180 2024-04-02T09:12:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.987+0000 7fa152200700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c1f6d80 0x7fa14c15f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.988+0000 7fa150e00700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa14c09a180 0x7fa14c0c6f10 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7fa138007310 tx=0x7fa138007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.988+0000 7fa149600700 1 -- 172.21.15.89:0/1530862200 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13800f5e0 con 0x7fa14c09a180 2024-04-02T09:12:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.988+0000 7fa149600700 1 -- 172.21.15.89:0/1530862200 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13800f760 con 0x7fa14c09a180 2024-04-02T09:12:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.988+0000 7fa149600700 1 -- 172.21.15.89:0/1530862200 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13800fb10 con 0x7fa14c09a180 2024-04-02T09:12:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.988+0000 7fa154200700 1 -- 172.21.15.89:0/1530862200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14c150f90 con 0x7fa14c09a180 2024-04-02T09:12:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.988+0000 7fa154200700 1 -- 172.21.15.89:0/1530862200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14c0c7630 con 0x7fa14c09a180 2024-04-02T09:12:36.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.989+0000 7fa154200700 1 -- 172.21.15.89:0/1530862200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa14c15d0f0 con 0x7fa14c09a180 2024-04-02T09:12:36.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.990+0000 7fa149600700 1 -- 172.21.15.89:0/1530862200 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa13800fc70 con 0x7fa14c09a180 2024-04-02T09:12:36.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.991+0000 7fa149600700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa12c06adb0 0x7fa12c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:36.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.991+0000 7fa149600700 1 -- 172.21.15.89:0/1530862200 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fa1380895e0 con 0x7fa14c09a180 2024-04-02T09:12:36.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.991+0000 7fa151800700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa12c06adb0 0x7fa12c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:36.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.992+0000 7fa151800700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa12c06adb0 0x7fa12c06d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa1400090b0 tx=0x7fa140008030 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:36.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:36.995+0000 7fa149600700 1 -- 172.21.15.89:0/1530862200 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa138054500 con 0x7fa14c09a180 2024-04-02T09:12:37.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.195+0000 7fa154200700 1 -- 172.21.15.89:0/1530862200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa14c067720 con 0x7fa14c09a180 2024-04-02T09:12:37.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.609+0000 7fa149600700 1 -- 172.21.15.89:0/1530862200 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v224) v1 ==== 144+0+0 (secure 0 0 0) 0x7fa138057b20 con 0x7fa14c09a180 2024-04-02T09:12:37.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.615+0000 7fa154200700 1 -- 172.21.15.89:0/1530862200 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa12c06adb0 msgr2=0x7fa12c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:37.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.615+0000 7fa154200700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa12c06adb0 0x7fa12c06d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa1400090b0 tx=0x7fa140008030 comp rx=0 tx=0).stop 2024-04-02T09:12:37.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.615+0000 7fa154200700 1 -- 172.21.15.89:0/1530862200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa14c09a180 msgr2=0x7fa14c0c6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:37.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.615+0000 7fa154200700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa14c09a180 0x7fa14c0c6f10 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7fa138007310 tx=0x7fa138007620 comp rx=0 tx=0).stop 2024-04-02T09:12:37.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.615+0000 7fa154200700 1 -- 172.21.15.89:0/1530862200 shutdown_connections 2024-04-02T09:12:37.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.615+0000 7fa154200700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa12c06adb0 0x7fa12c06d270 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:37.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.615+0000 7fa154200700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa134003a00 0x7fa14c099c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:37.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.615+0000 7fa154200700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa14c1f6d80 0x7fa14c15f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:37.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.615+0000 7fa154200700 1 --2- 172.21.15.89:0/1530862200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa14c09a180 0x7fa14c0c6f10 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:37.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.615+0000 7fa154200700 1 -- 172.21.15.89:0/1530862200 >> 172.21.15.89:0/1530862200 conn(0x7fa14c0b2e20 msgr2=0x7fa14c06efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:37.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.615+0000 7fa154200700 1 -- 172.21.15.89:0/1530862200 shutdown_connections 2024-04-02T09:12:37.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.616+0000 7fa154200700 1 -- 172.21.15.89:0/1530862200 wait complete. 2024-04-02T09:12:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'backup_fs_metadata' created 2024-04-02T09:12:37.634 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T09:12:37.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.834+0000 7fa06b800700 1 Processor -- start 2024-04-02T09:12:37.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.834+0000 7fa06b800700 1 -- start start 2024-04-02T09:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.835+0000 7fa06b800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa06406fa20 0x7fa06415e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.835+0000 7fa06b800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa06415cfe0 0x7fa06415d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.835+0000 7fa06b800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa06415eda0 con 0x7fa0640cd5c0 2024-04-02T09:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.835+0000 7fa06b800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa064099dd0 con 0x7fa06415cfe0 2024-04-02T09:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.835+0000 7fa06b800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06409a0b0 con 0x7fa06406fa20 2024-04-02T09:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.836+0000 7fa069800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa06415cfe0 0x7fa06415d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.836+0000 7fa069800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa06415cfe0 0x7fa06415d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52660/0 (socket says 172.21.15.89:52660) 2024-04-02T09:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.836+0000 7fa069800700 1 -- 172.21.15.89:0/4030309144 learned_addr learned my addr 172.21.15.89:0/4030309144 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.836+0000 7fa063e00700 1 --2- 172.21.15.89:0/4030309144 >> v2:172.21.15.113:3300/0 conn(0x7fa06406fa20 0x7fa06415e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.836+0000 7fa069800700 1 -- 172.21.15.89:0/4030309144 >> v2:172.21.15.113:3300/0 conn(0x7fa06406fa20 msgr2=0x7fa06415e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:37.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.836+0000 7fa069800700 1 --2- 172.21.15.89:0/4030309144 >> v2:172.21.15.113:3300/0 conn(0x7fa06406fa20 0x7fa06415e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:37.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.836+0000 7fa069800700 1 -- 172.21.15.89:0/4030309144 >> v1:172.21.15.89:6789/0 conn(0x7fa0640cd5c0 legacy=0x7fa06406f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:37.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.836+0000 7fa069800700 1 -- 172.21.15.89:0/4030309144 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06409a390 con 0x7fa06415cfe0 2024-04-02T09:12:37.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.836+0000 7fa069800700 1 --2- 172.21.15.89:0/4030309144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa06415cfe0 0x7fa06415d450 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7fa05c009e50 tx=0x7fa05c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a9fb2206a8bfa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:37.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.837+0000 7fa063400700 1 -- 172.21.15.89:0/4030309144 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c00e070 con 0x7fa06415cfe0 2024-04-02T09:12:37.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.837+0000 7fa063400700 1 -- 172.21.15.89:0/4030309144 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05c00b8d0 con 0x7fa06415cfe0 2024-04-02T09:12:37.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.837+0000 7fa063400700 1 -- 172.21.15.89:0/4030309144 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c00bc80 con 0x7fa06415cfe0 2024-04-02T09:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.837+0000 7fa06b800700 1 -- 172.21.15.89:0/4030309144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa06415cfe0 msgr2=0x7fa06415d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.837+0000 7fa06b800700 1 --2- 172.21.15.89:0/4030309144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa06415cfe0 0x7fa06415d450 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7fa05c009e50 tx=0x7fa05c00b330 comp rx=0 tx=0).stop 2024-04-02T09:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.837+0000 7fa06b800700 1 -- 172.21.15.89:0/4030309144 shutdown_connections 2024-04-02T09:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.837+0000 7fa06b800700 1 --2- 172.21.15.89:0/4030309144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa06415cfe0 0x7fa06415d450 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.837+0000 7fa06b800700 1 --2- 172.21.15.89:0/4030309144 >> v2:172.21.15.113:3300/0 conn(0x7fa06406fa20 0x7fa06415e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.837+0000 7fa06b800700 1 -- 172.21.15.89:0/4030309144 >> 172.21.15.89:0/4030309144 conn(0x7fa0640d1370 msgr2=0x7fa0640d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.837+0000 7fa06b800700 1 -- 172.21.15.89:0/4030309144 shutdown_connections 2024-04-02T09:12:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.837+0000 7fa06b800700 1 -- 172.21.15.89:0/4030309144 wait complete. 2024-04-02T09:12:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.840+0000 7fa06b800700 1 Processor -- start 2024-04-02T09:12:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.840+0000 7fa06b800700 1 -- start start 2024-04-02T09:12:37.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.841+0000 7fa06b800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa06406fa20 0x7fa0641f3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.841+0000 7fa06b800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0640cd5c0 0x7fa064155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.841+0000 7fa06b800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa06415cfe0 0x7fa0640d0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.841+0000 7fa06b800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0641574a0 con 0x7fa0640cd5c0 2024-04-02T09:12:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.841+0000 7fa06b800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa064157720 con 0x7fa06415cfe0 2024-04-02T09:12:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.841+0000 7fa06b800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa064157a00 con 0x7fa06406fa20 2024-04-02T09:12:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.841+0000 7fa063e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0640cd5c0 0x7fa064155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.841+0000 7fa063e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0640cd5c0 0x7fa064155900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52664/0 (socket says 172.21.15.89:52664) 2024-04-02T09:12:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.841+0000 7fa063e00700 1 -- 172.21.15.89:0/3692027032 learned_addr learned my addr 172.21.15.89:0/3692027032 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.841+0000 7fa068e00700 1 --2- 172.21.15.89:0/3692027032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa06406fa20 0x7fa0641f3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.841+0000 7fa069800700 1 --2- 172.21.15.89:0/3692027032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa06415cfe0 0x7fa0640d0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.842+0000 7fa063e00700 1 -- 172.21.15.89:0/3692027032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa06406fa20 msgr2=0x7fa0641f3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.842+0000 7fa063e00700 1 --2- 172.21.15.89:0/3692027032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa06406fa20 0x7fa0641f3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:37.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.842+0000 7fa063e00700 1 -- 172.21.15.89:0/3692027032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa06415cfe0 msgr2=0x7fa0640d0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:37.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.842+0000 7fa063e00700 1 --2- 172.21.15.89:0/3692027032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa06415cfe0 0x7fa0640d0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:37.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.842+0000 7fa063e00700 1 -- 172.21.15.89:0/3692027032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c009b30 con 0x7fa0640cd5c0 2024-04-02T09:12:37.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.842+0000 7fa063e00700 1 --2- 172.21.15.89:0/3692027032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0640cd5c0 0x7fa064155900 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fa05000e980 tx=0x7fa05000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:37.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.842+0000 7fa060c00700 1 -- 172.21.15.89:0/3692027032 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa050009710 con 0x7fa0640cd5c0 2024-04-02T09:12:37.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.842+0000 7fa060c00700 1 -- 172.21.15.89:0/3692027032 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05000c690 con 0x7fa0640cd5c0 2024-04-02T09:12:37.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.842+0000 7fa060c00700 1 -- 172.21.15.89:0/3692027032 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05000ca20 con 0x7fa0640cd5c0 2024-04-02T09:12:37.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.842+0000 7fa06b800700 1 -- 172.21.15.89:0/3692027032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa064157ce0 con 0x7fa0640cd5c0 2024-04-02T09:12:37.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.842+0000 7fa06b800700 1 -- 172.21.15.89:0/3692027032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0641582a0 con 0x7fa0640cd5c0 2024-04-02T09:12:37.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.843+0000 7fa06b800700 1 -- 172.21.15.89:0/3692027032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa064068240 con 0x7fa0640cd5c0 2024-04-02T09:12:37.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.845+0000 7fa060c00700 1 -- 172.21.15.89:0/3692027032 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa05000cb80 con 0x7fa0640cd5c0 2024-04-02T09:12:37.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.846+0000 7fa060c00700 1 --2- 172.21.15.89:0/3692027032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa04406adb0 0x7fa04406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:37.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.846+0000 7fa060c00700 1 -- 172.21.15.89:0/3692027032 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 6530+0+0 (secure 0 0 0) 0x7fa050089fe0 con 0x7fa0640cd5c0 2024-04-02T09:12:37.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.846+0000 7fa068e00700 1 --2- 172.21.15.89:0/3692027032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa04406adb0 0x7fa04406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:37.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.847+0000 7fa068e00700 1 --2- 172.21.15.89:0/3692027032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa04406adb0 0x7fa04406d270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa0580018a0 tx=0x7fa058009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:37.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:37.850+0000 7fa060c00700 1 -- 172.21.15.89:0/3692027032 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa050054d50 con 0x7fa0640cd5c0 2024-04-02T09:12:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.049+0000 7fa06b800700 1 -- 172.21.15.89:0/3692027032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa064067720 con 0x7fa0640cd5c0 2024-04-02T09:12:38.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.611+0000 7fa060c00700 1 -- 172.21.15.89:0/3692027032 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v225) v1 ==== 176+0+0 (secure 0 0 0) 0x7fa0500170d0 con 0x7fa0640cd5c0 2024-04-02T09:12:38.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.617+0000 7fa06b800700 1 -- 172.21.15.89:0/3692027032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa04406adb0 msgr2=0x7fa04406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:38.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.617+0000 7fa06b800700 1 --2- 172.21.15.89:0/3692027032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa04406adb0 0x7fa04406d270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa0580018a0 tx=0x7fa058009040 comp rx=0 tx=0).stop 2024-04-02T09:12:38.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.617+0000 7fa06b800700 1 -- 172.21.15.89:0/3692027032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0640cd5c0 msgr2=0x7fa064155900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:38.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.617+0000 7fa06b800700 1 --2- 172.21.15.89:0/3692027032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0640cd5c0 0x7fa064155900 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fa05000e980 tx=0x7fa05000ec90 comp rx=0 tx=0).stop 2024-04-02T09:12:38.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.618+0000 7fa06b800700 1 -- 172.21.15.89:0/3692027032 shutdown_connections 2024-04-02T09:12:38.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.618+0000 7fa06b800700 1 --2- 172.21.15.89:0/3692027032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa04406adb0 0x7fa04406d270 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa0580018a0 tx=0x7fa058009040 comp rx=0 tx=0).stop 2024-04-02T09:12:38.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.618+0000 7fa06b800700 1 --2- 172.21.15.89:0/3692027032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa06415cfe0 0x7fa0640d0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:38.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.618+0000 7fa06b800700 1 --2- 172.21.15.89:0/3692027032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa06406fa20 0x7fa0641f3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:38.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.618+0000 7fa06b800700 1 --2- 172.21.15.89:0/3692027032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0640cd5c0 0x7fa064155900 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:38.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.618+0000 7fa06b800700 1 -- 172.21.15.89:0/3692027032 >> 172.21.15.89:0/3692027032 conn(0x7fa0640d1370 msgr2=0x7fa06415c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:38.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.618+0000 7fa06b800700 1 -- 172.21.15.89:0/3692027032 shutdown_connections 2024-04-02T09:12:38.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.619+0000 7fa06b800700 1 -- 172.21.15.89:0/3692027032 wait complete. 2024-04-02T09:12:38.621 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'backup_fs_data' created 2024-04-02T09:12:38.649 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-04-02T09:12:38.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.844+0000 7fb839200700 1 Processor -- start 2024-04-02T09:12:38.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.844+0000 7fb839200700 1 -- start start 2024-04-02T09:12:38.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.845+0000 7fb839200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8340cd5c0 0x7fb83415e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:38.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.845+0000 7fb839200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb83406f310 0x7fb83406f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:38.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.845+0000 7fb839200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83415eda0 con 0x7fb83415cfe0 2024-04-02T09:12:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.845+0000 7fb839200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83406fcb0 con 0x7fb83406f310 2024-04-02T09:12:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.845+0000 7fb839200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb834099dd0 con 0x7fb8340cd5c0 2024-04-02T09:12:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.846+0000 7fb832800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb83415cfe0 0x7fb83415d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:54624/0 (socket says 172.21.15.89:54624) 2024-04-02T09:12:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.846+0000 7fb832800700 1 -- 172.21.15.89:0/2622340640 learned_addr learned my addr 172.21.15.89:0/2622340640 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.846+0000 7fb831e00700 1 --2- 172.21.15.89:0/2622340640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8340cd5c0 0x7fb83415e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.846+0000 7fb831400700 1 --2- 172.21.15.89:0/2622340640 >> v2:172.21.15.113:3300/0 conn(0x7fb83406f310 0x7fb83406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.846+0000 7fb830a00700 1 -- 172.21.15.89:0/2622340640 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625302413 0 0) 0x7fb82c001490 con 0x7fb83415cfe0 2024-04-02T09:12:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.846+0000 7fb830a00700 1 -- 172.21.15.89:0/2622340640 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8180035b0 con 0x7fb83415cfe0 2024-04-02T09:12:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.846+0000 7fb831e00700 1 -- 172.21.15.89:0/2622340640 >> v2:172.21.15.113:3300/0 conn(0x7fb83406f310 msgr2=0x7fb83406f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.846+0000 7fb831e00700 1 --2- 172.21.15.89:0/2622340640 >> v2:172.21.15.113:3300/0 conn(0x7fb83406f310 0x7fb83406f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.846+0000 7fb831e00700 1 -- 172.21.15.89:0/2622340640 >> v1:172.21.15.89:6789/0 conn(0x7fb83415cfe0 legacy=0x7fb83415d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.846+0000 7fb830a00700 1 -- 172.21.15.89:0/2622340640 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3570530651 0 0) 0x7fb8180035b0 con 0x7fb83415cfe0 2024-04-02T09:12:38.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.846+0000 7fb831e00700 1 -- 172.21.15.89:0/2622340640 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83409a0b0 con 0x7fb8340cd5c0 2024-04-02T09:12:38.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.847+0000 7fb831e00700 1 --2- 172.21.15.89:0/2622340640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8340cd5c0 0x7fb83415e860 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb824009840 tx=0x7fb824009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8960d1844135749d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:38.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.847+0000 7fb830a00700 1 -- 172.21.15.89:0/2622340640 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c001b80 con 0x7fb8340cd5c0 2024-04-02T09:12:38.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.847+0000 7fb830a00700 1 -- 172.21.15.89:0/2622340640 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb824004030 con 0x7fb8340cd5c0 2024-04-02T09:12:38.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.847+0000 7fb830a00700 1 -- 172.21.15.89:0/2622340640 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8240043a0 con 0x7fb8340cd5c0 2024-04-02T09:12:38.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.847+0000 7fb839200700 1 -- 172.21.15.89:0/2622340640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8340cd5c0 msgr2=0x7fb83415e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:38.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.847+0000 7fb839200700 1 --2- 172.21.15.89:0/2622340640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8340cd5c0 0x7fb83415e860 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb824009840 tx=0x7fb824009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:38.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.847+0000 7fb839200700 1 -- 172.21.15.89:0/2622340640 shutdown_connections 2024-04-02T09:12:38.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.847+0000 7fb839200700 1 --2- 172.21.15.89:0/2622340640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8340cd5c0 0x7fb83415e860 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:38.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.847+0000 7fb839200700 1 --2- 172.21.15.89:0/2622340640 >> v2:172.21.15.113:3300/0 conn(0x7fb83406f310 0x7fb83406f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:38.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.847+0000 7fb839200700 1 -- 172.21.15.89:0/2622340640 >> 172.21.15.89:0/2622340640 conn(0x7fb8340d1370 msgr2=0x7fb8340d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:38.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.847+0000 7fb839200700 1 -- 172.21.15.89:0/2622340640 shutdown_connections 2024-04-02T09:12:38.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.848+0000 7fb839200700 1 -- 172.21.15.89:0/2622340640 wait complete. 2024-04-02T09:12:38.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.849+0000 7fb839200700 1 Processor -- start 2024-04-02T09:12:38.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.849+0000 7fb839200700 1 -- start start 2024-04-02T09:12:38.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.850+0000 7fb839200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb83406f310 0x7fb8340c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:38.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.850+0000 7fb839200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8340cd5c0 0x7fb834160530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:38.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.850+0000 7fb839200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83415cfe0 0x7fb8341515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:38.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.850+0000 7fb839200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83415f490 con 0x7fb83406f310 2024-04-02T09:12:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.850+0000 7fb839200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83415f770 con 0x7fb83415cfe0 2024-04-02T09:12:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.850+0000 7fb839200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83415fa50 con 0x7fb8340cd5c0 2024-04-02T09:12:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.850+0000 7fb831e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb83406f310 0x7fb8340c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.850+0000 7fb831e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb83406f310 0x7fb8340c6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59508/0 (socket says 172.21.15.89:59508) 2024-04-02T09:12:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.851+0000 7fb831e00700 1 -- 172.21.15.89:0/4086859785 learned_addr learned my addr 172.21.15.89:0/4086859785 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.851+0000 7fb832800700 1 --2- 172.21.15.89:0/4086859785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83415cfe0 0x7fb8341515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.851+0000 7fb831400700 1 --2- 172.21.15.89:0/4086859785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8340cd5c0 0x7fb834160530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.851+0000 7fb831e00700 1 -- 172.21.15.89:0/4086859785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8340cd5c0 msgr2=0x7fb834160530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.851+0000 7fb831e00700 1 --2- 172.21.15.89:0/4086859785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8340cd5c0 0x7fb834160530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.851+0000 7fb831e00700 1 -- 172.21.15.89:0/4086859785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83415cfe0 msgr2=0x7fb8341515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.851+0000 7fb831e00700 1 --2- 172.21.15.89:0/4086859785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83415cfe0 0x7fb8341515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.851+0000 7fb831e00700 1 -- 172.21.15.89:0/4086859785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8240094a0 con 0x7fb83406f310 2024-04-02T09:12:38.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.851+0000 7fb831e00700 1 --2- 172.21.15.89:0/4086859785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb83406f310 0x7fb8340c6ec0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7fb824009790 tx=0x7fb824004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:38.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.851+0000 7fb82a000700 1 -- 172.21.15.89:0/4086859785 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824003ea0 con 0x7fb83406f310 2024-04-02T09:12:38.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.851+0000 7fb82a000700 1 -- 172.21.15.89:0/4086859785 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82401f070 con 0x7fb83406f310 2024-04-02T09:12:38.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.851+0000 7fb82a000700 1 -- 172.21.15.89:0/4086859785 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824011870 con 0x7fb83406f310 2024-04-02T09:12:38.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.852+0000 7fb839200700 1 -- 172.21.15.89:0/4086859785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83415fd30 con 0x7fb83406f310 2024-04-02T09:12:38.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.852+0000 7fb839200700 1 -- 172.21.15.89:0/4086859785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb834151b20 con 0x7fb83406f310 2024-04-02T09:12:38.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.853+0000 7fb839200700 1 -- 172.21.15.89:0/4086859785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb834151cb0 con 0x7fb83406f310 2024-04-02T09:12:38.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.854+0000 7fb82a000700 1 -- 172.21.15.89:0/4086859785 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb8240119d0 con 0x7fb83406f310 2024-04-02T09:12:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.855+0000 7fb82a000700 1 --2- 172.21.15.89:0/4086859785 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb81406adb0 0x7fb81406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.855+0000 7fb82a000700 1 -- 172.21.15.89:0/4086859785 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6915+0+0 (secure 0 0 0) 0x7fb824021070 con 0x7fb83406f310 2024-04-02T09:12:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.855+0000 7fb831400700 1 --2- 172.21.15.89:0/4086859785 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb81406adb0 0x7fb81406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:38.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.856+0000 7fb831400700 1 --2- 172.21.15.89:0/4086859785 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb81406adb0 0x7fb81406d270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb83406c9f0 tx=0x7fb820009210 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:38.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:38.859+0000 7fb82a000700 1 -- 172.21.15.89:0/4086859785 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb8240560d0 con 0x7fb83406f310 2024-04-02T09:12:39.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.103+0000 7fb839200700 1 -- 172.21.15.89:0/4086859785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7fb834151cb0 con 0x7fb83406f310 2024-04-02T09:12:39.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.109+0000 7fb82a000700 1 -- 172.21.15.89:0/4086859785 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 28 and data pool 29 v107) v1 ==== 183+0+0 (secure 0 0 0) 0x7fb834151cb0 con 0x7fb83406f310 2024-04-02T09:12:39.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.116+0000 7fb839200700 1 -- 172.21.15.89:0/4086859785 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb81406adb0 msgr2=0x7fb81406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:39.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.116+0000 7fb839200700 1 --2- 172.21.15.89:0/4086859785 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb81406adb0 0x7fb81406d270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb83406c9f0 tx=0x7fb820009210 comp rx=0 tx=0).stop 2024-04-02T09:12:39.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.116+0000 7fb839200700 1 -- 172.21.15.89:0/4086859785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb83406f310 msgr2=0x7fb8340c6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:39.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.116+0000 7fb839200700 1 --2- 172.21.15.89:0/4086859785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb83406f310 0x7fb8340c6ec0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7fb824009790 tx=0x7fb824004750 comp rx=0 tx=0).stop 2024-04-02T09:12:39.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.118+0000 7fb839200700 1 -- 172.21.15.89:0/4086859785 shutdown_connections 2024-04-02T09:12:39.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.118+0000 7fb839200700 1 --2- 172.21.15.89:0/4086859785 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb81406adb0 0x7fb81406d270 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.118+0000 7fb839200700 1 --2- 172.21.15.89:0/4086859785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83415cfe0 0x7fb8341515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.118+0000 7fb839200700 1 --2- 172.21.15.89:0/4086859785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb83406f310 0x7fb8340c6ec0 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.118+0000 7fb839200700 1 --2- 172.21.15.89:0/4086859785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8340cd5c0 0x7fb834160530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.118+0000 7fb839200700 1 -- 172.21.15.89:0/4086859785 >> 172.21.15.89:0/4086859785 conn(0x7fb8340d1370 msgr2=0x7fb83415c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:39.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.118+0000 7fb839200700 1 -- 172.21.15.89:0/4086859785 shutdown_connections 2024-04-02T09:12:39.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.118+0000 7fb839200700 1 -- 172.21.15.89:0/4086859785 wait complete. 2024-04-02T09:12:39.127 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 28 and data pool 29 2024-04-02T09:12:39.152 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:12:39.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.347+0000 7f3169200700 1 Processor -- start 2024-04-02T09:12:39.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.347+0000 7f3169200700 1 -- start start 2024-04-02T09:12:39.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.348+0000 7f3169200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f316415da50 0x7f31640ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:39.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.348+0000 7f3169200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31640cec70 0x7f31640d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:39.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.348+0000 7f3169200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f31640d0b20 0x7f31640b52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:39.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.348+0000 7f3169200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31640b95d0 con 0x7f316415da50 2024-04-02T09:12:39.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.348+0000 7f3169200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f316415de30 con 0x7f31640d0b20 2024-04-02T09:12:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.348+0000 7f3169200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31640d0fa0 con 0x7f31640cec70 2024-04-02T09:12:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.348+0000 7f3161e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f316415da50 0x7f31640ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.348+0000 7f3161e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f316415da50 0x7f31640ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59516/0 (socket says 172.21.15.89:59516) 2024-04-02T09:12:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.348+0000 7f3161e00700 1 -- 172.21.15.89:0/4220366815 learned_addr learned my addr 172.21.15.89:0/4220366815 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.349+0000 7f3161400700 1 --2- 172.21.15.89:0/4220366815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31640cec70 0x7f31640d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.349+0000 7f3162800700 1 --2- 172.21.15.89:0/4220366815 >> v2:172.21.15.113:3300/0 conn(0x7f31640d0b20 0x7f31640b52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.349+0000 7f3161e00700 1 -- 172.21.15.89:0/4220366815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31640cec70 msgr2=0x7f31640d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.349+0000 7f3161e00700 1 --2- 172.21.15.89:0/4220366815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31640cec70 0x7f31640d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.349+0000 7f3161e00700 1 -- 172.21.15.89:0/4220366815 >> v2:172.21.15.113:3300/0 conn(0x7f31640d0b20 msgr2=0x7f31640b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.349+0000 7f3161e00700 1 --2- 172.21.15.89:0/4220366815 >> v2:172.21.15.113:3300/0 conn(0x7f31640d0b20 0x7f31640b52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.349+0000 7f3161e00700 1 -- 172.21.15.89:0/4220366815 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31640d24e0 con 0x7f316415da50 2024-04-02T09:12:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.349+0000 7f3161e00700 1 --2- 172.21.15.89:0/4220366815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f316415da50 0x7f31640ce730 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f316408c190 tx=0x7f3154009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f63c688617107df server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.349+0000 7f3160a00700 1 -- 172.21.15.89:0/4220366815 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315400b870 con 0x7f316415da50 2024-04-02T09:12:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.349+0000 7f3160a00700 1 -- 172.21.15.89:0/4220366815 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315400b9d0 con 0x7f316415da50 2024-04-02T09:12:39.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.349+0000 7f3160a00700 1 -- 172.21.15.89:0/4220366815 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315400bd60 con 0x7f316415da50 2024-04-02T09:12:39.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.350+0000 7f3169200700 1 -- 172.21.15.89:0/4220366815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f316415da50 msgr2=0x7f31640ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:39.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.350+0000 7f3169200700 1 --2- 172.21.15.89:0/4220366815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f316415da50 0x7f31640ce730 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f316408c190 tx=0x7f3154009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.350+0000 7f3169200700 1 -- 172.21.15.89:0/4220366815 shutdown_connections 2024-04-02T09:12:39.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.350+0000 7f3169200700 1 --2- 172.21.15.89:0/4220366815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f316415da50 0x7f31640ce730 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.350+0000 7f3169200700 1 --2- 172.21.15.89:0/4220366815 >> v2:172.21.15.113:3300/0 conn(0x7f31640d0b20 0x7f31640b52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.350+0000 7f3169200700 1 --2- 172.21.15.89:0/4220366815 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31640cec70 0x7f31640d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.350+0000 7f3169200700 1 -- 172.21.15.89:0/4220366815 >> 172.21.15.89:0/4220366815 conn(0x7f31640666a0 msgr2=0x7f3164066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:39.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.350+0000 7f3169200700 1 -- 172.21.15.89:0/4220366815 shutdown_connections 2024-04-02T09:12:39.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.350+0000 7f3169200700 1 -- 172.21.15.89:0/4220366815 wait complete. 2024-04-02T09:12:39.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.351+0000 7f3169200700 1 Processor -- start 2024-04-02T09:12:39.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.352+0000 7f3169200700 1 -- start start 2024-04-02T09:12:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.352+0000 7f3169200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31640cec70 0x7f31641f2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.352+0000 7f3169200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31640d0b20 0x7f3164155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3169200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f316415da50 0x7f31640b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3169200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31640b6060 con 0x7f31640cec70 2024-04-02T09:12:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3169200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31640b5210 con 0x7f316415da50 2024-04-02T09:12:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3169200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31640b5450 con 0x7f31640d0b20 2024-04-02T09:12:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3161e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31640cec70 0x7f31641f2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3161e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31640cec70 0x7f31641f2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59524/0 (socket says 172.21.15.89:59524) 2024-04-02T09:12:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3161e00700 1 -- 172.21.15.89:0/14774477 learned_addr learned my addr 172.21.15.89:0/14774477 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3161400700 1 --2- 172.21.15.89:0/14774477 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31640d0b20 0x7f3164155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3162800700 1 --2- 172.21.15.89:0/14774477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f316415da50 0x7f31640b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3161e00700 1 -- 172.21.15.89:0/14774477 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31640d0b20 msgr2=0x7f3164155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3161e00700 1 --2- 172.21.15.89:0/14774477 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31640d0b20 0x7f3164155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3161e00700 1 -- 172.21.15.89:0/14774477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f316415da50 msgr2=0x7f31640b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3161e00700 1 --2- 172.21.15.89:0/14774477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f316415da50 0x7f31640b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3161e00700 1 -- 172.21.15.89:0/14774477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3154009550 con 0x7f31640cec70 2024-04-02T09:12:39.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.353+0000 7f3161e00700 1 --2- 172.21.15.89:0/14774477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31640cec70 0x7f31641f2c20 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f3154000c00 tx=0x7f3154004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:39.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.354+0000 7f315a000700 1 -- 172.21.15.89:0/14774477 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3154004520 con 0x7f31640cec70 2024-04-02T09:12:39.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.354+0000 7f315a000700 1 -- 172.21.15.89:0/14774477 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315401e070 con 0x7f31640cec70 2024-04-02T09:12:39.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.354+0000 7f3169200700 1 -- 172.21.15.89:0/14774477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31640b5730 con 0x7f31640cec70 2024-04-02T09:12:39.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.354+0000 7f315a000700 1 -- 172.21.15.89:0/14774477 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31540038c0 con 0x7f31640cec70 2024-04-02T09:12:39.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.354+0000 7f3169200700 1 -- 172.21.15.89:0/14774477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31640b5cf0 con 0x7f31640cec70 2024-04-02T09:12:39.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.355+0000 7f3169200700 1 -- 172.21.15.89:0/14774477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3164147710 con 0x7f31640cec70 2024-04-02T09:12:39.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.356+0000 7f315a000700 1 -- 172.21.15.89:0/14774477 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3154003a20 con 0x7f31640cec70 2024-04-02T09:12:39.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.357+0000 7f315a000700 1 --2- 172.21.15.89:0/14774477 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f314406adb0 0x7f314406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:39.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.357+0000 7f315a000700 1 -- 172.21.15.89:0/14774477 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f3154020070 con 0x7f31640cec70 2024-04-02T09:12:39.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.357+0000 7f3161400700 1 --2- 172.21.15.89:0/14774477 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f314406adb0 0x7f314406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:39.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.358+0000 7f3161400700 1 --2- 172.21.15.89:0/14774477 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f314406adb0 0x7f314406d270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f31641f6640 tx=0x7f3150008040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:39.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.362+0000 7f315a000700 1 -- 172.21.15.89:0/14774477 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3154055ba0 con 0x7f31640cec70 2024-04-02T09:12:39.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.567+0000 7f3169200700 1 -- 172.21.15.89:0/14774477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31640b5e80 con 0x7f31640cec70 2024-04-02T09:12:39.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.568+0000 7f315a000700 1 -- 172.21.15.89:0/14774477 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+20477 (secure 0 0 0) 0x7f315402a090 con 0x7f31640cec70 2024-04-02T09:12:39.568 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:39.568 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":226,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:12:39.105974+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:12:23.034869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-02T09:12:24.158560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:12:37.240568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:12:38.093999+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:12:39.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.571+0000 7f3169200700 1 -- 172.21.15.89:0/14774477 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f314406adb0 msgr2=0x7f314406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.571+0000 7f3169200700 1 --2- 172.21.15.89:0/14774477 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f314406adb0 0x7f314406d270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f31641f6640 tx=0x7f3150008040 comp rx=0 tx=0).stop 2024-04-02T09:12:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.571+0000 7f3169200700 1 -- 172.21.15.89:0/14774477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31640cec70 msgr2=0x7f31641f2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.571+0000 7f3169200700 1 --2- 172.21.15.89:0/14774477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31640cec70 0x7f31641f2c20 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f3154000c00 tx=0x7f3154004750 comp rx=0 tx=0).stop 2024-04-02T09:12:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.571+0000 7f3169200700 1 -- 172.21.15.89:0/14774477 shutdown_connections 2024-04-02T09:12:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.571+0000 7f3169200700 1 --2- 172.21.15.89:0/14774477 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f314406adb0 0x7f314406d270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.571+0000 7f3169200700 1 --2- 172.21.15.89:0/14774477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f316415da50 0x7f31640b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.571+0000 7f3169200700 1 --2- 172.21.15.89:0/14774477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31640cec70 0x7f31641f2c20 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.571+0000 7f3169200700 1 --2- 172.21.15.89:0/14774477 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31640d0b20 0x7f3164155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.571+0000 7f3169200700 1 -- 172.21.15.89:0/14774477 >> 172.21.15.89:0/14774477 conn(0x7f31640666a0 msgr2=0x7f316415cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.571+0000 7f3169200700 1 -- 172.21.15.89:0/14774477 shutdown_connections 2024-04-02T09:12:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.571+0000 7f3169200700 1 -- 172.21.15.89:0/14774477 wait complete. 2024-04-02T09:12:39.589 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:12:39.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.782+0000 7fd553200700 1 Processor -- start 2024-04-02T09:12:39.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.783+0000 7fd553200700 1 -- start start 2024-04-02T09:12:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd553200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd54c160be0 0x7fd54c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd553200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd54c0ce0e0 0x7fd54c0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd553200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd54c0cead0 0x7fd54c161240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd553200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54c0cef60 con 0x7fd54c160be0 2024-04-02T09:12:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd553200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54c0b65d0 con 0x7fd54c0cead0 2024-04-02T09:12:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd553200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0d2a80 con 0x7fd54c0ce0e0 2024-04-02T09:12:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd54be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd54c160be0 0x7fd54c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd54be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd54c160be0 0x7fd54c0cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59538/0 (socket says 172.21.15.89:59538) 2024-04-02T09:12:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd54be00700 1 -- 172.21.15.89:0/4270020005 learned_addr learned my addr 172.21.15.89:0/4270020005 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd54b400700 1 --2- 172.21.15.89:0/4270020005 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd54c0ce0e0 0x7fd54c0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd551200700 1 --2- 172.21.15.89:0/4270020005 >> v2:172.21.15.113:3300/0 conn(0x7fd54c0cead0 0x7fd54c161240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd54be00700 1 -- 172.21.15.89:0/4270020005 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd54c0ce0e0 msgr2=0x7fd54c0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd54be00700 1 --2- 172.21.15.89:0/4270020005 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd54c0ce0e0 0x7fd54c0ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd54be00700 1 -- 172.21.15.89:0/4270020005 >> v2:172.21.15.113:3300/0 conn(0x7fd54c0cead0 msgr2=0x7fd54c161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:39.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd54be00700 1 --2- 172.21.15.89:0/4270020005 >> v2:172.21.15.113:3300/0 conn(0x7fd54c0cead0 0x7fd54c161240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.784+0000 7fd54be00700 1 -- 172.21.15.89:0/4270020005 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c160fc0 con 0x7fd54c160be0 2024-04-02T09:12:39.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.785+0000 7fd54be00700 1 --2- 172.21.15.89:0/4270020005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd54c160be0 0x7fd54c0cdba0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7fd54c0b6540 tx=0x7fd53c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74caaab9782d777 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.785+0000 7fd54aa00700 1 -- 172.21.15.89:0/4270020005 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c00b860 con 0x7fd54c160be0 2024-04-02T09:12:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.785+0000 7fd54aa00700 1 -- 172.21.15.89:0/4270020005 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53c00b9c0 con 0x7fd54c160be0 2024-04-02T09:12:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.785+0000 7fd54aa00700 1 -- 172.21.15.89:0/4270020005 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c00bd50 con 0x7fd54c160be0 2024-04-02T09:12:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.785+0000 7fd553200700 1 -- 172.21.15.89:0/4270020005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd54c160be0 msgr2=0x7fd54c0cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.785+0000 7fd553200700 1 --2- 172.21.15.89:0/4270020005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd54c160be0 0x7fd54c0cdba0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7fd54c0b6540 tx=0x7fd53c009c00 comp rx=0 tx=0).stop 2024-04-02T09:12:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.785+0000 7fd553200700 1 -- 172.21.15.89:0/4270020005 shutdown_connections 2024-04-02T09:12:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.785+0000 7fd553200700 1 --2- 172.21.15.89:0/4270020005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd54c160be0 0x7fd54c0cdba0 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.785+0000 7fd553200700 1 --2- 172.21.15.89:0/4270020005 >> v2:172.21.15.113:3300/0 conn(0x7fd54c0cead0 0x7fd54c161240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.786+0000 7fd553200700 1 --2- 172.21.15.89:0/4270020005 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd54c0ce0e0 0x7fd54c0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.786+0000 7fd553200700 1 -- 172.21.15.89:0/4270020005 >> 172.21.15.89:0/4270020005 conn(0x7fd54c08aff0 msgr2=0x7fd54c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.786+0000 7fd553200700 1 -- 172.21.15.89:0/4270020005 shutdown_connections 2024-04-02T09:12:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.786+0000 7fd553200700 1 -- 172.21.15.89:0/4270020005 wait complete. 2024-04-02T09:12:39.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.787+0000 7fd553200700 1 Processor -- start 2024-04-02T09:12:39.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.787+0000 7fd553200700 1 -- start start 2024-04-02T09:12:39.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.788+0000 7fd553200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd54c0ce0e0 0x7fd54c1f3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:39.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.788+0000 7fd553200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd54c0cead0 0x7fd54c0c8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:39.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.788+0000 7fd553200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd54c160be0 0x7fd54c0cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.788+0000 7fd553200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c1f3e60 con 0x7fd54c160be0 2024-04-02T09:12:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.788+0000 7fd553200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0c7390 con 0x7fd54c0ce0e0 2024-04-02T09:12:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.788+0000 7fd553200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0c7670 con 0x7fd54c0cead0 2024-04-02T09:12:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.789+0000 7fd551200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd54c160be0 0x7fd54c0cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.789+0000 7fd54b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd54c0cead0 0x7fd54c0c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.789+0000 7fd551200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd54c160be0 0x7fd54c0cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59540/0 (socket says 172.21.15.89:59540) 2024-04-02T09:12:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.789+0000 7fd551200700 1 -- 172.21.15.89:0/790327965 learned_addr learned my addr 172.21.15.89:0/790327965 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.789+0000 7fd54be00700 1 --2- 172.21.15.89:0/790327965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd54c0ce0e0 0x7fd54c1f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.789+0000 7fd551200700 1 -- 172.21.15.89:0/790327965 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd54c0cead0 msgr2=0x7fd54c0c8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.789+0000 7fd551200700 1 --2- 172.21.15.89:0/790327965 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd54c0cead0 0x7fd54c0c8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.789+0000 7fd551200700 1 -- 172.21.15.89:0/790327965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd54c0ce0e0 msgr2=0x7fd54c1f3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.789+0000 7fd551200700 1 --2- 172.21.15.89:0/790327965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd54c0ce0e0 0x7fd54c1f3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.789+0000 7fd551200700 1 -- 172.21.15.89:0/790327965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c009550 con 0x7fd54c160be0 2024-04-02T09:12:39.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.789+0000 7fd551200700 1 --2- 172.21.15.89:0/790327965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd54c160be0 0x7fd54c0cc6d0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7fd544007e40 tx=0x7fd54400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.789+0000 7fd543e00700 1 -- 172.21.15.89:0/790327965 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54400ed30 con 0x7fd54c160be0 2024-04-02T09:12:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.790+0000 7fd543e00700 1 -- 172.21.15.89:0/790327965 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd544014420 con 0x7fd54c160be0 2024-04-02T09:12:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.790+0000 7fd553200700 1 -- 172.21.15.89:0/790327965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54c0c7950 con 0x7fd54c160be0 2024-04-02T09:12:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.790+0000 7fd543e00700 1 -- 172.21.15.89:0/790327965 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5440147d0 con 0x7fd54c160be0 2024-04-02T09:12:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.790+0000 7fd553200700 1 -- 172.21.15.89:0/790327965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c0c7f10 con 0x7fd54c160be0 2024-04-02T09:12:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.791+0000 7fd553200700 1 -- 172.21.15.89:0/790327965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c067720 con 0x7fd54c160be0 2024-04-02T09:12:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.792+0000 7fd543e00700 1 -- 172.21.15.89:0/790327965 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd544014930 con 0x7fd54c160be0 2024-04-02T09:12:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.793+0000 7fd543e00700 1 --2- 172.21.15.89:0/790327965 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd52c06adb0 0x7fd52c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.793+0000 7fd543e00700 1 -- 172.21.15.89:0/790327965 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fd54408a0d0 con 0x7fd54c160be0 2024-04-02T09:12:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.793+0000 7fd54be00700 1 --2- 172.21.15.89:0/790327965 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd52c06adb0 0x7fd52c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.794+0000 7fd54be00700 1 --2- 172.21.15.89:0/790327965 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd52c06adb0 0x7fd52c06d270 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd53c000c00 tx=0x7fd53c008030 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:39.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:39.798+0000 7fd543e00700 1 -- 172.21.15.89:0/790327965 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd544054be0 con 0x7fd54c160be0 2024-04-02T09:12:40.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.002+0000 7fd553200700 1 -- 172.21.15.89:0/790327965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd54c0c9b40 con 0x7fd54c160be0 2024-04-02T09:12:40.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.003+0000 7fd543e00700 1 -- 172.21.15.89:0/790327965 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+20477 (secure 0 0 0) 0x7fd544058200 con 0x7fd54c160be0 2024-04-02T09:12:40.003 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:40.003 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":226,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:12:39.105974+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:12:23.034869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-02T09:12:24.158560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:12:37.240568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:12:38.093999+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:12:40.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.005+0000 7fd553200700 1 -- 172.21.15.89:0/790327965 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd52c06adb0 msgr2=0x7fd52c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:40.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.005+0000 7fd553200700 1 --2- 172.21.15.89:0/790327965 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd52c06adb0 0x7fd52c06d270 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd53c000c00 tx=0x7fd53c008030 comp rx=0 tx=0).stop 2024-04-02T09:12:40.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.005+0000 7fd553200700 1 -- 172.21.15.89:0/790327965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd54c160be0 msgr2=0x7fd54c0cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:40.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.005+0000 7fd553200700 1 --2- 172.21.15.89:0/790327965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd54c160be0 0x7fd54c0cc6d0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7fd544007e40 tx=0x7fd54400e3c0 comp rx=0 tx=0).stop 2024-04-02T09:12:40.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.006+0000 7fd553200700 1 -- 172.21.15.89:0/790327965 shutdown_connections 2024-04-02T09:12:40.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.006+0000 7fd553200700 1 --2- 172.21.15.89:0/790327965 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd52c06adb0 0x7fd52c06d270 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:40.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.006+0000 7fd553200700 1 --2- 172.21.15.89:0/790327965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd54c0ce0e0 0x7fd54c1f3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:40.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.006+0000 7fd553200700 1 --2- 172.21.15.89:0/790327965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd54c160be0 0x7fd54c0cc6d0 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:40.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.006+0000 7fd553200700 1 --2- 172.21.15.89:0/790327965 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd54c0cead0 0x7fd54c0c8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:40.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.006+0000 7fd553200700 1 -- 172.21.15.89:0/790327965 >> 172.21.15.89:0/790327965 conn(0x7fd54c08aff0 msgr2=0x7fd54c0b7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:40.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.006+0000 7fd553200700 1 -- 172.21.15.89:0/790327965 shutdown_connections 2024-04-02T09:12:40.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.006+0000 7fd553200700 1 -- 172.21.15.89:0/790327965 wait complete. 2024-04-02T09:12:40.023 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-04-02T09:12:40.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.220+0000 7f852ec00700 1 Processor -- start 2024-04-02T09:12:40.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.220+0000 7f852ec00700 1 -- start start 2024-04-02T09:12:40.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.221+0000 7f852ec00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f85280ced80 0x7f85280d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:40.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.221+0000 7f852ec00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852815f820 con 0x7f852815fcb0 2024-04-02T09:12:40.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.221+0000 7f852ec00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85280cf420 con 0x7f852815dad0 2024-04-02T09:12:40.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.221+0000 7f852ec00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85280cfcc0 con 0x7f85280ced80 2024-04-02T09:12:40.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.221+0000 7f852cc00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f852815fcb0 0x7f85281600e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:54640/0 (socket says 172.21.15.89:54640) 2024-04-02T09:12:40.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.221+0000 7f852cc00700 1 -- 172.21.15.89:0/1666565009 learned_addr learned my addr 172.21.15.89:0/1666565009 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:40.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.222+0000 7f8527400700 1 --2- 172.21.15.89:0/1666565009 >> v2:172.21.15.113:3300/0 conn(0x7f85280ced80 0x7f85280d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:40.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.222+0000 7f8526a00700 1 -- 172.21.15.89:0/1666565009 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045717805 0 0) 0x7f852815f820 con 0x7f852815fcb0 2024-04-02T09:12:40.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.222+0000 7f8526a00700 1 -- 172.21.15.89:0/1666565009 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f850c0035b0 con 0x7f852815fcb0 2024-04-02T09:12:40.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.222+0000 7f8526a00700 1 -- 172.21.15.89:0/1666565009 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319816402 0 0) 0x7f85280cf420 con 0x7f852815dad0 2024-04-02T09:12:40.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.222+0000 7f8526a00700 1 -- 172.21.15.89:0/1666565009 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f852815f820 con 0x7f852815dad0 2024-04-02T09:12:40.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.222+0000 7f8527400700 1 -- 172.21.15.89:0/1666565009 >> v1:172.21.15.113:6789/0 conn(0x7f852815dad0 legacy=0x7f85280d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:40.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.222+0000 7f8527400700 1 -- 172.21.15.89:0/1666565009 >> v1:172.21.15.89:6789/0 conn(0x7f852815fcb0 legacy=0x7f85281600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:40.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.222+0000 7f8526a00700 1 -- 172.21.15.89:0/1666565009 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2244508966 0 0) 0x7f850c0035b0 con 0x7f852815fcb0 2024-04-02T09:12:40.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.222+0000 7f8527400700 1 -- 172.21.15.89:0/1666565009 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85280cf700 con 0x7f85280ced80 2024-04-02T09:12:40.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.222+0000 7f8527400700 1 --2- 172.21.15.89:0/1666565009 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85280ced80 0x7f85280d1d40 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f8514009840 tx=0x7f8514009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58653a0ed9f9d69b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.223+0000 7f8526a00700 1 -- 172.21.15.89:0/1666565009 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85200019f0 con 0x7f85280ced80 2024-04-02T09:12:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.223+0000 7f8526a00700 1 -- 172.21.15.89:0/1666565009 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8514004030 con 0x7f85280ced80 2024-04-02T09:12:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.223+0000 7f8526a00700 1 -- 172.21.15.89:0/1666565009 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85140043a0 con 0x7f85280ced80 2024-04-02T09:12:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.223+0000 7f852ec00700 1 -- 172.21.15.89:0/1666565009 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85280ced80 msgr2=0x7f85280d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.223+0000 7f852ec00700 1 --2- 172.21.15.89:0/1666565009 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85280ced80 0x7f85280d1d40 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f8514009840 tx=0x7f8514009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.223+0000 7f852ec00700 1 -- 172.21.15.89:0/1666565009 shutdown_connections 2024-04-02T09:12:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.223+0000 7f852ec00700 1 --2- 172.21.15.89:0/1666565009 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85280ced80 0x7f85280d1d40 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.223+0000 7f852ec00700 1 -- 172.21.15.89:0/1666565009 >> 172.21.15.89:0/1666565009 conn(0x7f852808aff0 msgr2=0x7f85280d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.223+0000 7f852ec00700 1 -- 172.21.15.89:0/1666565009 shutdown_connections 2024-04-02T09:12:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.224+0000 7f852ec00700 1 -- 172.21.15.89:0/1666565009 wait complete. 2024-04-02T09:12:40.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.225+0000 7f852ec00700 1 Processor -- start 2024-04-02T09:12:40.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.225+0000 7f852ec00700 1 -- start start 2024-04-02T09:12:40.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.226+0000 7f852ec00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85280ced80 0x7f85280c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:40.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.226+0000 7f852ec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f852815dad0 0x7f85281f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:40.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.226+0000 7f852ec00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f852815fcb0 0x7f85281f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:40.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.226+0000 7f852ec00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85281f2920 con 0x7f85280ced80 2024-04-02T09:12:40.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.226+0000 7f852ec00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85280b8dc0 con 0x7f852815fcb0 2024-04-02T09:12:40.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.226+0000 7f852ec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85280b8240 con 0x7f852815dad0 2024-04-02T09:12:40.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.226+0000 7f8527e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85280ced80 0x7f85280c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:40.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.226+0000 7f8527e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85280ced80 0x7f85280c9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59554/0 (socket says 172.21.15.89:59554) 2024-04-02T09:12:40.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.226+0000 7f8527e00700 1 -- 172.21.15.89:0/2304685477 learned_addr learned my addr 172.21.15.89:0/2304685477 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:40.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.227+0000 7f8527400700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f852815dad0 0x7f85281f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:40.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.227+0000 7f852cc00700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f852815fcb0 0x7f85281f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:40.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.227+0000 7f8527e00700 1 -- 172.21.15.89:0/2304685477 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f852815dad0 msgr2=0x7f85281f1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:40.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.227+0000 7f8527e00700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f852815dad0 0x7f85281f1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:40.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.227+0000 7f8527e00700 1 -- 172.21.15.89:0/2304685477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f852815fcb0 msgr2=0x7f85281f23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:40.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.227+0000 7f8527e00700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f852815fcb0 0x7f85281f23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:40.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.227+0000 7f8527e00700 1 -- 172.21.15.89:0/2304685477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85140094a0 con 0x7f85280ced80 2024-04-02T09:12:40.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.227+0000 7f852cc00700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f852815fcb0 0x7f85281f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:40.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.227+0000 7f8527e00700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85280ced80 0x7f85280c9a80 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f851800c520 tx=0x7f851800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:40.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.227+0000 7f851fe00700 1 -- 172.21.15.89:0/2304685477 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85180052d0 con 0x7f85280ced80 2024-04-02T09:12:40.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.227+0000 7f851fe00700 1 -- 172.21.15.89:0/2304685477 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8518005430 con 0x7f85280ced80 2024-04-02T09:12:40.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.227+0000 7f852ec00700 1 -- 172.21.15.89:0/2304685477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85280b8520 con 0x7f85280ced80 2024-04-02T09:12:40.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.227+0000 7f851fe00700 1 -- 172.21.15.89:0/2304685477 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85180038c0 con 0x7f85280ced80 2024-04-02T09:12:40.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.227+0000 7f852ec00700 1 -- 172.21.15.89:0/2304685477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85280b8ae0 con 0x7f85280ced80 2024-04-02T09:12:40.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.228+0000 7f852ec00700 1 -- 172.21.15.89:0/2304685477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8528067720 con 0x7f85280ced80 2024-04-02T09:12:40.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.229+0000 7f851fe00700 1 -- 172.21.15.89:0/2304685477 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8518003a20 con 0x7f85280ced80 2024-04-02T09:12:40.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.230+0000 7f851fe00700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f850806adb0 0x7f850806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:40.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.230+0000 7f851fe00700 1 -- 172.21.15.89:0/2304685477 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f8518088e70 con 0x7f85280ced80 2024-04-02T09:12:40.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.230+0000 7f8527400700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f850806adb0 0x7f850806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:40.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.231+0000 7f8527400700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f850806adb0 0x7f850806d270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f85140090b0 tx=0x7f8514008030 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:40.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.235+0000 7f851fe00700 1 -- 172.21.15.89:0/2304685477 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8518053ab0 con 0x7f85280ced80 2024-04-02T09:12:40.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:40.483+0000 7f852ec00700 1 -- 172.21.15.89:0/2304685477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f852806a840 con 0x7f85280ced80 2024-04-02T09:12:40.662 INFO:tasks.ceph.mds.d.smithi113.stderr:2024-04-02T09:12:40.663+0000 7fc139e00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T09:12:41.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.113+0000 7f851fe00700 1 -- 172.21.15.89:0/2304685477 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v109) v1 ==== 121+0+0 (secure 0 0 0) 0x7f85180170c0 con 0x7f85280ced80 2024-04-02T09:12:41.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.116+0000 7f852ec00700 1 -- 172.21.15.89:0/2304685477 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f850806adb0 msgr2=0x7f850806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:41.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.116+0000 7f852ec00700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f850806adb0 0x7f850806d270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f85140090b0 tx=0x7f8514008030 comp rx=0 tx=0).stop 2024-04-02T09:12:41.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.116+0000 7f852ec00700 1 -- 172.21.15.89:0/2304685477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85280ced80 msgr2=0x7f85280c9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:41.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.116+0000 7f852ec00700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85280ced80 0x7f85280c9a80 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f851800c520 tx=0x7f851800c8e0 comp rx=0 tx=0).stop 2024-04-02T09:12:41.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.116+0000 7f852ec00700 1 -- 172.21.15.89:0/2304685477 shutdown_connections 2024-04-02T09:12:41.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.116+0000 7f852ec00700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f850806adb0 0x7f850806d270 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:41.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.116+0000 7f852ec00700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f852815fcb0 0x7f85281f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:41.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.116+0000 7f852ec00700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85280ced80 0x7f85280c9a80 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:41.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.116+0000 7f852ec00700 1 --2- 172.21.15.89:0/2304685477 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f852815dad0 0x7f85281f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:41.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.116+0000 7f852ec00700 1 -- 172.21.15.89:0/2304685477 >> 172.21.15.89:0/2304685477 conn(0x7f852808aff0 msgr2=0x7f852815b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:41.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.117+0000 7f852ec00700 1 -- 172.21.15.89:0/2304685477 shutdown_connections 2024-04-02T09:12:41.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.117+0000 7f852ec00700 1 -- 172.21.15.89:0/2304685477 wait complete. 2024-04-02T09:12:41.137 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T09:12:41.138 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-04-02T09:12:41.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.339+0000 7f6c5da00700 1 Processor -- start 2024-04-02T09:12:41.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.340+0000 7f6c5da00700 1 -- start start 2024-04-02T09:12:41.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.341+0000 7f6c5da00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c580cbb30 0x7f6c580cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:41.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.341+0000 7f6c5da00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c580cc450 con 0x7f6c580cd4a0 2024-04-02T09:12:41.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.341+0000 7f6c5da00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c580cdae0 con 0x7f6c580cd020 2024-04-02T09:12:41.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.341+0000 7f6c5da00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c580cddc0 con 0x7f6c580cbb30 2024-04-02T09:12:41.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.341+0000 7f6c57400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6c580cd4a0 0x7f6c5806e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:54646/0 (socket says 172.21.15.89:54646) 2024-04-02T09:12:41.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.341+0000 7f6c57400700 1 -- 172.21.15.89:0/4214334321 learned_addr learned my addr 172.21.15.89:0/4214334321 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:41.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.341+0000 7f6c56a00700 1 --2- 172.21.15.89:0/4214334321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c580cbb30 0x7f6c580cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:41.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.341+0000 7f6c55600700 1 -- 172.21.15.89:0/4214334321 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252773933 0 0) 0x7f6c580cc450 con 0x7f6c580cd4a0 2024-04-02T09:12:41.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.341+0000 7f6c55600700 1 -- 172.21.15.89:0/4214334321 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c3c0035b0 con 0x7f6c580cd4a0 2024-04-02T09:12:41.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.342+0000 7f6c55600700 1 -- 172.21.15.89:0/4214334321 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743303995 0 0) 0x7f6c580cdae0 con 0x7f6c580cd020 2024-04-02T09:12:41.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.342+0000 7f6c55600700 1 -- 172.21.15.89:0/4214334321 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c580cc450 con 0x7f6c580cd020 2024-04-02T09:12:41.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.342+0000 7f6c56a00700 1 -- 172.21.15.89:0/4214334321 >> v1:172.21.15.113:6789/0 conn(0x7f6c580cd020 legacy=0x7f6c580b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:41.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.342+0000 7f6c56a00700 1 -- 172.21.15.89:0/4214334321 >> v1:172.21.15.89:6789/0 conn(0x7f6c580cd4a0 legacy=0x7f6c5806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:41.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.342+0000 7f6c56a00700 1 -- 172.21.15.89:0/4214334321 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c580ce0a0 con 0x7f6c580cbb30 2024-04-02T09:12:41.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.342+0000 7f6c56a00700 1 --2- 172.21.15.89:0/4214334321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c580cbb30 0x7f6c580cbf10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6c48009870 tx=0x7f6c48009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9516725e15058da4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:41.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.342+0000 7f6c55600700 1 -- 172.21.15.89:0/4214334321 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c48011830 con 0x7f6c580cbb30 2024-04-02T09:12:41.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.342+0000 7f6c55600700 1 -- 172.21.15.89:0/4214334321 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c48011990 con 0x7f6c580cbb30 2024-04-02T09:12:41.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.342+0000 7f6c55600700 1 -- 172.21.15.89:0/4214334321 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c48011d20 con 0x7f6c580cbb30 2024-04-02T09:12:41.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.343+0000 7f6c5da00700 1 -- 172.21.15.89:0/4214334321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c580cbb30 msgr2=0x7f6c580cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:41.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.343+0000 7f6c5da00700 1 --2- 172.21.15.89:0/4214334321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c580cbb30 0x7f6c580cbf10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6c48009870 tx=0x7f6c48009b80 comp rx=0 tx=0).stop 2024-04-02T09:12:41.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.343+0000 7f6c5da00700 1 -- 172.21.15.89:0/4214334321 shutdown_connections 2024-04-02T09:12:41.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.343+0000 7f6c5da00700 1 --2- 172.21.15.89:0/4214334321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c580cbb30 0x7f6c580cbf10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:41.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.343+0000 7f6c5da00700 1 -- 172.21.15.89:0/4214334321 >> 172.21.15.89:0/4214334321 conn(0x7f6c580b0d70 msgr2=0x7f6c580b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:41.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.343+0000 7f6c5da00700 1 -- 172.21.15.89:0/4214334321 shutdown_connections 2024-04-02T09:12:41.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.343+0000 7f6c5da00700 1 -- 172.21.15.89:0/4214334321 wait complete. 2024-04-02T09:12:41.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.345+0000 7f6c5da00700 1 Processor -- start 2024-04-02T09:12:41.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.345+0000 7f6c5da00700 1 -- start start 2024-04-02T09:12:41.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c5da00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c580cbb30 0x7f6c5814b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:41.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c5da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c580cd020 0x7f6c58072200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:41.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c5da00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c580cd4a0 0x7f6c580aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:41.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c5da00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5814bac0 con 0x7f6c580cd020 2024-04-02T09:12:41.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c5da00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c580c8e00 con 0x7f6c580cbb30 2024-04-02T09:12:41.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c5da00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c580c90c0 con 0x7f6c580cd4a0 2024-04-02T09:12:41.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c56000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c580cd020 0x7f6c58072200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:41.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c56000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c580cd020 0x7f6c58072200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59568/0 (socket says 172.21.15.89:59568) 2024-04-02T09:12:41.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c56000700 1 -- 172.21.15.89:0/407969539 learned_addr learned my addr 172.21.15.89:0/407969539 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:41.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c57400700 1 --2- 172.21.15.89:0/407969539 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c580cd4a0 0x7f6c580aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:41.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c56a00700 1 --2- 172.21.15.89:0/407969539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c580cbb30 0x7f6c5814b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:41.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c56000700 1 -- 172.21.15.89:0/407969539 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c580cd4a0 msgr2=0x7f6c580aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:41.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c56000700 1 --2- 172.21.15.89:0/407969539 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c580cd4a0 0x7f6c580aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:41.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c56000700 1 -- 172.21.15.89:0/407969539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c580cbb30 msgr2=0x7f6c5814b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:41.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c56000700 1 --2- 172.21.15.89:0/407969539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c580cbb30 0x7f6c5814b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:41.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.346+0000 7f6c56000700 1 -- 172.21.15.89:0/407969539 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c480094a0 con 0x7f6c580cd020 2024-04-02T09:12:41.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.347+0000 7f6c56000700 1 --2- 172.21.15.89:0/407969539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c580cd020 0x7f6c58072200 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f6c4400c500 tx=0x7f6c4400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:41.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.347+0000 7f6c4ea00700 1 -- 172.21.15.89:0/407969539 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c440052d0 con 0x7f6c580cd020 2024-04-02T09:12:41.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.347+0000 7f6c4ea00700 1 -- 172.21.15.89:0/407969539 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c44005450 con 0x7f6c580cd020 2024-04-02T09:12:41.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.347+0000 7f6c5da00700 1 -- 172.21.15.89:0/407969539 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c580c93a0 con 0x7f6c580cd020 2024-04-02T09:12:41.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.347+0000 7f6c4ea00700 1 -- 172.21.15.89:0/407969539 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c440038c0 con 0x7f6c580cd020 2024-04-02T09:12:41.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.347+0000 7f6c5da00700 1 -- 172.21.15.89:0/407969539 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c580c9960 con 0x7f6c580cd020 2024-04-02T09:12:41.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.348+0000 7f6c5da00700 1 -- 172.21.15.89:0/407969539 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c58067720 con 0x7f6c580cd020 2024-04-02T09:12:41.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.349+0000 7f6c4ea00700 1 -- 172.21.15.89:0/407969539 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6c44003a20 con 0x7f6c580cd020 2024-04-02T09:12:41.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.350+0000 7f6c4ea00700 1 --2- 172.21.15.89:0/407969539 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6c3806adb0 0x7f6c3806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:41.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.350+0000 7f6c4ea00700 1 -- 172.21.15.89:0/407969539 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f6c44088f50 con 0x7f6c580cd020 2024-04-02T09:12:41.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.350+0000 7f6c56a00700 1 --2- 172.21.15.89:0/407969539 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6c3806adb0 0x7f6c3806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:41.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.351+0000 7f6c56a00700 1 --2- 172.21.15.89:0/407969539 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6c3806adb0 0x7f6c3806d270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6c480097c0 tx=0x7f6c4801c040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:41.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.354+0000 7f6c4ea00700 1 -- 172.21.15.89:0/407969539 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c44053a60 con 0x7f6c580cd020 2024-04-02T09:12:41.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:41.601+0000 7f6c5da00700 1 -- 172.21.15.89:0/407969539 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6c5815bda0 con 0x7f6c580cd020 2024-04-02T09:12:42.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.114+0000 7f6c4ea00700 1 -- 172.21.15.89:0/407969539 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v110) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6c440170c0 con 0x7f6c580cd020 2024-04-02T09:12:42.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.117+0000 7f6c5da00700 1 -- 172.21.15.89:0/407969539 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6c3806adb0 msgr2=0x7f6c3806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:42.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.117+0000 7f6c5da00700 1 --2- 172.21.15.89:0/407969539 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6c3806adb0 0x7f6c3806d270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6c480097c0 tx=0x7f6c4801c040 comp rx=0 tx=0).stop 2024-04-02T09:12:42.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.117+0000 7f6c5da00700 1 -- 172.21.15.89:0/407969539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c580cd020 msgr2=0x7f6c58072200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:42.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.117+0000 7f6c5da00700 1 --2- 172.21.15.89:0/407969539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c580cd020 0x7f6c58072200 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f6c4400c500 tx=0x7f6c4400c8c0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.117+0000 7f6c5da00700 1 -- 172.21.15.89:0/407969539 shutdown_connections 2024-04-02T09:12:42.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.117+0000 7f6c5da00700 1 --2- 172.21.15.89:0/407969539 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6c3806adb0 0x7f6c3806d270 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.117+0000 7f6c5da00700 1 --2- 172.21.15.89:0/407969539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c580cbb30 0x7f6c5814b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.117+0000 7f6c5da00700 1 --2- 172.21.15.89:0/407969539 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c580cd4a0 0x7f6c580aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.117+0000 7f6c5da00700 1 --2- 172.21.15.89:0/407969539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c580cd020 0x7f6c58072200 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.118+0000 7f6c5da00700 1 -- 172.21.15.89:0/407969539 >> 172.21.15.89:0/407969539 conn(0x7f6c580b0d70 msgr2=0x7f6c5815d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:42.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.118+0000 7f6c5da00700 1 -- 172.21.15.89:0/407969539 shutdown_connections 2024-04-02T09:12:42.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.118+0000 7f6c5da00700 1 -- 172.21.15.89:0/407969539 wait complete. 2024-04-02T09:12:42.137 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2024-04-02T09:12:42.138 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:12:42.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.330+0000 7f3460a00700 1 Processor -- start 2024-04-02T09:12:42.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.330+0000 7f3460a00700 1 -- start start 2024-04-02T09:12:42.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.331+0000 7f3460a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f345c0b27f0 0x7f345c0ac960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:42.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.331+0000 7f3460a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f345c0718e0 0x7f345c0acea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:42.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.331+0000 7f3460a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345c0ac4d0 con 0x7f345c160c90 2024-04-02T09:12:42.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.331+0000 7f3460a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f345c071e50 con 0x7f345c0718e0 2024-04-02T09:12:42.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.331+0000 7f3460a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f345c072130 con 0x7f345c0b27f0 2024-04-02T09:12:42.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.331+0000 7f345a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f345c0718e0 0x7f345c0acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:42.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.331+0000 7f345a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f345c0718e0 0x7f345c0acea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59580/0 (socket says 172.21.15.89:59580) 2024-04-02T09:12:42.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.331+0000 7f3459e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f345c160c90 0x7f345c0b20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:54652/0 (socket says 172.21.15.89:54652) 2024-04-02T09:12:42.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.332+0000 7f345a800700 1 -- 172.21.15.89:0/2355287274 learned_addr learned my addr 172.21.15.89:0/2355287274 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:42.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.332+0000 7f3459400700 1 --2- 172.21.15.89:0/2355287274 >> v2:172.21.15.113:3300/0 conn(0x7f345c0b27f0 0x7f345c0ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:42.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.332+0000 7f345a800700 1 -- 172.21.15.89:0/2355287274 >> v2:172.21.15.113:3300/0 conn(0x7f345c0b27f0 msgr2=0x7f345c0ac960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:42.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.332+0000 7f345a800700 1 --2- 172.21.15.89:0/2355287274 >> v2:172.21.15.113:3300/0 conn(0x7f345c0b27f0 0x7f345c0ac960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.332+0000 7f345a800700 1 -- 172.21.15.89:0/2355287274 >> v1:172.21.15.89:6789/0 conn(0x7f345c160c90 legacy=0x7f345c0b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:42.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.332+0000 7f345a800700 1 -- 172.21.15.89:0/2355287274 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c072410 con 0x7f345c0718e0 2024-04-02T09:12:42.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.332+0000 7f3459400700 1 --2- 172.21.15.89:0/2355287274 >> v2:172.21.15.113:3300/0 conn(0x7f345c0b27f0 0x7f345c0ac960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:42.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.332+0000 7f345a800700 1 --2- 172.21.15.89:0/2355287274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345c0718e0 0x7f345c0acea0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f345c06c530 tx=0x7f345400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac461116a70231e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:42.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.332+0000 7f3458a00700 1 -- 172.21.15.89:0/2355287274 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345400e070 con 0x7f345c0718e0 2024-04-02T09:12:42.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.332+0000 7f3458a00700 1 -- 172.21.15.89:0/2355287274 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345400b9c0 con 0x7f345c0718e0 2024-04-02T09:12:42.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.332+0000 7f3458a00700 1 -- 172.21.15.89:0/2355287274 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345400bd70 con 0x7f345c0718e0 2024-04-02T09:12:42.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.333+0000 7f3460a00700 1 -- 172.21.15.89:0/2355287274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345c0718e0 msgr2=0x7f345c0acea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:42.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.333+0000 7f3460a00700 1 --2- 172.21.15.89:0/2355287274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345c0718e0 0x7f345c0acea0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f345c06c530 tx=0x7f345400b330 comp rx=0 tx=0).stop 2024-04-02T09:12:42.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.333+0000 7f3460a00700 1 -- 172.21.15.89:0/2355287274 shutdown_connections 2024-04-02T09:12:42.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.333+0000 7f3460a00700 1 --2- 172.21.15.89:0/2355287274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345c0718e0 0x7f345c0acea0 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.333+0000 7f3460a00700 1 --2- 172.21.15.89:0/2355287274 >> v2:172.21.15.113:3300/0 conn(0x7f345c0b27f0 0x7f345c0ac960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.333+0000 7f3460a00700 1 -- 172.21.15.89:0/2355287274 >> 172.21.15.89:0/2355287274 conn(0x7f345c0cf190 msgr2=0x7f345c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:42.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.333+0000 7f3460a00700 1 -- 172.21.15.89:0/2355287274 shutdown_connections 2024-04-02T09:12:42.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.333+0000 7f3460a00700 1 -- 172.21.15.89:0/2355287274 wait complete. 2024-04-02T09:12:42.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.335+0000 7f3460a00700 1 Processor -- start 2024-04-02T09:12:42.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.335+0000 7f3460a00700 1 -- start start 2024-04-02T09:12:42.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.336+0000 7f3460a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345c0718e0 0x7f345c1f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:42.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.336+0000 7f3460a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345c0b27f0 0x7f345c1f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:42.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.336+0000 7f3460a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f345c160c90 0x7f345c158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:42.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.336+0000 7f3460a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c157200 con 0x7f345c0718e0 2024-04-02T09:12:42.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.336+0000 7f3460a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c1f3e30 con 0x7f345c160c90 2024-04-02T09:12:42.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.336+0000 7f3460a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345c157e80 con 0x7f345c0b27f0 2024-04-02T09:12:42.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.336+0000 7f3459e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345c0718e0 0x7f345c1f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:42.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.336+0000 7f3459e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345c0718e0 0x7f345c1f38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59590/0 (socket says 172.21.15.89:59590) 2024-04-02T09:12:42.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.336+0000 7f3459e00700 1 -- 172.21.15.89:0/570834525 learned_addr learned my addr 172.21.15.89:0/570834525 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:42.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.336+0000 7f3459400700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345c0b27f0 0x7f345c1f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:42.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.336+0000 7f345a800700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f345c160c90 0x7f345c158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:42.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.337+0000 7f3459e00700 1 -- 172.21.15.89:0/570834525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345c0b27f0 msgr2=0x7f345c1f4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:42.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.337+0000 7f3459e00700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345c0b27f0 0x7f345c1f4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.337+0000 7f3459e00700 1 -- 172.21.15.89:0/570834525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f345c160c90 msgr2=0x7f345c158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:42.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.337+0000 7f3459e00700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f345c160c90 0x7f345c158f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.337+0000 7f3459e00700 1 -- 172.21.15.89:0/570834525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3454009b30 con 0x7f345c0718e0 2024-04-02T09:12:42.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.337+0000 7f3459400700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345c0b27f0 0x7f345c1f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:42.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.337+0000 7f345a800700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f345c160c90 0x7f345c158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:42.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.337+0000 7f3459e00700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345c0718e0 0x7f345c1f38f0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f344c00c500 tx=0x7f344c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:42.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.337+0000 7f3452000700 1 -- 172.21.15.89:0/570834525 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c005cb0 con 0x7f345c0718e0 2024-04-02T09:12:42.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.337+0000 7f3452000700 1 -- 172.21.15.89:0/570834525 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344c005e10 con 0x7f345c0718e0 2024-04-02T09:12:42.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.337+0000 7f3460a00700 1 -- 172.21.15.89:0/570834525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f345c158160 con 0x7f345c0718e0 2024-04-02T09:12:42.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.337+0000 7f3452000700 1 -- 172.21.15.89:0/570834525 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c0072d0 con 0x7f345c0718e0 2024-04-02T09:12:42.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.337+0000 7f3460a00700 1 -- 172.21.15.89:0/570834525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f345c158720 con 0x7f345c0718e0 2024-04-02T09:12:42.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.338+0000 7f3460a00700 1 -- 172.21.15.89:0/570834525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f345c067720 con 0x7f345c0718e0 2024-04-02T09:12:42.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.339+0000 7f3452000700 1 -- 172.21.15.89:0/570834525 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f344c007430 con 0x7f345c0718e0 2024-04-02T09:12:42.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.340+0000 7f3452000700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f343c06adb0 0x7f343c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:42.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.340+0000 7f3452000700 1 -- 172.21.15.89:0/570834525 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f344c088c50 con 0x7f345c0718e0 2024-04-02T09:12:42.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.340+0000 7f3459400700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f343c06adb0 0x7f343c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:42.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.341+0000 7f3459400700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f343c06adb0 0x7f343c06d270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3448006fd0 tx=0x7f3448008040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:42.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.345+0000 7f3452000700 1 -- 172.21.15.89:0/570834525 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f344c08e020 con 0x7f345c0718e0 2024-04-02T09:12:42.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.592+0000 7f3460a00700 1 -- 172.21.15.89:0/570834525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f345c0ce550 con 0x7f345c0718e0 2024-04-02T09:12:42.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.593+0000 7f3452000700 1 -- 172.21.15.89:0/570834525 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5971 (secure 0 0 0) 0x7f344c017020 con 0x7f345c0718e0 2024-04-02T09:12:42.592 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:42.592 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":110,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8969,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3650347444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3650347444},{"type":"v1","addr":"172.21.15.89:6836","nonce":3650347444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2152500685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2152500685},{"type":"v1","addr":"172.21.15.113:6834","nonce":2152500685}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:25.170849+0000","modified":"2024-04-02T09:12:28.185307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11157":{"gid":11157,"name":"a","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2615927215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2615927215},{"type":"v1","addr":"172.21.15.89:6837","nonce":2615927215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:39.106317+0000","modified":"2024-04-02T09:12:42.112504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10279":{"gid":10279,"name":"d","rank":0,"incarnation":107,"state":"up:active","state_seq":6,"addr":"172.21.15.113:6835/2457040702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2457040702},{"type":"v1","addr":"172.21.15.113:6835","nonce":2457040702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-02T09:12:42.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.595+0000 7f3460a00700 1 -- 172.21.15.89:0/570834525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f343c06adb0 msgr2=0x7f343c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:42.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.595+0000 7f3460a00700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f343c06adb0 0x7f343c06d270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3448006fd0 tx=0x7f3448008040 comp rx=0 tx=0).stop 2024-04-02T09:12:42.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.595+0000 7f3460a00700 1 -- 172.21.15.89:0/570834525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345c0718e0 msgr2=0x7f345c1f38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:42.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.595+0000 7f3460a00700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345c0718e0 0x7f345c1f38f0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f344c00c500 tx=0x7f344c00c8c0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.595+0000 7f3460a00700 1 -- 172.21.15.89:0/570834525 shutdown_connections 2024-04-02T09:12:42.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.595+0000 7f3460a00700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f343c06adb0 0x7f343c06d270 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.595+0000 7f3460a00700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f345c160c90 0x7f345c158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.595+0000 7f3460a00700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f345c0718e0 0x7f345c1f38f0 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.595+0000 7f3460a00700 1 --2- 172.21.15.89:0/570834525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f345c0b27f0 0x7f345c1f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.595+0000 7f3460a00700 1 -- 172.21.15.89:0/570834525 >> 172.21.15.89:0/570834525 conn(0x7f345c0cf190 msgr2=0x7f345c06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:42.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.595+0000 7f3460a00700 1 -- 172.21.15.89:0/570834525 shutdown_connections 2024-04-02T09:12:42.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.595+0000 7f3460a00700 1 -- 172.21.15.89:0/570834525 wait complete. 2024-04-02T09:12:42.596 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 110 2024-04-02T09:12:42.613 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:12:42.642 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:12:42.663 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:12:42.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.810+0000 7fd98a800700 1 Processor -- start 2024-04-02T09:12:42.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.810+0000 7fd98a800700 1 -- start start 2024-04-02T09:12:42.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.811+0000 7fd98a800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd984072580 0x7fd9840b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:42.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.811+0000 7fd98a800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9840b2180 con 0x7fd9840b3af0 2024-04-02T09:12:42.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.811+0000 7fd98a800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98415b100 con 0x7fd9840b2e50 2024-04-02T09:12:42.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.811+0000 7fd98a800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd98415b3e0 con 0x7fd984072580 2024-04-02T09:12:42.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.811+0000 7fd983400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd9840b3af0 0x7fd98415a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:54664/0 (socket says 172.21.15.89:54664) 2024-04-02T09:12:42.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.811+0000 7fd983400700 1 -- 172.21.15.89:0/225876966 learned_addr learned my addr 172.21.15.89:0/225876966 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:42.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.811+0000 7fd97aa00700 1 --2- 172.21.15.89:0/225876966 >> v2:172.21.15.113:3300/0 conn(0x7fd984072580 0x7fd9840b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:42.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.812+0000 7fd982a00700 1 -- 172.21.15.89:0/225876966 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555388591 0 0) 0x7fd9840b2180 con 0x7fd9840b3af0 2024-04-02T09:12:42.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.812+0000 7fd982a00700 1 -- 172.21.15.89:0/225876966 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9680057c0 con 0x7fd9840b3af0 2024-04-02T09:12:42.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.812+0000 7fd97aa00700 1 -- 172.21.15.89:0/225876966 >> v1:172.21.15.113:6789/0 conn(0x7fd9840b2e50 legacy=0x7fd9840709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:42.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.812+0000 7fd97aa00700 1 -- 172.21.15.89:0/225876966 >> v1:172.21.15.89:6789/0 conn(0x7fd9840b3af0 legacy=0x7fd98415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:42.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.812+0000 7fd97aa00700 1 -- 172.21.15.89:0/225876966 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9840b23f0 con 0x7fd984072580 2024-04-02T09:12:42.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.812+0000 7fd97aa00700 1 --2- 172.21.15.89:0/225876966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd984072580 0x7fd9840b2910 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fd96c009840 tx=0x7fd96c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10aad6cbbd4c8c12 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:42.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.812+0000 7fd982a00700 1 -- 172.21.15.89:0/225876966 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c011830 con 0x7fd984072580 2024-04-02T09:12:42.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.812+0000 7fd982a00700 1 -- 172.21.15.89:0/225876966 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96c011990 con 0x7fd984072580 2024-04-02T09:12:42.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.812+0000 7fd982a00700 1 -- 172.21.15.89:0/225876966 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c011d00 con 0x7fd984072580 2024-04-02T09:12:42.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.813+0000 7fd98a800700 1 -- 172.21.15.89:0/225876966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd984072580 msgr2=0x7fd9840b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:42.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.813+0000 7fd98a800700 1 --2- 172.21.15.89:0/225876966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd984072580 0x7fd9840b2910 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fd96c009840 tx=0x7fd96c009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:42.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.813+0000 7fd98a800700 1 -- 172.21.15.89:0/225876966 shutdown_connections 2024-04-02T09:12:42.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.813+0000 7fd98a800700 1 --2- 172.21.15.89:0/225876966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd984072580 0x7fd9840b2910 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.813+0000 7fd98a800700 1 -- 172.21.15.89:0/225876966 >> 172.21.15.89:0/225876966 conn(0x7fd9840cf190 msgr2=0x7fd9840cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:42.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.813+0000 7fd98a800700 1 -- 172.21.15.89:0/225876966 shutdown_connections 2024-04-02T09:12:42.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.813+0000 7fd98a800700 1 -- 172.21.15.89:0/225876966 wait complete. 2024-04-02T09:12:42.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.815+0000 7fd98a800700 1 Processor -- start 2024-04-02T09:12:42.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.815+0000 7fd98a800700 1 -- start start 2024-04-02T09:12:42.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.816+0000 7fd98a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd984072580 0x7fd984159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:42.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.816+0000 7fd98a800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9840b2e50 0x7fd98409c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:42.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.816+0000 7fd98a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9840b3af0 0x7fd9840afdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:42.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.816+0000 7fd98a800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98414f9f0 con 0x7fd9840b3af0 2024-04-02T09:12:42.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.816+0000 7fd98a800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd984158170 con 0x7fd9840b2e50 2024-04-02T09:12:42.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.816+0000 7fd98a800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd984158450 con 0x7fd984072580 2024-04-02T09:12:42.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.816+0000 7fd983e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9840b3af0 0x7fd9840afdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:42.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.816+0000 7fd983e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9840b3af0 0x7fd9840afdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59596/0 (socket says 172.21.15.89:59596) 2024-04-02T09:12:42.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.816+0000 7fd983e00700 1 -- 172.21.15.89:0/4064842852 learned_addr learned my addr 172.21.15.89:0/4064842852 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:42.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.816+0000 7fd983e00700 1 -- 172.21.15.89:0/4064842852 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd984072580 msgr2=0x7fd984159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:42.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.816+0000 7fd983e00700 1 --2- 172.21.15.89:0/4064842852 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd984072580 0x7fd984159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.816+0000 7fd983e00700 1 -- 172.21.15.89:0/4064842852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9840b2e50 msgr2=0x7fd98409c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:42.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.816+0000 7fd983e00700 1 --2- 172.21.15.89:0/4064842852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9840b2e50 0x7fd98409c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:42.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.816+0000 7fd983e00700 1 -- 172.21.15.89:0/4064842852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c0094a0 con 0x7fd9840b3af0 2024-04-02T09:12:42.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.817+0000 7fd983e00700 1 --2- 172.21.15.89:0/4064842852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9840b3af0 0x7fd9840afdd0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00cc50 tx=0x7fd97c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:42.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.817+0000 7fd97be00700 1 -- 172.21.15.89:0/4064842852 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c00ad80 con 0x7fd9840b3af0 2024-04-02T09:12:42.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.817+0000 7fd97be00700 1 -- 172.21.15.89:0/4064842852 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97c00ce50 con 0x7fd9840b3af0 2024-04-02T09:12:42.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.817+0000 7fd97be00700 1 -- 172.21.15.89:0/4064842852 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c003c40 con 0x7fd9840b3af0 2024-04-02T09:12:42.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.817+0000 7fd98a800700 1 -- 172.21.15.89:0/4064842852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd984158730 con 0x7fd9840b3af0 2024-04-02T09:12:42.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.817+0000 7fd98a800700 1 -- 172.21.15.89:0/4064842852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd984158cf0 con 0x7fd9840b3af0 2024-04-02T09:12:42.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.818+0000 7fd98a800700 1 -- 172.21.15.89:0/4064842852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd984158ee0 con 0x7fd9840b3af0 2024-04-02T09:12:42.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.819+0000 7fd97be00700 1 -- 172.21.15.89:0/4064842852 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd97c0050d0 con 0x7fd9840b3af0 2024-04-02T09:12:42.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.820+0000 7fd97be00700 1 --2- 172.21.15.89:0/4064842852 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd96006adb0 0x7fd96006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:42.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.820+0000 7fd97be00700 1 -- 172.21.15.89:0/4064842852 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fd97c089c10 con 0x7fd9840b3af0 2024-04-02T09:12:42.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.820+0000 7fd983400700 1 --2- 172.21.15.89:0/4064842852 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd96006adb0 0x7fd96006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:42.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.820+0000 7fd983400700 1 --2- 172.21.15.89:0/4064842852 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd96006adb0 0x7fd96006d270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd974001610 tx=0x7fd9740079c0 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:42.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:42.824+0000 7fd97be00700 1 -- 172.21.15.89:0/4064842852 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd97c054690 con 0x7fd9840b3af0 2024-04-02T09:12:43.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.024+0000 7fd98a800700 1 -- 172.21.15.89:0/4064842852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd984067720 con 0x7fd9840b3af0 2024-04-02T09:12:43.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.026+0000 7fd97be00700 1 -- 172.21.15.89:0/4064842852 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v227) v1 ==== 74+0+20464 (secure 0 0 0) 0x7fd97c057cb0 con 0x7fd9840b3af0 2024-04-02T09:12:43.026 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:43.026 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":227,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:12:40.108457+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:12:23.034869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-02T09:12:24.158560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:12:37.240568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:12:38.093999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:12:43.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.029+0000 7fd98a800700 1 -- 172.21.15.89:0/4064842852 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd96006adb0 msgr2=0x7fd96006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:43.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.029+0000 7fd98a800700 1 --2- 172.21.15.89:0/4064842852 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd96006adb0 0x7fd96006d270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd974001610 tx=0x7fd9740079c0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.029+0000 7fd98a800700 1 -- 172.21.15.89:0/4064842852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9840b3af0 msgr2=0x7fd9840afdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:43.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.029+0000 7fd98a800700 1 --2- 172.21.15.89:0/4064842852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9840b3af0 0x7fd9840afdd0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00cc50 tx=0x7fd97c00a340 comp rx=0 tx=0).stop 2024-04-02T09:12:43.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.029+0000 7fd98a800700 1 -- 172.21.15.89:0/4064842852 shutdown_connections 2024-04-02T09:12:43.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.029+0000 7fd98a800700 1 --2- 172.21.15.89:0/4064842852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9840b3af0 0x7fd9840afdd0 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.029+0000 7fd98a800700 1 --2- 172.21.15.89:0/4064842852 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd984072580 0x7fd984159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.029+0000 7fd98a800700 1 --2- 172.21.15.89:0/4064842852 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd96006adb0 0x7fd96006d270 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.029+0000 7fd98a800700 1 --2- 172.21.15.89:0/4064842852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9840b2e50 0x7fd98409c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.029+0000 7fd98a800700 1 -- 172.21.15.89:0/4064842852 >> 172.21.15.89:0/4064842852 conn(0x7fd9840cf190 msgr2=0x7fd98406dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:43.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.029+0000 7fd98a800700 1 -- 172.21.15.89:0/4064842852 shutdown_connections 2024-04-02T09:12:43.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.029+0000 7fd98a800700 1 -- 172.21.15.89:0/4064842852 wait complete. 2024-04-02T09:12:43.046 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T09:12:43.046 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:12:43.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.239+0000 7fbec2c00700 1 Processor -- start 2024-04-02T09:12:43.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.239+0000 7fbec2c00700 1 -- start start 2024-04-02T09:12:43.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.240+0000 7fbec2c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbebc0b3af0 0x7fbebc15a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:43.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.240+0000 7fbec2c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbebc072580 0x7fbebc0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:43.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.240+0000 7fbec2c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc0b2e50 0x7fbebc0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:43.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.240+0000 7fbec2c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbebc0b2180 con 0x7fbebc072580 2024-04-02T09:12:43.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.240+0000 7fbec2c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbebc15b100 con 0x7fbebc0b3af0 2024-04-02T09:12:43.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.240+0000 7fbec2c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc15b3e0 con 0x7fbebc0b2e50 2024-04-02T09:12:43.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.240+0000 7fbeb3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbebc072580 0x7fbebc0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:43.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.240+0000 7fbeb3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbebc072580 0x7fbebc0b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59606/0 (socket says 172.21.15.89:59606) 2024-04-02T09:12:43.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.240+0000 7fbeb3e00700 1 -- 172.21.15.89:0/4280549209 learned_addr learned my addr 172.21.15.89:0/4280549209 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:43.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.241+0000 7fbebbe00700 1 --2- 172.21.15.89:0/4280549209 >> v2:172.21.15.113:3300/0 conn(0x7fbebc0b3af0 0x7fbebc15a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:43.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.241+0000 7fbec0c00700 1 --2- 172.21.15.89:0/4280549209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc0b2e50 0x7fbebc0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:43.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.241+0000 7fbeb3e00700 1 -- 172.21.15.89:0/4280549209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc0b2e50 msgr2=0x7fbebc0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:43.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.241+0000 7fbeb3e00700 1 --2- 172.21.15.89:0/4280549209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc0b2e50 0x7fbebc0709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.241+0000 7fbeb3e00700 1 -- 172.21.15.89:0/4280549209 >> v2:172.21.15.113:3300/0 conn(0x7fbebc0b3af0 msgr2=0x7fbebc15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:43.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.241+0000 7fbeb3e00700 1 --2- 172.21.15.89:0/4280549209 >> v2:172.21.15.113:3300/0 conn(0x7fbebc0b3af0 0x7fbebc15a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.241+0000 7fbeb3e00700 1 -- 172.21.15.89:0/4280549209 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc0b23f0 con 0x7fbebc072580 2024-04-02T09:12:43.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.241+0000 7fbeb3e00700 1 --2- 172.21.15.89:0/4280549209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc072580 0x7fbebc0b2910 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7fbebc06b860 tx=0x7fbea4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fddf6bbab6c9d38 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:43.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.241+0000 7fbebb400700 1 -- 172.21.15.89:0/4280549209 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea400b870 con 0x7fbebc072580 2024-04-02T09:12:43.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.241+0000 7fbebb400700 1 -- 172.21.15.89:0/4280549209 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea400b9d0 con 0x7fbebc072580 2024-04-02T09:12:43.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.241+0000 7fbebb400700 1 -- 172.21.15.89:0/4280549209 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea400bd60 con 0x7fbebc072580 2024-04-02T09:12:43.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.242+0000 7fbec2c00700 1 -- 172.21.15.89:0/4280549209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc072580 msgr2=0x7fbebc0b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:43.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.242+0000 7fbec2c00700 1 --2- 172.21.15.89:0/4280549209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc072580 0x7fbebc0b2910 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7fbebc06b860 tx=0x7fbea4009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.242+0000 7fbec2c00700 1 -- 172.21.15.89:0/4280549209 shutdown_connections 2024-04-02T09:12:43.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.242+0000 7fbec2c00700 1 --2- 172.21.15.89:0/4280549209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc0b2e50 0x7fbebc0709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.242+0000 7fbec2c00700 1 --2- 172.21.15.89:0/4280549209 >> v2:172.21.15.113:3300/0 conn(0x7fbebc0b3af0 0x7fbebc15a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.242+0000 7fbec2c00700 1 --2- 172.21.15.89:0/4280549209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc072580 0x7fbebc0b2910 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.242+0000 7fbec2c00700 1 -- 172.21.15.89:0/4280549209 >> 172.21.15.89:0/4280549209 conn(0x7fbebc0cf190 msgr2=0x7fbebc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:43.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.242+0000 7fbec2c00700 1 -- 172.21.15.89:0/4280549209 shutdown_connections 2024-04-02T09:12:43.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.242+0000 7fbec2c00700 1 -- 172.21.15.89:0/4280549209 wait complete. 2024-04-02T09:12:43.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.244+0000 7fbec2c00700 1 Processor -- start 2024-04-02T09:12:43.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.244+0000 7fbec2c00700 1 -- start start 2024-04-02T09:12:43.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbec2c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc072580 0x7fbebc154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:43.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbec2c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc0b2e50 0x7fbebc150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:43.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbec2c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0b3af0 0x7fbebc14ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:43.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbec2c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc151360 con 0x7fbebc0b2e50 2024-04-02T09:12:43.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbec2c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc150470 con 0x7fbebc0b3af0 2024-04-02T09:12:43.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbec2c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc151590 con 0x7fbebc072580 2024-04-02T09:12:43.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbeb3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc0b2e50 0x7fbebc150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:43.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbeb3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc0b2e50 0x7fbebc150e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59614/0 (socket says 172.21.15.89:59614) 2024-04-02T09:12:43.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbeb3e00700 1 -- 172.21.15.89:0/444822004 learned_addr learned my addr 172.21.15.89:0/444822004 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:43.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbebbe00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc072580 0x7fbebc154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:43.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbec0c00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0b3af0 0x7fbebc14ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:43.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbeb3e00700 1 -- 172.21.15.89:0/444822004 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc072580 msgr2=0x7fbebc154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:43.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbeb3e00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc072580 0x7fbebc154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbeb3e00700 1 -- 172.21.15.89:0/444822004 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0b3af0 msgr2=0x7fbebc14ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:43.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbeb3e00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0b3af0 0x7fbebc14ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbeb3e00700 1 -- 172.21.15.89:0/444822004 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea4009550 con 0x7fbebc0b2e50 2024-04-02T09:12:43.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.245+0000 7fbec0c00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0b3af0 0x7fbebc14ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:43.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.246+0000 7fbeb3e00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc0b2e50 0x7fbebc150e20 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7fbea4000c00 tx=0x7fbea4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:43.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.246+0000 7fbeb8c00700 1 -- 172.21.15.89:0/444822004 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea4004520 con 0x7fbebc0b2e50 2024-04-02T09:12:43.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.246+0000 7fbeb8c00700 1 -- 172.21.15.89:0/444822004 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea401e070 con 0x7fbebc0b2e50 2024-04-02T09:12:43.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.246+0000 7fbeb8c00700 1 -- 172.21.15.89:0/444822004 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea40038c0 con 0x7fbebc0b2e50 2024-04-02T09:12:43.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.246+0000 7fbec2c00700 1 -- 172.21.15.89:0/444822004 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbebc151870 con 0x7fbebc0b2e50 2024-04-02T09:12:43.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.246+0000 7fbec2c00700 1 -- 172.21.15.89:0/444822004 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbebc151e30 con 0x7fbebc0b2e50 2024-04-02T09:12:43.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.247+0000 7fbec2c00700 1 -- 172.21.15.89:0/444822004 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbebc067720 con 0x7fbebc0b2e50 2024-04-02T09:12:43.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.248+0000 7fbeb8c00700 1 -- 172.21.15.89:0/444822004 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbea4003a20 con 0x7fbebc0b2e50 2024-04-02T09:12:43.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.249+0000 7fbeb8c00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe9806adb0 0x7fbe9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:43.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.249+0000 7fbeb8c00700 1 -- 172.21.15.89:0/444822004 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fbea4020070 con 0x7fbebc0b2e50 2024-04-02T09:12:43.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.249+0000 7fbebbe00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe9806adb0 0x7fbe9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:43.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.250+0000 7fbebbe00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe9806adb0 0x7fbe9806d270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0cccd0 tx=0x7fbeac008040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:43.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.254+0000 7fbeb8c00700 1 -- 172.21.15.89:0/444822004 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbea4090020 con 0x7fbebc0b2e50 2024-04-02T09:12:43.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.454+0000 7fbec2c00700 1 -- 172.21.15.89:0/444822004 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbebc15b6c0 con 0x7fbe9806adb0 2024-04-02T09:12:43.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.470+0000 7fbeb8c00700 1 -- 172.21.15.89:0/444822004 <== mgr.10923 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528387 (secure 0 0 0) 0x7fbeac009dc0 con 0x7fbe9806adb0 2024-04-02T09:12:43.476 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:43.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.476+0000 7fbec2c00700 1 -- 172.21.15.89:0/444822004 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe9806adb0 msgr2=0x7fbe9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:43.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.476+0000 7fbec2c00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe9806adb0 0x7fbe9806d270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0cccd0 tx=0x7fbeac008040 comp rx=0 tx=0).stop 2024-04-02T09:12:43.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.476+0000 7fbec2c00700 1 -- 172.21.15.89:0/444822004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc0b2e50 msgr2=0x7fbebc150e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:43.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.476+0000 7fbec2c00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc0b2e50 0x7fbebc150e20 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7fbea4000c00 tx=0x7fbea4004750 comp rx=0 tx=0).stop 2024-04-02T09:12:43.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.476+0000 7fbec2c00700 1 -- 172.21.15.89:0/444822004 shutdown_connections 2024-04-02T09:12:43.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.476+0000 7fbec2c00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe9806adb0 0x7fbe9806d270 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.476+0000 7fbec2c00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0b3af0 0x7fbebc14ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.476+0000 7fbec2c00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbebc072580 0x7fbebc154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.476+0000 7fbec2c00700 1 --2- 172.21.15.89:0/444822004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbebc0b2e50 0x7fbebc150e20 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.476+0000 7fbec2c00700 1 -- 172.21.15.89:0/444822004 >> 172.21.15.89:0/444822004 conn(0x7fbebc0cf190 msgr2=0x7fbebc0afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:43.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.476+0000 7fbec2c00700 1 -- 172.21.15.89:0/444822004 shutdown_connections 2024-04-02T09:12:43.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.476+0000 7fbec2c00700 1 -- 172.21.15.89:0/444822004 wait complete. 2024-04-02T09:12:43.481 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:12:43.494 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-02T09:12:42.802655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596333,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":671,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":360,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":407012,"kb_used_data":23520,"kb_used_omap":12,"kb_used_meta":383347,"kb_avail":754567708,"statfs":{"total":773094113280,"available":772677332992,"internally_reserved":0,"allocated":24084480,"data_stored":12786028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":392547896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.196694"},"pg_stats":[{"pgid":"1.0","version":"211'348","reported_seq":1215,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.126614+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:12:39.126614+0000","last_peered":"2024-04-02T09:12:39.126614+0000","last_clean":"2024-04-02T09:12:39.126614+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:12:39.126614+0000","last_undegraded":"2024-04-02T09:12:39.126614+0000","last_fullsized":"2024-04-02T09:12:39.126614+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1b","version":"222'1","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117125+0000","last_change":"2024-04-02T09:12:24.687221+0000","last_active":"2024-04-02T09:12:39.117125+0000","last_peered":"2024-04-02T09:12:39.117125+0000","last_clean":"2024-04-02T09:12:39.117125+0000","last_became_active":"2024-04-02T09:12:24.686953+0000","last_became_peered":"2024-04-02T09:12:24.686953+0000","last_unstale":"2024-04-02T09:12:39.117125+0000","last_undegraded":"2024-04-02T09:12:39.117125+0000","last_fullsized":"2024-04-02T09:12:39.117125+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:39:31.379977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.115858+0000","last_change":"2024-04-02T09:12:25.190434+0000","last_active":"2024-04-02T09:12:39.115858+0000","last_peered":"2024-04-02T09:12:39.115858+0000","last_clean":"2024-04-02T09:12:39.115858+0000","last_became_active":"2024-04-02T09:12:25.190187+0000","last_became_peered":"2024-04-02T09:12:25.190187+0000","last_unstale":"2024-04-02T09:12:39.115858+0000","last_undegraded":"2024-04-02T09:12:39.115858+0000","last_fullsized":"2024-04-02T09:12:39.115858+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:01.850942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1d","version":"226'2","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347881+0000","last_change":"2024-04-02T09:12:38.637685+0000","last_active":"2024-04-02T09:12:40.347881+0000","last_peered":"2024-04-02T09:12:40.347881+0000","last_clean":"2024-04-02T09:12:40.347881+0000","last_became_active":"2024-04-02T09:12:38.636649+0000","last_became_peered":"2024-04-02T09:12:38.636649+0000","last_unstale":"2024-04-02T09:12:40.347881+0000","last_undegraded":"2024-04-02T09:12:40.347881+0000","last_fullsized":"2024-04-02T09:12:40.347881+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:39:07.848314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347543+0000","last_change":"2024-04-02T09:12:39.135559+0000","last_active":"2024-04-02T09:12:40.347543+0000","last_peered":"2024-04-02T09:12:40.347543+0000","last_clean":"2024-04-02T09:12:40.347543+0000","last_became_active":"2024-04-02T09:12:39.135093+0000","last_became_peered":"2024-04-02T09:12:39.135093+0000","last_unstale":"2024-04-02T09:12:40.347543+0000","last_undegraded":"2024-04-02T09:12:40.347543+0000","last_fullsized":"2024-04-02T09:12:40.347543+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:39.344420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.126704+0000","last_change":"2024-04-02T09:12:24.698442+0000","last_active":"2024-04-02T09:12:39.126704+0000","last_peered":"2024-04-02T09:12:39.126704+0000","last_clean":"2024-04-02T09:12:39.126704+0000","last_became_active":"2024-04-02T09:12:24.698179+0000","last_became_peered":"2024-04-02T09:12:24.698179+0000","last_unstale":"2024-04-02T09:12:39.126704+0000","last_undegraded":"2024-04-02T09:12:39.126704+0000","last_fullsized":"2024-04-02T09:12:39.126704+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:48.278840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.126608+0000","last_change":"2024-04-02T09:12:25.203302+0000","last_active":"2024-04-02T09:12:39.126608+0000","last_peered":"2024-04-02T09:12:39.126608+0000","last_clean":"2024-04-02T09:12:39.126608+0000","last_became_active":"2024-04-02T09:12:25.202579+0000","last_became_peered":"2024-04-02T09:12:25.202579+0000","last_unstale":"2024-04-02T09:12:39.126608+0000","last_undegraded":"2024-04-02T09:12:39.126608+0000","last_fullsized":"2024-04-02T09:12:39.126608+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:25.603157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.6","version":"226'1","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348863+0000","last_change":"2024-04-02T09:12:38.637937+0000","last_active":"2024-04-02T09:12:40.348863+0000","last_peered":"2024-04-02T09:12:40.348863+0000","last_clean":"2024-04-02T09:12:40.348863+0000","last_became_active":"2024-04-02T09:12:38.636887+0000","last_became_peered":"2024-04-02T09:12:38.636887+0000","last_unstale":"2024-04-02T09:12:40.348863+0000","last_undegraded":"2024-04-02T09:12:40.348863+0000","last_fullsized":"2024-04-02T09:12:40.348863+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:19:29.015299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349429+0000","last_change":"2024-04-02T09:12:39.140604+0000","last_active":"2024-04-02T09:12:40.349429+0000","last_peered":"2024-04-02T09:12:40.349429+0000","last_clean":"2024-04-02T09:12:40.349429+0000","last_became_active":"2024-04-02T09:12:39.138536+0000","last_became_peered":"2024-04-02T09:12:39.138536+0000","last_unstale":"2024-04-02T09:12:40.349429+0000","last_undegraded":"2024-04-02T09:12:40.349429+0000","last_fullsized":"2024-04-02T09:12:40.349429+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:56:30.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116587+0000","last_change":"2024-04-02T09:12:24.695544+0000","last_active":"2024-04-02T09:12:39.116587+0000","last_peered":"2024-04-02T09:12:39.116587+0000","last_clean":"2024-04-02T09:12:39.116587+0000","last_became_active":"2024-04-02T09:12:24.695032+0000","last_became_peered":"2024-04-02T09:12:24.695032+0000","last_unstale":"2024-04-02T09:12:39.116587+0000","last_undegraded":"2024-04-02T09:12:39.116587+0000","last_fullsized":"2024-04-02T09:12:39.116587+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:41.705096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.120572+0000","last_change":"2024-04-02T09:12:25.203957+0000","last_active":"2024-04-02T09:12:39.120572+0000","last_peered":"2024-04-02T09:12:39.120572+0000","last_clean":"2024-04-02T09:12:39.120572+0000","last_became_active":"2024-04-02T09:12:25.203680+0000","last_became_peered":"2024-04-02T09:12:25.203680+0000","last_unstale":"2024-04-02T09:12:39.120572+0000","last_undegraded":"2024-04-02T09:12:39.120572+0000","last_fullsized":"2024-04-02T09:12:39.120572+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:31.267333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.7","version":"226'2","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349033+0000","last_change":"2024-04-02T09:12:38.633818+0000","last_active":"2024-04-02T09:12:40.349033+0000","last_peered":"2024-04-02T09:12:40.349033+0000","last_clean":"2024-04-02T09:12:40.349033+0000","last_became_active":"2024-04-02T09:12:38.633565+0000","last_became_peered":"2024-04-02T09:12:38.633565+0000","last_unstale":"2024-04-02T09:12:40.349033+0000","last_undegraded":"2024-04-02T09:12:40.349033+0000","last_fullsized":"2024-04-02T09:12:40.349033+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:21.508468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349105+0000","last_change":"2024-04-02T09:12:39.128790+0000","last_active":"2024-04-02T09:12:40.349105+0000","last_peered":"2024-04-02T09:12:40.349105+0000","last_clean":"2024-04-02T09:12:40.349105+0000","last_became_active":"2024-04-02T09:12:39.127115+0000","last_became_peered":"2024-04-02T09:12:39.127115+0000","last_unstale":"2024-04-02T09:12:40.349105+0000","last_undegraded":"2024-04-02T09:12:40.349105+0000","last_fullsized":"2024-04-02T09:12:40.349105+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:02.402506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116024+0000","last_change":"2024-04-02T09:12:24.694385+0000","last_active":"2024-04-02T09:12:39.116024+0000","last_peered":"2024-04-02T09:12:39.116024+0000","last_clean":"2024-04-02T09:12:39.116024+0000","last_became_active":"2024-04-02T09:12:24.693609+0000","last_became_peered":"2024-04-02T09:12:24.693609+0000","last_unstale":"2024-04-02T09:12:39.116024+0000","last_undegraded":"2024-04-02T09:12:39.116024+0000","last_fullsized":"2024-04-02T09:12:39.116024+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:46:08.099991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117041+0000","last_change":"2024-04-02T09:12:25.197899+0000","last_active":"2024-04-02T09:12:39.117041+0000","last_peered":"2024-04-02T09:12:39.117041+0000","last_clean":"2024-04-02T09:12:39.117041+0000","last_became_active":"2024-04-02T09:12:25.197631+0000","last_became_peered":"2024-04-02T09:12:25.197631+0000","last_unstale":"2024-04-02T09:12:39.117041+0000","last_undegraded":"2024-04-02T09:12:39.117041+0000","last_fullsized":"2024-04-02T09:12:39.117041+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:02.349107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112833+0000","last_change":"2024-04-02T09:12:38.629932+0000","last_active":"2024-04-02T09:12:40.112833+0000","last_peered":"2024-04-02T09:12:40.112833+0000","last_clean":"2024-04-02T09:12:40.112833+0000","last_became_active":"2024-04-02T09:12:38.629265+0000","last_became_peered":"2024-04-02T09:12:38.629265+0000","last_unstale":"2024-04-02T09:12:40.112833+0000","last_undegraded":"2024-04-02T09:12:40.112833+0000","last_fullsized":"2024-04-02T09:12:40.112833+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:16:41.585294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348316+0000","last_change":"2024-04-02T09:12:39.137028+0000","last_active":"2024-04-02T09:12:40.348316+0000","last_peered":"2024-04-02T09:12:40.348316+0000","last_clean":"2024-04-02T09:12:40.348316+0000","last_became_active":"2024-04-02T09:12:39.135462+0000","last_became_peered":"2024-04-02T09:12:39.135462+0000","last_unstale":"2024-04-02T09:12:40.348316+0000","last_undegraded":"2024-04-02T09:12:40.348316+0000","last_fullsized":"2024-04-02T09:12:40.348316+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:25.642123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.134179+0000","last_change":"2024-04-02T09:12:24.693539+0000","last_active":"2024-04-02T09:12:39.134179+0000","last_peered":"2024-04-02T09:12:39.134179+0000","last_clean":"2024-04-02T09:12:39.134179+0000","last_became_active":"2024-04-02T09:12:24.693275+0000","last_became_peered":"2024-04-02T09:12:24.693275+0000","last_unstale":"2024-04-02T09:12:39.134179+0000","last_undegraded":"2024-04-02T09:12:39.134179+0000","last_fullsized":"2024-04-02T09:12:39.134179+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:22:34.495765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114271+0000","last_change":"2024-04-02T09:12:25.194789+0000","last_active":"2024-04-02T09:12:39.114271+0000","last_peered":"2024-04-02T09:12:39.114271+0000","last_clean":"2024-04-02T09:12:39.114271+0000","last_became_active":"2024-04-02T09:12:25.192429+0000","last_became_peered":"2024-04-02T09:12:25.192429+0000","last_unstale":"2024-04-02T09:12:39.114271+0000","last_undegraded":"2024-04-02T09:12:39.114271+0000","last_fullsized":"2024-04-02T09:12:39.114271+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:11:59.558881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347920+0000","last_change":"2024-04-02T09:12:38.624346+0000","last_active":"2024-04-02T09:12:40.347920+0000","last_peered":"2024-04-02T09:12:40.347920+0000","last_clean":"2024-04-02T09:12:40.347920+0000","last_became_active":"2024-04-02T09:12:38.624090+0000","last_became_peered":"2024-04-02T09:12:38.624090+0000","last_unstale":"2024-04-02T09:12:40.347920+0000","last_undegraded":"2024-04-02T09:12:40.347920+0000","last_fullsized":"2024-04-02T09:12:40.347920+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:20:18.506907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349418+0000","last_change":"2024-04-02T09:12:39.143742+0000","last_active":"2024-04-02T09:12:40.349418+0000","last_peered":"2024-04-02T09:12:40.349418+0000","last_clean":"2024-04-02T09:12:40.349418+0000","last_became_active":"2024-04-02T09:12:39.143456+0000","last_became_peered":"2024-04-02T09:12:39.143456+0000","last_unstale":"2024-04-02T09:12:40.349418+0000","last_undegraded":"2024-04-02T09:12:40.349418+0000","last_fullsized":"2024-04-02T09:12:40.349418+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:33.582844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.113955+0000","last_change":"2024-04-02T09:12:24.693871+0000","last_active":"2024-04-02T09:12:39.113955+0000","last_peered":"2024-04-02T09:12:39.113955+0000","last_clean":"2024-04-02T09:12:39.113955+0000","last_became_active":"2024-04-02T09:12:24.693562+0000","last_became_peered":"2024-04-02T09:12:24.693562+0000","last_unstale":"2024-04-02T09:12:39.113955+0000","last_undegraded":"2024-04-02T09:12:39.113955+0000","last_fullsized":"2024-04-02T09:12:39.113955+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:48:24.676039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.119409+0000","last_change":"2024-04-02T09:12:25.196931+0000","last_active":"2024-04-02T09:12:39.119409+0000","last_peered":"2024-04-02T09:12:39.119409+0000","last_clean":"2024-04-02T09:12:39.119409+0000","last_became_active":"2024-04-02T09:12:25.194947+0000","last_became_peered":"2024-04-02T09:12:25.194947+0000","last_unstale":"2024-04-02T09:12:39.119409+0000","last_undegraded":"2024-04-02T09:12:39.119409+0000","last_fullsized":"2024-04-02T09:12:39.119409+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:21.964569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116880+0000","last_change":"2024-04-02T09:12:38.633018+0000","last_active":"2024-04-02T09:12:40.116880+0000","last_peered":"2024-04-02T09:12:40.116880+0000","last_clean":"2024-04-02T09:12:40.116880+0000","last_became_active":"2024-04-02T09:12:38.632717+0000","last_became_peered":"2024-04-02T09:12:38.632717+0000","last_unstale":"2024-04-02T09:12:40.116880+0000","last_undegraded":"2024-04-02T09:12:40.116880+0000","last_fullsized":"2024-04-02T09:12:40.116880+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:00:35.621780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349580+0000","last_change":"2024-04-02T09:12:39.143785+0000","last_active":"2024-04-02T09:12:40.349580+0000","last_peered":"2024-04-02T09:12:40.349580+0000","last_clean":"2024-04-02T09:12:40.349580+0000","last_became_active":"2024-04-02T09:12:39.142457+0000","last_became_peered":"2024-04-02T09:12:39.142457+0000","last_unstale":"2024-04-02T09:12:40.349580+0000","last_undegraded":"2024-04-02T09:12:40.349580+0000","last_fullsized":"2024-04-02T09:12:40.349580+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:35:02.087473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114480+0000","last_change":"2024-04-02T09:12:24.691094+0000","last_active":"2024-04-02T09:12:39.114480+0000","last_peered":"2024-04-02T09:12:39.114480+0000","last_clean":"2024-04-02T09:12:39.114480+0000","last_became_active":"2024-04-02T09:12:24.690782+0000","last_became_peered":"2024-04-02T09:12:24.690782+0000","last_unstale":"2024-04-02T09:12:39.114480+0000","last_undegraded":"2024-04-02T09:12:39.114480+0000","last_fullsized":"2024-04-02T09:12:39.114480+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:17.789082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116605+0000","last_change":"2024-04-02T09:12:25.200491+0000","last_active":"2024-04-02T09:12:39.116605+0000","last_peered":"2024-04-02T09:12:39.116605+0000","last_clean":"2024-04-02T09:12:39.116605+0000","last_became_active":"2024-04-02T09:12:25.200208+0000","last_became_peered":"2024-04-02T09:12:25.200208+0000","last_unstale":"2024-04-02T09:12:39.116605+0000","last_undegraded":"2024-04-02T09:12:39.116605+0000","last_fullsized":"2024-04-02T09:12:39.116605+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:53:44.514037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112619+0000","last_change":"2024-04-02T09:12:38.632816+0000","last_active":"2024-04-02T09:12:40.112619+0000","last_peered":"2024-04-02T09:12:40.112619+0000","last_clean":"2024-04-02T09:12:40.112619+0000","last_became_active":"2024-04-02T09:12:38.631996+0000","last_became_peered":"2024-04-02T09:12:38.631996+0000","last_unstale":"2024-04-02T09:12:40.112619+0000","last_undegraded":"2024-04-02T09:12:40.112619+0000","last_fullsized":"2024-04-02T09:12:40.112619+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:49:38.141564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349247+0000","last_change":"2024-04-02T09:12:39.141524+0000","last_active":"2024-04-02T09:12:40.349247+0000","last_peered":"2024-04-02T09:12:40.349247+0000","last_clean":"2024-04-02T09:12:40.349247+0000","last_became_active":"2024-04-02T09:12:39.140362+0000","last_became_peered":"2024-04-02T09:12:39.140362+0000","last_unstale":"2024-04-02T09:12:40.349247+0000","last_undegraded":"2024-04-02T09:12:40.349247+0000","last_fullsized":"2024-04-02T09:12:40.349247+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:48:50.055962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"223'1","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117479+0000","last_change":"2024-04-02T09:12:24.686970+0000","last_active":"2024-04-02T09:12:39.117479+0000","last_peered":"2024-04-02T09:12:39.117479+0000","last_clean":"2024-04-02T09:12:39.117479+0000","last_became_active":"2024-04-02T09:12:24.686686+0000","last_became_peered":"2024-04-02T09:12:24.686686+0000","last_unstale":"2024-04-02T09:12:39.117479+0000","last_undegraded":"2024-04-02T09:12:39.117479+0000","last_fullsized":"2024-04-02T09:12:39.117479+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:17.201623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116745+0000","last_change":"2024-04-02T09:12:25.196988+0000","last_active":"2024-04-02T09:12:39.116745+0000","last_peered":"2024-04-02T09:12:39.116745+0000","last_clean":"2024-04-02T09:12:39.116745+0000","last_became_active":"2024-04-02T09:12:25.196031+0000","last_became_peered":"2024-04-02T09:12:25.196031+0000","last_unstale":"2024-04-02T09:12:39.116745+0000","last_undegraded":"2024-04-02T09:12:39.116745+0000","last_fullsized":"2024-04-02T09:12:39.116745+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:50.975576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348967+0000","last_change":"2024-04-02T09:12:38.633373+0000","last_active":"2024-04-02T09:12:40.348967+0000","last_peered":"2024-04-02T09:12:40.348967+0000","last_clean":"2024-04-02T09:12:40.348967+0000","last_became_active":"2024-04-02T09:12:38.633108+0000","last_became_peered":"2024-04-02T09:12:38.633108+0000","last_unstale":"2024-04-02T09:12:40.348967+0000","last_undegraded":"2024-04-02T09:12:40.348967+0000","last_fullsized":"2024-04-02T09:12:40.348967+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:06.959393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348106+0000","last_change":"2024-04-02T09:12:39.126921+0000","last_active":"2024-04-02T09:12:40.348106+0000","last_peered":"2024-04-02T09:12:40.348106+0000","last_clean":"2024-04-02T09:12:40.348106+0000","last_became_active":"2024-04-02T09:12:39.126650+0000","last_became_peered":"2024-04-02T09:12:39.126650+0000","last_unstale":"2024-04-02T09:12:40.348106+0000","last_undegraded":"2024-04-02T09:12:40.348106+0000","last_fullsized":"2024-04-02T09:12:40.348106+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:11.640312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.7","version":"222'2","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.123499+0000","last_change":"2024-04-02T09:12:24.696113+0000","last_active":"2024-04-02T09:12:39.123499+0000","last_peered":"2024-04-02T09:12:39.123499+0000","last_clean":"2024-04-02T09:12:39.123499+0000","last_became_active":"2024-04-02T09:12:24.695191+0000","last_became_peered":"2024-04-02T09:12:24.695191+0000","last_unstale":"2024-04-02T09:12:39.123499+0000","last_undegraded":"2024-04-02T09:12:39.123499+0000","last_fullsized":"2024-04-02T09:12:39.123499+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:41:21.090616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117147+0000","last_change":"2024-04-02T09:12:25.196365+0000","last_active":"2024-04-02T09:12:39.117147+0000","last_peered":"2024-04-02T09:12:39.117147+0000","last_clean":"2024-04-02T09:12:39.117147+0000","last_became_active":"2024-04-02T09:12:25.191868+0000","last_became_peered":"2024-04-02T09:12:25.191868+0000","last_unstale":"2024-04-02T09:12:39.117147+0000","last_undegraded":"2024-04-02T09:12:39.117147+0000","last_fullsized":"2024-04-02T09:12:39.117147+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:43:29.229527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117688+0000","last_change":"2024-04-02T09:12:38.628570+0000","last_active":"2024-04-02T09:12:40.117688+0000","last_peered":"2024-04-02T09:12:40.117688+0000","last_clean":"2024-04-02T09:12:40.117688+0000","last_became_active":"2024-04-02T09:12:38.628252+0000","last_became_peered":"2024-04-02T09:12:38.628252+0000","last_unstale":"2024-04-02T09:12:40.117688+0000","last_undegraded":"2024-04-02T09:12:40.117688+0000","last_fullsized":"2024-04-02T09:12:40.117688+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:49:16.327081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112331+0000","last_change":"2024-04-02T09:12:39.144893+0000","last_active":"2024-04-02T09:12:40.112331+0000","last_peered":"2024-04-02T09:12:40.112331+0000","last_clean":"2024-04-02T09:12:40.112331+0000","last_became_active":"2024-04-02T09:12:39.144615+0000","last_became_peered":"2024-04-02T09:12:39.144615+0000","last_unstale":"2024-04-02T09:12:40.112331+0000","last_undegraded":"2024-04-02T09:12:40.112331+0000","last_fullsized":"2024-04-02T09:12:40.112331+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:01.838261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.120667+0000","last_change":"2024-04-02T09:12:24.683516+0000","last_active":"2024-04-02T09:12:39.120667+0000","last_peered":"2024-04-02T09:12:39.120667+0000","last_clean":"2024-04-02T09:12:39.120667+0000","last_became_active":"2024-04-02T09:12:24.683207+0000","last_became_peered":"2024-04-02T09:12:24.683207+0000","last_unstale":"2024-04-02T09:12:39.120667+0000","last_undegraded":"2024-04-02T09:12:39.120667+0000","last_fullsized":"2024-04-02T09:12:39.120667+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:19:15.076490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.127052+0000","last_change":"2024-04-02T09:12:25.205206+0000","last_active":"2024-04-02T09:12:39.127052+0000","last_peered":"2024-04-02T09:12:39.127052+0000","last_clean":"2024-04-02T09:12:39.127052+0000","last_became_active":"2024-04-02T09:12:25.204526+0000","last_became_peered":"2024-04-02T09:12:25.204526+0000","last_unstale":"2024-04-02T09:12:39.127052+0000","last_undegraded":"2024-04-02T09:12:39.127052+0000","last_fullsized":"2024-04-02T09:12:39.127052+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:33:41.191493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"226'2","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349262+0000","last_change":"2024-04-02T09:12:38.632806+0000","last_active":"2024-04-02T09:12:40.349262+0000","last_peered":"2024-04-02T09:12:40.349262+0000","last_clean":"2024-04-02T09:12:40.349262+0000","last_became_active":"2024-04-02T09:12:38.632489+0000","last_became_peered":"2024-04-02T09:12:38.632489+0000","last_unstale":"2024-04-02T09:12:40.349262+0000","last_undegraded":"2024-04-02T09:12:40.349262+0000","last_fullsized":"2024-04-02T09:12:40.349262+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:18:04.222535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349208+0000","last_change":"2024-04-02T09:12:39.123626+0000","last_active":"2024-04-02T09:12:40.349208+0000","last_peered":"2024-04-02T09:12:40.349208+0000","last_clean":"2024-04-02T09:12:40.349208+0000","last_became_active":"2024-04-02T09:12:39.123310+0000","last_became_peered":"2024-04-02T09:12:39.123310+0000","last_unstale":"2024-04-02T09:12:40.349208+0000","last_undegraded":"2024-04-02T09:12:40.349208+0000","last_fullsized":"2024-04-02T09:12:40.349208+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:18.395664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.119361+0000","last_change":"2024-04-02T09:12:24.699586+0000","last_active":"2024-04-02T09:12:39.119361+0000","last_peered":"2024-04-02T09:12:39.119361+0000","last_clean":"2024-04-02T09:12:39.119361+0000","last_became_active":"2024-04-02T09:12:24.697592+0000","last_became_peered":"2024-04-02T09:12:24.697592+0000","last_unstale":"2024-04-02T09:12:39.119361+0000","last_undegraded":"2024-04-02T09:12:39.119361+0000","last_fullsized":"2024-04-02T09:12:39.119361+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:53.870135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.126786+0000","last_change":"2024-04-02T09:12:25.189098+0000","last_active":"2024-04-02T09:12:39.126786+0000","last_peered":"2024-04-02T09:12:39.126786+0000","last_clean":"2024-04-02T09:12:39.126786+0000","last_became_active":"2024-04-02T09:12:25.187377+0000","last_became_peered":"2024-04-02T09:12:25.187377+0000","last_unstale":"2024-04-02T09:12:39.126786+0000","last_undegraded":"2024-04-02T09:12:39.126786+0000","last_fullsized":"2024-04-02T09:12:39.126786+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:58:52.752985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.f","version":"226'1","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349684+0000","last_change":"2024-04-02T09:12:38.638073+0000","last_active":"2024-04-02T09:12:40.349684+0000","last_peered":"2024-04-02T09:12:40.349684+0000","last_clean":"2024-04-02T09:12:40.349684+0000","last_became_active":"2024-04-02T09:12:38.637830+0000","last_became_peered":"2024-04-02T09:12:38.637830+0000","last_unstale":"2024-04-02T09:12:40.349684+0000","last_undegraded":"2024-04-02T09:12:40.349684+0000","last_fullsized":"2024-04-02T09:12:40.349684+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:11:20.511970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349415+0000","last_change":"2024-04-02T09:12:39.135926+0000","last_active":"2024-04-02T09:12:40.349415+0000","last_peered":"2024-04-02T09:12:40.349415+0000","last_clean":"2024-04-02T09:12:40.349415+0000","last_became_active":"2024-04-02T09:12:39.135625+0000","last_became_peered":"2024-04-02T09:12:39.135625+0000","last_unstale":"2024-04-02T09:12:40.349415+0000","last_undegraded":"2024-04-02T09:12:40.349415+0000","last_fullsized":"2024-04-02T09:12:40.349415+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:49:00.944821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.126297+0000","last_change":"2024-04-02T09:12:24.691812+0000","last_active":"2024-04-02T09:12:39.126297+0000","last_peered":"2024-04-02T09:12:39.126297+0000","last_clean":"2024-04-02T09:12:39.126297+0000","last_became_active":"2024-04-02T09:12:24.688935+0000","last_became_peered":"2024-04-02T09:12:24.688935+0000","last_unstale":"2024-04-02T09:12:39.126297+0000","last_undegraded":"2024-04-02T09:12:39.126297+0000","last_fullsized":"2024-04-02T09:12:39.126297+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:53.020585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.113633+0000","last_change":"2024-04-02T09:12:25.195866+0000","last_active":"2024-04-02T09:12:39.113633+0000","last_peered":"2024-04-02T09:12:39.113633+0000","last_clean":"2024-04-02T09:12:39.113633+0000","last_became_active":"2024-04-02T09:12:25.195430+0000","last_became_peered":"2024-04-02T09:12:25.195430+0000","last_unstale":"2024-04-02T09:12:39.113633+0000","last_undegraded":"2024-04-02T09:12:39.113633+0000","last_fullsized":"2024-04-02T09:12:39.113633+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:42:28.562906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348451+0000","last_change":"2024-04-02T09:12:38.638164+0000","last_active":"2024-04-02T09:12:40.348451+0000","last_peered":"2024-04-02T09:12:40.348451+0000","last_clean":"2024-04-02T09:12:40.348451+0000","last_became_active":"2024-04-02T09:12:38.637882+0000","last_became_peered":"2024-04-02T09:12:38.637882+0000","last_unstale":"2024-04-02T09:12:40.348451+0000","last_undegraded":"2024-04-02T09:12:40.348451+0000","last_fullsized":"2024-04-02T09:12:40.348451+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:34.894177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348111+0000","last_change":"2024-04-02T09:12:39.139332+0000","last_active":"2024-04-02T09:12:40.348111+0000","last_peered":"2024-04-02T09:12:40.348111+0000","last_clean":"2024-04-02T09:12:40.348111+0000","last_became_active":"2024-04-02T09:12:39.139084+0000","last_became_peered":"2024-04-02T09:12:39.139084+0000","last_unstale":"2024-04-02T09:12:40.348111+0000","last_undegraded":"2024-04-02T09:12:40.348111+0000","last_fullsized":"2024-04-02T09:12:40.348111+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:06:26.150667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"222'2","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.123006+0000","last_change":"2024-04-02T09:12:24.693402+0000","last_active":"2024-04-02T09:12:39.123006+0000","last_peered":"2024-04-02T09:12:39.123006+0000","last_clean":"2024-04-02T09:12:39.123006+0000","last_became_active":"2024-04-02T09:12:24.693047+0000","last_became_peered":"2024-04-02T09:12:24.693047+0000","last_unstale":"2024-04-02T09:12:39.123006+0000","last_undegraded":"2024-04-02T09:12:39.123006+0000","last_fullsized":"2024-04-02T09:12:39.123006+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:15:17.948098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.115937+0000","last_change":"2024-04-02T09:12:25.195964+0000","last_active":"2024-04-02T09:12:39.115937+0000","last_peered":"2024-04-02T09:12:39.115937+0000","last_clean":"2024-04-02T09:12:39.115937+0000","last_became_active":"2024-04-02T09:12:25.195716+0000","last_became_peered":"2024-04-02T09:12:25.195716+0000","last_unstale":"2024-04-02T09:12:39.115937+0000","last_undegraded":"2024-04-02T09:12:39.115937+0000","last_fullsized":"2024-04-02T09:12:39.115937+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:25.408279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.d","version":"226'1","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348038+0000","last_change":"2024-04-02T09:12:38.633567+0000","last_active":"2024-04-02T09:12:40.348038+0000","last_peered":"2024-04-02T09:12:40.348038+0000","last_clean":"2024-04-02T09:12:40.348038+0000","last_became_active":"2024-04-02T09:12:38.633209+0000","last_became_peered":"2024-04-02T09:12:38.633209+0000","last_unstale":"2024-04-02T09:12:40.348038+0000","last_undegraded":"2024-04-02T09:12:40.348038+0000","last_fullsized":"2024-04-02T09:12:40.348038+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:25:28.960133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112513+0000","last_change":"2024-04-02T09:12:39.124846+0000","last_active":"2024-04-02T09:12:40.112513+0000","last_peered":"2024-04-02T09:12:40.112513+0000","last_clean":"2024-04-02T09:12:40.112513+0000","last_became_active":"2024-04-02T09:12:39.124595+0000","last_became_peered":"2024-04-02T09:12:39.124595+0000","last_unstale":"2024-04-02T09:12:40.112513+0000","last_undegraded":"2024-04-02T09:12:40.112513+0000","last_fullsized":"2024-04-02T09:12:40.112513+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:58.288706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114578+0000","last_change":"2024-04-02T09:12:24.692444+0000","last_active":"2024-04-02T09:12:39.114578+0000","last_peered":"2024-04-02T09:12:39.114578+0000","last_clean":"2024-04-02T09:12:39.114578+0000","last_became_active":"2024-04-02T09:12:24.692165+0000","last_became_peered":"2024-04-02T09:12:24.692165+0000","last_unstale":"2024-04-02T09:12:39.114578+0000","last_undegraded":"2024-04-02T09:12:39.114578+0000","last_fullsized":"2024-04-02T09:12:39.114578+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:29.826236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114388+0000","last_change":"2024-04-02T09:12:25.190936+0000","last_active":"2024-04-02T09:12:39.114388+0000","last_peered":"2024-04-02T09:12:39.114388+0000","last_clean":"2024-04-02T09:12:39.114388+0000","last_became_active":"2024-04-02T09:12:25.189970+0000","last_became_peered":"2024-04-02T09:12:25.189970+0000","last_unstale":"2024-04-02T09:12:39.114388+0000","last_undegraded":"2024-04-02T09:12:39.114388+0000","last_fullsized":"2024-04-02T09:12:39.114388+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:57.372310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347519+0000","last_change":"2024-04-02T09:12:38.630574+0000","last_active":"2024-04-02T09:12:40.347519+0000","last_peered":"2024-04-02T09:12:40.347519+0000","last_clean":"2024-04-02T09:12:40.347519+0000","last_became_active":"2024-04-02T09:12:38.628873+0000","last_became_peered":"2024-04-02T09:12:38.628873+0000","last_unstale":"2024-04-02T09:12:40.347519+0000","last_undegraded":"2024-04-02T09:12:40.347519+0000","last_fullsized":"2024-04-02T09:12:40.347519+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:04.426115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348238+0000","last_change":"2024-04-02T09:12:39.134066+0000","last_active":"2024-04-02T09:12:40.348238+0000","last_peered":"2024-04-02T09:12:40.348238+0000","last_clean":"2024-04-02T09:12:40.348238+0000","last_became_active":"2024-04-02T09:12:39.131593+0000","last_became_peered":"2024-04-02T09:12:39.131593+0000","last_unstale":"2024-04-02T09:12:40.348238+0000","last_undegraded":"2024-04-02T09:12:40.348238+0000","last_fullsized":"2024-04-02T09:12:40.348238+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:45.203475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"222'1","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.123187+0000","last_change":"2024-04-02T09:12:24.688637+0000","last_active":"2024-04-02T09:12:39.123187+0000","last_peered":"2024-04-02T09:12:39.123187+0000","last_clean":"2024-04-02T09:12:39.123187+0000","last_became_active":"2024-04-02T09:12:24.688321+0000","last_became_peered":"2024-04-02T09:12:24.688321+0000","last_unstale":"2024-04-02T09:12:39.123187+0000","last_undegraded":"2024-04-02T09:12:39.123187+0000","last_fullsized":"2024-04-02T09:12:39.123187+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:41:28.231249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.118045+0000","last_change":"2024-04-02T09:12:25.193723+0000","last_active":"2024-04-02T09:12:39.118045+0000","last_peered":"2024-04-02T09:12:39.118045+0000","last_clean":"2024-04-02T09:12:39.118045+0000","last_became_active":"2024-04-02T09:12:25.193367+0000","last_became_peered":"2024-04-02T09:12:25.193367+0000","last_unstale":"2024-04-02T09:12:39.118045+0000","last_undegraded":"2024-04-02T09:12:39.118045+0000","last_fullsized":"2024-04-02T09:12:39.118045+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:48.218780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.b","version":"226'2","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349531+0000","last_change":"2024-04-02T09:12:38.636198+0000","last_active":"2024-04-02T09:12:40.349531+0000","last_peered":"2024-04-02T09:12:40.349531+0000","last_clean":"2024-04-02T09:12:40.349531+0000","last_became_active":"2024-04-02T09:12:38.633531+0000","last_became_peered":"2024-04-02T09:12:38.633531+0000","last_unstale":"2024-04-02T09:12:40.349531+0000","last_undegraded":"2024-04-02T09:12:40.349531+0000","last_fullsized":"2024-04-02T09:12:40.349531+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:06:25.144133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348478+0000","last_change":"2024-04-02T09:12:39.124009+0000","last_active":"2024-04-02T09:12:40.348478+0000","last_peered":"2024-04-02T09:12:40.348478+0000","last_clean":"2024-04-02T09:12:40.348478+0000","last_became_active":"2024-04-02T09:12:39.123484+0000","last_became_peered":"2024-04-02T09:12:39.123484+0000","last_unstale":"2024-04-02T09:12:40.348478+0000","last_undegraded":"2024-04-02T09:12:40.348478+0000","last_fullsized":"2024-04-02T09:12:40.348478+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:24:10.675482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"222'2","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.118077+0000","last_change":"2024-04-02T09:12:24.690213+0000","last_active":"2024-04-02T09:12:39.118077+0000","last_peered":"2024-04-02T09:12:39.118077+0000","last_clean":"2024-04-02T09:12:39.118077+0000","last_became_active":"2024-04-02T09:12:24.689813+0000","last_became_peered":"2024-04-02T09:12:24.689813+0000","last_unstale":"2024-04-02T09:12:39.118077+0000","last_undegraded":"2024-04-02T09:12:39.118077+0000","last_fullsized":"2024-04-02T09:12:39.118077+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:11.366124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.130606+0000","last_change":"2024-04-02T09:12:25.196053+0000","last_active":"2024-04-02T09:12:39.130606+0000","last_peered":"2024-04-02T09:12:39.130606+0000","last_clean":"2024-04-02T09:12:39.130606+0000","last_became_active":"2024-04-02T09:12:25.195799+0000","last_became_peered":"2024-04-02T09:12:25.195799+0000","last_unstale":"2024-04-02T09:12:39.130606+0000","last_undegraded":"2024-04-02T09:12:39.130606+0000","last_fullsized":"2024-04-02T09:12:39.130606+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:26:33.117159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112459+0000","last_change":"2024-04-02T09:12:38.638977+0000","last_active":"2024-04-02T09:12:40.112459+0000","last_peered":"2024-04-02T09:12:40.112459+0000","last_clean":"2024-04-02T09:12:40.112459+0000","last_became_active":"2024-04-02T09:12:38.638666+0000","last_became_peered":"2024-04-02T09:12:38.638666+0000","last_unstale":"2024-04-02T09:12:40.112459+0000","last_undegraded":"2024-04-02T09:12:40.112459+0000","last_fullsized":"2024-04-02T09:12:40.112459+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:40:47.207533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117654+0000","last_change":"2024-04-02T09:12:39.142960+0000","last_active":"2024-04-02T09:12:40.117654+0000","last_peered":"2024-04-02T09:12:40.117654+0000","last_clean":"2024-04-02T09:12:40.117654+0000","last_became_active":"2024-04-02T09:12:39.142500+0000","last_became_peered":"2024-04-02T09:12:39.142500+0000","last_unstale":"2024-04-02T09:12:40.117654+0000","last_undegraded":"2024-04-02T09:12:40.117654+0000","last_fullsized":"2024-04-02T09:12:40.117654+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:25:29.966667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"222'1","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116664+0000","last_change":"2024-04-02T09:12:24.695567+0000","last_active":"2024-04-02T09:12:39.116664+0000","last_peered":"2024-04-02T09:12:39.116664+0000","last_clean":"2024-04-02T09:12:39.116664+0000","last_became_active":"2024-04-02T09:12:24.695267+0000","last_became_peered":"2024-04-02T09:12:24.695267+0000","last_unstale":"2024-04-02T09:12:39.116664+0000","last_undegraded":"2024-04-02T09:12:39.116664+0000","last_fullsized":"2024-04-02T09:12:39.116664+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:02.378346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117284+0000","last_change":"2024-04-02T09:12:25.196196+0000","last_active":"2024-04-02T09:12:39.117284+0000","last_peered":"2024-04-02T09:12:39.117284+0000","last_clean":"2024-04-02T09:12:39.117284+0000","last_became_active":"2024-04-02T09:12:25.195967+0000","last_became_peered":"2024-04-02T09:12:25.195967+0000","last_unstale":"2024-04-02T09:12:39.117284+0000","last_undegraded":"2024-04-02T09:12:39.117284+0000","last_fullsized":"2024-04-02T09:12:39.117284+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:41.683998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112356+0000","last_change":"2024-04-02T09:12:38.633999+0000","last_active":"2024-04-02T09:12:40.112356+0000","last_peered":"2024-04-02T09:12:40.112356+0000","last_clean":"2024-04-02T09:12:40.112356+0000","last_became_active":"2024-04-02T09:12:38.632478+0000","last_became_peered":"2024-04-02T09:12:38.632478+0000","last_unstale":"2024-04-02T09:12:40.112356+0000","last_undegraded":"2024-04-02T09:12:40.112356+0000","last_fullsized":"2024-04-02T09:12:40.112356+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:18:04.222535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347683+0000","last_change":"2024-04-02T09:12:39.128396+0000","last_active":"2024-04-02T09:12:40.347683+0000","last_peered":"2024-04-02T09:12:40.347683+0000","last_clean":"2024-04-02T09:12:40.347683+0000","last_became_active":"2024-04-02T09:12:39.128118+0000","last_became_peered":"2024-04-02T09:12:39.128118+0000","last_unstale":"2024-04-02T09:12:40.347683+0000","last_undegraded":"2024-04-02T09:12:40.347683+0000","last_fullsized":"2024-04-02T09:12:40.347683+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:39:46.325320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"222'8","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.120472+0000","last_change":"2024-04-02T09:12:24.692205+0000","last_active":"2024-04-02T09:12:39.120472+0000","last_peered":"2024-04-02T09:12:39.120472+0000","last_clean":"2024-04-02T09:12:39.120472+0000","last_became_active":"2024-04-02T09:12:24.690462+0000","last_became_peered":"2024-04-02T09:12:24.690462+0000","last_unstale":"2024-04-02T09:12:39.120472+0000","last_undegraded":"2024-04-02T09:12:39.120472+0000","last_fullsized":"2024-04-02T09:12:39.120472+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:00:14.796518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.115999+0000","last_change":"2024-04-02T09:12:25.201746+0000","last_active":"2024-04-02T09:12:39.115999+0000","last_peered":"2024-04-02T09:12:39.115999+0000","last_clean":"2024-04-02T09:12:39.115999+0000","last_became_active":"2024-04-02T09:12:25.201445+0000","last_became_peered":"2024-04-02T09:12:25.201445+0000","last_unstale":"2024-04-02T09:12:39.115999+0000","last_undegraded":"2024-04-02T09:12:39.115999+0000","last_fullsized":"2024-04-02T09:12:39.115999+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:06:08.648823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349299+0000","last_change":"2024-04-02T09:12:38.633797+0000","last_active":"2024-04-02T09:12:40.349299+0000","last_peered":"2024-04-02T09:12:40.349299+0000","last_clean":"2024-04-02T09:12:40.349299+0000","last_became_active":"2024-04-02T09:12:38.632294+0000","last_became_peered":"2024-04-02T09:12:38.632294+0000","last_unstale":"2024-04-02T09:12:40.349299+0000","last_undegraded":"2024-04-02T09:12:40.349299+0000","last_fullsized":"2024-04-02T09:12:40.349299+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:43:42.159134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.113378+0000","last_change":"2024-04-02T09:12:39.132856+0000","last_active":"2024-04-02T09:12:40.113378+0000","last_peered":"2024-04-02T09:12:40.113378+0000","last_clean":"2024-04-02T09:12:40.113378+0000","last_became_active":"2024-04-02T09:12:39.128056+0000","last_became_peered":"2024-04-02T09:12:39.128056+0000","last_unstale":"2024-04-02T09:12:40.113378+0000","last_undegraded":"2024-04-02T09:12:40.113378+0000","last_fullsized":"2024-04-02T09:12:40.113378+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:05.432649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.123091+0000","last_change":"2024-04-02T09:12:24.691834+0000","last_active":"2024-04-02T09:12:39.123091+0000","last_peered":"2024-04-02T09:12:39.123091+0000","last_clean":"2024-04-02T09:12:39.123091+0000","last_became_active":"2024-04-02T09:12:24.689829+0000","last_became_peered":"2024-04-02T09:12:24.689829+0000","last_unstale":"2024-04-02T09:12:39.123091+0000","last_undegraded":"2024-04-02T09:12:39.123091+0000","last_fullsized":"2024-04-02T09:12:39.123091+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:56.694970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114297+0000","last_change":"2024-04-02T09:12:25.203805+0000","last_active":"2024-04-02T09:12:39.114297+0000","last_peered":"2024-04-02T09:12:39.114297+0000","last_clean":"2024-04-02T09:12:39.114297+0000","last_became_active":"2024-04-02T09:12:25.203516+0000","last_became_peered":"2024-04-02T09:12:25.203516+0000","last_unstale":"2024-04-02T09:12:39.114297+0000","last_undegraded":"2024-04-02T09:12:39.114297+0000","last_fullsized":"2024-04-02T09:12:39.114297+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:00:15.805719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348958+0000","last_change":"2024-04-02T09:12:38.638310+0000","last_active":"2024-04-02T09:12:40.348958+0000","last_peered":"2024-04-02T09:12:40.348958+0000","last_clean":"2024-04-02T09:12:40.348958+0000","last_became_active":"2024-04-02T09:12:38.638018+0000","last_became_peered":"2024-04-02T09:12:38.638018+0000","last_unstale":"2024-04-02T09:12:40.348958+0000","last_undegraded":"2024-04-02T09:12:40.348958+0000","last_fullsized":"2024-04-02T09:12:40.348958+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:38.337886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348836+0000","last_change":"2024-04-02T09:12:39.142582+0000","last_active":"2024-04-02T09:12:40.348836+0000","last_peered":"2024-04-02T09:12:40.348836+0000","last_clean":"2024-04-02T09:12:40.348836+0000","last_became_active":"2024-04-02T09:12:39.142245+0000","last_became_peered":"2024-04-02T09:12:39.142245+0000","last_unstale":"2024-04-02T09:12:40.348836+0000","last_undegraded":"2024-04-02T09:12:40.348836+0000","last_fullsized":"2024-04-02T09:12:40.348836+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:45:28.462172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116430+0000","last_change":"2024-04-02T09:12:24.695778+0000","last_active":"2024-04-02T09:12:39.116430+0000","last_peered":"2024-04-02T09:12:39.116430+0000","last_clean":"2024-04-02T09:12:39.116430+0000","last_became_active":"2024-04-02T09:12:24.695349+0000","last_became_peered":"2024-04-02T09:12:24.695349+0000","last_unstale":"2024-04-02T09:12:39.116430+0000","last_undegraded":"2024-04-02T09:12:39.116430+0000","last_fullsized":"2024-04-02T09:12:39.116430+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:33:51.021792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.121582+0000","last_change":"2024-04-02T09:12:25.205479+0000","last_active":"2024-04-02T09:12:39.121582+0000","last_peered":"2024-04-02T09:12:39.121582+0000","last_clean":"2024-04-02T09:12:39.121582+0000","last_became_active":"2024-04-02T09:12:25.205210+0000","last_became_peered":"2024-04-02T09:12:25.205210+0000","last_unstale":"2024-04-02T09:12:39.121582+0000","last_undegraded":"2024-04-02T09:12:39.121582+0000","last_fullsized":"2024-04-02T09:12:39.121582+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:03.985819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"226'6","reported_seq":17,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348721+0000","last_change":"2024-04-02T09:12:38.633159+0000","last_active":"2024-04-02T09:12:40.348721+0000","last_peered":"2024-04-02T09:12:40.348721+0000","last_clean":"2024-04-02T09:12:40.348721+0000","last_became_active":"2024-04-02T09:12:38.632508+0000","last_became_peered":"2024-04-02T09:12:38.632508+0000","last_unstale":"2024-04-02T09:12:40.348721+0000","last_undegraded":"2024-04-02T09:12:40.348721+0000","last_fullsized":"2024-04-02T09:12:40.348721+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:49:38.141564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348595+0000","last_change":"2024-04-02T09:12:39.142408+0000","last_active":"2024-04-02T09:12:40.348595+0000","last_peered":"2024-04-02T09:12:40.348595+0000","last_clean":"2024-04-02T09:12:40.348595+0000","last_became_active":"2024-04-02T09:12:39.142140+0000","last_became_peered":"2024-04-02T09:12:39.142140+0000","last_unstale":"2024-04-02T09:12:40.348595+0000","last_undegraded":"2024-04-02T09:12:40.348595+0000","last_fullsized":"2024-04-02T09:12:40.348595+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:42:49.185298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.13","version":"222'2","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116884+0000","last_change":"2024-04-02T09:12:24.694118+0000","last_active":"2024-04-02T09:12:39.116884+0000","last_peered":"2024-04-02T09:12:39.116884+0000","last_clean":"2024-04-02T09:12:39.116884+0000","last_became_active":"2024-04-02T09:12:24.693829+0000","last_became_peered":"2024-04-02T09:12:24.693829+0000","last_unstale":"2024-04-02T09:12:39.116884+0000","last_undegraded":"2024-04-02T09:12:39.116884+0000","last_fullsized":"2024-04-02T09:12:39.116884+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:58:51.743784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.122178+0000","last_change":"2024-04-02T09:12:25.192240+0000","last_active":"2024-04-02T09:12:39.122178+0000","last_peered":"2024-04-02T09:12:39.122178+0000","last_clean":"2024-04-02T09:12:39.122178+0000","last_became_active":"2024-04-02T09:12:25.191954+0000","last_became_peered":"2024-04-02T09:12:25.191954+0000","last_unstale":"2024-04-02T09:12:39.122178+0000","last_undegraded":"2024-04-02T09:12:39.122178+0000","last_fullsized":"2024-04-02T09:12:39.122178+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:26:33.117159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"226'3","reported_seq":14,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.118052+0000","last_change":"2024-04-02T09:12:38.626779+0000","last_active":"2024-04-02T09:12:40.118052+0000","last_peered":"2024-04-02T09:12:40.118052+0000","last_clean":"2024-04-02T09:12:40.118052+0000","last_became_active":"2024-04-02T09:12:38.626438+0000","last_became_peered":"2024-04-02T09:12:38.626438+0000","last_unstale":"2024-04-02T09:12:40.118052+0000","last_undegraded":"2024-04-02T09:12:40.118052+0000","last_fullsized":"2024-04-02T09:12:40.118052+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:20:18.506907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349069+0000","last_change":"2024-04-02T09:12:39.121376+0000","last_active":"2024-04-02T09:12:40.349069+0000","last_peered":"2024-04-02T09:12:40.349069+0000","last_clean":"2024-04-02T09:12:40.349069+0000","last_became_active":"2024-04-02T09:12:39.121100+0000","last_became_peered":"2024-04-02T09:12:39.121100+0000","last_unstale":"2024-04-02T09:12:40.349069+0000","last_undegraded":"2024-04-02T09:12:40.349069+0000","last_fullsized":"2024-04-02T09:12:40.349069+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:01.838261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.14","version":"223'8","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.118056+0000","last_change":"2024-04-02T09:12:24.689123+0000","last_active":"2024-04-02T09:12:39.118056+0000","last_peered":"2024-04-02T09:12:39.118056+0000","last_clean":"2024-04-02T09:12:39.118056+0000","last_became_active":"2024-04-02T09:12:24.688848+0000","last_became_peered":"2024-04-02T09:12:24.688848+0000","last_unstale":"2024-04-02T09:12:39.118056+0000","last_undegraded":"2024-04-02T09:12:39.118056+0000","last_fullsized":"2024-04-02T09:12:39.118056+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:24.593956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2805,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.122691+0000","last_change":"2024-04-02T09:12:25.199901+0000","last_active":"2024-04-02T09:12:39.122691+0000","last_peered":"2024-04-02T09:12:39.122691+0000","last_clean":"2024-04-02T09:12:39.122691+0000","last_became_active":"2024-04-02T09:12:25.198989+0000","last_became_peered":"2024-04-02T09:12:25.198989+0000","last_unstale":"2024-04-02T09:12:39.122691+0000","last_undegraded":"2024-04-02T09:12:39.122691+0000","last_fullsized":"2024-04-02T09:12:39.122691+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:30.835437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116796+0000","last_change":"2024-04-02T09:12:38.634647+0000","last_active":"2024-04-02T09:12:40.116796+0000","last_peered":"2024-04-02T09:12:40.116796+0000","last_clean":"2024-04-02T09:12:40.116796+0000","last_became_active":"2024-04-02T09:12:38.634254+0000","last_became_peered":"2024-04-02T09:12:38.634254+0000","last_unstale":"2024-04-02T09:12:40.116796+0000","last_undegraded":"2024-04-02T09:12:40.116796+0000","last_fullsized":"2024-04-02T09:12:40.116796+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:51:50.442138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348361+0000","last_change":"2024-04-02T09:12:39.132916+0000","last_active":"2024-04-02T09:12:40.348361+0000","last_peered":"2024-04-02T09:12:40.348361+0000","last_clean":"2024-04-02T09:12:40.348361+0000","last_became_active":"2024-04-02T09:12:39.132638+0000","last_became_peered":"2024-04-02T09:12:39.132638+0000","last_unstale":"2024-04-02T09:12:40.348361+0000","last_undegraded":"2024-04-02T09:12:40.348361+0000","last_fullsized":"2024-04-02T09:12:40.348361+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:37:05.275434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"222'3","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.122883+0000","last_change":"2024-04-02T09:12:24.694400+0000","last_active":"2024-04-02T09:12:39.122883+0000","last_peered":"2024-04-02T09:12:39.122883+0000","last_clean":"2024-04-02T09:12:39.122883+0000","last_became_active":"2024-04-02T09:12:24.694118+0000","last_became_peered":"2024-04-02T09:12:24.694118+0000","last_unstale":"2024-04-02T09:12:39.122883+0000","last_undegraded":"2024-04-02T09:12:39.122883+0000","last_fullsized":"2024-04-02T09:12:39.122883+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:00.841741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117972+0000","last_change":"2024-04-02T09:12:25.202617+0000","last_active":"2024-04-02T09:12:39.117972+0000","last_peered":"2024-04-02T09:12:39.117972+0000","last_clean":"2024-04-02T09:12:39.117972+0000","last_became_active":"2024-04-02T09:12:25.201786+0000","last_became_peered":"2024-04-02T09:12:25.201786+0000","last_unstale":"2024-04-02T09:12:39.117972+0000","last_undegraded":"2024-04-02T09:12:39.117972+0000","last_fullsized":"2024-04-02T09:12:39.117972+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:32:03.502011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.13","version":"226'2","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117980+0000","last_change":"2024-04-02T09:12:38.626907+0000","last_active":"2024-04-02T09:12:40.117980+0000","last_peered":"2024-04-02T09:12:40.117980+0000","last_clean":"2024-04-02T09:12:40.117980+0000","last_became_active":"2024-04-02T09:12:38.626639+0000","last_became_peered":"2024-04-02T09:12:38.626639+0000","last_unstale":"2024-04-02T09:12:40.117980+0000","last_undegraded":"2024-04-02T09:12:40.117980+0000","last_fullsized":"2024-04-02T09:12:40.117980+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:09:35.892912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116986+0000","last_change":"2024-04-02T09:12:39.128580+0000","last_active":"2024-04-02T09:12:40.116986+0000","last_peered":"2024-04-02T09:12:40.116986+0000","last_clean":"2024-04-02T09:12:40.116986+0000","last_became_active":"2024-04-02T09:12:39.128324+0000","last_became_peered":"2024-04-02T09:12:39.128324+0000","last_unstale":"2024-04-02T09:12:40.116986+0000","last_undegraded":"2024-04-02T09:12:40.116986+0000","last_fullsized":"2024-04-02T09:12:40.116986+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:16:42.591828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.128748+0000","last_change":"2024-04-02T09:12:24.690012+0000","last_active":"2024-04-02T09:12:39.128748+0000","last_peered":"2024-04-02T09:12:39.128748+0000","last_clean":"2024-04-02T09:12:39.128748+0000","last_became_active":"2024-04-02T09:12:24.689551+0000","last_became_peered":"2024-04-02T09:12:24.689551+0000","last_unstale":"2024-04-02T09:12:39.128748+0000","last_undegraded":"2024-04-02T09:12:39.128748+0000","last_fullsized":"2024-04-02T09:12:39.128748+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:15.316332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.126471+0000","last_change":"2024-04-02T09:12:25.198058+0000","last_active":"2024-04-02T09:12:39.126471+0000","last_peered":"2024-04-02T09:12:39.126471+0000","last_clean":"2024-04-02T09:12:39.126471+0000","last_became_active":"2024-04-02T09:12:25.197791+0000","last_became_peered":"2024-04-02T09:12:25.197791+0000","last_unstale":"2024-04-02T09:12:39.126471+0000","last_undegraded":"2024-04-02T09:12:39.126471+0000","last_fullsized":"2024-04-02T09:12:39.126471+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:38.903726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.10","version":"226'8","reported_seq":19,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112332+0000","last_change":"2024-04-02T09:12:38.629329+0000","last_active":"2024-04-02T09:12:40.112332+0000","last_peered":"2024-04-02T09:12:40.112332+0000","last_clean":"2024-04-02T09:12:40.112332+0000","last_became_active":"2024-04-02T09:12:38.628342+0000","last_became_peered":"2024-04-02T09:12:38.628342+0000","last_unstale":"2024-04-02T09:12:40.112332+0000","last_undegraded":"2024-04-02T09:12:40.112332+0000","last_fullsized":"2024-04-02T09:12:40.112332+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:21.508468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112653+0000","last_change":"2024-04-02T09:12:39.126411+0000","last_active":"2024-04-02T09:12:40.112653+0000","last_peered":"2024-04-02T09:12:40.112653+0000","last_clean":"2024-04-02T09:12:40.112653+0000","last_became_active":"2024-04-02T09:12:39.126179+0000","last_became_peered":"2024-04-02T09:12:39.126179+0000","last_unstale":"2024-04-02T09:12:40.112653+0000","last_undegraded":"2024-04-02T09:12:40.112653+0000","last_fullsized":"2024-04-02T09:12:40.112653+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:00.936946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.113926+0000","last_change":"2024-04-02T09:12:24.697017+0000","last_active":"2024-04-02T09:12:39.113926+0000","last_peered":"2024-04-02T09:12:39.113926+0000","last_clean":"2024-04-02T09:12:39.113926+0000","last_became_active":"2024-04-02T09:12:24.696537+0000","last_became_peered":"2024-04-02T09:12:24.696537+0000","last_unstale":"2024-04-02T09:12:39.113926+0000","last_undegraded":"2024-04-02T09:12:39.113926+0000","last_fullsized":"2024-04-02T09:12:39.113926+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:16.861790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.128655+0000","last_change":"2024-04-02T09:12:25.192446+0000","last_active":"2024-04-02T09:12:39.128655+0000","last_peered":"2024-04-02T09:12:39.128655+0000","last_clean":"2024-04-02T09:12:39.128655+0000","last_became_active":"2024-04-02T09:12:25.192159+0000","last_became_peered":"2024-04-02T09:12:25.192159+0000","last_unstale":"2024-04-02T09:12:39.128655+0000","last_undegraded":"2024-04-02T09:12:39.128655+0000","last_fullsized":"2024-04-02T09:12:39.128655+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:38.903726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117486+0000","last_change":"2024-04-02T09:12:38.634219+0000","last_active":"2024-04-02T09:12:40.117486+0000","last_peered":"2024-04-02T09:12:40.117486+0000","last_clean":"2024-04-02T09:12:40.117486+0000","last_became_active":"2024-04-02T09:12:38.633910+0000","last_became_peered":"2024-04-02T09:12:38.633910+0000","last_unstale":"2024-04-02T09:12:40.117486+0000","last_undegraded":"2024-04-02T09:12:40.117486+0000","last_fullsized":"2024-04-02T09:12:40.117486+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:08:49.341798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347060+0000","last_change":"2024-04-02T09:12:39.125324+0000","last_active":"2024-04-02T09:12:40.347060+0000","last_peered":"2024-04-02T09:12:40.347060+0000","last_clean":"2024-04-02T09:12:40.347060+0000","last_became_active":"2024-04-02T09:12:39.124764+0000","last_became_peered":"2024-04-02T09:12:39.124764+0000","last_unstale":"2024-04-02T09:12:40.347060+0000","last_undegraded":"2024-04-02T09:12:40.347060+0000","last_fullsized":"2024-04-02T09:12:40.347060+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:48:31.765033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.126879+0000","last_change":"2024-04-02T09:12:24.698699+0000","last_active":"2024-04-02T09:12:39.126879+0000","last_peered":"2024-04-02T09:12:39.126879+0000","last_clean":"2024-04-02T09:12:39.126879+0000","last_became_active":"2024-04-02T09:12:24.698347+0000","last_became_peered":"2024-04-02T09:12:24.698347+0000","last_unstale":"2024-04-02T09:12:39.126879+0000","last_undegraded":"2024-04-02T09:12:39.126879+0000","last_fullsized":"2024-04-02T09:12:39.126879+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:44:22.620155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116766+0000","last_change":"2024-04-02T09:12:25.198098+0000","last_active":"2024-04-02T09:12:39.116766+0000","last_peered":"2024-04-02T09:12:39.116766+0000","last_clean":"2024-04-02T09:12:39.116766+0000","last_became_active":"2024-04-02T09:12:25.197848+0000","last_became_peered":"2024-04-02T09:12:25.197848+0000","last_unstale":"2024-04-02T09:12:39.116766+0000","last_undegraded":"2024-04-02T09:12:39.116766+0000","last_fullsized":"2024-04-02T09:12:39.116766+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:02.893062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"226'1","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348766+0000","last_change":"2024-04-02T09:12:38.635301+0000","last_active":"2024-04-02T09:12:40.348766+0000","last_peered":"2024-04-02T09:12:40.348766+0000","last_clean":"2024-04-02T09:12:40.348766+0000","last_became_active":"2024-04-02T09:12:38.634575+0000","last_became_peered":"2024-04-02T09:12:38.634575+0000","last_unstale":"2024-04-02T09:12:40.348766+0000","last_undegraded":"2024-04-02T09:12:40.348766+0000","last_fullsized":"2024-04-02T09:12:40.348766+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:20:17.549794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.118489+0000","last_change":"2024-04-02T09:12:39.141529+0000","last_active":"2024-04-02T09:12:40.118489+0000","last_peered":"2024-04-02T09:12:40.118489+0000","last_clean":"2024-04-02T09:12:40.118489+0000","last_became_active":"2024-04-02T09:12:39.141238+0000","last_became_peered":"2024-04-02T09:12:39.141238+0000","last_unstale":"2024-04-02T09:12:40.118489+0000","last_undegraded":"2024-04-02T09:12:40.118489+0000","last_fullsized":"2024-04-02T09:12:40.118489+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:18:05.229069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.123191+0000","last_change":"2024-04-02T09:12:24.698972+0000","last_active":"2024-04-02T09:12:39.123191+0000","last_peered":"2024-04-02T09:12:39.123191+0000","last_clean":"2024-04-02T09:12:39.123191+0000","last_became_active":"2024-04-02T09:12:24.698664+0000","last_became_peered":"2024-04-02T09:12:24.698664+0000","last_unstale":"2024-04-02T09:12:39.123191+0000","last_undegraded":"2024-04-02T09:12:39.123191+0000","last_fullsized":"2024-04-02T09:12:39.123191+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:11:06.573162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.113545+0000","last_change":"2024-04-02T09:12:25.191145+0000","last_active":"2024-04-02T09:12:39.113545+0000","last_peered":"2024-04-02T09:12:39.113545+0000","last_clean":"2024-04-02T09:12:39.113545+0000","last_became_active":"2024-04-02T09:12:25.190868+0000","last_became_peered":"2024-04-02T09:12:25.190868+0000","last_unstale":"2024-04-02T09:12:39.113545+0000","last_undegraded":"2024-04-02T09:12:39.113545+0000","last_fullsized":"2024-04-02T09:12:39.113545+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:41.683998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1f","version":"226'4","reported_seq":15,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.118385+0000","last_change":"2024-04-02T09:12:38.640727+0000","last_active":"2024-04-02T09:12:40.118385+0000","last_peered":"2024-04-02T09:12:40.118385+0000","last_clean":"2024-04-02T09:12:40.118385+0000","last_became_active":"2024-04-02T09:12:38.640469+0000","last_became_peered":"2024-04-02T09:12:38.640469+0000","last_unstale":"2024-04-02T09:12:40.118385+0000","last_undegraded":"2024-04-02T09:12:40.118385+0000","last_fullsized":"2024-04-02T09:12:40.118385+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:59:12.891582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.113379+0000","last_change":"2024-04-02T09:12:39.144611+0000","last_active":"2024-04-02T09:12:40.113379+0000","last_peered":"2024-04-02T09:12:40.113379+0000","last_clean":"2024-04-02T09:12:40.113379+0000","last_became_active":"2024-04-02T09:12:39.144284+0000","last_became_peered":"2024-04-02T09:12:39.144284+0000","last_unstale":"2024-04-02T09:12:40.113379+0000","last_undegraded":"2024-04-02T09:12:40.113379+0000","last_fullsized":"2024-04-02T09:12:40.113379+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:22:50.985250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"222'1","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114379+0000","last_change":"2024-04-02T09:12:24.690596+0000","last_active":"2024-04-02T09:12:39.114379+0000","last_peered":"2024-04-02T09:12:39.114379+0000","last_clean":"2024-04-02T09:12:39.114379+0000","last_became_active":"2024-04-02T09:12:24.690029+0000","last_became_peered":"2024-04-02T09:12:24.690029+0000","last_unstale":"2024-04-02T09:12:39.114379+0000","last_undegraded":"2024-04-02T09:12:39.114379+0000","last_fullsized":"2024-04-02T09:12:39.114379+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:00.608890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.134279+0000","last_change":"2024-04-02T09:12:25.192982+0000","last_active":"2024-04-02T09:12:39.134279+0000","last_peered":"2024-04-02T09:12:39.134279+0000","last_clean":"2024-04-02T09:12:39.134279+0000","last_became_active":"2024-04-02T09:12:25.192737+0000","last_became_peered":"2024-04-02T09:12:25.192737+0000","last_unstale":"2024-04-02T09:12:39.134279+0000","last_undegraded":"2024-04-02T09:12:39.134279+0000","last_fullsized":"2024-04-02T09:12:39.134279+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:28.343401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1c","version":"226'2","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348484+0000","last_change":"2024-04-02T09:12:38.630289+0000","last_active":"2024-04-02T09:12:40.348484+0000","last_peered":"2024-04-02T09:12:40.348484+0000","last_clean":"2024-04-02T09:12:40.348484+0000","last_became_active":"2024-04-02T09:12:38.628695+0000","last_became_peered":"2024-04-02T09:12:38.628695+0000","last_unstale":"2024-04-02T09:12:40.348484+0000","last_undegraded":"2024-04-02T09:12:40.348484+0000","last_fullsized":"2024-04-02T09:12:40.348484+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:08:59.930412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348040+0000","last_change":"2024-04-02T09:12:39.138377+0000","last_active":"2024-04-02T09:12:40.348040+0000","last_peered":"2024-04-02T09:12:40.348040+0000","last_clean":"2024-04-02T09:12:40.348040+0000","last_became_active":"2024-04-02T09:12:39.138107+0000","last_became_peered":"2024-04-02T09:12:39.138107+0000","last_unstale":"2024-04-02T09:12:40.348040+0000","last_undegraded":"2024-04-02T09:12:40.348040+0000","last_fullsized":"2024-04-02T09:12:40.348040+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:32.734424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1c","version":"222'2","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.133207+0000","last_change":"2024-04-02T09:12:24.692574+0000","last_active":"2024-04-02T09:12:39.133207+0000","last_peered":"2024-04-02T09:12:39.133207+0000","last_clean":"2024-04-02T09:12:39.133207+0000","last_became_active":"2024-04-02T09:12:24.692283+0000","last_became_peered":"2024-04-02T09:12:24.692283+0000","last_unstale":"2024-04-02T09:12:39.133207+0000","last_undegraded":"2024-04-02T09:12:39.133207+0000","last_fullsized":"2024-04-02T09:12:39.133207+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:56:59.029298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.113910+0000","last_change":"2024-04-02T09:12:25.194989+0000","last_active":"2024-04-02T09:12:39.113910+0000","last_peered":"2024-04-02T09:12:39.113910+0000","last_clean":"2024-04-02T09:12:39.113910+0000","last_became_active":"2024-04-02T09:12:25.194466+0000","last_became_peered":"2024-04-02T09:12:25.194466+0000","last_unstale":"2024-04-02T09:12:39.113910+0000","last_undegraded":"2024-04-02T09:12:39.113910+0000","last_fullsized":"2024-04-02T09:12:39.113910+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:01.618091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1a","version":"226'1","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348635+0000","last_change":"2024-04-02T09:12:38.635903+0000","last_active":"2024-04-02T09:12:40.348635+0000","last_peered":"2024-04-02T09:12:40.348635+0000","last_clean":"2024-04-02T09:12:40.348635+0000","last_became_active":"2024-04-02T09:12:38.635616+0000","last_became_peered":"2024-04-02T09:12:38.635616+0000","last_unstale":"2024-04-02T09:12:40.348635+0000","last_undegraded":"2024-04-02T09:12:40.348635+0000","last_fullsized":"2024-04-02T09:12:40.348635+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:10.633778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116558+0000","last_change":"2024-04-02T09:12:39.147583+0000","last_active":"2024-04-02T09:12:40.116558+0000","last_peered":"2024-04-02T09:12:40.116558+0000","last_clean":"2024-04-02T09:12:40.116558+0000","last_became_active":"2024-04-02T09:12:39.147229+0000","last_became_peered":"2024-04-02T09:12:39.147229+0000","last_unstale":"2024-04-02T09:12:40.116558+0000","last_undegraded":"2024-04-02T09:12:40.116558+0000","last_fullsized":"2024-04-02T09:12:40.116558+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:53:58.450179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"222'2","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.122782+0000","last_change":"2024-04-02T09:12:24.691669+0000","last_active":"2024-04-02T09:12:39.122782+0000","last_peered":"2024-04-02T09:12:39.122782+0000","last_clean":"2024-04-02T09:12:39.122782+0000","last_became_active":"2024-04-02T09:12:24.689803+0000","last_became_peered":"2024-04-02T09:12:24.689803+0000","last_unstale":"2024-04-02T09:12:39.122782+0000","last_undegraded":"2024-04-02T09:12:39.122782+0000","last_fullsized":"2024-04-02T09:12:39.122782+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:25:15.021325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117957+0000","last_change":"2024-04-02T09:12:25.204722+0000","last_active":"2024-04-02T09:12:39.117957+0000","last_peered":"2024-04-02T09:12:39.117957+0000","last_clean":"2024-04-02T09:12:39.117957+0000","last_became_active":"2024-04-02T09:12:25.204031+0000","last_became_peered":"2024-04-02T09:12:25.204031+0000","last_unstale":"2024-04-02T09:12:39.117957+0000","last_undegraded":"2024-04-02T09:12:39.117957+0000","last_fullsized":"2024-04-02T09:12:39.117957+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:00:19.545624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1b","version":"226'1","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347797+0000","last_change":"2024-04-02T09:12:38.631122+0000","last_active":"2024-04-02T09:12:40.347797+0000","last_peered":"2024-04-02T09:12:40.347797+0000","last_clean":"2024-04-02T09:12:40.347797+0000","last_became_active":"2024-04-02T09:12:38.630207+0000","last_became_peered":"2024-04-02T09:12:38.630207+0000","last_unstale":"2024-04-02T09:12:40.347797+0000","last_undegraded":"2024-04-02T09:12:40.347797+0000","last_fullsized":"2024-04-02T09:12:40.347797+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:44:41.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112664+0000","last_change":"2024-04-02T09:12:39.143199+0000","last_active":"2024-04-02T09:12:40.112664+0000","last_peered":"2024-04-02T09:12:40.112664+0000","last_clean":"2024-04-02T09:12:40.112664+0000","last_became_active":"2024-04-02T09:12:39.142929+0000","last_became_peered":"2024-04-02T09:12:39.142929+0000","last_unstale":"2024-04-02T09:12:40.112664+0000","last_undegraded":"2024-04-02T09:12:40.112664+0000","last_fullsized":"2024-04-02T09:12:40.112664+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:11:52.229332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1e","version":"222'1","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117390+0000","last_change":"2024-04-02T09:12:24.684617+0000","last_active":"2024-04-02T09:12:39.117390+0000","last_peered":"2024-04-02T09:12:39.117390+0000","last_clean":"2024-04-02T09:12:39.117390+0000","last_became_active":"2024-04-02T09:12:24.684367+0000","last_became_peered":"2024-04-02T09:12:24.684367+0000","last_unstale":"2024-04-02T09:12:39.117390+0000","last_undegraded":"2024-04-02T09:12:39.117390+0000","last_fullsized":"2024-04-02T09:12:39.117390+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:48:45.999478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.130741+0000","last_change":"2024-04-02T09:12:25.190209+0000","last_active":"2024-04-02T09:12:39.130741+0000","last_peered":"2024-04-02T09:12:39.130741+0000","last_clean":"2024-04-02T09:12:39.130741+0000","last_became_active":"2024-04-02T09:12:25.189908+0000","last_became_peered":"2024-04-02T09:12:25.189908+0000","last_unstale":"2024-04-02T09:12:39.130741+0000","last_undegraded":"2024-04-02T09:12:39.130741+0000","last_fullsized":"2024-04-02T09:12:39.130741+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:12.375324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347333+0000","last_change":"2024-04-02T09:12:38.623441+0000","last_active":"2024-04-02T09:12:40.347333+0000","last_peered":"2024-04-02T09:12:40.347333+0000","last_clean":"2024-04-02T09:12:40.347333+0000","last_became_active":"2024-04-02T09:12:38.622492+0000","last_became_peered":"2024-04-02T09:12:38.622492+0000","last_unstale":"2024-04-02T09:12:40.347333+0000","last_undegraded":"2024-04-02T09:12:40.347333+0000","last_fullsized":"2024-04-02T09:12:40.347333+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:16:41.585294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349891+0000","last_change":"2024-04-02T09:12:39.141266+0000","last_active":"2024-04-02T09:12:40.349891+0000","last_peered":"2024-04-02T09:12:40.349891+0000","last_clean":"2024-04-02T09:12:40.349891+0000","last_became_active":"2024-04-02T09:12:39.140845+0000","last_became_peered":"2024-04-02T09:12:39.140845+0000","last_unstale":"2024-04-02T09:12:40.349891+0000","last_undegraded":"2024-04-02T09:12:40.349891+0000","last_fullsized":"2024-04-02T09:12:40.349891+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:22:56.853967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1f","version":"222'4","reported_seq":19,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.121616+0000","last_change":"2024-04-02T09:12:24.695565+0000","last_active":"2024-04-02T09:12:39.121616+0000","last_peered":"2024-04-02T09:12:39.121616+0000","last_clean":"2024-04-02T09:12:39.121616+0000","last_became_active":"2024-04-02T09:12:24.695304+0000","last_became_peered":"2024-04-02T09:12:24.695304+0000","last_unstale":"2024-04-02T09:12:39.121616+0000","last_undegraded":"2024-04-02T09:12:39.121616+0000","last_fullsized":"2024-04-02T09:12:39.121616+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:56:16.018943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117066+0000","last_change":"2024-04-02T09:12:25.195780+0000","last_active":"2024-04-02T09:12:39.117066+0000","last_peered":"2024-04-02T09:12:39.117066+0000","last_clean":"2024-04-02T09:12:39.117066+0000","last_became_active":"2024-04-02T09:12:25.194939+0000","last_became_peered":"2024-04-02T09:12:25.194939+0000","last_unstale":"2024-04-02T09:12:39.117066+0000","last_undegraded":"2024-04-02T09:12:39.117066+0000","last_fullsized":"2024-04-02T09:12:39.117066+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:20:21.844925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348155+0000","last_change":"2024-04-02T09:12:38.625812+0000","last_active":"2024-04-02T09:12:40.348155+0000","last_peered":"2024-04-02T09:12:40.348155+0000","last_clean":"2024-04-02T09:12:40.348155+0000","last_became_active":"2024-04-02T09:12:38.625536+0000","last_became_peered":"2024-04-02T09:12:38.625536+0000","last_unstale":"2024-04-02T09:12:40.348155+0000","last_undegraded":"2024-04-02T09:12:40.348155+0000","last_fullsized":"2024-04-02T09:12:40.348155+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:49:16.327081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116523+0000","last_change":"2024-04-02T09:12:39.143363+0000","last_active":"2024-04-02T09:12:40.116523+0000","last_peered":"2024-04-02T09:12:40.116523+0000","last_clean":"2024-04-02T09:12:40.116523+0000","last_became_active":"2024-04-02T09:12:39.142967+0000","last_became_peered":"2024-04-02T09:12:39.142967+0000","last_unstale":"2024-04-02T09:12:40.116523+0000","last_undegraded":"2024-04-02T09:12:40.116523+0000","last_fullsized":"2024-04-02T09:12:40.116523+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:22.515002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.118239+0000","last_change":"2024-04-02T09:12:39.130951+0000","last_active":"2024-04-02T09:12:40.118239+0000","last_peered":"2024-04-02T09:12:40.118239+0000","last_clean":"2024-04-02T09:12:40.118239+0000","last_became_active":"2024-04-02T09:12:39.130211+0000","last_became_peered":"2024-04-02T09:12:39.130211+0000","last_unstale":"2024-04-02T09:12:40.118239+0000","last_undegraded":"2024-04-02T09:12:40.118239+0000","last_fullsized":"2024-04-02T09:12:40.118239+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:41.395363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.122770+0000","last_change":"2024-04-02T09:12:25.193316+0000","last_active":"2024-04-02T09:12:39.122770+0000","last_peered":"2024-04-02T09:12:39.122770+0000","last_clean":"2024-04-02T09:12:39.122770+0000","last_became_active":"2024-04-02T09:12:25.192672+0000","last_became_peered":"2024-04-02T09:12:25.192672+0000","last_unstale":"2024-04-02T09:12:39.122770+0000","last_undegraded":"2024-04-02T09:12:39.122770+0000","last_fullsized":"2024-04-02T09:12:39.122770+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:28.343401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.113030+0000","last_change":"2024-04-02T09:12:39.132655+0000","last_active":"2024-04-02T09:12:40.113030+0000","last_peered":"2024-04-02T09:12:40.113030+0000","last_clean":"2024-04-02T09:12:40.113030+0000","last_became_active":"2024-04-02T09:12:39.128795+0000","last_became_peered":"2024-04-02T09:12:39.128795+0000","last_unstale":"2024-04-02T09:12:40.113030+0000","last_undegraded":"2024-04-02T09:12:40.113030+0000","last_fullsized":"2024-04-02T09:12:40.113030+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:13:15.590165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.119524+0000","last_change":"2024-04-02T09:12:25.203525+0000","last_active":"2024-04-02T09:12:39.119524+0000","last_peered":"2024-04-02T09:12:39.119524+0000","last_clean":"2024-04-02T09:12:39.119524+0000","last_became_active":"2024-04-02T09:12:25.202409+0000","last_became_peered":"2024-04-02T09:12:25.202409+0000","last_unstale":"2024-04-02T09:12:39.119524+0000","last_undegraded":"2024-04-02T09:12:39.119524+0000","last_fullsized":"2024-04-02T09:12:39.119524+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:38:54.918706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.124181+0000","last_change":"2024-04-02T09:12:25.194990+0000","last_active":"2024-04-02T09:12:39.124181+0000","last_peered":"2024-04-02T09:12:39.124181+0000","last_clean":"2024-04-02T09:12:39.124181+0000","last_became_active":"2024-04-02T09:12:25.191623+0000","last_became_peered":"2024-04-02T09:12:25.191623+0000","last_unstale":"2024-04-02T09:12:39.124181+0000","last_undegraded":"2024-04-02T09:12:39.124181+0000","last_fullsized":"2024-04-02T09:12:39.124181+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:42:10.151562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117908+0000","last_change":"2024-04-02T09:12:39.133147+0000","last_active":"2024-04-02T09:12:40.117908+0000","last_peered":"2024-04-02T09:12:40.117908+0000","last_clean":"2024-04-02T09:12:40.117908+0000","last_became_active":"2024-04-02T09:12:39.132883+0000","last_became_peered":"2024-04-02T09:12:39.132883+0000","last_unstale":"2024-04-02T09:12:40.117908+0000","last_undegraded":"2024-04-02T09:12:40.117908+0000","last_fullsized":"2024-04-02T09:12:40.117908+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:19:30.021833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.113219+0000","last_change":"2024-04-02T09:12:39.130628+0000","last_active":"2024-04-02T09:12:40.113219+0000","last_peered":"2024-04-02T09:12:40.113219+0000","last_clean":"2024-04-02T09:12:40.113219+0000","last_became_active":"2024-04-02T09:12:39.130023+0000","last_became_peered":"2024-04-02T09:12:39.130023+0000","last_unstale":"2024-04-02T09:12:40.113219+0000","last_undegraded":"2024-04-02T09:12:40.113219+0000","last_fullsized":"2024-04-02T09:12:40.113219+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:28.062436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116957+0000","last_change":"2024-04-02T09:12:25.205454+0000","last_active":"2024-04-02T09:12:39.116957+0000","last_peered":"2024-04-02T09:12:39.116957+0000","last_clean":"2024-04-02T09:12:39.116957+0000","last_became_active":"2024-04-02T09:12:25.204157+0000","last_became_peered":"2024-04-02T09:12:25.204157+0000","last_unstale":"2024-04-02T09:12:39.116957+0000","last_undegraded":"2024-04-02T09:12:39.116957+0000","last_fullsized":"2024-04-02T09:12:39.116957+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:46:50.298119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.124619+0000","last_change":"2024-04-02T09:12:25.192359+0000","last_active":"2024-04-02T09:12:39.124619+0000","last_peered":"2024-04-02T09:12:39.124619+0000","last_clean":"2024-04-02T09:12:39.124619+0000","last_became_active":"2024-04-02T09:12:25.192092+0000","last_became_peered":"2024-04-02T09:12:25.192092+0000","last_unstale":"2024-04-02T09:12:39.124619+0000","last_undegraded":"2024-04-02T09:12:39.124619+0000","last_fullsized":"2024-04-02T09:12:39.124619+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:01.618091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347858+0000","last_change":"2024-04-02T09:12:39.143757+0000","last_active":"2024-04-02T09:12:40.347858+0000","last_peered":"2024-04-02T09:12:40.347858+0000","last_clean":"2024-04-02T09:12:40.347858+0000","last_became_active":"2024-04-02T09:12:39.142148+0000","last_became_peered":"2024-04-02T09:12:39.142148+0000","last_unstale":"2024-04-02T09:12:40.347858+0000","last_undegraded":"2024-04-02T09:12:40.347858+0000","last_fullsized":"2024-04-02T09:12:40.347858+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:03:55.514882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.119332+0000","last_change":"2024-04-02T09:12:25.196804+0000","last_active":"2024-04-02T09:12:39.119332+0000","last_peered":"2024-04-02T09:12:39.119332+0000","last_clean":"2024-04-02T09:12:39.119332+0000","last_became_active":"2024-04-02T09:12:25.194411+0000","last_became_peered":"2024-04-02T09:12:25.194411+0000","last_unstale":"2024-04-02T09:12:39.119332+0000","last_undegraded":"2024-04-02T09:12:39.119332+0000","last_fullsized":"2024-04-02T09:12:39.119332+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:45:21.225286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348334+0000","last_change":"2024-04-02T09:12:39.125335+0000","last_active":"2024-04-02T09:12:40.348334+0000","last_peered":"2024-04-02T09:12:40.348334+0000","last_clean":"2024-04-02T09:12:40.348334+0000","last_became_active":"2024-04-02T09:12:39.124780+0000","last_became_peered":"2024-04-02T09:12:39.124780+0000","last_unstale":"2024-04-02T09:12:40.348334+0000","last_undegraded":"2024-04-02T09:12:40.348334+0000","last_fullsized":"2024-04-02T09:12:40.348334+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:15.554233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.123254+0000","last_change":"2024-04-02T09:12:25.200885+0000","last_active":"2024-04-02T09:12:39.123254+0000","last_peered":"2024-04-02T09:12:39.123254+0000","last_clean":"2024-04-02T09:12:39.123254+0000","last_became_active":"2024-04-02T09:12:25.200630+0000","last_became_peered":"2024-04-02T09:12:25.200630+0000","last_unstale":"2024-04-02T09:12:39.123254+0000","last_undegraded":"2024-04-02T09:12:39.123254+0000","last_fullsized":"2024-04-02T09:12:39.123254+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:18.798282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117294+0000","last_change":"2024-04-02T09:12:39.142959+0000","last_active":"2024-04-02T09:12:40.117294+0000","last_peered":"2024-04-02T09:12:40.117294+0000","last_clean":"2024-04-02T09:12:40.117294+0000","last_became_active":"2024-04-02T09:12:39.142426+0000","last_became_peered":"2024-04-02T09:12:39.142426+0000","last_unstale":"2024-04-02T09:12:40.117294+0000","last_undegraded":"2024-04-02T09:12:40.117294+0000","last_fullsized":"2024-04-02T09:12:40.117294+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:07.965927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.130830+0000","last_change":"2024-04-02T09:12:25.195404+0000","last_active":"2024-04-02T09:12:39.130830+0000","last_peered":"2024-04-02T09:12:39.130830+0000","last_clean":"2024-04-02T09:12:39.130830+0000","last_became_active":"2024-04-02T09:12:25.192051+0000","last_became_peered":"2024-04-02T09:12:25.192051+0000","last_unstale":"2024-04-02T09:12:39.130830+0000","last_undegraded":"2024-04-02T09:12:39.130830+0000","last_fullsized":"2024-04-02T09:12:39.130830+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:02.893062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347686+0000","last_change":"2024-04-02T09:12:39.133572+0000","last_active":"2024-04-02T09:12:40.347686+0000","last_peered":"2024-04-02T09:12:40.347686+0000","last_clean":"2024-04-02T09:12:40.347686+0000","last_became_active":"2024-04-02T09:12:39.132574+0000","last_became_peered":"2024-04-02T09:12:39.132574+0000","last_unstale":"2024-04-02T09:12:40.347686+0000","last_undegraded":"2024-04-02T09:12:40.347686+0000","last_fullsized":"2024-04-02T09:12:40.347686+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:13:38.360757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348986+0000","last_change":"2024-04-02T09:12:39.142178+0000","last_active":"2024-04-02T09:12:40.348986+0000","last_peered":"2024-04-02T09:12:40.348986+0000","last_clean":"2024-04-02T09:12:40.348986+0000","last_became_active":"2024-04-02T09:12:39.141888+0000","last_became_peered":"2024-04-02T09:12:39.141888+0000","last_unstale":"2024-04-02T09:12:40.348986+0000","last_undegraded":"2024-04-02T09:12:40.348986+0000","last_fullsized":"2024-04-02T09:12:40.348986+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:55.944314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.113868+0000","last_change":"2024-04-02T09:12:25.185102+0000","last_active":"2024-04-02T09:12:39.113868+0000","last_peered":"2024-04-02T09:12:39.113868+0000","last_clean":"2024-04-02T09:12:39.113868+0000","last_became_active":"2024-04-02T09:12:25.184829+0000","last_became_peered":"2024-04-02T09:12:25.184829+0000","last_unstale":"2024-04-02T09:12:39.113868+0000","last_undegraded":"2024-04-02T09:12:39.113868+0000","last_fullsized":"2024-04-02T09:12:39.113868+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:28.343401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348863+0000","last_change":"2024-04-02T09:12:39.140834+0000","last_active":"2024-04-02T09:12:40.348863+0000","last_peered":"2024-04-02T09:12:40.348863+0000","last_clean":"2024-04-02T09:12:40.348863+0000","last_became_active":"2024-04-02T09:12:39.140430+0000","last_became_peered":"2024-04-02T09:12:39.140430+0000","last_unstale":"2024-04-02T09:12:40.348863+0000","last_undegraded":"2024-04-02T09:12:40.348863+0000","last_fullsized":"2024-04-02T09:12:40.348863+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:38:03.793164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.127120+0000","last_change":"2024-04-02T09:12:25.205832+0000","last_active":"2024-04-02T09:12:39.127120+0000","last_peered":"2024-04-02T09:12:39.127120+0000","last_clean":"2024-04-02T09:12:39.127120+0000","last_became_active":"2024-04-02T09:12:25.205206+0000","last_became_peered":"2024-04-02T09:12:25.205206+0000","last_unstale":"2024-04-02T09:12:39.127120+0000","last_undegraded":"2024-04-02T09:12:39.127120+0000","last_fullsized":"2024-04-02T09:12:39.127120+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:17.511411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348672+0000","last_change":"2024-04-02T09:12:39.123156+0000","last_active":"2024-04-02T09:12:40.348672+0000","last_peered":"2024-04-02T09:12:40.348672+0000","last_clean":"2024-04-02T09:12:40.348672+0000","last_became_active":"2024-04-02T09:12:39.122879+0000","last_became_peered":"2024-04-02T09:12:39.122879+0000","last_unstale":"2024-04-02T09:12:40.348672+0000","last_undegraded":"2024-04-02T09:12:40.348672+0000","last_fullsized":"2024-04-02T09:12:40.348672+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:35:05.722753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116264+0000","last_change":"2024-04-02T09:12:25.197238+0000","last_active":"2024-04-02T09:12:39.116264+0000","last_peered":"2024-04-02T09:12:39.116264+0000","last_clean":"2024-04-02T09:12:39.116264+0000","last_became_active":"2024-04-02T09:12:25.196921+0000","last_became_peered":"2024-04-02T09:12:25.196921+0000","last_unstale":"2024-04-02T09:12:39.116264+0000","last_undegraded":"2024-04-02T09:12:39.116264+0000","last_fullsized":"2024-04-02T09:12:39.116264+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:40:34.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117627+0000","last_change":"2024-04-02T09:12:39.125280+0000","last_active":"2024-04-02T09:12:40.117627+0000","last_peered":"2024-04-02T09:12:40.117627+0000","last_clean":"2024-04-02T09:12:40.117627+0000","last_became_active":"2024-04-02T09:12:39.123651+0000","last_became_peered":"2024-04-02T09:12:39.123651+0000","last_unstale":"2024-04-02T09:12:40.117627+0000","last_undegraded":"2024-04-02T09:12:40.117627+0000","last_fullsized":"2024-04-02T09:12:40.117627+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:04.234260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.115677+0000","last_change":"2024-04-02T09:12:25.201010+0000","last_active":"2024-04-02T09:12:39.115677+0000","last_peered":"2024-04-02T09:12:39.115677+0000","last_clean":"2024-04-02T09:12:39.115677+0000","last_became_active":"2024-04-02T09:12:25.198423+0000","last_became_peered":"2024-04-02T09:12:25.198423+0000","last_unstale":"2024-04-02T09:12:39.115677+0000","last_undegraded":"2024-04-02T09:12:39.115677+0000","last_fullsized":"2024-04-02T09:12:39.115677+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:44:28.685030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116695+0000","last_change":"2024-04-02T09:12:39.142382+0000","last_active":"2024-04-02T09:12:40.116695+0000","last_peered":"2024-04-02T09:12:40.116695+0000","last_clean":"2024-04-02T09:12:40.116695+0000","last_became_active":"2024-04-02T09:12:39.142079+0000","last_became_peered":"2024-04-02T09:12:39.142079+0000","last_unstale":"2024-04-02T09:12:40.116695+0000","last_undegraded":"2024-04-02T09:12:40.116695+0000","last_fullsized":"2024-04-02T09:12:40.116695+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:40:48.214067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117659+0000","last_change":"2024-04-02T09:12:25.196567+0000","last_active":"2024-04-02T09:12:39.117659+0000","last_peered":"2024-04-02T09:12:39.117659+0000","last_clean":"2024-04-02T09:12:39.117659+0000","last_became_active":"2024-04-02T09:12:25.196264+0000","last_became_peered":"2024-04-02T09:12:25.196264+0000","last_unstale":"2024-04-02T09:12:39.117659+0000","last_undegraded":"2024-04-02T09:12:39.117659+0000","last_fullsized":"2024-04-02T09:12:39.117659+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:03.387546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348467+0000","last_change":"2024-04-02T09:12:39.147190+0000","last_active":"2024-04-02T09:12:40.348467+0000","last_peered":"2024-04-02T09:12:40.348467+0000","last_clean":"2024-04-02T09:12:40.348467+0000","last_became_active":"2024-04-02T09:12:39.146900+0000","last_became_peered":"2024-04-02T09:12:39.146900+0000","last_unstale":"2024-04-02T09:12:40.348467+0000","last_undegraded":"2024-04-02T09:12:40.348467+0000","last_fullsized":"2024-04-02T09:12:40.348467+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:28.062436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116479+0000","last_change":"2024-04-02T09:12:25.196715+0000","last_active":"2024-04-02T09:12:39.116479+0000","last_peered":"2024-04-02T09:12:39.116479+0000","last_clean":"2024-04-02T09:12:39.116479+0000","last_became_active":"2024-04-02T09:12:25.196433+0000","last_became_peered":"2024-04-02T09:12:25.196433+0000","last_unstale":"2024-04-02T09:12:39.116479+0000","last_undegraded":"2024-04-02T09:12:39.116479+0000","last_fullsized":"2024-04-02T09:12:39.116479+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:30.835437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347349+0000","last_change":"2024-04-02T09:12:39.137405+0000","last_active":"2024-04-02T09:12:40.347349+0000","last_peered":"2024-04-02T09:12:40.347349+0000","last_clean":"2024-04-02T09:12:40.347349+0000","last_became_active":"2024-04-02T09:12:39.136852+0000","last_became_peered":"2024-04-02T09:12:39.136852+0000","last_unstale":"2024-04-02T09:12:40.347349+0000","last_undegraded":"2024-04-02T09:12:40.347349+0000","last_fullsized":"2024-04-02T09:12:40.347349+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:59:13.898116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.119988+0000","last_change":"2024-04-02T09:12:25.207159+0000","last_active":"2024-04-02T09:12:39.119988+0000","last_peered":"2024-04-02T09:12:39.119988+0000","last_clean":"2024-04-02T09:12:39.119988+0000","last_became_active":"2024-04-02T09:12:25.206804+0000","last_became_peered":"2024-04-02T09:12:25.206804+0000","last_unstale":"2024-04-02T09:12:39.119988+0000","last_undegraded":"2024-04-02T09:12:39.119988+0000","last_fullsized":"2024-04-02T09:12:39.119988+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:19.646703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349878+0000","last_change":"2024-04-02T09:12:39.142032+0000","last_active":"2024-04-02T09:12:40.349878+0000","last_peered":"2024-04-02T09:12:40.349878+0000","last_clean":"2024-04-02T09:12:40.349878+0000","last_became_active":"2024-04-02T09:12:39.141722+0000","last_became_peered":"2024-04-02T09:12:39.141722+0000","last_unstale":"2024-04-02T09:12:40.349878+0000","last_undegraded":"2024-04-02T09:12:40.349878+0000","last_fullsized":"2024-04-02T09:12:40.349878+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:17.921961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116835+0000","last_change":"2024-04-02T09:12:25.196993+0000","last_active":"2024-04-02T09:12:39.116835+0000","last_peered":"2024-04-02T09:12:39.116835+0000","last_clean":"2024-04-02T09:12:39.116835+0000","last_became_active":"2024-04-02T09:12:25.196316+0000","last_became_peered":"2024-04-02T09:12:25.196316+0000","last_unstale":"2024-04-02T09:12:39.116835+0000","last_undegraded":"2024-04-02T09:12:39.116835+0000","last_fullsized":"2024-04-02T09:12:39.116835+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:36:48.733839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348669+0000","last_change":"2024-04-02T09:12:39.134057+0000","last_active":"2024-04-02T09:12:40.348669+0000","last_peered":"2024-04-02T09:12:40.348669+0000","last_clean":"2024-04-02T09:12:40.348669+0000","last_became_active":"2024-04-02T09:12:39.132976+0000","last_became_peered":"2024-04-02T09:12:39.132976+0000","last_unstale":"2024-04-02T09:12:40.348669+0000","last_undegraded":"2024-04-02T09:12:40.348669+0000","last_fullsized":"2024-04-02T09:12:40.348669+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:45:43.597770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.113451+0000","last_change":"2024-04-02T09:12:25.188711+0000","last_active":"2024-04-02T09:12:39.113451+0000","last_peered":"2024-04-02T09:12:39.113451+0000","last_clean":"2024-04-02T09:12:39.113451+0000","last_became_active":"2024-04-02T09:12:25.188399+0000","last_became_peered":"2024-04-02T09:12:25.188399+0000","last_unstale":"2024-04-02T09:12:39.113451+0000","last_undegraded":"2024-04-02T09:12:39.113451+0000","last_fullsized":"2024-04-02T09:12:39.113451+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:20:21.844925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348815+0000","last_change":"2024-04-02T09:12:39.126125+0000","last_active":"2024-04-02T09:12:40.348815+0000","last_peered":"2024-04-02T09:12:40.348815+0000","last_clean":"2024-04-02T09:12:40.348815+0000","last_became_active":"2024-04-02T09:12:39.125868+0000","last_became_peered":"2024-04-02T09:12:39.125868+0000","last_unstale":"2024-04-02T09:12:40.348815+0000","last_undegraded":"2024-04-02T09:12:40.348815+0000","last_fullsized":"2024-04-02T09:12:40.348815+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:48:39.621381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.115933+0000","last_change":"2024-04-02T09:12:25.186757+0000","last_active":"2024-04-02T09:12:39.115933+0000","last_peered":"2024-04-02T09:12:39.115933+0000","last_clean":"2024-04-02T09:12:39.115933+0000","last_became_active":"2024-04-02T09:12:25.186436+0000","last_became_peered":"2024-04-02T09:12:25.186436+0000","last_unstale":"2024-04-02T09:12:39.115933+0000","last_undegraded":"2024-04-02T09:12:39.115933+0000","last_fullsized":"2024-04-02T09:12:39.115933+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:21.964569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349159+0000","last_change":"2024-04-02T09:12:39.123464+0000","last_active":"2024-04-02T09:12:40.349159+0000","last_peered":"2024-04-02T09:12:40.349159+0000","last_clean":"2024-04-02T09:12:40.349159+0000","last_became_active":"2024-04-02T09:12:39.123189+0000","last_became_peered":"2024-04-02T09:12:39.123189+0000","last_unstale":"2024-04-02T09:12:40.349159+0000","last_undegraded":"2024-04-02T09:12:40.349159+0000","last_fullsized":"2024-04-02T09:12:40.349159+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:26:54.971921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.121991+0000","last_change":"2024-04-02T09:12:25.198312+0000","last_active":"2024-04-02T09:12:39.121991+0000","last_peered":"2024-04-02T09:12:39.121991+0000","last_clean":"2024-04-02T09:12:39.121991+0000","last_became_active":"2024-04-02T09:12:25.198040+0000","last_became_peered":"2024-04-02T09:12:25.198040+0000","last_unstale":"2024-04-02T09:12:39.121991+0000","last_undegraded":"2024-04-02T09:12:39.121991+0000","last_fullsized":"2024-04-02T09:12:39.121991+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:02.893062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112978+0000","last_change":"2024-04-02T09:12:39.132043+0000","last_active":"2024-04-02T09:12:40.112978+0000","last_peered":"2024-04-02T09:12:40.112978+0000","last_clean":"2024-04-02T09:12:40.112978+0000","last_became_active":"2024-04-02T09:12:39.130384+0000","last_became_peered":"2024-04-02T09:12:39.130384+0000","last_unstale":"2024-04-02T09:12:40.112978+0000","last_undegraded":"2024-04-02T09:12:40.112978+0000","last_fullsized":"2024-04-02T09:12:40.112978+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:10.514935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.134359+0000","last_change":"2024-04-02T09:12:25.205465+0000","last_active":"2024-04-02T09:12:39.134359+0000","last_peered":"2024-04-02T09:12:39.134359+0000","last_clean":"2024-04-02T09:12:39.134359+0000","last_became_active":"2024-04-02T09:12:25.204248+0000","last_became_peered":"2024-04-02T09:12:25.204248+0000","last_unstale":"2024-04-02T09:12:39.134359+0000","last_undegraded":"2024-04-02T09:12:39.134359+0000","last_fullsized":"2024-04-02T09:12:39.134359+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:32:03.502011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112641+0000","last_change":"2024-04-02T09:12:39.118797+0000","last_active":"2024-04-02T09:12:40.112641+0000","last_peered":"2024-04-02T09:12:40.112641+0000","last_clean":"2024-04-02T09:12:40.112641+0000","last_became_active":"2024-04-02T09:12:39.118542+0000","last_became_peered":"2024-04-02T09:12:39.118542+0000","last_unstale":"2024-04-02T09:12:40.112641+0000","last_undegraded":"2024-04-02T09:12:40.112641+0000","last_fullsized":"2024-04-02T09:12:40.112641+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:35:05.722753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116558+0000","last_change":"2024-04-02T09:12:25.186623+0000","last_active":"2024-04-02T09:12:39.116558+0000","last_peered":"2024-04-02T09:12:39.116558+0000","last_clean":"2024-04-02T09:12:39.116558+0000","last_became_active":"2024-04-02T09:12:25.186297+0000","last_became_peered":"2024-04-02T09:12:25.186297+0000","last_unstale":"2024-04-02T09:12:39.116558+0000","last_undegraded":"2024-04-02T09:12:39.116558+0000","last_fullsized":"2024-04-02T09:12:39.116558+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:36:48.733839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349352+0000","last_change":"2024-04-02T09:12:39.143922+0000","last_active":"2024-04-02T09:12:40.349352+0000","last_peered":"2024-04-02T09:12:40.349352+0000","last_clean":"2024-04-02T09:12:40.349352+0000","last_became_active":"2024-04-02T09:12:39.141641+0000","last_became_peered":"2024-04-02T09:12:39.141641+0000","last_unstale":"2024-04-02T09:12:40.349352+0000","last_undegraded":"2024-04-02T09:12:40.349352+0000","last_fullsized":"2024-04-02T09:12:40.349352+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:56.650439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114225+0000","last_change":"2024-04-02T09:12:25.203968+0000","last_active":"2024-04-02T09:12:39.114225+0000","last_peered":"2024-04-02T09:12:39.114225+0000","last_clean":"2024-04-02T09:12:39.114225+0000","last_became_active":"2024-04-02T09:12:25.203714+0000","last_became_peered":"2024-04-02T09:12:25.203714+0000","last_unstale":"2024-04-02T09:12:39.114225+0000","last_undegraded":"2024-04-02T09:12:39.114225+0000","last_fullsized":"2024-04-02T09:12:39.114225+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:44:28.685030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347121+0000","last_change":"2024-04-02T09:12:39.129719+0000","last_active":"2024-04-02T09:12:40.347121+0000","last_peered":"2024-04-02T09:12:40.347121+0000","last_clean":"2024-04-02T09:12:40.347121+0000","last_became_active":"2024-04-02T09:12:39.129414+0000","last_became_peered":"2024-04-02T09:12:39.129414+0000","last_unstale":"2024-04-02T09:12:40.347121+0000","last_undegraded":"2024-04-02T09:12:40.347121+0000","last_fullsized":"2024-04-02T09:12:40.347121+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:44.771579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.119899+0000","last_change":"2024-04-02T09:12:25.202349+0000","last_active":"2024-04-02T09:12:39.119899+0000","last_peered":"2024-04-02T09:12:39.119899+0000","last_clean":"2024-04-02T09:12:39.119899+0000","last_became_active":"2024-04-02T09:12:25.200134+0000","last_became_peered":"2024-04-02T09:12:25.200134+0000","last_unstale":"2024-04-02T09:12:39.119899+0000","last_undegraded":"2024-04-02T09:12:39.119899+0000","last_fullsized":"2024-04-02T09:12:39.119899+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:45:54.007817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117390+0000","last_change":"2024-04-02T09:12:39.136859+0000","last_active":"2024-04-02T09:12:40.117390+0000","last_peered":"2024-04-02T09:12:40.117390+0000","last_clean":"2024-04-02T09:12:40.117390+0000","last_became_active":"2024-04-02T09:12:39.136602+0000","last_became_peered":"2024-04-02T09:12:39.136602+0000","last_unstale":"2024-04-02T09:12:40.117390+0000","last_undegraded":"2024-04-02T09:12:40.117390+0000","last_fullsized":"2024-04-02T09:12:40.117390+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:16.285248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.118534+0000","last_change":"2024-04-02T09:12:25.201593+0000","last_active":"2024-04-02T09:12:39.118534+0000","last_peered":"2024-04-02T09:12:39.118534+0000","last_clean":"2024-04-02T09:12:39.118534+0000","last_became_active":"2024-04-02T09:12:25.200851+0000","last_became_peered":"2024-04-02T09:12:25.200851+0000","last_unstale":"2024-04-02T09:12:39.118534+0000","last_undegraded":"2024-04-02T09:12:39.118534+0000","last_fullsized":"2024-04-02T09:12:39.118534+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:26:33.117159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.113284+0000","last_change":"2024-04-02T09:12:39.144609+0000","last_active":"2024-04-02T09:12:40.113284+0000","last_peered":"2024-04-02T09:12:40.113284+0000","last_clean":"2024-04-02T09:12:40.113284+0000","last_became_active":"2024-04-02T09:12:39.144321+0000","last_became_peered":"2024-04-02T09:12:39.144321+0000","last_unstale":"2024-04-02T09:12:40.113284+0000","last_undegraded":"2024-04-02T09:12:40.113284+0000","last_fullsized":"2024-04-02T09:12:40.113284+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:37:05.275434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.120759+0000","last_change":"2024-04-02T09:12:25.191970+0000","last_active":"2024-04-02T09:12:39.120759+0000","last_peered":"2024-04-02T09:12:39.120759+0000","last_clean":"2024-04-02T09:12:39.120759+0000","last_became_active":"2024-04-02T09:12:25.191572+0000","last_became_peered":"2024-04-02T09:12:25.191572+0000","last_unstale":"2024-04-02T09:12:39.120759+0000","last_undegraded":"2024-04-02T09:12:39.120759+0000","last_fullsized":"2024-04-02T09:12:39.120759+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:03.387546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348766+0000","last_change":"2024-04-02T09:12:39.140664+0000","last_active":"2024-04-02T09:12:40.348766+0000","last_peered":"2024-04-02T09:12:40.348766+0000","last_clean":"2024-04-02T09:12:40.348766+0000","last_became_active":"2024-04-02T09:12:39.140336+0000","last_became_peered":"2024-04-02T09:12:39.140336+0000","last_unstale":"2024-04-02T09:12:40.348766+0000","last_undegraded":"2024-04-02T09:12:40.348766+0000","last_fullsized":"2024-04-02T09:12:40.348766+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:23:36.313057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.115897+0000","last_change":"2024-04-02T09:12:25.201485+0000","last_active":"2024-04-02T09:12:39.115897+0000","last_peered":"2024-04-02T09:12:39.115897+0000","last_clean":"2024-04-02T09:12:39.115897+0000","last_became_active":"2024-04-02T09:12:25.200542+0000","last_became_peered":"2024-04-02T09:12:25.200542+0000","last_unstale":"2024-04-02T09:12:39.115897+0000","last_undegraded":"2024-04-02T09:12:39.115897+0000","last_fullsized":"2024-04-02T09:12:39.115897+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:19:30.245770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348141+0000","last_change":"2024-04-02T09:12:39.129069+0000","last_active":"2024-04-02T09:12:40.348141+0000","last_peered":"2024-04-02T09:12:40.348141+0000","last_clean":"2024-04-02T09:12:40.348141+0000","last_became_active":"2024-04-02T09:12:39.128806+0000","last_became_peered":"2024-04-02T09:12:39.128806+0000","last_unstale":"2024-04-02T09:12:40.348141+0000","last_undegraded":"2024-04-02T09:12:40.348141+0000","last_fullsized":"2024-04-02T09:12:40.348141+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:32.146966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114164+0000","last_change":"2024-04-02T09:12:25.200889+0000","last_active":"2024-04-02T09:12:39.114164+0000","last_peered":"2024-04-02T09:12:39.114164+0000","last_clean":"2024-04-02T09:12:39.114164+0000","last_became_active":"2024-04-02T09:12:25.200635+0000","last_became_peered":"2024-04-02T09:12:25.200635+0000","last_unstale":"2024-04-02T09:12:39.114164+0000","last_undegraded":"2024-04-02T09:12:39.114164+0000","last_fullsized":"2024-04-02T09:12:39.114164+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:43:29.229527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.113023+0000","last_change":"2024-04-02T09:12:39.125881+0000","last_active":"2024-04-02T09:12:40.113023+0000","last_peered":"2024-04-02T09:12:40.113023+0000","last_clean":"2024-04-02T09:12:40.113023+0000","last_became_active":"2024-04-02T09:12:39.124319+0000","last_became_peered":"2024-04-02T09:12:39.124319+0000","last_unstale":"2024-04-02T09:12:40.113023+0000","last_undegraded":"2024-04-02T09:12:40.113023+0000","last_fullsized":"2024-04-02T09:12:40.113023+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:55:45.430413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.121484+0000","last_change":"2024-04-02T09:12:25.191955+0000","last_active":"2024-04-02T09:12:39.121484+0000","last_peered":"2024-04-02T09:12:39.121484+0000","last_clean":"2024-04-02T09:12:39.121484+0000","last_became_active":"2024-04-02T09:12:25.191699+0000","last_became_peered":"2024-04-02T09:12:25.191699+0000","last_unstale":"2024-04-02T09:12:39.121484+0000","last_undegraded":"2024-04-02T09:12:39.121484+0000","last_fullsized":"2024-04-02T09:12:39.121484+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:22:42.917826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116986+0000","last_change":"2024-04-02T09:12:39.132319+0000","last_active":"2024-04-02T09:12:40.116986+0000","last_peered":"2024-04-02T09:12:40.116986+0000","last_clean":"2024-04-02T09:12:40.116986+0000","last_became_active":"2024-04-02T09:12:39.131855+0000","last_became_peered":"2024-04-02T09:12:39.131855+0000","last_unstale":"2024-04-02T09:12:40.116986+0000","last_undegraded":"2024-04-02T09:12:40.116986+0000","last_fullsized":"2024-04-02T09:12:40.116986+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:25.746845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.133348+0000","last_change":"2024-04-02T09:12:25.190996+0000","last_active":"2024-04-02T09:12:39.133348+0000","last_peered":"2024-04-02T09:12:39.133348+0000","last_clean":"2024-04-02T09:12:39.133348+0000","last_became_active":"2024-04-02T09:12:25.190655+0000","last_became_peered":"2024-04-02T09:12:25.190655+0000","last_unstale":"2024-04-02T09:12:39.133348+0000","last_undegraded":"2024-04-02T09:12:39.133348+0000","last_fullsized":"2024-04-02T09:12:39.133348+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:48:22.188323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347985+0000","last_change":"2024-04-02T09:12:39.132496+0000","last_active":"2024-04-02T09:12:40.347985+0000","last_peered":"2024-04-02T09:12:40.347985+0000","last_clean":"2024-04-02T09:12:40.347985+0000","last_became_active":"2024-04-02T09:12:39.132240+0000","last_became_peered":"2024-04-02T09:12:39.132240+0000","last_unstale":"2024-04-02T09:12:40.347985+0000","last_undegraded":"2024-04-02T09:12:40.347985+0000","last_fullsized":"2024-04-02T09:12:40.347985+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:14:26.132847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116570+0000","last_change":"2024-04-02T09:12:25.194789+0000","last_active":"2024-04-02T09:12:39.116570+0000","last_peered":"2024-04-02T09:12:39.116570+0000","last_clean":"2024-04-02T09:12:39.116570+0000","last_became_active":"2024-04-02T09:12:25.194433+0000","last_became_peered":"2024-04-02T09:12:25.194433+0000","last_unstale":"2024-04-02T09:12:39.116570+0000","last_undegraded":"2024-04-02T09:12:39.116570+0000","last_fullsized":"2024-04-02T09:12:39.116570+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:18.798282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116611+0000","last_change":"2024-04-02T09:12:39.134852+0000","last_active":"2024-04-02T09:12:40.116611+0000","last_peered":"2024-04-02T09:12:40.116611+0000","last_clean":"2024-04-02T09:12:40.116611+0000","last_became_active":"2024-04-02T09:12:39.134565+0000","last_became_peered":"2024-04-02T09:12:39.134565+0000","last_unstale":"2024-04-02T09:12:40.116611+0000","last_undegraded":"2024-04-02T09:12:40.116611+0000","last_fullsized":"2024-04-02T09:12:40.116611+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:11.640312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117583+0000","last_change":"2024-04-02T09:12:25.208060+0000","last_active":"2024-04-02T09:12:39.117583+0000","last_peered":"2024-04-02T09:12:39.117583+0000","last_clean":"2024-04-02T09:12:39.117583+0000","last_became_active":"2024-04-02T09:12:25.206932+0000","last_became_peered":"2024-04-02T09:12:25.206932+0000","last_unstale":"2024-04-02T09:12:39.117583+0000","last_undegraded":"2024-04-02T09:12:39.117583+0000","last_fullsized":"2024-04-02T09:12:39.117583+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:11.810703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112484+0000","last_change":"2024-04-02T09:12:39.122041+0000","last_active":"2024-04-02T09:12:40.112484+0000","last_peered":"2024-04-02T09:12:40.112484+0000","last_clean":"2024-04-02T09:12:40.112484+0000","last_became_active":"2024-04-02T09:12:39.119213+0000","last_became_peered":"2024-04-02T09:12:39.119213+0000","last_unstale":"2024-04-02T09:12:40.112484+0000","last_undegraded":"2024-04-02T09:12:40.112484+0000","last_fullsized":"2024-04-02T09:12:40.112484+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:37:26.600229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116911+0000","last_change":"2024-04-02T09:12:25.187698+0000","last_active":"2024-04-02T09:12:39.116911+0000","last_peered":"2024-04-02T09:12:39.116911+0000","last_clean":"2024-04-02T09:12:39.116911+0000","last_became_active":"2024-04-02T09:12:25.187420+0000","last_became_peered":"2024-04-02T09:12:25.187420+0000","last_unstale":"2024-04-02T09:12:39.116911+0000","last_undegraded":"2024-04-02T09:12:39.116911+0000","last_fullsized":"2024-04-02T09:12:39.116911+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:45:21.225286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3603,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575991,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60148,"kb_used_data":2792,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311692,"statfs":{"total":96636764160,"available":96575172608,"internally_reserved":0,"allocated":2859008,"data_stored":1450119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.68500000000000005,"15min":0.71199999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.93899999999999995,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68700000000000006,"15min":0.71999999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91800000000000004,"5min":0.94199999999999995,"15min":1.165},"last":0.65100000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.70599999999999996,"15min":0.73099999999999998},"min":{"1min":0.48899999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.95399999999999996,"5min":0.95699999999999996,"15min":1.125},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68100000000000005,"15min":0.71399999999999997},"min":{"1min":0.51700000000000002,"5min":0.497,"15min":0.42399999999999999},"max":{"1min":0.89100000000000001,"5min":0.92700000000000005,"15min":1.042},"last":0.621}]},{"osd":2,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.749,"15min":0.76100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.873,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.78200000000000003,"15min":0.80500000000000005},"min":{"1min":0.54900000000000004,"5min":0.504,"15min":0.504},"max":{"1min":0.98299999999999998,"5min":1.036,"15min":1.109},"last":0.68200000000000005}]},{"osd":3,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.72399999999999998,"15min":0.74099999999999999},"min":{"1min":0.497,"5min":0.45800000000000002,"15min":0.41399999999999998},"max":{"1min":0.94399999999999995,"5min":1.452,"15min":1.452},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.748,"15min":0.76400000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.0740000000000001},"last":0.70999999999999996}]},{"osd":4,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61699999999999999,"15min":0.60799999999999998},"min":{"1min":0.42799999999999999,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.95299999999999996,"15min":1.01},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.73699999999999999,"15min":0.71699999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.38300000000000001},"max":{"1min":1.006,"5min":1.536,"15min":1.536},"last":0.80300000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.68799999999999994,"15min":0.67500000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":0.93400000000000005,"5min":1.0169999999999999,"15min":1.1850000000000001},"last":0.746},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.72499999999999998,"15min":0.71699999999999997},"min":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.40600000000000003},"max":{"1min":0.88500000000000001,"5min":1.097,"15min":1.1319999999999999},"last":0.73799999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45800000000000002},"max":{"1min":0.871,"5min":1.034,"15min":1.3919999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.72299999999999998,"15min":0.69099999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.97299999999999998,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.77400000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575989,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42112,"kb_used_data":2796,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329728,"statfs":{"total":96636764160,"available":96593641472,"internally_reserved":0,"allocated":2863104,"data_stored":1449641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.73699999999999999},"min":{"1min":0.54900000000000004,"5min":0.50800000000000001,"15min":0.42699999999999999},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73199999999999998,"15min":0.747},"min":{"1min":0.48699999999999999,"5min":0.47899999999999998,"15min":0.45800000000000002},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":1.077},"last":0.59399999999999997}]},{"osd":1,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70399999999999996,"15min":0.72099999999999997},"min":{"1min":0.57199999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.212},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.73499999999999999,"15min":0.75800000000000001},"min":{"1min":0.55700000000000005,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.65300000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.73599999999999999,"15min":0.75800000000000001},"min":{"1min":0.53500000000000003,"5min":0.47699999999999998,"15min":0.47199999999999998},"max":{"1min":1.016,"5min":1.016,"15min":1.2789999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.73899999999999999,"15min":0.748},"min":{"1min":0.50900000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.93300000000000005,"5min":0.95999999999999996,"15min":1.226},"last":0.50700000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.74399999999999999,"15min":0.76900000000000002},"min":{"1min":0.50800000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.141,"5min":1.141,"15min":1.2549999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76900000000000002,"15min":0.77500000000000002},"min":{"1min":0.63100000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.94099999999999995,"5min":0.95399999999999996,"15min":1.3200000000000001},"last":0.82399999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60799999999999998,"15min":0.623},"min":{"1min":0.45300000000000001,"5min":0.42199999999999999,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.1259999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.627,"5min":0.65100000000000002,"15min":0.66700000000000004},"min":{"1min":0.51100000000000001,"5min":0.47399999999999998,"15min":0.41999999999999998},"max":{"1min":0.84399999999999997,"5min":0.92300000000000004,"15min":1.1879999999999999},"last":0.72999999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64500000000000002,"15min":0.65600000000000003},"min":{"1min":0.47899999999999998,"5min":0.42799999999999999,"15min":0.42199999999999999},"max":{"1min":0.86599999999999999,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70099999999999996,"15min":0.72599999999999998},"min":{"1min":0.505,"5min":0.48299999999999998,"15min":0.46700000000000003},"max":{"1min":0.92000000000000004,"5min":1.034,"15min":1.204},"last":0.747}]},{"osd":7,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.2370000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.72199999999999998,"15min":0.73899999999999999},"min":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.48299999999999998},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.77500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575990,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52544,"kb_used_data":3372,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319296,"statfs":{"total":96636764160,"available":96582959104,"internally_reserved":0,"allocated":3452928,"data_stored":2040467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44400000000000001,"15min":0.45300000000000001},"min":{"1min":0.35499999999999998,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.67000000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.502,"15min":0.51200000000000001},"min":{"1min":0.371,"5min":0.312,"15min":0.312},"max":{"1min":0.76000000000000001,"5min":0.78600000000000003,"15min":0.79800000000000004},"last":0.39500000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.50900000000000001,"15min":0.505},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.753,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58299999999999996,"15min":0.59699999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74099999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53100000000000003}]},{"osd":3,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.42699999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76200000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.505},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.40100000000000002,"5min":0.39000000000000001,"15min":0.378},"max":{"1min":0.73199999999999998,"5min":0.86699999999999999,"15min":0.878},"last":0.47999999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.79700000000000004,"15min":0.79200000000000004},"min":{"1min":0.61099999999999999,"5min":0.57599999999999996,"15min":0.433},"max":{"1min":0.98499999999999999,"5min":1.1120000000000001,"15min":1.1479999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.78000000000000003,"15min":0.78900000000000003},"min":{"1min":0.61299999999999999,"5min":0.57199999999999995,"15min":0.41799999999999998},"max":{"1min":1.05,"5min":1.05,"15min":1.25},"last":0.91200000000000003}]},{"osd":5,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.77700000000000002,"15min":0.79600000000000004},"min":{"1min":0.59799999999999998,"5min":0.54300000000000004,"15min":0.52900000000000003},"max":{"1min":1.01,"5min":1.115,"15min":1.321},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.65800000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.173},"last":0.79300000000000004}]},{"osd":6,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.81200000000000006,"15min":0.81799999999999995},"min":{"1min":0.67300000000000004,"5min":0.53900000000000003,"15min":0.48099999999999998},"max":{"1min":1.0649999999999999,"5min":1.159,"15min":1.2210000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.82199999999999995,"15min":0.81299999999999994},"min":{"1min":0.61899999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.087,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":1.0009999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.82099999999999995,"15min":0.82799999999999996},"min":{"1min":0.64200000000000002,"5min":0.58499999999999996,"15min":0.53700000000000003},"max":{"1min":1.0680000000000001,"5min":1.1080000000000001,"15min":1.236},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.83899999999999997,"15min":0.83999999999999997},"min":{"1min":0.63700000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0329999999999999,"5min":1.0720000000000001,"15min":1.2789999999999999},"last":0.84999999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575990,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47860,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323980,"statfs":{"total":96636764160,"available":96587755520,"internally_reserved":0,"allocated":2846720,"data_stored":1448988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45400000000000001,"15min":0.45800000000000002},"min":{"1min":0.33700000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.69899999999999995,"5min":0.76500000000000001,"15min":0.86699999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.5,"15min":0.51000000000000001},"min":{"1min":0.32800000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.54900000000000004}]},{"osd":2,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.42099999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66100000000000003,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.503},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53900000000000003,"15min":0.55400000000000005},"min":{"1min":0.36699999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72499999999999998,"5min":0.999,"15min":0.999},"last":0.59499999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.40100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.77800000000000002,"5min":0.80500000000000005,"15min":0.85699999999999998},"last":0.437},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.63900000000000001,"15min":0.64300000000000002},"min":{"1min":0.5,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.82799999999999996,"5min":0.92000000000000004,"15min":0.96399999999999997},"last":0.64600000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.82499999999999996,"15min":0.83299999999999996},"min":{"1min":0.65700000000000003,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.2769999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81000000000000005,"15min":0.82399999999999995},"min":{"1min":0.621,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.95999999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.80100000000000005,"15min":0.82699999999999996},"min":{"1min":0.58899999999999997,"5min":0.497,"15min":0.497},"max":{"1min":1.1419999999999999,"5min":1.145,"15min":1.232},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.83499999999999996,"15min":0.83399999999999996},"min":{"1min":0.67700000000000005,"5min":0.57699999999999996,"15min":0.55900000000000005},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.89900000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.58099999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1830000000000001,"5min":1.196,"15min":1.196},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.80500000000000005,"15min":0.81200000000000006},"min":{"1min":0.60499999999999998,"5min":0.52900000000000003,"15min":0.499},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.747}]},{"osd":7,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.84399999999999997,"15min":0.85399999999999998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.875},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.84099999999999997,"15min":0.85099999999999998},"min":{"1min":0.67700000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.1100000000000001,"5min":1.1719999999999999,"15min":1.2589999999999999},"last":0.91200000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575989,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51972,"kb_used_data":2804,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319868,"statfs":{"total":96636764160,"available":96583544832,"internally_reserved":0,"allocated":2871296,"data_stored":1452368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44},"min":{"1min":0.32200000000000001,"5min":0.30499999999999999,"15min":0.29199999999999998},"max":{"1min":0.61499999999999999,"5min":0.66600000000000004,"15min":0.72999999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.502,"15min":0.503},"min":{"1min":0.375,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64800000000000002,"5min":0.79900000000000004,"15min":1.4930000000000001},"last":0.44700000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.40999999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65800000000000003,"5min":0.75800000000000001,"15min":0.80600000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.40100000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.68799999999999994,"5min":0.81999999999999995,"15min":1.4530000000000001},"last":0.52200000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.56000000000000005},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.80700000000000005,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.497},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.628,"15min":0.63500000000000001},"min":{"1min":0.50900000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77700000000000002,"5min":0.86799999999999999,"15min":1.0740000000000001},"last":0.59099999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.77700000000000002,"15min":0.77900000000000003},"min":{"1min":0.53700000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.008,"5min":1.121,"15min":1.121},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79300000000000004,"15min":0.79800000000000004},"min":{"1min":0.65200000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.1259999999999999,"5min":1.179,"15min":1.5329999999999999},"last":0.871}]},{"osd":5,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80000000000000004,"15min":0.79300000000000004},"min":{"1min":0.69899999999999995,"5min":0.53100000000000003,"15min":0.52600000000000002},"max":{"1min":1.054,"5min":1.99,"15min":1.99},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80500000000000005,"15min":0.80800000000000005},"min":{"1min":0.60499999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0660000000000001,"5min":1.101,"15min":1.1259999999999999},"last":0.70099999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.81599999999999995,"15min":0.81799999999999995},"min":{"1min":0.63500000000000001,"5min":0.54700000000000004,"15min":0.54600000000000004},"max":{"1min":0.96899999999999997,"5min":1.173,"15min":1.5940000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80400000000000005,"15min":0.81100000000000005},"min":{"1min":0.63200000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.99199999999999999,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.61899999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.83199999999999996,"15min":0.83499999999999996},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55100000000000005},"max":{"1min":1.165,"5min":1.2110000000000001,"15min":1.5649999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.81999999999999995,"15min":0.83099999999999996},"min":{"1min":0.70499999999999996,"5min":0.497,"15min":0.497},"max":{"1min":1.0269999999999999,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.68500000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575989,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51960,"kb_used_data":2792,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319880,"statfs":{"total":96636764160,"available":96583557120,"internally_reserved":0,"allocated":2859008,"data_stored":1452230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44400000000000001,"15min":0.45300000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.48699999999999999,"15min":0.496},"min":{"1min":0.36099999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.876},"last":0.42899999999999999}]},{"osd":1,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.49399999999999999,"15min":0.502},"min":{"1min":0.34300000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.64000000000000001,"5min":0.71499999999999997,"15min":0.90300000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.57699999999999996},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.34100000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.622,"15min":0.63500000000000001},"min":{"1min":0.42599999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.61199999999999999,"15min":0.623},"min":{"1min":0.48099999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81699999999999995,"5min":0.82299999999999995,"15min":1.0840000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.74399999999999999,"15min":0.75900000000000001},"min":{"1min":0.503,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.071,"5min":1.071,"15min":1.0800000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.59099999999999997,"5min":0.57199999999999995,"15min":0.49199999999999999},"max":{"1min":1.018,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.67000000000000004}]},{"osd":5,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.59599999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.054,"5min":1.1459999999999999,"15min":1.157},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.56599999999999995,"5min":0.505,"15min":0.505},"max":{"1min":0.99099999999999999,"5min":1.266,"15min":1.266},"last":0.64700000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.78500000000000003,"15min":0.79100000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.52600000000000002},"max":{"1min":1.0489999999999999,"5min":1.284,"15min":1.284},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.80400000000000005,"15min":0.81200000000000006},"min":{"1min":0.54900000000000004,"5min":0.495,"15min":0.495},"max":{"1min":1.04,"5min":1.2,"15min":1.2},"last":0.72299999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.80400000000000005,"15min":0.81499999999999995},"min":{"1min":0.61399999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.024,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.57999999999999996,"5min":0.55700000000000005,"15min":0.53100000000000003},"max":{"1min":1.103,"5min":1.248,"15min":1.248},"last":0.71199999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575989,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47864,"kb_used_data":2796,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323976,"statfs":{"total":96636764160,"available":96587751424,"internally_reserved":0,"allocated":2863104,"data_stored":1451079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68000000000000005,"15min":0.69899999999999995},"min":{"1min":0.55000000000000004,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66500000000000004,"15min":0.68899999999999995},"min":{"1min":0.40999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.83499999999999996,"5min":0.86199999999999999,"15min":1.075},"last":0.78100000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.73599999999999999,"15min":0.76100000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":1.0329999999999999,"5min":1.0549999999999999,"15min":1.069},"last":0.877},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.75,"15min":0.77700000000000002},"min":{"1min":0.53900000000000003,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.91400000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.69599999999999995,"15min":0.71799999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70499999999999996,"15min":0.73499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98999999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.85199999999999998}]},{"osd":3,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75600000000000001,"15min":0.77000000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.47399999999999998},"max":{"1min":0.98099999999999998,"5min":0.998,"15min":1.0700000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.51800000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.002,"5min":1.002,"15min":1.028},"last":0.81299999999999994}]},{"osd":5,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.60199999999999998,"15min":0.61099999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.91900000000000004,"5min":1.008,"15min":1.016},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60399999999999998,"15min":0.61699999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.56299999999999994}]},{"osd":6,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.64100000000000001,"15min":0.64200000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67300000000000004,"15min":0.69099999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":0.98299999999999998,"5min":1.0269999999999999,"15min":1.0469999999999999},"last":0.74199999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.0329999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.76600000000000001,"15min":0.78300000000000003},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0960000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.82799999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575992,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52552,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319288,"statfs":{"total":96636764160,"available":96582950912,"internally_reserved":0,"allocated":3469312,"data_stored":2041136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69699999999999995,"15min":0.70999999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.89600000000000002,"5min":0.98699999999999999,"15min":1.004},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69899999999999995,"15min":0.72499999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.88600000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.70099999999999996,"15min":0.72199999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85999999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.71399999999999997,"15min":0.73199999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.052},"last":0.86699999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73199999999999998,"15min":0.745},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93200000000000005,"5min":1.004,"15min":1.014},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72599999999999998,"15min":0.746},"min":{"1min":0.46600000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.018,"5min":1.018,"15min":1.0680000000000001},"last":0.65900000000000003}]},{"osd":3,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77000000000000002,"15min":0.78600000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":0.98899999999999999,"15min":1.083},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.59099999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94399999999999995,"5min":1.008,"15min":1.1020000000000001},"last":0.75}]},{"osd":4,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.629,"15min":0.629},"min":{"1min":0.56499999999999995,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.89100000000000001,"5min":0.96899999999999997,"15min":1.054},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.68400000000000005,"15min":0.67500000000000004},"min":{"1min":0.47999999999999998,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.85099999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.434,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71199999999999997,"15min":0.71099999999999997},"min":{"1min":0.501,"5min":0.44600000000000001,"15min":0.42699999999999999},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.82099999999999995}]},{"osd":7,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.73999999999999999,"15min":0.73499999999999999},"min":{"1min":0.60699999999999998,"5min":0.441,"15min":0.441},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.64000000000000001,"5min":0.48199999999999998,"15min":0.45700000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.79800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:12:43.503 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:12:43.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.727+0000 7f0bd9400700 1 Processor -- start 2024-04-02T09:12:43.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.727+0000 7f0bd9400700 1 -- start start 2024-04-02T09:12:43.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.728+0000 7f0bd9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0bd40ceec0 0x7f0bd41612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:43.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.728+0000 7f0bd9400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0bd4161800 0x7f0bd40b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:43.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.728+0000 7f0bd9400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd40d30d0 con 0x7f0bd415d640 2024-04-02T09:12:43.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.728+0000 7f0bd9400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd4161c90 con 0x7f0bd4161800 2024-04-02T09:12:43.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.728+0000 7f0bd9400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd40b6840 con 0x7f0bd40ceec0 2024-04-02T09:12:43.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.728+0000 7f0bd3200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0bd4161800 0x7f0bd40b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:43.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.728+0000 7f0bd3200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0bd4161800 0x7f0bd40b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59626/0 (socket says 172.21.15.89:59626) 2024-04-02T09:12:43.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.728+0000 7f0bd3200700 1 -- 172.21.15.89:0/2917600069 learned_addr learned my addr 172.21.15.89:0/2917600069 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:43.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.728+0000 7f0bd3200700 1 -- 172.21.15.89:0/2917600069 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0bd40ceec0 msgr2=0x7f0bd41612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:43.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.728+0000 7f0bd3200700 1 --2- 172.21.15.89:0/2917600069 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0bd40ceec0 0x7f0bd41612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.728+0000 7f0bd3200700 1 -- 172.21.15.89:0/2917600069 >> v1:172.21.15.89:6789/0 conn(0x7f0bd415d640 legacy=0x7f0bd40d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:43.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.728+0000 7f0bd3200700 1 -- 172.21.15.89:0/2917600069 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd40d2b30 con 0x7f0bd4161800 2024-04-02T09:12:43.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.729+0000 7f0bd3200700 1 --2- 172.21.15.89:0/2917600069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd4161800 0x7f0bd40b6cd0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009e50 tx=0x7f0bcc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab37add6fc33658 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:43.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.729+0000 7f0bd1400700 1 -- 172.21.15.89:0/2917600069 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc00e070 con 0x7f0bd4161800 2024-04-02T09:12:43.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.729+0000 7f0bd1400700 1 -- 172.21.15.89:0/2917600069 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc00b9c0 con 0x7f0bd4161800 2024-04-02T09:12:43.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.729+0000 7f0bd1400700 1 -- 172.21.15.89:0/2917600069 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc00bd70 con 0x7f0bd4161800 2024-04-02T09:12:43.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.729+0000 7f0bd9400700 1 -- 172.21.15.89:0/2917600069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd4161800 msgr2=0x7f0bd40b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:43.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.729+0000 7f0bd9400700 1 --2- 172.21.15.89:0/2917600069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd4161800 0x7f0bd40b6cd0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009e50 tx=0x7f0bcc00b330 comp rx=0 tx=0).stop 2024-04-02T09:12:43.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.729+0000 7f0bd9400700 1 -- 172.21.15.89:0/2917600069 shutdown_connections 2024-04-02T09:12:43.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.729+0000 7f0bd9400700 1 --2- 172.21.15.89:0/2917600069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd4161800 0x7f0bd40b6cd0 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.729+0000 7f0bd9400700 1 --2- 172.21.15.89:0/2917600069 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0bd40ceec0 0x7f0bd41612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.730+0000 7f0bd9400700 1 -- 172.21.15.89:0/2917600069 >> 172.21.15.89:0/2917600069 conn(0x7f0bd408aff0 msgr2=0x7f0bd40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:43.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.730+0000 7f0bd9400700 1 -- 172.21.15.89:0/2917600069 shutdown_connections 2024-04-02T09:12:43.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.730+0000 7f0bd9400700 1 -- 172.21.15.89:0/2917600069 wait complete. 2024-04-02T09:12:43.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.731+0000 7f0bd9400700 1 Processor -- start 2024-04-02T09:12:43.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.731+0000 7f0bd9400700 1 -- start start 2024-04-02T09:12:43.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.732+0000 7f0bd9400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bd40ceec0 0x7f0bd40c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:43.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.732+0000 7f0bd9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd415d640 0x7f0bd415eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:43.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.732+0000 7f0bd9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0bd4161800 0x7f0bd415f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:43.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.732+0000 7f0bd9400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd40c8370 con 0x7f0bd415d640 2024-04-02T09:12:43.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.732+0000 7f0bd9400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd40c8930 con 0x7f0bd40ceec0 2024-04-02T09:12:43.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.732+0000 7f0bd9400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd41f3b80 con 0x7f0bd4161800 2024-04-02T09:12:43.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.732+0000 7f0bd1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd415d640 0x7f0bd415eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:43.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.732+0000 7f0bd1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd415d640 0x7f0bd415eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59630/0 (socket says 172.21.15.89:59630) 2024-04-02T09:12:43.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.732+0000 7f0bd1e00700 1 -- 172.21.15.89:0/1373024257 learned_addr learned my addr 172.21.15.89:0/1373024257 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:43.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.733+0000 7f0bd2800700 1 --2- 172.21.15.89:0/1373024257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bd40ceec0 0x7f0bd40c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:43.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.733+0000 7f0bd3200700 1 --2- 172.21.15.89:0/1373024257 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0bd4161800 0x7f0bd415f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:43.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.733+0000 7f0bd1e00700 1 -- 172.21.15.89:0/1373024257 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0bd4161800 msgr2=0x7f0bd415f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:43.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.733+0000 7f0bd1e00700 1 --2- 172.21.15.89:0/1373024257 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0bd4161800 0x7f0bd415f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.733+0000 7f0bd1e00700 1 -- 172.21.15.89:0/1373024257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bd40ceec0 msgr2=0x7f0bd40c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:43.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.733+0000 7f0bd1e00700 1 --2- 172.21.15.89:0/1373024257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bd40ceec0 0x7f0bd40c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.733+0000 7f0bd1e00700 1 -- 172.21.15.89:0/1373024257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc009b30 con 0x7f0bd415d640 2024-04-02T09:12:43.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.733+0000 7f0bd1e00700 1 --2- 172.21.15.89:0/1373024257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd415d640 0x7f0bd415eda0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000b4d0 tx=0x7f0bc000b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:43.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.733+0000 7f0bcaa00700 1 -- 172.21.15.89:0/1373024257 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc000bea0 con 0x7f0bd415d640 2024-04-02T09:12:43.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.733+0000 7f0bcaa00700 1 -- 172.21.15.89:0/1373024257 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc0011830 con 0x7f0bd415d640 2024-04-02T09:12:43.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.733+0000 7f0bd9400700 1 -- 172.21.15.89:0/1373024257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd41f3e60 con 0x7f0bd415d640 2024-04-02T09:12:43.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.733+0000 7f0bcaa00700 1 -- 172.21.15.89:0/1373024257 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0011be0 con 0x7f0bd415d640 2024-04-02T09:12:43.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.734+0000 7f0bd9400700 1 -- 172.21.15.89:0/1373024257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd40ca800 con 0x7f0bd415d640 2024-04-02T09:12:43.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.734+0000 7f0bd9400700 1 -- 172.21.15.89:0/1373024257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd40c8ac0 con 0x7f0bd415d640 2024-04-02T09:12:43.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.735+0000 7f0bcaa00700 1 -- 172.21.15.89:0/1373024257 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0bc0011d40 con 0x7f0bd415d640 2024-04-02T09:12:43.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.736+0000 7f0bcaa00700 1 --2- 172.21.15.89:0/1373024257 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0bb406adb0 0x7f0bb406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:43.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.736+0000 7f0bcaa00700 1 -- 172.21.15.89:0/1373024257 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f0bc00898f0 con 0x7f0bd415d640 2024-04-02T09:12:43.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.736+0000 7f0bd2800700 1 --2- 172.21.15.89:0/1373024257 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0bb406adb0 0x7f0bb406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:43.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.737+0000 7f0bd2800700 1 --2- 172.21.15.89:0/1373024257 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0bb406adb0 0x7f0bb406d270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0bd40b9450 tx=0x7f0bc4009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:43.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.741+0000 7f0bcaa00700 1 -- 172.21.15.89:0/1373024257 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0bc0054410 con 0x7f0bd415d640 2024-04-02T09:12:43.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.938+0000 7f0bd9400700 1 -- 172.21.15.89:0/1373024257 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0bd40b64d0 con 0x7f0bb406adb0 2024-04-02T09:12:43.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.952+0000 7f0bcaa00700 1 -- 172.21.15.89:0/1373024257 <== mgr.10923 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528387 (secure 0 0 0) 0x7f0bc4001580 con 0x7f0bb406adb0 2024-04-02T09:12:43.958 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:43.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.958+0000 7f0bd9400700 1 -- 172.21.15.89:0/1373024257 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0bb406adb0 msgr2=0x7f0bb406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:43.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.958+0000 7f0bd9400700 1 --2- 172.21.15.89:0/1373024257 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0bb406adb0 0x7f0bb406d270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0bd40b9450 tx=0x7f0bc4009040 comp rx=0 tx=0).stop 2024-04-02T09:12:43.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.958+0000 7f0bd9400700 1 -- 172.21.15.89:0/1373024257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd415d640 msgr2=0x7f0bd415eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:43.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.958+0000 7f0bd9400700 1 --2- 172.21.15.89:0/1373024257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd415d640 0x7f0bd415eda0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000b4d0 tx=0x7f0bc000b7e0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.958+0000 7f0bd9400700 1 -- 172.21.15.89:0/1373024257 shutdown_connections 2024-04-02T09:12:43.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.958+0000 7f0bd9400700 1 --2- 172.21.15.89:0/1373024257 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0bb406adb0 0x7f0bb406d270 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.958+0000 7f0bd9400700 1 --2- 172.21.15.89:0/1373024257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bd40ceec0 0x7f0bd40c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.958+0000 7f0bd9400700 1 --2- 172.21.15.89:0/1373024257 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0bd4161800 0x7f0bd415f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.958+0000 7f0bd9400700 1 --2- 172.21.15.89:0/1373024257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bd415d640 0x7f0bd415eda0 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:43.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.958+0000 7f0bd9400700 1 -- 172.21.15.89:0/1373024257 >> 172.21.15.89:0/1373024257 conn(0x7f0bd408aff0 msgr2=0x7f0bd40b7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:43.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.958+0000 7f0bd9400700 1 -- 172.21.15.89:0/1373024257 shutdown_connections 2024-04-02T09:12:43.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:43.958+0000 7f0bd9400700 1 -- 172.21.15.89:0/1373024257 wait complete. 2024-04-02T09:12:43.964 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:12:43.976 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-02T09:12:42.802655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596333,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":671,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":360,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":407012,"kb_used_data":23520,"kb_used_omap":12,"kb_used_meta":383347,"kb_avail":754567708,"statfs":{"total":773094113280,"available":772677332992,"internally_reserved":0,"allocated":24084480,"data_stored":12786028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":392547896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.196694"},"pg_stats":[{"pgid":"1.0","version":"211'348","reported_seq":1215,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.126614+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:12:39.126614+0000","last_peered":"2024-04-02T09:12:39.126614+0000","last_clean":"2024-04-02T09:12:39.126614+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:12:39.126614+0000","last_undegraded":"2024-04-02T09:12:39.126614+0000","last_fullsized":"2024-04-02T09:12:39.126614+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1b","version":"222'1","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117125+0000","last_change":"2024-04-02T09:12:24.687221+0000","last_active":"2024-04-02T09:12:39.117125+0000","last_peered":"2024-04-02T09:12:39.117125+0000","last_clean":"2024-04-02T09:12:39.117125+0000","last_became_active":"2024-04-02T09:12:24.686953+0000","last_became_peered":"2024-04-02T09:12:24.686953+0000","last_unstale":"2024-04-02T09:12:39.117125+0000","last_undegraded":"2024-04-02T09:12:39.117125+0000","last_fullsized":"2024-04-02T09:12:39.117125+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:39:31.379977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.115858+0000","last_change":"2024-04-02T09:12:25.190434+0000","last_active":"2024-04-02T09:12:39.115858+0000","last_peered":"2024-04-02T09:12:39.115858+0000","last_clean":"2024-04-02T09:12:39.115858+0000","last_became_active":"2024-04-02T09:12:25.190187+0000","last_became_peered":"2024-04-02T09:12:25.190187+0000","last_unstale":"2024-04-02T09:12:39.115858+0000","last_undegraded":"2024-04-02T09:12:39.115858+0000","last_fullsized":"2024-04-02T09:12:39.115858+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:01.850942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1d","version":"226'2","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347881+0000","last_change":"2024-04-02T09:12:38.637685+0000","last_active":"2024-04-02T09:12:40.347881+0000","last_peered":"2024-04-02T09:12:40.347881+0000","last_clean":"2024-04-02T09:12:40.347881+0000","last_became_active":"2024-04-02T09:12:38.636649+0000","last_became_peered":"2024-04-02T09:12:38.636649+0000","last_unstale":"2024-04-02T09:12:40.347881+0000","last_undegraded":"2024-04-02T09:12:40.347881+0000","last_fullsized":"2024-04-02T09:12:40.347881+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:39:07.848314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347543+0000","last_change":"2024-04-02T09:12:39.135559+0000","last_active":"2024-04-02T09:12:40.347543+0000","last_peered":"2024-04-02T09:12:40.347543+0000","last_clean":"2024-04-02T09:12:40.347543+0000","last_became_active":"2024-04-02T09:12:39.135093+0000","last_became_peered":"2024-04-02T09:12:39.135093+0000","last_unstale":"2024-04-02T09:12:40.347543+0000","last_undegraded":"2024-04-02T09:12:40.347543+0000","last_fullsized":"2024-04-02T09:12:40.347543+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:39.344420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.126704+0000","last_change":"2024-04-02T09:12:24.698442+0000","last_active":"2024-04-02T09:12:39.126704+0000","last_peered":"2024-04-02T09:12:39.126704+0000","last_clean":"2024-04-02T09:12:39.126704+0000","last_became_active":"2024-04-02T09:12:24.698179+0000","last_became_peered":"2024-04-02T09:12:24.698179+0000","last_unstale":"2024-04-02T09:12:39.126704+0000","last_undegraded":"2024-04-02T09:12:39.126704+0000","last_fullsized":"2024-04-02T09:12:39.126704+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:45:48.278840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.126608+0000","last_change":"2024-04-02T09:12:25.203302+0000","last_active":"2024-04-02T09:12:39.126608+0000","last_peered":"2024-04-02T09:12:39.126608+0000","last_clean":"2024-04-02T09:12:39.126608+0000","last_became_active":"2024-04-02T09:12:25.202579+0000","last_became_peered":"2024-04-02T09:12:25.202579+0000","last_unstale":"2024-04-02T09:12:39.126608+0000","last_undegraded":"2024-04-02T09:12:39.126608+0000","last_fullsized":"2024-04-02T09:12:39.126608+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:25.603157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.6","version":"226'1","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348863+0000","last_change":"2024-04-02T09:12:38.637937+0000","last_active":"2024-04-02T09:12:40.348863+0000","last_peered":"2024-04-02T09:12:40.348863+0000","last_clean":"2024-04-02T09:12:40.348863+0000","last_became_active":"2024-04-02T09:12:38.636887+0000","last_became_peered":"2024-04-02T09:12:38.636887+0000","last_unstale":"2024-04-02T09:12:40.348863+0000","last_undegraded":"2024-04-02T09:12:40.348863+0000","last_fullsized":"2024-04-02T09:12:40.348863+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:19:29.015299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349429+0000","last_change":"2024-04-02T09:12:39.140604+0000","last_active":"2024-04-02T09:12:40.349429+0000","last_peered":"2024-04-02T09:12:40.349429+0000","last_clean":"2024-04-02T09:12:40.349429+0000","last_became_active":"2024-04-02T09:12:39.138536+0000","last_became_peered":"2024-04-02T09:12:39.138536+0000","last_unstale":"2024-04-02T09:12:40.349429+0000","last_undegraded":"2024-04-02T09:12:40.349429+0000","last_fullsized":"2024-04-02T09:12:40.349429+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:56:30.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116587+0000","last_change":"2024-04-02T09:12:24.695544+0000","last_active":"2024-04-02T09:12:39.116587+0000","last_peered":"2024-04-02T09:12:39.116587+0000","last_clean":"2024-04-02T09:12:39.116587+0000","last_became_active":"2024-04-02T09:12:24.695032+0000","last_became_peered":"2024-04-02T09:12:24.695032+0000","last_unstale":"2024-04-02T09:12:39.116587+0000","last_undegraded":"2024-04-02T09:12:39.116587+0000","last_fullsized":"2024-04-02T09:12:39.116587+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:41.705096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.120572+0000","last_change":"2024-04-02T09:12:25.203957+0000","last_active":"2024-04-02T09:12:39.120572+0000","last_peered":"2024-04-02T09:12:39.120572+0000","last_clean":"2024-04-02T09:12:39.120572+0000","last_became_active":"2024-04-02T09:12:25.203680+0000","last_became_peered":"2024-04-02T09:12:25.203680+0000","last_unstale":"2024-04-02T09:12:39.120572+0000","last_undegraded":"2024-04-02T09:12:39.120572+0000","last_fullsized":"2024-04-02T09:12:39.120572+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:31.267333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.7","version":"226'2","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349033+0000","last_change":"2024-04-02T09:12:38.633818+0000","last_active":"2024-04-02T09:12:40.349033+0000","last_peered":"2024-04-02T09:12:40.349033+0000","last_clean":"2024-04-02T09:12:40.349033+0000","last_became_active":"2024-04-02T09:12:38.633565+0000","last_became_peered":"2024-04-02T09:12:38.633565+0000","last_unstale":"2024-04-02T09:12:40.349033+0000","last_undegraded":"2024-04-02T09:12:40.349033+0000","last_fullsized":"2024-04-02T09:12:40.349033+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:21.508468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349105+0000","last_change":"2024-04-02T09:12:39.128790+0000","last_active":"2024-04-02T09:12:40.349105+0000","last_peered":"2024-04-02T09:12:40.349105+0000","last_clean":"2024-04-02T09:12:40.349105+0000","last_became_active":"2024-04-02T09:12:39.127115+0000","last_became_peered":"2024-04-02T09:12:39.127115+0000","last_unstale":"2024-04-02T09:12:40.349105+0000","last_undegraded":"2024-04-02T09:12:40.349105+0000","last_fullsized":"2024-04-02T09:12:40.349105+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:02.402506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116024+0000","last_change":"2024-04-02T09:12:24.694385+0000","last_active":"2024-04-02T09:12:39.116024+0000","last_peered":"2024-04-02T09:12:39.116024+0000","last_clean":"2024-04-02T09:12:39.116024+0000","last_became_active":"2024-04-02T09:12:24.693609+0000","last_became_peered":"2024-04-02T09:12:24.693609+0000","last_unstale":"2024-04-02T09:12:39.116024+0000","last_undegraded":"2024-04-02T09:12:39.116024+0000","last_fullsized":"2024-04-02T09:12:39.116024+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:46:08.099991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117041+0000","last_change":"2024-04-02T09:12:25.197899+0000","last_active":"2024-04-02T09:12:39.117041+0000","last_peered":"2024-04-02T09:12:39.117041+0000","last_clean":"2024-04-02T09:12:39.117041+0000","last_became_active":"2024-04-02T09:12:25.197631+0000","last_became_peered":"2024-04-02T09:12:25.197631+0000","last_unstale":"2024-04-02T09:12:39.117041+0000","last_undegraded":"2024-04-02T09:12:39.117041+0000","last_fullsized":"2024-04-02T09:12:39.117041+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:02.349107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112833+0000","last_change":"2024-04-02T09:12:38.629932+0000","last_active":"2024-04-02T09:12:40.112833+0000","last_peered":"2024-04-02T09:12:40.112833+0000","last_clean":"2024-04-02T09:12:40.112833+0000","last_became_active":"2024-04-02T09:12:38.629265+0000","last_became_peered":"2024-04-02T09:12:38.629265+0000","last_unstale":"2024-04-02T09:12:40.112833+0000","last_undegraded":"2024-04-02T09:12:40.112833+0000","last_fullsized":"2024-04-02T09:12:40.112833+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:16:41.585294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348316+0000","last_change":"2024-04-02T09:12:39.137028+0000","last_active":"2024-04-02T09:12:40.348316+0000","last_peered":"2024-04-02T09:12:40.348316+0000","last_clean":"2024-04-02T09:12:40.348316+0000","last_became_active":"2024-04-02T09:12:39.135462+0000","last_became_peered":"2024-04-02T09:12:39.135462+0000","last_unstale":"2024-04-02T09:12:40.348316+0000","last_undegraded":"2024-04-02T09:12:40.348316+0000","last_fullsized":"2024-04-02T09:12:40.348316+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:01:25.642123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.134179+0000","last_change":"2024-04-02T09:12:24.693539+0000","last_active":"2024-04-02T09:12:39.134179+0000","last_peered":"2024-04-02T09:12:39.134179+0000","last_clean":"2024-04-02T09:12:39.134179+0000","last_became_active":"2024-04-02T09:12:24.693275+0000","last_became_peered":"2024-04-02T09:12:24.693275+0000","last_unstale":"2024-04-02T09:12:39.134179+0000","last_undegraded":"2024-04-02T09:12:39.134179+0000","last_fullsized":"2024-04-02T09:12:39.134179+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:22:34.495765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114271+0000","last_change":"2024-04-02T09:12:25.194789+0000","last_active":"2024-04-02T09:12:39.114271+0000","last_peered":"2024-04-02T09:12:39.114271+0000","last_clean":"2024-04-02T09:12:39.114271+0000","last_became_active":"2024-04-02T09:12:25.192429+0000","last_became_peered":"2024-04-02T09:12:25.192429+0000","last_unstale":"2024-04-02T09:12:39.114271+0000","last_undegraded":"2024-04-02T09:12:39.114271+0000","last_fullsized":"2024-04-02T09:12:39.114271+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:11:59.558881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347920+0000","last_change":"2024-04-02T09:12:38.624346+0000","last_active":"2024-04-02T09:12:40.347920+0000","last_peered":"2024-04-02T09:12:40.347920+0000","last_clean":"2024-04-02T09:12:40.347920+0000","last_became_active":"2024-04-02T09:12:38.624090+0000","last_became_peered":"2024-04-02T09:12:38.624090+0000","last_unstale":"2024-04-02T09:12:40.347920+0000","last_undegraded":"2024-04-02T09:12:40.347920+0000","last_fullsized":"2024-04-02T09:12:40.347920+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:20:18.506907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349418+0000","last_change":"2024-04-02T09:12:39.143742+0000","last_active":"2024-04-02T09:12:40.349418+0000","last_peered":"2024-04-02T09:12:40.349418+0000","last_clean":"2024-04-02T09:12:40.349418+0000","last_became_active":"2024-04-02T09:12:39.143456+0000","last_became_peered":"2024-04-02T09:12:39.143456+0000","last_unstale":"2024-04-02T09:12:40.349418+0000","last_undegraded":"2024-04-02T09:12:40.349418+0000","last_fullsized":"2024-04-02T09:12:40.349418+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:33.582844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.113955+0000","last_change":"2024-04-02T09:12:24.693871+0000","last_active":"2024-04-02T09:12:39.113955+0000","last_peered":"2024-04-02T09:12:39.113955+0000","last_clean":"2024-04-02T09:12:39.113955+0000","last_became_active":"2024-04-02T09:12:24.693562+0000","last_became_peered":"2024-04-02T09:12:24.693562+0000","last_unstale":"2024-04-02T09:12:39.113955+0000","last_undegraded":"2024-04-02T09:12:39.113955+0000","last_fullsized":"2024-04-02T09:12:39.113955+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:48:24.676039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.119409+0000","last_change":"2024-04-02T09:12:25.196931+0000","last_active":"2024-04-02T09:12:39.119409+0000","last_peered":"2024-04-02T09:12:39.119409+0000","last_clean":"2024-04-02T09:12:39.119409+0000","last_became_active":"2024-04-02T09:12:25.194947+0000","last_became_peered":"2024-04-02T09:12:25.194947+0000","last_unstale":"2024-04-02T09:12:39.119409+0000","last_undegraded":"2024-04-02T09:12:39.119409+0000","last_fullsized":"2024-04-02T09:12:39.119409+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:21.964569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116880+0000","last_change":"2024-04-02T09:12:38.633018+0000","last_active":"2024-04-02T09:12:40.116880+0000","last_peered":"2024-04-02T09:12:40.116880+0000","last_clean":"2024-04-02T09:12:40.116880+0000","last_became_active":"2024-04-02T09:12:38.632717+0000","last_became_peered":"2024-04-02T09:12:38.632717+0000","last_unstale":"2024-04-02T09:12:40.116880+0000","last_undegraded":"2024-04-02T09:12:40.116880+0000","last_fullsized":"2024-04-02T09:12:40.116880+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:00:35.621780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349580+0000","last_change":"2024-04-02T09:12:39.143785+0000","last_active":"2024-04-02T09:12:40.349580+0000","last_peered":"2024-04-02T09:12:40.349580+0000","last_clean":"2024-04-02T09:12:40.349580+0000","last_became_active":"2024-04-02T09:12:39.142457+0000","last_became_peered":"2024-04-02T09:12:39.142457+0000","last_unstale":"2024-04-02T09:12:40.349580+0000","last_undegraded":"2024-04-02T09:12:40.349580+0000","last_fullsized":"2024-04-02T09:12:40.349580+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:35:02.087473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114480+0000","last_change":"2024-04-02T09:12:24.691094+0000","last_active":"2024-04-02T09:12:39.114480+0000","last_peered":"2024-04-02T09:12:39.114480+0000","last_clean":"2024-04-02T09:12:39.114480+0000","last_became_active":"2024-04-02T09:12:24.690782+0000","last_became_peered":"2024-04-02T09:12:24.690782+0000","last_unstale":"2024-04-02T09:12:39.114480+0000","last_undegraded":"2024-04-02T09:12:39.114480+0000","last_fullsized":"2024-04-02T09:12:39.114480+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:17.789082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116605+0000","last_change":"2024-04-02T09:12:25.200491+0000","last_active":"2024-04-02T09:12:39.116605+0000","last_peered":"2024-04-02T09:12:39.116605+0000","last_clean":"2024-04-02T09:12:39.116605+0000","last_became_active":"2024-04-02T09:12:25.200208+0000","last_became_peered":"2024-04-02T09:12:25.200208+0000","last_unstale":"2024-04-02T09:12:39.116605+0000","last_undegraded":"2024-04-02T09:12:39.116605+0000","last_fullsized":"2024-04-02T09:12:39.116605+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:53:44.514037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112619+0000","last_change":"2024-04-02T09:12:38.632816+0000","last_active":"2024-04-02T09:12:40.112619+0000","last_peered":"2024-04-02T09:12:40.112619+0000","last_clean":"2024-04-02T09:12:40.112619+0000","last_became_active":"2024-04-02T09:12:38.631996+0000","last_became_peered":"2024-04-02T09:12:38.631996+0000","last_unstale":"2024-04-02T09:12:40.112619+0000","last_undegraded":"2024-04-02T09:12:40.112619+0000","last_fullsized":"2024-04-02T09:12:40.112619+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:49:38.141564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349247+0000","last_change":"2024-04-02T09:12:39.141524+0000","last_active":"2024-04-02T09:12:40.349247+0000","last_peered":"2024-04-02T09:12:40.349247+0000","last_clean":"2024-04-02T09:12:40.349247+0000","last_became_active":"2024-04-02T09:12:39.140362+0000","last_became_peered":"2024-04-02T09:12:39.140362+0000","last_unstale":"2024-04-02T09:12:40.349247+0000","last_undegraded":"2024-04-02T09:12:40.349247+0000","last_fullsized":"2024-04-02T09:12:40.349247+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:48:50.055962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"223'1","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117479+0000","last_change":"2024-04-02T09:12:24.686970+0000","last_active":"2024-04-02T09:12:39.117479+0000","last_peered":"2024-04-02T09:12:39.117479+0000","last_clean":"2024-04-02T09:12:39.117479+0000","last_became_active":"2024-04-02T09:12:24.686686+0000","last_became_peered":"2024-04-02T09:12:24.686686+0000","last_unstale":"2024-04-02T09:12:39.117479+0000","last_undegraded":"2024-04-02T09:12:39.117479+0000","last_fullsized":"2024-04-02T09:12:39.117479+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:17.201623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116745+0000","last_change":"2024-04-02T09:12:25.196988+0000","last_active":"2024-04-02T09:12:39.116745+0000","last_peered":"2024-04-02T09:12:39.116745+0000","last_clean":"2024-04-02T09:12:39.116745+0000","last_became_active":"2024-04-02T09:12:25.196031+0000","last_became_peered":"2024-04-02T09:12:25.196031+0000","last_unstale":"2024-04-02T09:12:39.116745+0000","last_undegraded":"2024-04-02T09:12:39.116745+0000","last_fullsized":"2024-04-02T09:12:39.116745+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:50.975576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348967+0000","last_change":"2024-04-02T09:12:38.633373+0000","last_active":"2024-04-02T09:12:40.348967+0000","last_peered":"2024-04-02T09:12:40.348967+0000","last_clean":"2024-04-02T09:12:40.348967+0000","last_became_active":"2024-04-02T09:12:38.633108+0000","last_became_peered":"2024-04-02T09:12:38.633108+0000","last_unstale":"2024-04-02T09:12:40.348967+0000","last_undegraded":"2024-04-02T09:12:40.348967+0000","last_fullsized":"2024-04-02T09:12:40.348967+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:06.959393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348106+0000","last_change":"2024-04-02T09:12:39.126921+0000","last_active":"2024-04-02T09:12:40.348106+0000","last_peered":"2024-04-02T09:12:40.348106+0000","last_clean":"2024-04-02T09:12:40.348106+0000","last_became_active":"2024-04-02T09:12:39.126650+0000","last_became_peered":"2024-04-02T09:12:39.126650+0000","last_unstale":"2024-04-02T09:12:40.348106+0000","last_undegraded":"2024-04-02T09:12:40.348106+0000","last_fullsized":"2024-04-02T09:12:40.348106+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:11.640312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.7","version":"222'2","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.123499+0000","last_change":"2024-04-02T09:12:24.696113+0000","last_active":"2024-04-02T09:12:39.123499+0000","last_peered":"2024-04-02T09:12:39.123499+0000","last_clean":"2024-04-02T09:12:39.123499+0000","last_became_active":"2024-04-02T09:12:24.695191+0000","last_became_peered":"2024-04-02T09:12:24.695191+0000","last_unstale":"2024-04-02T09:12:39.123499+0000","last_undegraded":"2024-04-02T09:12:39.123499+0000","last_fullsized":"2024-04-02T09:12:39.123499+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:41:21.090616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117147+0000","last_change":"2024-04-02T09:12:25.196365+0000","last_active":"2024-04-02T09:12:39.117147+0000","last_peered":"2024-04-02T09:12:39.117147+0000","last_clean":"2024-04-02T09:12:39.117147+0000","last_became_active":"2024-04-02T09:12:25.191868+0000","last_became_peered":"2024-04-02T09:12:25.191868+0000","last_unstale":"2024-04-02T09:12:39.117147+0000","last_undegraded":"2024-04-02T09:12:39.117147+0000","last_fullsized":"2024-04-02T09:12:39.117147+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:43:29.229527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117688+0000","last_change":"2024-04-02T09:12:38.628570+0000","last_active":"2024-04-02T09:12:40.117688+0000","last_peered":"2024-04-02T09:12:40.117688+0000","last_clean":"2024-04-02T09:12:40.117688+0000","last_became_active":"2024-04-02T09:12:38.628252+0000","last_became_peered":"2024-04-02T09:12:38.628252+0000","last_unstale":"2024-04-02T09:12:40.117688+0000","last_undegraded":"2024-04-02T09:12:40.117688+0000","last_fullsized":"2024-04-02T09:12:40.117688+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:49:16.327081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112331+0000","last_change":"2024-04-02T09:12:39.144893+0000","last_active":"2024-04-02T09:12:40.112331+0000","last_peered":"2024-04-02T09:12:40.112331+0000","last_clean":"2024-04-02T09:12:40.112331+0000","last_became_active":"2024-04-02T09:12:39.144615+0000","last_became_peered":"2024-04-02T09:12:39.144615+0000","last_unstale":"2024-04-02T09:12:40.112331+0000","last_undegraded":"2024-04-02T09:12:40.112331+0000","last_fullsized":"2024-04-02T09:12:40.112331+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:01.838261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.120667+0000","last_change":"2024-04-02T09:12:24.683516+0000","last_active":"2024-04-02T09:12:39.120667+0000","last_peered":"2024-04-02T09:12:39.120667+0000","last_clean":"2024-04-02T09:12:39.120667+0000","last_became_active":"2024-04-02T09:12:24.683207+0000","last_became_peered":"2024-04-02T09:12:24.683207+0000","last_unstale":"2024-04-02T09:12:39.120667+0000","last_undegraded":"2024-04-02T09:12:39.120667+0000","last_fullsized":"2024-04-02T09:12:39.120667+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:19:15.076490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.127052+0000","last_change":"2024-04-02T09:12:25.205206+0000","last_active":"2024-04-02T09:12:39.127052+0000","last_peered":"2024-04-02T09:12:39.127052+0000","last_clean":"2024-04-02T09:12:39.127052+0000","last_became_active":"2024-04-02T09:12:25.204526+0000","last_became_peered":"2024-04-02T09:12:25.204526+0000","last_unstale":"2024-04-02T09:12:39.127052+0000","last_undegraded":"2024-04-02T09:12:39.127052+0000","last_fullsized":"2024-04-02T09:12:39.127052+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:33:41.191493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.e","version":"226'2","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349262+0000","last_change":"2024-04-02T09:12:38.632806+0000","last_active":"2024-04-02T09:12:40.349262+0000","last_peered":"2024-04-02T09:12:40.349262+0000","last_clean":"2024-04-02T09:12:40.349262+0000","last_became_active":"2024-04-02T09:12:38.632489+0000","last_became_peered":"2024-04-02T09:12:38.632489+0000","last_unstale":"2024-04-02T09:12:40.349262+0000","last_undegraded":"2024-04-02T09:12:40.349262+0000","last_fullsized":"2024-04-02T09:12:40.349262+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:18:04.222535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349208+0000","last_change":"2024-04-02T09:12:39.123626+0000","last_active":"2024-04-02T09:12:40.349208+0000","last_peered":"2024-04-02T09:12:40.349208+0000","last_clean":"2024-04-02T09:12:40.349208+0000","last_became_active":"2024-04-02T09:12:39.123310+0000","last_became_peered":"2024-04-02T09:12:39.123310+0000","last_unstale":"2024-04-02T09:12:40.349208+0000","last_undegraded":"2024-04-02T09:12:40.349208+0000","last_fullsized":"2024-04-02T09:12:40.349208+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:18.395664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.119361+0000","last_change":"2024-04-02T09:12:24.699586+0000","last_active":"2024-04-02T09:12:39.119361+0000","last_peered":"2024-04-02T09:12:39.119361+0000","last_clean":"2024-04-02T09:12:39.119361+0000","last_became_active":"2024-04-02T09:12:24.697592+0000","last_became_peered":"2024-04-02T09:12:24.697592+0000","last_unstale":"2024-04-02T09:12:39.119361+0000","last_undegraded":"2024-04-02T09:12:39.119361+0000","last_fullsized":"2024-04-02T09:12:39.119361+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:22:53.870135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.126786+0000","last_change":"2024-04-02T09:12:25.189098+0000","last_active":"2024-04-02T09:12:39.126786+0000","last_peered":"2024-04-02T09:12:39.126786+0000","last_clean":"2024-04-02T09:12:39.126786+0000","last_became_active":"2024-04-02T09:12:25.187377+0000","last_became_peered":"2024-04-02T09:12:25.187377+0000","last_unstale":"2024-04-02T09:12:39.126786+0000","last_undegraded":"2024-04-02T09:12:39.126786+0000","last_fullsized":"2024-04-02T09:12:39.126786+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:58:52.752985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.f","version":"226'1","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349684+0000","last_change":"2024-04-02T09:12:38.638073+0000","last_active":"2024-04-02T09:12:40.349684+0000","last_peered":"2024-04-02T09:12:40.349684+0000","last_clean":"2024-04-02T09:12:40.349684+0000","last_became_active":"2024-04-02T09:12:38.637830+0000","last_became_peered":"2024-04-02T09:12:38.637830+0000","last_unstale":"2024-04-02T09:12:40.349684+0000","last_undegraded":"2024-04-02T09:12:40.349684+0000","last_fullsized":"2024-04-02T09:12:40.349684+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:11:20.511970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349415+0000","last_change":"2024-04-02T09:12:39.135926+0000","last_active":"2024-04-02T09:12:40.349415+0000","last_peered":"2024-04-02T09:12:40.349415+0000","last_clean":"2024-04-02T09:12:40.349415+0000","last_became_active":"2024-04-02T09:12:39.135625+0000","last_became_peered":"2024-04-02T09:12:39.135625+0000","last_unstale":"2024-04-02T09:12:40.349415+0000","last_undegraded":"2024-04-02T09:12:40.349415+0000","last_fullsized":"2024-04-02T09:12:40.349415+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:49:00.944821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.126297+0000","last_change":"2024-04-02T09:12:24.691812+0000","last_active":"2024-04-02T09:12:39.126297+0000","last_peered":"2024-04-02T09:12:39.126297+0000","last_clean":"2024-04-02T09:12:39.126297+0000","last_became_active":"2024-04-02T09:12:24.688935+0000","last_became_peered":"2024-04-02T09:12:24.688935+0000","last_unstale":"2024-04-02T09:12:39.126297+0000","last_undegraded":"2024-04-02T09:12:39.126297+0000","last_fullsized":"2024-04-02T09:12:39.126297+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:53.020585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.113633+0000","last_change":"2024-04-02T09:12:25.195866+0000","last_active":"2024-04-02T09:12:39.113633+0000","last_peered":"2024-04-02T09:12:39.113633+0000","last_clean":"2024-04-02T09:12:39.113633+0000","last_became_active":"2024-04-02T09:12:25.195430+0000","last_became_peered":"2024-04-02T09:12:25.195430+0000","last_unstale":"2024-04-02T09:12:39.113633+0000","last_undegraded":"2024-04-02T09:12:39.113633+0000","last_fullsized":"2024-04-02T09:12:39.113633+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:42:28.562906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348451+0000","last_change":"2024-04-02T09:12:38.638164+0000","last_active":"2024-04-02T09:12:40.348451+0000","last_peered":"2024-04-02T09:12:40.348451+0000","last_clean":"2024-04-02T09:12:40.348451+0000","last_became_active":"2024-04-02T09:12:38.637882+0000","last_became_peered":"2024-04-02T09:12:38.637882+0000","last_unstale":"2024-04-02T09:12:40.348451+0000","last_undegraded":"2024-04-02T09:12:40.348451+0000","last_fullsized":"2024-04-02T09:12:40.348451+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:34.894177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348111+0000","last_change":"2024-04-02T09:12:39.139332+0000","last_active":"2024-04-02T09:12:40.348111+0000","last_peered":"2024-04-02T09:12:40.348111+0000","last_clean":"2024-04-02T09:12:40.348111+0000","last_became_active":"2024-04-02T09:12:39.139084+0000","last_became_peered":"2024-04-02T09:12:39.139084+0000","last_unstale":"2024-04-02T09:12:40.348111+0000","last_undegraded":"2024-04-02T09:12:40.348111+0000","last_fullsized":"2024-04-02T09:12:40.348111+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:06:26.150667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"222'2","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.123006+0000","last_change":"2024-04-02T09:12:24.693402+0000","last_active":"2024-04-02T09:12:39.123006+0000","last_peered":"2024-04-02T09:12:39.123006+0000","last_clean":"2024-04-02T09:12:39.123006+0000","last_became_active":"2024-04-02T09:12:24.693047+0000","last_became_peered":"2024-04-02T09:12:24.693047+0000","last_unstale":"2024-04-02T09:12:39.123006+0000","last_undegraded":"2024-04-02T09:12:39.123006+0000","last_fullsized":"2024-04-02T09:12:39.123006+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:15:17.948098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.115937+0000","last_change":"2024-04-02T09:12:25.195964+0000","last_active":"2024-04-02T09:12:39.115937+0000","last_peered":"2024-04-02T09:12:39.115937+0000","last_clean":"2024-04-02T09:12:39.115937+0000","last_became_active":"2024-04-02T09:12:25.195716+0000","last_became_peered":"2024-04-02T09:12:25.195716+0000","last_unstale":"2024-04-02T09:12:39.115937+0000","last_undegraded":"2024-04-02T09:12:39.115937+0000","last_fullsized":"2024-04-02T09:12:39.115937+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:25.408279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.d","version":"226'1","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348038+0000","last_change":"2024-04-02T09:12:38.633567+0000","last_active":"2024-04-02T09:12:40.348038+0000","last_peered":"2024-04-02T09:12:40.348038+0000","last_clean":"2024-04-02T09:12:40.348038+0000","last_became_active":"2024-04-02T09:12:38.633209+0000","last_became_peered":"2024-04-02T09:12:38.633209+0000","last_unstale":"2024-04-02T09:12:40.348038+0000","last_undegraded":"2024-04-02T09:12:40.348038+0000","last_fullsized":"2024-04-02T09:12:40.348038+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:25:28.960133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112513+0000","last_change":"2024-04-02T09:12:39.124846+0000","last_active":"2024-04-02T09:12:40.112513+0000","last_peered":"2024-04-02T09:12:40.112513+0000","last_clean":"2024-04-02T09:12:40.112513+0000","last_became_active":"2024-04-02T09:12:39.124595+0000","last_became_peered":"2024-04-02T09:12:39.124595+0000","last_unstale":"2024-04-02T09:12:40.112513+0000","last_undegraded":"2024-04-02T09:12:40.112513+0000","last_fullsized":"2024-04-02T09:12:40.112513+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:58.288706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114578+0000","last_change":"2024-04-02T09:12:24.692444+0000","last_active":"2024-04-02T09:12:39.114578+0000","last_peered":"2024-04-02T09:12:39.114578+0000","last_clean":"2024-04-02T09:12:39.114578+0000","last_became_active":"2024-04-02T09:12:24.692165+0000","last_became_peered":"2024-04-02T09:12:24.692165+0000","last_unstale":"2024-04-02T09:12:39.114578+0000","last_undegraded":"2024-04-02T09:12:39.114578+0000","last_fullsized":"2024-04-02T09:12:39.114578+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:29.826236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114388+0000","last_change":"2024-04-02T09:12:25.190936+0000","last_active":"2024-04-02T09:12:39.114388+0000","last_peered":"2024-04-02T09:12:39.114388+0000","last_clean":"2024-04-02T09:12:39.114388+0000","last_became_active":"2024-04-02T09:12:25.189970+0000","last_became_peered":"2024-04-02T09:12:25.189970+0000","last_unstale":"2024-04-02T09:12:39.114388+0000","last_undegraded":"2024-04-02T09:12:39.114388+0000","last_fullsized":"2024-04-02T09:12:39.114388+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:57.372310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347519+0000","last_change":"2024-04-02T09:12:38.630574+0000","last_active":"2024-04-02T09:12:40.347519+0000","last_peered":"2024-04-02T09:12:40.347519+0000","last_clean":"2024-04-02T09:12:40.347519+0000","last_became_active":"2024-04-02T09:12:38.628873+0000","last_became_peered":"2024-04-02T09:12:38.628873+0000","last_unstale":"2024-04-02T09:12:40.347519+0000","last_undegraded":"2024-04-02T09:12:40.347519+0000","last_fullsized":"2024-04-02T09:12:40.347519+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:04.426115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348238+0000","last_change":"2024-04-02T09:12:39.134066+0000","last_active":"2024-04-02T09:12:40.348238+0000","last_peered":"2024-04-02T09:12:40.348238+0000","last_clean":"2024-04-02T09:12:40.348238+0000","last_became_active":"2024-04-02T09:12:39.131593+0000","last_became_peered":"2024-04-02T09:12:39.131593+0000","last_unstale":"2024-04-02T09:12:40.348238+0000","last_undegraded":"2024-04-02T09:12:40.348238+0000","last_fullsized":"2024-04-02T09:12:40.348238+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:36:45.203475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"222'1","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.123187+0000","last_change":"2024-04-02T09:12:24.688637+0000","last_active":"2024-04-02T09:12:39.123187+0000","last_peered":"2024-04-02T09:12:39.123187+0000","last_clean":"2024-04-02T09:12:39.123187+0000","last_became_active":"2024-04-02T09:12:24.688321+0000","last_became_peered":"2024-04-02T09:12:24.688321+0000","last_unstale":"2024-04-02T09:12:39.123187+0000","last_undegraded":"2024-04-02T09:12:39.123187+0000","last_fullsized":"2024-04-02T09:12:39.123187+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:41:28.231249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.118045+0000","last_change":"2024-04-02T09:12:25.193723+0000","last_active":"2024-04-02T09:12:39.118045+0000","last_peered":"2024-04-02T09:12:39.118045+0000","last_clean":"2024-04-02T09:12:39.118045+0000","last_became_active":"2024-04-02T09:12:25.193367+0000","last_became_peered":"2024-04-02T09:12:25.193367+0000","last_unstale":"2024-04-02T09:12:39.118045+0000","last_undegraded":"2024-04-02T09:12:39.118045+0000","last_fullsized":"2024-04-02T09:12:39.118045+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:48.218780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.b","version":"226'2","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349531+0000","last_change":"2024-04-02T09:12:38.636198+0000","last_active":"2024-04-02T09:12:40.349531+0000","last_peered":"2024-04-02T09:12:40.349531+0000","last_clean":"2024-04-02T09:12:40.349531+0000","last_became_active":"2024-04-02T09:12:38.633531+0000","last_became_peered":"2024-04-02T09:12:38.633531+0000","last_unstale":"2024-04-02T09:12:40.349531+0000","last_undegraded":"2024-04-02T09:12:40.349531+0000","last_fullsized":"2024-04-02T09:12:40.349531+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:06:25.144133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348478+0000","last_change":"2024-04-02T09:12:39.124009+0000","last_active":"2024-04-02T09:12:40.348478+0000","last_peered":"2024-04-02T09:12:40.348478+0000","last_clean":"2024-04-02T09:12:40.348478+0000","last_became_active":"2024-04-02T09:12:39.123484+0000","last_became_peered":"2024-04-02T09:12:39.123484+0000","last_unstale":"2024-04-02T09:12:40.348478+0000","last_undegraded":"2024-04-02T09:12:40.348478+0000","last_fullsized":"2024-04-02T09:12:40.348478+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:24:10.675482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"222'2","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.118077+0000","last_change":"2024-04-02T09:12:24.690213+0000","last_active":"2024-04-02T09:12:39.118077+0000","last_peered":"2024-04-02T09:12:39.118077+0000","last_clean":"2024-04-02T09:12:39.118077+0000","last_became_active":"2024-04-02T09:12:24.689813+0000","last_became_peered":"2024-04-02T09:12:24.689813+0000","last_unstale":"2024-04-02T09:12:39.118077+0000","last_undegraded":"2024-04-02T09:12:39.118077+0000","last_fullsized":"2024-04-02T09:12:39.118077+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:11.366124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.130606+0000","last_change":"2024-04-02T09:12:25.196053+0000","last_active":"2024-04-02T09:12:39.130606+0000","last_peered":"2024-04-02T09:12:39.130606+0000","last_clean":"2024-04-02T09:12:39.130606+0000","last_became_active":"2024-04-02T09:12:25.195799+0000","last_became_peered":"2024-04-02T09:12:25.195799+0000","last_unstale":"2024-04-02T09:12:39.130606+0000","last_undegraded":"2024-04-02T09:12:39.130606+0000","last_fullsized":"2024-04-02T09:12:39.130606+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:26:33.117159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112459+0000","last_change":"2024-04-02T09:12:38.638977+0000","last_active":"2024-04-02T09:12:40.112459+0000","last_peered":"2024-04-02T09:12:40.112459+0000","last_clean":"2024-04-02T09:12:40.112459+0000","last_became_active":"2024-04-02T09:12:38.638666+0000","last_became_peered":"2024-04-02T09:12:38.638666+0000","last_unstale":"2024-04-02T09:12:40.112459+0000","last_undegraded":"2024-04-02T09:12:40.112459+0000","last_fullsized":"2024-04-02T09:12:40.112459+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:40:47.207533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117654+0000","last_change":"2024-04-02T09:12:39.142960+0000","last_active":"2024-04-02T09:12:40.117654+0000","last_peered":"2024-04-02T09:12:40.117654+0000","last_clean":"2024-04-02T09:12:40.117654+0000","last_became_active":"2024-04-02T09:12:39.142500+0000","last_became_peered":"2024-04-02T09:12:39.142500+0000","last_unstale":"2024-04-02T09:12:40.117654+0000","last_undegraded":"2024-04-02T09:12:40.117654+0000","last_fullsized":"2024-04-02T09:12:40.117654+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:25:29.966667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"222'1","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116664+0000","last_change":"2024-04-02T09:12:24.695567+0000","last_active":"2024-04-02T09:12:39.116664+0000","last_peered":"2024-04-02T09:12:39.116664+0000","last_clean":"2024-04-02T09:12:39.116664+0000","last_became_active":"2024-04-02T09:12:24.695267+0000","last_became_peered":"2024-04-02T09:12:24.695267+0000","last_unstale":"2024-04-02T09:12:39.116664+0000","last_undegraded":"2024-04-02T09:12:39.116664+0000","last_fullsized":"2024-04-02T09:12:39.116664+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:02.378346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117284+0000","last_change":"2024-04-02T09:12:25.196196+0000","last_active":"2024-04-02T09:12:39.117284+0000","last_peered":"2024-04-02T09:12:39.117284+0000","last_clean":"2024-04-02T09:12:39.117284+0000","last_became_active":"2024-04-02T09:12:25.195967+0000","last_became_peered":"2024-04-02T09:12:25.195967+0000","last_unstale":"2024-04-02T09:12:39.117284+0000","last_undegraded":"2024-04-02T09:12:39.117284+0000","last_fullsized":"2024-04-02T09:12:39.117284+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:41.683998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112356+0000","last_change":"2024-04-02T09:12:38.633999+0000","last_active":"2024-04-02T09:12:40.112356+0000","last_peered":"2024-04-02T09:12:40.112356+0000","last_clean":"2024-04-02T09:12:40.112356+0000","last_became_active":"2024-04-02T09:12:38.632478+0000","last_became_peered":"2024-04-02T09:12:38.632478+0000","last_unstale":"2024-04-02T09:12:40.112356+0000","last_undegraded":"2024-04-02T09:12:40.112356+0000","last_fullsized":"2024-04-02T09:12:40.112356+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:18:04.222535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347683+0000","last_change":"2024-04-02T09:12:39.128396+0000","last_active":"2024-04-02T09:12:40.347683+0000","last_peered":"2024-04-02T09:12:40.347683+0000","last_clean":"2024-04-02T09:12:40.347683+0000","last_became_active":"2024-04-02T09:12:39.128118+0000","last_became_peered":"2024-04-02T09:12:39.128118+0000","last_unstale":"2024-04-02T09:12:40.347683+0000","last_undegraded":"2024-04-02T09:12:40.347683+0000","last_fullsized":"2024-04-02T09:12:40.347683+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:39:46.325320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"222'8","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.120472+0000","last_change":"2024-04-02T09:12:24.692205+0000","last_active":"2024-04-02T09:12:39.120472+0000","last_peered":"2024-04-02T09:12:39.120472+0000","last_clean":"2024-04-02T09:12:39.120472+0000","last_became_active":"2024-04-02T09:12:24.690462+0000","last_became_peered":"2024-04-02T09:12:24.690462+0000","last_unstale":"2024-04-02T09:12:39.120472+0000","last_undegraded":"2024-04-02T09:12:39.120472+0000","last_fullsized":"2024-04-02T09:12:39.120472+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:00:14.796518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.115999+0000","last_change":"2024-04-02T09:12:25.201746+0000","last_active":"2024-04-02T09:12:39.115999+0000","last_peered":"2024-04-02T09:12:39.115999+0000","last_clean":"2024-04-02T09:12:39.115999+0000","last_became_active":"2024-04-02T09:12:25.201445+0000","last_became_peered":"2024-04-02T09:12:25.201445+0000","last_unstale":"2024-04-02T09:12:39.115999+0000","last_undegraded":"2024-04-02T09:12:39.115999+0000","last_fullsized":"2024-04-02T09:12:39.115999+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:06:08.648823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349299+0000","last_change":"2024-04-02T09:12:38.633797+0000","last_active":"2024-04-02T09:12:40.349299+0000","last_peered":"2024-04-02T09:12:40.349299+0000","last_clean":"2024-04-02T09:12:40.349299+0000","last_became_active":"2024-04-02T09:12:38.632294+0000","last_became_peered":"2024-04-02T09:12:38.632294+0000","last_unstale":"2024-04-02T09:12:40.349299+0000","last_undegraded":"2024-04-02T09:12:40.349299+0000","last_fullsized":"2024-04-02T09:12:40.349299+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:43:42.159134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.113378+0000","last_change":"2024-04-02T09:12:39.132856+0000","last_active":"2024-04-02T09:12:40.113378+0000","last_peered":"2024-04-02T09:12:40.113378+0000","last_clean":"2024-04-02T09:12:40.113378+0000","last_became_active":"2024-04-02T09:12:39.128056+0000","last_became_peered":"2024-04-02T09:12:39.128056+0000","last_unstale":"2024-04-02T09:12:40.113378+0000","last_undegraded":"2024-04-02T09:12:40.113378+0000","last_fullsized":"2024-04-02T09:12:40.113378+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:05.432649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.123091+0000","last_change":"2024-04-02T09:12:24.691834+0000","last_active":"2024-04-02T09:12:39.123091+0000","last_peered":"2024-04-02T09:12:39.123091+0000","last_clean":"2024-04-02T09:12:39.123091+0000","last_became_active":"2024-04-02T09:12:24.689829+0000","last_became_peered":"2024-04-02T09:12:24.689829+0000","last_unstale":"2024-04-02T09:12:39.123091+0000","last_undegraded":"2024-04-02T09:12:39.123091+0000","last_fullsized":"2024-04-02T09:12:39.123091+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:56.694970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114297+0000","last_change":"2024-04-02T09:12:25.203805+0000","last_active":"2024-04-02T09:12:39.114297+0000","last_peered":"2024-04-02T09:12:39.114297+0000","last_clean":"2024-04-02T09:12:39.114297+0000","last_became_active":"2024-04-02T09:12:25.203516+0000","last_became_peered":"2024-04-02T09:12:25.203516+0000","last_unstale":"2024-04-02T09:12:39.114297+0000","last_undegraded":"2024-04-02T09:12:39.114297+0000","last_fullsized":"2024-04-02T09:12:39.114297+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:00:15.805719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348958+0000","last_change":"2024-04-02T09:12:38.638310+0000","last_active":"2024-04-02T09:12:40.348958+0000","last_peered":"2024-04-02T09:12:40.348958+0000","last_clean":"2024-04-02T09:12:40.348958+0000","last_became_active":"2024-04-02T09:12:38.638018+0000","last_became_peered":"2024-04-02T09:12:38.638018+0000","last_unstale":"2024-04-02T09:12:40.348958+0000","last_undegraded":"2024-04-02T09:12:40.348958+0000","last_fullsized":"2024-04-02T09:12:40.348958+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:38.337886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348836+0000","last_change":"2024-04-02T09:12:39.142582+0000","last_active":"2024-04-02T09:12:40.348836+0000","last_peered":"2024-04-02T09:12:40.348836+0000","last_clean":"2024-04-02T09:12:40.348836+0000","last_became_active":"2024-04-02T09:12:39.142245+0000","last_became_peered":"2024-04-02T09:12:39.142245+0000","last_unstale":"2024-04-02T09:12:40.348836+0000","last_undegraded":"2024-04-02T09:12:40.348836+0000","last_fullsized":"2024-04-02T09:12:40.348836+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:45:28.462172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116430+0000","last_change":"2024-04-02T09:12:24.695778+0000","last_active":"2024-04-02T09:12:39.116430+0000","last_peered":"2024-04-02T09:12:39.116430+0000","last_clean":"2024-04-02T09:12:39.116430+0000","last_became_active":"2024-04-02T09:12:24.695349+0000","last_became_peered":"2024-04-02T09:12:24.695349+0000","last_unstale":"2024-04-02T09:12:39.116430+0000","last_undegraded":"2024-04-02T09:12:39.116430+0000","last_fullsized":"2024-04-02T09:12:39.116430+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:33:51.021792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.121582+0000","last_change":"2024-04-02T09:12:25.205479+0000","last_active":"2024-04-02T09:12:39.121582+0000","last_peered":"2024-04-02T09:12:39.121582+0000","last_clean":"2024-04-02T09:12:39.121582+0000","last_became_active":"2024-04-02T09:12:25.205210+0000","last_became_peered":"2024-04-02T09:12:25.205210+0000","last_unstale":"2024-04-02T09:12:39.121582+0000","last_undegraded":"2024-04-02T09:12:39.121582+0000","last_fullsized":"2024-04-02T09:12:39.121582+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:03.985819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"226'6","reported_seq":17,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348721+0000","last_change":"2024-04-02T09:12:38.633159+0000","last_active":"2024-04-02T09:12:40.348721+0000","last_peered":"2024-04-02T09:12:40.348721+0000","last_clean":"2024-04-02T09:12:40.348721+0000","last_became_active":"2024-04-02T09:12:38.632508+0000","last_became_peered":"2024-04-02T09:12:38.632508+0000","last_unstale":"2024-04-02T09:12:40.348721+0000","last_undegraded":"2024-04-02T09:12:40.348721+0000","last_fullsized":"2024-04-02T09:12:40.348721+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:49:38.141564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348595+0000","last_change":"2024-04-02T09:12:39.142408+0000","last_active":"2024-04-02T09:12:40.348595+0000","last_peered":"2024-04-02T09:12:40.348595+0000","last_clean":"2024-04-02T09:12:40.348595+0000","last_became_active":"2024-04-02T09:12:39.142140+0000","last_became_peered":"2024-04-02T09:12:39.142140+0000","last_unstale":"2024-04-02T09:12:40.348595+0000","last_undegraded":"2024-04-02T09:12:40.348595+0000","last_fullsized":"2024-04-02T09:12:40.348595+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:42:49.185298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.13","version":"222'2","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116884+0000","last_change":"2024-04-02T09:12:24.694118+0000","last_active":"2024-04-02T09:12:39.116884+0000","last_peered":"2024-04-02T09:12:39.116884+0000","last_clean":"2024-04-02T09:12:39.116884+0000","last_became_active":"2024-04-02T09:12:24.693829+0000","last_became_peered":"2024-04-02T09:12:24.693829+0000","last_unstale":"2024-04-02T09:12:39.116884+0000","last_undegraded":"2024-04-02T09:12:39.116884+0000","last_fullsized":"2024-04-02T09:12:39.116884+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:58:51.743784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.122178+0000","last_change":"2024-04-02T09:12:25.192240+0000","last_active":"2024-04-02T09:12:39.122178+0000","last_peered":"2024-04-02T09:12:39.122178+0000","last_clean":"2024-04-02T09:12:39.122178+0000","last_became_active":"2024-04-02T09:12:25.191954+0000","last_became_peered":"2024-04-02T09:12:25.191954+0000","last_unstale":"2024-04-02T09:12:39.122178+0000","last_undegraded":"2024-04-02T09:12:39.122178+0000","last_fullsized":"2024-04-02T09:12:39.122178+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:26:33.117159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.15","version":"226'3","reported_seq":14,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.118052+0000","last_change":"2024-04-02T09:12:38.626779+0000","last_active":"2024-04-02T09:12:40.118052+0000","last_peered":"2024-04-02T09:12:40.118052+0000","last_clean":"2024-04-02T09:12:40.118052+0000","last_became_active":"2024-04-02T09:12:38.626438+0000","last_became_peered":"2024-04-02T09:12:38.626438+0000","last_unstale":"2024-04-02T09:12:40.118052+0000","last_undegraded":"2024-04-02T09:12:40.118052+0000","last_fullsized":"2024-04-02T09:12:40.118052+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:20:18.506907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349069+0000","last_change":"2024-04-02T09:12:39.121376+0000","last_active":"2024-04-02T09:12:40.349069+0000","last_peered":"2024-04-02T09:12:40.349069+0000","last_clean":"2024-04-02T09:12:40.349069+0000","last_became_active":"2024-04-02T09:12:39.121100+0000","last_became_peered":"2024-04-02T09:12:39.121100+0000","last_unstale":"2024-04-02T09:12:40.349069+0000","last_undegraded":"2024-04-02T09:12:40.349069+0000","last_fullsized":"2024-04-02T09:12:40.349069+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:01.838261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.14","version":"223'8","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.118056+0000","last_change":"2024-04-02T09:12:24.689123+0000","last_active":"2024-04-02T09:12:39.118056+0000","last_peered":"2024-04-02T09:12:39.118056+0000","last_clean":"2024-04-02T09:12:39.118056+0000","last_became_active":"2024-04-02T09:12:24.688848+0000","last_became_peered":"2024-04-02T09:12:24.688848+0000","last_unstale":"2024-04-02T09:12:39.118056+0000","last_undegraded":"2024-04-02T09:12:39.118056+0000","last_fullsized":"2024-04-02T09:12:39.118056+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:24.593956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2805,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.122691+0000","last_change":"2024-04-02T09:12:25.199901+0000","last_active":"2024-04-02T09:12:39.122691+0000","last_peered":"2024-04-02T09:12:39.122691+0000","last_clean":"2024-04-02T09:12:39.122691+0000","last_became_active":"2024-04-02T09:12:25.198989+0000","last_became_peered":"2024-04-02T09:12:25.198989+0000","last_unstale":"2024-04-02T09:12:39.122691+0000","last_undegraded":"2024-04-02T09:12:39.122691+0000","last_fullsized":"2024-04-02T09:12:39.122691+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:30.835437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116796+0000","last_change":"2024-04-02T09:12:38.634647+0000","last_active":"2024-04-02T09:12:40.116796+0000","last_peered":"2024-04-02T09:12:40.116796+0000","last_clean":"2024-04-02T09:12:40.116796+0000","last_became_active":"2024-04-02T09:12:38.634254+0000","last_became_peered":"2024-04-02T09:12:38.634254+0000","last_unstale":"2024-04-02T09:12:40.116796+0000","last_undegraded":"2024-04-02T09:12:40.116796+0000","last_fullsized":"2024-04-02T09:12:40.116796+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:51:50.442138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348361+0000","last_change":"2024-04-02T09:12:39.132916+0000","last_active":"2024-04-02T09:12:40.348361+0000","last_peered":"2024-04-02T09:12:40.348361+0000","last_clean":"2024-04-02T09:12:40.348361+0000","last_became_active":"2024-04-02T09:12:39.132638+0000","last_became_peered":"2024-04-02T09:12:39.132638+0000","last_unstale":"2024-04-02T09:12:40.348361+0000","last_undegraded":"2024-04-02T09:12:40.348361+0000","last_fullsized":"2024-04-02T09:12:40.348361+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:37:05.275434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"222'3","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.122883+0000","last_change":"2024-04-02T09:12:24.694400+0000","last_active":"2024-04-02T09:12:39.122883+0000","last_peered":"2024-04-02T09:12:39.122883+0000","last_clean":"2024-04-02T09:12:39.122883+0000","last_became_active":"2024-04-02T09:12:24.694118+0000","last_became_peered":"2024-04-02T09:12:24.694118+0000","last_unstale":"2024-04-02T09:12:39.122883+0000","last_undegraded":"2024-04-02T09:12:39.122883+0000","last_fullsized":"2024-04-02T09:12:39.122883+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:00.841741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117972+0000","last_change":"2024-04-02T09:12:25.202617+0000","last_active":"2024-04-02T09:12:39.117972+0000","last_peered":"2024-04-02T09:12:39.117972+0000","last_clean":"2024-04-02T09:12:39.117972+0000","last_became_active":"2024-04-02T09:12:25.201786+0000","last_became_peered":"2024-04-02T09:12:25.201786+0000","last_unstale":"2024-04-02T09:12:39.117972+0000","last_undegraded":"2024-04-02T09:12:39.117972+0000","last_fullsized":"2024-04-02T09:12:39.117972+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:32:03.502011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.13","version":"226'2","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117980+0000","last_change":"2024-04-02T09:12:38.626907+0000","last_active":"2024-04-02T09:12:40.117980+0000","last_peered":"2024-04-02T09:12:40.117980+0000","last_clean":"2024-04-02T09:12:40.117980+0000","last_became_active":"2024-04-02T09:12:38.626639+0000","last_became_peered":"2024-04-02T09:12:38.626639+0000","last_unstale":"2024-04-02T09:12:40.117980+0000","last_undegraded":"2024-04-02T09:12:40.117980+0000","last_fullsized":"2024-04-02T09:12:40.117980+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:09:35.892912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116986+0000","last_change":"2024-04-02T09:12:39.128580+0000","last_active":"2024-04-02T09:12:40.116986+0000","last_peered":"2024-04-02T09:12:40.116986+0000","last_clean":"2024-04-02T09:12:40.116986+0000","last_became_active":"2024-04-02T09:12:39.128324+0000","last_became_peered":"2024-04-02T09:12:39.128324+0000","last_unstale":"2024-04-02T09:12:40.116986+0000","last_undegraded":"2024-04-02T09:12:40.116986+0000","last_fullsized":"2024-04-02T09:12:40.116986+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:16:42.591828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.128748+0000","last_change":"2024-04-02T09:12:24.690012+0000","last_active":"2024-04-02T09:12:39.128748+0000","last_peered":"2024-04-02T09:12:39.128748+0000","last_clean":"2024-04-02T09:12:39.128748+0000","last_became_active":"2024-04-02T09:12:24.689551+0000","last_became_peered":"2024-04-02T09:12:24.689551+0000","last_unstale":"2024-04-02T09:12:39.128748+0000","last_undegraded":"2024-04-02T09:12:39.128748+0000","last_fullsized":"2024-04-02T09:12:39.128748+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:59:15.316332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.126471+0000","last_change":"2024-04-02T09:12:25.198058+0000","last_active":"2024-04-02T09:12:39.126471+0000","last_peered":"2024-04-02T09:12:39.126471+0000","last_clean":"2024-04-02T09:12:39.126471+0000","last_became_active":"2024-04-02T09:12:25.197791+0000","last_became_peered":"2024-04-02T09:12:25.197791+0000","last_unstale":"2024-04-02T09:12:39.126471+0000","last_undegraded":"2024-04-02T09:12:39.126471+0000","last_fullsized":"2024-04-02T09:12:39.126471+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:38.903726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.10","version":"226'8","reported_seq":19,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112332+0000","last_change":"2024-04-02T09:12:38.629329+0000","last_active":"2024-04-02T09:12:40.112332+0000","last_peered":"2024-04-02T09:12:40.112332+0000","last_clean":"2024-04-02T09:12:40.112332+0000","last_became_active":"2024-04-02T09:12:38.628342+0000","last_became_peered":"2024-04-02T09:12:38.628342+0000","last_unstale":"2024-04-02T09:12:40.112332+0000","last_undegraded":"2024-04-02T09:12:40.112332+0000","last_fullsized":"2024-04-02T09:12:40.112332+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:21.508468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112653+0000","last_change":"2024-04-02T09:12:39.126411+0000","last_active":"2024-04-02T09:12:40.112653+0000","last_peered":"2024-04-02T09:12:40.112653+0000","last_clean":"2024-04-02T09:12:40.112653+0000","last_became_active":"2024-04-02T09:12:39.126179+0000","last_became_peered":"2024-04-02T09:12:39.126179+0000","last_unstale":"2024-04-02T09:12:40.112653+0000","last_undegraded":"2024-04-02T09:12:40.112653+0000","last_fullsized":"2024-04-02T09:12:40.112653+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:00.936946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.113926+0000","last_change":"2024-04-02T09:12:24.697017+0000","last_active":"2024-04-02T09:12:39.113926+0000","last_peered":"2024-04-02T09:12:39.113926+0000","last_clean":"2024-04-02T09:12:39.113926+0000","last_became_active":"2024-04-02T09:12:24.696537+0000","last_became_peered":"2024-04-02T09:12:24.696537+0000","last_unstale":"2024-04-02T09:12:39.113926+0000","last_undegraded":"2024-04-02T09:12:39.113926+0000","last_fullsized":"2024-04-02T09:12:39.113926+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:16.861790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.128655+0000","last_change":"2024-04-02T09:12:25.192446+0000","last_active":"2024-04-02T09:12:39.128655+0000","last_peered":"2024-04-02T09:12:39.128655+0000","last_clean":"2024-04-02T09:12:39.128655+0000","last_became_active":"2024-04-02T09:12:25.192159+0000","last_became_peered":"2024-04-02T09:12:25.192159+0000","last_unstale":"2024-04-02T09:12:39.128655+0000","last_undegraded":"2024-04-02T09:12:39.128655+0000","last_fullsized":"2024-04-02T09:12:39.128655+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:38.903726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117486+0000","last_change":"2024-04-02T09:12:38.634219+0000","last_active":"2024-04-02T09:12:40.117486+0000","last_peered":"2024-04-02T09:12:40.117486+0000","last_clean":"2024-04-02T09:12:40.117486+0000","last_became_active":"2024-04-02T09:12:38.633910+0000","last_became_peered":"2024-04-02T09:12:38.633910+0000","last_unstale":"2024-04-02T09:12:40.117486+0000","last_undegraded":"2024-04-02T09:12:40.117486+0000","last_fullsized":"2024-04-02T09:12:40.117486+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:08:49.341798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347060+0000","last_change":"2024-04-02T09:12:39.125324+0000","last_active":"2024-04-02T09:12:40.347060+0000","last_peered":"2024-04-02T09:12:40.347060+0000","last_clean":"2024-04-02T09:12:40.347060+0000","last_became_active":"2024-04-02T09:12:39.124764+0000","last_became_peered":"2024-04-02T09:12:39.124764+0000","last_unstale":"2024-04-02T09:12:40.347060+0000","last_undegraded":"2024-04-02T09:12:40.347060+0000","last_fullsized":"2024-04-02T09:12:40.347060+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:48:31.765033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.126879+0000","last_change":"2024-04-02T09:12:24.698699+0000","last_active":"2024-04-02T09:12:39.126879+0000","last_peered":"2024-04-02T09:12:39.126879+0000","last_clean":"2024-04-02T09:12:39.126879+0000","last_became_active":"2024-04-02T09:12:24.698347+0000","last_became_peered":"2024-04-02T09:12:24.698347+0000","last_unstale":"2024-04-02T09:12:39.126879+0000","last_undegraded":"2024-04-02T09:12:39.126879+0000","last_fullsized":"2024-04-02T09:12:39.126879+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:44:22.620155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116766+0000","last_change":"2024-04-02T09:12:25.198098+0000","last_active":"2024-04-02T09:12:39.116766+0000","last_peered":"2024-04-02T09:12:39.116766+0000","last_clean":"2024-04-02T09:12:39.116766+0000","last_became_active":"2024-04-02T09:12:25.197848+0000","last_became_peered":"2024-04-02T09:12:25.197848+0000","last_unstale":"2024-04-02T09:12:39.116766+0000","last_undegraded":"2024-04-02T09:12:39.116766+0000","last_fullsized":"2024-04-02T09:12:39.116766+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:02.893062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"226'1","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348766+0000","last_change":"2024-04-02T09:12:38.635301+0000","last_active":"2024-04-02T09:12:40.348766+0000","last_peered":"2024-04-02T09:12:40.348766+0000","last_clean":"2024-04-02T09:12:40.348766+0000","last_became_active":"2024-04-02T09:12:38.634575+0000","last_became_peered":"2024-04-02T09:12:38.634575+0000","last_unstale":"2024-04-02T09:12:40.348766+0000","last_undegraded":"2024-04-02T09:12:40.348766+0000","last_fullsized":"2024-04-02T09:12:40.348766+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:20:17.549794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.118489+0000","last_change":"2024-04-02T09:12:39.141529+0000","last_active":"2024-04-02T09:12:40.118489+0000","last_peered":"2024-04-02T09:12:40.118489+0000","last_clean":"2024-04-02T09:12:40.118489+0000","last_became_active":"2024-04-02T09:12:39.141238+0000","last_became_peered":"2024-04-02T09:12:39.141238+0000","last_unstale":"2024-04-02T09:12:40.118489+0000","last_undegraded":"2024-04-02T09:12:40.118489+0000","last_fullsized":"2024-04-02T09:12:40.118489+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:18:05.229069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.123191+0000","last_change":"2024-04-02T09:12:24.698972+0000","last_active":"2024-04-02T09:12:39.123191+0000","last_peered":"2024-04-02T09:12:39.123191+0000","last_clean":"2024-04-02T09:12:39.123191+0000","last_became_active":"2024-04-02T09:12:24.698664+0000","last_became_peered":"2024-04-02T09:12:24.698664+0000","last_unstale":"2024-04-02T09:12:39.123191+0000","last_undegraded":"2024-04-02T09:12:39.123191+0000","last_fullsized":"2024-04-02T09:12:39.123191+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:11:06.573162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.113545+0000","last_change":"2024-04-02T09:12:25.191145+0000","last_active":"2024-04-02T09:12:39.113545+0000","last_peered":"2024-04-02T09:12:39.113545+0000","last_clean":"2024-04-02T09:12:39.113545+0000","last_became_active":"2024-04-02T09:12:25.190868+0000","last_became_peered":"2024-04-02T09:12:25.190868+0000","last_unstale":"2024-04-02T09:12:39.113545+0000","last_undegraded":"2024-04-02T09:12:39.113545+0000","last_fullsized":"2024-04-02T09:12:39.113545+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:41.683998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1f","version":"226'4","reported_seq":15,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.118385+0000","last_change":"2024-04-02T09:12:38.640727+0000","last_active":"2024-04-02T09:12:40.118385+0000","last_peered":"2024-04-02T09:12:40.118385+0000","last_clean":"2024-04-02T09:12:40.118385+0000","last_became_active":"2024-04-02T09:12:38.640469+0000","last_became_peered":"2024-04-02T09:12:38.640469+0000","last_unstale":"2024-04-02T09:12:40.118385+0000","last_undegraded":"2024-04-02T09:12:40.118385+0000","last_fullsized":"2024-04-02T09:12:40.118385+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:59:12.891582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.113379+0000","last_change":"2024-04-02T09:12:39.144611+0000","last_active":"2024-04-02T09:12:40.113379+0000","last_peered":"2024-04-02T09:12:40.113379+0000","last_clean":"2024-04-02T09:12:40.113379+0000","last_became_active":"2024-04-02T09:12:39.144284+0000","last_became_peered":"2024-04-02T09:12:39.144284+0000","last_unstale":"2024-04-02T09:12:40.113379+0000","last_undegraded":"2024-04-02T09:12:40.113379+0000","last_fullsized":"2024-04-02T09:12:40.113379+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:22:50.985250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"222'1","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114379+0000","last_change":"2024-04-02T09:12:24.690596+0000","last_active":"2024-04-02T09:12:39.114379+0000","last_peered":"2024-04-02T09:12:39.114379+0000","last_clean":"2024-04-02T09:12:39.114379+0000","last_became_active":"2024-04-02T09:12:24.690029+0000","last_became_peered":"2024-04-02T09:12:24.690029+0000","last_unstale":"2024-04-02T09:12:39.114379+0000","last_undegraded":"2024-04-02T09:12:39.114379+0000","last_fullsized":"2024-04-02T09:12:39.114379+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:00.608890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.134279+0000","last_change":"2024-04-02T09:12:25.192982+0000","last_active":"2024-04-02T09:12:39.134279+0000","last_peered":"2024-04-02T09:12:39.134279+0000","last_clean":"2024-04-02T09:12:39.134279+0000","last_became_active":"2024-04-02T09:12:25.192737+0000","last_became_peered":"2024-04-02T09:12:25.192737+0000","last_unstale":"2024-04-02T09:12:39.134279+0000","last_undegraded":"2024-04-02T09:12:39.134279+0000","last_fullsized":"2024-04-02T09:12:39.134279+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:28.343401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1c","version":"226'2","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348484+0000","last_change":"2024-04-02T09:12:38.630289+0000","last_active":"2024-04-02T09:12:40.348484+0000","last_peered":"2024-04-02T09:12:40.348484+0000","last_clean":"2024-04-02T09:12:40.348484+0000","last_became_active":"2024-04-02T09:12:38.628695+0000","last_became_peered":"2024-04-02T09:12:38.628695+0000","last_unstale":"2024-04-02T09:12:40.348484+0000","last_undegraded":"2024-04-02T09:12:40.348484+0000","last_fullsized":"2024-04-02T09:12:40.348484+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:08:59.930412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348040+0000","last_change":"2024-04-02T09:12:39.138377+0000","last_active":"2024-04-02T09:12:40.348040+0000","last_peered":"2024-04-02T09:12:40.348040+0000","last_clean":"2024-04-02T09:12:40.348040+0000","last_became_active":"2024-04-02T09:12:39.138107+0000","last_became_peered":"2024-04-02T09:12:39.138107+0000","last_unstale":"2024-04-02T09:12:40.348040+0000","last_undegraded":"2024-04-02T09:12:40.348040+0000","last_fullsized":"2024-04-02T09:12:40.348040+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:32.734424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1c","version":"222'2","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.133207+0000","last_change":"2024-04-02T09:12:24.692574+0000","last_active":"2024-04-02T09:12:39.133207+0000","last_peered":"2024-04-02T09:12:39.133207+0000","last_clean":"2024-04-02T09:12:39.133207+0000","last_became_active":"2024-04-02T09:12:24.692283+0000","last_became_peered":"2024-04-02T09:12:24.692283+0000","last_unstale":"2024-04-02T09:12:39.133207+0000","last_undegraded":"2024-04-02T09:12:39.133207+0000","last_fullsized":"2024-04-02T09:12:39.133207+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:56:59.029298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.113910+0000","last_change":"2024-04-02T09:12:25.194989+0000","last_active":"2024-04-02T09:12:39.113910+0000","last_peered":"2024-04-02T09:12:39.113910+0000","last_clean":"2024-04-02T09:12:39.113910+0000","last_became_active":"2024-04-02T09:12:25.194466+0000","last_became_peered":"2024-04-02T09:12:25.194466+0000","last_unstale":"2024-04-02T09:12:39.113910+0000","last_undegraded":"2024-04-02T09:12:39.113910+0000","last_fullsized":"2024-04-02T09:12:39.113910+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:01.618091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1a","version":"226'1","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348635+0000","last_change":"2024-04-02T09:12:38.635903+0000","last_active":"2024-04-02T09:12:40.348635+0000","last_peered":"2024-04-02T09:12:40.348635+0000","last_clean":"2024-04-02T09:12:40.348635+0000","last_became_active":"2024-04-02T09:12:38.635616+0000","last_became_peered":"2024-04-02T09:12:38.635616+0000","last_unstale":"2024-04-02T09:12:40.348635+0000","last_undegraded":"2024-04-02T09:12:40.348635+0000","last_fullsized":"2024-04-02T09:12:40.348635+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:10.633778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116558+0000","last_change":"2024-04-02T09:12:39.147583+0000","last_active":"2024-04-02T09:12:40.116558+0000","last_peered":"2024-04-02T09:12:40.116558+0000","last_clean":"2024-04-02T09:12:40.116558+0000","last_became_active":"2024-04-02T09:12:39.147229+0000","last_became_peered":"2024-04-02T09:12:39.147229+0000","last_unstale":"2024-04-02T09:12:40.116558+0000","last_undegraded":"2024-04-02T09:12:40.116558+0000","last_fullsized":"2024-04-02T09:12:40.116558+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:53:58.450179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"222'2","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.122782+0000","last_change":"2024-04-02T09:12:24.691669+0000","last_active":"2024-04-02T09:12:39.122782+0000","last_peered":"2024-04-02T09:12:39.122782+0000","last_clean":"2024-04-02T09:12:39.122782+0000","last_became_active":"2024-04-02T09:12:24.689803+0000","last_became_peered":"2024-04-02T09:12:24.689803+0000","last_unstale":"2024-04-02T09:12:39.122782+0000","last_undegraded":"2024-04-02T09:12:39.122782+0000","last_fullsized":"2024-04-02T09:12:39.122782+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:25:15.021325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117957+0000","last_change":"2024-04-02T09:12:25.204722+0000","last_active":"2024-04-02T09:12:39.117957+0000","last_peered":"2024-04-02T09:12:39.117957+0000","last_clean":"2024-04-02T09:12:39.117957+0000","last_became_active":"2024-04-02T09:12:25.204031+0000","last_became_peered":"2024-04-02T09:12:25.204031+0000","last_unstale":"2024-04-02T09:12:39.117957+0000","last_undegraded":"2024-04-02T09:12:39.117957+0000","last_fullsized":"2024-04-02T09:12:39.117957+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:00:19.545624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1b","version":"226'1","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347797+0000","last_change":"2024-04-02T09:12:38.631122+0000","last_active":"2024-04-02T09:12:40.347797+0000","last_peered":"2024-04-02T09:12:40.347797+0000","last_clean":"2024-04-02T09:12:40.347797+0000","last_became_active":"2024-04-02T09:12:38.630207+0000","last_became_peered":"2024-04-02T09:12:38.630207+0000","last_unstale":"2024-04-02T09:12:40.347797+0000","last_undegraded":"2024-04-02T09:12:40.347797+0000","last_fullsized":"2024-04-02T09:12:40.347797+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:44:41.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112664+0000","last_change":"2024-04-02T09:12:39.143199+0000","last_active":"2024-04-02T09:12:40.112664+0000","last_peered":"2024-04-02T09:12:40.112664+0000","last_clean":"2024-04-02T09:12:40.112664+0000","last_became_active":"2024-04-02T09:12:39.142929+0000","last_became_peered":"2024-04-02T09:12:39.142929+0000","last_unstale":"2024-04-02T09:12:40.112664+0000","last_undegraded":"2024-04-02T09:12:40.112664+0000","last_fullsized":"2024-04-02T09:12:40.112664+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:11:52.229332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1e","version":"222'1","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117390+0000","last_change":"2024-04-02T09:12:24.684617+0000","last_active":"2024-04-02T09:12:39.117390+0000","last_peered":"2024-04-02T09:12:39.117390+0000","last_clean":"2024-04-02T09:12:39.117390+0000","last_became_active":"2024-04-02T09:12:24.684367+0000","last_became_peered":"2024-04-02T09:12:24.684367+0000","last_unstale":"2024-04-02T09:12:39.117390+0000","last_undegraded":"2024-04-02T09:12:39.117390+0000","last_fullsized":"2024-04-02T09:12:39.117390+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:48:45.999478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.130741+0000","last_change":"2024-04-02T09:12:25.190209+0000","last_active":"2024-04-02T09:12:39.130741+0000","last_peered":"2024-04-02T09:12:39.130741+0000","last_clean":"2024-04-02T09:12:39.130741+0000","last_became_active":"2024-04-02T09:12:25.189908+0000","last_became_peered":"2024-04-02T09:12:25.189908+0000","last_unstale":"2024-04-02T09:12:39.130741+0000","last_undegraded":"2024-04-02T09:12:39.130741+0000","last_fullsized":"2024-04-02T09:12:39.130741+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:54:12.375324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347333+0000","last_change":"2024-04-02T09:12:38.623441+0000","last_active":"2024-04-02T09:12:40.347333+0000","last_peered":"2024-04-02T09:12:40.347333+0000","last_clean":"2024-04-02T09:12:40.347333+0000","last_became_active":"2024-04-02T09:12:38.622492+0000","last_became_peered":"2024-04-02T09:12:38.622492+0000","last_unstale":"2024-04-02T09:12:40.347333+0000","last_undegraded":"2024-04-02T09:12:40.347333+0000","last_fullsized":"2024-04-02T09:12:40.347333+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:16:41.585294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349891+0000","last_change":"2024-04-02T09:12:39.141266+0000","last_active":"2024-04-02T09:12:40.349891+0000","last_peered":"2024-04-02T09:12:40.349891+0000","last_clean":"2024-04-02T09:12:40.349891+0000","last_became_active":"2024-04-02T09:12:39.140845+0000","last_became_peered":"2024-04-02T09:12:39.140845+0000","last_unstale":"2024-04-02T09:12:40.349891+0000","last_undegraded":"2024-04-02T09:12:40.349891+0000","last_fullsized":"2024-04-02T09:12:40.349891+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:22:56.853967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1f","version":"222'4","reported_seq":19,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.121616+0000","last_change":"2024-04-02T09:12:24.695565+0000","last_active":"2024-04-02T09:12:39.121616+0000","last_peered":"2024-04-02T09:12:39.121616+0000","last_clean":"2024-04-02T09:12:39.121616+0000","last_became_active":"2024-04-02T09:12:24.695304+0000","last_became_peered":"2024-04-02T09:12:24.695304+0000","last_unstale":"2024-04-02T09:12:39.121616+0000","last_undegraded":"2024-04-02T09:12:39.121616+0000","last_fullsized":"2024-04-02T09:12:39.121616+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:23.659795+0000","last_clean_scrub_stamp":"2024-04-02T09:12:23.659795+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:56:16.018943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117066+0000","last_change":"2024-04-02T09:12:25.195780+0000","last_active":"2024-04-02T09:12:39.117066+0000","last_peered":"2024-04-02T09:12:39.117066+0000","last_clean":"2024-04-02T09:12:39.117066+0000","last_became_active":"2024-04-02T09:12:25.194939+0000","last_became_peered":"2024-04-02T09:12:25.194939+0000","last_unstale":"2024-04-02T09:12:39.117066+0000","last_undegraded":"2024-04-02T09:12:39.117066+0000","last_fullsized":"2024-04-02T09:12:39.117066+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:20:21.844925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348155+0000","last_change":"2024-04-02T09:12:38.625812+0000","last_active":"2024-04-02T09:12:40.348155+0000","last_peered":"2024-04-02T09:12:40.348155+0000","last_clean":"2024-04-02T09:12:40.348155+0000","last_became_active":"2024-04-02T09:12:38.625536+0000","last_became_peered":"2024-04-02T09:12:38.625536+0000","last_unstale":"2024-04-02T09:12:40.348155+0000","last_undegraded":"2024-04-02T09:12:40.348155+0000","last_fullsized":"2024-04-02T09:12:40.348155+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:37.598604+0000","last_clean_scrub_stamp":"2024-04-02T09:12:37.598604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:49:16.327081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116523+0000","last_change":"2024-04-02T09:12:39.143363+0000","last_active":"2024-04-02T09:12:40.116523+0000","last_peered":"2024-04-02T09:12:40.116523+0000","last_clean":"2024-04-02T09:12:40.116523+0000","last_became_active":"2024-04-02T09:12:39.142967+0000","last_became_peered":"2024-04-02T09:12:39.142967+0000","last_unstale":"2024-04-02T09:12:40.116523+0000","last_undegraded":"2024-04-02T09:12:40.116523+0000","last_fullsized":"2024-04-02T09:12:40.116523+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:18:22.515002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.118239+0000","last_change":"2024-04-02T09:12:39.130951+0000","last_active":"2024-04-02T09:12:40.118239+0000","last_peered":"2024-04-02T09:12:40.118239+0000","last_clean":"2024-04-02T09:12:40.118239+0000","last_became_active":"2024-04-02T09:12:39.130211+0000","last_became_peered":"2024-04-02T09:12:39.130211+0000","last_unstale":"2024-04-02T09:12:40.118239+0000","last_undegraded":"2024-04-02T09:12:40.118239+0000","last_fullsized":"2024-04-02T09:12:40.118239+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:41.395363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.122770+0000","last_change":"2024-04-02T09:12:25.193316+0000","last_active":"2024-04-02T09:12:39.122770+0000","last_peered":"2024-04-02T09:12:39.122770+0000","last_clean":"2024-04-02T09:12:39.122770+0000","last_became_active":"2024-04-02T09:12:25.192672+0000","last_became_peered":"2024-04-02T09:12:25.192672+0000","last_unstale":"2024-04-02T09:12:39.122770+0000","last_undegraded":"2024-04-02T09:12:39.122770+0000","last_fullsized":"2024-04-02T09:12:39.122770+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:28.343401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.113030+0000","last_change":"2024-04-02T09:12:39.132655+0000","last_active":"2024-04-02T09:12:40.113030+0000","last_peered":"2024-04-02T09:12:40.113030+0000","last_clean":"2024-04-02T09:12:40.113030+0000","last_became_active":"2024-04-02T09:12:39.128795+0000","last_became_peered":"2024-04-02T09:12:39.128795+0000","last_unstale":"2024-04-02T09:12:40.113030+0000","last_undegraded":"2024-04-02T09:12:40.113030+0000","last_fullsized":"2024-04-02T09:12:40.113030+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:13:15.590165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.119524+0000","last_change":"2024-04-02T09:12:25.203525+0000","last_active":"2024-04-02T09:12:39.119524+0000","last_peered":"2024-04-02T09:12:39.119524+0000","last_clean":"2024-04-02T09:12:39.119524+0000","last_became_active":"2024-04-02T09:12:25.202409+0000","last_became_peered":"2024-04-02T09:12:25.202409+0000","last_unstale":"2024-04-02T09:12:39.119524+0000","last_undegraded":"2024-04-02T09:12:39.119524+0000","last_fullsized":"2024-04-02T09:12:39.119524+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:38:54.918706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.124181+0000","last_change":"2024-04-02T09:12:25.194990+0000","last_active":"2024-04-02T09:12:39.124181+0000","last_peered":"2024-04-02T09:12:39.124181+0000","last_clean":"2024-04-02T09:12:39.124181+0000","last_became_active":"2024-04-02T09:12:25.191623+0000","last_became_peered":"2024-04-02T09:12:25.191623+0000","last_unstale":"2024-04-02T09:12:39.124181+0000","last_undegraded":"2024-04-02T09:12:39.124181+0000","last_fullsized":"2024-04-02T09:12:39.124181+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:42:10.151562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117908+0000","last_change":"2024-04-02T09:12:39.133147+0000","last_active":"2024-04-02T09:12:40.117908+0000","last_peered":"2024-04-02T09:12:40.117908+0000","last_clean":"2024-04-02T09:12:40.117908+0000","last_became_active":"2024-04-02T09:12:39.132883+0000","last_became_peered":"2024-04-02T09:12:39.132883+0000","last_unstale":"2024-04-02T09:12:40.117908+0000","last_undegraded":"2024-04-02T09:12:40.117908+0000","last_fullsized":"2024-04-02T09:12:40.117908+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:19:30.021833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.113219+0000","last_change":"2024-04-02T09:12:39.130628+0000","last_active":"2024-04-02T09:12:40.113219+0000","last_peered":"2024-04-02T09:12:40.113219+0000","last_clean":"2024-04-02T09:12:40.113219+0000","last_became_active":"2024-04-02T09:12:39.130023+0000","last_became_peered":"2024-04-02T09:12:39.130023+0000","last_unstale":"2024-04-02T09:12:40.113219+0000","last_undegraded":"2024-04-02T09:12:40.113219+0000","last_fullsized":"2024-04-02T09:12:40.113219+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:28.062436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116957+0000","last_change":"2024-04-02T09:12:25.205454+0000","last_active":"2024-04-02T09:12:39.116957+0000","last_peered":"2024-04-02T09:12:39.116957+0000","last_clean":"2024-04-02T09:12:39.116957+0000","last_became_active":"2024-04-02T09:12:25.204157+0000","last_became_peered":"2024-04-02T09:12:25.204157+0000","last_unstale":"2024-04-02T09:12:39.116957+0000","last_undegraded":"2024-04-02T09:12:39.116957+0000","last_fullsized":"2024-04-02T09:12:39.116957+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:46:50.298119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.124619+0000","last_change":"2024-04-02T09:12:25.192359+0000","last_active":"2024-04-02T09:12:39.124619+0000","last_peered":"2024-04-02T09:12:39.124619+0000","last_clean":"2024-04-02T09:12:39.124619+0000","last_became_active":"2024-04-02T09:12:25.192092+0000","last_became_peered":"2024-04-02T09:12:25.192092+0000","last_unstale":"2024-04-02T09:12:39.124619+0000","last_undegraded":"2024-04-02T09:12:39.124619+0000","last_fullsized":"2024-04-02T09:12:39.124619+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:01.618091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347858+0000","last_change":"2024-04-02T09:12:39.143757+0000","last_active":"2024-04-02T09:12:40.347858+0000","last_peered":"2024-04-02T09:12:40.347858+0000","last_clean":"2024-04-02T09:12:40.347858+0000","last_became_active":"2024-04-02T09:12:39.142148+0000","last_became_peered":"2024-04-02T09:12:39.142148+0000","last_unstale":"2024-04-02T09:12:40.347858+0000","last_undegraded":"2024-04-02T09:12:40.347858+0000","last_fullsized":"2024-04-02T09:12:40.347858+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:03:55.514882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.119332+0000","last_change":"2024-04-02T09:12:25.196804+0000","last_active":"2024-04-02T09:12:39.119332+0000","last_peered":"2024-04-02T09:12:39.119332+0000","last_clean":"2024-04-02T09:12:39.119332+0000","last_became_active":"2024-04-02T09:12:25.194411+0000","last_became_peered":"2024-04-02T09:12:25.194411+0000","last_unstale":"2024-04-02T09:12:39.119332+0000","last_undegraded":"2024-04-02T09:12:39.119332+0000","last_fullsized":"2024-04-02T09:12:39.119332+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:45:21.225286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348334+0000","last_change":"2024-04-02T09:12:39.125335+0000","last_active":"2024-04-02T09:12:40.348334+0000","last_peered":"2024-04-02T09:12:40.348334+0000","last_clean":"2024-04-02T09:12:40.348334+0000","last_became_active":"2024-04-02T09:12:39.124780+0000","last_became_peered":"2024-04-02T09:12:39.124780+0000","last_unstale":"2024-04-02T09:12:40.348334+0000","last_undegraded":"2024-04-02T09:12:40.348334+0000","last_fullsized":"2024-04-02T09:12:40.348334+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:51:15.554233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.123254+0000","last_change":"2024-04-02T09:12:25.200885+0000","last_active":"2024-04-02T09:12:39.123254+0000","last_peered":"2024-04-02T09:12:39.123254+0000","last_clean":"2024-04-02T09:12:39.123254+0000","last_became_active":"2024-04-02T09:12:25.200630+0000","last_became_peered":"2024-04-02T09:12:25.200630+0000","last_unstale":"2024-04-02T09:12:39.123254+0000","last_undegraded":"2024-04-02T09:12:39.123254+0000","last_fullsized":"2024-04-02T09:12:39.123254+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:18.798282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117294+0000","last_change":"2024-04-02T09:12:39.142959+0000","last_active":"2024-04-02T09:12:40.117294+0000","last_peered":"2024-04-02T09:12:40.117294+0000","last_clean":"2024-04-02T09:12:40.117294+0000","last_became_active":"2024-04-02T09:12:39.142426+0000","last_became_peered":"2024-04-02T09:12:39.142426+0000","last_unstale":"2024-04-02T09:12:40.117294+0000","last_undegraded":"2024-04-02T09:12:40.117294+0000","last_fullsized":"2024-04-02T09:12:40.117294+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:07.965927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.130830+0000","last_change":"2024-04-02T09:12:25.195404+0000","last_active":"2024-04-02T09:12:39.130830+0000","last_peered":"2024-04-02T09:12:39.130830+0000","last_clean":"2024-04-02T09:12:39.130830+0000","last_became_active":"2024-04-02T09:12:25.192051+0000","last_became_peered":"2024-04-02T09:12:25.192051+0000","last_unstale":"2024-04-02T09:12:39.130830+0000","last_undegraded":"2024-04-02T09:12:39.130830+0000","last_fullsized":"2024-04-02T09:12:39.130830+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:02.893062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347686+0000","last_change":"2024-04-02T09:12:39.133572+0000","last_active":"2024-04-02T09:12:40.347686+0000","last_peered":"2024-04-02T09:12:40.347686+0000","last_clean":"2024-04-02T09:12:40.347686+0000","last_became_active":"2024-04-02T09:12:39.132574+0000","last_became_peered":"2024-04-02T09:12:39.132574+0000","last_unstale":"2024-04-02T09:12:40.347686+0000","last_undegraded":"2024-04-02T09:12:40.347686+0000","last_fullsized":"2024-04-02T09:12:40.347686+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:13:38.360757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348986+0000","last_change":"2024-04-02T09:12:39.142178+0000","last_active":"2024-04-02T09:12:40.348986+0000","last_peered":"2024-04-02T09:12:40.348986+0000","last_clean":"2024-04-02T09:12:40.348986+0000","last_became_active":"2024-04-02T09:12:39.141888+0000","last_became_peered":"2024-04-02T09:12:39.141888+0000","last_unstale":"2024-04-02T09:12:40.348986+0000","last_undegraded":"2024-04-02T09:12:40.348986+0000","last_fullsized":"2024-04-02T09:12:40.348986+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:35:55.944314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.113868+0000","last_change":"2024-04-02T09:12:25.185102+0000","last_active":"2024-04-02T09:12:39.113868+0000","last_peered":"2024-04-02T09:12:39.113868+0000","last_clean":"2024-04-02T09:12:39.113868+0000","last_became_active":"2024-04-02T09:12:25.184829+0000","last_became_peered":"2024-04-02T09:12:25.184829+0000","last_unstale":"2024-04-02T09:12:39.113868+0000","last_undegraded":"2024-04-02T09:12:39.113868+0000","last_fullsized":"2024-04-02T09:12:39.113868+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:28.343401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348863+0000","last_change":"2024-04-02T09:12:39.140834+0000","last_active":"2024-04-02T09:12:40.348863+0000","last_peered":"2024-04-02T09:12:40.348863+0000","last_clean":"2024-04-02T09:12:40.348863+0000","last_became_active":"2024-04-02T09:12:39.140430+0000","last_became_peered":"2024-04-02T09:12:39.140430+0000","last_unstale":"2024-04-02T09:12:40.348863+0000","last_undegraded":"2024-04-02T09:12:40.348863+0000","last_fullsized":"2024-04-02T09:12:40.348863+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:38:03.793164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.127120+0000","last_change":"2024-04-02T09:12:25.205832+0000","last_active":"2024-04-02T09:12:39.127120+0000","last_peered":"2024-04-02T09:12:39.127120+0000","last_clean":"2024-04-02T09:12:39.127120+0000","last_became_active":"2024-04-02T09:12:25.205206+0000","last_became_peered":"2024-04-02T09:12:25.205206+0000","last_unstale":"2024-04-02T09:12:39.127120+0000","last_undegraded":"2024-04-02T09:12:39.127120+0000","last_fullsized":"2024-04-02T09:12:39.127120+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:03:17.511411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348672+0000","last_change":"2024-04-02T09:12:39.123156+0000","last_active":"2024-04-02T09:12:40.348672+0000","last_peered":"2024-04-02T09:12:40.348672+0000","last_clean":"2024-04-02T09:12:40.348672+0000","last_became_active":"2024-04-02T09:12:39.122879+0000","last_became_peered":"2024-04-02T09:12:39.122879+0000","last_unstale":"2024-04-02T09:12:40.348672+0000","last_undegraded":"2024-04-02T09:12:40.348672+0000","last_fullsized":"2024-04-02T09:12:40.348672+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:35:05.722753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116264+0000","last_change":"2024-04-02T09:12:25.197238+0000","last_active":"2024-04-02T09:12:39.116264+0000","last_peered":"2024-04-02T09:12:39.116264+0000","last_clean":"2024-04-02T09:12:39.116264+0000","last_became_active":"2024-04-02T09:12:25.196921+0000","last_became_peered":"2024-04-02T09:12:25.196921+0000","last_unstale":"2024-04-02T09:12:39.116264+0000","last_undegraded":"2024-04-02T09:12:39.116264+0000","last_fullsized":"2024-04-02T09:12:39.116264+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:40:34.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117627+0000","last_change":"2024-04-02T09:12:39.125280+0000","last_active":"2024-04-02T09:12:40.117627+0000","last_peered":"2024-04-02T09:12:40.117627+0000","last_clean":"2024-04-02T09:12:40.117627+0000","last_became_active":"2024-04-02T09:12:39.123651+0000","last_became_peered":"2024-04-02T09:12:39.123651+0000","last_unstale":"2024-04-02T09:12:40.117627+0000","last_undegraded":"2024-04-02T09:12:40.117627+0000","last_fullsized":"2024-04-02T09:12:40.117627+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:47:04.234260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.115677+0000","last_change":"2024-04-02T09:12:25.201010+0000","last_active":"2024-04-02T09:12:39.115677+0000","last_peered":"2024-04-02T09:12:39.115677+0000","last_clean":"2024-04-02T09:12:39.115677+0000","last_became_active":"2024-04-02T09:12:25.198423+0000","last_became_peered":"2024-04-02T09:12:25.198423+0000","last_unstale":"2024-04-02T09:12:39.115677+0000","last_undegraded":"2024-04-02T09:12:39.115677+0000","last_fullsized":"2024-04-02T09:12:39.115677+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:44:28.685030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116695+0000","last_change":"2024-04-02T09:12:39.142382+0000","last_active":"2024-04-02T09:12:40.116695+0000","last_peered":"2024-04-02T09:12:40.116695+0000","last_clean":"2024-04-02T09:12:40.116695+0000","last_became_active":"2024-04-02T09:12:39.142079+0000","last_became_peered":"2024-04-02T09:12:39.142079+0000","last_unstale":"2024-04-02T09:12:40.116695+0000","last_undegraded":"2024-04-02T09:12:40.116695+0000","last_fullsized":"2024-04-02T09:12:40.116695+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:40:48.214067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117659+0000","last_change":"2024-04-02T09:12:25.196567+0000","last_active":"2024-04-02T09:12:39.117659+0000","last_peered":"2024-04-02T09:12:39.117659+0000","last_clean":"2024-04-02T09:12:39.117659+0000","last_became_active":"2024-04-02T09:12:25.196264+0000","last_became_peered":"2024-04-02T09:12:25.196264+0000","last_unstale":"2024-04-02T09:12:39.117659+0000","last_undegraded":"2024-04-02T09:12:39.117659+0000","last_fullsized":"2024-04-02T09:12:39.117659+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:03.387546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348467+0000","last_change":"2024-04-02T09:12:39.147190+0000","last_active":"2024-04-02T09:12:40.348467+0000","last_peered":"2024-04-02T09:12:40.348467+0000","last_clean":"2024-04-02T09:12:40.348467+0000","last_became_active":"2024-04-02T09:12:39.146900+0000","last_became_peered":"2024-04-02T09:12:39.146900+0000","last_unstale":"2024-04-02T09:12:40.348467+0000","last_undegraded":"2024-04-02T09:12:40.348467+0000","last_fullsized":"2024-04-02T09:12:40.348467+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:28.062436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116479+0000","last_change":"2024-04-02T09:12:25.196715+0000","last_active":"2024-04-02T09:12:39.116479+0000","last_peered":"2024-04-02T09:12:39.116479+0000","last_clean":"2024-04-02T09:12:39.116479+0000","last_became_active":"2024-04-02T09:12:25.196433+0000","last_became_peered":"2024-04-02T09:12:25.196433+0000","last_unstale":"2024-04-02T09:12:39.116479+0000","last_undegraded":"2024-04-02T09:12:39.116479+0000","last_fullsized":"2024-04-02T09:12:39.116479+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:30.835437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347349+0000","last_change":"2024-04-02T09:12:39.137405+0000","last_active":"2024-04-02T09:12:40.347349+0000","last_peered":"2024-04-02T09:12:40.347349+0000","last_clean":"2024-04-02T09:12:40.347349+0000","last_became_active":"2024-04-02T09:12:39.136852+0000","last_became_peered":"2024-04-02T09:12:39.136852+0000","last_unstale":"2024-04-02T09:12:40.347349+0000","last_undegraded":"2024-04-02T09:12:40.347349+0000","last_fullsized":"2024-04-02T09:12:40.347349+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:59:13.898116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.119988+0000","last_change":"2024-04-02T09:12:25.207159+0000","last_active":"2024-04-02T09:12:39.119988+0000","last_peered":"2024-04-02T09:12:39.119988+0000","last_clean":"2024-04-02T09:12:39.119988+0000","last_became_active":"2024-04-02T09:12:25.206804+0000","last_became_peered":"2024-04-02T09:12:25.206804+0000","last_unstale":"2024-04-02T09:12:39.119988+0000","last_undegraded":"2024-04-02T09:12:39.119988+0000","last_fullsized":"2024-04-02T09:12:39.119988+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:19.646703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349878+0000","last_change":"2024-04-02T09:12:39.142032+0000","last_active":"2024-04-02T09:12:40.349878+0000","last_peered":"2024-04-02T09:12:40.349878+0000","last_clean":"2024-04-02T09:12:40.349878+0000","last_became_active":"2024-04-02T09:12:39.141722+0000","last_became_peered":"2024-04-02T09:12:39.141722+0000","last_unstale":"2024-04-02T09:12:40.349878+0000","last_undegraded":"2024-04-02T09:12:40.349878+0000","last_fullsized":"2024-04-02T09:12:40.349878+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:17.921961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116835+0000","last_change":"2024-04-02T09:12:25.196993+0000","last_active":"2024-04-02T09:12:39.116835+0000","last_peered":"2024-04-02T09:12:39.116835+0000","last_clean":"2024-04-02T09:12:39.116835+0000","last_became_active":"2024-04-02T09:12:25.196316+0000","last_became_peered":"2024-04-02T09:12:25.196316+0000","last_unstale":"2024-04-02T09:12:39.116835+0000","last_undegraded":"2024-04-02T09:12:39.116835+0000","last_fullsized":"2024-04-02T09:12:39.116835+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:36:48.733839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348669+0000","last_change":"2024-04-02T09:12:39.134057+0000","last_active":"2024-04-02T09:12:40.348669+0000","last_peered":"2024-04-02T09:12:40.348669+0000","last_clean":"2024-04-02T09:12:40.348669+0000","last_became_active":"2024-04-02T09:12:39.132976+0000","last_became_peered":"2024-04-02T09:12:39.132976+0000","last_unstale":"2024-04-02T09:12:40.348669+0000","last_undegraded":"2024-04-02T09:12:40.348669+0000","last_fullsized":"2024-04-02T09:12:40.348669+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:45:43.597770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.113451+0000","last_change":"2024-04-02T09:12:25.188711+0000","last_active":"2024-04-02T09:12:39.113451+0000","last_peered":"2024-04-02T09:12:39.113451+0000","last_clean":"2024-04-02T09:12:39.113451+0000","last_became_active":"2024-04-02T09:12:25.188399+0000","last_became_peered":"2024-04-02T09:12:25.188399+0000","last_unstale":"2024-04-02T09:12:39.113451+0000","last_undegraded":"2024-04-02T09:12:39.113451+0000","last_fullsized":"2024-04-02T09:12:39.113451+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:20:21.844925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348815+0000","last_change":"2024-04-02T09:12:39.126125+0000","last_active":"2024-04-02T09:12:40.348815+0000","last_peered":"2024-04-02T09:12:40.348815+0000","last_clean":"2024-04-02T09:12:40.348815+0000","last_became_active":"2024-04-02T09:12:39.125868+0000","last_became_peered":"2024-04-02T09:12:39.125868+0000","last_unstale":"2024-04-02T09:12:40.348815+0000","last_undegraded":"2024-04-02T09:12:40.348815+0000","last_fullsized":"2024-04-02T09:12:40.348815+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:48:39.621381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.115933+0000","last_change":"2024-04-02T09:12:25.186757+0000","last_active":"2024-04-02T09:12:39.115933+0000","last_peered":"2024-04-02T09:12:39.115933+0000","last_clean":"2024-04-02T09:12:39.115933+0000","last_became_active":"2024-04-02T09:12:25.186436+0000","last_became_peered":"2024-04-02T09:12:25.186436+0000","last_unstale":"2024-04-02T09:12:39.115933+0000","last_undegraded":"2024-04-02T09:12:39.115933+0000","last_fullsized":"2024-04-02T09:12:39.115933+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:18:21.964569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349159+0000","last_change":"2024-04-02T09:12:39.123464+0000","last_active":"2024-04-02T09:12:40.349159+0000","last_peered":"2024-04-02T09:12:40.349159+0000","last_clean":"2024-04-02T09:12:40.349159+0000","last_became_active":"2024-04-02T09:12:39.123189+0000","last_became_peered":"2024-04-02T09:12:39.123189+0000","last_unstale":"2024-04-02T09:12:40.349159+0000","last_undegraded":"2024-04-02T09:12:40.349159+0000","last_fullsized":"2024-04-02T09:12:40.349159+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:26:54.971921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.121991+0000","last_change":"2024-04-02T09:12:25.198312+0000","last_active":"2024-04-02T09:12:39.121991+0000","last_peered":"2024-04-02T09:12:39.121991+0000","last_clean":"2024-04-02T09:12:39.121991+0000","last_became_active":"2024-04-02T09:12:25.198040+0000","last_became_peered":"2024-04-02T09:12:25.198040+0000","last_unstale":"2024-04-02T09:12:39.121991+0000","last_undegraded":"2024-04-02T09:12:39.121991+0000","last_fullsized":"2024-04-02T09:12:39.121991+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:02.893062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112978+0000","last_change":"2024-04-02T09:12:39.132043+0000","last_active":"2024-04-02T09:12:40.112978+0000","last_peered":"2024-04-02T09:12:40.112978+0000","last_clean":"2024-04-02T09:12:40.112978+0000","last_became_active":"2024-04-02T09:12:39.130384+0000","last_became_peered":"2024-04-02T09:12:39.130384+0000","last_unstale":"2024-04-02T09:12:40.112978+0000","last_undegraded":"2024-04-02T09:12:40.112978+0000","last_fullsized":"2024-04-02T09:12:40.112978+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:10.514935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.134359+0000","last_change":"2024-04-02T09:12:25.205465+0000","last_active":"2024-04-02T09:12:39.134359+0000","last_peered":"2024-04-02T09:12:39.134359+0000","last_clean":"2024-04-02T09:12:39.134359+0000","last_became_active":"2024-04-02T09:12:25.204248+0000","last_became_peered":"2024-04-02T09:12:25.204248+0000","last_unstale":"2024-04-02T09:12:39.134359+0000","last_undegraded":"2024-04-02T09:12:39.134359+0000","last_fullsized":"2024-04-02T09:12:39.134359+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:32:03.502011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112641+0000","last_change":"2024-04-02T09:12:39.118797+0000","last_active":"2024-04-02T09:12:40.112641+0000","last_peered":"2024-04-02T09:12:40.112641+0000","last_clean":"2024-04-02T09:12:40.112641+0000","last_became_active":"2024-04-02T09:12:39.118542+0000","last_became_peered":"2024-04-02T09:12:39.118542+0000","last_unstale":"2024-04-02T09:12:40.112641+0000","last_undegraded":"2024-04-02T09:12:40.112641+0000","last_fullsized":"2024-04-02T09:12:40.112641+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:35:05.722753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116558+0000","last_change":"2024-04-02T09:12:25.186623+0000","last_active":"2024-04-02T09:12:39.116558+0000","last_peered":"2024-04-02T09:12:39.116558+0000","last_clean":"2024-04-02T09:12:39.116558+0000","last_became_active":"2024-04-02T09:12:25.186297+0000","last_became_peered":"2024-04-02T09:12:25.186297+0000","last_unstale":"2024-04-02T09:12:39.116558+0000","last_undegraded":"2024-04-02T09:12:39.116558+0000","last_fullsized":"2024-04-02T09:12:39.116558+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:36:48.733839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.349352+0000","last_change":"2024-04-02T09:12:39.143922+0000","last_active":"2024-04-02T09:12:40.349352+0000","last_peered":"2024-04-02T09:12:40.349352+0000","last_clean":"2024-04-02T09:12:40.349352+0000","last_became_active":"2024-04-02T09:12:39.141641+0000","last_became_peered":"2024-04-02T09:12:39.141641+0000","last_unstale":"2024-04-02T09:12:40.349352+0000","last_undegraded":"2024-04-02T09:12:40.349352+0000","last_fullsized":"2024-04-02T09:12:40.349352+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:10:56.650439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114225+0000","last_change":"2024-04-02T09:12:25.203968+0000","last_active":"2024-04-02T09:12:39.114225+0000","last_peered":"2024-04-02T09:12:39.114225+0000","last_clean":"2024-04-02T09:12:39.114225+0000","last_became_active":"2024-04-02T09:12:25.203714+0000","last_became_peered":"2024-04-02T09:12:25.203714+0000","last_unstale":"2024-04-02T09:12:39.114225+0000","last_undegraded":"2024-04-02T09:12:39.114225+0000","last_fullsized":"2024-04-02T09:12:39.114225+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:44:28.685030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347121+0000","last_change":"2024-04-02T09:12:39.129719+0000","last_active":"2024-04-02T09:12:40.347121+0000","last_peered":"2024-04-02T09:12:40.347121+0000","last_clean":"2024-04-02T09:12:40.347121+0000","last_became_active":"2024-04-02T09:12:39.129414+0000","last_became_peered":"2024-04-02T09:12:39.129414+0000","last_unstale":"2024-04-02T09:12:40.347121+0000","last_undegraded":"2024-04-02T09:12:40.347121+0000","last_fullsized":"2024-04-02T09:12:40.347121+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:01:44.771579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.119899+0000","last_change":"2024-04-02T09:12:25.202349+0000","last_active":"2024-04-02T09:12:39.119899+0000","last_peered":"2024-04-02T09:12:39.119899+0000","last_clean":"2024-04-02T09:12:39.119899+0000","last_became_active":"2024-04-02T09:12:25.200134+0000","last_became_peered":"2024-04-02T09:12:25.200134+0000","last_unstale":"2024-04-02T09:12:39.119899+0000","last_undegraded":"2024-04-02T09:12:39.119899+0000","last_fullsized":"2024-04-02T09:12:39.119899+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:45:54.007817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.117390+0000","last_change":"2024-04-02T09:12:39.136859+0000","last_active":"2024-04-02T09:12:40.117390+0000","last_peered":"2024-04-02T09:12:40.117390+0000","last_clean":"2024-04-02T09:12:40.117390+0000","last_became_active":"2024-04-02T09:12:39.136602+0000","last_became_peered":"2024-04-02T09:12:39.136602+0000","last_unstale":"2024-04-02T09:12:40.117390+0000","last_undegraded":"2024-04-02T09:12:40.117390+0000","last_fullsized":"2024-04-02T09:12:40.117390+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:10:16.285248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.118534+0000","last_change":"2024-04-02T09:12:25.201593+0000","last_active":"2024-04-02T09:12:39.118534+0000","last_peered":"2024-04-02T09:12:39.118534+0000","last_clean":"2024-04-02T09:12:39.118534+0000","last_became_active":"2024-04-02T09:12:25.200851+0000","last_became_peered":"2024-04-02T09:12:25.200851+0000","last_unstale":"2024-04-02T09:12:39.118534+0000","last_undegraded":"2024-04-02T09:12:39.118534+0000","last_fullsized":"2024-04-02T09:12:39.118534+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:26:33.117159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.113284+0000","last_change":"2024-04-02T09:12:39.144609+0000","last_active":"2024-04-02T09:12:40.113284+0000","last_peered":"2024-04-02T09:12:40.113284+0000","last_clean":"2024-04-02T09:12:40.113284+0000","last_became_active":"2024-04-02T09:12:39.144321+0000","last_became_peered":"2024-04-02T09:12:39.144321+0000","last_unstale":"2024-04-02T09:12:40.113284+0000","last_undegraded":"2024-04-02T09:12:40.113284+0000","last_fullsized":"2024-04-02T09:12:40.113284+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:37:05.275434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.120759+0000","last_change":"2024-04-02T09:12:25.191970+0000","last_active":"2024-04-02T09:12:39.120759+0000","last_peered":"2024-04-02T09:12:39.120759+0000","last_clean":"2024-04-02T09:12:39.120759+0000","last_became_active":"2024-04-02T09:12:25.191572+0000","last_became_peered":"2024-04-02T09:12:25.191572+0000","last_unstale":"2024-04-02T09:12:39.120759+0000","last_undegraded":"2024-04-02T09:12:39.120759+0000","last_fullsized":"2024-04-02T09:12:39.120759+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:03.387546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348766+0000","last_change":"2024-04-02T09:12:39.140664+0000","last_active":"2024-04-02T09:12:40.348766+0000","last_peered":"2024-04-02T09:12:40.348766+0000","last_clean":"2024-04-02T09:12:40.348766+0000","last_became_active":"2024-04-02T09:12:39.140336+0000","last_became_peered":"2024-04-02T09:12:39.140336+0000","last_unstale":"2024-04-02T09:12:40.348766+0000","last_undegraded":"2024-04-02T09:12:40.348766+0000","last_fullsized":"2024-04-02T09:12:40.348766+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:23:36.313057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.115897+0000","last_change":"2024-04-02T09:12:25.201485+0000","last_active":"2024-04-02T09:12:39.115897+0000","last_peered":"2024-04-02T09:12:39.115897+0000","last_clean":"2024-04-02T09:12:39.115897+0000","last_became_active":"2024-04-02T09:12:25.200542+0000","last_became_peered":"2024-04-02T09:12:25.200542+0000","last_unstale":"2024-04-02T09:12:39.115897+0000","last_undegraded":"2024-04-02T09:12:39.115897+0000","last_fullsized":"2024-04-02T09:12:39.115897+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:19:30.245770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.348141+0000","last_change":"2024-04-02T09:12:39.129069+0000","last_active":"2024-04-02T09:12:40.348141+0000","last_peered":"2024-04-02T09:12:40.348141+0000","last_clean":"2024-04-02T09:12:40.348141+0000","last_became_active":"2024-04-02T09:12:39.128806+0000","last_became_peered":"2024-04-02T09:12:39.128806+0000","last_unstale":"2024-04-02T09:12:40.348141+0000","last_undegraded":"2024-04-02T09:12:40.348141+0000","last_fullsized":"2024-04-02T09:12:40.348141+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:32.146966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.114164+0000","last_change":"2024-04-02T09:12:25.200889+0000","last_active":"2024-04-02T09:12:39.114164+0000","last_peered":"2024-04-02T09:12:39.114164+0000","last_clean":"2024-04-02T09:12:39.114164+0000","last_became_active":"2024-04-02T09:12:25.200635+0000","last_became_peered":"2024-04-02T09:12:25.200635+0000","last_unstale":"2024-04-02T09:12:39.114164+0000","last_undegraded":"2024-04-02T09:12:39.114164+0000","last_fullsized":"2024-04-02T09:12:39.114164+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:43:29.229527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.113023+0000","last_change":"2024-04-02T09:12:39.125881+0000","last_active":"2024-04-02T09:12:40.113023+0000","last_peered":"2024-04-02T09:12:40.113023+0000","last_clean":"2024-04-02T09:12:40.113023+0000","last_became_active":"2024-04-02T09:12:39.124319+0000","last_became_peered":"2024-04-02T09:12:39.124319+0000","last_unstale":"2024-04-02T09:12:40.113023+0000","last_undegraded":"2024-04-02T09:12:40.113023+0000","last_fullsized":"2024-04-02T09:12:40.113023+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:55:45.430413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.121484+0000","last_change":"2024-04-02T09:12:25.191955+0000","last_active":"2024-04-02T09:12:39.121484+0000","last_peered":"2024-04-02T09:12:39.121484+0000","last_clean":"2024-04-02T09:12:39.121484+0000","last_became_active":"2024-04-02T09:12:25.191699+0000","last_became_peered":"2024-04-02T09:12:25.191699+0000","last_unstale":"2024-04-02T09:12:39.121484+0000","last_undegraded":"2024-04-02T09:12:39.121484+0000","last_fullsized":"2024-04-02T09:12:39.121484+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:22:42.917826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116986+0000","last_change":"2024-04-02T09:12:39.132319+0000","last_active":"2024-04-02T09:12:40.116986+0000","last_peered":"2024-04-02T09:12:40.116986+0000","last_clean":"2024-04-02T09:12:40.116986+0000","last_became_active":"2024-04-02T09:12:39.131855+0000","last_became_peered":"2024-04-02T09:12:39.131855+0000","last_unstale":"2024-04-02T09:12:40.116986+0000","last_undegraded":"2024-04-02T09:12:40.116986+0000","last_fullsized":"2024-04-02T09:12:40.116986+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:25.746845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.133348+0000","last_change":"2024-04-02T09:12:25.190996+0000","last_active":"2024-04-02T09:12:39.133348+0000","last_peered":"2024-04-02T09:12:39.133348+0000","last_clean":"2024-04-02T09:12:39.133348+0000","last_became_active":"2024-04-02T09:12:25.190655+0000","last_became_peered":"2024-04-02T09:12:25.190655+0000","last_unstale":"2024-04-02T09:12:39.133348+0000","last_undegraded":"2024-04-02T09:12:39.133348+0000","last_fullsized":"2024-04-02T09:12:39.133348+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:48:22.188323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.347985+0000","last_change":"2024-04-02T09:12:39.132496+0000","last_active":"2024-04-02T09:12:40.347985+0000","last_peered":"2024-04-02T09:12:40.347985+0000","last_clean":"2024-04-02T09:12:40.347985+0000","last_became_active":"2024-04-02T09:12:39.132240+0000","last_became_peered":"2024-04-02T09:12:39.132240+0000","last_unstale":"2024-04-02T09:12:40.347985+0000","last_undegraded":"2024-04-02T09:12:40.347985+0000","last_fullsized":"2024-04-02T09:12:40.347985+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:14:26.132847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116570+0000","last_change":"2024-04-02T09:12:25.194789+0000","last_active":"2024-04-02T09:12:39.116570+0000","last_peered":"2024-04-02T09:12:39.116570+0000","last_clean":"2024-04-02T09:12:39.116570+0000","last_became_active":"2024-04-02T09:12:25.194433+0000","last_became_peered":"2024-04-02T09:12:25.194433+0000","last_unstale":"2024-04-02T09:12:39.116570+0000","last_undegraded":"2024-04-02T09:12:39.116570+0000","last_fullsized":"2024-04-02T09:12:39.116570+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:18.798282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.116611+0000","last_change":"2024-04-02T09:12:39.134852+0000","last_active":"2024-04-02T09:12:40.116611+0000","last_peered":"2024-04-02T09:12:40.116611+0000","last_clean":"2024-04-02T09:12:40.116611+0000","last_became_active":"2024-04-02T09:12:39.134565+0000","last_became_peered":"2024-04-02T09:12:39.134565+0000","last_unstale":"2024-04-02T09:12:40.116611+0000","last_undegraded":"2024-04-02T09:12:40.116611+0000","last_fullsized":"2024-04-02T09:12:40.116611+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:11.640312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.117583+0000","last_change":"2024-04-02T09:12:25.208060+0000","last_active":"2024-04-02T09:12:39.117583+0000","last_peered":"2024-04-02T09:12:39.117583+0000","last_clean":"2024-04-02T09:12:39.117583+0000","last_became_active":"2024-04-02T09:12:25.206932+0000","last_became_peered":"2024-04-02T09:12:25.206932+0000","last_unstale":"2024-04-02T09:12:39.117583+0000","last_undegraded":"2024-04-02T09:12:39.117583+0000","last_fullsized":"2024-04-02T09:12:39.117583+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:42:11.810703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-02T09:12:40.112484+0000","last_change":"2024-04-02T09:12:39.122041+0000","last_active":"2024-04-02T09:12:40.112484+0000","last_peered":"2024-04-02T09:12:40.112484+0000","last_clean":"2024-04-02T09:12:40.112484+0000","last_became_active":"2024-04-02T09:12:39.119213+0000","last_became_peered":"2024-04-02T09:12:39.119213+0000","last_unstale":"2024-04-02T09:12:40.112484+0000","last_undegraded":"2024-04-02T09:12:40.112484+0000","last_fullsized":"2024-04-02T09:12:40.112484+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:38.605138+0000","last_clean_scrub_stamp":"2024-04-02T09:12:38.605138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:37:26.600229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-02T09:12:39.116911+0000","last_change":"2024-04-02T09:12:25.187698+0000","last_active":"2024-04-02T09:12:39.116911+0000","last_peered":"2024-04-02T09:12:39.116911+0000","last_clean":"2024-04-02T09:12:39.116911+0000","last_became_active":"2024-04-02T09:12:25.187420+0000","last_became_peered":"2024-04-02T09:12:25.187420+0000","last_unstale":"2024-04-02T09:12:39.116911+0000","last_undegraded":"2024-04-02T09:12:39.116911+0000","last_fullsized":"2024-04-02T09:12:39.116911+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:12:24.668996+0000","last_clean_scrub_stamp":"2024-04-02T09:12:24.668996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:45:21.225286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3603,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575991,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60148,"kb_used_data":2792,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311692,"statfs":{"total":96636764160,"available":96575172608,"internally_reserved":0,"allocated":2859008,"data_stored":1450119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.68500000000000005,"15min":0.71199999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.93899999999999995,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68700000000000006,"15min":0.71999999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91800000000000004,"5min":0.94199999999999995,"15min":1.165},"last":0.65100000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.70599999999999996,"15min":0.73099999999999998},"min":{"1min":0.48899999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.95399999999999996,"5min":0.95699999999999996,"15min":1.125},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68100000000000005,"15min":0.71399999999999997},"min":{"1min":0.51700000000000002,"5min":0.497,"15min":0.42399999999999999},"max":{"1min":0.89100000000000001,"5min":0.92700000000000005,"15min":1.042},"last":0.621}]},{"osd":2,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.749,"15min":0.76100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.873,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.78200000000000003,"15min":0.80500000000000005},"min":{"1min":0.54900000000000004,"5min":0.504,"15min":0.504},"max":{"1min":0.98299999999999998,"5min":1.036,"15min":1.109},"last":0.68200000000000005}]},{"osd":3,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.72399999999999998,"15min":0.74099999999999999},"min":{"1min":0.497,"5min":0.45800000000000002,"15min":0.41399999999999998},"max":{"1min":0.94399999999999995,"5min":1.452,"15min":1.452},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.748,"15min":0.76400000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.0740000000000001},"last":0.70999999999999996}]},{"osd":4,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61699999999999999,"15min":0.60799999999999998},"min":{"1min":0.42799999999999999,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.95299999999999996,"15min":1.01},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.73699999999999999,"15min":0.71699999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.38300000000000001},"max":{"1min":1.006,"5min":1.536,"15min":1.536},"last":0.80300000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.68799999999999994,"15min":0.67500000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":0.93400000000000005,"5min":1.0169999999999999,"15min":1.1850000000000001},"last":0.746},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.72499999999999998,"15min":0.71699999999999997},"min":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.40600000000000003},"max":{"1min":0.88500000000000001,"5min":1.097,"15min":1.1319999999999999},"last":0.73799999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45800000000000002},"max":{"1min":0.871,"5min":1.034,"15min":1.3919999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.72299999999999998,"15min":0.69099999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.97299999999999998,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.77400000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575989,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42112,"kb_used_data":2796,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329728,"statfs":{"total":96636764160,"available":96593641472,"internally_reserved":0,"allocated":2863104,"data_stored":1449641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.73699999999999999},"min":{"1min":0.54900000000000004,"5min":0.50800000000000001,"15min":0.42699999999999999},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73199999999999998,"15min":0.747},"min":{"1min":0.48699999999999999,"5min":0.47899999999999998,"15min":0.45800000000000002},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":1.077},"last":0.59399999999999997}]},{"osd":1,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70399999999999996,"15min":0.72099999999999997},"min":{"1min":0.57199999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.212},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.73499999999999999,"15min":0.75800000000000001},"min":{"1min":0.55700000000000005,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.65300000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.73599999999999999,"15min":0.75800000000000001},"min":{"1min":0.53500000000000003,"5min":0.47699999999999998,"15min":0.47199999999999998},"max":{"1min":1.016,"5min":1.016,"15min":1.2789999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.73899999999999999,"15min":0.748},"min":{"1min":0.50900000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.93300000000000005,"5min":0.95999999999999996,"15min":1.226},"last":0.50700000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.74399999999999999,"15min":0.76900000000000002},"min":{"1min":0.50800000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.141,"5min":1.141,"15min":1.2549999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76900000000000002,"15min":0.77500000000000002},"min":{"1min":0.63100000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.94099999999999995,"5min":0.95399999999999996,"15min":1.3200000000000001},"last":0.82399999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60799999999999998,"15min":0.623},"min":{"1min":0.45300000000000001,"5min":0.42199999999999999,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.1259999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.627,"5min":0.65100000000000002,"15min":0.66700000000000004},"min":{"1min":0.51100000000000001,"5min":0.47399999999999998,"15min":0.41999999999999998},"max":{"1min":0.84399999999999997,"5min":0.92300000000000004,"15min":1.1879999999999999},"last":0.72999999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64500000000000002,"15min":0.65600000000000003},"min":{"1min":0.47899999999999998,"5min":0.42799999999999999,"15min":0.42199999999999999},"max":{"1min":0.86599999999999999,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70099999999999996,"15min":0.72599999999999998},"min":{"1min":0.505,"5min":0.48299999999999998,"15min":0.46700000000000003},"max":{"1min":0.92000000000000004,"5min":1.034,"15min":1.204},"last":0.747}]},{"osd":7,"last update":"Tue Apr 2 09:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.2370000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.72199999999999998,"15min":0.73899999999999999},"min":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.48299999999999998},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.77500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575990,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52544,"kb_used_data":3372,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319296,"statfs":{"total":96636764160,"available":96582959104,"internally_reserved":0,"allocated":3452928,"data_stored":2040467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44400000000000001,"15min":0.45300000000000001},"min":{"1min":0.35499999999999998,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.67000000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.502,"15min":0.51200000000000001},"min":{"1min":0.371,"5min":0.312,"15min":0.312},"max":{"1min":0.76000000000000001,"5min":0.78600000000000003,"15min":0.79800000000000004},"last":0.39500000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.50900000000000001,"15min":0.505},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.753,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58299999999999996,"15min":0.59699999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74099999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53100000000000003}]},{"osd":3,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.42699999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76200000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.505},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.40100000000000002,"5min":0.39000000000000001,"15min":0.378},"max":{"1min":0.73199999999999998,"5min":0.86699999999999999,"15min":0.878},"last":0.47999999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.79700000000000004,"15min":0.79200000000000004},"min":{"1min":0.61099999999999999,"5min":0.57599999999999996,"15min":0.433},"max":{"1min":0.98499999999999999,"5min":1.1120000000000001,"15min":1.1479999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.78000000000000003,"15min":0.78900000000000003},"min":{"1min":0.61299999999999999,"5min":0.57199999999999995,"15min":0.41799999999999998},"max":{"1min":1.05,"5min":1.05,"15min":1.25},"last":0.91200000000000003}]},{"osd":5,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.77700000000000002,"15min":0.79600000000000004},"min":{"1min":0.59799999999999998,"5min":0.54300000000000004,"15min":0.52900000000000003},"max":{"1min":1.01,"5min":1.115,"15min":1.321},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.65800000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.173},"last":0.79300000000000004}]},{"osd":6,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.81200000000000006,"15min":0.81799999999999995},"min":{"1min":0.67300000000000004,"5min":0.53900000000000003,"15min":0.48099999999999998},"max":{"1min":1.0649999999999999,"5min":1.159,"15min":1.2210000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.82199999999999995,"15min":0.81299999999999994},"min":{"1min":0.61899999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.087,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":1.0009999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.82099999999999995,"15min":0.82799999999999996},"min":{"1min":0.64200000000000002,"5min":0.58499999999999996,"15min":0.53700000000000003},"max":{"1min":1.0680000000000001,"5min":1.1080000000000001,"15min":1.236},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.83899999999999997,"15min":0.83999999999999997},"min":{"1min":0.63700000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0329999999999999,"5min":1.0720000000000001,"15min":1.2789999999999999},"last":0.84999999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575990,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47860,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323980,"statfs":{"total":96636764160,"available":96587755520,"internally_reserved":0,"allocated":2846720,"data_stored":1448988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45400000000000001,"15min":0.45800000000000002},"min":{"1min":0.33700000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.69899999999999995,"5min":0.76500000000000001,"15min":0.86699999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.5,"15min":0.51000000000000001},"min":{"1min":0.32800000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.54900000000000004}]},{"osd":2,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.42099999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.66100000000000003,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.503},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53900000000000003,"15min":0.55400000000000005},"min":{"1min":0.36699999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72499999999999998,"5min":0.999,"15min":0.999},"last":0.59499999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.40100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.77800000000000002,"5min":0.80500000000000005,"15min":0.85699999999999998},"last":0.437},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.63900000000000001,"15min":0.64300000000000002},"min":{"1min":0.5,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.82799999999999996,"5min":0.92000000000000004,"15min":0.96399999999999997},"last":0.64600000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.82499999999999996,"15min":0.83299999999999996},"min":{"1min":0.65700000000000003,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.2769999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81000000000000005,"15min":0.82399999999999995},"min":{"1min":0.621,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.95999999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.80100000000000005,"15min":0.82699999999999996},"min":{"1min":0.58899999999999997,"5min":0.497,"15min":0.497},"max":{"1min":1.1419999999999999,"5min":1.145,"15min":1.232},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.83499999999999996,"15min":0.83399999999999996},"min":{"1min":0.67700000000000005,"5min":0.57699999999999996,"15min":0.55900000000000005},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.89900000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.58099999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1830000000000001,"5min":1.196,"15min":1.196},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.80500000000000005,"15min":0.81200000000000006},"min":{"1min":0.60499999999999998,"5min":0.52900000000000003,"15min":0.499},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.747}]},{"osd":7,"last update":"Tue Apr 2 09:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.84399999999999997,"15min":0.85399999999999998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.875},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.84099999999999997,"15min":0.85099999999999998},"min":{"1min":0.67700000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.1100000000000001,"5min":1.1719999999999999,"15min":1.2589999999999999},"last":0.91200000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575989,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51972,"kb_used_data":2804,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319868,"statfs":{"total":96636764160,"available":96583544832,"internally_reserved":0,"allocated":2871296,"data_stored":1452368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44},"min":{"1min":0.32200000000000001,"5min":0.30499999999999999,"15min":0.29199999999999998},"max":{"1min":0.61499999999999999,"5min":0.66600000000000004,"15min":0.72999999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.502,"15min":0.503},"min":{"1min":0.375,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64800000000000002,"5min":0.79900000000000004,"15min":1.4930000000000001},"last":0.44700000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.40999999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65800000000000003,"5min":0.75800000000000001,"15min":0.80600000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.40100000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.68799999999999994,"5min":0.81999999999999995,"15min":1.4530000000000001},"last":0.52200000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.56000000000000005},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.80700000000000005,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.497},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.628,"15min":0.63500000000000001},"min":{"1min":0.50900000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77700000000000002,"5min":0.86799999999999999,"15min":1.0740000000000001},"last":0.59099999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.77700000000000002,"15min":0.77900000000000003},"min":{"1min":0.53700000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.008,"5min":1.121,"15min":1.121},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79300000000000004,"15min":0.79800000000000004},"min":{"1min":0.65200000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.1259999999999999,"5min":1.179,"15min":1.5329999999999999},"last":0.871}]},{"osd":5,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80000000000000004,"15min":0.79300000000000004},"min":{"1min":0.69899999999999995,"5min":0.53100000000000003,"15min":0.52600000000000002},"max":{"1min":1.054,"5min":1.99,"15min":1.99},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80500000000000005,"15min":0.80800000000000005},"min":{"1min":0.60499999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0660000000000001,"5min":1.101,"15min":1.1259999999999999},"last":0.70099999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.81599999999999995,"15min":0.81799999999999995},"min":{"1min":0.63500000000000001,"5min":0.54700000000000004,"15min":0.54600000000000004},"max":{"1min":0.96899999999999997,"5min":1.173,"15min":1.5940000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80400000000000005,"15min":0.81100000000000005},"min":{"1min":0.63200000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.99199999999999999,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.61899999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.83199999999999996,"15min":0.83499999999999996},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55100000000000005},"max":{"1min":1.165,"5min":1.2110000000000001,"15min":1.5649999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.81999999999999995,"15min":0.83099999999999996},"min":{"1min":0.70499999999999996,"5min":0.497,"15min":0.497},"max":{"1min":1.0269999999999999,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.68500000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575989,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51960,"kb_used_data":2792,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319880,"statfs":{"total":96636764160,"available":96583557120,"internally_reserved":0,"allocated":2859008,"data_stored":1452230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44400000000000001,"15min":0.45300000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.48699999999999999,"15min":0.496},"min":{"1min":0.36099999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.876},"last":0.42899999999999999}]},{"osd":1,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.49399999999999999,"15min":0.502},"min":{"1min":0.34300000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.64000000000000001,"5min":0.71499999999999997,"15min":0.90300000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.57699999999999996},"min":{"1min":0.46100000000000002,"5min":0.39100000000000001,"15min":0.34100000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.622,"15min":0.63500000000000001},"min":{"1min":0.42599999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.61199999999999999,"15min":0.623},"min":{"1min":0.48099999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81699999999999995,"5min":0.82299999999999995,"15min":1.0840000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.74399999999999999,"15min":0.75900000000000001},"min":{"1min":0.503,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.071,"5min":1.071,"15min":1.0800000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76300000000000001,"15min":0.77700000000000002},"min":{"1min":0.59099999999999997,"5min":0.57199999999999995,"15min":0.49199999999999999},"max":{"1min":1.018,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.67000000000000004}]},{"osd":5,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.59599999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.054,"5min":1.1459999999999999,"15min":1.157},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.56599999999999995,"5min":0.505,"15min":0.505},"max":{"1min":0.99099999999999999,"5min":1.266,"15min":1.266},"last":0.64700000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.78500000000000003,"15min":0.79100000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.52600000000000002},"max":{"1min":1.0489999999999999,"5min":1.284,"15min":1.284},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.80400000000000005,"15min":0.81200000000000006},"min":{"1min":0.54900000000000004,"5min":0.495,"15min":0.495},"max":{"1min":1.04,"5min":1.2,"15min":1.2},"last":0.72299999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.80400000000000005,"15min":0.81499999999999995},"min":{"1min":0.61399999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.024,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.57999999999999996,"5min":0.55700000000000005,"15min":0.53100000000000003},"max":{"1min":1.103,"5min":1.248,"15min":1.248},"last":0.71199999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575989,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47864,"kb_used_data":2796,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323976,"statfs":{"total":96636764160,"available":96587751424,"internally_reserved":0,"allocated":2863104,"data_stored":1451079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68000000000000005,"15min":0.69899999999999995},"min":{"1min":0.55000000000000004,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66500000000000004,"15min":0.68899999999999995},"min":{"1min":0.40999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.83499999999999996,"5min":0.86199999999999999,"15min":1.075},"last":0.78100000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.73599999999999999,"15min":0.76100000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":1.0329999999999999,"5min":1.0549999999999999,"15min":1.069},"last":0.877},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.75,"15min":0.77700000000000002},"min":{"1min":0.53900000000000003,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.91400000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.69599999999999995,"15min":0.71799999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70499999999999996,"15min":0.73499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98999999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.85199999999999998}]},{"osd":3,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75600000000000001,"15min":0.77000000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.47399999999999998},"max":{"1min":0.98099999999999998,"5min":0.998,"15min":1.0700000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.51800000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.002,"5min":1.002,"15min":1.028},"last":0.81299999999999994}]},{"osd":5,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.60199999999999998,"15min":0.61099999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.91900000000000004,"5min":1.008,"15min":1.016},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60399999999999998,"15min":0.61699999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.56299999999999994}]},{"osd":6,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.64100000000000001,"15min":0.64200000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67300000000000004,"15min":0.69099999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":0.98299999999999998,"5min":1.0269999999999999,"15min":1.0469999999999999},"last":0.74199999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:12:20 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.0329999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.76600000000000001,"15min":0.78300000000000003},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0960000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.82799999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575992,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52552,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319288,"statfs":{"total":96636764160,"available":96582950912,"internally_reserved":0,"allocated":3469312,"data_stored":2041136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69699999999999995,"15min":0.70999999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.89600000000000002,"5min":0.98699999999999999,"15min":1.004},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69899999999999995,"15min":0.72499999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.88600000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.70099999999999996,"15min":0.72199999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85999999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.71399999999999997,"15min":0.73199999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.052},"last":0.86699999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73199999999999998,"15min":0.745},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93200000000000005,"5min":1.004,"15min":1.014},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72599999999999998,"15min":0.746},"min":{"1min":0.46600000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.018,"5min":1.018,"15min":1.0680000000000001},"last":0.65900000000000003}]},{"osd":3,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77000000000000002,"15min":0.78600000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":0.98899999999999999,"15min":1.083},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.59099999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94399999999999995,"5min":1.008,"15min":1.1020000000000001},"last":0.75}]},{"osd":4,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.629,"15min":0.629},"min":{"1min":0.56499999999999995,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.89100000000000001,"5min":0.96899999999999997,"15min":1.054},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.68400000000000005,"15min":0.67500000000000004},"min":{"1min":0.47999999999999998,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.85099999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.434,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71199999999999997,"15min":0.71099999999999997},"min":{"1min":0.501,"5min":0.44600000000000001,"15min":0.42699999999999999},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.82099999999999995}]},{"osd":7,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.73999999999999999,"15min":0.73499999999999999},"min":{"1min":0.60699999999999998,"5min":0.441,"15min":0.441},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.64000000000000001,"5min":0.48199999999999998,"15min":0.45700000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.79800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:12:43.984 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T09:12:43.984 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:12:44.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.210+0000 7fc45e000700 1 Processor -- start 2024-04-02T09:12:44.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.210+0000 7fc45e000700 1 -- start start 2024-04-02T09:12:44.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.211+0000 7fc45e000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4580ce590 0x7fc4580ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:44.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.211+0000 7fc45e000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc4580b6a20 0x7fc4580b6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:44.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.211+0000 7fc45e000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4580ceef0 con 0x7fc458160be0 2024-04-02T09:12:44.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.211+0000 7fc45e000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4580b65d0 con 0x7fc4580ce590 2024-04-02T09:12:44.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.211+0000 7fc45e000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4580d2a80 con 0x7fc4580b6a20 2024-04-02T09:12:44.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.212+0000 7fc456a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc4580ce590 0x7fc4580ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:44.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.212+0000 7fc457400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc458160be0 0x7fc4580cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:54688/0 (socket says 172.21.15.89:54688) 2024-04-02T09:12:44.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.212+0000 7fc457400700 1 -- 172.21.15.89:0/1603658000 learned_addr learned my addr 172.21.15.89:0/1603658000 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:44.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.212+0000 7fc457e00700 1 --2- 172.21.15.89:0/1603658000 >> v2:172.21.15.113:3300/0 conn(0x7fc4580b6a20 0x7fc4580b6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:44.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.212+0000 7fc456a00700 1 -- 172.21.15.89:0/1603658000 >> v2:172.21.15.113:3300/0 conn(0x7fc4580b6a20 msgr2=0x7fc4580b6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:44.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.212+0000 7fc456a00700 1 --2- 172.21.15.89:0/1603658000 >> v2:172.21.15.113:3300/0 conn(0x7fc4580b6a20 0x7fc4580b6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:44.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.212+0000 7fc456a00700 1 -- 172.21.15.89:0/1603658000 >> v1:172.21.15.89:6789/0 conn(0x7fc458160be0 legacy=0x7fc4580cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:44.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.212+0000 7fc456a00700 1 -- 172.21.15.89:0/1603658000 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc458160fc0 con 0x7fc4580ce590 2024-04-02T09:12:44.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.212+0000 7fc456a00700 1 --2- 172.21.15.89:0/1603658000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4580ce590 0x7fc4580ce9b0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fc45808d150 tx=0x7fc444009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ad1f971b36211 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:44.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.212+0000 7fc456000700 1 -- 172.21.15.89:0/1603658000 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444011830 con 0x7fc4580ce590 2024-04-02T09:12:44.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.213+0000 7fc456000700 1 -- 172.21.15.89:0/1603658000 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc444011990 con 0x7fc4580ce590 2024-04-02T09:12:44.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.213+0000 7fc456000700 1 -- 172.21.15.89:0/1603658000 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444011d20 con 0x7fc4580ce590 2024-04-02T09:12:44.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.213+0000 7fc45e000700 1 -- 172.21.15.89:0/1603658000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4580ce590 msgr2=0x7fc4580ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:44.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.213+0000 7fc45e000700 1 --2- 172.21.15.89:0/1603658000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4580ce590 0x7fc4580ce9b0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fc45808d150 tx=0x7fc444009b20 comp rx=0 tx=0).stop 2024-04-02T09:12:44.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.213+0000 7fc45e000700 1 -- 172.21.15.89:0/1603658000 shutdown_connections 2024-04-02T09:12:44.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.213+0000 7fc45e000700 1 --2- 172.21.15.89:0/1603658000 >> v2:172.21.15.113:3300/0 conn(0x7fc4580b6a20 0x7fc4580b6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:44.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.213+0000 7fc45e000700 1 --2- 172.21.15.89:0/1603658000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4580ce590 0x7fc4580ce9b0 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:44.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.213+0000 7fc45e000700 1 -- 172.21.15.89:0/1603658000 >> 172.21.15.89:0/1603658000 conn(0x7fc45808aff0 msgr2=0x7fc4580d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:44.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.213+0000 7fc45e000700 1 -- 172.21.15.89:0/1603658000 shutdown_connections 2024-04-02T09:12:44.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.213+0000 7fc45e000700 1 -- 172.21.15.89:0/1603658000 wait complete. 2024-04-02T09:12:44.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.215+0000 7fc45e000700 1 Processor -- start 2024-04-02T09:12:44.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.215+0000 7fc45e000700 1 -- start start 2024-04-02T09:12:44.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.216+0000 7fc45e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4580b6a20 0x7fc45815df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:44.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.216+0000 7fc45e000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4580ce590 0x7fc4580cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:44.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.216+0000 7fc45e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc458160be0 0x7fc45806cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:44.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.216+0000 7fc45e000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45809d230 con 0x7fc458160be0 2024-04-02T09:12:44.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.216+0000 7fc45e000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45815dd60 con 0x7fc4580ce590 2024-04-02T09:12:44.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.216+0000 7fc45e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45809d4d0 con 0x7fc4580b6a20 2024-04-02T09:12:44.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.216+0000 7fc457e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc458160be0 0x7fc45806cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:44.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.216+0000 7fc457e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc458160be0 0x7fc45806cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59644/0 (socket says 172.21.15.89:59644) 2024-04-02T09:12:44.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.216+0000 7fc457e00700 1 -- 172.21.15.89:0/2581636313 learned_addr learned my addr 172.21.15.89:0/2581636313 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:44.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.216+0000 7fc456a00700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4580ce590 0x7fc4580cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:44.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.216+0000 7fc457400700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4580b6a20 0x7fc45815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:44.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.217+0000 7fc457e00700 1 -- 172.21.15.89:0/2581636313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4580b6a20 msgr2=0x7fc45815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:44.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.217+0000 7fc457e00700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4580b6a20 0x7fc45815df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:44.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.217+0000 7fc457e00700 1 -- 172.21.15.89:0/2581636313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4580ce590 msgr2=0x7fc4580cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:44.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.217+0000 7fc457e00700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4580ce590 0x7fc4580cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:44.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.217+0000 7fc457e00700 1 -- 172.21.15.89:0/2581636313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4440094a0 con 0x7fc458160be0 2024-04-02T09:12:44.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.217+0000 7fc456a00700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4580ce590 0x7fc4580cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:44.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.217+0000 7fc457e00700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc458160be0 0x7fc45806cc30 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7fc450007e40 tx=0x7fc45000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:44.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.217+0000 7fc44f400700 1 -- 172.21.15.89:0/2581636313 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45000eea0 con 0x7fc458160be0 2024-04-02T09:12:44.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.217+0000 7fc44f400700 1 -- 172.21.15.89:0/2581636313 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc450014420 con 0x7fc458160be0 2024-04-02T09:12:44.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.217+0000 7fc44f400700 1 -- 172.21.15.89:0/2581636313 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4500147b0 con 0x7fc458160be0 2024-04-02T09:12:44.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.217+0000 7fc45e000700 1 -- 172.21.15.89:0/2581636313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4580cc810 con 0x7fc458160be0 2024-04-02T09:12:44.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.217+0000 7fc45e000700 1 -- 172.21.15.89:0/2581636313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4580ccdd0 con 0x7fc458160be0 2024-04-02T09:12:44.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.219+0000 7fc45e000700 1 -- 172.21.15.89:0/2581636313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc458067720 con 0x7fc458160be0 2024-04-02T09:12:44.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.219+0000 7fc44f400700 1 -- 172.21.15.89:0/2581636313 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc450014910 con 0x7fc458160be0 2024-04-02T09:12:44.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.221+0000 7fc44f400700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc43806adb0 0x7fc43806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:44.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.221+0000 7fc44f400700 1 -- 172.21.15.89:0/2581636313 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fc45008a1e0 con 0x7fc458160be0 2024-04-02T09:12:44.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.221+0000 7fc457400700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc43806adb0 0x7fc43806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:44.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.221+0000 7fc457400700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc43806adb0 0x7fc43806d270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc448005910 tx=0x7fc448009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:44.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.226+0000 7fc44f400700 1 -- 172.21.15.89:0/2581636313 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc450054d90 con 0x7fc458160be0 2024-04-02T09:12:44.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.470+0000 7fc45e000700 1 -- 172.21.15.89:0/2581636313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4580ccf60 con 0x7fc458160be0 2024-04-02T09:12:44.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.471+0000 7fc44f400700 1 -- 172.21.15.89:0/2581636313 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5971 (secure 0 0 0) 0x7fc4500583b0 con 0x7fc458160be0 2024-04-02T09:12:44.470 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:44.470 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":110,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8969,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3650347444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3650347444},{"type":"v1","addr":"172.21.15.89:6836","nonce":3650347444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2152500685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2152500685},{"type":"v1","addr":"172.21.15.113:6834","nonce":2152500685}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:25.170849+0000","modified":"2024-04-02T09:12:28.185307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11157":{"gid":11157,"name":"a","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2615927215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2615927215},{"type":"v1","addr":"172.21.15.89:6837","nonce":2615927215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:39.106317+0000","modified":"2024-04-02T09:12:42.112504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10279":{"gid":10279,"name":"d","rank":0,"incarnation":107,"state":"up:active","state_seq":6,"addr":"172.21.15.113:6835/2457040702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2457040702},{"type":"v1","addr":"172.21.15.113:6835","nonce":2457040702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-02T09:12:44.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.473+0000 7fc45e000700 1 -- 172.21.15.89:0/2581636313 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc43806adb0 msgr2=0x7fc43806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:44.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.473+0000 7fc45e000700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc43806adb0 0x7fc43806d270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc448005910 tx=0x7fc448009040 comp rx=0 tx=0).stop 2024-04-02T09:12:44.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.473+0000 7fc45e000700 1 -- 172.21.15.89:0/2581636313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc458160be0 msgr2=0x7fc45806cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:44.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.473+0000 7fc45e000700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc458160be0 0x7fc45806cc30 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7fc450007e40 tx=0x7fc45000e3c0 comp rx=0 tx=0).stop 2024-04-02T09:12:44.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.473+0000 7fc45e000700 1 -- 172.21.15.89:0/2581636313 shutdown_connections 2024-04-02T09:12:44.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.473+0000 7fc45e000700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc458160be0 0x7fc45806cc30 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:44.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.473+0000 7fc45e000700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4580b6a20 0x7fc45815df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:44.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.473+0000 7fc45e000700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc43806adb0 0x7fc43806d270 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:44.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.473+0000 7fc45e000700 1 --2- 172.21.15.89:0/2581636313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4580ce590 0x7fc4580cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:44.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.473+0000 7fc45e000700 1 -- 172.21.15.89:0/2581636313 >> 172.21.15.89:0/2581636313 conn(0x7fc45808aff0 msgr2=0x7fc45815e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:44.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.473+0000 7fc45e000700 1 -- 172.21.15.89:0/2581636313 shutdown_connections 2024-04-02T09:12:44.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:44.473+0000 7fc45e000700 1 -- 172.21.15.89:0/2581636313 wait complete. 2024-04-02T09:12:44.474 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 110 2024-04-02T09:12:44.893 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 110, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T09:12:39.106317+0000', 'modified': '2024-04-02T09:12:42.112504+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10279}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10279': {'gid': 10279, 'name': 'd', 'rank': 0, 'incarnation': 107, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.113:6835/2457040702', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6832', 'nonce': 2457040702}, {'type': 'v1', 'addr': '172.21.15.113:6835', 'nonce': 2457040702}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T09:12:44.893 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T09:12:44.893 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-02T09:12:45.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.088+0000 7f22b1800700 1 Processor -- start 2024-04-02T09:12:45.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.088+0000 7f22b1800700 1 -- start start 2024-04-02T09:12:45.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.089+0000 7f22b1800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f22ac15dfe0 0x7f22ac0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.089+0000 7f22b1800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22ac0ce670 0x7f22ac0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.089+0000 7f22b1800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ac0cb310 con 0x7f22ac15a5c0 2024-04-02T09:12:45.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.089+0000 7f22b1800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22ac0ceb00 con 0x7f22ac15dfe0 2024-04-02T09:12:45.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.089+0000 7f22b1800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac0711a0 con 0x7f22ac0ce670 2024-04-02T09:12:45.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.090+0000 7f22aaa00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f22ac15a5c0 0x7f22ac0709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:54694/0 (socket says 172.21.15.89:54694) 2024-04-02T09:12:45.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.090+0000 7f22aaa00700 1 -- 172.21.15.89:0/141910265 learned_addr learned my addr 172.21.15.89:0/141910265 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:45.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.090+0000 7f22aa000700 1 --2- 172.21.15.89:0/141910265 >> v2:172.21.15.113:3300/0 conn(0x7f22ac15dfe0 0x7f22ac0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.090+0000 7f22ab400700 1 --2- 172.21.15.89:0/141910265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22ac0ce670 0x7f22ac0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.090+0000 7f22a9600700 1 -- 172.21.15.89:0/141910265 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061467319 0 0) 0x7f22ac0cb310 con 0x7f22ac15a5c0 2024-04-02T09:12:45.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.090+0000 7f22a9600700 1 -- 172.21.15.89:0/141910265 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22900035b0 con 0x7f22ac15a5c0 2024-04-02T09:12:45.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.090+0000 7f22aa000700 1 -- 172.21.15.89:0/141910265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22ac0ce670 msgr2=0x7f22ac0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:45.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.090+0000 7f22aa000700 1 --2- 172.21.15.89:0/141910265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22ac0ce670 0x7f22ac0cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.090+0000 7f22aa000700 1 -- 172.21.15.89:0/141910265 >> v1:172.21.15.89:6789/0 conn(0x7f22ac15a5c0 legacy=0x7f22ac0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.090+0000 7f22aa000700 1 -- 172.21.15.89:0/141910265 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ac071480 con 0x7f22ac15dfe0 2024-04-02T09:12:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.091+0000 7f22ab400700 1 --2- 172.21.15.89:0/141910265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22ac0ce670 0x7f22ac0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:12:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.091+0000 7f22aa000700 1 --2- 172.21.15.89:0/141910265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22ac15dfe0 0x7f22ac0ce130 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2298009840 tx=0x7f2298009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd36492e992755ed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.091+0000 7f22a9600700 1 -- 172.21.15.89:0/141910265 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298011830 con 0x7f22ac15dfe0 2024-04-02T09:12:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.091+0000 7f22a9600700 1 -- 172.21.15.89:0/141910265 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2298011990 con 0x7f22ac15dfe0 2024-04-02T09:12:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.091+0000 7f22a9600700 1 -- 172.21.15.89:0/141910265 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298011d00 con 0x7f22ac15dfe0 2024-04-02T09:12:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.092+0000 7f22b1800700 1 -- 172.21.15.89:0/141910265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22ac15dfe0 msgr2=0x7f22ac0ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.092+0000 7f22b1800700 1 --2- 172.21.15.89:0/141910265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22ac15dfe0 0x7f22ac0ce130 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2298009840 tx=0x7f2298009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.092+0000 7f22b1800700 1 -- 172.21.15.89:0/141910265 shutdown_connections 2024-04-02T09:12:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.092+0000 7f22b1800700 1 --2- 172.21.15.89:0/141910265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22ac0ce670 0x7f22ac0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.092+0000 7f22b1800700 1 --2- 172.21.15.89:0/141910265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22ac15dfe0 0x7f22ac0ce130 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.092+0000 7f22b1800700 1 -- 172.21.15.89:0/141910265 >> 172.21.15.89:0/141910265 conn(0x7f22ac0b0d70 msgr2=0x7f22ac0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:45.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.092+0000 7f22b1800700 1 -- 172.21.15.89:0/141910265 shutdown_connections 2024-04-02T09:12:45.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.092+0000 7f22b1800700 1 -- 172.21.15.89:0/141910265 wait complete. 2024-04-02T09:12:45.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.093+0000 7f22b1800700 1 Processor -- start 2024-04-02T09:12:45.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.094+0000 7f22b1800700 1 -- start start 2024-04-02T09:12:45.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.094+0000 7f22b1800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22ac0ce670 0x7f22ac0c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.094+0000 7f22b1800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22ac15a5c0 0x7f22ac0ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22b1800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22ac15dfe0 0x7f22ac149330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22b1800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac0b0b90 con 0x7f22ac15a5c0 2024-04-02T09:12:45.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22b1800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac0ac590 con 0x7f22ac0ce670 2024-04-02T09:12:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22b1800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac0ac780 con 0x7f22ac15dfe0 2024-04-02T09:12:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22aa000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22ac15a5c0 0x7f22ac0ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22aa000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22ac15a5c0 0x7f22ac0ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59650/0 (socket says 172.21.15.89:59650) 2024-04-02T09:12:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22aa000700 1 -- 172.21.15.89:0/4288303446 learned_addr learned my addr 172.21.15.89:0/4288303446 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22aa000700 1 -- 172.21.15.89:0/4288303446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22ac15dfe0 msgr2=0x7f22ac149330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22ab400700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22ac15dfe0 0x7f22ac149330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22aaa00700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22ac0ce670 0x7f22ac0c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22aa000700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22ac15dfe0 0x7f22ac149330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22aa000700 1 -- 172.21.15.89:0/4288303446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22ac0ce670 msgr2=0x7f22ac0c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22aa000700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22ac0ce670 0x7f22ac0c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22aa000700 1 -- 172.21.15.89:0/4288303446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a4009a70 con 0x7f22ac15a5c0 2024-04-02T09:12:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22aaa00700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22ac0ce670 0x7f22ac0c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.095+0000 7f22aa000700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22ac15a5c0 0x7f22ac0ac050 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f2298009b90 tx=0x7f2298004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:45.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.096+0000 7f22a2a00700 1 -- 172.21.15.89:0/4288303446 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22980036a0 con 0x7f22ac15a5c0 2024-04-02T09:12:45.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.096+0000 7f22a2a00700 1 -- 172.21.15.89:0/4288303446 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229801e070 con 0x7f22ac15a5c0 2024-04-02T09:12:45.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.096+0000 7f22b1800700 1 -- 172.21.15.89:0/4288303446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22980094a0 con 0x7f22ac15a5c0 2024-04-02T09:12:45.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.096+0000 7f22a2a00700 1 -- 172.21.15.89:0/4288303446 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298003a60 con 0x7f22ac15a5c0 2024-04-02T09:12:45.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.096+0000 7f22b1800700 1 -- 172.21.15.89:0/4288303446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ac0b04d0 con 0x7f22ac15a5c0 2024-04-02T09:12:45.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.098+0000 7f22a2a00700 1 -- 172.21.15.89:0/4288303446 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2298003bc0 con 0x7f22ac15a5c0 2024-04-02T09:12:45.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.099+0000 7f22a2a00700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f228c06adb0 0x7f228c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.099+0000 7f22a2a00700 1 -- 172.21.15.89:0/4288303446 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f229808a940 con 0x7f22ac15a5c0 2024-04-02T09:12:45.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.099+0000 7f22aaa00700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f228c06adb0 0x7f228c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.099+0000 7f22aaa00700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f228c06adb0 0x7f228c06d270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f229c005910 tx=0x7f229c009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:45.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.100+0000 7f22c53b3b80 1 Processor -- start 2024-04-02T09:12:45.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.101+0000 7f22c53b3b80 1 -- start start 2024-04-02T09:12:45.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.101+0000 7f22c53b3b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3965830 0x558cd3965c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.101+0000 7f22c53b3b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3938be0 0x558cd393ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.101+0000 7f22c53b3b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3936b60 0x558cd3962a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.101+0000 7f22c53b3b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cd38b0650 con 0x558cd3938be0 2024-04-02T09:12:45.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.101+0000 7f22c53b3b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cd3891c30 con 0x558cd3965830 2024-04-02T09:12:45.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.101+0000 7f22c53b3b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x558cd388f430 con 0x558cd3936b60 2024-04-02T09:12:45.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f22aa000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3938be0 0x558cd393ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f22aa000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3938be0 0x558cd393ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59652/0 (socket says 172.21.15.89:59652) 2024-04-02T09:12:45.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f22aa000700 1 -- 172.21.15.89:0/2101157947 learned_addr learned my addr 172.21.15.89:0/2101157947 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:45.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f22aa000700 1 -- 172.21.15.89:0/2101157947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3936b60 msgr2=0x558cd3962a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:45.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f22aaa00700 1 --2- 172.21.15.89:0/2101157947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3965830 0x558cd3965c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f22ab400700 1 --2- 172.21.15.89:0/2101157947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3936b60 0x558cd3962a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f22aa000700 1 --2- 172.21.15.89:0/2101157947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3936b60 0x558cd3962a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f22aa000700 1 -- 172.21.15.89:0/2101157947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3965830 msgr2=0x558cd3965c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:45.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f22aa000700 1 --2- 172.21.15.89:0/2101157947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3965830 0x558cd3965c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f22aa000700 1 -- 172.21.15.89:0/2101157947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558cd367e790 con 0x558cd3938be0 2024-04-02T09:12:45.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f22aaa00700 1 --2- 172.21.15.89:0/2101157947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3965830 0x558cd3965c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:45.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f22ab400700 1 --2- 172.21.15.89:0/2101157947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3936b60 0x558cd3962a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:45.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f22aa000700 1 --2- 172.21.15.89:0/2101157947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3938be0 0x558cd393ba10 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f229800b270 tx=0x7f2298058b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ee76fc4bd2a20b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:45.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f2297400700 1 -- 172.21.15.89:0/2101157947 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22980598a0 con 0x558cd3938be0 2024-04-02T09:12:45.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f2297400700 1 -- 172.21.15.89:0/2101157947 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22980591a0 con 0x558cd3938be0 2024-04-02T09:12:45.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.102+0000 7f2297400700 1 -- 172.21.15.89:0/2101157947 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229805cfd0 con 0x558cd3938be0 2024-04-02T09:12:45.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.103+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2101157947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3938be0 msgr2=0x558cd393ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.103+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/2101157947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3938be0 0x558cd393ba10 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f229800b270 tx=0x7f2298058b80 comp rx=0 tx=0).stop 2024-04-02T09:12:45.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.103+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2101157947 shutdown_connections 2024-04-02T09:12:45.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.103+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/2101157947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3965830 0x558cd3965c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.103+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/2101157947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3936b60 0x558cd3962a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.103+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/2101157947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3938be0 0x558cd393ba10 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.103+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2101157947 >> 172.21.15.89:0/2101157947 conn(0x558cd38996a0 msgr2=0x558cd3980640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:45.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.103+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2101157947 shutdown_connections 2024-04-02T09:12:45.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.103+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2101157947 wait complete. 2024-04-02T09:12:45.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.104+0000 7f22c53b3b80 1 -- start start 2024-04-02T09:12:45.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.104+0000 7f22c53b3b80 1 Processor -- start 2024-04-02T09:12:45.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.105+0000 7f22c53b3b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3936b60 0x558cd395c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.105+0000 7f22c53b3b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3938be0 0x558cd3968f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.105+0000 7f22c53b3b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3965830 0x558cd3969480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.105+0000 7f22c53b3b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cd388ad20 con 0x558cd3936b60 2024-04-02T09:12:45.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.105+0000 7f22c53b3b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cd3826770 con 0x558cd3965830 2024-04-02T09:12:45.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.105+0000 7f22c53b3b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x558cd38496f0 con 0x558cd3938be0 2024-04-02T09:12:45.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.105+0000 7f22c53b3b80 10 client.0 _ll_register_callbacks cb 0x558cd38305e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T09:12:45.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.105+0000 7f22aaa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3936b60 0x558cd395c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.105+0000 7f22aaa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3936b60 0x558cd395c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59664/0 (socket says 172.21.15.89:59664) 2024-04-02T09:12:45.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.105+0000 7f22aaa00700 1 -- 172.21.15.89:0/2606896985 learned_addr learned my addr 172.21.15.89:0/2606896985 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:45.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.106+0000 7f22aaa00700 1 -- 172.21.15.89:0/2606896985 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3938be0 msgr2=0x558cd3968f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:45.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.106+0000 7f22aaa00700 1 --2- 172.21.15.89:0/2606896985 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3938be0 0x558cd3968f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.106+0000 7f22aaa00700 1 -- 172.21.15.89:0/2606896985 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3965830 msgr2=0x558cd3969480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:45.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.106+0000 7f22aaa00700 1 --2- 172.21.15.89:0/2606896985 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3965830 0x558cd3969480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.106+0000 7f22aaa00700 1 -- 172.21.15.89:0/2606896985 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ac0b04d0 con 0x558cd3936b60 2024-04-02T09:12:45.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.106+0000 7f22aaa00700 1 --2- 172.21.15.89:0/2606896985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3936b60 0x558cd395c4e0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f229c007380 tx=0x7f229c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bcae7d05d481886 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:45.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.106+0000 7f228be00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T09:12:45.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.106+0000 7f228be00700 1 -- 172.21.15.89:0/2606896985 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c00de70 con 0x558cd3936b60 2024-04-02T09:12:45.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.106+0000 7f228be00700 1 -- 172.21.15.89:0/2606896985 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f229c021430 con 0x558cd3936b60 2024-04-02T09:12:45.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.106+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2606896985 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558cd392d920 con 0x558cd3936b60 2024-04-02T09:12:45.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.107+0000 7f228be00700 1 -- 172.21.15.89:0/2606896985 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f229c0077f0 con 0x558cd3936b60 2024-04-02T09:12:45.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.107+0000 7f22c53b3b80 10 client.11436 fetch_fsmap learned FSMap version 110 2024-04-02T09:12:45.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.107+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2606896985 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=2+}) v3 -- 0x558cd376b0a0 con 0x558cd3936b60 2024-04-02T09:12:45.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.107+0000 7f228be00700 1 -- 172.21.15.89:0/2606896985 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558cd392d920 con 0x558cd3936b60 2024-04-02T09:12:45.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.107+0000 7f228be00700 1 -- 172.21.15.89:0/2606896985 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 110) v1 ==== 3681+0+0 (secure 0 0 0) 0x7f229c0215f0 con 0x558cd3936b60 2024-04-02T09:12:45.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.107+0000 7f22c53b3b80 10 client.11436 fetch_fsmap finished waiting for FSMap version 110 2024-04-02T09:12:45.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.107+0000 7f22c53b3b80 10 client.11436 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-02T09:12:45.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.107+0000 7f22c53b3b80 20 client.11436 populate_metadata read hostname 'smithi089' 2024-04-02T09:12:45.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.107+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/2606896985 >> [v2:172.21.15.113:6832/2457040702,v1:172.21.15.113:6835/2457040702] conn(0x558cd3967640 0x558cd3967a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.107+0000 7f22c53b3b80 4 client.11436 mds_command: new command op to 10279 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T09:12:45.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.107+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2606896985 --> [v2:172.21.15.113:6832/2457040702,v1:172.21.15.113:6835/2457040702] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558cd390a6f0 con 0x558cd3967640 2024-04-02T09:12:45.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.107+0000 7f22aa000700 1 --2- 172.21.15.89:0/2606896985 >> [v2:172.21.15.113:6832/2457040702,v1:172.21.15.113:6835/2457040702] conn(0x558cd3967640 0x558cd3967a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.108+0000 7f228be00700 10 client.11436 ms_handle_connect on v2:172.21.15.113:6832/2457040702 2024-04-02T09:12:45.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.108+0000 7f22aa000700 1 --2- 172.21.15.89:0/2606896985 >> [v2:172.21.15.113:6832/2457040702,v1:172.21.15.113:6835/2457040702] conn(0x558cd3967640 0x558cd3967a20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b78b293544794a server_cookie=62ee976ab074f202 in_seq=0 out_seq=0 2024-04-02T09:12:45.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.109+0000 7f228be00700 1 -- 172.21.15.89:0/2606896985 <== mds.0 v2:172.21.15.113:6832/2457040702 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558cd390a6f0 con 0x558cd3967640 2024-04-02T09:12:45.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.109+0000 7f228be00700 10 client.11436 handle_command_reply: tid=0 2024-04-02T09:12:45.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.109+0000 7f22c53b3b80 1 client.11436 shutdown 2024-04-02T09:12:45.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.109+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2606896985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3936b60 msgr2=0x558cd395c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.109+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/2606896985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3936b60 0x558cd395c4e0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f229c007380 tx=0x7f229c000bc0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.109+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2606896985 shutdown_connections 2024-04-02T09:12:45.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.109+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/2606896985 >> [v2:172.21.15.113:6832/2457040702,v1:172.21.15.113:6835/2457040702] conn(0x558cd3967640 0x558cd3967a20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.109+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/2606896985 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3965830 0x558cd3969480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.109+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/2606896985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3936b60 0x558cd395c4e0 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.109+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/2606896985 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3938be0 0x558cd3968f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.109+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2606896985 >> 172.21.15.89:0/2606896985 conn(0x558cd38996a0 msgr2=0x558cd397fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:45.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.109+0000 7f228be00700 0 client.11436 ms_handle_reset on v2:172.21.15.113:6832/2457040702 2024-04-02T09:12:45.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.109+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2606896985 shutdown_connections 2024-04-02T09:12:45.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.109+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2606896985 wait complete. 2024-04-02T09:12:45.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.110+0000 7f22c53b3b80 20 client.11436 trim_cache size 0 max 16384 2024-04-02T09:12:45.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.128+0000 7f22c53b3b80 1 Processor -- start 2024-04-02T09:12:45.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.128+0000 7f22c53b3b80 1 -- start start 2024-04-02T09:12:45.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.129+0000 7f22c53b3b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3936b60 0x558cd393c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.129+0000 7f22c53b3b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3938be0 0x558cd39357a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.129+0000 7f22c53b3b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3965830 0x558cd39345a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.129+0000 7f22c53b3b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cd367f390 con 0x558cd3965830 2024-04-02T09:12:45.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.129+0000 7f22c53b3b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cd395b710 con 0x558cd3936b60 2024-04-02T09:12:45.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.129+0000 7f22c53b3b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x558cd3984900 con 0x558cd3938be0 2024-04-02T09:12:45.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.129+0000 7f22ab400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3965830 0x558cd39345a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.129+0000 7f22ab400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3965830 0x558cd39345a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59670/0 (socket says 172.21.15.89:59670) 2024-04-02T09:12:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.129+0000 7f22ab400700 1 -- 172.21.15.89:0/2209529065 learned_addr learned my addr 172.21.15.89:0/2209529065 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.129+0000 7f22aaa00700 1 --2- 172.21.15.89:0/2209529065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3936b60 0x558cd393c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.129+0000 7f22aa000700 1 --2- 172.21.15.89:0/2209529065 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3938be0 0x558cd39357a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.130+0000 7f22ab400700 1 -- 172.21.15.89:0/2209529065 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3938be0 msgr2=0x558cd39357a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.130+0000 7f22ab400700 1 --2- 172.21.15.89:0/2209529065 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3938be0 0x558cd39357a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.130+0000 7f22ab400700 1 -- 172.21.15.89:0/2209529065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3936b60 msgr2=0x558cd393c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.130+0000 7f22ab400700 1 --2- 172.21.15.89:0/2209529065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3936b60 0x558cd393c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.130+0000 7f22ab400700 1 -- 172.21.15.89:0/2209529065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558cd3851010 con 0x558cd3965830 2024-04-02T09:12:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.130+0000 7f22aa000700 1 --2- 172.21.15.89:0/2209529065 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3938be0 0x558cd39357a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.130+0000 7f22aaa00700 1 --2- 172.21.15.89:0/2209529065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3936b60 0x558cd393c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.130+0000 7f22ab400700 1 --2- 172.21.15.89:0/2209529065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3965830 0x558cd39345a0 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f22a4003c30 tx=0x7f22a4003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e001dc803462f4e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.130+0000 7f2297e00700 1 -- 172.21.15.89:0/2209529065 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a4004890 con 0x558cd3965830 2024-04-02T09:12:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.130+0000 7f2297e00700 1 -- 172.21.15.89:0/2209529065 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a4015420 con 0x558cd3965830 2024-04-02T09:12:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.130+0000 7f2297e00700 1 -- 172.21.15.89:0/2209529065 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a40157d0 con 0x558cd3965830 2024-04-02T09:12:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.131+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2209529065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3965830 msgr2=0x558cd39345a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.131+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/2209529065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3965830 0x558cd39345a0 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f22a4003c30 tx=0x7f22a4003f40 comp rx=0 tx=0).stop 2024-04-02T09:12:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.131+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2209529065 shutdown_connections 2024-04-02T09:12:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.131+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/2209529065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3936b60 0x558cd393c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.131+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/2209529065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3965830 0x558cd39345a0 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.131+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/2209529065 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3938be0 0x558cd39357a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.131+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2209529065 >> 172.21.15.89:0/2209529065 conn(0x558cd38996a0 msgr2=0x558cd3931880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.131+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2209529065 shutdown_connections 2024-04-02T09:12:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.131+0000 7f22c53b3b80 1 -- 172.21.15.89:0/2209529065 wait complete. 2024-04-02T09:12:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.131+0000 7f22c53b3b80 1 -- start start 2024-04-02T09:12:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.132+0000 7f22c53b3b80 1 Processor -- start 2024-04-02T09:12:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.132+0000 7f22c53b3b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3936b60 0x558cd39e6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.132+0000 7f22c53b3b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3938be0 0x558cd397cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f22c53b3b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3965830 0x558cd39676b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f22c53b3b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cd3939ad0 con 0x558cd3965830 2024-04-02T09:12:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f22c53b3b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cd39e6b80 con 0x558cd3936b60 2024-04-02T09:12:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f22c53b3b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x558cd3997b10 con 0x558cd3938be0 2024-04-02T09:12:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f22c53b3b80 10 client.0 _ll_register_callbacks cb 0x7f229c005d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T09:12:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f22ab400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3965830 0x558cd39676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f22ab400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3965830 0x558cd39676b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59684/0 (socket says 172.21.15.89:59684) 2024-04-02T09:12:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f22ab400700 1 -- 172.21.15.89:0/3274877310 learned_addr learned my addr 172.21.15.89:0/3274877310 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f22ab400700 1 -- 172.21.15.89:0/3274877310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3938be0 msgr2=0x558cd397cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f22ab400700 1 --2- 172.21.15.89:0/3274877310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3938be0 0x558cd397cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f22ab400700 1 -- 172.21.15.89:0/3274877310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3936b60 msgr2=0x558cd39e6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f22ab400700 1 --2- 172.21.15.89:0/3274877310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3936b60 0x558cd39e6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f22ab400700 1 -- 172.21.15.89:0/3274877310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a4011040 con 0x558cd3965830 2024-04-02T09:12:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f22ab400700 1 --2- 172.21.15.89:0/3274877310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3965830 0x558cd39676b0 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f22a4000f80 tx=0x7f22a400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71712f0db5cd3f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.133+0000 7f228be00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T09:12:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.134+0000 7f228be00700 1 -- 172.21.15.89:0/3274877310 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a4015750 con 0x558cd3965830 2024-04-02T09:12:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.134+0000 7f228be00700 1 -- 172.21.15.89:0/3274877310 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f22a40158b0 con 0x558cd3965830 2024-04-02T09:12:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.134+0000 7f22c53b3b80 1 -- 172.21.15.89:0/3274877310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22ac0af850 con 0x558cd3965830 2024-04-02T09:12:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.134+0000 7f228be00700 1 -- 172.21.15.89:0/3274877310 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22a4015a90 con 0x558cd3965830 2024-04-02T09:12:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.134+0000 7f22c53b3b80 10 client.11442 fetch_fsmap learned FSMap version 110 2024-04-02T09:12:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.134+0000 7f22c53b3b80 1 -- 172.21.15.89:0/3274877310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=2+}) v3 -- 0x558cd36df1d0 con 0x558cd3965830 2024-04-02T09:12:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.134+0000 7f228be00700 1 -- 172.21.15.89:0/3274877310 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ac0af850 con 0x558cd3965830 2024-04-02T09:12:45.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.134+0000 7f228be00700 1 -- 172.21.15.89:0/3274877310 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 110) v1 ==== 3681+0+0 (secure 0 0 0) 0x7f22a4015bf0 con 0x558cd3965830 2024-04-02T09:12:45.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.134+0000 7f22c53b3b80 10 client.11442 fetch_fsmap finished waiting for FSMap version 110 2024-04-02T09:12:45.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.134+0000 7f22c53b3b80 10 client.11442 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-02T09:12:45.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.134+0000 7f22c53b3b80 20 client.11442 populate_metadata read hostname 'smithi089' 2024-04-02T09:12:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.134+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/3274877310 >> [v2:172.21.15.113:6832/2457040702,v1:172.21.15.113:6835/2457040702] conn(0x558cd397e3e0 0x558cd397e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.134+0000 7f22c53b3b80 4 client.11442 mds_command: new command op to 10279 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T09:12:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.134+0000 7f22c53b3b80 1 -- 172.21.15.89:0/3274877310 --> [v2:172.21.15.113:6832/2457040702,v1:172.21.15.113:6835/2457040702] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f22980591a0 con 0x558cd397e3e0 2024-04-02T09:12:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.135+0000 7f22aaa00700 1 --2- 172.21.15.89:0/3274877310 >> [v2:172.21.15.113:6832/2457040702,v1:172.21.15.113:6835/2457040702] conn(0x558cd397e3e0 0x558cd397e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.135+0000 7f22aaa00700 1 --2- 172.21.15.89:0/3274877310 >> [v2:172.21.15.113:6832/2457040702,v1:172.21.15.113:6835/2457040702] conn(0x558cd397e3e0 0x558cd397e7c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f955553ebd7051b server_cookie=93a77fa764bfffeb in_seq=0 out_seq=0 2024-04-02T09:12:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.135+0000 7f228be00700 10 client.11442 ms_handle_connect on v2:172.21.15.113:6832/2457040702 2024-04-02T09:12:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.135+0000 7f228be00700 1 -- 172.21.15.89:0/3274877310 <== mds.0 v2:172.21.15.113:6832/2457040702 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+338 (crc 0 0 0) 0x7f22980591a0 con 0x558cd397e3e0 2024-04-02T09:12:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.135+0000 7f228be00700 10 client.11442 handle_command_reply: tid=0 2024-04-02T09:12:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.135+0000 7f22c53b3b80 1 client.11442 shutdown 2024-04-02T09:12:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.136+0000 7f22c53b3b80 1 -- 172.21.15.89:0/3274877310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3965830 msgr2=0x558cd39676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.136+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/3274877310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3965830 0x558cd39676b0 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f22a4000f80 tx=0x7f22a400b040 comp rx=0 tx=0).stop 2024-04-02T09:12:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.136+0000 7f22c53b3b80 1 -- 172.21.15.89:0/3274877310 shutdown_connections 2024-04-02T09:12:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.136+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/3274877310 >> [v2:172.21.15.113:6832/2457040702,v1:172.21.15.113:6835/2457040702] conn(0x558cd397e3e0 0x558cd397e7c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.136+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/3274877310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558cd3936b60 0x558cd39e6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.136+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/3274877310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558cd3965830 0x558cd39676b0 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.136+0000 7f22c53b3b80 1 --2- 172.21.15.89:0/3274877310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x558cd3938be0 0x558cd397cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.136+0000 7f228be00700 0 client.11442 ms_handle_reset on v2:172.21.15.113:6832/2457040702 2024-04-02T09:12:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.136+0000 7f22c53b3b80 1 -- 172.21.15.89:0/3274877310 >> 172.21.15.89:0/3274877310 conn(0x558cd38996a0 msgr2=0x558cd39345a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.136+0000 7f22c53b3b80 1 -- 172.21.15.89:0/3274877310 shutdown_connections 2024-04-02T09:12:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.136+0000 7f22c53b3b80 1 -- 172.21.15.89:0/3274877310 wait complete. 2024-04-02T09:12:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.136+0000 7f22c53b3b80 20 client.11442 trim_cache size 0 max 16384 2024-04-02T09:12:45.139 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:12:45.139 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T09:12:45.139 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T09:12:45.140 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 10279, 2024-04-02T09:12:45.140 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T09:12:45.140 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T09:12:45.140 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "backup_fs", 2024-04-02T09:12:45.140 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 6.0273181679999999, 2024-04-02T09:12:45.140 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 110, 2024-04-02T09:12:45.140 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 227, 2024-04-02T09:12:45.140 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 227, 2024-04-02T09:12:45.140 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 25.417267560999999 2024-04-02T09:12:45.140 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:12:45.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.137+0000 7f22b1800700 1 -- 172.21.15.89:0/4288303446 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f228c06adb0 msgr2=0x7f228c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.137+0000 7f22b1800700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f228c06adb0 0x7f228c06d270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f229c005910 tx=0x7f229c009040 comp rx=0 tx=0).stop 2024-04-02T09:12:45.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.137+0000 7f22b1800700 1 -- 172.21.15.89:0/4288303446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22ac15a5c0 msgr2=0x7f22ac0ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.137+0000 7f22b1800700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22ac15a5c0 0x7f22ac0ac050 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f2298009b90 tx=0x7f2298004750 comp rx=0 tx=0).stop 2024-04-02T09:12:45.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.137+0000 7f22b1800700 1 -- 172.21.15.89:0/4288303446 shutdown_connections 2024-04-02T09:12:45.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.137+0000 7f22b1800700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f228c06adb0 0x7f228c06d270 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.137+0000 7f22b1800700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22ac0ce670 0x7f22ac0c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.137+0000 7f22b1800700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22ac15dfe0 0x7f22ac149330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.137+0000 7f22b1800700 1 --2- 172.21.15.89:0/4288303446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22ac15a5c0 0x7f22ac0ac050 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.137+0000 7f22b1800700 1 -- 172.21.15.89:0/4288303446 >> 172.21.15.89:0/4288303446 conn(0x7f22ac0b0d70 msgr2=0x7f22ac0b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:45.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.137+0000 7f22b1800700 1 -- 172.21.15.89:0/4288303446 shutdown_connections 2024-04-02T09:12:45.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.137+0000 7f22b1800700 1 -- 172.21.15.89:0/4288303446 wait complete. 2024-04-02T09:12:45.154 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T09:12:45.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.347+0000 7fedaf600700 1 Processor -- start 2024-04-02T09:12:45.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.348+0000 7fedaf600700 1 -- start start 2024-04-02T09:12:45.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.349+0000 7fedaf600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feda8157f90 0x7feda80b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.349+0000 7fedaf600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7feda81613e0 0x7feda8161860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.349+0000 7fedaf600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda80b3070 con 0x7feda80b3420 2024-04-02T09:12:45.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.349+0000 7fedaf600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feda8158370 con 0x7feda81613e0 2024-04-02T09:12:45.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.349+0000 7fedaf600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda80b2710 con 0x7feda8157f90 2024-04-02T09:12:45.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.349+0000 7feda7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7feda81613e0 0x7feda8161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.349+0000 7feda7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7feda81613e0 0x7feda8161860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59698/0 (socket says 172.21.15.89:59698) 2024-04-02T09:12:45.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.349+0000 7feda7e00700 1 -- 172.21.15.89:0/3051278209 learned_addr learned my addr 172.21.15.89:0/3051278209 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:45.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.349+0000 7fedacc00700 1 --2- 172.21.15.89:0/3051278209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feda8157f90 0x7feda80b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.349+0000 7feda7e00700 1 -- 172.21.15.89:0/3051278209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feda8157f90 msgr2=0x7feda80b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:45.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.349+0000 7feda7e00700 1 --2- 172.21.15.89:0/3051278209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feda8157f90 0x7feda80b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.349+0000 7feda7e00700 1 -- 172.21.15.89:0/3051278209 >> v1:172.21.15.113:6789/0 conn(0x7feda80b3420 legacy=0x7feda80cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.349+0000 7feda7e00700 1 -- 172.21.15.89:0/3051278209 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda80b2070 con 0x7feda81613e0 2024-04-02T09:12:45.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.349+0000 7fedacc00700 1 --2- 172.21.15.89:0/3051278209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feda8157f90 0x7feda80b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:45.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.350+0000 7feda7e00700 1 --2- 172.21.15.89:0/3051278209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feda81613e0 0x7feda8161860 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7fed940098f0 tx=0x7fed94009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc562c5d299fcbf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:45.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.350+0000 7feda7400700 1 -- 172.21.15.89:0/3051278209 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9400b8a0 con 0x7feda81613e0 2024-04-02T09:12:45.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.350+0000 7feda7400700 1 -- 172.21.15.89:0/3051278209 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9400ba00 con 0x7feda81613e0 2024-04-02T09:12:45.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.350+0000 7feda7400700 1 -- 172.21.15.89:0/3051278209 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9400bd90 con 0x7feda81613e0 2024-04-02T09:12:45.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.350+0000 7fedaf600700 1 -- 172.21.15.89:0/3051278209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feda81613e0 msgr2=0x7feda8161860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.350+0000 7fedaf600700 1 --2- 172.21.15.89:0/3051278209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feda81613e0 0x7feda8161860 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7fed940098f0 tx=0x7fed94009c00 comp rx=0 tx=0).stop 2024-04-02T09:12:45.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.351+0000 7fedaf600700 1 -- 172.21.15.89:0/3051278209 shutdown_connections 2024-04-02T09:12:45.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.351+0000 7fedaf600700 1 --2- 172.21.15.89:0/3051278209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feda8157f90 0x7feda80b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.351+0000 7fedaf600700 1 --2- 172.21.15.89:0/3051278209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feda81613e0 0x7feda8161860 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.351+0000 7fedaf600700 1 -- 172.21.15.89:0/3051278209 >> 172.21.15.89:0/3051278209 conn(0x7feda80b0d70 msgr2=0x7feda80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:45.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.351+0000 7fedaf600700 1 -- 172.21.15.89:0/3051278209 shutdown_connections 2024-04-02T09:12:45.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.351+0000 7fedaf600700 1 -- 172.21.15.89:0/3051278209 wait complete. 2024-04-02T09:12:45.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.352+0000 7fedaf600700 1 Processor -- start 2024-04-02T09:12:45.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.352+0000 7fedaf600700 1 -- start start 2024-04-02T09:12:45.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.353+0000 7fedaf600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feda80b3420 0x7feda809cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.353+0000 7fedaf600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feda8157f90 0x7feda8145cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.353+0000 7fedaf600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feda81613e0 0x7feda80b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.353+0000 7fedaf600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda809c880 con 0x7feda8157f90 2024-04-02T09:12:45.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.353+0000 7fedaf600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda809bc70 con 0x7feda80b3420 2024-04-02T09:12:45.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.353+0000 7fedaf600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda809bf50 con 0x7feda81613e0 2024-04-02T09:12:45.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.353+0000 7feda7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feda8157f90 0x7feda8145cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.354+0000 7feda7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feda8157f90 0x7feda8145cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59712/0 (socket says 172.21.15.89:59712) 2024-04-02T09:12:45.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.354+0000 7feda7e00700 1 -- 172.21.15.89:0/2471463928 learned_addr learned my addr 172.21.15.89:0/2471463928 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:45.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.354+0000 7fedacc00700 1 --2- 172.21.15.89:0/2471463928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feda80b3420 0x7feda809cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.354+0000 7fedad600700 1 --2- 172.21.15.89:0/2471463928 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feda81613e0 0x7feda80b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.354+0000 7feda7e00700 1 -- 172.21.15.89:0/2471463928 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feda81613e0 msgr2=0x7feda80b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.354+0000 7feda7e00700 1 --2- 172.21.15.89:0/2471463928 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feda81613e0 0x7feda80b0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.354+0000 7feda7e00700 1 -- 172.21.15.89:0/2471463928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feda80b3420 msgr2=0x7feda809cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.354+0000 7feda7e00700 1 --2- 172.21.15.89:0/2471463928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feda80b3420 0x7feda809cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.354+0000 7feda7e00700 1 -- 172.21.15.89:0/2471463928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed94009550 con 0x7feda8157f90 2024-04-02T09:12:45.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.354+0000 7feda7e00700 1 --2- 172.21.15.89:0/2471463928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feda8157f90 0x7feda8145cc0 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7fed94009c40 tx=0x7fed94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:45.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.354+0000 7feda4c00700 1 -- 172.21.15.89:0/2471463928 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed940036a0 con 0x7feda8157f90 2024-04-02T09:12:45.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.354+0000 7feda4c00700 1 -- 172.21.15.89:0/2471463928 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed94003800 con 0x7feda8157f90 2024-04-02T09:12:45.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.354+0000 7feda4c00700 1 -- 172.21.15.89:0/2471463928 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed94003bb0 con 0x7feda8157f90 2024-04-02T09:12:45.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.354+0000 7fedaf600700 1 -- 172.21.15.89:0/2471463928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda809c230 con 0x7feda8157f90 2024-04-02T09:12:45.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.355+0000 7fedaf600700 1 -- 172.21.15.89:0/2471463928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda8146200 con 0x7feda8157f90 2024-04-02T09:12:45.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.356+0000 7fedaf600700 1 -- 172.21.15.89:0/2471463928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda809ca30 con 0x7feda8157f90 2024-04-02T09:12:45.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.357+0000 7feda4c00700 1 -- 172.21.15.89:0/2471463928 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fed94003d10 con 0x7feda8157f90 2024-04-02T09:12:45.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.357+0000 7feda4c00700 1 --2- 172.21.15.89:0/2471463928 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed8806adb0 0x7fed8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.357+0000 7feda4c00700 1 -- 172.21.15.89:0/2471463928 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fed9408b1a0 con 0x7feda8157f90 2024-04-02T09:12:45.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.358+0000 7fedacc00700 1 --2- 172.21.15.89:0/2471463928 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed8806adb0 0x7fed8806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.358+0000 7fedacc00700 1 --2- 172.21.15.89:0/2471463928 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed8806adb0 0x7fed8806d270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed9c009930 tx=0x7fed9c008040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:45.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.362+0000 7feda4c00700 1 -- 172.21.15.89:0/2471463928 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feda809ca30 con 0x7feda8157f90 2024-04-02T09:12:45.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.591+0000 7fedaf600700 1 -- 172.21.15.89:0/2471463928 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7feda8162230 con 0x7fed8806adb0 2024-04-02T09:12:45.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.593+0000 7feda4c00700 1 -- 172.21.15.89:0/2471463928 <== mgr.10923 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7fed9c0095c0 con 0x7fed8806adb0 2024-04-02T09:12:45.592 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T09:12:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T09:12:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:12:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:12:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "backup_fs" 2024-04-02T09:12:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:12:45.593 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T09:12:45.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.595+0000 7fedaf600700 1 -- 172.21.15.89:0/2471463928 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed8806adb0 msgr2=0x7fed8806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.595+0000 7fedaf600700 1 --2- 172.21.15.89:0/2471463928 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed8806adb0 0x7fed8806d270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed9c009930 tx=0x7fed9c008040 comp rx=0 tx=0).stop 2024-04-02T09:12:45.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.595+0000 7fedaf600700 1 -- 172.21.15.89:0/2471463928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feda8157f90 msgr2=0x7feda8145cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.595+0000 7fedaf600700 1 --2- 172.21.15.89:0/2471463928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feda8157f90 0x7feda8145cc0 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7fed94009c40 tx=0x7fed94004750 comp rx=0 tx=0).stop 2024-04-02T09:12:45.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.595+0000 7fedaf600700 1 -- 172.21.15.89:0/2471463928 shutdown_connections 2024-04-02T09:12:45.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.595+0000 7fedaf600700 1 --2- 172.21.15.89:0/2471463928 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed8806adb0 0x7fed8806d270 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.595+0000 7fedaf600700 1 --2- 172.21.15.89:0/2471463928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feda80b3420 0x7feda809cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.595+0000 7fedaf600700 1 --2- 172.21.15.89:0/2471463928 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feda81613e0 0x7feda80b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.595+0000 7fedaf600700 1 --2- 172.21.15.89:0/2471463928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feda8157f90 0x7feda8145cc0 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.595+0000 7fedaf600700 1 -- 172.21.15.89:0/2471463928 >> 172.21.15.89:0/2471463928 conn(0x7feda80b0d70 msgr2=0x7feda8160830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:45.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.596+0000 7fedaf600700 1 -- 172.21.15.89:0/2471463928 shutdown_connections 2024-04-02T09:12:45.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.596+0000 7fedaf600700 1 -- 172.21.15.89:0/2471463928 wait complete. 2024-04-02T09:12:45.613 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T09:12:45.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.808+0000 7f69c5400700 1 Processor -- start 2024-04-02T09:12:45.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.808+0000 7f69c5400700 1 -- start start 2024-04-02T09:12:45.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.809+0000 7f69c5400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c00d16a0 0x7f69c00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.809+0000 7f69c5400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f69c00d2370 0x7f69c00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.810+0000 7f69c5400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c0161420 con 0x7f69c015d9d0 2024-04-02T09:12:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.810+0000 7f69c5400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c00d2820 con 0x7f69c00d2370 2024-04-02T09:12:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.810+0000 7f69c5400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c00d4340 con 0x7f69c00d16a0 2024-04-02T09:12:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.810+0000 7f69be800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f69c015d9d0 0x7f69c00d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:54706/0 (socket says 172.21.15.89:54706) 2024-04-02T09:12:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.810+0000 7f69be800700 1 -- 172.21.15.89:0/4277877295 learned_addr learned my addr 172.21.15.89:0/4277877295 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.810+0000 7f69bf200700 1 --2- 172.21.15.89:0/4277877295 >> v2:172.21.15.113:3300/0 conn(0x7f69c00d2370 0x7f69c00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.810+0000 7f69b7e00700 1 --2- 172.21.15.89:0/4277877295 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c00d16a0 0x7f69c00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.810+0000 7f69bde00700 1 -- 172.21.15.89:0/4277877295 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733216412 0 0) 0x7f69c0161420 con 0x7f69c015d9d0 2024-04-02T09:12:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.810+0000 7f69bde00700 1 -- 172.21.15.89:0/4277877295 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69a40035b0 con 0x7f69c015d9d0 2024-04-02T09:12:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.810+0000 7f69bf200700 1 -- 172.21.15.89:0/4277877295 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c00d16a0 msgr2=0x7f69c00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.810+0000 7f69bf200700 1 --2- 172.21.15.89:0/4277877295 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c00d16a0 0x7f69c00d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.810+0000 7f69bf200700 1 -- 172.21.15.89:0/4277877295 >> v1:172.21.15.89:6789/0 conn(0x7f69c015d9d0 legacy=0x7f69c00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.811+0000 7f69bf200700 1 -- 172.21.15.89:0/4277877295 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c0161700 con 0x7f69c00d2370 2024-04-02T09:12:45.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.811+0000 7f69b7e00700 1 --2- 172.21.15.89:0/4277877295 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c00d16a0 0x7f69c00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:12:45.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.811+0000 7f69bf200700 1 --2- 172.21.15.89:0/4277877295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c00d2370 0x7f69c00ce360 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009dc0 tx=0x7f69b800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6428051d6e31005c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:45.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.811+0000 7f69bde00700 1 -- 172.21.15.89:0/4277877295 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b800e070 con 0x7f69c00d2370 2024-04-02T09:12:45.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.811+0000 7f69bde00700 1 -- 172.21.15.89:0/4277877295 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b800bae0 con 0x7f69c00d2370 2024-04-02T09:12:45.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.811+0000 7f69bde00700 1 -- 172.21.15.89:0/4277877295 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b800be50 con 0x7f69c00d2370 2024-04-02T09:12:45.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.812+0000 7f69c5400700 1 -- 172.21.15.89:0/4277877295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c00d2370 msgr2=0x7f69c00ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.812+0000 7f69c5400700 1 --2- 172.21.15.89:0/4277877295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c00d2370 0x7f69c00ce360 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009dc0 tx=0x7f69b800b330 comp rx=0 tx=0).stop 2024-04-02T09:12:45.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.812+0000 7f69c5400700 1 -- 172.21.15.89:0/4277877295 shutdown_connections 2024-04-02T09:12:45.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.812+0000 7f69c5400700 1 --2- 172.21.15.89:0/4277877295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c00d2370 0x7f69c00ce360 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.812+0000 7f69c5400700 1 --2- 172.21.15.89:0/4277877295 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c00d16a0 0x7f69c00d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.812+0000 7f69c5400700 1 -- 172.21.15.89:0/4277877295 >> 172.21.15.89:0/4277877295 conn(0x7f69c008aff0 msgr2=0x7f69c00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:45.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.812+0000 7f69c5400700 1 -- 172.21.15.89:0/4277877295 shutdown_connections 2024-04-02T09:12:45.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.812+0000 7f69c5400700 1 -- 172.21.15.89:0/4277877295 wait complete. 2024-04-02T09:12:45.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.813+0000 7f69c5400700 1 Processor -- start 2024-04-02T09:12:45.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.814+0000 7f69c5400700 1 -- start start 2024-04-02T09:12:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.814+0000 7f69c5400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c00d16a0 0x7f69c015ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.814+0000 7f69c5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c00d2370 0x7f69c00b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.814+0000 7f69c5400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c015d9d0 0x7f69c0158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.814+0000 7f69c5400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c00b92f0 con 0x7f69c00d2370 2024-04-02T09:12:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.814+0000 7f69c5400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c01590c0 con 0x7f69c015d9d0 2024-04-02T09:12:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.814+0000 7f69c5400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c015f5a0 con 0x7f69c00d16a0 2024-04-02T09:12:45.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.815+0000 7f69b7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c00d2370 0x7f69c00b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.815+0000 7f69b7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c00d2370 0x7f69c00b8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59714/0 (socket says 172.21.15.89:59714) 2024-04-02T09:12:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.815+0000 7f69b7e00700 1 -- 172.21.15.89:0/3122193641 learned_addr learned my addr 172.21.15.89:0/3122193641 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.815+0000 7f69be800700 1 --2- 172.21.15.89:0/3122193641 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c00d16a0 0x7f69c015ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.815+0000 7f69bf200700 1 --2- 172.21.15.89:0/3122193641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c015d9d0 0x7f69c0158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.815+0000 7f69b7e00700 1 -- 172.21.15.89:0/3122193641 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c00d16a0 msgr2=0x7f69c015ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.815+0000 7f69b7e00700 1 --2- 172.21.15.89:0/3122193641 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c00d16a0 0x7f69c015ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.815+0000 7f69b7e00700 1 -- 172.21.15.89:0/3122193641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c015d9d0 msgr2=0x7f69c0158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.815+0000 7f69b7e00700 1 --2- 172.21.15.89:0/3122193641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c015d9d0 0x7f69c0158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.815+0000 7f69b7e00700 1 -- 172.21.15.89:0/3122193641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a8009430 con 0x7f69c00d2370 2024-04-02T09:12:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.815+0000 7f69b7e00700 1 --2- 172.21.15.89:0/3122193641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c00d2370 0x7f69c00b8db0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f69a800ebe0 tx=0x7f69a800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.816+0000 7f69b6a00700 1 -- 172.21.15.89:0/3122193641 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8009780 con 0x7f69c00d2370 2024-04-02T09:12:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.816+0000 7f69b6a00700 1 -- 172.21.15.89:0/3122193641 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a8005a20 con 0x7f69c00d2370 2024-04-02T09:12:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.816+0000 7f69b6a00700 1 -- 172.21.15.89:0/3122193641 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8005dd0 con 0x7f69c00d2370 2024-04-02T09:12:45.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.816+0000 7f69c5400700 1 -- 172.21.15.89:0/3122193641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b8009a70 con 0x7f69c00d2370 2024-04-02T09:12:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.816+0000 7f69c5400700 1 -- 172.21.15.89:0/3122193641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c015fb60 con 0x7f69c00d2370 2024-04-02T09:12:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.817+0000 7f69c5400700 1 -- 172.21.15.89:0/3122193641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c0160490 con 0x7f69c00d2370 2024-04-02T09:12:45.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.818+0000 7f69b6a00700 1 -- 172.21.15.89:0/3122193641 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 146) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f69a800c850 con 0x7f69c00d2370 2024-04-02T09:12:45.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.819+0000 7f69b6a00700 1 --2- 172.21.15.89:0/3122193641 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f699c06adb0 0x7f699c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:45.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.819+0000 7f69b6a00700 1 -- 172.21.15.89:0/3122193641 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f69a8089be0 con 0x7f69c00d2370 2024-04-02T09:12:45.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.819+0000 7f69be800700 1 --2- 172.21.15.89:0/3122193641 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f699c06adb0 0x7f699c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:45.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.819+0000 7f69be800700 1 --2- 172.21.15.89:0/3122193641 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f699c06adb0 0x7f699c06d270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f69b0007fd0 tx=0x7f69b0009040 comp rx=0 tx=0).ready entity=mgr.10923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:45.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:45.823+0000 7f69b6a00700 1 -- 172.21.15.89:0/3122193641 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f69a80546f0 con 0x7f69c00d2370 2024-04-02T09:12:46.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.084+0000 7f69c5400700 1 -- 172.21.15.89:0/3122193641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f69c00d1ca0 con 0x7f69c00d2370 2024-04-02T09:12:46.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.224+0000 7f69b6a00700 1 -- 172.21.15.89:0/3122193641 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 147) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f69a8054ac0 con 0x7f69c00d2370 2024-04-02T09:12:46.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.226+0000 7f69b6a00700 1 -- 172.21.15.89:0/3122193641 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v147) v1 ==== 92+0+0 (secure 0 0 0) 0x7f69a8087190 con 0x7f69c00d2370 2024-04-02T09:12:46.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.229+0000 7f69c5400700 1 -- 172.21.15.89:0/3122193641 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f699c06adb0 msgr2=0x7f699c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:46.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.229+0000 7f69c5400700 1 --2- 172.21.15.89:0/3122193641 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f699c06adb0 0x7f699c06d270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f69b0007fd0 tx=0x7f69b0009040 comp rx=0 tx=0).stop 2024-04-02T09:12:46.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.229+0000 7f69c5400700 1 -- 172.21.15.89:0/3122193641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c00d2370 msgr2=0x7f69c00b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:46.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.229+0000 7f69c5400700 1 --2- 172.21.15.89:0/3122193641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c00d2370 0x7f69c00b8db0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f69a800ebe0 tx=0x7f69a800eef0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.229+0000 7f69c5400700 1 -- 172.21.15.89:0/3122193641 shutdown_connections 2024-04-02T09:12:46.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.229+0000 7f69c5400700 1 --2- 172.21.15.89:0/3122193641 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f699c06adb0 0x7f699c06d270 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.229+0000 7f69c5400700 1 --2- 172.21.15.89:0/3122193641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69c015d9d0 0x7f69c0158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.229+0000 7f69c5400700 1 --2- 172.21.15.89:0/3122193641 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69c00d16a0 0x7f69c015ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.229+0000 7f69c5400700 1 --2- 172.21.15.89:0/3122193641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c00d2370 0x7f69c00b8db0 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.229+0000 7f69c5400700 1 -- 172.21.15.89:0/3122193641 >> 172.21.15.89:0/3122193641 conn(0x7f69c008aff0 msgr2=0x7f69c015b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:46.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.229+0000 7f69c5400700 1 -- 172.21.15.89:0/3122193641 shutdown_connections 2024-04-02T09:12:46.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.229+0000 7f69c5400700 1 -- 172.21.15.89:0/3122193641 wait complete. 2024-04-02T09:12:46.255 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T09:12:46.440 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:46.442+0000 7f3fd7e48240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:12:46.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.459+0000 7f72d3e00700 1 Processor -- start 2024-04-02T09:12:46.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.459+0000 7f72d3e00700 1 -- start start 2024-04-02T09:12:46.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.461+0000 7f72d3e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f72cc160d70 0x7f72cc0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:46.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.461+0000 7f72d3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f72cc06a790 0x7f72cc06ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:46.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.461+0000 7f72d3e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72cc161150 con 0x7f72cc15ef70 2024-04-02T09:12:46.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.461+0000 7f72d3e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72cc0b4cd0 con 0x7f72cc06a790 2024-04-02T09:12:46.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.461+0000 7f72d3e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72cc06b140 con 0x7f72cc160d70 2024-04-02T09:12:46.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.461+0000 7f72d1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f72cc06a790 0x7f72cc06ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:46.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.461+0000 7f72d1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f72cc06a790 0x7f72cc06ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59720/0 (socket says 172.21.15.89:59720) 2024-04-02T09:12:46.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.461+0000 7f72d1e00700 1 -- 172.21.15.89:0/4112008143 learned_addr learned my addr 172.21.15.89:0/4112008143 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:46.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.461+0000 7f72d1400700 1 --2- 172.21.15.89:0/4112008143 >> v2:172.21.15.113:3300/0 conn(0x7f72cc160d70 0x7f72cc0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:46.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.462+0000 7f72d1e00700 1 -- 172.21.15.89:0/4112008143 >> v2:172.21.15.113:3300/0 conn(0x7f72cc160d70 msgr2=0x7f72cc0b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:46.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.462+0000 7f72d1e00700 1 --2- 172.21.15.89:0/4112008143 >> v2:172.21.15.113:3300/0 conn(0x7f72cc160d70 0x7f72cc0b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.462+0000 7f72d1e00700 1 -- 172.21.15.89:0/4112008143 >> v1:172.21.15.89:6789/0 conn(0x7f72cc15ef70 legacy=0x7f72cc15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:46.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.462+0000 7f72d1e00700 1 -- 172.21.15.89:0/4112008143 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72cc06dba0 con 0x7f72cc06a790 2024-04-02T09:12:46.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.462+0000 7f72d1e00700 1 --2- 172.21.15.89:0/4112008143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72cc06a790 0x7f72cc06ac00 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f72cc06c880 tx=0x7f72c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b69d96d86dc12c97 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:46.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.462+0000 7f72cbe00700 1 -- 172.21.15.89:0/4112008143 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c400e070 con 0x7f72cc06a790 2024-04-02T09:12:46.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.462+0000 7f72cbe00700 1 -- 172.21.15.89:0/4112008143 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c400b910 con 0x7f72cc06a790 2024-04-02T09:12:46.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.462+0000 7f72cbe00700 1 -- 172.21.15.89:0/4112008143 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c400bcc0 con 0x7f72cc06a790 2024-04-02T09:12:46.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.463+0000 7f72d3e00700 1 -- 172.21.15.89:0/4112008143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72cc06a790 msgr2=0x7f72cc06ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:46.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.463+0000 7f72d3e00700 1 --2- 172.21.15.89:0/4112008143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72cc06a790 0x7f72cc06ac00 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f72cc06c880 tx=0x7f72c400b330 comp rx=0 tx=0).stop 2024-04-02T09:12:46.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.463+0000 7f72d3e00700 1 -- 172.21.15.89:0/4112008143 shutdown_connections 2024-04-02T09:12:46.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.463+0000 7f72d3e00700 1 --2- 172.21.15.89:0/4112008143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72cc06a790 0x7f72cc06ac00 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.463+0000 7f72d3e00700 1 --2- 172.21.15.89:0/4112008143 >> v2:172.21.15.113:3300/0 conn(0x7f72cc160d70 0x7f72cc0b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.463+0000 7f72d3e00700 1 -- 172.21.15.89:0/4112008143 >> 172.21.15.89:0/4112008143 conn(0x7f72cc0b5720 msgr2=0x7f72cc0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:46.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.463+0000 7f72d3e00700 1 -- 172.21.15.89:0/4112008143 shutdown_connections 2024-04-02T09:12:46.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.463+0000 7f72d3e00700 1 -- 172.21.15.89:0/4112008143 wait complete. 2024-04-02T09:12:46.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.466+0000 7f72d3e00700 1 Processor -- start 2024-04-02T09:12:46.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.466+0000 7f72d3e00700 1 -- start start 2024-04-02T09:12:46.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.468+0000 7f72d3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72cc06a790 0x7f72cc0b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:46.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.468+0000 7f72d3e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72cc15ef70 0x7f72cc1593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:46.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.468+0000 7f72d3e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72cc160d70 0x7f72cc144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:46.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.468+0000 7f72d3e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc1f2e50 con 0x7f72cc06a790 2024-04-02T09:12:46.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.468+0000 7f72d3e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc158c70 con 0x7f72cc15ef70 2024-04-02T09:12:46.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.468+0000 7f72d3e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc157d80 con 0x7f72cc160d70 2024-04-02T09:12:46.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.468+0000 7f72d1400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72cc06a790 0x7f72cc0b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:46.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.468+0000 7f72d1400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72cc06a790 0x7f72cc0b2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59734/0 (socket says 172.21.15.89:59734) 2024-04-02T09:12:46.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.468+0000 7f72d1400700 1 -- 172.21.15.89:0/3683520047 learned_addr learned my addr 172.21.15.89:0/3683520047 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:46.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.468+0000 7f72d0a00700 1 --2- 172.21.15.89:0/3683520047 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72cc15ef70 0x7f72cc1593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:46.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.468+0000 7f72d1e00700 1 --2- 172.21.15.89:0/3683520047 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72cc160d70 0x7f72cc144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:46.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.469+0000 7f72d1400700 1 -- 172.21.15.89:0/3683520047 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72cc160d70 msgr2=0x7f72cc144760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:46.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.469+0000 7f72d1400700 1 --2- 172.21.15.89:0/3683520047 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72cc160d70 0x7f72cc144760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.469+0000 7f72d1400700 1 -- 172.21.15.89:0/3683520047 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72cc15ef70 msgr2=0x7f72cc1593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:46.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.469+0000 7f72d1400700 1 --2- 172.21.15.89:0/3683520047 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72cc15ef70 0x7f72cc1593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.469+0000 7f72d1400700 1 -- 172.21.15.89:0/3683520047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c4009b30 con 0x7f72cc06a790 2024-04-02T09:12:46.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.469+0000 7f72d1400700 1 --2- 172.21.15.89:0/3683520047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72cc06a790 0x7f72cc0b2e10 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f72c000e870 tx=0x7f72c000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:46.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.469+0000 7f72c9600700 1 -- 172.21.15.89:0/3683520047 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c0009590 con 0x7f72cc06a790 2024-04-02T09:12:46.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.469+0000 7f72c9600700 1 -- 172.21.15.89:0/3683520047 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c0013680 con 0x7f72cc06a790 2024-04-02T09:12:46.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.469+0000 7f72c9600700 1 -- 172.21.15.89:0/3683520047 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c00139f0 con 0x7f72cc06a790 2024-04-02T09:12:46.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.469+0000 7f72d3e00700 1 -- 172.21.15.89:0/3683520047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72cc158060 con 0x7f72cc06a790 2024-04-02T09:12:46.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.469+0000 7f72d3e00700 1 -- 172.21.15.89:0/3683520047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72cc158620 con 0x7f72cc06a790 2024-04-02T09:12:46.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.471+0000 7f72d3e00700 1 -- 172.21.15.89:0/3683520047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72cc0b51c0 con 0x7f72cc06a790 2024-04-02T09:12:46.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.471+0000 7f72c9600700 1 -- 172.21.15.89:0/3683520047 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 147) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f72c0013b50 con 0x7f72cc06a790 2024-04-02T09:12:46.471 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:46.472+0000 7efdb317a240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:12:46.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.472+0000 7f72c9600700 1 --2- 172.21.15.89:0/3683520047 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72ac06ae00 0x7f72ac06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:46.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.472+0000 7f72c9600700 1 -- 172.21.15.89:0/3683520047 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f72c0089850 con 0x7f72cc06a790 2024-04-02T09:12:46.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.472+0000 7f72d0a00700 1 -- 172.21.15.89:0/3683520047 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72ac06ae00 msgr2=0x7f72ac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:46.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.472+0000 7f72d0a00700 1 --2- 172.21.15.89:0/3683520047 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72ac06ae00 0x7f72ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:46.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.476+0000 7f72c9600700 1 -- 172.21.15.89:0/3683520047 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f72cc0b51c0 con 0x7f72cc06a790 2024-04-02T09:12:46.625 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:46.627+0000 7f3fd7e48240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:12:46.661 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:46.662+0000 7efdb317a240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:12:46.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.673+0000 7f72d0a00700 1 -- 172.21.15.89:0/3683520047 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72ac06ae00 msgr2=0x7f72ac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:46.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.673+0000 7f72d0a00700 1 --2- 172.21.15.89:0/3683520047 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72ac06ae00 0x7f72ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:46.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.722+0000 7f72d3e00700 1 -- 172.21.15.89:0/3683520047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f72cc0b51c0 con 0x7f72cc06a790 2024-04-02T09:12:46.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.727+0000 7f72c9600700 1 -- 172.21.15.89:0/3683520047 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v77)=0 v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7f72cc0b51c0 con 0x7f72cc06a790 2024-04-02T09:12:46.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.729+0000 7f72d3e00700 1 -- 172.21.15.89:0/3683520047 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72ac06ae00 msgr2=0x7f72ac06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:46.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.729+0000 7f72d3e00700 1 --2- 172.21.15.89:0/3683520047 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72ac06ae00 0x7f72ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.729+0000 7f72d3e00700 1 -- 172.21.15.89:0/3683520047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72cc06a790 msgr2=0x7f72cc0b2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:46.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.729+0000 7f72d3e00700 1 --2- 172.21.15.89:0/3683520047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72cc06a790 0x7f72cc0b2e10 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f72c000e870 tx=0x7f72c000eb80 comp rx=0 tx=0).stop 2024-04-02T09:12:46.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.730+0000 7f72d3e00700 1 -- 172.21.15.89:0/3683520047 shutdown_connections 2024-04-02T09:12:46.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.730+0000 7f72d3e00700 1 --2- 172.21.15.89:0/3683520047 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72cc160d70 0x7f72cc144760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.730+0000 7f72d3e00700 1 --2- 172.21.15.89:0/3683520047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72cc06a790 0x7f72cc0b2e10 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.730+0000 7f72d3e00700 1 --2- 172.21.15.89:0/3683520047 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72ac06ae00 0x7f72ac06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.730+0000 7f72d3e00700 1 --2- 172.21.15.89:0/3683520047 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72cc15ef70 0x7f72cc1593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.730+0000 7f72d3e00700 1 -- 172.21.15.89:0/3683520047 >> 172.21.15.89:0/3683520047 conn(0x7f72cc0b5720 msgr2=0x7f72cc09c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:46.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.730+0000 7f72d3e00700 1 -- 172.21.15.89:0/3683520047 shutdown_connections 2024-04-02T09:12:46.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.730+0000 7f72d3e00700 1 -- 172.21.15.89:0/3683520047 wait complete. 2024-04-02T09:12:46.747 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T09:12:46.942 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:46.943+0000 7f3fd7e48240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:12:46.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.947+0000 7f2a70800700 1 Processor -- start 2024-04-02T09:12:46.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.947+0000 7f2a70800700 1 -- start start 2024-04-02T09:12:46.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.949+0000 7f2a70800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2a6815dfe0 0x7f2a680ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:46.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.949+0000 7f2a70800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a680ce670 0x7f2a680cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:46.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.949+0000 7f2a70800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a680cb310 con 0x7f2a6815a5c0 2024-04-02T09:12:46.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.949+0000 7f2a70800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a680ceb00 con 0x7f2a6815dfe0 2024-04-02T09:12:46.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.949+0000 7f2a70800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a680711a0 con 0x7f2a680ce670 2024-04-02T09:12:46.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.949+0000 7f2a6de00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2a6815a5c0 0x7f2a680709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:54742/0 (socket says 172.21.15.89:54742) 2024-04-02T09:12:46.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.949+0000 7f2a6de00700 1 -- 172.21.15.89:0/2504762028 learned_addr learned my addr 172.21.15.89:0/2504762028 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:46.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.949+0000 7f2a6e800700 1 --2- 172.21.15.89:0/2504762028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a680ce670 0x7f2a680cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:46.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.949+0000 7f2a6d400700 1 --2- 172.21.15.89:0/2504762028 >> v2:172.21.15.113:3300/0 conn(0x7f2a6815dfe0 0x7f2a680ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:46.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.949+0000 7f2a6ca00700 1 -- 172.21.15.89:0/2504762028 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372695423 0 0) 0x7f2a680cb310 con 0x7f2a6815a5c0 2024-04-02T09:12:46.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.949+0000 7f2a6ca00700 1 -- 172.21.15.89:0/2504762028 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a500035b0 con 0x7f2a6815a5c0 2024-04-02T09:12:46.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.950+0000 7f2a6e800700 1 -- 172.21.15.89:0/2504762028 >> v2:172.21.15.113:3300/0 conn(0x7f2a6815dfe0 msgr2=0x7f2a680ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:46.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.950+0000 7f2a6e800700 1 --2- 172.21.15.89:0/2504762028 >> v2:172.21.15.113:3300/0 conn(0x7f2a6815dfe0 0x7f2a680ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.950+0000 7f2a6e800700 1 -- 172.21.15.89:0/2504762028 >> v1:172.21.15.89:6789/0 conn(0x7f2a6815a5c0 legacy=0x7f2a680709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:46.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.950+0000 7f2a6e800700 1 -- 172.21.15.89:0/2504762028 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a68071480 con 0x7f2a680ce670 2024-04-02T09:12:46.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.950+0000 7f2a6d400700 1 --2- 172.21.15.89:0/2504762028 >> v2:172.21.15.113:3300/0 conn(0x7f2a6815dfe0 0x7f2a680ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:12:46.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.950+0000 7f2a6e800700 1 --2- 172.21.15.89:0/2504762028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a680ce670 0x7f2a680cba80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2a64009d90 tx=0x7f2a6400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e547b2a7f74c7bc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:46.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.950+0000 7f2a6ca00700 1 -- 172.21.15.89:0/2504762028 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6400e070 con 0x7f2a680ce670 2024-04-02T09:12:46.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.950+0000 7f2a6ca00700 1 -- 172.21.15.89:0/2504762028 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6400bb20 con 0x7f2a680ce670 2024-04-02T09:12:46.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.950+0000 7f2a6ca00700 1 -- 172.21.15.89:0/2504762028 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64012430 con 0x7f2a680ce670 2024-04-02T09:12:46.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.951+0000 7f2a70800700 1 -- 172.21.15.89:0/2504762028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a680ce670 msgr2=0x7f2a680cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:46.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.951+0000 7f2a70800700 1 --2- 172.21.15.89:0/2504762028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a680ce670 0x7f2a680cba80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2a64009d90 tx=0x7f2a6400b330 comp rx=0 tx=0).stop 2024-04-02T09:12:46.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.951+0000 7f2a70800700 1 -- 172.21.15.89:0/2504762028 shutdown_connections 2024-04-02T09:12:46.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.951+0000 7f2a70800700 1 --2- 172.21.15.89:0/2504762028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a680ce670 0x7f2a680cba80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.951+0000 7f2a70800700 1 --2- 172.21.15.89:0/2504762028 >> v2:172.21.15.113:3300/0 conn(0x7f2a6815dfe0 0x7f2a680ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.951+0000 7f2a70800700 1 -- 172.21.15.89:0/2504762028 >> 172.21.15.89:0/2504762028 conn(0x7f2a680b0d70 msgr2=0x7f2a680b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:46.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.951+0000 7f2a70800700 1 -- 172.21.15.89:0/2504762028 shutdown_connections 2024-04-02T09:12:46.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.951+0000 7f2a70800700 1 -- 172.21.15.89:0/2504762028 wait complete. 2024-04-02T09:12:46.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.953+0000 7f2a70800700 1 Processor -- start 2024-04-02T09:12:46.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.953+0000 7f2a70800700 1 -- start start 2024-04-02T09:12:46.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.954+0000 7f2a70800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680ce670 0x7f2a68158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:46.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.954+0000 7f2a70800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a6815a5c0 0x7f2a68145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:46.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.954+0000 7f2a70800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a6815dfe0 0x7f2a68145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:46.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.954+0000 7f2a70800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a68146550 con 0x7f2a680ce670 2024-04-02T09:12:46.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.954+0000 7f2a70800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a680ac050 con 0x7f2a6815a5c0 2024-04-02T09:12:46.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.954+0000 7f2a70800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a680cd4e0 con 0x7f2a6815dfe0 2024-04-02T09:12:46.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.954+0000 7f2a6de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680ce670 0x7f2a68158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:46.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.954+0000 7f2a6d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a6815a5c0 0x7f2a68145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:46.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.954+0000 7f2a6de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680ce670 0x7f2a68158e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59748/0 (socket says 172.21.15.89:59748) 2024-04-02T09:12:46.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.954+0000 7f2a6d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a6815a5c0 0x7f2a68145640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:49646/0 (socket says 172.21.15.89:49646) 2024-04-02T09:12:46.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.954+0000 7f2a6de00700 1 -- 172.21.15.89:0/1455165404 learned_addr learned my addr 172.21.15.89:0/1455165404 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:46.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.954+0000 7f2a6e800700 1 --2- 172.21.15.89:0/1455165404 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a6815dfe0 0x7f2a68145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:46.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.955+0000 7f2a6de00700 1 -- 172.21.15.89:0/1455165404 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a6815dfe0 msgr2=0x7f2a68145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:46.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.955+0000 7f2a6de00700 1 --2- 172.21.15.89:0/1455165404 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a6815dfe0 0x7f2a68145b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.955+0000 7f2a6de00700 1 -- 172.21.15.89:0/1455165404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a6815a5c0 msgr2=0x7f2a68145640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:46.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.955+0000 7f2a6de00700 1 --2- 172.21.15.89:0/1455165404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a6815a5c0 0x7f2a68145640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:46.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.955+0000 7f2a6de00700 1 -- 172.21.15.89:0/1455165404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a580094a0 con 0x7f2a680ce670 2024-04-02T09:12:46.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.955+0000 7f2a6de00700 1 --2- 172.21.15.89:0/1455165404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680ce670 0x7f2a68158e70 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c00c700 tx=0x7f2a5c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:46.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.955+0000 7f2a62000700 1 -- 172.21.15.89:0/1455165404 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c005d70 con 0x7f2a680ce670 2024-04-02T09:12:46.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.955+0000 7f2a62000700 1 -- 172.21.15.89:0/1455165404 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5c0070d0 con 0x7f2a680ce670 2024-04-02T09:12:46.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.955+0000 7f2a70800700 1 -- 172.21.15.89:0/1455165404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a64009a70 con 0x7f2a680ce670 2024-04-02T09:12:46.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.955+0000 7f2a62000700 1 -- 172.21.15.89:0/1455165404 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c007460 con 0x7f2a680ce670 2024-04-02T09:12:46.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.955+0000 7f2a70800700 1 -- 172.21.15.89:0/1455165404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a68159c60 con 0x7f2a680ce670 2024-04-02T09:12:46.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.956+0000 7f2a70800700 1 -- 172.21.15.89:0/1455165404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a68067720 con 0x7f2a680ce670 2024-04-02T09:12:46.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.958+0000 7f2a62000700 1 -- 172.21.15.89:0/1455165404 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 147) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f2a5c0075c0 con 0x7f2a680ce670 2024-04-02T09:12:46.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.959+0000 7f2a62000700 1 --2- 172.21.15.89:0/1455165404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2a4c06ae00 0x7f2a4c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:46.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.959+0000 7f2a62000700 1 -- 172.21.15.89:0/1455165404 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f2a5c088e00 con 0x7f2a680ce670 2024-04-02T09:12:46.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.959+0000 7f2a6d400700 1 -- 172.21.15.89:0/1455165404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2a4c06ae00 msgr2=0x7f2a4c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:46.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.959+0000 7f2a6d400700 1 --2- 172.21.15.89:0/1455165404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2a4c06ae00 0x7f2a4c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:46.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:46.963+0000 7f2a62000700 1 -- 172.21.15.89:0/1455165404 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2a5c053a40 con 0x7f2a680ce670 2024-04-02T09:12:47.000 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:47.001+0000 7efdb317a240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:12:47.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.160+0000 7f2a6d400700 1 -- 172.21.15.89:0/1455165404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2a4c06ae00 msgr2=0x7f2a4c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:47.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.160+0000 7f2a6d400700 1 --2- 172.21.15.89:0/1455165404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2a4c06ae00 0x7f2a4c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:47.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.175+0000 7f2a70800700 1 -- 172.21.15.89:0/1455165404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f2a680b1ad0 con 0x7f2a680ce670 2024-04-02T09:12:47.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.179+0000 7f2a62000700 1 -- 172.21.15.89:0/1455165404 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v78)=0 v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2a5c017020 con 0x7f2a680ce670 2024-04-02T09:12:47.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.181+0000 7f2a70800700 1 -- 172.21.15.89:0/1455165404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2a4c06ae00 msgr2=0x7f2a4c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:47.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.181+0000 7f2a70800700 1 --2- 172.21.15.89:0/1455165404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2a4c06ae00 0x7f2a4c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:47.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.181+0000 7f2a70800700 1 -- 172.21.15.89:0/1455165404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680ce670 msgr2=0x7f2a68158e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:47.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.181+0000 7f2a70800700 1 --2- 172.21.15.89:0/1455165404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680ce670 0x7f2a68158e70 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c00c700 tx=0x7f2a5c00ca10 comp rx=0 tx=0).stop 2024-04-02T09:12:47.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.181+0000 7f2a70800700 1 -- 172.21.15.89:0/1455165404 shutdown_connections 2024-04-02T09:12:47.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.181+0000 7f2a70800700 1 --2- 172.21.15.89:0/1455165404 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a6815dfe0 0x7f2a68145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:47.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.181+0000 7f2a70800700 1 --2- 172.21.15.89:0/1455165404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680ce670 0x7f2a68158e70 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:47.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.181+0000 7f2a70800700 1 --2- 172.21.15.89:0/1455165404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2a4c06ae00 0x7f2a4c06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:47.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.181+0000 7f2a70800700 1 --2- 172.21.15.89:0/1455165404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a6815a5c0 0x7f2a68145640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:47.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.181+0000 7f2a70800700 1 -- 172.21.15.89:0/1455165404 >> 172.21.15.89:0/1455165404 conn(0x7f2a680b0d70 msgr2=0x7f2a680cb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:47.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.182+0000 7f2a70800700 1 -- 172.21.15.89:0/1455165404 shutdown_connections 2024-04-02T09:12:47.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.182+0000 7f2a70800700 1 -- 172.21.15.89:0/1455165404 wait complete. 2024-04-02T09:12:47.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:47.199 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-04-02T09:12:47.248 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-02T09:12:47.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.452+0000 7fa41ce00700 1 Processor -- start 2024-04-02T09:12:47.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.452+0000 7fa41ce00700 1 -- start start 2024-04-02T09:12:47.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.453+0000 7fa41ce00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa4180d22a0 0x7fa4180cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:47.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.453+0000 7fa41ce00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41815dd10 con 0x7fa418162030 2024-04-02T09:12:47.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.453+0000 7fa41ce00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41815dfc0 con 0x7fa41815d930 2024-04-02T09:12:47.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.453+0000 7fa41ce00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41815ec40 con 0x7fa4180d22a0 2024-04-02T09:12:47.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.454+0000 7fa416800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa4180d22a0 0x7fa4180cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:47.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.454+0000 7fa416800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa4180d22a0 0x7fa4180cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59760/0 (socket says 172.21.15.89:59760) 2024-04-02T09:12:47.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.454+0000 7fa40d400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa418162030 0x7fa4180d1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:54748/0 (socket says 172.21.15.89:54748) 2024-04-02T09:12:47.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.454+0000 7fa416800700 1 -- 172.21.15.89:0/2239673680 learned_addr learned my addr 172.21.15.89:0/2239673680 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:47.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.454+0000 7fa416800700 1 -- 172.21.15.89:0/2239673680 >> v1:172.21.15.113:6789/0 conn(0x7fa41815d930 legacy=0x7fa4180d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:47.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.454+0000 7fa416800700 1 -- 172.21.15.89:0/2239673680 >> v1:172.21.15.89:6789/0 conn(0x7fa418162030 legacy=0x7fa4180d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:47.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.454+0000 7fa415400700 1 -- 172.21.15.89:0/2239673680 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180154483 0 0) 0x7fa41815dfc0 con 0x7fa41815d930 2024-04-02T09:12:47.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.454+0000 7fa416800700 1 -- 172.21.15.89:0/2239673680 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41815e2a0 con 0x7fa4180d22a0 2024-04-02T09:12:47.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.454+0000 7fa416800700 1 --2- 172.21.15.89:0/2239673680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4180d22a0 0x7fa4180cf310 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fa410009e50 tx=0x7fa41000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb842ee85839563e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:47.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.455+0000 7fa415400700 1 -- 172.21.15.89:0/2239673680 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41000e070 con 0x7fa4180d22a0 2024-04-02T09:12:47.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.455+0000 7fa415400700 1 -- 172.21.15.89:0/2239673680 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa410003a20 con 0x7fa4180d22a0 2024-04-02T09:12:47.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.455+0000 7fa415400700 1 -- 172.21.15.89:0/2239673680 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa410003dd0 con 0x7fa4180d22a0 2024-04-02T09:12:47.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.455+0000 7fa41ce00700 1 -- 172.21.15.89:0/2239673680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4180d22a0 msgr2=0x7fa4180cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:47.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.455+0000 7fa41ce00700 1 --2- 172.21.15.89:0/2239673680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4180d22a0 0x7fa4180cf310 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fa410009e50 tx=0x7fa41000b330 comp rx=0 tx=0).stop 2024-04-02T09:12:47.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.455+0000 7fa41ce00700 1 -- 172.21.15.89:0/2239673680 shutdown_connections 2024-04-02T09:12:47.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.455+0000 7fa41ce00700 1 --2- 172.21.15.89:0/2239673680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4180d22a0 0x7fa4180cf310 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:47.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.455+0000 7fa41ce00700 1 -- 172.21.15.89:0/2239673680 >> 172.21.15.89:0/2239673680 conn(0x7fa41808aff0 msgr2=0x7fa4180d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:47.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.455+0000 7fa41ce00700 1 -- 172.21.15.89:0/2239673680 shutdown_connections 2024-04-02T09:12:47.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.456+0000 7fa41ce00700 1 -- 172.21.15.89:0/2239673680 wait complete. 2024-04-02T09:12:47.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.457+0000 7fa41ce00700 1 Processor -- start 2024-04-02T09:12:47.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.457+0000 7fa41ce00700 1 -- start start 2024-04-02T09:12:47.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.458+0000 7fa41ce00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4180d22a0 0x7fa4181533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:47.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.458+0000 7fa41ce00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa41815d930 0x7fa4181f6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:47.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.458+0000 7fa41ce00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa418162030 0x7fa4181f72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:47.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.458+0000 7fa41ce00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa418160740 con 0x7fa4180d22a0 2024-04-02T09:12:47.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.458+0000 7fa41ce00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41815fb30 con 0x7fa418162030 2024-04-02T09:12:47.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.458+0000 7fa41ce00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41815fe10 con 0x7fa41815d930 2024-04-02T09:12:47.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.458+0000 7fa415e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4180d22a0 0x7fa4181533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:47.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.458+0000 7fa415e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4180d22a0 0x7fa4181533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59772/0 (socket says 172.21.15.89:59772) 2024-04-02T09:12:47.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.458+0000 7fa415e00700 1 -- 172.21.15.89:0/641879622 learned_addr learned my addr 172.21.15.89:0/641879622 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:47.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.459+0000 7fa416800700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa418162030 0x7fa4181f72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.459+0000 7fa40d400700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa41815d930 0x7fa4181f6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.459+0000 7fa415e00700 1 -- 172.21.15.89:0/641879622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa41815d930 msgr2=0x7fa4181f6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.459+0000 7fa415e00700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa41815d930 0x7fa4181f6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.459+0000 7fa415e00700 1 -- 172.21.15.89:0/641879622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa418162030 msgr2=0x7fa4181f72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.459+0000 7fa415e00700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa418162030 0x7fa4181f72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.459+0000 7fa415e00700 1 -- 172.21.15.89:0/641879622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa410009b30 con 0x7fa4180d22a0 2024-04-02T09:12:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.459+0000 7fa415e00700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4180d22a0 0x7fa4181533d0 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fa4080011e0 tx=0x7fa40800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.459+0000 7fa40ea00700 1 -- 172.21.15.89:0/641879622 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4080054e0 con 0x7fa4180d22a0 2024-04-02T09:12:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.459+0000 7fa40ea00700 1 -- 172.21.15.89:0/641879622 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa408003680 con 0x7fa4180d22a0 2024-04-02T09:12:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.459+0000 7fa40ea00700 1 -- 172.21.15.89:0/641879622 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa408003a30 con 0x7fa4180d22a0 2024-04-02T09:12:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.459+0000 7fa41ce00700 1 -- 172.21.15.89:0/641879622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4181600f0 con 0x7fa4180d22a0 2024-04-02T09:12:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.460+0000 7fa41ce00700 1 -- 172.21.15.89:0/641879622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa418155fb0 con 0x7fa4180d22a0 2024-04-02T09:12:47.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.461+0000 7fa41ce00700 1 -- 172.21.15.89:0/641879622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa418160a20 con 0x7fa4180d22a0 2024-04-02T09:12:47.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.461+0000 7fa40ea00700 1 -- 172.21.15.89:0/641879622 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 147) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fa408003b90 con 0x7fa4180d22a0 2024-04-02T09:12:47.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.462+0000 7fa40ea00700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 0x7fa3f406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:47.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.462+0000 7fa40ea00700 1 -- 172.21.15.89:0/641879622 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fa408088e60 con 0x7fa4180d22a0 2024-04-02T09:12:47.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.462+0000 7fa40d400700 1 -- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 msgr2=0x7fa3f406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:47.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.462+0000 7fa40d400700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 0x7fa3f406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:47.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.467+0000 7fa40ea00700 1 -- 172.21.15.89:0/641879622 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa408053a00 con 0x7fa4180d22a0 2024-04-02T09:12:47.476 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:47.477+0000 7f3fd7e48240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:12:47.561 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:47.562+0000 7efdb317a240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:12:47.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.663+0000 7fa40d400700 1 -- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 msgr2=0x7fa3f406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:47.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.663+0000 7fa40d400700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 0x7fa3f406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:47.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:47.702+0000 7fa41ce00700 1 -- 172.21.15.89:0/641879622 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fa41815e860 con 0x7fa3f406ae00 2024-04-02T09:12:47.731 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:47.733+0000 7f3fd7e48240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:12:47.821 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:47.822+0000 7efdb317a240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:12:48.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:48.064+0000 7fa40d400700 1 -- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 msgr2=0x7fa3f406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:48.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:48.064+0000 7fa40d400700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 0x7fa3f406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T09:12:48.188 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:48.190+0000 7f3fd7e48240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:12:48.276 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:48.277+0000 7efdb317a240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:12:48.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:48.865+0000 7fa40d400700 1 -- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 msgr2=0x7fa3f406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:48.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:48.865+0000 7fa40d400700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 0x7fa3f406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T09:12:48.899 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:48.900+0000 7f3fd7e48240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:12:48.986 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:48.985+0000 7efdb317a240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:12:48.991 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:48.992+0000 7f3fd7e48240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:12:49.075 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:49.075+0000 7efdb317a240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:12:49.150 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:49.151+0000 7f3fd7e48240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:12:49.238 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:49.239+0000 7efdb317a240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:12:49.367 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:49.368+0000 7f3fd7e48240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:12:49.454 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:49.455+0000 7efdb317a240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:12:49.458 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:49.459+0000 7f3fd7e48240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:12:49.545 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:49.546+0000 7efdb317a240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:12:49.556 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:49.557+0000 7f3fd7e48240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:12:49.643 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:49.644+0000 7efdb317a240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:12:49.653 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:49.655+0000 7f3fd7e48240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:12:49.743 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:49.744+0000 7efdb317a240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:12:49.753 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:49.754+0000 7f3fd7e48240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:12:49.844 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:49.844+0000 7efdb317a240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:12:49.963 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:49.964+0000 7f3fd7e48240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:12:50.057 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:50.058+0000 7efdb317a240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:12:50.218 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:50.219+0000 7f3fd7e48240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:12:50.313 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:50.314+0000 7efdb317a240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:12:50.409 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:50.410+0000 7f3fd7e48240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:12:50.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:50.467+0000 7fa40d400700 1 -- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 msgr2=0x7fa3f406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:50.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:50.467+0000 7fa40d400700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 0x7fa3f406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T09:12:50.502 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:50.504+0000 7f3fd7e48240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:12:50.506 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:50.507+0000 7efdb317a240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:12:50.599 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:50.600+0000 7efdb317a240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:12:51.228 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:51.230+0000 7f3fd7e48240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:12:51.331 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:51.331+0000 7efdb317a240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:12:51.441 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:51.442+0000 7f3fd7e48240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:12:51.546 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:51.547+0000 7efdb317a240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:12:51.646 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:51.647+0000 7f3fd7e48240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:12:51.758 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:51.759+0000 7efdb317a240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:12:52.118 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:52.119+0000 7f3fd7e48240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:12:52.237 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:52.238+0000 7efdb317a240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:12:52.324 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:52.326+0000 7f3fd7e48240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:12:52.446 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:52.447+0000 7efdb317a240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:12:52.764 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:52.765+0000 7f3fd7e48240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:12:52.877 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:52.877+0000 7f3fd7e48240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:12:52.885 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:52.887+0000 7efdb317a240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:12:52.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:52.937+0000 7fa40ea00700 1 -- 172.21.15.89:0/641879622 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 148) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fa408053dd0 con 0x7fa4180d22a0 2024-04-02T09:12:52.999 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.001+0000 7efdb317a240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:12:53.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:53.011+0000 7fa40ea00700 1 -- 172.21.15.89:0/641879622 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 149) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fa408086dc0 con 0x7fa4180d22a0 2024-04-02T09:12:53.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:53.012+0000 7fa40ea00700 1 -- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 msgr2=0x7fa3f406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:53.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:53.012+0000 7fa40ea00700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 0x7fa3f406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:53.284 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.286+0000 7efd1ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.284 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.286+0000 7efd1ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.284 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.286+0000 7efd1ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.284 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.286+0000 7efd1ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.284 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.286+0000 7efd1ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.285 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.286+0000 7efd1ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.285 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.286+0000 7efd1ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.285 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.286+0000 7efd1ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.285 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.286+0000 7efd1ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.285 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.286+0000 7efd1ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.297 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.299+0000 7efd1ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.297 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.299+0000 7efd1ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.297 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.299+0000 7efd1ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.297 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.299+0000 7efd1ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.298 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.299+0000 7efd1ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.298 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.299+0000 7efd1ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.298 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.299+0000 7efd1ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.298 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.299+0000 7efd1ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.298 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.299+0000 7efd1ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:53.298 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:53.299+0000 7efd1ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:12:54.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.014+0000 7fa40ea00700 1 -- 172.21.15.89:0/641879622 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mgrmap(e 150) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fa4080546a0 con 0x7fa4180d22a0 2024-04-02T09:12:54.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.015+0000 7fa40ea00700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 0x7fa3f406d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:54.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.015+0000 7fa40ea00700 1 -- 172.21.15.89:0/641879622 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3f401d060 con 0x7fa3f406ae00 2024-04-02T09:12:54.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.018+0000 7fa40d400700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 0x7fa3f406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:54.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.021+0000 7fa40d400700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 0x7fa3f406d240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa400001340 tx=0x7fa4000012d0 comp rx=0 tx=0).ready entity=mgr.11466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:54.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.084+0000 7fa40ea00700 1 -- 172.21.15.89:0/641879622 <== mgr.11466 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+53 (secure 0 0 0) 0x7fa3f401d060 con 0x7fa3f406ae00 2024-04-02T09:12:54.082 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path mandatory_fs_argument_test_dir 2024-04-02T09:12:54.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.086+0000 7fa41ce00700 1 -- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 msgr2=0x7fa3f406d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:54.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.086+0000 7fa41ce00700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 0x7fa3f406d240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa400001340 tx=0x7fa4000012d0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.086+0000 7fa41ce00700 1 -- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4180d22a0 msgr2=0x7fa4181533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:54.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.086+0000 7fa41ce00700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4180d22a0 0x7fa4181533d0 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fa4080011e0 tx=0x7fa40800ca30 comp rx=0 tx=0).stop 2024-04-02T09:12:54.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.086+0000 7fa41ce00700 1 -- 172.21.15.89:0/641879622 shutdown_connections 2024-04-02T09:12:54.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.086+0000 7fa41ce00700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3f406ae00 0x7fa3f406d240 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.086+0000 7fa41ce00700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa418162030 0x7fa4181f72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.086+0000 7fa41ce00700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4180d22a0 0x7fa4181533d0 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.086+0000 7fa41ce00700 1 --2- 172.21.15.89:0/641879622 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa41815d930 0x7fa4181f6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.086+0000 7fa41ce00700 1 -- 172.21.15.89:0/641879622 >> 172.21.15.89:0/641879622 conn(0x7fa41808aff0 msgr2=0x7fa41806d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:54.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.087+0000 7fa41ce00700 1 -- 172.21.15.89:0/641879622 shutdown_connections 2024-04-02T09:12:54.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.087+0000 7fa41ce00700 1 -- 172.21.15.89:0/641879622 wait complete. 2024-04-02T09:12:54.103 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2024-04-02T09:12:54.104 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-02T09:12:54.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.304+0000 7f8821800700 1 Processor -- start 2024-04-02T09:12:54.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.304+0000 7f8821800700 1 -- start start 2024-04-02T09:12:54.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.305+0000 7f8821800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f881c155ac0 0x7f881c15c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:54.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.305+0000 7f8821800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f881c06c910 0x7f881c0adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:54.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.305+0000 7f8821800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f881c0ae4a0 0x7f881c15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:54.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.305+0000 7f8821800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f881c06dd00 con 0x7f881c155ac0 2024-04-02T09:12:54.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.305+0000 7f8821800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f881c06d0f0 con 0x7f881c0ae4a0 2024-04-02T09:12:54.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.305+0000 7f8821800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881c06d3d0 con 0x7f881c06c910 2024-04-02T09:12:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.305+0000 7f881aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f881c155ac0 0x7f881c15c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.305+0000 7f881aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f881c155ac0 0x7f881c15c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42786/0 (socket says 172.21.15.89:42786) 2024-04-02T09:12:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.305+0000 7f881aa00700 1 -- 172.21.15.89:0/2269379270 learned_addr learned my addr 172.21.15.89:0/2269379270 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.305+0000 7f881b400700 1 --2- 172.21.15.89:0/2269379270 >> v2:172.21.15.113:3300/0 conn(0x7f881c0ae4a0 0x7f881c15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.305+0000 7f881a000700 1 --2- 172.21.15.89:0/2269379270 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f881c06c910 0x7f881c0adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.306+0000 7f881aa00700 1 -- 172.21.15.89:0/2269379270 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f881c06c910 msgr2=0x7f881c0adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.306+0000 7f881aa00700 1 --2- 172.21.15.89:0/2269379270 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f881c06c910 0x7f881c0adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.306+0000 7f881aa00700 1 -- 172.21.15.89:0/2269379270 >> v2:172.21.15.113:3300/0 conn(0x7f881c0ae4a0 msgr2=0x7f881c15f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.306+0000 7f881aa00700 1 --2- 172.21.15.89:0/2269379270 >> v2:172.21.15.113:3300/0 conn(0x7f881c0ae4a0 0x7f881c15f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.306+0000 7f881aa00700 1 -- 172.21.15.89:0/2269379270 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881c06d6b0 con 0x7f881c155ac0 2024-04-02T09:12:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.306+0000 7f881aa00700 1 --2- 172.21.15.89:0/2269379270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881c155ac0 0x7f881c15c700 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f881c15cc40 tx=0x7f880c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0aa9a5f1e52a2fd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.306+0000 7f8819600700 1 -- 172.21.15.89:0/2269379270 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c00b870 con 0x7f881c155ac0 2024-04-02T09:12:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.306+0000 7f8819600700 1 -- 172.21.15.89:0/2269379270 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880c00b9d0 con 0x7f881c155ac0 2024-04-02T09:12:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.306+0000 7f8819600700 1 -- 172.21.15.89:0/2269379270 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c00bd60 con 0x7f881c155ac0 2024-04-02T09:12:54.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.307+0000 7f8821800700 1 -- 172.21.15.89:0/2269379270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881c155ac0 msgr2=0x7f881c15c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:54.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.307+0000 7f8821800700 1 --2- 172.21.15.89:0/2269379270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881c155ac0 0x7f881c15c700 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f881c15cc40 tx=0x7f880c009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.307+0000 7f8821800700 1 -- 172.21.15.89:0/2269379270 shutdown_connections 2024-04-02T09:12:54.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.307+0000 7f8821800700 1 --2- 172.21.15.89:0/2269379270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881c155ac0 0x7f881c15c700 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.307+0000 7f8821800700 1 --2- 172.21.15.89:0/2269379270 >> v2:172.21.15.113:3300/0 conn(0x7f881c0ae4a0 0x7f881c15f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.307+0000 7f8821800700 1 --2- 172.21.15.89:0/2269379270 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f881c06c910 0x7f881c0adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.307+0000 7f8821800700 1 -- 172.21.15.89:0/2269379270 >> 172.21.15.89:0/2269379270 conn(0x7f881c0cf210 msgr2=0x7f881c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:54.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.307+0000 7f8821800700 1 -- 172.21.15.89:0/2269379270 shutdown_connections 2024-04-02T09:12:54.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.307+0000 7f8821800700 1 -- 172.21.15.89:0/2269379270 wait complete. 2024-04-02T09:12:54.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.308+0000 7f8821800700 1 Processor -- start 2024-04-02T09:12:54.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.309+0000 7f8821800700 1 -- start start 2024-04-02T09:12:54.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.309+0000 7f8821800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881c06c910 0x7f881c098b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f8821800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f881c0ae4a0 0x7f881c14b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f8821800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f881c155ac0 0x7f881c14d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f8821800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c14c240 con 0x7f881c06c910 2024-04-02T09:12:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f8821800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c14c520 con 0x7f881c155ac0 2024-04-02T09:12:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f8821800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881c14c800 con 0x7f881c0ae4a0 2024-04-02T09:12:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f881aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881c06c910 0x7f881c098b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f881aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881c06c910 0x7f881c098b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42790/0 (socket says 172.21.15.89:42790) 2024-04-02T09:12:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f881aa00700 1 -- 172.21.15.89:0/3130828575 learned_addr learned my addr 172.21.15.89:0/3130828575 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f881b400700 1 --2- 172.21.15.89:0/3130828575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f881c155ac0 0x7f881c14d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f881a000700 1 --2- 172.21.15.89:0/3130828575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f881c0ae4a0 0x7f881c14b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f881aa00700 1 -- 172.21.15.89:0/3130828575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f881c0ae4a0 msgr2=0x7f881c14b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f881aa00700 1 --2- 172.21.15.89:0/3130828575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f881c0ae4a0 0x7f881c14b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f881aa00700 1 -- 172.21.15.89:0/3130828575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f881c155ac0 msgr2=0x7f881c14d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:54.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f881aa00700 1 --2- 172.21.15.89:0/3130828575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f881c155ac0 0x7f881c14d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.310+0000 7f881aa00700 1 -- 172.21.15.89:0/3130828575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f880c009550 con 0x7f881c06c910 2024-04-02T09:12:54.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.311+0000 7f881aa00700 1 --2- 172.21.15.89:0/3130828575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881c06c910 0x7f881c098b40 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f880c000c00 tx=0x7f880c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:54.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.311+0000 7f8812a00700 1 -- 172.21.15.89:0/3130828575 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c004310 con 0x7f881c06c910 2024-04-02T09:12:54.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.311+0000 7f8812a00700 1 -- 172.21.15.89:0/3130828575 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880c01e070 con 0x7f881c06c910 2024-04-02T09:12:54.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.311+0000 7f8812a00700 1 -- 172.21.15.89:0/3130828575 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c0036a0 con 0x7f881c06c910 2024-04-02T09:12:54.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.311+0000 7f8821800700 1 -- 172.21.15.89:0/3130828575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f881c14cae0 con 0x7f881c06c910 2024-04-02T09:12:54.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.311+0000 7f8821800700 1 -- 172.21.15.89:0/3130828575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f881c14bd80 con 0x7f881c06c910 2024-04-02T09:12:54.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.312+0000 7f8821800700 1 -- 172.21.15.89:0/3130828575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f881c14dec0 con 0x7f881c06c910 2024-04-02T09:12:54.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.313+0000 7f8812a00700 1 -- 172.21.15.89:0/3130828575 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 150) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f880c020030 con 0x7f881c06c910 2024-04-02T09:12:54.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.314+0000 7f8812a00700 1 --2- 172.21.15.89:0/3130828575 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f87fc06ab30 0x7f87fc06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:54.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.314+0000 7f8812a00700 1 -- 172.21.15.89:0/3130828575 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f880c08a4e0 con 0x7f881c06c910 2024-04-02T09:12:54.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.314+0000 7f881a000700 1 --2- 172.21.15.89:0/3130828575 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f87fc06ab30 0x7f87fc06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:54.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.314+0000 7f881a000700 1 --2- 172.21.15.89:0/3130828575 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f87fc06ab30 0x7f87fc06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f881c072780 tx=0x7f8808008040 comp rx=0 tx=0).ready entity=mgr.11466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:54.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.319+0000 7f8812a00700 1 -- 172.21.15.89:0/3130828575 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f881c14dec0 con 0x7f881c06c910 2024-04-02T09:12:54.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.547+0000 7f8821800700 1 -- 172.21.15.89:0/3130828575 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f881c14c690 con 0x7f87fc06ab30 2024-04-02T09:12:54.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.597+0000 7f8812a00700 1 -- 172.21.15.89:0/3130828575 <== mgr.11466 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f881c14c690 con 0x7f87fc06ab30 2024-04-02T09:12:54.595 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2024-04-02T09:12:54.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.599+0000 7f8821800700 1 -- 172.21.15.89:0/3130828575 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f87fc06ab30 msgr2=0x7f87fc06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:54.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.599+0000 7f8821800700 1 --2- 172.21.15.89:0/3130828575 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f87fc06ab30 0x7f87fc06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f881c072780 tx=0x7f8808008040 comp rx=0 tx=0).stop 2024-04-02T09:12:54.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.599+0000 7f8821800700 1 -- 172.21.15.89:0/3130828575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881c06c910 msgr2=0x7f881c098b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:54.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.599+0000 7f8821800700 1 --2- 172.21.15.89:0/3130828575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881c06c910 0x7f881c098b40 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f880c000c00 tx=0x7f880c004750 comp rx=0 tx=0).stop 2024-04-02T09:12:54.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.599+0000 7f8821800700 1 -- 172.21.15.89:0/3130828575 shutdown_connections 2024-04-02T09:12:54.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.599+0000 7f8821800700 1 --2- 172.21.15.89:0/3130828575 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f87fc06ab30 0x7f87fc06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.599+0000 7f8821800700 1 --2- 172.21.15.89:0/3130828575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f881c155ac0 0x7f881c14d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.599+0000 7f8821800700 1 --2- 172.21.15.89:0/3130828575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f881c06c910 0x7f881c098b40 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.599+0000 7f8821800700 1 --2- 172.21.15.89:0/3130828575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f881c0ae4a0 0x7f881c14b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.600+0000 7f8821800700 1 -- 172.21.15.89:0/3130828575 >> 172.21.15.89:0/3130828575 conn(0x7f881c0cf210 msgr2=0x7f881c071c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:54.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.600+0000 7f8821800700 1 -- 172.21.15.89:0/3130828575 shutdown_connections 2024-04-02T09:12:54.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.600+0000 7f8821800700 1 -- 172.21.15.89:0/3130828575 wait complete. 2024-04-02T09:12:54.616 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2024-04-02T09:12:54.616 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-02T09:12:54.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.819+0000 7f7b35e00700 1 Processor -- start 2024-04-02T09:12:54.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.819+0000 7f7b35e00700 1 -- start start 2024-04-02T09:12:54.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.820+0000 7f7b35e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b300d3f20 0x7f7b300cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:54.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.820+0000 7f7b35e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b30161230 con 0x7f7b30160e50 2024-04-02T09:12:54.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.820+0000 7f7b35e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b300d2a80 con 0x7f7b300b3f80 2024-04-02T09:12:54.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.820+0000 7f7b35e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b300d43b0 con 0x7f7b300d3f20 2024-04-02T09:12:54.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.820+0000 7f7b2e000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b300d3f20 0x7f7b300cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:54.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.820+0000 7f7b2e000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b300d3f20 0x7f7b300cdef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42798/0 (socket says 172.21.15.89:42798) 2024-04-02T09:12:54.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.820+0000 7f7b2e000700 1 -- 172.21.15.89:0/1363688511 learned_addr learned my addr 172.21.15.89:0/1363688511 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:54.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.820+0000 7f7b2ea00700 1 --1- 172.21.15.89:0/1363688511 >> v1:172.21.15.89:6789/0 conn(0x7f7b30160e50 0x7f7b300b4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40702/0 (socket says 172.21.15.89:40702) 2024-04-02T09:12:54.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.820+0000 7f7b2e000700 1 -- 172.21.15.89:0/1363688511 >> v1:172.21.15.113:6789/0 conn(0x7f7b300b3f80 legacy=0x7f7b300ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:54.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.821+0000 7f7b2e000700 1 -- 172.21.15.89:0/1363688511 >> v1:172.21.15.89:6789/0 conn(0x7f7b30160e50 legacy=0x7f7b300b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:54.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.821+0000 7f7b2e000700 1 -- 172.21.15.89:0/1363688511 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b300d21e0 con 0x7f7b300d3f20 2024-04-02T09:12:54.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.821+0000 7f7b2d600700 1 -- 172.21.15.89:0/1363688511 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942502958 0 0) 0x7f7b30161230 con 0x7f7b30160e50 2024-04-02T09:12:54.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.821+0000 7f7b2e000700 1 --2- 172.21.15.89:0/1363688511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b300d3f20 0x7f7b300cdef0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c0098f0 tx=0x7f7b1c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f9d4c95e1aab081 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:54.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.821+0000 7f7b2d600700 1 -- 172.21.15.89:0/1363688511 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c004230 con 0x7f7b300d3f20 2024-04-02T09:12:54.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.821+0000 7f7b2d600700 1 -- 172.21.15.89:0/1363688511 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c004390 con 0x7f7b300d3f20 2024-04-02T09:12:54.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.821+0000 7f7b2d600700 1 -- 172.21.15.89:0/1363688511 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c0036a0 con 0x7f7b300d3f20 2024-04-02T09:12:54.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.822+0000 7f7b35e00700 1 -- 172.21.15.89:0/1363688511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b300d3f20 msgr2=0x7f7b300cdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:54.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.822+0000 7f7b35e00700 1 --2- 172.21.15.89:0/1363688511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b300d3f20 0x7f7b300cdef0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c0098f0 tx=0x7f7b1c009c00 comp rx=0 tx=0).stop 2024-04-02T09:12:54.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.822+0000 7f7b35e00700 1 -- 172.21.15.89:0/1363688511 shutdown_connections 2024-04-02T09:12:54.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.822+0000 7f7b35e00700 1 --2- 172.21.15.89:0/1363688511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b300d3f20 0x7f7b300cdef0 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.822+0000 7f7b35e00700 1 -- 172.21.15.89:0/1363688511 >> 172.21.15.89:0/1363688511 conn(0x7f7b3008aff0 msgr2=0x7f7b300d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:54.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.822+0000 7f7b35e00700 1 -- 172.21.15.89:0/1363688511 shutdown_connections 2024-04-02T09:12:54.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.822+0000 7f7b35e00700 1 -- 172.21.15.89:0/1363688511 wait complete. 2024-04-02T09:12:54.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.823+0000 7f7b35e00700 1 Processor -- start 2024-04-02T09:12:54.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.824+0000 7f7b35e00700 1 -- start start 2024-04-02T09:12:54.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.824+0000 7f7b35e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b300b3f80 0x7f7b3015b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:54.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.824+0000 7f7b35e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b30160e50 0x7f7b3006c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:54.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b35e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b300ca0d0 0x7f7b300cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:54.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b35e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b300cb8e0 con 0x7f7b30160e50 2024-04-02T09:12:54.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b35e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b300cb0b0 con 0x7f7b300ca0d0 2024-04-02T09:12:54.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b35e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b300cb320 con 0x7f7b300b3f80 2024-04-02T09:12:54.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b2e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b30160e50 0x7f7b3006c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:54.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b2e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b30160e50 0x7f7b3006c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42808/0 (socket says 172.21.15.89:42808) 2024-04-02T09:12:54.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b2e000700 1 -- 172.21.15.89:0/320717253 learned_addr learned my addr 172.21.15.89:0/320717253 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:54.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b2f400700 1 --2- 172.21.15.89:0/320717253 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b300ca0d0 0x7f7b300cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b2ea00700 1 --2- 172.21.15.89:0/320717253 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b300b3f80 0x7f7b3015b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b2e000700 1 -- 172.21.15.89:0/320717253 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b300b3f80 msgr2=0x7f7b3015b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b2e000700 1 --2- 172.21.15.89:0/320717253 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b300b3f80 0x7f7b3015b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b2e000700 1 -- 172.21.15.89:0/320717253 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b300ca0d0 msgr2=0x7f7b300cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b2e000700 1 --2- 172.21.15.89:0/320717253 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b300ca0d0 0x7f7b300cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b2e000700 1 -- 172.21.15.89:0/320717253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1c009550 con 0x7f7b30160e50 2024-04-02T09:12:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.825+0000 7f7b2e000700 1 --2- 172.21.15.89:0/320717253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b30160e50 0x7f7b3006c4a0 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f7b3008ba30 tx=0x7f7b1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.826+0000 7f7b27400700 1 -- 172.21.15.89:0/320717253 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c003e90 con 0x7f7b30160e50 2024-04-02T09:12:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.826+0000 7f7b27400700 1 -- 172.21.15.89:0/320717253 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c00b660 con 0x7f7b30160e50 2024-04-02T09:12:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.826+0000 7f7b27400700 1 -- 172.21.15.89:0/320717253 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c00ba10 con 0x7f7b30160e50 2024-04-02T09:12:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.826+0000 7f7b35e00700 1 -- 172.21.15.89:0/320717253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b300cb600 con 0x7f7b30160e50 2024-04-02T09:12:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.826+0000 7f7b35e00700 1 -- 172.21.15.89:0/320717253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3015be90 con 0x7f7b30160e50 2024-04-02T09:12:54.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.827+0000 7f7b35e00700 1 -- 172.21.15.89:0/320717253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b300b7510 con 0x7f7b30160e50 2024-04-02T09:12:54.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.828+0000 7f7b27400700 1 -- 172.21.15.89:0/320717253 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 150) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f7b1c00bb70 con 0x7f7b30160e50 2024-04-02T09:12:54.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.828+0000 7f7b27400700 1 --2- 172.21.15.89:0/320717253 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7b1006ab30 0x7f7b1006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:54.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.829+0000 7f7b27400700 1 -- 172.21.15.89:0/320717253 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f7b1c08ae80 con 0x7f7b30160e50 2024-04-02T09:12:54.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.829+0000 7f7b2ea00700 1 --2- 172.21.15.89:0/320717253 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7b1006ab30 0x7f7b1006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:54.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.829+0000 7f7b2ea00700 1 --2- 172.21.15.89:0/320717253 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7b1006ab30 0x7f7b1006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7b20000f60 tx=0x7f7b20007c40 comp rx=0 tx=0).ready entity=mgr.11466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:54.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:54.833+0000 7f7b27400700 1 -- 172.21.15.89:0/320717253 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7b300b7510 con 0x7f7b30160e50 2024-04-02T09:12:55.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.063+0000 7f7b35e00700 1 -- 172.21.15.89:0/320717253 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b300b0100 con 0x7f7b1006ab30 2024-04-02T09:12:55.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.126+0000 7f7b27400700 1 -- 172.21.15.89:0/320717253 <== mgr.11466 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f7b20005aa0 con 0x7f7b1006ab30 2024-04-02T09:12:55.125 INFO:teuthology.orchestra.run.smithi089.stdout:Retention added to path mandatory_fs_argument_test_dir 2024-04-02T09:12:55.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.129+0000 7f7b35e00700 1 -- 172.21.15.89:0/320717253 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7b1006ab30 msgr2=0x7f7b1006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:55.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.129+0000 7f7b35e00700 1 --2- 172.21.15.89:0/320717253 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7b1006ab30 0x7f7b1006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7b20000f60 tx=0x7f7b20007c40 comp rx=0 tx=0).stop 2024-04-02T09:12:55.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.129+0000 7f7b35e00700 1 -- 172.21.15.89:0/320717253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b30160e50 msgr2=0x7f7b3006c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:55.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.129+0000 7f7b35e00700 1 --2- 172.21.15.89:0/320717253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b30160e50 0x7f7b3006c4a0 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f7b3008ba30 tx=0x7f7b1c004750 comp rx=0 tx=0).stop 2024-04-02T09:12:55.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.129+0000 7f7b35e00700 1 -- 172.21.15.89:0/320717253 shutdown_connections 2024-04-02T09:12:55.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.129+0000 7f7b35e00700 1 --2- 172.21.15.89:0/320717253 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7b1006ab30 0x7f7b1006cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.129+0000 7f7b35e00700 1 --2- 172.21.15.89:0/320717253 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b300ca0d0 0x7f7b300cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.129+0000 7f7b35e00700 1 --2- 172.21.15.89:0/320717253 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b300b3f80 0x7f7b3015b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.129+0000 7f7b35e00700 1 --2- 172.21.15.89:0/320717253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b30160e50 0x7f7b3006c4a0 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.129+0000 7f7b35e00700 1 -- 172.21.15.89:0/320717253 >> 172.21.15.89:0/320717253 conn(0x7f7b3008aff0 msgr2=0x7f7b300ce6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:55.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.129+0000 7f7b35e00700 1 -- 172.21.15.89:0/320717253 shutdown_connections 2024-04-02T09:12:55.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.129+0000 7f7b35e00700 1 -- 172.21.15.89:0/320717253 wait complete. 2024-04-02T09:12:55.147 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2024-04-02T09:12:55.147 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-04-02T09:12:55.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.350+0000 7fb38a600700 1 Processor -- start 2024-04-02T09:12:55.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.350+0000 7fb38a600700 1 -- start start 2024-04-02T09:12:55.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.351+0000 7fb38a600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb38415d930 0x7fb3840d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:55.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.351+0000 7fb38a600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb3840d22a0 0x7fb3840ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:55.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.351+0000 7fb38a600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38415dd10 con 0x7fb384162030 2024-04-02T09:12:55.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.351+0000 7fb38a600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38415ec40 con 0x7fb3840d22a0 2024-04-02T09:12:55.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.351+0000 7fb38a600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38415dfc0 con 0x7fb38415d930 2024-04-02T09:12:55.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.352+0000 7fb383e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb3840d22a0 0x7fb3840ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:55.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.352+0000 7fb383400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb38415d930 0x7fb3840d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:55.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.352+0000 7fb382a00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb384162030 0x7fb3840d1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:35978/0 (socket says 172.21.15.89:35978) 2024-04-02T09:12:55.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.352+0000 7fb382a00700 1 -- 172.21.15.89:0/4252201232 learned_addr learned my addr 172.21.15.89:0/4252201232 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:55.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.352+0000 7fb383e00700 1 -- 172.21.15.89:0/4252201232 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb38415d930 msgr2=0x7fb3840d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:55.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.352+0000 7fb383e00700 1 --2- 172.21.15.89:0/4252201232 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb38415d930 0x7fb3840d3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.352+0000 7fb383e00700 1 -- 172.21.15.89:0/4252201232 >> v1:172.21.15.113:6789/0 conn(0x7fb384162030 legacy=0x7fb3840d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:55.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.352+0000 7fb383e00700 1 -- 172.21.15.89:0/4252201232 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38415e2a0 con 0x7fb3840d22a0 2024-04-02T09:12:55.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.352+0000 7fb383400700 1 --2- 172.21.15.89:0/4252201232 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb38415d930 0x7fb3840d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:12:55.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.352+0000 7fb382000700 1 -- 172.21.15.89:0/4252201232 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729730402 0 0) 0x7fb38415dd10 con 0x7fb384162030 2024-04-02T09:12:55.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.352+0000 7fb383e00700 1 --2- 172.21.15.89:0/4252201232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3840d22a0 0x7fb3840ce300 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009d90 tx=0x7fb37c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50bd188a9a3950f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:55.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.353+0000 7fb382000700 1 -- 172.21.15.89:0/4252201232 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c00e070 con 0x7fb3840d22a0 2024-04-02T09:12:55.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.353+0000 7fb382000700 1 -- 172.21.15.89:0/4252201232 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37c00bb20 con 0x7fb3840d22a0 2024-04-02T09:12:55.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.353+0000 7fb382000700 1 -- 172.21.15.89:0/4252201232 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c012430 con 0x7fb3840d22a0 2024-04-02T09:12:55.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.353+0000 7fb38a600700 1 -- 172.21.15.89:0/4252201232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3840d22a0 msgr2=0x7fb3840ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:55.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.353+0000 7fb38a600700 1 --2- 172.21.15.89:0/4252201232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3840d22a0 0x7fb3840ce300 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009d90 tx=0x7fb37c00b330 comp rx=0 tx=0).stop 2024-04-02T09:12:55.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.353+0000 7fb38a600700 1 -- 172.21.15.89:0/4252201232 shutdown_connections 2024-04-02T09:12:55.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.353+0000 7fb38a600700 1 --2- 172.21.15.89:0/4252201232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3840d22a0 0x7fb3840ce300 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.353+0000 7fb38a600700 1 --2- 172.21.15.89:0/4252201232 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb38415d930 0x7fb3840d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.353+0000 7fb38a600700 1 -- 172.21.15.89:0/4252201232 >> 172.21.15.89:0/4252201232 conn(0x7fb38408aff0 msgr2=0x7fb3840d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:55.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.353+0000 7fb38a600700 1 -- 172.21.15.89:0/4252201232 shutdown_connections 2024-04-02T09:12:55.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.353+0000 7fb38a600700 1 -- 172.21.15.89:0/4252201232 wait complete. 2024-04-02T09:12:55.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.355+0000 7fb38a600700 1 Processor -- start 2024-04-02T09:12:55.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.355+0000 7fb38a600700 1 -- start start 2024-04-02T09:12:55.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.356+0000 7fb38a600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3840d22a0 0x7fb3840cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:55.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.356+0000 7fb38a600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb38415d930 0x7fb38415aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:55.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.356+0000 7fb38a600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb384162030 0x7fb3840b8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:55.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.356+0000 7fb38a600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3840b2f80 con 0x7fb3840d22a0 2024-04-02T09:12:55.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.356+0000 7fb38a600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3840cc3a0 con 0x7fb38415d930 2024-04-02T09:12:55.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.356+0000 7fb38a600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38415af70 con 0x7fb384162030 2024-04-02T09:12:55.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.356+0000 7fb383400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3840d22a0 0x7fb3840cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:55.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.356+0000 7fb383400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3840d22a0 0x7fb3840cbe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42818/0 (socket says 172.21.15.89:42818) 2024-04-02T09:12:55.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.356+0000 7fb383400700 1 -- 172.21.15.89:0/1977952593 learned_addr learned my addr 172.21.15.89:0/1977952593 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:55.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.357+0000 7fb382a00700 1 --2- 172.21.15.89:0/1977952593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb38415d930 0x7fb38415aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:55.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.357+0000 7fb383e00700 1 --2- 172.21.15.89:0/1977952593 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb384162030 0x7fb3840b8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:55.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.357+0000 7fb383400700 1 -- 172.21.15.89:0/1977952593 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb384162030 msgr2=0x7fb3840b8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:55.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.357+0000 7fb383400700 1 --2- 172.21.15.89:0/1977952593 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb384162030 0x7fb3840b8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.357+0000 7fb383400700 1 -- 172.21.15.89:0/1977952593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb38415d930 msgr2=0x7fb38415aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:55.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.357+0000 7fb383400700 1 --2- 172.21.15.89:0/1977952593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb38415d930 0x7fb38415aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.357+0000 7fb383400700 1 -- 172.21.15.89:0/1977952593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3740094a0 con 0x7fb3840d22a0 2024-04-02T09:12:55.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.357+0000 7fb383400700 1 --2- 172.21.15.89:0/1977952593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3840d22a0 0x7fb3840cbe60 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7fb37400ebe0 tx=0x7fb37400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:55.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.357+0000 7fb37b400700 1 -- 172.21.15.89:0/1977952593 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37400c850 con 0x7fb3840d22a0 2024-04-02T09:12:55.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.357+0000 7fb37b400700 1 -- 172.21.15.89:0/1977952593 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37400c9b0 con 0x7fb3840d22a0 2024-04-02T09:12:55.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.357+0000 7fb37b400700 1 -- 172.21.15.89:0/1977952593 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37400cd40 con 0x7fb3840d22a0 2024-04-02T09:12:55.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.357+0000 7fb38a600700 1 -- 172.21.15.89:0/1977952593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37c009a70 con 0x7fb3840d22a0 2024-04-02T09:12:55.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.358+0000 7fb38a600700 1 -- 172.21.15.89:0/1977952593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3840b34d0 con 0x7fb3840d22a0 2024-04-02T09:12:55.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.359+0000 7fb38a600700 1 -- 172.21.15.89:0/1977952593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb384067720 con 0x7fb3840d22a0 2024-04-02T09:12:55.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.359+0000 7fb37b400700 1 -- 172.21.15.89:0/1977952593 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 151) v1 ==== 89359+0+0 (secure 0 0 0) 0x7fb374013470 con 0x7fb3840d22a0 2024-04-02T09:12:55.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.360+0000 7fb37b400700 1 --2- 172.21.15.89:0/1977952593 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb36406af20 0x7fb36406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:55.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.360+0000 7fb37b400700 1 -- 172.21.15.89:0/1977952593 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7fb374089f00 con 0x7fb3840d22a0 2024-04-02T09:12:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.361+0000 7fb382a00700 1 --2- 172.21.15.89:0/1977952593 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb36406af20 0x7fb36406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.361+0000 7fb382a00700 1 --2- 172.21.15.89:0/1977952593 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb36406af20 0x7fb36406d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb3840a13c0 tx=0x7fb370007e70 comp rx=0 tx=0).ready entity=mgr.11466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:55.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.365+0000 7fb37b400700 1 -- 172.21.15.89:0/1977952593 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb3740546f0 con 0x7fb3840d22a0 2024-04-02T09:12:55.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.590+0000 7fb38a600700 1 -- 172.21.15.89:0/1977952593 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb3840d4410 con 0x7fb36406af20 2024-04-02T09:12:55.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.597+0000 7fb37b400700 1 -- 172.21.15.89:0/1977952593 <== mgr.11466 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+88 (secure 0 0 0) 0x7fb370004ab0 con 0x7fb36406af20 2024-04-02T09:12:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:55.595 INFO:teuthology.orchestra.run.smithi089.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-04-02T09:12:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.599+0000 7fb38a600700 1 -- 172.21.15.89:0/1977952593 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb36406af20 msgr2=0x7fb36406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.599+0000 7fb38a600700 1 --2- 172.21.15.89:0/1977952593 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb36406af20 0x7fb36406d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb3840a13c0 tx=0x7fb370007e70 comp rx=0 tx=0).stop 2024-04-02T09:12:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.599+0000 7fb38a600700 1 -- 172.21.15.89:0/1977952593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3840d22a0 msgr2=0x7fb3840cbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.599+0000 7fb38a600700 1 --2- 172.21.15.89:0/1977952593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3840d22a0 0x7fb3840cbe60 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7fb37400ebe0 tx=0x7fb37400eef0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.599+0000 7fb38a600700 1 -- 172.21.15.89:0/1977952593 shutdown_connections 2024-04-02T09:12:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.599+0000 7fb38a600700 1 --2- 172.21.15.89:0/1977952593 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb384162030 0x7fb3840b8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.599+0000 7fb38a600700 1 --2- 172.21.15.89:0/1977952593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3840d22a0 0x7fb3840cbe60 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.599+0000 7fb38a600700 1 --2- 172.21.15.89:0/1977952593 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb36406af20 0x7fb36406d3e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.599+0000 7fb38a600700 1 --2- 172.21.15.89:0/1977952593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb38415d930 0x7fb38415aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.599+0000 7fb38a600700 1 -- 172.21.15.89:0/1977952593 >> 172.21.15.89:0/1977952593 conn(0x7fb38408aff0 msgr2=0x7fb3840b0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.599+0000 7fb38a600700 1 -- 172.21.15.89:0/1977952593 shutdown_connections 2024-04-02T09:12:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.599+0000 7fb38a600700 1 -- 172.21.15.89:0/1977952593 wait complete. 2024-04-02T09:12:55.616 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-04-02T09:12:55.616 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-04-02T09:12:55.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.813+0000 7f7400400700 1 Processor -- start 2024-04-02T09:12:55.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.813+0000 7f7400400700 1 -- start start 2024-04-02T09:12:55.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.814+0000 7f7400400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73f8071340 0x7f73f80b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:55.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.814+0000 7f7400400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f73f8070980 0x7f73f806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:55.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.815+0000 7f7400400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f80b2ff0 con 0x7f73f8159d10 2024-04-02T09:12:55.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.815+0000 7f7400400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73f80b3e50 con 0x7f73f8070980 2024-04-02T09:12:55.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.815+0000 7f7400400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f80b3240 con 0x7f73f8071340 2024-04-02T09:12:55.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.815+0000 7f73fda00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73f8071340 0x7f73f80b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:55.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.815+0000 7f73fe400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f73f8070980 0x7f73f806d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:55.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.815+0000 7f73fe400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f73f8070980 0x7f73f806d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:48972/0 (socket says 172.21.15.89:48972) 2024-04-02T09:12:55.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.815+0000 7f73fe400700 1 -- 172.21.15.89:0/2967860813 learned_addr learned my addr 172.21.15.89:0/2967860813 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:55.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.816+0000 7f73f7e00700 1 -- 172.21.15.89:0/2967860813 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270570497 0 0) 0x7f73f80b2ff0 con 0x7f73f8159d10 2024-04-02T09:12:55.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.816+0000 7f73fda00700 1 -- 172.21.15.89:0/2967860813 >> v2:172.21.15.113:3300/0 conn(0x7f73f8070980 msgr2=0x7f73f806d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:55.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.816+0000 7f73fda00700 1 --2- 172.21.15.89:0/2967860813 >> v2:172.21.15.113:3300/0 conn(0x7f73f8070980 0x7f73f806d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.816+0000 7f73fda00700 1 -- 172.21.15.89:0/2967860813 >> v1:172.21.15.113:6789/0 conn(0x7f73f8159d10 legacy=0x7f73f8071720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:55.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.816+0000 7f73fda00700 1 -- 172.21.15.89:0/2967860813 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f80b3520 con 0x7f73f8071340 2024-04-02T09:12:55.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.816+0000 7f73fda00700 1 --2- 172.21.15.89:0/2967860813 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73f8071340 0x7f73f80b42e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f73ec009840 tx=0x7f73ec009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc968c40cd4d408c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:55.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.816+0000 7f73f7e00700 1 -- 172.21.15.89:0/2967860813 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec011830 con 0x7f73f8071340 2024-04-02T09:12:55.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.816+0000 7f73f7e00700 1 -- 172.21.15.89:0/2967860813 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ec011990 con 0x7f73f8071340 2024-04-02T09:12:55.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.816+0000 7f73f7e00700 1 -- 172.21.15.89:0/2967860813 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec011d20 con 0x7f73f8071340 2024-04-02T09:12:55.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.817+0000 7f7400400700 1 -- 172.21.15.89:0/2967860813 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73f8071340 msgr2=0x7f73f80b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:55.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.817+0000 7f7400400700 1 --2- 172.21.15.89:0/2967860813 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73f8071340 0x7f73f80b42e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f73ec009840 tx=0x7f73ec009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:55.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.817+0000 7f7400400700 1 -- 172.21.15.89:0/2967860813 shutdown_connections 2024-04-02T09:12:55.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.817+0000 7f7400400700 1 --2- 172.21.15.89:0/2967860813 >> v2:172.21.15.113:3300/0 conn(0x7f73f8070980 0x7f73f806d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.817+0000 7f7400400700 1 --2- 172.21.15.89:0/2967860813 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73f8071340 0x7f73f80b42e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.817+0000 7f7400400700 1 -- 172.21.15.89:0/2967860813 >> 172.21.15.89:0/2967860813 conn(0x7f73f80cf210 msgr2=0x7f73f80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:55.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.817+0000 7f7400400700 1 -- 172.21.15.89:0/2967860813 shutdown_connections 2024-04-02T09:12:55.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.817+0000 7f7400400700 1 -- 172.21.15.89:0/2967860813 wait complete. 2024-04-02T09:12:55.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.819+0000 7f7400400700 1 Processor -- start 2024-04-02T09:12:55.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.819+0000 7f7400400700 1 -- start start 2024-04-02T09:12:55.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f7400400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73f8070980 0x7f73f814d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:55.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f7400400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73f8071340 0x7f73f81e4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:55.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f7400400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73f8159d10 0x7f73f80cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:55.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f7400400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f815b930 con 0x7f73f8070980 2024-04-02T09:12:55.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f7400400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f81e5b90 con 0x7f73f8071340 2024-04-02T09:12:55.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f7400400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f81e4f80 con 0x7f73f8159d10 2024-04-02T09:12:55.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f73fda00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73f8070980 0x7f73f814d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f73fda00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73f8070980 0x7f73f814d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42820/0 (socket says 172.21.15.89:42820) 2024-04-02T09:12:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f73fda00700 1 -- 172.21.15.89:0/1085348862 learned_addr learned my addr 172.21.15.89:0/1085348862 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f73fe400700 1 --2- 172.21.15.89:0/1085348862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73f8159d10 0x7f73f80cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f73fd000700 1 --2- 172.21.15.89:0/1085348862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73f8071340 0x7f73f81e4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f73fda00700 1 -- 172.21.15.89:0/1085348862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73f8159d10 msgr2=0x7f73f80cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f73fda00700 1 --2- 172.21.15.89:0/1085348862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73f8159d10 0x7f73f80cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f73fda00700 1 -- 172.21.15.89:0/1085348862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73f8071340 msgr2=0x7f73f81e4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f73fda00700 1 --2- 172.21.15.89:0/1085348862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73f8071340 0x7f73f81e4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.820+0000 7f73fda00700 1 -- 172.21.15.89:0/1085348862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ec0094a0 con 0x7f73f8070980 2024-04-02T09:12:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.821+0000 7f73fda00700 1 --2- 172.21.15.89:0/1085348862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73f8070980 0x7f73f814d090 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f73ec009fd0 tx=0x7f73ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.821+0000 7f73f5600700 1 -- 172.21.15.89:0/1085348862 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec0036a0 con 0x7f73f8070980 2024-04-02T09:12:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.821+0000 7f73f5600700 1 -- 172.21.15.89:0/1085348862 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ec01f070 con 0x7f73f8070980 2024-04-02T09:12:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.821+0000 7f7400400700 1 -- 172.21.15.89:0/1085348862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f81e5260 con 0x7f73f8070980 2024-04-02T09:12:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.821+0000 7f73f5600700 1 -- 172.21.15.89:0/1085348862 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec003a60 con 0x7f73f8070980 2024-04-02T09:12:55.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.821+0000 7f7400400700 1 -- 172.21.15.89:0/1085348862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f81e5820 con 0x7f73f8070980 2024-04-02T09:12:55.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.822+0000 7f7400400700 1 -- 172.21.15.89:0/1085348862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f81e59b0 con 0x7f73f8070980 2024-04-02T09:12:55.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.823+0000 7f73f5600700 1 -- 172.21.15.89:0/1085348862 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 151) v1 ==== 89359+0+0 (secure 0 0 0) 0x7f73ec003bc0 con 0x7f73f8070980 2024-04-02T09:12:55.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.824+0000 7f73f5600700 1 --2- 172.21.15.89:0/1085348862 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f73d806af20 0x7f73d806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:55.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.824+0000 7f73f5600700 1 -- 172.21.15.89:0/1085348862 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f73ec021070 con 0x7f73f8070980 2024-04-02T09:12:55.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.824+0000 7f73fd000700 1 --2- 172.21.15.89:0/1085348862 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f73d806af20 0x7f73d806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:55.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.825+0000 7f73fd000700 1 --2- 172.21.15.89:0/1085348862 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f73d806af20 0x7f73d806d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f73f80cb850 tx=0x7f73e4009040 comp rx=0 tx=0).ready entity=mgr.11466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:55.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:55.829+0000 7f73f5600700 1 -- 172.21.15.89:0/1085348862 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f73ec055ff0 con 0x7f73f8070980 2024-04-02T09:12:56.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.056+0000 7f7400400700 1 -- 172.21.15.89:0/1085348862 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f73f815afe0 con 0x7f73d806af20 2024-04-02T09:12:56.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.062+0000 7f73f5600700 1 -- 172.21.15.89:0/1085348862 <== mgr.11466 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+353 (secure 0 0 0) 0x7f73e4001320 con 0x7f73d806af20 2024-04-02T09:12:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:56.061 INFO:teuthology.orchestra.run.smithi089.stdout:[{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T09:12:54", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-04-02T09:12:56.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.065+0000 7f7400400700 1 -- 172.21.15.89:0/1085348862 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f73d806af20 msgr2=0x7f73d806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.065+0000 7f7400400700 1 --2- 172.21.15.89:0/1085348862 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f73d806af20 0x7f73d806d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f73f80cb850 tx=0x7f73e4009040 comp rx=0 tx=0).stop 2024-04-02T09:12:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.065+0000 7f7400400700 1 -- 172.21.15.89:0/1085348862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73f8070980 msgr2=0x7f73f814d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.065+0000 7f7400400700 1 --2- 172.21.15.89:0/1085348862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73f8070980 0x7f73f814d090 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f73ec009fd0 tx=0x7f73ec004750 comp rx=0 tx=0).stop 2024-04-02T09:12:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.065+0000 7f7400400700 1 -- 172.21.15.89:0/1085348862 shutdown_connections 2024-04-02T09:12:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.065+0000 7f7400400700 1 --2- 172.21.15.89:0/1085348862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f73f8159d10 0x7f73f80cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.065+0000 7f7400400700 1 --2- 172.21.15.89:0/1085348862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73f8070980 0x7f73f814d090 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.065+0000 7f7400400700 1 --2- 172.21.15.89:0/1085348862 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f73d806af20 0x7f73d806d3e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.065+0000 7f7400400700 1 --2- 172.21.15.89:0/1085348862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73f8071340 0x7f73f81e4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.065+0000 7f7400400700 1 -- 172.21.15.89:0/1085348862 >> 172.21.15.89:0/1085348862 conn(0x7f73f80cf210 msgr2=0x7f73f80ac4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.065+0000 7f7400400700 1 -- 172.21.15.89:0/1085348862 shutdown_connections 2024-04-02T09:12:56.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.065+0000 7f7400400700 1 -- 172.21.15.89:0/1085348862 wait complete. 2024-04-02T09:12:56.081 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T09:12:54", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-04-02T09:12:56.081 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-02T09:12:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.281+0000 7f445d800700 1 Processor -- start 2024-04-02T09:12:56.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.281+0000 7f445d800700 1 -- start start 2024-04-02T09:12:56.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.282+0000 7f445d800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f445809a750 0x7f44580d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:56.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.282+0000 7f445d800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44580d0970 con 0x7f4458099a60 2024-04-02T09:12:56.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.282+0000 7f445d800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445806c020 con 0x7f445815d990 2024-04-02T09:12:56.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.282+0000 7f445d800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44580c81a0 con 0x7f445809a750 2024-04-02T09:12:56.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.282+0000 7f4457400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f445809a750 0x7f44580d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:56.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.282+0000 7f4456a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4458099a60 0x7f4458099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40716/0 (socket says 172.21.15.89:40716) 2024-04-02T09:12:56.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.282+0000 7f4456a00700 1 -- 172.21.15.89:0/25882249 learned_addr learned my addr 172.21.15.89:0/25882249 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:56.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.283+0000 7f4457400700 1 -- 172.21.15.89:0/25882249 >> v1:172.21.15.113:6789/0 conn(0x7f445815d990 legacy=0x7f445806c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:56.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.283+0000 7f4457400700 1 -- 172.21.15.89:0/25882249 >> v1:172.21.15.89:6789/0 conn(0x7f4458099a60 legacy=0x7f4458099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:56.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.283+0000 7f4455600700 1 -- 172.21.15.89:0/25882249 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827210810 0 0) 0x7f44580d0970 con 0x7f4458099a60 2024-04-02T09:12:56.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.283+0000 7f4457400700 1 -- 172.21.15.89:0/25882249 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44580d0690 con 0x7f445809a750 2024-04-02T09:12:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.283+0000 7f4457400700 1 --2- 172.21.15.89:0/25882249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f445809a750 0x7f44580d0060 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f4450009d90 tx=0x7f445000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb93f4c4ef3e37c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.283+0000 7f4455600700 1 -- 172.21.15.89:0/25882249 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445000e070 con 0x7f445809a750 2024-04-02T09:12:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.283+0000 7f4455600700 1 -- 172.21.15.89:0/25882249 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445000bbd0 con 0x7f445809a750 2024-04-02T09:12:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.283+0000 7f4455600700 1 -- 172.21.15.89:0/25882249 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450003a20 con 0x7f445809a750 2024-04-02T09:12:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.284+0000 7f445d800700 1 -- 172.21.15.89:0/25882249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f445809a750 msgr2=0x7f44580d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.284+0000 7f445d800700 1 --2- 172.21.15.89:0/25882249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f445809a750 0x7f44580d0060 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f4450009d90 tx=0x7f445000b330 comp rx=0 tx=0).stop 2024-04-02T09:12:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.284+0000 7f445d800700 1 -- 172.21.15.89:0/25882249 shutdown_connections 2024-04-02T09:12:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.284+0000 7f445d800700 1 --2- 172.21.15.89:0/25882249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f445809a750 0x7f44580d0060 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.284+0000 7f445d800700 1 -- 172.21.15.89:0/25882249 >> 172.21.15.89:0/25882249 conn(0x7f44580b2e20 msgr2=0x7f44580b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:56.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.284+0000 7f445d800700 1 -- 172.21.15.89:0/25882249 shutdown_connections 2024-04-02T09:12:56.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.284+0000 7f445d800700 1 -- 172.21.15.89:0/25882249 wait complete. 2024-04-02T09:12:56.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.285+0000 7f445d800700 1 Processor -- start 2024-04-02T09:12:56.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.286+0000 7f445d800700 1 -- start start 2024-04-02T09:12:56.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f445d800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4458099a60 0x7f44581f28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f445d800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f445809a750 0x7f44580c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f445d800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f445815d990 0x7f44580c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f445d800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445814c0f0 con 0x7f445815d990 2024-04-02T09:12:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f445d800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44581f2e30 con 0x7f445809a750 2024-04-02T09:12:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f445d800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445814b470 con 0x7f4458099a60 2024-04-02T09:12:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f4457400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f445815d990 0x7f44580c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f4457400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f445815d990 0x7f44580c67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42830/0 (socket says 172.21.15.89:42830) 2024-04-02T09:12:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f4457400700 1 -- 172.21.15.89:0/3508790061 learned_addr learned my addr 172.21.15.89:0/3508790061 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f4456000700 1 --2- 172.21.15.89:0/3508790061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f445809a750 0x7f44580c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f4456a00700 1 --2- 172.21.15.89:0/3508790061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4458099a60 0x7f44581f28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f4457400700 1 -- 172.21.15.89:0/3508790061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4458099a60 msgr2=0x7f44581f28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f4457400700 1 --2- 172.21.15.89:0/3508790061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4458099a60 0x7f44581f28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f4457400700 1 -- 172.21.15.89:0/3508790061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f445809a750 msgr2=0x7f44580c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f4457400700 1 --2- 172.21.15.89:0/3508790061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f445809a750 0x7f44580c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.287+0000 7f4457400700 1 -- 172.21.15.89:0/3508790061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4450009a70 con 0x7f445815d990 2024-04-02T09:12:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.288+0000 7f4457400700 1 --2- 172.21.15.89:0/3508790061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f445815d990 0x7f44580c67f0 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f44580724f0 tx=0x7f4450004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.288+0000 7f444ea00700 1 -- 172.21.15.89:0/3508790061 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450012430 con 0x7f445815d990 2024-04-02T09:12:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.288+0000 7f444ea00700 1 -- 172.21.15.89:0/3508790061 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44500125b0 con 0x7f445815d990 2024-04-02T09:12:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.288+0000 7f445d800700 1 -- 172.21.15.89:0/3508790061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445814b750 con 0x7f445815d990 2024-04-02T09:12:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.288+0000 7f444ea00700 1 -- 172.21.15.89:0/3508790061 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450012960 con 0x7f445815d990 2024-04-02T09:12:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.288+0000 7f445d800700 1 -- 172.21.15.89:0/3508790061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f445814bd10 con 0x7f445815d990 2024-04-02T09:12:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.289+0000 7f445d800700 1 -- 172.21.15.89:0/3508790061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445814c230 con 0x7f445815d990 2024-04-02T09:12:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.290+0000 7f444ea00700 1 -- 172.21.15.89:0/3508790061 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 151) v1 ==== 89359+0+0 (secure 0 0 0) 0x7f4450012ac0 con 0x7f445815d990 2024-04-02T09:12:56.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.291+0000 7f444ea00700 1 --2- 172.21.15.89:0/3508790061 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f443806af20 0x7f443806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:56.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.291+0000 7f444ea00700 1 -- 172.21.15.89:0/3508790061 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f445008c5b0 con 0x7f445815d990 2024-04-02T09:12:56.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.291+0000 7f4456a00700 1 --2- 172.21.15.89:0/3508790061 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f443806af20 0x7f443806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:56.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.292+0000 7f4456a00700 1 --2- 172.21.15.89:0/3508790061 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f443806af20 0x7f443806d3e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4448007fd0 tx=0x7f4448007de0 comp rx=0 tx=0).ready entity=mgr.11466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:56.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.295+0000 7f444ea00700 1 -- 172.21.15.89:0/3508790061 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f445814c230 con 0x7f445815d990 2024-04-02T09:12:56.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.521+0000 7f445d800700 1 -- 172.21.15.89:0/3508790061 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f445806cbc0 con 0x7f443806af20 2024-04-02T09:12:56.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.583+0000 7f444ea00700 1 -- 172.21.15.89:0/3508790061 <== mgr.11466 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f4448004b10 con 0x7f443806af20 2024-04-02T09:12:56.581 INFO:teuthology.orchestra.run.smithi089.stdout:Retention removed from path mandatory_fs_argument_test_dir 2024-04-02T09:12:56.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.585+0000 7f445d800700 1 -- 172.21.15.89:0/3508790061 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f443806af20 msgr2=0x7f443806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:56.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.585+0000 7f445d800700 1 --2- 172.21.15.89:0/3508790061 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f443806af20 0x7f443806d3e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4448007fd0 tx=0x7f4448007de0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.585+0000 7f445d800700 1 -- 172.21.15.89:0/3508790061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f445815d990 msgr2=0x7f44580c67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:56.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.585+0000 7f445d800700 1 --2- 172.21.15.89:0/3508790061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f445815d990 0x7f44580c67f0 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f44580724f0 tx=0x7f4450004ad0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.585+0000 7f445d800700 1 -- 172.21.15.89:0/3508790061 shutdown_connections 2024-04-02T09:12:56.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.585+0000 7f445d800700 1 --2- 172.21.15.89:0/3508790061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f445815d990 0x7f44580c67f0 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.585+0000 7f445d800700 1 --2- 172.21.15.89:0/3508790061 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4458099a60 0x7f44581f28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.585+0000 7f445d800700 1 --2- 172.21.15.89:0/3508790061 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f443806af20 0x7f443806d3e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.585+0000 7f445d800700 1 --2- 172.21.15.89:0/3508790061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f445809a750 0x7f44580c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.585+0000 7f445d800700 1 -- 172.21.15.89:0/3508790061 >> 172.21.15.89:0/3508790061 conn(0x7f44580b2e20 msgr2=0x7f44580b6800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:56.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.585+0000 7f445d800700 1 -- 172.21.15.89:0/3508790061 shutdown_connections 2024-04-02T09:12:56.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.585+0000 7f445d800700 1 -- 172.21.15.89:0/3508790061 wait complete. 2024-04-02T09:12:56.601 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2024-04-02T09:12:56.601 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-02T09:12:56.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.800+0000 7f2245a00700 1 Processor -- start 2024-04-02T09:12:56.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.801+0000 7f2245a00700 1 -- start start 2024-04-02T09:12:56.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.802+0000 7f2245a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f224015db20 0x7f22400d1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:56.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.802+0000 7f2245a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22400ced30 con 0x7f224015d740 2024-04-02T09:12:56.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.802+0000 7f2245a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22400cef80 con 0x7f22400d1fe0 2024-04-02T09:12:56.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.802+0000 7f2245a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22400cf260 con 0x7f224015db20 2024-04-02T09:12:56.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.802+0000 7f223ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f224015d740 0x7f22400d18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40724/0 (socket says 172.21.15.89:40724) 2024-04-02T09:12:56.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.802+0000 7f223ea00700 1 -- 172.21.15.89:0/1926086947 learned_addr learned my addr 172.21.15.89:0/1926086947 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:56.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.802+0000 7f223f400700 1 --2- 172.21.15.89:0/1926086947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f224015db20 0x7f22400d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:56.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.802+0000 7f223d600700 1 -- 172.21.15.89:0/1926086947 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304715314 0 0) 0x7f22400ced30 con 0x7f224015d740 2024-04-02T09:12:56.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.802+0000 7f223d600700 1 -- 172.21.15.89:0/1926086947 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22240035b0 con 0x7f224015d740 2024-04-02T09:12:56.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223d600700 1 -- 172.21.15.89:0/1926086947 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1276390442 0 0) 0x7f22240035b0 con 0x7f224015d740 2024-04-02T09:12:56.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223d600700 1 -- 172.21.15.89:0/1926086947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f224015db20 msgr2=0x7f22400d1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:56.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223d600700 1 --2- 172.21.15.89:0/1926086947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f224015db20 0x7f22400d1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223d600700 1 -- 172.21.15.89:0/1926086947 >> v1:172.21.15.113:6789/0 conn(0x7f22400d1fe0 legacy=0x7f22400b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:56.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223d600700 1 -- 172.21.15.89:0/1926086947 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22400cf540 con 0x7f224015d740 2024-04-02T09:12:56.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223d600700 1 -- 172.21.15.89:0/1926086947 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f2230001670 con 0x7f224015d740 2024-04-02T09:12:56.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223d600700 1 -- 172.21.15.89:0/1926086947 >> v1:172.21.15.89:6789/0 conn(0x7f224015d740 legacy=0x7f22400d18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:56.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223d600700 1 --2- 172.21.15.89:0/1926086947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2224003a30 0x7f2224003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:56.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223d600700 1 --2- 172.21.15.89:0/1926086947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2224004850 0x7f2224004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:56.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223d600700 1 --2- 172.21.15.89:0/1926086947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f224015db20 0x7f2224005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:56.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223d600700 1 -- 172.21.15.89:0/1926086947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400cf260 con 0x7f2224004850 2024-04-02T09:12:56.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223d600700 1 -- 172.21.15.89:0/1926086947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222400bac0 con 0x7f2224003a30 2024-04-02T09:12:56.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223d600700 1 -- 172.21.15.89:0/1926086947 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222400bc80 con 0x7f224015db20 2024-04-02T09:12:56.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223e000700 1 --2- 172.21.15.89:0/1926086947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2224004850 0x7f2224004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:56.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223ea00700 1 --2- 172.21.15.89:0/1926086947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2224003a30 0x7f2224003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:56.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.803+0000 7f223f400700 1 --2- 172.21.15.89:0/1926086947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f224015db20 0x7f2224005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:56.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.804+0000 7f223e000700 1 -- 172.21.15.89:0/1926086947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f224015db20 msgr2=0x7f2224005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:56.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.804+0000 7f223e000700 1 --2- 172.21.15.89:0/1926086947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f224015db20 0x7f2224005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.804+0000 7f223e000700 1 -- 172.21.15.89:0/1926086947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2224003a30 msgr2=0x7f2224003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:56.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.804+0000 7f223e000700 1 --2- 172.21.15.89:0/1926086947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2224003a30 0x7f2224003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.804+0000 7f223e000700 1 -- 172.21.15.89:0/1926086947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22240035b0 con 0x7f2224004850 2024-04-02T09:12:56.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.804+0000 7f223e000700 1 --2- 172.21.15.89:0/1926086947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2224004850 0x7f2224004cd0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f222c007fd0 tx=0x7f2224005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecd44c6518ad97f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:56.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.804+0000 7f223d600700 1 -- 172.21.15.89:0/1926086947 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c0052d0 con 0x7f2224004850 2024-04-02T09:12:56.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.804+0000 7f223d600700 1 -- 172.21.15.89:0/1926086947 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222c005470 con 0x7f2224004850 2024-04-02T09:12:56.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.804+0000 7f223d600700 1 -- 172.21.15.89:0/1926086947 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c0038c0 con 0x7f2224004850 2024-04-02T09:12:56.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.804+0000 7f2245a00700 1 -- 172.21.15.89:0/1926086947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2224004850 msgr2=0x7f2224004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:56.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.804+0000 7f2245a00700 1 --2- 172.21.15.89:0/1926086947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2224004850 0x7f2224004cd0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f222c007fd0 tx=0x7f2224005880 comp rx=0 tx=0).stop 2024-04-02T09:12:56.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.804+0000 7f223f400700 1 -- 172.21.15.89:0/1926086947 reap_dead start 2024-04-02T09:12:56.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.805+0000 7f2245a00700 1 -- 172.21.15.89:0/1926086947 shutdown_connections 2024-04-02T09:12:56.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.805+0000 7f2245a00700 1 -- 172.21.15.89:0/1926086947 >> 172.21.15.89:0/1926086947 conn(0x7f224008aff0 msgr2=0x7f22400d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:56.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.805+0000 7f2245a00700 1 -- 172.21.15.89:0/1926086947 shutdown_connections 2024-04-02T09:12:56.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.805+0000 7f2245a00700 1 -- 172.21.15.89:0/1926086947 wait complete. 2024-04-02T09:12:56.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.806+0000 7f2245a00700 1 Processor -- start 2024-04-02T09:12:56.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.807+0000 7f2245a00700 1 -- start start 2024-04-02T09:12:56.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.807+0000 7f2245a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2224004850 0x7f224015c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:56.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.807+0000 7f2245a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22400a0d20 0x7f22400cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:56.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.807+0000 7f2245a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400ccce0 0x7f22400caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:56.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f2245a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400cd160 con 0x7f2224004850 2024-04-02T09:12:56.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f2245a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400b3270 con 0x7f22400ccce0 2024-04-02T09:12:56.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f2245a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22400a11a0 con 0x7f22400a0d20 2024-04-02T09:12:56.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f223ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2224004850 0x7f224015c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:56.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f223ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2224004850 0x7f224015c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42860/0 (socket says 172.21.15.89:42860) 2024-04-02T09:12:56.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f223ea00700 1 -- 172.21.15.89:0/827975804 learned_addr learned my addr 172.21.15.89:0/827975804 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:56.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f223e000700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22400a0d20 0x7f22400cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:56.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f223f400700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400ccce0 0x7f22400caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:56.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f223ea00700 1 -- 172.21.15.89:0/827975804 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22400a0d20 msgr2=0x7f22400cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:56.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f223ea00700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22400a0d20 0x7f22400cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f223ea00700 1 -- 172.21.15.89:0/827975804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400ccce0 msgr2=0x7f22400caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:56.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f223ea00700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400ccce0 0x7f22400caa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:56.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f223ea00700 1 -- 172.21.15.89:0/827975804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22400b5720 con 0x7f2224004850 2024-04-02T09:12:56.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f223f400700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400ccce0 0x7f22400caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:12:56.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.808+0000 7f223ea00700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2224004850 0x7f224015c130 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f22300072f0 tx=0x7f22300076b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:56.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.809+0000 7f2236a00700 1 -- 172.21.15.89:0/827975804 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22300077d0 con 0x7f2224004850 2024-04-02T09:12:56.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.809+0000 7f2236a00700 1 -- 172.21.15.89:0/827975804 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223000f8d0 con 0x7f2224004850 2024-04-02T09:12:56.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.809+0000 7f2245a00700 1 -- 172.21.15.89:0/827975804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22400b34c0 con 0x7f2224004850 2024-04-02T09:12:56.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.809+0000 7f2236a00700 1 -- 172.21.15.89:0/827975804 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223000fc80 con 0x7f2224004850 2024-04-02T09:12:56.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.809+0000 7f2245a00700 1 -- 172.21.15.89:0/827975804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22400b3a80 con 0x7f2224004850 2024-04-02T09:12:56.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.810+0000 7f2245a00700 1 -- 172.21.15.89:0/827975804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2240067720 con 0x7f2224004850 2024-04-02T09:12:56.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.811+0000 7f2236a00700 1 -- 172.21.15.89:0/827975804 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 151) v1 ==== 89359+0+0 (secure 0 0 0) 0x7f223001e3b0 con 0x7f2224004850 2024-04-02T09:12:56.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.812+0000 7f2236a00700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f222006af20 0x7f222006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:56.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.812+0000 7f2236a00700 1 -- 172.21.15.89:0/827975804 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f22300898e0 con 0x7f2224004850 2024-04-02T09:12:56.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.812+0000 7f223e000700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f222006af20 0x7f222006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:56.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.813+0000 7f223e000700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f222006af20 0x7f222006d3e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f222c007fd0 tx=0x7f222c009030 comp rx=0 tx=0).ready entity=mgr.11466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:56.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:56.816+0000 7f2236a00700 1 -- 172.21.15.89:0/827975804 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f22300540d0 con 0x7f2224004850 2024-04-02T09:12:57.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.049+0000 7f2245a00700 1 -- 172.21.15.89:0/827975804 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f22400cfb00 con 0x7f222006af20 2024-04-02T09:12:57.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.106+0000 7f2236a00700 1 -- 172.21.15.89:0/827975804 <== mgr.11466 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+61 (secure 0 0 0) 0x7f222c0051d0 con 0x7f222006af20 2024-04-02T09:12:57.105 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2024-04-02T09:12:57.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.109+0000 7f2245a00700 1 -- 172.21.15.89:0/827975804 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f222006af20 msgr2=0x7f222006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:57.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.109+0000 7f2245a00700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f222006af20 0x7f222006d3e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f222c007fd0 tx=0x7f222c009030 comp rx=0 tx=0).stop 2024-04-02T09:12:57.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.109+0000 7f2245a00700 1 -- 172.21.15.89:0/827975804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2224004850 msgr2=0x7f224015c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:57.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.109+0000 7f2245a00700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2224004850 0x7f224015c130 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f22300072f0 tx=0x7f22300076b0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.109+0000 7f2245a00700 1 -- 172.21.15.89:0/827975804 shutdown_connections 2024-04-02T09:12:57.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.109+0000 7f2245a00700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f222006af20 0x7f222006d3e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.109+0000 7f2245a00700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400ccce0 0x7f22400caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.109+0000 7f2245a00700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2224004850 0x7f224015c130 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.109+0000 7f2245a00700 1 --2- 172.21.15.89:0/827975804 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22400a0d20 0x7f22400cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.109+0000 7f2245a00700 1 -- 172.21.15.89:0/827975804 >> 172.21.15.89:0/827975804 conn(0x7f224008aff0 msgr2=0x7f22400c6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:57.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.109+0000 7f2245a00700 1 -- 172.21.15.89:0/827975804 shutdown_connections 2024-04-02T09:12:57.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.109+0000 7f2245a00700 1 -- 172.21.15.89:0/827975804 wait complete. 2024-04-02T09:12:57.125 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2024-04-02T09:12:57.125 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-02T09:12:57.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.329+0000 7f44f0800700 1 Processor -- start 2024-04-02T09:12:57.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.329+0000 7f44f0800700 1 -- start start 2024-04-02T09:12:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.330+0000 7f44f0800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f44e8155ac0 0x7f44e815c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.330+0000 7f44f0800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f44e806c910 0x7f44e80adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.330+0000 7f44f0800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e815dcc0 con 0x7f44e80ae4a0 2024-04-02T09:12:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.330+0000 7f44f0800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e806dd00 con 0x7f44e806c910 2024-04-02T09:12:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.330+0000 7f44f0800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e806d0f0 con 0x7f44e8155ac0 2024-04-02T09:12:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.331+0000 7f44ed400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f44e806c910 0x7f44e80adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.331+0000 7f44ed400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f44e806c910 0x7f44e80adf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42866/0 (socket says 172.21.15.89:42866) 2024-04-02T09:12:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.331+0000 7f44ed400700 1 -- 172.21.15.89:0/752799653 learned_addr learned my addr 172.21.15.89:0/752799653 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.331+0000 7f44ede00700 1 --2- 172.21.15.89:0/752799653 >> v2:172.21.15.113:3300/0 conn(0x7f44e8155ac0 0x7f44e815c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.331+0000 7f44ed400700 1 -- 172.21.15.89:0/752799653 >> v2:172.21.15.113:3300/0 conn(0x7f44e8155ac0 msgr2=0x7f44e815c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.331+0000 7f44ed400700 1 --2- 172.21.15.89:0/752799653 >> v2:172.21.15.113:3300/0 conn(0x7f44e8155ac0 0x7f44e815c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.331+0000 7f44ed400700 1 -- 172.21.15.89:0/752799653 >> v1:172.21.15.89:6789/0 conn(0x7f44e80ae4a0 legacy=0x7f44e815f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.331+0000 7f44ed400700 1 -- 172.21.15.89:0/752799653 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e806d3d0 con 0x7f44e806c910 2024-04-02T09:12:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.331+0000 7f44ed400700 1 --2- 172.21.15.89:0/752799653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e806c910 0x7f44e80adf60 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f44e806c4d0 tx=0x7f44d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e8d31517549486 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.332+0000 7f44eca00700 1 -- 172.21.15.89:0/752799653 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d8011830 con 0x7f44e806c910 2024-04-02T09:12:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.332+0000 7f44eca00700 1 -- 172.21.15.89:0/752799653 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d8011990 con 0x7f44e806c910 2024-04-02T09:12:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.332+0000 7f44eca00700 1 -- 172.21.15.89:0/752799653 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d8011d20 con 0x7f44e806c910 2024-04-02T09:12:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.332+0000 7f44f0800700 1 -- 172.21.15.89:0/752799653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e806c910 msgr2=0x7f44e80adf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.332+0000 7f44f0800700 1 --2- 172.21.15.89:0/752799653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e806c910 0x7f44e80adf60 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f44e806c4d0 tx=0x7f44d8009bd0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.332+0000 7f44f0800700 1 -- 172.21.15.89:0/752799653 shutdown_connections 2024-04-02T09:12:57.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.332+0000 7f44f0800700 1 --2- 172.21.15.89:0/752799653 >> v2:172.21.15.113:3300/0 conn(0x7f44e8155ac0 0x7f44e815c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.332+0000 7f44f0800700 1 --2- 172.21.15.89:0/752799653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e806c910 0x7f44e80adf60 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.332+0000 7f44f0800700 1 -- 172.21.15.89:0/752799653 >> 172.21.15.89:0/752799653 conn(0x7f44e80cf210 msgr2=0x7f44e80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:57.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.332+0000 7f44f0800700 1 -- 172.21.15.89:0/752799653 shutdown_connections 2024-04-02T09:12:57.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.332+0000 7f44f0800700 1 -- 172.21.15.89:0/752799653 wait complete. 2024-04-02T09:12:57.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.334+0000 7f44f0800700 1 Processor -- start 2024-04-02T09:12:57.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.334+0000 7f44f0800700 1 -- start start 2024-04-02T09:12:57.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.335+0000 7f44f0800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44e806c910 0x7f44e8145ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.335+0000 7f44f0800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e80ae4a0 0x7f44e80719d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.335+0000 7f44f0800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f44e8155ac0 0x7f44e8071f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.335+0000 7f44f0800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e8141df0 con 0x7f44e80ae4a0 2024-04-02T09:12:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.335+0000 7f44f0800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e8143600 con 0x7f44e806c910 2024-04-02T09:12:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.335+0000 7f44f0800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e81438e0 con 0x7f44e8155ac0 2024-04-02T09:12:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.335+0000 7f44ed400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e80ae4a0 0x7f44e80719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.335+0000 7f44ed400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e80ae4a0 0x7f44e80719d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42878/0 (socket says 172.21.15.89:42878) 2024-04-02T09:12:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.335+0000 7f44ed400700 1 -- 172.21.15.89:0/185882421 learned_addr learned my addr 172.21.15.89:0/185882421 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.336+0000 7f44ede00700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44e806c910 0x7f44e8145ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.336+0000 7f44ee800700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f44e8155ac0 0x7f44e8071f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.336+0000 7f44ed400700 1 -- 172.21.15.89:0/185882421 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f44e8155ac0 msgr2=0x7f44e8071f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.336+0000 7f44ed400700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f44e8155ac0 0x7f44e8071f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.336+0000 7f44ed400700 1 -- 172.21.15.89:0/185882421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44e806c910 msgr2=0x7f44e8145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:57.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.336+0000 7f44ed400700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44e806c910 0x7f44e8145ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.336+0000 7f44ed400700 1 -- 172.21.15.89:0/185882421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d8009550 con 0x7f44e80ae4a0 2024-04-02T09:12:57.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.336+0000 7f44ede00700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44e806c910 0x7f44e8145ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:57.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.336+0000 7f44ed400700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e80ae4a0 0x7f44e80719d0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f44d8000c00 tx=0x7f44d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:57.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.336+0000 7f44e2000700 1 -- 172.21.15.89:0/185882421 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d8004570 con 0x7f44e80ae4a0 2024-04-02T09:12:57.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.336+0000 7f44e2000700 1 -- 172.21.15.89:0/185882421 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d801f070 con 0x7f44e80ae4a0 2024-04-02T09:12:57.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.336+0000 7f44f0800700 1 -- 172.21.15.89:0/185882421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e8143bc0 con 0x7f44e80ae4a0 2024-04-02T09:12:57.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.336+0000 7f44e2000700 1 -- 172.21.15.89:0/185882421 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d80038e0 con 0x7f44e80ae4a0 2024-04-02T09:12:57.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.337+0000 7f44f0800700 1 -- 172.21.15.89:0/185882421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e8146420 con 0x7f44e80ae4a0 2024-04-02T09:12:57.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.338+0000 7f44f0800700 1 -- 172.21.15.89:0/185882421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e81440e0 con 0x7f44e80ae4a0 2024-04-02T09:12:57.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.339+0000 7f44e2000700 1 -- 172.21.15.89:0/185882421 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 152) v1 ==== 89415+0+0 (secure 0 0 0) 0x7f44d8021030 con 0x7f44e80ae4a0 2024-04-02T09:12:57.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.340+0000 7f44e2000700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f44cc06afb0 0x7f44cc06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:57.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.340+0000 7f44e2000700 1 -- 172.21.15.89:0/185882421 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f44d808b2b0 con 0x7f44e80ae4a0 2024-04-02T09:12:57.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.340+0000 7f44ede00700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f44cc06afb0 0x7f44cc06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:57.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.341+0000 7f44ede00700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f44cc06afb0 0x7f44cc06d470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f44dc009b90 tx=0x7f44dc008040 comp rx=0 tx=0).ready entity=mgr.11466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:57.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.344+0000 7f44e2000700 1 -- 172.21.15.89:0/185882421 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f44e81440e0 con 0x7f44e80ae4a0 2024-04-02T09:12:57.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.570+0000 7f44f0800700 1 -- 172.21.15.89:0/185882421 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f44e81465b0 con 0x7f44cc06afb0 2024-04-02T09:12:57.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.631+0000 7f44e2000700 1 -- 172.21.15.89:0/185882421 <== mgr.11466 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f44e81465b0 con 0x7f44cc06afb0 2024-04-02T09:12:57.629 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2024-04-02T09:12:57.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.633+0000 7f44f0800700 1 -- 172.21.15.89:0/185882421 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f44cc06afb0 msgr2=0x7f44cc06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:57.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.633+0000 7f44f0800700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f44cc06afb0 0x7f44cc06d470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f44dc009b90 tx=0x7f44dc008040 comp rx=0 tx=0).stop 2024-04-02T09:12:57.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.633+0000 7f44f0800700 1 -- 172.21.15.89:0/185882421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e80ae4a0 msgr2=0x7f44e80719d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:57.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.633+0000 7f44f0800700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e80ae4a0 0x7f44e80719d0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f44d8000c00 tx=0x7f44d8004750 comp rx=0 tx=0).stop 2024-04-02T09:12:57.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.633+0000 7f44f0800700 1 -- 172.21.15.89:0/185882421 shutdown_connections 2024-04-02T09:12:57.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.633+0000 7f44f0800700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f44cc06afb0 0x7f44cc06d470 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.633+0000 7f44f0800700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44e806c910 0x7f44e8145ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.633+0000 7f44f0800700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f44e8155ac0 0x7f44e8071f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.633+0000 7f44f0800700 1 --2- 172.21.15.89:0/185882421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e80ae4a0 0x7f44e80719d0 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.633+0000 7f44f0800700 1 -- 172.21.15.89:0/185882421 >> 172.21.15.89:0/185882421 conn(0x7f44e80cf210 msgr2=0x7f44e815e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:57.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.633+0000 7f44f0800700 1 -- 172.21.15.89:0/185882421 shutdown_connections 2024-04-02T09:12:57.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.633+0000 7f44f0800700 1 -- 172.21.15.89:0/185882421 wait complete. 2024-04-02T09:12:57.649 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2024-04-02T09:12:57.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:12:57.649 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-04-02T09:12:57.695 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T09:12:57.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.931+0000 7f7280e00700 1 Processor -- start 2024-04-02T09:12:57.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.931+0000 7f7280e00700 1 -- start start 2024-04-02T09:12:57.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.932+0000 7f7280e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f727c0cd5c0 0x7f727c15e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:57.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.932+0000 7f7280e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727c15eda0 con 0x7f727c06f310 2024-04-02T09:12:57.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.932+0000 7f7280e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727c099dd0 con 0x7f727c15cfe0 2024-04-02T09:12:57.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.932+0000 7f7280e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f727c09a0b0 con 0x7f727c0cd5c0 2024-04-02T09:12:57.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.932+0000 7f7279e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f727c0cd5c0 0x7f727c15e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:57.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.932+0000 7f7279400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f727c06f310 0x7f727c06f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40756/0 (socket says 172.21.15.89:40756) 2024-04-02T09:12:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.933+0000 7f7279400700 1 -- 172.21.15.89:0/1780622739 learned_addr learned my addr 172.21.15.89:0/1780622739 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.933+0000 7f7279e00700 1 -- 172.21.15.89:0/1780622739 >> v1:172.21.15.113:6789/0 conn(0x7f727c15cfe0 legacy=0x7f727c15d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.933+0000 7f7279e00700 1 -- 172.21.15.89:0/1780622739 >> v1:172.21.15.89:6789/0 conn(0x7f727c06f310 legacy=0x7f727c06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.933+0000 7f7279e00700 1 -- 172.21.15.89:0/1780622739 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727c09a390 con 0x7f727c0cd5c0 2024-04-02T09:12:57.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.933+0000 7f7279e00700 1 --2- 172.21.15.89:0/1780622739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f727c0cd5c0 0x7f727c15e860 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f726c009870 tx=0x7f726c009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e0bc092932b282 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.933+0000 7f7278a00700 1 -- 172.21.15.89:0/1780622739 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c011830 con 0x7f727c0cd5c0 2024-04-02T09:12:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.933+0000 7f7278a00700 1 -- 172.21.15.89:0/1780622739 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726c011990 con 0x7f727c0cd5c0 2024-04-02T09:12:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.933+0000 7f7278a00700 1 -- 172.21.15.89:0/1780622739 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c011d20 con 0x7f727c0cd5c0 2024-04-02T09:12:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.934+0000 7f7280e00700 1 -- 172.21.15.89:0/1780622739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f727c0cd5c0 msgr2=0x7f727c15e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.934+0000 7f7280e00700 1 --2- 172.21.15.89:0/1780622739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f727c0cd5c0 0x7f727c15e860 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f726c009870 tx=0x7f726c009b80 comp rx=0 tx=0).stop 2024-04-02T09:12:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.934+0000 7f7280e00700 1 -- 172.21.15.89:0/1780622739 shutdown_connections 2024-04-02T09:12:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.934+0000 7f7280e00700 1 --2- 172.21.15.89:0/1780622739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f727c0cd5c0 0x7f727c15e860 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.934+0000 7f7280e00700 1 -- 172.21.15.89:0/1780622739 >> 172.21.15.89:0/1780622739 conn(0x7f727c0d1370 msgr2=0x7f727c0d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:57.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.934+0000 7f7280e00700 1 -- 172.21.15.89:0/1780622739 shutdown_connections 2024-04-02T09:12:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.934+0000 7f7280e00700 1 -- 172.21.15.89:0/1780622739 wait complete. 2024-04-02T09:12:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.936+0000 7f7280e00700 1 Processor -- start 2024-04-02T09:12:57.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.936+0000 7f7280e00700 1 -- start start 2024-04-02T09:12:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.937+0000 7f7280e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f727c06f310 0x7f727c151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.937+0000 7f7280e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f727c0cd5c0 0x7f727c15a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.937+0000 7f7280e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f727c15cfe0 0x7f727c0b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.937+0000 7f7280e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727c151c50 con 0x7f727c15cfe0 2024-04-02T09:12:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.937+0000 7f7280e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727c0ae690 con 0x7f727c0cd5c0 2024-04-02T09:12:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.937+0000 7f7280e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f727c0ae970 con 0x7f727c06f310 2024-04-02T09:12:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.937+0000 7f727a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f727c15cfe0 0x7f727c0b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.937+0000 7f727a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f727c15cfe0 0x7f727c0b1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42886/0 (socket says 172.21.15.89:42886) 2024-04-02T09:12:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.937+0000 7f727a800700 1 -- 172.21.15.89:0/1785467402 learned_addr learned my addr 172.21.15.89:0/1785467402 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.937+0000 7f7279400700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f727c0cd5c0 0x7f727c15a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.937+0000 7f7279e00700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f727c06f310 0x7f727c151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.938+0000 7f727a800700 1 -- 172.21.15.89:0/1785467402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f727c06f310 msgr2=0x7f727c151710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:57.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.938+0000 7f727a800700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f727c06f310 0x7f727c151710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.938+0000 7f727a800700 1 -- 172.21.15.89:0/1785467402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f727c0cd5c0 msgr2=0x7f727c15a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.938+0000 7f727a800700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f727c0cd5c0 0x7f727c15a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.938+0000 7f727a800700 1 -- 172.21.15.89:0/1785467402 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726c0094a0 con 0x7f727c15cfe0 2024-04-02T09:12:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.938+0000 7f7279e00700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f727c06f310 0x7f727c151710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:12:57.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.938+0000 7f727a800700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f727c15cfe0 0x7f727c0b1e90 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f727400cb50 tx=0x7f727400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:57.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.938+0000 7f7272000700 1 -- 172.21.15.89:0/1785467402 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7274003c00 con 0x7f727c15cfe0 2024-04-02T09:12:57.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.938+0000 7f7272000700 1 -- 172.21.15.89:0/1785467402 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7274003d60 con 0x7f727c15cfe0 2024-04-02T09:12:57.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.938+0000 7f7272000700 1 -- 172.21.15.89:0/1785467402 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727400aac0 con 0x7f727c15cfe0 2024-04-02T09:12:57.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.938+0000 7f7280e00700 1 -- 172.21.15.89:0/1785467402 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f727c0aec50 con 0x7f727c15cfe0 2024-04-02T09:12:57.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.938+0000 7f7280e00700 1 -- 172.21.15.89:0/1785467402 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f727c0c6330 con 0x7f727c15cfe0 2024-04-02T09:12:57.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.939+0000 7f7280e00700 1 -- 172.21.15.89:0/1785467402 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f727c0c65a0 con 0x7f727c15cfe0 2024-04-02T09:12:57.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.940+0000 7f7272000700 1 -- 172.21.15.89:0/1785467402 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 152) v1 ==== 89415+0+0 (secure 0 0 0) 0x7f727400ac20 con 0x7f727c15cfe0 2024-04-02T09:12:57.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.941+0000 7f7272000700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f725c06afb0 0x7f725c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:57.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.941+0000 7f7272000700 1 -- 172.21.15.89:0/1785467402 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f7274089db0 con 0x7f727c15cfe0 2024-04-02T09:12:57.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.941+0000 7f7279e00700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f725c06afb0 0x7f725c06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:57.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.942+0000 7f7279e00700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f725c06afb0 0x7f725c06d470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f726c009790 tx=0x7f726c008030 comp rx=0 tx=0).ready entity=mgr.11466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:57.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:57.945+0000 7f7272000700 1 -- 172.21.15.89:0/1785467402 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f727c0c65a0 con 0x7f727c15cfe0 2024-04-02T09:12:58.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.205+0000 7f7280e00700 1 -- 172.21.15.89:0/1785467402 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f727c0c65a0 con 0x7f727c15cfe0 2024-04-02T09:12:58.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.232+0000 7f7272000700 1 -- 172.21.15.89:0/1785467402 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f7274010dc0 con 0x7f727c15cfe0 2024-04-02T09:12:58.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.232+0000 7f7272000700 1 -- 172.21.15.89:0/1785467402 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v153) v1 ==== 93+0+0 (secure 0 0 0) 0x7f727404ff90 con 0x7f727c15cfe0 2024-04-02T09:12:58.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.235+0000 7f7280e00700 1 -- 172.21.15.89:0/1785467402 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f725c06afb0 msgr2=0x7f725c06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:58.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.235+0000 7f7280e00700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f725c06afb0 0x7f725c06d470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f726c009790 tx=0x7f726c008030 comp rx=0 tx=0).stop 2024-04-02T09:12:58.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.235+0000 7f7280e00700 1 -- 172.21.15.89:0/1785467402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f727c15cfe0 msgr2=0x7f727c0b1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:58.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.235+0000 7f7280e00700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f727c15cfe0 0x7f727c0b1e90 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f727400cb50 tx=0x7f727400ce60 comp rx=0 tx=0).stop 2024-04-02T09:12:58.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.235+0000 7f7280e00700 1 -- 172.21.15.89:0/1785467402 shutdown_connections 2024-04-02T09:12:58.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.235+0000 7f7280e00700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f727c15cfe0 0x7f727c0b1e90 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:58.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.235+0000 7f7280e00700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f727c06f310 0x7f727c151710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:58.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.235+0000 7f7280e00700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f725c06afb0 0x7f725c06d470 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:58.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.235+0000 7f7280e00700 1 --2- 172.21.15.89:0/1785467402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f727c0cd5c0 0x7f727c15a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:58.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.235+0000 7f7280e00700 1 -- 172.21.15.89:0/1785467402 >> 172.21.15.89:0/1785467402 conn(0x7f727c0d1370 msgr2=0x7f727c0d1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:58.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.235+0000 7f7280e00700 1 -- 172.21.15.89:0/1785467402 shutdown_connections 2024-04-02T09:12:58.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.235+0000 7f7280e00700 1 -- 172.21.15.89:0/1785467402 wait complete. 2024-04-02T09:12:58.253 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T09:12:58.253 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:12:58.345 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:12:58.439 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T09:12:58.439 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:12:58.439 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:12:58.455 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:58.456+0000 7f590ab3a240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:12:58.467 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:58.469+0000 7fc13ea36240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:12:58.483 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:12:58.484 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:12:58.573 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:12:58.640 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:58.642+0000 7f590ab3a240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:12:58.662 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:58.664+0000 7fc13ea36240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:12:58.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.802+0000 7f0ff6600700 1 Processor -- start 2024-04-02T09:12:58.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.802+0000 7f0ff6600700 1 -- start start 2024-04-02T09:12:58.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.804+0000 7f0ff6600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ff015b5d0 0x7f0ff00ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:58.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.804+0000 7f0ff6600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ff00cb3f0 0x7f0ff00ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:58.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.804+0000 7f0ff6600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff015fd50 con 0x7f0ff00ae930 2024-04-02T09:12:58.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.804+0000 7f0ff6600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff015b9b0 con 0x7f0ff00cb3f0 2024-04-02T09:12:58.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.804+0000 7f0ff6600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff015f140 con 0x7f0ff015b5d0 2024-04-02T09:12:58.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.804+0000 7f0feea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ff00cb3f0 0x7f0ff00ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:58.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.804+0000 7f0fefe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0ff00ae930 0x7f0ff0140dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47642/0 (socket says 172.21.15.89:47642) 2024-04-02T09:12:58.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.804+0000 7f0fefe00700 1 -- 172.21.15.89:0/3049716083 learned_addr learned my addr 172.21.15.89:0/3049716083 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:58.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.804+0000 7f0fef400700 1 --2- 172.21.15.89:0/3049716083 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ff015b5d0 0x7f0ff00ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:58.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.804+0000 7f0feea00700 1 -- 172.21.15.89:0/3049716083 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ff015b5d0 msgr2=0x7f0ff00ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:58.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.804+0000 7f0feea00700 1 --2- 172.21.15.89:0/3049716083 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ff015b5d0 0x7f0ff00ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:58.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.804+0000 7f0feea00700 1 -- 172.21.15.89:0/3049716083 >> v1:172.21.15.89:6789/0 conn(0x7f0ff00ae930 legacy=0x7f0ff0140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:58.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.804+0000 7f0feea00700 1 -- 172.21.15.89:0/3049716083 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff015f420 con 0x7f0ff00cb3f0 2024-04-02T09:12:58.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.804+0000 7f0fee000700 1 -- 172.21.15.89:0/3049716083 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757080430 0 0) 0x7f0ff015fd50 con 0x7f0ff00ae930 2024-04-02T09:12:58.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.805+0000 7f0feea00700 1 --2- 172.21.15.89:0/3049716083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ff00cb3f0 0x7f0ff00ac0c0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009840 tx=0x7f0fdc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ccd9a01c93e4c3f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:58.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.805+0000 7f0fee000700 1 -- 172.21.15.89:0/3049716083 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc011830 con 0x7f0ff00cb3f0 2024-04-02T09:12:58.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.805+0000 7f0fee000700 1 -- 172.21.15.89:0/3049716083 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fdc011990 con 0x7f0ff00cb3f0 2024-04-02T09:12:58.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.805+0000 7f0fee000700 1 -- 172.21.15.89:0/3049716083 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc011d20 con 0x7f0ff00cb3f0 2024-04-02T09:12:58.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.806+0000 7f0ff6600700 1 -- 172.21.15.89:0/3049716083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ff00cb3f0 msgr2=0x7f0ff00ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:58.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.806+0000 7f0ff6600700 1 --2- 172.21.15.89:0/3049716083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ff00cb3f0 0x7f0ff00ac0c0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009840 tx=0x7f0fdc009b50 comp rx=0 tx=0).stop 2024-04-02T09:12:58.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.806+0000 7f0ff6600700 1 -- 172.21.15.89:0/3049716083 shutdown_connections 2024-04-02T09:12:58.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.806+0000 7f0ff6600700 1 --2- 172.21.15.89:0/3049716083 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ff015b5d0 0x7f0ff00ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:58.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.806+0000 7f0ff6600700 1 --2- 172.21.15.89:0/3049716083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ff00cb3f0 0x7f0ff00ac0c0 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:58.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.806+0000 7f0ff6600700 1 -- 172.21.15.89:0/3049716083 >> 172.21.15.89:0/3049716083 conn(0x7f0ff00cf190 msgr2=0x7f0ff00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:58.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.806+0000 7f0ff6600700 1 -- 172.21.15.89:0/3049716083 shutdown_connections 2024-04-02T09:12:58.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.806+0000 7f0ff6600700 1 -- 172.21.15.89:0/3049716083 wait complete. 2024-04-02T09:12:58.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.808+0000 7f0ff6600700 1 Processor -- start 2024-04-02T09:12:58.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.808+0000 7f0ff6600700 1 -- start start 2024-04-02T09:12:58.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0ff6600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ff00ae930 0x7f0ff0156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:58.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0ff6600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ff015b5d0 0x7f0ff015a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0ff6600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ff00b01c0 0x7f0ff015a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0ff6600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff0155240 con 0x7f0ff015b5d0 2024-04-02T09:12:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0ff6600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff0155da0 con 0x7f0ff00ae930 2024-04-02T09:12:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0ff6600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff015ad90 con 0x7f0ff00b01c0 2024-04-02T09:12:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0feea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ff015b5d0 0x7f0ff015a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0feea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ff015b5d0 0x7f0ff015a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52594/0 (socket says 172.21.15.89:52594) 2024-04-02T09:12:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0fef400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ff00ae930 0x7f0ff0156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0feea00700 1 -- 172.21.15.89:0/1022563882 learned_addr learned my addr 172.21.15.89:0/1022563882 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0fefe00700 1 --2- 172.21.15.89:0/1022563882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ff00b01c0 0x7f0ff015a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0feea00700 1 -- 172.21.15.89:0/1022563882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ff00b01c0 msgr2=0x7f0ff015a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0feea00700 1 --2- 172.21.15.89:0/1022563882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ff00b01c0 0x7f0ff015a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0feea00700 1 -- 172.21.15.89:0/1022563882 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ff00ae930 msgr2=0x7f0ff0156230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0feea00700 1 --2- 172.21.15.89:0/1022563882 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ff00ae930 0x7f0ff0156230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.809+0000 7f0feea00700 1 -- 172.21.15.89:0/1022563882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc0094a0 con 0x7f0ff015b5d0 2024-04-02T09:12:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.810+0000 7f0feea00700 1 --2- 172.21.15.89:0/1022563882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ff015b5d0 0x7f0ff015a310 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009fd0 tx=0x7f0fdc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.810+0000 7f0fe7400700 1 -- 172.21.15.89:0/1022563882 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc0036a0 con 0x7f0ff015b5d0 2024-04-02T09:12:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.810+0000 7f0fe7400700 1 -- 172.21.15.89:0/1022563882 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fdc01f070 con 0x7f0ff015b5d0 2024-04-02T09:12:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.810+0000 7f0fe7400700 1 -- 172.21.15.89:0/1022563882 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc003a20 con 0x7f0ff015b5d0 2024-04-02T09:12:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.810+0000 7f0ff6600700 1 -- 172.21.15.89:0/1022563882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff0156770 con 0x7f0ff015b5d0 2024-04-02T09:12:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.810+0000 7f0ff6600700 1 -- 172.21.15.89:0/1022563882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff0155750 con 0x7f0ff015b5d0 2024-04-02T09:12:58.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.811+0000 7f0ff6600700 1 -- 172.21.15.89:0/1022563882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff00cba10 con 0x7f0ff015b5d0 2024-04-02T09:12:58.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.813+0000 7f0fe7400700 1 -- 172.21.15.89:0/1022563882 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f0fdc003b80 con 0x7f0ff015b5d0 2024-04-02T09:12:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.814+0000 7f0fe7400700 1 --2- 172.21.15.89:0/1022563882 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0fd006af60 0x7f0fd006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.814+0000 7f0fe7400700 1 -- 172.21.15.89:0/1022563882 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f0fdc021070 con 0x7f0ff015b5d0 2024-04-02T09:12:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.814+0000 7f0fef400700 1 -- 172.21.15.89:0/1022563882 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0fd006af60 msgr2=0x7f0fd006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:58.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.814+0000 7f0fef400700 1 --2- 172.21.15.89:0/1022563882 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0fd006af60 0x7f0fd006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:58.823+0000 7f0fe7400700 1 -- 172.21.15.89:0/1022563882 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0fdc055f80 con 0x7f0ff015b5d0 2024-04-02T09:12:58.955 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:58.956+0000 7f590ab3a240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:12:58.997 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:58.999+0000 7fc13ea36240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:12:59.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.015+0000 7f0fef400700 1 -- 172.21.15.89:0/1022563882 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0fd006af60 msgr2=0x7f0fd006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:59.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.015+0000 7f0fef400700 1 --2- 172.21.15.89:0/1022563882 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0fd006af60 0x7f0fd006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:59.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.037+0000 7f0ff6600700 1 -- 172.21.15.89:0/1022563882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ff0155f10 con 0x7f0ff015b5d0 2024-04-02T09:12:59.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.040+0000 7f0fe7400700 1 -- 172.21.15.89:0/1022563882 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) v1 ==== 74+0+20955 (secure 0 0 0) 0x7f0fdc0595a0 con 0x7f0ff015b5d0 2024-04-02T09:12:59.038 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:12:59.039 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":228,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:12:53.003725+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:12:23.034869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-02T09:12:24.158560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:12:37.240568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:12:38.093999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/2216461018":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/3268555703":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/3481438563":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/783107677":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/2439246170":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/965177671":"2024-04-03T09:12:53.003657+0000","172.21.15.89:6833/108855":"2024-04-03T09:12:53.003657+0000","172.21.15.89:6832/108855":"2024-04-03T09:12:53.003657+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:12:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.042+0000 7f0ff6600700 1 -- 172.21.15.89:0/1022563882 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0fd006af60 msgr2=0x7f0fd006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.042+0000 7f0ff6600700 1 --2- 172.21.15.89:0/1022563882 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0fd006af60 0x7f0fd006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.042+0000 7f0ff6600700 1 -- 172.21.15.89:0/1022563882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ff015b5d0 msgr2=0x7f0ff015a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.042+0000 7f0ff6600700 1 --2- 172.21.15.89:0/1022563882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ff015b5d0 0x7f0ff015a310 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009fd0 tx=0x7f0fdc004750 comp rx=0 tx=0).stop 2024-04-02T09:12:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.042+0000 7f0ff6600700 1 -- 172.21.15.89:0/1022563882 shutdown_connections 2024-04-02T09:12:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.042+0000 7f0ff6600700 1 --2- 172.21.15.89:0/1022563882 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0fd006af60 0x7f0fd006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.042+0000 7f0ff6600700 1 --2- 172.21.15.89:0/1022563882 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ff00ae930 0x7f0ff0156230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.042+0000 7f0ff6600700 1 --2- 172.21.15.89:0/1022563882 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ff00b01c0 0x7f0ff015a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.042+0000 7f0ff6600700 1 --2- 172.21.15.89:0/1022563882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ff015b5d0 0x7f0ff015a310 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.042+0000 7f0ff6600700 1 -- 172.21.15.89:0/1022563882 >> 172.21.15.89:0/1022563882 conn(0x7f0ff00cf190 msgr2=0x7f0ff015d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.042+0000 7f0ff6600700 1 -- 172.21.15.89:0/1022563882 shutdown_connections 2024-04-02T09:12:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.042+0000 7f0ff6600700 1 -- 172.21.15.89:0/1022563882 wait complete. 2024-04-02T09:12:59.060 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T09:12:23.034869+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T09:12:24.158560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '223', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-02T09:12:37.240568+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-02T09:12:38.093999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:12:59.060 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:12:59.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.264+0000 7f3f64200700 1 Processor -- start 2024-04-02T09:12:59.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.265+0000 7f3f64200700 1 -- start start 2024-04-02T09:12:59.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.266+0000 7f3f64200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3f5c0cd5c0 0x7f3f5c15e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:59.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.266+0000 7f3f64200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5c06f310 0x7f3f5c06f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:59.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.266+0000 7f3f64200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5c15eda0 con 0x7f3f5c15cfe0 2024-04-02T09:12:59.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.266+0000 7f3f64200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5c06fce0 con 0x7f3f5c0cd5c0 2024-04-02T09:12:59.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.266+0000 7f3f64200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c099dd0 con 0x7f3f5c06f310 2024-04-02T09:12:59.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.267+0000 7f3f61800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3f5c0cd5c0 0x7f3f5c15e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:59.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.267+0000 7f3f61800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3f5c0cd5c0 0x7f3f5c15e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52596/0 (socket says 172.21.15.89:52596) 2024-04-02T09:12:59.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.267+0000 7f3f62200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3f5c15cfe0 0x7f3f5c15d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47654/0 (socket says 172.21.15.89:47654) 2024-04-02T09:12:59.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.267+0000 7f3f61800700 1 -- 172.21.15.89:0/2440493391 learned_addr learned my addr 172.21.15.89:0/2440493391 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:59.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.267+0000 7f3f60e00700 1 --2- 172.21.15.89:0/2440493391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5c06f310 0x7f3f5c06f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:59.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.267+0000 7f3f61800700 1 -- 172.21.15.89:0/2440493391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5c06f310 msgr2=0x7f3f5c06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:59.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.267+0000 7f3f61800700 1 --2- 172.21.15.89:0/2440493391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5c06f310 0x7f3f5c06f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.267+0000 7f3f61800700 1 -- 172.21.15.89:0/2440493391 >> v1:172.21.15.89:6789/0 conn(0x7f3f5c15cfe0 legacy=0x7f3f5c15d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:59.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.267+0000 7f3f61800700 1 -- 172.21.15.89:0/2440493391 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5c09a0b0 con 0x7f3f5c0cd5c0 2024-04-02T09:12:59.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.267+0000 7f3f61800700 1 --2- 172.21.15.89:0/2440493391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5c0cd5c0 0x7f3f5c15e860 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f3f500098f0 tx=0x7f3f50009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3457299bbd75bd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:59.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.268+0000 7f3f5be00700 1 -- 172.21.15.89:0/2440493391 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f50011830 con 0x7f3f5c0cd5c0 2024-04-02T09:12:59.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.268+0000 7f3f5be00700 1 -- 172.21.15.89:0/2440493391 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f50011990 con 0x7f3f5c0cd5c0 2024-04-02T09:12:59.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.268+0000 7f3f5be00700 1 -- 172.21.15.89:0/2440493391 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f50011d20 con 0x7f3f5c0cd5c0 2024-04-02T09:12:59.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.268+0000 7f3f64200700 1 -- 172.21.15.89:0/2440493391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5c0cd5c0 msgr2=0x7f3f5c15e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:59.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.268+0000 7f3f64200700 1 --2- 172.21.15.89:0/2440493391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5c0cd5c0 0x7f3f5c15e860 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f3f500098f0 tx=0x7f3f50009c00 comp rx=0 tx=0).stop 2024-04-02T09:12:59.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.268+0000 7f3f64200700 1 -- 172.21.15.89:0/2440493391 shutdown_connections 2024-04-02T09:12:59.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.268+0000 7f3f64200700 1 --2- 172.21.15.89:0/2440493391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5c0cd5c0 0x7f3f5c15e860 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.268+0000 7f3f64200700 1 --2- 172.21.15.89:0/2440493391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5c06f310 0x7f3f5c06f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.268+0000 7f3f64200700 1 -- 172.21.15.89:0/2440493391 >> 172.21.15.89:0/2440493391 conn(0x7f3f5c0d1370 msgr2=0x7f3f5c0d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:59.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.268+0000 7f3f64200700 1 -- 172.21.15.89:0/2440493391 shutdown_connections 2024-04-02T09:12:59.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.268+0000 7f3f64200700 1 -- 172.21.15.89:0/2440493391 wait complete. 2024-04-02T09:12:59.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.270+0000 7f3f64200700 1 Processor -- start 2024-04-02T09:12:59.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.270+0000 7f3f64200700 1 -- start start 2024-04-02T09:12:59.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.271+0000 7f3f64200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c06f310 0x7f3f5c160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:59.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.271+0000 7f3f64200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5c0cd5c0 0x7f3f5c15c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:59.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.271+0000 7f3f64200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5c15cfe0 0x7f3f5c15c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:59.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.271+0000 7f3f64200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c155180 con 0x7f3f5c15cfe0 2024-04-02T09:12:59.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.271+0000 7f3f64200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c0b5c20 con 0x7f3f5c06f310 2024-04-02T09:12:59.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.271+0000 7f3f64200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c160030 con 0x7f3f5c0cd5c0 2024-04-02T09:12:59.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.271+0000 7f3f62200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5c15cfe0 0x7f3f5c15c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.272+0000 7f3f62200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5c15cfe0 0x7f3f5c15c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52608/0 (socket says 172.21.15.89:52608) 2024-04-02T09:12:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.272+0000 7f3f62200700 1 -- 172.21.15.89:0/4196484427 learned_addr learned my addr 172.21.15.89:0/4196484427 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.272+0000 7f3f61800700 1 --2- 172.21.15.89:0/4196484427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c06f310 0x7f3f5c160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.272+0000 7f3f60e00700 1 --2- 172.21.15.89:0/4196484427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5c0cd5c0 0x7f3f5c15c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.272+0000 7f3f62200700 1 -- 172.21.15.89:0/4196484427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5c0cd5c0 msgr2=0x7f3f5c15c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.272+0000 7f3f62200700 1 --2- 172.21.15.89:0/4196484427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5c0cd5c0 0x7f3f5c15c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.272+0000 7f3f62200700 1 -- 172.21.15.89:0/4196484427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c06f310 msgr2=0x7f3f5c160440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.272+0000 7f3f62200700 1 --2- 172.21.15.89:0/4196484427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c06f310 0x7f3f5c160440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.272+0000 7f3f62200700 1 -- 172.21.15.89:0/4196484427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f50009550 con 0x7f3f5c15cfe0 2024-04-02T09:12:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.272+0000 7f3f62200700 1 --2- 172.21.15.89:0/4196484427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5c15cfe0 0x7f3f5c15c8d0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400cbc0 tx=0x7f3f5400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:59.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.272+0000 7f3f59600700 1 -- 172.21.15.89:0/4196484427 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5400a9e0 con 0x7f3f5c15cfe0 2024-04-02T09:12:59.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.272+0000 7f3f59600700 1 -- 172.21.15.89:0/4196484427 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5400ab40 con 0x7f3f5c15cfe0 2024-04-02T09:12:59.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.272+0000 7f3f59600700 1 -- 172.21.15.89:0/4196484427 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54007070 con 0x7f3f5c15cfe0 2024-04-02T09:12:59.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.272+0000 7f3f64200700 1 -- 172.21.15.89:0/4196484427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5c15f420 con 0x7f3f5c15cfe0 2024-04-02T09:12:59.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.273+0000 7f3f64200700 1 -- 172.21.15.89:0/4196484427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5c15f9e0 con 0x7f3f5c15cfe0 2024-04-02T09:12:59.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.274+0000 7f3f64200700 1 -- 172.21.15.89:0/4196484427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5c0b5d90 con 0x7f3f5c15cfe0 2024-04-02T09:12:59.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.275+0000 7f3f59600700 1 -- 172.21.15.89:0/4196484427 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f3f540071d0 con 0x7f3f5c15cfe0 2024-04-02T09:12:59.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.275+0000 7f3f59600700 1 --2- 172.21.15.89:0/4196484427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3f3c06af60 0x7f3f3c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:59.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.275+0000 7f3f59600700 1 -- 172.21.15.89:0/4196484427 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f3f540897e0 con 0x7f3f5c15cfe0 2024-04-02T09:12:59.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.275+0000 7f3f61800700 1 -- 172.21.15.89:0/4196484427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3f3c06af60 msgr2=0x7f3f3c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:59.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.275+0000 7f3f61800700 1 --2- 172.21.15.89:0/4196484427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3f3c06af60 0x7f3f3c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.280+0000 7f3f59600700 1 -- 172.21.15.89:0/4196484427 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3f540540a0 con 0x7f3f5c15cfe0 2024-04-02T09:12:59.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.476+0000 7f3f61800700 1 -- 172.21.15.89:0/4196484427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3f3c06af60 msgr2=0x7f3f3c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:59.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.476+0000 7f3f61800700 1 --2- 172.21.15.89:0/4196484427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3f3c06af60 0x7f3f3c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:59.483 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:59.485+0000 7f590ab3a240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:12:59.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.513+0000 7f3f64200700 1 -- 172.21.15.89:0/4196484427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3f5c067720 con 0x7f3f5c15cfe0 2024-04-02T09:12:59.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.514+0000 7f3f59600700 1 -- 172.21.15.89:0/4196484427 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v228) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3f540170a0 con 0x7f3f5c15cfe0 2024-04-02T09:12:59.512 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:12:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.516+0000 7f3f64200700 1 -- 172.21.15.89:0/4196484427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3f3c06af60 msgr2=0x7f3f3c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.516+0000 7f3f64200700 1 --2- 172.21.15.89:0/4196484427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3f3c06af60 0x7f3f3c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.516+0000 7f3f64200700 1 -- 172.21.15.89:0/4196484427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5c15cfe0 msgr2=0x7f3f5c15c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.516+0000 7f3f64200700 1 --2- 172.21.15.89:0/4196484427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5c15cfe0 0x7f3f5c15c8d0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400cbc0 tx=0x7f3f5400ced0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.516+0000 7f3f64200700 1 -- 172.21.15.89:0/4196484427 shutdown_connections 2024-04-02T09:12:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.516+0000 7f3f64200700 1 --2- 172.21.15.89:0/4196484427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3f3c06af60 0x7f3f3c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.516+0000 7f3f64200700 1 --2- 172.21.15.89:0/4196484427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f5c06f310 0x7f3f5c160440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.516+0000 7f3f64200700 1 --2- 172.21.15.89:0/4196484427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f5c15cfe0 0x7f3f5c15c8d0 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.516+0000 7f3f64200700 1 --2- 172.21.15.89:0/4196484427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3f5c0cd5c0 0x7f3f5c15c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.516+0000 7f3f64200700 1 -- 172.21.15.89:0/4196484427 >> 172.21.15.89:0/4196484427 conn(0x7f3f5c0d1370 msgr2=0x7f3f5c06b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:59.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.516+0000 7f3f64200700 1 -- 172.21.15.89:0/4196484427 shutdown_connections 2024-04-02T09:12:59.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.516+0000 7f3f64200700 1 -- 172.21.15.89:0/4196484427 wait complete. 2024-04-02T09:12:59.532 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T09:12:59.578 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:59.580+0000 7fc13ea36240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:12:59.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.733+0000 7fe98f600700 1 Processor -- start 2024-04-02T09:12:59.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.733+0000 7fe98f600700 1 -- start start 2024-04-02T09:12:59.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.735+0000 7fe98f600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe9880d16a0 0x7fe9880d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:59.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.735+0000 7fe98f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9880d2370 0x7fe9880ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:59.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.735+0000 7fe98f600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe988161420 con 0x7fe98815d9d0 2024-04-02T09:12:59.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.735+0000 7fe98f600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9880d2820 con 0x7fe9880d16a0 2024-04-02T09:12:59.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.735+0000 7fe98f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9880d4340 con 0x7fe9880d2370 2024-04-02T09:12:59.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.735+0000 7fe98cc00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe98815d9d0 0x7fe9880d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47656/0 (socket says 172.21.15.89:47656) 2024-04-02T09:12:59.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.735+0000 7fe98cc00700 1 -- 172.21.15.89:0/3778324178 learned_addr learned my addr 172.21.15.89:0/3778324178 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:12:59.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.735+0000 7fe98d600700 1 --2- 172.21.15.89:0/3778324178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9880d2370 0x7fe9880ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:59.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.735+0000 7fe987e00700 1 --2- 172.21.15.89:0/3778324178 >> v2:172.21.15.113:3300/0 conn(0x7fe9880d16a0 0x7fe9880d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:59.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.735+0000 7fe987400700 1 -- 172.21.15.89:0/3778324178 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882961316 0 0) 0x7fe988161420 con 0x7fe98815d9d0 2024-04-02T09:12:59.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.735+0000 7fe987400700 1 -- 172.21.15.89:0/3778324178 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9700035b0 con 0x7fe98815d9d0 2024-04-02T09:12:59.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.735+0000 7fe98d600700 1 -- 172.21.15.89:0/3778324178 >> v2:172.21.15.113:3300/0 conn(0x7fe9880d16a0 msgr2=0x7fe9880d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:12:59.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.735+0000 7fe98d600700 1 --2- 172.21.15.89:0/3778324178 >> v2:172.21.15.113:3300/0 conn(0x7fe9880d16a0 0x7fe9880d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.735+0000 7fe98d600700 1 -- 172.21.15.89:0/3778324178 >> v1:172.21.15.89:6789/0 conn(0x7fe98815d9d0 legacy=0x7fe9880d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:59.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.736+0000 7fe98d600700 1 -- 172.21.15.89:0/3778324178 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe988161700 con 0x7fe9880d2370 2024-04-02T09:12:59.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.736+0000 7fe98d600700 1 --2- 172.21.15.89:0/3778324178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9880d2370 0x7fe9880ce360 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe980009d90 tx=0x7fe98000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85098c26d77c8aa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:59.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.736+0000 7fe987400700 1 -- 172.21.15.89:0/3778324178 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98000e070 con 0x7fe9880d2370 2024-04-02T09:12:59.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.736+0000 7fe987400700 1 -- 172.21.15.89:0/3778324178 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98000bb20 con 0x7fe9880d2370 2024-04-02T09:12:59.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.736+0000 7fe987400700 1 -- 172.21.15.89:0/3778324178 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98000be90 con 0x7fe9880d2370 2024-04-02T09:12:59.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.737+0000 7fe98f600700 1 -- 172.21.15.89:0/3778324178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9880d2370 msgr2=0x7fe9880ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:59.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.737+0000 7fe98f600700 1 --2- 172.21.15.89:0/3778324178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9880d2370 0x7fe9880ce360 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe980009d90 tx=0x7fe98000b330 comp rx=0 tx=0).stop 2024-04-02T09:12:59.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.737+0000 7fe98f600700 1 -- 172.21.15.89:0/3778324178 shutdown_connections 2024-04-02T09:12:59.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.737+0000 7fe98f600700 1 --2- 172.21.15.89:0/3778324178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9880d2370 0x7fe9880ce360 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.737+0000 7fe98f600700 1 --2- 172.21.15.89:0/3778324178 >> v2:172.21.15.113:3300/0 conn(0x7fe9880d16a0 0x7fe9880d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.737+0000 7fe98f600700 1 -- 172.21.15.89:0/3778324178 >> 172.21.15.89:0/3778324178 conn(0x7fe98808aff0 msgr2=0x7fe9880d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:59.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.737+0000 7fe98f600700 1 -- 172.21.15.89:0/3778324178 shutdown_connections 2024-04-02T09:12:59.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.737+0000 7fe98f600700 1 -- 172.21.15.89:0/3778324178 wait complete. 2024-04-02T09:12:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.740+0000 7fe98f600700 1 Processor -- start 2024-04-02T09:12:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.740+0000 7fe98f600700 1 -- start start 2024-04-02T09:12:59.739 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:12:59.740+0000 7f590ab3a240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:12:59.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98f600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9880d16a0 0x7fe9881f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:59.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9880d2370 0x7fe9880cb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:59.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98f600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe98815d9d0 0x7fe9880cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:59.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98f600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9880cbf80 con 0x7fe9880d16a0 2024-04-02T09:12:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98f600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9881e8f10 con 0x7fe98815d9d0 2024-04-02T09:12:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9880b6150 con 0x7fe9880d2370 2024-04-02T09:12:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9880d16a0 0x7fe9881f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9880d16a0 0x7fe9881f2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52616/0 (socket says 172.21.15.89:52616) 2024-04-02T09:12:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98cc00700 1 -- 172.21.15.89:0/2860941303 learned_addr learned my addr 172.21.15.89:0/2860941303 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:12:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe987e00700 1 --2- 172.21.15.89:0/2860941303 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9880d2370 0x7fe9880cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98d600700 1 --2- 172.21.15.89:0/2860941303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe98815d9d0 0x7fe9880cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:12:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98cc00700 1 -- 172.21.15.89:0/2860941303 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9880d2370 msgr2=0x7fe9880cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98cc00700 1 --2- 172.21.15.89:0/2860941303 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9880d2370 0x7fe9880cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98cc00700 1 -- 172.21.15.89:0/2860941303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe98815d9d0 msgr2=0x7fe9880cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98cc00700 1 --2- 172.21.15.89:0/2860941303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe98815d9d0 0x7fe9880cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.742+0000 7fe98cc00700 1 -- 172.21.15.89:0/2860941303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe980009a70 con 0x7fe9880d16a0 2024-04-02T09:12:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.743+0000 7fe98cc00700 1 --2- 172.21.15.89:0/2860941303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9880d16a0 0x7fe9881f2dc0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00c420 tx=0x7fe97c00c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:12:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.743+0000 7fe984c00700 1 -- 172.21.15.89:0/2860941303 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c005d70 con 0x7fe9880d16a0 2024-04-02T09:12:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.743+0000 7fe984c00700 1 -- 172.21.15.89:0/2860941303 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97c00ce30 con 0x7fe9880d16a0 2024-04-02T09:12:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.743+0000 7fe984c00700 1 -- 172.21.15.89:0/2860941303 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c0072d0 con 0x7fe9880d16a0 2024-04-02T09:12:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.743+0000 7fe98f600700 1 -- 172.21.15.89:0/2860941303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9880b6430 con 0x7fe9880d16a0 2024-04-02T09:12:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.743+0000 7fe98f600700 1 -- 172.21.15.89:0/2860941303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9880b69f0 con 0x7fe9880d16a0 2024-04-02T09:12:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.745+0000 7fe98f600700 1 -- 172.21.15.89:0/2860941303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9880d1ca0 con 0x7fe9880d16a0 2024-04-02T09:12:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.745+0000 7fe984c00700 1 -- 172.21.15.89:0/2860941303 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fe97c007430 con 0x7fe9880d16a0 2024-04-02T09:12:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.746+0000 7fe984c00700 1 --2- 172.21.15.89:0/2860941303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96806af60 0x7fe96806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:12:59.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.746+0000 7fe984c00700 1 -- 172.21.15.89:0/2860941303 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7fe97c088f70 con 0x7fe9880d16a0 2024-04-02T09:12:59.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.746+0000 7fe987e00700 1 -- 172.21.15.89:0/2860941303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96806af60 msgr2=0x7fe96806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:59.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.746+0000 7fe987e00700 1 --2- 172.21.15.89:0/2860941303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96806af60 0x7fe96806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:12:59.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.751+0000 7fe984c00700 1 -- 172.21.15.89:0/2860941303 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe97c0538c0 con 0x7fe9880d16a0 2024-04-02T09:12:59.850 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:12:59.852+0000 7fc13ea36240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:12:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.947+0000 7fe987e00700 1 -- 172.21.15.89:0/2860941303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96806af60 msgr2=0x7fe96806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:12:59.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.947+0000 7fe987e00700 1 --2- 172.21.15.89:0/2860941303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96806af60 0x7fe96806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:12:59.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.959+0000 7fe98f600700 1 -- 172.21.15.89:0/2860941303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe988067720 con 0x7fe9880d16a0 2024-04-02T09:12:59.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.960+0000 7fe984c00700 1 -- 172.21.15.89:0/2860941303 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v228) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe97c0170a0 con 0x7fe9880d16a0 2024-04-02T09:12:59.958 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:12:59.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.962+0000 7fe98f600700 1 -- 172.21.15.89:0/2860941303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96806af60 msgr2=0x7fe96806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:12:59.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.962+0000 7fe98f600700 1 --2- 172.21.15.89:0/2860941303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96806af60 0x7fe96806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.962+0000 7fe98f600700 1 -- 172.21.15.89:0/2860941303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9880d16a0 msgr2=0x7fe9881f2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:12:59.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.962+0000 7fe98f600700 1 --2- 172.21.15.89:0/2860941303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9880d16a0 0x7fe9881f2dc0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00c420 tx=0x7fe97c00c730 comp rx=0 tx=0).stop 2024-04-02T09:12:59.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.962+0000 7fe98f600700 1 -- 172.21.15.89:0/2860941303 shutdown_connections 2024-04-02T09:12:59.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.962+0000 7fe98f600700 1 --2- 172.21.15.89:0/2860941303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96806af60 0x7fe96806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.962+0000 7fe98f600700 1 --2- 172.21.15.89:0/2860941303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe98815d9d0 0x7fe9880cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.962+0000 7fe98f600700 1 --2- 172.21.15.89:0/2860941303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9880d16a0 0x7fe9881f2dc0 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.962+0000 7fe98f600700 1 --2- 172.21.15.89:0/2860941303 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9880d2370 0x7fe9880cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:12:59.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.962+0000 7fe98f600700 1 -- 172.21.15.89:0/2860941303 >> 172.21.15.89:0/2860941303 conn(0x7fe98808aff0 msgr2=0x7fe98806d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:12:59.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.962+0000 7fe98f600700 1 -- 172.21.15.89:0/2860941303 shutdown_connections 2024-04-02T09:12:59.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:12:59.962+0000 7fe98f600700 1 -- 172.21.15.89:0/2860941303 wait complete. 2024-04-02T09:12:59.979 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T09:13:00.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.180+0000 7fe98be00700 1 Processor -- start 2024-04-02T09:13:00.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.180+0000 7fe98be00700 1 -- start start 2024-04-02T09:13:00.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.181+0000 7fe98be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9840cbb30 0x7fe9840cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:00.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.181+0000 7fe98be00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe9840cd4a0 0x7fe98406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:00.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.181+0000 7fe98be00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9840cc450 con 0x7fe9840cd020 2024-04-02T09:13:00.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.182+0000 7fe98be00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9840cdae0 con 0x7fe9840cd4a0 2024-04-02T09:13:00.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.182+0000 7fe98be00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9840cddc0 con 0x7fe9840cbb30 2024-04-02T09:13:00.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.182+0000 7fe989400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9840cbb30 0x7fe9840cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:00.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.182+0000 7fe989e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe9840cd4a0 0x7fe98406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:00.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.182+0000 7fe988a00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fe9840cd020 0x7fe9840b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:40054/0 (socket says 172.21.15.89:40054) 2024-04-02T09:13:00.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.182+0000 7fe988a00700 1 -- 172.21.15.89:0/116817361 learned_addr learned my addr 172.21.15.89:0/116817361 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:00.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.182+0000 7fe989400700 1 -- 172.21.15.89:0/116817361 >> v2:172.21.15.113:3300/0 conn(0x7fe9840cd4a0 msgr2=0x7fe98406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:00.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.182+0000 7fe989400700 1 --2- 172.21.15.89:0/116817361 >> v2:172.21.15.113:3300/0 conn(0x7fe9840cd4a0 0x7fe98406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.182+0000 7fe989400700 1 -- 172.21.15.89:0/116817361 >> v1:172.21.15.113:6789/0 conn(0x7fe9840cd020 legacy=0x7fe9840b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:00.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.182+0000 7fe989400700 1 -- 172.21.15.89:0/116817361 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9840ce0a0 con 0x7fe9840cbb30 2024-04-02T09:13:00.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.182+0000 7fe983e00700 1 -- 172.21.15.89:0/116817361 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837682641 0 0) 0x7fe9840cc450 con 0x7fe9840cd020 2024-04-02T09:13:00.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.183+0000 7fe989400700 1 --2- 172.21.15.89:0/116817361 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9840cbb30 0x7fe9840cbf10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe978009840 tx=0x7fe978009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=871ff9a86c01ba5b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:00.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.183+0000 7fe983e00700 1 -- 172.21.15.89:0/116817361 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe978011830 con 0x7fe9840cbb30 2024-04-02T09:13:00.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.183+0000 7fe983e00700 1 -- 172.21.15.89:0/116817361 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe978011990 con 0x7fe9840cbb30 2024-04-02T09:13:00.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.183+0000 7fe983e00700 1 -- 172.21.15.89:0/116817361 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe978011d20 con 0x7fe9840cbb30 2024-04-02T09:13:00.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.184+0000 7fe98be00700 1 -- 172.21.15.89:0/116817361 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9840cbb30 msgr2=0x7fe9840cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:00.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.184+0000 7fe98be00700 1 --2- 172.21.15.89:0/116817361 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9840cbb30 0x7fe9840cbf10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe978009840 tx=0x7fe978009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:00.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.184+0000 7fe98be00700 1 -- 172.21.15.89:0/116817361 shutdown_connections 2024-04-02T09:13:00.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.184+0000 7fe98be00700 1 --2- 172.21.15.89:0/116817361 >> v2:172.21.15.113:3300/0 conn(0x7fe9840cd4a0 0x7fe98406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.184+0000 7fe98be00700 1 --2- 172.21.15.89:0/116817361 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9840cbb30 0x7fe9840cbf10 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.184+0000 7fe98be00700 1 -- 172.21.15.89:0/116817361 >> 172.21.15.89:0/116817361 conn(0x7fe9840b0d70 msgr2=0x7fe9840b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:00.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.184+0000 7fe98be00700 1 -- 172.21.15.89:0/116817361 shutdown_connections 2024-04-02T09:13:00.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.184+0000 7fe98be00700 1 -- 172.21.15.89:0/116817361 wait complete. 2024-04-02T09:13:00.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.185+0000 7fe98be00700 1 Processor -- start 2024-04-02T09:13:00.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.186+0000 7fe98be00700 1 -- start start 2024-04-02T09:13:00.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.186+0000 7fe98be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9840cbb30 0x7fe984149d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:00.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.186+0000 7fe98be00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9840cd020 0x7fe9841495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:00.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.187+0000 7fe98be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9840cd4a0 0x7fe98409ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:00.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.187+0000 7fe98be00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe984149b30 con 0x7fe9840cd4a0 2024-04-02T09:13:00.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.187+0000 7fe98be00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9840991d0 con 0x7fe9840cd020 2024-04-02T09:13:00.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.187+0000 7fe98be00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9840994b0 con 0x7fe9840cbb30 2024-04-02T09:13:00.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.187+0000 7fe989e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9840cd4a0 0x7fe98409ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:00.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.187+0000 7fe988a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9840cd020 0x7fe9841495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.187+0000 7fe989400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9840cbb30 0x7fe984149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.187+0000 7fe989e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9840cd4a0 0x7fe98409ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52628/0 (socket says 172.21.15.89:52628) 2024-04-02T09:13:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.187+0000 7fe989e00700 1 -- 172.21.15.89:0/2496979273 learned_addr learned my addr 172.21.15.89:0/2496979273 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.187+0000 7fe989e00700 1 -- 172.21.15.89:0/2496979273 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9840cbb30 msgr2=0x7fe984149d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.187+0000 7fe989e00700 1 --2- 172.21.15.89:0/2496979273 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9840cbb30 0x7fe984149d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.187+0000 7fe989e00700 1 -- 172.21.15.89:0/2496979273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9840cd020 msgr2=0x7fe9841495f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.187+0000 7fe989e00700 1 --2- 172.21.15.89:0/2496979273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9840cd020 0x7fe9841495f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.187+0000 7fe989e00700 1 -- 172.21.15.89:0/2496979273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9780094a0 con 0x7fe9840cd4a0 2024-04-02T09:13:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.188+0000 7fe989e00700 1 --2- 172.21.15.89:0/2496979273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9840cd4a0 0x7fe98409ca80 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00efc0 tx=0x7fe97c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.188+0000 7fe981600700 1 -- 172.21.15.89:0/2496979273 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c009ea0 con 0x7fe9840cd4a0 2024-04-02T09:13:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.188+0000 7fe981600700 1 -- 172.21.15.89:0/2496979273 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97c00f040 con 0x7fe9840cd4a0 2024-04-02T09:13:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.188+0000 7fe98be00700 1 -- 172.21.15.89:0/2496979273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe984099790 con 0x7fe9840cd4a0 2024-04-02T09:13:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.188+0000 7fe981600700 1 -- 172.21.15.89:0/2496979273 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c00ced0 con 0x7fe9840cd4a0 2024-04-02T09:13:00.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.188+0000 7fe98be00700 1 -- 172.21.15.89:0/2496979273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe984099d50 con 0x7fe9840cd4a0 2024-04-02T09:13:00.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.189+0000 7fe98be00700 1 -- 172.21.15.89:0/2496979273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe984067720 con 0x7fe9840cd4a0 2024-04-02T09:13:00.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.190+0000 7fe981600700 1 -- 172.21.15.89:0/2496979273 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fe97c014480 con 0x7fe9840cd4a0 2024-04-02T09:13:00.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.191+0000 7fe981600700 1 --2- 172.21.15.89:0/2496979273 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96406af60 0x7fe96406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:00.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.191+0000 7fe989400700 1 -- 172.21.15.89:0/2496979273 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96406af60 msgr2=0x7fe96406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:00.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.191+0000 7fe981600700 1 -- 172.21.15.89:0/2496979273 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7fe97c08aaa0 con 0x7fe9840cd4a0 2024-04-02T09:13:00.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.191+0000 7fe989400700 1 --2- 172.21.15.89:0/2496979273 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96406af60 0x7fe96406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:13:00.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.195+0000 7fe981600700 1 -- 172.21.15.89:0/2496979273 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe97c055160 con 0x7fe9840cd4a0 2024-04-02T09:13:00.199 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:00.201+0000 7f590ab3a240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:13:00.337 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:00.338+0000 7fc13ea36240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:13:00.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.392+0000 7fe989400700 1 -- 172.21.15.89:0/2496979273 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96406af60 msgr2=0x7fe96406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:00.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.392+0000 7fe989400700 1 --2- 172.21.15.89:0/2496979273 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96406af60 0x7fe96406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:13:00.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.411+0000 7fe98be00700 1 -- 172.21.15.89:0/2496979273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe984158c50 con 0x7fe9840cd4a0 2024-04-02T09:13:00.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.411+0000 7fe981600700 1 -- 172.21.15.89:0/2496979273 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v228) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe97c058780 con 0x7fe9840cd4a0 2024-04-02T09:13:00.409 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:13:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.414+0000 7fe98be00700 1 -- 172.21.15.89:0/2496979273 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96406af60 msgr2=0x7fe96406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:13:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.414+0000 7fe98be00700 1 --2- 172.21.15.89:0/2496979273 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96406af60 0x7fe96406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.414+0000 7fe98be00700 1 -- 172.21.15.89:0/2496979273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9840cd4a0 msgr2=0x7fe98409ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:00.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.414+0000 7fe98be00700 1 --2- 172.21.15.89:0/2496979273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9840cd4a0 0x7fe98409ca80 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00efc0 tx=0x7fe97c00c3a0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.414+0000 7fe98be00700 1 -- 172.21.15.89:0/2496979273 shutdown_connections 2024-04-02T09:13:00.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.414+0000 7fe98be00700 1 --2- 172.21.15.89:0/2496979273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9840cd4a0 0x7fe98409ca80 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.414+0000 7fe98be00700 1 --2- 172.21.15.89:0/2496979273 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe9840cbb30 0x7fe984149d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.414+0000 7fe98be00700 1 --2- 172.21.15.89:0/2496979273 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe96406af60 0x7fe96406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.414+0000 7fe98be00700 1 --2- 172.21.15.89:0/2496979273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9840cd020 0x7fe9841495f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.414+0000 7fe98be00700 1 -- 172.21.15.89:0/2496979273 >> 172.21.15.89:0/2496979273 conn(0x7fe9840b0d70 msgr2=0x7fe98415da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:00.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.414+0000 7fe98be00700 1 -- 172.21.15.89:0/2496979273 shutdown_connections 2024-04-02T09:13:00.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.414+0000 7fe98be00700 1 -- 172.21.15.89:0/2496979273 wait complete. 2024-04-02T09:13:00.430 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-02T09:13:00.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.640+0000 7efec1000700 1 Processor -- start 2024-04-02T09:13:00.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.640+0000 7efec1000700 1 -- start start 2024-04-02T09:13:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.641+0000 7efec1000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efebc099a60 0x7efebc0d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.641+0000 7efec1000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efebc09a750 0x7efebc06c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.641+0000 7efec1000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efebc0d0970 con 0x7efebc15d990 2024-04-02T09:13:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.641+0000 7efec1000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efebc06c020 con 0x7efebc09a750 2024-04-02T09:13:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.641+0000 7efec1000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efebc0c81a0 con 0x7efebc099a60 2024-04-02T09:13:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.642+0000 7efeb9400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efebc15d990 0x7efebc099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47662/0 (socket says 172.21.15.89:47662) 2024-04-02T09:13:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.642+0000 7efeb9400700 1 -- 172.21.15.89:0/414291506 learned_addr learned my addr 172.21.15.89:0/414291506 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.642+0000 7efeba800700 1 --2- 172.21.15.89:0/414291506 >> v2:172.21.15.89:3300/0 conn(0x7efebc09a750 0x7efebc06c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.642+0000 7efeb9e00700 1 --2- 172.21.15.89:0/414291506 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efebc099a60 0x7efebc0d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:00.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.642+0000 7efeb8a00700 1 -- 172.21.15.89:0/414291506 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540129902 0 0) 0x7efebc0d0970 con 0x7efebc15d990 2024-04-02T09:13:00.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.642+0000 7efeba800700 1 -- 172.21.15.89:0/414291506 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efebc099a60 msgr2=0x7efebc0d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:00.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.642+0000 7efeba800700 1 --2- 172.21.15.89:0/414291506 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efebc099a60 0x7efebc0d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.642+0000 7efeba800700 1 -- 172.21.15.89:0/414291506 >> v1:172.21.15.89:6789/0 conn(0x7efebc15d990 legacy=0x7efebc099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:00.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.642+0000 7efeba800700 1 -- 172.21.15.89:0/414291506 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efebc0d0690 con 0x7efebc09a750 2024-04-02T09:13:00.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.642+0000 7efeba800700 1 --2- 172.21.15.89:0/414291506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efebc09a750 0x7efebc06c4b0 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7efeb4009d90 tx=0x7efeb400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0670c54991c863e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:00.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.643+0000 7efeb8a00700 1 -- 172.21.15.89:0/414291506 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb400e070 con 0x7efebc09a750 2024-04-02T09:13:00.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.643+0000 7efeb8a00700 1 -- 172.21.15.89:0/414291506 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb400bb20 con 0x7efebc09a750 2024-04-02T09:13:00.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.643+0000 7efeb8a00700 1 -- 172.21.15.89:0/414291506 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb4012430 con 0x7efebc09a750 2024-04-02T09:13:00.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.643+0000 7efec1000700 1 -- 172.21.15.89:0/414291506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efebc09a750 msgr2=0x7efebc06c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:00.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.643+0000 7efec1000700 1 --2- 172.21.15.89:0/414291506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efebc09a750 0x7efebc06c4b0 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7efeb4009d90 tx=0x7efeb400b330 comp rx=0 tx=0).stop 2024-04-02T09:13:00.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.643+0000 7efec1000700 1 -- 172.21.15.89:0/414291506 shutdown_connections 2024-04-02T09:13:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.643+0000 7efec1000700 1 --2- 172.21.15.89:0/414291506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efebc09a750 0x7efebc06c4b0 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.643+0000 7efec1000700 1 --2- 172.21.15.89:0/414291506 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efebc099a60 0x7efebc0d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.643+0000 7efec1000700 1 -- 172.21.15.89:0/414291506 >> 172.21.15.89:0/414291506 conn(0x7efebc0b2e20 msgr2=0x7efebc0b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.643+0000 7efec1000700 1 -- 172.21.15.89:0/414291506 shutdown_connections 2024-04-02T09:13:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.643+0000 7efec1000700 1 -- 172.21.15.89:0/414291506 wait complete. 2024-04-02T09:13:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.646+0000 7efec1000700 1 Processor -- start 2024-04-02T09:13:00.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.646+0000 7efec1000700 1 -- start start 2024-04-02T09:13:00.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.647+0000 7efec1000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efebc099a60 0x7efebc14f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:00.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.647+0000 7efec1000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efebc09a750 0x7efebc0c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:00.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.647+0000 7efec1000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efebc15d990 0x7efebc0cf510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:00.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.647+0000 7efec1000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc14fd80 con 0x7efebc099a60 2024-04-02T09:13:00.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.647+0000 7efec1000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc0c5770 con 0x7efebc15d990 2024-04-02T09:13:00.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.647+0000 7efec1000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efebc0c5a50 con 0x7efebc09a750 2024-04-02T09:13:00.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.647+0000 7efeb9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efebc099a60 0x7efebc14f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:00.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.647+0000 7efeb9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efebc099a60 0x7efebc14f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52648/0 (socket says 172.21.15.89:52648) 2024-04-02T09:13:00.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.647+0000 7efeb9e00700 1 -- 172.21.15.89:0/4162561424 learned_addr learned my addr 172.21.15.89:0/4162561424 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:00.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.647+0000 7efeb9400700 1 --2- 172.21.15.89:0/4162561424 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efebc09a750 0x7efebc0c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:00.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.647+0000 7efeba800700 1 --2- 172.21.15.89:0/4162561424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efebc15d990 0x7efebc0cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:00.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.648+0000 7efeb9e00700 1 -- 172.21.15.89:0/4162561424 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efebc09a750 msgr2=0x7efebc0c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:00.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.648+0000 7efeb9e00700 1 --2- 172.21.15.89:0/4162561424 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efebc09a750 0x7efebc0c4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.648+0000 7efeb9e00700 1 -- 172.21.15.89:0/4162561424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efebc15d990 msgr2=0x7efebc0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.648+0000 7efeb9e00700 1 --2- 172.21.15.89:0/4162561424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efebc15d990 0x7efebc0cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.648+0000 7efeb9e00700 1 -- 172.21.15.89:0/4162561424 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb4009a70 con 0x7efebc099a60 2024-04-02T09:13:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.648+0000 7efeb9e00700 1 --2- 172.21.15.89:0/4162561424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efebc099a60 0x7efebc14f840 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7efeac00e870 tx=0x7efeac00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.648+0000 7efeb2000700 1 -- 172.21.15.89:0/4162561424 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac009590 con 0x7efebc099a60 2024-04-02T09:13:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.648+0000 7efeb2000700 1 -- 172.21.15.89:0/4162561424 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeac013680 con 0x7efebc099a60 2024-04-02T09:13:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.648+0000 7efec1000700 1 -- 172.21.15.89:0/4162561424 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efebc0c5d30 con 0x7efebc099a60 2024-04-02T09:13:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.648+0000 7efeb2000700 1 -- 172.21.15.89:0/4162561424 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac0139f0 con 0x7efebc099a60 2024-04-02T09:13:00.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.648+0000 7efec1000700 1 -- 172.21.15.89:0/4162561424 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efebc0c62f0 con 0x7efebc099a60 2024-04-02T09:13:00.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.650+0000 7efec1000700 1 -- 172.21.15.89:0/4162561424 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efebc067720 con 0x7efebc099a60 2024-04-02T09:13:00.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.650+0000 7efeb2000700 1 -- 172.21.15.89:0/4162561424 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7efeac013b50 con 0x7efebc099a60 2024-04-02T09:13:00.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.652+0000 7efeb2000700 1 --2- 172.21.15.89:0/4162561424 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efe9c06af60 0x7efe9c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:00.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.652+0000 7efeb2000700 1 -- 172.21.15.89:0/4162561424 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7efeac089c50 con 0x7efebc099a60 2024-04-02T09:13:00.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.652+0000 7efeb9400700 1 -- 172.21.15.89:0/4162561424 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efe9c06af60 msgr2=0x7efe9c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:00.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.652+0000 7efeb9400700 1 --2- 172.21.15.89:0/4162561424 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efe9c06af60 0x7efe9c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:13:00.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.656+0000 7efeb2000700 1 -- 172.21.15.89:0/4162561424 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efeac054460 con 0x7efebc099a60 2024-04-02T09:13:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.853+0000 7efeb9400700 1 -- 172.21.15.89:0/4162561424 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efe9c06af60 msgr2=0x7efe9c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.853+0000 7efeb9400700 1 --2- 172.21.15.89:0/4162561424 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efe9c06af60 0x7efe9c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:13:00.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.864+0000 7efec1000700 1 -- 172.21.15.89:0/4162561424 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efebc0d0b90 con 0x7efebc099a60 2024-04-02T09:13:00.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.865+0000 7efeb2000700 1 -- 172.21.15.89:0/4162561424 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v228) v1 ==== 107+0+11 (secure 0 0 0) 0x7efeac057a80 con 0x7efebc099a60 2024-04-02T09:13:00.863 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:13:00.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.867+0000 7efec1000700 1 -- 172.21.15.89:0/4162561424 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efe9c06af60 msgr2=0x7efe9c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:13:00.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.867+0000 7efec1000700 1 --2- 172.21.15.89:0/4162561424 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efe9c06af60 0x7efe9c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.867+0000 7efec1000700 1 -- 172.21.15.89:0/4162561424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efebc099a60 msgr2=0x7efebc14f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:00.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.867+0000 7efec1000700 1 --2- 172.21.15.89:0/4162561424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efebc099a60 0x7efebc14f840 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7efeac00e870 tx=0x7efeac00eb80 comp rx=0 tx=0).stop 2024-04-02T09:13:00.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.867+0000 7efec1000700 1 -- 172.21.15.89:0/4162561424 shutdown_connections 2024-04-02T09:13:00.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.867+0000 7efec1000700 1 --2- 172.21.15.89:0/4162561424 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efe9c06af60 0x7efe9c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.867+0000 7efec1000700 1 --2- 172.21.15.89:0/4162561424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efebc15d990 0x7efebc0cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.867+0000 7efec1000700 1 --2- 172.21.15.89:0/4162561424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efebc099a60 0x7efebc14f840 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.867+0000 7efec1000700 1 --2- 172.21.15.89:0/4162561424 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efebc09a750 0x7efebc0c4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:00.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.867+0000 7efec1000700 1 -- 172.21.15.89:0/4162561424 >> 172.21.15.89:0/4162561424 conn(0x7efebc0b2e20 msgr2=0x7efebc06f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:00.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.867+0000 7efec1000700 1 -- 172.21.15.89:0/4162561424 shutdown_connections 2024-04-02T09:13:00.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:00.867+0000 7efec1000700 1 -- 172.21.15.89:0/4162561424 wait complete. 2024-04-02T09:13:00.886 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-02T09:13:00.913 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:00.914+0000 7f590ab3a240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:13:01.005 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:01.006+0000 7f590ab3a240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:13:01.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.090+0000 7fd6e9000700 1 Processor -- start 2024-04-02T09:13:01.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.090+0000 7fd6e9000700 1 -- start start 2024-04-02T09:13:01.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.091+0000 7fd6e9000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd6e40cbb30 0x7fd6e40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:01.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.091+0000 7fd6e9000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6e40cd020 0x7fd6e40cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:01.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.091+0000 7fd6e9000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e40cc450 con 0x7fd6e40b2070 2024-04-02T09:13:01.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.091+0000 7fd6e9000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e40cdae0 con 0x7fd6e40cd020 2024-04-02T09:13:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.091+0000 7fd6e9000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e40cddc0 con 0x7fd6e40cbb30 2024-04-02T09:13:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.091+0000 7fd6e1400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6e40cd020 0x7fd6e40cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.091+0000 7fd6e1400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6e40cd020 0x7fd6e40cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52652/0 (socket says 172.21.15.89:52652) 2024-04-02T09:13:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.091+0000 7fd6e1400700 1 -- 172.21.15.89:0/3062212015 learned_addr learned my addr 172.21.15.89:0/3062212015 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.091+0000 7fd6e1e00700 1 --2- 172.21.15.89:0/3062212015 >> v2:172.21.15.113:3300/0 conn(0x7fd6e40cbb30 0x7fd6e40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.092+0000 7fd6e1400700 1 -- 172.21.15.89:0/3062212015 >> v2:172.21.15.113:3300/0 conn(0x7fd6e40cbb30 msgr2=0x7fd6e40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.092+0000 7fd6e1400700 1 --2- 172.21.15.89:0/3062212015 >> v2:172.21.15.113:3300/0 conn(0x7fd6e40cbb30 0x7fd6e40cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.092+0000 7fd6e1400700 1 -- 172.21.15.89:0/3062212015 >> v1:172.21.15.113:6789/0 conn(0x7fd6e40b2070 legacy=0x7fd6e406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.092+0000 7fd6e1400700 1 -- 172.21.15.89:0/3062212015 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e40ce0a0 con 0x7fd6e40cd020 2024-04-02T09:13:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.092+0000 7fd6e1400700 1 --2- 172.21.15.89:0/3062212015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e40cd020 0x7fd6e40cd480 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7fd6e406fd00 tx=0x7fd6d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7342dbc787cc5ca4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.092+0000 7fd6e0a00700 1 -- 172.21.15.89:0/3062212015 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0011830 con 0x7fd6e40cd020 2024-04-02T09:13:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.092+0000 7fd6e0a00700 1 -- 172.21.15.89:0/3062212015 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d0011990 con 0x7fd6e40cd020 2024-04-02T09:13:01.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.092+0000 7fd6e0a00700 1 -- 172.21.15.89:0/3062212015 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0011d20 con 0x7fd6e40cd020 2024-04-02T09:13:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.093+0000 7fd6e9000700 1 -- 172.21.15.89:0/3062212015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e40cd020 msgr2=0x7fd6e40cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.093+0000 7fd6e9000700 1 --2- 172.21.15.89:0/3062212015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e40cd020 0x7fd6e40cd480 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7fd6e406fd00 tx=0x7fd6d0009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.093+0000 7fd6e9000700 1 -- 172.21.15.89:0/3062212015 shutdown_connections 2024-04-02T09:13:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.093+0000 7fd6e9000700 1 --2- 172.21.15.89:0/3062212015 >> v2:172.21.15.113:3300/0 conn(0x7fd6e40cbb30 0x7fd6e40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.093+0000 7fd6e9000700 1 --2- 172.21.15.89:0/3062212015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e40cd020 0x7fd6e40cd480 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.093+0000 7fd6e9000700 1 -- 172.21.15.89:0/3062212015 >> 172.21.15.89:0/3062212015 conn(0x7fd6e40b0d70 msgr2=0x7fd6e40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.093+0000 7fd6e9000700 1 -- 172.21.15.89:0/3062212015 shutdown_connections 2024-04-02T09:13:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.093+0000 7fd6e9000700 1 -- 172.21.15.89:0/3062212015 wait complete. 2024-04-02T09:13:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.095+0000 7fd6e9000700 1 Processor -- start 2024-04-02T09:13:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.095+0000 7fd6e9000700 1 -- start start 2024-04-02T09:13:01.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e9000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e40b2070 0x7fd6e415aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:01.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e9000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6e40cbb30 0x7fd6e40aca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:01.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e9000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6e40cd020 0x7fd6e41509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:01.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e9000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e40cae70 con 0x7fd6e40b2070 2024-04-02T09:13:01.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e9000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e415a030 con 0x7fd6e40cbb30 2024-04-02T09:13:01.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e9000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e40cb0c0 con 0x7fd6e40cd020 2024-04-02T09:13:01.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e40b2070 0x7fd6e415aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:01.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e40b2070 0x7fd6e415aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52668/0 (socket says 172.21.15.89:52668) 2024-04-02T09:13:01.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e1e00700 1 -- 172.21.15.89:0/1151840702 learned_addr learned my addr 172.21.15.89:0/1151840702 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:01.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e1400700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6e40cbb30 0x7fd6e40aca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:01.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e2800700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6e40cd020 0x7fd6e41509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:01.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e1e00700 1 -- 172.21.15.89:0/1151840702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6e40cd020 msgr2=0x7fd6e41509f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:01.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e1e00700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6e40cd020 0x7fd6e41509f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e1e00700 1 -- 172.21.15.89:0/1151840702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6e40cbb30 msgr2=0x7fd6e40aca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:01.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e1e00700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6e40cbb30 0x7fd6e40aca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e1e00700 1 -- 172.21.15.89:0/1151840702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d0009550 con 0x7fd6e40b2070 2024-04-02T09:13:01.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.096+0000 7fd6e2800700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6e40cd020 0x7fd6e41509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:01.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.097+0000 7fd6e1e00700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e40b2070 0x7fd6e415aaf0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400e870 tx=0x7fd6d400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:01.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.097+0000 7fd6da000700 1 -- 172.21.15.89:0/1151840702 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4009590 con 0x7fd6e40b2070 2024-04-02T09:13:01.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.097+0000 7fd6da000700 1 -- 172.21.15.89:0/1151840702 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d4013680 con 0x7fd6e40b2070 2024-04-02T09:13:01.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.097+0000 7fd6da000700 1 -- 172.21.15.89:0/1151840702 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d40139f0 con 0x7fd6e40b2070 2024-04-02T09:13:01.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.097+0000 7fd6e9000700 1 -- 172.21.15.89:0/1151840702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e40cb3a0 con 0x7fd6e40b2070 2024-04-02T09:13:01.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.097+0000 7fd6e9000700 1 -- 172.21.15.89:0/1151840702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e415a5d0 con 0x7fd6e40b2070 2024-04-02T09:13:01.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.098+0000 7fd6e9000700 1 -- 172.21.15.89:0/1151840702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e415a7c0 con 0x7fd6e40b2070 2024-04-02T09:13:01.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.099+0000 7fd6da000700 1 -- 172.21.15.89:0/1151840702 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fd6d4013b50 con 0x7fd6e40b2070 2024-04-02T09:13:01.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.100+0000 7fd6da000700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd6c406af90 0x7fd6c406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:01.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.100+0000 7fd6e1400700 1 -- 172.21.15.89:0/1151840702 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd6c406af90 msgr2=0x7fd6c406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:01.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.100+0000 7fd6da000700 1 -- 172.21.15.89:0/1151840702 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7fd6d4089d30 con 0x7fd6e40b2070 2024-04-02T09:13:01.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.100+0000 7fd6e1400700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd6c406af90 0x7fd6c406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:13:01.102 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:01.104+0000 7fc13ea36240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:13:01.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.105+0000 7fd6da000700 1 -- 172.21.15.89:0/1151840702 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd6d4054500 con 0x7fd6e40b2070 2024-04-02T09:13:01.165 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:01.167+0000 7f590ab3a240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:13:01.199 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:01.201+0000 7fc13ea36240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:13:01.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.301+0000 7fd6e1400700 1 -- 172.21.15.89:0/1151840702 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd6c406af90 msgr2=0x7fd6c406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:01.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.301+0000 7fd6e1400700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd6c406af90 0x7fd6c406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:13:01.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.315+0000 7fd6e9000700 1 -- 172.21.15.89:0/1151840702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fd6e415bda0 con 0x7fd6e40b2070 2024-04-02T09:13:01.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.315+0000 7fd6da000700 1 -- 172.21.15.89:0/1151840702 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v228) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd6d4057b20 con 0x7fd6e40b2070 2024-04-02T09:13:01.314 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:13:01.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.318+0000 7fd6e9000700 1 -- 172.21.15.89:0/1151840702 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd6c406af90 msgr2=0x7fd6c406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:13:01.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.318+0000 7fd6e9000700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd6c406af90 0x7fd6c406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.318+0000 7fd6e9000700 1 -- 172.21.15.89:0/1151840702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e40b2070 msgr2=0x7fd6e415aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:01.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.318+0000 7fd6e9000700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e40b2070 0x7fd6e415aaf0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400e870 tx=0x7fd6d400eb80 comp rx=0 tx=0).stop 2024-04-02T09:13:01.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.318+0000 7fd6e9000700 1 -- 172.21.15.89:0/1151840702 shutdown_connections 2024-04-02T09:13:01.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.318+0000 7fd6e9000700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6e40cd020 0x7fd6e41509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.318+0000 7fd6e9000700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6e40b2070 0x7fd6e415aaf0 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.318+0000 7fd6e9000700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd6c406af90 0x7fd6c406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.318+0000 7fd6e9000700 1 --2- 172.21.15.89:0/1151840702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6e40cbb30 0x7fd6e40aca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.318+0000 7fd6e9000700 1 -- 172.21.15.89:0/1151840702 >> 172.21.15.89:0/1151840702 conn(0x7fd6e40b0d70 msgr2=0x7fd6e40ce540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:01.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.318+0000 7fd6e9000700 1 -- 172.21.15.89:0/1151840702 shutdown_connections 2024-04-02T09:13:01.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.318+0000 7fd6e9000700 1 -- 172.21.15.89:0/1151840702 wait complete. 2024-04-02T09:13:01.334 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T09:13:01.373 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:01.375+0000 7fc13ea36240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:13:01.380 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:01.381+0000 7f590ab3a240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:13:01.471 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:01.472+0000 7f590ab3a240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:13:01.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.541+0000 7fe2cd200700 1 Processor -- start 2024-04-02T09:13:01.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.541+0000 7fe2cd200700 1 -- start start 2024-04-02T09:13:01.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.543+0000 7fe2cd200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe2c80cf3f0 0x7fe2c809a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:01.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.543+0000 7fe2cd200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c80ccdd0 con 0x7fe2c80cef40 2024-04-02T09:13:01.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.543+0000 7fe2cd200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c815cae0 con 0x7fe2c815f1e0 2024-04-02T09:13:01.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.543+0000 7fe2cd200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c80cd260 con 0x7fe2c80cf3f0 2024-04-02T09:13:01.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.543+0000 7fe2c6800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe2c80cf3f0 0x7fe2c809a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:01.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.543+0000 7fe2c5400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe2c80cef40 0x7fe2c8099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47676/0 (socket says 172.21.15.89:47676) 2024-04-02T09:13:01.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.543+0000 7fe2c5400700 1 -- 172.21.15.89:0/305807507 learned_addr learned my addr 172.21.15.89:0/305807507 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:01.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.543+0000 7fe2c6800700 1 -- 172.21.15.89:0/305807507 >> v1:172.21.15.113:6789/0 conn(0x7fe2c815f1e0 legacy=0x7fe2c815f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:01.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.543+0000 7fe2c6800700 1 -- 172.21.15.89:0/305807507 >> v1:172.21.15.89:6789/0 conn(0x7fe2c80cef40 legacy=0x7fe2c8099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:01.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.543+0000 7fe2c6800700 1 -- 172.21.15.89:0/305807507 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c80cc1c0 con 0x7fe2c80cf3f0 2024-04-02T09:13:01.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.543+0000 7fe2c4a00700 1 -- 172.21.15.89:0/305807507 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916155334 0 0) 0x7fe2c80ccdd0 con 0x7fe2c80cef40 2024-04-02T09:13:01.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.544+0000 7fe2c6800700 1 --2- 172.21.15.89:0/305807507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2c80cf3f0 0x7fe2c809a170 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009dc0 tx=0x7fe2c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98401ed452520ea4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:01.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.544+0000 7fe2c4a00700 1 -- 172.21.15.89:0/305807507 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c000e070 con 0x7fe2c80cf3f0 2024-04-02T09:13:01.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.544+0000 7fe2c4a00700 1 -- 172.21.15.89:0/305807507 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c000bb90 con 0x7fe2c80cf3f0 2024-04-02T09:13:01.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.544+0000 7fe2c4a00700 1 -- 172.21.15.89:0/305807507 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0012430 con 0x7fe2c80cf3f0 2024-04-02T09:13:01.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.545+0000 7fe2cd200700 1 -- 172.21.15.89:0/305807507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2c80cf3f0 msgr2=0x7fe2c809a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:01.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.545+0000 7fe2cd200700 1 --2- 172.21.15.89:0/305807507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2c80cf3f0 0x7fe2c809a170 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009dc0 tx=0x7fe2c000b330 comp rx=0 tx=0).stop 2024-04-02T09:13:01.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.545+0000 7fe2cd200700 1 -- 172.21.15.89:0/305807507 shutdown_connections 2024-04-02T09:13:01.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.545+0000 7fe2cd200700 1 --2- 172.21.15.89:0/305807507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2c80cf3f0 0x7fe2c809a170 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.545+0000 7fe2cd200700 1 -- 172.21.15.89:0/305807507 >> 172.21.15.89:0/305807507 conn(0x7fe2c80d1370 msgr2=0x7fe2c80d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:01.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.545+0000 7fe2cd200700 1 -- 172.21.15.89:0/305807507 shutdown_connections 2024-04-02T09:13:01.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.545+0000 7fe2cd200700 1 -- 172.21.15.89:0/305807507 wait complete. 2024-04-02T09:13:01.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.547+0000 7fe2cd200700 1 Processor -- start 2024-04-02T09:13:01.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.547+0000 7fe2cd200700 1 -- start start 2024-04-02T09:13:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.548+0000 7fe2cd200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2c80cef40 0x7fe2c806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.548+0000 7fe2cd200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80cf3f0 0x7fe2c806ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2cd200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2c815f1e0 0x7fe2c815d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2cd200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c80c9260 con 0x7fe2c80cef40 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2cd200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c8154310 con 0x7fe2c80cf3f0 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2cd200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c80c85e0 con 0x7fe2c815f1e0 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2c5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2c80cef40 0x7fe2c806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2c5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2c80cef40 0x7fe2c806c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52692/0 (socket says 172.21.15.89:52692) 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2c5e00700 1 -- 172.21.15.89:0/2888761148 learned_addr learned my addr 172.21.15.89:0/2888761148 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2c5400700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80cf3f0 0x7fe2c806ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2c6800700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2c815f1e0 0x7fe2c815d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2c5e00700 1 -- 172.21.15.89:0/2888761148 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2c815f1e0 msgr2=0x7fe2c815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2c5e00700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2c815f1e0 0x7fe2c815d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2c5e00700 1 -- 172.21.15.89:0/2888761148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80cf3f0 msgr2=0x7fe2c806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2c5e00700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80cf3f0 0x7fe2c806ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2c5e00700 1 -- 172.21.15.89:0/2888761148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c0009a70 con 0x7fe2c80cef40 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2c6800700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2c815f1e0 0x7fe2c815d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.549+0000 7fe2c5e00700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2c80cef40 0x7fe2c806c8f0 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fe2b800c450 tx=0x7fe2b800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:01.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.550+0000 7fe2be000700 1 -- 172.21.15.89:0/2888761148 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b80052d0 con 0x7fe2c80cef40 2024-04-02T09:13:01.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.550+0000 7fe2be000700 1 -- 172.21.15.89:0/2888761148 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b8005430 con 0x7fe2c80cef40 2024-04-02T09:13:01.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.550+0000 7fe2be000700 1 -- 172.21.15.89:0/2888761148 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b80038c0 con 0x7fe2c80cef40 2024-04-02T09:13:01.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.550+0000 7fe2cd200700 1 -- 172.21.15.89:0/2888761148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c80c88c0 con 0x7fe2c80cef40 2024-04-02T09:13:01.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.550+0000 7fe2cd200700 1 -- 172.21.15.89:0/2888761148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c80c8e80 con 0x7fe2c80cef40 2024-04-02T09:13:01.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.551+0000 7fe2cd200700 1 -- 172.21.15.89:0/2888761148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c8067720 con 0x7fe2c80cef40 2024-04-02T09:13:01.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.552+0000 7fe2be000700 1 -- 172.21.15.89:0/2888761148 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fe2b8003a20 con 0x7fe2c80cef40 2024-04-02T09:13:01.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.553+0000 7fe2be000700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe2a806af60 0x7fe2a806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:01.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.553+0000 7fe2be000700 1 -- 172.21.15.89:0/2888761148 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7fe2b8089180 con 0x7fe2c80cef40 2024-04-02T09:13:01.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.553+0000 7fe2c5400700 1 -- 172.21.15.89:0/2888761148 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe2a806af60 msgr2=0x7fe2a806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:01.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.553+0000 7fe2c5400700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe2a806af60 0x7fe2a806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:13:01.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.557+0000 7fe2be000700 1 -- 172.21.15.89:0/2888761148 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe2b80539e0 con 0x7fe2c80cef40 2024-04-02T09:13:01.569 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:01.570+0000 7f590ab3a240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:13:01.600 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:01.602+0000 7fc13ea36240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:13:01.667 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:01.668+0000 7f590ab3a240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:13:01.697 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:01.697+0000 7fc13ea36240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:13:01.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.754+0000 7fe2c5400700 1 -- 172.21.15.89:0/2888761148 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe2a806af60 msgr2=0x7fe2a806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:01.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.754+0000 7fe2c5400700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe2a806af60 0x7fe2a806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:13:01.764 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:01.766+0000 7f590ab3a240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:13:01.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.783+0000 7fe2cd200700 1 -- 172.21.15.89:0/2888761148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe2c80d20b0 con 0x7fe2c80cef40 2024-04-02T09:13:01.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.784+0000 7fe2be000700 1 -- 172.21.15.89:0/2888761148 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v228) v1 ==== 80+0+460 (secure 0 0 0) 0x7fe2b80171e0 con 0x7fe2c80cef40 2024-04-02T09:13:01.782 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2216461018 2024-04-03T09:12:53.003657+0000 2024-04-02T09:13:01.782 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3268555703 2024-04-03T09:12:53.003657+0000 2024-04-02T09:13:01.782 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3481438563 2024-04-03T09:12:53.003657+0000 2024-04-02T09:13:01.782 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/783107677 2024-04-03T09:12:53.003657+0000 2024-04-02T09:13:01.782 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2439246170 2024-04-03T09:12:53.003657+0000 2024-04-02T09:13:01.783 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/965177671 2024-04-03T09:12:53.003657+0000 2024-04-02T09:13:01.783 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/108855 2024-04-03T09:12:53.003657+0000 2024-04-02T09:13:01.783 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/108855 2024-04-03T09:12:53.003657+0000 2024-04-02T09:13:01.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.786+0000 7fe2cd200700 1 -- 172.21.15.89:0/2888761148 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe2a806af60 msgr2=0x7fe2a806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:13:01.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.786+0000 7fe2cd200700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe2a806af60 0x7fe2a806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.786+0000 7fe2cd200700 1 -- 172.21.15.89:0/2888761148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2c80cef40 msgr2=0x7fe2c806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:01.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.786+0000 7fe2cd200700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2c80cef40 0x7fe2c806c8f0 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fe2b800c450 tx=0x7fe2b800c760 comp rx=0 tx=0).stop 2024-04-02T09:13:01.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.786+0000 7fe2cd200700 1 -- 172.21.15.89:0/2888761148 shutdown_connections 2024-04-02T09:13:01.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.786+0000 7fe2cd200700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe2c815f1e0 0x7fe2c815d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.786+0000 7fe2cd200700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2c80cef40 0x7fe2c806c8f0 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.786+0000 7fe2cd200700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe2a806af60 0x7fe2a806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.786+0000 7fe2cd200700 1 --2- 172.21.15.89:0/2888761148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80cf3f0 0x7fe2c806ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:01.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.786+0000 7fe2cd200700 1 -- 172.21.15.89:0/2888761148 >> 172.21.15.89:0/2888761148 conn(0x7fe2c80d1370 msgr2=0x7fe2c8099fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:01.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.787+0000 7fe2cd200700 1 -- 172.21.15.89:0/2888761148 shutdown_connections 2024-04-02T09:13:01.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:01.787+0000 7fe2cd200700 1 -- 172.21.15.89:0/2888761148 wait complete. 2024-04-02T09:13:01.786 INFO:teuthology.orchestra.run.smithi089.stderr:listed 8 entries 2024-04-02T09:13:01.798 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:01.800+0000 7fc13ea36240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:13:01.803 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T09:13:01.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:13:01.803 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T09:13:01.908 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:01.909+0000 7fc13ea36240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:13:01.923 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T09:13:01.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:13:01.923 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T09:13:01.974 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:13:01.974 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:13:01.979 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:01.979+0000 7f590ab3a240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:13:02.019 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:02.021+0000 7fc13ea36240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:13:02.023 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:13:02.023 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:13:02.023 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:13:02.073 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:13:02.073 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:13:02.073 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:13:02.120 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:13:02.120 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:13:02.247 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:02.248+0000 7f590ab3a240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:13:02.257 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:02.259+0000 7fc13ea36240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:13:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.323+0000 7f2f0e600700 1 Processor -- start 2024-04-02T09:13:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.324+0000 7f2f0e600700 1 -- start start 2024-04-02T09:13:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.325+0000 7f2f0e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f0809a910 0x7f2f0806c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.325+0000 7f2f0e600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2f08099a60 0x7f2f08099eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.325+0000 7f2f0e600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0809a3f0 con 0x7f2f0815d920 2024-04-02T09:13:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.325+0000 7f2f0e600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f080d08d0 con 0x7f2f08099a60 2024-04-02T09:13:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.325+0000 7f2f0e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0806c300 con 0x7f2f0809a910 2024-04-02T09:13:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.325+0000 7f2f07e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2f08099a60 0x7f2f08099eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.325+0000 7f2effe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f0809a910 0x7f2f0806c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.325+0000 7f2f07400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f2f0815d920 0x7f2f080cfff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:40080/0 (socket says 172.21.15.89:40080) 2024-04-02T09:13:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.325+0000 7f2f07400700 1 -- 172.21.15.89:0/4138968766 learned_addr learned my addr 172.21.15.89:0/4138968766 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.326+0000 7f2f06a00700 1 -- 172.21.15.89:0/4138968766 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098128592 0 0) 0x7f2ef8001110 con 0x7f2f0815d920 2024-04-02T09:13:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.326+0000 7f2f06a00700 1 -- 172.21.15.89:0/4138968766 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eec0035b0 con 0x7f2f0815d920 2024-04-02T09:13:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.326+0000 7f2f07e00700 1 -- 172.21.15.89:0/4138968766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f0809a910 msgr2=0x7f2f0806c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.326+0000 7f2f07e00700 1 --2- 172.21.15.89:0/4138968766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f0809a910 0x7f2f0806c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.326+0000 7f2f07e00700 1 -- 172.21.15.89:0/4138968766 >> v1:172.21.15.113:6789/0 conn(0x7f2f0815d920 legacy=0x7f2f080cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.326+0000 7f2f07e00700 1 -- 172.21.15.89:0/4138968766 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0806b9d0 con 0x7f2f08099a60 2024-04-02T09:13:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.326+0000 7f2effe00700 1 --2- 172.21.15.89:0/4138968766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f0809a910 0x7f2f0806c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.326+0000 7f2f07e00700 1 --2- 172.21.15.89:0/4138968766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f08099a60 0x7f2f08099eb0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009d90 tx=0x7f2f0000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b83ad004d9757a9c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.326+0000 7f2f06a00700 1 -- 172.21.15.89:0/4138968766 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0000e070 con 0x7f2f08099a60 2024-04-02T09:13:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.326+0000 7f2f06a00700 1 -- 172.21.15.89:0/4138968766 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0000bb20 con 0x7f2f08099a60 2024-04-02T09:13:02.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.326+0000 7f2f06a00700 1 -- 172.21.15.89:0/4138968766 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f00012430 con 0x7f2f08099a60 2024-04-02T09:13:02.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.327+0000 7f2f0e600700 1 -- 172.21.15.89:0/4138968766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f08099a60 msgr2=0x7f2f08099eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:02.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.327+0000 7f2f0e600700 1 --2- 172.21.15.89:0/4138968766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f08099a60 0x7f2f08099eb0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009d90 tx=0x7f2f0000b330 comp rx=0 tx=0).stop 2024-04-02T09:13:02.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.327+0000 7f2f0e600700 1 -- 172.21.15.89:0/4138968766 shutdown_connections 2024-04-02T09:13:02.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.327+0000 7f2f0e600700 1 --2- 172.21.15.89:0/4138968766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f08099a60 0x7f2f08099eb0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.327+0000 7f2f0e600700 1 --2- 172.21.15.89:0/4138968766 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f0809a910 0x7f2f0806c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.327+0000 7f2f0e600700 1 -- 172.21.15.89:0/4138968766 >> 172.21.15.89:0/4138968766 conn(0x7f2f080d1350 msgr2=0x7f2f080d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:02.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.327+0000 7f2f0e600700 1 -- 172.21.15.89:0/4138968766 shutdown_connections 2024-04-02T09:13:02.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.327+0000 7f2f0e600700 1 -- 172.21.15.89:0/4138968766 wait complete. 2024-04-02T09:13:02.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.329+0000 7f2f0e600700 1 Processor -- start 2024-04-02T09:13:02.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.330+0000 7f2f0e600700 1 -- start start 2024-04-02T09:13:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.331+0000 7f2f0e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f08099a60 0x7f2f080ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.331+0000 7f2f0e600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f0809a910 0x7f2f080b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.331+0000 7f2f0e600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f0815d920 0x7f2f080b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.331+0000 7f2f0e600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0815f690 con 0x7f2f0815d920 2024-04-02T09:13:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.331+0000 7f2f0e600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0815ea80 con 0x7f2f0809a910 2024-04-02T09:13:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.331+0000 7f2f0e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0815ed60 con 0x7f2f08099a60 2024-04-02T09:13:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.331+0000 7f2effe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f0809a910 0x7f2f080b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.331+0000 7f2f07400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f08099a60 0x7f2f080ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.331+0000 7f2f07e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f0815d920 0x7f2f080b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.331+0000 7f2effe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f0809a910 0x7f2f080b4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:54002/0 (socket says 172.21.15.89:54002) 2024-04-02T09:13:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.331+0000 7f2effe00700 1 -- 172.21.15.89:0/1930424459 learned_addr learned my addr 172.21.15.89:0/1930424459 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:02.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.331+0000 7f2effe00700 1 -- 172.21.15.89:0/1930424459 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f08099a60 msgr2=0x7f2f080ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.331+0000 7f2effe00700 1 --2- 172.21.15.89:0/1930424459 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f08099a60 0x7f2f080ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.331+0000 7f2effe00700 1 -- 172.21.15.89:0/1930424459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f0815d920 msgr2=0x7f2f080b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.332+0000 7f2effe00700 1 --2- 172.21.15.89:0/1930424459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f0815d920 0x7f2f080b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.332+0000 7f2effe00700 1 -- 172.21.15.89:0/1930424459 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef00094a0 con 0x7f2f0809a910 2024-04-02T09:13:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.332+0000 7f2f07e00700 1 --2- 172.21.15.89:0/1930424459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f0815d920 0x7f2f080b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.332+0000 7f2f07400700 1 --2- 172.21.15.89:0/1930424459 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f08099a60 0x7f2f080ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.332+0000 7f2effe00700 1 --2- 172.21.15.89:0/1930424459 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f0809a910 0x7f2f080b4ce0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2ef000ebe0 tx=0x7f2ef000eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.332+0000 7f2eff400700 1 -- 172.21.15.89:0/1930424459 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef00097a0 con 0x7f2f0809a910 2024-04-02T09:13:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.332+0000 7f2eff400700 1 -- 172.21.15.89:0/1930424459 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef000c850 con 0x7f2f0809a910 2024-04-02T09:13:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.332+0000 7f2f0e600700 1 -- 172.21.15.89:0/1930424459 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f00009a70 con 0x7f2f0809a910 2024-04-02T09:13:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.332+0000 7f2eff400700 1 -- 172.21.15.89:0/1930424459 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef000cbe0 con 0x7f2f0809a910 2024-04-02T09:13:02.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.332+0000 7f2f0e600700 1 -- 172.21.15.89:0/1930424459 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0815f320 con 0x7f2f0809a910 2024-04-02T09:13:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.333+0000 7f2f0e600700 1 -- 172.21.15.89:0/1930424459 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f08067720 con 0x7f2f0809a910 2024-04-02T09:13:02.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.334+0000 7f2eff400700 1 -- 172.21.15.89:0/1930424459 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f2ef000cd40 con 0x7f2f0809a910 2024-04-02T09:13:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.335+0000 7f2eff400700 1 --2- 172.21.15.89:0/1930424459 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ee406af60 0x7f2ee406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.335+0000 7f2eff400700 1 -- 172.21.15.89:0/1930424459 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f2ef0089b90 con 0x7f2f0809a910 2024-04-02T09:13:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.335+0000 7f2f07400700 1 -- 172.21.15.89:0/1930424459 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ee406af60 msgr2=0x7f2ee406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:02.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.335+0000 7f2f07400700 1 --2- 172.21.15.89:0/1930424459 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ee406af60 0x7f2ee406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:13:02.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.340+0000 7f2eff400700 1 -- 172.21.15.89:0/1930424459 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2ef0054360 con 0x7f2f0809a910 2024-04-02T09:13:02.438 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:02.439+0000 7f590ab3a240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:13:02.525 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:02.527+0000 7fc13ea36240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:13:02.530 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:02.532+0000 7f590ab3a240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:13:02.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.536+0000 7f2f07400700 1 -- 172.21.15.89:0/1930424459 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ee406af60 msgr2=0x7f2ee406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:02.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.536+0000 7f2f07400700 1 --2- 172.21.15.89:0/1930424459 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ee406af60 0x7f2ee406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:13:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.594+0000 7f2f0e600700 1 -- 172.21.15.89:0/1930424459 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f0815f7d0 con 0x7f2f0809a910 2024-04-02T09:13:02.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.595+0000 7f2eff400700 1 -- 172.21.15.89:0/1930424459 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5971 (secure 0 0 0) 0x7f2ef0057980 con 0x7f2f0809a910 2024-04-02T09:13:02.594 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:02.594 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":110,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8969,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3650347444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3650347444},{"type":"v1","addr":"172.21.15.89:6836","nonce":3650347444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2152500685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2152500685},{"type":"v1","addr":"172.21.15.113:6834","nonce":2152500685}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:25.170849+0000","modified":"2024-04-02T09:12:28.185307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11157":{"gid":11157,"name":"a","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2615927215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2615927215},{"type":"v1","addr":"172.21.15.89:6837","nonce":2615927215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:39.106317+0000","modified":"2024-04-02T09:12:42.112504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10279":{"gid":10279,"name":"d","rank":0,"incarnation":107,"state":"up:active","state_seq":6,"addr":"172.21.15.113:6835/2457040702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2457040702},{"type":"v1","addr":"172.21.15.113:6835","nonce":2457040702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-02T09:13:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.598+0000 7f2f0e600700 1 -- 172.21.15.89:0/1930424459 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ee406af60 msgr2=0x7f2ee406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:13:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.598+0000 7f2f0e600700 1 --2- 172.21.15.89:0/1930424459 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ee406af60 0x7f2ee406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.598+0000 7f2f0e600700 1 -- 172.21.15.89:0/1930424459 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f0809a910 msgr2=0x7f2f080b4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:02.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.598+0000 7f2f0e600700 1 --2- 172.21.15.89:0/1930424459 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f0809a910 0x7f2f080b4ce0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2ef000ebe0 tx=0x7f2ef000eef0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.598+0000 7f2f0e600700 1 -- 172.21.15.89:0/1930424459 shutdown_connections 2024-04-02T09:13:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.598+0000 7f2f0e600700 1 --2- 172.21.15.89:0/1930424459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f0815d920 0x7f2f080b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.598+0000 7f2f0e600700 1 --2- 172.21.15.89:0/1930424459 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f08099a60 0x7f2f080ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.598+0000 7f2f0e600700 1 --2- 172.21.15.89:0/1930424459 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ee406af60 0x7f2ee406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.598+0000 7f2f0e600700 1 --2- 172.21.15.89:0/1930424459 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f0809a910 0x7f2f080b4ce0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.598+0000 7f2f0e600700 1 -- 172.21.15.89:0/1930424459 >> 172.21.15.89:0/1930424459 conn(0x7f2f080d1350 msgr2=0x7f2f08070020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.598+0000 7f2f0e600700 1 -- 172.21.15.89:0/1930424459 shutdown_connections 2024-04-02T09:13:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.598+0000 7f2f0e600700 1 -- 172.21.15.89:0/1930424459 wait complete. 2024-04-02T09:13:02.597 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 110 2024-04-02T09:13:02.616 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:13:02.740 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:02.742+0000 7fc13ea36240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:13:02.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.825+0000 7f1962200700 1 Processor -- start 2024-04-02T09:13:02.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.825+0000 7f1962200700 1 -- start start 2024-04-02T09:13:02.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.826+0000 7f1962200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f195c15d640 0x7f195c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:02.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.826+0000 7f1962200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f195c0ceec0 0x7f195c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:02.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.827+0000 7f1962200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195c0d30d0 con 0x7f195c161800 2024-04-02T09:13:02.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.827+0000 7f1962200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195c0b6840 con 0x7f195c0ceec0 2024-04-02T09:13:02.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.827+0000 7f1962200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195c161c20 con 0x7f195c15d640 2024-04-02T09:13:02.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.827+0000 7f195aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f195c0ceec0 0x7f195c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:02.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.827+0000 7f195aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f195c0ceec0 0x7f195c1612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52712/0 (socket says 172.21.15.89:52712) 2024-04-02T09:13:02.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.827+0000 7f195aa00700 1 -- 172.21.15.89:0/3639849529 learned_addr learned my addr 172.21.15.89:0/3639849529 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.827+0000 7f195b400700 1 --2- 172.21.15.89:0/3639849529 >> v2:172.21.15.113:3300/0 conn(0x7f195c15d640 0x7f195c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.827+0000 7f195aa00700 1 -- 172.21.15.89:0/3639849529 >> v2:172.21.15.113:3300/0 conn(0x7f195c15d640 msgr2=0x7f195c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.827+0000 7f195aa00700 1 --2- 172.21.15.89:0/3639849529 >> v2:172.21.15.113:3300/0 conn(0x7f195c15d640 0x7f195c0d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.827+0000 7f195aa00700 1 -- 172.21.15.89:0/3639849529 >> v1:172.21.15.113:6789/0 conn(0x7f195c161800 legacy=0x7f195c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.827+0000 7f195aa00700 1 -- 172.21.15.89:0/3639849529 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c0d2880 con 0x7f195c0ceec0 2024-04-02T09:13:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.828+0000 7f195aa00700 1 --2- 172.21.15.89:0/3639849529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f195c0ceec0 0x7f195c1612c0 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f195c0b5c20 tx=0x7f1948009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c687fec9fab46086 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.828+0000 7f195a000700 1 -- 172.21.15.89:0/3639849529 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1948011830 con 0x7f195c0ceec0 2024-04-02T09:13:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.828+0000 7f195a000700 1 -- 172.21.15.89:0/3639849529 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1948011990 con 0x7f195c0ceec0 2024-04-02T09:13:02.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.828+0000 7f195a000700 1 -- 172.21.15.89:0/3639849529 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1948011d20 con 0x7f195c0ceec0 2024-04-02T09:13:02.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.828+0000 7f1962200700 1 -- 172.21.15.89:0/3639849529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f195c0ceec0 msgr2=0x7f195c1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:02.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.828+0000 7f1962200700 1 --2- 172.21.15.89:0/3639849529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f195c0ceec0 0x7f195c1612c0 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f195c0b5c20 tx=0x7f1948009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.829+0000 7f1962200700 1 -- 172.21.15.89:0/3639849529 shutdown_connections 2024-04-02T09:13:02.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.829+0000 7f1962200700 1 --2- 172.21.15.89:0/3639849529 >> v2:172.21.15.113:3300/0 conn(0x7f195c15d640 0x7f195c0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.829+0000 7f1962200700 1 --2- 172.21.15.89:0/3639849529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f195c0ceec0 0x7f195c1612c0 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.829+0000 7f1962200700 1 -- 172.21.15.89:0/3639849529 >> 172.21.15.89:0/3639849529 conn(0x7f195c08aff0 msgr2=0x7f195c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:02.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.829+0000 7f1962200700 1 -- 172.21.15.89:0/3639849529 shutdown_connections 2024-04-02T09:13:02.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.829+0000 7f1962200700 1 -- 172.21.15.89:0/3639849529 wait complete. 2024-04-02T09:13:02.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.830+0000 7f1962200700 1 Processor -- start 2024-04-02T09:13:02.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.830+0000 7f1962200700 1 -- start start 2024-04-02T09:13:02.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.831+0000 7f1962200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f195c0ceec0 0x7f195c158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:02.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.831+0000 7f1962200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f195c15d640 0x7f195c09e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:02.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.831+0000 7f1962200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f195c161800 0x7f195c15f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:02.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.831+0000 7f1962200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c0a0be0 con 0x7f195c15d640 2024-04-02T09:13:02.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.831+0000 7f1962200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c158f80 con 0x7f195c0ceec0 2024-04-02T09:13:02.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.831+0000 7f1962200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195c0a02b0 con 0x7f195c161800 2024-04-02T09:13:02.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.832+0000 7f195aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f195c15d640 0x7f195c09e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:02.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.832+0000 7f195aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f195c15d640 0x7f195c09e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52716/0 (socket says 172.21.15.89:52716) 2024-04-02T09:13:02.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.832+0000 7f195aa00700 1 -- 172.21.15.89:0/3356546158 learned_addr learned my addr 172.21.15.89:0/3356546158 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:02.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.832+0000 7f195b400700 1 --2- 172.21.15.89:0/3356546158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f195c0ceec0 0x7f195c158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:02.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.832+0000 7f195be00700 1 --2- 172.21.15.89:0/3356546158 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f195c161800 0x7f195c15f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:02.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.832+0000 7f195aa00700 1 -- 172.21.15.89:0/3356546158 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f195c161800 msgr2=0x7f195c15f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:02.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.832+0000 7f195aa00700 1 --2- 172.21.15.89:0/3356546158 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f195c161800 0x7f195c15f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.832+0000 7f195aa00700 1 -- 172.21.15.89:0/3356546158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f195c0ceec0 msgr2=0x7f195c158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:02.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.832+0000 7f195aa00700 1 --2- 172.21.15.89:0/3356546158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f195c0ceec0 0x7f195c158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:02.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.832+0000 7f195aa00700 1 -- 172.21.15.89:0/3356546158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1948009550 con 0x7f195c15d640 2024-04-02T09:13:02.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.832+0000 7f195aa00700 1 --2- 172.21.15.89:0/3356546158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f195c15d640 0x7f195c09e1e0 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f1948009c10 tx=0x7f1948004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:02.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.833+0000 7f1953400700 1 -- 172.21.15.89:0/3356546158 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1948004510 con 0x7f195c15d640 2024-04-02T09:13:02.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.833+0000 7f1953400700 1 -- 172.21.15.89:0/3356546158 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194801e070 con 0x7f195c15d640 2024-04-02T09:13:02.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.833+0000 7f1962200700 1 -- 172.21.15.89:0/3356546158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195c0a0590 con 0x7f195c15d640 2024-04-02T09:13:02.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.833+0000 7f1953400700 1 -- 172.21.15.89:0/3356546158 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19480038e0 con 0x7f195c15d640 2024-04-02T09:13:02.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.833+0000 7f1962200700 1 -- 172.21.15.89:0/3356546158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195c0a1070 con 0x7f195c15d640 2024-04-02T09:13:02.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.834+0000 7f1962200700 1 -- 172.21.15.89:0/3356546158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f195c0a0d90 con 0x7f195c15d640 2024-04-02T09:13:02.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.835+0000 7f1953400700 1 -- 172.21.15.89:0/3356546158 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f1948003a40 con 0x7f195c15d640 2024-04-02T09:13:02.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.835+0000 7f1953400700 1 --2- 172.21.15.89:0/3356546158 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f193c06af60 0x7f193c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:02.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.836+0000 7f1953400700 1 -- 172.21.15.89:0/3356546158 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f194808ab50 con 0x7f195c15d640 2024-04-02T09:13:02.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.836+0000 7f195b400700 1 -- 172.21.15.89:0/3356546158 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f193c06af60 msgr2=0x7f193c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:02.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.836+0000 7f195b400700 1 --2- 172.21.15.89:0/3356546158 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f193c06af60 0x7f193c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:13:02.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:02.844+0000 7f1953400700 1 -- 172.21.15.89:0/3356546158 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f195c0a0d90 con 0x7f195c15d640 2024-04-02T09:13:02.843 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:02.845+0000 7fc13ea36240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:13:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.037+0000 7f195b400700 1 -- 172.21.15.89:0/3356546158 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f193c06af60 msgr2=0x7f193c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.037+0000 7f195b400700 1 --2- 172.21.15.89:0/3356546158 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f193c06af60 0x7f193c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:13:03.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.053+0000 7f1962200700 1 -- 172.21.15.89:0/3356546158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f195c067720 con 0x7f195c15d640 2024-04-02T09:13:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.055+0000 7f1953400700 1 -- 172.21.15.89:0/3356546158 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) v1 ==== 74+0+20955 (secure 0 0 0) 0x7f19480553b0 con 0x7f195c15d640 2024-04-02T09:13:03.054 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:03.054 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":228,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:12:53.003725+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:12:23.034869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-02T09:12:24.158560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:12:37.240568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:12:38.093999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/2216461018":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/3268555703":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/3481438563":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/783107677":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/2439246170":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/965177671":"2024-04-03T09:12:53.003657+0000","172.21.15.89:6833/108855":"2024-04-03T09:12:53.003657+0000","172.21.15.89:6832/108855":"2024-04-03T09:12:53.003657+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:13:03.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.058+0000 7f1962200700 1 -- 172.21.15.89:0/3356546158 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f193c06af60 msgr2=0x7f193c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:13:03.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.058+0000 7f1962200700 1 --2- 172.21.15.89:0/3356546158 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f193c06af60 0x7f193c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.058+0000 7f1962200700 1 -- 172.21.15.89:0/3356546158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f195c15d640 msgr2=0x7f195c09e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:03.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.058+0000 7f1962200700 1 --2- 172.21.15.89:0/3356546158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f195c15d640 0x7f195c09e1e0 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f1948009c10 tx=0x7f1948004750 comp rx=0 tx=0).stop 2024-04-02T09:13:03.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.058+0000 7f1962200700 1 -- 172.21.15.89:0/3356546158 shutdown_connections 2024-04-02T09:13:03.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.058+0000 7f1962200700 1 --2- 172.21.15.89:0/3356546158 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f193c06af60 0x7f193c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.058+0000 7f1962200700 1 --2- 172.21.15.89:0/3356546158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f195c0ceec0 0x7f195c158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.058+0000 7f1962200700 1 --2- 172.21.15.89:0/3356546158 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f195c161800 0x7f195c15f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.058+0000 7f1962200700 1 --2- 172.21.15.89:0/3356546158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f195c15d640 0x7f195c09e1e0 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.058+0000 7f1962200700 1 -- 172.21.15.89:0/3356546158 >> 172.21.15.89:0/3356546158 conn(0x7f195c08aff0 msgr2=0x7f195c15b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:03.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.058+0000 7f1962200700 1 -- 172.21.15.89:0/3356546158 shutdown_connections 2024-04-02T09:13:03.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.058+0000 7f1962200700 1 -- 172.21.15.89:0/3356546158 wait complete. 2024-04-02T09:13:03.077 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T09:12:23.034869+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T09:12:24.158560+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '223', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-02T09:12:37.240568+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-02T09:12:38.093999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:13:03.078 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:13:03.262 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:03.264+0000 7f590ab3a240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:13:03.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.286+0000 7fd114800700 1 Processor -- start 2024-04-02T09:13:03.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.286+0000 7fd114800700 1 -- start start 2024-04-02T09:13:03.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.287+0000 7fd114800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd10c0cbb30 0x7fd10c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:03.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.287+0000 7fd114800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd10c0cc450 con 0x7fd10c0cd4a0 2024-04-02T09:13:03.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.287+0000 7fd114800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd10c0cdae0 con 0x7fd10c0cd020 2024-04-02T09:13:03.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.287+0000 7fd114800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c0cddc0 con 0x7fd10c0cbb30 2024-04-02T09:13:03.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.288+0000 7fd112800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd10c0cd4a0 0x7fd10c06e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47680/0 (socket says 172.21.15.89:47680) 2024-04-02T09:13:03.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.288+0000 7fd112800700 1 -- 172.21.15.89:0/2109770181 learned_addr learned my addr 172.21.15.89:0/2109770181 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:03.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.288+0000 7fd111e00700 1 --2- 172.21.15.89:0/2109770181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd10c0cbb30 0x7fd10c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:03.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.288+0000 7fd110a00700 1 -- 172.21.15.89:0/2109770181 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999474119 0 0) 0x7fd10c0cc450 con 0x7fd10c0cd4a0 2024-04-02T09:13:03.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.288+0000 7fd110a00700 1 -- 172.21.15.89:0/2109770181 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0f40035b0 con 0x7fd10c0cd4a0 2024-04-02T09:13:03.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.288+0000 7fd110a00700 1 -- 172.21.15.89:0/2109770181 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309805099 0 0) 0x7fd10c0cdae0 con 0x7fd10c0cd020 2024-04-02T09:13:03.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.288+0000 7fd110a00700 1 -- 172.21.15.89:0/2109770181 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd10c0cc450 con 0x7fd10c0cd020 2024-04-02T09:13:03.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.288+0000 7fd111e00700 1 -- 172.21.15.89:0/2109770181 >> v1:172.21.15.113:6789/0 conn(0x7fd10c0cd020 legacy=0x7fd10c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:03.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.288+0000 7fd111e00700 1 -- 172.21.15.89:0/2109770181 >> v1:172.21.15.89:6789/0 conn(0x7fd10c0cd4a0 legacy=0x7fd10c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:03.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.288+0000 7fd111e00700 1 -- 172.21.15.89:0/2109770181 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10c0ce0a0 con 0x7fd10c0cbb30 2024-04-02T09:13:03.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.289+0000 7fd111e00700 1 --2- 172.21.15.89:0/2109770181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd10c0cbb30 0x7fd10c0cbf10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd100009870 tx=0x7fd100009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=335bdb1ad9acdec9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:03.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.289+0000 7fd110a00700 1 -- 172.21.15.89:0/2109770181 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100011830 con 0x7fd10c0cbb30 2024-04-02T09:13:03.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.289+0000 7fd110a00700 1 -- 172.21.15.89:0/2109770181 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd100011990 con 0x7fd10c0cbb30 2024-04-02T09:13:03.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.289+0000 7fd110a00700 1 -- 172.21.15.89:0/2109770181 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100011d20 con 0x7fd10c0cbb30 2024-04-02T09:13:03.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.289+0000 7fd114800700 1 -- 172.21.15.89:0/2109770181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd10c0cbb30 msgr2=0x7fd10c0cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:03.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.289+0000 7fd114800700 1 --2- 172.21.15.89:0/2109770181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd10c0cbb30 0x7fd10c0cbf10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd100009870 tx=0x7fd100009b80 comp rx=0 tx=0).stop 2024-04-02T09:13:03.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.290+0000 7fd114800700 1 -- 172.21.15.89:0/2109770181 shutdown_connections 2024-04-02T09:13:03.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.290+0000 7fd114800700 1 --2- 172.21.15.89:0/2109770181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd10c0cbb30 0x7fd10c0cbf10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.290+0000 7fd114800700 1 -- 172.21.15.89:0/2109770181 >> 172.21.15.89:0/2109770181 conn(0x7fd10c0b0d70 msgr2=0x7fd10c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:03.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.290+0000 7fd114800700 1 -- 172.21.15.89:0/2109770181 shutdown_connections 2024-04-02T09:13:03.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.290+0000 7fd114800700 1 -- 172.21.15.89:0/2109770181 wait complete. 2024-04-02T09:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.291+0000 7fd114800700 1 Processor -- start 2024-04-02T09:13:03.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.292+0000 7fd114800700 1 -- start start 2024-04-02T09:13:03.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.292+0000 7fd114800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0cbb30 0x7fd10c072510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:03.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.292+0000 7fd114800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd10c0cd020 0x7fd10c15d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:03.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.292+0000 7fd114800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd10c0cd4a0 0x7fd10c1f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:03.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.292+0000 7fd114800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c14fc20 con 0x7fd10c0cd020 2024-04-02T09:13:03.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.292+0000 7fd114800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c072a50 con 0x7fd10c0cbb30 2024-04-02T09:13:03.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.292+0000 7fd114800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c0c8970 con 0x7fd10c0cd4a0 2024-04-02T09:13:03.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.293+0000 7fd111400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd10c0cd020 0x7fd10c15d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:03.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.293+0000 7fd111400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd10c0cd020 0x7fd10c15d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52730/0 (socket says 172.21.15.89:52730) 2024-04-02T09:13:03.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.293+0000 7fd111400700 1 -- 172.21.15.89:0/225834974 learned_addr learned my addr 172.21.15.89:0/225834974 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:03.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.293+0000 7fd112800700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd10c0cd4a0 0x7fd10c1f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:03.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.293+0000 7fd111e00700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0cbb30 0x7fd10c072510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:03.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.293+0000 7fd111400700 1 -- 172.21.15.89:0/225834974 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd10c0cd4a0 msgr2=0x7fd10c1f2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:03.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.293+0000 7fd111400700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd10c0cd4a0 0x7fd10c1f2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.293+0000 7fd111400700 1 -- 172.21.15.89:0/225834974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0cbb30 msgr2=0x7fd10c072510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:03.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.293+0000 7fd111400700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0cbb30 0x7fd10c072510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.293+0000 7fd111400700 1 -- 172.21.15.89:0/225834974 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1000094a0 con 0x7fd10c0cd020 2024-04-02T09:13:03.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.293+0000 7fd111e00700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0cbb30 0x7fd10c072510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:03.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.293+0000 7fd111400700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd10c0cd020 0x7fd10c15d130 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc00c550 tx=0x7fd0fc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:03.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.294+0000 7fd106000700 1 -- 172.21.15.89:0/225834974 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0fc0052d0 con 0x7fd10c0cd020 2024-04-02T09:13:03.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.294+0000 7fd106000700 1 -- 172.21.15.89:0/225834974 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0fc005430 con 0x7fd10c0cd020 2024-04-02T09:13:03.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.294+0000 7fd114800700 1 -- 172.21.15.89:0/225834974 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10c1597e0 con 0x7fd10c0cd020 2024-04-02T09:13:03.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.294+0000 7fd106000700 1 -- 172.21.15.89:0/225834974 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0fc0038c0 con 0x7fd10c0cd020 2024-04-02T09:13:03.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.294+0000 7fd114800700 1 -- 172.21.15.89:0/225834974 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10c159da0 con 0x7fd10c0cd020 2024-04-02T09:13:03.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.295+0000 7fd114800700 1 -- 172.21.15.89:0/225834974 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd10c067720 con 0x7fd10c0cd020 2024-04-02T09:13:03.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.296+0000 7fd106000700 1 -- 172.21.15.89:0/225834974 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fd0fc003a20 con 0x7fd10c0cd020 2024-04-02T09:13:03.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.297+0000 7fd106000700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd0f006af60 0x7fd0f006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:03.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.297+0000 7fd111e00700 1 -- 172.21.15.89:0/225834974 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd0f006af60 msgr2=0x7fd0f006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:03.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.297+0000 7fd106000700 1 -- 172.21.15.89:0/225834974 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7fd0fc089180 con 0x7fd10c0cd020 2024-04-02T09:13:03.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.297+0000 7fd111e00700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd0f006af60 0x7fd0f006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:13:03.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.301+0000 7fd106000700 1 -- 172.21.15.89:0/225834974 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd0fc0539e0 con 0x7fd10c0cd020 2024-04-02T09:13:03.476 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:03.478+0000 7f590ab3a240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:13:03.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.498+0000 7fd111e00700 1 -- 172.21.15.89:0/225834974 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd0f006af60 msgr2=0x7fd0f006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:03.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.498+0000 7fd111e00700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd0f006af60 0x7fd0f006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:13:03.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.511+0000 7fd114800700 1 -- 172.21.15.89:0/225834974 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd10c15bda0 con 0x7fd10c0cd020 2024-04-02T09:13:03.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.511+0000 7fd106000700 1 -- 172.21.15.89:0/225834974 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v228) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd0fc057000 con 0x7fd10c0cd020 2024-04-02T09:13:03.510 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:13:03.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.514+0000 7fd114800700 1 -- 172.21.15.89:0/225834974 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd0f006af60 msgr2=0x7fd0f006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:13:03.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.514+0000 7fd114800700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd0f006af60 0x7fd0f006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.514+0000 7fd114800700 1 -- 172.21.15.89:0/225834974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd10c0cd020 msgr2=0x7fd10c15d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:03.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.514+0000 7fd114800700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd10c0cd020 0x7fd10c15d130 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc00c550 tx=0x7fd0fc00c860 comp rx=0 tx=0).stop 2024-04-02T09:13:03.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.514+0000 7fd114800700 1 -- 172.21.15.89:0/225834974 shutdown_connections 2024-04-02T09:13:03.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.514+0000 7fd114800700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd0f006af60 0x7fd0f006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.514+0000 7fd114800700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0cbb30 0x7fd10c072510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.514+0000 7fd114800700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd10c0cd4a0 0x7fd10c1f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.514+0000 7fd114800700 1 --2- 172.21.15.89:0/225834974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd10c0cd020 0x7fd10c15d130 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.514+0000 7fd114800700 1 -- 172.21.15.89:0/225834974 >> 172.21.15.89:0/225834974 conn(0x7fd10c0b0d70 msgr2=0x7fd10c06e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:03.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.514+0000 7fd114800700 1 -- 172.21.15.89:0/225834974 shutdown_connections 2024-04-02T09:13:03.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.514+0000 7fd114800700 1 -- 172.21.15.89:0/225834974 wait complete. 2024-04-02T09:13:03.535 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T09:13:03.624 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:03.626+0000 7fc13ea36240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:13:03.684 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:03.685+0000 7f590ab3a240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:13:03.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.736+0000 7f3e1fc00700 1 Processor -- start 2024-04-02T09:13:03.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.736+0000 7f3e1fc00700 1 -- start start 2024-04-02T09:13:03.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.737+0000 7f3e1fc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3e18160cc0 0x7f3e18168730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:03.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.737+0000 7f3e1fc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e1815f820 con 0x7f3e180ced80 2024-04-02T09:13:03.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.737+0000 7f3e1fc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e180d2450 con 0x7f3e1815dad0 2024-04-02T09:13:03.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.737+0000 7f3e1fc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e180cfcc0 con 0x7f3e18160cc0 2024-04-02T09:13:03.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.738+0000 7f3e17e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3e180ced80 0x7f3e180d1d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47696/0 (socket says 172.21.15.89:47696) 2024-04-02T09:13:03.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.738+0000 7f3e17e00700 1 -- 172.21.15.89:0/2669701533 learned_addr learned my addr 172.21.15.89:0/2669701533 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:03.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.738+0000 7f3e1dc00700 1 --2- 172.21.15.89:0/2669701533 >> v2:172.21.15.113:3300/0 conn(0x7f3e18160cc0 0x7f3e18168730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.738+0000 7f3e17400700 1 -- 172.21.15.89:0/2669701533 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248042390 0 0) 0x7f3e1815f820 con 0x7f3e180ced80 2024-04-02T09:13:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.738+0000 7f3e17400700 1 -- 172.21.15.89:0/2669701533 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e000035b0 con 0x7f3e180ced80 2024-04-02T09:13:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.738+0000 7f3e17400700 1 -- 172.21.15.89:0/2669701533 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135634437 0 0) 0x7f3e180d2450 con 0x7f3e1815dad0 2024-04-02T09:13:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.738+0000 7f3e17400700 1 -- 172.21.15.89:0/2669701533 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e1815f820 con 0x7f3e1815dad0 2024-04-02T09:13:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.738+0000 7f3e1dc00700 1 -- 172.21.15.89:0/2669701533 >> v1:172.21.15.113:6789/0 conn(0x7f3e1815dad0 legacy=0x7f3e180d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.738+0000 7f3e1dc00700 1 -- 172.21.15.89:0/2669701533 >> v1:172.21.15.89:6789/0 conn(0x7f3e180ced80 legacy=0x7f3e180d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.738+0000 7f3e1dc00700 1 -- 172.21.15.89:0/2669701533 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e180cf420 con 0x7f3e18160cc0 2024-04-02T09:13:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.739+0000 7f3e1dc00700 1 --2- 172.21.15.89:0/2669701533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e18160cc0 0x7f3e18168730 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3e10009d90 tx=0x7f3e1000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bc947d8a7f444e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.739+0000 7f3e17400700 1 -- 172.21.15.89:0/2669701533 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1000e070 con 0x7f3e18160cc0 2024-04-02T09:13:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.739+0000 7f3e17400700 1 -- 172.21.15.89:0/2669701533 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1000bb20 con 0x7f3e18160cc0 2024-04-02T09:13:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.739+0000 7f3e17400700 1 -- 172.21.15.89:0/2669701533 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e10012430 con 0x7f3e18160cc0 2024-04-02T09:13:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.740+0000 7f3e1fc00700 1 -- 172.21.15.89:0/2669701533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e18160cc0 msgr2=0x7f3e18168730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.740+0000 7f3e1fc00700 1 --2- 172.21.15.89:0/2669701533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e18160cc0 0x7f3e18168730 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3e10009d90 tx=0x7f3e1000b330 comp rx=0 tx=0).stop 2024-04-02T09:13:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.740+0000 7f3e1fc00700 1 -- 172.21.15.89:0/2669701533 shutdown_connections 2024-04-02T09:13:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.740+0000 7f3e1fc00700 1 --2- 172.21.15.89:0/2669701533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e18160cc0 0x7f3e18168730 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.740+0000 7f3e1fc00700 1 -- 172.21.15.89:0/2669701533 >> 172.21.15.89:0/2669701533 conn(0x7f3e1808aff0 msgr2=0x7f3e180d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.740+0000 7f3e1fc00700 1 -- 172.21.15.89:0/2669701533 shutdown_connections 2024-04-02T09:13:03.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.740+0000 7f3e1fc00700 1 -- 172.21.15.89:0/2669701533 wait complete. 2024-04-02T09:13:03.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.742+0000 7f3e1fc00700 1 Processor -- start 2024-04-02T09:13:03.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.742+0000 7f3e1fc00700 1 -- start start 2024-04-02T09:13:03.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.743+0000 7f3e1fc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e180ced80 0x7f3e18150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:03.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.743+0000 7f3e1fc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e1815dad0 0x7f3e18153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:03.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.743+0000 7f3e1fc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e18160cc0 0x7f3e18154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:03.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.743+0000 7f3e1fc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e18150f40 con 0x7f3e180ced80 2024-04-02T09:13:03.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.743+0000 7f3e1fc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e180c9860 con 0x7f3e1815dad0 2024-04-02T09:13:03.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.743+0000 7f3e1fc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e180c9b40 con 0x7f3e18160cc0 2024-04-02T09:13:03.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.744+0000 7f3e1d200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e180ced80 0x7f3e18150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:03.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.744+0000 7f3e1d200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e180ced80 0x7f3e18150a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52736/0 (socket says 172.21.15.89:52736) 2024-04-02T09:13:03.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.744+0000 7f3e1d200700 1 -- 172.21.15.89:0/3164117875 learned_addr learned my addr 172.21.15.89:0/3164117875 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.744+0000 7f3e17e00700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e1815dad0 0x7f3e18153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.744+0000 7f3e1dc00700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e18160cc0 0x7f3e18154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.744+0000 7f3e1d200700 1 -- 172.21.15.89:0/3164117875 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e18160cc0 msgr2=0x7f3e18154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.744+0000 7f3e1d200700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e18160cc0 0x7f3e18154250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.744+0000 7f3e1d200700 1 -- 172.21.15.89:0/3164117875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e1815dad0 msgr2=0x7f3e18153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.744+0000 7f3e1d200700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e1815dad0 0x7f3e18153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.744+0000 7f3e1d200700 1 -- 172.21.15.89:0/3164117875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e10009a70 con 0x7f3e180ced80 2024-04-02T09:13:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.744+0000 7f3e17e00700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e1815dad0 0x7f3e18153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.745+0000 7f3e1d200700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e180ced80 0x7f3e18150a00 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c00c520 tx=0x7f3e0c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.745+0000 7f3e14c00700 1 -- 172.21.15.89:0/3164117875 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c0052d0 con 0x7f3e180ced80 2024-04-02T09:13:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.745+0000 7f3e14c00700 1 -- 172.21.15.89:0/3164117875 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0c005430 con 0x7f3e180ced80 2024-04-02T09:13:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.745+0000 7f3e1fc00700 1 -- 172.21.15.89:0/3164117875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e180c9e20 con 0x7f3e180ced80 2024-04-02T09:13:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.745+0000 7f3e14c00700 1 -- 172.21.15.89:0/3164117875 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c0038c0 con 0x7f3e180ced80 2024-04-02T09:13:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.745+0000 7f3e1fc00700 1 -- 172.21.15.89:0/3164117875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e180b9110 con 0x7f3e180ced80 2024-04-02T09:13:03.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.746+0000 7f3e1fc00700 1 -- 172.21.15.89:0/3164117875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e180ca440 con 0x7f3e180ced80 2024-04-02T09:13:03.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.747+0000 7f3e14c00700 1 -- 172.21.15.89:0/3164117875 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f3e0c003a20 con 0x7f3e180ced80 2024-04-02T09:13:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.748+0000 7f3e14c00700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3df806af60 0x7f3df806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.748+0000 7f3e14c00700 1 -- 172.21.15.89:0/3164117875 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f3e0c089240 con 0x7f3e180ced80 2024-04-02T09:13:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.748+0000 7f3e17e00700 1 -- 172.21.15.89:0/3164117875 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3df806af60 msgr2=0x7f3df806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.748+0000 7f3e17e00700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3df806af60 0x7f3df806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:13:03.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.752+0000 7f3e14c00700 1 -- 172.21.15.89:0/3164117875 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3e0c053a10 con 0x7f3e180ced80 2024-04-02T09:13:03.865 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:03.867+0000 7fc13ea36240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:13:03.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.949+0000 7f3e17e00700 1 -- 172.21.15.89:0/3164117875 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3df806af60 msgr2=0x7f3df806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:03.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.949+0000 7f3e17e00700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3df806af60 0x7f3df806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:13:03.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.978+0000 7f3e1fc00700 1 -- 172.21.15.89:0/3164117875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3e18067720 con 0x7f3e180ced80 2024-04-02T09:13:03.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.979+0000 7f3e14c00700 1 -- 172.21.15.89:0/3164117875 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v228) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3e0c057030 con 0x7f3e180ced80 2024-04-02T09:13:03.977 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:13:03.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.981+0000 7f3e1fc00700 1 -- 172.21.15.89:0/3164117875 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3df806af60 msgr2=0x7f3df806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:13:03.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.981+0000 7f3e1fc00700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3df806af60 0x7f3df806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.981+0000 7f3e1fc00700 1 -- 172.21.15.89:0/3164117875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e180ced80 msgr2=0x7f3e18150a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:03.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.981+0000 7f3e1fc00700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e180ced80 0x7f3e18150a00 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c00c520 tx=0x7f3e0c00c830 comp rx=0 tx=0).stop 2024-04-02T09:13:03.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.981+0000 7f3e1fc00700 1 -- 172.21.15.89:0/3164117875 shutdown_connections 2024-04-02T09:13:03.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.981+0000 7f3e1fc00700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3e18160cc0 0x7f3e18154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.981+0000 7f3e1fc00700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e180ced80 0x7f3e18150a00 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.981+0000 7f3e1fc00700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3df806af60 0x7f3df806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.981+0000 7f3e1fc00700 1 --2- 172.21.15.89:0/3164117875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e1815dad0 0x7f3e18153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:03.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.981+0000 7f3e1fc00700 1 -- 172.21.15.89:0/3164117875 >> 172.21.15.89:0/3164117875 conn(0x7f3e1808aff0 msgr2=0x7f3e1806c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:03.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.981+0000 7f3e1fc00700 1 -- 172.21.15.89:0/3164117875 shutdown_connections 2024-04-02T09:13:03.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:03.982+0000 7f3e1fc00700 1 -- 172.21.15.89:0/3164117875 wait complete. 2024-04-02T09:13:03.998 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T09:13:04.083 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:04.085+0000 7fc13ea36240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:13:04.158 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:04.159+0000 7f590ab3a240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:13:04.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.199+0000 7f0941c00700 1 Processor -- start 2024-04-02T09:13:04.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.199+0000 7f0941c00700 1 -- start start 2024-04-02T09:13:04.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.200+0000 7f0941c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f093c15dad0 0x7f093c0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:04.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.200+0000 7f0941c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093c15f820 con 0x7f093c0d1630 2024-04-02T09:13:04.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.201+0000 7f0941c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093c0cf420 con 0x7f093c0d21a0 2024-04-02T09:13:04.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.201+0000 7f0941c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093c0cfcc0 con 0x7f093c15dad0 2024-04-02T09:13:04.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.201+0000 7f093a000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f093c0d1630 0x7f093c0d1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47700/0 (socket says 172.21.15.89:47700) 2024-04-02T09:13:04.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.201+0000 7f093a000700 1 -- 172.21.15.89:0/151742997 learned_addr learned my addr 172.21.15.89:0/151742997 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:04.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.201+0000 7f093aa00700 1 --2- 172.21.15.89:0/151742997 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f093c15dad0 0x7f093c0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:04.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.201+0000 7f0939600700 1 -- 172.21.15.89:0/151742997 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477106742 0 0) 0x7f093c15f820 con 0x7f093c0d1630 2024-04-02T09:13:04.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.201+0000 7f0939600700 1 -- 172.21.15.89:0/151742997 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09200035b0 con 0x7f093c0d1630 2024-04-02T09:13:04.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.201+0000 7f0939600700 1 -- 172.21.15.89:0/151742997 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421994322 0 0) 0x7f093c0cf420 con 0x7f093c0d21a0 2024-04-02T09:13:04.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.201+0000 7f0939600700 1 -- 172.21.15.89:0/151742997 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f093c15f820 con 0x7f093c0d21a0 2024-04-02T09:13:04.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.201+0000 7f093aa00700 1 -- 172.21.15.89:0/151742997 >> v1:172.21.15.113:6789/0 conn(0x7f093c0d21a0 legacy=0x7f093c15fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:04.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.201+0000 7f093aa00700 1 -- 172.21.15.89:0/151742997 >> v1:172.21.15.89:6789/0 conn(0x7f093c0d1630 legacy=0x7f093c0d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:04.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.201+0000 7f093aa00700 1 -- 172.21.15.89:0/151742997 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093c0cf700 con 0x7f093c15dad0 2024-04-02T09:13:04.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.202+0000 7f093aa00700 1 --2- 172.21.15.89:0/151742997 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f093c15dad0 0x7f093c0ced80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f092c009870 tx=0x7f092c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e383657c00ed855a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:04.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.202+0000 7f0939600700 1 -- 172.21.15.89:0/151742997 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c011830 con 0x7f093c15dad0 2024-04-02T09:13:04.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.202+0000 7f0939600700 1 -- 172.21.15.89:0/151742997 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092c011990 con 0x7f093c15dad0 2024-04-02T09:13:04.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.202+0000 7f0939600700 1 -- 172.21.15.89:0/151742997 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c011d00 con 0x7f093c15dad0 2024-04-02T09:13:04.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.202+0000 7f0941c00700 1 -- 172.21.15.89:0/151742997 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f093c15dad0 msgr2=0x7f093c0ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:04.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.202+0000 7f0941c00700 1 --2- 172.21.15.89:0/151742997 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f093c15dad0 0x7f093c0ced80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f092c009870 tx=0x7f092c009b80 comp rx=0 tx=0).stop 2024-04-02T09:13:04.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.202+0000 7f0941c00700 1 -- 172.21.15.89:0/151742997 shutdown_connections 2024-04-02T09:13:04.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.202+0000 7f0941c00700 1 --2- 172.21.15.89:0/151742997 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f093c15dad0 0x7f093c0ced80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.203+0000 7f0941c00700 1 -- 172.21.15.89:0/151742997 >> 172.21.15.89:0/151742997 conn(0x7f093c08aff0 msgr2=0x7f093c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:04.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.203+0000 7f0941c00700 1 -- 172.21.15.89:0/151742997 shutdown_connections 2024-04-02T09:13:04.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.203+0000 7f0941c00700 1 -- 172.21.15.89:0/151742997 wait complete. 2024-04-02T09:13:04.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.205+0000 7f0941c00700 1 Processor -- start 2024-04-02T09:13:04.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.205+0000 7f0941c00700 1 -- start start 2024-04-02T09:13:04.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.206+0000 7f0941c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f093c0d1630 0x7f093c1593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:04.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.206+0000 7f0941c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f093c0d21a0 0x7f093c0c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:04.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.206+0000 7f0941c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c15dad0 0x7f093c09d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:04.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.206+0000 7f0941c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c0c8fa0 con 0x7f093c0d21a0 2024-04-02T09:13:04.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.206+0000 7f0941c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c0b6dc0 con 0x7f093c15dad0 2024-04-02T09:13:04.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.206+0000 7f0941c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093c0c9220 con 0x7f093c0d1630 2024-04-02T09:13:04.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.207+0000 7f093a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f093c0d21a0 0x7f093c0c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:04.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.207+0000 7f093a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f093c0d21a0 0x7f093c0c8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52746/0 (socket says 172.21.15.89:52746) 2024-04-02T09:13:04.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.207+0000 7f093a000700 1 -- 172.21.15.89:0/3971968070 learned_addr learned my addr 172.21.15.89:0/3971968070 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:04.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.207+0000 7f093aa00700 1 --2- 172.21.15.89:0/3971968070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f093c0d1630 0x7f093c1593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:04.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.207+0000 7f093b400700 1 --2- 172.21.15.89:0/3971968070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c15dad0 0x7f093c09d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:04.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.207+0000 7f093a000700 1 -- 172.21.15.89:0/3971968070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f093c0d1630 msgr2=0x7f093c1593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:04.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.207+0000 7f093a000700 1 --2- 172.21.15.89:0/3971968070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f093c0d1630 0x7f093c1593a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.207+0000 7f093a000700 1 -- 172.21.15.89:0/3971968070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c15dad0 msgr2=0x7f093c09d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:04.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.207+0000 7f093a000700 1 --2- 172.21.15.89:0/3971968070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c15dad0 0x7f093c09d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.207+0000 7f093a000700 1 -- 172.21.15.89:0/3971968070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c0094a0 con 0x7f093c0d21a0 2024-04-02T09:13:04.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.207+0000 7f093a000700 1 --2- 172.21.15.89:0/3971968070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f093c0d21a0 0x7f093c0c8a60 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f092800c700 tx=0x7f092800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:04.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.207+0000 7f0932a00700 1 -- 172.21.15.89:0/3971968070 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928005d70 con 0x7f093c0d21a0 2024-04-02T09:13:04.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.208+0000 7f0932a00700 1 -- 172.21.15.89:0/3971968070 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09280070d0 con 0x7f093c0d21a0 2024-04-02T09:13:04.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.208+0000 7f0932a00700 1 -- 172.21.15.89:0/3971968070 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928007480 con 0x7f093c0d21a0 2024-04-02T09:13:04.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.208+0000 7f0941c00700 1 -- 172.21.15.89:0/3971968070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f093c0c9500 con 0x7f093c0d21a0 2024-04-02T09:13:04.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.208+0000 7f0941c00700 1 -- 172.21.15.89:0/3971968070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093c1599a0 con 0x7f093c0d21a0 2024-04-02T09:13:04.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.209+0000 7f0941c00700 1 -- 172.21.15.89:0/3971968070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093c0c9a20 con 0x7f093c0d21a0 2024-04-02T09:13:04.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.210+0000 7f0932a00700 1 -- 172.21.15.89:0/3971968070 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f09280075e0 con 0x7f093c0d21a0 2024-04-02T09:13:04.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.210+0000 7f0932a00700 1 --2- 172.21.15.89:0/3971968070 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f091c06af60 0x7f091c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:04.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.211+0000 7f0932a00700 1 -- 172.21.15.89:0/3971968070 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f0928089180 con 0x7f093c0d21a0 2024-04-02T09:13:04.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.211+0000 7f093aa00700 1 -- 172.21.15.89:0/3971968070 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f091c06af60 msgr2=0x7f091c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:04.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.211+0000 7f093aa00700 1 --2- 172.21.15.89:0/3971968070 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f091c06af60 0x7f091c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:13:04.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.215+0000 7f0932a00700 1 -- 172.21.15.89:0/3971968070 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0928053a20 con 0x7f093c0d21a0 2024-04-02T09:13:04.369 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:04.370+0000 7f590ab3a240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:13:04.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.412+0000 7f093aa00700 1 -- 172.21.15.89:0/3971968070 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f091c06af60 msgr2=0x7f091c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:04.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.412+0000 7f093aa00700 1 --2- 172.21.15.89:0/3971968070 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f091c06af60 0x7f091c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:13:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.428+0000 7f0941c00700 1 -- 172.21.15.89:0/3971968070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f093c067720 con 0x7f093c0d21a0 2024-04-02T09:13:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.429+0000 7f0932a00700 1 -- 172.21.15.89:0/3971968070 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v228) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0928017020 con 0x7f093c0d21a0 2024-04-02T09:13:04.427 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:13:04.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.431+0000 7f0941c00700 1 -- 172.21.15.89:0/3971968070 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f091c06af60 msgr2=0x7f091c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:13:04.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.431+0000 7f0941c00700 1 --2- 172.21.15.89:0/3971968070 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f091c06af60 0x7f091c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.431+0000 7f0941c00700 1 -- 172.21.15.89:0/3971968070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f093c0d21a0 msgr2=0x7f093c0c8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:04.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.431+0000 7f0941c00700 1 --2- 172.21.15.89:0/3971968070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f093c0d21a0 0x7f093c0c8a60 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f092800c700 tx=0x7f092800ca10 comp rx=0 tx=0).stop 2024-04-02T09:13:04.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.431+0000 7f0941c00700 1 -- 172.21.15.89:0/3971968070 shutdown_connections 2024-04-02T09:13:04.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.431+0000 7f0941c00700 1 --2- 172.21.15.89:0/3971968070 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f091c06af60 0x7f091c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.431+0000 7f0941c00700 1 --2- 172.21.15.89:0/3971968070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c15dad0 0x7f093c09d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.431+0000 7f0941c00700 1 --2- 172.21.15.89:0/3971968070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f093c0d1630 0x7f093c1593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.431+0000 7f0941c00700 1 --2- 172.21.15.89:0/3971968070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f093c0d21a0 0x7f093c0c8a60 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.431+0000 7f0941c00700 1 -- 172.21.15.89:0/3971968070 >> 172.21.15.89:0/3971968070 conn(0x7f093c08aff0 msgr2=0x7f093c06c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:04.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.432+0000 7f0941c00700 1 -- 172.21.15.89:0/3971968070 shutdown_connections 2024-04-02T09:13:04.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.432+0000 7f0941c00700 1 -- 172.21.15.89:0/3971968070 wait complete. 2024-04-02T09:13:04.448 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-02T09:13:04.593 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:04.594+0000 7fc13ea36240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:13:04.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.647+0000 7faea8200700 1 Processor -- start 2024-04-02T09:13:04.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.647+0000 7faea8200700 1 -- start start 2024-04-02T09:13:04.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.648+0000 7faea8200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faea00cbb30 0x7faea00cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:04.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.648+0000 7faea8200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7faea00cd020 0x7faea00cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:04.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.648+0000 7faea8200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faea00b2070 0x7faea00b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:04.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.648+0000 7faea8200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea00cc450 con 0x7faea00b2070 2024-04-02T09:13:04.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.648+0000 7faea8200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea00cdae0 con 0x7faea00cd020 2024-04-02T09:13:04.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.648+0000 7faea8200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea00cddc0 con 0x7faea00cbb30 2024-04-02T09:13:04.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.649+0000 7faea6200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faea00b2070 0x7faea00b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:04.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.649+0000 7faea6200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faea00b2070 0x7faea00b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52762/0 (socket says 172.21.15.89:52762) 2024-04-02T09:13:04.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.649+0000 7faea6200700 1 -- 172.21.15.89:0/1519546777 learned_addr learned my addr 172.21.15.89:0/1519546777 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:04.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.649+0000 7faea5800700 1 --2- 172.21.15.89:0/1519546777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faea00cbb30 0x7faea00cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:04.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.649+0000 7faea4e00700 1 --2- 172.21.15.89:0/1519546777 >> v2:172.21.15.113:3300/0 conn(0x7faea00cd020 0x7faea00cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:04.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.649+0000 7faea6200700 1 -- 172.21.15.89:0/1519546777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faea00cbb30 msgr2=0x7faea00cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:04.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.649+0000 7faea6200700 1 --2- 172.21.15.89:0/1519546777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faea00cbb30 0x7faea00cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.649+0000 7faea6200700 1 -- 172.21.15.89:0/1519546777 >> v2:172.21.15.113:3300/0 conn(0x7faea00cd020 msgr2=0x7faea00cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:04.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.649+0000 7faea6200700 1 --2- 172.21.15.89:0/1519546777 >> v2:172.21.15.113:3300/0 conn(0x7faea00cd020 0x7faea00cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.649+0000 7faea6200700 1 -- 172.21.15.89:0/1519546777 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea00ce0a0 con 0x7faea00b2070 2024-04-02T09:13:04.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.649+0000 7faea4e00700 1 --2- 172.21.15.89:0/1519546777 >> v2:172.21.15.113:3300/0 conn(0x7faea00cd020 0x7faea00cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:04.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.650+0000 7faea6200700 1 --2- 172.21.15.89:0/1519546777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faea00b2070 0x7faea00b2500 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7faea006fdb0 tx=0x7fae9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37170fb3b9b461d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:04.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.650+0000 7fae9fe00700 1 -- 172.21.15.89:0/1519546777 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9800e070 con 0x7faea00b2070 2024-04-02T09:13:04.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.650+0000 7fae9fe00700 1 -- 172.21.15.89:0/1519546777 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9800b910 con 0x7faea00b2070 2024-04-02T09:13:04.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.650+0000 7fae9fe00700 1 -- 172.21.15.89:0/1519546777 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9800bcc0 con 0x7faea00b2070 2024-04-02T09:13:04.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.651+0000 7faea8200700 1 -- 172.21.15.89:0/1519546777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faea00b2070 msgr2=0x7faea00b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:04.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.651+0000 7faea8200700 1 --2- 172.21.15.89:0/1519546777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faea00b2070 0x7faea00b2500 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7faea006fdb0 tx=0x7fae9800b330 comp rx=0 tx=0).stop 2024-04-02T09:13:04.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.651+0000 7faea8200700 1 -- 172.21.15.89:0/1519546777 shutdown_connections 2024-04-02T09:13:04.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.651+0000 7faea8200700 1 --2- 172.21.15.89:0/1519546777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faea00b2070 0x7faea00b2500 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.651+0000 7faea8200700 1 --2- 172.21.15.89:0/1519546777 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faea00cbb30 0x7faea00cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.651+0000 7faea8200700 1 --2- 172.21.15.89:0/1519546777 >> v2:172.21.15.113:3300/0 conn(0x7faea00cd020 0x7faea00cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.651+0000 7faea8200700 1 -- 172.21.15.89:0/1519546777 >> 172.21.15.89:0/1519546777 conn(0x7faea00b0d70 msgr2=0x7faea00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:04.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.651+0000 7faea8200700 1 -- 172.21.15.89:0/1519546777 shutdown_connections 2024-04-02T09:13:04.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.651+0000 7faea8200700 1 -- 172.21.15.89:0/1519546777 wait complete. 2024-04-02T09:13:04.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.653+0000 7faea8200700 1 Processor -- start 2024-04-02T09:13:04.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.653+0000 7faea8200700 1 -- start start 2024-04-02T09:13:04.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea8200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faea00b2070 0x7faea00cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:04.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea8200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faea00cbb30 0x7faea01f6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:04.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea8200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faea00cd020 0x7faea00adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:04.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea8200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea00caa40 con 0x7faea00cbb30 2024-04-02T09:13:04.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea8200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea0155b80 con 0x7faea00b2070 2024-04-02T09:13:04.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea8200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea00c9b90 con 0x7faea00cd020 2024-04-02T09:13:04.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea4e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faea00cbb30 0x7faea01f6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:04.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea4e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faea00cbb30 0x7faea01f6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52774/0 (socket says 172.21.15.89:52774) 2024-04-02T09:13:04.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea4e00700 1 -- 172.21.15.89:0/435166343 learned_addr learned my addr 172.21.15.89:0/435166343 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:04.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea6200700 1 --2- 172.21.15.89:0/435166343 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faea00cd020 0x7faea00adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:04.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea5800700 1 --2- 172.21.15.89:0/435166343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faea00b2070 0x7faea00cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:04.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea4e00700 1 -- 172.21.15.89:0/435166343 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faea00cd020 msgr2=0x7faea00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:04.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea4e00700 1 --2- 172.21.15.89:0/435166343 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faea00cd020 0x7faea00adfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea4e00700 1 -- 172.21.15.89:0/435166343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faea00b2070 msgr2=0x7faea00cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:04.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea4e00700 1 --2- 172.21.15.89:0/435166343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faea00b2070 0x7faea00cae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.655+0000 7faea4e00700 1 -- 172.21.15.89:0/435166343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c0094a0 con 0x7faea00cbb30 2024-04-02T09:13:04.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.656+0000 7faea4e00700 1 --2- 172.21.15.89:0/435166343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faea00cbb30 0x7faea01f6280 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00ec10 tx=0x7fae8c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:04.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.656+0000 7fae9d600700 1 -- 172.21.15.89:0/435166343 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c00c850 con 0x7faea00cbb30 2024-04-02T09:13:04.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.656+0000 7fae9d600700 1 -- 172.21.15.89:0/435166343 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c00c9b0 con 0x7faea00cbb30 2024-04-02T09:13:04.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.656+0000 7faea8200700 1 -- 172.21.15.89:0/435166343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae98009b30 con 0x7faea00cbb30 2024-04-02T09:13:04.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.656+0000 7fae9d600700 1 -- 172.21.15.89:0/435166343 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c00cd20 con 0x7faea00cbb30 2024-04-02T09:13:04.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.656+0000 7faea8200700 1 -- 172.21.15.89:0/435166343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea00ca0a0 con 0x7faea00cbb30 2024-04-02T09:13:04.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.657+0000 7faea8200700 1 -- 172.21.15.89:0/435166343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea0067720 con 0x7faea00cbb30 2024-04-02T09:13:04.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.658+0000 7fae9d600700 1 -- 172.21.15.89:0/435166343 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fae8c013470 con 0x7faea00cbb30 2024-04-02T09:13:04.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.659+0000 7fae9d600700 1 --2- 172.21.15.89:0/435166343 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fae8006af60 0x7fae8006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:04.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.659+0000 7fae9d600700 1 -- 172.21.15.89:0/435166343 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7fae8c089e90 con 0x7faea00cbb30 2024-04-02T09:13:04.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.659+0000 7faea5800700 1 -- 172.21.15.89:0/435166343 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fae8006af60 msgr2=0x7fae8006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:04.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.659+0000 7faea5800700 1 --2- 172.21.15.89:0/435166343 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fae8006af60 0x7fae8006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:13:04.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.664+0000 7fae9d600700 1 -- 172.21.15.89:0/435166343 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae8c054660 con 0x7faea00cbb30 2024-04-02T09:13:04.808 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:04.809+0000 7f590ab3a240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:13:04.815 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:04.817+0000 7fc13ea36240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:13:04.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.860+0000 7faea5800700 1 -- 172.21.15.89:0/435166343 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fae8006af60 msgr2=0x7fae8006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:04.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.860+0000 7faea5800700 1 --2- 172.21.15.89:0/435166343 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fae8006af60 0x7fae8006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:13:04.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.876+0000 7faea8200700 1 -- 172.21.15.89:0/435166343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faea00cb630 con 0x7faea00cbb30 2024-04-02T09:13:04.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.877+0000 7fae9d600700 1 -- 172.21.15.89:0/435166343 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v228) v1 ==== 107+0+11 (secure 0 0 0) 0x7fae8c0210b0 con 0x7faea00cbb30 2024-04-02T09:13:04.875 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:13:04.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.879+0000 7faea8200700 1 -- 172.21.15.89:0/435166343 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fae8006af60 msgr2=0x7fae8006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:13:04.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.880+0000 7faea8200700 1 --2- 172.21.15.89:0/435166343 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fae8006af60 0x7fae8006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.880+0000 7faea8200700 1 -- 172.21.15.89:0/435166343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faea00cbb30 msgr2=0x7faea01f6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:04.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.880+0000 7faea8200700 1 --2- 172.21.15.89:0/435166343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faea00cbb30 0x7faea01f6280 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00ec10 tx=0x7fae8c00ef20 comp rx=0 tx=0).stop 2024-04-02T09:13:04.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.880+0000 7faea8200700 1 -- 172.21.15.89:0/435166343 shutdown_connections 2024-04-02T09:13:04.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.880+0000 7faea8200700 1 --2- 172.21.15.89:0/435166343 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fae8006af60 0x7fae8006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.880+0000 7faea8200700 1 --2- 172.21.15.89:0/435166343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faea00b2070 0x7faea00cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.880+0000 7faea8200700 1 --2- 172.21.15.89:0/435166343 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faea00cd020 0x7faea00adfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.880+0000 7faea8200700 1 --2- 172.21.15.89:0/435166343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faea00cbb30 0x7faea01f6280 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:04.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.880+0000 7faea8200700 1 -- 172.21.15.89:0/435166343 >> 172.21.15.89:0/435166343 conn(0x7faea00b0d70 msgr2=0x7faea006e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:04.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.880+0000 7faea8200700 1 -- 172.21.15.89:0/435166343 shutdown_connections 2024-04-02T09:13:04.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:04.880+0000 7faea8200700 1 -- 172.21.15.89:0/435166343 wait complete. 2024-04-02T09:13:04.897 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-02T09:13:04.919 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:04.920+0000 7f590ab3a240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:13:05.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.102+0000 7f7892000700 1 Processor -- start 2024-04-02T09:13:05.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.102+0000 7f7892000700 1 -- start start 2024-04-02T09:13:05.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.103+0000 7f7892000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f788c15d9d0 0x7f788c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:05.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.103+0000 7f7892000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f788c0d16a0 0x7f788c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:05.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.103+0000 7f7892000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f788c0d2370 0x7f788c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:05.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.103+0000 7f7892000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788c0d2820 con 0x7f788c15d9d0 2024-04-02T09:13:05.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.103+0000 7f7892000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788c161420 con 0x7f788c0d16a0 2024-04-02T09:13:05.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.103+0000 7f7892000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788c161700 con 0x7f788c0d2370 2024-04-02T09:13:05.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.103+0000 7f788b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f788c15d9d0 0x7f788c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:05.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.103+0000 7f788b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f788c15d9d0 0x7f788c0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52780/0 (socket says 172.21.15.89:52780) 2024-04-02T09:13:05.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.103+0000 7f788b400700 1 -- 172.21.15.89:0/197225299 learned_addr learned my addr 172.21.15.89:0/197225299 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:05.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.103+0000 7f788aa00700 1 --2- 172.21.15.89:0/197225299 >> v2:172.21.15.113:3300/0 conn(0x7f788c0d16a0 0x7f788c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:05.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.103+0000 7f788be00700 1 --2- 172.21.15.89:0/197225299 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f788c0d2370 0x7f788c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:05.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.104+0000 7f788b400700 1 -- 172.21.15.89:0/197225299 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f788c0d2370 msgr2=0x7f788c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:05.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.104+0000 7f788b400700 1 --2- 172.21.15.89:0/197225299 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f788c0d2370 0x7f788c15ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.104+0000 7f788b400700 1 -- 172.21.15.89:0/197225299 >> v2:172.21.15.113:3300/0 conn(0x7f788c0d16a0 msgr2=0x7f788c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:05.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.104+0000 7f788b400700 1 --2- 172.21.15.89:0/197225299 >> v2:172.21.15.113:3300/0 conn(0x7f788c0d16a0 0x7f788c0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.104+0000 7f788b400700 1 -- 172.21.15.89:0/197225299 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c1619e0 con 0x7f788c15d9d0 2024-04-02T09:13:05.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.104+0000 7f788b400700 1 --2- 172.21.15.89:0/197225299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f788c15d9d0 0x7f788c0d3c30 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f788c0d12f0 tx=0x7f787c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e78734871f4378 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:05.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.104+0000 7f788a000700 1 -- 172.21.15.89:0/197225299 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c00b870 con 0x7f788c15d9d0 2024-04-02T09:13:05.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.104+0000 7f788a000700 1 -- 172.21.15.89:0/197225299 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787c00b9d0 con 0x7f788c15d9d0 2024-04-02T09:13:05.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.104+0000 7f788a000700 1 -- 172.21.15.89:0/197225299 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c00bd60 con 0x7f788c15d9d0 2024-04-02T09:13:05.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.105+0000 7f7892000700 1 -- 172.21.15.89:0/197225299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f788c15d9d0 msgr2=0x7f788c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:05.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.105+0000 7f7892000700 1 --2- 172.21.15.89:0/197225299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f788c15d9d0 0x7f788c0d3c30 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f788c0d12f0 tx=0x7f787c009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.105+0000 7f7892000700 1 -- 172.21.15.89:0/197225299 shutdown_connections 2024-04-02T09:13:05.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.105+0000 7f7892000700 1 --2- 172.21.15.89:0/197225299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f788c15d9d0 0x7f788c0d3c30 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.105+0000 7f7892000700 1 --2- 172.21.15.89:0/197225299 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f788c0d2370 0x7f788c15ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.105+0000 7f7892000700 1 --2- 172.21.15.89:0/197225299 >> v2:172.21.15.113:3300/0 conn(0x7f788c0d16a0 0x7f788c0d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.105+0000 7f7892000700 1 -- 172.21.15.89:0/197225299 >> 172.21.15.89:0/197225299 conn(0x7f788c08aff0 msgr2=0x7f788c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:05.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.105+0000 7f7892000700 1 -- 172.21.15.89:0/197225299 shutdown_connections 2024-04-02T09:13:05.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.105+0000 7f7892000700 1 -- 172.21.15.89:0/197225299 wait complete. 2024-04-02T09:13:05.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.108+0000 7f7892000700 1 Processor -- start 2024-04-02T09:13:05.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.108+0000 7f7892000700 1 -- start start 2024-04-02T09:13:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.109+0000 7f7892000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f788c0d16a0 0x7f788c0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.109+0000 7f7892000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f788c0d2370 0x7f788c14a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.109+0000 7f7892000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f788c15d9d0 0x7f788c14ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.109+0000 7f7892000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c0cac80 con 0x7f788c0d16a0 2024-04-02T09:13:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.109+0000 7f7892000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c0cd390 con 0x7f788c15d9d0 2024-04-02T09:13:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.109+0000 7f7892000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788c0cd670 con 0x7f788c0d2370 2024-04-02T09:13:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.109+0000 7f788b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f788c0d16a0 0x7f788c0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.109+0000 7f788b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f788c0d16a0 0x7f788c0ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52796/0 (socket says 172.21.15.89:52796) 2024-04-02T09:13:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.109+0000 7f788b400700 1 -- 172.21.15.89:0/791018487 learned_addr learned my addr 172.21.15.89:0/791018487 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.110+0000 7f788be00700 1 --2- 172.21.15.89:0/791018487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f788c15d9d0 0x7f788c14ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:05.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.110+0000 7f788aa00700 1 --2- 172.21.15.89:0/791018487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f788c0d2370 0x7f788c14a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:05.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.110+0000 7f788b400700 1 -- 172.21.15.89:0/791018487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f788c0d2370 msgr2=0x7f788c14a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:05.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.110+0000 7f788b400700 1 --2- 172.21.15.89:0/791018487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f788c0d2370 0x7f788c14a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.110+0000 7f788b400700 1 -- 172.21.15.89:0/791018487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f788c15d9d0 msgr2=0x7f788c14ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:05.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.110+0000 7f788b400700 1 --2- 172.21.15.89:0/791018487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f788c15d9d0 0x7f788c14ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.110+0000 7f788b400700 1 -- 172.21.15.89:0/791018487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787c009550 con 0x7f788c0d16a0 2024-04-02T09:13:05.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.110+0000 7f788b400700 1 --2- 172.21.15.89:0/791018487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f788c0d16a0 0x7f788c0ca5f0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f787c000c00 tx=0x7f787c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:05.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.110+0000 7f7883400700 1 -- 172.21.15.89:0/791018487 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c004520 con 0x7f788c0d16a0 2024-04-02T09:13:05.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.110+0000 7f7883400700 1 -- 172.21.15.89:0/791018487 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787c01e070 con 0x7f788c0d16a0 2024-04-02T09:13:05.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.110+0000 7f7883400700 1 -- 172.21.15.89:0/791018487 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c0038c0 con 0x7f788c0d16a0 2024-04-02T09:13:05.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.110+0000 7f7892000700 1 -- 172.21.15.89:0/791018487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788c0cd950 con 0x7f788c0d16a0 2024-04-02T09:13:05.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.111+0000 7f7892000700 1 -- 172.21.15.89:0/791018487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788c14a7f0 con 0x7f788c0d16a0 2024-04-02T09:13:05.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.112+0000 7f7892000700 1 -- 172.21.15.89:0/791018487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788c0d1ca0 con 0x7f788c0d16a0 2024-04-02T09:13:05.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.113+0000 7f7883400700 1 -- 172.21.15.89:0/791018487 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 153) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f787c003a20 con 0x7f788c0d16a0 2024-04-02T09:13:05.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.113+0000 7f7883400700 1 --2- 172.21.15.89:0/791018487 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f786c06af60 0x7f786c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:05.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.114+0000 7f7883400700 1 -- 172.21.15.89:0/791018487 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f787c020070 con 0x7f788c0d16a0 2024-04-02T09:13:05.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.114+0000 7f788aa00700 1 -- 172.21.15.89:0/791018487 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f786c06af60 msgr2=0x7f786c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:05.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.114+0000 7f788aa00700 1 --2- 172.21.15.89:0/791018487 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f786c06af60 0x7f786c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:13:05.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.118+0000 7f7883400700 1 -- 172.21.15.89:0/791018487 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f787c055c20 con 0x7f788c0d16a0 2024-04-02T09:13:05.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.255+0000 7f7883400700 1 -- 172.21.15.89:0/791018487 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 154) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f787c003cd0 con 0x7f788c0d16a0 2024-04-02T09:13:05.282 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.283+0000 7fc13ea36240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:13:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.315+0000 7f788aa00700 1 -- 172.21.15.89:0/791018487 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f786c06af60 msgr2=0x7f786c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.315+0000 7f788aa00700 1 --2- 172.21.15.89:0/791018487 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f786c06af60 0x7f786c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:13:05.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.334+0000 7f7892000700 1 -- 172.21.15.89:0/791018487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f788c0b7d60 con 0x7f788c0d16a0 2024-04-02T09:13:05.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.335+0000 7f7883400700 1 -- 172.21.15.89:0/791018487 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v228) v1 ==== 103+0+11 (secure 0 0 0) 0x7f787c0150a0 con 0x7f788c0d16a0 2024-04-02T09:13:05.333 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:13:05.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.337+0000 7f7892000700 1 -- 172.21.15.89:0/791018487 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f786c06af60 msgr2=0x7f786c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:13:05.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.337+0000 7f7892000700 1 --2- 172.21.15.89:0/791018487 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f786c06af60 0x7f786c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.337+0000 7f7892000700 1 -- 172.21.15.89:0/791018487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f788c0d16a0 msgr2=0x7f788c0ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:05.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.337+0000 7f7892000700 1 --2- 172.21.15.89:0/791018487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f788c0d16a0 0x7f788c0ca5f0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f787c000c00 tx=0x7f787c004750 comp rx=0 tx=0).stop 2024-04-02T09:13:05.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.337+0000 7f7892000700 1 -- 172.21.15.89:0/791018487 shutdown_connections 2024-04-02T09:13:05.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.337+0000 7f7892000700 1 --2- 172.21.15.89:0/791018487 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f786c06af60 0x7f786c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.337+0000 7f7892000700 1 --2- 172.21.15.89:0/791018487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f788c15d9d0 0x7f788c14ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.337+0000 7f7892000700 1 --2- 172.21.15.89:0/791018487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f788c0d16a0 0x7f788c0ca5f0 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.337+0000 7f7892000700 1 --2- 172.21.15.89:0/791018487 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f788c0d2370 0x7f788c14a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.337+0000 7f7892000700 1 -- 172.21.15.89:0/791018487 >> 172.21.15.89:0/791018487 conn(0x7f788c08aff0 msgr2=0x7f788c0ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:05.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.337+0000 7f7892000700 1 -- 172.21.15.89:0/791018487 shutdown_connections 2024-04-02T09:13:05.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.337+0000 7f7892000700 1 -- 172.21.15.89:0/791018487 wait complete. 2024-04-02T09:13:05.354 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:13:05.400 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.402+0000 7fc13ea36240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:13:05.491 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.492+0000 7fc0ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.491 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.492+0000 7fc0ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.491 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.492+0000 7fc0ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.491 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.492+0000 7fc0ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.491 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.492+0000 7fc0ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.491 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.493+0000 7fc0b1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.491 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.493+0000 7fc0b1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.491 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.493+0000 7fc0b1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.491 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.493+0000 7fc0b1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.491 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.493+0000 7fc0b1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.505 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.507+0000 7fc0b1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.505 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.507+0000 7fc0b1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.505 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.507+0000 7fc0b1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.506 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.507+0000 7fc0b1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.506 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.507+0000 7fc0b1e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.506 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.508+0000 7fc0ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.506 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.508+0000 7fc0ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.507 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.508+0000 7fc0ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.507 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.508+0000 7fc0ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.507 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:05.508+0000 7fc0ae200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:13:05.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.573+0000 7fbdd9c00700 1 Processor -- start 2024-04-02T09:13:05.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.573+0000 7fbdd9c00700 1 -- start start 2024-04-02T09:13:05.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.574+0000 7fbdd9c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbdd4160f10 0x7fbdd40d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:05.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd9c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbdd4161590 0x7fbdd4161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:05.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd9c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd40ceda0 con 0x7fbdd40cf230 2024-04-02T09:13:05.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd9c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd4161f80 con 0x7fbdd4160f10 2024-04-02T09:13:05.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd9c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd41612f0 con 0x7fbdd4161590 2024-04-02T09:13:05.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd3400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbdd4161590 0x7fbdd4161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:05.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd2a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbdd4160f10 0x7fbdd40d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:05.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd2a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbdd4160f10 0x7fbdd40d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:54222/0 (socket says 172.21.15.89:54222) 2024-04-02T09:13:05.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd2a00700 1 -- 172.21.15.89:0/2837813901 learned_addr learned my addr 172.21.15.89:0/2837813901 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:05.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd3400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbdd4161590 0x7fbdd4161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:48334/0 (socket says 172.21.15.89:48334) 2024-04-02T09:13:05.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd2000700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fbdd40cf230 0x7fbdd40d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:40118/0 (socket says 172.21.15.89:40118) 2024-04-02T09:13:05.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd2a00700 1 -- 172.21.15.89:0/2837813901 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbdd4161590 msgr2=0x7fbdd4161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:05.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd2a00700 1 --2- 172.21.15.89:0/2837813901 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbdd4161590 0x7fbdd4161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd2a00700 1 -- 172.21.15.89:0/2837813901 >> v1:172.21.15.113:6789/0 conn(0x7fbdd40cf230 legacy=0x7fbdd40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:05.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd2a00700 1 -- 172.21.15.89:0/2837813901 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd40ce470 con 0x7fbdd4160f10 2024-04-02T09:13:05.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.575+0000 7fbdd1600700 1 -- 172.21.15.89:0/2837813901 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182100144 0 0) 0x7fbdd40ceda0 con 0x7fbdd40cf230 2024-04-02T09:13:05.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.576+0000 7fbdd2a00700 1 --2- 172.21.15.89:0/2837813901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdd4160f10 0x7fbdd40d18b0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40098f0 tx=0x7fbdc4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d3c5050d08b173e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:05.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.576+0000 7fbdd1600700 1 -- 172.21.15.89:0/2837813901 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4011830 con 0x7fbdd4160f10 2024-04-02T09:13:05.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.576+0000 7fbdd1600700 1 -- 172.21.15.89:0/2837813901 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc4011990 con 0x7fbdd4160f10 2024-04-02T09:13:05.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.576+0000 7fbdd1600700 1 -- 172.21.15.89:0/2837813901 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4011d20 con 0x7fbdd4160f10 2024-04-02T09:13:05.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.576+0000 7fbdd9c00700 1 -- 172.21.15.89:0/2837813901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdd4160f10 msgr2=0x7fbdd40d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:05.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.576+0000 7fbdd9c00700 1 --2- 172.21.15.89:0/2837813901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdd4160f10 0x7fbdd40d18b0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40098f0 tx=0x7fbdc4009c00 comp rx=0 tx=0).stop 2024-04-02T09:13:05.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.576+0000 7fbdd9c00700 1 -- 172.21.15.89:0/2837813901 shutdown_connections 2024-04-02T09:13:05.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.576+0000 7fbdd9c00700 1 --2- 172.21.15.89:0/2837813901 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbdd4161590 0x7fbdd4161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.576+0000 7fbdd9c00700 1 --2- 172.21.15.89:0/2837813901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdd4160f10 0x7fbdd40d18b0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.576+0000 7fbdd9c00700 1 -- 172.21.15.89:0/2837813901 >> 172.21.15.89:0/2837813901 conn(0x7fbdd408aff0 msgr2=0x7fbdd40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:05.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.577+0000 7fbdd9c00700 1 -- 172.21.15.89:0/2837813901 shutdown_connections 2024-04-02T09:13:05.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.577+0000 7fbdd9c00700 1 -- 172.21.15.89:0/2837813901 wait complete. 2024-04-02T09:13:05.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.578+0000 7fbdd9c00700 1 Processor -- start 2024-04-02T09:13:05.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.578+0000 7fbdd9c00700 1 -- start start 2024-04-02T09:13:05.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.579+0000 7fbdd9c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdd40cf230 0x7fbdd41589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:05.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.579+0000 7fbdd9c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdd4160f10 0x7fbdd4157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:05.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.579+0000 7fbdd9c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbdd4161590 0x7fbdd415e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:05.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.579+0000 7fbdd9c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd41582c0 con 0x7fbdd4160f10 2024-04-02T09:13:05.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.579+0000 7fbdd9c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd415f2b0 con 0x7fbdd40cf230 2024-04-02T09:13:05.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.579+0000 7fbdd9c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd415f590 con 0x7fbdd4161590 2024-04-02T09:13:05.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.580+0000 7fbdd2a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdd40cf230 0x7fbdd41589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:05.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.580+0000 7fbdd2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdd4160f10 0x7fbdd4157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:05.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.580+0000 7fbdd2a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdd40cf230 0x7fbdd41589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:54238/0 (socket says 172.21.15.89:54238) 2024-04-02T09:13:05.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.580+0000 7fbdd2a00700 1 -- 172.21.15.89:0/3089361180 learned_addr learned my addr 172.21.15.89:0/3089361180 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:05.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.580+0000 7fbdd3400700 1 --2- 172.21.15.89:0/3089361180 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbdd4161590 0x7fbdd415e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:05.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.580+0000 7fbdd2a00700 1 -- 172.21.15.89:0/3089361180 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbdd4161590 msgr2=0x7fbdd415e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:05.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.580+0000 7fbdd2a00700 1 --2- 172.21.15.89:0/3089361180 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbdd4161590 0x7fbdd415e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.580+0000 7fbdd2a00700 1 -- 172.21.15.89:0/3089361180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdd4160f10 msgr2=0x7fbdd4157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:05.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.580+0000 7fbdd2a00700 1 --2- 172.21.15.89:0/3089361180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdd4160f10 0x7fbdd4157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.580+0000 7fbdd2a00700 1 -- 172.21.15.89:0/3089361180 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc4009550 con 0x7fbdd40cf230 2024-04-02T09:13:05.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.580+0000 7fbdd2a00700 1 --2- 172.21.15.89:0/3089361180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdd40cf230 0x7fbdd41589b0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4009c40 tx=0x7fbdc4004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:05.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.580+0000 7fbdcaa00700 1 -- 172.21.15.89:0/3089361180 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4004510 con 0x7fbdd40cf230 2024-04-02T09:13:05.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.581+0000 7fbdcaa00700 1 -- 172.21.15.89:0/3089361180 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc401e070 con 0x7fbdd40cf230 2024-04-02T09:13:05.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.581+0000 7fbdd9c00700 1 -- 172.21.15.89:0/3089361180 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd415f870 con 0x7fbdd40cf230 2024-04-02T09:13:05.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.581+0000 7fbdcaa00700 1 -- 172.21.15.89:0/3089361180 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40038c0 con 0x7fbdd40cf230 2024-04-02T09:13:05.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.581+0000 7fbdd9c00700 1 -- 172.21.15.89:0/3089361180 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd41e70b0 con 0x7fbdd40cf230 2024-04-02T09:13:05.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.582+0000 7fbdd9c00700 1 -- 172.21.15.89:0/3089361180 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd4067720 con 0x7fbdd40cf230 2024-04-02T09:13:05.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.583+0000 7fbdcaa00700 1 -- 172.21.15.89:0/3089361180 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 155) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fbdc4003a20 con 0x7fbdd40cf230 2024-04-02T09:13:05.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.583+0000 7fbdcaa00700 1 -- 172.21.15.89:0/3089361180 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7752+0+0 (secure 0 0 0) 0x7fbdc408a500 con 0x7fbdd40cf230 2024-04-02T09:13:05.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.588+0000 7fbdcaa00700 1 -- 172.21.15.89:0/3089361180 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbdc40551c0 con 0x7fbdd40cf230 2024-04-02T09:13:05.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.831+0000 7fbdd9c00700 1 -- 172.21.15.89:0/3089361180 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdd41e72a0 con 0x7fbdd40cf230 2024-04-02T09:13:05.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.832+0000 7fbdcaa00700 1 -- 172.21.15.89:0/3089361180 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5971 (secure 0 0 0) 0x7fbdc4020090 con 0x7fbdd40cf230 2024-04-02T09:13:05.831 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:05.831 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":110,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8969,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3650347444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3650347444},{"type":"v1","addr":"172.21.15.89:6836","nonce":3650347444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2152500685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2152500685},{"type":"v1","addr":"172.21.15.113:6834","nonce":2152500685}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:25.170849+0000","modified":"2024-04-02T09:12:28.185307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11157":{"gid":11157,"name":"a","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2615927215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2615927215},{"type":"v1","addr":"172.21.15.89:6837","nonce":2615927215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:39.106317+0000","modified":"2024-04-02T09:12:42.112504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10279":{"gid":10279,"name":"d","rank":0,"incarnation":107,"state":"up:active","state_seq":6,"addr":"172.21.15.113:6835/2457040702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2457040702},{"type":"v1","addr":"172.21.15.113:6835","nonce":2457040702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-02T09:13:05.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.835+0000 7fbdd9c00700 1 -- 172.21.15.89:0/3089361180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdd40cf230 msgr2=0x7fbdd41589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:05.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.835+0000 7fbdd9c00700 1 --2- 172.21.15.89:0/3089361180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdd40cf230 0x7fbdd41589b0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4009c40 tx=0x7fbdc4004750 comp rx=0 tx=0).stop 2024-04-02T09:13:05.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.835+0000 7fbdd9c00700 1 -- 172.21.15.89:0/3089361180 shutdown_connections 2024-04-02T09:13:05.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.835+0000 7fbdd9c00700 1 --2- 172.21.15.89:0/3089361180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdd40cf230 0x7fbdd41589b0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.835+0000 7fbdd9c00700 1 --2- 172.21.15.89:0/3089361180 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbdd4161590 0x7fbdd415e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.835+0000 7fbdd9c00700 1 --2- 172.21.15.89:0/3089361180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdd4160f10 0x7fbdd4157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:05.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.835+0000 7fbdd9c00700 1 -- 172.21.15.89:0/3089361180 >> 172.21.15.89:0/3089361180 conn(0x7fbdd408aff0 msgr2=0x7fbdd40b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:05.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.835+0000 7fbdd9c00700 1 -- 172.21.15.89:0/3089361180 shutdown_connections 2024-04-02T09:13:05.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:05.835+0000 7fbdd9c00700 1 -- 172.21.15.89:0/3089361180 wait complete. 2024-04-02T09:13:05.835 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 110 2024-04-02T09:13:05.851 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:13:06.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.082+0000 7ff402000700 1 Processor -- start 2024-04-02T09:13:06.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.082+0000 7ff402000700 1 -- start start 2024-04-02T09:13:06.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.084+0000 7ff402000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff3fc160f10 0x7ff3fc0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.084+0000 7ff402000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff3fc161590 0x7ff3fc161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.084+0000 7ff402000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc0ceda0 con 0x7ff3fc0cf230 2024-04-02T09:13:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.084+0000 7ff402000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3fc161f80 con 0x7ff3fc161590 2024-04-02T09:13:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.084+0000 7ff402000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3fc1612f0 con 0x7ff3fc160f10 2024-04-02T09:13:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.084+0000 7ff3fbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff3fc161590 0x7ff3fc161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.084+0000 7ff3fbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff3fc161590 0x7ff3fc161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52922/0 (socket says 172.21.15.89:52922) 2024-04-02T09:13:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.084+0000 7ff3fbe00700 1 -- 172.21.15.89:0/1470484345 learned_addr learned my addr 172.21.15.89:0/1470484345 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.084+0000 7ff3fb400700 1 --2- 172.21.15.89:0/1470484345 >> v2:172.21.15.113:3300/0 conn(0x7ff3fc160f10 0x7ff3fc0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.084+0000 7ff3fbe00700 1 -- 172.21.15.89:0/1470484345 >> v2:172.21.15.113:3300/0 conn(0x7ff3fc160f10 msgr2=0x7ff3fc0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.084+0000 7ff3fbe00700 1 --2- 172.21.15.89:0/1470484345 >> v2:172.21.15.113:3300/0 conn(0x7ff3fc160f10 0x7ff3fc0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.084+0000 7ff3fbe00700 1 -- 172.21.15.89:0/1470484345 >> v1:172.21.15.113:6789/0 conn(0x7ff3fc0cf230 legacy=0x7ff3fc0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.084+0000 7ff3fbe00700 1 -- 172.21.15.89:0/1470484345 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc0ce470 con 0x7ff3fc161590 2024-04-02T09:13:06.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.085+0000 7ff3fbe00700 1 --2- 172.21.15.89:0/1470484345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3fc161590 0x7ff3fc161a40 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0d1080 tx=0x7ff3f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10ab19725fb6408 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:06.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.085+0000 7ff3fa000700 1 -- 172.21.15.89:0/1470484345 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f400e070 con 0x7ff3fc161590 2024-04-02T09:13:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.085+0000 7ff3fa000700 1 -- 172.21.15.89:0/1470484345 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f4003a20 con 0x7ff3fc161590 2024-04-02T09:13:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.085+0000 7ff3fa000700 1 -- 172.21.15.89:0/1470484345 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4003dd0 con 0x7ff3fc161590 2024-04-02T09:13:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.085+0000 7ff402000700 1 -- 172.21.15.89:0/1470484345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3fc161590 msgr2=0x7ff3fc161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.085+0000 7ff402000700 1 --2- 172.21.15.89:0/1470484345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3fc161590 0x7ff3fc161a40 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0d1080 tx=0x7ff3f400b330 comp rx=0 tx=0).stop 2024-04-02T09:13:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.085+0000 7ff402000700 1 -- 172.21.15.89:0/1470484345 shutdown_connections 2024-04-02T09:13:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.085+0000 7ff402000700 1 --2- 172.21.15.89:0/1470484345 >> v2:172.21.15.113:3300/0 conn(0x7ff3fc160f10 0x7ff3fc0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.085+0000 7ff402000700 1 --2- 172.21.15.89:0/1470484345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3fc161590 0x7ff3fc161a40 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.085+0000 7ff402000700 1 -- 172.21.15.89:0/1470484345 >> 172.21.15.89:0/1470484345 conn(0x7ff3fc08aff0 msgr2=0x7ff3fc0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.086+0000 7ff402000700 1 -- 172.21.15.89:0/1470484345 shutdown_connections 2024-04-02T09:13:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.086+0000 7ff402000700 1 -- 172.21.15.89:0/1470484345 wait complete. 2024-04-02T09:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.087+0000 7ff402000700 1 Processor -- start 2024-04-02T09:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.087+0000 7ff402000700 1 -- start start 2024-04-02T09:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.088+0000 7ff402000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3fc0cf230 0x7ff3fc147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.088+0000 7ff402000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3fc160f10 0x7ff3fc160170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.088+0000 7ff402000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3fc161590 0x7ff3fc146b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.088+0000 7ff402000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc148530 con 0x7ff3fc0cf230 2024-04-02T09:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.088+0000 7ff402000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc145af0 con 0x7ff3fc161590 2024-04-02T09:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.088+0000 7ff402000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc145dd0 con 0x7ff3fc160f10 2024-04-02T09:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.088+0000 7ff3fb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3fc0cf230 0x7ff3fc147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.088+0000 7ff3fb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3fc0cf230 0x7ff3fc147ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52928/0 (socket says 172.21.15.89:52928) 2024-04-02T09:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.088+0000 7ff3fb400700 1 -- 172.21.15.89:0/3143074698 learned_addr learned my addr 172.21.15.89:0/3143074698 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.089+0000 7ff3faa00700 1 --2- 172.21.15.89:0/3143074698 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3fc160f10 0x7ff3fc160170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.089+0000 7ff3fbe00700 1 --2- 172.21.15.89:0/3143074698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3fc161590 0x7ff3fc146b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.089+0000 7ff3fb400700 1 -- 172.21.15.89:0/3143074698 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3fc160f10 msgr2=0x7ff3fc160170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.089+0000 7ff3fb400700 1 --2- 172.21.15.89:0/3143074698 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3fc160f10 0x7ff3fc160170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.089+0000 7ff3fb400700 1 -- 172.21.15.89:0/3143074698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3fc161590 msgr2=0x7ff3fc146b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.089+0000 7ff3fb400700 1 --2- 172.21.15.89:0/3143074698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3fc161590 0x7ff3fc146b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.089+0000 7ff3fb400700 1 -- 172.21.15.89:0/3143074698 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f4009b30 con 0x7ff3fc0cf230 2024-04-02T09:13:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.089+0000 7ff3fb400700 1 --2- 172.21.15.89:0/3143074698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3fc0cf230 0x7ff3fc147ff0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00d800 tx=0x7ff3ec00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.089+0000 7ff3f3400700 1 -- 172.21.15.89:0/3143074698 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec0125f0 con 0x7ff3fc0cf230 2024-04-02T09:13:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.089+0000 7ff3f3400700 1 -- 172.21.15.89:0/3143074698 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ec012750 con 0x7ff3fc0cf230 2024-04-02T09:13:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.089+0000 7ff3f3400700 1 -- 172.21.15.89:0/3143074698 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec012b00 con 0x7ff3fc0cf230 2024-04-02T09:13:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.089+0000 7ff402000700 1 -- 172.21.15.89:0/3143074698 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3fc1460b0 con 0x7ff3fc0cf230 2024-04-02T09:13:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.090+0000 7ff402000700 1 -- 172.21.15.89:0/3143074698 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3fc146670 con 0x7ff3fc0cf230 2024-04-02T09:13:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.091+0000 7ff402000700 1 -- 172.21.15.89:0/3143074698 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3fc146890 con 0x7ff3fc0cf230 2024-04-02T09:13:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.091+0000 7ff3f3400700 1 -- 172.21.15.89:0/3143074698 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 155) v1 ==== 88848+0+0 (secure 0 0 0) 0x7ff3ec012c60 con 0x7ff3fc0cf230 2024-04-02T09:13:06.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.092+0000 7ff3f3400700 1 -- 172.21.15.89:0/3143074698 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7752+0+0 (secure 0 0 0) 0x7ff3ec089710 con 0x7ff3fc0cf230 2024-04-02T09:13:06.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.097+0000 7ff3f3400700 1 -- 172.21.15.89:0/3143074698 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff3ec054450 con 0x7ff3fc0cf230 2024-04-02T09:13:06.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.300+0000 7ff402000700 1 -- 172.21.15.89:0/3143074698 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3fc067720 con 0x7ff3fc0cf230 2024-04-02T09:13:06.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.302+0000 7ff3f3400700 1 -- 172.21.15.89:0/3143074698 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+21507 (secure 0 0 0) 0x7ff3ec05ce60 con 0x7ff3fc0cf230 2024-04-02T09:13:06.300 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:06.300 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":229,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:13:05.404764+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:12:23.034869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-02T09:12:24.158560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:12:37.240568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:12:38.093999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/1531440110":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/92627425":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/3786436284":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/3784813489":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/38958757":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/544818595":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/1763496588":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/341049398":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/2216461018":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/3268555703":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/3481438563":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/783107677":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/2439246170":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/965177671":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/2516488332":"2024-04-03T09:13:05.404716+0000","172.21.15.89:6833/108855":"2024-04-03T09:12:53.003657+0000","172.21.15.89:6832/108855":"2024-04-03T09:12:53.003657+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:13:06.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.304+0000 7ff402000700 1 -- 172.21.15.89:0/3143074698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3fc0cf230 msgr2=0x7ff3fc147ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:06.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.304+0000 7ff402000700 1 --2- 172.21.15.89:0/3143074698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3fc0cf230 0x7ff3fc147ff0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00d800 tx=0x7ff3ec00db10 comp rx=0 tx=0).stop 2024-04-02T09:13:06.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.304+0000 7ff402000700 1 -- 172.21.15.89:0/3143074698 shutdown_connections 2024-04-02T09:13:06.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.304+0000 7ff402000700 1 --2- 172.21.15.89:0/3143074698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3fc161590 0x7ff3fc146b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.304+0000 7ff402000700 1 --2- 172.21.15.89:0/3143074698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3fc0cf230 0x7ff3fc147ff0 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.304+0000 7ff402000700 1 --2- 172.21.15.89:0/3143074698 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3fc160f10 0x7ff3fc160170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.304+0000 7ff402000700 1 -- 172.21.15.89:0/3143074698 >> 172.21.15.89:0/3143074698 conn(0x7ff3fc08aff0 msgr2=0x7ff3fc0ca730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:06.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.304+0000 7ff402000700 1 -- 172.21.15.89:0/3143074698 shutdown_connections 2024-04-02T09:13:06.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.304+0000 7ff402000700 1 -- 172.21.15.89:0/3143074698 wait complete. 2024-04-02T09:13:06.320 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T09:13:06.320 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:13:06.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.520+0000 7f1ea0e00700 1 Processor -- start 2024-04-02T09:13:06.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.520+0000 7f1ea0e00700 1 -- start start 2024-04-02T09:13:06.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.521+0000 7f1ea0e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1e9c15d920 0x7f1e9c06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:06.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.521+0000 7f1ea0e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1e9c099a60 0x7f1e9c099eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:06.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.521+0000 7f1ea0e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e9c09a3f0 con 0x7f1e9c09a910 2024-04-02T09:13:06.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.521+0000 7f1ea0e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e9c0d08d0 con 0x7f1e9c099a60 2024-04-02T09:13:06.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.521+0000 7f1ea0e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c06c300 con 0x7f1e9c15d920 2024-04-02T09:13:06.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.521+0000 7f1e9a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1e9c099a60 0x7f1e9c099eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:06.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.522+0000 7f1e9a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1e9c099a60 0x7f1e9c099eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52936/0 (socket says 172.21.15.89:52936) 2024-04-02T09:13:06.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.522+0000 7f1e9a800700 1 -- 172.21.15.89:0/883047145 learned_addr learned my addr 172.21.15.89:0/883047145 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:06.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.522+0000 7f1e99e00700 1 --2- 172.21.15.89:0/883047145 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1e9c15d920 0x7f1e9c06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:06.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.522+0000 7f1e9a800700 1 -- 172.21.15.89:0/883047145 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1e9c15d920 msgr2=0x7f1e9c06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:06.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.522+0000 7f1e9a800700 1 --2- 172.21.15.89:0/883047145 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1e9c15d920 0x7f1e9c06c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.522+0000 7f1e9a800700 1 -- 172.21.15.89:0/883047145 >> v1:172.21.15.113:6789/0 conn(0x7f1e9c09a910 legacy=0x7f1e9c0cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:06.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.522+0000 7f1e9a800700 1 -- 172.21.15.89:0/883047145 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9c06b9d0 con 0x7f1e9c099a60 2024-04-02T09:13:06.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.522+0000 7f1e9a800700 1 --2- 172.21.15.89:0/883047145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c099a60 0x7f1e9c099eb0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009e50 tx=0x7f1e9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=388001edd5fe160d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:06.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.522+0000 7f1e98a00700 1 -- 172.21.15.89:0/883047145 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9400e070 con 0x7f1e9c099a60 2024-04-02T09:13:06.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.522+0000 7f1e98a00700 1 -- 172.21.15.89:0/883047145 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9400b9c0 con 0x7f1e9c099a60 2024-04-02T09:13:06.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.522+0000 7f1e98a00700 1 -- 172.21.15.89:0/883047145 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9400bd70 con 0x7f1e9c099a60 2024-04-02T09:13:06.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.523+0000 7f1ea0e00700 1 -- 172.21.15.89:0/883047145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c099a60 msgr2=0x7f1e9c099eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.523+0000 7f1ea0e00700 1 --2- 172.21.15.89:0/883047145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c099a60 0x7f1e9c099eb0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009e50 tx=0x7f1e9400b330 comp rx=0 tx=0).stop 2024-04-02T09:13:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.523+0000 7f1ea0e00700 1 -- 172.21.15.89:0/883047145 shutdown_connections 2024-04-02T09:13:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.523+0000 7f1ea0e00700 1 --2- 172.21.15.89:0/883047145 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1e9c15d920 0x7f1e9c06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.523+0000 7f1ea0e00700 1 --2- 172.21.15.89:0/883047145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c099a60 0x7f1e9c099eb0 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.523+0000 7f1ea0e00700 1 -- 172.21.15.89:0/883047145 >> 172.21.15.89:0/883047145 conn(0x7f1e9c0d1350 msgr2=0x7f1e9c0d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.523+0000 7f1ea0e00700 1 -- 172.21.15.89:0/883047145 shutdown_connections 2024-04-02T09:13:06.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.523+0000 7f1ea0e00700 1 -- 172.21.15.89:0/883047145 wait complete. 2024-04-02T09:13:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.525+0000 7f1ea0e00700 1 Processor -- start 2024-04-02T09:13:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.525+0000 7f1ea0e00700 1 -- start start 2024-04-02T09:13:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.526+0000 7f1ea0e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e9c099a60 0x7f1e9c15c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.526+0000 7f1ea0e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c09a910 0x7f1e9c0ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.526+0000 7f1ea0e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1e9c15d920 0x7f1e9c0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.526+0000 7f1ea0e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c0ca4a0 con 0x7f1e9c09a910 2024-04-02T09:13:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.526+0000 7f1ea0e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c1f0ba0 con 0x7f1e9c099a60 2024-04-02T09:13:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.526+0000 7f1ea0e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c0c9ee0 con 0x7f1e9c15d920 2024-04-02T09:13:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.526+0000 7f1e99400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c09a910 0x7f1e9c0ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.526+0000 7f1e99400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c09a910 0x7f1e9c0ae1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52952/0 (socket says 172.21.15.89:52952) 2024-04-02T09:13:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.526+0000 7f1e99400700 1 -- 172.21.15.89:0/4294792462 learned_addr learned my addr 172.21.15.89:0/4294792462 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.526+0000 7f1e99400700 1 -- 172.21.15.89:0/4294792462 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1e9c15d920 msgr2=0x7f1e9c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:06.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.526+0000 7f1e99400700 1 --2- 172.21.15.89:0/4294792462 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1e9c15d920 0x7f1e9c0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.526+0000 7f1e99400700 1 -- 172.21.15.89:0/4294792462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e9c099a60 msgr2=0x7f1e9c15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.526+0000 7f1e99400700 1 --2- 172.21.15.89:0/4294792462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e9c099a60 0x7f1e9c15c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.526+0000 7f1e99400700 1 -- 172.21.15.89:0/4294792462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e94009b30 con 0x7f1e9c09a910 2024-04-02T09:13:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.527+0000 7f1e99400700 1 --2- 172.21.15.89:0/4294792462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c09a910 0x7f1e9c0ae1f0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f1e8800c4b0 tx=0x7f1e8800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:06.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.527+0000 7f1e92000700 1 -- 172.21.15.89:0/4294792462 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e880052d0 con 0x7f1e9c09a910 2024-04-02T09:13:06.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.527+0000 7f1e92000700 1 -- 172.21.15.89:0/4294792462 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e88005430 con 0x7f1e9c09a910 2024-04-02T09:13:06.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.527+0000 7f1e92000700 1 -- 172.21.15.89:0/4294792462 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e880038c0 con 0x7f1e9c09a910 2024-04-02T09:13:06.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.527+0000 7f1ea0e00700 1 -- 172.21.15.89:0/4294792462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9c0ca1c0 con 0x7f1e9c09a910 2024-04-02T09:13:06.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.527+0000 7f1ea0e00700 1 -- 172.21.15.89:0/4294792462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e9c15f1d0 con 0x7f1e9c09a910 2024-04-02T09:13:06.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.528+0000 7f1ea0e00700 1 -- 172.21.15.89:0/4294792462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e9c15ed50 con 0x7f1e9c09a910 2024-04-02T09:13:06.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.529+0000 7f1e92000700 1 -- 172.21.15.89:0/4294792462 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 156) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f1e88003a20 con 0x7f1e9c09a910 2024-04-02T09:13:06.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.530+0000 7f1e92000700 1 --2- 172.21.15.89:0/4294792462 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1e7c06aae0 0x7f1e7c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:06.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.530+0000 7f1e92000700 1 -- 172.21.15.89:0/4294792462 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7752+0+0 (secure 0 0 0) 0x7f1e88088ae0 con 0x7f1e9c09a910 2024-04-02T09:13:06.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.530+0000 7f1e99e00700 1 --2- 172.21.15.89:0/4294792462 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1e7c06aae0 0x7f1e7c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:06.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.531+0000 7f1e99e00700 1 --2- 172.21.15.89:0/4294792462 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1e7c06aae0 0x7f1e7c06cfa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c160e30 tx=0x7f1e8c008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:06.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.535+0000 7f1e92000700 1 -- 172.21.15.89:0/4294792462 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1e9c15ed50 con 0x7f1e9c09a910 2024-04-02T09:13:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.782+0000 7f1ea0e00700 1 -- 172.21.15.89:0/4294792462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e9c09a3f0 con 0x7f1e9c09a910 2024-04-02T09:13:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.782+0000 7f1e92000700 1 -- 172.21.15.89:0/4294792462 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5971 (secure 0 0 0) 0x7f1e9c09a3f0 con 0x7f1e9c09a910 2024-04-02T09:13:06.780 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:06.781 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":110,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8969,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3650347444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3650347444},{"type":"v1","addr":"172.21.15.89:6836","nonce":3650347444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2152500685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2152500685},{"type":"v1","addr":"172.21.15.113:6834","nonce":2152500685}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:25.170849+0000","modified":"2024-04-02T09:12:28.185307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11157":{"gid":11157,"name":"a","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2615927215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2615927215},{"type":"v1","addr":"172.21.15.89:6837","nonce":2615927215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:39.106317+0000","modified":"2024-04-02T09:12:42.112504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10279":{"gid":10279,"name":"d","rank":0,"incarnation":107,"state":"up:active","state_seq":6,"addr":"172.21.15.113:6835/2457040702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2457040702},{"type":"v1","addr":"172.21.15.113:6835","nonce":2457040702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-02T09:13:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.784+0000 7f1ea0e00700 1 -- 172.21.15.89:0/4294792462 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1e7c06aae0 msgr2=0x7f1e7c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.784+0000 7f1ea0e00700 1 --2- 172.21.15.89:0/4294792462 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1e7c06aae0 0x7f1e7c06cfa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c160e30 tx=0x7f1e8c008040 comp rx=0 tx=0).stop 2024-04-02T09:13:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.784+0000 7f1ea0e00700 1 -- 172.21.15.89:0/4294792462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c09a910 msgr2=0x7f1e9c0ae1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.785+0000 7f1ea0e00700 1 --2- 172.21.15.89:0/4294792462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c09a910 0x7f1e9c0ae1f0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f1e8800c4b0 tx=0x7f1e8800c870 comp rx=0 tx=0).stop 2024-04-02T09:13:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.785+0000 7f1ea0e00700 1 -- 172.21.15.89:0/4294792462 shutdown_connections 2024-04-02T09:13:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.785+0000 7f1ea0e00700 1 --2- 172.21.15.89:0/4294792462 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1e7c06aae0 0x7f1e7c06cfa0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.785+0000 7f1ea0e00700 1 --2- 172.21.15.89:0/4294792462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e9c099a60 0x7f1e9c15c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.785+0000 7f1ea0e00700 1 --2- 172.21.15.89:0/4294792462 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1e9c15d920 0x7f1e9c0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.785+0000 7f1ea0e00700 1 --2- 172.21.15.89:0/4294792462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9c09a910 0x7f1e9c0ae1f0 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.785+0000 7f1ea0e00700 1 -- 172.21.15.89:0/4294792462 >> 172.21.15.89:0/4294792462 conn(0x7f1e9c0d1350 msgr2=0x7f1e9c06ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.785+0000 7f1ea0e00700 1 -- 172.21.15.89:0/4294792462 shutdown_connections 2024-04-02T09:13:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.785+0000 7f1ea0e00700 1 -- 172.21.15.89:0/4294792462 wait complete. 2024-04-02T09:13:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 110 2024-04-02T09:13:06.801 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:13:06.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.995+0000 7f590b200700 1 Processor -- start 2024-04-02T09:13:06.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.996+0000 7f590b200700 1 -- start start 2024-04-02T09:13:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.997+0000 7f590b200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f590415d760 0x7f59040d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.997+0000 7f590b200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f59040cf3c0 0x7f59040cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.997+0000 7f590b200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59040d1330 con 0x7f590415db40 2024-04-02T09:13:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.997+0000 7f590b200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59040d1fb0 con 0x7f59040cf3c0 2024-04-02T09:13:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.997+0000 7f590b200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59040d1610 con 0x7f590415d760 2024-04-02T09:13:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.997+0000 7f5909200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f59040cf3c0 0x7f59040cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.997+0000 7f5909200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f59040cf3c0 0x7f59040cffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52968/0 (socket says 172.21.15.89:52968) 2024-04-02T09:13:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.997+0000 7f5909200700 1 -- 172.21.15.89:0/1590368430 learned_addr learned my addr 172.21.15.89:0/1590368430 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.997+0000 7f5909200700 1 -- 172.21.15.89:0/1590368430 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f590415d760 msgr2=0x7f59040d23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.997+0000 7f5909200700 1 --2- 172.21.15.89:0/1590368430 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f590415d760 0x7f59040d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.997+0000 7f5909200700 1 -- 172.21.15.89:0/1590368430 >> v1:172.21.15.113:6789/0 conn(0x7f590415db40 legacy=0x7f59040cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.998+0000 7f5909200700 1 -- 172.21.15.89:0/1590368430 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59040d18f0 con 0x7f59040cf3c0 2024-04-02T09:13:06.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.998+0000 7f5909200700 1 --2- 172.21.15.89:0/1590368430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59040cf3c0 0x7f59040cffd0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009e50 tx=0x7f58fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2476589299893a8c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.998+0000 7f5902a00700 1 -- 172.21.15.89:0/1590368430 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc00e070 con 0x7f59040cf3c0 2024-04-02T09:13:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.998+0000 7f5902a00700 1 -- 172.21.15.89:0/1590368430 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58fc003a20 con 0x7f59040cf3c0 2024-04-02T09:13:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.998+0000 7f5902a00700 1 -- 172.21.15.89:0/1590368430 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc003dd0 con 0x7f59040cf3c0 2024-04-02T09:13:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.999+0000 7f590b200700 1 -- 172.21.15.89:0/1590368430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59040cf3c0 msgr2=0x7f59040cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.999+0000 7f590b200700 1 --2- 172.21.15.89:0/1590368430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59040cf3c0 0x7f59040cffd0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009e50 tx=0x7f58fc00b330 comp rx=0 tx=0).stop 2024-04-02T09:13:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.999+0000 7f590b200700 1 -- 172.21.15.89:0/1590368430 shutdown_connections 2024-04-02T09:13:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.999+0000 7f590b200700 1 --2- 172.21.15.89:0/1590368430 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f590415d760 0x7f59040d23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.999+0000 7f590b200700 1 --2- 172.21.15.89:0/1590368430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59040cf3c0 0x7f59040cffd0 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.999+0000 7f590b200700 1 -- 172.21.15.89:0/1590368430 >> 172.21.15.89:0/1590368430 conn(0x7f590408aff0 msgr2=0x7f59040d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.999+0000 7f590b200700 1 -- 172.21.15.89:0/1590368430 shutdown_connections 2024-04-02T09:13:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:06.999+0000 7f590b200700 1 -- 172.21.15.89:0/1590368430 wait complete. 2024-04-02T09:13:06.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.001+0000 7f590b200700 1 Processor -- start 2024-04-02T09:13:06.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.001+0000 7f590b200700 1 -- start start 2024-04-02T09:13:07.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f590b200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59040cf3c0 0x7f59040b6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f590b200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590415d760 0x7f590415a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f590b200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f590415db40 0x7f59040a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f590b200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59040a10a0 con 0x7f590415d760 2024-04-02T09:13:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f590b200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59040cbd80 con 0x7f59040cf3c0 2024-04-02T09:13:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f590b200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59040b7ba0 con 0x7f590415db40 2024-04-02T09:13:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f5903400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590415d760 0x7f590415a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f5903400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590415d760 0x7f590415a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52978/0 (socket says 172.21.15.89:52978) 2024-04-02T09:13:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f5903400700 1 -- 172.21.15.89:0/2272444048 learned_addr learned my addr 172.21.15.89:0/2272444048 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f5903e00700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59040cf3c0 0x7f59040b6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f5909200700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f590415db40 0x7f59040a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f5903400700 1 -- 172.21.15.89:0/2272444048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f590415db40 msgr2=0x7f59040a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f5903400700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f590415db40 0x7f59040a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f5903400700 1 -- 172.21.15.89:0/2272444048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59040cf3c0 msgr2=0x7f59040b6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f5903400700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59040cf3c0 0x7f59040b6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f5903400700 1 -- 172.21.15.89:0/2272444048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc009b30 con 0x7f590415d760 2024-04-02T09:13:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f5909200700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f590415db40 0x7f59040a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:07.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f5903e00700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59040cf3c0 0x7f59040b6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:07.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.002+0000 7f5903400700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590415d760 0x7f590415a950 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f58f000b450 tx=0x7f58f000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:07.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.003+0000 7f58fbe00700 1 -- 172.21.15.89:0/2272444048 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0011830 con 0x7f590415d760 2024-04-02T09:13:07.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.003+0000 7f58fbe00700 1 -- 172.21.15.89:0/2272444048 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f0011990 con 0x7f590415d760 2024-04-02T09:13:07.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.003+0000 7f590b200700 1 -- 172.21.15.89:0/2272444048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59040b75e0 con 0x7f590415d760 2024-04-02T09:13:07.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.003+0000 7f58fbe00700 1 -- 172.21.15.89:0/2272444048 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0011d40 con 0x7f590415d760 2024-04-02T09:13:07.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.003+0000 7f590b200700 1 -- 172.21.15.89:0/2272444048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59040cc020 con 0x7f590415d760 2024-04-02T09:13:07.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.004+0000 7f590b200700 1 -- 172.21.15.89:0/2272444048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59040cbbe0 con 0x7f590415d760 2024-04-02T09:13:07.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.005+0000 7f58fbe00700 1 -- 172.21.15.89:0/2272444048 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 156) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f58f001a400 con 0x7f590415d760 2024-04-02T09:13:07.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.005+0000 7f58fbe00700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f58e406aae0 0x7f58e406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.006+0000 7f58fbe00700 1 -- 172.21.15.89:0/2272444048 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7752+0+0 (secure 0 0 0) 0x7f58f00894d0 con 0x7f590415d760 2024-04-02T09:13:07.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.006+0000 7f5903e00700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f58e406aae0 0x7f58e406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.006+0000 7f5903e00700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f58e406aae0 0x7f58e406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f58f4007900 tx=0x7f58f4008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:07.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.010+0000 7f58fbe00700 1 -- 172.21.15.89:0/2272444048 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59040cbbe0 con 0x7f590415d760 2024-04-02T09:13:07.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.260+0000 7f590b200700 1 -- 172.21.15.89:0/2272444048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59040cbbe0 con 0x7f590415d760 2024-04-02T09:13:07.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.261+0000 7f58fbe00700 1 -- 172.21.15.89:0/2272444048 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5971 (secure 0 0 0) 0x7f59040cbbe0 con 0x7f590415d760 2024-04-02T09:13:07.259 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:07.259 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":110,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8969,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3650347444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3650347444},{"type":"v1","addr":"172.21.15.89:6836","nonce":3650347444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2152500685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2152500685},{"type":"v1","addr":"172.21.15.113:6834","nonce":2152500685}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:25.170849+0000","modified":"2024-04-02T09:12:28.185307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11157":{"gid":11157,"name":"a","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2615927215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2615927215},{"type":"v1","addr":"172.21.15.89:6837","nonce":2615927215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:39.106317+0000","modified":"2024-04-02T09:12:42.112504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10279":{"gid":10279,"name":"d","rank":0,"incarnation":107,"state":"up:active","state_seq":6,"addr":"172.21.15.113:6835/2457040702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2457040702},{"type":"v1","addr":"172.21.15.113:6835","nonce":2457040702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-02T09:13:07.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.264+0000 7f590b200700 1 -- 172.21.15.89:0/2272444048 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f58e406aae0 msgr2=0x7f58e406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.264+0000 7f590b200700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f58e406aae0 0x7f58e406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f58f4007900 tx=0x7f58f4008040 comp rx=0 tx=0).stop 2024-04-02T09:13:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.264+0000 7f590b200700 1 -- 172.21.15.89:0/2272444048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590415d760 msgr2=0x7f590415a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.264+0000 7f590b200700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590415d760 0x7f590415a950 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f58f000b450 tx=0x7f58f000b810 comp rx=0 tx=0).stop 2024-04-02T09:13:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.264+0000 7f590b200700 1 -- 172.21.15.89:0/2272444048 shutdown_connections 2024-04-02T09:13:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.264+0000 7f590b200700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f58e406aae0 0x7f58e406cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.264+0000 7f590b200700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59040cf3c0 0x7f59040b6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.264+0000 7f590b200700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f590415db40 0x7f59040a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.264+0000 7f590b200700 1 --2- 172.21.15.89:0/2272444048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f590415d760 0x7f590415a950 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.264+0000 7f590b200700 1 -- 172.21.15.89:0/2272444048 >> 172.21.15.89:0/2272444048 conn(0x7f590408aff0 msgr2=0x7f59040cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.264+0000 7f590b200700 1 -- 172.21.15.89:0/2272444048 shutdown_connections 2024-04-02T09:13:07.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.264+0000 7f590b200700 1 -- 172.21.15.89:0/2272444048 wait complete. 2024-04-02T09:13:07.264 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 110 2024-04-02T09:13:07.280 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:13:07.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.483+0000 7ff7a1000700 1 Processor -- start 2024-04-02T09:13:07.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.483+0000 7ff7a1000700 1 -- start start 2024-04-02T09:13:07.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.484+0000 7ff7a1000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff79c160f10 0x7ff79c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.484+0000 7ff7a1000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff79c0cf230 0x7ff79c0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.484+0000 7ff7a1000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff79c161590 0x7ff79c161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.484+0000 7ff7a1000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79c161f60 con 0x7ff79c0cf230 2024-04-02T09:13:07.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.484+0000 7ff7a1000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79c0ceda0 con 0x7ff79c160f10 2024-04-02T09:13:07.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.484+0000 7ff7a1000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c0d23d0 con 0x7ff79c161590 2024-04-02T09:13:07.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.484+0000 7ff799400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff79c0cf230 0x7ff79c0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.485+0000 7ff799400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff79c0cf230 0x7ff79c0d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52988/0 (socket says 172.21.15.89:52988) 2024-04-02T09:13:07.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.485+0000 7ff799400700 1 -- 172.21.15.89:0/2734121139 learned_addr learned my addr 172.21.15.89:0/2734121139 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:07.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.485+0000 7ff799e00700 1 --2- 172.21.15.89:0/2734121139 >> v2:172.21.15.113:3300/0 conn(0x7ff79c160f10 0x7ff79c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.485+0000 7ff79a800700 1 --2- 172.21.15.89:0/2734121139 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff79c161590 0x7ff79c161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.485+0000 7ff799400700 1 -- 172.21.15.89:0/2734121139 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff79c161590 msgr2=0x7ff79c161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:07.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.485+0000 7ff799400700 1 --2- 172.21.15.89:0/2734121139 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff79c161590 0x7ff79c161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.485+0000 7ff799400700 1 -- 172.21.15.89:0/2734121139 >> v2:172.21.15.113:3300/0 conn(0x7ff79c160f10 msgr2=0x7ff79c0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:07.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.485+0000 7ff799400700 1 --2- 172.21.15.89:0/2734121139 >> v2:172.21.15.113:3300/0 conn(0x7ff79c160f10 0x7ff79c0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.485+0000 7ff799400700 1 -- 172.21.15.89:0/2734121139 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c1612f0 con 0x7ff79c0cf230 2024-04-02T09:13:07.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.485+0000 7ff799e00700 1 --2- 172.21.15.89:0/2734121139 >> v2:172.21.15.113:3300/0 conn(0x7ff79c160f10 0x7ff79c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:07.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.485+0000 7ff799400700 1 --2- 172.21.15.89:0/2734121139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff79c0cf230 0x7ff79c0d1e90 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0d1080 tx=0x7ff788009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f432212fbd7cf47f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:07.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.486+0000 7ff798a00700 1 -- 172.21.15.89:0/2734121139 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788011830 con 0x7ff79c0cf230 2024-04-02T09:13:07.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.486+0000 7ff798a00700 1 -- 172.21.15.89:0/2734121139 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff788011990 con 0x7ff79c0cf230 2024-04-02T09:13:07.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.486+0000 7ff798a00700 1 -- 172.21.15.89:0/2734121139 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788011d20 con 0x7ff79c0cf230 2024-04-02T09:13:07.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.486+0000 7ff7a1000700 1 -- 172.21.15.89:0/2734121139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff79c0cf230 msgr2=0x7ff79c0d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:07.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.486+0000 7ff7a1000700 1 --2- 172.21.15.89:0/2734121139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff79c0cf230 0x7ff79c0d1e90 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0d1080 tx=0x7ff788009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.486+0000 7ff7a1000700 1 -- 172.21.15.89:0/2734121139 shutdown_connections 2024-04-02T09:13:07.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.486+0000 7ff7a1000700 1 --2- 172.21.15.89:0/2734121139 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff79c161590 0x7ff79c161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.486+0000 7ff7a1000700 1 --2- 172.21.15.89:0/2734121139 >> v2:172.21.15.113:3300/0 conn(0x7ff79c160f10 0x7ff79c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.486+0000 7ff7a1000700 1 --2- 172.21.15.89:0/2734121139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff79c0cf230 0x7ff79c0d1e90 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.486+0000 7ff7a1000700 1 -- 172.21.15.89:0/2734121139 >> 172.21.15.89:0/2734121139 conn(0x7ff79c08aff0 msgr2=0x7ff79c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:07.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.486+0000 7ff7a1000700 1 -- 172.21.15.89:0/2734121139 shutdown_connections 2024-04-02T09:13:07.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.486+0000 7ff7a1000700 1 -- 172.21.15.89:0/2734121139 wait complete. 2024-04-02T09:13:07.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.489+0000 7ff7a1000700 1 Processor -- start 2024-04-02T09:13:07.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.489+0000 7ff7a1000700 1 -- start start 2024-04-02T09:13:07.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.491+0000 7ff7a1000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff79c0cf230 0x7ff79c14ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.491+0000 7ff7a1000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c160f10 0x7ff79c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.491+0000 7ff7a1000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff79c161590 0x7ff79c15f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.491+0000 7ff7a1000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c1568d0 con 0x7ff79c161590 2024-04-02T09:13:07.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.491+0000 7ff7a1000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c15fd00 con 0x7ff79c160f10 2024-04-02T09:13:07.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.491+0000 7ff7a1000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c15ffe0 con 0x7ff79c0cf230 2024-04-02T09:13:07.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.491+0000 7ff79a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff79c161590 0x7ff79c15f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.491+0000 7ff79a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff79c161590 0x7ff79c15f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53004/0 (socket says 172.21.15.89:53004) 2024-04-02T09:13:07.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.491+0000 7ff79a800700 1 -- 172.21.15.89:0/4005291028 learned_addr learned my addr 172.21.15.89:0/4005291028 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:07.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.491+0000 7ff799e00700 1 --2- 172.21.15.89:0/4005291028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff79c0cf230 0x7ff79c14ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.491+0000 7ff799400700 1 --2- 172.21.15.89:0/4005291028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c160f10 0x7ff79c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.492+0000 7ff79a800700 1 -- 172.21.15.89:0/4005291028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff79c0cf230 msgr2=0x7ff79c14ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:07.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.492+0000 7ff79a800700 1 --2- 172.21.15.89:0/4005291028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff79c0cf230 0x7ff79c14ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.492+0000 7ff79a800700 1 -- 172.21.15.89:0/4005291028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c160f10 msgr2=0x7ff79c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:07.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.492+0000 7ff79a800700 1 --2- 172.21.15.89:0/4005291028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c160f10 0x7ff79c15ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.492+0000 7ff79a800700 1 -- 172.21.15.89:0/4005291028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c0094a0 con 0x7ff79c161590 2024-04-02T09:13:07.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.492+0000 7ff79a800700 1 --2- 172.21.15.89:0/4005291028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff79c161590 0x7ff79c15f4d0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7ff79400efc0 tx=0x7ff79400c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:07.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.492+0000 7ff792000700 1 -- 172.21.15.89:0/4005291028 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79400cad0 con 0x7ff79c161590 2024-04-02T09:13:07.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.492+0000 7ff792000700 1 -- 172.21.15.89:0/4005291028 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79400f040 con 0x7ff79c161590 2024-04-02T09:13:07.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.492+0000 7ff7a1000700 1 -- 172.21.15.89:0/4005291028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff788009550 con 0x7ff79c161590 2024-04-02T09:13:07.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.492+0000 7ff792000700 1 -- 172.21.15.89:0/4005291028 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79400ce70 con 0x7ff79c161590 2024-04-02T09:13:07.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.492+0000 7ff7a1000700 1 -- 172.21.15.89:0/4005291028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79c156560 con 0x7ff79c161590 2024-04-02T09:13:07.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.494+0000 7ff7a1000700 1 -- 172.21.15.89:0/4005291028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff79c156a40 con 0x7ff79c161590 2024-04-02T09:13:07.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.494+0000 7ff792000700 1 -- 172.21.15.89:0/4005291028 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 156) v1 ==== 88975+0+0 (secure 0 0 0) 0x7ff794014480 con 0x7ff79c161590 2024-04-02T09:13:07.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.496+0000 7ff792000700 1 --2- 172.21.15.89:0/4005291028 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff77c06ab10 0x7ff77c06cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.496+0000 7ff792000700 1 -- 172.21.15.89:0/4005291028 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7752+0+0 (secure 0 0 0) 0x7ff794089e90 con 0x7ff79c161590 2024-04-02T09:13:07.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.496+0000 7ff799e00700 1 --2- 172.21.15.89:0/4005291028 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff77c06ab10 0x7ff77c06cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.497+0000 7ff799e00700 1 --2- 172.21.15.89:0/4005291028 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff77c06ab10 0x7ff77c06cfd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009cd0 tx=0x7ff78c009210 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:07.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.500+0000 7ff792000700 1 -- 172.21.15.89:0/4005291028 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff794054b20 con 0x7ff79c161590 2024-04-02T09:13:07.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.705+0000 7ff7a1000700 1 -- 172.21.15.89:0/4005291028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff79c067720 con 0x7ff79c161590 2024-04-02T09:13:07.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.706+0000 7ff792000700 1 -- 172.21.15.89:0/4005291028 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+21507 (secure 0 0 0) 0x7ff794018090 con 0x7ff79c161590 2024-04-02T09:13:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:07.705 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":229,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:13:05.404764+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:12:23.034869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-02T09:12:24.158560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:12:37.240568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:12:38.093999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/1531440110":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/92627425":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/3786436284":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/3784813489":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/38958757":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/544818595":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/1763496588":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/341049398":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/2216461018":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/3268555703":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/3481438563":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/783107677":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/2439246170":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/965177671":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/2516488332":"2024-04-03T09:13:05.404716+0000","172.21.15.89:6833/108855":"2024-04-03T09:12:53.003657+0000","172.21.15.89:6832/108855":"2024-04-03T09:12:53.003657+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:13:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.709+0000 7ff7a1000700 1 -- 172.21.15.89:0/4005291028 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff77c06ab10 msgr2=0x7ff77c06cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.709+0000 7ff7a1000700 1 --2- 172.21.15.89:0/4005291028 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff77c06ab10 0x7ff77c06cfd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009cd0 tx=0x7ff78c009210 comp rx=0 tx=0).stop 2024-04-02T09:13:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.709+0000 7ff7a1000700 1 -- 172.21.15.89:0/4005291028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff79c161590 msgr2=0x7ff79c15f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.709+0000 7ff7a1000700 1 --2- 172.21.15.89:0/4005291028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff79c161590 0x7ff79c15f4d0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7ff79400efc0 tx=0x7ff79400c0c0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.709+0000 7ff7a1000700 1 -- 172.21.15.89:0/4005291028 shutdown_connections 2024-04-02T09:13:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.709+0000 7ff7a1000700 1 --2- 172.21.15.89:0/4005291028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff79c161590 0x7ff79c15f4d0 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.709+0000 7ff7a1000700 1 --2- 172.21.15.89:0/4005291028 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff79c0cf230 0x7ff79c14ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.709+0000 7ff7a1000700 1 --2- 172.21.15.89:0/4005291028 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff77c06ab10 0x7ff77c06cfd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.709+0000 7ff7a1000700 1 --2- 172.21.15.89:0/4005291028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c160f10 0x7ff79c15ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.709+0000 7ff7a1000700 1 -- 172.21.15.89:0/4005291028 >> 172.21.15.89:0/4005291028 conn(0x7ff79c08aff0 msgr2=0x7ff79c0cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.709+0000 7ff7a1000700 1 -- 172.21.15.89:0/4005291028 shutdown_connections 2024-04-02T09:13:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.709+0000 7ff7a1000700 1 -- 172.21.15.89:0/4005291028 wait complete. 2024-04-02T09:13:07.725 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T09:13:07.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.920+0000 7fd2c1e00700 1 Processor -- start 2024-04-02T09:13:07.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.920+0000 7fd2c1e00700 1 -- start start 2024-04-02T09:13:07.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.922+0000 7fd2c1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2bc0cbb30 0x7fd2bc0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.922+0000 7fd2c1e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2bc0cd480 0x7fd2bc06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.922+0000 7fd2c1e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2bc0cc450 con 0x7fd2bc0cd020 2024-04-02T09:13:07.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.922+0000 7fd2c1e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2bc0cdae0 con 0x7fd2bc0cbb30 2024-04-02T09:13:07.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.922+0000 7fd2c1e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc0cddc0 con 0x7fd2bc0cd480 2024-04-02T09:13:07.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.922+0000 7fd2baa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2bc0cbb30 0x7fd2bc0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.922+0000 7fd2baa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2bc0cbb30 0x7fd2bc0cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53012/0 (socket says 172.21.15.89:53012) 2024-04-02T09:13:07.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.922+0000 7fd2baa00700 1 -- 172.21.15.89:0/1005363237 learned_addr learned my addr 172.21.15.89:0/1005363237 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:07.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.922+0000 7fd2baa00700 1 -- 172.21.15.89:0/1005363237 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2bc0cd480 msgr2=0x7fd2bc06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:07.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.922+0000 7fd2bb400700 1 --2- 172.21.15.89:0/1005363237 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2bc0cd480 0x7fd2bc06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.922+0000 7fd2baa00700 1 --2- 172.21.15.89:0/1005363237 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2bc0cd480 0x7fd2bc06e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.922+0000 7fd2baa00700 1 -- 172.21.15.89:0/1005363237 >> v1:172.21.15.113:6789/0 conn(0x7fd2bc0cd020 legacy=0x7fd2bc0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:07.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.922+0000 7fd2baa00700 1 -- 172.21.15.89:0/1005363237 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc0ce0a0 con 0x7fd2bc0cbb30 2024-04-02T09:13:07.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.922+0000 7fd2bb400700 1 --2- 172.21.15.89:0/1005363237 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2bc0cd480 0x7fd2bc06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:07.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.923+0000 7fd2baa00700 1 --2- 172.21.15.89:0/1005363237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0cbb30 0x7fd2bc0cbf10 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0098f0 tx=0x7fd2ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653fd5306e8d219f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:07.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.923+0000 7fd2b9600700 1 -- 172.21.15.89:0/1005363237 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac004230 con 0x7fd2bc0cbb30 2024-04-02T09:13:07.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.923+0000 7fd2b9600700 1 -- 172.21.15.89:0/1005363237 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ac004390 con 0x7fd2bc0cbb30 2024-04-02T09:13:07.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.923+0000 7fd2b9600700 1 -- 172.21.15.89:0/1005363237 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac0036a0 con 0x7fd2bc0cbb30 2024-04-02T09:13:07.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.923+0000 7fd2c1e00700 1 -- 172.21.15.89:0/1005363237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0cbb30 msgr2=0x7fd2bc0cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:07.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.923+0000 7fd2c1e00700 1 --2- 172.21.15.89:0/1005363237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0cbb30 0x7fd2bc0cbf10 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0098f0 tx=0x7fd2ac009c00 comp rx=0 tx=0).stop 2024-04-02T09:13:07.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.923+0000 7fd2c1e00700 1 -- 172.21.15.89:0/1005363237 shutdown_connections 2024-04-02T09:13:07.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.923+0000 7fd2c1e00700 1 --2- 172.21.15.89:0/1005363237 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2bc0cd480 0x7fd2bc06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.923+0000 7fd2c1e00700 1 --2- 172.21.15.89:0/1005363237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0cbb30 0x7fd2bc0cbf10 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.923+0000 7fd2c1e00700 1 -- 172.21.15.89:0/1005363237 >> 172.21.15.89:0/1005363237 conn(0x7fd2bc0b0d70 msgr2=0x7fd2bc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:07.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.923+0000 7fd2c1e00700 1 -- 172.21.15.89:0/1005363237 shutdown_connections 2024-04-02T09:13:07.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.923+0000 7fd2c1e00700 1 -- 172.21.15.89:0/1005363237 wait complete. 2024-04-02T09:13:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.925+0000 7fd2c1e00700 1 Processor -- start 2024-04-02T09:13:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.925+0000 7fd2c1e00700 1 -- start start 2024-04-02T09:13:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.926+0000 7fd2c1e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2bc0cbb30 0x7fd2bc0c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.926+0000 7fd2c1e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2bc0cd020 0x7fd2bc09c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.926+0000 7fd2c1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0cd480 0x7fd2bc157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.926+0000 7fd2c1e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc0991d0 con 0x7fd2bc0cd480 2024-04-02T09:13:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.926+0000 7fd2c1e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc09cc10 con 0x7fd2bc0cd020 2024-04-02T09:13:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.926+0000 7fd2c1e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc09cef0 con 0x7fd2bc0cbb30 2024-04-02T09:13:07.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.926+0000 7fd2bb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0cd480 0x7fd2bc157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.926+0000 7fd2bb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0cd480 0x7fd2bc157280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53028/0 (socket says 172.21.15.89:53028) 2024-04-02T09:13:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.926+0000 7fd2bb400700 1 -- 172.21.15.89:0/2325283745 learned_addr learned my addr 172.21.15.89:0/2325283745 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.926+0000 7fd2ba000700 1 --2- 172.21.15.89:0/2325283745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2bc0cd020 0x7fd2bc09c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.926+0000 7fd2baa00700 1 --2- 172.21.15.89:0/2325283745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2bc0cbb30 0x7fd2bc0c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.927+0000 7fd2bb400700 1 -- 172.21.15.89:0/2325283745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2bc0cbb30 msgr2=0x7fd2bc0c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.927+0000 7fd2bb400700 1 --2- 172.21.15.89:0/2325283745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2bc0cbb30 0x7fd2bc0c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.927+0000 7fd2bb400700 1 -- 172.21.15.89:0/2325283745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2bc0cd020 msgr2=0x7fd2bc09c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.927+0000 7fd2bb400700 1 --2- 172.21.15.89:0/2325283745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2bc0cd020 0x7fd2bc09c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.927+0000 7fd2bb400700 1 -- 172.21.15.89:0/2325283745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac009550 con 0x7fd2bc0cd480 2024-04-02T09:13:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.927+0000 7fd2bb400700 1 --2- 172.21.15.89:0/2325283745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0cd480 0x7fd2bc157280 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400cef0 tx=0x7fd2b400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.927+0000 7fd2b3400700 1 -- 172.21.15.89:0/2325283745 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b400ad00 con 0x7fd2bc0cd480 2024-04-02T09:13:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.927+0000 7fd2b3400700 1 -- 172.21.15.89:0/2325283745 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b400ae60 con 0x7fd2bc0cd480 2024-04-02T09:13:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.927+0000 7fd2b3400700 1 -- 172.21.15.89:0/2325283745 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4015680 con 0x7fd2bc0cd480 2024-04-02T09:13:07.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.927+0000 7fd2c1e00700 1 -- 172.21.15.89:0/2325283745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2bc0993e0 con 0x7fd2bc0cd480 2024-04-02T09:13:07.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.927+0000 7fd2c1e00700 1 -- 172.21.15.89:0/2325283745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2bc09c890 con 0x7fd2bc0cd480 2024-04-02T09:13:07.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.929+0000 7fd2c1e00700 1 -- 172.21.15.89:0/2325283745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2bc099900 con 0x7fd2bc0cd480 2024-04-02T09:13:07.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.930+0000 7fd2b3400700 1 -- 172.21.15.89:0/2325283745 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 156) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fd2b400d430 con 0x7fd2bc0cd480 2024-04-02T09:13:07.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.931+0000 7fd2b3400700 1 --2- 172.21.15.89:0/2325283745 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd29c06aae0 0x7fd29c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:07.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.931+0000 7fd2b3400700 1 -- 172.21.15.89:0/2325283745 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7752+0+0 (secure 0 0 0) 0x7fd2b4089be0 con 0x7fd2bc0cd480 2024-04-02T09:13:07.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.931+0000 7fd2baa00700 1 --2- 172.21.15.89:0/2325283745 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd29c06aae0 0x7fd29c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:07.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.931+0000 7fd2baa00700 1 --2- 172.21.15.89:0/2325283745 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd29c06aae0 0x7fd29c06cfa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac009130 tx=0x7fd2ac008030 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:07.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:07.935+0000 7fd2b3400700 1 -- 172.21.15.89:0/2325283745 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd2bc099900 con 0x7fd2bc0cd480 2024-04-02T09:13:08.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.184+0000 7fd2c1e00700 1 -- 172.21.15.89:0/2325283745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd2bc099900 con 0x7fd2bc0cd480 2024-04-02T09:13:08.183 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T09:13:08.185+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T09:13:08.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.197+0000 7fd2b3400700 1 -- 172.21.15.89:0/2325283745 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v111) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd2bc099900 con 0x7fd2bc0cd480 2024-04-02T09:13:08.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.200+0000 7fd2c1e00700 1 -- 172.21.15.89:0/2325283745 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd29c06aae0 msgr2=0x7fd29c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:08.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.200+0000 7fd2c1e00700 1 --2- 172.21.15.89:0/2325283745 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd29c06aae0 0x7fd29c06cfa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac009130 tx=0x7fd2ac008030 comp rx=0 tx=0).stop 2024-04-02T09:13:08.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.200+0000 7fd2c1e00700 1 -- 172.21.15.89:0/2325283745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0cd480 msgr2=0x7fd2bc157280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:08.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.200+0000 7fd2c1e00700 1 --2- 172.21.15.89:0/2325283745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0cd480 0x7fd2bc157280 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400cef0 tx=0x7fd2b400cf20 comp rx=0 tx=0).stop 2024-04-02T09:13:08.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.200+0000 7fd2c1e00700 1 -- 172.21.15.89:0/2325283745 shutdown_connections 2024-04-02T09:13:08.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.200+0000 7fd2c1e00700 1 --2- 172.21.15.89:0/2325283745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2bc0cd480 0x7fd2bc157280 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:08.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.200+0000 7fd2c1e00700 1 --2- 172.21.15.89:0/2325283745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd2bc0cbb30 0x7fd2bc0c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:08.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.200+0000 7fd2c1e00700 1 --2- 172.21.15.89:0/2325283745 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd29c06aae0 0x7fd29c06cfa0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:08.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.200+0000 7fd2c1e00700 1 --2- 172.21.15.89:0/2325283745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2bc0cd020 0x7fd2bc09c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:08.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.200+0000 7fd2c1e00700 1 -- 172.21.15.89:0/2325283745 >> 172.21.15.89:0/2325283745 conn(0x7fd2bc0b0d70 msgr2=0x7fd2bc06e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:08.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.200+0000 7fd2c1e00700 1 -- 172.21.15.89:0/2325283745 shutdown_connections 2024-04-02T09:13:08.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.200+0000 7fd2c1e00700 1 -- 172.21.15.89:0/2325283745 wait complete. 2024-04-02T09:13:08.200 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T09:13:08.206 INFO:tasks.ceph.mds.a.smithi089.stderr: -5733> 2024-04-02T09:12:30.664+0000 7fef9e800700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T09:13:08.217 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T09:13:08.265 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T09:13:08.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.420+0000 7fe38e600700 1 Processor -- start 2024-04-02T09:13:08.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.420+0000 7fe38e600700 1 -- start start 2024-04-02T09:13:08.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.421+0000 7fe38e600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe38815d9d0 0x7fe3880d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:08.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.421+0000 7fe38e600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe388161420 con 0x7fe3880d2540 2024-04-02T09:13:08.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.421+0000 7fe38e600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe388161700 con 0x7fe3880d16a0 2024-04-02T09:13:08.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.421+0000 7fe38e600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe38815edc0 con 0x7fe38815d9d0 2024-04-02T09:13:08.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.421+0000 7fe387e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe3880d2540 0x7fe3880ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47722/0 (socket says 172.21.15.89:47722) 2024-04-02T09:13:08.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.421+0000 7fe387e00700 1 -- 172.21.15.89:0/3880971100 learned_addr learned my addr 172.21.15.89:0/3880971100 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:08.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.421+0000 7fe387400700 1 --2- 172.21.15.89:0/3880971100 >> v2:172.21.15.113:3300/0 conn(0x7fe38815d9d0 0x7fe3880d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:08.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.422+0000 7fe386000700 1 -- 172.21.15.89:0/3880971100 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253688876 0 0) 0x7fe388161420 con 0x7fe3880d2540 2024-04-02T09:13:08.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.422+0000 7fe386000700 1 -- 172.21.15.89:0/3880971100 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe36c0035b0 con 0x7fe3880d2540 2024-04-02T09:13:08.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.422+0000 7fe387400700 1 -- 172.21.15.89:0/3880971100 >> v1:172.21.15.113:6789/0 conn(0x7fe3880d16a0 legacy=0x7fe3880d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:08.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.422+0000 7fe387400700 1 -- 172.21.15.89:0/3880971100 >> v1:172.21.15.89:6789/0 conn(0x7fe3880d2540 legacy=0x7fe3880ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:08.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.422+0000 7fe387400700 1 -- 172.21.15.89:0/3880971100 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3881619e0 con 0x7fe38815d9d0 2024-04-02T09:13:08.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.422+0000 7fe387400700 1 --2- 172.21.15.89:0/3880971100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe38815d9d0 0x7fe3880d3c30 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe378009840 tx=0x7fe378009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28eccc9e0128acf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:08.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.422+0000 7fe386000700 1 -- 172.21.15.89:0/3880971100 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe378011830 con 0x7fe38815d9d0 2024-04-02T09:13:08.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.423+0000 7fe386000700 1 -- 172.21.15.89:0/3880971100 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe378011990 con 0x7fe38815d9d0 2024-04-02T09:13:08.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.423+0000 7fe386000700 1 -- 172.21.15.89:0/3880971100 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe378011d20 con 0x7fe38815d9d0 2024-04-02T09:13:08.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.423+0000 7fe38e600700 1 -- 172.21.15.89:0/3880971100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe38815d9d0 msgr2=0x7fe3880d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:08.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.423+0000 7fe38e600700 1 --2- 172.21.15.89:0/3880971100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe38815d9d0 0x7fe3880d3c30 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe378009840 tx=0x7fe378009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:08.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.423+0000 7fe38e600700 1 -- 172.21.15.89:0/3880971100 shutdown_connections 2024-04-02T09:13:08.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.423+0000 7fe38e600700 1 --2- 172.21.15.89:0/3880971100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe38815d9d0 0x7fe3880d3c30 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:08.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.423+0000 7fe38e600700 1 -- 172.21.15.89:0/3880971100 >> 172.21.15.89:0/3880971100 conn(0x7fe38808aff0 msgr2=0x7fe3880d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:08.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.423+0000 7fe38e600700 1 -- 172.21.15.89:0/3880971100 shutdown_connections 2024-04-02T09:13:08.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.423+0000 7fe38e600700 1 -- 172.21.15.89:0/3880971100 wait complete. 2024-04-02T09:13:08.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.425+0000 7fe38e600700 1 Processor -- start 2024-04-02T09:13:08.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.425+0000 7fe38e600700 1 -- start start 2024-04-02T09:13:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.426+0000 7fe38e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3880d16a0 0x7fe3880b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.426+0000 7fe38e600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3880d2540 0x7fe38814f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.426+0000 7fe38e600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe38815d9d0 0x7fe38814f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.426+0000 7fe38e600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3880cba80 con 0x7fe38815d9d0 2024-04-02T09:13:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.426+0000 7fe38e600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38814fdd0 con 0x7fe3880d2540 2024-04-02T09:13:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.426+0000 7fe38e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3881e8ca0 con 0x7fe3880d16a0 2024-04-02T09:13:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.426+0000 7fe387e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe38815d9d0 0x7fe38814f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.426+0000 7fe387e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe38815d9d0 0x7fe38814f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53056/0 (socket says 172.21.15.89:53056) 2024-04-02T09:13:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.426+0000 7fe387e00700 1 -- 172.21.15.89:0/3356576917 learned_addr learned my addr 172.21.15.89:0/3356576917 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.426+0000 7fe386a00700 1 --2- 172.21.15.89:0/3356576917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3880d2540 0x7fe38814f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.426+0000 7fe387400700 1 --2- 172.21.15.89:0/3356576917 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3880d16a0 0x7fe3880b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.427+0000 7fe387e00700 1 -- 172.21.15.89:0/3356576917 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3880d16a0 msgr2=0x7fe3880b7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.427+0000 7fe387e00700 1 --2- 172.21.15.89:0/3356576917 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3880d16a0 0x7fe3880b7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.427+0000 7fe387e00700 1 -- 172.21.15.89:0/3356576917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3880d2540 msgr2=0x7fe38814f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.427+0000 7fe387e00700 1 --2- 172.21.15.89:0/3356576917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3880d2540 0x7fe38814f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.427+0000 7fe387e00700 1 -- 172.21.15.89:0/3356576917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3780094a0 con 0x7fe38815d9d0 2024-04-02T09:13:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.427+0000 7fe387e00700 1 --2- 172.21.15.89:0/3356576917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe38815d9d0 0x7fe38814f890 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7fe38000cbb0 tx=0x7fe38000cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.427+0000 7fe37f400700 1 -- 172.21.15.89:0/3356576917 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38000a9e0 con 0x7fe38815d9d0 2024-04-02T09:13:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.427+0000 7fe37f400700 1 -- 172.21.15.89:0/3356576917 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38000ab40 con 0x7fe38815d9d0 2024-04-02T09:13:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.427+0000 7fe38e600700 1 -- 172.21.15.89:0/3356576917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3880cb3c0 con 0x7fe38815d9d0 2024-04-02T09:13:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.427+0000 7fe37f400700 1 -- 172.21.15.89:0/3356576917 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380012400 con 0x7fe38815d9d0 2024-04-02T09:13:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.427+0000 7fe38e600700 1 -- 172.21.15.89:0/3356576917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3880cb070 con 0x7fe38815d9d0 2024-04-02T09:13:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.428+0000 7fe38e600700 1 -- 172.21.15.89:0/3356576917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3880d1ca0 con 0x7fe38815d9d0 2024-04-02T09:13:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.429+0000 7fe37f400700 1 -- 172.21.15.89:0/3356576917 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 157) v1 ==== 89285+0+0 (secure 0 0 0) 0x7fe38001b3b0 con 0x7fe38815d9d0 2024-04-02T09:13:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.430+0000 7fe37f400700 1 --2- 172.21.15.89:0/3356576917 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe36806ae40 0x7fe36806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.430+0000 7fe37f400700 1 -- 172.21.15.89:0/3356576917 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7838+0+0 (secure 0 0 0) 0x7fe380089a40 con 0x7fe38815d9d0 2024-04-02T09:13:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.430+0000 7fe387400700 1 --2- 172.21.15.89:0/3356576917 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe36806ae40 0x7fe36806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.431+0000 7fe387400700 1 --2- 172.21.15.89:0/3356576917 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe36806ae40 0x7fe36806d300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe3780090b0 tx=0x7fe37801c040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.435+0000 7fe37f400700 1 -- 172.21.15.89:0/3356576917 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe38008be60 con 0x7fe38815d9d0 2024-04-02T09:13:08.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:08.683+0000 7fe38e600700 1 -- 172.21.15.89:0/3356576917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe3880cbbc0 con 0x7fe38815d9d0 2024-04-02T09:13:09.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.271+0000 7fe37f400700 1 -- 172.21.15.89:0/3356576917 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v112) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe3880cbbc0 con 0x7fe38815d9d0 2024-04-02T09:13:09.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.275+0000 7fe38e600700 1 -- 172.21.15.89:0/3356576917 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe36806ae40 msgr2=0x7fe36806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:09.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.275+0000 7fe38e600700 1 --2- 172.21.15.89:0/3356576917 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe36806ae40 0x7fe36806d300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe3780090b0 tx=0x7fe37801c040 comp rx=0 tx=0).stop 2024-04-02T09:13:09.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.275+0000 7fe38e600700 1 -- 172.21.15.89:0/3356576917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe38815d9d0 msgr2=0x7fe38814f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:09.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.275+0000 7fe38e600700 1 --2- 172.21.15.89:0/3356576917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe38815d9d0 0x7fe38814f890 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7fe38000cbb0 tx=0x7fe38000cec0 comp rx=0 tx=0).stop 2024-04-02T09:13:09.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.275+0000 7fe38e600700 1 -- 172.21.15.89:0/3356576917 shutdown_connections 2024-04-02T09:13:09.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.275+0000 7fe38e600700 1 --2- 172.21.15.89:0/3356576917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe38815d9d0 0x7fe38814f890 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:09.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.275+0000 7fe38e600700 1 --2- 172.21.15.89:0/3356576917 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe3880d16a0 0x7fe3880b7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:09.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.275+0000 7fe38e600700 1 --2- 172.21.15.89:0/3356576917 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe36806ae40 0x7fe36806d300 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:09.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.275+0000 7fe38e600700 1 --2- 172.21.15.89:0/3356576917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3880d2540 0x7fe38814f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:09.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.275+0000 7fe38e600700 1 -- 172.21.15.89:0/3356576917 >> 172.21.15.89:0/3356576917 conn(0x7fe38808aff0 msgr2=0x7fe3880ceca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:09.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.275+0000 7fe38e600700 1 -- 172.21.15.89:0/3356576917 shutdown_connections 2024-04-02T09:13:09.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.275+0000 7fe38e600700 1 -- 172.21.15.89:0/3356576917 wait complete. 2024-04-02T09:13:09.296 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T09:13:09.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.499+0000 7fd544800700 1 Processor -- start 2024-04-02T09:13:09.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.499+0000 7fd544800700 1 -- start start 2024-04-02T09:13:09.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.500+0000 7fd544800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd53c0ced30 0x7fd53c0d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:09.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.500+0000 7fd544800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53c0ce8a0 con 0x7fd53c0d2710 2024-04-02T09:13:09.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.500+0000 7fd544800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53c0cdc90 con 0x7fd53c15ddf0 2024-04-02T09:13:09.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.500+0000 7fd544800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53c15ec60 con 0x7fd53c0ced30 2024-04-02T09:13:09.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.500+0000 7fd541400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd53c0ced30 0x7fd53c0d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:09.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.500+0000 7fd542800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd53c0d2710 0x7fd53c15f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47572/0 (socket says 172.21.15.89:47572) 2024-04-02T09:13:09.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.500+0000 7fd542800700 1 -- 172.21.15.89:0/4248296882 learned_addr learned my addr 172.21.15.89:0/4248296882 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:09.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.501+0000 7fd541400700 1 -- 172.21.15.89:0/4248296882 >> v1:172.21.15.113:6789/0 conn(0x7fd53c15ddf0 legacy=0x7fd53c15e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:09.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.501+0000 7fd541400700 1 -- 172.21.15.89:0/4248296882 >> v1:172.21.15.89:6789/0 conn(0x7fd53c0d2710 legacy=0x7fd53c15f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:09.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.501+0000 7fd541400700 1 -- 172.21.15.89:0/4248296882 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c0cdf70 con 0x7fd53c0ced30 2024-04-02T09:13:09.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.501+0000 7fd540a00700 1 -- 172.21.15.89:0/4248296882 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049924648 0 0) 0x7fd53c0ce8a0 con 0x7fd53c0d2710 2024-04-02T09:13:09.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.501+0000 7fd541400700 1 --2- 172.21.15.89:0/4248296882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd53c0ced30 0x7fd53c0d21d0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fd52c009840 tx=0x7fd52c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e29fa0873156305 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:09.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.501+0000 7fd540a00700 1 -- 172.21.15.89:0/4248296882 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c011830 con 0x7fd53c0ced30 2024-04-02T09:13:09.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.501+0000 7fd540a00700 1 -- 172.21.15.89:0/4248296882 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c011990 con 0x7fd53c0ced30 2024-04-02T09:13:09.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.501+0000 7fd540a00700 1 -- 172.21.15.89:0/4248296882 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c011d20 con 0x7fd53c0ced30 2024-04-02T09:13:09.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.502+0000 7fd544800700 1 -- 172.21.15.89:0/4248296882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd53c0ced30 msgr2=0x7fd53c0d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:09.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.502+0000 7fd544800700 1 --2- 172.21.15.89:0/4248296882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd53c0ced30 0x7fd53c0d21d0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fd52c009840 tx=0x7fd52c009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:09.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.502+0000 7fd544800700 1 -- 172.21.15.89:0/4248296882 shutdown_connections 2024-04-02T09:13:09.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.502+0000 7fd544800700 1 --2- 172.21.15.89:0/4248296882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd53c0ced30 0x7fd53c0d21d0 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:09.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.502+0000 7fd544800700 1 -- 172.21.15.89:0/4248296882 >> 172.21.15.89:0/4248296882 conn(0x7fd53c08aff0 msgr2=0x7fd53c0d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:09.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.502+0000 7fd544800700 1 -- 172.21.15.89:0/4248296882 shutdown_connections 2024-04-02T09:13:09.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.502+0000 7fd544800700 1 -- 172.21.15.89:0/4248296882 wait complete. 2024-04-02T09:13:09.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.504+0000 7fd544800700 1 Processor -- start 2024-04-02T09:13:09.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.504+0000 7fd544800700 1 -- start start 2024-04-02T09:13:09.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd544800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd53c0ced30 0x7fd53c156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:09.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd544800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd53c0d2710 0x7fd53c15fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:09.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd544800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd53c15ddf0 0x7fd53c160090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:09.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd544800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c1e9790 con 0x7fd53c15ddf0 2024-04-02T09:13:09.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd544800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c1605d0 con 0x7fd53c0ced30 2024-04-02T09:13:09.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd544800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c160820 con 0x7fd53c0d2710 2024-04-02T09:13:09.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd542800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd53c15ddf0 0x7fd53c160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:09.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd542800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd53c15ddf0 0x7fd53c160090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46592/0 (socket says 172.21.15.89:46592) 2024-04-02T09:13:09.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd542800700 1 -- 172.21.15.89:0/2178681866 learned_addr learned my addr 172.21.15.89:0/2178681866 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:09.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd541400700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd53c0d2710 0x7fd53c15fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:09.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd541e00700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd53c0ced30 0x7fd53c156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:09.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd542800700 1 -- 172.21.15.89:0/2178681866 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd53c0d2710 msgr2=0x7fd53c15fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:09.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd542800700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd53c0d2710 0x7fd53c15fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:09.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd542800700 1 -- 172.21.15.89:0/2178681866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd53c0ced30 msgr2=0x7fd53c156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:09.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd542800700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd53c0ced30 0x7fd53c156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:09.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd542800700 1 -- 172.21.15.89:0/2178681866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52c0094a0 con 0x7fd53c15ddf0 2024-04-02T09:13:09.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.505+0000 7fd541e00700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd53c0ced30 0x7fd53c156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:09.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.506+0000 7fd542800700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd53c15ddf0 0x7fd53c160090 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7fd538002230 tx=0x7fd53800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:09.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.506+0000 7fd536000700 1 -- 172.21.15.89:0/2178681866 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53800ac20 con 0x7fd53c15ddf0 2024-04-02T09:13:09.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.506+0000 7fd536000700 1 -- 172.21.15.89:0/2178681866 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53800ada0 con 0x7fd53c15ddf0 2024-04-02T09:13:09.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.506+0000 7fd536000700 1 -- 172.21.15.89:0/2178681866 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd538007310 con 0x7fd53c15ddf0 2024-04-02T09:13:09.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.506+0000 7fd544800700 1 -- 172.21.15.89:0/2178681866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53c1e99e0 con 0x7fd53c15ddf0 2024-04-02T09:13:09.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.506+0000 7fd544800700 1 -- 172.21.15.89:0/2178681866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53c160da0 con 0x7fd53c15ddf0 2024-04-02T09:13:09.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.507+0000 7fd544800700 1 -- 172.21.15.89:0/2178681866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd53c160fc0 con 0x7fd53c15ddf0 2024-04-02T09:13:09.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.508+0000 7fd536000700 1 -- 172.21.15.89:0/2178681866 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 157) v1 ==== 89285+0+0 (secure 0 0 0) 0x7fd538007470 con 0x7fd53c15ddf0 2024-04-02T09:13:09.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.509+0000 7fd536000700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd52006ae40 0x7fd52006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:09.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.509+0000 7fd536000700 1 -- 172.21.15.89:0/2178681866 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7838+0+0 (secure 0 0 0) 0x7fd538089aa0 con 0x7fd53c15ddf0 2024-04-02T09:13:09.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.509+0000 7fd541e00700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd52006ae40 0x7fd52006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:09.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.509+0000 7fd541e00700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd52006ae40 0x7fd52006d300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0b9240 tx=0x7fd530009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:09.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.513+0000 7fd536000700 1 -- 172.21.15.89:0/2178681866 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd538054110 con 0x7fd53c15ddf0 2024-04-02T09:13:09.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:09.714+0000 7fd544800700 1 -- 172.21.15.89:0/2178681866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd53c067720 con 0x7fd53c15ddf0 2024-04-02T09:13:10.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.291+0000 7fd536000700 1 -- 172.21.15.89:0/2178681866 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fd5380544e0 con 0x7fd53c15ddf0 2024-04-02T09:13:10.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.293+0000 7fd536000700 1 -- 172.21.15.89:0/2178681866 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v231) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd5380748c0 con 0x7fd53c15ddf0 2024-04-02T09:13:10.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.295+0000 7fd544800700 1 -- 172.21.15.89:0/2178681866 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd52006ae40 msgr2=0x7fd52006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:10.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.295+0000 7fd544800700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd52006ae40 0x7fd52006d300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0b9240 tx=0x7fd530009040 comp rx=0 tx=0).stop 2024-04-02T09:13:10.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.295+0000 7fd544800700 1 -- 172.21.15.89:0/2178681866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd53c15ddf0 msgr2=0x7fd53c160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:10.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.295+0000 7fd544800700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd53c15ddf0 0x7fd53c160090 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7fd538002230 tx=0x7fd53800a060 comp rx=0 tx=0).stop 2024-04-02T09:13:10.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.295+0000 7fd544800700 1 -- 172.21.15.89:0/2178681866 shutdown_connections 2024-04-02T09:13:10.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.295+0000 7fd544800700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd52006ae40 0x7fd52006d300 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:10.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.295+0000 7fd544800700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd53c0ced30 0x7fd53c156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:10.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.295+0000 7fd544800700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd53c15ddf0 0x7fd53c160090 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:10.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.295+0000 7fd544800700 1 --2- 172.21.15.89:0/2178681866 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd53c0d2710 0x7fd53c15fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:10.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.296+0000 7fd544800700 1 -- 172.21.15.89:0/2178681866 >> 172.21.15.89:0/2178681866 conn(0x7fd53c08aff0 msgr2=0x7fd53c0b6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:10.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.296+0000 7fd544800700 1 -- 172.21.15.89:0/2178681866 shutdown_connections 2024-04-02T09:13:10.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.296+0000 7fd544800700 1 -- 172.21.15.89:0/2178681866 wait complete. 2024-04-02T09:13:10.295 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T09:13:10.313 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T09:13:10.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.510+0000 7fa15ca00700 1 Processor -- start 2024-04-02T09:13:10.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.510+0000 7fa15ca00700 1 -- start start 2024-04-02T09:13:10.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.511+0000 7fa15ca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa158155ac0 0x7fa15815c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:10.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.511+0000 7fa15ca00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa15806c910 0x7fa1580adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:10.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.512+0000 7fa15ca00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15815dcc0 con 0x7fa1580ae4a0 2024-04-02T09:13:10.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.512+0000 7fa15ca00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa15806dd00 con 0x7fa15806c910 2024-04-02T09:13:10.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.512+0000 7fa15ca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15806d0f0 con 0x7fa158155ac0 2024-04-02T09:13:10.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.512+0000 7fa155400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa15806c910 0x7fa1580adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:10.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.512+0000 7fa155e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa158155ac0 0x7fa15815c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:10.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.512+0000 7fa156800700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fa1580ae4a0 0x7fa15815f0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:55462/0 (socket says 172.21.15.89:55462) 2024-04-02T09:13:10.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.512+0000 7fa156800700 1 -- 172.21.15.89:0/170396305 learned_addr learned my addr 172.21.15.89:0/170396305 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:10.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.512+0000 7fa154a00700 1 -- 172.21.15.89:0/170396305 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699078004 0 0) 0x7fa15815dcc0 con 0x7fa1580ae4a0 2024-04-02T09:13:10.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.512+0000 7fa154a00700 1 -- 172.21.15.89:0/170396305 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa13c0035b0 con 0x7fa1580ae4a0 2024-04-02T09:13:10.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.513+0000 7fa155e00700 1 -- 172.21.15.89:0/170396305 >> v2:172.21.15.113:3300/0 conn(0x7fa15806c910 msgr2=0x7fa1580adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.513+0000 7fa155e00700 1 --2- 172.21.15.89:0/170396305 >> v2:172.21.15.113:3300/0 conn(0x7fa15806c910 0x7fa1580adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.513+0000 7fa155e00700 1 -- 172.21.15.89:0/170396305 >> v1:172.21.15.113:6789/0 conn(0x7fa1580ae4a0 legacy=0x7fa15815f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.513+0000 7fa155e00700 1 -- 172.21.15.89:0/170396305 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15806d3d0 con 0x7fa158155ac0 2024-04-02T09:13:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.513+0000 7fa155400700 1 --2- 172.21.15.89:0/170396305 >> v2:172.21.15.113:3300/0 conn(0x7fa15806c910 0x7fa1580adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.513+0000 7fa155e00700 1 --2- 172.21.15.89:0/170396305 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa158155ac0 0x7fa15815c700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa148009840 tx=0x7fa148009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6c25ec92ab2f958 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.513+0000 7fa154a00700 1 -- 172.21.15.89:0/170396305 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148011830 con 0x7fa158155ac0 2024-04-02T09:13:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.513+0000 7fa154a00700 1 -- 172.21.15.89:0/170396305 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa148011990 con 0x7fa158155ac0 2024-04-02T09:13:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.513+0000 7fa154a00700 1 -- 172.21.15.89:0/170396305 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148011d20 con 0x7fa158155ac0 2024-04-02T09:13:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.514+0000 7fa15ca00700 1 -- 172.21.15.89:0/170396305 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa158155ac0 msgr2=0x7fa15815c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.514+0000 7fa15ca00700 1 --2- 172.21.15.89:0/170396305 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa158155ac0 0x7fa15815c700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa148009840 tx=0x7fa148009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.514+0000 7fa15ca00700 1 -- 172.21.15.89:0/170396305 shutdown_connections 2024-04-02T09:13:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.514+0000 7fa15ca00700 1 --2- 172.21.15.89:0/170396305 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa158155ac0 0x7fa15815c700 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.514+0000 7fa15ca00700 1 --2- 172.21.15.89:0/170396305 >> v2:172.21.15.113:3300/0 conn(0x7fa15806c910 0x7fa1580adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.514+0000 7fa15ca00700 1 -- 172.21.15.89:0/170396305 >> 172.21.15.89:0/170396305 conn(0x7fa1580cf210 msgr2=0x7fa1580cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:10.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.514+0000 7fa15ca00700 1 -- 172.21.15.89:0/170396305 shutdown_connections 2024-04-02T09:13:10.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.514+0000 7fa15ca00700 1 -- 172.21.15.89:0/170396305 wait complete. 2024-04-02T09:13:10.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.516+0000 7fa15ca00700 1 Processor -- start 2024-04-02T09:13:10.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.516+0000 7fa15ca00700 1 -- start start 2024-04-02T09:13:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa15ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa15806c910 0x7fa158152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa15ca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa1580ae4a0 0x7fa15814d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa15ca00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa158155ac0 0x7fa158150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa15ca00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15815e690 con 0x7fa15806c910 2024-04-02T09:13:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa15ca00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa158153260 con 0x7fa158155ac0 2024-04-02T09:13:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa15ca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15815dd60 con 0x7fa1580ae4a0 2024-04-02T09:13:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa155e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa15806c910 0x7fa158152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa155e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa15806c910 0x7fa158152d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46594/0 (socket says 172.21.15.89:46594) 2024-04-02T09:13:10.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa155e00700 1 -- 172.21.15.89:0/475345670 learned_addr learned my addr 172.21.15.89:0/475345670 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa155400700 1 --2- 172.21.15.89:0/475345670 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa1580ae4a0 0x7fa15814d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa156800700 1 --2- 172.21.15.89:0/475345670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa158155ac0 0x7fa158150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa155e00700 1 -- 172.21.15.89:0/475345670 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa1580ae4a0 msgr2=0x7fa15814d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa155e00700 1 --2- 172.21.15.89:0/475345670 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa1580ae4a0 0x7fa15814d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa155e00700 1 -- 172.21.15.89:0/475345670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa158155ac0 msgr2=0x7fa158150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa155e00700 1 --2- 172.21.15.89:0/475345670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa158155ac0 0x7fa158150c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.517+0000 7fa155e00700 1 -- 172.21.15.89:0/475345670 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1440094a0 con 0x7fa15806c910 2024-04-02T09:13:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.518+0000 7fa155e00700 1 --2- 172.21.15.89:0/475345670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa15806c910 0x7fa158152d20 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7fa148009fd0 tx=0x7fa148004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.518+0000 7fa14e000700 1 -- 172.21.15.89:0/475345670 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1480036a0 con 0x7fa15806c910 2024-04-02T09:13:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.518+0000 7fa14e000700 1 -- 172.21.15.89:0/475345670 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14801e070 con 0x7fa15806c910 2024-04-02T09:13:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.518+0000 7fa15ca00700 1 -- 172.21.15.89:0/475345670 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1480094a0 con 0x7fa15806c910 2024-04-02T09:13:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.518+0000 7fa14e000700 1 -- 172.21.15.89:0/475345670 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148003a60 con 0x7fa15806c910 2024-04-02T09:13:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.518+0000 7fa15ca00700 1 -- 172.21.15.89:0/475345670 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15815e320 con 0x7fa15806c910 2024-04-02T09:13:10.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.519+0000 7fa15ca00700 1 -- 172.21.15.89:0/475345670 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa158067720 con 0x7fa15806c910 2024-04-02T09:13:10.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.520+0000 7fa14e000700 1 -- 172.21.15.89:0/475345670 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa148003bc0 con 0x7fa15806c910 2024-04-02T09:13:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.522+0000 7fa14e000700 1 --2- 172.21.15.89:0/475345670 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa13806aed0 0x7fa13806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.522+0000 7fa14e000700 1 -- 172.21.15.89:0/475345670 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7400+0+0 (secure 0 0 0) 0x7fa14808ac90 con 0x7fa15806c910 2024-04-02T09:13:10.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.522+0000 7fa155400700 1 --2- 172.21.15.89:0/475345670 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa13806aed0 0x7fa13806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:10.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.523+0000 7fa155400700 1 --2- 172.21.15.89:0/475345670 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa13806aed0 0x7fa13806d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa15814bf90 tx=0x7fa144008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:10.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.526+0000 7fa14e000700 1 -- 172.21.15.89:0/475345670 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa148055500 con 0x7fa15806c910 2024-04-02T09:13:10.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:10.731+0000 7fa15ca00700 1 -- 172.21.15.89:0/475345670 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa15806e950 con 0x7fa15806c910 2024-04-02T09:13:11.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.288+0000 7fa14e000700 1 -- 172.21.15.89:0/475345670 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v232) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa148058b20 con 0x7fa15806c910 2024-04-02T09:13:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.292+0000 7fa15ca00700 1 -- 172.21.15.89:0/475345670 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa13806aed0 msgr2=0x7fa13806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.292+0000 7fa15ca00700 1 --2- 172.21.15.89:0/475345670 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa13806aed0 0x7fa13806d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa15814bf90 tx=0x7fa144008040 comp rx=0 tx=0).stop 2024-04-02T09:13:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.292+0000 7fa15ca00700 1 -- 172.21.15.89:0/475345670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa15806c910 msgr2=0x7fa158152d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.292+0000 7fa15ca00700 1 --2- 172.21.15.89:0/475345670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa15806c910 0x7fa158152d20 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7fa148009fd0 tx=0x7fa148004750 comp rx=0 tx=0).stop 2024-04-02T09:13:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.292+0000 7fa15ca00700 1 -- 172.21.15.89:0/475345670 shutdown_connections 2024-04-02T09:13:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.292+0000 7fa15ca00700 1 --2- 172.21.15.89:0/475345670 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa13806aed0 0x7fa13806d390 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.292+0000 7fa15ca00700 1 --2- 172.21.15.89:0/475345670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa158155ac0 0x7fa158150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.292+0000 7fa15ca00700 1 --2- 172.21.15.89:0/475345670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa15806c910 0x7fa158152d20 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.292+0000 7fa15ca00700 1 --2- 172.21.15.89:0/475345670 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa1580ae4a0 0x7fa15814d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.292+0000 7fa15ca00700 1 -- 172.21.15.89:0/475345670 >> 172.21.15.89:0/475345670 conn(0x7fa1580cf210 msgr2=0x7fa15815a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.292+0000 7fa15ca00700 1 -- 172.21.15.89:0/475345670 shutdown_connections 2024-04-02T09:13:11.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.293+0000 7fa15ca00700 1 -- 172.21.15.89:0/475345670 wait complete. 2024-04-02T09:13:11.292 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T09:13:11.310 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:13:11.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.514+0000 7f2551e00700 1 Processor -- start 2024-04-02T09:13:11.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.514+0000 7f2551e00700 1 -- start start 2024-04-02T09:13:11.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.515+0000 7f2551e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f254c160f10 0x7f254c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:11.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.515+0000 7f2551e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f254c161590 0x7f254c161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:11.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.515+0000 7f2551e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254c0ceda0 con 0x7f254c0cf230 2024-04-02T09:13:11.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.515+0000 7f2551e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254c161f80 con 0x7f254c160f10 2024-04-02T09:13:11.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.515+0000 7f2551e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254c1612f0 con 0x7f254c161590 2024-04-02T09:13:11.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.516+0000 7f254aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f254c160f10 0x7f254c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:11.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.516+0000 7f254aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f254c160f10 0x7f254c0d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46606/0 (socket says 172.21.15.89:46606) 2024-04-02T09:13:11.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.516+0000 7f254aa00700 1 -- 172.21.15.89:0/4294634516 learned_addr learned my addr 172.21.15.89:0/4294634516 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:11.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.516+0000 7f254b400700 1 --2- 172.21.15.89:0/4294634516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f254c161590 0x7f254c161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:11.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.516+0000 7f254aa00700 1 -- 172.21.15.89:0/4294634516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f254c161590 msgr2=0x7f254c161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:11.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.516+0000 7f254aa00700 1 --2- 172.21.15.89:0/4294634516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f254c161590 0x7f254c161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.516+0000 7f254aa00700 1 -- 172.21.15.89:0/4294634516 >> v1:172.21.15.113:6789/0 conn(0x7f254c0cf230 legacy=0x7f254c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:11.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.516+0000 7f254aa00700 1 -- 172.21.15.89:0/4294634516 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c0ce470 con 0x7f254c160f10 2024-04-02T09:13:11.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.516+0000 7f254aa00700 1 --2- 172.21.15.89:0/4294634516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f254c160f10 0x7f254c0d18b0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f253c0098f0 tx=0x7f253c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d11a1f56cf9d0ef server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:11.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.517+0000 7f2549600700 1 -- 172.21.15.89:0/4294634516 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253c00b8a0 con 0x7f254c160f10 2024-04-02T09:13:11.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.517+0000 7f2549600700 1 -- 172.21.15.89:0/4294634516 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253c00ba00 con 0x7f254c160f10 2024-04-02T09:13:11.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.517+0000 7f2549600700 1 -- 172.21.15.89:0/4294634516 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253c00bd90 con 0x7f254c160f10 2024-04-02T09:13:11.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.517+0000 7f2551e00700 1 -- 172.21.15.89:0/4294634516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f254c160f10 msgr2=0x7f254c0d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:11.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.517+0000 7f2551e00700 1 --2- 172.21.15.89:0/4294634516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f254c160f10 0x7f254c0d18b0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f253c0098f0 tx=0x7f253c009c00 comp rx=0 tx=0).stop 2024-04-02T09:13:11.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.517+0000 7f2551e00700 1 -- 172.21.15.89:0/4294634516 shutdown_connections 2024-04-02T09:13:11.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.517+0000 7f2551e00700 1 --2- 172.21.15.89:0/4294634516 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f254c161590 0x7f254c161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.517+0000 7f2551e00700 1 --2- 172.21.15.89:0/4294634516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f254c160f10 0x7f254c0d18b0 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.517+0000 7f2551e00700 1 -- 172.21.15.89:0/4294634516 >> 172.21.15.89:0/4294634516 conn(0x7f254c08aff0 msgr2=0x7f254c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:11.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.517+0000 7f2551e00700 1 -- 172.21.15.89:0/4294634516 shutdown_connections 2024-04-02T09:13:11.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.517+0000 7f2551e00700 1 -- 172.21.15.89:0/4294634516 wait complete. 2024-04-02T09:13:11.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.519+0000 7f2551e00700 1 Processor -- start 2024-04-02T09:13:11.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.519+0000 7f2551e00700 1 -- start start 2024-04-02T09:13:11.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.520+0000 7f2551e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f254c0cf230 0x7f254c155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:11.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.520+0000 7f2551e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f254c160f10 0x7f254c156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:11.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.520+0000 7f2551e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f254c161590 0x7f254c158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.520+0000 7f2551e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c157dd0 con 0x7f254c0cf230 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.520+0000 7f2551e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c157150 con 0x7f254c160f10 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.520+0000 7f2551e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254c157430 con 0x7f254c161590 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.520+0000 7f254aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f254c0cf230 0x7f254c155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.520+0000 7f254aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f254c0cf230 0x7f254c155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46618/0 (socket says 172.21.15.89:46618) 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.520+0000 7f254aa00700 1 -- 172.21.15.89:0/225113666 learned_addr learned my addr 172.21.15.89:0/225113666 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.520+0000 7f254a000700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f254c160f10 0x7f254c156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.520+0000 7f254b400700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f254c161590 0x7f254c158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.521+0000 7f254aa00700 1 -- 172.21.15.89:0/225113666 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f254c161590 msgr2=0x7f254c158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.521+0000 7f254aa00700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f254c161590 0x7f254c158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.521+0000 7f254aa00700 1 -- 172.21.15.89:0/225113666 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f254c160f10 msgr2=0x7f254c156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.521+0000 7f254aa00700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f254c160f10 0x7f254c156650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.521+0000 7f254aa00700 1 -- 172.21.15.89:0/225113666 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c009550 con 0x7f254c0cf230 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.521+0000 7f254b400700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f254c161590 0x7f254c158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.521+0000 7f254a000700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f254c160f10 0x7f254c156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:11.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.521+0000 7f254aa00700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f254c0cf230 0x7f254c155a40 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f253c009c40 tx=0x7f253c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:11.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.521+0000 7f2543400700 1 -- 172.21.15.89:0/225113666 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253c0036a0 con 0x7f254c0cf230 2024-04-02T09:13:11.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.521+0000 7f2543400700 1 -- 172.21.15.89:0/225113666 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253c01e070 con 0x7f254c0cf230 2024-04-02T09:13:11.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.521+0000 7f2543400700 1 -- 172.21.15.89:0/225113666 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253c003a40 con 0x7f254c0cf230 2024-04-02T09:13:11.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.521+0000 7f2551e00700 1 -- 172.21.15.89:0/225113666 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254c157710 con 0x7f254c0cf230 2024-04-02T09:13:11.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.521+0000 7f2551e00700 1 -- 172.21.15.89:0/225113666 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f254c15fcf0 con 0x7f254c0cf230 2024-04-02T09:13:11.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.522+0000 7f2551e00700 1 -- 172.21.15.89:0/225113666 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f254c067720 con 0x7f254c0cf230 2024-04-02T09:13:11.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.523+0000 7f2543400700 1 -- 172.21.15.89:0/225113666 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f253c003ba0 con 0x7f254c0cf230 2024-04-02T09:13:11.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.524+0000 7f2543400700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f252c06aed0 0x7f252c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:11.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.524+0000 7f2543400700 1 -- 172.21.15.89:0/225113666 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6986+0+0 (secure 0 0 0) 0x7f253c020070 con 0x7f254c0cf230 2024-04-02T09:13:11.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.524+0000 7f254a000700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f252c06aed0 0x7f252c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:11.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.525+0000 7f254a000700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f252c06aed0 0x7f252c06d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f254c160920 tx=0x7f2538009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:11.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.528+0000 7f2543400700 1 -- 172.21.15.89:0/225113666 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f253c055ab0 con 0x7f254c0cf230 2024-04-02T09:13:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.730+0000 7f2551e00700 1 -- 172.21.15.89:0/225113666 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f254c0d2ab0 con 0x7f254c0cf230 2024-04-02T09:13:11.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.732+0000 7f2543400700 1 -- 172.21.15.89:0/225113666 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+17989 (secure 0 0 0) 0x7f253c015090 con 0x7f254c0cf230 2024-04-02T09:13:11.730 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:11.731 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":232,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:13:11.284499+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:12:37.240568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:12:38.093999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6837/2615927215":"2024-04-03T09:13:08.185338+0000","172.21.15.89:0/1531440110":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/92627425":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/3786436284":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/3784813489":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/38958757":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/544818595":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/1763496588":"2024-04-03T09:13:05.404716+0000","172.21.15.89:6835/2615927215":"2024-04-03T09:13:08.185338+0000","172.21.15.89:0/341049398":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/2216461018":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/3268555703":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/3481438563":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/783107677":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/2439246170":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/965177671":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/2516488332":"2024-04-03T09:13:05.404716+0000","172.21.15.89:6833/108855":"2024-04-03T09:12:53.003657+0000","172.21.15.89:6832/108855":"2024-04-03T09:12:53.003657+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:13:11.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.734+0000 7f2551e00700 1 -- 172.21.15.89:0/225113666 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f252c06aed0 msgr2=0x7f252c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:11.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.734+0000 7f2551e00700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f252c06aed0 0x7f252c06d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f254c160920 tx=0x7f2538009040 comp rx=0 tx=0).stop 2024-04-02T09:13:11.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.734+0000 7f2551e00700 1 -- 172.21.15.89:0/225113666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f254c0cf230 msgr2=0x7f254c155a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:11.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.734+0000 7f2551e00700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f254c0cf230 0x7f254c155a40 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f253c009c40 tx=0x7f253c004750 comp rx=0 tx=0).stop 2024-04-02T09:13:11.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.734+0000 7f2551e00700 1 -- 172.21.15.89:0/225113666 shutdown_connections 2024-04-02T09:13:11.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.734+0000 7f2551e00700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f254c161590 0x7f254c158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.734+0000 7f2551e00700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f254c0cf230 0x7f254c155a40 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.734+0000 7f2551e00700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f252c06aed0 0x7f252c06d390 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.734+0000 7f2551e00700 1 --2- 172.21.15.89:0/225113666 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f254c160f10 0x7f254c156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.734+0000 7f2551e00700 1 -- 172.21.15.89:0/225113666 >> 172.21.15.89:0/225113666 conn(0x7f254c08aff0 msgr2=0x7f254c0ca9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:11.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.734+0000 7f2551e00700 1 -- 172.21.15.89:0/225113666 shutdown_connections 2024-04-02T09:13:11.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.735+0000 7f2551e00700 1 -- 172.21.15.89:0/225113666 wait complete. 2024-04-02T09:13:11.751 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-02T09:12:37.240568+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-02T09:12:38.093999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:13:11.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:13:11.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.947+0000 7f27c2a00700 1 Processor -- start 2024-04-02T09:13:11.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.948+0000 7f27c2a00700 1 -- start start 2024-04-02T09:13:11.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.949+0000 7f27c2a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f27bc0d16a0 0x7f27bc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:11.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.949+0000 7f27c2a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d2370 0x7f27bc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:11.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.949+0000 7f27c2a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc161420 con 0x7f27bc15d9d0 2024-04-02T09:13:11.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.949+0000 7f27c2a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc0d2820 con 0x7f27bc0d16a0 2024-04-02T09:13:11.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.949+0000 7f27c2a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0d4340 con 0x7f27bc0d2370 2024-04-02T09:13:11.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.950+0000 7f27bbe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f27bc15d9d0 0x7f27bc0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47584/0 (socket says 172.21.15.89:47584) 2024-04-02T09:13:11.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.950+0000 7f27bbe00700 1 -- 172.21.15.89:0/3682627063 learned_addr learned my addr 172.21.15.89:0/3682627063 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:11.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.950+0000 7f27bb400700 1 --2- 172.21.15.89:0/3682627063 >> v2:172.21.15.89:3300/0 conn(0x7f27bc0d16a0 0x7f27bc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:11.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.950+0000 7f27c0a00700 1 --2- 172.21.15.89:0/3682627063 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d2370 0x7f27bc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:11.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.950+0000 7f27baa00700 1 -- 172.21.15.89:0/3682627063 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107538395 0 0) 0x7f27bc161420 con 0x7f27bc15d9d0 2024-04-02T09:13:11.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.950+0000 7f27baa00700 1 -- 172.21.15.89:0/3682627063 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a00035b0 con 0x7f27bc15d9d0 2024-04-02T09:13:11.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.950+0000 7f27bb400700 1 -- 172.21.15.89:0/3682627063 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d2370 msgr2=0x7f27bc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:11.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.950+0000 7f27bb400700 1 --2- 172.21.15.89:0/3682627063 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d2370 0x7f27bc0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.950+0000 7f27bb400700 1 -- 172.21.15.89:0/3682627063 >> v1:172.21.15.89:6789/0 conn(0x7f27bc15d9d0 legacy=0x7f27bc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:11.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.950+0000 7f27bb400700 1 -- 172.21.15.89:0/3682627063 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc161700 con 0x7f27bc0d16a0 2024-04-02T09:13:11.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.950+0000 7f27bb400700 1 --2- 172.21.15.89:0/3682627063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27bc0d16a0 0x7f27bc0d1e30 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f27a8009820 tx=0x7f27a8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c57732e3139debb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:11.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.951+0000 7f27baa00700 1 -- 172.21.15.89:0/3682627063 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a8005e30 con 0x7f27bc0d16a0 2024-04-02T09:13:11.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.951+0000 7f27baa00700 1 -- 172.21.15.89:0/3682627063 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a8006030 con 0x7f27bc0d16a0 2024-04-02T09:13:11.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.951+0000 7f27baa00700 1 -- 172.21.15.89:0/3682627063 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a80063a0 con 0x7f27bc0d16a0 2024-04-02T09:13:11.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.951+0000 7f27c2a00700 1 -- 172.21.15.89:0/3682627063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27bc0d16a0 msgr2=0x7f27bc0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:11.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.951+0000 7f27c2a00700 1 --2- 172.21.15.89:0/3682627063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27bc0d16a0 0x7f27bc0d1e30 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f27a8009820 tx=0x7f27a8009b30 comp rx=0 tx=0).stop 2024-04-02T09:13:11.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.951+0000 7f27c2a00700 1 -- 172.21.15.89:0/3682627063 shutdown_connections 2024-04-02T09:13:11.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.951+0000 7f27c2a00700 1 --2- 172.21.15.89:0/3682627063 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d2370 0x7f27bc0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.951+0000 7f27c2a00700 1 --2- 172.21.15.89:0/3682627063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27bc0d16a0 0x7f27bc0d1e30 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.951+0000 7f27c2a00700 1 -- 172.21.15.89:0/3682627063 >> 172.21.15.89:0/3682627063 conn(0x7f27bc08aff0 msgr2=0x7f27bc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:11.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.951+0000 7f27c2a00700 1 -- 172.21.15.89:0/3682627063 shutdown_connections 2024-04-02T09:13:11.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.951+0000 7f27c2a00700 1 -- 172.21.15.89:0/3682627063 wait complete. 2024-04-02T09:13:11.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.954+0000 7f27c2a00700 1 Processor -- start 2024-04-02T09:13:11.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.954+0000 7f27c2a00700 1 -- start start 2024-04-02T09:13:11.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.955+0000 7f27c2a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27bc0d16a0 0x7f27bc1f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:11.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.955+0000 7f27c2a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d2370 0x7f27bc15fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:11.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.955+0000 7f27c2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27bc15d9d0 0x7f27bc160400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:11.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.955+0000 7f27c2a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc15f660 con 0x7f27bc15d9d0 2024-04-02T09:13:11.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.955+0000 7f27c2a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc160d10 con 0x7f27bc0d16a0 2024-04-02T09:13:11.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.955+0000 7f27c2a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc160a30 con 0x7f27bc0d2370 2024-04-02T09:13:11.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.955+0000 7f27c0a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27bc15d9d0 0x7f27bc160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:11.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.955+0000 7f27c0a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27bc15d9d0 0x7f27bc160400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46634/0 (socket says 172.21.15.89:46634) 2024-04-02T09:13:11.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.955+0000 7f27c0a00700 1 -- 172.21.15.89:0/848256183 learned_addr learned my addr 172.21.15.89:0/848256183 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:11.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.956+0000 7f27bbe00700 1 --2- 172.21.15.89:0/848256183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27bc0d16a0 0x7f27bc1f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:11.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.956+0000 7f27bb400700 1 --2- 172.21.15.89:0/848256183 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d2370 0x7f27bc15fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:11.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.956+0000 7f27c0a00700 1 -- 172.21.15.89:0/848256183 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d2370 msgr2=0x7f27bc15fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:11.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.956+0000 7f27c0a00700 1 --2- 172.21.15.89:0/848256183 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d2370 0x7f27bc15fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.956+0000 7f27c0a00700 1 -- 172.21.15.89:0/848256183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27bc0d16a0 msgr2=0x7f27bc1f2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:11.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.956+0000 7f27c0a00700 1 --2- 172.21.15.89:0/848256183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27bc0d16a0 0x7f27bc1f2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:11.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.956+0000 7f27c0a00700 1 -- 172.21.15.89:0/848256183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a8009430 con 0x7f27bc15d9d0 2024-04-02T09:13:11.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.956+0000 7f27c0a00700 1 --2- 172.21.15.89:0/848256183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27bc15d9d0 0x7f27bc160400 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f27b4007e40 tx=0x7f27b400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:11.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.956+0000 7f27b3e00700 1 -- 172.21.15.89:0/848256183 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b400ed30 con 0x7f27bc15d9d0 2024-04-02T09:13:11.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.956+0000 7f27b3e00700 1 -- 172.21.15.89:0/848256183 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b4014420 con 0x7f27bc15d9d0 2024-04-02T09:13:11.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.956+0000 7f27b3e00700 1 -- 172.21.15.89:0/848256183 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b40147d0 con 0x7f27bc15d9d0 2024-04-02T09:13:11.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.956+0000 7f27c2a00700 1 -- 172.21.15.89:0/848256183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc15efa0 con 0x7f27bc15d9d0 2024-04-02T09:13:11.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.957+0000 7f27c2a00700 1 -- 172.21.15.89:0/848256183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc14fd20 con 0x7f27bc15d9d0 2024-04-02T09:13:11.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.958+0000 7f27c2a00700 1 -- 172.21.15.89:0/848256183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27bc0d1ca0 con 0x7f27bc15d9d0 2024-04-02T09:13:11.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.958+0000 7f27b3e00700 1 -- 172.21.15.89:0/848256183 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f27b4014930 con 0x7f27bc15d9d0 2024-04-02T09:13:11.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.960+0000 7f27b3e00700 1 --2- 172.21.15.89:0/848256183 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f279c06aed0 0x7f279c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:11.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.960+0000 7f27b3e00700 1 -- 172.21.15.89:0/848256183 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6986+0+0 (secure 0 0 0) 0x7f27b408a240 con 0x7f27bc15d9d0 2024-04-02T09:13:11.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.960+0000 7f27bbe00700 1 --2- 172.21.15.89:0/848256183 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f279c06aed0 0x7f279c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:11.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.961+0000 7f27bbe00700 1 --2- 172.21.15.89:0/848256183 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f279c06aed0 0x7f279c06d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f27ac007fd0 tx=0x7f27ac009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:11.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:11.964+0000 7f27b3e00700 1 -- 172.21.15.89:0/848256183 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f27b4054c80 con 0x7f27bc15d9d0 2024-04-02T09:13:12.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.165+0000 7f27c2a00700 1 -- 172.21.15.89:0/848256183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f27bc15ee00 con 0x7f27bc15d9d0 2024-04-02T09:13:12.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.166+0000 7f27b3e00700 1 -- 172.21.15.89:0/848256183 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v232) v1 ==== 93+0+10 (secure 0 0 0) 0x7f27bc15ee00 con 0x7f27bc15d9d0 2024-04-02T09:13:12.164 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:13:12.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.168+0000 7f27c2a00700 1 -- 172.21.15.89:0/848256183 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f279c06aed0 msgr2=0x7f279c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:12.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.168+0000 7f27c2a00700 1 --2- 172.21.15.89:0/848256183 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f279c06aed0 0x7f279c06d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f27ac007fd0 tx=0x7f27ac009040 comp rx=0 tx=0).stop 2024-04-02T09:13:12.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.168+0000 7f27c2a00700 1 -- 172.21.15.89:0/848256183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27bc15d9d0 msgr2=0x7f27bc160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:12.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.168+0000 7f27c2a00700 1 --2- 172.21.15.89:0/848256183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27bc15d9d0 0x7f27bc160400 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f27b4007e40 tx=0x7f27b400e3c0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.168+0000 7f27c2a00700 1 -- 172.21.15.89:0/848256183 shutdown_connections 2024-04-02T09:13:12.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.168+0000 7f27c2a00700 1 --2- 172.21.15.89:0/848256183 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f279c06aed0 0x7f279c06d390 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.168+0000 7f27c2a00700 1 --2- 172.21.15.89:0/848256183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27bc0d16a0 0x7f27bc1f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.168+0000 7f27c2a00700 1 --2- 172.21.15.89:0/848256183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27bc15d9d0 0x7f27bc160400 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.168+0000 7f27c2a00700 1 --2- 172.21.15.89:0/848256183 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d2370 0x7f27bc15fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.168+0000 7f27c2a00700 1 -- 172.21.15.89:0/848256183 >> 172.21.15.89:0/848256183 conn(0x7f27bc08aff0 msgr2=0x7f27bc0b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:12.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.169+0000 7f27c2a00700 1 -- 172.21.15.89:0/848256183 shutdown_connections 2024-04-02T09:13:12.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.169+0000 7f27c2a00700 1 -- 172.21.15.89:0/848256183 wait complete. 2024-04-02T09:13:12.184 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-02T09:13:12.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.384+0000 7f65fc000700 1 Processor -- start 2024-04-02T09:13:12.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.384+0000 7f65fc000700 1 -- start start 2024-04-02T09:13:12.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.385+0000 7f65fc000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65f415d8a0 0x7f65f40b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:12.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.385+0000 7f65fc000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f65f40d3eb0 0x7f65f406b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:12.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.385+0000 7f65fc000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f40ced20 con 0x7f65f406bdd0 2024-04-02T09:13:12.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.385+0000 7f65fc000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f40d4340 con 0x7f65f40d3eb0 2024-04-02T09:13:12.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.385+0000 7f65fc000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f40ce6d0 con 0x7f65f415d8a0 2024-04-02T09:13:12.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.385+0000 7f65fa000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f65f406bdd0 0x7f65f40cf1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47596/0 (socket says 172.21.15.89:47596) 2024-04-02T09:13:12.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.385+0000 7f65fa000700 1 -- 172.21.15.89:0/2918470311 learned_addr learned my addr 172.21.15.89:0/2918470311 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:12.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.385+0000 7f65f8c00700 1 --2- 172.21.15.89:0/2918470311 >> v2:172.21.15.113:3300/0 conn(0x7f65f40d3eb0 0x7f65f406b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:12.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.385+0000 7f65f9600700 1 --2- 172.21.15.89:0/2918470311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65f415d8a0 0x7f65f40b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:12.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.385+0000 7f65f3e00700 1 -- 172.21.15.89:0/2918470311 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211273981 0 0) 0x7f65f40ced20 con 0x7f65f406bdd0 2024-04-02T09:13:12.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 -- 172.21.15.89:0/2918470311 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65dc0035b0 con 0x7f65f406bdd0 2024-04-02T09:13:12.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 -- 172.21.15.89:0/2918470311 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3284465512 0 0) 0x7f65dc0035b0 con 0x7f65f406bdd0 2024-04-02T09:13:12.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 -- 172.21.15.89:0/2918470311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65f415d8a0 msgr2=0x7f65f40b2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:12.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 --2- 172.21.15.89:0/2918470311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65f415d8a0 0x7f65f40b2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 -- 172.21.15.89:0/2918470311 >> v2:172.21.15.113:3300/0 conn(0x7f65f40d3eb0 msgr2=0x7f65f406b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 --2- 172.21.15.89:0/2918470311 >> v2:172.21.15.113:3300/0 conn(0x7f65f40d3eb0 0x7f65f406b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 -- 172.21.15.89:0/2918470311 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f40ce9b0 con 0x7f65f406bdd0 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 -- 172.21.15.89:0/2918470311 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f65ec0019f0 con 0x7f65f406bdd0 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 -- 172.21.15.89:0/2918470311 >> v1:172.21.15.89:6789/0 conn(0x7f65f406bdd0 legacy=0x7f65f40cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 --2- 172.21.15.89:0/2918470311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65dc003a00 0x7f65dc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 --2- 172.21.15.89:0/2918470311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65dc004850 0x7f65dc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 --2- 172.21.15.89:0/2918470311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65f415d8a0 0x7f65dc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 -- 172.21.15.89:0/2918470311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f40d4340 con 0x7f65dc004850 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 -- 172.21.15.89:0/2918470311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f40ce6d0 con 0x7f65dc003a00 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f3e00700 1 -- 172.21.15.89:0/2918470311 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc00ba10 con 0x7f65f415d8a0 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f8c00700 1 --2- 172.21.15.89:0/2918470311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65dc004850 0x7f65dc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65f9600700 1 --2- 172.21.15.89:0/2918470311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65dc003a00 0x7f65dc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.386+0000 7f65fa000700 1 --2- 172.21.15.89:0/2918470311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65f415d8a0 0x7f65dc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.387+0000 7f65f8c00700 1 -- 172.21.15.89:0/2918470311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65f415d8a0 msgr2=0x7f65dc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:12.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.387+0000 7f65f8c00700 1 --2- 172.21.15.89:0/2918470311 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65f415d8a0 0x7f65dc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.387+0000 7f65f8c00700 1 -- 172.21.15.89:0/2918470311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65dc003a00 msgr2=0x7f65dc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.387+0000 7f65f8c00700 1 --2- 172.21.15.89:0/2918470311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65dc003a00 0x7f65dc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.387+0000 7f65f8c00700 1 -- 172.21.15.89:0/2918470311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc0035b0 con 0x7f65dc004850 2024-04-02T09:13:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.387+0000 7f65f8c00700 1 --2- 172.21.15.89:0/2918470311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65dc004850 0x7f65dc004cd0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009b40 tx=0x7f65dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e234a853d55f1ccd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.387+0000 7f65f3e00700 1 -- 172.21.15.89:0/2918470311 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0005a20 con 0x7f65dc004850 2024-04-02T09:13:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.387+0000 7f65f3e00700 1 -- 172.21.15.89:0/2918470311 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e0005b80 con 0x7f65dc004850 2024-04-02T09:13:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.387+0000 7f65f3e00700 1 -- 172.21.15.89:0/2918470311 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e000e6c0 con 0x7f65dc004850 2024-04-02T09:13:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.388+0000 7f65fc000700 1 -- 172.21.15.89:0/2918470311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65dc004850 msgr2=0x7f65dc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.388+0000 7f65fc000700 1 --2- 172.21.15.89:0/2918470311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65dc004850 0x7f65dc004cd0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009b40 tx=0x7f65dc005880 comp rx=0 tx=0).stop 2024-04-02T09:13:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.388+0000 7f65fa000700 1 -- 172.21.15.89:0/2918470311 reap_dead start 2024-04-02T09:13:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.388+0000 7f65fc000700 1 -- 172.21.15.89:0/2918470311 shutdown_connections 2024-04-02T09:13:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.388+0000 7f65fc000700 1 -- 172.21.15.89:0/2918470311 >> 172.21.15.89:0/2918470311 conn(0x7f65f408aff0 msgr2=0x7f65f40d3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.388+0000 7f65fc000700 1 -- 172.21.15.89:0/2918470311 shutdown_connections 2024-04-02T09:13:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.388+0000 7f65fc000700 1 -- 172.21.15.89:0/2918470311 wait complete. 2024-04-02T09:13:12.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.389+0000 7f65fc000700 1 Processor -- start 2024-04-02T09:13:12.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.390+0000 7f65fc000700 1 -- start start 2024-04-02T09:13:12.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.390+0000 7f65fc000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65dc004850 0x7f65f40a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:12.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.390+0000 7f65fc000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65f409e1e0 0x7f65f40c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:12.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.390+0000 7f65fc000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65f4152660 0x7f65f40cc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:12.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.390+0000 7f65fc000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f40a1210 con 0x7f65dc004850 2024-04-02T09:13:12.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.390+0000 7f65fc000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f409f990 con 0x7f65f409e1e0 2024-04-02T09:13:12.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.390+0000 7f65fc000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f409fc30 con 0x7f65f4152660 2024-04-02T09:13:12.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.391+0000 7f65f9600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65dc004850 0x7f65f40a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:12.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.391+0000 7f65f9600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65dc004850 0x7f65f40a0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46658/0 (socket says 172.21.15.89:46658) 2024-04-02T09:13:12.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.391+0000 7f65f9600700 1 -- 172.21.15.89:0/2322115556 learned_addr learned my addr 172.21.15.89:0/2322115556 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:12.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.391+0000 7f65fa000700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65f4152660 0x7f65f40cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:12.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.391+0000 7f65f8c00700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65f409e1e0 0x7f65f40c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:12.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.391+0000 7f65f9600700 1 -- 172.21.15.89:0/2322115556 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65f4152660 msgr2=0x7f65f40cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:12.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.391+0000 7f65f9600700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65f4152660 0x7f65f40cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.391+0000 7f65f9600700 1 -- 172.21.15.89:0/2322115556 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65f409e1e0 msgr2=0x7f65f40c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:12.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.391+0000 7f65f9600700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65f409e1e0 0x7f65f40c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.391+0000 7f65f9600700 1 -- 172.21.15.89:0/2322115556 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f409ff10 con 0x7f65dc004850 2024-04-02T09:13:12.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.391+0000 7f65fa000700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65f4152660 0x7f65f40cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:12.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.391+0000 7f65f8c00700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65f409e1e0 0x7f65f40c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:12.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.391+0000 7f65f9600700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65dc004850 0x7f65f40a0cd0 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f65e80119a0 tx=0x7f65e8011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:12.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.392+0000 7f65f1600700 1 -- 172.21.15.89:0/2322115556 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e800f940 con 0x7f65dc004850 2024-04-02T09:13:12.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.392+0000 7f65f1600700 1 -- 172.21.15.89:0/2322115556 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e800faa0 con 0x7f65dc004850 2024-04-02T09:13:12.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.392+0000 7f65f1600700 1 -- 172.21.15.89:0/2322115556 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e800fe30 con 0x7f65dc004850 2024-04-02T09:13:12.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.392+0000 7f65fc000700 1 -- 172.21.15.89:0/2322115556 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f40a01f0 con 0x7f65dc004850 2024-04-02T09:13:12.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.392+0000 7f65fc000700 1 -- 172.21.15.89:0/2322115556 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f40a07b0 con 0x7f65dc004850 2024-04-02T09:13:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.393+0000 7f65fc000700 1 -- 172.21.15.89:0/2322115556 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f40a09a0 con 0x7f65dc004850 2024-04-02T09:13:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.394+0000 7f65f1600700 1 -- 172.21.15.89:0/2322115556 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f65e8014440 con 0x7f65dc004850 2024-04-02T09:13:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.394+0000 7f65f1600700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f65d406af00 0x7f65d406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.394+0000 7f65f1600700 1 -- 172.21.15.89:0/2322115556 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6986+0+0 (secure 0 0 0) 0x7f65e808a2a0 con 0x7f65dc004850 2024-04-02T09:13:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.395+0000 7f65f8c00700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f65d406af00 0x7f65d406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.395+0000 7f65f8c00700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f65d406af00 0x7f65d406d3c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f65f40d0730 tx=0x7f65e0008030 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.399+0000 7f65f1600700 1 -- 172.21.15.89:0/2322115556 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f65e8054c30 con 0x7f65dc004850 2024-04-02T09:13:12.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.598+0000 7f65fc000700 1 -- 172.21.15.89:0/2322115556 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f65f4067720 con 0x7f65dc004850 2024-04-02T09:13:12.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.599+0000 7f65f1600700 1 -- 172.21.15.89:0/2322115556 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v232) v1 ==== 107+0+11 (secure 0 0 0) 0x7f65e8058250 con 0x7f65dc004850 2024-04-02T09:13:12.597 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:13:12.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.601+0000 7f65fc000700 1 -- 172.21.15.89:0/2322115556 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f65d406af00 msgr2=0x7f65d406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:12.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.601+0000 7f65fc000700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f65d406af00 0x7f65d406d3c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f65f40d0730 tx=0x7f65e0008030 comp rx=0 tx=0).stop 2024-04-02T09:13:12.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.601+0000 7f65fc000700 1 -- 172.21.15.89:0/2322115556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65dc004850 msgr2=0x7f65f40a0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:12.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.601+0000 7f65fc000700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65dc004850 0x7f65f40a0cd0 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f65e80119a0 tx=0x7f65e8011d60 comp rx=0 tx=0).stop 2024-04-02T09:13:12.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.601+0000 7f65fc000700 1 -- 172.21.15.89:0/2322115556 shutdown_connections 2024-04-02T09:13:12.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.601+0000 7f65fc000700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65f4152660 0x7f65f40cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.601+0000 7f65fc000700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65dc004850 0x7f65f40a0cd0 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.601+0000 7f65fc000700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f65d406af00 0x7f65d406d3c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.601+0000 7f65fc000700 1 --2- 172.21.15.89:0/2322115556 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65f409e1e0 0x7f65f40c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.601+0000 7f65fc000700 1 -- 172.21.15.89:0/2322115556 >> 172.21.15.89:0/2322115556 conn(0x7f65f408aff0 msgr2=0x7f65f40b8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:12.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.601+0000 7f65fc000700 1 -- 172.21.15.89:0/2322115556 shutdown_connections 2024-04-02T09:13:12.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.601+0000 7f65fc000700 1 -- 172.21.15.89:0/2322115556 wait complete. 2024-04-02T09:13:12.617 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-02T09:13:12.780 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:13:12.784 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:13:12.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.815+0000 7f8033600700 1 Processor -- start 2024-04-02T09:13:12.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.815+0000 7f8033600700 1 -- start start 2024-04-02T09:13:12.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.816+0000 7f8033600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f802c0cee10 0x7f802c161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:12.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.816+0000 7f8033600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802c0ce270 con 0x7f802c1611b0 2024-04-02T09:13:12.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.816+0000 7f8033600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802c15b600 con 0x7f802c0d21f0 2024-04-02T09:13:12.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.816+0000 7f8033600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f802c0d2760 con 0x7f802c0cee10 2024-04-02T09:13:12.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.816+0000 7f8030c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f802c1611b0 0x7f802c15ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47600/0 (socket says 172.21.15.89:47600) 2024-04-02T09:13:12.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.816+0000 7f8030c00700 1 -- 172.21.15.89:0/2486860959 learned_addr learned my addr 172.21.15.89:0/2486860959 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:12.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.817+0000 7f8031600700 1 --2- 172.21.15.89:0/2486860959 >> v2:172.21.15.113:3300/0 conn(0x7f802c0cee10 0x7f802c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:12.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.817+0000 7f802b400700 1 -- 172.21.15.89:0/2486860959 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802974068 0 0) 0x7f802c0ce270 con 0x7f802c1611b0 2024-04-02T09:13:12.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.817+0000 7f802b400700 1 -- 172.21.15.89:0/2486860959 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80140035b0 con 0x7f802c1611b0 2024-04-02T09:13:12.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.817+0000 7f802b400700 1 -- 172.21.15.89:0/2486860959 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790631482 0 0) 0x7f802c15b600 con 0x7f802c0d21f0 2024-04-02T09:13:12.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.817+0000 7f802b400700 1 -- 172.21.15.89:0/2486860959 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f802c0ce270 con 0x7f802c0d21f0 2024-04-02T09:13:12.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.817+0000 7f8031600700 1 -- 172.21.15.89:0/2486860959 >> v1:172.21.15.113:6789/0 conn(0x7f802c0d21f0 legacy=0x7f802c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:12.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.817+0000 7f8031600700 1 -- 172.21.15.89:0/2486860959 >> v1:172.21.15.89:6789/0 conn(0x7f802c1611b0 legacy=0x7f802c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:12.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.817+0000 7f802b400700 1 -- 172.21.15.89:0/2486860959 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 994283320 0 0) 0x7f80140035b0 con 0x7f802c1611b0 2024-04-02T09:13:12.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.817+0000 7f8031600700 1 -- 172.21.15.89:0/2486860959 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c0d2a40 con 0x7f802c0cee10 2024-04-02T09:13:12.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.818+0000 7f8031600700 1 --2- 172.21.15.89:0/2486860959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f802c0cee10 0x7f802c161590 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8024009d90 tx=0x7f802400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b87afed6c3f87992 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:12.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.818+0000 7f802b400700 1 -- 172.21.15.89:0/2486860959 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802400e070 con 0x7f802c0cee10 2024-04-02T09:13:12.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.818+0000 7f802b400700 1 -- 172.21.15.89:0/2486860959 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8024003a20 con 0x7f802c0cee10 2024-04-02T09:13:12.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.818+0000 7f802b400700 1 -- 172.21.15.89:0/2486860959 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8020001670 con 0x7f802c0cee10 2024-04-02T09:13:12.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.818+0000 7f8033600700 1 -- 172.21.15.89:0/2486860959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f802c0cee10 msgr2=0x7f802c161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:12.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.818+0000 7f8033600700 1 --2- 172.21.15.89:0/2486860959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f802c0cee10 0x7f802c161590 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8024009d90 tx=0x7f802400b330 comp rx=0 tx=0).stop 2024-04-02T09:13:12.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.818+0000 7f8033600700 1 -- 172.21.15.89:0/2486860959 shutdown_connections 2024-04-02T09:13:12.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.818+0000 7f8033600700 1 --2- 172.21.15.89:0/2486860959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f802c0cee10 0x7f802c161590 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.818+0000 7f8033600700 1 -- 172.21.15.89:0/2486860959 >> 172.21.15.89:0/2486860959 conn(0x7f802c08aff0 msgr2=0x7f802c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:12.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.819+0000 7f8033600700 1 -- 172.21.15.89:0/2486860959 shutdown_connections 2024-04-02T09:13:12.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.819+0000 7f8033600700 1 -- 172.21.15.89:0/2486860959 wait complete. 2024-04-02T09:13:12.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.820+0000 7f8033600700 1 Processor -- start 2024-04-02T09:13:12.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.821+0000 7f8033600700 1 -- start start 2024-04-02T09:13:12.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.822+0000 7f8033600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cee10 0x7f802c157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:12.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.822+0000 7f8033600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f802c0d21f0 0x7f802c0c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.822+0000 7f8033600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c1611b0 0x7f802c0b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.822+0000 7f8033600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c0c7d80 con 0x7f802c0cee10 2024-04-02T09:13:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.822+0000 7f8033600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c0c81a0 con 0x7f802c0d21f0 2024-04-02T09:13:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.822+0000 7f8033600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802c0c7100 con 0x7f802c1611b0 2024-04-02T09:13:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.822+0000 7f8030c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cee10 0x7f802c157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.822+0000 7f8030c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cee10 0x7f802c157410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46660/0 (socket says 172.21.15.89:46660) 2024-04-02T09:13:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.822+0000 7f8030c00700 1 -- 172.21.15.89:0/4215723580 learned_addr learned my addr 172.21.15.89:0/4215723580 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.822+0000 7f802be00700 1 --2- 172.21.15.89:0/4215723580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f802c0d21f0 0x7f802c0c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.822+0000 7f8031600700 1 --2- 172.21.15.89:0/4215723580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c1611b0 0x7f802c0b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.823+0000 7f8030c00700 1 -- 172.21.15.89:0/4215723580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c1611b0 msgr2=0x7f802c0b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.823+0000 7f8030c00700 1 --2- 172.21.15.89:0/4215723580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c1611b0 0x7f802c0b8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.823+0000 7f8030c00700 1 -- 172.21.15.89:0/4215723580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f802c0d21f0 msgr2=0x7f802c0c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.823+0000 7f8030c00700 1 --2- 172.21.15.89:0/4215723580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f802c0d21f0 0x7f802c0c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:12.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.823+0000 7f8030c00700 1 -- 172.21.15.89:0/4215723580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8024009a70 con 0x7f802c0cee10 2024-04-02T09:13:12.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.823+0000 7f8030c00700 1 --2- 172.21.15.89:0/4215723580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cee10 0x7f802c157410 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f802000c6b0 tx=0x7f802000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:12.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.823+0000 7f8028c00700 1 -- 172.21.15.89:0/4215723580 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8020004c80 con 0x7f802c0cee10 2024-04-02T09:13:12.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.823+0000 7f8028c00700 1 -- 172.21.15.89:0/4215723580 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8020004de0 con 0x7f802c0cee10 2024-04-02T09:13:12.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.823+0000 7f8033600700 1 -- 172.21.15.89:0/4215723580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802c0c73e0 con 0x7f802c0cee10 2024-04-02T09:13:12.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.823+0000 7f8028c00700 1 -- 172.21.15.89:0/4215723580 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80200072f0 con 0x7f802c0cee10 2024-04-02T09:13:12.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.823+0000 7f8033600700 1 -- 172.21.15.89:0/4215723580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802c0c79a0 con 0x7f802c0cee10 2024-04-02T09:13:12.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.825+0000 7f8033600700 1 -- 172.21.15.89:0/4215723580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802c0d0d80 con 0x7f802c0cee10 2024-04-02T09:13:12.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.826+0000 7f8028c00700 1 -- 172.21.15.89:0/4215723580 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f8020007450 con 0x7f802c0cee10 2024-04-02T09:13:12.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.826+0000 7f8028c00700 1 --2- 172.21.15.89:0/4215723580 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f800c06aed0 0x7f800c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:12.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.826+0000 7f8028c00700 1 -- 172.21.15.89:0/4215723580 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6986+0+0 (secure 0 0 0) 0x7f8020089950 con 0x7f802c0cee10 2024-04-02T09:13:12.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.827+0000 7f802be00700 1 --2- 172.21.15.89:0/4215723580 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f800c06aed0 0x7f800c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:12.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.827+0000 7f802be00700 1 --2- 172.21.15.89:0/4215723580 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f800c06aed0 0x7f800c06d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f802c0c85e0 tx=0x7f8018009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:12.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:12.831+0000 7f8028c00700 1 -- 172.21.15.89:0/4215723580 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8020054300 con 0x7f802c0cee10 2024-04-02T09:13:13.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.033+0000 7f8033600700 1 -- 172.21.15.89:0/4215723580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f802c067720 con 0x7f802c0cee10 2024-04-02T09:13:13.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.033+0000 7f8028c00700 1 -- 172.21.15.89:0/4215723580 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v232) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8020007700 con 0x7f802c0cee10 2024-04-02T09:13:13.032 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:13:13.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.035+0000 7f8033600700 1 -- 172.21.15.89:0/4215723580 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f800c06aed0 msgr2=0x7f800c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:13.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.035+0000 7f8033600700 1 --2- 172.21.15.89:0/4215723580 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f800c06aed0 0x7f800c06d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f802c0c85e0 tx=0x7f8018009040 comp rx=0 tx=0).stop 2024-04-02T09:13:13.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.035+0000 7f8033600700 1 -- 172.21.15.89:0/4215723580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cee10 msgr2=0x7f802c157410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:13.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.035+0000 7f8033600700 1 --2- 172.21.15.89:0/4215723580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cee10 0x7f802c157410 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f802000c6b0 tx=0x7f802000c9c0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.035+0000 7f8033600700 1 -- 172.21.15.89:0/4215723580 shutdown_connections 2024-04-02T09:13:13.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.036+0000 7f8033600700 1 --2- 172.21.15.89:0/4215723580 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c1611b0 0x7f802c0b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.036+0000 7f8033600700 1 --2- 172.21.15.89:0/4215723580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cee10 0x7f802c157410 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.036+0000 7f8033600700 1 --2- 172.21.15.89:0/4215723580 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f800c06aed0 0x7f800c06d390 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.036+0000 7f8033600700 1 --2- 172.21.15.89:0/4215723580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f802c0d21f0 0x7f802c0c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.036+0000 7f8033600700 1 -- 172.21.15.89:0/4215723580 >> 172.21.15.89:0/4215723580 conn(0x7f802c08aff0 msgr2=0x7f802c06bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:13.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.036+0000 7f8033600700 1 -- 172.21.15.89:0/4215723580 shutdown_connections 2024-04-02T09:13:13.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.036+0000 7f8033600700 1 -- 172.21.15.89:0/4215723580 wait complete. 2024-04-02T09:13:13.052 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:13:13.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.245+0000 7effe2e00700 1 Processor -- start 2024-04-02T09:13:13.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.245+0000 7effe2e00700 1 -- start start 2024-04-02T09:13:13.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.246+0000 7effe2e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effdc1619d0 0x7effdc1685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:13.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.246+0000 7effe2e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effdc0d30d0 con 0x7effdc0ceec0 2024-04-02T09:13:13.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.246+0000 7effe2e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effdc0b6840 con 0x7effdc15d640 2024-04-02T09:13:13.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.246+0000 7effe2e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effdc0d2b30 con 0x7effdc1619d0 2024-04-02T09:13:13.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.247+0000 7effdb400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7effdc0ceec0 0x7effdc1612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47616/0 (socket says 172.21.15.89:47616) 2024-04-02T09:13:13.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.247+0000 7effdb400700 1 -- 172.21.15.89:0/578859515 learned_addr learned my addr 172.21.15.89:0/578859515 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:13.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.247+0000 7effe0e00700 1 --2- 172.21.15.89:0/578859515 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effdc1619d0 0x7effdc1685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:13.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.247+0000 7effdaa00700 1 -- 172.21.15.89:0/578859515 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809231499 0 0) 0x7effdc0d30d0 con 0x7effdc0ceec0 2024-04-02T09:13:13.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.247+0000 7effdaa00700 1 -- 172.21.15.89:0/578859515 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc00035b0 con 0x7effdc0ceec0 2024-04-02T09:13:13.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.247+0000 7effdaa00700 1 -- 172.21.15.89:0/578859515 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533700923 0 0) 0x7effdc0b6840 con 0x7effdc15d640 2024-04-02T09:13:13.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.247+0000 7effdaa00700 1 -- 172.21.15.89:0/578859515 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effdc0d30d0 con 0x7effdc15d640 2024-04-02T09:13:13.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.247+0000 7effe0e00700 1 -- 172.21.15.89:0/578859515 >> v1:172.21.15.113:6789/0 conn(0x7effdc15d640 legacy=0x7effdc0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:13.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.247+0000 7effe0e00700 1 -- 172.21.15.89:0/578859515 >> v1:172.21.15.89:6789/0 conn(0x7effdc0ceec0 legacy=0x7effdc1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:13.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.247+0000 7effe0e00700 1 -- 172.21.15.89:0/578859515 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc0d2df0 con 0x7effdc1619d0 2024-04-02T09:13:13.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.248+0000 7effe0e00700 1 --2- 172.21.15.89:0/578859515 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effdc1619d0 0x7effdc1685f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7effd4009dc0 tx=0x7effd400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54801f1dfcf6b60 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:13.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.248+0000 7effdaa00700 1 -- 172.21.15.89:0/578859515 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd400e070 con 0x7effdc1619d0 2024-04-02T09:13:13.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.248+0000 7effdaa00700 1 -- 172.21.15.89:0/578859515 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd400bb20 con 0x7effdc1619d0 2024-04-02T09:13:13.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.248+0000 7effdaa00700 1 -- 172.21.15.89:0/578859515 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4012430 con 0x7effdc1619d0 2024-04-02T09:13:13.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.248+0000 7effe2e00700 1 -- 172.21.15.89:0/578859515 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effdc1619d0 msgr2=0x7effdc1685f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:13.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.248+0000 7effe2e00700 1 --2- 172.21.15.89:0/578859515 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effdc1619d0 0x7effdc1685f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7effd4009dc0 tx=0x7effd400b330 comp rx=0 tx=0).stop 2024-04-02T09:13:13.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.249+0000 7effe2e00700 1 -- 172.21.15.89:0/578859515 shutdown_connections 2024-04-02T09:13:13.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.249+0000 7effe2e00700 1 --2- 172.21.15.89:0/578859515 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effdc1619d0 0x7effdc1685f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.249+0000 7effe2e00700 1 -- 172.21.15.89:0/578859515 >> 172.21.15.89:0/578859515 conn(0x7effdc08aff0 msgr2=0x7effdc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:13.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.249+0000 7effe2e00700 1 -- 172.21.15.89:0/578859515 shutdown_connections 2024-04-02T09:13:13.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.249+0000 7effe2e00700 1 -- 172.21.15.89:0/578859515 wait complete. 2024-04-02T09:13:13.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.250+0000 7effe2e00700 1 Processor -- start 2024-04-02T09:13:13.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.251+0000 7effe2e00700 1 -- start start 2024-04-02T09:13:13.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.251+0000 7effe2e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effdc0ceec0 0x7effdc155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:13.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.251+0000 7effe2e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effdc15d640 0x7effdc156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:13.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effe2e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effdc1619d0 0x7effdc15f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effe2e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc158210 con 0x7effdc0ceec0 2024-04-02T09:13:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effe2e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc1f32d0 con 0x7effdc1619d0 2024-04-02T09:13:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effe2e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effdc157690 con 0x7effdc15d640 2024-04-02T09:13:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effdbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effdc0ceec0 0x7effdc155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effdbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effdc0ceec0 0x7effdc155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46668/0 (socket says 172.21.15.89:46668) 2024-04-02T09:13:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effdbe00700 1 -- 172.21.15.89:0/3776910843 learned_addr learned my addr 172.21.15.89:0/3776910843 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effdbe00700 1 -- 172.21.15.89:0/3776910843 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effdc15d640 msgr2=0x7effdc156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effdb400700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effdc15d640 0x7effdc156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effe0e00700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effdc1619d0 0x7effdc15f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effdbe00700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effdc15d640 0x7effdc156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effdbe00700 1 -- 172.21.15.89:0/3776910843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effdc1619d0 msgr2=0x7effdc15f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effdbe00700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effdc1619d0 0x7effdc15f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effdbe00700 1 -- 172.21.15.89:0/3776910843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd4009a70 con 0x7effdc0ceec0 2024-04-02T09:13:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effdb400700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effdc15d640 0x7effdc156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effe0e00700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effdc1619d0 0x7effdc15f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.252+0000 7effdbe00700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effdc0ceec0 0x7effdc155850 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7effcc00c520 tx=0x7effcc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.253+0000 7effd3e00700 1 -- 172.21.15.89:0/3776910843 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc0052d0 con 0x7effdc0ceec0 2024-04-02T09:13:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.253+0000 7effd3e00700 1 -- 172.21.15.89:0/3776910843 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effcc005430 con 0x7effdc0ceec0 2024-04-02T09:13:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.253+0000 7effd3e00700 1 -- 172.21.15.89:0/3776910843 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc0038c0 con 0x7effdc0ceec0 2024-04-02T09:13:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.253+0000 7effe2e00700 1 -- 172.21.15.89:0/3776910843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effdc157970 con 0x7effdc0ceec0 2024-04-02T09:13:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.253+0000 7effe2e00700 1 -- 172.21.15.89:0/3776910843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effdc157f30 con 0x7effdc0ceec0 2024-04-02T09:13:13.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.254+0000 7effe2e00700 1 -- 172.21.15.89:0/3776910843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effdc0d3340 con 0x7effdc0ceec0 2024-04-02T09:13:13.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.255+0000 7effd3e00700 1 -- 172.21.15.89:0/3776910843 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7effcc003a20 con 0x7effdc0ceec0 2024-04-02T09:13:13.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.256+0000 7effd3e00700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7effbc06aed0 0x7effbc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:13.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.256+0000 7effd3e00700 1 -- 172.21.15.89:0/3776910843 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6986+0+0 (secure 0 0 0) 0x7effcc089140 con 0x7effdc0ceec0 2024-04-02T09:13:13.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.256+0000 7effdb400700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7effbc06aed0 0x7effbc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:13.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.256+0000 7effdb400700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7effbc06aed0 0x7effbc06d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7effc80011e0 tx=0x7effc8009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:13.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.260+0000 7effd3e00700 1 -- 172.21.15.89:0/3776910843 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7effcc053b80 con 0x7effdc0ceec0 2024-04-02T09:13:13.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.506+0000 7effe2e00700 1 -- 172.21.15.89:0/3776910843 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effdc067720 con 0x7effdc0ceec0 2024-04-02T09:13:13.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.507+0000 7effd3e00700 1 -- 172.21.15.89:0/3776910843 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4758 (secure 0 0 0) 0x7effcc0571a0 con 0x7effdc0ceec0 2024-04-02T09:13:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:13.505 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":112,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8969,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3650347444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3650347444},{"type":"v1","addr":"172.21.15.89:6836","nonce":3650347444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2152500685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2152500685},{"type":"v1","addr":"172.21.15.113:6834","nonce":2152500685}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":11784,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2785458438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2785458438},{"type":"v1","addr":"172.21.15.89:6837","nonce":2785458438}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:39.106317+0000","modified":"2024-04-02T09:12:42.112504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10279":{"gid":10279,"name":"d","rank":0,"incarnation":107,"state":"up:active","state_seq":6,"addr":"172.21.15.113:6835/2457040702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2457040702},{"type":"v1","addr":"172.21.15.113:6835","nonce":2457040702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-02T09:13:13.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.509+0000 7effe2e00700 1 -- 172.21.15.89:0/3776910843 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7effbc06aed0 msgr2=0x7effbc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:13.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.509+0000 7effe2e00700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7effbc06aed0 0x7effbc06d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7effc80011e0 tx=0x7effc8009040 comp rx=0 tx=0).stop 2024-04-02T09:13:13.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.509+0000 7effe2e00700 1 -- 172.21.15.89:0/3776910843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effdc0ceec0 msgr2=0x7effdc155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:13.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.509+0000 7effe2e00700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effdc0ceec0 0x7effdc155850 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7effcc00c520 tx=0x7effcc00c8e0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.509+0000 7effe2e00700 1 -- 172.21.15.89:0/3776910843 shutdown_connections 2024-04-02T09:13:13.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.509+0000 7effe2e00700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7effbc06aed0 0x7effbc06d390 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.509+0000 7effe2e00700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effdc1619d0 0x7effdc15f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.509+0000 7effe2e00700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effdc0ceec0 0x7effdc155850 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.509+0000 7effe2e00700 1 --2- 172.21.15.89:0/3776910843 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effdc15d640 0x7effdc156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.509+0000 7effe2e00700 1 -- 172.21.15.89:0/3776910843 >> 172.21.15.89:0/3776910843 conn(0x7effdc08aff0 msgr2=0x7effdc06c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:13.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.509+0000 7effe2e00700 1 -- 172.21.15.89:0/3776910843 shutdown_connections 2024-04-02T09:13:13.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.509+0000 7effe2e00700 1 -- 172.21.15.89:0/3776910843 wait complete. 2024-04-02T09:13:13.509 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 112 2024-04-02T09:13:13.525 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:13:13.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.717+0000 7f5b10a00700 1 Processor -- start 2024-04-02T09:13:13.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.717+0000 7f5b10a00700 1 -- start start 2024-04-02T09:13:13.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.719+0000 7f5b10a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b0c15d710 0x7f5b0c0d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:13.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.719+0000 7f5b10a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5b0c06c650 0x7f5b0c0d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:13.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.719+0000 7f5b10a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b0c06bb40 con 0x7f5b0c09aba0 2024-04-02T09:13:13.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.719+0000 7f5b10a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b0c099b00 con 0x7f5b0c15d710 2024-04-02T09:13:13.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.719+0000 7f5b10a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b0c099de0 con 0x7f5b0c06c650 2024-04-02T09:13:13.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.719+0000 7f5b09e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b0c15d710 0x7f5b0c0d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:13.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.719+0000 7f5b09e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b0c15d710 0x7f5b0c0d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46678/0 (socket says 172.21.15.89:46678) 2024-04-02T09:13:13.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.719+0000 7f5b09e00700 1 -- 172.21.15.89:0/3178232078 learned_addr learned my addr 172.21.15.89:0/3178232078 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:13.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.719+0000 7f5b0a800700 1 --2- 172.21.15.89:0/3178232078 >> v2:172.21.15.113:3300/0 conn(0x7f5b0c06c650 0x7f5b0c0d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.719+0000 7f5b09e00700 1 -- 172.21.15.89:0/3178232078 >> v2:172.21.15.113:3300/0 conn(0x7f5b0c06c650 msgr2=0x7f5b0c0d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.719+0000 7f5b09e00700 1 --2- 172.21.15.89:0/3178232078 >> v2:172.21.15.113:3300/0 conn(0x7f5b0c06c650 0x7f5b0c0d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.719+0000 7f5b09e00700 1 -- 172.21.15.89:0/3178232078 >> v1:172.21.15.113:6789/0 conn(0x7f5b0c09aba0 legacy=0x7f5b0c06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.719+0000 7f5b09e00700 1 -- 172.21.15.89:0/3178232078 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0c09a0c0 con 0x7f5b0c15d710 2024-04-02T09:13:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.720+0000 7f5b09e00700 1 --2- 172.21.15.89:0/3178232078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b0c15d710 0x7f5b0c0d0130 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c0722c0 tx=0x7f5afc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12263434cbefee6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.720+0000 7f5b08a00700 1 -- 172.21.15.89:0/3178232078 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc004230 con 0x7f5b0c15d710 2024-04-02T09:13:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.720+0000 7f5b08a00700 1 -- 172.21.15.89:0/3178232078 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5afc004390 con 0x7f5b0c15d710 2024-04-02T09:13:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.720+0000 7f5b08a00700 1 -- 172.21.15.89:0/3178232078 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc0036a0 con 0x7f5b0c15d710 2024-04-02T09:13:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.720+0000 7f5b10a00700 1 -- 172.21.15.89:0/3178232078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b0c15d710 msgr2=0x7f5b0c0d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.720+0000 7f5b10a00700 1 --2- 172.21.15.89:0/3178232078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b0c15d710 0x7f5b0c0d0130 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c0722c0 tx=0x7f5afc009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.720+0000 7f5b10a00700 1 -- 172.21.15.89:0/3178232078 shutdown_connections 2024-04-02T09:13:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.720+0000 7f5b10a00700 1 --2- 172.21.15.89:0/3178232078 >> v2:172.21.15.113:3300/0 conn(0x7f5b0c06c650 0x7f5b0c0d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.720+0000 7f5b10a00700 1 --2- 172.21.15.89:0/3178232078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b0c15d710 0x7f5b0c0d0130 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.721+0000 7f5b10a00700 1 -- 172.21.15.89:0/3178232078 >> 172.21.15.89:0/3178232078 conn(0x7f5b0c0d13e0 msgr2=0x7f5b0c0d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.721+0000 7f5b10a00700 1 -- 172.21.15.89:0/3178232078 shutdown_connections 2024-04-02T09:13:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.721+0000 7f5b10a00700 1 -- 172.21.15.89:0/3178232078 wait complete. 2024-04-02T09:13:13.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.722+0000 7f5b10a00700 1 Processor -- start 2024-04-02T09:13:13.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.722+0000 7f5b10a00700 1 -- start start 2024-04-02T09:13:13.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.723+0000 7f5b10a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b0c06c650 0x7f5b0c15de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:13.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.723+0000 7f5b10a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b0c09aba0 0x7f5b0c09eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:13.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.723+0000 7f5b10a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b0c15d710 0x7f5b0c0c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:13.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.723+0000 7f5b10a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c0b62c0 con 0x7f5b0c06c650 2024-04-02T09:13:13.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.723+0000 7f5b10a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c15e460 con 0x7f5b0c09aba0 2024-04-02T09:13:13.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.723+0000 7f5b10a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c15e740 con 0x7f5b0c15d710 2024-04-02T09:13:13.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.723+0000 7f5b09e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b0c06c650 0x7f5b0c15de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:13.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.723+0000 7f5b09e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b0c06c650 0x7f5b0c15de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46692/0 (socket says 172.21.15.89:46692) 2024-04-02T09:13:13.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.723+0000 7f5b09e00700 1 -- 172.21.15.89:0/4286638146 learned_addr learned my addr 172.21.15.89:0/4286638146 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:13.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.724+0000 7f5b09400700 1 --2- 172.21.15.89:0/4286638146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b0c09aba0 0x7f5b0c09eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:13.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.724+0000 7f5b0a800700 1 --2- 172.21.15.89:0/4286638146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b0c15d710 0x7f5b0c0c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:13.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.724+0000 7f5b09e00700 1 -- 172.21.15.89:0/4286638146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b0c15d710 msgr2=0x7f5b0c0c70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:13.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.724+0000 7f5b09e00700 1 --2- 172.21.15.89:0/4286638146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b0c15d710 0x7f5b0c0c70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.724+0000 7f5b09e00700 1 -- 172.21.15.89:0/4286638146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b0c09aba0 msgr2=0x7f5b0c09eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.724+0000 7f5b09e00700 1 --2- 172.21.15.89:0/4286638146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b0c09aba0 0x7f5b0c09eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.724+0000 7f5b09e00700 1 -- 172.21.15.89:0/4286638146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc009550 con 0x7f5b0c06c650 2024-04-02T09:13:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.724+0000 7f5b09e00700 1 --2- 172.21.15.89:0/4286638146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b0c06c650 0x7f5b0c15de80 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f5afc009c10 tx=0x7f5afc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.724+0000 7f5b02000700 1 -- 172.21.15.89:0/4286638146 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc011630 con 0x7f5b0c06c650 2024-04-02T09:13:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.724+0000 7f5b02000700 1 -- 172.21.15.89:0/4286638146 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5afc011790 con 0x7f5b0c06c650 2024-04-02T09:13:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.724+0000 7f5b10a00700 1 -- 172.21.15.89:0/4286638146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b0c15ea20 con 0x7f5b0c06c650 2024-04-02T09:13:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.724+0000 7f5b02000700 1 -- 172.21.15.89:0/4286638146 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc011b40 con 0x7f5b0c06c650 2024-04-02T09:13:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.725+0000 7f5b10a00700 1 -- 172.21.15.89:0/4286638146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b0c09f0e0 con 0x7f5b0c06c650 2024-04-02T09:13:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.725+0000 7f5b10a00700 1 -- 172.21.15.89:0/4286638146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b0c067720 con 0x7f5b0c06c650 2024-04-02T09:13:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.727+0000 7f5b02000700 1 -- 172.21.15.89:0/4286638146 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5afc01b440 con 0x7f5b0c06c650 2024-04-02T09:13:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.728+0000 7f5b02000700 1 --2- 172.21.15.89:0/4286638146 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5aec06aed0 0x7f5aec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.728+0000 7f5b02000700 1 -- 172.21.15.89:0/4286638146 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6986+0+0 (secure 0 0 0) 0x7f5afc08b1e0 con 0x7f5b0c06c650 2024-04-02T09:13:13.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.728+0000 7f5b09400700 1 --2- 172.21.15.89:0/4286638146 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5aec06aed0 0x7f5aec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:13.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.728+0000 7f5b09400700 1 --2- 172.21.15.89:0/4286638146 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5aec06aed0 0x7f5aec06d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c0b5e60 tx=0x7f5af8009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.732+0000 7f5b02000700 1 -- 172.21.15.89:0/4286638146 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5afc055c20 con 0x7f5b0c06c650 2024-04-02T09:13:13.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.935+0000 7f5b10a00700 1 -- 172.21.15.89:0/4286638146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b0c0b6640 con 0x7f5b0c06c650 2024-04-02T09:13:13.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.937+0000 7f5b02000700 1 -- 172.21.15.89:0/4286638146 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+17989 (secure 0 0 0) 0x7f5afc059240 con 0x7f5b0c06c650 2024-04-02T09:13:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:13.935 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":232,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:13:11.284499+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:12:37.240568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:12:38.093999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6837/2615927215":"2024-04-03T09:13:08.185338+0000","172.21.15.89:0/1531440110":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/92627425":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/3786436284":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/3784813489":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/38958757":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/544818595":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/1763496588":"2024-04-03T09:13:05.404716+0000","172.21.15.89:6835/2615927215":"2024-04-03T09:13:08.185338+0000","172.21.15.89:0/341049398":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/2216461018":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/3268555703":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/3481438563":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/783107677":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/2439246170":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/965177671":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/2516488332":"2024-04-03T09:13:05.404716+0000","172.21.15.89:6833/108855":"2024-04-03T09:12:53.003657+0000","172.21.15.89:6832/108855":"2024-04-03T09:12:53.003657+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:13:13.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.939+0000 7f5b10a00700 1 -- 172.21.15.89:0/4286638146 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5aec06aed0 msgr2=0x7f5aec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:13.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.939+0000 7f5b10a00700 1 --2- 172.21.15.89:0/4286638146 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5aec06aed0 0x7f5aec06d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c0b5e60 tx=0x7f5af8009040 comp rx=0 tx=0).stop 2024-04-02T09:13:13.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.939+0000 7f5b10a00700 1 -- 172.21.15.89:0/4286638146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b0c06c650 msgr2=0x7f5b0c15de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:13.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.939+0000 7f5b10a00700 1 --2- 172.21.15.89:0/4286638146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b0c06c650 0x7f5b0c15de80 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f5afc009c10 tx=0x7f5afc004750 comp rx=0 tx=0).stop 2024-04-02T09:13:13.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.939+0000 7f5b10a00700 1 -- 172.21.15.89:0/4286638146 shutdown_connections 2024-04-02T09:13:13.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.939+0000 7f5b10a00700 1 --2- 172.21.15.89:0/4286638146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5b0c15d710 0x7f5b0c0c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.939+0000 7f5b10a00700 1 --2- 172.21.15.89:0/4286638146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b0c06c650 0x7f5b0c15de80 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.939+0000 7f5b10a00700 1 --2- 172.21.15.89:0/4286638146 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5aec06aed0 0x7f5aec06d390 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.939+0000 7f5b10a00700 1 --2- 172.21.15.89:0/4286638146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b0c09aba0 0x7f5b0c09eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:13.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.939+0000 7f5b10a00700 1 -- 172.21.15.89:0/4286638146 >> 172.21.15.89:0/4286638146 conn(0x7f5b0c0d13e0 msgr2=0x7f5b0c0ae160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:13.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.939+0000 7f5b10a00700 1 -- 172.21.15.89:0/4286638146 shutdown_connections 2024-04-02T09:13:13.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:13.939+0000 7f5b10a00700 1 -- 172.21.15.89:0/4286638146 wait complete. 2024-04-02T09:13:13.956 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-04-02T09:13:13.956 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:13:14.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.148+0000 7f5366200700 1 Processor -- start 2024-04-02T09:13:14.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.148+0000 7f5366200700 1 -- start start 2024-04-02T09:13:14.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.149+0000 7f5366200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f53601619d0 0x7f53601685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:14.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.149+0000 7f5366200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53600d30d0 con 0x7f536015d640 2024-04-02T09:13:14.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.149+0000 7f5366200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53600b6840 con 0x7f53600ceec0 2024-04-02T09:13:14.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.149+0000 7f5366200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53600d2b30 con 0x7f53601619d0 2024-04-02T09:13:14.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.149+0000 7f535fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f53601619d0 0x7f53601685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.149+0000 7f535f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f536015d640 0x7f53600d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47632/0 (socket says 172.21.15.89:47632) 2024-04-02T09:13:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.149+0000 7f535f400700 1 -- 172.21.15.89:0/2690111895 learned_addr learned my addr 172.21.15.89:0/2690111895 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:14.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.150+0000 7f535fe00700 1 -- 172.21.15.89:0/2690111895 >> v1:172.21.15.113:6789/0 conn(0x7f53600ceec0 legacy=0x7f53601612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:14.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.150+0000 7f535fe00700 1 -- 172.21.15.89:0/2690111895 >> v1:172.21.15.89:6789/0 conn(0x7f536015d640 legacy=0x7f53600d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:14.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.150+0000 7f535fe00700 1 -- 172.21.15.89:0/2690111895 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53600d2df0 con 0x7f53601619d0 2024-04-02T09:13:14.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.150+0000 7f535fe00700 1 --2- 172.21.15.89:0/2690111895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53601619d0 0x7f53601685f0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f5358009d90 tx=0x7f535800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83eb16ed6ede144 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:14.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.150+0000 7f535e000700 1 -- 172.21.15.89:0/2690111895 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535800e070 con 0x7f53601619d0 2024-04-02T09:13:14.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.150+0000 7f535e000700 1 -- 172.21.15.89:0/2690111895 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535800bbd0 con 0x7f53601619d0 2024-04-02T09:13:14.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.150+0000 7f535e000700 1 -- 172.21.15.89:0/2690111895 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358012430 con 0x7f53601619d0 2024-04-02T09:13:14.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.151+0000 7f5366200700 1 -- 172.21.15.89:0/2690111895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53601619d0 msgr2=0x7f53601685f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:14.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.151+0000 7f5366200700 1 --2- 172.21.15.89:0/2690111895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53601619d0 0x7f53601685f0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f5358009d90 tx=0x7f535800b330 comp rx=0 tx=0).stop 2024-04-02T09:13:14.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.151+0000 7f5366200700 1 -- 172.21.15.89:0/2690111895 shutdown_connections 2024-04-02T09:13:14.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.151+0000 7f5366200700 1 --2- 172.21.15.89:0/2690111895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53601619d0 0x7f53601685f0 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.151+0000 7f5366200700 1 -- 172.21.15.89:0/2690111895 >> 172.21.15.89:0/2690111895 conn(0x7f536008aff0 msgr2=0x7f53600d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:14.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.151+0000 7f5366200700 1 -- 172.21.15.89:0/2690111895 shutdown_connections 2024-04-02T09:13:14.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.151+0000 7f5366200700 1 -- 172.21.15.89:0/2690111895 wait complete. 2024-04-02T09:13:14.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.153+0000 7f5366200700 1 Processor -- start 2024-04-02T09:13:14.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.153+0000 7f5366200700 1 -- start start 2024-04-02T09:13:14.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.154+0000 7f5366200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53600ceec0 0x7f536015bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:14.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.154+0000 7f5366200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f536015d640 0x7f536009d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:14.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.155+0000 7f5366200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53601619d0 0x7f536009f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:14.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.155+0000 7f5366200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53600a0240 con 0x7f53600ceec0 2024-04-02T09:13:14.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.155+0000 7f5366200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53600a0520 con 0x7f536015d640 2024-04-02T09:13:14.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.155+0000 7f5366200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53600a0800 con 0x7f53601619d0 2024-04-02T09:13:14.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.155+0000 7f535f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53600ceec0 0x7f536015bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:14.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.155+0000 7f535f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53600ceec0 0x7f536015bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46700/0 (socket says 172.21.15.89:46700) 2024-04-02T09:13:14.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.155+0000 7f535f400700 1 -- 172.21.15.89:0/1253468781 learned_addr learned my addr 172.21.15.89:0/1253468781 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:14.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.155+0000 7f535f400700 1 -- 172.21.15.89:0/1253468781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53601619d0 msgr2=0x7f536009f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:14.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.155+0000 7f535f400700 1 --2- 172.21.15.89:0/1253468781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53601619d0 0x7f536009f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.155+0000 7f535f400700 1 -- 172.21.15.89:0/1253468781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f536015d640 msgr2=0x7f536009d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:14.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.155+0000 7f535f400700 1 --2- 172.21.15.89:0/1253468781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f536015d640 0x7f536009d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.155+0000 7f535f400700 1 -- 172.21.15.89:0/1253468781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5358009a70 con 0x7f53600ceec0 2024-04-02T09:13:14.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.155+0000 7f535f400700 1 --2- 172.21.15.89:0/1253468781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53600ceec0 0x7f536015bae0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f535000c520 tx=0x7f535000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:14.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.156+0000 7f5357400700 1 -- 172.21.15.89:0/1253468781 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350005d70 con 0x7f53600ceec0 2024-04-02T09:13:14.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.156+0000 7f5357400700 1 -- 172.21.15.89:0/1253468781 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53500070d0 con 0x7f53600ceec0 2024-04-02T09:13:14.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.156+0000 7f5366200700 1 -- 172.21.15.89:0/1253468781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53600a0ae0 con 0x7f53600ceec0 2024-04-02T09:13:14.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.156+0000 7f5357400700 1 -- 172.21.15.89:0/1253468781 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350007440 con 0x7f53600ceec0 2024-04-02T09:13:14.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.156+0000 7f5366200700 1 -- 172.21.15.89:0/1253468781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536009ff10 con 0x7f53600ceec0 2024-04-02T09:13:14.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.157+0000 7f5366200700 1 -- 172.21.15.89:0/1253468781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53600b58a0 con 0x7f53600ceec0 2024-04-02T09:13:14.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.158+0000 7f5357400700 1 -- 172.21.15.89:0/1253468781 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f53500075a0 con 0x7f53600ceec0 2024-04-02T09:13:14.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.158+0000 7f5357400700 1 --2- 172.21.15.89:0/1253468781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f534006aed0 0x7f534006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:14.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.159+0000 7f5357400700 1 -- 172.21.15.89:0/1253468781 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6986+0+0 (secure 0 0 0) 0x7f5350089030 con 0x7f53600ceec0 2024-04-02T09:13:14.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.159+0000 7f535ea00700 1 --2- 172.21.15.89:0/1253468781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f534006aed0 0x7f534006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:14.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.159+0000 7f535ea00700 1 --2- 172.21.15.89:0/1253468781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f534006aed0 0x7f534006d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f534c000f60 tx=0x7f534c007a40 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:14.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.163+0000 7f5357400700 1 -- 172.21.15.89:0/1253468781 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5350053a60 con 0x7f53600ceec0 2024-04-02T09:13:14.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.415+0000 7f5366200700 1 -- 172.21.15.89:0/1253468781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5360067720 con 0x7f53600ceec0 2024-04-02T09:13:14.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.415+0000 7f5357400700 1 -- 172.21.15.89:0/1253468781 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4758 (secure 0 0 0) 0x7f5350057080 con 0x7f53600ceec0 2024-04-02T09:13:14.414 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:14.414 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":112,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8969,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3650347444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3650347444},{"type":"v1","addr":"172.21.15.89:6836","nonce":3650347444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2152500685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2152500685},{"type":"v1","addr":"172.21.15.113:6834","nonce":2152500685}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":11784,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2785458438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2785458438},{"type":"v1","addr":"172.21.15.89:6837","nonce":2785458438}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:39.106317+0000","modified":"2024-04-02T09:12:42.112504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10279":{"gid":10279,"name":"d","rank":0,"incarnation":107,"state":"up:active","state_seq":6,"addr":"172.21.15.113:6835/2457040702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2457040702},{"type":"v1","addr":"172.21.15.113:6835","nonce":2457040702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-02T09:13:14.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.418+0000 7f5366200700 1 -- 172.21.15.89:0/1253468781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f534006aed0 msgr2=0x7f534006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:14.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.418+0000 7f5366200700 1 --2- 172.21.15.89:0/1253468781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f534006aed0 0x7f534006d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f534c000f60 tx=0x7f534c007a40 comp rx=0 tx=0).stop 2024-04-02T09:13:14.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.418+0000 7f5366200700 1 -- 172.21.15.89:0/1253468781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53600ceec0 msgr2=0x7f536015bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:14.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.418+0000 7f5366200700 1 --2- 172.21.15.89:0/1253468781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53600ceec0 0x7f536015bae0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f535000c520 tx=0x7f535000c8e0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.418+0000 7f5366200700 1 -- 172.21.15.89:0/1253468781 shutdown_connections 2024-04-02T09:13:14.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.418+0000 7f5366200700 1 --2- 172.21.15.89:0/1253468781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53601619d0 0x7f536009f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.418+0000 7f5366200700 1 --2- 172.21.15.89:0/1253468781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53600ceec0 0x7f536015bae0 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.418+0000 7f5366200700 1 --2- 172.21.15.89:0/1253468781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f534006aed0 0x7f534006d390 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.418+0000 7f5366200700 1 --2- 172.21.15.89:0/1253468781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f536015d640 0x7f536009d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.418+0000 7f5366200700 1 -- 172.21.15.89:0/1253468781 >> 172.21.15.89:0/1253468781 conn(0x7f536008aff0 msgr2=0x7f536015ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:14.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.418+0000 7f5366200700 1 -- 172.21.15.89:0/1253468781 shutdown_connections 2024-04-02T09:13:14.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.418+0000 7f5366200700 1 -- 172.21.15.89:0/1253468781 wait complete. 2024-04-02T09:13:14.418 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 112 2024-04-02T09:13:14.434 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:13:14.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.626+0000 7fac0e800700 1 Processor -- start 2024-04-02T09:13:14.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.626+0000 7fac0e800700 1 -- start start 2024-04-02T09:13:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac0e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080cf3c0 0x7fac080d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac0e800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fac0815db40 0x7fac080cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac0e800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac080d1330 con 0x7fac0815d760 2024-04-02T09:13:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac0e800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac080d1fb0 con 0x7fac0815db40 2024-04-02T09:13:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac0e800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac080d1610 con 0x7fac080cf3c0 2024-04-02T09:13:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac07400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fac0815d760 0x7fac080cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47644/0 (socket says 172.21.15.89:47644) 2024-04-02T09:13:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac07400700 1 -- 172.21.15.89:0/2495008947 learned_addr learned my addr 172.21.15.89:0/2495008947 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac06a00700 1 --2- 172.21.15.89:0/2495008947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080cf3c0 0x7fac080d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac07e00700 1 --2- 172.21.15.89:0/2495008947 >> v2:172.21.15.113:3300/0 conn(0x7fac0815db40 0x7fac080cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac06000700 1 -- 172.21.15.89:0/2495008947 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450388643 0 0) 0x7fac080d1330 con 0x7fac0815d760 2024-04-02T09:13:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac06000700 1 -- 172.21.15.89:0/2495008947 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabec0035b0 con 0x7fac0815d760 2024-04-02T09:13:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac06a00700 1 -- 172.21.15.89:0/2495008947 >> v2:172.21.15.113:3300/0 conn(0x7fac0815db40 msgr2=0x7fac080cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:14.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac06a00700 1 --2- 172.21.15.89:0/2495008947 >> v2:172.21.15.113:3300/0 conn(0x7fac0815db40 0x7fac080cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac06a00700 1 -- 172.21.15.89:0/2495008947 >> v1:172.21.15.89:6789/0 conn(0x7fac0815d760 legacy=0x7fac080cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:14.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.628+0000 7fac06a00700 1 -- 172.21.15.89:0/2495008947 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac080d18f0 con 0x7fac080cf3c0 2024-04-02T09:13:14.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.629+0000 7fac06a00700 1 --2- 172.21.15.89:0/2495008947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080cf3c0 0x7fac080d23d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fabf4009840 tx=0x7fabf4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85e3f59ada95630a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:14.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.629+0000 7fac06000700 1 -- 172.21.15.89:0/2495008947 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf4011830 con 0x7fac080cf3c0 2024-04-02T09:13:14.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.629+0000 7fac06000700 1 -- 172.21.15.89:0/2495008947 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf4011990 con 0x7fac080cf3c0 2024-04-02T09:13:14.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.629+0000 7fac06000700 1 -- 172.21.15.89:0/2495008947 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf4011d20 con 0x7fac080cf3c0 2024-04-02T09:13:14.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.629+0000 7fac0e800700 1 -- 172.21.15.89:0/2495008947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080cf3c0 msgr2=0x7fac080d23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:14.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.629+0000 7fac0e800700 1 --2- 172.21.15.89:0/2495008947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080cf3c0 0x7fac080d23d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fabf4009840 tx=0x7fabf4009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:14.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.630+0000 7fac0e800700 1 -- 172.21.15.89:0/2495008947 shutdown_connections 2024-04-02T09:13:14.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.630+0000 7fac0e800700 1 --2- 172.21.15.89:0/2495008947 >> v2:172.21.15.113:3300/0 conn(0x7fac0815db40 0x7fac080cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.630+0000 7fac0e800700 1 --2- 172.21.15.89:0/2495008947 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080cf3c0 0x7fac080d23d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.630+0000 7fac0e800700 1 -- 172.21.15.89:0/2495008947 >> 172.21.15.89:0/2495008947 conn(0x7fac0808aff0 msgr2=0x7fac080d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:14.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.630+0000 7fac0e800700 1 -- 172.21.15.89:0/2495008947 shutdown_connections 2024-04-02T09:13:14.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.630+0000 7fac0e800700 1 -- 172.21.15.89:0/2495008947 wait complete. 2024-04-02T09:13:14.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.631+0000 7fac0e800700 1 Processor -- start 2024-04-02T09:13:14.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.632+0000 7fac0e800700 1 -- start start 2024-04-02T09:13:14.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.632+0000 7fac0e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080cf3c0 0x7fac08151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:14.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac0e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac0815d760 0x7fac080c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:14.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac0e800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac0815db40 0x7fac080b1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:14.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac0e800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac08152010 con 0x7fac0815d760 2024-04-02T09:13:14.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac0e800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac080c8c70 con 0x7fac0815db40 2024-04-02T09:13:14.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac0e800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac080c8f50 con 0x7fac080cf3c0 2024-04-02T09:13:14.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac06a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac0815d760 0x7fac080c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac06a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac0815d760 0x7fac080c8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46708/0 (socket says 172.21.15.89:46708) 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac06a00700 1 -- 172.21.15.89:0/3696619006 learned_addr learned my addr 172.21.15.89:0/3696619006 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac07400700 1 --2- 172.21.15.89:0/3696619006 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080cf3c0 0x7fac08151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac07e00700 1 --2- 172.21.15.89:0/3696619006 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac0815db40 0x7fac080b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac06a00700 1 -- 172.21.15.89:0/3696619006 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080cf3c0 msgr2=0x7fac08151ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac06a00700 1 --2- 172.21.15.89:0/3696619006 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080cf3c0 0x7fac08151ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac06a00700 1 -- 172.21.15.89:0/3696619006 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac0815db40 msgr2=0x7fac080b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac06a00700 1 --2- 172.21.15.89:0/3696619006 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac0815db40 0x7fac080b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.633+0000 7fac06a00700 1 -- 172.21.15.89:0/3696619006 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf40094a0 con 0x7fac0815d760 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.634+0000 7fac06a00700 1 --2- 172.21.15.89:0/3696619006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac0815d760 0x7fac080c8240 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fabf4009fd0 tx=0x7fabf4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.634+0000 7fabff400700 1 -- 172.21.15.89:0/3696619006 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf4004590 con 0x7fac0815d760 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.634+0000 7fabff400700 1 -- 172.21.15.89:0/3696619006 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf401e070 con 0x7fac0815d760 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.634+0000 7fac0e800700 1 -- 172.21.15.89:0/3696619006 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac080c9230 con 0x7fac0815d760 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.634+0000 7fabff400700 1 -- 172.21.15.89:0/3696619006 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf40038e0 con 0x7fac0815d760 2024-04-02T09:13:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.634+0000 7fac0e800700 1 -- 172.21.15.89:0/3696619006 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac080c97f0 con 0x7fac0815d760 2024-04-02T09:13:14.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.635+0000 7fac0e800700 1 -- 172.21.15.89:0/3696619006 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac08067720 con 0x7fac0815d760 2024-04-02T09:13:14.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.636+0000 7fabff400700 1 -- 172.21.15.89:0/3696619006 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fabf4003a40 con 0x7fac0815d760 2024-04-02T09:13:14.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.637+0000 7fabff400700 1 --2- 172.21.15.89:0/3696619006 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fabe806af00 0x7fabe806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:14.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.637+0000 7fabff400700 1 -- 172.21.15.89:0/3696619006 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6986+0+0 (secure 0 0 0) 0x7fabf408aaa0 con 0x7fac0815d760 2024-04-02T09:13:14.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.637+0000 7fac07400700 1 --2- 172.21.15.89:0/3696619006 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fabe806af00 0x7fabe806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:14.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.638+0000 7fac07400700 1 --2- 172.21.15.89:0/3696619006 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fabe806af00 0x7fabe806d3c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fabf8007fd0 tx=0x7fabf8007dc0 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:14.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.641+0000 7fabff400700 1 -- 172.21.15.89:0/3696619006 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fabf4055390 con 0x7fac0815d760 2024-04-02T09:13:14.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.888+0000 7fac0e800700 1 -- 172.21.15.89:0/3696619006 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac080c99e0 con 0x7fac0815d760 2024-04-02T09:13:14.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.889+0000 7fabff400700 1 -- 172.21.15.89:0/3696619006 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4758 (secure 0 0 0) 0x7fabf40589b0 con 0x7fac0815d760 2024-04-02T09:13:14.887 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:14.887 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":112,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8969,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3650347444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3650347444},{"type":"v1","addr":"172.21.15.89:6836","nonce":3650347444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2152500685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2152500685},{"type":"v1","addr":"172.21.15.113:6834","nonce":2152500685}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":11784,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2785458438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2785458438},{"type":"v1","addr":"172.21.15.89:6837","nonce":2785458438}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:12:39.106317+0000","modified":"2024-04-02T09:12:42.112504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10279":{"gid":10279,"name":"d","rank":0,"incarnation":107,"state":"up:active","state_seq":6,"addr":"172.21.15.113:6835/2457040702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2457040702},{"type":"v1","addr":"172.21.15.113:6835","nonce":2457040702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-04-02T09:13:14.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.891+0000 7fac0e800700 1 -- 172.21.15.89:0/3696619006 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fabe806af00 msgr2=0x7fabe806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:14.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.891+0000 7fac0e800700 1 --2- 172.21.15.89:0/3696619006 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fabe806af00 0x7fabe806d3c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fabf8007fd0 tx=0x7fabf8007dc0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.891+0000 7fac0e800700 1 -- 172.21.15.89:0/3696619006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac0815d760 msgr2=0x7fac080c8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:14.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.891+0000 7fac0e800700 1 --2- 172.21.15.89:0/3696619006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac0815d760 0x7fac080c8240 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fabf4009fd0 tx=0x7fabf4004750 comp rx=0 tx=0).stop 2024-04-02T09:13:14.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.891+0000 7fac0e800700 1 -- 172.21.15.89:0/3696619006 shutdown_connections 2024-04-02T09:13:14.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.891+0000 7fac0e800700 1 --2- 172.21.15.89:0/3696619006 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fabe806af00 0x7fabe806d3c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.891+0000 7fac0e800700 1 --2- 172.21.15.89:0/3696619006 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac0815db40 0x7fac080b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.891+0000 7fac0e800700 1 --2- 172.21.15.89:0/3696619006 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080cf3c0 0x7fac08151ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.891+0000 7fac0e800700 1 --2- 172.21.15.89:0/3696619006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac0815d760 0x7fac080c8240 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:14.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.891+0000 7fac0e800700 1 -- 172.21.15.89:0/3696619006 >> 172.21.15.89:0/3696619006 conn(0x7fac0808aff0 msgr2=0x7fac080b5330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:14.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.891+0000 7fac0e800700 1 -- 172.21.15.89:0/3696619006 shutdown_connections 2024-04-02T09:13:14.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:14.891+0000 7fac0e800700 1 -- 172.21.15.89:0/3696619006 wait complete. 2024-04-02T09:13:14.891 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 112 2024-04-02T09:13:14.908 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:13:15.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.102+0000 7ff4ffa00700 1 Processor -- start 2024-04-02T09:13:15.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.102+0000 7ff4ffa00700 1 -- start start 2024-04-02T09:13:15.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.103+0000 7ff4ffa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff4f80d0a40 0x7ff4f80b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:15.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.103+0000 7ff4ffa00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f80d0ef0 con 0x7ff4f80d05e0 2024-04-02T09:13:15.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.103+0000 7ff4ffa00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f80d24e0 con 0x7ff4f815da50 2024-04-02T09:13:15.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.103+0000 7ff4ffa00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f80b95d0 con 0x7ff4f80d0a40 2024-04-02T09:13:15.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.104+0000 7ff4fda00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff4f80d0a40 0x7ff4f80b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:15.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.104+0000 7ff4f7e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff4f80d05e0 0x7ff4f80b6990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47660/0 (socket says 172.21.15.89:47660) 2024-04-02T09:13:15.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.104+0000 7ff4f7e00700 1 -- 172.21.15.89:0/2014198733 learned_addr learned my addr 172.21.15.89:0/2014198733 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.104+0000 7ff4fda00700 1 -- 172.21.15.89:0/2014198733 >> v1:172.21.15.113:6789/0 conn(0x7ff4f815da50 legacy=0x7ff4f80ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.104+0000 7ff4fda00700 1 -- 172.21.15.89:0/2014198733 >> v1:172.21.15.89:6789/0 conn(0x7ff4f80d05e0 legacy=0x7ff4f80b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.104+0000 7ff4fda00700 1 -- 172.21.15.89:0/2014198733 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f80d27c0 con 0x7ff4f80d0a40 2024-04-02T09:13:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.104+0000 7ff4fda00700 1 --2- 172.21.15.89:0/2014198733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f80d0a40 0x7ff4f80b70a0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0009d90 tx=0x7ff4f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42cfde3e2ed1430d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.105+0000 7ff4f7400700 1 -- 172.21.15.89:0/2014198733 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f000e070 con 0x7ff4f80d0a40 2024-04-02T09:13:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.105+0000 7ff4f7400700 1 -- 172.21.15.89:0/2014198733 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f000bbd0 con 0x7ff4f80d0a40 2024-04-02T09:13:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.105+0000 7ff4f7400700 1 -- 172.21.15.89:0/2014198733 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f0012430 con 0x7ff4f80d0a40 2024-04-02T09:13:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.105+0000 7ff4ffa00700 1 -- 172.21.15.89:0/2014198733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f80d0a40 msgr2=0x7ff4f80b70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.105+0000 7ff4ffa00700 1 --2- 172.21.15.89:0/2014198733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f80d0a40 0x7ff4f80b70a0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0009d90 tx=0x7ff4f000b330 comp rx=0 tx=0).stop 2024-04-02T09:13:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.105+0000 7ff4ffa00700 1 -- 172.21.15.89:0/2014198733 shutdown_connections 2024-04-02T09:13:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.105+0000 7ff4ffa00700 1 --2- 172.21.15.89:0/2014198733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f80d0a40 0x7ff4f80b70a0 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.105+0000 7ff4ffa00700 1 -- 172.21.15.89:0/2014198733 >> 172.21.15.89:0/2014198733 conn(0x7ff4f80666a0 msgr2=0x7ff4f8066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.105+0000 7ff4ffa00700 1 -- 172.21.15.89:0/2014198733 shutdown_connections 2024-04-02T09:13:15.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.105+0000 7ff4ffa00700 1 -- 172.21.15.89:0/2014198733 wait complete. 2024-04-02T09:13:15.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.107+0000 7ff4ffa00700 1 Processor -- start 2024-04-02T09:13:15.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.107+0000 7ff4ffa00700 1 -- start start 2024-04-02T09:13:15.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.108+0000 7ff4ffa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4f80d05e0 0x7ff4f81f45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:15.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.108+0000 7ff4ffa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff4f80d0a40 0x7ff4f81f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:15.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.108+0000 7ff4ffa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f815da50 0x7ff4f80c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:15.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.108+0000 7ff4ffa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f81f67d0 con 0x7ff4f815da50 2024-04-02T09:13:15.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.108+0000 7ff4ffa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f80c92f0 con 0x7ff4f80d05e0 2024-04-02T09:13:15.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.108+0000 7ff4ffa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f80c8d30 con 0x7ff4f80d0a40 2024-04-02T09:13:15.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.108+0000 7ff4fda00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f815da50 0x7ff4f80c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:15.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.108+0000 7ff4fda00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f815da50 0x7ff4f80c83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46728/0 (socket says 172.21.15.89:46728) 2024-04-02T09:13:15.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.108+0000 7ff4fda00700 1 -- 172.21.15.89:0/540798820 learned_addr learned my addr 172.21.15.89:0/540798820 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:15.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.108+0000 7ff4fd000700 1 --2- 172.21.15.89:0/540798820 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4f80d05e0 0x7ff4f81f45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:15.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.108+0000 7ff4f7e00700 1 --2- 172.21.15.89:0/540798820 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff4f80d0a40 0x7ff4f81f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.109+0000 7ff4fda00700 1 -- 172.21.15.89:0/540798820 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff4f80d0a40 msgr2=0x7ff4f81f6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.109+0000 7ff4fda00700 1 --2- 172.21.15.89:0/540798820 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff4f80d0a40 0x7ff4f81f6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.109+0000 7ff4fda00700 1 -- 172.21.15.89:0/540798820 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4f80d05e0 msgr2=0x7ff4f81f45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.109+0000 7ff4fda00700 1 --2- 172.21.15.89:0/540798820 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4f80d05e0 0x7ff4f81f45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.109+0000 7ff4fda00700 1 -- 172.21.15.89:0/540798820 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f0009a70 con 0x7ff4f815da50 2024-04-02T09:13:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.109+0000 7ff4fda00700 1 --2- 172.21.15.89:0/540798820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f815da50 0x7ff4f80c83c0 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0014040 tx=0x7ff4f0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.109+0000 7ff4f4c00700 1 -- 172.21.15.89:0/540798820 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f0012d20 con 0x7ff4f815da50 2024-04-02T09:13:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.109+0000 7ff4f4c00700 1 -- 172.21.15.89:0/540798820 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f0020070 con 0x7ff4f815da50 2024-04-02T09:13:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.109+0000 7ff4f4c00700 1 -- 172.21.15.89:0/540798820 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f0012ea0 con 0x7ff4f815da50 2024-04-02T09:13:15.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.109+0000 7ff4ffa00700 1 -- 172.21.15.89:0/540798820 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f80c9010 con 0x7ff4f815da50 2024-04-02T09:13:15.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.109+0000 7ff4ffa00700 1 -- 172.21.15.89:0/540798820 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f8150a90 con 0x7ff4f815da50 2024-04-02T09:13:15.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.110+0000 7ff4ffa00700 1 -- 172.21.15.89:0/540798820 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f8150610 con 0x7ff4f815da50 2024-04-02T09:13:15.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.111+0000 7ff4f4c00700 1 -- 172.21.15.89:0/540798820 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7ff4f0004280 con 0x7ff4f815da50 2024-04-02T09:13:15.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.112+0000 7ff4f4c00700 1 --2- 172.21.15.89:0/540798820 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff4d806aed0 0x7ff4d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:15.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.112+0000 7ff4f4c00700 1 -- 172.21.15.89:0/540798820 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6986+0+0 (secure 0 0 0) 0x7ff4f0022070 con 0x7ff4f815da50 2024-04-02T09:13:15.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.112+0000 7ff4fd000700 1 --2- 172.21.15.89:0/540798820 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff4d806aed0 0x7ff4d806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:15.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.113+0000 7ff4fd000700 1 --2- 172.21.15.89:0/540798820 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff4d806aed0 0x7ff4d806d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff4f81613d0 tx=0x7ff4ec009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:15.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.116+0000 7ff4f4c00700 1 -- 172.21.15.89:0/540798820 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff4f8150610 con 0x7ff4f815da50 2024-04-02T09:13:15.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.325+0000 7ff4ffa00700 1 -- 172.21.15.89:0/540798820 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4f8150610 con 0x7ff4f815da50 2024-04-02T09:13:15.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.326+0000 7ff4f4c00700 1 -- 172.21.15.89:0/540798820 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+17989 (secure 0 0 0) 0x7ff4f8150610 con 0x7ff4f815da50 2024-04-02T09:13:15.324 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:15.325 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":232,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:13:11.284499+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:12:37.240568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:12:38.093999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6837/2615927215":"2024-04-03T09:13:08.185338+0000","172.21.15.89:0/1531440110":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/92627425":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/3786436284":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/3784813489":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/38958757":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/544818595":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/1763496588":"2024-04-03T09:13:05.404716+0000","172.21.15.89:6835/2615927215":"2024-04-03T09:13:08.185338+0000","172.21.15.89:0/341049398":"2024-04-03T09:13:05.404716+0000","172.21.15.89:0/2216461018":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/3268555703":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/3481438563":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/783107677":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/2439246170":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/965177671":"2024-04-03T09:12:53.003657+0000","172.21.15.89:0/2516488332":"2024-04-03T09:13:05.404716+0000","172.21.15.89:6833/108855":"2024-04-03T09:12:53.003657+0000","172.21.15.89:6832/108855":"2024-04-03T09:12:53.003657+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:13:15.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.328+0000 7ff4ffa00700 1 -- 172.21.15.89:0/540798820 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff4d806aed0 msgr2=0x7ff4d806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:15.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.328+0000 7ff4ffa00700 1 --2- 172.21.15.89:0/540798820 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff4d806aed0 0x7ff4d806d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff4f81613d0 tx=0x7ff4ec009040 comp rx=0 tx=0).stop 2024-04-02T09:13:15.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.328+0000 7ff4ffa00700 1 -- 172.21.15.89:0/540798820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f815da50 msgr2=0x7ff4f80c83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:15.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.328+0000 7ff4ffa00700 1 --2- 172.21.15.89:0/540798820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f815da50 0x7ff4f80c83c0 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0014040 tx=0x7ff4f0004ad0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.328+0000 7ff4ffa00700 1 -- 172.21.15.89:0/540798820 shutdown_connections 2024-04-02T09:13:15.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.328+0000 7ff4ffa00700 1 --2- 172.21.15.89:0/540798820 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff4d806aed0 0x7ff4d806d390 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.328+0000 7ff4ffa00700 1 --2- 172.21.15.89:0/540798820 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4f80d05e0 0x7ff4f81f45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.328+0000 7ff4ffa00700 1 --2- 172.21.15.89:0/540798820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4f815da50 0x7ff4f80c83c0 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.328+0000 7ff4ffa00700 1 --2- 172.21.15.89:0/540798820 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff4f80d0a40 0x7ff4f81f6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.328+0000 7ff4ffa00700 1 -- 172.21.15.89:0/540798820 >> 172.21.15.89:0/540798820 conn(0x7ff4f80666a0 msgr2=0x7ff4f80b7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:15.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.328+0000 7ff4ffa00700 1 -- 172.21.15.89:0/540798820 shutdown_connections 2024-04-02T09:13:15.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.329+0000 7ff4ffa00700 1 -- 172.21.15.89:0/540798820 wait complete. 2024-04-02T09:13:15.345 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-04-02T09:13:15.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.539+0000 7fef5e000700 1 Processor -- start 2024-04-02T09:13:15.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.539+0000 7fef5e000700 1 -- start start 2024-04-02T09:13:15.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.540+0000 7fef5e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef580b2b30 0x7fef580b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:15.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.540+0000 7fef5e000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fef580b3960 0x7fef580cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:15.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.540+0000 7fef5e000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef580b3db0 con 0x7fef58157f90 2024-04-02T09:13:15.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.540+0000 7fef5e000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef58158370 con 0x7fef580b3960 2024-04-02T09:13:15.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.540+0000 7fef5e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef580b2710 con 0x7fef580b2b30 2024-04-02T09:13:15.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.541+0000 7fef57400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fef58157f90 0x7fef581613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47664/0 (socket says 172.21.15.89:47664) 2024-04-02T09:13:15.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.541+0000 7fef57400700 1 -- 172.21.15.89:0/3901766542 learned_addr learned my addr 172.21.15.89:0/3901766542 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:15.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.541+0000 7fef57e00700 1 --2- 172.21.15.89:0/3901766542 >> v2:172.21.15.113:3300/0 conn(0x7fef580b3960 0x7fef580cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:15.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.541+0000 7fef56a00700 1 --2- 172.21.15.89:0/3901766542 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef580b2b30 0x7fef580b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:15.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.541+0000 7fef56000700 1 -- 172.21.15.89:0/3901766542 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185621092 0 0) 0x7fef48001110 con 0x7fef58157f90 2024-04-02T09:13:15.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.541+0000 7fef56000700 1 -- 172.21.15.89:0/3901766542 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef3c0035b0 con 0x7fef58157f90 2024-04-02T09:13:15.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.541+0000 7fef56000700 1 -- 172.21.15.89:0/3901766542 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1909232972 0 0) 0x7fef3c0035b0 con 0x7fef58157f90 2024-04-02T09:13:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.541+0000 7fef56a00700 1 -- 172.21.15.89:0/3901766542 >> v2:172.21.15.113:3300/0 conn(0x7fef580b3960 msgr2=0x7fef580cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.541+0000 7fef56a00700 1 --2- 172.21.15.89:0/3901766542 >> v2:172.21.15.113:3300/0 conn(0x7fef580b3960 0x7fef580cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.541+0000 7fef56a00700 1 -- 172.21.15.89:0/3901766542 >> v1:172.21.15.89:6789/0 conn(0x7fef58157f90 legacy=0x7fef581613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.541+0000 7fef56a00700 1 -- 172.21.15.89:0/3901766542 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef580b2070 con 0x7fef580b2b30 2024-04-02T09:13:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.541+0000 7fef57e00700 1 --2- 172.21.15.89:0/3901766542 >> v2:172.21.15.113:3300/0 conn(0x7fef580b3960 0x7fef580cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.542+0000 7fef56a00700 1 --2- 172.21.15.89:0/3901766542 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef580b2b30 0x7fef580b3420 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fef44009840 tx=0x7fef44009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7e06e13ff37de2d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.542+0000 7fef56000700 1 -- 172.21.15.89:0/3901766542 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef48001800 con 0x7fef580b2b30 2024-04-02T09:13:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.542+0000 7fef56000700 1 -- 172.21.15.89:0/3901766542 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef44004030 con 0x7fef580b2b30 2024-04-02T09:13:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.542+0000 7fef56000700 1 -- 172.21.15.89:0/3901766542 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef440043c0 con 0x7fef580b2b30 2024-04-02T09:13:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.543+0000 7fef5e000700 1 -- 172.21.15.89:0/3901766542 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef580b2b30 msgr2=0x7fef580b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.543+0000 7fef5e000700 1 --2- 172.21.15.89:0/3901766542 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef580b2b30 0x7fef580b3420 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fef44009840 tx=0x7fef44009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.543+0000 7fef5e000700 1 -- 172.21.15.89:0/3901766542 shutdown_connections 2024-04-02T09:13:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.543+0000 7fef5e000700 1 --2- 172.21.15.89:0/3901766542 >> v2:172.21.15.113:3300/0 conn(0x7fef580b3960 0x7fef580cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.543+0000 7fef5e000700 1 --2- 172.21.15.89:0/3901766542 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef580b2b30 0x7fef580b3420 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.543+0000 7fef5e000700 1 -- 172.21.15.89:0/3901766542 >> 172.21.15.89:0/3901766542 conn(0x7fef580b0d70 msgr2=0x7fef580b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.543+0000 7fef5e000700 1 -- 172.21.15.89:0/3901766542 shutdown_connections 2024-04-02T09:13:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.543+0000 7fef5e000700 1 -- 172.21.15.89:0/3901766542 wait complete. 2024-04-02T09:13:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.544+0000 7fef5e000700 1 Processor -- start 2024-04-02T09:13:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.545+0000 7fef5e000700 1 -- start start 2024-04-02T09:13:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.545+0000 7fef5e000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef580b2b30 0x7fef580c9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.545+0000 7fef5e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef580b3960 0x7fef58153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.545+0000 7fef5e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef58157f90 0x7fef580cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.545+0000 7fef5e000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef58153db0 con 0x7fef580b3960 2024-04-02T09:13:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.545+0000 7fef5e000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef580cc2e0 con 0x7fef580b2b30 2024-04-02T09:13:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.545+0000 7fef5e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef580cbd20 con 0x7fef58157f90 2024-04-02T09:13:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.546+0000 7fef56a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef580b3960 0x7fef58153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.546+0000 7fef56a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef580b3960 0x7fef58153870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46738/0 (socket says 172.21.15.89:46738) 2024-04-02T09:13:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.546+0000 7fef56a00700 1 -- 172.21.15.89:0/2164718268 learned_addr learned my addr 172.21.15.89:0/2164718268 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.546+0000 7fef57400700 1 --2- 172.21.15.89:0/2164718268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef580b2b30 0x7fef580c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.546+0000 7fef57e00700 1 --2- 172.21.15.89:0/2164718268 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef58157f90 0x7fef580cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.546+0000 7fef56a00700 1 -- 172.21.15.89:0/2164718268 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef58157f90 msgr2=0x7fef580cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.546+0000 7fef56a00700 1 --2- 172.21.15.89:0/2164718268 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef58157f90 0x7fef580cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.546+0000 7fef56a00700 1 -- 172.21.15.89:0/2164718268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef580b2b30 msgr2=0x7fef580c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.546+0000 7fef56a00700 1 --2- 172.21.15.89:0/2164718268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef580b2b30 0x7fef580c9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.546+0000 7fef56a00700 1 -- 172.21.15.89:0/2164718268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef50009a70 con 0x7fef580b3960 2024-04-02T09:13:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.546+0000 7fef56a00700 1 --2- 172.21.15.89:0/2164718268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef580b3960 0x7fef58153870 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7fef4400a010 tx=0x7fef44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.547+0000 7fef4f400700 1 -- 172.21.15.89:0/2164718268 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef44003c00 con 0x7fef580b3960 2024-04-02T09:13:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.547+0000 7fef4f400700 1 -- 172.21.15.89:0/2164718268 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4401f070 con 0x7fef580b3960 2024-04-02T09:13:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.547+0000 7fef4f400700 1 -- 172.21.15.89:0/2164718268 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef44011630 con 0x7fef580b3960 2024-04-02T09:13:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.547+0000 7fef5e000700 1 -- 172.21.15.89:0/2164718268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef440094a0 con 0x7fef580b3960 2024-04-02T09:13:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.547+0000 7fef5e000700 1 -- 172.21.15.89:0/2164718268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5809af50 con 0x7fef580b3960 2024-04-02T09:13:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.548+0000 7fef5e000700 1 -- 172.21.15.89:0/2164718268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef58067720 con 0x7fef580b3960 2024-04-02T09:13:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.549+0000 7fef4f400700 1 -- 172.21.15.89:0/2164718268 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fef44021030 con 0x7fef580b3960 2024-04-02T09:13:15.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.550+0000 7fef4f400700 1 --2- 172.21.15.89:0/2164718268 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef3806aed0 0x7fef3806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:15.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.550+0000 7fef4f400700 1 -- 172.21.15.89:0/2164718268 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6986+0+0 (secure 0 0 0) 0x7fef4408afc0 con 0x7fef580b3960 2024-04-02T09:13:15.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.550+0000 7fef57400700 1 --2- 172.21.15.89:0/2164718268 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef3806aed0 0x7fef3806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.551+0000 7fef57400700 1 --2- 172.21.15.89:0/2164718268 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef3806aed0 0x7fef3806d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fef48001eb0 tx=0x7fef48001640 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:15.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.555+0000 7fef4f400700 1 -- 172.21.15.89:0/2164718268 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fef440be580 con 0x7fef580b3960 2024-04-02T09:13:15.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.800+0000 7fef5e000700 1 -- 172.21.15.89:0/2164718268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7fef580ce2e0 con 0x7fef580b3960 2024-04-02T09:13:15.799 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T09:13:15.801+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T09:13:15.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.810+0000 7fef4f400700 1 -- 172.21.15.89:0/2164718268 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v113) v1 ==== 167+0+0 (secure 0 0 0) 0x7fef44055af0 con 0x7fef580b3960 2024-04-02T09:13:15.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.812+0000 7fef5e000700 1 -- 172.21.15.89:0/2164718268 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef3806aed0 msgr2=0x7fef3806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:15.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.812+0000 7fef5e000700 1 --2- 172.21.15.89:0/2164718268 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef3806aed0 0x7fef3806d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fef48001eb0 tx=0x7fef48001640 comp rx=0 tx=0).stop 2024-04-02T09:13:15.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.812+0000 7fef5e000700 1 -- 172.21.15.89:0/2164718268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef580b3960 msgr2=0x7fef58153870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:15.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.812+0000 7fef5e000700 1 --2- 172.21.15.89:0/2164718268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef580b3960 0x7fef58153870 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7fef4400a010 tx=0x7fef44004750 comp rx=0 tx=0).stop 2024-04-02T09:13:15.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.813+0000 7fef5e000700 1 -- 172.21.15.89:0/2164718268 shutdown_connections 2024-04-02T09:13:15.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.813+0000 7fef5e000700 1 --2- 172.21.15.89:0/2164718268 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef3806aed0 0x7fef3806d390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.813+0000 7fef5e000700 1 --2- 172.21.15.89:0/2164718268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef580b2b30 0x7fef580c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.813+0000 7fef5e000700 1 --2- 172.21.15.89:0/2164718268 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef58157f90 0x7fef580cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.813+0000 7fef5e000700 1 --2- 172.21.15.89:0/2164718268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef580b3960 0x7fef58153870 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:15.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.813+0000 7fef5e000700 1 -- 172.21.15.89:0/2164718268 >> 172.21.15.89:0/2164718268 conn(0x7fef580b0d70 msgr2=0x7fef5806e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:15.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.813+0000 7fef5e000700 1 -- 172.21.15.89:0/2164718268 shutdown_connections 2024-04-02T09:13:15.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:15.813+0000 7fef5e000700 1 -- 172.21.15.89:0/2164718268 wait complete. 2024-04-02T09:13:15.814 INFO:teuthology.orchestra.run.smithi089.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T09:13:15.815 INFO:tasks.ceph.mds.d.smithi113.stderr: -2017> 2024-04-02T09:12:40.663+0000 7fc139e00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T09:13:15.829 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-04-02T09:13:15.862 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T09:13:16.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.025+0000 7f4a74600700 1 Processor -- start 2024-04-02T09:13:16.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.026+0000 7f4a74600700 1 -- start start 2024-04-02T09:13:16.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.027+0000 7f4a74600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a6c0cbb30 0x7f4a6c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:16.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.027+0000 7f4a74600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4a6c0cd4a0 0x7f4a6c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:16.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.027+0000 7f4a74600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6c0cc450 con 0x7f4a6c0cd020 2024-04-02T09:13:16.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.027+0000 7f4a74600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a6c0cdae0 con 0x7f4a6c0cd4a0 2024-04-02T09:13:16.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.027+0000 7f4a74600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c0cddc0 con 0x7f4a6c0cbb30 2024-04-02T09:13:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.027+0000 7f4a71c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a6c0cbb30 0x7f4a6c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.027+0000 7f4a72600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4a6c0cd4a0 0x7f4a6c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.027+0000 7f4a72600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4a6c0cd4a0 0x7f4a6c06e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:35226/0 (socket says 172.21.15.89:35226) 2024-04-02T09:13:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.027+0000 7f4a72600700 1 -- 172.21.15.89:0/70293919 learned_addr learned my addr 172.21.15.89:0/70293919 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.027+0000 7f4a71c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a6c0cbb30 0x7f4a6c0cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:33282/0 (socket says 172.21.15.89:33282) 2024-04-02T09:13:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.028+0000 7f4a71c00700 1 -- 172.21.15.89:0/70293919 >> v2:172.21.15.113:3300/0 conn(0x7f4a6c0cd4a0 msgr2=0x7f4a6c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.028+0000 7f4a71c00700 1 --2- 172.21.15.89:0/70293919 >> v2:172.21.15.113:3300/0 conn(0x7f4a6c0cd4a0 0x7f4a6c06e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.028+0000 7f4a71c00700 1 -- 172.21.15.89:0/70293919 >> v1:172.21.15.113:6789/0 conn(0x7f4a6c0cd020 legacy=0x7f4a6c0b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.028+0000 7f4a71c00700 1 -- 172.21.15.89:0/70293919 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c0ce0a0 con 0x7f4a6c0cbb30 2024-04-02T09:13:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.028+0000 7f4a72600700 1 --2- 172.21.15.89:0/70293919 >> v2:172.21.15.113:3300/0 conn(0x7f4a6c0cd4a0 0x7f4a6c06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.028+0000 7f4a71c00700 1 --2- 172.21.15.89:0/70293919 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a6c0cbb30 0x7f4a6c0cbf10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4a60009840 tx=0x7f4a60009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d18d3bc9d86f0dd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.028+0000 7f4a6be00700 1 -- 172.21.15.89:0/70293919 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60011830 con 0x7f4a6c0cbb30 2024-04-02T09:13:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.028+0000 7f4a6be00700 1 -- 172.21.15.89:0/70293919 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a60011990 con 0x7f4a6c0cbb30 2024-04-02T09:13:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.028+0000 7f4a6be00700 1 -- 172.21.15.89:0/70293919 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60011d20 con 0x7f4a6c0cbb30 2024-04-02T09:13:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.029+0000 7f4a74600700 1 -- 172.21.15.89:0/70293919 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a6c0cbb30 msgr2=0x7f4a6c0cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.029+0000 7f4a74600700 1 --2- 172.21.15.89:0/70293919 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a6c0cbb30 0x7f4a6c0cbf10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4a60009840 tx=0x7f4a60009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.029+0000 7f4a74600700 1 -- 172.21.15.89:0/70293919 shutdown_connections 2024-04-02T09:13:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.029+0000 7f4a74600700 1 --2- 172.21.15.89:0/70293919 >> v2:172.21.15.113:3300/0 conn(0x7f4a6c0cd4a0 0x7f4a6c06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.029+0000 7f4a74600700 1 --2- 172.21.15.89:0/70293919 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a6c0cbb30 0x7f4a6c0cbf10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.029+0000 7f4a74600700 1 -- 172.21.15.89:0/70293919 >> 172.21.15.89:0/70293919 conn(0x7f4a6c0b0d70 msgr2=0x7f4a6c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.029+0000 7f4a74600700 1 -- 172.21.15.89:0/70293919 shutdown_connections 2024-04-02T09:13:16.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.029+0000 7f4a74600700 1 -- 172.21.15.89:0/70293919 wait complete. 2024-04-02T09:13:16.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.030+0000 7f4a74600700 1 Processor -- start 2024-04-02T09:13:16.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.031+0000 7f4a74600700 1 -- start start 2024-04-02T09:13:16.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.031+0000 7f4a74600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a6c0cbb30 0x7f4a6c154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:16.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.031+0000 7f4a74600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a6c0cd020 0x7f4a6c09abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:16.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.031+0000 7f4a74600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a6c0cd4a0 0x7f4a6c0b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:16.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.032+0000 7f4a74600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c09c2a0 con 0x7f4a6c0cd4a0 2024-04-02T09:13:16.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.032+0000 7f4a74600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c09b690 con 0x7f4a6c0cbb30 2024-04-02T09:13:16.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.032+0000 7f4a74600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c09b970 con 0x7f4a6c0cd020 2024-04-02T09:13:16.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.032+0000 7f4a72600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a6c0cd4a0 0x7f4a6c0b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:16.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.032+0000 7f4a72600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a6c0cd4a0 0x7f4a6c0b2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46752/0 (socket says 172.21.15.89:46752) 2024-04-02T09:13:16.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.032+0000 7f4a72600700 1 -- 172.21.15.89:0/1724721269 learned_addr learned my addr 172.21.15.89:0/1724721269 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:16.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.032+0000 7f4a71c00700 1 --2- 172.21.15.89:0/1724721269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a6c0cbb30 0x7f4a6c154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:16.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.032+0000 7f4a71200700 1 --2- 172.21.15.89:0/1724721269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a6c0cd020 0x7f4a6c09abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:16.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.032+0000 7f4a72600700 1 -- 172.21.15.89:0/1724721269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a6c0cd020 msgr2=0x7f4a6c09abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:16.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.032+0000 7f4a72600700 1 --2- 172.21.15.89:0/1724721269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a6c0cd020 0x7f4a6c09abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:16.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.032+0000 7f4a72600700 1 -- 172.21.15.89:0/1724721269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a6c0cbb30 msgr2=0x7f4a6c154cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:16.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.032+0000 7f4a72600700 1 --2- 172.21.15.89:0/1724721269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a6c0cbb30 0x7f4a6c154cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:16.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.032+0000 7f4a72600700 1 -- 172.21.15.89:0/1724721269 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a64009b30 con 0x7f4a6c0cd4a0 2024-04-02T09:13:16.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.032+0000 7f4a72600700 1 --2- 172.21.15.89:0/1724721269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a6c0cd4a0 0x7f4a6c0b2e80 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f4a6400efc0 tx=0x7f4a6400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:16.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.033+0000 7f4a69600700 1 -- 172.21.15.89:0/1724721269 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a64018070 con 0x7f4a6c0cd4a0 2024-04-02T09:13:16.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.033+0000 7f4a69600700 1 -- 172.21.15.89:0/1724721269 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6400f040 con 0x7f4a6c0cd4a0 2024-04-02T09:13:16.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.033+0000 7f4a69600700 1 -- 172.21.15.89:0/1724721269 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a64003c20 con 0x7f4a6c0cd4a0 2024-04-02T09:13:16.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.033+0000 7f4a74600700 1 -- 172.21.15.89:0/1724721269 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a600094a0 con 0x7f4a6c0cd4a0 2024-04-02T09:13:16.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.033+0000 7f4a74600700 1 -- 172.21.15.89:0/1724721269 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6c09bf30 con 0x7f4a6c0cd4a0 2024-04-02T09:13:16.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.034+0000 7f4a74600700 1 -- 172.21.15.89:0/1724721269 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6c09c450 con 0x7f4a6c0cd4a0 2024-04-02T09:13:16.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.035+0000 7f4a69600700 1 -- 172.21.15.89:0/1724721269 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4a6400c880 con 0x7f4a6c0cd4a0 2024-04-02T09:13:16.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.036+0000 7f4a69600700 1 --2- 172.21.15.89:0/1724721269 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4a4c06aed0 0x7f4a4c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:16.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.036+0000 7f4a69600700 1 -- 172.21.15.89:0/1724721269 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7072+0+0 (secure 0 0 0) 0x7f4a64089e70 con 0x7f4a6c0cd4a0 2024-04-02T09:13:16.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.036+0000 7f4a71c00700 1 --2- 172.21.15.89:0/1724721269 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4a4c06aed0 0x7f4a4c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:16.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.037+0000 7f4a71c00700 1 --2- 172.21.15.89:0/1724721269 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4a4c06aed0 0x7f4a4c06d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4a60000c00 tx=0x7f4a6001c040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:16.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.041+0000 7f4a69600700 1 -- 172.21.15.89:0/1724721269 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4a6c09c450 con 0x7f4a6c0cd4a0 2024-04-02T09:13:16.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.289+0000 7f4a74600700 1 -- 172.21.15.89:0/1724721269 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4a6c15bd50 con 0x7f4a6c0cd4a0 2024-04-02T09:13:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.813+0000 7f4a69600700 1 -- 172.21.15.89:0/1724721269 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v114) v1 ==== 107+0+0 (secure 0 0 0) 0x7f4a64054720 con 0x7f4a6c0cd4a0 2024-04-02T09:13:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.816+0000 7f4a74600700 1 -- 172.21.15.89:0/1724721269 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4a4c06aed0 msgr2=0x7f4a4c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.816+0000 7f4a74600700 1 --2- 172.21.15.89:0/1724721269 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4a4c06aed0 0x7f4a4c06d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4a60000c00 tx=0x7f4a6001c040 comp rx=0 tx=0).stop 2024-04-02T09:13:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.816+0000 7f4a74600700 1 -- 172.21.15.89:0/1724721269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a6c0cd4a0 msgr2=0x7f4a6c0b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.816+0000 7f4a74600700 1 --2- 172.21.15.89:0/1724721269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a6c0cd4a0 0x7f4a6c0b2e80 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f4a6400efc0 tx=0x7f4a6400c3a0 comp rx=0 tx=0).stop 2024-04-02T09:13:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.816+0000 7f4a74600700 1 -- 172.21.15.89:0/1724721269 shutdown_connections 2024-04-02T09:13:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.816+0000 7f4a74600700 1 --2- 172.21.15.89:0/1724721269 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4a4c06aed0 0x7f4a4c06d390 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.816+0000 7f4a74600700 1 --2- 172.21.15.89:0/1724721269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a6c0cbb30 0x7f4a6c154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.816+0000 7f4a74600700 1 --2- 172.21.15.89:0/1724721269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a6c0cd4a0 0x7f4a6c0b2e80 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.816+0000 7f4a74600700 1 --2- 172.21.15.89:0/1724721269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a6c0cd020 0x7f4a6c09abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.816+0000 7f4a74600700 1 -- 172.21.15.89:0/1724721269 >> 172.21.15.89:0/1724721269 conn(0x7f4a6c0b0d70 msgr2=0x7f4a6c06e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.816+0000 7f4a74600700 1 -- 172.21.15.89:0/1724721269 shutdown_connections 2024-04-02T09:13:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:16.816+0000 7f4a74600700 1 -- 172.21.15.89:0/1724721269 wait complete. 2024-04-02T09:13:16.835 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-04-02T09:13:17.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.036+0000 7f65b5c00700 1 Processor -- start 2024-04-02T09:13:17.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.036+0000 7f65b5c00700 1 -- start start 2024-04-02T09:13:17.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.037+0000 7f65b5c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f65b015c700 0x7f65b00adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:17.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.037+0000 7f65b5c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b006dd00 con 0x7f65b00ae4a0 2024-04-02T09:13:17.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.037+0000 7f65b5c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b006d0f0 con 0x7f65b0155ac0 2024-04-02T09:13:17.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.037+0000 7f65b5c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b006d3d0 con 0x7f65b015c700 2024-04-02T09:13:17.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.037+0000 7f65ae000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f65b015c700 0x7f65b00adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:17.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.037+0000 7f65af400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f65b00ae4a0 0x7f65b015ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47668/0 (socket says 172.21.15.89:47668) 2024-04-02T09:13:17.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.037+0000 7f65af400700 1 -- 172.21.15.89:0/2522531173 learned_addr learned my addr 172.21.15.89:0/2522531173 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:17.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.037+0000 7f65ad600700 1 -- 172.21.15.89:0/2522531173 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061577595 0 0) 0x7f65b006dd00 con 0x7f65b00ae4a0 2024-04-02T09:13:17.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.038+0000 7f65ad600700 1 -- 172.21.15.89:0/2522531173 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65940035b0 con 0x7f65b00ae4a0 2024-04-02T09:13:17.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.038+0000 7f65ae000700 1 -- 172.21.15.89:0/2522531173 >> v1:172.21.15.113:6789/0 conn(0x7f65b0155ac0 legacy=0x7f65b006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:17.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.038+0000 7f65ae000700 1 -- 172.21.15.89:0/2522531173 >> v1:172.21.15.89:6789/0 conn(0x7f65b00ae4a0 legacy=0x7f65b015ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:17.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.038+0000 7f65ae000700 1 -- 172.21.15.89:0/2522531173 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b006d6b0 con 0x7f65b015c700 2024-04-02T09:13:17.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.038+0000 7f65ae000700 1 --2- 172.21.15.89:0/2522531173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b015c700 0x7f65b00adf60 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f659c009840 tx=0x7f659c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee2142abdaf5b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:17.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.038+0000 7f65ad600700 1 -- 172.21.15.89:0/2522531173 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c011830 con 0x7f65b015c700 2024-04-02T09:13:17.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.038+0000 7f65ad600700 1 -- 172.21.15.89:0/2522531173 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659c011990 con 0x7f65b015c700 2024-04-02T09:13:17.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.038+0000 7f65ad600700 1 -- 172.21.15.89:0/2522531173 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c011d20 con 0x7f65b015c700 2024-04-02T09:13:17.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.039+0000 7f65b5c00700 1 -- 172.21.15.89:0/2522531173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b015c700 msgr2=0x7f65b00adf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:17.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.039+0000 7f65b5c00700 1 --2- 172.21.15.89:0/2522531173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b015c700 0x7f65b00adf60 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f659c009840 tx=0x7f659c009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:17.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.039+0000 7f65b5c00700 1 -- 172.21.15.89:0/2522531173 shutdown_connections 2024-04-02T09:13:17.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.039+0000 7f65b5c00700 1 --2- 172.21.15.89:0/2522531173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b015c700 0x7f65b00adf60 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:17.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.039+0000 7f65b5c00700 1 -- 172.21.15.89:0/2522531173 >> 172.21.15.89:0/2522531173 conn(0x7f65b00cf210 msgr2=0x7f65b00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:17.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.039+0000 7f65b5c00700 1 -- 172.21.15.89:0/2522531173 shutdown_connections 2024-04-02T09:13:17.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.039+0000 7f65b5c00700 1 -- 172.21.15.89:0/2522531173 wait complete. 2024-04-02T09:13:17.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.040+0000 7f65b5c00700 1 Processor -- start 2024-04-02T09:13:17.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.041+0000 7f65b5c00700 1 -- start start 2024-04-02T09:13:17.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.041+0000 7f65b5c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65b00ae4a0 0x7f65b0098560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:17.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.041+0000 7f65b5c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65b0155ac0 0x7f65b0150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:17.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.041+0000 7f65b5c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b015c700 0x7f65b0150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:17.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.041+0000 7f65b5c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b014ce60 con 0x7f65b015c700 2024-04-02T09:13:17.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.041+0000 7f65b5c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b014c4d0 con 0x7f65b00ae4a0 2024-04-02T09:13:17.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.041+0000 7f65b5c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b014ca90 con 0x7f65b0155ac0 2024-04-02T09:13:17.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.042+0000 7f65af400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b015c700 0x7f65b0150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:17.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.042+0000 7f65af400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b015c700 0x7f65b0150c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46778/0 (socket says 172.21.15.89:46778) 2024-04-02T09:13:17.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.042+0000 7f65af400700 1 -- 172.21.15.89:0/4008379988 learned_addr learned my addr 172.21.15.89:0/4008379988 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:17.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.042+0000 7f65aea00700 1 --2- 172.21.15.89:0/4008379988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65b00ae4a0 0x7f65b0098560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:17.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.042+0000 7f65ae000700 1 --2- 172.21.15.89:0/4008379988 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65b0155ac0 0x7f65b0150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:17.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.042+0000 7f65af400700 1 -- 172.21.15.89:0/4008379988 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65b0155ac0 msgr2=0x7f65b0150720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:17.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.042+0000 7f65af400700 1 --2- 172.21.15.89:0/4008379988 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65b0155ac0 0x7f65b0150720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:17.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.042+0000 7f65af400700 1 -- 172.21.15.89:0/4008379988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65b00ae4a0 msgr2=0x7f65b0098560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:17.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.042+0000 7f65af400700 1 --2- 172.21.15.89:0/4008379988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65b00ae4a0 0x7f65b0098560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:17.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.042+0000 7f65af400700 1 -- 172.21.15.89:0/4008379988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c0094a0 con 0x7f65b015c700 2024-04-02T09:13:17.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.042+0000 7f65af400700 1 --2- 172.21.15.89:0/4008379988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b015c700 0x7f65b0150c60 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f65a800cad0 tx=0x7f65a800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:17.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.043+0000 7f65a6a00700 1 -- 172.21.15.89:0/4008379988 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a800a9a0 con 0x7f65b015c700 2024-04-02T09:13:17.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.043+0000 7f65a6a00700 1 -- 172.21.15.89:0/4008379988 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a800ab00 con 0x7f65b015c700 2024-04-02T09:13:17.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.043+0000 7f65a6a00700 1 -- 172.21.15.89:0/4008379988 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a800ae90 con 0x7f65b015c700 2024-04-02T09:13:17.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.043+0000 7f65b5c00700 1 -- 172.21.15.89:0/4008379988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b0156020 con 0x7f65b015c700 2024-04-02T09:13:17.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.043+0000 7f65b5c00700 1 -- 172.21.15.89:0/4008379988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b014c720 con 0x7f65b015c700 2024-04-02T09:13:17.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.044+0000 7f65b5c00700 1 -- 172.21.15.89:0/4008379988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b0067720 con 0x7f65b015c700 2024-04-02T09:13:17.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.045+0000 7f65a6a00700 1 -- 172.21.15.89:0/4008379988 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f65a801b3b0 con 0x7f65b015c700 2024-04-02T09:13:17.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.045+0000 7f65a6a00700 1 --2- 172.21.15.89:0/4008379988 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f659006aed0 0x7f659006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:17.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.046+0000 7f65a6a00700 1 -- 172.21.15.89:0/4008379988 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7072+0+0 (secure 0 0 0) 0x7f65a8089840 con 0x7f65b015c700 2024-04-02T09:13:17.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.046+0000 7f65aea00700 1 --2- 172.21.15.89:0/4008379988 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f659006aed0 0x7f659006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:17.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.046+0000 7f65aea00700 1 --2- 172.21.15.89:0/4008379988 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f659006aed0 0x7f659006d390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f65a0007fd0 tx=0x7f65a0009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:17.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.050+0000 7f65a6a00700 1 -- 172.21.15.89:0/4008379988 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f65a8054180 con 0x7f65b015c700 2024-04-02T09:13:17.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.252+0000 7f65b5c00700 1 -- 172.21.15.89:0/4008379988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f65b014cfa0 con 0x7f65b015c700 2024-04-02T09:13:17.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.817+0000 7f65a6a00700 1 -- 172.21.15.89:0/4008379988 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v234) v1 ==== 190+0+0 (secure 0 0 0) 0x7f65a80577a0 con 0x7f65b015c700 2024-04-02T09:13:17.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.819+0000 7f65b5c00700 1 -- 172.21.15.89:0/4008379988 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f659006aed0 msgr2=0x7f659006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:17.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.819+0000 7f65b5c00700 1 --2- 172.21.15.89:0/4008379988 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f659006aed0 0x7f659006d390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f65a0007fd0 tx=0x7f65a0009040 comp rx=0 tx=0).stop 2024-04-02T09:13:17.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.819+0000 7f65b5c00700 1 -- 172.21.15.89:0/4008379988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b015c700 msgr2=0x7f65b0150c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:17.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.819+0000 7f65b5c00700 1 --2- 172.21.15.89:0/4008379988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b015c700 0x7f65b0150c60 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f65a800cad0 tx=0x7f65a800cde0 comp rx=0 tx=0).stop 2024-04-02T09:13:17.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.820+0000 7f65b5c00700 1 -- 172.21.15.89:0/4008379988 shutdown_connections 2024-04-02T09:13:17.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.820+0000 7f65b5c00700 1 --2- 172.21.15.89:0/4008379988 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f659006aed0 0x7f659006d390 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:17.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.820+0000 7f65b5c00700 1 --2- 172.21.15.89:0/4008379988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65b00ae4a0 0x7f65b0098560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:17.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.820+0000 7f65b5c00700 1 --2- 172.21.15.89:0/4008379988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65b015c700 0x7f65b0150c60 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:17.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.820+0000 7f65b5c00700 1 --2- 172.21.15.89:0/4008379988 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65b0155ac0 0x7f65b0150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:17.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.820+0000 7f65b5c00700 1 -- 172.21.15.89:0/4008379988 >> 172.21.15.89:0/4008379988 conn(0x7f65b00cf210 msgr2=0x7f65b00b0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:17.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.820+0000 7f65b5c00700 1 -- 172.21.15.89:0/4008379988 shutdown_connections 2024-04-02T09:13:17.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:17.820+0000 7f65b5c00700 1 -- 172.21.15.89:0/4008379988 wait complete. 2024-04-02T09:13:17.819 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'backup_fs_metadata' removed 2024-04-02T09:13:17.837 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-04-02T09:13:18.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.036+0000 7f913a800700 1 Processor -- start 2024-04-02T09:13:18.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.036+0000 7f913a800700 1 -- start start 2024-04-02T09:13:18.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.037+0000 7f913a800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f913415ddf0 0x7f91340ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:18.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.038+0000 7f913a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91340d28e0 0x7f913415f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:18.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.038+0000 7f913a800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91340ce8a0 con 0x7f913415e1d0 2024-04-02T09:13:18.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.038+0000 7f913a800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913415ec60 con 0x7f913415ddf0 2024-04-02T09:13:18.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.038+0000 7f913a800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91340cdc90 con 0x7f91340d28e0 2024-04-02T09:13:18.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.038+0000 7f9132a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f913415e1d0 0x7f91340d21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47678/0 (socket says 172.21.15.89:47678) 2024-04-02T09:13:18.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.038+0000 7f9132a00700 1 -- 172.21.15.89:0/2221571837 learned_addr learned my addr 172.21.15.89:0/2221571837 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:18.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.038+0000 7f9133e00700 1 --2- 172.21.15.89:0/2221571837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91340d28e0 0x7f913415f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:18.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.038+0000 7f9133400700 1 --2- 172.21.15.89:0/2221571837 >> v2:172.21.15.113:3300/0 conn(0x7f913415ddf0 0x7f91340ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:18.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.038+0000 7f9132000700 1 -- 172.21.15.89:0/2221571837 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881443346 0 0) 0x7f91340ce8a0 con 0x7f913415e1d0 2024-04-02T09:13:18.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.038+0000 7f9132000700 1 -- 172.21.15.89:0/2221571837 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91180035b0 con 0x7f913415e1d0 2024-04-02T09:13:18.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132000700 1 -- 172.21.15.89:0/2221571837 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1903248898 0 0) 0x7f91180035b0 con 0x7f913415e1d0 2024-04-02T09:13:18.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132000700 1 -- 172.21.15.89:0/2221571837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91340d28e0 msgr2=0x7f913415f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:18.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132000700 1 --2- 172.21.15.89:0/2221571837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91340d28e0 0x7f913415f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:18.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132000700 1 -- 172.21.15.89:0/2221571837 >> v2:172.21.15.113:3300/0 conn(0x7f913415ddf0 msgr2=0x7f91340ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:18.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132000700 1 --2- 172.21.15.89:0/2221571837 >> v2:172.21.15.113:3300/0 conn(0x7f913415ddf0 0x7f91340ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:18.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132000700 1 -- 172.21.15.89:0/2221571837 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91340cdf70 con 0x7f913415e1d0 2024-04-02T09:13:18.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132000700 1 -- 172.21.15.89:0/2221571837 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f9120001670 con 0x7f913415e1d0 2024-04-02T09:13:18.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132000700 1 -- 172.21.15.89:0/2221571837 >> v1:172.21.15.89:6789/0 conn(0x7f913415e1d0 legacy=0x7f91340d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:18.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132000700 1 --2- 172.21.15.89:0/2221571837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9118003a00 0x7f9118003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:18.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132000700 1 --2- 172.21.15.89:0/2221571837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9118004850 0x7f9118004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:18.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132000700 1 --2- 172.21.15.89:0/2221571837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91180051d0 0x7f911800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:18.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132000700 1 -- 172.21.15.89:0/2221571837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913415ec60 con 0x7f9118003a00 2024-04-02T09:13:18.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132000700 1 -- 172.21.15.89:0/2221571837 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91340cdc90 con 0x7f91180051d0 2024-04-02T09:13:18.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132000700 1 -- 172.21.15.89:0/2221571837 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9118004680 con 0x7f9118004850 2024-04-02T09:13:18.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9133400700 1 --2- 172.21.15.89:0/2221571837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9118003a00 0x7f9118003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:18.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9132a00700 1 --2- 172.21.15.89:0/2221571837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9118004850 0x7f9118004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:18.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9133e00700 1 --2- 172.21.15.89:0/2221571837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91180051d0 0x7f911800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:18.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9133400700 1 -- 172.21.15.89:0/2221571837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9118004850 msgr2=0x7f9118004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:18.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9133400700 1 --2- 172.21.15.89:0/2221571837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9118004850 0x7f9118004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:18.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9133400700 1 -- 172.21.15.89:0/2221571837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91180051d0 msgr2=0x7f911800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:18.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9133400700 1 --2- 172.21.15.89:0/2221571837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91180051d0 0x7f911800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:18.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.039+0000 7f9133400700 1 -- 172.21.15.89:0/2221571837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91180035b0 con 0x7f9118003a00 2024-04-02T09:13:18.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.040+0000 7f9133400700 1 --2- 172.21.15.89:0/2221571837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9118003a00 0x7f9118003e20 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7f9124009b90 tx=0x7f9118005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd159f05b8a3df79 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:18.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.040+0000 7f9132000700 1 -- 172.21.15.89:0/2221571837 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912400e6c0 con 0x7f9118003a00 2024-04-02T09:13:18.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.040+0000 7f9132000700 1 -- 172.21.15.89:0/2221571837 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912400e820 con 0x7f9118003a00 2024-04-02T09:13:18.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.040+0000 7f9132000700 1 -- 172.21.15.89:0/2221571837 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912400eb90 con 0x7f9118003a00 2024-04-02T09:13:18.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.040+0000 7f913a800700 1 -- 172.21.15.89:0/2221571837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9118003a00 msgr2=0x7f9118003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:18.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.040+0000 7f913a800700 1 --2- 172.21.15.89:0/2221571837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9118003a00 0x7f9118003e20 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7f9124009b90 tx=0x7f9118005880 comp rx=0 tx=0).stop 2024-04-02T09:13:18.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.040+0000 7f9133e00700 1 -- 172.21.15.89:0/2221571837 reap_dead start 2024-04-02T09:13:18.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.040+0000 7f913a800700 1 -- 172.21.15.89:0/2221571837 shutdown_connections 2024-04-02T09:13:18.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.040+0000 7f913a800700 1 -- 172.21.15.89:0/2221571837 >> 172.21.15.89:0/2221571837 conn(0x7f913408aff0 msgr2=0x7f91340d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:18.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.040+0000 7f913a800700 1 -- 172.21.15.89:0/2221571837 shutdown_connections 2024-04-02T09:13:18.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.040+0000 7f913a800700 1 -- 172.21.15.89:0/2221571837 wait complete. 2024-04-02T09:13:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.042+0000 7f913a800700 1 Processor -- start 2024-04-02T09:13:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.042+0000 7f913a800700 1 -- start start 2024-04-02T09:13:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.043+0000 7f913a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9118003a00 0x7f91340ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.043+0000 7f913a800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91340c7020 0x7f91340b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.043+0000 7f913a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91340b7a00 0x7f9134158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.043+0000 7f913a800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91340b3a50 con 0x7f9118003a00 2024-04-02T09:13:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.043+0000 7f913a800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91340c7e40 con 0x7f91340c7020 2024-04-02T09:13:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.043+0000 7f913a800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91340c75a0 con 0x7f91340b7a00 2024-04-02T09:13:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.043+0000 7f9133400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9118003a00 0x7f91340ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:18.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.043+0000 7f9133400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9118003a00 0x7f91340ca5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46798/0 (socket says 172.21.15.89:46798) 2024-04-02T09:13:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.043+0000 7f9133400700 1 -- 172.21.15.89:0/3025763392 learned_addr learned my addr 172.21.15.89:0/3025763392 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.044+0000 7f9132a00700 1 --2- 172.21.15.89:0/3025763392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91340c7020 0x7f91340b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.044+0000 7f9133e00700 1 --2- 172.21.15.89:0/3025763392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91340b7a00 0x7f9134158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.044+0000 7f9133400700 1 -- 172.21.15.89:0/3025763392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91340b7a00 msgr2=0x7f9134158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.044+0000 7f9133400700 1 --2- 172.21.15.89:0/3025763392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91340b7a00 0x7f9134158660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.044+0000 7f9133400700 1 -- 172.21.15.89:0/3025763392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91340c7020 msgr2=0x7f91340b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.044+0000 7f9133400700 1 --2- 172.21.15.89:0/3025763392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91340c7020 0x7f91340b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.044+0000 7f9133400700 1 -- 172.21.15.89:0/3025763392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91340c7880 con 0x7f9118003a00 2024-04-02T09:13:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.044+0000 7f9133400700 1 --2- 172.21.15.89:0/3025763392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9118003a00 0x7f91340ca5e0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7f91180045d0 tx=0x7f91240047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.044+0000 7f912b400700 1 -- 172.21.15.89:0/3025763392 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912400e6c0 con 0x7f9118003a00 2024-04-02T09:13:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.044+0000 7f912b400700 1 -- 172.21.15.89:0/3025763392 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9124003ab0 con 0x7f9118003a00 2024-04-02T09:13:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.044+0000 7f913a800700 1 -- 172.21.15.89:0/3025763392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91340c7b60 con 0x7f9118003a00 2024-04-02T09:13:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.044+0000 7f912b400700 1 -- 172.21.15.89:0/3025763392 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9124003e60 con 0x7f9118003a00 2024-04-02T09:13:18.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.044+0000 7f913a800700 1 -- 172.21.15.89:0/3025763392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91340b7e80 con 0x7f9118003a00 2024-04-02T09:13:18.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.045+0000 7f913a800700 1 -- 172.21.15.89:0/3025763392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9134067720 con 0x7f9118003a00 2024-04-02T09:13:18.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.046+0000 7f912b400700 1 -- 172.21.15.89:0/3025763392 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f9124013470 con 0x7f9118003a00 2024-04-02T09:13:18.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.047+0000 7f912b400700 1 --2- 172.21.15.89:0/3025763392 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f911406aed0 0x7f911406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:18.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.047+0000 7f912b400700 1 -- 172.21.15.89:0/3025763392 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f912408b410 con 0x7f9118003a00 2024-04-02T09:13:18.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.047+0000 7f9132a00700 1 --2- 172.21.15.89:0/3025763392 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f911406aed0 0x7f911406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:18.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.048+0000 7f9132a00700 1 --2- 172.21.15.89:0/3025763392 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f911406aed0 0x7f911406d390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f91340b84b0 tx=0x7f9120007040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:18.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.051+0000 7f912b400700 1 -- 172.21.15.89:0/3025763392 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9124056000 con 0x7f9118003a00 2024-04-02T09:13:18.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.254+0000 7f913a800700 1 -- 172.21.15.89:0/3025763392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f91340b80a0 con 0x7f9118003a00 2024-04-02T09:13:18.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.821+0000 7f912b400700 1 -- 172.21.15.89:0/3025763392 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v235) v1 ==== 178+0+0 (secure 0 0 0) 0x7f9124059620 con 0x7f9118003a00 2024-04-02T09:13:18.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.823+0000 7f913a800700 1 -- 172.21.15.89:0/3025763392 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f911406aed0 msgr2=0x7f911406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:18.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.823+0000 7f913a800700 1 --2- 172.21.15.89:0/3025763392 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f911406aed0 0x7f911406d390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f91340b84b0 tx=0x7f9120007040 comp rx=0 tx=0).stop 2024-04-02T09:13:18.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.823+0000 7f913a800700 1 -- 172.21.15.89:0/3025763392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9118003a00 msgr2=0x7f91340ca5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:18.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.823+0000 7f913a800700 1 --2- 172.21.15.89:0/3025763392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9118003a00 0x7f91340ca5e0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7f91180045d0 tx=0x7f91240047c0 comp rx=0 tx=0).stop 2024-04-02T09:13:18.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.823+0000 7f913a800700 1 -- 172.21.15.89:0/3025763392 shutdown_connections 2024-04-02T09:13:18.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.823+0000 7f913a800700 1 --2- 172.21.15.89:0/3025763392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91340b7a00 0x7f9134158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:18.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.823+0000 7f913a800700 1 --2- 172.21.15.89:0/3025763392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9118003a00 0x7f91340ca5e0 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:18.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.823+0000 7f913a800700 1 --2- 172.21.15.89:0/3025763392 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f911406aed0 0x7f911406d390 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:18.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.823+0000 7f913a800700 1 --2- 172.21.15.89:0/3025763392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91340c7020 0x7f91340b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:18.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.823+0000 7f913a800700 1 -- 172.21.15.89:0/3025763392 >> 172.21.15.89:0/3025763392 conn(0x7f913408aff0 msgr2=0x7f91340b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:18.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.824+0000 7f913a800700 1 -- 172.21.15.89:0/3025763392 shutdown_connections 2024-04-02T09:13:18.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:18.824+0000 7f913a800700 1 -- 172.21.15.89:0/3025763392 wait complete. 2024-04-02T09:13:18.823 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'backup_fs_data' removed 2024-04-02T09:13:18.841 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T09:13:19.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.037+0000 7fccf7200700 1 Processor -- start 2024-04-02T09:13:19.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.037+0000 7fccf7200700 1 -- start start 2024-04-02T09:13:19.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.038+0000 7fccf7200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fccf015d9d0 0x7fccf00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.038+0000 7fccf7200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccf00d2540 0x7fccf00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.038+0000 7fccf7200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf0161420 con 0x7fccf00d16a0 2024-04-02T09:13:19.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fccf7200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf015edc0 con 0x7fccf015d9d0 2024-04-02T09:13:19.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fccf7200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf0161700 con 0x7fccf00d2540 2024-04-02T09:13:19.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fccef400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fccf00d16a0 0x7fccf00d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48378/0 (socket says 172.21.15.89:48378) 2024-04-02T09:13:19.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fccef400700 1 -- 172.21.15.89:0/3190908827 learned_addr learned my addr 172.21.15.89:0/3190908827 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:19.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fccf5200700 1 --2- 172.21.15.89:0/3190908827 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccf00d2540 0x7fccf00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fccefe00700 1 --2- 172.21.15.89:0/3190908827 >> v2:172.21.15.113:3300/0 conn(0x7fccf015d9d0 0x7fccf00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fcceea00700 1 -- 172.21.15.89:0/3190908827 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864981130 0 0) 0x7fccf0161420 con 0x7fccf00d16a0 2024-04-02T09:13:19.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fcceea00700 1 -- 172.21.15.89:0/3190908827 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccd40035b0 con 0x7fccf00d16a0 2024-04-02T09:13:19.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fcceea00700 1 -- 172.21.15.89:0/3190908827 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2226367044 0 0) 0x7fccd40035b0 con 0x7fccf00d16a0 2024-04-02T09:13:19.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fcceea00700 1 -- 172.21.15.89:0/3190908827 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccf00d2540 msgr2=0x7fccf00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:19.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fcceea00700 1 --2- 172.21.15.89:0/3190908827 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccf00d2540 0x7fccf00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fcceea00700 1 -- 172.21.15.89:0/3190908827 >> v2:172.21.15.113:3300/0 conn(0x7fccf015d9d0 msgr2=0x7fccf00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:19.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fcceea00700 1 --2- 172.21.15.89:0/3190908827 >> v2:172.21.15.113:3300/0 conn(0x7fccf015d9d0 0x7fccf00d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fcceea00700 1 -- 172.21.15.89:0/3190908827 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf01619e0 con 0x7fccf00d16a0 2024-04-02T09:13:19.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.039+0000 7fcceea00700 1 -- 172.21.15.89:0/3190908827 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fccdc0017f0 con 0x7fccf00d16a0 2024-04-02T09:13:19.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fcceea00700 1 -- 172.21.15.89:0/3190908827 >> v1:172.21.15.89:6789/0 conn(0x7fccf00d16a0 legacy=0x7fccf00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fcceea00700 1 --2- 172.21.15.89:0/3190908827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd4003a00 0x7fccd4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fcceea00700 1 --2- 172.21.15.89:0/3190908827 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccf00d2540 0x7fccd4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fcceea00700 1 --2- 172.21.15.89:0/3190908827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd4004d90 0x7fccd4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fcceea00700 1 -- 172.21.15.89:0/3190908827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf0161700 con 0x7fccd4004d90 2024-04-02T09:13:19.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fcceea00700 1 -- 172.21.15.89:0/3190908827 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd400ba10 con 0x7fccd4003a00 2024-04-02T09:13:19.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fcceea00700 1 -- 172.21.15.89:0/3190908827 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd400bb50 con 0x7fccf00d2540 2024-04-02T09:13:19.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fccf5200700 1 --2- 172.21.15.89:0/3190908827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd4004d90 0x7fccd4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fccef400700 1 --2- 172.21.15.89:0/3190908827 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccf00d2540 0x7fccd4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fccefe00700 1 --2- 172.21.15.89:0/3190908827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd4003a00 0x7fccd4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fccf5200700 1 -- 172.21.15.89:0/3190908827 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccf00d2540 msgr2=0x7fccd4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:19.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fccf5200700 1 --2- 172.21.15.89:0/3190908827 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccf00d2540 0x7fccd4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fccf5200700 1 -- 172.21.15.89:0/3190908827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd4003a00 msgr2=0x7fccd4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:19.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fccf5200700 1 --2- 172.21.15.89:0/3190908827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd4003a00 0x7fccd4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.040+0000 7fccf5200700 1 -- 172.21.15.89:0/3190908827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd40035b0 con 0x7fccd4004d90 2024-04-02T09:13:19.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.041+0000 7fccf5200700 1 --2- 172.21.15.89:0/3190908827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd4004d90 0x7fccd4005210 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fccf00ca5a0 tx=0x7fccd4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a37b7900709eaee1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:19.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.041+0000 7fcceea00700 1 -- 172.21.15.89:0/3190908827 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce800ebd0 con 0x7fccd4004d90 2024-04-02T09:13:19.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.041+0000 7fcceea00700 1 -- 172.21.15.89:0/3190908827 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce800f040 con 0x7fccd4004d90 2024-04-02T09:13:19.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.041+0000 7fcceea00700 1 -- 172.21.15.89:0/3190908827 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce8014480 con 0x7fccd4004d90 2024-04-02T09:13:19.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.041+0000 7fccf7200700 1 -- 172.21.15.89:0/3190908827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd4004d90 msgr2=0x7fccd4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.041+0000 7fccf7200700 1 --2- 172.21.15.89:0/3190908827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd4004d90 0x7fccd4005210 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fccf00ca5a0 tx=0x7fccd4005880 comp rx=0 tx=0).stop 2024-04-02T09:13:19.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.041+0000 7fccf5200700 1 -- 172.21.15.89:0/3190908827 reap_dead start 2024-04-02T09:13:19.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.041+0000 7fccf7200700 1 -- 172.21.15.89:0/3190908827 shutdown_connections 2024-04-02T09:13:19.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.041+0000 7fccf7200700 1 -- 172.21.15.89:0/3190908827 >> 172.21.15.89:0/3190908827 conn(0x7fccf008aff0 msgr2=0x7fccf00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:19.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.042+0000 7fccf7200700 1 -- 172.21.15.89:0/3190908827 shutdown_connections 2024-04-02T09:13:19.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.042+0000 7fccf7200700 1 -- 172.21.15.89:0/3190908827 wait complete. 2024-04-02T09:13:19.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.043+0000 7fccf7200700 1 Processor -- start 2024-04-02T09:13:19.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.044+0000 7fccf7200700 1 -- start start 2024-04-02T09:13:19.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.044+0000 7fccf7200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd4004d90 0x7fccf015eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.044+0000 7fccf7200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccf015fed0 0x7fccf0156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.044+0000 7fccf7200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccf00b1c70 0x7fccf00b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.044+0000 7fccf7200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf0160350 con 0x7fccf015fed0 2024-04-02T09:13:19.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.044+0000 7fccf7200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf00b2630 con 0x7fccd4004d90 2024-04-02T09:13:19.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.044+0000 7fccf7200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf00ca5f0 con 0x7fccf00b1c70 2024-04-02T09:13:19.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.045+0000 7fccef400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccf015fed0 0x7fccf0156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.045+0000 7fccef400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccf015fed0 0x7fccf0156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54328/0 (socket says 172.21.15.89:54328) 2024-04-02T09:13:19.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.045+0000 7fccef400700 1 -- 172.21.15.89:0/66729377 learned_addr learned my addr 172.21.15.89:0/66729377 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:19.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.045+0000 7fccefe00700 1 --2- 172.21.15.89:0/66729377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd4004d90 0x7fccf015eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.045+0000 7fccf5200700 1 --2- 172.21.15.89:0/66729377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccf00b1c70 0x7fccf00b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.045+0000 7fccef400700 1 -- 172.21.15.89:0/66729377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccf00b1c70 msgr2=0x7fccf00b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.045+0000 7fccef400700 1 --2- 172.21.15.89:0/66729377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccf00b1c70 0x7fccf00b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.045+0000 7fccef400700 1 -- 172.21.15.89:0/66729377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd4004d90 msgr2=0x7fccf015eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.045+0000 7fccef400700 1 --2- 172.21.15.89:0/66729377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd4004d90 0x7fccf015eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.045+0000 7fccef400700 1 -- 172.21.15.89:0/66729377 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf00ca890 con 0x7fccf015fed0 2024-04-02T09:13:19.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.045+0000 7fccef400700 1 --2- 172.21.15.89:0/66729377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccf015fed0 0x7fccf0156960 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00f940 tx=0x7fccdc00fc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:19.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.046+0000 7fcce7e00700 1 -- 172.21.15.89:0/66729377 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc0070d0 con 0x7fccf015fed0 2024-04-02T09:13:19.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.046+0000 7fcce7e00700 1 -- 172.21.15.89:0/66729377 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccdc007230 con 0x7fccf015fed0 2024-04-02T09:13:19.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.046+0000 7fcce7e00700 1 -- 172.21.15.89:0/66729377 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc0075e0 con 0x7fccf015fed0 2024-04-02T09:13:19.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.046+0000 7fccf7200700 1 -- 172.21.15.89:0/66729377 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf00cab70 con 0x7fccf015fed0 2024-04-02T09:13:19.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.046+0000 7fccf7200700 1 -- 172.21.15.89:0/66729377 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf00cb130 con 0x7fccf015fed0 2024-04-02T09:13:19.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.047+0000 7fccf7200700 1 -- 172.21.15.89:0/66729377 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf00d1ca0 con 0x7fccf015fed0 2024-04-02T09:13:19.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.047+0000 7fcce7e00700 1 -- 172.21.15.89:0/66729377 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fccdc005830 con 0x7fccf015fed0 2024-04-02T09:13:19.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.048+0000 7fcce7e00700 1 --2- 172.21.15.89:0/66729377 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fccd006af00 0x7fccd006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.048+0000 7fcce7e00700 1 -- 172.21.15.89:0/66729377 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6208+0+0 (secure 0 0 0) 0x7fccdc089930 con 0x7fccf015fed0 2024-04-02T09:13:19.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.049+0000 7fccefe00700 1 --2- 172.21.15.89:0/66729377 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fccd006af00 0x7fccd006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.049+0000 7fccefe00700 1 --2- 172.21.15.89:0/66729377 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fccd006af00 0x7fccd006d3c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009c60 tx=0x7fcce0009210 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:19.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.053+0000 7fcce7e00700 1 -- 172.21.15.89:0/66729377 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fccdc054530 con 0x7fccf015fed0 2024-04-02T09:13:19.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.263+0000 7fccf7200700 1 -- 172.21.15.89:0/66729377 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fccf0067720 con 0x7fccf015fed0 2024-04-02T09:13:19.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.268+0000 7fcce7e00700 1 -- 172.21.15.89:0/66729377 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v79) v1 ==== 115+0+0 (secure 0 0 0) 0x7fccdc057b50 con 0x7fccf015fed0 2024-04-02T09:13:19.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.270+0000 7fccf7200700 1 -- 172.21.15.89:0/66729377 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fccd006af00 msgr2=0x7fccd006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.270+0000 7fccf7200700 1 --2- 172.21.15.89:0/66729377 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fccd006af00 0x7fccd006d3c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009c60 tx=0x7fcce0009210 comp rx=0 tx=0).stop 2024-04-02T09:13:19.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.270+0000 7fccf7200700 1 -- 172.21.15.89:0/66729377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccf015fed0 msgr2=0x7fccf0156960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.270+0000 7fccf7200700 1 --2- 172.21.15.89:0/66729377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccf015fed0 0x7fccf0156960 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00f940 tx=0x7fccdc00fc50 comp rx=0 tx=0).stop 2024-04-02T09:13:19.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.270+0000 7fccf7200700 1 -- 172.21.15.89:0/66729377 shutdown_connections 2024-04-02T09:13:19.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.270+0000 7fccf7200700 1 --2- 172.21.15.89:0/66729377 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fccd006af00 0x7fccd006d3c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.270+0000 7fccf7200700 1 --2- 172.21.15.89:0/66729377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccd4004d90 0x7fccf015eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.270+0000 7fccf7200700 1 --2- 172.21.15.89:0/66729377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccf00b1c70 0x7fccf00b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.270+0000 7fccf7200700 1 --2- 172.21.15.89:0/66729377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccf015fed0 0x7fccf0156960 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.270+0000 7fccf7200700 1 -- 172.21.15.89:0/66729377 >> 172.21.15.89:0/66729377 conn(0x7fccf008aff0 msgr2=0x7fccf00d1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:19.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.270+0000 7fccf7200700 1 -- 172.21.15.89:0/66729377 shutdown_connections 2024-04-02T09:13:19.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.270+0000 7fccf7200700 1 -- 172.21.15.89:0/66729377 wait complete. 2024-04-02T09:13:19.291 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T09:13:19.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.488+0000 7fed5ae00700 1 Processor -- start 2024-04-02T09:13:19.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.488+0000 7fed5ae00700 1 -- start start 2024-04-02T09:13:19.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.489+0000 7fed5ae00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed541611b0 0x7fed5415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.489+0000 7fed5ae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed540cee10 0x7fed54161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.489+0000 7fed5ae00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed540ce270 con 0x7fed5415f4d0 2024-04-02T09:13:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.489+0000 7fed5ae00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed5415b600 con 0x7fed541611b0 2024-04-02T09:13:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.489+0000 7fed5ae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed540d2760 con 0x7fed540cee10 2024-04-02T09:13:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.490+0000 7fed53e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fed541611b0 0x7fed5415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.490+0000 7fed53400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fed5415f4d0 0x7fed540ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48388/0 (socket says 172.21.15.89:48388) 2024-04-02T09:13:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.490+0000 7fed53400700 1 -- 172.21.15.89:0/1987654181 learned_addr learned my addr 172.21.15.89:0/1987654181 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.490+0000 7fed58e00700 1 --2- 172.21.15.89:0/1987654181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed540cee10 0x7fed54161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.490+0000 7fed53e00700 1 -- 172.21.15.89:0/1987654181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed540cee10 msgr2=0x7fed54161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.490+0000 7fed53e00700 1 --2- 172.21.15.89:0/1987654181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed540cee10 0x7fed54161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.490+0000 7fed53e00700 1 -- 172.21.15.89:0/1987654181 >> v1:172.21.15.89:6789/0 conn(0x7fed5415f4d0 legacy=0x7fed540ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.490+0000 7fed53e00700 1 -- 172.21.15.89:0/1987654181 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed540d2a40 con 0x7fed541611b0 2024-04-02T09:13:19.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.490+0000 7fed52a00700 1 -- 172.21.15.89:0/1987654181 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800213666 0 0) 0x7fed540ce270 con 0x7fed5415f4d0 2024-04-02T09:13:19.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.490+0000 7fed53e00700 1 --2- 172.21.15.89:0/1987654181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed541611b0 0x7fed5415ef90 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fed44009840 tx=0x7fed44009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed71e21de9a47c5e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:19.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.491+0000 7fed52a00700 1 -- 172.21.15.89:0/1987654181 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44011830 con 0x7fed541611b0 2024-04-02T09:13:19.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.491+0000 7fed52a00700 1 -- 172.21.15.89:0/1987654181 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed44011990 con 0x7fed541611b0 2024-04-02T09:13:19.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.491+0000 7fed52a00700 1 -- 172.21.15.89:0/1987654181 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44011d20 con 0x7fed541611b0 2024-04-02T09:13:19.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.491+0000 7fed5ae00700 1 -- 172.21.15.89:0/1987654181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed541611b0 msgr2=0x7fed5415ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.491+0000 7fed5ae00700 1 --2- 172.21.15.89:0/1987654181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed541611b0 0x7fed5415ef90 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fed44009840 tx=0x7fed44009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:19.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.491+0000 7fed5ae00700 1 -- 172.21.15.89:0/1987654181 shutdown_connections 2024-04-02T09:13:19.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.491+0000 7fed5ae00700 1 --2- 172.21.15.89:0/1987654181 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed540cee10 0x7fed54161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.491+0000 7fed5ae00700 1 --2- 172.21.15.89:0/1987654181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed541611b0 0x7fed5415ef90 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.491+0000 7fed5ae00700 1 -- 172.21.15.89:0/1987654181 >> 172.21.15.89:0/1987654181 conn(0x7fed5408aff0 msgr2=0x7fed540d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:19.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.491+0000 7fed5ae00700 1 -- 172.21.15.89:0/1987654181 shutdown_connections 2024-04-02T09:13:19.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.491+0000 7fed5ae00700 1 -- 172.21.15.89:0/1987654181 wait complete. 2024-04-02T09:13:19.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.493+0000 7fed5ae00700 1 Processor -- start 2024-04-02T09:13:19.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.493+0000 7fed5ae00700 1 -- start start 2024-04-02T09:13:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.494+0000 7fed5ae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed540cee10 0x7fed54153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.494+0000 7fed5ae00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed5415f4d0 0x7fed540b75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.494+0000 7fed5ae00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed541611b0 0x7fed540cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.494+0000 7fed5ae00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed540b71b0 con 0x7fed5415f4d0 2024-04-02T09:13:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.494+0000 7fed5ae00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed541544d0 con 0x7fed541611b0 2024-04-02T09:13:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.494+0000 7fed5ae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed540b6530 con 0x7fed540cee10 2024-04-02T09:13:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.494+0000 7fed53400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed5415f4d0 0x7fed540b75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.494+0000 7fed53400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed5415f4d0 0x7fed540b75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54344/0 (socket says 172.21.15.89:54344) 2024-04-02T09:13:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.494+0000 7fed53400700 1 -- 172.21.15.89:0/1197378131 learned_addr learned my addr 172.21.15.89:0/1197378131 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.494+0000 7fed53e00700 1 --2- 172.21.15.89:0/1197378131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed540cee10 0x7fed54153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.495+0000 7fed58e00700 1 --2- 172.21.15.89:0/1197378131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed541611b0 0x7fed540cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.495+0000 7fed53400700 1 -- 172.21.15.89:0/1197378131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed540cee10 msgr2=0x7fed54153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.495+0000 7fed53400700 1 --2- 172.21.15.89:0/1197378131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed540cee10 0x7fed54153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.495+0000 7fed53400700 1 -- 172.21.15.89:0/1197378131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed541611b0 msgr2=0x7fed540cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.495+0000 7fed53400700 1 --2- 172.21.15.89:0/1197378131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed541611b0 0x7fed540cafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.495+0000 7fed53400700 1 -- 172.21.15.89:0/1197378131 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed440094a0 con 0x7fed5415f4d0 2024-04-02T09:13:19.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.495+0000 7fed53400700 1 --2- 172.21.15.89:0/1197378131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed5415f4d0 0x7fed540b75d0 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fed4000c520 tx=0x7fed4000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:19.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.495+0000 7fed4be00700 1 -- 172.21.15.89:0/1197378131 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40004c80 con 0x7fed5415f4d0 2024-04-02T09:13:19.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.495+0000 7fed4be00700 1 -- 172.21.15.89:0/1197378131 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed40004de0 con 0x7fed5415f4d0 2024-04-02T09:13:19.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.495+0000 7fed4be00700 1 -- 172.21.15.89:0/1197378131 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40007310 con 0x7fed5415f4d0 2024-04-02T09:13:19.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.495+0000 7fed5ae00700 1 -- 172.21.15.89:0/1197378131 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed540b6810 con 0x7fed5415f4d0 2024-04-02T09:13:19.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.495+0000 7fed5ae00700 1 -- 172.21.15.89:0/1197378131 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed540b6dd0 con 0x7fed5415f4d0 2024-04-02T09:13:19.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.496+0000 7fed5ae00700 1 -- 172.21.15.89:0/1197378131 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed540b72f0 con 0x7fed5415f4d0 2024-04-02T09:13:19.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.497+0000 7fed4be00700 1 -- 172.21.15.89:0/1197378131 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fed40007470 con 0x7fed5415f4d0 2024-04-02T09:13:19.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.498+0000 7fed4be00700 1 --2- 172.21.15.89:0/1197378131 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed3406aed0 0x7fed3406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.498+0000 7fed4be00700 1 -- 172.21.15.89:0/1197378131 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6208+0+0 (secure 0 0 0) 0x7fed40089510 con 0x7fed5415f4d0 2024-04-02T09:13:19.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.498+0000 7fed53e00700 1 --2- 172.21.15.89:0/1197378131 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed3406aed0 0x7fed3406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.499+0000 7fed53e00700 1 --2- 172.21.15.89:0/1197378131 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed3406aed0 0x7fed3406d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fed440090b0 tx=0x7fed4401c040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:19.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.503+0000 7fed4be00700 1 -- 172.21.15.89:0/1197378131 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fed540b72f0 con 0x7fed5415f4d0 2024-04-02T09:13:19.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.717+0000 7fed5ae00700 1 -- 172.21.15.89:0/1197378131 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fed540b72f0 con 0x7fed5415f4d0 2024-04-02T09:13:19.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.722+0000 7fed4be00700 1 -- 172.21.15.89:0/1197378131 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v80) v1 ==== 120+0+0 (secure 0 0 0) 0x7fed540b72f0 con 0x7fed5415f4d0 2024-04-02T09:13:19.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.724+0000 7fed5ae00700 1 -- 172.21.15.89:0/1197378131 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed3406aed0 msgr2=0x7fed3406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.724+0000 7fed5ae00700 1 --2- 172.21.15.89:0/1197378131 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed3406aed0 0x7fed3406d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fed440090b0 tx=0x7fed4401c040 comp rx=0 tx=0).stop 2024-04-02T09:13:19.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.724+0000 7fed5ae00700 1 -- 172.21.15.89:0/1197378131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed5415f4d0 msgr2=0x7fed540b75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.724+0000 7fed5ae00700 1 --2- 172.21.15.89:0/1197378131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed5415f4d0 0x7fed540b75d0 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fed4000c520 tx=0x7fed4000c830 comp rx=0 tx=0).stop 2024-04-02T09:13:19.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.724+0000 7fed5ae00700 1 -- 172.21.15.89:0/1197378131 shutdown_connections 2024-04-02T09:13:19.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.724+0000 7fed5ae00700 1 --2- 172.21.15.89:0/1197378131 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed3406aed0 0x7fed3406d390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.724+0000 7fed5ae00700 1 --2- 172.21.15.89:0/1197378131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed541611b0 0x7fed540cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.724+0000 7fed5ae00700 1 --2- 172.21.15.89:0/1197378131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed540cee10 0x7fed54153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.724+0000 7fed5ae00700 1 --2- 172.21.15.89:0/1197378131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed5415f4d0 0x7fed540b75d0 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.725+0000 7fed5ae00700 1 -- 172.21.15.89:0/1197378131 >> 172.21.15.89:0/1197378131 conn(0x7fed5408aff0 msgr2=0x7fed540ce700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:19.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.725+0000 7fed5ae00700 1 -- 172.21.15.89:0/1197378131 shutdown_connections 2024-04-02T09:13:19.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.725+0000 7fed5ae00700 1 -- 172.21.15.89:0/1197378131 wait complete. 2024-04-02T09:13:19.743 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T09:13:19.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.938+0000 7f5df1c00700 1 Processor -- start 2024-04-02T09:13:19.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.939+0000 7f5df1c00700 1 -- start start 2024-04-02T09:13:19.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.940+0000 7f5df1c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5dec1611b0 0x7f5dec15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.940+0000 7f5df1c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dec15f4d0 0x7f5dec0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.940+0000 7f5df1c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec0ce270 con 0x7f5dec0d21f0 2024-04-02T09:13:19.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.940+0000 7f5df1c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dec15b600 con 0x7f5dec1611b0 2024-04-02T09:13:19.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.940+0000 7f5df1c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec0d2760 con 0x7f5dec15f4d0 2024-04-02T09:13:19.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.940+0000 7f5deaa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5dec1611b0 0x7f5dec15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.940+0000 7f5deaa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5dec1611b0 0x7f5dec15ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54352/0 (socket says 172.21.15.89:54352) 2024-04-02T09:13:19.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.940+0000 7f5deaa00700 1 -- 172.21.15.89:0/2770907454 learned_addr learned my addr 172.21.15.89:0/2770907454 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:19.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.940+0000 7f5de3e00700 1 --2- 172.21.15.89:0/2770907454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dec15f4d0 0x7f5dec0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.941+0000 7f5deaa00700 1 -- 172.21.15.89:0/2770907454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dec15f4d0 msgr2=0x7f5dec0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:19.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.941+0000 7f5deaa00700 1 --2- 172.21.15.89:0/2770907454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dec15f4d0 0x7f5dec0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.941+0000 7f5deaa00700 1 -- 172.21.15.89:0/2770907454 >> v1:172.21.15.113:6789/0 conn(0x7f5dec0d21f0 legacy=0x7f5dec161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.941+0000 7f5deaa00700 1 -- 172.21.15.89:0/2770907454 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec0d2a40 con 0x7f5dec1611b0 2024-04-02T09:13:19.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.941+0000 7f5deaa00700 1 --2- 172.21.15.89:0/2770907454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dec1611b0 0x7f5dec15ef90 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc0098f0 tx=0x7f5ddc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9206e657be19b1b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:19.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.941+0000 7f5dea000700 1 -- 172.21.15.89:0/2770907454 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc00b8a0 con 0x7f5dec1611b0 2024-04-02T09:13:19.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.941+0000 7f5dea000700 1 -- 172.21.15.89:0/2770907454 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ddc00ba00 con 0x7f5dec1611b0 2024-04-02T09:13:19.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.941+0000 7f5dea000700 1 -- 172.21.15.89:0/2770907454 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc00bd90 con 0x7f5dec1611b0 2024-04-02T09:13:19.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.942+0000 7f5df1c00700 1 -- 172.21.15.89:0/2770907454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dec1611b0 msgr2=0x7f5dec15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.942+0000 7f5df1c00700 1 --2- 172.21.15.89:0/2770907454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dec1611b0 0x7f5dec15ef90 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc0098f0 tx=0x7f5ddc009c00 comp rx=0 tx=0).stop 2024-04-02T09:13:19.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.942+0000 7f5df1c00700 1 -- 172.21.15.89:0/2770907454 shutdown_connections 2024-04-02T09:13:19.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.942+0000 7f5df1c00700 1 --2- 172.21.15.89:0/2770907454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dec1611b0 0x7f5dec15ef90 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.942+0000 7f5df1c00700 1 --2- 172.21.15.89:0/2770907454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dec15f4d0 0x7f5dec0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.942+0000 7f5df1c00700 1 -- 172.21.15.89:0/2770907454 >> 172.21.15.89:0/2770907454 conn(0x7f5dec08aff0 msgr2=0x7f5dec0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:19.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.942+0000 7f5df1c00700 1 -- 172.21.15.89:0/2770907454 shutdown_connections 2024-04-02T09:13:19.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.942+0000 7f5df1c00700 1 -- 172.21.15.89:0/2770907454 wait complete. 2024-04-02T09:13:19.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.944+0000 7f5df1c00700 1 Processor -- start 2024-04-02T09:13:19.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.944+0000 7f5df1c00700 1 -- start start 2024-04-02T09:13:19.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5df1c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dec0d21f0 0x7f5dec1f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5df1c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dec15f4d0 0x7f5dec1f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5df1c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dec1611b0 0x7f5dec09d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5df1c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec0caf10 con 0x7f5dec15f4d0 2024-04-02T09:13:19.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5df1c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec0ca600 con 0x7f5dec0d21f0 2024-04-02T09:13:19.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5df1c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec09e130 con 0x7f5dec1611b0 2024-04-02T09:13:19.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5de3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dec15f4d0 0x7f5dec1f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5de3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dec15f4d0 0x7f5dec1f69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54360/0 (socket says 172.21.15.89:54360) 2024-04-02T09:13:19.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5de3e00700 1 -- 172.21.15.89:0/3189067658 learned_addr learned my addr 172.21.15.89:0/3189067658 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:19.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5deaa00700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dec0d21f0 0x7f5dec1f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5deb400700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dec1611b0 0x7f5dec09d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5de3e00700 1 -- 172.21.15.89:0/3189067658 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dec1611b0 msgr2=0x7f5dec09d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5de3e00700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dec1611b0 0x7f5dec09d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5de3e00700 1 -- 172.21.15.89:0/3189067658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dec0d21f0 msgr2=0x7f5dec1f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:19.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5de3e00700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dec0d21f0 0x7f5dec1f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:19.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5de3e00700 1 -- 172.21.15.89:0/3189067658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ddc009550 con 0x7f5dec15f4d0 2024-04-02T09:13:19.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.945+0000 7f5deaa00700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dec0d21f0 0x7f5dec1f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:19.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.946+0000 7f5de3e00700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dec15f4d0 0x7f5dec1f69c0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400e810 tx=0x7f5dd400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:19.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.946+0000 7f5de2a00700 1 -- 172.21.15.89:0/3189067658 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4013610 con 0x7f5dec15f4d0 2024-04-02T09:13:19.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.946+0000 7f5de2a00700 1 -- 172.21.15.89:0/3189067658 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd4013770 con 0x7f5dec15f4d0 2024-04-02T09:13:19.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.946+0000 7f5df1c00700 1 -- 172.21.15.89:0/3189067658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dec0ca850 con 0x7f5dec15f4d0 2024-04-02T09:13:19.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.946+0000 7f5de2a00700 1 -- 172.21.15.89:0/3189067658 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4013b20 con 0x7f5dec15f4d0 2024-04-02T09:13:19.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.946+0000 7f5df1c00700 1 -- 172.21.15.89:0/3189067658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dec09d920 con 0x7f5dec15f4d0 2024-04-02T09:13:19.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.947+0000 7f5df1c00700 1 -- 172.21.15.89:0/3189067658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dec15d610 con 0x7f5dec15f4d0 2024-04-02T09:13:19.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.948+0000 7f5de2a00700 1 -- 172.21.15.89:0/3189067658 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5dd4013c80 con 0x7f5dec15f4d0 2024-04-02T09:13:19.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.949+0000 7f5de2a00700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dc806aed0 0x7f5dc806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:19.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.949+0000 7f5de2a00700 1 -- 172.21.15.89:0/3189067658 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f5dd40896c0 con 0x7f5dec15f4d0 2024-04-02T09:13:19.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.950+0000 7f5deaa00700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dc806aed0 0x7f5dc806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:19.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.950+0000 7f5deaa00700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dc806aed0 0x7f5dc806d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc014040 tx=0x7f5ddc008030 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:19.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:19.954+0000 7f5de2a00700 1 -- 172.21.15.89:0/3189067658 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5dd4054250 con 0x7f5dec15f4d0 2024-04-02T09:13:20.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.153+0000 7f5df1c00700 1 -- 172.21.15.89:0/3189067658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f5dec067720 con 0x7f5dec15f4d0 2024-04-02T09:13:20.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.158+0000 7f5de2a00700 1 -- 172.21.15.89:0/3189067658 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v81) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5dd4057870 con 0x7f5dec15f4d0 2024-04-02T09:13:20.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.161+0000 7f5df1c00700 1 -- 172.21.15.89:0/3189067658 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dc806aed0 msgr2=0x7f5dc806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:20.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.161+0000 7f5df1c00700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dc806aed0 0x7f5dc806d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc014040 tx=0x7f5ddc008030 comp rx=0 tx=0).stop 2024-04-02T09:13:20.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.161+0000 7f5df1c00700 1 -- 172.21.15.89:0/3189067658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dec15f4d0 msgr2=0x7f5dec1f69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:20.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.161+0000 7f5df1c00700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dec15f4d0 0x7f5dec1f69c0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400e810 tx=0x7f5dd400ebd0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.161+0000 7f5df1c00700 1 -- 172.21.15.89:0/3189067658 shutdown_connections 2024-04-02T09:13:20.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.161+0000 7f5df1c00700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5dc806aed0 0x7f5dc806d390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.161+0000 7f5df1c00700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dec0d21f0 0x7f5dec1f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.161+0000 7f5df1c00700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5dec1611b0 0x7f5dec09d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.161+0000 7f5df1c00700 1 --2- 172.21.15.89:0/3189067658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dec15f4d0 0x7f5dec1f69c0 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.161+0000 7f5df1c00700 1 -- 172.21.15.89:0/3189067658 >> 172.21.15.89:0/3189067658 conn(0x7f5dec08aff0 msgr2=0x7f5dec15da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:20.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.161+0000 7f5df1c00700 1 -- 172.21.15.89:0/3189067658 shutdown_connections 2024-04-02T09:13:20.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.161+0000 7f5df1c00700 1 -- 172.21.15.89:0/3189067658 wait complete. 2024-04-02T09:13:20.177 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-04-02T09:13:20.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.372+0000 7f7207a00700 1 Processor -- start 2024-04-02T09:13:20.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.372+0000 7f7207a00700 1 -- start start 2024-04-02T09:13:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.373+0000 7f7207a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72000cb3f0 0x7f72000ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.373+0000 7f7207a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f72000ac600 0x7f720006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.373+0000 7f7207a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720015fd50 con 0x7f720015b5d0 2024-04-02T09:13:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.373+0000 7f7207a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f720015b9b0 con 0x7f72000ac600 2024-04-02T09:13:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.373+0000 7f7207a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72000aca70 con 0x7f72000cb3f0 2024-04-02T09:13:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.374+0000 7f7205000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f720015b5d0 0x7f72000ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48402/0 (socket says 172.21.15.89:48402) 2024-04-02T09:13:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.374+0000 7f7205000700 1 -- 172.21.15.89:0/65984014 learned_addr learned my addr 172.21.15.89:0/65984014 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.374+0000 7f7205a00700 1 --2- 172.21.15.89:0/65984014 >> v2:172.21.15.113:3300/0 conn(0x7f72000ac600 0x7f720006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.374+0000 7f71ffe00700 1 --2- 172.21.15.89:0/65984014 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72000cb3f0 0x7f72000ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.374+0000 7f71ff400700 1 -- 172.21.15.89:0/65984014 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415825438 0 0) 0x7f720015fd50 con 0x7f720015b5d0 2024-04-02T09:13:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.374+0000 7f71ff400700 1 -- 172.21.15.89:0/65984014 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71e80079c0 con 0x7f720015b5d0 2024-04-02T09:13:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.374+0000 7f71ff400700 1 -- 172.21.15.89:0/65984014 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1930413487 0 0) 0x7f71e80079c0 con 0x7f720015b5d0 2024-04-02T09:13:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.374+0000 7f71ff400700 1 -- 172.21.15.89:0/65984014 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72000cb3f0 msgr2=0x7f72000ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.374+0000 7f71ff400700 1 --2- 172.21.15.89:0/65984014 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72000cb3f0 0x7f72000ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.374+0000 7f71ff400700 1 -- 172.21.15.89:0/65984014 >> v2:172.21.15.113:3300/0 conn(0x7f72000ac600 msgr2=0x7f720006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.374+0000 7f71ff400700 1 --2- 172.21.15.89:0/65984014 >> v2:172.21.15.113:3300/0 conn(0x7f72000ac600 0x7f720006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ff400700 1 -- 172.21.15.89:0/65984014 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720015f140 con 0x7f720015b5d0 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ff400700 1 -- 172.21.15.89:0/65984014 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f71f4001670 con 0x7f720015b5d0 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ffe00700 1 --2- 172.21.15.89:0/65984014 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72000cb3f0 0x7f72000ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ff400700 1 -- 172.21.15.89:0/65984014 >> v1:172.21.15.89:6789/0 conn(0x7f720015b5d0 legacy=0x7f72000ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ff400700 1 --2- 172.21.15.89:0/65984014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71e8007e10 0x7f71e8008230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ff400700 1 --2- 172.21.15.89:0/65984014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71e8008c60 0x7f71e80090e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ff400700 1 --2- 172.21.15.89:0/65984014 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71e8009620 0x7f71e800fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ff400700 1 -- 172.21.15.89:0/65984014 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720015b9b0 con 0x7f71e8008c60 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ff400700 1 -- 172.21.15.89:0/65984014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72000aca70 con 0x7f71e8007e10 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ff400700 1 -- 172.21.15.89:0/65984014 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e8010360 con 0x7f71e8009620 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f7205a00700 1 --2- 172.21.15.89:0/65984014 >> v2:172.21.15.113:3300/0 conn(0x7f72000ac600 0x7f720006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ffe00700 1 --2- 172.21.15.89:0/65984014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71e8008c60 0x7f71e80090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f7205000700 1 --2- 172.21.15.89:0/65984014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71e8007e10 0x7f71e8008230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f7205a00700 1 --2- 172.21.15.89:0/65984014 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71e8009620 0x7f71e800fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ffe00700 1 -- 172.21.15.89:0/65984014 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71e8009620 msgr2=0x7f71e800fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ffe00700 1 --2- 172.21.15.89:0/65984014 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71e8009620 0x7f71e800fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ffe00700 1 -- 172.21.15.89:0/65984014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71e8007e10 msgr2=0x7f71e8008230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ffe00700 1 --2- 172.21.15.89:0/65984014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71e8007e10 0x7f71e8008230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f71ffe00700 1 -- 172.21.15.89:0/65984014 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ec0094a0 con 0x7f71e8008c60 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.375+0000 7f7205a00700 1 --2- 172.21.15.89:0/65984014 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71e8009620 0x7f71e800fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.376+0000 7f71ffe00700 1 --2- 172.21.15.89:0/65984014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71e8008c60 0x7f71e80090e0 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f71ec009d30 tx=0x7f71e8009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603a15c909f1d20d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.376+0000 7f71ff400700 1 -- 172.21.15.89:0/65984014 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec009e80 con 0x7f71e8008c60 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.376+0000 7f71ff400700 1 -- 172.21.15.89:0/65984014 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ec00e9a0 con 0x7f71e8008c60 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.376+0000 7f71ff400700 1 -- 172.21.15.89:0/65984014 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec00ed10 con 0x7f71e8008c60 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.376+0000 7f7207a00700 1 -- 172.21.15.89:0/65984014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71e8008c60 msgr2=0x7f71e80090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.376+0000 7f7207a00700 1 --2- 172.21.15.89:0/65984014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71e8008c60 0x7f71e80090e0 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f71ec009d30 tx=0x7f71e8009c90 comp rx=0 tx=0).stop 2024-04-02T09:13:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.376+0000 7f7205a00700 1 -- 172.21.15.89:0/65984014 reap_dead start 2024-04-02T09:13:20.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.376+0000 7f7207a00700 1 -- 172.21.15.89:0/65984014 shutdown_connections 2024-04-02T09:13:20.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.376+0000 7f7207a00700 1 -- 172.21.15.89:0/65984014 >> 172.21.15.89:0/65984014 conn(0x7f72000cf190 msgr2=0x7f72000cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:20.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.377+0000 7f7207a00700 1 -- 172.21.15.89:0/65984014 shutdown_connections 2024-04-02T09:13:20.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.377+0000 7f7207a00700 1 -- 172.21.15.89:0/65984014 wait complete. 2024-04-02T09:13:20.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.379+0000 7f7207a00700 1 Processor -- start 2024-04-02T09:13:20.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.379+0000 7f7207a00700 1 -- start start 2024-04-02T09:13:20.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.380+0000 7f7207a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71e8008c60 0x7f7200152e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:20.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.380+0000 7f7207a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72000b3840 0x7f720015baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:20.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.380+0000 7f7207a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72000b3cc0 0x7f720015c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:20.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.380+0000 7f7207a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72000cda10 con 0x7f72000b3cc0 2024-04-02T09:13:20.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.380+0000 7f7207a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72000cdcf0 con 0x7f71e8008c60 2024-04-02T09:13:20.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.380+0000 7f7207a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72000cdfd0 con 0x7f72000b3840 2024-04-02T09:13:20.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.380+0000 7f7205a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72000b3cc0 0x7f720015c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:20.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.381+0000 7f7205a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72000b3cc0 0x7f720015c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54384/0 (socket says 172.21.15.89:54384) 2024-04-02T09:13:20.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.381+0000 7f7205a00700 1 -- 172.21.15.89:0/3440565943 learned_addr learned my addr 172.21.15.89:0/3440565943 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:20.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.381+0000 7f7205000700 1 --2- 172.21.15.89:0/3440565943 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71e8008c60 0x7f7200152e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:20.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.381+0000 7f71ffe00700 1 --2- 172.21.15.89:0/3440565943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72000b3840 0x7f720015baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:20.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.381+0000 7f7205a00700 1 -- 172.21.15.89:0/3440565943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72000b3840 msgr2=0x7f720015baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:20.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.381+0000 7f7205a00700 1 --2- 172.21.15.89:0/3440565943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72000b3840 0x7f720015baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.381+0000 7f7205a00700 1 -- 172.21.15.89:0/3440565943 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71e8008c60 msgr2=0x7f7200152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:20.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.381+0000 7f7205a00700 1 --2- 172.21.15.89:0/3440565943 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71e8008c60 0x7f7200152e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.381+0000 7f7205a00700 1 -- 172.21.15.89:0/3440565943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7200154a30 con 0x7f72000b3cc0 2024-04-02T09:13:20.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.381+0000 7f7205a00700 1 --2- 172.21.15.89:0/3440565943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72000b3cc0 0x7f720015c030 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f72000700c0 tx=0x7f71f80114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:20.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.381+0000 7f71fcc00700 1 -- 172.21.15.89:0/3440565943 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f800e070 con 0x7f72000b3cc0 2024-04-02T09:13:20.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.381+0000 7f71fcc00700 1 -- 172.21.15.89:0/3440565943 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f8003c30 con 0x7f72000b3cc0 2024-04-02T09:13:20.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.381+0000 7f7207a00700 1 -- 172.21.15.89:0/3440565943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72000cd630 con 0x7f72000b3cc0 2024-04-02T09:13:20.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.381+0000 7f71fcc00700 1 -- 172.21.15.89:0/3440565943 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8003de0 con 0x7f72000b3cc0 2024-04-02T09:13:20.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.382+0000 7f7207a00700 1 -- 172.21.15.89:0/3440565943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7200153530 con 0x7f72000b3cc0 2024-04-02T09:13:20.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.383+0000 7f7207a00700 1 -- 172.21.15.89:0/3440565943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7200154890 con 0x7f72000b3cc0 2024-04-02T09:13:20.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.383+0000 7f71fcc00700 1 -- 172.21.15.89:0/3440565943 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f71f8011a20 con 0x7f72000b3cc0 2024-04-02T09:13:20.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.385+0000 7f71fcc00700 1 --2- 172.21.15.89:0/3440565943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f71e006aed0 0x7f71e006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:20.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.385+0000 7f71fcc00700 1 -- 172.21.15.89:0/3440565943 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f71f8089e30 con 0x7f72000b3cc0 2024-04-02T09:13:20.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.385+0000 7f7205000700 1 --2- 172.21.15.89:0/3440565943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f71e006aed0 0x7f71e006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:20.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.386+0000 7f7205000700 1 --2- 172.21.15.89:0/3440565943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f71e006aed0 0x7f71e006d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71f4001bf0 tx=0x7f71f4001dd0 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:20.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.389+0000 7f71fcc00700 1 -- 172.21.15.89:0/3440565943 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7200154890 con 0x7f72000b3cc0 2024-04-02T09:13:20.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.673+0000 7f7207a00700 1 -- 172.21.15.89:0/3440565943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f7200153530 con 0x7f72000b3cc0 2024-04-02T09:13:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.828+0000 7f71fcc00700 1 -- 172.21.15.89:0/3440565943 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3180) v1 ==== 186+0+0 (secure 0 0 0) 0x7f7200153530 con 0x7f72000b3cc0 2024-04-02T09:13:20.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.830+0000 7f7207a00700 1 -- 172.21.15.89:0/3440565943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f71e006aed0 msgr2=0x7f71e006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:20.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.830+0000 7f7207a00700 1 --2- 172.21.15.89:0/3440565943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f71e006aed0 0x7f71e006d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71f4001bf0 tx=0x7f71f4001dd0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.830+0000 7f7207a00700 1 -- 172.21.15.89:0/3440565943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72000b3cc0 msgr2=0x7f720015c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:20.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.830+0000 7f7207a00700 1 --2- 172.21.15.89:0/3440565943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72000b3cc0 0x7f720015c030 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f72000700c0 tx=0x7f71f80114b0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.831+0000 7f7207a00700 1 -- 172.21.15.89:0/3440565943 shutdown_connections 2024-04-02T09:13:20.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.831+0000 7f7207a00700 1 --2- 172.21.15.89:0/3440565943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f71e006aed0 0x7f71e006d390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.831+0000 7f7207a00700 1 --2- 172.21.15.89:0/3440565943 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71e8008c60 0x7f7200152e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.831+0000 7f7207a00700 1 --2- 172.21.15.89:0/3440565943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72000b3cc0 0x7f720015c030 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.831+0000 7f7207a00700 1 --2- 172.21.15.89:0/3440565943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72000b3840 0x7f720015baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:20.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.831+0000 7f7207a00700 1 -- 172.21.15.89:0/3440565943 >> 172.21.15.89:0/3440565943 conn(0x7f72000cf190 msgr2=0x7f72000aed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:20.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.831+0000 7f7207a00700 1 -- 172.21.15.89:0/3440565943 shutdown_connections 2024-04-02T09:13:20.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:20.831+0000 7f7207a00700 1 -- 172.21.15.89:0/3440565943 wait complete. 2024-04-02T09:13:20.848 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2024-04-02T09:13:20.848 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2024-04-02T09:13:20.848 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-04-02T09:13:20.848 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-04-02T09:13:21.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.042+0000 7ff38a800700 1 Processor -- start 2024-04-02T09:13:21.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.042+0000 7ff38a800700 1 -- start start 2024-04-02T09:13:21.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.043+0000 7ff38a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff384160f10 0x7ff3840d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:21.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.043+0000 7ff38a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff384161590 0x7ff384161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:21.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.043+0000 7ff38a800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3840ceda0 con 0x7ff3840cf230 2024-04-02T09:13:21.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.043+0000 7ff38a800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff384161f80 con 0x7ff384160f10 2024-04-02T09:13:21.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.043+0000 7ff38a800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3841612f0 con 0x7ff384161590 2024-04-02T09:13:21.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.043+0000 7ff382a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff3840cf230 0x7ff3840d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48410/0 (socket says 172.21.15.89:48410) 2024-04-02T09:13:21.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.043+0000 7ff382a00700 1 -- 172.21.15.89:0/1045660843 learned_addr learned my addr 172.21.15.89:0/1045660843 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:21.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.043+0000 7ff383400700 1 --2- 172.21.15.89:0/1045660843 >> v2:172.21.15.89:3300/0 conn(0x7ff384160f10 0x7ff3840d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:21.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.044+0000 7ff383e00700 1 --2- 172.21.15.89:0/1045660843 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff384161590 0x7ff384161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:21.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.044+0000 7ff382000700 1 -- 172.21.15.89:0/1045660843 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588742272 0 0) 0x7ff3840ceda0 con 0x7ff3840cf230 2024-04-02T09:13:21.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.044+0000 7ff382000700 1 -- 172.21.15.89:0/1045660843 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3680035b0 con 0x7ff3840cf230 2024-04-02T09:13:21.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.044+0000 7ff383400700 1 -- 172.21.15.89:0/1045660843 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff384161590 msgr2=0x7ff384161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:21.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.044+0000 7ff383400700 1 --2- 172.21.15.89:0/1045660843 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff384161590 0x7ff384161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:21.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.044+0000 7ff383400700 1 -- 172.21.15.89:0/1045660843 >> v1:172.21.15.89:6789/0 conn(0x7ff3840cf230 legacy=0x7ff3840d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:21.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.044+0000 7ff383400700 1 -- 172.21.15.89:0/1045660843 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3840ce470 con 0x7ff384160f10 2024-04-02T09:13:21.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.044+0000 7ff383e00700 1 --2- 172.21.15.89:0/1045660843 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff384161590 0x7ff384161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:21.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.044+0000 7ff383400700 1 --2- 172.21.15.89:0/1045660843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff384160f10 0x7ff3840d18b0 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7ff374009840 tx=0x7ff374009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e86998393a5657 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:21.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.044+0000 7ff382000700 1 -- 172.21.15.89:0/1045660843 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374011830 con 0x7ff384160f10 2024-04-02T09:13:21.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.044+0000 7ff382000700 1 -- 172.21.15.89:0/1045660843 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff374011990 con 0x7ff384160f10 2024-04-02T09:13:21.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.044+0000 7ff382000700 1 -- 172.21.15.89:0/1045660843 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374011d20 con 0x7ff384160f10 2024-04-02T09:13:21.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.045+0000 7ff38a800700 1 -- 172.21.15.89:0/1045660843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff384160f10 msgr2=0x7ff3840d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:21.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.045+0000 7ff38a800700 1 --2- 172.21.15.89:0/1045660843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff384160f10 0x7ff3840d18b0 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7ff374009840 tx=0x7ff374009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:21.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.045+0000 7ff38a800700 1 -- 172.21.15.89:0/1045660843 shutdown_connections 2024-04-02T09:13:21.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.045+0000 7ff38a800700 1 --2- 172.21.15.89:0/1045660843 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff384161590 0x7ff384161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:21.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.045+0000 7ff38a800700 1 --2- 172.21.15.89:0/1045660843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff384160f10 0x7ff3840d18b0 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:21.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.045+0000 7ff38a800700 1 -- 172.21.15.89:0/1045660843 >> 172.21.15.89:0/1045660843 conn(0x7ff38408aff0 msgr2=0x7ff3840d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:21.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.045+0000 7ff38a800700 1 -- 172.21.15.89:0/1045660843 shutdown_connections 2024-04-02T09:13:21.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.045+0000 7ff38a800700 1 -- 172.21.15.89:0/1045660843 wait complete. 2024-04-02T09:13:21.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.047+0000 7ff38a800700 1 Processor -- start 2024-04-02T09:13:21.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.047+0000 7ff38a800700 1 -- start start 2024-04-02T09:13:21.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.048+0000 7ff38a800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3840cf230 0x7ff3840b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:21.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.048+0000 7ff38a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff384160f10 0x7ff3841f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:21.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.048+0000 7ff38a800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff384161590 0x7ff38414e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:21.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.048+0000 7ff38a800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3841f73c0 con 0x7ff384160f10 2024-04-02T09:13:21.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.048+0000 7ff38a800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38415d020 con 0x7ff384161590 2024-04-02T09:13:21.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.048+0000 7ff38a800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38415d300 con 0x7ff3840cf230 2024-04-02T09:13:21.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.048+0000 7ff382a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff384160f10 0x7ff3841f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:21.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.048+0000 7ff382a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff384160f10 0x7ff3841f3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54402/0 (socket says 172.21.15.89:54402) 2024-04-02T09:13:21.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.048+0000 7ff382a00700 1 -- 172.21.15.89:0/4064129563 learned_addr learned my addr 172.21.15.89:0/4064129563 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:21.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.048+0000 7ff383400700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3840cf230 0x7ff3840b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:21.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.048+0000 7ff383e00700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff384161590 0x7ff38414e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:21.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.049+0000 7ff382a00700 1 -- 172.21.15.89:0/4064129563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3840cf230 msgr2=0x7ff3840b4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:21.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.049+0000 7ff382a00700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3840cf230 0x7ff3840b4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:21.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.049+0000 7ff382a00700 1 -- 172.21.15.89:0/4064129563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff384161590 msgr2=0x7ff38414e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:21.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.049+0000 7ff382a00700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff384161590 0x7ff38414e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:21.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.049+0000 7ff382a00700 1 -- 172.21.15.89:0/4064129563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3740094a0 con 0x7ff384160f10 2024-04-02T09:13:21.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.049+0000 7ff383400700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3840cf230 0x7ff3840b4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:21.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.049+0000 7ff382a00700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff384160f10 0x7ff3841f3020 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7ff370005910 tx=0x7ff37000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:21.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.049+0000 7ff37b400700 1 -- 172.21.15.89:0/4064129563 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370005d70 con 0x7ff384160f10 2024-04-02T09:13:21.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.049+0000 7ff37b400700 1 -- 172.21.15.89:0/4064129563 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3700070d0 con 0x7ff384160f10 2024-04-02T09:13:21.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.049+0000 7ff37b400700 1 -- 172.21.15.89:0/4064129563 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370007440 con 0x7ff384160f10 2024-04-02T09:13:21.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.049+0000 7ff38a800700 1 -- 172.21.15.89:0/4064129563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38415d5e0 con 0x7ff384160f10 2024-04-02T09:13:21.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.049+0000 7ff38a800700 1 -- 172.21.15.89:0/4064129563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3840b5530 con 0x7ff384160f10 2024-04-02T09:13:21.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.050+0000 7ff38a800700 1 -- 172.21.15.89:0/4064129563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38415db00 con 0x7ff384160f10 2024-04-02T09:13:21.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.051+0000 7ff37b400700 1 -- 172.21.15.89:0/4064129563 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7ff3700075a0 con 0x7ff384160f10 2024-04-02T09:13:21.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.052+0000 7ff37b400700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff36406aed0 0x7ff36406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:21.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.052+0000 7ff37b400700 1 -- 172.21.15.89:0/4064129563 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6208+0+0 (secure 0 0 0) 0x7ff370088d80 con 0x7ff384160f10 2024-04-02T09:13:21.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.052+0000 7ff383400700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff36406aed0 0x7ff36406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:21.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.053+0000 7ff383400700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff36406aed0 0x7ff36406d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff374000c00 tx=0x7ff37401c040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:21.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.056+0000 7ff37b400700 1 -- 172.21.15.89:0/4064129563 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff38415db00 con 0x7ff384160f10 2024-04-02T09:13:21.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.337+0000 7ff38a800700 1 -- 172.21.15.89:0/4064129563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7ff38415db00 con 0x7ff384160f10 2024-04-02T09:13:21.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.833+0000 7ff37b400700 1 -- 172.21.15.89:0/4064129563 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3181) v1 ==== 190+0+0 (secure 0 0 0) 0x7ff38415db00 con 0x7ff384160f10 2024-04-02T09:13:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.836+0000 7ff38a800700 1 -- 172.21.15.89:0/4064129563 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff36406aed0 msgr2=0x7ff36406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.836+0000 7ff38a800700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff36406aed0 0x7ff36406d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff374000c00 tx=0x7ff37401c040 comp rx=0 tx=0).stop 2024-04-02T09:13:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.836+0000 7ff38a800700 1 -- 172.21.15.89:0/4064129563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff384160f10 msgr2=0x7ff3841f3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:21.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.836+0000 7ff38a800700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff384160f10 0x7ff3841f3020 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7ff370005910 tx=0x7ff37000c860 comp rx=0 tx=0).stop 2024-04-02T09:13:21.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.836+0000 7ff38a800700 1 -- 172.21.15.89:0/4064129563 shutdown_connections 2024-04-02T09:13:21.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.836+0000 7ff38a800700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff36406aed0 0x7ff36406d390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:21.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.836+0000 7ff38a800700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff384161590 0x7ff38414e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:21.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.836+0000 7ff38a800700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3840cf230 0x7ff3840b4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:21.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.836+0000 7ff38a800700 1 --2- 172.21.15.89:0/4064129563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff384160f10 0x7ff3841f3020 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:21.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.836+0000 7ff38a800700 1 -- 172.21.15.89:0/4064129563 >> 172.21.15.89:0/4064129563 conn(0x7ff38408aff0 msgr2=0x7ff3840cbb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:21.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.836+0000 7ff38a800700 1 -- 172.21.15.89:0/4064129563 shutdown_connections 2024-04-02T09:13:21.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:21.836+0000 7ff38a800700 1 -- 172.21.15.89:0/4064129563 wait complete. 2024-04-02T09:13:21.855 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T09:13:22.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.052+0000 7f8872000700 1 Processor -- start 2024-04-02T09:13:22.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.052+0000 7f8872000700 1 -- start start 2024-04-02T09:13:22.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.053+0000 7f8872000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f886c0cd020 0x7f886c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:22.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.053+0000 7f8872000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886c0cdae0 con 0x7f886c0cbb30 2024-04-02T09:13:22.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.053+0000 7f8872000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886c0cddc0 con 0x7f886c0b2070 2024-04-02T09:13:22.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.053+0000 7f8872000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f886c0ce0a0 con 0x7f886c0cd020 2024-04-02T09:13:22.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.053+0000 7f886b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f886c0cbb30 0x7f886c0cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48424/0 (socket says 172.21.15.89:48424) 2024-04-02T09:13:22.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.053+0000 7f886b400700 1 -- 172.21.15.89:0/771191686 learned_addr learned my addr 172.21.15.89:0/771191686 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:22.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.053+0000 7f886aa00700 1 --2- 172.21.15.89:0/771191686 >> v2:172.21.15.89:3300/0 conn(0x7f886c0cd020 0x7f886c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:22.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.054+0000 7f886aa00700 1 -- 172.21.15.89:0/771191686 >> v1:172.21.15.113:6789/0 conn(0x7f886c0b2070 legacy=0x7f886c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:22.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.054+0000 7f886aa00700 1 -- 172.21.15.89:0/771191686 >> v1:172.21.15.89:6789/0 conn(0x7f886c0cbb30 legacy=0x7f886c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:22.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.053+0000 7f886a000700 1 -- 172.21.15.89:0/771191686 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564415195 0 0) 0x7f886c0cdae0 con 0x7f886c0cbb30 2024-04-02T09:13:22.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.054+0000 7f886aa00700 1 -- 172.21.15.89:0/771191686 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c06dc20 con 0x7f886c0cd020 2024-04-02T09:13:22.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.054+0000 7f886aa00700 1 --2- 172.21.15.89:0/771191686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f886c0cd020 0x7f886c0cd480 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f8858009840 tx=0x7f8858009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9940418e2ebb25 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:22.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.054+0000 7f886a000700 1 -- 172.21.15.89:0/771191686 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8858011830 con 0x7f886c0cd020 2024-04-02T09:13:22.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.054+0000 7f886a000700 1 -- 172.21.15.89:0/771191686 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8858011990 con 0x7f886c0cd020 2024-04-02T09:13:22.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.054+0000 7f886a000700 1 -- 172.21.15.89:0/771191686 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8858011d20 con 0x7f886c0cd020 2024-04-02T09:13:22.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.055+0000 7f8872000700 1 -- 172.21.15.89:0/771191686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f886c0cd020 msgr2=0x7f886c0cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:22.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.055+0000 7f8872000700 1 --2- 172.21.15.89:0/771191686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f886c0cd020 0x7f886c0cd480 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f8858009840 tx=0x7f8858009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:22.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.055+0000 7f8872000700 1 -- 172.21.15.89:0/771191686 shutdown_connections 2024-04-02T09:13:22.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.055+0000 7f8872000700 1 --2- 172.21.15.89:0/771191686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f886c0cd020 0x7f886c0cd480 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:22.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.055+0000 7f8872000700 1 -- 172.21.15.89:0/771191686 >> 172.21.15.89:0/771191686 conn(0x7f886c0b0d70 msgr2=0x7f886c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:22.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.055+0000 7f8872000700 1 -- 172.21.15.89:0/771191686 shutdown_connections 2024-04-02T09:13:22.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.055+0000 7f8872000700 1 -- 172.21.15.89:0/771191686 wait complete. 2024-04-02T09:13:22.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.056+0000 7f8872000700 1 Processor -- start 2024-04-02T09:13:22.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.057+0000 7f8872000700 1 -- start start 2024-04-02T09:13:22.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.057+0000 7f8872000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f886c0b2070 0x7f886c071b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:22.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.057+0000 7f8872000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f886c0cbb30 0x7f886c14a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:22.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.057+0000 7f8872000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f886c0cd020 0x7f886c0c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.057+0000 7f8872000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c072040 con 0x7f886c0cbb30 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.057+0000 7f8872000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c157d90 con 0x7f886c0b2070 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.057+0000 7f8872000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886c070a60 con 0x7f886c0cd020 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.058+0000 7f886aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f886c0cbb30 0x7f886c14a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.058+0000 7f886aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f886c0cbb30 0x7f886c14a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54420/0 (socket says 172.21.15.89:54420) 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.058+0000 7f886aa00700 1 -- 172.21.15.89:0/4013324039 learned_addr learned my addr 172.21.15.89:0/4013324039 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.058+0000 7f886be00700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f886c0cd020 0x7f886c0c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.058+0000 7f886b400700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f886c0b2070 0x7f886c071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.058+0000 7f886aa00700 1 -- 172.21.15.89:0/4013324039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f886c0cd020 msgr2=0x7f886c0c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.058+0000 7f886aa00700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f886c0cd020 0x7f886c0c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.058+0000 7f886aa00700 1 -- 172.21.15.89:0/4013324039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f886c0b2070 msgr2=0x7f886c071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.058+0000 7f886aa00700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f886c0b2070 0x7f886c071b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.058+0000 7f886aa00700 1 -- 172.21.15.89:0/4013324039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88580094a0 con 0x7f886c0cbb30 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.058+0000 7f886be00700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f886c0cd020 0x7f886c0c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.058+0000 7f886aa00700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f886c0cbb30 0x7f886c14a4e0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f8858014040 tx=0x7f8858004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:22.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.059+0000 7f8863400700 1 -- 172.21.15.89:0/4013324039 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885801a900 con 0x7f886c0cbb30 2024-04-02T09:13:22.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.059+0000 7f8863400700 1 -- 172.21.15.89:0/4013324039 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885801aa60 con 0x7f886c0cbb30 2024-04-02T09:13:22.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.059+0000 7f8863400700 1 -- 172.21.15.89:0/4013324039 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885801add0 con 0x7f886c0cbb30 2024-04-02T09:13:22.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.059+0000 7f8872000700 1 -- 172.21.15.89:0/4013324039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f886c070d40 con 0x7f886c0cbb30 2024-04-02T09:13:22.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.059+0000 7f8872000700 1 -- 172.21.15.89:0/4013324039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886c071300 con 0x7f886c0cbb30 2024-04-02T09:13:22.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.060+0000 7f8872000700 1 -- 172.21.15.89:0/4013324039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f886c067720 con 0x7f886c0cbb30 2024-04-02T09:13:22.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.061+0000 7f8863400700 1 -- 172.21.15.89:0/4013324039 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f8858022070 con 0x7f886c0cbb30 2024-04-02T09:13:22.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.061+0000 7f8863400700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f884c06aed0 0x7f884c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:22.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.061+0000 7f8863400700 1 -- 172.21.15.89:0/4013324039 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f885808a730 con 0x7f886c0cbb30 2024-04-02T09:13:22.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.062+0000 7f886b400700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f884c06aed0 0x7f884c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:22.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.062+0000 7f886b400700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f884c06aed0 0x7f884c06d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f885c007fd0 tx=0x7f885c007c20 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:22.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.066+0000 7f8863400700 1 -- 172.21.15.89:0/4013324039 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8858055500 con 0x7f886c0cbb30 2024-04-02T09:13:22.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.266+0000 7f8872000700 1 -- 172.21.15.89:0/4013324039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f886c0ce2e0 con 0x7f886c0cbb30 2024-04-02T09:13:22.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.272+0000 7f8863400700 1 -- 172.21.15.89:0/4013324039 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v82)=0 v82) v1 ==== 122+0+0 (secure 0 0 0) 0x7f885805de60 con 0x7f886c0cbb30 2024-04-02T09:13:22.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.275+0000 7f8872000700 1 -- 172.21.15.89:0/4013324039 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f884c06aed0 msgr2=0x7f884c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:22.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.275+0000 7f8872000700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f884c06aed0 0x7f884c06d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f885c007fd0 tx=0x7f885c007c20 comp rx=0 tx=0).stop 2024-04-02T09:13:22.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.275+0000 7f8872000700 1 -- 172.21.15.89:0/4013324039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f886c0cbb30 msgr2=0x7f886c14a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:22.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.275+0000 7f8872000700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f886c0cbb30 0x7f886c14a4e0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f8858014040 tx=0x7f8858004750 comp rx=0 tx=0).stop 2024-04-02T09:13:22.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.275+0000 7f8872000700 1 -- 172.21.15.89:0/4013324039 shutdown_connections 2024-04-02T09:13:22.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.275+0000 7f8872000700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f884c06aed0 0x7f884c06d390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:22.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.275+0000 7f8872000700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f886c0b2070 0x7f886c071b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:22.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.275+0000 7f8872000700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f886c0cd020 0x7f886c0c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:22.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.275+0000 7f8872000700 1 --2- 172.21.15.89:0/4013324039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f886c0cbb30 0x7f886c14a4e0 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:22.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.275+0000 7f8872000700 1 -- 172.21.15.89:0/4013324039 >> 172.21.15.89:0/4013324039 conn(0x7f886c0b0d70 msgr2=0x7f886c0af9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:22.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.275+0000 7f8872000700 1 -- 172.21.15.89:0/4013324039 shutdown_connections 2024-04-02T09:13:22.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.275+0000 7f8872000700 1 -- 172.21.15.89:0/4013324039 wait complete. 2024-04-02T09:13:22.292 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T09:13:22.292 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:13:22.372 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:13:22.465 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T09:13:22.465 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:13:22.465 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:13:22.509 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:13:22.510 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:13:22.510 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:13:22.556 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:13:22.556 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:13:22.556 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:13:22.600 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:13:22.600 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:13:22.600 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:13:22.645 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:13:22.645 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T09:13:22.645 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:13:22.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.838+0000 7f9463600700 1 Processor -- start 2024-04-02T09:13:22.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.838+0000 7f9463600700 1 -- start start 2024-04-02T09:13:22.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.839+0000 7f9463600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f945c0d18b0 0x7f945c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:22.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.840+0000 7f9463600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f945c161590 0x7f945c161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:22.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.840+0000 7f9463600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945c0ceda0 con 0x7f945c160f10 2024-04-02T09:13:22.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.840+0000 7f9463600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c0d1d30 con 0x7f945c161590 2024-04-02T09:13:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.840+0000 7f9463600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c161f60 con 0x7f945c0d18b0 2024-04-02T09:13:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.840+0000 7f9461600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f945c161590 0x7f945c161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.840+0000 7f9460c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f945c160f10 0x7f945c0d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48436/0 (socket says 172.21.15.89:48436) 2024-04-02T09:13:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.840+0000 7f9460c00700 1 -- 172.21.15.89:0/1433085295 learned_addr learned my addr 172.21.15.89:0/1433085295 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.840+0000 7f945be00700 1 --2- 172.21.15.89:0/1433085295 >> v2:172.21.15.113:3300/0 conn(0x7f945c0d18b0 0x7f945c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.840+0000 7f9461600700 1 -- 172.21.15.89:0/1433085295 >> v2:172.21.15.113:3300/0 conn(0x7f945c0d18b0 msgr2=0x7f945c0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.840+0000 7f9461600700 1 --2- 172.21.15.89:0/1433085295 >> v2:172.21.15.113:3300/0 conn(0x7f945c0d18b0 0x7f945c0cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.840+0000 7f9461600700 1 -- 172.21.15.89:0/1433085295 >> v1:172.21.15.89:6789/0 conn(0x7f945c160f10 legacy=0x7f945c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.840+0000 7f9461600700 1 -- 172.21.15.89:0/1433085295 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c1612f0 con 0x7f945c161590 2024-04-02T09:13:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.841+0000 7f9461600700 1 --2- 172.21.15.89:0/1433085295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f945c161590 0x7f945c161a20 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f945c08ca30 tx=0x7f945400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af574412571df36 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.841+0000 7f945b400700 1 -- 172.21.15.89:0/1433085295 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945400e070 con 0x7f945c161590 2024-04-02T09:13:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.841+0000 7f945b400700 1 -- 172.21.15.89:0/1433085295 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945400bb20 con 0x7f945c161590 2024-04-02T09:13:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.841+0000 7f945b400700 1 -- 172.21.15.89:0/1433085295 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9454012430 con 0x7f945c161590 2024-04-02T09:13:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.841+0000 7f9463600700 1 -- 172.21.15.89:0/1433085295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f945c161590 msgr2=0x7f945c161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.841+0000 7f9463600700 1 --2- 172.21.15.89:0/1433085295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f945c161590 0x7f945c161a20 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f945c08ca30 tx=0x7f945400b330 comp rx=0 tx=0).stop 2024-04-02T09:13:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.841+0000 7f9463600700 1 -- 172.21.15.89:0/1433085295 shutdown_connections 2024-04-02T09:13:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.841+0000 7f9463600700 1 --2- 172.21.15.89:0/1433085295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f945c161590 0x7f945c161a20 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.841+0000 7f9463600700 1 --2- 172.21.15.89:0/1433085295 >> v2:172.21.15.113:3300/0 conn(0x7f945c0d18b0 0x7f945c0cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.841+0000 7f9463600700 1 -- 172.21.15.89:0/1433085295 >> 172.21.15.89:0/1433085295 conn(0x7f945c08aff0 msgr2=0x7f945c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.842+0000 7f9463600700 1 -- 172.21.15.89:0/1433085295 shutdown_connections 2024-04-02T09:13:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.842+0000 7f9463600700 1 -- 172.21.15.89:0/1433085295 wait complete. 2024-04-02T09:13:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.843+0000 7f9463600700 1 Processor -- start 2024-04-02T09:13:22.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.843+0000 7f9463600700 1 -- start start 2024-04-02T09:13:22.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.844+0000 7f9463600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f945c0d18b0 0x7f945c155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:22.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.844+0000 7f9463600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f945c160f10 0x7f945c15cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:22.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.844+0000 7f9463600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f945c161590 0x7f945c0b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:22.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.844+0000 7f9463600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c1571c0 con 0x7f945c0d18b0 2024-04-02T09:13:22.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.844+0000 7f9463600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c1565b0 con 0x7f945c161590 2024-04-02T09:13:22.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.844+0000 7f9463600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c156890 con 0x7f945c160f10 2024-04-02T09:13:22.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9460c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f945c0d18b0 0x7f945c155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:22.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9460c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f945c0d18b0 0x7f945c155b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54430/0 (socket says 172.21.15.89:54430) 2024-04-02T09:13:22.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9460c00700 1 -- 172.21.15.89:0/2277162299 learned_addr learned my addr 172.21.15.89:0/2277162299 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9460c00700 1 -- 172.21.15.89:0/2277162299 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f945c160f10 msgr2=0x7f945c15cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f945be00700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f945c160f10 0x7f945c15cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9461600700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f945c161590 0x7f945c0b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9460c00700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f945c160f10 0x7f945c15cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9460c00700 1 -- 172.21.15.89:0/2277162299 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f945c161590 msgr2=0x7f945c0b82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9460c00700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f945c161590 0x7f945c0b82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9460c00700 1 -- 172.21.15.89:0/2277162299 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9454009a70 con 0x7f945c0d18b0 2024-04-02T09:13:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9461600700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f945c161590 0x7f945c0b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f945be00700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f945c160f10 0x7f945c15cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9460c00700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f945c0d18b0 0x7f945c155b40 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f945000c480 tx=0x7f945000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9458c00700 1 -- 172.21.15.89:0/2277162299 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450005d70 con 0x7f945c0d18b0 2024-04-02T09:13:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9458c00700 1 -- 172.21.15.89:0/2277162299 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94500070d0 con 0x7f945c0d18b0 2024-04-02T09:13:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9463600700 1 -- 172.21.15.89:0/2277162299 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c156b70 con 0x7f945c0d18b0 2024-04-02T09:13:22.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.845+0000 7f9458c00700 1 -- 172.21.15.89:0/2277162299 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450007440 con 0x7f945c0d18b0 2024-04-02T09:13:22.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.846+0000 7f9463600700 1 -- 172.21.15.89:0/2277162299 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c0cb490 con 0x7f945c0d18b0 2024-04-02T09:13:22.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.847+0000 7f9463600700 1 -- 172.21.15.89:0/2277162299 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945c067720 con 0x7f945c0d18b0 2024-04-02T09:13:22.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.847+0000 7f9458c00700 1 -- 172.21.15.89:0/2277162299 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f94500075a0 con 0x7f945c0d18b0 2024-04-02T09:13:22.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.848+0000 7f9458c00700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f943c06aed0 0x7f943c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:22.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.848+0000 7f9458c00700 1 -- 172.21.15.89:0/2277162299 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f9450088c10 con 0x7f945c0d18b0 2024-04-02T09:13:22.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.849+0000 7f945be00700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f943c06aed0 0x7f943c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:22.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.849+0000 7f945be00700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f943c06aed0 0x7f943c06d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f945c15bad0 tx=0x7f9448008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:22.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:22.853+0000 7f9458c00700 1 -- 172.21.15.89:0/2277162299 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9450053ab0 con 0x7f945c0d18b0 2024-04-02T09:13:23.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.100+0000 7f9463600700 1 -- 172.21.15.89:0/2277162299 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f945c0d1610 con 0x7f945c0d18b0 2024-04-02T09:13:23.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.100+0000 7f9458c00700 1 -- 172.21.15.89:0/2277162299 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+3539 (secure 0 0 0) 0x7f9450017090 con 0x7f945c0d18b0 2024-04-02T09:13:23.100 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:23.100 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":114,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8969,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3650347444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3650347444},{"type":"v1","addr":"172.21.15.89:6836","nonce":3650347444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":9464,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/516286475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":516286475},{"type":"v1","addr":"172.21.15.113:6835","nonce":516286475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114},{"gid":10276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2152500685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":2152500685},{"type":"v1","addr":"172.21.15.113:6834","nonce":2152500685}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":11784,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2785458438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":2785458438},{"type":"v1","addr":"172.21.15.89:6837","nonce":2785458438}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":112}],"filesystems":[]} 2024-04-02T09:13:23.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.102+0000 7f9463600700 1 -- 172.21.15.89:0/2277162299 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f943c06aed0 msgr2=0x7f943c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:23.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.103+0000 7f9463600700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f943c06aed0 0x7f943c06d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f945c15bad0 tx=0x7f9448008040 comp rx=0 tx=0).stop 2024-04-02T09:13:23.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.103+0000 7f9463600700 1 -- 172.21.15.89:0/2277162299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f945c0d18b0 msgr2=0x7f945c155b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:23.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.103+0000 7f9463600700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f945c0d18b0 0x7f945c155b40 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f945000c480 tx=0x7f945000c840 comp rx=0 tx=0).stop 2024-04-02T09:13:23.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.103+0000 7f9463600700 1 -- 172.21.15.89:0/2277162299 shutdown_connections 2024-04-02T09:13:23.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.103+0000 7f9463600700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f943c06aed0 0x7f943c06d390 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:23.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.103+0000 7f9463600700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f945c161590 0x7f945c0b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:23.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.103+0000 7f9463600700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f945c0d18b0 0x7f945c155b40 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:23.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.103+0000 7f9463600700 1 --2- 172.21.15.89:0/2277162299 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f945c160f10 0x7f945c15cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:23.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.103+0000 7f9463600700 1 -- 172.21.15.89:0/2277162299 >> 172.21.15.89:0/2277162299 conn(0x7f945c08aff0 msgr2=0x7f945c0ccd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:23.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.103+0000 7f9463600700 1 -- 172.21.15.89:0/2277162299 shutdown_connections 2024-04-02T09:13:23.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.103+0000 7f9463600700 1 -- 172.21.15.89:0/2277162299 wait complete. 2024-04-02T09:13:23.103 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 114 2024-04-02T09:13:23.121 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T09:13:23.121 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T09:13:23.121 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T09:13:23.121 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:13:23.121 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T09:13:23.121 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T09:13:23.121 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T09:13:23.122 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:13:23.122 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T09:13:23.122 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T09:13:23.122 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T09:13:23.122 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:13:23.122 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T09:13:23.122 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T09:13:23.123 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T09:13:23.123 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:13:23.137 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T09:13:23.137 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T09:13:23.138 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T09:13:23.139 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T09:13:23.141 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T09:13:23.141 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T09:13:23.142 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T09:13:23.142 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T09:13:23.183 INFO:tasks.ceph.mds.a:Started 2024-04-02T09:13:23.183 DEBUG:teuthology.parallel:result is None 2024-04-02T09:13:23.186 INFO:tasks.ceph.mds.c:Started 2024-04-02T09:13:23.186 DEBUG:teuthology.parallel:result is None 2024-04-02T09:13:23.187 INFO:tasks.ceph.mds.b:Started 2024-04-02T09:13:23.187 DEBUG:teuthology.parallel:result is None 2024-04-02T09:13:23.189 INFO:tasks.ceph.mds.d:Started 2024-04-02T09:13:23.189 DEBUG:teuthology.parallel:result is None 2024-04-02T09:13:23.190 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T09:13:23.330 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T09:13:23.330 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T09:13:23.334 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T09:13:23.334 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T09:13:23.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.396+0000 7f6850000700 1 Processor -- start 2024-04-02T09:13:23.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.396+0000 7f6850000700 1 -- start start 2024-04-02T09:13:23.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.397+0000 7f6850000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f684815d9d0 0x7f68480d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:23.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.397+0000 7f6850000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f68480d16a0 0x7f68480d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:23.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.397+0000 7f6850000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6848161420 con 0x7f68480d2370 2024-04-02T09:13:23.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.397+0000 7f6850000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68480d27e0 con 0x7f68480d16a0 2024-04-02T09:13:23.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.397+0000 7f6850000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6848161700 con 0x7f684815d9d0 2024-04-02T09:13:23.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.397+0000 7f684cc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f68480d16a0 0x7f68480d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:23.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.397+0000 7f684cc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f68480d16a0 0x7f68480d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54448/0 (socket says 172.21.15.89:54448) 2024-04-02T09:13:23.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.397+0000 7f684cc00700 1 -- 172.21.15.89:0/2811358306 learned_addr learned my addr 172.21.15.89:0/2811358306 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:23.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.397+0000 7f684d600700 1 --2- 172.21.15.89:0/2811358306 >> v2:172.21.15.113:3300/0 conn(0x7f684815d9d0 0x7f68480d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:23.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.398+0000 7f684cc00700 1 -- 172.21.15.89:0/2811358306 >> v2:172.21.15.113:3300/0 conn(0x7f684815d9d0 msgr2=0x7f68480d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:23.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.398+0000 7f684cc00700 1 --2- 172.21.15.89:0/2811358306 >> v2:172.21.15.113:3300/0 conn(0x7f684815d9d0 0x7f68480d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:23.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.398+0000 7f684cc00700 1 -- 172.21.15.89:0/2811358306 >> v1:172.21.15.113:6789/0 conn(0x7f68480d2370 legacy=0x7f684815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:23.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.398+0000 7f684cc00700 1 -- 172.21.15.89:0/2811358306 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68481619e0 con 0x7f68480d16a0 2024-04-02T09:13:23.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.398+0000 7f684cc00700 1 --2- 172.21.15.89:0/2811358306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68480d16a0 0x7f68480d1e30 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f68480d0ff0 tx=0x7f6834009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0b6f3a3539d74d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:23.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.398+0000 7f6847e00700 1 -- 172.21.15.89:0/2811358306 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834005e30 con 0x7f68480d16a0 2024-04-02T09:13:23.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.398+0000 7f6847e00700 1 -- 172.21.15.89:0/2811358306 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6834004030 con 0x7f68480d16a0 2024-04-02T09:13:23.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.398+0000 7f6847e00700 1 -- 172.21.15.89:0/2811358306 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68340043e0 con 0x7f68480d16a0 2024-04-02T09:13:23.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.399+0000 7f6850000700 1 -- 172.21.15.89:0/2811358306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68480d16a0 msgr2=0x7f68480d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:23.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.399+0000 7f6850000700 1 --2- 172.21.15.89:0/2811358306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68480d16a0 0x7f68480d1e30 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f68480d0ff0 tx=0x7f6834009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:23.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.399+0000 7f6850000700 1 -- 172.21.15.89:0/2811358306 shutdown_connections 2024-04-02T09:13:23.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.399+0000 7f6850000700 1 --2- 172.21.15.89:0/2811358306 >> v2:172.21.15.113:3300/0 conn(0x7f684815d9d0 0x7f68480d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:23.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.399+0000 7f6850000700 1 --2- 172.21.15.89:0/2811358306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68480d16a0 0x7f68480d1e30 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:23.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.399+0000 7f6850000700 1 -- 172.21.15.89:0/2811358306 >> 172.21.15.89:0/2811358306 conn(0x7f684808aff0 msgr2=0x7f68480d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:23.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.399+0000 7f6850000700 1 -- 172.21.15.89:0/2811358306 shutdown_connections 2024-04-02T09:13:23.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.399+0000 7f6850000700 1 -- 172.21.15.89:0/2811358306 wait complete. 2024-04-02T09:13:23.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.401+0000 7f6850000700 1 Processor -- start 2024-04-02T09:13:23.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.401+0000 7f6850000700 1 -- start start 2024-04-02T09:13:23.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f6850000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68480d16a0 0x7f684815f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:23.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f6850000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68480d2370 0x7f68481f09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:23.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f6850000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684815d9d0 0x7f68480b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:23.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f6850000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684815ed20 con 0x7f684815d9d0 2024-04-02T09:13:23.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f6850000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68480cb590 con 0x7f68480d2370 2024-04-02T09:13:23.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f6850000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68481f0f10 con 0x7f68480d16a0 2024-04-02T09:13:23.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f684e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684815d9d0 0x7f68480b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:23.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f684e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684815d9d0 0x7f68480b4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54450/0 (socket says 172.21.15.89:54450) 2024-04-02T09:13:23.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f684e000700 1 -- 172.21.15.89:0/2313228340 learned_addr learned my addr 172.21.15.89:0/2313228340 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:23.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f684d600700 1 --2- 172.21.15.89:0/2313228340 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68480d16a0 0x7f684815f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:23.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f684cc00700 1 --2- 172.21.15.89:0/2313228340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68480d2370 0x7f68481f09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:23.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f684e000700 1 -- 172.21.15.89:0/2313228340 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68480d16a0 msgr2=0x7f684815f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:23.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f684e000700 1 --2- 172.21.15.89:0/2313228340 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68480d16a0 0x7f684815f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:23.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f684e000700 1 -- 172.21.15.89:0/2313228340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68480d2370 msgr2=0x7f68481f09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:23.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f684e000700 1 --2- 172.21.15.89:0/2313228340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68480d2370 0x7f68481f09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:23.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.402+0000 7f684e000700 1 -- 172.21.15.89:0/2313228340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6834009500 con 0x7f684815d9d0 2024-04-02T09:13:23.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.403+0000 7f684e000700 1 --2- 172.21.15.89:0/2313228340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684815d9d0 0x7f68480b4270 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f684000cc50 tx=0x7f684000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:23.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.403+0000 7f6845600700 1 -- 172.21.15.89:0/2313228340 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6840003c00 con 0x7f684815d9d0 2024-04-02T09:13:23.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.403+0000 7f6845600700 1 -- 172.21.15.89:0/2313228340 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6840003d60 con 0x7f684815d9d0 2024-04-02T09:13:23.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.403+0000 7f6850000700 1 -- 172.21.15.89:0/2313228340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68480cb2b0 con 0x7f684815d9d0 2024-04-02T09:13:23.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.403+0000 7f6845600700 1 -- 172.21.15.89:0/2313228340 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684000aab0 con 0x7f684815d9d0 2024-04-02T09:13:23.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.403+0000 7f6850000700 1 -- 172.21.15.89:0/2313228340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684815e9b0 con 0x7f684815d9d0 2024-04-02T09:13:23.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.404+0000 7f6850000700 1 -- 172.21.15.89:0/2313228340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68480d1ca0 con 0x7f684815d9d0 2024-04-02T09:13:23.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.405+0000 7f6845600700 1 -- 172.21.15.89:0/2313228340 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f684000ac10 con 0x7f684815d9d0 2024-04-02T09:13:23.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.406+0000 7f6845600700 1 --2- 172.21.15.89:0/2313228340 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f682806aed0 0x7f682806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:23.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.406+0000 7f6845600700 1 -- 172.21.15.89:0/2313228340 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f68400898b0 con 0x7f684815d9d0 2024-04-02T09:13:23.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.406+0000 7f684d600700 1 --2- 172.21.15.89:0/2313228340 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f682806aed0 0x7f682806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:23.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.406+0000 7f684d600700 1 --2- 172.21.15.89:0/2313228340 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f682806aed0 0x7f682806d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f68480b5960 tx=0x7f683c009210 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:23.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.410+0000 7f6845600700 1 -- 172.21.15.89:0/2313228340 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6840054550 con 0x7f684815d9d0 2024-04-02T09:13:23.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:23.629+0000 7f6850000700 1 -- 172.21.15.89:0/2313228340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f684806afa0 con 0x7f684815d9d0 2024-04-02T09:13:24.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.367+0000 7f6845600700 1 -- 172.21.15.89:0/2313228340 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v240) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6840057b70 con 0x7f684815d9d0 2024-04-02T09:13:24.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.370+0000 7f6850000700 1 -- 172.21.15.89:0/2313228340 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f682806aed0 msgr2=0x7f682806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:24.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.370+0000 7f6850000700 1 --2- 172.21.15.89:0/2313228340 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f682806aed0 0x7f682806d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f68480b5960 tx=0x7f683c009210 comp rx=0 tx=0).stop 2024-04-02T09:13:24.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.370+0000 7f6850000700 1 -- 172.21.15.89:0/2313228340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684815d9d0 msgr2=0x7f68480b4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:24.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.370+0000 7f6850000700 1 --2- 172.21.15.89:0/2313228340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684815d9d0 0x7f68480b4270 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f684000cc50 tx=0x7f684000cf60 comp rx=0 tx=0).stop 2024-04-02T09:13:24.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.370+0000 7f6850000700 1 -- 172.21.15.89:0/2313228340 shutdown_connections 2024-04-02T09:13:24.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.370+0000 7f6850000700 1 --2- 172.21.15.89:0/2313228340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684815d9d0 0x7f68480b4270 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:24.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.370+0000 7f6850000700 1 --2- 172.21.15.89:0/2313228340 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68480d16a0 0x7f684815f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:24.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.370+0000 7f6850000700 1 --2- 172.21.15.89:0/2313228340 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f682806aed0 0x7f682806d390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:24.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.370+0000 7f6850000700 1 --2- 172.21.15.89:0/2313228340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68480d2370 0x7f68481f09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:24.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.370+0000 7f6850000700 1 -- 172.21.15.89:0/2313228340 >> 172.21.15.89:0/2313228340 conn(0x7f684808aff0 msgr2=0x7f684815bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:24.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.370+0000 7f6850000700 1 -- 172.21.15.89:0/2313228340 shutdown_connections 2024-04-02T09:13:24.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.370+0000 7f6850000700 1 -- 172.21.15.89:0/2313228340 wait complete. 2024-04-02T09:13:24.371 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T09:13:24.390 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T09:13:24.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.585+0000 7f9f85000700 1 Processor -- start 2024-04-02T09:13:24.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.586+0000 7f9f85000700 1 -- start start 2024-04-02T09:13:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.587+0000 7f9f85000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f800ac7d0 0x7f9f80166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.587+0000 7f9f85000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8015fd50 con 0x7f9f800cb3f0 2024-04-02T09:13:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.587+0000 7f9f85000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8015f140 con 0x7f9f8015b5d0 2024-04-02T09:13:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.587+0000 7f9f85000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8015b9b0 con 0x7f9f800ac7d0 2024-04-02T09:13:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.587+0000 7f9f7d400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9f800cb3f0 0x7f9f800ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48460/0 (socket says 172.21.15.89:48460) 2024-04-02T09:13:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.587+0000 7f9f7d400700 1 -- 172.21.15.89:0/2550930520 learned_addr learned my addr 172.21.15.89:0/2550930520 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.587+0000 7f9f7ca00700 1 -- 172.21.15.89:0/2550930520 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656808441 0 0) 0x7f9f8015fd50 con 0x7f9f800cb3f0 2024-04-02T09:13:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.587+0000 7f9f7e800700 1 --2- 172.21.15.89:0/2550930520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f800ac7d0 0x7f9f80166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.587+0000 7f9f7ca00700 1 -- 172.21.15.89:0/2550930520 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f640035b0 con 0x7f9f800cb3f0 2024-04-02T09:13:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7ca00700 1 -- 172.21.15.89:0/2550930520 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3826401121 0 0) 0x7f9f640035b0 con 0x7f9f800cb3f0 2024-04-02T09:13:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7ca00700 1 -- 172.21.15.89:0/2550930520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f800ac7d0 msgr2=0x7f9f80166430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7ca00700 1 --2- 172.21.15.89:0/2550930520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f800ac7d0 0x7f9f80166430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7ca00700 1 -- 172.21.15.89:0/2550930520 >> v1:172.21.15.113:6789/0 conn(0x7f9f8015b5d0 legacy=0x7f9f800ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7ca00700 1 -- 172.21.15.89:0/2550930520 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8015f420 con 0x7f9f800cb3f0 2024-04-02T09:13:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7ca00700 1 -- 172.21.15.89:0/2550930520 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f9f6c001670 con 0x7f9f800cb3f0 2024-04-02T09:13:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7ca00700 1 -- 172.21.15.89:0/2550930520 >> v1:172.21.15.89:6789/0 conn(0x7f9f800cb3f0 legacy=0x7f9f800ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7ca00700 1 --2- 172.21.15.89:0/2550930520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f800ac7d0 0x7f9f64003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7ca00700 1 --2- 172.21.15.89:0/2550930520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f64004060 0x7f9f64004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7ca00700 1 --2- 172.21.15.89:0/2550930520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f64004d90 0x7f9f64005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7ca00700 1 -- 172.21.15.89:0/2550930520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8015b9b0 con 0x7f9f64004d90 2024-04-02T09:13:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7ca00700 1 -- 172.21.15.89:0/2550930520 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6400bc50 con 0x7f9f64004060 2024-04-02T09:13:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7ca00700 1 -- 172.21.15.89:0/2550930520 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6400be10 con 0x7f9f800ac7d0 2024-04-02T09:13:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7e800700 1 --2- 172.21.15.89:0/2550930520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f64004d90 0x7f9f64005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7d400700 1 --2- 172.21.15.89:0/2550930520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f64004060 0x7f9f64004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.588+0000 7f9f7de00700 1 --2- 172.21.15.89:0/2550930520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f800ac7d0 0x7f9f64003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.589+0000 7f9f7e800700 1 -- 172.21.15.89:0/2550930520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f800ac7d0 msgr2=0x7f9f64003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.589+0000 7f9f7e800700 1 --2- 172.21.15.89:0/2550930520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f800ac7d0 0x7f9f64003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.589+0000 7f9f7e800700 1 -- 172.21.15.89:0/2550930520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f64004060 msgr2=0x7f9f64004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.589+0000 7f9f7e800700 1 --2- 172.21.15.89:0/2550930520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f64004060 0x7f9f64004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.589+0000 7f9f7e800700 1 -- 172.21.15.89:0/2550930520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f640035b0 con 0x7f9f64004d90 2024-04-02T09:13:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.589+0000 7f9f7e800700 1 --2- 172.21.15.89:0/2550930520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f64004d90 0x7f9f64005210 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f9f78007330 tx=0x7f9f64005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea7ab65492c9b494 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.589+0000 7f9f7ca00700 1 -- 172.21.15.89:0/2550930520 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f78007e50 con 0x7f9f64004d90 2024-04-02T09:13:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.589+0000 7f9f7ca00700 1 -- 172.21.15.89:0/2550930520 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f78014400 con 0x7f9f64004d90 2024-04-02T09:13:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.589+0000 7f9f7ca00700 1 -- 172.21.15.89:0/2550930520 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f780147b0 con 0x7f9f64004d90 2024-04-02T09:13:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.589+0000 7f9f85000700 1 -- 172.21.15.89:0/2550930520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f64004d90 msgr2=0x7f9f64005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.589+0000 7f9f85000700 1 --2- 172.21.15.89:0/2550930520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f64004d90 0x7f9f64005210 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f9f78007330 tx=0x7f9f64005880 comp rx=0 tx=0).stop 2024-04-02T09:13:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.589+0000 7f9f7e800700 1 -- 172.21.15.89:0/2550930520 reap_dead start 2024-04-02T09:13:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.590+0000 7f9f85000700 1 -- 172.21.15.89:0/2550930520 shutdown_connections 2024-04-02T09:13:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.590+0000 7f9f85000700 1 -- 172.21.15.89:0/2550930520 >> 172.21.15.89:0/2550930520 conn(0x7f9f800cf190 msgr2=0x7f9f800cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.590+0000 7f9f85000700 1 -- 172.21.15.89:0/2550930520 shutdown_connections 2024-04-02T09:13:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.590+0000 7f9f85000700 1 -- 172.21.15.89:0/2550930520 wait complete. 2024-04-02T09:13:24.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.591+0000 7f9f85000700 1 Processor -- start 2024-04-02T09:13:24.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.591+0000 7f9f85000700 1 -- start start 2024-04-02T09:13:24.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.592+0000 7f9f85000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f64004d90 0x7f9f8014ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:24.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.592+0000 7f9f85000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f800b0770 0x7f9f800cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:24.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.592+0000 7f9f85000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f800cc910 0x7f9f800ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:24.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.592+0000 7f9f85000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f800b0bf0 con 0x7f9f64004d90 2024-04-02T09:13:24.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.592+0000 7f9f85000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f800c9230 con 0x7f9f800b0770 2024-04-02T09:13:24.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.592+0000 7f9f85000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f800c9510 con 0x7f9f800cc910 2024-04-02T09:13:24.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.593+0000 7f9f7de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f64004d90 0x7f9f8014ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:24.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.593+0000 7f9f7de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f64004d90 0x7f9f8014ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54468/0 (socket says 172.21.15.89:54468) 2024-04-02T09:13:24.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.593+0000 7f9f7de00700 1 -- 172.21.15.89:0/2947392484 learned_addr learned my addr 172.21.15.89:0/2947392484 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:24.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.593+0000 7f9f7de00700 1 -- 172.21.15.89:0/2947392484 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f800cc910 msgr2=0x7f9f800ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:24.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.593+0000 7f9f7e800700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f800cc910 0x7f9f800ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:24.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.593+0000 7f9f7d400700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f800b0770 0x7f9f800cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:24.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.593+0000 7f9f7de00700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f800cc910 0x7f9f800ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:24.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.593+0000 7f9f7de00700 1 -- 172.21.15.89:0/2947392484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f800b0770 msgr2=0x7f9f800cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:24.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.593+0000 7f9f7de00700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f800b0770 0x7f9f800cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:24.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.593+0000 7f9f7de00700 1 -- 172.21.15.89:0/2947392484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f800c97f0 con 0x7f9f64004d90 2024-04-02T09:13:24.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.593+0000 7f9f7d400700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f800b0770 0x7f9f800cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:24.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.593+0000 7f9f7e800700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f800cc910 0x7f9f800ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:24.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.593+0000 7f9f7de00700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f64004d90 0x7f9f8014ae00 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000e7b0 tx=0x7f9f7000eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:24.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.593+0000 7f9f76000700 1 -- 172.21.15.89:0/2947392484 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f700052e0 con 0x7f9f64004d90 2024-04-02T09:13:24.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.594+0000 7f9f76000700 1 -- 172.21.15.89:0/2947392484 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f70005440 con 0x7f9f64004d90 2024-04-02T09:13:24.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.594+0000 7f9f85000700 1 -- 172.21.15.89:0/2947392484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f800c9ad0 con 0x7f9f64004d90 2024-04-02T09:13:24.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.594+0000 7f9f76000700 1 -- 172.21.15.89:0/2947392484 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f700038a0 con 0x7f9f64004d90 2024-04-02T09:13:24.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.594+0000 7f9f85000700 1 -- 172.21.15.89:0/2947392484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f801530d0 con 0x7f9f64004d90 2024-04-02T09:13:24.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.595+0000 7f9f85000700 1 -- 172.21.15.89:0/2947392484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f800ca120 con 0x7f9f64004d90 2024-04-02T09:13:24.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.596+0000 7f9f76000700 1 -- 172.21.15.89:0/2947392484 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f9f70003a00 con 0x7f9f64004d90 2024-04-02T09:13:24.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.597+0000 7f9f76000700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9f6006aed0 0x7f9f6006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:24.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.598+0000 7f9f76000700 1 -- 172.21.15.89:0/2947392484 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f9f70088c80 con 0x7f9f64004d90 2024-04-02T09:13:24.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.598+0000 7f9f7d400700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9f6006aed0 0x7f9f6006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:24.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.598+0000 7f9f7d400700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9f6006aed0 0x7f9f6006d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c001200 tx=0x7f9f6c001150 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:24.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.602+0000 7f9f76000700 1 -- 172.21.15.89:0/2947392484 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9f70053dd0 con 0x7f9f64004d90 2024-04-02T09:13:24.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.883+0000 7f9f85000700 1 -- 172.21.15.89:0/2947392484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9f800cba10 con 0x7f9f64004d90 2024-04-02T09:13:24.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.884+0000 7f9f76000700 1 -- 172.21.15.89:0/2947392484 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v54) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f9f70019020 con 0x7f9f64004d90 2024-04-02T09:13:24.883 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:24.883 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:13:24.883 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T09:13:24.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.883 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T09:13:24.883 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:13:24.884 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T09:13:24.885 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T09:13:24.886 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.886 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:13:24.886 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:13:24.886 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:13:24.886 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.886 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.886 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T09:13:24.886 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T09:13:24.886 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.886 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:13:24.886 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:13:24.886 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.887 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:13:24.888 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:13:24.888 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.888 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.888 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T09:13:24.888 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T09:13:24.888 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.888 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:13:24.888 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:13:24.888 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.888 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.890 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T09:13:24.890 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T09:13:24.890 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.890 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:13:24.890 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:13:24.890 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.890 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T09:13:24.890 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T09:13:24.891 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.891 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:13:24.891 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:13:24.891 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.891 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.891 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T09:13:24.891 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T09:13:24.891 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.891 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:13:24.891 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2024-04-02T09:13:24.891 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T09:13:24.891 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T09:13:24.892 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:13:24.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.894 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T09:13:24.894 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T09:13:24.894 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.894 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:13:24.894 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:13:24.894 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:13:24.894 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.894 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.895 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T09:13:24.895 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T09:13:24.895 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.895 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:13:24.895 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T09:13:24.895 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.895 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:24.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:24.895 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T09:13:24.895 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T09:13:24.895 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:13:24.895 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:13:24.896 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T09:13:24.896 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:13:24.896 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.896 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:24.896 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T09:13:24.896 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:13:24.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.886+0000 7f9f85000700 1 -- 172.21.15.89:0/2947392484 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9f6006aed0 msgr2=0x7f9f6006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:24.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.886+0000 7f9f85000700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9f6006aed0 0x7f9f6006d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c001200 tx=0x7f9f6c001150 comp rx=0 tx=0).stop 2024-04-02T09:13:24.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.886+0000 7f9f85000700 1 -- 172.21.15.89:0/2947392484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f64004d90 msgr2=0x7f9f8014ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:24.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.886+0000 7f9f85000700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f64004d90 0x7f9f8014ae00 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000e7b0 tx=0x7f9f7000eb70 comp rx=0 tx=0).stop 2024-04-02T09:13:24.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.886+0000 7f9f85000700 1 -- 172.21.15.89:0/2947392484 shutdown_connections 2024-04-02T09:13:24.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.886+0000 7f9f85000700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f800cc910 0x7f9f800ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:24.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.886+0000 7f9f85000700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f64004d90 0x7f9f8014ae00 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:24.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.886+0000 7f9f85000700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9f6006aed0 0x7f9f6006d390 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:24.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.886+0000 7f9f85000700 1 --2- 172.21.15.89:0/2947392484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f800b0770 0x7f9f800cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:24.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.886+0000 7f9f85000700 1 -- 172.21.15.89:0/2947392484 >> 172.21.15.89:0/2947392484 conn(0x7f9f800cf190 msgr2=0x7f9f8015dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:24.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.886+0000 7f9f85000700 1 -- 172.21.15.89:0/2947392484 shutdown_connections 2024-04-02T09:13:24.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:24.886+0000 7f9f85000700 1 -- 172.21.15.89:0/2947392484 wait complete. 2024-04-02T09:13:24.903 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:13:25.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.094+0000 7f0c30400700 1 Processor -- start 2024-04-02T09:13:25.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.095+0000 7f0c30400700 1 -- start start 2024-04-02T09:13:25.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.096+0000 7f0c30400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0c2815d640 0x7f0c280d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.096+0000 7f0c30400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0c280ceec0 0x7f0c281612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.096+0000 7f0c30400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c280d30d0 con 0x7f0c28161800 2024-04-02T09:13:25.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.096+0000 7f0c30400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c280b6840 con 0x7f0c2815d640 2024-04-02T09:13:25.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.096+0000 7f0c30400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c28161c20 con 0x7f0c280ceec0 2024-04-02T09:13:25.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.096+0000 7f0c2da00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0c2815d640 0x7f0c280d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.096+0000 7f0c2da00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0c2815d640 0x7f0c280d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54480/0 (socket says 172.21.15.89:54480) 2024-04-02T09:13:25.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.096+0000 7f0c2da00700 1 -- 172.21.15.89:0/3737148290 learned_addr learned my addr 172.21.15.89:0/3737148290 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:25.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.096+0000 7f0c2d000700 1 --2- 172.21.15.89:0/3737148290 >> v2:172.21.15.113:3300/0 conn(0x7f0c280ceec0 0x7f0c281612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.096+0000 7f0c2da00700 1 -- 172.21.15.89:0/3737148290 >> v2:172.21.15.113:3300/0 conn(0x7f0c280ceec0 msgr2=0x7f0c281612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:25.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.096+0000 7f0c2da00700 1 --2- 172.21.15.89:0/3737148290 >> v2:172.21.15.113:3300/0 conn(0x7f0c280ceec0 0x7f0c281612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.096+0000 7f0c2da00700 1 -- 172.21.15.89:0/3737148290 >> v1:172.21.15.113:6789/0 conn(0x7f0c28161800 legacy=0x7f0c280cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:25.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.096+0000 7f0c2da00700 1 -- 172.21.15.89:0/3737148290 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c280d2880 con 0x7f0c2815d640 2024-04-02T09:13:25.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.097+0000 7f0c2da00700 1 --2- 172.21.15.89:0/3737148290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c2815d640 0x7f0c280d2340 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f0c280cfbf0 tx=0x7f0c1c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0cd14748cc35050 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:25.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.097+0000 7f0c27e00700 1 -- 172.21.15.89:0/3737148290 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c004230 con 0x7f0c2815d640 2024-04-02T09:13:25.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.097+0000 7f0c27e00700 1 -- 172.21.15.89:0/3737148290 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1c004390 con 0x7f0c2815d640 2024-04-02T09:13:25.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.097+0000 7f0c27e00700 1 -- 172.21.15.89:0/3737148290 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c0036a0 con 0x7f0c2815d640 2024-04-02T09:13:25.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.097+0000 7f0c30400700 1 -- 172.21.15.89:0/3737148290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c2815d640 msgr2=0x7f0c280d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:25.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.097+0000 7f0c30400700 1 --2- 172.21.15.89:0/3737148290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c2815d640 0x7f0c280d2340 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f0c280cfbf0 tx=0x7f0c1c009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.098+0000 7f0c30400700 1 -- 172.21.15.89:0/3737148290 shutdown_connections 2024-04-02T09:13:25.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.098+0000 7f0c30400700 1 --2- 172.21.15.89:0/3737148290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c2815d640 0x7f0c280d2340 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.098+0000 7f0c30400700 1 --2- 172.21.15.89:0/3737148290 >> v2:172.21.15.113:3300/0 conn(0x7f0c280ceec0 0x7f0c281612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.098+0000 7f0c30400700 1 -- 172.21.15.89:0/3737148290 >> 172.21.15.89:0/3737148290 conn(0x7f0c2808aff0 msgr2=0x7f0c280d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:25.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.098+0000 7f0c30400700 1 -- 172.21.15.89:0/3737148290 shutdown_connections 2024-04-02T09:13:25.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.098+0000 7f0c30400700 1 -- 172.21.15.89:0/3737148290 wait complete. 2024-04-02T09:13:25.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.099+0000 7f0c30400700 1 Processor -- start 2024-04-02T09:13:25.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.099+0000 7f0c30400700 1 -- start start 2024-04-02T09:13:25.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.100+0000 7f0c30400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c280ceec0 0x7f0c280c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.100+0000 7f0c30400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c2815d640 0x7f0c2814e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.100+0000 7f0c30400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c28161800 0x7f0c28152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.100+0000 7f0c30400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c281f7720 con 0x7f0c280ceec0 2024-04-02T09:13:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.100+0000 7f0c30400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c280c81a0 con 0x7f0c2815d640 2024-04-02T09:13:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.100+0000 7f0c30400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c280c8480 con 0x7f0c28161800 2024-04-02T09:13:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.100+0000 7f0c2da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c280ceec0 0x7f0c280c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.100+0000 7f0c2da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c280ceec0 0x7f0c280c7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54488/0 (socket says 172.21.15.89:54488) 2024-04-02T09:13:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.100+0000 7f0c2da00700 1 -- 172.21.15.89:0/1575493190 learned_addr learned my addr 172.21.15.89:0/1575493190 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.101+0000 7f0c2d000700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c2815d640 0x7f0c2814e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.101+0000 7f0c2e400700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c28161800 0x7f0c28152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.101+0000 7f0c2da00700 1 -- 172.21.15.89:0/1575493190 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c28161800 msgr2=0x7f0c28152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.101+0000 7f0c2da00700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c28161800 0x7f0c28152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.101+0000 7f0c2da00700 1 -- 172.21.15.89:0/1575493190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c2815d640 msgr2=0x7f0c2814e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.101+0000 7f0c2da00700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c2815d640 0x7f0c2814e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.101+0000 7f0c2da00700 1 -- 172.21.15.89:0/1575493190 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c1c009550 con 0x7f0c280ceec0 2024-04-02T09:13:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.101+0000 7f0c2d000700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c2815d640 0x7f0c2814e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.101+0000 7f0c2da00700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c280ceec0 0x7f0c280c7b30 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c009130 tx=0x7f0c1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.101+0000 7f0c25600700 1 -- 172.21.15.89:0/1575493190 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c003d40 con 0x7f0c280ceec0 2024-04-02T09:13:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.101+0000 7f0c25600700 1 -- 172.21.15.89:0/1575493190 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1c01f070 con 0x7f0c280ceec0 2024-04-02T09:13:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.101+0000 7f0c30400700 1 -- 172.21.15.89:0/1575493190 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c280c8760 con 0x7f0c280ceec0 2024-04-02T09:13:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.101+0000 7f0c25600700 1 -- 172.21.15.89:0/1575493190 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c011870 con 0x7f0c280ceec0 2024-04-02T09:13:25.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.102+0000 7f0c30400700 1 -- 172.21.15.89:0/1575493190 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c280c8d20 con 0x7f0c280ceec0 2024-04-02T09:13:25.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.103+0000 7f0c30400700 1 -- 172.21.15.89:0/1575493190 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c280c8f40 con 0x7f0c280ceec0 2024-04-02T09:13:25.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.103+0000 7f0c25600700 1 -- 172.21.15.89:0/1575493190 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0c1c021030 con 0x7f0c280ceec0 2024-04-02T09:13:25.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.104+0000 7f0c25600700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0c0806aed0 0x7f0c0806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.104+0000 7f0c25600700 1 -- 172.21.15.89:0/1575493190 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f0c1c08a950 con 0x7f0c280ceec0 2024-04-02T09:13:25.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.104+0000 7f0c2d000700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0c0806aed0 0x7f0c0806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.105+0000 7f0c2d000700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0c0806aed0 0x7f0c0806d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0c28160980 tx=0x7f0c14008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:25.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.109+0000 7f0c25600700 1 -- 172.21.15.89:0/1575493190 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0c1c055ac0 con 0x7f0c280ceec0 2024-04-02T09:13:25.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.313+0000 7f0c30400700 1 -- 172.21.15.89:0/1575493190 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c28067720 con 0x7f0c280ceec0 2024-04-02T09:13:25.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.314+0000 7f0c25600700 1 -- 172.21.15.89:0/1575493190 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v240) v1 ==== 74+0+13088 (secure 0 0 0) 0x7f0c1c029090 con 0x7f0c280ceec0 2024-04-02T09:13:25.313 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:25.313 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":240,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:13:24.359373+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:13:25.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.316+0000 7f0c30400700 1 -- 172.21.15.89:0/1575493190 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0c0806aed0 msgr2=0x7f0c0806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:25.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.316+0000 7f0c30400700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0c0806aed0 0x7f0c0806d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0c28160980 tx=0x7f0c14008040 comp rx=0 tx=0).stop 2024-04-02T09:13:25.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.316+0000 7f0c30400700 1 -- 172.21.15.89:0/1575493190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c280ceec0 msgr2=0x7f0c280c7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:25.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.316+0000 7f0c30400700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c280ceec0 0x7f0c280c7b30 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c009130 tx=0x7f0c1c004750 comp rx=0 tx=0).stop 2024-04-02T09:13:25.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.316+0000 7f0c30400700 1 -- 172.21.15.89:0/1575493190 shutdown_connections 2024-04-02T09:13:25.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.316+0000 7f0c30400700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0c28161800 0x7f0c28152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.316+0000 7f0c30400700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c280ceec0 0x7f0c280c7b30 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.316+0000 7f0c30400700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0c0806aed0 0x7f0c0806d390 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.316+0000 7f0c30400700 1 --2- 172.21.15.89:0/1575493190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c2815d640 0x7f0c2814e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.316+0000 7f0c30400700 1 -- 172.21.15.89:0/1575493190 >> 172.21.15.89:0/1575493190 conn(0x7f0c2808aff0 msgr2=0x7f0c280b8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:25.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.316+0000 7f0c30400700 1 -- 172.21.15.89:0/1575493190 shutdown_connections 2024-04-02T09:13:25.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.316+0000 7f0c30400700 1 -- 172.21.15.89:0/1575493190 wait complete. 2024-04-02T09:13:25.334 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:13:25.334 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:13:25.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.530+0000 7f5fefa00700 1 Processor -- start 2024-04-02T09:13:25.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.531+0000 7f5fefa00700 1 -- start start 2024-04-02T09:13:25.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.532+0000 7f5fefa00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5fe81611b0 0x7f5fe815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.532+0000 7f5fefa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fe815f4d0 0x7f5fe80ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.532+0000 7f5fefa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5fe80d21f0 0x7f5fe80cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.532+0000 7f5fefa00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe80cf180 con 0x7f5fe80d21f0 2024-04-02T09:13:25.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.532+0000 7f5fefa00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe80ce270 con 0x7f5fe81611b0 2024-04-02T09:13:25.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.532+0000 7f5fefa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe815b600 con 0x7f5fe815f4d0 2024-04-02T09:13:25.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.532+0000 7f5feda00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5fe80d21f0 0x7f5fe80cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.532+0000 7f5feda00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5fe80d21f0 0x7f5fe80cec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54492/0 (socket says 172.21.15.89:54492) 2024-04-02T09:13:25.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.532+0000 7f5feda00700 1 -- 172.21.15.89:0/3760161532 learned_addr learned my addr 172.21.15.89:0/3760161532 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:25.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.532+0000 7f5fed000700 1 --2- 172.21.15.89:0/3760161532 >> v2:172.21.15.113:3300/0 conn(0x7f5fe81611b0 0x7f5fe815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.532+0000 7f5fe7e00700 1 --2- 172.21.15.89:0/3760161532 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fe815f4d0 0x7f5fe80ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.533+0000 7f5feda00700 1 -- 172.21.15.89:0/3760161532 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fe815f4d0 msgr2=0x7f5fe80ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:25.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.533+0000 7f5feda00700 1 --2- 172.21.15.89:0/3760161532 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fe815f4d0 0x7f5fe80ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.533+0000 7f5feda00700 1 -- 172.21.15.89:0/3760161532 >> v2:172.21.15.113:3300/0 conn(0x7f5fe81611b0 msgr2=0x7f5fe815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:25.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.533+0000 7f5feda00700 1 --2- 172.21.15.89:0/3760161532 >> v2:172.21.15.113:3300/0 conn(0x7f5fe81611b0 0x7f5fe815ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.533+0000 7f5feda00700 1 -- 172.21.15.89:0/3760161532 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe80d2760 con 0x7f5fe80d21f0 2024-04-02T09:13:25.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.533+0000 7f5feda00700 1 --2- 172.21.15.89:0/3760161532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fe80d21f0 0x7f5fe80cec40 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f5fe808b520 tx=0x7f5fe000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ab95657b0dd166 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:25.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.533+0000 7f5fe7400700 1 -- 172.21.15.89:0/3760161532 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe000e070 con 0x7f5fe80d21f0 2024-04-02T09:13:25.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.533+0000 7f5fe7400700 1 -- 172.21.15.89:0/3760161532 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe000b910 con 0x7f5fe80d21f0 2024-04-02T09:13:25.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.533+0000 7f5fe7400700 1 -- 172.21.15.89:0/3760161532 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe000bcc0 con 0x7f5fe80d21f0 2024-04-02T09:13:25.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.534+0000 7f5fefa00700 1 -- 172.21.15.89:0/3760161532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fe80d21f0 msgr2=0x7f5fe80cec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:25.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.534+0000 7f5fefa00700 1 --2- 172.21.15.89:0/3760161532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fe80d21f0 0x7f5fe80cec40 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f5fe808b520 tx=0x7f5fe000b330 comp rx=0 tx=0).stop 2024-04-02T09:13:25.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.534+0000 7f5fefa00700 1 -- 172.21.15.89:0/3760161532 shutdown_connections 2024-04-02T09:13:25.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.534+0000 7f5fefa00700 1 --2- 172.21.15.89:0/3760161532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fe80d21f0 0x7f5fe80cec40 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.534+0000 7f5fefa00700 1 --2- 172.21.15.89:0/3760161532 >> v2:172.21.15.113:3300/0 conn(0x7f5fe81611b0 0x7f5fe815ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.534+0000 7f5fefa00700 1 --2- 172.21.15.89:0/3760161532 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fe815f4d0 0x7f5fe80ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.534+0000 7f5fefa00700 1 -- 172.21.15.89:0/3760161532 >> 172.21.15.89:0/3760161532 conn(0x7f5fe808aff0 msgr2=0x7f5fe80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:25.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.534+0000 7f5fefa00700 1 -- 172.21.15.89:0/3760161532 shutdown_connections 2024-04-02T09:13:25.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.534+0000 7f5fefa00700 1 -- 172.21.15.89:0/3760161532 wait complete. 2024-04-02T09:13:25.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.536+0000 7f5fefa00700 1 Processor -- start 2024-04-02T09:13:25.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.536+0000 7f5fefa00700 1 -- start start 2024-04-02T09:13:25.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.537+0000 7f5fefa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fe80d21f0 0x7f5fe80cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.537+0000 7f5fefa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fe815f4d0 0x7f5fe8157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.538+0000 7f5fefa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5fe81611b0 0x7f5fe80cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.538+0000 7f5fefa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe80cc1a0 con 0x7f5fe815f4d0 2024-04-02T09:13:25.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.538+0000 7f5fefa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe80cc480 con 0x7f5fe81611b0 2024-04-02T09:13:25.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.538+0000 7f5fefa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe80cc760 con 0x7f5fe80d21f0 2024-04-02T09:13:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.538+0000 7f5fe7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fe815f4d0 0x7f5fe8157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.538+0000 7f5fe7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fe815f4d0 0x7f5fe8157cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54502/0 (socket says 172.21.15.89:54502) 2024-04-02T09:13:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.538+0000 7f5fe7e00700 1 -- 172.21.15.89:0/2209434730 learned_addr learned my addr 172.21.15.89:0/2209434730 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.538+0000 7f5fed000700 1 --2- 172.21.15.89:0/2209434730 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fe80d21f0 0x7f5fe80cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.538+0000 7f5feda00700 1 --2- 172.21.15.89:0/2209434730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5fe81611b0 0x7f5fe80cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.538+0000 7f5fe7e00700 1 -- 172.21.15.89:0/2209434730 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fe80d21f0 msgr2=0x7f5fe80cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.538+0000 7f5fe7e00700 1 --2- 172.21.15.89:0/2209434730 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fe80d21f0 0x7f5fe80cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.538+0000 7f5fe7e00700 1 -- 172.21.15.89:0/2209434730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5fe81611b0 msgr2=0x7f5fe80cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.538+0000 7f5fe7e00700 1 --2- 172.21.15.89:0/2209434730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5fe81611b0 0x7f5fe80cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.538+0000 7f5fe7e00700 1 -- 172.21.15.89:0/2209434730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe0009b30 con 0x7f5fe815f4d0 2024-04-02T09:13:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.539+0000 7f5fe7e00700 1 --2- 172.21.15.89:0/2209434730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fe815f4d0 0x7f5fe8157cc0 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400e8d0 tx=0x7f5fd400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.539+0000 7f5fe4c00700 1 -- 172.21.15.89:0/2209434730 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd400eed0 con 0x7f5fe815f4d0 2024-04-02T09:13:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.539+0000 7f5fe4c00700 1 -- 172.21.15.89:0/2209434730 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd400c850 con 0x7f5fe815f4d0 2024-04-02T09:13:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.539+0000 7f5fefa00700 1 -- 172.21.15.89:0/2209434730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe80cca40 con 0x7f5fe815f4d0 2024-04-02T09:13:25.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.539+0000 7f5fe4c00700 1 -- 172.21.15.89:0/2209434730 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd400cbc0 con 0x7f5fe815f4d0 2024-04-02T09:13:25.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.539+0000 7f5fefa00700 1 -- 172.21.15.89:0/2209434730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe8158200 con 0x7f5fe815f4d0 2024-04-02T09:13:25.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.540+0000 7f5fefa00700 1 -- 172.21.15.89:0/2209434730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe80cd090 con 0x7f5fe815f4d0 2024-04-02T09:13:25.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.541+0000 7f5fe4c00700 1 -- 172.21.15.89:0/2209434730 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5fd400cd20 con 0x7f5fe815f4d0 2024-04-02T09:13:25.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.542+0000 7f5fe4c00700 1 --2- 172.21.15.89:0/2209434730 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5fc806aed0 0x7f5fc806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.542+0000 7f5fe4c00700 1 -- 172.21.15.89:0/2209434730 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f5fd40896b0 con 0x7f5fe815f4d0 2024-04-02T09:13:25.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.542+0000 7f5fed000700 1 --2- 172.21.15.89:0/2209434730 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5fc806aed0 0x7f5fc806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.543+0000 7f5fed000700 1 --2- 172.21.15.89:0/2209434730 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5fc806aed0 0x7f5fc806d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc009ab0 tx=0x7f5fdc008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:25.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.546+0000 7f5fe4c00700 1 -- 172.21.15.89:0/2209434730 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5fd4054620 con 0x7f5fe815f4d0 2024-04-02T09:13:25.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.747+0000 7f5fefa00700 1 -- 172.21.15.89:0/2209434730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5fe8067720 con 0x7f5fe815f4d0 2024-04-02T09:13:25.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.747+0000 7f5fe4c00700 1 -- 172.21.15.89:0/2209434730 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v240) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5fd4057c40 con 0x7f5fe815f4d0 2024-04-02T09:13:25.747 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:13:25.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.750+0000 7f5fefa00700 1 -- 172.21.15.89:0/2209434730 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5fc806aed0 msgr2=0x7f5fc806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:25.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.750+0000 7f5fefa00700 1 --2- 172.21.15.89:0/2209434730 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5fc806aed0 0x7f5fc806d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc009ab0 tx=0x7f5fdc008040 comp rx=0 tx=0).stop 2024-04-02T09:13:25.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.750+0000 7f5fefa00700 1 -- 172.21.15.89:0/2209434730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fe815f4d0 msgr2=0x7f5fe8157cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:25.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.750+0000 7f5fefa00700 1 --2- 172.21.15.89:0/2209434730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fe815f4d0 0x7f5fe8157cc0 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400e8d0 tx=0x7f5fd400ec90 comp rx=0 tx=0).stop 2024-04-02T09:13:25.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.750+0000 7f5fefa00700 1 -- 172.21.15.89:0/2209434730 shutdown_connections 2024-04-02T09:13:25.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.750+0000 7f5fefa00700 1 --2- 172.21.15.89:0/2209434730 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5fc806aed0 0x7f5fc806d390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.750+0000 7f5fefa00700 1 --2- 172.21.15.89:0/2209434730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5fe81611b0 0x7f5fe80cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.750+0000 7f5fefa00700 1 --2- 172.21.15.89:0/2209434730 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5fe80d21f0 0x7f5fe80cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.750+0000 7f5fefa00700 1 --2- 172.21.15.89:0/2209434730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fe815f4d0 0x7f5fe8157cc0 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.750+0000 7f5fefa00700 1 -- 172.21.15.89:0/2209434730 >> 172.21.15.89:0/2209434730 conn(0x7f5fe808aff0 msgr2=0x7f5fe806c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:25.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.750+0000 7f5fefa00700 1 -- 172.21.15.89:0/2209434730 shutdown_connections 2024-04-02T09:13:25.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.750+0000 7f5fefa00700 1 -- 172.21.15.89:0/2209434730 wait complete. 2024-04-02T09:13:25.767 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T09:13:25.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.959+0000 7fa3e3200700 1 Processor -- start 2024-04-02T09:13:25.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.959+0000 7fa3e3200700 1 -- start start 2024-04-02T09:13:25.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.960+0000 7fa3e3200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa3dc15b5d0 0x7fa3dc0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.960+0000 7fa3e3200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa3dc0cb3f0 0x7fa3dc0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.960+0000 7fa3e3200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3dc15fd50 con 0x7fa3dc0ae930 2024-04-02T09:13:25.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.960+0000 7fa3e3200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3dc15b9b0 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.960+0000 7fa3e3200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3dc15f140 con 0x7fa3dc0cb3f0 2024-04-02T09:13:25.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.960+0000 7fa3dbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa3dc15b5d0 0x7fa3dc0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.960+0000 7fa3dbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa3dc15b5d0 0x7fa3dc0ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54504/0 (socket says 172.21.15.89:54504) 2024-04-02T09:13:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.960+0000 7fa3dbe00700 1 -- 172.21.15.89:0/73632833 learned_addr learned my addr 172.21.15.89:0/73632833 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.961+0000 7fa3db400700 1 --2- 172.21.15.89:0/73632833 >> v2:172.21.15.113:3300/0 conn(0x7fa3dc0cb3f0 0x7fa3dc0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.961+0000 7fa3dbe00700 1 -- 172.21.15.89:0/73632833 >> v2:172.21.15.113:3300/0 conn(0x7fa3dc0cb3f0 msgr2=0x7fa3dc0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.961+0000 7fa3dbe00700 1 --2- 172.21.15.89:0/73632833 >> v2:172.21.15.113:3300/0 conn(0x7fa3dc0cb3f0 0x7fa3dc0ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.961+0000 7fa3dbe00700 1 -- 172.21.15.89:0/73632833 >> v1:172.21.15.113:6789/0 conn(0x7fa3dc0ae930 legacy=0x7fa3dc140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.961+0000 7fa3dbe00700 1 -- 172.21.15.89:0/73632833 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3dc15f420 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.961+0000 7fa3dbe00700 1 --2- 172.21.15.89:0/73632833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3dc15b5d0 0x7fa3dc0ae3f0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc06c6c0 tx=0x7fa3cc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e954e71064cbfdbf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:25.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.961+0000 7fa3daa00700 1 -- 172.21.15.89:0/73632833 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc00b870 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.961+0000 7fa3daa00700 1 -- 172.21.15.89:0/73632833 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3cc00b9d0 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.961+0000 7fa3daa00700 1 -- 172.21.15.89:0/73632833 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc00bd60 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.962+0000 7fa3e3200700 1 -- 172.21.15.89:0/73632833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3dc15b5d0 msgr2=0x7fa3dc0ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:25.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.962+0000 7fa3e3200700 1 --2- 172.21.15.89:0/73632833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3dc15b5d0 0x7fa3dc0ae3f0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc06c6c0 tx=0x7fa3cc009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.962+0000 7fa3e3200700 1 -- 172.21.15.89:0/73632833 shutdown_connections 2024-04-02T09:13:25.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.962+0000 7fa3e3200700 1 --2- 172.21.15.89:0/73632833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3dc15b5d0 0x7fa3dc0ae3f0 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.962+0000 7fa3e3200700 1 --2- 172.21.15.89:0/73632833 >> v2:172.21.15.113:3300/0 conn(0x7fa3dc0cb3f0 0x7fa3dc0ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.962+0000 7fa3e3200700 1 -- 172.21.15.89:0/73632833 >> 172.21.15.89:0/73632833 conn(0x7fa3dc0cf190 msgr2=0x7fa3dc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:25.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.962+0000 7fa3e3200700 1 -- 172.21.15.89:0/73632833 shutdown_connections 2024-04-02T09:13:25.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.962+0000 7fa3e3200700 1 -- 172.21.15.89:0/73632833 wait complete. 2024-04-02T09:13:25.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.964+0000 7fa3e3200700 1 Processor -- start 2024-04-02T09:13:25.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.964+0000 7fa3e3200700 1 -- start start 2024-04-02T09:13:25.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3e3200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3dc0ae930 0x7fa3dc0ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3e3200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3dc15b5d0 0x7fa3dc155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3e3200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3dc0c8460 0x7fa3dc1561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3e3200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc1581e0 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3e3200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc0c8c20 con 0x7fa3dc0c8460 2024-04-02T09:13:25.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3e3200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc0c8f00 con 0x7fa3dc0ae930 2024-04-02T09:13:25.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3db400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3dc15b5d0 0x7fa3dc155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3db400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3dc15b5d0 0x7fa3dc155410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54514/0 (socket says 172.21.15.89:54514) 2024-04-02T09:13:25.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3db400700 1 -- 172.21.15.89:0/483906390 learned_addr learned my addr 172.21.15.89:0/483906390 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:25.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3dbe00700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3dc0ae930 0x7fa3dc0ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3e1200700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3dc0c8460 0x7fa3dc1561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3db400700 1 -- 172.21.15.89:0/483906390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3dc0ae930 msgr2=0x7fa3dc0ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:25.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3db400700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3dc0ae930 0x7fa3dc0ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3db400700 1 -- 172.21.15.89:0/483906390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3dc0c8460 msgr2=0x7fa3dc1561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3db400700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3dc0c8460 0x7fa3dc1561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3db400700 1 -- 172.21.15.89:0/483906390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc009550 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.965+0000 7fa3e1200700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3dc0c8460 0x7fa3dc1561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.966+0000 7fa3db400700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3dc15b5d0 0x7fa3dc155410 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800e810 tx=0x7fa3c800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.966+0000 7fa3d3e00700 1 -- 172.21.15.89:0/483906390 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8013610 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.966+0000 7fa3d3e00700 1 -- 172.21.15.89:0/483906390 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c8013770 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.966+0000 7fa3d3e00700 1 -- 172.21.15.89:0/483906390 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8013b20 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.966+0000 7fa3e3200700 1 -- 172.21.15.89:0/483906390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3dc0c91e0 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.966+0000 7fa3e3200700 1 -- 172.21.15.89:0/483906390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3dc1566f0 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.967+0000 7fa3e3200700 1 -- 172.21.15.89:0/483906390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3dc1586a0 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.968+0000 7fa3d3e00700 1 -- 172.21.15.89:0/483906390 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa3c8013c80 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.969+0000 7fa3d3e00700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3bc06aed0 0x7fa3bc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:25.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.969+0000 7fa3d3e00700 1 -- 172.21.15.89:0/483906390 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fa3c80894c0 con 0x7fa3dc15b5d0 2024-04-02T09:13:25.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.969+0000 7fa3dbe00700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3bc06aed0 0x7fa3bc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:25.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.970+0000 7fa3dbe00700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3bc06aed0 0x7fa3bc06d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc000c00 tx=0x7fa3cc008030 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:25.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:25.973+0000 7fa3d3e00700 1 -- 172.21.15.89:0/483906390 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa3dc1586a0 con 0x7fa3dc15b5d0 2024-04-02T09:13:26.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.174+0000 7fa3e3200700 1 -- 172.21.15.89:0/483906390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa3dc1586a0 con 0x7fa3dc15b5d0 2024-04-02T09:13:26.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.175+0000 7fa3d3e00700 1 -- 172.21.15.89:0/483906390 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v240) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa3dc1586a0 con 0x7fa3dc15b5d0 2024-04-02T09:13:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:26.174 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T09:13:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:26.174 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T09:13:26.175 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T09:13:26.175 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:26.175 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T09:13:26.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.177+0000 7fa3e3200700 1 -- 172.21.15.89:0/483906390 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3bc06aed0 msgr2=0x7fa3bc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:26.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.177+0000 7fa3e3200700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3bc06aed0 0x7fa3bc06d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc000c00 tx=0x7fa3cc008030 comp rx=0 tx=0).stop 2024-04-02T09:13:26.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.177+0000 7fa3e3200700 1 -- 172.21.15.89:0/483906390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3dc15b5d0 msgr2=0x7fa3dc155410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:26.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.177+0000 7fa3e3200700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3dc15b5d0 0x7fa3dc155410 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800e810 tx=0x7fa3c800ebd0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.177+0000 7fa3e3200700 1 -- 172.21.15.89:0/483906390 shutdown_connections 2024-04-02T09:13:26.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.177+0000 7fa3e3200700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa3bc06aed0 0x7fa3bc06d390 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.177+0000 7fa3e3200700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3dc0c8460 0x7fa3dc1561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.177+0000 7fa3e3200700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa3dc0ae930 0x7fa3dc0ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.177+0000 7fa3e3200700 1 --2- 172.21.15.89:0/483906390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3dc15b5d0 0x7fa3dc155410 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.177+0000 7fa3e3200700 1 -- 172.21.15.89:0/483906390 >> 172.21.15.89:0/483906390 conn(0x7fa3dc0cf190 msgr2=0x7fa3dc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:26.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.177+0000 7fa3e3200700 1 -- 172.21.15.89:0/483906390 shutdown_connections 2024-04-02T09:13:26.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.177+0000 7fa3e3200700 1 -- 172.21.15.89:0/483906390 wait complete. 2024-04-02T09:13:26.195 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T09:13:26.195 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T09:13:26.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.391+0000 7fe802e00700 1 Processor -- start 2024-04-02T09:13:26.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.391+0000 7fe802e00700 1 -- start start 2024-04-02T09:13:26.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.392+0000 7fe802e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7fc1580c0 0x7fe7fc06e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:26.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.392+0000 7fe802e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe7fc15b330 0x7fe7fc0b2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:26.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.393+0000 7fe802e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc070980 con 0x7fe7fc06e7f0 2024-04-02T09:13:26.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.393+0000 7fe802e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7fc071570 con 0x7fe7fc15b330 2024-04-02T09:13:26.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.393+0000 7fe802e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc071990 con 0x7fe7fc1580c0 2024-04-02T09:13:26.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.393+0000 7fe800e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe7fc15b330 0x7fe7fc0b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:26.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.393+0000 7fe800e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe7fc15b330 0x7fe7fc0b2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54518/0 (socket says 172.21.15.89:54518) 2024-04-02T09:13:26.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.393+0000 7fe800e00700 1 -- 172.21.15.89:0/2811357950 learned_addr learned my addr 172.21.15.89:0/2811357950 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:26.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.393+0000 7fe7fbe00700 1 --2- 172.21.15.89:0/2811357950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7fc1580c0 0x7fe7fc06e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:26.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.393+0000 7fe800e00700 1 -- 172.21.15.89:0/2811357950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7fc1580c0 msgr2=0x7fe7fc06e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:26.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.393+0000 7fe800e00700 1 --2- 172.21.15.89:0/2811357950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7fc1580c0 0x7fe7fc06e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.393+0000 7fe800e00700 1 -- 172.21.15.89:0/2811357950 >> v1:172.21.15.113:6789/0 conn(0x7fe7fc06e7f0 legacy=0x7fe7fc15ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:26.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.393+0000 7fe800e00700 1 -- 172.21.15.89:0/2811357950 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc15b760 con 0x7fe7fc15b330 2024-04-02T09:13:26.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.394+0000 7fe800e00700 1 --2- 172.21.15.89:0/2811357950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7fc15b330 0x7fe7fc0b2180 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400d150 tx=0x7fe7f400d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f865f27bf8b16ae server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:26.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.394+0000 7fe7fb400700 1 -- 172.21.15.89:0/2811357950 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4010070 con 0x7fe7fc15b330 2024-04-02T09:13:26.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.394+0000 7fe7fb400700 1 -- 172.21.15.89:0/2811357950 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f400ddc0 con 0x7fe7fc15b330 2024-04-02T09:13:26.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.394+0000 7fe7fb400700 1 -- 172.21.15.89:0/2811357950 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f400b310 con 0x7fe7fc15b330 2024-04-02T09:13:26.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.394+0000 7fe802e00700 1 -- 172.21.15.89:0/2811357950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7fc15b330 msgr2=0x7fe7fc0b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:26.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.394+0000 7fe802e00700 1 --2- 172.21.15.89:0/2811357950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7fc15b330 0x7fe7fc0b2180 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400d150 tx=0x7fe7f400d460 comp rx=0 tx=0).stop 2024-04-02T09:13:26.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.394+0000 7fe802e00700 1 -- 172.21.15.89:0/2811357950 shutdown_connections 2024-04-02T09:13:26.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.394+0000 7fe802e00700 1 --2- 172.21.15.89:0/2811357950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7fc1580c0 0x7fe7fc06e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.394+0000 7fe802e00700 1 --2- 172.21.15.89:0/2811357950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7fc15b330 0x7fe7fc0b2180 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.394+0000 7fe802e00700 1 -- 172.21.15.89:0/2811357950 >> 172.21.15.89:0/2811357950 conn(0x7fe7fc0cf210 msgr2=0x7fe7fc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:26.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.394+0000 7fe802e00700 1 -- 172.21.15.89:0/2811357950 shutdown_connections 2024-04-02T09:13:26.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.395+0000 7fe802e00700 1 -- 172.21.15.89:0/2811357950 wait complete. 2024-04-02T09:13:26.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.396+0000 7fe802e00700 1 Processor -- start 2024-04-02T09:13:26.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.396+0000 7fe802e00700 1 -- start start 2024-04-02T09:13:26.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.397+0000 7fe802e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7fc06e7f0 0x7fe7fc0992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:26.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.397+0000 7fe802e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7fc1580c0 0x7fe7fc09c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:26.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.397+0000 7fe802e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7fc15b330 0x7fe7fc1f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:26.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.397+0000 7fe802e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc099970 con 0x7fe7fc06e7f0 2024-04-02T09:13:26.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.397+0000 7fe802e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc099c50 con 0x7fe7fc1580c0 2024-04-02T09:13:26.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.397+0000 7fe802e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc099f30 con 0x7fe7fc15b330 2024-04-02T09:13:26.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.397+0000 7fe7fbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7fc06e7f0 0x7fe7fc0992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:26.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.397+0000 7fe7fbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7fc06e7f0 0x7fe7fc0992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54528/0 (socket says 172.21.15.89:54528) 2024-04-02T09:13:26.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.397+0000 7fe7fbe00700 1 -- 172.21.15.89:0/3324421188 learned_addr learned my addr 172.21.15.89:0/3324421188 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:26.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.398+0000 7fe7f3e00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7fc1580c0 0x7fe7fc09c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:26.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.398+0000 7fe800e00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7fc15b330 0x7fe7fc1f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.398+0000 7fe7fbe00700 1 -- 172.21.15.89:0/3324421188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7fc15b330 msgr2=0x7fe7fc1f2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.398+0000 7fe7fbe00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7fc15b330 0x7fe7fc1f2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.398+0000 7fe7fbe00700 1 -- 172.21.15.89:0/3324421188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7fc1580c0 msgr2=0x7fe7fc09c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.398+0000 7fe7fbe00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7fc1580c0 0x7fe7fc09c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.398+0000 7fe7fbe00700 1 -- 172.21.15.89:0/3324421188 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f400bd40 con 0x7fe7fc06e7f0 2024-04-02T09:13:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.398+0000 7fe800e00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7fc15b330 0x7fe7fc1f2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.398+0000 7fe7fbe00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7fc06e7f0 0x7fe7fc0992e0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00e810 tx=0x7fe7ec00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.398+0000 7fe7f8c00700 1 -- 172.21.15.89:0/3324421188 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec013610 con 0x7fe7fc06e7f0 2024-04-02T09:13:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.398+0000 7fe7f8c00700 1 -- 172.21.15.89:0/3324421188 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec013770 con 0x7fe7fc06e7f0 2024-04-02T09:13:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.398+0000 7fe802e00700 1 -- 172.21.15.89:0/3324421188 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc09a210 con 0x7fe7fc06e7f0 2024-04-02T09:13:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.398+0000 7fe7f8c00700 1 -- 172.21.15.89:0/3324421188 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec013b20 con 0x7fe7fc06e7f0 2024-04-02T09:13:26.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.398+0000 7fe802e00700 1 -- 172.21.15.89:0/3324421188 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc09cc20 con 0x7fe7fc06e7f0 2024-04-02T09:13:26.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.399+0000 7fe802e00700 1 -- 172.21.15.89:0/3324421188 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc09a860 con 0x7fe7fc06e7f0 2024-04-02T09:13:26.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.400+0000 7fe7f8c00700 1 -- 172.21.15.89:0/3324421188 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe7ec013c80 con 0x7fe7fc06e7f0 2024-04-02T09:13:26.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.401+0000 7fe7f8c00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe7d806aed0 0x7fe7d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:26.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.401+0000 7fe7f8c00700 1 -- 172.21.15.89:0/3324421188 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fe7ec0893d0 con 0x7fe7fc06e7f0 2024-04-02T09:13:26.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.402+0000 7fe7f3e00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe7d806aed0 0x7fe7d806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:26.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.402+0000 7fe7f3e00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe7d806aed0 0x7fe7d806d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc06d9b0 tx=0x7fe7e4009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:26.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.406+0000 7fe7f8c00700 1 -- 172.21.15.89:0/3324421188 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe7ec054240 con 0x7fe7fc06e7f0 2024-04-02T09:13:26.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.607+0000 7fe802e00700 1 -- 172.21.15.89:0/3324421188 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe7fc0b4710 con 0x7fe7fc06e7f0 2024-04-02T09:13:26.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.855+0000 7fe7f8c00700 1 -- 172.21.15.89:0/3324421188 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v241) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe7ec057860 con 0x7fe7fc06e7f0 2024-04-02T09:13:26.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.858+0000 7fe802e00700 1 -- 172.21.15.89:0/3324421188 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe7d806aed0 msgr2=0x7fe7d806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:26.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.858+0000 7fe802e00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe7d806aed0 0x7fe7d806d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc06d9b0 tx=0x7fe7e4009040 comp rx=0 tx=0).stop 2024-04-02T09:13:26.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.858+0000 7fe802e00700 1 -- 172.21.15.89:0/3324421188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7fc06e7f0 msgr2=0x7fe7fc0992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:26.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.858+0000 7fe802e00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7fc06e7f0 0x7fe7fc0992e0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00e810 tx=0x7fe7ec00ebd0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.859+0000 7fe802e00700 1 -- 172.21.15.89:0/3324421188 shutdown_connections 2024-04-02T09:13:26.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.859+0000 7fe802e00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe7fc15b330 0x7fe7fc1f2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.859+0000 7fe802e00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7fc06e7f0 0x7fe7fc0992e0 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.859+0000 7fe802e00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe7d806aed0 0x7fe7d806d390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.859+0000 7fe802e00700 1 --2- 172.21.15.89:0/3324421188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7fc1580c0 0x7fe7fc09c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:26.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.859+0000 7fe802e00700 1 -- 172.21.15.89:0/3324421188 >> 172.21.15.89:0/3324421188 conn(0x7fe7fc0cf210 msgr2=0x7fe7fc06dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:26.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.859+0000 7fe802e00700 1 -- 172.21.15.89:0/3324421188 shutdown_connections 2024-04-02T09:13:26.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:26.860+0000 7fe802e00700 1 -- 172.21.15.89:0/3324421188 wait complete. 2024-04-02T09:13:26.861 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T09:13:26.879 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T09:13:27.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.077+0000 7f6bbc800700 1 Processor -- start 2024-04-02T09:13:27.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.077+0000 7f6bbc800700 1 -- start start 2024-04-02T09:13:27.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.078+0000 7f6bbc800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6bb415a5c0 0x7f6bb40709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:27.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.078+0000 7f6bbc800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6bb40ce840 0x7f6bb40cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:27.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.078+0000 7f6bbc800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb40cb310 con 0x7f6bb415dfe0 2024-04-02T09:13:27.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.078+0000 7f6bbc800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb4070f00 con 0x7f6bb40ce840 2024-04-02T09:13:27.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.078+0000 7f6bbc800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb40711a0 con 0x7f6bb415a5c0 2024-04-02T09:13:27.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.079+0000 7f6bb9400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6bb415dfe0 0x7f6bb40ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48464/0 (socket says 172.21.15.89:48464) 2024-04-02T09:13:27.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.079+0000 7f6bb9400700 1 -- 172.21.15.89:0/4213998093 learned_addr learned my addr 172.21.15.89:0/4213998093 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:27.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.079+0000 7f6bba800700 1 --2- 172.21.15.89:0/4213998093 >> v2:172.21.15.89:3300/0 conn(0x7f6bb40ce840 0x7f6bb40cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:27.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.079+0000 7f6bb9e00700 1 --2- 172.21.15.89:0/4213998093 >> v2:172.21.15.113:3300/0 conn(0x7f6bb415a5c0 0x7f6bb40709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:27.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.079+0000 7f6bb8a00700 1 -- 172.21.15.89:0/4213998093 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539266581 0 0) 0x7f6bb40cb310 con 0x7f6bb415dfe0 2024-04-02T09:13:27.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.079+0000 7f6bb8a00700 1 -- 172.21.15.89:0/4213998093 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b9c0035b0 con 0x7f6bb415dfe0 2024-04-02T09:13:27.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.079+0000 7f6bba800700 1 -- 172.21.15.89:0/4213998093 >> v2:172.21.15.113:3300/0 conn(0x7f6bb415a5c0 msgr2=0x7f6bb40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:27.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.079+0000 7f6bba800700 1 --2- 172.21.15.89:0/4213998093 >> v2:172.21.15.113:3300/0 conn(0x7f6bb415a5c0 0x7f6bb40709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:27.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.079+0000 7f6bba800700 1 -- 172.21.15.89:0/4213998093 >> v1:172.21.15.89:6789/0 conn(0x7f6bb415dfe0 legacy=0x7f6bb40ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:27.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.079+0000 7f6bba800700 1 -- 172.21.15.89:0/4213998093 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb4071480 con 0x7f6bb40ce840 2024-04-02T09:13:27.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.080+0000 7f6bba800700 1 --2- 172.21.15.89:0/4213998093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb40ce840 0x7f6bb40cba80 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4072e70 tx=0x7f6bb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e7b18b66d6cb97 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:27.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.080+0000 7f6bb8a00700 1 -- 172.21.15.89:0/4213998093 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb000e070 con 0x7f6bb40ce840 2024-04-02T09:13:27.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.080+0000 7f6bb8a00700 1 -- 172.21.15.89:0/4213998093 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb000bb20 con 0x7f6bb40ce840 2024-04-02T09:13:27.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.080+0000 7f6bb8a00700 1 -- 172.21.15.89:0/4213998093 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0012430 con 0x7f6bb40ce840 2024-04-02T09:13:27.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.080+0000 7f6bbc800700 1 -- 172.21.15.89:0/4213998093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb40ce840 msgr2=0x7f6bb40cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:27.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.080+0000 7f6bbc800700 1 --2- 172.21.15.89:0/4213998093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb40ce840 0x7f6bb40cba80 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4072e70 tx=0x7f6bb000b330 comp rx=0 tx=0).stop 2024-04-02T09:13:27.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.080+0000 7f6bbc800700 1 -- 172.21.15.89:0/4213998093 shutdown_connections 2024-04-02T09:13:27.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.080+0000 7f6bbc800700 1 --2- 172.21.15.89:0/4213998093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb40ce840 0x7f6bb40cba80 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:27.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.080+0000 7f6bbc800700 1 --2- 172.21.15.89:0/4213998093 >> v2:172.21.15.113:3300/0 conn(0x7f6bb415a5c0 0x7f6bb40709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:27.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.080+0000 7f6bbc800700 1 -- 172.21.15.89:0/4213998093 >> 172.21.15.89:0/4213998093 conn(0x7f6bb40b0d70 msgr2=0x7f6bb40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:27.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.081+0000 7f6bbc800700 1 -- 172.21.15.89:0/4213998093 shutdown_connections 2024-04-02T09:13:27.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.081+0000 7f6bbc800700 1 -- 172.21.15.89:0/4213998093 wait complete. 2024-04-02T09:13:27.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.082+0000 7f6bbc800700 1 Processor -- start 2024-04-02T09:13:27.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.082+0000 7f6bbc800700 1 -- start start 2024-04-02T09:13:27.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.083+0000 7f6bbc800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bb40ce840 0x7f6bb4099520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:27.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.083+0000 7f6bbc800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb415a5c0 0x7f6bb41465e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:27.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.083+0000 7f6bbc800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb415dfe0 0x7f6bb4147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.083+0000 7f6bbc800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb4146b20 con 0x7f6bb415a5c0 2024-04-02T09:13:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.083+0000 7f6bbc800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb4146dc0 con 0x7f6bb40ce840 2024-04-02T09:13:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.083+0000 7f6bbc800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb41470a0 con 0x7f6bb415dfe0 2024-04-02T09:13:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.084+0000 7f6bb9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb415a5c0 0x7f6bb41465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.084+0000 7f6bb9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb415a5c0 0x7f6bb41465e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54554/0 (socket says 172.21.15.89:54554) 2024-04-02T09:13:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.084+0000 7f6bb9400700 1 -- 172.21.15.89:0/1076497781 learned_addr learned my addr 172.21.15.89:0/1076497781 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.084+0000 7f6bb9e00700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bb40ce840 0x7f6bb4099520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.084+0000 7f6bba800700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb415dfe0 0x7f6bb4147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.084+0000 7f6bb9400700 1 -- 172.21.15.89:0/1076497781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb415dfe0 msgr2=0x7f6bb4147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.084+0000 7f6bb9400700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb415dfe0 0x7f6bb4147e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.084+0000 7f6bb9400700 1 -- 172.21.15.89:0/1076497781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bb40ce840 msgr2=0x7f6bb4099520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.084+0000 7f6bb9400700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bb40ce840 0x7f6bb4099520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.084+0000 7f6bb9400700 1 -- 172.21.15.89:0/1076497781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb0009a70 con 0x7f6bb415a5c0 2024-04-02T09:13:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.084+0000 7f6bb9e00700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bb40ce840 0x7f6bb4099520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.084+0000 7f6bb9400700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb415a5c0 0x7f6bb41465e0 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4001eb0 tx=0x7f6ba400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.084+0000 7f6bae000700 1 -- 172.21.15.89:0/1076497781 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4005d70 con 0x7f6bb415a5c0 2024-04-02T09:13:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.085+0000 7f6bae000700 1 -- 172.21.15.89:0/1076497781 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba40070d0 con 0x7f6bb415a5c0 2024-04-02T09:13:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.085+0000 7f6bae000700 1 -- 172.21.15.89:0/1076497781 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4007460 con 0x7f6bb415a5c0 2024-04-02T09:13:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.085+0000 7f6bbc800700 1 -- 172.21.15.89:0/1076497781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb4147380 con 0x7f6bb415a5c0 2024-04-02T09:13:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.085+0000 7f6bbc800700 1 -- 172.21.15.89:0/1076497781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb4147940 con 0x7f6bb415a5c0 2024-04-02T09:13:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.086+0000 7f6bbc800700 1 -- 172.21.15.89:0/1076497781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb406f360 con 0x7f6bb415a5c0 2024-04-02T09:13:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.087+0000 7f6bae000700 1 -- 172.21.15.89:0/1076497781 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f6ba40075c0 con 0x7f6bb415a5c0 2024-04-02T09:13:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.088+0000 7f6bae000700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6b9806aed0 0x7f6b9806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.088+0000 7f6bae000700 1 -- 172.21.15.89:0/1076497781 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f6ba4089050 con 0x7f6bb415a5c0 2024-04-02T09:13:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.088+0000 7f6bb9e00700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6b9806aed0 0x7f6b9806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.089+0000 7f6bb9e00700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6b9806aed0 0x7f6b9806d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8009c30 tx=0x7f6ba8008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:27.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.092+0000 7f6bae000700 1 -- 172.21.15.89:0/1076497781 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ba4053fd0 con 0x7f6bb415a5c0 2024-04-02T09:13:27.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.296+0000 7f6bbc800700 1 -- 172.21.15.89:0/1076497781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f6bb4066ff0 con 0x7f6bb415a5c0 2024-04-02T09:13:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.864+0000 7f6bae000700 1 -- 172.21.15.89:0/1076497781 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v242) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6ba40170d0 con 0x7f6bb415a5c0 2024-04-02T09:13:27.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.869+0000 7f6bbc800700 1 -- 172.21.15.89:0/1076497781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6b9806aed0 msgr2=0x7f6b9806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:27.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.869+0000 7f6bbc800700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6b9806aed0 0x7f6b9806d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8009c30 tx=0x7f6ba8008040 comp rx=0 tx=0).stop 2024-04-02T09:13:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.869+0000 7f6bbc800700 1 -- 172.21.15.89:0/1076497781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb415a5c0 msgr2=0x7f6bb41465e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.869+0000 7f6bbc800700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb415a5c0 0x7f6bb41465e0 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4001eb0 tx=0x7f6ba400c860 comp rx=0 tx=0).stop 2024-04-02T09:13:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.869+0000 7f6bbc800700 1 -- 172.21.15.89:0/1076497781 shutdown_connections 2024-04-02T09:13:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.869+0000 7f6bbc800700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6b9806aed0 0x7f6b9806d390 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8009c30 tx=0x7f6ba8008040 comp rx=0 tx=0).stop 2024-04-02T09:13:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.869+0000 7f6bbc800700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bb40ce840 0x7f6bb4099520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.869+0000 7f6bbc800700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6bb415dfe0 0x7f6bb4147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.869+0000 7f6bbc800700 1 --2- 172.21.15.89:0/1076497781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bb415a5c0 0x7f6bb41465e0 secure :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4001eb0 tx=0x7f6ba400c860 comp rx=0 tx=0).stop 2024-04-02T09:13:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.869+0000 7f6bbc800700 1 -- 172.21.15.89:0/1076497781 >> 172.21.15.89:0/1076497781 conn(0x7f6bb40b0d70 msgr2=0x7f6bb406e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:27.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.872+0000 7f6bbc800700 1 -- 172.21.15.89:0/1076497781 shutdown_connections 2024-04-02T09:13:27.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:27.873+0000 7f6bbc800700 1 -- 172.21.15.89:0/1076497781 wait complete. 2024-04-02T09:13:27.876 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T09:13:27.899 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T09:13:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.096+0000 7f5cbe400700 1 Processor -- start 2024-04-02T09:13:28.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.096+0000 7f5cbe400700 1 -- start start 2024-04-02T09:13:28.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.097+0000 7f5cbe400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5cb80d16a0 0x7f5cb80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.097+0000 7f5cbe400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5cb80d2370 0x7f5cb80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.097+0000 7f5cbe400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb8161420 con 0x7f5cb815d9d0 2024-04-02T09:13:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.097+0000 7f5cbe400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cb80d4340 con 0x7f5cb80d2370 2024-04-02T09:13:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.097+0000 7f5cbe400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cb815edc0 con 0x7f5cb80d16a0 2024-04-02T09:13:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.097+0000 7f5cb7400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5cb815d9d0 0x7f5cb80d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48476/0 (socket says 172.21.15.89:48476) 2024-04-02T09:13:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.097+0000 7f5cb7400700 1 -- 172.21.15.89:0/1847605013 learned_addr learned my addr 172.21.15.89:0/1847605013 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.097+0000 7f5cb7e00700 1 --2- 172.21.15.89:0/1847605013 >> v2:172.21.15.89:3300/0 conn(0x7f5cb80d2370 0x7f5cb80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.098+0000 7f5cb6a00700 1 --2- 172.21.15.89:0/1847605013 >> v2:172.21.15.113:3300/0 conn(0x7f5cb80d16a0 0x7f5cb80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.098+0000 7f5cb6000700 1 -- 172.21.15.89:0/1847605013 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416219481 0 0) 0x7f5cb8161420 con 0x7f5cb815d9d0 2024-04-02T09:13:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.098+0000 7f5cb6000700 1 -- 172.21.15.89:0/1847605013 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c9c0035b0 con 0x7f5cb815d9d0 2024-04-02T09:13:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.098+0000 7f5cb7e00700 1 -- 172.21.15.89:0/1847605013 >> v2:172.21.15.113:3300/0 conn(0x7f5cb80d16a0 msgr2=0x7f5cb80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.098+0000 7f5cb7e00700 1 --2- 172.21.15.89:0/1847605013 >> v2:172.21.15.113:3300/0 conn(0x7f5cb80d16a0 0x7f5cb80d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.098+0000 7f5cb7e00700 1 -- 172.21.15.89:0/1847605013 >> v1:172.21.15.89:6789/0 conn(0x7f5cb815d9d0 legacy=0x7f5cb80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:28.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.098+0000 7f5cb7e00700 1 -- 172.21.15.89:0/1847605013 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb8161700 con 0x7f5cb80d2370 2024-04-02T09:13:28.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.098+0000 7f5cb7e00700 1 --2- 172.21.15.89:0/1847605013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cb80d2370 0x7f5cb80ce360 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f5cb80d12f0 tx=0x7f5cb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247486072a44fbb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:28.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.098+0000 7f5cb6000700 1 -- 172.21.15.89:0/1847605013 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb000e070 con 0x7f5cb80d2370 2024-04-02T09:13:28.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.098+0000 7f5cb6000700 1 -- 172.21.15.89:0/1847605013 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb000bb20 con 0x7f5cb80d2370 2024-04-02T09:13:28.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.098+0000 7f5cb6000700 1 -- 172.21.15.89:0/1847605013 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb0012430 con 0x7f5cb80d2370 2024-04-02T09:13:28.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.099+0000 7f5cbe400700 1 -- 172.21.15.89:0/1847605013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cb80d2370 msgr2=0x7f5cb80ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:28.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.099+0000 7f5cbe400700 1 --2- 172.21.15.89:0/1847605013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cb80d2370 0x7f5cb80ce360 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f5cb80d12f0 tx=0x7f5cb000b330 comp rx=0 tx=0).stop 2024-04-02T09:13:28.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.099+0000 7f5cbe400700 1 -- 172.21.15.89:0/1847605013 shutdown_connections 2024-04-02T09:13:28.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.099+0000 7f5cbe400700 1 --2- 172.21.15.89:0/1847605013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cb80d2370 0x7f5cb80ce360 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.099+0000 7f5cbe400700 1 --2- 172.21.15.89:0/1847605013 >> v2:172.21.15.113:3300/0 conn(0x7f5cb80d16a0 0x7f5cb80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.099+0000 7f5cbe400700 1 -- 172.21.15.89:0/1847605013 >> 172.21.15.89:0/1847605013 conn(0x7f5cb808aff0 msgr2=0x7f5cb80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:28.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.099+0000 7f5cbe400700 1 -- 172.21.15.89:0/1847605013 shutdown_connections 2024-04-02T09:13:28.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.099+0000 7f5cbe400700 1 -- 172.21.15.89:0/1847605013 wait complete. 2024-04-02T09:13:28.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.101+0000 7f5cbe400700 1 Processor -- start 2024-04-02T09:13:28.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.101+0000 7f5cbe400700 1 -- start start 2024-04-02T09:13:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cbe400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cb80d16a0 0x7f5cb815cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cbe400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cb80d2370 0x7f5cb806b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cbe400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cb815d9d0 0x7f5cb806be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cbe400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80b70f0 con 0x7f5cb80d16a0 2024-04-02T09:13:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cbe400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb815d000 con 0x7f5cb80d2370 2024-04-02T09:13:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cbe400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80b64e0 con 0x7f5cb815d9d0 2024-04-02T09:13:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cb7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cb80d16a0 0x7f5cb815cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cb7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cb80d16a0 0x7f5cb815cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54570/0 (socket says 172.21.15.89:54570) 2024-04-02T09:13:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cb7400700 1 -- 172.21.15.89:0/3871792165 learned_addr learned my addr 172.21.15.89:0/3871792165 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cb6a00700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cb80d2370 0x7f5cb806b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cb7e00700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cb815d9d0 0x7f5cb806be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:28.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cb7400700 1 -- 172.21.15.89:0/3871792165 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cb815d9d0 msgr2=0x7f5cb806be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cb7400700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cb815d9d0 0x7f5cb806be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cb7400700 1 -- 172.21.15.89:0/3871792165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cb80d2370 msgr2=0x7f5cb806b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cb7400700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cb80d2370 0x7f5cb806b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.102+0000 7f5cb7400700 1 -- 172.21.15.89:0/3871792165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb0009a70 con 0x7f5cb80d16a0 2024-04-02T09:13:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.103+0000 7f5cb6a00700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cb80d2370 0x7f5cb806b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.103+0000 7f5cb7e00700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cb815d9d0 0x7f5cb806be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.103+0000 7f5cb7400700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cb80d16a0 0x7f5cb815cac0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8001eb0 tx=0x7f5ca800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.103+0000 7f5caf400700 1 -- 172.21.15.89:0/3871792165 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8005d70 con 0x7f5cb80d16a0 2024-04-02T09:13:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.103+0000 7f5caf400700 1 -- 172.21.15.89:0/3871792165 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca80070d0 con 0x7f5cb80d16a0 2024-04-02T09:13:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.103+0000 7f5cbe400700 1 -- 172.21.15.89:0/3871792165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb80b67c0 con 0x7f5cb80d16a0 2024-04-02T09:13:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.103+0000 7f5caf400700 1 -- 172.21.15.89:0/3871792165 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8007460 con 0x7f5cb80d16a0 2024-04-02T09:13:28.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.103+0000 7f5cbe400700 1 -- 172.21.15.89:0/3871792165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb80b6d80 con 0x7f5cb80d16a0 2024-04-02T09:13:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.104+0000 7f5cbe400700 1 -- 172.21.15.89:0/3871792165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb80d1ca0 con 0x7f5cb80d16a0 2024-04-02T09:13:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.105+0000 7f5caf400700 1 -- 172.21.15.89:0/3871792165 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5ca80075c0 con 0x7f5cb80d16a0 2024-04-02T09:13:28.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.106+0000 7f5caf400700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c9806aed0 0x7f5c9806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:28.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.106+0000 7f5caf400700 1 -- 172.21.15.89:0/3871792165 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5ca8089160 con 0x7f5cb80d16a0 2024-04-02T09:13:28.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.106+0000 7f5cb6a00700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c9806aed0 0x7f5c9806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:28.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.107+0000 7f5cb6a00700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c9806aed0 0x7f5c9806d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5cb80b5580 tx=0x7f5ca4008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:28.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.111+0000 7f5caf400700 1 -- 172.21.15.89:0/3871792165 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5ca8053f60 con 0x7f5cb80d16a0 2024-04-02T09:13:28.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.355+0000 7f5cbe400700 1 -- 172.21.15.89:0/3871792165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f5cb815d200 con 0x7f5cb80d16a0 2024-04-02T09:13:28.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.364+0000 7f5caf400700 1 -- 172.21.15.89:0/3871792165 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 30 and data pool 31 v116) v1 ==== 174+0+0 (secure 0 0 0) 0x7f5ca8057580 con 0x7f5cb80d16a0 2024-04-02T09:13:28.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.368+0000 7f5cbe400700 1 -- 172.21.15.89:0/3871792165 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c9806aed0 msgr2=0x7f5c9806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:28.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.368+0000 7f5cbe400700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c9806aed0 0x7f5c9806d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5cb80b5580 tx=0x7f5ca4008040 comp rx=0 tx=0).stop 2024-04-02T09:13:28.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.368+0000 7f5cbe400700 1 -- 172.21.15.89:0/3871792165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cb80d16a0 msgr2=0x7f5cb815cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:28.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.368+0000 7f5cbe400700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cb80d16a0 0x7f5cb815cac0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8001eb0 tx=0x7f5ca800c860 comp rx=0 tx=0).stop 2024-04-02T09:13:28.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.369+0000 7f5cbe400700 1 -- 172.21.15.89:0/3871792165 shutdown_connections 2024-04-02T09:13:28.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.369+0000 7f5cbe400700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5cb815d9d0 0x7f5cb806be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.369+0000 7f5cbe400700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cb80d16a0 0x7f5cb815cac0 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.369+0000 7f5cbe400700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c9806aed0 0x7f5c9806d390 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.369+0000 7f5cbe400700 1 --2- 172.21.15.89:0/3871792165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cb80d2370 0x7f5cb806b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.369+0000 7f5cbe400700 1 -- 172.21.15.89:0/3871792165 >> 172.21.15.89:0/3871792165 conn(0x7f5cb808aff0 msgr2=0x7f5cb80b1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:28.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.369+0000 7f5cbe400700 1 -- 172.21.15.89:0/3871792165 shutdown_connections 2024-04-02T09:13:28.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.369+0000 7f5cbe400700 1 -- 172.21.15.89:0/3871792165 wait complete. 2024-04-02T09:13:28.372 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 30 and data pool 31 2024-04-02T09:13:28.398 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:13:28.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.591+0000 7f2ef6800700 1 Processor -- start 2024-04-02T09:13:28.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.591+0000 7f2ef6800700 1 -- start start 2024-04-02T09:13:28.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.592+0000 7f2ef6800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef00cb3f0 0x7f2ef00ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:28.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.592+0000 7f2ef6800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef015fd50 con 0x7f2ef00ac600 2024-04-02T09:13:28.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.592+0000 7f2ef6800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef015f140 con 0x7f2ef015b5d0 2024-04-02T09:13:28.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.592+0000 7f2ef6800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef015b9b0 con 0x7f2ef00cb3f0 2024-04-02T09:13:28.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.592+0000 7f2eefe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2ef00ac600 0x7f2ef006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48486/0 (socket says 172.21.15.89:48486) 2024-04-02T09:13:28.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.592+0000 7f2eefe00700 1 -- 172.21.15.89:0/3266680300 learned_addr learned my addr 172.21.15.89:0/3266680300 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:28.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.593+0000 7f2eeea00700 1 --2- 172.21.15.89:0/3266680300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef00cb3f0 0x7f2ef00ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:28.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.593+0000 7f2eee000700 1 -- 172.21.15.89:0/3266680300 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474042003 0 0) 0x7f2ef015fd50 con 0x7f2ef00ac600 2024-04-02T09:13:28.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.593+0000 7f2eee000700 1 -- 172.21.15.89:0/3266680300 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed40035b0 con 0x7f2ef00ac600 2024-04-02T09:13:28.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.593+0000 7f2eee000700 1 -- 172.21.15.89:0/3266680300 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237324397 0 0) 0x7f2ef015f140 con 0x7f2ef015b5d0 2024-04-02T09:13:28.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.593+0000 7f2eee000700 1 -- 172.21.15.89:0/3266680300 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef015fd50 con 0x7f2ef015b5d0 2024-04-02T09:13:28.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.593+0000 7f2eeea00700 1 -- 172.21.15.89:0/3266680300 >> v1:172.21.15.113:6789/0 conn(0x7f2ef015b5d0 legacy=0x7f2ef00ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:28.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.593+0000 7f2eeea00700 1 -- 172.21.15.89:0/3266680300 >> v1:172.21.15.89:6789/0 conn(0x7f2ef00ac600 legacy=0x7f2ef006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:28.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.593+0000 7f2eee000700 1 -- 172.21.15.89:0/3266680300 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 921526990 0 0) 0x7f2ed40035b0 con 0x7f2ef00ac600 2024-04-02T09:13:28.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.593+0000 7f2eeea00700 1 -- 172.21.15.89:0/3266680300 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef015f420 con 0x7f2ef00cb3f0 2024-04-02T09:13:28.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.593+0000 7f2eeea00700 1 --2- 172.21.15.89:0/3266680300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef00cb3f0 0x7f2ef00ac0c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2edc009870 tx=0x7f2edc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e92c19fdf8103ebc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:28.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.594+0000 7f2eee000700 1 -- 172.21.15.89:0/3266680300 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc011830 con 0x7f2ef00cb3f0 2024-04-02T09:13:28.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.594+0000 7f2eee000700 1 -- 172.21.15.89:0/3266680300 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2edc011990 con 0x7f2ef00cb3f0 2024-04-02T09:13:28.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.594+0000 7f2eee000700 1 -- 172.21.15.89:0/3266680300 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc011d20 con 0x7f2ef00cb3f0 2024-04-02T09:13:28.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.594+0000 7f2ef6800700 1 -- 172.21.15.89:0/3266680300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef00cb3f0 msgr2=0x7f2ef00ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:28.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.594+0000 7f2ef6800700 1 --2- 172.21.15.89:0/3266680300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef00cb3f0 0x7f2ef00ac0c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2edc009870 tx=0x7f2edc009b80 comp rx=0 tx=0).stop 2024-04-02T09:13:28.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.594+0000 7f2ef6800700 1 -- 172.21.15.89:0/3266680300 shutdown_connections 2024-04-02T09:13:28.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.594+0000 7f2ef6800700 1 --2- 172.21.15.89:0/3266680300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef00cb3f0 0x7f2ef00ac0c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.594+0000 7f2ef6800700 1 -- 172.21.15.89:0/3266680300 >> 172.21.15.89:0/3266680300 conn(0x7f2ef00cf190 msgr2=0x7f2ef00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:28.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.594+0000 7f2ef6800700 1 -- 172.21.15.89:0/3266680300 shutdown_connections 2024-04-02T09:13:28.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.594+0000 7f2ef6800700 1 -- 172.21.15.89:0/3266680300 wait complete. 2024-04-02T09:13:28.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.596+0000 7f2ef6800700 1 Processor -- start 2024-04-02T09:13:28.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.596+0000 7f2ef6800700 1 -- start start 2024-04-02T09:13:28.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.597+0000 7f2ef6800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ef00ac600 0x7f2ef0144560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:28.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.597+0000 7f2ef6800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef015b5d0 0x7f2ef015a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:28.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.597+0000 7f2ef6800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef01f1150 0x7f2ef015a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:28.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.597+0000 7f2ef6800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef015ac80 con 0x7f2ef00ac600 2024-04-02T09:13:28.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.597+0000 7f2ef6800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef00cd680 con 0x7f2ef015b5d0 2024-04-02T09:13:28.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.597+0000 7f2ef6800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef00ccd50 con 0x7f2ef01f1150 2024-04-02T09:13:28.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.597+0000 7f2eef400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ef00ac600 0x7f2ef0144560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:28.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.597+0000 7f2eef400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ef00ac600 0x7f2ef0144560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54576/0 (socket says 172.21.15.89:54576) 2024-04-02T09:13:28.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.597+0000 7f2eef400700 1 -- 172.21.15.89:0/2696354637 learned_addr learned my addr 172.21.15.89:0/2696354637 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:28.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.597+0000 7f2eefe00700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef01f1150 0x7f2ef015a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:28.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.597+0000 7f2eeea00700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef015b5d0 0x7f2ef015a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:28.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.598+0000 7f2eef400700 1 -- 172.21.15.89:0/2696354637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef01f1150 msgr2=0x7f2ef015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:28.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.598+0000 7f2eef400700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef01f1150 0x7f2ef015a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.598+0000 7f2eef400700 1 -- 172.21.15.89:0/2696354637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef015b5d0 msgr2=0x7f2ef015a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.598+0000 7f2eef400700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef015b5d0 0x7f2ef015a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.598+0000 7f2eef400700 1 -- 172.21.15.89:0/2696354637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc0094a0 con 0x7f2ef00ac600 2024-04-02T09:13:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.598+0000 7f2eefe00700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef01f1150 0x7f2ef015a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.598+0000 7f2eeea00700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef015b5d0 0x7f2ef015a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.598+0000 7f2eef400700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ef00ac600 0x7f2ef0144560 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f2ee000c550 tx=0x7f2ee000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.598+0000 7f2ee7400700 1 -- 172.21.15.89:0/2696354637 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee00052d0 con 0x7f2ef00ac600 2024-04-02T09:13:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.598+0000 7f2ee7400700 1 -- 172.21.15.89:0/2696354637 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee0005430 con 0x7f2ef00ac600 2024-04-02T09:13:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.598+0000 7f2ef6800700 1 -- 172.21.15.89:0/2696354637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef00cd030 con 0x7f2ef00ac600 2024-04-02T09:13:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.598+0000 7f2ee7400700 1 -- 172.21.15.89:0/2696354637 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee00038c0 con 0x7f2ef00ac600 2024-04-02T09:13:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.598+0000 7f2ef6800700 1 -- 172.21.15.89:0/2696354637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef0158aa0 con 0x7f2ef00ac600 2024-04-02T09:13:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.599+0000 7f2ef6800700 1 -- 172.21.15.89:0/2696354637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef0067720 con 0x7f2ef00ac600 2024-04-02T09:13:28.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.600+0000 7f2ee7400700 1 -- 172.21.15.89:0/2696354637 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f2ee0003a20 con 0x7f2ef00ac600 2024-04-02T09:13:28.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.601+0000 7f2ee7400700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ed006af00 0x7f2ed006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:28.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.601+0000 7f2ee7400700 1 -- 172.21.15.89:0/2696354637 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f2ee0088dd0 con 0x7f2ef00ac600 2024-04-02T09:13:28.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.601+0000 7f2eeea00700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ed006af00 0x7f2ed006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:28.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.602+0000 7f2eeea00700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ed006af00 0x7f2ed006d3c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0097c0 tx=0x7f2edc01c040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:28.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.606+0000 7f2ee7400700 1 -- 172.21.15.89:0/2696354637 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2ee0053ad0 con 0x7f2ef00ac600 2024-04-02T09:13:28.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.808+0000 7f2ef6800700 1 -- 172.21.15.89:0/2696354637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ef006ea60 con 0x7f2ef00ac600 2024-04-02T09:13:28.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.809+0000 7f2ee7400700 1 -- 172.21.15.89:0/2696354637 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v243) v1 ==== 74+0+16749 (secure 0 0 0) 0x7f2ee008a090 con 0x7f2ef00ac600 2024-04-02T09:13:28.809 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:28.809 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":243,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:13:28.357646+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:13:26.654647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-02T09:13:27.342285+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:13:28.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.811+0000 7f2ef6800700 1 -- 172.21.15.89:0/2696354637 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ed006af00 msgr2=0x7f2ed006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:28.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.811+0000 7f2ef6800700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ed006af00 0x7f2ed006d3c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0097c0 tx=0x7f2edc01c040 comp rx=0 tx=0).stop 2024-04-02T09:13:28.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.811+0000 7f2ef6800700 1 -- 172.21.15.89:0/2696354637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ef00ac600 msgr2=0x7f2ef0144560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:28.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.811+0000 7f2ef6800700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ef00ac600 0x7f2ef0144560 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f2ee000c550 tx=0x7f2ee000c910 comp rx=0 tx=0).stop 2024-04-02T09:13:28.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.811+0000 7f2ef6800700 1 -- 172.21.15.89:0/2696354637 shutdown_connections 2024-04-02T09:13:28.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.811+0000 7f2ef6800700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef01f1150 0x7f2ef015a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.811+0000 7f2ef6800700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ef00ac600 0x7f2ef0144560 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.811+0000 7f2ef6800700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ed006af00 0x7f2ed006d3c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.811+0000 7f2ef6800700 1 --2- 172.21.15.89:0/2696354637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef015b5d0 0x7f2ef015a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:28.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.811+0000 7f2ef6800700 1 -- 172.21.15.89:0/2696354637 >> 172.21.15.89:0/2696354637 conn(0x7f2ef00cf190 msgr2=0x7f2ef015bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:28.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.812+0000 7f2ef6800700 1 -- 172.21.15.89:0/2696354637 shutdown_connections 2024-04-02T09:13:28.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:28.812+0000 7f2ef6800700 1 -- 172.21.15.89:0/2696354637 wait complete. 2024-04-02T09:13:28.830 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:13:29.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.022+0000 7f0448000700 1 Processor -- start 2024-04-02T09:13:29.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.022+0000 7f0448000700 1 -- start start 2024-04-02T09:13:29.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.023+0000 7f0448000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f044015d640 0x7f04400d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:29.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.023+0000 7f0448000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f04401619d0 0x7f04400b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:29.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.023+0000 7f0448000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04400d30d0 con 0x7f04400ceec0 2024-04-02T09:13:29.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.023+0000 7f0448000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04400b6840 con 0x7f04401619d0 2024-04-02T09:13:29.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.023+0000 7f0448000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04400d2880 con 0x7f044015d640 2024-04-02T09:13:29.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.023+0000 7f0446000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f04401619d0 0x7f04400b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:29.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.024+0000 7f0446000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f04401619d0 0x7f04400b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60640/0 (socket says 172.21.15.89:60640) 2024-04-02T09:13:29.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.024+0000 7f0446000700 1 -- 172.21.15.89:0/767094421 learned_addr learned my addr 172.21.15.89:0/767094421 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:29.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.024+0000 7f0445600700 1 --2- 172.21.15.89:0/767094421 >> v2:172.21.15.113:3300/0 conn(0x7f044015d640 0x7f04400d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:29.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.024+0000 7f0446000700 1 -- 172.21.15.89:0/767094421 >> v2:172.21.15.113:3300/0 conn(0x7f044015d640 msgr2=0x7f04400d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:29.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.024+0000 7f0446000700 1 --2- 172.21.15.89:0/767094421 >> v2:172.21.15.113:3300/0 conn(0x7f044015d640 0x7f04400d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.024+0000 7f0446000700 1 -- 172.21.15.89:0/767094421 >> v1:172.21.15.113:6789/0 conn(0x7f04400ceec0 legacy=0x7f04401612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:29.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.024+0000 7f0446000700 1 -- 172.21.15.89:0/767094421 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04400d2b30 con 0x7f04401619d0 2024-04-02T09:13:29.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.024+0000 7f0446000700 1 --2- 172.21.15.89:0/767094421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04401619d0 0x7f04400b6cd0 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f04400b5c20 tx=0x7f043800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e029ebc8409cc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:29.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.024+0000 7f043fe00700 1 -- 172.21.15.89:0/767094421 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043800e070 con 0x7f04401619d0 2024-04-02T09:13:29.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.024+0000 7f043fe00700 1 -- 172.21.15.89:0/767094421 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043800b9c0 con 0x7f04401619d0 2024-04-02T09:13:29.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.024+0000 7f043fe00700 1 -- 172.21.15.89:0/767094421 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043800bd70 con 0x7f04401619d0 2024-04-02T09:13:29.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.025+0000 7f0448000700 1 -- 172.21.15.89:0/767094421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04401619d0 msgr2=0x7f04400b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:29.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.025+0000 7f0448000700 1 --2- 172.21.15.89:0/767094421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04401619d0 0x7f04400b6cd0 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f04400b5c20 tx=0x7f043800b330 comp rx=0 tx=0).stop 2024-04-02T09:13:29.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.025+0000 7f0448000700 1 -- 172.21.15.89:0/767094421 shutdown_connections 2024-04-02T09:13:29.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.025+0000 7f0448000700 1 --2- 172.21.15.89:0/767094421 >> v2:172.21.15.113:3300/0 conn(0x7f044015d640 0x7f04400d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.025+0000 7f0448000700 1 --2- 172.21.15.89:0/767094421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04401619d0 0x7f04400b6cd0 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.025+0000 7f0448000700 1 -- 172.21.15.89:0/767094421 >> 172.21.15.89:0/767094421 conn(0x7f044008aff0 msgr2=0x7f04400d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:29.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.025+0000 7f0448000700 1 -- 172.21.15.89:0/767094421 shutdown_connections 2024-04-02T09:13:29.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.025+0000 7f0448000700 1 -- 172.21.15.89:0/767094421 wait complete. 2024-04-02T09:13:29.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.027+0000 7f0448000700 1 Processor -- start 2024-04-02T09:13:29.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.027+0000 7f0448000700 1 -- start start 2024-04-02T09:13:29.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.028+0000 7f0448000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04400ceec0 0x7f044009d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:29.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.028+0000 7f0448000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f044015d640 0x7f04400c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:29.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.028+0000 7f0448000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04401619d0 0x7f04400c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:29.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.028+0000 7f0448000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044014c820 con 0x7f04401619d0 2024-04-02T09:13:29.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.028+0000 7f0448000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04400a0690 con 0x7f044015d640 2024-04-02T09:13:29.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.028+0000 7f0448000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044009d770 con 0x7f04400ceec0 2024-04-02T09:13:29.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.028+0000 7f0446000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04401619d0 0x7f04400c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:29.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.028+0000 7f0446000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04401619d0 0x7f04400c9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60642/0 (socket says 172.21.15.89:60642) 2024-04-02T09:13:29.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.028+0000 7f0446000700 1 -- 172.21.15.89:0/3484306867 learned_addr learned my addr 172.21.15.89:0/3484306867 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:29.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.028+0000 7f0445600700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04400ceec0 0x7f044009d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:29.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.029+0000 7f0444c00700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f044015d640 0x7f04400c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:29.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.029+0000 7f0446000700 1 -- 172.21.15.89:0/3484306867 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04400ceec0 msgr2=0x7f044009d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:29.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.029+0000 7f0446000700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04400ceec0 0x7f044009d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.029+0000 7f0446000700 1 -- 172.21.15.89:0/3484306867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f044015d640 msgr2=0x7f04400c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:29.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.029+0000 7f0446000700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f044015d640 0x7f04400c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.029+0000 7f0446000700 1 -- 172.21.15.89:0/3484306867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0438009b30 con 0x7f04401619d0 2024-04-02T09:13:29.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.029+0000 7f0444c00700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f044015d640 0x7f04400c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:29.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.029+0000 7f0445600700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04400ceec0 0x7f044009d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:29.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.029+0000 7f0446000700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04401619d0 0x7f04400c9790 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f044008c490 tx=0x7f0438004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:29.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.029+0000 7f043d600700 1 -- 172.21.15.89:0/3484306867 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04380092e0 con 0x7f04401619d0 2024-04-02T09:13:29.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.029+0000 7f043d600700 1 -- 172.21.15.89:0/3484306867 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043801c440 con 0x7f04401619d0 2024-04-02T09:13:29.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.029+0000 7f0448000700 1 -- 172.21.15.89:0/3484306867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044014ca20 con 0x7f04401619d0 2024-04-02T09:13:29.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.029+0000 7f043d600700 1 -- 172.21.15.89:0/3484306867 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043801c7f0 con 0x7f04401619d0 2024-04-02T09:13:29.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.029+0000 7f0448000700 1 -- 172.21.15.89:0/3484306867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f044014cfe0 con 0x7f04401619d0 2024-04-02T09:13:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.030+0000 7f0448000700 1 -- 172.21.15.89:0/3484306867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0440067720 con 0x7f04401619d0 2024-04-02T09:13:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.031+0000 7f043d600700 1 -- 172.21.15.89:0/3484306867 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f043801c950 con 0x7f04401619d0 2024-04-02T09:13:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.032+0000 7f043d600700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f042006aed0 0x7f042006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.032+0000 7f043d600700 1 -- 172.21.15.89:0/3484306867 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f043808b440 con 0x7f04401619d0 2024-04-02T09:13:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.032+0000 7f0445600700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f042006aed0 0x7f042006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.033+0000 7f0445600700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f042006aed0 0x7f042006d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0434007a50 tx=0x7f0434008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:29.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.037+0000 7f043d600700 1 -- 172.21.15.89:0/3484306867 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0438055fe0 con 0x7f04401619d0 2024-04-02T09:13:29.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.241+0000 7f0448000700 1 -- 172.21.15.89:0/3484306867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04400c6e30 con 0x7f04401619d0 2024-04-02T09:13:29.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.243+0000 7f043d600700 1 -- 172.21.15.89:0/3484306867 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v243) v1 ==== 74+0+16749 (secure 0 0 0) 0x7f0438059600 con 0x7f04401619d0 2024-04-02T09:13:29.242 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:29.242 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":243,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:13:28.357646+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:13:26.654647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-02T09:13:27.342285+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:13:29.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.245+0000 7f0448000700 1 -- 172.21.15.89:0/3484306867 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f042006aed0 msgr2=0x7f042006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:29.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.245+0000 7f0448000700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f042006aed0 0x7f042006d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0434007a50 tx=0x7f0434008040 comp rx=0 tx=0).stop 2024-04-02T09:13:29.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.245+0000 7f0448000700 1 -- 172.21.15.89:0/3484306867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04401619d0 msgr2=0x7f04400c9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:29.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.245+0000 7f0448000700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04401619d0 0x7f04400c9790 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f044008c490 tx=0x7f0438004ad0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.245+0000 7f0448000700 1 -- 172.21.15.89:0/3484306867 shutdown_connections 2024-04-02T09:13:29.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.245+0000 7f0448000700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04401619d0 0x7f04400c9790 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.245+0000 7f0448000700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f04400ceec0 0x7f044009d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.245+0000 7f0448000700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f042006aed0 0x7f042006d390 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.245+0000 7f0448000700 1 --2- 172.21.15.89:0/3484306867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f044015d640 0x7f04400c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.245+0000 7f0448000700 1 -- 172.21.15.89:0/3484306867 >> 172.21.15.89:0/3484306867 conn(0x7f044008aff0 msgr2=0x7f044015bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:29.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.246+0000 7f0448000700 1 -- 172.21.15.89:0/3484306867 shutdown_connections 2024-04-02T09:13:29.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.246+0000 7f0448000700 1 -- 172.21.15.89:0/3484306867 wait complete. 2024-04-02T09:13:29.263 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T09:13:29.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.461+0000 7ffb2e600700 1 Processor -- start 2024-04-02T09:13:29.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.462+0000 7ffb2e600700 1 -- start start 2024-04-02T09:13:29.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb2e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb2815d9d0 0x7ffb280d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:29.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb2e600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb280d16a0 0x7ffb280d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:29.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb2e600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ffb280d2370 0x7ffb2815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:29.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb2e600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb280d2820 con 0x7ffb280d16a0 2024-04-02T09:13:29.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb2e600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb28161420 con 0x7ffb280d2370 2024-04-02T09:13:29.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb2e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb28161700 con 0x7ffb2815d9d0 2024-04-02T09:13:29.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb1fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb280d16a0 0x7ffb280d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:29.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb1fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffb280d16a0 0x7ffb280d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60654/0 (socket says 172.21.15.89:60654) 2024-04-02T09:13:29.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb1fe00700 1 -- 172.21.15.89:0/2784829261 learned_addr learned my addr 172.21.15.89:0/2784829261 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:29.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb27400700 1 --2- 172.21.15.89:0/2784829261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb2815d9d0 0x7ffb280d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:29.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb27e00700 1 --2- 172.21.15.89:0/2784829261 >> v2:172.21.15.113:3300/0 conn(0x7ffb280d2370 0x7ffb2815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:29.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb1fe00700 1 -- 172.21.15.89:0/2784829261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb2815d9d0 msgr2=0x7ffb280d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:29.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb1fe00700 1 --2- 172.21.15.89:0/2784829261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb2815d9d0 0x7ffb280d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb1fe00700 1 -- 172.21.15.89:0/2784829261 >> v2:172.21.15.113:3300/0 conn(0x7ffb280d2370 msgr2=0x7ffb2815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:29.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb1fe00700 1 --2- 172.21.15.89:0/2784829261 >> v2:172.21.15.113:3300/0 conn(0x7ffb280d2370 0x7ffb2815ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.463+0000 7ffb1fe00700 1 -- 172.21.15.89:0/2784829261 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb281619e0 con 0x7ffb280d16a0 2024-04-02T09:13:29.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.464+0000 7ffb1fe00700 1 --2- 172.21.15.89:0/2784829261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb280d16a0 0x7ffb280d1e30 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7ffb280cdc00 tx=0x7ffb10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c683a6a14055e9a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:29.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.464+0000 7ffb26a00700 1 -- 172.21.15.89:0/2784829261 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10005e30 con 0x7ffb280d16a0 2024-04-02T09:13:29.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.464+0000 7ffb26a00700 1 -- 172.21.15.89:0/2784829261 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb10004030 con 0x7ffb280d16a0 2024-04-02T09:13:29.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.464+0000 7ffb26a00700 1 -- 172.21.15.89:0/2784829261 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb100043e0 con 0x7ffb280d16a0 2024-04-02T09:13:29.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.464+0000 7ffb2e600700 1 -- 172.21.15.89:0/2784829261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb280d16a0 msgr2=0x7ffb280d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:29.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.464+0000 7ffb2e600700 1 --2- 172.21.15.89:0/2784829261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb280d16a0 0x7ffb280d1e30 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7ffb280cdc00 tx=0x7ffb10009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.464+0000 7ffb2e600700 1 -- 172.21.15.89:0/2784829261 shutdown_connections 2024-04-02T09:13:29.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.464+0000 7ffb2e600700 1 --2- 172.21.15.89:0/2784829261 >> v2:172.21.15.113:3300/0 conn(0x7ffb280d2370 0x7ffb2815ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.464+0000 7ffb2e600700 1 --2- 172.21.15.89:0/2784829261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb2815d9d0 0x7ffb280d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.464+0000 7ffb2e600700 1 --2- 172.21.15.89:0/2784829261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb280d16a0 0x7ffb280d1e30 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.464+0000 7ffb2e600700 1 -- 172.21.15.89:0/2784829261 >> 172.21.15.89:0/2784829261 conn(0x7ffb2808aff0 msgr2=0x7ffb280d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:29.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.465+0000 7ffb2e600700 1 -- 172.21.15.89:0/2784829261 shutdown_connections 2024-04-02T09:13:29.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.465+0000 7ffb2e600700 1 -- 172.21.15.89:0/2784829261 wait complete. 2024-04-02T09:13:29.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.466+0000 7ffb2e600700 1 Processor -- start 2024-04-02T09:13:29.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.466+0000 7ffb2e600700 1 -- start start 2024-04-02T09:13:29.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.467+0000 7ffb2e600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb280d16a0 0x7ffb281e9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:29.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.467+0000 7ffb2e600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb280d2370 0x7ffb280cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:29.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.467+0000 7ffb2e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb2815d9d0 0x7ffb280b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:29.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.467+0000 7ffb2e600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb280ccef0 con 0x7ffb280d2370 2024-04-02T09:13:29.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.467+0000 7ffb2e600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb280ca5f0 con 0x7ffb280d16a0 2024-04-02T09:13:29.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.467+0000 7ffb2e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb280ca840 con 0x7ffb2815d9d0 2024-04-02T09:13:29.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.468+0000 7ffb1fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb280d2370 0x7ffb280cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:29.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.468+0000 7ffb1fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb280d2370 0x7ffb280cc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60656/0 (socket says 172.21.15.89:60656) 2024-04-02T09:13:29.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.468+0000 7ffb1fe00700 1 -- 172.21.15.89:0/3612027655 learned_addr learned my addr 172.21.15.89:0/3612027655 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:29.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.468+0000 7ffb27400700 1 --2- 172.21.15.89:0/3612027655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb280d16a0 0x7ffb281e9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:29.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.468+0000 7ffb27e00700 1 --2- 172.21.15.89:0/3612027655 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb2815d9d0 0x7ffb280b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:29.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.468+0000 7ffb1fe00700 1 -- 172.21.15.89:0/3612027655 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb2815d9d0 msgr2=0x7ffb280b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:29.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.468+0000 7ffb1fe00700 1 --2- 172.21.15.89:0/3612027655 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb2815d9d0 0x7ffb280b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.468+0000 7ffb1fe00700 1 -- 172.21.15.89:0/3612027655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb280d16a0 msgr2=0x7ffb281e9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:29.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.468+0000 7ffb1fe00700 1 --2- 172.21.15.89:0/3612027655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb280d16a0 0x7ffb281e9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:29.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.468+0000 7ffb1fe00700 1 -- 172.21.15.89:0/3612027655 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb10009500 con 0x7ffb280d2370 2024-04-02T09:13:29.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.468+0000 7ffb1fe00700 1 --2- 172.21.15.89:0/3612027655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb280d2370 0x7ffb280cc9b0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009c10 tx=0x7ffb10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:29.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.468+0000 7ffb1f400700 1 -- 172.21.15.89:0/3612027655 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10003c70 con 0x7ffb280d2370 2024-04-02T09:13:29.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.469+0000 7ffb1f400700 1 -- 172.21.15.89:0/3612027655 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb10003dd0 con 0x7ffb280d2370 2024-04-02T09:13:29.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.469+0000 7ffb2e600700 1 -- 172.21.15.89:0/3612027655 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb280cab20 con 0x7ffb280d2370 2024-04-02T09:13:29.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.469+0000 7ffb1f400700 1 -- 172.21.15.89:0/3612027655 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10011660 con 0x7ffb280d2370 2024-04-02T09:13:29.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.469+0000 7ffb2e600700 1 -- 172.21.15.89:0/3612027655 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb280cb0e0 con 0x7ffb280d2370 2024-04-02T09:13:29.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.470+0000 7ffb2e600700 1 -- 172.21.15.89:0/3612027655 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb280d1ca0 con 0x7ffb280d2370 2024-04-02T09:13:29.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.471+0000 7ffb1f400700 1 -- 172.21.15.89:0/3612027655 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7ffb100117c0 con 0x7ffb280d2370 2024-04-02T09:13:29.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.471+0000 7ffb1f400700 1 --2- 172.21.15.89:0/3612027655 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffb0406aed0 0x7ffb0406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:29.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.471+0000 7ffb1f400700 1 -- 172.21.15.89:0/3612027655 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7ffb1008a9d0 con 0x7ffb280d2370 2024-04-02T09:13:29.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.472+0000 7ffb27400700 1 --2- 172.21.15.89:0/3612027655 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffb0406aed0 0x7ffb0406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:29.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.472+0000 7ffb27400700 1 --2- 172.21.15.89:0/3612027655 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffb0406aed0 0x7ffb0406d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffb18006b20 tx=0x7ffb18008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:29.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.476+0000 7ffb1f400700 1 -- 172.21.15.89:0/3612027655 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffb10055720 con 0x7ffb280d2370 2024-04-02T09:13:29.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:29.724+0000 7ffb2e600700 1 -- 172.21.15.89:0/3612027655 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ffb28067720 con 0x7ffb280d2370 2024-04-02T09:13:30.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.368+0000 7ffb1f400700 1 -- 172.21.15.89:0/3612027655 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v118) v1 ==== 118+0+0 (secure 0 0 0) 0x7ffb100160b0 con 0x7ffb280d2370 2024-04-02T09:13:30.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.371+0000 7ffb2e600700 1 -- 172.21.15.89:0/3612027655 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffb0406aed0 msgr2=0x7ffb0406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:30.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.371+0000 7ffb2e600700 1 --2- 172.21.15.89:0/3612027655 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffb0406aed0 0x7ffb0406d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffb18006b20 tx=0x7ffb18008040 comp rx=0 tx=0).stop 2024-04-02T09:13:30.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.371+0000 7ffb2e600700 1 -- 172.21.15.89:0/3612027655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb280d2370 msgr2=0x7ffb280cc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:30.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.371+0000 7ffb2e600700 1 --2- 172.21.15.89:0/3612027655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb280d2370 0x7ffb280cc9b0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009c10 tx=0x7ffb10004750 comp rx=0 tx=0).stop 2024-04-02T09:13:30.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.371+0000 7ffb2e600700 1 -- 172.21.15.89:0/3612027655 shutdown_connections 2024-04-02T09:13:30.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.371+0000 7ffb2e600700 1 --2- 172.21.15.89:0/3612027655 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffb0406aed0 0x7ffb0406d390 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:30.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.371+0000 7ffb2e600700 1 --2- 172.21.15.89:0/3612027655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb280d16a0 0x7ffb281e9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:30.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.371+0000 7ffb2e600700 1 --2- 172.21.15.89:0/3612027655 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffb2815d9d0 0x7ffb280b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:30.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.371+0000 7ffb2e600700 1 --2- 172.21.15.89:0/3612027655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb280d2370 0x7ffb280cc9b0 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:30.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.371+0000 7ffb2e600700 1 -- 172.21.15.89:0/3612027655 >> 172.21.15.89:0/3612027655 conn(0x7ffb2808aff0 msgr2=0x7ffb281608b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:30.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.371+0000 7ffb2e600700 1 -- 172.21.15.89:0/3612027655 shutdown_connections 2024-04-02T09:13:30.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.371+0000 7ffb2e600700 1 -- 172.21.15.89:0/3612027655 wait complete. 2024-04-02T09:13:30.390 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T09:13:30.391 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T09:13:30.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.589+0000 7f302de00700 1 Processor -- start 2024-04-02T09:13:30.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.590+0000 7f302de00700 1 -- start start 2024-04-02T09:13:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.591+0000 7f302de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f302815d9d0 0x7f30280d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.591+0000 7f302de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f30280d16a0 0x7f30280d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.591+0000 7f302de00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3028161420 con 0x7f30280d2370 2024-04-02T09:13:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.591+0000 7f302de00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30280d2820 con 0x7f30280d16a0 2024-04-02T09:13:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.591+0000 7f302de00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3028161700 con 0x7f302815d9d0 2024-04-02T09:13:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.591+0000 7f3026000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f30280d16a0 0x7f30280d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.591+0000 7f3026000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f30280d16a0 0x7f30280d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60670/0 (socket says 172.21.15.89:60670) 2024-04-02T09:13:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.591+0000 7f3026000700 1 -- 172.21.15.89:0/1025293105 learned_addr learned my addr 172.21.15.89:0/1025293105 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.591+0000 7f3026a00700 1 --2- 172.21.15.89:0/1025293105 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f302815d9d0 0x7f30280d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.591+0000 7f3026000700 1 -- 172.21.15.89:0/1025293105 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f302815d9d0 msgr2=0x7f30280d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.591+0000 7f3026000700 1 --2- 172.21.15.89:0/1025293105 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f302815d9d0 0x7f30280d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.591+0000 7f3026000700 1 -- 172.21.15.89:0/1025293105 >> v1:172.21.15.113:6789/0 conn(0x7f30280d2370 legacy=0x7f302815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:30.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.591+0000 7f3026000700 1 -- 172.21.15.89:0/1025293105 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30281619e0 con 0x7f30280d16a0 2024-04-02T09:13:30.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.592+0000 7f3026000700 1 --2- 172.21.15.89:0/1025293105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30280d16a0 0x7f30280d1e30 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f3014006fd0 tx=0x7f3014009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f65c4a1df09e9583 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:30.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.592+0000 7f3025600700 1 -- 172.21.15.89:0/1025293105 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014005e30 con 0x7f30280d16a0 2024-04-02T09:13:30.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.592+0000 7f3025600700 1 -- 172.21.15.89:0/1025293105 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3014004030 con 0x7f30280d16a0 2024-04-02T09:13:30.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.592+0000 7f3025600700 1 -- 172.21.15.89:0/1025293105 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30140043e0 con 0x7f30280d16a0 2024-04-02T09:13:30.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.592+0000 7f302de00700 1 -- 172.21.15.89:0/1025293105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30280d16a0 msgr2=0x7f30280d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:30.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.592+0000 7f302de00700 1 --2- 172.21.15.89:0/1025293105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30280d16a0 0x7f30280d1e30 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f3014006fd0 tx=0x7f3014009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:30.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.593+0000 7f302de00700 1 -- 172.21.15.89:0/1025293105 shutdown_connections 2024-04-02T09:13:30.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.593+0000 7f302de00700 1 --2- 172.21.15.89:0/1025293105 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f302815d9d0 0x7f30280d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:30.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.593+0000 7f302de00700 1 --2- 172.21.15.89:0/1025293105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30280d16a0 0x7f30280d1e30 unknown :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:30.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.593+0000 7f302de00700 1 -- 172.21.15.89:0/1025293105 >> 172.21.15.89:0/1025293105 conn(0x7f302808aff0 msgr2=0x7f30280d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:30.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.593+0000 7f302de00700 1 -- 172.21.15.89:0/1025293105 shutdown_connections 2024-04-02T09:13:30.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.593+0000 7f302de00700 1 -- 172.21.15.89:0/1025293105 wait complete. 2024-04-02T09:13:30.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.594+0000 7f302de00700 1 Processor -- start 2024-04-02T09:13:30.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.595+0000 7f302de00700 1 -- start start 2024-04-02T09:13:30.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.595+0000 7f302de00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30280d16a0 0x7f30280b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:30.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f302de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30280d2370 0x7f30281607d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:30.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f302de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f302815d9d0 0x7f302815afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:30.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f302de00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30280b7210 con 0x7f30280d2370 2024-04-02T09:13:30.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f302de00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3028160d10 con 0x7f30280d16a0 2024-04-02T09:13:30.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f302de00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30280b6600 con 0x7f302815d9d0 2024-04-02T09:13:30.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f3026000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30280d2370 0x7f30281607d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f3026000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30280d2370 0x7f30281607d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60682/0 (socket says 172.21.15.89:60682) 2024-04-02T09:13:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f3026000700 1 -- 172.21.15.89:0/316588781 learned_addr learned my addr 172.21.15.89:0/316588781 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f3026a00700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30280d16a0 0x7f30280b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f3027400700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f302815d9d0 0x7f302815afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f3026000700 1 -- 172.21.15.89:0/316588781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f302815d9d0 msgr2=0x7f302815afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f3026000700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f302815d9d0 0x7f302815afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f3026000700 1 -- 172.21.15.89:0/316588781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30280d16a0 msgr2=0x7f30280b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f3026000700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30280d16a0 0x7f30280b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f3026000700 1 -- 172.21.15.89:0/316588781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3014009500 con 0x7f30280d2370 2024-04-02T09:13:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f3026a00700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30280d16a0 0x7f30280b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.596+0000 7f3026000700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30280d2370 0x7f30281607d0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f3014000c00 tx=0x7f3014004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.597+0000 7f301f400700 1 -- 172.21.15.89:0/316588781 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014003e10 con 0x7f30280d2370 2024-04-02T09:13:30.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.597+0000 7f301f400700 1 -- 172.21.15.89:0/316588781 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301401d430 con 0x7f30280d2370 2024-04-02T09:13:30.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.597+0000 7f302de00700 1 -- 172.21.15.89:0/316588781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30280b68e0 con 0x7f30280d2370 2024-04-02T09:13:30.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.597+0000 7f301f400700 1 -- 172.21.15.89:0/316588781 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301401d7e0 con 0x7f30280d2370 2024-04-02T09:13:30.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.597+0000 7f302de00700 1 -- 172.21.15.89:0/316588781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30280b6ea0 con 0x7f30280d2370 2024-04-02T09:13:30.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.598+0000 7f302de00700 1 -- 172.21.15.89:0/316588781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30280d1ca0 con 0x7f30280d2370 2024-04-02T09:13:30.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.599+0000 7f301f400700 1 -- 172.21.15.89:0/316588781 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f301401d940 con 0x7f30280d2370 2024-04-02T09:13:30.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.599+0000 7f301f400700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f300806aed0 0x7f300806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:30.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.600+0000 7f301f400700 1 -- 172.21.15.89:0/316588781 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f301408af20 con 0x7f30280d2370 2024-04-02T09:13:30.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.600+0000 7f3026a00700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f300806aed0 0x7f300806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:30.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.600+0000 7f3026a00700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f300806aed0 0x7f300806d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f30281561d0 tx=0x7f3018008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:30.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.604+0000 7f301f400700 1 -- 172.21.15.89:0/316588781 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3014055bd0 con 0x7f30280d2370 2024-04-02T09:13:30.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:30.852+0000 7f302de00700 1 -- 172.21.15.89:0/316588781 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3028067720 con 0x7f30280d2370 2024-04-02T09:13:31.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.370+0000 7f301f400700 1 -- 172.21.15.89:0/316588781 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v119) v1 ==== 122+0+0 (secure 0 0 0) 0x7f30140591f0 con 0x7f30280d2370 2024-04-02T09:13:31.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.373+0000 7f302de00700 1 -- 172.21.15.89:0/316588781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f300806aed0 msgr2=0x7f300806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:31.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.373+0000 7f302de00700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f300806aed0 0x7f300806d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f30281561d0 tx=0x7f3018008040 comp rx=0 tx=0).stop 2024-04-02T09:13:31.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.373+0000 7f302de00700 1 -- 172.21.15.89:0/316588781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30280d2370 msgr2=0x7f30281607d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:31.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.373+0000 7f302de00700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30280d2370 0x7f30281607d0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f3014000c00 tx=0x7f3014004750 comp rx=0 tx=0).stop 2024-04-02T09:13:31.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.373+0000 7f302de00700 1 -- 172.21.15.89:0/316588781 shutdown_connections 2024-04-02T09:13:31.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.373+0000 7f302de00700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f300806aed0 0x7f300806d390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:31.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.373+0000 7f302de00700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30280d16a0 0x7f30280b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:31.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.373+0000 7f302de00700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f302815d9d0 0x7f302815afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:31.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.373+0000 7f302de00700 1 --2- 172.21.15.89:0/316588781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30280d2370 0x7f30281607d0 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:31.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.373+0000 7f302de00700 1 -- 172.21.15.89:0/316588781 >> 172.21.15.89:0/316588781 conn(0x7f302808aff0 msgr2=0x7f30280b5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:31.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.373+0000 7f302de00700 1 -- 172.21.15.89:0/316588781 shutdown_connections 2024-04-02T09:13:31.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.373+0000 7f302de00700 1 -- 172.21.15.89:0/316588781 wait complete. 2024-04-02T09:13:31.392 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T09:13:31.392 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:13:31.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.585+0000 7f9f31400700 1 Processor -- start 2024-04-02T09:13:31.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.586+0000 7f9f31400700 1 -- start start 2024-04-02T09:13:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.587+0000 7f9f31400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9f2c0ceec0 0x7f9f2c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.587+0000 7f9f31400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9f2c161800 0x7f9f2c0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.587+0000 7f9f31400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2c161c50 con 0x7f9f2c15d640 2024-04-02T09:13:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.587+0000 7f9f31400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f2c0d30d0 con 0x7f9f2c161800 2024-04-02T09:13:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.587+0000 7f9f31400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f2c0b6840 con 0x7f9f2c0ceec0 2024-04-02T09:13:31.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.587+0000 7f9f2a800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9f2c15d640 0x7f9f2c0d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56086/0 (socket says 172.21.15.89:56086) 2024-04-02T09:13:31.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.587+0000 7f9f2a800700 1 -- 172.21.15.89:0/999860290 learned_addr learned my addr 172.21.15.89:0/999860290 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:31.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.587+0000 7f9f2b200700 1 --2- 172.21.15.89:0/999860290 >> v2:172.21.15.89:3300/0 conn(0x7f9f2c161800 0x7f9f2c0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:31.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.587+0000 7f9f29e00700 1 --2- 172.21.15.89:0/999860290 >> v2:172.21.15.113:3300/0 conn(0x7f9f2c0ceec0 0x7f9f2c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:31.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.588+0000 7f9f2b200700 1 -- 172.21.15.89:0/999860290 >> v2:172.21.15.113:3300/0 conn(0x7f9f2c0ceec0 msgr2=0x7f9f2c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:31.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.588+0000 7f9f2b200700 1 --2- 172.21.15.89:0/999860290 >> v2:172.21.15.113:3300/0 conn(0x7f9f2c0ceec0 0x7f9f2c1612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:31.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.588+0000 7f9f2b200700 1 -- 172.21.15.89:0/999860290 >> v1:172.21.15.89:6789/0 conn(0x7f9f2c15d640 legacy=0x7f9f2c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:31.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.588+0000 7f9f2b200700 1 -- 172.21.15.89:0/999860290 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2c0d2b30 con 0x7f9f2c161800 2024-04-02T09:13:31.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.588+0000 7f9f2b200700 1 --2- 172.21.15.89:0/999860290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2c161800 0x7f9f2c0b6cd0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c0cfbf0 tx=0x7f9f2400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c6e10a88a92643 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:31.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.588+0000 7f9f29400700 1 -- 172.21.15.89:0/999860290 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2400e070 con 0x7f9f2c161800 2024-04-02T09:13:31.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.588+0000 7f9f29400700 1 -- 172.21.15.89:0/999860290 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2400bbd0 con 0x7f9f2c161800 2024-04-02T09:13:31.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.588+0000 7f9f29400700 1 -- 172.21.15.89:0/999860290 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f24012430 con 0x7f9f2c161800 2024-04-02T09:13:31.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.589+0000 7f9f31400700 1 -- 172.21.15.89:0/999860290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2c161800 msgr2=0x7f9f2c0b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:31.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.589+0000 7f9f31400700 1 --2- 172.21.15.89:0/999860290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2c161800 0x7f9f2c0b6cd0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c0cfbf0 tx=0x7f9f2400b330 comp rx=0 tx=0).stop 2024-04-02T09:13:31.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.589+0000 7f9f31400700 1 -- 172.21.15.89:0/999860290 shutdown_connections 2024-04-02T09:13:31.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.589+0000 7f9f31400700 1 --2- 172.21.15.89:0/999860290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2c161800 0x7f9f2c0b6cd0 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:31.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.589+0000 7f9f31400700 1 --2- 172.21.15.89:0/999860290 >> v2:172.21.15.113:3300/0 conn(0x7f9f2c0ceec0 0x7f9f2c1612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:31.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.589+0000 7f9f31400700 1 -- 172.21.15.89:0/999860290 >> 172.21.15.89:0/999860290 conn(0x7f9f2c08aff0 msgr2=0x7f9f2c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:31.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.589+0000 7f9f31400700 1 -- 172.21.15.89:0/999860290 shutdown_connections 2024-04-02T09:13:31.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.589+0000 7f9f31400700 1 -- 172.21.15.89:0/999860290 wait complete. 2024-04-02T09:13:31.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.590+0000 7f9f31400700 1 Processor -- start 2024-04-02T09:13:31.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.591+0000 7f9f31400700 1 -- start start 2024-04-02T09:13:31.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f31400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f2c0ceec0 0x7f9f2c0c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:31.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f31400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f2c15d640 0x7f9f2c153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:31.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f31400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2c161800 0x7f9f2c1f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:31.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f31400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2c0c8d80 con 0x7f9f2c161800 2024-04-02T09:13:31.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f31400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2c1f42a0 con 0x7f9f2c0ceec0 2024-04-02T09:13:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f31400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2c1f4580 con 0x7f9f2c15d640 2024-04-02T09:13:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f2b200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2c161800 0x7f9f2c1f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f2b200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2c161800 0x7f9f2c1f3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60698/0 (socket says 172.21.15.89:60698) 2024-04-02T09:13:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f2b200700 1 -- 172.21.15.89:0/1616196561 learned_addr learned my addr 172.21.15.89:0/1616196561 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f2a800700 1 --2- 172.21.15.89:0/1616196561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f2c0ceec0 0x7f9f2c0c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f29e00700 1 --2- 172.21.15.89:0/1616196561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f2c15d640 0x7f9f2c153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f2b200700 1 -- 172.21.15.89:0/1616196561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f2c15d640 msgr2=0x7f9f2c153130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f2b200700 1 --2- 172.21.15.89:0/1616196561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f2c15d640 0x7f9f2c153130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f2b200700 1 -- 172.21.15.89:0/1616196561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f2c0ceec0 msgr2=0x7f9f2c0c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f2b200700 1 --2- 172.21.15.89:0/1616196561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f2c0ceec0 0x7f9f2c0c7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:31.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.592+0000 7f9f2b200700 1 -- 172.21.15.89:0/1616196561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f24009a70 con 0x7f9f2c161800 2024-04-02T09:13:31.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.593+0000 7f9f2b200700 1 --2- 172.21.15.89:0/1616196561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2c161800 0x7f9f2c1f3770 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f9f24014040 tx=0x7f9f24004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:31.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.593+0000 7f9f22a00700 1 -- 172.21.15.89:0/1616196561 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f24012430 con 0x7f9f2c161800 2024-04-02T09:13:31.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.593+0000 7f9f22a00700 1 -- 172.21.15.89:0/1616196561 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f24020070 con 0x7f9f2c161800 2024-04-02T09:13:31.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.593+0000 7f9f31400700 1 -- 172.21.15.89:0/1616196561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2c0c8a10 con 0x7f9f2c161800 2024-04-02T09:13:31.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.593+0000 7f9f22a00700 1 -- 172.21.15.89:0/1616196561 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f24003ea0 con 0x7f9f2c161800 2024-04-02T09:13:31.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.593+0000 7f9f31400700 1 -- 172.21.15.89:0/1616196561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2c1f4860 con 0x7f9f2c161800 2024-04-02T09:13:31.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.594+0000 7f9f31400700 1 -- 172.21.15.89:0/1616196561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f2c0c8ef0 con 0x7f9f2c161800 2024-04-02T09:13:31.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.595+0000 7f9f22a00700 1 -- 172.21.15.89:0/1616196561 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f9f24004280 con 0x7f9f2c161800 2024-04-02T09:13:31.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.596+0000 7f9f22a00700 1 --2- 172.21.15.89:0/1616196561 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9f0c06aed0 0x7f9f0c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:31.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.597+0000 7f9f22a00700 1 -- 172.21.15.89:0/1616196561 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f9f24022070 con 0x7f9f2c161800 2024-04-02T09:13:31.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.597+0000 7f9f2a800700 1 --2- 172.21.15.89:0/1616196561 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9f0c06aed0 0x7f9f0c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:31.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.597+0000 7f9f2a800700 1 --2- 172.21.15.89:0/1616196561 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9f0c06aed0 0x7f9f0c06d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c005910 tx=0x7f9f1c009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:31.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.601+0000 7f9f22a00700 1 -- 172.21.15.89:0/1616196561 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9f24056880 con 0x7f9f2c161800 2024-04-02T09:13:31.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.845+0000 7f9f31400700 1 -- 172.21.15.89:0/1616196561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f2c067720 con 0x7f9f2c161800 2024-04-02T09:13:31.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.846+0000 7f9f22a00700 1 -- 172.21.15.89:0/1616196561 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4752 (secure 0 0 0) 0x7f9f24059ea0 con 0x7f9f2c161800 2024-04-02T09:13:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:31.845 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":119,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9518,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3094781297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3094781297},{"type":"v1","addr":"172.21.15.89:6836","nonce":3094781297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":10978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1799351491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1799351491},{"type":"v1","addr":"172.21.15.113:6834","nonce":1799351491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":11958,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3112580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3112580185},{"type":"v1","addr":"172.21.15.89:6837","nonce":3112580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:28.357996+0000","modified":"2024-04-02T09:13:31.369015+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11964":{"gid":11964,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6835/703852199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":703852199},{"type":"v1","addr":"172.21.15.113:6835","nonce":703852199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-04-02T09:13:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.848+0000 7f9f31400700 1 -- 172.21.15.89:0/1616196561 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9f0c06aed0 msgr2=0x7f9f0c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.848+0000 7f9f31400700 1 --2- 172.21.15.89:0/1616196561 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9f0c06aed0 0x7f9f0c06d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c005910 tx=0x7f9f1c009040 comp rx=0 tx=0).stop 2024-04-02T09:13:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.848+0000 7f9f31400700 1 -- 172.21.15.89:0/1616196561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2c161800 msgr2=0x7f9f2c1f3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.848+0000 7f9f31400700 1 --2- 172.21.15.89:0/1616196561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2c161800 0x7f9f2c1f3770 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f9f24014040 tx=0x7f9f24004ad0 comp rx=0 tx=0).stop 2024-04-02T09:13:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.848+0000 7f9f31400700 1 -- 172.21.15.89:0/1616196561 shutdown_connections 2024-04-02T09:13:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.849+0000 7f9f31400700 1 --2- 172.21.15.89:0/1616196561 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9f0c06aed0 0x7f9f0c06d390 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.849+0000 7f9f31400700 1 --2- 172.21.15.89:0/1616196561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f2c0ceec0 0x7f9f2c0c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:31.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.849+0000 7f9f31400700 1 --2- 172.21.15.89:0/1616196561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2c161800 0x7f9f2c1f3770 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.849+0000 7f9f31400700 1 --2- 172.21.15.89:0/1616196561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9f2c15d640 0x7f9f2c153130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.849+0000 7f9f31400700 1 -- 172.21.15.89:0/1616196561 >> 172.21.15.89:0/1616196561 conn(0x7f9f2c08aff0 msgr2=0x7f9f2c0b6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.849+0000 7f9f31400700 1 -- 172.21.15.89:0/1616196561 shutdown_connections 2024-04-02T09:13:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:31.849+0000 7f9f31400700 1 -- 172.21.15.89:0/1616196561 wait complete. 2024-04-02T09:13:31.849 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 119 2024-04-02T09:13:31.866 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:13:32.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.058+0000 7f54ae800700 1 Processor -- start 2024-04-02T09:13:32.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.059+0000 7f54ae800700 1 -- start start 2024-04-02T09:13:32.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.060+0000 7f54ae800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54a80d18b0 0x7f54a80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.060+0000 7f54ae800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a80ceda0 con 0x7f54a8160f10 2024-04-02T09:13:32.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.060+0000 7f54ae800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a81612f0 con 0x7f54a8161590 2024-04-02T09:13:32.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.060+0000 7f54ae800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a80d1d30 con 0x7f54a80d18b0 2024-04-02T09:13:32.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.060+0000 7f54a7400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f54a8160f10 0x7f54a80d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56090/0 (socket says 172.21.15.89:56090) 2024-04-02T09:13:32.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.060+0000 7f54a7400700 1 -- 172.21.15.89:0/3594830371 learned_addr learned my addr 172.21.15.89:0/3594830371 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:32.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.060+0000 7f54a6a00700 1 --2- 172.21.15.89:0/3594830371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54a80d18b0 0x7f54a80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.060+0000 7f54a6000700 1 -- 172.21.15.89:0/3594830371 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050211622 0 0) 0x7f54a80ceda0 con 0x7f54a8160f10 2024-04-02T09:13:32.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6000700 1 -- 172.21.15.89:0/3594830371 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f548c0035b0 con 0x7f54a8160f10 2024-04-02T09:13:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6000700 1 -- 172.21.15.89:0/3594830371 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 446538923 0 0) 0x7f548c0035b0 con 0x7f54a8160f10 2024-04-02T09:13:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6000700 1 -- 172.21.15.89:0/3594830371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54a80d18b0 msgr2=0x7f54a80cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6000700 1 --2- 172.21.15.89:0/3594830371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54a80d18b0 0x7f54a80cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6000700 1 -- 172.21.15.89:0/3594830371 >> v1:172.21.15.113:6789/0 conn(0x7f54a8161590 legacy=0x7f54a80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6000700 1 -- 172.21.15.89:0/3594830371 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a80ce470 con 0x7f54a8160f10 2024-04-02T09:13:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6000700 1 -- 172.21.15.89:0/3594830371 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f5498001670 con 0x7f54a8160f10 2024-04-02T09:13:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6000700 1 -- 172.21.15.89:0/3594830371 >> v1:172.21.15.89:6789/0 conn(0x7f54a8160f10 legacy=0x7f54a80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6000700 1 --2- 172.21.15.89:0/3594830371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c003a30 0x7f548c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6000700 1 --2- 172.21.15.89:0/3594830371 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f548c004850 0x7f548c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6000700 1 --2- 172.21.15.89:0/3594830371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54a80d18b0 0x7f548c005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6000700 1 -- 172.21.15.89:0/3594830371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a80d1d30 con 0x7f548c003a30 2024-04-02T09:13:32.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6000700 1 -- 172.21.15.89:0/3594830371 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c00bb40 con 0x7f548c004850 2024-04-02T09:13:32.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6000700 1 -- 172.21.15.89:0/3594830371 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548c00bd00 con 0x7f54a80d18b0 2024-04-02T09:13:32.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a7400700 1 --2- 172.21.15.89:0/3594830371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c003a30 0x7f548c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a6a00700 1 --2- 172.21.15.89:0/3594830371 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f548c004850 0x7f548c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.061+0000 7f54a7e00700 1 --2- 172.21.15.89:0/3594830371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54a80d18b0 0x7f548c005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.062+0000 7f54a7400700 1 -- 172.21.15.89:0/3594830371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54a80d18b0 msgr2=0x7f548c005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:32.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.062+0000 7f54a7400700 1 --2- 172.21.15.89:0/3594830371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54a80d18b0 0x7f548c005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.062+0000 7f54a7400700 1 -- 172.21.15.89:0/3594830371 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f548c004850 msgr2=0x7f548c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:32.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.062+0000 7f54a7400700 1 --2- 172.21.15.89:0/3594830371 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f548c004850 0x7f548c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.062+0000 7f54a7400700 1 -- 172.21.15.89:0/3594830371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c0035b0 con 0x7f548c003a30 2024-04-02T09:13:32.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.062+0000 7f54a7e00700 1 --2- 172.21.15.89:0/3594830371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54a80d18b0 0x7f548c005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:32.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.062+0000 7f54a6a00700 1 --2- 172.21.15.89:0/3594830371 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f548c004850 0x7f548c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:32.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.062+0000 7f54a7400700 1 --2- 172.21.15.89:0/3594830371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c003a30 0x7f548c003e50 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f549800c240 tx=0x7f548c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95529c8c6fff8ad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:32.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.062+0000 7f54a6000700 1 -- 172.21.15.89:0/3594830371 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498007c10 con 0x7f548c003a30 2024-04-02T09:13:32.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.062+0000 7f54a6000700 1 -- 172.21.15.89:0/3594830371 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549800ce00 con 0x7f548c003a30 2024-04-02T09:13:32.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.062+0000 7f54a6000700 1 -- 172.21.15.89:0/3594830371 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498005ba0 con 0x7f548c003a30 2024-04-02T09:13:32.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.062+0000 7f54ae800700 1 -- 172.21.15.89:0/3594830371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c003a30 msgr2=0x7f548c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.062+0000 7f54ae800700 1 --2- 172.21.15.89:0/3594830371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c003a30 0x7f548c003e50 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f549800c240 tx=0x7f548c005880 comp rx=0 tx=0).stop 2024-04-02T09:13:32.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.063+0000 7f54a7e00700 1 -- 172.21.15.89:0/3594830371 reap_dead start 2024-04-02T09:13:32.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.063+0000 7f54ae800700 1 -- 172.21.15.89:0/3594830371 shutdown_connections 2024-04-02T09:13:32.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.063+0000 7f54ae800700 1 -- 172.21.15.89:0/3594830371 >> 172.21.15.89:0/3594830371 conn(0x7f54a808aff0 msgr2=0x7f54a80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:32.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.063+0000 7f54ae800700 1 -- 172.21.15.89:0/3594830371 shutdown_connections 2024-04-02T09:13:32.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.063+0000 7f54ae800700 1 -- 172.21.15.89:0/3594830371 wait complete. 2024-04-02T09:13:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.064+0000 7f54ae800700 1 Processor -- start 2024-04-02T09:13:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.065+0000 7f54ae800700 1 -- start start 2024-04-02T09:13:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.065+0000 7f54ae800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c003a30 0x7f54a80b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.065+0000 7f54ae800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54a80b7ee0 0x7f54a814a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.065+0000 7f54ae800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54a814ae90 0x7f54a814b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.065+0000 7f54ae800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a814b850 con 0x7f548c003a30 2024-04-02T09:13:32.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.065+0000 7f54ae800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a80b8360 con 0x7f54a80b7ee0 2024-04-02T09:13:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.065+0000 7f54ae800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a815fb70 con 0x7f54a814ae90 2024-04-02T09:13:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f54a7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c003a30 0x7f54a80b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f54a7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c003a30 0x7f54a80b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60724/0 (socket says 172.21.15.89:60724) 2024-04-02T09:13:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f54a7400700 1 -- 172.21.15.89:0/975396816 learned_addr learned my addr 172.21.15.89:0/975396816 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f54a7e00700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54a814ae90 0x7f54a814b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f54a6a00700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54a80b7ee0 0x7f54a814a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f54a7400700 1 -- 172.21.15.89:0/975396816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54a814ae90 msgr2=0x7f54a814b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f54a7400700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54a814ae90 0x7f54a814b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f54a7400700 1 -- 172.21.15.89:0/975396816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54a80b7ee0 msgr2=0x7f54a814a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f54a7400700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54a80b7ee0 0x7f54a814a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f54a7400700 1 -- 172.21.15.89:0/975396816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a815f890 con 0x7f548c003a30 2024-04-02T09:13:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f54a6a00700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54a80b7ee0 0x7f54a814a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:32.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f54a7400700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c003a30 0x7f54a80b79a0 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f5498000ed0 tx=0x7f5498001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:32.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f549f400700 1 -- 172.21.15.89:0/975396816 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498001630 con 0x7f548c003a30 2024-04-02T09:13:32.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f549f400700 1 -- 172.21.15.89:0/975396816 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54980070d0 con 0x7f548c003a30 2024-04-02T09:13:32.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f549f400700 1 -- 172.21.15.89:0/975396816 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54980074a0 con 0x7f548c003a30 2024-04-02T09:13:32.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.066+0000 7f54ae800700 1 -- 172.21.15.89:0/975396816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a8148a50 con 0x7f548c003a30 2024-04-02T09:13:32.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.067+0000 7f54ae800700 1 -- 172.21.15.89:0/975396816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a8149010 con 0x7f548c003a30 2024-04-02T09:13:32.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.068+0000 7f54ae800700 1 -- 172.21.15.89:0/975396816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a81491d0 con 0x7f548c003a30 2024-04-02T09:13:32.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.068+0000 7f549f400700 1 -- 172.21.15.89:0/975396816 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5498019030 con 0x7f548c003a30 2024-04-02T09:13:32.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.069+0000 7f549f400700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f548806aed0 0x7f548806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.069+0000 7f549f400700 1 -- 172.21.15.89:0/975396816 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f549808ab70 con 0x7f548c003a30 2024-04-02T09:13:32.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.070+0000 7f54a6a00700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f548806aed0 0x7f548806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.070+0000 7f54a6a00700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f548806aed0 0x7f548806d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f54a814f030 tx=0x7f5494008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:32.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.074+0000 7f549f400700 1 -- 172.21.15.89:0/975396816 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f54980558c0 con 0x7f548c003a30 2024-04-02T09:13:32.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.279+0000 7f54ae800700 1 -- 172.21.15.89:0/975396816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54a808d650 con 0x7f548c003a30 2024-04-02T09:13:32.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.280+0000 7f549f400700 1 -- 172.21.15.89:0/975396816 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v244) v1 ==== 74+0+16736 (secure 0 0 0) 0x7f5498058ee0 con 0x7f548c003a30 2024-04-02T09:13:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:32.280 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":244,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:13:29.360152+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:13:26.654647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-02T09:13:27.342285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:13:32.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.282+0000 7f54ae800700 1 -- 172.21.15.89:0/975396816 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f548806aed0 msgr2=0x7f548806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.282+0000 7f54ae800700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f548806aed0 0x7f548806d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f54a814f030 tx=0x7f5494008040 comp rx=0 tx=0).stop 2024-04-02T09:13:32.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.282+0000 7f54ae800700 1 -- 172.21.15.89:0/975396816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c003a30 msgr2=0x7f54a80b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.282+0000 7f54ae800700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c003a30 0x7f54a80b79a0 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f5498000ed0 tx=0x7f5498001a00 comp rx=0 tx=0).stop 2024-04-02T09:13:32.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.283+0000 7f54ae800700 1 -- 172.21.15.89:0/975396816 shutdown_connections 2024-04-02T09:13:32.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.283+0000 7f54ae800700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54a814ae90 0x7f54a814b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.283+0000 7f54ae800700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f548c003a30 0x7f54a80b79a0 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.283+0000 7f54ae800700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f548806aed0 0x7f548806d390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.283+0000 7f54ae800700 1 --2- 172.21.15.89:0/975396816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54a80b7ee0 0x7f54a814a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.283+0000 7f54ae800700 1 -- 172.21.15.89:0/975396816 >> 172.21.15.89:0/975396816 conn(0x7f54a808aff0 msgr2=0x7f54a80ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:32.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.283+0000 7f54ae800700 1 -- 172.21.15.89:0/975396816 shutdown_connections 2024-04-02T09:13:32.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.283+0000 7f54ae800700 1 -- 172.21.15.89:0/975396816 wait complete. 2024-04-02T09:13:32.301 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T09:13:32.302 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:13:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.495+0000 7fc3e6200700 1 Processor -- start 2024-04-02T09:13:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.495+0000 7fc3e6200700 1 -- start start 2024-04-02T09:13:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.496+0000 7fc3e6200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc3e00709c0 0x7fc3e0070e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.496+0000 7fc3e6200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e00b2180 con 0x7fc3e0072580 2024-04-02T09:13:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.496+0000 7fc3e6200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e015b100 con 0x7fc3e00b3af0 2024-04-02T09:13:32.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.496+0000 7fc3e6200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e015b3e0 con 0x7fc3e00709c0 2024-04-02T09:13:32.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.496+0000 7fc3dea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc3e0072580 0x7fc3e00b2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56098/0 (socket says 172.21.15.89:56098) 2024-04-02T09:13:32.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.496+0000 7fc3dea00700 1 -- 172.21.15.89:0/4291393805 learned_addr learned my addr 172.21.15.89:0/4291393805 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:32.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.496+0000 7fc3dfe00700 1 --2- 172.21.15.89:0/4291393805 >> v2:172.21.15.89:3300/0 conn(0x7fc3e00709c0 0x7fc3e0070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.497+0000 7fc3de000700 1 -- 172.21.15.89:0/4291393805 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632903877 0 0) 0x7fc3e00b2180 con 0x7fc3e0072580 2024-04-02T09:13:32.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.497+0000 7fc3de000700 1 -- 172.21.15.89:0/4291393805 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c40035b0 con 0x7fc3e0072580 2024-04-02T09:13:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.497+0000 7fc3dfe00700 1 -- 172.21.15.89:0/4291393805 >> v1:172.21.15.113:6789/0 conn(0x7fc3e00b3af0 legacy=0x7fc3e015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.497+0000 7fc3dfe00700 1 -- 172.21.15.89:0/4291393805 >> v1:172.21.15.89:6789/0 conn(0x7fc3e0072580 legacy=0x7fc3e00b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.497+0000 7fc3dfe00700 1 -- 172.21.15.89:0/4291393805 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e00b23f0 con 0x7fc3e00709c0 2024-04-02T09:13:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.497+0000 7fc3dfe00700 1 --2- 172.21.15.89:0/4291393805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e00709c0 0x7fc3e0070e00 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009d90 tx=0x7fc3d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b097ba8fb0561e4f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.497+0000 7fc3de000700 1 -- 172.21.15.89:0/4291393805 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d800e070 con 0x7fc3e00709c0 2024-04-02T09:13:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.497+0000 7fc3de000700 1 -- 172.21.15.89:0/4291393805 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d800bb20 con 0x7fc3e00709c0 2024-04-02T09:13:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.497+0000 7fc3de000700 1 -- 172.21.15.89:0/4291393805 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d800be90 con 0x7fc3e00709c0 2024-04-02T09:13:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.498+0000 7fc3e6200700 1 -- 172.21.15.89:0/4291393805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e00709c0 msgr2=0x7fc3e0070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.498+0000 7fc3e6200700 1 --2- 172.21.15.89:0/4291393805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e00709c0 0x7fc3e0070e00 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009d90 tx=0x7fc3d800b330 comp rx=0 tx=0).stop 2024-04-02T09:13:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.498+0000 7fc3e6200700 1 -- 172.21.15.89:0/4291393805 shutdown_connections 2024-04-02T09:13:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.498+0000 7fc3e6200700 1 --2- 172.21.15.89:0/4291393805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e00709c0 0x7fc3e0070e00 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.498+0000 7fc3e6200700 1 -- 172.21.15.89:0/4291393805 >> 172.21.15.89:0/4291393805 conn(0x7fc3e00cf190 msgr2=0x7fc3e00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.498+0000 7fc3e6200700 1 -- 172.21.15.89:0/4291393805 shutdown_connections 2024-04-02T09:13:32.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.498+0000 7fc3e6200700 1 -- 172.21.15.89:0/4291393805 wait complete. 2024-04-02T09:13:32.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.500+0000 7fc3e6200700 1 Processor -- start 2024-04-02T09:13:32.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.500+0000 7fc3e6200700 1 -- start start 2024-04-02T09:13:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3e6200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3e00709c0 0x7fc3e015c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3e6200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3e0072580 0x7fc3e01f2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3e6200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e00b3af0 0x7fc3e0156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3e6200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e015c070 con 0x7fc3e00b3af0 2024-04-02T09:13:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3e6200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e015c9d0 con 0x7fc3e0072580 2024-04-02T09:13:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3e6200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e01f32a0 con 0x7fc3e00709c0 2024-04-02T09:13:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3dfe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e00b3af0 0x7fc3e0156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3dfe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e00b3af0 0x7fc3e0156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60748/0 (socket says 172.21.15.89:60748) 2024-04-02T09:13:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3dfe00700 1 -- 172.21.15.89:0/1391466364 learned_addr learned my addr 172.21.15.89:0/1391466364 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3df400700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3e00709c0 0x7fc3e015c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3dea00700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3e0072580 0x7fc3e01f2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3dfe00700 1 -- 172.21.15.89:0/1391466364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3e00709c0 msgr2=0x7fc3e015c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3dfe00700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3e00709c0 0x7fc3e015c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3dfe00700 1 -- 172.21.15.89:0/1391466364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3e0072580 msgr2=0x7fc3e01f2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3dfe00700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3e0072580 0x7fc3e01f2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3dfe00700 1 -- 172.21.15.89:0/1391466364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d8009a70 con 0x7fc3e00b3af0 2024-04-02T09:13:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.501+0000 7fc3dea00700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3e0072580 0x7fc3e01f2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.502+0000 7fc3dfe00700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e00b3af0 0x7fc3e0156ed0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7fc3d800b370 tx=0x7fc3d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.502+0000 7fc3d7400700 1 -- 172.21.15.89:0/1391466364 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8004960 con 0x7fc3e00b3af0 2024-04-02T09:13:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.502+0000 7fc3d7400700 1 -- 172.21.15.89:0/1391466364 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d8020070 con 0x7fc3e00b3af0 2024-04-02T09:13:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.502+0000 7fc3d7400700 1 -- 172.21.15.89:0/1391466364 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d80276c0 con 0x7fc3e00b3af0 2024-04-02T09:13:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.502+0000 7fc3e6200700 1 -- 172.21.15.89:0/1391466364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e0157c90 con 0x7fc3e00b3af0 2024-04-02T09:13:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.502+0000 7fc3e6200700 1 -- 172.21.15.89:0/1391466364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e015b9b0 con 0x7fc3e00b3af0 2024-04-02T09:13:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.503+0000 7fc3e6200700 1 -- 172.21.15.89:0/1391466364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e015c1b0 con 0x7fc3e00b3af0 2024-04-02T09:13:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.504+0000 7fc3d7400700 1 -- 172.21.15.89:0/1391466364 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fc3d8012430 con 0x7fc3e00b3af0 2024-04-02T09:13:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.505+0000 7fc3d7400700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc3c006aed0 0x7fc3c006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.505+0000 7fc3d7400700 1 -- 172.21.15.89:0/1391466364 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fc3d8022070 con 0x7fc3e00b3af0 2024-04-02T09:13:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.505+0000 7fc3df400700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc3c006aed0 0x7fc3c006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.506+0000 7fc3df400700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc3c006aed0 0x7fc3c006d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0000f60 tx=0x7fc3d0009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.509+0000 7fc3d7400700 1 -- 172.21.15.89:0/1391466364 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc3e015c1b0 con 0x7fc3e00b3af0 2024-04-02T09:13:32.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.706+0000 7fc3e6200700 1 -- 172.21.15.89:0/1391466364 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc3e00b3ed0 con 0x7fc3c006aed0 2024-04-02T09:13:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.717+0000 7fc3d7400700 1 -- 172.21.15.89:0/1391466364 <== mgr.11613 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289489 (secure 0 0 0) 0x7fc3d00013f0 con 0x7fc3c006aed0 2024-04-02T09:13:32.720 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.721+0000 7fc3e6200700 1 -- 172.21.15.89:0/1391466364 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc3c006aed0 msgr2=0x7fc3c006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.721+0000 7fc3e6200700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc3c006aed0 0x7fc3c006d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0000f60 tx=0x7fc3d0009040 comp rx=0 tx=0).stop 2024-04-02T09:13:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.721+0000 7fc3e6200700 1 -- 172.21.15.89:0/1391466364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e00b3af0 msgr2=0x7fc3e0156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.721+0000 7fc3e6200700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e00b3af0 0x7fc3e0156ed0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7fc3d800b370 tx=0x7fc3d8004ad0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.721+0000 7fc3e6200700 1 -- 172.21.15.89:0/1391466364 shutdown_connections 2024-04-02T09:13:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.721+0000 7fc3e6200700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e00b3af0 0x7fc3e0156ed0 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.721+0000 7fc3e6200700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3e00709c0 0x7fc3e015c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.721+0000 7fc3e6200700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc3c006aed0 0x7fc3c006d390 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.721+0000 7fc3e6200700 1 --2- 172.21.15.89:0/1391466364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3e0072580 0x7fc3e01f2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.721+0000 7fc3e6200700 1 -- 172.21.15.89:0/1391466364 >> 172.21.15.89:0/1391466364 conn(0x7fc3e00cf190 msgr2=0x7fc3e015e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.722+0000 7fc3e6200700 1 -- 172.21.15.89:0/1391466364 shutdown_connections 2024-04-02T09:13:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.722+0000 7fc3e6200700 1 -- 172.21.15.89:0/1391466364 wait complete. 2024-04-02T09:13:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:13:32.739 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-04-02T09:13:31.429034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":663,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":412780,"kb_used_data":25168,"kb_used_omap":12,"kb_used_meta":387443,"kb_avail":754561940,"statfs":{"total":773094113280,"available":772671426560,"internally_reserved":0,"allocated":25772032,"data_stored":13715166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":396742200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.572919"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367610+0000","last_change":"2024-04-02T09:13:28.396642+0000","last_active":"2024-04-02T09:13:29.367610+0000","last_peered":"2024-04-02T09:13:29.367610+0000","last_clean":"2024-04-02T09:13:29.367610+0000","last_became_active":"2024-04-02T09:13:28.396356+0000","last_became_peered":"2024-04-02T09:13:28.396356+0000","last_unstale":"2024-04-02T09:13:29.367610+0000","last_undegraded":"2024-04-02T09:13:29.367610+0000","last_fullsized":"2024-04-02T09:13:29.367610+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:34.295480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201968+0000","last_change":"2024-04-02T09:13:28.381100+0000","last_active":"2024-04-02T09:13:31.201968+0000","last_peered":"2024-04-02T09:13:31.201968+0000","last_clean":"2024-04-02T09:13:31.201968+0000","last_became_active":"2024-04-02T09:13:28.380832+0000","last_became_peered":"2024-04-02T09:13:28.380832+0000","last_unstale":"2024-04-02T09:13:31.201968+0000","last_undegraded":"2024-04-02T09:13:31.201968+0000","last_fullsized":"2024-04-02T09:13:31.201968+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:21.987215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367599+0000","last_change":"2024-04-02T09:13:28.393942+0000","last_active":"2024-04-02T09:13:29.367599+0000","last_peered":"2024-04-02T09:13:29.367599+0000","last_clean":"2024-04-02T09:13:29.367599+0000","last_became_active":"2024-04-02T09:13:28.393467+0000","last_became_peered":"2024-04-02T09:13:28.393467+0000","last_unstale":"2024-04-02T09:13:29.367599+0000","last_undegraded":"2024-04-02T09:13:29.367599+0000","last_fullsized":"2024-04-02T09:13:29.367599+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:24.836500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367032+0000","last_change":"2024-04-02T09:13:28.394098+0000","last_active":"2024-04-02T09:13:29.367032+0000","last_peered":"2024-04-02T09:13:29.367032+0000","last_clean":"2024-04-02T09:13:29.367032+0000","last_became_active":"2024-04-02T09:13:28.393837+0000","last_became_peered":"2024-04-02T09:13:28.393837+0000","last_unstale":"2024-04-02T09:13:29.367032+0000","last_undegraded":"2024-04-02T09:13:29.367032+0000","last_fullsized":"2024-04-02T09:13:29.367032+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:40:27.068896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.030967+0000","last_change":"2024-04-02T09:13:28.379239+0000","last_active":"2024-04-02T09:13:31.030967+0000","last_peered":"2024-04-02T09:13:31.030967+0000","last_clean":"2024-04-02T09:13:31.030967+0000","last_became_active":"2024-04-02T09:13:28.378970+0000","last_became_peered":"2024-04-02T09:13:28.378970+0000","last_unstale":"2024-04-02T09:13:31.030967+0000","last_undegraded":"2024-04-02T09:13:31.030967+0000","last_fullsized":"2024-04-02T09:13:31.030967+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:37.960638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366504+0000","last_change":"2024-04-02T09:13:28.393747+0000","last_active":"2024-04-02T09:13:29.366504+0000","last_peered":"2024-04-02T09:13:29.366504+0000","last_clean":"2024-04-02T09:13:29.366504+0000","last_became_active":"2024-04-02T09:13:28.393515+0000","last_became_peered":"2024-04-02T09:13:28.393515+0000","last_unstale":"2024-04-02T09:13:29.366504+0000","last_undegraded":"2024-04-02T09:13:29.366504+0000","last_fullsized":"2024-04-02T09:13:29.366504+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:28:37.462844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.400699+0000","last_change":"2024-04-02T09:13:28.399975+0000","last_active":"2024-04-02T09:13:28.400699+0000","last_peered":"2024-04-02T09:13:28.400699+0000","last_clean":"2024-04-02T09:13:28.400699+0000","last_became_active":"2024-04-02T09:13:28.399335+0000","last_became_peered":"2024-04-02T09:13:28.399335+0000","last_unstale":"2024-04-02T09:13:28.400699+0000","last_undegraded":"2024-04-02T09:13:28.400699+0000","last_fullsized":"2024-04-02T09:13:28.400699+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:16:14.265813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.030889+0000","last_change":"2024-04-02T09:13:28.392551+0000","last_active":"2024-04-02T09:13:31.030889+0000","last_peered":"2024-04-02T09:13:31.030889+0000","last_clean":"2024-04-02T09:13:31.030889+0000","last_became_active":"2024-04-02T09:13:28.392159+0000","last_became_peered":"2024-04-02T09:13:28.392159+0000","last_unstale":"2024-04-02T09:13:31.030889+0000","last_undegraded":"2024-04-02T09:13:31.030889+0000","last_fullsized":"2024-04-02T09:13:31.030889+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:42.551696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520858+0000","last_change":"2024-04-02T09:13:28.383891+0000","last_active":"2024-04-02T09:13:30.520858+0000","last_peered":"2024-04-02T09:13:30.520858+0000","last_clean":"2024-04-02T09:13:30.520858+0000","last_became_active":"2024-04-02T09:13:28.383628+0000","last_became_peered":"2024-04-02T09:13:28.383628+0000","last_unstale":"2024-04-02T09:13:30.520858+0000","last_undegraded":"2024-04-02T09:13:30.520858+0000","last_fullsized":"2024-04-02T09:13:30.520858+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:44.222993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.405010+0000","last_change":"2024-04-02T09:13:28.404390+0000","last_active":"2024-04-02T09:13:28.405010+0000","last_peered":"2024-04-02T09:13:28.405010+0000","last_clean":"2024-04-02T09:13:28.405010+0000","last_became_active":"2024-04-02T09:13:28.404038+0000","last_became_peered":"2024-04-02T09:13:28.404038+0000","last_unstale":"2024-04-02T09:13:28.405010+0000","last_undegraded":"2024-04-02T09:13:28.405010+0000","last_fullsized":"2024-04-02T09:13:28.405010+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:47.359176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367663+0000","last_change":"2024-04-02T09:13:28.403517+0000","last_active":"2024-04-02T09:13:29.367663+0000","last_peered":"2024-04-02T09:13:29.367663+0000","last_clean":"2024-04-02T09:13:29.367663+0000","last_became_active":"2024-04-02T09:13:28.401353+0000","last_became_peered":"2024-04-02T09:13:28.401353+0000","last_unstale":"2024-04-02T09:13:29.367663+0000","last_undegraded":"2024-04-02T09:13:29.367663+0000","last_fullsized":"2024-04-02T09:13:29.367663+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:08.980542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.400667+0000","last_change":"2024-04-02T09:13:28.399938+0000","last_active":"2024-04-02T09:13:28.400667+0000","last_peered":"2024-04-02T09:13:28.400667+0000","last_clean":"2024-04-02T09:13:28.400667+0000","last_became_active":"2024-04-02T09:13:28.399350+0000","last_became_peered":"2024-04-02T09:13:28.399350+0000","last_unstale":"2024-04-02T09:13:28.400667+0000","last_undegraded":"2024-04-02T09:13:28.400667+0000","last_fullsized":"2024-04-02T09:13:28.400667+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:05.933699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520714+0000","last_change":"2024-04-02T09:13:28.386688+0000","last_active":"2024-04-02T09:13:30.520714+0000","last_peered":"2024-04-02T09:13:30.520714+0000","last_clean":"2024-04-02T09:13:30.520714+0000","last_became_active":"2024-04-02T09:13:28.386363+0000","last_became_peered":"2024-04-02T09:13:28.386363+0000","last_unstale":"2024-04-02T09:13:30.520714+0000","last_undegraded":"2024-04-02T09:13:30.520714+0000","last_fullsized":"2024-04-02T09:13:30.520714+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:12:41.480404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366464+0000","last_change":"2024-04-02T09:13:28.401903+0000","last_active":"2024-04-02T09:13:29.366464+0000","last_peered":"2024-04-02T09:13:29.366464+0000","last_clean":"2024-04-02T09:13:29.366464+0000","last_became_active":"2024-04-02T09:13:28.399098+0000","last_became_peered":"2024-04-02T09:13:28.399098+0000","last_unstale":"2024-04-02T09:13:29.366464+0000","last_undegraded":"2024-04-02T09:13:29.366464+0000","last_fullsized":"2024-04-02T09:13:29.366464+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:33:56.420978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.365755+0000","last_change":"2024-04-02T09:13:28.383146+0000","last_active":"2024-04-02T09:13:29.365755+0000","last_peered":"2024-04-02T09:13:29.365755+0000","last_clean":"2024-04-02T09:13:29.365755+0000","last_became_active":"2024-04-02T09:13:28.382879+0000","last_became_peered":"2024-04-02T09:13:28.382879+0000","last_unstale":"2024-04-02T09:13:29.365755+0000","last_undegraded":"2024-04-02T09:13:29.365755+0000","last_fullsized":"2024-04-02T09:13:29.365755+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:57:59.273299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201693+0000","last_change":"2024-04-02T09:13:28.381499+0000","last_active":"2024-04-02T09:13:31.201693+0000","last_peered":"2024-04-02T09:13:31.201693+0000","last_clean":"2024-04-02T09:13:31.201693+0000","last_became_active":"2024-04-02T09:13:28.377983+0000","last_became_peered":"2024-04-02T09:13:28.377983+0000","last_unstale":"2024-04-02T09:13:31.201693+0000","last_undegraded":"2024-04-02T09:13:31.201693+0000","last_fullsized":"2024-04-02T09:13:31.201693+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:45.021173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.385314+0000","last_change":"2024-04-02T09:13:28.382899+0000","last_active":"2024-04-02T09:13:28.385314+0000","last_peered":"2024-04-02T09:13:28.385314+0000","last_clean":"2024-04-02T09:13:28.385314+0000","last_became_active":"2024-04-02T09:13:28.382023+0000","last_became_peered":"2024-04-02T09:13:28.382023+0000","last_unstale":"2024-04-02T09:13:28.385314+0000","last_undegraded":"2024-04-02T09:13:28.385314+0000","last_fullsized":"2024-04-02T09:13:28.385314+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:12.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.394157+0000","last_change":"2024-04-02T09:13:28.390594+0000","last_active":"2024-04-02T09:13:28.394157+0000","last_peered":"2024-04-02T09:13:28.394157+0000","last_clean":"2024-04-02T09:13:28.394157+0000","last_became_active":"2024-04-02T09:13:28.388061+0000","last_became_peered":"2024-04-02T09:13:28.388061+0000","last_unstale":"2024-04-02T09:13:28.394157+0000","last_undegraded":"2024-04-02T09:13:28.394157+0000","last_fullsized":"2024-04-02T09:13:28.394157+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:14:04.841237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520986+0000","last_change":"2024-04-02T09:13:28.382883+0000","last_active":"2024-04-02T09:13:30.520986+0000","last_peered":"2024-04-02T09:13:30.520986+0000","last_clean":"2024-04-02T09:13:30.520986+0000","last_became_active":"2024-04-02T09:13:28.382187+0000","last_became_peered":"2024-04-02T09:13:28.382187+0000","last_unstale":"2024-04-02T09:13:30.520986+0000","last_undegraded":"2024-04-02T09:13:30.520986+0000","last_fullsized":"2024-04-02T09:13:30.520986+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:21:26.362248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201650+0000","last_change":"2024-04-02T09:13:28.384295+0000","last_active":"2024-04-02T09:13:31.201650+0000","last_peered":"2024-04-02T09:13:31.201650+0000","last_clean":"2024-04-02T09:13:31.201650+0000","last_became_active":"2024-04-02T09:13:28.384010+0000","last_became_peered":"2024-04-02T09:13:28.384010+0000","last_unstale":"2024-04-02T09:13:31.201650+0000","last_undegraded":"2024-04-02T09:13:31.201650+0000","last_fullsized":"2024-04-02T09:13:31.201650+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:47:33.241308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.030973+0000","last_change":"2024-04-02T09:13:28.380209+0000","last_active":"2024-04-02T09:13:31.030973+0000","last_peered":"2024-04-02T09:13:31.030973+0000","last_clean":"2024-04-02T09:13:31.030973+0000","last_became_active":"2024-04-02T09:13:28.379739+0000","last_became_peered":"2024-04-02T09:13:28.379739+0000","last_unstale":"2024-04-02T09:13:31.030973+0000","last_undegraded":"2024-04-02T09:13:31.030973+0000","last_fullsized":"2024-04-02T09:13:31.030973+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:46:14.163290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520758+0000","last_change":"2024-04-02T09:13:28.392993+0000","last_active":"2024-04-02T09:13:30.520758+0000","last_peered":"2024-04-02T09:13:30.520758+0000","last_clean":"2024-04-02T09:13:30.520758+0000","last_became_active":"2024-04-02T09:13:28.392712+0000","last_became_peered":"2024-04-02T09:13:28.392712+0000","last_unstale":"2024-04-02T09:13:30.520758+0000","last_undegraded":"2024-04-02T09:13:30.520758+0000","last_fullsized":"2024-04-02T09:13:30.520758+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:38.436370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520548+0000","last_change":"2024-04-02T09:13:28.378030+0000","last_active":"2024-04-02T09:13:30.520548+0000","last_peered":"2024-04-02T09:13:30.520548+0000","last_clean":"2024-04-02T09:13:30.520548+0000","last_became_active":"2024-04-02T09:13:28.377700+0000","last_became_peered":"2024-04-02T09:13:28.377700+0000","last_unstale":"2024-04-02T09:13:30.520548+0000","last_undegraded":"2024-04-02T09:13:30.520548+0000","last_fullsized":"2024-04-02T09:13:30.520548+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:40.236322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366260+0000","last_change":"2024-04-02T09:13:28.395364+0000","last_active":"2024-04-02T09:13:29.366260+0000","last_peered":"2024-04-02T09:13:29.366260+0000","last_clean":"2024-04-02T09:13:29.366260+0000","last_became_active":"2024-04-02T09:13:28.395100+0000","last_became_peered":"2024-04-02T09:13:28.395100+0000","last_unstale":"2024-04-02T09:13:29.366260+0000","last_undegraded":"2024-04-02T09:13:29.366260+0000","last_fullsized":"2024-04-02T09:13:29.366260+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:12.863676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367672+0000","last_change":"2024-04-02T09:13:28.399604+0000","last_active":"2024-04-02T09:13:29.367672+0000","last_peered":"2024-04-02T09:13:29.367672+0000","last_clean":"2024-04-02T09:13:29.367672+0000","last_became_active":"2024-04-02T09:13:28.399333+0000","last_became_peered":"2024-04-02T09:13:28.399333+0000","last_unstale":"2024-04-02T09:13:29.367672+0000","last_undegraded":"2024-04-02T09:13:29.367672+0000","last_fullsized":"2024-04-02T09:13:29.367672+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:18.748067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.031206+0000","last_change":"2024-04-02T09:13:28.378770+0000","last_active":"2024-04-02T09:13:31.031206+0000","last_peered":"2024-04-02T09:13:31.031206+0000","last_clean":"2024-04-02T09:13:31.031206+0000","last_became_active":"2024-04-02T09:13:28.378521+0000","last_became_peered":"2024-04-02T09:13:28.378521+0000","last_unstale":"2024-04-02T09:13:31.031206+0000","last_undegraded":"2024-04-02T09:13:31.031206+0000","last_fullsized":"2024-04-02T09:13:31.031206+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:24:11.210144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201827+0000","last_change":"2024-04-02T09:13:28.389084+0000","last_active":"2024-04-02T09:13:31.201827+0000","last_peered":"2024-04-02T09:13:31.201827+0000","last_clean":"2024-04-02T09:13:31.201827+0000","last_became_active":"2024-04-02T09:13:28.388624+0000","last_became_peered":"2024-04-02T09:13:28.388624+0000","last_unstale":"2024-04-02T09:13:31.201827+0000","last_undegraded":"2024-04-02T09:13:31.201827+0000","last_fullsized":"2024-04-02T09:13:31.201827+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:23.766461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366809+0000","last_change":"2024-04-02T09:13:28.395286+0000","last_active":"2024-04-02T09:13:29.366809+0000","last_peered":"2024-04-02T09:13:29.366809+0000","last_clean":"2024-04-02T09:13:29.366809+0000","last_became_active":"2024-04-02T09:13:28.394377+0000","last_became_peered":"2024-04-02T09:13:28.394377+0000","last_unstale":"2024-04-02T09:13:29.366809+0000","last_undegraded":"2024-04-02T09:13:29.366809+0000","last_fullsized":"2024-04-02T09:13:29.366809+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:37:39.028435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366453+0000","last_change":"2024-04-02T09:13:28.391267+0000","last_active":"2024-04-02T09:13:29.366453+0000","last_peered":"2024-04-02T09:13:29.366453+0000","last_clean":"2024-04-02T09:13:29.366453+0000","last_became_active":"2024-04-02T09:13:28.390985+0000","last_became_peered":"2024-04-02T09:13:28.390985+0000","last_unstale":"2024-04-02T09:13:29.366453+0000","last_undegraded":"2024-04-02T09:13:29.366453+0000","last_fullsized":"2024-04-02T09:13:29.366453+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:30.210394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520638+0000","last_change":"2024-04-02T09:13:28.380650+0000","last_active":"2024-04-02T09:13:30.520638+0000","last_peered":"2024-04-02T09:13:30.520638+0000","last_clean":"2024-04-02T09:13:30.520638+0000","last_became_active":"2024-04-02T09:13:28.380376+0000","last_became_peered":"2024-04-02T09:13:28.380376+0000","last_unstale":"2024-04-02T09:13:30.520638+0000","last_undegraded":"2024-04-02T09:13:30.520638+0000","last_fullsized":"2024-04-02T09:13:30.520638+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:46:17.713244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366034+0000","last_change":"2024-04-02T09:13:28.390273+0000","last_active":"2024-04-02T09:13:29.366034+0000","last_peered":"2024-04-02T09:13:29.366034+0000","last_clean":"2024-04-02T09:13:29.366034+0000","last_became_active":"2024-04-02T09:13:28.389990+0000","last_became_peered":"2024-04-02T09:13:28.389990+0000","last_unstale":"2024-04-02T09:13:29.366034+0000","last_undegraded":"2024-04-02T09:13:29.366034+0000","last_fullsized":"2024-04-02T09:13:29.366034+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:30:33.534225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201504+0000","last_change":"2024-04-02T09:13:28.395003+0000","last_active":"2024-04-02T09:13:31.201504+0000","last_peered":"2024-04-02T09:13:31.201504+0000","last_clean":"2024-04-02T09:13:31.201504+0000","last_became_active":"2024-04-02T09:13:28.394635+0000","last_became_peered":"2024-04-02T09:13:28.394635+0000","last_unstale":"2024-04-02T09:13:31.201504+0000","last_undegraded":"2024-04-02T09:13:31.201504+0000","last_fullsized":"2024-04-02T09:13:31.201504+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:54.758118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.393384+0000","last_change":"2024-04-02T09:13:28.391302+0000","last_active":"2024-04-02T09:13:28.393384+0000","last_peered":"2024-04-02T09:13:28.393384+0000","last_clean":"2024-04-02T09:13:28.393384+0000","last_became_active":"2024-04-02T09:13:28.388976+0000","last_became_peered":"2024-04-02T09:13:28.388976+0000","last_unstale":"2024-04-02T09:13:28.393384+0000","last_undegraded":"2024-04-02T09:13:28.393384+0000","last_fullsized":"2024-04-02T09:13:28.393384+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:02.556232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"243'1","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366012+0000","last_change":"2024-04-02T09:13:27.881269+0000","last_active":"2024-04-02T09:13:29.366012+0000","last_peered":"2024-04-02T09:13:29.366012+0000","last_clean":"2024-04-02T09:13:29.366012+0000","last_became_active":"2024-04-02T09:13:27.880664+0000","last_became_peered":"2024-04-02T09:13:27.880664+0000","last_unstale":"2024-04-02T09:13:29.366012+0000","last_undegraded":"2024-04-02T09:13:29.366012+0000","last_fullsized":"2024-04-02T09:13:29.366012+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:53.750473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366969+0000","last_change":"2024-04-02T09:13:28.399529+0000","last_active":"2024-04-02T09:13:29.366969+0000","last_peered":"2024-04-02T09:13:29.366969+0000","last_clean":"2024-04-02T09:13:29.366969+0000","last_became_active":"2024-04-02T09:13:28.399042+0000","last_became_peered":"2024-04-02T09:13:28.399042+0000","last_unstale":"2024-04-02T09:13:29.366969+0000","last_undegraded":"2024-04-02T09:13:29.366969+0000","last_fullsized":"2024-04-02T09:13:29.366969+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:58:19.109584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1d","version":"243'2","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367699+0000","last_change":"2024-04-02T09:13:27.889299+0000","last_active":"2024-04-02T09:13:29.367699+0000","last_peered":"2024-04-02T09:13:29.367699+0000","last_clean":"2024-04-02T09:13:29.367699+0000","last_became_active":"2024-04-02T09:13:27.888249+0000","last_became_peered":"2024-04-02T09:13:27.888249+0000","last_unstale":"2024-04-02T09:13:29.367699+0000","last_undegraded":"2024-04-02T09:13:29.367699+0000","last_fullsized":"2024-04-02T09:13:29.367699+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:01:31.866958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520985+0000","last_change":"2024-04-02T09:13:28.382423+0000","last_active":"2024-04-02T09:13:30.520985+0000","last_peered":"2024-04-02T09:13:30.520985+0000","last_clean":"2024-04-02T09:13:30.520985+0000","last_became_active":"2024-04-02T09:13:28.381840+0000","last_became_peered":"2024-04-02T09:13:28.381840+0000","last_unstale":"2024-04-02T09:13:30.520985+0000","last_undegraded":"2024-04-02T09:13:30.520985+0000","last_fullsized":"2024-04-02T09:13:30.520985+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:04:44.765954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"243'2","reported_seq":12,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.396509+0000","last_change":"2024-04-02T09:13:27.885629+0000","last_active":"2024-04-02T09:13:28.396509+0000","last_peered":"2024-04-02T09:13:28.396509+0000","last_clean":"2024-04-02T09:13:28.396509+0000","last_became_active":"2024-04-02T09:13:27.884921+0000","last_became_peered":"2024-04-02T09:13:27.884921+0000","last_unstale":"2024-04-02T09:13:28.396509+0000","last_undegraded":"2024-04-02T09:13:28.396509+0000","last_fullsized":"2024-04-02T09:13:28.396509+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:49:58.033026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.387545+0000","last_change":"2024-04-02T09:13:28.381558+0000","last_active":"2024-04-02T09:13:28.387545+0000","last_peered":"2024-04-02T09:13:28.387545+0000","last_clean":"2024-04-02T09:13:28.387545+0000","last_became_active":"2024-04-02T09:13:28.381306+0000","last_became_peered":"2024-04-02T09:13:28.381306+0000","last_unstale":"2024-04-02T09:13:28.387545+0000","last_undegraded":"2024-04-02T09:13:28.387545+0000","last_fullsized":"2024-04-02T09:13:28.387545+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:12:17.313507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1b","version":"243'1","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.397372+0000","last_change":"2024-04-02T09:13:27.884062+0000","last_active":"2024-04-02T09:13:28.397372+0000","last_peered":"2024-04-02T09:13:28.397372+0000","last_clean":"2024-04-02T09:13:28.397372+0000","last_became_active":"2024-04-02T09:13:27.883677+0000","last_became_peered":"2024-04-02T09:13:27.883677+0000","last_unstale":"2024-04-02T09:13:28.397372+0000","last_undegraded":"2024-04-02T09:13:28.397372+0000","last_fullsized":"2024-04-02T09:13:28.397372+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:14.774442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.365865+0000","last_change":"2024-04-02T09:13:28.389833+0000","last_active":"2024-04-02T09:13:29.365865+0000","last_peered":"2024-04-02T09:13:29.365865+0000","last_clean":"2024-04-02T09:13:29.365865+0000","last_became_active":"2024-04-02T09:13:28.389495+0000","last_became_peered":"2024-04-02T09:13:28.389495+0000","last_unstale":"2024-04-02T09:13:29.365865+0000","last_undegraded":"2024-04-02T09:13:29.365865+0000","last_fullsized":"2024-04-02T09:13:29.365865+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:27.049697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1a","version":"243'1","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.397106+0000","last_change":"2024-04-02T09:13:27.883532+0000","last_active":"2024-04-02T09:13:28.397106+0000","last_peered":"2024-04-02T09:13:28.397106+0000","last_clean":"2024-04-02T09:13:28.397106+0000","last_became_active":"2024-04-02T09:13:27.883252+0000","last_became_peered":"2024-04-02T09:13:27.883252+0000","last_unstale":"2024-04-02T09:13:28.397106+0000","last_undegraded":"2024-04-02T09:13:28.397106+0000","last_fullsized":"2024-04-02T09:13:28.397106+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.950371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366139+0000","last_change":"2024-04-02T09:13:28.403943+0000","last_active":"2024-04-02T09:13:29.366139+0000","last_peered":"2024-04-02T09:13:29.366139+0000","last_clean":"2024-04-02T09:13:29.366139+0000","last_became_active":"2024-04-02T09:13:28.402197+0000","last_became_peered":"2024-04-02T09:13:28.402197+0000","last_unstale":"2024-04-02T09:13:29.366139+0000","last_undegraded":"2024-04-02T09:13:29.366139+0000","last_fullsized":"2024-04-02T09:13:29.366139+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:50.579695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.030923+0000","last_change":"2024-04-02T09:13:27.883508+0000","last_active":"2024-04-02T09:13:31.030923+0000","last_peered":"2024-04-02T09:13:31.030923+0000","last_clean":"2024-04-02T09:13:31.030923+0000","last_became_active":"2024-04-02T09:13:27.883263+0000","last_became_peered":"2024-04-02T09:13:27.883263+0000","last_unstale":"2024-04-02T09:13:31.030923+0000","last_undegraded":"2024-04-02T09:13:31.030923+0000","last_fullsized":"2024-04-02T09:13:31.030923+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:37.428725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366154+0000","last_change":"2024-04-02T09:13:28.403768+0000","last_active":"2024-04-02T09:13:29.366154+0000","last_peered":"2024-04-02T09:13:29.366154+0000","last_clean":"2024-04-02T09:13:29.366154+0000","last_became_active":"2024-04-02T09:13:28.399745+0000","last_became_peered":"2024-04-02T09:13:28.399745+0000","last_unstale":"2024-04-02T09:13:29.366154+0000","last_undegraded":"2024-04-02T09:13:29.366154+0000","last_fullsized":"2024-04-02T09:13:29.366154+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:39.541325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.369408+0000","last_change":"2024-04-02T09:13:27.888598+0000","last_active":"2024-04-02T09:13:28.369408+0000","last_peered":"2024-04-02T09:13:28.369408+0000","last_clean":"2024-04-02T09:13:28.369408+0000","last_became_active":"2024-04-02T09:13:27.888115+0000","last_became_peered":"2024-04-02T09:13:27.888115+0000","last_unstale":"2024-04-02T09:13:28.369408+0000","last_undegraded":"2024-04-02T09:13:28.369408+0000","last_fullsized":"2024-04-02T09:13:28.369408+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:57:03.609779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367965+0000","last_change":"2024-04-02T09:13:28.404602+0000","last_active":"2024-04-02T09:13:29.367965+0000","last_peered":"2024-04-02T09:13:29.367965+0000","last_clean":"2024-04-02T09:13:29.367965+0000","last_became_active":"2024-04-02T09:13:28.403446+0000","last_became_peered":"2024-04-02T09:13:28.403446+0000","last_unstale":"2024-04-02T09:13:29.367965+0000","last_undegraded":"2024-04-02T09:13:29.367965+0000","last_fullsized":"2024-04-02T09:13:29.367965+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:24.279179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366051+0000","last_change":"2024-04-02T09:13:27.887286+0000","last_active":"2024-04-02T09:13:29.366051+0000","last_peered":"2024-04-02T09:13:29.366051+0000","last_clean":"2024-04-02T09:13:29.366051+0000","last_became_active":"2024-04-02T09:13:27.886999+0000","last_became_peered":"2024-04-02T09:13:27.886999+0000","last_unstale":"2024-04-02T09:13:29.366051+0000","last_undegraded":"2024-04-02T09:13:29.366051+0000","last_fullsized":"2024-04-02T09:13:29.366051+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:44.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.385852+0000","last_change":"2024-04-02T09:13:28.378767+0000","last_active":"2024-04-02T09:13:28.385852+0000","last_peered":"2024-04-02T09:13:28.385852+0000","last_clean":"2024-04-02T09:13:28.385852+0000","last_became_active":"2024-04-02T09:13:28.378487+0000","last_became_peered":"2024-04-02T09:13:28.378487+0000","last_unstale":"2024-04-02T09:13:28.385852+0000","last_undegraded":"2024-04-02T09:13:28.385852+0000","last_fullsized":"2024-04-02T09:13:28.385852+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:26:47.539777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.368012+0000","last_change":"2024-04-02T09:13:27.884469+0000","last_active":"2024-04-02T09:13:29.368012+0000","last_peered":"2024-04-02T09:13:29.368012+0000","last_clean":"2024-04-02T09:13:29.368012+0000","last_became_active":"2024-04-02T09:13:27.882570+0000","last_became_peered":"2024-04-02T09:13:27.882570+0000","last_unstale":"2024-04-02T09:13:29.368012+0000","last_undegraded":"2024-04-02T09:13:29.368012+0000","last_fullsized":"2024-04-02T09:13:29.368012+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:57:12.395680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201703+0000","last_change":"2024-04-02T09:13:28.387205+0000","last_active":"2024-04-02T09:13:31.201703+0000","last_peered":"2024-04-02T09:13:31.201703+0000","last_clean":"2024-04-02T09:13:31.201703+0000","last_became_active":"2024-04-02T09:13:28.384126+0000","last_became_peered":"2024-04-02T09:13:28.384126+0000","last_unstale":"2024-04-02T09:13:31.201703+0000","last_undegraded":"2024-04-02T09:13:31.201703+0000","last_fullsized":"2024-04-02T09:13:31.201703+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:52.330349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.15","version":"243'3","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201885+0000","last_change":"2024-04-02T09:13:27.883818+0000","last_active":"2024-04-02T09:13:31.201885+0000","last_peered":"2024-04-02T09:13:31.201885+0000","last_clean":"2024-04-02T09:13:31.201885+0000","last_became_active":"2024-04-02T09:13:27.883527+0000","last_became_peered":"2024-04-02T09:13:27.883527+0000","last_unstale":"2024-04-02T09:13:31.201885+0000","last_undegraded":"2024-04-02T09:13:31.201885+0000","last_fullsized":"2024-04-02T09:13:31.201885+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:49.572049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.031123+0000","last_change":"2024-04-02T09:13:28.379346+0000","last_active":"2024-04-02T09:13:31.031123+0000","last_peered":"2024-04-02T09:13:31.031123+0000","last_clean":"2024-04-02T09:13:31.031123+0000","last_became_active":"2024-04-02T09:13:28.378741+0000","last_became_peered":"2024-04-02T09:13:28.378741+0000","last_unstale":"2024-04-02T09:13:31.031123+0000","last_undegraded":"2024-04-02T09:13:31.031123+0000","last_fullsized":"2024-04-02T09:13:31.031123+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:44.092529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.14","version":"243'6","reported_seq":17,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367490+0000","last_change":"2024-04-02T09:13:27.885270+0000","last_active":"2024-04-02T09:13:29.367490+0000","last_peered":"2024-04-02T09:13:29.367490+0000","last_clean":"2024-04-02T09:13:29.367490+0000","last_became_active":"2024-04-02T09:13:27.884953+0000","last_became_peered":"2024-04-02T09:13:27.884953+0000","last_unstale":"2024-04-02T09:13:29.367490+0000","last_undegraded":"2024-04-02T09:13:29.367490+0000","last_fullsized":"2024-04-02T09:13:29.367490+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:48.103277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.365957+0000","last_change":"2024-04-02T09:13:28.393735+0000","last_active":"2024-04-02T09:13:29.365957+0000","last_peered":"2024-04-02T09:13:29.365957+0000","last_clean":"2024-04-02T09:13:29.365957+0000","last_became_active":"2024-04-02T09:13:28.393387+0000","last_became_peered":"2024-04-02T09:13:28.393387+0000","last_unstale":"2024-04-02T09:13:29.365957+0000","last_undegraded":"2024-04-02T09:13:29.365957+0000","last_fullsized":"2024-04-02T09:13:29.365957+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:47.359176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.13","version":"243'2","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367787+0000","last_change":"2024-04-02T09:13:27.885596+0000","last_active":"2024-04-02T09:13:29.367787+0000","last_peered":"2024-04-02T09:13:29.367787+0000","last_clean":"2024-04-02T09:13:29.367787+0000","last_became_active":"2024-04-02T09:13:27.885310+0000","last_became_peered":"2024-04-02T09:13:27.885310+0000","last_unstale":"2024-04-02T09:13:29.367787+0000","last_undegraded":"2024-04-02T09:13:29.367787+0000","last_fullsized":"2024-04-02T09:13:29.367787+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:02.497118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520825+0000","last_change":"2024-04-02T09:13:28.378384+0000","last_active":"2024-04-02T09:13:30.520825+0000","last_peered":"2024-04-02T09:13:30.520825+0000","last_clean":"2024-04-02T09:13:30.520825+0000","last_became_active":"2024-04-02T09:13:28.377833+0000","last_became_peered":"2024-04-02T09:13:28.377833+0000","last_unstale":"2024-04-02T09:13:30.520825+0000","last_undegraded":"2024-04-02T09:13:30.520825+0000","last_fullsized":"2024-04-02T09:13:30.520825+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:37:54.526506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366553+0000","last_change":"2024-04-02T09:13:27.880069+0000","last_active":"2024-04-02T09:13:29.366553+0000","last_peered":"2024-04-02T09:13:29.366553+0000","last_clean":"2024-04-02T09:13:29.366553+0000","last_became_active":"2024-04-02T09:13:27.879793+0000","last_became_peered":"2024-04-02T09:13:27.879793+0000","last_unstale":"2024-04-02T09:13:29.366553+0000","last_undegraded":"2024-04-02T09:13:29.366553+0000","last_fullsized":"2024-04-02T09:13:29.366553+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:10.198371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.396599+0000","last_change":"2024-04-02T09:13:28.394088+0000","last_active":"2024-04-02T09:13:28.396599+0000","last_peered":"2024-04-02T09:13:28.396599+0000","last_clean":"2024-04-02T09:13:28.396599+0000","last_became_active":"2024-04-02T09:13:28.392932+0000","last_became_peered":"2024-04-02T09:13:28.392932+0000","last_unstale":"2024-04-02T09:13:28.396599+0000","last_undegraded":"2024-04-02T09:13:28.396599+0000","last_fullsized":"2024-04-02T09:13:28.396599+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:54.683721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.030976+0000","last_change":"2024-04-02T09:13:27.881714+0000","last_active":"2024-04-02T09:13:31.030976+0000","last_peered":"2024-04-02T09:13:31.030976+0000","last_clean":"2024-04-02T09:13:31.030976+0000","last_became_active":"2024-04-02T09:13:27.881428+0000","last_became_peered":"2024-04-02T09:13:27.881428+0000","last_unstale":"2024-04-02T09:13:31.030976+0000","last_undegraded":"2024-04-02T09:13:31.030976+0000","last_fullsized":"2024-04-02T09:13:31.030976+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:50.081688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366047+0000","last_change":"2024-04-02T09:13:28.391188+0000","last_active":"2024-04-02T09:13:29.366047+0000","last_peered":"2024-04-02T09:13:29.366047+0000","last_clean":"2024-04-02T09:13:29.366047+0000","last_became_active":"2024-04-02T09:13:28.390867+0000","last_became_peered":"2024-04-02T09:13:28.390867+0000","last_unstale":"2024-04-02T09:13:29.366047+0000","last_undegraded":"2024-04-02T09:13:29.366047+0000","last_fullsized":"2024-04-02T09:13:29.366047+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:26:58.531401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.10","version":"243'8","reported_seq":19,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201677+0000","last_change":"2024-04-02T09:13:27.886503+0000","last_active":"2024-04-02T09:13:31.201677+0000","last_peered":"2024-04-02T09:13:31.201677+0000","last_clean":"2024-04-02T09:13:31.201677+0000","last_became_active":"2024-04-02T09:13:27.886121+0000","last_became_peered":"2024-04-02T09:13:27.886121+0000","last_unstale":"2024-04-02T09:13:31.201677+0000","last_undegraded":"2024-04-02T09:13:31.201677+0000","last_fullsized":"2024-04-02T09:13:31.201677+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:38.533680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.379586+0000","last_change":"2024-04-02T09:13:28.376609+0000","last_active":"2024-04-02T09:13:28.379586+0000","last_peered":"2024-04-02T09:13:28.379586+0000","last_clean":"2024-04-02T09:13:28.379586+0000","last_became_active":"2024-04-02T09:13:28.376226+0000","last_became_peered":"2024-04-02T09:13:28.376226+0000","last_unstale":"2024-04-02T09:13:28.379586+0000","last_undegraded":"2024-04-02T09:13:28.379586+0000","last_fullsized":"2024-04-02T09:13:28.379586+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:59.766007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"243'1","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366132+0000","last_change":"2024-04-02T09:13:27.883080+0000","last_active":"2024-04-02T09:13:29.366132+0000","last_peered":"2024-04-02T09:13:29.366132+0000","last_clean":"2024-04-02T09:13:29.366132+0000","last_became_active":"2024-04-02T09:13:27.882612+0000","last_became_peered":"2024-04-02T09:13:27.882612+0000","last_unstale":"2024-04-02T09:13:29.366132+0000","last_undegraded":"2024-04-02T09:13:29.366132+0000","last_fullsized":"2024-04-02T09:13:29.366132+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:20.979570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201892+0000","last_change":"2024-04-02T09:13:28.381499+0000","last_active":"2024-04-02T09:13:31.201892+0000","last_peered":"2024-04-02T09:13:31.201892+0000","last_clean":"2024-04-02T09:13:31.201892+0000","last_became_active":"2024-04-02T09:13:28.381286+0000","last_became_peered":"2024-04-02T09:13:28.381286+0000","last_unstale":"2024-04-02T09:13:31.201892+0000","last_undegraded":"2024-04-02T09:13:31.201892+0000","last_fullsized":"2024-04-02T09:13:31.201892+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:18.748067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.e","version":"243'2","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.368061+0000","last_change":"2024-04-02T09:13:27.888558+0000","last_active":"2024-04-02T09:13:29.368061+0000","last_peered":"2024-04-02T09:13:29.368061+0000","last_clean":"2024-04-02T09:13:29.368061+0000","last_became_active":"2024-04-02T09:13:27.888288+0000","last_became_peered":"2024-04-02T09:13:27.888288+0000","last_unstale":"2024-04-02T09:13:29.368061+0000","last_undegraded":"2024-04-02T09:13:29.368061+0000","last_fullsized":"2024-04-02T09:13:29.368061+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:15.386238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.365942+0000","last_change":"2024-04-02T09:13:28.372849+0000","last_active":"2024-04-02T09:13:29.365942+0000","last_peered":"2024-04-02T09:13:29.365942+0000","last_clean":"2024-04-02T09:13:29.365942+0000","last_became_active":"2024-04-02T09:13:28.372595+0000","last_became_peered":"2024-04-02T09:13:28.372595+0000","last_unstale":"2024-04-02T09:13:29.365942+0000","last_undegraded":"2024-04-02T09:13:29.365942+0000","last_fullsized":"2024-04-02T09:13:29.365942+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:12.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.d","version":"243'1","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366061+0000","last_change":"2024-04-02T09:13:27.879417+0000","last_active":"2024-04-02T09:13:29.366061+0000","last_peered":"2024-04-02T09:13:29.366061+0000","last_clean":"2024-04-02T09:13:29.366061+0000","last_became_active":"2024-04-02T09:13:27.879165+0000","last_became_peered":"2024-04-02T09:13:27.879165+0000","last_unstale":"2024-04-02T09:13:29.366061+0000","last_undegraded":"2024-04-02T09:13:29.366061+0000","last_fullsized":"2024-04-02T09:13:29.366061+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:47:32.233662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.393129+0000","last_change":"2024-04-02T09:13:28.392130+0000","last_active":"2024-04-02T09:13:28.393129+0000","last_peered":"2024-04-02T09:13:28.393129+0000","last_clean":"2024-04-02T09:13:28.393129+0000","last_became_active":"2024-04-02T09:13:28.391690+0000","last_became_peered":"2024-04-02T09:13:28.391690+0000","last_unstale":"2024-04-02T09:13:28.393129+0000","last_undegraded":"2024-04-02T09:13:28.393129+0000","last_fullsized":"2024-04-02T09:13:28.393129+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:24.319747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367068+0000","last_change":"2024-04-02T09:13:27.884781+0000","last_active":"2024-04-02T09:13:29.367068+0000","last_peered":"2024-04-02T09:13:29.367068+0000","last_clean":"2024-04-02T09:13:29.367068+0000","last_became_active":"2024-04-02T09:13:27.884507+0000","last_became_peered":"2024-04-02T09:13:27.884507+0000","last_unstale":"2024-04-02T09:13:29.367068+0000","last_undegraded":"2024-04-02T09:13:29.367068+0000","last_fullsized":"2024-04-02T09:13:29.367068+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:31:18.889273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520561+0000","last_change":"2024-04-02T09:13:28.381580+0000","last_active":"2024-04-02T09:13:30.520561+0000","last_peered":"2024-04-02T09:13:30.520561+0000","last_clean":"2024-04-02T09:13:30.520561+0000","last_became_active":"2024-04-02T09:13:28.381300+0000","last_became_peered":"2024-04-02T09:13:28.381300+0000","last_unstale":"2024-04-02T09:13:30.520561+0000","last_undegraded":"2024-04-02T09:13:30.520561+0000","last_fullsized":"2024-04-02T09:13:30.520561+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:51.089333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"243'2","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366666+0000","last_change":"2024-04-02T09:13:27.875291+0000","last_active":"2024-04-02T09:13:29.366666+0000","last_peered":"2024-04-02T09:13:29.366666+0000","last_clean":"2024-04-02T09:13:29.366666+0000","last_became_active":"2024-04-02T09:13:27.875003+0000","last_became_peered":"2024-04-02T09:13:27.875003+0000","last_unstale":"2024-04-02T09:13:29.366666+0000","last_undegraded":"2024-04-02T09:13:29.366666+0000","last_fullsized":"2024-04-02T09:13:29.366666+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:00.637981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366563+0000","last_change":"2024-04-02T09:13:28.404265+0000","last_active":"2024-04-02T09:13:29.366563+0000","last_peered":"2024-04-02T09:13:29.366563+0000","last_clean":"2024-04-02T09:13:29.366563+0000","last_became_active":"2024-04-02T09:13:28.403894+0000","last_became_peered":"2024-04-02T09:13:28.403894+0000","last_unstale":"2024-04-02T09:13:29.366563+0000","last_undegraded":"2024-04-02T09:13:29.366563+0000","last_fullsized":"2024-04-02T09:13:29.366563+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:05:23.433426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366765+0000","last_change":"2024-04-02T09:13:27.881604+0000","last_active":"2024-04-02T09:13:29.366765+0000","last_peered":"2024-04-02T09:13:29.366765+0000","last_clean":"2024-04-02T09:13:29.366765+0000","last_became_active":"2024-04-02T09:13:27.881304+0000","last_became_peered":"2024-04-02T09:13:27.881304+0000","last_unstale":"2024-04-02T09:13:29.366765+0000","last_undegraded":"2024-04-02T09:13:29.366765+0000","last_fullsized":"2024-04-02T09:13:29.366765+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:02.497118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.377099+0000","last_change":"2024-04-02T09:13:28.374960+0000","last_active":"2024-04-02T09:13:28.377099+0000","last_peered":"2024-04-02T09:13:28.377099+0000","last_clean":"2024-04-02T09:13:28.377099+0000","last_became_active":"2024-04-02T09:13:28.373683+0000","last_became_peered":"2024-04-02T09:13:28.373683+0000","last_unstale":"2024-04-02T09:13:28.377099+0000","last_undegraded":"2024-04-02T09:13:28.377099+0000","last_fullsized":"2024-04-02T09:13:28.377099+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:15.851301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.369657+0000","last_change":"2024-04-02T09:13:27.877837+0000","last_active":"2024-04-02T09:13:28.369657+0000","last_peered":"2024-04-02T09:13:28.369657+0000","last_clean":"2024-04-02T09:13:28.369657+0000","last_became_active":"2024-04-02T09:13:27.876792+0000","last_became_peered":"2024-04-02T09:13:27.876792+0000","last_unstale":"2024-04-02T09:13:28.369657+0000","last_undegraded":"2024-04-02T09:13:28.369657+0000","last_fullsized":"2024-04-02T09:13:28.369657+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:17.740422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.030922+0000","last_change":"2024-04-02T09:13:28.379243+0000","last_active":"2024-04-02T09:13:31.030922+0000","last_peered":"2024-04-02T09:13:31.030922+0000","last_clean":"2024-04-02T09:13:31.030922+0000","last_became_active":"2024-04-02T09:13:28.378297+0000","last_became_peered":"2024-04-02T09:13:28.378297+0000","last_unstale":"2024-04-02T09:13:31.030922+0000","last_undegraded":"2024-04-02T09:13:31.030922+0000","last_fullsized":"2024-04-02T09:13:31.030922+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:02.761190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.369516+0000","last_change":"2024-04-02T09:13:27.876051+0000","last_active":"2024-04-02T09:13:28.369516+0000","last_peered":"2024-04-02T09:13:28.369516+0000","last_clean":"2024-04-02T09:13:28.369516+0000","last_became_active":"2024-04-02T09:13:27.875628+0000","last_became_peered":"2024-04-02T09:13:27.875628+0000","last_unstale":"2024-04-02T09:13:28.369516+0000","last_undegraded":"2024-04-02T09:13:28.369516+0000","last_fullsized":"2024-04-02T09:13:28.369516+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:53.750473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.365848+0000","last_change":"2024-04-02T09:13:28.383204+0000","last_active":"2024-04-02T09:13:29.365848+0000","last_peered":"2024-04-02T09:13:29.365848+0000","last_clean":"2024-04-02T09:13:29.365848+0000","last_became_active":"2024-04-02T09:13:28.382961+0000","last_became_peered":"2024-04-02T09:13:28.382961+0000","last_unstale":"2024-04-02T09:13:29.365848+0000","last_undegraded":"2024-04-02T09:13:29.365848+0000","last_fullsized":"2024-04-02T09:13:29.365848+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:16:14.265813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.7","version":"243'2","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367774+0000","last_change":"2024-04-02T09:13:27.886740+0000","last_active":"2024-04-02T09:13:29.367774+0000","last_peered":"2024-04-02T09:13:29.367774+0000","last_clean":"2024-04-02T09:13:29.367774+0000","last_became_active":"2024-04-02T09:13:27.886385+0000","last_became_peered":"2024-04-02T09:13:27.886385+0000","last_unstale":"2024-04-02T09:13:29.367774+0000","last_undegraded":"2024-04-02T09:13:29.367774+0000","last_fullsized":"2024-04-02T09:13:29.367774+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:20.383235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367723+0000","last_change":"2024-04-02T09:13:28.402497+0000","last_active":"2024-04-02T09:13:29.367723+0000","last_peered":"2024-04-02T09:13:29.367723+0000","last_clean":"2024-04-02T09:13:29.367723+0000","last_became_active":"2024-04-02T09:13:28.402235+0000","last_became_peered":"2024-04-02T09:13:28.402235+0000","last_unstale":"2024-04-02T09:13:29.367723+0000","last_undegraded":"2024-04-02T09:13:29.367723+0000","last_fullsized":"2024-04-02T09:13:29.367723+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:15.072346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.521037+0000","last_change":"2024-04-02T09:13:27.884225+0000","last_active":"2024-04-02T09:13:30.521037+0000","last_peered":"2024-04-02T09:13:30.521037+0000","last_clean":"2024-04-02T09:13:30.521037+0000","last_became_active":"2024-04-02T09:13:27.883536+0000","last_became_peered":"2024-04-02T09:13:27.883536+0000","last_unstale":"2024-04-02T09:13:30.521037+0000","last_undegraded":"2024-04-02T09:13:30.521037+0000","last_fullsized":"2024-04-02T09:13:30.521037+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:50.645932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.390464+0000","last_change":"2024-04-02T09:13:28.387139+0000","last_active":"2024-04-02T09:13:28.390464+0000","last_peered":"2024-04-02T09:13:28.390464+0000","last_clean":"2024-04-02T09:13:28.390464+0000","last_became_active":"2024-04-02T09:13:28.386878+0000","last_became_peered":"2024-04-02T09:13:28.386878+0000","last_unstale":"2024-04-02T09:13:28.390464+0000","last_undegraded":"2024-04-02T09:13:28.390464+0000","last_fullsized":"2024-04-02T09:13:28.390464+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:56:34.681485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.031185+0000","last_change":"2024-04-02T09:13:27.882615+0000","last_active":"2024-04-02T09:13:31.031185+0000","last_peered":"2024-04-02T09:13:31.031185+0000","last_clean":"2024-04-02T09:13:31.031185+0000","last_became_active":"2024-04-02T09:13:27.882334+0000","last_became_peered":"2024-04-02T09:13:27.882334+0000","last_unstale":"2024-04-02T09:13:31.031185+0000","last_undegraded":"2024-04-02T09:13:31.031185+0000","last_fullsized":"2024-04-02T09:13:31.031185+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:43.215347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367663+0000","last_change":"2024-04-02T09:13:28.402303+0000","last_active":"2024-04-02T09:13:29.367663+0000","last_peered":"2024-04-02T09:13:29.367663+0000","last_clean":"2024-04-02T09:13:29.367663+0000","last_became_active":"2024-04-02T09:13:28.399305+0000","last_became_peered":"2024-04-02T09:13:28.399305+0000","last_unstale":"2024-04-02T09:13:29.367663+0000","last_undegraded":"2024-04-02T09:13:29.367663+0000","last_fullsized":"2024-04-02T09:13:29.367663+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:58.927786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367158+0000","last_change":"2024-04-02T09:13:27.881268+0000","last_active":"2024-04-02T09:13:29.367158+0000","last_peered":"2024-04-02T09:13:29.367158+0000","last_clean":"2024-04-02T09:13:29.367158+0000","last_became_active":"2024-04-02T09:13:27.880983+0000","last_became_peered":"2024-04-02T09:13:27.880983+0000","last_unstale":"2024-04-02T09:13:29.367158+0000","last_undegraded":"2024-04-02T09:13:29.367158+0000","last_fullsized":"2024-04-02T09:13:29.367158+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:09.803095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520642+0000","last_change":"2024-04-02T09:13:28.374285+0000","last_active":"2024-04-02T09:13:30.520642+0000","last_peered":"2024-04-02T09:13:30.520642+0000","last_clean":"2024-04-02T09:13:30.520642+0000","last_became_active":"2024-04-02T09:13:28.373980+0000","last_became_peered":"2024-04-02T09:13:28.373980+0000","last_unstale":"2024-04-02T09:13:30.520642+0000","last_undegraded":"2024-04-02T09:13:30.520642+0000","last_fullsized":"2024-04-02T09:13:30.520642+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:56:34.681485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.365962+0000","last_change":"2024-04-02T09:13:27.882249+0000","last_active":"2024-04-02T09:13:29.365962+0000","last_peered":"2024-04-02T09:13:29.365962+0000","last_clean":"2024-04-02T09:13:29.365962+0000","last_became_active":"2024-04-02T09:13:27.881833+0000","last_became_peered":"2024-04-02T09:13:27.881833+0000","last_unstale":"2024-04-02T09:13:29.365962+0000","last_undegraded":"2024-04-02T09:13:29.365962+0000","last_fullsized":"2024-04-02T09:13:29.365962+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:22.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.031083+0000","last_change":"2024-04-02T09:13:28.389208+0000","last_active":"2024-04-02T09:13:31.031083+0000","last_peered":"2024-04-02T09:13:31.031083+0000","last_clean":"2024-04-02T09:13:31.031083+0000","last_became_active":"2024-04-02T09:13:28.388923+0000","last_became_peered":"2024-04-02T09:13:28.388923+0000","last_unstale":"2024-04-02T09:13:31.031083+0000","last_undegraded":"2024-04-02T09:13:31.031083+0000","last_fullsized":"2024-04-02T09:13:31.031083+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:35:55.299814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.369998+0000","last_change":"2024-04-02T09:13:27.887509+0000","last_active":"2024-04-02T09:13:28.369998+0000","last_peered":"2024-04-02T09:13:28.369998+0000","last_clean":"2024-04-02T09:13:28.369998+0000","last_became_active":"2024-04-02T09:13:27.887260+0000","last_became_peered":"2024-04-02T09:13:27.887260+0000","last_unstale":"2024-04-02T09:13:28.369998+0000","last_undegraded":"2024-04-02T09:13:28.369998+0000","last_fullsized":"2024-04-02T09:13:28.369998+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:28.581007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.375213+0000","last_change":"2024-04-02T09:13:28.374292+0000","last_active":"2024-04-02T09:13:28.375213+0000","last_peered":"2024-04-02T09:13:28.375213+0000","last_clean":"2024-04-02T09:13:28.375213+0000","last_became_active":"2024-04-02T09:13:28.373689+0000","last_became_peered":"2024-04-02T09:13:28.373689+0000","last_unstale":"2024-04-02T09:13:28.375213+0000","last_undegraded":"2024-04-02T09:13:28.375213+0000","last_fullsized":"2024-04-02T09:13:28.375213+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:08.953388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366809+0000","last_change":"2024-04-02T09:13:27.882492+0000","last_active":"2024-04-02T09:13:29.366809+0000","last_peered":"2024-04-02T09:13:29.366809+0000","last_clean":"2024-04-02T09:13:29.366809+0000","last_became_active":"2024-04-02T09:13:27.882217+0000","last_became_peered":"2024-04-02T09:13:27.882217+0000","last_unstale":"2024-04-02T09:13:29.366809+0000","last_undegraded":"2024-04-02T09:13:29.366809+0000","last_fullsized":"2024-04-02T09:13:29.366809+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:22:56.505431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366817+0000","last_change":"2024-04-02T09:13:28.381355+0000","last_active":"2024-04-02T09:13:29.366817+0000","last_peered":"2024-04-02T09:13:29.366817+0000","last_clean":"2024-04-02T09:13:29.366817+0000","last_became_active":"2024-04-02T09:13:28.381068+0000","last_became_peered":"2024-04-02T09:13:28.381068+0000","last_unstale":"2024-04-02T09:13:29.366817+0000","last_undegraded":"2024-04-02T09:13:29.366817+0000","last_fullsized":"2024-04-02T09:13:29.366817+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:50.895638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201945+0000","last_change":"2024-04-02T09:13:27.883425+0000","last_active":"2024-04-02T09:13:31.201945+0000","last_peered":"2024-04-02T09:13:31.201945+0000","last_clean":"2024-04-02T09:13:31.201945+0000","last_became_active":"2024-04-02T09:13:27.883106+0000","last_became_peered":"2024-04-02T09:13:27.883106+0000","last_unstale":"2024-04-02T09:13:31.201945+0000","last_undegraded":"2024-04-02T09:13:31.201945+0000","last_fullsized":"2024-04-02T09:13:31.201945+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:23.271534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.403396+0000","last_change":"2024-04-02T09:13:28.402819+0000","last_active":"2024-04-02T09:13:28.403396+0000","last_peered":"2024-04-02T09:13:28.403396+0000","last_clean":"2024-04-02T09:13:28.403396+0000","last_became_active":"2024-04-02T09:13:28.402563+0000","last_became_peered":"2024-04-02T09:13:28.402563+0000","last_unstale":"2024-04-02T09:13:28.403396+0000","last_undegraded":"2024-04-02T09:13:28.403396+0000","last_fullsized":"2024-04-02T09:13:28.403396+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:27.049697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1f","version":"243'4","reported_seq":15,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367814+0000","last_change":"2024-04-02T09:13:27.883417+0000","last_active":"2024-04-02T09:13:29.367814+0000","last_peered":"2024-04-02T09:13:29.367814+0000","last_clean":"2024-04-02T09:13:29.367814+0000","last_became_active":"2024-04-02T09:13:27.882944+0000","last_became_peered":"2024-04-02T09:13:27.882944+0000","last_unstale":"2024-04-02T09:13:29.367814+0000","last_undegraded":"2024-04-02T09:13:29.367814+0000","last_fullsized":"2024-04-02T09:13:29.367814+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:05.331734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"229'370","reported_seq":1308,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.377004+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:13:28.377004+0000","last_peered":"2024-04-02T09:13:28.377004+0000","last_clean":"2024-04-02T09:13:28.377004+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:13:28.377004+0000","last_undegraded":"2024-04-02T09:13:28.377004+0000","last_fullsized":"2024-04-02T09:13:28.377004+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575998,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60384,"kb_used_data":3020,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311456,"statfs":{"total":96636764160,"available":96574930944,"internally_reserved":0,"allocated":3092480,"data_stored":1572022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.40400000000000003,"5min":0.309,"15min":0.309},"max":{"1min":8.016,"5min":8.016,"15min":8.016},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.71999999999999997,"15min":0.72999999999999998},"min":{"1min":0.39900000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.1790000000000003,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.41199999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.76800000000000002,"15min":0.751},"min":{"1min":0.44800000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.5819999999999999,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.70899999999999996,"15min":0.71999999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.42399999999999999},"max":{"1min":4.2569999999999997,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.442}]},{"osd":2,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.79600000000000004,"15min":0.77800000000000002},"min":{"1min":0.437,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.9790000000000001,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.80700000000000005,"15min":0.81200000000000006},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":4.8819999999999997,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.622}]},{"osd":3,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.77100000000000002,"15min":0.75900000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.0530000000000008,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.999,"5min":0.79900000000000004,"15min":0.77700000000000002},"min":{"1min":0.54100000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":5.093,"5min":5.093,"15min":5.093},"last":0.66400000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.66700000000000004,"15min":0.628},"min":{"1min":0.38300000000000001,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":5.4589999999999996,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.1060000000000001,"5min":0.79700000000000004,"15min":0.74399999999999999},"min":{"1min":0.50600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.9409999999999998,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.64600000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.73799999999999999,"15min":0.69299999999999995},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":5.1340000000000003,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.77400000000000002,"15min":0.73599999999999999},"min":{"1min":0.58899999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.492,"5min":3.492,"15min":3.492},"last":0.67900000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.72199999999999998,"15min":0.69699999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45800000000000002},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.075,"5min":0.78500000000000003,"15min":0.71699999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41099999999999998},"max":{"1min":7.8899999999999997,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.73299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575996,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42308,"kb_used_data":2992,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329532,"statfs":{"total":96636764160,"available":96593440768,"internally_reserved":0,"allocated":3063808,"data_stored":1563184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.69799999999999995,"15min":0.72899999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.90600000000000003,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.71099999999999997,"15min":0.73899999999999999},"min":{"1min":0.53600000000000003,"5min":0.47899999999999998,"15min":0.45800000000000002},"max":{"1min":0.88300000000000001,"5min":0.93200000000000005,"15min":1.0600000000000001},"last":0.68100000000000005}]},{"osd":1,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68799999999999994,"15min":0.71299999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86299999999999999,"5min":1.044,"15min":1.212},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.71299999999999997,"15min":0.752},"min":{"1min":0.45800000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.018,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.56899999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.71799999999999997,"15min":0.747},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.046,"5min":1.046,"15min":1.2789999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.71999999999999997,"15min":0.73899999999999999},"min":{"1min":0.44,"5min":0.439,"15min":0.439},"max":{"1min":0.93500000000000005,"5min":0.95999999999999996,"15min":1.226},"last":0.58699999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72399999999999998,"15min":0.76300000000000001},"min":{"1min":0.53900000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97799999999999998,"5min":1.141,"15min":1.2549999999999999},"last":0.624},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.755,"15min":0.77000000000000002},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.747}]},{"osd":4,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.60999999999999999,"15min":0.624},"min":{"1min":0.442,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.91200000000000003,"5min":0.96699999999999997,"15min":1.1259999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.41999999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.1879999999999999},"last":0.80900000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84199999999999997,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69999999999999996,"15min":0.72499999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":0.96399999999999997,"5min":1.034,"15min":1.204},"last":0.71599999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.72999999999999998,"15min":0.74099999999999999},"min":{"1min":0.53600000000000003,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.012,"5min":1.21,"15min":1.2370000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72599999999999998,"15min":0.73899999999999999},"min":{"1min":0.52400000000000002,"5min":0.51400000000000001,"15min":0.49399999999999999},"max":{"1min":0.91900000000000004,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.73399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575997,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52760,"kb_used_data":3572,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319080,"statfs":{"total":96636764160,"available":96582737920,"internally_reserved":0,"allocated":3657728,"data_stored":2154044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.433,"15min":0.45000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.58999999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48999999999999999,"15min":0.50900000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73199999999999998,"5min":0.76000000000000001,"15min":0.79800000000000004},"last":0.54100000000000004}]},{"osd":2,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.504,"15min":0.505},"min":{"1min":0.38900000000000001,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56699999999999995,"15min":0.59199999999999997},"min":{"1min":0.44500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68100000000000005,"5min":0.77500000000000002,"15min":0.85599999999999998},"last":0.52900000000000003}]},{"osd":3,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.41099999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.69999999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.55200000000000005,"15min":0.56100000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.58899999999999997,"5min":0.86699999999999999,"15min":0.878},"last":0.505}]},{"osd":4,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76900000000000002,"15min":0.78400000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.433},"max":{"1min":0.90700000000000003,"5min":1.1120000000000001,"15min":1.1479999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.76400000000000001,"15min":0.78500000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.41799999999999998},"max":{"1min":0.95699999999999996,"5min":1.05,"15min":1.25},"last":0.66700000000000004}]},{"osd":5,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76100000000000001,"15min":0.78800000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.96399999999999997,"5min":1.115,"15min":1.321},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.79700000000000004,"15min":0.80700000000000005},"min":{"1min":0.56899999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93799999999999994,"5min":1.0980000000000001,"15min":1.173},"last":0.61599999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.79100000000000004,"15min":0.81299999999999994},"min":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.48099999999999998},"max":{"1min":0.96099999999999997,"5min":1.159,"15min":1.2210000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80800000000000005,"15min":0.80900000000000005},"min":{"1min":0.56799999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0009999999999999,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.77000000000000002}]},{"osd":7,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.79900000000000004,"15min":0.81999999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0029999999999999,"5min":1.1080000000000001,"15min":1.236},"last":0.755},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81899999999999995,"15min":0.83399999999999996},"min":{"1min":0.60899999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":0.97499999999999998,"5min":1.0720000000000001,"15min":1.2789999999999999},"last":0.84699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575997,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52172,"kb_used_data":3004,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319668,"statfs":{"total":96636764160,"available":96583340032,"internally_reserved":0,"allocated":3076096,"data_stored":1568915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.441,"15min":0.45500000000000002},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.66700000000000004,"5min":0.69899999999999995,"15min":0.86699999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.49399999999999999,"15min":0.50700000000000001},"min":{"1min":0.33900000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64800000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.41499999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.502,"15min":0.51100000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.82699999999999996},"last":0.621},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53400000000000003,"15min":0.55500000000000005},"min":{"1min":0.38300000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72699999999999998,"5min":0.999,"15min":0.999},"last":0.51200000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54200000000000004,"15min":0.56699999999999995},"min":{"1min":0.35699999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.71899999999999997,"5min":0.79800000000000004,"15min":0.85699999999999998},"last":0.495},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.61599999999999999,"15min":0.63800000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.73599999999999999,"5min":0.92000000000000004,"15min":0.96399999999999997},"last":0.66000000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.80500000000000005,"15min":0.83099999999999996},"min":{"1min":0.55000000000000004,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":0.97499999999999998,"5min":1.1950000000000001,"15min":1.2769999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.78500000000000003,"15min":0.82199999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99299999999999999,"5min":1.169,"15min":1.169},"last":0.56100000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.77900000000000003,"15min":0.82399999999999995},"min":{"1min":0.54000000000000004,"5min":0.497,"15min":0.497},"max":{"1min":1,"5min":1.145,"15min":1.232},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.96599999999999997,"5min":1.21,"15min":1.21},"last":0.69099999999999995}]},{"osd":6,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.77600000000000002,"15min":0.79800000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91100000000000003,"5min":1.1830000000000001,"15min":1.196},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.754,"5min":0.79200000000000004,"15min":0.81200000000000006},"min":{"1min":0.56499999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96199999999999997,"5min":1.246,"15min":1.246},"last":0.57799999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82699999999999996,"15min":0.85499999999999998},"min":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0069999999999999,"5min":1.292,"15min":1.292},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.84799999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.95499999999999996,"5min":1.1719999999999999,"15min":1.2589999999999999},"last":0.64800000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575996,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52168,"kb_used_data":3004,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319672,"statfs":{"total":96636764160,"available":96583344128,"internally_reserved":0,"allocated":3076096,"data_stored":1569912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.44600000000000001,"15min":0.44},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.72999999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.48399999999999999,"15min":0.498},"min":{"1min":0.29199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":1.4930000000000001},"last":0.501}]},{"osd":1,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.504},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69699999999999995,"5min":0.75800000000000001,"15min":0.80600000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53600000000000003,"15min":0.55400000000000005},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.59599999999999997,"5min":0.75,"15min":1.4530000000000001},"last":0.45000000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52900000000000003,"15min":0.55700000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.745,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60499999999999998,"15min":0.63200000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.64000000000000001,"5min":0.83699999999999997,"15min":1.0740000000000001},"last":0.52800000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.82099999999999995,"5min":1.121,"15min":1.121},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.77300000000000002,"15min":0.79100000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.96799999999999997,"5min":1.179,"15min":1.5329999999999999},"last":0.81399999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.78400000000000003,"15min":0.78600000000000003},"min":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.82999999999999996,"5min":1.99,"15min":1.99},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.78500000000000003,"15min":0.79900000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.89600000000000002,"5min":1.101,"15min":1.1259999999999999},"last":0.88300000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.997,"5min":1.173,"15min":1.5940000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":0.80400000000000005},"min":{"1min":0.56200000000000006,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.92300000000000004,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.70699999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.877,"5min":1.2110000000000001,"15min":1.5649999999999999},"last":0.876},{"interface":"front","average":{"1min":0.752,"5min":0.80700000000000005,"15min":0.82699999999999996},"min":{"1min":0.61499999999999999,"5min":0.497,"15min":0.497},"max":{"1min":0.95899999999999996,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.79800000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52172,"kb_used_data":3000,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319668,"statfs":{"total":96636764160,"available":96583340032,"internally_reserved":0,"allocated":3072000,"data_stored":1564658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.42799999999999999,"15min":0.44600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57199999999999995,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.46899999999999997,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60199999999999998,"5min":0.71199999999999997,"15min":0.876},"last":0.375}]},{"osd":1,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48699999999999999,"15min":0.499},"min":{"1min":0.36799999999999999,"5min":0.33200000000000002,"15min":0.32500000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.90300000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55100000000000005,"15min":0.57199999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67700000000000005,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.54900000000000004}]},{"osd":2,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.60099999999999998,"15min":0.626},"min":{"1min":0.40100000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71399999999999997,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.58799999999999997,"15min":0.61599999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.69499999999999995,"5min":0.81699999999999995,"15min":1.0840000000000001},"last":0.56899999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.72299999999999998,"15min":0.752},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89100000000000001,"5min":1.071,"15min":1.0800000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.74299999999999999,"15min":0.76800000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.879,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.73999999999999999}]},{"osd":5,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.74399999999999999,"15min":0.77500000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91800000000000004,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.755},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.76300000000000001,"15min":0.78100000000000003},"min":{"1min":0.54200000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.85699999999999998,"5min":1.266,"15min":1.266},"last":0.627}]},{"osd":6,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.76500000000000001,"15min":0.78400000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97099999999999997,"5min":1.284,"15min":1.284},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.54800000000000004,"5min":0.495,"15min":0.495},"max":{"1min":0.95999999999999996,"5min":1.2,"15min":1.2},"last":0.71199999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.77900000000000003,"15min":0.80800000000000005},"min":{"1min":0.58699999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.93500000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.79300000000000004,"15min":0.81799999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.014,"5min":1.248,"15min":1.248},"last":0.69799999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48080,"kb_used_data":3004,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323760,"statfs":{"total":96636764160,"available":96587530240,"internally_reserved":0,"allocated":3076096,"data_stored":1569864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.66400000000000003,"15min":0.68700000000000006},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":0.80000000000000004,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.65100000000000002,"15min":0.67700000000000005},"min":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.78100000000000003,"5min":0.86199999999999999,"15min":1.075},"last":0.47099999999999997}]},{"osd":1,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.72199999999999998,"15min":0.754},"min":{"1min":0.49399999999999999,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":0.90500000000000003,"5min":1.0549999999999999,"15min":1.069},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.73599999999999999,"15min":0.76500000000000001},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":0.92500000000000004,"5min":1.048,"15min":1.048},"last":0.63700000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.67300000000000004,"15min":0.70799999999999996},"min":{"1min":0.47099999999999997,"5min":0.438,"15min":0.438},"max":{"1min":0.81299999999999994,"5min":1.077,"15min":1.077},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.68999999999999995,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85599999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.52700000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.51500000000000001,"5min":0.501,"15min":0.47399999999999998},"max":{"1min":0.94599999999999995,"5min":0.998,"15min":1.018},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.028},"last":0.749}]},{"osd":5,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.81899999999999995,"5min":1.008,"15min":1.008},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.56100000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65500000000000003,"15min":0.64400000000000002},"min":{"1min":0.48499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85099999999999998,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68999999999999995,"15min":0.69199999999999995},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":1.0049999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.83099999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.48399999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96099999999999997,"5min":1.002,"15min":1.0329999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.77500000000000002,"15min":0.78100000000000003},"min":{"1min":0.56100000000000005,"5min":0.5,"15min":0.5},"max":{"1min":1.038,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.85199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575999,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":3572,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":3657728,"data_stored":2152567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69699999999999995,"15min":0.70999999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.89600000000000002,"5min":0.98699999999999999,"15min":1.004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69899999999999995,"15min":0.72499999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.76200000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.70099999999999996,"15min":0.72199999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85999999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.71399999999999997,"15min":0.73199999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.052},"last":0.66100000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73199999999999998,"15min":0.745},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93200000000000005,"5min":1.004,"15min":1.014},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72599999999999998,"15min":0.746},"min":{"1min":0.46600000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.018,"5min":1.018,"15min":1.0680000000000001},"last":0.63800000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77000000000000002,"15min":0.78600000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":0.98899999999999999,"15min":1.083},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.59099999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94399999999999995,"5min":1.008,"15min":1.1020000000000001},"last":0.58399999999999996}]},{"osd":4,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.629,"15min":0.629},"min":{"1min":0.56499999999999995,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.89100000000000001,"5min":0.96899999999999997,"15min":1.054},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.68400000000000005,"15min":0.67500000000000004},"min":{"1min":0.47999999999999998,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.89300000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.434,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71199999999999997,"15min":0.71099999999999997},"min":{"1min":0.501,"5min":0.44600000000000001,"15min":0.42699999999999999},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.74299999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.73999999999999999,"15min":0.73499999999999999},"min":{"1min":0.60699999999999998,"5min":0.441,"15min":0.441},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.64000000000000001,"5min":0.48199999999999998,"15min":0.45700000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:13:32.745 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:13:32.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.969+0000 7f7091000700 1 Processor -- start 2024-04-02T09:13:32.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.970+0000 7f7091000700 1 -- start start 2024-04-02T09:13:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.971+0000 7f7091000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f708c1619d0 0x7f708c1685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.971+0000 7f7091000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708c0d30d0 con 0x7f708c15d640 2024-04-02T09:13:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.971+0000 7f7091000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708c0b6840 con 0x7f708c0ceec0 2024-04-02T09:13:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.971+0000 7f7091000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f708c0d2b30 con 0x7f708c1619d0 2024-04-02T09:13:32.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.971+0000 7f7089e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f708c15d640 0x7f708c0d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56104/0 (socket says 172.21.15.89:56104) 2024-04-02T09:13:32.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.971+0000 7f7089e00700 1 -- 172.21.15.89:0/3240274466 learned_addr learned my addr 172.21.15.89:0/3240274466 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:32.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.971+0000 7f708a800700 1 --2- 172.21.15.89:0/3240274466 >> v2:172.21.15.113:3300/0 conn(0x7f708c1619d0 0x7f708c1685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.971+0000 7f7088a00700 1 -- 172.21.15.89:0/3240274466 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843325787 0 0) 0x7f708c0d30d0 con 0x7f708c15d640 2024-04-02T09:13:32.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.972+0000 7f7088a00700 1 -- 172.21.15.89:0/3240274466 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70700035b0 con 0x7f708c15d640 2024-04-02T09:13:32.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.972+0000 7f708a800700 1 -- 172.21.15.89:0/3240274466 >> v1:172.21.15.113:6789/0 conn(0x7f708c0ceec0 legacy=0x7f708c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.972+0000 7f708a800700 1 -- 172.21.15.89:0/3240274466 >> v1:172.21.15.89:6789/0 conn(0x7f708c15d640 legacy=0x7f708c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.972+0000 7f7088a00700 1 -- 172.21.15.89:0/3240274466 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2987445215 0 0) 0x7f70700035b0 con 0x7f708c15d640 2024-04-02T09:13:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.972+0000 7f708a800700 1 -- 172.21.15.89:0/3240274466 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708c0d2df0 con 0x7f708c1619d0 2024-04-02T09:13:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.972+0000 7f708a800700 1 --2- 172.21.15.89:0/3240274466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f708c1619d0 0x7f708c1685f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f7084009d90 tx=0x7f708400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d24f8a75fedbd053 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.972+0000 7f7088a00700 1 -- 172.21.15.89:0/3240274466 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708400e070 con 0x7f708c1619d0 2024-04-02T09:13:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.972+0000 7f7088a00700 1 -- 172.21.15.89:0/3240274466 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708400bb20 con 0x7f708c1619d0 2024-04-02T09:13:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.972+0000 7f7088a00700 1 -- 172.21.15.89:0/3240274466 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708400bed0 con 0x7f708c1619d0 2024-04-02T09:13:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.973+0000 7f7091000700 1 -- 172.21.15.89:0/3240274466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f708c1619d0 msgr2=0x7f708c1685f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.973+0000 7f7091000700 1 --2- 172.21.15.89:0/3240274466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f708c1619d0 0x7f708c1685f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f7084009d90 tx=0x7f708400b330 comp rx=0 tx=0).stop 2024-04-02T09:13:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.973+0000 7f7091000700 1 -- 172.21.15.89:0/3240274466 shutdown_connections 2024-04-02T09:13:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.973+0000 7f7091000700 1 --2- 172.21.15.89:0/3240274466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f708c1619d0 0x7f708c1685f0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.973+0000 7f7091000700 1 -- 172.21.15.89:0/3240274466 >> 172.21.15.89:0/3240274466 conn(0x7f708c08aff0 msgr2=0x7f708c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:32.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.973+0000 7f7091000700 1 -- 172.21.15.89:0/3240274466 shutdown_connections 2024-04-02T09:13:32.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.973+0000 7f7091000700 1 -- 172.21.15.89:0/3240274466 wait complete. 2024-04-02T09:13:32.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.975+0000 7f7091000700 1 Processor -- start 2024-04-02T09:13:32.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.975+0000 7f7091000700 1 -- start start 2024-04-02T09:13:32.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7091000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f708c0ceec0 0x7f708c1498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7091000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f708c15d640 0x7f708c159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7091000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f708c1619d0 0x7f708c160110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7091000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c159fc0 con 0x7f708c0ceec0 2024-04-02T09:13:32.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7091000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c0c7800 con 0x7f708c15d640 2024-04-02T09:13:32.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7091000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708c159ce0 con 0x7f708c1619d0 2024-04-02T09:13:32.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7089e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f708c0ceec0 0x7f708c1498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7089e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f708c0ceec0 0x7f708c1498e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60764/0 (socket says 172.21.15.89:60764) 2024-04-02T09:13:32.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7089e00700 1 -- 172.21.15.89:0/1225377147 learned_addr learned my addr 172.21.15.89:0/1225377147 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7089e00700 1 -- 172.21.15.89:0/1225377147 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f708c1619d0 msgr2=0x7f708c160110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7089400700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f708c15d640 0x7f708c159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f708a800700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f708c1619d0 0x7f708c160110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7089e00700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f708c1619d0 0x7f708c160110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7089e00700 1 -- 172.21.15.89:0/1225377147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f708c15d640 msgr2=0x7f708c159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7089e00700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f708c15d640 0x7f708c159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7089e00700 1 -- 172.21.15.89:0/1225377147 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7084009a70 con 0x7f708c0ceec0 2024-04-02T09:13:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f708a800700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f708c1619d0 0x7f708c160110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.976+0000 7f7089400700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f708c15d640 0x7f708c159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.977+0000 7f7089e00700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f708c0ceec0 0x7f708c1498e0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f707c00c7b0 tx=0x7f707c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.977+0000 7f7082000700 1 -- 172.21.15.89:0/1225377147 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c005d70 con 0x7f708c0ceec0 2024-04-02T09:13:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.977+0000 7f7082000700 1 -- 172.21.15.89:0/1225377147 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707c0070d0 con 0x7f708c0ceec0 2024-04-02T09:13:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.977+0000 7f7082000700 1 -- 172.21.15.89:0/1225377147 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c007460 con 0x7f708c0ceec0 2024-04-02T09:13:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.977+0000 7f7091000700 1 -- 172.21.15.89:0/1225377147 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f708c0c71b0 con 0x7f708c0ceec0 2024-04-02T09:13:32.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.977+0000 7f7091000700 1 -- 172.21.15.89:0/1225377147 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f708c149e20 con 0x7f708c0ceec0 2024-04-02T09:13:32.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.978+0000 7f7091000700 1 -- 172.21.15.89:0/1225377147 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f708c067720 con 0x7f708c0ceec0 2024-04-02T09:13:32.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.979+0000 7f7082000700 1 -- 172.21.15.89:0/1225377147 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f707c0075c0 con 0x7f708c0ceec0 2024-04-02T09:13:32.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.980+0000 7f7082000700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f706c06aed0 0x7f706c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:32.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.980+0000 7f7082000700 1 -- 172.21.15.89:0/1225377147 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f707c088f70 con 0x7f708c0ceec0 2024-04-02T09:13:32.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.980+0000 7f7089400700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f706c06aed0 0x7f706c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:32.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.981+0000 7f7089400700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f706c06aed0 0x7f706c06d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f708c06c780 tx=0x7f7078009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:32.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:32.984+0000 7f7082000700 1 -- 172.21.15.89:0/1225377147 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f707c053ba0 con 0x7f708c0ceec0 2024-04-02T09:13:33.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.182+0000 7f7091000700 1 -- 172.21.15.89:0/1225377147 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f708c0b7ce0 con 0x7f706c06aed0 2024-04-02T09:13:33.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.190+0000 7f7082000700 1 -- 172.21.15.89:0/1225377147 <== mgr.11613 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289489 (secure 0 0 0) 0x7f7078014420 con 0x7f706c06aed0 2024-04-02T09:13:33.193 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:33.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.195+0000 7f7091000700 1 -- 172.21.15.89:0/1225377147 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f706c06aed0 msgr2=0x7f706c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:33.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.195+0000 7f7091000700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f706c06aed0 0x7f706c06d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f708c06c780 tx=0x7f7078009040 comp rx=0 tx=0).stop 2024-04-02T09:13:33.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.195+0000 7f7091000700 1 -- 172.21.15.89:0/1225377147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f708c0ceec0 msgr2=0x7f708c1498e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:33.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.195+0000 7f7091000700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f708c0ceec0 0x7f708c1498e0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f707c00c7b0 tx=0x7f707c00cb70 comp rx=0 tx=0).stop 2024-04-02T09:13:33.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.195+0000 7f7091000700 1 -- 172.21.15.89:0/1225377147 shutdown_connections 2024-04-02T09:13:33.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.195+0000 7f7091000700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f708c1619d0 0x7f708c160110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.195+0000 7f7091000700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f708c0ceec0 0x7f708c1498e0 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.195+0000 7f7091000700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f706c06aed0 0x7f706c06d390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.195+0000 7f7091000700 1 --2- 172.21.15.89:0/1225377147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f708c15d640 0x7f708c159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.195+0000 7f7091000700 1 -- 172.21.15.89:0/1225377147 >> 172.21.15.89:0/1225377147 conn(0x7f708c08aff0 msgr2=0x7f708c06cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:33.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.195+0000 7f7091000700 1 -- 172.21.15.89:0/1225377147 shutdown_connections 2024-04-02T09:13:33.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.195+0000 7f7091000700 1 -- 172.21.15.89:0/1225377147 wait complete. 2024-04-02T09:13:33.200 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:13:33.212 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-04-02T09:13:31.429034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":663,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":412780,"kb_used_data":25168,"kb_used_omap":12,"kb_used_meta":387443,"kb_avail":754561940,"statfs":{"total":773094113280,"available":772671426560,"internally_reserved":0,"allocated":25772032,"data_stored":13715166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":396742200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.572919"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367610+0000","last_change":"2024-04-02T09:13:28.396642+0000","last_active":"2024-04-02T09:13:29.367610+0000","last_peered":"2024-04-02T09:13:29.367610+0000","last_clean":"2024-04-02T09:13:29.367610+0000","last_became_active":"2024-04-02T09:13:28.396356+0000","last_became_peered":"2024-04-02T09:13:28.396356+0000","last_unstale":"2024-04-02T09:13:29.367610+0000","last_undegraded":"2024-04-02T09:13:29.367610+0000","last_fullsized":"2024-04-02T09:13:29.367610+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:34.295480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201968+0000","last_change":"2024-04-02T09:13:28.381100+0000","last_active":"2024-04-02T09:13:31.201968+0000","last_peered":"2024-04-02T09:13:31.201968+0000","last_clean":"2024-04-02T09:13:31.201968+0000","last_became_active":"2024-04-02T09:13:28.380832+0000","last_became_peered":"2024-04-02T09:13:28.380832+0000","last_unstale":"2024-04-02T09:13:31.201968+0000","last_undegraded":"2024-04-02T09:13:31.201968+0000","last_fullsized":"2024-04-02T09:13:31.201968+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:21.987215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367599+0000","last_change":"2024-04-02T09:13:28.393942+0000","last_active":"2024-04-02T09:13:29.367599+0000","last_peered":"2024-04-02T09:13:29.367599+0000","last_clean":"2024-04-02T09:13:29.367599+0000","last_became_active":"2024-04-02T09:13:28.393467+0000","last_became_peered":"2024-04-02T09:13:28.393467+0000","last_unstale":"2024-04-02T09:13:29.367599+0000","last_undegraded":"2024-04-02T09:13:29.367599+0000","last_fullsized":"2024-04-02T09:13:29.367599+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:24.836500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367032+0000","last_change":"2024-04-02T09:13:28.394098+0000","last_active":"2024-04-02T09:13:29.367032+0000","last_peered":"2024-04-02T09:13:29.367032+0000","last_clean":"2024-04-02T09:13:29.367032+0000","last_became_active":"2024-04-02T09:13:28.393837+0000","last_became_peered":"2024-04-02T09:13:28.393837+0000","last_unstale":"2024-04-02T09:13:29.367032+0000","last_undegraded":"2024-04-02T09:13:29.367032+0000","last_fullsized":"2024-04-02T09:13:29.367032+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:40:27.068896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.030967+0000","last_change":"2024-04-02T09:13:28.379239+0000","last_active":"2024-04-02T09:13:31.030967+0000","last_peered":"2024-04-02T09:13:31.030967+0000","last_clean":"2024-04-02T09:13:31.030967+0000","last_became_active":"2024-04-02T09:13:28.378970+0000","last_became_peered":"2024-04-02T09:13:28.378970+0000","last_unstale":"2024-04-02T09:13:31.030967+0000","last_undegraded":"2024-04-02T09:13:31.030967+0000","last_fullsized":"2024-04-02T09:13:31.030967+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:37.960638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366504+0000","last_change":"2024-04-02T09:13:28.393747+0000","last_active":"2024-04-02T09:13:29.366504+0000","last_peered":"2024-04-02T09:13:29.366504+0000","last_clean":"2024-04-02T09:13:29.366504+0000","last_became_active":"2024-04-02T09:13:28.393515+0000","last_became_peered":"2024-04-02T09:13:28.393515+0000","last_unstale":"2024-04-02T09:13:29.366504+0000","last_undegraded":"2024-04-02T09:13:29.366504+0000","last_fullsized":"2024-04-02T09:13:29.366504+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:28:37.462844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.400699+0000","last_change":"2024-04-02T09:13:28.399975+0000","last_active":"2024-04-02T09:13:28.400699+0000","last_peered":"2024-04-02T09:13:28.400699+0000","last_clean":"2024-04-02T09:13:28.400699+0000","last_became_active":"2024-04-02T09:13:28.399335+0000","last_became_peered":"2024-04-02T09:13:28.399335+0000","last_unstale":"2024-04-02T09:13:28.400699+0000","last_undegraded":"2024-04-02T09:13:28.400699+0000","last_fullsized":"2024-04-02T09:13:28.400699+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:16:14.265813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.030889+0000","last_change":"2024-04-02T09:13:28.392551+0000","last_active":"2024-04-02T09:13:31.030889+0000","last_peered":"2024-04-02T09:13:31.030889+0000","last_clean":"2024-04-02T09:13:31.030889+0000","last_became_active":"2024-04-02T09:13:28.392159+0000","last_became_peered":"2024-04-02T09:13:28.392159+0000","last_unstale":"2024-04-02T09:13:31.030889+0000","last_undegraded":"2024-04-02T09:13:31.030889+0000","last_fullsized":"2024-04-02T09:13:31.030889+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:42.551696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520858+0000","last_change":"2024-04-02T09:13:28.383891+0000","last_active":"2024-04-02T09:13:30.520858+0000","last_peered":"2024-04-02T09:13:30.520858+0000","last_clean":"2024-04-02T09:13:30.520858+0000","last_became_active":"2024-04-02T09:13:28.383628+0000","last_became_peered":"2024-04-02T09:13:28.383628+0000","last_unstale":"2024-04-02T09:13:30.520858+0000","last_undegraded":"2024-04-02T09:13:30.520858+0000","last_fullsized":"2024-04-02T09:13:30.520858+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:44.222993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.405010+0000","last_change":"2024-04-02T09:13:28.404390+0000","last_active":"2024-04-02T09:13:28.405010+0000","last_peered":"2024-04-02T09:13:28.405010+0000","last_clean":"2024-04-02T09:13:28.405010+0000","last_became_active":"2024-04-02T09:13:28.404038+0000","last_became_peered":"2024-04-02T09:13:28.404038+0000","last_unstale":"2024-04-02T09:13:28.405010+0000","last_undegraded":"2024-04-02T09:13:28.405010+0000","last_fullsized":"2024-04-02T09:13:28.405010+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:47.359176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367663+0000","last_change":"2024-04-02T09:13:28.403517+0000","last_active":"2024-04-02T09:13:29.367663+0000","last_peered":"2024-04-02T09:13:29.367663+0000","last_clean":"2024-04-02T09:13:29.367663+0000","last_became_active":"2024-04-02T09:13:28.401353+0000","last_became_peered":"2024-04-02T09:13:28.401353+0000","last_unstale":"2024-04-02T09:13:29.367663+0000","last_undegraded":"2024-04-02T09:13:29.367663+0000","last_fullsized":"2024-04-02T09:13:29.367663+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:08.980542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.400667+0000","last_change":"2024-04-02T09:13:28.399938+0000","last_active":"2024-04-02T09:13:28.400667+0000","last_peered":"2024-04-02T09:13:28.400667+0000","last_clean":"2024-04-02T09:13:28.400667+0000","last_became_active":"2024-04-02T09:13:28.399350+0000","last_became_peered":"2024-04-02T09:13:28.399350+0000","last_unstale":"2024-04-02T09:13:28.400667+0000","last_undegraded":"2024-04-02T09:13:28.400667+0000","last_fullsized":"2024-04-02T09:13:28.400667+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:05.933699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520714+0000","last_change":"2024-04-02T09:13:28.386688+0000","last_active":"2024-04-02T09:13:30.520714+0000","last_peered":"2024-04-02T09:13:30.520714+0000","last_clean":"2024-04-02T09:13:30.520714+0000","last_became_active":"2024-04-02T09:13:28.386363+0000","last_became_peered":"2024-04-02T09:13:28.386363+0000","last_unstale":"2024-04-02T09:13:30.520714+0000","last_undegraded":"2024-04-02T09:13:30.520714+0000","last_fullsized":"2024-04-02T09:13:30.520714+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:12:41.480404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366464+0000","last_change":"2024-04-02T09:13:28.401903+0000","last_active":"2024-04-02T09:13:29.366464+0000","last_peered":"2024-04-02T09:13:29.366464+0000","last_clean":"2024-04-02T09:13:29.366464+0000","last_became_active":"2024-04-02T09:13:28.399098+0000","last_became_peered":"2024-04-02T09:13:28.399098+0000","last_unstale":"2024-04-02T09:13:29.366464+0000","last_undegraded":"2024-04-02T09:13:29.366464+0000","last_fullsized":"2024-04-02T09:13:29.366464+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:33:56.420978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.365755+0000","last_change":"2024-04-02T09:13:28.383146+0000","last_active":"2024-04-02T09:13:29.365755+0000","last_peered":"2024-04-02T09:13:29.365755+0000","last_clean":"2024-04-02T09:13:29.365755+0000","last_became_active":"2024-04-02T09:13:28.382879+0000","last_became_peered":"2024-04-02T09:13:28.382879+0000","last_unstale":"2024-04-02T09:13:29.365755+0000","last_undegraded":"2024-04-02T09:13:29.365755+0000","last_fullsized":"2024-04-02T09:13:29.365755+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:57:59.273299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201693+0000","last_change":"2024-04-02T09:13:28.381499+0000","last_active":"2024-04-02T09:13:31.201693+0000","last_peered":"2024-04-02T09:13:31.201693+0000","last_clean":"2024-04-02T09:13:31.201693+0000","last_became_active":"2024-04-02T09:13:28.377983+0000","last_became_peered":"2024-04-02T09:13:28.377983+0000","last_unstale":"2024-04-02T09:13:31.201693+0000","last_undegraded":"2024-04-02T09:13:31.201693+0000","last_fullsized":"2024-04-02T09:13:31.201693+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:45.021173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.385314+0000","last_change":"2024-04-02T09:13:28.382899+0000","last_active":"2024-04-02T09:13:28.385314+0000","last_peered":"2024-04-02T09:13:28.385314+0000","last_clean":"2024-04-02T09:13:28.385314+0000","last_became_active":"2024-04-02T09:13:28.382023+0000","last_became_peered":"2024-04-02T09:13:28.382023+0000","last_unstale":"2024-04-02T09:13:28.385314+0000","last_undegraded":"2024-04-02T09:13:28.385314+0000","last_fullsized":"2024-04-02T09:13:28.385314+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:12.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.394157+0000","last_change":"2024-04-02T09:13:28.390594+0000","last_active":"2024-04-02T09:13:28.394157+0000","last_peered":"2024-04-02T09:13:28.394157+0000","last_clean":"2024-04-02T09:13:28.394157+0000","last_became_active":"2024-04-02T09:13:28.388061+0000","last_became_peered":"2024-04-02T09:13:28.388061+0000","last_unstale":"2024-04-02T09:13:28.394157+0000","last_undegraded":"2024-04-02T09:13:28.394157+0000","last_fullsized":"2024-04-02T09:13:28.394157+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:14:04.841237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520986+0000","last_change":"2024-04-02T09:13:28.382883+0000","last_active":"2024-04-02T09:13:30.520986+0000","last_peered":"2024-04-02T09:13:30.520986+0000","last_clean":"2024-04-02T09:13:30.520986+0000","last_became_active":"2024-04-02T09:13:28.382187+0000","last_became_peered":"2024-04-02T09:13:28.382187+0000","last_unstale":"2024-04-02T09:13:30.520986+0000","last_undegraded":"2024-04-02T09:13:30.520986+0000","last_fullsized":"2024-04-02T09:13:30.520986+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:21:26.362248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201650+0000","last_change":"2024-04-02T09:13:28.384295+0000","last_active":"2024-04-02T09:13:31.201650+0000","last_peered":"2024-04-02T09:13:31.201650+0000","last_clean":"2024-04-02T09:13:31.201650+0000","last_became_active":"2024-04-02T09:13:28.384010+0000","last_became_peered":"2024-04-02T09:13:28.384010+0000","last_unstale":"2024-04-02T09:13:31.201650+0000","last_undegraded":"2024-04-02T09:13:31.201650+0000","last_fullsized":"2024-04-02T09:13:31.201650+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:47:33.241308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.030973+0000","last_change":"2024-04-02T09:13:28.380209+0000","last_active":"2024-04-02T09:13:31.030973+0000","last_peered":"2024-04-02T09:13:31.030973+0000","last_clean":"2024-04-02T09:13:31.030973+0000","last_became_active":"2024-04-02T09:13:28.379739+0000","last_became_peered":"2024-04-02T09:13:28.379739+0000","last_unstale":"2024-04-02T09:13:31.030973+0000","last_undegraded":"2024-04-02T09:13:31.030973+0000","last_fullsized":"2024-04-02T09:13:31.030973+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:46:14.163290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520758+0000","last_change":"2024-04-02T09:13:28.392993+0000","last_active":"2024-04-02T09:13:30.520758+0000","last_peered":"2024-04-02T09:13:30.520758+0000","last_clean":"2024-04-02T09:13:30.520758+0000","last_became_active":"2024-04-02T09:13:28.392712+0000","last_became_peered":"2024-04-02T09:13:28.392712+0000","last_unstale":"2024-04-02T09:13:30.520758+0000","last_undegraded":"2024-04-02T09:13:30.520758+0000","last_fullsized":"2024-04-02T09:13:30.520758+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:38.436370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520548+0000","last_change":"2024-04-02T09:13:28.378030+0000","last_active":"2024-04-02T09:13:30.520548+0000","last_peered":"2024-04-02T09:13:30.520548+0000","last_clean":"2024-04-02T09:13:30.520548+0000","last_became_active":"2024-04-02T09:13:28.377700+0000","last_became_peered":"2024-04-02T09:13:28.377700+0000","last_unstale":"2024-04-02T09:13:30.520548+0000","last_undegraded":"2024-04-02T09:13:30.520548+0000","last_fullsized":"2024-04-02T09:13:30.520548+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:40.236322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366260+0000","last_change":"2024-04-02T09:13:28.395364+0000","last_active":"2024-04-02T09:13:29.366260+0000","last_peered":"2024-04-02T09:13:29.366260+0000","last_clean":"2024-04-02T09:13:29.366260+0000","last_became_active":"2024-04-02T09:13:28.395100+0000","last_became_peered":"2024-04-02T09:13:28.395100+0000","last_unstale":"2024-04-02T09:13:29.366260+0000","last_undegraded":"2024-04-02T09:13:29.366260+0000","last_fullsized":"2024-04-02T09:13:29.366260+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:12.863676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367672+0000","last_change":"2024-04-02T09:13:28.399604+0000","last_active":"2024-04-02T09:13:29.367672+0000","last_peered":"2024-04-02T09:13:29.367672+0000","last_clean":"2024-04-02T09:13:29.367672+0000","last_became_active":"2024-04-02T09:13:28.399333+0000","last_became_peered":"2024-04-02T09:13:28.399333+0000","last_unstale":"2024-04-02T09:13:29.367672+0000","last_undegraded":"2024-04-02T09:13:29.367672+0000","last_fullsized":"2024-04-02T09:13:29.367672+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:18.748067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.031206+0000","last_change":"2024-04-02T09:13:28.378770+0000","last_active":"2024-04-02T09:13:31.031206+0000","last_peered":"2024-04-02T09:13:31.031206+0000","last_clean":"2024-04-02T09:13:31.031206+0000","last_became_active":"2024-04-02T09:13:28.378521+0000","last_became_peered":"2024-04-02T09:13:28.378521+0000","last_unstale":"2024-04-02T09:13:31.031206+0000","last_undegraded":"2024-04-02T09:13:31.031206+0000","last_fullsized":"2024-04-02T09:13:31.031206+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:24:11.210144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201827+0000","last_change":"2024-04-02T09:13:28.389084+0000","last_active":"2024-04-02T09:13:31.201827+0000","last_peered":"2024-04-02T09:13:31.201827+0000","last_clean":"2024-04-02T09:13:31.201827+0000","last_became_active":"2024-04-02T09:13:28.388624+0000","last_became_peered":"2024-04-02T09:13:28.388624+0000","last_unstale":"2024-04-02T09:13:31.201827+0000","last_undegraded":"2024-04-02T09:13:31.201827+0000","last_fullsized":"2024-04-02T09:13:31.201827+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:23.766461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366809+0000","last_change":"2024-04-02T09:13:28.395286+0000","last_active":"2024-04-02T09:13:29.366809+0000","last_peered":"2024-04-02T09:13:29.366809+0000","last_clean":"2024-04-02T09:13:29.366809+0000","last_became_active":"2024-04-02T09:13:28.394377+0000","last_became_peered":"2024-04-02T09:13:28.394377+0000","last_unstale":"2024-04-02T09:13:29.366809+0000","last_undegraded":"2024-04-02T09:13:29.366809+0000","last_fullsized":"2024-04-02T09:13:29.366809+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:37:39.028435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366453+0000","last_change":"2024-04-02T09:13:28.391267+0000","last_active":"2024-04-02T09:13:29.366453+0000","last_peered":"2024-04-02T09:13:29.366453+0000","last_clean":"2024-04-02T09:13:29.366453+0000","last_became_active":"2024-04-02T09:13:28.390985+0000","last_became_peered":"2024-04-02T09:13:28.390985+0000","last_unstale":"2024-04-02T09:13:29.366453+0000","last_undegraded":"2024-04-02T09:13:29.366453+0000","last_fullsized":"2024-04-02T09:13:29.366453+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:30.210394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520638+0000","last_change":"2024-04-02T09:13:28.380650+0000","last_active":"2024-04-02T09:13:30.520638+0000","last_peered":"2024-04-02T09:13:30.520638+0000","last_clean":"2024-04-02T09:13:30.520638+0000","last_became_active":"2024-04-02T09:13:28.380376+0000","last_became_peered":"2024-04-02T09:13:28.380376+0000","last_unstale":"2024-04-02T09:13:30.520638+0000","last_undegraded":"2024-04-02T09:13:30.520638+0000","last_fullsized":"2024-04-02T09:13:30.520638+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:46:17.713244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366034+0000","last_change":"2024-04-02T09:13:28.390273+0000","last_active":"2024-04-02T09:13:29.366034+0000","last_peered":"2024-04-02T09:13:29.366034+0000","last_clean":"2024-04-02T09:13:29.366034+0000","last_became_active":"2024-04-02T09:13:28.389990+0000","last_became_peered":"2024-04-02T09:13:28.389990+0000","last_unstale":"2024-04-02T09:13:29.366034+0000","last_undegraded":"2024-04-02T09:13:29.366034+0000","last_fullsized":"2024-04-02T09:13:29.366034+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:30:33.534225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201504+0000","last_change":"2024-04-02T09:13:28.395003+0000","last_active":"2024-04-02T09:13:31.201504+0000","last_peered":"2024-04-02T09:13:31.201504+0000","last_clean":"2024-04-02T09:13:31.201504+0000","last_became_active":"2024-04-02T09:13:28.394635+0000","last_became_peered":"2024-04-02T09:13:28.394635+0000","last_unstale":"2024-04-02T09:13:31.201504+0000","last_undegraded":"2024-04-02T09:13:31.201504+0000","last_fullsized":"2024-04-02T09:13:31.201504+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:54.758118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.393384+0000","last_change":"2024-04-02T09:13:28.391302+0000","last_active":"2024-04-02T09:13:28.393384+0000","last_peered":"2024-04-02T09:13:28.393384+0000","last_clean":"2024-04-02T09:13:28.393384+0000","last_became_active":"2024-04-02T09:13:28.388976+0000","last_became_peered":"2024-04-02T09:13:28.388976+0000","last_unstale":"2024-04-02T09:13:28.393384+0000","last_undegraded":"2024-04-02T09:13:28.393384+0000","last_fullsized":"2024-04-02T09:13:28.393384+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:02.556232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"243'1","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366012+0000","last_change":"2024-04-02T09:13:27.881269+0000","last_active":"2024-04-02T09:13:29.366012+0000","last_peered":"2024-04-02T09:13:29.366012+0000","last_clean":"2024-04-02T09:13:29.366012+0000","last_became_active":"2024-04-02T09:13:27.880664+0000","last_became_peered":"2024-04-02T09:13:27.880664+0000","last_unstale":"2024-04-02T09:13:29.366012+0000","last_undegraded":"2024-04-02T09:13:29.366012+0000","last_fullsized":"2024-04-02T09:13:29.366012+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:53.750473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366969+0000","last_change":"2024-04-02T09:13:28.399529+0000","last_active":"2024-04-02T09:13:29.366969+0000","last_peered":"2024-04-02T09:13:29.366969+0000","last_clean":"2024-04-02T09:13:29.366969+0000","last_became_active":"2024-04-02T09:13:28.399042+0000","last_became_peered":"2024-04-02T09:13:28.399042+0000","last_unstale":"2024-04-02T09:13:29.366969+0000","last_undegraded":"2024-04-02T09:13:29.366969+0000","last_fullsized":"2024-04-02T09:13:29.366969+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:58:19.109584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1d","version":"243'2","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367699+0000","last_change":"2024-04-02T09:13:27.889299+0000","last_active":"2024-04-02T09:13:29.367699+0000","last_peered":"2024-04-02T09:13:29.367699+0000","last_clean":"2024-04-02T09:13:29.367699+0000","last_became_active":"2024-04-02T09:13:27.888249+0000","last_became_peered":"2024-04-02T09:13:27.888249+0000","last_unstale":"2024-04-02T09:13:29.367699+0000","last_undegraded":"2024-04-02T09:13:29.367699+0000","last_fullsized":"2024-04-02T09:13:29.367699+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:01:31.866958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520985+0000","last_change":"2024-04-02T09:13:28.382423+0000","last_active":"2024-04-02T09:13:30.520985+0000","last_peered":"2024-04-02T09:13:30.520985+0000","last_clean":"2024-04-02T09:13:30.520985+0000","last_became_active":"2024-04-02T09:13:28.381840+0000","last_became_peered":"2024-04-02T09:13:28.381840+0000","last_unstale":"2024-04-02T09:13:30.520985+0000","last_undegraded":"2024-04-02T09:13:30.520985+0000","last_fullsized":"2024-04-02T09:13:30.520985+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:04:44.765954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"243'2","reported_seq":12,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.396509+0000","last_change":"2024-04-02T09:13:27.885629+0000","last_active":"2024-04-02T09:13:28.396509+0000","last_peered":"2024-04-02T09:13:28.396509+0000","last_clean":"2024-04-02T09:13:28.396509+0000","last_became_active":"2024-04-02T09:13:27.884921+0000","last_became_peered":"2024-04-02T09:13:27.884921+0000","last_unstale":"2024-04-02T09:13:28.396509+0000","last_undegraded":"2024-04-02T09:13:28.396509+0000","last_fullsized":"2024-04-02T09:13:28.396509+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:49:58.033026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.387545+0000","last_change":"2024-04-02T09:13:28.381558+0000","last_active":"2024-04-02T09:13:28.387545+0000","last_peered":"2024-04-02T09:13:28.387545+0000","last_clean":"2024-04-02T09:13:28.387545+0000","last_became_active":"2024-04-02T09:13:28.381306+0000","last_became_peered":"2024-04-02T09:13:28.381306+0000","last_unstale":"2024-04-02T09:13:28.387545+0000","last_undegraded":"2024-04-02T09:13:28.387545+0000","last_fullsized":"2024-04-02T09:13:28.387545+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:12:17.313507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1b","version":"243'1","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.397372+0000","last_change":"2024-04-02T09:13:27.884062+0000","last_active":"2024-04-02T09:13:28.397372+0000","last_peered":"2024-04-02T09:13:28.397372+0000","last_clean":"2024-04-02T09:13:28.397372+0000","last_became_active":"2024-04-02T09:13:27.883677+0000","last_became_peered":"2024-04-02T09:13:27.883677+0000","last_unstale":"2024-04-02T09:13:28.397372+0000","last_undegraded":"2024-04-02T09:13:28.397372+0000","last_fullsized":"2024-04-02T09:13:28.397372+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:14.774442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.365865+0000","last_change":"2024-04-02T09:13:28.389833+0000","last_active":"2024-04-02T09:13:29.365865+0000","last_peered":"2024-04-02T09:13:29.365865+0000","last_clean":"2024-04-02T09:13:29.365865+0000","last_became_active":"2024-04-02T09:13:28.389495+0000","last_became_peered":"2024-04-02T09:13:28.389495+0000","last_unstale":"2024-04-02T09:13:29.365865+0000","last_undegraded":"2024-04-02T09:13:29.365865+0000","last_fullsized":"2024-04-02T09:13:29.365865+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:27.049697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1a","version":"243'1","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.397106+0000","last_change":"2024-04-02T09:13:27.883532+0000","last_active":"2024-04-02T09:13:28.397106+0000","last_peered":"2024-04-02T09:13:28.397106+0000","last_clean":"2024-04-02T09:13:28.397106+0000","last_became_active":"2024-04-02T09:13:27.883252+0000","last_became_peered":"2024-04-02T09:13:27.883252+0000","last_unstale":"2024-04-02T09:13:28.397106+0000","last_undegraded":"2024-04-02T09:13:28.397106+0000","last_fullsized":"2024-04-02T09:13:28.397106+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.950371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366139+0000","last_change":"2024-04-02T09:13:28.403943+0000","last_active":"2024-04-02T09:13:29.366139+0000","last_peered":"2024-04-02T09:13:29.366139+0000","last_clean":"2024-04-02T09:13:29.366139+0000","last_became_active":"2024-04-02T09:13:28.402197+0000","last_became_peered":"2024-04-02T09:13:28.402197+0000","last_unstale":"2024-04-02T09:13:29.366139+0000","last_undegraded":"2024-04-02T09:13:29.366139+0000","last_fullsized":"2024-04-02T09:13:29.366139+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:50.579695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.030923+0000","last_change":"2024-04-02T09:13:27.883508+0000","last_active":"2024-04-02T09:13:31.030923+0000","last_peered":"2024-04-02T09:13:31.030923+0000","last_clean":"2024-04-02T09:13:31.030923+0000","last_became_active":"2024-04-02T09:13:27.883263+0000","last_became_peered":"2024-04-02T09:13:27.883263+0000","last_unstale":"2024-04-02T09:13:31.030923+0000","last_undegraded":"2024-04-02T09:13:31.030923+0000","last_fullsized":"2024-04-02T09:13:31.030923+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:37.428725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366154+0000","last_change":"2024-04-02T09:13:28.403768+0000","last_active":"2024-04-02T09:13:29.366154+0000","last_peered":"2024-04-02T09:13:29.366154+0000","last_clean":"2024-04-02T09:13:29.366154+0000","last_became_active":"2024-04-02T09:13:28.399745+0000","last_became_peered":"2024-04-02T09:13:28.399745+0000","last_unstale":"2024-04-02T09:13:29.366154+0000","last_undegraded":"2024-04-02T09:13:29.366154+0000","last_fullsized":"2024-04-02T09:13:29.366154+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:39.541325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.369408+0000","last_change":"2024-04-02T09:13:27.888598+0000","last_active":"2024-04-02T09:13:28.369408+0000","last_peered":"2024-04-02T09:13:28.369408+0000","last_clean":"2024-04-02T09:13:28.369408+0000","last_became_active":"2024-04-02T09:13:27.888115+0000","last_became_peered":"2024-04-02T09:13:27.888115+0000","last_unstale":"2024-04-02T09:13:28.369408+0000","last_undegraded":"2024-04-02T09:13:28.369408+0000","last_fullsized":"2024-04-02T09:13:28.369408+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:57:03.609779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367965+0000","last_change":"2024-04-02T09:13:28.404602+0000","last_active":"2024-04-02T09:13:29.367965+0000","last_peered":"2024-04-02T09:13:29.367965+0000","last_clean":"2024-04-02T09:13:29.367965+0000","last_became_active":"2024-04-02T09:13:28.403446+0000","last_became_peered":"2024-04-02T09:13:28.403446+0000","last_unstale":"2024-04-02T09:13:29.367965+0000","last_undegraded":"2024-04-02T09:13:29.367965+0000","last_fullsized":"2024-04-02T09:13:29.367965+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:24.279179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366051+0000","last_change":"2024-04-02T09:13:27.887286+0000","last_active":"2024-04-02T09:13:29.366051+0000","last_peered":"2024-04-02T09:13:29.366051+0000","last_clean":"2024-04-02T09:13:29.366051+0000","last_became_active":"2024-04-02T09:13:27.886999+0000","last_became_peered":"2024-04-02T09:13:27.886999+0000","last_unstale":"2024-04-02T09:13:29.366051+0000","last_undegraded":"2024-04-02T09:13:29.366051+0000","last_fullsized":"2024-04-02T09:13:29.366051+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:44.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.385852+0000","last_change":"2024-04-02T09:13:28.378767+0000","last_active":"2024-04-02T09:13:28.385852+0000","last_peered":"2024-04-02T09:13:28.385852+0000","last_clean":"2024-04-02T09:13:28.385852+0000","last_became_active":"2024-04-02T09:13:28.378487+0000","last_became_peered":"2024-04-02T09:13:28.378487+0000","last_unstale":"2024-04-02T09:13:28.385852+0000","last_undegraded":"2024-04-02T09:13:28.385852+0000","last_fullsized":"2024-04-02T09:13:28.385852+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:26:47.539777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.368012+0000","last_change":"2024-04-02T09:13:27.884469+0000","last_active":"2024-04-02T09:13:29.368012+0000","last_peered":"2024-04-02T09:13:29.368012+0000","last_clean":"2024-04-02T09:13:29.368012+0000","last_became_active":"2024-04-02T09:13:27.882570+0000","last_became_peered":"2024-04-02T09:13:27.882570+0000","last_unstale":"2024-04-02T09:13:29.368012+0000","last_undegraded":"2024-04-02T09:13:29.368012+0000","last_fullsized":"2024-04-02T09:13:29.368012+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:57:12.395680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201703+0000","last_change":"2024-04-02T09:13:28.387205+0000","last_active":"2024-04-02T09:13:31.201703+0000","last_peered":"2024-04-02T09:13:31.201703+0000","last_clean":"2024-04-02T09:13:31.201703+0000","last_became_active":"2024-04-02T09:13:28.384126+0000","last_became_peered":"2024-04-02T09:13:28.384126+0000","last_unstale":"2024-04-02T09:13:31.201703+0000","last_undegraded":"2024-04-02T09:13:31.201703+0000","last_fullsized":"2024-04-02T09:13:31.201703+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:52.330349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.15","version":"243'3","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201885+0000","last_change":"2024-04-02T09:13:27.883818+0000","last_active":"2024-04-02T09:13:31.201885+0000","last_peered":"2024-04-02T09:13:31.201885+0000","last_clean":"2024-04-02T09:13:31.201885+0000","last_became_active":"2024-04-02T09:13:27.883527+0000","last_became_peered":"2024-04-02T09:13:27.883527+0000","last_unstale":"2024-04-02T09:13:31.201885+0000","last_undegraded":"2024-04-02T09:13:31.201885+0000","last_fullsized":"2024-04-02T09:13:31.201885+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:49.572049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.031123+0000","last_change":"2024-04-02T09:13:28.379346+0000","last_active":"2024-04-02T09:13:31.031123+0000","last_peered":"2024-04-02T09:13:31.031123+0000","last_clean":"2024-04-02T09:13:31.031123+0000","last_became_active":"2024-04-02T09:13:28.378741+0000","last_became_peered":"2024-04-02T09:13:28.378741+0000","last_unstale":"2024-04-02T09:13:31.031123+0000","last_undegraded":"2024-04-02T09:13:31.031123+0000","last_fullsized":"2024-04-02T09:13:31.031123+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:44.092529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.14","version":"243'6","reported_seq":17,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367490+0000","last_change":"2024-04-02T09:13:27.885270+0000","last_active":"2024-04-02T09:13:29.367490+0000","last_peered":"2024-04-02T09:13:29.367490+0000","last_clean":"2024-04-02T09:13:29.367490+0000","last_became_active":"2024-04-02T09:13:27.884953+0000","last_became_peered":"2024-04-02T09:13:27.884953+0000","last_unstale":"2024-04-02T09:13:29.367490+0000","last_undegraded":"2024-04-02T09:13:29.367490+0000","last_fullsized":"2024-04-02T09:13:29.367490+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:48.103277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.365957+0000","last_change":"2024-04-02T09:13:28.393735+0000","last_active":"2024-04-02T09:13:29.365957+0000","last_peered":"2024-04-02T09:13:29.365957+0000","last_clean":"2024-04-02T09:13:29.365957+0000","last_became_active":"2024-04-02T09:13:28.393387+0000","last_became_peered":"2024-04-02T09:13:28.393387+0000","last_unstale":"2024-04-02T09:13:29.365957+0000","last_undegraded":"2024-04-02T09:13:29.365957+0000","last_fullsized":"2024-04-02T09:13:29.365957+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:47.359176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.13","version":"243'2","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367787+0000","last_change":"2024-04-02T09:13:27.885596+0000","last_active":"2024-04-02T09:13:29.367787+0000","last_peered":"2024-04-02T09:13:29.367787+0000","last_clean":"2024-04-02T09:13:29.367787+0000","last_became_active":"2024-04-02T09:13:27.885310+0000","last_became_peered":"2024-04-02T09:13:27.885310+0000","last_unstale":"2024-04-02T09:13:29.367787+0000","last_undegraded":"2024-04-02T09:13:29.367787+0000","last_fullsized":"2024-04-02T09:13:29.367787+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:02.497118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520825+0000","last_change":"2024-04-02T09:13:28.378384+0000","last_active":"2024-04-02T09:13:30.520825+0000","last_peered":"2024-04-02T09:13:30.520825+0000","last_clean":"2024-04-02T09:13:30.520825+0000","last_became_active":"2024-04-02T09:13:28.377833+0000","last_became_peered":"2024-04-02T09:13:28.377833+0000","last_unstale":"2024-04-02T09:13:30.520825+0000","last_undegraded":"2024-04-02T09:13:30.520825+0000","last_fullsized":"2024-04-02T09:13:30.520825+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:37:54.526506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366553+0000","last_change":"2024-04-02T09:13:27.880069+0000","last_active":"2024-04-02T09:13:29.366553+0000","last_peered":"2024-04-02T09:13:29.366553+0000","last_clean":"2024-04-02T09:13:29.366553+0000","last_became_active":"2024-04-02T09:13:27.879793+0000","last_became_peered":"2024-04-02T09:13:27.879793+0000","last_unstale":"2024-04-02T09:13:29.366553+0000","last_undegraded":"2024-04-02T09:13:29.366553+0000","last_fullsized":"2024-04-02T09:13:29.366553+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:10.198371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.396599+0000","last_change":"2024-04-02T09:13:28.394088+0000","last_active":"2024-04-02T09:13:28.396599+0000","last_peered":"2024-04-02T09:13:28.396599+0000","last_clean":"2024-04-02T09:13:28.396599+0000","last_became_active":"2024-04-02T09:13:28.392932+0000","last_became_peered":"2024-04-02T09:13:28.392932+0000","last_unstale":"2024-04-02T09:13:28.396599+0000","last_undegraded":"2024-04-02T09:13:28.396599+0000","last_fullsized":"2024-04-02T09:13:28.396599+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:54.683721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.030976+0000","last_change":"2024-04-02T09:13:27.881714+0000","last_active":"2024-04-02T09:13:31.030976+0000","last_peered":"2024-04-02T09:13:31.030976+0000","last_clean":"2024-04-02T09:13:31.030976+0000","last_became_active":"2024-04-02T09:13:27.881428+0000","last_became_peered":"2024-04-02T09:13:27.881428+0000","last_unstale":"2024-04-02T09:13:31.030976+0000","last_undegraded":"2024-04-02T09:13:31.030976+0000","last_fullsized":"2024-04-02T09:13:31.030976+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:50.081688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366047+0000","last_change":"2024-04-02T09:13:28.391188+0000","last_active":"2024-04-02T09:13:29.366047+0000","last_peered":"2024-04-02T09:13:29.366047+0000","last_clean":"2024-04-02T09:13:29.366047+0000","last_became_active":"2024-04-02T09:13:28.390867+0000","last_became_peered":"2024-04-02T09:13:28.390867+0000","last_unstale":"2024-04-02T09:13:29.366047+0000","last_undegraded":"2024-04-02T09:13:29.366047+0000","last_fullsized":"2024-04-02T09:13:29.366047+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:26:58.531401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.10","version":"243'8","reported_seq":19,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201677+0000","last_change":"2024-04-02T09:13:27.886503+0000","last_active":"2024-04-02T09:13:31.201677+0000","last_peered":"2024-04-02T09:13:31.201677+0000","last_clean":"2024-04-02T09:13:31.201677+0000","last_became_active":"2024-04-02T09:13:27.886121+0000","last_became_peered":"2024-04-02T09:13:27.886121+0000","last_unstale":"2024-04-02T09:13:31.201677+0000","last_undegraded":"2024-04-02T09:13:31.201677+0000","last_fullsized":"2024-04-02T09:13:31.201677+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:38.533680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.379586+0000","last_change":"2024-04-02T09:13:28.376609+0000","last_active":"2024-04-02T09:13:28.379586+0000","last_peered":"2024-04-02T09:13:28.379586+0000","last_clean":"2024-04-02T09:13:28.379586+0000","last_became_active":"2024-04-02T09:13:28.376226+0000","last_became_peered":"2024-04-02T09:13:28.376226+0000","last_unstale":"2024-04-02T09:13:28.379586+0000","last_undegraded":"2024-04-02T09:13:28.379586+0000","last_fullsized":"2024-04-02T09:13:28.379586+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:59.766007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"243'1","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366132+0000","last_change":"2024-04-02T09:13:27.883080+0000","last_active":"2024-04-02T09:13:29.366132+0000","last_peered":"2024-04-02T09:13:29.366132+0000","last_clean":"2024-04-02T09:13:29.366132+0000","last_became_active":"2024-04-02T09:13:27.882612+0000","last_became_peered":"2024-04-02T09:13:27.882612+0000","last_unstale":"2024-04-02T09:13:29.366132+0000","last_undegraded":"2024-04-02T09:13:29.366132+0000","last_fullsized":"2024-04-02T09:13:29.366132+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:20.979570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201892+0000","last_change":"2024-04-02T09:13:28.381499+0000","last_active":"2024-04-02T09:13:31.201892+0000","last_peered":"2024-04-02T09:13:31.201892+0000","last_clean":"2024-04-02T09:13:31.201892+0000","last_became_active":"2024-04-02T09:13:28.381286+0000","last_became_peered":"2024-04-02T09:13:28.381286+0000","last_unstale":"2024-04-02T09:13:31.201892+0000","last_undegraded":"2024-04-02T09:13:31.201892+0000","last_fullsized":"2024-04-02T09:13:31.201892+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:18.748067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.e","version":"243'2","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.368061+0000","last_change":"2024-04-02T09:13:27.888558+0000","last_active":"2024-04-02T09:13:29.368061+0000","last_peered":"2024-04-02T09:13:29.368061+0000","last_clean":"2024-04-02T09:13:29.368061+0000","last_became_active":"2024-04-02T09:13:27.888288+0000","last_became_peered":"2024-04-02T09:13:27.888288+0000","last_unstale":"2024-04-02T09:13:29.368061+0000","last_undegraded":"2024-04-02T09:13:29.368061+0000","last_fullsized":"2024-04-02T09:13:29.368061+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:15.386238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.365942+0000","last_change":"2024-04-02T09:13:28.372849+0000","last_active":"2024-04-02T09:13:29.365942+0000","last_peered":"2024-04-02T09:13:29.365942+0000","last_clean":"2024-04-02T09:13:29.365942+0000","last_became_active":"2024-04-02T09:13:28.372595+0000","last_became_peered":"2024-04-02T09:13:28.372595+0000","last_unstale":"2024-04-02T09:13:29.365942+0000","last_undegraded":"2024-04-02T09:13:29.365942+0000","last_fullsized":"2024-04-02T09:13:29.365942+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:12.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.d","version":"243'1","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366061+0000","last_change":"2024-04-02T09:13:27.879417+0000","last_active":"2024-04-02T09:13:29.366061+0000","last_peered":"2024-04-02T09:13:29.366061+0000","last_clean":"2024-04-02T09:13:29.366061+0000","last_became_active":"2024-04-02T09:13:27.879165+0000","last_became_peered":"2024-04-02T09:13:27.879165+0000","last_unstale":"2024-04-02T09:13:29.366061+0000","last_undegraded":"2024-04-02T09:13:29.366061+0000","last_fullsized":"2024-04-02T09:13:29.366061+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:47:32.233662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.393129+0000","last_change":"2024-04-02T09:13:28.392130+0000","last_active":"2024-04-02T09:13:28.393129+0000","last_peered":"2024-04-02T09:13:28.393129+0000","last_clean":"2024-04-02T09:13:28.393129+0000","last_became_active":"2024-04-02T09:13:28.391690+0000","last_became_peered":"2024-04-02T09:13:28.391690+0000","last_unstale":"2024-04-02T09:13:28.393129+0000","last_undegraded":"2024-04-02T09:13:28.393129+0000","last_fullsized":"2024-04-02T09:13:28.393129+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:24.319747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367068+0000","last_change":"2024-04-02T09:13:27.884781+0000","last_active":"2024-04-02T09:13:29.367068+0000","last_peered":"2024-04-02T09:13:29.367068+0000","last_clean":"2024-04-02T09:13:29.367068+0000","last_became_active":"2024-04-02T09:13:27.884507+0000","last_became_peered":"2024-04-02T09:13:27.884507+0000","last_unstale":"2024-04-02T09:13:29.367068+0000","last_undegraded":"2024-04-02T09:13:29.367068+0000","last_fullsized":"2024-04-02T09:13:29.367068+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:31:18.889273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520561+0000","last_change":"2024-04-02T09:13:28.381580+0000","last_active":"2024-04-02T09:13:30.520561+0000","last_peered":"2024-04-02T09:13:30.520561+0000","last_clean":"2024-04-02T09:13:30.520561+0000","last_became_active":"2024-04-02T09:13:28.381300+0000","last_became_peered":"2024-04-02T09:13:28.381300+0000","last_unstale":"2024-04-02T09:13:30.520561+0000","last_undegraded":"2024-04-02T09:13:30.520561+0000","last_fullsized":"2024-04-02T09:13:30.520561+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:51.089333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"243'2","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366666+0000","last_change":"2024-04-02T09:13:27.875291+0000","last_active":"2024-04-02T09:13:29.366666+0000","last_peered":"2024-04-02T09:13:29.366666+0000","last_clean":"2024-04-02T09:13:29.366666+0000","last_became_active":"2024-04-02T09:13:27.875003+0000","last_became_peered":"2024-04-02T09:13:27.875003+0000","last_unstale":"2024-04-02T09:13:29.366666+0000","last_undegraded":"2024-04-02T09:13:29.366666+0000","last_fullsized":"2024-04-02T09:13:29.366666+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:00.637981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366563+0000","last_change":"2024-04-02T09:13:28.404265+0000","last_active":"2024-04-02T09:13:29.366563+0000","last_peered":"2024-04-02T09:13:29.366563+0000","last_clean":"2024-04-02T09:13:29.366563+0000","last_became_active":"2024-04-02T09:13:28.403894+0000","last_became_peered":"2024-04-02T09:13:28.403894+0000","last_unstale":"2024-04-02T09:13:29.366563+0000","last_undegraded":"2024-04-02T09:13:29.366563+0000","last_fullsized":"2024-04-02T09:13:29.366563+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:05:23.433426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366765+0000","last_change":"2024-04-02T09:13:27.881604+0000","last_active":"2024-04-02T09:13:29.366765+0000","last_peered":"2024-04-02T09:13:29.366765+0000","last_clean":"2024-04-02T09:13:29.366765+0000","last_became_active":"2024-04-02T09:13:27.881304+0000","last_became_peered":"2024-04-02T09:13:27.881304+0000","last_unstale":"2024-04-02T09:13:29.366765+0000","last_undegraded":"2024-04-02T09:13:29.366765+0000","last_fullsized":"2024-04-02T09:13:29.366765+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:02.497118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.377099+0000","last_change":"2024-04-02T09:13:28.374960+0000","last_active":"2024-04-02T09:13:28.377099+0000","last_peered":"2024-04-02T09:13:28.377099+0000","last_clean":"2024-04-02T09:13:28.377099+0000","last_became_active":"2024-04-02T09:13:28.373683+0000","last_became_peered":"2024-04-02T09:13:28.373683+0000","last_unstale":"2024-04-02T09:13:28.377099+0000","last_undegraded":"2024-04-02T09:13:28.377099+0000","last_fullsized":"2024-04-02T09:13:28.377099+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:15.851301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.369657+0000","last_change":"2024-04-02T09:13:27.877837+0000","last_active":"2024-04-02T09:13:28.369657+0000","last_peered":"2024-04-02T09:13:28.369657+0000","last_clean":"2024-04-02T09:13:28.369657+0000","last_became_active":"2024-04-02T09:13:27.876792+0000","last_became_peered":"2024-04-02T09:13:27.876792+0000","last_unstale":"2024-04-02T09:13:28.369657+0000","last_undegraded":"2024-04-02T09:13:28.369657+0000","last_fullsized":"2024-04-02T09:13:28.369657+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:17.740422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.030922+0000","last_change":"2024-04-02T09:13:28.379243+0000","last_active":"2024-04-02T09:13:31.030922+0000","last_peered":"2024-04-02T09:13:31.030922+0000","last_clean":"2024-04-02T09:13:31.030922+0000","last_became_active":"2024-04-02T09:13:28.378297+0000","last_became_peered":"2024-04-02T09:13:28.378297+0000","last_unstale":"2024-04-02T09:13:31.030922+0000","last_undegraded":"2024-04-02T09:13:31.030922+0000","last_fullsized":"2024-04-02T09:13:31.030922+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:02.761190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.369516+0000","last_change":"2024-04-02T09:13:27.876051+0000","last_active":"2024-04-02T09:13:28.369516+0000","last_peered":"2024-04-02T09:13:28.369516+0000","last_clean":"2024-04-02T09:13:28.369516+0000","last_became_active":"2024-04-02T09:13:27.875628+0000","last_became_peered":"2024-04-02T09:13:27.875628+0000","last_unstale":"2024-04-02T09:13:28.369516+0000","last_undegraded":"2024-04-02T09:13:28.369516+0000","last_fullsized":"2024-04-02T09:13:28.369516+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:53.750473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.365848+0000","last_change":"2024-04-02T09:13:28.383204+0000","last_active":"2024-04-02T09:13:29.365848+0000","last_peered":"2024-04-02T09:13:29.365848+0000","last_clean":"2024-04-02T09:13:29.365848+0000","last_became_active":"2024-04-02T09:13:28.382961+0000","last_became_peered":"2024-04-02T09:13:28.382961+0000","last_unstale":"2024-04-02T09:13:29.365848+0000","last_undegraded":"2024-04-02T09:13:29.365848+0000","last_fullsized":"2024-04-02T09:13:29.365848+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:16:14.265813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.7","version":"243'2","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367774+0000","last_change":"2024-04-02T09:13:27.886740+0000","last_active":"2024-04-02T09:13:29.367774+0000","last_peered":"2024-04-02T09:13:29.367774+0000","last_clean":"2024-04-02T09:13:29.367774+0000","last_became_active":"2024-04-02T09:13:27.886385+0000","last_became_peered":"2024-04-02T09:13:27.886385+0000","last_unstale":"2024-04-02T09:13:29.367774+0000","last_undegraded":"2024-04-02T09:13:29.367774+0000","last_fullsized":"2024-04-02T09:13:29.367774+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:20.383235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367723+0000","last_change":"2024-04-02T09:13:28.402497+0000","last_active":"2024-04-02T09:13:29.367723+0000","last_peered":"2024-04-02T09:13:29.367723+0000","last_clean":"2024-04-02T09:13:29.367723+0000","last_became_active":"2024-04-02T09:13:28.402235+0000","last_became_peered":"2024-04-02T09:13:28.402235+0000","last_unstale":"2024-04-02T09:13:29.367723+0000","last_undegraded":"2024-04-02T09:13:29.367723+0000","last_fullsized":"2024-04-02T09:13:29.367723+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:15.072346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.521037+0000","last_change":"2024-04-02T09:13:27.884225+0000","last_active":"2024-04-02T09:13:30.521037+0000","last_peered":"2024-04-02T09:13:30.521037+0000","last_clean":"2024-04-02T09:13:30.521037+0000","last_became_active":"2024-04-02T09:13:27.883536+0000","last_became_peered":"2024-04-02T09:13:27.883536+0000","last_unstale":"2024-04-02T09:13:30.521037+0000","last_undegraded":"2024-04-02T09:13:30.521037+0000","last_fullsized":"2024-04-02T09:13:30.521037+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:50.645932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.390464+0000","last_change":"2024-04-02T09:13:28.387139+0000","last_active":"2024-04-02T09:13:28.390464+0000","last_peered":"2024-04-02T09:13:28.390464+0000","last_clean":"2024-04-02T09:13:28.390464+0000","last_became_active":"2024-04-02T09:13:28.386878+0000","last_became_peered":"2024-04-02T09:13:28.386878+0000","last_unstale":"2024-04-02T09:13:28.390464+0000","last_undegraded":"2024-04-02T09:13:28.390464+0000","last_fullsized":"2024-04-02T09:13:28.390464+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:56:34.681485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.031185+0000","last_change":"2024-04-02T09:13:27.882615+0000","last_active":"2024-04-02T09:13:31.031185+0000","last_peered":"2024-04-02T09:13:31.031185+0000","last_clean":"2024-04-02T09:13:31.031185+0000","last_became_active":"2024-04-02T09:13:27.882334+0000","last_became_peered":"2024-04-02T09:13:27.882334+0000","last_unstale":"2024-04-02T09:13:31.031185+0000","last_undegraded":"2024-04-02T09:13:31.031185+0000","last_fullsized":"2024-04-02T09:13:31.031185+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:43.215347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367663+0000","last_change":"2024-04-02T09:13:28.402303+0000","last_active":"2024-04-02T09:13:29.367663+0000","last_peered":"2024-04-02T09:13:29.367663+0000","last_clean":"2024-04-02T09:13:29.367663+0000","last_became_active":"2024-04-02T09:13:28.399305+0000","last_became_peered":"2024-04-02T09:13:28.399305+0000","last_unstale":"2024-04-02T09:13:29.367663+0000","last_undegraded":"2024-04-02T09:13:29.367663+0000","last_fullsized":"2024-04-02T09:13:29.367663+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:58.927786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367158+0000","last_change":"2024-04-02T09:13:27.881268+0000","last_active":"2024-04-02T09:13:29.367158+0000","last_peered":"2024-04-02T09:13:29.367158+0000","last_clean":"2024-04-02T09:13:29.367158+0000","last_became_active":"2024-04-02T09:13:27.880983+0000","last_became_peered":"2024-04-02T09:13:27.880983+0000","last_unstale":"2024-04-02T09:13:29.367158+0000","last_undegraded":"2024-04-02T09:13:29.367158+0000","last_fullsized":"2024-04-02T09:13:29.367158+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:09.803095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:30.520642+0000","last_change":"2024-04-02T09:13:28.374285+0000","last_active":"2024-04-02T09:13:30.520642+0000","last_peered":"2024-04-02T09:13:30.520642+0000","last_clean":"2024-04-02T09:13:30.520642+0000","last_became_active":"2024-04-02T09:13:28.373980+0000","last_became_peered":"2024-04-02T09:13:28.373980+0000","last_unstale":"2024-04-02T09:13:30.520642+0000","last_undegraded":"2024-04-02T09:13:30.520642+0000","last_fullsized":"2024-04-02T09:13:30.520642+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:56:34.681485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.365962+0000","last_change":"2024-04-02T09:13:27.882249+0000","last_active":"2024-04-02T09:13:29.365962+0000","last_peered":"2024-04-02T09:13:29.365962+0000","last_clean":"2024-04-02T09:13:29.365962+0000","last_became_active":"2024-04-02T09:13:27.881833+0000","last_became_peered":"2024-04-02T09:13:27.881833+0000","last_unstale":"2024-04-02T09:13:29.365962+0000","last_undegraded":"2024-04-02T09:13:29.365962+0000","last_fullsized":"2024-04-02T09:13:29.365962+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:22.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.031083+0000","last_change":"2024-04-02T09:13:28.389208+0000","last_active":"2024-04-02T09:13:31.031083+0000","last_peered":"2024-04-02T09:13:31.031083+0000","last_clean":"2024-04-02T09:13:31.031083+0000","last_became_active":"2024-04-02T09:13:28.388923+0000","last_became_peered":"2024-04-02T09:13:28.388923+0000","last_unstale":"2024-04-02T09:13:31.031083+0000","last_undegraded":"2024-04-02T09:13:31.031083+0000","last_fullsized":"2024-04-02T09:13:31.031083+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:35:55.299814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.369998+0000","last_change":"2024-04-02T09:13:27.887509+0000","last_active":"2024-04-02T09:13:28.369998+0000","last_peered":"2024-04-02T09:13:28.369998+0000","last_clean":"2024-04-02T09:13:28.369998+0000","last_became_active":"2024-04-02T09:13:27.887260+0000","last_became_peered":"2024-04-02T09:13:27.887260+0000","last_unstale":"2024-04-02T09:13:28.369998+0000","last_undegraded":"2024-04-02T09:13:28.369998+0000","last_fullsized":"2024-04-02T09:13:28.369998+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:28.581007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.375213+0000","last_change":"2024-04-02T09:13:28.374292+0000","last_active":"2024-04-02T09:13:28.375213+0000","last_peered":"2024-04-02T09:13:28.375213+0000","last_clean":"2024-04-02T09:13:28.375213+0000","last_became_active":"2024-04-02T09:13:28.373689+0000","last_became_peered":"2024-04-02T09:13:28.373689+0000","last_unstale":"2024-04-02T09:13:28.375213+0000","last_undegraded":"2024-04-02T09:13:28.375213+0000","last_fullsized":"2024-04-02T09:13:28.375213+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:08.953388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366809+0000","last_change":"2024-04-02T09:13:27.882492+0000","last_active":"2024-04-02T09:13:29.366809+0000","last_peered":"2024-04-02T09:13:29.366809+0000","last_clean":"2024-04-02T09:13:29.366809+0000","last_became_active":"2024-04-02T09:13:27.882217+0000","last_became_peered":"2024-04-02T09:13:27.882217+0000","last_unstale":"2024-04-02T09:13:29.366809+0000","last_undegraded":"2024-04-02T09:13:29.366809+0000","last_fullsized":"2024-04-02T09:13:29.366809+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:22:56.505431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":10,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.366817+0000","last_change":"2024-04-02T09:13:28.381355+0000","last_active":"2024-04-02T09:13:29.366817+0000","last_peered":"2024-04-02T09:13:29.366817+0000","last_clean":"2024-04-02T09:13:29.366817+0000","last_became_active":"2024-04-02T09:13:28.381068+0000","last_became_peered":"2024-04-02T09:13:28.381068+0000","last_unstale":"2024-04-02T09:13:29.366817+0000","last_undegraded":"2024-04-02T09:13:29.366817+0000","last_fullsized":"2024-04-02T09:13:29.366817+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:50.895638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.201945+0000","last_change":"2024-04-02T09:13:27.883425+0000","last_active":"2024-04-02T09:13:31.201945+0000","last_peered":"2024-04-02T09:13:31.201945+0000","last_clean":"2024-04-02T09:13:31.201945+0000","last_became_active":"2024-04-02T09:13:27.883106+0000","last_became_peered":"2024-04-02T09:13:27.883106+0000","last_unstale":"2024-04-02T09:13:31.201945+0000","last_undegraded":"2024-04-02T09:13:31.201945+0000","last_fullsized":"2024-04-02T09:13:31.201945+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:23.271534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":9,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.403396+0000","last_change":"2024-04-02T09:13:28.402819+0000","last_active":"2024-04-02T09:13:28.403396+0000","last_peered":"2024-04-02T09:13:28.403396+0000","last_clean":"2024-04-02T09:13:28.403396+0000","last_became_active":"2024-04-02T09:13:28.402563+0000","last_became_peered":"2024-04-02T09:13:28.402563+0000","last_unstale":"2024-04-02T09:13:28.403396+0000","last_undegraded":"2024-04-02T09:13:28.403396+0000","last_fullsized":"2024-04-02T09:13:28.403396+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:27.049697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1f","version":"243'4","reported_seq":15,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:29.367814+0000","last_change":"2024-04-02T09:13:27.883417+0000","last_active":"2024-04-02T09:13:29.367814+0000","last_peered":"2024-04-02T09:13:29.367814+0000","last_clean":"2024-04-02T09:13:29.367814+0000","last_became_active":"2024-04-02T09:13:27.882944+0000","last_became_peered":"2024-04-02T09:13:27.882944+0000","last_unstale":"2024-04-02T09:13:29.367814+0000","last_undegraded":"2024-04-02T09:13:29.367814+0000","last_fullsized":"2024-04-02T09:13:29.367814+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:05.331734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"229'370","reported_seq":1308,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-04-02T09:13:28.377004+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:13:28.377004+0000","last_peered":"2024-04-02T09:13:28.377004+0000","last_clean":"2024-04-02T09:13:28.377004+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:13:28.377004+0000","last_undegraded":"2024-04-02T09:13:28.377004+0000","last_fullsized":"2024-04-02T09:13:28.377004+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575998,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60384,"kb_used_data":3020,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311456,"statfs":{"total":96636764160,"available":96574930944,"internally_reserved":0,"allocated":3092480,"data_stored":1572022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.40400000000000003,"5min":0.309,"15min":0.309},"max":{"1min":8.016,"5min":8.016,"15min":8.016},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.71999999999999997,"15min":0.72999999999999998},"min":{"1min":0.39900000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.1790000000000003,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.41199999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.76800000000000002,"15min":0.751},"min":{"1min":0.44800000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.5819999999999999,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.70899999999999996,"15min":0.71999999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.42399999999999999},"max":{"1min":4.2569999999999997,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.442}]},{"osd":2,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.79600000000000004,"15min":0.77800000000000002},"min":{"1min":0.437,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.9790000000000001,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.80700000000000005,"15min":0.81200000000000006},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":4.8819999999999997,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.622}]},{"osd":3,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.77100000000000002,"15min":0.75900000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.0530000000000008,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.999,"5min":0.79900000000000004,"15min":0.77700000000000002},"min":{"1min":0.54100000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":5.093,"5min":5.093,"15min":5.093},"last":0.66400000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.66700000000000004,"15min":0.628},"min":{"1min":0.38300000000000001,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":5.4589999999999996,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.1060000000000001,"5min":0.79700000000000004,"15min":0.74399999999999999},"min":{"1min":0.50600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.9409999999999998,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.64600000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.73799999999999999,"15min":0.69299999999999995},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":5.1340000000000003,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.77400000000000002,"15min":0.73599999999999999},"min":{"1min":0.58899999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.492,"5min":3.492,"15min":3.492},"last":0.67900000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.72199999999999998,"15min":0.69699999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45800000000000002},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.075,"5min":0.78500000000000003,"15min":0.71699999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41099999999999998},"max":{"1min":7.8899999999999997,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.73299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575996,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42308,"kb_used_data":2992,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329532,"statfs":{"total":96636764160,"available":96593440768,"internally_reserved":0,"allocated":3063808,"data_stored":1563184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.69799999999999995,"15min":0.72899999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.90600000000000003,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.71099999999999997,"15min":0.73899999999999999},"min":{"1min":0.53600000000000003,"5min":0.47899999999999998,"15min":0.45800000000000002},"max":{"1min":0.88300000000000001,"5min":0.93200000000000005,"15min":1.0600000000000001},"last":0.68100000000000005}]},{"osd":1,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68799999999999994,"15min":0.71299999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86299999999999999,"5min":1.044,"15min":1.212},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.71299999999999997,"15min":0.752},"min":{"1min":0.45800000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.018,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.56899999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.71799999999999997,"15min":0.747},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.046,"5min":1.046,"15min":1.2789999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.71999999999999997,"15min":0.73899999999999999},"min":{"1min":0.44,"5min":0.439,"15min":0.439},"max":{"1min":0.93500000000000005,"5min":0.95999999999999996,"15min":1.226},"last":0.58699999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72399999999999998,"15min":0.76300000000000001},"min":{"1min":0.53900000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97799999999999998,"5min":1.141,"15min":1.2549999999999999},"last":0.624},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.755,"15min":0.77000000000000002},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.747}]},{"osd":4,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.60999999999999999,"15min":0.624},"min":{"1min":0.442,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.91200000000000003,"5min":0.96699999999999997,"15min":1.1259999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.41999999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.1879999999999999},"last":0.80900000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84199999999999997,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69999999999999996,"15min":0.72499999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":0.96399999999999997,"5min":1.034,"15min":1.204},"last":0.71599999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.72999999999999998,"15min":0.74099999999999999},"min":{"1min":0.53600000000000003,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.012,"5min":1.21,"15min":1.2370000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72599999999999998,"15min":0.73899999999999999},"min":{"1min":0.52400000000000002,"5min":0.51400000000000001,"15min":0.49399999999999999},"max":{"1min":0.91900000000000004,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.73399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575997,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52760,"kb_used_data":3572,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319080,"statfs":{"total":96636764160,"available":96582737920,"internally_reserved":0,"allocated":3657728,"data_stored":2154044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.433,"15min":0.45000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.58999999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48999999999999999,"15min":0.50900000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73199999999999998,"5min":0.76000000000000001,"15min":0.79800000000000004},"last":0.54100000000000004}]},{"osd":2,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.504,"15min":0.505},"min":{"1min":0.38900000000000001,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56699999999999995,"15min":0.59199999999999997},"min":{"1min":0.44500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68100000000000005,"5min":0.77500000000000002,"15min":0.85599999999999998},"last":0.52900000000000003}]},{"osd":3,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.41099999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.69999999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.55200000000000005,"15min":0.56100000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.58899999999999997,"5min":0.86699999999999999,"15min":0.878},"last":0.505}]},{"osd":4,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76900000000000002,"15min":0.78400000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.433},"max":{"1min":0.90700000000000003,"5min":1.1120000000000001,"15min":1.1479999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.76400000000000001,"15min":0.78500000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.41799999999999998},"max":{"1min":0.95699999999999996,"5min":1.05,"15min":1.25},"last":0.66700000000000004}]},{"osd":5,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76100000000000001,"15min":0.78800000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.96399999999999997,"5min":1.115,"15min":1.321},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.79700000000000004,"15min":0.80700000000000005},"min":{"1min":0.56899999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93799999999999994,"5min":1.0980000000000001,"15min":1.173},"last":0.61599999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.79100000000000004,"15min":0.81299999999999994},"min":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.48099999999999998},"max":{"1min":0.96099999999999997,"5min":1.159,"15min":1.2210000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80800000000000005,"15min":0.80900000000000005},"min":{"1min":0.56799999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0009999999999999,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.77000000000000002}]},{"osd":7,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.79900000000000004,"15min":0.81999999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0029999999999999,"5min":1.1080000000000001,"15min":1.236},"last":0.755},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81899999999999995,"15min":0.83399999999999996},"min":{"1min":0.60899999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":0.97499999999999998,"5min":1.0720000000000001,"15min":1.2789999999999999},"last":0.84699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575997,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52172,"kb_used_data":3004,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319668,"statfs":{"total":96636764160,"available":96583340032,"internally_reserved":0,"allocated":3076096,"data_stored":1568915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.441,"15min":0.45500000000000002},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.66700000000000004,"5min":0.69899999999999995,"15min":0.86699999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.49399999999999999,"15min":0.50700000000000001},"min":{"1min":0.33900000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64800000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.41499999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.502,"15min":0.51100000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.82699999999999996},"last":0.621},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53400000000000003,"15min":0.55500000000000005},"min":{"1min":0.38300000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72699999999999998,"5min":0.999,"15min":0.999},"last":0.51200000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54200000000000004,"15min":0.56699999999999995},"min":{"1min":0.35699999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.71899999999999997,"5min":0.79800000000000004,"15min":0.85699999999999998},"last":0.495},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.61599999999999999,"15min":0.63800000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.73599999999999999,"5min":0.92000000000000004,"15min":0.96399999999999997},"last":0.66000000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.80500000000000005,"15min":0.83099999999999996},"min":{"1min":0.55000000000000004,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":0.97499999999999998,"5min":1.1950000000000001,"15min":1.2769999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.78500000000000003,"15min":0.82199999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99299999999999999,"5min":1.169,"15min":1.169},"last":0.56100000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.77900000000000003,"15min":0.82399999999999995},"min":{"1min":0.54000000000000004,"5min":0.497,"15min":0.497},"max":{"1min":1,"5min":1.145,"15min":1.232},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.96599999999999997,"5min":1.21,"15min":1.21},"last":0.69099999999999995}]},{"osd":6,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.77600000000000002,"15min":0.79800000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91100000000000003,"5min":1.1830000000000001,"15min":1.196},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.754,"5min":0.79200000000000004,"15min":0.81200000000000006},"min":{"1min":0.56499999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96199999999999997,"5min":1.246,"15min":1.246},"last":0.57799999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82699999999999996,"15min":0.85499999999999998},"min":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0069999999999999,"5min":1.292,"15min":1.292},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.84799999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.95499999999999996,"5min":1.1719999999999999,"15min":1.2589999999999999},"last":0.64800000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575996,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52168,"kb_used_data":3004,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319672,"statfs":{"total":96636764160,"available":96583344128,"internally_reserved":0,"allocated":3076096,"data_stored":1569912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.44600000000000001,"15min":0.44},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.72999999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.48399999999999999,"15min":0.498},"min":{"1min":0.29199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":1.4930000000000001},"last":0.501}]},{"osd":1,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.504},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69699999999999995,"5min":0.75800000000000001,"15min":0.80600000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53600000000000003,"15min":0.55400000000000005},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.59599999999999997,"5min":0.75,"15min":1.4530000000000001},"last":0.45000000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52900000000000003,"15min":0.55700000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.745,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60499999999999998,"15min":0.63200000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.64000000000000001,"5min":0.83699999999999997,"15min":1.0740000000000001},"last":0.52800000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.82099999999999995,"5min":1.121,"15min":1.121},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.77300000000000002,"15min":0.79100000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.96799999999999997,"5min":1.179,"15min":1.5329999999999999},"last":0.81399999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.78400000000000003,"15min":0.78600000000000003},"min":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.82999999999999996,"5min":1.99,"15min":1.99},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.78500000000000003,"15min":0.79900000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.89600000000000002,"5min":1.101,"15min":1.1259999999999999},"last":0.88300000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.997,"5min":1.173,"15min":1.5940000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":0.80400000000000005},"min":{"1min":0.56200000000000006,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.92300000000000004,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.70699999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.877,"5min":1.2110000000000001,"15min":1.5649999999999999},"last":0.876},{"interface":"front","average":{"1min":0.752,"5min":0.80700000000000005,"15min":0.82699999999999996},"min":{"1min":0.61499999999999999,"5min":0.497,"15min":0.497},"max":{"1min":0.95899999999999996,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.79800000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52172,"kb_used_data":3000,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319668,"statfs":{"total":96636764160,"available":96583340032,"internally_reserved":0,"allocated":3072000,"data_stored":1564658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.42799999999999999,"15min":0.44600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57199999999999995,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.46899999999999997,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60199999999999998,"5min":0.71199999999999997,"15min":0.876},"last":0.375}]},{"osd":1,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48699999999999999,"15min":0.499},"min":{"1min":0.36799999999999999,"5min":0.33200000000000002,"15min":0.32500000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.90300000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55100000000000005,"15min":0.57199999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67700000000000005,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.54900000000000004}]},{"osd":2,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.60099999999999998,"15min":0.626},"min":{"1min":0.40100000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71399999999999997,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.58799999999999997,"15min":0.61599999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.69499999999999995,"5min":0.81699999999999995,"15min":1.0840000000000001},"last":0.56899999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.72299999999999998,"15min":0.752},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89100000000000001,"5min":1.071,"15min":1.0800000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.74299999999999999,"15min":0.76800000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.879,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.73999999999999999}]},{"osd":5,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.74399999999999999,"15min":0.77500000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91800000000000004,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.755},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.76300000000000001,"15min":0.78100000000000003},"min":{"1min":0.54200000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.85699999999999998,"5min":1.266,"15min":1.266},"last":0.627}]},{"osd":6,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.76500000000000001,"15min":0.78400000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97099999999999997,"5min":1.284,"15min":1.284},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.54800000000000004,"5min":0.495,"15min":0.495},"max":{"1min":0.95999999999999996,"5min":1.2,"15min":1.2},"last":0.71199999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.77900000000000003,"15min":0.80800000000000005},"min":{"1min":0.58699999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.93500000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.79300000000000004,"15min":0.81799999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.014,"5min":1.248,"15min":1.248},"last":0.69799999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48080,"kb_used_data":3004,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323760,"statfs":{"total":96636764160,"available":96587530240,"internally_reserved":0,"allocated":3076096,"data_stored":1569864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.66400000000000003,"15min":0.68700000000000006},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":0.80000000000000004,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.65100000000000002,"15min":0.67700000000000005},"min":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.78100000000000003,"5min":0.86199999999999999,"15min":1.075},"last":0.47099999999999997}]},{"osd":1,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.72199999999999998,"15min":0.754},"min":{"1min":0.49399999999999999,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":0.90500000000000003,"5min":1.0549999999999999,"15min":1.069},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.73599999999999999,"15min":0.76500000000000001},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":0.92500000000000004,"5min":1.048,"15min":1.048},"last":0.63700000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.67300000000000004,"15min":0.70799999999999996},"min":{"1min":0.47099999999999997,"5min":0.438,"15min":0.438},"max":{"1min":0.81299999999999994,"5min":1.077,"15min":1.077},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.68999999999999995,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85599999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.52700000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.51500000000000001,"5min":0.501,"15min":0.47399999999999998},"max":{"1min":0.94599999999999995,"5min":0.998,"15min":1.018},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.028},"last":0.749}]},{"osd":5,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.81899999999999995,"5min":1.008,"15min":1.008},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.56100000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65500000000000003,"15min":0.64400000000000002},"min":{"1min":0.48499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85099999999999998,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68999999999999995,"15min":0.69199999999999995},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":1.0049999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.83099999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.48399999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96099999999999997,"5min":1.002,"15min":1.0329999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.77500000000000002,"15min":0.78100000000000003},"min":{"1min":0.56100000000000005,"5min":0.5,"15min":0.5},"max":{"1min":1.038,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.85199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575999,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":3572,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":3657728,"data_stored":2152567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69699999999999995,"15min":0.70999999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.89600000000000002,"5min":0.98699999999999999,"15min":1.004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69899999999999995,"15min":0.72499999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.76200000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.70099999999999996,"15min":0.72199999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85999999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.71399999999999997,"15min":0.73199999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.052},"last":0.66100000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73199999999999998,"15min":0.745},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93200000000000005,"5min":1.004,"15min":1.014},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72599999999999998,"15min":0.746},"min":{"1min":0.46600000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.018,"5min":1.018,"15min":1.0680000000000001},"last":0.63800000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77000000000000002,"15min":0.78600000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":0.98899999999999999,"15min":1.083},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.59099999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94399999999999995,"5min":1.008,"15min":1.1020000000000001},"last":0.58399999999999996}]},{"osd":4,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.629,"15min":0.629},"min":{"1min":0.56499999999999995,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.89100000000000001,"5min":0.96899999999999997,"15min":1.054},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.68400000000000005,"15min":0.67500000000000004},"min":{"1min":0.47999999999999998,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.89300000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.434,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71199999999999997,"15min":0.71099999999999997},"min":{"1min":0.501,"5min":0.44600000000000001,"15min":0.42699999999999999},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.74299999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.73999999999999999,"15min":0.73499999999999999},"min":{"1min":0.60699999999999998,"5min":0.441,"15min":0.441},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.64000000000000001,"5min":0.48199999999999998,"15min":0.45700000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:13:33.217 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T09:13:33.217 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T09:13:33.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.454+0000 7fb795200700 1 Processor -- start 2024-04-02T09:13:33.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.454+0000 7fb795200700 1 -- start start 2024-04-02T09:13:33.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.455+0000 7fb795200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb7900d3f20 0x7fb7900cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:33.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.455+0000 7fb795200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7900b3f80 0x7fb7900ce430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:33.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.455+0000 7fb795200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb790161230 con 0x7fb790160e50 2024-04-02T09:13:33.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.455+0000 7fb795200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7900d2a80 con 0x7fb7900b3f80 2024-04-02T09:13:33.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.455+0000 7fb795200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7900d43b0 con 0x7fb7900d3f20 2024-04-02T09:13:33.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.455+0000 7fb78e800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7900b3f80 0x7fb7900ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:33.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.455+0000 7fb78e800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb7900b3f80 0x7fb7900ce430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60772/0 (socket says 172.21.15.89:60772) 2024-04-02T09:13:33.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.455+0000 7fb78e800700 1 -- 172.21.15.89:0/2712753095 learned_addr learned my addr 172.21.15.89:0/2712753095 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:33.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.455+0000 7fb78d400700 1 --2- 172.21.15.89:0/2712753095 >> v2:172.21.15.113:3300/0 conn(0x7fb7900d3f20 0x7fb7900cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:33.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.456+0000 7fb78e800700 1 -- 172.21.15.89:0/2712753095 >> v2:172.21.15.113:3300/0 conn(0x7fb7900d3f20 msgr2=0x7fb7900cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:33.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.456+0000 7fb78e800700 1 --2- 172.21.15.89:0/2712753095 >> v2:172.21.15.113:3300/0 conn(0x7fb7900d3f20 0x7fb7900cdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.456+0000 7fb78e800700 1 -- 172.21.15.89:0/2712753095 >> v1:172.21.15.113:6789/0 conn(0x7fb790160e50 legacy=0x7fb7900b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:33.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.456+0000 7fb78e800700 1 -- 172.21.15.89:0/2712753095 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7900d21e0 con 0x7fb7900b3f80 2024-04-02T09:13:33.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.456+0000 7fb78e800700 1 --2- 172.21.15.89:0/2712753095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7900b3f80 0x7fb7900ce430 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7fb79008b910 tx=0x7fb78800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e8818c8582bc66 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:33.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.456+0000 7fb78ca00700 1 -- 172.21.15.89:0/2712753095 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78800e070 con 0x7fb7900b3f80 2024-04-02T09:13:33.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.456+0000 7fb78ca00700 1 -- 172.21.15.89:0/2712753095 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78800b910 con 0x7fb7900b3f80 2024-04-02T09:13:33.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.456+0000 7fb78ca00700 1 -- 172.21.15.89:0/2712753095 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78800bcc0 con 0x7fb7900b3f80 2024-04-02T09:13:33.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.457+0000 7fb795200700 1 -- 172.21.15.89:0/2712753095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7900b3f80 msgr2=0x7fb7900ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:33.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.457+0000 7fb795200700 1 --2- 172.21.15.89:0/2712753095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7900b3f80 0x7fb7900ce430 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7fb79008b910 tx=0x7fb78800b330 comp rx=0 tx=0).stop 2024-04-02T09:13:33.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.457+0000 7fb795200700 1 -- 172.21.15.89:0/2712753095 shutdown_connections 2024-04-02T09:13:33.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.457+0000 7fb795200700 1 --2- 172.21.15.89:0/2712753095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7900b3f80 0x7fb7900ce430 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.457+0000 7fb795200700 1 --2- 172.21.15.89:0/2712753095 >> v2:172.21.15.113:3300/0 conn(0x7fb7900d3f20 0x7fb7900cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.457+0000 7fb795200700 1 -- 172.21.15.89:0/2712753095 >> 172.21.15.89:0/2712753095 conn(0x7fb79008aff0 msgr2=0x7fb7900d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:33.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.457+0000 7fb795200700 1 -- 172.21.15.89:0/2712753095 shutdown_connections 2024-04-02T09:13:33.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.457+0000 7fb795200700 1 -- 172.21.15.89:0/2712753095 wait complete. 2024-04-02T09:13:33.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.459+0000 7fb795200700 1 Processor -- start 2024-04-02T09:13:33.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.459+0000 7fb795200700 1 -- start start 2024-04-02T09:13:33.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.460+0000 7fb795200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7900b3f80 0x7fb79015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:33.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.460+0000 7fb795200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb790160e50 0x7fb79006ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:33.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.460+0000 7fb795200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7900b7a80 0x7fb7900ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:33.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.460+0000 7fb795200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79006cf50 con 0x7fb7900b3f80 2024-04-02T09:13:33.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.460+0000 7fb795200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7900b8020 con 0x7fb790160e50 2024-04-02T09:13:33.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.460+0000 7fb795200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7900b8300 con 0x7fb7900b7a80 2024-04-02T09:13:33.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.460+0000 7fb78de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7900b3f80 0x7fb79015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:33.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.460+0000 7fb78de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7900b3f80 0x7fb79015bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60788/0 (socket says 172.21.15.89:60788) 2024-04-02T09:13:33.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.460+0000 7fb78de00700 1 -- 172.21.15.89:0/105159598 learned_addr learned my addr 172.21.15.89:0/105159598 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:33.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.460+0000 7fb78de00700 1 -- 172.21.15.89:0/105159598 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7900b7a80 msgr2=0x7fb7900ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:33.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.460+0000 7fb78de00700 1 --2- 172.21.15.89:0/105159598 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7900b7a80 0x7fb7900ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.460+0000 7fb78de00700 1 -- 172.21.15.89:0/105159598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb790160e50 msgr2=0x7fb79006ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:33.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.460+0000 7fb78de00700 1 --2- 172.21.15.89:0/105159598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb790160e50 0x7fb79006ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.460+0000 7fb78de00700 1 -- 172.21.15.89:0/105159598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb788009b30 con 0x7fb7900b3f80 2024-04-02T09:13:33.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.461+0000 7fb78de00700 1 --2- 172.21.15.89:0/105159598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7900b3f80 0x7fb79015bd70 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fb78000c450 tx=0x7fb78000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:33.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.461+0000 7fb786000700 1 -- 172.21.15.89:0/105159598 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7800052d0 con 0x7fb7900b3f80 2024-04-02T09:13:33.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.461+0000 7fb786000700 1 -- 172.21.15.89:0/105159598 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb780005430 con 0x7fb7900b3f80 2024-04-02T09:13:33.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.461+0000 7fb795200700 1 -- 172.21.15.89:0/105159598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7900b85e0 con 0x7fb7900b3f80 2024-04-02T09:13:33.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.461+0000 7fb786000700 1 -- 172.21.15.89:0/105159598 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7800038c0 con 0x7fb7900b3f80 2024-04-02T09:13:33.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.461+0000 7fb795200700 1 -- 172.21.15.89:0/105159598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7900c77c0 con 0x7fb7900b3f80 2024-04-02T09:13:33.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.462+0000 7fb795200700 1 -- 172.21.15.89:0/105159598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb790067720 con 0x7fb7900b3f80 2024-04-02T09:13:33.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.463+0000 7fb786000700 1 -- 172.21.15.89:0/105159598 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fb780003a20 con 0x7fb7900b3f80 2024-04-02T09:13:33.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.464+0000 7fb786000700 1 --2- 172.21.15.89:0/105159598 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb77006aed0 0x7fb77006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:33.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.464+0000 7fb786000700 1 -- 172.21.15.89:0/105159598 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fb780088d20 con 0x7fb7900b3f80 2024-04-02T09:13:33.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.464+0000 7fb78d400700 1 --2- 172.21.15.89:0/105159598 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb77006aed0 0x7fb77006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:33.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.465+0000 7fb78d400700 1 --2- 172.21.15.89:0/105159598 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb77006aed0 0x7fb77006d390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb77c006b20 tx=0x7fb77c006ab0 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:33.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.468+0000 7fb786000700 1 -- 172.21.15.89:0/105159598 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb780053a10 con 0x7fb7900b3f80 2024-04-02T09:13:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.752+0000 7fb795200700 1 -- 172.21.15.89:0/105159598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb79015d3b0 con 0x7fb7900b3f80 2024-04-02T09:13:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.758+0000 7fb786000700 1 -- 172.21.15.89:0/105159598 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v55) v1 ==== 190+0+0 (secure 0 0 0) 0x7fb780057030 con 0x7fb7900b3f80 2024-04-02T09:13:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.761+0000 7fb795200700 1 -- 172.21.15.89:0/105159598 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb77006aed0 msgr2=0x7fb77006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.761+0000 7fb795200700 1 --2- 172.21.15.89:0/105159598 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb77006aed0 0x7fb77006d390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb77c006b20 tx=0x7fb77c006ab0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.761+0000 7fb795200700 1 -- 172.21.15.89:0/105159598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7900b3f80 msgr2=0x7fb79015bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.761+0000 7fb795200700 1 --2- 172.21.15.89:0/105159598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7900b3f80 0x7fb79015bd70 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fb78000c450 tx=0x7fb78000c810 comp rx=0 tx=0).stop 2024-04-02T09:13:33.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.761+0000 7fb795200700 1 -- 172.21.15.89:0/105159598 shutdown_connections 2024-04-02T09:13:33.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.761+0000 7fb795200700 1 --2- 172.21.15.89:0/105159598 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb7900b7a80 0x7fb7900ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.761+0000 7fb795200700 1 --2- 172.21.15.89:0/105159598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7900b3f80 0x7fb79015bd70 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.761+0000 7fb795200700 1 --2- 172.21.15.89:0/105159598 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb77006aed0 0x7fb77006d390 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.761+0000 7fb795200700 1 --2- 172.21.15.89:0/105159598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb790160e50 0x7fb79006ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.761+0000 7fb795200700 1 -- 172.21.15.89:0/105159598 >> 172.21.15.89:0/105159598 conn(0x7fb79008aff0 msgr2=0x7fb7900cff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:33.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.761+0000 7fb795200700 1 -- 172.21.15.89:0/105159598 shutdown_connections 2024-04-02T09:13:33.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.761+0000 7fb795200700 1 -- 172.21.15.89:0/105159598 wait complete. 2024-04-02T09:13:33.761 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T09:13:33.779 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:13:33.844 INFO:tasks.ceph.mds.d.smithi113.stderr:2024-04-02T09:13:33.845+0000 7f3675000700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T09:13:33.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.973+0000 7fbeaac00700 1 Processor -- start 2024-04-02T09:13:33.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.973+0000 7fbeaac00700 1 -- start start 2024-04-02T09:13:33.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.974+0000 7fbeaac00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbea40b3af0 0x7fbea415a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:33.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.974+0000 7fbeaac00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea40b2180 con 0x7fbea4072580 2024-04-02T09:13:33.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.974+0000 7fbeaac00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea415b100 con 0x7fbea40709c0 2024-04-02T09:13:33.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.974+0000 7fbeaac00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbea415b3e0 con 0x7fbea40b3af0 2024-04-02T09:13:33.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea3400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbea4072580 0x7fbea40b2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56116/0 (socket says 172.21.15.89:56116) 2024-04-02T09:13:33.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea3400700 1 -- 172.21.15.89:0/2720506890 learned_addr learned my addr 172.21.15.89:0/2720506890 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:33.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea3e00700 1 --2- 172.21.15.89:0/2720506890 >> v2:172.21.15.113:3300/0 conn(0x7fbea40b3af0 0x7fbea415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:33.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea2a00700 1 -- 172.21.15.89:0/2720506890 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941939820 0 0) 0x7fbea40b2180 con 0x7fbea4072580 2024-04-02T09:13:33.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea2a00700 1 -- 172.21.15.89:0/2720506890 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe880035b0 con 0x7fbea4072580 2024-04-02T09:13:33.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea2a00700 1 -- 172.21.15.89:0/2720506890 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3773083794 0 0) 0x7fbe880035b0 con 0x7fbea4072580 2024-04-02T09:13:33.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea2a00700 1 -- 172.21.15.89:0/2720506890 >> v2:172.21.15.113:3300/0 conn(0x7fbea40b3af0 msgr2=0x7fbea415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:33.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea2a00700 1 --2- 172.21.15.89:0/2720506890 >> v2:172.21.15.113:3300/0 conn(0x7fbea40b3af0 0x7fbea415a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea2a00700 1 -- 172.21.15.89:0/2720506890 >> v1:172.21.15.113:6789/0 conn(0x7fbea40709c0 legacy=0x7fbea406e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:33.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea2a00700 1 -- 172.21.15.89:0/2720506890 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea40b23f0 con 0x7fbea4072580 2024-04-02T09:13:33.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea2a00700 1 -- 172.21.15.89:0/2720506890 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fbe90001670 con 0x7fbea4072580 2024-04-02T09:13:33.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea3e00700 1 --2- 172.21.15.89:0/2720506890 >> v2:172.21.15.113:3300/0 conn(0x7fbea40b3af0 0x7fbea415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea2a00700 1 -- 172.21.15.89:0/2720506890 >> v1:172.21.15.89:6789/0 conn(0x7fbea4072580 legacy=0x7fbea40b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea2a00700 1 --2- 172.21.15.89:0/2720506890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe88003a00 0x7fbe88003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea2a00700 1 --2- 172.21.15.89:0/2720506890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe88004850 0x7fbe88004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.975+0000 7fbea2a00700 1 --2- 172.21.15.89:0/2720506890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe88005210 0x7fbe8800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.976+0000 7fbea2a00700 1 -- 172.21.15.89:0/2720506890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea415b3e0 con 0x7fbe88003a00 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.976+0000 7fbea2a00700 1 -- 172.21.15.89:0/2720506890 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8800c0b0 con 0x7fbe88004850 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.976+0000 7fbea2a00700 1 -- 172.21.15.89:0/2720506890 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8800e480 con 0x7fbe88005210 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.976+0000 7fbea3e00700 1 --2- 172.21.15.89:0/2720506890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe88003a00 0x7fbe88003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.976+0000 7fbea3400700 1 --2- 172.21.15.89:0/2720506890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe88004850 0x7fbe88004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.976+0000 7fbea8c00700 1 --2- 172.21.15.89:0/2720506890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe88005210 0x7fbe8800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.976+0000 7fbea3e00700 1 -- 172.21.15.89:0/2720506890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe88005210 msgr2=0x7fbe8800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.976+0000 7fbea3e00700 1 --2- 172.21.15.89:0/2720506890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe88005210 0x7fbe8800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.976+0000 7fbea3e00700 1 -- 172.21.15.89:0/2720506890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe88004850 msgr2=0x7fbe88004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.976+0000 7fbea3e00700 1 --2- 172.21.15.89:0/2720506890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe88004850 0x7fbe88004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.976+0000 7fbea3e00700 1 -- 172.21.15.89:0/2720506890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe880035b0 con 0x7fbe88003a00 2024-04-02T09:13:33.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.976+0000 7fbea8c00700 1 -- 172.21.15.89:0/2720506890 reap_dead start 2024-04-02T09:13:33.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.976+0000 7fbea3e00700 1 --2- 172.21.15.89:0/2720506890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe88003a00 0x7fbe88003e20 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7fbe94009d30 tx=0x7fbe88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff802c0c71b49e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:33.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.977+0000 7fbea2a00700 1 -- 172.21.15.89:0/2720506890 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe94009e80 con 0x7fbe88003a00 2024-04-02T09:13:33.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.977+0000 7fbea2a00700 1 -- 172.21.15.89:0/2720506890 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9400e9a0 con 0x7fbe88003a00 2024-04-02T09:13:33.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.977+0000 7fbea2a00700 1 -- 172.21.15.89:0/2720506890 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9400ed10 con 0x7fbe88003a00 2024-04-02T09:13:33.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.977+0000 7fbeaac00700 1 -- 172.21.15.89:0/2720506890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe88003a00 msgr2=0x7fbe88003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:33.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.977+0000 7fbeaac00700 1 --2- 172.21.15.89:0/2720506890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe88003a00 0x7fbe88003e20 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7fbe94009d30 tx=0x7fbe88005880 comp rx=0 tx=0).stop 2024-04-02T09:13:33.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.977+0000 7fbeaac00700 1 -- 172.21.15.89:0/2720506890 shutdown_connections 2024-04-02T09:13:33.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.977+0000 7fbeaac00700 1 --2- 172.21.15.89:0/2720506890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe88003a00 0x7fbe88003e20 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.977+0000 7fbeaac00700 1 -- 172.21.15.89:0/2720506890 >> 172.21.15.89:0/2720506890 conn(0x7fbea40cf190 msgr2=0x7fbea40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:33.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.977+0000 7fbeaac00700 1 -- 172.21.15.89:0/2720506890 shutdown_connections 2024-04-02T09:13:33.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.977+0000 7fbeaac00700 1 -- 172.21.15.89:0/2720506890 wait complete. 2024-04-02T09:13:33.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.979+0000 7fbeaac00700 1 Processor -- start 2024-04-02T09:13:33.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.979+0000 7fbeaac00700 1 -- start start 2024-04-02T09:13:33.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.980+0000 7fbeaac00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe88003a00 0x7fbea414bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:33.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.980+0000 7fbeaac00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbea4154e90 0x7fbea4159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.980+0000 7fbeaac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbea4071970 0x7fbea415c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.980+0000 7fbeaac00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea406df30 con 0x7fbea4154e90 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.980+0000 7fbeaac00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea414c560 con 0x7fbe88003a00 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.980+0000 7fbeaac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea415d530 con 0x7fbea4071970 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.980+0000 7fbea3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbea4154e90 0x7fbea4159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.980+0000 7fbea3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbea4154e90 0x7fbea4159f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60796/0 (socket says 172.21.15.89:60796) 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.980+0000 7fbea3400700 1 -- 172.21.15.89:0/3952656856 learned_addr learned my addr 172.21.15.89:0/3952656856 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.981+0000 7fbea8c00700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbea4071970 0x7fbea415c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.981+0000 7fbea3e00700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe88003a00 0x7fbea414bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.981+0000 7fbea3400700 1 -- 172.21.15.89:0/3952656856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbea4071970 msgr2=0x7fbea415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.981+0000 7fbea3400700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbea4071970 0x7fbea415c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.981+0000 7fbea3400700 1 -- 172.21.15.89:0/3952656856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe88003a00 msgr2=0x7fbea414bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.981+0000 7fbea3400700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe88003a00 0x7fbea414bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.981+0000 7fbea3400700 1 -- 172.21.15.89:0/3952656856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea406dc50 con 0x7fbea4154e90 2024-04-02T09:13:33.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.981+0000 7fbea3e00700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe88003a00 0x7fbea414bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:33.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.981+0000 7fbea3400700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbea4154e90 0x7fbea4159f60 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7fbe90007290 tx=0x7fbe900075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:33.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.981+0000 7fbe9be00700 1 -- 172.21.15.89:0/3952656856 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe90011580 con 0x7fbea4154e90 2024-04-02T09:13:33.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.981+0000 7fbe9be00700 1 -- 172.21.15.89:0/3952656856 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe90011700 con 0x7fbea4154e90 2024-04-02T09:13:33.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.981+0000 7fbe9be00700 1 -- 172.21.15.89:0/3952656856 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe90011ab0 con 0x7fbea4154e90 2024-04-02T09:13:33.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.981+0000 7fbeaac00700 1 -- 172.21.15.89:0/3952656856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea41577e0 con 0x7fbea4154e90 2024-04-02T09:13:33.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.982+0000 7fbeaac00700 1 -- 172.21.15.89:0/3952656856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea414c1f0 con 0x7fbea4154e90 2024-04-02T09:13:33.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.983+0000 7fbeaac00700 1 -- 172.21.15.89:0/3952656856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea406d0b0 con 0x7fbea4154e90 2024-04-02T09:13:33.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.983+0000 7fbe9be00700 1 -- 172.21.15.89:0/3952656856 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fbe90011c10 con 0x7fbea4154e90 2024-04-02T09:13:33.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.985+0000 7fbe9be00700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe8406aed0 0x7fbe8406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:33.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.985+0000 7fbe9be00700 1 -- 172.21.15.89:0/3952656856 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fbe900897a0 con 0x7fbea4154e90 2024-04-02T09:13:33.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.985+0000 7fbea3e00700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe8406aed0 0x7fbe8406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:33.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.985+0000 7fbea3e00700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe8406aed0 0x7fbe8406d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe940090b0 tx=0x7fbe9401e040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:33.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:33.989+0000 7fbe9be00700 1 -- 172.21.15.89:0/3952656856 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbe90054470 con 0x7fbea4154e90 2024-04-02T09:13:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.235+0000 7fbeaac00700 1 -- 172.21.15.89:0/3952656856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbea4157d00 con 0x7fbea4154e90 2024-04-02T09:13:34.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.236+0000 7fbe9be00700 1 -- 172.21.15.89:0/3952656856 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4752 (secure 0 0 0) 0x7fbea4157d00 con 0x7fbea4154e90 2024-04-02T09:13:34.236 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:34.237 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":119,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9518,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3094781297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3094781297},{"type":"v1","addr":"172.21.15.89:6836","nonce":3094781297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":10978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1799351491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1799351491},{"type":"v1","addr":"172.21.15.113:6834","nonce":1799351491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":11958,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3112580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3112580185},{"type":"v1","addr":"172.21.15.89:6837","nonce":3112580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:28.357996+0000","modified":"2024-04-02T09:13:31.369015+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11964":{"gid":11964,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6835/703852199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":703852199},{"type":"v1","addr":"172.21.15.113:6835","nonce":703852199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-04-02T09:13:34.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.238+0000 7fbeaac00700 1 -- 172.21.15.89:0/3952656856 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe8406aed0 msgr2=0x7fbe8406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:34.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.238+0000 7fbeaac00700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe8406aed0 0x7fbe8406d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe940090b0 tx=0x7fbe9401e040 comp rx=0 tx=0).stop 2024-04-02T09:13:34.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.238+0000 7fbeaac00700 1 -- 172.21.15.89:0/3952656856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbea4154e90 msgr2=0x7fbea4159f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:34.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.238+0000 7fbeaac00700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbea4154e90 0x7fbea4159f60 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7fbe90007290 tx=0x7fbe900075a0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.238+0000 7fbeaac00700 1 -- 172.21.15.89:0/3952656856 shutdown_connections 2024-04-02T09:13:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.238+0000 7fbeaac00700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe8406aed0 0x7fbe8406d390 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.238+0000 7fbeaac00700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe88003a00 0x7fbea414bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.238+0000 7fbeaac00700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbea4071970 0x7fbea415c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.238+0000 7fbeaac00700 1 --2- 172.21.15.89:0/3952656856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbea4154e90 0x7fbea4159f60 unknown :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.238+0000 7fbeaac00700 1 -- 172.21.15.89:0/3952656856 >> 172.21.15.89:0/3952656856 conn(0x7fbea40cf190 msgr2=0x7fbea406e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.239+0000 7fbeaac00700 1 -- 172.21.15.89:0/3952656856 shutdown_connections 2024-04-02T09:13:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.239+0000 7fbeaac00700 1 -- 172.21.15.89:0/3952656856 wait complete. 2024-04-02T09:13:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 119 2024-04-02T09:13:34.659 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 119, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T09:13:28.357996+0000', 'modified': '2024-04-02T09:13:31.369015+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11964}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11964': {'gid': 11964, 'name': 'd', 'rank': 0, 'incarnation': 116, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.113:6835/703852199', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6833', 'nonce': 703852199}, {'type': 'v1', 'addr': '172.21.15.113:6835', 'nonce': 703852199}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T09:13:34.659 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T09:13:34.659 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-02T09:13:34.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.853+0000 7fec85a00700 1 Processor -- start 2024-04-02T09:13:34.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.854+0000 7fec85a00700 1 -- start start 2024-04-02T09:13:34.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.855+0000 7fec85a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fec80160e50 0x7fec800b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.855+0000 7fec85a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec800b3cb0 con 0x7fec800b3f80 2024-04-02T09:13:34.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.855+0000 7fec85a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec800cef00 con 0x7fec800d3f20 2024-04-02T09:13:34.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.855+0000 7fec85a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec80161230 con 0x7fec80160e50 2024-04-02T09:13:34.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.855+0000 7fec7f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fec800b3f80 0x7fec800b35a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56126/0 (socket says 172.21.15.89:56126) 2024-04-02T09:13:34.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.855+0000 7fec7ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fec80160e50 0x7fec800b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.855+0000 7fec7f400700 1 -- 172.21.15.89:0/3796256028 learned_addr learned my addr 172.21.15.89:0/3796256028 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:34.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.855+0000 7fec7e000700 1 -- 172.21.15.89:0/3796256028 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800106731 0 0) 0x7fec800b3cb0 con 0x7fec800b3f80 2024-04-02T09:13:34.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.855+0000 7fec7ea00700 1 -- 172.21.15.89:0/3796256028 >> v1:172.21.15.113:6789/0 conn(0x7fec800d3f20 legacy=0x7fec800b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:34.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.856+0000 7fec7ea00700 1 -- 172.21.15.89:0/3796256028 >> v1:172.21.15.89:6789/0 conn(0x7fec800b3f80 legacy=0x7fec800b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:34.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.856+0000 7fec7ea00700 1 -- 172.21.15.89:0/3796256028 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec800d2a80 con 0x7fec80160e50 2024-04-02T09:13:34.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.856+0000 7fec7ea00700 1 --2- 172.21.15.89:0/3796256028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec80160e50 0x7fec800b4e90 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7fec70009840 tx=0x7fec70009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9ecd1166d9aad5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:34.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.856+0000 7fec7e000700 1 -- 172.21.15.89:0/3796256028 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70011830 con 0x7fec80160e50 2024-04-02T09:13:34.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.856+0000 7fec7e000700 1 -- 172.21.15.89:0/3796256028 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec70011990 con 0x7fec80160e50 2024-04-02T09:13:34.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.856+0000 7fec7e000700 1 -- 172.21.15.89:0/3796256028 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70011d20 con 0x7fec80160e50 2024-04-02T09:13:34.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.856+0000 7fec85a00700 1 -- 172.21.15.89:0/3796256028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec80160e50 msgr2=0x7fec800b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:34.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.856+0000 7fec85a00700 1 --2- 172.21.15.89:0/3796256028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec80160e50 0x7fec800b4e90 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7fec70009840 tx=0x7fec70009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:34.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.857+0000 7fec85a00700 1 -- 172.21.15.89:0/3796256028 shutdown_connections 2024-04-02T09:13:34.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.857+0000 7fec85a00700 1 --2- 172.21.15.89:0/3796256028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec80160e50 0x7fec800b4e90 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.857+0000 7fec85a00700 1 -- 172.21.15.89:0/3796256028 >> 172.21.15.89:0/3796256028 conn(0x7fec8008aff0 msgr2=0x7fec800d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:34.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.857+0000 7fec85a00700 1 -- 172.21.15.89:0/3796256028 shutdown_connections 2024-04-02T09:13:34.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.857+0000 7fec85a00700 1 -- 172.21.15.89:0/3796256028 wait complete. 2024-04-02T09:13:34.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.858+0000 7fec85a00700 1 Processor -- start 2024-04-02T09:13:34.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.858+0000 7fec85a00700 1 -- start start 2024-04-02T09:13:34.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.859+0000 7fec85a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec800b3f80 0x7fec800c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.859+0000 7fec85a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec80160e50 0x7fec800b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.859+0000 7fec85a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec800c7090 0x7fec8015fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.859+0000 7fec85a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec800c8380 con 0x7fec80160e50 2024-04-02T09:13:34.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.859+0000 7fec85a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8015f720 con 0x7fec800c7090 2024-04-02T09:13:34.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.859+0000 7fec85a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec800ca020 con 0x7fec800b3f80 2024-04-02T09:13:34.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.860+0000 7fec77e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec80160e50 0x7fec800b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.860+0000 7fec77e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec80160e50 0x7fec800b6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60812/0 (socket says 172.21.15.89:60812) 2024-04-02T09:13:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.860+0000 7fec77e00700 1 -- 172.21.15.89:0/2323774725 learned_addr learned my addr 172.21.15.89:0/2323774725 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.860+0000 7fec77e00700 1 -- 172.21.15.89:0/2323774725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec800b3f80 msgr2=0x7fec800c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.860+0000 7fec77e00700 1 --2- 172.21.15.89:0/2323774725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec800b3f80 0x7fec800c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.860+0000 7fec77e00700 1 -- 172.21.15.89:0/2323774725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec800c7090 msgr2=0x7fec8015fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.860+0000 7fec7f400700 1 --2- 172.21.15.89:0/2323774725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec800c7090 0x7fec8015fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.860+0000 7fec77e00700 1 --2- 172.21.15.89:0/2323774725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec800c7090 0x7fec8015fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.860+0000 7fec77e00700 1 -- 172.21.15.89:0/2323774725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec700094a0 con 0x7fec80160e50 2024-04-02T09:13:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.860+0000 7fec77e00700 1 --2- 172.21.15.89:0/2323774725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec80160e50 0x7fec800b6130 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7fec6800c4e0 tx=0x7fec6800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.860+0000 7fec76a00700 1 -- 172.21.15.89:0/2323774725 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec680052d0 con 0x7fec80160e50 2024-04-02T09:13:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.860+0000 7fec76a00700 1 -- 172.21.15.89:0/2323774725 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec68005430 con 0x7fec80160e50 2024-04-02T09:13:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.860+0000 7fec85a00700 1 -- 172.21.15.89:0/2323774725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8015ed80 con 0x7fec80160e50 2024-04-02T09:13:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.861+0000 7fec76a00700 1 -- 172.21.15.89:0/2323774725 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec680038c0 con 0x7fec80160e50 2024-04-02T09:13:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.861+0000 7fec85a00700 1 -- 172.21.15.89:0/2323774725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8015f340 con 0x7fec80160e50 2024-04-02T09:13:34.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.863+0000 7fec76a00700 1 -- 172.21.15.89:0/2323774725 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fec68003a20 con 0x7fec80160e50 2024-04-02T09:13:34.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.864+0000 7fec76a00700 1 --2- 172.21.15.89:0/2323774725 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fec5c06aed0 0x7fec5c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.864+0000 7fec76a00700 1 -- 172.21.15.89:0/2323774725 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fec68088d10 con 0x7fec80160e50 2024-04-02T09:13:34.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.864+0000 7fec7ea00700 1 --2- 172.21.15.89:0/2323774725 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fec5c06aed0 0x7fec5c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.865+0000 7fec7ea00700 1 --2- 172.21.15.89:0/2323774725 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fec5c06aed0 0x7fec5c06d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fec800cb700 tx=0x7fec70010000 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:34.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.866+0000 7fec994dbb80 1 Processor -- start 2024-04-02T09:13:34.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.866+0000 7fec994dbb80 1 -- start start 2024-04-02T09:13:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec994dbb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a826830 0x560a8a826c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec994dbb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a7fca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec994dbb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 0x560a8a823a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec994dbb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a8a771650 con 0x560a8a7f9be0 2024-04-02T09:13:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec994dbb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a8a752c30 con 0x560a8a826830 2024-04-02T09:13:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec994dbb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x560a8a750430 con 0x560a8a7f7b60 2024-04-02T09:13:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec77e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a7fca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec77e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a7fca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60816/0 (socket says 172.21.15.89:60816) 2024-04-02T09:13:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec77e00700 1 -- 172.21.15.89:0/3437507319 learned_addr learned my addr 172.21.15.89:0/3437507319 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec7ea00700 1 --2- 172.21.15.89:0/3437507319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a826830 0x560a8a826c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec7f400700 1 --2- 172.21.15.89:0/3437507319 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 0x560a8a823a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec77e00700 1 -- 172.21.15.89:0/3437507319 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 msgr2=0x560a8a823a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:34.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec77e00700 1 --2- 172.21.15.89:0/3437507319 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 0x560a8a823a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec77e00700 1 -- 172.21.15.89:0/3437507319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a826830 msgr2=0x560a8a826c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:34.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec77e00700 1 --2- 172.21.15.89:0/3437507319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a826830 0x560a8a826c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec77e00700 1 -- 172.21.15.89:0/3437507319 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a8a53f790 con 0x560a8a7f9be0 2024-04-02T09:13:34.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.867+0000 7fec7f400700 1 --2- 172.21.15.89:0/3437507319 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 0x560a8a823a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:34.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.868+0000 7fec77e00700 1 --2- 172.21.15.89:0/3437507319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a7fca10 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7fec6800c4b0 tx=0x7fec680571a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c6b9afe67b826 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:34.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.868+0000 7fec6f400700 1 -- 172.21.15.89:0/3437507319 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6805b580 con 0x560a8a7f9be0 2024-04-02T09:13:34.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.868+0000 7fec6f400700 1 -- 172.21.15.89:0/3437507319 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6805fb40 con 0x560a8a7f9be0 2024-04-02T09:13:34.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.868+0000 7fec6f400700 1 -- 172.21.15.89:0/3437507319 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6805b370 con 0x560a8a7f9be0 2024-04-02T09:13:34.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.869+0000 7fec994dbb80 1 -- 172.21.15.89:0/3437507319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 msgr2=0x560a8a7fca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:34.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.869+0000 7fec994dbb80 1 --2- 172.21.15.89:0/3437507319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a7fca10 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7fec6800c4b0 tx=0x7fec680571a0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.869+0000 7fec994dbb80 1 -- 172.21.15.89:0/3437507319 shutdown_connections 2024-04-02T09:13:34.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.869+0000 7fec994dbb80 1 --2- 172.21.15.89:0/3437507319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a826830 0x560a8a826c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.869+0000 7fec994dbb80 1 --2- 172.21.15.89:0/3437507319 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 0x560a8a823a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.869+0000 7fec994dbb80 1 --2- 172.21.15.89:0/3437507319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a7fca10 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.869+0000 7fec994dbb80 1 -- 172.21.15.89:0/3437507319 >> 172.21.15.89:0/3437507319 conn(0x560a8a75a6a0 msgr2=0x560a8a841640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:34.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.869+0000 7fec994dbb80 1 -- 172.21.15.89:0/3437507319 shutdown_connections 2024-04-02T09:13:34.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.869+0000 7fec994dbb80 1 -- 172.21.15.89:0/3437507319 wait complete. 2024-04-02T09:13:34.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.870+0000 7fec994dbb80 1 -- start start 2024-04-02T09:13:34.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.870+0000 7fec994dbb80 1 Processor -- start 2024-04-02T09:13:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec994dbb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a7f7b60 0x560a8a842ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec994dbb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a843000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec994dbb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a826830 0x560a8a853e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec994dbb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a8a74bd20 con 0x560a8a7f9be0 2024-04-02T09:13:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec994dbb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a8a6e7770 con 0x560a8a7f7b60 2024-04-02T09:13:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec994dbb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x560a8a70a6f0 con 0x560a8a826830 2024-04-02T09:13:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec994dbb80 10 client.0 _ll_register_callbacks cb 0x560a8a6f15e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T09:13:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec77e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a843000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec77e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a843000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60822/0 (socket says 172.21.15.89:60822) 2024-04-02T09:13:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec77e00700 1 -- 172.21.15.89:0/67737238 learned_addr learned my addr 172.21.15.89:0/67737238 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec7ea00700 1 --2- 172.21.15.89:0/67737238 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a7f7b60 0x560a8a842ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec7f400700 1 --2- 172.21.15.89:0/67737238 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a826830 0x560a8a853e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec77e00700 1 -- 172.21.15.89:0/67737238 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a826830 msgr2=0x560a8a853e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec77e00700 1 --2- 172.21.15.89:0/67737238 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a826830 0x560a8a853e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec77e00700 1 -- 172.21.15.89:0/67737238 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a7f7b60 msgr2=0x560a8a842ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.871+0000 7fec77e00700 1 --2- 172.21.15.89:0/67737238 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a7f7b60 0x560a8a842ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec77e00700 1 -- 172.21.15.89:0/67737238 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8015f340 con 0x560a8a7f9be0 2024-04-02T09:13:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec7ea00700 1 --2- 172.21.15.89:0/67737238 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a7f7b60 0x560a8a842ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec77e00700 1 --2- 172.21.15.89:0/67737238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a843000 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7fec68053c50 tx=0x7fec6801e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed883ad6df80ccef server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec63e00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T09:13:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec63e00700 1 -- 172.21.15.89:0/67737238 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68056e40 con 0x560a8a7f9be0 2024-04-02T09:13:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec63e00700 1 -- 172.21.15.89:0/67737238 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fec6805b580 con 0x560a8a7f9be0 2024-04-02T09:13:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec994dbb80 1 -- 172.21.15.89:0/67737238 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560a8a7ee920 con 0x560a8a7f9be0 2024-04-02T09:13:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec63e00700 1 -- 172.21.15.89:0/67737238 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec6805b380 con 0x560a8a7f9be0 2024-04-02T09:13:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec994dbb80 10 client.12099 fetch_fsmap learned FSMap version 119 2024-04-02T09:13:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec994dbb80 1 -- 172.21.15.89:0/67737238 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) v3 -- 0x560a8a62c0a0 con 0x560a8a7f9be0 2024-04-02T09:13:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec63e00700 1 -- 172.21.15.89:0/67737238 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6807a020 con 0x560a8a7f9be0 2024-04-02T09:13:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec63e00700 1 -- 172.21.15.89:0/67737238 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 119) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fec6805f6b0 con 0x560a8a7f9be0 2024-04-02T09:13:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec994dbb80 10 client.12099 fetch_fsmap finished waiting for FSMap version 119 2024-04-02T09:13:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec994dbb80 10 client.12099 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-02T09:13:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.872+0000 7fec994dbb80 20 client.12099 populate_metadata read hostname 'smithi089' 2024-04-02T09:13:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.873+0000 7fec994dbb80 1 --2- 172.21.15.89:0/67737238 >> [v2:172.21.15.113:6833/703852199,v1:172.21.15.113:6835/703852199] conn(0x560a8a854550 0x560a8a854930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.873+0000 7fec994dbb80 4 client.12099 mds_command: new command op to 11964 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T09:13:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.873+0000 7fec994dbb80 1 -- 172.21.15.89:0/67737238 --> [v2:172.21.15.113:6833/703852199,v1:172.21.15.113:6835/703852199] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a8a7cb6f0 con 0x560a8a854550 2024-04-02T09:13:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.873+0000 7fec7ea00700 1 --2- 172.21.15.89:0/67737238 >> [v2:172.21.15.113:6833/703852199,v1:172.21.15.113:6835/703852199] conn(0x560a8a854550 0x560a8a854930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.874+0000 7fec7ea00700 1 --2- 172.21.15.89:0/67737238 >> [v2:172.21.15.113:6833/703852199,v1:172.21.15.113:6835/703852199] conn(0x560a8a854550 0x560a8a854930 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7205192f915e536e server_cookie=1b3f7fd0dec34304 in_seq=0 out_seq=0 2024-04-02T09:13:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.874+0000 7fec63e00700 10 client.12099 ms_handle_connect on v2:172.21.15.113:6833/703852199 2024-04-02T09:13:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.875+0000 7fec63e00700 1 -- 172.21.15.89:0/67737238 <== mds.0 v2:172.21.15.113:6833/703852199 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560a8a7cb6f0 con 0x560a8a854550 2024-04-02T09:13:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.875+0000 7fec63e00700 10 client.12099 handle_command_reply: tid=0 2024-04-02T09:13:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.875+0000 7fec994dbb80 1 client.12099 shutdown 2024-04-02T09:13:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.875+0000 7fec994dbb80 1 -- 172.21.15.89:0/67737238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 msgr2=0x560a8a843000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.875+0000 7fec994dbb80 1 --2- 172.21.15.89:0/67737238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a843000 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7fec68053c50 tx=0x7fec6801e070 comp rx=0 tx=0).stop 2024-04-02T09:13:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.875+0000 7fec994dbb80 1 -- 172.21.15.89:0/67737238 shutdown_connections 2024-04-02T09:13:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.875+0000 7fec994dbb80 1 --2- 172.21.15.89:0/67737238 >> [v2:172.21.15.113:6833/703852199,v1:172.21.15.113:6835/703852199] conn(0x560a8a854550 0x560a8a854930 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.875+0000 7fec994dbb80 1 --2- 172.21.15.89:0/67737238 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a7f7b60 0x560a8a842ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.875+0000 7fec994dbb80 1 --2- 172.21.15.89:0/67737238 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a826830 0x560a8a853e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.875+0000 7fec63e00700 0 client.12099 ms_handle_reset on v2:172.21.15.113:6833/703852199 2024-04-02T09:13:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.875+0000 7fec994dbb80 1 --2- 172.21.15.89:0/67737238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a843000 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.875+0000 7fec994dbb80 1 -- 172.21.15.89:0/67737238 >> 172.21.15.89:0/67737238 conn(0x560a8a75a6a0 msgr2=0x560a8a7fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.875+0000 7fec994dbb80 1 -- 172.21.15.89:0/67737238 shutdown_connections 2024-04-02T09:13:34.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.875+0000 7fec994dbb80 1 -- 172.21.15.89:0/67737238 wait complete. 2024-04-02T09:13:34.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.876+0000 7fec994dbb80 20 client.12099 trim_cache size 0 max 16384 2024-04-02T09:13:34.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.894+0000 7fec994dbb80 1 Processor -- start 2024-04-02T09:13:34.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.894+0000 7fec994dbb80 1 -- start start 2024-04-02T09:13:34.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.894+0000 7fec994dbb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 0x560a8a7f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.894+0000 7fec994dbb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a7f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec994dbb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a826830 0x560a8a842ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec994dbb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a8a540390 con 0x560a8a7f9be0 2024-04-02T09:13:34.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec994dbb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a8a7f8830 con 0x560a8a826830 2024-04-02T09:13:34.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec994dbb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x560a8a845940 con 0x560a8a7f7b60 2024-04-02T09:13:34.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec77e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a7f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec77e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a7f5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60834/0 (socket says 172.21.15.89:60834) 2024-04-02T09:13:34.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec77e00700 1 -- 172.21.15.89:0/69537682 learned_addr learned my addr 172.21.15.89:0/69537682 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:34.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec7ea00700 1 --2- 172.21.15.89:0/69537682 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 0x560a8a7f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec7f400700 1 --2- 172.21.15.89:0/69537682 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a826830 0x560a8a842ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec77e00700 1 -- 172.21.15.89:0/69537682 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 msgr2=0x560a8a7f55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:34.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec77e00700 1 --2- 172.21.15.89:0/69537682 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 0x560a8a7f55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec77e00700 1 -- 172.21.15.89:0/69537682 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a826830 msgr2=0x560a8a842ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:34.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec77e00700 1 --2- 172.21.15.89:0/69537682 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a826830 0x560a8a842ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec77e00700 1 -- 172.21.15.89:0/69537682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a8a712010 con 0x560a8a7f9be0 2024-04-02T09:13:34.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec7f400700 1 --2- 172.21.15.89:0/69537682 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a826830 0x560a8a842ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:34.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.895+0000 7fec77e00700 1 --2- 172.21.15.89:0/69537682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a7f5ae0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7fec6800c4b0 tx=0x7fec68083070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47bb468411393e55 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:34.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.896+0000 7fec6fe00700 1 -- 172.21.15.89:0/69537682 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6800e040 con 0x560a8a7f9be0 2024-04-02T09:13:34.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.896+0000 7fec6fe00700 1 -- 172.21.15.89:0/69537682 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6807a040 con 0x560a8a7f9be0 2024-04-02T09:13:34.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.896+0000 7fec6fe00700 1 -- 172.21.15.89:0/69537682 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68056e60 con 0x560a8a7f9be0 2024-04-02T09:13:34.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.896+0000 7fec994dbb80 1 -- 172.21.15.89:0/69537682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 msgr2=0x560a8a7f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:34.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.896+0000 7fec994dbb80 1 --2- 172.21.15.89:0/69537682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a7f5ae0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7fec6800c4b0 tx=0x7fec68083070 comp rx=0 tx=0).stop 2024-04-02T09:13:34.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.896+0000 7fec994dbb80 1 -- 172.21.15.89:0/69537682 shutdown_connections 2024-04-02T09:13:34.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.896+0000 7fec994dbb80 1 --2- 172.21.15.89:0/69537682 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a826830 0x560a8a842ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.896+0000 7fec994dbb80 1 --2- 172.21.15.89:0/69537682 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 0x560a8a7f55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.896+0000 7fec994dbb80 1 --2- 172.21.15.89:0/69537682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a7f9be0 0x560a8a7f5ae0 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.896+0000 7fec994dbb80 1 -- 172.21.15.89:0/69537682 >> 172.21.15.89:0/69537682 conn(0x560a8a75a6a0 msgr2=0x560a8a844cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:34.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.896+0000 7fec994dbb80 1 -- 172.21.15.89:0/69537682 shutdown_connections 2024-04-02T09:13:34.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.896+0000 7fec994dbb80 1 -- 172.21.15.89:0/69537682 wait complete. 2024-04-02T09:13:34.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.897+0000 7fec994dbb80 1 -- start start 2024-04-02T09:13:34.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.897+0000 7fec994dbb80 1 Processor -- start 2024-04-02T09:13:34.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.898+0000 7fec994dbb80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 0x560a8a8231f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.898+0000 7fec994dbb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a7f9be0 0x560a8a8271d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.898+0000 7fec994dbb80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a826830 0x560a8a842ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.898+0000 7fec994dbb80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a8a826db0 con 0x560a8a826830 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.898+0000 7fec994dbb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a8a8a8ce0 con 0x560a8a7f9be0 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.898+0000 7fec994dbb80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x560a8a7f7240 con 0x560a8a7f7b60 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.898+0000 7fec994dbb80 10 client.0 _ll_register_callbacks cb 0x7fec68001310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.898+0000 7fec7f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a826830 0x560a8a842ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.898+0000 7fec7f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a826830 0x560a8a842ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60846/0 (socket says 172.21.15.89:60846) 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.898+0000 7fec7f400700 1 -- 172.21.15.89:0/168950771 learned_addr learned my addr 172.21.15.89:0/168950771 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.898+0000 7fec7ea00700 1 --2- 172.21.15.89:0/168950771 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 0x560a8a8231f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.898+0000 7fec77e00700 1 --2- 172.21.15.89:0/168950771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a7f9be0 0x560a8a8271d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.899+0000 7fec7f400700 1 -- 172.21.15.89:0/168950771 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 msgr2=0x560a8a8231f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.899+0000 7fec7f400700 1 --2- 172.21.15.89:0/168950771 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 0x560a8a8231f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.899+0000 7fec7f400700 1 -- 172.21.15.89:0/168950771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a7f9be0 msgr2=0x560a8a8271d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.899+0000 7fec7f400700 1 --2- 172.21.15.89:0/168950771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a7f9be0 0x560a8a8271d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.899+0000 7fec7f400700 1 -- 172.21.15.89:0/168950771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a8a62c0a0 con 0x560a8a826830 2024-04-02T09:13:34.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.899+0000 7fec7f400700 1 --2- 172.21.15.89:0/168950771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a826830 0x560a8a842ac0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7fec780012e0 tx=0x7fec780077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e396f3b8086f089 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:34.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.899+0000 7fec63e00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T09:13:34.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.899+0000 7fec63e00700 1 -- 172.21.15.89:0/168950771 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec780128e0 con 0x560a8a826830 2024-04-02T09:13:34.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.899+0000 7fec63e00700 1 -- 172.21.15.89:0/168950771 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fec78012a40 con 0x560a8a826830 2024-04-02T09:13:34.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.899+0000 7fec994dbb80 1 -- 172.21.15.89:0/168950771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec80153760 con 0x560a8a826830 2024-04-02T09:13:34.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.899+0000 7fec63e00700 1 -- 172.21.15.89:0/168950771 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec78012c00 con 0x560a8a826830 2024-04-02T09:13:34.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.899+0000 7fec994dbb80 10 client.12105 fetch_fsmap learned FSMap version 119 2024-04-02T09:13:34.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.899+0000 7fec994dbb80 1 -- 172.21.15.89:0/168950771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) v3 -- 0x560a8a5a01d0 con 0x560a8a826830 2024-04-02T09:13:34.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.899+0000 7fec63e00700 1 -- 172.21.15.89:0/168950771 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec80153760 con 0x560a8a826830 2024-04-02T09:13:34.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.900+0000 7fec63e00700 1 -- 172.21.15.89:0/168950771 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 119) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fec78012d60 con 0x560a8a826830 2024-04-02T09:13:34.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.900+0000 7fec994dbb80 10 client.12105 fetch_fsmap finished waiting for FSMap version 119 2024-04-02T09:13:34.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.900+0000 7fec994dbb80 10 client.12105 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-02T09:13:34.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.900+0000 7fec994dbb80 20 client.12105 populate_metadata read hostname 'smithi089' 2024-04-02T09:13:34.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.900+0000 7fec994dbb80 1 --2- 172.21.15.89:0/168950771 >> [v2:172.21.15.113:6833/703852199,v1:172.21.15.113:6835/703852199] conn(0x560a8a8a7f40 0x560a8a8a8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:34.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.900+0000 7fec994dbb80 4 client.12105 mds_command: new command op to 11964 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T09:13:34.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.900+0000 7fec994dbb80 1 -- 172.21.15.89:0/168950771 --> [v2:172.21.15.113:6833/703852199,v1:172.21.15.113:6835/703852199] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fec6805fb40 con 0x560a8a8a7f40 2024-04-02T09:13:34.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.900+0000 7fec7ea00700 1 --2- 172.21.15.89:0/168950771 >> [v2:172.21.15.113:6833/703852199,v1:172.21.15.113:6835/703852199] conn(0x560a8a8a7f40 0x560a8a8a8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:34.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec63e00700 10 client.12105 ms_handle_connect on v2:172.21.15.113:6833/703852199 2024-04-02T09:13:34.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec7ea00700 1 --2- 172.21.15.89:0/168950771 >> [v2:172.21.15.113:6833/703852199,v1:172.21.15.113:6835/703852199] conn(0x560a8a8a7f40 0x560a8a8a8320 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c8833d311e4a4f8 server_cookie=663b7c267dbb47bd in_seq=0 out_seq=0 2024-04-02T09:13:34.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec63e00700 1 -- 172.21.15.89:0/168950771 <== mds.0 v2:172.21.15.113:6833/703852199 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x7fec6805fb40 con 0x560a8a8a7f40 2024-04-02T09:13:34.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec63e00700 10 client.12105 handle_command_reply: tid=0 2024-04-02T09:13:34.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec994dbb80 1 client.12105 shutdown 2024-04-02T09:13:34.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec994dbb80 1 -- 172.21.15.89:0/168950771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a826830 msgr2=0x560a8a842ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:34.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec994dbb80 1 --2- 172.21.15.89:0/168950771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a826830 0x560a8a842ac0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7fec780012e0 tx=0x7fec780077e0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec994dbb80 1 -- 172.21.15.89:0/168950771 shutdown_connections 2024-04-02T09:13:34.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec994dbb80 1 --2- 172.21.15.89:0/168950771 >> [v2:172.21.15.113:6833/703852199,v1:172.21.15.113:6835/703852199] conn(0x560a8a8a7f40 0x560a8a8a8320 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec994dbb80 1 --2- 172.21.15.89:0/168950771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a8a826830 0x560a8a842ac0 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec994dbb80 1 --2- 172.21.15.89:0/168950771 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x560a8a7f7b60 0x560a8a8231f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec63e00700 0 client.12105 ms_handle_reset on v2:172.21.15.113:6833/703852199 2024-04-02T09:13:34.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec994dbb80 1 --2- 172.21.15.89:0/168950771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a8a7f9be0 0x560a8a8271d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec994dbb80 1 -- 172.21.15.89:0/168950771 >> 172.21.15.89:0/168950771 conn(0x560a8a75a6a0 msgr2=0x560a8a843c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:34.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec994dbb80 1 -- 172.21.15.89:0/168950771 shutdown_connections 2024-04-02T09:13:34.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec994dbb80 1 -- 172.21.15.89:0/168950771 wait complete. 2024-04-02T09:13:34.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.901+0000 7fec994dbb80 20 client.12105 trim_cache size 0 max 16384 2024-04-02T09:13:34.905 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:13:34.905 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T09:13:34.905 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T09:13:34.905 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 11964, 2024-04-02T09:13:34.905 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T09:13:34.906 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T09:13:34.906 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T09:13:34.906 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 6.5416314230000001, 2024-04-02T09:13:34.906 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 119, 2024-04-02T09:13:34.906 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 244, 2024-04-02T09:13:34.906 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 244, 2024-04-02T09:13:34.906 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 11.562333533 2024-04-02T09:13:34.906 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:13:34.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.902+0000 7fec85a00700 1 -- 172.21.15.89:0/2323774725 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fec5c06aed0 msgr2=0x7fec5c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:34.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.902+0000 7fec85a00700 1 --2- 172.21.15.89:0/2323774725 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fec5c06aed0 0x7fec5c06d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fec800cb700 tx=0x7fec70010000 comp rx=0 tx=0).stop 2024-04-02T09:13:34.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.902+0000 7fec85a00700 1 -- 172.21.15.89:0/2323774725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec80160e50 msgr2=0x7fec800b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:34.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.902+0000 7fec85a00700 1 --2- 172.21.15.89:0/2323774725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec80160e50 0x7fec800b6130 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7fec6800c4e0 tx=0x7fec6800c8a0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.902+0000 7fec85a00700 1 -- 172.21.15.89:0/2323774725 shutdown_connections 2024-04-02T09:13:34.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.902+0000 7fec85a00700 1 --2- 172.21.15.89:0/2323774725 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fec5c06aed0 0x7fec5c06d390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.902+0000 7fec85a00700 1 --2- 172.21.15.89:0/2323774725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec800c7090 0x7fec8015fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.902+0000 7fec85a00700 1 --2- 172.21.15.89:0/2323774725 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec800b3f80 0x7fec800c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.902+0000 7fec85a00700 1 --2- 172.21.15.89:0/2323774725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec80160e50 0x7fec800b6130 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:34.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.902+0000 7fec85a00700 1 -- 172.21.15.89:0/2323774725 >> 172.21.15.89:0/2323774725 conn(0x7fec8008aff0 msgr2=0x7fec800ce1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:34.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.902+0000 7fec85a00700 1 -- 172.21.15.89:0/2323774725 shutdown_connections 2024-04-02T09:13:34.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:34.902+0000 7fec85a00700 1 -- 172.21.15.89:0/2323774725 wait complete. 2024-04-02T09:13:34.919 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T09:13:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.110+0000 7fec16e00700 1 Processor -- start 2024-04-02T09:13:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.110+0000 7fec16e00700 1 -- start start 2024-04-02T09:13:35.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.111+0000 7fec16e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100ac580 0x7fec100b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:35.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.111+0000 7fec16e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fec100b3000 0x7fec1006dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:35.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.111+0000 7fec16e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec100b3890 con 0x7fec1015d670 2024-04-02T09:13:35.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.111+0000 7fec16e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec10159f50 con 0x7fec100b3000 2024-04-02T09:13:35.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.111+0000 7fec16e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec100b35b0 con 0x7fec100ac580 2024-04-02T09:13:35.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.111+0000 7fec0fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fec1015d670 0x7fec1015da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56140/0 (socket says 172.21.15.89:56140) 2024-04-02T09:13:35.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.111+0000 7fec0fe00700 1 -- 172.21.15.89:0/2392445060 learned_addr learned my addr 172.21.15.89:0/2392445060 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:35.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0f400700 1 --2- 172.21.15.89:0/2392445060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100ac580 0x7fec100b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:35.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec14e00700 1 --2- 172.21.15.89:0/2392445060 >> v2:172.21.15.113:3300/0 conn(0x7fec100b3000 0x7fec1006dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:35.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 -- 172.21.15.89:0/2392445060 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475636314 0 0) 0x7fec100b3890 con 0x7fec1015d670 2024-04-02T09:13:35.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 -- 172.21.15.89:0/2392445060 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf40035b0 con 0x7fec1015d670 2024-04-02T09:13:35.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 -- 172.21.15.89:0/2392445060 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3527042008 0 0) 0x7febf40035b0 con 0x7fec1015d670 2024-04-02T09:13:35.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 -- 172.21.15.89:0/2392445060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100ac580 msgr2=0x7fec100b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:35.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 --2- 172.21.15.89:0/2392445060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100ac580 0x7fec100b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:35.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 -- 172.21.15.89:0/2392445060 >> v2:172.21.15.113:3300/0 conn(0x7fec100b3000 msgr2=0x7fec1006dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:35.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 --2- 172.21.15.89:0/2392445060 >> v2:172.21.15.113:3300/0 conn(0x7fec100b3000 0x7fec1006dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:35.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 -- 172.21.15.89:0/2392445060 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1006d400 con 0x7fec1015d670 2024-04-02T09:13:35.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 -- 172.21.15.89:0/2392445060 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fec00001670 con 0x7fec1015d670 2024-04-02T09:13:35.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0f400700 1 --2- 172.21.15.89:0/2392445060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100ac580 0x7fec100b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:35.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 -- 172.21.15.89:0/2392445060 >> v1:172.21.15.89:6789/0 conn(0x7fec1015d670 legacy=0x7fec1015da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:35.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 --2- 172.21.15.89:0/2392445060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100ac580 0x7febf4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:35.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 --2- 172.21.15.89:0/2392445060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febf4003f40 0x7febf4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:35.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 --2- 172.21.15.89:0/2392445060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febf4004d90 0x7febf4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:35.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 -- 172.21.15.89:0/2392445060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec10159f50 con 0x7febf4003f40 2024-04-02T09:13:35.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 -- 172.21.15.89:0/2392445060 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec100b35b0 con 0x7febf4004d90 2024-04-02T09:13:35.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.112+0000 7fec0ea00700 1 -- 172.21.15.89:0/2392445060 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf400ba10 con 0x7fec100ac580 2024-04-02T09:13:35.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.113+0000 7fec0f400700 1 --2- 172.21.15.89:0/2392445060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febf4003f40 0x7febf4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:35.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.113+0000 7fec0fe00700 1 --2- 172.21.15.89:0/2392445060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100ac580 0x7febf4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:35.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.113+0000 7fec14e00700 1 --2- 172.21.15.89:0/2392445060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febf4004d90 0x7febf4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:35.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.113+0000 7fec0f400700 1 -- 172.21.15.89:0/2392445060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100ac580 msgr2=0x7febf4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:35.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.113+0000 7fec0f400700 1 --2- 172.21.15.89:0/2392445060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100ac580 0x7febf4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:35.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.113+0000 7fec0f400700 1 -- 172.21.15.89:0/2392445060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febf4004d90 msgr2=0x7febf4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:35.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.113+0000 7fec0f400700 1 --2- 172.21.15.89:0/2392445060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febf4004d90 0x7febf4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:35.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.113+0000 7fec0f400700 1 -- 172.21.15.89:0/2392445060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc00a640 con 0x7febf4003f40 2024-04-02T09:13:35.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.113+0000 7fec0fe00700 1 --2- 172.21.15.89:0/2392445060 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100ac580 0x7febf4003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:35.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.113+0000 7fec0f400700 1 --2- 172.21.15.89:0/2392445060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febf4003f40 0x7febf4004850 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7febfc00ad30 tx=0x7febf4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a15d349f581aa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:35.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.113+0000 7fec0ea00700 1 -- 172.21.15.89:0/2392445060 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc00fcb0 con 0x7febf4003f40 2024-04-02T09:13:35.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.113+0000 7fec0ea00700 1 -- 172.21.15.89:0/2392445060 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febfc00fe10 con 0x7febf4003f40 2024-04-02T09:13:35.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.113+0000 7fec0ea00700 1 -- 172.21.15.89:0/2392445060 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc012680 con 0x7febf4003f40 2024-04-02T09:13:35.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.114+0000 7fec16e00700 1 -- 172.21.15.89:0/2392445060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febf4003f40 msgr2=0x7febf4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:35.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.114+0000 7fec16e00700 1 --2- 172.21.15.89:0/2392445060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febf4003f40 0x7febf4004850 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7febfc00ad30 tx=0x7febf4005880 comp rx=0 tx=0).stop 2024-04-02T09:13:35.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.114+0000 7fec14e00700 1 -- 172.21.15.89:0/2392445060 reap_dead start 2024-04-02T09:13:35.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.114+0000 7fec16e00700 1 -- 172.21.15.89:0/2392445060 shutdown_connections 2024-04-02T09:13:35.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.114+0000 7fec16e00700 1 -- 172.21.15.89:0/2392445060 >> 172.21.15.89:0/2392445060 conn(0x7fec100cf190 msgr2=0x7fec100cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:35.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.114+0000 7fec16e00700 1 -- 172.21.15.89:0/2392445060 shutdown_connections 2024-04-02T09:13:35.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.114+0000 7fec16e00700 1 -- 172.21.15.89:0/2392445060 wait complete. 2024-04-02T09:13:35.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.116+0000 7fec16e00700 1 Processor -- start 2024-04-02T09:13:35.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.116+0000 7fec16e00700 1 -- start start 2024-04-02T09:13:35.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.117+0000 7fec16e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febf4003f40 0x7fec1006ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:35.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.117+0000 7fec16e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100979b0 0x7fec100c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:35.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.117+0000 7fec16e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec10156090 0x7fec10156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:35.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.117+0000 7fec16e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec100ca2d0 con 0x7febf4003f40 2024-04-02T09:13:35.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.117+0000 7fec16e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec100cb080 con 0x7fec10156090 2024-04-02T09:13:35.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.117+0000 7fec16e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec100ca500 con 0x7fec100979b0 2024-04-02T09:13:35.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.117+0000 7fec0fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febf4003f40 0x7fec1006ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:35.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.117+0000 7fec0fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febf4003f40 0x7fec1006ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60860/0 (socket says 172.21.15.89:60860) 2024-04-02T09:13:35.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.117+0000 7fec0fe00700 1 -- 172.21.15.89:0/1802087488 learned_addr learned my addr 172.21.15.89:0/1802087488 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:35.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.117+0000 7fec0f400700 1 --2- 172.21.15.89:0/1802087488 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100979b0 0x7fec100c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:35.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.117+0000 7fec14e00700 1 --2- 172.21.15.89:0/1802087488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec10156090 0x7fec10156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:35.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.118+0000 7fec0fe00700 1 -- 172.21.15.89:0/1802087488 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100979b0 msgr2=0x7fec100c86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:35.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.118+0000 7fec0fe00700 1 --2- 172.21.15.89:0/1802087488 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100979b0 0x7fec100c86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:35.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.118+0000 7fec0fe00700 1 -- 172.21.15.89:0/1802087488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec10156090 msgr2=0x7fec10156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:35.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.118+0000 7fec0fe00700 1 --2- 172.21.15.89:0/1802087488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec10156090 0x7fec10156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:35.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.118+0000 7fec0fe00700 1 -- 172.21.15.89:0/1802087488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec100ca7e0 con 0x7febf4003f40 2024-04-02T09:13:35.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.118+0000 7fec0fe00700 1 --2- 172.21.15.89:0/1802087488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febf4003f40 0x7fec1006ad30 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7fec0000eaf0 tx=0x7fec0000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:35.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.118+0000 7fec07e00700 1 -- 172.21.15.89:0/1802087488 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0000c7b0 con 0x7febf4003f40 2024-04-02T09:13:35.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.118+0000 7fec07e00700 1 -- 172.21.15.89:0/1802087488 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0000c910 con 0x7febf4003f40 2024-04-02T09:13:35.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.118+0000 7fec07e00700 1 -- 172.21.15.89:0/1802087488 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0000cc80 con 0x7febf4003f40 2024-04-02T09:13:35.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.118+0000 7fec16e00700 1 -- 172.21.15.89:0/1802087488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec100caac0 con 0x7febf4003f40 2024-04-02T09:13:35.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.118+0000 7fec16e00700 1 -- 172.21.15.89:0/1802087488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1006b270 con 0x7febf4003f40 2024-04-02T09:13:35.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.119+0000 7fec16e00700 1 -- 172.21.15.89:0/1802087488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1015a090 con 0x7febf4003f40 2024-04-02T09:13:35.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.120+0000 7fec07e00700 1 -- 172.21.15.89:0/1802087488 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fec0001d460 con 0x7febf4003f40 2024-04-02T09:13:35.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.121+0000 7fec07e00700 1 --2- 172.21.15.89:0/1802087488 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7febf006aed0 0x7febf006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:35.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.121+0000 7fec07e00700 1 -- 172.21.15.89:0/1802087488 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fec00089780 con 0x7febf4003f40 2024-04-02T09:13:35.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.121+0000 7fec0f400700 1 --2- 172.21.15.89:0/1802087488 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7febf006aed0 0x7febf006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:35.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.122+0000 7fec0f400700 1 --2- 172.21.15.89:0/1802087488 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7febf006aed0 0x7febf006d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7febfc00a1c0 tx=0x7febfc01d040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:35.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.126+0000 7fec07e00700 1 -- 172.21.15.89:0/1802087488 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fec1015a090 con 0x7febf4003f40 2024-04-02T09:13:35.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.369+0000 7fec16e00700 1 -- 172.21.15.89:0/1802087488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fec1015a090 con 0x7febf4003f40 2024-04-02T09:13:35.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.369+0000 7fec07e00700 1 -- 172.21.15.89:0/1802087488 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v119) v1 ==== 53+0+73 (secure 0 0 0) 0x7fec1015a090 con 0x7febf4003f40 2024-04-02T09:13:35.369 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T09:13:35.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.372+0000 7fec16e00700 1 -- 172.21.15.89:0/1802087488 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7febf006aed0 msgr2=0x7febf006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:35.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.372+0000 7fec16e00700 1 --2- 172.21.15.89:0/1802087488 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7febf006aed0 0x7febf006d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7febfc00a1c0 tx=0x7febfc01d040 comp rx=0 tx=0).stop 2024-04-02T09:13:35.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.372+0000 7fec16e00700 1 -- 172.21.15.89:0/1802087488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febf4003f40 msgr2=0x7fec1006ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:35.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.372+0000 7fec16e00700 1 --2- 172.21.15.89:0/1802087488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febf4003f40 0x7fec1006ad30 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7fec0000eaf0 tx=0x7fec0000eeb0 comp rx=0 tx=0).stop 2024-04-02T09:13:35.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.372+0000 7fec16e00700 1 -- 172.21.15.89:0/1802087488 shutdown_connections 2024-04-02T09:13:35.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.372+0000 7fec16e00700 1 --2- 172.21.15.89:0/1802087488 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7febf006aed0 0x7febf006d390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:35.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.372+0000 7fec16e00700 1 --2- 172.21.15.89:0/1802087488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec10156090 0x7fec10156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:35.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.372+0000 7fec16e00700 1 --2- 172.21.15.89:0/1802087488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7febf4003f40 0x7fec1006ad30 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:35.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.372+0000 7fec16e00700 1 --2- 172.21.15.89:0/1802087488 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec100979b0 0x7fec100c86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:35.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.372+0000 7fec16e00700 1 -- 172.21.15.89:0/1802087488 >> 172.21.15.89:0/1802087488 conn(0x7fec100cf190 msgr2=0x7fec10072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:35.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.372+0000 7fec16e00700 1 -- 172.21.15.89:0/1802087488 shutdown_connections 2024-04-02T09:13:35.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:35.372+0000 7fec16e00700 1 -- 172.21.15.89:0/1802087488 wait complete. 2024-04-02T09:13:35.389 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T09:13:35.389 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T09:13:35.389 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T09:13:35.390 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T09:13:35.390 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T09:13:35.390 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T09:13:35.390 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T09:13:35.390 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T09:13:35.390 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:13:35.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:13:35.390 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T09:13:35.435 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T09:13:35.436 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T09:13:35.436 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T09:13:35.436 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:13:35.436 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T09:13:35.436 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:13:35.436 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:13:35.436 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:13:35.436 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T09:13:35.436 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:13:35.436 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:13:35.436 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T09:13:35.437 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:13:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:13:35.437 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T09:13:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:13:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T09:13:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 36613sec preferred_lft 36613sec 2024-04-02T09:13:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T09:13:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:13:35.437 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T09:13:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:13:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T09:13:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:13:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T09:13:35.437 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:13:35.438 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T09:13:35.438 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T09:13:35.438 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T09:13:35.438 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:13:35.438 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T09:13:35.438 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T09:13:35.438 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T09:13:35.438 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:13:35.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:13:35.439 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T09:13:35.543 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T09:13:35.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:13:35.546 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T09:13:35.593 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T09:13:35.594 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T09:13:35.594 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T09:13:35.594 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T09:13:35.594 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T09:13:35.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:13:35.595 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T09:13:35.595 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T09:13:35.595 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T09:13:35.595 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T09:13:35.595 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T09:13:35.595 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T09:13:35.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:13:35.755 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T09:13:35.799 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T09:13:35.799 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T09:13:35.800 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:13:35.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:13:35.801 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T09:13:35.909 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T09:13:35.909 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T09:13:35.953 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:13:35.953 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T09:13:35.953 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T09:13:36.037 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T09:13:36.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:13:36.037 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:13:36.176 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:13:36.176 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T09:13:36.203 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:13:36.204 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T09:13:36.204 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T09:13:36.204 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T09:13:36.204 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T09:13:36.208 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T09:13:36.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:13:36.209 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T09:13:36.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:13:36.276 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T09:13:36.381 INFO:teuthology.orchestra.run.smithi089.stdout:client12114 2024-04-02T09:13:36.382 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:13:36.382 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:13:36.427 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-04-02T09:13:36.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.664+0000 7f508dc00700 1 Processor -- start 2024-04-02T09:13:36.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.664+0000 7f508dc00700 1 -- start start 2024-04-02T09:13:36.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.665+0000 7f508dc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508815ddf0 0x7f50880ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.665+0000 7f508dc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f508815e1d0 0x7f50880d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.666+0000 7f508dc00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50880ce8a0 con 0x7f50880d2710 2024-04-02T09:13:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.666+0000 7f508dc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50880d2b80 con 0x7f508815e1d0 2024-04-02T09:13:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.666+0000 7f508dc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f508815ec60 con 0x7f508815ddf0 2024-04-02T09:13:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.666+0000 7f5086000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f508815e1d0 0x7f50880d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.666+0000 7f5087400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f50880d2710 0x7f50881612f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56148/0 (socket says 172.21.15.89:56148) 2024-04-02T09:13:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.666+0000 7f5087400700 1 -- 172.21.15.89:0/3459671961 learned_addr learned my addr 172.21.15.89:0/3459671961 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.666+0000 7f5086a00700 1 --2- 172.21.15.89:0/3459671961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508815ddf0 0x7f50880ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.666+0000 7f5086000700 1 -- 172.21.15.89:0/3459671961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508815ddf0 msgr2=0x7f50880ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.666+0000 7f5086000700 1 --2- 172.21.15.89:0/3459671961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508815ddf0 0x7f50880ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:36.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.666+0000 7f5086000700 1 -- 172.21.15.89:0/3459671961 >> v1:172.21.15.89:6789/0 conn(0x7f50880d2710 legacy=0x7f50881612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.666+0000 7f5086000700 1 -- 172.21.15.89:0/3459671961 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50880cdc90 con 0x7f508815e1d0 2024-04-02T09:13:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.666+0000 7f5085600700 1 -- 172.21.15.89:0/3459671961 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103698754 0 0) 0x7f50880ce8a0 con 0x7f50880d2710 2024-04-02T09:13:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.667+0000 7f5086000700 1 --2- 172.21.15.89:0/3459671961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508815e1d0 0x7f50880d21d0 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f5074009840 tx=0x7f5074009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b5f2e9f9e5fb7fb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.667+0000 7f5085600700 1 -- 172.21.15.89:0/3459671961 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074011830 con 0x7f508815e1d0 2024-04-02T09:13:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.667+0000 7f5085600700 1 -- 172.21.15.89:0/3459671961 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5074011990 con 0x7f508815e1d0 2024-04-02T09:13:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.667+0000 7f5085600700 1 -- 172.21.15.89:0/3459671961 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074011d20 con 0x7f508815e1d0 2024-04-02T09:13:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.667+0000 7f508dc00700 1 -- 172.21.15.89:0/3459671961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508815e1d0 msgr2=0x7f50880d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.667+0000 7f508dc00700 1 --2- 172.21.15.89:0/3459671961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508815e1d0 0x7f50880d21d0 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f5074009840 tx=0x7f5074009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.667+0000 7f508dc00700 1 -- 172.21.15.89:0/3459671961 shutdown_connections 2024-04-02T09:13:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.667+0000 7f508dc00700 1 --2- 172.21.15.89:0/3459671961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f508815ddf0 0x7f50880ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.667+0000 7f508dc00700 1 --2- 172.21.15.89:0/3459671961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508815e1d0 0x7f50880d21d0 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.667+0000 7f508dc00700 1 -- 172.21.15.89:0/3459671961 >> 172.21.15.89:0/3459671961 conn(0x7f508808aff0 msgr2=0x7f50880d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.667+0000 7f508dc00700 1 -- 172.21.15.89:0/3459671961 shutdown_connections 2024-04-02T09:13:36.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.667+0000 7f508dc00700 1 -- 172.21.15.89:0/3459671961 wait complete. 2024-04-02T09:13:36.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.669+0000 7f508dc00700 1 Processor -- start 2024-04-02T09:13:36.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.669+0000 7f508dc00700 1 -- start start 2024-04-02T09:13:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.670+0000 7f508dc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f50880d2710 0x7f5088159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.670+0000 7f508dc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508815ddf0 0x7f50880b6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f508dc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f508815e1d0 0x7f50880ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f508dc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508815a250 con 0x7f508815ddf0 2024-04-02T09:13:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f508dc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50880cb030 con 0x7f508815e1d0 2024-04-02T09:13:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f508dc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50881607c0 con 0x7f50880d2710 2024-04-02T09:13:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f5086000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508815ddf0 0x7f50880b6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:36.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f5086000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508815ddf0 0x7f50880b6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60874/0 (socket says 172.21.15.89:60874) 2024-04-02T09:13:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f5086000700 1 -- 172.21.15.89:0/958738923 learned_addr learned my addr 172.21.15.89:0/958738923 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f5086a00700 1 --2- 172.21.15.89:0/958738923 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f50880d2710 0x7f5088159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f5087400700 1 --2- 172.21.15.89:0/958738923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f508815e1d0 0x7f50880ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f5086000700 1 -- 172.21.15.89:0/958738923 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f50880d2710 msgr2=0x7f5088159d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f5086000700 1 --2- 172.21.15.89:0/958738923 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f50880d2710 0x7f5088159d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f5086000700 1 -- 172.21.15.89:0/958738923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f508815e1d0 msgr2=0x7f50880ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f5086000700 1 --2- 172.21.15.89:0/958738923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f508815e1d0 0x7f50880ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f5086000700 1 -- 172.21.15.89:0/958738923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50740094a0 con 0x7f508815ddf0 2024-04-02T09:13:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.671+0000 7f5086000700 1 --2- 172.21.15.89:0/958738923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508815ddf0 0x7f50880b6fd0 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f5074014040 tx=0x7f5074004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.672+0000 7f507ea00700 1 -- 172.21.15.89:0/958738923 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50740036a0 con 0x7f508815ddf0 2024-04-02T09:13:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.672+0000 7f507ea00700 1 -- 172.21.15.89:0/958738923 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507401e070 con 0x7f508815ddf0 2024-04-02T09:13:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.672+0000 7f507ea00700 1 -- 172.21.15.89:0/958738923 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074003a00 con 0x7f508815ddf0 2024-04-02T09:13:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.672+0000 7f508dc00700 1 -- 172.21.15.89:0/958738923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50880cb2d0 con 0x7f508815ddf0 2024-04-02T09:13:36.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.672+0000 7f508dc00700 1 -- 172.21.15.89:0/958738923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5088160a60 con 0x7f508815ddf0 2024-04-02T09:13:36.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.673+0000 7f508dc00700 1 -- 172.21.15.89:0/958738923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5088160c50 con 0x7f508815ddf0 2024-04-02T09:13:36.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.674+0000 7f507ea00700 1 -- 172.21.15.89:0/958738923 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5074003b60 con 0x7f508815ddf0 2024-04-02T09:13:36.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.675+0000 7f507ea00700 1 --2- 172.21.15.89:0/958738923 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f506806aed0 0x7f506806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:36.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.675+0000 7f507ea00700 1 -- 172.21.15.89:0/958738923 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f507408a990 con 0x7f508815ddf0 2024-04-02T09:13:36.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.676+0000 7f5086a00700 1 --2- 172.21.15.89:0/958738923 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f506806aed0 0x7f506806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:36.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.676+0000 7f5086a00700 1 --2- 172.21.15.89:0/958738923 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f506806aed0 0x7f506806d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5078006b20 tx=0x7f5078008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:36.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.680+0000 7f507ea00700 1 -- 172.21.15.89:0/958738923 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5074055640 con 0x7f508815ddf0 2024-04-02T09:13:36.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:36.925+0000 7f508dc00700 1 -- 172.21.15.89:0/958738923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5088067720 con 0x7f508815ddf0 2024-04-02T09:13:37.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.401+0000 7f507ea00700 1 -- 172.21.15.89:0/958738923 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v120) v1 ==== 136+0+0 (secure 0 0 0) 0x7f50740200c0 con 0x7f508815ddf0 2024-04-02T09:13:37.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.404+0000 7f508dc00700 1 -- 172.21.15.89:0/958738923 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f506806aed0 msgr2=0x7f506806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:37.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.404+0000 7f508dc00700 1 --2- 172.21.15.89:0/958738923 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f506806aed0 0x7f506806d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5078006b20 tx=0x7f5078008040 comp rx=0 tx=0).stop 2024-04-02T09:13:37.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.404+0000 7f508dc00700 1 -- 172.21.15.89:0/958738923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508815ddf0 msgr2=0x7f50880b6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:37.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.404+0000 7f508dc00700 1 --2- 172.21.15.89:0/958738923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508815ddf0 0x7f50880b6fd0 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f5074014040 tx=0x7f5074004750 comp rx=0 tx=0).stop 2024-04-02T09:13:37.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.404+0000 7f508dc00700 1 -- 172.21.15.89:0/958738923 shutdown_connections 2024-04-02T09:13:37.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.404+0000 7f508dc00700 1 --2- 172.21.15.89:0/958738923 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f506806aed0 0x7f506806d390 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:37.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.404+0000 7f508dc00700 1 --2- 172.21.15.89:0/958738923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f508815e1d0 0x7f50880ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:37.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.404+0000 7f508dc00700 1 --2- 172.21.15.89:0/958738923 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f50880d2710 0x7f5088159d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:37.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.404+0000 7f508dc00700 1 --2- 172.21.15.89:0/958738923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f508815ddf0 0x7f50880b6fd0 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:37.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.404+0000 7f508dc00700 1 -- 172.21.15.89:0/958738923 >> 172.21.15.89:0/958738923 conn(0x7f508808aff0 msgr2=0x7f50880b8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:37.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.404+0000 7f508dc00700 1 -- 172.21.15.89:0/958738923 shutdown_connections 2024-04-02T09:13:37.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.404+0000 7f508dc00700 1 -- 172.21.15.89:0/958738923 wait complete. 2024-04-02T09:13:37.420 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:13:37.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.613+0000 7f9a11600700 1 Processor -- start 2024-04-02T09:13:37.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.614+0000 7f9a11600700 1 -- start start 2024-04-02T09:13:37.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.615+0000 7f9a11600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a0c0d18b0 0x7f9a0c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:37.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.615+0000 7f9a11600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c0ceda0 con 0x7f9a0c161590 2024-04-02T09:13:37.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.615+0000 7f9a11600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c1612f0 con 0x7f9a0c160f10 2024-04-02T09:13:37.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.615+0000 7f9a11600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c0d1d30 con 0x7f9a0c0d18b0 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.615+0000 7f9a0b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9a0c161590 0x7f9a0c0b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56164/0 (socket says 172.21.15.89:56164) 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.615+0000 7f9a0b400700 1 -- 172.21.15.89:0/178174068 learned_addr learned my addr 172.21.15.89:0/178174068 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.615+0000 7f9a0a000700 1 --2- 172.21.15.89:0/178174068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a0c0d18b0 0x7f9a0c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858824727 0 0) 0x7f9a0c0ceda0 con 0x7f9a0c161590 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f00035b0 con 0x7f9a0c161590 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033932292 0 0) 0x7f9a0c1612f0 con 0x7f9a0c160f10 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a0c0ceda0 con 0x7f9a0c160f10 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2381712624 0 0) 0x7f99f00035b0 con 0x7f9a0c161590 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a0c0d18b0 msgr2=0x7f9a0c0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 --2- 172.21.15.89:0/178174068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a0c0d18b0 0x7f9a0c0cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 >> v1:172.21.15.113:6789/0 conn(0x7f9a0c160f10 legacy=0x7f9a0c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c0ce470 con 0x7f9a0c161590 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f9a040019f0 con 0x7f9a0c161590 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 >> v1:172.21.15.89:6789/0 conn(0x7f9a0c161590 legacy=0x7f9a0c0b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:37.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 --2- 172.21.15.89:0/178174068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a0c0d18b0 0x7f99f0003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 --2- 172.21.15.89:0/178174068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99f0004170 0x7f99f0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 --2- 172.21.15.89:0/178174068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f0004d90 0x7f99f0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c0d1d30 con 0x7f99f0004d90 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f000bbd0 con 0x7f99f0004170 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f000bd90 con 0x7f9a0c0d18b0 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a0b400700 1 --2- 172.21.15.89:0/178174068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f0004d90 0x7f99f0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a0aa00700 1 --2- 172.21.15.89:0/178174068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a0c0d18b0 0x7f99f0003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.616+0000 7f9a0a000700 1 --2- 172.21.15.89:0/178174068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99f0004170 0x7f99f0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.617+0000 7f9a0b400700 1 -- 172.21.15.89:0/178174068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a0c0d18b0 msgr2=0x7f99f0003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.617+0000 7f9a0b400700 1 --2- 172.21.15.89:0/178174068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a0c0d18b0 0x7f99f0003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.617+0000 7f9a0b400700 1 -- 172.21.15.89:0/178174068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99f0004170 msgr2=0x7f99f0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.617+0000 7f9a0b400700 1 --2- 172.21.15.89:0/178174068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99f0004170 0x7f99f0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.617+0000 7f9a0b400700 1 -- 172.21.15.89:0/178174068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f00035b0 con 0x7f99f0004d90 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.617+0000 7f9a0b400700 1 --2- 172.21.15.89:0/178174068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f0004d90 0x7f99f0005210 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f9a0400a710 tx=0x7f99f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b1bb59043ef82b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:37.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.617+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a040072d0 con 0x7f99f0004d90 2024-04-02T09:13:37.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.617+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a04007430 con 0x7f99f0004d90 2024-04-02T09:13:37.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.617+0000 7f9a09600700 1 -- 172.21.15.89:0/178174068 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a040077e0 con 0x7f99f0004d90 2024-04-02T09:13:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.618+0000 7f9a11600700 1 -- 172.21.15.89:0/178174068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f0004d90 msgr2=0x7f99f0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.618+0000 7f9a11600700 1 --2- 172.21.15.89:0/178174068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f0004d90 0x7f99f0005210 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f9a0400a710 tx=0x7f99f0005880 comp rx=0 tx=0).stop 2024-04-02T09:13:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.618+0000 7f9a0b400700 1 -- 172.21.15.89:0/178174068 reap_dead start 2024-04-02T09:13:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.618+0000 7f9a11600700 1 -- 172.21.15.89:0/178174068 shutdown_connections 2024-04-02T09:13:37.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.618+0000 7f9a11600700 1 -- 172.21.15.89:0/178174068 >> 172.21.15.89:0/178174068 conn(0x7f9a0c08aff0 msgr2=0x7f9a0c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.618+0000 7f9a11600700 1 -- 172.21.15.89:0/178174068 shutdown_connections 2024-04-02T09:13:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.618+0000 7f9a11600700 1 -- 172.21.15.89:0/178174068 wait complete. 2024-04-02T09:13:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.620+0000 7f9a11600700 1 Processor -- start 2024-04-02T09:13:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.620+0000 7f9a11600700 1 -- start start 2024-04-02T09:13:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a11600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99f0004d90 0x7f9a0c15b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a11600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a0c1e9510 0x7f9a0c0a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a11600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a0c15f5c0 0x7f9a0c15fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a11600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c15ff90 con 0x7f9a0c1e9510 2024-04-02T09:13:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a11600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c15b3d0 con 0x7f99f0004d90 2024-04-02T09:13:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a11600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c1e9990 con 0x7f9a0c15f5c0 2024-04-02T09:13:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a0a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a0c1e9510 0x7f9a0c0a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a0a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a0c1e9510 0x7f9a0c0a0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60892/0 (socket says 172.21.15.89:60892) 2024-04-02T09:13:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a0a000700 1 -- 172.21.15.89:0/3337929449 learned_addr learned my addr 172.21.15.89:0/3337929449 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:37.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a0b400700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a0c15f5c0 0x7f9a0c15fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:37.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a0aa00700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99f0004d90 0x7f9a0c15b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:37.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a0a000700 1 -- 172.21.15.89:0/3337929449 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a0c15f5c0 msgr2=0x7f9a0c15fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:37.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a0a000700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a0c15f5c0 0x7f9a0c15fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:37.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a0a000700 1 -- 172.21.15.89:0/3337929449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99f0004d90 msgr2=0x7f9a0c15b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:37.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a0a000700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99f0004d90 0x7f9a0c15b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:37.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a0a000700 1 -- 172.21.15.89:0/3337929449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c1e8250 con 0x7f9a0c1e9510 2024-04-02T09:13:37.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.621+0000 7f9a0b400700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a0c15f5c0 0x7f9a0c15fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:37.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.622+0000 7f9a0a000700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a0c1e9510 0x7f9a0c0a0fd0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f99f8011b00 tx=0x7f99f8011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:37.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.622+0000 7f9a02a00700 1 -- 172.21.15.89:0/3337929449 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f800f8e0 con 0x7f9a0c1e9510 2024-04-02T09:13:37.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.622+0000 7f9a02a00700 1 -- 172.21.15.89:0/3337929449 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f800fa40 con 0x7f9a0c1e9510 2024-04-02T09:13:37.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.622+0000 7f9a11600700 1 -- 172.21.15.89:0/3337929449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a0c1e7640 con 0x7f9a0c1e9510 2024-04-02T09:13:37.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.622+0000 7f9a02a00700 1 -- 172.21.15.89:0/3337929449 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f800fdf0 con 0x7f9a0c1e9510 2024-04-02T09:13:37.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.622+0000 7f9a11600700 1 -- 172.21.15.89:0/3337929449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a0c1e7c00 con 0x7f9a0c1e9510 2024-04-02T09:13:37.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.623+0000 7f9a11600700 1 -- 172.21.15.89:0/3337929449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a0c067720 con 0x7f9a0c1e9510 2024-04-02T09:13:37.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.624+0000 7f9a02a00700 1 -- 172.21.15.89:0/3337929449 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f99f8014440 con 0x7f9a0c1e9510 2024-04-02T09:13:37.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.625+0000 7f9a02a00700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f99ec06aed0 0x7f99ec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:37.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.625+0000 7f9a02a00700 1 -- 172.21.15.89:0/3337929449 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f99f808a0d0 con 0x7f9a0c1e9510 2024-04-02T09:13:37.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.625+0000 7f9a0aa00700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f99ec06aed0 0x7f99ec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:37.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.625+0000 7f9a0aa00700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f99ec06aed0 0x7f99ec06d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f99fc007fd0 tx=0x7f99fc009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:37.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.629+0000 7f9a02a00700 1 -- 172.21.15.89:0/3337929449 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f99f8054dc0 con 0x7f9a0c1e9510 2024-04-02T09:13:37.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.831+0000 7f9a11600700 1 -- 172.21.15.89:0/3337929449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a0c15bac0 con 0x7f9a0c1e9510 2024-04-02T09:13:37.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.833+0000 7f9a02a00700 1 -- 172.21.15.89:0/3337929449 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v244) v1 ==== 74+0+16736 (secure 0 0 0) 0x7f99f8022090 con 0x7f9a0c1e9510 2024-04-02T09:13:37.831 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:37.831 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":244,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:13:29.360152+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:13:26.654647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-02T09:13:27.342285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:13:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.835+0000 7f9a11600700 1 -- 172.21.15.89:0/3337929449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f99ec06aed0 msgr2=0x7f99ec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.835+0000 7f9a11600700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f99ec06aed0 0x7f99ec06d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f99fc007fd0 tx=0x7f99fc009040 comp rx=0 tx=0).stop 2024-04-02T09:13:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.835+0000 7f9a11600700 1 -- 172.21.15.89:0/3337929449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a0c1e9510 msgr2=0x7f9a0c0a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.835+0000 7f9a11600700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a0c1e9510 0x7f9a0c0a0fd0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f99f8011b00 tx=0x7f99f8011e10 comp rx=0 tx=0).stop 2024-04-02T09:13:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.835+0000 7f9a11600700 1 -- 172.21.15.89:0/3337929449 shutdown_connections 2024-04-02T09:13:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.835+0000 7f9a11600700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f99ec06aed0 0x7f99ec06d390 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.835+0000 7f9a11600700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99f0004d90 0x7f9a0c15b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.835+0000 7f9a11600700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9a0c15f5c0 0x7f9a0c15fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.835+0000 7f9a11600700 1 --2- 172.21.15.89:0/3337929449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a0c1e9510 0x7f9a0c0a0fd0 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.835+0000 7f9a11600700 1 -- 172.21.15.89:0/3337929449 >> 172.21.15.89:0/3337929449 conn(0x7f9a0c08aff0 msgr2=0x7f9a0c0cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.835+0000 7f9a11600700 1 -- 172.21.15.89:0/3337929449 shutdown_connections 2024-04-02T09:13:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:37.835+0000 7f9a11600700 1 -- 172.21.15.89:0/3337929449 wait complete. 2024-04-02T09:13:37.852 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T09:13:26.654647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '243', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T09:13:27.342285+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '244', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:13:37.853 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:13:38.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.047+0000 7f0b3f800700 1 Processor -- start 2024-04-02T09:13:38.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.047+0000 7f0b3f800700 1 -- start start 2024-04-02T09:13:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.048+0000 7f0b3f800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b380d16a0 0x7f0b380d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.048+0000 7f0b3f800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b38161420 con 0x7f0b380d2370 2024-04-02T09:13:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.048+0000 7f0b3f800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b38161700 con 0x7f0b3815d9d0 2024-04-02T09:13:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.048+0000 7f0b3f800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b380d4340 con 0x7f0b380d16a0 2024-04-02T09:13:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.049+0000 7f0b37e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b380d16a0 0x7f0b380d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.049+0000 7f0b37e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b380d16a0 0x7f0b380d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60896/0 (socket says 172.21.15.89:60896) 2024-04-02T09:13:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.049+0000 7f0b37e00700 1 -- 172.21.15.89:0/1468174556 learned_addr learned my addr 172.21.15.89:0/1468174556 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:38.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.049+0000 7f0b37e00700 1 -- 172.21.15.89:0/1468174556 >> v1:172.21.15.113:6789/0 conn(0x7f0b3815d9d0 legacy=0x7f0b380d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:38.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.049+0000 7f0b37e00700 1 -- 172.21.15.89:0/1468174556 >> v1:172.21.15.89:6789/0 conn(0x7f0b380d2370 legacy=0x7f0b380ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.049+0000 7f0b37e00700 1 -- 172.21.15.89:0/1468174556 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b381619e0 con 0x7f0b380d16a0 2024-04-02T09:13:38.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.049+0000 7f0b37e00700 1 --2- 172.21.15.89:0/1468174556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b380d16a0 0x7f0b380d1e30 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f0b20006fd0 tx=0x7f0b20009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd2f5fd88dfa93f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:38.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.049+0000 7f0b37400700 1 -- 172.21.15.89:0/1468174556 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20005e30 con 0x7f0b380d16a0 2024-04-02T09:13:38.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.050+0000 7f0b37400700 1 -- 172.21.15.89:0/1468174556 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b20006030 con 0x7f0b380d16a0 2024-04-02T09:13:38.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.050+0000 7f0b37400700 1 -- 172.21.15.89:0/1468174556 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b200063e0 con 0x7f0b380d16a0 2024-04-02T09:13:38.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.050+0000 7f0b3f800700 1 -- 172.21.15.89:0/1468174556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b380d16a0 msgr2=0x7f0b380d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.050+0000 7f0b3f800700 1 --2- 172.21.15.89:0/1468174556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b380d16a0 0x7f0b380d1e30 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f0b20006fd0 tx=0x7f0b20009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.050+0000 7f0b3f800700 1 -- 172.21.15.89:0/1468174556 shutdown_connections 2024-04-02T09:13:38.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.050+0000 7f0b3f800700 1 --2- 172.21.15.89:0/1468174556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b380d16a0 0x7f0b380d1e30 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.050+0000 7f0b3f800700 1 -- 172.21.15.89:0/1468174556 >> 172.21.15.89:0/1468174556 conn(0x7f0b3808aff0 msgr2=0x7f0b380d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:38.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.050+0000 7f0b3f800700 1 -- 172.21.15.89:0/1468174556 shutdown_connections 2024-04-02T09:13:38.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.050+0000 7f0b3f800700 1 -- 172.21.15.89:0/1468174556 wait complete. 2024-04-02T09:13:38.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.052+0000 7f0b3f800700 1 Processor -- start 2024-04-02T09:13:38.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.052+0000 7f0b3f800700 1 -- start start 2024-04-02T09:13:38.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.053+0000 7f0b3f800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b380d16a0 0x7f0b381e73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.053+0000 7f0b3f800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b380d2370 0x7f0b380b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.053+0000 7f0b3f800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3815d9d0 0x7f0b380b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.053+0000 7f0b3f800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3815eb60 con 0x7f0b3815d9d0 2024-04-02T09:13:38.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.053+0000 7f0b3f800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3815df50 con 0x7f0b380d16a0 2024-04-02T09:13:38.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.053+0000 7f0b3f800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3815e230 con 0x7f0b380d2370 2024-04-02T09:13:38.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.053+0000 7f0b3d800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3815d9d0 0x7f0b380b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.053+0000 7f0b3d800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3815d9d0 0x7f0b380b4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60906/0 (socket says 172.21.15.89:60906) 2024-04-02T09:13:38.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.053+0000 7f0b3d800700 1 -- 172.21.15.89:0/2238439701 learned_addr learned my addr 172.21.15.89:0/2238439701 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:38.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.054+0000 7f0b3d800700 1 -- 172.21.15.89:0/2238439701 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b380d2370 msgr2=0x7f0b380b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.053+0000 7f0b3ce00700 1 --2- 172.21.15.89:0/2238439701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b380d16a0 0x7f0b381e73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.054+0000 7f0b3d800700 1 --2- 172.21.15.89:0/2238439701 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b380d2370 0x7f0b380b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.054+0000 7f0b3d800700 1 -- 172.21.15.89:0/2238439701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b380d16a0 msgr2=0x7f0b381e73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.054+0000 7f0b3d800700 1 --2- 172.21.15.89:0/2238439701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b380d16a0 0x7f0b381e73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.054+0000 7f0b3d800700 1 -- 172.21.15.89:0/2238439701 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b20009500 con 0x7f0b3815d9d0 2024-04-02T09:13:38.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.054+0000 7f0b3ce00700 1 --2- 172.21.15.89:0/2238439701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b380d16a0 0x7f0b381e73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:38.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.054+0000 7f0b3d800700 1 --2- 172.21.15.89:0/2238439701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3815d9d0 0x7f0b380b4d50 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000c500 tx=0x7f0b3000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:38.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.054+0000 7f0b34c00700 1 -- 172.21.15.89:0/2238439701 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b300052d0 con 0x7f0b3815d9d0 2024-04-02T09:13:38.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.054+0000 7f0b34c00700 1 -- 172.21.15.89:0/2238439701 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b30005430 con 0x7f0b3815d9d0 2024-04-02T09:13:38.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.054+0000 7f0b3f800700 1 -- 172.21.15.89:0/2238439701 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3815e510 con 0x7f0b3815d9d0 2024-04-02T09:13:38.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.054+0000 7f0b34c00700 1 -- 172.21.15.89:0/2238439701 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b300038c0 con 0x7f0b3815d9d0 2024-04-02T09:13:38.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.054+0000 7f0b3f800700 1 -- 172.21.15.89:0/2238439701 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b381e78e0 con 0x7f0b3815d9d0 2024-04-02T09:13:38.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.055+0000 7f0b3f800700 1 -- 172.21.15.89:0/2238439701 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b380d1ca0 con 0x7f0b3815d9d0 2024-04-02T09:13:38.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.056+0000 7f0b34c00700 1 -- 172.21.15.89:0/2238439701 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0b30003a20 con 0x7f0b3815d9d0 2024-04-02T09:13:38.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.057+0000 7f0b34c00700 1 --2- 172.21.15.89:0/2238439701 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b1406aed0 0x7f0b1406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.057+0000 7f0b34c00700 1 -- 172.21.15.89:0/2238439701 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f0b30088cd0 con 0x7f0b3815d9d0 2024-04-02T09:13:38.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.057+0000 7f0b3ce00700 1 --2- 172.21.15.89:0/2238439701 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b1406aed0 0x7f0b1406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.058+0000 7f0b3ce00700 1 --2- 172.21.15.89:0/2238439701 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b1406aed0 0x7f0b1406d390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0b381f6a90 tx=0x7f0b280073c0 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:38.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.061+0000 7f0b34c00700 1 -- 172.21.15.89:0/2238439701 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b30053a50 con 0x7f0b3815d9d0 2024-04-02T09:13:38.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.265+0000 7f0b3f800700 1 -- 172.21.15.89:0/2238439701 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0b381e7a70 con 0x7f0b3815d9d0 2024-04-02T09:13:38.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.266+0000 7f0b34c00700 1 -- 172.21.15.89:0/2238439701 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v244) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0b300170a0 con 0x7f0b3815d9d0 2024-04-02T09:13:38.264 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:13:38.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.268+0000 7f0b3f800700 1 -- 172.21.15.89:0/2238439701 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b1406aed0 msgr2=0x7f0b1406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.268+0000 7f0b3f800700 1 --2- 172.21.15.89:0/2238439701 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b1406aed0 0x7f0b1406d390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0b381f6a90 tx=0x7f0b280073c0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.268+0000 7f0b3f800700 1 -- 172.21.15.89:0/2238439701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3815d9d0 msgr2=0x7f0b380b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.268+0000 7f0b3f800700 1 --2- 172.21.15.89:0/2238439701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3815d9d0 0x7f0b380b4d50 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000c500 tx=0x7f0b3000c8c0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.268+0000 7f0b3f800700 1 -- 172.21.15.89:0/2238439701 shutdown_connections 2024-04-02T09:13:38.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.268+0000 7f0b3f800700 1 --2- 172.21.15.89:0/2238439701 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b1406aed0 0x7f0b1406d390 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.268+0000 7f0b3f800700 1 --2- 172.21.15.89:0/2238439701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b380d16a0 0x7f0b381e73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.268+0000 7f0b3f800700 1 --2- 172.21.15.89:0/2238439701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3815d9d0 0x7f0b380b4d50 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.268+0000 7f0b3f800700 1 --2- 172.21.15.89:0/2238439701 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b380d2370 0x7f0b380b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.268+0000 7f0b3f800700 1 -- 172.21.15.89:0/2238439701 >> 172.21.15.89:0/2238439701 conn(0x7f0b3808aff0 msgr2=0x7f0b3806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:38.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.268+0000 7f0b3f800700 1 -- 172.21.15.89:0/2238439701 shutdown_connections 2024-04-02T09:13:38.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.268+0000 7f0b3f800700 1 -- 172.21.15.89:0/2238439701 wait complete. 2024-04-02T09:13:38.285 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T09:13:38.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.485+0000 7f0f51e00700 1 Processor -- start 2024-04-02T09:13:38.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.486+0000 7f0f51e00700 1 -- start start 2024-04-02T09:13:38.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.487+0000 7f0f51e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0f4c1611b0 0x7f0f4c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.487+0000 7f0f51e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f4c15f4d0 0x7f0f4c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.487+0000 7f0f51e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4c0ce270 con 0x7f0f4c0d21f0 2024-04-02T09:13:38.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.487+0000 7f0f51e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f4c15b600 con 0x7f0f4c1611b0 2024-04-02T09:13:38.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.487+0000 7f0f51e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c0d2760 con 0x7f0f4c15f4d0 2024-04-02T09:13:38.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.487+0000 7f0f4b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0f4c0d21f0 0x7f0f4c161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56178/0 (socket says 172.21.15.89:56178) 2024-04-02T09:13:38.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.487+0000 7f0f4b400700 1 -- 172.21.15.89:0/1876668780 learned_addr learned my addr 172.21.15.89:0/1876668780 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:38.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.487+0000 7f0f49600700 1 -- 172.21.15.89:0/1876668780 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460258205 0 0) 0x7f0f4c0ce270 con 0x7f0f4c0d21f0 2024-04-02T09:13:38.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.487+0000 7f0f4aa00700 1 --2- 172.21.15.89:0/1876668780 >> v2:172.21.15.113:3300/0 conn(0x7f0f4c1611b0 0x7f0f4c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.487+0000 7f0f49600700 1 -- 172.21.15.89:0/1876668780 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f300035b0 con 0x7f0f4c0d21f0 2024-04-02T09:13:38.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.487+0000 7f0f4a000700 1 --2- 172.21.15.89:0/1876668780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f4c15f4d0 0x7f0f4c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f49600700 1 -- 172.21.15.89:0/1876668780 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3284536900 0 0) 0x7f0f300035b0 con 0x7f0f4c0d21f0 2024-04-02T09:13:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f49600700 1 -- 172.21.15.89:0/1876668780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f4c15f4d0 msgr2=0x7f0f4c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f49600700 1 --2- 172.21.15.89:0/1876668780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f4c15f4d0 0x7f0f4c0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f49600700 1 -- 172.21.15.89:0/1876668780 >> v2:172.21.15.113:3300/0 conn(0x7f0f4c1611b0 msgr2=0x7f0f4c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f49600700 1 --2- 172.21.15.89:0/1876668780 >> v2:172.21.15.113:3300/0 conn(0x7f0f4c1611b0 0x7f0f4c15ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f49600700 1 -- 172.21.15.89:0/1876668780 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c0d2a40 con 0x7f0f4c0d21f0 2024-04-02T09:13:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f49600700 1 -- 172.21.15.89:0/1876668780 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f0f440019f0 con 0x7f0f4c0d21f0 2024-04-02T09:13:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f4a000700 1 --2- 172.21.15.89:0/1876668780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f4c15f4d0 0x7f0f4c0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f49600700 1 -- 172.21.15.89:0/1876668780 >> v1:172.21.15.89:6789/0 conn(0x7f0f4c0d21f0 legacy=0x7f0f4c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f49600700 1 --2- 172.21.15.89:0/1876668780 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f30003a00 0x7f0f30003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f49600700 1 --2- 172.21.15.89:0/1876668780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f4c15f4d0 0x7f0f30004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f49600700 1 --2- 172.21.15.89:0/1876668780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f30004d90 0x7f0f30005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f49600700 1 -- 172.21.15.89:0/1876668780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c15b600 con 0x7f0f30004d90 2024-04-02T09:13:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f49600700 1 -- 172.21.15.89:0/1876668780 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c0d2760 con 0x7f0f30003a00 2024-04-02T09:13:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f49600700 1 -- 172.21.15.89:0/1876668780 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3000ba10 con 0x7f0f4c15f4d0 2024-04-02T09:13:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f4b400700 1 --2- 172.21.15.89:0/1876668780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f30004d90 0x7f0f30005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f4aa00700 1 --2- 172.21.15.89:0/1876668780 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f30003a00 0x7f0f30003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f4a000700 1 --2- 172.21.15.89:0/1876668780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f4c15f4d0 0x7f0f30004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f4b400700 1 -- 172.21.15.89:0/1876668780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f4c15f4d0 msgr2=0x7f0f30004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f4b400700 1 --2- 172.21.15.89:0/1876668780 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f4c15f4d0 0x7f0f30004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f4b400700 1 -- 172.21.15.89:0/1876668780 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f30003a00 msgr2=0x7f0f30003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f4b400700 1 --2- 172.21.15.89:0/1876668780 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f30003a00 0x7f0f30003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.488+0000 7f0f4b400700 1 -- 172.21.15.89:0/1876668780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f300035b0 con 0x7f0f30004d90 2024-04-02T09:13:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.489+0000 7f0f4b400700 1 --2- 172.21.15.89:0/1876668780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f30004d90 0x7f0f30005210 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400a790 tx=0x7f0f30005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5776ac11028661e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.489+0000 7f0f49600700 1 -- 172.21.15.89:0/1876668780 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4400a250 con 0x7f0f30004d90 2024-04-02T09:13:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.489+0000 7f0f49600700 1 -- 172.21.15.89:0/1876668780 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f44012600 con 0x7f0f30004d90 2024-04-02T09:13:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.489+0000 7f0f49600700 1 -- 172.21.15.89:0/1876668780 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f440129b0 con 0x7f0f30004d90 2024-04-02T09:13:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.489+0000 7f0f51e00700 1 -- 172.21.15.89:0/1876668780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f30004d90 msgr2=0x7f0f30005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.489+0000 7f0f51e00700 1 --2- 172.21.15.89:0/1876668780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f30004d90 0x7f0f30005210 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400a790 tx=0x7f0f30005880 comp rx=0 tx=0).stop 2024-04-02T09:13:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.489+0000 7f0f4b400700 1 -- 172.21.15.89:0/1876668780 reap_dead start 2024-04-02T09:13:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.489+0000 7f0f51e00700 1 -- 172.21.15.89:0/1876668780 shutdown_connections 2024-04-02T09:13:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.489+0000 7f0f51e00700 1 -- 172.21.15.89:0/1876668780 >> 172.21.15.89:0/1876668780 conn(0x7f0f4c08aff0 msgr2=0x7f0f4c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.490+0000 7f0f51e00700 1 -- 172.21.15.89:0/1876668780 shutdown_connections 2024-04-02T09:13:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.490+0000 7f0f51e00700 1 -- 172.21.15.89:0/1876668780 wait complete. 2024-04-02T09:13:38.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.491+0000 7f0f51e00700 1 Processor -- start 2024-04-02T09:13:38.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.491+0000 7f0f51e00700 1 -- start start 2024-04-02T09:13:38.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.492+0000 7f0f51e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f30004d90 0x7f0f4c0b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.492+0000 7f0f51e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f4c0cd1b0 0x7f0f4c14c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.492+0000 7f0f51e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f4c0cbbe0 0x7f0f4c159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.492+0000 7f0f51e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c0ccd90 con 0x7f0f4c0cbbe0 2024-04-02T09:13:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.492+0000 7f0f51e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c0cc110 con 0x7f0f30004d90 2024-04-02T09:13:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.492+0000 7f0f51e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c0cc3f0 con 0x7f0f4c0cd1b0 2024-04-02T09:13:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.493+0000 7f0f4b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f4c0cbbe0 0x7f0f4c159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.493+0000 7f0f4b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f4c0cbbe0 0x7f0f4c159930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60924/0 (socket says 172.21.15.89:60924) 2024-04-02T09:13:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.493+0000 7f0f4b400700 1 -- 172.21.15.89:0/1929393730 learned_addr learned my addr 172.21.15.89:0/1929393730 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.493+0000 7f0f4a000700 1 --2- 172.21.15.89:0/1929393730 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f4c0cd1b0 0x7f0f4c14c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.493+0000 7f0f4aa00700 1 --2- 172.21.15.89:0/1929393730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f30004d90 0x7f0f4c0b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.493+0000 7f0f4b400700 1 -- 172.21.15.89:0/1929393730 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f4c0cd1b0 msgr2=0x7f0f4c14c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.493+0000 7f0f4b400700 1 --2- 172.21.15.89:0/1929393730 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f4c0cd1b0 0x7f0f4c14c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.493+0000 7f0f4b400700 1 -- 172.21.15.89:0/1929393730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f30004d90 msgr2=0x7f0f4c0b6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.493+0000 7f0f4b400700 1 --2- 172.21.15.89:0/1929393730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f30004d90 0x7f0f4c0b6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.493+0000 7f0f4b400700 1 -- 172.21.15.89:0/1929393730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c0cc6d0 con 0x7f0f4c0cbbe0 2024-04-02T09:13:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.493+0000 7f0f4b400700 1 --2- 172.21.15.89:0/1929393730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f4c0cbbe0 0x7f0f4c159930 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400afd0 tx=0x7f0f44001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.493+0000 7f0f43400700 1 -- 172.21.15.89:0/1929393730 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4400a250 con 0x7f0f4c0cbbe0 2024-04-02T09:13:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.494+0000 7f0f43400700 1 -- 172.21.15.89:0/1929393730 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4401e610 con 0x7f0f4c0cbbe0 2024-04-02T09:13:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.494+0000 7f0f51e00700 1 -- 172.21.15.89:0/1929393730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f4c0cc9b0 con 0x7f0f4c0cbbe0 2024-04-02T09:13:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.494+0000 7f0f43400700 1 -- 172.21.15.89:0/1929393730 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4401e7e0 con 0x7f0f4c0cbbe0 2024-04-02T09:13:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.494+0000 7f0f51e00700 1 -- 172.21.15.89:0/1929393730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4c14cce0 con 0x7f0f4c0cbbe0 2024-04-02T09:13:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.495+0000 7f0f51e00700 1 -- 172.21.15.89:0/1929393730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f4c0cdec0 con 0x7f0f4c0cbbe0 2024-04-02T09:13:38.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.496+0000 7f0f43400700 1 -- 172.21.15.89:0/1929393730 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0f44019030 con 0x7f0f4c0cbbe0 2024-04-02T09:13:38.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.496+0000 7f0f43400700 1 --2- 172.21.15.89:0/1929393730 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0f2c06aed0 0x7f0f2c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.497+0000 7f0f43400700 1 -- 172.21.15.89:0/1929393730 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f0f4408b740 con 0x7f0f4c0cbbe0 2024-04-02T09:13:38.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.497+0000 7f0f4aa00700 1 --2- 172.21.15.89:0/1929393730 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0f2c06aed0 0x7f0f2c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.497+0000 7f0f4aa00700 1 --2- 172.21.15.89:0/1929393730 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0f2c06aed0 0x7f0f2c06d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c009c60 tx=0x7f0f3c009210 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:38.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.501+0000 7f0f43400700 1 -- 172.21.15.89:0/1929393730 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0f440563f0 con 0x7f0f4c0cbbe0 2024-04-02T09:13:38.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.702+0000 7f0f51e00700 1 -- 172.21.15.89:0/1929393730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0f4c0ccb40 con 0x7f0f4c0cbbe0 2024-04-02T09:13:38.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.702+0000 7f0f43400700 1 -- 172.21.15.89:0/1929393730 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v244) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0f44059a10 con 0x7f0f4c0cbbe0 2024-04-02T09:13:38.701 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:13:38.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.705+0000 7f0f51e00700 1 -- 172.21.15.89:0/1929393730 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0f2c06aed0 msgr2=0x7f0f2c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.705+0000 7f0f51e00700 1 --2- 172.21.15.89:0/1929393730 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0f2c06aed0 0x7f0f2c06d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c009c60 tx=0x7f0f3c009210 comp rx=0 tx=0).stop 2024-04-02T09:13:38.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.705+0000 7f0f51e00700 1 -- 172.21.15.89:0/1929393730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f4c0cbbe0 msgr2=0x7f0f4c159930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.705+0000 7f0f51e00700 1 --2- 172.21.15.89:0/1929393730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f4c0cbbe0 0x7f0f4c159930 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400afd0 tx=0x7f0f44001cb0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.705+0000 7f0f51e00700 1 -- 172.21.15.89:0/1929393730 shutdown_connections 2024-04-02T09:13:38.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.705+0000 7f0f51e00700 1 --2- 172.21.15.89:0/1929393730 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0f2c06aed0 0x7f0f2c06d390 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.705+0000 7f0f51e00700 1 --2- 172.21.15.89:0/1929393730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f30004d90 0x7f0f4c0b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.705+0000 7f0f51e00700 1 --2- 172.21.15.89:0/1929393730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f4c0cbbe0 0x7f0f4c159930 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.705+0000 7f0f51e00700 1 --2- 172.21.15.89:0/1929393730 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0f4c0cd1b0 0x7f0f4c14c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.705+0000 7f0f51e00700 1 -- 172.21.15.89:0/1929393730 >> 172.21.15.89:0/1929393730 conn(0x7f0f4c08aff0 msgr2=0x7f0f4c06c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:38.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.705+0000 7f0f51e00700 1 -- 172.21.15.89:0/1929393730 shutdown_connections 2024-04-02T09:13:38.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.705+0000 7f0f51e00700 1 -- 172.21.15.89:0/1929393730 wait complete. 2024-04-02T09:13:38.721 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T09:13:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.917+0000 7fd1f6000700 1 Processor -- start 2024-04-02T09:13:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.917+0000 7fd1f6000700 1 -- start start 2024-04-02T09:13:38.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.919+0000 7fd1f6000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd1f01611b0 0x7fd1f015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.919+0000 7fd1f6000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f00ce270 con 0x7fd1f015f4d0 2024-04-02T09:13:38.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.919+0000 7fd1f6000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f015b600 con 0x7fd1f00cee10 2024-04-02T09:13:38.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.919+0000 7fd1f6000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1f00d2760 con 0x7fd1f01611b0 2024-04-02T09:13:38.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.919+0000 7fd1eea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd1f015f4d0 0x7fd1f00ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58290/0 (socket says 172.21.15.89:58290) 2024-04-02T09:13:38.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.919+0000 7fd1eea00700 1 -- 172.21.15.89:0/2468358534 learned_addr learned my addr 172.21.15.89:0/2468358534 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:38.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.919+0000 7fd1ef400700 1 --2- 172.21.15.89:0/2468358534 >> v2:172.21.15.113:3300/0 conn(0x7fd1f01611b0 0x7fd1f015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.919+0000 7fd1ee000700 1 -- 172.21.15.89:0/2468358534 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923988574 0 0) 0x7fd1f00ce270 con 0x7fd1f015f4d0 2024-04-02T09:13:38.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.919+0000 7fd1ee000700 1 -- 172.21.15.89:0/2468358534 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1d40035b0 con 0x7fd1f015f4d0 2024-04-02T09:13:38.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ee000700 1 -- 172.21.15.89:0/2468358534 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520209192 0 0) 0x7fd1d40035b0 con 0x7fd1f015f4d0 2024-04-02T09:13:38.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ee000700 1 -- 172.21.15.89:0/2468358534 >> v2:172.21.15.113:3300/0 conn(0x7fd1f01611b0 msgr2=0x7fd1f015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:38.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ee000700 1 --2- 172.21.15.89:0/2468358534 >> v2:172.21.15.113:3300/0 conn(0x7fd1f01611b0 0x7fd1f015ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ee000700 1 -- 172.21.15.89:0/2468358534 >> v1:172.21.15.113:6789/0 conn(0x7fd1f00cee10 legacy=0x7fd1f0161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ee000700 1 -- 172.21.15.89:0/2468358534 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f00d2a40 con 0x7fd1f015f4d0 2024-04-02T09:13:38.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ee000700 1 -- 172.21.15.89:0/2468358534 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fd1dc001670 con 0x7fd1f015f4d0 2024-04-02T09:13:38.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ee000700 1 -- 172.21.15.89:0/2468358534 >> v1:172.21.15.89:6789/0 conn(0x7fd1f015f4d0 legacy=0x7fd1f00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ee000700 1 --2- 172.21.15.89:0/2468358534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d4003a00 0x7fd1d4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ee000700 1 --2- 172.21.15.89:0/2468358534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1d4004850 0x7fd1d4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ee000700 1 --2- 172.21.15.89:0/2468358534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1d4005210 0x7fd1d400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ee000700 1 -- 172.21.15.89:0/2468358534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f00d2760 con 0x7fd1d4003a00 2024-04-02T09:13:38.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ee000700 1 -- 172.21.15.89:0/2468358534 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d400c0b0 con 0x7fd1d4005210 2024-04-02T09:13:38.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ee000700 1 -- 172.21.15.89:0/2468358534 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d400c270 con 0x7fd1d4004850 2024-04-02T09:13:38.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ef400700 1 --2- 172.21.15.89:0/2468358534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d4003a00 0x7fd1d4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1eea00700 1 --2- 172.21.15.89:0/2468358534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1d4004850 0x7fd1d4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1efe00700 1 --2- 172.21.15.89:0/2468358534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1d4005210 0x7fd1d400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ef400700 1 -- 172.21.15.89:0/2468358534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1d4004850 msgr2=0x7fd1d4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:38.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ef400700 1 --2- 172.21.15.89:0/2468358534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1d4004850 0x7fd1d4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ef400700 1 -- 172.21.15.89:0/2468358534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1d4005210 msgr2=0x7fd1d400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:38.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ef400700 1 --2- 172.21.15.89:0/2468358534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1d4005210 0x7fd1d400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1ef400700 1 -- 172.21.15.89:0/2468358534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d40035b0 con 0x7fd1d4003a00 2024-04-02T09:13:38.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1efe00700 1 -- 172.21.15.89:0/2468358534 reap_dead start 2024-04-02T09:13:38.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.920+0000 7fd1eea00700 1 --2- 172.21.15.89:0/2468358534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1d4004850 0x7fd1d4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:38.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.921+0000 7fd1ef400700 1 --2- 172.21.15.89:0/2468358534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d4003a00 0x7fd1d4003e20 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009b60 tx=0x7fd1d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7d5b4c4df5ed08 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:38.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.921+0000 7fd1ee000700 1 -- 172.21.15.89:0/2468358534 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e000e6c0 con 0x7fd1d4003a00 2024-04-02T09:13:38.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.921+0000 7fd1ee000700 1 -- 172.21.15.89:0/2468358534 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e000e820 con 0x7fd1d4003a00 2024-04-02T09:13:38.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.921+0000 7fd1ee000700 1 -- 172.21.15.89:0/2468358534 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e000eb90 con 0x7fd1d4003a00 2024-04-02T09:13:38.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.921+0000 7fd1f6000700 1 -- 172.21.15.89:0/2468358534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d4003a00 msgr2=0x7fd1d4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.921+0000 7fd1f6000700 1 --2- 172.21.15.89:0/2468358534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d4003a00 0x7fd1d4003e20 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009b60 tx=0x7fd1d4005880 comp rx=0 tx=0).stop 2024-04-02T09:13:38.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.921+0000 7fd1f6000700 1 -- 172.21.15.89:0/2468358534 shutdown_connections 2024-04-02T09:13:38.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.921+0000 7fd1f6000700 1 --2- 172.21.15.89:0/2468358534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d4003a00 0x7fd1d4003e20 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.921+0000 7fd1f6000700 1 -- 172.21.15.89:0/2468358534 >> 172.21.15.89:0/2468358534 conn(0x7fd1f008aff0 msgr2=0x7fd1f00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:38.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.922+0000 7fd1f6000700 1 -- 172.21.15.89:0/2468358534 shutdown_connections 2024-04-02T09:13:38.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.922+0000 7fd1f6000700 1 -- 172.21.15.89:0/2468358534 wait complete. 2024-04-02T09:13:38.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.923+0000 7fd1f6000700 1 Processor -- start 2024-04-02T09:13:38.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.923+0000 7fd1f6000700 1 -- start start 2024-04-02T09:13:38.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.924+0000 7fd1f6000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1d4003a00 0x7fd1f00c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.924+0000 7fd1f6000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f015abd0 0x7fd1f00b7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.924+0000 7fd1f6000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1f00cbf90 0x7fd1f00cc440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.924+0000 7fd1f6000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f00cc980 con 0x7fd1f015abd0 2024-04-02T09:13:38.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.924+0000 7fd1f6000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f015b050 con 0x7fd1d4003a00 2024-04-02T09:13:38.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.924+0000 7fd1f6000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f00b8c80 con 0x7fd1f00cbf90 2024-04-02T09:13:38.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1eea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f015abd0 0x7fd1f00b7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1eea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f015abd0 0x7fd1f00b7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55114/0 (socket says 172.21.15.89:55114) 2024-04-02T09:13:38.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1eea00700 1 -- 172.21.15.89:0/2333654653 learned_addr learned my addr 172.21.15.89:0/2333654653 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:38.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1ef400700 1 --2- 172.21.15.89:0/2333654653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1d4003a00 0x7fd1f00c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1efe00700 1 --2- 172.21.15.89:0/2333654653 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1f00cbf90 0x7fd1f00cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1eea00700 1 -- 172.21.15.89:0/2333654653 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1f00cbf90 msgr2=0x7fd1f00cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1eea00700 1 --2- 172.21.15.89:0/2333654653 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1f00cbf90 0x7fd1f00cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1eea00700 1 -- 172.21.15.89:0/2333654653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1d4003a00 msgr2=0x7fd1f00c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:38.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1eea00700 1 --2- 172.21.15.89:0/2333654653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1d4003a00 0x7fd1f00c9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:38.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1eea00700 1 -- 172.21.15.89:0/2333654653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f009e060 con 0x7fd1f015abd0 2024-04-02T09:13:38.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1eea00700 1 --2- 172.21.15.89:0/2333654653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f015abd0 0x7fd1f00b7bb0 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc000c00 tx=0x7fd1dc00edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:38.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1e7400700 1 -- 172.21.15.89:0/2333654653 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc005e00 con 0x7fd1f015abd0 2024-04-02T09:13:38.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1e7400700 1 -- 172.21.15.89:0/2333654653 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1dc00c5d0 con 0x7fd1f015abd0 2024-04-02T09:13:38.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1f6000700 1 -- 172.21.15.89:0/2333654653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f00b83e0 con 0x7fd1f015abd0 2024-04-02T09:13:38.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.925+0000 7fd1e7400700 1 -- 172.21.15.89:0/2333654653 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc00c960 con 0x7fd1f015abd0 2024-04-02T09:13:38.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.926+0000 7fd1f6000700 1 -- 172.21.15.89:0/2333654653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f00b89a0 con 0x7fd1f015abd0 2024-04-02T09:13:38.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.927+0000 7fd1f6000700 1 -- 172.21.15.89:0/2333654653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f009e1f0 con 0x7fd1f015abd0 2024-04-02T09:13:38.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.928+0000 7fd1e7400700 1 -- 172.21.15.89:0/2333654653 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fd1dc00cac0 con 0x7fd1f015abd0 2024-04-02T09:13:38.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.928+0000 7fd1e7400700 1 --2- 172.21.15.89:0/2333654653 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd1d006aed0 0x7fd1d006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:38.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.929+0000 7fd1e7400700 1 -- 172.21.15.89:0/2333654653 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fd1dc0893a0 con 0x7fd1f015abd0 2024-04-02T09:13:38.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.929+0000 7fd1ef400700 1 --2- 172.21.15.89:0/2333654653 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd1d006aed0 0x7fd1d006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:38.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.929+0000 7fd1ef400700 1 --2- 172.21.15.89:0/2333654653 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd1d006aed0 0x7fd1d006d390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00090b0 tx=0x7fd1e001e040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:38.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:38.933+0000 7fd1e7400700 1 -- 172.21.15.89:0/2333654653 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd1dc0541a0 con 0x7fd1f015abd0 2024-04-02T09:13:39.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.135+0000 7fd1f6000700 1 -- 172.21.15.89:0/2333654653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd1f009de80 con 0x7fd1f015abd0 2024-04-02T09:13:39.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.135+0000 7fd1e7400700 1 -- 172.21.15.89:0/2333654653 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v244) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd1dc0577c0 con 0x7fd1f015abd0 2024-04-02T09:13:39.133 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:13:39.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.137+0000 7fd1f6000700 1 -- 172.21.15.89:0/2333654653 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd1d006aed0 msgr2=0x7fd1d006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:39.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.137+0000 7fd1f6000700 1 --2- 172.21.15.89:0/2333654653 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd1d006aed0 0x7fd1d006d390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00090b0 tx=0x7fd1e001e040 comp rx=0 tx=0).stop 2024-04-02T09:13:39.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.137+0000 7fd1f6000700 1 -- 172.21.15.89:0/2333654653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f015abd0 msgr2=0x7fd1f00b7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:39.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.137+0000 7fd1f6000700 1 --2- 172.21.15.89:0/2333654653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f015abd0 0x7fd1f00b7bb0 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc000c00 tx=0x7fd1dc00edd0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.137+0000 7fd1f6000700 1 -- 172.21.15.89:0/2333654653 shutdown_connections 2024-04-02T09:13:39.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.137+0000 7fd1f6000700 1 --2- 172.21.15.89:0/2333654653 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd1d006aed0 0x7fd1d006d390 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.137+0000 7fd1f6000700 1 --2- 172.21.15.89:0/2333654653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1d4003a00 0x7fd1f00c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.137+0000 7fd1f6000700 1 --2- 172.21.15.89:0/2333654653 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd1f00cbf90 0x7fd1f00cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.137+0000 7fd1f6000700 1 --2- 172.21.15.89:0/2333654653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f015abd0 0x7fd1f00b7bb0 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.138+0000 7fd1f6000700 1 -- 172.21.15.89:0/2333654653 >> 172.21.15.89:0/2333654653 conn(0x7fd1f008aff0 msgr2=0x7fd1f006bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:39.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.138+0000 7fd1f6000700 1 -- 172.21.15.89:0/2333654653 shutdown_connections 2024-04-02T09:13:39.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.138+0000 7fd1f6000700 1 -- 172.21.15.89:0/2333654653 wait complete. 2024-04-02T09:13:39.154 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T09:13:39.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.347+0000 7fc6bcc00700 1 Processor -- start 2024-04-02T09:13:39.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.347+0000 7fc6bcc00700 1 -- start start 2024-04-02T09:13:39.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.348+0000 7fc6bcc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc6b8072580 0x7fc6b80b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:39.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.348+0000 7fc6bcc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80b2e50 0x7fc6b80709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:39.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.348+0000 7fc6bcc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b80b2180 con 0x7fc6b80b3af0 2024-04-02T09:13:39.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.348+0000 7fc6bcc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b815b100 con 0x7fc6b8072580 2024-04-02T09:13:39.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.348+0000 7fc6bcc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b815b3e0 con 0x7fc6b80b2e50 2024-04-02T09:13:39.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.349+0000 7fc6b5400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc6b8072580 0x7fc6b80b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:39.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.349+0000 7fc6b6800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80b2e50 0x7fc6b80709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:39.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.349+0000 7fc6b5400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc6b8072580 0x7fc6b80b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:51240/0 (socket says 172.21.15.89:51240) 2024-04-02T09:13:39.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.349+0000 7fc6b5e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fc6b80b3af0 0x7fc6b815a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:53218/0 (socket says 172.21.15.89:53218) 2024-04-02T09:13:39.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.349+0000 7fc6b5400700 1 -- 172.21.15.89:0/2187942375 learned_addr learned my addr 172.21.15.89:0/2187942375 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.349+0000 7fc6b5400700 1 -- 172.21.15.89:0/2187942375 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80b2e50 msgr2=0x7fc6b80709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.349+0000 7fc6b5400700 1 --2- 172.21.15.89:0/2187942375 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80b2e50 0x7fc6b80709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.349+0000 7fc6b5400700 1 -- 172.21.15.89:0/2187942375 >> v1:172.21.15.113:6789/0 conn(0x7fc6b80b3af0 legacy=0x7fc6b815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.349+0000 7fc6b5400700 1 -- 172.21.15.89:0/2187942375 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b80b23f0 con 0x7fc6b8072580 2024-04-02T09:13:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.349+0000 7fc6b5400700 1 --2- 172.21.15.89:0/2187942375 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b8072580 0x7fc6b80b2910 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc6a40098f0 tx=0x7fc6a4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74ba38c188e12c69 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.350+0000 7fc6b4a00700 1 -- 172.21.15.89:0/2187942375 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4011830 con 0x7fc6b8072580 2024-04-02T09:13:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.350+0000 7fc6b4a00700 1 -- 172.21.15.89:0/2187942375 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a4011990 con 0x7fc6b8072580 2024-04-02T09:13:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.350+0000 7fc6b4a00700 1 -- 172.21.15.89:0/2187942375 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4011d20 con 0x7fc6b8072580 2024-04-02T09:13:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.350+0000 7fc6bcc00700 1 -- 172.21.15.89:0/2187942375 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b8072580 msgr2=0x7fc6b80b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.350+0000 7fc6bcc00700 1 --2- 172.21.15.89:0/2187942375 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b8072580 0x7fc6b80b2910 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc6a40098f0 tx=0x7fc6a4009c00 comp rx=0 tx=0).stop 2024-04-02T09:13:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.350+0000 7fc6bcc00700 1 -- 172.21.15.89:0/2187942375 shutdown_connections 2024-04-02T09:13:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.350+0000 7fc6bcc00700 1 --2- 172.21.15.89:0/2187942375 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80b2e50 0x7fc6b80709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.350+0000 7fc6bcc00700 1 --2- 172.21.15.89:0/2187942375 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b8072580 0x7fc6b80b2910 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.350+0000 7fc6bcc00700 1 -- 172.21.15.89:0/2187942375 >> 172.21.15.89:0/2187942375 conn(0x7fc6b80cf190 msgr2=0x7fc6b80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:39.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.350+0000 7fc6bcc00700 1 -- 172.21.15.89:0/2187942375 shutdown_connections 2024-04-02T09:13:39.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.350+0000 7fc6bcc00700 1 -- 172.21.15.89:0/2187942375 wait complete. 2024-04-02T09:13:39.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.352+0000 7fc6bcc00700 1 Processor -- start 2024-04-02T09:13:39.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.352+0000 7fc6bcc00700 1 -- start start 2024-04-02T09:13:39.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.353+0000 7fc6bcc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b8072580 0x7fc6b80cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.353+0000 7fc6bcc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6b80b2e50 0x7fc6b80ce690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.353+0000 7fc6bcc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80b3af0 0x7fc6b80987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.353+0000 7fc6bcc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80ccb00 con 0x7fc6b80b2e50 2024-04-02T09:13:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.353+0000 7fc6bcc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80ccde0 con 0x7fc6b8072580 2024-04-02T09:13:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.353+0000 7fc6bcc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80cd0c0 con 0x7fc6b80b3af0 2024-04-02T09:13:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.353+0000 7fc6b5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6b80b2e50 0x7fc6b80ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.353+0000 7fc6b5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6b80b2e50 0x7fc6b80ce690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55128/0 (socket says 172.21.15.89:55128) 2024-04-02T09:13:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.353+0000 7fc6b5400700 1 -- 172.21.15.89:0/3591471592 learned_addr learned my addr 172.21.15.89:0/3591471592 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.353+0000 7fc6b5e00700 1 --2- 172.21.15.89:0/3591471592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b8072580 0x7fc6b80cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.354+0000 7fc6b6800700 1 --2- 172.21.15.89:0/3591471592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80b3af0 0x7fc6b80987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.354+0000 7fc6b5400700 1 -- 172.21.15.89:0/3591471592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80b3af0 msgr2=0x7fc6b80987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.354+0000 7fc6b5400700 1 --2- 172.21.15.89:0/3591471592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80b3af0 0x7fc6b80987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.354+0000 7fc6b5400700 1 -- 172.21.15.89:0/3591471592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b8072580 msgr2=0x7fc6b80cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.354+0000 7fc6b5400700 1 --2- 172.21.15.89:0/3591471592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b8072580 0x7fc6b80cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.354+0000 7fc6b5400700 1 -- 172.21.15.89:0/3591471592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a4009550 con 0x7fc6b80b2e50 2024-04-02T09:13:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.354+0000 7fc6b5400700 1 --2- 172.21.15.89:0/3591471592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6b80b2e50 0x7fc6b80ce690 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4000c00 tx=0x7fc6a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.354+0000 7fc6ae000700 1 -- 172.21.15.89:0/3591471592 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4004400 con 0x7fc6b80b2e50 2024-04-02T09:13:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.354+0000 7fc6ae000700 1 -- 172.21.15.89:0/3591471592 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a401f070 con 0x7fc6b80b2e50 2024-04-02T09:13:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.354+0000 7fc6ae000700 1 -- 172.21.15.89:0/3591471592 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a40038e0 con 0x7fc6b80b2e50 2024-04-02T09:13:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.354+0000 7fc6bcc00700 1 -- 172.21.15.89:0/3591471592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b80cd3a0 con 0x7fc6b80b2e50 2024-04-02T09:13:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.355+0000 7fc6bcc00700 1 -- 172.21.15.89:0/3591471592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b80cc740 con 0x7fc6b80b2e50 2024-04-02T09:13:39.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.356+0000 7fc6bcc00700 1 -- 172.21.15.89:0/3591471592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b80cc8d0 con 0x7fc6b80b2e50 2024-04-02T09:13:39.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.356+0000 7fc6ae000700 1 -- 172.21.15.89:0/3591471592 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fc6a4021030 con 0x7fc6b80b2e50 2024-04-02T09:13:39.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.358+0000 7fc6ae000700 1 --2- 172.21.15.89:0/3591471592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc69806aed0 0x7fc69806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:39.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.358+0000 7fc6ae000700 1 -- 172.21.15.89:0/3591471592 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fc6a408ad50 con 0x7fc6b80b2e50 2024-04-02T09:13:39.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.358+0000 7fc6b5e00700 1 --2- 172.21.15.89:0/3591471592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc69806aed0 0x7fc69806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:39.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.358+0000 7fc6b5e00700 1 --2- 172.21.15.89:0/3591471592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc69806aed0 0x7fc69806d390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8000f60 tx=0x7fc6a8007ed0 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:39.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.363+0000 7fc6ae000700 1 -- 172.21.15.89:0/3591471592 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc6a4055980 con 0x7fc6b80b2e50 2024-04-02T09:13:39.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.568+0000 7fc6bcc00700 1 -- 172.21.15.89:0/3591471592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc6b80cd8c0 con 0x7fc6b80b2e50 2024-04-02T09:13:39.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.569+0000 7fc6ae000700 1 -- 172.21.15.89:0/3591471592 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v244) v1 ==== 84+0+209 (secure 0 0 0) 0x7fc6a40150f0 con 0x7fc6b80b2e50 2024-04-02T09:13:39.567 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:39.567 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T09:13:39.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:39.568 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T09:13:39.568 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T09:13:39.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:39.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:39.568 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 30, 2024-04-02T09:13:39.568 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": "cephfs_metadata" 2024-04-02T09:13:39.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:39.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:39.568 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 31, 2024-04-02T09:13:39.568 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": "cephfs_data" 2024-04-02T09:13:39.568 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:39.568 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T09:13:39.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.571+0000 7fc6bcc00700 1 -- 172.21.15.89:0/3591471592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc69806aed0 msgr2=0x7fc69806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:39.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.571+0000 7fc6bcc00700 1 --2- 172.21.15.89:0/3591471592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc69806aed0 0x7fc69806d390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8000f60 tx=0x7fc6a8007ed0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.571+0000 7fc6bcc00700 1 -- 172.21.15.89:0/3591471592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6b80b2e50 msgr2=0x7fc6b80ce690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:39.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.571+0000 7fc6bcc00700 1 --2- 172.21.15.89:0/3591471592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6b80b2e50 0x7fc6b80ce690 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4000c00 tx=0x7fc6a4004750 comp rx=0 tx=0).stop 2024-04-02T09:13:39.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.571+0000 7fc6bcc00700 1 -- 172.21.15.89:0/3591471592 shutdown_connections 2024-04-02T09:13:39.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.571+0000 7fc6bcc00700 1 --2- 172.21.15.89:0/3591471592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc69806aed0 0x7fc69806d390 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.571+0000 7fc6bcc00700 1 --2- 172.21.15.89:0/3591471592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b8072580 0x7fc6b80cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.571+0000 7fc6bcc00700 1 --2- 172.21.15.89:0/3591471592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc6b80b3af0 0x7fc6b80987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.571+0000 7fc6bcc00700 1 --2- 172.21.15.89:0/3591471592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6b80b2e50 0x7fc6b80ce690 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.571+0000 7fc6bcc00700 1 -- 172.21.15.89:0/3591471592 >> 172.21.15.89:0/3591471592 conn(0x7fc6b80cf190 msgr2=0x7fc6b80ae460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.571+0000 7fc6bcc00700 1 -- 172.21.15.89:0/3591471592 shutdown_connections 2024-04-02T09:13:39.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.572+0000 7fc6bcc00700 1 -- 172.21.15.89:0/3591471592 wait complete. 2024-04-02T09:13:39.588 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-04-02T09:13:39.588 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-04-02T09:13:39.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.784+0000 7fb90ca00700 1 Processor -- start 2024-04-02T09:13:39.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.785+0000 7fb90ca00700 1 -- start start 2024-04-02T09:13:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.786+0000 7fb90ca00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb908160e10 0x7fb908099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.786+0000 7fb90ca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9080b2380 0x7fb908099fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.786+0000 7fb90ca00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90809ada0 con 0x7fb90809a4e0 2024-04-02T09:13:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.786+0000 7fb90ca00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9081611f0 con 0x7fb908160e10 2024-04-02T09:13:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.786+0000 7fb90ca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90806cc80 con 0x7fb9080b2380 2024-04-02T09:13:39.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.786+0000 7fb905e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb908160e10 0x7fb908099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:39.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.786+0000 7fb906800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb90809a4e0 0x7fb9080cf530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58292/0 (socket says 172.21.15.89:58292) 2024-04-02T09:13:39.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.786+0000 7fb906800700 1 -- 172.21.15.89:0/1694970933 learned_addr learned my addr 172.21.15.89:0/1694970933 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.786+0000 7fb905400700 1 --2- 172.21.15.89:0/1694970933 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9080b2380 0x7fb908099fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.786+0000 7fb905e00700 1 -- 172.21.15.89:0/1694970933 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9080b2380 msgr2=0x7fb908099fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.786+0000 7fb905e00700 1 --2- 172.21.15.89:0/1694970933 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9080b2380 0x7fb908099fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.786+0000 7fb905e00700 1 -- 172.21.15.89:0/1694970933 >> v1:172.21.15.89:6789/0 conn(0x7fb90809a4e0 legacy=0x7fb9080cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.787+0000 7fb905e00700 1 -- 172.21.15.89:0/1694970933 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9080d0840 con 0x7fb908160e10 2024-04-02T09:13:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.787+0000 7fb904a00700 1 -- 172.21.15.89:0/1694970933 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210994103 0 0) 0x7fb90809ada0 con 0x7fb90809a4e0 2024-04-02T09:13:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.787+0000 7fb905e00700 1 --2- 172.21.15.89:0/1694970933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb908160e10 0x7fb908099a60 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8009840 tx=0x7fb8f8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3200bccad4d32a45 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.787+0000 7fb904a00700 1 -- 172.21.15.89:0/1694970933 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8011830 con 0x7fb908160e10 2024-04-02T09:13:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.787+0000 7fb904a00700 1 -- 172.21.15.89:0/1694970933 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f8011990 con 0x7fb908160e10 2024-04-02T09:13:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.787+0000 7fb904a00700 1 -- 172.21.15.89:0/1694970933 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8011d20 con 0x7fb908160e10 2024-04-02T09:13:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.788+0000 7fb90ca00700 1 -- 172.21.15.89:0/1694970933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb908160e10 msgr2=0x7fb908099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.788+0000 7fb90ca00700 1 --2- 172.21.15.89:0/1694970933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb908160e10 0x7fb908099a60 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8009840 tx=0x7fb8f8009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.788+0000 7fb90ca00700 1 -- 172.21.15.89:0/1694970933 shutdown_connections 2024-04-02T09:13:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.788+0000 7fb90ca00700 1 --2- 172.21.15.89:0/1694970933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb908160e10 0x7fb908099a60 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.788+0000 7fb90ca00700 1 --2- 172.21.15.89:0/1694970933 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9080b2380 0x7fb908099fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.788+0000 7fb90ca00700 1 -- 172.21.15.89:0/1694970933 >> 172.21.15.89:0/1694970933 conn(0x7fb9080b2e20 msgr2=0x7fb9080b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.788+0000 7fb90ca00700 1 -- 172.21.15.89:0/1694970933 shutdown_connections 2024-04-02T09:13:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.788+0000 7fb90ca00700 1 -- 172.21.15.89:0/1694970933 wait complete. 2024-04-02T09:13:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.789+0000 7fb90ca00700 1 Processor -- start 2024-04-02T09:13:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.790+0000 7fb90ca00700 1 -- start start 2024-04-02T09:13:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.790+0000 7fb90ca00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb90809a4e0 0x7fb908148380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.790+0000 7fb90ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9080b2380 0x7fb908148ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.790+0000 7fb90ca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb908160e10 0x7fb908149000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.790+0000 7fb90ca00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9080c7a40 con 0x7fb9080b2380 2024-04-02T09:13:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.790+0000 7fb90ca00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90815edf0 con 0x7fb90809a4e0 2024-04-02T09:13:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.790+0000 7fb90ca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90815f0d0 con 0x7fb908160e10 2024-04-02T09:13:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.791+0000 7fb905400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9080b2380 0x7fb908148ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.791+0000 7fb905400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9080b2380 0x7fb908148ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55148/0 (socket says 172.21.15.89:55148) 2024-04-02T09:13:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.791+0000 7fb905400700 1 -- 172.21.15.89:0/2925316336 learned_addr learned my addr 172.21.15.89:0/2925316336 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.791+0000 7fb906800700 1 --2- 172.21.15.89:0/2925316336 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb908160e10 0x7fb908149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.791+0000 7fb905e00700 1 --2- 172.21.15.89:0/2925316336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb90809a4e0 0x7fb908148380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.791+0000 7fb905400700 1 -- 172.21.15.89:0/2925316336 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb908160e10 msgr2=0x7fb908149000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.791+0000 7fb905400700 1 --2- 172.21.15.89:0/2925316336 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb908160e10 0x7fb908149000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.791+0000 7fb905400700 1 -- 172.21.15.89:0/2925316336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb90809a4e0 msgr2=0x7fb908148380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.791+0000 7fb905400700 1 --2- 172.21.15.89:0/2925316336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb90809a4e0 0x7fb908148380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.791+0000 7fb905400700 1 -- 172.21.15.89:0/2925316336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f80094a0 con 0x7fb9080b2380 2024-04-02T09:13:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.791+0000 7fb905400700 1 --2- 172.21.15.89:0/2925316336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9080b2380 0x7fb908148ac0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7fb8f400e810 tx=0x7fb8f400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.791+0000 7fb8fe000700 1 -- 172.21.15.89:0/2925316336 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4013610 con 0x7fb9080b2380 2024-04-02T09:13:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.792+0000 7fb8fe000700 1 -- 172.21.15.89:0/2925316336 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f4013770 con 0x7fb9080b2380 2024-04-02T09:13:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.792+0000 7fb8fe000700 1 -- 172.21.15.89:0/2925316336 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4013b20 con 0x7fb9080b2380 2024-04-02T09:13:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.792+0000 7fb90ca00700 1 -- 172.21.15.89:0/2925316336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90815f3b0 con 0x7fb9080b2380 2024-04-02T09:13:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.792+0000 7fb90ca00700 1 -- 172.21.15.89:0/2925316336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90815ae40 con 0x7fb9080b2380 2024-04-02T09:13:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.793+0000 7fb90ca00700 1 -- 172.21.15.89:0/2925316336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90815ec50 con 0x7fb9080b2380 2024-04-02T09:13:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.794+0000 7fb8fe000700 1 -- 172.21.15.89:0/2925316336 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fb8f4013c80 con 0x7fb9080b2380 2024-04-02T09:13:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.794+0000 7fb8fe000700 1 --2- 172.21.15.89:0/2925316336 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb8e806aed0 0x7fb8e806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.795+0000 7fb8fe000700 1 -- 172.21.15.89:0/2925316336 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fb8f40896b0 con 0x7fb9080b2380 2024-04-02T09:13:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.795+0000 7fb905e00700 1 --2- 172.21.15.89:0/2925316336 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb8e806aed0 0x7fb8e806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.795+0000 7fb905e00700 1 --2- 172.21.15.89:0/2925316336 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb8e806aed0 0x7fb8e806d390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb8f80090b0 tx=0x7fb8f801c040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:39.799+0000 7fb8fe000700 1 -- 172.21.15.89:0/2925316336 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb90815ec50 con 0x7fb9080b2380 2024-04-02T09:13:40.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.002+0000 7fb90ca00700 1 -- 172.21.15.89:0/2925316336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb90815ec50 con 0x7fb9080b2380 2024-04-02T09:13:40.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.422+0000 7fb8fe000700 1 -- 172.21.15.89:0/2925316336 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v245) v1 ==== 144+0+0 (secure 0 0 0) 0x7fb90815ec50 con 0x7fb9080b2380 2024-04-02T09:13:40.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.427+0000 7fb90ca00700 1 -- 172.21.15.89:0/2925316336 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb8e806aed0 msgr2=0x7fb8e806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:40.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.427+0000 7fb90ca00700 1 --2- 172.21.15.89:0/2925316336 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb8e806aed0 0x7fb8e806d390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb8f80090b0 tx=0x7fb8f801c040 comp rx=0 tx=0).stop 2024-04-02T09:13:40.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.427+0000 7fb90ca00700 1 -- 172.21.15.89:0/2925316336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9080b2380 msgr2=0x7fb908148ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:40.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.427+0000 7fb90ca00700 1 --2- 172.21.15.89:0/2925316336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9080b2380 0x7fb908148ac0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7fb8f400e810 tx=0x7fb8f400eb20 comp rx=0 tx=0).stop 2024-04-02T09:13:40.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.428+0000 7fb90ca00700 1 -- 172.21.15.89:0/2925316336 shutdown_connections 2024-04-02T09:13:40.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.428+0000 7fb90ca00700 1 --2- 172.21.15.89:0/2925316336 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb8e806aed0 0x7fb8e806d390 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:40.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.428+0000 7fb90ca00700 1 --2- 172.21.15.89:0/2925316336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb90809a4e0 0x7fb908148380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:40.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.428+0000 7fb90ca00700 1 --2- 172.21.15.89:0/2925316336 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb908160e10 0x7fb908149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:40.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.428+0000 7fb90ca00700 1 --2- 172.21.15.89:0/2925316336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9080b2380 0x7fb908148ac0 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:40.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.428+0000 7fb90ca00700 1 -- 172.21.15.89:0/2925316336 >> 172.21.15.89:0/2925316336 conn(0x7fb9080b2e20 msgr2=0x7fb90815c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:40.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.428+0000 7fb90ca00700 1 -- 172.21.15.89:0/2925316336 shutdown_connections 2024-04-02T09:13:40.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.429+0000 7fb90ca00700 1 -- 172.21.15.89:0/2925316336 wait complete. 2024-04-02T09:13:40.428 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'backup_fs_metadata' created 2024-04-02T09:13:40.446 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T09:13:40.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.647+0000 7fea3c400700 1 Processor -- start 2024-04-02T09:13:40.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.647+0000 7fea3c400700 1 -- start start 2024-04-02T09:13:40.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.648+0000 7fea3c400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fea34160cc0 0x7fea34168730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:40.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.648+0000 7fea3c400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3415f820 con 0x7fea3415dad0 2024-04-02T09:13:40.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.648+0000 7fea3c400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea340d2450 con 0x7fea340ced80 2024-04-02T09:13:40.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.648+0000 7fea3c400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea340cfcc0 con 0x7fea34160cc0 2024-04-02T09:13:40.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.648+0000 7fea3a400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fea34160cc0 0x7fea34168730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:40.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.649+0000 7fea3a400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fea34160cc0 0x7fea34168730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55160/0 (socket says 172.21.15.89:55160) 2024-04-02T09:13:40.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.649+0000 7fea3a400700 1 -- 172.21.15.89:0/1430283822 learned_addr learned my addr 172.21.15.89:0/1430283822 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.649+0000 7fea3a400700 1 -- 172.21.15.89:0/1430283822 >> v1:172.21.15.113:6789/0 conn(0x7fea340ced80 legacy=0x7fea340d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.649+0000 7fea3a400700 1 -- 172.21.15.89:0/1430283822 >> v1:172.21.15.89:6789/0 conn(0x7fea3415dad0 legacy=0x7fea340d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.649+0000 7fea3a400700 1 -- 172.21.15.89:0/1430283822 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea340cf420 con 0x7fea34160cc0 2024-04-02T09:13:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.649+0000 7fea33e00700 1 -- 172.21.15.89:0/1430283822 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861752819 0 0) 0x7fea3415f820 con 0x7fea3415dad0 2024-04-02T09:13:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.649+0000 7fea3a400700 1 --2- 172.21.15.89:0/1430283822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea34160cc0 0x7fea34168730 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009e50 tx=0x7fea2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563c3ad93f87ae6d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.650+0000 7fea33e00700 1 -- 172.21.15.89:0/1430283822 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c00e070 con 0x7fea34160cc0 2024-04-02T09:13:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.650+0000 7fea33e00700 1 -- 172.21.15.89:0/1430283822 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2c00b9c0 con 0x7fea34160cc0 2024-04-02T09:13:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.650+0000 7fea33e00700 1 -- 172.21.15.89:0/1430283822 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c00bd70 con 0x7fea34160cc0 2024-04-02T09:13:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.650+0000 7fea3c400700 1 -- 172.21.15.89:0/1430283822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea34160cc0 msgr2=0x7fea34168730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.650+0000 7fea3c400700 1 --2- 172.21.15.89:0/1430283822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea34160cc0 0x7fea34168730 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009e50 tx=0x7fea2c00b330 comp rx=0 tx=0).stop 2024-04-02T09:13:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.650+0000 7fea3c400700 1 -- 172.21.15.89:0/1430283822 shutdown_connections 2024-04-02T09:13:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.650+0000 7fea3c400700 1 --2- 172.21.15.89:0/1430283822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea34160cc0 0x7fea34168730 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:40.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.650+0000 7fea3c400700 1 -- 172.21.15.89:0/1430283822 >> 172.21.15.89:0/1430283822 conn(0x7fea3408aff0 msgr2=0x7fea340d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:40.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.650+0000 7fea3c400700 1 -- 172.21.15.89:0/1430283822 shutdown_connections 2024-04-02T09:13:40.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.650+0000 7fea3c400700 1 -- 172.21.15.89:0/1430283822 wait complete. 2024-04-02T09:13:40.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.652+0000 7fea3c400700 1 Processor -- start 2024-04-02T09:13:40.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.652+0000 7fea3c400700 1 -- start start 2024-04-02T09:13:40.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.653+0000 7fea3c400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea340ced80 0x7fea340c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:40.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.653+0000 7fea3c400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea3415dad0 0x7fea340b8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:40.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.653+0000 7fea3c400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fea34160cc0 0x7fea3414f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:40.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.653+0000 7fea3c400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea341f34a0 con 0x7fea340ced80 2024-04-02T09:13:40.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.653+0000 7fea3c400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea340cb640 con 0x7fea3415dad0 2024-04-02T09:13:40.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.653+0000 7fea3c400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea340c9c40 con 0x7fea34160cc0 2024-04-02T09:13:40.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.653+0000 7fea39a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea340ced80 0x7fea340c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:40.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.653+0000 7fea39a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea340ced80 0x7fea340c7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55176/0 (socket says 172.21.15.89:55176) 2024-04-02T09:13:40.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.653+0000 7fea39a00700 1 -- 172.21.15.89:0/1281189470 learned_addr learned my addr 172.21.15.89:0/1281189470 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:40.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.654+0000 7fea39000700 1 --2- 172.21.15.89:0/1281189470 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea3415dad0 0x7fea340b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:40.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.654+0000 7fea3a400700 1 --2- 172.21.15.89:0/1281189470 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fea34160cc0 0x7fea3414f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:40.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.654+0000 7fea39a00700 1 -- 172.21.15.89:0/1281189470 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fea34160cc0 msgr2=0x7fea3414f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:40.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.654+0000 7fea39a00700 1 --2- 172.21.15.89:0/1281189470 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fea34160cc0 0x7fea3414f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:40.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.654+0000 7fea39a00700 1 -- 172.21.15.89:0/1281189470 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea3415dad0 msgr2=0x7fea340b8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:40.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.654+0000 7fea39a00700 1 --2- 172.21.15.89:0/1281189470 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea3415dad0 0x7fea340b8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:40.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.654+0000 7fea39a00700 1 -- 172.21.15.89:0/1281189470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2c009b30 con 0x7fea340ced80 2024-04-02T09:13:40.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.654+0000 7fea39a00700 1 --2- 172.21.15.89:0/1281189470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea340ced80 0x7fea340c7ab0 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fea28001eb0 tx=0x7fea2800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:40.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.654+0000 7fea31600700 1 -- 172.21.15.89:0/1281189470 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea280054e0 con 0x7fea340ced80 2024-04-02T09:13:40.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.654+0000 7fea31600700 1 -- 172.21.15.89:0/1281189470 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea28003680 con 0x7fea340ced80 2024-04-02T09:13:40.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.654+0000 7fea31600700 1 -- 172.21.15.89:0/1281189470 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea280039f0 con 0x7fea340ced80 2024-04-02T09:13:40.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.654+0000 7fea3c400700 1 -- 172.21.15.89:0/1281189470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea340c9f20 con 0x7fea340ced80 2024-04-02T09:13:40.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.654+0000 7fea3c400700 1 -- 172.21.15.89:0/1281189470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea340cb8e0 con 0x7fea340ced80 2024-04-02T09:13:40.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.655+0000 7fea3c400700 1 -- 172.21.15.89:0/1281189470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea34067720 con 0x7fea340ced80 2024-04-02T09:13:40.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.657+0000 7fea31600700 1 -- 172.21.15.89:0/1281189470 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fea28003b50 con 0x7fea340ced80 2024-04-02T09:13:40.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.658+0000 7fea31600700 1 --2- 172.21.15.89:0/1281189470 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fea1406aed0 0x7fea1406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:40.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.658+0000 7fea31600700 1 -- 172.21.15.89:0/1281189470 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6530+0+0 (secure 0 0 0) 0x7fea28089310 con 0x7fea340ced80 2024-04-02T09:13:40.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.658+0000 7fea39000700 1 --2- 172.21.15.89:0/1281189470 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fea1406aed0 0x7fea1406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:40.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.659+0000 7fea39000700 1 --2- 172.21.15.89:0/1281189470 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fea1406aed0 0x7fea1406d390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fea340b70a0 tx=0x7fea20009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:40.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.662+0000 7fea31600700 1 -- 172.21.15.89:0/1281189470 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fea28053f70 con 0x7fea340ced80 2024-04-02T09:13:40.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:40.863+0000 7fea3c400700 1 -- 172.21.15.89:0/1281189470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fea3406c180 con 0x7fea340ced80 2024-04-02T09:13:41.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.434+0000 7fea31600700 1 -- 172.21.15.89:0/1281189470 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v246) v1 ==== 176+0+0 (secure 0 0 0) 0x7fea28017020 con 0x7fea340ced80 2024-04-02T09:13:41.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.442+0000 7fea3c400700 1 -- 172.21.15.89:0/1281189470 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fea1406aed0 msgr2=0x7fea1406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:41.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.442+0000 7fea3c400700 1 --2- 172.21.15.89:0/1281189470 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fea1406aed0 0x7fea1406d390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fea340b70a0 tx=0x7fea20009040 comp rx=0 tx=0).stop 2024-04-02T09:13:41.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.442+0000 7fea3c400700 1 -- 172.21.15.89:0/1281189470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea340ced80 msgr2=0x7fea340c7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:41.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.442+0000 7fea3c400700 1 --2- 172.21.15.89:0/1281189470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea340ced80 0x7fea340c7ab0 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fea28001eb0 tx=0x7fea2800c8b0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.445+0000 7fea3c400700 1 -- 172.21.15.89:0/1281189470 shutdown_connections 2024-04-02T09:13:41.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.445+0000 7fea3c400700 1 --2- 172.21.15.89:0/1281189470 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fea34160cc0 0x7fea3414f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.445+0000 7fea3c400700 1 --2- 172.21.15.89:0/1281189470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fea340ced80 0x7fea340c7ab0 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.445+0000 7fea3c400700 1 --2- 172.21.15.89:0/1281189470 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fea1406aed0 0x7fea1406d390 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.445+0000 7fea3c400700 1 --2- 172.21.15.89:0/1281189470 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea3415dad0 0x7fea340b8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.445+0000 7fea3c400700 1 -- 172.21.15.89:0/1281189470 >> 172.21.15.89:0/1281189470 conn(0x7fea3408aff0 msgr2=0x7fea340ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:41.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.446+0000 7fea3c400700 1 -- 172.21.15.89:0/1281189470 shutdown_connections 2024-04-02T09:13:41.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.447+0000 7fea3c400700 1 -- 172.21.15.89:0/1281189470 wait complete. 2024-04-02T09:13:41.448 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'backup_fs_data' created 2024-04-02T09:13:41.472 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-04-02T09:13:41.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.670+0000 7f4021c00700 1 Processor -- start 2024-04-02T09:13:41.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.670+0000 7f4021c00700 1 -- start start 2024-04-02T09:13:41.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.672+0000 7f4021c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c0d2540 0x7f401c1687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:41.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.672+0000 7f4021c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401c161420 con 0x7f401c15d9d0 2024-04-02T09:13:41.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.672+0000 7f4021c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401c161700 con 0x7f401c0d16a0 2024-04-02T09:13:41.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.672+0000 7f4021c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401c0d4340 con 0x7f401c0d2540 2024-04-02T09:13:41.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.672+0000 7f401aa00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f401c15d9d0 0x7f401c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58304/0 (socket says 172.21.15.89:58304) 2024-04-02T09:13:41.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.672+0000 7f401aa00700 1 -- 172.21.15.89:0/3297263898 learned_addr learned my addr 172.21.15.89:0/3297263898 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:41.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.672+0000 7f401b400700 1 --2- 172.21.15.89:0/3297263898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c0d2540 0x7f401c1687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:41.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.672+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610667117 0 0) 0x7f401c161420 con 0x7f401c15d9d0 2024-04-02T09:13:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.672+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40000035b0 con 0x7f401c15d9d0 2024-04-02T09:13:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786389061 0 0) 0x7f401c161700 con 0x7f401c0d16a0 2024-04-02T09:13:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f401c161420 con 0x7f401c0d16a0 2024-04-02T09:13:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2846727163 0 0) 0x7f40000035b0 con 0x7f401c15d9d0 2024-04-02T09:13:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c0d2540 msgr2=0x7f401c1687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 --2- 172.21.15.89:0/3297263898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c0d2540 0x7f401c1687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 >> v1:172.21.15.113:6789/0 conn(0x7f401c0d16a0 legacy=0x7f401c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c1619e0 con 0x7f401c15d9d0 2024-04-02T09:13:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f401b400700 1 --2- 172.21.15.89:0/3297263898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c0d2540 0x7f401c1687b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:13:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f400c001670 con 0x7f401c15d9d0 2024-04-02T09:13:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 >> v1:172.21.15.89:6789/0 conn(0x7f401c15d9d0 legacy=0x7f401c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 --2- 172.21.15.89:0/3297263898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c0d2540 0x7f4000003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 --2- 172.21.15.89:0/3297263898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000004100 0x7f4000004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 --2- 172.21.15.89:0/3297263898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4000004d90 0x7f4000005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400000bac0 con 0x7f4000004d90 2024-04-02T09:13:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400000bc00 con 0x7f4000004100 2024-04-02T09:13:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400000bdc0 con 0x7f401c0d2540 2024-04-02T09:13:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f401b400700 1 --2- 172.21.15.89:0/3297263898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4000004d90 0x7f4000005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f401aa00700 1 --2- 172.21.15.89:0/3297263898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c0d2540 0x7f4000003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f401a000700 1 --2- 172.21.15.89:0/3297263898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000004100 0x7f4000004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f401b400700 1 -- 172.21.15.89:0/3297263898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c0d2540 msgr2=0x7f4000003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.673+0000 7f401b400700 1 --2- 172.21.15.89:0/3297263898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c0d2540 0x7f4000003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.674+0000 7f401b400700 1 -- 172.21.15.89:0/3297263898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000004100 msgr2=0x7f4000004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.674+0000 7f401b400700 1 --2- 172.21.15.89:0/3297263898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000004100 0x7f4000004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.674+0000 7f401b400700 1 -- 172.21.15.89:0/3297263898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4014009a70 con 0x7f4000004d90 2024-04-02T09:13:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.674+0000 7f401aa00700 1 --2- 172.21.15.89:0/3297263898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c0d2540 0x7f4000003bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.674+0000 7f401b400700 1 --2- 172.21.15.89:0/3297263898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4000004d90 0x7f4000005210 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f401c08c320 tx=0x7f4000005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a485dc5a125972 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.674+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401400ed80 con 0x7f4000004d90 2024-04-02T09:13:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.674+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401400f040 con 0x7f4000004d90 2024-04-02T09:13:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.674+0000 7f4019600700 1 -- 172.21.15.89:0/3297263898 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401400eee0 con 0x7f4000004d90 2024-04-02T09:13:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.674+0000 7f4021c00700 1 -- 172.21.15.89:0/3297263898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4000004d90 msgr2=0x7f4000005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.674+0000 7f4021c00700 1 --2- 172.21.15.89:0/3297263898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4000004d90 0x7f4000005210 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f401c08c320 tx=0x7f4000005880 comp rx=0 tx=0).stop 2024-04-02T09:13:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.674+0000 7f401b400700 1 -- 172.21.15.89:0/3297263898 reap_dead start 2024-04-02T09:13:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.675+0000 7f4021c00700 1 -- 172.21.15.89:0/3297263898 shutdown_connections 2024-04-02T09:13:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.675+0000 7f4021c00700 1 -- 172.21.15.89:0/3297263898 >> 172.21.15.89:0/3297263898 conn(0x7f401c08aff0 msgr2=0x7f401c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.675+0000 7f4021c00700 1 -- 172.21.15.89:0/3297263898 shutdown_connections 2024-04-02T09:13:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.675+0000 7f4021c00700 1 -- 172.21.15.89:0/3297263898 wait complete. 2024-04-02T09:13:41.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.676+0000 7f4021c00700 1 Processor -- start 2024-04-02T09:13:41.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.677+0000 7f4021c00700 1 -- start start 2024-04-02T09:13:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.677+0000 7f4021c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4000004d90 0x7f401c14eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.677+0000 7f4021c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f401c1f4630 0x7f401c15eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.678+0000 7f4021c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15f0c0 0x7f401c0b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.678+0000 7f4021c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c0cc380 con 0x7f4000004d90 2024-04-02T09:13:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.678+0000 7f4021c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c15f640 con 0x7f401c1f4630 2024-04-02T09:13:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.678+0000 7f4021c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401c0cba50 con 0x7f401c15f0c0 2024-04-02T09:13:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.678+0000 7f401aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4000004d90 0x7f401c14eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.678+0000 7f401aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4000004d90 0x7f401c14eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55186/0 (socket says 172.21.15.89:55186) 2024-04-02T09:13:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.678+0000 7f401aa00700 1 -- 172.21.15.89:0/3744191880 learned_addr learned my addr 172.21.15.89:0/3744191880 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.678+0000 7f401aa00700 1 -- 172.21.15.89:0/3744191880 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15f0c0 msgr2=0x7f401c0b40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.678+0000 7f401aa00700 1 --2- 172.21.15.89:0/3744191880 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15f0c0 0x7f401c0b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.678+0000 7f401aa00700 1 -- 172.21.15.89:0/3744191880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f401c1f4630 msgr2=0x7f401c15eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.678+0000 7f401aa00700 1 --2- 172.21.15.89:0/3744191880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f401c1f4630 0x7f401c15eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.678+0000 7f401aa00700 1 -- 172.21.15.89:0/3744191880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c0cbd30 con 0x7f4000004d90 2024-04-02T09:13:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.678+0000 7f401aa00700 1 --2- 172.21.15.89:0/3744191880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4000004d90 0x7f401c14eb00 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f400c00eaf0 tx=0x7f400c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.679+0000 7f4012a00700 1 -- 172.21.15.89:0/3744191880 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c00c730 con 0x7f4000004d90 2024-04-02T09:13:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.679+0000 7f4012a00700 1 -- 172.21.15.89:0/3744191880 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400c00c890 con 0x7f4000004d90 2024-04-02T09:13:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.679+0000 7f4012a00700 1 -- 172.21.15.89:0/3744191880 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c00cc00 con 0x7f4000004d90 2024-04-02T09:13:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.679+0000 7f4021c00700 1 -- 172.21.15.89:0/3744191880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401c0cc010 con 0x7f4000004d90 2024-04-02T09:13:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.679+0000 7f4021c00700 1 -- 172.21.15.89:0/3744191880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401c0cb4c0 con 0x7f4000004d90 2024-04-02T09:13:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.680+0000 7f4021c00700 1 -- 172.21.15.89:0/3744191880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401c0d1ca0 con 0x7f4000004d90 2024-04-02T09:13:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.681+0000 7f4012a00700 1 -- 172.21.15.89:0/3744191880 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f400c01d460 con 0x7f4000004d90 2024-04-02T09:13:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.682+0000 7f4012a00700 1 --2- 172.21.15.89:0/3744191880 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ffc06aed0 0x7f3ffc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.682+0000 7f4012a00700 1 -- 172.21.15.89:0/3744191880 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 6915+0+0 (secure 0 0 0) 0x7f400c089990 con 0x7f4000004d90 2024-04-02T09:13:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.682+0000 7f401a000700 1 --2- 172.21.15.89:0/3744191880 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ffc06aed0 0x7f3ffc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.682+0000 7f401a000700 1 --2- 172.21.15.89:0/3744191880 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ffc06aed0 0x7f3ffc06d390 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4008007fd0 tx=0x7f4008009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.687+0000 7f4012a00700 1 -- 172.21.15.89:0/3744191880 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f400c054380 con 0x7f4000004d90 2024-04-02T09:13:41.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.931+0000 7f4021c00700 1 -- 172.21.15.89:0/3744191880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f401c0cc1a0 con 0x7f4000004d90 2024-04-02T09:13:41.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.938+0000 7f4012a00700 1 -- 172.21.15.89:0/3744191880 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 32 and data pool 33 v121) v1 ==== 183+0+0 (secure 0 0 0) 0x7f400c0579a0 con 0x7f4000004d90 2024-04-02T09:13:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.948+0000 7f4021c00700 1 -- 172.21.15.89:0/3744191880 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ffc06aed0 msgr2=0x7f3ffc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.948+0000 7f4021c00700 1 --2- 172.21.15.89:0/3744191880 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ffc06aed0 0x7f3ffc06d390 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4008007fd0 tx=0x7f4008009040 comp rx=0 tx=0).stop 2024-04-02T09:13:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.948+0000 7f4021c00700 1 -- 172.21.15.89:0/3744191880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4000004d90 msgr2=0x7f401c14eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:41.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.948+0000 7f4021c00700 1 --2- 172.21.15.89:0/3744191880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4000004d90 0x7f401c14eb00 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f400c00eaf0 tx=0x7f400c00eeb0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.949+0000 7f4021c00700 1 -- 172.21.15.89:0/3744191880 shutdown_connections 2024-04-02T09:13:41.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.949+0000 7f4021c00700 1 --2- 172.21.15.89:0/3744191880 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15f0c0 0x7f401c0b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.949+0000 7f4021c00700 1 --2- 172.21.15.89:0/3744191880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4000004d90 0x7f401c14eb00 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.949+0000 7f4021c00700 1 --2- 172.21.15.89:0/3744191880 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ffc06aed0 0x7f3ffc06d390 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.949+0000 7f4021c00700 1 --2- 172.21.15.89:0/3744191880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f401c1f4630 0x7f401c15eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:41.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.949+0000 7f4021c00700 1 -- 172.21.15.89:0/3744191880 >> 172.21.15.89:0/3744191880 conn(0x7f401c08aff0 msgr2=0x7f401c0ce360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:41.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.951+0000 7f4021c00700 1 -- 172.21.15.89:0/3744191880 shutdown_connections 2024-04-02T09:13:41.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:41.951+0000 7f4021c00700 1 -- 172.21.15.89:0/3744191880 wait complete. 2024-04-02T09:13:41.956 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 32 and data pool 33 2024-04-02T09:13:41.978 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:13:42.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.170+0000 7f0e8b200700 1 Processor -- start 2024-04-02T09:13:42.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.171+0000 7f0e8b200700 1 -- start start 2024-04-02T09:13:42.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.172+0000 7f0e8b200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0e84160f10 0x7f0e840d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:42.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.172+0000 7f0e8b200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e840cf230 0x7f0e840d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:42.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.172+0000 7f0e8b200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e840ceda0 con 0x7f0e84161590 2024-04-02T09:13:42.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.172+0000 7f0e8b200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e840d23d0 con 0x7f0e84160f10 2024-04-02T09:13:42.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.172+0000 7f0e8b200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e841612f0 con 0x7f0e840cf230 2024-04-02T09:13:42.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.172+0000 7f0e83e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0e84160f10 0x7f0e840d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:42.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.172+0000 7f0e83e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0e84160f10 0x7f0e840d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55202/0 (socket says 172.21.15.89:55202) 2024-04-02T09:13:42.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.172+0000 7f0e83e00700 1 -- 172.21.15.89:0/956438693 learned_addr learned my addr 172.21.15.89:0/956438693 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:42.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.172+0000 7f0e83e00700 1 -- 172.21.15.89:0/956438693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e840cf230 msgr2=0x7f0e840d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:42.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.172+0000 7f0e83400700 1 --2- 172.21.15.89:0/956438693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e840cf230 0x7f0e840d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:42.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.172+0000 7f0e83e00700 1 --2- 172.21.15.89:0/956438693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e840cf230 0x7f0e840d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.172+0000 7f0e83e00700 1 -- 172.21.15.89:0/956438693 >> v1:172.21.15.113:6789/0 conn(0x7f0e84161590 legacy=0x7f0e840b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:42.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.172+0000 7f0e83e00700 1 -- 172.21.15.89:0/956438693 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e840ce470 con 0x7f0e84160f10 2024-04-02T09:13:42.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.172+0000 7f0e83400700 1 --2- 172.21.15.89:0/956438693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e840cf230 0x7f0e840d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:42.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.173+0000 7f0e83e00700 1 --2- 172.21.15.89:0/956438693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e84160f10 0x7f0e840d18b0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f0e740098f0 tx=0x7f0e74009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b8c3bdf19492ff server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:42.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.173+0000 7f0e82a00700 1 -- 172.21.15.89:0/956438693 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e74004230 con 0x7f0e84160f10 2024-04-02T09:13:42.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.173+0000 7f0e82a00700 1 -- 172.21.15.89:0/956438693 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e74004390 con 0x7f0e84160f10 2024-04-02T09:13:42.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.173+0000 7f0e82a00700 1 -- 172.21.15.89:0/956438693 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e740036a0 con 0x7f0e84160f10 2024-04-02T09:13:42.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.173+0000 7f0e8b200700 1 -- 172.21.15.89:0/956438693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e84160f10 msgr2=0x7f0e840d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:42.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.173+0000 7f0e8b200700 1 --2- 172.21.15.89:0/956438693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e84160f10 0x7f0e840d18b0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f0e740098f0 tx=0x7f0e74009c00 comp rx=0 tx=0).stop 2024-04-02T09:13:42.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.174+0000 7f0e8b200700 1 -- 172.21.15.89:0/956438693 shutdown_connections 2024-04-02T09:13:42.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.174+0000 7f0e8b200700 1 --2- 172.21.15.89:0/956438693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e84160f10 0x7f0e840d18b0 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.174+0000 7f0e8b200700 1 --2- 172.21.15.89:0/956438693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e840cf230 0x7f0e840d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.174+0000 7f0e8b200700 1 -- 172.21.15.89:0/956438693 >> 172.21.15.89:0/956438693 conn(0x7f0e8408aff0 msgr2=0x7f0e840d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:42.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.174+0000 7f0e8b200700 1 -- 172.21.15.89:0/956438693 shutdown_connections 2024-04-02T09:13:42.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.174+0000 7f0e8b200700 1 -- 172.21.15.89:0/956438693 wait complete. 2024-04-02T09:13:42.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.175+0000 7f0e8b200700 1 Processor -- start 2024-04-02T09:13:42.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.175+0000 7f0e8b200700 1 -- start start 2024-04-02T09:13:42.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.176+0000 7f0e8b200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e840cf230 0x7f0e841585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:42.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.176+0000 7f0e8b200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e84160f10 0x7f0e841f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:42.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.176+0000 7f0e8b200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e84161590 0x7f0e841f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:42.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.176+0000 7f0e8b200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e841f31e0 con 0x7f0e840cf230 2024-04-02T09:13:42.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.176+0000 7f0e8b200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e84157ba0 con 0x7f0e84160f10 2024-04-02T09:13:42.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.176+0000 7f0e8b200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e840b8c20 con 0x7f0e84161590 2024-04-02T09:13:42.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e83e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e840cf230 0x7f0e841585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:42.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e83e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e840cf230 0x7f0e841585e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55204/0 (socket says 172.21.15.89:55204) 2024-04-02T09:13:42.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e83e00700 1 -- 172.21.15.89:0/2160443420 learned_addr learned my addr 172.21.15.89:0/2160443420 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:42.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e83400700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e84160f10 0x7f0e841f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:42.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e89200700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e84161590 0x7f0e841f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:42.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e83e00700 1 -- 172.21.15.89:0/2160443420 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e84161590 msgr2=0x7f0e841f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:42.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e83e00700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e84161590 0x7f0e841f6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e83e00700 1 -- 172.21.15.89:0/2160443420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e84160f10 msgr2=0x7f0e841f6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:42.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e83e00700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e84160f10 0x7f0e841f6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e83e00700 1 -- 172.21.15.89:0/2160443420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e74009550 con 0x7f0e840cf230 2024-04-02T09:13:42.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e83400700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e84160f10 0x7f0e841f6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:42.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e83e00700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e840cf230 0x7f0e841585e0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f0e74009130 tx=0x7f0e74004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:42.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e7be00700 1 -- 172.21.15.89:0/2160443420 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7400b860 con 0x7f0e840cf230 2024-04-02T09:13:42.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e7be00700 1 -- 172.21.15.89:0/2160443420 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7400b9c0 con 0x7f0e840cf230 2024-04-02T09:13:42.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.177+0000 7f0e7be00700 1 -- 172.21.15.89:0/2160443420 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7400bd70 con 0x7f0e840cf230 2024-04-02T09:13:42.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.178+0000 7f0e8b200700 1 -- 172.21.15.89:0/2160443420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e840b8f00 con 0x7f0e840cf230 2024-04-02T09:13:42.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.178+0000 7f0e8b200700 1 -- 172.21.15.89:0/2160443420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e841580c0 con 0x7f0e840cf230 2024-04-02T09:13:42.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.179+0000 7f0e8b200700 1 -- 172.21.15.89:0/2160443420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e84067720 con 0x7f0e840cf230 2024-04-02T09:13:42.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.180+0000 7f0e7be00700 1 -- 172.21.15.89:0/2160443420 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0e7401b440 con 0x7f0e840cf230 2024-04-02T09:13:42.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.181+0000 7f0e7be00700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0e6406aed0 0x7f0e6406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:42.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.181+0000 7f0e7be00700 1 -- 172.21.15.89:0/2160443420 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f0e7408b690 con 0x7f0e840cf230 2024-04-02T09:13:42.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.181+0000 7f0e83400700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0e6406aed0 0x7f0e6406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:42.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.181+0000 7f0e83400700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0e6406aed0 0x7f0e6406d390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0e70006fd0 tx=0x7f0e70008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:42.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.185+0000 7f0e7be00700 1 -- 172.21.15.89:0/2160443420 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0e740560b0 con 0x7f0e840cf230 2024-04-02T09:13:42.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.387+0000 7f0e8b200700 1 -- 172.21.15.89:0/2160443420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e8406bc50 con 0x7f0e840cf230 2024-04-02T09:13:42.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.389+0000 7f0e7be00700 1 -- 172.21.15.89:0/2160443420 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v247) v1 ==== 74+0+20417 (secure 0 0 0) 0x7f0e740596d0 con 0x7f0e840cf230 2024-04-02T09:13:42.387 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:42.388 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":247,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:13:41.933015+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:13:26.654647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-02T09:13:27.342285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:13:40.048039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:13:40.907706+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:13:42.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.391+0000 7f0e8b200700 1 -- 172.21.15.89:0/2160443420 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0e6406aed0 msgr2=0x7f0e6406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:42.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.391+0000 7f0e8b200700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0e6406aed0 0x7f0e6406d390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0e70006fd0 tx=0x7f0e70008040 comp rx=0 tx=0).stop 2024-04-02T09:13:42.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.391+0000 7f0e8b200700 1 -- 172.21.15.89:0/2160443420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e840cf230 msgr2=0x7f0e841585e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:42.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.391+0000 7f0e8b200700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e840cf230 0x7f0e841585e0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f0e74009130 tx=0x7f0e74004750 comp rx=0 tx=0).stop 2024-04-02T09:13:42.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.391+0000 7f0e8b200700 1 -- 172.21.15.89:0/2160443420 shutdown_connections 2024-04-02T09:13:42.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.391+0000 7f0e8b200700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0e84161590 0x7f0e841f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.391+0000 7f0e8b200700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e840cf230 0x7f0e841585e0 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.391+0000 7f0e8b200700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0e6406aed0 0x7f0e6406d390 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.391+0000 7f0e8b200700 1 --2- 172.21.15.89:0/2160443420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e84160f10 0x7f0e841f6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.391+0000 7f0e8b200700 1 -- 172.21.15.89:0/2160443420 >> 172.21.15.89:0/2160443420 conn(0x7f0e8408aff0 msgr2=0x7f0e84160070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:42.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.391+0000 7f0e8b200700 1 -- 172.21.15.89:0/2160443420 shutdown_connections 2024-04-02T09:13:42.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.392+0000 7f0e8b200700 1 -- 172.21.15.89:0/2160443420 wait complete. 2024-04-02T09:13:42.408 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:13:42.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.600+0000 7efd59200700 1 Processor -- start 2024-04-02T09:13:42.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.601+0000 7efd59200700 1 -- start start 2024-04-02T09:13:42.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.602+0000 7efd59200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efd540d05e0 0x7efd540d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:42.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.602+0000 7efd59200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd540b6990 0x7efd540b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:42.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.602+0000 7efd59200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd540d0f80 con 0x7efd5415da50 2024-04-02T09:13:42.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.602+0000 7efd59200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd540b95d0 con 0x7efd540d05e0 2024-04-02T09:13:42.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.602+0000 7efd59200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5415de30 con 0x7efd540b6990 2024-04-02T09:13:42.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.602+0000 7efd52800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd540b6990 0x7efd540b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:42.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.602+0000 7efd51400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efd540d05e0 0x7efd540d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:42.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.602+0000 7efd52800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd540b6990 0x7efd540b6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:33442/0 (socket says 172.21.15.89:33442) 2024-04-02T09:13:42.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.602+0000 7efd52800700 1 -- 172.21.15.89:0/3427977324 learned_addr learned my addr 172.21.15.89:0/3427977324 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:42.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.603+0000 7efd52800700 1 -- 172.21.15.89:0/3427977324 >> v2:172.21.15.113:3300/0 conn(0x7efd540d05e0 msgr2=0x7efd540d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:42.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.603+0000 7efd52800700 1 --2- 172.21.15.89:0/3427977324 >> v2:172.21.15.113:3300/0 conn(0x7efd540d05e0 0x7efd540d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.603+0000 7efd52800700 1 -- 172.21.15.89:0/3427977324 >> v1:172.21.15.113:6789/0 conn(0x7efd5415da50 legacy=0x7efd540ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:42.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.603+0000 7efd52800700 1 -- 172.21.15.89:0/3427977324 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd540d24e0 con 0x7efd540b6990 2024-04-02T09:13:42.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.603+0000 7efd52800700 1 --2- 172.21.15.89:0/3427977324 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd540b6990 0x7efd540b6e40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efd4c009e50 tx=0x7efd4c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4c1a378f6a3864a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:42.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.603+0000 7efd50a00700 1 -- 172.21.15.89:0/3427977324 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c00e070 con 0x7efd540b6990 2024-04-02T09:13:42.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.603+0000 7efd50a00700 1 -- 172.21.15.89:0/3427977324 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4c00b910 con 0x7efd540b6990 2024-04-02T09:13:42.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.603+0000 7efd50a00700 1 -- 172.21.15.89:0/3427977324 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c00bcc0 con 0x7efd540b6990 2024-04-02T09:13:42.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.604+0000 7efd59200700 1 -- 172.21.15.89:0/3427977324 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd540b6990 msgr2=0x7efd540b6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:42.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.604+0000 7efd59200700 1 --2- 172.21.15.89:0/3427977324 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd540b6990 0x7efd540b6e40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efd4c009e50 tx=0x7efd4c00b330 comp rx=0 tx=0).stop 2024-04-02T09:13:42.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.604+0000 7efd59200700 1 -- 172.21.15.89:0/3427977324 shutdown_connections 2024-04-02T09:13:42.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.604+0000 7efd59200700 1 --2- 172.21.15.89:0/3427977324 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd540b6990 0x7efd540b6e40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.604+0000 7efd59200700 1 --2- 172.21.15.89:0/3427977324 >> v2:172.21.15.113:3300/0 conn(0x7efd540d05e0 0x7efd540d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.604+0000 7efd59200700 1 -- 172.21.15.89:0/3427977324 >> 172.21.15.89:0/3427977324 conn(0x7efd540666a0 msgr2=0x7efd54066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:42.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.604+0000 7efd59200700 1 -- 172.21.15.89:0/3427977324 shutdown_connections 2024-04-02T09:13:42.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.604+0000 7efd59200700 1 -- 172.21.15.89:0/3427977324 wait complete. 2024-04-02T09:13:42.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.606+0000 7efd59200700 1 Processor -- start 2024-04-02T09:13:42.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.606+0000 7efd59200700 1 -- start start 2024-04-02T09:13:42.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd59200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd540b6990 0x7efd5415c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:42.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd59200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd540d05e0 0x7efd540cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:42.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd59200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd5415da50 0x7efd540cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:42.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd59200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd540ccf30 con 0x7efd540b6990 2024-04-02T09:13:42.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd59200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd54157290 con 0x7efd5415da50 2024-04-02T09:13:42.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd59200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd54157570 con 0x7efd540d05e0 2024-04-02T09:13:42.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd51e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd540b6990 0x7efd5415c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:42.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd51e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd540b6990 0x7efd5415c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55220/0 (socket says 172.21.15.89:55220) 2024-04-02T09:13:42.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd51e00700 1 -- 172.21.15.89:0/3961368334 learned_addr learned my addr 172.21.15.89:0/3961368334 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:42.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd51e00700 1 -- 172.21.15.89:0/3961368334 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd540d05e0 msgr2=0x7efd540cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:42.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd51e00700 1 --2- 172.21.15.89:0/3961368334 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd540d05e0 0x7efd540cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd52800700 1 --2- 172.21.15.89:0/3961368334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd5415da50 0x7efd540cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:42.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd51e00700 1 -- 172.21.15.89:0/3961368334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd5415da50 msgr2=0x7efd540cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:42.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd51e00700 1 --2- 172.21.15.89:0/3961368334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd5415da50 0x7efd540cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.607+0000 7efd51e00700 1 -- 172.21.15.89:0/3961368334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4c009b30 con 0x7efd540b6990 2024-04-02T09:13:42.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.608+0000 7efd51e00700 1 --2- 172.21.15.89:0/3961368334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd540b6990 0x7efd5415c860 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7efd4400c450 tx=0x7efd4400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:42.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.608+0000 7efd4a000700 1 -- 172.21.15.89:0/3961368334 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd440052d0 con 0x7efd540b6990 2024-04-02T09:13:42.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.608+0000 7efd4a000700 1 -- 172.21.15.89:0/3961368334 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd44005430 con 0x7efd540b6990 2024-04-02T09:13:42.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.608+0000 7efd4a000700 1 -- 172.21.15.89:0/3961368334 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd440038c0 con 0x7efd540b6990 2024-04-02T09:13:42.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.608+0000 7efd59200700 1 -- 172.21.15.89:0/3961368334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd54157850 con 0x7efd540b6990 2024-04-02T09:13:42.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.608+0000 7efd59200700 1 -- 172.21.15.89:0/3961368334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd540b3ca0 con 0x7efd540b6990 2024-04-02T09:13:42.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.609+0000 7efd59200700 1 -- 172.21.15.89:0/3961368334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd540b3f10 con 0x7efd540b6990 2024-04-02T09:13:42.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.610+0000 7efd4a000700 1 -- 172.21.15.89:0/3961368334 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7efd44003a20 con 0x7efd540b6990 2024-04-02T09:13:42.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.611+0000 7efd4a000700 1 --2- 172.21.15.89:0/3961368334 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efd3406af00 0x7efd3406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:42.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.611+0000 7efd4a000700 1 -- 172.21.15.89:0/3961368334 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 7021+0+0 (secure 0 0 0) 0x7efd440890f0 con 0x7efd540b6990 2024-04-02T09:13:42.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.611+0000 7efd51400700 1 --2- 172.21.15.89:0/3961368334 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efd3406af00 0x7efd3406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:42.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.612+0000 7efd51400700 1 --2- 172.21.15.89:0/3961368334 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efd3406af00 0x7efd3406d3c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efd5415d200 tx=0x7efd40009640 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:42.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.614+0000 7efd4a000700 1 -- 172.21.15.89:0/3961368334 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efd44053b00 con 0x7efd540b6990 2024-04-02T09:13:42.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.817+0000 7efd59200700 1 -- 172.21.15.89:0/3961368334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd54067720 con 0x7efd540b6990 2024-04-02T09:13:42.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.819+0000 7efd4a000700 1 -- 172.21.15.89:0/3961368334 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v247) v1 ==== 74+0+20417 (secure 0 0 0) 0x7efd44057120 con 0x7efd540b6990 2024-04-02T09:13:42.817 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:42.818 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":247,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:13:41.933015+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:13:26.654647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-02T09:13:27.342285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:13:40.048039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:13:40.907706+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:13:42.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.821+0000 7efd59200700 1 -- 172.21.15.89:0/3961368334 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efd3406af00 msgr2=0x7efd3406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:42.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.821+0000 7efd59200700 1 --2- 172.21.15.89:0/3961368334 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efd3406af00 0x7efd3406d3c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efd5415d200 tx=0x7efd40009640 comp rx=0 tx=0).stop 2024-04-02T09:13:42.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.821+0000 7efd59200700 1 -- 172.21.15.89:0/3961368334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd540b6990 msgr2=0x7efd5415c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:42.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.821+0000 7efd59200700 1 --2- 172.21.15.89:0/3961368334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd540b6990 0x7efd5415c860 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7efd4400c450 tx=0x7efd4400c810 comp rx=0 tx=0).stop 2024-04-02T09:13:42.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.821+0000 7efd59200700 1 -- 172.21.15.89:0/3961368334 shutdown_connections 2024-04-02T09:13:42.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.821+0000 7efd59200700 1 --2- 172.21.15.89:0/3961368334 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efd3406af00 0x7efd3406d3c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.821+0000 7efd59200700 1 --2- 172.21.15.89:0/3961368334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd5415da50 0x7efd540cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.821+0000 7efd59200700 1 --2- 172.21.15.89:0/3961368334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd540b6990 0x7efd5415c860 unknown :-1 s=CLOSED pgs=2595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.821+0000 7efd59200700 1 --2- 172.21.15.89:0/3961368334 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd540d05e0 0x7efd540cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:42.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.821+0000 7efd59200700 1 -- 172.21.15.89:0/3961368334 >> 172.21.15.89:0/3961368334 conn(0x7efd540666a0 msgr2=0x7efd540d3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:42.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.821+0000 7efd59200700 1 -- 172.21.15.89:0/3961368334 shutdown_connections 2024-04-02T09:13:42.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:42.822+0000 7efd59200700 1 -- 172.21.15.89:0/3961368334 wait complete. 2024-04-02T09:13:42.839 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-04-02T09:13:42.892 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:13:42.896 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:13:43.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.042+0000 7f36cfc00700 1 Processor -- start 2024-04-02T09:13:43.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.043+0000 7f36cfc00700 1 -- start start 2024-04-02T09:13:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.044+0000 7f36cfc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f36c815d9d0 0x7f36c80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.044+0000 7f36cfc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36c80d16a0 0x7f36c80d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.044+0000 7f36cfc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f36c80d2370 0x7f36c815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.044+0000 7f36cfc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c80d2820 con 0x7f36c80d2370 2024-04-02T09:13:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.044+0000 7f36cfc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c8161420 con 0x7f36c815d9d0 2024-04-02T09:13:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.044+0000 7f36cfc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c8161700 con 0x7f36c80d16a0 2024-04-02T09:13:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.044+0000 7f36cdc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f36c80d2370 0x7f36c815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.044+0000 7f36cdc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f36c80d2370 0x7f36c815ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55234/0 (socket says 172.21.15.89:55234) 2024-04-02T09:13:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.044+0000 7f36cdc00700 1 -- 172.21.15.89:0/2874363757 learned_addr learned my addr 172.21.15.89:0/2874363757 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.044+0000 7f36cd200700 1 --2- 172.21.15.89:0/2874363757 >> v2:172.21.15.113:3300/0 conn(0x7f36c815d9d0 0x7f36c80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.044+0000 7f36c7e00700 1 --2- 172.21.15.89:0/2874363757 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36c80d16a0 0x7f36c80d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:43.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.045+0000 7f36cdc00700 1 -- 172.21.15.89:0/2874363757 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36c80d16a0 msgr2=0x7f36c80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:43.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.045+0000 7f36cdc00700 1 --2- 172.21.15.89:0/2874363757 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36c80d16a0 0x7f36c80d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:43.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.045+0000 7f36cdc00700 1 -- 172.21.15.89:0/2874363757 >> v2:172.21.15.113:3300/0 conn(0x7f36c815d9d0 msgr2=0x7f36c80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:43.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.045+0000 7f36cdc00700 1 --2- 172.21.15.89:0/2874363757 >> v2:172.21.15.113:3300/0 conn(0x7f36c815d9d0 0x7f36c80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:43.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.045+0000 7f36cdc00700 1 -- 172.21.15.89:0/2874363757 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c81619e0 con 0x7f36c80d2370 2024-04-02T09:13:43.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.045+0000 7f36c7e00700 1 --2- 172.21.15.89:0/2874363757 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36c80d16a0 0x7f36c80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:43.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.045+0000 7f36cdc00700 1 --2- 172.21.15.89:0/2874363757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c80d2370 0x7f36c815ddb0 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7f36c80d0ff0 tx=0x7f36c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35eaaa1b2ae1f90e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:43.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.045+0000 7f36c7400700 1 -- 172.21.15.89:0/2874363757 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c000e070 con 0x7f36c80d2370 2024-04-02T09:13:43.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.045+0000 7f36c7400700 1 -- 172.21.15.89:0/2874363757 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c000b910 con 0x7f36c80d2370 2024-04-02T09:13:43.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.045+0000 7f36c7400700 1 -- 172.21.15.89:0/2874363757 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c000bcc0 con 0x7f36c80d2370 2024-04-02T09:13:43.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.046+0000 7f36cfc00700 1 -- 172.21.15.89:0/2874363757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c80d2370 msgr2=0x7f36c815ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:43.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.046+0000 7f36cfc00700 1 --2- 172.21.15.89:0/2874363757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c80d2370 0x7f36c815ddb0 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7f36c80d0ff0 tx=0x7f36c000b330 comp rx=0 tx=0).stop 2024-04-02T09:13:43.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.046+0000 7f36cfc00700 1 -- 172.21.15.89:0/2874363757 shutdown_connections 2024-04-02T09:13:43.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.046+0000 7f36cfc00700 1 --2- 172.21.15.89:0/2874363757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c80d2370 0x7f36c815ddb0 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:43.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.046+0000 7f36cfc00700 1 --2- 172.21.15.89:0/2874363757 >> v2:172.21.15.113:3300/0 conn(0x7f36c815d9d0 0x7f36c80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:43.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.046+0000 7f36cfc00700 1 --2- 172.21.15.89:0/2874363757 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36c80d16a0 0x7f36c80d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:43.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.046+0000 7f36cfc00700 1 -- 172.21.15.89:0/2874363757 >> 172.21.15.89:0/2874363757 conn(0x7f36c808aff0 msgr2=0x7f36c80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:43.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.046+0000 7f36cfc00700 1 -- 172.21.15.89:0/2874363757 shutdown_connections 2024-04-02T09:13:43.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.046+0000 7f36cfc00700 1 -- 172.21.15.89:0/2874363757 wait complete. 2024-04-02T09:13:43.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.047+0000 7f36cfc00700 1 Processor -- start 2024-04-02T09:13:43.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.048+0000 7f36cfc00700 1 -- start start 2024-04-02T09:13:43.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.048+0000 7f36cfc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36c80d16a0 0x7f36c815c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:43.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.048+0000 7f36cfc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c80d2370 0x7f36c80b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36cfc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c815d9d0 0x7f36c80b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36cfc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c80caa40 con 0x7f36c80d2370 2024-04-02T09:13:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36cfc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c80b7640 con 0x7f36c815d9d0 2024-04-02T09:13:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36cfc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c80b6ac0 con 0x7f36c80d16a0 2024-04-02T09:13:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36c7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c80d2370 0x7f36c80b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36c7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c80d2370 0x7f36c80b6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55248/0 (socket says 172.21.15.89:55248) 2024-04-02T09:13:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36c7e00700 1 -- 172.21.15.89:0/1484533769 learned_addr learned my addr 172.21.15.89:0/1484533769 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36cd200700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36c80d16a0 0x7f36c815c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36cdc00700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c815d9d0 0x7f36c80b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36c7e00700 1 -- 172.21.15.89:0/1484533769 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36c80d16a0 msgr2=0x7f36c815c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36c7e00700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36c80d16a0 0x7f36c815c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36c7e00700 1 -- 172.21.15.89:0/1484533769 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c815d9d0 msgr2=0x7f36c80b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:43.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36c7e00700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c815d9d0 0x7f36c80b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:43.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36c7e00700 1 -- 172.21.15.89:0/1484533769 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c0009b30 con 0x7f36c80d2370 2024-04-02T09:13:43.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36cdc00700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c815d9d0 0x7f36c80b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:43.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.049+0000 7f36c7e00700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c80d2370 0x7f36c80b6020 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f36b400e7e0 tx=0x7f36b400eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:43.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.050+0000 7f36c4c00700 1 -- 172.21.15.89:0/1484533769 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4005c20 con 0x7f36c80d2370 2024-04-02T09:13:43.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.050+0000 7f36c4c00700 1 -- 172.21.15.89:0/1484533769 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b4005d80 con 0x7f36c80d2370 2024-04-02T09:13:43.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.050+0000 7f36c4c00700 1 -- 172.21.15.89:0/1484533769 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4004250 con 0x7f36c80d2370 2024-04-02T09:13:43.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.050+0000 7f36cfc00700 1 -- 172.21.15.89:0/1484533769 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c80b6da0 con 0x7f36c80d2370 2024-04-02T09:13:43.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.050+0000 7f36cfc00700 1 -- 172.21.15.89:0/1484533769 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c80b7360 con 0x7f36c80d2370 2024-04-02T09:13:43.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.051+0000 7f36cfc00700 1 -- 172.21.15.89:0/1484533769 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c80cad00 con 0x7f36c80d2370 2024-04-02T09:13:43.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.052+0000 7f36c4c00700 1 -- 172.21.15.89:0/1484533769 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f36b40043b0 con 0x7f36c80d2370 2024-04-02T09:13:43.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.052+0000 7f36c4c00700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f36a806aed0 0x7f36a806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:43.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.053+0000 7f36c4c00700 1 -- 172.21.15.89:0/1484533769 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f36b4089be0 con 0x7f36c80d2370 2024-04-02T09:13:43.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.053+0000 7f36cd200700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f36a806aed0 0x7f36a806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:43.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.053+0000 7f36cd200700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f36a806aed0 0x7f36a806d390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f36c81e9130 tx=0x7f36bc008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:43.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.057+0000 7f36c4c00700 1 -- 172.21.15.89:0/1484533769 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f36b4054390 con 0x7f36c80d2370 2024-04-02T09:13:43.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.303+0000 7f36cfc00700 1 -- 172.21.15.89:0/1484533769 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f36c80d1ca0 con 0x7f36c80d2370 2024-04-02T09:13:43.844 INFO:tasks.ceph.mds.a.smithi089.stderr:2024-04-02T09:13:43.845+0000 7f8b3c000700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T09:13:43.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.953+0000 7f36c4c00700 1 -- 172.21.15.89:0/1484533769 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v124) v1 ==== 121+0+0 (secure 0 0 0) 0x7f36b40579b0 con 0x7f36c80d2370 2024-04-02T09:13:43.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.955+0000 7f36cfc00700 1 -- 172.21.15.89:0/1484533769 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f36a806aed0 msgr2=0x7f36a806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:43.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.955+0000 7f36cfc00700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f36a806aed0 0x7f36a806d390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f36c81e9130 tx=0x7f36bc008040 comp rx=0 tx=0).stop 2024-04-02T09:13:43.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.955+0000 7f36cfc00700 1 -- 172.21.15.89:0/1484533769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c80d2370 msgr2=0x7f36c80b6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:43.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.955+0000 7f36cfc00700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c80d2370 0x7f36c80b6020 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f36b400e7e0 tx=0x7f36b400eba0 comp rx=0 tx=0).stop 2024-04-02T09:13:43.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.955+0000 7f36cfc00700 1 -- 172.21.15.89:0/1484533769 shutdown_connections 2024-04-02T09:13:43.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.955+0000 7f36cfc00700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f36a806aed0 0x7f36a806d390 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:43.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.955+0000 7f36cfc00700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c815d9d0 0x7f36c80b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:43.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.955+0000 7f36cfc00700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36c80d16a0 0x7f36c815c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:43.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.955+0000 7f36cfc00700 1 --2- 172.21.15.89:0/1484533769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36c80d2370 0x7f36c80b6020 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:43.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.956+0000 7f36cfc00700 1 -- 172.21.15.89:0/1484533769 >> 172.21.15.89:0/1484533769 conn(0x7f36c808aff0 msgr2=0x7f36c80c9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:43.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.956+0000 7f36cfc00700 1 -- 172.21.15.89:0/1484533769 shutdown_connections 2024-04-02T09:13:43.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:43.956+0000 7f36cfc00700 1 -- 172.21.15.89:0/1484533769 wait complete. 2024-04-02T09:13:43.973 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T09:13:43.974 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-04-02T09:13:44.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.171+0000 7f6848000700 1 Processor -- start 2024-04-02T09:13:44.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.172+0000 7f6848000700 1 -- start start 2024-04-02T09:13:44.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6848000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f684015d9d0 0x7f68400d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:44.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6848000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f68400d16a0 0x7f68400d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:44.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6848000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68400d2370 0x7f684015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:44.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6848000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68400d2820 con 0x7f684015d9d0 2024-04-02T09:13:44.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6848000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6840161420 con 0x7f68400d16a0 2024-04-02T09:13:44.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6848000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6840161700 con 0x7f68400d2370 2024-04-02T09:13:44.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6845600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f684015d9d0 0x7f68400d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:44.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6845600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f684015d9d0 0x7f68400d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55256/0 (socket says 172.21.15.89:55256) 2024-04-02T09:13:44.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6845600700 1 -- 172.21.15.89:0/601266754 learned_addr learned my addr 172.21.15.89:0/601266754 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:44.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6844c00700 1 --2- 172.21.15.89:0/601266754 >> v2:172.21.15.113:3300/0 conn(0x7f68400d16a0 0x7f68400d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:44.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6846000700 1 --2- 172.21.15.89:0/601266754 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68400d2370 0x7f684015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:44.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6845600700 1 -- 172.21.15.89:0/601266754 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68400d2370 msgr2=0x7f684015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:44.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6845600700 1 --2- 172.21.15.89:0/601266754 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68400d2370 0x7f684015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:44.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6845600700 1 -- 172.21.15.89:0/601266754 >> v2:172.21.15.113:3300/0 conn(0x7f68400d16a0 msgr2=0x7f68400d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:44.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6845600700 1 --2- 172.21.15.89:0/601266754 >> v2:172.21.15.113:3300/0 conn(0x7f68400d16a0 0x7f68400d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:44.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.173+0000 7f6845600700 1 -- 172.21.15.89:0/601266754 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68401619e0 con 0x7f684015d9d0 2024-04-02T09:13:44.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.174+0000 7f6845600700 1 --2- 172.21.15.89:0/601266754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684015d9d0 0x7f68400d3c30 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7f68400d12f0 tx=0x7f6834009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aaa1d7822423fdf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:44.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.174+0000 7f683fe00700 1 -- 172.21.15.89:0/601266754 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683400b870 con 0x7f684015d9d0 2024-04-02T09:13:44.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.174+0000 7f683fe00700 1 -- 172.21.15.89:0/601266754 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683400b9d0 con 0x7f684015d9d0 2024-04-02T09:13:44.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.174+0000 7f683fe00700 1 -- 172.21.15.89:0/601266754 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683400bd60 con 0x7f684015d9d0 2024-04-02T09:13:44.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.174+0000 7f6848000700 1 -- 172.21.15.89:0/601266754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684015d9d0 msgr2=0x7f68400d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:44.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.174+0000 7f6848000700 1 --2- 172.21.15.89:0/601266754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684015d9d0 0x7f68400d3c30 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7f68400d12f0 tx=0x7f6834009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:44.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.175+0000 7f6848000700 1 -- 172.21.15.89:0/601266754 shutdown_connections 2024-04-02T09:13:44.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.175+0000 7f6848000700 1 --2- 172.21.15.89:0/601266754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684015d9d0 0x7f68400d3c30 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:44.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.175+0000 7f6848000700 1 --2- 172.21.15.89:0/601266754 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68400d2370 0x7f684015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:44.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.175+0000 7f6848000700 1 --2- 172.21.15.89:0/601266754 >> v2:172.21.15.113:3300/0 conn(0x7f68400d16a0 0x7f68400d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:44.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.175+0000 7f6848000700 1 -- 172.21.15.89:0/601266754 >> 172.21.15.89:0/601266754 conn(0x7f684008aff0 msgr2=0x7f68400d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:44.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.175+0000 7f6848000700 1 -- 172.21.15.89:0/601266754 shutdown_connections 2024-04-02T09:13:44.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.175+0000 7f6848000700 1 -- 172.21.15.89:0/601266754 wait complete. 2024-04-02T09:13:44.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.176+0000 7f6848000700 1 Processor -- start 2024-04-02T09:13:44.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.177+0000 7f6848000700 1 -- start start 2024-04-02T09:13:44.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.177+0000 7f6848000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68400d16a0 0x7f68400ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:44.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.177+0000 7f6848000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68400d2370 0x7f684014a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:44.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.177+0000 7f6848000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684015d9d0 0x7f684014ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:44.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.177+0000 7f6848000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68400cac80 con 0x7f684015d9d0 2024-04-02T09:13:44.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.177+0000 7f6848000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68400cd390 con 0x7f68400d2370 2024-04-02T09:13:44.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.177+0000 7f6848000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68400cd670 con 0x7f68400d16a0 2024-04-02T09:13:44.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.178+0000 7f6846000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684015d9d0 0x7f684014ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:44.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.178+0000 7f6846000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684015d9d0 0x7f684014ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55260/0 (socket says 172.21.15.89:55260) 2024-04-02T09:13:44.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.178+0000 7f6846000700 1 -- 172.21.15.89:0/1724122874 learned_addr learned my addr 172.21.15.89:0/1724122874 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:44.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.178+0000 7f6845600700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68400d16a0 0x7f68400ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:44.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.178+0000 7f6844c00700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68400d2370 0x7f684014a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:44.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.178+0000 7f6846000700 1 -- 172.21.15.89:0/1724122874 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68400d16a0 msgr2=0x7f68400ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:44.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.178+0000 7f6846000700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68400d16a0 0x7f68400ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:44.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.178+0000 7f6846000700 1 -- 172.21.15.89:0/1724122874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68400d2370 msgr2=0x7f684014a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:44.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.178+0000 7f6846000700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68400d2370 0x7f684014a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:44.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.178+0000 7f6846000700 1 -- 172.21.15.89:0/1724122874 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6834009550 con 0x7f684015d9d0 2024-04-02T09:13:44.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.178+0000 7f6844c00700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68400d2370 0x7f684014a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:44.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.178+0000 7f6846000700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684015d9d0 0x7f684014ac00 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f6838007e40 tx=0x7f683800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:44.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.179+0000 7f683d600700 1 -- 172.21.15.89:0/1724122874 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683800ede0 con 0x7f684015d9d0 2024-04-02T09:13:44.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.179+0000 7f683d600700 1 -- 172.21.15.89:0/1724122874 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6838014420 con 0x7f684015d9d0 2024-04-02T09:13:44.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.179+0000 7f683d600700 1 -- 172.21.15.89:0/1724122874 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68380147d0 con 0x7f684015d9d0 2024-04-02T09:13:44.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.179+0000 7f6848000700 1 -- 172.21.15.89:0/1724122874 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68400cd950 con 0x7f684015d9d0 2024-04-02T09:13:44.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.179+0000 7f6848000700 1 -- 172.21.15.89:0/1724122874 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684014a7f0 con 0x7f684015d9d0 2024-04-02T09:13:44.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.180+0000 7f6848000700 1 -- 172.21.15.89:0/1724122874 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68400d1ca0 con 0x7f684015d9d0 2024-04-02T09:13:44.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.181+0000 7f683d600700 1 -- 172.21.15.89:0/1724122874 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f6838014930 con 0x7f684015d9d0 2024-04-02T09:13:44.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.182+0000 7f683d600700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f682006aed0 0x7f682006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:44.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.182+0000 7f683d600700 1 -- 172.21.15.89:0/1724122874 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f683808a2c0 con 0x7f684015d9d0 2024-04-02T09:13:44.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.182+0000 7f6845600700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f682006aed0 0x7f682006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:44.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.182+0000 7f6845600700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f682006aed0 0x7f682006d390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6834014040 tx=0x7f6834008030 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:44.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.186+0000 7f683d600700 1 -- 172.21.15.89:0/1724122874 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6838054cf0 con 0x7f684015d9d0 2024-04-02T09:13:44.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:44.442+0000 7f6848000700 1 -- 172.21.15.89:0/1724122874 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f68400b7d60 con 0x7f684015d9d0 2024-04-02T09:13:45.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.447+0000 7f683d600700 1 -- 172.21.15.89:0/1724122874 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v125) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6838058310 con 0x7f684015d9d0 2024-04-02T09:13:45.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.450+0000 7f6848000700 1 -- 172.21.15.89:0/1724122874 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f682006aed0 msgr2=0x7f682006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:45.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.450+0000 7f6848000700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f682006aed0 0x7f682006d390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6834014040 tx=0x7f6834008030 comp rx=0 tx=0).stop 2024-04-02T09:13:45.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.450+0000 7f6848000700 1 -- 172.21.15.89:0/1724122874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684015d9d0 msgr2=0x7f684014ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:45.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.450+0000 7f6848000700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684015d9d0 0x7f684014ac00 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f6838007e40 tx=0x7f683800e3c0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.450+0000 7f6848000700 1 -- 172.21.15.89:0/1724122874 shutdown_connections 2024-04-02T09:13:45.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.450+0000 7f6848000700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f684015d9d0 0x7f684014ac00 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.450+0000 7f6848000700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68400d16a0 0x7f68400ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.450+0000 7f6848000700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f682006aed0 0x7f682006d390 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.450+0000 7f6848000700 1 --2- 172.21.15.89:0/1724122874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68400d2370 0x7f684014a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.450+0000 7f6848000700 1 -- 172.21.15.89:0/1724122874 >> 172.21.15.89:0/1724122874 conn(0x7f684008aff0 msgr2=0x7f68400ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:45.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.450+0000 7f6848000700 1 -- 172.21.15.89:0/1724122874 shutdown_connections 2024-04-02T09:13:45.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.450+0000 7f6848000700 1 -- 172.21.15.89:0/1724122874 wait complete. 2024-04-02T09:13:45.469 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2024-04-02T09:13:45.469 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:13:45.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.664+0000 7f8390600700 1 Processor -- start 2024-04-02T09:13:45.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.664+0000 7f8390600700 1 -- start start 2024-04-02T09:13:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.665+0000 7f8390600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f83880cb3f0 0x7f83880ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.665+0000 7f8390600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f83880ac600 0x7f838806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:45.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.665+0000 7f8390600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f838815fd50 con 0x7f838815b5d0 2024-04-02T09:13:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.665+0000 7f8390600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f838815b9b0 con 0x7f83880cb3f0 2024-04-02T09:13:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.665+0000 7f8390600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83880aca10 con 0x7f83880ac600 2024-04-02T09:13:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.665+0000 7f838d200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f83880cb3f0 0x7f83880ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.665+0000 7f838d200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f83880cb3f0 0x7f83880ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55270/0 (socket says 172.21.15.89:55270) 2024-04-02T09:13:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.665+0000 7f838d200700 1 -- 172.21.15.89:0/2188698484 learned_addr learned my addr 172.21.15.89:0/2188698484 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.666+0000 7f838e600700 1 --2- 172.21.15.89:0/2188698484 >> v2:172.21.15.113:3300/0 conn(0x7f83880ac600 0x7f838806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.666+0000 7f838d200700 1 -- 172.21.15.89:0/2188698484 >> v2:172.21.15.113:3300/0 conn(0x7f83880ac600 msgr2=0x7f838806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.666+0000 7f838d200700 1 --2- 172.21.15.89:0/2188698484 >> v2:172.21.15.113:3300/0 conn(0x7f83880ac600 0x7f838806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.666+0000 7f838d200700 1 -- 172.21.15.89:0/2188698484 >> v1:172.21.15.113:6789/0 conn(0x7f838815b5d0 legacy=0x7f83880ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.666+0000 7f838d200700 1 -- 172.21.15.89:0/2188698484 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838815f140 con 0x7f83880cb3f0 2024-04-02T09:13:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.666+0000 7f838d200700 1 --2- 172.21.15.89:0/2188698484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83880cb3f0 0x7f83880ac0c0 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f83880cfdc0 tx=0x7f8374009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cfb022dd2ac8aad server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.666+0000 7f8387e00700 1 -- 172.21.15.89:0/2188698484 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8374004230 con 0x7f83880cb3f0 2024-04-02T09:13:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.666+0000 7f8387e00700 1 -- 172.21.15.89:0/2188698484 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8374004390 con 0x7f83880cb3f0 2024-04-02T09:13:45.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.666+0000 7f8387e00700 1 -- 172.21.15.89:0/2188698484 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83740036a0 con 0x7f83880cb3f0 2024-04-02T09:13:45.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.667+0000 7f8390600700 1 -- 172.21.15.89:0/2188698484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83880cb3f0 msgr2=0x7f83880ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:45.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.667+0000 7f8390600700 1 --2- 172.21.15.89:0/2188698484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83880cb3f0 0x7f83880ac0c0 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f83880cfdc0 tx=0x7f8374009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.667+0000 7f8390600700 1 -- 172.21.15.89:0/2188698484 shutdown_connections 2024-04-02T09:13:45.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.667+0000 7f8390600700 1 --2- 172.21.15.89:0/2188698484 >> v2:172.21.15.113:3300/0 conn(0x7f83880ac600 0x7f838806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.667+0000 7f8390600700 1 --2- 172.21.15.89:0/2188698484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83880cb3f0 0x7f83880ac0c0 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.667+0000 7f8390600700 1 -- 172.21.15.89:0/2188698484 >> 172.21.15.89:0/2188698484 conn(0x7f83880cf190 msgr2=0x7f83880cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:45.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.667+0000 7f8390600700 1 -- 172.21.15.89:0/2188698484 shutdown_connections 2024-04-02T09:13:45.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.667+0000 7f8390600700 1 -- 172.21.15.89:0/2188698484 wait complete. 2024-04-02T09:13:45.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.669+0000 7f8390600700 1 Processor -- start 2024-04-02T09:13:45.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.669+0000 7f8390600700 1 -- start start 2024-04-02T09:13:45.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.670+0000 7f8390600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83880ac600 0x7f838815a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:45.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.670+0000 7f8390600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f838815b5d0 0x7f83881f6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:45.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.670+0000 7f8390600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83880c8e70 0x7f83881f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:45.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.670+0000 7f8390600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838815aea0 con 0x7f83880c8e70 2024-04-02T09:13:45.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.670+0000 7f8390600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880c9f90 con 0x7f838815b5d0 2024-04-02T09:13:45.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.670+0000 7f8390600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83880af580 con 0x7f83880ac600 2024-04-02T09:13:45.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.670+0000 7f838e600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83880c8e70 0x7f83881f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:45.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.670+0000 7f838e600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83880c8e70 0x7f83881f6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55280/0 (socket says 172.21.15.89:55280) 2024-04-02T09:13:45.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.670+0000 7f838e600700 1 -- 172.21.15.89:0/3510164152 learned_addr learned my addr 172.21.15.89:0/3510164152 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:45.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.670+0000 7f838e600700 1 -- 172.21.15.89:0/3510164152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83880ac600 msgr2=0x7f838815a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:45.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.670+0000 7f838e600700 1 --2- 172.21.15.89:0/3510164152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83880ac600 0x7f838815a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.670+0000 7f838e600700 1 -- 172.21.15.89:0/3510164152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f838815b5d0 msgr2=0x7f83881f6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:45.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.670+0000 7f838e600700 1 --2- 172.21.15.89:0/3510164152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f838815b5d0 0x7f83881f6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.670+0000 7f838e600700 1 -- 172.21.15.89:0/3510164152 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8374009550 con 0x7f83880c8e70 2024-04-02T09:13:45.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.671+0000 7f838e600700 1 --2- 172.21.15.89:0/3510164152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83880c8e70 0x7f83881f6890 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f8380007e60 tx=0x7f838000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:45.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.671+0000 7f8385600700 1 -- 172.21.15.89:0/3510164152 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838000ede0 con 0x7f83880c8e70 2024-04-02T09:13:45.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.671+0000 7f8385600700 1 -- 172.21.15.89:0/3510164152 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8380014420 con 0x7f83880c8e70 2024-04-02T09:13:45.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.671+0000 7f8385600700 1 -- 172.21.15.89:0/3510164152 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83800147d0 con 0x7f83880c8e70 2024-04-02T09:13:45.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.671+0000 7f8390600700 1 -- 172.21.15.89:0/3510164152 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83880729a0 con 0x7f83880c8e70 2024-04-02T09:13:45.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.671+0000 7f8390600700 1 -- 172.21.15.89:0/3510164152 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83880c9660 con 0x7f83880c8e70 2024-04-02T09:13:45.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.672+0000 7f8390600700 1 -- 172.21.15.89:0/3510164152 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8388067720 con 0x7f83880c8e70 2024-04-02T09:13:45.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.673+0000 7f8385600700 1 -- 172.21.15.89:0/3510164152 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f8380014930 con 0x7f83880c8e70 2024-04-02T09:13:45.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.674+0000 7f8385600700 1 --2- 172.21.15.89:0/3510164152 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f836806aed0 0x7f836806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:45.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.674+0000 7f8385600700 1 -- 172.21.15.89:0/3510164152 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f838008a420 con 0x7f83880c8e70 2024-04-02T09:13:45.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.674+0000 7f838dc00700 1 --2- 172.21.15.89:0/3510164152 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f836806aed0 0x7f836806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:45.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.674+0000 7f838dc00700 1 --2- 172.21.15.89:0/3510164152 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f836806aed0 0x7f836806d390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f83880b0880 tx=0x7f837c007a20 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:45.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.678+0000 7f8385600700 1 -- 172.21.15.89:0/3510164152 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8380054db0 con 0x7f83880c8e70 2024-04-02T09:13:45.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.922+0000 7f8390600700 1 -- 172.21.15.89:0/3510164152 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83880ca0d0 con 0x7f83880c8e70 2024-04-02T09:13:45.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.923+0000 7f8385600700 1 -- 172.21.15.89:0/3510164152 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5968 (secure 0 0 0) 0x7f83800583d0 con 0x7f83880c8e70 2024-04-02T09:13:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:45.921 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":125,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9518,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3094781297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3094781297},{"type":"v1","addr":"172.21.15.89:6836","nonce":3094781297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":10978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1799351491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1799351491},{"type":"v1","addr":"172.21.15.113:6834","nonce":1799351491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:28.357996+0000","modified":"2024-04-02T09:13:31.369015+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11964":{"gid":11964,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6835/703852199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":703852199},{"type":"v1","addr":"172.21.15.113:6835","nonce":703852199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:41.933360+0000","modified":"2024-04-02T09:13:45.439462+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11958},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11958":{"gid":11958,"name":"a","rank":0,"incarnation":121,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6837/3112580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3112580185},{"type":"v1","addr":"172.21.15.89:6837","nonce":3112580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-02T09:13:45.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.925+0000 7f8390600700 1 -- 172.21.15.89:0/3510164152 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f836806aed0 msgr2=0x7f836806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:45.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.925+0000 7f8390600700 1 --2- 172.21.15.89:0/3510164152 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f836806aed0 0x7f836806d390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f83880b0880 tx=0x7f837c007a20 comp rx=0 tx=0).stop 2024-04-02T09:13:45.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.925+0000 7f8390600700 1 -- 172.21.15.89:0/3510164152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83880c8e70 msgr2=0x7f83881f6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:45.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.925+0000 7f8390600700 1 --2- 172.21.15.89:0/3510164152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83880c8e70 0x7f83881f6890 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f8380007e60 tx=0x7f838000e3c0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.925+0000 7f8390600700 1 -- 172.21.15.89:0/3510164152 shutdown_connections 2024-04-02T09:13:45.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.925+0000 7f8390600700 1 --2- 172.21.15.89:0/3510164152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83880c8e70 0x7f83881f6890 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.925+0000 7f8390600700 1 --2- 172.21.15.89:0/3510164152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83880ac600 0x7f838815a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.925+0000 7f8390600700 1 --2- 172.21.15.89:0/3510164152 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f836806aed0 0x7f836806d390 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.925+0000 7f8390600700 1 --2- 172.21.15.89:0/3510164152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f838815b5d0 0x7f83881f6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:45.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.925+0000 7f8390600700 1 -- 172.21.15.89:0/3510164152 >> 172.21.15.89:0/3510164152 conn(0x7f83880cf190 msgr2=0x7f83880afd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:45.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.925+0000 7f8390600700 1 -- 172.21.15.89:0/3510164152 shutdown_connections 2024-04-02T09:13:45.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:45.925+0000 7f8390600700 1 -- 172.21.15.89:0/3510164152 wait complete. 2024-04-02T09:13:45.925 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 125 2024-04-02T09:13:45.942 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:13:46.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.134+0000 7f231da00700 1 Processor -- start 2024-04-02T09:13:46.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.134+0000 7f231da00700 1 -- start start 2024-04-02T09:13:46.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.135+0000 7f231da00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23180d18b0 0x7f23180cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:46.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.135+0000 7f231da00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2318161590 0x7f2318161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:46.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.135+0000 7f231da00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23180ceda0 con 0x7f2318160f10 2024-04-02T09:13:46.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.135+0000 7f231da00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23180d1d30 con 0x7f2318161590 2024-04-02T09:13:46.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.135+0000 7f231da00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2318161f60 con 0x7f23180d18b0 2024-04-02T09:13:46.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.136+0000 7f2317400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2318161590 0x7f2318161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:46.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.136+0000 7f2317400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2318161590 0x7f2318161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55286/0 (socket says 172.21.15.89:55286) 2024-04-02T09:13:46.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.136+0000 7f2317400700 1 -- 172.21.15.89:0/2196365115 learned_addr learned my addr 172.21.15.89:0/2196365115 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:46.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.136+0000 7f2316000700 1 --2- 172.21.15.89:0/2196365115 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23180d18b0 0x7f23180cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:46.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.136+0000 7f2317400700 1 -- 172.21.15.89:0/2196365115 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23180d18b0 msgr2=0x7f23180cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:46.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.136+0000 7f2317400700 1 --2- 172.21.15.89:0/2196365115 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23180d18b0 0x7f23180cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.136+0000 7f2317400700 1 -- 172.21.15.89:0/2196365115 >> v1:172.21.15.113:6789/0 conn(0x7f2318160f10 legacy=0x7f23180d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:46.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.136+0000 7f2317400700 1 -- 172.21.15.89:0/2196365115 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23181612f0 con 0x7f2318161590 2024-04-02T09:13:46.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.136+0000 7f2317400700 1 --2- 172.21.15.89:0/2196365115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318161590 0x7f2318161a20 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f2310009e80 tx=0x7f231000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402d90e45e51d655 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:46.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.137+0000 7f2315600700 1 -- 172.21.15.89:0/2196365115 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231000e070 con 0x7f2318161590 2024-04-02T09:13:46.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.137+0000 7f2315600700 1 -- 172.21.15.89:0/2196365115 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231000b8d0 con 0x7f2318161590 2024-04-02T09:13:46.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.137+0000 7f2315600700 1 -- 172.21.15.89:0/2196365115 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231000bc80 con 0x7f2318161590 2024-04-02T09:13:46.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.137+0000 7f231da00700 1 -- 172.21.15.89:0/2196365115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318161590 msgr2=0x7f2318161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:46.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.137+0000 7f231da00700 1 --2- 172.21.15.89:0/2196365115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318161590 0x7f2318161a20 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f2310009e80 tx=0x7f231000b330 comp rx=0 tx=0).stop 2024-04-02T09:13:46.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.137+0000 7f231da00700 1 -- 172.21.15.89:0/2196365115 shutdown_connections 2024-04-02T09:13:46.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.137+0000 7f231da00700 1 --2- 172.21.15.89:0/2196365115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318161590 0x7f2318161a20 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.137+0000 7f231da00700 1 --2- 172.21.15.89:0/2196365115 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23180d18b0 0x7f23180cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.137+0000 7f231da00700 1 -- 172.21.15.89:0/2196365115 >> 172.21.15.89:0/2196365115 conn(0x7f231808aff0 msgr2=0x7f23180d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:46.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.137+0000 7f231da00700 1 -- 172.21.15.89:0/2196365115 shutdown_connections 2024-04-02T09:13:46.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.137+0000 7f231da00700 1 -- 172.21.15.89:0/2196365115 wait complete. 2024-04-02T09:13:46.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.139+0000 7f231da00700 1 Processor -- start 2024-04-02T09:13:46.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.139+0000 7f231da00700 1 -- start start 2024-04-02T09:13:46.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.140+0000 7f231da00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23180d18b0 0x7f23181f0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:46.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.140+0000 7f231da00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318160f10 0x7f23181f39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:46.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.140+0000 7f231da00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2318161590 0x7f2318158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:46.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.140+0000 7f231da00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23181f3ee0 con 0x7f2318160f10 2024-04-02T09:13:46.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.140+0000 7f231da00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2318156a80 con 0x7f23180d18b0 2024-04-02T09:13:46.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.140+0000 7f231da00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2318156d60 con 0x7f2318161590 2024-04-02T09:13:46.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.140+0000 7f2316000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318160f10 0x7f23181f39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:46.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.140+0000 7f2316000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318160f10 0x7f23181f39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55292/0 (socket says 172.21.15.89:55292) 2024-04-02T09:13:46.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.140+0000 7f2316000700 1 -- 172.21.15.89:0/3879583278 learned_addr learned my addr 172.21.15.89:0/3879583278 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:46.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f2317400700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2318161590 0x7f2318158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:46.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f2316000700 1 -- 172.21.15.89:0/3879583278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2318161590 msgr2=0x7f2318158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:46.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f2316a00700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23180d18b0 0x7f23181f0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:46.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f2316000700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2318161590 0x7f2318158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f2316000700 1 -- 172.21.15.89:0/3879583278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23180d18b0 msgr2=0x7f23181f0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:46.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f2316000700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23180d18b0 0x7f23181f0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f2316000700 1 -- 172.21.15.89:0/3879583278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2310009b30 con 0x7f2318160f10 2024-04-02T09:13:46.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f2317400700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2318161590 0x7f2318158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:46.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f2316a00700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23180d18b0 0x7f23181f0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:46.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f2316000700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318160f10 0x7f23181f39a0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f230400e870 tx=0x7f230400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:46.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f230ea00700 1 -- 172.21.15.89:0/3879583278 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2304009590 con 0x7f2318160f10 2024-04-02T09:13:46.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f230ea00700 1 -- 172.21.15.89:0/3879583278 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230400ee70 con 0x7f2318160f10 2024-04-02T09:13:46.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f230ea00700 1 -- 172.21.15.89:0/3879583278 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230400ca50 con 0x7f2318160f10 2024-04-02T09:13:46.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f231da00700 1 -- 172.21.15.89:0/3879583278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2318157040 con 0x7f2318160f10 2024-04-02T09:13:46.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.141+0000 7f231da00700 1 -- 172.21.15.89:0/3879583278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2318157600 con 0x7f2318160f10 2024-04-02T09:13:46.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.142+0000 7f231da00700 1 -- 172.21.15.89:0/3879583278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2318067720 con 0x7f2318160f10 2024-04-02T09:13:46.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.143+0000 7f230ea00700 1 -- 172.21.15.89:0/3879583278 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f230400cbb0 con 0x7f2318160f10 2024-04-02T09:13:46.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.144+0000 7f230ea00700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f22f806aed0 0x7f22f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:46.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.144+0000 7f230ea00700 1 -- 172.21.15.89:0/3879583278 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f2304089d10 con 0x7f2318160f10 2024-04-02T09:13:46.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.144+0000 7f2316a00700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f22f806aed0 0x7f22f806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:46.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.145+0000 7f2316a00700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f22f806aed0 0x7f22f806d390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2318157fd0 tx=0x7f2308009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:46.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.149+0000 7f230ea00700 1 -- 172.21.15.89:0/3879583278 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f230408f020 con 0x7f2318160f10 2024-04-02T09:13:46.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.356+0000 7f231da00700 1 -- 172.21.15.89:0/3879583278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f231806c890 con 0x7f2318160f10 2024-04-02T09:13:46.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.357+0000 7f230ea00700 1 -- 172.21.15.89:0/3879583278 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) v1 ==== 74+0+20404 (secure 0 0 0) 0x7f23040545d0 con 0x7f2318160f10 2024-04-02T09:13:46.356 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:46.356 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":248,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:13:42.935848+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:13:26.654647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-02T09:13:27.342285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:13:40.048039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:13:40.907706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:13:46.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.360+0000 7f231da00700 1 -- 172.21.15.89:0/3879583278 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f22f806aed0 msgr2=0x7f22f806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:46.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.360+0000 7f231da00700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f22f806aed0 0x7f22f806d390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2318157fd0 tx=0x7f2308009040 comp rx=0 tx=0).stop 2024-04-02T09:13:46.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.360+0000 7f231da00700 1 -- 172.21.15.89:0/3879583278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318160f10 msgr2=0x7f23181f39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:46.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.360+0000 7f231da00700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318160f10 0x7f23181f39a0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f230400e870 tx=0x7f230400ec30 comp rx=0 tx=0).stop 2024-04-02T09:13:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.360+0000 7f231da00700 1 -- 172.21.15.89:0/3879583278 shutdown_connections 2024-04-02T09:13:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.360+0000 7f231da00700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f22f806aed0 0x7f22f806d390 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.360+0000 7f231da00700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23180d18b0 0x7f23181f0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.360+0000 7f231da00700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2318161590 0x7f2318158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.360+0000 7f231da00700 1 --2- 172.21.15.89:0/3879583278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318160f10 0x7f23181f39a0 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.360+0000 7f231da00700 1 -- 172.21.15.89:0/3879583278 >> 172.21.15.89:0/3879583278 conn(0x7f231808aff0 msgr2=0x7f231806bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.360+0000 7f231da00700 1 -- 172.21.15.89:0/3879583278 shutdown_connections 2024-04-02T09:13:46.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.360+0000 7f231da00700 1 -- 172.21.15.89:0/3879583278 wait complete. 2024-04-02T09:13:46.377 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T09:13:46.377 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:13:46.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.572+0000 7f5961e00700 1 Processor -- start 2024-04-02T09:13:46.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.572+0000 7f5961e00700 1 -- start start 2024-04-02T09:13:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.573+0000 7f5961e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f595c15d920 0x7f595c06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.574+0000 7f5961e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f595c09a910 0x7f595c0cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.574+0000 7f5961e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f595c0d08d0 con 0x7f595c099a60 2024-04-02T09:13:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.574+0000 7f5961e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f595c06c300 con 0x7f595c15d920 2024-04-02T09:13:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.574+0000 7f5961e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f595c06b9d0 con 0x7f595c09a910 2024-04-02T09:13:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.574+0000 7f595aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f595c15d920 0x7f595c06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.574+0000 7f595aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f595c15d920 0x7f595c06c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55304/0 (socket says 172.21.15.89:55304) 2024-04-02T09:13:46.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.574+0000 7f595b400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f595c099a60 0x7f595c099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58314/0 (socket says 172.21.15.89:58314) 2024-04-02T09:13:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.574+0000 7f595aa00700 1 -- 172.21.15.89:0/2710743401 learned_addr learned my addr 172.21.15.89:0/2710743401 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:46.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.574+0000 7f595a000700 1 --2- 172.21.15.89:0/2710743401 >> v2:172.21.15.113:3300/0 conn(0x7f595c09a910 0x7f595c0cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:46.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.574+0000 7f595aa00700 1 -- 172.21.15.89:0/2710743401 >> v2:172.21.15.113:3300/0 conn(0x7f595c09a910 msgr2=0x7f595c0cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:46.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.574+0000 7f595aa00700 1 --2- 172.21.15.89:0/2710743401 >> v2:172.21.15.113:3300/0 conn(0x7f595c09a910 0x7f595c0cfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.574+0000 7f595aa00700 1 -- 172.21.15.89:0/2710743401 >> v1:172.21.15.89:6789/0 conn(0x7f595c099a60 legacy=0x7f595c099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:46.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.574+0000 7f595aa00700 1 -- 172.21.15.89:0/2710743401 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595c06bcb0 con 0x7f595c15d920 2024-04-02T09:13:46.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.575+0000 7f595aa00700 1 --2- 172.21.15.89:0/2710743401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f595c15d920 0x7f595c06c790 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f595c0c4e50 tx=0x7f594c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500eb7002be73eb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:46.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.575+0000 7f5959600700 1 -- 172.21.15.89:0/2710743401 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c011830 con 0x7f595c15d920 2024-04-02T09:13:46.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.575+0000 7f5959600700 1 -- 172.21.15.89:0/2710743401 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594c011990 con 0x7f595c15d920 2024-04-02T09:13:46.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.575+0000 7f5959600700 1 -- 172.21.15.89:0/2710743401 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c011d20 con 0x7f595c15d920 2024-04-02T09:13:46.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.575+0000 7f5961e00700 1 -- 172.21.15.89:0/2710743401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f595c15d920 msgr2=0x7f595c06c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:46.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.575+0000 7f5961e00700 1 --2- 172.21.15.89:0/2710743401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f595c15d920 0x7f595c06c790 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f595c0c4e50 tx=0x7f594c009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.575+0000 7f5961e00700 1 -- 172.21.15.89:0/2710743401 shutdown_connections 2024-04-02T09:13:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.575+0000 7f5961e00700 1 --2- 172.21.15.89:0/2710743401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f595c15d920 0x7f595c06c790 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.575+0000 7f5961e00700 1 --2- 172.21.15.89:0/2710743401 >> v2:172.21.15.113:3300/0 conn(0x7f595c09a910 0x7f595c0cfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.575+0000 7f5961e00700 1 -- 172.21.15.89:0/2710743401 >> 172.21.15.89:0/2710743401 conn(0x7f595c0d1350 msgr2=0x7f595c0d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.576+0000 7f5961e00700 1 -- 172.21.15.89:0/2710743401 shutdown_connections 2024-04-02T09:13:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.576+0000 7f5961e00700 1 -- 172.21.15.89:0/2710743401 wait complete. 2024-04-02T09:13:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.577+0000 7f5961e00700 1 Processor -- start 2024-04-02T09:13:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.577+0000 7f5961e00700 1 -- start start 2024-04-02T09:13:46.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.578+0000 7f5961e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f595c099a60 0x7f595c15e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:46.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.578+0000 7f5961e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f595c09a910 0x7f595c1f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:46.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.578+0000 7f5961e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f595c15d920 0x7f595c0b1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:46.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.578+0000 7f5961e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595c0c92f0 con 0x7f595c09a910 2024-04-02T09:13:46.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.578+0000 7f5961e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595c0b08e0 con 0x7f595c099a60 2024-04-02T09:13:46.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.578+0000 7f5961e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595c0c8420 con 0x7f595c15d920 2024-04-02T09:13:46.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.578+0000 7f595a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f595c09a910 0x7f595c1f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:46.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.578+0000 7f595a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f595c09a910 0x7f595c1f67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55308/0 (socket says 172.21.15.89:55308) 2024-04-02T09:13:46.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.578+0000 7f595a000700 1 -- 172.21.15.89:0/1264882069 learned_addr learned my addr 172.21.15.89:0/1264882069 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:46.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.579+0000 7f595aa00700 1 --2- 172.21.15.89:0/1264882069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f595c099a60 0x7f595c15e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:46.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.579+0000 7f595b400700 1 --2- 172.21.15.89:0/1264882069 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f595c15d920 0x7f595c0b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:46.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.579+0000 7f595a000700 1 -- 172.21.15.89:0/1264882069 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f595c15d920 msgr2=0x7f595c0b1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:46.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.579+0000 7f595a000700 1 --2- 172.21.15.89:0/1264882069 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f595c15d920 0x7f595c0b1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.579+0000 7f595a000700 1 -- 172.21.15.89:0/1264882069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f595c099a60 msgr2=0x7f595c15e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:46.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.579+0000 7f595a000700 1 --2- 172.21.15.89:0/1264882069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f595c099a60 0x7f595c15e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.579+0000 7f595a000700 1 -- 172.21.15.89:0/1264882069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c009550 con 0x7f595c09a910 2024-04-02T09:13:46.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.579+0000 7f595a000700 1 --2- 172.21.15.89:0/1264882069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f595c09a910 0x7f595c1f67d0 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f594800e980 tx=0x7f594800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:46.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.579+0000 7f5953400700 1 -- 172.21.15.89:0/1264882069 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5948009710 con 0x7f595c09a910 2024-04-02T09:13:46.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.579+0000 7f5953400700 1 -- 172.21.15.89:0/1264882069 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594800eed0 con 0x7f595c09a910 2024-04-02T09:13:46.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.579+0000 7f5961e00700 1 -- 172.21.15.89:0/1264882069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f595c15e740 con 0x7f595c09a910 2024-04-02T09:13:46.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.579+0000 7f5953400700 1 -- 172.21.15.89:0/1264882069 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594800ca90 con 0x7f595c09a910 2024-04-02T09:13:46.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.579+0000 7f5961e00700 1 -- 172.21.15.89:0/1264882069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f595c0c8950 con 0x7f595c09a910 2024-04-02T09:13:46.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.580+0000 7f5961e00700 1 -- 172.21.15.89:0/1264882069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f595c0c9430 con 0x7f595c09a910 2024-04-02T09:13:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.581+0000 7f5953400700 1 -- 172.21.15.89:0/1264882069 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f594800cbf0 con 0x7f595c09a910 2024-04-02T09:13:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.583+0000 7f5953400700 1 --2- 172.21.15.89:0/1264882069 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f593c06aed0 0x7f593c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.583+0000 7f5953400700 1 -- 172.21.15.89:0/1264882069 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f5948089ff0 con 0x7f595c09a910 2024-04-02T09:13:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.583+0000 7f595aa00700 1 --2- 172.21.15.89:0/1264882069 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f593c06aed0 0x7f593c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.584+0000 7f595aa00700 1 --2- 172.21.15.89:0/1264882069 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f593c06aed0 0x7f593c06d390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f594c014040 tx=0x7f594c008030 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:46.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.587+0000 7f5953400700 1 -- 172.21.15.89:0/1264882069 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59480bd580 con 0x7f595c09a910 2024-04-02T09:13:46.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.781+0000 7f5961e00700 1 -- 172.21.15.89:0/1264882069 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f595c067720 con 0x7f593c06aed0 2024-04-02T09:13:46.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.796+0000 7f5953400700 1 -- 172.21.15.89:0/1264882069 <== mgr.11613 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+517525 (secure 0 0 0) 0x7f595c067720 con 0x7f593c06aed0 2024-04-02T09:13:46.801 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:46.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.802+0000 7f5961e00700 1 -- 172.21.15.89:0/1264882069 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f593c06aed0 msgr2=0x7f593c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:46.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.802+0000 7f5961e00700 1 --2- 172.21.15.89:0/1264882069 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f593c06aed0 0x7f593c06d390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f594c014040 tx=0x7f594c008030 comp rx=0 tx=0).stop 2024-04-02T09:13:46.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.802+0000 7f5961e00700 1 -- 172.21.15.89:0/1264882069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f595c09a910 msgr2=0x7f595c1f67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:46.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.802+0000 7f5961e00700 1 --2- 172.21.15.89:0/1264882069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f595c09a910 0x7f595c1f67d0 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f594800e980 tx=0x7f594800ec90 comp rx=0 tx=0).stop 2024-04-02T09:13:46.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.802+0000 7f5961e00700 1 -- 172.21.15.89:0/1264882069 shutdown_connections 2024-04-02T09:13:46.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.802+0000 7f5961e00700 1 --2- 172.21.15.89:0/1264882069 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f593c06aed0 0x7f593c06d390 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.802+0000 7f5961e00700 1 --2- 172.21.15.89:0/1264882069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f595c099a60 0x7f595c15e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.802+0000 7f5961e00700 1 --2- 172.21.15.89:0/1264882069 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f595c15d920 0x7f595c0b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.802+0000 7f5961e00700 1 --2- 172.21.15.89:0/1264882069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f595c09a910 0x7f595c1f67d0 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:46.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.802+0000 7f5961e00700 1 -- 172.21.15.89:0/1264882069 >> 172.21.15.89:0/1264882069 conn(0x7f595c0d1350 msgr2=0x7f595c06f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:46.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.802+0000 7f5961e00700 1 -- 172.21.15.89:0/1264882069 shutdown_connections 2024-04-02T09:13:46.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:46.803+0000 7f5961e00700 1 -- 172.21.15.89:0/1264882069 wait complete. 2024-04-02T09:13:46.807 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:13:46.819 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-04-02T09:13:45.432285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593971,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":666,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":203,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":412864,"kb_used_data":25312,"kb_used_omap":12,"kb_used_meta":387443,"kb_avail":754561856,"statfs":{"total":773094113280,"available":772671340544,"internally_reserved":0,"allocated":25919488,"data_stored":13801624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":396742200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":987,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.011247"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425059+0000","last_change":"2024-04-02T09:13:40.425059+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425059+0000","last_undegraded":"2024-04-02T09:13:40.425059+0000","last_fullsized":"2024-04-02T09:13:40.425059+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:12:22.769587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426693+0000","last_change":"2024-04-02T09:13:28.396642+0000","last_active":"2024-04-02T09:13:31.426693+0000","last_peered":"2024-04-02T09:13:31.426693+0000","last_clean":"2024-04-02T09:13:31.426693+0000","last_became_active":"2024-04-02T09:13:28.396356+0000","last_became_peered":"2024-04-02T09:13:28.396356+0000","last_unstale":"2024-04-02T09:13:31.426693+0000","last_undegraded":"2024-04-02T09:13:31.426693+0000","last_fullsized":"2024-04-02T09:13:31.426693+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:34.295480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422195+0000","last_change":"2024-04-02T09:13:40.422195+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422195+0000","last_undegraded":"2024-04-02T09:13:40.422195+0000","last_fullsized":"2024-04-02T09:13:40.422195+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:24:28.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424078+0000","last_change":"2024-04-02T09:13:28.381100+0000","last_active":"2024-04-02T09:13:31.424078+0000","last_peered":"2024-04-02T09:13:31.424078+0000","last_clean":"2024-04-02T09:13:31.424078+0000","last_became_active":"2024-04-02T09:13:28.380832+0000","last_became_peered":"2024-04-02T09:13:28.380832+0000","last_unstale":"2024-04-02T09:13:31.424078+0000","last_undegraded":"2024-04-02T09:13:31.424078+0000","last_fullsized":"2024-04-02T09:13:31.424078+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:21.987215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.426705+0000","last_change":"2024-04-02T09:13:40.426705+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.426705+0000","last_undegraded":"2024-04-02T09:13:40.426705+0000","last_fullsized":"2024-04-02T09:13:40.426705+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:31:39.320935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426652+0000","last_change":"2024-04-02T09:13:28.393942+0000","last_active":"2024-04-02T09:13:31.426652+0000","last_peered":"2024-04-02T09:13:31.426652+0000","last_clean":"2024-04-02T09:13:31.426652+0000","last_became_active":"2024-04-02T09:13:28.393467+0000","last_became_peered":"2024-04-02T09:13:28.393467+0000","last_unstale":"2024-04-02T09:13:31.426652+0000","last_undegraded":"2024-04-02T09:13:31.426652+0000","last_fullsized":"2024-04-02T09:13:31.426652+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:24.836500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.427104+0000","last_change":"2024-04-02T09:13:40.427104+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.427104+0000","last_undegraded":"2024-04-02T09:13:40.427104+0000","last_fullsized":"2024-04-02T09:13:40.427104+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:13:02.182890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426886+0000","last_change":"2024-04-02T09:13:28.394098+0000","last_active":"2024-04-02T09:13:31.426886+0000","last_peered":"2024-04-02T09:13:31.426886+0000","last_clean":"2024-04-02T09:13:31.426886+0000","last_became_active":"2024-04-02T09:13:28.393837+0000","last_became_peered":"2024-04-02T09:13:28.393837+0000","last_unstale":"2024-04-02T09:13:31.426886+0000","last_undegraded":"2024-04-02T09:13:31.426886+0000","last_fullsized":"2024-04-02T09:13:31.426886+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:40:27.068896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425796+0000","last_change":"2024-04-02T09:13:40.425796+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425796+0000","last_undegraded":"2024-04-02T09:13:40.425796+0000","last_fullsized":"2024-04-02T09:13:40.425796+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:53.327540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427560+0000","last_change":"2024-04-02T09:13:28.379239+0000","last_active":"2024-04-02T09:13:31.427560+0000","last_peered":"2024-04-02T09:13:31.427560+0000","last_clean":"2024-04-02T09:13:31.427560+0000","last_became_active":"2024-04-02T09:13:28.378970+0000","last_became_peered":"2024-04-02T09:13:28.378970+0000","last_unstale":"2024-04-02T09:13:31.427560+0000","last_undegraded":"2024-04-02T09:13:31.427560+0000","last_fullsized":"2024-04-02T09:13:31.427560+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:37.960638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422210+0000","last_change":"2024-04-02T09:13:40.422210+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422210+0000","last_undegraded":"2024-04-02T09:13:40.422210+0000","last_fullsized":"2024-04-02T09:13:40.422210+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:18.892470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426567+0000","last_change":"2024-04-02T09:13:28.393747+0000","last_active":"2024-04-02T09:13:31.426567+0000","last_peered":"2024-04-02T09:13:31.426567+0000","last_clean":"2024-04-02T09:13:31.426567+0000","last_became_active":"2024-04-02T09:13:28.393515+0000","last_became_peered":"2024-04-02T09:13:28.393515+0000","last_unstale":"2024-04-02T09:13:31.426567+0000","last_undegraded":"2024-04-02T09:13:31.426567+0000","last_fullsized":"2024-04-02T09:13:31.426567+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:28:37.462844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422411+0000","last_change":"2024-04-02T09:13:40.422411+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422411+0000","last_undegraded":"2024-04-02T09:13:40.422411+0000","last_fullsized":"2024-04-02T09:13:40.422411+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:53.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629507+0000","last_change":"2024-04-02T09:13:28.399975+0000","last_active":"2024-04-02T09:13:31.629507+0000","last_peered":"2024-04-02T09:13:31.629507+0000","last_clean":"2024-04-02T09:13:31.629507+0000","last_became_active":"2024-04-02T09:13:28.399335+0000","last_became_peered":"2024-04-02T09:13:28.399335+0000","last_unstale":"2024-04-02T09:13:31.629507+0000","last_undegraded":"2024-04-02T09:13:31.629507+0000","last_fullsized":"2024-04-02T09:13:31.629507+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:16:14.265813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422275+0000","last_change":"2024-04-02T09:13:40.422275+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422275+0000","last_undegraded":"2024-04-02T09:13:40.422275+0000","last_fullsized":"2024-04-02T09:13:40.422275+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:40.922210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427506+0000","last_change":"2024-04-02T09:13:28.392551+0000","last_active":"2024-04-02T09:13:31.427506+0000","last_peered":"2024-04-02T09:13:31.427506+0000","last_clean":"2024-04-02T09:13:31.427506+0000","last_became_active":"2024-04-02T09:13:28.392159+0000","last_became_peered":"2024-04-02T09:13:28.392159+0000","last_unstale":"2024-04-02T09:13:31.427506+0000","last_undegraded":"2024-04-02T09:13:31.427506+0000","last_fullsized":"2024-04-02T09:13:31.427506+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:42.551696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.427207+0000","last_change":"2024-04-02T09:13:40.427207+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.427207+0000","last_undegraded":"2024-04-02T09:13:40.427207+0000","last_fullsized":"2024-04-02T09:13:40.427207+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:08.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424242+0000","last_change":"2024-04-02T09:13:28.383891+0000","last_active":"2024-04-02T09:13:31.424242+0000","last_peered":"2024-04-02T09:13:31.424242+0000","last_clean":"2024-04-02T09:13:31.424242+0000","last_became_active":"2024-04-02T09:13:28.383628+0000","last_became_peered":"2024-04-02T09:13:28.383628+0000","last_unstale":"2024-04-02T09:13:31.424242+0000","last_undegraded":"2024-04-02T09:13:31.424242+0000","last_fullsized":"2024-04-02T09:13:31.424242+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:44.222993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422239+0000","last_change":"2024-04-02T09:13:40.422239+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422239+0000","last_undegraded":"2024-04-02T09:13:40.422239+0000","last_fullsized":"2024-04-02T09:13:40.422239+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:10.066167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629713+0000","last_change":"2024-04-02T09:13:28.404390+0000","last_active":"2024-04-02T09:13:31.629713+0000","last_peered":"2024-04-02T09:13:31.629713+0000","last_clean":"2024-04-02T09:13:31.629713+0000","last_became_active":"2024-04-02T09:13:28.404038+0000","last_became_peered":"2024-04-02T09:13:28.404038+0000","last_unstale":"2024-04-02T09:13:31.629713+0000","last_undegraded":"2024-04-02T09:13:31.629713+0000","last_fullsized":"2024-04-02T09:13:31.629713+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:47.359176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422238+0000","last_change":"2024-04-02T09:13:40.422238+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422238+0000","last_undegraded":"2024-04-02T09:13:40.422238+0000","last_fullsized":"2024-04-02T09:13:40.422238+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:50:01.664013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426922+0000","last_change":"2024-04-02T09:13:28.403517+0000","last_active":"2024-04-02T09:13:31.426922+0000","last_peered":"2024-04-02T09:13:31.426922+0000","last_clean":"2024-04-02T09:13:31.426922+0000","last_became_active":"2024-04-02T09:13:28.401353+0000","last_became_peered":"2024-04-02T09:13:28.401353+0000","last_unstale":"2024-04-02T09:13:31.426922+0000","last_undegraded":"2024-04-02T09:13:31.426922+0000","last_fullsized":"2024-04-02T09:13:31.426922+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:08.980542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425711+0000","last_change":"2024-04-02T09:13:40.425711+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425711+0000","last_undegraded":"2024-04-02T09:13:40.425711+0000","last_fullsized":"2024-04-02T09:13:40.425711+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:08.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629692+0000","last_change":"2024-04-02T09:13:28.399938+0000","last_active":"2024-04-02T09:13:31.629692+0000","last_peered":"2024-04-02T09:13:31.629692+0000","last_clean":"2024-04-02T09:13:31.629692+0000","last_became_active":"2024-04-02T09:13:28.399350+0000","last_became_peered":"2024-04-02T09:13:28.399350+0000","last_unstale":"2024-04-02T09:13:31.629692+0000","last_undegraded":"2024-04-02T09:13:31.629692+0000","last_fullsized":"2024-04-02T09:13:31.629692+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:05.933699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422193+0000","last_change":"2024-04-02T09:13:40.422193+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422193+0000","last_undegraded":"2024-04-02T09:13:40.422193+0000","last_fullsized":"2024-04-02T09:13:40.422193+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:53.842095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424157+0000","last_change":"2024-04-02T09:13:28.386688+0000","last_active":"2024-04-02T09:13:31.424157+0000","last_peered":"2024-04-02T09:13:31.424157+0000","last_clean":"2024-04-02T09:13:31.424157+0000","last_became_active":"2024-04-02T09:13:28.386363+0000","last_became_peered":"2024-04-02T09:13:28.386363+0000","last_unstale":"2024-04-02T09:13:31.424157+0000","last_undegraded":"2024-04-02T09:13:31.424157+0000","last_fullsized":"2024-04-02T09:13:31.424157+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:12:41.480404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422972+0000","last_change":"2024-04-02T09:13:40.422972+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422972+0000","last_undegraded":"2024-04-02T09:13:40.422972+0000","last_fullsized":"2024-04-02T09:13:40.422972+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:22:20.170595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426790+0000","last_change":"2024-04-02T09:13:28.401903+0000","last_active":"2024-04-02T09:13:31.426790+0000","last_peered":"2024-04-02T09:13:31.426790+0000","last_clean":"2024-04-02T09:13:31.426790+0000","last_became_active":"2024-04-02T09:13:28.399098+0000","last_became_peered":"2024-04-02T09:13:28.399098+0000","last_unstale":"2024-04-02T09:13:31.426790+0000","last_undegraded":"2024-04-02T09:13:31.426790+0000","last_fullsized":"2024-04-02T09:13:31.426790+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:33:56.420978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425751+0000","last_change":"2024-04-02T09:13:40.425751+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425751+0000","last_undegraded":"2024-04-02T09:13:40.425751+0000","last_fullsized":"2024-04-02T09:13:40.425751+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:53.688559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422619+0000","last_change":"2024-04-02T09:13:28.383146+0000","last_active":"2024-04-02T09:13:31.422619+0000","last_peered":"2024-04-02T09:13:31.422619+0000","last_clean":"2024-04-02T09:13:31.422619+0000","last_became_active":"2024-04-02T09:13:28.382879+0000","last_became_peered":"2024-04-02T09:13:28.382879+0000","last_unstale":"2024-04-02T09:13:31.422619+0000","last_undegraded":"2024-04-02T09:13:31.422619+0000","last_fullsized":"2024-04-02T09:13:31.422619+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:57:59.273299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425074+0000","last_change":"2024-04-02T09:13:40.425074+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425074+0000","last_undegraded":"2024-04-02T09:13:40.425074+0000","last_fullsized":"2024-04-02T09:13:40.425074+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:56.693535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.423992+0000","last_change":"2024-04-02T09:13:28.381499+0000","last_active":"2024-04-02T09:13:31.423992+0000","last_peered":"2024-04-02T09:13:31.423992+0000","last_clean":"2024-04-02T09:13:31.423992+0000","last_became_active":"2024-04-02T09:13:28.377983+0000","last_became_peered":"2024-04-02T09:13:28.377983+0000","last_unstale":"2024-04-02T09:13:31.423992+0000","last_undegraded":"2024-04-02T09:13:31.423992+0000","last_fullsized":"2024-04-02T09:13:31.423992+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:45.021173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.426751+0000","last_change":"2024-04-02T09:13:40.426751+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.426751+0000","last_undegraded":"2024-04-02T09:13:40.426751+0000","last_fullsized":"2024-04-02T09:13:40.426751+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:14.301601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629812+0000","last_change":"2024-04-02T09:13:28.382899+0000","last_active":"2024-04-02T09:13:31.629812+0000","last_peered":"2024-04-02T09:13:31.629812+0000","last_clean":"2024-04-02T09:13:31.629812+0000","last_became_active":"2024-04-02T09:13:28.382023+0000","last_became_peered":"2024-04-02T09:13:28.382023+0000","last_unstale":"2024-04-02T09:13:31.629812+0000","last_undegraded":"2024-04-02T09:13:31.629812+0000","last_fullsized":"2024-04-02T09:13:31.629812+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:12.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425230+0000","last_change":"2024-04-02T09:13:40.425230+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425230+0000","last_undegraded":"2024-04-02T09:13:40.425230+0000","last_fullsized":"2024-04-02T09:13:40.425230+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:40.922210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630208+0000","last_change":"2024-04-02T09:13:28.390594+0000","last_active":"2024-04-02T09:13:31.630208+0000","last_peered":"2024-04-02T09:13:31.630208+0000","last_clean":"2024-04-02T09:13:31.630208+0000","last_became_active":"2024-04-02T09:13:28.388061+0000","last_became_peered":"2024-04-02T09:13:28.388061+0000","last_unstale":"2024-04-02T09:13:31.630208+0000","last_undegraded":"2024-04-02T09:13:31.630208+0000","last_fullsized":"2024-04-02T09:13:31.630208+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:14:04.841237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422434+0000","last_change":"2024-04-02T09:13:40.422434+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422434+0000","last_undegraded":"2024-04-02T09:13:40.422434+0000","last_fullsized":"2024-04-02T09:13:40.422434+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:56.693535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424191+0000","last_change":"2024-04-02T09:13:28.382883+0000","last_active":"2024-04-02T09:13:31.424191+0000","last_peered":"2024-04-02T09:13:31.424191+0000","last_clean":"2024-04-02T09:13:31.424191+0000","last_became_active":"2024-04-02T09:13:28.382187+0000","last_became_peered":"2024-04-02T09:13:28.382187+0000","last_unstale":"2024-04-02T09:13:31.424191+0000","last_undegraded":"2024-04-02T09:13:31.424191+0000","last_fullsized":"2024-04-02T09:13:31.424191+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:21:26.362248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.426415+0000","last_change":"2024-04-02T09:13:40.426415+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.426415+0000","last_undegraded":"2024-04-02T09:13:40.426415+0000","last_fullsized":"2024-04-02T09:13:40.426415+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:19.429201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424015+0000","last_change":"2024-04-02T09:13:28.384295+0000","last_active":"2024-04-02T09:13:31.424015+0000","last_peered":"2024-04-02T09:13:31.424015+0000","last_clean":"2024-04-02T09:13:31.424015+0000","last_became_active":"2024-04-02T09:13:28.384010+0000","last_became_peered":"2024-04-02T09:13:28.384010+0000","last_unstale":"2024-04-02T09:13:31.424015+0000","last_undegraded":"2024-04-02T09:13:31.424015+0000","last_fullsized":"2024-04-02T09:13:31.424015+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:47:33.241308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425141+0000","last_change":"2024-04-02T09:13:40.425141+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425141+0000","last_undegraded":"2024-04-02T09:13:40.425141+0000","last_fullsized":"2024-04-02T09:13:40.425141+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:08.249339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427569+0000","last_change":"2024-04-02T09:13:28.380209+0000","last_active":"2024-04-02T09:13:31.427569+0000","last_peered":"2024-04-02T09:13:31.427569+0000","last_clean":"2024-04-02T09:13:31.427569+0000","last_became_active":"2024-04-02T09:13:28.379739+0000","last_became_peered":"2024-04-02T09:13:28.379739+0000","last_unstale":"2024-04-02T09:13:31.427569+0000","last_undegraded":"2024-04-02T09:13:31.427569+0000","last_fullsized":"2024-04-02T09:13:31.427569+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:46:14.163290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422802+0000","last_change":"2024-04-02T09:13:40.422802+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422802+0000","last_undegraded":"2024-04-02T09:13:40.422802+0000","last_fullsized":"2024-04-02T09:13:40.422802+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:47:51.824824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424141+0000","last_change":"2024-04-02T09:13:28.392993+0000","last_active":"2024-04-02T09:13:31.424141+0000","last_peered":"2024-04-02T09:13:31.424141+0000","last_clean":"2024-04-02T09:13:31.424141+0000","last_became_active":"2024-04-02T09:13:28.392712+0000","last_became_peered":"2024-04-02T09:13:28.392712+0000","last_unstale":"2024-04-02T09:13:31.424141+0000","last_undegraded":"2024-04-02T09:13:31.424141+0000","last_fullsized":"2024-04-02T09:13:31.424141+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:38.436370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422422+0000","last_change":"2024-04-02T09:13:40.422422+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422422+0000","last_undegraded":"2024-04-02T09:13:40.422422+0000","last_fullsized":"2024-04-02T09:13:40.422422+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:48:38.200651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424060+0000","last_change":"2024-04-02T09:13:28.378030+0000","last_active":"2024-04-02T09:13:31.424060+0000","last_peered":"2024-04-02T09:13:31.424060+0000","last_clean":"2024-04-02T09:13:31.424060+0000","last_became_active":"2024-04-02T09:13:28.377700+0000","last_became_peered":"2024-04-02T09:13:28.377700+0000","last_unstale":"2024-04-02T09:13:31.424060+0000","last_undegraded":"2024-04-02T09:13:31.424060+0000","last_fullsized":"2024-04-02T09:13:31.424060+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:40.236322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.428373+0000","last_change":"2024-04-02T09:13:40.428373+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.428373+0000","last_undegraded":"2024-04-02T09:13:40.428373+0000","last_fullsized":"2024-04-02T09:13:40.428373+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:27.266184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426698+0000","last_change":"2024-04-02T09:13:28.395364+0000","last_active":"2024-04-02T09:13:31.426698+0000","last_peered":"2024-04-02T09:13:31.426698+0000","last_clean":"2024-04-02T09:13:31.426698+0000","last_became_active":"2024-04-02T09:13:28.395100+0000","last_became_peered":"2024-04-02T09:13:28.395100+0000","last_unstale":"2024-04-02T09:13:31.426698+0000","last_undegraded":"2024-04-02T09:13:31.426698+0000","last_fullsized":"2024-04-02T09:13:31.426698+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:12.863676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425209+0000","last_change":"2024-04-02T09:13:40.425209+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425209+0000","last_undegraded":"2024-04-02T09:13:40.425209+0000","last_fullsized":"2024-04-02T09:13:40.425209+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:47:51.824824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426979+0000","last_change":"2024-04-02T09:13:28.399604+0000","last_active":"2024-04-02T09:13:31.426979+0000","last_peered":"2024-04-02T09:13:31.426979+0000","last_clean":"2024-04-02T09:13:31.426979+0000","last_became_active":"2024-04-02T09:13:28.399333+0000","last_became_peered":"2024-04-02T09:13:28.399333+0000","last_unstale":"2024-04-02T09:13:31.426979+0000","last_undegraded":"2024-04-02T09:13:31.426979+0000","last_fullsized":"2024-04-02T09:13:31.426979+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:18.748067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.428242+0000","last_change":"2024-04-02T09:13:40.428242+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.428242+0000","last_undegraded":"2024-04-02T09:13:40.428242+0000","last_fullsized":"2024-04-02T09:13:40.428242+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:19.429201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427635+0000","last_change":"2024-04-02T09:13:28.378770+0000","last_active":"2024-04-02T09:13:31.427635+0000","last_peered":"2024-04-02T09:13:31.427635+0000","last_clean":"2024-04-02T09:13:31.427635+0000","last_became_active":"2024-04-02T09:13:28.378521+0000","last_became_peered":"2024-04-02T09:13:28.378521+0000","last_unstale":"2024-04-02T09:13:31.427635+0000","last_undegraded":"2024-04-02T09:13:31.427635+0000","last_fullsized":"2024-04-02T09:13:31.427635+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:24:11.210144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.423010+0000","last_change":"2024-04-02T09:13:40.423010+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.423010+0000","last_undegraded":"2024-04-02T09:13:40.423010+0000","last_fullsized":"2024-04-02T09:13:40.423010+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:45:49.868260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424038+0000","last_change":"2024-04-02T09:13:28.389084+0000","last_active":"2024-04-02T09:13:31.424038+0000","last_peered":"2024-04-02T09:13:31.424038+0000","last_clean":"2024-04-02T09:13:31.424038+0000","last_became_active":"2024-04-02T09:13:28.388624+0000","last_became_peered":"2024-04-02T09:13:28.388624+0000","last_unstale":"2024-04-02T09:13:31.424038+0000","last_undegraded":"2024-04-02T09:13:31.424038+0000","last_fullsized":"2024-04-02T09:13:31.424038+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:23.766461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422211+0000","last_change":"2024-04-02T09:13:40.422211+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422211+0000","last_undegraded":"2024-04-02T09:13:40.422211+0000","last_fullsized":"2024-04-02T09:13:40.422211+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:45.017947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426864+0000","last_change":"2024-04-02T09:13:28.395286+0000","last_active":"2024-04-02T09:13:31.426864+0000","last_peered":"2024-04-02T09:13:31.426864+0000","last_clean":"2024-04-02T09:13:31.426864+0000","last_became_active":"2024-04-02T09:13:28.394377+0000","last_became_peered":"2024-04-02T09:13:28.394377+0000","last_unstale":"2024-04-02T09:13:31.426864+0000","last_undegraded":"2024-04-02T09:13:31.426864+0000","last_fullsized":"2024-04-02T09:13:31.426864+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:37:39.028435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425714+0000","last_change":"2024-04-02T09:13:40.425714+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425714+0000","last_undegraded":"2024-04-02T09:13:40.425714+0000","last_fullsized":"2024-04-02T09:13:40.425714+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:52.540226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426829+0000","last_change":"2024-04-02T09:13:28.391267+0000","last_active":"2024-04-02T09:13:31.426829+0000","last_peered":"2024-04-02T09:13:31.426829+0000","last_clean":"2024-04-02T09:13:31.426829+0000","last_became_active":"2024-04-02T09:13:28.390985+0000","last_became_peered":"2024-04-02T09:13:28.390985+0000","last_unstale":"2024-04-02T09:13:31.426829+0000","last_undegraded":"2024-04-02T09:13:31.426829+0000","last_fullsized":"2024-04-02T09:13:31.426829+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:30.210394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.427271+0000","last_change":"2024-04-02T09:13:40.427271+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.427271+0000","last_undegraded":"2024-04-02T09:13:40.427271+0000","last_fullsized":"2024-04-02T09:13:40.427271+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:23:34.898152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424106+0000","last_change":"2024-04-02T09:13:28.380650+0000","last_active":"2024-04-02T09:13:31.424106+0000","last_peered":"2024-04-02T09:13:31.424106+0000","last_clean":"2024-04-02T09:13:31.424106+0000","last_became_active":"2024-04-02T09:13:28.380376+0000","last_became_peered":"2024-04-02T09:13:28.380376+0000","last_unstale":"2024-04-02T09:13:31.424106+0000","last_undegraded":"2024-04-02T09:13:31.424106+0000","last_fullsized":"2024-04-02T09:13:31.424106+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:46:17.713244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.427715+0000","last_change":"2024-04-02T09:13:40.427715+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.427715+0000","last_undegraded":"2024-04-02T09:13:40.427715+0000","last_fullsized":"2024-04-02T09:13:40.427715+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:21.506479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426750+0000","last_change":"2024-04-02T09:13:28.390273+0000","last_active":"2024-04-02T09:13:31.426750+0000","last_peered":"2024-04-02T09:13:31.426750+0000","last_clean":"2024-04-02T09:13:31.426750+0000","last_became_active":"2024-04-02T09:13:28.389990+0000","last_became_peered":"2024-04-02T09:13:28.389990+0000","last_unstale":"2024-04-02T09:13:31.426750+0000","last_undegraded":"2024-04-02T09:13:31.426750+0000","last_fullsized":"2024-04-02T09:13:31.426750+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:30:33.534225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.429062+0000","last_change":"2024-04-02T09:13:40.429062+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.429062+0000","last_undegraded":"2024-04-02T09:13:40.429062+0000","last_fullsized":"2024-04-02T09:13:40.429062+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:39.602788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.423962+0000","last_change":"2024-04-02T09:13:28.395003+0000","last_active":"2024-04-02T09:13:31.423962+0000","last_peered":"2024-04-02T09:13:31.423962+0000","last_clean":"2024-04-02T09:13:31.423962+0000","last_became_active":"2024-04-02T09:13:28.394635+0000","last_became_peered":"2024-04-02T09:13:28.394635+0000","last_unstale":"2024-04-02T09:13:31.423962+0000","last_undegraded":"2024-04-02T09:13:31.423962+0000","last_fullsized":"2024-04-02T09:13:31.423962+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:54.758118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630389+0000","last_change":"2024-04-02T09:13:28.391302+0000","last_active":"2024-04-02T09:13:31.630389+0000","last_peered":"2024-04-02T09:13:31.630389+0000","last_clean":"2024-04-02T09:13:31.630389+0000","last_became_active":"2024-04-02T09:13:28.388976+0000","last_became_peered":"2024-04-02T09:13:28.388976+0000","last_unstale":"2024-04-02T09:13:31.630389+0000","last_undegraded":"2024-04-02T09:13:31.630389+0000","last_fullsized":"2024-04-02T09:13:31.630389+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:02.556232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"243'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422779+0000","last_change":"2024-04-02T09:13:27.881269+0000","last_active":"2024-04-02T09:13:31.422779+0000","last_peered":"2024-04-02T09:13:31.422779+0000","last_clean":"2024-04-02T09:13:31.422779+0000","last_became_active":"2024-04-02T09:13:27.880664+0000","last_became_peered":"2024-04-02T09:13:27.880664+0000","last_unstale":"2024-04-02T09:13:31.422779+0000","last_undegraded":"2024-04-02T09:13:31.422779+0000","last_fullsized":"2024-04-02T09:13:31.422779+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:53.750473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426876+0000","last_change":"2024-04-02T09:13:28.399529+0000","last_active":"2024-04-02T09:13:31.426876+0000","last_peered":"2024-04-02T09:13:31.426876+0000","last_clean":"2024-04-02T09:13:31.426876+0000","last_became_active":"2024-04-02T09:13:28.399042+0000","last_became_peered":"2024-04-02T09:13:28.399042+0000","last_unstale":"2024-04-02T09:13:31.426876+0000","last_undegraded":"2024-04-02T09:13:31.426876+0000","last_fullsized":"2024-04-02T09:13:31.426876+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:58:19.109584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1d","version":"243'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426670+0000","last_change":"2024-04-02T09:13:27.889299+0000","last_active":"2024-04-02T09:13:31.426670+0000","last_peered":"2024-04-02T09:13:31.426670+0000","last_clean":"2024-04-02T09:13:31.426670+0000","last_became_active":"2024-04-02T09:13:27.888249+0000","last_became_peered":"2024-04-02T09:13:27.888249+0000","last_unstale":"2024-04-02T09:13:31.426670+0000","last_undegraded":"2024-04-02T09:13:31.426670+0000","last_fullsized":"2024-04-02T09:13:31.426670+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:01:31.866958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424210+0000","last_change":"2024-04-02T09:13:28.382423+0000","last_active":"2024-04-02T09:13:31.424210+0000","last_peered":"2024-04-02T09:13:31.424210+0000","last_clean":"2024-04-02T09:13:31.424210+0000","last_became_active":"2024-04-02T09:13:28.381840+0000","last_became_peered":"2024-04-02T09:13:28.381840+0000","last_unstale":"2024-04-02T09:13:31.424210+0000","last_undegraded":"2024-04-02T09:13:31.424210+0000","last_fullsized":"2024-04-02T09:13:31.424210+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:04:44.765954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"243'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630265+0000","last_change":"2024-04-02T09:13:27.885629+0000","last_active":"2024-04-02T09:13:31.630265+0000","last_peered":"2024-04-02T09:13:31.630265+0000","last_clean":"2024-04-02T09:13:31.630265+0000","last_became_active":"2024-04-02T09:13:27.884921+0000","last_became_peered":"2024-04-02T09:13:27.884921+0000","last_unstale":"2024-04-02T09:13:31.630265+0000","last_undegraded":"2024-04-02T09:13:31.630265+0000","last_fullsized":"2024-04-02T09:13:31.630265+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:49:58.033026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630095+0000","last_change":"2024-04-02T09:13:28.381558+0000","last_active":"2024-04-02T09:13:31.630095+0000","last_peered":"2024-04-02T09:13:31.630095+0000","last_clean":"2024-04-02T09:13:31.630095+0000","last_became_active":"2024-04-02T09:13:28.381306+0000","last_became_peered":"2024-04-02T09:13:28.381306+0000","last_unstale":"2024-04-02T09:13:31.630095+0000","last_undegraded":"2024-04-02T09:13:31.630095+0000","last_fullsized":"2024-04-02T09:13:31.630095+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:12:17.313507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1b","version":"243'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630114+0000","last_change":"2024-04-02T09:13:27.884062+0000","last_active":"2024-04-02T09:13:31.630114+0000","last_peered":"2024-04-02T09:13:31.630114+0000","last_clean":"2024-04-02T09:13:31.630114+0000","last_became_active":"2024-04-02T09:13:27.883677+0000","last_became_peered":"2024-04-02T09:13:27.883677+0000","last_unstale":"2024-04-02T09:13:31.630114+0000","last_undegraded":"2024-04-02T09:13:31.630114+0000","last_fullsized":"2024-04-02T09:13:31.630114+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:14.774442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422658+0000","last_change":"2024-04-02T09:13:28.389833+0000","last_active":"2024-04-02T09:13:31.422658+0000","last_peered":"2024-04-02T09:13:31.422658+0000","last_clean":"2024-04-02T09:13:31.422658+0000","last_became_active":"2024-04-02T09:13:28.389495+0000","last_became_peered":"2024-04-02T09:13:28.389495+0000","last_unstale":"2024-04-02T09:13:31.422658+0000","last_undegraded":"2024-04-02T09:13:31.422658+0000","last_fullsized":"2024-04-02T09:13:31.422658+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:27.049697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1a","version":"243'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630023+0000","last_change":"2024-04-02T09:13:27.883532+0000","last_active":"2024-04-02T09:13:31.630023+0000","last_peered":"2024-04-02T09:13:31.630023+0000","last_clean":"2024-04-02T09:13:31.630023+0000","last_became_active":"2024-04-02T09:13:27.883252+0000","last_became_peered":"2024-04-02T09:13:27.883252+0000","last_unstale":"2024-04-02T09:13:31.630023+0000","last_undegraded":"2024-04-02T09:13:31.630023+0000","last_fullsized":"2024-04-02T09:13:31.630023+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.950371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426724+0000","last_change":"2024-04-02T09:13:28.403943+0000","last_active":"2024-04-02T09:13:31.426724+0000","last_peered":"2024-04-02T09:13:31.426724+0000","last_clean":"2024-04-02T09:13:31.426724+0000","last_became_active":"2024-04-02T09:13:28.402197+0000","last_became_peered":"2024-04-02T09:13:28.402197+0000","last_unstale":"2024-04-02T09:13:31.426724+0000","last_undegraded":"2024-04-02T09:13:31.426724+0000","last_fullsized":"2024-04-02T09:13:31.426724+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:50.579695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427538+0000","last_change":"2024-04-02T09:13:27.883508+0000","last_active":"2024-04-02T09:13:31.427538+0000","last_peered":"2024-04-02T09:13:31.427538+0000","last_clean":"2024-04-02T09:13:31.427538+0000","last_became_active":"2024-04-02T09:13:27.883263+0000","last_became_peered":"2024-04-02T09:13:27.883263+0000","last_unstale":"2024-04-02T09:13:31.427538+0000","last_undegraded":"2024-04-02T09:13:31.427538+0000","last_fullsized":"2024-04-02T09:13:31.427538+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:37.428725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426770+0000","last_change":"2024-04-02T09:13:28.403768+0000","last_active":"2024-04-02T09:13:31.426770+0000","last_peered":"2024-04-02T09:13:31.426770+0000","last_clean":"2024-04-02T09:13:31.426770+0000","last_became_active":"2024-04-02T09:13:28.399745+0000","last_became_peered":"2024-04-02T09:13:28.399745+0000","last_unstale":"2024-04-02T09:13:31.426770+0000","last_undegraded":"2024-04-02T09:13:31.426770+0000","last_fullsized":"2024-04-02T09:13:31.426770+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:39.541325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629961+0000","last_change":"2024-04-02T09:13:27.888598+0000","last_active":"2024-04-02T09:13:31.629961+0000","last_peered":"2024-04-02T09:13:31.629961+0000","last_clean":"2024-04-02T09:13:31.629961+0000","last_became_active":"2024-04-02T09:13:27.888115+0000","last_became_peered":"2024-04-02T09:13:27.888115+0000","last_unstale":"2024-04-02T09:13:31.629961+0000","last_undegraded":"2024-04-02T09:13:31.629961+0000","last_fullsized":"2024-04-02T09:13:31.629961+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:57:03.609779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426945+0000","last_change":"2024-04-02T09:13:28.404602+0000","last_active":"2024-04-02T09:13:31.426945+0000","last_peered":"2024-04-02T09:13:31.426945+0000","last_clean":"2024-04-02T09:13:31.426945+0000","last_became_active":"2024-04-02T09:13:28.403446+0000","last_became_peered":"2024-04-02T09:13:28.403446+0000","last_unstale":"2024-04-02T09:13:31.426945+0000","last_undegraded":"2024-04-02T09:13:31.426945+0000","last_fullsized":"2024-04-02T09:13:31.426945+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:24.279179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422798+0000","last_change":"2024-04-02T09:13:27.887286+0000","last_active":"2024-04-02T09:13:31.422798+0000","last_peered":"2024-04-02T09:13:31.422798+0000","last_clean":"2024-04-02T09:13:31.422798+0000","last_became_active":"2024-04-02T09:13:27.886999+0000","last_became_peered":"2024-04-02T09:13:27.886999+0000","last_unstale":"2024-04-02T09:13:31.422798+0000","last_undegraded":"2024-04-02T09:13:31.422798+0000","last_fullsized":"2024-04-02T09:13:31.422798+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:44.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630366+0000","last_change":"2024-04-02T09:13:28.378767+0000","last_active":"2024-04-02T09:13:31.630366+0000","last_peered":"2024-04-02T09:13:31.630366+0000","last_clean":"2024-04-02T09:13:31.630366+0000","last_became_active":"2024-04-02T09:13:28.378487+0000","last_became_peered":"2024-04-02T09:13:28.378487+0000","last_unstale":"2024-04-02T09:13:31.630366+0000","last_undegraded":"2024-04-02T09:13:31.630366+0000","last_fullsized":"2024-04-02T09:13:31.630366+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:26:47.539777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426956+0000","last_change":"2024-04-02T09:13:27.884469+0000","last_active":"2024-04-02T09:13:31.426956+0000","last_peered":"2024-04-02T09:13:31.426956+0000","last_clean":"2024-04-02T09:13:31.426956+0000","last_became_active":"2024-04-02T09:13:27.882570+0000","last_became_peered":"2024-04-02T09:13:27.882570+0000","last_unstale":"2024-04-02T09:13:31.426956+0000","last_undegraded":"2024-04-02T09:13:31.426956+0000","last_fullsized":"2024-04-02T09:13:31.426956+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:57:12.395680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424028+0000","last_change":"2024-04-02T09:13:28.387205+0000","last_active":"2024-04-02T09:13:31.424028+0000","last_peered":"2024-04-02T09:13:31.424028+0000","last_clean":"2024-04-02T09:13:31.424028+0000","last_became_active":"2024-04-02T09:13:28.384126+0000","last_became_peered":"2024-04-02T09:13:28.384126+0000","last_unstale":"2024-04-02T09:13:31.424028+0000","last_undegraded":"2024-04-02T09:13:31.424028+0000","last_fullsized":"2024-04-02T09:13:31.424028+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:52.330349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.15","version":"243'3","reported_seq":15,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424048+0000","last_change":"2024-04-02T09:13:27.883818+0000","last_active":"2024-04-02T09:13:31.424048+0000","last_peered":"2024-04-02T09:13:31.424048+0000","last_clean":"2024-04-02T09:13:31.424048+0000","last_became_active":"2024-04-02T09:13:27.883527+0000","last_became_peered":"2024-04-02T09:13:27.883527+0000","last_unstale":"2024-04-02T09:13:31.424048+0000","last_undegraded":"2024-04-02T09:13:31.424048+0000","last_fullsized":"2024-04-02T09:13:31.424048+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:49.572049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427603+0000","last_change":"2024-04-02T09:13:28.379346+0000","last_active":"2024-04-02T09:13:31.427603+0000","last_peered":"2024-04-02T09:13:31.427603+0000","last_clean":"2024-04-02T09:13:31.427603+0000","last_became_active":"2024-04-02T09:13:28.378741+0000","last_became_peered":"2024-04-02T09:13:28.378741+0000","last_unstale":"2024-04-02T09:13:31.427603+0000","last_undegraded":"2024-04-02T09:13:31.427603+0000","last_fullsized":"2024-04-02T09:13:31.427603+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:44.092529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.14","version":"244'8","reported_seq":20,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:38.348183+0000","last_change":"2024-04-02T09:13:27.885270+0000","last_active":"2024-04-02T09:13:38.348183+0000","last_peered":"2024-04-02T09:13:38.348183+0000","last_clean":"2024-04-02T09:13:38.348183+0000","last_became_active":"2024-04-02T09:13:27.884953+0000","last_became_peered":"2024-04-02T09:13:27.884953+0000","last_unstale":"2024-04-02T09:13:38.348183+0000","last_undegraded":"2024-04-02T09:13:38.348183+0000","last_fullsized":"2024-04-02T09:13:38.348183+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:48.103277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2805,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422639+0000","last_change":"2024-04-02T09:13:28.393735+0000","last_active":"2024-04-02T09:13:31.422639+0000","last_peered":"2024-04-02T09:13:31.422639+0000","last_clean":"2024-04-02T09:13:31.422639+0000","last_became_active":"2024-04-02T09:13:28.393387+0000","last_became_peered":"2024-04-02T09:13:28.393387+0000","last_unstale":"2024-04-02T09:13:31.422639+0000","last_undegraded":"2024-04-02T09:13:31.422639+0000","last_fullsized":"2024-04-02T09:13:31.422639+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:47.359176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.13","version":"243'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426591+0000","last_change":"2024-04-02T09:13:27.885596+0000","last_active":"2024-04-02T09:13:31.426591+0000","last_peered":"2024-04-02T09:13:31.426591+0000","last_clean":"2024-04-02T09:13:31.426591+0000","last_became_active":"2024-04-02T09:13:27.885310+0000","last_became_peered":"2024-04-02T09:13:27.885310+0000","last_unstale":"2024-04-02T09:13:31.426591+0000","last_undegraded":"2024-04-02T09:13:31.426591+0000","last_fullsized":"2024-04-02T09:13:31.426591+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:02.497118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424172+0000","last_change":"2024-04-02T09:13:28.378384+0000","last_active":"2024-04-02T09:13:31.424172+0000","last_peered":"2024-04-02T09:13:31.424172+0000","last_clean":"2024-04-02T09:13:31.424172+0000","last_became_active":"2024-04-02T09:13:28.377833+0000","last_became_peered":"2024-04-02T09:13:28.377833+0000","last_unstale":"2024-04-02T09:13:31.424172+0000","last_undegraded":"2024-04-02T09:13:31.424172+0000","last_fullsized":"2024-04-02T09:13:31.424172+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:37:54.526506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426808+0000","last_change":"2024-04-02T09:13:27.880069+0000","last_active":"2024-04-02T09:13:31.426808+0000","last_peered":"2024-04-02T09:13:31.426808+0000","last_clean":"2024-04-02T09:13:31.426808+0000","last_became_active":"2024-04-02T09:13:27.879793+0000","last_became_peered":"2024-04-02T09:13:27.879793+0000","last_unstale":"2024-04-02T09:13:31.426808+0000","last_undegraded":"2024-04-02T09:13:31.426808+0000","last_fullsized":"2024-04-02T09:13:31.426808+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:10.198371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629856+0000","last_change":"2024-04-02T09:13:28.394088+0000","last_active":"2024-04-02T09:13:31.629856+0000","last_peered":"2024-04-02T09:13:31.629856+0000","last_clean":"2024-04-02T09:13:31.629856+0000","last_became_active":"2024-04-02T09:13:28.392932+0000","last_became_peered":"2024-04-02T09:13:28.392932+0000","last_unstale":"2024-04-02T09:13:31.629856+0000","last_undegraded":"2024-04-02T09:13:31.629856+0000","last_fullsized":"2024-04-02T09:13:31.629856+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:54.683721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427550+0000","last_change":"2024-04-02T09:13:27.881714+0000","last_active":"2024-04-02T09:13:31.427550+0000","last_peered":"2024-04-02T09:13:31.427550+0000","last_clean":"2024-04-02T09:13:31.427550+0000","last_became_active":"2024-04-02T09:13:27.881428+0000","last_became_peered":"2024-04-02T09:13:27.881428+0000","last_unstale":"2024-04-02T09:13:31.427550+0000","last_undegraded":"2024-04-02T09:13:31.427550+0000","last_fullsized":"2024-04-02T09:13:31.427550+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:50.081688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422590+0000","last_change":"2024-04-02T09:13:28.391188+0000","last_active":"2024-04-02T09:13:31.422590+0000","last_peered":"2024-04-02T09:13:31.422590+0000","last_clean":"2024-04-02T09:13:31.422590+0000","last_became_active":"2024-04-02T09:13:28.390867+0000","last_became_peered":"2024-04-02T09:13:28.390867+0000","last_unstale":"2024-04-02T09:13:31.422590+0000","last_undegraded":"2024-04-02T09:13:31.422590+0000","last_fullsized":"2024-04-02T09:13:31.422590+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:26:58.531401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.10","version":"243'8","reported_seq":20,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.423981+0000","last_change":"2024-04-02T09:13:27.886503+0000","last_active":"2024-04-02T09:13:31.423981+0000","last_peered":"2024-04-02T09:13:31.423981+0000","last_clean":"2024-04-02T09:13:31.423981+0000","last_became_active":"2024-04-02T09:13:27.886121+0000","last_became_peered":"2024-04-02T09:13:27.886121+0000","last_unstale":"2024-04-02T09:13:31.423981+0000","last_undegraded":"2024-04-02T09:13:31.423981+0000","last_fullsized":"2024-04-02T09:13:31.423981+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:38.533680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630223+0000","last_change":"2024-04-02T09:13:28.376609+0000","last_active":"2024-04-02T09:13:31.630223+0000","last_peered":"2024-04-02T09:13:31.630223+0000","last_clean":"2024-04-02T09:13:31.630223+0000","last_became_active":"2024-04-02T09:13:28.376226+0000","last_became_peered":"2024-04-02T09:13:28.376226+0000","last_unstale":"2024-04-02T09:13:31.630223+0000","last_undegraded":"2024-04-02T09:13:31.630223+0000","last_fullsized":"2024-04-02T09:13:31.630223+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:59.766007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"243'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422817+0000","last_change":"2024-04-02T09:13:27.883080+0000","last_active":"2024-04-02T09:13:31.422817+0000","last_peered":"2024-04-02T09:13:31.422817+0000","last_clean":"2024-04-02T09:13:31.422817+0000","last_became_active":"2024-04-02T09:13:27.882612+0000","last_became_peered":"2024-04-02T09:13:27.882612+0000","last_unstale":"2024-04-02T09:13:31.422817+0000","last_undegraded":"2024-04-02T09:13:31.422817+0000","last_fullsized":"2024-04-02T09:13:31.422817+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:20.979570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424093+0000","last_change":"2024-04-02T09:13:28.381499+0000","last_active":"2024-04-02T09:13:31.424093+0000","last_peered":"2024-04-02T09:13:31.424093+0000","last_clean":"2024-04-02T09:13:31.424093+0000","last_became_active":"2024-04-02T09:13:28.381286+0000","last_became_peered":"2024-04-02T09:13:28.381286+0000","last_unstale":"2024-04-02T09:13:31.424093+0000","last_undegraded":"2024-04-02T09:13:31.424093+0000","last_fullsized":"2024-04-02T09:13:31.424093+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:18.748067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.e","version":"243'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426966+0000","last_change":"2024-04-02T09:13:27.888558+0000","last_active":"2024-04-02T09:13:31.426966+0000","last_peered":"2024-04-02T09:13:31.426966+0000","last_clean":"2024-04-02T09:13:31.426966+0000","last_became_active":"2024-04-02T09:13:27.888288+0000","last_became_peered":"2024-04-02T09:13:27.888288+0000","last_unstale":"2024-04-02T09:13:31.426966+0000","last_undegraded":"2024-04-02T09:13:31.426966+0000","last_fullsized":"2024-04-02T09:13:31.426966+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:15.386238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422721+0000","last_change":"2024-04-02T09:13:28.372849+0000","last_active":"2024-04-02T09:13:31.422721+0000","last_peered":"2024-04-02T09:13:31.422721+0000","last_clean":"2024-04-02T09:13:31.422721+0000","last_became_active":"2024-04-02T09:13:28.372595+0000","last_became_peered":"2024-04-02T09:13:28.372595+0000","last_unstale":"2024-04-02T09:13:31.422721+0000","last_undegraded":"2024-04-02T09:13:31.422721+0000","last_fullsized":"2024-04-02T09:13:31.422721+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:12.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.d","version":"243'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422739+0000","last_change":"2024-04-02T09:13:27.879417+0000","last_active":"2024-04-02T09:13:31.422739+0000","last_peered":"2024-04-02T09:13:31.422739+0000","last_clean":"2024-04-02T09:13:31.422739+0000","last_became_active":"2024-04-02T09:13:27.879165+0000","last_became_peered":"2024-04-02T09:13:27.879165+0000","last_unstale":"2024-04-02T09:13:31.422739+0000","last_undegraded":"2024-04-02T09:13:31.422739+0000","last_fullsized":"2024-04-02T09:13:31.422739+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:47:32.233662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629634+0000","last_change":"2024-04-02T09:13:28.392130+0000","last_active":"2024-04-02T09:13:31.629634+0000","last_peered":"2024-04-02T09:13:31.629634+0000","last_clean":"2024-04-02T09:13:31.629634+0000","last_became_active":"2024-04-02T09:13:28.391690+0000","last_became_peered":"2024-04-02T09:13:28.391690+0000","last_unstale":"2024-04-02T09:13:31.629634+0000","last_undegraded":"2024-04-02T09:13:31.629634+0000","last_fullsized":"2024-04-02T09:13:31.629634+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:24.319747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426897+0000","last_change":"2024-04-02T09:13:27.884781+0000","last_active":"2024-04-02T09:13:31.426897+0000","last_peered":"2024-04-02T09:13:31.426897+0000","last_clean":"2024-04-02T09:13:31.426897+0000","last_became_active":"2024-04-02T09:13:27.884507+0000","last_became_peered":"2024-04-02T09:13:27.884507+0000","last_unstale":"2024-04-02T09:13:31.426897+0000","last_undegraded":"2024-04-02T09:13:31.426897+0000","last_fullsized":"2024-04-02T09:13:31.426897+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:31:18.889273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424087+0000","last_change":"2024-04-02T09:13:28.381580+0000","last_active":"2024-04-02T09:13:31.424087+0000","last_peered":"2024-04-02T09:13:31.424087+0000","last_clean":"2024-04-02T09:13:31.424087+0000","last_became_active":"2024-04-02T09:13:28.381300+0000","last_became_peered":"2024-04-02T09:13:28.381300+0000","last_unstale":"2024-04-02T09:13:31.424087+0000","last_undegraded":"2024-04-02T09:13:31.424087+0000","last_fullsized":"2024-04-02T09:13:31.424087+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:51.089333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"243'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426854+0000","last_change":"2024-04-02T09:13:27.875291+0000","last_active":"2024-04-02T09:13:31.426854+0000","last_peered":"2024-04-02T09:13:31.426854+0000","last_clean":"2024-04-02T09:13:31.426854+0000","last_became_active":"2024-04-02T09:13:27.875003+0000","last_became_peered":"2024-04-02T09:13:27.875003+0000","last_unstale":"2024-04-02T09:13:31.426854+0000","last_undegraded":"2024-04-02T09:13:31.426854+0000","last_fullsized":"2024-04-02T09:13:31.426854+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:00.637981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426839+0000","last_change":"2024-04-02T09:13:28.404265+0000","last_active":"2024-04-02T09:13:31.426839+0000","last_peered":"2024-04-02T09:13:31.426839+0000","last_clean":"2024-04-02T09:13:31.426839+0000","last_became_active":"2024-04-02T09:13:28.403894+0000","last_became_peered":"2024-04-02T09:13:28.403894+0000","last_unstale":"2024-04-02T09:13:31.426839+0000","last_undegraded":"2024-04-02T09:13:31.426839+0000","last_fullsized":"2024-04-02T09:13:31.426839+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:05:23.433426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426819+0000","last_change":"2024-04-02T09:13:27.881604+0000","last_active":"2024-04-02T09:13:31.426819+0000","last_peered":"2024-04-02T09:13:31.426819+0000","last_clean":"2024-04-02T09:13:31.426819+0000","last_became_active":"2024-04-02T09:13:27.881304+0000","last_became_peered":"2024-04-02T09:13:27.881304+0000","last_unstale":"2024-04-02T09:13:31.426819+0000","last_undegraded":"2024-04-02T09:13:31.426819+0000","last_fullsized":"2024-04-02T09:13:31.426819+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:02.497118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629870+0000","last_change":"2024-04-02T09:13:28.374960+0000","last_active":"2024-04-02T09:13:31.629870+0000","last_peered":"2024-04-02T09:13:31.629870+0000","last_clean":"2024-04-02T09:13:31.629870+0000","last_became_active":"2024-04-02T09:13:28.373683+0000","last_became_peered":"2024-04-02T09:13:28.373683+0000","last_unstale":"2024-04-02T09:13:31.629870+0000","last_undegraded":"2024-04-02T09:13:31.629870+0000","last_fullsized":"2024-04-02T09:13:31.629870+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:15.851301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629524+0000","last_change":"2024-04-02T09:13:27.877837+0000","last_active":"2024-04-02T09:13:31.629524+0000","last_peered":"2024-04-02T09:13:31.629524+0000","last_clean":"2024-04-02T09:13:31.629524+0000","last_became_active":"2024-04-02T09:13:27.876792+0000","last_became_peered":"2024-04-02T09:13:27.876792+0000","last_unstale":"2024-04-02T09:13:31.629524+0000","last_undegraded":"2024-04-02T09:13:31.629524+0000","last_fullsized":"2024-04-02T09:13:31.629524+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:17.740422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427528+0000","last_change":"2024-04-02T09:13:28.379243+0000","last_active":"2024-04-02T09:13:31.427528+0000","last_peered":"2024-04-02T09:13:31.427528+0000","last_clean":"2024-04-02T09:13:31.427528+0000","last_became_active":"2024-04-02T09:13:28.378297+0000","last_became_peered":"2024-04-02T09:13:28.378297+0000","last_unstale":"2024-04-02T09:13:31.427528+0000","last_undegraded":"2024-04-02T09:13:31.427528+0000","last_fullsized":"2024-04-02T09:13:31.427528+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:02.761190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629474+0000","last_change":"2024-04-02T09:13:27.876051+0000","last_active":"2024-04-02T09:13:31.629474+0000","last_peered":"2024-04-02T09:13:31.629474+0000","last_clean":"2024-04-02T09:13:31.629474+0000","last_became_active":"2024-04-02T09:13:27.875628+0000","last_became_peered":"2024-04-02T09:13:27.875628+0000","last_unstale":"2024-04-02T09:13:31.629474+0000","last_undegraded":"2024-04-02T09:13:31.629474+0000","last_fullsized":"2024-04-02T09:13:31.629474+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:53.750473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422758+0000","last_change":"2024-04-02T09:13:28.383204+0000","last_active":"2024-04-02T09:13:31.422758+0000","last_peered":"2024-04-02T09:13:31.422758+0000","last_clean":"2024-04-02T09:13:31.422758+0000","last_became_active":"2024-04-02T09:13:28.382961+0000","last_became_peered":"2024-04-02T09:13:28.382961+0000","last_unstale":"2024-04-02T09:13:31.422758+0000","last_undegraded":"2024-04-02T09:13:31.422758+0000","last_fullsized":"2024-04-02T09:13:31.422758+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:16:14.265813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.7","version":"243'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426992+0000","last_change":"2024-04-02T09:13:27.886740+0000","last_active":"2024-04-02T09:13:31.426992+0000","last_peered":"2024-04-02T09:13:31.426992+0000","last_clean":"2024-04-02T09:13:31.426992+0000","last_became_active":"2024-04-02T09:13:27.886385+0000","last_became_peered":"2024-04-02T09:13:27.886385+0000","last_unstale":"2024-04-02T09:13:31.426992+0000","last_undegraded":"2024-04-02T09:13:31.426992+0000","last_fullsized":"2024-04-02T09:13:31.426992+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:20.383235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426708+0000","last_change":"2024-04-02T09:13:28.402497+0000","last_active":"2024-04-02T09:13:31.426708+0000","last_peered":"2024-04-02T09:13:31.426708+0000","last_clean":"2024-04-02T09:13:31.426708+0000","last_became_active":"2024-04-02T09:13:28.402235+0000","last_became_peered":"2024-04-02T09:13:28.402235+0000","last_unstale":"2024-04-02T09:13:31.426708+0000","last_undegraded":"2024-04-02T09:13:31.426708+0000","last_fullsized":"2024-04-02T09:13:31.426708+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:15.072346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.6","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:33.349612+0000","last_change":"2024-04-02T09:13:27.884225+0000","last_active":"2024-04-02T09:13:33.349612+0000","last_peered":"2024-04-02T09:13:33.349612+0000","last_clean":"2024-04-02T09:13:33.349612+0000","last_became_active":"2024-04-02T09:13:27.883536+0000","last_became_peered":"2024-04-02T09:13:27.883536+0000","last_unstale":"2024-04-02T09:13:33.349612+0000","last_undegraded":"2024-04-02T09:13:33.349612+0000","last_fullsized":"2024-04-02T09:13:33.349612+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:50.645932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630206+0000","last_change":"2024-04-02T09:13:28.387139+0000","last_active":"2024-04-02T09:13:31.630206+0000","last_peered":"2024-04-02T09:13:31.630206+0000","last_clean":"2024-04-02T09:13:31.630206+0000","last_became_active":"2024-04-02T09:13:28.386878+0000","last_became_peered":"2024-04-02T09:13:28.386878+0000","last_unstale":"2024-04-02T09:13:31.630206+0000","last_undegraded":"2024-04-02T09:13:31.630206+0000","last_fullsized":"2024-04-02T09:13:31.630206+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:56:34.681485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427622+0000","last_change":"2024-04-02T09:13:27.882615+0000","last_active":"2024-04-02T09:13:31.427622+0000","last_peered":"2024-04-02T09:13:31.427622+0000","last_clean":"2024-04-02T09:13:31.427622+0000","last_became_active":"2024-04-02T09:13:27.882334+0000","last_became_peered":"2024-04-02T09:13:27.882334+0000","last_unstale":"2024-04-02T09:13:31.427622+0000","last_undegraded":"2024-04-02T09:13:31.427622+0000","last_fullsized":"2024-04-02T09:13:31.427622+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:43.215347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426932+0000","last_change":"2024-04-02T09:13:28.402303+0000","last_active":"2024-04-02T09:13:31.426932+0000","last_peered":"2024-04-02T09:13:31.426932+0000","last_clean":"2024-04-02T09:13:31.426932+0000","last_became_active":"2024-04-02T09:13:28.399305+0000","last_became_peered":"2024-04-02T09:13:28.399305+0000","last_unstale":"2024-04-02T09:13:31.426932+0000","last_undegraded":"2024-04-02T09:13:31.426932+0000","last_fullsized":"2024-04-02T09:13:31.426932+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:58.927786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426908+0000","last_change":"2024-04-02T09:13:27.881268+0000","last_active":"2024-04-02T09:13:31.426908+0000","last_peered":"2024-04-02T09:13:31.426908+0000","last_clean":"2024-04-02T09:13:31.426908+0000","last_became_active":"2024-04-02T09:13:27.880983+0000","last_became_peered":"2024-04-02T09:13:27.880983+0000","last_unstale":"2024-04-02T09:13:31.426908+0000","last_undegraded":"2024-04-02T09:13:31.426908+0000","last_fullsized":"2024-04-02T09:13:31.426908+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:09.803095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424125+0000","last_change":"2024-04-02T09:13:28.374285+0000","last_active":"2024-04-02T09:13:31.424125+0000","last_peered":"2024-04-02T09:13:31.424125+0000","last_clean":"2024-04-02T09:13:31.424125+0000","last_became_active":"2024-04-02T09:13:28.373980+0000","last_became_peered":"2024-04-02T09:13:28.373980+0000","last_unstale":"2024-04-02T09:13:31.424125+0000","last_undegraded":"2024-04-02T09:13:31.424125+0000","last_fullsized":"2024-04-02T09:13:31.424125+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:56:34.681485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422700+0000","last_change":"2024-04-02T09:13:27.882249+0000","last_active":"2024-04-02T09:13:31.422700+0000","last_peered":"2024-04-02T09:13:31.422700+0000","last_clean":"2024-04-02T09:13:31.422700+0000","last_became_active":"2024-04-02T09:13:27.881833+0000","last_became_peered":"2024-04-02T09:13:27.881833+0000","last_unstale":"2024-04-02T09:13:31.422700+0000","last_undegraded":"2024-04-02T09:13:31.422700+0000","last_fullsized":"2024-04-02T09:13:31.422700+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:22.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427581+0000","last_change":"2024-04-02T09:13:28.389208+0000","last_active":"2024-04-02T09:13:31.427581+0000","last_peered":"2024-04-02T09:13:31.427581+0000","last_clean":"2024-04-02T09:13:31.427581+0000","last_became_active":"2024-04-02T09:13:28.388923+0000","last_became_peered":"2024-04-02T09:13:28.388923+0000","last_unstale":"2024-04-02T09:13:31.427581+0000","last_undegraded":"2024-04-02T09:13:31.427581+0000","last_fullsized":"2024-04-02T09:13:31.427581+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:35:55.299814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630151+0000","last_change":"2024-04-02T09:13:27.887509+0000","last_active":"2024-04-02T09:13:31.630151+0000","last_peered":"2024-04-02T09:13:31.630151+0000","last_clean":"2024-04-02T09:13:31.630151+0000","last_became_active":"2024-04-02T09:13:27.887260+0000","last_became_peered":"2024-04-02T09:13:27.887260+0000","last_unstale":"2024-04-02T09:13:31.630151+0000","last_undegraded":"2024-04-02T09:13:31.630151+0000","last_fullsized":"2024-04-02T09:13:31.630151+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:28.581007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629486+0000","last_change":"2024-04-02T09:13:28.374292+0000","last_active":"2024-04-02T09:13:31.629486+0000","last_peered":"2024-04-02T09:13:31.629486+0000","last_clean":"2024-04-02T09:13:31.629486+0000","last_became_active":"2024-04-02T09:13:28.373689+0000","last_became_peered":"2024-04-02T09:13:28.373689+0000","last_unstale":"2024-04-02T09:13:31.629486+0000","last_undegraded":"2024-04-02T09:13:31.629486+0000","last_fullsized":"2024-04-02T09:13:31.629486+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:08.953388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426541+0000","last_change":"2024-04-02T09:13:27.882492+0000","last_active":"2024-04-02T09:13:31.426541+0000","last_peered":"2024-04-02T09:13:31.426541+0000","last_clean":"2024-04-02T09:13:31.426541+0000","last_became_active":"2024-04-02T09:13:27.882217+0000","last_became_peered":"2024-04-02T09:13:27.882217+0000","last_unstale":"2024-04-02T09:13:31.426541+0000","last_undegraded":"2024-04-02T09:13:31.426541+0000","last_fullsized":"2024-04-02T09:13:31.426541+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:22:56.505431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426510+0000","last_change":"2024-04-02T09:13:28.381355+0000","last_active":"2024-04-02T09:13:31.426510+0000","last_peered":"2024-04-02T09:13:31.426510+0000","last_clean":"2024-04-02T09:13:31.426510+0000","last_became_active":"2024-04-02T09:13:28.381068+0000","last_became_peered":"2024-04-02T09:13:28.381068+0000","last_unstale":"2024-04-02T09:13:31.426510+0000","last_undegraded":"2024-04-02T09:13:31.426510+0000","last_fullsized":"2024-04-02T09:13:31.426510+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:50.895638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424004+0000","last_change":"2024-04-02T09:13:27.883425+0000","last_active":"2024-04-02T09:13:31.424004+0000","last_peered":"2024-04-02T09:13:31.424004+0000","last_clean":"2024-04-02T09:13:31.424004+0000","last_became_active":"2024-04-02T09:13:27.883106+0000","last_became_peered":"2024-04-02T09:13:27.883106+0000","last_unstale":"2024-04-02T09:13:31.424004+0000","last_undegraded":"2024-04-02T09:13:31.424004+0000","last_fullsized":"2024-04-02T09:13:31.424004+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:23.271534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629742+0000","last_change":"2024-04-02T09:13:28.402819+0000","last_active":"2024-04-02T09:13:31.629742+0000","last_peered":"2024-04-02T09:13:31.629742+0000","last_clean":"2024-04-02T09:13:31.629742+0000","last_became_active":"2024-04-02T09:13:28.402563+0000","last_became_peered":"2024-04-02T09:13:28.402563+0000","last_unstale":"2024-04-02T09:13:31.629742+0000","last_undegraded":"2024-04-02T09:13:31.629742+0000","last_fullsized":"2024-04-02T09:13:31.629742+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:27.049697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1f","version":"243'4","reported_seq":16,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426728+0000","last_change":"2024-04-02T09:13:27.883417+0000","last_active":"2024-04-02T09:13:31.426728+0000","last_peered":"2024-04-02T09:13:31.426728+0000","last_clean":"2024-04-02T09:13:31.426728+0000","last_became_active":"2024-04-02T09:13:27.882944+0000","last_became_peered":"2024-04-02T09:13:27.882944+0000","last_unstale":"2024-04-02T09:13:31.426728+0000","last_undegraded":"2024-04-02T09:13:31.426728+0000","last_fullsized":"2024-04-02T09:13:31.426728+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:05.331734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"229'370","reported_seq":1309,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.030879+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:13:31.030879+0000","last_peered":"2024-04-02T09:13:31.030879+0000","last_clean":"2024-04-02T09:13:31.030879+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:13:31.030879+0000","last_undegraded":"2024-04-02T09:13:31.030879+0000","last_fullsized":"2024-04-02T09:13:31.030879+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3603,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576000,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60388,"kb_used_data":3032,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311452,"statfs":{"total":96636764160,"available":96574926848,"internally_reserved":0,"allocated":3104768,"data_stored":1580386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.40400000000000003,"5min":0.309,"15min":0.309},"max":{"1min":8.016,"5min":8.016,"15min":8.016},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.71999999999999997,"15min":0.72999999999999998},"min":{"1min":0.39900000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.1790000000000003,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.81999999999999995}]},{"osd":1,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.76800000000000002,"15min":0.751},"min":{"1min":0.44800000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.5819999999999999,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.70899999999999996,"15min":0.71999999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.42399999999999999},"max":{"1min":4.2569999999999997,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.48999999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.79600000000000004,"15min":0.77800000000000002},"min":{"1min":0.437,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.9790000000000001,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.80700000000000005,"15min":0.81200000000000006},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":4.8819999999999997,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.66900000000000004}]},{"osd":3,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.77100000000000002,"15min":0.75900000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.0530000000000008,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.999,"5min":0.79900000000000004,"15min":0.77700000000000002},"min":{"1min":0.54100000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":5.093,"5min":5.093,"15min":5.093},"last":0.78300000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.66700000000000004,"15min":0.628},"min":{"1min":0.38300000000000001,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":5.4589999999999996,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.1060000000000001,"5min":0.79700000000000004,"15min":0.74399999999999999},"min":{"1min":0.50600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.9409999999999998,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.79900000000000004}]},{"osd":5,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.73799999999999999,"15min":0.69299999999999995},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":5.1340000000000003,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.748},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.77400000000000002,"15min":0.73599999999999999},"min":{"1min":0.58899999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.492,"5min":3.492,"15min":3.492},"last":0.76600000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.72199999999999998,"15min":0.69699999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45800000000000002},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.075,"5min":0.78500000000000003,"15min":0.71699999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41099999999999998},"max":{"1min":7.8899999999999997,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.84899999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575998,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42324,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329516,"statfs":{"total":96636764160,"available":96593424384,"internally_reserved":0,"allocated":3088384,"data_stored":1577055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.69799999999999995,"15min":0.72899999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.90600000000000003,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.71099999999999997,"15min":0.73899999999999999},"min":{"1min":0.53600000000000003,"5min":0.47899999999999998,"15min":0.45800000000000002},"max":{"1min":0.88300000000000001,"5min":0.93200000000000005,"15min":1.0600000000000001},"last":0.70799999999999996}]},{"osd":1,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68799999999999994,"15min":0.71299999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86299999999999999,"5min":1.044,"15min":1.212},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.71299999999999997,"15min":0.752},"min":{"1min":0.45800000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.018,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.52100000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.71799999999999997,"15min":0.747},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.046,"5min":1.046,"15min":1.2789999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.71999999999999997,"15min":0.73899999999999999},"min":{"1min":0.44,"5min":0.439,"15min":0.439},"max":{"1min":0.93500000000000005,"5min":0.95999999999999996,"15min":1.226},"last":0.48899999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72399999999999998,"15min":0.76300000000000001},"min":{"1min":0.53900000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97799999999999998,"5min":1.141,"15min":1.2549999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.755,"15min":0.77000000000000002},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.67500000000000004}]},{"osd":4,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.60999999999999999,"15min":0.624},"min":{"1min":0.442,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.91200000000000003,"5min":0.96699999999999997,"15min":1.1259999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.41999999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.1879999999999999},"last":0.77200000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84199999999999997,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69999999999999996,"15min":0.72499999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":0.96399999999999997,"5min":1.034,"15min":1.204},"last":0.72499999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.72999999999999998,"15min":0.74099999999999999},"min":{"1min":0.53600000000000003,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.012,"5min":1.21,"15min":1.2370000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72599999999999998,"15min":0.73899999999999999},"min":{"1min":0.52400000000000002,"5min":0.51400000000000001,"15min":0.49399999999999999},"max":{"1min":0.91900000000000004,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.68799999999999994}]}]},{"osd":1,"up_from":13,"seq":55834575999,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52760,"kb_used_data":3596,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319080,"statfs":{"total":96636764160,"available":96582737920,"internally_reserved":0,"allocated":3682304,"data_stored":2167915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.433,"15min":0.45000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.58999999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48999999999999999,"15min":0.50900000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73199999999999998,"5min":0.76000000000000001,"15min":0.79800000000000004},"last":0.51400000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.504,"15min":0.505},"min":{"1min":0.38900000000000001,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56699999999999995,"15min":0.59199999999999997},"min":{"1min":0.44500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68100000000000005,"5min":0.77500000000000002,"15min":0.85599999999999998},"last":0.59499999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.41099999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.69999999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.55200000000000005,"15min":0.56100000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.58899999999999997,"5min":0.86699999999999999,"15min":0.878},"last":0.56399999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76900000000000002,"15min":0.78400000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.433},"max":{"1min":0.90700000000000003,"5min":1.1120000000000001,"15min":1.1479999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.76400000000000001,"15min":0.78500000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.41799999999999998},"max":{"1min":0.95699999999999996,"5min":1.05,"15min":1.25},"last":0.58099999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76100000000000001,"15min":0.78800000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.96399999999999997,"5min":1.115,"15min":1.321},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.79700000000000004,"15min":0.80700000000000005},"min":{"1min":0.56899999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93799999999999994,"5min":1.0980000000000001,"15min":1.173},"last":0.63800000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.79100000000000004,"15min":0.81299999999999994},"min":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.48099999999999998},"max":{"1min":0.96099999999999997,"5min":1.159,"15min":1.2210000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80800000000000005,"15min":0.80900000000000005},"min":{"1min":0.56799999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0009999999999999,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.69099999999999995}]},{"osd":7,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.79900000000000004,"15min":0.81999999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0029999999999999,"5min":1.1080000000000001,"15min":1.236},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81899999999999995,"15min":0.83399999999999996},"min":{"1min":0.60899999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":0.97499999999999998,"5min":1.0720000000000001,"15min":1.2789999999999999},"last":0.68600000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575999,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52180,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319660,"statfs":{"total":96636764160,"available":96583331840,"internally_reserved":0,"allocated":3088384,"data_stored":1576038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.441,"15min":0.45500000000000002},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.66700000000000004,"5min":0.69899999999999995,"15min":0.86699999999999999},"last":0.498},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.49399999999999999,"15min":0.50700000000000001},"min":{"1min":0.33900000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64800000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.53100000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.502,"15min":0.51100000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.82699999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53400000000000003,"15min":0.55500000000000005},"min":{"1min":0.38300000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72699999999999998,"5min":0.999,"15min":0.999},"last":0.63300000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54200000000000004,"15min":0.56699999999999995},"min":{"1min":0.35699999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.71899999999999997,"5min":0.79800000000000004,"15min":0.85699999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.61599999999999999,"15min":0.63800000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.73599999999999999,"5min":0.92000000000000004,"15min":0.96399999999999997},"last":0.65500000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.80500000000000005,"15min":0.83099999999999996},"min":{"1min":0.55000000000000004,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":0.97499999999999998,"5min":1.1950000000000001,"15min":1.2769999999999999},"last":0.745},{"interface":"front","average":{"1min":0.751,"5min":0.78500000000000003,"15min":0.82199999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99299999999999999,"5min":1.169,"15min":1.169},"last":0.873}]},{"osd":5,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.77900000000000003,"15min":0.82399999999999995},"min":{"1min":0.54000000000000004,"5min":0.497,"15min":0.497},"max":{"1min":1,"5min":1.145,"15min":1.232},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.96599999999999997,"5min":1.21,"15min":1.21},"last":0.90600000000000003}]},{"osd":6,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.77600000000000002,"15min":0.79800000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91100000000000003,"5min":1.1830000000000001,"15min":1.196},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.79200000000000004,"15min":0.81200000000000006},"min":{"1min":0.56499999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96199999999999997,"5min":1.246,"15min":1.246},"last":0.96799999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82699999999999996,"15min":0.85499999999999998},"min":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0069999999999999,"5min":1.292,"15min":1.292},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.84799999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.95499999999999996,"5min":1.1719999999999999,"15min":1.2589999999999999},"last":0.92300000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575998,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52180,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319660,"statfs":{"total":96636764160,"available":96583331840,"internally_reserved":0,"allocated":3088384,"data_stored":1577035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.44600000000000001,"15min":0.44},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.72999999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.48399999999999999,"15min":0.498},"min":{"1min":0.29199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":1.4930000000000001},"last":0.34000000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.504},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69699999999999995,"5min":0.75800000000000001,"15min":0.80600000000000005},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53600000000000003,"15min":0.55400000000000005},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.59599999999999997,"5min":0.75,"15min":1.4530000000000001},"last":0.41799999999999998}]},{"osd":3,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52900000000000003,"15min":0.55700000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.745,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60499999999999998,"15min":0.63200000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.64000000000000001,"5min":0.83699999999999997,"15min":1.0740000000000001},"last":0.51400000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.82099999999999995,"5min":1.121,"15min":1.121},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.77300000000000002,"15min":0.79100000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.96799999999999997,"5min":1.179,"15min":1.5329999999999999},"last":0.874}]},{"osd":5,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.78400000000000003,"15min":0.78600000000000003},"min":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.82999999999999996,"5min":1.99,"15min":1.99},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.78500000000000003,"15min":0.79900000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.89600000000000002,"5min":1.101,"15min":1.1259999999999999},"last":0.79100000000000004}]},{"osd":6,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.997,"5min":1.173,"15min":1.5940000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":0.80400000000000005},"min":{"1min":0.56200000000000006,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.92300000000000004,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.89800000000000002}]},{"osd":7,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.877,"5min":1.2110000000000001,"15min":1.5649999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.752,"5min":0.80700000000000005,"15min":0.82699999999999996},"min":{"1min":0.61499999999999999,"5min":0.497,"15min":0.497},"max":{"1min":0.95899999999999996,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.749}]}]},{"osd":3,"up_from":13,"seq":55834575998,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52188,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319652,"statfs":{"total":96636764160,"available":96583323648,"internally_reserved":0,"allocated":3096576,"data_stored":1579770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.42799999999999999,"15min":0.44600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57199999999999995,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.46899999999999997,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60199999999999998,"5min":0.71199999999999997,"15min":0.876},"last":0.52100000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48699999999999999,"15min":0.499},"min":{"1min":0.36799999999999999,"5min":0.33200000000000002,"15min":0.32500000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.90300000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55100000000000005,"15min":0.57199999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67700000000000005,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.55300000000000005}]},{"osd":2,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.60099999999999998,"15min":0.626},"min":{"1min":0.40100000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71399999999999997,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.628},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.58799999999999997,"15min":0.61599999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.69499999999999995,"5min":0.81699999999999995,"15min":1.0840000000000001},"last":0.60399999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.72299999999999998,"15min":0.752},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89100000000000001,"5min":1.071,"15min":1.0800000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.74299999999999999,"15min":0.76800000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.879,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.74399999999999999,"15min":0.77500000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91800000000000004,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.76300000000000001,"15min":0.78100000000000003},"min":{"1min":0.54200000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.85699999999999998,"5min":1.266,"15min":1.266},"last":0.59499999999999997}]},{"osd":6,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.76500000000000001,"15min":0.78400000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97099999999999997,"5min":1.284,"15min":1.284},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.54800000000000004,"5min":0.495,"15min":0.495},"max":{"1min":0.95999999999999996,"5min":1.2,"15min":1.2},"last":0.67500000000000004}]},{"osd":7,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.77900000000000003,"15min":0.80800000000000005},"min":{"1min":0.58699999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.93500000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.79300000000000004,"15min":0.81799999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.014,"5min":1.248,"15min":1.248},"last":0.72899999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575998,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48084,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323756,"statfs":{"total":96636764160,"available":96587526144,"internally_reserved":0,"allocated":3088384,"data_stored":1576987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.66400000000000003,"15min":0.68700000000000006},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":0.80000000000000004,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.432},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.65100000000000002,"15min":0.67700000000000005},"min":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.78100000000000003,"5min":0.86199999999999999,"15min":1.075},"last":0.66100000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.72199999999999998,"15min":0.754},"min":{"1min":0.49399999999999999,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":0.90500000000000003,"5min":1.0549999999999999,"15min":1.069},"last":0.63},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.73599999999999999,"15min":0.76500000000000001},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":0.92500000000000004,"5min":1.048,"15min":1.048},"last":0.49099999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.67300000000000004,"15min":0.70799999999999996},"min":{"1min":0.47099999999999997,"5min":0.438,"15min":0.438},"max":{"1min":0.81299999999999994,"5min":1.077,"15min":1.077},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.68999999999999995,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85599999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.58899999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.51500000000000001,"5min":0.501,"15min":0.47399999999999998},"max":{"1min":0.94599999999999995,"5min":0.998,"15min":1.018},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.028},"last":0.60899999999999999}]},{"osd":5,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.81899999999999995,"5min":1.008,"15min":1.008},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.50600000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65500000000000003,"15min":0.64400000000000002},"min":{"1min":0.48499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85099999999999998,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68999999999999995,"15min":0.69199999999999995},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":1.0049999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.623}]},{"osd":7,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.48399999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96099999999999997,"5min":1.002,"15min":1.0329999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.77500000000000002,"15min":0.78100000000000003},"min":{"1min":0.56100000000000005,"5min":0.5,"15min":0.5},"max":{"1min":1.038,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.70599999999999996}]}]},{"osd":5,"up_from":13,"seq":55834576001,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52760,"kb_used_data":3596,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319080,"statfs":{"total":96636764160,"available":96582737920,"internally_reserved":0,"allocated":3682304,"data_stored":2166438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69699999999999995,"15min":0.70999999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.89600000000000002,"5min":0.98699999999999999,"15min":1.004},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69899999999999995,"15min":0.72499999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.625}]},{"osd":1,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.70099999999999996,"15min":0.72199999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85999999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.71399999999999997,"15min":0.73199999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.052},"last":0.88500000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73199999999999998,"15min":0.745},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93200000000000005,"5min":1.004,"15min":1.014},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72599999999999998,"15min":0.746},"min":{"1min":0.46600000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.018,"5min":1.018,"15min":1.0680000000000001},"last":0.69499999999999995}]},{"osd":3,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77000000000000002,"15min":0.78600000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":0.98899999999999999,"15min":1.083},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.59099999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94399999999999995,"5min":1.008,"15min":1.1020000000000001},"last":0.81899999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.629,"15min":0.629},"min":{"1min":0.56499999999999995,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.89100000000000001,"5min":0.96899999999999997,"15min":1.054},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.68400000000000005,"15min":0.67500000000000004},"min":{"1min":0.47999999999999998,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.68300000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.434,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.751},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71199999999999997,"15min":0.71099999999999997},"min":{"1min":0.501,"5min":0.44600000000000001,"15min":0.42699999999999999},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.94899999999999995}]},{"osd":7,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.73999999999999999,"15min":0.73499999999999999},"min":{"1min":0.60699999999999998,"5min":0.441,"15min":0.441},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.64000000000000001,"5min":0.48199999999999998,"15min":0.45700000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.84599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:13:46.828 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:13:47.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.053+0000 7f5d2be00700 1 Processor -- start 2024-04-02T09:13:47.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.053+0000 7f5d2be00700 1 -- start start 2024-04-02T09:13:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.055+0000 7f5d2be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5d240b3af0 0x7f5d2415a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.055+0000 7f5d2be00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5d240709c0 0x7f5d24070e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.055+0000 7f5d2be00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d240b2180 con 0x7f5d24072580 2024-04-02T09:13:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.055+0000 7f5d2be00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d2415b100 con 0x7f5d240b3af0 2024-04-02T09:13:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.055+0000 7f5d2be00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d2415b3e0 con 0x7f5d240709c0 2024-04-02T09:13:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.055+0000 7f5d29400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5d240b3af0 0x7f5d2415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.055+0000 7f5d29400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5d240b3af0 0x7f5d2415a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55322/0 (socket says 172.21.15.89:55322) 2024-04-02T09:13:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.055+0000 7f5d29400700 1 -- 172.21.15.89:0/3758664970 learned_addr learned my addr 172.21.15.89:0/3758664970 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.055+0000 7f5d29e00700 1 --2- 172.21.15.89:0/3758664970 >> v2:172.21.15.113:3300/0 conn(0x7f5d240709c0 0x7f5d24070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:47.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.055+0000 7f5d29400700 1 -- 172.21.15.89:0/3758664970 >> v2:172.21.15.113:3300/0 conn(0x7f5d240709c0 msgr2=0x7f5d24070e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:47.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.055+0000 7f5d29400700 1 --2- 172.21.15.89:0/3758664970 >> v2:172.21.15.113:3300/0 conn(0x7f5d240709c0 0x7f5d24070e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.055+0000 7f5d29400700 1 -- 172.21.15.89:0/3758664970 >> v1:172.21.15.113:6789/0 conn(0x7f5d24072580 legacy=0x7f5d240b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:47.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.055+0000 7f5d29400700 1 -- 172.21.15.89:0/3758664970 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d240b23f0 con 0x7f5d240b3af0 2024-04-02T09:13:47.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.055+0000 7f5d29400700 1 --2- 172.21.15.89:0/3758664970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d240b3af0 0x7f5d2415a860 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f5d24158390 tx=0x7f5d18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a337be9a34a89b20 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:47.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.056+0000 7f5d23e00700 1 -- 172.21.15.89:0/3758664970 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d18004230 con 0x7f5d240b3af0 2024-04-02T09:13:47.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.056+0000 7f5d23e00700 1 -- 172.21.15.89:0/3758664970 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d18004390 con 0x7f5d240b3af0 2024-04-02T09:13:47.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.056+0000 7f5d23e00700 1 -- 172.21.15.89:0/3758664970 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d180036a0 con 0x7f5d240b3af0 2024-04-02T09:13:47.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.056+0000 7f5d2be00700 1 -- 172.21.15.89:0/3758664970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d240b3af0 msgr2=0x7f5d2415a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:47.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.056+0000 7f5d2be00700 1 --2- 172.21.15.89:0/3758664970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d240b3af0 0x7f5d2415a860 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f5d24158390 tx=0x7f5d18009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.056+0000 7f5d2be00700 1 -- 172.21.15.89:0/3758664970 shutdown_connections 2024-04-02T09:13:47.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.056+0000 7f5d2be00700 1 --2- 172.21.15.89:0/3758664970 >> v2:172.21.15.113:3300/0 conn(0x7f5d240709c0 0x7f5d24070e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.056+0000 7f5d2be00700 1 --2- 172.21.15.89:0/3758664970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d240b3af0 0x7f5d2415a860 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.056+0000 7f5d2be00700 1 -- 172.21.15.89:0/3758664970 >> 172.21.15.89:0/3758664970 conn(0x7f5d240cf190 msgr2=0x7f5d240cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:47.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.056+0000 7f5d2be00700 1 -- 172.21.15.89:0/3758664970 shutdown_connections 2024-04-02T09:13:47.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.057+0000 7f5d2be00700 1 -- 172.21.15.89:0/3758664970 wait complete. 2024-04-02T09:13:47.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.058+0000 7f5d2be00700 1 Processor -- start 2024-04-02T09:13:47.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.058+0000 7f5d2be00700 1 -- start start 2024-04-02T09:13:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.059+0000 7f5d2be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d240709c0 0x7f5d240cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.059+0000 7f5d2be00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d24072580 0x7f5d2414cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.059+0000 7f5d2be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d240b3af0 0x7f5d24153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.059+0000 7f5d2be00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d24150470 con 0x7f5d240b3af0 2024-04-02T09:13:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.059+0000 7f5d2be00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d24150750 con 0x7f5d24072580 2024-04-02T09:13:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.059+0000 7f5d2be00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d24150a30 con 0x7f5d240709c0 2024-04-02T09:13:47.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.059+0000 7f5d29e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d240b3af0 0x7f5d24153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:47.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.059+0000 7f5d29e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d240b3af0 0x7f5d24153ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55324/0 (socket says 172.21.15.89:55324) 2024-04-02T09:13:47.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.059+0000 7f5d29e00700 1 -- 172.21.15.89:0/791082207 learned_addr learned my addr 172.21.15.89:0/791082207 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:47.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.060+0000 7f5d29400700 1 --2- 172.21.15.89:0/791082207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d240709c0 0x7f5d240cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:47.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.060+0000 7f5d28a00700 1 --2- 172.21.15.89:0/791082207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d24072580 0x7f5d2414cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:47.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.060+0000 7f5d29e00700 1 -- 172.21.15.89:0/791082207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d240709c0 msgr2=0x7f5d240cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:47.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.060+0000 7f5d29e00700 1 --2- 172.21.15.89:0/791082207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d240709c0 0x7f5d240cc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.060+0000 7f5d29e00700 1 -- 172.21.15.89:0/791082207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d24072580 msgr2=0x7f5d2414cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:47.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.060+0000 7f5d29e00700 1 --2- 172.21.15.89:0/791082207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d24072580 0x7f5d2414cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.060+0000 7f5d29e00700 1 -- 172.21.15.89:0/791082207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d18009550 con 0x7f5d240b3af0 2024-04-02T09:13:47.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.060+0000 7f5d29e00700 1 --2- 172.21.15.89:0/791082207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d240b3af0 0x7f5d24153ce0 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c007e40 tx=0x7f5d1c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:47.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.060+0000 7f5d21600700 1 -- 172.21.15.89:0/791082207 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c00ed30 con 0x7f5d240b3af0 2024-04-02T09:13:47.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.060+0000 7f5d21600700 1 -- 172.21.15.89:0/791082207 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c014420 con 0x7f5d240b3af0 2024-04-02T09:13:47.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.060+0000 7f5d21600700 1 -- 172.21.15.89:0/791082207 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c0147d0 con 0x7f5d240b3af0 2024-04-02T09:13:47.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.060+0000 7f5d2be00700 1 -- 172.21.15.89:0/791082207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d24150d10 con 0x7f5d240b3af0 2024-04-02T09:13:47.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.061+0000 7f5d2be00700 1 -- 172.21.15.89:0/791082207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d240ccaf0 con 0x7f5d240b3af0 2024-04-02T09:13:47.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.061+0000 7f5d2be00700 1 -- 172.21.15.89:0/791082207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d24067720 con 0x7f5d240b3af0 2024-04-02T09:13:47.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.062+0000 7f5d21600700 1 -- 172.21.15.89:0/791082207 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5d1c014930 con 0x7f5d240b3af0 2024-04-02T09:13:47.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.063+0000 7f5d21600700 1 --2- 172.21.15.89:0/791082207 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5d0406aed0 0x7f5d0406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:47.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.063+0000 7f5d21600700 1 -- 172.21.15.89:0/791082207 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f5d1c08a2f0 con 0x7f5d240b3af0 2024-04-02T09:13:47.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.063+0000 7f5d29400700 1 --2- 172.21.15.89:0/791082207 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5d0406aed0 0x7f5d0406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:47.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.064+0000 7f5d29400700 1 --2- 172.21.15.89:0/791082207 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5d0406aed0 0x7f5d0406d390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5d18009130 tx=0x7f5d18008030 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:47.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.068+0000 7f5d21600700 1 -- 172.21.15.89:0/791082207 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5d1c054c80 con 0x7f5d240b3af0 2024-04-02T09:13:47.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.266+0000 7f5d2be00700 1 -- 172.21.15.89:0/791082207 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5d240cccb0 con 0x7f5d0406aed0 2024-04-02T09:13:47.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.279+0000 7f5d21600700 1 -- 172.21.15.89:0/791082207 <== mgr.11613 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+517525 (secure 0 0 0) 0x7f5d240cccb0 con 0x7f5d0406aed0 2024-04-02T09:13:47.284 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:47.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.285+0000 7f5d2be00700 1 -- 172.21.15.89:0/791082207 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5d0406aed0 msgr2=0x7f5d0406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:47.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.285+0000 7f5d2be00700 1 --2- 172.21.15.89:0/791082207 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5d0406aed0 0x7f5d0406d390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5d18009130 tx=0x7f5d18008030 comp rx=0 tx=0).stop 2024-04-02T09:13:47.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.285+0000 7f5d2be00700 1 -- 172.21.15.89:0/791082207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d240b3af0 msgr2=0x7f5d24153ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:47.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.285+0000 7f5d2be00700 1 --2- 172.21.15.89:0/791082207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d240b3af0 0x7f5d24153ce0 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c007e40 tx=0x7f5d1c00e3c0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.285+0000 7f5d2be00700 1 -- 172.21.15.89:0/791082207 shutdown_connections 2024-04-02T09:13:47.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.285+0000 7f5d2be00700 1 --2- 172.21.15.89:0/791082207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d240b3af0 0x7f5d24153ce0 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.285+0000 7f5d2be00700 1 --2- 172.21.15.89:0/791082207 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d240709c0 0x7f5d240cc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.285+0000 7f5d2be00700 1 --2- 172.21.15.89:0/791082207 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5d0406aed0 0x7f5d0406d390 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.285+0000 7f5d2be00700 1 --2- 172.21.15.89:0/791082207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d24072580 0x7f5d2414cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.285+0000 7f5d2be00700 1 -- 172.21.15.89:0/791082207 >> 172.21.15.89:0/791082207 conn(0x7f5d240cf190 msgr2=0x7f5d2406d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:47.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.286+0000 7f5d2be00700 1 -- 172.21.15.89:0/791082207 shutdown_connections 2024-04-02T09:13:47.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.286+0000 7f5d2be00700 1 -- 172.21.15.89:0/791082207 wait complete. 2024-04-02T09:13:47.290 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:13:47.302 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-04-02T09:13:45.432285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593971,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":666,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":203,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":412864,"kb_used_data":25312,"kb_used_omap":12,"kb_used_meta":387443,"kb_avail":754561856,"statfs":{"total":773094113280,"available":772671340544,"internally_reserved":0,"allocated":25919488,"data_stored":13801624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":396742200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":987,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.011247"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425059+0000","last_change":"2024-04-02T09:13:40.425059+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425059+0000","last_undegraded":"2024-04-02T09:13:40.425059+0000","last_fullsized":"2024-04-02T09:13:40.425059+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:12:22.769587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426693+0000","last_change":"2024-04-02T09:13:28.396642+0000","last_active":"2024-04-02T09:13:31.426693+0000","last_peered":"2024-04-02T09:13:31.426693+0000","last_clean":"2024-04-02T09:13:31.426693+0000","last_became_active":"2024-04-02T09:13:28.396356+0000","last_became_peered":"2024-04-02T09:13:28.396356+0000","last_unstale":"2024-04-02T09:13:31.426693+0000","last_undegraded":"2024-04-02T09:13:31.426693+0000","last_fullsized":"2024-04-02T09:13:31.426693+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:34.295480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422195+0000","last_change":"2024-04-02T09:13:40.422195+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422195+0000","last_undegraded":"2024-04-02T09:13:40.422195+0000","last_fullsized":"2024-04-02T09:13:40.422195+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:24:28.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424078+0000","last_change":"2024-04-02T09:13:28.381100+0000","last_active":"2024-04-02T09:13:31.424078+0000","last_peered":"2024-04-02T09:13:31.424078+0000","last_clean":"2024-04-02T09:13:31.424078+0000","last_became_active":"2024-04-02T09:13:28.380832+0000","last_became_peered":"2024-04-02T09:13:28.380832+0000","last_unstale":"2024-04-02T09:13:31.424078+0000","last_undegraded":"2024-04-02T09:13:31.424078+0000","last_fullsized":"2024-04-02T09:13:31.424078+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:21.987215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.426705+0000","last_change":"2024-04-02T09:13:40.426705+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.426705+0000","last_undegraded":"2024-04-02T09:13:40.426705+0000","last_fullsized":"2024-04-02T09:13:40.426705+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:31:39.320935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426652+0000","last_change":"2024-04-02T09:13:28.393942+0000","last_active":"2024-04-02T09:13:31.426652+0000","last_peered":"2024-04-02T09:13:31.426652+0000","last_clean":"2024-04-02T09:13:31.426652+0000","last_became_active":"2024-04-02T09:13:28.393467+0000","last_became_peered":"2024-04-02T09:13:28.393467+0000","last_unstale":"2024-04-02T09:13:31.426652+0000","last_undegraded":"2024-04-02T09:13:31.426652+0000","last_fullsized":"2024-04-02T09:13:31.426652+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:24.836500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.427104+0000","last_change":"2024-04-02T09:13:40.427104+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.427104+0000","last_undegraded":"2024-04-02T09:13:40.427104+0000","last_fullsized":"2024-04-02T09:13:40.427104+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:13:02.182890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426886+0000","last_change":"2024-04-02T09:13:28.394098+0000","last_active":"2024-04-02T09:13:31.426886+0000","last_peered":"2024-04-02T09:13:31.426886+0000","last_clean":"2024-04-02T09:13:31.426886+0000","last_became_active":"2024-04-02T09:13:28.393837+0000","last_became_peered":"2024-04-02T09:13:28.393837+0000","last_unstale":"2024-04-02T09:13:31.426886+0000","last_undegraded":"2024-04-02T09:13:31.426886+0000","last_fullsized":"2024-04-02T09:13:31.426886+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:40:27.068896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425796+0000","last_change":"2024-04-02T09:13:40.425796+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425796+0000","last_undegraded":"2024-04-02T09:13:40.425796+0000","last_fullsized":"2024-04-02T09:13:40.425796+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:53.327540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427560+0000","last_change":"2024-04-02T09:13:28.379239+0000","last_active":"2024-04-02T09:13:31.427560+0000","last_peered":"2024-04-02T09:13:31.427560+0000","last_clean":"2024-04-02T09:13:31.427560+0000","last_became_active":"2024-04-02T09:13:28.378970+0000","last_became_peered":"2024-04-02T09:13:28.378970+0000","last_unstale":"2024-04-02T09:13:31.427560+0000","last_undegraded":"2024-04-02T09:13:31.427560+0000","last_fullsized":"2024-04-02T09:13:31.427560+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:37.960638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422210+0000","last_change":"2024-04-02T09:13:40.422210+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422210+0000","last_undegraded":"2024-04-02T09:13:40.422210+0000","last_fullsized":"2024-04-02T09:13:40.422210+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:18.892470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426567+0000","last_change":"2024-04-02T09:13:28.393747+0000","last_active":"2024-04-02T09:13:31.426567+0000","last_peered":"2024-04-02T09:13:31.426567+0000","last_clean":"2024-04-02T09:13:31.426567+0000","last_became_active":"2024-04-02T09:13:28.393515+0000","last_became_peered":"2024-04-02T09:13:28.393515+0000","last_unstale":"2024-04-02T09:13:31.426567+0000","last_undegraded":"2024-04-02T09:13:31.426567+0000","last_fullsized":"2024-04-02T09:13:31.426567+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:28:37.462844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422411+0000","last_change":"2024-04-02T09:13:40.422411+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422411+0000","last_undegraded":"2024-04-02T09:13:40.422411+0000","last_fullsized":"2024-04-02T09:13:40.422411+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:53.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629507+0000","last_change":"2024-04-02T09:13:28.399975+0000","last_active":"2024-04-02T09:13:31.629507+0000","last_peered":"2024-04-02T09:13:31.629507+0000","last_clean":"2024-04-02T09:13:31.629507+0000","last_became_active":"2024-04-02T09:13:28.399335+0000","last_became_peered":"2024-04-02T09:13:28.399335+0000","last_unstale":"2024-04-02T09:13:31.629507+0000","last_undegraded":"2024-04-02T09:13:31.629507+0000","last_fullsized":"2024-04-02T09:13:31.629507+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:16:14.265813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422275+0000","last_change":"2024-04-02T09:13:40.422275+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422275+0000","last_undegraded":"2024-04-02T09:13:40.422275+0000","last_fullsized":"2024-04-02T09:13:40.422275+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:40.922210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427506+0000","last_change":"2024-04-02T09:13:28.392551+0000","last_active":"2024-04-02T09:13:31.427506+0000","last_peered":"2024-04-02T09:13:31.427506+0000","last_clean":"2024-04-02T09:13:31.427506+0000","last_became_active":"2024-04-02T09:13:28.392159+0000","last_became_peered":"2024-04-02T09:13:28.392159+0000","last_unstale":"2024-04-02T09:13:31.427506+0000","last_undegraded":"2024-04-02T09:13:31.427506+0000","last_fullsized":"2024-04-02T09:13:31.427506+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:42.551696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.427207+0000","last_change":"2024-04-02T09:13:40.427207+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.427207+0000","last_undegraded":"2024-04-02T09:13:40.427207+0000","last_fullsized":"2024-04-02T09:13:40.427207+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:08.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424242+0000","last_change":"2024-04-02T09:13:28.383891+0000","last_active":"2024-04-02T09:13:31.424242+0000","last_peered":"2024-04-02T09:13:31.424242+0000","last_clean":"2024-04-02T09:13:31.424242+0000","last_became_active":"2024-04-02T09:13:28.383628+0000","last_became_peered":"2024-04-02T09:13:28.383628+0000","last_unstale":"2024-04-02T09:13:31.424242+0000","last_undegraded":"2024-04-02T09:13:31.424242+0000","last_fullsized":"2024-04-02T09:13:31.424242+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:44.222993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422239+0000","last_change":"2024-04-02T09:13:40.422239+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422239+0000","last_undegraded":"2024-04-02T09:13:40.422239+0000","last_fullsized":"2024-04-02T09:13:40.422239+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:10.066167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629713+0000","last_change":"2024-04-02T09:13:28.404390+0000","last_active":"2024-04-02T09:13:31.629713+0000","last_peered":"2024-04-02T09:13:31.629713+0000","last_clean":"2024-04-02T09:13:31.629713+0000","last_became_active":"2024-04-02T09:13:28.404038+0000","last_became_peered":"2024-04-02T09:13:28.404038+0000","last_unstale":"2024-04-02T09:13:31.629713+0000","last_undegraded":"2024-04-02T09:13:31.629713+0000","last_fullsized":"2024-04-02T09:13:31.629713+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:47.359176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422238+0000","last_change":"2024-04-02T09:13:40.422238+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422238+0000","last_undegraded":"2024-04-02T09:13:40.422238+0000","last_fullsized":"2024-04-02T09:13:40.422238+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:50:01.664013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426922+0000","last_change":"2024-04-02T09:13:28.403517+0000","last_active":"2024-04-02T09:13:31.426922+0000","last_peered":"2024-04-02T09:13:31.426922+0000","last_clean":"2024-04-02T09:13:31.426922+0000","last_became_active":"2024-04-02T09:13:28.401353+0000","last_became_peered":"2024-04-02T09:13:28.401353+0000","last_unstale":"2024-04-02T09:13:31.426922+0000","last_undegraded":"2024-04-02T09:13:31.426922+0000","last_fullsized":"2024-04-02T09:13:31.426922+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:08.980542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425711+0000","last_change":"2024-04-02T09:13:40.425711+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425711+0000","last_undegraded":"2024-04-02T09:13:40.425711+0000","last_fullsized":"2024-04-02T09:13:40.425711+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:08.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629692+0000","last_change":"2024-04-02T09:13:28.399938+0000","last_active":"2024-04-02T09:13:31.629692+0000","last_peered":"2024-04-02T09:13:31.629692+0000","last_clean":"2024-04-02T09:13:31.629692+0000","last_became_active":"2024-04-02T09:13:28.399350+0000","last_became_peered":"2024-04-02T09:13:28.399350+0000","last_unstale":"2024-04-02T09:13:31.629692+0000","last_undegraded":"2024-04-02T09:13:31.629692+0000","last_fullsized":"2024-04-02T09:13:31.629692+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:05.933699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422193+0000","last_change":"2024-04-02T09:13:40.422193+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422193+0000","last_undegraded":"2024-04-02T09:13:40.422193+0000","last_fullsized":"2024-04-02T09:13:40.422193+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:53.842095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424157+0000","last_change":"2024-04-02T09:13:28.386688+0000","last_active":"2024-04-02T09:13:31.424157+0000","last_peered":"2024-04-02T09:13:31.424157+0000","last_clean":"2024-04-02T09:13:31.424157+0000","last_became_active":"2024-04-02T09:13:28.386363+0000","last_became_peered":"2024-04-02T09:13:28.386363+0000","last_unstale":"2024-04-02T09:13:31.424157+0000","last_undegraded":"2024-04-02T09:13:31.424157+0000","last_fullsized":"2024-04-02T09:13:31.424157+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:12:41.480404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422972+0000","last_change":"2024-04-02T09:13:40.422972+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422972+0000","last_undegraded":"2024-04-02T09:13:40.422972+0000","last_fullsized":"2024-04-02T09:13:40.422972+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:22:20.170595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426790+0000","last_change":"2024-04-02T09:13:28.401903+0000","last_active":"2024-04-02T09:13:31.426790+0000","last_peered":"2024-04-02T09:13:31.426790+0000","last_clean":"2024-04-02T09:13:31.426790+0000","last_became_active":"2024-04-02T09:13:28.399098+0000","last_became_peered":"2024-04-02T09:13:28.399098+0000","last_unstale":"2024-04-02T09:13:31.426790+0000","last_undegraded":"2024-04-02T09:13:31.426790+0000","last_fullsized":"2024-04-02T09:13:31.426790+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:33:56.420978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425751+0000","last_change":"2024-04-02T09:13:40.425751+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425751+0000","last_undegraded":"2024-04-02T09:13:40.425751+0000","last_fullsized":"2024-04-02T09:13:40.425751+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:53.688559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422619+0000","last_change":"2024-04-02T09:13:28.383146+0000","last_active":"2024-04-02T09:13:31.422619+0000","last_peered":"2024-04-02T09:13:31.422619+0000","last_clean":"2024-04-02T09:13:31.422619+0000","last_became_active":"2024-04-02T09:13:28.382879+0000","last_became_peered":"2024-04-02T09:13:28.382879+0000","last_unstale":"2024-04-02T09:13:31.422619+0000","last_undegraded":"2024-04-02T09:13:31.422619+0000","last_fullsized":"2024-04-02T09:13:31.422619+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:57:59.273299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425074+0000","last_change":"2024-04-02T09:13:40.425074+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425074+0000","last_undegraded":"2024-04-02T09:13:40.425074+0000","last_fullsized":"2024-04-02T09:13:40.425074+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:56.693535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.423992+0000","last_change":"2024-04-02T09:13:28.381499+0000","last_active":"2024-04-02T09:13:31.423992+0000","last_peered":"2024-04-02T09:13:31.423992+0000","last_clean":"2024-04-02T09:13:31.423992+0000","last_became_active":"2024-04-02T09:13:28.377983+0000","last_became_peered":"2024-04-02T09:13:28.377983+0000","last_unstale":"2024-04-02T09:13:31.423992+0000","last_undegraded":"2024-04-02T09:13:31.423992+0000","last_fullsized":"2024-04-02T09:13:31.423992+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:45.021173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.426751+0000","last_change":"2024-04-02T09:13:40.426751+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.426751+0000","last_undegraded":"2024-04-02T09:13:40.426751+0000","last_fullsized":"2024-04-02T09:13:40.426751+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:14.301601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629812+0000","last_change":"2024-04-02T09:13:28.382899+0000","last_active":"2024-04-02T09:13:31.629812+0000","last_peered":"2024-04-02T09:13:31.629812+0000","last_clean":"2024-04-02T09:13:31.629812+0000","last_became_active":"2024-04-02T09:13:28.382023+0000","last_became_peered":"2024-04-02T09:13:28.382023+0000","last_unstale":"2024-04-02T09:13:31.629812+0000","last_undegraded":"2024-04-02T09:13:31.629812+0000","last_fullsized":"2024-04-02T09:13:31.629812+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:12.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425230+0000","last_change":"2024-04-02T09:13:40.425230+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425230+0000","last_undegraded":"2024-04-02T09:13:40.425230+0000","last_fullsized":"2024-04-02T09:13:40.425230+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:40.922210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630208+0000","last_change":"2024-04-02T09:13:28.390594+0000","last_active":"2024-04-02T09:13:31.630208+0000","last_peered":"2024-04-02T09:13:31.630208+0000","last_clean":"2024-04-02T09:13:31.630208+0000","last_became_active":"2024-04-02T09:13:28.388061+0000","last_became_peered":"2024-04-02T09:13:28.388061+0000","last_unstale":"2024-04-02T09:13:31.630208+0000","last_undegraded":"2024-04-02T09:13:31.630208+0000","last_fullsized":"2024-04-02T09:13:31.630208+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:14:04.841237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422434+0000","last_change":"2024-04-02T09:13:40.422434+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422434+0000","last_undegraded":"2024-04-02T09:13:40.422434+0000","last_fullsized":"2024-04-02T09:13:40.422434+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:56.693535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424191+0000","last_change":"2024-04-02T09:13:28.382883+0000","last_active":"2024-04-02T09:13:31.424191+0000","last_peered":"2024-04-02T09:13:31.424191+0000","last_clean":"2024-04-02T09:13:31.424191+0000","last_became_active":"2024-04-02T09:13:28.382187+0000","last_became_peered":"2024-04-02T09:13:28.382187+0000","last_unstale":"2024-04-02T09:13:31.424191+0000","last_undegraded":"2024-04-02T09:13:31.424191+0000","last_fullsized":"2024-04-02T09:13:31.424191+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:21:26.362248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.426415+0000","last_change":"2024-04-02T09:13:40.426415+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.426415+0000","last_undegraded":"2024-04-02T09:13:40.426415+0000","last_fullsized":"2024-04-02T09:13:40.426415+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:19.429201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424015+0000","last_change":"2024-04-02T09:13:28.384295+0000","last_active":"2024-04-02T09:13:31.424015+0000","last_peered":"2024-04-02T09:13:31.424015+0000","last_clean":"2024-04-02T09:13:31.424015+0000","last_became_active":"2024-04-02T09:13:28.384010+0000","last_became_peered":"2024-04-02T09:13:28.384010+0000","last_unstale":"2024-04-02T09:13:31.424015+0000","last_undegraded":"2024-04-02T09:13:31.424015+0000","last_fullsized":"2024-04-02T09:13:31.424015+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:47:33.241308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425141+0000","last_change":"2024-04-02T09:13:40.425141+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425141+0000","last_undegraded":"2024-04-02T09:13:40.425141+0000","last_fullsized":"2024-04-02T09:13:40.425141+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:08.249339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427569+0000","last_change":"2024-04-02T09:13:28.380209+0000","last_active":"2024-04-02T09:13:31.427569+0000","last_peered":"2024-04-02T09:13:31.427569+0000","last_clean":"2024-04-02T09:13:31.427569+0000","last_became_active":"2024-04-02T09:13:28.379739+0000","last_became_peered":"2024-04-02T09:13:28.379739+0000","last_unstale":"2024-04-02T09:13:31.427569+0000","last_undegraded":"2024-04-02T09:13:31.427569+0000","last_fullsized":"2024-04-02T09:13:31.427569+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:46:14.163290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422802+0000","last_change":"2024-04-02T09:13:40.422802+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422802+0000","last_undegraded":"2024-04-02T09:13:40.422802+0000","last_fullsized":"2024-04-02T09:13:40.422802+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:47:51.824824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424141+0000","last_change":"2024-04-02T09:13:28.392993+0000","last_active":"2024-04-02T09:13:31.424141+0000","last_peered":"2024-04-02T09:13:31.424141+0000","last_clean":"2024-04-02T09:13:31.424141+0000","last_became_active":"2024-04-02T09:13:28.392712+0000","last_became_peered":"2024-04-02T09:13:28.392712+0000","last_unstale":"2024-04-02T09:13:31.424141+0000","last_undegraded":"2024-04-02T09:13:31.424141+0000","last_fullsized":"2024-04-02T09:13:31.424141+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:38.436370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422422+0000","last_change":"2024-04-02T09:13:40.422422+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422422+0000","last_undegraded":"2024-04-02T09:13:40.422422+0000","last_fullsized":"2024-04-02T09:13:40.422422+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:48:38.200651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424060+0000","last_change":"2024-04-02T09:13:28.378030+0000","last_active":"2024-04-02T09:13:31.424060+0000","last_peered":"2024-04-02T09:13:31.424060+0000","last_clean":"2024-04-02T09:13:31.424060+0000","last_became_active":"2024-04-02T09:13:28.377700+0000","last_became_peered":"2024-04-02T09:13:28.377700+0000","last_unstale":"2024-04-02T09:13:31.424060+0000","last_undegraded":"2024-04-02T09:13:31.424060+0000","last_fullsized":"2024-04-02T09:13:31.424060+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:40.236322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.428373+0000","last_change":"2024-04-02T09:13:40.428373+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.428373+0000","last_undegraded":"2024-04-02T09:13:40.428373+0000","last_fullsized":"2024-04-02T09:13:40.428373+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:27.266184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426698+0000","last_change":"2024-04-02T09:13:28.395364+0000","last_active":"2024-04-02T09:13:31.426698+0000","last_peered":"2024-04-02T09:13:31.426698+0000","last_clean":"2024-04-02T09:13:31.426698+0000","last_became_active":"2024-04-02T09:13:28.395100+0000","last_became_peered":"2024-04-02T09:13:28.395100+0000","last_unstale":"2024-04-02T09:13:31.426698+0000","last_undegraded":"2024-04-02T09:13:31.426698+0000","last_fullsized":"2024-04-02T09:13:31.426698+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:12.863676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425209+0000","last_change":"2024-04-02T09:13:40.425209+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425209+0000","last_undegraded":"2024-04-02T09:13:40.425209+0000","last_fullsized":"2024-04-02T09:13:40.425209+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:47:51.824824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426979+0000","last_change":"2024-04-02T09:13:28.399604+0000","last_active":"2024-04-02T09:13:31.426979+0000","last_peered":"2024-04-02T09:13:31.426979+0000","last_clean":"2024-04-02T09:13:31.426979+0000","last_became_active":"2024-04-02T09:13:28.399333+0000","last_became_peered":"2024-04-02T09:13:28.399333+0000","last_unstale":"2024-04-02T09:13:31.426979+0000","last_undegraded":"2024-04-02T09:13:31.426979+0000","last_fullsized":"2024-04-02T09:13:31.426979+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:18.748067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.428242+0000","last_change":"2024-04-02T09:13:40.428242+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.428242+0000","last_undegraded":"2024-04-02T09:13:40.428242+0000","last_fullsized":"2024-04-02T09:13:40.428242+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:19.429201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427635+0000","last_change":"2024-04-02T09:13:28.378770+0000","last_active":"2024-04-02T09:13:31.427635+0000","last_peered":"2024-04-02T09:13:31.427635+0000","last_clean":"2024-04-02T09:13:31.427635+0000","last_became_active":"2024-04-02T09:13:28.378521+0000","last_became_peered":"2024-04-02T09:13:28.378521+0000","last_unstale":"2024-04-02T09:13:31.427635+0000","last_undegraded":"2024-04-02T09:13:31.427635+0000","last_fullsized":"2024-04-02T09:13:31.427635+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:24:11.210144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.423010+0000","last_change":"2024-04-02T09:13:40.423010+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.423010+0000","last_undegraded":"2024-04-02T09:13:40.423010+0000","last_fullsized":"2024-04-02T09:13:40.423010+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:45:49.868260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424038+0000","last_change":"2024-04-02T09:13:28.389084+0000","last_active":"2024-04-02T09:13:31.424038+0000","last_peered":"2024-04-02T09:13:31.424038+0000","last_clean":"2024-04-02T09:13:31.424038+0000","last_became_active":"2024-04-02T09:13:28.388624+0000","last_became_peered":"2024-04-02T09:13:28.388624+0000","last_unstale":"2024-04-02T09:13:31.424038+0000","last_undegraded":"2024-04-02T09:13:31.424038+0000","last_fullsized":"2024-04-02T09:13:31.424038+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:23.766461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.422211+0000","last_change":"2024-04-02T09:13:40.422211+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.422211+0000","last_undegraded":"2024-04-02T09:13:40.422211+0000","last_fullsized":"2024-04-02T09:13:40.422211+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:45.017947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426864+0000","last_change":"2024-04-02T09:13:28.395286+0000","last_active":"2024-04-02T09:13:31.426864+0000","last_peered":"2024-04-02T09:13:31.426864+0000","last_clean":"2024-04-02T09:13:31.426864+0000","last_became_active":"2024-04-02T09:13:28.394377+0000","last_became_peered":"2024-04-02T09:13:28.394377+0000","last_unstale":"2024-04-02T09:13:31.426864+0000","last_undegraded":"2024-04-02T09:13:31.426864+0000","last_fullsized":"2024-04-02T09:13:31.426864+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:37:39.028435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.425714+0000","last_change":"2024-04-02T09:13:40.425714+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.425714+0000","last_undegraded":"2024-04-02T09:13:40.425714+0000","last_fullsized":"2024-04-02T09:13:40.425714+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:52.540226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426829+0000","last_change":"2024-04-02T09:13:28.391267+0000","last_active":"2024-04-02T09:13:31.426829+0000","last_peered":"2024-04-02T09:13:31.426829+0000","last_clean":"2024-04-02T09:13:31.426829+0000","last_became_active":"2024-04-02T09:13:28.390985+0000","last_became_peered":"2024-04-02T09:13:28.390985+0000","last_unstale":"2024-04-02T09:13:31.426829+0000","last_undegraded":"2024-04-02T09:13:31.426829+0000","last_fullsized":"2024-04-02T09:13:31.426829+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:30.210394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.427271+0000","last_change":"2024-04-02T09:13:40.427271+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.427271+0000","last_undegraded":"2024-04-02T09:13:40.427271+0000","last_fullsized":"2024-04-02T09:13:40.427271+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:23:34.898152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424106+0000","last_change":"2024-04-02T09:13:28.380650+0000","last_active":"2024-04-02T09:13:31.424106+0000","last_peered":"2024-04-02T09:13:31.424106+0000","last_clean":"2024-04-02T09:13:31.424106+0000","last_became_active":"2024-04-02T09:13:28.380376+0000","last_became_peered":"2024-04-02T09:13:28.380376+0000","last_unstale":"2024-04-02T09:13:31.424106+0000","last_undegraded":"2024-04-02T09:13:31.424106+0000","last_fullsized":"2024-04-02T09:13:31.424106+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:46:17.713244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.427715+0000","last_change":"2024-04-02T09:13:40.427715+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.427715+0000","last_undegraded":"2024-04-02T09:13:40.427715+0000","last_fullsized":"2024-04-02T09:13:40.427715+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:21.506479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426750+0000","last_change":"2024-04-02T09:13:28.390273+0000","last_active":"2024-04-02T09:13:31.426750+0000","last_peered":"2024-04-02T09:13:31.426750+0000","last_clean":"2024-04-02T09:13:31.426750+0000","last_became_active":"2024-04-02T09:13:28.389990+0000","last_became_peered":"2024-04-02T09:13:28.389990+0000","last_unstale":"2024-04-02T09:13:31.426750+0000","last_undegraded":"2024-04-02T09:13:31.426750+0000","last_fullsized":"2024-04-02T09:13:31.426750+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:30:33.534225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-04-02T09:13:40.429062+0000","last_change":"2024-04-02T09:13:40.429062+0000","last_active":"2024-04-02T09:13:40.409301+0000","last_peered":"2024-04-02T09:13:40.409301+0000","last_clean":"2024-04-02T09:13:40.409301+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:40.429062+0000","last_undegraded":"2024-04-02T09:13:40.429062+0000","last_fullsized":"2024-04-02T09:13:40.429062+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:39.602788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.423962+0000","last_change":"2024-04-02T09:13:28.395003+0000","last_active":"2024-04-02T09:13:31.423962+0000","last_peered":"2024-04-02T09:13:31.423962+0000","last_clean":"2024-04-02T09:13:31.423962+0000","last_became_active":"2024-04-02T09:13:28.394635+0000","last_became_peered":"2024-04-02T09:13:28.394635+0000","last_unstale":"2024-04-02T09:13:31.423962+0000","last_undegraded":"2024-04-02T09:13:31.423962+0000","last_fullsized":"2024-04-02T09:13:31.423962+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:54.758118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630389+0000","last_change":"2024-04-02T09:13:28.391302+0000","last_active":"2024-04-02T09:13:31.630389+0000","last_peered":"2024-04-02T09:13:31.630389+0000","last_clean":"2024-04-02T09:13:31.630389+0000","last_became_active":"2024-04-02T09:13:28.388976+0000","last_became_peered":"2024-04-02T09:13:28.388976+0000","last_unstale":"2024-04-02T09:13:31.630389+0000","last_undegraded":"2024-04-02T09:13:31.630389+0000","last_fullsized":"2024-04-02T09:13:31.630389+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:02.556232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"243'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422779+0000","last_change":"2024-04-02T09:13:27.881269+0000","last_active":"2024-04-02T09:13:31.422779+0000","last_peered":"2024-04-02T09:13:31.422779+0000","last_clean":"2024-04-02T09:13:31.422779+0000","last_became_active":"2024-04-02T09:13:27.880664+0000","last_became_peered":"2024-04-02T09:13:27.880664+0000","last_unstale":"2024-04-02T09:13:31.422779+0000","last_undegraded":"2024-04-02T09:13:31.422779+0000","last_fullsized":"2024-04-02T09:13:31.422779+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:53.750473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426876+0000","last_change":"2024-04-02T09:13:28.399529+0000","last_active":"2024-04-02T09:13:31.426876+0000","last_peered":"2024-04-02T09:13:31.426876+0000","last_clean":"2024-04-02T09:13:31.426876+0000","last_became_active":"2024-04-02T09:13:28.399042+0000","last_became_peered":"2024-04-02T09:13:28.399042+0000","last_unstale":"2024-04-02T09:13:31.426876+0000","last_undegraded":"2024-04-02T09:13:31.426876+0000","last_fullsized":"2024-04-02T09:13:31.426876+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:58:19.109584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1d","version":"243'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426670+0000","last_change":"2024-04-02T09:13:27.889299+0000","last_active":"2024-04-02T09:13:31.426670+0000","last_peered":"2024-04-02T09:13:31.426670+0000","last_clean":"2024-04-02T09:13:31.426670+0000","last_became_active":"2024-04-02T09:13:27.888249+0000","last_became_peered":"2024-04-02T09:13:27.888249+0000","last_unstale":"2024-04-02T09:13:31.426670+0000","last_undegraded":"2024-04-02T09:13:31.426670+0000","last_fullsized":"2024-04-02T09:13:31.426670+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:01:31.866958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424210+0000","last_change":"2024-04-02T09:13:28.382423+0000","last_active":"2024-04-02T09:13:31.424210+0000","last_peered":"2024-04-02T09:13:31.424210+0000","last_clean":"2024-04-02T09:13:31.424210+0000","last_became_active":"2024-04-02T09:13:28.381840+0000","last_became_peered":"2024-04-02T09:13:28.381840+0000","last_unstale":"2024-04-02T09:13:31.424210+0000","last_undegraded":"2024-04-02T09:13:31.424210+0000","last_fullsized":"2024-04-02T09:13:31.424210+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:04:44.765954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"243'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630265+0000","last_change":"2024-04-02T09:13:27.885629+0000","last_active":"2024-04-02T09:13:31.630265+0000","last_peered":"2024-04-02T09:13:31.630265+0000","last_clean":"2024-04-02T09:13:31.630265+0000","last_became_active":"2024-04-02T09:13:27.884921+0000","last_became_peered":"2024-04-02T09:13:27.884921+0000","last_unstale":"2024-04-02T09:13:31.630265+0000","last_undegraded":"2024-04-02T09:13:31.630265+0000","last_fullsized":"2024-04-02T09:13:31.630265+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:49:58.033026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630095+0000","last_change":"2024-04-02T09:13:28.381558+0000","last_active":"2024-04-02T09:13:31.630095+0000","last_peered":"2024-04-02T09:13:31.630095+0000","last_clean":"2024-04-02T09:13:31.630095+0000","last_became_active":"2024-04-02T09:13:28.381306+0000","last_became_peered":"2024-04-02T09:13:28.381306+0000","last_unstale":"2024-04-02T09:13:31.630095+0000","last_undegraded":"2024-04-02T09:13:31.630095+0000","last_fullsized":"2024-04-02T09:13:31.630095+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:12:17.313507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1b","version":"243'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630114+0000","last_change":"2024-04-02T09:13:27.884062+0000","last_active":"2024-04-02T09:13:31.630114+0000","last_peered":"2024-04-02T09:13:31.630114+0000","last_clean":"2024-04-02T09:13:31.630114+0000","last_became_active":"2024-04-02T09:13:27.883677+0000","last_became_peered":"2024-04-02T09:13:27.883677+0000","last_unstale":"2024-04-02T09:13:31.630114+0000","last_undegraded":"2024-04-02T09:13:31.630114+0000","last_fullsized":"2024-04-02T09:13:31.630114+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:14.774442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422658+0000","last_change":"2024-04-02T09:13:28.389833+0000","last_active":"2024-04-02T09:13:31.422658+0000","last_peered":"2024-04-02T09:13:31.422658+0000","last_clean":"2024-04-02T09:13:31.422658+0000","last_became_active":"2024-04-02T09:13:28.389495+0000","last_became_peered":"2024-04-02T09:13:28.389495+0000","last_unstale":"2024-04-02T09:13:31.422658+0000","last_undegraded":"2024-04-02T09:13:31.422658+0000","last_fullsized":"2024-04-02T09:13:31.422658+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:27.049697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1a","version":"243'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630023+0000","last_change":"2024-04-02T09:13:27.883532+0000","last_active":"2024-04-02T09:13:31.630023+0000","last_peered":"2024-04-02T09:13:31.630023+0000","last_clean":"2024-04-02T09:13:31.630023+0000","last_became_active":"2024-04-02T09:13:27.883252+0000","last_became_peered":"2024-04-02T09:13:27.883252+0000","last_unstale":"2024-04-02T09:13:31.630023+0000","last_undegraded":"2024-04-02T09:13:31.630023+0000","last_fullsized":"2024-04-02T09:13:31.630023+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.950371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426724+0000","last_change":"2024-04-02T09:13:28.403943+0000","last_active":"2024-04-02T09:13:31.426724+0000","last_peered":"2024-04-02T09:13:31.426724+0000","last_clean":"2024-04-02T09:13:31.426724+0000","last_became_active":"2024-04-02T09:13:28.402197+0000","last_became_peered":"2024-04-02T09:13:28.402197+0000","last_unstale":"2024-04-02T09:13:31.426724+0000","last_undegraded":"2024-04-02T09:13:31.426724+0000","last_fullsized":"2024-04-02T09:13:31.426724+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:50.579695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427538+0000","last_change":"2024-04-02T09:13:27.883508+0000","last_active":"2024-04-02T09:13:31.427538+0000","last_peered":"2024-04-02T09:13:31.427538+0000","last_clean":"2024-04-02T09:13:31.427538+0000","last_became_active":"2024-04-02T09:13:27.883263+0000","last_became_peered":"2024-04-02T09:13:27.883263+0000","last_unstale":"2024-04-02T09:13:31.427538+0000","last_undegraded":"2024-04-02T09:13:31.427538+0000","last_fullsized":"2024-04-02T09:13:31.427538+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:37.428725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426770+0000","last_change":"2024-04-02T09:13:28.403768+0000","last_active":"2024-04-02T09:13:31.426770+0000","last_peered":"2024-04-02T09:13:31.426770+0000","last_clean":"2024-04-02T09:13:31.426770+0000","last_became_active":"2024-04-02T09:13:28.399745+0000","last_became_peered":"2024-04-02T09:13:28.399745+0000","last_unstale":"2024-04-02T09:13:31.426770+0000","last_undegraded":"2024-04-02T09:13:31.426770+0000","last_fullsized":"2024-04-02T09:13:31.426770+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:39.541325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629961+0000","last_change":"2024-04-02T09:13:27.888598+0000","last_active":"2024-04-02T09:13:31.629961+0000","last_peered":"2024-04-02T09:13:31.629961+0000","last_clean":"2024-04-02T09:13:31.629961+0000","last_became_active":"2024-04-02T09:13:27.888115+0000","last_became_peered":"2024-04-02T09:13:27.888115+0000","last_unstale":"2024-04-02T09:13:31.629961+0000","last_undegraded":"2024-04-02T09:13:31.629961+0000","last_fullsized":"2024-04-02T09:13:31.629961+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:57:03.609779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426945+0000","last_change":"2024-04-02T09:13:28.404602+0000","last_active":"2024-04-02T09:13:31.426945+0000","last_peered":"2024-04-02T09:13:31.426945+0000","last_clean":"2024-04-02T09:13:31.426945+0000","last_became_active":"2024-04-02T09:13:28.403446+0000","last_became_peered":"2024-04-02T09:13:28.403446+0000","last_unstale":"2024-04-02T09:13:31.426945+0000","last_undegraded":"2024-04-02T09:13:31.426945+0000","last_fullsized":"2024-04-02T09:13:31.426945+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:24.279179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422798+0000","last_change":"2024-04-02T09:13:27.887286+0000","last_active":"2024-04-02T09:13:31.422798+0000","last_peered":"2024-04-02T09:13:31.422798+0000","last_clean":"2024-04-02T09:13:31.422798+0000","last_became_active":"2024-04-02T09:13:27.886999+0000","last_became_peered":"2024-04-02T09:13:27.886999+0000","last_unstale":"2024-04-02T09:13:31.422798+0000","last_undegraded":"2024-04-02T09:13:31.422798+0000","last_fullsized":"2024-04-02T09:13:31.422798+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:44.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630366+0000","last_change":"2024-04-02T09:13:28.378767+0000","last_active":"2024-04-02T09:13:31.630366+0000","last_peered":"2024-04-02T09:13:31.630366+0000","last_clean":"2024-04-02T09:13:31.630366+0000","last_became_active":"2024-04-02T09:13:28.378487+0000","last_became_peered":"2024-04-02T09:13:28.378487+0000","last_unstale":"2024-04-02T09:13:31.630366+0000","last_undegraded":"2024-04-02T09:13:31.630366+0000","last_fullsized":"2024-04-02T09:13:31.630366+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:26:47.539777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426956+0000","last_change":"2024-04-02T09:13:27.884469+0000","last_active":"2024-04-02T09:13:31.426956+0000","last_peered":"2024-04-02T09:13:31.426956+0000","last_clean":"2024-04-02T09:13:31.426956+0000","last_became_active":"2024-04-02T09:13:27.882570+0000","last_became_peered":"2024-04-02T09:13:27.882570+0000","last_unstale":"2024-04-02T09:13:31.426956+0000","last_undegraded":"2024-04-02T09:13:31.426956+0000","last_fullsized":"2024-04-02T09:13:31.426956+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:57:12.395680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424028+0000","last_change":"2024-04-02T09:13:28.387205+0000","last_active":"2024-04-02T09:13:31.424028+0000","last_peered":"2024-04-02T09:13:31.424028+0000","last_clean":"2024-04-02T09:13:31.424028+0000","last_became_active":"2024-04-02T09:13:28.384126+0000","last_became_peered":"2024-04-02T09:13:28.384126+0000","last_unstale":"2024-04-02T09:13:31.424028+0000","last_undegraded":"2024-04-02T09:13:31.424028+0000","last_fullsized":"2024-04-02T09:13:31.424028+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:52.330349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.15","version":"243'3","reported_seq":15,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424048+0000","last_change":"2024-04-02T09:13:27.883818+0000","last_active":"2024-04-02T09:13:31.424048+0000","last_peered":"2024-04-02T09:13:31.424048+0000","last_clean":"2024-04-02T09:13:31.424048+0000","last_became_active":"2024-04-02T09:13:27.883527+0000","last_became_peered":"2024-04-02T09:13:27.883527+0000","last_unstale":"2024-04-02T09:13:31.424048+0000","last_undegraded":"2024-04-02T09:13:31.424048+0000","last_fullsized":"2024-04-02T09:13:31.424048+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:49.572049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427603+0000","last_change":"2024-04-02T09:13:28.379346+0000","last_active":"2024-04-02T09:13:31.427603+0000","last_peered":"2024-04-02T09:13:31.427603+0000","last_clean":"2024-04-02T09:13:31.427603+0000","last_became_active":"2024-04-02T09:13:28.378741+0000","last_became_peered":"2024-04-02T09:13:28.378741+0000","last_unstale":"2024-04-02T09:13:31.427603+0000","last_undegraded":"2024-04-02T09:13:31.427603+0000","last_fullsized":"2024-04-02T09:13:31.427603+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:44.092529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.14","version":"244'8","reported_seq":20,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:38.348183+0000","last_change":"2024-04-02T09:13:27.885270+0000","last_active":"2024-04-02T09:13:38.348183+0000","last_peered":"2024-04-02T09:13:38.348183+0000","last_clean":"2024-04-02T09:13:38.348183+0000","last_became_active":"2024-04-02T09:13:27.884953+0000","last_became_peered":"2024-04-02T09:13:27.884953+0000","last_unstale":"2024-04-02T09:13:38.348183+0000","last_undegraded":"2024-04-02T09:13:38.348183+0000","last_fullsized":"2024-04-02T09:13:38.348183+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:48.103277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2805,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422639+0000","last_change":"2024-04-02T09:13:28.393735+0000","last_active":"2024-04-02T09:13:31.422639+0000","last_peered":"2024-04-02T09:13:31.422639+0000","last_clean":"2024-04-02T09:13:31.422639+0000","last_became_active":"2024-04-02T09:13:28.393387+0000","last_became_peered":"2024-04-02T09:13:28.393387+0000","last_unstale":"2024-04-02T09:13:31.422639+0000","last_undegraded":"2024-04-02T09:13:31.422639+0000","last_fullsized":"2024-04-02T09:13:31.422639+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:47.359176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.13","version":"243'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426591+0000","last_change":"2024-04-02T09:13:27.885596+0000","last_active":"2024-04-02T09:13:31.426591+0000","last_peered":"2024-04-02T09:13:31.426591+0000","last_clean":"2024-04-02T09:13:31.426591+0000","last_became_active":"2024-04-02T09:13:27.885310+0000","last_became_peered":"2024-04-02T09:13:27.885310+0000","last_unstale":"2024-04-02T09:13:31.426591+0000","last_undegraded":"2024-04-02T09:13:31.426591+0000","last_fullsized":"2024-04-02T09:13:31.426591+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:02.497118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424172+0000","last_change":"2024-04-02T09:13:28.378384+0000","last_active":"2024-04-02T09:13:31.424172+0000","last_peered":"2024-04-02T09:13:31.424172+0000","last_clean":"2024-04-02T09:13:31.424172+0000","last_became_active":"2024-04-02T09:13:28.377833+0000","last_became_peered":"2024-04-02T09:13:28.377833+0000","last_unstale":"2024-04-02T09:13:31.424172+0000","last_undegraded":"2024-04-02T09:13:31.424172+0000","last_fullsized":"2024-04-02T09:13:31.424172+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:37:54.526506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426808+0000","last_change":"2024-04-02T09:13:27.880069+0000","last_active":"2024-04-02T09:13:31.426808+0000","last_peered":"2024-04-02T09:13:31.426808+0000","last_clean":"2024-04-02T09:13:31.426808+0000","last_became_active":"2024-04-02T09:13:27.879793+0000","last_became_peered":"2024-04-02T09:13:27.879793+0000","last_unstale":"2024-04-02T09:13:31.426808+0000","last_undegraded":"2024-04-02T09:13:31.426808+0000","last_fullsized":"2024-04-02T09:13:31.426808+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:10.198371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629856+0000","last_change":"2024-04-02T09:13:28.394088+0000","last_active":"2024-04-02T09:13:31.629856+0000","last_peered":"2024-04-02T09:13:31.629856+0000","last_clean":"2024-04-02T09:13:31.629856+0000","last_became_active":"2024-04-02T09:13:28.392932+0000","last_became_peered":"2024-04-02T09:13:28.392932+0000","last_unstale":"2024-04-02T09:13:31.629856+0000","last_undegraded":"2024-04-02T09:13:31.629856+0000","last_fullsized":"2024-04-02T09:13:31.629856+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:54.683721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427550+0000","last_change":"2024-04-02T09:13:27.881714+0000","last_active":"2024-04-02T09:13:31.427550+0000","last_peered":"2024-04-02T09:13:31.427550+0000","last_clean":"2024-04-02T09:13:31.427550+0000","last_became_active":"2024-04-02T09:13:27.881428+0000","last_became_peered":"2024-04-02T09:13:27.881428+0000","last_unstale":"2024-04-02T09:13:31.427550+0000","last_undegraded":"2024-04-02T09:13:31.427550+0000","last_fullsized":"2024-04-02T09:13:31.427550+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:50.081688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422590+0000","last_change":"2024-04-02T09:13:28.391188+0000","last_active":"2024-04-02T09:13:31.422590+0000","last_peered":"2024-04-02T09:13:31.422590+0000","last_clean":"2024-04-02T09:13:31.422590+0000","last_became_active":"2024-04-02T09:13:28.390867+0000","last_became_peered":"2024-04-02T09:13:28.390867+0000","last_unstale":"2024-04-02T09:13:31.422590+0000","last_undegraded":"2024-04-02T09:13:31.422590+0000","last_fullsized":"2024-04-02T09:13:31.422590+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:26:58.531401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.10","version":"243'8","reported_seq":20,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.423981+0000","last_change":"2024-04-02T09:13:27.886503+0000","last_active":"2024-04-02T09:13:31.423981+0000","last_peered":"2024-04-02T09:13:31.423981+0000","last_clean":"2024-04-02T09:13:31.423981+0000","last_became_active":"2024-04-02T09:13:27.886121+0000","last_became_peered":"2024-04-02T09:13:27.886121+0000","last_unstale":"2024-04-02T09:13:31.423981+0000","last_undegraded":"2024-04-02T09:13:31.423981+0000","last_fullsized":"2024-04-02T09:13:31.423981+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:38.533680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630223+0000","last_change":"2024-04-02T09:13:28.376609+0000","last_active":"2024-04-02T09:13:31.630223+0000","last_peered":"2024-04-02T09:13:31.630223+0000","last_clean":"2024-04-02T09:13:31.630223+0000","last_became_active":"2024-04-02T09:13:28.376226+0000","last_became_peered":"2024-04-02T09:13:28.376226+0000","last_unstale":"2024-04-02T09:13:31.630223+0000","last_undegraded":"2024-04-02T09:13:31.630223+0000","last_fullsized":"2024-04-02T09:13:31.630223+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:59.766007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"243'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422817+0000","last_change":"2024-04-02T09:13:27.883080+0000","last_active":"2024-04-02T09:13:31.422817+0000","last_peered":"2024-04-02T09:13:31.422817+0000","last_clean":"2024-04-02T09:13:31.422817+0000","last_became_active":"2024-04-02T09:13:27.882612+0000","last_became_peered":"2024-04-02T09:13:27.882612+0000","last_unstale":"2024-04-02T09:13:31.422817+0000","last_undegraded":"2024-04-02T09:13:31.422817+0000","last_fullsized":"2024-04-02T09:13:31.422817+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:20.979570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424093+0000","last_change":"2024-04-02T09:13:28.381499+0000","last_active":"2024-04-02T09:13:31.424093+0000","last_peered":"2024-04-02T09:13:31.424093+0000","last_clean":"2024-04-02T09:13:31.424093+0000","last_became_active":"2024-04-02T09:13:28.381286+0000","last_became_peered":"2024-04-02T09:13:28.381286+0000","last_unstale":"2024-04-02T09:13:31.424093+0000","last_undegraded":"2024-04-02T09:13:31.424093+0000","last_fullsized":"2024-04-02T09:13:31.424093+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:18.748067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.e","version":"243'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426966+0000","last_change":"2024-04-02T09:13:27.888558+0000","last_active":"2024-04-02T09:13:31.426966+0000","last_peered":"2024-04-02T09:13:31.426966+0000","last_clean":"2024-04-02T09:13:31.426966+0000","last_became_active":"2024-04-02T09:13:27.888288+0000","last_became_peered":"2024-04-02T09:13:27.888288+0000","last_unstale":"2024-04-02T09:13:31.426966+0000","last_undegraded":"2024-04-02T09:13:31.426966+0000","last_fullsized":"2024-04-02T09:13:31.426966+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:15.386238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422721+0000","last_change":"2024-04-02T09:13:28.372849+0000","last_active":"2024-04-02T09:13:31.422721+0000","last_peered":"2024-04-02T09:13:31.422721+0000","last_clean":"2024-04-02T09:13:31.422721+0000","last_became_active":"2024-04-02T09:13:28.372595+0000","last_became_peered":"2024-04-02T09:13:28.372595+0000","last_unstale":"2024-04-02T09:13:31.422721+0000","last_undegraded":"2024-04-02T09:13:31.422721+0000","last_fullsized":"2024-04-02T09:13:31.422721+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:12.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.d","version":"243'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422739+0000","last_change":"2024-04-02T09:13:27.879417+0000","last_active":"2024-04-02T09:13:31.422739+0000","last_peered":"2024-04-02T09:13:31.422739+0000","last_clean":"2024-04-02T09:13:31.422739+0000","last_became_active":"2024-04-02T09:13:27.879165+0000","last_became_peered":"2024-04-02T09:13:27.879165+0000","last_unstale":"2024-04-02T09:13:31.422739+0000","last_undegraded":"2024-04-02T09:13:31.422739+0000","last_fullsized":"2024-04-02T09:13:31.422739+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:47:32.233662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629634+0000","last_change":"2024-04-02T09:13:28.392130+0000","last_active":"2024-04-02T09:13:31.629634+0000","last_peered":"2024-04-02T09:13:31.629634+0000","last_clean":"2024-04-02T09:13:31.629634+0000","last_became_active":"2024-04-02T09:13:28.391690+0000","last_became_peered":"2024-04-02T09:13:28.391690+0000","last_unstale":"2024-04-02T09:13:31.629634+0000","last_undegraded":"2024-04-02T09:13:31.629634+0000","last_fullsized":"2024-04-02T09:13:31.629634+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:24.319747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426897+0000","last_change":"2024-04-02T09:13:27.884781+0000","last_active":"2024-04-02T09:13:31.426897+0000","last_peered":"2024-04-02T09:13:31.426897+0000","last_clean":"2024-04-02T09:13:31.426897+0000","last_became_active":"2024-04-02T09:13:27.884507+0000","last_became_peered":"2024-04-02T09:13:27.884507+0000","last_unstale":"2024-04-02T09:13:31.426897+0000","last_undegraded":"2024-04-02T09:13:31.426897+0000","last_fullsized":"2024-04-02T09:13:31.426897+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:31:18.889273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424087+0000","last_change":"2024-04-02T09:13:28.381580+0000","last_active":"2024-04-02T09:13:31.424087+0000","last_peered":"2024-04-02T09:13:31.424087+0000","last_clean":"2024-04-02T09:13:31.424087+0000","last_became_active":"2024-04-02T09:13:28.381300+0000","last_became_peered":"2024-04-02T09:13:28.381300+0000","last_unstale":"2024-04-02T09:13:31.424087+0000","last_undegraded":"2024-04-02T09:13:31.424087+0000","last_fullsized":"2024-04-02T09:13:31.424087+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:51.089333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"243'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426854+0000","last_change":"2024-04-02T09:13:27.875291+0000","last_active":"2024-04-02T09:13:31.426854+0000","last_peered":"2024-04-02T09:13:31.426854+0000","last_clean":"2024-04-02T09:13:31.426854+0000","last_became_active":"2024-04-02T09:13:27.875003+0000","last_became_peered":"2024-04-02T09:13:27.875003+0000","last_unstale":"2024-04-02T09:13:31.426854+0000","last_undegraded":"2024-04-02T09:13:31.426854+0000","last_fullsized":"2024-04-02T09:13:31.426854+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:00.637981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426839+0000","last_change":"2024-04-02T09:13:28.404265+0000","last_active":"2024-04-02T09:13:31.426839+0000","last_peered":"2024-04-02T09:13:31.426839+0000","last_clean":"2024-04-02T09:13:31.426839+0000","last_became_active":"2024-04-02T09:13:28.403894+0000","last_became_peered":"2024-04-02T09:13:28.403894+0000","last_unstale":"2024-04-02T09:13:31.426839+0000","last_undegraded":"2024-04-02T09:13:31.426839+0000","last_fullsized":"2024-04-02T09:13:31.426839+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:05:23.433426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426819+0000","last_change":"2024-04-02T09:13:27.881604+0000","last_active":"2024-04-02T09:13:31.426819+0000","last_peered":"2024-04-02T09:13:31.426819+0000","last_clean":"2024-04-02T09:13:31.426819+0000","last_became_active":"2024-04-02T09:13:27.881304+0000","last_became_peered":"2024-04-02T09:13:27.881304+0000","last_unstale":"2024-04-02T09:13:31.426819+0000","last_undegraded":"2024-04-02T09:13:31.426819+0000","last_fullsized":"2024-04-02T09:13:31.426819+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:02.497118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629870+0000","last_change":"2024-04-02T09:13:28.374960+0000","last_active":"2024-04-02T09:13:31.629870+0000","last_peered":"2024-04-02T09:13:31.629870+0000","last_clean":"2024-04-02T09:13:31.629870+0000","last_became_active":"2024-04-02T09:13:28.373683+0000","last_became_peered":"2024-04-02T09:13:28.373683+0000","last_unstale":"2024-04-02T09:13:31.629870+0000","last_undegraded":"2024-04-02T09:13:31.629870+0000","last_fullsized":"2024-04-02T09:13:31.629870+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:15.851301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629524+0000","last_change":"2024-04-02T09:13:27.877837+0000","last_active":"2024-04-02T09:13:31.629524+0000","last_peered":"2024-04-02T09:13:31.629524+0000","last_clean":"2024-04-02T09:13:31.629524+0000","last_became_active":"2024-04-02T09:13:27.876792+0000","last_became_peered":"2024-04-02T09:13:27.876792+0000","last_unstale":"2024-04-02T09:13:31.629524+0000","last_undegraded":"2024-04-02T09:13:31.629524+0000","last_fullsized":"2024-04-02T09:13:31.629524+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:17.740422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427528+0000","last_change":"2024-04-02T09:13:28.379243+0000","last_active":"2024-04-02T09:13:31.427528+0000","last_peered":"2024-04-02T09:13:31.427528+0000","last_clean":"2024-04-02T09:13:31.427528+0000","last_became_active":"2024-04-02T09:13:28.378297+0000","last_became_peered":"2024-04-02T09:13:28.378297+0000","last_unstale":"2024-04-02T09:13:31.427528+0000","last_undegraded":"2024-04-02T09:13:31.427528+0000","last_fullsized":"2024-04-02T09:13:31.427528+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:02.761190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629474+0000","last_change":"2024-04-02T09:13:27.876051+0000","last_active":"2024-04-02T09:13:31.629474+0000","last_peered":"2024-04-02T09:13:31.629474+0000","last_clean":"2024-04-02T09:13:31.629474+0000","last_became_active":"2024-04-02T09:13:27.875628+0000","last_became_peered":"2024-04-02T09:13:27.875628+0000","last_unstale":"2024-04-02T09:13:31.629474+0000","last_undegraded":"2024-04-02T09:13:31.629474+0000","last_fullsized":"2024-04-02T09:13:31.629474+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:53.750473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422758+0000","last_change":"2024-04-02T09:13:28.383204+0000","last_active":"2024-04-02T09:13:31.422758+0000","last_peered":"2024-04-02T09:13:31.422758+0000","last_clean":"2024-04-02T09:13:31.422758+0000","last_became_active":"2024-04-02T09:13:28.382961+0000","last_became_peered":"2024-04-02T09:13:28.382961+0000","last_unstale":"2024-04-02T09:13:31.422758+0000","last_undegraded":"2024-04-02T09:13:31.422758+0000","last_fullsized":"2024-04-02T09:13:31.422758+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:16:14.265813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.7","version":"243'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426992+0000","last_change":"2024-04-02T09:13:27.886740+0000","last_active":"2024-04-02T09:13:31.426992+0000","last_peered":"2024-04-02T09:13:31.426992+0000","last_clean":"2024-04-02T09:13:31.426992+0000","last_became_active":"2024-04-02T09:13:27.886385+0000","last_became_peered":"2024-04-02T09:13:27.886385+0000","last_unstale":"2024-04-02T09:13:31.426992+0000","last_undegraded":"2024-04-02T09:13:31.426992+0000","last_fullsized":"2024-04-02T09:13:31.426992+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:20.383235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426708+0000","last_change":"2024-04-02T09:13:28.402497+0000","last_active":"2024-04-02T09:13:31.426708+0000","last_peered":"2024-04-02T09:13:31.426708+0000","last_clean":"2024-04-02T09:13:31.426708+0000","last_became_active":"2024-04-02T09:13:28.402235+0000","last_became_peered":"2024-04-02T09:13:28.402235+0000","last_unstale":"2024-04-02T09:13:31.426708+0000","last_undegraded":"2024-04-02T09:13:31.426708+0000","last_fullsized":"2024-04-02T09:13:31.426708+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:15.072346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.6","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:33.349612+0000","last_change":"2024-04-02T09:13:27.884225+0000","last_active":"2024-04-02T09:13:33.349612+0000","last_peered":"2024-04-02T09:13:33.349612+0000","last_clean":"2024-04-02T09:13:33.349612+0000","last_became_active":"2024-04-02T09:13:27.883536+0000","last_became_peered":"2024-04-02T09:13:27.883536+0000","last_unstale":"2024-04-02T09:13:33.349612+0000","last_undegraded":"2024-04-02T09:13:33.349612+0000","last_fullsized":"2024-04-02T09:13:33.349612+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:50.645932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630206+0000","last_change":"2024-04-02T09:13:28.387139+0000","last_active":"2024-04-02T09:13:31.630206+0000","last_peered":"2024-04-02T09:13:31.630206+0000","last_clean":"2024-04-02T09:13:31.630206+0000","last_became_active":"2024-04-02T09:13:28.386878+0000","last_became_peered":"2024-04-02T09:13:28.386878+0000","last_unstale":"2024-04-02T09:13:31.630206+0000","last_undegraded":"2024-04-02T09:13:31.630206+0000","last_fullsized":"2024-04-02T09:13:31.630206+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:56:34.681485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427622+0000","last_change":"2024-04-02T09:13:27.882615+0000","last_active":"2024-04-02T09:13:31.427622+0000","last_peered":"2024-04-02T09:13:31.427622+0000","last_clean":"2024-04-02T09:13:31.427622+0000","last_became_active":"2024-04-02T09:13:27.882334+0000","last_became_peered":"2024-04-02T09:13:27.882334+0000","last_unstale":"2024-04-02T09:13:31.427622+0000","last_undegraded":"2024-04-02T09:13:31.427622+0000","last_fullsized":"2024-04-02T09:13:31.427622+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:43.215347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426932+0000","last_change":"2024-04-02T09:13:28.402303+0000","last_active":"2024-04-02T09:13:31.426932+0000","last_peered":"2024-04-02T09:13:31.426932+0000","last_clean":"2024-04-02T09:13:31.426932+0000","last_became_active":"2024-04-02T09:13:28.399305+0000","last_became_peered":"2024-04-02T09:13:28.399305+0000","last_unstale":"2024-04-02T09:13:31.426932+0000","last_undegraded":"2024-04-02T09:13:31.426932+0000","last_fullsized":"2024-04-02T09:13:31.426932+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:58.927786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426908+0000","last_change":"2024-04-02T09:13:27.881268+0000","last_active":"2024-04-02T09:13:31.426908+0000","last_peered":"2024-04-02T09:13:31.426908+0000","last_clean":"2024-04-02T09:13:31.426908+0000","last_became_active":"2024-04-02T09:13:27.880983+0000","last_became_peered":"2024-04-02T09:13:27.880983+0000","last_unstale":"2024-04-02T09:13:31.426908+0000","last_undegraded":"2024-04-02T09:13:31.426908+0000","last_fullsized":"2024-04-02T09:13:31.426908+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:09.803095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424125+0000","last_change":"2024-04-02T09:13:28.374285+0000","last_active":"2024-04-02T09:13:31.424125+0000","last_peered":"2024-04-02T09:13:31.424125+0000","last_clean":"2024-04-02T09:13:31.424125+0000","last_became_active":"2024-04-02T09:13:28.373980+0000","last_became_peered":"2024-04-02T09:13:28.373980+0000","last_unstale":"2024-04-02T09:13:31.424125+0000","last_undegraded":"2024-04-02T09:13:31.424125+0000","last_fullsized":"2024-04-02T09:13:31.424125+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:56:34.681485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.422700+0000","last_change":"2024-04-02T09:13:27.882249+0000","last_active":"2024-04-02T09:13:31.422700+0000","last_peered":"2024-04-02T09:13:31.422700+0000","last_clean":"2024-04-02T09:13:31.422700+0000","last_became_active":"2024-04-02T09:13:27.881833+0000","last_became_peered":"2024-04-02T09:13:27.881833+0000","last_unstale":"2024-04-02T09:13:31.422700+0000","last_undegraded":"2024-04-02T09:13:31.422700+0000","last_fullsized":"2024-04-02T09:13:31.422700+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:22.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.427581+0000","last_change":"2024-04-02T09:13:28.389208+0000","last_active":"2024-04-02T09:13:31.427581+0000","last_peered":"2024-04-02T09:13:31.427581+0000","last_clean":"2024-04-02T09:13:31.427581+0000","last_became_active":"2024-04-02T09:13:28.388923+0000","last_became_peered":"2024-04-02T09:13:28.388923+0000","last_unstale":"2024-04-02T09:13:31.427581+0000","last_undegraded":"2024-04-02T09:13:31.427581+0000","last_fullsized":"2024-04-02T09:13:31.427581+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:35:55.299814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.630151+0000","last_change":"2024-04-02T09:13:27.887509+0000","last_active":"2024-04-02T09:13:31.630151+0000","last_peered":"2024-04-02T09:13:31.630151+0000","last_clean":"2024-04-02T09:13:31.630151+0000","last_became_active":"2024-04-02T09:13:27.887260+0000","last_became_peered":"2024-04-02T09:13:27.887260+0000","last_unstale":"2024-04-02T09:13:31.630151+0000","last_undegraded":"2024-04-02T09:13:31.630151+0000","last_fullsized":"2024-04-02T09:13:31.630151+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:28.581007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629486+0000","last_change":"2024-04-02T09:13:28.374292+0000","last_active":"2024-04-02T09:13:31.629486+0000","last_peered":"2024-04-02T09:13:31.629486+0000","last_clean":"2024-04-02T09:13:31.629486+0000","last_became_active":"2024-04-02T09:13:28.373689+0000","last_became_peered":"2024-04-02T09:13:28.373689+0000","last_unstale":"2024-04-02T09:13:31.629486+0000","last_undegraded":"2024-04-02T09:13:31.629486+0000","last_fullsized":"2024-04-02T09:13:31.629486+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:08.953388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426541+0000","last_change":"2024-04-02T09:13:27.882492+0000","last_active":"2024-04-02T09:13:31.426541+0000","last_peered":"2024-04-02T09:13:31.426541+0000","last_clean":"2024-04-02T09:13:31.426541+0000","last_became_active":"2024-04-02T09:13:27.882217+0000","last_became_peered":"2024-04-02T09:13:27.882217+0000","last_unstale":"2024-04-02T09:13:31.426541+0000","last_undegraded":"2024-04-02T09:13:31.426541+0000","last_fullsized":"2024-04-02T09:13:31.426541+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:22:56.505431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426510+0000","last_change":"2024-04-02T09:13:28.381355+0000","last_active":"2024-04-02T09:13:31.426510+0000","last_peered":"2024-04-02T09:13:31.426510+0000","last_clean":"2024-04-02T09:13:31.426510+0000","last_became_active":"2024-04-02T09:13:28.381068+0000","last_became_peered":"2024-04-02T09:13:28.381068+0000","last_unstale":"2024-04-02T09:13:31.426510+0000","last_undegraded":"2024-04-02T09:13:31.426510+0000","last_fullsized":"2024-04-02T09:13:31.426510+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:50.895638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.424004+0000","last_change":"2024-04-02T09:13:27.883425+0000","last_active":"2024-04-02T09:13:31.424004+0000","last_peered":"2024-04-02T09:13:31.424004+0000","last_clean":"2024-04-02T09:13:31.424004+0000","last_became_active":"2024-04-02T09:13:27.883106+0000","last_became_peered":"2024-04-02T09:13:27.883106+0000","last_unstale":"2024-04-02T09:13:31.424004+0000","last_undegraded":"2024-04-02T09:13:31.424004+0000","last_fullsized":"2024-04-02T09:13:31.424004+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:23.271534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:13:41.418587+0000","last_change":"2024-04-02T09:13:41.418587+0000","last_active":"2024-04-02T09:13:41.418587+0000","last_peered":"2024-04-02T09:13:41.418587+0000","last_clean":"2024-04-02T09:13:41.418587+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:13:41.418587+0000","last_undegraded":"2024-04-02T09:13:41.418587+0000","last_fullsized":"2024-04-02T09:13:41.418587+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":11,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.629742+0000","last_change":"2024-04-02T09:13:28.402819+0000","last_active":"2024-04-02T09:13:31.629742+0000","last_peered":"2024-04-02T09:13:31.629742+0000","last_clean":"2024-04-02T09:13:31.629742+0000","last_became_active":"2024-04-02T09:13:28.402563+0000","last_became_peered":"2024-04-02T09:13:28.402563+0000","last_unstale":"2024-04-02T09:13:31.629742+0000","last_undegraded":"2024-04-02T09:13:31.629742+0000","last_fullsized":"2024-04-02T09:13:31.629742+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:27.049697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1f","version":"243'4","reported_seq":16,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.426728+0000","last_change":"2024-04-02T09:13:27.883417+0000","last_active":"2024-04-02T09:13:31.426728+0000","last_peered":"2024-04-02T09:13:31.426728+0000","last_clean":"2024-04-02T09:13:31.426728+0000","last_became_active":"2024-04-02T09:13:27.882944+0000","last_became_peered":"2024-04-02T09:13:27.882944+0000","last_unstale":"2024-04-02T09:13:31.426728+0000","last_undegraded":"2024-04-02T09:13:31.426728+0000","last_fullsized":"2024-04-02T09:13:31.426728+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:05.331734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"229'370","reported_seq":1309,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-04-02T09:13:31.030879+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:13:31.030879+0000","last_peered":"2024-04-02T09:13:31.030879+0000","last_clean":"2024-04-02T09:13:31.030879+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:13:31.030879+0000","last_undegraded":"2024-04-02T09:13:31.030879+0000","last_fullsized":"2024-04-02T09:13:31.030879+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3603,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576000,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60388,"kb_used_data":3032,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311452,"statfs":{"total":96636764160,"available":96574926848,"internally_reserved":0,"allocated":3104768,"data_stored":1580386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.40400000000000003,"5min":0.309,"15min":0.309},"max":{"1min":8.016,"5min":8.016,"15min":8.016},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.71999999999999997,"15min":0.72999999999999998},"min":{"1min":0.39900000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.1790000000000003,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.81999999999999995}]},{"osd":1,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.76800000000000002,"15min":0.751},"min":{"1min":0.44800000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.5819999999999999,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.70899999999999996,"15min":0.71999999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.42399999999999999},"max":{"1min":4.2569999999999997,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.48999999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.79600000000000004,"15min":0.77800000000000002},"min":{"1min":0.437,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.9790000000000001,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.80700000000000005,"15min":0.81200000000000006},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":4.8819999999999997,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.66900000000000004}]},{"osd":3,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.77100000000000002,"15min":0.75900000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.0530000000000008,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.999,"5min":0.79900000000000004,"15min":0.77700000000000002},"min":{"1min":0.54100000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":5.093,"5min":5.093,"15min":5.093},"last":0.78300000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.66700000000000004,"15min":0.628},"min":{"1min":0.38300000000000001,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":5.4589999999999996,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.1060000000000001,"5min":0.79700000000000004,"15min":0.74399999999999999},"min":{"1min":0.50600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.9409999999999998,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.79900000000000004}]},{"osd":5,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.73799999999999999,"15min":0.69299999999999995},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":5.1340000000000003,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.748},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.77400000000000002,"15min":0.73599999999999999},"min":{"1min":0.58899999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.492,"5min":3.492,"15min":3.492},"last":0.76600000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.72199999999999998,"15min":0.69699999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45800000000000002},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.075,"5min":0.78500000000000003,"15min":0.71699999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41099999999999998},"max":{"1min":7.8899999999999997,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.84899999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575998,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42324,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329516,"statfs":{"total":96636764160,"available":96593424384,"internally_reserved":0,"allocated":3088384,"data_stored":1577055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.69799999999999995,"15min":0.72899999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.90600000000000003,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.71099999999999997,"15min":0.73899999999999999},"min":{"1min":0.53600000000000003,"5min":0.47899999999999998,"15min":0.45800000000000002},"max":{"1min":0.88300000000000001,"5min":0.93200000000000005,"15min":1.0600000000000001},"last":0.70799999999999996}]},{"osd":1,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68799999999999994,"15min":0.71299999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86299999999999999,"5min":1.044,"15min":1.212},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.71299999999999997,"15min":0.752},"min":{"1min":0.45800000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.018,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.52100000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.71799999999999997,"15min":0.747},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.046,"5min":1.046,"15min":1.2789999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.71999999999999997,"15min":0.73899999999999999},"min":{"1min":0.44,"5min":0.439,"15min":0.439},"max":{"1min":0.93500000000000005,"5min":0.95999999999999996,"15min":1.226},"last":0.48899999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72399999999999998,"15min":0.76300000000000001},"min":{"1min":0.53900000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97799999999999998,"5min":1.141,"15min":1.2549999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.755,"15min":0.77000000000000002},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.67500000000000004}]},{"osd":4,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.60999999999999999,"15min":0.624},"min":{"1min":0.442,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.91200000000000003,"5min":0.96699999999999997,"15min":1.1259999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.41999999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.1879999999999999},"last":0.77200000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84199999999999997,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69999999999999996,"15min":0.72499999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":0.96399999999999997,"5min":1.034,"15min":1.204},"last":0.72499999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.72999999999999998,"15min":0.74099999999999999},"min":{"1min":0.53600000000000003,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.012,"5min":1.21,"15min":1.2370000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72599999999999998,"15min":0.73899999999999999},"min":{"1min":0.52400000000000002,"5min":0.51400000000000001,"15min":0.49399999999999999},"max":{"1min":0.91900000000000004,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.68799999999999994}]}]},{"osd":1,"up_from":13,"seq":55834575999,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52760,"kb_used_data":3596,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319080,"statfs":{"total":96636764160,"available":96582737920,"internally_reserved":0,"allocated":3682304,"data_stored":2167915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.433,"15min":0.45000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.58999999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48999999999999999,"15min":0.50900000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73199999999999998,"5min":0.76000000000000001,"15min":0.79800000000000004},"last":0.51400000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.504,"15min":0.505},"min":{"1min":0.38900000000000001,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56699999999999995,"15min":0.59199999999999997},"min":{"1min":0.44500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68100000000000005,"5min":0.77500000000000002,"15min":0.85599999999999998},"last":0.59499999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.41099999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.69999999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.55200000000000005,"15min":0.56100000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.58899999999999997,"5min":0.86699999999999999,"15min":0.878},"last":0.56399999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76900000000000002,"15min":0.78400000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.433},"max":{"1min":0.90700000000000003,"5min":1.1120000000000001,"15min":1.1479999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.76400000000000001,"15min":0.78500000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.41799999999999998},"max":{"1min":0.95699999999999996,"5min":1.05,"15min":1.25},"last":0.58099999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76100000000000001,"15min":0.78800000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.96399999999999997,"5min":1.115,"15min":1.321},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.79700000000000004,"15min":0.80700000000000005},"min":{"1min":0.56899999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93799999999999994,"5min":1.0980000000000001,"15min":1.173},"last":0.63800000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.79100000000000004,"15min":0.81299999999999994},"min":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.48099999999999998},"max":{"1min":0.96099999999999997,"5min":1.159,"15min":1.2210000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80800000000000005,"15min":0.80900000000000005},"min":{"1min":0.56799999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0009999999999999,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.69099999999999995}]},{"osd":7,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.79900000000000004,"15min":0.81999999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0029999999999999,"5min":1.1080000000000001,"15min":1.236},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81899999999999995,"15min":0.83399999999999996},"min":{"1min":0.60899999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":0.97499999999999998,"5min":1.0720000000000001,"15min":1.2789999999999999},"last":0.68600000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575999,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52180,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319660,"statfs":{"total":96636764160,"available":96583331840,"internally_reserved":0,"allocated":3088384,"data_stored":1576038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.441,"15min":0.45500000000000002},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.66700000000000004,"5min":0.69899999999999995,"15min":0.86699999999999999},"last":0.498},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.49399999999999999,"15min":0.50700000000000001},"min":{"1min":0.33900000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64800000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.53100000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.502,"15min":0.51100000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.82699999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53400000000000003,"15min":0.55500000000000005},"min":{"1min":0.38300000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72699999999999998,"5min":0.999,"15min":0.999},"last":0.63300000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54200000000000004,"15min":0.56699999999999995},"min":{"1min":0.35699999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.71899999999999997,"5min":0.79800000000000004,"15min":0.85699999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.61599999999999999,"15min":0.63800000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.73599999999999999,"5min":0.92000000000000004,"15min":0.96399999999999997},"last":0.65500000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.80500000000000005,"15min":0.83099999999999996},"min":{"1min":0.55000000000000004,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":0.97499999999999998,"5min":1.1950000000000001,"15min":1.2769999999999999},"last":0.745},{"interface":"front","average":{"1min":0.751,"5min":0.78500000000000003,"15min":0.82199999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99299999999999999,"5min":1.169,"15min":1.169},"last":0.873}]},{"osd":5,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.77900000000000003,"15min":0.82399999999999995},"min":{"1min":0.54000000000000004,"5min":0.497,"15min":0.497},"max":{"1min":1,"5min":1.145,"15min":1.232},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.96599999999999997,"5min":1.21,"15min":1.21},"last":0.90600000000000003}]},{"osd":6,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.77600000000000002,"15min":0.79800000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91100000000000003,"5min":1.1830000000000001,"15min":1.196},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.79200000000000004,"15min":0.81200000000000006},"min":{"1min":0.56499999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96199999999999997,"5min":1.246,"15min":1.246},"last":0.96799999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82699999999999996,"15min":0.85499999999999998},"min":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0069999999999999,"5min":1.292,"15min":1.292},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.84799999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.95499999999999996,"5min":1.1719999999999999,"15min":1.2589999999999999},"last":0.92300000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575998,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52180,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319660,"statfs":{"total":96636764160,"available":96583331840,"internally_reserved":0,"allocated":3088384,"data_stored":1577035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.44600000000000001,"15min":0.44},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.72999999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.48399999999999999,"15min":0.498},"min":{"1min":0.29199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":1.4930000000000001},"last":0.34000000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.504},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69699999999999995,"5min":0.75800000000000001,"15min":0.80600000000000005},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53600000000000003,"15min":0.55400000000000005},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.59599999999999997,"5min":0.75,"15min":1.4530000000000001},"last":0.41799999999999998}]},{"osd":3,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52900000000000003,"15min":0.55700000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.745,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60499999999999998,"15min":0.63200000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.64000000000000001,"5min":0.83699999999999997,"15min":1.0740000000000001},"last":0.51400000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.82099999999999995,"5min":1.121,"15min":1.121},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.77300000000000002,"15min":0.79100000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.96799999999999997,"5min":1.179,"15min":1.5329999999999999},"last":0.874}]},{"osd":5,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.78400000000000003,"15min":0.78600000000000003},"min":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.82999999999999996,"5min":1.99,"15min":1.99},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.78500000000000003,"15min":0.79900000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.89600000000000002,"5min":1.101,"15min":1.1259999999999999},"last":0.79100000000000004}]},{"osd":6,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.997,"5min":1.173,"15min":1.5940000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":0.80400000000000005},"min":{"1min":0.56200000000000006,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.92300000000000004,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.89800000000000002}]},{"osd":7,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.877,"5min":1.2110000000000001,"15min":1.5649999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.752,"5min":0.80700000000000005,"15min":0.82699999999999996},"min":{"1min":0.61499999999999999,"5min":0.497,"15min":0.497},"max":{"1min":0.95899999999999996,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.749}]}]},{"osd":3,"up_from":13,"seq":55834575998,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52188,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319652,"statfs":{"total":96636764160,"available":96583323648,"internally_reserved":0,"allocated":3096576,"data_stored":1579770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.42799999999999999,"15min":0.44600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57199999999999995,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.46899999999999997,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60199999999999998,"5min":0.71199999999999997,"15min":0.876},"last":0.52100000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48699999999999999,"15min":0.499},"min":{"1min":0.36799999999999999,"5min":0.33200000000000002,"15min":0.32500000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.90300000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55100000000000005,"15min":0.57199999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67700000000000005,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.55300000000000005}]},{"osd":2,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.60099999999999998,"15min":0.626},"min":{"1min":0.40100000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71399999999999997,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.628},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.58799999999999997,"15min":0.61599999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.69499999999999995,"5min":0.81699999999999995,"15min":1.0840000000000001},"last":0.60399999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.72299999999999998,"15min":0.752},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89100000000000001,"5min":1.071,"15min":1.0800000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.74299999999999999,"15min":0.76800000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.879,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.74399999999999999,"15min":0.77500000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91800000000000004,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.76300000000000001,"15min":0.78100000000000003},"min":{"1min":0.54200000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.85699999999999998,"5min":1.266,"15min":1.266},"last":0.59499999999999997}]},{"osd":6,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.76500000000000001,"15min":0.78400000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97099999999999997,"5min":1.284,"15min":1.284},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.54800000000000004,"5min":0.495,"15min":0.495},"max":{"1min":0.95999999999999996,"5min":1.2,"15min":1.2},"last":0.67500000000000004}]},{"osd":7,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.77900000000000003,"15min":0.80800000000000005},"min":{"1min":0.58699999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.93500000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.79300000000000004,"15min":0.81799999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.014,"5min":1.248,"15min":1.248},"last":0.72899999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575998,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48084,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323756,"statfs":{"total":96636764160,"available":96587526144,"internally_reserved":0,"allocated":3088384,"data_stored":1576987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.66400000000000003,"15min":0.68700000000000006},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":0.80000000000000004,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.432},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.65100000000000002,"15min":0.67700000000000005},"min":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.78100000000000003,"5min":0.86199999999999999,"15min":1.075},"last":0.66100000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.72199999999999998,"15min":0.754},"min":{"1min":0.49399999999999999,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":0.90500000000000003,"5min":1.0549999999999999,"15min":1.069},"last":0.63},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.73599999999999999,"15min":0.76500000000000001},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":0.92500000000000004,"5min":1.048,"15min":1.048},"last":0.49099999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.67300000000000004,"15min":0.70799999999999996},"min":{"1min":0.47099999999999997,"5min":0.438,"15min":0.438},"max":{"1min":0.81299999999999994,"5min":1.077,"15min":1.077},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.68999999999999995,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85599999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.58899999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.51500000000000001,"5min":0.501,"15min":0.47399999999999998},"max":{"1min":0.94599999999999995,"5min":0.998,"15min":1.018},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.028},"last":0.60899999999999999}]},{"osd":5,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.81899999999999995,"5min":1.008,"15min":1.008},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.50600000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65500000000000003,"15min":0.64400000000000002},"min":{"1min":0.48499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85099999999999998,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68999999999999995,"15min":0.69199999999999995},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":1.0049999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.623}]},{"osd":7,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.48399999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96099999999999997,"5min":1.002,"15min":1.0329999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.77500000000000002,"15min":0.78100000000000003},"min":{"1min":0.56100000000000005,"5min":0.5,"15min":0.5},"max":{"1min":1.038,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.70599999999999996}]}]},{"osd":5,"up_from":13,"seq":55834576001,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52760,"kb_used_data":3596,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319080,"statfs":{"total":96636764160,"available":96582737920,"internally_reserved":0,"allocated":3682304,"data_stored":2166438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69699999999999995,"15min":0.70999999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.89600000000000002,"5min":0.98699999999999999,"15min":1.004},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69899999999999995,"15min":0.72499999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.625}]},{"osd":1,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.70099999999999996,"15min":0.72199999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85999999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.71399999999999997,"15min":0.73199999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.052},"last":0.88500000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73199999999999998,"15min":0.745},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93200000000000005,"5min":1.004,"15min":1.014},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72599999999999998,"15min":0.746},"min":{"1min":0.46600000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.018,"5min":1.018,"15min":1.0680000000000001},"last":0.69499999999999995}]},{"osd":3,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77000000000000002,"15min":0.78600000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":0.98899999999999999,"15min":1.083},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.59099999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94399999999999995,"5min":1.008,"15min":1.1020000000000001},"last":0.81899999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.629,"15min":0.629},"min":{"1min":0.56499999999999995,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.89100000000000001,"5min":0.96899999999999997,"15min":1.054},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.68400000000000005,"15min":0.67500000000000004},"min":{"1min":0.47999999999999998,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.68300000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.434,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.751},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71199999999999997,"15min":0.71099999999999997},"min":{"1min":0.501,"5min":0.44600000000000001,"15min":0.42699999999999999},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.94899999999999995}]},{"osd":7,"last update":"Tue Apr 2 09:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.73999999999999999,"15min":0.73499999999999999},"min":{"1min":0.60699999999999998,"5min":0.441,"15min":0.441},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.64000000000000001,"5min":0.48199999999999998,"15min":0.45700000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.84599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:13:47.310 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:13:47.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.538+0000 7f2870a00700 1 Processor -- start 2024-04-02T09:13:47.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.538+0000 7f2870a00700 1 -- start start 2024-04-02T09:13:47.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.539+0000 7f2870a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f286c15b5d0 0x7f286c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:47.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.539+0000 7f2870a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f286c0cb3f0 0x7f286c0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:47.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.539+0000 7f2870a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286c15fd50 con 0x7f286c0ae930 2024-04-02T09:13:47.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.540+0000 7f2870a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286c15b9b0 con 0x7f286c15b5d0 2024-04-02T09:13:47.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.540+0000 7f2870a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286c15f140 con 0x7f286c0cb3f0 2024-04-02T09:13:47.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.540+0000 7f2869e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f286c15b5d0 0x7f286c0ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:47.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.540+0000 7f2869e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f286c15b5d0 0x7f286c0ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55340/0 (socket says 172.21.15.89:55340) 2024-04-02T09:13:47.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.540+0000 7f2869e00700 1 -- 172.21.15.89:0/2630677379 learned_addr learned my addr 172.21.15.89:0/2630677379 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:47.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.540+0000 7f2869e00700 1 -- 172.21.15.89:0/2630677379 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f286c0cb3f0 msgr2=0x7f286c0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:47.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.540+0000 7f2869400700 1 --2- 172.21.15.89:0/2630677379 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f286c0cb3f0 0x7f286c0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:47.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.540+0000 7f2869e00700 1 --2- 172.21.15.89:0/2630677379 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f286c0cb3f0 0x7f286c0ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.540+0000 7f2869e00700 1 -- 172.21.15.89:0/2630677379 >> v1:172.21.15.113:6789/0 conn(0x7f286c0ae930 legacy=0x7f286c140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:47.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.540+0000 7f2869e00700 1 -- 172.21.15.89:0/2630677379 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c15f420 con 0x7f286c15b5d0 2024-04-02T09:13:47.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.540+0000 7f2869400700 1 --2- 172.21.15.89:0/2630677379 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f286c0cb3f0 0x7f286c0ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:47.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.541+0000 7f2869e00700 1 --2- 172.21.15.89:0/2630677379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f286c15b5d0 0x7f286c0ae3f0 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f285c0098f0 tx=0x7f285c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86779dac1444ed31 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:47.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.541+0000 7f2868a00700 1 -- 172.21.15.89:0/2630677379 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c004230 con 0x7f286c15b5d0 2024-04-02T09:13:47.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.541+0000 7f2868a00700 1 -- 172.21.15.89:0/2630677379 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285c004390 con 0x7f286c15b5d0 2024-04-02T09:13:47.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.541+0000 7f2868a00700 1 -- 172.21.15.89:0/2630677379 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c0036a0 con 0x7f286c15b5d0 2024-04-02T09:13:47.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.541+0000 7f2870a00700 1 -- 172.21.15.89:0/2630677379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f286c15b5d0 msgr2=0x7f286c0ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:47.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.541+0000 7f2870a00700 1 --2- 172.21.15.89:0/2630677379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f286c15b5d0 0x7f286c0ae3f0 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f285c0098f0 tx=0x7f285c009c00 comp rx=0 tx=0).stop 2024-04-02T09:13:47.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.541+0000 7f2870a00700 1 -- 172.21.15.89:0/2630677379 shutdown_connections 2024-04-02T09:13:47.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.541+0000 7f2870a00700 1 --2- 172.21.15.89:0/2630677379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f286c15b5d0 0x7f286c0ae3f0 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.541+0000 7f2870a00700 1 --2- 172.21.15.89:0/2630677379 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f286c0cb3f0 0x7f286c0ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.541+0000 7f2870a00700 1 -- 172.21.15.89:0/2630677379 >> 172.21.15.89:0/2630677379 conn(0x7f286c0cf190 msgr2=0x7f286c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:47.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.541+0000 7f2870a00700 1 -- 172.21.15.89:0/2630677379 shutdown_connections 2024-04-02T09:13:47.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.542+0000 7f2870a00700 1 -- 172.21.15.89:0/2630677379 wait complete. 2024-04-02T09:13:47.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.544+0000 7f2870a00700 1 Processor -- start 2024-04-02T09:13:47.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.544+0000 7f2870a00700 1 -- start start 2024-04-02T09:13:47.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.545+0000 7f2870a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f286c0ae930 0x7f286c156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:47.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.545+0000 7f2870a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f286c15b5d0 0x7f286c0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:47.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.545+0000 7f2870a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f286c15d000 0x7f286c0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:47.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.545+0000 7f2870a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c156cc0 con 0x7f286c15d000 2024-04-02T09:13:47.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.545+0000 7f2870a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c0ccd40 con 0x7f286c0ae930 2024-04-02T09:13:47.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.545+0000 7f2870a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286c0cc4a0 con 0x7f286c15b5d0 2024-04-02T09:13:47.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.545+0000 7f286a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f286c15d000 0x7f286c0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:47.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.545+0000 7f286a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f286c15d000 0x7f286c0ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55344/0 (socket says 172.21.15.89:55344) 2024-04-02T09:13:47.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.545+0000 7f286a800700 1 -- 172.21.15.89:0/1591301505 learned_addr learned my addr 172.21.15.89:0/1591301505 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:47.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.546+0000 7f2869e00700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f286c0ae930 0x7f286c156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:47.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.546+0000 7f2869400700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f286c15b5d0 0x7f286c0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:47.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.546+0000 7f286a800700 1 -- 172.21.15.89:0/1591301505 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f286c15b5d0 msgr2=0x7f286c0cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:47.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.546+0000 7f286a800700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f286c15b5d0 0x7f286c0cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.546+0000 7f286a800700 1 -- 172.21.15.89:0/1591301505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f286c0ae930 msgr2=0x7f286c156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:47.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.546+0000 7f286a800700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f286c0ae930 0x7f286c156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.546+0000 7f286a800700 1 -- 172.21.15.89:0/1591301505 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c009550 con 0x7f286c15d000 2024-04-02T09:13:47.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.546+0000 7f2869e00700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f286c0ae930 0x7f286c156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:47.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.546+0000 7f286a800700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f286c15d000 0x7f286c0ac0c0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f286400cc10 tx=0x7f2864004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:47.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.546+0000 7f2862000700 1 -- 172.21.15.89:0/1591301505 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286400ab80 con 0x7f286c15d000 2024-04-02T09:13:47.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.546+0000 7f2862000700 1 -- 172.21.15.89:0/1591301505 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286400ace0 con 0x7f286c15d000 2024-04-02T09:13:47.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.546+0000 7f2862000700 1 -- 172.21.15.89:0/1591301505 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864012660 con 0x7f286c15d000 2024-04-02T09:13:47.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.546+0000 7f2870a00700 1 -- 172.21.15.89:0/1591301505 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286c0cc780 con 0x7f286c15d000 2024-04-02T09:13:47.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.546+0000 7f2870a00700 1 -- 172.21.15.89:0/1591301505 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286c1587a0 con 0x7f286c15d000 2024-04-02T09:13:47.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.547+0000 7f2870a00700 1 -- 172.21.15.89:0/1591301505 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286c158600 con 0x7f286c15d000 2024-04-02T09:13:47.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.548+0000 7f2862000700 1 -- 172.21.15.89:0/1591301505 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f28640127c0 con 0x7f286c15d000 2024-04-02T09:13:47.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.549+0000 7f2862000700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f284c06aed0 0x7f284c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:47.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.549+0000 7f2862000700 1 -- 172.21.15.89:0/1591301505 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f286408a440 con 0x7f286c15d000 2024-04-02T09:13:47.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.549+0000 7f2869e00700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f284c06aed0 0x7f284c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:47.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.550+0000 7f2869e00700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f284c06aed0 0x7f284c06d390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f285c009130 tx=0x7f285c008030 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:47.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.554+0000 7f2862000700 1 -- 172.21.15.89:0/1591301505 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f286c158600 con 0x7f286c15d000 2024-04-02T09:13:47.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.750+0000 7f2870a00700 1 -- 172.21.15.89:0/1591301505 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f286c15f9e0 con 0x7f284c06aed0 2024-04-02T09:13:47.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.765+0000 7f2862000700 1 -- 172.21.15.89:0/1591301505 <== mgr.11613 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528603 (secure 0 0 0) 0x7f285c020070 con 0x7f284c06aed0 2024-04-02T09:13:47.770 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:47.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.771+0000 7f2870a00700 1 -- 172.21.15.89:0/1591301505 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f284c06aed0 msgr2=0x7f284c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:47.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.771+0000 7f2870a00700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f284c06aed0 0x7f284c06d390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f285c009130 tx=0x7f285c008030 comp rx=0 tx=0).stop 2024-04-02T09:13:47.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.771+0000 7f2870a00700 1 -- 172.21.15.89:0/1591301505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f286c15d000 msgr2=0x7f286c0ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:47.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.771+0000 7f2870a00700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f286c15d000 0x7f286c0ac0c0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f286400cc10 tx=0x7f2864004990 comp rx=0 tx=0).stop 2024-04-02T09:13:47.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.771+0000 7f2870a00700 1 -- 172.21.15.89:0/1591301505 shutdown_connections 2024-04-02T09:13:47.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.771+0000 7f2870a00700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f284c06aed0 0x7f284c06d390 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.771+0000 7f2870a00700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f286c0ae930 0x7f286c156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.771+0000 7f2870a00700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f286c15d000 0x7f286c0ac0c0 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.771+0000 7f2870a00700 1 --2- 172.21.15.89:0/1591301505 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f286c15b5d0 0x7f286c0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:47.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.771+0000 7f2870a00700 1 -- 172.21.15.89:0/1591301505 >> 172.21.15.89:0/1591301505 conn(0x7f286c0cf190 msgr2=0x7f286c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:47.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.771+0000 7f2870a00700 1 -- 172.21.15.89:0/1591301505 shutdown_connections 2024-04-02T09:13:47.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:47.771+0000 7f2870a00700 1 -- 172.21.15.89:0/1591301505 wait complete. 2024-04-02T09:13:47.776 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:13:47.787 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-04-02T09:13:47.432749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596333,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":711,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":450,"ondisk_log_size":450,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":361,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413236,"kb_used_data":25664,"kb_used_omap":12,"kb_used_meta":387443,"kb_avail":754561484,"statfs":{"total":773094113280,"available":772670959616,"internally_reserved":0,"allocated":26279936,"data_stored":13992684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":396742200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001855"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.947418+0000","last_change":"2024-04-02T09:13:41.944375+0000","last_active":"2024-04-02T09:13:42.947418+0000","last_peered":"2024-04-02T09:13:42.947418+0000","last_clean":"2024-04-02T09:13:42.947418+0000","last_became_active":"2024-04-02T09:13:41.944110+0000","last_became_peered":"2024-04-02T09:13:41.944110+0000","last_unstale":"2024-04-02T09:13:42.947418+0000","last_undegraded":"2024-04-02T09:13:42.947418+0000","last_fullsized":"2024-04-02T09:13:42.947418+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:02:20.460847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942652+0000","last_change":"2024-04-02T09:13:41.458005+0000","last_active":"2024-04-02T09:13:41.942652+0000","last_peered":"2024-04-02T09:13:41.942652+0000","last_clean":"2024-04-02T09:13:41.942652+0000","last_became_active":"2024-04-02T09:13:41.457753+0000","last_became_peered":"2024-04-02T09:13:41.457753+0000","last_unstale":"2024-04-02T09:13:41.942652+0000","last_undegraded":"2024-04-02T09:13:41.942652+0000","last_fullsized":"2024-04-02T09:13:41.942652+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:12:22.769587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942756+0000","last_change":"2024-04-02T09:13:28.396642+0000","last_active":"2024-04-02T09:13:41.942756+0000","last_peered":"2024-04-02T09:13:41.942756+0000","last_clean":"2024-04-02T09:13:41.942756+0000","last_became_active":"2024-04-02T09:13:28.396356+0000","last_became_peered":"2024-04-02T09:13:28.396356+0000","last_unstale":"2024-04-02T09:13:41.942756+0000","last_undegraded":"2024-04-02T09:13:41.942756+0000","last_fullsized":"2024-04-02T09:13:41.942756+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:34.295480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345466+0000","last_change":"2024-04-02T09:13:41.961471+0000","last_active":"2024-04-02T09:13:43.345466+0000","last_peered":"2024-04-02T09:13:43.345466+0000","last_clean":"2024-04-02T09:13:43.345466+0000","last_became_active":"2024-04-02T09:13:41.961181+0000","last_became_peered":"2024-04-02T09:13:41.961181+0000","last_unstale":"2024-04-02T09:13:43.345466+0000","last_undegraded":"2024-04-02T09:13:43.345466+0000","last_fullsized":"2024-04-02T09:13:43.345466+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:13.220603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952545+0000","last_change":"2024-04-02T09:13:41.454077+0000","last_active":"2024-04-02T09:13:41.952545+0000","last_peered":"2024-04-02T09:13:41.952545+0000","last_clean":"2024-04-02T09:13:41.952545+0000","last_became_active":"2024-04-02T09:13:41.453804+0000","last_became_peered":"2024-04-02T09:13:41.453804+0000","last_unstale":"2024-04-02T09:13:41.952545+0000","last_undegraded":"2024-04-02T09:13:41.952545+0000","last_fullsized":"2024-04-02T09:13:41.952545+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:24:28.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.961240+0000","last_change":"2024-04-02T09:13:28.381100+0000","last_active":"2024-04-02T09:13:41.961240+0000","last_peered":"2024-04-02T09:13:41.961240+0000","last_clean":"2024-04-02T09:13:41.961240+0000","last_became_active":"2024-04-02T09:13:28.380832+0000","last_became_peered":"2024-04-02T09:13:28.380832+0000","last_unstale":"2024-04-02T09:13:41.961240+0000","last_undegraded":"2024-04-02T09:13:41.961240+0000","last_fullsized":"2024-04-02T09:13:41.961240+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:21.987215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734898+0000","last_change":"2024-04-02T09:13:41.970419+0000","last_active":"2024-04-02T09:13:43.734898+0000","last_peered":"2024-04-02T09:13:43.734898+0000","last_clean":"2024-04-02T09:13:43.734898+0000","last_became_active":"2024-04-02T09:13:41.965663+0000","last_became_peered":"2024-04-02T09:13:41.965663+0000","last_unstale":"2024-04-02T09:13:43.734898+0000","last_undegraded":"2024-04-02T09:13:43.734898+0000","last_fullsized":"2024-04-02T09:13:43.734898+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:28:51.025221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942697+0000","last_change":"2024-04-02T09:13:41.457188+0000","last_active":"2024-04-02T09:13:41.942697+0000","last_peered":"2024-04-02T09:13:41.942697+0000","last_clean":"2024-04-02T09:13:41.942697+0000","last_became_active":"2024-04-02T09:13:41.456916+0000","last_became_peered":"2024-04-02T09:13:41.456916+0000","last_unstale":"2024-04-02T09:13:41.942697+0000","last_undegraded":"2024-04-02T09:13:41.942697+0000","last_fullsized":"2024-04-02T09:13:41.942697+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:31:39.320935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.941485+0000","last_change":"2024-04-02T09:13:28.393942+0000","last_active":"2024-04-02T09:13:41.941485+0000","last_peered":"2024-04-02T09:13:41.941485+0000","last_clean":"2024-04-02T09:13:41.941485+0000","last_became_active":"2024-04-02T09:13:28.393467+0000","last_became_peered":"2024-04-02T09:13:28.393467+0000","last_unstale":"2024-04-02T09:13:41.941485+0000","last_undegraded":"2024-04-02T09:13:41.941485+0000","last_fullsized":"2024-04-02T09:13:41.941485+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:24.836500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952847+0000","last_change":"2024-04-02T09:13:41.961739+0000","last_active":"2024-04-02T09:13:42.952847+0000","last_peered":"2024-04-02T09:13:42.952847+0000","last_clean":"2024-04-02T09:13:42.952847+0000","last_became_active":"2024-04-02T09:13:41.961468+0000","last_became_peered":"2024-04-02T09:13:41.961468+0000","last_unstale":"2024-04-02T09:13:42.952847+0000","last_undegraded":"2024-04-02T09:13:42.952847+0000","last_fullsized":"2024-04-02T09:13:42.952847+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:33:35.636916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940817+0000","last_change":"2024-04-02T09:13:41.452325+0000","last_active":"2024-04-02T09:13:41.940817+0000","last_peered":"2024-04-02T09:13:41.940817+0000","last_clean":"2024-04-02T09:13:41.940817+0000","last_became_active":"2024-04-02T09:13:41.449714+0000","last_became_peered":"2024-04-02T09:13:41.449714+0000","last_unstale":"2024-04-02T09:13:41.940817+0000","last_undegraded":"2024-04-02T09:13:41.940817+0000","last_fullsized":"2024-04-02T09:13:41.940817+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:13:02.182890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945254+0000","last_change":"2024-04-02T09:13:28.394098+0000","last_active":"2024-04-02T09:13:41.945254+0000","last_peered":"2024-04-02T09:13:41.945254+0000","last_clean":"2024-04-02T09:13:41.945254+0000","last_became_active":"2024-04-02T09:13:28.393837+0000","last_became_peered":"2024-04-02T09:13:28.393837+0000","last_unstale":"2024-04-02T09:13:41.945254+0000","last_undegraded":"2024-04-02T09:13:41.945254+0000","last_fullsized":"2024-04-02T09:13:41.945254+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:40:27.068896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.948026+0000","last_change":"2024-04-02T09:13:41.951117+0000","last_active":"2024-04-02T09:13:42.948026+0000","last_peered":"2024-04-02T09:13:42.948026+0000","last_clean":"2024-04-02T09:13:42.948026+0000","last_became_active":"2024-04-02T09:13:41.948760+0000","last_became_peered":"2024-04-02T09:13:41.948760+0000","last_unstale":"2024-04-02T09:13:42.948026+0000","last_undegraded":"2024-04-02T09:13:42.948026+0000","last_fullsized":"2024-04-02T09:13:42.948026+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:53:09.361699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944874+0000","last_change":"2024-04-02T09:13:41.458809+0000","last_active":"2024-04-02T09:13:41.944874+0000","last_peered":"2024-04-02T09:13:41.944874+0000","last_clean":"2024-04-02T09:13:41.944874+0000","last_became_active":"2024-04-02T09:13:41.458392+0000","last_became_peered":"2024-04-02T09:13:41.458392+0000","last_unstale":"2024-04-02T09:13:41.944874+0000","last_undegraded":"2024-04-02T09:13:41.944874+0000","last_fullsized":"2024-04-02T09:13:41.944874+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:53.327540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947921+0000","last_change":"2024-04-02T09:13:28.379239+0000","last_active":"2024-04-02T09:13:41.947921+0000","last_peered":"2024-04-02T09:13:41.947921+0000","last_clean":"2024-04-02T09:13:41.947921+0000","last_became_active":"2024-04-02T09:13:28.378970+0000","last_became_peered":"2024-04-02T09:13:28.378970+0000","last_unstale":"2024-04-02T09:13:41.947921+0000","last_undegraded":"2024-04-02T09:13:41.947921+0000","last_fullsized":"2024-04-02T09:13:41.947921+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:37.960638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342554+0000","last_change":"2024-04-02T09:13:41.959102+0000","last_active":"2024-04-02T09:13:43.342554+0000","last_peered":"2024-04-02T09:13:43.342554+0000","last_clean":"2024-04-02T09:13:43.342554+0000","last_became_active":"2024-04-02T09:13:41.956524+0000","last_became_peered":"2024-04-02T09:13:41.956524+0000","last_unstale":"2024-04-02T09:13:43.342554+0000","last_undegraded":"2024-04-02T09:13:43.342554+0000","last_fullsized":"2024-04-02T09:13:43.342554+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:47.857858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946223+0000","last_change":"2024-04-02T09:13:41.451612+0000","last_active":"2024-04-02T09:13:41.946223+0000","last_peered":"2024-04-02T09:13:41.946223+0000","last_clean":"2024-04-02T09:13:41.946223+0000","last_became_active":"2024-04-02T09:13:41.451089+0000","last_became_peered":"2024-04-02T09:13:41.451089+0000","last_unstale":"2024-04-02T09:13:41.946223+0000","last_undegraded":"2024-04-02T09:13:41.946223+0000","last_fullsized":"2024-04-02T09:13:41.946223+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:18.892470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.939756+0000","last_change":"2024-04-02T09:13:28.393747+0000","last_active":"2024-04-02T09:13:41.939756+0000","last_peered":"2024-04-02T09:13:41.939756+0000","last_clean":"2024-04-02T09:13:41.939756+0000","last_became_active":"2024-04-02T09:13:28.393515+0000","last_became_peered":"2024-04-02T09:13:28.393515+0000","last_unstale":"2024-04-02T09:13:41.939756+0000","last_undegraded":"2024-04-02T09:13:41.939756+0000","last_fullsized":"2024-04-02T09:13:41.939756+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:28:37.462844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954473+0000","last_change":"2024-04-02T09:13:41.948810+0000","last_active":"2024-04-02T09:13:42.954473+0000","last_peered":"2024-04-02T09:13:42.954473+0000","last_clean":"2024-04-02T09:13:42.954473+0000","last_became_active":"2024-04-02T09:13:41.948523+0000","last_became_peered":"2024-04-02T09:13:41.948523+0000","last_unstale":"2024-04-02T09:13:42.954473+0000","last_undegraded":"2024-04-02T09:13:42.954473+0000","last_fullsized":"2024-04-02T09:13:42.954473+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:54.336827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.6","version":"248'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342742+0000","last_change":"2024-04-02T09:13:41.460002+0000","last_active":"2024-04-02T09:13:43.342742+0000","last_peered":"2024-04-02T09:13:43.342742+0000","last_clean":"2024-04-02T09:13:43.342742+0000","last_became_active":"2024-04-02T09:13:41.459491+0000","last_became_peered":"2024-04-02T09:13:41.459491+0000","last_unstale":"2024-04-02T09:13:43.342742+0000","last_undegraded":"2024-04-02T09:13:43.342742+0000","last_fullsized":"2024-04-02T09:13:43.342742+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:53.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.938200+0000","last_change":"2024-04-02T09:13:28.399975+0000","last_active":"2024-04-02T09:13:41.938200+0000","last_peered":"2024-04-02T09:13:41.938200+0000","last_clean":"2024-04-02T09:13:41.938200+0000","last_became_active":"2024-04-02T09:13:28.399335+0000","last_became_peered":"2024-04-02T09:13:28.399335+0000","last_unstale":"2024-04-02T09:13:41.938200+0000","last_undegraded":"2024-04-02T09:13:41.938200+0000","last_fullsized":"2024-04-02T09:13:41.938200+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:16:14.265813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954211+0000","last_change":"2024-04-02T09:13:41.954505+0000","last_active":"2024-04-02T09:13:42.954211+0000","last_peered":"2024-04-02T09:13:42.954211+0000","last_clean":"2024-04-02T09:13:42.954211+0000","last_became_active":"2024-04-02T09:13:41.952682+0000","last_became_peered":"2024-04-02T09:13:41.952682+0000","last_unstale":"2024-04-02T09:13:42.954211+0000","last_undegraded":"2024-04-02T09:13:42.954211+0000","last_fullsized":"2024-04-02T09:13:42.954211+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:52.343662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.7","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.971184+0000","last_change":"2024-04-02T09:13:41.459550+0000","last_active":"2024-04-02T09:13:41.971184+0000","last_peered":"2024-04-02T09:13:41.971184+0000","last_clean":"2024-04-02T09:13:41.971184+0000","last_became_active":"2024-04-02T09:13:41.459249+0000","last_became_peered":"2024-04-02T09:13:41.459249+0000","last_unstale":"2024-04-02T09:13:41.971184+0000","last_undegraded":"2024-04-02T09:13:41.971184+0000","last_fullsized":"2024-04-02T09:13:41.971184+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:40.922210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947893+0000","last_change":"2024-04-02T09:13:28.392551+0000","last_active":"2024-04-02T09:13:41.947893+0000","last_peered":"2024-04-02T09:13:41.947893+0000","last_clean":"2024-04-02T09:13:41.947893+0000","last_became_active":"2024-04-02T09:13:28.392159+0000","last_became_peered":"2024-04-02T09:13:28.392159+0000","last_unstale":"2024-04-02T09:13:41.947893+0000","last_undegraded":"2024-04-02T09:13:41.947893+0000","last_fullsized":"2024-04-02T09:13:41.947893+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:42.551696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952740+0000","last_change":"2024-04-02T09:13:41.951315+0000","last_active":"2024-04-02T09:13:42.952740+0000","last_peered":"2024-04-02T09:13:42.952740+0000","last_clean":"2024-04-02T09:13:42.952740+0000","last_became_active":"2024-04-02T09:13:41.950989+0000","last_became_peered":"2024-04-02T09:13:41.950989+0000","last_unstale":"2024-04-02T09:13:42.952740+0000","last_undegraded":"2024-04-02T09:13:42.952740+0000","last_fullsized":"2024-04-02T09:13:42.952740+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:09.529216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944222+0000","last_change":"2024-04-02T09:13:41.451519+0000","last_active":"2024-04-02T09:13:41.944222+0000","last_peered":"2024-04-02T09:13:41.944222+0000","last_clean":"2024-04-02T09:13:41.944222+0000","last_became_active":"2024-04-02T09:13:41.450207+0000","last_became_peered":"2024-04-02T09:13:41.450207+0000","last_unstale":"2024-04-02T09:13:41.944222+0000","last_undegraded":"2024-04-02T09:13:41.944222+0000","last_fullsized":"2024-04-02T09:13:41.944222+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:08.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950200+0000","last_change":"2024-04-02T09:13:28.383891+0000","last_active":"2024-04-02T09:13:41.950200+0000","last_peered":"2024-04-02T09:13:41.950200+0000","last_clean":"2024-04-02T09:13:41.950200+0000","last_became_active":"2024-04-02T09:13:28.383628+0000","last_became_peered":"2024-04-02T09:13:28.383628+0000","last_unstale":"2024-04-02T09:13:41.950200+0000","last_undegraded":"2024-04-02T09:13:41.950200+0000","last_fullsized":"2024-04-02T09:13:41.950200+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:44.222993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734669+0000","last_change":"2024-04-02T09:13:41.969003+0000","last_active":"2024-04-02T09:13:43.734669+0000","last_peered":"2024-04-02T09:13:43.734669+0000","last_clean":"2024-04-02T09:13:43.734669+0000","last_became_active":"2024-04-02T09:13:41.968108+0000","last_became_peered":"2024-04-02T09:13:41.968108+0000","last_unstale":"2024-04-02T09:13:43.734669+0000","last_undegraded":"2024-04-02T09:13:43.734669+0000","last_fullsized":"2024-04-02T09:13:43.734669+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:11.075454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945467+0000","last_change":"2024-04-02T09:13:41.450917+0000","last_active":"2024-04-02T09:13:41.945467+0000","last_peered":"2024-04-02T09:13:41.945467+0000","last_clean":"2024-04-02T09:13:41.945467+0000","last_became_active":"2024-04-02T09:13:41.450645+0000","last_became_peered":"2024-04-02T09:13:41.450645+0000","last_unstale":"2024-04-02T09:13:41.945467+0000","last_undegraded":"2024-04-02T09:13:41.945467+0000","last_fullsized":"2024-04-02T09:13:41.945467+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:10.066167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940531+0000","last_change":"2024-04-02T09:13:28.404390+0000","last_active":"2024-04-02T09:13:41.940531+0000","last_peered":"2024-04-02T09:13:41.940531+0000","last_clean":"2024-04-02T09:13:41.940531+0000","last_became_active":"2024-04-02T09:13:28.404038+0000","last_became_peered":"2024-04-02T09:13:28.404038+0000","last_unstale":"2024-04-02T09:13:41.940531+0000","last_undegraded":"2024-04-02T09:13:41.940531+0000","last_fullsized":"2024-04-02T09:13:41.940531+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:47.359176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.735085+0000","last_change":"2024-04-02T09:13:41.968705+0000","last_active":"2024-04-02T09:13:43.735085+0000","last_peered":"2024-04-02T09:13:43.735085+0000","last_clean":"2024-04-02T09:13:43.735085+0000","last_became_active":"2024-04-02T09:13:41.968079+0000","last_became_peered":"2024-04-02T09:13:41.968079+0000","last_unstale":"2024-04-02T09:13:43.735085+0000","last_undegraded":"2024-04-02T09:13:43.735085+0000","last_fullsized":"2024-04-02T09:13:43.735085+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:27:04.458016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944274+0000","last_change":"2024-04-02T09:13:41.451223+0000","last_active":"2024-04-02T09:13:41.944274+0000","last_peered":"2024-04-02T09:13:41.944274+0000","last_clean":"2024-04-02T09:13:41.944274+0000","last_became_active":"2024-04-02T09:13:41.449328+0000","last_became_peered":"2024-04-02T09:13:41.449328+0000","last_unstale":"2024-04-02T09:13:41.944274+0000","last_undegraded":"2024-04-02T09:13:41.944274+0000","last_fullsized":"2024-04-02T09:13:41.944274+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:50:01.664013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946964+0000","last_change":"2024-04-02T09:13:28.403517+0000","last_active":"2024-04-02T09:13:41.946964+0000","last_peered":"2024-04-02T09:13:41.946964+0000","last_clean":"2024-04-02T09:13:41.946964+0000","last_became_active":"2024-04-02T09:13:28.401353+0000","last_became_peered":"2024-04-02T09:13:28.401353+0000","last_unstale":"2024-04-02T09:13:41.946964+0000","last_undegraded":"2024-04-02T09:13:41.946964+0000","last_fullsized":"2024-04-02T09:13:41.946964+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:08.980542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345809+0000","last_change":"2024-04-02T09:13:41.957583+0000","last_active":"2024-04-02T09:13:43.345809+0000","last_peered":"2024-04-02T09:13:43.345809+0000","last_clean":"2024-04-02T09:13:43.345809+0000","last_became_active":"2024-04-02T09:13:41.953951+0000","last_became_peered":"2024-04-02T09:13:41.953951+0000","last_unstale":"2024-04-02T09:13:43.345809+0000","last_undegraded":"2024-04-02T09:13:43.345809+0000","last_fullsized":"2024-04-02T09:13:43.345809+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:14:22.720024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.b","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.969813+0000","last_change":"2024-04-02T09:13:41.446786+0000","last_active":"2024-04-02T09:13:41.969813+0000","last_peered":"2024-04-02T09:13:41.969813+0000","last_clean":"2024-04-02T09:13:41.969813+0000","last_became_active":"2024-04-02T09:13:41.446482+0000","last_became_peered":"2024-04-02T09:13:41.446482+0000","last_unstale":"2024-04-02T09:13:41.969813+0000","last_undegraded":"2024-04-02T09:13:41.969813+0000","last_fullsized":"2024-04-02T09:13:41.969813+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:08.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.938581+0000","last_change":"2024-04-02T09:13:28.399938+0000","last_active":"2024-04-02T09:13:41.938581+0000","last_peered":"2024-04-02T09:13:41.938581+0000","last_clean":"2024-04-02T09:13:41.938581+0000","last_became_active":"2024-04-02T09:13:28.399350+0000","last_became_peered":"2024-04-02T09:13:28.399350+0000","last_unstale":"2024-04-02T09:13:41.938581+0000","last_undegraded":"2024-04-02T09:13:41.938581+0000","last_fullsized":"2024-04-02T09:13:41.938581+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:05.933699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734569+0000","last_change":"2024-04-02T09:13:41.959689+0000","last_active":"2024-04-02T09:13:43.734569+0000","last_peered":"2024-04-02T09:13:43.734569+0000","last_clean":"2024-04-02T09:13:43.734569+0000","last_became_active":"2024-04-02T09:13:41.959238+0000","last_became_peered":"2024-04-02T09:13:41.959238+0000","last_unstale":"2024-04-02T09:13:43.734569+0000","last_undegraded":"2024-04-02T09:13:43.734569+0000","last_fullsized":"2024-04-02T09:13:43.734569+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:45:09.502243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953945+0000","last_change":"2024-04-02T09:13:41.450544+0000","last_active":"2024-04-02T09:13:41.953945+0000","last_peered":"2024-04-02T09:13:41.953945+0000","last_clean":"2024-04-02T09:13:41.953945+0000","last_became_active":"2024-04-02T09:13:41.450233+0000","last_became_peered":"2024-04-02T09:13:41.450233+0000","last_unstale":"2024-04-02T09:13:41.953945+0000","last_undegraded":"2024-04-02T09:13:41.953945+0000","last_fullsized":"2024-04-02T09:13:41.953945+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:53.842095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950527+0000","last_change":"2024-04-02T09:13:28.386688+0000","last_active":"2024-04-02T09:13:41.950527+0000","last_peered":"2024-04-02T09:13:41.950527+0000","last_clean":"2024-04-02T09:13:41.950527+0000","last_became_active":"2024-04-02T09:13:28.386363+0000","last_became_peered":"2024-04-02T09:13:28.386363+0000","last_unstale":"2024-04-02T09:13:41.950527+0000","last_undegraded":"2024-04-02T09:13:41.950527+0000","last_fullsized":"2024-04-02T09:13:41.950527+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:12:41.480404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954148+0000","last_change":"2024-04-02T09:13:41.952898+0000","last_active":"2024-04-02T09:13:42.954148+0000","last_peered":"2024-04-02T09:13:42.954148+0000","last_clean":"2024-04-02T09:13:42.954148+0000","last_became_active":"2024-04-02T09:13:41.952619+0000","last_became_peered":"2024-04-02T09:13:41.952619+0000","last_unstale":"2024-04-02T09:13:42.954148+0000","last_undegraded":"2024-04-02T09:13:42.954148+0000","last_fullsized":"2024-04-02T09:13:42.954148+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:14:22.720024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.d","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.966777+0000","last_change":"2024-04-02T09:13:41.455641+0000","last_active":"2024-04-02T09:13:41.966777+0000","last_peered":"2024-04-02T09:13:41.966777+0000","last_clean":"2024-04-02T09:13:41.966777+0000","last_became_active":"2024-04-02T09:13:41.455348+0000","last_became_peered":"2024-04-02T09:13:41.455348+0000","last_unstale":"2024-04-02T09:13:41.966777+0000","last_undegraded":"2024-04-02T09:13:41.966777+0000","last_fullsized":"2024-04-02T09:13:41.966777+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:22:20.170595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942920+0000","last_change":"2024-04-02T09:13:28.401903+0000","last_active":"2024-04-02T09:13:41.942920+0000","last_peered":"2024-04-02T09:13:41.942920+0000","last_clean":"2024-04-02T09:13:41.942920+0000","last_became_active":"2024-04-02T09:13:28.399098+0000","last_became_peered":"2024-04-02T09:13:28.399098+0000","last_unstale":"2024-04-02T09:13:41.942920+0000","last_undegraded":"2024-04-02T09:13:41.942920+0000","last_fullsized":"2024-04-02T09:13:41.942920+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:33:56.420978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734685+0000","last_change":"2024-04-02T09:13:41.957617+0000","last_active":"2024-04-02T09:13:43.734685+0000","last_peered":"2024-04-02T09:13:43.734685+0000","last_clean":"2024-04-02T09:13:43.734685+0000","last_became_active":"2024-04-02T09:13:41.957374+0000","last_became_peered":"2024-04-02T09:13:41.957374+0000","last_unstale":"2024-04-02T09:13:43.734685+0000","last_undegraded":"2024-04-02T09:13:43.734685+0000","last_fullsized":"2024-04-02T09:13:43.734685+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:02:20.460847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.e","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.974578+0000","last_change":"2024-04-02T09:13:41.456993+0000","last_active":"2024-04-02T09:13:41.974578+0000","last_peered":"2024-04-02T09:13:41.974578+0000","last_clean":"2024-04-02T09:13:41.974578+0000","last_became_active":"2024-04-02T09:13:41.456687+0000","last_became_peered":"2024-04-02T09:13:41.456687+0000","last_unstale":"2024-04-02T09:13:41.974578+0000","last_undegraded":"2024-04-02T09:13:41.974578+0000","last_fullsized":"2024-04-02T09:13:41.974578+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:53.688559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944668+0000","last_change":"2024-04-02T09:13:28.383146+0000","last_active":"2024-04-02T09:13:41.944668+0000","last_peered":"2024-04-02T09:13:41.944668+0000","last_clean":"2024-04-02T09:13:41.944668+0000","last_became_active":"2024-04-02T09:13:28.382879+0000","last_became_peered":"2024-04-02T09:13:28.382879+0000","last_unstale":"2024-04-02T09:13:41.944668+0000","last_undegraded":"2024-04-02T09:13:41.944668+0000","last_fullsized":"2024-04-02T09:13:41.944668+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:57:59.273299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.735294+0000","last_change":"2024-04-02T09:13:41.963375+0000","last_active":"2024-04-02T09:13:43.735294+0000","last_peered":"2024-04-02T09:13:43.735294+0000","last_clean":"2024-04-02T09:13:43.735294+0000","last_became_active":"2024-04-02T09:13:41.961332+0000","last_became_peered":"2024-04-02T09:13:41.961332+0000","last_unstale":"2024-04-02T09:13:43.735294+0000","last_undegraded":"2024-04-02T09:13:43.735294+0000","last_fullsized":"2024-04-02T09:13:43.735294+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:38.398878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.f","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.967015+0000","last_change":"2024-04-02T09:13:41.450114+0000","last_active":"2024-04-02T09:13:41.967015+0000","last_peered":"2024-04-02T09:13:41.967015+0000","last_clean":"2024-04-02T09:13:41.967015+0000","last_became_active":"2024-04-02T09:13:41.449847+0000","last_became_peered":"2024-04-02T09:13:41.449847+0000","last_unstale":"2024-04-02T09:13:41.967015+0000","last_undegraded":"2024-04-02T09:13:41.967015+0000","last_fullsized":"2024-04-02T09:13:41.967015+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:56.693535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953001+0000","last_change":"2024-04-02T09:13:28.381499+0000","last_active":"2024-04-02T09:13:41.953001+0000","last_peered":"2024-04-02T09:13:41.953001+0000","last_clean":"2024-04-02T09:13:41.953001+0000","last_became_active":"2024-04-02T09:13:28.377983+0000","last_became_peered":"2024-04-02T09:13:28.377983+0000","last_unstale":"2024-04-02T09:13:41.953001+0000","last_undegraded":"2024-04-02T09:13:41.953001+0000","last_fullsized":"2024-04-02T09:13:41.953001+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:45.021173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734898+0000","last_change":"2024-04-02T09:13:41.958180+0000","last_active":"2024-04-02T09:13:43.734898+0000","last_peered":"2024-04-02T09:13:43.734898+0000","last_clean":"2024-04-02T09:13:43.734898+0000","last_became_active":"2024-04-02T09:13:41.957880+0000","last_became_peered":"2024-04-02T09:13:41.957880+0000","last_unstale":"2024-04-02T09:13:43.734898+0000","last_undegraded":"2024-04-02T09:13:43.734898+0000","last_fullsized":"2024-04-02T09:13:43.734898+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:28.634723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.10","version":"247'8","reported_seq":19,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.975332+0000","last_change":"2024-04-02T09:13:41.453631+0000","last_active":"2024-04-02T09:13:41.975332+0000","last_peered":"2024-04-02T09:13:41.975332+0000","last_clean":"2024-04-02T09:13:41.975332+0000","last_became_active":"2024-04-02T09:13:41.453353+0000","last_became_peered":"2024-04-02T09:13:41.453353+0000","last_unstale":"2024-04-02T09:13:41.975332+0000","last_undegraded":"2024-04-02T09:13:41.975332+0000","last_fullsized":"2024-04-02T09:13:41.975332+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:14.301601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940637+0000","last_change":"2024-04-02T09:13:28.382899+0000","last_active":"2024-04-02T09:13:41.940637+0000","last_peered":"2024-04-02T09:13:41.940637+0000","last_clean":"2024-04-02T09:13:41.940637+0000","last_became_active":"2024-04-02T09:13:28.382023+0000","last_became_peered":"2024-04-02T09:13:28.382023+0000","last_unstale":"2024-04-02T09:13:41.940637+0000","last_undegraded":"2024-04-02T09:13:41.940637+0000","last_fullsized":"2024-04-02T09:13:41.940637+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:12.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345329+0000","last_change":"2024-04-02T09:13:41.972132+0000","last_active":"2024-04-02T09:13:43.345329+0000","last_peered":"2024-04-02T09:13:43.345329+0000","last_clean":"2024-04-02T09:13:43.345329+0000","last_became_active":"2024-04-02T09:13:41.965773+0000","last_became_peered":"2024-04-02T09:13:41.965773+0000","last_unstale":"2024-04-02T09:13:43.345329+0000","last_undegraded":"2024-04-02T09:13:43.345329+0000","last_fullsized":"2024-04-02T09:13:43.345329+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:38.686528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.949402+0000","last_change":"2024-04-02T09:13:41.445504+0000","last_active":"2024-04-02T09:13:41.949402+0000","last_peered":"2024-04-02T09:13:41.949402+0000","last_clean":"2024-04-02T09:13:41.949402+0000","last_became_active":"2024-04-02T09:13:41.445211+0000","last_became_peered":"2024-04-02T09:13:41.445211+0000","last_unstale":"2024-04-02T09:13:41.949402+0000","last_undegraded":"2024-04-02T09:13:41.949402+0000","last_fullsized":"2024-04-02T09:13:41.949402+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:40.922210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.955279+0000","last_change":"2024-04-02T09:13:28.390594+0000","last_active":"2024-04-02T09:13:41.955279+0000","last_peered":"2024-04-02T09:13:41.955279+0000","last_clean":"2024-04-02T09:13:41.955279+0000","last_became_active":"2024-04-02T09:13:28.388061+0000","last_became_peered":"2024-04-02T09:13:28.388061+0000","last_unstale":"2024-04-02T09:13:41.955279+0000","last_undegraded":"2024-04-02T09:13:41.955279+0000","last_fullsized":"2024-04-02T09:13:41.955279+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:14:04.841237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345714+0000","last_change":"2024-04-02T09:13:41.966357+0000","last_active":"2024-04-02T09:13:43.345714+0000","last_peered":"2024-04-02T09:13:43.345714+0000","last_clean":"2024-04-02T09:13:43.345714+0000","last_became_active":"2024-04-02T09:13:41.961810+0000","last_became_peered":"2024-04-02T09:13:41.961810+0000","last_unstale":"2024-04-02T09:13:43.345714+0000","last_undegraded":"2024-04-02T09:13:43.345714+0000","last_fullsized":"2024-04-02T09:13:43.345714+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:37.458648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953956+0000","last_change":"2024-04-02T09:13:41.457989+0000","last_active":"2024-04-02T09:13:41.953956+0000","last_peered":"2024-04-02T09:13:41.953956+0000","last_clean":"2024-04-02T09:13:41.953956+0000","last_became_active":"2024-04-02T09:13:41.456512+0000","last_became_peered":"2024-04-02T09:13:41.456512+0000","last_unstale":"2024-04-02T09:13:41.953956+0000","last_undegraded":"2024-04-02T09:13:41.953956+0000","last_fullsized":"2024-04-02T09:13:41.953956+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:56.693535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950481+0000","last_change":"2024-04-02T09:13:28.382883+0000","last_active":"2024-04-02T09:13:41.950481+0000","last_peered":"2024-04-02T09:13:41.950481+0000","last_clean":"2024-04-02T09:13:41.950481+0000","last_became_active":"2024-04-02T09:13:28.382187+0000","last_became_peered":"2024-04-02T09:13:28.382187+0000","last_unstale":"2024-04-02T09:13:41.950481+0000","last_undegraded":"2024-04-02T09:13:41.950481+0000","last_fullsized":"2024-04-02T09:13:41.950481+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:21:26.362248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734816+0000","last_change":"2024-04-02T09:13:41.961588+0000","last_active":"2024-04-02T09:13:43.734816+0000","last_peered":"2024-04-02T09:13:43.734816+0000","last_clean":"2024-04-02T09:13:43.734816+0000","last_became_active":"2024-04-02T09:13:41.958669+0000","last_became_peered":"2024-04-02T09:13:41.958669+0000","last_unstale":"2024-04-02T09:13:43.734816+0000","last_undegraded":"2024-04-02T09:13:43.734816+0000","last_fullsized":"2024-04-02T09:13:43.734816+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:46.027233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.13","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.971712+0000","last_change":"2024-04-02T09:13:41.450452+0000","last_active":"2024-04-02T09:13:41.971712+0000","last_peered":"2024-04-02T09:13:41.971712+0000","last_clean":"2024-04-02T09:13:41.971712+0000","last_became_active":"2024-04-02T09:13:41.449005+0000","last_became_peered":"2024-04-02T09:13:41.449005+0000","last_unstale":"2024-04-02T09:13:41.971712+0000","last_undegraded":"2024-04-02T09:13:41.971712+0000","last_fullsized":"2024-04-02T09:13:41.971712+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:19.429201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.961389+0000","last_change":"2024-04-02T09:13:28.384295+0000","last_active":"2024-04-02T09:13:41.961389+0000","last_peered":"2024-04-02T09:13:41.961389+0000","last_clean":"2024-04-02T09:13:41.961389+0000","last_became_active":"2024-04-02T09:13:28.384010+0000","last_became_peered":"2024-04-02T09:13:28.384010+0000","last_unstale":"2024-04-02T09:13:41.961389+0000","last_undegraded":"2024-04-02T09:13:41.961389+0000","last_fullsized":"2024-04-02T09:13:41.961389+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:47:33.241308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.735273+0000","last_change":"2024-04-02T09:13:41.965796+0000","last_active":"2024-04-02T09:13:43.735273+0000","last_peered":"2024-04-02T09:13:43.735273+0000","last_clean":"2024-04-02T09:13:43.735273+0000","last_became_active":"2024-04-02T09:13:41.965262+0000","last_became_peered":"2024-04-02T09:13:41.965262+0000","last_unstale":"2024-04-02T09:13:43.735273+0000","last_undegraded":"2024-04-02T09:13:43.735273+0000","last_fullsized":"2024-04-02T09:13:43.735273+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:47.857858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.14","version":"247'6","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.971032+0000","last_change":"2024-04-02T09:13:41.456890+0000","last_active":"2024-04-02T09:13:41.971032+0000","last_peered":"2024-04-02T09:13:41.971032+0000","last_clean":"2024-04-02T09:13:41.971032+0000","last_became_active":"2024-04-02T09:13:41.453059+0000","last_became_peered":"2024-04-02T09:13:41.453059+0000","last_unstale":"2024-04-02T09:13:41.971032+0000","last_undegraded":"2024-04-02T09:13:41.971032+0000","last_fullsized":"2024-04-02T09:13:41.971032+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:08.249339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948077+0000","last_change":"2024-04-02T09:13:28.380209+0000","last_active":"2024-04-02T09:13:41.948077+0000","last_peered":"2024-04-02T09:13:41.948077+0000","last_clean":"2024-04-02T09:13:41.948077+0000","last_became_active":"2024-04-02T09:13:28.379739+0000","last_became_peered":"2024-04-02T09:13:28.379739+0000","last_unstale":"2024-04-02T09:13:41.948077+0000","last_undegraded":"2024-04-02T09:13:41.948077+0000","last_fullsized":"2024-04-02T09:13:41.948077+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:46:14.163290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.346038+0000","last_change":"2024-04-02T09:13:41.956661+0000","last_active":"2024-04-02T09:13:43.346038+0000","last_peered":"2024-04-02T09:13:43.346038+0000","last_clean":"2024-04-02T09:13:43.346038+0000","last_became_active":"2024-04-02T09:13:41.953910+0000","last_became_peered":"2024-04-02T09:13:41.953910+0000","last_unstale":"2024-04-02T09:13:43.346038+0000","last_undegraded":"2024-04-02T09:13:43.346038+0000","last_fullsized":"2024-04-02T09:13:43.346038+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:20.438488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.15","version":"247'3","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.966676+0000","last_change":"2024-04-02T09:13:41.458001+0000","last_active":"2024-04-02T09:13:41.966676+0000","last_peered":"2024-04-02T09:13:41.966676+0000","last_clean":"2024-04-02T09:13:41.966676+0000","last_became_active":"2024-04-02T09:13:41.457318+0000","last_became_peered":"2024-04-02T09:13:41.457318+0000","last_unstale":"2024-04-02T09:13:41.966676+0000","last_undegraded":"2024-04-02T09:13:41.966676+0000","last_fullsized":"2024-04-02T09:13:41.966676+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:47:51.824824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.949330+0000","last_change":"2024-04-02T09:13:28.392993+0000","last_active":"2024-04-02T09:13:41.949330+0000","last_peered":"2024-04-02T09:13:41.949330+0000","last_clean":"2024-04-02T09:13:41.949330+0000","last_became_active":"2024-04-02T09:13:28.392712+0000","last_became_peered":"2024-04-02T09:13:28.392712+0000","last_unstale":"2024-04-02T09:13:41.949330+0000","last_undegraded":"2024-04-02T09:13:41.949330+0000","last_fullsized":"2024-04-02T09:13:41.949330+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:38.436370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.955195+0000","last_change":"2024-04-02T09:13:41.960213+0000","last_active":"2024-04-02T09:13:42.955195+0000","last_peered":"2024-04-02T09:13:42.955195+0000","last_clean":"2024-04-02T09:13:42.955195+0000","last_became_active":"2024-04-02T09:13:41.959935+0000","last_became_peered":"2024-04-02T09:13:41.959935+0000","last_unstale":"2024-04-02T09:13:42.955195+0000","last_undegraded":"2024-04-02T09:13:42.955195+0000","last_fullsized":"2024-04-02T09:13:42.955195+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:41.505910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.955082+0000","last_change":"2024-04-02T09:13:41.451596+0000","last_active":"2024-04-02T09:13:41.955082+0000","last_peered":"2024-04-02T09:13:41.955082+0000","last_clean":"2024-04-02T09:13:41.955082+0000","last_became_active":"2024-04-02T09:13:41.451329+0000","last_became_peered":"2024-04-02T09:13:41.451329+0000","last_unstale":"2024-04-02T09:13:41.955082+0000","last_undegraded":"2024-04-02T09:13:41.955082+0000","last_fullsized":"2024-04-02T09:13:41.955082+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:48:38.200651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948538+0000","last_change":"2024-04-02T09:13:28.378030+0000","last_active":"2024-04-02T09:13:41.948538+0000","last_peered":"2024-04-02T09:13:41.948538+0000","last_clean":"2024-04-02T09:13:41.948538+0000","last_became_active":"2024-04-02T09:13:28.377700+0000","last_became_peered":"2024-04-02T09:13:28.377700+0000","last_unstale":"2024-04-02T09:13:41.948538+0000","last_undegraded":"2024-04-02T09:13:41.948538+0000","last_fullsized":"2024-04-02T09:13:41.948538+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:40.236322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734926+0000","last_change":"2024-04-02T09:13:41.956307+0000","last_active":"2024-04-02T09:13:43.734926+0000","last_peered":"2024-04-02T09:13:43.734926+0000","last_clean":"2024-04-02T09:13:43.734926+0000","last_became_active":"2024-04-02T09:13:41.955455+0000","last_became_peered":"2024-04-02T09:13:41.955455+0000","last_unstale":"2024-04-02T09:13:43.734926+0000","last_undegraded":"2024-04-02T09:13:43.734926+0000","last_fullsized":"2024-04-02T09:13:43.734926+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:11.075454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942471+0000","last_change":"2024-04-02T09:13:41.455756+0000","last_active":"2024-04-02T09:13:41.942471+0000","last_peered":"2024-04-02T09:13:41.942471+0000","last_clean":"2024-04-02T09:13:41.942471+0000","last_became_active":"2024-04-02T09:13:41.455267+0000","last_became_peered":"2024-04-02T09:13:41.455267+0000","last_unstale":"2024-04-02T09:13:41.942471+0000","last_undegraded":"2024-04-02T09:13:41.942471+0000","last_fullsized":"2024-04-02T09:13:41.942471+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:27.266184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943109+0000","last_change":"2024-04-02T09:13:28.395364+0000","last_active":"2024-04-02T09:13:41.943109+0000","last_peered":"2024-04-02T09:13:41.943109+0000","last_clean":"2024-04-02T09:13:41.943109+0000","last_became_active":"2024-04-02T09:13:28.395100+0000","last_became_peered":"2024-04-02T09:13:28.395100+0000","last_unstale":"2024-04-02T09:13:41.943109+0000","last_undegraded":"2024-04-02T09:13:41.943109+0000","last_fullsized":"2024-04-02T09:13:41.943109+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:12.863676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.733818+0000","last_change":"2024-04-02T09:13:41.956248+0000","last_active":"2024-04-02T09:13:43.733818+0000","last_peered":"2024-04-02T09:13:43.733818+0000","last_clean":"2024-04-02T09:13:43.733818+0000","last_became_active":"2024-04-02T09:13:41.955983+0000","last_became_peered":"2024-04-02T09:13:41.955983+0000","last_unstale":"2024-04-02T09:13:43.733818+0000","last_undegraded":"2024-04-02T09:13:43.733818+0000","last_fullsized":"2024-04-02T09:13:43.733818+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:50:02.673300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946966+0000","last_change":"2024-04-02T09:13:41.442991+0000","last_active":"2024-04-02T09:13:41.946966+0000","last_peered":"2024-04-02T09:13:41.946966+0000","last_clean":"2024-04-02T09:13:41.946966+0000","last_became_active":"2024-04-02T09:13:41.442743+0000","last_became_peered":"2024-04-02T09:13:41.442743+0000","last_unstale":"2024-04-02T09:13:41.946966+0000","last_undegraded":"2024-04-02T09:13:41.946966+0000","last_fullsized":"2024-04-02T09:13:41.946966+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:47:51.824824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947036+0000","last_change":"2024-04-02T09:13:28.399604+0000","last_active":"2024-04-02T09:13:41.947036+0000","last_peered":"2024-04-02T09:13:41.947036+0000","last_clean":"2024-04-02T09:13:41.947036+0000","last_became_active":"2024-04-02T09:13:28.399333+0000","last_became_peered":"2024-04-02T09:13:28.399333+0000","last_unstale":"2024-04-02T09:13:41.947036+0000","last_undegraded":"2024-04-02T09:13:41.947036+0000","last_fullsized":"2024-04-02T09:13:41.947036+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:18.748067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345299+0000","last_change":"2024-04-02T09:13:41.955992+0000","last_active":"2024-04-02T09:13:43.345299+0000","last_peered":"2024-04-02T09:13:43.345299+0000","last_clean":"2024-04-02T09:13:43.345299+0000","last_became_active":"2024-04-02T09:13:41.955696+0000","last_became_peered":"2024-04-02T09:13:41.955696+0000","last_unstale":"2024-04-02T09:13:43.345299+0000","last_undegraded":"2024-04-02T09:13:43.345299+0000","last_fullsized":"2024-04-02T09:13:43.345299+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:50:19.183542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944518+0000","last_change":"2024-04-02T09:13:41.454745+0000","last_active":"2024-04-02T09:13:41.944518+0000","last_peered":"2024-04-02T09:13:41.944518+0000","last_clean":"2024-04-02T09:13:41.944518+0000","last_became_active":"2024-04-02T09:13:41.454416+0000","last_became_peered":"2024-04-02T09:13:41.454416+0000","last_unstale":"2024-04-02T09:13:41.944518+0000","last_undegraded":"2024-04-02T09:13:41.944518+0000","last_fullsized":"2024-04-02T09:13:41.944518+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:19.429201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953068+0000","last_change":"2024-04-02T09:13:28.378770+0000","last_active":"2024-04-02T09:13:41.953068+0000","last_peered":"2024-04-02T09:13:41.953068+0000","last_clean":"2024-04-02T09:13:41.953068+0000","last_became_active":"2024-04-02T09:13:28.378521+0000","last_became_peered":"2024-04-02T09:13:28.378521+0000","last_unstale":"2024-04-02T09:13:41.953068+0000","last_undegraded":"2024-04-02T09:13:41.953068+0000","last_fullsized":"2024-04-02T09:13:41.953068+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:24:11.210144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952812+0000","last_change":"2024-04-02T09:13:41.962211+0000","last_active":"2024-04-02T09:13:42.952812+0000","last_peered":"2024-04-02T09:13:42.952812+0000","last_clean":"2024-04-02T09:13:42.952812+0000","last_became_active":"2024-04-02T09:13:41.961032+0000","last_became_peered":"2024-04-02T09:13:41.961032+0000","last_unstale":"2024-04-02T09:13:42.952812+0000","last_undegraded":"2024-04-02T09:13:42.952812+0000","last_fullsized":"2024-04-02T09:13:42.952812+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:43:41.499934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1a","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.971085+0000","last_change":"2024-04-02T09:13:41.455291+0000","last_active":"2024-04-02T09:13:41.971085+0000","last_peered":"2024-04-02T09:13:41.971085+0000","last_clean":"2024-04-02T09:13:41.971085+0000","last_became_active":"2024-04-02T09:13:41.453960+0000","last_became_peered":"2024-04-02T09:13:41.453960+0000","last_unstale":"2024-04-02T09:13:41.971085+0000","last_undegraded":"2024-04-02T09:13:41.971085+0000","last_fullsized":"2024-04-02T09:13:41.971085+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:45:49.868260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.962039+0000","last_change":"2024-04-02T09:13:28.389084+0000","last_active":"2024-04-02T09:13:41.962039+0000","last_peered":"2024-04-02T09:13:41.962039+0000","last_clean":"2024-04-02T09:13:41.962039+0000","last_became_active":"2024-04-02T09:13:28.388624+0000","last_became_peered":"2024-04-02T09:13:28.388624+0000","last_unstale":"2024-04-02T09:13:41.962039+0000","last_undegraded":"2024-04-02T09:13:41.962039+0000","last_fullsized":"2024-04-02T09:13:41.962039+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:23.766461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952740+0000","last_change":"2024-04-02T09:13:41.954960+0000","last_active":"2024-04-02T09:13:42.952740+0000","last_peered":"2024-04-02T09:13:42.952740+0000","last_clean":"2024-04-02T09:13:42.952740+0000","last_became_active":"2024-04-02T09:13:41.952119+0000","last_became_peered":"2024-04-02T09:13:41.952119+0000","last_unstale":"2024-04-02T09:13:42.952740+0000","last_undegraded":"2024-04-02T09:13:42.952740+0000","last_fullsized":"2024-04-02T09:13:42.952740+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:20.438488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1b","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.964365+0000","last_change":"2024-04-02T09:13:41.455663+0000","last_active":"2024-04-02T09:13:41.964365+0000","last_peered":"2024-04-02T09:13:41.964365+0000","last_clean":"2024-04-02T09:13:41.964365+0000","last_became_active":"2024-04-02T09:13:41.455401+0000","last_became_peered":"2024-04-02T09:13:41.455401+0000","last_unstale":"2024-04-02T09:13:41.964365+0000","last_undegraded":"2024-04-02T09:13:41.964365+0000","last_fullsized":"2024-04-02T09:13:41.964365+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:45.017947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944969+0000","last_change":"2024-04-02T09:13:28.395286+0000","last_active":"2024-04-02T09:13:41.944969+0000","last_peered":"2024-04-02T09:13:41.944969+0000","last_clean":"2024-04-02T09:13:41.944969+0000","last_became_active":"2024-04-02T09:13:28.394377+0000","last_became_peered":"2024-04-02T09:13:28.394377+0000","last_unstale":"2024-04-02T09:13:41.944969+0000","last_undegraded":"2024-04-02T09:13:41.944969+0000","last_fullsized":"2024-04-02T09:13:41.944969+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:37:39.028435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734569+0000","last_change":"2024-04-02T09:13:41.954547+0000","last_active":"2024-04-02T09:13:43.734569+0000","last_peered":"2024-04-02T09:13:43.734569+0000","last_clean":"2024-04-02T09:13:43.734569+0000","last_became_active":"2024-04-02T09:13:41.954306+0000","last_became_peered":"2024-04-02T09:13:41.954306+0000","last_unstale":"2024-04-02T09:13:43.734569+0000","last_undegraded":"2024-04-02T09:13:43.734569+0000","last_fullsized":"2024-04-02T09:13:43.734569+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:53:09.361699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1c","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.974035+0000","last_change":"2024-04-02T09:13:41.456420+0000","last_active":"2024-04-02T09:13:41.974035+0000","last_peered":"2024-04-02T09:13:41.974035+0000","last_clean":"2024-04-02T09:13:41.974035+0000","last_became_active":"2024-04-02T09:13:41.456055+0000","last_became_peered":"2024-04-02T09:13:41.456055+0000","last_unstale":"2024-04-02T09:13:41.974035+0000","last_undegraded":"2024-04-02T09:13:41.974035+0000","last_fullsized":"2024-04-02T09:13:41.974035+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:52.540226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943657+0000","last_change":"2024-04-02T09:13:28.391267+0000","last_active":"2024-04-02T09:13:41.943657+0000","last_peered":"2024-04-02T09:13:41.943657+0000","last_clean":"2024-04-02T09:13:41.943657+0000","last_became_active":"2024-04-02T09:13:28.390985+0000","last_became_peered":"2024-04-02T09:13:28.390985+0000","last_unstale":"2024-04-02T09:13:41.943657+0000","last_undegraded":"2024-04-02T09:13:41.943657+0000","last_fullsized":"2024-04-02T09:13:41.943657+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:30.210394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342493+0000","last_change":"2024-04-02T09:13:41.963616+0000","last_active":"2024-04-02T09:13:43.342493+0000","last_peered":"2024-04-02T09:13:43.342493+0000","last_clean":"2024-04-02T09:13:43.342493+0000","last_became_active":"2024-04-02T09:13:41.963371+0000","last_became_peered":"2024-04-02T09:13:41.963371+0000","last_unstale":"2024-04-02T09:13:43.342493+0000","last_undegraded":"2024-04-02T09:13:43.342493+0000","last_fullsized":"2024-04-02T09:13:43.342493+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:30:47.096602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1d","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.972845+0000","last_change":"2024-04-02T09:13:41.457926+0000","last_active":"2024-04-02T09:13:41.972845+0000","last_peered":"2024-04-02T09:13:41.972845+0000","last_clean":"2024-04-02T09:13:41.972845+0000","last_became_active":"2024-04-02T09:13:41.457081+0000","last_became_peered":"2024-04-02T09:13:41.457081+0000","last_unstale":"2024-04-02T09:13:41.972845+0000","last_undegraded":"2024-04-02T09:13:41.972845+0000","last_fullsized":"2024-04-02T09:13:41.972845+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:23:34.898152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948952+0000","last_change":"2024-04-02T09:13:28.380650+0000","last_active":"2024-04-02T09:13:41.948952+0000","last_peered":"2024-04-02T09:13:41.948952+0000","last_clean":"2024-04-02T09:13:41.948952+0000","last_became_active":"2024-04-02T09:13:28.380376+0000","last_became_peered":"2024-04-02T09:13:28.380376+0000","last_unstale":"2024-04-02T09:13:41.948952+0000","last_undegraded":"2024-04-02T09:13:41.948952+0000","last_fullsized":"2024-04-02T09:13:41.948952+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:46:17.713244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.346444+0000","last_change":"2024-04-02T09:13:41.961661+0000","last_active":"2024-04-02T09:13:43.346444+0000","last_peered":"2024-04-02T09:13:43.346444+0000","last_clean":"2024-04-02T09:13:43.346444+0000","last_became_active":"2024-04-02T09:13:41.959158+0000","last_became_peered":"2024-04-02T09:13:41.959158+0000","last_unstale":"2024-04-02T09:13:43.346444+0000","last_undegraded":"2024-04-02T09:13:43.346444+0000","last_fullsized":"2024-04-02T09:13:43.346444+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:57:28.703863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.1e","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.974495+0000","last_change":"2024-04-02T09:13:41.448497+0000","last_active":"2024-04-02T09:13:41.974495+0000","last_peered":"2024-04-02T09:13:41.974495+0000","last_clean":"2024-04-02T09:13:41.974495+0000","last_became_active":"2024-04-02T09:13:41.447385+0000","last_became_peered":"2024-04-02T09:13:41.447385+0000","last_unstale":"2024-04-02T09:13:41.974495+0000","last_undegraded":"2024-04-02T09:13:41.974495+0000","last_fullsized":"2024-04-02T09:13:41.974495+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:21.506479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943796+0000","last_change":"2024-04-02T09:13:28.390273+0000","last_active":"2024-04-02T09:13:41.943796+0000","last_peered":"2024-04-02T09:13:41.943796+0000","last_clean":"2024-04-02T09:13:41.943796+0000","last_became_active":"2024-04-02T09:13:28.389990+0000","last_became_peered":"2024-04-02T09:13:28.389990+0000","last_unstale":"2024-04-02T09:13:41.943796+0000","last_undegraded":"2024-04-02T09:13:41.943796+0000","last_fullsized":"2024-04-02T09:13:41.943796+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:30:33.534225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.948220+0000","last_change":"2024-04-02T09:13:41.958664+0000","last_active":"2024-04-02T09:13:42.948220+0000","last_peered":"2024-04-02T09:13:42.948220+0000","last_clean":"2024-04-02T09:13:42.948220+0000","last_became_active":"2024-04-02T09:13:41.958365+0000","last_became_peered":"2024-04-02T09:13:41.958365+0000","last_unstale":"2024-04-02T09:13:42.948220+0000","last_undegraded":"2024-04-02T09:13:42.948220+0000","last_fullsized":"2024-04-02T09:13:42.948220+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:37.458648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1f","version":"247'4","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.972969+0000","last_change":"2024-04-02T09:13:41.448711+0000","last_active":"2024-04-02T09:13:41.972969+0000","last_peered":"2024-04-02T09:13:41.972969+0000","last_clean":"2024-04-02T09:13:41.972969+0000","last_became_active":"2024-04-02T09:13:41.448398+0000","last_became_peered":"2024-04-02T09:13:41.448398+0000","last_unstale":"2024-04-02T09:13:41.972969+0000","last_undegraded":"2024-04-02T09:13:41.972969+0000","last_fullsized":"2024-04-02T09:13:41.972969+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:39.602788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952747+0000","last_change":"2024-04-02T09:13:28.395003+0000","last_active":"2024-04-02T09:13:41.952747+0000","last_peered":"2024-04-02T09:13:41.952747+0000","last_clean":"2024-04-02T09:13:41.952747+0000","last_became_active":"2024-04-02T09:13:28.394635+0000","last_became_peered":"2024-04-02T09:13:28.394635+0000","last_unstale":"2024-04-02T09:13:41.952747+0000","last_undegraded":"2024-04-02T09:13:41.952747+0000","last_fullsized":"2024-04-02T09:13:41.952747+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:54.758118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.733918+0000","last_change":"2024-04-02T09:13:41.949873+0000","last_active":"2024-04-02T09:13:43.733918+0000","last_peered":"2024-04-02T09:13:43.733918+0000","last_clean":"2024-04-02T09:13:43.733918+0000","last_became_active":"2024-04-02T09:13:41.949617+0000","last_became_peered":"2024-04-02T09:13:41.949617+0000","last_unstale":"2024-04-02T09:13:43.733918+0000","last_undegraded":"2024-04-02T09:13:43.733918+0000","last_fullsized":"2024-04-02T09:13:43.733918+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:19.901756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.954901+0000","last_change":"2024-04-02T09:13:28.391302+0000","last_active":"2024-04-02T09:13:41.954901+0000","last_peered":"2024-04-02T09:13:41.954901+0000","last_clean":"2024-04-02T09:13:41.954901+0000","last_became_active":"2024-04-02T09:13:28.388976+0000","last_became_peered":"2024-04-02T09:13:28.388976+0000","last_unstale":"2024-04-02T09:13:41.954901+0000","last_undegraded":"2024-04-02T09:13:41.954901+0000","last_fullsized":"2024-04-02T09:13:41.954901+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:02.556232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947787+0000","last_change":"2024-04-02T09:13:27.881269+0000","last_active":"2024-04-02T09:13:41.947787+0000","last_peered":"2024-04-02T09:13:41.947787+0000","last_clean":"2024-04-02T09:13:41.947787+0000","last_became_active":"2024-04-02T09:13:27.880664+0000","last_became_peered":"2024-04-02T09:13:27.880664+0000","last_unstale":"2024-04-02T09:13:41.947787+0000","last_undegraded":"2024-04-02T09:13:41.947787+0000","last_fullsized":"2024-04-02T09:13:41.947787+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:53.750473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734852+0000","last_change":"2024-04-02T09:13:41.967103+0000","last_active":"2024-04-02T09:13:43.734852+0000","last_peered":"2024-04-02T09:13:43.734852+0000","last_clean":"2024-04-02T09:13:43.734852+0000","last_became_active":"2024-04-02T09:13:41.966828+0000","last_became_peered":"2024-04-02T09:13:41.966828+0000","last_unstale":"2024-04-02T09:13:43.734852+0000","last_undegraded":"2024-04-02T09:13:43.734852+0000","last_fullsized":"2024-04-02T09:13:43.734852+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:44:48.623486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945056+0000","last_change":"2024-04-02T09:13:28.399529+0000","last_active":"2024-04-02T09:13:41.945056+0000","last_peered":"2024-04-02T09:13:41.945056+0000","last_clean":"2024-04-02T09:13:41.945056+0000","last_became_active":"2024-04-02T09:13:28.399042+0000","last_became_peered":"2024-04-02T09:13:28.399042+0000","last_unstale":"2024-04-02T09:13:41.945056+0000","last_undegraded":"2024-04-02T09:13:41.945056+0000","last_fullsized":"2024-04-02T09:13:41.945056+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:58:19.109584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1d","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.941583+0000","last_change":"2024-04-02T09:13:27.889299+0000","last_active":"2024-04-02T09:13:41.941583+0000","last_peered":"2024-04-02T09:13:41.941583+0000","last_clean":"2024-04-02T09:13:41.941583+0000","last_became_active":"2024-04-02T09:13:27.888249+0000","last_became_peered":"2024-04-02T09:13:27.888249+0000","last_unstale":"2024-04-02T09:13:41.941583+0000","last_undegraded":"2024-04-02T09:13:41.941583+0000","last_fullsized":"2024-04-02T09:13:41.941583+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:01:31.866958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345673+0000","last_change":"2024-04-02T09:13:41.966096+0000","last_active":"2024-04-02T09:13:43.345673+0000","last_peered":"2024-04-02T09:13:43.345673+0000","last_clean":"2024-04-02T09:13:43.345673+0000","last_became_active":"2024-04-02T09:13:41.959139+0000","last_became_peered":"2024-04-02T09:13:41.959139+0000","last_unstale":"2024-04-02T09:13:43.345673+0000","last_undegraded":"2024-04-02T09:13:43.345673+0000","last_fullsized":"2024-04-02T09:13:43.345673+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:54:06.041578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950575+0000","last_change":"2024-04-02T09:13:28.382423+0000","last_active":"2024-04-02T09:13:41.950575+0000","last_peered":"2024-04-02T09:13:41.950575+0000","last_clean":"2024-04-02T09:13:41.950575+0000","last_became_active":"2024-04-02T09:13:28.381840+0000","last_became_peered":"2024-04-02T09:13:28.381840+0000","last_unstale":"2024-04-02T09:13:41.950575+0000","last_undegraded":"2024-04-02T09:13:41.950575+0000","last_fullsized":"2024-04-02T09:13:41.950575+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:04:44.765954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.954520+0000","last_change":"2024-04-02T09:13:27.885629+0000","last_active":"2024-04-02T09:13:41.954520+0000","last_peered":"2024-04-02T09:13:41.954520+0000","last_clean":"2024-04-02T09:13:41.954520+0000","last_became_active":"2024-04-02T09:13:27.884921+0000","last_became_peered":"2024-04-02T09:13:27.884921+0000","last_unstale":"2024-04-02T09:13:41.954520+0000","last_undegraded":"2024-04-02T09:13:41.954520+0000","last_fullsized":"2024-04-02T09:13:41.954520+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:49:58.033026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734939+0000","last_change":"2024-04-02T09:13:41.963609+0000","last_active":"2024-04-02T09:13:43.734939+0000","last_peered":"2024-04-02T09:13:43.734939+0000","last_clean":"2024-04-02T09:13:43.734939+0000","last_became_active":"2024-04-02T09:13:41.963275+0000","last_became_peered":"2024-04-02T09:13:41.963275+0000","last_unstale":"2024-04-02T09:13:43.734939+0000","last_undegraded":"2024-04-02T09:13:43.734939+0000","last_fullsized":"2024-04-02T09:13:43.734939+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:50:02.673300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952776+0000","last_change":"2024-04-02T09:13:28.381558+0000","last_active":"2024-04-02T09:13:41.952776+0000","last_peered":"2024-04-02T09:13:41.952776+0000","last_clean":"2024-04-02T09:13:41.952776+0000","last_became_active":"2024-04-02T09:13:28.381306+0000","last_became_peered":"2024-04-02T09:13:28.381306+0000","last_unstale":"2024-04-02T09:13:41.952776+0000","last_undegraded":"2024-04-02T09:13:41.952776+0000","last_fullsized":"2024-04-02T09:13:41.952776+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:12:17.313507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1b","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.955013+0000","last_change":"2024-04-02T09:13:27.884062+0000","last_active":"2024-04-02T09:13:41.955013+0000","last_peered":"2024-04-02T09:13:41.955013+0000","last_clean":"2024-04-02T09:13:41.955013+0000","last_became_active":"2024-04-02T09:13:27.883677+0000","last_became_peered":"2024-04-02T09:13:27.883677+0000","last_unstale":"2024-04-02T09:13:41.955013+0000","last_undegraded":"2024-04-02T09:13:41.955013+0000","last_fullsized":"2024-04-02T09:13:41.955013+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:14.774442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.948048+0000","last_change":"2024-04-02T09:13:41.955593+0000","last_active":"2024-04-02T09:13:42.948048+0000","last_peered":"2024-04-02T09:13:42.948048+0000","last_clean":"2024-04-02T09:13:42.948048+0000","last_became_active":"2024-04-02T09:13:41.955337+0000","last_became_peered":"2024-04-02T09:13:41.955337+0000","last_unstale":"2024-04-02T09:13:42.948048+0000","last_undegraded":"2024-04-02T09:13:42.948048+0000","last_fullsized":"2024-04-02T09:13:42.948048+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:39:05.298707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943513+0000","last_change":"2024-04-02T09:13:28.389833+0000","last_active":"2024-04-02T09:13:41.943513+0000","last_peered":"2024-04-02T09:13:41.943513+0000","last_clean":"2024-04-02T09:13:41.943513+0000","last_became_active":"2024-04-02T09:13:28.389495+0000","last_became_peered":"2024-04-02T09:13:28.389495+0000","last_unstale":"2024-04-02T09:13:41.943513+0000","last_undegraded":"2024-04-02T09:13:41.943513+0000","last_fullsized":"2024-04-02T09:13:41.943513+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:27.049697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1a","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952655+0000","last_change":"2024-04-02T09:13:27.883532+0000","last_active":"2024-04-02T09:13:41.952655+0000","last_peered":"2024-04-02T09:13:41.952655+0000","last_clean":"2024-04-02T09:13:41.952655+0000","last_became_active":"2024-04-02T09:13:27.883252+0000","last_became_peered":"2024-04-02T09:13:27.883252+0000","last_unstale":"2024-04-02T09:13:41.952655+0000","last_undegraded":"2024-04-02T09:13:41.952655+0000","last_fullsized":"2024-04-02T09:13:41.952655+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.950371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.948318+0000","last_change":"2024-04-02T09:13:41.963566+0000","last_active":"2024-04-02T09:13:42.948318+0000","last_peered":"2024-04-02T09:13:42.948318+0000","last_clean":"2024-04-02T09:13:42.948318+0000","last_became_active":"2024-04-02T09:13:41.962503+0000","last_became_peered":"2024-04-02T09:13:41.962503+0000","last_unstale":"2024-04-02T09:13:42.948318+0000","last_undegraded":"2024-04-02T09:13:42.948318+0000","last_fullsized":"2024-04-02T09:13:42.948318+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:13.220603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943231+0000","last_change":"2024-04-02T09:13:28.403943+0000","last_active":"2024-04-02T09:13:41.943231+0000","last_peered":"2024-04-02T09:13:41.943231+0000","last_clean":"2024-04-02T09:13:41.943231+0000","last_became_active":"2024-04-02T09:13:28.402197+0000","last_became_peered":"2024-04-02T09:13:28.402197+0000","last_unstale":"2024-04-02T09:13:41.943231+0000","last_undegraded":"2024-04-02T09:13:41.943231+0000","last_fullsized":"2024-04-02T09:13:41.943231+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:50.579695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945549+0000","last_change":"2024-04-02T09:13:27.883508+0000","last_active":"2024-04-02T09:13:41.945549+0000","last_peered":"2024-04-02T09:13:41.945549+0000","last_clean":"2024-04-02T09:13:41.945549+0000","last_became_active":"2024-04-02T09:13:27.883263+0000","last_became_peered":"2024-04-02T09:13:27.883263+0000","last_unstale":"2024-04-02T09:13:41.945549+0000","last_undegraded":"2024-04-02T09:13:41.945549+0000","last_fullsized":"2024-04-02T09:13:41.945549+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:37.428725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954612+0000","last_change":"2024-04-02T09:13:41.965533+0000","last_active":"2024-04-02T09:13:42.954612+0000","last_peered":"2024-04-02T09:13:42.954612+0000","last_clean":"2024-04-02T09:13:42.954612+0000","last_became_active":"2024-04-02T09:13:41.965270+0000","last_became_peered":"2024-04-02T09:13:41.965270+0000","last_unstale":"2024-04-02T09:13:42.954612+0000","last_undegraded":"2024-04-02T09:13:42.954612+0000","last_fullsized":"2024-04-02T09:13:42.954612+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:53.549513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943892+0000","last_change":"2024-04-02T09:13:28.403768+0000","last_active":"2024-04-02T09:13:41.943892+0000","last_peered":"2024-04-02T09:13:41.943892+0000","last_clean":"2024-04-02T09:13:41.943892+0000","last_became_active":"2024-04-02T09:13:28.399745+0000","last_became_peered":"2024-04-02T09:13:28.399745+0000","last_unstale":"2024-04-02T09:13:41.943892+0000","last_undegraded":"2024-04-02T09:13:41.943892+0000","last_fullsized":"2024-04-02T09:13:41.943892+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:39.541325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953065+0000","last_change":"2024-04-02T09:13:27.888598+0000","last_active":"2024-04-02T09:13:41.953065+0000","last_peered":"2024-04-02T09:13:41.953065+0000","last_clean":"2024-04-02T09:13:41.953065+0000","last_became_active":"2024-04-02T09:13:27.888115+0000","last_became_peered":"2024-04-02T09:13:27.888115+0000","last_unstale":"2024-04-02T09:13:41.953065+0000","last_undegraded":"2024-04-02T09:13:41.953065+0000","last_fullsized":"2024-04-02T09:13:41.953065+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:57:03.609779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.953354+0000","last_change":"2024-04-02T09:13:41.957572+0000","last_active":"2024-04-02T09:13:42.953354+0000","last_peered":"2024-04-02T09:13:42.953354+0000","last_clean":"2024-04-02T09:13:42.953354+0000","last_became_active":"2024-04-02T09:13:41.956534+0000","last_became_peered":"2024-04-02T09:13:41.956534+0000","last_unstale":"2024-04-02T09:13:42.953354+0000","last_undegraded":"2024-04-02T09:13:42.953354+0000","last_fullsized":"2024-04-02T09:13:42.953354+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:09.258626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947327+0000","last_change":"2024-04-02T09:13:28.404602+0000","last_active":"2024-04-02T09:13:41.947327+0000","last_peered":"2024-04-02T09:13:41.947327+0000","last_clean":"2024-04-02T09:13:41.947327+0000","last_became_active":"2024-04-02T09:13:28.403446+0000","last_became_peered":"2024-04-02T09:13:28.403446+0000","last_unstale":"2024-04-02T09:13:41.947327+0000","last_undegraded":"2024-04-02T09:13:41.947327+0000","last_fullsized":"2024-04-02T09:13:41.947327+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:24.279179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947218+0000","last_change":"2024-04-02T09:13:27.887286+0000","last_active":"2024-04-02T09:13:41.947218+0000","last_peered":"2024-04-02T09:13:41.947218+0000","last_clean":"2024-04-02T09:13:41.947218+0000","last_became_active":"2024-04-02T09:13:27.886999+0000","last_became_peered":"2024-04-02T09:13:27.886999+0000","last_unstale":"2024-04-02T09:13:41.947218+0000","last_undegraded":"2024-04-02T09:13:41.947218+0000","last_fullsized":"2024-04-02T09:13:41.947218+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:44.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342182+0000","last_change":"2024-04-02T09:13:41.961802+0000","last_active":"2024-04-02T09:13:43.342182+0000","last_peered":"2024-04-02T09:13:43.342182+0000","last_clean":"2024-04-02T09:13:43.342182+0000","last_became_active":"2024-04-02T09:13:41.961522+0000","last_became_peered":"2024-04-02T09:13:41.961522+0000","last_unstale":"2024-04-02T09:13:43.342182+0000","last_undegraded":"2024-04-02T09:13:43.342182+0000","last_fullsized":"2024-04-02T09:13:43.342182+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:51:04.142072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.954773+0000","last_change":"2024-04-02T09:13:28.378767+0000","last_active":"2024-04-02T09:13:41.954773+0000","last_peered":"2024-04-02T09:13:41.954773+0000","last_clean":"2024-04-02T09:13:41.954773+0000","last_became_active":"2024-04-02T09:13:28.378487+0000","last_became_peered":"2024-04-02T09:13:28.378487+0000","last_unstale":"2024-04-02T09:13:41.954773+0000","last_undegraded":"2024-04-02T09:13:41.954773+0000","last_fullsized":"2024-04-02T09:13:41.954773+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:26:47.539777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947422+0000","last_change":"2024-04-02T09:13:27.884469+0000","last_active":"2024-04-02T09:13:41.947422+0000","last_peered":"2024-04-02T09:13:41.947422+0000","last_clean":"2024-04-02T09:13:41.947422+0000","last_became_active":"2024-04-02T09:13:27.882570+0000","last_became_peered":"2024-04-02T09:13:27.882570+0000","last_unstale":"2024-04-02T09:13:41.947422+0000","last_undegraded":"2024-04-02T09:13:41.947422+0000","last_fullsized":"2024-04-02T09:13:41.947422+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:57:12.395680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.953682+0000","last_change":"2024-04-02T09:13:41.959972+0000","last_active":"2024-04-02T09:13:42.953682+0000","last_peered":"2024-04-02T09:13:42.953682+0000","last_clean":"2024-04-02T09:13:42.953682+0000","last_became_active":"2024-04-02T09:13:41.959655+0000","last_became_peered":"2024-04-02T09:13:41.959655+0000","last_unstale":"2024-04-02T09:13:42.953682+0000","last_undegraded":"2024-04-02T09:13:42.953682+0000","last_fullsized":"2024-04-02T09:13:42.953682+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:23:35.907439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.959311+0000","last_change":"2024-04-02T09:13:28.387205+0000","last_active":"2024-04-02T09:13:41.959311+0000","last_peered":"2024-04-02T09:13:41.959311+0000","last_clean":"2024-04-02T09:13:41.959311+0000","last_became_active":"2024-04-02T09:13:28.384126+0000","last_became_peered":"2024-04-02T09:13:28.384126+0000","last_unstale":"2024-04-02T09:13:41.959311+0000","last_undegraded":"2024-04-02T09:13:41.959311+0000","last_fullsized":"2024-04-02T09:13:41.959311+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:52.330349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.15","version":"243'3","reported_seq":18,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.962102+0000","last_change":"2024-04-02T09:13:27.883818+0000","last_active":"2024-04-02T09:13:41.962102+0000","last_peered":"2024-04-02T09:13:41.962102+0000","last_clean":"2024-04-02T09:13:41.962102+0000","last_became_active":"2024-04-02T09:13:27.883527+0000","last_became_peered":"2024-04-02T09:13:27.883527+0000","last_unstale":"2024-04-02T09:13:41.962102+0000","last_undegraded":"2024-04-02T09:13:41.962102+0000","last_fullsized":"2024-04-02T09:13:41.962102+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:49.572049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954621+0000","last_change":"2024-04-02T09:13:41.948944+0000","last_active":"2024-04-02T09:13:42.954621+0000","last_peered":"2024-04-02T09:13:42.954621+0000","last_clean":"2024-04-02T09:13:42.954621+0000","last_became_active":"2024-04-02T09:13:41.948653+0000","last_became_peered":"2024-04-02T09:13:41.948653+0000","last_unstale":"2024-04-02T09:13:42.954621+0000","last_undegraded":"2024-04-02T09:13:42.954621+0000","last_fullsized":"2024-04-02T09:13:42.954621+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:12:23.778874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948000+0000","last_change":"2024-04-02T09:13:28.379346+0000","last_active":"2024-04-02T09:13:41.948000+0000","last_peered":"2024-04-02T09:13:41.948000+0000","last_clean":"2024-04-02T09:13:41.948000+0000","last_became_active":"2024-04-02T09:13:28.378741+0000","last_became_peered":"2024-04-02T09:13:28.378741+0000","last_unstale":"2024-04-02T09:13:41.948000+0000","last_undegraded":"2024-04-02T09:13:41.948000+0000","last_fullsized":"2024-04-02T09:13:41.948000+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:44.092529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.14","version":"244'8","reported_seq":23,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.941516+0000","last_change":"2024-04-02T09:13:27.885270+0000","last_active":"2024-04-02T09:13:41.941516+0000","last_peered":"2024-04-02T09:13:41.941516+0000","last_clean":"2024-04-02T09:13:41.941516+0000","last_became_active":"2024-04-02T09:13:27.884953+0000","last_became_peered":"2024-04-02T09:13:27.884953+0000","last_unstale":"2024-04-02T09:13:41.941516+0000","last_undegraded":"2024-04-02T09:13:41.941516+0000","last_fullsized":"2024-04-02T09:13:41.941516+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:48.103277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2805,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.953123+0000","last_change":"2024-04-02T09:13:41.955589+0000","last_active":"2024-04-02T09:13:42.953123+0000","last_peered":"2024-04-02T09:13:42.953123+0000","last_clean":"2024-04-02T09:13:42.953123+0000","last_became_active":"2024-04-02T09:13:41.955331+0000","last_became_peered":"2024-04-02T09:13:41.955331+0000","last_unstale":"2024-04-02T09:13:42.953123+0000","last_undegraded":"2024-04-02T09:13:42.953123+0000","last_fullsized":"2024-04-02T09:13:42.953123+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:35.863141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943802+0000","last_change":"2024-04-02T09:13:28.393735+0000","last_active":"2024-04-02T09:13:41.943802+0000","last_peered":"2024-04-02T09:13:41.943802+0000","last_clean":"2024-04-02T09:13:41.943802+0000","last_became_active":"2024-04-02T09:13:28.393387+0000","last_became_peered":"2024-04-02T09:13:28.393387+0000","last_unstale":"2024-04-02T09:13:41.943802+0000","last_undegraded":"2024-04-02T09:13:41.943802+0000","last_fullsized":"2024-04-02T09:13:41.943802+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:47.359176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.13","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940581+0000","last_change":"2024-04-02T09:13:27.885596+0000","last_active":"2024-04-02T09:13:41.940581+0000","last_peered":"2024-04-02T09:13:41.940581+0000","last_clean":"2024-04-02T09:13:41.940581+0000","last_became_active":"2024-04-02T09:13:27.885310+0000","last_became_peered":"2024-04-02T09:13:27.885310+0000","last_unstale":"2024-04-02T09:13:41.940581+0000","last_undegraded":"2024-04-02T09:13:41.940581+0000","last_fullsized":"2024-04-02T09:13:41.940581+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:02.497118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342453+0000","last_change":"2024-04-02T09:13:41.962021+0000","last_active":"2024-04-02T09:13:43.342453+0000","last_peered":"2024-04-02T09:13:43.342453+0000","last_clean":"2024-04-02T09:13:43.342453+0000","last_became_active":"2024-04-02T09:13:41.961212+0000","last_became_peered":"2024-04-02T09:13:41.961212+0000","last_unstale":"2024-04-02T09:13:43.342453+0000","last_undegraded":"2024-04-02T09:13:43.342453+0000","last_fullsized":"2024-04-02T09:13:43.342453+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:37.841557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950657+0000","last_change":"2024-04-02T09:13:28.378384+0000","last_active":"2024-04-02T09:13:41.950657+0000","last_peered":"2024-04-02T09:13:41.950657+0000","last_clean":"2024-04-02T09:13:41.950657+0000","last_became_active":"2024-04-02T09:13:28.377833+0000","last_became_peered":"2024-04-02T09:13:28.377833+0000","last_unstale":"2024-04-02T09:13:41.950657+0000","last_undegraded":"2024-04-02T09:13:41.950657+0000","last_fullsized":"2024-04-02T09:13:41.950657+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:37:54.526506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942840+0000","last_change":"2024-04-02T09:13:27.880069+0000","last_active":"2024-04-02T09:13:41.942840+0000","last_peered":"2024-04-02T09:13:41.942840+0000","last_clean":"2024-04-02T09:13:41.942840+0000","last_became_active":"2024-04-02T09:13:27.879793+0000","last_became_peered":"2024-04-02T09:13:27.879793+0000","last_unstale":"2024-04-02T09:13:41.942840+0000","last_undegraded":"2024-04-02T09:13:41.942840+0000","last_fullsized":"2024-04-02T09:13:41.942840+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:10.198371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734950+0000","last_change":"2024-04-02T09:13:41.951761+0000","last_active":"2024-04-02T09:13:43.734950+0000","last_peered":"2024-04-02T09:13:43.734950+0000","last_clean":"2024-04-02T09:13:43.734950+0000","last_became_active":"2024-04-02T09:13:41.951447+0000","last_became_peered":"2024-04-02T09:13:41.951447+0000","last_unstale":"2024-04-02T09:13:43.734950+0000","last_undegraded":"2024-04-02T09:13:43.734950+0000","last_fullsized":"2024-04-02T09:13:43.734950+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:37.458648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952962+0000","last_change":"2024-04-02T09:13:28.394088+0000","last_active":"2024-04-02T09:13:41.952962+0000","last_peered":"2024-04-02T09:13:41.952962+0000","last_clean":"2024-04-02T09:13:41.952962+0000","last_became_active":"2024-04-02T09:13:28.392932+0000","last_became_peered":"2024-04-02T09:13:28.392932+0000","last_unstale":"2024-04-02T09:13:41.952962+0000","last_undegraded":"2024-04-02T09:13:41.952962+0000","last_fullsized":"2024-04-02T09:13:41.952962+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:54.683721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945630+0000","last_change":"2024-04-02T09:13:27.881714+0000","last_active":"2024-04-02T09:13:41.945630+0000","last_peered":"2024-04-02T09:13:41.945630+0000","last_clean":"2024-04-02T09:13:41.945630+0000","last_became_active":"2024-04-02T09:13:27.881428+0000","last_became_peered":"2024-04-02T09:13:27.881428+0000","last_unstale":"2024-04-02T09:13:41.945630+0000","last_undegraded":"2024-04-02T09:13:41.945630+0000","last_fullsized":"2024-04-02T09:13:41.945630+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:50.081688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.346442+0000","last_change":"2024-04-02T09:13:41.972497+0000","last_active":"2024-04-02T09:13:43.346442+0000","last_peered":"2024-04-02T09:13:43.346442+0000","last_clean":"2024-04-02T09:13:43.346442+0000","last_became_active":"2024-04-02T09:13:41.972137+0000","last_became_peered":"2024-04-02T09:13:41.972137+0000","last_unstale":"2024-04-02T09:13:43.346442+0000","last_undegraded":"2024-04-02T09:13:43.346442+0000","last_fullsized":"2024-04-02T09:13:43.346442+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:19.631975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944595+0000","last_change":"2024-04-02T09:13:28.391188+0000","last_active":"2024-04-02T09:13:41.944595+0000","last_peered":"2024-04-02T09:13:41.944595+0000","last_clean":"2024-04-02T09:13:41.944595+0000","last_became_active":"2024-04-02T09:13:28.390867+0000","last_became_peered":"2024-04-02T09:13:28.390867+0000","last_unstale":"2024-04-02T09:13:41.944595+0000","last_undegraded":"2024-04-02T09:13:41.944595+0000","last_fullsized":"2024-04-02T09:13:41.944595+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:26:58.531401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.10","version":"243'8","reported_seq":23,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952918+0000","last_change":"2024-04-02T09:13:27.886503+0000","last_active":"2024-04-02T09:13:41.952918+0000","last_peered":"2024-04-02T09:13:41.952918+0000","last_clean":"2024-04-02T09:13:41.952918+0000","last_became_active":"2024-04-02T09:13:27.886121+0000","last_became_peered":"2024-04-02T09:13:27.886121+0000","last_unstale":"2024-04-02T09:13:41.952918+0000","last_undegraded":"2024-04-02T09:13:41.952918+0000","last_fullsized":"2024-04-02T09:13:41.952918+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:38.533680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952724+0000","last_change":"2024-04-02T09:13:41.954229+0000","last_active":"2024-04-02T09:13:42.952724+0000","last_peered":"2024-04-02T09:13:42.952724+0000","last_clean":"2024-04-02T09:13:42.952724+0000","last_became_active":"2024-04-02T09:13:41.950847+0000","last_became_peered":"2024-04-02T09:13:41.950847+0000","last_unstale":"2024-04-02T09:13:42.952724+0000","last_undegraded":"2024-04-02T09:13:42.952724+0000","last_fullsized":"2024-04-02T09:13:42.952724+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:58.664898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.955366+0000","last_change":"2024-04-02T09:13:28.376609+0000","last_active":"2024-04-02T09:13:41.955366+0000","last_peered":"2024-04-02T09:13:41.955366+0000","last_clean":"2024-04-02T09:13:41.955366+0000","last_became_active":"2024-04-02T09:13:28.376226+0000","last_became_peered":"2024-04-02T09:13:28.376226+0000","last_unstale":"2024-04-02T09:13:41.955366+0000","last_undegraded":"2024-04-02T09:13:41.955366+0000","last_fullsized":"2024-04-02T09:13:41.955366+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:59.766007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948352+0000","last_change":"2024-04-02T09:13:27.883080+0000","last_active":"2024-04-02T09:13:41.948352+0000","last_peered":"2024-04-02T09:13:41.948352+0000","last_clean":"2024-04-02T09:13:41.948352+0000","last_became_active":"2024-04-02T09:13:27.882612+0000","last_became_peered":"2024-04-02T09:13:27.882612+0000","last_unstale":"2024-04-02T09:13:41.948352+0000","last_undegraded":"2024-04-02T09:13:41.948352+0000","last_fullsized":"2024-04-02T09:13:41.948352+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:20.979570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.735089+0000","last_change":"2024-04-02T09:13:41.960521+0000","last_active":"2024-04-02T09:13:43.735089+0000","last_peered":"2024-04-02T09:13:43.735089+0000","last_clean":"2024-04-02T09:13:43.735089+0000","last_became_active":"2024-04-02T09:13:41.960258+0000","last_became_peered":"2024-04-02T09:13:41.960258+0000","last_unstale":"2024-04-02T09:13:43.735089+0000","last_undegraded":"2024-04-02T09:13:43.735089+0000","last_fullsized":"2024-04-02T09:13:43.735089+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:45:09.502243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.961541+0000","last_change":"2024-04-02T09:13:28.381499+0000","last_active":"2024-04-02T09:13:41.961541+0000","last_peered":"2024-04-02T09:13:41.961541+0000","last_clean":"2024-04-02T09:13:41.961541+0000","last_became_active":"2024-04-02T09:13:28.381286+0000","last_became_peered":"2024-04-02T09:13:28.381286+0000","last_unstale":"2024-04-02T09:13:41.961541+0000","last_undegraded":"2024-04-02T09:13:41.961541+0000","last_fullsized":"2024-04-02T09:13:41.961541+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:18.748067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.e","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947518+0000","last_change":"2024-04-02T09:13:27.888558+0000","last_active":"2024-04-02T09:13:41.947518+0000","last_peered":"2024-04-02T09:13:41.947518+0000","last_clean":"2024-04-02T09:13:41.947518+0000","last_became_active":"2024-04-02T09:13:27.888288+0000","last_became_peered":"2024-04-02T09:13:27.888288+0000","last_unstale":"2024-04-02T09:13:41.947518+0000","last_undegraded":"2024-04-02T09:13:41.947518+0000","last_fullsized":"2024-04-02T09:13:41.947518+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:15.386238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734712+0000","last_change":"2024-04-02T09:13:41.967882+0000","last_active":"2024-04-02T09:13:43.734712+0000","last_peered":"2024-04-02T09:13:43.734712+0000","last_clean":"2024-04-02T09:13:43.734712+0000","last_became_active":"2024-04-02T09:13:41.967598+0000","last_became_peered":"2024-04-02T09:13:41.967598+0000","last_unstale":"2024-04-02T09:13:43.734712+0000","last_undegraded":"2024-04-02T09:13:43.734712+0000","last_fullsized":"2024-04-02T09:13:43.734712+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:18.838465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948337+0000","last_change":"2024-04-02T09:13:28.372849+0000","last_active":"2024-04-02T09:13:41.948337+0000","last_peered":"2024-04-02T09:13:41.948337+0000","last_clean":"2024-04-02T09:13:41.948337+0000","last_became_active":"2024-04-02T09:13:28.372595+0000","last_became_peered":"2024-04-02T09:13:28.372595+0000","last_unstale":"2024-04-02T09:13:41.948337+0000","last_undegraded":"2024-04-02T09:13:41.948337+0000","last_fullsized":"2024-04-02T09:13:41.948337+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:12.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.d","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947395+0000","last_change":"2024-04-02T09:13:27.879417+0000","last_active":"2024-04-02T09:13:41.947395+0000","last_peered":"2024-04-02T09:13:41.947395+0000","last_clean":"2024-04-02T09:13:41.947395+0000","last_became_active":"2024-04-02T09:13:27.879165+0000","last_became_peered":"2024-04-02T09:13:27.879165+0000","last_unstale":"2024-04-02T09:13:41.947395+0000","last_undegraded":"2024-04-02T09:13:41.947395+0000","last_fullsized":"2024-04-02T09:13:41.947395+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:47:32.233662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734901+0000","last_change":"2024-04-02T09:13:41.968877+0000","last_active":"2024-04-02T09:13:43.734901+0000","last_peered":"2024-04-02T09:13:43.734901+0000","last_clean":"2024-04-02T09:13:43.734901+0000","last_became_active":"2024-04-02T09:13:41.968456+0000","last_became_peered":"2024-04-02T09:13:41.968456+0000","last_unstale":"2024-04-02T09:13:43.734901+0000","last_undegraded":"2024-04-02T09:13:43.734901+0000","last_fullsized":"2024-04-02T09:13:43.734901+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:54.806757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.939446+0000","last_change":"2024-04-02T09:13:28.392130+0000","last_active":"2024-04-02T09:13:41.939446+0000","last_peered":"2024-04-02T09:13:41.939446+0000","last_clean":"2024-04-02T09:13:41.939446+0000","last_became_active":"2024-04-02T09:13:28.391690+0000","last_became_peered":"2024-04-02T09:13:28.391690+0000","last_unstale":"2024-04-02T09:13:41.939446+0000","last_undegraded":"2024-04-02T09:13:41.939446+0000","last_fullsized":"2024-04-02T09:13:41.939446+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:24.319747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945343+0000","last_change":"2024-04-02T09:13:27.884781+0000","last_active":"2024-04-02T09:13:41.945343+0000","last_peered":"2024-04-02T09:13:41.945343+0000","last_clean":"2024-04-02T09:13:41.945343+0000","last_became_active":"2024-04-02T09:13:27.884507+0000","last_became_peered":"2024-04-02T09:13:27.884507+0000","last_unstale":"2024-04-02T09:13:41.945343+0000","last_undegraded":"2024-04-02T09:13:41.945343+0000","last_fullsized":"2024-04-02T09:13:41.945343+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:31:18.889273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342453+0000","last_change":"2024-04-02T09:13:41.958635+0000","last_active":"2024-04-02T09:13:43.342453+0000","last_peered":"2024-04-02T09:13:43.342453+0000","last_clean":"2024-04-02T09:13:43.342453+0000","last_became_active":"2024-04-02T09:13:41.955459+0000","last_became_peered":"2024-04-02T09:13:41.955459+0000","last_unstale":"2024-04-02T09:13:43.342453+0000","last_undegraded":"2024-04-02T09:13:43.342453+0000","last_fullsized":"2024-04-02T09:13:43.342453+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:53.103702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947024+0000","last_change":"2024-04-02T09:13:28.381580+0000","last_active":"2024-04-02T09:13:41.947024+0000","last_peered":"2024-04-02T09:13:41.947024+0000","last_clean":"2024-04-02T09:13:41.947024+0000","last_became_active":"2024-04-02T09:13:28.381300+0000","last_became_peered":"2024-04-02T09:13:28.381300+0000","last_unstale":"2024-04-02T09:13:41.947024+0000","last_undegraded":"2024-04-02T09:13:41.947024+0000","last_fullsized":"2024-04-02T09:13:41.947024+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:51.089333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944130+0000","last_change":"2024-04-02T09:13:27.875291+0000","last_active":"2024-04-02T09:13:41.944130+0000","last_peered":"2024-04-02T09:13:41.944130+0000","last_clean":"2024-04-02T09:13:41.944130+0000","last_became_active":"2024-04-02T09:13:27.875003+0000","last_became_peered":"2024-04-02T09:13:27.875003+0000","last_unstale":"2024-04-02T09:13:41.944130+0000","last_undegraded":"2024-04-02T09:13:41.944130+0000","last_fullsized":"2024-04-02T09:13:41.944130+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:00.637981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734207+0000","last_change":"2024-04-02T09:13:41.949115+0000","last_active":"2024-04-02T09:13:43.734207+0000","last_peered":"2024-04-02T09:13:43.734207+0000","last_clean":"2024-04-02T09:13:43.734207+0000","last_became_active":"2024-04-02T09:13:41.948850+0000","last_became_peered":"2024-04-02T09:13:41.948850+0000","last_unstale":"2024-04-02T09:13:43.734207+0000","last_undegraded":"2024-04-02T09:13:43.734207+0000","last_fullsized":"2024-04-02T09:13:43.734207+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:54.806757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944021+0000","last_change":"2024-04-02T09:13:28.404265+0000","last_active":"2024-04-02T09:13:41.944021+0000","last_peered":"2024-04-02T09:13:41.944021+0000","last_clean":"2024-04-02T09:13:41.944021+0000","last_became_active":"2024-04-02T09:13:28.403894+0000","last_became_peered":"2024-04-02T09:13:28.403894+0000","last_unstale":"2024-04-02T09:13:41.944021+0000","last_undegraded":"2024-04-02T09:13:41.944021+0000","last_fullsized":"2024-04-02T09:13:41.944021+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:05:23.433426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943067+0000","last_change":"2024-04-02T09:13:27.881604+0000","last_active":"2024-04-02T09:13:41.943067+0000","last_peered":"2024-04-02T09:13:41.943067+0000","last_clean":"2024-04-02T09:13:41.943067+0000","last_became_active":"2024-04-02T09:13:27.881304+0000","last_became_peered":"2024-04-02T09:13:27.881304+0000","last_unstale":"2024-04-02T09:13:41.943067+0000","last_undegraded":"2024-04-02T09:13:41.943067+0000","last_fullsized":"2024-04-02T09:13:41.943067+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:02.497118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734857+0000","last_change":"2024-04-02T09:13:41.956788+0000","last_active":"2024-04-02T09:13:43.734857+0000","last_peered":"2024-04-02T09:13:43.734857+0000","last_clean":"2024-04-02T09:13:43.734857+0000","last_became_active":"2024-04-02T09:13:41.956513+0000","last_became_peered":"2024-04-02T09:13:41.956513+0000","last_unstale":"2024-04-02T09:13:43.734857+0000","last_undegraded":"2024-04-02T09:13:43.734857+0000","last_fullsized":"2024-04-02T09:13:43.734857+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:39:05.298707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953053+0000","last_change":"2024-04-02T09:13:28.374960+0000","last_active":"2024-04-02T09:13:41.953053+0000","last_peered":"2024-04-02T09:13:41.953053+0000","last_clean":"2024-04-02T09:13:41.953053+0000","last_became_active":"2024-04-02T09:13:28.373683+0000","last_became_peered":"2024-04-02T09:13:28.373683+0000","last_unstale":"2024-04-02T09:13:41.953053+0000","last_undegraded":"2024-04-02T09:13:41.953053+0000","last_fullsized":"2024-04-02T09:13:41.953053+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:15.851301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.938571+0000","last_change":"2024-04-02T09:13:27.877837+0000","last_active":"2024-04-02T09:13:41.938571+0000","last_peered":"2024-04-02T09:13:41.938571+0000","last_clean":"2024-04-02T09:13:41.938571+0000","last_became_active":"2024-04-02T09:13:27.876792+0000","last_became_peered":"2024-04-02T09:13:27.876792+0000","last_unstale":"2024-04-02T09:13:41.938571+0000","last_undegraded":"2024-04-02T09:13:41.938571+0000","last_fullsized":"2024-04-02T09:13:41.938571+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:17.740422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954331+0000","last_change":"2024-04-02T09:13:41.949429+0000","last_active":"2024-04-02T09:13:42.954331+0000","last_peered":"2024-04-02T09:13:42.954331+0000","last_clean":"2024-04-02T09:13:42.954331+0000","last_became_active":"2024-04-02T09:13:41.948422+0000","last_became_peered":"2024-04-02T09:13:41.948422+0000","last_unstale":"2024-04-02T09:13:42.954331+0000","last_undegraded":"2024-04-02T09:13:42.954331+0000","last_fullsized":"2024-04-02T09:13:42.954331+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:31:40.330222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945371+0000","last_change":"2024-04-02T09:13:28.379243+0000","last_active":"2024-04-02T09:13:41.945371+0000","last_peered":"2024-04-02T09:13:41.945371+0000","last_clean":"2024-04-02T09:13:41.945371+0000","last_became_active":"2024-04-02T09:13:28.378297+0000","last_became_peered":"2024-04-02T09:13:28.378297+0000","last_unstale":"2024-04-02T09:13:41.945371+0000","last_undegraded":"2024-04-02T09:13:41.945371+0000","last_fullsized":"2024-04-02T09:13:41.945371+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:02.761190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.938490+0000","last_change":"2024-04-02T09:13:27.876051+0000","last_active":"2024-04-02T09:13:41.938490+0000","last_peered":"2024-04-02T09:13:41.938490+0000","last_clean":"2024-04-02T09:13:41.938490+0000","last_became_active":"2024-04-02T09:13:27.875628+0000","last_became_peered":"2024-04-02T09:13:27.875628+0000","last_unstale":"2024-04-02T09:13:41.938490+0000","last_undegraded":"2024-04-02T09:13:41.938490+0000","last_fullsized":"2024-04-02T09:13:41.938490+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:53.750473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.947186+0000","last_change":"2024-04-02T09:13:41.961652+0000","last_active":"2024-04-02T09:13:42.947186+0000","last_peered":"2024-04-02T09:13:42.947186+0000","last_clean":"2024-04-02T09:13:42.947186+0000","last_became_active":"2024-04-02T09:13:41.961378+0000","last_became_peered":"2024-04-02T09:13:41.961378+0000","last_unstale":"2024-04-02T09:13:42.947186+0000","last_undegraded":"2024-04-02T09:13:42.947186+0000","last_fullsized":"2024-04-02T09:13:42.947186+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:51:51.721965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947638+0000","last_change":"2024-04-02T09:13:28.383204+0000","last_active":"2024-04-02T09:13:41.947638+0000","last_peered":"2024-04-02T09:13:41.947638+0000","last_clean":"2024-04-02T09:13:41.947638+0000","last_became_active":"2024-04-02T09:13:28.382961+0000","last_became_peered":"2024-04-02T09:13:28.382961+0000","last_unstale":"2024-04-02T09:13:41.947638+0000","last_undegraded":"2024-04-02T09:13:41.947638+0000","last_fullsized":"2024-04-02T09:13:41.947638+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:16:14.265813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.7","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946858+0000","last_change":"2024-04-02T09:13:27.886740+0000","last_active":"2024-04-02T09:13:41.946858+0000","last_peered":"2024-04-02T09:13:41.946858+0000","last_clean":"2024-04-02T09:13:41.946858+0000","last_became_active":"2024-04-02T09:13:27.886385+0000","last_became_peered":"2024-04-02T09:13:27.886385+0000","last_unstale":"2024-04-02T09:13:41.946858+0000","last_undegraded":"2024-04-02T09:13:41.946858+0000","last_fullsized":"2024-04-02T09:13:41.946858+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:20.383235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734173+0000","last_change":"2024-04-02T09:13:41.954104+0000","last_active":"2024-04-02T09:13:43.734173+0000","last_peered":"2024-04-02T09:13:43.734173+0000","last_clean":"2024-04-02T09:13:43.734173+0000","last_became_active":"2024-04-02T09:13:41.953393+0000","last_became_peered":"2024-04-02T09:13:41.953393+0000","last_unstale":"2024-04-02T09:13:43.734173+0000","last_undegraded":"2024-04-02T09:13:43.734173+0000","last_fullsized":"2024-04-02T09:13:43.734173+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:01:46.436981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942847+0000","last_change":"2024-04-02T09:13:28.402497+0000","last_active":"2024-04-02T09:13:41.942847+0000","last_peered":"2024-04-02T09:13:41.942847+0000","last_clean":"2024-04-02T09:13:41.942847+0000","last_became_active":"2024-04-02T09:13:28.402235+0000","last_became_peered":"2024-04-02T09:13:28.402235+0000","last_unstale":"2024-04-02T09:13:41.942847+0000","last_undegraded":"2024-04-02T09:13:41.942847+0000","last_fullsized":"2024-04-02T09:13:41.942847+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:15.072346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.6","version":"244'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950712+0000","last_change":"2024-04-02T09:13:27.884225+0000","last_active":"2024-04-02T09:13:41.950712+0000","last_peered":"2024-04-02T09:13:41.950712+0000","last_clean":"2024-04-02T09:13:41.950712+0000","last_became_active":"2024-04-02T09:13:27.883536+0000","last_became_peered":"2024-04-02T09:13:27.883536+0000","last_unstale":"2024-04-02T09:13:41.950712+0000","last_undegraded":"2024-04-02T09:13:41.950712+0000","last_fullsized":"2024-04-02T09:13:41.950712+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:50.645932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345681+0000","last_change":"2024-04-02T09:13:41.955484+0000","last_active":"2024-04-02T09:13:43.345681+0000","last_peered":"2024-04-02T09:13:43.345681+0000","last_clean":"2024-04-02T09:13:43.345681+0000","last_became_active":"2024-04-02T09:13:41.953502+0000","last_became_peered":"2024-04-02T09:13:41.953502+0000","last_unstale":"2024-04-02T09:13:43.345681+0000","last_undegraded":"2024-04-02T09:13:43.345681+0000","last_fullsized":"2024-04-02T09:13:43.345681+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:50.626394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953832+0000","last_change":"2024-04-02T09:13:28.387139+0000","last_active":"2024-04-02T09:13:41.953832+0000","last_peered":"2024-04-02T09:13:41.953832+0000","last_clean":"2024-04-02T09:13:41.953832+0000","last_became_active":"2024-04-02T09:13:28.386878+0000","last_became_peered":"2024-04-02T09:13:28.386878+0000","last_unstale":"2024-04-02T09:13:41.953832+0000","last_undegraded":"2024-04-02T09:13:41.953832+0000","last_fullsized":"2024-04-02T09:13:41.953832+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:56:34.681485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948085+0000","last_change":"2024-04-02T09:13:27.882615+0000","last_active":"2024-04-02T09:13:41.948085+0000","last_peered":"2024-04-02T09:13:41.948085+0000","last_clean":"2024-04-02T09:13:41.948085+0000","last_became_active":"2024-04-02T09:13:27.882334+0000","last_became_peered":"2024-04-02T09:13:27.882334+0000","last_unstale":"2024-04-02T09:13:41.948085+0000","last_undegraded":"2024-04-02T09:13:41.948085+0000","last_fullsized":"2024-04-02T09:13:41.948085+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:43.215347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.733778+0000","last_change":"2024-04-02T09:13:41.962699+0000","last_active":"2024-04-02T09:13:43.733778+0000","last_peered":"2024-04-02T09:13:43.733778+0000","last_clean":"2024-04-02T09:13:43.733778+0000","last_became_active":"2024-04-02T09:13:41.960572+0000","last_became_peered":"2024-04-02T09:13:41.960572+0000","last_unstale":"2024-04-02T09:13:43.733778+0000","last_undegraded":"2024-04-02T09:13:43.733778+0000","last_fullsized":"2024-04-02T09:13:43.733778+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:22:21.179882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946870+0000","last_change":"2024-04-02T09:13:28.402303+0000","last_active":"2024-04-02T09:13:41.946870+0000","last_peered":"2024-04-02T09:13:41.946870+0000","last_clean":"2024-04-02T09:13:41.946870+0000","last_became_active":"2024-04-02T09:13:28.399305+0000","last_became_peered":"2024-04-02T09:13:28.399305+0000","last_unstale":"2024-04-02T09:13:41.946870+0000","last_undegraded":"2024-04-02T09:13:41.946870+0000","last_fullsized":"2024-04-02T09:13:41.946870+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:58.927786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945487+0000","last_change":"2024-04-02T09:13:27.881268+0000","last_active":"2024-04-02T09:13:41.945487+0000","last_peered":"2024-04-02T09:13:41.945487+0000","last_clean":"2024-04-02T09:13:41.945487+0000","last_became_active":"2024-04-02T09:13:27.880983+0000","last_became_peered":"2024-04-02T09:13:27.880983+0000","last_unstale":"2024-04-02T09:13:41.945487+0000","last_undegraded":"2024-04-02T09:13:41.945487+0000","last_fullsized":"2024-04-02T09:13:41.945487+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:09.803095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734037+0000","last_change":"2024-04-02T09:13:41.955819+0000","last_active":"2024-04-02T09:13:43.734037+0000","last_peered":"2024-04-02T09:13:43.734037+0000","last_clean":"2024-04-02T09:13:43.734037+0000","last_became_active":"2024-04-02T09:13:41.955568+0000","last_became_peered":"2024-04-02T09:13:41.955568+0000","last_unstale":"2024-04-02T09:13:43.734037+0000","last_undegraded":"2024-04-02T09:13:43.734037+0000","last_fullsized":"2024-04-02T09:13:43.734037+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:44:48.623486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.949252+0000","last_change":"2024-04-02T09:13:28.374285+0000","last_active":"2024-04-02T09:13:41.949252+0000","last_peered":"2024-04-02T09:13:41.949252+0000","last_clean":"2024-04-02T09:13:41.949252+0000","last_became_active":"2024-04-02T09:13:28.373980+0000","last_became_peered":"2024-04-02T09:13:28.373980+0000","last_unstale":"2024-04-02T09:13:41.949252+0000","last_undegraded":"2024-04-02T09:13:41.949252+0000","last_fullsized":"2024-04-02T09:13:41.949252+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:56:34.681485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947216+0000","last_change":"2024-04-02T09:13:27.882249+0000","last_active":"2024-04-02T09:13:41.947216+0000","last_peered":"2024-04-02T09:13:41.947216+0000","last_clean":"2024-04-02T09:13:41.947216+0000","last_became_active":"2024-04-02T09:13:27.881833+0000","last_became_peered":"2024-04-02T09:13:27.881833+0000","last_unstale":"2024-04-02T09:13:41.947216+0000","last_undegraded":"2024-04-02T09:13:41.947216+0000","last_fullsized":"2024-04-02T09:13:41.947216+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:22.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342582+0000","last_change":"2024-04-02T09:13:41.960270+0000","last_active":"2024-04-02T09:13:43.342582+0000","last_peered":"2024-04-02T09:13:43.342582+0000","last_clean":"2024-04-02T09:13:43.342582+0000","last_became_active":"2024-04-02T09:13:41.960009+0000","last_became_peered":"2024-04-02T09:13:41.960009+0000","last_unstale":"2024-04-02T09:13:43.342582+0000","last_undegraded":"2024-04-02T09:13:43.342582+0000","last_fullsized":"2024-04-02T09:13:43.342582+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:37:52.590813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948199+0000","last_change":"2024-04-02T09:13:28.389208+0000","last_active":"2024-04-02T09:13:41.948199+0000","last_peered":"2024-04-02T09:13:41.948199+0000","last_clean":"2024-04-02T09:13:41.948199+0000","last_became_active":"2024-04-02T09:13:28.388923+0000","last_became_peered":"2024-04-02T09:13:28.388923+0000","last_unstale":"2024-04-02T09:13:41.948199+0000","last_undegraded":"2024-04-02T09:13:41.948199+0000","last_fullsized":"2024-04-02T09:13:41.948199+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:35:55.299814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953418+0000","last_change":"2024-04-02T09:13:27.887509+0000","last_active":"2024-04-02T09:13:41.953418+0000","last_peered":"2024-04-02T09:13:41.953418+0000","last_clean":"2024-04-02T09:13:41.953418+0000","last_became_active":"2024-04-02T09:13:27.887260+0000","last_became_peered":"2024-04-02T09:13:27.887260+0000","last_unstale":"2024-04-02T09:13:41.953418+0000","last_undegraded":"2024-04-02T09:13:41.953418+0000","last_fullsized":"2024-04-02T09:13:41.953418+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:28.581007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734767+0000","last_change":"2024-04-02T09:13:41.960376+0000","last_active":"2024-04-02T09:13:43.734767+0000","last_peered":"2024-04-02T09:13:43.734767+0000","last_clean":"2024-04-02T09:13:43.734767+0000","last_became_active":"2024-04-02T09:13:41.960062+0000","last_became_peered":"2024-04-02T09:13:41.960062+0000","last_unstale":"2024-04-02T09:13:43.734767+0000","last_undegraded":"2024-04-02T09:13:43.734767+0000","last_fullsized":"2024-04-02T09:13:43.734767+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:13.220603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.937842+0000","last_change":"2024-04-02T09:13:28.374292+0000","last_active":"2024-04-02T09:13:41.937842+0000","last_peered":"2024-04-02T09:13:41.937842+0000","last_clean":"2024-04-02T09:13:41.937842+0000","last_became_active":"2024-04-02T09:13:28.373689+0000","last_became_peered":"2024-04-02T09:13:28.373689+0000","last_unstale":"2024-04-02T09:13:41.937842+0000","last_undegraded":"2024-04-02T09:13:41.937842+0000","last_fullsized":"2024-04-02T09:13:41.937842+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:08.953388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940440+0000","last_change":"2024-04-02T09:13:27.882492+0000","last_active":"2024-04-02T09:13:41.940440+0000","last_peered":"2024-04-02T09:13:41.940440+0000","last_clean":"2024-04-02T09:13:41.940440+0000","last_became_active":"2024-04-02T09:13:27.882217+0000","last_became_peered":"2024-04-02T09:13:27.882217+0000","last_unstale":"2024-04-02T09:13:41.940440+0000","last_undegraded":"2024-04-02T09:13:41.940440+0000","last_fullsized":"2024-04-02T09:13:41.940440+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:22:56.505431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734928+0000","last_change":"2024-04-02T09:13:41.958277+0000","last_active":"2024-04-02T09:13:43.734928+0000","last_peered":"2024-04-02T09:13:43.734928+0000","last_clean":"2024-04-02T09:13:43.734928+0000","last_became_active":"2024-04-02T09:13:41.956555+0000","last_became_peered":"2024-04-02T09:13:41.956555+0000","last_unstale":"2024-04-02T09:13:43.734928+0000","last_undegraded":"2024-04-02T09:13:43.734928+0000","last_fullsized":"2024-04-02T09:13:43.734928+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:46.027233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940350+0000","last_change":"2024-04-02T09:13:28.381355+0000","last_active":"2024-04-02T09:13:41.940350+0000","last_peered":"2024-04-02T09:13:41.940350+0000","last_clean":"2024-04-02T09:13:41.940350+0000","last_became_active":"2024-04-02T09:13:28.381068+0000","last_became_peered":"2024-04-02T09:13:28.381068+0000","last_unstale":"2024-04-02T09:13:41.940350+0000","last_undegraded":"2024-04-02T09:13:41.940350+0000","last_fullsized":"2024-04-02T09:13:41.940350+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:50.895638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953078+0000","last_change":"2024-04-02T09:13:27.883425+0000","last_active":"2024-04-02T09:13:41.953078+0000","last_peered":"2024-04-02T09:13:41.953078+0000","last_clean":"2024-04-02T09:13:41.953078+0000","last_became_active":"2024-04-02T09:13:27.883106+0000","last_became_peered":"2024-04-02T09:13:27.883106+0000","last_unstale":"2024-04-02T09:13:41.953078+0000","last_undegraded":"2024-04-02T09:13:41.953078+0000","last_fullsized":"2024-04-02T09:13:41.953078+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:23.271534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.733563+0000","last_change":"2024-04-02T09:13:41.960258+0000","last_active":"2024-04-02T09:13:43.733563+0000","last_peered":"2024-04-02T09:13:43.733563+0000","last_clean":"2024-04-02T09:13:43.733563+0000","last_became_active":"2024-04-02T09:13:41.960011+0000","last_became_peered":"2024-04-02T09:13:41.960011+0000","last_unstale":"2024-04-02T09:13:43.733563+0000","last_undegraded":"2024-04-02T09:13:43.733563+0000","last_fullsized":"2024-04-02T09:13:43.733563+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:18.838465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940567+0000","last_change":"2024-04-02T09:13:28.402819+0000","last_active":"2024-04-02T09:13:41.940567+0000","last_peered":"2024-04-02T09:13:41.940567+0000","last_clean":"2024-04-02T09:13:41.940567+0000","last_became_active":"2024-04-02T09:13:28.402563+0000","last_became_peered":"2024-04-02T09:13:28.402563+0000","last_unstale":"2024-04-02T09:13:41.940567+0000","last_undegraded":"2024-04-02T09:13:41.940567+0000","last_fullsized":"2024-04-02T09:13:41.940567+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:27.049697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1f","version":"243'4","reported_seq":19,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943081+0000","last_change":"2024-04-02T09:13:27.883417+0000","last_active":"2024-04-02T09:13:41.943081+0000","last_peered":"2024-04-02T09:13:41.943081+0000","last_clean":"2024-04-02T09:13:41.943081+0000","last_became_active":"2024-04-02T09:13:27.882944+0000","last_became_peered":"2024-04-02T09:13:27.882944+0000","last_unstale":"2024-04-02T09:13:41.943081+0000","last_undegraded":"2024-04-02T09:13:41.943081+0000","last_fullsized":"2024-04-02T09:13:41.943081+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:05.331734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"229'370","reported_seq":1312,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948041+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:13:41.948041+0000","last_peered":"2024-04-02T09:13:41.948041+0000","last_clean":"2024-04-02T09:13:41.948041+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:13:41.948041+0000","last_undegraded":"2024-04-02T09:13:41.948041+0000","last_fullsized":"2024-04-02T09:13:41.948041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3603,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576001,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60436,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311404,"statfs":{"total":96636764160,"available":96574877696,"internally_reserved":0,"allocated":3153920,"data_stored":1604364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.40400000000000003,"5min":0.309,"15min":0.309},"max":{"1min":8.016,"5min":8.016,"15min":8.016},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.71999999999999997,"15min":0.72999999999999998},"min":{"1min":0.39900000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.1790000000000003,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.70699999999999996}]},{"osd":1,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.76800000000000002,"15min":0.751},"min":{"1min":0.44800000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.5819999999999999,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.70899999999999996,"15min":0.71999999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.42399999999999999},"max":{"1min":4.2569999999999997,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.72099999999999997}]},{"osd":2,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.79600000000000004,"15min":0.77800000000000002},"min":{"1min":0.437,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.9790000000000001,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.80700000000000005,"15min":0.81200000000000006},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":4.8819999999999997,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.871}]},{"osd":3,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.77100000000000002,"15min":0.75900000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.0530000000000008,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.999,"5min":0.79900000000000004,"15min":0.77700000000000002},"min":{"1min":0.54100000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":5.093,"5min":5.093,"15min":5.093},"last":0.81999999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.66700000000000004,"15min":0.628},"min":{"1min":0.38300000000000001,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":5.4589999999999996,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.1060000000000001,"5min":0.79700000000000004,"15min":0.74399999999999999},"min":{"1min":0.50600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.9409999999999998,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.63200000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.73799999999999999,"15min":0.69299999999999995},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":5.1340000000000003,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.77400000000000002,"15min":0.73599999999999999},"min":{"1min":0.58899999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.492,"5min":3.492,"15min":3.492},"last":0.60699999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.72199999999999998,"15min":0.69699999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45800000000000002},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.075,"5min":0.78500000000000003,"15min":0.71699999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41099999999999998},"max":{"1min":7.8899999999999997,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575999,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42368,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329472,"statfs":{"total":96636764160,"available":96593379328,"internally_reserved":0,"allocated":3129344,"data_stored":1600437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.69799999999999995,"15min":0.72899999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.90600000000000003,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.71099999999999997,"15min":0.73899999999999999},"min":{"1min":0.53600000000000003,"5min":0.47899999999999998,"15min":0.45800000000000002},"max":{"1min":0.88300000000000001,"5min":0.93200000000000005,"15min":1.0600000000000001},"last":0.61399999999999999}]},{"osd":1,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68799999999999994,"15min":0.71299999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86299999999999999,"5min":1.044,"15min":1.212},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.71299999999999997,"15min":0.752},"min":{"1min":0.45800000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.018,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.51500000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.71799999999999997,"15min":0.747},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.046,"5min":1.046,"15min":1.2789999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.71999999999999997,"15min":0.73899999999999999},"min":{"1min":0.44,"5min":0.439,"15min":0.439},"max":{"1min":0.93500000000000005,"5min":0.95999999999999996,"15min":1.226},"last":0.77200000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72399999999999998,"15min":0.76300000000000001},"min":{"1min":0.53900000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97799999999999998,"5min":1.141,"15min":1.2549999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.755,"15min":0.77000000000000002},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.67400000000000004}]},{"osd":4,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.60999999999999999,"15min":0.624},"min":{"1min":0.442,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.91200000000000003,"5min":0.96699999999999997,"15min":1.1259999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.41999999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.1879999999999999},"last":0.75600000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84199999999999997,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69999999999999996,"15min":0.72499999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":0.96399999999999997,"5min":1.034,"15min":1.204},"last":0.74199999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.72999999999999998,"15min":0.74099999999999999},"min":{"1min":0.53600000000000003,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.012,"5min":1.21,"15min":1.2370000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72599999999999998,"15min":0.73899999999999999},"min":{"1min":0.52400000000000002,"5min":0.51400000000000001,"15min":0.49399999999999999},"max":{"1min":0.91900000000000004,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.81699999999999995}]}]},{"osd":1,"up_from":13,"seq":55834576000,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52812,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319028,"statfs":{"total":96636764160,"available":96582684672,"internally_reserved":0,"allocated":3727360,"data_stored":2191299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.433,"15min":0.45000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.58999999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48999999999999999,"15min":0.50900000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73199999999999998,"5min":0.76000000000000001,"15min":0.79800000000000004},"last":0.64700000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.504,"15min":0.505},"min":{"1min":0.38900000000000001,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56699999999999995,"15min":0.59199999999999997},"min":{"1min":0.44500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68100000000000005,"5min":0.77500000000000002,"15min":0.85599999999999998},"last":0.67400000000000004}]},{"osd":3,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.41099999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.69999999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.55200000000000005,"15min":0.56100000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.58899999999999997,"5min":0.86699999999999999,"15min":0.878},"last":0.59799999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76900000000000002,"15min":0.78400000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.433},"max":{"1min":0.90700000000000003,"5min":1.1120000000000001,"15min":1.1479999999999999},"last":1.1879999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.76400000000000001,"15min":0.78500000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.41799999999999998},"max":{"1min":0.95699999999999996,"5min":1.05,"15min":1.25},"last":0.873}]},{"osd":5,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76100000000000001,"15min":0.78800000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.96399999999999997,"5min":1.115,"15min":1.321},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.79700000000000004,"15min":0.80700000000000005},"min":{"1min":0.56899999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93799999999999994,"5min":1.0980000000000001,"15min":1.173},"last":1.0049999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.79100000000000004,"15min":0.81299999999999994},"min":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.48099999999999998},"max":{"1min":0.96099999999999997,"5min":1.159,"15min":1.2210000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80800000000000005,"15min":0.80900000000000005},"min":{"1min":0.56799999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0009999999999999,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.94099999999999995}]},{"osd":7,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.79900000000000004,"15min":0.81999999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0029999999999999,"5min":1.1080000000000001,"15min":1.236},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81899999999999995,"15min":0.83399999999999996},"min":{"1min":0.60899999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":0.97499999999999998,"5min":1.0720000000000001,"15min":1.2789999999999999},"last":1.1020000000000001}]}]},{"osd":0,"up_from":13,"seq":55834576000,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52220,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319620,"statfs":{"total":96636764160,"available":96583290880,"internally_reserved":0,"allocated":3129344,"data_stored":1599420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.441,"15min":0.45500000000000002},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.66700000000000004,"5min":0.69899999999999995,"15min":0.86699999999999999},"last":0.498},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.49399999999999999,"15min":0.50700000000000001},"min":{"1min":0.33900000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64800000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.53100000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.502,"15min":0.51100000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.82699999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53400000000000003,"15min":0.55500000000000005},"min":{"1min":0.38300000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72699999999999998,"5min":0.999,"15min":0.999},"last":0.63300000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54200000000000004,"15min":0.56699999999999995},"min":{"1min":0.35699999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.71899999999999997,"5min":0.79800000000000004,"15min":0.85699999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.61599999999999999,"15min":0.63800000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.73599999999999999,"5min":0.92000000000000004,"15min":0.96399999999999997},"last":0.65500000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.80500000000000005,"15min":0.83099999999999996},"min":{"1min":0.55000000000000004,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":0.97499999999999998,"5min":1.1950000000000001,"15min":1.2769999999999999},"last":0.745},{"interface":"front","average":{"1min":0.751,"5min":0.78500000000000003,"15min":0.82199999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99299999999999999,"5min":1.169,"15min":1.169},"last":0.873}]},{"osd":5,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.77900000000000003,"15min":0.82399999999999995},"min":{"1min":0.54000000000000004,"5min":0.497,"15min":0.497},"max":{"1min":1,"5min":1.145,"15min":1.232},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.96599999999999997,"5min":1.21,"15min":1.21},"last":0.90600000000000003}]},{"osd":6,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.77600000000000002,"15min":0.79800000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91100000000000003,"5min":1.1830000000000001,"15min":1.196},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.79200000000000004,"15min":0.81200000000000006},"min":{"1min":0.56499999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96199999999999997,"5min":1.246,"15min":1.246},"last":0.96799999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82699999999999996,"15min":0.85499999999999998},"min":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0069999999999999,"5min":1.292,"15min":1.292},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.84799999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.95499999999999996,"5min":1.1719999999999999,"15min":1.2589999999999999},"last":0.92300000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575999,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52236,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319604,"statfs":{"total":96636764160,"available":96583274496,"internally_reserved":0,"allocated":3141632,"data_stored":1601913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.44600000000000001,"15min":0.44},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.72999999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.48399999999999999,"15min":0.498},"min":{"1min":0.29199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":1.4930000000000001},"last":0.53600000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.504},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69699999999999995,"5min":0.75800000000000001,"15min":0.80600000000000005},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53600000000000003,"15min":0.55400000000000005},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.59599999999999997,"5min":0.75,"15min":1.4530000000000001},"last":0.58999999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52900000000000003,"15min":0.55700000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.745,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60499999999999998,"15min":0.63200000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.64000000000000001,"5min":0.83699999999999997,"15min":1.0740000000000001},"last":0.60799999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.82099999999999995,"5min":1.121,"15min":1.121},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.77300000000000002,"15min":0.79100000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.96799999999999997,"5min":1.179,"15min":1.5329999999999999},"last":0.93600000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.78400000000000003,"15min":0.78600000000000003},"min":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.82999999999999996,"5min":1.99,"15min":1.99},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.78500000000000003,"15min":0.79900000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.89600000000000002,"5min":1.101,"15min":1.1259999999999999},"last":0.88}]},{"osd":6,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.997,"5min":1.173,"15min":1.5940000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":0.80400000000000005},"min":{"1min":0.56200000000000006,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.92300000000000004,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.83499999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.877,"5min":1.2110000000000001,"15min":1.5649999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.80700000000000005,"15min":0.82699999999999996},"min":{"1min":0.61499999999999999,"5min":0.497,"15min":0.497},"max":{"1min":0.95899999999999996,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.90600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575999,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52232,"kb_used_data":3064,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319608,"statfs":{"total":96636764160,"available":96583278592,"internally_reserved":0,"allocated":3137536,"data_stored":1603096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.42799999999999999,"15min":0.44600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57199999999999995,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.46899999999999997,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60199999999999998,"5min":0.71199999999999997,"15min":0.876},"last":0.34699999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48699999999999999,"15min":0.499},"min":{"1min":0.36799999999999999,"5min":0.33200000000000002,"15min":0.32500000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.90300000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55100000000000005,"15min":0.57199999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67700000000000005,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.56399999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.60099999999999998,"15min":0.626},"min":{"1min":0.40100000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71399999999999997,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.58799999999999997,"15min":0.61599999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.69499999999999995,"5min":0.81699999999999995,"15min":1.0840000000000001},"last":0.61399999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.72299999999999998,"15min":0.752},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89100000000000001,"5min":1.071,"15min":1.0800000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.74299999999999999,"15min":0.76800000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.879,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.57399999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.74399999999999999,"15min":0.77500000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91800000000000004,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.76300000000000001,"15min":0.78100000000000003},"min":{"1min":0.54200000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.85699999999999998,"5min":1.266,"15min":1.266},"last":0.51100000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.76500000000000001,"15min":0.78400000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97099999999999997,"5min":1.284,"15min":1.284},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.54800000000000004,"5min":0.495,"15min":0.495},"max":{"1min":0.95999999999999996,"5min":1.2,"15min":1.2},"last":0.64700000000000002}]},{"osd":7,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.77900000000000003,"15min":0.80800000000000005},"min":{"1min":0.58699999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.93500000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.79300000000000004,"15min":0.81799999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.014,"5min":1.248,"15min":1.248},"last":0.70099999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575999,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48136,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323704,"statfs":{"total":96636764160,"available":96587472896,"internally_reserved":0,"allocated":3141632,"data_stored":1602425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.66400000000000003,"15min":0.68700000000000006},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":0.80000000000000004,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.65100000000000002,"15min":0.67700000000000005},"min":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.78100000000000003,"5min":0.86199999999999999,"15min":1.075},"last":0.58499999999999996}]},{"osd":1,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.72199999999999998,"15min":0.754},"min":{"1min":0.49399999999999999,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":0.90500000000000003,"5min":1.0549999999999999,"15min":1.069},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.73599999999999999,"15min":0.76500000000000001},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":0.92500000000000004,"5min":1.048,"15min":1.048},"last":0.72699999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.67300000000000004,"15min":0.70799999999999996},"min":{"1min":0.47099999999999997,"5min":0.438,"15min":0.438},"max":{"1min":0.81299999999999994,"5min":1.077,"15min":1.077},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.68999999999999995,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85599999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.70499999999999996}]},{"osd":3,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.51500000000000001,"5min":0.501,"15min":0.47399999999999998},"max":{"1min":0.94599999999999995,"5min":0.998,"15min":1.018},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.028},"last":0.55100000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.81899999999999995,"5min":1.008,"15min":1.008},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.497}]},{"osd":6,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65500000000000003,"15min":0.64400000000000002},"min":{"1min":0.48499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85099999999999998,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68999999999999995,"15min":0.69199999999999995},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":1.0049999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.48399999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96099999999999997,"5min":1.002,"15min":1.0329999999999999},"last":0.627},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.77500000000000002,"15min":0.78100000000000003},"min":{"1min":0.56100000000000005,"5min":0.5,"15min":0.5},"max":{"1min":1.038,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.73799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576002,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52796,"kb_used_data":3632,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319044,"statfs":{"total":96636764160,"available":96582701056,"internally_reserved":0,"allocated":3719168,"data_stored":2189730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.82699999999999996,"5min":0.98699999999999999,"15min":1.004},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.68999999999999995,"15min":0.71999999999999997},"min":{"1min":0.48399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.028,"5min":1.085,"15min":1.085},"last":0.67300000000000004}]},{"osd":1,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.68100000000000005,"15min":0.71499999999999997},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84399999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70999999999999996,"15min":0.72499999999999998},"min":{"1min":0.52300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88500000000000001,"5min":0.95299999999999996,"15min":1.052},"last":0.81599999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.71599999999999997,"15min":0.73799999999999999},"min":{"1min":0.47199999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97299999999999998,"5min":1.004,"15min":1.014},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.69499999999999995,"15min":0.73699999999999999},"min":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.81799999999999995,"5min":1.018,"15min":1.0680000000000001},"last":0.66100000000000003}]},{"osd":3,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.747,"15min":0.77700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.88500000000000001,"5min":0.98899999999999999,"15min":1.083},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77000000000000002,"15min":0.79000000000000004},"min":{"1min":0.58399999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1020000000000001},"last":0.71199999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.40699999999999997,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.054},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":0.98799999999999999,"5min":1.145,"15min":1.145},"last":0.84499999999999997}]},{"osd":6,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67900000000000005,"15min":0.67300000000000004},"min":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96799999999999997,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71399999999999997,"15min":0.71099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.94899999999999995,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.91900000000000004}]},{"osd":7,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73999999999999999,"15min":0.73799999999999999},"min":{"1min":0.46500000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.0820000000000001,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76300000000000001,"15min":0.76600000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1519999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.76300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:13:47.797 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-02T09:13:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.020+0000 7faf71200700 1 Processor -- start 2024-04-02T09:13:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.020+0000 7faf71200700 1 -- start start 2024-04-02T09:13:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.021+0000 7faf71200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faf6c0cb3f0 0x7faf6c0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.021+0000 7faf71200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf6c0ac600 0x7faf6c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.021+0000 7faf71200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6c15fd50 con 0x7faf6c15b5d0 2024-04-02T09:13:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.021+0000 7faf71200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf6c15b9b0 con 0x7faf6c0cb3f0 2024-04-02T09:13:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.021+0000 7faf71200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c0aca50 con 0x7faf6c0ac600 2024-04-02T09:13:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.022+0000 7faf69400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faf6c0cb3f0 0x7faf6c0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.022+0000 7faf69400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faf6c0cb3f0 0x7faf6c0ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55358/0 (socket says 172.21.15.89:55358) 2024-04-02T09:13:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.022+0000 7faf69400700 1 -- 172.21.15.89:0/254624539 learned_addr learned my addr 172.21.15.89:0/254624539 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.022+0000 7faf69e00700 1 --1- 172.21.15.89:0/254624539 >> v1:172.21.15.89:6789/0 conn(0x7faf6c15b5d0 0x7faf6c0ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58324/0 (socket says 172.21.15.89:58324) 2024-04-02T09:13:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.022+0000 7faf6a800700 1 --2- 172.21.15.89:0/254624539 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf6c0ac600 0x7faf6c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.022+0000 7faf69400700 1 -- 172.21.15.89:0/254624539 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf6c0ac600 msgr2=0x7faf6c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.022+0000 7faf69400700 1 --2- 172.21.15.89:0/254624539 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf6c0ac600 0x7faf6c06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.022+0000 7faf69400700 1 -- 172.21.15.89:0/254624539 >> v1:172.21.15.89:6789/0 conn(0x7faf6c15b5d0 legacy=0x7faf6c0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.022+0000 7faf69400700 1 -- 172.21.15.89:0/254624539 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c15f140 con 0x7faf6c0cb3f0 2024-04-02T09:13:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.022+0000 7faf68a00700 1 -- 172.21.15.89:0/254624539 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231247661 0 0) 0x7faf6c15fd50 con 0x7faf6c15b5d0 2024-04-02T09:13:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.022+0000 7faf69400700 1 --2- 172.21.15.89:0/254624539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf6c0cb3f0 0x7faf6c0ac0c0 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7faf580098f0 tx=0x7faf58009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2325ad0c70afdfc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.023+0000 7faf68a00700 1 -- 172.21.15.89:0/254624539 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5800b8a0 con 0x7faf6c0cb3f0 2024-04-02T09:13:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.023+0000 7faf68a00700 1 -- 172.21.15.89:0/254624539 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5800ba00 con 0x7faf6c0cb3f0 2024-04-02T09:13:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.023+0000 7faf68a00700 1 -- 172.21.15.89:0/254624539 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5800bd90 con 0x7faf6c0cb3f0 2024-04-02T09:13:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.023+0000 7faf71200700 1 -- 172.21.15.89:0/254624539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf6c0cb3f0 msgr2=0x7faf6c0ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.023+0000 7faf71200700 1 --2- 172.21.15.89:0/254624539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf6c0cb3f0 0x7faf6c0ac0c0 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7faf580098f0 tx=0x7faf58009c00 comp rx=0 tx=0).stop 2024-04-02T09:13:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.023+0000 7faf71200700 1 -- 172.21.15.89:0/254624539 shutdown_connections 2024-04-02T09:13:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.023+0000 7faf71200700 1 --2- 172.21.15.89:0/254624539 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf6c0ac600 0x7faf6c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.023+0000 7faf71200700 1 --2- 172.21.15.89:0/254624539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf6c0cb3f0 0x7faf6c0ac0c0 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.023+0000 7faf71200700 1 -- 172.21.15.89:0/254624539 >> 172.21.15.89:0/254624539 conn(0x7faf6c0cf190 msgr2=0x7faf6c0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.023+0000 7faf71200700 1 -- 172.21.15.89:0/254624539 shutdown_connections 2024-04-02T09:13:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.023+0000 7faf71200700 1 -- 172.21.15.89:0/254624539 wait complete. 2024-04-02T09:13:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.025+0000 7faf71200700 1 Processor -- start 2024-04-02T09:13:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.025+0000 7faf71200700 1 -- start start 2024-04-02T09:13:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.026+0000 7faf71200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf6c0ac600 0x7faf6c0ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.026+0000 7faf71200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf6c15b5d0 0x7faf6c0cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.026+0000 7faf71200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf6c0ccee0 0x7faf6c1f6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.026+0000 7faf71200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c070af0 con 0x7faf6c15b5d0 2024-04-02T09:13:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.026+0000 7faf71200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c0cdc40 con 0x7faf6c0ccee0 2024-04-02T09:13:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.026+0000 7faf71200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c0cd3a0 con 0x7faf6c0ac600 2024-04-02T09:13:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.026+0000 7faf69400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf6c15b5d0 0x7faf6c0cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.026+0000 7faf69400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf6c15b5d0 0x7faf6c0cc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55366/0 (socket says 172.21.15.89:55366) 2024-04-02T09:13:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.026+0000 7faf69400700 1 -- 172.21.15.89:0/2480005756 learned_addr learned my addr 172.21.15.89:0/2480005756 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf69e00700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf6c0ac600 0x7faf6c0ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf6a800700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf6c0ccee0 0x7faf6c1f6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf69400700 1 -- 172.21.15.89:0/2480005756 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf6c0ac600 msgr2=0x7faf6c0ca030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf69400700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf6c0ac600 0x7faf6c0ca030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf69400700 1 -- 172.21.15.89:0/2480005756 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf6c0ccee0 msgr2=0x7faf6c1f6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf69400700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf6c0ccee0 0x7faf6c1f6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf69400700 1 -- 172.21.15.89:0/2480005756 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf58009550 con 0x7faf6c15b5d0 2024-04-02T09:13:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf6a800700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf6c0ccee0 0x7faf6c1f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf69e00700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf6c0ac600 0x7faf6c0ca030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf69400700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf6c15b5d0 0x7faf6c0cc9a0 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7faf58009c40 tx=0x7faf58004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf62000700 1 -- 172.21.15.89:0/2480005756 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58004380 con 0x7faf6c15b5d0 2024-04-02T09:13:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf62000700 1 -- 172.21.15.89:0/2480005756 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5801f070 con 0x7faf6c15b5d0 2024-04-02T09:13:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf71200700 1 -- 172.21.15.89:0/2480005756 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6c0cd680 con 0x7faf6c15b5d0 2024-04-02T09:13:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf62000700 1 -- 172.21.15.89:0/2480005756 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf580036a0 con 0x7faf6c15b5d0 2024-04-02T09:13:48.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.027+0000 7faf71200700 1 -- 172.21.15.89:0/2480005756 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6c0ca570 con 0x7faf6c15b5d0 2024-04-02T09:13:48.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.028+0000 7faf71200700 1 -- 172.21.15.89:0/2480005756 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6c0adb10 con 0x7faf6c15b5d0 2024-04-02T09:13:48.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.029+0000 7faf62000700 1 -- 172.21.15.89:0/2480005756 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7faf58021030 con 0x7faf6c15b5d0 2024-04-02T09:13:48.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.031+0000 7faf62000700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7faf4c06aed0 0x7faf4c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:48.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.031+0000 7faf62000700 1 -- 172.21.15.89:0/2480005756 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7faf5808b100 con 0x7faf6c15b5d0 2024-04-02T09:13:48.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.032+0000 7faf69e00700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7faf4c06aed0 0x7faf4c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:48.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.032+0000 7faf69e00700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7faf4c06aed0 0x7faf4c06d390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faf5c000f60 tx=0x7faf5c007cc0 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.036+0000 7faf62000700 1 -- 172.21.15.89:0/2480005756 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faf58055a50 con 0x7faf6c15b5d0 2024-04-02T09:13:48.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.317+0000 7faf71200700 1 -- 172.21.15.89:0/2480005756 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7faf6c0b4640 con 0x7faf6c15b5d0 2024-04-02T09:13:48.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.318+0000 7faf62000700 1 -- 172.21.15.89:0/2480005756 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1125 (secure 0 0 0) 0x7faf58015480 con 0x7faf6c15b5d0 2024-04-02T09:13:48.316 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:48.317 INFO:teuthology.orchestra.run.smithi089.stdout:{"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5900,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":248,"num_osds":8,"num_up_osds":8,"osd_up_since":1712043340,"num_in_osds":8,"osd_in_since":1712043332,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":193}],"num_pgs":193,"num_pools":5,"num_objects":46,"data_bytes":596333,"bytes_used":423153664,"bytes_avail":772670959616,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":125,"id":15,"up":1,"in":1,"max":1,"id":16,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":16,"rank":0,"name":"a","status":"up:active","gid":11958},{"filesystem_id":15,"rank":0,"name":"d","status":"up:active","gid":11964}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":23,"modified":"2024-04-02T09:11:01.562287+0000","services":{}},"progress_events":{}} 2024-04-02T09:13:48.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.320+0000 7faf71200700 1 -- 172.21.15.89:0/2480005756 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7faf4c06aed0 msgr2=0x7faf4c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:48.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.320+0000 7faf71200700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7faf4c06aed0 0x7faf4c06d390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faf5c000f60 tx=0x7faf5c007cc0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.320+0000 7faf71200700 1 -- 172.21.15.89:0/2480005756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf6c15b5d0 msgr2=0x7faf6c0cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:48.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.320+0000 7faf71200700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf6c15b5d0 0x7faf6c0cc9a0 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7faf58009c40 tx=0x7faf58004750 comp rx=0 tx=0).stop 2024-04-02T09:13:48.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.320+0000 7faf71200700 1 -- 172.21.15.89:0/2480005756 shutdown_connections 2024-04-02T09:13:48.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.320+0000 7faf71200700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7faf4c06aed0 0x7faf4c06d390 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.320+0000 7faf71200700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf6c0ccee0 0x7faf6c1f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.320+0000 7faf71200700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faf6c0ac600 0x7faf6c0ca030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.320+0000 7faf71200700 1 --2- 172.21.15.89:0/2480005756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf6c15b5d0 0x7faf6c0cc9a0 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.320+0000 7faf71200700 1 -- 172.21.15.89:0/2480005756 >> 172.21.15.89:0/2480005756 conn(0x7faf6c0cf190 msgr2=0x7faf6c0af090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:48.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.320+0000 7faf71200700 1 -- 172.21.15.89:0/2480005756 shutdown_connections 2024-04-02T09:13:48.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.320+0000 7faf71200700 1 -- 172.21.15.89:0/2480005756 wait complete. 2024-04-02T09:13:48.336 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-02T09:13:48.336 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:13:48.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.530+0000 7f2e77c00700 1 Processor -- start 2024-04-02T09:13:48.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.530+0000 7f2e77c00700 1 -- start start 2024-04-02T09:13:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e77c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e7006a790 0x7f2e7006ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e77c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2e700cebd0 0x7f2e700cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e77c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7006f850 con 0x7f2e700afbe0 2024-04-02T09:13:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e77c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7006e3f0 con 0x7f2e700cebd0 2024-04-02T09:13:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e77c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7006b0b0 con 0x7f2e7006a790 2024-04-02T09:13:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e6fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2e700afbe0 0x7f2e7006dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58334/0 (socket says 172.21.15.89:58334) 2024-04-02T09:13:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e6fe00700 1 -- 172.21.15.89:0/3263132706 learned_addr learned my addr 172.21.15.89:0/3263132706 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e75c00700 1 --2- 172.21.15.89:0/3263132706 >> v2:172.21.15.89:3300/0 conn(0x7f2e700cebd0 0x7f2e700cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:48.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e75200700 1 --2- 172.21.15.89:0/3263132706 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e7006a790 0x7f2e7006ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e6f400700 1 -- 172.21.15.89:0/3263132706 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842434892 0 0) 0x7f2e7006f850 con 0x7f2e700afbe0 2024-04-02T09:13:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e6f400700 1 -- 172.21.15.89:0/3263132706 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e580035b0 con 0x7f2e700afbe0 2024-04-02T09:13:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e75c00700 1 -- 172.21.15.89:0/3263132706 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e7006a790 msgr2=0x7f2e7006ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e75c00700 1 --2- 172.21.15.89:0/3263132706 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e7006a790 0x7f2e7006ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e75c00700 1 -- 172.21.15.89:0/3263132706 >> v1:172.21.15.89:6789/0 conn(0x7f2e700afbe0 legacy=0x7f2e7006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e75c00700 1 -- 172.21.15.89:0/3263132706 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7006fad0 con 0x7f2e700cebd0 2024-04-02T09:13:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.532+0000 7f2e75200700 1 --2- 172.21.15.89:0/3263132706 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e7006a790 0x7f2e7006ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.533+0000 7f2e75c00700 1 --2- 172.21.15.89:0/3263132706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e700cebd0 0x7f2e700cf040 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009d90 tx=0x7f2e6800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1b872c6224ab0c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.533+0000 7f2e6f400700 1 -- 172.21.15.89:0/3263132706 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6800e070 con 0x7f2e700cebd0 2024-04-02T09:13:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.533+0000 7f2e6f400700 1 -- 172.21.15.89:0/3263132706 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6800bb20 con 0x7f2e700cebd0 2024-04-02T09:13:48.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.533+0000 7f2e6f400700 1 -- 172.21.15.89:0/3263132706 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e68012430 con 0x7f2e700cebd0 2024-04-02T09:13:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.533+0000 7f2e77c00700 1 -- 172.21.15.89:0/3263132706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e700cebd0 msgr2=0x7f2e700cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.533+0000 7f2e77c00700 1 --2- 172.21.15.89:0/3263132706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e700cebd0 0x7f2e700cf040 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009d90 tx=0x7f2e6800b330 comp rx=0 tx=0).stop 2024-04-02T09:13:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.534+0000 7f2e77c00700 1 -- 172.21.15.89:0/3263132706 shutdown_connections 2024-04-02T09:13:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.534+0000 7f2e77c00700 1 --2- 172.21.15.89:0/3263132706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e700cebd0 0x7f2e700cf040 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.534+0000 7f2e77c00700 1 --2- 172.21.15.89:0/3263132706 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e7006a790 0x7f2e7006ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.534+0000 7f2e77c00700 1 -- 172.21.15.89:0/3263132706 >> 172.21.15.89:0/3263132706 conn(0x7f2e700b5ab0 msgr2=0x7f2e700b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.534+0000 7f2e77c00700 1 -- 172.21.15.89:0/3263132706 shutdown_connections 2024-04-02T09:13:48.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.534+0000 7f2e77c00700 1 -- 172.21.15.89:0/3263132706 wait complete. 2024-04-02T09:13:48.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.535+0000 7f2e77c00700 1 Processor -- start 2024-04-02T09:13:48.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.536+0000 7f2e77c00700 1 -- start start 2024-04-02T09:13:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.536+0000 7f2e77c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e7006a790 0x7f2e701f6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.536+0000 7f2e77c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e700afbe0 0x7f2e700ad350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.536+0000 7f2e77c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e700cebd0 0x7f2e700ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.536+0000 7f2e77c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e701f64e0 con 0x7f2e700afbe0 2024-04-02T09:13:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.536+0000 7f2e77c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e701f39d0 con 0x7f2e7006a790 2024-04-02T09:13:48.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.536+0000 7f2e77c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e701f3cb0 con 0x7f2e700cebd0 2024-04-02T09:13:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.537+0000 7f2e6fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e700afbe0 0x7f2e700ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.537+0000 7f2e6fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e700afbe0 0x7f2e700ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55386/0 (socket says 172.21.15.89:55386) 2024-04-02T09:13:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.537+0000 7f2e6fe00700 1 -- 172.21.15.89:0/1368887806 learned_addr learned my addr 172.21.15.89:0/1368887806 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.537+0000 7f2e75c00700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e700cebd0 0x7f2e700ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.537+0000 7f2e75200700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e7006a790 0x7f2e701f6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.537+0000 7f2e6fe00700 1 -- 172.21.15.89:0/1368887806 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e700cebd0 msgr2=0x7f2e700ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.537+0000 7f2e6fe00700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e700cebd0 0x7f2e700ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.537+0000 7f2e6fe00700 1 -- 172.21.15.89:0/1368887806 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e7006a790 msgr2=0x7f2e701f6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.537+0000 7f2e6fe00700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e7006a790 0x7f2e701f6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.537+0000 7f2e6fe00700 1 -- 172.21.15.89:0/1368887806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68009a70 con 0x7f2e700afbe0 2024-04-02T09:13:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.537+0000 7f2e75c00700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e700cebd0 0x7f2e700ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.537+0000 7f2e75200700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e7006a790 0x7f2e701f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.537+0000 7f2e6fe00700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e700afbe0 0x7f2e700ad350 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c001eb0 tx=0x7f2e5c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:48.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.537+0000 7f2e6cc00700 1 -- 172.21.15.89:0/1368887806 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c005d70 con 0x7f2e700afbe0 2024-04-02T09:13:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.538+0000 7f2e6cc00700 1 -- 172.21.15.89:0/1368887806 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e5c0070d0 con 0x7f2e700afbe0 2024-04-02T09:13:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.538+0000 7f2e77c00700 1 -- 172.21.15.89:0/1368887806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e701f6730 con 0x7f2e700afbe0 2024-04-02T09:13:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.538+0000 7f2e6cc00700 1 -- 172.21.15.89:0/1368887806 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c007460 con 0x7f2e700afbe0 2024-04-02T09:13:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.538+0000 7f2e77c00700 1 -- 172.21.15.89:0/1368887806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e701541b0 con 0x7f2e700afbe0 2024-04-02T09:13:48.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.539+0000 7f2e77c00700 1 -- 172.21.15.89:0/1368887806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e70067720 con 0x7f2e700afbe0 2024-04-02T09:13:48.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.540+0000 7f2e6cc00700 1 -- 172.21.15.89:0/1368887806 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f2e5c0075c0 con 0x7f2e700afbe0 2024-04-02T09:13:48.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.541+0000 7f2e6cc00700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2e5006aed0 0x7f2e5006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:48.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.541+0000 7f2e6cc00700 1 -- 172.21.15.89:0/1368887806 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f2e5c0890e0 con 0x7f2e700afbe0 2024-04-02T09:13:48.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.541+0000 7f2e75200700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2e5006aed0 0x7f2e5006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:48.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.542+0000 7f2e75200700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2e5006aed0 0x7f2e5006d390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2e64009960 tx=0x7f2e64008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:48.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.545+0000 7f2e6cc00700 1 -- 172.21.15.89:0/1368887806 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e5c053ad0 con 0x7f2e700afbe0 2024-04-02T09:13:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.742+0000 7f2e77c00700 1 -- 172.21.15.89:0/1368887806 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2e700cfb70 con 0x7f2e5006aed0 2024-04-02T09:13:48.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.757+0000 7f2e6cc00700 1 -- 172.21.15.89:0/1368887806 <== mgr.11613 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528603 (secure 0 0 0) 0x7f2e64009570 con 0x7f2e5006aed0 2024-04-02T09:13:48.763 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:48.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.763+0000 7f2e77c00700 1 -- 172.21.15.89:0/1368887806 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2e5006aed0 msgr2=0x7f2e5006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:48.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.763+0000 7f2e77c00700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2e5006aed0 0x7f2e5006d390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2e64009960 tx=0x7f2e64008040 comp rx=0 tx=0).stop 2024-04-02T09:13:48.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.763+0000 7f2e77c00700 1 -- 172.21.15.89:0/1368887806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e700afbe0 msgr2=0x7f2e700ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:48.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.763+0000 7f2e77c00700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e700afbe0 0x7f2e700ad350 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c001eb0 tx=0x7f2e5c00c860 comp rx=0 tx=0).stop 2024-04-02T09:13:48.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.763+0000 7f2e77c00700 1 -- 172.21.15.89:0/1368887806 shutdown_connections 2024-04-02T09:13:48.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.763+0000 7f2e77c00700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2e5006aed0 0x7f2e5006d390 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.763+0000 7f2e77c00700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e7006a790 0x7f2e701f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.763+0000 7f2e77c00700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2e700cebd0 0x7f2e700ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.763+0000 7f2e77c00700 1 --2- 172.21.15.89:0/1368887806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e700afbe0 0x7f2e700ad350 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:48.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.763+0000 7f2e77c00700 1 -- 172.21.15.89:0/1368887806 >> 172.21.15.89:0/1368887806 conn(0x7f2e700b5ab0 msgr2=0x7f2e700ae780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:48.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.763+0000 7f2e77c00700 1 -- 172.21.15.89:0/1368887806 shutdown_connections 2024-04-02T09:13:48.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:48.763+0000 7f2e77c00700 1 -- 172.21.15.89:0/1368887806 wait complete. 2024-04-02T09:13:48.768 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:13:48.780 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-04-02T09:13:47.432749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596333,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":711,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":450,"ondisk_log_size":450,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":361,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413236,"kb_used_data":25664,"kb_used_omap":12,"kb_used_meta":387443,"kb_avail":754561484,"statfs":{"total":773094113280,"available":772670959616,"internally_reserved":0,"allocated":26279936,"data_stored":13992684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":396742200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001855"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.947418+0000","last_change":"2024-04-02T09:13:41.944375+0000","last_active":"2024-04-02T09:13:42.947418+0000","last_peered":"2024-04-02T09:13:42.947418+0000","last_clean":"2024-04-02T09:13:42.947418+0000","last_became_active":"2024-04-02T09:13:41.944110+0000","last_became_peered":"2024-04-02T09:13:41.944110+0000","last_unstale":"2024-04-02T09:13:42.947418+0000","last_undegraded":"2024-04-02T09:13:42.947418+0000","last_fullsized":"2024-04-02T09:13:42.947418+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:02:20.460847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942652+0000","last_change":"2024-04-02T09:13:41.458005+0000","last_active":"2024-04-02T09:13:41.942652+0000","last_peered":"2024-04-02T09:13:41.942652+0000","last_clean":"2024-04-02T09:13:41.942652+0000","last_became_active":"2024-04-02T09:13:41.457753+0000","last_became_peered":"2024-04-02T09:13:41.457753+0000","last_unstale":"2024-04-02T09:13:41.942652+0000","last_undegraded":"2024-04-02T09:13:41.942652+0000","last_fullsized":"2024-04-02T09:13:41.942652+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:12:22.769587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942756+0000","last_change":"2024-04-02T09:13:28.396642+0000","last_active":"2024-04-02T09:13:41.942756+0000","last_peered":"2024-04-02T09:13:41.942756+0000","last_clean":"2024-04-02T09:13:41.942756+0000","last_became_active":"2024-04-02T09:13:28.396356+0000","last_became_peered":"2024-04-02T09:13:28.396356+0000","last_unstale":"2024-04-02T09:13:41.942756+0000","last_undegraded":"2024-04-02T09:13:41.942756+0000","last_fullsized":"2024-04-02T09:13:41.942756+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:34.295480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345466+0000","last_change":"2024-04-02T09:13:41.961471+0000","last_active":"2024-04-02T09:13:43.345466+0000","last_peered":"2024-04-02T09:13:43.345466+0000","last_clean":"2024-04-02T09:13:43.345466+0000","last_became_active":"2024-04-02T09:13:41.961181+0000","last_became_peered":"2024-04-02T09:13:41.961181+0000","last_unstale":"2024-04-02T09:13:43.345466+0000","last_undegraded":"2024-04-02T09:13:43.345466+0000","last_fullsized":"2024-04-02T09:13:43.345466+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:13.220603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952545+0000","last_change":"2024-04-02T09:13:41.454077+0000","last_active":"2024-04-02T09:13:41.952545+0000","last_peered":"2024-04-02T09:13:41.952545+0000","last_clean":"2024-04-02T09:13:41.952545+0000","last_became_active":"2024-04-02T09:13:41.453804+0000","last_became_peered":"2024-04-02T09:13:41.453804+0000","last_unstale":"2024-04-02T09:13:41.952545+0000","last_undegraded":"2024-04-02T09:13:41.952545+0000","last_fullsized":"2024-04-02T09:13:41.952545+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:24:28.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.961240+0000","last_change":"2024-04-02T09:13:28.381100+0000","last_active":"2024-04-02T09:13:41.961240+0000","last_peered":"2024-04-02T09:13:41.961240+0000","last_clean":"2024-04-02T09:13:41.961240+0000","last_became_active":"2024-04-02T09:13:28.380832+0000","last_became_peered":"2024-04-02T09:13:28.380832+0000","last_unstale":"2024-04-02T09:13:41.961240+0000","last_undegraded":"2024-04-02T09:13:41.961240+0000","last_fullsized":"2024-04-02T09:13:41.961240+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:21.987215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734898+0000","last_change":"2024-04-02T09:13:41.970419+0000","last_active":"2024-04-02T09:13:43.734898+0000","last_peered":"2024-04-02T09:13:43.734898+0000","last_clean":"2024-04-02T09:13:43.734898+0000","last_became_active":"2024-04-02T09:13:41.965663+0000","last_became_peered":"2024-04-02T09:13:41.965663+0000","last_unstale":"2024-04-02T09:13:43.734898+0000","last_undegraded":"2024-04-02T09:13:43.734898+0000","last_fullsized":"2024-04-02T09:13:43.734898+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:28:51.025221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942697+0000","last_change":"2024-04-02T09:13:41.457188+0000","last_active":"2024-04-02T09:13:41.942697+0000","last_peered":"2024-04-02T09:13:41.942697+0000","last_clean":"2024-04-02T09:13:41.942697+0000","last_became_active":"2024-04-02T09:13:41.456916+0000","last_became_peered":"2024-04-02T09:13:41.456916+0000","last_unstale":"2024-04-02T09:13:41.942697+0000","last_undegraded":"2024-04-02T09:13:41.942697+0000","last_fullsized":"2024-04-02T09:13:41.942697+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:31:39.320935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.941485+0000","last_change":"2024-04-02T09:13:28.393942+0000","last_active":"2024-04-02T09:13:41.941485+0000","last_peered":"2024-04-02T09:13:41.941485+0000","last_clean":"2024-04-02T09:13:41.941485+0000","last_became_active":"2024-04-02T09:13:28.393467+0000","last_became_peered":"2024-04-02T09:13:28.393467+0000","last_unstale":"2024-04-02T09:13:41.941485+0000","last_undegraded":"2024-04-02T09:13:41.941485+0000","last_fullsized":"2024-04-02T09:13:41.941485+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:24.836500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952847+0000","last_change":"2024-04-02T09:13:41.961739+0000","last_active":"2024-04-02T09:13:42.952847+0000","last_peered":"2024-04-02T09:13:42.952847+0000","last_clean":"2024-04-02T09:13:42.952847+0000","last_became_active":"2024-04-02T09:13:41.961468+0000","last_became_peered":"2024-04-02T09:13:41.961468+0000","last_unstale":"2024-04-02T09:13:42.952847+0000","last_undegraded":"2024-04-02T09:13:42.952847+0000","last_fullsized":"2024-04-02T09:13:42.952847+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:33:35.636916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940817+0000","last_change":"2024-04-02T09:13:41.452325+0000","last_active":"2024-04-02T09:13:41.940817+0000","last_peered":"2024-04-02T09:13:41.940817+0000","last_clean":"2024-04-02T09:13:41.940817+0000","last_became_active":"2024-04-02T09:13:41.449714+0000","last_became_peered":"2024-04-02T09:13:41.449714+0000","last_unstale":"2024-04-02T09:13:41.940817+0000","last_undegraded":"2024-04-02T09:13:41.940817+0000","last_fullsized":"2024-04-02T09:13:41.940817+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:13:02.182890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945254+0000","last_change":"2024-04-02T09:13:28.394098+0000","last_active":"2024-04-02T09:13:41.945254+0000","last_peered":"2024-04-02T09:13:41.945254+0000","last_clean":"2024-04-02T09:13:41.945254+0000","last_became_active":"2024-04-02T09:13:28.393837+0000","last_became_peered":"2024-04-02T09:13:28.393837+0000","last_unstale":"2024-04-02T09:13:41.945254+0000","last_undegraded":"2024-04-02T09:13:41.945254+0000","last_fullsized":"2024-04-02T09:13:41.945254+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:40:27.068896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.948026+0000","last_change":"2024-04-02T09:13:41.951117+0000","last_active":"2024-04-02T09:13:42.948026+0000","last_peered":"2024-04-02T09:13:42.948026+0000","last_clean":"2024-04-02T09:13:42.948026+0000","last_became_active":"2024-04-02T09:13:41.948760+0000","last_became_peered":"2024-04-02T09:13:41.948760+0000","last_unstale":"2024-04-02T09:13:42.948026+0000","last_undegraded":"2024-04-02T09:13:42.948026+0000","last_fullsized":"2024-04-02T09:13:42.948026+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:53:09.361699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944874+0000","last_change":"2024-04-02T09:13:41.458809+0000","last_active":"2024-04-02T09:13:41.944874+0000","last_peered":"2024-04-02T09:13:41.944874+0000","last_clean":"2024-04-02T09:13:41.944874+0000","last_became_active":"2024-04-02T09:13:41.458392+0000","last_became_peered":"2024-04-02T09:13:41.458392+0000","last_unstale":"2024-04-02T09:13:41.944874+0000","last_undegraded":"2024-04-02T09:13:41.944874+0000","last_fullsized":"2024-04-02T09:13:41.944874+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:53.327540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947921+0000","last_change":"2024-04-02T09:13:28.379239+0000","last_active":"2024-04-02T09:13:41.947921+0000","last_peered":"2024-04-02T09:13:41.947921+0000","last_clean":"2024-04-02T09:13:41.947921+0000","last_became_active":"2024-04-02T09:13:28.378970+0000","last_became_peered":"2024-04-02T09:13:28.378970+0000","last_unstale":"2024-04-02T09:13:41.947921+0000","last_undegraded":"2024-04-02T09:13:41.947921+0000","last_fullsized":"2024-04-02T09:13:41.947921+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:37.960638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342554+0000","last_change":"2024-04-02T09:13:41.959102+0000","last_active":"2024-04-02T09:13:43.342554+0000","last_peered":"2024-04-02T09:13:43.342554+0000","last_clean":"2024-04-02T09:13:43.342554+0000","last_became_active":"2024-04-02T09:13:41.956524+0000","last_became_peered":"2024-04-02T09:13:41.956524+0000","last_unstale":"2024-04-02T09:13:43.342554+0000","last_undegraded":"2024-04-02T09:13:43.342554+0000","last_fullsized":"2024-04-02T09:13:43.342554+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:47.857858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946223+0000","last_change":"2024-04-02T09:13:41.451612+0000","last_active":"2024-04-02T09:13:41.946223+0000","last_peered":"2024-04-02T09:13:41.946223+0000","last_clean":"2024-04-02T09:13:41.946223+0000","last_became_active":"2024-04-02T09:13:41.451089+0000","last_became_peered":"2024-04-02T09:13:41.451089+0000","last_unstale":"2024-04-02T09:13:41.946223+0000","last_undegraded":"2024-04-02T09:13:41.946223+0000","last_fullsized":"2024-04-02T09:13:41.946223+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:18.892470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.939756+0000","last_change":"2024-04-02T09:13:28.393747+0000","last_active":"2024-04-02T09:13:41.939756+0000","last_peered":"2024-04-02T09:13:41.939756+0000","last_clean":"2024-04-02T09:13:41.939756+0000","last_became_active":"2024-04-02T09:13:28.393515+0000","last_became_peered":"2024-04-02T09:13:28.393515+0000","last_unstale":"2024-04-02T09:13:41.939756+0000","last_undegraded":"2024-04-02T09:13:41.939756+0000","last_fullsized":"2024-04-02T09:13:41.939756+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:28:37.462844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954473+0000","last_change":"2024-04-02T09:13:41.948810+0000","last_active":"2024-04-02T09:13:42.954473+0000","last_peered":"2024-04-02T09:13:42.954473+0000","last_clean":"2024-04-02T09:13:42.954473+0000","last_became_active":"2024-04-02T09:13:41.948523+0000","last_became_peered":"2024-04-02T09:13:41.948523+0000","last_unstale":"2024-04-02T09:13:42.954473+0000","last_undegraded":"2024-04-02T09:13:42.954473+0000","last_fullsized":"2024-04-02T09:13:42.954473+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:54.336827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.6","version":"248'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342742+0000","last_change":"2024-04-02T09:13:41.460002+0000","last_active":"2024-04-02T09:13:43.342742+0000","last_peered":"2024-04-02T09:13:43.342742+0000","last_clean":"2024-04-02T09:13:43.342742+0000","last_became_active":"2024-04-02T09:13:41.459491+0000","last_became_peered":"2024-04-02T09:13:41.459491+0000","last_unstale":"2024-04-02T09:13:43.342742+0000","last_undegraded":"2024-04-02T09:13:43.342742+0000","last_fullsized":"2024-04-02T09:13:43.342742+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:53.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.938200+0000","last_change":"2024-04-02T09:13:28.399975+0000","last_active":"2024-04-02T09:13:41.938200+0000","last_peered":"2024-04-02T09:13:41.938200+0000","last_clean":"2024-04-02T09:13:41.938200+0000","last_became_active":"2024-04-02T09:13:28.399335+0000","last_became_peered":"2024-04-02T09:13:28.399335+0000","last_unstale":"2024-04-02T09:13:41.938200+0000","last_undegraded":"2024-04-02T09:13:41.938200+0000","last_fullsized":"2024-04-02T09:13:41.938200+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:16:14.265813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954211+0000","last_change":"2024-04-02T09:13:41.954505+0000","last_active":"2024-04-02T09:13:42.954211+0000","last_peered":"2024-04-02T09:13:42.954211+0000","last_clean":"2024-04-02T09:13:42.954211+0000","last_became_active":"2024-04-02T09:13:41.952682+0000","last_became_peered":"2024-04-02T09:13:41.952682+0000","last_unstale":"2024-04-02T09:13:42.954211+0000","last_undegraded":"2024-04-02T09:13:42.954211+0000","last_fullsized":"2024-04-02T09:13:42.954211+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:52.343662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.7","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.971184+0000","last_change":"2024-04-02T09:13:41.459550+0000","last_active":"2024-04-02T09:13:41.971184+0000","last_peered":"2024-04-02T09:13:41.971184+0000","last_clean":"2024-04-02T09:13:41.971184+0000","last_became_active":"2024-04-02T09:13:41.459249+0000","last_became_peered":"2024-04-02T09:13:41.459249+0000","last_unstale":"2024-04-02T09:13:41.971184+0000","last_undegraded":"2024-04-02T09:13:41.971184+0000","last_fullsized":"2024-04-02T09:13:41.971184+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:40.922210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947893+0000","last_change":"2024-04-02T09:13:28.392551+0000","last_active":"2024-04-02T09:13:41.947893+0000","last_peered":"2024-04-02T09:13:41.947893+0000","last_clean":"2024-04-02T09:13:41.947893+0000","last_became_active":"2024-04-02T09:13:28.392159+0000","last_became_peered":"2024-04-02T09:13:28.392159+0000","last_unstale":"2024-04-02T09:13:41.947893+0000","last_undegraded":"2024-04-02T09:13:41.947893+0000","last_fullsized":"2024-04-02T09:13:41.947893+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:42.551696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952740+0000","last_change":"2024-04-02T09:13:41.951315+0000","last_active":"2024-04-02T09:13:42.952740+0000","last_peered":"2024-04-02T09:13:42.952740+0000","last_clean":"2024-04-02T09:13:42.952740+0000","last_became_active":"2024-04-02T09:13:41.950989+0000","last_became_peered":"2024-04-02T09:13:41.950989+0000","last_unstale":"2024-04-02T09:13:42.952740+0000","last_undegraded":"2024-04-02T09:13:42.952740+0000","last_fullsized":"2024-04-02T09:13:42.952740+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:09.529216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944222+0000","last_change":"2024-04-02T09:13:41.451519+0000","last_active":"2024-04-02T09:13:41.944222+0000","last_peered":"2024-04-02T09:13:41.944222+0000","last_clean":"2024-04-02T09:13:41.944222+0000","last_became_active":"2024-04-02T09:13:41.450207+0000","last_became_peered":"2024-04-02T09:13:41.450207+0000","last_unstale":"2024-04-02T09:13:41.944222+0000","last_undegraded":"2024-04-02T09:13:41.944222+0000","last_fullsized":"2024-04-02T09:13:41.944222+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:08.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950200+0000","last_change":"2024-04-02T09:13:28.383891+0000","last_active":"2024-04-02T09:13:41.950200+0000","last_peered":"2024-04-02T09:13:41.950200+0000","last_clean":"2024-04-02T09:13:41.950200+0000","last_became_active":"2024-04-02T09:13:28.383628+0000","last_became_peered":"2024-04-02T09:13:28.383628+0000","last_unstale":"2024-04-02T09:13:41.950200+0000","last_undegraded":"2024-04-02T09:13:41.950200+0000","last_fullsized":"2024-04-02T09:13:41.950200+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:44.222993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734669+0000","last_change":"2024-04-02T09:13:41.969003+0000","last_active":"2024-04-02T09:13:43.734669+0000","last_peered":"2024-04-02T09:13:43.734669+0000","last_clean":"2024-04-02T09:13:43.734669+0000","last_became_active":"2024-04-02T09:13:41.968108+0000","last_became_peered":"2024-04-02T09:13:41.968108+0000","last_unstale":"2024-04-02T09:13:43.734669+0000","last_undegraded":"2024-04-02T09:13:43.734669+0000","last_fullsized":"2024-04-02T09:13:43.734669+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:11.075454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945467+0000","last_change":"2024-04-02T09:13:41.450917+0000","last_active":"2024-04-02T09:13:41.945467+0000","last_peered":"2024-04-02T09:13:41.945467+0000","last_clean":"2024-04-02T09:13:41.945467+0000","last_became_active":"2024-04-02T09:13:41.450645+0000","last_became_peered":"2024-04-02T09:13:41.450645+0000","last_unstale":"2024-04-02T09:13:41.945467+0000","last_undegraded":"2024-04-02T09:13:41.945467+0000","last_fullsized":"2024-04-02T09:13:41.945467+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:10.066167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940531+0000","last_change":"2024-04-02T09:13:28.404390+0000","last_active":"2024-04-02T09:13:41.940531+0000","last_peered":"2024-04-02T09:13:41.940531+0000","last_clean":"2024-04-02T09:13:41.940531+0000","last_became_active":"2024-04-02T09:13:28.404038+0000","last_became_peered":"2024-04-02T09:13:28.404038+0000","last_unstale":"2024-04-02T09:13:41.940531+0000","last_undegraded":"2024-04-02T09:13:41.940531+0000","last_fullsized":"2024-04-02T09:13:41.940531+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:47.359176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.735085+0000","last_change":"2024-04-02T09:13:41.968705+0000","last_active":"2024-04-02T09:13:43.735085+0000","last_peered":"2024-04-02T09:13:43.735085+0000","last_clean":"2024-04-02T09:13:43.735085+0000","last_became_active":"2024-04-02T09:13:41.968079+0000","last_became_peered":"2024-04-02T09:13:41.968079+0000","last_unstale":"2024-04-02T09:13:43.735085+0000","last_undegraded":"2024-04-02T09:13:43.735085+0000","last_fullsized":"2024-04-02T09:13:43.735085+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:27:04.458016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944274+0000","last_change":"2024-04-02T09:13:41.451223+0000","last_active":"2024-04-02T09:13:41.944274+0000","last_peered":"2024-04-02T09:13:41.944274+0000","last_clean":"2024-04-02T09:13:41.944274+0000","last_became_active":"2024-04-02T09:13:41.449328+0000","last_became_peered":"2024-04-02T09:13:41.449328+0000","last_unstale":"2024-04-02T09:13:41.944274+0000","last_undegraded":"2024-04-02T09:13:41.944274+0000","last_fullsized":"2024-04-02T09:13:41.944274+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:50:01.664013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946964+0000","last_change":"2024-04-02T09:13:28.403517+0000","last_active":"2024-04-02T09:13:41.946964+0000","last_peered":"2024-04-02T09:13:41.946964+0000","last_clean":"2024-04-02T09:13:41.946964+0000","last_became_active":"2024-04-02T09:13:28.401353+0000","last_became_peered":"2024-04-02T09:13:28.401353+0000","last_unstale":"2024-04-02T09:13:41.946964+0000","last_undegraded":"2024-04-02T09:13:41.946964+0000","last_fullsized":"2024-04-02T09:13:41.946964+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:08.980542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345809+0000","last_change":"2024-04-02T09:13:41.957583+0000","last_active":"2024-04-02T09:13:43.345809+0000","last_peered":"2024-04-02T09:13:43.345809+0000","last_clean":"2024-04-02T09:13:43.345809+0000","last_became_active":"2024-04-02T09:13:41.953951+0000","last_became_peered":"2024-04-02T09:13:41.953951+0000","last_unstale":"2024-04-02T09:13:43.345809+0000","last_undegraded":"2024-04-02T09:13:43.345809+0000","last_fullsized":"2024-04-02T09:13:43.345809+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:14:22.720024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.b","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.969813+0000","last_change":"2024-04-02T09:13:41.446786+0000","last_active":"2024-04-02T09:13:41.969813+0000","last_peered":"2024-04-02T09:13:41.969813+0000","last_clean":"2024-04-02T09:13:41.969813+0000","last_became_active":"2024-04-02T09:13:41.446482+0000","last_became_peered":"2024-04-02T09:13:41.446482+0000","last_unstale":"2024-04-02T09:13:41.969813+0000","last_undegraded":"2024-04-02T09:13:41.969813+0000","last_fullsized":"2024-04-02T09:13:41.969813+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:08.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.938581+0000","last_change":"2024-04-02T09:13:28.399938+0000","last_active":"2024-04-02T09:13:41.938581+0000","last_peered":"2024-04-02T09:13:41.938581+0000","last_clean":"2024-04-02T09:13:41.938581+0000","last_became_active":"2024-04-02T09:13:28.399350+0000","last_became_peered":"2024-04-02T09:13:28.399350+0000","last_unstale":"2024-04-02T09:13:41.938581+0000","last_undegraded":"2024-04-02T09:13:41.938581+0000","last_fullsized":"2024-04-02T09:13:41.938581+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:05.933699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734569+0000","last_change":"2024-04-02T09:13:41.959689+0000","last_active":"2024-04-02T09:13:43.734569+0000","last_peered":"2024-04-02T09:13:43.734569+0000","last_clean":"2024-04-02T09:13:43.734569+0000","last_became_active":"2024-04-02T09:13:41.959238+0000","last_became_peered":"2024-04-02T09:13:41.959238+0000","last_unstale":"2024-04-02T09:13:43.734569+0000","last_undegraded":"2024-04-02T09:13:43.734569+0000","last_fullsized":"2024-04-02T09:13:43.734569+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:45:09.502243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953945+0000","last_change":"2024-04-02T09:13:41.450544+0000","last_active":"2024-04-02T09:13:41.953945+0000","last_peered":"2024-04-02T09:13:41.953945+0000","last_clean":"2024-04-02T09:13:41.953945+0000","last_became_active":"2024-04-02T09:13:41.450233+0000","last_became_peered":"2024-04-02T09:13:41.450233+0000","last_unstale":"2024-04-02T09:13:41.953945+0000","last_undegraded":"2024-04-02T09:13:41.953945+0000","last_fullsized":"2024-04-02T09:13:41.953945+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:53.842095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950527+0000","last_change":"2024-04-02T09:13:28.386688+0000","last_active":"2024-04-02T09:13:41.950527+0000","last_peered":"2024-04-02T09:13:41.950527+0000","last_clean":"2024-04-02T09:13:41.950527+0000","last_became_active":"2024-04-02T09:13:28.386363+0000","last_became_peered":"2024-04-02T09:13:28.386363+0000","last_unstale":"2024-04-02T09:13:41.950527+0000","last_undegraded":"2024-04-02T09:13:41.950527+0000","last_fullsized":"2024-04-02T09:13:41.950527+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:12:41.480404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954148+0000","last_change":"2024-04-02T09:13:41.952898+0000","last_active":"2024-04-02T09:13:42.954148+0000","last_peered":"2024-04-02T09:13:42.954148+0000","last_clean":"2024-04-02T09:13:42.954148+0000","last_became_active":"2024-04-02T09:13:41.952619+0000","last_became_peered":"2024-04-02T09:13:41.952619+0000","last_unstale":"2024-04-02T09:13:42.954148+0000","last_undegraded":"2024-04-02T09:13:42.954148+0000","last_fullsized":"2024-04-02T09:13:42.954148+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:14:22.720024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.d","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.966777+0000","last_change":"2024-04-02T09:13:41.455641+0000","last_active":"2024-04-02T09:13:41.966777+0000","last_peered":"2024-04-02T09:13:41.966777+0000","last_clean":"2024-04-02T09:13:41.966777+0000","last_became_active":"2024-04-02T09:13:41.455348+0000","last_became_peered":"2024-04-02T09:13:41.455348+0000","last_unstale":"2024-04-02T09:13:41.966777+0000","last_undegraded":"2024-04-02T09:13:41.966777+0000","last_fullsized":"2024-04-02T09:13:41.966777+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:22:20.170595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942920+0000","last_change":"2024-04-02T09:13:28.401903+0000","last_active":"2024-04-02T09:13:41.942920+0000","last_peered":"2024-04-02T09:13:41.942920+0000","last_clean":"2024-04-02T09:13:41.942920+0000","last_became_active":"2024-04-02T09:13:28.399098+0000","last_became_peered":"2024-04-02T09:13:28.399098+0000","last_unstale":"2024-04-02T09:13:41.942920+0000","last_undegraded":"2024-04-02T09:13:41.942920+0000","last_fullsized":"2024-04-02T09:13:41.942920+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:33:56.420978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734685+0000","last_change":"2024-04-02T09:13:41.957617+0000","last_active":"2024-04-02T09:13:43.734685+0000","last_peered":"2024-04-02T09:13:43.734685+0000","last_clean":"2024-04-02T09:13:43.734685+0000","last_became_active":"2024-04-02T09:13:41.957374+0000","last_became_peered":"2024-04-02T09:13:41.957374+0000","last_unstale":"2024-04-02T09:13:43.734685+0000","last_undegraded":"2024-04-02T09:13:43.734685+0000","last_fullsized":"2024-04-02T09:13:43.734685+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:02:20.460847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.e","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.974578+0000","last_change":"2024-04-02T09:13:41.456993+0000","last_active":"2024-04-02T09:13:41.974578+0000","last_peered":"2024-04-02T09:13:41.974578+0000","last_clean":"2024-04-02T09:13:41.974578+0000","last_became_active":"2024-04-02T09:13:41.456687+0000","last_became_peered":"2024-04-02T09:13:41.456687+0000","last_unstale":"2024-04-02T09:13:41.974578+0000","last_undegraded":"2024-04-02T09:13:41.974578+0000","last_fullsized":"2024-04-02T09:13:41.974578+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:53.688559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944668+0000","last_change":"2024-04-02T09:13:28.383146+0000","last_active":"2024-04-02T09:13:41.944668+0000","last_peered":"2024-04-02T09:13:41.944668+0000","last_clean":"2024-04-02T09:13:41.944668+0000","last_became_active":"2024-04-02T09:13:28.382879+0000","last_became_peered":"2024-04-02T09:13:28.382879+0000","last_unstale":"2024-04-02T09:13:41.944668+0000","last_undegraded":"2024-04-02T09:13:41.944668+0000","last_fullsized":"2024-04-02T09:13:41.944668+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:57:59.273299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.735294+0000","last_change":"2024-04-02T09:13:41.963375+0000","last_active":"2024-04-02T09:13:43.735294+0000","last_peered":"2024-04-02T09:13:43.735294+0000","last_clean":"2024-04-02T09:13:43.735294+0000","last_became_active":"2024-04-02T09:13:41.961332+0000","last_became_peered":"2024-04-02T09:13:41.961332+0000","last_unstale":"2024-04-02T09:13:43.735294+0000","last_undegraded":"2024-04-02T09:13:43.735294+0000","last_fullsized":"2024-04-02T09:13:43.735294+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:38.398878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.f","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.967015+0000","last_change":"2024-04-02T09:13:41.450114+0000","last_active":"2024-04-02T09:13:41.967015+0000","last_peered":"2024-04-02T09:13:41.967015+0000","last_clean":"2024-04-02T09:13:41.967015+0000","last_became_active":"2024-04-02T09:13:41.449847+0000","last_became_peered":"2024-04-02T09:13:41.449847+0000","last_unstale":"2024-04-02T09:13:41.967015+0000","last_undegraded":"2024-04-02T09:13:41.967015+0000","last_fullsized":"2024-04-02T09:13:41.967015+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:56.693535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953001+0000","last_change":"2024-04-02T09:13:28.381499+0000","last_active":"2024-04-02T09:13:41.953001+0000","last_peered":"2024-04-02T09:13:41.953001+0000","last_clean":"2024-04-02T09:13:41.953001+0000","last_became_active":"2024-04-02T09:13:28.377983+0000","last_became_peered":"2024-04-02T09:13:28.377983+0000","last_unstale":"2024-04-02T09:13:41.953001+0000","last_undegraded":"2024-04-02T09:13:41.953001+0000","last_fullsized":"2024-04-02T09:13:41.953001+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:45.021173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734898+0000","last_change":"2024-04-02T09:13:41.958180+0000","last_active":"2024-04-02T09:13:43.734898+0000","last_peered":"2024-04-02T09:13:43.734898+0000","last_clean":"2024-04-02T09:13:43.734898+0000","last_became_active":"2024-04-02T09:13:41.957880+0000","last_became_peered":"2024-04-02T09:13:41.957880+0000","last_unstale":"2024-04-02T09:13:43.734898+0000","last_undegraded":"2024-04-02T09:13:43.734898+0000","last_fullsized":"2024-04-02T09:13:43.734898+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:28.634723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.10","version":"247'8","reported_seq":19,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.975332+0000","last_change":"2024-04-02T09:13:41.453631+0000","last_active":"2024-04-02T09:13:41.975332+0000","last_peered":"2024-04-02T09:13:41.975332+0000","last_clean":"2024-04-02T09:13:41.975332+0000","last_became_active":"2024-04-02T09:13:41.453353+0000","last_became_peered":"2024-04-02T09:13:41.453353+0000","last_unstale":"2024-04-02T09:13:41.975332+0000","last_undegraded":"2024-04-02T09:13:41.975332+0000","last_fullsized":"2024-04-02T09:13:41.975332+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:14.301601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940637+0000","last_change":"2024-04-02T09:13:28.382899+0000","last_active":"2024-04-02T09:13:41.940637+0000","last_peered":"2024-04-02T09:13:41.940637+0000","last_clean":"2024-04-02T09:13:41.940637+0000","last_became_active":"2024-04-02T09:13:28.382023+0000","last_became_peered":"2024-04-02T09:13:28.382023+0000","last_unstale":"2024-04-02T09:13:41.940637+0000","last_undegraded":"2024-04-02T09:13:41.940637+0000","last_fullsized":"2024-04-02T09:13:41.940637+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:12.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345329+0000","last_change":"2024-04-02T09:13:41.972132+0000","last_active":"2024-04-02T09:13:43.345329+0000","last_peered":"2024-04-02T09:13:43.345329+0000","last_clean":"2024-04-02T09:13:43.345329+0000","last_became_active":"2024-04-02T09:13:41.965773+0000","last_became_peered":"2024-04-02T09:13:41.965773+0000","last_unstale":"2024-04-02T09:13:43.345329+0000","last_undegraded":"2024-04-02T09:13:43.345329+0000","last_fullsized":"2024-04-02T09:13:43.345329+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:38.686528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.949402+0000","last_change":"2024-04-02T09:13:41.445504+0000","last_active":"2024-04-02T09:13:41.949402+0000","last_peered":"2024-04-02T09:13:41.949402+0000","last_clean":"2024-04-02T09:13:41.949402+0000","last_became_active":"2024-04-02T09:13:41.445211+0000","last_became_peered":"2024-04-02T09:13:41.445211+0000","last_unstale":"2024-04-02T09:13:41.949402+0000","last_undegraded":"2024-04-02T09:13:41.949402+0000","last_fullsized":"2024-04-02T09:13:41.949402+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:40.922210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.955279+0000","last_change":"2024-04-02T09:13:28.390594+0000","last_active":"2024-04-02T09:13:41.955279+0000","last_peered":"2024-04-02T09:13:41.955279+0000","last_clean":"2024-04-02T09:13:41.955279+0000","last_became_active":"2024-04-02T09:13:28.388061+0000","last_became_peered":"2024-04-02T09:13:28.388061+0000","last_unstale":"2024-04-02T09:13:41.955279+0000","last_undegraded":"2024-04-02T09:13:41.955279+0000","last_fullsized":"2024-04-02T09:13:41.955279+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:14:04.841237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345714+0000","last_change":"2024-04-02T09:13:41.966357+0000","last_active":"2024-04-02T09:13:43.345714+0000","last_peered":"2024-04-02T09:13:43.345714+0000","last_clean":"2024-04-02T09:13:43.345714+0000","last_became_active":"2024-04-02T09:13:41.961810+0000","last_became_peered":"2024-04-02T09:13:41.961810+0000","last_unstale":"2024-04-02T09:13:43.345714+0000","last_undegraded":"2024-04-02T09:13:43.345714+0000","last_fullsized":"2024-04-02T09:13:43.345714+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:37.458648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953956+0000","last_change":"2024-04-02T09:13:41.457989+0000","last_active":"2024-04-02T09:13:41.953956+0000","last_peered":"2024-04-02T09:13:41.953956+0000","last_clean":"2024-04-02T09:13:41.953956+0000","last_became_active":"2024-04-02T09:13:41.456512+0000","last_became_peered":"2024-04-02T09:13:41.456512+0000","last_unstale":"2024-04-02T09:13:41.953956+0000","last_undegraded":"2024-04-02T09:13:41.953956+0000","last_fullsized":"2024-04-02T09:13:41.953956+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:56.693535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950481+0000","last_change":"2024-04-02T09:13:28.382883+0000","last_active":"2024-04-02T09:13:41.950481+0000","last_peered":"2024-04-02T09:13:41.950481+0000","last_clean":"2024-04-02T09:13:41.950481+0000","last_became_active":"2024-04-02T09:13:28.382187+0000","last_became_peered":"2024-04-02T09:13:28.382187+0000","last_unstale":"2024-04-02T09:13:41.950481+0000","last_undegraded":"2024-04-02T09:13:41.950481+0000","last_fullsized":"2024-04-02T09:13:41.950481+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:21:26.362248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734816+0000","last_change":"2024-04-02T09:13:41.961588+0000","last_active":"2024-04-02T09:13:43.734816+0000","last_peered":"2024-04-02T09:13:43.734816+0000","last_clean":"2024-04-02T09:13:43.734816+0000","last_became_active":"2024-04-02T09:13:41.958669+0000","last_became_peered":"2024-04-02T09:13:41.958669+0000","last_unstale":"2024-04-02T09:13:43.734816+0000","last_undegraded":"2024-04-02T09:13:43.734816+0000","last_fullsized":"2024-04-02T09:13:43.734816+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:46.027233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.13","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.971712+0000","last_change":"2024-04-02T09:13:41.450452+0000","last_active":"2024-04-02T09:13:41.971712+0000","last_peered":"2024-04-02T09:13:41.971712+0000","last_clean":"2024-04-02T09:13:41.971712+0000","last_became_active":"2024-04-02T09:13:41.449005+0000","last_became_peered":"2024-04-02T09:13:41.449005+0000","last_unstale":"2024-04-02T09:13:41.971712+0000","last_undegraded":"2024-04-02T09:13:41.971712+0000","last_fullsized":"2024-04-02T09:13:41.971712+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:19.429201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.961389+0000","last_change":"2024-04-02T09:13:28.384295+0000","last_active":"2024-04-02T09:13:41.961389+0000","last_peered":"2024-04-02T09:13:41.961389+0000","last_clean":"2024-04-02T09:13:41.961389+0000","last_became_active":"2024-04-02T09:13:28.384010+0000","last_became_peered":"2024-04-02T09:13:28.384010+0000","last_unstale":"2024-04-02T09:13:41.961389+0000","last_undegraded":"2024-04-02T09:13:41.961389+0000","last_fullsized":"2024-04-02T09:13:41.961389+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:47:33.241308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.735273+0000","last_change":"2024-04-02T09:13:41.965796+0000","last_active":"2024-04-02T09:13:43.735273+0000","last_peered":"2024-04-02T09:13:43.735273+0000","last_clean":"2024-04-02T09:13:43.735273+0000","last_became_active":"2024-04-02T09:13:41.965262+0000","last_became_peered":"2024-04-02T09:13:41.965262+0000","last_unstale":"2024-04-02T09:13:43.735273+0000","last_undegraded":"2024-04-02T09:13:43.735273+0000","last_fullsized":"2024-04-02T09:13:43.735273+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:47.857858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.14","version":"247'6","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.971032+0000","last_change":"2024-04-02T09:13:41.456890+0000","last_active":"2024-04-02T09:13:41.971032+0000","last_peered":"2024-04-02T09:13:41.971032+0000","last_clean":"2024-04-02T09:13:41.971032+0000","last_became_active":"2024-04-02T09:13:41.453059+0000","last_became_peered":"2024-04-02T09:13:41.453059+0000","last_unstale":"2024-04-02T09:13:41.971032+0000","last_undegraded":"2024-04-02T09:13:41.971032+0000","last_fullsized":"2024-04-02T09:13:41.971032+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:08.249339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948077+0000","last_change":"2024-04-02T09:13:28.380209+0000","last_active":"2024-04-02T09:13:41.948077+0000","last_peered":"2024-04-02T09:13:41.948077+0000","last_clean":"2024-04-02T09:13:41.948077+0000","last_became_active":"2024-04-02T09:13:28.379739+0000","last_became_peered":"2024-04-02T09:13:28.379739+0000","last_unstale":"2024-04-02T09:13:41.948077+0000","last_undegraded":"2024-04-02T09:13:41.948077+0000","last_fullsized":"2024-04-02T09:13:41.948077+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:46:14.163290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.346038+0000","last_change":"2024-04-02T09:13:41.956661+0000","last_active":"2024-04-02T09:13:43.346038+0000","last_peered":"2024-04-02T09:13:43.346038+0000","last_clean":"2024-04-02T09:13:43.346038+0000","last_became_active":"2024-04-02T09:13:41.953910+0000","last_became_peered":"2024-04-02T09:13:41.953910+0000","last_unstale":"2024-04-02T09:13:43.346038+0000","last_undegraded":"2024-04-02T09:13:43.346038+0000","last_fullsized":"2024-04-02T09:13:43.346038+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:20.438488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.15","version":"247'3","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.966676+0000","last_change":"2024-04-02T09:13:41.458001+0000","last_active":"2024-04-02T09:13:41.966676+0000","last_peered":"2024-04-02T09:13:41.966676+0000","last_clean":"2024-04-02T09:13:41.966676+0000","last_became_active":"2024-04-02T09:13:41.457318+0000","last_became_peered":"2024-04-02T09:13:41.457318+0000","last_unstale":"2024-04-02T09:13:41.966676+0000","last_undegraded":"2024-04-02T09:13:41.966676+0000","last_fullsized":"2024-04-02T09:13:41.966676+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:47:51.824824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.949330+0000","last_change":"2024-04-02T09:13:28.392993+0000","last_active":"2024-04-02T09:13:41.949330+0000","last_peered":"2024-04-02T09:13:41.949330+0000","last_clean":"2024-04-02T09:13:41.949330+0000","last_became_active":"2024-04-02T09:13:28.392712+0000","last_became_peered":"2024-04-02T09:13:28.392712+0000","last_unstale":"2024-04-02T09:13:41.949330+0000","last_undegraded":"2024-04-02T09:13:41.949330+0000","last_fullsized":"2024-04-02T09:13:41.949330+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:38.436370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.955195+0000","last_change":"2024-04-02T09:13:41.960213+0000","last_active":"2024-04-02T09:13:42.955195+0000","last_peered":"2024-04-02T09:13:42.955195+0000","last_clean":"2024-04-02T09:13:42.955195+0000","last_became_active":"2024-04-02T09:13:41.959935+0000","last_became_peered":"2024-04-02T09:13:41.959935+0000","last_unstale":"2024-04-02T09:13:42.955195+0000","last_undegraded":"2024-04-02T09:13:42.955195+0000","last_fullsized":"2024-04-02T09:13:42.955195+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:41.505910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.955082+0000","last_change":"2024-04-02T09:13:41.451596+0000","last_active":"2024-04-02T09:13:41.955082+0000","last_peered":"2024-04-02T09:13:41.955082+0000","last_clean":"2024-04-02T09:13:41.955082+0000","last_became_active":"2024-04-02T09:13:41.451329+0000","last_became_peered":"2024-04-02T09:13:41.451329+0000","last_unstale":"2024-04-02T09:13:41.955082+0000","last_undegraded":"2024-04-02T09:13:41.955082+0000","last_fullsized":"2024-04-02T09:13:41.955082+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:48:38.200651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948538+0000","last_change":"2024-04-02T09:13:28.378030+0000","last_active":"2024-04-02T09:13:41.948538+0000","last_peered":"2024-04-02T09:13:41.948538+0000","last_clean":"2024-04-02T09:13:41.948538+0000","last_became_active":"2024-04-02T09:13:28.377700+0000","last_became_peered":"2024-04-02T09:13:28.377700+0000","last_unstale":"2024-04-02T09:13:41.948538+0000","last_undegraded":"2024-04-02T09:13:41.948538+0000","last_fullsized":"2024-04-02T09:13:41.948538+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:40.236322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734926+0000","last_change":"2024-04-02T09:13:41.956307+0000","last_active":"2024-04-02T09:13:43.734926+0000","last_peered":"2024-04-02T09:13:43.734926+0000","last_clean":"2024-04-02T09:13:43.734926+0000","last_became_active":"2024-04-02T09:13:41.955455+0000","last_became_peered":"2024-04-02T09:13:41.955455+0000","last_unstale":"2024-04-02T09:13:43.734926+0000","last_undegraded":"2024-04-02T09:13:43.734926+0000","last_fullsized":"2024-04-02T09:13:43.734926+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:11.075454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942471+0000","last_change":"2024-04-02T09:13:41.455756+0000","last_active":"2024-04-02T09:13:41.942471+0000","last_peered":"2024-04-02T09:13:41.942471+0000","last_clean":"2024-04-02T09:13:41.942471+0000","last_became_active":"2024-04-02T09:13:41.455267+0000","last_became_peered":"2024-04-02T09:13:41.455267+0000","last_unstale":"2024-04-02T09:13:41.942471+0000","last_undegraded":"2024-04-02T09:13:41.942471+0000","last_fullsized":"2024-04-02T09:13:41.942471+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:27.266184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943109+0000","last_change":"2024-04-02T09:13:28.395364+0000","last_active":"2024-04-02T09:13:41.943109+0000","last_peered":"2024-04-02T09:13:41.943109+0000","last_clean":"2024-04-02T09:13:41.943109+0000","last_became_active":"2024-04-02T09:13:28.395100+0000","last_became_peered":"2024-04-02T09:13:28.395100+0000","last_unstale":"2024-04-02T09:13:41.943109+0000","last_undegraded":"2024-04-02T09:13:41.943109+0000","last_fullsized":"2024-04-02T09:13:41.943109+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:12.863676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.733818+0000","last_change":"2024-04-02T09:13:41.956248+0000","last_active":"2024-04-02T09:13:43.733818+0000","last_peered":"2024-04-02T09:13:43.733818+0000","last_clean":"2024-04-02T09:13:43.733818+0000","last_became_active":"2024-04-02T09:13:41.955983+0000","last_became_peered":"2024-04-02T09:13:41.955983+0000","last_unstale":"2024-04-02T09:13:43.733818+0000","last_undegraded":"2024-04-02T09:13:43.733818+0000","last_fullsized":"2024-04-02T09:13:43.733818+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:50:02.673300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946966+0000","last_change":"2024-04-02T09:13:41.442991+0000","last_active":"2024-04-02T09:13:41.946966+0000","last_peered":"2024-04-02T09:13:41.946966+0000","last_clean":"2024-04-02T09:13:41.946966+0000","last_became_active":"2024-04-02T09:13:41.442743+0000","last_became_peered":"2024-04-02T09:13:41.442743+0000","last_unstale":"2024-04-02T09:13:41.946966+0000","last_undegraded":"2024-04-02T09:13:41.946966+0000","last_fullsized":"2024-04-02T09:13:41.946966+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:47:51.824824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947036+0000","last_change":"2024-04-02T09:13:28.399604+0000","last_active":"2024-04-02T09:13:41.947036+0000","last_peered":"2024-04-02T09:13:41.947036+0000","last_clean":"2024-04-02T09:13:41.947036+0000","last_became_active":"2024-04-02T09:13:28.399333+0000","last_became_peered":"2024-04-02T09:13:28.399333+0000","last_unstale":"2024-04-02T09:13:41.947036+0000","last_undegraded":"2024-04-02T09:13:41.947036+0000","last_fullsized":"2024-04-02T09:13:41.947036+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:18.748067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345299+0000","last_change":"2024-04-02T09:13:41.955992+0000","last_active":"2024-04-02T09:13:43.345299+0000","last_peered":"2024-04-02T09:13:43.345299+0000","last_clean":"2024-04-02T09:13:43.345299+0000","last_became_active":"2024-04-02T09:13:41.955696+0000","last_became_peered":"2024-04-02T09:13:41.955696+0000","last_unstale":"2024-04-02T09:13:43.345299+0000","last_undegraded":"2024-04-02T09:13:43.345299+0000","last_fullsized":"2024-04-02T09:13:43.345299+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:50:19.183542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944518+0000","last_change":"2024-04-02T09:13:41.454745+0000","last_active":"2024-04-02T09:13:41.944518+0000","last_peered":"2024-04-02T09:13:41.944518+0000","last_clean":"2024-04-02T09:13:41.944518+0000","last_became_active":"2024-04-02T09:13:41.454416+0000","last_became_peered":"2024-04-02T09:13:41.454416+0000","last_unstale":"2024-04-02T09:13:41.944518+0000","last_undegraded":"2024-04-02T09:13:41.944518+0000","last_fullsized":"2024-04-02T09:13:41.944518+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:19.429201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953068+0000","last_change":"2024-04-02T09:13:28.378770+0000","last_active":"2024-04-02T09:13:41.953068+0000","last_peered":"2024-04-02T09:13:41.953068+0000","last_clean":"2024-04-02T09:13:41.953068+0000","last_became_active":"2024-04-02T09:13:28.378521+0000","last_became_peered":"2024-04-02T09:13:28.378521+0000","last_unstale":"2024-04-02T09:13:41.953068+0000","last_undegraded":"2024-04-02T09:13:41.953068+0000","last_fullsized":"2024-04-02T09:13:41.953068+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:24:11.210144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952812+0000","last_change":"2024-04-02T09:13:41.962211+0000","last_active":"2024-04-02T09:13:42.952812+0000","last_peered":"2024-04-02T09:13:42.952812+0000","last_clean":"2024-04-02T09:13:42.952812+0000","last_became_active":"2024-04-02T09:13:41.961032+0000","last_became_peered":"2024-04-02T09:13:41.961032+0000","last_unstale":"2024-04-02T09:13:42.952812+0000","last_undegraded":"2024-04-02T09:13:42.952812+0000","last_fullsized":"2024-04-02T09:13:42.952812+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:43:41.499934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1a","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.971085+0000","last_change":"2024-04-02T09:13:41.455291+0000","last_active":"2024-04-02T09:13:41.971085+0000","last_peered":"2024-04-02T09:13:41.971085+0000","last_clean":"2024-04-02T09:13:41.971085+0000","last_became_active":"2024-04-02T09:13:41.453960+0000","last_became_peered":"2024-04-02T09:13:41.453960+0000","last_unstale":"2024-04-02T09:13:41.971085+0000","last_undegraded":"2024-04-02T09:13:41.971085+0000","last_fullsized":"2024-04-02T09:13:41.971085+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:45:49.868260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.962039+0000","last_change":"2024-04-02T09:13:28.389084+0000","last_active":"2024-04-02T09:13:41.962039+0000","last_peered":"2024-04-02T09:13:41.962039+0000","last_clean":"2024-04-02T09:13:41.962039+0000","last_became_active":"2024-04-02T09:13:28.388624+0000","last_became_peered":"2024-04-02T09:13:28.388624+0000","last_unstale":"2024-04-02T09:13:41.962039+0000","last_undegraded":"2024-04-02T09:13:41.962039+0000","last_fullsized":"2024-04-02T09:13:41.962039+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:23.766461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952740+0000","last_change":"2024-04-02T09:13:41.954960+0000","last_active":"2024-04-02T09:13:42.952740+0000","last_peered":"2024-04-02T09:13:42.952740+0000","last_clean":"2024-04-02T09:13:42.952740+0000","last_became_active":"2024-04-02T09:13:41.952119+0000","last_became_peered":"2024-04-02T09:13:41.952119+0000","last_unstale":"2024-04-02T09:13:42.952740+0000","last_undegraded":"2024-04-02T09:13:42.952740+0000","last_fullsized":"2024-04-02T09:13:42.952740+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:20.438488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1b","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.964365+0000","last_change":"2024-04-02T09:13:41.455663+0000","last_active":"2024-04-02T09:13:41.964365+0000","last_peered":"2024-04-02T09:13:41.964365+0000","last_clean":"2024-04-02T09:13:41.964365+0000","last_became_active":"2024-04-02T09:13:41.455401+0000","last_became_peered":"2024-04-02T09:13:41.455401+0000","last_unstale":"2024-04-02T09:13:41.964365+0000","last_undegraded":"2024-04-02T09:13:41.964365+0000","last_fullsized":"2024-04-02T09:13:41.964365+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:45.017947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944969+0000","last_change":"2024-04-02T09:13:28.395286+0000","last_active":"2024-04-02T09:13:41.944969+0000","last_peered":"2024-04-02T09:13:41.944969+0000","last_clean":"2024-04-02T09:13:41.944969+0000","last_became_active":"2024-04-02T09:13:28.394377+0000","last_became_peered":"2024-04-02T09:13:28.394377+0000","last_unstale":"2024-04-02T09:13:41.944969+0000","last_undegraded":"2024-04-02T09:13:41.944969+0000","last_fullsized":"2024-04-02T09:13:41.944969+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:37:39.028435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734569+0000","last_change":"2024-04-02T09:13:41.954547+0000","last_active":"2024-04-02T09:13:43.734569+0000","last_peered":"2024-04-02T09:13:43.734569+0000","last_clean":"2024-04-02T09:13:43.734569+0000","last_became_active":"2024-04-02T09:13:41.954306+0000","last_became_peered":"2024-04-02T09:13:41.954306+0000","last_unstale":"2024-04-02T09:13:43.734569+0000","last_undegraded":"2024-04-02T09:13:43.734569+0000","last_fullsized":"2024-04-02T09:13:43.734569+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:53:09.361699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1c","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.974035+0000","last_change":"2024-04-02T09:13:41.456420+0000","last_active":"2024-04-02T09:13:41.974035+0000","last_peered":"2024-04-02T09:13:41.974035+0000","last_clean":"2024-04-02T09:13:41.974035+0000","last_became_active":"2024-04-02T09:13:41.456055+0000","last_became_peered":"2024-04-02T09:13:41.456055+0000","last_unstale":"2024-04-02T09:13:41.974035+0000","last_undegraded":"2024-04-02T09:13:41.974035+0000","last_fullsized":"2024-04-02T09:13:41.974035+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:52.540226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943657+0000","last_change":"2024-04-02T09:13:28.391267+0000","last_active":"2024-04-02T09:13:41.943657+0000","last_peered":"2024-04-02T09:13:41.943657+0000","last_clean":"2024-04-02T09:13:41.943657+0000","last_became_active":"2024-04-02T09:13:28.390985+0000","last_became_peered":"2024-04-02T09:13:28.390985+0000","last_unstale":"2024-04-02T09:13:41.943657+0000","last_undegraded":"2024-04-02T09:13:41.943657+0000","last_fullsized":"2024-04-02T09:13:41.943657+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:30.210394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342493+0000","last_change":"2024-04-02T09:13:41.963616+0000","last_active":"2024-04-02T09:13:43.342493+0000","last_peered":"2024-04-02T09:13:43.342493+0000","last_clean":"2024-04-02T09:13:43.342493+0000","last_became_active":"2024-04-02T09:13:41.963371+0000","last_became_peered":"2024-04-02T09:13:41.963371+0000","last_unstale":"2024-04-02T09:13:43.342493+0000","last_undegraded":"2024-04-02T09:13:43.342493+0000","last_fullsized":"2024-04-02T09:13:43.342493+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:30:47.096602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1d","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.972845+0000","last_change":"2024-04-02T09:13:41.457926+0000","last_active":"2024-04-02T09:13:41.972845+0000","last_peered":"2024-04-02T09:13:41.972845+0000","last_clean":"2024-04-02T09:13:41.972845+0000","last_became_active":"2024-04-02T09:13:41.457081+0000","last_became_peered":"2024-04-02T09:13:41.457081+0000","last_unstale":"2024-04-02T09:13:41.972845+0000","last_undegraded":"2024-04-02T09:13:41.972845+0000","last_fullsized":"2024-04-02T09:13:41.972845+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:23:34.898152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948952+0000","last_change":"2024-04-02T09:13:28.380650+0000","last_active":"2024-04-02T09:13:41.948952+0000","last_peered":"2024-04-02T09:13:41.948952+0000","last_clean":"2024-04-02T09:13:41.948952+0000","last_became_active":"2024-04-02T09:13:28.380376+0000","last_became_peered":"2024-04-02T09:13:28.380376+0000","last_unstale":"2024-04-02T09:13:41.948952+0000","last_undegraded":"2024-04-02T09:13:41.948952+0000","last_fullsized":"2024-04-02T09:13:41.948952+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:46:17.713244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.346444+0000","last_change":"2024-04-02T09:13:41.961661+0000","last_active":"2024-04-02T09:13:43.346444+0000","last_peered":"2024-04-02T09:13:43.346444+0000","last_clean":"2024-04-02T09:13:43.346444+0000","last_became_active":"2024-04-02T09:13:41.959158+0000","last_became_peered":"2024-04-02T09:13:41.959158+0000","last_unstale":"2024-04-02T09:13:43.346444+0000","last_undegraded":"2024-04-02T09:13:43.346444+0000","last_fullsized":"2024-04-02T09:13:43.346444+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:57:28.703863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.1e","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.974495+0000","last_change":"2024-04-02T09:13:41.448497+0000","last_active":"2024-04-02T09:13:41.974495+0000","last_peered":"2024-04-02T09:13:41.974495+0000","last_clean":"2024-04-02T09:13:41.974495+0000","last_became_active":"2024-04-02T09:13:41.447385+0000","last_became_peered":"2024-04-02T09:13:41.447385+0000","last_unstale":"2024-04-02T09:13:41.974495+0000","last_undegraded":"2024-04-02T09:13:41.974495+0000","last_fullsized":"2024-04-02T09:13:41.974495+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:21.506479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943796+0000","last_change":"2024-04-02T09:13:28.390273+0000","last_active":"2024-04-02T09:13:41.943796+0000","last_peered":"2024-04-02T09:13:41.943796+0000","last_clean":"2024-04-02T09:13:41.943796+0000","last_became_active":"2024-04-02T09:13:28.389990+0000","last_became_peered":"2024-04-02T09:13:28.389990+0000","last_unstale":"2024-04-02T09:13:41.943796+0000","last_undegraded":"2024-04-02T09:13:41.943796+0000","last_fullsized":"2024-04-02T09:13:41.943796+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:30:33.534225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.948220+0000","last_change":"2024-04-02T09:13:41.958664+0000","last_active":"2024-04-02T09:13:42.948220+0000","last_peered":"2024-04-02T09:13:42.948220+0000","last_clean":"2024-04-02T09:13:42.948220+0000","last_became_active":"2024-04-02T09:13:41.958365+0000","last_became_peered":"2024-04-02T09:13:41.958365+0000","last_unstale":"2024-04-02T09:13:42.948220+0000","last_undegraded":"2024-04-02T09:13:42.948220+0000","last_fullsized":"2024-04-02T09:13:42.948220+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:37.458648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1f","version":"247'4","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.972969+0000","last_change":"2024-04-02T09:13:41.448711+0000","last_active":"2024-04-02T09:13:41.972969+0000","last_peered":"2024-04-02T09:13:41.972969+0000","last_clean":"2024-04-02T09:13:41.972969+0000","last_became_active":"2024-04-02T09:13:41.448398+0000","last_became_peered":"2024-04-02T09:13:41.448398+0000","last_unstale":"2024-04-02T09:13:41.972969+0000","last_undegraded":"2024-04-02T09:13:41.972969+0000","last_fullsized":"2024-04-02T09:13:41.972969+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:39.602788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952747+0000","last_change":"2024-04-02T09:13:28.395003+0000","last_active":"2024-04-02T09:13:41.952747+0000","last_peered":"2024-04-02T09:13:41.952747+0000","last_clean":"2024-04-02T09:13:41.952747+0000","last_became_active":"2024-04-02T09:13:28.394635+0000","last_became_peered":"2024-04-02T09:13:28.394635+0000","last_unstale":"2024-04-02T09:13:41.952747+0000","last_undegraded":"2024-04-02T09:13:41.952747+0000","last_fullsized":"2024-04-02T09:13:41.952747+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:54.758118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.733918+0000","last_change":"2024-04-02T09:13:41.949873+0000","last_active":"2024-04-02T09:13:43.733918+0000","last_peered":"2024-04-02T09:13:43.733918+0000","last_clean":"2024-04-02T09:13:43.733918+0000","last_became_active":"2024-04-02T09:13:41.949617+0000","last_became_peered":"2024-04-02T09:13:41.949617+0000","last_unstale":"2024-04-02T09:13:43.733918+0000","last_undegraded":"2024-04-02T09:13:43.733918+0000","last_fullsized":"2024-04-02T09:13:43.733918+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:19.901756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.954901+0000","last_change":"2024-04-02T09:13:28.391302+0000","last_active":"2024-04-02T09:13:41.954901+0000","last_peered":"2024-04-02T09:13:41.954901+0000","last_clean":"2024-04-02T09:13:41.954901+0000","last_became_active":"2024-04-02T09:13:28.388976+0000","last_became_peered":"2024-04-02T09:13:28.388976+0000","last_unstale":"2024-04-02T09:13:41.954901+0000","last_undegraded":"2024-04-02T09:13:41.954901+0000","last_fullsized":"2024-04-02T09:13:41.954901+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:02.556232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947787+0000","last_change":"2024-04-02T09:13:27.881269+0000","last_active":"2024-04-02T09:13:41.947787+0000","last_peered":"2024-04-02T09:13:41.947787+0000","last_clean":"2024-04-02T09:13:41.947787+0000","last_became_active":"2024-04-02T09:13:27.880664+0000","last_became_peered":"2024-04-02T09:13:27.880664+0000","last_unstale":"2024-04-02T09:13:41.947787+0000","last_undegraded":"2024-04-02T09:13:41.947787+0000","last_fullsized":"2024-04-02T09:13:41.947787+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:53.750473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734852+0000","last_change":"2024-04-02T09:13:41.967103+0000","last_active":"2024-04-02T09:13:43.734852+0000","last_peered":"2024-04-02T09:13:43.734852+0000","last_clean":"2024-04-02T09:13:43.734852+0000","last_became_active":"2024-04-02T09:13:41.966828+0000","last_became_peered":"2024-04-02T09:13:41.966828+0000","last_unstale":"2024-04-02T09:13:43.734852+0000","last_undegraded":"2024-04-02T09:13:43.734852+0000","last_fullsized":"2024-04-02T09:13:43.734852+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:44:48.623486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945056+0000","last_change":"2024-04-02T09:13:28.399529+0000","last_active":"2024-04-02T09:13:41.945056+0000","last_peered":"2024-04-02T09:13:41.945056+0000","last_clean":"2024-04-02T09:13:41.945056+0000","last_became_active":"2024-04-02T09:13:28.399042+0000","last_became_peered":"2024-04-02T09:13:28.399042+0000","last_unstale":"2024-04-02T09:13:41.945056+0000","last_undegraded":"2024-04-02T09:13:41.945056+0000","last_fullsized":"2024-04-02T09:13:41.945056+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:58:19.109584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1d","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.941583+0000","last_change":"2024-04-02T09:13:27.889299+0000","last_active":"2024-04-02T09:13:41.941583+0000","last_peered":"2024-04-02T09:13:41.941583+0000","last_clean":"2024-04-02T09:13:41.941583+0000","last_became_active":"2024-04-02T09:13:27.888249+0000","last_became_peered":"2024-04-02T09:13:27.888249+0000","last_unstale":"2024-04-02T09:13:41.941583+0000","last_undegraded":"2024-04-02T09:13:41.941583+0000","last_fullsized":"2024-04-02T09:13:41.941583+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:01:31.866958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345673+0000","last_change":"2024-04-02T09:13:41.966096+0000","last_active":"2024-04-02T09:13:43.345673+0000","last_peered":"2024-04-02T09:13:43.345673+0000","last_clean":"2024-04-02T09:13:43.345673+0000","last_became_active":"2024-04-02T09:13:41.959139+0000","last_became_peered":"2024-04-02T09:13:41.959139+0000","last_unstale":"2024-04-02T09:13:43.345673+0000","last_undegraded":"2024-04-02T09:13:43.345673+0000","last_fullsized":"2024-04-02T09:13:43.345673+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:54:06.041578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950575+0000","last_change":"2024-04-02T09:13:28.382423+0000","last_active":"2024-04-02T09:13:41.950575+0000","last_peered":"2024-04-02T09:13:41.950575+0000","last_clean":"2024-04-02T09:13:41.950575+0000","last_became_active":"2024-04-02T09:13:28.381840+0000","last_became_peered":"2024-04-02T09:13:28.381840+0000","last_unstale":"2024-04-02T09:13:41.950575+0000","last_undegraded":"2024-04-02T09:13:41.950575+0000","last_fullsized":"2024-04-02T09:13:41.950575+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:04:44.765954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.954520+0000","last_change":"2024-04-02T09:13:27.885629+0000","last_active":"2024-04-02T09:13:41.954520+0000","last_peered":"2024-04-02T09:13:41.954520+0000","last_clean":"2024-04-02T09:13:41.954520+0000","last_became_active":"2024-04-02T09:13:27.884921+0000","last_became_peered":"2024-04-02T09:13:27.884921+0000","last_unstale":"2024-04-02T09:13:41.954520+0000","last_undegraded":"2024-04-02T09:13:41.954520+0000","last_fullsized":"2024-04-02T09:13:41.954520+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:49:58.033026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734939+0000","last_change":"2024-04-02T09:13:41.963609+0000","last_active":"2024-04-02T09:13:43.734939+0000","last_peered":"2024-04-02T09:13:43.734939+0000","last_clean":"2024-04-02T09:13:43.734939+0000","last_became_active":"2024-04-02T09:13:41.963275+0000","last_became_peered":"2024-04-02T09:13:41.963275+0000","last_unstale":"2024-04-02T09:13:43.734939+0000","last_undegraded":"2024-04-02T09:13:43.734939+0000","last_fullsized":"2024-04-02T09:13:43.734939+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:50:02.673300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952776+0000","last_change":"2024-04-02T09:13:28.381558+0000","last_active":"2024-04-02T09:13:41.952776+0000","last_peered":"2024-04-02T09:13:41.952776+0000","last_clean":"2024-04-02T09:13:41.952776+0000","last_became_active":"2024-04-02T09:13:28.381306+0000","last_became_peered":"2024-04-02T09:13:28.381306+0000","last_unstale":"2024-04-02T09:13:41.952776+0000","last_undegraded":"2024-04-02T09:13:41.952776+0000","last_fullsized":"2024-04-02T09:13:41.952776+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:12:17.313507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1b","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.955013+0000","last_change":"2024-04-02T09:13:27.884062+0000","last_active":"2024-04-02T09:13:41.955013+0000","last_peered":"2024-04-02T09:13:41.955013+0000","last_clean":"2024-04-02T09:13:41.955013+0000","last_became_active":"2024-04-02T09:13:27.883677+0000","last_became_peered":"2024-04-02T09:13:27.883677+0000","last_unstale":"2024-04-02T09:13:41.955013+0000","last_undegraded":"2024-04-02T09:13:41.955013+0000","last_fullsized":"2024-04-02T09:13:41.955013+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:14.774442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.948048+0000","last_change":"2024-04-02T09:13:41.955593+0000","last_active":"2024-04-02T09:13:42.948048+0000","last_peered":"2024-04-02T09:13:42.948048+0000","last_clean":"2024-04-02T09:13:42.948048+0000","last_became_active":"2024-04-02T09:13:41.955337+0000","last_became_peered":"2024-04-02T09:13:41.955337+0000","last_unstale":"2024-04-02T09:13:42.948048+0000","last_undegraded":"2024-04-02T09:13:42.948048+0000","last_fullsized":"2024-04-02T09:13:42.948048+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:39:05.298707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943513+0000","last_change":"2024-04-02T09:13:28.389833+0000","last_active":"2024-04-02T09:13:41.943513+0000","last_peered":"2024-04-02T09:13:41.943513+0000","last_clean":"2024-04-02T09:13:41.943513+0000","last_became_active":"2024-04-02T09:13:28.389495+0000","last_became_peered":"2024-04-02T09:13:28.389495+0000","last_unstale":"2024-04-02T09:13:41.943513+0000","last_undegraded":"2024-04-02T09:13:41.943513+0000","last_fullsized":"2024-04-02T09:13:41.943513+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:27.049697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1a","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952655+0000","last_change":"2024-04-02T09:13:27.883532+0000","last_active":"2024-04-02T09:13:41.952655+0000","last_peered":"2024-04-02T09:13:41.952655+0000","last_clean":"2024-04-02T09:13:41.952655+0000","last_became_active":"2024-04-02T09:13:27.883252+0000","last_became_peered":"2024-04-02T09:13:27.883252+0000","last_unstale":"2024-04-02T09:13:41.952655+0000","last_undegraded":"2024-04-02T09:13:41.952655+0000","last_fullsized":"2024-04-02T09:13:41.952655+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.950371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.948318+0000","last_change":"2024-04-02T09:13:41.963566+0000","last_active":"2024-04-02T09:13:42.948318+0000","last_peered":"2024-04-02T09:13:42.948318+0000","last_clean":"2024-04-02T09:13:42.948318+0000","last_became_active":"2024-04-02T09:13:41.962503+0000","last_became_peered":"2024-04-02T09:13:41.962503+0000","last_unstale":"2024-04-02T09:13:42.948318+0000","last_undegraded":"2024-04-02T09:13:42.948318+0000","last_fullsized":"2024-04-02T09:13:42.948318+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:13.220603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943231+0000","last_change":"2024-04-02T09:13:28.403943+0000","last_active":"2024-04-02T09:13:41.943231+0000","last_peered":"2024-04-02T09:13:41.943231+0000","last_clean":"2024-04-02T09:13:41.943231+0000","last_became_active":"2024-04-02T09:13:28.402197+0000","last_became_peered":"2024-04-02T09:13:28.402197+0000","last_unstale":"2024-04-02T09:13:41.943231+0000","last_undegraded":"2024-04-02T09:13:41.943231+0000","last_fullsized":"2024-04-02T09:13:41.943231+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:50.579695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945549+0000","last_change":"2024-04-02T09:13:27.883508+0000","last_active":"2024-04-02T09:13:41.945549+0000","last_peered":"2024-04-02T09:13:41.945549+0000","last_clean":"2024-04-02T09:13:41.945549+0000","last_became_active":"2024-04-02T09:13:27.883263+0000","last_became_peered":"2024-04-02T09:13:27.883263+0000","last_unstale":"2024-04-02T09:13:41.945549+0000","last_undegraded":"2024-04-02T09:13:41.945549+0000","last_fullsized":"2024-04-02T09:13:41.945549+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:37.428725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954612+0000","last_change":"2024-04-02T09:13:41.965533+0000","last_active":"2024-04-02T09:13:42.954612+0000","last_peered":"2024-04-02T09:13:42.954612+0000","last_clean":"2024-04-02T09:13:42.954612+0000","last_became_active":"2024-04-02T09:13:41.965270+0000","last_became_peered":"2024-04-02T09:13:41.965270+0000","last_unstale":"2024-04-02T09:13:42.954612+0000","last_undegraded":"2024-04-02T09:13:42.954612+0000","last_fullsized":"2024-04-02T09:13:42.954612+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:53.549513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943892+0000","last_change":"2024-04-02T09:13:28.403768+0000","last_active":"2024-04-02T09:13:41.943892+0000","last_peered":"2024-04-02T09:13:41.943892+0000","last_clean":"2024-04-02T09:13:41.943892+0000","last_became_active":"2024-04-02T09:13:28.399745+0000","last_became_peered":"2024-04-02T09:13:28.399745+0000","last_unstale":"2024-04-02T09:13:41.943892+0000","last_undegraded":"2024-04-02T09:13:41.943892+0000","last_fullsized":"2024-04-02T09:13:41.943892+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:39.541325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953065+0000","last_change":"2024-04-02T09:13:27.888598+0000","last_active":"2024-04-02T09:13:41.953065+0000","last_peered":"2024-04-02T09:13:41.953065+0000","last_clean":"2024-04-02T09:13:41.953065+0000","last_became_active":"2024-04-02T09:13:27.888115+0000","last_became_peered":"2024-04-02T09:13:27.888115+0000","last_unstale":"2024-04-02T09:13:41.953065+0000","last_undegraded":"2024-04-02T09:13:41.953065+0000","last_fullsized":"2024-04-02T09:13:41.953065+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:57:03.609779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.953354+0000","last_change":"2024-04-02T09:13:41.957572+0000","last_active":"2024-04-02T09:13:42.953354+0000","last_peered":"2024-04-02T09:13:42.953354+0000","last_clean":"2024-04-02T09:13:42.953354+0000","last_became_active":"2024-04-02T09:13:41.956534+0000","last_became_peered":"2024-04-02T09:13:41.956534+0000","last_unstale":"2024-04-02T09:13:42.953354+0000","last_undegraded":"2024-04-02T09:13:42.953354+0000","last_fullsized":"2024-04-02T09:13:42.953354+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:09.258626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947327+0000","last_change":"2024-04-02T09:13:28.404602+0000","last_active":"2024-04-02T09:13:41.947327+0000","last_peered":"2024-04-02T09:13:41.947327+0000","last_clean":"2024-04-02T09:13:41.947327+0000","last_became_active":"2024-04-02T09:13:28.403446+0000","last_became_peered":"2024-04-02T09:13:28.403446+0000","last_unstale":"2024-04-02T09:13:41.947327+0000","last_undegraded":"2024-04-02T09:13:41.947327+0000","last_fullsized":"2024-04-02T09:13:41.947327+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:24.279179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947218+0000","last_change":"2024-04-02T09:13:27.887286+0000","last_active":"2024-04-02T09:13:41.947218+0000","last_peered":"2024-04-02T09:13:41.947218+0000","last_clean":"2024-04-02T09:13:41.947218+0000","last_became_active":"2024-04-02T09:13:27.886999+0000","last_became_peered":"2024-04-02T09:13:27.886999+0000","last_unstale":"2024-04-02T09:13:41.947218+0000","last_undegraded":"2024-04-02T09:13:41.947218+0000","last_fullsized":"2024-04-02T09:13:41.947218+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:44.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342182+0000","last_change":"2024-04-02T09:13:41.961802+0000","last_active":"2024-04-02T09:13:43.342182+0000","last_peered":"2024-04-02T09:13:43.342182+0000","last_clean":"2024-04-02T09:13:43.342182+0000","last_became_active":"2024-04-02T09:13:41.961522+0000","last_became_peered":"2024-04-02T09:13:41.961522+0000","last_unstale":"2024-04-02T09:13:43.342182+0000","last_undegraded":"2024-04-02T09:13:43.342182+0000","last_fullsized":"2024-04-02T09:13:43.342182+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:51:04.142072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.954773+0000","last_change":"2024-04-02T09:13:28.378767+0000","last_active":"2024-04-02T09:13:41.954773+0000","last_peered":"2024-04-02T09:13:41.954773+0000","last_clean":"2024-04-02T09:13:41.954773+0000","last_became_active":"2024-04-02T09:13:28.378487+0000","last_became_peered":"2024-04-02T09:13:28.378487+0000","last_unstale":"2024-04-02T09:13:41.954773+0000","last_undegraded":"2024-04-02T09:13:41.954773+0000","last_fullsized":"2024-04-02T09:13:41.954773+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:26:47.539777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947422+0000","last_change":"2024-04-02T09:13:27.884469+0000","last_active":"2024-04-02T09:13:41.947422+0000","last_peered":"2024-04-02T09:13:41.947422+0000","last_clean":"2024-04-02T09:13:41.947422+0000","last_became_active":"2024-04-02T09:13:27.882570+0000","last_became_peered":"2024-04-02T09:13:27.882570+0000","last_unstale":"2024-04-02T09:13:41.947422+0000","last_undegraded":"2024-04-02T09:13:41.947422+0000","last_fullsized":"2024-04-02T09:13:41.947422+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:57:12.395680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.953682+0000","last_change":"2024-04-02T09:13:41.959972+0000","last_active":"2024-04-02T09:13:42.953682+0000","last_peered":"2024-04-02T09:13:42.953682+0000","last_clean":"2024-04-02T09:13:42.953682+0000","last_became_active":"2024-04-02T09:13:41.959655+0000","last_became_peered":"2024-04-02T09:13:41.959655+0000","last_unstale":"2024-04-02T09:13:42.953682+0000","last_undegraded":"2024-04-02T09:13:42.953682+0000","last_fullsized":"2024-04-02T09:13:42.953682+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:23:35.907439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.959311+0000","last_change":"2024-04-02T09:13:28.387205+0000","last_active":"2024-04-02T09:13:41.959311+0000","last_peered":"2024-04-02T09:13:41.959311+0000","last_clean":"2024-04-02T09:13:41.959311+0000","last_became_active":"2024-04-02T09:13:28.384126+0000","last_became_peered":"2024-04-02T09:13:28.384126+0000","last_unstale":"2024-04-02T09:13:41.959311+0000","last_undegraded":"2024-04-02T09:13:41.959311+0000","last_fullsized":"2024-04-02T09:13:41.959311+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:52.330349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.15","version":"243'3","reported_seq":18,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.962102+0000","last_change":"2024-04-02T09:13:27.883818+0000","last_active":"2024-04-02T09:13:41.962102+0000","last_peered":"2024-04-02T09:13:41.962102+0000","last_clean":"2024-04-02T09:13:41.962102+0000","last_became_active":"2024-04-02T09:13:27.883527+0000","last_became_peered":"2024-04-02T09:13:27.883527+0000","last_unstale":"2024-04-02T09:13:41.962102+0000","last_undegraded":"2024-04-02T09:13:41.962102+0000","last_fullsized":"2024-04-02T09:13:41.962102+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:49.572049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954621+0000","last_change":"2024-04-02T09:13:41.948944+0000","last_active":"2024-04-02T09:13:42.954621+0000","last_peered":"2024-04-02T09:13:42.954621+0000","last_clean":"2024-04-02T09:13:42.954621+0000","last_became_active":"2024-04-02T09:13:41.948653+0000","last_became_peered":"2024-04-02T09:13:41.948653+0000","last_unstale":"2024-04-02T09:13:42.954621+0000","last_undegraded":"2024-04-02T09:13:42.954621+0000","last_fullsized":"2024-04-02T09:13:42.954621+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:12:23.778874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948000+0000","last_change":"2024-04-02T09:13:28.379346+0000","last_active":"2024-04-02T09:13:41.948000+0000","last_peered":"2024-04-02T09:13:41.948000+0000","last_clean":"2024-04-02T09:13:41.948000+0000","last_became_active":"2024-04-02T09:13:28.378741+0000","last_became_peered":"2024-04-02T09:13:28.378741+0000","last_unstale":"2024-04-02T09:13:41.948000+0000","last_undegraded":"2024-04-02T09:13:41.948000+0000","last_fullsized":"2024-04-02T09:13:41.948000+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:44.092529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.14","version":"244'8","reported_seq":23,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.941516+0000","last_change":"2024-04-02T09:13:27.885270+0000","last_active":"2024-04-02T09:13:41.941516+0000","last_peered":"2024-04-02T09:13:41.941516+0000","last_clean":"2024-04-02T09:13:41.941516+0000","last_became_active":"2024-04-02T09:13:27.884953+0000","last_became_peered":"2024-04-02T09:13:27.884953+0000","last_unstale":"2024-04-02T09:13:41.941516+0000","last_undegraded":"2024-04-02T09:13:41.941516+0000","last_fullsized":"2024-04-02T09:13:41.941516+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:48.103277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2805,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.953123+0000","last_change":"2024-04-02T09:13:41.955589+0000","last_active":"2024-04-02T09:13:42.953123+0000","last_peered":"2024-04-02T09:13:42.953123+0000","last_clean":"2024-04-02T09:13:42.953123+0000","last_became_active":"2024-04-02T09:13:41.955331+0000","last_became_peered":"2024-04-02T09:13:41.955331+0000","last_unstale":"2024-04-02T09:13:42.953123+0000","last_undegraded":"2024-04-02T09:13:42.953123+0000","last_fullsized":"2024-04-02T09:13:42.953123+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:35.863141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943802+0000","last_change":"2024-04-02T09:13:28.393735+0000","last_active":"2024-04-02T09:13:41.943802+0000","last_peered":"2024-04-02T09:13:41.943802+0000","last_clean":"2024-04-02T09:13:41.943802+0000","last_became_active":"2024-04-02T09:13:28.393387+0000","last_became_peered":"2024-04-02T09:13:28.393387+0000","last_unstale":"2024-04-02T09:13:41.943802+0000","last_undegraded":"2024-04-02T09:13:41.943802+0000","last_fullsized":"2024-04-02T09:13:41.943802+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:47.359176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.13","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940581+0000","last_change":"2024-04-02T09:13:27.885596+0000","last_active":"2024-04-02T09:13:41.940581+0000","last_peered":"2024-04-02T09:13:41.940581+0000","last_clean":"2024-04-02T09:13:41.940581+0000","last_became_active":"2024-04-02T09:13:27.885310+0000","last_became_peered":"2024-04-02T09:13:27.885310+0000","last_unstale":"2024-04-02T09:13:41.940581+0000","last_undegraded":"2024-04-02T09:13:41.940581+0000","last_fullsized":"2024-04-02T09:13:41.940581+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:02.497118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342453+0000","last_change":"2024-04-02T09:13:41.962021+0000","last_active":"2024-04-02T09:13:43.342453+0000","last_peered":"2024-04-02T09:13:43.342453+0000","last_clean":"2024-04-02T09:13:43.342453+0000","last_became_active":"2024-04-02T09:13:41.961212+0000","last_became_peered":"2024-04-02T09:13:41.961212+0000","last_unstale":"2024-04-02T09:13:43.342453+0000","last_undegraded":"2024-04-02T09:13:43.342453+0000","last_fullsized":"2024-04-02T09:13:43.342453+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:37.841557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950657+0000","last_change":"2024-04-02T09:13:28.378384+0000","last_active":"2024-04-02T09:13:41.950657+0000","last_peered":"2024-04-02T09:13:41.950657+0000","last_clean":"2024-04-02T09:13:41.950657+0000","last_became_active":"2024-04-02T09:13:28.377833+0000","last_became_peered":"2024-04-02T09:13:28.377833+0000","last_unstale":"2024-04-02T09:13:41.950657+0000","last_undegraded":"2024-04-02T09:13:41.950657+0000","last_fullsized":"2024-04-02T09:13:41.950657+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:37:54.526506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942840+0000","last_change":"2024-04-02T09:13:27.880069+0000","last_active":"2024-04-02T09:13:41.942840+0000","last_peered":"2024-04-02T09:13:41.942840+0000","last_clean":"2024-04-02T09:13:41.942840+0000","last_became_active":"2024-04-02T09:13:27.879793+0000","last_became_peered":"2024-04-02T09:13:27.879793+0000","last_unstale":"2024-04-02T09:13:41.942840+0000","last_undegraded":"2024-04-02T09:13:41.942840+0000","last_fullsized":"2024-04-02T09:13:41.942840+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:10.198371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734950+0000","last_change":"2024-04-02T09:13:41.951761+0000","last_active":"2024-04-02T09:13:43.734950+0000","last_peered":"2024-04-02T09:13:43.734950+0000","last_clean":"2024-04-02T09:13:43.734950+0000","last_became_active":"2024-04-02T09:13:41.951447+0000","last_became_peered":"2024-04-02T09:13:41.951447+0000","last_unstale":"2024-04-02T09:13:43.734950+0000","last_undegraded":"2024-04-02T09:13:43.734950+0000","last_fullsized":"2024-04-02T09:13:43.734950+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:37.458648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952962+0000","last_change":"2024-04-02T09:13:28.394088+0000","last_active":"2024-04-02T09:13:41.952962+0000","last_peered":"2024-04-02T09:13:41.952962+0000","last_clean":"2024-04-02T09:13:41.952962+0000","last_became_active":"2024-04-02T09:13:28.392932+0000","last_became_peered":"2024-04-02T09:13:28.392932+0000","last_unstale":"2024-04-02T09:13:41.952962+0000","last_undegraded":"2024-04-02T09:13:41.952962+0000","last_fullsized":"2024-04-02T09:13:41.952962+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:54.683721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945630+0000","last_change":"2024-04-02T09:13:27.881714+0000","last_active":"2024-04-02T09:13:41.945630+0000","last_peered":"2024-04-02T09:13:41.945630+0000","last_clean":"2024-04-02T09:13:41.945630+0000","last_became_active":"2024-04-02T09:13:27.881428+0000","last_became_peered":"2024-04-02T09:13:27.881428+0000","last_unstale":"2024-04-02T09:13:41.945630+0000","last_undegraded":"2024-04-02T09:13:41.945630+0000","last_fullsized":"2024-04-02T09:13:41.945630+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:50.081688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.346442+0000","last_change":"2024-04-02T09:13:41.972497+0000","last_active":"2024-04-02T09:13:43.346442+0000","last_peered":"2024-04-02T09:13:43.346442+0000","last_clean":"2024-04-02T09:13:43.346442+0000","last_became_active":"2024-04-02T09:13:41.972137+0000","last_became_peered":"2024-04-02T09:13:41.972137+0000","last_unstale":"2024-04-02T09:13:43.346442+0000","last_undegraded":"2024-04-02T09:13:43.346442+0000","last_fullsized":"2024-04-02T09:13:43.346442+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:19.631975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944595+0000","last_change":"2024-04-02T09:13:28.391188+0000","last_active":"2024-04-02T09:13:41.944595+0000","last_peered":"2024-04-02T09:13:41.944595+0000","last_clean":"2024-04-02T09:13:41.944595+0000","last_became_active":"2024-04-02T09:13:28.390867+0000","last_became_peered":"2024-04-02T09:13:28.390867+0000","last_unstale":"2024-04-02T09:13:41.944595+0000","last_undegraded":"2024-04-02T09:13:41.944595+0000","last_fullsized":"2024-04-02T09:13:41.944595+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:26:58.531401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.10","version":"243'8","reported_seq":23,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952918+0000","last_change":"2024-04-02T09:13:27.886503+0000","last_active":"2024-04-02T09:13:41.952918+0000","last_peered":"2024-04-02T09:13:41.952918+0000","last_clean":"2024-04-02T09:13:41.952918+0000","last_became_active":"2024-04-02T09:13:27.886121+0000","last_became_peered":"2024-04-02T09:13:27.886121+0000","last_unstale":"2024-04-02T09:13:41.952918+0000","last_undegraded":"2024-04-02T09:13:41.952918+0000","last_fullsized":"2024-04-02T09:13:41.952918+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:38.533680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952724+0000","last_change":"2024-04-02T09:13:41.954229+0000","last_active":"2024-04-02T09:13:42.952724+0000","last_peered":"2024-04-02T09:13:42.952724+0000","last_clean":"2024-04-02T09:13:42.952724+0000","last_became_active":"2024-04-02T09:13:41.950847+0000","last_became_peered":"2024-04-02T09:13:41.950847+0000","last_unstale":"2024-04-02T09:13:42.952724+0000","last_undegraded":"2024-04-02T09:13:42.952724+0000","last_fullsized":"2024-04-02T09:13:42.952724+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:58.664898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.955366+0000","last_change":"2024-04-02T09:13:28.376609+0000","last_active":"2024-04-02T09:13:41.955366+0000","last_peered":"2024-04-02T09:13:41.955366+0000","last_clean":"2024-04-02T09:13:41.955366+0000","last_became_active":"2024-04-02T09:13:28.376226+0000","last_became_peered":"2024-04-02T09:13:28.376226+0000","last_unstale":"2024-04-02T09:13:41.955366+0000","last_undegraded":"2024-04-02T09:13:41.955366+0000","last_fullsized":"2024-04-02T09:13:41.955366+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:59.766007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948352+0000","last_change":"2024-04-02T09:13:27.883080+0000","last_active":"2024-04-02T09:13:41.948352+0000","last_peered":"2024-04-02T09:13:41.948352+0000","last_clean":"2024-04-02T09:13:41.948352+0000","last_became_active":"2024-04-02T09:13:27.882612+0000","last_became_peered":"2024-04-02T09:13:27.882612+0000","last_unstale":"2024-04-02T09:13:41.948352+0000","last_undegraded":"2024-04-02T09:13:41.948352+0000","last_fullsized":"2024-04-02T09:13:41.948352+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:20.979570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.735089+0000","last_change":"2024-04-02T09:13:41.960521+0000","last_active":"2024-04-02T09:13:43.735089+0000","last_peered":"2024-04-02T09:13:43.735089+0000","last_clean":"2024-04-02T09:13:43.735089+0000","last_became_active":"2024-04-02T09:13:41.960258+0000","last_became_peered":"2024-04-02T09:13:41.960258+0000","last_unstale":"2024-04-02T09:13:43.735089+0000","last_undegraded":"2024-04-02T09:13:43.735089+0000","last_fullsized":"2024-04-02T09:13:43.735089+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:45:09.502243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.961541+0000","last_change":"2024-04-02T09:13:28.381499+0000","last_active":"2024-04-02T09:13:41.961541+0000","last_peered":"2024-04-02T09:13:41.961541+0000","last_clean":"2024-04-02T09:13:41.961541+0000","last_became_active":"2024-04-02T09:13:28.381286+0000","last_became_peered":"2024-04-02T09:13:28.381286+0000","last_unstale":"2024-04-02T09:13:41.961541+0000","last_undegraded":"2024-04-02T09:13:41.961541+0000","last_fullsized":"2024-04-02T09:13:41.961541+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:18.748067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.e","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947518+0000","last_change":"2024-04-02T09:13:27.888558+0000","last_active":"2024-04-02T09:13:41.947518+0000","last_peered":"2024-04-02T09:13:41.947518+0000","last_clean":"2024-04-02T09:13:41.947518+0000","last_became_active":"2024-04-02T09:13:27.888288+0000","last_became_peered":"2024-04-02T09:13:27.888288+0000","last_unstale":"2024-04-02T09:13:41.947518+0000","last_undegraded":"2024-04-02T09:13:41.947518+0000","last_fullsized":"2024-04-02T09:13:41.947518+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:15.386238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734712+0000","last_change":"2024-04-02T09:13:41.967882+0000","last_active":"2024-04-02T09:13:43.734712+0000","last_peered":"2024-04-02T09:13:43.734712+0000","last_clean":"2024-04-02T09:13:43.734712+0000","last_became_active":"2024-04-02T09:13:41.967598+0000","last_became_peered":"2024-04-02T09:13:41.967598+0000","last_unstale":"2024-04-02T09:13:43.734712+0000","last_undegraded":"2024-04-02T09:13:43.734712+0000","last_fullsized":"2024-04-02T09:13:43.734712+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:18.838465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948337+0000","last_change":"2024-04-02T09:13:28.372849+0000","last_active":"2024-04-02T09:13:41.948337+0000","last_peered":"2024-04-02T09:13:41.948337+0000","last_clean":"2024-04-02T09:13:41.948337+0000","last_became_active":"2024-04-02T09:13:28.372595+0000","last_became_peered":"2024-04-02T09:13:28.372595+0000","last_unstale":"2024-04-02T09:13:41.948337+0000","last_undegraded":"2024-04-02T09:13:41.948337+0000","last_fullsized":"2024-04-02T09:13:41.948337+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:12.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.d","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947395+0000","last_change":"2024-04-02T09:13:27.879417+0000","last_active":"2024-04-02T09:13:41.947395+0000","last_peered":"2024-04-02T09:13:41.947395+0000","last_clean":"2024-04-02T09:13:41.947395+0000","last_became_active":"2024-04-02T09:13:27.879165+0000","last_became_peered":"2024-04-02T09:13:27.879165+0000","last_unstale":"2024-04-02T09:13:41.947395+0000","last_undegraded":"2024-04-02T09:13:41.947395+0000","last_fullsized":"2024-04-02T09:13:41.947395+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:47:32.233662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734901+0000","last_change":"2024-04-02T09:13:41.968877+0000","last_active":"2024-04-02T09:13:43.734901+0000","last_peered":"2024-04-02T09:13:43.734901+0000","last_clean":"2024-04-02T09:13:43.734901+0000","last_became_active":"2024-04-02T09:13:41.968456+0000","last_became_peered":"2024-04-02T09:13:41.968456+0000","last_unstale":"2024-04-02T09:13:43.734901+0000","last_undegraded":"2024-04-02T09:13:43.734901+0000","last_fullsized":"2024-04-02T09:13:43.734901+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:54.806757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.939446+0000","last_change":"2024-04-02T09:13:28.392130+0000","last_active":"2024-04-02T09:13:41.939446+0000","last_peered":"2024-04-02T09:13:41.939446+0000","last_clean":"2024-04-02T09:13:41.939446+0000","last_became_active":"2024-04-02T09:13:28.391690+0000","last_became_peered":"2024-04-02T09:13:28.391690+0000","last_unstale":"2024-04-02T09:13:41.939446+0000","last_undegraded":"2024-04-02T09:13:41.939446+0000","last_fullsized":"2024-04-02T09:13:41.939446+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:24.319747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945343+0000","last_change":"2024-04-02T09:13:27.884781+0000","last_active":"2024-04-02T09:13:41.945343+0000","last_peered":"2024-04-02T09:13:41.945343+0000","last_clean":"2024-04-02T09:13:41.945343+0000","last_became_active":"2024-04-02T09:13:27.884507+0000","last_became_peered":"2024-04-02T09:13:27.884507+0000","last_unstale":"2024-04-02T09:13:41.945343+0000","last_undegraded":"2024-04-02T09:13:41.945343+0000","last_fullsized":"2024-04-02T09:13:41.945343+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:31:18.889273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342453+0000","last_change":"2024-04-02T09:13:41.958635+0000","last_active":"2024-04-02T09:13:43.342453+0000","last_peered":"2024-04-02T09:13:43.342453+0000","last_clean":"2024-04-02T09:13:43.342453+0000","last_became_active":"2024-04-02T09:13:41.955459+0000","last_became_peered":"2024-04-02T09:13:41.955459+0000","last_unstale":"2024-04-02T09:13:43.342453+0000","last_undegraded":"2024-04-02T09:13:43.342453+0000","last_fullsized":"2024-04-02T09:13:43.342453+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:53.103702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947024+0000","last_change":"2024-04-02T09:13:28.381580+0000","last_active":"2024-04-02T09:13:41.947024+0000","last_peered":"2024-04-02T09:13:41.947024+0000","last_clean":"2024-04-02T09:13:41.947024+0000","last_became_active":"2024-04-02T09:13:28.381300+0000","last_became_peered":"2024-04-02T09:13:28.381300+0000","last_unstale":"2024-04-02T09:13:41.947024+0000","last_undegraded":"2024-04-02T09:13:41.947024+0000","last_fullsized":"2024-04-02T09:13:41.947024+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:51.089333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944130+0000","last_change":"2024-04-02T09:13:27.875291+0000","last_active":"2024-04-02T09:13:41.944130+0000","last_peered":"2024-04-02T09:13:41.944130+0000","last_clean":"2024-04-02T09:13:41.944130+0000","last_became_active":"2024-04-02T09:13:27.875003+0000","last_became_peered":"2024-04-02T09:13:27.875003+0000","last_unstale":"2024-04-02T09:13:41.944130+0000","last_undegraded":"2024-04-02T09:13:41.944130+0000","last_fullsized":"2024-04-02T09:13:41.944130+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:00.637981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734207+0000","last_change":"2024-04-02T09:13:41.949115+0000","last_active":"2024-04-02T09:13:43.734207+0000","last_peered":"2024-04-02T09:13:43.734207+0000","last_clean":"2024-04-02T09:13:43.734207+0000","last_became_active":"2024-04-02T09:13:41.948850+0000","last_became_peered":"2024-04-02T09:13:41.948850+0000","last_unstale":"2024-04-02T09:13:43.734207+0000","last_undegraded":"2024-04-02T09:13:43.734207+0000","last_fullsized":"2024-04-02T09:13:43.734207+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:54.806757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944021+0000","last_change":"2024-04-02T09:13:28.404265+0000","last_active":"2024-04-02T09:13:41.944021+0000","last_peered":"2024-04-02T09:13:41.944021+0000","last_clean":"2024-04-02T09:13:41.944021+0000","last_became_active":"2024-04-02T09:13:28.403894+0000","last_became_peered":"2024-04-02T09:13:28.403894+0000","last_unstale":"2024-04-02T09:13:41.944021+0000","last_undegraded":"2024-04-02T09:13:41.944021+0000","last_fullsized":"2024-04-02T09:13:41.944021+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:05:23.433426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943067+0000","last_change":"2024-04-02T09:13:27.881604+0000","last_active":"2024-04-02T09:13:41.943067+0000","last_peered":"2024-04-02T09:13:41.943067+0000","last_clean":"2024-04-02T09:13:41.943067+0000","last_became_active":"2024-04-02T09:13:27.881304+0000","last_became_peered":"2024-04-02T09:13:27.881304+0000","last_unstale":"2024-04-02T09:13:41.943067+0000","last_undegraded":"2024-04-02T09:13:41.943067+0000","last_fullsized":"2024-04-02T09:13:41.943067+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:02.497118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734857+0000","last_change":"2024-04-02T09:13:41.956788+0000","last_active":"2024-04-02T09:13:43.734857+0000","last_peered":"2024-04-02T09:13:43.734857+0000","last_clean":"2024-04-02T09:13:43.734857+0000","last_became_active":"2024-04-02T09:13:41.956513+0000","last_became_peered":"2024-04-02T09:13:41.956513+0000","last_unstale":"2024-04-02T09:13:43.734857+0000","last_undegraded":"2024-04-02T09:13:43.734857+0000","last_fullsized":"2024-04-02T09:13:43.734857+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:39:05.298707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953053+0000","last_change":"2024-04-02T09:13:28.374960+0000","last_active":"2024-04-02T09:13:41.953053+0000","last_peered":"2024-04-02T09:13:41.953053+0000","last_clean":"2024-04-02T09:13:41.953053+0000","last_became_active":"2024-04-02T09:13:28.373683+0000","last_became_peered":"2024-04-02T09:13:28.373683+0000","last_unstale":"2024-04-02T09:13:41.953053+0000","last_undegraded":"2024-04-02T09:13:41.953053+0000","last_fullsized":"2024-04-02T09:13:41.953053+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:15.851301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.938571+0000","last_change":"2024-04-02T09:13:27.877837+0000","last_active":"2024-04-02T09:13:41.938571+0000","last_peered":"2024-04-02T09:13:41.938571+0000","last_clean":"2024-04-02T09:13:41.938571+0000","last_became_active":"2024-04-02T09:13:27.876792+0000","last_became_peered":"2024-04-02T09:13:27.876792+0000","last_unstale":"2024-04-02T09:13:41.938571+0000","last_undegraded":"2024-04-02T09:13:41.938571+0000","last_fullsized":"2024-04-02T09:13:41.938571+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:17.740422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954331+0000","last_change":"2024-04-02T09:13:41.949429+0000","last_active":"2024-04-02T09:13:42.954331+0000","last_peered":"2024-04-02T09:13:42.954331+0000","last_clean":"2024-04-02T09:13:42.954331+0000","last_became_active":"2024-04-02T09:13:41.948422+0000","last_became_peered":"2024-04-02T09:13:41.948422+0000","last_unstale":"2024-04-02T09:13:42.954331+0000","last_undegraded":"2024-04-02T09:13:42.954331+0000","last_fullsized":"2024-04-02T09:13:42.954331+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:31:40.330222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945371+0000","last_change":"2024-04-02T09:13:28.379243+0000","last_active":"2024-04-02T09:13:41.945371+0000","last_peered":"2024-04-02T09:13:41.945371+0000","last_clean":"2024-04-02T09:13:41.945371+0000","last_became_active":"2024-04-02T09:13:28.378297+0000","last_became_peered":"2024-04-02T09:13:28.378297+0000","last_unstale":"2024-04-02T09:13:41.945371+0000","last_undegraded":"2024-04-02T09:13:41.945371+0000","last_fullsized":"2024-04-02T09:13:41.945371+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:02.761190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.938490+0000","last_change":"2024-04-02T09:13:27.876051+0000","last_active":"2024-04-02T09:13:41.938490+0000","last_peered":"2024-04-02T09:13:41.938490+0000","last_clean":"2024-04-02T09:13:41.938490+0000","last_became_active":"2024-04-02T09:13:27.875628+0000","last_became_peered":"2024-04-02T09:13:27.875628+0000","last_unstale":"2024-04-02T09:13:41.938490+0000","last_undegraded":"2024-04-02T09:13:41.938490+0000","last_fullsized":"2024-04-02T09:13:41.938490+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:53.750473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.947186+0000","last_change":"2024-04-02T09:13:41.961652+0000","last_active":"2024-04-02T09:13:42.947186+0000","last_peered":"2024-04-02T09:13:42.947186+0000","last_clean":"2024-04-02T09:13:42.947186+0000","last_became_active":"2024-04-02T09:13:41.961378+0000","last_became_peered":"2024-04-02T09:13:41.961378+0000","last_unstale":"2024-04-02T09:13:42.947186+0000","last_undegraded":"2024-04-02T09:13:42.947186+0000","last_fullsized":"2024-04-02T09:13:42.947186+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:51:51.721965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947638+0000","last_change":"2024-04-02T09:13:28.383204+0000","last_active":"2024-04-02T09:13:41.947638+0000","last_peered":"2024-04-02T09:13:41.947638+0000","last_clean":"2024-04-02T09:13:41.947638+0000","last_became_active":"2024-04-02T09:13:28.382961+0000","last_became_peered":"2024-04-02T09:13:28.382961+0000","last_unstale":"2024-04-02T09:13:41.947638+0000","last_undegraded":"2024-04-02T09:13:41.947638+0000","last_fullsized":"2024-04-02T09:13:41.947638+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:16:14.265813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.7","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946858+0000","last_change":"2024-04-02T09:13:27.886740+0000","last_active":"2024-04-02T09:13:41.946858+0000","last_peered":"2024-04-02T09:13:41.946858+0000","last_clean":"2024-04-02T09:13:41.946858+0000","last_became_active":"2024-04-02T09:13:27.886385+0000","last_became_peered":"2024-04-02T09:13:27.886385+0000","last_unstale":"2024-04-02T09:13:41.946858+0000","last_undegraded":"2024-04-02T09:13:41.946858+0000","last_fullsized":"2024-04-02T09:13:41.946858+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:20.383235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734173+0000","last_change":"2024-04-02T09:13:41.954104+0000","last_active":"2024-04-02T09:13:43.734173+0000","last_peered":"2024-04-02T09:13:43.734173+0000","last_clean":"2024-04-02T09:13:43.734173+0000","last_became_active":"2024-04-02T09:13:41.953393+0000","last_became_peered":"2024-04-02T09:13:41.953393+0000","last_unstale":"2024-04-02T09:13:43.734173+0000","last_undegraded":"2024-04-02T09:13:43.734173+0000","last_fullsized":"2024-04-02T09:13:43.734173+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:01:46.436981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942847+0000","last_change":"2024-04-02T09:13:28.402497+0000","last_active":"2024-04-02T09:13:41.942847+0000","last_peered":"2024-04-02T09:13:41.942847+0000","last_clean":"2024-04-02T09:13:41.942847+0000","last_became_active":"2024-04-02T09:13:28.402235+0000","last_became_peered":"2024-04-02T09:13:28.402235+0000","last_unstale":"2024-04-02T09:13:41.942847+0000","last_undegraded":"2024-04-02T09:13:41.942847+0000","last_fullsized":"2024-04-02T09:13:41.942847+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:15.072346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.6","version":"244'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950712+0000","last_change":"2024-04-02T09:13:27.884225+0000","last_active":"2024-04-02T09:13:41.950712+0000","last_peered":"2024-04-02T09:13:41.950712+0000","last_clean":"2024-04-02T09:13:41.950712+0000","last_became_active":"2024-04-02T09:13:27.883536+0000","last_became_peered":"2024-04-02T09:13:27.883536+0000","last_unstale":"2024-04-02T09:13:41.950712+0000","last_undegraded":"2024-04-02T09:13:41.950712+0000","last_fullsized":"2024-04-02T09:13:41.950712+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:50.645932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345681+0000","last_change":"2024-04-02T09:13:41.955484+0000","last_active":"2024-04-02T09:13:43.345681+0000","last_peered":"2024-04-02T09:13:43.345681+0000","last_clean":"2024-04-02T09:13:43.345681+0000","last_became_active":"2024-04-02T09:13:41.953502+0000","last_became_peered":"2024-04-02T09:13:41.953502+0000","last_unstale":"2024-04-02T09:13:43.345681+0000","last_undegraded":"2024-04-02T09:13:43.345681+0000","last_fullsized":"2024-04-02T09:13:43.345681+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:50.626394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953832+0000","last_change":"2024-04-02T09:13:28.387139+0000","last_active":"2024-04-02T09:13:41.953832+0000","last_peered":"2024-04-02T09:13:41.953832+0000","last_clean":"2024-04-02T09:13:41.953832+0000","last_became_active":"2024-04-02T09:13:28.386878+0000","last_became_peered":"2024-04-02T09:13:28.386878+0000","last_unstale":"2024-04-02T09:13:41.953832+0000","last_undegraded":"2024-04-02T09:13:41.953832+0000","last_fullsized":"2024-04-02T09:13:41.953832+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:56:34.681485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948085+0000","last_change":"2024-04-02T09:13:27.882615+0000","last_active":"2024-04-02T09:13:41.948085+0000","last_peered":"2024-04-02T09:13:41.948085+0000","last_clean":"2024-04-02T09:13:41.948085+0000","last_became_active":"2024-04-02T09:13:27.882334+0000","last_became_peered":"2024-04-02T09:13:27.882334+0000","last_unstale":"2024-04-02T09:13:41.948085+0000","last_undegraded":"2024-04-02T09:13:41.948085+0000","last_fullsized":"2024-04-02T09:13:41.948085+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:43.215347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.733778+0000","last_change":"2024-04-02T09:13:41.962699+0000","last_active":"2024-04-02T09:13:43.733778+0000","last_peered":"2024-04-02T09:13:43.733778+0000","last_clean":"2024-04-02T09:13:43.733778+0000","last_became_active":"2024-04-02T09:13:41.960572+0000","last_became_peered":"2024-04-02T09:13:41.960572+0000","last_unstale":"2024-04-02T09:13:43.733778+0000","last_undegraded":"2024-04-02T09:13:43.733778+0000","last_fullsized":"2024-04-02T09:13:43.733778+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:22:21.179882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946870+0000","last_change":"2024-04-02T09:13:28.402303+0000","last_active":"2024-04-02T09:13:41.946870+0000","last_peered":"2024-04-02T09:13:41.946870+0000","last_clean":"2024-04-02T09:13:41.946870+0000","last_became_active":"2024-04-02T09:13:28.399305+0000","last_became_peered":"2024-04-02T09:13:28.399305+0000","last_unstale":"2024-04-02T09:13:41.946870+0000","last_undegraded":"2024-04-02T09:13:41.946870+0000","last_fullsized":"2024-04-02T09:13:41.946870+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:58.927786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945487+0000","last_change":"2024-04-02T09:13:27.881268+0000","last_active":"2024-04-02T09:13:41.945487+0000","last_peered":"2024-04-02T09:13:41.945487+0000","last_clean":"2024-04-02T09:13:41.945487+0000","last_became_active":"2024-04-02T09:13:27.880983+0000","last_became_peered":"2024-04-02T09:13:27.880983+0000","last_unstale":"2024-04-02T09:13:41.945487+0000","last_undegraded":"2024-04-02T09:13:41.945487+0000","last_fullsized":"2024-04-02T09:13:41.945487+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:09.803095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734037+0000","last_change":"2024-04-02T09:13:41.955819+0000","last_active":"2024-04-02T09:13:43.734037+0000","last_peered":"2024-04-02T09:13:43.734037+0000","last_clean":"2024-04-02T09:13:43.734037+0000","last_became_active":"2024-04-02T09:13:41.955568+0000","last_became_peered":"2024-04-02T09:13:41.955568+0000","last_unstale":"2024-04-02T09:13:43.734037+0000","last_undegraded":"2024-04-02T09:13:43.734037+0000","last_fullsized":"2024-04-02T09:13:43.734037+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:44:48.623486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.949252+0000","last_change":"2024-04-02T09:13:28.374285+0000","last_active":"2024-04-02T09:13:41.949252+0000","last_peered":"2024-04-02T09:13:41.949252+0000","last_clean":"2024-04-02T09:13:41.949252+0000","last_became_active":"2024-04-02T09:13:28.373980+0000","last_became_peered":"2024-04-02T09:13:28.373980+0000","last_unstale":"2024-04-02T09:13:41.949252+0000","last_undegraded":"2024-04-02T09:13:41.949252+0000","last_fullsized":"2024-04-02T09:13:41.949252+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:56:34.681485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947216+0000","last_change":"2024-04-02T09:13:27.882249+0000","last_active":"2024-04-02T09:13:41.947216+0000","last_peered":"2024-04-02T09:13:41.947216+0000","last_clean":"2024-04-02T09:13:41.947216+0000","last_became_active":"2024-04-02T09:13:27.881833+0000","last_became_peered":"2024-04-02T09:13:27.881833+0000","last_unstale":"2024-04-02T09:13:41.947216+0000","last_undegraded":"2024-04-02T09:13:41.947216+0000","last_fullsized":"2024-04-02T09:13:41.947216+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:22.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342582+0000","last_change":"2024-04-02T09:13:41.960270+0000","last_active":"2024-04-02T09:13:43.342582+0000","last_peered":"2024-04-02T09:13:43.342582+0000","last_clean":"2024-04-02T09:13:43.342582+0000","last_became_active":"2024-04-02T09:13:41.960009+0000","last_became_peered":"2024-04-02T09:13:41.960009+0000","last_unstale":"2024-04-02T09:13:43.342582+0000","last_undegraded":"2024-04-02T09:13:43.342582+0000","last_fullsized":"2024-04-02T09:13:43.342582+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:37:52.590813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948199+0000","last_change":"2024-04-02T09:13:28.389208+0000","last_active":"2024-04-02T09:13:41.948199+0000","last_peered":"2024-04-02T09:13:41.948199+0000","last_clean":"2024-04-02T09:13:41.948199+0000","last_became_active":"2024-04-02T09:13:28.388923+0000","last_became_peered":"2024-04-02T09:13:28.388923+0000","last_unstale":"2024-04-02T09:13:41.948199+0000","last_undegraded":"2024-04-02T09:13:41.948199+0000","last_fullsized":"2024-04-02T09:13:41.948199+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:35:55.299814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953418+0000","last_change":"2024-04-02T09:13:27.887509+0000","last_active":"2024-04-02T09:13:41.953418+0000","last_peered":"2024-04-02T09:13:41.953418+0000","last_clean":"2024-04-02T09:13:41.953418+0000","last_became_active":"2024-04-02T09:13:27.887260+0000","last_became_peered":"2024-04-02T09:13:27.887260+0000","last_unstale":"2024-04-02T09:13:41.953418+0000","last_undegraded":"2024-04-02T09:13:41.953418+0000","last_fullsized":"2024-04-02T09:13:41.953418+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:28.581007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734767+0000","last_change":"2024-04-02T09:13:41.960376+0000","last_active":"2024-04-02T09:13:43.734767+0000","last_peered":"2024-04-02T09:13:43.734767+0000","last_clean":"2024-04-02T09:13:43.734767+0000","last_became_active":"2024-04-02T09:13:41.960062+0000","last_became_peered":"2024-04-02T09:13:41.960062+0000","last_unstale":"2024-04-02T09:13:43.734767+0000","last_undegraded":"2024-04-02T09:13:43.734767+0000","last_fullsized":"2024-04-02T09:13:43.734767+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:13.220603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.937842+0000","last_change":"2024-04-02T09:13:28.374292+0000","last_active":"2024-04-02T09:13:41.937842+0000","last_peered":"2024-04-02T09:13:41.937842+0000","last_clean":"2024-04-02T09:13:41.937842+0000","last_became_active":"2024-04-02T09:13:28.373689+0000","last_became_peered":"2024-04-02T09:13:28.373689+0000","last_unstale":"2024-04-02T09:13:41.937842+0000","last_undegraded":"2024-04-02T09:13:41.937842+0000","last_fullsized":"2024-04-02T09:13:41.937842+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:08.953388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940440+0000","last_change":"2024-04-02T09:13:27.882492+0000","last_active":"2024-04-02T09:13:41.940440+0000","last_peered":"2024-04-02T09:13:41.940440+0000","last_clean":"2024-04-02T09:13:41.940440+0000","last_became_active":"2024-04-02T09:13:27.882217+0000","last_became_peered":"2024-04-02T09:13:27.882217+0000","last_unstale":"2024-04-02T09:13:41.940440+0000","last_undegraded":"2024-04-02T09:13:41.940440+0000","last_fullsized":"2024-04-02T09:13:41.940440+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:22:56.505431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734928+0000","last_change":"2024-04-02T09:13:41.958277+0000","last_active":"2024-04-02T09:13:43.734928+0000","last_peered":"2024-04-02T09:13:43.734928+0000","last_clean":"2024-04-02T09:13:43.734928+0000","last_became_active":"2024-04-02T09:13:41.956555+0000","last_became_peered":"2024-04-02T09:13:41.956555+0000","last_unstale":"2024-04-02T09:13:43.734928+0000","last_undegraded":"2024-04-02T09:13:43.734928+0000","last_fullsized":"2024-04-02T09:13:43.734928+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:46.027233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940350+0000","last_change":"2024-04-02T09:13:28.381355+0000","last_active":"2024-04-02T09:13:41.940350+0000","last_peered":"2024-04-02T09:13:41.940350+0000","last_clean":"2024-04-02T09:13:41.940350+0000","last_became_active":"2024-04-02T09:13:28.381068+0000","last_became_peered":"2024-04-02T09:13:28.381068+0000","last_unstale":"2024-04-02T09:13:41.940350+0000","last_undegraded":"2024-04-02T09:13:41.940350+0000","last_fullsized":"2024-04-02T09:13:41.940350+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:50.895638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953078+0000","last_change":"2024-04-02T09:13:27.883425+0000","last_active":"2024-04-02T09:13:41.953078+0000","last_peered":"2024-04-02T09:13:41.953078+0000","last_clean":"2024-04-02T09:13:41.953078+0000","last_became_active":"2024-04-02T09:13:27.883106+0000","last_became_peered":"2024-04-02T09:13:27.883106+0000","last_unstale":"2024-04-02T09:13:41.953078+0000","last_undegraded":"2024-04-02T09:13:41.953078+0000","last_fullsized":"2024-04-02T09:13:41.953078+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:23.271534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.733563+0000","last_change":"2024-04-02T09:13:41.960258+0000","last_active":"2024-04-02T09:13:43.733563+0000","last_peered":"2024-04-02T09:13:43.733563+0000","last_clean":"2024-04-02T09:13:43.733563+0000","last_became_active":"2024-04-02T09:13:41.960011+0000","last_became_peered":"2024-04-02T09:13:41.960011+0000","last_unstale":"2024-04-02T09:13:43.733563+0000","last_undegraded":"2024-04-02T09:13:43.733563+0000","last_fullsized":"2024-04-02T09:13:43.733563+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:18.838465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940567+0000","last_change":"2024-04-02T09:13:28.402819+0000","last_active":"2024-04-02T09:13:41.940567+0000","last_peered":"2024-04-02T09:13:41.940567+0000","last_clean":"2024-04-02T09:13:41.940567+0000","last_became_active":"2024-04-02T09:13:28.402563+0000","last_became_peered":"2024-04-02T09:13:28.402563+0000","last_unstale":"2024-04-02T09:13:41.940567+0000","last_undegraded":"2024-04-02T09:13:41.940567+0000","last_fullsized":"2024-04-02T09:13:41.940567+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:27.049697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1f","version":"243'4","reported_seq":19,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943081+0000","last_change":"2024-04-02T09:13:27.883417+0000","last_active":"2024-04-02T09:13:41.943081+0000","last_peered":"2024-04-02T09:13:41.943081+0000","last_clean":"2024-04-02T09:13:41.943081+0000","last_became_active":"2024-04-02T09:13:27.882944+0000","last_became_peered":"2024-04-02T09:13:27.882944+0000","last_unstale":"2024-04-02T09:13:41.943081+0000","last_undegraded":"2024-04-02T09:13:41.943081+0000","last_fullsized":"2024-04-02T09:13:41.943081+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:05.331734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"229'370","reported_seq":1312,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948041+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:13:41.948041+0000","last_peered":"2024-04-02T09:13:41.948041+0000","last_clean":"2024-04-02T09:13:41.948041+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:13:41.948041+0000","last_undegraded":"2024-04-02T09:13:41.948041+0000","last_fullsized":"2024-04-02T09:13:41.948041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3603,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576001,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60436,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311404,"statfs":{"total":96636764160,"available":96574877696,"internally_reserved":0,"allocated":3153920,"data_stored":1604364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.40400000000000003,"5min":0.309,"15min":0.309},"max":{"1min":8.016,"5min":8.016,"15min":8.016},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.71999999999999997,"15min":0.72999999999999998},"min":{"1min":0.39900000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.1790000000000003,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.70699999999999996}]},{"osd":1,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.76800000000000002,"15min":0.751},"min":{"1min":0.44800000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.5819999999999999,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.70899999999999996,"15min":0.71999999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.42399999999999999},"max":{"1min":4.2569999999999997,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.72099999999999997}]},{"osd":2,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.79600000000000004,"15min":0.77800000000000002},"min":{"1min":0.437,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.9790000000000001,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.80700000000000005,"15min":0.81200000000000006},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":4.8819999999999997,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.871}]},{"osd":3,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.77100000000000002,"15min":0.75900000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.0530000000000008,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.999,"5min":0.79900000000000004,"15min":0.77700000000000002},"min":{"1min":0.54100000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":5.093,"5min":5.093,"15min":5.093},"last":0.81999999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.66700000000000004,"15min":0.628},"min":{"1min":0.38300000000000001,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":5.4589999999999996,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.1060000000000001,"5min":0.79700000000000004,"15min":0.74399999999999999},"min":{"1min":0.50600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.9409999999999998,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.63200000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.73799999999999999,"15min":0.69299999999999995},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":5.1340000000000003,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.77400000000000002,"15min":0.73599999999999999},"min":{"1min":0.58899999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.492,"5min":3.492,"15min":3.492},"last":0.60699999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.72199999999999998,"15min":0.69699999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45800000000000002},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.075,"5min":0.78500000000000003,"15min":0.71699999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41099999999999998},"max":{"1min":7.8899999999999997,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575999,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42368,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329472,"statfs":{"total":96636764160,"available":96593379328,"internally_reserved":0,"allocated":3129344,"data_stored":1600437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.69799999999999995,"15min":0.72899999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.90600000000000003,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.71099999999999997,"15min":0.73899999999999999},"min":{"1min":0.53600000000000003,"5min":0.47899999999999998,"15min":0.45800000000000002},"max":{"1min":0.88300000000000001,"5min":0.93200000000000005,"15min":1.0600000000000001},"last":0.61399999999999999}]},{"osd":1,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68799999999999994,"15min":0.71299999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86299999999999999,"5min":1.044,"15min":1.212},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.71299999999999997,"15min":0.752},"min":{"1min":0.45800000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.018,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.51500000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.71799999999999997,"15min":0.747},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.046,"5min":1.046,"15min":1.2789999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.71999999999999997,"15min":0.73899999999999999},"min":{"1min":0.44,"5min":0.439,"15min":0.439},"max":{"1min":0.93500000000000005,"5min":0.95999999999999996,"15min":1.226},"last":0.77200000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72399999999999998,"15min":0.76300000000000001},"min":{"1min":0.53900000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97799999999999998,"5min":1.141,"15min":1.2549999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.755,"15min":0.77000000000000002},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.67400000000000004}]},{"osd":4,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.60999999999999999,"15min":0.624},"min":{"1min":0.442,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.91200000000000003,"5min":0.96699999999999997,"15min":1.1259999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.41999999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.1879999999999999},"last":0.75600000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84199999999999997,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69999999999999996,"15min":0.72499999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":0.96399999999999997,"5min":1.034,"15min":1.204},"last":0.74199999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.72999999999999998,"15min":0.74099999999999999},"min":{"1min":0.53600000000000003,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.012,"5min":1.21,"15min":1.2370000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72599999999999998,"15min":0.73899999999999999},"min":{"1min":0.52400000000000002,"5min":0.51400000000000001,"15min":0.49399999999999999},"max":{"1min":0.91900000000000004,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.81699999999999995}]}]},{"osd":1,"up_from":13,"seq":55834576000,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52812,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319028,"statfs":{"total":96636764160,"available":96582684672,"internally_reserved":0,"allocated":3727360,"data_stored":2191299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.433,"15min":0.45000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.58999999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48999999999999999,"15min":0.50900000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73199999999999998,"5min":0.76000000000000001,"15min":0.79800000000000004},"last":0.64700000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.504,"15min":0.505},"min":{"1min":0.38900000000000001,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56699999999999995,"15min":0.59199999999999997},"min":{"1min":0.44500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68100000000000005,"5min":0.77500000000000002,"15min":0.85599999999999998},"last":0.67400000000000004}]},{"osd":3,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.41099999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.69999999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.55200000000000005,"15min":0.56100000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.58899999999999997,"5min":0.86699999999999999,"15min":0.878},"last":0.59799999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76900000000000002,"15min":0.78400000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.433},"max":{"1min":0.90700000000000003,"5min":1.1120000000000001,"15min":1.1479999999999999},"last":1.1879999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.76400000000000001,"15min":0.78500000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.41799999999999998},"max":{"1min":0.95699999999999996,"5min":1.05,"15min":1.25},"last":0.873}]},{"osd":5,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76100000000000001,"15min":0.78800000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.96399999999999997,"5min":1.115,"15min":1.321},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.79700000000000004,"15min":0.80700000000000005},"min":{"1min":0.56899999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93799999999999994,"5min":1.0980000000000001,"15min":1.173},"last":1.0049999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.79100000000000004,"15min":0.81299999999999994},"min":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.48099999999999998},"max":{"1min":0.96099999999999997,"5min":1.159,"15min":1.2210000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80800000000000005,"15min":0.80900000000000005},"min":{"1min":0.56799999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0009999999999999,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.94099999999999995}]},{"osd":7,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.79900000000000004,"15min":0.81999999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0029999999999999,"5min":1.1080000000000001,"15min":1.236},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81899999999999995,"15min":0.83399999999999996},"min":{"1min":0.60899999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":0.97499999999999998,"5min":1.0720000000000001,"15min":1.2789999999999999},"last":1.1020000000000001}]}]},{"osd":0,"up_from":13,"seq":55834576000,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52220,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319620,"statfs":{"total":96636764160,"available":96583290880,"internally_reserved":0,"allocated":3129344,"data_stored":1599420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.441,"15min":0.45500000000000002},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.66700000000000004,"5min":0.69899999999999995,"15min":0.86699999999999999},"last":0.498},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.49399999999999999,"15min":0.50700000000000001},"min":{"1min":0.33900000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64800000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.53100000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.502,"15min":0.51100000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.82699999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53400000000000003,"15min":0.55500000000000005},"min":{"1min":0.38300000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72699999999999998,"5min":0.999,"15min":0.999},"last":0.63300000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54200000000000004,"15min":0.56699999999999995},"min":{"1min":0.35699999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.71899999999999997,"5min":0.79800000000000004,"15min":0.85699999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.61599999999999999,"15min":0.63800000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.73599999999999999,"5min":0.92000000000000004,"15min":0.96399999999999997},"last":0.65500000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.80500000000000005,"15min":0.83099999999999996},"min":{"1min":0.55000000000000004,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":0.97499999999999998,"5min":1.1950000000000001,"15min":1.2769999999999999},"last":0.745},{"interface":"front","average":{"1min":0.751,"5min":0.78500000000000003,"15min":0.82199999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99299999999999999,"5min":1.169,"15min":1.169},"last":0.873}]},{"osd":5,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.77900000000000003,"15min":0.82399999999999995},"min":{"1min":0.54000000000000004,"5min":0.497,"15min":0.497},"max":{"1min":1,"5min":1.145,"15min":1.232},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.96599999999999997,"5min":1.21,"15min":1.21},"last":0.90600000000000003}]},{"osd":6,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.77600000000000002,"15min":0.79800000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91100000000000003,"5min":1.1830000000000001,"15min":1.196},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.79200000000000004,"15min":0.81200000000000006},"min":{"1min":0.56499999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96199999999999997,"5min":1.246,"15min":1.246},"last":0.96799999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82699999999999996,"15min":0.85499999999999998},"min":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0069999999999999,"5min":1.292,"15min":1.292},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.84799999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.95499999999999996,"5min":1.1719999999999999,"15min":1.2589999999999999},"last":0.92300000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575999,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52236,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319604,"statfs":{"total":96636764160,"available":96583274496,"internally_reserved":0,"allocated":3141632,"data_stored":1601913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.44600000000000001,"15min":0.44},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.72999999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.48399999999999999,"15min":0.498},"min":{"1min":0.29199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":1.4930000000000001},"last":0.53600000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.504},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69699999999999995,"5min":0.75800000000000001,"15min":0.80600000000000005},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53600000000000003,"15min":0.55400000000000005},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.59599999999999997,"5min":0.75,"15min":1.4530000000000001},"last":0.58999999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52900000000000003,"15min":0.55700000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.745,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60499999999999998,"15min":0.63200000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.64000000000000001,"5min":0.83699999999999997,"15min":1.0740000000000001},"last":0.60799999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.82099999999999995,"5min":1.121,"15min":1.121},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.77300000000000002,"15min":0.79100000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.96799999999999997,"5min":1.179,"15min":1.5329999999999999},"last":0.93600000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.78400000000000003,"15min":0.78600000000000003},"min":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.82999999999999996,"5min":1.99,"15min":1.99},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.78500000000000003,"15min":0.79900000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.89600000000000002,"5min":1.101,"15min":1.1259999999999999},"last":0.88}]},{"osd":6,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.997,"5min":1.173,"15min":1.5940000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":0.80400000000000005},"min":{"1min":0.56200000000000006,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.92300000000000004,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.83499999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.877,"5min":1.2110000000000001,"15min":1.5649999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.80700000000000005,"15min":0.82699999999999996},"min":{"1min":0.61499999999999999,"5min":0.497,"15min":0.497},"max":{"1min":0.95899999999999996,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.90600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575999,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52232,"kb_used_data":3064,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319608,"statfs":{"total":96636764160,"available":96583278592,"internally_reserved":0,"allocated":3137536,"data_stored":1603096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.42799999999999999,"15min":0.44600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57199999999999995,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.46899999999999997,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60199999999999998,"5min":0.71199999999999997,"15min":0.876},"last":0.34699999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48699999999999999,"15min":0.499},"min":{"1min":0.36799999999999999,"5min":0.33200000000000002,"15min":0.32500000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.90300000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55100000000000005,"15min":0.57199999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67700000000000005,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.56399999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.60099999999999998,"15min":0.626},"min":{"1min":0.40100000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71399999999999997,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.58799999999999997,"15min":0.61599999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.69499999999999995,"5min":0.81699999999999995,"15min":1.0840000000000001},"last":0.61399999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.72299999999999998,"15min":0.752},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89100000000000001,"5min":1.071,"15min":1.0800000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.74299999999999999,"15min":0.76800000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.879,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.57399999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.74399999999999999,"15min":0.77500000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91800000000000004,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.76300000000000001,"15min":0.78100000000000003},"min":{"1min":0.54200000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.85699999999999998,"5min":1.266,"15min":1.266},"last":0.51100000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.76500000000000001,"15min":0.78400000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97099999999999997,"5min":1.284,"15min":1.284},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.54800000000000004,"5min":0.495,"15min":0.495},"max":{"1min":0.95999999999999996,"5min":1.2,"15min":1.2},"last":0.64700000000000002}]},{"osd":7,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.77900000000000003,"15min":0.80800000000000005},"min":{"1min":0.58699999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.93500000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.79300000000000004,"15min":0.81799999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.014,"5min":1.248,"15min":1.248},"last":0.70099999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575999,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48136,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323704,"statfs":{"total":96636764160,"available":96587472896,"internally_reserved":0,"allocated":3141632,"data_stored":1602425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.66400000000000003,"15min":0.68700000000000006},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":0.80000000000000004,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.65100000000000002,"15min":0.67700000000000005},"min":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.78100000000000003,"5min":0.86199999999999999,"15min":1.075},"last":0.58499999999999996}]},{"osd":1,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.72199999999999998,"15min":0.754},"min":{"1min":0.49399999999999999,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":0.90500000000000003,"5min":1.0549999999999999,"15min":1.069},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.73599999999999999,"15min":0.76500000000000001},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":0.92500000000000004,"5min":1.048,"15min":1.048},"last":0.72699999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.67300000000000004,"15min":0.70799999999999996},"min":{"1min":0.47099999999999997,"5min":0.438,"15min":0.438},"max":{"1min":0.81299999999999994,"5min":1.077,"15min":1.077},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.68999999999999995,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85599999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.70499999999999996}]},{"osd":3,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.51500000000000001,"5min":0.501,"15min":0.47399999999999998},"max":{"1min":0.94599999999999995,"5min":0.998,"15min":1.018},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.028},"last":0.55100000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.81899999999999995,"5min":1.008,"15min":1.008},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.497}]},{"osd":6,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65500000000000003,"15min":0.64400000000000002},"min":{"1min":0.48499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85099999999999998,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68999999999999995,"15min":0.69199999999999995},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":1.0049999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.48399999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96099999999999997,"5min":1.002,"15min":1.0329999999999999},"last":0.627},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.77500000000000002,"15min":0.78100000000000003},"min":{"1min":0.56100000000000005,"5min":0.5,"15min":0.5},"max":{"1min":1.038,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.73799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576002,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52796,"kb_used_data":3632,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319044,"statfs":{"total":96636764160,"available":96582701056,"internally_reserved":0,"allocated":3719168,"data_stored":2189730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.82699999999999996,"5min":0.98699999999999999,"15min":1.004},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.68999999999999995,"15min":0.71999999999999997},"min":{"1min":0.48399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.028,"5min":1.085,"15min":1.085},"last":0.67300000000000004}]},{"osd":1,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.68100000000000005,"15min":0.71499999999999997},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84399999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70999999999999996,"15min":0.72499999999999998},"min":{"1min":0.52300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88500000000000001,"5min":0.95299999999999996,"15min":1.052},"last":0.81599999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.71599999999999997,"15min":0.73799999999999999},"min":{"1min":0.47199999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97299999999999998,"5min":1.004,"15min":1.014},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.69499999999999995,"15min":0.73699999999999999},"min":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.81799999999999995,"5min":1.018,"15min":1.0680000000000001},"last":0.66100000000000003}]},{"osd":3,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.747,"15min":0.77700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.88500000000000001,"5min":0.98899999999999999,"15min":1.083},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77000000000000002,"15min":0.79000000000000004},"min":{"1min":0.58399999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1020000000000001},"last":0.71199999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.40699999999999997,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.054},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":0.98799999999999999,"5min":1.145,"15min":1.145},"last":0.84499999999999997}]},{"osd":6,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67900000000000005,"15min":0.67300000000000004},"min":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96799999999999997,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71399999999999997,"15min":0.71099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.94899999999999995,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.91900000000000004}]},{"osd":7,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73999999999999999,"15min":0.73799999999999999},"min":{"1min":0.46500000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.0820000000000001,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76300000000000001,"15min":0.76600000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1519999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.76300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:13:51.788 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:13:51.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.992+0000 7ffb01e00700 1 Processor -- start 2024-04-02T09:13:51.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.992+0000 7ffb01e00700 1 -- start start 2024-04-02T09:13:51.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.993+0000 7ffb01e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffafc0d0a40 0x7ffafc0b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:51.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.993+0000 7ffb01e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffafc0d0ef0 con 0x7ffafc15da50 2024-04-02T09:13:51.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.993+0000 7ffb01e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffafc0d24e0 con 0x7ffafc0d05e0 2024-04-02T09:13:51.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.993+0000 7ffb01e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffafc0b95d0 con 0x7ffafc0d0a40 2024-04-02T09:13:51.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.993+0000 7ffafb400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffafc0d0a40 0x7ffafc0b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:51.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.993+0000 7ffafb400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffafc0d0a40 0x7ffafc0b70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59208/0 (socket says 172.21.15.89:59208) 2024-04-02T09:13:51.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.993+0000 7ffafb400700 1 -- 172.21.15.89:0/3816718118 learned_addr learned my addr 172.21.15.89:0/3816718118 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:51.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.994+0000 7ffafb400700 1 -- 172.21.15.89:0/3816718118 >> v1:172.21.15.113:6789/0 conn(0x7ffafc0d05e0 legacy=0x7ffafc0b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:51.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.994+0000 7ffafb400700 1 -- 172.21.15.89:0/3816718118 >> v1:172.21.15.89:6789/0 conn(0x7ffafc15da50 legacy=0x7ffafc0ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:51.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.994+0000 7ffafb400700 1 -- 172.21.15.89:0/3816718118 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffafc0d27c0 con 0x7ffafc0d0a40 2024-04-02T09:13:51.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.994+0000 7ffaf9600700 1 -- 172.21.15.89:0/3816718118 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039227104 0 0) 0x7ffafc0d0ef0 con 0x7ffafc15da50 2024-04-02T09:13:51.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.994+0000 7ffafb400700 1 --2- 172.21.15.89:0/3816718118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffafc0d0a40 0x7ffafc0b70a0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4009e50 tx=0x7ffaf400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8459e21ac0b13e76 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:51.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.994+0000 7ffaf9600700 1 -- 172.21.15.89:0/3816718118 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf400e070 con 0x7ffafc0d0a40 2024-04-02T09:13:51.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.994+0000 7ffaf9600700 1 -- 172.21.15.89:0/3816718118 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf4003a20 con 0x7ffafc0d0a40 2024-04-02T09:13:51.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.994+0000 7ffaf9600700 1 -- 172.21.15.89:0/3816718118 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4003dd0 con 0x7ffafc0d0a40 2024-04-02T09:13:51.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.995+0000 7ffb01e00700 1 -- 172.21.15.89:0/3816718118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffafc0d0a40 msgr2=0x7ffafc0b70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:51.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.995+0000 7ffb01e00700 1 --2- 172.21.15.89:0/3816718118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffafc0d0a40 0x7ffafc0b70a0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4009e50 tx=0x7ffaf400b330 comp rx=0 tx=0).stop 2024-04-02T09:13:51.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.995+0000 7ffb01e00700 1 -- 172.21.15.89:0/3816718118 shutdown_connections 2024-04-02T09:13:51.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.995+0000 7ffb01e00700 1 --2- 172.21.15.89:0/3816718118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffafc0d0a40 0x7ffafc0b70a0 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:51.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.995+0000 7ffb01e00700 1 -- 172.21.15.89:0/3816718118 >> 172.21.15.89:0/3816718118 conn(0x7ffafc0666a0 msgr2=0x7ffafc066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:51.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.995+0000 7ffb01e00700 1 -- 172.21.15.89:0/3816718118 shutdown_connections 2024-04-02T09:13:51.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.995+0000 7ffb01e00700 1 -- 172.21.15.89:0/3816718118 wait complete. 2024-04-02T09:13:51.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.997+0000 7ffb01e00700 1 Processor -- start 2024-04-02T09:13:51.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.997+0000 7ffb01e00700 1 -- start start 2024-04-02T09:13:51.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffb01e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffafc0d05e0 0x7ffafc0c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:51.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffb01e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffafc0d0a40 0x7ffafc160400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:51.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffb01e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffafc15da50 0x7ffafc0d1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:51.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffb01e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc1f2c30 con 0x7ffafc15da50 2024-04-02T09:13:51.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffb01e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc160100 con 0x7ffafc0d0a40 2024-04-02T09:13:51.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffb01e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc15f060 con 0x7ffafc0d05e0 2024-04-02T09:13:51.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffafb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffafc15da50 0x7ffafc0d1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:51.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffafb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffafc15da50 0x7ffafc0d1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59224/0 (socket says 172.21.15.89:59224) 2024-04-02T09:13:51.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffafb400700 1 -- 172.21.15.89:0/394187588 learned_addr learned my addr 172.21.15.89:0/394187588 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:51.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffafb400700 1 -- 172.21.15.89:0/394187588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffafc0d05e0 msgr2=0x7ffafc0c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:51.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffafaa00700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffafc0d05e0 0x7ffafc0c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:51.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffafa000700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffafc0d0a40 0x7ffafc160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:51.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffafb400700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffafc0d05e0 0x7ffafc0c9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:51.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffafb400700 1 -- 172.21.15.89:0/394187588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffafc0d0a40 msgr2=0x7ffafc160400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:51.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffafb400700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffafc0d0a40 0x7ffafc160400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:51.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffafb400700 1 -- 172.21.15.89:0/394187588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf4009b30 con 0x7ffafc15da50 2024-04-02T09:13:51.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffafa000700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffafc0d0a40 0x7ffafc160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:51.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.998+0000 7ffafaa00700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffafc0d05e0 0x7ffafc0c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:52.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.999+0000 7ffafb400700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffafc15da50 0x7ffafc0d1690 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7ffafc0741d0 tx=0x7ffaf4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:52.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.999+0000 7ffaf3400700 1 -- 172.21.15.89:0/394187588 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4004440 con 0x7ffafc15da50 2024-04-02T09:13:52.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.999+0000 7ffaf3400700 1 -- 172.21.15.89:0/394187588 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf40045a0 con 0x7ffafc15da50 2024-04-02T09:13:52.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.999+0000 7ffaf3400700 1 -- 172.21.15.89:0/394187588 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4004950 con 0x7ffafc15da50 2024-04-02T09:13:52.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.999+0000 7ffb01e00700 1 -- 172.21.15.89:0/394187588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffafc15f340 con 0x7ffafc15da50 2024-04-02T09:13:52.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:51.999+0000 7ffb01e00700 1 -- 172.21.15.89:0/394187588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffafc15f900 con 0x7ffafc15da50 2024-04-02T09:13:52.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.000+0000 7ffb01e00700 1 -- 172.21.15.89:0/394187588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffafc067720 con 0x7ffafc15da50 2024-04-02T09:13:52.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.001+0000 7ffaf3400700 1 -- 172.21.15.89:0/394187588 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7ffaf4026450 con 0x7ffafc15da50 2024-04-02T09:13:52.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.002+0000 7ffaf3400700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffadc06aed0 0x7ffadc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:52.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.002+0000 7ffaf3400700 1 -- 172.21.15.89:0/394187588 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7ffaf408ba50 con 0x7ffafc15da50 2024-04-02T09:13:52.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.002+0000 7ffafaa00700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffadc06aed0 0x7ffadc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:52.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.003+0000 7ffafaa00700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffadc06aed0 0x7ffadc06d390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffaec000f60 tx=0x7ffaec009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:52.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.007+0000 7ffaf3400700 1 -- 172.21.15.89:0/394187588 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffaf4056450 con 0x7ffafc15da50 2024-04-02T09:13:52.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.204+0000 7ffb01e00700 1 -- 172.21.15.89:0/394187588 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffafc0d3060 con 0x7ffadc06aed0 2024-04-02T09:13:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.218+0000 7ffaf3400700 1 -- 172.21.15.89:0/394187588 <== mgr.11613 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528479 (secure 0 0 0) 0x7ffaec0050d0 con 0x7ffadc06aed0 2024-04-02T09:13:52.229 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:52.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.224+0000 7ffb01e00700 1 -- 172.21.15.89:0/394187588 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffadc06aed0 msgr2=0x7ffadc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:52.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.224+0000 7ffb01e00700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffadc06aed0 0x7ffadc06d390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffaec000f60 tx=0x7ffaec009040 comp rx=0 tx=0).stop 2024-04-02T09:13:52.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.224+0000 7ffb01e00700 1 -- 172.21.15.89:0/394187588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffafc15da50 msgr2=0x7ffafc0d1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:52.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.224+0000 7ffb01e00700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffafc15da50 0x7ffafc0d1690 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7ffafc0741d0 tx=0x7ffaf4004ad0 comp rx=0 tx=0).stop 2024-04-02T09:13:52.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.224+0000 7ffb01e00700 1 -- 172.21.15.89:0/394187588 shutdown_connections 2024-04-02T09:13:52.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.224+0000 7ffb01e00700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffafc15da50 0x7ffafc0d1690 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:52.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.224+0000 7ffb01e00700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffafc0d05e0 0x7ffafc0c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:52.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.224+0000 7ffb01e00700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffadc06aed0 0x7ffadc06d390 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:52.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.224+0000 7ffb01e00700 1 --2- 172.21.15.89:0/394187588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffafc0d0a40 0x7ffafc160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:52.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.224+0000 7ffb01e00700 1 -- 172.21.15.89:0/394187588 >> 172.21.15.89:0/394187588 conn(0x7ffafc0666a0 msgr2=0x7ffafc0b52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:52.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.224+0000 7ffb01e00700 1 -- 172.21.15.89:0/394187588 shutdown_connections 2024-04-02T09:13:52.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.224+0000 7ffb01e00700 1 -- 172.21.15.89:0/394187588 wait complete. 2024-04-02T09:13:52.238 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:13:52.242 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-04-02T09:13:51.433884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596333,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":711,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":450,"ondisk_log_size":450,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413236,"kb_used_data":25664,"kb_used_omap":12,"kb_used_meta":387443,"kb_avail":754561484,"statfs":{"total":773094113280,"available":772670959616,"internally_reserved":0,"allocated":26279936,"data_stored":13992684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":396742200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.497737"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.947418+0000","last_change":"2024-04-02T09:13:41.944375+0000","last_active":"2024-04-02T09:13:42.947418+0000","last_peered":"2024-04-02T09:13:42.947418+0000","last_clean":"2024-04-02T09:13:42.947418+0000","last_became_active":"2024-04-02T09:13:41.944110+0000","last_became_peered":"2024-04-02T09:13:41.944110+0000","last_unstale":"2024-04-02T09:13:42.947418+0000","last_undegraded":"2024-04-02T09:13:42.947418+0000","last_fullsized":"2024-04-02T09:13:42.947418+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:02:20.460847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942652+0000","last_change":"2024-04-02T09:13:41.458005+0000","last_active":"2024-04-02T09:13:41.942652+0000","last_peered":"2024-04-02T09:13:41.942652+0000","last_clean":"2024-04-02T09:13:41.942652+0000","last_became_active":"2024-04-02T09:13:41.457753+0000","last_became_peered":"2024-04-02T09:13:41.457753+0000","last_unstale":"2024-04-02T09:13:41.942652+0000","last_undegraded":"2024-04-02T09:13:41.942652+0000","last_fullsized":"2024-04-02T09:13:41.942652+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:12:22.769587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942756+0000","last_change":"2024-04-02T09:13:28.396642+0000","last_active":"2024-04-02T09:13:41.942756+0000","last_peered":"2024-04-02T09:13:41.942756+0000","last_clean":"2024-04-02T09:13:41.942756+0000","last_became_active":"2024-04-02T09:13:28.396356+0000","last_became_peered":"2024-04-02T09:13:28.396356+0000","last_unstale":"2024-04-02T09:13:41.942756+0000","last_undegraded":"2024-04-02T09:13:41.942756+0000","last_fullsized":"2024-04-02T09:13:41.942756+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:34.295480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345466+0000","last_change":"2024-04-02T09:13:41.961471+0000","last_active":"2024-04-02T09:13:43.345466+0000","last_peered":"2024-04-02T09:13:43.345466+0000","last_clean":"2024-04-02T09:13:43.345466+0000","last_became_active":"2024-04-02T09:13:41.961181+0000","last_became_peered":"2024-04-02T09:13:41.961181+0000","last_unstale":"2024-04-02T09:13:43.345466+0000","last_undegraded":"2024-04-02T09:13:43.345466+0000","last_fullsized":"2024-04-02T09:13:43.345466+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:13.220603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952545+0000","last_change":"2024-04-02T09:13:41.454077+0000","last_active":"2024-04-02T09:13:41.952545+0000","last_peered":"2024-04-02T09:13:41.952545+0000","last_clean":"2024-04-02T09:13:41.952545+0000","last_became_active":"2024-04-02T09:13:41.453804+0000","last_became_peered":"2024-04-02T09:13:41.453804+0000","last_unstale":"2024-04-02T09:13:41.952545+0000","last_undegraded":"2024-04-02T09:13:41.952545+0000","last_fullsized":"2024-04-02T09:13:41.952545+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:24:28.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.961240+0000","last_change":"2024-04-02T09:13:28.381100+0000","last_active":"2024-04-02T09:13:41.961240+0000","last_peered":"2024-04-02T09:13:41.961240+0000","last_clean":"2024-04-02T09:13:41.961240+0000","last_became_active":"2024-04-02T09:13:28.380832+0000","last_became_peered":"2024-04-02T09:13:28.380832+0000","last_unstale":"2024-04-02T09:13:41.961240+0000","last_undegraded":"2024-04-02T09:13:41.961240+0000","last_fullsized":"2024-04-02T09:13:41.961240+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:21.987215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734898+0000","last_change":"2024-04-02T09:13:41.970419+0000","last_active":"2024-04-02T09:13:43.734898+0000","last_peered":"2024-04-02T09:13:43.734898+0000","last_clean":"2024-04-02T09:13:43.734898+0000","last_became_active":"2024-04-02T09:13:41.965663+0000","last_became_peered":"2024-04-02T09:13:41.965663+0000","last_unstale":"2024-04-02T09:13:43.734898+0000","last_undegraded":"2024-04-02T09:13:43.734898+0000","last_fullsized":"2024-04-02T09:13:43.734898+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:28:51.025221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942697+0000","last_change":"2024-04-02T09:13:41.457188+0000","last_active":"2024-04-02T09:13:41.942697+0000","last_peered":"2024-04-02T09:13:41.942697+0000","last_clean":"2024-04-02T09:13:41.942697+0000","last_became_active":"2024-04-02T09:13:41.456916+0000","last_became_peered":"2024-04-02T09:13:41.456916+0000","last_unstale":"2024-04-02T09:13:41.942697+0000","last_undegraded":"2024-04-02T09:13:41.942697+0000","last_fullsized":"2024-04-02T09:13:41.942697+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:31:39.320935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.941485+0000","last_change":"2024-04-02T09:13:28.393942+0000","last_active":"2024-04-02T09:13:41.941485+0000","last_peered":"2024-04-02T09:13:41.941485+0000","last_clean":"2024-04-02T09:13:41.941485+0000","last_became_active":"2024-04-02T09:13:28.393467+0000","last_became_peered":"2024-04-02T09:13:28.393467+0000","last_unstale":"2024-04-02T09:13:41.941485+0000","last_undegraded":"2024-04-02T09:13:41.941485+0000","last_fullsized":"2024-04-02T09:13:41.941485+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:24.836500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952847+0000","last_change":"2024-04-02T09:13:41.961739+0000","last_active":"2024-04-02T09:13:42.952847+0000","last_peered":"2024-04-02T09:13:42.952847+0000","last_clean":"2024-04-02T09:13:42.952847+0000","last_became_active":"2024-04-02T09:13:41.961468+0000","last_became_peered":"2024-04-02T09:13:41.961468+0000","last_unstale":"2024-04-02T09:13:42.952847+0000","last_undegraded":"2024-04-02T09:13:42.952847+0000","last_fullsized":"2024-04-02T09:13:42.952847+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:33:35.636916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940817+0000","last_change":"2024-04-02T09:13:41.452325+0000","last_active":"2024-04-02T09:13:41.940817+0000","last_peered":"2024-04-02T09:13:41.940817+0000","last_clean":"2024-04-02T09:13:41.940817+0000","last_became_active":"2024-04-02T09:13:41.449714+0000","last_became_peered":"2024-04-02T09:13:41.449714+0000","last_unstale":"2024-04-02T09:13:41.940817+0000","last_undegraded":"2024-04-02T09:13:41.940817+0000","last_fullsized":"2024-04-02T09:13:41.940817+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:13:02.182890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945254+0000","last_change":"2024-04-02T09:13:28.394098+0000","last_active":"2024-04-02T09:13:41.945254+0000","last_peered":"2024-04-02T09:13:41.945254+0000","last_clean":"2024-04-02T09:13:41.945254+0000","last_became_active":"2024-04-02T09:13:28.393837+0000","last_became_peered":"2024-04-02T09:13:28.393837+0000","last_unstale":"2024-04-02T09:13:41.945254+0000","last_undegraded":"2024-04-02T09:13:41.945254+0000","last_fullsized":"2024-04-02T09:13:41.945254+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:40:27.068896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.948026+0000","last_change":"2024-04-02T09:13:41.951117+0000","last_active":"2024-04-02T09:13:42.948026+0000","last_peered":"2024-04-02T09:13:42.948026+0000","last_clean":"2024-04-02T09:13:42.948026+0000","last_became_active":"2024-04-02T09:13:41.948760+0000","last_became_peered":"2024-04-02T09:13:41.948760+0000","last_unstale":"2024-04-02T09:13:42.948026+0000","last_undegraded":"2024-04-02T09:13:42.948026+0000","last_fullsized":"2024-04-02T09:13:42.948026+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:53:09.361699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944874+0000","last_change":"2024-04-02T09:13:41.458809+0000","last_active":"2024-04-02T09:13:41.944874+0000","last_peered":"2024-04-02T09:13:41.944874+0000","last_clean":"2024-04-02T09:13:41.944874+0000","last_became_active":"2024-04-02T09:13:41.458392+0000","last_became_peered":"2024-04-02T09:13:41.458392+0000","last_unstale":"2024-04-02T09:13:41.944874+0000","last_undegraded":"2024-04-02T09:13:41.944874+0000","last_fullsized":"2024-04-02T09:13:41.944874+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:53.327540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947921+0000","last_change":"2024-04-02T09:13:28.379239+0000","last_active":"2024-04-02T09:13:41.947921+0000","last_peered":"2024-04-02T09:13:41.947921+0000","last_clean":"2024-04-02T09:13:41.947921+0000","last_became_active":"2024-04-02T09:13:28.378970+0000","last_became_peered":"2024-04-02T09:13:28.378970+0000","last_unstale":"2024-04-02T09:13:41.947921+0000","last_undegraded":"2024-04-02T09:13:41.947921+0000","last_fullsized":"2024-04-02T09:13:41.947921+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:12:37.960638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342554+0000","last_change":"2024-04-02T09:13:41.959102+0000","last_active":"2024-04-02T09:13:43.342554+0000","last_peered":"2024-04-02T09:13:43.342554+0000","last_clean":"2024-04-02T09:13:43.342554+0000","last_became_active":"2024-04-02T09:13:41.956524+0000","last_became_peered":"2024-04-02T09:13:41.956524+0000","last_unstale":"2024-04-02T09:13:43.342554+0000","last_undegraded":"2024-04-02T09:13:43.342554+0000","last_fullsized":"2024-04-02T09:13:43.342554+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:47.857858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946223+0000","last_change":"2024-04-02T09:13:41.451612+0000","last_active":"2024-04-02T09:13:41.946223+0000","last_peered":"2024-04-02T09:13:41.946223+0000","last_clean":"2024-04-02T09:13:41.946223+0000","last_became_active":"2024-04-02T09:13:41.451089+0000","last_became_peered":"2024-04-02T09:13:41.451089+0000","last_unstale":"2024-04-02T09:13:41.946223+0000","last_undegraded":"2024-04-02T09:13:41.946223+0000","last_fullsized":"2024-04-02T09:13:41.946223+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:18.892470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.939756+0000","last_change":"2024-04-02T09:13:28.393747+0000","last_active":"2024-04-02T09:13:41.939756+0000","last_peered":"2024-04-02T09:13:41.939756+0000","last_clean":"2024-04-02T09:13:41.939756+0000","last_became_active":"2024-04-02T09:13:28.393515+0000","last_became_peered":"2024-04-02T09:13:28.393515+0000","last_unstale":"2024-04-02T09:13:41.939756+0000","last_undegraded":"2024-04-02T09:13:41.939756+0000","last_fullsized":"2024-04-02T09:13:41.939756+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:28:37.462844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954473+0000","last_change":"2024-04-02T09:13:41.948810+0000","last_active":"2024-04-02T09:13:42.954473+0000","last_peered":"2024-04-02T09:13:42.954473+0000","last_clean":"2024-04-02T09:13:42.954473+0000","last_became_active":"2024-04-02T09:13:41.948523+0000","last_became_peered":"2024-04-02T09:13:41.948523+0000","last_unstale":"2024-04-02T09:13:42.954473+0000","last_undegraded":"2024-04-02T09:13:42.954473+0000","last_fullsized":"2024-04-02T09:13:42.954473+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:54.336827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.6","version":"248'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342742+0000","last_change":"2024-04-02T09:13:41.460002+0000","last_active":"2024-04-02T09:13:43.342742+0000","last_peered":"2024-04-02T09:13:43.342742+0000","last_clean":"2024-04-02T09:13:43.342742+0000","last_became_active":"2024-04-02T09:13:41.459491+0000","last_became_peered":"2024-04-02T09:13:41.459491+0000","last_unstale":"2024-04-02T09:13:43.342742+0000","last_undegraded":"2024-04-02T09:13:43.342742+0000","last_fullsized":"2024-04-02T09:13:43.342742+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:53.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.938200+0000","last_change":"2024-04-02T09:13:28.399975+0000","last_active":"2024-04-02T09:13:41.938200+0000","last_peered":"2024-04-02T09:13:41.938200+0000","last_clean":"2024-04-02T09:13:41.938200+0000","last_became_active":"2024-04-02T09:13:28.399335+0000","last_became_peered":"2024-04-02T09:13:28.399335+0000","last_unstale":"2024-04-02T09:13:41.938200+0000","last_undegraded":"2024-04-02T09:13:41.938200+0000","last_fullsized":"2024-04-02T09:13:41.938200+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:16:14.265813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954211+0000","last_change":"2024-04-02T09:13:41.954505+0000","last_active":"2024-04-02T09:13:42.954211+0000","last_peered":"2024-04-02T09:13:42.954211+0000","last_clean":"2024-04-02T09:13:42.954211+0000","last_became_active":"2024-04-02T09:13:41.952682+0000","last_became_peered":"2024-04-02T09:13:41.952682+0000","last_unstale":"2024-04-02T09:13:42.954211+0000","last_undegraded":"2024-04-02T09:13:42.954211+0000","last_fullsized":"2024-04-02T09:13:42.954211+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:32:52.343662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.7","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.971184+0000","last_change":"2024-04-02T09:13:41.459550+0000","last_active":"2024-04-02T09:13:41.971184+0000","last_peered":"2024-04-02T09:13:41.971184+0000","last_clean":"2024-04-02T09:13:41.971184+0000","last_became_active":"2024-04-02T09:13:41.459249+0000","last_became_peered":"2024-04-02T09:13:41.459249+0000","last_unstale":"2024-04-02T09:13:41.971184+0000","last_undegraded":"2024-04-02T09:13:41.971184+0000","last_fullsized":"2024-04-02T09:13:41.971184+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:40.922210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947893+0000","last_change":"2024-04-02T09:13:28.392551+0000","last_active":"2024-04-02T09:13:41.947893+0000","last_peered":"2024-04-02T09:13:41.947893+0000","last_clean":"2024-04-02T09:13:41.947893+0000","last_became_active":"2024-04-02T09:13:28.392159+0000","last_became_peered":"2024-04-02T09:13:28.392159+0000","last_unstale":"2024-04-02T09:13:41.947893+0000","last_undegraded":"2024-04-02T09:13:41.947893+0000","last_fullsized":"2024-04-02T09:13:41.947893+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:42.551696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952740+0000","last_change":"2024-04-02T09:13:41.951315+0000","last_active":"2024-04-02T09:13:42.952740+0000","last_peered":"2024-04-02T09:13:42.952740+0000","last_clean":"2024-04-02T09:13:42.952740+0000","last_became_active":"2024-04-02T09:13:41.950989+0000","last_became_peered":"2024-04-02T09:13:41.950989+0000","last_unstale":"2024-04-02T09:13:42.952740+0000","last_undegraded":"2024-04-02T09:13:42.952740+0000","last_fullsized":"2024-04-02T09:13:42.952740+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:09.529216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944222+0000","last_change":"2024-04-02T09:13:41.451519+0000","last_active":"2024-04-02T09:13:41.944222+0000","last_peered":"2024-04-02T09:13:41.944222+0000","last_clean":"2024-04-02T09:13:41.944222+0000","last_became_active":"2024-04-02T09:13:41.450207+0000","last_became_peered":"2024-04-02T09:13:41.450207+0000","last_unstale":"2024-04-02T09:13:41.944222+0000","last_undegraded":"2024-04-02T09:13:41.944222+0000","last_fullsized":"2024-04-02T09:13:41.944222+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:08.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950200+0000","last_change":"2024-04-02T09:13:28.383891+0000","last_active":"2024-04-02T09:13:41.950200+0000","last_peered":"2024-04-02T09:13:41.950200+0000","last_clean":"2024-04-02T09:13:41.950200+0000","last_became_active":"2024-04-02T09:13:28.383628+0000","last_became_peered":"2024-04-02T09:13:28.383628+0000","last_unstale":"2024-04-02T09:13:41.950200+0000","last_undegraded":"2024-04-02T09:13:41.950200+0000","last_fullsized":"2024-04-02T09:13:41.950200+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:44.222993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734669+0000","last_change":"2024-04-02T09:13:41.969003+0000","last_active":"2024-04-02T09:13:43.734669+0000","last_peered":"2024-04-02T09:13:43.734669+0000","last_clean":"2024-04-02T09:13:43.734669+0000","last_became_active":"2024-04-02T09:13:41.968108+0000","last_became_peered":"2024-04-02T09:13:41.968108+0000","last_unstale":"2024-04-02T09:13:43.734669+0000","last_undegraded":"2024-04-02T09:13:43.734669+0000","last_fullsized":"2024-04-02T09:13:43.734669+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:11.075454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945467+0000","last_change":"2024-04-02T09:13:41.450917+0000","last_active":"2024-04-02T09:13:41.945467+0000","last_peered":"2024-04-02T09:13:41.945467+0000","last_clean":"2024-04-02T09:13:41.945467+0000","last_became_active":"2024-04-02T09:13:41.450645+0000","last_became_peered":"2024-04-02T09:13:41.450645+0000","last_unstale":"2024-04-02T09:13:41.945467+0000","last_undegraded":"2024-04-02T09:13:41.945467+0000","last_fullsized":"2024-04-02T09:13:41.945467+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:10.066167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940531+0000","last_change":"2024-04-02T09:13:28.404390+0000","last_active":"2024-04-02T09:13:41.940531+0000","last_peered":"2024-04-02T09:13:41.940531+0000","last_clean":"2024-04-02T09:13:41.940531+0000","last_became_active":"2024-04-02T09:13:28.404038+0000","last_became_peered":"2024-04-02T09:13:28.404038+0000","last_unstale":"2024-04-02T09:13:41.940531+0000","last_undegraded":"2024-04-02T09:13:41.940531+0000","last_fullsized":"2024-04-02T09:13:41.940531+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:47.359176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.735085+0000","last_change":"2024-04-02T09:13:41.968705+0000","last_active":"2024-04-02T09:13:43.735085+0000","last_peered":"2024-04-02T09:13:43.735085+0000","last_clean":"2024-04-02T09:13:43.735085+0000","last_became_active":"2024-04-02T09:13:41.968079+0000","last_became_peered":"2024-04-02T09:13:41.968079+0000","last_unstale":"2024-04-02T09:13:43.735085+0000","last_undegraded":"2024-04-02T09:13:43.735085+0000","last_fullsized":"2024-04-02T09:13:43.735085+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:27:04.458016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944274+0000","last_change":"2024-04-02T09:13:41.451223+0000","last_active":"2024-04-02T09:13:41.944274+0000","last_peered":"2024-04-02T09:13:41.944274+0000","last_clean":"2024-04-02T09:13:41.944274+0000","last_became_active":"2024-04-02T09:13:41.449328+0000","last_became_peered":"2024-04-02T09:13:41.449328+0000","last_unstale":"2024-04-02T09:13:41.944274+0000","last_undegraded":"2024-04-02T09:13:41.944274+0000","last_fullsized":"2024-04-02T09:13:41.944274+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:50:01.664013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946964+0000","last_change":"2024-04-02T09:13:28.403517+0000","last_active":"2024-04-02T09:13:41.946964+0000","last_peered":"2024-04-02T09:13:41.946964+0000","last_clean":"2024-04-02T09:13:41.946964+0000","last_became_active":"2024-04-02T09:13:28.401353+0000","last_became_peered":"2024-04-02T09:13:28.401353+0000","last_unstale":"2024-04-02T09:13:41.946964+0000","last_undegraded":"2024-04-02T09:13:41.946964+0000","last_fullsized":"2024-04-02T09:13:41.946964+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:08.980542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345809+0000","last_change":"2024-04-02T09:13:41.957583+0000","last_active":"2024-04-02T09:13:43.345809+0000","last_peered":"2024-04-02T09:13:43.345809+0000","last_clean":"2024-04-02T09:13:43.345809+0000","last_became_active":"2024-04-02T09:13:41.953951+0000","last_became_peered":"2024-04-02T09:13:41.953951+0000","last_unstale":"2024-04-02T09:13:43.345809+0000","last_undegraded":"2024-04-02T09:13:43.345809+0000","last_fullsized":"2024-04-02T09:13:43.345809+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:14:22.720024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.b","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.969813+0000","last_change":"2024-04-02T09:13:41.446786+0000","last_active":"2024-04-02T09:13:41.969813+0000","last_peered":"2024-04-02T09:13:41.969813+0000","last_clean":"2024-04-02T09:13:41.969813+0000","last_became_active":"2024-04-02T09:13:41.446482+0000","last_became_peered":"2024-04-02T09:13:41.446482+0000","last_unstale":"2024-04-02T09:13:41.969813+0000","last_undegraded":"2024-04-02T09:13:41.969813+0000","last_fullsized":"2024-04-02T09:13:41.969813+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:04:08.519929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.938581+0000","last_change":"2024-04-02T09:13:28.399938+0000","last_active":"2024-04-02T09:13:41.938581+0000","last_peered":"2024-04-02T09:13:41.938581+0000","last_clean":"2024-04-02T09:13:41.938581+0000","last_became_active":"2024-04-02T09:13:28.399350+0000","last_became_peered":"2024-04-02T09:13:28.399350+0000","last_unstale":"2024-04-02T09:13:41.938581+0000","last_undegraded":"2024-04-02T09:13:41.938581+0000","last_fullsized":"2024-04-02T09:13:41.938581+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:05:05.933699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734569+0000","last_change":"2024-04-02T09:13:41.959689+0000","last_active":"2024-04-02T09:13:43.734569+0000","last_peered":"2024-04-02T09:13:43.734569+0000","last_clean":"2024-04-02T09:13:43.734569+0000","last_became_active":"2024-04-02T09:13:41.959238+0000","last_became_peered":"2024-04-02T09:13:41.959238+0000","last_unstale":"2024-04-02T09:13:43.734569+0000","last_undegraded":"2024-04-02T09:13:43.734569+0000","last_fullsized":"2024-04-02T09:13:43.734569+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:45:09.502243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953945+0000","last_change":"2024-04-02T09:13:41.450544+0000","last_active":"2024-04-02T09:13:41.953945+0000","last_peered":"2024-04-02T09:13:41.953945+0000","last_clean":"2024-04-02T09:13:41.953945+0000","last_became_active":"2024-04-02T09:13:41.450233+0000","last_became_peered":"2024-04-02T09:13:41.450233+0000","last_unstale":"2024-04-02T09:13:41.953945+0000","last_undegraded":"2024-04-02T09:13:41.953945+0000","last_fullsized":"2024-04-02T09:13:41.953945+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:53.842095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950527+0000","last_change":"2024-04-02T09:13:28.386688+0000","last_active":"2024-04-02T09:13:41.950527+0000","last_peered":"2024-04-02T09:13:41.950527+0000","last_clean":"2024-04-02T09:13:41.950527+0000","last_became_active":"2024-04-02T09:13:28.386363+0000","last_became_peered":"2024-04-02T09:13:28.386363+0000","last_unstale":"2024-04-02T09:13:41.950527+0000","last_undegraded":"2024-04-02T09:13:41.950527+0000","last_fullsized":"2024-04-02T09:13:41.950527+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:12:41.480404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954148+0000","last_change":"2024-04-02T09:13:41.952898+0000","last_active":"2024-04-02T09:13:42.954148+0000","last_peered":"2024-04-02T09:13:42.954148+0000","last_clean":"2024-04-02T09:13:42.954148+0000","last_became_active":"2024-04-02T09:13:41.952619+0000","last_became_peered":"2024-04-02T09:13:41.952619+0000","last_unstale":"2024-04-02T09:13:42.954148+0000","last_undegraded":"2024-04-02T09:13:42.954148+0000","last_fullsized":"2024-04-02T09:13:42.954148+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:14:22.720024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.d","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.966777+0000","last_change":"2024-04-02T09:13:41.455641+0000","last_active":"2024-04-02T09:13:41.966777+0000","last_peered":"2024-04-02T09:13:41.966777+0000","last_clean":"2024-04-02T09:13:41.966777+0000","last_became_active":"2024-04-02T09:13:41.455348+0000","last_became_peered":"2024-04-02T09:13:41.455348+0000","last_unstale":"2024-04-02T09:13:41.966777+0000","last_undegraded":"2024-04-02T09:13:41.966777+0000","last_fullsized":"2024-04-02T09:13:41.966777+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:22:20.170595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942920+0000","last_change":"2024-04-02T09:13:28.401903+0000","last_active":"2024-04-02T09:13:41.942920+0000","last_peered":"2024-04-02T09:13:41.942920+0000","last_clean":"2024-04-02T09:13:41.942920+0000","last_became_active":"2024-04-02T09:13:28.399098+0000","last_became_peered":"2024-04-02T09:13:28.399098+0000","last_unstale":"2024-04-02T09:13:41.942920+0000","last_undegraded":"2024-04-02T09:13:41.942920+0000","last_fullsized":"2024-04-02T09:13:41.942920+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:33:56.420978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734685+0000","last_change":"2024-04-02T09:13:41.957617+0000","last_active":"2024-04-02T09:13:43.734685+0000","last_peered":"2024-04-02T09:13:43.734685+0000","last_clean":"2024-04-02T09:13:43.734685+0000","last_became_active":"2024-04-02T09:13:41.957374+0000","last_became_peered":"2024-04-02T09:13:41.957374+0000","last_unstale":"2024-04-02T09:13:43.734685+0000","last_undegraded":"2024-04-02T09:13:43.734685+0000","last_fullsized":"2024-04-02T09:13:43.734685+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:02:20.460847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.e","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.974578+0000","last_change":"2024-04-02T09:13:41.456993+0000","last_active":"2024-04-02T09:13:41.974578+0000","last_peered":"2024-04-02T09:13:41.974578+0000","last_clean":"2024-04-02T09:13:41.974578+0000","last_became_active":"2024-04-02T09:13:41.456687+0000","last_became_peered":"2024-04-02T09:13:41.456687+0000","last_unstale":"2024-04-02T09:13:41.974578+0000","last_undegraded":"2024-04-02T09:13:41.974578+0000","last_fullsized":"2024-04-02T09:13:41.974578+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:19:53.688559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944668+0000","last_change":"2024-04-02T09:13:28.383146+0000","last_active":"2024-04-02T09:13:41.944668+0000","last_peered":"2024-04-02T09:13:41.944668+0000","last_clean":"2024-04-02T09:13:41.944668+0000","last_became_active":"2024-04-02T09:13:28.382879+0000","last_became_peered":"2024-04-02T09:13:28.382879+0000","last_unstale":"2024-04-02T09:13:41.944668+0000","last_undegraded":"2024-04-02T09:13:41.944668+0000","last_fullsized":"2024-04-02T09:13:41.944668+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:57:59.273299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.735294+0000","last_change":"2024-04-02T09:13:41.963375+0000","last_active":"2024-04-02T09:13:43.735294+0000","last_peered":"2024-04-02T09:13:43.735294+0000","last_clean":"2024-04-02T09:13:43.735294+0000","last_became_active":"2024-04-02T09:13:41.961332+0000","last_became_peered":"2024-04-02T09:13:41.961332+0000","last_unstale":"2024-04-02T09:13:43.735294+0000","last_undegraded":"2024-04-02T09:13:43.735294+0000","last_fullsized":"2024-04-02T09:13:43.735294+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:03:38.398878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.f","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.967015+0000","last_change":"2024-04-02T09:13:41.450114+0000","last_active":"2024-04-02T09:13:41.967015+0000","last_peered":"2024-04-02T09:13:41.967015+0000","last_clean":"2024-04-02T09:13:41.967015+0000","last_became_active":"2024-04-02T09:13:41.449847+0000","last_became_peered":"2024-04-02T09:13:41.449847+0000","last_unstale":"2024-04-02T09:13:41.967015+0000","last_undegraded":"2024-04-02T09:13:41.967015+0000","last_fullsized":"2024-04-02T09:13:41.967015+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:56.693535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953001+0000","last_change":"2024-04-02T09:13:28.381499+0000","last_active":"2024-04-02T09:13:41.953001+0000","last_peered":"2024-04-02T09:13:41.953001+0000","last_clean":"2024-04-02T09:13:41.953001+0000","last_became_active":"2024-04-02T09:13:28.377983+0000","last_became_peered":"2024-04-02T09:13:28.377983+0000","last_unstale":"2024-04-02T09:13:41.953001+0000","last_undegraded":"2024-04-02T09:13:41.953001+0000","last_fullsized":"2024-04-02T09:13:41.953001+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:45.021173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734898+0000","last_change":"2024-04-02T09:13:41.958180+0000","last_active":"2024-04-02T09:13:43.734898+0000","last_peered":"2024-04-02T09:13:43.734898+0000","last_clean":"2024-04-02T09:13:43.734898+0000","last_became_active":"2024-04-02T09:13:41.957880+0000","last_became_peered":"2024-04-02T09:13:41.957880+0000","last_unstale":"2024-04-02T09:13:43.734898+0000","last_undegraded":"2024-04-02T09:13:43.734898+0000","last_fullsized":"2024-04-02T09:13:43.734898+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:28.634723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.10","version":"247'8","reported_seq":19,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.975332+0000","last_change":"2024-04-02T09:13:41.453631+0000","last_active":"2024-04-02T09:13:41.975332+0000","last_peered":"2024-04-02T09:13:41.975332+0000","last_clean":"2024-04-02T09:13:41.975332+0000","last_became_active":"2024-04-02T09:13:41.453353+0000","last_became_peered":"2024-04-02T09:13:41.453353+0000","last_unstale":"2024-04-02T09:13:41.975332+0000","last_undegraded":"2024-04-02T09:13:41.975332+0000","last_fullsized":"2024-04-02T09:13:41.975332+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:14.301601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940637+0000","last_change":"2024-04-02T09:13:28.382899+0000","last_active":"2024-04-02T09:13:41.940637+0000","last_peered":"2024-04-02T09:13:41.940637+0000","last_clean":"2024-04-02T09:13:41.940637+0000","last_became_active":"2024-04-02T09:13:28.382023+0000","last_became_peered":"2024-04-02T09:13:28.382023+0000","last_unstale":"2024-04-02T09:13:41.940637+0000","last_undegraded":"2024-04-02T09:13:41.940637+0000","last_fullsized":"2024-04-02T09:13:41.940637+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:12.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345329+0000","last_change":"2024-04-02T09:13:41.972132+0000","last_active":"2024-04-02T09:13:43.345329+0000","last_peered":"2024-04-02T09:13:43.345329+0000","last_clean":"2024-04-02T09:13:43.345329+0000","last_became_active":"2024-04-02T09:13:41.965773+0000","last_became_peered":"2024-04-02T09:13:41.965773+0000","last_unstale":"2024-04-02T09:13:43.345329+0000","last_undegraded":"2024-04-02T09:13:43.345329+0000","last_fullsized":"2024-04-02T09:13:43.345329+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:38.686528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.949402+0000","last_change":"2024-04-02T09:13:41.445504+0000","last_active":"2024-04-02T09:13:41.949402+0000","last_peered":"2024-04-02T09:13:41.949402+0000","last_clean":"2024-04-02T09:13:41.949402+0000","last_became_active":"2024-04-02T09:13:41.445211+0000","last_became_peered":"2024-04-02T09:13:41.445211+0000","last_unstale":"2024-04-02T09:13:41.949402+0000","last_undegraded":"2024-04-02T09:13:41.949402+0000","last_fullsized":"2024-04-02T09:13:41.949402+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:34:40.922210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.955279+0000","last_change":"2024-04-02T09:13:28.390594+0000","last_active":"2024-04-02T09:13:41.955279+0000","last_peered":"2024-04-02T09:13:41.955279+0000","last_clean":"2024-04-02T09:13:41.955279+0000","last_became_active":"2024-04-02T09:13:28.388061+0000","last_became_peered":"2024-04-02T09:13:28.388061+0000","last_unstale":"2024-04-02T09:13:41.955279+0000","last_undegraded":"2024-04-02T09:13:41.955279+0000","last_fullsized":"2024-04-02T09:13:41.955279+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:14:04.841237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345714+0000","last_change":"2024-04-02T09:13:41.966357+0000","last_active":"2024-04-02T09:13:43.345714+0000","last_peered":"2024-04-02T09:13:43.345714+0000","last_clean":"2024-04-02T09:13:43.345714+0000","last_became_active":"2024-04-02T09:13:41.961810+0000","last_became_peered":"2024-04-02T09:13:41.961810+0000","last_unstale":"2024-04-02T09:13:43.345714+0000","last_undegraded":"2024-04-02T09:13:43.345714+0000","last_fullsized":"2024-04-02T09:13:43.345714+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:37.458648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953956+0000","last_change":"2024-04-02T09:13:41.457989+0000","last_active":"2024-04-02T09:13:41.953956+0000","last_peered":"2024-04-02T09:13:41.953956+0000","last_clean":"2024-04-02T09:13:41.953956+0000","last_became_active":"2024-04-02T09:13:41.456512+0000","last_became_peered":"2024-04-02T09:13:41.456512+0000","last_unstale":"2024-04-02T09:13:41.953956+0000","last_undegraded":"2024-04-02T09:13:41.953956+0000","last_fullsized":"2024-04-02T09:13:41.953956+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:28:56.693535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950481+0000","last_change":"2024-04-02T09:13:28.382883+0000","last_active":"2024-04-02T09:13:41.950481+0000","last_peered":"2024-04-02T09:13:41.950481+0000","last_clean":"2024-04-02T09:13:41.950481+0000","last_became_active":"2024-04-02T09:13:28.382187+0000","last_became_peered":"2024-04-02T09:13:28.382187+0000","last_unstale":"2024-04-02T09:13:41.950481+0000","last_undegraded":"2024-04-02T09:13:41.950481+0000","last_fullsized":"2024-04-02T09:13:41.950481+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:21:26.362248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734816+0000","last_change":"2024-04-02T09:13:41.961588+0000","last_active":"2024-04-02T09:13:43.734816+0000","last_peered":"2024-04-02T09:13:43.734816+0000","last_clean":"2024-04-02T09:13:43.734816+0000","last_became_active":"2024-04-02T09:13:41.958669+0000","last_became_peered":"2024-04-02T09:13:41.958669+0000","last_unstale":"2024-04-02T09:13:43.734816+0000","last_undegraded":"2024-04-02T09:13:43.734816+0000","last_fullsized":"2024-04-02T09:13:43.734816+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:46.027233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.13","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.971712+0000","last_change":"2024-04-02T09:13:41.450452+0000","last_active":"2024-04-02T09:13:41.971712+0000","last_peered":"2024-04-02T09:13:41.971712+0000","last_clean":"2024-04-02T09:13:41.971712+0000","last_became_active":"2024-04-02T09:13:41.449005+0000","last_became_peered":"2024-04-02T09:13:41.449005+0000","last_unstale":"2024-04-02T09:13:41.971712+0000","last_undegraded":"2024-04-02T09:13:41.971712+0000","last_fullsized":"2024-04-02T09:13:41.971712+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:19.429201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.961389+0000","last_change":"2024-04-02T09:13:28.384295+0000","last_active":"2024-04-02T09:13:41.961389+0000","last_peered":"2024-04-02T09:13:41.961389+0000","last_clean":"2024-04-02T09:13:41.961389+0000","last_became_active":"2024-04-02T09:13:28.384010+0000","last_became_peered":"2024-04-02T09:13:28.384010+0000","last_unstale":"2024-04-02T09:13:41.961389+0000","last_undegraded":"2024-04-02T09:13:41.961389+0000","last_fullsized":"2024-04-02T09:13:41.961389+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:47:33.241308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.735273+0000","last_change":"2024-04-02T09:13:41.965796+0000","last_active":"2024-04-02T09:13:43.735273+0000","last_peered":"2024-04-02T09:13:43.735273+0000","last_clean":"2024-04-02T09:13:43.735273+0000","last_became_active":"2024-04-02T09:13:41.965262+0000","last_became_peered":"2024-04-02T09:13:41.965262+0000","last_unstale":"2024-04-02T09:13:43.735273+0000","last_undegraded":"2024-04-02T09:13:43.735273+0000","last_fullsized":"2024-04-02T09:13:43.735273+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:35:47.857858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.14","version":"247'6","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.971032+0000","last_change":"2024-04-02T09:13:41.456890+0000","last_active":"2024-04-02T09:13:41.971032+0000","last_peered":"2024-04-02T09:13:41.971032+0000","last_clean":"2024-04-02T09:13:41.971032+0000","last_became_active":"2024-04-02T09:13:41.453059+0000","last_became_peered":"2024-04-02T09:13:41.453059+0000","last_unstale":"2024-04-02T09:13:41.971032+0000","last_undegraded":"2024-04-02T09:13:41.971032+0000","last_fullsized":"2024-04-02T09:13:41.971032+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:08.249339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948077+0000","last_change":"2024-04-02T09:13:28.380209+0000","last_active":"2024-04-02T09:13:41.948077+0000","last_peered":"2024-04-02T09:13:41.948077+0000","last_clean":"2024-04-02T09:13:41.948077+0000","last_became_active":"2024-04-02T09:13:28.379739+0000","last_became_peered":"2024-04-02T09:13:28.379739+0000","last_unstale":"2024-04-02T09:13:41.948077+0000","last_undegraded":"2024-04-02T09:13:41.948077+0000","last_fullsized":"2024-04-02T09:13:41.948077+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:46:14.163290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.346038+0000","last_change":"2024-04-02T09:13:41.956661+0000","last_active":"2024-04-02T09:13:43.346038+0000","last_peered":"2024-04-02T09:13:43.346038+0000","last_clean":"2024-04-02T09:13:43.346038+0000","last_became_active":"2024-04-02T09:13:41.953910+0000","last_became_peered":"2024-04-02T09:13:41.953910+0000","last_unstale":"2024-04-02T09:13:43.346038+0000","last_undegraded":"2024-04-02T09:13:43.346038+0000","last_fullsized":"2024-04-02T09:13:43.346038+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:20.438488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.15","version":"247'3","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.966676+0000","last_change":"2024-04-02T09:13:41.458001+0000","last_active":"2024-04-02T09:13:41.966676+0000","last_peered":"2024-04-02T09:13:41.966676+0000","last_clean":"2024-04-02T09:13:41.966676+0000","last_became_active":"2024-04-02T09:13:41.457318+0000","last_became_peered":"2024-04-02T09:13:41.457318+0000","last_unstale":"2024-04-02T09:13:41.966676+0000","last_undegraded":"2024-04-02T09:13:41.966676+0000","last_fullsized":"2024-04-02T09:13:41.966676+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:47:51.824824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.949330+0000","last_change":"2024-04-02T09:13:28.392993+0000","last_active":"2024-04-02T09:13:41.949330+0000","last_peered":"2024-04-02T09:13:41.949330+0000","last_clean":"2024-04-02T09:13:41.949330+0000","last_became_active":"2024-04-02T09:13:28.392712+0000","last_became_peered":"2024-04-02T09:13:28.392712+0000","last_unstale":"2024-04-02T09:13:41.949330+0000","last_undegraded":"2024-04-02T09:13:41.949330+0000","last_fullsized":"2024-04-02T09:13:41.949330+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:38.436370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.955195+0000","last_change":"2024-04-02T09:13:41.960213+0000","last_active":"2024-04-02T09:13:42.955195+0000","last_peered":"2024-04-02T09:13:42.955195+0000","last_clean":"2024-04-02T09:13:42.955195+0000","last_became_active":"2024-04-02T09:13:41.959935+0000","last_became_peered":"2024-04-02T09:13:41.959935+0000","last_unstale":"2024-04-02T09:13:42.955195+0000","last_undegraded":"2024-04-02T09:13:42.955195+0000","last_fullsized":"2024-04-02T09:13:42.955195+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:11:41.505910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.955082+0000","last_change":"2024-04-02T09:13:41.451596+0000","last_active":"2024-04-02T09:13:41.955082+0000","last_peered":"2024-04-02T09:13:41.955082+0000","last_clean":"2024-04-02T09:13:41.955082+0000","last_became_active":"2024-04-02T09:13:41.451329+0000","last_became_peered":"2024-04-02T09:13:41.451329+0000","last_unstale":"2024-04-02T09:13:41.955082+0000","last_undegraded":"2024-04-02T09:13:41.955082+0000","last_fullsized":"2024-04-02T09:13:41.955082+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:48:38.200651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948538+0000","last_change":"2024-04-02T09:13:28.378030+0000","last_active":"2024-04-02T09:13:41.948538+0000","last_peered":"2024-04-02T09:13:41.948538+0000","last_clean":"2024-04-02T09:13:41.948538+0000","last_became_active":"2024-04-02T09:13:28.377700+0000","last_became_peered":"2024-04-02T09:13:28.377700+0000","last_unstale":"2024-04-02T09:13:41.948538+0000","last_undegraded":"2024-04-02T09:13:41.948538+0000","last_fullsized":"2024-04-02T09:13:41.948538+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:40.236322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734926+0000","last_change":"2024-04-02T09:13:41.956307+0000","last_active":"2024-04-02T09:13:43.734926+0000","last_peered":"2024-04-02T09:13:43.734926+0000","last_clean":"2024-04-02T09:13:43.734926+0000","last_became_active":"2024-04-02T09:13:41.955455+0000","last_became_peered":"2024-04-02T09:13:41.955455+0000","last_unstale":"2024-04-02T09:13:43.734926+0000","last_undegraded":"2024-04-02T09:13:43.734926+0000","last_fullsized":"2024-04-02T09:13:43.734926+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:11.075454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942471+0000","last_change":"2024-04-02T09:13:41.455756+0000","last_active":"2024-04-02T09:13:41.942471+0000","last_peered":"2024-04-02T09:13:41.942471+0000","last_clean":"2024-04-02T09:13:41.942471+0000","last_became_active":"2024-04-02T09:13:41.455267+0000","last_became_peered":"2024-04-02T09:13:41.455267+0000","last_unstale":"2024-04-02T09:13:41.942471+0000","last_undegraded":"2024-04-02T09:13:41.942471+0000","last_fullsized":"2024-04-02T09:13:41.942471+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:34:27.266184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943109+0000","last_change":"2024-04-02T09:13:28.395364+0000","last_active":"2024-04-02T09:13:41.943109+0000","last_peered":"2024-04-02T09:13:41.943109+0000","last_clean":"2024-04-02T09:13:41.943109+0000","last_became_active":"2024-04-02T09:13:28.395100+0000","last_became_peered":"2024-04-02T09:13:28.395100+0000","last_unstale":"2024-04-02T09:13:41.943109+0000","last_undegraded":"2024-04-02T09:13:41.943109+0000","last_fullsized":"2024-04-02T09:13:41.943109+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:55:12.863676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.733818+0000","last_change":"2024-04-02T09:13:41.956248+0000","last_active":"2024-04-02T09:13:43.733818+0000","last_peered":"2024-04-02T09:13:43.733818+0000","last_clean":"2024-04-02T09:13:43.733818+0000","last_became_active":"2024-04-02T09:13:41.955983+0000","last_became_peered":"2024-04-02T09:13:41.955983+0000","last_unstale":"2024-04-02T09:13:43.733818+0000","last_undegraded":"2024-04-02T09:13:43.733818+0000","last_fullsized":"2024-04-02T09:13:43.733818+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:50:02.673300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946966+0000","last_change":"2024-04-02T09:13:41.442991+0000","last_active":"2024-04-02T09:13:41.946966+0000","last_peered":"2024-04-02T09:13:41.946966+0000","last_clean":"2024-04-02T09:13:41.946966+0000","last_became_active":"2024-04-02T09:13:41.442743+0000","last_became_peered":"2024-04-02T09:13:41.442743+0000","last_unstale":"2024-04-02T09:13:41.946966+0000","last_undegraded":"2024-04-02T09:13:41.946966+0000","last_fullsized":"2024-04-02T09:13:41.946966+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:47:51.824824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947036+0000","last_change":"2024-04-02T09:13:28.399604+0000","last_active":"2024-04-02T09:13:41.947036+0000","last_peered":"2024-04-02T09:13:41.947036+0000","last_clean":"2024-04-02T09:13:41.947036+0000","last_became_active":"2024-04-02T09:13:28.399333+0000","last_became_peered":"2024-04-02T09:13:28.399333+0000","last_unstale":"2024-04-02T09:13:41.947036+0000","last_undegraded":"2024-04-02T09:13:41.947036+0000","last_fullsized":"2024-04-02T09:13:41.947036+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:18.748067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345299+0000","last_change":"2024-04-02T09:13:41.955992+0000","last_active":"2024-04-02T09:13:43.345299+0000","last_peered":"2024-04-02T09:13:43.345299+0000","last_clean":"2024-04-02T09:13:43.345299+0000","last_became_active":"2024-04-02T09:13:41.955696+0000","last_became_peered":"2024-04-02T09:13:41.955696+0000","last_unstale":"2024-04-02T09:13:43.345299+0000","last_undegraded":"2024-04-02T09:13:43.345299+0000","last_fullsized":"2024-04-02T09:13:43.345299+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:50:19.183542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944518+0000","last_change":"2024-04-02T09:13:41.454745+0000","last_active":"2024-04-02T09:13:41.944518+0000","last_peered":"2024-04-02T09:13:41.944518+0000","last_clean":"2024-04-02T09:13:41.944518+0000","last_became_active":"2024-04-02T09:13:41.454416+0000","last_became_peered":"2024-04-02T09:13:41.454416+0000","last_unstale":"2024-04-02T09:13:41.944518+0000","last_undegraded":"2024-04-02T09:13:41.944518+0000","last_fullsized":"2024-04-02T09:13:41.944518+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:19.429201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953068+0000","last_change":"2024-04-02T09:13:28.378770+0000","last_active":"2024-04-02T09:13:41.953068+0000","last_peered":"2024-04-02T09:13:41.953068+0000","last_clean":"2024-04-02T09:13:41.953068+0000","last_became_active":"2024-04-02T09:13:28.378521+0000","last_became_peered":"2024-04-02T09:13:28.378521+0000","last_unstale":"2024-04-02T09:13:41.953068+0000","last_undegraded":"2024-04-02T09:13:41.953068+0000","last_fullsized":"2024-04-02T09:13:41.953068+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:24:11.210144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952812+0000","last_change":"2024-04-02T09:13:41.962211+0000","last_active":"2024-04-02T09:13:42.952812+0000","last_peered":"2024-04-02T09:13:42.952812+0000","last_clean":"2024-04-02T09:13:42.952812+0000","last_became_active":"2024-04-02T09:13:41.961032+0000","last_became_peered":"2024-04-02T09:13:41.961032+0000","last_unstale":"2024-04-02T09:13:42.952812+0000","last_undegraded":"2024-04-02T09:13:42.952812+0000","last_fullsized":"2024-04-02T09:13:42.952812+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:43:41.499934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1a","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.971085+0000","last_change":"2024-04-02T09:13:41.455291+0000","last_active":"2024-04-02T09:13:41.971085+0000","last_peered":"2024-04-02T09:13:41.971085+0000","last_clean":"2024-04-02T09:13:41.971085+0000","last_became_active":"2024-04-02T09:13:41.453960+0000","last_became_peered":"2024-04-02T09:13:41.453960+0000","last_unstale":"2024-04-02T09:13:41.971085+0000","last_undegraded":"2024-04-02T09:13:41.971085+0000","last_fullsized":"2024-04-02T09:13:41.971085+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:45:49.868260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.962039+0000","last_change":"2024-04-02T09:13:28.389084+0000","last_active":"2024-04-02T09:13:41.962039+0000","last_peered":"2024-04-02T09:13:41.962039+0000","last_clean":"2024-04-02T09:13:41.962039+0000","last_became_active":"2024-04-02T09:13:28.388624+0000","last_became_peered":"2024-04-02T09:13:28.388624+0000","last_unstale":"2024-04-02T09:13:41.962039+0000","last_undegraded":"2024-04-02T09:13:41.962039+0000","last_fullsized":"2024-04-02T09:13:41.962039+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:23.766461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952740+0000","last_change":"2024-04-02T09:13:41.954960+0000","last_active":"2024-04-02T09:13:42.952740+0000","last_peered":"2024-04-02T09:13:42.952740+0000","last_clean":"2024-04-02T09:13:42.952740+0000","last_became_active":"2024-04-02T09:13:41.952119+0000","last_became_peered":"2024-04-02T09:13:41.952119+0000","last_unstale":"2024-04-02T09:13:42.952740+0000","last_undegraded":"2024-04-02T09:13:42.952740+0000","last_fullsized":"2024-04-02T09:13:42.952740+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:55:20.438488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1b","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.964365+0000","last_change":"2024-04-02T09:13:41.455663+0000","last_active":"2024-04-02T09:13:41.964365+0000","last_peered":"2024-04-02T09:13:41.964365+0000","last_clean":"2024-04-02T09:13:41.964365+0000","last_became_active":"2024-04-02T09:13:41.455401+0000","last_became_peered":"2024-04-02T09:13:41.455401+0000","last_unstale":"2024-04-02T09:13:41.964365+0000","last_undegraded":"2024-04-02T09:13:41.964365+0000","last_fullsized":"2024-04-02T09:13:41.964365+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:45.017947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944969+0000","last_change":"2024-04-02T09:13:28.395286+0000","last_active":"2024-04-02T09:13:41.944969+0000","last_peered":"2024-04-02T09:13:41.944969+0000","last_clean":"2024-04-02T09:13:41.944969+0000","last_became_active":"2024-04-02T09:13:28.394377+0000","last_became_peered":"2024-04-02T09:13:28.394377+0000","last_unstale":"2024-04-02T09:13:41.944969+0000","last_undegraded":"2024-04-02T09:13:41.944969+0000","last_fullsized":"2024-04-02T09:13:41.944969+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:37:39.028435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734569+0000","last_change":"2024-04-02T09:13:41.954547+0000","last_active":"2024-04-02T09:13:43.734569+0000","last_peered":"2024-04-02T09:13:43.734569+0000","last_clean":"2024-04-02T09:13:43.734569+0000","last_became_active":"2024-04-02T09:13:41.954306+0000","last_became_peered":"2024-04-02T09:13:41.954306+0000","last_unstale":"2024-04-02T09:13:43.734569+0000","last_undegraded":"2024-04-02T09:13:43.734569+0000","last_fullsized":"2024-04-02T09:13:43.734569+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:53:09.361699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1c","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.974035+0000","last_change":"2024-04-02T09:13:41.456420+0000","last_active":"2024-04-02T09:13:41.974035+0000","last_peered":"2024-04-02T09:13:41.974035+0000","last_clean":"2024-04-02T09:13:41.974035+0000","last_became_active":"2024-04-02T09:13:41.456055+0000","last_became_peered":"2024-04-02T09:13:41.456055+0000","last_unstale":"2024-04-02T09:13:41.974035+0000","last_undegraded":"2024-04-02T09:13:41.974035+0000","last_fullsized":"2024-04-02T09:13:41.974035+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:52.540226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943657+0000","last_change":"2024-04-02T09:13:28.391267+0000","last_active":"2024-04-02T09:13:41.943657+0000","last_peered":"2024-04-02T09:13:41.943657+0000","last_clean":"2024-04-02T09:13:41.943657+0000","last_became_active":"2024-04-02T09:13:28.390985+0000","last_became_peered":"2024-04-02T09:13:28.390985+0000","last_unstale":"2024-04-02T09:13:41.943657+0000","last_undegraded":"2024-04-02T09:13:41.943657+0000","last_fullsized":"2024-04-02T09:13:41.943657+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:00:30.210394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342493+0000","last_change":"2024-04-02T09:13:41.963616+0000","last_active":"2024-04-02T09:13:43.342493+0000","last_peered":"2024-04-02T09:13:43.342493+0000","last_clean":"2024-04-02T09:13:43.342493+0000","last_became_active":"2024-04-02T09:13:41.963371+0000","last_became_peered":"2024-04-02T09:13:41.963371+0000","last_unstale":"2024-04-02T09:13:43.342493+0000","last_undegraded":"2024-04-02T09:13:43.342493+0000","last_fullsized":"2024-04-02T09:13:43.342493+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:30:47.096602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1d","version":"247'2","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.972845+0000","last_change":"2024-04-02T09:13:41.457926+0000","last_active":"2024-04-02T09:13:41.972845+0000","last_peered":"2024-04-02T09:13:41.972845+0000","last_clean":"2024-04-02T09:13:41.972845+0000","last_became_active":"2024-04-02T09:13:41.457081+0000","last_became_peered":"2024-04-02T09:13:41.457081+0000","last_unstale":"2024-04-02T09:13:41.972845+0000","last_undegraded":"2024-04-02T09:13:41.972845+0000","last_fullsized":"2024-04-02T09:13:41.972845+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:23:34.898152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948952+0000","last_change":"2024-04-02T09:13:28.380650+0000","last_active":"2024-04-02T09:13:41.948952+0000","last_peered":"2024-04-02T09:13:41.948952+0000","last_clean":"2024-04-02T09:13:41.948952+0000","last_became_active":"2024-04-02T09:13:28.380376+0000","last_became_peered":"2024-04-02T09:13:28.380376+0000","last_unstale":"2024-04-02T09:13:41.948952+0000","last_undegraded":"2024-04-02T09:13:41.948952+0000","last_fullsized":"2024-04-02T09:13:41.948952+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:46:17.713244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.346444+0000","last_change":"2024-04-02T09:13:41.961661+0000","last_active":"2024-04-02T09:13:43.346444+0000","last_peered":"2024-04-02T09:13:43.346444+0000","last_clean":"2024-04-02T09:13:43.346444+0000","last_became_active":"2024-04-02T09:13:41.959158+0000","last_became_peered":"2024-04-02T09:13:41.959158+0000","last_unstale":"2024-04-02T09:13:43.346444+0000","last_undegraded":"2024-04-02T09:13:43.346444+0000","last_fullsized":"2024-04-02T09:13:43.346444+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:57:28.703863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.1e","version":"247'1","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.974495+0000","last_change":"2024-04-02T09:13:41.448497+0000","last_active":"2024-04-02T09:13:41.974495+0000","last_peered":"2024-04-02T09:13:41.974495+0000","last_clean":"2024-04-02T09:13:41.974495+0000","last_became_active":"2024-04-02T09:13:41.447385+0000","last_became_peered":"2024-04-02T09:13:41.447385+0000","last_unstale":"2024-04-02T09:13:41.974495+0000","last_undegraded":"2024-04-02T09:13:41.974495+0000","last_fullsized":"2024-04-02T09:13:41.974495+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:21.506479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943796+0000","last_change":"2024-04-02T09:13:28.390273+0000","last_active":"2024-04-02T09:13:41.943796+0000","last_peered":"2024-04-02T09:13:41.943796+0000","last_clean":"2024-04-02T09:13:41.943796+0000","last_became_active":"2024-04-02T09:13:28.389990+0000","last_became_peered":"2024-04-02T09:13:28.389990+0000","last_unstale":"2024-04-02T09:13:41.943796+0000","last_undegraded":"2024-04-02T09:13:41.943796+0000","last_fullsized":"2024-04-02T09:13:41.943796+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:30:33.534225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.948220+0000","last_change":"2024-04-02T09:13:41.958664+0000","last_active":"2024-04-02T09:13:42.948220+0000","last_peered":"2024-04-02T09:13:42.948220+0000","last_clean":"2024-04-02T09:13:42.948220+0000","last_became_active":"2024-04-02T09:13:41.958365+0000","last_became_peered":"2024-04-02T09:13:41.958365+0000","last_unstale":"2024-04-02T09:13:42.948220+0000","last_undegraded":"2024-04-02T09:13:42.948220+0000","last_fullsized":"2024-04-02T09:13:42.948220+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:37.458648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1f","version":"247'4","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.972969+0000","last_change":"2024-04-02T09:13:41.448711+0000","last_active":"2024-04-02T09:13:41.972969+0000","last_peered":"2024-04-02T09:13:41.972969+0000","last_clean":"2024-04-02T09:13:41.972969+0000","last_became_active":"2024-04-02T09:13:41.448398+0000","last_became_peered":"2024-04-02T09:13:41.448398+0000","last_unstale":"2024-04-02T09:13:41.972969+0000","last_undegraded":"2024-04-02T09:13:41.972969+0000","last_fullsized":"2024-04-02T09:13:41.972969+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:40.409301+0000","last_clean_scrub_stamp":"2024-04-02T09:13:40.409301+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:39.602788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952747+0000","last_change":"2024-04-02T09:13:28.395003+0000","last_active":"2024-04-02T09:13:41.952747+0000","last_peered":"2024-04-02T09:13:41.952747+0000","last_clean":"2024-04-02T09:13:41.952747+0000","last_became_active":"2024-04-02T09:13:28.394635+0000","last_became_peered":"2024-04-02T09:13:28.394635+0000","last_unstale":"2024-04-02T09:13:41.952747+0000","last_undegraded":"2024-04-02T09:13:41.952747+0000","last_fullsized":"2024-04-02T09:13:41.952747+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:54.758118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.733918+0000","last_change":"2024-04-02T09:13:41.949873+0000","last_active":"2024-04-02T09:13:43.733918+0000","last_peered":"2024-04-02T09:13:43.733918+0000","last_clean":"2024-04-02T09:13:43.733918+0000","last_became_active":"2024-04-02T09:13:41.949617+0000","last_became_peered":"2024-04-02T09:13:41.949617+0000","last_unstale":"2024-04-02T09:13:43.733918+0000","last_undegraded":"2024-04-02T09:13:43.733918+0000","last_fullsized":"2024-04-02T09:13:43.733918+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:19.901756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.954901+0000","last_change":"2024-04-02T09:13:28.391302+0000","last_active":"2024-04-02T09:13:41.954901+0000","last_peered":"2024-04-02T09:13:41.954901+0000","last_clean":"2024-04-02T09:13:41.954901+0000","last_became_active":"2024-04-02T09:13:28.388976+0000","last_became_peered":"2024-04-02T09:13:28.388976+0000","last_unstale":"2024-04-02T09:13:41.954901+0000","last_undegraded":"2024-04-02T09:13:41.954901+0000","last_fullsized":"2024-04-02T09:13:41.954901+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:02.556232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947787+0000","last_change":"2024-04-02T09:13:27.881269+0000","last_active":"2024-04-02T09:13:41.947787+0000","last_peered":"2024-04-02T09:13:41.947787+0000","last_clean":"2024-04-02T09:13:41.947787+0000","last_became_active":"2024-04-02T09:13:27.880664+0000","last_became_peered":"2024-04-02T09:13:27.880664+0000","last_unstale":"2024-04-02T09:13:41.947787+0000","last_undegraded":"2024-04-02T09:13:41.947787+0000","last_fullsized":"2024-04-02T09:13:41.947787+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:53.750473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734852+0000","last_change":"2024-04-02T09:13:41.967103+0000","last_active":"2024-04-02T09:13:43.734852+0000","last_peered":"2024-04-02T09:13:43.734852+0000","last_clean":"2024-04-02T09:13:43.734852+0000","last_became_active":"2024-04-02T09:13:41.966828+0000","last_became_peered":"2024-04-02T09:13:41.966828+0000","last_unstale":"2024-04-02T09:13:43.734852+0000","last_undegraded":"2024-04-02T09:13:43.734852+0000","last_fullsized":"2024-04-02T09:13:43.734852+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:44:48.623486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945056+0000","last_change":"2024-04-02T09:13:28.399529+0000","last_active":"2024-04-02T09:13:41.945056+0000","last_peered":"2024-04-02T09:13:41.945056+0000","last_clean":"2024-04-02T09:13:41.945056+0000","last_became_active":"2024-04-02T09:13:28.399042+0000","last_became_peered":"2024-04-02T09:13:28.399042+0000","last_unstale":"2024-04-02T09:13:41.945056+0000","last_undegraded":"2024-04-02T09:13:41.945056+0000","last_fullsized":"2024-04-02T09:13:41.945056+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:58:19.109584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1d","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.941583+0000","last_change":"2024-04-02T09:13:27.889299+0000","last_active":"2024-04-02T09:13:41.941583+0000","last_peered":"2024-04-02T09:13:41.941583+0000","last_clean":"2024-04-02T09:13:41.941583+0000","last_became_active":"2024-04-02T09:13:27.888249+0000","last_became_peered":"2024-04-02T09:13:27.888249+0000","last_unstale":"2024-04-02T09:13:41.941583+0000","last_undegraded":"2024-04-02T09:13:41.941583+0000","last_fullsized":"2024-04-02T09:13:41.941583+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:01:31.866958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345673+0000","last_change":"2024-04-02T09:13:41.966096+0000","last_active":"2024-04-02T09:13:43.345673+0000","last_peered":"2024-04-02T09:13:43.345673+0000","last_clean":"2024-04-02T09:13:43.345673+0000","last_became_active":"2024-04-02T09:13:41.959139+0000","last_became_peered":"2024-04-02T09:13:41.959139+0000","last_unstale":"2024-04-02T09:13:43.345673+0000","last_undegraded":"2024-04-02T09:13:43.345673+0000","last_fullsized":"2024-04-02T09:13:43.345673+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:54:06.041578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950575+0000","last_change":"2024-04-02T09:13:28.382423+0000","last_active":"2024-04-02T09:13:41.950575+0000","last_peered":"2024-04-02T09:13:41.950575+0000","last_clean":"2024-04-02T09:13:41.950575+0000","last_became_active":"2024-04-02T09:13:28.381840+0000","last_became_peered":"2024-04-02T09:13:28.381840+0000","last_unstale":"2024-04-02T09:13:41.950575+0000","last_undegraded":"2024-04-02T09:13:41.950575+0000","last_fullsized":"2024-04-02T09:13:41.950575+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:04:44.765954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.954520+0000","last_change":"2024-04-02T09:13:27.885629+0000","last_active":"2024-04-02T09:13:41.954520+0000","last_peered":"2024-04-02T09:13:41.954520+0000","last_clean":"2024-04-02T09:13:41.954520+0000","last_became_active":"2024-04-02T09:13:27.884921+0000","last_became_peered":"2024-04-02T09:13:27.884921+0000","last_unstale":"2024-04-02T09:13:41.954520+0000","last_undegraded":"2024-04-02T09:13:41.954520+0000","last_fullsized":"2024-04-02T09:13:41.954520+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:49:58.033026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734939+0000","last_change":"2024-04-02T09:13:41.963609+0000","last_active":"2024-04-02T09:13:43.734939+0000","last_peered":"2024-04-02T09:13:43.734939+0000","last_clean":"2024-04-02T09:13:43.734939+0000","last_became_active":"2024-04-02T09:13:41.963275+0000","last_became_peered":"2024-04-02T09:13:41.963275+0000","last_unstale":"2024-04-02T09:13:43.734939+0000","last_undegraded":"2024-04-02T09:13:43.734939+0000","last_fullsized":"2024-04-02T09:13:43.734939+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:50:02.673300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952776+0000","last_change":"2024-04-02T09:13:28.381558+0000","last_active":"2024-04-02T09:13:41.952776+0000","last_peered":"2024-04-02T09:13:41.952776+0000","last_clean":"2024-04-02T09:13:41.952776+0000","last_became_active":"2024-04-02T09:13:28.381306+0000","last_became_peered":"2024-04-02T09:13:28.381306+0000","last_unstale":"2024-04-02T09:13:41.952776+0000","last_undegraded":"2024-04-02T09:13:41.952776+0000","last_fullsized":"2024-04-02T09:13:41.952776+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:12:17.313507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1b","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.955013+0000","last_change":"2024-04-02T09:13:27.884062+0000","last_active":"2024-04-02T09:13:41.955013+0000","last_peered":"2024-04-02T09:13:41.955013+0000","last_clean":"2024-04-02T09:13:41.955013+0000","last_became_active":"2024-04-02T09:13:27.883677+0000","last_became_peered":"2024-04-02T09:13:27.883677+0000","last_unstale":"2024-04-02T09:13:41.955013+0000","last_undegraded":"2024-04-02T09:13:41.955013+0000","last_fullsized":"2024-04-02T09:13:41.955013+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:51:14.774442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.948048+0000","last_change":"2024-04-02T09:13:41.955593+0000","last_active":"2024-04-02T09:13:42.948048+0000","last_peered":"2024-04-02T09:13:42.948048+0000","last_clean":"2024-04-02T09:13:42.948048+0000","last_became_active":"2024-04-02T09:13:41.955337+0000","last_became_peered":"2024-04-02T09:13:41.955337+0000","last_unstale":"2024-04-02T09:13:42.948048+0000","last_undegraded":"2024-04-02T09:13:42.948048+0000","last_fullsized":"2024-04-02T09:13:42.948048+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:39:05.298707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943513+0000","last_change":"2024-04-02T09:13:28.389833+0000","last_active":"2024-04-02T09:13:41.943513+0000","last_peered":"2024-04-02T09:13:41.943513+0000","last_clean":"2024-04-02T09:13:41.943513+0000","last_became_active":"2024-04-02T09:13:28.389495+0000","last_became_peered":"2024-04-02T09:13:28.389495+0000","last_unstale":"2024-04-02T09:13:41.943513+0000","last_undegraded":"2024-04-02T09:13:41.943513+0000","last_fullsized":"2024-04-02T09:13:41.943513+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:27.049697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1a","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952655+0000","last_change":"2024-04-02T09:13:27.883532+0000","last_active":"2024-04-02T09:13:41.952655+0000","last_peered":"2024-04-02T09:13:41.952655+0000","last_clean":"2024-04-02T09:13:41.952655+0000","last_became_active":"2024-04-02T09:13:27.883252+0000","last_became_peered":"2024-04-02T09:13:27.883252+0000","last_unstale":"2024-04-02T09:13:41.952655+0000","last_undegraded":"2024-04-02T09:13:41.952655+0000","last_fullsized":"2024-04-02T09:13:41.952655+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:11.950371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.948318+0000","last_change":"2024-04-02T09:13:41.963566+0000","last_active":"2024-04-02T09:13:42.948318+0000","last_peered":"2024-04-02T09:13:42.948318+0000","last_clean":"2024-04-02T09:13:42.948318+0000","last_became_active":"2024-04-02T09:13:41.962503+0000","last_became_peered":"2024-04-02T09:13:41.962503+0000","last_unstale":"2024-04-02T09:13:42.948318+0000","last_undegraded":"2024-04-02T09:13:42.948318+0000","last_fullsized":"2024-04-02T09:13:42.948318+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:13.220603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943231+0000","last_change":"2024-04-02T09:13:28.403943+0000","last_active":"2024-04-02T09:13:41.943231+0000","last_peered":"2024-04-02T09:13:41.943231+0000","last_clean":"2024-04-02T09:13:41.943231+0000","last_became_active":"2024-04-02T09:13:28.402197+0000","last_became_peered":"2024-04-02T09:13:28.402197+0000","last_unstale":"2024-04-02T09:13:41.943231+0000","last_undegraded":"2024-04-02T09:13:41.943231+0000","last_fullsized":"2024-04-02T09:13:41.943231+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:50.579695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945549+0000","last_change":"2024-04-02T09:13:27.883508+0000","last_active":"2024-04-02T09:13:41.945549+0000","last_peered":"2024-04-02T09:13:41.945549+0000","last_clean":"2024-04-02T09:13:41.945549+0000","last_became_active":"2024-04-02T09:13:27.883263+0000","last_became_peered":"2024-04-02T09:13:27.883263+0000","last_unstale":"2024-04-02T09:13:41.945549+0000","last_undegraded":"2024-04-02T09:13:41.945549+0000","last_fullsized":"2024-04-02T09:13:41.945549+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:43:37.428725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954612+0000","last_change":"2024-04-02T09:13:41.965533+0000","last_active":"2024-04-02T09:13:42.954612+0000","last_peered":"2024-04-02T09:13:42.954612+0000","last_clean":"2024-04-02T09:13:42.954612+0000","last_became_active":"2024-04-02T09:13:41.965270+0000","last_became_peered":"2024-04-02T09:13:41.965270+0000","last_unstale":"2024-04-02T09:13:42.954612+0000","last_undegraded":"2024-04-02T09:13:42.954612+0000","last_fullsized":"2024-04-02T09:13:42.954612+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:48:53.549513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943892+0000","last_change":"2024-04-02T09:13:28.403768+0000","last_active":"2024-04-02T09:13:41.943892+0000","last_peered":"2024-04-02T09:13:41.943892+0000","last_clean":"2024-04-02T09:13:41.943892+0000","last_became_active":"2024-04-02T09:13:28.399745+0000","last_became_peered":"2024-04-02T09:13:28.399745+0000","last_unstale":"2024-04-02T09:13:41.943892+0000","last_undegraded":"2024-04-02T09:13:41.943892+0000","last_fullsized":"2024-04-02T09:13:41.943892+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:39.541325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953065+0000","last_change":"2024-04-02T09:13:27.888598+0000","last_active":"2024-04-02T09:13:41.953065+0000","last_peered":"2024-04-02T09:13:41.953065+0000","last_clean":"2024-04-02T09:13:41.953065+0000","last_became_active":"2024-04-02T09:13:27.888115+0000","last_became_peered":"2024-04-02T09:13:27.888115+0000","last_unstale":"2024-04-02T09:13:41.953065+0000","last_undegraded":"2024-04-02T09:13:41.953065+0000","last_fullsized":"2024-04-02T09:13:41.953065+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:57:03.609779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.953354+0000","last_change":"2024-04-02T09:13:41.957572+0000","last_active":"2024-04-02T09:13:42.953354+0000","last_peered":"2024-04-02T09:13:42.953354+0000","last_clean":"2024-04-02T09:13:42.953354+0000","last_became_active":"2024-04-02T09:13:41.956534+0000","last_became_peered":"2024-04-02T09:13:41.956534+0000","last_unstale":"2024-04-02T09:13:42.953354+0000","last_undegraded":"2024-04-02T09:13:42.953354+0000","last_fullsized":"2024-04-02T09:13:42.953354+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:09.258626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947327+0000","last_change":"2024-04-02T09:13:28.404602+0000","last_active":"2024-04-02T09:13:41.947327+0000","last_peered":"2024-04-02T09:13:41.947327+0000","last_clean":"2024-04-02T09:13:41.947327+0000","last_became_active":"2024-04-02T09:13:28.403446+0000","last_became_peered":"2024-04-02T09:13:28.403446+0000","last_unstale":"2024-04-02T09:13:41.947327+0000","last_undegraded":"2024-04-02T09:13:41.947327+0000","last_fullsized":"2024-04-02T09:13:41.947327+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:24.279179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947218+0000","last_change":"2024-04-02T09:13:27.887286+0000","last_active":"2024-04-02T09:13:41.947218+0000","last_peered":"2024-04-02T09:13:41.947218+0000","last_clean":"2024-04-02T09:13:41.947218+0000","last_became_active":"2024-04-02T09:13:27.886999+0000","last_became_peered":"2024-04-02T09:13:27.886999+0000","last_unstale":"2024-04-02T09:13:41.947218+0000","last_undegraded":"2024-04-02T09:13:41.947218+0000","last_fullsized":"2024-04-02T09:13:41.947218+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:44.013528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342182+0000","last_change":"2024-04-02T09:13:41.961802+0000","last_active":"2024-04-02T09:13:43.342182+0000","last_peered":"2024-04-02T09:13:43.342182+0000","last_clean":"2024-04-02T09:13:43.342182+0000","last_became_active":"2024-04-02T09:13:41.961522+0000","last_became_peered":"2024-04-02T09:13:41.961522+0000","last_unstale":"2024-04-02T09:13:43.342182+0000","last_undegraded":"2024-04-02T09:13:43.342182+0000","last_fullsized":"2024-04-02T09:13:43.342182+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:51:04.142072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.954773+0000","last_change":"2024-04-02T09:13:28.378767+0000","last_active":"2024-04-02T09:13:41.954773+0000","last_peered":"2024-04-02T09:13:41.954773+0000","last_clean":"2024-04-02T09:13:41.954773+0000","last_became_active":"2024-04-02T09:13:28.378487+0000","last_became_peered":"2024-04-02T09:13:28.378487+0000","last_unstale":"2024-04-02T09:13:41.954773+0000","last_undegraded":"2024-04-02T09:13:41.954773+0000","last_fullsized":"2024-04-02T09:13:41.954773+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:26:47.539777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947422+0000","last_change":"2024-04-02T09:13:27.884469+0000","last_active":"2024-04-02T09:13:41.947422+0000","last_peered":"2024-04-02T09:13:41.947422+0000","last_clean":"2024-04-02T09:13:41.947422+0000","last_became_active":"2024-04-02T09:13:27.882570+0000","last_became_peered":"2024-04-02T09:13:27.882570+0000","last_unstale":"2024-04-02T09:13:41.947422+0000","last_undegraded":"2024-04-02T09:13:41.947422+0000","last_fullsized":"2024-04-02T09:13:41.947422+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:57:12.395680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.953682+0000","last_change":"2024-04-02T09:13:41.959972+0000","last_active":"2024-04-02T09:13:42.953682+0000","last_peered":"2024-04-02T09:13:42.953682+0000","last_clean":"2024-04-02T09:13:42.953682+0000","last_became_active":"2024-04-02T09:13:41.959655+0000","last_became_peered":"2024-04-02T09:13:41.959655+0000","last_unstale":"2024-04-02T09:13:42.953682+0000","last_undegraded":"2024-04-02T09:13:42.953682+0000","last_fullsized":"2024-04-02T09:13:42.953682+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:23:35.907439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.959311+0000","last_change":"2024-04-02T09:13:28.387205+0000","last_active":"2024-04-02T09:13:41.959311+0000","last_peered":"2024-04-02T09:13:41.959311+0000","last_clean":"2024-04-02T09:13:41.959311+0000","last_became_active":"2024-04-02T09:13:28.384126+0000","last_became_peered":"2024-04-02T09:13:28.384126+0000","last_unstale":"2024-04-02T09:13:41.959311+0000","last_undegraded":"2024-04-02T09:13:41.959311+0000","last_fullsized":"2024-04-02T09:13:41.959311+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:52.330349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.15","version":"243'3","reported_seq":18,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.962102+0000","last_change":"2024-04-02T09:13:27.883818+0000","last_active":"2024-04-02T09:13:41.962102+0000","last_peered":"2024-04-02T09:13:41.962102+0000","last_clean":"2024-04-02T09:13:41.962102+0000","last_became_active":"2024-04-02T09:13:27.883527+0000","last_became_peered":"2024-04-02T09:13:27.883527+0000","last_unstale":"2024-04-02T09:13:41.962102+0000","last_undegraded":"2024-04-02T09:13:41.962102+0000","last_fullsized":"2024-04-02T09:13:41.962102+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:50:49.572049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954621+0000","last_change":"2024-04-02T09:13:41.948944+0000","last_active":"2024-04-02T09:13:42.954621+0000","last_peered":"2024-04-02T09:13:42.954621+0000","last_clean":"2024-04-02T09:13:42.954621+0000","last_became_active":"2024-04-02T09:13:41.948653+0000","last_became_peered":"2024-04-02T09:13:41.948653+0000","last_unstale":"2024-04-02T09:13:42.954621+0000","last_undegraded":"2024-04-02T09:13:42.954621+0000","last_fullsized":"2024-04-02T09:13:42.954621+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:12:23.778874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948000+0000","last_change":"2024-04-02T09:13:28.379346+0000","last_active":"2024-04-02T09:13:41.948000+0000","last_peered":"2024-04-02T09:13:41.948000+0000","last_clean":"2024-04-02T09:13:41.948000+0000","last_became_active":"2024-04-02T09:13:28.378741+0000","last_became_peered":"2024-04-02T09:13:28.378741+0000","last_unstale":"2024-04-02T09:13:41.948000+0000","last_undegraded":"2024-04-02T09:13:41.948000+0000","last_fullsized":"2024-04-02T09:13:41.948000+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:44.092529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.14","version":"244'8","reported_seq":23,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.941516+0000","last_change":"2024-04-02T09:13:27.885270+0000","last_active":"2024-04-02T09:13:41.941516+0000","last_peered":"2024-04-02T09:13:41.941516+0000","last_clean":"2024-04-02T09:13:41.941516+0000","last_became_active":"2024-04-02T09:13:27.884953+0000","last_became_peered":"2024-04-02T09:13:27.884953+0000","last_unstale":"2024-04-02T09:13:41.941516+0000","last_undegraded":"2024-04-02T09:13:41.941516+0000","last_fullsized":"2024-04-02T09:13:41.941516+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:48.103277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2805,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.953123+0000","last_change":"2024-04-02T09:13:41.955589+0000","last_active":"2024-04-02T09:13:42.953123+0000","last_peered":"2024-04-02T09:13:42.953123+0000","last_clean":"2024-04-02T09:13:42.953123+0000","last_became_active":"2024-04-02T09:13:41.955331+0000","last_became_peered":"2024-04-02T09:13:41.955331+0000","last_unstale":"2024-04-02T09:13:42.953123+0000","last_undegraded":"2024-04-02T09:13:42.953123+0000","last_fullsized":"2024-04-02T09:13:42.953123+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:53:35.863141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943802+0000","last_change":"2024-04-02T09:13:28.393735+0000","last_active":"2024-04-02T09:13:41.943802+0000","last_peered":"2024-04-02T09:13:41.943802+0000","last_clean":"2024-04-02T09:13:41.943802+0000","last_became_active":"2024-04-02T09:13:28.393387+0000","last_became_peered":"2024-04-02T09:13:28.393387+0000","last_unstale":"2024-04-02T09:13:41.943802+0000","last_undegraded":"2024-04-02T09:13:41.943802+0000","last_fullsized":"2024-04-02T09:13:41.943802+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:02:47.359176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.13","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940581+0000","last_change":"2024-04-02T09:13:27.885596+0000","last_active":"2024-04-02T09:13:41.940581+0000","last_peered":"2024-04-02T09:13:41.940581+0000","last_clean":"2024-04-02T09:13:41.940581+0000","last_became_active":"2024-04-02T09:13:27.885310+0000","last_became_peered":"2024-04-02T09:13:27.885310+0000","last_unstale":"2024-04-02T09:13:41.940581+0000","last_undegraded":"2024-04-02T09:13:41.940581+0000","last_fullsized":"2024-04-02T09:13:41.940581+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:02.497118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342453+0000","last_change":"2024-04-02T09:13:41.962021+0000","last_active":"2024-04-02T09:13:43.342453+0000","last_peered":"2024-04-02T09:13:43.342453+0000","last_clean":"2024-04-02T09:13:43.342453+0000","last_became_active":"2024-04-02T09:13:41.961212+0000","last_became_peered":"2024-04-02T09:13:41.961212+0000","last_unstale":"2024-04-02T09:13:43.342453+0000","last_undegraded":"2024-04-02T09:13:43.342453+0000","last_fullsized":"2024-04-02T09:13:43.342453+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:37.841557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950657+0000","last_change":"2024-04-02T09:13:28.378384+0000","last_active":"2024-04-02T09:13:41.950657+0000","last_peered":"2024-04-02T09:13:41.950657+0000","last_clean":"2024-04-02T09:13:41.950657+0000","last_became_active":"2024-04-02T09:13:28.377833+0000","last_became_peered":"2024-04-02T09:13:28.377833+0000","last_unstale":"2024-04-02T09:13:41.950657+0000","last_undegraded":"2024-04-02T09:13:41.950657+0000","last_fullsized":"2024-04-02T09:13:41.950657+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:37:54.526506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942840+0000","last_change":"2024-04-02T09:13:27.880069+0000","last_active":"2024-04-02T09:13:41.942840+0000","last_peered":"2024-04-02T09:13:41.942840+0000","last_clean":"2024-04-02T09:13:41.942840+0000","last_became_active":"2024-04-02T09:13:27.879793+0000","last_became_peered":"2024-04-02T09:13:27.879793+0000","last_unstale":"2024-04-02T09:13:41.942840+0000","last_undegraded":"2024-04-02T09:13:41.942840+0000","last_fullsized":"2024-04-02T09:13:41.942840+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:10.198371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734950+0000","last_change":"2024-04-02T09:13:41.951761+0000","last_active":"2024-04-02T09:13:43.734950+0000","last_peered":"2024-04-02T09:13:43.734950+0000","last_clean":"2024-04-02T09:13:43.734950+0000","last_became_active":"2024-04-02T09:13:41.951447+0000","last_became_peered":"2024-04-02T09:13:41.951447+0000","last_unstale":"2024-04-02T09:13:43.734950+0000","last_undegraded":"2024-04-02T09:13:43.734950+0000","last_fullsized":"2024-04-02T09:13:43.734950+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:38:37.458648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952962+0000","last_change":"2024-04-02T09:13:28.394088+0000","last_active":"2024-04-02T09:13:41.952962+0000","last_peered":"2024-04-02T09:13:41.952962+0000","last_clean":"2024-04-02T09:13:41.952962+0000","last_became_active":"2024-04-02T09:13:28.392932+0000","last_became_peered":"2024-04-02T09:13:28.392932+0000","last_unstale":"2024-04-02T09:13:41.952962+0000","last_undegraded":"2024-04-02T09:13:41.952962+0000","last_fullsized":"2024-04-02T09:13:41.952962+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:54.683721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945630+0000","last_change":"2024-04-02T09:13:27.881714+0000","last_active":"2024-04-02T09:13:41.945630+0000","last_peered":"2024-04-02T09:13:41.945630+0000","last_clean":"2024-04-02T09:13:41.945630+0000","last_became_active":"2024-04-02T09:13:27.881428+0000","last_became_peered":"2024-04-02T09:13:27.881428+0000","last_unstale":"2024-04-02T09:13:41.945630+0000","last_undegraded":"2024-04-02T09:13:41.945630+0000","last_fullsized":"2024-04-02T09:13:41.945630+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:50.081688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.346442+0000","last_change":"2024-04-02T09:13:41.972497+0000","last_active":"2024-04-02T09:13:43.346442+0000","last_peered":"2024-04-02T09:13:43.346442+0000","last_clean":"2024-04-02T09:13:43.346442+0000","last_became_active":"2024-04-02T09:13:41.972137+0000","last_became_peered":"2024-04-02T09:13:41.972137+0000","last_unstale":"2024-04-02T09:13:43.346442+0000","last_undegraded":"2024-04-02T09:13:43.346442+0000","last_fullsized":"2024-04-02T09:13:43.346442+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:19.631975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944595+0000","last_change":"2024-04-02T09:13:28.391188+0000","last_active":"2024-04-02T09:13:41.944595+0000","last_peered":"2024-04-02T09:13:41.944595+0000","last_clean":"2024-04-02T09:13:41.944595+0000","last_became_active":"2024-04-02T09:13:28.390867+0000","last_became_peered":"2024-04-02T09:13:28.390867+0000","last_unstale":"2024-04-02T09:13:41.944595+0000","last_undegraded":"2024-04-02T09:13:41.944595+0000","last_fullsized":"2024-04-02T09:13:41.944595+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:26:58.531401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.10","version":"243'8","reported_seq":23,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.952918+0000","last_change":"2024-04-02T09:13:27.886503+0000","last_active":"2024-04-02T09:13:41.952918+0000","last_peered":"2024-04-02T09:13:41.952918+0000","last_clean":"2024-04-02T09:13:41.952918+0000","last_became_active":"2024-04-02T09:13:27.886121+0000","last_became_peered":"2024-04-02T09:13:27.886121+0000","last_unstale":"2024-04-02T09:13:41.952918+0000","last_undegraded":"2024-04-02T09:13:41.952918+0000","last_fullsized":"2024-04-02T09:13:41.952918+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:38.533680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.952724+0000","last_change":"2024-04-02T09:13:41.954229+0000","last_active":"2024-04-02T09:13:42.952724+0000","last_peered":"2024-04-02T09:13:42.952724+0000","last_clean":"2024-04-02T09:13:42.952724+0000","last_became_active":"2024-04-02T09:13:41.950847+0000","last_became_peered":"2024-04-02T09:13:41.950847+0000","last_unstale":"2024-04-02T09:13:42.952724+0000","last_undegraded":"2024-04-02T09:13:42.952724+0000","last_fullsized":"2024-04-02T09:13:42.952724+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:55:58.664898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.955366+0000","last_change":"2024-04-02T09:13:28.376609+0000","last_active":"2024-04-02T09:13:41.955366+0000","last_peered":"2024-04-02T09:13:41.955366+0000","last_clean":"2024-04-02T09:13:41.955366+0000","last_became_active":"2024-04-02T09:13:28.376226+0000","last_became_peered":"2024-04-02T09:13:28.376226+0000","last_unstale":"2024-04-02T09:13:41.955366+0000","last_undegraded":"2024-04-02T09:13:41.955366+0000","last_fullsized":"2024-04-02T09:13:41.955366+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:27:59.766007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948352+0000","last_change":"2024-04-02T09:13:27.883080+0000","last_active":"2024-04-02T09:13:41.948352+0000","last_peered":"2024-04-02T09:13:41.948352+0000","last_clean":"2024-04-02T09:13:41.948352+0000","last_became_active":"2024-04-02T09:13:27.882612+0000","last_became_peered":"2024-04-02T09:13:27.882612+0000","last_unstale":"2024-04-02T09:13:41.948352+0000","last_undegraded":"2024-04-02T09:13:41.948352+0000","last_fullsized":"2024-04-02T09:13:41.948352+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:20.979570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.735089+0000","last_change":"2024-04-02T09:13:41.960521+0000","last_active":"2024-04-02T09:13:43.735089+0000","last_peered":"2024-04-02T09:13:43.735089+0000","last_clean":"2024-04-02T09:13:43.735089+0000","last_became_active":"2024-04-02T09:13:41.960258+0000","last_became_peered":"2024-04-02T09:13:41.960258+0000","last_unstale":"2024-04-02T09:13:43.735089+0000","last_undegraded":"2024-04-02T09:13:43.735089+0000","last_fullsized":"2024-04-02T09:13:43.735089+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:45:09.502243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.961541+0000","last_change":"2024-04-02T09:13:28.381499+0000","last_active":"2024-04-02T09:13:41.961541+0000","last_peered":"2024-04-02T09:13:41.961541+0000","last_clean":"2024-04-02T09:13:41.961541+0000","last_became_active":"2024-04-02T09:13:28.381286+0000","last_became_peered":"2024-04-02T09:13:28.381286+0000","last_unstale":"2024-04-02T09:13:41.961541+0000","last_undegraded":"2024-04-02T09:13:41.961541+0000","last_fullsized":"2024-04-02T09:13:41.961541+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:18.748067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.e","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947518+0000","last_change":"2024-04-02T09:13:27.888558+0000","last_active":"2024-04-02T09:13:41.947518+0000","last_peered":"2024-04-02T09:13:41.947518+0000","last_clean":"2024-04-02T09:13:41.947518+0000","last_became_active":"2024-04-02T09:13:27.888288+0000","last_became_peered":"2024-04-02T09:13:27.888288+0000","last_unstale":"2024-04-02T09:13:41.947518+0000","last_undegraded":"2024-04-02T09:13:41.947518+0000","last_fullsized":"2024-04-02T09:13:41.947518+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:15.386238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734712+0000","last_change":"2024-04-02T09:13:41.967882+0000","last_active":"2024-04-02T09:13:43.734712+0000","last_peered":"2024-04-02T09:13:43.734712+0000","last_clean":"2024-04-02T09:13:43.734712+0000","last_became_active":"2024-04-02T09:13:41.967598+0000","last_became_peered":"2024-04-02T09:13:41.967598+0000","last_unstale":"2024-04-02T09:13:43.734712+0000","last_undegraded":"2024-04-02T09:13:43.734712+0000","last_fullsized":"2024-04-02T09:13:43.734712+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:18.838465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948337+0000","last_change":"2024-04-02T09:13:28.372849+0000","last_active":"2024-04-02T09:13:41.948337+0000","last_peered":"2024-04-02T09:13:41.948337+0000","last_clean":"2024-04-02T09:13:41.948337+0000","last_became_active":"2024-04-02T09:13:28.372595+0000","last_became_peered":"2024-04-02T09:13:28.372595+0000","last_unstale":"2024-04-02T09:13:41.948337+0000","last_undegraded":"2024-04-02T09:13:41.948337+0000","last_fullsized":"2024-04-02T09:13:41.948337+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:27:12.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.d","version":"243'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947395+0000","last_change":"2024-04-02T09:13:27.879417+0000","last_active":"2024-04-02T09:13:41.947395+0000","last_peered":"2024-04-02T09:13:41.947395+0000","last_clean":"2024-04-02T09:13:41.947395+0000","last_became_active":"2024-04-02T09:13:27.879165+0000","last_became_peered":"2024-04-02T09:13:27.879165+0000","last_unstale":"2024-04-02T09:13:41.947395+0000","last_undegraded":"2024-04-02T09:13:41.947395+0000","last_fullsized":"2024-04-02T09:13:41.947395+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:47:32.233662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734901+0000","last_change":"2024-04-02T09:13:41.968877+0000","last_active":"2024-04-02T09:13:43.734901+0000","last_peered":"2024-04-02T09:13:43.734901+0000","last_clean":"2024-04-02T09:13:43.734901+0000","last_became_active":"2024-04-02T09:13:41.968456+0000","last_became_peered":"2024-04-02T09:13:41.968456+0000","last_unstale":"2024-04-02T09:13:43.734901+0000","last_undegraded":"2024-04-02T09:13:43.734901+0000","last_fullsized":"2024-04-02T09:13:43.734901+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:54.806757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.939446+0000","last_change":"2024-04-02T09:13:28.392130+0000","last_active":"2024-04-02T09:13:41.939446+0000","last_peered":"2024-04-02T09:13:41.939446+0000","last_clean":"2024-04-02T09:13:41.939446+0000","last_became_active":"2024-04-02T09:13:28.391690+0000","last_became_peered":"2024-04-02T09:13:28.391690+0000","last_unstale":"2024-04-02T09:13:41.939446+0000","last_undegraded":"2024-04-02T09:13:41.939446+0000","last_fullsized":"2024-04-02T09:13:41.939446+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:24.319747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945343+0000","last_change":"2024-04-02T09:13:27.884781+0000","last_active":"2024-04-02T09:13:41.945343+0000","last_peered":"2024-04-02T09:13:41.945343+0000","last_clean":"2024-04-02T09:13:41.945343+0000","last_became_active":"2024-04-02T09:13:27.884507+0000","last_became_peered":"2024-04-02T09:13:27.884507+0000","last_unstale":"2024-04-02T09:13:41.945343+0000","last_undegraded":"2024-04-02T09:13:41.945343+0000","last_fullsized":"2024-04-02T09:13:41.945343+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:31:18.889273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342453+0000","last_change":"2024-04-02T09:13:41.958635+0000","last_active":"2024-04-02T09:13:43.342453+0000","last_peered":"2024-04-02T09:13:43.342453+0000","last_clean":"2024-04-02T09:13:43.342453+0000","last_became_active":"2024-04-02T09:13:41.955459+0000","last_became_peered":"2024-04-02T09:13:41.955459+0000","last_unstale":"2024-04-02T09:13:43.342453+0000","last_undegraded":"2024-04-02T09:13:43.342453+0000","last_fullsized":"2024-04-02T09:13:43.342453+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:58:53.103702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947024+0000","last_change":"2024-04-02T09:13:28.381580+0000","last_active":"2024-04-02T09:13:41.947024+0000","last_peered":"2024-04-02T09:13:41.947024+0000","last_clean":"2024-04-02T09:13:41.947024+0000","last_became_active":"2024-04-02T09:13:28.381300+0000","last_became_peered":"2024-04-02T09:13:28.381300+0000","last_unstale":"2024-04-02T09:13:41.947024+0000","last_undegraded":"2024-04-02T09:13:41.947024+0000","last_fullsized":"2024-04-02T09:13:41.947024+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:51.089333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944130+0000","last_change":"2024-04-02T09:13:27.875291+0000","last_active":"2024-04-02T09:13:41.944130+0000","last_peered":"2024-04-02T09:13:41.944130+0000","last_clean":"2024-04-02T09:13:41.944130+0000","last_became_active":"2024-04-02T09:13:27.875003+0000","last_became_peered":"2024-04-02T09:13:27.875003+0000","last_unstale":"2024-04-02T09:13:41.944130+0000","last_undegraded":"2024-04-02T09:13:41.944130+0000","last_fullsized":"2024-04-02T09:13:41.944130+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:40:00.637981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734207+0000","last_change":"2024-04-02T09:13:41.949115+0000","last_active":"2024-04-02T09:13:43.734207+0000","last_peered":"2024-04-02T09:13:43.734207+0000","last_clean":"2024-04-02T09:13:43.734207+0000","last_became_active":"2024-04-02T09:13:41.948850+0000","last_became_peered":"2024-04-02T09:13:41.948850+0000","last_unstale":"2024-04-02T09:13:43.734207+0000","last_undegraded":"2024-04-02T09:13:43.734207+0000","last_fullsized":"2024-04-02T09:13:43.734207+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:54.806757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.944021+0000","last_change":"2024-04-02T09:13:28.404265+0000","last_active":"2024-04-02T09:13:41.944021+0000","last_peered":"2024-04-02T09:13:41.944021+0000","last_clean":"2024-04-02T09:13:41.944021+0000","last_became_active":"2024-04-02T09:13:28.403894+0000","last_became_peered":"2024-04-02T09:13:28.403894+0000","last_unstale":"2024-04-02T09:13:41.944021+0000","last_undegraded":"2024-04-02T09:13:41.944021+0000","last_fullsized":"2024-04-02T09:13:41.944021+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:05:23.433426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943067+0000","last_change":"2024-04-02T09:13:27.881604+0000","last_active":"2024-04-02T09:13:41.943067+0000","last_peered":"2024-04-02T09:13:41.943067+0000","last_clean":"2024-04-02T09:13:41.943067+0000","last_became_active":"2024-04-02T09:13:27.881304+0000","last_became_peered":"2024-04-02T09:13:27.881304+0000","last_unstale":"2024-04-02T09:13:41.943067+0000","last_undegraded":"2024-04-02T09:13:41.943067+0000","last_fullsized":"2024-04-02T09:13:41.943067+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:25:02.497118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734857+0000","last_change":"2024-04-02T09:13:41.956788+0000","last_active":"2024-04-02T09:13:43.734857+0000","last_peered":"2024-04-02T09:13:43.734857+0000","last_clean":"2024-04-02T09:13:43.734857+0000","last_became_active":"2024-04-02T09:13:41.956513+0000","last_became_peered":"2024-04-02T09:13:41.956513+0000","last_unstale":"2024-04-02T09:13:43.734857+0000","last_undegraded":"2024-04-02T09:13:43.734857+0000","last_fullsized":"2024-04-02T09:13:43.734857+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:39:05.298707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953053+0000","last_change":"2024-04-02T09:13:28.374960+0000","last_active":"2024-04-02T09:13:41.953053+0000","last_peered":"2024-04-02T09:13:41.953053+0000","last_clean":"2024-04-02T09:13:41.953053+0000","last_became_active":"2024-04-02T09:13:28.373683+0000","last_became_peered":"2024-04-02T09:13:28.373683+0000","last_unstale":"2024-04-02T09:13:41.953053+0000","last_undegraded":"2024-04-02T09:13:41.953053+0000","last_fullsized":"2024-04-02T09:13:41.953053+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:15.851301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.938571+0000","last_change":"2024-04-02T09:13:27.877837+0000","last_active":"2024-04-02T09:13:41.938571+0000","last_peered":"2024-04-02T09:13:41.938571+0000","last_clean":"2024-04-02T09:13:41.938571+0000","last_became_active":"2024-04-02T09:13:27.876792+0000","last_became_peered":"2024-04-02T09:13:27.876792+0000","last_unstale":"2024-04-02T09:13:41.938571+0000","last_undegraded":"2024-04-02T09:13:41.938571+0000","last_fullsized":"2024-04-02T09:13:41.938571+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:01:17.740422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.954331+0000","last_change":"2024-04-02T09:13:41.949429+0000","last_active":"2024-04-02T09:13:42.954331+0000","last_peered":"2024-04-02T09:13:42.954331+0000","last_clean":"2024-04-02T09:13:42.954331+0000","last_became_active":"2024-04-02T09:13:41.948422+0000","last_became_peered":"2024-04-02T09:13:41.948422+0000","last_unstale":"2024-04-02T09:13:42.954331+0000","last_undegraded":"2024-04-02T09:13:42.954331+0000","last_fullsized":"2024-04-02T09:13:42.954331+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:31:40.330222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945371+0000","last_change":"2024-04-02T09:13:28.379243+0000","last_active":"2024-04-02T09:13:41.945371+0000","last_peered":"2024-04-02T09:13:41.945371+0000","last_clean":"2024-04-02T09:13:41.945371+0000","last_became_active":"2024-04-02T09:13:28.378297+0000","last_became_peered":"2024-04-02T09:13:28.378297+0000","last_unstale":"2024-04-02T09:13:41.945371+0000","last_undegraded":"2024-04-02T09:13:41.945371+0000","last_fullsized":"2024-04-02T09:13:41.945371+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:02.761190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.938490+0000","last_change":"2024-04-02T09:13:27.876051+0000","last_active":"2024-04-02T09:13:41.938490+0000","last_peered":"2024-04-02T09:13:41.938490+0000","last_clean":"2024-04-02T09:13:41.938490+0000","last_became_active":"2024-04-02T09:13:27.875628+0000","last_became_peered":"2024-04-02T09:13:27.875628+0000","last_unstale":"2024-04-02T09:13:41.938490+0000","last_undegraded":"2024-04-02T09:13:41.938490+0000","last_fullsized":"2024-04-02T09:13:41.938490+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:09:53.750473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:42.947186+0000","last_change":"2024-04-02T09:13:41.961652+0000","last_active":"2024-04-02T09:13:42.947186+0000","last_peered":"2024-04-02T09:13:42.947186+0000","last_clean":"2024-04-02T09:13:42.947186+0000","last_became_active":"2024-04-02T09:13:41.961378+0000","last_became_peered":"2024-04-02T09:13:41.961378+0000","last_unstale":"2024-04-02T09:13:42.947186+0000","last_undegraded":"2024-04-02T09:13:42.947186+0000","last_fullsized":"2024-04-02T09:13:42.947186+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:51:51.721965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947638+0000","last_change":"2024-04-02T09:13:28.383204+0000","last_active":"2024-04-02T09:13:41.947638+0000","last_peered":"2024-04-02T09:13:41.947638+0000","last_clean":"2024-04-02T09:13:41.947638+0000","last_became_active":"2024-04-02T09:13:28.382961+0000","last_became_peered":"2024-04-02T09:13:28.382961+0000","last_unstale":"2024-04-02T09:13:41.947638+0000","last_undegraded":"2024-04-02T09:13:41.947638+0000","last_fullsized":"2024-04-02T09:13:41.947638+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:16:14.265813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.7","version":"243'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946858+0000","last_change":"2024-04-02T09:13:27.886740+0000","last_active":"2024-04-02T09:13:41.946858+0000","last_peered":"2024-04-02T09:13:41.946858+0000","last_clean":"2024-04-02T09:13:41.946858+0000","last_became_active":"2024-04-02T09:13:27.886385+0000","last_became_peered":"2024-04-02T09:13:27.886385+0000","last_unstale":"2024-04-02T09:13:41.946858+0000","last_undegraded":"2024-04-02T09:13:41.946858+0000","last_fullsized":"2024-04-02T09:13:41.946858+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:23:20.383235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734173+0000","last_change":"2024-04-02T09:13:41.954104+0000","last_active":"2024-04-02T09:13:43.734173+0000","last_peered":"2024-04-02T09:13:43.734173+0000","last_clean":"2024-04-02T09:13:43.734173+0000","last_became_active":"2024-04-02T09:13:41.953393+0000","last_became_peered":"2024-04-02T09:13:41.953393+0000","last_unstale":"2024-04-02T09:13:43.734173+0000","last_undegraded":"2024-04-02T09:13:43.734173+0000","last_fullsized":"2024-04-02T09:13:43.734173+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:01:46.436981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.942847+0000","last_change":"2024-04-02T09:13:28.402497+0000","last_active":"2024-04-02T09:13:41.942847+0000","last_peered":"2024-04-02T09:13:41.942847+0000","last_clean":"2024-04-02T09:13:41.942847+0000","last_became_active":"2024-04-02T09:13:28.402235+0000","last_became_peered":"2024-04-02T09:13:28.402235+0000","last_unstale":"2024-04-02T09:13:41.942847+0000","last_undegraded":"2024-04-02T09:13:41.942847+0000","last_fullsized":"2024-04-02T09:13:41.942847+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:20:15.072346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.6","version":"244'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.950712+0000","last_change":"2024-04-02T09:13:27.884225+0000","last_active":"2024-04-02T09:13:41.950712+0000","last_peered":"2024-04-02T09:13:41.950712+0000","last_clean":"2024-04-02T09:13:41.950712+0000","last_became_active":"2024-04-02T09:13:27.883536+0000","last_became_peered":"2024-04-02T09:13:27.883536+0000","last_unstale":"2024-04-02T09:13:41.950712+0000","last_undegraded":"2024-04-02T09:13:41.950712+0000","last_fullsized":"2024-04-02T09:13:41.950712+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:39:50.645932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.345681+0000","last_change":"2024-04-02T09:13:41.955484+0000","last_active":"2024-04-02T09:13:43.345681+0000","last_peered":"2024-04-02T09:13:43.345681+0000","last_clean":"2024-04-02T09:13:43.345681+0000","last_became_active":"2024-04-02T09:13:41.953502+0000","last_became_peered":"2024-04-02T09:13:41.953502+0000","last_unstale":"2024-04-02T09:13:43.345681+0000","last_undegraded":"2024-04-02T09:13:43.345681+0000","last_fullsized":"2024-04-02T09:13:43.345681+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:05:50.626394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953832+0000","last_change":"2024-04-02T09:13:28.387139+0000","last_active":"2024-04-02T09:13:41.953832+0000","last_peered":"2024-04-02T09:13:41.953832+0000","last_clean":"2024-04-02T09:13:41.953832+0000","last_became_active":"2024-04-02T09:13:28.386878+0000","last_became_peered":"2024-04-02T09:13:28.386878+0000","last_unstale":"2024-04-02T09:13:41.953832+0000","last_undegraded":"2024-04-02T09:13:41.953832+0000","last_fullsized":"2024-04-02T09:13:41.953832+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:56:34.681485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948085+0000","last_change":"2024-04-02T09:13:27.882615+0000","last_active":"2024-04-02T09:13:41.948085+0000","last_peered":"2024-04-02T09:13:41.948085+0000","last_clean":"2024-04-02T09:13:41.948085+0000","last_became_active":"2024-04-02T09:13:27.882334+0000","last_became_peered":"2024-04-02T09:13:27.882334+0000","last_unstale":"2024-04-02T09:13:41.948085+0000","last_undegraded":"2024-04-02T09:13:41.948085+0000","last_fullsized":"2024-04-02T09:13:41.948085+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:43.215347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.733778+0000","last_change":"2024-04-02T09:13:41.962699+0000","last_active":"2024-04-02T09:13:43.733778+0000","last_peered":"2024-04-02T09:13:43.733778+0000","last_clean":"2024-04-02T09:13:43.733778+0000","last_became_active":"2024-04-02T09:13:41.960572+0000","last_became_peered":"2024-04-02T09:13:41.960572+0000","last_unstale":"2024-04-02T09:13:43.733778+0000","last_undegraded":"2024-04-02T09:13:43.733778+0000","last_fullsized":"2024-04-02T09:13:43.733778+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:22:21.179882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.946870+0000","last_change":"2024-04-02T09:13:28.402303+0000","last_active":"2024-04-02T09:13:41.946870+0000","last_peered":"2024-04-02T09:13:41.946870+0000","last_clean":"2024-04-02T09:13:41.946870+0000","last_became_active":"2024-04-02T09:13:28.399305+0000","last_became_peered":"2024-04-02T09:13:28.399305+0000","last_unstale":"2024-04-02T09:13:41.946870+0000","last_undegraded":"2024-04-02T09:13:41.946870+0000","last_fullsized":"2024-04-02T09:13:41.946870+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:30:58.927786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.945487+0000","last_change":"2024-04-02T09:13:27.881268+0000","last_active":"2024-04-02T09:13:41.945487+0000","last_peered":"2024-04-02T09:13:41.945487+0000","last_clean":"2024-04-02T09:13:41.945487+0000","last_became_active":"2024-04-02T09:13:27.880983+0000","last_became_peered":"2024-04-02T09:13:27.880983+0000","last_unstale":"2024-04-02T09:13:41.945487+0000","last_undegraded":"2024-04-02T09:13:41.945487+0000","last_fullsized":"2024-04-02T09:13:41.945487+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:09.803095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734037+0000","last_change":"2024-04-02T09:13:41.955819+0000","last_active":"2024-04-02T09:13:43.734037+0000","last_peered":"2024-04-02T09:13:43.734037+0000","last_clean":"2024-04-02T09:13:43.734037+0000","last_became_active":"2024-04-02T09:13:41.955568+0000","last_became_peered":"2024-04-02T09:13:41.955568+0000","last_unstale":"2024-04-02T09:13:43.734037+0000","last_undegraded":"2024-04-02T09:13:43.734037+0000","last_fullsized":"2024-04-02T09:13:43.734037+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:44:48.623486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.949252+0000","last_change":"2024-04-02T09:13:28.374285+0000","last_active":"2024-04-02T09:13:41.949252+0000","last_peered":"2024-04-02T09:13:41.949252+0000","last_clean":"2024-04-02T09:13:41.949252+0000","last_became_active":"2024-04-02T09:13:28.373980+0000","last_became_peered":"2024-04-02T09:13:28.373980+0000","last_unstale":"2024-04-02T09:13:41.949252+0000","last_undegraded":"2024-04-02T09:13:41.949252+0000","last_fullsized":"2024-04-02T09:13:41.949252+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:56:34.681485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.947216+0000","last_change":"2024-04-02T09:13:27.882249+0000","last_active":"2024-04-02T09:13:41.947216+0000","last_peered":"2024-04-02T09:13:41.947216+0000","last_clean":"2024-04-02T09:13:41.947216+0000","last_became_active":"2024-04-02T09:13:27.881833+0000","last_became_peered":"2024-04-02T09:13:27.881833+0000","last_unstale":"2024-04-02T09:13:41.947216+0000","last_undegraded":"2024-04-02T09:13:41.947216+0000","last_fullsized":"2024-04-02T09:13:41.947216+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:22.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.342582+0000","last_change":"2024-04-02T09:13:41.960270+0000","last_active":"2024-04-02T09:13:43.342582+0000","last_peered":"2024-04-02T09:13:43.342582+0000","last_clean":"2024-04-02T09:13:43.342582+0000","last_became_active":"2024-04-02T09:13:41.960009+0000","last_became_peered":"2024-04-02T09:13:41.960009+0000","last_unstale":"2024-04-02T09:13:43.342582+0000","last_undegraded":"2024-04-02T09:13:43.342582+0000","last_fullsized":"2024-04-02T09:13:43.342582+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:37:52.590813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948199+0000","last_change":"2024-04-02T09:13:28.389208+0000","last_active":"2024-04-02T09:13:41.948199+0000","last_peered":"2024-04-02T09:13:41.948199+0000","last_clean":"2024-04-02T09:13:41.948199+0000","last_became_active":"2024-04-02T09:13:28.388923+0000","last_became_peered":"2024-04-02T09:13:28.388923+0000","last_unstale":"2024-04-02T09:13:41.948199+0000","last_undegraded":"2024-04-02T09:13:41.948199+0000","last_fullsized":"2024-04-02T09:13:41.948199+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:35:55.299814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953418+0000","last_change":"2024-04-02T09:13:27.887509+0000","last_active":"2024-04-02T09:13:41.953418+0000","last_peered":"2024-04-02T09:13:41.953418+0000","last_clean":"2024-04-02T09:13:41.953418+0000","last_became_active":"2024-04-02T09:13:27.887260+0000","last_became_peered":"2024-04-02T09:13:27.887260+0000","last_unstale":"2024-04-02T09:13:41.953418+0000","last_undegraded":"2024-04-02T09:13:41.953418+0000","last_fullsized":"2024-04-02T09:13:41.953418+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:28.581007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734767+0000","last_change":"2024-04-02T09:13:41.960376+0000","last_active":"2024-04-02T09:13:43.734767+0000","last_peered":"2024-04-02T09:13:43.734767+0000","last_clean":"2024-04-02T09:13:43.734767+0000","last_became_active":"2024-04-02T09:13:41.960062+0000","last_became_peered":"2024-04-02T09:13:41.960062+0000","last_unstale":"2024-04-02T09:13:43.734767+0000","last_undegraded":"2024-04-02T09:13:43.734767+0000","last_fullsized":"2024-04-02T09:13:43.734767+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:13.220603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.937842+0000","last_change":"2024-04-02T09:13:28.374292+0000","last_active":"2024-04-02T09:13:41.937842+0000","last_peered":"2024-04-02T09:13:41.937842+0000","last_clean":"2024-04-02T09:13:41.937842+0000","last_became_active":"2024-04-02T09:13:28.373689+0000","last_became_peered":"2024-04-02T09:13:28.373689+0000","last_unstale":"2024-04-02T09:13:41.937842+0000","last_undegraded":"2024-04-02T09:13:41.937842+0000","last_fullsized":"2024-04-02T09:13:41.937842+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:08.953388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940440+0000","last_change":"2024-04-02T09:13:27.882492+0000","last_active":"2024-04-02T09:13:41.940440+0000","last_peered":"2024-04-02T09:13:41.940440+0000","last_clean":"2024-04-02T09:13:41.940440+0000","last_became_active":"2024-04-02T09:13:27.882217+0000","last_became_peered":"2024-04-02T09:13:27.882217+0000","last_unstale":"2024-04-02T09:13:41.940440+0000","last_undegraded":"2024-04-02T09:13:41.940440+0000","last_fullsized":"2024-04-02T09:13:41.940440+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:22:56.505431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.734928+0000","last_change":"2024-04-02T09:13:41.958277+0000","last_active":"2024-04-02T09:13:43.734928+0000","last_peered":"2024-04-02T09:13:43.734928+0000","last_clean":"2024-04-02T09:13:43.734928+0000","last_became_active":"2024-04-02T09:13:41.956555+0000","last_became_peered":"2024-04-02T09:13:41.956555+0000","last_unstale":"2024-04-02T09:13:43.734928+0000","last_undegraded":"2024-04-02T09:13:43.734928+0000","last_fullsized":"2024-04-02T09:13:43.734928+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:00:46.027233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940350+0000","last_change":"2024-04-02T09:13:28.381355+0000","last_active":"2024-04-02T09:13:41.940350+0000","last_peered":"2024-04-02T09:13:41.940350+0000","last_clean":"2024-04-02T09:13:41.940350+0000","last_became_active":"2024-04-02T09:13:28.381068+0000","last_became_peered":"2024-04-02T09:13:28.381068+0000","last_unstale":"2024-04-02T09:13:41.940350+0000","last_undegraded":"2024-04-02T09:13:41.940350+0000","last_fullsized":"2024-04-02T09:13:41.940350+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:26:50.895638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.953078+0000","last_change":"2024-04-02T09:13:27.883425+0000","last_active":"2024-04-02T09:13:41.953078+0000","last_peered":"2024-04-02T09:13:41.953078+0000","last_clean":"2024-04-02T09:13:41.953078+0000","last_became_active":"2024-04-02T09:13:27.883106+0000","last_became_peered":"2024-04-02T09:13:27.883106+0000","last_unstale":"2024-04-02T09:13:41.953078+0000","last_undegraded":"2024-04-02T09:13:41.953078+0000","last_fullsized":"2024-04-02T09:13:41.953078+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:29:23.271534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":10,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-04-02T09:13:43.733563+0000","last_change":"2024-04-02T09:13:41.960258+0000","last_active":"2024-04-02T09:13:43.733563+0000","last_peered":"2024-04-02T09:13:43.733563+0000","last_clean":"2024-04-02T09:13:43.733563+0000","last_became_active":"2024-04-02T09:13:41.960011+0000","last_became_peered":"2024-04-02T09:13:41.960011+0000","last_unstale":"2024-04-02T09:13:43.733563+0000","last_undegraded":"2024-04-02T09:13:43.733563+0000","last_fullsized":"2024-04-02T09:13:43.733563+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:41.418587+0000","last_clean_scrub_stamp":"2024-04-02T09:13:41.418587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:18.838465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.940567+0000","last_change":"2024-04-02T09:13:28.402819+0000","last_active":"2024-04-02T09:13:41.940567+0000","last_peered":"2024-04-02T09:13:41.940567+0000","last_clean":"2024-04-02T09:13:41.940567+0000","last_became_active":"2024-04-02T09:13:28.402563+0000","last_became_peered":"2024-04-02T09:13:28.402563+0000","last_unstale":"2024-04-02T09:13:41.940567+0000","last_undegraded":"2024-04-02T09:13:41.940567+0000","last_fullsized":"2024-04-02T09:13:41.940567+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:27.856210+0000","last_clean_scrub_stamp":"2024-04-02T09:13:27.856210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:09:27.049697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1f","version":"243'4","reported_seq":19,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.943081+0000","last_change":"2024-04-02T09:13:27.883417+0000","last_active":"2024-04-02T09:13:41.943081+0000","last_peered":"2024-04-02T09:13:41.943081+0000","last_clean":"2024-04-02T09:13:41.943081+0000","last_became_active":"2024-04-02T09:13:27.882944+0000","last_became_peered":"2024-04-02T09:13:27.882944+0000","last_unstale":"2024-04-02T09:13:41.943081+0000","last_undegraded":"2024-04-02T09:13:41.943081+0000","last_fullsized":"2024-04-02T09:13:41.943081+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:13:26.848564+0000","last_clean_scrub_stamp":"2024-04-02T09:13:26.848564+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:51:05.331734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"229'370","reported_seq":1312,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-04-02T09:13:41.948041+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:13:41.948041+0000","last_peered":"2024-04-02T09:13:41.948041+0000","last_clean":"2024-04-02T09:13:41.948041+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:13:41.948041+0000","last_undegraded":"2024-04-02T09:13:41.948041+0000","last_fullsized":"2024-04-02T09:13:41.948041+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3603,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576002,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60436,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311404,"statfs":{"total":96636764160,"available":96574877696,"internally_reserved":0,"allocated":3153920,"data_stored":1604364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.40400000000000003,"5min":0.309,"15min":0.309},"max":{"1min":8.016,"5min":8.016,"15min":8.016},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.71999999999999997,"15min":0.72999999999999998},"min":{"1min":0.39900000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.1790000000000003,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.51800000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.76800000000000002,"15min":0.751},"min":{"1min":0.44800000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.5819999999999999,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.70899999999999996,"15min":0.71999999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.42399999999999999},"max":{"1min":4.2569999999999997,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.60299999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.79600000000000004,"15min":0.77800000000000002},"min":{"1min":0.437,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.9790000000000001,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.80700000000000005,"15min":0.81200000000000006},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":4.8819999999999997,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.63400000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.77100000000000002,"15min":0.75900000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.0530000000000008,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.999,"5min":0.79900000000000004,"15min":0.77700000000000002},"min":{"1min":0.54100000000000004,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":5.093,"5min":5.093,"15min":5.093},"last":0.71999999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.66700000000000004,"15min":0.628},"min":{"1min":0.38300000000000001,"5min":0.38100000000000001,"15min":0.36499999999999999},"max":{"1min":5.4589999999999996,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.1060000000000001,"5min":0.79700000000000004,"15min":0.74399999999999999},"min":{"1min":0.50600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.9409999999999998,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.70099999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.73799999999999999,"15min":0.69299999999999995},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":5.1340000000000003,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.754},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.77400000000000002,"15min":0.73599999999999999},"min":{"1min":0.58899999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.492,"5min":3.492,"15min":3.492},"last":0.54000000000000004}]},{"osd":6,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.72199999999999998,"15min":0.69699999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.45800000000000002},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.075,"5min":0.78500000000000003,"15min":0.71699999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41099999999999998},"max":{"1min":7.8899999999999997,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.73799999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576000,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42368,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329472,"statfs":{"total":96636764160,"available":96593379328,"internally_reserved":0,"allocated":3129344,"data_stored":1600437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.69799999999999995,"15min":0.72899999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.90600000000000003,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.71099999999999997,"15min":0.73899999999999999},"min":{"1min":0.53600000000000003,"5min":0.47899999999999998,"15min":0.45800000000000002},"max":{"1min":0.88300000000000001,"5min":0.93200000000000005,"15min":1.0600000000000001},"last":0.78700000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68799999999999994,"15min":0.71299999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86299999999999999,"5min":1.044,"15min":1.212},"last":0.745},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.71299999999999997,"15min":0.752},"min":{"1min":0.45800000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.018,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.71799999999999997,"15min":0.747},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.046,"5min":1.046,"15min":1.2789999999999999},"last":0.754},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.71999999999999997,"15min":0.73899999999999999},"min":{"1min":0.44,"5min":0.439,"15min":0.439},"max":{"1min":0.93500000000000005,"5min":0.95999999999999996,"15min":1.226},"last":0.70799999999999996}]},{"osd":3,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72399999999999998,"15min":0.76300000000000001},"min":{"1min":0.53900000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97799999999999998,"5min":1.141,"15min":1.2549999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.755,"15min":0.77000000000000002},"min":{"1min":0.51000000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.67100000000000004}]},{"osd":4,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.60999999999999999,"15min":0.624},"min":{"1min":0.442,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.91200000000000003,"5min":0.96699999999999997,"15min":1.1259999999999999},"last":0.625},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.41999999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.1879999999999999},"last":0.57399999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84199999999999997,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69999999999999996,"15min":0.72499999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":0.96399999999999997,"5min":1.034,"15min":1.204},"last":0.73599999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.72999999999999998,"15min":0.74099999999999999},"min":{"1min":0.53600000000000003,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.012,"5min":1.21,"15min":1.2370000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72599999999999998,"15min":0.73899999999999999},"min":{"1min":0.52400000000000002,"5min":0.51400000000000001,"15min":0.49399999999999999},"max":{"1min":0.91900000000000004,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.79900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834576001,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52812,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319028,"statfs":{"total":96636764160,"available":96582684672,"internally_reserved":0,"allocated":3727360,"data_stored":2191299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.433,"15min":0.45000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28199999999999997,"15min":0.27400000000000002},"max":{"1min":0.58999999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48999999999999999,"15min":0.50900000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73199999999999998,"5min":0.76000000000000001,"15min":0.79800000000000004},"last":0.38800000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.504,"15min":0.505},"min":{"1min":0.38900000000000001,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56699999999999995,"15min":0.59199999999999997},"min":{"1min":0.44500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68100000000000005,"5min":0.77500000000000002,"15min":0.85599999999999998},"last":0.49399999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.41099999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.69999999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.55200000000000005,"15min":0.56100000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.58899999999999997,"5min":0.86699999999999999,"15min":0.878},"last":0.56699999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76900000000000002,"15min":0.78400000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.433},"max":{"1min":0.90700000000000003,"5min":1.1120000000000001,"15min":1.1479999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.76400000000000001,"15min":0.78500000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.41799999999999998},"max":{"1min":0.95699999999999996,"5min":1.05,"15min":1.25},"last":0.78300000000000003}]},{"osd":5,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76100000000000001,"15min":0.78800000000000003},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.96399999999999997,"5min":1.115,"15min":1.321},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.79700000000000004,"15min":0.80700000000000005},"min":{"1min":0.56899999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93799999999999994,"5min":1.0980000000000001,"15min":1.173},"last":0.85599999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.79100000000000004,"15min":0.81299999999999994},"min":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.48099999999999998},"max":{"1min":0.96099999999999997,"5min":1.159,"15min":1.2210000000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.80800000000000005,"15min":0.80900000000000005},"min":{"1min":0.56799999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0009999999999999,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.81100000000000005}]},{"osd":7,"last update":"Tue Apr 2 09:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.79900000000000004,"15min":0.81999999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0029999999999999,"5min":1.1080000000000001,"15min":1.236},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81899999999999995,"15min":0.83399999999999996},"min":{"1min":0.60899999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":0.97499999999999998,"5min":1.0720000000000001,"15min":1.2789999999999999},"last":0.90700000000000003}]}]},{"osd":0,"up_from":13,"seq":55834576001,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52220,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319620,"statfs":{"total":96636764160,"available":96583290880,"internally_reserved":0,"allocated":3129344,"data_stored":1599420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.441,"15min":0.45500000000000002},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.66700000000000004,"5min":0.69899999999999995,"15min":0.86699999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.49399999999999999,"15min":0.50700000000000001},"min":{"1min":0.33900000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64800000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.39300000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.502,"15min":0.51100000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.82699999999999996},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53400000000000003,"15min":0.55500000000000005},"min":{"1min":0.38300000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72699999999999998,"5min":0.999,"15min":0.999},"last":0.51000000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54200000000000004,"15min":0.56699999999999995},"min":{"1min":0.35699999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.71899999999999997,"5min":0.79800000000000004,"15min":0.85699999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.61599999999999999,"15min":0.63800000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.73599999999999999,"5min":0.92000000000000004,"15min":0.96399999999999997},"last":0.48399999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.80500000000000005,"15min":0.83099999999999996},"min":{"1min":0.55000000000000004,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":0.97499999999999998,"5min":1.1950000000000001,"15min":1.2769999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.78500000000000003,"15min":0.82199999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99299999999999999,"5min":1.169,"15min":1.169},"last":0.63700000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.77900000000000003,"15min":0.82399999999999995},"min":{"1min":0.54000000000000004,"5min":0.497,"15min":0.497},"max":{"1min":1,"5min":1.145,"15min":1.232},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.96599999999999997,"5min":1.21,"15min":1.21},"last":0.85199999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.77600000000000002,"15min":0.79800000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91100000000000003,"5min":1.1830000000000001,"15min":1.196},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.79200000000000004,"15min":0.81200000000000006},"min":{"1min":0.56499999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96199999999999997,"5min":1.246,"15min":1.246},"last":0.81799999999999995}]},{"osd":7,"last update":"Tue Apr 2 09:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82699999999999996,"15min":0.85499999999999998},"min":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0069999999999999,"5min":1.292,"15min":1.292},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.84799999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.95499999999999996,"5min":1.1719999999999999,"15min":1.2589999999999999},"last":0.65600000000000003}]}]},{"osd":2,"up_from":13,"seq":55834576000,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52236,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319604,"statfs":{"total":96636764160,"available":96583274496,"internally_reserved":0,"allocated":3141632,"data_stored":1601913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.44600000000000001,"15min":0.44},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.72999999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.48399999999999999,"15min":0.498},"min":{"1min":0.29199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63100000000000001,"5min":0.79900000000000004,"15min":1.4930000000000001},"last":0.31}]},{"osd":1,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.504},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69699999999999995,"5min":0.75800000000000001,"15min":0.80600000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53600000000000003,"15min":0.55400000000000005},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.59599999999999997,"5min":0.75,"15min":1.4530000000000001},"last":0.43099999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52900000000000003,"15min":0.55700000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.745,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60499999999999998,"15min":0.63200000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.64000000000000001,"5min":0.83699999999999997,"15min":1.0740000000000001},"last":0.46899999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.82099999999999995,"5min":1.121,"15min":1.121},"last":0.753},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.77300000000000002,"15min":0.79100000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.96799999999999997,"5min":1.179,"15min":1.5329999999999999},"last":0.74299999999999999}]},{"osd":5,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.78400000000000003,"15min":0.78600000000000003},"min":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.82999999999999996,"5min":1.99,"15min":1.99},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.78500000000000003,"15min":0.79900000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.89600000000000002,"5min":1.101,"15min":1.1259999999999999},"last":0.65800000000000003}]},{"osd":6,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.997,"5min":1.173,"15min":1.5940000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":0.80400000000000005},"min":{"1min":0.56200000000000006,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.92300000000000004,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.83099999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.877,"5min":1.2110000000000001,"15min":1.5649999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.80700000000000005,"15min":0.82699999999999996},"min":{"1min":0.61499999999999999,"5min":0.497,"15min":0.497},"max":{"1min":0.95899999999999996,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.80600000000000005}]}]},{"osd":3,"up_from":13,"seq":55834576000,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52232,"kb_used_data":3064,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319608,"statfs":{"total":96636764160,"available":96583278592,"internally_reserved":0,"allocated":3137536,"data_stored":1603096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.42799999999999999,"15min":0.44600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57199999999999995,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.46899999999999997,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60199999999999998,"5min":0.71199999999999997,"15min":0.876},"last":0.58699999999999997}]},{"osd":1,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48699999999999999,"15min":0.499},"min":{"1min":0.36799999999999999,"5min":0.33200000000000002,"15min":0.32500000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.90300000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55100000000000005,"15min":0.57199999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67700000000000005,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.66100000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.60099999999999998,"15min":0.626},"min":{"1min":0.40100000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71399999999999997,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.58799999999999997,"15min":0.61599999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.69499999999999995,"5min":0.81699999999999995,"15min":1.0840000000000001},"last":0.72999999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.72299999999999998,"15min":0.752},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89100000000000001,"5min":1.071,"15min":1.0800000000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.74299999999999999,"15min":0.76800000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.879,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.90000000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.74399999999999999,"15min":0.77500000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91800000000000004,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.76300000000000001,"15min":0.78100000000000003},"min":{"1min":0.54200000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.85699999999999998,"5min":1.266,"15min":1.266},"last":0.86099999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.76500000000000001,"15min":0.78400000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97099999999999997,"5min":1.284,"15min":1.284},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.54800000000000004,"5min":0.495,"15min":0.495},"max":{"1min":0.95999999999999996,"5min":1.2,"15min":1.2},"last":0.98799999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.77900000000000003,"15min":0.80800000000000005},"min":{"1min":0.58699999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.93500000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.79300000000000004,"15min":0.81799999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.014,"5min":1.248,"15min":1.248},"last":1.002}]}]},{"osd":4,"up_from":13,"seq":55834576000,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48136,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323704,"statfs":{"total":96636764160,"available":96587472896,"internally_reserved":0,"allocated":3141632,"data_stored":1602425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.66400000000000003,"15min":0.68700000000000006},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":0.80000000000000004,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.65100000000000002,"15min":0.67700000000000005},"min":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.78100000000000003,"5min":0.86199999999999999,"15min":1.075},"last":0.97099999999999997}]},{"osd":1,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.72199999999999998,"15min":0.754},"min":{"1min":0.49399999999999999,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":0.90500000000000003,"5min":1.0549999999999999,"15min":1.069},"last":1.014},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.73599999999999999,"15min":0.76500000000000001},"min":{"1min":0.46100000000000002,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":0.92500000000000004,"5min":1.048,"15min":1.048},"last":1.0620000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.67300000000000004,"15min":0.70799999999999996},"min":{"1min":0.47099999999999997,"5min":0.438,"15min":0.438},"max":{"1min":0.81299999999999994,"5min":1.077,"15min":1.077},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.68999999999999995,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85599999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":1.032}]},{"osd":3,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.51500000000000001,"5min":0.501,"15min":0.47399999999999998},"max":{"1min":0.94599999999999995,"5min":0.998,"15min":1.018},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.72999999999999998,"15min":0.752},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":0.81299999999999994,"5min":1.002,"15min":1.028},"last":1.0489999999999999}]},{"osd":5,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.81899999999999995,"5min":1.008,"15min":1.008},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.71799999999999997}]},{"osd":6,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65500000000000003,"15min":0.64400000000000002},"min":{"1min":0.48499999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85099999999999998,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68999999999999995,"15min":0.69199999999999995},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":1.0049999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.99399999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.48399999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96099999999999997,"5min":1.002,"15min":1.0329999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.77500000000000002,"15min":0.78100000000000003},"min":{"1min":0.56100000000000005,"5min":0.5,"15min":0.5},"max":{"1min":1.038,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":1.0700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576003,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52796,"kb_used_data":3632,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319044,"statfs":{"total":96636764160,"available":96582701056,"internally_reserved":0,"allocated":3719168,"data_stored":2189730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.82699999999999996,"5min":0.98699999999999999,"15min":1.004},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.68999999999999995,"15min":0.71999999999999997},"min":{"1min":0.48399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.028,"5min":1.085,"15min":1.085},"last":0.55300000000000005}]},{"osd":1,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.68100000000000005,"15min":0.71499999999999997},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84399999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70999999999999996,"15min":0.72499999999999998},"min":{"1min":0.52300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88500000000000001,"5min":0.95299999999999996,"15min":1.052},"last":0.78300000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.71599999999999997,"15min":0.73799999999999999},"min":{"1min":0.47199999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97299999999999998,"5min":1.004,"15min":1.014},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.69499999999999995,"15min":0.73699999999999999},"min":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.81799999999999995,"5min":1.018,"15min":1.0680000000000001},"last":0.61499999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.747,"15min":0.77700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.88500000000000001,"5min":0.98899999999999999,"15min":1.083},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77000000000000002,"15min":0.79000000000000004},"min":{"1min":0.58399999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1020000000000001},"last":0.67400000000000004}]},{"osd":4,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.40699999999999997,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.054},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":0.98799999999999999,"5min":1.145,"15min":1.145},"last":0.76300000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67900000000000005,"15min":0.67300000000000004},"min":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96799999999999997,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71399999999999997,"15min":0.71099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.94899999999999995,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.84899999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73999999999999999,"15min":0.73799999999999999},"min":{"1min":0.46500000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.0820000000000001,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76300000000000001,"15min":0.76600000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1519999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.83299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:13:52.256 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T09:13:52.257 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:13:52.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.480+0000 7f18d8e00700 1 Processor -- start 2024-04-02T09:13:52.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.480+0000 7f18d8e00700 1 -- start start 2024-04-02T09:13:52.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.481+0000 7f18d8e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f18d40d18b0 0x7f18d40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:52.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.481+0000 7f18d8e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f18d4161590 0x7f18d4161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:52.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.481+0000 7f18d8e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d40ceda0 con 0x7f18d4160f10 2024-04-02T09:13:52.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.481+0000 7f18d8e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d40d1d30 con 0x7f18d4161590 2024-04-02T09:13:52.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.481+0000 7f18d8e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d4161f60 con 0x7f18d40d18b0 2024-04-02T09:13:52.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.481+0000 7f18d1e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f18d4160f10 0x7f18d40d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39454/0 (socket says 172.21.15.89:39454) 2024-04-02T09:13:52.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.481+0000 7f18d1e00700 1 -- 172.21.15.89:0/3042407704 learned_addr learned my addr 172.21.15.89:0/3042407704 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:52.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.481+0000 7f18d2800700 1 --2- 172.21.15.89:0/3042407704 >> v2:172.21.15.89:3300/0 conn(0x7f18d4161590 0x7f18d4161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:52.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.482+0000 7f18d1400700 1 --2- 172.21.15.89:0/3042407704 >> v2:172.21.15.113:3300/0 conn(0x7f18d40d18b0 0x7f18d40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:52.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.482+0000 7f18d0a00700 1 -- 172.21.15.89:0/3042407704 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534425800 0 0) 0x7f18d40ceda0 con 0x7f18d4160f10 2024-04-02T09:13:52.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.482+0000 7f18d0a00700 1 -- 172.21.15.89:0/3042407704 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18b80035b0 con 0x7f18d4160f10 2024-04-02T09:13:52.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.482+0000 7f18d2800700 1 -- 172.21.15.89:0/3042407704 >> v2:172.21.15.113:3300/0 conn(0x7f18d40d18b0 msgr2=0x7f18d40cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:52.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.482+0000 7f18d2800700 1 --2- 172.21.15.89:0/3042407704 >> v2:172.21.15.113:3300/0 conn(0x7f18d40d18b0 0x7f18d40cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:52.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.482+0000 7f18d2800700 1 -- 172.21.15.89:0/3042407704 >> v1:172.21.15.89:6789/0 conn(0x7f18d4160f10 legacy=0x7f18d40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:52.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.482+0000 7f18d2800700 1 -- 172.21.15.89:0/3042407704 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d41612f0 con 0x7f18d4161590 2024-04-02T09:13:52.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.482+0000 7f18d1400700 1 --2- 172.21.15.89:0/3042407704 >> v2:172.21.15.113:3300/0 conn(0x7f18d40d18b0 0x7f18d40cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:52.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.482+0000 7f18d2800700 1 --2- 172.21.15.89:0/3042407704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18d4161590 0x7f18d4161a20 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f18d408ca30 tx=0x7f18cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca806f7ec2315645 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:52.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.483+0000 7f18d0a00700 1 -- 172.21.15.89:0/3042407704 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc00e070 con 0x7f18d4161590 2024-04-02T09:13:52.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.483+0000 7f18d0a00700 1 -- 172.21.15.89:0/3042407704 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18cc00bb20 con 0x7f18d4161590 2024-04-02T09:13:52.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.483+0000 7f18d0a00700 1 -- 172.21.15.89:0/3042407704 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc012430 con 0x7f18d4161590 2024-04-02T09:13:52.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.483+0000 7f18d8e00700 1 -- 172.21.15.89:0/3042407704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18d4161590 msgr2=0x7f18d4161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:52.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.483+0000 7f18d8e00700 1 --2- 172.21.15.89:0/3042407704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18d4161590 0x7f18d4161a20 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f18d408ca30 tx=0x7f18cc00b330 comp rx=0 tx=0).stop 2024-04-02T09:13:52.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.483+0000 7f18d8e00700 1 -- 172.21.15.89:0/3042407704 shutdown_connections 2024-04-02T09:13:52.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.483+0000 7f18d8e00700 1 --2- 172.21.15.89:0/3042407704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18d4161590 0x7f18d4161a20 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:52.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.483+0000 7f18d8e00700 1 --2- 172.21.15.89:0/3042407704 >> v2:172.21.15.113:3300/0 conn(0x7f18d40d18b0 0x7f18d40cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:52.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.483+0000 7f18d8e00700 1 -- 172.21.15.89:0/3042407704 >> 172.21.15.89:0/3042407704 conn(0x7f18d408aff0 msgr2=0x7f18d40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:52.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.483+0000 7f18d8e00700 1 -- 172.21.15.89:0/3042407704 shutdown_connections 2024-04-02T09:13:52.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.483+0000 7f18d8e00700 1 -- 172.21.15.89:0/3042407704 wait complete. 2024-04-02T09:13:52.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.485+0000 7f18d8e00700 1 Processor -- start 2024-04-02T09:13:52.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.485+0000 7f18d8e00700 1 -- start start 2024-04-02T09:13:52.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.486+0000 7f18d8e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18d40d18b0 0x7f18d41553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:52.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.486+0000 7f18d8e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f18d4160f10 0x7f18d41e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:52.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.486+0000 7f18d8e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18d4161590 0x7f18d406c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:52.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.486+0000 7f18d8e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d41e86e0 con 0x7f18d40d18b0 2024-04-02T09:13:52.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.486+0000 7f18d8e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d41e89c0 con 0x7f18d4161590 2024-04-02T09:13:52.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.486+0000 7f18d8e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d41e8ca0 con 0x7f18d4160f10 2024-04-02T09:13:52.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.486+0000 7f18d1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18d40d18b0 0x7f18d41553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:52.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.486+0000 7f18d1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18d40d18b0 0x7f18d41553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59238/0 (socket says 172.21.15.89:59238) 2024-04-02T09:13:52.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.486+0000 7f18d1e00700 1 -- 172.21.15.89:0/459364552 learned_addr learned my addr 172.21.15.89:0/459364552 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:52.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.486+0000 7f18d1400700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f18d4160f10 0x7f18d41e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:52.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.486+0000 7f18d1e00700 1 -- 172.21.15.89:0/459364552 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f18d4160f10 msgr2=0x7f18d41e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:52.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.486+0000 7f18d1e00700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f18d4160f10 0x7f18d41e7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:52.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.486+0000 7f18d1e00700 1 -- 172.21.15.89:0/459364552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18d4161590 msgr2=0x7f18d406c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:52.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.486+0000 7f18d2800700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18d4161590 0x7f18d406c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:52.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.487+0000 7f18d1e00700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18d4161590 0x7f18d406c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:52.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.487+0000 7f18d1e00700 1 -- 172.21.15.89:0/459364552 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18cc009a70 con 0x7f18d40d18b0 2024-04-02T09:13:52.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.487+0000 7f18d1400700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f18d4160f10 0x7f18d41e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:52.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.487+0000 7f18d2800700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18d4161590 0x7f18d406c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:52.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.487+0000 7f18d1e00700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18d40d18b0 0x7f18d41553c0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7f18c4001eb0 tx=0x7f18c400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:52.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.487+0000 7f18ca000700 1 -- 172.21.15.89:0/459364552 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4005d70 con 0x7f18d40d18b0 2024-04-02T09:13:52.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.487+0000 7f18ca000700 1 -- 172.21.15.89:0/459364552 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c40070d0 con 0x7f18d40d18b0 2024-04-02T09:13:52.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.487+0000 7f18d8e00700 1 -- 172.21.15.89:0/459364552 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d41e8f80 con 0x7f18d40d18b0 2024-04-02T09:13:52.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.487+0000 7f18ca000700 1 -- 172.21.15.89:0/459364552 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4007460 con 0x7f18d40d18b0 2024-04-02T09:13:52.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.487+0000 7f18d8e00700 1 -- 172.21.15.89:0/459364552 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d41566c0 con 0x7f18d40d18b0 2024-04-02T09:13:52.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.488+0000 7f18d8e00700 1 -- 172.21.15.89:0/459364552 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d4067720 con 0x7f18d40d18b0 2024-04-02T09:13:52.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.489+0000 7f18ca000700 1 -- 172.21.15.89:0/459364552 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f18c40075c0 con 0x7f18d40d18b0 2024-04-02T09:13:52.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.491+0000 7f18ca000700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f18b406aed0 0x7f18b406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:52.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.491+0000 7f18ca000700 1 -- 172.21.15.89:0/459364552 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f18c40891c0 con 0x7f18d40d18b0 2024-04-02T09:13:52.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.491+0000 7f18d1400700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f18b406aed0 0x7f18b406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:52.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.492+0000 7f18d1400700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f18b406aed0 0x7f18b406d390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f18c0009820 tx=0x7f18c0008040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:52.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.495+0000 7f18ca000700 1 -- 172.21.15.89:0/459364552 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f18c4053b10 con 0x7f18d40d18b0 2024-04-02T09:13:52.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.744+0000 7f18d8e00700 1 -- 172.21.15.89:0/459364552 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18d41564a0 con 0x7f18d40d18b0 2024-04-02T09:13:52.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.744+0000 7f18ca000700 1 -- 172.21.15.89:0/459364552 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5968 (secure 0 0 0) 0x7f18c4057130 con 0x7f18d40d18b0 2024-04-02T09:13:52.743 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:13:52.743 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":125,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9518,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3094781297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3094781297},{"type":"v1","addr":"172.21.15.89:6836","nonce":3094781297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":10978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1799351491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1799351491},{"type":"v1","addr":"172.21.15.113:6834","nonce":1799351491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:28.357996+0000","modified":"2024-04-02T09:13:31.369015+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11964":{"gid":11964,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6835/703852199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":703852199},{"type":"v1","addr":"172.21.15.113:6835","nonce":703852199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:41.933360+0000","modified":"2024-04-02T09:13:45.439462+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11958},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11958":{"gid":11958,"name":"a","rank":0,"incarnation":121,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6837/3112580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3112580185},{"type":"v1","addr":"172.21.15.89:6837","nonce":3112580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-02T09:13:52.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.747+0000 7f18d8e00700 1 -- 172.21.15.89:0/459364552 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f18b406aed0 msgr2=0x7f18b406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:52.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.747+0000 7f18d8e00700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f18b406aed0 0x7f18b406d390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f18c0009820 tx=0x7f18c0008040 comp rx=0 tx=0).stop 2024-04-02T09:13:52.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.747+0000 7f18d8e00700 1 -- 172.21.15.89:0/459364552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18d40d18b0 msgr2=0x7f18d41553c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:52.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.747+0000 7f18d8e00700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18d40d18b0 0x7f18d41553c0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7f18c4001eb0 tx=0x7f18c400c860 comp rx=0 tx=0).stop 2024-04-02T09:13:52.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.747+0000 7f18d8e00700 1 -- 172.21.15.89:0/459364552 shutdown_connections 2024-04-02T09:13:52.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.747+0000 7f18d8e00700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f18b406aed0 0x7f18b406d390 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:52.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.747+0000 7f18d8e00700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18d4161590 0x7f18d406c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:52.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.747+0000 7f18d8e00700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18d40d18b0 0x7f18d41553c0 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:52.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.747+0000 7f18d8e00700 1 --2- 172.21.15.89:0/459364552 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f18d4160f10 0x7f18d41e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:52.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.747+0000 7f18d8e00700 1 -- 172.21.15.89:0/459364552 >> 172.21.15.89:0/459364552 conn(0x7f18d408aff0 msgr2=0x7f18d40cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:52.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.747+0000 7f18d8e00700 1 -- 172.21.15.89:0/459364552 shutdown_connections 2024-04-02T09:13:52.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:52.747+0000 7f18d8e00700 1 -- 172.21.15.89:0/459364552 wait complete. 2024-04-02T09:13:52.747 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 125 2024-04-02T09:13:53.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 125, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T09:13:41.933360+0000', 'modified': '2024-04-02T09:13:45.439462+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11958}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11958': {'gid': 11958, 'name': 'a', 'rank': 0, 'incarnation': 121, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.89:6837/3112580185', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6834', 'nonce': 3112580185}, {'type': 'v1', 'addr': '172.21.15.89:6837', 'nonce': 3112580185}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T09:13:53.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T09:13:53.167 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T09:13:53.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.362+0000 7f4983e00700 1 Processor -- start 2024-04-02T09:13:53.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.363+0000 7f4983e00700 1 -- start start 2024-04-02T09:13:53.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.364+0000 7f4983e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f497c15dad0 0x7f497c0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.364+0000 7f4983e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f497c0d1630 0x7f497c0d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.364+0000 7f4983e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497c15f820 con 0x7f497c0d1fd0 2024-04-02T09:13:53.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.364+0000 7f4983e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f497c0cfcc0 con 0x7f497c15dad0 2024-04-02T09:13:53.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.364+0000 7f4983e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497c0cd9f0 con 0x7f497c0d1630 2024-04-02T09:13:53.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.364+0000 7f4981400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f497c15dad0 0x7f497c0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.364+0000 7f4981400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f497c15dad0 0x7f497c0ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59240/0 (socket says 172.21.15.89:59240) 2024-04-02T09:13:53.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.364+0000 7f4981400700 1 -- 172.21.15.89:0/199675544 learned_addr learned my addr 172.21.15.89:0/199675544 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:53.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.364+0000 7f4981400700 1 -- 172.21.15.89:0/199675544 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f497c0d1630 msgr2=0x7f497c0d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:53.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.364+0000 7f4980a00700 1 --2- 172.21.15.89:0/199675544 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f497c0d1630 0x7f497c0d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.364+0000 7f4981400700 1 --2- 172.21.15.89:0/199675544 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f497c0d1630 0x7f497c0d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.365+0000 7f4981400700 1 -- 172.21.15.89:0/199675544 >> v1:172.21.15.113:6789/0 conn(0x7f497c0d1fd0 legacy=0x7f497c15deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:53.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.365+0000 7f4981400700 1 -- 172.21.15.89:0/199675544 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c0d2460 con 0x7f497c15dad0 2024-04-02T09:13:53.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.365+0000 7f4980a00700 1 --2- 172.21.15.89:0/199675544 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f497c0d1630 0x7f497c0d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:13:53.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.365+0000 7f4981400700 1 --2- 172.21.15.89:0/199675544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f497c15dad0 0x7f497c0ced80 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7f49700098f0 tx=0x7f4970009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97bea3666ae524b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:53.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.365+0000 7f497be00700 1 -- 172.21.15.89:0/199675544 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4970004230 con 0x7f497c15dad0 2024-04-02T09:13:53.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.365+0000 7f497be00700 1 -- 172.21.15.89:0/199675544 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4970004390 con 0x7f497c15dad0 2024-04-02T09:13:53.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.365+0000 7f497be00700 1 -- 172.21.15.89:0/199675544 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49700036a0 con 0x7f497c15dad0 2024-04-02T09:13:53.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.365+0000 7f4983e00700 1 -- 172.21.15.89:0/199675544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f497c15dad0 msgr2=0x7f497c0ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:53.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.366+0000 7f4983e00700 1 --2- 172.21.15.89:0/199675544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f497c15dad0 0x7f497c0ced80 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7f49700098f0 tx=0x7f4970009c00 comp rx=0 tx=0).stop 2024-04-02T09:13:53.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.366+0000 7f4983e00700 1 -- 172.21.15.89:0/199675544 shutdown_connections 2024-04-02T09:13:53.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.366+0000 7f4983e00700 1 --2- 172.21.15.89:0/199675544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f497c15dad0 0x7f497c0ced80 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.366+0000 7f4983e00700 1 --2- 172.21.15.89:0/199675544 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f497c0d1630 0x7f497c0d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.366+0000 7f4983e00700 1 -- 172.21.15.89:0/199675544 >> 172.21.15.89:0/199675544 conn(0x7f497c08aff0 msgr2=0x7f497c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:53.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.366+0000 7f4983e00700 1 -- 172.21.15.89:0/199675544 shutdown_connections 2024-04-02T09:13:53.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.366+0000 7f4983e00700 1 -- 172.21.15.89:0/199675544 wait complete. 2024-04-02T09:13:53.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.367+0000 7f4983e00700 1 Processor -- start 2024-04-02T09:13:53.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.367+0000 7f4983e00700 1 -- start start 2024-04-02T09:13:53.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.368+0000 7f4983e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f497c0d1630 0x7f497c1f20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.368+0000 7f4983e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f497c0d1fd0 0x7f497c1f2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.368+0000 7f4983e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f497c15dad0 0x7f497c1f6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.368+0000 7f4983e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c1f2b50 con 0x7f497c0d1fd0 2024-04-02T09:13:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.368+0000 7f4983e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c0c9fc0 con 0x7f497c15dad0 2024-04-02T09:13:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.368+0000 7f4983e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497c0b8ef0 con 0x7f497c0d1630 2024-04-02T09:13:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.369+0000 7f4980a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f497c0d1fd0 0x7f497c1f2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.369+0000 7f4980a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f497c0d1fd0 0x7f497c1f2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59244/0 (socket says 172.21.15.89:59244) 2024-04-02T09:13:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.369+0000 7f4980a00700 1 -- 172.21.15.89:0/6406400 learned_addr learned my addr 172.21.15.89:0/6406400 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.369+0000 7f4981400700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f497c0d1630 0x7f497c1f20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.369+0000 7f4981e00700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f497c15dad0 0x7f497c1f6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.369+0000 7f4980a00700 1 -- 172.21.15.89:0/6406400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f497c0d1630 msgr2=0x7f497c1f20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.369+0000 7f4980a00700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f497c0d1630 0x7f497c1f20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.369+0000 7f4980a00700 1 -- 172.21.15.89:0/6406400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f497c15dad0 msgr2=0x7f497c1f6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.369+0000 7f4980a00700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f497c15dad0 0x7f497c1f6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.369+0000 7f4980a00700 1 -- 172.21.15.89:0/6406400 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4970009550 con 0x7f497c0d1fd0 2024-04-02T09:13:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.369+0000 7f4981e00700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f497c15dad0 0x7f497c1f6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:13:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.369+0000 7f4980a00700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f497c0d1fd0 0x7f497c1f2610 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7f496800c750 tx=0x7f496800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.370+0000 7f4979600700 1 -- 172.21.15.89:0/6406400 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4968011800 con 0x7f497c0d1fd0 2024-04-02T09:13:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.370+0000 7f4979600700 1 -- 172.21.15.89:0/6406400 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4968011960 con 0x7f497c0d1fd0 2024-04-02T09:13:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.370+0000 7f4983e00700 1 -- 172.21.15.89:0/6406400 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497c0b91d0 con 0x7f497c0d1fd0 2024-04-02T09:13:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.370+0000 7f4979600700 1 -- 172.21.15.89:0/6406400 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4968011cd0 con 0x7f497c0d1fd0 2024-04-02T09:13:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.370+0000 7f4983e00700 1 -- 172.21.15.89:0/6406400 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f497c0c9c50 con 0x7f497c0d1fd0 2024-04-02T09:13:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.373+0000 7f4979600700 1 -- 172.21.15.89:0/6406400 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4968021350 con 0x7f497c0d1fd0 2024-04-02T09:13:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.373+0000 7f4979600700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f495c06aed0 0x7f495c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.374+0000 7f4979600700 1 -- 172.21.15.89:0/6406400 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f4968089ce0 con 0x7f497c0d1fd0 2024-04-02T09:13:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.374+0000 7f4981400700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f495c06aed0 0x7f495c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.374+0000 7f4981400700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f495c06aed0 0x7f495c06d390 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4970009130 tx=0x7f4970008030 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.375+0000 7f49979b7b80 1 Processor -- start 2024-04-02T09:13:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.375+0000 7f49979b7b80 1 -- start start 2024-04-02T09:13:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.376+0000 7f49979b7b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e552ffac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.376+0000 7f49979b7b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcdbe0 0x55e552fd0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.376+0000 7f49979b7b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcbb60 0x55e552ff7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.376+0000 7f49979b7b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e552f45650 con 0x55e552ffa830 2024-04-02T09:13:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.376+0000 7f49979b7b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e552f26c30 con 0x55e552fcbb60 2024-04-02T09:13:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.376+0000 7f49979b7b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e552f24430 con 0x55e552fcdbe0 2024-04-02T09:13:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.376+0000 7f4981400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e552ffac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.376+0000 7f4981400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e552ffac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59246/0 (socket says 172.21.15.89:59246) 2024-04-02T09:13:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.377+0000 7f4981400700 1 -- 172.21.15.89:0/1489121319 learned_addr learned my addr 172.21.15.89:0/1489121319 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.377+0000 7f4980a00700 1 --2- 172.21.15.89:0/1489121319 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcdbe0 0x55e552fd0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.377+0000 7f4981e00700 1 --2- 172.21.15.89:0/1489121319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcbb60 0x55e552ff7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.377+0000 7f4981400700 1 -- 172.21.15.89:0/1489121319 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcdbe0 msgr2=0x55e552fd0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.377+0000 7f4981400700 1 --2- 172.21.15.89:0/1489121319 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcdbe0 0x55e552fd0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.377+0000 7f4981400700 1 -- 172.21.15.89:0/1489121319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcbb60 msgr2=0x55e552ff7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.377+0000 7f4981400700 1 --2- 172.21.15.89:0/1489121319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcbb60 0x55e552ff7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.377+0000 7f4981400700 1 -- 172.21.15.89:0/1489121319 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e552d13790 con 0x55e552ffa830 2024-04-02T09:13:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.377+0000 7f4981400700 1 --2- 172.21.15.89:0/1489121319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e552ffac10 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f497c08ce40 tx=0x7f4970010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b17cbfde2464ba4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.377+0000 7f496dc00700 1 -- 172.21.15.89:0/1489121319 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497000b660 con 0x55e552ffa830 2024-04-02T09:13:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.377+0000 7f496dc00700 1 -- 172.21.15.89:0/1489121319 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497000b7c0 con 0x55e552ffa830 2024-04-02T09:13:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.377+0000 7f496dc00700 1 -- 172.21.15.89:0/1489121319 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497000bb70 con 0x55e552ffa830 2024-04-02T09:13:53.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.378+0000 7f49979b7b80 1 -- 172.21.15.89:0/1489121319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 msgr2=0x55e552ffac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:53.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.378+0000 7f49979b7b80 1 --2- 172.21.15.89:0/1489121319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e552ffac10 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f497c08ce40 tx=0x7f4970010000 comp rx=0 tx=0).stop 2024-04-02T09:13:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.378+0000 7f49979b7b80 1 -- 172.21.15.89:0/1489121319 shutdown_connections 2024-04-02T09:13:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.378+0000 7f49979b7b80 1 --2- 172.21.15.89:0/1489121319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcbb60 0x55e552ff7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.378+0000 7f49979b7b80 1 --2- 172.21.15.89:0/1489121319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e552ffac10 unknown :-1 s=CLOSED pgs=2625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.378+0000 7f49979b7b80 1 --2- 172.21.15.89:0/1489121319 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcdbe0 0x55e552fd0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.378+0000 7f49979b7b80 1 -- 172.21.15.89:0/1489121319 >> 172.21.15.89:0/1489121319 conn(0x55e552f2e6a0 msgr2=0x55e553015640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.378+0000 7f49979b7b80 1 -- 172.21.15.89:0/1489121319 shutdown_connections 2024-04-02T09:13:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.378+0000 7f49979b7b80 1 -- 172.21.15.89:0/1489121319 wait complete. 2024-04-02T09:13:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.379+0000 7f49979b7b80 1 -- start start 2024-04-02T09:13:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.379+0000 7f49979b7b80 1 Processor -- start 2024-04-02T09:13:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.380+0000 7f49979b7b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcbb60 0x55e552ff6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.380+0000 7f49979b7b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcdbe0 0x55e553028840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.380+0000 7f49979b7b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e552ffd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.380+0000 7f49979b7b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e552f1fd20 con 0x55e552ffa830 2024-04-02T09:13:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.380+0000 7f49979b7b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e552ebb770 con 0x55e552fcdbe0 2024-04-02T09:13:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.380+0000 7f49979b7b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e552ede6f0 con 0x55e552fcbb60 2024-04-02T09:13:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.380+0000 7f49979b7b80 10 client.0 _ll_register_callbacks cb 0x55e552ec55e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T09:13:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.380+0000 7f4981e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e552ffd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.380+0000 7f4981e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e552ffd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59254/0 (socket says 172.21.15.89:59254) 2024-04-02T09:13:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.380+0000 7f4981e00700 1 -- 172.21.15.89:0/3503317621 learned_addr learned my addr 172.21.15.89:0/3503317621 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.380+0000 7f4980a00700 1 --2- 172.21.15.89:0/3503317621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcdbe0 0x55e553028840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.380+0000 7f4981400700 1 --2- 172.21.15.89:0/3503317621 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcbb60 0x55e552ff6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.381+0000 7f4981e00700 1 -- 172.21.15.89:0/3503317621 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcbb60 msgr2=0x55e552ff6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.381+0000 7f4981e00700 1 --2- 172.21.15.89:0/3503317621 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcbb60 0x55e552ff6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.381+0000 7f4981e00700 1 -- 172.21.15.89:0/3503317621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcdbe0 msgr2=0x55e553028840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.381+0000 7f4981e00700 1 --2- 172.21.15.89:0/3503317621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcdbe0 0x55e553028840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.381+0000 7f4981e00700 1 -- 172.21.15.89:0/3503317621 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f497c0d2460 con 0x55e552ffa830 2024-04-02T09:13:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.381+0000 7f4981e00700 1 --2- 172.21.15.89:0/3503317621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e552ffd6c0 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f4974011d60 tx=0x7f497400f4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c7fdff0ecc526f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.381+0000 7f4962000700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T09:13:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.381+0000 7f4962000700 1 -- 172.21.15.89:0/3503317621 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4974005030 con 0x55e552ffa830 2024-04-02T09:13:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.381+0000 7f4962000700 1 -- 172.21.15.89:0/3503317621 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f4974005190 con 0x55e552ffa830 2024-04-02T09:13:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.381+0000 7f49979b7b80 1 -- 172.21.15.89:0/3503317621 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e552fc2920 con 0x55e552ffa830 2024-04-02T09:13:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.381+0000 7f4962000700 1 -- 172.21.15.89:0/3503317621 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4974005350 con 0x55e552ffa830 2024-04-02T09:13:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.381+0000 7f49979b7b80 10 client.12279 fetch_fsmap learned FSMap version 125 2024-04-02T09:13:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.381+0000 7f49979b7b80 1 -- 172.21.15.89:0/3503317621 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=125,monmap=2+}) v3 -- 0x55e552e000a0 con 0x55e552ffa830 2024-04-02T09:13:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.382+0000 7f4962000700 1 -- 172.21.15.89:0/3503317621 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e552fc2920 con 0x55e552ffa830 2024-04-02T09:13:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.382+0000 7f4962000700 1 -- 172.21.15.89:0/3503317621 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 125) v1 ==== 3681+0+0 (secure 0 0 0) 0x7f497400fa10 con 0x55e552ffa830 2024-04-02T09:13:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.382+0000 7f49979b7b80 10 client.12279 fetch_fsmap finished waiting for FSMap version 125 2024-04-02T09:13:53.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.382+0000 7f49979b7b80 10 client.12279 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T09:13:53.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.382+0000 7f49979b7b80 20 client.12279 populate_metadata read hostname 'smithi089' 2024-04-02T09:13:53.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.382+0000 7f49979b7b80 1 --2- 172.21.15.89:0/3503317621 >> [v2:172.21.15.89:6834/3112580185,v1:172.21.15.89:6837/3112580185] conn(0x55e552ffbed0 0x55e552ffc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.382+0000 7f49979b7b80 4 client.12279 mds_command: new command op to 11958 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T09:13:53.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.382+0000 7f49979b7b80 1 -- 172.21.15.89:0/3503317621 --> [v2:172.21.15.89:6834/3112580185,v1:172.21.15.89:6837/3112580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e552f9f6f0 con 0x55e552ffbed0 2024-04-02T09:13:53.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.382+0000 7f4981400700 1 --2- 172.21.15.89:0/3503317621 >> [v2:172.21.15.89:6834/3112580185,v1:172.21.15.89:6837/3112580185] conn(0x55e552ffbed0 0x55e552ffc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.383+0000 7f4962000700 10 client.12279 ms_handle_connect on v2:172.21.15.89:6834/3112580185 2024-04-02T09:13:53.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.383+0000 7f4981400700 1 --2- 172.21.15.89:0/3503317621 >> [v2:172.21.15.89:6834/3112580185,v1:172.21.15.89:6837/3112580185] conn(0x55e552ffbed0 0x55e552ffc2b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bece4fa23a65fd62 server_cookie=a8dddfca59aacf7 in_seq=0 out_seq=0 2024-04-02T09:13:53.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.383+0000 7f4962000700 1 -- 172.21.15.89:0/3503317621 <== mds.0 v2:172.21.15.89:6834/3112580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e552f9f6f0 con 0x55e552ffbed0 2024-04-02T09:13:53.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.383+0000 7f4962000700 10 client.12279 handle_command_reply: tid=0 2024-04-02T09:13:53.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.383+0000 7f49979b7b80 1 client.12279 shutdown 2024-04-02T09:13:53.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.384+0000 7f49979b7b80 1 -- 172.21.15.89:0/3503317621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 msgr2=0x55e552ffd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.384+0000 7f49979b7b80 1 --2- 172.21.15.89:0/3503317621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e552ffd6c0 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f4974011d60 tx=0x7f497400f4b0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.384+0000 7f49979b7b80 1 -- 172.21.15.89:0/3503317621 shutdown_connections 2024-04-02T09:13:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.384+0000 7f49979b7b80 1 --2- 172.21.15.89:0/3503317621 >> [v2:172.21.15.89:6834/3112580185,v1:172.21.15.89:6837/3112580185] conn(0x55e552ffbed0 0x55e552ffc2b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.384+0000 7f49979b7b80 1 --2- 172.21.15.89:0/3503317621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e552ffd6c0 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.384+0000 7f49979b7b80 1 --2- 172.21.15.89:0/3503317621 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcbb60 0x55e552ff6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.384+0000 7f49979b7b80 1 --2- 172.21.15.89:0/3503317621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcdbe0 0x55e553028840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.384+0000 7f49979b7b80 1 -- 172.21.15.89:0/3503317621 >> 172.21.15.89:0/3503317621 conn(0x55e552f2e6a0 msgr2=0x55e553018250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.384+0000 7f4962000700 0 client.12279 ms_handle_reset on v2:172.21.15.89:6834/3112580185 2024-04-02T09:13:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.384+0000 7f49979b7b80 1 -- 172.21.15.89:0/3503317621 shutdown_connections 2024-04-02T09:13:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.384+0000 7f49979b7b80 1 -- 172.21.15.89:0/3503317621 wait complete. 2024-04-02T09:13:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.384+0000 7f49979b7b80 20 client.12279 trim_cache size 0 max 16384 2024-04-02T09:13:53.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.402+0000 7f49979b7b80 1 Processor -- start 2024-04-02T09:13:53.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.402+0000 7f49979b7b80 1 -- start start 2024-04-02T09:13:53.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.403+0000 7f49979b7b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcbb60 0x55e552ffbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.403+0000 7f49979b7b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552fcdbe0 0x55e552ffc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.403+0000 7f49979b7b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552ffa830 0x55e553023ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.403+0000 7f49979b7b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e552d14390 con 0x55e552fcdbe0 2024-04-02T09:13:53.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.403+0000 7f49979b7b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e552ff96a0 con 0x55e552ffa830 2024-04-02T09:13:53.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.403+0000 7f49979b7b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e552fcc260 con 0x55e552fcbb60 2024-04-02T09:13:53.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.403+0000 7f4980a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552fcdbe0 0x55e552ffc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.403+0000 7f4980a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552fcdbe0 0x55e552ffc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59266/0 (socket says 172.21.15.89:59266) 2024-04-02T09:13:53.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.403+0000 7f4980a00700 1 -- 172.21.15.89:0/453525100 learned_addr learned my addr 172.21.15.89:0/453525100 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:53.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.403+0000 7f4980a00700 1 -- 172.21.15.89:0/453525100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcbb60 msgr2=0x55e552ffbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:53.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.403+0000 7f4980a00700 1 --2- 172.21.15.89:0/453525100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcbb60 0x55e552ffbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.403+0000 7f4980a00700 1 -- 172.21.15.89:0/453525100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552ffa830 msgr2=0x55e553023ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:53.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.403+0000 7f4980a00700 1 --2- 172.21.15.89:0/453525100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552ffa830 0x55e553023ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.403+0000 7f4980a00700 1 -- 172.21.15.89:0/453525100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e552ee6010 con 0x55e552fcdbe0 2024-04-02T09:13:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.404+0000 7f4980a00700 1 --2- 172.21.15.89:0/453525100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552fcdbe0 0x55e552ffc410 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f49680879f0 tx=0x7f496805e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85bc2481ef3bab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.404+0000 7f496e600700 1 -- 172.21.15.89:0/453525100 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496805c020 con 0x55e552fcdbe0 2024-04-02T09:13:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.404+0000 7f496e600700 1 -- 172.21.15.89:0/453525100 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f496806f050 con 0x55e552fcdbe0 2024-04-02T09:13:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.404+0000 7f496e600700 1 -- 172.21.15.89:0/453525100 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49680605e0 con 0x55e552fcdbe0 2024-04-02T09:13:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.404+0000 7f49979b7b80 1 -- 172.21.15.89:0/453525100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552fcdbe0 msgr2=0x55e552ffc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.404+0000 7f49979b7b80 1 --2- 172.21.15.89:0/453525100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552fcdbe0 0x55e552ffc410 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f49680879f0 tx=0x7f496805e040 comp rx=0 tx=0).stop 2024-04-02T09:13:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.404+0000 7f49979b7b80 1 -- 172.21.15.89:0/453525100 shutdown_connections 2024-04-02T09:13:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.404+0000 7f49979b7b80 1 --2- 172.21.15.89:0/453525100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552ffa830 0x55e553023ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.404+0000 7f49979b7b80 1 --2- 172.21.15.89:0/453525100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcbb60 0x55e552ffbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.404+0000 7f49979b7b80 1 --2- 172.21.15.89:0/453525100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552fcdbe0 0x55e552ffc410 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.404+0000 7f49979b7b80 1 -- 172.21.15.89:0/453525100 >> 172.21.15.89:0/453525100 conn(0x55e552f2e6a0 msgr2=0x55e553018250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:53.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.404+0000 7f49979b7b80 1 -- 172.21.15.89:0/453525100 shutdown_connections 2024-04-02T09:13:53.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.405+0000 7f49979b7b80 1 -- 172.21.15.89:0/453525100 wait complete. 2024-04-02T09:13:53.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.405+0000 7f49979b7b80 1 -- start start 2024-04-02T09:13:53.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.405+0000 7f49979b7b80 1 Processor -- start 2024-04-02T09:13:53.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.406+0000 7f49979b7b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcbb60 0x55e552ffbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.406+0000 7f49979b7b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcdbe0 0x55e552ffc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.406+0000 7f49979b7b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e553010bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.406+0000 7f49979b7b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5530236b0 con 0x55e552ffa830 2024-04-02T09:13:53.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.406+0000 7f49979b7b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e552fc6af0 con 0x55e552fcbb60 2024-04-02T09:13:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.406+0000 7f49979b7b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e553010730 con 0x55e552fcdbe0 2024-04-02T09:13:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.406+0000 7f49979b7b80 10 client.0 _ll_register_callbacks cb 0x7f49740115f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T09:13:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.407+0000 7f4981e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e553010bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.407+0000 7f4981e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e553010bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59276/0 (socket says 172.21.15.89:59276) 2024-04-02T09:13:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.407+0000 7f4981e00700 1 -- 172.21.15.89:0/1188480468 learned_addr learned my addr 172.21.15.89:0/1188480468 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.407+0000 7f4981400700 1 --2- 172.21.15.89:0/1188480468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcbb60 0x55e552ffbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.407+0000 7f4980a00700 1 --2- 172.21.15.89:0/1188480468 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcdbe0 0x55e552ffc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.407+0000 7f4981e00700 1 -- 172.21.15.89:0/1188480468 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcdbe0 msgr2=0x55e552ffc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.407+0000 7f4981e00700 1 --2- 172.21.15.89:0/1188480468 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcdbe0 0x55e552ffc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.407+0000 7f4981e00700 1 -- 172.21.15.89:0/1188480468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcbb60 msgr2=0x55e552ffbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.407+0000 7f4981e00700 1 --2- 172.21.15.89:0/1188480468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcbb60 0x55e552ffbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.407+0000 7f4981e00700 1 -- 172.21.15.89:0/1188480468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f497c0c9c50 con 0x55e552ffa830 2024-04-02T09:13:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.407+0000 7f4981e00700 1 --2- 172.21.15.89:0/1188480468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e553010bc0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f4974000f80 tx=0x7f4974003b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9504f2081b4a72b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:53.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.407+0000 7f4962000700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.408+0000 7f4962000700 1 -- 172.21.15.89:0/1188480468 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4974005700 con 0x55e552ffa830 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.408+0000 7f4962000700 1 -- 172.21.15.89:0/1188480468 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f497401ec10 con 0x55e552ffa830 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.408+0000 7f49979b7b80 1 -- 172.21.15.89:0/1188480468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f497c06b8e0 con 0x55e552ffa830 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.408+0000 7f4962000700 1 -- 172.21.15.89:0/1188480468 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49740041c0 con 0x55e552ffa830 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.408+0000 7f49979b7b80 10 client.12285 fetch_fsmap learned FSMap version 125 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.408+0000 7f49979b7b80 1 -- 172.21.15.89:0/1188480468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=125,monmap=2+}) v3 -- 0x55e552d741d0 con 0x55e552ffa830 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.408+0000 7f4962000700 1 -- 172.21.15.89:0/1188480468 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497c06b8e0 con 0x55e552ffa830 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.408+0000 7f4962000700 1 -- 172.21.15.89:0/1188480468 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 125) v1 ==== 3681+0+0 (secure 0 0 0) 0x7f497400f800 con 0x55e552ffa830 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.408+0000 7f49979b7b80 10 client.12285 fetch_fsmap finished waiting for FSMap version 125 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.408+0000 7f49979b7b80 10 client.12285 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.408+0000 7f49979b7b80 20 client.12285 populate_metadata read hostname 'smithi089' 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.408+0000 7f49979b7b80 1 --2- 172.21.15.89:0/1188480468 >> [v2:172.21.15.89:6834/3112580185,v1:172.21.15.89:6837/3112580185] conn(0x55e553011100 0x55e552fcdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.408+0000 7f49979b7b80 4 client.12285 mds_command: new command op to 11958 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.408+0000 7f49979b7b80 1 -- 172.21.15.89:0/1188480468 --> [v2:172.21.15.89:6834/3112580185,v1:172.21.15.89:6837/3112580185] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f497000b7c0 con 0x55e553011100 2024-04-02T09:13:53.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f4981400700 1 --2- 172.21.15.89:0/1188480468 >> [v2:172.21.15.89:6834/3112580185,v1:172.21.15.89:6837/3112580185] conn(0x55e553011100 0x55e552fcdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f4981400700 1 --2- 172.21.15.89:0/1188480468 >> [v2:172.21.15.89:6834/3112580185,v1:172.21.15.89:6837/3112580185] conn(0x55e553011100 0x55e552fcdfc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1f86b3130ac669e server_cookie=fcc70cd16d1cbe7b in_seq=0 out_seq=0 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f4962000700 10 client.12285 ms_handle_connect on v2:172.21.15.89:6834/3112580185 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f4962000700 1 -- 172.21.15.89:0/1188480468 <== mds.0 v2:172.21.15.89:6834/3112580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x7f497000b7c0 con 0x55e553011100 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f4962000700 10 client.12285 handle_command_reply: tid=0 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f49979b7b80 1 client.12285 shutdown 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f49979b7b80 1 -- 172.21.15.89:0/1188480468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 msgr2=0x55e553010bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f49979b7b80 1 --2- 172.21.15.89:0/1188480468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e553010bc0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f4974000f80 tx=0x7f4974003b00 comp rx=0 tx=0).stop 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f49979b7b80 1 -- 172.21.15.89:0/1188480468 shutdown_connections 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f49979b7b80 1 --2- 172.21.15.89:0/1188480468 >> [v2:172.21.15.89:6834/3112580185,v1:172.21.15.89:6837/3112580185] conn(0x55e553011100 0x55e552fcdfc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f49979b7b80 1 --2- 172.21.15.89:0/1188480468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e552fcbb60 0x55e552ffbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f4962000700 0 client.12285 ms_handle_reset on v2:172.21.15.89:6834/3112580185 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f49979b7b80 1 --2- 172.21.15.89:0/1188480468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e552ffa830 0x55e553010bc0 unknown :-1 s=CLOSED pgs=2628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f49979b7b80 1 --2- 172.21.15.89:0/1188480468 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55e552fcdbe0 0x55e552ffc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.409+0000 7f49979b7b80 1 -- 172.21.15.89:0/1188480468 >> 172.21.15.89:0/1188480468 conn(0x55e552f2e6a0 msgr2=0x55e553018230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.410+0000 7f49979b7b80 1 -- 172.21.15.89:0/1188480468 shutdown_connections 2024-04-02T09:13:53.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.410+0000 7f49979b7b80 1 -- 172.21.15.89:0/1188480468 wait complete. 2024-04-02T09:13:53.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.410+0000 7f49979b7b80 20 client.12285 trim_cache size 0 max 16384 2024-04-02T09:13:53.412 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:13:53.412 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T09:13:53.412 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T09:13:53.412 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 11958, 2024-04-02T09:13:53.412 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T09:13:53.412 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T09:13:53.412 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "backup_fs", 2024-04-02T09:13:53.412 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 11.4742444, 2024-04-02T09:13:53.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 125, 2024-04-02T09:13:53.412 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 248, 2024-04-02T09:13:53.412 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 248, 2024-04-02T09:13:53.413 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 30.073839305 2024-04-02T09:13:53.413 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:13:53.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.410+0000 7f4983e00700 1 -- 172.21.15.89:0/6406400 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f495c06aed0 msgr2=0x7f495c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:53.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.410+0000 7f4983e00700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f495c06aed0 0x7f495c06d390 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4970009130 tx=0x7f4970008030 comp rx=0 tx=0).stop 2024-04-02T09:13:53.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.410+0000 7f4983e00700 1 -- 172.21.15.89:0/6406400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f497c0d1fd0 msgr2=0x7f497c1f2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:53.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.410+0000 7f4983e00700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f497c0d1fd0 0x7f497c1f2610 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7f496800c750 tx=0x7f496800ca60 comp rx=0 tx=0).stop 2024-04-02T09:13:53.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.410+0000 7f4983e00700 1 -- 172.21.15.89:0/6406400 shutdown_connections 2024-04-02T09:13:53.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.410+0000 7f4983e00700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f495c06aed0 0x7f495c06d390 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.410+0000 7f4983e00700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f497c15dad0 0x7f497c1f6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.410+0000 7f4983e00700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f497c0d1630 0x7f497c1f20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.410+0000 7f4983e00700 1 --2- 172.21.15.89:0/6406400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f497c0d1fd0 0x7f497c1f2610 unknown :-1 s=CLOSED pgs=2624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.410+0000 7f4983e00700 1 -- 172.21.15.89:0/6406400 >> 172.21.15.89:0/6406400 conn(0x7f497c08aff0 msgr2=0x7f497c15ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:53.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.411+0000 7f4983e00700 1 -- 172.21.15.89:0/6406400 shutdown_connections 2024-04-02T09:13:53.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.411+0000 7f4983e00700 1 -- 172.21.15.89:0/6406400 wait complete. 2024-04-02T09:13:53.426 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T09:13:53.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.620+0000 7ffad5a00700 1 Processor -- start 2024-04-02T09:13:53.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.620+0000 7ffad5a00700 1 -- start start 2024-04-02T09:13:53.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.621+0000 7ffad5a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffad01437f0 0x7ffad00b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.621+0000 7ffad5a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ffad0143bd0 0x7ffad00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.621+0000 7ffad5a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad00ace10 0x7ffad006c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.621+0000 7ffad5a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad00ad280 con 0x7ffad01437f0 2024-04-02T09:13:53.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.621+0000 7ffad5a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad00ae4d0 con 0x7ffad0143bd0 2024-04-02T09:13:53.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.621+0000 7ffad5a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00ad490 con 0x7ffad00ace10 2024-04-02T09:13:53.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.621+0000 7ffacfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffad01437f0 0x7ffad00b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.621+0000 7ffacfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffad01437f0 0x7ffad00b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59286/0 (socket says 172.21.15.89:59286) 2024-04-02T09:13:53.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.621+0000 7ffacfe00700 1 -- 172.21.15.89:0/3150388684 learned_addr learned my addr 172.21.15.89:0/3150388684 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:53.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.622+0000 7ffacf400700 1 --2- 172.21.15.89:0/3150388684 >> v2:172.21.15.113:3300/0 conn(0x7ffad0143bd0 0x7ffad00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.622+0000 7ffad5000700 1 --2- 172.21.15.89:0/3150388684 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad00ace10 0x7ffad006c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.622+0000 7ffacfe00700 1 -- 172.21.15.89:0/3150388684 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad00ace10 msgr2=0x7ffad006c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:53.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.622+0000 7ffacfe00700 1 --2- 172.21.15.89:0/3150388684 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad00ace10 0x7ffad006c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.622+0000 7ffacfe00700 1 -- 172.21.15.89:0/3150388684 >> v2:172.21.15.113:3300/0 conn(0x7ffad0143bd0 msgr2=0x7ffad00ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:53.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.622+0000 7ffacfe00700 1 --2- 172.21.15.89:0/3150388684 >> v2:172.21.15.113:3300/0 conn(0x7ffad0143bd0 0x7ffad00ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.622+0000 7ffacfe00700 1 -- 172.21.15.89:0/3150388684 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00ad770 con 0x7ffad01437f0 2024-04-02T09:13:53.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.622+0000 7ffacfe00700 1 --2- 172.21.15.89:0/3150388684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad01437f0 0x7ffad00b3fc0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7ffad006b5e0 tx=0x7ffac0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b87a0339c47c66 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:53.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.622+0000 7ffacea00700 1 -- 172.21.15.89:0/3150388684 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac0011830 con 0x7ffad01437f0 2024-04-02T09:13:53.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.622+0000 7ffacea00700 1 -- 172.21.15.89:0/3150388684 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac0011990 con 0x7ffad01437f0 2024-04-02T09:13:53.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.622+0000 7ffacea00700 1 -- 172.21.15.89:0/3150388684 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac0011d20 con 0x7ffad01437f0 2024-04-02T09:13:53.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.623+0000 7ffad5a00700 1 -- 172.21.15.89:0/3150388684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad01437f0 msgr2=0x7ffad00b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:53.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.623+0000 7ffad5a00700 1 --2- 172.21.15.89:0/3150388684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad01437f0 0x7ffad00b3fc0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7ffad006b5e0 tx=0x7ffac0009bd0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.623+0000 7ffad5a00700 1 -- 172.21.15.89:0/3150388684 shutdown_connections 2024-04-02T09:13:53.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.623+0000 7ffad5a00700 1 --2- 172.21.15.89:0/3150388684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad01437f0 0x7ffad00b3fc0 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.623+0000 7ffad5a00700 1 --2- 172.21.15.89:0/3150388684 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad00ace10 0x7ffad006c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.623+0000 7ffad5a00700 1 --2- 172.21.15.89:0/3150388684 >> v2:172.21.15.113:3300/0 conn(0x7ffad0143bd0 0x7ffad00ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.623+0000 7ffad5a00700 1 -- 172.21.15.89:0/3150388684 >> 172.21.15.89:0/3150388684 conn(0x7ffad006df90 msgr2=0x7ffad006a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:53.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.623+0000 7ffad5a00700 1 -- 172.21.15.89:0/3150388684 shutdown_connections 2024-04-02T09:13:53.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.623+0000 7ffad5a00700 1 -- 172.21.15.89:0/3150388684 wait complete. 2024-04-02T09:13:53.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.625+0000 7ffad5a00700 1 Processor -- start 2024-04-02T09:13:53.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.625+0000 7ffad5a00700 1 -- start start 2024-04-02T09:13:53.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.626+0000 7ffad5a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad00ace10 0x7ffad015ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.626+0000 7ffad5a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad01437f0 0x7ffad01f1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.626+0000 7ffad5a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffad0143bd0 0x7ffad01f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.626+0000 7ffad5a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad01603f0 con 0x7ffad01437f0 2024-04-02T09:13:53.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.626+0000 7ffad5a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad015ff60 con 0x7ffad0143bd0 2024-04-02T09:13:53.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.626+0000 7ffad5a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad015f350 con 0x7ffad00ace10 2024-04-02T09:13:53.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.626+0000 7ffacf400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad01437f0 0x7ffad01f1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.626+0000 7ffacf400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad01437f0 0x7ffad01f1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59290/0 (socket says 172.21.15.89:59290) 2024-04-02T09:13:53.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.626+0000 7ffacf400700 1 -- 172.21.15.89:0/2747333577 learned_addr learned my addr 172.21.15.89:0/2747333577 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:53.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.627+0000 7ffad5000700 1 --2- 172.21.15.89:0/2747333577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffad0143bd0 0x7ffad01f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.627+0000 7ffacfe00700 1 --2- 172.21.15.89:0/2747333577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad00ace10 0x7ffad015ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.627+0000 7ffacf400700 1 -- 172.21.15.89:0/2747333577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad00ace10 msgr2=0x7ffad015ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:53.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.627+0000 7ffacf400700 1 --2- 172.21.15.89:0/2747333577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad00ace10 0x7ffad015ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.627+0000 7ffacf400700 1 -- 172.21.15.89:0/2747333577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffad0143bd0 msgr2=0x7ffad01f24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:53.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.627+0000 7ffacf400700 1 --2- 172.21.15.89:0/2747333577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffad0143bd0 0x7ffad01f24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.627+0000 7ffacf400700 1 -- 172.21.15.89:0/2747333577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac0009550 con 0x7ffad01437f0 2024-04-02T09:13:53.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.627+0000 7ffacf400700 1 --2- 172.21.15.89:0/2747333577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad01437f0 0x7ffad01f1f80 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00e870 tx=0x7ffabc00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:53.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.627+0000 7ffac7e00700 1 -- 172.21.15.89:0/2747333577 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc009590 con 0x7ffad01437f0 2024-04-02T09:13:53.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.627+0000 7ffac7e00700 1 -- 172.21.15.89:0/2747333577 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffabc013680 con 0x7ffad01437f0 2024-04-02T09:13:53.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.627+0000 7ffac7e00700 1 -- 172.21.15.89:0/2747333577 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc0139f0 con 0x7ffad01437f0 2024-04-02T09:13:53.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.627+0000 7ffad5a00700 1 -- 172.21.15.89:0/2747333577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad015f630 con 0x7ffad01437f0 2024-04-02T09:13:53.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.628+0000 7ffad5a00700 1 -- 172.21.15.89:0/2747333577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad015fbf0 con 0x7ffad01437f0 2024-04-02T09:13:53.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.629+0000 7ffad5a00700 1 -- 172.21.15.89:0/2747333577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad004ea80 con 0x7ffad01437f0 2024-04-02T09:13:53.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.630+0000 7ffac7e00700 1 -- 172.21.15.89:0/2747333577 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7ffabc013b50 con 0x7ffad01437f0 2024-04-02T09:13:53.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.631+0000 7ffac7e00700 1 --2- 172.21.15.89:0/2747333577 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffab006aed0 0x7ffab006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:53.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.631+0000 7ffac7e00700 1 -- 172.21.15.89:0/2747333577 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7ffabc089a90 con 0x7ffad01437f0 2024-04-02T09:13:53.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.631+0000 7ffacfe00700 1 --2- 172.21.15.89:0/2747333577 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffab006aed0 0x7ffab006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:53.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.632+0000 7ffacfe00700 1 --2- 172.21.15.89:0/2747333577 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffab006aed0 0x7ffab006d390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffac0009c10 tx=0x7ffac0008030 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:53.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.635+0000 7ffac7e00700 1 -- 172.21.15.89:0/2747333577 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffabc0543e0 con 0x7ffad01437f0 2024-04-02T09:13:53.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.864+0000 7ffad5a00700 1 -- 172.21.15.89:0/2747333577 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ffad00b3aa0 con 0x7ffab006aed0 2024-04-02T09:13:53.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.865+0000 7ffac7e00700 1 -- 172.21.15.89:0/2747333577 <== mgr.11613 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7ffac0011810 con 0x7ffab006aed0 2024-04-02T09:13:53.864 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T09:13:53.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:53.864 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T09:13:53.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:13:53.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:13:53.864 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "backup_fs" 2024-04-02T09:13:53.864 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:13:53.864 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T09:13:53.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.868+0000 7ffad5a00700 1 -- 172.21.15.89:0/2747333577 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffab006aed0 msgr2=0x7ffab006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:53.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.868+0000 7ffad5a00700 1 --2- 172.21.15.89:0/2747333577 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffab006aed0 0x7ffab006d390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffac0009c10 tx=0x7ffac0008030 comp rx=0 tx=0).stop 2024-04-02T09:13:53.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.868+0000 7ffad5a00700 1 -- 172.21.15.89:0/2747333577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad01437f0 msgr2=0x7ffad01f1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:53.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.868+0000 7ffad5a00700 1 --2- 172.21.15.89:0/2747333577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad01437f0 0x7ffad01f1f80 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00e870 tx=0x7ffabc00eb80 comp rx=0 tx=0).stop 2024-04-02T09:13:53.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.868+0000 7ffad5a00700 1 -- 172.21.15.89:0/2747333577 shutdown_connections 2024-04-02T09:13:53.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.868+0000 7ffad5a00700 1 --2- 172.21.15.89:0/2747333577 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffab006aed0 0x7ffab006d390 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.868+0000 7ffad5a00700 1 --2- 172.21.15.89:0/2747333577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffad0143bd0 0x7ffad01f24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.868+0000 7ffad5a00700 1 --2- 172.21.15.89:0/2747333577 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad00ace10 0x7ffad015ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.868+0000 7ffad5a00700 1 --2- 172.21.15.89:0/2747333577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad01437f0 0x7ffad01f1f80 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:53.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.868+0000 7ffad5a00700 1 -- 172.21.15.89:0/2747333577 >> 172.21.15.89:0/2747333577 conn(0x7ffad006df90 msgr2=0x7ffad015d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:53.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.868+0000 7ffad5a00700 1 -- 172.21.15.89:0/2747333577 shutdown_connections 2024-04-02T09:13:53.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:53.868+0000 7ffad5a00700 1 -- 172.21.15.89:0/2747333577 wait complete. 2024-04-02T09:13:53.885 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T09:13:54.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.081+0000 7f6c19000700 1 Processor -- start 2024-04-02T09:13:54.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.081+0000 7f6c19000700 1 -- start start 2024-04-02T09:13:54.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.082+0000 7f6c19000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6c14160e50 0x7f6c140b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:54.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.082+0000 7f6c19000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c140b3cb0 con 0x7f6c140d3f20 2024-04-02T09:13:54.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.082+0000 7f6c19000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c140cef00 con 0x7f6c140b3f80 2024-04-02T09:13:54.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.082+0000 7f6c19000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c14161230 con 0x7f6c14160e50 2024-04-02T09:13:54.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.083+0000 7f6c11e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6c14160e50 0x7f6c140b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:54.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.083+0000 7f6c11400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6c140d3f20 0x7f6c140b2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39462/0 (socket says 172.21.15.89:39462) 2024-04-02T09:13:54.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.083+0000 7f6c11400700 1 -- 172.21.15.89:0/4176423137 learned_addr learned my addr 172.21.15.89:0/4176423137 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.083+0000 7f6c11e00700 1 -- 172.21.15.89:0/4176423137 >> v1:172.21.15.113:6789/0 conn(0x7f6c140b3f80 legacy=0x7f6c140b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.083+0000 7f6c11e00700 1 -- 172.21.15.89:0/4176423137 >> v1:172.21.15.89:6789/0 conn(0x7f6c140d3f20 legacy=0x7f6c140b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.083+0000 7f6c11e00700 1 -- 172.21.15.89:0/4176423137 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c140d2a80 con 0x7f6c14160e50 2024-04-02T09:13:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.083+0000 7f6c11e00700 1 --2- 172.21.15.89:0/4176423137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c14160e50 0x7f6c140b4e90 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009840 tx=0x7f6c04009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244a760497ec5ba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.084+0000 7f6c10a00700 1 -- 172.21.15.89:0/4176423137 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04011830 con 0x7f6c14160e50 2024-04-02T09:13:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.084+0000 7f6c10a00700 1 -- 172.21.15.89:0/4176423137 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c04011990 con 0x7f6c14160e50 2024-04-02T09:13:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.084+0000 7f6c10a00700 1 -- 172.21.15.89:0/4176423137 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04011d20 con 0x7f6c14160e50 2024-04-02T09:13:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.084+0000 7f6c19000700 1 -- 172.21.15.89:0/4176423137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c14160e50 msgr2=0x7f6c140b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.084+0000 7f6c19000700 1 --2- 172.21.15.89:0/4176423137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c14160e50 0x7f6c140b4e90 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009840 tx=0x7f6c04009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.084+0000 7f6c19000700 1 -- 172.21.15.89:0/4176423137 shutdown_connections 2024-04-02T09:13:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.084+0000 7f6c19000700 1 --2- 172.21.15.89:0/4176423137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c14160e50 0x7f6c140b4e90 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.084+0000 7f6c19000700 1 -- 172.21.15.89:0/4176423137 >> 172.21.15.89:0/4176423137 conn(0x7f6c1408aff0 msgr2=0x7f6c140d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:54.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.084+0000 7f6c19000700 1 -- 172.21.15.89:0/4176423137 shutdown_connections 2024-04-02T09:13:54.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.084+0000 7f6c19000700 1 -- 172.21.15.89:0/4176423137 wait complete. 2024-04-02T09:13:54.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.086+0000 7f6c19000700 1 Processor -- start 2024-04-02T09:13:54.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.086+0000 7f6c19000700 1 -- start start 2024-04-02T09:13:54.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.087+0000 7f6c19000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c140b3f80 0x7f6c1415fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:54.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.087+0000 7f6c19000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c14160e50 0x7f6c14155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:54.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.087+0000 7f6c19000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c14155c10 0x7f6c140d1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:54.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.087+0000 7f6c19000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c140b6f20 con 0x7f6c140b3f80 2024-04-02T09:13:54.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.087+0000 7f6c19000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c140b8330 con 0x7f6c14155c10 2024-04-02T09:13:54.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.087+0000 7f6c19000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c140b6050 con 0x7f6c14160e50 2024-04-02T09:13:54.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.087+0000 7f6c11e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c140b3f80 0x7f6c1415fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:54.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.087+0000 7f6c11e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c140b3f80 0x7f6c1415fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59308/0 (socket says 172.21.15.89:59308) 2024-04-02T09:13:54.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.087+0000 7f6c11e00700 1 -- 172.21.15.89:0/1225243994 learned_addr learned my addr 172.21.15.89:0/1225243994 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:54.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.087+0000 7f6c11400700 1 --2- 172.21.15.89:0/1225243994 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c14160e50 0x7f6c14155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:54.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.088+0000 7f6c12800700 1 --2- 172.21.15.89:0/1225243994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c14155c10 0x7f6c140d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:54.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.088+0000 7f6c11e00700 1 -- 172.21.15.89:0/1225243994 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c14160e50 msgr2=0x7f6c14155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:54.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.088+0000 7f6c11e00700 1 --2- 172.21.15.89:0/1225243994 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c14160e50 0x7f6c14155370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:54.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.088+0000 7f6c11e00700 1 -- 172.21.15.89:0/1225243994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c14155c10 msgr2=0x7f6c140d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:54.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.088+0000 7f6c11e00700 1 --2- 172.21.15.89:0/1225243994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c14155c10 0x7f6c140d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:54.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.088+0000 7f6c11e00700 1 -- 172.21.15.89:0/1225243994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c040094a0 con 0x7f6c140b3f80 2024-04-02T09:13:54.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.088+0000 7f6c11e00700 1 --2- 172.21.15.89:0/1225243994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c140b3f80 0x7f6c1415fcd0 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f6c040090b0 tx=0x7f6c04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:54.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.088+0000 7f6c0a000700 1 -- 172.21.15.89:0/1225243994 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c040045f0 con 0x7f6c140b3f80 2024-04-02T09:13:54.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.088+0000 7f6c0a000700 1 -- 172.21.15.89:0/1225243994 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0401e070 con 0x7f6c140b3f80 2024-04-02T09:13:54.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.088+0000 7f6c0a000700 1 -- 172.21.15.89:0/1225243994 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04003b00 con 0x7f6c140b3f80 2024-04-02T09:13:54.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.088+0000 7f6c19000700 1 -- 172.21.15.89:0/1225243994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c140b62a0 con 0x7f6c140b3f80 2024-04-02T09:13:54.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.088+0000 7f6c19000700 1 -- 172.21.15.89:0/1225243994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c140b6860 con 0x7f6c140b3f80 2024-04-02T09:13:54.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.089+0000 7f6c19000700 1 -- 172.21.15.89:0/1225243994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c140b7060 con 0x7f6c140b3f80 2024-04-02T09:13:54.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.090+0000 7f6c0a000700 1 -- 172.21.15.89:0/1225243994 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 158) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f6c04003c60 con 0x7f6c140b3f80 2024-04-02T09:13:54.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.091+0000 7f6c0a000700 1 --2- 172.21.15.89:0/1225243994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6bf406aed0 0x7f6bf406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:54.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.091+0000 7f6c0a000700 1 -- 172.21.15.89:0/1225243994 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f6c0408ae80 con 0x7f6c140b3f80 2024-04-02T09:13:54.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.091+0000 7f6c11400700 1 --2- 172.21.15.89:0/1225243994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6bf406aed0 0x7f6bf406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:54.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.092+0000 7f6c11400700 1 --2- 172.21.15.89:0/1225243994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6bf406aed0 0x7f6bf406d390 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6c14156130 tx=0x7f6c00009040 comp rx=0 tx=0).ready entity=mgr.11613 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:54.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.096+0000 7f6c0a000700 1 -- 172.21.15.89:0/1225243994 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c140b7060 con 0x7f6c140b3f80 2024-04-02T09:13:54.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:54.356+0000 7f6c19000700 1 -- 172.21.15.89:0/1225243994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f6c140b7060 con 0x7f6c140b3f80 2024-04-02T09:13:55.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.258+0000 7f6c0a000700 1 -- 172.21.15.89:0/1225243994 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 159) v1 ==== 89358+0+0 (secure 0 0 0) 0x7f6c04055ba0 con 0x7f6c140b3f80 2024-04-02T09:13:55.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.260+0000 7f6c0a000700 1 -- 172.21.15.89:0/1225243994 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v159) v1 ==== 92+0+0 (secure 0 0 0) 0x7f6c040710a0 con 0x7f6c140b3f80 2024-04-02T09:13:55.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.265+0000 7f6c19000700 1 -- 172.21.15.89:0/1225243994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6bf406aed0 msgr2=0x7f6bf406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:55.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.265+0000 7f6c19000700 1 --2- 172.21.15.89:0/1225243994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6bf406aed0 0x7f6bf406d390 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6c14156130 tx=0x7f6c00009040 comp rx=0 tx=0).stop 2024-04-02T09:13:55.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.265+0000 7f6c19000700 1 -- 172.21.15.89:0/1225243994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c140b3f80 msgr2=0x7f6c1415fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:55.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.265+0000 7f6c19000700 1 --2- 172.21.15.89:0/1225243994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c140b3f80 0x7f6c1415fcd0 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f6c040090b0 tx=0x7f6c04004750 comp rx=0 tx=0).stop 2024-04-02T09:13:55.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.265+0000 7f6c19000700 1 -- 172.21.15.89:0/1225243994 shutdown_connections 2024-04-02T09:13:55.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.265+0000 7f6c19000700 1 --2- 172.21.15.89:0/1225243994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6bf406aed0 0x7f6bf406d390 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.265+0000 7f6c19000700 1 --2- 172.21.15.89:0/1225243994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c14155c10 0x7f6c140d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.265+0000 7f6c19000700 1 --2- 172.21.15.89:0/1225243994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c140b3f80 0x7f6c1415fcd0 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.265+0000 7f6c19000700 1 --2- 172.21.15.89:0/1225243994 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6c14160e50 0x7f6c14155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.265+0000 7f6c19000700 1 -- 172.21.15.89:0/1225243994 >> 172.21.15.89:0/1225243994 conn(0x7f6c1408aff0 msgr2=0x7f6c1406d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:55.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.265+0000 7f6c19000700 1 -- 172.21.15.89:0/1225243994 shutdown_connections 2024-04-02T09:13:55.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.265+0000 7f6c19000700 1 -- 172.21.15.89:0/1225243994 wait complete. 2024-04-02T09:13:55.285 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T09:13:55.475 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:55.477+0000 7fc6098b6240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:13:55.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.490+0000 7fedb2000700 1 Processor -- start 2024-04-02T09:13:55.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.490+0000 7fedb2000700 1 -- start start 2024-04-02T09:13:55.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.491+0000 7fedb2000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fedac15d9d0 0x7fedac0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:55.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.491+0000 7fedb2000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac161420 con 0x7fedac0d2540 2024-04-02T09:13:55.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.491+0000 7fedb2000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac161700 con 0x7fedac0d16a0 2024-04-02T09:13:55.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.491+0000 7fedb2000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedac15edc0 con 0x7fedac15d9d0 2024-04-02T09:13:55.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.491+0000 7fedab400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fedac15d9d0 0x7fedac0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:55.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.491+0000 7fedabe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fedac0d2540 0x7fedac0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39482/0 (socket says 172.21.15.89:39482) 2024-04-02T09:13:55.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.491+0000 7fedabe00700 1 -- 172.21.15.89:0/1262767418 learned_addr learned my addr 172.21.15.89:0/1262767418 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:55.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.492+0000 7fedaa000700 1 -- 172.21.15.89:0/1262767418 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061623581 0 0) 0x7fedac161420 con 0x7fedac0d2540 2024-04-02T09:13:55.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.492+0000 7fedaa000700 1 -- 172.21.15.89:0/1262767418 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed900035b0 con 0x7fedac0d2540 2024-04-02T09:13:55.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.492+0000 7fedaa000700 1 -- 172.21.15.89:0/1262767418 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784210963 0 0) 0x7fedac161700 con 0x7fedac0d16a0 2024-04-02T09:13:55.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.492+0000 7fedab400700 1 -- 172.21.15.89:0/1262767418 >> v1:172.21.15.113:6789/0 conn(0x7fedac0d16a0 legacy=0x7fedac0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:55.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.492+0000 7fedab400700 1 -- 172.21.15.89:0/1262767418 >> v1:172.21.15.89:6789/0 conn(0x7fedac0d2540 legacy=0x7fedac0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:55.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.492+0000 7fedab400700 1 -- 172.21.15.89:0/1262767418 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac1619e0 con 0x7fedac15d9d0 2024-04-02T09:13:55.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.492+0000 7fedab400700 1 --2- 172.21.15.89:0/1262767418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac15d9d0 0x7fedac0d3c30 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fed9c009840 tx=0x7fed9c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfe5cac32f31cb1b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:55.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.493+0000 7fedaa000700 1 -- 172.21.15.89:0/1262767418 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c011830 con 0x7fedac15d9d0 2024-04-02T09:13:55.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.493+0000 7fedaa000700 1 -- 172.21.15.89:0/1262767418 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9c011990 con 0x7fedac15d9d0 2024-04-02T09:13:55.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.493+0000 7fedaa000700 1 -- 172.21.15.89:0/1262767418 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c011d00 con 0x7fedac15d9d0 2024-04-02T09:13:55.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.493+0000 7fedb2000700 1 -- 172.21.15.89:0/1262767418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac15d9d0 msgr2=0x7fedac0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:55.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.493+0000 7fedb2000700 1 --2- 172.21.15.89:0/1262767418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac15d9d0 0x7fedac0d3c30 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fed9c009840 tx=0x7fed9c009b50 comp rx=0 tx=0).stop 2024-04-02T09:13:55.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.493+0000 7fedb2000700 1 -- 172.21.15.89:0/1262767418 shutdown_connections 2024-04-02T09:13:55.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.493+0000 7fedb2000700 1 --2- 172.21.15.89:0/1262767418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac15d9d0 0x7fedac0d3c30 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.493+0000 7fedb2000700 1 -- 172.21.15.89:0/1262767418 >> 172.21.15.89:0/1262767418 conn(0x7fedac08aff0 msgr2=0x7fedac0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:55.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.493+0000 7fedb2000700 1 -- 172.21.15.89:0/1262767418 shutdown_connections 2024-04-02T09:13:55.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.493+0000 7fedb2000700 1 -- 172.21.15.89:0/1262767418 wait complete. 2024-04-02T09:13:55.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.495+0000 7fedb2000700 1 Processor -- start 2024-04-02T09:13:55.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.495+0000 7fedb2000700 1 -- start start 2024-04-02T09:13:55.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.496+0000 7fedb2000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0d16a0 0x7fedac0b75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:55.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.496+0000 7fedb2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac0d2540 0x7fedac0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:55.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.496+0000 7fedb2000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac15d9d0 0x7fedac0b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:55.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.496+0000 7fedb2000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac0b3e90 con 0x7fedac0d2540 2024-04-02T09:13:55.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.496+0000 7fedb2000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac0b4170 con 0x7fedac15d9d0 2024-04-02T09:13:55.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.496+0000 7fedb2000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedac0b4450 con 0x7fedac0d16a0 2024-04-02T09:13:55.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.496+0000 7fedaaa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac0d2540 0x7fedac0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:55.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.496+0000 7fedaaa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac0d2540 0x7fedac0b3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59328/0 (socket says 172.21.15.89:59328) 2024-04-02T09:13:55.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.496+0000 7fedaaa00700 1 -- 172.21.15.89:0/1697477092 learned_addr learned my addr 172.21.15.89:0/1697477092 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:55.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.496+0000 7fedabe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac15d9d0 0x7fedac0b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:55.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.496+0000 7fedab400700 1 --2- 172.21.15.89:0/1697477092 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0d16a0 0x7fedac0b75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:55.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.497+0000 7fedaaa00700 1 -- 172.21.15.89:0/1697477092 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0d16a0 msgr2=0x7fedac0b75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:55.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.497+0000 7fedaaa00700 1 --2- 172.21.15.89:0/1697477092 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0d16a0 0x7fedac0b75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.497+0000 7fedaaa00700 1 -- 172.21.15.89:0/1697477092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac15d9d0 msgr2=0x7fedac0b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.497+0000 7fedaaa00700 1 --2- 172.21.15.89:0/1697477092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac15d9d0 0x7fedac0b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.497+0000 7fedaaa00700 1 -- 172.21.15.89:0/1697477092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9c0094a0 con 0x7fedac0d2540 2024-04-02T09:13:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.497+0000 7fedaaa00700 1 --2- 172.21.15.89:0/1697477092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac0d2540 0x7fedac0b3660 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fed9800c450 tx=0x7fed9800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.497+0000 7feda3400700 1 -- 172.21.15.89:0/1697477092 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed980054e0 con 0x7fedac0d2540 2024-04-02T09:13:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.497+0000 7feda3400700 1 -- 172.21.15.89:0/1697477092 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9800ce60 con 0x7fedac0d2540 2024-04-02T09:13:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.497+0000 7feda3400700 1 -- 172.21.15.89:0/1697477092 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed98003910 con 0x7fedac0d2540 2024-04-02T09:13:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.497+0000 7fedb2000700 1 -- 172.21.15.89:0/1697477092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedac0b4730 con 0x7fedac0d2540 2024-04-02T09:13:55.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.498+0000 7fedb2000700 1 -- 172.21.15.89:0/1697477092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedac0b3ba0 con 0x7fedac0d2540 2024-04-02T09:13:55.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.499+0000 7fedb2000700 1 -- 172.21.15.89:0/1697477092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedac0d1ca0 con 0x7fedac0d2540 2024-04-02T09:13:55.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.500+0000 7feda3400700 1 -- 172.21.15.89:0/1697477092 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 159) v1 ==== 89358+0+0 (secure 0 0 0) 0x7fed98003a70 con 0x7fedac0d2540 2024-04-02T09:13:55.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.500+0000 7feda3400700 1 --2- 172.21.15.89:0/1697477092 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed8c06af20 0x7fed8c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:55.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.500+0000 7feda3400700 1 -- 172.21.15.89:0/1697477092 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fed98089030 con 0x7fedac0d2540 2024-04-02T09:13:55.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.500+0000 7fedab400700 1 -- 172.21.15.89:0/1697477092 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed8c06af20 msgr2=0x7fed8c06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:55.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.501+0000 7fedab400700 1 --2- 172.21.15.89:0/1697477092 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed8c06af20 0x7fed8c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:13:55.501 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:55.503+0000 7fca907b9240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:13:55.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.505+0000 7feda3400700 1 -- 172.21.15.89:0/1697477092 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fed98053990 con 0x7fedac0d2540 2024-04-02T09:13:55.660 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:55.662+0000 7fc6098b6240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:13:55.691 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:55.693+0000 7fca907b9240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:13:55.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.702+0000 7fedab400700 1 -- 172.21.15.89:0/1697477092 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed8c06af20 msgr2=0x7fed8c06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:55.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.702+0000 7fedab400700 1 --2- 172.21.15.89:0/1697477092 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed8c06af20 0x7fed8c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:13:55.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.753+0000 7fedb2000700 1 -- 172.21.15.89:0/1697477092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fedac0b4c50 con 0x7fedac0d2540 2024-04-02T09:13:55.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.757+0000 7feda3400700 1 -- 172.21.15.89:0/1697477092 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v83)=0 v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7fedac0b4c50 con 0x7fedac0d2540 2024-04-02T09:13:55.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.759+0000 7fedb2000700 1 -- 172.21.15.89:0/1697477092 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed8c06af20 msgr2=0x7fed8c06d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:13:55.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.759+0000 7fedb2000700 1 --2- 172.21.15.89:0/1697477092 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed8c06af20 0x7fed8c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.759+0000 7fedb2000700 1 -- 172.21.15.89:0/1697477092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac0d2540 msgr2=0x7fedac0b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:55.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.759+0000 7fedb2000700 1 --2- 172.21.15.89:0/1697477092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac0d2540 0x7fedac0b3660 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fed9800c450 tx=0x7fed9800c760 comp rx=0 tx=0).stop 2024-04-02T09:13:55.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.760+0000 7fedb2000700 1 -- 172.21.15.89:0/1697477092 shutdown_connections 2024-04-02T09:13:55.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.760+0000 7fedb2000700 1 --2- 172.21.15.89:0/1697477092 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fed8c06af20 0x7fed8c06d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.760+0000 7fedb2000700 1 --2- 172.21.15.89:0/1697477092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedac15d9d0 0x7fedac0b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.760+0000 7fedb2000700 1 --2- 172.21.15.89:0/1697477092 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fedac0d16a0 0x7fedac0b75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.760+0000 7fedb2000700 1 --2- 172.21.15.89:0/1697477092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac0d2540 0x7fedac0b3660 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.760+0000 7fedb2000700 1 -- 172.21.15.89:0/1697477092 >> 172.21.15.89:0/1697477092 conn(0x7fedac08aff0 msgr2=0x7fedac0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:55.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.760+0000 7fedb2000700 1 -- 172.21.15.89:0/1697477092 shutdown_connections 2024-04-02T09:13:55.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.760+0000 7fedb2000700 1 -- 172.21.15.89:0/1697477092 wait complete. 2024-04-02T09:13:55.777 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T09:13:55.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.975+0000 7f5449800700 1 Processor -- start 2024-04-02T09:13:55.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.975+0000 7f5449800700 1 -- start start 2024-04-02T09:13:55.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.977+0000 7f5449800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f54441611b0 0x7f544415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:55.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.977+0000 7f5449800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f54440cee10 0x7f5444161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:55.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.977+0000 7f5449800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54440ce270 con 0x7f544415f4d0 2024-04-02T09:13:55.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.977+0000 7f5449800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f544415b600 con 0x7f54440cee10 2024-04-02T09:13:55.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.977+0000 7f5449800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54440d2760 con 0x7f54441611b0 2024-04-02T09:13:55.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.977+0000 7f5443400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f54440cee10 0x7f5444161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:55.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.977+0000 7f5443400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f54440cee10 0x7f5444161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59334/0 (socket says 172.21.15.89:59334) 2024-04-02T09:13:55.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.977+0000 7f5443400700 1 -- 172.21.15.89:0/436787641 learned_addr learned my addr 172.21.15.89:0/436787641 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:55.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.977+0000 7f5442a00700 1 --2- 172.21.15.89:0/436787641 >> v2:172.21.15.113:3300/0 conn(0x7f54441611b0 0x7f544415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:55.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.977+0000 7f5443400700 1 -- 172.21.15.89:0/436787641 >> v2:172.21.15.113:3300/0 conn(0x7f54441611b0 msgr2=0x7f544415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:55.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.977+0000 7f5443400700 1 --2- 172.21.15.89:0/436787641 >> v2:172.21.15.113:3300/0 conn(0x7f54441611b0 0x7f544415ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.977+0000 7f5443400700 1 -- 172.21.15.89:0/436787641 >> v1:172.21.15.113:6789/0 conn(0x7f544415f4d0 legacy=0x7f54440ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:55.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.977+0000 7f5443400700 1 -- 172.21.15.89:0/436787641 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54440d2a40 con 0x7f54440cee10 2024-04-02T09:13:55.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.978+0000 7f5443400700 1 --2- 172.21.15.89:0/436787641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54440cee10 0x7f5444161590 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f544408b520 tx=0x7f543c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34628e1e133727b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:55.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.978+0000 7f5442000700 1 -- 172.21.15.89:0/436787641 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c00e070 con 0x7f54440cee10 2024-04-02T09:13:55.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.978+0000 7f5442000700 1 -- 172.21.15.89:0/436787641 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543c00b910 con 0x7f54440cee10 2024-04-02T09:13:55.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.978+0000 7f5442000700 1 -- 172.21.15.89:0/436787641 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c00bcc0 con 0x7f54440cee10 2024-04-02T09:13:55.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.978+0000 7f5449800700 1 -- 172.21.15.89:0/436787641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54440cee10 msgr2=0x7f5444161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:55.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.978+0000 7f5449800700 1 --2- 172.21.15.89:0/436787641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54440cee10 0x7f5444161590 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f544408b520 tx=0x7f543c00b330 comp rx=0 tx=0).stop 2024-04-02T09:13:55.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.979+0000 7f5449800700 1 -- 172.21.15.89:0/436787641 shutdown_connections 2024-04-02T09:13:55.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.979+0000 7f5449800700 1 --2- 172.21.15.89:0/436787641 >> v2:172.21.15.113:3300/0 conn(0x7f54441611b0 0x7f544415ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.979+0000 7f5449800700 1 --2- 172.21.15.89:0/436787641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54440cee10 0x7f5444161590 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.979+0000 7f5449800700 1 -- 172.21.15.89:0/436787641 >> 172.21.15.89:0/436787641 conn(0x7f544408aff0 msgr2=0x7f54440d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:55.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.979+0000 7f5449800700 1 -- 172.21.15.89:0/436787641 shutdown_connections 2024-04-02T09:13:55.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.979+0000 7f5449800700 1 -- 172.21.15.89:0/436787641 wait complete. 2024-04-02T09:13:55.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.980+0000 7f5449800700 1 Processor -- start 2024-04-02T09:13:55.979 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:55.980+0000 7fc6098b6240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:13:55.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.981+0000 7f5449800700 1 -- start start 2024-04-02T09:13:55.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.981+0000 7f5449800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54440cee10 0x7f54440c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:55.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.981+0000 7f5449800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f544415f4d0 0x7f54440b8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:55.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.981+0000 7f5449800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54441611b0 0x7f54440c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:55.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.982+0000 7f5449800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54440b7960 con 0x7f54440cee10 2024-04-02T09:13:55.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.982+0000 7f5449800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54440b6ce0 con 0x7f54441611b0 2024-04-02T09:13:55.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.982+0000 7f5449800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54440b6fc0 con 0x7f544415f4d0 2024-04-02T09:13:55.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.982+0000 7f5442a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54440cee10 0x7f54440c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:55.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.982+0000 7f5442a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54440cee10 0x7f54440c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59344/0 (socket says 172.21.15.89:59344) 2024-04-02T09:13:55.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.982+0000 7f5442a00700 1 -- 172.21.15.89:0/972827674 learned_addr learned my addr 172.21.15.89:0/972827674 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:55.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.982+0000 7f543be00700 1 --2- 172.21.15.89:0/972827674 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f544415f4d0 0x7f54440b8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:55.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.982+0000 7f5443400700 1 --2- 172.21.15.89:0/972827674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54441611b0 0x7f54440c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:55.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.982+0000 7f5442a00700 1 -- 172.21.15.89:0/972827674 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f544415f4d0 msgr2=0x7f54440b8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:55.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.982+0000 7f5442a00700 1 --2- 172.21.15.89:0/972827674 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f544415f4d0 0x7f54440b8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.982+0000 7f5442a00700 1 -- 172.21.15.89:0/972827674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54441611b0 msgr2=0x7f54440c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:55.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.982+0000 7f5442a00700 1 --2- 172.21.15.89:0/972827674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54441611b0 0x7f54440c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:55.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.982+0000 7f5442a00700 1 -- 172.21.15.89:0/972827674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c009b30 con 0x7f54440cee10 2024-04-02T09:13:55.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.983+0000 7f5442a00700 1 --2- 172.21.15.89:0/972827674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54440cee10 0x7f54440c9830 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f543400e870 tx=0x7f543400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:55.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.983+0000 7f543aa00700 1 -- 172.21.15.89:0/972827674 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434009590 con 0x7f54440cee10 2024-04-02T09:13:55.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.983+0000 7f543aa00700 1 -- 172.21.15.89:0/972827674 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5434013680 con 0x7f54440cee10 2024-04-02T09:13:55.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.983+0000 7f543aa00700 1 -- 172.21.15.89:0/972827674 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54340139f0 con 0x7f54440cee10 2024-04-02T09:13:55.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.983+0000 7f5449800700 1 -- 172.21.15.89:0/972827674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54440b72a0 con 0x7f54440cee10 2024-04-02T09:13:55.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.983+0000 7f5449800700 1 -- 172.21.15.89:0/972827674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54440c9440 con 0x7f54440cee10 2024-04-02T09:13:55.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.984+0000 7f5449800700 1 -- 172.21.15.89:0/972827674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5444067720 con 0x7f54440cee10 2024-04-02T09:13:55.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.985+0000 7f543aa00700 1 -- 172.21.15.89:0/972827674 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 159) v1 ==== 89358+0+0 (secure 0 0 0) 0x7f5434013b50 con 0x7f54440cee10 2024-04-02T09:13:55.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.986+0000 7f543aa00700 1 --2- 172.21.15.89:0/972827674 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f542006af20 0x7f542006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:55.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.986+0000 7f543aa00700 1 -- 172.21.15.89:0/972827674 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f5434089bc0 con 0x7f54440cee10 2024-04-02T09:13:55.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.986+0000 7f543be00700 1 -- 172.21.15.89:0/972827674 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f542006af20 msgr2=0x7f542006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:55.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.986+0000 7f543be00700 1 --2- 172.21.15.89:0/972827674 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f542006af20 0x7f542006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:13:55.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:55.991+0000 7f543aa00700 1 -- 172.21.15.89:0/972827674 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5434054480 con 0x7f54440cee10 2024-04-02T09:13:56.030 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:56.032+0000 7fca907b9240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:13:56.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.187+0000 7f543be00700 1 -- 172.21.15.89:0/972827674 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f542006af20 msgr2=0x7f542006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:56.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.187+0000 7f543be00700 1 --2- 172.21.15.89:0/972827674 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f542006af20 0x7f542006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:13:56.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.199+0000 7f5449800700 1 -- 172.21.15.89:0/972827674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f54440b5d00 con 0x7f54440cee10 2024-04-02T09:13:56.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.203+0000 7f543aa00700 1 -- 172.21.15.89:0/972827674 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v84)=0 v84) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5434057aa0 con 0x7f54440cee10 2024-04-02T09:13:56.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.206+0000 7f5449800700 1 -- 172.21.15.89:0/972827674 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f542006af20 msgr2=0x7f542006d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:13:56.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.206+0000 7f5449800700 1 --2- 172.21.15.89:0/972827674 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f542006af20 0x7f542006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:56.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.206+0000 7f5449800700 1 -- 172.21.15.89:0/972827674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54440cee10 msgr2=0x7f54440c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:56.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.206+0000 7f5449800700 1 --2- 172.21.15.89:0/972827674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54440cee10 0x7f54440c9830 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f543400e870 tx=0x7f543400eb80 comp rx=0 tx=0).stop 2024-04-02T09:13:56.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.206+0000 7f5449800700 1 -- 172.21.15.89:0/972827674 shutdown_connections 2024-04-02T09:13:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.206+0000 7f5449800700 1 --2- 172.21.15.89:0/972827674 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f542006af20 0x7f542006d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.206+0000 7f5449800700 1 --2- 172.21.15.89:0/972827674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54441611b0 0x7f54440c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.206+0000 7f5449800700 1 --2- 172.21.15.89:0/972827674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54440cee10 0x7f54440c9830 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.206+0000 7f5449800700 1 --2- 172.21.15.89:0/972827674 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f544415f4d0 0x7f54440b8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.206+0000 7f5449800700 1 -- 172.21.15.89:0/972827674 >> 172.21.15.89:0/972827674 conn(0x7f544408aff0 msgr2=0x7f544415da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.206+0000 7f5449800700 1 -- 172.21.15.89:0/972827674 shutdown_connections 2024-04-02T09:13:56.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.206+0000 7f5449800700 1 -- 172.21.15.89:0/972827674 wait complete. 2024-04-02T09:13:56.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:13:56.223 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-04-02T09:13:56.273 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2024-04-02T09:13:56.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.476+0000 7fca9bc00700 1 Processor -- start 2024-04-02T09:13:56.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.476+0000 7fca9bc00700 1 -- start start 2024-04-02T09:13:56.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.477+0000 7fca9bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca940d16a0 0x7fca940d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.477+0000 7fca9bc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fca940d2370 0x7fca940ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.477+0000 7fca9bc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca94161420 con 0x7fca9415d9d0 2024-04-02T09:13:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.477+0000 7fca9bc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca940d2820 con 0x7fca940d2370 2024-04-02T09:13:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.477+0000 7fca9bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca940d4340 con 0x7fca940d16a0 2024-04-02T09:13:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.478+0000 7fca99c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fca940d2370 0x7fca940ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.478+0000 7fca99200700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fca9415d9d0 0x7fca940d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:48278/0 (socket says 172.21.15.89:48278) 2024-04-02T09:13:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.478+0000 7fca99200700 1 -- 172.21.15.89:0/1713254080 learned_addr learned my addr 172.21.15.89:0/1713254080 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:13:56.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.478+0000 7fca93e00700 1 --2- 172.21.15.89:0/1713254080 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca940d16a0 0x7fca940d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:56.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.478+0000 7fca99c00700 1 -- 172.21.15.89:0/1713254080 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca940d16a0 msgr2=0x7fca940d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:13:56.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.478+0000 7fca99c00700 1 --2- 172.21.15.89:0/1713254080 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca940d16a0 0x7fca940d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:56.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.478+0000 7fca99c00700 1 -- 172.21.15.89:0/1713254080 >> v1:172.21.15.113:6789/0 conn(0x7fca9415d9d0 legacy=0x7fca940d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:56.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.478+0000 7fca99c00700 1 -- 172.21.15.89:0/1713254080 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca94161700 con 0x7fca940d2370 2024-04-02T09:13:56.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.478+0000 7fca93400700 1 -- 172.21.15.89:0/1713254080 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538571644 0 0) 0x7fca94161420 con 0x7fca9415d9d0 2024-04-02T09:13:56.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.478+0000 7fca99c00700 1 --2- 172.21.15.89:0/1713254080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca940d2370 0x7fca940ce360 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fca8c009dc0 tx=0x7fca8c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f04ec1bbc3f5809 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:56.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.479+0000 7fca93400700 1 -- 172.21.15.89:0/1713254080 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c00e070 con 0x7fca940d2370 2024-04-02T09:13:56.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.479+0000 7fca93400700 1 -- 172.21.15.89:0/1713254080 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8c00bae0 con 0x7fca940d2370 2024-04-02T09:13:56.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.479+0000 7fca93400700 1 -- 172.21.15.89:0/1713254080 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c00be90 con 0x7fca940d2370 2024-04-02T09:13:56.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.479+0000 7fca9bc00700 1 -- 172.21.15.89:0/1713254080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca940d2370 msgr2=0x7fca940ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:56.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.479+0000 7fca9bc00700 1 --2- 172.21.15.89:0/1713254080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca940d2370 0x7fca940ce360 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fca8c009dc0 tx=0x7fca8c00b330 comp rx=0 tx=0).stop 2024-04-02T09:13:56.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.479+0000 7fca9bc00700 1 -- 172.21.15.89:0/1713254080 shutdown_connections 2024-04-02T09:13:56.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.480+0000 7fca9bc00700 1 --2- 172.21.15.89:0/1713254080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca940d2370 0x7fca940ce360 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:56.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.480+0000 7fca9bc00700 1 --2- 172.21.15.89:0/1713254080 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca940d16a0 0x7fca940d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:56.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.480+0000 7fca9bc00700 1 -- 172.21.15.89:0/1713254080 >> 172.21.15.89:0/1713254080 conn(0x7fca9408aff0 msgr2=0x7fca940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:13:56.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.480+0000 7fca9bc00700 1 -- 172.21.15.89:0/1713254080 shutdown_connections 2024-04-02T09:13:56.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.480+0000 7fca9bc00700 1 -- 172.21.15.89:0/1713254080 wait complete. 2024-04-02T09:13:56.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.482+0000 7fca9bc00700 1 Processor -- start 2024-04-02T09:13:56.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.483+0000 7fca9bc00700 1 -- start start 2024-04-02T09:13:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.484+0000 7fca9bc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca940d16a0 0x7fca941f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.484+0000 7fca9bc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca940d2370 0x7fca9415f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.484+0000 7fca9bc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca9415d9d0 0x7fca940b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.484+0000 7fca9bc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca94160a80 con 0x7fca940d16a0 2024-04-02T09:13:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.484+0000 7fca9bc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca941f64e0 con 0x7fca9415d9d0 2024-04-02T09:13:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.484+0000 7fca9bc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca9415fe00 con 0x7fca940d2370 2024-04-02T09:13:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.484+0000 7fca93e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca940d2370 0x7fca9415f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.485+0000 7fca99c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca9415d9d0 0x7fca940b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.485+0000 7fca93e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca940d2370 0x7fca9415f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:60978/0 (socket says 172.21.15.89:60978) 2024-04-02T09:13:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.485+0000 7fca93e00700 1 -- 172.21.15.89:0/3455611700 learned_addr learned my addr 172.21.15.89:0/3455611700 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:13:56.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.485+0000 7fca99200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca940d16a0 0x7fca941f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:13:56.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.485+0000 7fca93e00700 1 -- 172.21.15.89:0/3455611700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca9415d9d0 msgr2=0x7fca940b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:56.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.485+0000 7fca93e00700 1 --2- 172.21.15.89:0/3455611700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca9415d9d0 0x7fca940b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:56.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.485+0000 7fca93e00700 1 -- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca940d16a0 msgr2=0x7fca941f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:13:56.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.485+0000 7fca93e00700 1 --2- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca940d16a0 0x7fca941f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:13:56.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.485+0000 7fca93e00700 1 -- 172.21.15.89:0/3455611700 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8c009a70 con 0x7fca940d2370 2024-04-02T09:13:56.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.485+0000 7fca93e00700 1 --2- 172.21.15.89:0/3455611700 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca940d2370 0x7fca9415f420 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fca8000e960 tx=0x7fca8000ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:13:56.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.485+0000 7fca90c00700 1 -- 172.21.15.89:0/3455611700 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca800096f0 con 0x7fca940d2370 2024-04-02T09:13:56.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.486+0000 7fca90c00700 1 -- 172.21.15.89:0/3455611700 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca80013680 con 0x7fca940d2370 2024-04-02T09:13:56.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.486+0000 7fca90c00700 1 -- 172.21.15.89:0/3455611700 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca80013a30 con 0x7fca940d2370 2024-04-02T09:13:56.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.486+0000 7fca9bc00700 1 -- 172.21.15.89:0/3455611700 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca941600e0 con 0x7fca940d2370 2024-04-02T09:13:56.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.486+0000 7fca9bc00700 1 -- 172.21.15.89:0/3455611700 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca941606a0 con 0x7fca940d2370 2024-04-02T09:13:56.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.487+0000 7fca9bc00700 1 -- 172.21.15.89:0/3455611700 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca940d1ca0 con 0x7fca940d2370 2024-04-02T09:13:56.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.488+0000 7fca90c00700 1 -- 172.21.15.89:0/3455611700 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 159) v1 ==== 89358+0+0 (secure 0 0 0) 0x7fca80013b90 con 0x7fca940d2370 2024-04-02T09:13:56.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.489+0000 7fca90c00700 1 --2- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 0x7fca7406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:13:56.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.489+0000 7fca99200700 1 -- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 msgr2=0x7fca7406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:56.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.489+0000 7fca90c00700 1 -- 172.21.15.89:0/3455611700 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fca80089fd0 con 0x7fca940d2370 2024-04-02T09:13:56.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.489+0000 7fca99200700 1 --2- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 0x7fca7406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:13:56.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.493+0000 7fca90c00700 1 -- 172.21.15.89:0/3455611700 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fca80054920 con 0x7fca940d2370 2024-04-02T09:13:56.508 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:56.510+0000 7fc6098b6240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:13:56.595 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:56.596+0000 7fca907b9240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:13:56.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.690+0000 7fca99200700 1 -- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 msgr2=0x7fca7406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:56.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.690+0000 7fca99200700 1 --2- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 0x7fca7406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:13:56.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:56.730+0000 7fca9bc00700 1 -- 172.21.15.89:0/3455611700 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fca94161fa0 con 0x7fca7406af20 2024-04-02T09:13:56.762 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:56.764+0000 7fc6098b6240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:13:56.853 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:56.855+0000 7fca907b9240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:13:57.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:57.091+0000 7fca99200700 1 -- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 msgr2=0x7fca7406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:57.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:57.091+0000 7fca99200700 1 --2- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 0x7fca7406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T09:13:57.216 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:57.218+0000 7fc6098b6240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:13:57.310 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:57.312+0000 7fca907b9240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:13:57.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:57.892+0000 7fca99200700 1 -- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 msgr2=0x7fca7406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:57.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:57.892+0000 7fca99200700 1 --2- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 0x7fca7406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T09:13:57.923 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:57.925+0000 7fc6098b6240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:13:58.018 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:58.021+0000 7fc6098b6240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:13:58.025 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:58.027+0000 7fca907b9240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:13:58.117 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:58.119+0000 7fca907b9240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:13:58.176 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:58.178+0000 7fc6098b6240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:13:58.277 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:58.279+0000 7fca907b9240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:13:58.389 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:58.392+0000 7fc6098b6240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:13:58.481 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:58.483+0000 7fc6098b6240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:13:58.490 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:58.492+0000 7fca907b9240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:13:58.580 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:58.582+0000 7fc6098b6240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:13:58.584 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:58.586+0000 7fca907b9240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:13:58.677 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:58.679+0000 7fc6098b6240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:13:58.683 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:58.685+0000 7fca907b9240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:13:58.774 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:58.777+0000 7fc6098b6240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:13:58.781 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:58.783+0000 7fca907b9240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:13:58.880 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:58.882+0000 7fca907b9240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:13:58.986 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:58.988+0000 7fc6098b6240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:13:59.095 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:59.097+0000 7fca907b9240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:13:59.242 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:59.245+0000 7fc6098b6240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:13:59.352 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:59.353+0000 7fca907b9240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:13:59.434 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:59.436+0000 7fc6098b6240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:13:59.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:59.494+0000 7fca99200700 1 -- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 msgr2=0x7fca7406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:13:59.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:13:59.494+0000 7fca99200700 1 --2- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 0x7fca7406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T09:13:59.526 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:13:59.528+0000 7fc6098b6240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:13:59.545 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:59.546+0000 7fca907b9240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:13:59.638 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:13:59.640+0000 7fca907b9240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:14:00.256 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:00.258+0000 7fc6098b6240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:14:00.371 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:00.373+0000 7fca907b9240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:14:00.472 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:00.475+0000 7fc6098b6240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:14:00.588 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:00.590+0000 7fca907b9240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:14:00.677 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:00.679+0000 7fc6098b6240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:14:00.798 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:00.799+0000 7fca907b9240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:14:01.152 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:01.155+0000 7fc6098b6240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:14:01.275 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:01.277+0000 7fca907b9240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:14:01.360 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:01.363+0000 7fc6098b6240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:14:01.486 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:01.488+0000 7fca907b9240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:14:01.798 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:01.800+0000 7fc6098b6240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:14:01.909 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:01.911+0000 7fc6098b6240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:14:01.927 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:01.929+0000 7fca907b9240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:14:01.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:01.972+0000 7fca90c00700 1 -- 172.21.15.89:0/3455611700 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mgrmap(e 160) v1 ==== 89358+0+0 (secure 0 0 0) 0x7fca80054cf0 con 0x7fca940d2370 2024-04-02T09:14:02.045 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.047+0000 7fca907b9240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:14:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:02.058+0000 7fca90c00700 1 -- 172.21.15.89:0/3455611700 <== mon.2 v2:172.21.15.113:3301/0 8 ==== mgrmap(e 161) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fca80087e10 con 0x7fca940d2370 2024-04-02T09:14:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:02.058+0000 7fca90c00700 1 -- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 msgr2=0x7fca7406d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:02.058+0000 7fca90c00700 1 --2- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 0x7fca7406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:02.312 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.314+0000 7fc9f8600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.312 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.314+0000 7fc9f8600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.312 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.314+0000 7fc9f8600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.312 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.314+0000 7fc9f8600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.312 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.314+0000 7fc9f8600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.312 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.314+0000 7fc9fb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.313 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.314+0000 7fc9fb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.313 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.314+0000 7fc9fb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.313 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.314+0000 7fc9fb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.313 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.314+0000 7fc9fb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.325 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.327+0000 7fc9fb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.325 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.327+0000 7fc9fb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.325 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.327+0000 7fc9fb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.325 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.327+0000 7fc9fb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.325 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.327+0000 7fc9fb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.326 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.328+0000 7fc9f8600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.326 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.328+0000 7fc9f8600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.326 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.328+0000 7fc9f8600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.326 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.328+0000 7fc9f8600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:02.326 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:02.328+0000 7fc9f8600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:03.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.069+0000 7fca90c00700 1 -- 172.21.15.89:0/3455611700 <== mon.2 v2:172.21.15.113:3301/0 9 ==== mgrmap(e 162) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fca800555c0 con 0x7fca940d2370 2024-04-02T09:14:03.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.070+0000 7fca90c00700 1 --2- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 0x7fca7406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:03.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.070+0000 7fca90c00700 1 -- 172.21.15.89:0/3455611700 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fca7401d3f0 con 0x7fca7406af20 2024-04-02T09:14:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.071+0000 7fca99200700 1 --2- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 0x7fca7406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.074+0000 7fca99200700 1 --2- 172.21.15.89:0/3455611700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca7406af20 0x7fca7406d3e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fca88000f60 tx=0x7fca88007e70 comp rx=0 tx=0).ready entity=mgr.12312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:03.077 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:03.077+0000 7fca04e00700 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2024-04-02T09:14:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.077+0000 7fca90c00700 1 -- 172.21.15.89:0/3455611700 <== mgr.12312 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) v1 ==== 79+0+0 (secure 0 0 0) 0x7fca7401d3f0 con 0x7fca7406af20 2024-04-02T09:14:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2024-04-02T09:14:03.085 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-02T09:14:03.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:03.086 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-04-02T09:14:03.134 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T09:14:03.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.330+0000 7fb38f400700 1 Processor -- start 2024-04-02T09:14:03.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.330+0000 7fb38f400700 1 -- start start 2024-04-02T09:14:03.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.331+0000 7fb38f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3880ce230 0x7fb3880ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:03.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.331+0000 7fb38f400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3880709c0 con 0x7fb38815d800 2024-04-02T09:14:03.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.331+0000 7fb38f400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb388070c80 con 0x7fb3880b2070 2024-04-02T09:14:03.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.331+0000 7fb38f400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3880b40f0 con 0x7fb3880ce230 2024-04-02T09:14:03.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb38ca00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb38815d800 0x7fb3880725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34118/0 (socket says 172.21.15.89:34118) 2024-04-02T09:14:03.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb38ca00700 1 -- 172.21.15.89:0/4034643572 learned_addr learned my addr 172.21.15.89:0/4034643572 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:03.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387e00700 1 --2- 172.21.15.89:0/4034643572 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3880ce230 0x7fb3880ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:03.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387400700 1 -- 172.21.15.89:0/4034643572 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591227841 0 0) 0x7fb3880709c0 con 0x7fb38815d800 2024-04-02T09:14:03.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387400700 1 -- 172.21.15.89:0/4034643572 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3700035b0 con 0x7fb38815d800 2024-04-02T09:14:03.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387400700 1 -- 172.21.15.89:0/4034643572 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3768551678 0 0) 0x7fb3700035b0 con 0x7fb38815d800 2024-04-02T09:14:03.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387400700 1 -- 172.21.15.89:0/4034643572 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3880ce230 msgr2=0x7fb3880ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:03.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387400700 1 --2- 172.21.15.89:0/4034643572 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3880ce230 0x7fb3880ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:03.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387400700 1 -- 172.21.15.89:0/4034643572 >> v1:172.21.15.113:6789/0 conn(0x7fb3880b2070 legacy=0x7fb38815dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:03.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387400700 1 -- 172.21.15.89:0/4034643572 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb388070f60 con 0x7fb38815d800 2024-04-02T09:14:03.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387400700 1 -- 172.21.15.89:0/4034643572 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fb37c001670 con 0x7fb38815d800 2024-04-02T09:14:03.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387400700 1 -- 172.21.15.89:0/4034643572 >> v1:172.21.15.89:6789/0 conn(0x7fb38815d800 legacy=0x7fb3880725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:03.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387400700 1 --2- 172.21.15.89:0/4034643572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb370003c00 0x7fb370004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:03.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387400700 1 --2- 172.21.15.89:0/4034643572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb370008870 0x7fb37000ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:03.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387400700 1 --2- 172.21.15.89:0/4034643572 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3880ce230 0x7fb37000d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:03.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387400700 1 -- 172.21.15.89:0/4034643572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3880b40f0 con 0x7fb370003c00 2024-04-02T09:14:03.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.332+0000 7fb387400700 1 -- 172.21.15.89:0/4034643572 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37000dd50 con 0x7fb370008870 2024-04-02T09:14:03.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.333+0000 7fb387400700 1 -- 172.21.15.89:0/4034643572 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37000df10 con 0x7fb3880ce230 2024-04-02T09:14:03.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.333+0000 7fb38ca00700 1 --2- 172.21.15.89:0/4034643572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb370003c00 0x7fb370004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:03.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.333+0000 7fb387e00700 1 --2- 172.21.15.89:0/4034643572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb370008870 0x7fb37000ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:03.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.333+0000 7fb38d400700 1 --2- 172.21.15.89:0/4034643572 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3880ce230 0x7fb37000d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:03.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.333+0000 7fb38ca00700 1 -- 172.21.15.89:0/4034643572 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3880ce230 msgr2=0x7fb37000d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:03.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.333+0000 7fb38ca00700 1 --2- 172.21.15.89:0/4034643572 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3880ce230 0x7fb37000d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:03.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.333+0000 7fb38ca00700 1 -- 172.21.15.89:0/4034643572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb370008870 msgr2=0x7fb37000ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:03.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.333+0000 7fb38ca00700 1 --2- 172.21.15.89:0/4034643572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb370008870 0x7fb37000ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:03.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.333+0000 7fb38ca00700 1 -- 172.21.15.89:0/4034643572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3700035b0 con 0x7fb370003c00 2024-04-02T09:14:03.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.333+0000 7fb38ca00700 1 --2- 172.21.15.89:0/4034643572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb370003c00 0x7fb370004020 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00c240 tx=0x7fb370003860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8670d6229a34e99 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:03.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.333+0000 7fb387400700 1 -- 172.21.15.89:0/4034643572 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c007c10 con 0x7fb370003c00 2024-04-02T09:14:03.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.334+0000 7fb387400700 1 -- 172.21.15.89:0/4034643572 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37c00ce00 con 0x7fb370003c00 2024-04-02T09:14:03.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.334+0000 7fb387400700 1 -- 172.21.15.89:0/4034643572 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c005ba0 con 0x7fb370003c00 2024-04-02T09:14:03.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.334+0000 7fb38f400700 1 -- 172.21.15.89:0/4034643572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb370003c00 msgr2=0x7fb370004020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:03.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.334+0000 7fb38f400700 1 --2- 172.21.15.89:0/4034643572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb370003c00 0x7fb370004020 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00c240 tx=0x7fb370003860 comp rx=0 tx=0).stop 2024-04-02T09:14:03.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.334+0000 7fb38d400700 1 -- 172.21.15.89:0/4034643572 reap_dead start 2024-04-02T09:14:03.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.334+0000 7fb38f400700 1 -- 172.21.15.89:0/4034643572 shutdown_connections 2024-04-02T09:14:03.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.334+0000 7fb38f400700 1 -- 172.21.15.89:0/4034643572 >> 172.21.15.89:0/4034643572 conn(0x7fb3880b0d70 msgr2=0x7fb3880b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:03.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.334+0000 7fb38f400700 1 -- 172.21.15.89:0/4034643572 shutdown_connections 2024-04-02T09:14:03.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.334+0000 7fb38f400700 1 -- 172.21.15.89:0/4034643572 wait complete. 2024-04-02T09:14:03.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.336+0000 7fb38f400700 1 Processor -- start 2024-04-02T09:14:03.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.336+0000 7fb38f400700 1 -- start start 2024-04-02T09:14:03.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.337+0000 7fb38f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb370003c00 0x7fb3880ad990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:03.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.337+0000 7fb38f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb38806d990 0x7fb38806de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:03.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.337+0000 7fb38f400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3880ac0c0 0x7fb3880ac540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:03.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.337+0000 7fb38f400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38806e350 con 0x7fb370003c00 2024-04-02T09:14:03.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.337+0000 7fb38f400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3880aded0 con 0x7fb38806d990 2024-04-02T09:14:03.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.337+0000 7fb38f400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3881f3e20 con 0x7fb3880ac0c0 2024-04-02T09:14:03.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.337+0000 7fb38ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb370003c00 0x7fb3880ad990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:03.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.337+0000 7fb38ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb370003c00 0x7fb3880ad990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44198/0 (socket says 172.21.15.89:44198) 2024-04-02T09:14:03.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.337+0000 7fb38ca00700 1 -- 172.21.15.89:0/2305312070 learned_addr learned my addr 172.21.15.89:0/2305312070 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:03.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.337+0000 7fb38d400700 1 --2- 172.21.15.89:0/2305312070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3880ac0c0 0x7fb3880ac540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:03.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.337+0000 7fb387e00700 1 --2- 172.21.15.89:0/2305312070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb38806d990 0x7fb38806de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:03.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.338+0000 7fb38ca00700 1 -- 172.21.15.89:0/2305312070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3880ac0c0 msgr2=0x7fb3880ac540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:03.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.338+0000 7fb38ca00700 1 --2- 172.21.15.89:0/2305312070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3880ac0c0 0x7fb3880ac540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:03.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.338+0000 7fb38ca00700 1 -- 172.21.15.89:0/2305312070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb38806d990 msgr2=0x7fb38806de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:03.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.338+0000 7fb38ca00700 1 --2- 172.21.15.89:0/2305312070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb38806d990 0x7fb38806de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:03.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.338+0000 7fb38ca00700 1 -- 172.21.15.89:0/2305312070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3881f34f0 con 0x7fb370003c00 2024-04-02T09:14:03.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.338+0000 7fb38ca00700 1 --2- 172.21.15.89:0/2305312070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb370003c00 0x7fb3880ad990 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7fb37000dd20 tx=0x7fb37c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:03.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.338+0000 7fb384c00700 1 -- 172.21.15.89:0/2305312070 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c005ea0 con 0x7fb370003c00 2024-04-02T09:14:03.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.338+0000 7fb384c00700 1 -- 172.21.15.89:0/2305312070 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37c001630 con 0x7fb370003c00 2024-04-02T09:14:03.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.338+0000 7fb38f400700 1 -- 172.21.15.89:0/2305312070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3881f37d0 con 0x7fb370003c00 2024-04-02T09:14:03.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.338+0000 7fb384c00700 1 -- 172.21.15.89:0/2305312070 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c007310 con 0x7fb370003c00 2024-04-02T09:14:03.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.338+0000 7fb38f400700 1 -- 172.21.15.89:0/2305312070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3880ace10 con 0x7fb370003c00 2024-04-02T09:14:03.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.339+0000 7fb38f400700 1 -- 172.21.15.89:0/2305312070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb388067720 con 0x7fb370003c00 2024-04-02T09:14:03.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.340+0000 7fb384c00700 1 -- 172.21.15.89:0/2305312070 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 162) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fb37c019030 con 0x7fb370003c00 2024-04-02T09:14:03.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.341+0000 7fb384c00700 1 --2- 172.21.15.89:0/2305312070 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb36806ab30 0x7fb36806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:03.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.341+0000 7fb384c00700 1 -- 172.21.15.89:0/2305312070 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7fb37c08ac20 con 0x7fb370003c00 2024-04-02T09:14:03.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.341+0000 7fb387e00700 1 --2- 172.21.15.89:0/2305312070 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb36806ab30 0x7fb36806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:03.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.342+0000 7fb387e00700 1 --2- 172.21.15.89:0/2305312070 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb36806ab30 0x7fb36806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb38814f260 tx=0x7fb374009210 comp rx=0 tx=0).ready entity=mgr.12312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:03.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.346+0000 7fb384c00700 1 -- 172.21.15.89:0/2305312070 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb37c0559a0 con 0x7fb370003c00 2024-04-02T09:14:03.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:03.604+0000 7fb38f400700 1 -- 172.21.15.89:0/2305312070 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fb3880b06d0 con 0x7fb370003c00 2024-04-02T09:14:04.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.074+0000 7fb384c00700 1 -- 172.21.15.89:0/2305312070 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7fb37c055d70 con 0x7fb370003c00 2024-04-02T09:14:04.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.075+0000 7fb384c00700 1 -- 172.21.15.89:0/2305312070 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v163) v1 ==== 93+0+0 (secure 0 0 0) 0x7fb37c06e5f0 con 0x7fb370003c00 2024-04-02T09:14:04.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.078+0000 7fb38f400700 1 -- 172.21.15.89:0/2305312070 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb36806ab30 msgr2=0x7fb36806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:04.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.078+0000 7fb38f400700 1 --2- 172.21.15.89:0/2305312070 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb36806ab30 0x7fb36806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb38814f260 tx=0x7fb374009210 comp rx=0 tx=0).stop 2024-04-02T09:14:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.078+0000 7fb38f400700 1 -- 172.21.15.89:0/2305312070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb370003c00 msgr2=0x7fb3880ad990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.078+0000 7fb38f400700 1 --2- 172.21.15.89:0/2305312070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb370003c00 0x7fb3880ad990 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7fb37000dd20 tx=0x7fb37c001a00 comp rx=0 tx=0).stop 2024-04-02T09:14:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.078+0000 7fb38f400700 1 -- 172.21.15.89:0/2305312070 shutdown_connections 2024-04-02T09:14:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.078+0000 7fb38f400700 1 --2- 172.21.15.89:0/2305312070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3880ac0c0 0x7fb3880ac540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.078+0000 7fb38f400700 1 --2- 172.21.15.89:0/2305312070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb370003c00 0x7fb3880ad990 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.078+0000 7fb38f400700 1 --2- 172.21.15.89:0/2305312070 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb36806ab30 0x7fb36806cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.078+0000 7fb38f400700 1 --2- 172.21.15.89:0/2305312070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb38806d990 0x7fb38806de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.078+0000 7fb38f400700 1 -- 172.21.15.89:0/2305312070 >> 172.21.15.89:0/2305312070 conn(0x7fb3880b0d70 msgr2=0x7fb38814b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.078+0000 7fb38f400700 1 -- 172.21.15.89:0/2305312070 shutdown_connections 2024-04-02T09:14:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.078+0000 7fb38f400700 1 -- 172.21.15.89:0/2305312070 wait complete. 2024-04-02T09:14:04.097 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T09:14:04.097 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:14:04.184 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:14:04.267 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T09:14:04.267 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:14:04.267 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:14:04.295 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:04.296+0000 7f90b8167240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:14:04.308 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:04.310+0000 7f75006c9240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:14:04.315 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:14:04.315 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:14:04.405 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:14:04.483 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:04.484+0000 7f90b8167240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:14:04.503 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:04.505+0000 7f75006c9240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:14:04.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.634+0000 7ff1ac600700 1 Processor -- start 2024-04-02T09:14:04.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.634+0000 7ff1ac600700 1 -- start start 2024-04-02T09:14:04.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.636+0000 7ff1ac600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff1a40ac580 0x7ff1a406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:04.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.636+0000 7ff1ac600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1a415d670 0x7ff1a415db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:04.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.636+0000 7ff1ac600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a40b3890 con 0x7ff1a406cf10 2024-04-02T09:14:04.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.636+0000 7ff1ac600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a415e060 con 0x7ff1a40ac580 2024-04-02T09:14:04.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.636+0000 7ff1ac600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a4159f50 con 0x7ff1a415d670 2024-04-02T09:14:04.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.636+0000 7ff1a9c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff1a40ac580 0x7ff1a406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:04.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.636+0000 7ff1a9c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff1a40ac580 0x7ff1a406c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44210/0 (socket says 172.21.15.89:44210) 2024-04-02T09:14:04.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.636+0000 7ff1a9c00700 1 -- 172.21.15.89:0/3973342863 learned_addr learned my addr 172.21.15.89:0/3973342863 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:04.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.637+0000 7ff1aa600700 1 --2- 172.21.15.89:0/3973342863 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1a415d670 0x7ff1a415db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:04.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.637+0000 7ff1a9c00700 1 -- 172.21.15.89:0/3973342863 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1a415d670 msgr2=0x7ff1a415db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:04.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.637+0000 7ff1a9c00700 1 --2- 172.21.15.89:0/3973342863 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1a415d670 0x7ff1a415db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.637+0000 7ff1a9c00700 1 -- 172.21.15.89:0/3973342863 >> v1:172.21.15.113:6789/0 conn(0x7ff1a406cf10 legacy=0x7ff1a40b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:04.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.637+0000 7ff1a9c00700 1 -- 172.21.15.89:0/3973342863 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a40b35b0 con 0x7ff1a40ac580 2024-04-02T09:14:04.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.637+0000 7ff1a9c00700 1 --2- 172.21.15.89:0/3973342863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a40ac580 0x7ff1a406c9d0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7ff1980098f0 tx=0x7ff198009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df772adc93690b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:04.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.637+0000 7ff1a3e00700 1 -- 172.21.15.89:0/3973342863 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198011830 con 0x7ff1a40ac580 2024-04-02T09:14:04.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.637+0000 7ff1a3e00700 1 -- 172.21.15.89:0/3973342863 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff198011990 con 0x7ff1a40ac580 2024-04-02T09:14:04.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.637+0000 7ff1a3e00700 1 -- 172.21.15.89:0/3973342863 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198011d20 con 0x7ff1a40ac580 2024-04-02T09:14:04.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.638+0000 7ff1ac600700 1 -- 172.21.15.89:0/3973342863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a40ac580 msgr2=0x7ff1a406c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:04.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.638+0000 7ff1ac600700 1 --2- 172.21.15.89:0/3973342863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a40ac580 0x7ff1a406c9d0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7ff1980098f0 tx=0x7ff198009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:04.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.638+0000 7ff1ac600700 1 -- 172.21.15.89:0/3973342863 shutdown_connections 2024-04-02T09:14:04.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.638+0000 7ff1ac600700 1 --2- 172.21.15.89:0/3973342863 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1a415d670 0x7ff1a415db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.638+0000 7ff1ac600700 1 --2- 172.21.15.89:0/3973342863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a40ac580 0x7ff1a406c9d0 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.638+0000 7ff1ac600700 1 -- 172.21.15.89:0/3973342863 >> 172.21.15.89:0/3973342863 conn(0x7ff1a40cf190 msgr2=0x7ff1a40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:04.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.638+0000 7ff1ac600700 1 -- 172.21.15.89:0/3973342863 shutdown_connections 2024-04-02T09:14:04.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.639+0000 7ff1ac600700 1 -- 172.21.15.89:0/3973342863 wait complete. 2024-04-02T09:14:04.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.640+0000 7ff1ac600700 1 Processor -- start 2024-04-02T09:14:04.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.640+0000 7ff1ac600700 1 -- start start 2024-04-02T09:14:04.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.641+0000 7ff1ac600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1a406cf10 0x7ff1a40c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:04.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.641+0000 7ff1ac600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1a40ac580 0x7ff1a4152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:04.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.641+0000 7ff1ac600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a415d670 0x7ff1a40cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:04.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.641+0000 7ff1ac600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a40b05c0 con 0x7ff1a415d670 2024-04-02T09:14:04.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.641+0000 7ff1ac600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a40ca5a0 con 0x7ff1a406cf10 2024-04-02T09:14:04.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.641+0000 7ff1ac600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a40c9500 con 0x7ff1a40ac580 2024-04-02T09:14:04.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.642+0000 7ff1aa600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a415d670 0x7ff1a40cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:04.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.642+0000 7ff1aa600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a415d670 0x7ff1a40cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44220/0 (socket says 172.21.15.89:44220) 2024-04-02T09:14:04.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.642+0000 7ff1aa600700 1 -- 172.21.15.89:0/321545440 learned_addr learned my addr 172.21.15.89:0/321545440 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:04.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.642+0000 7ff1a9c00700 1 --2- 172.21.15.89:0/321545440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1a406cf10 0x7ff1a40c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:04.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.642+0000 7ff1a9200700 1 --2- 172.21.15.89:0/321545440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1a40ac580 0x7ff1a4152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:04.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.642+0000 7ff1aa600700 1 -- 172.21.15.89:0/321545440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1a40ac580 msgr2=0x7ff1a4152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:04.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.642+0000 7ff1aa600700 1 --2- 172.21.15.89:0/321545440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1a40ac580 0x7ff1a4152590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.642+0000 7ff1aa600700 1 -- 172.21.15.89:0/321545440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1a406cf10 msgr2=0x7ff1a40c8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:04.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.642+0000 7ff1aa600700 1 --2- 172.21.15.89:0/321545440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1a406cf10 0x7ff1a40c8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.642+0000 7ff1aa600700 1 -- 172.21.15.89:0/321545440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff198009550 con 0x7ff1a415d670 2024-04-02T09:14:04.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.642+0000 7ff1aa600700 1 --2- 172.21.15.89:0/321545440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a415d670 0x7ff1a40cda50 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00efc0 tx=0x7ff19c00c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:04.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.643+0000 7ff1a1600700 1 -- 172.21.15.89:0/321545440 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c00cad0 con 0x7ff1a415d670 2024-04-02T09:14:04.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.643+0000 7ff1a1600700 1 -- 172.21.15.89:0/321545440 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19c00f040 con 0x7ff1a415d670 2024-04-02T09:14:04.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.643+0000 7ff1a1600700 1 -- 172.21.15.89:0/321545440 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c00ce70 con 0x7ff1a415d670 2024-04-02T09:14:04.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.643+0000 7ff1ac600700 1 -- 172.21.15.89:0/321545440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a40c97e0 con 0x7ff1a415d670 2024-04-02T09:14:04.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.643+0000 7ff1ac600700 1 -- 172.21.15.89:0/321545440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a40c9da0 con 0x7ff1a415d670 2024-04-02T09:14:04.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.644+0000 7ff1ac600700 1 -- 172.21.15.89:0/321545440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a415e380 con 0x7ff1a415d670 2024-04-02T09:14:04.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.645+0000 7ff1a1600700 1 -- 172.21.15.89:0/321545440 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7ff19c014480 con 0x7ff1a415d670 2024-04-02T09:14:04.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.645+0000 7ff1a1600700 1 --2- 172.21.15.89:0/321545440 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff18406aed0 0x7ff18406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:04.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.646+0000 7ff1a1600700 1 -- 172.21.15.89:0/321545440 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7ff19c08a700 con 0x7ff1a415d670 2024-04-02T09:14:04.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.646+0000 7ff1a9c00700 1 -- 172.21.15.89:0/321545440 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff18406aed0 msgr2=0x7ff18406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:04.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.646+0000 7ff1a9c00700 1 --2- 172.21.15.89:0/321545440 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff18406aed0 0x7ff18406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:04.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.650+0000 7ff1a1600700 1 -- 172.21.15.89:0/321545440 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff19c054d00 con 0x7ff1a415d670 2024-04-02T09:14:04.801 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:04.802+0000 7f90b8167240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:14:04.836 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:04.838+0000 7f75006c9240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:14:04.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.847+0000 7ff1a9c00700 1 -- 172.21.15.89:0/321545440 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff18406aed0 msgr2=0x7ff18406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:04.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.847+0000 7ff1a9c00700 1 --2- 172.21.15.89:0/321545440 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff18406aed0 0x7ff18406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:04.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.860+0000 7ff1ac600700 1 -- 172.21.15.89:0/321545440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1a4066ff0 con 0x7ff1a415d670 2024-04-02T09:14:04.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.861+0000 7ff1a1600700 1 -- 172.21.15.89:0/321545440 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) v1 ==== 74+0+21019 (secure 0 0 0) 0x7ff19c058320 con 0x7ff1a415d670 2024-04-02T09:14:04.860 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:04.860 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":249,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:14:02.049435+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:13:26.654647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-02T09:13:27.342285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:13:40.048039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:13:40.907706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/1998199546":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/492795889":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/3750385348":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/2067457213":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/1275405725":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/3355557522":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/4204293224":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/176419408":"2024-04-03T09:14:02.049396+0000","172.21.15.89:6833/108855":"2024-04-03T09:14:02.049396+0000","172.21.15.89:6832/108855":"2024-04-03T09:14:02.049396+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:14:04.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.864+0000 7ff1ac600700 1 -- 172.21.15.89:0/321545440 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff18406aed0 msgr2=0x7ff18406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:04.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.864+0000 7ff1ac600700 1 --2- 172.21.15.89:0/321545440 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff18406aed0 0x7ff18406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.864+0000 7ff1ac600700 1 -- 172.21.15.89:0/321545440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a415d670 msgr2=0x7ff1a40cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:04.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.864+0000 7ff1ac600700 1 --2- 172.21.15.89:0/321545440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a415d670 0x7ff1a40cda50 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00efc0 tx=0x7ff19c00c0c0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.864+0000 7ff1ac600700 1 -- 172.21.15.89:0/321545440 shutdown_connections 2024-04-02T09:14:04.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.864+0000 7ff1ac600700 1 --2- 172.21.15.89:0/321545440 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff18406aed0 0x7ff18406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.864+0000 7ff1ac600700 1 --2- 172.21.15.89:0/321545440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1a406cf10 0x7ff1a40c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.864+0000 7ff1ac600700 1 --2- 172.21.15.89:0/321545440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1a415d670 0x7ff1a40cda50 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.864+0000 7ff1ac600700 1 --2- 172.21.15.89:0/321545440 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff1a40ac580 0x7ff1a4152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:04.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.864+0000 7ff1ac600700 1 -- 172.21.15.89:0/321545440 >> 172.21.15.89:0/321545440 conn(0x7ff1a40cf190 msgr2=0x7ff1a4159040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:04.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.864+0000 7ff1ac600700 1 -- 172.21.15.89:0/321545440 shutdown_connections 2024-04-02T09:14:04.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:04.864+0000 7ff1ac600700 1 -- 172.21.15.89:0/321545440 wait complete. 2024-04-02T09:14:04.881 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T09:13:26.654647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '243', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T09:13:27.342285+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '244', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-02T09:13:40.048039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-02T09:13:40.907706+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:14:04.882 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:14:05.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.090+0000 7f8cbc600700 1 Processor -- start 2024-04-02T09:14:05.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.091+0000 7f8cbc600700 1 -- start start 2024-04-02T09:14:05.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.092+0000 7f8cbc600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8cb40cd020 0x7f8cb40cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:05.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.092+0000 7f8cbc600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8cb40b2070 0x7f8cb40b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:05.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.092+0000 7f8cbc600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb40cdae0 con 0x7f8cb40cbb30 2024-04-02T09:14:05.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.092+0000 7f8cbc600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb40cddc0 con 0x7f8cb40cd020 2024-04-02T09:14:05.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.092+0000 7f8cbc600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb40ce0a0 con 0x7f8cb40b2070 2024-04-02T09:14:05.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.092+0000 7f8cb9200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8cb40cd020 0x7f8cb40cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:05.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.092+0000 7f8cb9200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8cb40cd020 0x7f8cb40cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44222/0 (socket says 172.21.15.89:44222) 2024-04-02T09:14:05.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.092+0000 7f8cb9200700 1 -- 172.21.15.89:0/934079165 learned_addr learned my addr 172.21.15.89:0/934079165 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:05.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.092+0000 7f8cba600700 1 --2- 172.21.15.89:0/934079165 >> v2:172.21.15.113:3300/0 conn(0x7f8cb40b2070 0x7f8cb40b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:05.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.092+0000 7f8cb9200700 1 -- 172.21.15.89:0/934079165 >> v2:172.21.15.113:3300/0 conn(0x7f8cb40b2070 msgr2=0x7f8cb40b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:05.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.092+0000 7f8cb9200700 1 --2- 172.21.15.89:0/934079165 >> v2:172.21.15.113:3300/0 conn(0x7f8cb40b2070 0x7f8cb40b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.092+0000 7f8cb9200700 1 -- 172.21.15.89:0/934079165 >> v1:172.21.15.89:6789/0 conn(0x7f8cb40cbb30 legacy=0x7f8cb40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:05.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.092+0000 7f8cb9200700 1 -- 172.21.15.89:0/934079165 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb406dc20 con 0x7f8cb40cd020 2024-04-02T09:14:05.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.093+0000 7f8cb9200700 1 --2- 172.21.15.89:0/934079165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8cb40cd020 0x7f8cb40cd480 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f8cb40b1d60 tx=0x7f8ca0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1658371b99edd0e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:05.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.093+0000 7f8cb3e00700 1 -- 172.21.15.89:0/934079165 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca000b870 con 0x7f8cb40cd020 2024-04-02T09:14:05.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.093+0000 7f8cb3e00700 1 -- 172.21.15.89:0/934079165 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca000b9d0 con 0x7f8cb40cd020 2024-04-02T09:14:05.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.093+0000 7f8cb3e00700 1 -- 172.21.15.89:0/934079165 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca000bd60 con 0x7f8cb40cd020 2024-04-02T09:14:05.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.094+0000 7f8cbc600700 1 -- 172.21.15.89:0/934079165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8cb40cd020 msgr2=0x7f8cb40cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:05.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.094+0000 7f8cbc600700 1 --2- 172.21.15.89:0/934079165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8cb40cd020 0x7f8cb40cd480 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f8cb40b1d60 tx=0x7f8ca0009bd0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.094+0000 7f8cbc600700 1 -- 172.21.15.89:0/934079165 shutdown_connections 2024-04-02T09:14:05.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.094+0000 7f8cbc600700 1 --2- 172.21.15.89:0/934079165 >> v2:172.21.15.113:3300/0 conn(0x7f8cb40b2070 0x7f8cb40b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.094+0000 7f8cbc600700 1 --2- 172.21.15.89:0/934079165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8cb40cd020 0x7f8cb40cd480 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.094+0000 7f8cbc600700 1 -- 172.21.15.89:0/934079165 >> 172.21.15.89:0/934079165 conn(0x7f8cb40b0d70 msgr2=0x7f8cb40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:05.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.094+0000 7f8cbc600700 1 -- 172.21.15.89:0/934079165 shutdown_connections 2024-04-02T09:14:05.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.094+0000 7f8cbc600700 1 -- 172.21.15.89:0/934079165 wait complete. 2024-04-02T09:14:05.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.095+0000 7f8cbc600700 1 Processor -- start 2024-04-02T09:14:05.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.096+0000 7f8cbc600700 1 -- start start 2024-04-02T09:14:05.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.096+0000 7f8cbc600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cb40b2070 0x7f8cb4071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:05.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.096+0000 7f8cbc600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8cb40cbb30 0x7f8cb414e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:05.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.097+0000 7f8cbc600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8cb40cd020 0x7f8cb415e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:05.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.097+0000 7f8cbc600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb4072130 con 0x7f8cb40cbb30 2024-04-02T09:14:05.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.097+0000 7f8cbc600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb40c9430 con 0x7f8cb40b2070 2024-04-02T09:14:05.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.097+0000 7f8cbc600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb414ead0 con 0x7f8cb40cd020 2024-04-02T09:14:05.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.097+0000 7f8cb9200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8cb40cbb30 0x7f8cb414e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:05.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.097+0000 7f8cb9200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8cb40cbb30 0x7f8cb414e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44234/0 (socket says 172.21.15.89:44234) 2024-04-02T09:14:05.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.097+0000 7f8cb9200700 1 -- 172.21.15.89:0/2488239135 learned_addr learned my addr 172.21.15.89:0/2488239135 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:05.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.097+0000 7f8cb9c00700 1 --2- 172.21.15.89:0/2488239135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cb40b2070 0x7f8cb4071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:05.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.097+0000 7f8cba600700 1 --2- 172.21.15.89:0/2488239135 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8cb40cd020 0x7f8cb415e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:05.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.097+0000 7f8cb9200700 1 -- 172.21.15.89:0/2488239135 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8cb40cd020 msgr2=0x7f8cb415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:05.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.097+0000 7f8cb9200700 1 --2- 172.21.15.89:0/2488239135 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8cb40cd020 0x7f8cb415e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.097+0000 7f8cb9200700 1 -- 172.21.15.89:0/2488239135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cb40b2070 msgr2=0x7f8cb4071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:05.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.097+0000 7f8cb9200700 1 --2- 172.21.15.89:0/2488239135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cb40b2070 0x7f8cb4071bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.097+0000 7f8cb9200700 1 -- 172.21.15.89:0/2488239135 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca0009550 con 0x7f8cb40cbb30 2024-04-02T09:14:05.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.098+0000 7f8cb9200700 1 --2- 172.21.15.89:0/2488239135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8cb40cbb30 0x7f8cb414e590 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0014040 tx=0x7f8ca0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:05.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.098+0000 7f8cb1600700 1 -- 172.21.15.89:0/2488239135 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0004310 con 0x7f8cb40cbb30 2024-04-02T09:14:05.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.098+0000 7f8cb1600700 1 -- 172.21.15.89:0/2488239135 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca001e070 con 0x7f8cb40cbb30 2024-04-02T09:14:05.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.098+0000 7f8cbc600700 1 -- 172.21.15.89:0/2488239135 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb41f33b0 con 0x7f8cb40cbb30 2024-04-02T09:14:05.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.098+0000 7f8cb1600700 1 -- 172.21.15.89:0/2488239135 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca00036a0 con 0x7f8cb40cbb30 2024-04-02T09:14:05.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.098+0000 7f8cbc600700 1 -- 172.21.15.89:0/2488239135 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb41f3970 con 0x7f8cb40cbb30 2024-04-02T09:14:05.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.099+0000 7f8cbc600700 1 -- 172.21.15.89:0/2488239135 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb4067720 con 0x7f8cb40cbb30 2024-04-02T09:14:05.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.100+0000 7f8cb1600700 1 -- 172.21.15.89:0/2488239135 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f8ca0020030 con 0x7f8cb40cbb30 2024-04-02T09:14:05.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.101+0000 7f8cb1600700 1 --2- 172.21.15.89:0/2488239135 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8c9406aed0 0x7f8c9406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:05.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.101+0000 7f8cb1600700 1 -- 172.21.15.89:0/2488239135 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f8ca008ade0 con 0x7f8cb40cbb30 2024-04-02T09:14:05.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.101+0000 7f8cb9c00700 1 -- 172.21.15.89:0/2488239135 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8c9406aed0 msgr2=0x7f8c9406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:05.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.101+0000 7f8cb9c00700 1 --2- 172.21.15.89:0/2488239135 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8c9406aed0 0x7f8c9406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:05.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.105+0000 7f8cb1600700 1 -- 172.21.15.89:0/2488239135 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ca0055790 con 0x7f8cb40cbb30 2024-04-02T09:14:05.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.302+0000 7f8cb9c00700 1 -- 172.21.15.89:0/2488239135 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8c9406aed0 msgr2=0x7f8c9406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:05.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.302+0000 7f8cb9c00700 1 --2- 172.21.15.89:0/2488239135 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8c9406aed0 0x7f8c9406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.315+0000 7f8cbc600700 1 -- 172.21.15.89:0/2488239135 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8cb40ce2e0 con 0x7f8cb40cbb30 2024-04-02T09:14:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.315+0000 7f8cb1600700 1 -- 172.21.15.89:0/2488239135 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v249) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8ca0058db0 con 0x7f8cb40cbb30 2024-04-02T09:14:05.313 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:14:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.317+0000 7f8cbc600700 1 -- 172.21.15.89:0/2488239135 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8c9406aed0 msgr2=0x7f8c9406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.317+0000 7f8cbc600700 1 --2- 172.21.15.89:0/2488239135 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8c9406aed0 0x7f8c9406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.317+0000 7f8cbc600700 1 -- 172.21.15.89:0/2488239135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8cb40cbb30 msgr2=0x7f8cb414e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.317+0000 7f8cbc600700 1 --2- 172.21.15.89:0/2488239135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8cb40cbb30 0x7f8cb414e590 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0014040 tx=0x7f8ca0004750 comp rx=0 tx=0).stop 2024-04-02T09:14:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.318+0000 7f8cbc600700 1 -- 172.21.15.89:0/2488239135 shutdown_connections 2024-04-02T09:14:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.318+0000 7f8cbc600700 1 --2- 172.21.15.89:0/2488239135 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8c9406aed0 0x7f8c9406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.318+0000 7f8cbc600700 1 --2- 172.21.15.89:0/2488239135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cb40b2070 0x7f8cb4071bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.318+0000 7f8cbc600700 1 --2- 172.21.15.89:0/2488239135 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8cb40cd020 0x7f8cb415e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.318+0000 7f8cbc600700 1 --2- 172.21.15.89:0/2488239135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8cb40cbb30 0x7f8cb414e590 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.318+0000 7f8cbc600700 1 -- 172.21.15.89:0/2488239135 >> 172.21.15.89:0/2488239135 conn(0x7f8cb40b0d70 msgr2=0x7f8cb40aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.318+0000 7f8cbc600700 1 -- 172.21.15.89:0/2488239135 shutdown_connections 2024-04-02T09:14:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.318+0000 7f8cbc600700 1 -- 172.21.15.89:0/2488239135 wait complete. 2024-04-02T09:14:05.327 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:05.328+0000 7f90b8167240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:14:05.334 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T09:14:05.401 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:05.403+0000 7f75006c9240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:14:05.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.537+0000 7fb9ca000700 1 Processor -- start 2024-04-02T09:14:05.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.537+0000 7fb9ca000700 1 -- start start 2024-04-02T09:14:05.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.538+0000 7fb9ca000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb9c4164330 0x7fb9c4166720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:05.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.538+0000 7fb9ca000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c40cc870 con 0x7fb9c40cec00 2024-04-02T09:14:05.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.538+0000 7fb9ca000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c40cbca0 con 0x7fb9c415b760 2024-04-02T09:14:05.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.538+0000 7fb9ca000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c40b4570 con 0x7fb9c4164330 2024-04-02T09:14:05.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.538+0000 7fb9c2a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb9c40cec00 0x7fb9c40cf080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34138/0 (socket says 172.21.15.89:34138) 2024-04-02T09:14:05.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.538+0000 7fb9c2a00700 1 -- 172.21.15.89:0/2632763705 learned_addr learned my addr 172.21.15.89:0/2632763705 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:05.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.538+0000 7fb9c3e00700 1 --2- 172.21.15.89:0/2632763705 >> v2:172.21.15.89:3300/0 conn(0x7fb9c4164330 0x7fb9c4166720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:05.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.539+0000 7fb9c3e00700 1 -- 172.21.15.89:0/2632763705 >> v1:172.21.15.113:6789/0 conn(0x7fb9c415b760 legacy=0x7fb9c40724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:05.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.539+0000 7fb9c3e00700 1 -- 172.21.15.89:0/2632763705 >> v1:172.21.15.89:6789/0 conn(0x7fb9c40cec00 legacy=0x7fb9c40cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:05.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.539+0000 7fb9c3e00700 1 -- 172.21.15.89:0/2632763705 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c415bb40 con 0x7fb9c4164330 2024-04-02T09:14:05.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.539+0000 7fb9c2000700 1 -- 172.21.15.89:0/2632763705 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152976735 0 0) 0x7fb9c40cbca0 con 0x7fb9c415b760 2024-04-02T09:14:05.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.539+0000 7fb9c3e00700 1 --2- 172.21.15.89:0/2632763705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9c4164330 0x7fb9c4166720 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc009d90 tx=0x7fb9bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0fc095c01a88182 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:05.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.539+0000 7fb9c2000700 1 -- 172.21.15.89:0/2632763705 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc00e070 con 0x7fb9c4164330 2024-04-02T09:14:05.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.539+0000 7fb9c2000700 1 -- 172.21.15.89:0/2632763705 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9bc00bbd0 con 0x7fb9c4164330 2024-04-02T09:14:05.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.539+0000 7fb9c2000700 1 -- 172.21.15.89:0/2632763705 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc012430 con 0x7fb9c4164330 2024-04-02T09:14:05.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.540+0000 7fb9ca000700 1 -- 172.21.15.89:0/2632763705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9c4164330 msgr2=0x7fb9c4166720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:05.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.540+0000 7fb9ca000700 1 --2- 172.21.15.89:0/2632763705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9c4164330 0x7fb9c4166720 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc009d90 tx=0x7fb9bc00b330 comp rx=0 tx=0).stop 2024-04-02T09:14:05.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.540+0000 7fb9ca000700 1 -- 172.21.15.89:0/2632763705 shutdown_connections 2024-04-02T09:14:05.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.540+0000 7fb9ca000700 1 --2- 172.21.15.89:0/2632763705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9c4164330 0x7fb9c4166720 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.540+0000 7fb9ca000700 1 -- 172.21.15.89:0/2632763705 >> 172.21.15.89:0/2632763705 conn(0x7fb9c4070f00 msgr2=0x7fb9c4071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:05.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.540+0000 7fb9ca000700 1 -- 172.21.15.89:0/2632763705 shutdown_connections 2024-04-02T09:14:05.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.540+0000 7fb9ca000700 1 -- 172.21.15.89:0/2632763705 wait complete. 2024-04-02T09:14:05.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.542+0000 7fb9ca000700 1 Processor -- start 2024-04-02T09:14:05.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.543+0000 7fb9ca000700 1 -- start start 2024-04-02T09:14:05.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.544+0000 7fb9ca000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9c40cec00 0x7fb9c4098a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:05.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.544+0000 7fb9ca000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9c415b760 0x7fb9c4156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:05.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.544+0000 7fb9ca000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9c4164330 0x7fb9c4157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.544+0000 7fb9ca000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c4151c40 con 0x7fb9c415b760 2024-04-02T09:14:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.544+0000 7fb9ca000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c4151030 con 0x7fb9c4164330 2024-04-02T09:14:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.544+0000 7fb9ca000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c4151310 con 0x7fb9c40cec00 2024-04-02T09:14:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.544+0000 7fb9c2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9c415b760 0x7fb9c4156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.544+0000 7fb9c2a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9c415b760 0x7fb9c4156cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44258/0 (socket says 172.21.15.89:44258) 2024-04-02T09:14:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.544+0000 7fb9c2a00700 1 -- 172.21.15.89:0/527925743 learned_addr learned my addr 172.21.15.89:0/527925743 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.544+0000 7fb9c3e00700 1 --2- 172.21.15.89:0/527925743 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9c4164330 0x7fb9c4157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.544+0000 7fb9c3400700 1 --2- 172.21.15.89:0/527925743 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9c40cec00 0x7fb9c4098a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.545+0000 7fb9c2a00700 1 -- 172.21.15.89:0/527925743 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9c40cec00 msgr2=0x7fb9c4098a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.545+0000 7fb9c2a00700 1 --2- 172.21.15.89:0/527925743 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9c40cec00 0x7fb9c4098a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.545+0000 7fb9c2a00700 1 -- 172.21.15.89:0/527925743 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9c4164330 msgr2=0x7fb9c4157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.545+0000 7fb9c2a00700 1 --2- 172.21.15.89:0/527925743 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9c4164330 0x7fb9c4157200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.545+0000 7fb9c2a00700 1 -- 172.21.15.89:0/527925743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc009a70 con 0x7fb9c415b760 2024-04-02T09:14:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.545+0000 7fb9c2a00700 1 --2- 172.21.15.89:0/527925743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9c415b760 0x7fb9c4156cc0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000c430 tx=0x7fb9b000c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:05.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.545+0000 7fb9bb400700 1 -- 172.21.15.89:0/527925743 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0005b70 con 0x7fb9c415b760 2024-04-02T09:14:05.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.545+0000 7fb9bb400700 1 -- 172.21.15.89:0/527925743 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b0005cd0 con 0x7fb9c415b760 2024-04-02T09:14:05.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.545+0000 7fb9ca000700 1 -- 172.21.15.89:0/527925743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c41515f0 con 0x7fb9c415b760 2024-04-02T09:14:05.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.545+0000 7fb9bb400700 1 -- 172.21.15.89:0/527925743 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b00072f0 con 0x7fb9c415b760 2024-04-02T09:14:05.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.545+0000 7fb9ca000700 1 -- 172.21.15.89:0/527925743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c41520d0 con 0x7fb9c415b760 2024-04-02T09:14:05.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.547+0000 7fb9ca000700 1 -- 172.21.15.89:0/527925743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c406b7d0 con 0x7fb9c415b760 2024-04-02T09:14:05.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.547+0000 7fb9bb400700 1 -- 172.21.15.89:0/527925743 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7fb9b0007450 con 0x7fb9c415b760 2024-04-02T09:14:05.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.548+0000 7fb9bb400700 1 --2- 172.21.15.89:0/527925743 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb9a406aed0 0x7fb9a406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:05.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.548+0000 7fb9bb400700 1 -- 172.21.15.89:0/527925743 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7fb9b0089100 con 0x7fb9c415b760 2024-04-02T09:14:05.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.549+0000 7fb9c3400700 1 -- 172.21.15.89:0/527925743 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb9a406aed0 msgr2=0x7fb9a406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:05.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.549+0000 7fb9c3400700 1 --2- 172.21.15.89:0/527925743 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb9a406aed0 0x7fb9a406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:05.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.556+0000 7fb9bb400700 1 -- 172.21.15.89:0/527925743 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb9b00538b0 con 0x7fb9c415b760 2024-04-02T09:14:05.585 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:05.586+0000 7f90b8167240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:14:05.695 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:05.697+0000 7f75006c9240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:14:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.750+0000 7fb9c3400700 1 -- 172.21.15.89:0/527925743 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb9a406aed0 msgr2=0x7fb9a406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.750+0000 7fb9c3400700 1 --2- 172.21.15.89:0/527925743 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb9a406aed0 0x7fb9a406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:05.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.790+0000 7fb9ca000700 1 -- 172.21.15.89:0/527925743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb9c4067720 con 0x7fb9c415b760 2024-04-02T09:14:05.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.790+0000 7fb9bb400700 1 -- 172.21.15.89:0/527925743 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v249) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb9b0017020 con 0x7fb9c415b760 2024-04-02T09:14:05.789 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:14:05.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.793+0000 7fb9ca000700 1 -- 172.21.15.89:0/527925743 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb9a406aed0 msgr2=0x7fb9a406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:05.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.793+0000 7fb9ca000700 1 --2- 172.21.15.89:0/527925743 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb9a406aed0 0x7fb9a406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.793+0000 7fb9ca000700 1 -- 172.21.15.89:0/527925743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9c415b760 msgr2=0x7fb9c4156cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:05.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.793+0000 7fb9ca000700 1 --2- 172.21.15.89:0/527925743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9c415b760 0x7fb9c4156cc0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000c430 tx=0x7fb9b000c740 comp rx=0 tx=0).stop 2024-04-02T09:14:05.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.793+0000 7fb9ca000700 1 -- 172.21.15.89:0/527925743 shutdown_connections 2024-04-02T09:14:05.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.793+0000 7fb9ca000700 1 --2- 172.21.15.89:0/527925743 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb9a406aed0 0x7fb9a406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.793+0000 7fb9ca000700 1 --2- 172.21.15.89:0/527925743 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9c4164330 0x7fb9c4157200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.793+0000 7fb9ca000700 1 --2- 172.21.15.89:0/527925743 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb9c40cec00 0x7fb9c4098a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.793+0000 7fb9ca000700 1 --2- 172.21.15.89:0/527925743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9c415b760 0x7fb9c4156cc0 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:05.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.793+0000 7fb9ca000700 1 -- 172.21.15.89:0/527925743 >> 172.21.15.89:0/527925743 conn(0x7fb9c4070f00 msgr2=0x7fb9c40cda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:05.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.793+0000 7fb9ca000700 1 -- 172.21.15.89:0/527925743 shutdown_connections 2024-04-02T09:14:05.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:05.793+0000 7fb9ca000700 1 -- 172.21.15.89:0/527925743 wait complete. 2024-04-02T09:14:05.810 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T09:14:06.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.011+0000 7fef65e00700 1 Processor -- start 2024-04-02T09:14:06.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.011+0000 7fef65e00700 1 -- start start 2024-04-02T09:14:06.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.012+0000 7fef65e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef6015d9d0 0x7fef600d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.012+0000 7fef65e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef60161420 con 0x7fef600d16a0 2024-04-02T09:14:06.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.012+0000 7fef65e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef60161700 con 0x7fef600d2540 2024-04-02T09:14:06.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.012+0000 7fef65e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef6015edc0 con 0x7fef6015d9d0 2024-04-02T09:14:06.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.012+0000 7fef5ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef6015d9d0 0x7fef600d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:06.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.012+0000 7fef5f400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fef600d2540 0x7fef600ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:44514/0 (socket says 172.21.15.89:44514) 2024-04-02T09:14:06.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.012+0000 7fef5ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef6015d9d0 0x7fef600d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44262/0 (socket says 172.21.15.89:44262) 2024-04-02T09:14:06.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.012+0000 7fef5f400700 1 -- 172.21.15.89:0/927137330 learned_addr learned my addr 172.21.15.89:0/927137330 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:06.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.013+0000 7fef5d600700 1 -- 172.21.15.89:0/927137330 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367514947 0 0) 0x7fef60161700 con 0x7fef600d2540 2024-04-02T09:14:06.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.013+0000 7fef5d600700 1 -- 172.21.15.89:0/927137330 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef440035b0 con 0x7fef600d2540 2024-04-02T09:14:06.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.013+0000 7fef5ea00700 1 -- 172.21.15.89:0/927137330 >> v1:172.21.15.113:6789/0 conn(0x7fef600d2540 legacy=0x7fef600ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.013+0000 7fef5ea00700 1 -- 172.21.15.89:0/927137330 >> v1:172.21.15.89:6789/0 conn(0x7fef600d16a0 legacy=0x7fef600d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.013+0000 7fef5ea00700 1 -- 172.21.15.89:0/927137330 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef601619e0 con 0x7fef6015d9d0 2024-04-02T09:14:06.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.013+0000 7fef5ea00700 1 --2- 172.21.15.89:0/927137330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef6015d9d0 0x7fef600d3c30 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7fef50009840 tx=0x7fef50009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cedc9724ad39afd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:06.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.013+0000 7fef5d600700 1 -- 172.21.15.89:0/927137330 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef50011830 con 0x7fef6015d9d0 2024-04-02T09:14:06.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.013+0000 7fef5d600700 1 -- 172.21.15.89:0/927137330 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef50011990 con 0x7fef6015d9d0 2024-04-02T09:14:06.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.013+0000 7fef5d600700 1 -- 172.21.15.89:0/927137330 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef50011d20 con 0x7fef6015d9d0 2024-04-02T09:14:06.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.014+0000 7fef65e00700 1 -- 172.21.15.89:0/927137330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef6015d9d0 msgr2=0x7fef600d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.014+0000 7fef65e00700 1 --2- 172.21.15.89:0/927137330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef6015d9d0 0x7fef600d3c30 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7fef50009840 tx=0x7fef50009b50 comp rx=0 tx=0).stop 2024-04-02T09:14:06.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.014+0000 7fef65e00700 1 -- 172.21.15.89:0/927137330 shutdown_connections 2024-04-02T09:14:06.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.014+0000 7fef65e00700 1 --2- 172.21.15.89:0/927137330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef6015d9d0 0x7fef600d3c30 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.014+0000 7fef65e00700 1 -- 172.21.15.89:0/927137330 >> 172.21.15.89:0/927137330 conn(0x7fef6008aff0 msgr2=0x7fef600d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:06.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.014+0000 7fef65e00700 1 -- 172.21.15.89:0/927137330 shutdown_connections 2024-04-02T09:14:06.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.014+0000 7fef65e00700 1 -- 172.21.15.89:0/927137330 wait complete. 2024-04-02T09:14:06.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.016+0000 7fef65e00700 1 Processor -- start 2024-04-02T09:14:06.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.016+0000 7fef65e00700 1 -- start start 2024-04-02T09:14:06.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.017+0000 7fef65e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef600d16a0 0x7fef601e9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.017+0000 7fef65e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef600d2540 0x7fef601565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.017+0000 7fef65e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef6015d9d0 0x7fef600b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.017+0000 7fef65e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef600b4b60 con 0x7fef600d2540 2024-04-02T09:14:06.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.017+0000 7fef65e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef60156ff0 con 0x7fef600d16a0 2024-04-02T09:14:06.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.017+0000 7fef65e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef601e8870 con 0x7fef6015d9d0 2024-04-02T09:14:06.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.017+0000 7fef5e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef600d2540 0x7fef601565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:06.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.017+0000 7fef5ea00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef600d16a0 0x7fef601e9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:06.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.017+0000 7fef5ea00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef600d16a0 0x7fef601e9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:53584/0 (socket says 172.21.15.89:53584) 2024-04-02T09:14:06.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.017+0000 7fef5ea00700 1 -- 172.21.15.89:0/3603740226 learned_addr learned my addr 172.21.15.89:0/3603740226 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:06.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.018+0000 7fef5f400700 1 --2- 172.21.15.89:0/3603740226 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef6015d9d0 0x7fef600b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:06.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.018+0000 7fef5e000700 1 -- 172.21.15.89:0/3603740226 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef6015d9d0 msgr2=0x7fef600b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.018+0000 7fef5e000700 1 --2- 172.21.15.89:0/3603740226 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef6015d9d0 0x7fef600b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.018+0000 7fef5e000700 1 -- 172.21.15.89:0/3603740226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef600d16a0 msgr2=0x7fef601e9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.018+0000 7fef5e000700 1 --2- 172.21.15.89:0/3603740226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef600d16a0 0x7fef601e9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.018+0000 7fef5e000700 1 -- 172.21.15.89:0/3603740226 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef500094a0 con 0x7fef600d2540 2024-04-02T09:14:06.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.018+0000 7fef5f400700 1 --2- 172.21.15.89:0/3603740226 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef6015d9d0 0x7fef600b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:06.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.018+0000 7fef5e000700 1 --2- 172.21.15.89:0/3603740226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef600d2540 0x7fef601565a0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7fef600d1ae0 tx=0x7fef4c00c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:06.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.018+0000 7fef57400700 1 -- 172.21.15.89:0/3603740226 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c0054e0 con 0x7fef600d2540 2024-04-02T09:14:06.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.018+0000 7fef57400700 1 -- 172.21.15.89:0/3603740226 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4c00ce50 con 0x7fef600d2540 2024-04-02T09:14:06.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.018+0000 7fef65e00700 1 -- 172.21.15.89:0/3603740226 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef601e8b50 con 0x7fef600d2540 2024-04-02T09:14:06.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.018+0000 7fef57400700 1 -- 172.21.15.89:0/3603740226 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c0038c0 con 0x7fef600d2540 2024-04-02T09:14:06.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.018+0000 7fef65e00700 1 -- 172.21.15.89:0/3603740226 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef601e9110 con 0x7fef600d2540 2024-04-02T09:14:06.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.020+0000 7fef65e00700 1 -- 172.21.15.89:0/3603740226 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef600d1ca0 con 0x7fef600d2540 2024-04-02T09:14:06.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.020+0000 7fef57400700 1 -- 172.21.15.89:0/3603740226 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7fef4c003a20 con 0x7fef600d2540 2024-04-02T09:14:06.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.021+0000 7fef57400700 1 --2- 172.21.15.89:0/3603740226 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef4006aed0 0x7fef4006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.021+0000 7fef57400700 1 -- 172.21.15.89:0/3603740226 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7fef4c089270 con 0x7fef600d2540 2024-04-02T09:14:06.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.021+0000 7fef5ea00700 1 -- 172.21.15.89:0/3603740226 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef4006aed0 msgr2=0x7fef4006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:06.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.021+0000 7fef5ea00700 1 --2- 172.21.15.89:0/3603740226 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef4006aed0 0x7fef4006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:06.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.026+0000 7fef57400700 1 -- 172.21.15.89:0/3603740226 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fef4c053910 con 0x7fef600d2540 2024-04-02T09:14:06.044 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:06.045+0000 7f90b8167240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:14:06.185 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:06.185+0000 7f75006c9240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:14:06.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.222+0000 7fef5ea00700 1 -- 172.21.15.89:0/3603740226 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef4006aed0 msgr2=0x7fef4006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:06.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.222+0000 7fef5ea00700 1 --2- 172.21.15.89:0/3603740226 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef4006aed0 0x7fef4006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:06.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.234+0000 7fef65e00700 1 -- 172.21.15.89:0/3603740226 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fef60067720 con 0x7fef600d2540 2024-04-02T09:14:06.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.234+0000 7fef57400700 1 -- 172.21.15.89:0/3603740226 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v249) v1 ==== 100+0+11 (secure 0 0 0) 0x7fef4c056f30 con 0x7fef600d2540 2024-04-02T09:14:06.234 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:14:06.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.237+0000 7fef65e00700 1 -- 172.21.15.89:0/3603740226 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef4006aed0 msgr2=0x7fef4006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:06.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.237+0000 7fef65e00700 1 --2- 172.21.15.89:0/3603740226 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef4006aed0 0x7fef4006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.237+0000 7fef65e00700 1 -- 172.21.15.89:0/3603740226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef600d2540 msgr2=0x7fef601565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.237+0000 7fef65e00700 1 --2- 172.21.15.89:0/3603740226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef600d2540 0x7fef601565a0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7fef600d1ae0 tx=0x7fef4c00c6e0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.237+0000 7fef65e00700 1 -- 172.21.15.89:0/3603740226 shutdown_connections 2024-04-02T09:14:06.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.237+0000 7fef65e00700 1 --2- 172.21.15.89:0/3603740226 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef4006aed0 0x7fef4006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.237+0000 7fef65e00700 1 --2- 172.21.15.89:0/3603740226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef600d16a0 0x7fef601e9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.237+0000 7fef65e00700 1 --2- 172.21.15.89:0/3603740226 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef6015d9d0 0x7fef600b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.237+0000 7fef65e00700 1 --2- 172.21.15.89:0/3603740226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef600d2540 0x7fef601565a0 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.237+0000 7fef65e00700 1 -- 172.21.15.89:0/3603740226 >> 172.21.15.89:0/3603740226 conn(0x7fef6008aff0 msgr2=0x7fef6006cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:06.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.237+0000 7fef65e00700 1 -- 172.21.15.89:0/3603740226 shutdown_connections 2024-04-02T09:14:06.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.237+0000 7fef65e00700 1 -- 172.21.15.89:0/3603740226 wait complete. 2024-04-02T09:14:06.255 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-02T09:14:06.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.454+0000 7f0148200700 1 Processor -- start 2024-04-02T09:14:06.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.454+0000 7f0148200700 1 -- start start 2024-04-02T09:14:06.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.455+0000 7f0148200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01400d16a0 0x7f01400d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.455+0000 7f0148200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0140161420 con 0x7f01400d2370 2024-04-02T09:14:06.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.455+0000 7f0148200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0140161700 con 0x7f014015d9d0 2024-04-02T09:14:06.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.455+0000 7f0148200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01400d2820 con 0x7f01400d16a0 2024-04-02T09:14:06.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.456+0000 7f0146200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f01400d2370 0x7f01400ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34156/0 (socket says 172.21.15.89:34156) 2024-04-02T09:14:06.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.456+0000 7f0146200700 1 -- 172.21.15.89:0/328611861 learned_addr learned my addr 172.21.15.89:0/328611861 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:06.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.456+0000 7f0144e00700 1 --2- 172.21.15.89:0/328611861 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01400d16a0 0x7f01400d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:06.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.456+0000 7f013fe00700 1 -- 172.21.15.89:0/328611861 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750015419 0 0) 0x7f0140161420 con 0x7f01400d2370 2024-04-02T09:14:06.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.456+0000 7f013fe00700 1 -- 172.21.15.89:0/328611861 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01280035b0 con 0x7f01400d2370 2024-04-02T09:14:06.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.456+0000 7f013fe00700 1 -- 172.21.15.89:0/328611861 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995034356 0 0) 0x7f0140161700 con 0x7f014015d9d0 2024-04-02T09:14:06.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.456+0000 7f0144e00700 1 -- 172.21.15.89:0/328611861 >> v1:172.21.15.113:6789/0 conn(0x7f014015d9d0 legacy=0x7f01400d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.456+0000 7f0144e00700 1 -- 172.21.15.89:0/328611861 >> v1:172.21.15.89:6789/0 conn(0x7f01400d2370 legacy=0x7f01400ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.456+0000 7f0144e00700 1 -- 172.21.15.89:0/328611861 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01401619e0 con 0x7f01400d16a0 2024-04-02T09:14:06.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.456+0000 7f0144e00700 1 --2- 172.21.15.89:0/328611861 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01400d16a0 0x7f01400d1e30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f012c009850 tx=0x7f012c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771ffdce4e21cba1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:06.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.457+0000 7f013fe00700 1 -- 172.21.15.89:0/328611861 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c005e30 con 0x7f01400d16a0 2024-04-02T09:14:06.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.457+0000 7f013fe00700 1 -- 172.21.15.89:0/328611861 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012c006030 con 0x7f01400d16a0 2024-04-02T09:14:06.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.457+0000 7f013fe00700 1 -- 172.21.15.89:0/328611861 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c0063e0 con 0x7f01400d16a0 2024-04-02T09:14:06.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.457+0000 7f0148200700 1 -- 172.21.15.89:0/328611861 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01400d16a0 msgr2=0x7f01400d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.457+0000 7f0148200700 1 --2- 172.21.15.89:0/328611861 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01400d16a0 0x7f01400d1e30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f012c009850 tx=0x7f012c009b60 comp rx=0 tx=0).stop 2024-04-02T09:14:06.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.458+0000 7f0148200700 1 -- 172.21.15.89:0/328611861 shutdown_connections 2024-04-02T09:14:06.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.458+0000 7f0148200700 1 --2- 172.21.15.89:0/328611861 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01400d16a0 0x7f01400d1e30 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.458+0000 7f0148200700 1 -- 172.21.15.89:0/328611861 >> 172.21.15.89:0/328611861 conn(0x7f014008aff0 msgr2=0x7f01400d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:06.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.458+0000 7f0148200700 1 -- 172.21.15.89:0/328611861 shutdown_connections 2024-04-02T09:14:06.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.458+0000 7f0148200700 1 -- 172.21.15.89:0/328611861 wait complete. 2024-04-02T09:14:06.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.460+0000 7f0148200700 1 Processor -- start 2024-04-02T09:14:06.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.460+0000 7f0148200700 1 -- start start 2024-04-02T09:14:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.461+0000 7f0148200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01400d16a0 0x7f014014f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.461+0000 7f0148200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01400d2370 0x7f01401605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.461+0000 7f0148200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f014015d9d0 0x7f014015bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.461+0000 7f0148200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01400ca5f0 con 0x7f01400d16a0 2024-04-02T09:14:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.461+0000 7f0148200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01400cb450 con 0x7f01400d2370 2024-04-02T09:14:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.461+0000 7f0148200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01400ca840 con 0x7f014015d9d0 2024-04-02T09:14:06.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.462+0000 7f0145800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01400d16a0 0x7f014014f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:06.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.462+0000 7f0145800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01400d16a0 0x7f014014f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44290/0 (socket says 172.21.15.89:44290) 2024-04-02T09:14:06.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.462+0000 7f0145800700 1 -- 172.21.15.89:0/1281626466 learned_addr learned my addr 172.21.15.89:0/1281626466 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:06.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.462+0000 7f0144e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01400d2370 0x7f01401605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:06.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.462+0000 7f0146200700 1 --2- 172.21.15.89:0/1281626466 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f014015d9d0 0x7f014015bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:06.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.462+0000 7f0145800700 1 -- 172.21.15.89:0/1281626466 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f014015d9d0 msgr2=0x7f014015bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.462+0000 7f0145800700 1 --2- 172.21.15.89:0/1281626466 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f014015d9d0 0x7f014015bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.462+0000 7f0145800700 1 -- 172.21.15.89:0/1281626466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01400d2370 msgr2=0x7f01401605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.462+0000 7f0145800700 1 --2- 172.21.15.89:0/1281626466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01400d2370 0x7f01401605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.462+0000 7f0145800700 1 -- 172.21.15.89:0/1281626466 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c009430 con 0x7f01400d16a0 2024-04-02T09:14:06.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.462+0000 7f0144e00700 1 --2- 172.21.15.89:0/1281626466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01400d2370 0x7f01401605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:14:06.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.462+0000 7f0145800700 1 --2- 172.21.15.89:0/1281626466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01400d16a0 0x7f014014f8f0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f013400c580 tx=0x7f013400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:06.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.463+0000 7f013d600700 1 -- 172.21.15.89:0/1281626466 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01340054e0 con 0x7f01400d16a0 2024-04-02T09:14:06.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.463+0000 7f013d600700 1 -- 172.21.15.89:0/1281626466 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0134003680 con 0x7f01400d16a0 2024-04-02T09:14:06.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.463+0000 7f013d600700 1 -- 172.21.15.89:0/1281626466 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0134003a30 con 0x7f01400d16a0 2024-04-02T09:14:06.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.463+0000 7f0148200700 1 -- 172.21.15.89:0/1281626466 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01400cab20 con 0x7f01400d16a0 2024-04-02T09:14:06.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.463+0000 7f0148200700 1 -- 172.21.15.89:0/1281626466 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01400cb0e0 con 0x7f01400d16a0 2024-04-02T09:14:06.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.464+0000 7f0148200700 1 -- 172.21.15.89:0/1281626466 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01400d1ca0 con 0x7f01400d16a0 2024-04-02T09:14:06.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.465+0000 7f013d600700 1 -- 172.21.15.89:0/1281626466 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f0134003b90 con 0x7f01400d16a0 2024-04-02T09:14:06.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.466+0000 7f013d600700 1 --2- 172.21.15.89:0/1281626466 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f012006aed0 0x7f012006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.466+0000 7f013d600700 1 -- 172.21.15.89:0/1281626466 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f0134089140 con 0x7f01400d16a0 2024-04-02T09:14:06.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.466+0000 7f0144e00700 1 -- 172.21.15.89:0/1281626466 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f012006aed0 msgr2=0x7f012006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:06.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.466+0000 7f0144e00700 1 --2- 172.21.15.89:0/1281626466 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f012006aed0 0x7f012006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:06.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.470+0000 7f013d600700 1 -- 172.21.15.89:0/1281626466 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f01340538f0 con 0x7f01400d16a0 2024-04-02T09:14:06.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.667+0000 7f0144e00700 1 -- 172.21.15.89:0/1281626466 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f012006aed0 msgr2=0x7f012006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:06.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.667+0000 7f0144e00700 1 --2- 172.21.15.89:0/1281626466 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f012006aed0 0x7f012006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:06.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.691+0000 7f0148200700 1 -- 172.21.15.89:0/1281626466 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0140067720 con 0x7f01400d16a0 2024-04-02T09:14:06.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.691+0000 7f013d600700 1 -- 172.21.15.89:0/1281626466 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v249) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0134017020 con 0x7f01400d16a0 2024-04-02T09:14:06.691 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:14:06.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.694+0000 7f0148200700 1 -- 172.21.15.89:0/1281626466 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f012006aed0 msgr2=0x7f012006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.694+0000 7f0148200700 1 --2- 172.21.15.89:0/1281626466 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f012006aed0 0x7f012006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.694+0000 7f0148200700 1 -- 172.21.15.89:0/1281626466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01400d16a0 msgr2=0x7f014014f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.694+0000 7f0148200700 1 --2- 172.21.15.89:0/1281626466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01400d16a0 0x7f014014f8f0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f013400c580 tx=0x7f013400c890 comp rx=0 tx=0).stop 2024-04-02T09:14:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.694+0000 7f0148200700 1 -- 172.21.15.89:0/1281626466 shutdown_connections 2024-04-02T09:14:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.694+0000 7f0148200700 1 --2- 172.21.15.89:0/1281626466 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f014015d9d0 0x7f014015bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.694+0000 7f0148200700 1 --2- 172.21.15.89:0/1281626466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01400d16a0 0x7f014014f8f0 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.694+0000 7f0148200700 1 --2- 172.21.15.89:0/1281626466 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f012006aed0 0x7f012006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.694+0000 7f0148200700 1 --2- 172.21.15.89:0/1281626466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01400d2370 0x7f01401605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.694+0000 7f0148200700 1 -- 172.21.15.89:0/1281626466 >> 172.21.15.89:0/1281626466 conn(0x7f014008aff0 msgr2=0x7f014006c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.694+0000 7f0148200700 1 -- 172.21.15.89:0/1281626466 shutdown_connections 2024-04-02T09:14:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.694+0000 7f0148200700 1 -- 172.21.15.89:0/1281626466 wait complete. 2024-04-02T09:14:06.713 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-02T09:14:06.757 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:06.759+0000 7f90b8167240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:14:06.847 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:06.848+0000 7f90b8167240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:14:06.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.912+0000 7f83f3200700 1 Processor -- start 2024-04-02T09:14:06.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.913+0000 7f83f3200700 1 -- start start 2024-04-02T09:14:06.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.914+0000 7f83f3200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f83ec15dfe0 0x7f83ec0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.914+0000 7f83f3200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83ec0ce670 0x7f83ec0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.915+0000 7f83f3200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ec0cb310 con 0x7f83ec15a5c0 2024-04-02T09:14:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.915+0000 7f83f3200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83ec0ceb00 con 0x7f83ec15dfe0 2024-04-02T09:14:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.915+0000 7f83f3200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec0711a0 con 0x7f83ec0ce670 2024-04-02T09:14:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.915+0000 7f83ebe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f83ec15a5c0 0x7f83ec0709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34172/0 (socket says 172.21.15.89:34172) 2024-04-02T09:14:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.915+0000 7f83ebe00700 1 -- 172.21.15.89:0/3881328399 learned_addr learned my addr 172.21.15.89:0/3881328399 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.915+0000 7f83e3e00700 1 --2- 172.21.15.89:0/3881328399 >> v2:172.21.15.113:3300/0 conn(0x7f83ec15dfe0 0x7f83ec0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.915+0000 7f83f1200700 1 --2- 172.21.15.89:0/3881328399 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83ec0ce670 0x7f83ec0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.915+0000 7f83eb400700 1 -- 172.21.15.89:0/3881328399 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050337574 0 0) 0x7f83ec0cb310 con 0x7f83ec15a5c0 2024-04-02T09:14:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.915+0000 7f83eb400700 1 -- 172.21.15.89:0/3881328399 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83d00035b0 con 0x7f83ec15a5c0 2024-04-02T09:14:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.915+0000 7f83e3e00700 1 -- 172.21.15.89:0/3881328399 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83ec0ce670 msgr2=0x7f83ec0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.915+0000 7f83e3e00700 1 --2- 172.21.15.89:0/3881328399 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83ec0ce670 0x7f83ec0cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.915+0000 7f83e3e00700 1 -- 172.21.15.89:0/3881328399 >> v1:172.21.15.89:6789/0 conn(0x7f83ec15a5c0 legacy=0x7f83ec0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.915+0000 7f83e3e00700 1 -- 172.21.15.89:0/3881328399 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ec071480 con 0x7f83ec15dfe0 2024-04-02T09:14:06.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.916+0000 7f83e3e00700 1 --2- 172.21.15.89:0/3881328399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83ec15dfe0 0x7f83ec0ce130 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009840 tx=0x7f83d4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c925503c3cdb4428 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:06.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.916+0000 7f83eb400700 1 -- 172.21.15.89:0/3881328399 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d4011830 con 0x7f83ec15dfe0 2024-04-02T09:14:06.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.916+0000 7f83eb400700 1 -- 172.21.15.89:0/3881328399 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d4011990 con 0x7f83ec15dfe0 2024-04-02T09:14:06.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.916+0000 7f83eb400700 1 -- 172.21.15.89:0/3881328399 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d4011d20 con 0x7f83ec15dfe0 2024-04-02T09:14:06.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.916+0000 7f83f3200700 1 -- 172.21.15.89:0/3881328399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83ec15dfe0 msgr2=0x7f83ec0ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.916+0000 7f83f3200700 1 --2- 172.21.15.89:0/3881328399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83ec15dfe0 0x7f83ec0ce130 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009840 tx=0x7f83d4009b50 comp rx=0 tx=0).stop 2024-04-02T09:14:06.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.917+0000 7f83f3200700 1 -- 172.21.15.89:0/3881328399 shutdown_connections 2024-04-02T09:14:06.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.917+0000 7f83f3200700 1 --2- 172.21.15.89:0/3881328399 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83ec0ce670 0x7f83ec0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.917+0000 7f83f3200700 1 --2- 172.21.15.89:0/3881328399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83ec15dfe0 0x7f83ec0ce130 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.917+0000 7f83f3200700 1 -- 172.21.15.89:0/3881328399 >> 172.21.15.89:0/3881328399 conn(0x7f83ec0b0d70 msgr2=0x7f83ec0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:06.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.917+0000 7f83f3200700 1 -- 172.21.15.89:0/3881328399 shutdown_connections 2024-04-02T09:14:06.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.917+0000 7f83f3200700 1 -- 172.21.15.89:0/3881328399 wait complete. 2024-04-02T09:14:06.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.919+0000 7f83f3200700 1 Processor -- start 2024-04-02T09:14:06.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.919+0000 7f83f3200700 1 -- start start 2024-04-02T09:14:06.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.921+0000 7f83f3200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83ec0ce670 0x7f83ec1f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.921+0000 7f83f3200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83ec15a5c0 0x7f83ec0991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.921+0000 7f83f3200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83ec15dfe0 0x7f83ec09aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.921+0000 7f83f3200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec15ae00 con 0x7f83ec15dfe0 2024-04-02T09:14:06.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.921+0000 7f83f3200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec15b980 con 0x7f83ec0ce670 2024-04-02T09:14:06.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.921+0000 7f83f3200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec15b050 con 0x7f83ec15a5c0 2024-04-02T09:14:06.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.921+0000 7f83f1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83ec15dfe0 0x7f83ec09aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:06.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.921+0000 7f83f1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83ec15dfe0 0x7f83ec09aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44306/0 (socket says 172.21.15.89:44306) 2024-04-02T09:14:06.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.921+0000 7f83e3e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83ec15a5c0 0x7f83ec0991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:06.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.921+0000 7f83f1200700 1 -- 172.21.15.89:0/2489940589 learned_addr learned my addr 172.21.15.89:0/2489940589 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:06.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.921+0000 7f83ebe00700 1 --2- 172.21.15.89:0/2489940589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83ec0ce670 0x7f83ec1f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:06.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.922+0000 7f83e3e00700 1 -- 172.21.15.89:0/2489940589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83ec0ce670 msgr2=0x7f83ec1f6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.922+0000 7f83e3e00700 1 --2- 172.21.15.89:0/2489940589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83ec0ce670 0x7f83ec1f6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.922+0000 7f83e3e00700 1 -- 172.21.15.89:0/2489940589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83ec15dfe0 msgr2=0x7f83ec09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:06.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.922+0000 7f83e3e00700 1 --2- 172.21.15.89:0/2489940589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83ec15dfe0 0x7f83ec09aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:06.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.922+0000 7f83e3e00700 1 -- 172.21.15.89:0/2489940589 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d40094a0 con 0x7f83ec15a5c0 2024-04-02T09:14:06.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.922+0000 7f83e3e00700 1 --2- 172.21.15.89:0/2489940589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83ec15a5c0 0x7f83ec0991d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f83d4016000 tx=0x7f83d4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:06.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.922+0000 7f83e8c00700 1 -- 172.21.15.89:0/2489940589 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d4004590 con 0x7f83ec15a5c0 2024-04-02T09:14:06.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.922+0000 7f83e8c00700 1 -- 172.21.15.89:0/2489940589 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d401e070 con 0x7f83ec15a5c0 2024-04-02T09:14:06.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.922+0000 7f83f3200700 1 -- 172.21.15.89:0/2489940589 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ec15b330 con 0x7f83ec15a5c0 2024-04-02T09:14:06.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.922+0000 7f83e8c00700 1 -- 172.21.15.89:0/2489940589 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d40038e0 con 0x7f83ec15a5c0 2024-04-02T09:14:06.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.922+0000 7f83f3200700 1 -- 172.21.15.89:0/2489940589 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ec09b0e0 con 0x7f83ec15a5c0 2024-04-02T09:14:06.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.924+0000 7f83f3200700 1 -- 172.21.15.89:0/2489940589 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ec15baf0 con 0x7f83ec15a5c0 2024-04-02T09:14:06.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.924+0000 7f83e8c00700 1 -- 172.21.15.89:0/2489940589 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f83d4003a40 con 0x7f83ec15a5c0 2024-04-02T09:14:06.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.925+0000 7f83e8c00700 1 --2- 172.21.15.89:0/2489940589 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f83c806aed0 0x7f83c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:06.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.925+0000 7f83e8c00700 1 -- 172.21.15.89:0/2489940589 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f83d408ad70 con 0x7f83ec15a5c0 2024-04-02T09:14:06.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.925+0000 7f83ebe00700 1 -- 172.21.15.89:0/2489940589 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f83c806aed0 msgr2=0x7f83c806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:06.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.925+0000 7f83ebe00700 1 --2- 172.21.15.89:0/2489940589 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f83c806aed0 0x7f83c806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:06.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:06.930+0000 7f83e8c00700 1 -- 172.21.15.89:0/2489940589 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f83d4055490 con 0x7f83ec15a5c0 2024-04-02T09:14:06.959 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:06.960+0000 7f75006c9240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:14:07.007 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:07.008+0000 7f90b8167240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:14:07.054 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:07.055+0000 7f75006c9240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:14:07.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.126+0000 7f83ebe00700 1 -- 172.21.15.89:0/2489940589 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f83c806aed0 msgr2=0x7f83c806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:07.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.126+0000 7f83ebe00700 1 --2- 172.21.15.89:0/2489940589 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f83c806aed0 0x7f83c806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:07.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.147+0000 7f83f3200700 1 -- 172.21.15.89:0/2489940589 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f83ec067720 con 0x7f83ec15a5c0 2024-04-02T09:14:07.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.148+0000 7f83e8c00700 1 -- 172.21.15.89:0/2489940589 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v249) v1 ==== 103+0+11 (secure 0 0 0) 0x7f83d4058ab0 con 0x7f83ec15a5c0 2024-04-02T09:14:07.147 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:14:07.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.151+0000 7f83f3200700 1 -- 172.21.15.89:0/2489940589 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f83c806aed0 msgr2=0x7f83c806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:07.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.151+0000 7f83f3200700 1 --2- 172.21.15.89:0/2489940589 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f83c806aed0 0x7f83c806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.151+0000 7f83f3200700 1 -- 172.21.15.89:0/2489940589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83ec15a5c0 msgr2=0x7f83ec0991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:07.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.151+0000 7f83f3200700 1 --2- 172.21.15.89:0/2489940589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83ec15a5c0 0x7f83ec0991d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f83d4016000 tx=0x7f83d4004750 comp rx=0 tx=0).stop 2024-04-02T09:14:07.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.151+0000 7f83f3200700 1 -- 172.21.15.89:0/2489940589 shutdown_connections 2024-04-02T09:14:07.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.151+0000 7f83f3200700 1 --2- 172.21.15.89:0/2489940589 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f83c806aed0 0x7f83c806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.151+0000 7f83f3200700 1 --2- 172.21.15.89:0/2489940589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83ec0ce670 0x7f83ec1f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.151+0000 7f83f3200700 1 --2- 172.21.15.89:0/2489940589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83ec15dfe0 0x7f83ec09aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.151+0000 7f83f3200700 1 --2- 172.21.15.89:0/2489940589 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83ec15a5c0 0x7f83ec0991d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.151+0000 7f83f3200700 1 -- 172.21.15.89:0/2489940589 >> 172.21.15.89:0/2489940589 conn(0x7f83ec0b0d70 msgr2=0x7f83ec0cc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:07.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.151+0000 7f83f3200700 1 -- 172.21.15.89:0/2489940589 shutdown_connections 2024-04-02T09:14:07.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.151+0000 7f83f3200700 1 -- 172.21.15.89:0/2489940589 wait complete. 2024-04-02T09:14:07.170 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T09:14:07.220 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:07.222+0000 7f90b8167240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:14:07.222 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:07.223+0000 7f75006c9240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:14:07.312 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:07.313+0000 7f90b8167240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:14:07.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.369+0000 7fef94200700 1 Processor -- start 2024-04-02T09:14:07.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.370+0000 7fef94200700 1 -- start start 2024-04-02T09:14:07.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.371+0000 7fef94200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef8c1611b0 0x7fef8c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:07.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.371+0000 7fef94200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef8c15f4d0 0x7fef8c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:07.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.371+0000 7fef94200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef8c0ce270 con 0x7fef8c0d21f0 2024-04-02T09:14:07.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.371+0000 7fef94200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef8c15b600 con 0x7fef8c1611b0 2024-04-02T09:14:07.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.371+0000 7fef94200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c0d2760 con 0x7fef8c15f4d0 2024-04-02T09:14:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.371+0000 7fef91800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef8c1611b0 0x7fef8c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.371+0000 7fef91800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef8c1611b0 0x7fef8c15ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44308/0 (socket says 172.21.15.89:44308) 2024-04-02T09:14:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.371+0000 7fef91800700 1 -- 172.21.15.89:0/1576009837 learned_addr learned my addr 172.21.15.89:0/1576009837 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.371+0000 7fef90e00700 1 --2- 172.21.15.89:0/1576009837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef8c15f4d0 0x7fef8c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.372+0000 7fef91800700 1 -- 172.21.15.89:0/1576009837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef8c15f4d0 msgr2=0x7fef8c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.372+0000 7fef91800700 1 --2- 172.21.15.89:0/1576009837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef8c15f4d0 0x7fef8c0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.372+0000 7fef91800700 1 -- 172.21.15.89:0/1576009837 >> v1:172.21.15.89:6789/0 conn(0x7fef8c0d21f0 legacy=0x7fef8c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.372+0000 7fef91800700 1 -- 172.21.15.89:0/1576009837 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c0d2a40 con 0x7fef8c1611b0 2024-04-02T09:14:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.372+0000 7fef91800700 1 --2- 172.21.15.89:0/1576009837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8c1611b0 0x7fef8c15ef90 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fef800098f0 tx=0x7fef80009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d34bfe5ef7390b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.372+0000 7fef8be00700 1 -- 172.21.15.89:0/1576009837 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef80011830 con 0x7fef8c1611b0 2024-04-02T09:14:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.372+0000 7fef8be00700 1 -- 172.21.15.89:0/1576009837 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef80011990 con 0x7fef8c1611b0 2024-04-02T09:14:07.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.372+0000 7fef8be00700 1 -- 172.21.15.89:0/1576009837 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef80011d20 con 0x7fef8c1611b0 2024-04-02T09:14:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.373+0000 7fef94200700 1 -- 172.21.15.89:0/1576009837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8c1611b0 msgr2=0x7fef8c15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.373+0000 7fef94200700 1 --2- 172.21.15.89:0/1576009837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8c1611b0 0x7fef8c15ef90 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fef800098f0 tx=0x7fef80009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.373+0000 7fef94200700 1 -- 172.21.15.89:0/1576009837 shutdown_connections 2024-04-02T09:14:07.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.373+0000 7fef94200700 1 --2- 172.21.15.89:0/1576009837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8c1611b0 0x7fef8c15ef90 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.373+0000 7fef94200700 1 --2- 172.21.15.89:0/1576009837 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef8c15f4d0 0x7fef8c0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.373+0000 7fef94200700 1 -- 172.21.15.89:0/1576009837 >> 172.21.15.89:0/1576009837 conn(0x7fef8c08aff0 msgr2=0x7fef8c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.373+0000 7fef94200700 1 -- 172.21.15.89:0/1576009837 shutdown_connections 2024-04-02T09:14:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.373+0000 7fef94200700 1 -- 172.21.15.89:0/1576009837 wait complete. 2024-04-02T09:14:07.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.375+0000 7fef94200700 1 Processor -- start 2024-04-02T09:14:07.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.376+0000 7fef94200700 1 -- start start 2024-04-02T09:14:07.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.377+0000 7fef94200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef8c0d21f0 0x7fef8c1f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.377+0000 7fef94200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef8c15f4d0 0x7fef8c1f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.377+0000 7fef94200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8c1611b0 0x7fef8c09d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.377+0000 7fef94200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c0caf10 con 0x7fef8c1611b0 2024-04-02T09:14:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.377+0000 7fef94200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c0ca600 con 0x7fef8c15f4d0 2024-04-02T09:14:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.377+0000 7fef94200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c09e130 con 0x7fef8c0d21f0 2024-04-02T09:14:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.377+0000 7fef92200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8c1611b0 0x7fef8c09d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.377+0000 7fef91800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef8c0d21f0 0x7fef8c1f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.377+0000 7fef92200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8c1611b0 0x7fef8c09d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44314/0 (socket says 172.21.15.89:44314) 2024-04-02T09:14:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.377+0000 7fef91800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef8c0d21f0 0x7fef8c1f6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:36588/0 (socket says 172.21.15.89:36588) 2024-04-02T09:14:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.377+0000 7fef92200700 1 -- 172.21.15.89:0/719078392 learned_addr learned my addr 172.21.15.89:0/719078392 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.377+0000 7fef90e00700 1 --2- 172.21.15.89:0/719078392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef8c15f4d0 0x7fef8c1f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:07.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.378+0000 7fef91800700 1 -- 172.21.15.89:0/719078392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef8c15f4d0 msgr2=0x7fef8c1f69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.378+0000 7fef91800700 1 --2- 172.21.15.89:0/719078392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef8c15f4d0 0x7fef8c1f69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.378+0000 7fef91800700 1 -- 172.21.15.89:0/719078392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8c1611b0 msgr2=0x7fef8c09d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.378+0000 7fef91800700 1 --2- 172.21.15.89:0/719078392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8c1611b0 0x7fef8c09d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.378+0000 7fef91800700 1 -- 172.21.15.89:0/719078392 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef80009550 con 0x7fef8c0d21f0 2024-04-02T09:14:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.378+0000 7fef91800700 1 --2- 172.21.15.89:0/719078392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef8c0d21f0 0x7fef8c1f6480 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fef80016000 tx=0x7fef80004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.378+0000 7fef89600700 1 -- 172.21.15.89:0/719078392 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef80004590 con 0x7fef8c0d21f0 2024-04-02T09:14:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.378+0000 7fef89600700 1 -- 172.21.15.89:0/719078392 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8001e070 con 0x7fef8c0d21f0 2024-04-02T09:14:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.378+0000 7fef94200700 1 -- 172.21.15.89:0/719078392 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef8c0ca850 con 0x7fef8c0d21f0 2024-04-02T09:14:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.378+0000 7fef89600700 1 -- 172.21.15.89:0/719078392 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef800038a0 con 0x7fef8c0d21f0 2024-04-02T09:14:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.378+0000 7fef94200700 1 -- 172.21.15.89:0/719078392 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef8c09d920 con 0x7fef8c0d21f0 2024-04-02T09:14:07.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.379+0000 7fef94200700 1 -- 172.21.15.89:0/719078392 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef8c15d610 con 0x7fef8c0d21f0 2024-04-02T09:14:07.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.381+0000 7fef89600700 1 -- 172.21.15.89:0/719078392 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7fef80003a00 con 0x7fef8c0d21f0 2024-04-02T09:14:07.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.382+0000 7fef89600700 1 --2- 172.21.15.89:0/719078392 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef6c06aed0 0x7fef6c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:07.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.382+0000 7fef89600700 1 -- 172.21.15.89:0/719078392 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7fef8008ac70 con 0x7fef8c0d21f0 2024-04-02T09:14:07.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.382+0000 7fef90e00700 1 -- 172.21.15.89:0/719078392 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef6c06aed0 msgr2=0x7fef6c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:07.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.382+0000 7fef90e00700 1 --2- 172.21.15.89:0/719078392 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef6c06aed0 0x7fef6c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:07.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.387+0000 7fef89600700 1 -- 172.21.15.89:0/719078392 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fef80055580 con 0x7fef8c0d21f0 2024-04-02T09:14:07.412 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:07.414+0000 7f90b8167240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:14:07.444 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:07.445+0000 7f75006c9240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:14:07.511 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:07.513+0000 7f90b8167240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:14:07.540 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:07.541+0000 7f75006c9240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:14:07.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.583+0000 7fef90e00700 1 -- 172.21.15.89:0/719078392 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef6c06aed0 msgr2=0x7fef6c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:07.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.583+0000 7fef90e00700 1 --2- 172.21.15.89:0/719078392 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef6c06aed0 0x7fef6c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:07.610 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:07.611+0000 7f90b8167240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:14:07.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.615+0000 7fef94200700 1 -- 172.21.15.89:0/719078392 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fef8c067720 con 0x7fef8c0d21f0 2024-04-02T09:14:07.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.616+0000 7fef89600700 1 -- 172.21.15.89:0/719078392 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v249) v1 ==== 81+0+576 (secure 0 0 0) 0x7fef80020250 con 0x7fef8c0d21f0 2024-04-02T09:14:07.615 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1998199546 2024-04-03T09:14:02.049396+0000 2024-04-02T09:14:07.615 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/492795889 2024-04-03T09:14:02.049396+0000 2024-04-02T09:14:07.615 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3750385348 2024-04-03T09:14:02.049396+0000 2024-04-02T09:14:07.616 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2067457213 2024-04-03T09:14:02.049396+0000 2024-04-02T09:14:07.616 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1275405725 2024-04-03T09:14:02.049396+0000 2024-04-02T09:14:07.616 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3355557522 2024-04-03T09:14:02.049396+0000 2024-04-02T09:14:07.616 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/4204293224 2024-04-03T09:14:02.049396+0000 2024-04-02T09:14:07.616 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/176419408 2024-04-03T09:14:02.049396+0000 2024-04-02T09:14:07.616 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/108855 2024-04-03T09:14:02.049396+0000 2024-04-02T09:14:07.616 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/108855 2024-04-03T09:14:02.049396+0000 2024-04-02T09:14:07.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.619+0000 7fef94200700 1 -- 172.21.15.89:0/719078392 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef6c06aed0 msgr2=0x7fef6c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:07.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.619+0000 7fef94200700 1 --2- 172.21.15.89:0/719078392 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef6c06aed0 0x7fef6c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.619+0000 7fef94200700 1 -- 172.21.15.89:0/719078392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef8c0d21f0 msgr2=0x7fef8c1f6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:07.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.619+0000 7fef94200700 1 --2- 172.21.15.89:0/719078392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef8c0d21f0 0x7fef8c1f6480 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fef80016000 tx=0x7fef80004750 comp rx=0 tx=0).stop 2024-04-02T09:14:07.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.619+0000 7fef94200700 1 -- 172.21.15.89:0/719078392 shutdown_connections 2024-04-02T09:14:07.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.619+0000 7fef94200700 1 --2- 172.21.15.89:0/719078392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef8c1611b0 0x7fef8c09d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.619+0000 7fef94200700 1 --2- 172.21.15.89:0/719078392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fef8c0d21f0 0x7fef8c1f6480 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.619+0000 7fef94200700 1 --2- 172.21.15.89:0/719078392 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fef6c06aed0 0x7fef6c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.619+0000 7fef94200700 1 --2- 172.21.15.89:0/719078392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef8c15f4d0 0x7fef8c1f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:07.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.619+0000 7fef94200700 1 -- 172.21.15.89:0/719078392 >> 172.21.15.89:0/719078392 conn(0x7fef8c08aff0 msgr2=0x7fef8c15da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:07.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.619+0000 7fef94200700 1 -- 172.21.15.89:0/719078392 shutdown_connections 2024-04-02T09:14:07.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:07.619+0000 7fef94200700 1 -- 172.21.15.89:0/719078392 wait complete. 2024-04-02T09:14:07.619 INFO:teuthology.orchestra.run.smithi089.stderr:listed 10 entries 2024-04-02T09:14:07.636 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T09:14:07.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:07.637 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T09:14:07.644 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:07.645+0000 7f75006c9240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:14:07.709 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T09:14:07.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:07.710 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T09:14:07.751 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:07.751+0000 7f75006c9240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:14:07.794 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:14:07.795 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:14:07.826 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:07.828+0000 7f90b8167240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:14:07.838 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:14:07.839 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:14:07.839 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:14:07.854 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:07.855+0000 7f75006c9240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:14:07.884 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:14:07.884 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:14:07.885 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:14:07.930 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:14:07.930 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:14:08.091 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:08.092+0000 7f90b8167240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:14:08.094 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:08.095+0000 7f75006c9240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:14:08.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.142+0000 7f570c200700 1 Processor -- start 2024-04-02T09:14:08.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.142+0000 7f570c200700 1 -- start start 2024-04-02T09:14:08.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.143+0000 7f570c200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f570415d920 0x7f570406c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:08.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.143+0000 7f570c200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f570409a910 0x7f57040cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:08.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.143+0000 7f570c200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5704099a60 0x7f5704099eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:08.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.143+0000 7f570c200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570409a3f0 con 0x7f570415d920 2024-04-02T09:14:08.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.143+0000 7f570c200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57040d08d0 con 0x7f5704099a60 2024-04-02T09:14:08.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.143+0000 7f570c200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570406c300 con 0x7f570409a910 2024-04-02T09:14:08.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.144+0000 7f5709800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f570415d920 0x7f570406c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:08.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.144+0000 7f5709800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f570415d920 0x7f570406c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44326/0 (socket says 172.21.15.89:44326) 2024-04-02T09:14:08.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.144+0000 7f5709800700 1 -- 172.21.15.89:0/1303252122 learned_addr learned my addr 172.21.15.89:0/1303252122 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:08.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.144+0000 7f5708e00700 1 --2- 172.21.15.89:0/1303252122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f570409a910 0x7f57040cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:08.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.144+0000 7f570a200700 1 --2- 172.21.15.89:0/1303252122 >> v2:172.21.15.113:3300/0 conn(0x7f5704099a60 0x7f5704099eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:08.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.144+0000 7f5709800700 1 -- 172.21.15.89:0/1303252122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f570409a910 msgr2=0x7f57040cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:08.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.144+0000 7f5709800700 1 --2- 172.21.15.89:0/1303252122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f570409a910 0x7f57040cfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.144+0000 7f5709800700 1 -- 172.21.15.89:0/1303252122 >> v2:172.21.15.113:3300/0 conn(0x7f5704099a60 msgr2=0x7f5704099eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:08.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.144+0000 7f5709800700 1 --2- 172.21.15.89:0/1303252122 >> v2:172.21.15.113:3300/0 conn(0x7f5704099a60 0x7f5704099eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.144+0000 7f5709800700 1 -- 172.21.15.89:0/1303252122 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570406b9d0 con 0x7f570415d920 2024-04-02T09:14:08.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.144+0000 7f570a200700 1 --2- 172.21.15.89:0/1303252122 >> v2:172.21.15.113:3300/0 conn(0x7f5704099a60 0x7f5704099eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:08.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.144+0000 7f5709800700 1 --2- 172.21.15.89:0/1303252122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570415d920 0x7f570406c790 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f5704074590 tx=0x7f56f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3852595a6f7fb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:08.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.145+0000 7f5703e00700 1 -- 172.21.15.89:0/1303252122 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8011830 con 0x7f570415d920 2024-04-02T09:14:08.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.145+0000 7f5703e00700 1 -- 172.21.15.89:0/1303252122 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f8011990 con 0x7f570415d920 2024-04-02T09:14:08.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.145+0000 7f5703e00700 1 -- 172.21.15.89:0/1303252122 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8011d20 con 0x7f570415d920 2024-04-02T09:14:08.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.145+0000 7f570c200700 1 -- 172.21.15.89:0/1303252122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570415d920 msgr2=0x7f570406c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:08.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.145+0000 7f570c200700 1 --2- 172.21.15.89:0/1303252122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570415d920 0x7f570406c790 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f5704074590 tx=0x7f56f8009bd0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.145+0000 7f570c200700 1 -- 172.21.15.89:0/1303252122 shutdown_connections 2024-04-02T09:14:08.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.145+0000 7f570c200700 1 --2- 172.21.15.89:0/1303252122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f570415d920 0x7f570406c790 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.145+0000 7f570c200700 1 --2- 172.21.15.89:0/1303252122 >> v2:172.21.15.113:3300/0 conn(0x7f5704099a60 0x7f5704099eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.145+0000 7f570c200700 1 --2- 172.21.15.89:0/1303252122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f570409a910 0x7f57040cfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.145+0000 7f570c200700 1 -- 172.21.15.89:0/1303252122 >> 172.21.15.89:0/1303252122 conn(0x7f57040d1350 msgr2=0x7f57040d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:08.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.145+0000 7f570c200700 1 -- 172.21.15.89:0/1303252122 shutdown_connections 2024-04-02T09:14:08.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.146+0000 7f570c200700 1 -- 172.21.15.89:0/1303252122 wait complete. 2024-04-02T09:14:08.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.148+0000 7f570c200700 1 Processor -- start 2024-04-02T09:14:08.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.148+0000 7f570c200700 1 -- start start 2024-04-02T09:14:08.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.149+0000 7f570c200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5704099a60 0x7f57040b6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:08.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.149+0000 7f570c200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f570409a910 0x7f570409eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:08.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.149+0000 7f570c200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f570415d920 0x7f570409b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:08.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.149+0000 7f570c200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57040b56f0 con 0x7f5704099a60 2024-04-02T09:14:08.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.149+0000 7f570c200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57040b59d0 con 0x7f570415d920 2024-04-02T09:14:08.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.149+0000 7f570c200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57040b5cb0 con 0x7f570409a910 2024-04-02T09:14:08.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.149+0000 7f5709800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5704099a60 0x7f57040b6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.149+0000 7f5709800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5704099a60 0x7f57040b6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44328/0 (socket says 172.21.15.89:44328) 2024-04-02T09:14:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.150+0000 7f5709800700 1 -- 172.21.15.89:0/2124113734 learned_addr learned my addr 172.21.15.89:0/2124113734 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.150+0000 7f5708e00700 1 --2- 172.21.15.89:0/2124113734 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f570409a910 0x7f570409eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.150+0000 7f570a200700 1 --2- 172.21.15.89:0/2124113734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f570415d920 0x7f570409b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.150+0000 7f5709800700 1 -- 172.21.15.89:0/2124113734 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f570409a910 msgr2=0x7f570409eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.150+0000 7f5709800700 1 --2- 172.21.15.89:0/2124113734 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f570409a910 0x7f570409eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.150+0000 7f5709800700 1 -- 172.21.15.89:0/2124113734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f570415d920 msgr2=0x7f570409b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.150+0000 7f5709800700 1 --2- 172.21.15.89:0/2124113734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f570415d920 0x7f570409b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.150+0000 7f5709800700 1 -- 172.21.15.89:0/2124113734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f8009550 con 0x7f5704099a60 2024-04-02T09:14:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.150+0000 7f5709800700 1 --2- 172.21.15.89:0/2124113734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5704099a60 0x7f57040b6790 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009c10 tx=0x7f56f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.150+0000 7f5701600700 1 -- 172.21.15.89:0/2124113734 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8004450 con 0x7f5704099a60 2024-04-02T09:14:08.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.150+0000 7f5701600700 1 -- 172.21.15.89:0/2124113734 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f801e070 con 0x7f5704099a60 2024-04-02T09:14:08.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.150+0000 7f570c200700 1 -- 172.21.15.89:0/2124113734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57040b5f90 con 0x7f5704099a60 2024-04-02T09:14:08.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.150+0000 7f5701600700 1 -- 172.21.15.89:0/2124113734 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f80038e0 con 0x7f5704099a60 2024-04-02T09:14:08.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.151+0000 7f570c200700 1 -- 172.21.15.89:0/2124113734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570409f060 con 0x7f5704099a60 2024-04-02T09:14:08.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.152+0000 7f570c200700 1 -- 172.21.15.89:0/2124113734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57040b64b0 con 0x7f5704099a60 2024-04-02T09:14:08.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.153+0000 7f5701600700 1 -- 172.21.15.89:0/2124113734 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f56f8003a40 con 0x7f5704099a60 2024-04-02T09:14:08.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.154+0000 7f5701600700 1 --2- 172.21.15.89:0/2124113734 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f56e406aed0 0x7f56e406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:08.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.154+0000 7f5701600700 1 -- 172.21.15.89:0/2124113734 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f56f808aa60 con 0x7f5704099a60 2024-04-02T09:14:08.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.154+0000 7f5708e00700 1 -- 172.21.15.89:0/2124113734 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f56e406aed0 msgr2=0x7f56e406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:08.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.154+0000 7f5708e00700 1 --2- 172.21.15.89:0/2124113734 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f56e406aed0 0x7f56e406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:08.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.162+0000 7f5701600700 1 -- 172.21.15.89:0/2124113734 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f56f8055370 con 0x7f5704099a60 2024-04-02T09:14:08.284 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:08.286+0000 7f90b8167240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:14:08.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.355+0000 7f5708e00700 1 -- 172.21.15.89:0/2124113734 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f56e406aed0 msgr2=0x7f56e406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:08.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.355+0000 7f5708e00700 1 --2- 172.21.15.89:0/2124113734 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f56e406aed0 0x7f56e406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:08.366 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:08.367+0000 7f75006c9240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:14:08.381 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:08.382+0000 7f90b8167240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:14:08.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.423+0000 7f570c200700 1 -- 172.21.15.89:0/2124113734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57040d0260 con 0x7f5704099a60 2024-04-02T09:14:08.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.423+0000 7f5701600700 1 -- 172.21.15.89:0/2124113734 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5968 (secure 0 0 0) 0x7f56f8058990 con 0x7f5704099a60 2024-04-02T09:14:08.423 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:08.423 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":125,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9518,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3094781297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3094781297},{"type":"v1","addr":"172.21.15.89:6836","nonce":3094781297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":10978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1799351491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1799351491},{"type":"v1","addr":"172.21.15.113:6834","nonce":1799351491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:28.357996+0000","modified":"2024-04-02T09:13:31.369015+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11964":{"gid":11964,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6835/703852199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":703852199},{"type":"v1","addr":"172.21.15.113:6835","nonce":703852199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:41.933360+0000","modified":"2024-04-02T09:13:45.439462+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11958},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11958":{"gid":11958,"name":"a","rank":0,"incarnation":121,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6837/3112580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3112580185},{"type":"v1","addr":"172.21.15.89:6837","nonce":3112580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-02T09:14:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.426+0000 7f570c200700 1 -- 172.21.15.89:0/2124113734 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f56e406aed0 msgr2=0x7f56e406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.426+0000 7f570c200700 1 --2- 172.21.15.89:0/2124113734 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f56e406aed0 0x7f56e406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.426+0000 7f570c200700 1 -- 172.21.15.89:0/2124113734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5704099a60 msgr2=0x7f57040b6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.426+0000 7f570c200700 1 --2- 172.21.15.89:0/2124113734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5704099a60 0x7f57040b6790 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009c10 tx=0x7f56f8004750 comp rx=0 tx=0).stop 2024-04-02T09:14:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.426+0000 7f570c200700 1 -- 172.21.15.89:0/2124113734 shutdown_connections 2024-04-02T09:14:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.426+0000 7f570c200700 1 --2- 172.21.15.89:0/2124113734 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f56e406aed0 0x7f56e406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.426+0000 7f570c200700 1 --2- 172.21.15.89:0/2124113734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f570415d920 0x7f570409b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.426+0000 7f570c200700 1 --2- 172.21.15.89:0/2124113734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5704099a60 0x7f57040b6790 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.426+0000 7f570c200700 1 --2- 172.21.15.89:0/2124113734 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f570409a910 0x7f570409eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.426+0000 7f570c200700 1 -- 172.21.15.89:0/2124113734 >> 172.21.15.89:0/2124113734 conn(0x7f57040d1350 msgr2=0x7f57040b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.426+0000 7f570c200700 1 -- 172.21.15.89:0/2124113734 shutdown_connections 2024-04-02T09:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.426+0000 7f570c200700 1 -- 172.21.15.89:0/2124113734 wait complete. 2024-04-02T09:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 125 2024-04-02T09:14:08.446 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:14:08.578 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:08.578+0000 7f75006c9240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:14:08.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.640+0000 7f5232400700 1 Processor -- start 2024-04-02T09:14:08.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.640+0000 7f5232400700 1 -- start start 2024-04-02T09:14:08.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.642+0000 7f5232400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f522c15d710 0x7f522c0d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:08.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.642+0000 7f5232400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f522c09aba0 0x7f522c0d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:08.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.642+0000 7f5232400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522c06bb40 con 0x7f522c06bf40 2024-04-02T09:14:08.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.642+0000 7f5232400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f522c099b00 con 0x7f522c09aba0 2024-04-02T09:14:08.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.642+0000 7f5232400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f522c099de0 con 0x7f522c15d710 2024-04-02T09:14:08.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.642+0000 7f522aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f522c09aba0 0x7f522c0d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:08.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.642+0000 7f522aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f522c09aba0 0x7f522c0d0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44340/0 (socket says 172.21.15.89:44340) 2024-04-02T09:14:08.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.642+0000 7f522aa00700 1 -- 172.21.15.89:0/3434804197 learned_addr learned my addr 172.21.15.89:0/3434804197 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:08.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.643+0000 7f522b400700 1 --2- 172.21.15.89:0/3434804197 >> v2:172.21.15.113:3300/0 conn(0x7f522c15d710 0x7f522c0d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:08.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.643+0000 7f522aa00700 1 -- 172.21.15.89:0/3434804197 >> v2:172.21.15.113:3300/0 conn(0x7f522c15d710 msgr2=0x7f522c0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:08.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.643+0000 7f522aa00700 1 --2- 172.21.15.89:0/3434804197 >> v2:172.21.15.113:3300/0 conn(0x7f522c15d710 0x7f522c0d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.643+0000 7f522aa00700 1 -- 172.21.15.89:0/3434804197 >> v1:172.21.15.113:6789/0 conn(0x7f522c06bf40 legacy=0x7f522c0ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:08.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.643+0000 7f522aa00700 1 -- 172.21.15.89:0/3434804197 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c09a0c0 con 0x7f522c09aba0 2024-04-02T09:14:08.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.643+0000 7f522b400700 1 --2- 172.21.15.89:0/3434804197 >> v2:172.21.15.113:3300/0 conn(0x7f522c15d710 0x7f522c0d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:08.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.643+0000 7f522aa00700 1 --2- 172.21.15.89:0/3434804197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f522c09aba0 0x7f522c0d0670 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f522c071ec0 tx=0x7f5218009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da7ac21ff8dd9f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:08.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.643+0000 7f522a000700 1 -- 172.21.15.89:0/3434804197 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521800b870 con 0x7f522c09aba0 2024-04-02T09:14:08.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.643+0000 7f522a000700 1 -- 172.21.15.89:0/3434804197 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521800b9d0 con 0x7f522c09aba0 2024-04-02T09:14:08.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.643+0000 7f522a000700 1 -- 172.21.15.89:0/3434804197 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521800bd60 con 0x7f522c09aba0 2024-04-02T09:14:08.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.644+0000 7f5232400700 1 -- 172.21.15.89:0/3434804197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f522c09aba0 msgr2=0x7f522c0d0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:08.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.644+0000 7f5232400700 1 --2- 172.21.15.89:0/3434804197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f522c09aba0 0x7f522c0d0670 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f522c071ec0 tx=0x7f5218009bd0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.644+0000 7f5232400700 1 -- 172.21.15.89:0/3434804197 shutdown_connections 2024-04-02T09:14:08.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.644+0000 7f5232400700 1 --2- 172.21.15.89:0/3434804197 >> v2:172.21.15.113:3300/0 conn(0x7f522c15d710 0x7f522c0d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.644+0000 7f5232400700 1 --2- 172.21.15.89:0/3434804197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f522c09aba0 0x7f522c0d0670 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.644+0000 7f5232400700 1 -- 172.21.15.89:0/3434804197 >> 172.21.15.89:0/3434804197 conn(0x7f522c0d13e0 msgr2=0x7f522c0d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:08.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.644+0000 7f5232400700 1 -- 172.21.15.89:0/3434804197 shutdown_connections 2024-04-02T09:14:08.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.644+0000 7f5232400700 1 -- 172.21.15.89:0/3434804197 wait complete. 2024-04-02T09:14:08.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.647+0000 7f5232400700 1 Processor -- start 2024-04-02T09:14:08.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.647+0000 7f5232400700 1 -- start start 2024-04-02T09:14:08.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.648+0000 7f5232400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f522c06bf40 0x7f522c0b1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:08.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f5232400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f522c09aba0 0x7f522c153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:08.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f5232400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f522c15d710 0x7f522c1542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:08.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f5232400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c15df80 con 0x7f522c15d710 2024-04-02T09:14:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f5232400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c1575e0 con 0x7f522c09aba0 2024-04-02T09:14:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f5232400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522c1578c0 con 0x7f522c06bf40 2024-04-02T09:14:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f522be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f522c15d710 0x7f522c1542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f522be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f522c15d710 0x7f522c1542e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44344/0 (socket says 172.21.15.89:44344) 2024-04-02T09:14:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f522b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f522c06bf40 0x7f522c0b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f522be00700 1 -- 172.21.15.89:0/1373055607 learned_addr learned my addr 172.21.15.89:0/1373055607 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f522aa00700 1 --2- 172.21.15.89:0/1373055607 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f522c09aba0 0x7f522c153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f522be00700 1 -- 172.21.15.89:0/1373055607 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f522c06bf40 msgr2=0x7f522c0b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f522be00700 1 --2- 172.21.15.89:0/1373055607 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f522c06bf40 0x7f522c0b1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f522be00700 1 -- 172.21.15.89:0/1373055607 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f522c09aba0 msgr2=0x7f522c153da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f522be00700 1 --2- 172.21.15.89:0/1373055607 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f522c09aba0 0x7f522c153da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f522be00700 1 -- 172.21.15.89:0/1373055607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5218009550 con 0x7f522c15d710 2024-04-02T09:14:08.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.649+0000 7f522aa00700 1 --2- 172.21.15.89:0/1373055607 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f522c09aba0 0x7f522c153da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:14:08.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.650+0000 7f522be00700 1 --2- 172.21.15.89:0/1373055607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f522c15d710 0x7f522c1542e0 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f522400cc50 tx=0x7f522400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:08.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.650+0000 7f5223400700 1 -- 172.21.15.89:0/1373055607 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5224003c00 con 0x7f522c15d710 2024-04-02T09:14:08.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.650+0000 7f5223400700 1 -- 172.21.15.89:0/1373055607 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5224003d60 con 0x7f522c15d710 2024-04-02T09:14:08.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.650+0000 7f5232400700 1 -- 172.21.15.89:0/1373055607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522c157ba0 con 0x7f522c15d710 2024-04-02T09:14:08.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.650+0000 7f5223400700 1 -- 172.21.15.89:0/1373055607 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522400aab0 con 0x7f522c15d710 2024-04-02T09:14:08.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.650+0000 7f5232400700 1 -- 172.21.15.89:0/1373055607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522c155100 con 0x7f522c15d710 2024-04-02T09:14:08.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.651+0000 7f5232400700 1 -- 172.21.15.89:0/1373055607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f522c15e0f0 con 0x7f522c15d710 2024-04-02T09:14:08.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.652+0000 7f5223400700 1 -- 172.21.15.89:0/1373055607 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f522400ac10 con 0x7f522c15d710 2024-04-02T09:14:08.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.653+0000 7f5223400700 1 --2- 172.21.15.89:0/1373055607 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f520c06aed0 0x7f520c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:08.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.653+0000 7f5223400700 1 -- 172.21.15.89:0/1373055607 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f5224089e80 con 0x7f522c15d710 2024-04-02T09:14:08.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.653+0000 7f522b400700 1 -- 172.21.15.89:0/1373055607 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f520c06aed0 msgr2=0x7f520c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:08.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.653+0000 7f522b400700 1 --2- 172.21.15.89:0/1373055607 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f520c06aed0 0x7f520c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:08.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.657+0000 7f5223400700 1 -- 172.21.15.89:0/1373055607 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f52240545a0 con 0x7f522c15d710 2024-04-02T09:14:08.679 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:08.680+0000 7f75006c9240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:14:08.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.854+0000 7f522b400700 1 -- 172.21.15.89:0/1373055607 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f520c06aed0 msgr2=0x7f520c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:08.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.854+0000 7f522b400700 1 --2- 172.21.15.89:0/1373055607 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f520c06aed0 0x7f520c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:08.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.869+0000 7f5232400700 1 -- 172.21.15.89:0/1373055607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f522c067720 con 0x7f522c15d710 2024-04-02T09:14:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.871+0000 7f5223400700 1 -- 172.21.15.89:0/1373055607 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) v1 ==== 74+0+21019 (secure 0 0 0) 0x7f5224057bc0 con 0x7f522c15d710 2024-04-02T09:14:08.871 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:08.871 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":249,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:14:02.049435+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:13:26.654647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-02T09:13:27.342285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:13:40.048039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:13:40.907706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/1998199546":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/492795889":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/3750385348":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/2067457213":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/1275405725":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/3355557522":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/4204293224":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/176419408":"2024-04-03T09:14:02.049396+0000","172.21.15.89:6833/108855":"2024-04-03T09:14:02.049396+0000","172.21.15.89:6832/108855":"2024-04-03T09:14:02.049396+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:14:08.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.873+0000 7f5232400700 1 -- 172.21.15.89:0/1373055607 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f520c06aed0 msgr2=0x7f520c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:08.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.873+0000 7f5232400700 1 --2- 172.21.15.89:0/1373055607 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f520c06aed0 0x7f520c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.873+0000 7f5232400700 1 -- 172.21.15.89:0/1373055607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f522c15d710 msgr2=0x7f522c1542e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:08.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.873+0000 7f5232400700 1 --2- 172.21.15.89:0/1373055607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f522c15d710 0x7f522c1542e0 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f522400cc50 tx=0x7f522400cf60 comp rx=0 tx=0).stop 2024-04-02T09:14:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.874+0000 7f5232400700 1 -- 172.21.15.89:0/1373055607 shutdown_connections 2024-04-02T09:14:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.874+0000 7f5232400700 1 --2- 172.21.15.89:0/1373055607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f522c15d710 0x7f522c1542e0 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.874+0000 7f5232400700 1 --2- 172.21.15.89:0/1373055607 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f522c06bf40 0x7f522c0b1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.874+0000 7f5232400700 1 --2- 172.21.15.89:0/1373055607 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f520c06aed0 0x7f520c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.874+0000 7f5232400700 1 --2- 172.21.15.89:0/1373055607 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f522c09aba0 0x7f522c153da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.874+0000 7f5232400700 1 -- 172.21.15.89:0/1373055607 >> 172.21.15.89:0/1373055607 conn(0x7f522c0d13e0 msgr2=0x7f522c0b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.874+0000 7f5232400700 1 -- 172.21.15.89:0/1373055607 shutdown_connections 2024-04-02T09:14:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:08.874+0000 7f5232400700 1 -- 172.21.15.89:0/1373055607 wait complete. 2024-04-02T09:14:08.891 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T09:13:26.654647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '243', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T09:13:27.342285+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '244', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-02T09:13:40.048039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-02T09:13:40.907706+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:14:08.892 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:14:09.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.092+0000 7f782aa00700 1 Processor -- start 2024-04-02T09:14:09.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.092+0000 7f782aa00700 1 -- start start 2024-04-02T09:14:09.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.093+0000 7f782aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78241611b0 0x7f782415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:09.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.093+0000 7f782aa00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f78240cee10 0x7f7824161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:09.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.093+0000 7f782aa00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78240ce270 con 0x7f782415f4d0 2024-04-02T09:14:09.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.093+0000 7f782aa00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f782415b600 con 0x7f78241611b0 2024-04-02T09:14:09.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.093+0000 7f782aa00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78240d2760 con 0x7f78240cee10 2024-04-02T09:14:09.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.093+0000 7f7823e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78241611b0 0x7f782415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:09.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.094+0000 7f7823e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78241611b0 0x7f782415ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49742/0 (socket says 172.21.15.89:49742) 2024-04-02T09:14:09.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.094+0000 7f7823e00700 1 -- 172.21.15.89:0/396605823 learned_addr learned my addr 172.21.15.89:0/396605823 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:09.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.094+0000 7f7828a00700 1 --2- 172.21.15.89:0/396605823 >> v2:172.21.15.113:3300/0 conn(0x7f78240cee10 0x7f7824161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:09.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.094+0000 7f7823e00700 1 -- 172.21.15.89:0/396605823 >> v2:172.21.15.113:3300/0 conn(0x7f78240cee10 msgr2=0x7f7824161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:09.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.094+0000 7f7823e00700 1 --2- 172.21.15.89:0/396605823 >> v2:172.21.15.113:3300/0 conn(0x7f78240cee10 0x7f7824161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.094+0000 7f7823e00700 1 -- 172.21.15.89:0/396605823 >> v1:172.21.15.113:6789/0 conn(0x7f782415f4d0 legacy=0x7f78240ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:09.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.094+0000 7f7823e00700 1 -- 172.21.15.89:0/396605823 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78240d2a40 con 0x7f78241611b0 2024-04-02T09:14:09.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.094+0000 7f7823e00700 1 --2- 172.21.15.89:0/396605823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78241611b0 0x7f782415ef90 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f78240d2f60 tx=0x7f7814009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b29842878b6c4d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:09.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.094+0000 7f7822a00700 1 -- 172.21.15.89:0/396605823 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781400b870 con 0x7f78241611b0 2024-04-02T09:14:09.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.094+0000 7f7822a00700 1 -- 172.21.15.89:0/396605823 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781400b9d0 con 0x7f78241611b0 2024-04-02T09:14:09.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.094+0000 7f7822a00700 1 -- 172.21.15.89:0/396605823 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781400bd60 con 0x7f78241611b0 2024-04-02T09:14:09.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.095+0000 7f782aa00700 1 -- 172.21.15.89:0/396605823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78241611b0 msgr2=0x7f782415ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:09.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.095+0000 7f782aa00700 1 --2- 172.21.15.89:0/396605823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78241611b0 0x7f782415ef90 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f78240d2f60 tx=0x7f7814009bd0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.095+0000 7f782aa00700 1 -- 172.21.15.89:0/396605823 shutdown_connections 2024-04-02T09:14:09.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.095+0000 7f782aa00700 1 --2- 172.21.15.89:0/396605823 >> v2:172.21.15.113:3300/0 conn(0x7f78240cee10 0x7f7824161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.095+0000 7f782aa00700 1 --2- 172.21.15.89:0/396605823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78241611b0 0x7f782415ef90 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.095+0000 7f782aa00700 1 -- 172.21.15.89:0/396605823 >> 172.21.15.89:0/396605823 conn(0x7f782408aff0 msgr2=0x7f78240d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:09.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.095+0000 7f782aa00700 1 -- 172.21.15.89:0/396605823 shutdown_connections 2024-04-02T09:14:09.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.095+0000 7f782aa00700 1 -- 172.21.15.89:0/396605823 wait complete. 2024-04-02T09:14:09.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.097+0000 7f782aa00700 1 Processor -- start 2024-04-02T09:14:09.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.097+0000 7f782aa00700 1 -- start start 2024-04-02T09:14:09.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.098+0000 7f782aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78240cee10 0x7f7824152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:09.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.098+0000 7f782aa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f782415f4d0 0x7f78240cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:09.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.098+0000 7f782aa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78241611b0 0x7f78240c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:09.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.098+0000 7f782aa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78240cc5f0 con 0x7f78240cee10 2024-04-02T09:14:09.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.098+0000 7f782aa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78240cc030 con 0x7f78241611b0 2024-04-02T09:14:09.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.098+0000 7f782aa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78240cc310 con 0x7f782415f4d0 2024-04-02T09:14:09.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.098+0000 7f7823e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78240cee10 0x7f7824152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:09.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.098+0000 7f7823e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78240cee10 0x7f7824152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49744/0 (socket says 172.21.15.89:49744) 2024-04-02T09:14:09.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.098+0000 7f7823e00700 1 -- 172.21.15.89:0/4008030473 learned_addr learned my addr 172.21.15.89:0/4008030473 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:09.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.098+0000 7f7828a00700 1 --2- 172.21.15.89:0/4008030473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78241611b0 0x7f78240c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:09.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.098+0000 7f7823400700 1 --2- 172.21.15.89:0/4008030473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f782415f4d0 0x7f78240cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:09.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.099+0000 7f7823e00700 1 -- 172.21.15.89:0/4008030473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f782415f4d0 msgr2=0x7f78240cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:09.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.099+0000 7f7823e00700 1 --2- 172.21.15.89:0/4008030473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f782415f4d0 0x7f78240cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.099+0000 7f7823e00700 1 -- 172.21.15.89:0/4008030473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78241611b0 msgr2=0x7f78240c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:09.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.099+0000 7f7823e00700 1 --2- 172.21.15.89:0/4008030473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78241611b0 0x7f78240c93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.099+0000 7f7823e00700 1 -- 172.21.15.89:0/4008030473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7814009550 con 0x7f78240cee10 2024-04-02T09:14:09.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.099+0000 7f7823e00700 1 --2- 172.21.15.89:0/4008030473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78240cee10 0x7f7824152470 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f7814000c00 tx=0x7f7814004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:09.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.099+0000 7f781be00700 1 -- 172.21.15.89:0/4008030473 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7814004310 con 0x7f78240cee10 2024-04-02T09:14:09.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.099+0000 7f781be00700 1 -- 172.21.15.89:0/4008030473 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781401e070 con 0x7f78240cee10 2024-04-02T09:14:09.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.099+0000 7f781be00700 1 -- 172.21.15.89:0/4008030473 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78140036a0 con 0x7f78240cee10 2024-04-02T09:14:09.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.099+0000 7f782aa00700 1 -- 172.21.15.89:0/4008030473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78240c7100 con 0x7f78240cee10 2024-04-02T09:14:09.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.099+0000 7f782aa00700 1 -- 172.21.15.89:0/4008030473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78241529b0 con 0x7f78240cee10 2024-04-02T09:14:09.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.101+0000 7f782aa00700 1 -- 172.21.15.89:0/4008030473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7824067720 con 0x7f78240cee10 2024-04-02T09:14:09.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.102+0000 7f781be00700 1 -- 172.21.15.89:0/4008030473 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f7814020030 con 0x7f78240cee10 2024-04-02T09:14:09.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.103+0000 7f781be00700 1 --2- 172.21.15.89:0/4008030473 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f780406aed0 0x7f780406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:09.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.103+0000 7f7823400700 1 -- 172.21.15.89:0/4008030473 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f780406aed0 msgr2=0x7f780406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:09.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.103+0000 7f781be00700 1 -- 172.21.15.89:0/4008030473 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f781408ac80 con 0x7f78240cee10 2024-04-02T09:14:09.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.103+0000 7f7823400700 1 --2- 172.21.15.89:0/4008030473 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f780406aed0 0x7f780406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:09.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.107+0000 7f781be00700 1 -- 172.21.15.89:0/4008030473 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7814055510 con 0x7f78240cee10 2024-04-02T09:14:09.115 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:09.116+0000 7f90b8167240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:14:09.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.304+0000 7f7823400700 1 -- 172.21.15.89:0/4008030473 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f780406aed0 msgr2=0x7f780406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:09.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.304+0000 7f7823400700 1 --2- 172.21.15.89:0/4008030473 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f780406aed0 0x7f780406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:09.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.320+0000 7f782aa00700 1 -- 172.21.15.89:0/4008030473 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f78240b5d00 con 0x7f78240cee10 2024-04-02T09:14:09.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.321+0000 7f781be00700 1 -- 172.21.15.89:0/4008030473 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v249) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7814058b30 con 0x7f78240cee10 2024-04-02T09:14:09.320 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:14:09.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.323+0000 7f782aa00700 1 -- 172.21.15.89:0/4008030473 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f780406aed0 msgr2=0x7f780406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:09.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.323+0000 7f782aa00700 1 --2- 172.21.15.89:0/4008030473 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f780406aed0 0x7f780406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.323+0000 7f782aa00700 1 -- 172.21.15.89:0/4008030473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78240cee10 msgr2=0x7f7824152470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:09.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.323+0000 7f782aa00700 1 --2- 172.21.15.89:0/4008030473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78240cee10 0x7f7824152470 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f7814000c00 tx=0x7f7814004750 comp rx=0 tx=0).stop 2024-04-02T09:14:09.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.323+0000 7f782aa00700 1 -- 172.21.15.89:0/4008030473 shutdown_connections 2024-04-02T09:14:09.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.323+0000 7f782aa00700 1 --2- 172.21.15.89:0/4008030473 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f780406aed0 0x7f780406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.323+0000 7f782aa00700 1 --2- 172.21.15.89:0/4008030473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78241611b0 0x7f78240c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.323+0000 7f782aa00700 1 --2- 172.21.15.89:0/4008030473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78240cee10 0x7f7824152470 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.323+0000 7f782aa00700 1 --2- 172.21.15.89:0/4008030473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f782415f4d0 0x7f78240cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.323+0000 7f782aa00700 1 -- 172.21.15.89:0/4008030473 >> 172.21.15.89:0/4008030473 conn(0x7f782408aff0 msgr2=0x7f782406c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:09.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.323+0000 7f782aa00700 1 -- 172.21.15.89:0/4008030473 shutdown_connections 2024-04-02T09:14:09.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.323+0000 7f782aa00700 1 -- 172.21.15.89:0/4008030473 wait complete. 2024-04-02T09:14:09.332 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:09.333+0000 7f90b8167240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:14:09.343 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T09:14:09.457 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:09.458+0000 7f75006c9240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:14:09.538 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:09.539+0000 7f90b8167240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:14:09.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.545+0000 7f7c99200700 1 Processor -- start 2024-04-02T09:14:09.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.545+0000 7f7c99200700 1 -- start start 2024-04-02T09:14:09.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.546+0000 7f7c99200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c940d16a0 0x7f7c940d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:09.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.546+0000 7f7c99200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7c940d2370 0x7f7c940ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:09.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.546+0000 7f7c99200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c94161420 con 0x7f7c9415d9d0 2024-04-02T09:14:09.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.546+0000 7f7c99200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c940d2820 con 0x7f7c940d2370 2024-04-02T09:14:09.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.546+0000 7f7c99200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c940d4340 con 0x7f7c940d16a0 2024-04-02T09:14:09.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.547+0000 7f7c92800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7c940d2370 0x7f7c940ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:09.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.547+0000 7f7c91e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f7c9415d9d0 0x7f7c940d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:57364/0 (socket says 172.21.15.89:57364) 2024-04-02T09:14:09.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.547+0000 7f7c91e00700 1 -- 172.21.15.89:0/2521028520 learned_addr learned my addr 172.21.15.89:0/2521028520 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:09.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.547+0000 7f7c91400700 1 --2- 172.21.15.89:0/2521028520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c940d16a0 0x7f7c940d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:09.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.547+0000 7f7c92800700 1 -- 172.21.15.89:0/2521028520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c940d16a0 msgr2=0x7f7c940d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:09.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.547+0000 7f7c92800700 1 --2- 172.21.15.89:0/2521028520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c940d16a0 0x7f7c940d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.547+0000 7f7c92800700 1 -- 172.21.15.89:0/2521028520 >> v1:172.21.15.113:6789/0 conn(0x7f7c9415d9d0 legacy=0x7f7c940d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.547+0000 7f7c92800700 1 -- 172.21.15.89:0/2521028520 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c94161700 con 0x7f7c940d2370 2024-04-02T09:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.547+0000 7f7c90a00700 1 -- 172.21.15.89:0/2521028520 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449632965 0 0) 0x7f7c94161420 con 0x7f7c9415d9d0 2024-04-02T09:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.548+0000 7f7c92800700 1 --2- 172.21.15.89:0/2521028520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c940d2370 0x7f7c940ce360 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c009dc0 tx=0x7f7c8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=514bdf19531c32c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.548+0000 7f7c90a00700 1 -- 172.21.15.89:0/2521028520 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c00e070 con 0x7f7c940d2370 2024-04-02T09:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.548+0000 7f7c90a00700 1 -- 172.21.15.89:0/2521028520 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8c00bae0 con 0x7f7c940d2370 2024-04-02T09:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.548+0000 7f7c90a00700 1 -- 172.21.15.89:0/2521028520 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c00be90 con 0x7f7c940d2370 2024-04-02T09:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.548+0000 7f7c99200700 1 -- 172.21.15.89:0/2521028520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c940d2370 msgr2=0x7f7c940ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.548+0000 7f7c99200700 1 --2- 172.21.15.89:0/2521028520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c940d2370 0x7f7c940ce360 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c009dc0 tx=0x7f7c8c00b330 comp rx=0 tx=0).stop 2024-04-02T09:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.549+0000 7f7c99200700 1 -- 172.21.15.89:0/2521028520 shutdown_connections 2024-04-02T09:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.549+0000 7f7c99200700 1 --2- 172.21.15.89:0/2521028520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c940d2370 0x7f7c940ce360 unknown :-1 s=CLOSED pgs=2681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.549+0000 7f7c99200700 1 --2- 172.21.15.89:0/2521028520 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c940d16a0 0x7f7c940d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.549+0000 7f7c99200700 1 -- 172.21.15.89:0/2521028520 >> 172.21.15.89:0/2521028520 conn(0x7f7c9408aff0 msgr2=0x7f7c940d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.549+0000 7f7c99200700 1 -- 172.21.15.89:0/2521028520 shutdown_connections 2024-04-02T09:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.549+0000 7f7c99200700 1 -- 172.21.15.89:0/2521028520 wait complete. 2024-04-02T09:14:09.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.550+0000 7f7c99200700 1 Processor -- start 2024-04-02T09:14:09.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.551+0000 7f7c99200700 1 -- start start 2024-04-02T09:14:09.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.551+0000 7f7c99200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c940d16a0 0x7f7c940b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:09.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.551+0000 7f7c99200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c940d2370 0x7f7c94158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:09.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.551+0000 7f7c99200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c9415d9d0 0x7f7c94158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:09.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c99200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c941e8470 con 0x7f7c940d2370 2024-04-02T09:14:09.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c99200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c94159390 con 0x7f7c940d16a0 2024-04-02T09:14:09.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c99200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c941e8190 con 0x7f7c9415d9d0 2024-04-02T09:14:09.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c91400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c940d2370 0x7f7c94158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:09.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c91e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c940d16a0 0x7f7c940b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:09.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c91e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c940d16a0 0x7f7c940b4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:48630/0 (socket says 172.21.15.89:48630) 2024-04-02T09:14:09.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c91400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c940d2370 0x7f7c94158910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49772/0 (socket says 172.21.15.89:49772) 2024-04-02T09:14:09.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c91e00700 1 -- 172.21.15.89:0/662109884 learned_addr learned my addr 172.21.15.89:0/662109884 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:09.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c92800700 1 --2- 172.21.15.89:0/662109884 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c9415d9d0 0x7f7c94158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:09.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c91e00700 1 -- 172.21.15.89:0/662109884 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c9415d9d0 msgr2=0x7f7c94158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:09.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c91e00700 1 --2- 172.21.15.89:0/662109884 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c9415d9d0 0x7f7c94158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c91e00700 1 -- 172.21.15.89:0/662109884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c940d2370 msgr2=0x7f7c94158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:09.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c91e00700 1 --2- 172.21.15.89:0/662109884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c940d2370 0x7f7c94158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c91e00700 1 -- 172.21.15.89:0/662109884 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8c009a70 con 0x7f7c940d16a0 2024-04-02T09:14:09.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.552+0000 7f7c92800700 1 --2- 172.21.15.89:0/662109884 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c9415d9d0 0x7f7c94158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:14:09.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.553+0000 7f7c91e00700 1 --2- 172.21.15.89:0/662109884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c940d16a0 0x7f7c940b4860 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f7c8400c550 tx=0x7f7c8400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:09.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.553+0000 7f7c8a000700 1 -- 172.21.15.89:0/662109884 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c84004c80 con 0x7f7c940d16a0 2024-04-02T09:14:09.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.553+0000 7f7c8a000700 1 -- 172.21.15.89:0/662109884 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c84004e00 con 0x7f7c940d16a0 2024-04-02T09:14:09.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.553+0000 7f7c99200700 1 -- 172.21.15.89:0/662109884 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c9415df50 con 0x7f7c940d16a0 2024-04-02T09:14:09.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.553+0000 7f7c8a000700 1 -- 172.21.15.89:0/662109884 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c84007310 con 0x7f7c940d16a0 2024-04-02T09:14:09.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.553+0000 7f7c99200700 1 -- 172.21.15.89:0/662109884 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9415e510 con 0x7f7c940d16a0 2024-04-02T09:14:09.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.554+0000 7f7c99200700 1 -- 172.21.15.89:0/662109884 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c940d1ca0 con 0x7f7c940d16a0 2024-04-02T09:14:09.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.555+0000 7f7c8a000700 1 -- 172.21.15.89:0/662109884 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f7c84007470 con 0x7f7c940d16a0 2024-04-02T09:14:09.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.556+0000 7f7c8a000700 1 --2- 172.21.15.89:0/662109884 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7c7406aed0 0x7f7c7406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:09.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.556+0000 7f7c91400700 1 -- 172.21.15.89:0/662109884 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7c7406aed0 msgr2=0x7f7c7406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:09.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.556+0000 7f7c8a000700 1 -- 172.21.15.89:0/662109884 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f7c840899a0 con 0x7f7c940d16a0 2024-04-02T09:14:09.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.556+0000 7f7c91400700 1 --2- 172.21.15.89:0/662109884 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7c7406aed0 0x7f7c7406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:09.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.561+0000 7f7c8a000700 1 -- 172.21.15.89:0/662109884 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7c84054140 con 0x7f7c940d16a0 2024-04-02T09:14:09.684 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:09.685+0000 7f75006c9240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:14:09.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.757+0000 7f7c91400700 1 -- 172.21.15.89:0/662109884 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7c7406aed0 msgr2=0x7f7c7406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:09.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.757+0000 7f7c91400700 1 --2- 172.21.15.89:0/662109884 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7c7406aed0 0x7f7c7406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:09.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.768+0000 7f7c99200700 1 -- 172.21.15.89:0/662109884 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7c9415e730 con 0x7f7c940d16a0 2024-04-02T09:14:09.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.769+0000 7f7c8a000700 1 -- 172.21.15.89:0/662109884 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v249) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7c84057760 con 0x7f7c940d16a0 2024-04-02T09:14:09.768 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:14:09.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.771+0000 7f7c99200700 1 -- 172.21.15.89:0/662109884 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7c7406aed0 msgr2=0x7f7c7406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:09.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.771+0000 7f7c99200700 1 --2- 172.21.15.89:0/662109884 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7c7406aed0 0x7f7c7406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.772+0000 7f7c99200700 1 -- 172.21.15.89:0/662109884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c940d16a0 msgr2=0x7f7c940b4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:09.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.772+0000 7f7c99200700 1 --2- 172.21.15.89:0/662109884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c940d16a0 0x7f7c940b4860 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f7c8400c550 tx=0x7f7c8400c860 comp rx=0 tx=0).stop 2024-04-02T09:14:09.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.772+0000 7f7c99200700 1 -- 172.21.15.89:0/662109884 shutdown_connections 2024-04-02T09:14:09.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.772+0000 7f7c99200700 1 --2- 172.21.15.89:0/662109884 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7c7406aed0 0x7f7c7406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.772+0000 7f7c99200700 1 --2- 172.21.15.89:0/662109884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c940d16a0 0x7f7c940b4860 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.772+0000 7f7c99200700 1 --2- 172.21.15.89:0/662109884 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c9415d9d0 0x7f7c94158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.772+0000 7f7c99200700 1 --2- 172.21.15.89:0/662109884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c940d2370 0x7f7c94158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.772+0000 7f7c99200700 1 -- 172.21.15.89:0/662109884 >> 172.21.15.89:0/662109884 conn(0x7f7c9408aff0 msgr2=0x7f7c9415b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:09.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.772+0000 7f7c99200700 1 -- 172.21.15.89:0/662109884 shutdown_connections 2024-04-02T09:14:09.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.772+0000 7f7c99200700 1 -- 172.21.15.89:0/662109884 wait complete. 2024-04-02T09:14:09.789 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T09:14:09.910 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:09.910+0000 7f75006c9240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:14:09.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.992+0000 7fca80800700 1 Processor -- start 2024-04-02T09:14:09.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.992+0000 7fca80800700 1 -- start start 2024-04-02T09:14:09.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.994+0000 7fca80800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fca780ce840 0x7fca78165370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:09.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.994+0000 7fca80800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca780cb310 con 0x7fca7815dfe0 2024-04-02T09:14:09.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.994+0000 7fca80800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca780cca90 con 0x7fca7815a5c0 2024-04-02T09:14:09.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.994+0000 7fca80800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca780711a0 con 0x7fca780ce840 2024-04-02T09:14:09.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.994+0000 7fca7d400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fca7815dfe0 0x7fca780ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53116/0 (socket says 172.21.15.89:53116) 2024-04-02T09:14:09.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.994+0000 7fca7d400700 1 -- 172.21.15.89:0/150725106 learned_addr learned my addr 172.21.15.89:0/150725106 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:09.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.994+0000 7fca7e800700 1 --2- 172.21.15.89:0/150725106 >> v2:172.21.15.113:3300/0 conn(0x7fca780ce840 0x7fca78165370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:09.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.994+0000 7fca7ca00700 1 -- 172.21.15.89:0/150725106 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319616475 0 0) 0x7fca780cb310 con 0x7fca7815dfe0 2024-04-02T09:14:09.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.994+0000 7fca7ca00700 1 -- 172.21.15.89:0/150725106 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca600035b0 con 0x7fca7815dfe0 2024-04-02T09:14:09.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.995+0000 7fca7e800700 1 -- 172.21.15.89:0/150725106 >> v1:172.21.15.113:6789/0 conn(0x7fca7815a5c0 legacy=0x7fca780709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:09.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.995+0000 7fca7e800700 1 -- 172.21.15.89:0/150725106 >> v1:172.21.15.89:6789/0 conn(0x7fca7815dfe0 legacy=0x7fca780ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.995+0000 7fca7e800700 1 -- 172.21.15.89:0/150725106 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca78071480 con 0x7fca780ce840 2024-04-02T09:14:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.995+0000 7fca7e800700 1 --2- 172.21.15.89:0/150725106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca780ce840 0x7fca78165370 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fca74009d90 tx=0x7fca7400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cc343d89c2abb6b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.995+0000 7fca7ca00700 1 -- 172.21.15.89:0/150725106 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7400e070 con 0x7fca780ce840 2024-04-02T09:14:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.995+0000 7fca7ca00700 1 -- 172.21.15.89:0/150725106 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7400bb20 con 0x7fca780ce840 2024-04-02T09:14:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.995+0000 7fca7ca00700 1 -- 172.21.15.89:0/150725106 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7400be90 con 0x7fca780ce840 2024-04-02T09:14:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.996+0000 7fca80800700 1 -- 172.21.15.89:0/150725106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca780ce840 msgr2=0x7fca78165370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.996+0000 7fca80800700 1 --2- 172.21.15.89:0/150725106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca780ce840 0x7fca78165370 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fca74009d90 tx=0x7fca7400b330 comp rx=0 tx=0).stop 2024-04-02T09:14:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.996+0000 7fca80800700 1 -- 172.21.15.89:0/150725106 shutdown_connections 2024-04-02T09:14:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.996+0000 7fca80800700 1 --2- 172.21.15.89:0/150725106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca780ce840 0x7fca78165370 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.996+0000 7fca80800700 1 -- 172.21.15.89:0/150725106 >> 172.21.15.89:0/150725106 conn(0x7fca780b0d70 msgr2=0x7fca780b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.996+0000 7fca80800700 1 -- 172.21.15.89:0/150725106 shutdown_connections 2024-04-02T09:14:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.996+0000 7fca80800700 1 -- 172.21.15.89:0/150725106 wait complete. 2024-04-02T09:14:09.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.999+0000 7fca80800700 1 Processor -- start 2024-04-02T09:14:09.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:09.999+0000 7fca80800700 1 -- start start 2024-04-02T09:14:10.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca80800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca780ce840 0x7fca780ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca80800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca7815a5c0 0x7fca7814f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca80800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca7815dfe0 0x7fca780afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca80800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca780ae460 con 0x7fca780ce840 2024-04-02T09:14:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca80800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca780ca590 con 0x7fca7815a5c0 2024-04-02T09:14:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca80800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca780adda0 con 0x7fca7815dfe0 2024-04-02T09:14:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca7de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca780ce840 0x7fca780ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca7de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca780ce840 0x7fca780ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49776/0 (socket says 172.21.15.89:49776) 2024-04-02T09:14:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca7de00700 1 -- 172.21.15.89:0/1607130813 learned_addr learned my addr 172.21.15.89:0/1607130813 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca7d400700 1 --2- 172.21.15.89:0/1607130813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca7815a5c0 0x7fca7814f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca7e800700 1 --2- 172.21.15.89:0/1607130813 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca7815dfe0 0x7fca780afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca7de00700 1 -- 172.21.15.89:0/1607130813 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca7815dfe0 msgr2=0x7fca780afd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca7de00700 1 --2- 172.21.15.89:0/1607130813 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca7815dfe0 0x7fca780afd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca7de00700 1 -- 172.21.15.89:0/1607130813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca7815a5c0 msgr2=0x7fca7814f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:10.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca7de00700 1 --2- 172.21.15.89:0/1607130813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca7815a5c0 0x7fca7814f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.001+0000 7fca7de00700 1 -- 172.21.15.89:0/1607130813 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca74009a70 con 0x7fca780ce840 2024-04-02T09:14:10.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.002+0000 7fca7de00700 1 --2- 172.21.15.89:0/1607130813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca780ce840 0x7fca780ac050 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00c490 tx=0x7fca6c00c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:10.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.002+0000 7fca72000700 1 -- 172.21.15.89:0/1607130813 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c00cea0 con 0x7fca780ce840 2024-04-02T09:14:10.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.002+0000 7fca72000700 1 -- 172.21.15.89:0/1607130813 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6c0052f0 con 0x7fca780ce840 2024-04-02T09:14:10.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.002+0000 7fca80800700 1 -- 172.21.15.89:0/1607130813 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca780ae080 con 0x7fca780ce840 2024-04-02T09:14:10.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.002+0000 7fca72000700 1 -- 172.21.15.89:0/1607130813 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c006860 con 0x7fca780ce840 2024-04-02T09:14:10.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.002+0000 7fca80800700 1 -- 172.21.15.89:0/1607130813 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca780ada90 con 0x7fca780ce840 2024-04-02T09:14:10.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.003+0000 7fca80800700 1 -- 172.21.15.89:0/1607130813 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca78067720 con 0x7fca780ce840 2024-04-02T09:14:10.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.004+0000 7fca72000700 1 -- 172.21.15.89:0/1607130813 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7fca6c003680 con 0x7fca780ce840 2024-04-02T09:14:10.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.005+0000 7fca72000700 1 --2- 172.21.15.89:0/1607130813 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca5c06aed0 0x7fca5c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.005+0000 7fca7d400700 1 -- 172.21.15.89:0/1607130813 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca5c06aed0 msgr2=0x7fca5c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:10.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.005+0000 7fca72000700 1 -- 172.21.15.89:0/1607130813 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7fca6c088f70 con 0x7fca780ce840 2024-04-02T09:14:10.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.005+0000 7fca7d400700 1 --2- 172.21.15.89:0/1607130813 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca5c06aed0 0x7fca5c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:10.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.010+0000 7fca72000700 1 -- 172.21.15.89:0/1607130813 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fca6c0bc580 con 0x7fca780ce840 2024-04-02T09:14:10.013 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:10.015+0000 7f90b8167240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:14:10.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.206+0000 7fca7d400700 1 -- 172.21.15.89:0/1607130813 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca5c06aed0 msgr2=0x7fca5c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:10.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.206+0000 7fca7d400700 1 --2- 172.21.15.89:0/1607130813 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca5c06aed0 0x7fca5c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:10.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.214+0000 7fca80800700 1 -- 172.21.15.89:0/1607130813 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fca780b1ad0 con 0x7fca780ce840 2024-04-02T09:14:10.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.215+0000 7fca72000700 1 -- 172.21.15.89:0/1607130813 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v249) v1 ==== 100+0+11 (secure 0 0 0) 0x7fca6c053770 con 0x7fca780ce840 2024-04-02T09:14:10.214 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:14:10.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.217+0000 7fca80800700 1 -- 172.21.15.89:0/1607130813 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca5c06aed0 msgr2=0x7fca5c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:10.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.217+0000 7fca80800700 1 --2- 172.21.15.89:0/1607130813 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca5c06aed0 0x7fca5c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.217+0000 7fca80800700 1 -- 172.21.15.89:0/1607130813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca780ce840 msgr2=0x7fca780ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:10.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.217+0000 7fca80800700 1 --2- 172.21.15.89:0/1607130813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca780ce840 0x7fca780ac050 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00c490 tx=0x7fca6c00c7a0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.217+0000 7fca80800700 1 -- 172.21.15.89:0/1607130813 shutdown_connections 2024-04-02T09:14:10.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.217+0000 7fca80800700 1 --2- 172.21.15.89:0/1607130813 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca7815dfe0 0x7fca780afd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.217+0000 7fca80800700 1 --2- 172.21.15.89:0/1607130813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca780ce840 0x7fca780ac050 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.217+0000 7fca80800700 1 --2- 172.21.15.89:0/1607130813 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fca5c06aed0 0x7fca5c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.217+0000 7fca80800700 1 --2- 172.21.15.89:0/1607130813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca7815a5c0 0x7fca7814f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.217+0000 7fca80800700 1 -- 172.21.15.89:0/1607130813 >> 172.21.15.89:0/1607130813 conn(0x7fca780b0d70 msgr2=0x7fca780cdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:10.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.217+0000 7fca80800700 1 -- 172.21.15.89:0/1607130813 shutdown_connections 2024-04-02T09:14:10.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.217+0000 7fca80800700 1 -- 172.21.15.89:0/1607130813 wait complete. 2024-04-02T09:14:10.220 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:10.221+0000 7f90b8167240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:14:10.235 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-02T09:14:10.416 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:10.417+0000 7f75006c9240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:14:10.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.435+0000 7f8d61000700 1 Processor -- start 2024-04-02T09:14:10.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.435+0000 7f8d61000700 1 -- start start 2024-04-02T09:14:10.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.436+0000 7f8d61000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8d5c0cef40 0x7f8d5c099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.437+0000 7f8d61000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8d5c0cf3f0 0x7f8d5c099fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.437+0000 7f8d61000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c0ccdd0 con 0x7f8d5c15f1e0 2024-04-02T09:14:10.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.437+0000 7f8d61000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5c0cd260 con 0x7f8d5c0cef40 2024-04-02T09:14:10.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.437+0000 7f8d61000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5c1600c0 con 0x7f8d5c0cf3f0 2024-04-02T09:14:10.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.437+0000 7f8d59400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8d5c0cef40 0x7f8d5c099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:10.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.437+0000 7f8d59400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8d5c0cef40 0x7f8d5c099a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49790/0 (socket says 172.21.15.89:49790) 2024-04-02T09:14:10.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.437+0000 7f8d59400700 1 -- 172.21.15.89:0/2097740482 learned_addr learned my addr 172.21.15.89:0/2097740482 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:10.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.437+0000 7f8d5a800700 1 --2- 172.21.15.89:0/2097740482 >> v2:172.21.15.113:3300/0 conn(0x7f8d5c0cf3f0 0x7f8d5c099fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:10.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.437+0000 7f8d59400700 1 -- 172.21.15.89:0/2097740482 >> v2:172.21.15.113:3300/0 conn(0x7f8d5c0cf3f0 msgr2=0x7f8d5c099fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:10.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.437+0000 7f8d59400700 1 --2- 172.21.15.89:0/2097740482 >> v2:172.21.15.113:3300/0 conn(0x7f8d5c0cf3f0 0x7f8d5c099fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.437+0000 7f8d59400700 1 -- 172.21.15.89:0/2097740482 >> v1:172.21.15.113:6789/0 conn(0x7f8d5c15f1e0 legacy=0x7f8d5c15f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:10.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.437+0000 7f8d59400700 1 -- 172.21.15.89:0/2097740482 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c15cae0 con 0x7f8d5c0cef40 2024-04-02T09:14:10.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.438+0000 7f8d59400700 1 --2- 172.21.15.89:0/2097740482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d5c0cef40 0x7f8d5c099a60 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f8d480098f0 tx=0x7f8d48009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=904e6e2f591087cc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:10.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.438+0000 7f8d58a00700 1 -- 172.21.15.89:0/2097740482 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d48011830 con 0x7f8d5c0cef40 2024-04-02T09:14:10.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.438+0000 7f8d58a00700 1 -- 172.21.15.89:0/2097740482 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d48011990 con 0x7f8d5c0cef40 2024-04-02T09:14:10.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.438+0000 7f8d58a00700 1 -- 172.21.15.89:0/2097740482 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d48011d20 con 0x7f8d5c0cef40 2024-04-02T09:14:10.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.438+0000 7f8d61000700 1 -- 172.21.15.89:0/2097740482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d5c0cef40 msgr2=0x7f8d5c099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:10.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.438+0000 7f8d61000700 1 --2- 172.21.15.89:0/2097740482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d5c0cef40 0x7f8d5c099a60 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f8d480098f0 tx=0x7f8d48009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:10.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.439+0000 7f8d61000700 1 -- 172.21.15.89:0/2097740482 shutdown_connections 2024-04-02T09:14:10.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.439+0000 7f8d61000700 1 --2- 172.21.15.89:0/2097740482 >> v2:172.21.15.113:3300/0 conn(0x7f8d5c0cf3f0 0x7f8d5c099fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.439+0000 7f8d61000700 1 --2- 172.21.15.89:0/2097740482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d5c0cef40 0x7f8d5c099a60 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.439+0000 7f8d61000700 1 -- 172.21.15.89:0/2097740482 >> 172.21.15.89:0/2097740482 conn(0x7f8d5c0d1370 msgr2=0x7f8d5c0d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:10.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.439+0000 7f8d61000700 1 -- 172.21.15.89:0/2097740482 shutdown_connections 2024-04-02T09:14:10.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.439+0000 7f8d61000700 1 -- 172.21.15.89:0/2097740482 wait complete. 2024-04-02T09:14:10.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.440+0000 7f8d61000700 1 Processor -- start 2024-04-02T09:14:10.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.441+0000 7f8d61000700 1 -- start start 2024-04-02T09:14:10.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.441+0000 7f8d61000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d5c0cef40 0x7f8d5c15ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d61000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d5c0cf3f0 0x7f8d5c06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d61000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d5c15f1e0 0x7f8d5c0c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d61000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c06c080 con 0x7f8d5c15f1e0 2024-04-02T09:14:10.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d61000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c06cec0 con 0x7f8d5c0cef40 2024-04-02T09:14:10.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d61000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c06c2b0 con 0x7f8d5c0cf3f0 2024-04-02T09:14:10.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d5a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d5c15f1e0 0x7f8d5c0c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:10.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d5a800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d5c15f1e0 0x7f8d5c0c62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49800/0 (socket says 172.21.15.89:49800) 2024-04-02T09:14:10.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d5a800700 1 -- 172.21.15.89:0/1727673327 learned_addr learned my addr 172.21.15.89:0/1727673327 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:10.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d59400700 1 --2- 172.21.15.89:0/1727673327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d5c0cf3f0 0x7f8d5c06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:10.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d59e00700 1 --2- 172.21.15.89:0/1727673327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d5c0cef40 0x7f8d5c15ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:10.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d5a800700 1 -- 172.21.15.89:0/1727673327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d5c0cf3f0 msgr2=0x7f8d5c06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:10.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d5a800700 1 --2- 172.21.15.89:0/1727673327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d5c0cf3f0 0x7f8d5c06bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d5a800700 1 -- 172.21.15.89:0/1727673327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d5c0cef40 msgr2=0x7f8d5c15ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:10.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d5a800700 1 --2- 172.21.15.89:0/1727673327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d5c0cef40 0x7f8d5c15ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.442+0000 7f8d5a800700 1 -- 172.21.15.89:0/1727673327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d48009550 con 0x7f8d5c15f1e0 2024-04-02T09:14:10.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.443+0000 7f8d5a800700 1 --2- 172.21.15.89:0/1727673327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d5c15f1e0 0x7f8d5c0c62b0 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f8d5400efc0 tx=0x7f8d5400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:10.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.443+0000 7f8d52000700 1 -- 172.21.15.89:0/1727673327 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54009ea0 con 0x7f8d5c15f1e0 2024-04-02T09:14:10.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.443+0000 7f8d52000700 1 -- 172.21.15.89:0/1727673327 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5400f040 con 0x7f8d5c15f1e0 2024-04-02T09:14:10.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.443+0000 7f8d61000700 1 -- 172.21.15.89:0/1727673327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c06c590 con 0x7f8d5c15f1e0 2024-04-02T09:14:10.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.443+0000 7f8d52000700 1 -- 172.21.15.89:0/1727673327 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5400ced0 con 0x7f8d5c15f1e0 2024-04-02T09:14:10.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.443+0000 7f8d61000700 1 -- 172.21.15.89:0/1727673327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5c06cb50 con 0x7f8d5c15f1e0 2024-04-02T09:14:10.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.444+0000 7f8d61000700 1 -- 172.21.15.89:0/1727673327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5c06d050 con 0x7f8d5c15f1e0 2024-04-02T09:14:10.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.445+0000 7f8d52000700 1 -- 172.21.15.89:0/1727673327 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f8d54014480 con 0x7f8d5c15f1e0 2024-04-02T09:14:10.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.446+0000 7f8d52000700 1 --2- 172.21.15.89:0/1727673327 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8d3c06aed0 0x7f8d3c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.447+0000 7f8d52000700 1 -- 172.21.15.89:0/1727673327 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f8d5408a690 con 0x7f8d5c15f1e0 2024-04-02T09:14:10.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.447+0000 7f8d59e00700 1 -- 172.21.15.89:0/1727673327 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8d3c06aed0 msgr2=0x7f8d3c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:10.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.447+0000 7f8d59e00700 1 --2- 172.21.15.89:0/1727673327 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8d3c06aed0 0x7f8d3c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:10.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.450+0000 7f8d52000700 1 -- 172.21.15.89:0/1727673327 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8d54054dc0 con 0x7f8d5c15f1e0 2024-04-02T09:14:10.633 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:10.634+0000 7f75006c9240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:14:10.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.648+0000 7f8d59e00700 1 -- 172.21.15.89:0/1727673327 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8d3c06aed0 msgr2=0x7f8d3c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:10.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.648+0000 7f8d59e00700 1 --2- 172.21.15.89:0/1727673327 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8d3c06aed0 0x7f8d3c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:10.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.656+0000 7f8d61000700 1 -- 172.21.15.89:0/1727673327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8d5c067720 con 0x7f8d5c15f1e0 2024-04-02T09:14:10.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.656+0000 7f8d52000700 1 -- 172.21.15.89:0/1727673327 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v249) v1 ==== 107+0+11 (secure 0 0 0) 0x7f8d540583e0 con 0x7f8d5c15f1e0 2024-04-02T09:14:10.656 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:14:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.659+0000 7f8d61000700 1 -- 172.21.15.89:0/1727673327 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8d3c06aed0 msgr2=0x7f8d3c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.659+0000 7f8d61000700 1 --2- 172.21.15.89:0/1727673327 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8d3c06aed0 0x7f8d3c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.659+0000 7f8d61000700 1 -- 172.21.15.89:0/1727673327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d5c15f1e0 msgr2=0x7f8d5c0c62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:10.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.659+0000 7f8d61000700 1 --2- 172.21.15.89:0/1727673327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d5c15f1e0 0x7f8d5c0c62b0 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f8d5400efc0 tx=0x7f8d5400c3a0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.659+0000 7f8d61000700 1 -- 172.21.15.89:0/1727673327 shutdown_connections 2024-04-02T09:14:10.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.659+0000 7f8d61000700 1 --2- 172.21.15.89:0/1727673327 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8d3c06aed0 0x7f8d3c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.659+0000 7f8d61000700 1 --2- 172.21.15.89:0/1727673327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d5c0cef40 0x7f8d5c15ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.659+0000 7f8d61000700 1 --2- 172.21.15.89:0/1727673327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d5c15f1e0 0x7f8d5c0c62b0 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.659+0000 7f8d61000700 1 --2- 172.21.15.89:0/1727673327 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d5c0cf3f0 0x7f8d5c06bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.659+0000 7f8d61000700 1 -- 172.21.15.89:0/1727673327 >> 172.21.15.89:0/1727673327 conn(0x7f8d5c0d1370 msgr2=0x7f8d5c06e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:10.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.659+0000 7f8d61000700 1 -- 172.21.15.89:0/1727673327 shutdown_connections 2024-04-02T09:14:10.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.659+0000 7f8d61000700 1 -- 172.21.15.89:0/1727673327 wait complete. 2024-04-02T09:14:10.660 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:10.662+0000 7f90b8167240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:14:10.677 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-02T09:14:10.771 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:10.773+0000 7f90b8167240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:14:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.882+0000 7ff570e00700 1 Processor -- start 2024-04-02T09:14:10.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.882+0000 7ff570e00700 1 -- start start 2024-04-02T09:14:10.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.883+0000 7ff570e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff56c160d70 0x7ff56c0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.883+0000 7ff570e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff56c15ef70 0x7ff56c15e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.884+0000 7ff570e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56c161150 con 0x7ff56c06a790 2024-04-02T09:14:10.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.884+0000 7ff570e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56c0b4cd0 con 0x7ff56c15ef70 2024-04-02T09:14:10.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.884+0000 7ff570e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c06dba0 con 0x7ff56c160d70 2024-04-02T09:14:10.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.884+0000 7ff569400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff56c15ef70 0x7ff56c15e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:10.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.884+0000 7ff569400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff56c15ef70 0x7ff56c15e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49814/0 (socket says 172.21.15.89:49814) 2024-04-02T09:14:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.884+0000 7ff569400700 1 -- 172.21.15.89:0/3511229155 learned_addr learned my addr 172.21.15.89:0/3511229155 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.884+0000 7ff569e00700 1 --2- 172.21.15.89:0/3511229155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff56c160d70 0x7ff56c0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.884+0000 7ff569400700 1 -- 172.21.15.89:0/3511229155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff56c160d70 msgr2=0x7ff56c0b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.884+0000 7ff569400700 1 --2- 172.21.15.89:0/3511229155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff56c160d70 0x7ff56c0b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.884+0000 7ff569400700 1 -- 172.21.15.89:0/3511229155 >> v1:172.21.15.89:6789/0 conn(0x7ff56c06a790 legacy=0x7ff56c1613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.884+0000 7ff569400700 1 -- 172.21.15.89:0/3511229155 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c15ed30 con 0x7ff56c15ef70 2024-04-02T09:14:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.885+0000 7ff569400700 1 --2- 172.21.15.89:0/3511229155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c15ef70 0x7ff56c15e7f0 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7ff5580098f0 tx=0x7ff558009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21fe527af80b7f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.885+0000 7ff568a00700 1 -- 172.21.15.89:0/3511229155 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55800b8a0 con 0x7ff56c15ef70 2024-04-02T09:14:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.885+0000 7ff568a00700 1 -- 172.21.15.89:0/3511229155 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55800ba00 con 0x7ff56c15ef70 2024-04-02T09:14:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.885+0000 7ff568a00700 1 -- 172.21.15.89:0/3511229155 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55800bd90 con 0x7ff56c15ef70 2024-04-02T09:14:10.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.886+0000 7ff570e00700 1 -- 172.21.15.89:0/3511229155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c15ef70 msgr2=0x7ff56c15e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:10.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.886+0000 7ff570e00700 1 --2- 172.21.15.89:0/3511229155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c15ef70 0x7ff56c15e7f0 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7ff5580098f0 tx=0x7ff558009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:10.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.886+0000 7ff570e00700 1 -- 172.21.15.89:0/3511229155 shutdown_connections 2024-04-02T09:14:10.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.886+0000 7ff570e00700 1 --2- 172.21.15.89:0/3511229155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff56c160d70 0x7ff56c0b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.886+0000 7ff570e00700 1 --2- 172.21.15.89:0/3511229155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c15ef70 0x7ff56c15e7f0 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.886+0000 7ff570e00700 1 -- 172.21.15.89:0/3511229155 >> 172.21.15.89:0/3511229155 conn(0x7ff56c0b5720 msgr2=0x7ff56c0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:10.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.886+0000 7ff570e00700 1 -- 172.21.15.89:0/3511229155 shutdown_connections 2024-04-02T09:14:10.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.886+0000 7ff570e00700 1 -- 172.21.15.89:0/3511229155 wait complete. 2024-04-02T09:14:10.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.887+0000 7ff570e00700 1 Processor -- start 2024-04-02T09:14:10.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.888+0000 7ff570e00700 1 -- start start 2024-04-02T09:14:10.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.888+0000 7ff570e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff56c06a790 0x7ff56c1f2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.888+0000 7ff570e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c15ef70 0x7ff56c09d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff570e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff56c160d70 0x7ff56c0caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff570e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0c9580 con 0x7ff56c15ef70 2024-04-02T09:14:10.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff570e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0cb930 con 0x7ff56c160d70 2024-04-02T09:14:10.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff570e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0c97d0 con 0x7ff56c06a790 2024-04-02T09:14:10.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff569400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c15ef70 0x7ff56c09d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff569e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff56c06a790 0x7ff56c1f2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff569400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c15ef70 0x7ff56c09d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49824/0 (socket says 172.21.15.89:49824) 2024-04-02T09:14:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff569400700 1 -- 172.21.15.89:0/4074002832 learned_addr learned my addr 172.21.15.89:0/4074002832 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff56a800700 1 --2- 172.21.15.89:0/4074002832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff56c160d70 0x7ff56c0caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff569400700 1 -- 172.21.15.89:0/4074002832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff56c06a790 msgr2=0x7ff56c1f2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff569400700 1 --2- 172.21.15.89:0/4074002832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff56c06a790 0x7ff56c1f2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff569400700 1 -- 172.21.15.89:0/4074002832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff56c160d70 msgr2=0x7ff56c0caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff569400700 1 --2- 172.21.15.89:0/4074002832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff56c160d70 0x7ff56c0caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff569400700 1 -- 172.21.15.89:0/4074002832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff558009550 con 0x7ff56c15ef70 2024-04-02T09:14:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.889+0000 7ff569e00700 1 --2- 172.21.15.89:0/4074002832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff56c06a790 0x7ff56c1f2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:14:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.890+0000 7ff569400700 1 --2- 172.21.15.89:0/4074002832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c15ef70 0x7ff56c09d9b0 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7ff558009c40 tx=0x7ff558004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.890+0000 7ff562000700 1 -- 172.21.15.89:0/4074002832 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff558004300 con 0x7ff56c15ef70 2024-04-02T09:14:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.890+0000 7ff562000700 1 -- 172.21.15.89:0/4074002832 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55801f070 con 0x7ff56c15ef70 2024-04-02T09:14:10.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.890+0000 7ff570e00700 1 -- 172.21.15.89:0/4074002832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56c0c9ab0 con 0x7ff56c15ef70 2024-04-02T09:14:10.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.890+0000 7ff562000700 1 -- 172.21.15.89:0/4074002832 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5580036a0 con 0x7ff56c15ef70 2024-04-02T09:14:10.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.890+0000 7ff570e00700 1 -- 172.21.15.89:0/4074002832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56c15e1d0 con 0x7ff56c15ef70 2024-04-02T09:14:10.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.891+0000 7ff570e00700 1 -- 172.21.15.89:0/4074002832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56c0c9fd0 con 0x7ff56c15ef70 2024-04-02T09:14:10.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.892+0000 7ff562000700 1 -- 172.21.15.89:0/4074002832 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 163) v1 ==== 89342+0+0 (secure 0 0 0) 0x7ff558021030 con 0x7ff56c15ef70 2024-04-02T09:14:10.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.893+0000 7ff562000700 1 --2- 172.21.15.89:0/4074002832 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff54c06aed0 0x7ff54c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:10.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.893+0000 7ff562000700 1 -- 172.21.15.89:0/4074002832 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7451+0+0 (secure 0 0 0) 0x7ff55808b0a0 con 0x7ff56c15ef70 2024-04-02T09:14:10.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.893+0000 7ff569e00700 1 -- 172.21.15.89:0/4074002832 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff54c06aed0 msgr2=0x7ff54c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:10.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.893+0000 7ff569e00700 1 --2- 172.21.15.89:0/4074002832 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff54c06aed0 0x7ff54c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:10.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:10.897+0000 7ff562000700 1 -- 172.21.15.89:0/4074002832 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff56c0c9fd0 con 0x7ff56c15ef70 2024-04-02T09:14:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.094+0000 7ff569e00700 1 -- 172.21.15.89:0/4074002832 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff54c06aed0 msgr2=0x7ff54c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.094+0000 7ff569e00700 1 --2- 172.21.15.89:0/4074002832 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff54c06aed0 0x7ff54c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:11.101 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.101+0000 7f75006c9240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:14:11.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.106+0000 7ff570e00700 1 -- 172.21.15.89:0/4074002832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7ff56c0c9fd0 con 0x7ff56c15ef70 2024-04-02T09:14:11.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.107+0000 7ff562000700 1 -- 172.21.15.89:0/4074002832 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v249) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff56c0c9fd0 con 0x7ff56c15ef70 2024-04-02T09:14:11.106 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:14:11.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.110+0000 7ff570e00700 1 -- 172.21.15.89:0/4074002832 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff54c06aed0 msgr2=0x7ff54c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:11.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.110+0000 7ff570e00700 1 --2- 172.21.15.89:0/4074002832 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff54c06aed0 0x7ff54c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.110+0000 7ff570e00700 1 -- 172.21.15.89:0/4074002832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c15ef70 msgr2=0x7ff56c09d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:11.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.110+0000 7ff570e00700 1 --2- 172.21.15.89:0/4074002832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c15ef70 0x7ff56c09d9b0 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7ff558009c40 tx=0x7ff558004750 comp rx=0 tx=0).stop 2024-04-02T09:14:11.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.110+0000 7ff570e00700 1 -- 172.21.15.89:0/4074002832 shutdown_connections 2024-04-02T09:14:11.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.110+0000 7ff570e00700 1 --2- 172.21.15.89:0/4074002832 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff54c06aed0 0x7ff54c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.110+0000 7ff570e00700 1 --2- 172.21.15.89:0/4074002832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff56c160d70 0x7ff56c0caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.110+0000 7ff570e00700 1 --2- 172.21.15.89:0/4074002832 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff56c06a790 0x7ff56c1f2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.110+0000 7ff570e00700 1 --2- 172.21.15.89:0/4074002832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff56c15ef70 0x7ff56c09d9b0 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.110+0000 7ff570e00700 1 -- 172.21.15.89:0/4074002832 >> 172.21.15.89:0/4074002832 conn(0x7ff56c0b5720 msgr2=0x7ff56c146080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:11.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.110+0000 7ff570e00700 1 -- 172.21.15.89:0/4074002832 shutdown_connections 2024-04-02T09:14:11.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.110+0000 7ff570e00700 1 -- 172.21.15.89:0/4074002832 wait complete. 2024-04-02T09:14:11.127 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:14:11.225 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.226+0000 7f75006c9240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:14:11.315 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.316+0000 7f746fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.315 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.316+0000 7f746fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.316 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.316+0000 7f746fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.316 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.316+0000 7f746fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.316 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.316+0000 7f746fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.316 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.316+0000 7f7473800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.316 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.316+0000 7f7473800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.316 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.316+0000 7f7473800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.316 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.316+0000 7f7473800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.316 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.316+0000 7f7473800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.329 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.330+0000 7f7473800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.330 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.330+0000 7f7473800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.330 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.330+0000 7f7473800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.330 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.330+0000 7f7473800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.330 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.330+0000 7f7473800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.330 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.331+0000 7f746fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.331 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.331+0000 7f746fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.331 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.331+0000 7f746fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.331 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.331+0000 7f746fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.331 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:11.331+0000 7f746fc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:11.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.344+0000 7fb373c00700 1 Processor -- start 2024-04-02T09:14:11.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.345+0000 7fb373c00700 1 -- start start 2024-04-02T09:14:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb373c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb36c15d640 0x7fb36c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb373c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb36c1619d0 0x7fb36c0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb373c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36c0d30d0 con 0x7fb36c0ceec0 2024-04-02T09:14:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb373c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36c0b6840 con 0x7fb36c1619d0 2024-04-02T09:14:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb373c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c0d2880 con 0x7fb36c15d640 2024-04-02T09:14:11.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb36be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb36c0ceec0 0x7fb36c1612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53138/0 (socket says 172.21.15.89:53138) 2024-04-02T09:14:11.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb36be00700 1 -- 172.21.15.89:0/3610757542 learned_addr learned my addr 172.21.15.89:0/3610757542 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:11.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb371200700 1 --2- 172.21.15.89:0/3610757542 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb36c15d640 0x7fb36c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:11.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb371c00700 1 --2- 172.21.15.89:0/3610757542 >> v2:172.21.15.113:3300/0 conn(0x7fb36c1619d0 0x7fb36c0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb36b400700 1 -- 172.21.15.89:0/3610757542 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094008497 0 0) 0x7fb36c0d30d0 con 0x7fb36c0ceec0 2024-04-02T09:14:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb36b400700 1 -- 172.21.15.89:0/3610757542 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3540035b0 con 0x7fb36c0ceec0 2024-04-02T09:14:11.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb371200700 1 -- 172.21.15.89:0/3610757542 >> v2:172.21.15.113:3300/0 conn(0x7fb36c1619d0 msgr2=0x7fb36c0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:11.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb371200700 1 --2- 172.21.15.89:0/3610757542 >> v2:172.21.15.113:3300/0 conn(0x7fb36c1619d0 0x7fb36c0b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb371200700 1 -- 172.21.15.89:0/3610757542 >> v1:172.21.15.89:6789/0 conn(0x7fb36c0ceec0 legacy=0x7fb36c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:11.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.346+0000 7fb371200700 1 -- 172.21.15.89:0/3610757542 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c0d2b30 con 0x7fb36c15d640 2024-04-02T09:14:11.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.347+0000 7fb371200700 1 --2- 172.21.15.89:0/3610757542 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb36c15d640 0x7fb36c0d2340 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb360009840 tx=0x7fb360009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=842d940687ee7081 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:11.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.347+0000 7fb36b400700 1 -- 172.21.15.89:0/3610757542 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb360011830 con 0x7fb36c15d640 2024-04-02T09:14:11.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.347+0000 7fb36b400700 1 -- 172.21.15.89:0/3610757542 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb360011990 con 0x7fb36c15d640 2024-04-02T09:14:11.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.347+0000 7fb36b400700 1 -- 172.21.15.89:0/3610757542 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb360011d20 con 0x7fb36c15d640 2024-04-02T09:14:11.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.348+0000 7fb373c00700 1 -- 172.21.15.89:0/3610757542 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb36c15d640 msgr2=0x7fb36c0d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:11.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.348+0000 7fb373c00700 1 --2- 172.21.15.89:0/3610757542 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb36c15d640 0x7fb36c0d2340 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb360009840 tx=0x7fb360009b50 comp rx=0 tx=0).stop 2024-04-02T09:14:11.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.348+0000 7fb373c00700 1 -- 172.21.15.89:0/3610757542 shutdown_connections 2024-04-02T09:14:11.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.348+0000 7fb373c00700 1 --2- 172.21.15.89:0/3610757542 >> v2:172.21.15.113:3300/0 conn(0x7fb36c1619d0 0x7fb36c0b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.348+0000 7fb373c00700 1 --2- 172.21.15.89:0/3610757542 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb36c15d640 0x7fb36c0d2340 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.348+0000 7fb373c00700 1 -- 172.21.15.89:0/3610757542 >> 172.21.15.89:0/3610757542 conn(0x7fb36c08aff0 msgr2=0x7fb36c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:11.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.348+0000 7fb373c00700 1 -- 172.21.15.89:0/3610757542 shutdown_connections 2024-04-02T09:14:11.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.348+0000 7fb373c00700 1 -- 172.21.15.89:0/3610757542 wait complete. 2024-04-02T09:14:11.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.349+0000 7fb373c00700 1 Processor -- start 2024-04-02T09:14:11.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.350+0000 7fb373c00700 1 -- start start 2024-04-02T09:14:11.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.350+0000 7fb373c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb36c0ceec0 0x7fb36c15a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:11.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.350+0000 7fb373c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb36c15d640 0x7fb36c0ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:11.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.350+0000 7fb373c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb36c1619d0 0x7fb36c0a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:11.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.351+0000 7fb373c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c1f6f00 con 0x7fb36c1619d0 2024-04-02T09:14:11.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.351+0000 7fb373c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c09d230 con 0x7fb36c15d640 2024-04-02T09:14:11.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.351+0000 7fb373c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c09d480 con 0x7fb36c0ceec0 2024-04-02T09:14:11.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.351+0000 7fb371c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb36c1619d0 0x7fb36c0a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:11.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.351+0000 7fb371c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb36c1619d0 0x7fb36c0a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49932/0 (socket says 172.21.15.89:49932) 2024-04-02T09:14:11.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.351+0000 7fb371c00700 1 -- 172.21.15.89:0/3391730696 learned_addr learned my addr 172.21.15.89:0/3391730696 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:11.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.351+0000 7fb371200700 1 --2- 172.21.15.89:0/3391730696 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb36c0ceec0 0x7fb36c15a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:11.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.351+0000 7fb36be00700 1 --2- 172.21.15.89:0/3391730696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb36c15d640 0x7fb36c0ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:11.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.351+0000 7fb371c00700 1 -- 172.21.15.89:0/3391730696 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb36c0ceec0 msgr2=0x7fb36c15a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:11.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.351+0000 7fb371c00700 1 --2- 172.21.15.89:0/3391730696 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb36c0ceec0 0x7fb36c15a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.351+0000 7fb371c00700 1 -- 172.21.15.89:0/3391730696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb36c15d640 msgr2=0x7fb36c0ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:11.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.351+0000 7fb371c00700 1 --2- 172.21.15.89:0/3391730696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb36c15d640 0x7fb36c0ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.351+0000 7fb371c00700 1 -- 172.21.15.89:0/3391730696 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3600094a0 con 0x7fb36c1619d0 2024-04-02T09:14:11.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.351+0000 7fb371c00700 1 --2- 172.21.15.89:0/3391730696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb36c1619d0 0x7fb36c0a0780 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7fb36400ef90 tx=0x7fb36400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:11.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.352+0000 7fb368c00700 1 -- 172.21.15.89:0/3391730696 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36400cc90 con 0x7fb36c1619d0 2024-04-02T09:14:11.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.352+0000 7fb368c00700 1 -- 172.21.15.89:0/3391730696 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36400f040 con 0x7fb36c1619d0 2024-04-02T09:14:11.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.352+0000 7fb373c00700 1 -- 172.21.15.89:0/3391730696 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36c09d760 con 0x7fb36c1619d0 2024-04-02T09:14:11.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.352+0000 7fb368c00700 1 -- 172.21.15.89:0/3391730696 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364014480 con 0x7fb36c1619d0 2024-04-02T09:14:11.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.352+0000 7fb373c00700 1 -- 172.21.15.89:0/3391730696 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c09dd20 con 0x7fb36c1619d0 2024-04-02T09:14:11.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.353+0000 7fb373c00700 1 -- 172.21.15.89:0/3391730696 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36c09df40 con 0x7fb36c1619d0 2024-04-02T09:14:11.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.354+0000 7fb368c00700 1 -- 172.21.15.89:0/3391730696 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 164) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fb3640145e0 con 0x7fb36c1619d0 2024-04-02T09:14:11.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.355+0000 7fb368c00700 1 -- 172.21.15.89:0/3391730696 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7795+0+0 (secure 0 0 0) 0x7fb36408a180 con 0x7fb36c1619d0 2024-04-02T09:14:11.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.359+0000 7fb368c00700 1 -- 172.21.15.89:0/3391730696 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb364054e90 con 0x7fb36c1619d0 2024-04-02T09:14:11.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.605+0000 7fb373c00700 1 -- 172.21.15.89:0/3391730696 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb36c067720 con 0x7fb36c1619d0 2024-04-02T09:14:11.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.606+0000 7fb368c00700 1 -- 172.21.15.89:0/3391730696 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5968 (secure 0 0 0) 0x7fb364018070 con 0x7fb36c1619d0 2024-04-02T09:14:11.605 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:11.606 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":125,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9518,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3094781297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3094781297},{"type":"v1","addr":"172.21.15.89:6836","nonce":3094781297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":10978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1799351491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1799351491},{"type":"v1","addr":"172.21.15.113:6834","nonce":1799351491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:28.357996+0000","modified":"2024-04-02T09:13:31.369015+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11964":{"gid":11964,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6835/703852199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":703852199},{"type":"v1","addr":"172.21.15.113:6835","nonce":703852199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:41.933360+0000","modified":"2024-04-02T09:13:45.439462+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11958},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11958":{"gid":11958,"name":"a","rank":0,"incarnation":121,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6837/3112580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3112580185},{"type":"v1","addr":"172.21.15.89:6837","nonce":3112580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-02T09:14:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.608+0000 7fb373c00700 1 -- 172.21.15.89:0/3391730696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb36c1619d0 msgr2=0x7fb36c0a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.608+0000 7fb373c00700 1 --2- 172.21.15.89:0/3391730696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb36c1619d0 0x7fb36c0a0780 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7fb36400ef90 tx=0x7fb36400c3a0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.608+0000 7fb373c00700 1 -- 172.21.15.89:0/3391730696 shutdown_connections 2024-04-02T09:14:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.608+0000 7fb373c00700 1 --2- 172.21.15.89:0/3391730696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb36c1619d0 0x7fb36c0a0780 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.608+0000 7fb373c00700 1 --2- 172.21.15.89:0/3391730696 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb36c0ceec0 0x7fb36c15a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.608+0000 7fb373c00700 1 --2- 172.21.15.89:0/3391730696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb36c15d640 0x7fb36c0ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.608+0000 7fb373c00700 1 -- 172.21.15.89:0/3391730696 >> 172.21.15.89:0/3391730696 conn(0x7fb36c08aff0 msgr2=0x7fb36c0b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.609+0000 7fb373c00700 1 -- 172.21.15.89:0/3391730696 shutdown_connections 2024-04-02T09:14:11.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.609+0000 7fb373c00700 1 -- 172.21.15.89:0/3391730696 wait complete. 2024-04-02T09:14:11.609 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 125 2024-04-02T09:14:11.627 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:14:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.819+0000 7f83c0a00700 1 Processor -- start 2024-04-02T09:14:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.819+0000 7f83c0a00700 1 -- start start 2024-04-02T09:14:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.821+0000 7f83c0a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83bc0d18b0 0x7f83bc0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.821+0000 7f83c0a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f83bc161590 0x7f83bc161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.821+0000 7f83c0a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83bc0ceda0 con 0x7f83bc160f10 2024-04-02T09:14:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.821+0000 7f83c0a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83bc0d1d30 con 0x7f83bc161590 2024-04-02T09:14:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.821+0000 7f83c0a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc161f60 con 0x7f83bc0d18b0 2024-04-02T09:14:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.821+0000 7f83ba800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f83bc161590 0x7f83bc161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.821+0000 7f83ba800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f83bc161590 0x7f83bc161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49940/0 (socket says 172.21.15.89:49940) 2024-04-02T09:14:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.821+0000 7f83ba800700 1 -- 172.21.15.89:0/297778094 learned_addr learned my addr 172.21.15.89:0/297778094 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.821+0000 7f83b9400700 1 --2- 172.21.15.89:0/297778094 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83bc0d18b0 0x7f83bc0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.821+0000 7f83ba800700 1 -- 172.21.15.89:0/297778094 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83bc0d18b0 msgr2=0x7f83bc0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.821+0000 7f83ba800700 1 --2- 172.21.15.89:0/297778094 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83bc0d18b0 0x7f83bc0cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.821+0000 7f83ba800700 1 -- 172.21.15.89:0/297778094 >> v1:172.21.15.113:6789/0 conn(0x7f83bc160f10 legacy=0x7f83bc0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.821+0000 7f83ba800700 1 -- 172.21.15.89:0/297778094 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc1612f0 con 0x7f83bc161590 2024-04-02T09:14:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.822+0000 7f83ba800700 1 --2- 172.21.15.89:0/297778094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc161590 0x7f83bc161a20 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009e80 tx=0x7f83b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6fb9e1906adeec server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.822+0000 7f83b8a00700 1 -- 172.21.15.89:0/297778094 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b400e070 con 0x7f83bc161590 2024-04-02T09:14:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.822+0000 7f83b8a00700 1 -- 172.21.15.89:0/297778094 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b400b980 con 0x7f83bc161590 2024-04-02T09:14:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.822+0000 7f83b8a00700 1 -- 172.21.15.89:0/297778094 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b400bd30 con 0x7f83bc161590 2024-04-02T09:14:11.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.822+0000 7f83c0a00700 1 -- 172.21.15.89:0/297778094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc161590 msgr2=0x7f83bc161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:11.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.822+0000 7f83c0a00700 1 --2- 172.21.15.89:0/297778094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc161590 0x7f83bc161a20 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009e80 tx=0x7f83b400b330 comp rx=0 tx=0).stop 2024-04-02T09:14:11.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.822+0000 7f83c0a00700 1 -- 172.21.15.89:0/297778094 shutdown_connections 2024-04-02T09:14:11.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.822+0000 7f83c0a00700 1 --2- 172.21.15.89:0/297778094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc161590 0x7f83bc161a20 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.822+0000 7f83c0a00700 1 --2- 172.21.15.89:0/297778094 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83bc0d18b0 0x7f83bc0cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.822+0000 7f83c0a00700 1 -- 172.21.15.89:0/297778094 >> 172.21.15.89:0/297778094 conn(0x7f83bc08aff0 msgr2=0x7f83bc0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.823+0000 7f83c0a00700 1 -- 172.21.15.89:0/297778094 shutdown_connections 2024-04-02T09:14:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.823+0000 7f83c0a00700 1 -- 172.21.15.89:0/297778094 wait complete. 2024-04-02T09:14:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.824+0000 7f83c0a00700 1 Processor -- start 2024-04-02T09:14:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.824+0000 7f83c0a00700 1 -- start start 2024-04-02T09:14:11.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.825+0000 7f83c0a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc0d18b0 0x7f83bc158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:11.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.825+0000 7f83c0a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83bc160f10 0x7f83bc0b75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:11.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.825+0000 7f83c0a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83bc161590 0x7f83bc156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:11.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.825+0000 7f83c0a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0b85b0 con 0x7f83bc0d18b0 2024-04-02T09:14:11.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.825+0000 7f83c0a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0b7ff0 con 0x7f83bc160f10 2024-04-02T09:14:11.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.825+0000 7f83c0a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0b82d0 con 0x7f83bc161590 2024-04-02T09:14:11.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.825+0000 7f83b9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc0d18b0 0x7f83bc158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:11.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.826+0000 7f83b9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc0d18b0 0x7f83bc158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49944/0 (socket says 172.21.15.89:49944) 2024-04-02T09:14:11.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.826+0000 7f83b9e00700 1 -- 172.21.15.89:0/1844785073 learned_addr learned my addr 172.21.15.89:0/1844785073 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:11.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.826+0000 7f83b9400700 1 --2- 172.21.15.89:0/1844785073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83bc160f10 0x7f83bc0b75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:11.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.826+0000 7f83ba800700 1 --2- 172.21.15.89:0/1844785073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83bc161590 0x7f83bc156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:11.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.826+0000 7f83b9e00700 1 -- 172.21.15.89:0/1844785073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83bc161590 msgr2=0x7f83bc156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:11.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.826+0000 7f83b9e00700 1 --2- 172.21.15.89:0/1844785073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83bc161590 0x7f83bc156f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.826+0000 7f83b9e00700 1 -- 172.21.15.89:0/1844785073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83bc160f10 msgr2=0x7f83bc0b75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:11.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.826+0000 7f83b9e00700 1 --2- 172.21.15.89:0/1844785073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83bc160f10 0x7f83bc0b75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:11.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.826+0000 7f83b9e00700 1 -- 172.21.15.89:0/1844785073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b4009b30 con 0x7f83bc0d18b0 2024-04-02T09:14:11.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.826+0000 7f83b9e00700 1 --2- 172.21.15.89:0/1844785073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc0d18b0 0x7f83bc158750 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00c450 tx=0x7f83ac00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:11.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.826+0000 7f83b2000700 1 -- 172.21.15.89:0/1844785073 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac0052d0 con 0x7f83bc0d18b0 2024-04-02T09:14:11.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.826+0000 7f83b2000700 1 -- 172.21.15.89:0/1844785073 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac005430 con 0x7f83bc0d18b0 2024-04-02T09:14:11.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.827+0000 7f83c0a00700 1 -- 172.21.15.89:0/1844785073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83bc156710 con 0x7f83bc0d18b0 2024-04-02T09:14:11.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.827+0000 7f83b2000700 1 -- 172.21.15.89:0/1844785073 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac0038c0 con 0x7f83bc0d18b0 2024-04-02T09:14:11.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.827+0000 7f83c0a00700 1 -- 172.21.15.89:0/1844785073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83bc15d1f0 con 0x7f83bc0d18b0 2024-04-02T09:14:11.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.828+0000 7f83c0a00700 1 -- 172.21.15.89:0/1844785073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83bc15d380 con 0x7f83bc0d18b0 2024-04-02T09:14:11.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.829+0000 7f83b2000700 1 -- 172.21.15.89:0/1844785073 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 164) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f83ac003a20 con 0x7f83bc0d18b0 2024-04-02T09:14:11.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.830+0000 7f83b2000700 1 -- 172.21.15.89:0/1844785073 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7795+0+0 (secure 0 0 0) 0x7f83ac0888f0 con 0x7f83bc0d18b0 2024-04-02T09:14:11.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:11.834+0000 7f83b2000700 1 -- 172.21.15.89:0/1844785073 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f83ac0536f0 con 0x7f83bc0d18b0 2024-04-02T09:14:12.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.037+0000 7f83c0a00700 1 -- 172.21.15.89:0/1844785073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83bc156c30 con 0x7f83bc0d18b0 2024-04-02T09:14:12.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.039+0000 7f83b2000700 1 -- 172.21.15.89:0/1844785073 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v250) v1 ==== 74+0+21515 (secure 0 0 0) 0x7f83bc156c30 con 0x7f83bc0d18b0 2024-04-02T09:14:12.039 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:12.039 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":250,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:14:11.230841+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:13:26.654647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-02T09:13:27.342285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:13:40.048039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:13:40.907706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/3750432255":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/2974789087":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/1231626446":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/2398338283":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/4088911225":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3553222249":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3687373115":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/1998199546":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/492795889":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/1337615482":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3750385348":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/2067457213":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/1275405725":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/3355557522":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/4204293224":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/176419408":"2024-04-03T09:14:02.049396+0000","172.21.15.89:6833/108855":"2024-04-03T09:14:02.049396+0000","172.21.15.89:6832/108855":"2024-04-03T09:14:02.049396+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:14:12.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.041+0000 7f83c0a00700 1 -- 172.21.15.89:0/1844785073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc0d18b0 msgr2=0x7f83bc158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:12.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.041+0000 7f83c0a00700 1 --2- 172.21.15.89:0/1844785073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc0d18b0 0x7f83bc158750 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00c450 tx=0x7f83ac00c760 comp rx=0 tx=0).stop 2024-04-02T09:14:12.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.041+0000 7f83c0a00700 1 -- 172.21.15.89:0/1844785073 shutdown_connections 2024-04-02T09:14:12.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.041+0000 7f83c0a00700 1 --2- 172.21.15.89:0/1844785073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f83bc161590 0x7f83bc156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.041+0000 7f83c0a00700 1 --2- 172.21.15.89:0/1844785073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc0d18b0 0x7f83bc158750 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.041+0000 7f83c0a00700 1 --2- 172.21.15.89:0/1844785073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83bc160f10 0x7f83bc0b75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.041+0000 7f83c0a00700 1 -- 172.21.15.89:0/1844785073 >> 172.21.15.89:0/1844785073 conn(0x7f83bc08aff0 msgr2=0x7f83bc06bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:12.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.042+0000 7f83c0a00700 1 -- 172.21.15.89:0/1844785073 shutdown_connections 2024-04-02T09:14:12.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.042+0000 7f83c0a00700 1 -- 172.21.15.89:0/1844785073 wait complete. 2024-04-02T09:14:12.059 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T09:14:12.060 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:14:12.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.252+0000 7f01f0a00700 1 Processor -- start 2024-04-02T09:14:12.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.252+0000 7f01f0a00700 1 -- start start 2024-04-02T09:14:12.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.254+0000 7f01f0a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f01ec0d2010 0x7f01ec0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:12.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.254+0000 7f01f0a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec0cebd0 con 0x7f01ec0d1b90 2024-04-02T09:14:12.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.254+0000 7f01f0a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec0b3280 con 0x7f01ec160ed0 2024-04-02T09:14:12.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.254+0000 7f01f0a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ec1612b0 con 0x7f01ec0d2010 2024-04-02T09:14:12.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.254+0000 7f01e9e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f01ec160ed0 0x7f01ec0d3cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:57384/0 (socket says 172.21.15.89:57384) 2024-04-02T09:14:12.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.254+0000 7f01e9e00700 1 -- 172.21.15.89:0/2881686669 learned_addr learned my addr 172.21.15.89:0/2881686669 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:12.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.255+0000 7f01ea800700 1 --2- 172.21.15.89:0/2881686669 >> v2:172.21.15.89:3300/0 conn(0x7f01ec0d2010 0x7f01ec0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:12.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.257+0000 7f01ea800700 1 -- 172.21.15.89:0/2881686669 >> v1:172.21.15.113:6789/0 conn(0x7f01ec160ed0 legacy=0x7f01ec0d3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:12.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.257+0000 7f01ea800700 1 -- 172.21.15.89:0/2881686669 >> v1:172.21.15.89:6789/0 conn(0x7f01ec0d1b90 legacy=0x7f01ec0b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:12.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.257+0000 7f01ea800700 1 -- 172.21.15.89:0/2881686669 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec0b2cf0 con 0x7f01ec0d2010 2024-04-02T09:14:12.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.258+0000 7f01ea800700 1 --2- 172.21.15.89:0/2881686669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01ec0d2010 0x7f01ec0b3e20 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009d90 tx=0x7f01e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6408fbcf83aafe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:12.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.259+0000 7f01e8a00700 1 -- 172.21.15.89:0/2881686669 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e400e070 con 0x7f01ec0d2010 2024-04-02T09:14:12.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.260+0000 7f01e8a00700 1 -- 172.21.15.89:0/2881686669 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e400bbd0 con 0x7f01ec0d2010 2024-04-02T09:14:12.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.260+0000 7f01e8a00700 1 -- 172.21.15.89:0/2881686669 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e4012430 con 0x7f01ec0d2010 2024-04-02T09:14:12.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.260+0000 7f01f0a00700 1 -- 172.21.15.89:0/2881686669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01ec0d2010 msgr2=0x7f01ec0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.260+0000 7f01f0a00700 1 --2- 172.21.15.89:0/2881686669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01ec0d2010 0x7f01ec0b3e20 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009d90 tx=0x7f01e400b330 comp rx=0 tx=0).stop 2024-04-02T09:14:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.261+0000 7f01f0a00700 1 -- 172.21.15.89:0/2881686669 shutdown_connections 2024-04-02T09:14:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.261+0000 7f01f0a00700 1 --2- 172.21.15.89:0/2881686669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01ec0d2010 0x7f01ec0b3e20 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.261+0000 7f01f0a00700 1 -- 172.21.15.89:0/2881686669 >> 172.21.15.89:0/2881686669 conn(0x7f01ec08aff0 msgr2=0x7f01ec0d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:12.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.261+0000 7f01f0a00700 1 -- 172.21.15.89:0/2881686669 shutdown_connections 2024-04-02T09:14:12.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.261+0000 7f01f0a00700 1 -- 172.21.15.89:0/2881686669 wait complete. 2024-04-02T09:14:12.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.264+0000 7f01f0a00700 1 Processor -- start 2024-04-02T09:14:12.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.265+0000 7f01f0a00700 1 -- start start 2024-04-02T09:14:12.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.266+0000 7f01f0a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec0d1b90 0x7f01ec09ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.266+0000 7f01f0a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01ec0d2010 0x7f01ec1f6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.266+0000 7f01f0a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec160ed0 0x7f01ec0b8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.266+0000 7f01f0a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec09efd0 con 0x7f01ec0d2010 2024-04-02T09:14:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.266+0000 7f01f0a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec14d6f0 con 0x7f01ec160ed0 2024-04-02T09:14:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.266+0000 7f01f0a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec14d9d0 con 0x7f01ec0d1b90 2024-04-02T09:14:12.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.267+0000 7f01e9e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec0d1b90 0x7f01ec09ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.267+0000 7f01ea800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec160ed0 0x7f01ec0b8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.267+0000 7f01e9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01ec0d2010 0x7f01ec1f6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.267+0000 7f01e9e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec0d1b90 0x7f01ec09ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:55954/0 (socket says 172.21.15.89:55954) 2024-04-02T09:14:12.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.267+0000 7f01e9e00700 1 -- 172.21.15.89:0/642992436 learned_addr learned my addr 172.21.15.89:0/642992436 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.267+0000 7f01e9400700 1 -- 172.21.15.89:0/642992436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec0d1b90 msgr2=0x7f01ec09ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.267+0000 7f01e9400700 1 --2- 172.21.15.89:0/642992436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec0d1b90 0x7f01ec09ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.267+0000 7f01e9400700 1 -- 172.21.15.89:0/642992436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec160ed0 msgr2=0x7f01ec0b8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.267+0000 7f01e9400700 1 --2- 172.21.15.89:0/642992436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec160ed0 0x7f01ec0b8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.267+0000 7f01e9400700 1 -- 172.21.15.89:0/642992436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e4009a70 con 0x7f01ec0d2010 2024-04-02T09:14:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.268+0000 7f01e9400700 1 --2- 172.21.15.89:0/642992436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01ec0d2010 0x7f01ec1f6860 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f01d8000fd0 tx=0x7f01d800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.268+0000 7f01e2000700 1 -- 172.21.15.89:0/642992436 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d80054e0 con 0x7f01ec0d2010 2024-04-02T09:14:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.268+0000 7f01e2000700 1 -- 172.21.15.89:0/642992436 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d800ce50 con 0x7f01ec0d2010 2024-04-02T09:14:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.268+0000 7f01f0a00700 1 -- 172.21.15.89:0/642992436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ec14dcb0 con 0x7f01ec0d2010 2024-04-02T09:14:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.268+0000 7f01e2000700 1 -- 172.21.15.89:0/642992436 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d80038a0 con 0x7f01ec0d2010 2024-04-02T09:14:12.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.268+0000 7f01f0a00700 1 -- 172.21.15.89:0/642992436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ec14e270 con 0x7f01ec0d2010 2024-04-02T09:14:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.269+0000 7f01f0a00700 1 -- 172.21.15.89:0/642992436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ec0d2480 con 0x7f01ec0d2010 2024-04-02T09:14:12.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.270+0000 7f01e2000700 1 -- 172.21.15.89:0/642992436 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 165) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f01d8003a00 con 0x7f01ec0d2010 2024-04-02T09:14:12.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.271+0000 7f01e2000700 1 --2- 172.21.15.89:0/642992436 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f01cc06ab10 0x7f01cc06cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:12.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.271+0000 7f01e2000700 1 -- 172.21.15.89:0/642992436 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7795+0+0 (secure 0 0 0) 0x7f01d8088bf0 con 0x7f01ec0d2010 2024-04-02T09:14:12.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.272+0000 7f01e9e00700 1 --2- 172.21.15.89:0/642992436 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f01cc06ab10 0x7f01cc06cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:12.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.272+0000 7f01e9e00700 1 --2- 172.21.15.89:0/642992436 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f01cc06ab10 0x7f01cc06cfd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f01dc005910 tx=0x7f01dc009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:12.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.276+0000 7f01e2000700 1 -- 172.21.15.89:0/642992436 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f01d8053780 con 0x7f01ec0d2010 2024-04-02T09:14:12.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.521+0000 7f01f0a00700 1 -- 172.21.15.89:0/642992436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01ec160a10 con 0x7f01ec0d2010 2024-04-02T09:14:12.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.522+0000 7f01e2000700 1 -- 172.21.15.89:0/642992436 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5968 (secure 0 0 0) 0x7f01d8017030 con 0x7f01ec0d2010 2024-04-02T09:14:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:12.521 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":125,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9518,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3094781297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3094781297},{"type":"v1","addr":"172.21.15.89:6836","nonce":3094781297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":10978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1799351491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1799351491},{"type":"v1","addr":"172.21.15.113:6834","nonce":1799351491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:28.357996+0000","modified":"2024-04-02T09:13:31.369015+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11964":{"gid":11964,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6835/703852199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":703852199},{"type":"v1","addr":"172.21.15.113:6835","nonce":703852199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:41.933360+0000","modified":"2024-04-02T09:13:45.439462+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11958},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11958":{"gid":11958,"name":"a","rank":0,"incarnation":121,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6837/3112580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3112580185},{"type":"v1","addr":"172.21.15.89:6837","nonce":3112580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-02T09:14:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.524+0000 7f01f0a00700 1 -- 172.21.15.89:0/642992436 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f01cc06ab10 msgr2=0x7f01cc06cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.524+0000 7f01f0a00700 1 --2- 172.21.15.89:0/642992436 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f01cc06ab10 0x7f01cc06cfd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f01dc005910 tx=0x7f01dc009040 comp rx=0 tx=0).stop 2024-04-02T09:14:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.524+0000 7f01f0a00700 1 -- 172.21.15.89:0/642992436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01ec0d2010 msgr2=0x7f01ec1f6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.524+0000 7f01f0a00700 1 --2- 172.21.15.89:0/642992436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01ec0d2010 0x7f01ec1f6860 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f01d8000fd0 tx=0x7f01d800c7b0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.524+0000 7f01f0a00700 1 -- 172.21.15.89:0/642992436 shutdown_connections 2024-04-02T09:14:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.524+0000 7f01f0a00700 1 --2- 172.21.15.89:0/642992436 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f01cc06ab10 0x7f01cc06cfd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.524+0000 7f01f0a00700 1 --2- 172.21.15.89:0/642992436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec160ed0 0x7f01ec0b8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.524+0000 7f01f0a00700 1 --2- 172.21.15.89:0/642992436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec0d1b90 0x7f01ec09ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.524+0000 7f01f0a00700 1 --2- 172.21.15.89:0/642992436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01ec0d2010 0x7f01ec1f6860 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.524+0000 7f01f0a00700 1 -- 172.21.15.89:0/642992436 >> 172.21.15.89:0/642992436 conn(0x7f01ec08aff0 msgr2=0x7f01ec0b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.524+0000 7f01f0a00700 1 -- 172.21.15.89:0/642992436 shutdown_connections 2024-04-02T09:14:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.525+0000 7f01f0a00700 1 -- 172.21.15.89:0/642992436 wait complete. 2024-04-02T09:14:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 125 2024-04-02T09:14:12.541 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:14:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.734+0000 7fbfe3600700 1 Processor -- start 2024-04-02T09:14:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.734+0000 7fbfe3600700 1 -- start start 2024-04-02T09:14:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.735+0000 7fbfe3600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfdc15d760 0x7fbfdc0d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.735+0000 7fbfe3600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbfdc15db40 0x7fbfdc0cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.735+0000 7fbfe3600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc0cf5f0 con 0x7fbfdc0cf1f0 2024-04-02T09:14:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.735+0000 7fbfe3600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfdc0d1fb0 con 0x7fbfdc15db40 2024-04-02T09:14:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.735+0000 7fbfe3600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0d1330 con 0x7fbfdc15d760 2024-04-02T09:14:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.735+0000 7fbfdbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbfdc15db40 0x7fbfdc0cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.735+0000 7fbfdbe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbfdc15db40 0x7fbfdc0cecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49988/0 (socket says 172.21.15.89:49988) 2024-04-02T09:14:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.735+0000 7fbfdbe00700 1 -- 172.21.15.89:0/1163503137 learned_addr learned my addr 172.21.15.89:0/1163503137 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.736+0000 7fbfe0c00700 1 --2- 172.21.15.89:0/1163503137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfdc15d760 0x7fbfdc0d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.736+0000 7fbfdbe00700 1 -- 172.21.15.89:0/1163503137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfdc15d760 msgr2=0x7fbfdc0d23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.736+0000 7fbfdbe00700 1 --2- 172.21.15.89:0/1163503137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfdc15d760 0x7fbfdc0d23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.736+0000 7fbfdbe00700 1 -- 172.21.15.89:0/1163503137 >> v1:172.21.15.113:6789/0 conn(0x7fbfdc0cf1f0 legacy=0x7fbfdc0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.736+0000 7fbfdbe00700 1 -- 172.21.15.89:0/1163503137 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc0d1610 con 0x7fbfdc15db40 2024-04-02T09:14:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.736+0000 7fbfdbe00700 1 --2- 172.21.15.89:0/1163503137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfdc15db40 0x7fbfdc0cecb0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7fbfc80098f0 tx=0x7fbfc8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40103199ce112b2e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.736+0000 7fbfdb400700 1 -- 172.21.15.89:0/1163503137 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc800b8a0 con 0x7fbfdc15db40 2024-04-02T09:14:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.736+0000 7fbfdb400700 1 -- 172.21.15.89:0/1163503137 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc800ba00 con 0x7fbfdc15db40 2024-04-02T09:14:12.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.736+0000 7fbfdb400700 1 -- 172.21.15.89:0/1163503137 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc800bd90 con 0x7fbfdc15db40 2024-04-02T09:14:12.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.737+0000 7fbfe3600700 1 -- 172.21.15.89:0/1163503137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfdc15db40 msgr2=0x7fbfdc0cecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:12.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.737+0000 7fbfe3600700 1 --2- 172.21.15.89:0/1163503137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfdc15db40 0x7fbfdc0cecb0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7fbfc80098f0 tx=0x7fbfc8009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:12.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.737+0000 7fbfe3600700 1 -- 172.21.15.89:0/1163503137 shutdown_connections 2024-04-02T09:14:12.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.737+0000 7fbfe3600700 1 --2- 172.21.15.89:0/1163503137 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfdc15d760 0x7fbfdc0d23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.737+0000 7fbfe3600700 1 --2- 172.21.15.89:0/1163503137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfdc15db40 0x7fbfdc0cecb0 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.737+0000 7fbfe3600700 1 -- 172.21.15.89:0/1163503137 >> 172.21.15.89:0/1163503137 conn(0x7fbfdc08aff0 msgr2=0x7fbfdc0d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:12.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.737+0000 7fbfe3600700 1 -- 172.21.15.89:0/1163503137 shutdown_connections 2024-04-02T09:14:12.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.737+0000 7fbfe3600700 1 -- 172.21.15.89:0/1163503137 wait complete. 2024-04-02T09:14:12.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.739+0000 7fbfe3600700 1 Processor -- start 2024-04-02T09:14:12.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.740+0000 7fbfe3600700 1 -- start start 2024-04-02T09:14:12.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe3600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfdc0cf1f0 0x7fbfdc14fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:12.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe3600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfdc15d760 0x7fbfdc0b8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:12.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe3600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfdc15db40 0x7fbfdc0b8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:12.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe3600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc14fff0 con 0x7fbfdc15db40 2024-04-02T09:14:12.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe3600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0b7050 con 0x7fbfdc0cf1f0 2024-04-02T09:14:12.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe3600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0b7330 con 0x7fbfdc15d760 2024-04-02T09:14:12.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe1600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfdc15db40 0x7fbfdc0b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:12.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe1600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfdc15db40 0x7fbfdc0b8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50000/0 (socket says 172.21.15.89:50000) 2024-04-02T09:14:12.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe1600700 1 -- 172.21.15.89:0/277760637 learned_addr learned my addr 172.21.15.89:0/277760637 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:12.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfdbe00700 1 --2- 172.21.15.89:0/277760637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfdc15d760 0x7fbfdc0b8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:12.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe0c00700 1 --2- 172.21.15.89:0/277760637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfdc0cf1f0 0x7fbfdc14fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:12.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe1600700 1 -- 172.21.15.89:0/277760637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfdc15d760 msgr2=0x7fbfdc0b8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:12.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe1600700 1 --2- 172.21.15.89:0/277760637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfdc15d760 0x7fbfdc0b8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe1600700 1 -- 172.21.15.89:0/277760637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfdc0cf1f0 msgr2=0x7fbfdc14fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:12.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe1600700 1 --2- 172.21.15.89:0/277760637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfdc0cf1f0 0x7fbfdc14fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.741+0000 7fbfe1600700 1 -- 172.21.15.89:0/277760637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc8009550 con 0x7fbfdc15db40 2024-04-02T09:14:12.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.742+0000 7fbfe1600700 1 --2- 172.21.15.89:0/277760637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfdc15db40 0x7fbfdc0b8970 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400cc50 tx=0x7fbfd400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:12.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.742+0000 7fbfd8c00700 1 -- 172.21.15.89:0/277760637 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4003c00 con 0x7fbfdc15db40 2024-04-02T09:14:12.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.742+0000 7fbfd8c00700 1 -- 172.21.15.89:0/277760637 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd4003d60 con 0x7fbfdc15db40 2024-04-02T09:14:12.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.742+0000 7fbfe3600700 1 -- 172.21.15.89:0/277760637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfdc0b7610 con 0x7fbfdc15db40 2024-04-02T09:14:12.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.742+0000 7fbfd8c00700 1 -- 172.21.15.89:0/277760637 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd400aab0 con 0x7fbfdc15db40 2024-04-02T09:14:12.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.742+0000 7fbfe3600700 1 -- 172.21.15.89:0/277760637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfdc0b7bd0 con 0x7fbfdc15db40 2024-04-02T09:14:12.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.743+0000 7fbfe3600700 1 -- 172.21.15.89:0/277760637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfdc067720 con 0x7fbfdc15db40 2024-04-02T09:14:12.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.744+0000 7fbfd8c00700 1 -- 172.21.15.89:0/277760637 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 165) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fbfd400ac10 con 0x7fbfdc15db40 2024-04-02T09:14:12.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.745+0000 7fbfd8c00700 1 --2- 172.21.15.89:0/277760637 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbfbc06ab10 0x7fbfbc06cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:12.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.745+0000 7fbfd8c00700 1 -- 172.21.15.89:0/277760637 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7795+0+0 (secure 0 0 0) 0x7fbfd4089840 con 0x7fbfdc15db40 2024-04-02T09:14:12.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.746+0000 7fbfe0c00700 1 --2- 172.21.15.89:0/277760637 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbfbc06ab10 0x7fbfbc06cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:12.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.746+0000 7fbfe0c00700 1 --2- 172.21.15.89:0/277760637 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbfbc06ab10 0x7fbfbc06cfd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0006b20 tx=0x7fbfd0008040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:12.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.750+0000 7fbfd8c00700 1 -- 172.21.15.89:0/277760637 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbfd4054460 con 0x7fbfdc15db40 2024-04-02T09:14:12.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.993+0000 7fbfe3600700 1 -- 172.21.15.89:0/277760637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfdc0b9460 con 0x7fbfdc15db40 2024-04-02T09:14:12.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.993+0000 7fbfd8c00700 1 -- 172.21.15.89:0/277760637 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5968 (secure 0 0 0) 0x7fbfd4057a80 con 0x7fbfdc15db40 2024-04-02T09:14:12.992 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:12.993 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":125,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9518,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3094781297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3094781297},{"type":"v1","addr":"172.21.15.89:6836","nonce":3094781297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":10978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1799351491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1799351491},{"type":"v1","addr":"172.21.15.113:6834","nonce":1799351491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:28.357996+0000","modified":"2024-04-02T09:13:31.369015+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11964":{"gid":11964,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.113:6835/703852199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":703852199},{"type":"v1","addr":"172.21.15.113:6835","nonce":703852199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:41.933360+0000","modified":"2024-04-02T09:13:45.439462+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11958},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11958":{"gid":11958,"name":"a","rank":0,"incarnation":121,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6837/3112580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3112580185},{"type":"v1","addr":"172.21.15.89:6837","nonce":3112580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-02T09:14:12.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.996+0000 7fbfe3600700 1 -- 172.21.15.89:0/277760637 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbfbc06ab10 msgr2=0x7fbfbc06cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:12.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.996+0000 7fbfe3600700 1 --2- 172.21.15.89:0/277760637 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbfbc06ab10 0x7fbfbc06cfd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0006b20 tx=0x7fbfd0008040 comp rx=0 tx=0).stop 2024-04-02T09:14:12.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.996+0000 7fbfe3600700 1 -- 172.21.15.89:0/277760637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfdc15db40 msgr2=0x7fbfdc0b8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:12.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.996+0000 7fbfe3600700 1 --2- 172.21.15.89:0/277760637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfdc15db40 0x7fbfdc0b8970 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400cc50 tx=0x7fbfd400cf60 comp rx=0 tx=0).stop 2024-04-02T09:14:12.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.996+0000 7fbfe3600700 1 -- 172.21.15.89:0/277760637 shutdown_connections 2024-04-02T09:14:12.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.996+0000 7fbfe3600700 1 --2- 172.21.15.89:0/277760637 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbfbc06ab10 0x7fbfbc06cfd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.996+0000 7fbfe3600700 1 --2- 172.21.15.89:0/277760637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfdc0cf1f0 0x7fbfdc14fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.996+0000 7fbfe3600700 1 --2- 172.21.15.89:0/277760637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfdc15db40 0x7fbfdc0b8970 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.996+0000 7fbfe3600700 1 --2- 172.21.15.89:0/277760637 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfdc15d760 0x7fbfdc0b8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:12.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.996+0000 7fbfe3600700 1 -- 172.21.15.89:0/277760637 >> 172.21.15.89:0/277760637 conn(0x7fbfdc08aff0 msgr2=0x7fbfdc0b3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:12.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.996+0000 7fbfe3600700 1 -- 172.21.15.89:0/277760637 shutdown_connections 2024-04-02T09:14:12.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:12.996+0000 7fbfe3600700 1 -- 172.21.15.89:0/277760637 wait complete. 2024-04-02T09:14:12.997 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:14:12.997 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 125 2024-04-02T09:14:13.001 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:14:13.015 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:14:13.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.249+0000 7efcd2800700 1 Processor -- start 2024-04-02T09:14:13.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.249+0000 7efcd2800700 1 -- start start 2024-04-02T09:14:13.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.250+0000 7efcd2800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efccc0cb3f0 0x7efccc0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:13.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.251+0000 7efcd2800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efccc0ac600 0x7efccc06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:13.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.251+0000 7efcd2800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efccc15fd50 con 0x7efccc15b5d0 2024-04-02T09:14:13.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.251+0000 7efcd2800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efccc15b9b0 con 0x7efccc0cb3f0 2024-04-02T09:14:13.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.251+0000 7efcd2800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efccc0aca50 con 0x7efccc0ac600 2024-04-02T09:14:13.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.251+0000 7efccbe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efccc0ac600 0x7efccc06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:13.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.251+0000 7efcc3e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efccc0cb3f0 0x7efccc0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:13.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.251+0000 7efcc3e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efccc0cb3f0 0x7efccc0ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:48892/0 (socket says 172.21.15.89:48892) 2024-04-02T09:14:13.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.251+0000 7efcc3e00700 1 -- 172.21.15.89:0/3999056710 learned_addr learned my addr 172.21.15.89:0/3999056710 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.251+0000 7efccbe00700 1 -- 172.21.15.89:0/3999056710 >> v2:172.21.15.113:3300/0 conn(0x7efccc0cb3f0 msgr2=0x7efccc0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.251+0000 7efccbe00700 1 --2- 172.21.15.89:0/3999056710 >> v2:172.21.15.113:3300/0 conn(0x7efccc0cb3f0 0x7efccc0ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.251+0000 7efccbe00700 1 -- 172.21.15.89:0/3999056710 >> v1:172.21.15.113:6789/0 conn(0x7efccc15b5d0 legacy=0x7efccc0ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.251+0000 7efccbe00700 1 -- 172.21.15.89:0/3999056710 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efccc15f140 con 0x7efccc0ac600 2024-04-02T09:14:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.251+0000 7efcc3e00700 1 --2- 172.21.15.89:0/3999056710 >> v2:172.21.15.113:3300/0 conn(0x7efccc0cb3f0 0x7efccc0ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:14:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.252+0000 7efccbe00700 1 --2- 172.21.15.89:0/3999056710 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efccc0ac600 0x7efccc06c9d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7efcc4009d90 tx=0x7efcc400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16d39d9862d0f0a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.252+0000 7efccaa00700 1 -- 172.21.15.89:0/3999056710 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc400e070 con 0x7efccc0ac600 2024-04-02T09:14:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.252+0000 7efccaa00700 1 -- 172.21.15.89:0/3999056710 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc400bb20 con 0x7efccc0ac600 2024-04-02T09:14:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.252+0000 7efccaa00700 1 -- 172.21.15.89:0/3999056710 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4012430 con 0x7efccc0ac600 2024-04-02T09:14:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.253+0000 7efcd2800700 1 -- 172.21.15.89:0/3999056710 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efccc0ac600 msgr2=0x7efccc06c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.253+0000 7efcd2800700 1 --2- 172.21.15.89:0/3999056710 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efccc0ac600 0x7efccc06c9d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7efcc4009d90 tx=0x7efcc400b330 comp rx=0 tx=0).stop 2024-04-02T09:14:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.253+0000 7efcd2800700 1 -- 172.21.15.89:0/3999056710 shutdown_connections 2024-04-02T09:14:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.253+0000 7efcd2800700 1 --2- 172.21.15.89:0/3999056710 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efccc0ac600 0x7efccc06c9d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.253+0000 7efcd2800700 1 --2- 172.21.15.89:0/3999056710 >> v2:172.21.15.113:3300/0 conn(0x7efccc0cb3f0 0x7efccc0ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.253+0000 7efcd2800700 1 -- 172.21.15.89:0/3999056710 >> 172.21.15.89:0/3999056710 conn(0x7efccc0cf190 msgr2=0x7efccc0cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.253+0000 7efcd2800700 1 -- 172.21.15.89:0/3999056710 shutdown_connections 2024-04-02T09:14:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.253+0000 7efcd2800700 1 -- 172.21.15.89:0/3999056710 wait complete. 2024-04-02T09:14:13.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.255+0000 7efcd2800700 1 Processor -- start 2024-04-02T09:14:13.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.255+0000 7efcd2800700 1 -- start start 2024-04-02T09:14:13.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.256+0000 7efcd2800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efccc0ac600 0x7efccc152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:13.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.256+0000 7efcd2800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc15b5d0 0x7efccc0b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.256+0000 7efcd2800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efccc0b3b30 0x7efccc0c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.256+0000 7efcd2800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efccc0b3fc0 con 0x7efccc15b5d0 2024-04-02T09:14:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.256+0000 7efcd2800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efccc158c20 con 0x7efccc0b3b30 2024-04-02T09:14:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.256+0000 7efcd2800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efccc0ca8f0 con 0x7efccc0ac600 2024-04-02T09:14:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.256+0000 7efcc3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc15b5d0 0x7efccc0b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.256+0000 7efcc3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc15b5d0 0x7efccc0b35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50006/0 (socket says 172.21.15.89:50006) 2024-04-02T09:14:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.256+0000 7efcc3e00700 1 -- 172.21.15.89:0/3387398074 learned_addr learned my addr 172.21.15.89:0/3387398074 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.256+0000 7efccb400700 1 --2- 172.21.15.89:0/3387398074 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efccc0ac600 0x7efccc152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.256+0000 7efccbe00700 1 --2- 172.21.15.89:0/3387398074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efccc0b3b30 0x7efccc0c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.257+0000 7efcc3e00700 1 -- 172.21.15.89:0/3387398074 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efccc0ac600 msgr2=0x7efccc152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.257+0000 7efcc3e00700 1 --2- 172.21.15.89:0/3387398074 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efccc0ac600 0x7efccc152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.257+0000 7efcc3e00700 1 -- 172.21.15.89:0/3387398074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efccc0b3b30 msgr2=0x7efccc0c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.257+0000 7efcc3e00700 1 --2- 172.21.15.89:0/3387398074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efccc0b3b30 0x7efccc0c95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.257+0000 7efcc3e00700 1 -- 172.21.15.89:0/3387398074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb4009550 con 0x7efccc15b5d0 2024-04-02T09:14:13.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.257+0000 7efcc3e00700 1 --2- 172.21.15.89:0/3387398074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc15b5d0 0x7efccc0b35f0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7efcb400ec10 tx=0x7efcb400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:13.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.257+0000 7efcc3400700 1 -- 172.21.15.89:0/3387398074 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb400c850 con 0x7efccc15b5d0 2024-04-02T09:14:13.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.257+0000 7efcc3400700 1 -- 172.21.15.89:0/3387398074 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb400c9b0 con 0x7efccc15b5d0 2024-04-02T09:14:13.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.257+0000 7efcc3400700 1 -- 172.21.15.89:0/3387398074 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb400cd40 con 0x7efccc15b5d0 2024-04-02T09:14:13.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.257+0000 7efcd2800700 1 -- 172.21.15.89:0/3387398074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc4009a70 con 0x7efccc15b5d0 2024-04-02T09:14:13.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.258+0000 7efcd2800700 1 -- 172.21.15.89:0/3387398074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efccc0ca610 con 0x7efccc15b5d0 2024-04-02T09:14:13.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.259+0000 7efcd2800700 1 -- 172.21.15.89:0/3387398074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efccc15f9a0 con 0x7efccc15b5d0 2024-04-02T09:14:13.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.260+0000 7efcc3400700 1 -- 172.21.15.89:0/3387398074 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 165) v1 ==== 88975+0+0 (secure 0 0 0) 0x7efcb4013470 con 0x7efccc15b5d0 2024-04-02T09:14:13.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.260+0000 7efcc3400700 1 --2- 172.21.15.89:0/3387398074 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efca806aae0 0x7efca806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:13.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.260+0000 7efcc3400700 1 -- 172.21.15.89:0/3387398074 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7795+0+0 (secure 0 0 0) 0x7efcb4089c10 con 0x7efccc15b5d0 2024-04-02T09:14:13.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.261+0000 7efccb400700 1 --2- 172.21.15.89:0/3387398074 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efca806aae0 0x7efca806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:13.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.261+0000 7efccb400700 1 --2- 172.21.15.89:0/3387398074 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efca806aae0 0x7efca806cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efcbc007fd0 tx=0x7efcbc009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:13.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.267+0000 7efcc3400700 1 -- 172.21.15.89:0/3387398074 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mgrmap(e 166) v1 ==== 89285+0+0 (secure 0 0 0) 0x7efcb4054cd0 con 0x7efccc15b5d0 2024-04-02T09:14:13.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.273+0000 7efcc3400700 1 -- 172.21.15.89:0/3387398074 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efcb4086380 con 0x7efccc15b5d0 2024-04-02T09:14:13.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.490+0000 7efcd2800700 1 -- 172.21.15.89:0/3387398074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efccc145090 con 0x7efccc15b5d0 2024-04-02T09:14:13.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.491+0000 7efcc3400700 1 -- 172.21.15.89:0/3387398074 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v250) v1 ==== 74+0+21515 (secure 0 0 0) 0x7efcb4058100 con 0x7efccc15b5d0 2024-04-02T09:14:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:13.491 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":250,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:14:11.230841+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:13:26.654647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-02T09:13:27.342285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:13:40.048039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:13:40.907706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/3750432255":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/2974789087":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/1231626446":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/2398338283":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/4088911225":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3553222249":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3687373115":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/1998199546":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/492795889":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/1337615482":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3750385348":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/2067457213":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/1275405725":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/3355557522":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/4204293224":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/176419408":"2024-04-03T09:14:02.049396+0000","172.21.15.89:6833/108855":"2024-04-03T09:14:02.049396+0000","172.21.15.89:6832/108855":"2024-04-03T09:14:02.049396+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:14:13.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.494+0000 7efcd2800700 1 -- 172.21.15.89:0/3387398074 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efca806aae0 msgr2=0x7efca806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:13.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.494+0000 7efcd2800700 1 --2- 172.21.15.89:0/3387398074 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efca806aae0 0x7efca806cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efcbc007fd0 tx=0x7efcbc009040 comp rx=0 tx=0).stop 2024-04-02T09:14:13.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.494+0000 7efcd2800700 1 -- 172.21.15.89:0/3387398074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc15b5d0 msgr2=0x7efccc0b35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.494+0000 7efcd2800700 1 --2- 172.21.15.89:0/3387398074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc15b5d0 0x7efccc0b35f0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7efcb400ec10 tx=0x7efcb400ef20 comp rx=0 tx=0).stop 2024-04-02T09:14:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.494+0000 7efcd2800700 1 -- 172.21.15.89:0/3387398074 shutdown_connections 2024-04-02T09:14:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.494+0000 7efcd2800700 1 --2- 172.21.15.89:0/3387398074 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efca806aae0 0x7efca806cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.494+0000 7efcd2800700 1 --2- 172.21.15.89:0/3387398074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efccc0b3b30 0x7efccc0c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.494+0000 7efcd2800700 1 --2- 172.21.15.89:0/3387398074 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efccc0ac600 0x7efccc152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.494+0000 7efcd2800700 1 --2- 172.21.15.89:0/3387398074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc15b5d0 0x7efccc0b35f0 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.494+0000 7efcd2800700 1 -- 172.21.15.89:0/3387398074 >> 172.21.15.89:0/3387398074 conn(0x7efccc0cf190 msgr2=0x7efccc06c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.494+0000 7efcd2800700 1 -- 172.21.15.89:0/3387398074 shutdown_connections 2024-04-02T09:14:13.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.494+0000 7efcd2800700 1 -- 172.21.15.89:0/3387398074 wait complete. 2024-04-02T09:14:13.512 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T09:14:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.704+0000 7f13f5000700 1 Processor -- start 2024-04-02T09:14:13.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.704+0000 7f13f5000700 1 -- start start 2024-04-02T09:14:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.705+0000 7f13f5000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f13f0157f90 0x7f13f00b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.705+0000 7f13f5000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f13f00b3b30 0x7f13f0161890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.705+0000 7f13f5000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f00b3070 con 0x7f13f01613e0 2024-04-02T09:14:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.705+0000 7f13f5000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f0158370 con 0x7f13f00b3b30 2024-04-02T09:14:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.705+0000 7f13f5000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f00b2710 con 0x7f13f0157f90 2024-04-02T09:14:13.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.705+0000 7f13ee800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f13f00b3b30 0x7f13f0161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.705+0000 7f13ee800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f13f00b3b30 0x7f13f0161890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50024/0 (socket says 172.21.15.89:50024) 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.705+0000 7f13ee800700 1 -- 172.21.15.89:0/466199080 learned_addr learned my addr 172.21.15.89:0/466199080 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.705+0000 7f13ede00700 1 --2- 172.21.15.89:0/466199080 >> v2:172.21.15.113:3300/0 conn(0x7f13f0157f90 0x7f13f00b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.705+0000 7f13ee800700 1 -- 172.21.15.89:0/466199080 >> v2:172.21.15.113:3300/0 conn(0x7f13f0157f90 msgr2=0x7f13f00b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.706+0000 7f13ee800700 1 --2- 172.21.15.89:0/466199080 >> v2:172.21.15.113:3300/0 conn(0x7f13f0157f90 0x7f13f00b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.706+0000 7f13ee800700 1 -- 172.21.15.89:0/466199080 >> v1:172.21.15.89:6789/0 conn(0x7f13f01613e0 legacy=0x7f13f00b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.706+0000 7f13ee800700 1 -- 172.21.15.89:0/466199080 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f00b2070 con 0x7f13f00b3b30 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.706+0000 7f13ee800700 1 --2- 172.21.15.89:0/466199080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13f00b3b30 0x7f13f0161890 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f13f006fd80 tx=0x7f13e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209092ff65039f69 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.706+0000 7f13eca00700 1 -- 172.21.15.89:0/466199080 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e800e070 con 0x7f13f00b3b30 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.706+0000 7f13eca00700 1 -- 172.21.15.89:0/466199080 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e800b9c0 con 0x7f13f00b3b30 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.706+0000 7f13eca00700 1 -- 172.21.15.89:0/466199080 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e800bd70 con 0x7f13f00b3b30 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.707+0000 7f13f5000700 1 -- 172.21.15.89:0/466199080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13f00b3b30 msgr2=0x7f13f0161890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.707+0000 7f13f5000700 1 --2- 172.21.15.89:0/466199080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13f00b3b30 0x7f13f0161890 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f13f006fd80 tx=0x7f13e800b330 comp rx=0 tx=0).stop 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.707+0000 7f13f5000700 1 -- 172.21.15.89:0/466199080 shutdown_connections 2024-04-02T09:14:13.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.707+0000 7f13f5000700 1 --2- 172.21.15.89:0/466199080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13f00b3b30 0x7f13f0161890 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.707+0000 7f13f5000700 1 --2- 172.21.15.89:0/466199080 >> v2:172.21.15.113:3300/0 conn(0x7f13f0157f90 0x7f13f00b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.707+0000 7f13f5000700 1 -- 172.21.15.89:0/466199080 >> 172.21.15.89:0/466199080 conn(0x7f13f00b0d70 msgr2=0x7f13f00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.707+0000 7f13f5000700 1 -- 172.21.15.89:0/466199080 shutdown_connections 2024-04-02T09:14:13.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.707+0000 7f13f5000700 1 -- 172.21.15.89:0/466199080 wait complete. 2024-04-02T09:14:13.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.708+0000 7f13f5000700 1 Processor -- start 2024-04-02T09:14:13.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.708+0000 7f13f5000700 1 -- start start 2024-04-02T09:14:13.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.709+0000 7f13f5000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13f00b3b30 0x7f13f0153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:13.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.709+0000 7f13f5000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13f0157f90 0x7f13f0159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:13.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.709+0000 7f13f5000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13f01613e0 0x7f13f01584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:13.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.709+0000 7f13f5000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f0158e90 con 0x7f13f01613e0 2024-04-02T09:14:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.709+0000 7f13f5000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f0159170 con 0x7f13f00b3b30 2024-04-02T09:14:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.709+0000 7f13f5000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f0159450 con 0x7f13f0157f90 2024-04-02T09:14:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.710+0000 7f13ee800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13f01613e0 0x7f13f01584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.710+0000 7f13ee800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13f01613e0 0x7f13f01584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50038/0 (socket says 172.21.15.89:50038) 2024-04-02T09:14:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.710+0000 7f13ee800700 1 -- 172.21.15.89:0/536828340 learned_addr learned my addr 172.21.15.89:0/536828340 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.710+0000 7f13ede00700 1 --2- 172.21.15.89:0/536828340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13f00b3b30 0x7f13f0153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.710+0000 7f13ed400700 1 --2- 172.21.15.89:0/536828340 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13f0157f90 0x7f13f0159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.710+0000 7f13ee800700 1 -- 172.21.15.89:0/536828340 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13f0157f90 msgr2=0x7f13f0159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.710+0000 7f13ee800700 1 --2- 172.21.15.89:0/536828340 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13f0157f90 0x7f13f0159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.710+0000 7f13ee800700 1 -- 172.21.15.89:0/536828340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13f00b3b30 msgr2=0x7f13f0153790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.710+0000 7f13ee800700 1 --2- 172.21.15.89:0/536828340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13f00b3b30 0x7f13f0153790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.710+0000 7f13ee800700 1 -- 172.21.15.89:0/536828340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e8009b30 con 0x7f13f01613e0 2024-04-02T09:14:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.710+0000 7f13ee800700 1 --2- 172.21.15.89:0/536828340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13f01613e0 0x7f13f01584b0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f13f006c130 tx=0x7f13e8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:13.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.710+0000 7f13e6000700 1 -- 172.21.15.89:0/536828340 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e8004880 con 0x7f13f01613e0 2024-04-02T09:14:13.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.710+0000 7f13e6000700 1 -- 172.21.15.89:0/536828340 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e80099b0 con 0x7f13f01613e0 2024-04-02T09:14:13.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.711+0000 7f13f5000700 1 -- 172.21.15.89:0/536828340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f0159730 con 0x7f13f01613e0 2024-04-02T09:14:13.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.711+0000 7f13e6000700 1 -- 172.21.15.89:0/536828340 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e801b670 con 0x7f13f01613e0 2024-04-02T09:14:13.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.711+0000 7f13f5000700 1 -- 172.21.15.89:0/536828340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f015bb40 con 0x7f13f01613e0 2024-04-02T09:14:13.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.712+0000 7f13f5000700 1 -- 172.21.15.89:0/536828340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f015bd00 con 0x7f13f01613e0 2024-04-02T09:14:13.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.713+0000 7f13e6000700 1 -- 172.21.15.89:0/536828340 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 166) v1 ==== 89285+0+0 (secure 0 0 0) 0x7f13e801b7d0 con 0x7f13f01613e0 2024-04-02T09:14:13.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.714+0000 7f13e6000700 1 --2- 172.21.15.89:0/536828340 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f13d006ae40 0x7f13d006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:13.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.714+0000 7f13e6000700 1 -- 172.21.15.89:0/536828340 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7795+0+0 (secure 0 0 0) 0x7f13e808b700 con 0x7f13f01613e0 2024-04-02T09:14:13.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.714+0000 7f13ede00700 1 --2- 172.21.15.89:0/536828340 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f13d006ae40 0x7f13d006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:13.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.715+0000 7f13ede00700 1 --2- 172.21.15.89:0/536828340 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f13d006ae40 0x7f13d006d300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f13f006e6b0 tx=0x7f13e0008040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:13.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.718+0000 7f13e6000700 1 -- 172.21.15.89:0/536828340 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f13e8055db0 con 0x7f13f01613e0 2024-04-02T09:14:13.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.965+0000 7f13f5000700 1 -- 172.21.15.89:0/536828340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f13f0067720 con 0x7f13f01613e0 2024-04-02T09:14:13.965 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T09:14:13.966+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T09:14:13.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.978+0000 7f13e6000700 1 -- 172.21.15.89:0/536828340 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v126) v1 ==== 161+0+0 (secure 0 0 0) 0x7f13e80593d0 con 0x7f13f01613e0 2024-04-02T09:14:13.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.981+0000 7f13f5000700 1 -- 172.21.15.89:0/536828340 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f13d006ae40 msgr2=0x7f13d006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:13.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.981+0000 7f13f5000700 1 --2- 172.21.15.89:0/536828340 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f13d006ae40 0x7f13d006d300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f13f006e6b0 tx=0x7f13e0008040 comp rx=0 tx=0).stop 2024-04-02T09:14:13.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.981+0000 7f13f5000700 1 -- 172.21.15.89:0/536828340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13f01613e0 msgr2=0x7f13f01584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:13.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.981+0000 7f13f5000700 1 --2- 172.21.15.89:0/536828340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13f01613e0 0x7f13f01584b0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f13f006c130 tx=0x7f13e8004ad0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.981+0000 7f13f5000700 1 -- 172.21.15.89:0/536828340 shutdown_connections 2024-04-02T09:14:13.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.981+0000 7f13f5000700 1 --2- 172.21.15.89:0/536828340 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f13d006ae40 0x7f13d006d300 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.981+0000 7f13f5000700 1 --2- 172.21.15.89:0/536828340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13f00b3b30 0x7f13f0153790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.981+0000 7f13f5000700 1 --2- 172.21.15.89:0/536828340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13f01613e0 0x7f13f01584b0 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.981+0000 7f13f5000700 1 --2- 172.21.15.89:0/536828340 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13f0157f90 0x7f13f0159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:13.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.981+0000 7f13f5000700 1 -- 172.21.15.89:0/536828340 >> 172.21.15.89:0/536828340 conn(0x7f13f00b0d70 msgr2=0x7f13f00cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:13.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.981+0000 7f13f5000700 1 -- 172.21.15.89:0/536828340 shutdown_connections 2024-04-02T09:14:13.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:13.981+0000 7f13f5000700 1 -- 172.21.15.89:0/536828340 wait complete. 2024-04-02T09:14:13.982 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T09:14:13.988 INFO:tasks.ceph.mds.d.smithi113.stderr: -5739> 2024-04-02T09:13:33.845+0000 7f3675000700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T09:14:13.998 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T09:14:14.038 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T09:14:14.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.193+0000 7fa809200700 1 Processor -- start 2024-04-02T09:14:14.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.193+0000 7fa809200700 1 -- start start 2024-04-02T09:14:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.194+0000 7fa809200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa8040d21f0 0x7fa8040ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.194+0000 7fa809200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8040cec40 0x7fa804161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.194+0000 7fa809200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8040ce270 con 0x7fa8041611b0 2024-04-02T09:14:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.194+0000 7fa809200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa80415b600 con 0x7fa8040d21f0 2024-04-02T09:14:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.194+0000 7fa809200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8040d2760 con 0x7fa8040cec40 2024-04-02T09:14:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.195+0000 7fa801400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa8040d21f0 0x7fa8040ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.195+0000 7fa801400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa8040d21f0 0x7fa8040ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50044/0 (socket says 172.21.15.89:50044) 2024-04-02T09:14:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.195+0000 7fa801400700 1 -- 172.21.15.89:0/1906331209 learned_addr learned my addr 172.21.15.89:0/1906331209 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.195+0000 7fa802800700 1 --2- 172.21.15.89:0/1906331209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8040cec40 0x7fa804161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.195+0000 7fa801400700 1 -- 172.21.15.89:0/1906331209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8040cec40 msgr2=0x7fa804161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.195+0000 7fa801400700 1 --2- 172.21.15.89:0/1906331209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8040cec40 0x7fa804161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:14.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.195+0000 7fa801400700 1 -- 172.21.15.89:0/1906331209 >> v1:172.21.15.89:6789/0 conn(0x7fa8041611b0 legacy=0x7fa80415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:14.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.195+0000 7fa801400700 1 -- 172.21.15.89:0/1906331209 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8040d2a40 con 0x7fa8040d21f0 2024-04-02T09:14:14.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.195+0000 7fa801400700 1 --2- 172.21.15.89:0/1906331209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8040d21f0 0x7fa8040ce700 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7fa7f00098f0 tx=0x7fa7f0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f46a00e8e5d499 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:14.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.196+0000 7fa800a00700 1 -- 172.21.15.89:0/1906331209 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f0011830 con 0x7fa8040d21f0 2024-04-02T09:14:14.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.196+0000 7fa800a00700 1 -- 172.21.15.89:0/1906331209 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f0011990 con 0x7fa8040d21f0 2024-04-02T09:14:14.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.196+0000 7fa800a00700 1 -- 172.21.15.89:0/1906331209 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f0011d20 con 0x7fa8040d21f0 2024-04-02T09:14:14.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.196+0000 7fa809200700 1 -- 172.21.15.89:0/1906331209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8040d21f0 msgr2=0x7fa8040ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:14.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.196+0000 7fa809200700 1 --2- 172.21.15.89:0/1906331209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8040d21f0 0x7fa8040ce700 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7fa7f00098f0 tx=0x7fa7f0009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:14.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.196+0000 7fa809200700 1 -- 172.21.15.89:0/1906331209 shutdown_connections 2024-04-02T09:14:14.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.196+0000 7fa809200700 1 --2- 172.21.15.89:0/1906331209 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8040cec40 0x7fa804161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:14.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.196+0000 7fa809200700 1 --2- 172.21.15.89:0/1906331209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8040d21f0 0x7fa8040ce700 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:14.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.196+0000 7fa809200700 1 -- 172.21.15.89:0/1906331209 >> 172.21.15.89:0/1906331209 conn(0x7fa80408aff0 msgr2=0x7fa8040d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:14.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.196+0000 7fa809200700 1 -- 172.21.15.89:0/1906331209 shutdown_connections 2024-04-02T09:14:14.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.196+0000 7fa809200700 1 -- 172.21.15.89:0/1906331209 wait complete. 2024-04-02T09:14:14.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.198+0000 7fa809200700 1 Processor -- start 2024-04-02T09:14:14.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.198+0000 7fa809200700 1 -- start start 2024-04-02T09:14:14.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.199+0000 7fa809200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8040cec40 0x7fa8041e6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:14.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.199+0000 7fa809200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8040d21f0 0x7fa8040b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:14.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.199+0000 7fa809200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8041611b0 0x7fa8040b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:14.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.199+0000 7fa809200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8040cbae0 con 0x7fa8040cec40 2024-04-02T09:14:14.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.199+0000 7fa809200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8040cbdc0 con 0x7fa8041611b0 2024-04-02T09:14:14.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.199+0000 7fa809200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8040cc0a0 con 0x7fa8040d21f0 2024-04-02T09:14:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.199+0000 7fa801e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8040cec40 0x7fa8041e6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.199+0000 7fa801e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8040cec40 0x7fa8041e6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50054/0 (socket says 172.21.15.89:50054) 2024-04-02T09:14:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.199+0000 7fa801e00700 1 -- 172.21.15.89:0/1736063829 learned_addr learned my addr 172.21.15.89:0/1736063829 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.199+0000 7fa802800700 1 --2- 172.21.15.89:0/1736063829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8041611b0 0x7fa8040b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.199+0000 7fa801400700 1 --2- 172.21.15.89:0/1736063829 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8040d21f0 0x7fa8040b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.200+0000 7fa801e00700 1 -- 172.21.15.89:0/1736063829 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8040d21f0 msgr2=0x7fa8040b85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.200+0000 7fa801e00700 1 --2- 172.21.15.89:0/1736063829 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8040d21f0 0x7fa8040b85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.200+0000 7fa801e00700 1 -- 172.21.15.89:0/1736063829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8041611b0 msgr2=0x7fa8040b8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.200+0000 7fa801e00700 1 --2- 172.21.15.89:0/1736063829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8041611b0 0x7fa8040b8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.200+0000 7fa801e00700 1 -- 172.21.15.89:0/1736063829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f0009550 con 0x7fa8040cec40 2024-04-02T09:14:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.200+0000 7fa801e00700 1 --2- 172.21.15.89:0/1736063829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8040cec40 0x7fa8041e6ff0 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7fa7f400c450 tx=0x7fa7f400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.200+0000 7fa7fa000700 1 -- 172.21.15.89:0/1736063829 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f40052d0 con 0x7fa8040cec40 2024-04-02T09:14:14.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.200+0000 7fa7fa000700 1 -- 172.21.15.89:0/1736063829 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f4005430 con 0x7fa8040cec40 2024-04-02T09:14:14.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.200+0000 7fa7fa000700 1 -- 172.21.15.89:0/1736063829 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f40038c0 con 0x7fa8040cec40 2024-04-02T09:14:14.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.200+0000 7fa809200700 1 -- 172.21.15.89:0/1736063829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8040cc380 con 0x7fa8040cec40 2024-04-02T09:14:14.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.200+0000 7fa809200700 1 -- 172.21.15.89:0/1736063829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa804156d90 con 0x7fa8040cec40 2024-04-02T09:14:14.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.201+0000 7fa809200700 1 -- 172.21.15.89:0/1736063829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8040cc8a0 con 0x7fa8040cec40 2024-04-02T09:14:14.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.202+0000 7fa7fa000700 1 -- 172.21.15.89:0/1736063829 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 166) v1 ==== 89285+0+0 (secure 0 0 0) 0x7fa7f4003a20 con 0x7fa8040cec40 2024-04-02T09:14:14.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.203+0000 7fa7fa000700 1 --2- 172.21.15.89:0/1736063829 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa7e406ae40 0x7fa7e406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:14.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.203+0000 7fa7fa000700 1 -- 172.21.15.89:0/1736063829 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7881+0+0 (secure 0 0 0) 0x7fa7f40891e0 con 0x7fa8040cec40 2024-04-02T09:14:14.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.203+0000 7fa801400700 1 --2- 172.21.15.89:0/1736063829 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa7e406ae40 0x7fa7e406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:14.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.204+0000 7fa801400700 1 --2- 172.21.15.89:0/1736063829 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa7e406ae40 0x7fa7e406d300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0009c40 tx=0x7fa7f0008030 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:14.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.208+0000 7fa7fa000700 1 -- 172.21.15.89:0/1736063829 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa7f40538b0 con 0x7fa8040cec40 2024-04-02T09:14:14.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.455+0000 7fa809200700 1 -- 172.21.15.89:0/1736063829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa804067720 con 0x7fa8040cec40 2024-04-02T09:14:14.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.980+0000 7fa7fa000700 1 -- 172.21.15.89:0/1736063829 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v127) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa7f400ce60 con 0x7fa8040cec40 2024-04-02T09:14:14.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.983+0000 7fa809200700 1 -- 172.21.15.89:0/1736063829 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa7e406ae40 msgr2=0x7fa7e406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:14.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.983+0000 7fa809200700 1 --2- 172.21.15.89:0/1736063829 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa7e406ae40 0x7fa7e406d300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0009c40 tx=0x7fa7f0008030 comp rx=0 tx=0).stop 2024-04-02T09:14:14.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.983+0000 7fa809200700 1 -- 172.21.15.89:0/1736063829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8040cec40 msgr2=0x7fa8041e6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:14.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.983+0000 7fa809200700 1 --2- 172.21.15.89:0/1736063829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8040cec40 0x7fa8041e6ff0 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7fa7f400c450 tx=0x7fa7f400c760 comp rx=0 tx=0).stop 2024-04-02T09:14:14.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.983+0000 7fa809200700 1 -- 172.21.15.89:0/1736063829 shutdown_connections 2024-04-02T09:14:14.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.983+0000 7fa809200700 1 --2- 172.21.15.89:0/1736063829 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa7e406ae40 0x7fa7e406d300 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:14.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.983+0000 7fa809200700 1 --2- 172.21.15.89:0/1736063829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8041611b0 0x7fa8040b8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:14.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.983+0000 7fa809200700 1 --2- 172.21.15.89:0/1736063829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8040cec40 0x7fa8041e6ff0 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:14.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.983+0000 7fa809200700 1 --2- 172.21.15.89:0/1736063829 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa8040d21f0 0x7fa8040b85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:14.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.983+0000 7fa809200700 1 -- 172.21.15.89:0/1736063829 >> 172.21.15.89:0/1736063829 conn(0x7fa80408aff0 msgr2=0x7fa80415f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:14.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.984+0000 7fa809200700 1 -- 172.21.15.89:0/1736063829 shutdown_connections 2024-04-02T09:14:14.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:14.984+0000 7fa809200700 1 -- 172.21.15.89:0/1736063829 wait complete. 2024-04-02T09:14:15.003 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T09:14:15.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.202+0000 7efca6400700 1 Processor -- start 2024-04-02T09:14:15.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.202+0000 7efca6400700 1 -- start start 2024-04-02T09:14:15.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.203+0000 7efca6400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efca015ddf0 0x7efca00ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:15.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.203+0000 7efca6400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efca015e1d0 0x7efca00d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:15.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.203+0000 7efca6400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca00d2b40 con 0x7efca00d2710 2024-04-02T09:14:15.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.203+0000 7efca6400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca015ec60 con 0x7efca015ddf0 2024-04-02T09:14:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.203+0000 7efca6400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca00ce8a0 con 0x7efca015e1d0 2024-04-02T09:14:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.203+0000 7efc9f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efca015ddf0 0x7efca00ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.203+0000 7efc9f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efca015ddf0 0x7efca00ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50070/0 (socket says 172.21.15.89:50070) 2024-04-02T09:14:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.203+0000 7efc9f400700 1 -- 172.21.15.89:0/815136510 learned_addr learned my addr 172.21.15.89:0/815136510 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.204+0000 7efc9ea00700 1 --2- 172.21.15.89:0/815136510 >> v2:172.21.15.113:3300/0 conn(0x7efca015e1d0 0x7efca00d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.204+0000 7efc9f400700 1 -- 172.21.15.89:0/815136510 >> v2:172.21.15.113:3300/0 conn(0x7efca015e1d0 msgr2=0x7efca00d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.204+0000 7efc9f400700 1 --2- 172.21.15.89:0/815136510 >> v2:172.21.15.113:3300/0 conn(0x7efca015e1d0 0x7efca00d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.204+0000 7efc9f400700 1 -- 172.21.15.89:0/815136510 >> v1:172.21.15.89:6789/0 conn(0x7efca00d2710 legacy=0x7efca01612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.204+0000 7efc9f400700 1 -- 172.21.15.89:0/815136510 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca00cdc90 con 0x7efca015ddf0 2024-04-02T09:14:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.204+0000 7efc9f400700 1 --2- 172.21.15.89:0/815136510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efca015ddf0 0x7efca00ced30 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7efca00d0fe0 tx=0x7efc90009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba325df398b1194 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.204+0000 7efc9e000700 1 -- 172.21.15.89:0/815136510 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9000b870 con 0x7efca015ddf0 2024-04-02T09:14:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.204+0000 7efc9e000700 1 -- 172.21.15.89:0/815136510 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9000b9d0 con 0x7efca015ddf0 2024-04-02T09:14:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.204+0000 7efc9e000700 1 -- 172.21.15.89:0/815136510 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9000bd60 con 0x7efca015ddf0 2024-04-02T09:14:15.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.205+0000 7efca6400700 1 -- 172.21.15.89:0/815136510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efca015ddf0 msgr2=0x7efca00ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:15.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.205+0000 7efca6400700 1 --2- 172.21.15.89:0/815136510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efca015ddf0 0x7efca00ced30 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7efca00d0fe0 tx=0x7efc90009bd0 comp rx=0 tx=0).stop 2024-04-02T09:14:15.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.205+0000 7efca6400700 1 -- 172.21.15.89:0/815136510 shutdown_connections 2024-04-02T09:14:15.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.205+0000 7efca6400700 1 --2- 172.21.15.89:0/815136510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efca015ddf0 0x7efca00ced30 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:15.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.205+0000 7efca6400700 1 --2- 172.21.15.89:0/815136510 >> v2:172.21.15.113:3300/0 conn(0x7efca015e1d0 0x7efca00d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:15.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.205+0000 7efca6400700 1 -- 172.21.15.89:0/815136510 >> 172.21.15.89:0/815136510 conn(0x7efca008aff0 msgr2=0x7efca00d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:15.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.205+0000 7efca6400700 1 -- 172.21.15.89:0/815136510 shutdown_connections 2024-04-02T09:14:15.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.205+0000 7efca6400700 1 -- 172.21.15.89:0/815136510 wait complete. 2024-04-02T09:14:15.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.207+0000 7efca6400700 1 Processor -- start 2024-04-02T09:14:15.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.207+0000 7efca6400700 1 -- start start 2024-04-02T09:14:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efca6400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca00d2710 0x7efca0156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efca6400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca015ddf0 0x7efca01e87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efca6400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efca015e1d0 0x7efca00c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efca6400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca0157bc0 con 0x7efca015e1d0 2024-04-02T09:14:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efca6400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca015a980 con 0x7efca00d2710 2024-04-02T09:14:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efca6400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca01e91f0 con 0x7efca015ddf0 2024-04-02T09:14:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efc9fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efca015e1d0 0x7efca00c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efc9fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efca015e1d0 0x7efca00c91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50078/0 (socket says 172.21.15.89:50078) 2024-04-02T09:14:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efc9fe00700 1 -- 172.21.15.89:0/2335937640 learned_addr learned my addr 172.21.15.89:0/2335937640 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efc9ea00700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca015ddf0 0x7efca01e87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efc9f400700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca00d2710 0x7efca0156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efc9fe00700 1 -- 172.21.15.89:0/2335937640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca015ddf0 msgr2=0x7efca01e87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efc9fe00700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca015ddf0 0x7efca01e87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efc9fe00700 1 -- 172.21.15.89:0/2335937640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca00d2710 msgr2=0x7efca0156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efc9fe00700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca00d2710 0x7efca0156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efc9fe00700 1 -- 172.21.15.89:0/2335937640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc90009550 con 0x7efca015e1d0 2024-04-02T09:14:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efc9ea00700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca015ddf0 0x7efca01e87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.208+0000 7efc9f400700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca00d2710 0x7efca0156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.209+0000 7efc9fe00700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efca015e1d0 0x7efca00c91f0 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7efc9800cc50 tx=0x7efc9800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.209+0000 7efc97400700 1 -- 172.21.15.89:0/2335937640 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9800ad80 con 0x7efca015e1d0 2024-04-02T09:14:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.209+0000 7efc97400700 1 -- 172.21.15.89:0/2335937640 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9800ce50 con 0x7efca015e1d0 2024-04-02T09:14:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.209+0000 7efc97400700 1 -- 172.21.15.89:0/2335937640 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98003c40 con 0x7efca015e1d0 2024-04-02T09:14:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.209+0000 7efca6400700 1 -- 172.21.15.89:0/2335937640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca015eed0 con 0x7efca015e1d0 2024-04-02T09:14:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.209+0000 7efca6400700 1 -- 172.21.15.89:0/2335937640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca015f490 con 0x7efca015e1d0 2024-04-02T09:14:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.210+0000 7efca6400700 1 -- 172.21.15.89:0/2335937640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca00ce4f0 con 0x7efca015e1d0 2024-04-02T09:14:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.211+0000 7efc97400700 1 -- 172.21.15.89:0/2335937640 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 166) v1 ==== 89285+0+0 (secure 0 0 0) 0x7efc980050d0 con 0x7efca015e1d0 2024-04-02T09:14:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.212+0000 7efc97400700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efc8006ae40 0x7efc8006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.212+0000 7efc97400700 1 -- 172.21.15.89:0/2335937640 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7881+0+0 (secure 0 0 0) 0x7efc98089f00 con 0x7efca015e1d0 2024-04-02T09:14:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.212+0000 7efc9f400700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efc8006ae40 0x7efc8006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.212+0000 7efc9f400700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efc8006ae40 0x7efc8006d300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efc90000c00 tx=0x7efc90008030 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.216+0000 7efc97400700 1 -- 172.21.15.89:0/2335937640 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efc98054670 con 0x7efca015e1d0 2024-04-02T09:14:15.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.424+0000 7efca6400700 1 -- 172.21.15.89:0/2335937640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efca0067720 con 0x7efca015e1d0 2024-04-02T09:14:15.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:15.990+0000 7efc97400700 1 -- 172.21.15.89:0/2335937640 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7efc98054a40 con 0x7efca015e1d0 2024-04-02T09:14:16.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.272+0000 7efc97400700 1 -- 172.21.15.89:0/2335937640 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v252) v1 ==== 181+0+0 (secure 0 0 0) 0x7efc98071d60 con 0x7efca015e1d0 2024-04-02T09:14:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.277+0000 7efca6400700 1 -- 172.21.15.89:0/2335937640 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efc8006ae40 msgr2=0x7efc8006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.277+0000 7efca6400700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efc8006ae40 0x7efc8006d300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efc90000c00 tx=0x7efc90008030 comp rx=0 tx=0).stop 2024-04-02T09:14:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.277+0000 7efca6400700 1 -- 172.21.15.89:0/2335937640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efca015e1d0 msgr2=0x7efca00c91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.277+0000 7efca6400700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efca015e1d0 0x7efca00c91f0 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7efc9800cc50 tx=0x7efc9800a340 comp rx=0 tx=0).stop 2024-04-02T09:14:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.277+0000 7efca6400700 1 -- 172.21.15.89:0/2335937640 shutdown_connections 2024-04-02T09:14:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.277+0000 7efca6400700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efc8006ae40 0x7efc8006d300 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.277+0000 7efca6400700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca00d2710 0x7efca0156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.278+0000 7efca6400700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efca015e1d0 0x7efca00c91f0 unknown :-1 s=CLOSED pgs=2719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.278+0000 7efca6400700 1 --2- 172.21.15.89:0/2335937640 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efca015ddf0 0x7efca01e87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.278+0000 7efca6400700 1 -- 172.21.15.89:0/2335937640 >> 172.21.15.89:0/2335937640 conn(0x7efca008aff0 msgr2=0x7efca00b79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:16.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.278+0000 7efca6400700 1 -- 172.21.15.89:0/2335937640 shutdown_connections 2024-04-02T09:14:16.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.278+0000 7efca6400700 1 -- 172.21.15.89:0/2335937640 wait complete. 2024-04-02T09:14:16.278 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T09:14:16.298 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T09:14:16.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.506+0000 7fdb7ec00700 1 Processor -- start 2024-04-02T09:14:16.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.507+0000 7fdb7ec00700 1 -- start start 2024-04-02T09:14:16.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.508+0000 7fdb7ec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb780d1fe0 0x7fdb780d1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:16.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.508+0000 7fdb7ec00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb780ced30 con 0x7fdb7815db20 2024-04-02T09:14:16.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.508+0000 7fdb7ec00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb780cef80 con 0x7fdb7815d740 2024-04-02T09:14:16.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.508+0000 7fdb7ec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb780cf260 con 0x7fdb780d1fe0 2024-04-02T09:14:16.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.508+0000 7fdb7cc00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdb7815db20 0x7fdb780b0320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53170/0 (socket says 172.21.15.89:53170) 2024-04-02T09:14:16.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.508+0000 7fdb7cc00700 1 -- 172.21.15.89:0/3876106089 learned_addr learned my addr 172.21.15.89:0/3876106089 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:16.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.508+0000 7fdb77400700 1 --2- 172.21.15.89:0/3876106089 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb780d1fe0 0x7fdb780d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:16.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.508+0000 7fdb76a00700 1 -- 172.21.15.89:0/3876106089 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726146721 0 0) 0x7fdb780ced30 con 0x7fdb7815db20 2024-04-02T09:14:16.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.509+0000 7fdb76a00700 1 -- 172.21.15.89:0/3876106089 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb5c0035b0 con 0x7fdb7815db20 2024-04-02T09:14:16.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.509+0000 7fdb76a00700 1 -- 172.21.15.89:0/3876106089 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671437907 0 0) 0x7fdb780cef80 con 0x7fdb7815d740 2024-04-02T09:14:16.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.509+0000 7fdb76a00700 1 -- 172.21.15.89:0/3876106089 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb780ced30 con 0x7fdb7815d740 2024-04-02T09:14:16.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.509+0000 7fdb77400700 1 -- 172.21.15.89:0/3876106089 >> v1:172.21.15.113:6789/0 conn(0x7fdb7815d740 legacy=0x7fdb780d18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:16.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.509+0000 7fdb77400700 1 -- 172.21.15.89:0/3876106089 >> v1:172.21.15.89:6789/0 conn(0x7fdb7815db20 legacy=0x7fdb780b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:16.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.509+0000 7fdb77400700 1 -- 172.21.15.89:0/3876106089 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb780cf540 con 0x7fdb780d1fe0 2024-04-02T09:14:16.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.509+0000 7fdb77400700 1 --2- 172.21.15.89:0/3876106089 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb780d1fe0 0x7fdb780d1280 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009870 tx=0x7fdb64009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56aff55a996b9b2d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:16.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.509+0000 7fdb76a00700 1 -- 172.21.15.89:0/3876106089 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb64011830 con 0x7fdb780d1fe0 2024-04-02T09:14:16.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.509+0000 7fdb76a00700 1 -- 172.21.15.89:0/3876106089 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb64011990 con 0x7fdb780d1fe0 2024-04-02T09:14:16.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.509+0000 7fdb76a00700 1 -- 172.21.15.89:0/3876106089 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb64011d20 con 0x7fdb780d1fe0 2024-04-02T09:14:16.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.510+0000 7fdb7ec00700 1 -- 172.21.15.89:0/3876106089 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb780d1fe0 msgr2=0x7fdb780d1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:16.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.510+0000 7fdb7ec00700 1 --2- 172.21.15.89:0/3876106089 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb780d1fe0 0x7fdb780d1280 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009870 tx=0x7fdb64009b80 comp rx=0 tx=0).stop 2024-04-02T09:14:16.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.510+0000 7fdb7ec00700 1 -- 172.21.15.89:0/3876106089 shutdown_connections 2024-04-02T09:14:16.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.510+0000 7fdb7ec00700 1 --2- 172.21.15.89:0/3876106089 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb780d1fe0 0x7fdb780d1280 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:16.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.510+0000 7fdb7ec00700 1 -- 172.21.15.89:0/3876106089 >> 172.21.15.89:0/3876106089 conn(0x7fdb7808aff0 msgr2=0x7fdb780d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:16.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.510+0000 7fdb7ec00700 1 -- 172.21.15.89:0/3876106089 shutdown_connections 2024-04-02T09:14:16.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.510+0000 7fdb7ec00700 1 -- 172.21.15.89:0/3876106089 wait complete. 2024-04-02T09:14:16.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.512+0000 7fdb7ec00700 1 Processor -- start 2024-04-02T09:14:16.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.512+0000 7fdb7ec00700 1 -- start start 2024-04-02T09:14:16.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb7ec00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb780d1fe0 0x7fdb780b7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:16.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb7ec00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb7815d740 0x7fdb780b8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:16.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb7ec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb7815db20 0x7fdb780cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:16.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb7ec00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb780b2880 con 0x7fdb7815d740 2024-04-02T09:14:16.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb7ec00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb780b3890 con 0x7fdb780d1fe0 2024-04-02T09:14:16.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb7ec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb780b2ad0 con 0x7fdb7815db20 2024-04-02T09:14:16.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb77400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb7815d740 0x7fdb780b8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:16.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb77400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb7815d740 0x7fdb780b8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50082/0 (socket says 172.21.15.89:50082) 2024-04-02T09:14:16.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb77400700 1 -- 172.21.15.89:0/373042125 learned_addr learned my addr 172.21.15.89:0/373042125 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:16.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb7cc00700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb7815db20 0x7fdb780cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:16.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb77e00700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb780d1fe0 0x7fdb780b7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:16.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb77400700 1 -- 172.21.15.89:0/373042125 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb7815db20 msgr2=0x7fdb780cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:16.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb77400700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb7815db20 0x7fdb780cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:16.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb77400700 1 -- 172.21.15.89:0/373042125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb780d1fe0 msgr2=0x7fdb780b7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:16.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb77400700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb780d1fe0 0x7fdb780b7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:16.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb77400700 1 -- 172.21.15.89:0/373042125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb640094a0 con 0x7fdb7815d740 2024-04-02T09:14:16.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb7cc00700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb7815db20 0x7fdb780cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:16.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.513+0000 7fdb77e00700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb780d1fe0 0x7fdb780b7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:16.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.514+0000 7fdb77400700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb7815d740 0x7fdb780b8470 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fdb64014040 tx=0x7fdb64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:16.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.514+0000 7fdb6fe00700 1 -- 172.21.15.89:0/373042125 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6401a860 con 0x7fdb7815d740 2024-04-02T09:14:16.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.514+0000 7fdb6fe00700 1 -- 172.21.15.89:0/373042125 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6401a9c0 con 0x7fdb7815d740 2024-04-02T09:14:16.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.514+0000 7fdb6fe00700 1 -- 172.21.15.89:0/373042125 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6401ad30 con 0x7fdb7815d740 2024-04-02T09:14:16.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.514+0000 7fdb7ec00700 1 -- 172.21.15.89:0/373042125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb780b2db0 con 0x7fdb7815d740 2024-04-02T09:14:16.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.514+0000 7fdb7ec00700 1 -- 172.21.15.89:0/373042125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb780b3370 con 0x7fdb7815d740 2024-04-02T09:14:16.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.515+0000 7fdb7ec00700 1 -- 172.21.15.89:0/373042125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb78067720 con 0x7fdb7815d740 2024-04-02T09:14:16.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.516+0000 7fdb6fe00700 1 -- 172.21.15.89:0/373042125 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fdb64022070 con 0x7fdb7815d740 2024-04-02T09:14:16.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.517+0000 7fdb6fe00700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb5806af00 0x7fdb5806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:16.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.517+0000 7fdb6fe00700 1 -- 172.21.15.89:0/373042125 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7443+0+0 (secure 0 0 0) 0x7fdb6408a9f0 con 0x7fdb7815d740 2024-04-02T09:14:16.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.517+0000 7fdb77e00700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb5806af00 0x7fdb5806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:16.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.517+0000 7fdb77e00700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb5806af00 0x7fdb5806d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdb680011e0 tx=0x7fdb68009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:16.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.521+0000 7fdb6fe00700 1 -- 172.21.15.89:0/373042125 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdb640552e0 con 0x7fdb7815d740 2024-04-02T09:14:16.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:16.724+0000 7fdb7ec00700 1 -- 172.21.15.89:0/373042125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdb7815b4e0 con 0x7fdb7815d740 2024-04-02T09:14:17.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.280+0000 7fdb6fe00700 1 -- 172.21.15.89:0/373042125 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v253) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdb640150d0 con 0x7fdb7815d740 2024-04-02T09:14:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.284+0000 7fdb7ec00700 1 -- 172.21.15.89:0/373042125 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb5806af00 msgr2=0x7fdb5806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.284+0000 7fdb7ec00700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb5806af00 0x7fdb5806d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdb680011e0 tx=0x7fdb68009040 comp rx=0 tx=0).stop 2024-04-02T09:14:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.284+0000 7fdb7ec00700 1 -- 172.21.15.89:0/373042125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb7815d740 msgr2=0x7fdb780b8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.284+0000 7fdb7ec00700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb7815d740 0x7fdb780b8470 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fdb64014040 tx=0x7fdb64004750 comp rx=0 tx=0).stop 2024-04-02T09:14:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.285+0000 7fdb7ec00700 1 -- 172.21.15.89:0/373042125 shutdown_connections 2024-04-02T09:14:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.285+0000 7fdb7ec00700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb5806af00 0x7fdb5806d3c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.285+0000 7fdb7ec00700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb780d1fe0 0x7fdb780b7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.285+0000 7fdb7ec00700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb7815db20 0x7fdb780cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.285+0000 7fdb7ec00700 1 --2- 172.21.15.89:0/373042125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb7815d740 0x7fdb780b8470 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.285+0000 7fdb7ec00700 1 -- 172.21.15.89:0/373042125 >> 172.21.15.89:0/373042125 conn(0x7fdb7808aff0 msgr2=0x7fdb7806d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:17.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.285+0000 7fdb7ec00700 1 -- 172.21.15.89:0/373042125 shutdown_connections 2024-04-02T09:14:17.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.285+0000 7fdb7ec00700 1 -- 172.21.15.89:0/373042125 wait complete. 2024-04-02T09:14:17.285 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T09:14:17.303 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:14:17.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.506+0000 7fd686400700 1 Processor -- start 2024-04-02T09:14:17.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.506+0000 7fd686400700 1 -- start start 2024-04-02T09:14:17.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.507+0000 7fd686400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd680160f10 0x7fd6800d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:17.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.507+0000 7fd686400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd6800cf230 0x7fd6800d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:17.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.507+0000 7fd686400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6800ceda0 con 0x7fd680161590 2024-04-02T09:14:17.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.507+0000 7fd686400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6800d23d0 con 0x7fd6800cf230 2024-04-02T09:14:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.507+0000 7fd686400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6801612f0 con 0x7fd680160f10 2024-04-02T09:14:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.507+0000 7fd67fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd680161590 0x7fd6800b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53180/0 (socket says 172.21.15.89:53180) 2024-04-02T09:14:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.507+0000 7fd67fe00700 1 -- 172.21.15.89:0/2216932984 learned_addr learned my addr 172.21.15.89:0/2216932984 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.507+0000 7fd67f400700 1 --2- 172.21.15.89:0/2216932984 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd680160f10 0x7fd6800d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.507+0000 7fd67ea00700 1 --2- 172.21.15.89:0/2216932984 >> v2:172.21.15.113:3300/0 conn(0x7fd6800cf230 0x7fd6800d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.508+0000 7fd67e000700 1 -- 172.21.15.89:0/2216932984 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596920435 0 0) 0x7fd6800ceda0 con 0x7fd680161590 2024-04-02T09:14:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.508+0000 7fd67e000700 1 -- 172.21.15.89:0/2216932984 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6640035b0 con 0x7fd680161590 2024-04-02T09:14:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.508+0000 7fd67f400700 1 -- 172.21.15.89:0/2216932984 >> v2:172.21.15.113:3300/0 conn(0x7fd6800cf230 msgr2=0x7fd6800d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.508+0000 7fd67f400700 1 --2- 172.21.15.89:0/2216932984 >> v2:172.21.15.113:3300/0 conn(0x7fd6800cf230 0x7fd6800d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.508+0000 7fd67f400700 1 -- 172.21.15.89:0/2216932984 >> v1:172.21.15.89:6789/0 conn(0x7fd680161590 legacy=0x7fd6800b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:17.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.508+0000 7fd67f400700 1 -- 172.21.15.89:0/2216932984 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6800ce470 con 0x7fd680160f10 2024-04-02T09:14:17.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.508+0000 7fd67f400700 1 --2- 172.21.15.89:0/2216932984 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd680160f10 0x7fd6800d18b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd670009840 tx=0x7fd670009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=779660ba04f8e6ad server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:17.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.509+0000 7fd67e000700 1 -- 172.21.15.89:0/2216932984 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd670011830 con 0x7fd680160f10 2024-04-02T09:14:17.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.509+0000 7fd67e000700 1 -- 172.21.15.89:0/2216932984 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd670011990 con 0x7fd680160f10 2024-04-02T09:14:17.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.509+0000 7fd67e000700 1 -- 172.21.15.89:0/2216932984 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd670011d20 con 0x7fd680160f10 2024-04-02T09:14:17.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.509+0000 7fd686400700 1 -- 172.21.15.89:0/2216932984 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd680160f10 msgr2=0x7fd6800d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.509+0000 7fd686400700 1 --2- 172.21.15.89:0/2216932984 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd680160f10 0x7fd6800d18b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd670009840 tx=0x7fd670009b50 comp rx=0 tx=0).stop 2024-04-02T09:14:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.509+0000 7fd686400700 1 -- 172.21.15.89:0/2216932984 shutdown_connections 2024-04-02T09:14:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.509+0000 7fd686400700 1 --2- 172.21.15.89:0/2216932984 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd680160f10 0x7fd6800d18b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.509+0000 7fd686400700 1 --2- 172.21.15.89:0/2216932984 >> v2:172.21.15.113:3300/0 conn(0x7fd6800cf230 0x7fd6800d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.509+0000 7fd686400700 1 -- 172.21.15.89:0/2216932984 >> 172.21.15.89:0/2216932984 conn(0x7fd68008aff0 msgr2=0x7fd6800d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.509+0000 7fd686400700 1 -- 172.21.15.89:0/2216932984 shutdown_connections 2024-04-02T09:14:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.509+0000 7fd686400700 1 -- 172.21.15.89:0/2216932984 wait complete. 2024-04-02T09:14:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.511+0000 7fd686400700 1 Processor -- start 2024-04-02T09:14:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.511+0000 7fd686400700 1 -- start start 2024-04-02T09:14:17.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.512+0000 7fd686400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6800cf230 0x7fd680155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:17.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.512+0000 7fd686400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd680160f10 0x7fd6800c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.512+0000 7fd686400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd680161590 0x7fd680157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.512+0000 7fd686400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6800b64a0 con 0x7fd680161590 2024-04-02T09:14:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.512+0000 7fd686400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd680157f00 con 0x7fd680160f10 2024-04-02T09:14:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.512+0000 7fd686400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6800b6150 con 0x7fd6800cf230 2024-04-02T09:14:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.512+0000 7fd67fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd680161590 0x7fd680157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.512+0000 7fd67fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd680161590 0x7fd680157030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50090/0 (socket says 172.21.15.89:50090) 2024-04-02T09:14:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.512+0000 7fd67fe00700 1 -- 172.21.15.89:0/2943940639 learned_addr learned my addr 172.21.15.89:0/2943940639 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.513+0000 7fd67f400700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6800cf230 0x7fd680155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.513+0000 7fd67ea00700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd680160f10 0x7fd6800c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.513+0000 7fd67fe00700 1 -- 172.21.15.89:0/2943940639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6800cf230 msgr2=0x7fd680155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.513+0000 7fd67fe00700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6800cf230 0x7fd680155c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.513+0000 7fd67fe00700 1 -- 172.21.15.89:0/2943940639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd680160f10 msgr2=0x7fd6800c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.513+0000 7fd67fe00700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd680160f10 0x7fd6800c91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.513+0000 7fd67fe00700 1 -- 172.21.15.89:0/2943940639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6700094a0 con 0x7fd680161590 2024-04-02T09:14:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.513+0000 7fd67ea00700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd680160f10 0x7fd6800c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.513+0000 7fd67fe00700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd680161590 0x7fd680157030 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7fd67800caa0 tx=0x7fd67800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.513+0000 7fd677400700 1 -- 172.21.15.89:0/2943940639 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67800aa80 con 0x7fd680161590 2024-04-02T09:14:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.513+0000 7fd677400700 1 -- 172.21.15.89:0/2943940639 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67800abe0 con 0x7fd680161590 2024-04-02T09:14:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.513+0000 7fd677400700 1 -- 172.21.15.89:0/2943940639 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678012400 con 0x7fd680161590 2024-04-02T09:14:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.513+0000 7fd686400700 1 -- 172.21.15.89:0/2943940639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd680158150 con 0x7fd680161590 2024-04-02T09:14:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.513+0000 7fd686400700 1 -- 172.21.15.89:0/2943940639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6801e6740 con 0x7fd680161590 2024-04-02T09:14:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.514+0000 7fd686400700 1 -- 172.21.15.89:0/2943940639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6800b65e0 con 0x7fd680161590 2024-04-02T09:14:17.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.515+0000 7fd677400700 1 -- 172.21.15.89:0/2943940639 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fd67801b3b0 con 0x7fd680161590 2024-04-02T09:14:17.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.516+0000 7fd677400700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd66006aed0 0x7fd66006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.516+0000 7fd677400700 1 -- 172.21.15.89:0/2943940639 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7029+0+0 (secure 0 0 0) 0x7fd6780899b0 con 0x7fd680161590 2024-04-02T09:14:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.516+0000 7fd67f400700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd66006aed0 0x7fd66006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.517+0000 7fd67f400700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd66006aed0 0x7fd66006d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd670000c00 tx=0x7fd67001c040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:17.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.521+0000 7fd677400700 1 -- 172.21.15.89:0/2943940639 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd6800b65e0 con 0x7fd680161590 2024-04-02T09:14:17.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.724+0000 7fd686400700 1 -- 172.21.15.89:0/2943940639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6800b65e0 con 0x7fd680161590 2024-04-02T09:14:17.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.726+0000 7fd677400700 1 -- 172.21.15.89:0/2943940639 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) v1 ==== 74+0+17997 (secure 0 0 0) 0x7fd6800b65e0 con 0x7fd680161590 2024-04-02T09:14:17.725 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:17.725 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":253,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:14:17.272062+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:13:40.048039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:13:40.907706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.113:6833/703852199":"2024-04-03T09:14:13.967181+0000","172.21.15.89:0/3750432255":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/2974789087":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/1231626446":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/2398338283":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/4088911225":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3553222249":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3687373115":"2024-04-03T09:14:11.230784+0000","172.21.15.113:6835/703852199":"2024-04-03T09:14:13.967181+0000","172.21.15.89:0/1998199546":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/492795889":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/1337615482":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3750385348":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/2067457213":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/1275405725":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/3355557522":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/4204293224":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/176419408":"2024-04-03T09:14:02.049396+0000","172.21.15.89:6833/108855":"2024-04-03T09:14:02.049396+0000","172.21.15.89:6832/108855":"2024-04-03T09:14:02.049396+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:14:17.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.728+0000 7fd686400700 1 -- 172.21.15.89:0/2943940639 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd66006aed0 msgr2=0x7fd66006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:17.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.728+0000 7fd686400700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd66006aed0 0x7fd66006d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd670000c00 tx=0x7fd67001c040 comp rx=0 tx=0).stop 2024-04-02T09:14:17.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.728+0000 7fd686400700 1 -- 172.21.15.89:0/2943940639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd680161590 msgr2=0x7fd680157030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:17.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.728+0000 7fd686400700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd680161590 0x7fd680157030 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7fd67800caa0 tx=0x7fd67800ce60 comp rx=0 tx=0).stop 2024-04-02T09:14:17.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.728+0000 7fd686400700 1 -- 172.21.15.89:0/2943940639 shutdown_connections 2024-04-02T09:14:17.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.728+0000 7fd686400700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd680161590 0x7fd680157030 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.728+0000 7fd686400700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6800cf230 0x7fd680155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.728+0000 7fd686400700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd66006aed0 0x7fd66006d390 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.728+0000 7fd686400700 1 --2- 172.21.15.89:0/2943940639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd680160f10 0x7fd6800c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.728+0000 7fd686400700 1 -- 172.21.15.89:0/2943940639 >> 172.21.15.89:0/2943940639 conn(0x7fd68008aff0 msgr2=0x7fd6800cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:17.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.728+0000 7fd686400700 1 -- 172.21.15.89:0/2943940639 shutdown_connections 2024-04-02T09:14:17.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.728+0000 7fd686400700 1 -- 172.21.15.89:0/2943940639 wait complete. 2024-04-02T09:14:17.745 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-02T09:13:40.048039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-02T09:13:40.907706+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:14:17.746 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:14:17.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.940+0000 7f86d8200700 1 Processor -- start 2024-04-02T09:14:17.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.940+0000 7f86d8200700 1 -- start start 2024-04-02T09:14:17.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.941+0000 7f86d8200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f86d01611b0 0x7f86d015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:17.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.941+0000 7f86d8200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f86d015f4d0 0x7f86d00ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:17.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.941+0000 7f86d8200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d00d21f0 0x7f86d00cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:17.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.941+0000 7f86d8200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d00cf180 con 0x7f86d01611b0 2024-04-02T09:14:17.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.941+0000 7f86d8200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d00ce270 con 0x7f86d015f4d0 2024-04-02T09:14:17.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.941+0000 7f86d8200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d015b600 con 0x7f86d00d21f0 2024-04-02T09:14:17.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.941+0000 7f86d5800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f86d01611b0 0x7f86d015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:17.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.941+0000 7f86d5800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f86d01611b0 0x7f86d015ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50098/0 (socket says 172.21.15.89:50098) 2024-04-02T09:14:17.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.941+0000 7f86d5800700 1 -- 172.21.15.89:0/1785086234 learned_addr learned my addr 172.21.15.89:0/1785086234 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:17.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.942+0000 7f86d4e00700 1 --2- 172.21.15.89:0/1785086234 >> v2:172.21.15.113:3300/0 conn(0x7f86d015f4d0 0x7f86d00ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:17.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.942+0000 7f86d5800700 1 -- 172.21.15.89:0/1785086234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d00d21f0 msgr2=0x7f86d00cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:17.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.942+0000 7f86d5800700 1 --2- 172.21.15.89:0/1785086234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d00d21f0 0x7f86d00cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.942+0000 7f86d5800700 1 -- 172.21.15.89:0/1785086234 >> v2:172.21.15.113:3300/0 conn(0x7f86d015f4d0 msgr2=0x7f86d00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:17.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.942+0000 7f86d5800700 1 --2- 172.21.15.89:0/1785086234 >> v2:172.21.15.113:3300/0 conn(0x7f86d015f4d0 0x7f86d00ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.942+0000 7f86d5800700 1 -- 172.21.15.89:0/1785086234 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d00d2760 con 0x7f86d01611b0 2024-04-02T09:14:17.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.942+0000 7f86d4e00700 1 --2- 172.21.15.89:0/1785086234 >> v2:172.21.15.113:3300/0 conn(0x7f86d015f4d0 0x7f86d00ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:17.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.942+0000 7f86d5800700 1 --2- 172.21.15.89:0/1785086234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d01611b0 0x7f86d015ef90 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f86d008c030 tx=0x7f86c4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989ea1dc4ab10755 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:17.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.942+0000 7f86cfe00700 1 -- 172.21.15.89:0/1785086234 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c400b870 con 0x7f86d01611b0 2024-04-02T09:14:17.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.942+0000 7f86cfe00700 1 -- 172.21.15.89:0/1785086234 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c400b9d0 con 0x7f86d01611b0 2024-04-02T09:14:17.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.942+0000 7f86cfe00700 1 -- 172.21.15.89:0/1785086234 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c400bd60 con 0x7f86d01611b0 2024-04-02T09:14:17.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.943+0000 7f86d8200700 1 -- 172.21.15.89:0/1785086234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d01611b0 msgr2=0x7f86d015ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:17.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.943+0000 7f86d8200700 1 --2- 172.21.15.89:0/1785086234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d01611b0 0x7f86d015ef90 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f86d008c030 tx=0x7f86c4009bd0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.943+0000 7f86d8200700 1 -- 172.21.15.89:0/1785086234 shutdown_connections 2024-04-02T09:14:17.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.943+0000 7f86d8200700 1 --2- 172.21.15.89:0/1785086234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d01611b0 0x7f86d015ef90 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.943+0000 7f86d8200700 1 --2- 172.21.15.89:0/1785086234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d00d21f0 0x7f86d00cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.943+0000 7f86d8200700 1 --2- 172.21.15.89:0/1785086234 >> v2:172.21.15.113:3300/0 conn(0x7f86d015f4d0 0x7f86d00ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.943+0000 7f86d8200700 1 -- 172.21.15.89:0/1785086234 >> 172.21.15.89:0/1785086234 conn(0x7f86d008aff0 msgr2=0x7f86d00d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:17.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.943+0000 7f86d8200700 1 -- 172.21.15.89:0/1785086234 shutdown_connections 2024-04-02T09:14:17.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.943+0000 7f86d8200700 1 -- 172.21.15.89:0/1785086234 wait complete. 2024-04-02T09:14:17.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.944+0000 7f86d8200700 1 Processor -- start 2024-04-02T09:14:17.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.945+0000 7f86d8200700 1 -- start start 2024-04-02T09:14:17.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.945+0000 7f86d8200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d00d21f0 0x7f86d00a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:17.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.945+0000 7f86d8200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86d015f4d0 0x7f86d00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:17.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.945+0000 7f86d8200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d01611b0 0x7f86d0159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:17.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.945+0000 7f86d8200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d009d6f0 con 0x7f86d01611b0 2024-04-02T09:14:17.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.945+0000 7f86d8200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d00cd4e0 con 0x7f86d015f4d0 2024-04-02T09:14:17.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.945+0000 7f86d8200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d009d350 con 0x7f86d00d21f0 2024-04-02T09:14:17.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.946+0000 7f86d6200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d01611b0 0x7f86d0159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:17.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.946+0000 7f86d6200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d01611b0 0x7f86d0159a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50114/0 (socket says 172.21.15.89:50114) 2024-04-02T09:14:17.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.946+0000 7f86d6200700 1 -- 172.21.15.89:0/3025834454 learned_addr learned my addr 172.21.15.89:0/3025834454 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:17.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.946+0000 7f86d5800700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d00d21f0 0x7f86d00a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:17.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.946+0000 7f86d4e00700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86d015f4d0 0x7f86d00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:17.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.946+0000 7f86d6200700 1 -- 172.21.15.89:0/3025834454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d00d21f0 msgr2=0x7f86d00a0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:17.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.946+0000 7f86d6200700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d00d21f0 0x7f86d00a0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.946+0000 7f86d6200700 1 -- 172.21.15.89:0/3025834454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86d015f4d0 msgr2=0x7f86d00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:17.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.946+0000 7f86d6200700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86d015f4d0 0x7f86d00cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:17.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.946+0000 7f86d6200700 1 -- 172.21.15.89:0/3025834454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c4009550 con 0x7f86d01611b0 2024-04-02T09:14:17.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.946+0000 7f86d4e00700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86d015f4d0 0x7f86d00cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:17.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.946+0000 7f86d6200700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d01611b0 0x7f86d0159a70 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f86c800b970 tx=0x7f86c800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:17.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.947+0000 7f86cd600700 1 -- 172.21.15.89:0/3025834454 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c800c930 con 0x7f86d01611b0 2024-04-02T09:14:17.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.947+0000 7f86cd600700 1 -- 172.21.15.89:0/3025834454 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c800cab0 con 0x7f86d01611b0 2024-04-02T09:14:17.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.947+0000 7f86cd600700 1 -- 172.21.15.89:0/3025834454 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c800ce60 con 0x7f86d01611b0 2024-04-02T09:14:17.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.947+0000 7f86d8200700 1 -- 172.21.15.89:0/3025834454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d00ccf20 con 0x7f86d01611b0 2024-04-02T09:14:17.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.947+0000 7f86d8200700 1 -- 172.21.15.89:0/3025834454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d00a0fa0 con 0x7f86d01611b0 2024-04-02T09:14:17.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.948+0000 7f86d8200700 1 -- 172.21.15.89:0/3025834454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d00a1190 con 0x7f86d01611b0 2024-04-02T09:14:17.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.949+0000 7f86cd600700 1 -- 172.21.15.89:0/3025834454 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f86c801d3b0 con 0x7f86d01611b0 2024-04-02T09:14:17.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.949+0000 7f86cd600700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f86b006aed0 0x7f86b006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:17.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.950+0000 7f86cd600700 1 -- 172.21.15.89:0/3025834454 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f86c8006e40 con 0x7f86d01611b0 2024-04-02T09:14:17.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.950+0000 7f86d5800700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f86b006aed0 0x7f86b006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:17.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.950+0000 7f86d5800700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f86b006aed0 0x7f86b006d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f86c4014040 tx=0x7f86c4008030 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:17.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:17.954+0000 7f86cd600700 1 -- 172.21.15.89:0/3025834454 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f86c8054270 con 0x7f86d01611b0 2024-04-02T09:14:18.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.154+0000 7f86d8200700 1 -- 172.21.15.89:0/3025834454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f86d009d830 con 0x7f86d01611b0 2024-04-02T09:14:18.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.154+0000 7f86cd600700 1 -- 172.21.15.89:0/3025834454 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v253) v1 ==== 93+0+10 (secure 0 0 0) 0x7f86d009d830 con 0x7f86d01611b0 2024-04-02T09:14:18.154 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:14:18.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.157+0000 7f86d8200700 1 -- 172.21.15.89:0/3025834454 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f86b006aed0 msgr2=0x7f86b006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:18.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.157+0000 7f86d8200700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f86b006aed0 0x7f86b006d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f86c4014040 tx=0x7f86c4008030 comp rx=0 tx=0).stop 2024-04-02T09:14:18.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.157+0000 7f86d8200700 1 -- 172.21.15.89:0/3025834454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d01611b0 msgr2=0x7f86d0159a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:18.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.157+0000 7f86d8200700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d01611b0 0x7f86d0159a70 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f86c800b970 tx=0x7f86c800bc80 comp rx=0 tx=0).stop 2024-04-02T09:14:18.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.157+0000 7f86d8200700 1 -- 172.21.15.89:0/3025834454 shutdown_connections 2024-04-02T09:14:18.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.157+0000 7f86d8200700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d01611b0 0x7f86d0159a70 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.157+0000 7f86d8200700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d00d21f0 0x7f86d00a0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.157+0000 7f86d8200700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f86b006aed0 0x7f86b006d390 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.157+0000 7f86d8200700 1 --2- 172.21.15.89:0/3025834454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86d015f4d0 0x7f86d00cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.157+0000 7f86d8200700 1 -- 172.21.15.89:0/3025834454 >> 172.21.15.89:0/3025834454 conn(0x7f86d008aff0 msgr2=0x7f86d0161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:18.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.157+0000 7f86d8200700 1 -- 172.21.15.89:0/3025834454 shutdown_connections 2024-04-02T09:14:18.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.157+0000 7f86d8200700 1 -- 172.21.15.89:0/3025834454 wait complete. 2024-04-02T09:14:18.174 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-02T09:14:18.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.370+0000 7f6a62a00700 1 Processor -- start 2024-04-02T09:14:18.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.370+0000 7f6a62a00700 1 -- start start 2024-04-02T09:14:18.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.371+0000 7f6a62a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a5c0d21f0 0x7f6a5c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:18.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.371+0000 7f6a62a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6a5c0cec40 0x7f6a5c161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.371+0000 7f6a62a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a5c0ce270 con 0x7f6a5c1611b0 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.371+0000 7f6a62a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a5c15b600 con 0x7f6a5c0cec40 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.371+0000 7f6a62a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c0cf0b0 con 0x7f6a5c0d21f0 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.371+0000 7f6a60a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6a5c0cec40 0x7f6a5c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.371+0000 7f6a60a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6a5c0cec40 0x7f6a5c161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50120/0 (socket says 172.21.15.89:50120) 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.371+0000 7f6a60a00700 1 -- 172.21.15.89:0/3156682962 learned_addr learned my addr 172.21.15.89:0/3156682962 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.371+0000 7f6a5b400700 1 --2- 172.21.15.89:0/3156682962 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a5c0d21f0 0x7f6a5c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.372+0000 7f6a60a00700 1 -- 172.21.15.89:0/3156682962 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a5c0d21f0 msgr2=0x7f6a5c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.372+0000 7f6a60a00700 1 --2- 172.21.15.89:0/3156682962 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a5c0d21f0 0x7f6a5c0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.372+0000 7f6a60a00700 1 -- 172.21.15.89:0/3156682962 >> v1:172.21.15.113:6789/0 conn(0x7f6a5c1611b0 legacy=0x7f6a5c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.372+0000 7f6a60a00700 1 -- 172.21.15.89:0/3156682962 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5c0d2760 con 0x7f6a5c0cec40 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.372+0000 7f6a60a00700 1 --2- 172.21.15.89:0/3156682962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a5c0cec40 0x7f6a5c161590 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009e50 tx=0x7f6a5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e02954cff5f1f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.372+0000 7f6a5aa00700 1 -- 172.21.15.89:0/3156682962 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5400e070 con 0x7f6a5c0cec40 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.372+0000 7f6a5aa00700 1 -- 172.21.15.89:0/3156682962 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5400b9c0 con 0x7f6a5c0cec40 2024-04-02T09:14:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.372+0000 7f6a5aa00700 1 -- 172.21.15.89:0/3156682962 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5400bd70 con 0x7f6a5c0cec40 2024-04-02T09:14:18.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.373+0000 7f6a62a00700 1 -- 172.21.15.89:0/3156682962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a5c0cec40 msgr2=0x7f6a5c161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:18.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.373+0000 7f6a62a00700 1 --2- 172.21.15.89:0/3156682962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a5c0cec40 0x7f6a5c161590 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009e50 tx=0x7f6a5400b330 comp rx=0 tx=0).stop 2024-04-02T09:14:18.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.373+0000 7f6a62a00700 1 -- 172.21.15.89:0/3156682962 shutdown_connections 2024-04-02T09:14:18.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.373+0000 7f6a62a00700 1 --2- 172.21.15.89:0/3156682962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a5c0cec40 0x7f6a5c161590 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.373+0000 7f6a62a00700 1 --2- 172.21.15.89:0/3156682962 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a5c0d21f0 0x7f6a5c0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.373+0000 7f6a62a00700 1 -- 172.21.15.89:0/3156682962 >> 172.21.15.89:0/3156682962 conn(0x7f6a5c08aff0 msgr2=0x7f6a5c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:18.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.373+0000 7f6a62a00700 1 -- 172.21.15.89:0/3156682962 shutdown_connections 2024-04-02T09:14:18.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.373+0000 7f6a62a00700 1 -- 172.21.15.89:0/3156682962 wait complete. 2024-04-02T09:14:18.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.374+0000 7f6a62a00700 1 Processor -- start 2024-04-02T09:14:18.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.375+0000 7f6a62a00700 1 -- start start 2024-04-02T09:14:18.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.375+0000 7f6a62a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a5c0cec40 0x7f6a5c1587c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:18.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.376+0000 7f6a62a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a5c0d21f0 0x7f6a5c156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:18.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.376+0000 7f6a62a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a5c1611b0 0x7f6a5c156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:18.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.376+0000 7f6a62a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c0b8d40 con 0x7f6a5c0cec40 2024-04-02T09:14:18.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.376+0000 7f6a62a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c157380 con 0x7f6a5c1611b0 2024-04-02T09:14:18.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.376+0000 7f6a62a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c0b9160 con 0x7f6a5c0d21f0 2024-04-02T09:14:18.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.376+0000 7f6a5be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a5c0cec40 0x7f6a5c1587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:18.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.376+0000 7f6a5be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a5c0cec40 0x7f6a5c1587c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50122/0 (socket says 172.21.15.89:50122) 2024-04-02T09:14:18.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.376+0000 7f6a5be00700 1 -- 172.21.15.89:0/3119223960 learned_addr learned my addr 172.21.15.89:0/3119223960 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:18.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.376+0000 7f6a5be00700 1 -- 172.21.15.89:0/3119223960 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a5c0d21f0 msgr2=0x7f6a5c156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:18.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.376+0000 7f6a5be00700 1 --2- 172.21.15.89:0/3119223960 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a5c0d21f0 0x7f6a5c156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.376+0000 7f6a5be00700 1 -- 172.21.15.89:0/3119223960 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a5c1611b0 msgr2=0x7f6a5c156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:18.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.376+0000 7f6a5be00700 1 --2- 172.21.15.89:0/3119223960 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a5c1611b0 0x7f6a5c156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.376+0000 7f6a5be00700 1 -- 172.21.15.89:0/3119223960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a54009b30 con 0x7f6a5c0cec40 2024-04-02T09:14:18.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.376+0000 7f6a5be00700 1 --2- 172.21.15.89:0/3119223960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a5c0cec40 0x7f6a5c1587c0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c00c450 tx=0x7f6a4c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:18.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.377+0000 7f6a53e00700 1 -- 172.21.15.89:0/3119223960 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c0052d0 con 0x7f6a5c0cec40 2024-04-02T09:14:18.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.377+0000 7f6a53e00700 1 -- 172.21.15.89:0/3119223960 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4c005430 con 0x7f6a5c0cec40 2024-04-02T09:14:18.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.377+0000 7f6a53e00700 1 -- 172.21.15.89:0/3119223960 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c0038c0 con 0x7f6a5c0cec40 2024-04-02T09:14:18.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.377+0000 7f6a62a00700 1 -- 172.21.15.89:0/3119223960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5c0b80c0 con 0x7f6a5c0cec40 2024-04-02T09:14:18.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.377+0000 7f6a62a00700 1 -- 172.21.15.89:0/3119223960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a5c0b8680 con 0x7f6a5c0cec40 2024-04-02T09:14:18.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.378+0000 7f6a62a00700 1 -- 172.21.15.89:0/3119223960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a5c067720 con 0x7f6a5c0cec40 2024-04-02T09:14:18.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.379+0000 7f6a53e00700 1 -- 172.21.15.89:0/3119223960 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f6a4c003a20 con 0x7f6a5c0cec40 2024-04-02T09:14:18.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.380+0000 7f6a53e00700 1 --2- 172.21.15.89:0/3119223960 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6a3c06aed0 0x7f6a3c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:18.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.380+0000 7f6a53e00700 1 -- 172.21.15.89:0/3119223960 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f6a4c088f50 con 0x7f6a5c0cec40 2024-04-02T09:14:18.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.380+0000 7f6a5b400700 1 --2- 172.21.15.89:0/3119223960 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6a3c06aed0 0x7f6a3c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:18.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.380+0000 7f6a5b400700 1 --2- 172.21.15.89:0/3119223960 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6a3c06aed0 0x7f6a3c06d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c1594f0 tx=0x7f6a48008040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:18.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.384+0000 7f6a53e00700 1 -- 172.21.15.89:0/3119223960 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a4c053970 con 0x7f6a5c0cec40 2024-04-02T09:14:18.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.584+0000 7f6a62a00700 1 -- 172.21.15.89:0/3119223960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6a5c0b5d00 con 0x7f6a5c0cec40 2024-04-02T09:14:18.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.584+0000 7f6a53e00700 1 -- 172.21.15.89:0/3119223960 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v253) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6a4c056f90 con 0x7f6a5c0cec40 2024-04-02T09:14:18.584 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:14:18.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.587+0000 7f6a62a00700 1 -- 172.21.15.89:0/3119223960 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6a3c06aed0 msgr2=0x7f6a3c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:18.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.587+0000 7f6a62a00700 1 --2- 172.21.15.89:0/3119223960 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6a3c06aed0 0x7f6a3c06d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c1594f0 tx=0x7f6a48008040 comp rx=0 tx=0).stop 2024-04-02T09:14:18.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.587+0000 7f6a62a00700 1 -- 172.21.15.89:0/3119223960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a5c0cec40 msgr2=0x7f6a5c1587c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:18.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.587+0000 7f6a62a00700 1 --2- 172.21.15.89:0/3119223960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a5c0cec40 0x7f6a5c1587c0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c00c450 tx=0x7f6a4c00c810 comp rx=0 tx=0).stop 2024-04-02T09:14:18.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.587+0000 7f6a62a00700 1 -- 172.21.15.89:0/3119223960 shutdown_connections 2024-04-02T09:14:18.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.587+0000 7f6a62a00700 1 --2- 172.21.15.89:0/3119223960 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6a3c06aed0 0x7f6a3c06d390 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.587+0000 7f6a62a00700 1 --2- 172.21.15.89:0/3119223960 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a5c1611b0 0x7f6a5c156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.587+0000 7f6a62a00700 1 --2- 172.21.15.89:0/3119223960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a5c0cec40 0x7f6a5c1587c0 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.587+0000 7f6a62a00700 1 --2- 172.21.15.89:0/3119223960 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a5c0d21f0 0x7f6a5c156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.587+0000 7f6a62a00700 1 -- 172.21.15.89:0/3119223960 >> 172.21.15.89:0/3119223960 conn(0x7f6a5c08aff0 msgr2=0x7f6a5c15da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:18.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.587+0000 7f6a62a00700 1 -- 172.21.15.89:0/3119223960 shutdown_connections 2024-04-02T09:14:18.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.587+0000 7f6a62a00700 1 -- 172.21.15.89:0/3119223960 wait complete. 2024-04-02T09:14:18.604 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-02T09:14:18.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.799+0000 7fe0e3a00700 1 Processor -- start 2024-04-02T09:14:18.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.799+0000 7fe0e3a00700 1 -- start start 2024-04-02T09:14:18.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.800+0000 7fe0e3a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0dc0d2540 0x7fe0dc1687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:18.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.800+0000 7fe0e3a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0dc161420 con 0x7fe0dc0d16a0 2024-04-02T09:14:18.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.801+0000 7fe0e3a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0dc161700 con 0x7fe0dc15d9d0 2024-04-02T09:14:18.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.801+0000 7fe0e3a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc0d4340 con 0x7fe0dc0d2540 2024-04-02T09:14:18.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.801+0000 7fe0dbe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe0dc0d16a0 0x7fe0dc0d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56800/0 (socket says 172.21.15.89:56800) 2024-04-02T09:14:18.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.801+0000 7fe0dbe00700 1 -- 172.21.15.89:0/3503215250 learned_addr learned my addr 172.21.15.89:0/3503215250 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:18.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.801+0000 7fe0e1a00700 1 --2- 172.21.15.89:0/3503215250 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0dc0d2540 0x7fe0dc1687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:18.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.801+0000 7fe0db400700 1 -- 172.21.15.89:0/3503215250 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75238225 0 0) 0x7fe0dc161420 con 0x7fe0dc0d16a0 2024-04-02T09:14:18.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.801+0000 7fe0db400700 1 -- 172.21.15.89:0/3503215250 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c40035b0 con 0x7fe0dc0d16a0 2024-04-02T09:14:18.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.801+0000 7fe0db400700 1 -- 172.21.15.89:0/3503215250 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2551150925 0 0) 0x7fe0c40035b0 con 0x7fe0dc0d16a0 2024-04-02T09:14:18.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.801+0000 7fe0db400700 1 -- 172.21.15.89:0/3503215250 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0dc0d2540 msgr2=0x7fe0dc1687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:18.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.801+0000 7fe0db400700 1 --2- 172.21.15.89:0/3503215250 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0dc0d2540 0x7fe0dc1687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0db400700 1 -- 172.21.15.89:0/3503215250 >> v1:172.21.15.113:6789/0 conn(0x7fe0dc15d9d0 legacy=0x7fe0dc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:18.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0db400700 1 -- 172.21.15.89:0/3503215250 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc1619e0 con 0x7fe0dc0d16a0 2024-04-02T09:14:18.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0db400700 1 -- 172.21.15.89:0/3503215250 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fe0c80017f0 con 0x7fe0dc0d16a0 2024-04-02T09:14:18.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0db400700 1 -- 172.21.15.89:0/3503215250 >> v1:172.21.15.89:6789/0 conn(0x7fe0dc0d16a0 legacy=0x7fe0dc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:18.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0db400700 1 --2- 172.21.15.89:0/3503215250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0c4003a30 0x7fe0c4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:18.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0db400700 1 --2- 172.21.15.89:0/3503215250 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0dc0d2540 0x7fe0c4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:18.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0db400700 1 --2- 172.21.15.89:0/3503215250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c4004d90 0x7fe0c4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:18.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0db400700 1 -- 172.21.15.89:0/3503215250 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c400ba10 con 0x7fe0c4003a30 2024-04-02T09:14:18.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0db400700 1 -- 172.21.15.89:0/3503215250 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c400bb80 con 0x7fe0c4004d90 2024-04-02T09:14:18.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0db400700 1 -- 172.21.15.89:0/3503215250 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c400bd40 con 0x7fe0dc0d2540 2024-04-02T09:14:18.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0e1000700 1 --2- 172.21.15.89:0/3503215250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0c4003a30 0x7fe0c4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:18.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0e1a00700 1 --2- 172.21.15.89:0/3503215250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c4004d90 0x7fe0c4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:18.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0dbe00700 1 --2- 172.21.15.89:0/3503215250 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0dc0d2540 0x7fe0c4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:18.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0e1000700 1 -- 172.21.15.89:0/3503215250 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0dc0d2540 msgr2=0x7fe0c4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:18.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0e1000700 1 --2- 172.21.15.89:0/3503215250 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0dc0d2540 0x7fe0c4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0e1000700 1 -- 172.21.15.89:0/3503215250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c4004d90 msgr2=0x7fe0c4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:18.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0e1000700 1 --2- 172.21.15.89:0/3503215250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c4004d90 0x7fe0c4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.802+0000 7fe0e1000700 1 -- 172.21.15.89:0/3503215250 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c40035b0 con 0x7fe0c4003a30 2024-04-02T09:14:18.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.803+0000 7fe0e1000700 1 --2- 172.21.15.89:0/3503215250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0c4003a30 0x7fe0c4003e10 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7fe0d00018d0 tx=0x7fe0c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3db24f4814a676 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:18.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.803+0000 7fe0db400700 1 -- 172.21.15.89:0/3503215250 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d0012620 con 0x7fe0c4003a30 2024-04-02T09:14:18.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.803+0000 7fe0db400700 1 -- 172.21.15.89:0/3503215250 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d00127a0 con 0x7fe0c4003a30 2024-04-02T09:14:18.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.803+0000 7fe0db400700 1 -- 172.21.15.89:0/3503215250 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d0012b50 con 0x7fe0c4003a30 2024-04-02T09:14:18.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.803+0000 7fe0e3a00700 1 -- 172.21.15.89:0/3503215250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0c4003a30 msgr2=0x7fe0c4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:18.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.803+0000 7fe0e3a00700 1 --2- 172.21.15.89:0/3503215250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0c4003a30 0x7fe0c4003e10 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7fe0d00018d0 tx=0x7fe0c4005880 comp rx=0 tx=0).stop 2024-04-02T09:14:18.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.803+0000 7fe0e1a00700 1 -- 172.21.15.89:0/3503215250 reap_dead start 2024-04-02T09:14:18.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.803+0000 7fe0e3a00700 1 -- 172.21.15.89:0/3503215250 shutdown_connections 2024-04-02T09:14:18.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.803+0000 7fe0e3a00700 1 -- 172.21.15.89:0/3503215250 >> 172.21.15.89:0/3503215250 conn(0x7fe0dc08aff0 msgr2=0x7fe0dc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:18.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.803+0000 7fe0e3a00700 1 -- 172.21.15.89:0/3503215250 shutdown_connections 2024-04-02T09:14:18.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.803+0000 7fe0e3a00700 1 -- 172.21.15.89:0/3503215250 wait complete. 2024-04-02T09:14:18.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.805+0000 7fe0e3a00700 1 Processor -- start 2024-04-02T09:14:18.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.805+0000 7fe0e3a00700 1 -- start start 2024-04-02T09:14:18.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.806+0000 7fe0e3a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c4003a30 0x7fe0dc14fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:18.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.806+0000 7fe0e3a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0dc15ddb0 0x7fe0dc15e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:18.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.806+0000 7fe0e3a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0dc0b4350 0x7fe0dc0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:18.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.806+0000 7fe0e3a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc15f400 con 0x7fe0dc15ddb0 2024-04-02T09:14:18.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.806+0000 7fe0e3a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc0b4d10 con 0x7fe0c4003a30 2024-04-02T09:14:18.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.806+0000 7fe0e3a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc15f950 con 0x7fe0dc0b4350 2024-04-02T09:14:18.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.806+0000 7fe0dbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0dc15ddb0 0x7fe0dc15e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:18.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.806+0000 7fe0dbe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0dc15ddb0 0x7fe0dc15e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41686/0 (socket says 172.21.15.89:41686) 2024-04-02T09:14:18.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.806+0000 7fe0dbe00700 1 -- 172.21.15.89:0/919097042 learned_addr learned my addr 172.21.15.89:0/919097042 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:18.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.806+0000 7fe0e1000700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c4003a30 0x7fe0dc14fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:18.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.806+0000 7fe0e1a00700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0dc0b4350 0x7fe0dc0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:18.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.807+0000 7fe0dbe00700 1 -- 172.21.15.89:0/919097042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0dc0b4350 msgr2=0x7fe0dc0b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:18.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.807+0000 7fe0dbe00700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0dc0b4350 0x7fe0dc0b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.807+0000 7fe0dbe00700 1 -- 172.21.15.89:0/919097042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c4003a30 msgr2=0x7fe0dc14fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:18.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.807+0000 7fe0dbe00700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c4003a30 0x7fe0dc14fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:18.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.807+0000 7fe0dbe00700 1 -- 172.21.15.89:0/919097042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc150250 con 0x7fe0dc15ddb0 2024-04-02T09:14:18.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.807+0000 7fe0e1a00700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0dc0b4350 0x7fe0dc0b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:18.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.807+0000 7fe0dbe00700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0dc15ddb0 0x7fe0dc15e230 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7fe0c800e830 tx=0x7fe0c800eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:18.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.807+0000 7fe0d8c00700 1 -- 172.21.15.89:0/919097042 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c80072e0 con 0x7fe0dc15ddb0 2024-04-02T09:14:18.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.807+0000 7fe0d8c00700 1 -- 172.21.15.89:0/919097042 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c8007440 con 0x7fe0dc15ddb0 2024-04-02T09:14:18.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.807+0000 7fe0e3a00700 1 -- 172.21.15.89:0/919097042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0dc1f3550 con 0x7fe0dc15ddb0 2024-04-02T09:14:18.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.807+0000 7fe0d8c00700 1 -- 172.21.15.89:0/919097042 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8005830 con 0x7fe0dc15ddb0 2024-04-02T09:14:18.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.807+0000 7fe0e3a00700 1 -- 172.21.15.89:0/919097042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0dc15eb90 con 0x7fe0dc15ddb0 2024-04-02T09:14:18.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.808+0000 7fe0e3a00700 1 -- 172.21.15.89:0/919097042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0dc0d1ca0 con 0x7fe0dc15ddb0 2024-04-02T09:14:18.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.809+0000 7fe0d8c00700 1 -- 172.21.15.89:0/919097042 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe0c8005990 con 0x7fe0dc15ddb0 2024-04-02T09:14:18.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.810+0000 7fe0d8c00700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe0bc06aed0 0x7fe0bc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:18.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.810+0000 7fe0d8c00700 1 -- 172.21.15.89:0/919097042 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7029+0+0 (secure 0 0 0) 0x7fe0c8089570 con 0x7fe0dc15ddb0 2024-04-02T09:14:18.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.810+0000 7fe0e1000700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe0bc06aed0 0x7fe0bc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:18.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.811+0000 7fe0e1000700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe0bc06aed0 0x7fe0bc06d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0014040 tx=0x7fe0d0005040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:18.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:18.814+0000 7fe0d8c00700 1 -- 172.21.15.89:0/919097042 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe0c80578c0 con 0x7fe0dc15ddb0 2024-04-02T09:14:19.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.016+0000 7fe0e3a00700 1 -- 172.21.15.89:0/919097042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fe0dc15edb0 con 0x7fe0dc15ddb0 2024-04-02T09:14:19.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.017+0000 7fe0d8c00700 1 -- 172.21.15.89:0/919097042 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v253) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe0c8057450 con 0x7fe0dc15ddb0 2024-04-02T09:14:19.016 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:14:19.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.019+0000 7fe0e3a00700 1 -- 172.21.15.89:0/919097042 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe0bc06aed0 msgr2=0x7fe0bc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:19.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.019+0000 7fe0e3a00700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe0bc06aed0 0x7fe0bc06d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0014040 tx=0x7fe0d0005040 comp rx=0 tx=0).stop 2024-04-02T09:14:19.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.019+0000 7fe0e3a00700 1 -- 172.21.15.89:0/919097042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0dc15ddb0 msgr2=0x7fe0dc15e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:19.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.019+0000 7fe0e3a00700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0dc15ddb0 0x7fe0dc15e230 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7fe0c800e830 tx=0x7fe0c800eb40 comp rx=0 tx=0).stop 2024-04-02T09:14:19.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.019+0000 7fe0e3a00700 1 -- 172.21.15.89:0/919097042 shutdown_connections 2024-04-02T09:14:19.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.019+0000 7fe0e3a00700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe0bc06aed0 0x7fe0bc06d390 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.019+0000 7fe0e3a00700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0c4003a30 0x7fe0dc14fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.019+0000 7fe0e3a00700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe0dc0b4350 0x7fe0dc0b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.019+0000 7fe0e3a00700 1 --2- 172.21.15.89:0/919097042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0dc15ddb0 0x7fe0dc15e230 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.019+0000 7fe0e3a00700 1 -- 172.21.15.89:0/919097042 >> 172.21.15.89:0/919097042 conn(0x7fe0dc08aff0 msgr2=0x7fe0dc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:19.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.019+0000 7fe0e3a00700 1 -- 172.21.15.89:0/919097042 shutdown_connections 2024-04-02T09:14:19.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.020+0000 7fe0e3a00700 1 -- 172.21.15.89:0/919097042 wait complete. 2024-04-02T09:14:19.036 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:14:19.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.227+0000 7f0cf7e00700 1 Processor -- start 2024-04-02T09:14:19.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.227+0000 7f0cf7e00700 1 -- start start 2024-04-02T09:14:19.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.228+0000 7f0cf7e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0cf0160be0 0x7f0cf00cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.228+0000 7f0cf7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0cf00ce0e0 0x7f0cf00ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.228+0000 7f0cf7e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0cf00cead0 0x7f0cf0161240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.228+0000 7f0cf7e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf00cef60 con 0x7f0cf00ce0e0 2024-04-02T09:14:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.229+0000 7f0cf7e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf00b65d0 con 0x7f0cf0160be0 2024-04-02T09:14:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.229+0000 7f0cf7e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf00d2a80 con 0x7f0cf00cead0 2024-04-02T09:14:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.229+0000 7f0cf4a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0cf00ce0e0 0x7f0cf00ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.229+0000 7f0cf4a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0cf00ce0e0 0x7f0cf00ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41702/0 (socket says 172.21.15.89:41702) 2024-04-02T09:14:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.229+0000 7f0cf4a00700 1 -- 172.21.15.89:0/3698750771 learned_addr learned my addr 172.21.15.89:0/3698750771 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.229+0000 7f0cf5400700 1 --2- 172.21.15.89:0/3698750771 >> v2:172.21.15.113:3300/0 conn(0x7f0cf0160be0 0x7f0cf00cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.229+0000 7f0cf5e00700 1 --2- 172.21.15.89:0/3698750771 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0cf00cead0 0x7f0cf0161240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.229+0000 7f0cf4a00700 1 -- 172.21.15.89:0/3698750771 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0cf00cead0 msgr2=0x7f0cf0161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.229+0000 7f0cf4a00700 1 --2- 172.21.15.89:0/3698750771 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0cf00cead0 0x7f0cf0161240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.229+0000 7f0cf4a00700 1 -- 172.21.15.89:0/3698750771 >> v2:172.21.15.113:3300/0 conn(0x7f0cf0160be0 msgr2=0x7f0cf00cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:19.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.229+0000 7f0cf4a00700 1 --2- 172.21.15.89:0/3698750771 >> v2:172.21.15.113:3300/0 conn(0x7f0cf0160be0 0x7f0cf00cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.229+0000 7f0cf4a00700 1 -- 172.21.15.89:0/3698750771 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf0160fc0 con 0x7f0cf00ce0e0 2024-04-02T09:14:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.229+0000 7f0cf5400700 1 --2- 172.21.15.89:0/3698750771 >> v2:172.21.15.113:3300/0 conn(0x7f0cf0160be0 0x7f0cf00cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.230+0000 7f0cf4a00700 1 --2- 172.21.15.89:0/3698750771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf00ce0e0 0x7f0cf00ce590 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f0cf008b590 tx=0x7f0cdc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b450c21f760bd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.230+0000 7f0cefe00700 1 -- 172.21.15.89:0/3698750771 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc00b860 con 0x7f0cf00ce0e0 2024-04-02T09:14:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.230+0000 7f0cefe00700 1 -- 172.21.15.89:0/3698750771 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cdc00b9c0 con 0x7f0cf00ce0e0 2024-04-02T09:14:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.230+0000 7f0cefe00700 1 -- 172.21.15.89:0/3698750771 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc00bd50 con 0x7f0cf00ce0e0 2024-04-02T09:14:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.230+0000 7f0cf7e00700 1 -- 172.21.15.89:0/3698750771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf00ce0e0 msgr2=0x7f0cf00ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.230+0000 7f0cf7e00700 1 --2- 172.21.15.89:0/3698750771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf00ce0e0 0x7f0cf00ce590 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f0cf008b590 tx=0x7f0cdc009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.231+0000 7f0cf7e00700 1 -- 172.21.15.89:0/3698750771 shutdown_connections 2024-04-02T09:14:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.231+0000 7f0cf7e00700 1 --2- 172.21.15.89:0/3698750771 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0cf00cead0 0x7f0cf0161240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.231+0000 7f0cf7e00700 1 --2- 172.21.15.89:0/3698750771 >> v2:172.21.15.113:3300/0 conn(0x7f0cf0160be0 0x7f0cf00cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.231+0000 7f0cf7e00700 1 --2- 172.21.15.89:0/3698750771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf00ce0e0 0x7f0cf00ce590 unknown :-1 s=CLOSED pgs=2731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.231+0000 7f0cf7e00700 1 -- 172.21.15.89:0/3698750771 >> 172.21.15.89:0/3698750771 conn(0x7f0cf008aff0 msgr2=0x7f0cf00d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:19.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.231+0000 7f0cf7e00700 1 -- 172.21.15.89:0/3698750771 shutdown_connections 2024-04-02T09:14:19.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.231+0000 7f0cf7e00700 1 -- 172.21.15.89:0/3698750771 wait complete. 2024-04-02T09:14:19.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.233+0000 7f0cf7e00700 1 Processor -- start 2024-04-02T09:14:19.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.233+0000 7f0cf7e00700 1 -- start start 2024-04-02T09:14:19.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.234+0000 7f0cf7e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0cf00ce0e0 0x7f0cf00c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:19.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.234+0000 7f0cf7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf00cead0 0x7f0cf01f6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:19.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.234+0000 7f0cf7e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cf0160be0 0x7f0cf00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:19.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.234+0000 7f0cf7e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf00ca620 con 0x7f0cf00cead0 2024-04-02T09:14:19.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.234+0000 7f0cf7e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf01f6c60 con 0x7f0cf0160be0 2024-04-02T09:14:19.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.234+0000 7f0cf7e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf01f6ee0 con 0x7f0cf00ce0e0 2024-04-02T09:14:19.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.235+0000 7f0cf4a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf00cead0 0x7f0cf01f6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:19.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.235+0000 7f0cf4a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf00cead0 0x7f0cf01f6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41718/0 (socket says 172.21.15.89:41718) 2024-04-02T09:14:19.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.235+0000 7f0cf4a00700 1 -- 172.21.15.89:0/39001049 learned_addr learned my addr 172.21.15.89:0/39001049 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:19.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.235+0000 7f0cf5400700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0cf00ce0e0 0x7f0cf00c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:19.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.235+0000 7f0cf5e00700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cf0160be0 0x7f0cf00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:19.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.235+0000 7f0cf4a00700 1 -- 172.21.15.89:0/39001049 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0cf00ce0e0 msgr2=0x7f0cf00c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:19.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.235+0000 7f0cf4a00700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0cf00ce0e0 0x7f0cf00c9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.235+0000 7f0cf4a00700 1 -- 172.21.15.89:0/39001049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cf0160be0 msgr2=0x7f0cf00b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:19.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.235+0000 7f0cf4a00700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cf0160be0 0x7f0cf00b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.235+0000 7f0cf4a00700 1 -- 172.21.15.89:0/39001049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc009550 con 0x7f0cf00cead0 2024-04-02T09:14:19.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.235+0000 7f0cf5e00700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cf0160be0 0x7f0cf00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:19.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.235+0000 7f0cf4a00700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf00cead0 0x7f0cf01f6720 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009130 tx=0x7f0cdc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:19.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.237+0000 7f0ced600700 1 -- 172.21.15.89:0/39001049 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc004450 con 0x7f0cf00cead0 2024-04-02T09:14:19.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.237+0000 7f0ced600700 1 -- 172.21.15.89:0/39001049 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cdc01e070 con 0x7f0cf00cead0 2024-04-02T09:14:19.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.237+0000 7f0cf7e00700 1 -- 172.21.15.89:0/39001049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf01f71c0 con 0x7f0cf00cead0 2024-04-02T09:14:19.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.237+0000 7f0ced600700 1 -- 172.21.15.89:0/39001049 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc0038e0 con 0x7f0cf00cead0 2024-04-02T09:14:19.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.237+0000 7f0cf7e00700 1 -- 172.21.15.89:0/39001049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf00ca870 con 0x7f0cf00cead0 2024-04-02T09:14:19.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.238+0000 7f0cf7e00700 1 -- 172.21.15.89:0/39001049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf00caa90 con 0x7f0cf00cead0 2024-04-02T09:14:19.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.239+0000 7f0ced600700 1 -- 172.21.15.89:0/39001049 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0cdc003a40 con 0x7f0cf00cead0 2024-04-02T09:14:19.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.240+0000 7f0ced600700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0cd006aed0 0x7f0cd006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:19.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.240+0000 7f0ced600700 1 -- 172.21.15.89:0/39001049 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f0cdc020070 con 0x7f0cf00cead0 2024-04-02T09:14:19.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.240+0000 7f0cf5400700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0cd006aed0 0x7f0cd006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:19.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.241+0000 7f0cf5400700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0cd006aed0 0x7f0cd006d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40099d0 tx=0x7f0ce4008040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:19.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.245+0000 7f0ced600700 1 -- 172.21.15.89:0/39001049 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0cdc055940 con 0x7f0cf00cead0 2024-04-02T09:14:19.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.492+0000 7f0cf7e00700 1 -- 172.21.15.89:0/39001049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cf0067720 con 0x7f0cf00cead0 2024-04-02T09:14:19.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.493+0000 7f0ced600700 1 -- 172.21.15.89:0/39001049 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+4757 (secure 0 0 0) 0x7f0cdc02a090 con 0x7f0cf00cead0 2024-04-02T09:14:19.492 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:19.492 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":127,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9518,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3094781297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3094781297},{"type":"v1","addr":"172.21.15.89:6836","nonce":3094781297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":9920,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1284027162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1284027162},{"type":"v1","addr":"172.21.15.113:6835","nonce":1284027162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":10978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1799351491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1799351491},{"type":"v1","addr":"172.21.15.113:6834","nonce":1799351491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:41.933360+0000","modified":"2024-04-02T09:13:45.439462+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11958},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11958":{"gid":11958,"name":"a","rank":0,"incarnation":121,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6837/3112580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3112580185},{"type":"v1","addr":"172.21.15.89:6837","nonce":3112580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-02T09:14:19.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.495+0000 7f0cf7e00700 1 -- 172.21.15.89:0/39001049 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0cd006aed0 msgr2=0x7f0cd006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:19.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.495+0000 7f0cf7e00700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0cd006aed0 0x7f0cd006d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40099d0 tx=0x7f0ce4008040 comp rx=0 tx=0).stop 2024-04-02T09:14:19.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.495+0000 7f0cf7e00700 1 -- 172.21.15.89:0/39001049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf00cead0 msgr2=0x7f0cf01f6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:19.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.495+0000 7f0cf7e00700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf00cead0 0x7f0cf01f6720 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009130 tx=0x7f0cdc004750 comp rx=0 tx=0).stop 2024-04-02T09:14:19.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.495+0000 7f0cf7e00700 1 -- 172.21.15.89:0/39001049 shutdown_connections 2024-04-02T09:14:19.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.495+0000 7f0cf7e00700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0cd006aed0 0x7f0cd006d390 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.495+0000 7f0cf7e00700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cf0160be0 0x7f0cf00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.495+0000 7f0cf7e00700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0cf00ce0e0 0x7f0cf00c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.495+0000 7f0cf7e00700 1 --2- 172.21.15.89:0/39001049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf00cead0 0x7f0cf01f6720 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.496+0000 7f0cf7e00700 1 -- 172.21.15.89:0/39001049 >> 172.21.15.89:0/39001049 conn(0x7f0cf008aff0 msgr2=0x7f0cf00b89f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.496+0000 7f0cf7e00700 1 -- 172.21.15.89:0/39001049 shutdown_connections 2024-04-02T09:14:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.496+0000 7f0cf7e00700 1 -- 172.21.15.89:0/39001049 wait complete. 2024-04-02T09:14:19.496 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 127 2024-04-02T09:14:19.512 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:14:19.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.703+0000 7f76b5400700 1 Processor -- start 2024-04-02T09:14:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.704+0000 7f76b5400700 1 -- start start 2024-04-02T09:14:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.705+0000 7f76b5400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f76b00b3af0 0x7f76b015a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.705+0000 7f76b5400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f76b0072580 0x7f76b00b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.705+0000 7f76b5400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76b00b2e50 0x7f76b00709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.705+0000 7f76b5400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76b00b2180 con 0x7f76b0072580 2024-04-02T09:14:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.705+0000 7f76b5400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76b015b100 con 0x7f76b00b3af0 2024-04-02T09:14:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.705+0000 7f76b5400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b015b3e0 con 0x7f76b00b2e50 2024-04-02T09:14:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.705+0000 7f76a7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f76b0072580 0x7f76b00b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.705+0000 7f76a7e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f76b0072580 0x7f76b00b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41724/0 (socket says 172.21.15.89:41724) 2024-04-02T09:14:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.705+0000 7f76a7e00700 1 -- 172.21.15.89:0/816792694 learned_addr learned my addr 172.21.15.89:0/816792694 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.705+0000 7f76ae800700 1 --2- 172.21.15.89:0/816792694 >> v2:172.21.15.113:3300/0 conn(0x7f76b00b3af0 0x7f76b015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.705+0000 7f76af200700 1 --2- 172.21.15.89:0/816792694 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76b00b2e50 0x7f76b00709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.706+0000 7f76a7e00700 1 -- 172.21.15.89:0/816792694 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76b00b2e50 msgr2=0x7f76b00709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.706+0000 7f76a7e00700 1 --2- 172.21.15.89:0/816792694 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76b00b2e50 0x7f76b00709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.706+0000 7f76a7e00700 1 -- 172.21.15.89:0/816792694 >> v2:172.21.15.113:3300/0 conn(0x7f76b00b3af0 msgr2=0x7f76b015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.706+0000 7f76a7e00700 1 --2- 172.21.15.89:0/816792694 >> v2:172.21.15.113:3300/0 conn(0x7f76b00b3af0 0x7f76b015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.706+0000 7f76a7e00700 1 -- 172.21.15.89:0/816792694 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b00b23f0 con 0x7f76b0072580 2024-04-02T09:14:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.706+0000 7f76ae800700 1 --2- 172.21.15.89:0/816792694 >> v2:172.21.15.113:3300/0 conn(0x7f76b00b3af0 0x7f76b015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.706+0000 7f76a7e00700 1 --2- 172.21.15.89:0/816792694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b0072580 0x7f76b00b2910 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f76b006b860 tx=0x7f7698009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d76be16279883d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.706+0000 7f76ade00700 1 -- 172.21.15.89:0/816792694 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769800b870 con 0x7f76b0072580 2024-04-02T09:14:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.706+0000 7f76ade00700 1 -- 172.21.15.89:0/816792694 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769800b9d0 con 0x7f76b0072580 2024-04-02T09:14:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.706+0000 7f76ade00700 1 -- 172.21.15.89:0/816792694 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769800bd60 con 0x7f76b0072580 2024-04-02T09:14:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.706+0000 7f76b5400700 1 -- 172.21.15.89:0/816792694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b0072580 msgr2=0x7f76b00b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.706+0000 7f76b5400700 1 --2- 172.21.15.89:0/816792694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b0072580 0x7f76b00b2910 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f76b006b860 tx=0x7f7698009bd0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.707+0000 7f76b5400700 1 -- 172.21.15.89:0/816792694 shutdown_connections 2024-04-02T09:14:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.707+0000 7f76b5400700 1 --2- 172.21.15.89:0/816792694 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76b00b2e50 0x7f76b00709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.707+0000 7f76b5400700 1 --2- 172.21.15.89:0/816792694 >> v2:172.21.15.113:3300/0 conn(0x7f76b00b3af0 0x7f76b015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.707+0000 7f76b5400700 1 --2- 172.21.15.89:0/816792694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b0072580 0x7f76b00b2910 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.707+0000 7f76b5400700 1 -- 172.21.15.89:0/816792694 >> 172.21.15.89:0/816792694 conn(0x7f76b00cf190 msgr2=0x7f76b00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:19.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.707+0000 7f76b5400700 1 -- 172.21.15.89:0/816792694 shutdown_connections 2024-04-02T09:14:19.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.707+0000 7f76b5400700 1 -- 172.21.15.89:0/816792694 wait complete. 2024-04-02T09:14:19.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.708+0000 7f76b5400700 1 Processor -- start 2024-04-02T09:14:19.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.709+0000 7f76b5400700 1 -- start start 2024-04-02T09:14:19.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.709+0000 7f76b5400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76b0072580 0x7f76b015b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:19.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76b5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b00b2e50 0x7f76b01475b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:19.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76b5400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76b00b3af0 0x7f76b015d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:19.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76b5400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b015c070 con 0x7f76b00b2e50 2024-04-02T09:14:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76b5400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b015c350 con 0x7f76b00b3af0 2024-04-02T09:14:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76b5400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b015c630 con 0x7f76b0072580 2024-04-02T09:14:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76a7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b00b2e50 0x7f76b01475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76a7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b00b2e50 0x7f76b01475b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41728/0 (socket says 172.21.15.89:41728) 2024-04-02T09:14:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76ae800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76b0072580 0x7f76b015b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76a7e00700 1 -- 172.21.15.89:0/4162666357 learned_addr learned my addr 172.21.15.89:0/4162666357 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76af200700 1 --2- 172.21.15.89:0/4162666357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76b00b3af0 0x7f76b015d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76a7e00700 1 -- 172.21.15.89:0/4162666357 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76b0072580 msgr2=0x7f76b015b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76a7e00700 1 --2- 172.21.15.89:0/4162666357 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76b0072580 0x7f76b015b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76a7e00700 1 -- 172.21.15.89:0/4162666357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76b00b3af0 msgr2=0x7f76b015d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76a7e00700 1 --2- 172.21.15.89:0/4162666357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76b00b3af0 0x7f76b015d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76a7e00700 1 -- 172.21.15.89:0/4162666357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7698009550 con 0x7f76b00b2e50 2024-04-02T09:14:19.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76af200700 1 --2- 172.21.15.89:0/4162666357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76b00b3af0 0x7f76b015d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:19.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.710+0000 7f76a7e00700 1 --2- 172.21.15.89:0/4162666357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b00b2e50 0x7f76b01475b0 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f7698000c00 tx=0x7f7698004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:19.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.711+0000 7f76a6a00700 1 -- 172.21.15.89:0/4162666357 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7698004520 con 0x7f76b00b2e50 2024-04-02T09:14:19.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.711+0000 7f76a6a00700 1 -- 172.21.15.89:0/4162666357 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769801e070 con 0x7f76b00b2e50 2024-04-02T09:14:19.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.711+0000 7f76b5400700 1 -- 172.21.15.89:0/4162666357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b015c910 con 0x7f76b00b2e50 2024-04-02T09:14:19.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.711+0000 7f76a6a00700 1 -- 172.21.15.89:0/4162666357 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76980038c0 con 0x7f76b00b2e50 2024-04-02T09:14:19.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.711+0000 7f76b5400700 1 -- 172.21.15.89:0/4162666357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b015bd80 con 0x7f76b00b2e50 2024-04-02T09:14:19.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.712+0000 7f76b5400700 1 -- 172.21.15.89:0/4162666357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b015ce30 con 0x7f76b00b2e50 2024-04-02T09:14:19.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.713+0000 7f76a6a00700 1 -- 172.21.15.89:0/4162666357 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f7698003a20 con 0x7f76b00b2e50 2024-04-02T09:14:19.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.713+0000 7f76a6a00700 1 --2- 172.21.15.89:0/4162666357 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f768c06aed0 0x7f768c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:19.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.714+0000 7f76a6a00700 1 -- 172.21.15.89:0/4162666357 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f7698020070 con 0x7f76b00b2e50 2024-04-02T09:14:19.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.714+0000 7f76ae800700 1 --2- 172.21.15.89:0/4162666357 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f768c06aed0 0x7f768c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:19.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.714+0000 7f76ae800700 1 --2- 172.21.15.89:0/4162666357 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f768c06aed0 0x7f768c06d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f76a0006f10 tx=0x7f76a0008040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:19.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.718+0000 7f76a6a00700 1 -- 172.21.15.89:0/4162666357 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f76b015ce30 con 0x7f76b00b2e50 2024-04-02T09:14:19.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.919+0000 7f76b5400700 1 -- 172.21.15.89:0/4162666357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76b015ce30 con 0x7f76b00b2e50 2024-04-02T09:14:19.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.920+0000 7f76a6a00700 1 -- 172.21.15.89:0/4162666357 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) v1 ==== 74+0+17997 (secure 0 0 0) 0x7f76b015ce30 con 0x7f76b00b2e50 2024-04-02T09:14:19.920 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:19.920 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":253,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:14:17.272062+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:13:40.048039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:13:40.907706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.113:6833/703852199":"2024-04-03T09:14:13.967181+0000","172.21.15.89:0/3750432255":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/2974789087":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/1231626446":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/2398338283":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/4088911225":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3553222249":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3687373115":"2024-04-03T09:14:11.230784+0000","172.21.15.113:6835/703852199":"2024-04-03T09:14:13.967181+0000","172.21.15.89:0/1998199546":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/492795889":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/1337615482":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3750385348":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/2067457213":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/1275405725":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/3355557522":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/4204293224":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/176419408":"2024-04-03T09:14:02.049396+0000","172.21.15.89:6833/108855":"2024-04-03T09:14:02.049396+0000","172.21.15.89:6832/108855":"2024-04-03T09:14:02.049396+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:14:19.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.923+0000 7f76b5400700 1 -- 172.21.15.89:0/4162666357 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f768c06aed0 msgr2=0x7f768c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:19.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.923+0000 7f76b5400700 1 --2- 172.21.15.89:0/4162666357 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f768c06aed0 0x7f768c06d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f76a0006f10 tx=0x7f76a0008040 comp rx=0 tx=0).stop 2024-04-02T09:14:19.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.923+0000 7f76b5400700 1 -- 172.21.15.89:0/4162666357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b00b2e50 msgr2=0x7f76b01475b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:19.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.923+0000 7f76b5400700 1 --2- 172.21.15.89:0/4162666357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b00b2e50 0x7f76b01475b0 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f7698000c00 tx=0x7f7698004750 comp rx=0 tx=0).stop 2024-04-02T09:14:19.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.923+0000 7f76b5400700 1 -- 172.21.15.89:0/4162666357 shutdown_connections 2024-04-02T09:14:19.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.923+0000 7f76b5400700 1 --2- 172.21.15.89:0/4162666357 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f768c06aed0 0x7f768c06d390 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.923+0000 7f76b5400700 1 --2- 172.21.15.89:0/4162666357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76b00b3af0 0x7f76b015d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.923+0000 7f76b5400700 1 --2- 172.21.15.89:0/4162666357 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f76b0072580 0x7f76b015b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.923+0000 7f76b5400700 1 --2- 172.21.15.89:0/4162666357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b00b2e50 0x7f76b01475b0 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:19.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.923+0000 7f76b5400700 1 -- 172.21.15.89:0/4162666357 >> 172.21.15.89:0/4162666357 conn(0x7f76b00cf190 msgr2=0x7f76b00b0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:19.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.923+0000 7f76b5400700 1 -- 172.21.15.89:0/4162666357 shutdown_connections 2024-04-02T09:14:19.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:19.923+0000 7f76b5400700 1 -- 172.21.15.89:0/4162666357 wait complete. 2024-04-02T09:14:19.941 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-04-02T09:14:19.941 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:14:20.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.132+0000 7f5082600700 1 Processor -- start 2024-04-02T09:14:20.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.133+0000 7f5082600700 1 -- start start 2024-04-02T09:14:20.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.134+0000 7f5082600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f507c15d640 0x7f507c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:20.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.134+0000 7f5082600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f507c1619d0 0x7f507c0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:20.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.134+0000 7f5082600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507c0d30d0 con 0x7f507c0ceec0 2024-04-02T09:14:20.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.134+0000 7f5082600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507c0b6840 con 0x7f507c1619d0 2024-04-02T09:14:20.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.134+0000 7f5082600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507c0d2880 con 0x7f507c15d640 2024-04-02T09:14:20.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.134+0000 7f507aa00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f507c0ceec0 0x7f507c1612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56804/0 (socket says 172.21.15.89:56804) 2024-04-02T09:14:20.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.134+0000 7f507aa00700 1 -- 172.21.15.89:0/2285612839 learned_addr learned my addr 172.21.15.89:0/2285612839 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:20.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.134+0000 7f507be00700 1 --2- 172.21.15.89:0/2285612839 >> v2:172.21.15.89:3300/0 conn(0x7f507c1619d0 0x7f507c0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:20.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.134+0000 7f507b400700 1 --2- 172.21.15.89:0/2285612839 >> v2:172.21.15.113:3300/0 conn(0x7f507c15d640 0x7f507c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:20.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.134+0000 7f507a000700 1 -- 172.21.15.89:0/2285612839 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177172322 0 0) 0x7f507c0d30d0 con 0x7f507c0ceec0 2024-04-02T09:14:20.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.135+0000 7f507a000700 1 -- 172.21.15.89:0/2285612839 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50600035b0 con 0x7f507c0ceec0 2024-04-02T09:14:20.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.135+0000 7f507be00700 1 -- 172.21.15.89:0/2285612839 >> v2:172.21.15.113:3300/0 conn(0x7f507c15d640 msgr2=0x7f507c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:20.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.135+0000 7f507be00700 1 --2- 172.21.15.89:0/2285612839 >> v2:172.21.15.113:3300/0 conn(0x7f507c15d640 0x7f507c0d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.135+0000 7f507be00700 1 -- 172.21.15.89:0/2285612839 >> v1:172.21.15.89:6789/0 conn(0x7f507c0ceec0 legacy=0x7f507c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:20.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.135+0000 7f507be00700 1 -- 172.21.15.89:0/2285612839 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c0d2b30 con 0x7f507c1619d0 2024-04-02T09:14:20.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.135+0000 7f507be00700 1 --2- 172.21.15.89:0/2285612839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f507c1619d0 0x7f507c0b6cd0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7f507c0b5c20 tx=0x7f507400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bbaa38265c925a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:20.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.135+0000 7f507a000700 1 -- 172.21.15.89:0/2285612839 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507400e070 con 0x7f507c1619d0 2024-04-02T09:14:20.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.135+0000 7f507a000700 1 -- 172.21.15.89:0/2285612839 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507400bb20 con 0x7f507c1619d0 2024-04-02T09:14:20.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.135+0000 7f507a000700 1 -- 172.21.15.89:0/2285612839 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507400be90 con 0x7f507c1619d0 2024-04-02T09:14:20.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.136+0000 7f5082600700 1 -- 172.21.15.89:0/2285612839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f507c1619d0 msgr2=0x7f507c0b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:20.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.136+0000 7f5082600700 1 --2- 172.21.15.89:0/2285612839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f507c1619d0 0x7f507c0b6cd0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7f507c0b5c20 tx=0x7f507400b330 comp rx=0 tx=0).stop 2024-04-02T09:14:20.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.136+0000 7f5082600700 1 -- 172.21.15.89:0/2285612839 shutdown_connections 2024-04-02T09:14:20.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.136+0000 7f5082600700 1 --2- 172.21.15.89:0/2285612839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f507c1619d0 0x7f507c0b6cd0 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.136+0000 7f5082600700 1 --2- 172.21.15.89:0/2285612839 >> v2:172.21.15.113:3300/0 conn(0x7f507c15d640 0x7f507c0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.136+0000 7f5082600700 1 -- 172.21.15.89:0/2285612839 >> 172.21.15.89:0/2285612839 conn(0x7f507c08aff0 msgr2=0x7f507c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:20.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.136+0000 7f5082600700 1 -- 172.21.15.89:0/2285612839 shutdown_connections 2024-04-02T09:14:20.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.136+0000 7f5082600700 1 -- 172.21.15.89:0/2285612839 wait complete. 2024-04-02T09:14:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.137+0000 7f5082600700 1 Processor -- start 2024-04-02T09:14:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.138+0000 7f5082600700 1 -- start start 2024-04-02T09:14:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.138+0000 7f5082600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f507c0ceec0 0x7f507c14f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f5082600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f507c15d640 0x7f507c0b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f5082600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c1619d0 0x7f507c0b79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f5082600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0c90e0 con 0x7f507c0ceec0 2024-04-02T09:14:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f5082600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0c80d0 con 0x7f507c1619d0 2024-04-02T09:14:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f5082600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0c8c50 con 0x7f507c15d640 2024-04-02T09:14:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f507b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f507c0ceec0 0x7f507c14f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f507b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f507c0ceec0 0x7f507c14f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41750/0 (socket says 172.21.15.89:41750) 2024-04-02T09:14:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f507b400700 1 -- 172.21.15.89:0/2773732344 learned_addr learned my addr 172.21.15.89:0/2773732344 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f507be00700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c1619d0 0x7f507c0b79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f507aa00700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f507c15d640 0x7f507c0b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f507b400700 1 -- 172.21.15.89:0/2773732344 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f507c15d640 msgr2=0x7f507c0b74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f507b400700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f507c15d640 0x7f507c0b74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f507b400700 1 -- 172.21.15.89:0/2773732344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c1619d0 msgr2=0x7f507c0b79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f507b400700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c1619d0 0x7f507c0b79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f507b400700 1 -- 172.21.15.89:0/2773732344 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5074009a70 con 0x7f507c0ceec0 2024-04-02T09:14:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f507aa00700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f507c15d640 0x7f507c0b74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.139+0000 7f507b400700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f507c0ceec0 0x7f507c14f810 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f506c00e870 tx=0x7f506c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.140+0000 7f5073400700 1 -- 172.21.15.89:0/2773732344 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c009590 con 0x7f507c0ceec0 2024-04-02T09:14:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.140+0000 7f5073400700 1 -- 172.21.15.89:0/2773732344 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506c00ee70 con 0x7f507c0ceec0 2024-04-02T09:14:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.140+0000 7f5073400700 1 -- 172.21.15.89:0/2773732344 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c00ca50 con 0x7f507c0ceec0 2024-04-02T09:14:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.140+0000 7f5082600700 1 -- 172.21.15.89:0/2773732344 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507c0c8320 con 0x7f507c0ceec0 2024-04-02T09:14:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.140+0000 7f5082600700 1 -- 172.21.15.89:0/2773732344 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507c0c88e0 con 0x7f507c0ceec0 2024-04-02T09:14:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.141+0000 7f5082600700 1 -- 172.21.15.89:0/2773732344 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507c0c8f30 con 0x7f507c0ceec0 2024-04-02T09:14:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.142+0000 7f5073400700 1 -- 172.21.15.89:0/2773732344 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f506c00cbb0 con 0x7f507c0ceec0 2024-04-02T09:14:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.143+0000 7f5073400700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f505c06aed0 0x7f505c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.143+0000 7f5073400700 1 -- 172.21.15.89:0/2773732344 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f506c089c40 con 0x7f507c0ceec0 2024-04-02T09:14:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.143+0000 7f507aa00700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f505c06aed0 0x7f505c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.143+0000 7f507aa00700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f505c06aed0 0x7f505c06d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f507c15b890 tx=0x7f5068009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:20.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.147+0000 7f5073400700 1 -- 172.21.15.89:0/2773732344 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f506c054580 con 0x7f507c0ceec0 2024-04-02T09:14:20.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.392+0000 7f5082600700 1 -- 172.21.15.89:0/2773732344 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f507c067720 con 0x7f507c0ceec0 2024-04-02T09:14:20.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.392+0000 7f5073400700 1 -- 172.21.15.89:0/2773732344 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+4757 (secure 0 0 0) 0x7f506c00ce60 con 0x7f507c0ceec0 2024-04-02T09:14:20.391 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:20.391 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":127,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9518,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3094781297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3094781297},{"type":"v1","addr":"172.21.15.89:6836","nonce":3094781297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":9920,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1284027162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1284027162},{"type":"v1","addr":"172.21.15.113:6835","nonce":1284027162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":10978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1799351491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1799351491},{"type":"v1","addr":"172.21.15.113:6834","nonce":1799351491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:41.933360+0000","modified":"2024-04-02T09:13:45.439462+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11958},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11958":{"gid":11958,"name":"a","rank":0,"incarnation":121,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6837/3112580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3112580185},{"type":"v1","addr":"172.21.15.89:6837","nonce":3112580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-02T09:14:20.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.395+0000 7f5082600700 1 -- 172.21.15.89:0/2773732344 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f505c06aed0 msgr2=0x7f505c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:20.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.395+0000 7f5082600700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f505c06aed0 0x7f505c06d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f507c15b890 tx=0x7f5068009040 comp rx=0 tx=0).stop 2024-04-02T09:14:20.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.395+0000 7f5082600700 1 -- 172.21.15.89:0/2773732344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f507c0ceec0 msgr2=0x7f507c14f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:20.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.395+0000 7f5082600700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f507c0ceec0 0x7f507c14f810 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f506c00e870 tx=0x7f506c00ec30 comp rx=0 tx=0).stop 2024-04-02T09:14:20.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.395+0000 7f5082600700 1 -- 172.21.15.89:0/2773732344 shutdown_connections 2024-04-02T09:14:20.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.395+0000 7f5082600700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f505c06aed0 0x7f505c06d390 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.395+0000 7f5082600700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c1619d0 0x7f507c0b79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.395+0000 7f5082600700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f507c0ceec0 0x7f507c14f810 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.395+0000 7f5082600700 1 --2- 172.21.15.89:0/2773732344 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f507c15d640 0x7f507c0b74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.395+0000 7f5082600700 1 -- 172.21.15.89:0/2773732344 >> 172.21.15.89:0/2773732344 conn(0x7f507c08aff0 msgr2=0x7f507c15bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:20.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.395+0000 7f5082600700 1 -- 172.21.15.89:0/2773732344 shutdown_connections 2024-04-02T09:14:20.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.395+0000 7f5082600700 1 -- 172.21.15.89:0/2773732344 wait complete. 2024-04-02T09:14:20.394 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 127 2024-04-02T09:14:20.411 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:14:20.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.604+0000 7fbc8ca00700 1 Processor -- start 2024-04-02T09:14:20.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.604+0000 7fbc8ca00700 1 -- start start 2024-04-02T09:14:20.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.606+0000 7fbc8ca00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbc880d18b0 0x7fbc880cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:20.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.606+0000 7fbc8ca00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbc88161590 0x7fbc88161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:20.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.606+0000 7fbc8ca00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc880ceda0 con 0x7fbc88160f10 2024-04-02T09:14:20.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.606+0000 7fbc8ca00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc880d1d30 con 0x7fbc880d18b0 2024-04-02T09:14:20.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.606+0000 7fbc8ca00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc88161f60 con 0x7fbc88161590 2024-04-02T09:14:20.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.606+0000 7fbc85400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbc880d18b0 0x7fbc880cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:20.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.606+0000 7fbc85400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbc880d18b0 0x7fbc880cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41754/0 (socket says 172.21.15.89:41754) 2024-04-02T09:14:20.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.606+0000 7fbc85400700 1 -- 172.21.15.89:0/3203038596 learned_addr learned my addr 172.21.15.89:0/3203038596 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:20.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.606+0000 7fbc86800700 1 --2- 172.21.15.89:0/3203038596 >> v2:172.21.15.113:3300/0 conn(0x7fbc88161590 0x7fbc88161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:20.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.606+0000 7fbc85400700 1 -- 172.21.15.89:0/3203038596 >> v2:172.21.15.113:3300/0 conn(0x7fbc88161590 msgr2=0x7fbc88161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:20.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.606+0000 7fbc85400700 1 --2- 172.21.15.89:0/3203038596 >> v2:172.21.15.113:3300/0 conn(0x7fbc88161590 0x7fbc88161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.606+0000 7fbc85400700 1 -- 172.21.15.89:0/3203038596 >> v1:172.21.15.89:6789/0 conn(0x7fbc88160f10 legacy=0x7fbc880d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:20.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.606+0000 7fbc85400700 1 -- 172.21.15.89:0/3203038596 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc881612f0 con 0x7fbc880d18b0 2024-04-02T09:14:20.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.606+0000 7fbc86800700 1 --2- 172.21.15.89:0/3203038596 >> v2:172.21.15.113:3300/0 conn(0x7fbc88161590 0x7fbc88161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:20.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.607+0000 7fbc85400700 1 --2- 172.21.15.89:0/3203038596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc880d18b0 0x7fbc880cf230 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7fbc880d1030 tx=0x7fbc74009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec3937fd90580fc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:20.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.607+0000 7fbc84a00700 1 -- 172.21.15.89:0/3203038596 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7400b870 con 0x7fbc880d18b0 2024-04-02T09:14:20.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.607+0000 7fbc84a00700 1 -- 172.21.15.89:0/3203038596 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7400b9d0 con 0x7fbc880d18b0 2024-04-02T09:14:20.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.607+0000 7fbc84a00700 1 -- 172.21.15.89:0/3203038596 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7400bd60 con 0x7fbc880d18b0 2024-04-02T09:14:20.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.607+0000 7fbc8ca00700 1 -- 172.21.15.89:0/3203038596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc880d18b0 msgr2=0x7fbc880cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:20.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.607+0000 7fbc8ca00700 1 --2- 172.21.15.89:0/3203038596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc880d18b0 0x7fbc880cf230 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7fbc880d1030 tx=0x7fbc74009bd0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.607+0000 7fbc8ca00700 1 -- 172.21.15.89:0/3203038596 shutdown_connections 2024-04-02T09:14:20.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.607+0000 7fbc8ca00700 1 --2- 172.21.15.89:0/3203038596 >> v2:172.21.15.113:3300/0 conn(0x7fbc88161590 0x7fbc88161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.607+0000 7fbc8ca00700 1 --2- 172.21.15.89:0/3203038596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc880d18b0 0x7fbc880cf230 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.607+0000 7fbc8ca00700 1 -- 172.21.15.89:0/3203038596 >> 172.21.15.89:0/3203038596 conn(0x7fbc8808aff0 msgr2=0x7fbc880d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:20.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.608+0000 7fbc8ca00700 1 -- 172.21.15.89:0/3203038596 shutdown_connections 2024-04-02T09:14:20.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.608+0000 7fbc8ca00700 1 -- 172.21.15.89:0/3203038596 wait complete. 2024-04-02T09:14:20.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.609+0000 7fbc8ca00700 1 Processor -- start 2024-04-02T09:14:20.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.609+0000 7fbc8ca00700 1 -- start start 2024-04-02T09:14:20.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.610+0000 7fbc8ca00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc880d18b0 0x7fbc881603f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:20.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.610+0000 7fbc8ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc88160f10 0x7fbc880b6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:20.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.610+0000 7fbc8ca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc88161590 0x7fbc8815f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:20.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.610+0000 7fbc8ca00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc880b8230 con 0x7fbc88160f10 2024-04-02T09:14:20.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.610+0000 7fbc8ca00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc88154f90 con 0x7fbc880d18b0 2024-04-02T09:14:20.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.610+0000 7fbc8ca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc880b8460 con 0x7fbc88161590 2024-04-02T09:14:20.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.610+0000 7fbc85400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc88160f10 0x7fbc880b6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:20.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.610+0000 7fbc85400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc88160f10 0x7fbc880b6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41756/0 (socket says 172.21.15.89:41756) 2024-04-02T09:14:20.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.610+0000 7fbc85400700 1 -- 172.21.15.89:0/668480700 learned_addr learned my addr 172.21.15.89:0/668480700 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:20.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.611+0000 7fbc86800700 1 --2- 172.21.15.89:0/668480700 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc88161590 0x7fbc8815f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:20.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.611+0000 7fbc85e00700 1 --2- 172.21.15.89:0/668480700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc880d18b0 0x7fbc881603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:20.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.611+0000 7fbc85400700 1 -- 172.21.15.89:0/668480700 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc88161590 msgr2=0x7fbc8815f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:20.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.611+0000 7fbc85400700 1 --2- 172.21.15.89:0/668480700 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc88161590 0x7fbc8815f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.611+0000 7fbc85400700 1 -- 172.21.15.89:0/668480700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc880d18b0 msgr2=0x7fbc881603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:20.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.611+0000 7fbc85400700 1 --2- 172.21.15.89:0/668480700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc880d18b0 0x7fbc881603f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.611+0000 7fbc85400700 1 -- 172.21.15.89:0/668480700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc74009550 con 0x7fbc88160f10 2024-04-02T09:14:20.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.611+0000 7fbc85400700 1 --2- 172.21.15.89:0/668480700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc88160f10 0x7fbc880b6570 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7fbc74009130 tx=0x7fbc74004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:20.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.611+0000 7fbc7e000700 1 -- 172.21.15.89:0/668480700 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc74004520 con 0x7fbc88160f10 2024-04-02T09:14:20.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.611+0000 7fbc7e000700 1 -- 172.21.15.89:0/668480700 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7401e070 con 0x7fbc88160f10 2024-04-02T09:14:20.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.611+0000 7fbc7e000700 1 -- 172.21.15.89:0/668480700 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc740038c0 con 0x7fbc88160f10 2024-04-02T09:14:20.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.611+0000 7fbc8ca00700 1 -- 172.21.15.89:0/668480700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc880b8740 con 0x7fbc88160f10 2024-04-02T09:14:20.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.612+0000 7fbc8ca00700 1 -- 172.21.15.89:0/668480700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc880b8d00 con 0x7fbc88160f10 2024-04-02T09:14:20.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.612+0000 7fbc8ca00700 1 -- 172.21.15.89:0/668480700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc88067720 con 0x7fbc88160f10 2024-04-02T09:14:20.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.613+0000 7fbc7e000700 1 -- 172.21.15.89:0/668480700 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fbc74003a20 con 0x7fbc88160f10 2024-04-02T09:14:20.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.614+0000 7fbc7e000700 1 --2- 172.21.15.89:0/668480700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbc6806aed0 0x7fbc6806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:20.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.614+0000 7fbc7e000700 1 -- 172.21.15.89:0/668480700 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7029+0+0 (secure 0 0 0) 0x7fbc74020070 con 0x7fbc88160f10 2024-04-02T09:14:20.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.614+0000 7fbc85e00700 1 --2- 172.21.15.89:0/668480700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbc6806aed0 0x7fbc6806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:20.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.615+0000 7fbc85e00700 1 --2- 172.21.15.89:0/668480700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbc6806aed0 0x7fbc6806d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbc78000f60 tx=0x7fbc78009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:20.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.619+0000 7fbc7e000700 1 -- 172.21.15.89:0/668480700 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbc74055d50 con 0x7fbc88160f10 2024-04-02T09:14:20.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.862+0000 7fbc8ca00700 1 -- 172.21.15.89:0/668480700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc880b9400 con 0x7fbc88160f10 2024-04-02T09:14:20.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.863+0000 7fbc7e000700 1 -- 172.21.15.89:0/668480700 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+4757 (secure 0 0 0) 0x7fbc74059370 con 0x7fbc88160f10 2024-04-02T09:14:20.861 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:20.861 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":127,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9518,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3094781297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3094781297},{"type":"v1","addr":"172.21.15.89:6836","nonce":3094781297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":9920,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1284027162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1284027162},{"type":"v1","addr":"172.21.15.113:6835","nonce":1284027162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":10978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1799351491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1799351491},{"type":"v1","addr":"172.21.15.113:6834","nonce":1799351491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:13:41.933360+0000","modified":"2024-04-02T09:13:45.439462+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11958},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11958":{"gid":11958,"name":"a","rank":0,"incarnation":121,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6837/3112580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3112580185},{"type":"v1","addr":"172.21.15.89:6837","nonce":3112580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-04-02T09:14:20.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.865+0000 7fbc8ca00700 1 -- 172.21.15.89:0/668480700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbc6806aed0 msgr2=0x7fbc6806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:20.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.865+0000 7fbc8ca00700 1 --2- 172.21.15.89:0/668480700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbc6806aed0 0x7fbc6806d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbc78000f60 tx=0x7fbc78009040 comp rx=0 tx=0).stop 2024-04-02T09:14:20.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.865+0000 7fbc8ca00700 1 -- 172.21.15.89:0/668480700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc88160f10 msgr2=0x7fbc880b6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:20.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.865+0000 7fbc8ca00700 1 --2- 172.21.15.89:0/668480700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc88160f10 0x7fbc880b6570 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7fbc74009130 tx=0x7fbc74004750 comp rx=0 tx=0).stop 2024-04-02T09:14:20.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.865+0000 7fbc8ca00700 1 -- 172.21.15.89:0/668480700 shutdown_connections 2024-04-02T09:14:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.865+0000 7fbc8ca00700 1 --2- 172.21.15.89:0/668480700 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbc6806aed0 0x7fbc6806d390 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.865+0000 7fbc8ca00700 1 --2- 172.21.15.89:0/668480700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc880d18b0 0x7fbc881603f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.865+0000 7fbc8ca00700 1 --2- 172.21.15.89:0/668480700 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbc88161590 0x7fbc8815f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.865+0000 7fbc8ca00700 1 --2- 172.21.15.89:0/668480700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbc88160f10 0x7fbc880b6570 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.865+0000 7fbc8ca00700 1 -- 172.21.15.89:0/668480700 >> 172.21.15.89:0/668480700 conn(0x7fbc8808aff0 msgr2=0x7fbc880cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.865+0000 7fbc8ca00700 1 -- 172.21.15.89:0/668480700 shutdown_connections 2024-04-02T09:14:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:20.865+0000 7fbc8ca00700 1 -- 172.21.15.89:0/668480700 wait complete. 2024-04-02T09:14:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 127 2024-04-02T09:14:20.882 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:14:21.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.075+0000 7f38ed000700 1 Processor -- start 2024-04-02T09:14:21.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.075+0000 7f38ed000700 1 -- start start 2024-04-02T09:14:21.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.077+0000 7f38ed000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f38e809aba0 0x7f38e806bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.077+0000 7f38ed000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f38e806c480 0x7f38e80ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.077+0000 7f38ed000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38e806bb40 con 0x7f38e815d710 2024-04-02T09:14:21.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.077+0000 7f38ed000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e806c910 con 0x7f38e806c480 2024-04-02T09:14:21.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.077+0000 7f38ed000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e8099b00 con 0x7f38e809aba0 2024-04-02T09:14:21.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.077+0000 7f38e5e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f38e815d710 0x7f38e80d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56812/0 (socket says 172.21.15.89:56812) 2024-04-02T09:14:21.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.077+0000 7f38e5e00700 1 -- 172.21.15.89:0/3611242500 learned_addr learned my addr 172.21.15.89:0/3611242500 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:21.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.077+0000 7f38e5400700 1 --2- 172.21.15.89:0/3611242500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f38e809aba0 0x7f38e806bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.077+0000 7f38e6800700 1 --2- 172.21.15.89:0/3611242500 >> v2:172.21.15.113:3300/0 conn(0x7f38e806c480 0x7f38e80ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.077+0000 7f38e4a00700 1 -- 172.21.15.89:0/3611242500 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808921742 0 0) 0x7f38e806bb40 con 0x7f38e815d710 2024-04-02T09:14:21.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.077+0000 7f38e4a00700 1 -- 172.21.15.89:0/3611242500 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38cc0035b0 con 0x7f38e815d710 2024-04-02T09:14:21.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e4a00700 1 -- 172.21.15.89:0/3611242500 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 635993761 0 0) 0x7f38cc0035b0 con 0x7f38e815d710 2024-04-02T09:14:21.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e4a00700 1 -- 172.21.15.89:0/3611242500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f38e809aba0 msgr2=0x7f38e806bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:21.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e4a00700 1 --2- 172.21.15.89:0/3611242500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f38e809aba0 0x7f38e806bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e4a00700 1 -- 172.21.15.89:0/3611242500 >> v2:172.21.15.113:3300/0 conn(0x7f38e806c480 msgr2=0x7f38e80ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:21.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e4a00700 1 --2- 172.21.15.89:0/3611242500 >> v2:172.21.15.113:3300/0 conn(0x7f38e806c480 0x7f38e80ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e4a00700 1 -- 172.21.15.89:0/3611242500 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e8099de0 con 0x7f38e815d710 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e4a00700 1 -- 172.21.15.89:0/3611242500 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f38d8001670 con 0x7f38e815d710 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e4a00700 1 -- 172.21.15.89:0/3611242500 >> v1:172.21.15.89:6789/0 conn(0x7f38e815d710 legacy=0x7f38e80d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e4a00700 1 --2- 172.21.15.89:0/3611242500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38cc003a00 0x7f38cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e4a00700 1 --2- 172.21.15.89:0/3611242500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38cc004850 0x7f38cc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e4a00700 1 --2- 172.21.15.89:0/3611242500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f38e809aba0 0x7f38cc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e4a00700 1 -- 172.21.15.89:0/3611242500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e806c910 con 0x7f38cc004850 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e4a00700 1 -- 172.21.15.89:0/3611242500 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e8099b00 con 0x7f38cc003a00 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e4a00700 1 -- 172.21.15.89:0/3611242500 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc00ba10 con 0x7f38e809aba0 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e5400700 1 --2- 172.21.15.89:0/3611242500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38cc004850 0x7f38cc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e6800700 1 --2- 172.21.15.89:0/3611242500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f38e809aba0 0x7f38cc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e5e00700 1 --2- 172.21.15.89:0/3611242500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38cc003a00 0x7f38cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e5400700 1 -- 172.21.15.89:0/3611242500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f38e809aba0 msgr2=0x7f38cc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e5400700 1 --2- 172.21.15.89:0/3611242500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f38e809aba0 0x7f38cc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e5400700 1 -- 172.21.15.89:0/3611242500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38cc003a00 msgr2=0x7f38cc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:21.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e5400700 1 --2- 172.21.15.89:0/3611242500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38cc003a00 0x7f38cc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e5400700 1 -- 172.21.15.89:0/3611242500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38cc0035b0 con 0x7f38cc004850 2024-04-02T09:14:21.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.078+0000 7f38e5e00700 1 --2- 172.21.15.89:0/3611242500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38cc003a00 0x7f38cc003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:21.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.079+0000 7f38e5400700 1 --2- 172.21.15.89:0/3611242500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38cc004850 0x7f38cc004cd0 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f38d4009b30 tx=0x7f38cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c5aa336f3f4e4a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:21.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.079+0000 7f38e4a00700 1 -- 172.21.15.89:0/3611242500 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d400e660 con 0x7f38cc004850 2024-04-02T09:14:21.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.079+0000 7f38e4a00700 1 -- 172.21.15.89:0/3611242500 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d400e7c0 con 0x7f38cc004850 2024-04-02T09:14:21.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.079+0000 7f38e4a00700 1 -- 172.21.15.89:0/3611242500 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d400eb30 con 0x7f38cc004850 2024-04-02T09:14:21.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.079+0000 7f38ed000700 1 -- 172.21.15.89:0/3611242500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38cc004850 msgr2=0x7f38cc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:21.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.079+0000 7f38ed000700 1 --2- 172.21.15.89:0/3611242500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38cc004850 0x7f38cc004cd0 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f38d4009b30 tx=0x7f38cc005880 comp rx=0 tx=0).stop 2024-04-02T09:14:21.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.079+0000 7f38e6800700 1 -- 172.21.15.89:0/3611242500 reap_dead start 2024-04-02T09:14:21.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.079+0000 7f38ed000700 1 -- 172.21.15.89:0/3611242500 shutdown_connections 2024-04-02T09:14:21.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.079+0000 7f38ed000700 1 -- 172.21.15.89:0/3611242500 >> 172.21.15.89:0/3611242500 conn(0x7f38e80d13e0 msgr2=0x7f38e80d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:21.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.079+0000 7f38ed000700 1 -- 172.21.15.89:0/3611242500 shutdown_connections 2024-04-02T09:14:21.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.079+0000 7f38ed000700 1 -- 172.21.15.89:0/3611242500 wait complete. 2024-04-02T09:14:21.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.081+0000 7f38ed000700 1 Processor -- start 2024-04-02T09:14:21.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.081+0000 7f38ed000700 1 -- start start 2024-04-02T09:14:21.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.082+0000 7f38ed000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38cc004850 0x7f38e81f7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.082+0000 7f38ed000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38e81f3270 0x7f38e80b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.082+0000 7f38ed000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f38e80b0ed0 0x7f38e8156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.082+0000 7f38ed000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e81f36f0 con 0x7f38cc004850 2024-04-02T09:14:21.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.082+0000 7f38ed000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e80b1500 con 0x7f38e81f3270 2024-04-02T09:14:21.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.082+0000 7f38ed000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e80b17e0 con 0x7f38e80b0ed0 2024-04-02T09:14:21.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.082+0000 7f38e5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38cc004850 0x7f38e81f7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38e5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38cc004850 0x7f38e81f7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41776/0 (socket says 172.21.15.89:41776) 2024-04-02T09:14:21.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38e5e00700 1 -- 172.21.15.89:0/3060676887 learned_addr learned my addr 172.21.15.89:0/3060676887 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:21.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38e6800700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f38e80b0ed0 0x7f38e8156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38e5400700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38e81f3270 0x7f38e80b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38e5e00700 1 -- 172.21.15.89:0/3060676887 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f38e80b0ed0 msgr2=0x7f38e8156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:21.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38e5e00700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f38e80b0ed0 0x7f38e8156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38e5e00700 1 -- 172.21.15.89:0/3060676887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38e81f3270 msgr2=0x7f38e80b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:21.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38e5e00700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38e81f3270 0x7f38e80b0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38e5e00700 1 -- 172.21.15.89:0/3060676887 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e81f6f30 con 0x7f38cc004850 2024-04-02T09:14:21.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38e5400700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38e81f3270 0x7f38e80b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:21.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38e6800700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f38e80b0ed0 0x7f38e8156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:21.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38e5e00700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38cc004850 0x7f38e81f7a10 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f38d800db10 tx=0x7f38d800ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:21.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38de000700 1 -- 172.21.15.89:0/3060676887 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d8004a80 con 0x7f38cc004850 2024-04-02T09:14:21.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38de000700 1 -- 172.21.15.89:0/3060676887 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d8004be0 con 0x7f38cc004850 2024-04-02T09:14:21.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38de000700 1 -- 172.21.15.89:0/3060676887 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d80070d0 con 0x7f38cc004850 2024-04-02T09:14:21.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.083+0000 7f38ed000700 1 -- 172.21.15.89:0/3060676887 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e81f7210 con 0x7f38cc004850 2024-04-02T09:14:21.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.084+0000 7f38ed000700 1 -- 172.21.15.89:0/3060676887 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e814e860 con 0x7f38cc004850 2024-04-02T09:14:21.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.084+0000 7f38ed000700 1 -- 172.21.15.89:0/3060676887 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e81f7860 con 0x7f38cc004850 2024-04-02T09:14:21.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.085+0000 7f38de000700 1 -- 172.21.15.89:0/3060676887 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f38d8007230 con 0x7f38cc004850 2024-04-02T09:14:21.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.086+0000 7f38de000700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f38c806aed0 0x7f38c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.086+0000 7f38de000700 1 -- 172.21.15.89:0/3060676887 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7029+0+0 (secure 0 0 0) 0x7f38d8089670 con 0x7f38cc004850 2024-04-02T09:14:21.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.086+0000 7f38e5400700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f38c806aed0 0x7f38c806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.087+0000 7f38e5400700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f38c806aed0 0x7f38c806d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f38d40090b0 tx=0x7f38d4008030 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:21.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.090+0000 7f38de000700 1 -- 172.21.15.89:0/3060676887 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f38d8053e10 con 0x7f38cc004850 2024-04-02T09:14:21.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.297+0000 7f38ed000700 1 -- 172.21.15.89:0/3060676887 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38e80b12b0 con 0x7f38cc004850 2024-04-02T09:14:21.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.298+0000 7f38de000700 1 -- 172.21.15.89:0/3060676887 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) v1 ==== 74+0+17997 (secure 0 0 0) 0x7f38d8057430 con 0x7f38cc004850 2024-04-02T09:14:21.297 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:21.297 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":253,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:14:17.272062+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-02T09:13:40.048039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-02T09:13:40.907706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.113:6833/703852199":"2024-04-03T09:14:13.967181+0000","172.21.15.89:0/3750432255":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/2974789087":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/1231626446":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/2398338283":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/4088911225":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3553222249":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3687373115":"2024-04-03T09:14:11.230784+0000","172.21.15.113:6835/703852199":"2024-04-03T09:14:13.967181+0000","172.21.15.89:0/1998199546":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/492795889":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/1337615482":"2024-04-03T09:14:11.230784+0000","172.21.15.89:0/3750385348":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/2067457213":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/1275405725":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/3355557522":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/4204293224":"2024-04-03T09:14:02.049396+0000","172.21.15.89:0/176419408":"2024-04-03T09:14:02.049396+0000","172.21.15.89:6833/108855":"2024-04-03T09:14:02.049396+0000","172.21.15.89:6832/108855":"2024-04-03T09:14:02.049396+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:14:21.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.301+0000 7f38ed000700 1 -- 172.21.15.89:0/3060676887 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f38c806aed0 msgr2=0x7f38c806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:21.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.301+0000 7f38ed000700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f38c806aed0 0x7f38c806d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f38d40090b0 tx=0x7f38d4008030 comp rx=0 tx=0).stop 2024-04-02T09:14:21.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.301+0000 7f38ed000700 1 -- 172.21.15.89:0/3060676887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38cc004850 msgr2=0x7f38e81f7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:21.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.301+0000 7f38ed000700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38cc004850 0x7f38e81f7a10 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f38d800db10 tx=0x7f38d800ded0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.301+0000 7f38ed000700 1 -- 172.21.15.89:0/3060676887 shutdown_connections 2024-04-02T09:14:21.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.301+0000 7f38ed000700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f38e80b0ed0 0x7f38e8156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.301+0000 7f38ed000700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38cc004850 0x7f38e81f7a10 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.301+0000 7f38ed000700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f38c806aed0 0x7f38c806d390 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.301+0000 7f38ed000700 1 --2- 172.21.15.89:0/3060676887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38e81f3270 0x7f38e80b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.301+0000 7f38ed000700 1 -- 172.21.15.89:0/3060676887 >> 172.21.15.89:0/3060676887 conn(0x7f38e80d13e0 msgr2=0x7f38e81595d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:21.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.301+0000 7f38ed000700 1 -- 172.21.15.89:0/3060676887 shutdown_connections 2024-04-02T09:14:21.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.301+0000 7f38ed000700 1 -- 172.21.15.89:0/3060676887 wait complete. 2024-04-02T09:14:21.318 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-04-02T09:14:21.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.514+0000 7fa576200700 1 Processor -- start 2024-04-02T09:14:21.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.514+0000 7fa576200700 1 -- start start 2024-04-02T09:14:21.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.515+0000 7fa576200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5700d16a0 0x7fa5700d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.515+0000 7fa576200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa5700d2370 0x7fa5700ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.515+0000 7fa576200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa570161420 con 0x7fa57015d9d0 2024-04-02T09:14:21.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.515+0000 7fa576200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5700d2820 con 0x7fa5700d2370 2024-04-02T09:14:21.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.515+0000 7fa576200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700d4340 con 0x7fa5700d16a0 2024-04-02T09:14:21.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.515+0000 7fa56fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa5700d2370 0x7fa5700ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.515+0000 7fa56fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa5700d2370 0x7fa5700ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41790/0 (socket says 172.21.15.89:41790) 2024-04-02T09:14:21.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.515+0000 7fa56fe00700 1 -- 172.21.15.89:0/2306546153 learned_addr learned my addr 172.21.15.89:0/2306546153 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:21.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.516+0000 7fa56ea00700 1 --2- 172.21.15.89:0/2306546153 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5700d16a0 0x7fa5700d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.516+0000 7fa56fe00700 1 -- 172.21.15.89:0/2306546153 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5700d16a0 msgr2=0x7fa5700d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:21.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.516+0000 7fa56fe00700 1 --2- 172.21.15.89:0/2306546153 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5700d16a0 0x7fa5700d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.516+0000 7fa56fe00700 1 -- 172.21.15.89:0/2306546153 >> v1:172.21.15.113:6789/0 conn(0x7fa57015d9d0 legacy=0x7fa5700d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:21.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.516+0000 7fa56fe00700 1 -- 172.21.15.89:0/2306546153 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa570161700 con 0x7fa5700d2370 2024-04-02T09:14:21.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.516+0000 7fa56ea00700 1 --2- 172.21.15.89:0/2306546153 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5700d16a0 0x7fa5700d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:21.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.516+0000 7fa56fe00700 1 --2- 172.21.15.89:0/2306546153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5700d2370 0x7fa5700ce360 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7fa568009e80 tx=0x7fa56800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d00f7de811eb82f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:21.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.516+0000 7fa56e000700 1 -- 172.21.15.89:0/2306546153 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56800e070 con 0x7fa5700d2370 2024-04-02T09:14:21.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.516+0000 7fa56e000700 1 -- 172.21.15.89:0/2306546153 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa568003a20 con 0x7fa5700d2370 2024-04-02T09:14:21.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.516+0000 7fa56e000700 1 -- 172.21.15.89:0/2306546153 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa568003dd0 con 0x7fa5700d2370 2024-04-02T09:14:21.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.517+0000 7fa576200700 1 -- 172.21.15.89:0/2306546153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5700d2370 msgr2=0x7fa5700ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:21.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.517+0000 7fa576200700 1 --2- 172.21.15.89:0/2306546153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5700d2370 0x7fa5700ce360 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7fa568009e80 tx=0x7fa56800b330 comp rx=0 tx=0).stop 2024-04-02T09:14:21.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.517+0000 7fa576200700 1 -- 172.21.15.89:0/2306546153 shutdown_connections 2024-04-02T09:14:21.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.517+0000 7fa576200700 1 --2- 172.21.15.89:0/2306546153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5700d2370 0x7fa5700ce360 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.517+0000 7fa576200700 1 --2- 172.21.15.89:0/2306546153 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5700d16a0 0x7fa5700d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.517+0000 7fa576200700 1 -- 172.21.15.89:0/2306546153 >> 172.21.15.89:0/2306546153 conn(0x7fa57008aff0 msgr2=0x7fa5700d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:21.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.517+0000 7fa576200700 1 -- 172.21.15.89:0/2306546153 shutdown_connections 2024-04-02T09:14:21.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.517+0000 7fa576200700 1 -- 172.21.15.89:0/2306546153 wait complete. 2024-04-02T09:14:21.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.519+0000 7fa576200700 1 Processor -- start 2024-04-02T09:14:21.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.519+0000 7fa576200700 1 -- start start 2024-04-02T09:14:21.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa576200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5700d16a0 0x7fa5701564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa576200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5700d2370 0x7fa570155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa576200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa57015d9d0 0x7fa5700b4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa576200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700cd4b0 con 0x7fa5700d16a0 2024-04-02T09:14:21.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa576200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5701e7520 con 0x7fa5700d2370 2024-04-02T09:14:21.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa576200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700ccc10 con 0x7fa57015d9d0 2024-04-02T09:14:21.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa56f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5700d16a0 0x7fa5701564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa56f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5700d16a0 0x7fa5701564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41796/0 (socket says 172.21.15.89:41796) 2024-04-02T09:14:21.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa56f400700 1 -- 172.21.15.89:0/3926595339 learned_addr learned my addr 172.21.15.89:0/3926595339 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:21.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa56fe00700 1 --2- 172.21.15.89:0/3926595339 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa57015d9d0 0x7fa5700b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa56ea00700 1 --2- 172.21.15.89:0/3926595339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5700d2370 0x7fa570155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa56f400700 1 -- 172.21.15.89:0/3926595339 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa57015d9d0 msgr2=0x7fa5700b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:21.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa56f400700 1 --2- 172.21.15.89:0/3926595339 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa57015d9d0 0x7fa5700b4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa56f400700 1 -- 172.21.15.89:0/3926595339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5700d2370 msgr2=0x7fa570155c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:21.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa56f400700 1 --2- 172.21.15.89:0/3926595339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5700d2370 0x7fa570155c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.520+0000 7fa56f400700 1 -- 172.21.15.89:0/3926595339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa568009b30 con 0x7fa5700d16a0 2024-04-02T09:14:21.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.521+0000 7fa56f400700 1 --2- 172.21.15.89:0/3926595339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5700d16a0 0x7fa5701564c0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fa56000b4d0 tx=0x7fa56000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:21.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.521+0000 7fa567400700 1 -- 172.21.15.89:0/3926595339 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa560011830 con 0x7fa5700d16a0 2024-04-02T09:14:21.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.521+0000 7fa567400700 1 -- 172.21.15.89:0/3926595339 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa560011990 con 0x7fa5700d16a0 2024-04-02T09:14:21.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.521+0000 7fa567400700 1 -- 172.21.15.89:0/3926595339 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa560011d40 con 0x7fa5700d16a0 2024-04-02T09:14:21.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.521+0000 7fa576200700 1 -- 172.21.15.89:0/3926595339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5700ccef0 con 0x7fa5700d16a0 2024-04-02T09:14:21.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.521+0000 7fa576200700 1 -- 172.21.15.89:0/3926595339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5700cc830 con 0x7fa5700d16a0 2024-04-02T09:14:21.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.522+0000 7fa576200700 1 -- 172.21.15.89:0/3926595339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5700d1ca0 con 0x7fa5700d16a0 2024-04-02T09:14:21.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.523+0000 7fa567400700 1 -- 172.21.15.89:0/3926595339 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa56001a400 con 0x7fa5700d16a0 2024-04-02T09:14:21.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.524+0000 7fa567400700 1 --2- 172.21.15.89:0/3926595339 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa55006aed0 0x7fa55006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:21.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.524+0000 7fa567400700 1 -- 172.21.15.89:0/3926595339 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7029+0+0 (secure 0 0 0) 0x7fa560089a00 con 0x7fa5700d16a0 2024-04-02T09:14:21.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.524+0000 7fa56ea00700 1 --2- 172.21.15.89:0/3926595339 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa55006aed0 0x7fa55006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:21.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.525+0000 7fa56ea00700 1 --2- 172.21.15.89:0/3926595339 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa55006aed0 0x7fa55006d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa5701f6b70 tx=0x7fa55c008040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:21.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.529+0000 7fa567400700 1 -- 172.21.15.89:0/3926595339 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa5600542b0 con 0x7fa5700d16a0 2024-04-02T09:14:21.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.774+0000 7fa576200700 1 -- 172.21.15.89:0/3926595339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7fa5701e7280 con 0x7fa5700d16a0 2024-04-02T09:14:21.773 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T09:14:21.775+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T09:14:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.788+0000 7fa567400700 1 -- 172.21.15.89:0/3926595339 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v128) v1 ==== 167+0+0 (secure 0 0 0) 0x7fa5701e7280 con 0x7fa5700d16a0 2024-04-02T09:14:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.790+0000 7fa576200700 1 -- 172.21.15.89:0/3926595339 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa55006aed0 msgr2=0x7fa55006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.790+0000 7fa576200700 1 --2- 172.21.15.89:0/3926595339 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa55006aed0 0x7fa55006d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa5701f6b70 tx=0x7fa55c008040 comp rx=0 tx=0).stop 2024-04-02T09:14:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.790+0000 7fa576200700 1 -- 172.21.15.89:0/3926595339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5700d16a0 msgr2=0x7fa5701564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.790+0000 7fa576200700 1 --2- 172.21.15.89:0/3926595339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5700d16a0 0x7fa5701564c0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fa56000b4d0 tx=0x7fa56000b890 comp rx=0 tx=0).stop 2024-04-02T09:14:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.790+0000 7fa576200700 1 -- 172.21.15.89:0/3926595339 shutdown_connections 2024-04-02T09:14:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.790+0000 7fa576200700 1 --2- 172.21.15.89:0/3926595339 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa57015d9d0 0x7fa5700b4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.790+0000 7fa576200700 1 --2- 172.21.15.89:0/3926595339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5700d16a0 0x7fa5701564c0 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.790+0000 7fa576200700 1 --2- 172.21.15.89:0/3926595339 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa55006aed0 0x7fa55006d390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.790+0000 7fa576200700 1 --2- 172.21.15.89:0/3926595339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5700d2370 0x7fa570155c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.790+0000 7fa576200700 1 -- 172.21.15.89:0/3926595339 >> 172.21.15.89:0/3926595339 conn(0x7fa57008aff0 msgr2=0x7fa5700b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:21.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.790+0000 7fa576200700 1 -- 172.21.15.89:0/3926595339 shutdown_connections 2024-04-02T09:14:21.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:21.791+0000 7fa576200700 1 -- 172.21.15.89:0/3926595339 wait complete. 2024-04-02T09:14:21.790 INFO:teuthology.orchestra.run.smithi089.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T09:14:21.792 INFO:tasks.ceph.mds.a.smithi089.stderr: -1969> 2024-04-02T09:13:43.845+0000 7f8b3c000700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T09:14:21.807 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-04-02T09:14:21.838 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T09:14:22.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.006+0000 7f875e000700 1 Processor -- start 2024-04-02T09:14:22.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.007+0000 7f875e000700 1 -- start start 2024-04-02T09:14:22.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.008+0000 7f875e000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f875815d9d0 0x7f87580d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:22.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.008+0000 7f875e000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8758161420 con 0x7f87580d16a0 2024-04-02T09:14:22.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.008+0000 7f875e000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8758161700 con 0x7f87580d2540 2024-04-02T09:14:22.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.008+0000 7f875e000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f875815edc0 con 0x7f875815d9d0 2024-04-02T09:14:22.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.008+0000 7f8757400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f875815d9d0 0x7f87580d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:22.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.008+0000 7f8757400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f875815d9d0 0x7f87580d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41818/0 (socket says 172.21.15.89:41818) 2024-04-02T09:14:22.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.008+0000 7f8757400700 1 -- 172.21.15.89:0/2695739319 learned_addr learned my addr 172.21.15.89:0/2695739319 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:22.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.008+0000 7f8757400700 1 -- 172.21.15.89:0/2695739319 >> v1:172.21.15.113:6789/0 conn(0x7f87580d2540 legacy=0x7f87580ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:22.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.008+0000 7f8757400700 1 -- 172.21.15.89:0/2695739319 >> v1:172.21.15.89:6789/0 conn(0x7f87580d16a0 legacy=0x7f87580d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:22.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.009+0000 7f8757400700 1 -- 172.21.15.89:0/2695739319 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87581619e0 con 0x7f875815d9d0 2024-04-02T09:14:22.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.009+0000 7f8757400700 1 --2- 172.21.15.89:0/2695739319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f875815d9d0 0x7f87580d3c30 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f87480098f0 tx=0x7f8748009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9552942c6bb3ff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:22.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.009+0000 7f8756000700 1 -- 172.21.15.89:0/2695739319 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874800b8a0 con 0x7f875815d9d0 2024-04-02T09:14:22.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.009+0000 7f8756000700 1 -- 172.21.15.89:0/2695739319 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874800ba00 con 0x7f875815d9d0 2024-04-02T09:14:22.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.009+0000 7f8756000700 1 -- 172.21.15.89:0/2695739319 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874800bd90 con 0x7f875815d9d0 2024-04-02T09:14:22.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.010+0000 7f875e000700 1 -- 172.21.15.89:0/2695739319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f875815d9d0 msgr2=0x7f87580d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:22.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.010+0000 7f875e000700 1 --2- 172.21.15.89:0/2695739319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f875815d9d0 0x7f87580d3c30 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f87480098f0 tx=0x7f8748009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:22.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.010+0000 7f875e000700 1 -- 172.21.15.89:0/2695739319 shutdown_connections 2024-04-02T09:14:22.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.010+0000 7f875e000700 1 --2- 172.21.15.89:0/2695739319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f875815d9d0 0x7f87580d3c30 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:22.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.010+0000 7f875e000700 1 -- 172.21.15.89:0/2695739319 >> 172.21.15.89:0/2695739319 conn(0x7f875808aff0 msgr2=0x7f87580d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:22.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.010+0000 7f875e000700 1 -- 172.21.15.89:0/2695739319 shutdown_connections 2024-04-02T09:14:22.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.010+0000 7f875e000700 1 -- 172.21.15.89:0/2695739319 wait complete. 2024-04-02T09:14:22.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.011+0000 7f875e000700 1 Processor -- start 2024-04-02T09:14:22.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.012+0000 7f875e000700 1 -- start start 2024-04-02T09:14:22.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.012+0000 7f875e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87580d16a0 0x7f87580b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:22.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f875e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87580d2540 0x7f87581e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:22.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f875e000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f875815d9d0 0x7f87581e77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:22.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f875e000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87580b3980 con 0x7f87580d16a0 2024-04-02T09:14:22.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f875e000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87580b7b20 con 0x7f875815d9d0 2024-04-02T09:14:22.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f875e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87580b3050 con 0x7f87580d2540 2024-04-02T09:14:22.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f8757400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87580d16a0 0x7f87580b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:22.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f8757400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87580d16a0 0x7f87580b3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41830/0 (socket says 172.21.15.89:41830) 2024-04-02T09:14:22.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f8757400700 1 -- 172.21.15.89:0/2380141921 learned_addr learned my addr 172.21.15.89:0/2380141921 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:22.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f8757e00700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f875815d9d0 0x7f87581e77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:22.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f8756a00700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87580d2540 0x7f87581e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:22.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f8757400700 1 -- 172.21.15.89:0/2380141921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87580d2540 msgr2=0x7f87581e7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:22.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f8757400700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87580d2540 0x7f87581e7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:22.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f8757400700 1 -- 172.21.15.89:0/2380141921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f875815d9d0 msgr2=0x7f87581e77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:22.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f8757400700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f875815d9d0 0x7f87581e77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:22.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f8757400700 1 -- 172.21.15.89:0/2380141921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8748009550 con 0x7f87580d16a0 2024-04-02T09:14:22.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f8756a00700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87580d2540 0x7f87581e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:22.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f8757e00700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f875815d9d0 0x7f87581e77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:22.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.013+0000 7f8757400700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87580d16a0 0x7f87580b3e10 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f8748014040 tx=0x7f8748004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:22.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.014+0000 7f874f400700 1 -- 172.21.15.89:0/2380141921 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87480038a0 con 0x7f87580d16a0 2024-04-02T09:14:22.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.014+0000 7f874f400700 1 -- 172.21.15.89:0/2380141921 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874801e070 con 0x7f87580d16a0 2024-04-02T09:14:22.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.014+0000 7f875e000700 1 -- 172.21.15.89:0/2380141921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87580b3330 con 0x7f87580d16a0 2024-04-02T09:14:22.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.014+0000 7f874f400700 1 -- 172.21.15.89:0/2380141921 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8748003c60 con 0x7f87580d16a0 2024-04-02T09:14:22.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.014+0000 7f875e000700 1 -- 172.21.15.89:0/2380141921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87580b4350 con 0x7f87580d16a0 2024-04-02T09:14:22.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.015+0000 7f875e000700 1 -- 172.21.15.89:0/2380141921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87580d1ca0 con 0x7f87580d16a0 2024-04-02T09:14:22.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.016+0000 7f874f400700 1 -- 172.21.15.89:0/2380141921 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f8748025480 con 0x7f87580d16a0 2024-04-02T09:14:22.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.017+0000 7f874f400700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f873806aed0 0x7f873806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:22.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.017+0000 7f874f400700 1 -- 172.21.15.89:0/2380141921 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f8748020070 con 0x7f87580d16a0 2024-04-02T09:14:22.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.017+0000 7f8756a00700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f873806aed0 0x7f873806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:22.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.017+0000 7f8756a00700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f873806aed0 0x7f873806d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f87580cdae0 tx=0x7f8744009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:22.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.021+0000 7f874f400700 1 -- 172.21.15.89:0/2380141921 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8748055b60 con 0x7f87580d16a0 2024-04-02T09:14:22.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.271+0000 7f875e000700 1 -- 172.21.15.89:0/2380141921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f87580b3c60 con 0x7f87580d16a0 2024-04-02T09:14:22.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.790+0000 7f874f400700 1 -- 172.21.15.89:0/2380141921 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v129) v1 ==== 107+0+0 (secure 0 0 0) 0x7f8748059180 con 0x7f87580d16a0 2024-04-02T09:14:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.793+0000 7f875e000700 1 -- 172.21.15.89:0/2380141921 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f873806aed0 msgr2=0x7f873806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.793+0000 7f875e000700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f873806aed0 0x7f873806d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f87580cdae0 tx=0x7f8744009040 comp rx=0 tx=0).stop 2024-04-02T09:14:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.793+0000 7f875e000700 1 -- 172.21.15.89:0/2380141921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87580d16a0 msgr2=0x7f87580b3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.793+0000 7f875e000700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87580d16a0 0x7f87580b3e10 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f8748014040 tx=0x7f8748004750 comp rx=0 tx=0).stop 2024-04-02T09:14:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.793+0000 7f875e000700 1 -- 172.21.15.89:0/2380141921 shutdown_connections 2024-04-02T09:14:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.793+0000 7f875e000700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f873806aed0 0x7f873806d390 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.793+0000 7f875e000700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f875815d9d0 0x7f87581e77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.793+0000 7f875e000700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87580d16a0 0x7f87580b3e10 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.793+0000 7f875e000700 1 --2- 172.21.15.89:0/2380141921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87580d2540 0x7f87581e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.793+0000 7f875e000700 1 -- 172.21.15.89:0/2380141921 >> 172.21.15.89:0/2380141921 conn(0x7f875808aff0 msgr2=0x7f875815ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.793+0000 7f875e000700 1 -- 172.21.15.89:0/2380141921 shutdown_connections 2024-04-02T09:14:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:22.794+0000 7f875e000700 1 -- 172.21.15.89:0/2380141921 wait complete. 2024-04-02T09:14:22.810 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-04-02T09:14:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.047+0000 7f5d1ec00700 1 Processor -- start 2024-04-02T09:14:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.048+0000 7f5d1ec00700 1 -- start start 2024-04-02T09:14:23.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.049+0000 7f5d1ec00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5d18155ac0 0x7f5d1815c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:23.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.049+0000 7f5d1ec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d180ae670 0x7f5d1815ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:23.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.049+0000 7f5d1ec00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1806dd00 con 0x7f5d1806c910 2024-04-02T09:14:23.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.049+0000 7f5d1ec00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d1806d0f0 con 0x7f5d18155ac0 2024-04-02T09:14:23.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.049+0000 7f5d1ec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1806d3d0 con 0x7f5d180ae670 2024-04-02T09:14:23.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.049+0000 7f5d17e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5d18155ac0 0x7f5d1815c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:23.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.049+0000 7f5d17e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5d18155ac0 0x7f5d1815c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41846/0 (socket says 172.21.15.89:41846) 2024-04-02T09:14:23.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.049+0000 7f5d17e00700 1 -- 172.21.15.89:0/429321483 learned_addr learned my addr 172.21.15.89:0/429321483 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:23.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.049+0000 7f5d17400700 1 --1- 172.21.15.89:0/429321483 >> v1:172.21.15.89:6789/0 conn(0x7f5d1806c910 0x7f5d180adf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56840/0 (socket says 172.21.15.89:56840) 2024-04-02T09:14:23.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.049+0000 7f5d1cc00700 1 --2- 172.21.15.89:0/429321483 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d180ae670 0x7f5d1815ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:23.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.049+0000 7f5d17e00700 1 -- 172.21.15.89:0/429321483 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d180ae670 msgr2=0x7f5d1815ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:23.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.049+0000 7f5d17e00700 1 --2- 172.21.15.89:0/429321483 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d180ae670 0x7f5d1815ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:23.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.049+0000 7f5d17e00700 1 -- 172.21.15.89:0/429321483 >> v1:172.21.15.89:6789/0 conn(0x7f5d1806c910 legacy=0x7f5d180adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:23.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.050+0000 7f5d17e00700 1 -- 172.21.15.89:0/429321483 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1806d6b0 con 0x7f5d18155ac0 2024-04-02T09:14:23.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.050+0000 7f5d1cc00700 1 --2- 172.21.15.89:0/429321483 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d180ae670 0x7f5d1815ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:23.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.050+0000 7f5d17e00700 1 --2- 172.21.15.89:0/429321483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d18155ac0 0x7f5d1815c700 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f5d080098f0 tx=0x7f5d08009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83defbfb39689a5d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:23.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.050+0000 7f5d16a00700 1 -- 172.21.15.89:0/429321483 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0800b8a0 con 0x7f5d18155ac0 2024-04-02T09:14:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.050+0000 7f5d16a00700 1 -- 172.21.15.89:0/429321483 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0800ba00 con 0x7f5d18155ac0 2024-04-02T09:14:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.050+0000 7f5d16a00700 1 -- 172.21.15.89:0/429321483 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0800bd90 con 0x7f5d18155ac0 2024-04-02T09:14:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.050+0000 7f5d1ec00700 1 -- 172.21.15.89:0/429321483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d18155ac0 msgr2=0x7f5d1815c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.050+0000 7f5d1ec00700 1 --2- 172.21.15.89:0/429321483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d18155ac0 0x7f5d1815c700 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f5d080098f0 tx=0x7f5d08009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.051+0000 7f5d1ec00700 1 -- 172.21.15.89:0/429321483 shutdown_connections 2024-04-02T09:14:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.051+0000 7f5d1ec00700 1 --2- 172.21.15.89:0/429321483 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d180ae670 0x7f5d1815ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.051+0000 7f5d1ec00700 1 --2- 172.21.15.89:0/429321483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d18155ac0 0x7f5d1815c700 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.051+0000 7f5d1ec00700 1 -- 172.21.15.89:0/429321483 >> 172.21.15.89:0/429321483 conn(0x7f5d180cf210 msgr2=0x7f5d180cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.051+0000 7f5d1ec00700 1 -- 172.21.15.89:0/429321483 shutdown_connections 2024-04-02T09:14:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.051+0000 7f5d1ec00700 1 -- 172.21.15.89:0/429321483 wait complete. 2024-04-02T09:14:23.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.052+0000 7f5d1ec00700 1 Processor -- start 2024-04-02T09:14:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.052+0000 7f5d1ec00700 1 -- start start 2024-04-02T09:14:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.053+0000 7f5d1ec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d1806c910 0x7f5d1814d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.053+0000 7f5d1ec00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d180ae670 0x7f5d1815a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.053+0000 7f5d1ec00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d18155ac0 0x7f5d180b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.053+0000 7f5d1ec00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d181505a0 con 0x7f5d180ae670 2024-04-02T09:14:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.053+0000 7f5d1ec00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1814f920 con 0x7f5d18155ac0 2024-04-02T09:14:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.053+0000 7f5d1ec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1814fc00 con 0x7f5d1806c910 2024-04-02T09:14:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d17400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d180ae670 0x7f5d1815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d17400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d180ae670 0x7f5d1815a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41862/0 (socket says 172.21.15.89:41862) 2024-04-02T09:14:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d17400700 1 -- 172.21.15.89:0/4195823455 learned_addr learned my addr 172.21.15.89:0/4195823455 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d1cc00700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d18155ac0 0x7f5d180b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d17e00700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d1806c910 0x7f5d1814d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d17400700 1 -- 172.21.15.89:0/4195823455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d1806c910 msgr2=0x7f5d1814d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d17400700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d1806c910 0x7f5d1814d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d17400700 1 -- 172.21.15.89:0/4195823455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d18155ac0 msgr2=0x7f5d180b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d17400700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d18155ac0 0x7f5d180b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d17400700 1 -- 172.21.15.89:0/4195823455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d08009550 con 0x7f5d180ae670 2024-04-02T09:14:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d1cc00700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d18155ac0 0x7f5d180b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d17e00700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d1806c910 0x7f5d1814d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d17400700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d180ae670 0x7f5d1815a860 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400c520 tx=0x7f5d0400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d0fe00700 1 -- 172.21.15.89:0/4195823455 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d04005cb0 con 0x7f5d180ae670 2024-04-02T09:14:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d0fe00700 1 -- 172.21.15.89:0/4195823455 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d04005e10 con 0x7f5d180ae670 2024-04-02T09:14:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.054+0000 7f5d1ec00700 1 -- 172.21.15.89:0/4195823455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d18156020 con 0x7f5d180ae670 2024-04-02T09:14:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.055+0000 7f5d0fe00700 1 -- 172.21.15.89:0/4195823455 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d040072d0 con 0x7f5d180ae670 2024-04-02T09:14:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.055+0000 7f5d1ec00700 1 -- 172.21.15.89:0/4195823455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d181501c0 con 0x7f5d180ae670 2024-04-02T09:14:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.056+0000 7f5d1ec00700 1 -- 172.21.15.89:0/4195823455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d181506e0 con 0x7f5d180ae670 2024-04-02T09:14:23.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.057+0000 7f5d0fe00700 1 -- 172.21.15.89:0/4195823455 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5d04007430 con 0x7f5d180ae670 2024-04-02T09:14:23.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.057+0000 7f5d0fe00700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5cf806aed0 0x7f5cf806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:23.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.057+0000 7f5d0fe00700 1 -- 172.21.15.89:0/4195823455 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f5d04089150 con 0x7f5d180ae670 2024-04-02T09:14:23.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.058+0000 7f5d17e00700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5cf806aed0 0x7f5cf806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:23.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.058+0000 7f5d17e00700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5cf806aed0 0x7f5cf806d390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5d08014040 tx=0x7f5d08008030 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:23.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.062+0000 7f5d0fe00700 1 -- 172.21.15.89:0/4195823455 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5d181506e0 con 0x7f5d180ae670 2024-04-02T09:14:23.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.262+0000 7f5d1ec00700 1 -- 172.21.15.89:0/4195823455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5d181506e0 con 0x7f5d180ae670 2024-04-02T09:14:23.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.797+0000 7f5d0fe00700 1 -- 172.21.15.89:0/4195823455 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v255) v1 ==== 190+0+0 (secure 0 0 0) 0x7f5d181506e0 con 0x7f5d180ae670 2024-04-02T09:14:23.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.801+0000 7f5d1ec00700 1 -- 172.21.15.89:0/4195823455 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5cf806aed0 msgr2=0x7f5cf806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:23.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.801+0000 7f5d1ec00700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5cf806aed0 0x7f5cf806d390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5d08014040 tx=0x7f5d08008030 comp rx=0 tx=0).stop 2024-04-02T09:14:23.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.801+0000 7f5d1ec00700 1 -- 172.21.15.89:0/4195823455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d180ae670 msgr2=0x7f5d1815a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:23.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.801+0000 7f5d1ec00700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d180ae670 0x7f5d1815a860 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400c520 tx=0x7f5d0400c8e0 comp rx=0 tx=0).stop 2024-04-02T09:14:23.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.801+0000 7f5d1ec00700 1 -- 172.21.15.89:0/4195823455 shutdown_connections 2024-04-02T09:14:23.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.801+0000 7f5d1ec00700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5cf806aed0 0x7f5cf806d390 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:23.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.801+0000 7f5d1ec00700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d18155ac0 0x7f5d180b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:23.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.801+0000 7f5d1ec00700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5d1806c910 0x7f5d1814d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:23.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.801+0000 7f5d1ec00700 1 --2- 172.21.15.89:0/4195823455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d180ae670 0x7f5d1815a860 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:23.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.801+0000 7f5d1ec00700 1 -- 172.21.15.89:0/4195823455 >> 172.21.15.89:0/4195823455 conn(0x7f5d180cf210 msgr2=0x7f5d18071d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:23.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.801+0000 7f5d1ec00700 1 -- 172.21.15.89:0/4195823455 shutdown_connections 2024-04-02T09:14:23.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:23.801+0000 7f5d1ec00700 1 -- 172.21.15.89:0/4195823455 wait complete. 2024-04-02T09:14:23.801 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'backup_fs_metadata' removed 2024-04-02T09:14:23.819 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-04-02T09:14:24.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.017+0000 7f194ea00700 1 Processor -- start 2024-04-02T09:14:24.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.017+0000 7f194ea00700 1 -- start start 2024-04-02T09:14:24.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.018+0000 7f194ea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f194815e8d0 0x7f194806a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:24.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.018+0000 7f194ea00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f194806acd0 0x7f194806dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:24.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.018+0000 7f194ea00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f194806b120 con 0x7f19480d0bd0 2024-04-02T09:14:24.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.018+0000 7f194ea00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194815de10 con 0x7f194815e8d0 2024-04-02T09:14:24.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.018+0000 7f194ea00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19480b4fd0 con 0x7f194806acd0 2024-04-02T09:14:24.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.018+0000 7f1947400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f194815e8d0 0x7f194806a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:24.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.018+0000 7f1947400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f194815e8d0 0x7f194806a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41864/0 (socket says 172.21.15.89:41864) 2024-04-02T09:14:24.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.018+0000 7f1947400700 1 -- 172.21.15.89:0/1517911480 learned_addr learned my addr 172.21.15.89:0/1517911480 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:24.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.018+0000 7f1947e00700 1 --1- 172.21.15.89:0/1517911480 >> v1:172.21.15.89:6789/0 conn(0x7f19480d0bd0 0x7f19480b3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56850/0 (socket says 172.21.15.89:56850) 2024-04-02T09:14:24.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.019+0000 7f194ca00700 1 --2- 172.21.15.89:0/1517911480 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f194806acd0 0x7f194806dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:24.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.019+0000 7f1947400700 1 -- 172.21.15.89:0/1517911480 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f194806acd0 msgr2=0x7f194806dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:24.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.019+0000 7f1947400700 1 --2- 172.21.15.89:0/1517911480 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f194806acd0 0x7f194806dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:24.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.019+0000 7f1947400700 1 -- 172.21.15.89:0/1517911480 >> v1:172.21.15.89:6789/0 conn(0x7f19480d0bd0 legacy=0x7f19480b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:24.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.019+0000 7f1947400700 1 -- 172.21.15.89:0/1517911480 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19480b52b0 con 0x7f194815e8d0 2024-04-02T09:14:24.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.019+0000 7f1947400700 1 --2- 172.21.15.89:0/1517911480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f194815e8d0 0x7f194806a790 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7f19340098f0 tx=0x7f1934009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413949032aee775 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:24.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.019+0000 7f1946a00700 1 -- 172.21.15.89:0/1517911480 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193400b8a0 con 0x7f194815e8d0 2024-04-02T09:14:24.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.019+0000 7f1946a00700 1 -- 172.21.15.89:0/1517911480 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193400ba00 con 0x7f194815e8d0 2024-04-02T09:14:24.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.019+0000 7f1946a00700 1 -- 172.21.15.89:0/1517911480 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193400bd90 con 0x7f194815e8d0 2024-04-02T09:14:24.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.020+0000 7f194ea00700 1 -- 172.21.15.89:0/1517911480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f194815e8d0 msgr2=0x7f194806a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:24.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.020+0000 7f194ea00700 1 --2- 172.21.15.89:0/1517911480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f194815e8d0 0x7f194806a790 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7f19340098f0 tx=0x7f1934009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:24.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.020+0000 7f194ea00700 1 -- 172.21.15.89:0/1517911480 shutdown_connections 2024-04-02T09:14:24.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.020+0000 7f194ea00700 1 --2- 172.21.15.89:0/1517911480 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f194806acd0 0x7f194806dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:24.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.020+0000 7f194ea00700 1 --2- 172.21.15.89:0/1517911480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f194815e8d0 0x7f194806a790 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:24.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.020+0000 7f194ea00700 1 -- 172.21.15.89:0/1517911480 >> 172.21.15.89:0/1517911480 conn(0x7f19480b5640 msgr2=0x7f19480b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:24.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.020+0000 7f194ea00700 1 -- 172.21.15.89:0/1517911480 shutdown_connections 2024-04-02T09:14:24.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.020+0000 7f194ea00700 1 -- 172.21.15.89:0/1517911480 wait complete. 2024-04-02T09:14:24.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.022+0000 7f194ea00700 1 Processor -- start 2024-04-02T09:14:24.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.022+0000 7f194ea00700 1 -- start start 2024-04-02T09:14:24.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f194ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f194806acd0 0x7f19481571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:24.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f194ea00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19480d0bd0 0x7f19481579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:24.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f194ea00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f194815e8d0 0x7f1948156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:24.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f194ea00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1948158e10 con 0x7f194806acd0 2024-04-02T09:14:24.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f194ea00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1948157ef0 con 0x7f19480d0bd0 2024-04-02T09:14:24.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f194ea00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1948158190 con 0x7f194815e8d0 2024-04-02T09:14:24.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f1947e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f194806acd0 0x7f19481571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:24.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f1947e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f194806acd0 0x7f19481571e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41868/0 (socket says 172.21.15.89:41868) 2024-04-02T09:14:24.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f1947e00700 1 -- 172.21.15.89:0/854116722 learned_addr learned my addr 172.21.15.89:0/854116722 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:24.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f194ca00700 1 --2- 172.21.15.89:0/854116722 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f194815e8d0 0x7f1948156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:24.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f1947400700 1 --2- 172.21.15.89:0/854116722 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19480d0bd0 0x7f19481579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:24.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f1947e00700 1 -- 172.21.15.89:0/854116722 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f194815e8d0 msgr2=0x7f1948156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:24.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f1947e00700 1 --2- 172.21.15.89:0/854116722 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f194815e8d0 0x7f1948156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:24.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f1947e00700 1 -- 172.21.15.89:0/854116722 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19480d0bd0 msgr2=0x7f19481579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:24.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f1947e00700 1 --2- 172.21.15.89:0/854116722 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19480d0bd0 0x7f19481579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:24.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.023+0000 7f1947e00700 1 -- 172.21.15.89:0/854116722 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1934009550 con 0x7f194806acd0 2024-04-02T09:14:24.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.024+0000 7f1947e00700 1 --2- 172.21.15.89:0/854116722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f194806acd0 0x7f19481571e0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f193800c520 tx=0x7f193800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:24.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.024+0000 7f193fe00700 1 -- 172.21.15.89:0/854116722 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1938005cb0 con 0x7f194806acd0 2024-04-02T09:14:24.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.024+0000 7f193fe00700 1 -- 172.21.15.89:0/854116722 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1938005e10 con 0x7f194806acd0 2024-04-02T09:14:24.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.024+0000 7f194ea00700 1 -- 172.21.15.89:0/854116722 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1948158470 con 0x7f194806acd0 2024-04-02T09:14:24.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.024+0000 7f193fe00700 1 -- 172.21.15.89:0/854116722 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19380072d0 con 0x7f194806acd0 2024-04-02T09:14:24.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.024+0000 7f194ea00700 1 -- 172.21.15.89:0/854116722 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1948158a30 con 0x7f194806acd0 2024-04-02T09:14:24.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.025+0000 7f194ea00700 1 -- 172.21.15.89:0/854116722 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1948067720 con 0x7f194806acd0 2024-04-02T09:14:24.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.026+0000 7f193fe00700 1 -- 172.21.15.89:0/854116722 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f1938007430 con 0x7f194806acd0 2024-04-02T09:14:24.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.027+0000 7f193fe00700 1 --2- 172.21.15.89:0/854116722 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f192806af00 0x7f192806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:24.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.027+0000 7f193fe00700 1 -- 172.21.15.89:0/854116722 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6671+0+0 (secure 0 0 0) 0x7f1938088d00 con 0x7f194806acd0 2024-04-02T09:14:24.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.027+0000 7f1947400700 1 --2- 172.21.15.89:0/854116722 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f192806af00 0x7f192806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:24.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.028+0000 7f1947400700 1 --2- 172.21.15.89:0/854116722 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f192806af00 0x7f192806d3c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1934014040 tx=0x7f1934008030 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:24.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.031+0000 7f193fe00700 1 -- 172.21.15.89:0/854116722 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1938053870 con 0x7f194806acd0 2024-04-02T09:14:24.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.232+0000 7f194ea00700 1 -- 172.21.15.89:0/854116722 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f194806b120 con 0x7f194806acd0 2024-04-02T09:14:24.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.809+0000 7f193fe00700 1 -- 172.21.15.89:0/854116722 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v256) v1 ==== 178+0+0 (secure 0 0 0) 0x7f194806b120 con 0x7f194806acd0 2024-04-02T09:14:24.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.812+0000 7f194ea00700 1 -- 172.21.15.89:0/854116722 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f192806af00 msgr2=0x7f192806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:24.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.812+0000 7f194ea00700 1 --2- 172.21.15.89:0/854116722 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f192806af00 0x7f192806d3c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1934014040 tx=0x7f1934008030 comp rx=0 tx=0).stop 2024-04-02T09:14:24.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.812+0000 7f194ea00700 1 -- 172.21.15.89:0/854116722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f194806acd0 msgr2=0x7f19481571e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:24.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.812+0000 7f194ea00700 1 --2- 172.21.15.89:0/854116722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f194806acd0 0x7f19481571e0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f193800c520 tx=0x7f193800c8e0 comp rx=0 tx=0).stop 2024-04-02T09:14:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.812+0000 7f194ea00700 1 -- 172.21.15.89:0/854116722 shutdown_connections 2024-04-02T09:14:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.812+0000 7f194ea00700 1 --2- 172.21.15.89:0/854116722 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f194815e8d0 0x7f1948156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.812+0000 7f194ea00700 1 --2- 172.21.15.89:0/854116722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f194806acd0 0x7f19481571e0 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.812+0000 7f194ea00700 1 --2- 172.21.15.89:0/854116722 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f192806af00 0x7f192806d3c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.812+0000 7f194ea00700 1 --2- 172.21.15.89:0/854116722 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19480d0bd0 0x7f19481579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.812+0000 7f194ea00700 1 -- 172.21.15.89:0/854116722 >> 172.21.15.89:0/854116722 conn(0x7f19480b5640 msgr2=0x7f194806dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.812+0000 7f194ea00700 1 -- 172.21.15.89:0/854116722 shutdown_connections 2024-04-02T09:14:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:24.812+0000 7f194ea00700 1 -- 172.21.15.89:0/854116722 wait complete. 2024-04-02T09:14:24.811 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'backup_fs_data' removed 2024-04-02T09:14:24.832 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T09:14:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.033+0000 7f2ef7c00700 1 Processor -- start 2024-04-02T09:14:25.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.033+0000 7f2ef7c00700 1 -- start start 2024-04-02T09:14:25.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.034+0000 7f2ef7c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef00ce230 0x7f2ef00ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.035+0000 7f2ef7c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef00709c0 con 0x7f2ef015d800 2024-04-02T09:14:25.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.035+0000 7f2ef7c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef0070c80 con 0x7f2ef00b2070 2024-04-02T09:14:25.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.035+0000 7f2ef7c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef00b40f0 con 0x7f2ef00ce230 2024-04-02T09:14:25.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.035+0000 7f2ef5200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2ef015d800 0x7f2ef00725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56854/0 (socket says 172.21.15.89:56854) 2024-04-02T09:14:25.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.035+0000 7f2ef5200700 1 -- 172.21.15.89:0/1721519527 learned_addr learned my addr 172.21.15.89:0/1721519527 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:25.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.035+0000 7f2eefe00700 1 --2- 172.21.15.89:0/1721519527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef00ce230 0x7f2ef00ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.035+0000 7f2eef400700 1 -- 172.21.15.89:0/1721519527 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115871744 0 0) 0x7f2ef00709c0 con 0x7f2ef015d800 2024-04-02T09:14:25.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.035+0000 7f2eef400700 1 -- 172.21.15.89:0/1721519527 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed80035b0 con 0x7f2ef015d800 2024-04-02T09:14:25.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.035+0000 7f2eef400700 1 -- 172.21.15.89:0/1721519527 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 257924380 0 0) 0x7f2ed80035b0 con 0x7f2ef015d800 2024-04-02T09:14:25.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2eef400700 1 -- 172.21.15.89:0/1721519527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef00ce230 msgr2=0x7f2ef00ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:25.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2eef400700 1 --2- 172.21.15.89:0/1721519527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef00ce230 0x7f2ef00ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2eef400700 1 -- 172.21.15.89:0/1721519527 >> v1:172.21.15.113:6789/0 conn(0x7f2ef00b2070 legacy=0x7f2ef015dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2eef400700 1 -- 172.21.15.89:0/1721519527 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef0070f60 con 0x7f2ef015d800 2024-04-02T09:14:25.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2eef400700 1 -- 172.21.15.89:0/1721519527 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f2ee4001670 con 0x7f2ef015d800 2024-04-02T09:14:25.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2eef400700 1 -- 172.21.15.89:0/1721519527 >> v1:172.21.15.89:6789/0 conn(0x7f2ef015d800 legacy=0x7f2ef00725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2eef400700 1 --2- 172.21.15.89:0/1721519527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef00ce230 0x7f2ed8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2eef400700 1 --2- 172.21.15.89:0/1721519527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ed8003f70 0x7f2ed8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2eef400700 1 --2- 172.21.15.89:0/1721519527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ed8004d90 0x7f2ed8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2eef400700 1 -- 172.21.15.89:0/1721519527 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef00b40f0 con 0x7f2ed8004d90 2024-04-02T09:14:25.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2eef400700 1 -- 172.21.15.89:0/1721519527 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed800bac0 con 0x7f2ed8003f70 2024-04-02T09:14:25.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2eef400700 1 -- 172.21.15.89:0/1721519527 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed800bc80 con 0x7f2ef00ce230 2024-04-02T09:14:25.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2ef5c00700 1 --2- 172.21.15.89:0/1721519527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ed8004d90 0x7f2ed8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2ef5200700 1 --2- 172.21.15.89:0/1721519527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef00ce230 0x7f2ed8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.036+0000 7f2eefe00700 1 --2- 172.21.15.89:0/1721519527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ed8003f70 0x7f2ed8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.037+0000 7f2ef5c00700 1 -- 172.21.15.89:0/1721519527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef00ce230 msgr2=0x7f2ed8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:25.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.037+0000 7f2ef5c00700 1 --2- 172.21.15.89:0/1721519527 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ef00ce230 0x7f2ed8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.037+0000 7f2ef5c00700 1 -- 172.21.15.89:0/1721519527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ed8003f70 msgr2=0x7f2ed8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:25.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.037+0000 7f2ef5c00700 1 --2- 172.21.15.89:0/1721519527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ed8003f70 0x7f2ed8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.037+0000 7f2ef5c00700 1 -- 172.21.15.89:0/1721519527 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef0070f60 con 0x7f2ed8004d90 2024-04-02T09:14:25.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.037+0000 7f2eefe00700 1 --2- 172.21.15.89:0/1721519527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ed8003f70 0x7f2ed8004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:25.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.037+0000 7f2ef5c00700 1 --2- 172.21.15.89:0/1721519527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ed8004d90 0x7f2ed8005210 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7f2ee800c290 tx=0x7f2ed8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc75cde4426c400e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:25.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.037+0000 7f2eef400700 1 -- 172.21.15.89:0/1721519527 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8001e70 con 0x7f2ed8004d90 2024-04-02T09:14:25.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.037+0000 7f2eef400700 1 -- 172.21.15.89:0/1721519527 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee800ce50 con 0x7f2ed8004d90 2024-04-02T09:14:25.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.037+0000 7f2eef400700 1 -- 172.21.15.89:0/1721519527 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8012660 con 0x7f2ed8004d90 2024-04-02T09:14:25.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.038+0000 7f2ef7c00700 1 -- 172.21.15.89:0/1721519527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ed8004d90 msgr2=0x7f2ed8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.038+0000 7f2ef7c00700 1 --2- 172.21.15.89:0/1721519527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ed8004d90 0x7f2ed8005210 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7f2ee800c290 tx=0x7f2ed8005880 comp rx=0 tx=0).stop 2024-04-02T09:14:25.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.038+0000 7f2ef5c00700 1 -- 172.21.15.89:0/1721519527 reap_dead start 2024-04-02T09:14:25.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.038+0000 7f2ef7c00700 1 -- 172.21.15.89:0/1721519527 shutdown_connections 2024-04-02T09:14:25.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.038+0000 7f2ef7c00700 1 -- 172.21.15.89:0/1721519527 >> 172.21.15.89:0/1721519527 conn(0x7f2ef00b0d70 msgr2=0x7f2ef00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:25.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.038+0000 7f2ef7c00700 1 -- 172.21.15.89:0/1721519527 shutdown_connections 2024-04-02T09:14:25.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.038+0000 7f2ef7c00700 1 -- 172.21.15.89:0/1721519527 wait complete. 2024-04-02T09:14:25.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.041+0000 7f2ef7c00700 1 Processor -- start 2024-04-02T09:14:25.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.041+0000 7f2ef7c00700 1 -- start start 2024-04-02T09:14:25.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.042+0000 7f2ef7c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ed8004d90 0x7f2ef0158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.042+0000 7f2ef7c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef0152680 0x7f2ef0152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.042+0000 7f2ef7c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ef0153430 0x7f2ef015dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.042+0000 7f2ef7c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef00ad860 con 0x7f2ef0153430 2024-04-02T09:14:25.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.042+0000 7f2ef7c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef00ad230 con 0x7f2ef0152680 2024-04-02T09:14:25.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.042+0000 7f2ef7c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef00cb8b0 con 0x7f2ed8004d90 2024-04-02T09:14:25.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.043+0000 7f2eefe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef0152680 0x7f2ef0152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.043+0000 7f2ef5c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ef0153430 0x7f2ef015dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.043+0000 7f2ef5200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ed8004d90 0x7f2ef0158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.043+0000 7f2ef5200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ed8004d90 0x7f2ef0158720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:53220/0 (socket says 172.21.15.89:53220) 2024-04-02T09:14:25.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.043+0000 7f2ef5200700 1 -- 172.21.15.89:0/2626849004 learned_addr learned my addr 172.21.15.89:0/2626849004 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:25.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.044+0000 7f2ef5200700 1 -- 172.21.15.89:0/2626849004 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef0152680 msgr2=0x7f2ef0152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.044+0000 7f2ef5200700 1 --2- 172.21.15.89:0/2626849004 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef0152680 0x7f2ef0152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.044+0000 7f2ef5200700 1 -- 172.21.15.89:0/2626849004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ef0153430 msgr2=0x7f2ef015dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.044+0000 7f2ef5200700 1 --2- 172.21.15.89:0/2626849004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ef0153430 0x7f2ef015dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.044+0000 7f2ef5200700 1 -- 172.21.15.89:0/2626849004 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef0157e60 con 0x7f2ed8004d90 2024-04-02T09:14:25.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.044+0000 7f2ef5200700 1 --2- 172.21.15.89:0/2626849004 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ed8004d90 0x7f2ef0158720 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4000fd0 tx=0x7f2ee400ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:25.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.045+0000 7f2eecc00700 1 -- 172.21.15.89:0/2626849004 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee400c620 con 0x7f2ed8004d90 2024-04-02T09:14:25.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.045+0000 7f2eecc00700 1 -- 172.21.15.89:0/2626849004 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee400c780 con 0x7f2ed8004d90 2024-04-02T09:14:25.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.045+0000 7f2eecc00700 1 -- 172.21.15.89:0/2626849004 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee400caf0 con 0x7f2ed8004d90 2024-04-02T09:14:25.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.045+0000 7f2ef7c00700 1 -- 172.21.15.89:0/2626849004 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef0158140 con 0x7f2ed8004d90 2024-04-02T09:14:25.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.045+0000 7f2ef7c00700 1 -- 172.21.15.89:0/2626849004 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef0158c60 con 0x7f2ed8004d90 2024-04-02T09:14:25.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.046+0000 7f2ef7c00700 1 -- 172.21.15.89:0/2626849004 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef0067720 con 0x7f2ed8004d90 2024-04-02T09:14:25.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.049+0000 7f2eecc00700 1 -- 172.21.15.89:0/2626849004 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f2ee400cc50 con 0x7f2ed8004d90 2024-04-02T09:14:25.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.050+0000 7f2eecc00700 1 --2- 172.21.15.89:0/2626849004 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ed006af00 0x7f2ed006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.050+0000 7f2eecc00700 1 -- 172.21.15.89:0/2626849004 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f2ee4089680 con 0x7f2ed8004d90 2024-04-02T09:14:25.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.051+0000 7f2eefe00700 1 --2- 172.21.15.89:0/2626849004 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ed006af00 0x7f2ed006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.052+0000 7f2eefe00700 1 --2- 172.21.15.89:0/2626849004 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ed006af00 0x7f2ed006d3c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2ef01576f0 tx=0x7f2edc008040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:25.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.055+0000 7f2eecc00700 1 -- 172.21.15.89:0/2626849004 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2ee40543a0 con 0x7f2ed8004d90 2024-04-02T09:14:25.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.256+0000 7f2ef7c00700 1 -- 172.21.15.89:0/2626849004 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f2ef006b9a0 con 0x7f2ed8004d90 2024-04-02T09:14:25.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.262+0000 7f2eecc00700 1 -- 172.21.15.89:0/2626849004 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v85) v1 ==== 115+0+0 (secure 0 0 0) 0x7f2ee40579c0 con 0x7f2ed8004d90 2024-04-02T09:14:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.264+0000 7f2ef7c00700 1 -- 172.21.15.89:0/2626849004 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ed006af00 msgr2=0x7f2ed006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.264+0000 7f2ef7c00700 1 --2- 172.21.15.89:0/2626849004 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ed006af00 0x7f2ed006d3c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2ef01576f0 tx=0x7f2edc008040 comp rx=0 tx=0).stop 2024-04-02T09:14:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.265+0000 7f2ef7c00700 1 -- 172.21.15.89:0/2626849004 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ed8004d90 msgr2=0x7f2ef0158720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.265+0000 7f2ef7c00700 1 --2- 172.21.15.89:0/2626849004 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ed8004d90 0x7f2ef0158720 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4000fd0 tx=0x7f2ee400ebf0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.265+0000 7f2ef7c00700 1 -- 172.21.15.89:0/2626849004 shutdown_connections 2024-04-02T09:14:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.265+0000 7f2ef7c00700 1 --2- 172.21.15.89:0/2626849004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ef0153430 0x7f2ef015dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.265+0000 7f2ef7c00700 1 --2- 172.21.15.89:0/2626849004 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2ed8004d90 0x7f2ef0158720 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.265+0000 7f2ef7c00700 1 --2- 172.21.15.89:0/2626849004 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2ed006af00 0x7f2ed006d3c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.265+0000 7f2ef7c00700 1 --2- 172.21.15.89:0/2626849004 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef0152680 0x7f2ef0152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.265+0000 7f2ef7c00700 1 -- 172.21.15.89:0/2626849004 >> 172.21.15.89:0/2626849004 conn(0x7f2ef00b0d70 msgr2=0x7f2ef00b0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.265+0000 7f2ef7c00700 1 -- 172.21.15.89:0/2626849004 shutdown_connections 2024-04-02T09:14:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.265+0000 7f2ef7c00700 1 -- 172.21.15.89:0/2626849004 wait complete. 2024-04-02T09:14:25.282 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T09:14:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.477+0000 7f8d9b800700 1 Processor -- start 2024-04-02T09:14:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.478+0000 7f8d9b800700 1 -- start start 2024-04-02T09:14:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.479+0000 7f8d9b800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d941611b0 0x7f8d9415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.479+0000 7f8d9b800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8d9415f4d0 0x7f8d940ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.479+0000 7f8d9b800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8d940d21f0 0x7f8d940cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.479+0000 7f8d9b800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d940cf180 con 0x7f8d940d21f0 2024-04-02T09:14:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.479+0000 7f8d9b800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d940ce270 con 0x7f8d9415f4d0 2024-04-02T09:14:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.479+0000 7f8d9b800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9415b600 con 0x7f8d941611b0 2024-04-02T09:14:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.479+0000 7f8d99800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8d940d21f0 0x7f8d940cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.479+0000 7f8d99800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8d940d21f0 0x7f8d940cec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41878/0 (socket says 172.21.15.89:41878) 2024-04-02T09:14:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.479+0000 7f8d99800700 1 -- 172.21.15.89:0/1730118286 learned_addr learned my addr 172.21.15.89:0/1730118286 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.479+0000 7f8d98e00700 1 --2- 172.21.15.89:0/1730118286 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d941611b0 0x7f8d9415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.479+0000 7f8d93e00700 1 --2- 172.21.15.89:0/1730118286 >> v2:172.21.15.113:3300/0 conn(0x7f8d9415f4d0 0x7f8d940ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.480+0000 7f8d99800700 1 -- 172.21.15.89:0/1730118286 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d941611b0 msgr2=0x7f8d9415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.480+0000 7f8d99800700 1 --2- 172.21.15.89:0/1730118286 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d941611b0 0x7f8d9415ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.480+0000 7f8d99800700 1 -- 172.21.15.89:0/1730118286 >> v2:172.21.15.113:3300/0 conn(0x7f8d9415f4d0 msgr2=0x7f8d940ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.480+0000 7f8d99800700 1 --2- 172.21.15.89:0/1730118286 >> v2:172.21.15.113:3300/0 conn(0x7f8d9415f4d0 0x7f8d940ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.480+0000 7f8d99800700 1 -- 172.21.15.89:0/1730118286 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d940d2760 con 0x7f8d940d21f0 2024-04-02T09:14:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.480+0000 7f8d99800700 1 --2- 172.21.15.89:0/1730118286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d940d21f0 0x7f8d940cec40 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f8d9408d540 tx=0x7f8d8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901df6ff7e1083ba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.480+0000 7f8d93400700 1 -- 172.21.15.89:0/1730118286 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c00e070 con 0x7f8d940d21f0 2024-04-02T09:14:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.480+0000 7f8d93400700 1 -- 172.21.15.89:0/1730118286 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8c00b9c0 con 0x7f8d940d21f0 2024-04-02T09:14:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.480+0000 7f8d93400700 1 -- 172.21.15.89:0/1730118286 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c00bd70 con 0x7f8d940d21f0 2024-04-02T09:14:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.481+0000 7f8d9b800700 1 -- 172.21.15.89:0/1730118286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d940d21f0 msgr2=0x7f8d940cec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.481+0000 7f8d9b800700 1 --2- 172.21.15.89:0/1730118286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d940d21f0 0x7f8d940cec40 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f8d9408d540 tx=0x7f8d8c00b330 comp rx=0 tx=0).stop 2024-04-02T09:14:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.481+0000 7f8d9b800700 1 -- 172.21.15.89:0/1730118286 shutdown_connections 2024-04-02T09:14:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.481+0000 7f8d9b800700 1 --2- 172.21.15.89:0/1730118286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d940d21f0 0x7f8d940cec40 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.481+0000 7f8d9b800700 1 --2- 172.21.15.89:0/1730118286 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d941611b0 0x7f8d9415ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.481+0000 7f8d9b800700 1 --2- 172.21.15.89:0/1730118286 >> v2:172.21.15.113:3300/0 conn(0x7f8d9415f4d0 0x7f8d940ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.481+0000 7f8d9b800700 1 -- 172.21.15.89:0/1730118286 >> 172.21.15.89:0/1730118286 conn(0x7f8d9408aff0 msgr2=0x7f8d940d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.481+0000 7f8d9b800700 1 -- 172.21.15.89:0/1730118286 shutdown_connections 2024-04-02T09:14:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.481+0000 7f8d9b800700 1 -- 172.21.15.89:0/1730118286 wait complete. 2024-04-02T09:14:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.483+0000 7f8d9b800700 1 Processor -- start 2024-04-02T09:14:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.483+0000 7f8d9b800700 1 -- start start 2024-04-02T09:14:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d9b800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d940d21f0 0x7f8d9415ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d9b800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d9415f4d0 0x7f8d940cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d9b800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d941611b0 0x7f8d940cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d9b800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d940cd0d0 con 0x7f8d9415f4d0 2024-04-02T09:14:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d9b800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d940b91c0 con 0x7f8d941611b0 2024-04-02T09:14:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d9b800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d940cc4c0 con 0x7f8d940d21f0 2024-04-02T09:14:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d93e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d9415f4d0 0x7f8d940cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d93e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d9415f4d0 0x7f8d940cb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41892/0 (socket says 172.21.15.89:41892) 2024-04-02T09:14:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d93e00700 1 -- 172.21.15.89:0/3424095702 learned_addr learned my addr 172.21.15.89:0/3424095702 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d99800700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d941611b0 0x7f8d940cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d98e00700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d940d21f0 0x7f8d9415ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d93e00700 1 -- 172.21.15.89:0/3424095702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d940d21f0 msgr2=0x7f8d9415ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d93e00700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d940d21f0 0x7f8d9415ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d93e00700 1 -- 172.21.15.89:0/3424095702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d941611b0 msgr2=0x7f8d940cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d93e00700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d941611b0 0x7f8d940cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d93e00700 1 -- 172.21.15.89:0/3424095702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c009b30 con 0x7f8d9415f4d0 2024-04-02T09:14:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.485+0000 7f8d99800700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d941611b0 0x7f8d940cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.486+0000 7f8d93e00700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d9415f4d0 0x7f8d940cb900 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000e7e0 tx=0x7f8d8000eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.486+0000 7f8d90c00700 1 -- 172.21.15.89:0/3424095702 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d80013610 con 0x7f8d9415f4d0 2024-04-02T09:14:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.486+0000 7f8d90c00700 1 -- 172.21.15.89:0/3424095702 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d80013770 con 0x7f8d9415f4d0 2024-04-02T09:14:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.486+0000 7f8d9b800700 1 -- 172.21.15.89:0/3424095702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d940cc7a0 con 0x7f8d9415f4d0 2024-04-02T09:14:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.486+0000 7f8d90c00700 1 -- 172.21.15.89:0/3424095702 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d80013b00 con 0x7f8d9415f4d0 2024-04-02T09:14:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.486+0000 7f8d9b800700 1 -- 172.21.15.89:0/3424095702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d940ccd60 con 0x7f8d9415f4d0 2024-04-02T09:14:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.487+0000 7f8d9b800700 1 -- 172.21.15.89:0/3424095702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d940cd240 con 0x7f8d9415f4d0 2024-04-02T09:14:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.488+0000 7f8d90c00700 1 -- 172.21.15.89:0/3424095702 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f8d80013c60 con 0x7f8d9415f4d0 2024-04-02T09:14:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.489+0000 7f8d90c00700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8d7406aed0 0x7f8d7406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.489+0000 7f8d90c00700 1 -- 172.21.15.89:0/3424095702 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f8d80089770 con 0x7f8d9415f4d0 2024-04-02T09:14:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.489+0000 7f8d98e00700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8d7406aed0 0x7f8d7406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.490+0000 7f8d98e00700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8d7406aed0 0x7f8d7406d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800b9b0 tx=0x7f8d88008040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.493+0000 7f8d90c00700 1 -- 172.21.15.89:0/3424095702 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8d80054220 con 0x7f8d9415f4d0 2024-04-02T09:14:25.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.696+0000 7f8d9b800700 1 -- 172.21.15.89:0/3424095702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f8d94067720 con 0x7f8d9415f4d0 2024-04-02T09:14:25.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.702+0000 7f8d90c00700 1 -- 172.21.15.89:0/3424095702 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v86) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8d80057840 con 0x7f8d9415f4d0 2024-04-02T09:14:25.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.705+0000 7f8d9b800700 1 -- 172.21.15.89:0/3424095702 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8d7406aed0 msgr2=0x7f8d7406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.705+0000 7f8d9b800700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8d7406aed0 0x7f8d7406d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800b9b0 tx=0x7f8d88008040 comp rx=0 tx=0).stop 2024-04-02T09:14:25.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.705+0000 7f8d9b800700 1 -- 172.21.15.89:0/3424095702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d9415f4d0 msgr2=0x7f8d940cb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.705+0000 7f8d9b800700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d9415f4d0 0x7f8d940cb900 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000e7e0 tx=0x7f8d8000eba0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.705+0000 7f8d9b800700 1 -- 172.21.15.89:0/3424095702 shutdown_connections 2024-04-02T09:14:25.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.705+0000 7f8d9b800700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8d7406aed0 0x7f8d7406d390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.705+0000 7f8d9b800700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d941611b0 0x7f8d940cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.705+0000 7f8d9b800700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8d940d21f0 0x7f8d9415ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.705+0000 7f8d9b800700 1 --2- 172.21.15.89:0/3424095702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d9415f4d0 0x7f8d940cb900 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.705+0000 7f8d9b800700 1 -- 172.21.15.89:0/3424095702 >> 172.21.15.89:0/3424095702 conn(0x7f8d9408aff0 msgr2=0x7f8d9415da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:25.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.705+0000 7f8d9b800700 1 -- 172.21.15.89:0/3424095702 shutdown_connections 2024-04-02T09:14:25.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.705+0000 7f8d9b800700 1 -- 172.21.15.89:0/3424095702 wait complete. 2024-04-02T09:14:25.722 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T09:14:25.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.919+0000 7f27c0a00700 1 Processor -- start 2024-04-02T09:14:25.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.920+0000 7f27c0a00700 1 -- start start 2024-04-02T09:14:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.921+0000 7f27c0a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d16a0 0x7f27bc0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.921+0000 7f27c0a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc161420 con 0x7f27bc0d2370 2024-04-02T09:14:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.921+0000 7f27c0a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc161700 con 0x7f27bc15d9d0 2024-04-02T09:14:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.921+0000 7f27c0a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0d2820 con 0x7f27bc0d16a0 2024-04-02T09:14:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.921+0000 7f27ba800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f27bc0d2370 0x7f27bc0ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56864/0 (socket says 172.21.15.89:56864) 2024-04-02T09:14:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.921+0000 7f27ba800700 1 -- 172.21.15.89:0/3215000408 learned_addr learned my addr 172.21.15.89:0/3215000408 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:25.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.921+0000 7f27b9400700 1 --2- 172.21.15.89:0/3215000408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d16a0 0x7f27bc0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.921+0000 7f27b8a00700 1 -- 172.21.15.89:0/3215000408 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639348029 0 0) 0x7f27bc161420 con 0x7f27bc0d2370 2024-04-02T09:14:25.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.921+0000 7f27b8a00700 1 -- 172.21.15.89:0/3215000408 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a00035b0 con 0x7f27bc0d2370 2024-04-02T09:14:25.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b8a00700 1 -- 172.21.15.89:0/3215000408 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 610618034 0 0) 0x7f27a00035b0 con 0x7f27bc0d2370 2024-04-02T09:14:25.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b8a00700 1 -- 172.21.15.89:0/3215000408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d16a0 msgr2=0x7f27bc0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:25.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b8a00700 1 --2- 172.21.15.89:0/3215000408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d16a0 0x7f27bc0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b8a00700 1 -- 172.21.15.89:0/3215000408 >> v1:172.21.15.113:6789/0 conn(0x7f27bc15d9d0 legacy=0x7f27bc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b8a00700 1 -- 172.21.15.89:0/3215000408 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc1619e0 con 0x7f27bc0d2370 2024-04-02T09:14:25.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b8a00700 1 -- 172.21.15.89:0/3215000408 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f27b40019f0 con 0x7f27bc0d2370 2024-04-02T09:14:25.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b8a00700 1 -- 172.21.15.89:0/3215000408 >> v1:172.21.15.89:6789/0 conn(0x7f27bc0d2370 legacy=0x7f27bc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b8a00700 1 --2- 172.21.15.89:0/3215000408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27a0003a30 0x7f27a0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b8a00700 1 --2- 172.21.15.89:0/3215000408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d16a0 0x7f27a0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b8a00700 1 --2- 172.21.15.89:0/3215000408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27a0004d90 0x7f27a0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b8a00700 1 -- 172.21.15.89:0/3215000408 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0d2820 con 0x7f27a0003a30 2024-04-02T09:14:25.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b8a00700 1 -- 172.21.15.89:0/3215000408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a000baf0 con 0x7f27a0004d90 2024-04-02T09:14:25.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b8a00700 1 -- 172.21.15.89:0/3215000408 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a000bcb0 con 0x7f27bc0d16a0 2024-04-02T09:14:25.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b9e00700 1 --2- 172.21.15.89:0/3215000408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27a0003a30 0x7f27a0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b9400700 1 --2- 172.21.15.89:0/3215000408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d16a0 0x7f27a0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27ba800700 1 --2- 172.21.15.89:0/3215000408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27a0004d90 0x7f27a0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b9e00700 1 -- 172.21.15.89:0/3215000408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d16a0 msgr2=0x7f27a0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:25.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b9e00700 1 --2- 172.21.15.89:0/3215000408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0d16a0 0x7f27a0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.922+0000 7f27b9e00700 1 -- 172.21.15.89:0/3215000408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27a0004d90 msgr2=0x7f27a0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.923+0000 7f27b9e00700 1 --2- 172.21.15.89:0/3215000408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27a0004d90 0x7f27a0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.923+0000 7f27b9e00700 1 -- 172.21.15.89:0/3215000408 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a00035b0 con 0x7f27a0003a30 2024-04-02T09:14:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.923+0000 7f27b9e00700 1 --2- 172.21.15.89:0/3215000408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27a0003a30 0x7f27a0003e50 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f27ac007fd0 tx=0x7f27a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=258006055a16f635 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.923+0000 7f27b8a00700 1 -- 172.21.15.89:0/3215000408 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac0052d0 con 0x7f27a0003a30 2024-04-02T09:14:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.923+0000 7f27b8a00700 1 -- 172.21.15.89:0/3215000408 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ac005470 con 0x7f27a0003a30 2024-04-02T09:14:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.923+0000 7f27b8a00700 1 -- 172.21.15.89:0/3215000408 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac0038c0 con 0x7f27a0003a30 2024-04-02T09:14:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.923+0000 7f27c0a00700 1 -- 172.21.15.89:0/3215000408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27a0003a30 msgr2=0x7f27a0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.923+0000 7f27c0a00700 1 --2- 172.21.15.89:0/3215000408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27a0003a30 0x7f27a0003e50 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f27ac007fd0 tx=0x7f27a0005880 comp rx=0 tx=0).stop 2024-04-02T09:14:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.923+0000 7f27ba800700 1 -- 172.21.15.89:0/3215000408 reap_dead start 2024-04-02T09:14:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.924+0000 7f27c0a00700 1 -- 172.21.15.89:0/3215000408 shutdown_connections 2024-04-02T09:14:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.924+0000 7f27c0a00700 1 -- 172.21.15.89:0/3215000408 >> 172.21.15.89:0/3215000408 conn(0x7f27bc08aff0 msgr2=0x7f27bc0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.924+0000 7f27c0a00700 1 -- 172.21.15.89:0/3215000408 shutdown_connections 2024-04-02T09:14:25.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.924+0000 7f27c0a00700 1 -- 172.21.15.89:0/3215000408 wait complete. 2024-04-02T09:14:25.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.925+0000 7f27c0a00700 1 Processor -- start 2024-04-02T09:14:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.926+0000 7f27c0a00700 1 -- start start 2024-04-02T09:14:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.926+0000 7f27c0a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27a0003a30 0x7f27bc158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.926+0000 7f27c0a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0cafc0 0x7f27bc0cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.926+0000 7f27c0a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27bc0b4300 0x7f27bc0b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27c0a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0b4cf0 con 0x7f27a0003a30 2024-04-02T09:14:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27c0a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0cba60 con 0x7f27bc0b4300 2024-04-02T09:14:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27c0a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0cbd40 con 0x7f27bc0cafc0 2024-04-02T09:14:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27b9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27a0003a30 0x7f27bc158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27b9e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27a0003a30 0x7f27bc158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41906/0 (socket says 172.21.15.89:41906) 2024-04-02T09:14:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27b9e00700 1 -- 172.21.15.89:0/1883745420 learned_addr learned my addr 172.21.15.89:0/1883745420 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27ba800700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27bc0b4300 0x7f27bc0b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27b9400700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0cafc0 0x7f27bc0cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27b9e00700 1 -- 172.21.15.89:0/1883745420 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0cafc0 msgr2=0x7f27bc0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27b9e00700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0cafc0 0x7f27bc0cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27b9e00700 1 -- 172.21.15.89:0/1883745420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27bc0b4300 msgr2=0x7f27bc0b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:25.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27b9e00700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27bc0b4300 0x7f27bc0b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:25.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27b9e00700 1 -- 172.21.15.89:0/1883745420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc1f4d80 con 0x7f27a0003a30 2024-04-02T09:14:25.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27b9400700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0cafc0 0x7f27bc0cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:25.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.927+0000 7f27b9e00700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27a0003a30 0x7f27bc158a10 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f27ac004000 tx=0x7f27ac001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:25.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.928+0000 7f27b2000700 1 -- 172.21.15.89:0/1883745420 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac007240 con 0x7f27a0003a30 2024-04-02T09:14:25.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.928+0000 7f27b2000700 1 -- 172.21.15.89:0/1883745420 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ac0073a0 con 0x7f27a0003a30 2024-04-02T09:14:25.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.928+0000 7f27b2000700 1 -- 172.21.15.89:0/1883745420 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac001290 con 0x7f27a0003a30 2024-04-02T09:14:25.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.928+0000 7f27c0a00700 1 -- 172.21.15.89:0/1883745420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc1f5060 con 0x7f27a0003a30 2024-04-02T09:14:25.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.928+0000 7f27c0a00700 1 -- 172.21.15.89:0/1883745420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc1f3d20 con 0x7f27a0003a30 2024-04-02T09:14:25.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.929+0000 7f27c0a00700 1 -- 172.21.15.89:0/1883745420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27bc067720 con 0x7f27a0003a30 2024-04-02T09:14:25.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.930+0000 7f27b2000700 1 -- 172.21.15.89:0/1883745420 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f27ac003680 con 0x7f27a0003a30 2024-04-02T09:14:25.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.931+0000 7f27b2000700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f279c06aed0 0x7f279c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:25.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.931+0000 7f27b2000700 1 -- 172.21.15.89:0/1883745420 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f27ac08a450 con 0x7f27a0003a30 2024-04-02T09:14:25.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.931+0000 7f27b9400700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f279c06aed0 0x7f279c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:25.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.932+0000 7f27b9400700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f279c06aed0 0x7f279c06d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f27bc160f90 tx=0x7f27a8008040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:25.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:25.935+0000 7f27b2000700 1 -- 172.21.15.89:0/1883745420 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f27ac055330 con 0x7f27a0003a30 2024-04-02T09:14:26.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.139+0000 7f27c0a00700 1 -- 172.21.15.89:0/1883745420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f27bc0cf670 con 0x7f27a0003a30 2024-04-02T09:14:26.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.145+0000 7f27b2000700 1 -- 172.21.15.89:0/1883745420 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v87) v1 ==== 104+0+0 (secure 0 0 0) 0x7f27ac058950 con 0x7f27a0003a30 2024-04-02T09:14:26.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.147+0000 7f27c0a00700 1 -- 172.21.15.89:0/1883745420 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f279c06aed0 msgr2=0x7f279c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:26.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.147+0000 7f27c0a00700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f279c06aed0 0x7f279c06d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f27bc160f90 tx=0x7f27a8008040 comp rx=0 tx=0).stop 2024-04-02T09:14:26.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.147+0000 7f27c0a00700 1 -- 172.21.15.89:0/1883745420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27a0003a30 msgr2=0x7f27bc158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:26.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.147+0000 7f27c0a00700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27a0003a30 0x7f27bc158a10 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f27ac004000 tx=0x7f27ac001120 comp rx=0 tx=0).stop 2024-04-02T09:14:26.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.147+0000 7f27c0a00700 1 -- 172.21.15.89:0/1883745420 shutdown_connections 2024-04-02T09:14:26.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.147+0000 7f27c0a00700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f279c06aed0 0x7f279c06d390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:26.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.147+0000 7f27c0a00700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27bc0b4300 0x7f27bc0b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:26.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.147+0000 7f27c0a00700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27a0003a30 0x7f27bc158a10 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:26.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.147+0000 7f27c0a00700 1 --2- 172.21.15.89:0/1883745420 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f27bc0cafc0 0x7f27bc0cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:26.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.147+0000 7f27c0a00700 1 -- 172.21.15.89:0/1883745420 >> 172.21.15.89:0/1883745420 conn(0x7f27bc08aff0 msgr2=0x7f27bc0d3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:26.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.148+0000 7f27c0a00700 1 -- 172.21.15.89:0/1883745420 shutdown_connections 2024-04-02T09:14:26.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.148+0000 7f27c0a00700 1 -- 172.21.15.89:0/1883745420 wait complete. 2024-04-02T09:14:26.163 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-04-02T09:14:26.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.358+0000 7f7b00000700 1 Processor -- start 2024-04-02T09:14:26.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.358+0000 7f7b00000700 1 -- start start 2024-04-02T09:14:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.359+0000 7f7b00000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7af80cf3c0 0x7f7af80d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.359+0000 7f7b00000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7af815db40 0x7f7af80cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.359+0000 7f7b00000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af80d1330 con 0x7f7af815d760 2024-04-02T09:14:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.359+0000 7f7b00000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af80d1fb0 con 0x7f7af815db40 2024-04-02T09:14:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.359+0000 7f7b00000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80d1610 con 0x7f7af80cf3c0 2024-04-02T09:14:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.359+0000 7f7afd600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7af815d760 0x7f7af80cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56878/0 (socket says 172.21.15.89:56878) 2024-04-02T09:14:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.359+0000 7f7afd600700 1 -- 172.21.15.89:0/1465891503 learned_addr learned my addr 172.21.15.89:0/1465891503 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.360+0000 7f7afcc00700 1 --2- 172.21.15.89:0/1465891503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7af80cf3c0 0x7f7af80d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.360+0000 7f7afe000700 1 --2- 172.21.15.89:0/1465891503 >> v2:172.21.15.113:3300/0 conn(0x7f7af815db40 0x7f7af80cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.360+0000 7f7af7e00700 1 -- 172.21.15.89:0/1465891503 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384305921 0 0) 0x7f7af80d1330 con 0x7f7af815d760 2024-04-02T09:14:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.360+0000 7f7af7e00700 1 -- 172.21.15.89:0/1465891503 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ae00035b0 con 0x7f7af815d760 2024-04-02T09:14:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.360+0000 7f7afcc00700 1 -- 172.21.15.89:0/1465891503 >> v2:172.21.15.113:3300/0 conn(0x7f7af815db40 msgr2=0x7f7af80cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.360+0000 7f7afcc00700 1 --2- 172.21.15.89:0/1465891503 >> v2:172.21.15.113:3300/0 conn(0x7f7af815db40 0x7f7af80cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:26.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.360+0000 7f7afcc00700 1 -- 172.21.15.89:0/1465891503 >> v1:172.21.15.89:6789/0 conn(0x7f7af815d760 legacy=0x7f7af80cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.360+0000 7f7afcc00700 1 -- 172.21.15.89:0/1465891503 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af80d18f0 con 0x7f7af80cf3c0 2024-04-02T09:14:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.360+0000 7f7afcc00700 1 --2- 172.21.15.89:0/1465891503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7af80cf3c0 0x7f7af80d23d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009840 tx=0x7f7ae4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb812edb4e55b211 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.361+0000 7f7af7e00700 1 -- 172.21.15.89:0/1465891503 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4011830 con 0x7f7af80cf3c0 2024-04-02T09:14:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.361+0000 7f7af7e00700 1 -- 172.21.15.89:0/1465891503 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae4011990 con 0x7f7af80cf3c0 2024-04-02T09:14:26.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.361+0000 7f7af7e00700 1 -- 172.21.15.89:0/1465891503 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4011d20 con 0x7f7af80cf3c0 2024-04-02T09:14:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.361+0000 7f7b00000700 1 -- 172.21.15.89:0/1465891503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7af80cf3c0 msgr2=0x7f7af80d23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.361+0000 7f7b00000700 1 --2- 172.21.15.89:0/1465891503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7af80cf3c0 0x7f7af80d23d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009840 tx=0x7f7ae4009b50 comp rx=0 tx=0).stop 2024-04-02T09:14:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.361+0000 7f7b00000700 1 -- 172.21.15.89:0/1465891503 shutdown_connections 2024-04-02T09:14:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.361+0000 7f7b00000700 1 --2- 172.21.15.89:0/1465891503 >> v2:172.21.15.113:3300/0 conn(0x7f7af815db40 0x7f7af80cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.361+0000 7f7b00000700 1 --2- 172.21.15.89:0/1465891503 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7af80cf3c0 0x7f7af80d23d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.361+0000 7f7b00000700 1 -- 172.21.15.89:0/1465891503 >> 172.21.15.89:0/1465891503 conn(0x7f7af808aff0 msgr2=0x7f7af80d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.361+0000 7f7b00000700 1 -- 172.21.15.89:0/1465891503 shutdown_connections 2024-04-02T09:14:26.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.361+0000 7f7b00000700 1 -- 172.21.15.89:0/1465891503 wait complete. 2024-04-02T09:14:26.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.363+0000 7f7b00000700 1 Processor -- start 2024-04-02T09:14:26.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.363+0000 7f7b00000700 1 -- start start 2024-04-02T09:14:26.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.364+0000 7f7b00000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7af80cf3c0 0x7f7af80cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:26.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.364+0000 7f7b00000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7af815d760 0x7f7af81439a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:26.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.364+0000 7f7b00000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af815db40 0x7f7af8143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.364+0000 7f7b00000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80cc3b0 con 0x7f7af815db40 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.364+0000 7f7b00000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80cc010 con 0x7f7af815d760 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.364+0000 7f7b00000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80c6ee0 con 0x7f7af80cf3c0 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.364+0000 7f7afe000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af815db40 0x7f7af8143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.364+0000 7f7afe000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af815db40 0x7f7af8143ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41908/0 (socket says 172.21.15.89:41908) 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.364+0000 7f7afe000700 1 -- 172.21.15.89:0/716718590 learned_addr learned my addr 172.21.15.89:0/716718590 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.364+0000 7f7afd600700 1 --2- 172.21.15.89:0/716718590 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7af80cf3c0 0x7f7af80cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.365+0000 7f7afcc00700 1 --2- 172.21.15.89:0/716718590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7af815d760 0x7f7af81439a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.365+0000 7f7afe000700 1 -- 172.21.15.89:0/716718590 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7af80cf3c0 msgr2=0x7f7af80cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.365+0000 7f7afe000700 1 --2- 172.21.15.89:0/716718590 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7af80cf3c0 0x7f7af80cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.365+0000 7f7afe000700 1 -- 172.21.15.89:0/716718590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7af815d760 msgr2=0x7f7af81439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.365+0000 7f7afe000700 1 --2- 172.21.15.89:0/716718590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7af815d760 0x7f7af81439a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.365+0000 7f7afe000700 1 -- 172.21.15.89:0/716718590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae40094a0 con 0x7f7af815db40 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.365+0000 7f7afe000700 1 --2- 172.21.15.89:0/716718590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af815db40 0x7f7af8143ee0 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7f7af000efc0 tx=0x7f7af000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:26.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.365+0000 7f7af5600700 1 -- 172.21.15.89:0/716718590 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0009ca0 con 0x7f7af815db40 2024-04-02T09:14:26.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.365+0000 7f7af5600700 1 -- 172.21.15.89:0/716718590 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af000f040 con 0x7f7af815db40 2024-04-02T09:14:26.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.365+0000 7f7af5600700 1 -- 172.21.15.89:0/716718590 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0014480 con 0x7f7af815db40 2024-04-02T09:14:26.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.365+0000 7f7b00000700 1 -- 172.21.15.89:0/716718590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af80c71c0 con 0x7f7af815db40 2024-04-02T09:14:26.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.365+0000 7f7b00000700 1 -- 172.21.15.89:0/716718590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af80b90a0 con 0x7f7af815db40 2024-04-02T09:14:26.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.366+0000 7f7b00000700 1 -- 172.21.15.89:0/716718590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af80cc4f0 con 0x7f7af815db40 2024-04-02T09:14:26.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.367+0000 7f7af5600700 1 -- 172.21.15.89:0/716718590 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f7af00145e0 con 0x7f7af815db40 2024-04-02T09:14:26.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.368+0000 7f7af5600700 1 --2- 172.21.15.89:0/716718590 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7ad806aed0 0x7f7ad806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:26.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.369+0000 7f7af5600700 1 -- 172.21.15.89:0/716718590 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f7af008a840 con 0x7f7af815db40 2024-04-02T09:14:26.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.369+0000 7f7afd600700 1 --2- 172.21.15.89:0/716718590 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7ad806aed0 0x7f7ad806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:26.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.369+0000 7f7afd600700 1 --2- 172.21.15.89:0/716718590 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7ad806aed0 0x7f7ad806d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7aec0011e0 tx=0x7f7aec009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:26.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.373+0000 7f7af5600700 1 -- 172.21.15.89:0/716718590 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7af80cc4f0 con 0x7f7af815db40 2024-04-02T09:14:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.652+0000 7f7b00000700 1 -- 172.21.15.89:0/716718590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7f7af80cc4f0 con 0x7f7af815db40 2024-04-02T09:14:26.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.819+0000 7f7af5600700 1 -- 172.21.15.89:0/716718590 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3237) v1 ==== 187+0+0 (secure 0 0 0) 0x7f7af80cc4f0 con 0x7f7af815db40 2024-04-02T09:14:26.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.821+0000 7f7b00000700 1 -- 172.21.15.89:0/716718590 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7ad806aed0 msgr2=0x7f7ad806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:26.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.821+0000 7f7b00000700 1 --2- 172.21.15.89:0/716718590 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7ad806aed0 0x7f7ad806d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7aec0011e0 tx=0x7f7aec009040 comp rx=0 tx=0).stop 2024-04-02T09:14:26.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.821+0000 7f7b00000700 1 -- 172.21.15.89:0/716718590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af815db40 msgr2=0x7f7af8143ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:26.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.821+0000 7f7b00000700 1 --2- 172.21.15.89:0/716718590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af815db40 0x7f7af8143ee0 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7f7af000efc0 tx=0x7f7af000c3a0 comp rx=0 tx=0).stop 2024-04-02T09:14:26.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.822+0000 7f7b00000700 1 -- 172.21.15.89:0/716718590 shutdown_connections 2024-04-02T09:14:26.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.822+0000 7f7b00000700 1 --2- 172.21.15.89:0/716718590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7af815db40 0x7f7af8143ee0 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:26.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.822+0000 7f7b00000700 1 --2- 172.21.15.89:0/716718590 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7af80cf3c0 0x7f7af80cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:26.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.822+0000 7f7b00000700 1 --2- 172.21.15.89:0/716718590 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7ad806aed0 0x7f7ad806d390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:26.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.822+0000 7f7b00000700 1 --2- 172.21.15.89:0/716718590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7af815d760 0x7f7af81439a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:26.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.822+0000 7f7b00000700 1 -- 172.21.15.89:0/716718590 >> 172.21.15.89:0/716718590 conn(0x7f7af808aff0 msgr2=0x7f7af80b30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:26.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.822+0000 7f7b00000700 1 -- 172.21.15.89:0/716718590 shutdown_connections 2024-04-02T09:14:26.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:26.822+0000 7f7b00000700 1 -- 172.21.15.89:0/716718590 wait complete. 2024-04-02T09:14:26.839 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2024-04-02T09:14:26.839 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2024-04-02T09:14:26.839 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2024-04-02T09:14:26.840 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-04-02T09:14:27.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.032+0000 7f56fc800700 1 Processor -- start 2024-04-02T09:14:27.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.033+0000 7f56fc800700 1 -- start start 2024-04-02T09:14:27.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.034+0000 7f56fc800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f56f415d640 0x7f56f40d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:27.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.034+0000 7f56fc800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f56f41619d0 0x7f56f40b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:27.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.034+0000 7f56fc800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f40d30d0 con 0x7f56f40ceec0 2024-04-02T09:14:27.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.034+0000 7f56fc800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f40b6840 con 0x7f56f415d640 2024-04-02T09:14:27.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.034+0000 7f56fc800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f40d2880 con 0x7f56f41619d0 2024-04-02T09:14:27.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.035+0000 7f56f9e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f56f415d640 0x7f56f40d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:27.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.035+0000 7f56f9e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f56f415d640 0x7f56f40d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41910/0 (socket says 172.21.15.89:41910) 2024-04-02T09:14:27.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.035+0000 7f56f9e00700 1 -- 172.21.15.89:0/1794260476 learned_addr learned my addr 172.21.15.89:0/1794260476 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:27.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.035+0000 7f56fa800700 1 --2- 172.21.15.89:0/1794260476 >> v2:172.21.15.113:3300/0 conn(0x7f56f41619d0 0x7f56f40b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:27.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.035+0000 7f56f9e00700 1 -- 172.21.15.89:0/1794260476 >> v2:172.21.15.113:3300/0 conn(0x7f56f41619d0 msgr2=0x7f56f40b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:27.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.035+0000 7f56f9e00700 1 --2- 172.21.15.89:0/1794260476 >> v2:172.21.15.113:3300/0 conn(0x7f56f41619d0 0x7f56f40b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:27.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.035+0000 7f56f9e00700 1 -- 172.21.15.89:0/1794260476 >> v1:172.21.15.113:6789/0 conn(0x7f56f40ceec0 legacy=0x7f56f41612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:27.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.035+0000 7f56f9e00700 1 -- 172.21.15.89:0/1794260476 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f40d2b30 con 0x7f56f415d640 2024-04-02T09:14:27.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.035+0000 7f56f9e00700 1 --2- 172.21.15.89:0/1794260476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56f415d640 0x7f56f40d2340 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f56f40c7020 tx=0x7f56e8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8ffa84a2d1f174 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:27.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.035+0000 7f56f8a00700 1 -- 172.21.15.89:0/1794260476 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e8011830 con 0x7f56f415d640 2024-04-02T09:14:27.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.036+0000 7f56f8a00700 1 -- 172.21.15.89:0/1794260476 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e8011990 con 0x7f56f415d640 2024-04-02T09:14:27.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.036+0000 7f56f8a00700 1 -- 172.21.15.89:0/1794260476 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e8011d20 con 0x7f56f415d640 2024-04-02T09:14:27.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.036+0000 7f56fc800700 1 -- 172.21.15.89:0/1794260476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56f415d640 msgr2=0x7f56f40d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:27.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.036+0000 7f56fc800700 1 --2- 172.21.15.89:0/1794260476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56f415d640 0x7f56f40d2340 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f56f40c7020 tx=0x7f56e8009bd0 comp rx=0 tx=0).stop 2024-04-02T09:14:27.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.036+0000 7f56fc800700 1 -- 172.21.15.89:0/1794260476 shutdown_connections 2024-04-02T09:14:27.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.036+0000 7f56fc800700 1 --2- 172.21.15.89:0/1794260476 >> v2:172.21.15.113:3300/0 conn(0x7f56f41619d0 0x7f56f40b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:27.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.036+0000 7f56fc800700 1 --2- 172.21.15.89:0/1794260476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56f415d640 0x7f56f40d2340 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:27.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.036+0000 7f56fc800700 1 -- 172.21.15.89:0/1794260476 >> 172.21.15.89:0/1794260476 conn(0x7f56f408aff0 msgr2=0x7f56f40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:27.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.036+0000 7f56fc800700 1 -- 172.21.15.89:0/1794260476 shutdown_connections 2024-04-02T09:14:27.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.036+0000 7f56fc800700 1 -- 172.21.15.89:0/1794260476 wait complete. 2024-04-02T09:14:27.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.038+0000 7f56fc800700 1 Processor -- start 2024-04-02T09:14:27.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.038+0000 7f56fc800700 1 -- start start 2024-04-02T09:14:27.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.039+0000 7f56fc800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f56f40ceec0 0x7f56f4158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:27.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.039+0000 7f56fc800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56f415d640 0x7f56f40a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:27.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.039+0000 7f56fc800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56f41619d0 0x7f56f415ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:27.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.039+0000 7f56fc800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f4158d50 con 0x7f56f415d640 2024-04-02T09:14:27.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.039+0000 7f56fc800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f40c85a0 con 0x7f56f41619d0 2024-04-02T09:14:27.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.039+0000 7f56fc800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f40a1180 con 0x7f56f40ceec0 2024-04-02T09:14:27.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.039+0000 7f56f9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56f415d640 0x7f56f40a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:27.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.039+0000 7f56f9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56f415d640 0x7f56f40a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41914/0 (socket says 172.21.15.89:41914) 2024-04-02T09:14:27.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.039+0000 7f56f9400700 1 -- 172.21.15.89:0/1724381525 learned_addr learned my addr 172.21.15.89:0/1724381525 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:27.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.039+0000 7f56f9e00700 1 --2- 172.21.15.89:0/1724381525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f56f40ceec0 0x7f56f4158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:27.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.039+0000 7f56fa800700 1 --2- 172.21.15.89:0/1724381525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56f41619d0 0x7f56f415ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:27.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.040+0000 7f56f9400700 1 -- 172.21.15.89:0/1724381525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f56f40ceec0 msgr2=0x7f56f4158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:27.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.040+0000 7f56f9400700 1 --2- 172.21.15.89:0/1724381525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f56f40ceec0 0x7f56f4158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:27.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.040+0000 7f56f9400700 1 -- 172.21.15.89:0/1724381525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56f41619d0 msgr2=0x7f56f415ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:27.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.040+0000 7f56f9400700 1 --2- 172.21.15.89:0/1724381525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56f41619d0 0x7f56f415ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:27.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.040+0000 7f56f9400700 1 -- 172.21.15.89:0/1724381525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e8009550 con 0x7f56f415d640 2024-04-02T09:14:27.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.040+0000 7f56f9400700 1 --2- 172.21.15.89:0/1724381525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56f415d640 0x7f56f40a06c0 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f56e400c430 tx=0x7f56e400c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:27.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.040+0000 7f56ee000700 1 -- 172.21.15.89:0/1724381525 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e40050d0 con 0x7f56f415d640 2024-04-02T09:14:27.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.040+0000 7f56ee000700 1 -- 172.21.15.89:0/1724381525 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e4005250 con 0x7f56f415d640 2024-04-02T09:14:27.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.040+0000 7f56ee000700 1 -- 172.21.15.89:0/1724381525 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e4006860 con 0x7f56f415d640 2024-04-02T09:14:27.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.040+0000 7f56fc800700 1 -- 172.21.15.89:0/1724381525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f40c8840 con 0x7f56f415d640 2024-04-02T09:14:27.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.040+0000 7f56fc800700 1 -- 172.21.15.89:0/1724381525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f40c8e00 con 0x7f56f415d640 2024-04-02T09:14:27.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.041+0000 7f56fc800700 1 -- 172.21.15.89:0/1724381525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f4067720 con 0x7f56f415d640 2024-04-02T09:14:27.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.042+0000 7f56ee000700 1 -- 172.21.15.89:0/1724381525 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f56e4003680 con 0x7f56f415d640 2024-04-02T09:14:27.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.043+0000 7f56ee000700 1 --2- 172.21.15.89:0/1724381525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f56d806aed0 0x7f56d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:27.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.043+0000 7f56ee000700 1 -- 172.21.15.89:0/1724381525 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f56e4088a30 con 0x7f56f415d640 2024-04-02T09:14:27.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.043+0000 7f56f9e00700 1 --2- 172.21.15.89:0/1724381525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f56d806aed0 0x7f56d806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:27.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.044+0000 7f56f9e00700 1 --2- 172.21.15.89:0/1724381525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f56d806aed0 0x7f56d806d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f56e8014040 tx=0x7f56e8008030 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:27.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.047+0000 7f56ee000700 1 -- 172.21.15.89:0/1724381525 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f56e40536c0 con 0x7f56f415d640 2024-04-02T09:14:27.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.328+0000 7f56fc800700 1 -- 172.21.15.89:0/1724381525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f56f40b5c70 con 0x7f56f415d640 2024-04-02T09:14:27.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.826+0000 7f56ee000700 1 -- 172.21.15.89:0/1724381525 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3238) v1 ==== 158+0+0 (secure 0 0 0) 0x7f56e4056ce0 con 0x7f56f415d640 2024-04-02T09:14:27.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.829+0000 7f56fc800700 1 -- 172.21.15.89:0/1724381525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f56d806aed0 msgr2=0x7f56d806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:27.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.829+0000 7f56fc800700 1 --2- 172.21.15.89:0/1724381525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f56d806aed0 0x7f56d806d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f56e8014040 tx=0x7f56e8008030 comp rx=0 tx=0).stop 2024-04-02T09:14:27.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.829+0000 7f56fc800700 1 -- 172.21.15.89:0/1724381525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56f415d640 msgr2=0x7f56f40a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:27.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.829+0000 7f56fc800700 1 --2- 172.21.15.89:0/1724381525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56f415d640 0x7f56f40a06c0 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f56e400c430 tx=0x7f56e400c740 comp rx=0 tx=0).stop 2024-04-02T09:14:27.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.829+0000 7f56fc800700 1 -- 172.21.15.89:0/1724381525 shutdown_connections 2024-04-02T09:14:27.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.829+0000 7f56fc800700 1 --2- 172.21.15.89:0/1724381525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f56d806aed0 0x7f56d806d390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:27.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.829+0000 7f56fc800700 1 --2- 172.21.15.89:0/1724381525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56f41619d0 0x7f56f415ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:27.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.829+0000 7f56fc800700 1 --2- 172.21.15.89:0/1724381525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f56f40ceec0 0x7f56f4158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:27.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.829+0000 7f56fc800700 1 --2- 172.21.15.89:0/1724381525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56f415d640 0x7f56f40a06c0 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:27.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.829+0000 7f56fc800700 1 -- 172.21.15.89:0/1724381525 >> 172.21.15.89:0/1724381525 conn(0x7f56f408aff0 msgr2=0x7f56f40cffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:27.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.829+0000 7f56fc800700 1 -- 172.21.15.89:0/1724381525 shutdown_connections 2024-04-02T09:14:27.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:27.829+0000 7f56fc800700 1 -- 172.21.15.89:0/1724381525 wait complete. 2024-04-02T09:14:27.847 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T09:14:28.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.042+0000 7fa61f200700 1 Processor -- start 2024-04-02T09:14:28.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.043+0000 7fa61f200700 1 -- start start 2024-04-02T09:14:28.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.044+0000 7fa61f200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa61815d9d0 0x7fa6180d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:28.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.044+0000 7fa61f200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa618161420 con 0x7fa6180d16a0 2024-04-02T09:14:28.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.044+0000 7fa61f200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa618161700 con 0x7fa6180d2540 2024-04-02T09:14:28.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.044+0000 7fa61f200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61815edc0 con 0x7fa61815d9d0 2024-04-02T09:14:28.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.044+0000 7fa617400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa6180d16a0 0x7fa6180d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56894/0 (socket says 172.21.15.89:56894) 2024-04-02T09:14:28.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.044+0000 7fa617400700 1 -- 172.21.15.89:0/1042290656 learned_addr learned my addr 172.21.15.89:0/1042290656 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:28.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.044+0000 7fa617e00700 1 --2- 172.21.15.89:0/1042290656 >> v2:172.21.15.113:3300/0 conn(0x7fa61815d9d0 0x7fa6180d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.044+0000 7fa616a00700 1 -- 172.21.15.89:0/1042290656 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16380986 0 0) 0x7fa618161420 con 0x7fa6180d16a0 2024-04-02T09:14:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.045+0000 7fa616a00700 1 -- 172.21.15.89:0/1042290656 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5fc0035b0 con 0x7fa6180d16a0 2024-04-02T09:14:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.045+0000 7fa616a00700 1 -- 172.21.15.89:0/1042290656 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037904244 0 0) 0x7fa618161700 con 0x7fa6180d2540 2024-04-02T09:14:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.045+0000 7fa616a00700 1 -- 172.21.15.89:0/1042290656 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa618161420 con 0x7fa6180d2540 2024-04-02T09:14:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.045+0000 7fa617e00700 1 -- 172.21.15.89:0/1042290656 >> v1:172.21.15.113:6789/0 conn(0x7fa6180d2540 legacy=0x7fa6180ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.045+0000 7fa616a00700 1 -- 172.21.15.89:0/1042290656 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 196288613 0 0) 0x7fa5fc0035b0 con 0x7fa6180d16a0 2024-04-02T09:14:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.045+0000 7fa617e00700 1 -- 172.21.15.89:0/1042290656 >> v1:172.21.15.89:6789/0 conn(0x7fa6180d16a0 legacy=0x7fa6180d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.045+0000 7fa617e00700 1 -- 172.21.15.89:0/1042290656 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6181619e0 con 0x7fa61815d9d0 2024-04-02T09:14:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.045+0000 7fa617e00700 1 --2- 172.21.15.89:0/1042290656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa61815d9d0 0x7fa6180d3c30 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fa608009840 tx=0x7fa608009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ecd0a53d38e5987 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.045+0000 7fa616a00700 1 -- 172.21.15.89:0/1042290656 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa604001770 con 0x7fa61815d9d0 2024-04-02T09:14:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.046+0000 7fa616a00700 1 -- 172.21.15.89:0/1042290656 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa608004030 con 0x7fa61815d9d0 2024-04-02T09:14:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.046+0000 7fa616a00700 1 -- 172.21.15.89:0/1042290656 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6080043c0 con 0x7fa61815d9d0 2024-04-02T09:14:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.046+0000 7fa61f200700 1 -- 172.21.15.89:0/1042290656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa61815d9d0 msgr2=0x7fa6180d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.046+0000 7fa61f200700 1 --2- 172.21.15.89:0/1042290656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa61815d9d0 0x7fa6180d3c30 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fa608009840 tx=0x7fa608009b50 comp rx=0 tx=0).stop 2024-04-02T09:14:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.046+0000 7fa61f200700 1 -- 172.21.15.89:0/1042290656 shutdown_connections 2024-04-02T09:14:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.046+0000 7fa61f200700 1 --2- 172.21.15.89:0/1042290656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa61815d9d0 0x7fa6180d3c30 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.046+0000 7fa61f200700 1 -- 172.21.15.89:0/1042290656 >> 172.21.15.89:0/1042290656 conn(0x7fa61808aff0 msgr2=0x7fa6180d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.046+0000 7fa61f200700 1 -- 172.21.15.89:0/1042290656 shutdown_connections 2024-04-02T09:14:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.046+0000 7fa61f200700 1 -- 172.21.15.89:0/1042290656 wait complete. 2024-04-02T09:14:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.048+0000 7fa61f200700 1 Processor -- start 2024-04-02T09:14:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.048+0000 7fa61f200700 1 -- start start 2024-04-02T09:14:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.049+0000 7fa61f200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6180d16a0 0x7fa61815c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.049+0000 7fa61f200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6180d2540 0x7fa6180b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.049+0000 7fa61f200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa61815d9d0 0x7fa6180b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.049+0000 7fa61f200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6181f4900 con 0x7fa6180d16a0 2024-04-02T09:14:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.049+0000 7fa61f200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6181f3a30 con 0x7fa61815d9d0 2024-04-02T09:14:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.049+0000 7fa61f200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6181f3c80 con 0x7fa6180d2540 2024-04-02T09:14:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.049+0000 7fa617e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6180d16a0 0x7fa61815c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.049+0000 7fa617e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6180d16a0 0x7fa61815c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41918/0 (socket says 172.21.15.89:41918) 2024-04-02T09:14:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.049+0000 7fa617e00700 1 -- 172.21.15.89:0/793018537 learned_addr learned my addr 172.21.15.89:0/793018537 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.050+0000 7fa617400700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6180d2540 0x7fa6180b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.050+0000 7fa61d200700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa61815d9d0 0x7fa6180b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.050+0000 7fa617e00700 1 -- 172.21.15.89:0/793018537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6180d2540 msgr2=0x7fa6180b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.050+0000 7fa617e00700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6180d2540 0x7fa6180b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.050+0000 7fa617e00700 1 -- 172.21.15.89:0/793018537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa61815d9d0 msgr2=0x7fa6180b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.050+0000 7fa617e00700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa61815d9d0 0x7fa6180b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.050+0000 7fa617e00700 1 -- 172.21.15.89:0/793018537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6080094a0 con 0x7fa6180d16a0 2024-04-02T09:14:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.050+0000 7fa617400700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6180d2540 0x7fa6180b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:28.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.050+0000 7fa617e00700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6180d16a0 0x7fa61815c550 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7fa608009790 tx=0x7fa608004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:28.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.050+0000 7fa60fe00700 1 -- 172.21.15.89:0/793018537 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa608003e30 con 0x7fa6180d16a0 2024-04-02T09:14:28.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.050+0000 7fa60fe00700 1 -- 172.21.15.89:0/793018537 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60801f070 con 0x7fa6180d16a0 2024-04-02T09:14:28.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.050+0000 7fa61f200700 1 -- 172.21.15.89:0/793018537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6181f3f60 con 0x7fa6180d16a0 2024-04-02T09:14:28.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.050+0000 7fa60fe00700 1 -- 172.21.15.89:0/793018537 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa608011870 con 0x7fa6180d16a0 2024-04-02T09:14:28.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.051+0000 7fa61f200700 1 -- 172.21.15.89:0/793018537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6181f4520 con 0x7fa6180d16a0 2024-04-02T09:14:28.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.052+0000 7fa61f200700 1 -- 172.21.15.89:0/793018537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6180d1ca0 con 0x7fa6180d16a0 2024-04-02T09:14:28.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.053+0000 7fa60fe00700 1 -- 172.21.15.89:0/793018537 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa608021030 con 0x7fa6180d16a0 2024-04-02T09:14:28.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.053+0000 7fa60fe00700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa5f806aed0 0x7fa5f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:28.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.053+0000 7fa60fe00700 1 -- 172.21.15.89:0/793018537 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6251+0+0 (secure 0 0 0) 0x7fa60808ac60 con 0x7fa6180d16a0 2024-04-02T09:14:28.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.054+0000 7fa617400700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa5f806aed0 0x7fa5f806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:28.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.054+0000 7fa617400700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa5f806aed0 0x7fa5f806d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa61815d1b0 tx=0x7fa604009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:28.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.058+0000 7fa60fe00700 1 -- 172.21.15.89:0/793018537 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa608055950 con 0x7fa6180d16a0 2024-04-02T09:14:28.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.258+0000 7fa61f200700 1 -- 172.21.15.89:0/793018537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa61815d510 con 0x7fa6180d16a0 2024-04-02T09:14:28.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.264+0000 7fa60fe00700 1 -- 172.21.15.89:0/793018537 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v88)=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa608058f70 con 0x7fa6180d16a0 2024-04-02T09:14:28.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.267+0000 7fa61f200700 1 -- 172.21.15.89:0/793018537 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa5f806aed0 msgr2=0x7fa5f806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:28.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.267+0000 7fa61f200700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa5f806aed0 0x7fa5f806d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa61815d1b0 tx=0x7fa604009040 comp rx=0 tx=0).stop 2024-04-02T09:14:28.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.267+0000 7fa61f200700 1 -- 172.21.15.89:0/793018537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6180d16a0 msgr2=0x7fa61815c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:28.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.267+0000 7fa61f200700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6180d16a0 0x7fa61815c550 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7fa608009790 tx=0x7fa608004750 comp rx=0 tx=0).stop 2024-04-02T09:14:28.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.267+0000 7fa61f200700 1 -- 172.21.15.89:0/793018537 shutdown_connections 2024-04-02T09:14:28.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.267+0000 7fa61f200700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa5f806aed0 0x7fa5f806d390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:28.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.267+0000 7fa61f200700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa61815d9d0 0x7fa6180b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:28.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.267+0000 7fa61f200700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6180d16a0 0x7fa61815c550 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:28.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.267+0000 7fa61f200700 1 --2- 172.21.15.89:0/793018537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa6180d2540 0x7fa6180b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:28.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.267+0000 7fa61f200700 1 -- 172.21.15.89:0/793018537 >> 172.21.15.89:0/793018537 conn(0x7fa61808aff0 msgr2=0x7fa618160120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:28.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.267+0000 7fa61f200700 1 -- 172.21.15.89:0/793018537 shutdown_connections 2024-04-02T09:14:28.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.267+0000 7fa61f200700 1 -- 172.21.15.89:0/793018537 wait complete. 2024-04-02T09:14:28.284 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T09:14:28.284 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:14:28.367 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:14:28.461 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T09:14:28.461 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:14:28.461 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:14:28.505 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:14:28.506 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:14:28.506 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:14:28.550 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:14:28.550 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:14:28.550 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:14:28.594 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:14:28.595 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:14:28.595 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:14:28.639 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:14:28.639 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T09:14:28.639 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:14:28.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.833+0000 7f5ef1a00700 1 Processor -- start 2024-04-02T09:14:28.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.833+0000 7f5ef1a00700 1 -- start start 2024-04-02T09:14:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.834+0000 7f5ef1a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5eec0d18b0 0x7f5eec0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.834+0000 7f5ef1a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5eec161590 0x7f5eec161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.834+0000 7f5ef1a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eec0ceda0 con 0x7f5eec160f10 2024-04-02T09:14:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.834+0000 7f5ef1a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eec0d1d30 con 0x7f5eec0d18b0 2024-04-02T09:14:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.834+0000 7f5ef1a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec161f60 con 0x7f5eec161590 2024-04-02T09:14:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.834+0000 7f5eea000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5eec0d18b0 0x7f5eec0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.834+0000 7f5eea000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5eec0d18b0 0x7f5eec0cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59476/0 (socket says 172.21.15.89:59476) 2024-04-02T09:14:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.834+0000 7f5eea000700 1 -- 172.21.15.89:0/2528139070 learned_addr learned my addr 172.21.15.89:0/2528139070 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.835+0000 7f5eea000700 1 -- 172.21.15.89:0/2528139070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5eec161590 msgr2=0x7f5eec161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.835+0000 7f5eeb400700 1 --2- 172.21.15.89:0/2528139070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5eec161590 0x7f5eec161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.835+0000 7f5eea000700 1 --2- 172.21.15.89:0/2528139070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5eec161590 0x7f5eec161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.835+0000 7f5eea000700 1 -- 172.21.15.89:0/2528139070 >> v1:172.21.15.89:6789/0 conn(0x7f5eec160f10 legacy=0x7f5eec0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.835+0000 7f5eea000700 1 -- 172.21.15.89:0/2528139070 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec1612f0 con 0x7f5eec0d18b0 2024-04-02T09:14:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.835+0000 7f5eeb400700 1 --2- 172.21.15.89:0/2528139070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5eec161590 0x7f5eec161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.835+0000 7f5eea000700 1 --2- 172.21.15.89:0/2528139070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eec0d18b0 0x7f5eec0cf230 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8009920 tx=0x7f5ed8009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f866bd126076efc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.835+0000 7f5ee9600700 1 -- 172.21.15.89:0/2528139070 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed800ba20 con 0x7f5eec0d18b0 2024-04-02T09:14:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.835+0000 7f5ee9600700 1 -- 172.21.15.89:0/2528139070 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed800bb80 con 0x7f5eec0d18b0 2024-04-02T09:14:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.835+0000 7f5ee9600700 1 -- 172.21.15.89:0/2528139070 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8009e70 con 0x7f5eec0d18b0 2024-04-02T09:14:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.836+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2528139070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eec0d18b0 msgr2=0x7f5eec0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.836+0000 7f5ef1a00700 1 --2- 172.21.15.89:0/2528139070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eec0d18b0 0x7f5eec0cf230 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8009920 tx=0x7f5ed8009c30 comp rx=0 tx=0).stop 2024-04-02T09:14:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.836+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2528139070 shutdown_connections 2024-04-02T09:14:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.836+0000 7f5ef1a00700 1 --2- 172.21.15.89:0/2528139070 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5eec161590 0x7f5eec161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.836+0000 7f5ef1a00700 1 --2- 172.21.15.89:0/2528139070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eec0d18b0 0x7f5eec0cf230 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.836+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2528139070 >> 172.21.15.89:0/2528139070 conn(0x7f5eec08aff0 msgr2=0x7f5eec0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.836+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2528139070 shutdown_connections 2024-04-02T09:14:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.836+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2528139070 wait complete. 2024-04-02T09:14:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.838+0000 7f5ef1a00700 1 Processor -- start 2024-04-02T09:14:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.838+0000 7f5ef1a00700 1 -- start start 2024-04-02T09:14:28.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5ef1a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5eec0d18b0 0x7f5eec1594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5ef1a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5eec160f10 0x7f5eec15af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5ef1a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eec161590 0x7f5eec15ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5ef1a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec0b7c90 con 0x7f5eec161590 2024-04-02T09:14:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5ef1a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec1599e0 con 0x7f5eec0d18b0 2024-04-02T09:14:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5ef1a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec0b7080 con 0x7f5eec160f10 2024-04-02T09:14:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5eeb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eec161590 0x7f5eec15ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5eeb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eec161590 0x7f5eec15ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59484/0 (socket says 172.21.15.89:59484) 2024-04-02T09:14:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5eeb400700 1 -- 172.21.15.89:0/2418414787 learned_addr learned my addr 172.21.15.89:0/2418414787 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5eea000700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5eec160f10 0x7f5eec15af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5eeaa00700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5eec0d18b0 0x7f5eec1594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5eeb400700 1 -- 172.21.15.89:0/2418414787 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5eec160f10 msgr2=0x7f5eec15af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5eeb400700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5eec160f10 0x7f5eec15af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:28.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5eeb400700 1 -- 172.21.15.89:0/2418414787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5eec0d18b0 msgr2=0x7f5eec1594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:28.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5eeb400700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5eec0d18b0 0x7f5eec1594a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:28.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5eeb400700 1 -- 172.21.15.89:0/2418414787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed8009550 con 0x7f5eec161590 2024-04-02T09:14:28.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.839+0000 7f5eeaa00700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5eec0d18b0 0x7f5eec1594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:28.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.840+0000 7f5eeb400700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eec161590 0x7f5eec15ca40 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400cd50 tx=0x7f5ee400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:28.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.840+0000 7f5ee2a00700 1 -- 172.21.15.89:0/2418414787 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee400cea0 con 0x7f5eec161590 2024-04-02T09:14:28.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.840+0000 7f5ee2a00700 1 -- 172.21.15.89:0/2418414787 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee400f480 con 0x7f5eec161590 2024-04-02T09:14:28.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.840+0000 7f5ee2a00700 1 -- 172.21.15.89:0/2418414787 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee400f810 con 0x7f5eec161590 2024-04-02T09:14:28.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.840+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2418414787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eec0b7360 con 0x7f5eec161590 2024-04-02T09:14:28.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.840+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2418414787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eec0b7920 con 0x7f5eec161590 2024-04-02T09:14:28.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.841+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2418414787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eec067720 con 0x7f5eec161590 2024-04-02T09:14:28.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.842+0000 7f5ee2a00700 1 -- 172.21.15.89:0/2418414787 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5ee400b040 con 0x7f5eec161590 2024-04-02T09:14:28.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.843+0000 7f5ee2a00700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ecc06aed0 0x7f5ecc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:28.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.843+0000 7f5ee2a00700 1 -- 172.21.15.89:0/2418414787 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6251+0+0 (secure 0 0 0) 0x7f5ee408a1b0 con 0x7f5eec161590 2024-04-02T09:14:28.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.843+0000 7f5eeaa00700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ecc06aed0 0x7f5ecc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:28.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.844+0000 7f5eeaa00700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ecc06aed0 0x7f5ecc06d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5edc000f60 tx=0x7f5edc009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:28.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:28.847+0000 7f5ee2a00700 1 -- 172.21.15.89:0/2418414787 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5ee4054d90 con 0x7f5eec161590 2024-04-02T09:14:29.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.089+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2418414787 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5eec0cf4d0 con 0x7f5eec161590 2024-04-02T09:14:29.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.090+0000 7f5ee2a00700 1 -- 172.21.15.89:0/2418414787 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+3539 (secure 0 0 0) 0x7f5ee401ddf0 con 0x7f5eec161590 2024-04-02T09:14:29.088 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:29.089 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":129,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9518,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6836/3094781297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6835","nonce":3094781297},{"type":"v1","addr":"172.21.15.89:6836","nonce":3094781297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":9920,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1284027162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1284027162},{"type":"v1","addr":"172.21.15.113:6835","nonce":1284027162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":127},{"gid":10978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1799351491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1799351491},{"type":"v1","addr":"172.21.15.113:6834","nonce":1799351491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":11557,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/246142562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":246142562},{"type":"v1","addr":"172.21.15.89:6837","nonce":246142562}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":129}],"filesystems":[]} 2024-04-02T09:14:29.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.092+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2418414787 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ecc06aed0 msgr2=0x7f5ecc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:29.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.092+0000 7f5ef1a00700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ecc06aed0 0x7f5ecc06d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5edc000f60 tx=0x7f5edc009040 comp rx=0 tx=0).stop 2024-04-02T09:14:29.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.092+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2418414787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eec161590 msgr2=0x7f5eec15ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:29.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.092+0000 7f5ef1a00700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eec161590 0x7f5eec15ca40 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400cd50 tx=0x7f5ee400a560 comp rx=0 tx=0).stop 2024-04-02T09:14:29.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.092+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2418414787 shutdown_connections 2024-04-02T09:14:29.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.092+0000 7f5ef1a00700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ecc06aed0 0x7f5ecc06d390 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:29.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.092+0000 7f5ef1a00700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5eec0d18b0 0x7f5eec1594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:29.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.092+0000 7f5ef1a00700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5eec161590 0x7f5eec15ca40 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:29.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.092+0000 7f5ef1a00700 1 --2- 172.21.15.89:0/2418414787 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5eec160f10 0x7f5eec15af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:29.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.092+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2418414787 >> 172.21.15.89:0/2418414787 conn(0x7f5eec08aff0 msgr2=0x7f5eec0cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:29.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.093+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2418414787 shutdown_connections 2024-04-02T09:14:29.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.093+0000 7f5ef1a00700 1 -- 172.21.15.89:0/2418414787 wait complete. 2024-04-02T09:14:29.092 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 129 2024-04-02T09:14:29.109 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T09:14:29.109 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T09:14:29.109 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T09:14:29.109 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:14:29.109 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T09:14:29.110 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T09:14:29.110 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T09:14:29.110 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:14:29.110 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T09:14:29.110 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T09:14:29.111 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T09:14:29.111 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:14:29.111 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T09:14:29.111 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T09:14:29.111 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T09:14:29.111 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:14:29.123 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T09:14:29.123 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T09:14:29.124 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T09:14:29.124 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T09:14:29.125 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T09:14:29.126 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T09:14:29.126 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T09:14:29.126 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T09:14:29.167 INFO:tasks.ceph.mds.a:Started 2024-04-02T09:14:29.168 DEBUG:teuthology.parallel:result is None 2024-04-02T09:14:29.169 INFO:tasks.ceph.mds.b:Started 2024-04-02T09:14:29.169 DEBUG:teuthology.parallel:result is None 2024-04-02T09:14:29.170 INFO:tasks.ceph.mds.c:Started 2024-04-02T09:14:29.170 DEBUG:teuthology.parallel:result is None 2024-04-02T09:14:29.171 INFO:tasks.ceph.mds.d:Started 2024-04-02T09:14:29.172 DEBUG:teuthology.parallel:result is None 2024-04-02T09:14:29.172 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T09:14:29.313 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T09:14:29.313 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T09:14:29.315 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T09:14:29.319 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T09:14:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.382+0000 7fa08e200700 1 Processor -- start 2024-04-02T09:14:29.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.382+0000 7fa08e200700 1 -- start start 2024-04-02T09:14:29.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.383+0000 7fa08e200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa0880d2540 0x7fa0881687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.383+0000 7fa08e200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa088161420 con 0x7fa0880d16a0 2024-04-02T09:14:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.383+0000 7fa08e200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa088161700 con 0x7fa08815d9d0 2024-04-02T09:14:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.383+0000 7fa08e200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0880d4340 con 0x7fa0880d2540 2024-04-02T09:14:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.384+0000 7fa087e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa0880d2540 0x7fa0881687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.384+0000 7fa087e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa0880d2540 0x7fa0881687b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59532/0 (socket says 172.21.15.89:59532) 2024-04-02T09:14:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.384+0000 7fa087e00700 1 -- 172.21.15.89:0/1648949750 learned_addr learned my addr 172.21.15.89:0/1648949750 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.384+0000 7fa087e00700 1 -- 172.21.15.89:0/1648949750 >> v1:172.21.15.113:6789/0 conn(0x7fa08815d9d0 legacy=0x7fa0880d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.384+0000 7fa087e00700 1 -- 172.21.15.89:0/1648949750 >> v1:172.21.15.89:6789/0 conn(0x7fa0880d16a0 legacy=0x7fa0880d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.384+0000 7fa087e00700 1 -- 172.21.15.89:0/1648949750 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0881619e0 con 0x7fa0880d2540 2024-04-02T09:14:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.384+0000 7fa087e00700 1 --2- 172.21.15.89:0/1648949750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0880d2540 0x7fa0881687b0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7fa080009e50 tx=0x7fa08000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163f0657ae7735e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.384+0000 7fa086a00700 1 -- 172.21.15.89:0/1648949750 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08000e070 con 0x7fa0880d2540 2024-04-02T09:14:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.385+0000 7fa086a00700 1 -- 172.21.15.89:0/1648949750 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa080003a20 con 0x7fa0880d2540 2024-04-02T09:14:29.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.385+0000 7fa086a00700 1 -- 172.21.15.89:0/1648949750 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa080003dd0 con 0x7fa0880d2540 2024-04-02T09:14:29.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.385+0000 7fa08e200700 1 -- 172.21.15.89:0/1648949750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0880d2540 msgr2=0x7fa0881687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:29.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.385+0000 7fa08e200700 1 --2- 172.21.15.89:0/1648949750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0880d2540 0x7fa0881687b0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7fa080009e50 tx=0x7fa08000b330 comp rx=0 tx=0).stop 2024-04-02T09:14:29.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.385+0000 7fa08e200700 1 -- 172.21.15.89:0/1648949750 shutdown_connections 2024-04-02T09:14:29.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.385+0000 7fa08e200700 1 --2- 172.21.15.89:0/1648949750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0880d2540 0x7fa0881687b0 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:29.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.385+0000 7fa08e200700 1 -- 172.21.15.89:0/1648949750 >> 172.21.15.89:0/1648949750 conn(0x7fa08808aff0 msgr2=0x7fa0880d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:29.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.385+0000 7fa08e200700 1 -- 172.21.15.89:0/1648949750 shutdown_connections 2024-04-02T09:14:29.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.385+0000 7fa08e200700 1 -- 172.21.15.89:0/1648949750 wait complete. 2024-04-02T09:14:29.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.387+0000 7fa08e200700 1 Processor -- start 2024-04-02T09:14:29.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.388+0000 7fa08e200700 1 -- start start 2024-04-02T09:14:29.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa08e200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0880d16a0 0x7fa08814f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:29.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa08e200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0880d2540 0x7fa0880cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:29.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa08e200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa08815d9d0 0x7fa0880cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:29.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa08e200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0881e7dc0 con 0x7fa0880d2540 2024-04-02T09:14:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa08e200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0880cc450 con 0x7fa08815d9d0 2024-04-02T09:14:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa08e200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0881e7420 con 0x7fa0880d16a0 2024-04-02T09:14:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa07fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0880d2540 0x7fa0880cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa07fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0880d2540 0x7fa0880cb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59542/0 (socket says 172.21.15.89:59542) 2024-04-02T09:14:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa07fe00700 1 -- 172.21.15.89:0/3480982630 learned_addr learned my addr 172.21.15.89:0/3480982630 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa087e00700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa08815d9d0 0x7fa0880cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa087400700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0880d16a0 0x7fa08814f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa07fe00700 1 -- 172.21.15.89:0/3480982630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0880d16a0 msgr2=0x7fa08814f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa07fe00700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0880d16a0 0x7fa08814f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa07fe00700 1 -- 172.21.15.89:0/3480982630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa08815d9d0 msgr2=0x7fa0880cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.389+0000 7fa07fe00700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa08815d9d0 0x7fa0880cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.390+0000 7fa07fe00700 1 -- 172.21.15.89:0/3480982630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa080009b30 con 0x7fa0880d2540 2024-04-02T09:14:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.390+0000 7fa087e00700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa08815d9d0 0x7fa0880cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:29.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.390+0000 7fa07fe00700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0880d2540 0x7fa0880cb9d0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7fa07000c450 tx=0x7fa07000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:29.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.390+0000 7fa07f400700 1 -- 172.21.15.89:0/3480982630 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0700054e0 con 0x7fa0880d2540 2024-04-02T09:14:29.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.390+0000 7fa07f400700 1 -- 172.21.15.89:0/3480982630 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa070003680 con 0x7fa0880d2540 2024-04-02T09:14:29.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.390+0000 7fa07f400700 1 -- 172.21.15.89:0/3480982630 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa070003a30 con 0x7fa0880d2540 2024-04-02T09:14:29.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.390+0000 7fa08e200700 1 -- 172.21.15.89:0/3480982630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0881e7700 con 0x7fa0880d2540 2024-04-02T09:14:29.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.390+0000 7fa08e200700 1 -- 172.21.15.89:0/3480982630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08814fe50 con 0x7fa0880d2540 2024-04-02T09:14:29.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.391+0000 7fa08e200700 1 -- 172.21.15.89:0/3480982630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0880d1ca0 con 0x7fa0880d2540 2024-04-02T09:14:29.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.392+0000 7fa07f400700 1 -- 172.21.15.89:0/3480982630 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa070003b90 con 0x7fa0880d2540 2024-04-02T09:14:29.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.393+0000 7fa07f400700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa06406aed0 0x7fa06406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:29.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.393+0000 7fa07f400700 1 -- 172.21.15.89:0/3480982630 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 6251+0+0 (secure 0 0 0) 0x7fa0700889d0 con 0x7fa0880d2540 2024-04-02T09:14:29.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.393+0000 7fa087400700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa06406aed0 0x7fa06406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:29.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.394+0000 7fa087400700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa06406aed0 0x7fa06406d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa078009870 tx=0x7fa078008040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:29.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.398+0000 7fa07f400700 1 -- 172.21.15.89:0/3480982630 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa070053590 con 0x7fa0880d2540 2024-04-02T09:14:29.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:29.615+0000 7fa08e200700 1 -- 172.21.15.89:0/3480982630 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa0881e7f00 con 0x7fa0880d2540 2024-04-02T09:14:30.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.362+0000 7fa07f400700 1 -- 172.21.15.89:0/3480982630 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v261) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa0881e7f00 con 0x7fa0880d2540 2024-04-02T09:14:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.365+0000 7fa08e200700 1 -- 172.21.15.89:0/3480982630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa06406aed0 msgr2=0x7fa06406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:30.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.365+0000 7fa08e200700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa06406aed0 0x7fa06406d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa078009870 tx=0x7fa078008040 comp rx=0 tx=0).stop 2024-04-02T09:14:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.365+0000 7fa08e200700 1 -- 172.21.15.89:0/3480982630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0880d2540 msgr2=0x7fa0880cb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.365+0000 7fa08e200700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0880d2540 0x7fa0880cb9d0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7fa07000c450 tx=0x7fa07000c810 comp rx=0 tx=0).stop 2024-04-02T09:14:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.365+0000 7fa08e200700 1 -- 172.21.15.89:0/3480982630 shutdown_connections 2024-04-02T09:14:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.365+0000 7fa08e200700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa06406aed0 0x7fa06406d390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.365+0000 7fa08e200700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa08815d9d0 0x7fa0880cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.365+0000 7fa08e200700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0880d16a0 0x7fa08814f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.365+0000 7fa08e200700 1 --2- 172.21.15.89:0/3480982630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0880d2540 0x7fa0880cb9d0 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.365+0000 7fa08e200700 1 -- 172.21.15.89:0/3480982630 >> 172.21.15.89:0/3480982630 conn(0x7fa08808aff0 msgr2=0x7fa0880b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.365+0000 7fa08e200700 1 -- 172.21.15.89:0/3480982630 shutdown_connections 2024-04-02T09:14:30.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.365+0000 7fa08e200700 1 -- 172.21.15.89:0/3480982630 wait complete. 2024-04-02T09:14:30.365 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T09:14:30.383 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T09:14:30.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.576+0000 7fad36200700 1 Processor -- start 2024-04-02T09:14:30.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.576+0000 7fad36200700 1 -- start start 2024-04-02T09:14:30.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.577+0000 7fad36200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad300cb3f0 0x7fad300ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:30.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.577+0000 7fad36200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fad300ac600 0x7fad3006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:30.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.577+0000 7fad36200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3015fd50 con 0x7fad3015b5d0 2024-04-02T09:14:30.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.577+0000 7fad36200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad3015b9b0 con 0x7fad300ac600 2024-04-02T09:14:30.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.577+0000 7fad36200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad300aca70 con 0x7fad300cb3f0 2024-04-02T09:14:30.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fad3015b5d0 0x7fad300ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50186/0 (socket says 172.21.15.89:50186) 2024-04-02T09:14:30.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2f400700 1 -- 172.21.15.89:0/680108444 learned_addr learned my addr 172.21.15.89:0/680108444 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:30.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2ea00700 1 --2- 172.21.15.89:0/680108444 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad300cb3f0 0x7fad300ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:30.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2fe00700 1 --2- 172.21.15.89:0/680108444 >> v2:172.21.15.113:3300/0 conn(0x7fad300ac600 0x7fad3006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:30.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2e000700 1 -- 172.21.15.89:0/680108444 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462999719 0 0) 0x7fad3015fd50 con 0x7fad3015b5d0 2024-04-02T09:14:30.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2e000700 1 -- 172.21.15.89:0/680108444 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad140035b0 con 0x7fad3015b5d0 2024-04-02T09:14:30.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2e000700 1 -- 172.21.15.89:0/680108444 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 95972892 0 0) 0x7fad140035b0 con 0x7fad3015b5d0 2024-04-02T09:14:30.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2e000700 1 -- 172.21.15.89:0/680108444 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad300cb3f0 msgr2=0x7fad300ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:30.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2e000700 1 --2- 172.21.15.89:0/680108444 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad300cb3f0 0x7fad300ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:30.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2e000700 1 -- 172.21.15.89:0/680108444 >> v2:172.21.15.113:3300/0 conn(0x7fad300ac600 msgr2=0x7fad3006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:30.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2e000700 1 --2- 172.21.15.89:0/680108444 >> v2:172.21.15.113:3300/0 conn(0x7fad300ac600 0x7fad3006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:30.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2e000700 1 -- 172.21.15.89:0/680108444 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3015f140 con 0x7fad3015b5d0 2024-04-02T09:14:30.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2e000700 1 -- 172.21.15.89:0/680108444 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fad20001670 con 0x7fad3015b5d0 2024-04-02T09:14:30.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2ea00700 1 --2- 172.21.15.89:0/680108444 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad300cb3f0 0x7fad300ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:14:30.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2fe00700 1 --2- 172.21.15.89:0/680108444 >> v2:172.21.15.113:3300/0 conn(0x7fad300ac600 0x7fad3006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:14:30.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2e000700 1 -- 172.21.15.89:0/680108444 >> v1:172.21.15.89:6789/0 conn(0x7fad3015b5d0 legacy=0x7fad300ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:30.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.578+0000 7fad2e000700 1 --2- 172.21.15.89:0/680108444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad14003a00 0x7fad14003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:30.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.579+0000 7fad2e000700 1 --2- 172.21.15.89:0/680108444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad14004850 0x7fad14004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:30.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.579+0000 7fad2e000700 1 --2- 172.21.15.89:0/680108444 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad14005210 0x7fad1400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:30.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.579+0000 7fad2e000700 1 -- 172.21.15.89:0/680108444 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3015b9b0 con 0x7fad14004850 2024-04-02T09:14:30.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.579+0000 7fad2e000700 1 -- 172.21.15.89:0/680108444 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad300aca70 con 0x7fad14003a00 2024-04-02T09:14:30.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.579+0000 7fad2e000700 1 -- 172.21.15.89:0/680108444 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad140056c0 con 0x7fad14005210 2024-04-02T09:14:30.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.579+0000 7fad2ea00700 1 --2- 172.21.15.89:0/680108444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad14004850 0x7fad14004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:30.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.579+0000 7fad2f400700 1 --2- 172.21.15.89:0/680108444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad14003a00 0x7fad14003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:30.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.579+0000 7fad2fe00700 1 --2- 172.21.15.89:0/680108444 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad14005210 0x7fad1400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:30.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.579+0000 7fad2ea00700 1 -- 172.21.15.89:0/680108444 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad14005210 msgr2=0x7fad1400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:30.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.579+0000 7fad2ea00700 1 --2- 172.21.15.89:0/680108444 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad14005210 0x7fad1400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:30.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.579+0000 7fad2ea00700 1 -- 172.21.15.89:0/680108444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad14003a00 msgr2=0x7fad14003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:30.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.579+0000 7fad2ea00700 1 --2- 172.21.15.89:0/680108444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad14003a00 0x7fad14003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:30.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.579+0000 7fad2ea00700 1 -- 172.21.15.89:0/680108444 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c0094a0 con 0x7fad14004850 2024-04-02T09:14:30.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.579+0000 7fad2ea00700 1 --2- 172.21.15.89:0/680108444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad14004850 0x7fad14004cd0 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009d30 tx=0x7fad14005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665237d2a6568b8c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:30.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.580+0000 7fad2e000700 1 -- 172.21.15.89:0/680108444 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c009e80 con 0x7fad14004850 2024-04-02T09:14:30.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.580+0000 7fad2e000700 1 -- 172.21.15.89:0/680108444 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c00e940 con 0x7fad14004850 2024-04-02T09:14:30.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.580+0000 7fad2e000700 1 -- 172.21.15.89:0/680108444 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c00ecb0 con 0x7fad14004850 2024-04-02T09:14:30.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.580+0000 7fad36200700 1 -- 172.21.15.89:0/680108444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad14004850 msgr2=0x7fad14004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:30.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.580+0000 7fad36200700 1 --2- 172.21.15.89:0/680108444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad14004850 0x7fad14004cd0 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009d30 tx=0x7fad14005880 comp rx=0 tx=0).stop 2024-04-02T09:14:30.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.580+0000 7fad2fe00700 1 -- 172.21.15.89:0/680108444 reap_dead start 2024-04-02T09:14:30.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.580+0000 7fad36200700 1 -- 172.21.15.89:0/680108444 shutdown_connections 2024-04-02T09:14:30.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.580+0000 7fad36200700 1 -- 172.21.15.89:0/680108444 >> 172.21.15.89:0/680108444 conn(0x7fad300cf190 msgr2=0x7fad300cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:30.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.580+0000 7fad36200700 1 -- 172.21.15.89:0/680108444 shutdown_connections 2024-04-02T09:14:30.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.580+0000 7fad36200700 1 -- 172.21.15.89:0/680108444 wait complete. 2024-04-02T09:14:30.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.583+0000 7fad36200700 1 Processor -- start 2024-04-02T09:14:30.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.583+0000 7fad36200700 1 -- start start 2024-04-02T09:14:30.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.584+0000 7fad36200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad14004850 0x7fad300cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:30.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.584+0000 7fad36200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad300ca3c0 0x7fad3006cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:30.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.584+0000 7fad36200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad30157ec0 0x7fad30158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:30.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.584+0000 7fad36200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad300cd150 con 0x7fad30157ec0 2024-04-02T09:14:30.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.584+0000 7fad36200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad300cc4d0 con 0x7fad14004850 2024-04-02T09:14:30.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.584+0000 7fad36200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad300cc7b0 con 0x7fad300ca3c0 2024-04-02T09:14:30.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.584+0000 7fad2fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad30157ec0 0x7fad30158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:30.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.584+0000 7fad2fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad30157ec0 0x7fad30158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59570/0 (socket says 172.21.15.89:59570) 2024-04-02T09:14:30.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.584+0000 7fad2fe00700 1 -- 172.21.15.89:0/2481921626 learned_addr learned my addr 172.21.15.89:0/2481921626 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:30.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.584+0000 7fad2f400700 1 --2- 172.21.15.89:0/2481921626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad14004850 0x7fad300cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:30.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.584+0000 7fad2ea00700 1 --2- 172.21.15.89:0/2481921626 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad300ca3c0 0x7fad3006cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:30.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.585+0000 7fad2fe00700 1 -- 172.21.15.89:0/2481921626 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad300ca3c0 msgr2=0x7fad3006cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:30.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.585+0000 7fad2fe00700 1 --2- 172.21.15.89:0/2481921626 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad300ca3c0 0x7fad3006cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:30.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.585+0000 7fad2fe00700 1 -- 172.21.15.89:0/2481921626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad14004850 msgr2=0x7fad300cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:30.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.585+0000 7fad2fe00700 1 --2- 172.21.15.89:0/2481921626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad14004850 0x7fad300cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:30.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.585+0000 7fad2fe00700 1 -- 172.21.15.89:0/2481921626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad300cca90 con 0x7fad30157ec0 2024-04-02T09:14:30.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.585+0000 7fad2fe00700 1 --2- 172.21.15.89:0/2481921626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad30157ec0 0x7fad30158350 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7fad3006c790 tx=0x7fad28009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:30.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.585+0000 7fad27400700 1 -- 172.21.15.89:0/2481921626 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2800e070 con 0x7fad30157ec0 2024-04-02T09:14:30.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.585+0000 7fad27400700 1 -- 172.21.15.89:0/2481921626 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad28003c30 con 0x7fad30157ec0 2024-04-02T09:14:30.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.585+0000 7fad27400700 1 -- 172.21.15.89:0/2481921626 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad28003dc0 con 0x7fad30157ec0 2024-04-02T09:14:30.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.585+0000 7fad36200700 1 -- 172.21.15.89:0/2481921626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad300ccd70 con 0x7fad30157ec0 2024-04-02T09:14:30.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.585+0000 7fad36200700 1 -- 172.21.15.89:0/2481921626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3006d180 con 0x7fad30157ec0 2024-04-02T09:14:30.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.586+0000 7fad36200700 1 -- 172.21.15.89:0/2481921626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3006d370 con 0x7fad30157ec0 2024-04-02T09:14:30.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.587+0000 7fad27400700 1 -- 172.21.15.89:0/2481921626 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fad280119b0 con 0x7fad30157ec0 2024-04-02T09:14:30.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.588+0000 7fad27400700 1 --2- 172.21.15.89:0/2481921626 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fad1006aed0 0x7fad1006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:30.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.588+0000 7fad27400700 1 -- 172.21.15.89:0/2481921626 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fad28089c50 con 0x7fad30157ec0 2024-04-02T09:14:30.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.588+0000 7fad2f400700 1 --2- 172.21.15.89:0/2481921626 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fad1006aed0 0x7fad1006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:30.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.589+0000 7fad2f400700 1 --2- 172.21.15.89:0/2481921626 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fad1006aed0 0x7fad1006d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fad200015d0 tx=0x7fad20007980 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:30.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.592+0000 7fad27400700 1 -- 172.21.15.89:0/2481921626 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fad28054ab0 con 0x7fad30157ec0 2024-04-02T09:14:30.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.875+0000 7fad36200700 1 -- 172.21.15.89:0/2481921626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fad300cd290 con 0x7fad30157ec0 2024-04-02T09:14:30.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.875+0000 7fad27400700 1 -- 172.21.15.89:0/2481921626 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v56) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fad300cd290 con 0x7fad30157ec0 2024-04-02T09:14:30.874 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:30.874 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:14:30.874 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T09:14:30.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.874 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T09:14:30.874 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T09:14:30.874 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.874 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:14:30.874 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:14:30.874 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:14:30.875 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.875 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.875 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T09:14:30.875 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T09:14:30.875 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.875 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T09:14:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T09:14:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:14:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:14:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:14:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T09:14:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T09:14:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.878 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:14:30.878 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:14:30.878 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.878 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.878 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T09:14:30.878 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T09:14:30.878 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.878 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:14:30.878 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:14:30.878 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.878 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.879 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:14:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:14:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T09:14:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T09:14:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:14:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:14:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T09:14:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T09:14:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:14:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:14:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T09:14:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T09:14:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:14:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:14:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T09:14:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T09:14:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:14:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:14:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T09:14:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T09:14:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:14:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2024-04-02T09:14:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T09:14:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T09:14:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:14:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T09:14:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T09:14:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:14:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:14:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:14:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:14:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T09:14:30.888 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:14:30.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.877+0000 7fad36200700 1 -- 172.21.15.89:0/2481921626 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fad1006aed0 msgr2=0x7fad1006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:30.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.877+0000 7fad36200700 1 --2- 172.21.15.89:0/2481921626 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fad1006aed0 0x7fad1006d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fad200015d0 tx=0x7fad20007980 comp rx=0 tx=0).stop 2024-04-02T09:14:30.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.877+0000 7fad36200700 1 -- 172.21.15.89:0/2481921626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad30157ec0 msgr2=0x7fad30158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:30.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.877+0000 7fad36200700 1 --2- 172.21.15.89:0/2481921626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad30157ec0 0x7fad30158350 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7fad3006c790 tx=0x7fad28009f90 comp rx=0 tx=0).stop 2024-04-02T09:14:30.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.878+0000 7fad36200700 1 -- 172.21.15.89:0/2481921626 shutdown_connections 2024-04-02T09:14:30.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.878+0000 7fad36200700 1 --2- 172.21.15.89:0/2481921626 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fad1006aed0 0x7fad1006d390 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:30.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.878+0000 7fad36200700 1 --2- 172.21.15.89:0/2481921626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad14004850 0x7fad300cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:30.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.878+0000 7fad36200700 1 --2- 172.21.15.89:0/2481921626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad30157ec0 0x7fad30158350 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:30.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.878+0000 7fad36200700 1 --2- 172.21.15.89:0/2481921626 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad300ca3c0 0x7fad3006cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:30.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.878+0000 7fad36200700 1 -- 172.21.15.89:0/2481921626 >> 172.21.15.89:0/2481921626 conn(0x7fad300cf190 msgr2=0x7fad30156d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:30.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.878+0000 7fad36200700 1 -- 172.21.15.89:0/2481921626 shutdown_connections 2024-04-02T09:14:30.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:30.878+0000 7fad36200700 1 -- 172.21.15.89:0/2481921626 wait complete. 2024-04-02T09:14:30.898 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:14:31.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.091+0000 7f69c2000700 1 Processor -- start 2024-04-02T09:14:31.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.091+0000 7f69c2000700 1 -- start start 2024-04-02T09:14:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.092+0000 7f69c2000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f69bc0d0a40 0x7f69bc0b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.092+0000 7f69c2000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69bc0d0ef0 con 0x7f69bc0d05e0 2024-04-02T09:14:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.092+0000 7f69c2000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69bc0d24e0 con 0x7f69bc15da50 2024-04-02T09:14:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.092+0000 7f69c2000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69bc0b95d0 con 0x7f69bc0d0a40 2024-04-02T09:14:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.093+0000 7f69baa00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f69bc0d05e0 0x7f69bc0b6990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50200/0 (socket says 172.21.15.89:50200) 2024-04-02T09:14:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.093+0000 7f69baa00700 1 -- 172.21.15.89:0/2353163206 learned_addr learned my addr 172.21.15.89:0/2353163206 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.093+0000 7f69bbe00700 1 --2- 172.21.15.89:0/2353163206 >> v2:172.21.15.113:3300/0 conn(0x7f69bc0d0a40 0x7f69bc0b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.093+0000 7f69ba000700 1 -- 172.21.15.89:0/2353163206 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267792620 0 0) 0x7f69bc0d0ef0 con 0x7f69bc0d05e0 2024-04-02T09:14:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.093+0000 7f69ba000700 1 -- 172.21.15.89:0/2353163206 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69a00035b0 con 0x7f69bc0d05e0 2024-04-02T09:14:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.093+0000 7f69bbe00700 1 -- 172.21.15.89:0/2353163206 >> v1:172.21.15.113:6789/0 conn(0x7f69bc15da50 legacy=0x7f69bc0ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.093+0000 7f69bbe00700 1 -- 172.21.15.89:0/2353163206 >> v1:172.21.15.89:6789/0 conn(0x7f69bc0d05e0 legacy=0x7f69bc0b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.093+0000 7f69bbe00700 1 -- 172.21.15.89:0/2353163206 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc0d27c0 con 0x7f69bc0d0a40 2024-04-02T09:14:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.094+0000 7f69bbe00700 1 --2- 172.21.15.89:0/2353163206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69bc0d0a40 0x7f69bc0b70a0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009d90 tx=0x7f69b400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be80769c4dd72f56 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.094+0000 7f69ba000700 1 -- 172.21.15.89:0/2353163206 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b400e070 con 0x7f69bc0d0a40 2024-04-02T09:14:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.094+0000 7f69ba000700 1 -- 172.21.15.89:0/2353163206 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b400bb20 con 0x7f69bc0d0a40 2024-04-02T09:14:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.094+0000 7f69ba000700 1 -- 172.21.15.89:0/2353163206 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b400be90 con 0x7f69bc0d0a40 2024-04-02T09:14:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.094+0000 7f69c2000700 1 -- 172.21.15.89:0/2353163206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69bc0d0a40 msgr2=0x7f69bc0b70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.094+0000 7f69c2000700 1 --2- 172.21.15.89:0/2353163206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69bc0d0a40 0x7f69bc0b70a0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009d90 tx=0x7f69b400b330 comp rx=0 tx=0).stop 2024-04-02T09:14:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.094+0000 7f69c2000700 1 -- 172.21.15.89:0/2353163206 shutdown_connections 2024-04-02T09:14:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.094+0000 7f69c2000700 1 --2- 172.21.15.89:0/2353163206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69bc0d0a40 0x7f69bc0b70a0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.094+0000 7f69c2000700 1 -- 172.21.15.89:0/2353163206 >> 172.21.15.89:0/2353163206 conn(0x7f69bc0666a0 msgr2=0x7f69bc066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.095+0000 7f69c2000700 1 -- 172.21.15.89:0/2353163206 shutdown_connections 2024-04-02T09:14:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.095+0000 7f69c2000700 1 -- 172.21.15.89:0/2353163206 wait complete. 2024-04-02T09:14:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.096+0000 7f69c2000700 1 Processor -- start 2024-04-02T09:14:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.096+0000 7f69c2000700 1 -- start start 2024-04-02T09:14:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.097+0000 7f69c2000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69bc0d05e0 0x7f69bc151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.097+0000 7f69c2000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69bc0d0a40 0x7f69bc0ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.097+0000 7f69c2000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69bc15da50 0x7f69bc1603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.097+0000 7f69c2000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc0ca930 con 0x7f69bc0d0a40 2024-04-02T09:14:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.097+0000 7f69c2000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc1f3880 con 0x7f69bc0d05e0 2024-04-02T09:14:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.097+0000 7f69c2000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc1f3b60 con 0x7f69bc15da50 2024-04-02T09:14:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.097+0000 7f69baa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69bc0d0a40 0x7f69bc0ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.097+0000 7f69baa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69bc0d0a40 0x7f69bc0ccb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59574/0 (socket says 172.21.15.89:59574) 2024-04-02T09:14:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.097+0000 7f69baa00700 1 -- 172.21.15.89:0/4012901409 learned_addr learned my addr 172.21.15.89:0/4012901409 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.098+0000 7f69bbe00700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69bc15da50 0x7f69bc1603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.098+0000 7f69bb400700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69bc0d05e0 0x7f69bc151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.098+0000 7f69baa00700 1 -- 172.21.15.89:0/4012901409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69bc15da50 msgr2=0x7f69bc1603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.098+0000 7f69baa00700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69bc15da50 0x7f69bc1603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.098+0000 7f69baa00700 1 -- 172.21.15.89:0/4012901409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69bc0d05e0 msgr2=0x7f69bc151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.098+0000 7f69baa00700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69bc0d05e0 0x7f69bc151e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.098+0000 7f69baa00700 1 -- 172.21.15.89:0/4012901409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b4009a70 con 0x7f69bc0d0a40 2024-04-02T09:14:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.098+0000 7f69bbe00700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69bc15da50 0x7f69bc1603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.098+0000 7f69baa00700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69bc0d0a40 0x7f69bc0ccb20 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f69a8005910 tx=0x7f69a800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.098+0000 7f69b3400700 1 -- 172.21.15.89:0/4012901409 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8005d70 con 0x7f69bc0d0a40 2024-04-02T09:14:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.098+0000 7f69b3400700 1 -- 172.21.15.89:0/4012901409 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a80070d0 con 0x7f69bc0d0a40 2024-04-02T09:14:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.098+0000 7f69c2000700 1 -- 172.21.15.89:0/4012901409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69bc1f3e40 con 0x7f69bc0d0a40 2024-04-02T09:14:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.098+0000 7f69b3400700 1 -- 172.21.15.89:0/4012901409 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8007440 con 0x7f69bc0d0a40 2024-04-02T09:14:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.098+0000 7f69c2000700 1 -- 172.21.15.89:0/4012901409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69bc1523b0 con 0x7f69bc0d0a40 2024-04-02T09:14:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.099+0000 7f69c2000700 1 -- 172.21.15.89:0/4012901409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69bc0caaa0 con 0x7f69bc0d0a40 2024-04-02T09:14:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.100+0000 7f69b3400700 1 -- 172.21.15.89:0/4012901409 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f69a80075a0 con 0x7f69bc0d0a40 2024-04-02T09:14:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.101+0000 7f69b3400700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f699c06aed0 0x7f699c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.101+0000 7f69b3400700 1 -- 172.21.15.89:0/4012901409 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f69a80888f0 con 0x7f69bc0d0a40 2024-04-02T09:14:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.101+0000 7f69bb400700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f699c06aed0 0x7f699c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:31.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.102+0000 7f69bb400700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f699c06aed0 0x7f699c06d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f69ac001800 tx=0x7f69ac009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.106+0000 7f69b3400700 1 -- 172.21.15.89:0/4012901409 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f69bc0caaa0 con 0x7f69bc0d0a40 2024-04-02T09:14:31.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.366+0000 7f69c2000700 1 -- 172.21.15.89:0/4012901409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69bc0caa70 con 0x7f69bc0d0a40 2024-04-02T09:14:31.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.368+0000 7f69b3400700 1 -- 172.21.15.89:0/4012901409 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v261) v1 ==== 74+0+13088 (secure 0 0 0) 0x7f69a8017020 con 0x7f69bc0d0a40 2024-04-02T09:14:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:31.367 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":261,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:14:30.355870+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:14:31.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.370+0000 7f69c2000700 1 -- 172.21.15.89:0/4012901409 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f699c06aed0 msgr2=0x7f699c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:31.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.370+0000 7f69c2000700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f699c06aed0 0x7f699c06d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f69ac001800 tx=0x7f69ac009040 comp rx=0 tx=0).stop 2024-04-02T09:14:31.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.370+0000 7f69c2000700 1 -- 172.21.15.89:0/4012901409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69bc0d0a40 msgr2=0x7f69bc0ccb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:31.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.370+0000 7f69c2000700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69bc0d0a40 0x7f69bc0ccb20 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f69a8005910 tx=0x7f69a800c9e0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.370+0000 7f69c2000700 1 -- 172.21.15.89:0/4012901409 shutdown_connections 2024-04-02T09:14:31.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.370+0000 7f69c2000700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f699c06aed0 0x7f699c06d390 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.370+0000 7f69c2000700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69bc0d05e0 0x7f69bc151e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.370+0000 7f69c2000700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69bc15da50 0x7f69bc1603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.370+0000 7f69c2000700 1 --2- 172.21.15.89:0/4012901409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69bc0d0a40 0x7f69bc0ccb20 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.370+0000 7f69c2000700 1 -- 172.21.15.89:0/4012901409 >> 172.21.15.89:0/4012901409 conn(0x7f69bc0666a0 msgr2=0x7f69bc15c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:31.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.371+0000 7f69c2000700 1 -- 172.21.15.89:0/4012901409 shutdown_connections 2024-04-02T09:14:31.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.371+0000 7f69c2000700 1 -- 172.21.15.89:0/4012901409 wait complete. 2024-04-02T09:14:31.387 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:14:31.387 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:14:31.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.583+0000 7f8605c00700 1 Processor -- start 2024-04-02T09:14:31.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.584+0000 7f8605c00700 1 -- start start 2024-04-02T09:14:31.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.585+0000 7f8605c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86001611b0 0x7f860015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:31.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.585+0000 7f8605c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f860015f4d0 0x7f86000ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:31.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.585+0000 7f8605c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86000ce270 con 0x7f86000d21f0 2024-04-02T09:14:31.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.585+0000 7f8605c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860015b600 con 0x7f860015f4d0 2024-04-02T09:14:31.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.585+0000 7f8605c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86000d2760 con 0x7f86001611b0 2024-04-02T09:14:31.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.585+0000 7f85fe000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f860015f4d0 0x7f86000ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:31.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.585+0000 7f85fe000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f860015f4d0 0x7f86000ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:47164/0 (socket says 172.21.15.89:47164) 2024-04-02T09:14:31.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.585+0000 7f85ff400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f86000d21f0 0x7f8600161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:56904/0 (socket says 172.21.15.89:56904) 2024-04-02T09:14:31.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.585+0000 7f85fe000700 1 -- 172.21.15.89:0/1581956753 learned_addr learned my addr 172.21.15.89:0/1581956753 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:31.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.585+0000 7f85fea00700 1 --2- 172.21.15.89:0/1581956753 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86001611b0 0x7f860015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.586+0000 7f85fe000700 1 -- 172.21.15.89:0/1581956753 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86001611b0 msgr2=0x7f860015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.586+0000 7f85fe000700 1 --2- 172.21.15.89:0/1581956753 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86001611b0 0x7f860015ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.586+0000 7f85fe000700 1 -- 172.21.15.89:0/1581956753 >> v1:172.21.15.113:6789/0 conn(0x7f86000d21f0 legacy=0x7f8600161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.586+0000 7f85fe000700 1 -- 172.21.15.89:0/1581956753 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86000d2a40 con 0x7f860015f4d0 2024-04-02T09:14:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.586+0000 7f85fd600700 1 -- 172.21.15.89:0/1581956753 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953930894 0 0) 0x7f86000ce270 con 0x7f86000d21f0 2024-04-02T09:14:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.586+0000 7f85fe000700 1 --2- 172.21.15.89:0/1581956753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f860015f4d0 0x7f86000ce700 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0098f0 tx=0x7f85ec009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e519df2476b82d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.586+0000 7f85fd600700 1 -- 172.21.15.89:0/1581956753 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec011830 con 0x7f860015f4d0 2024-04-02T09:14:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.587+0000 7f85fd600700 1 -- 172.21.15.89:0/1581956753 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ec011990 con 0x7f860015f4d0 2024-04-02T09:14:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.587+0000 7f85fd600700 1 -- 172.21.15.89:0/1581956753 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec011d20 con 0x7f860015f4d0 2024-04-02T09:14:31.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.587+0000 7f8605c00700 1 -- 172.21.15.89:0/1581956753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f860015f4d0 msgr2=0x7f86000ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:31.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.587+0000 7f8605c00700 1 --2- 172.21.15.89:0/1581956753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f860015f4d0 0x7f86000ce700 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0098f0 tx=0x7f85ec009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:31.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.587+0000 7f8605c00700 1 -- 172.21.15.89:0/1581956753 shutdown_connections 2024-04-02T09:14:31.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.587+0000 7f8605c00700 1 --2- 172.21.15.89:0/1581956753 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86001611b0 0x7f860015ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.587+0000 7f8605c00700 1 --2- 172.21.15.89:0/1581956753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f860015f4d0 0x7f86000ce700 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.587+0000 7f8605c00700 1 -- 172.21.15.89:0/1581956753 >> 172.21.15.89:0/1581956753 conn(0x7f860008aff0 msgr2=0x7f86000d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:31.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.587+0000 7f8605c00700 1 -- 172.21.15.89:0/1581956753 shutdown_connections 2024-04-02T09:14:31.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.587+0000 7f8605c00700 1 -- 172.21.15.89:0/1581956753 wait complete. 2024-04-02T09:14:31.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.589+0000 7f8605c00700 1 Processor -- start 2024-04-02T09:14:31.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.589+0000 7f8605c00700 1 -- start start 2024-04-02T09:14:31.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.590+0000 7f8605c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86000d21f0 0x7f8600156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:31.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.590+0000 7f8605c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f860015f4d0 0x7f86000cd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:31.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.590+0000 7f8605c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86001611b0 0x7f8600158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:31.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.590+0000 7f8605c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86000cd5b0 con 0x7f860015f4d0 2024-04-02T09:14:31.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.590+0000 7f8605c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86000b73e0 con 0x7f86000d21f0 2024-04-02T09:14:31.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.590+0000 7f8605c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86000b7630 con 0x7f86001611b0 2024-04-02T09:14:31.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.590+0000 7f85fe000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f860015f4d0 0x7f86000cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:31.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.590+0000 7f85fe000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f860015f4d0 0x7f86000cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59576/0 (socket says 172.21.15.89:59576) 2024-04-02T09:14:31.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.590+0000 7f85fe000700 1 -- 172.21.15.89:0/3731285165 learned_addr learned my addr 172.21.15.89:0/3731285165 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:31.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.590+0000 7f85fea00700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86000d21f0 0x7f8600156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:31.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.590+0000 7f85ff400700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86001611b0 0x7f8600158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:31.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.591+0000 7f85fe000700 1 -- 172.21.15.89:0/3731285165 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86001611b0 msgr2=0x7f8600158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:31.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.591+0000 7f85fe000700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86001611b0 0x7f8600158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.591+0000 7f85fe000700 1 -- 172.21.15.89:0/3731285165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86000d21f0 msgr2=0x7f8600156cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:31.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.591+0000 7f85fe000700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86000d21f0 0x7f8600156cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.591+0000 7f85fe000700 1 -- 172.21.15.89:0/3731285165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec009550 con 0x7f860015f4d0 2024-04-02T09:14:31.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.591+0000 7f85fea00700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86000d21f0 0x7f8600156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:31.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.591+0000 7f85fe000700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f860015f4d0 0x7f86000cd070 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f85ec009c40 tx=0x7f85ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:31.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.591+0000 7f85f6a00700 1 -- 172.21.15.89:0/3731285165 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec004400 con 0x7f860015f4d0 2024-04-02T09:14:31.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.591+0000 7f85f6a00700 1 -- 172.21.15.89:0/3731285165 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ec01f070 con 0x7f860015f4d0 2024-04-02T09:14:31.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.591+0000 7f85f6a00700 1 -- 172.21.15.89:0/3731285165 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec0038c0 con 0x7f860015f4d0 2024-04-02T09:14:31.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.591+0000 7f8605c00700 1 -- 172.21.15.89:0/3731285165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86000b7910 con 0x7f860015f4d0 2024-04-02T09:14:31.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.591+0000 7f8605c00700 1 -- 172.21.15.89:0/3731285165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86000b7ed0 con 0x7f860015f4d0 2024-04-02T09:14:31.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.592+0000 7f8605c00700 1 -- 172.21.15.89:0/3731285165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8600067720 con 0x7f860015f4d0 2024-04-02T09:14:31.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.593+0000 7f85f6a00700 1 -- 172.21.15.89:0/3731285165 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f85ec021030 con 0x7f860015f4d0 2024-04-02T09:14:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.595+0000 7f85f6a00700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f85e006aed0 0x7f85e006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.595+0000 7f85f6a00700 1 -- 172.21.15.89:0/3731285165 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f85ec08a9c0 con 0x7f860015f4d0 2024-04-02T09:14:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.595+0000 7f85fea00700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f85e006aed0 0x7f85e006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:31.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.596+0000 7f85fea00700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f85e006aed0 0x7f85e006d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009b90 tx=0x7f85f0008040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:31.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.599+0000 7f85f6a00700 1 -- 172.21.15.89:0/3731285165 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f85ec055ae0 con 0x7f860015f4d0 2024-04-02T09:14:31.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.798+0000 7f8605c00700 1 -- 172.21.15.89:0/3731285165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f86000b5d00 con 0x7f860015f4d0 2024-04-02T09:14:31.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.799+0000 7f85f6a00700 1 -- 172.21.15.89:0/3731285165 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v261) v1 ==== 93+0+10 (secure 0 0 0) 0x7f85ec015020 con 0x7f860015f4d0 2024-04-02T09:14:31.797 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:14:31.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.801+0000 7f8605c00700 1 -- 172.21.15.89:0/3731285165 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f85e006aed0 msgr2=0x7f85e006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:31.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.801+0000 7f8605c00700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f85e006aed0 0x7f85e006d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009b90 tx=0x7f85f0008040 comp rx=0 tx=0).stop 2024-04-02T09:14:31.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.801+0000 7f8605c00700 1 -- 172.21.15.89:0/3731285165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f860015f4d0 msgr2=0x7f86000cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:31.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.801+0000 7f8605c00700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f860015f4d0 0x7f86000cd070 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f85ec009c40 tx=0x7f85ec004750 comp rx=0 tx=0).stop 2024-04-02T09:14:31.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.801+0000 7f8605c00700 1 -- 172.21.15.89:0/3731285165 shutdown_connections 2024-04-02T09:14:31.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.801+0000 7f8605c00700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f85e006aed0 0x7f85e006d390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.801+0000 7f8605c00700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86000d21f0 0x7f8600156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.801+0000 7f8605c00700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86001611b0 0x7f8600158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.801+0000 7f8605c00700 1 --2- 172.21.15.89:0/3731285165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f860015f4d0 0x7f86000cd070 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:31.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.801+0000 7f8605c00700 1 -- 172.21.15.89:0/3731285165 >> 172.21.15.89:0/3731285165 conn(0x7f860008aff0 msgr2=0x7f860015da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:31.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.801+0000 7f8605c00700 1 -- 172.21.15.89:0/3731285165 shutdown_connections 2024-04-02T09:14:31.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:31.801+0000 7f8605c00700 1 -- 172.21.15.89:0/3731285165 wait complete. 2024-04-02T09:14:31.818 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T09:14:32.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.011+0000 7fb16b600700 1 Processor -- start 2024-04-02T09:14:32.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.011+0000 7fb16b600700 1 -- start start 2024-04-02T09:14:32.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.012+0000 7fb16b600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb164160be0 0x7fb1640cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:32.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.012+0000 7fb16b600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb1640ceca0 0x7fb1640b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:32.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.012+0000 7fb16b600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb164160fc0 con 0x7fb1640ce0e0 2024-04-02T09:14:32.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.012+0000 7fb16b600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1640b65d0 con 0x7fb164160be0 2024-04-02T09:14:32.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.012+0000 7fb16b600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1640d2a80 con 0x7fb1640ceca0 2024-04-02T09:14:32.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.013+0000 7fb168c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb164160be0 0x7fb1640cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:32.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.013+0000 7fb168c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb164160be0 0x7fb1640cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59590/0 (socket says 172.21.15.89:59590) 2024-04-02T09:14:32.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.013+0000 7fb168c00700 1 -- 172.21.15.89:0/1755864226 learned_addr learned my addr 172.21.15.89:0/1755864226 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:32.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.013+0000 7fb168c00700 1 -- 172.21.15.89:0/1755864226 >> v2:172.21.15.113:3300/0 conn(0x7fb1640ceca0 msgr2=0x7fb1640b6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:32.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.013+0000 7fb169600700 1 --2- 172.21.15.89:0/1755864226 >> v2:172.21.15.113:3300/0 conn(0x7fb1640ceca0 0x7fb1640b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:32.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.013+0000 7fb168c00700 1 --2- 172.21.15.89:0/1755864226 >> v2:172.21.15.113:3300/0 conn(0x7fb1640ceca0 0x7fb1640b6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.013+0000 7fb168c00700 1 -- 172.21.15.89:0/1755864226 >> v1:172.21.15.89:6789/0 conn(0x7fb1640ce0e0 legacy=0x7fb1640ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:32.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.013+0000 7fb168c00700 1 -- 172.21.15.89:0/1755864226 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1640d21e0 con 0x7fb164160be0 2024-04-02T09:14:32.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.013+0000 7fb169600700 1 --2- 172.21.15.89:0/1755864226 >> v2:172.21.15.113:3300/0 conn(0x7fb1640ceca0 0x7fb1640b6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:32.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.014+0000 7fb168c00700 1 --2- 172.21.15.89:0/1755864226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb164160be0 0x7fb1640cdba0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7fb16408d150 tx=0x7fb158009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901192c9a333ea11 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:32.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.014+0000 7fb163400700 1 -- 172.21.15.89:0/1755864226 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15800b870 con 0x7fb164160be0 2024-04-02T09:14:32.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.014+0000 7fb163400700 1 -- 172.21.15.89:0/1755864226 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15800b9d0 con 0x7fb164160be0 2024-04-02T09:14:32.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.014+0000 7fb163400700 1 -- 172.21.15.89:0/1755864226 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15800bd60 con 0x7fb164160be0 2024-04-02T09:14:32.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.014+0000 7fb16b600700 1 -- 172.21.15.89:0/1755864226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb164160be0 msgr2=0x7fb1640cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:32.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.014+0000 7fb16b600700 1 --2- 172.21.15.89:0/1755864226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb164160be0 0x7fb1640cdba0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7fb16408d150 tx=0x7fb158009bd0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.014+0000 7fb16b600700 1 -- 172.21.15.89:0/1755864226 shutdown_connections 2024-04-02T09:14:32.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.014+0000 7fb16b600700 1 --2- 172.21.15.89:0/1755864226 >> v2:172.21.15.113:3300/0 conn(0x7fb1640ceca0 0x7fb1640b6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.014+0000 7fb16b600700 1 --2- 172.21.15.89:0/1755864226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb164160be0 0x7fb1640cdba0 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.014+0000 7fb16b600700 1 -- 172.21.15.89:0/1755864226 >> 172.21.15.89:0/1755864226 conn(0x7fb16408aff0 msgr2=0x7fb1640d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:32.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.014+0000 7fb16b600700 1 -- 172.21.15.89:0/1755864226 shutdown_connections 2024-04-02T09:14:32.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.015+0000 7fb16b600700 1 -- 172.21.15.89:0/1755864226 wait complete. 2024-04-02T09:14:32.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.016+0000 7fb16b600700 1 Processor -- start 2024-04-02T09:14:32.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.016+0000 7fb16b600700 1 -- start start 2024-04-02T09:14:32.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.017+0000 7fb16b600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1640ce0e0 0x7fb1641f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:32.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.017+0000 7fb16b600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1640ceca0 0x7fb1641f5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:32.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.017+0000 7fb16b600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb164160be0 0x7fb1640c8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:32.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.017+0000 7fb16b600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1641f60d0 con 0x7fb164160be0 2024-04-02T09:14:32.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.017+0000 7fb16b600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1640c8740 con 0x7fb1640ceca0 2024-04-02T09:14:32.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.017+0000 7fb16b600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1640c8a10 con 0x7fb1640ce0e0 2024-04-02T09:14:32.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.017+0000 7fb169600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb164160be0 0x7fb1640c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:32.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.017+0000 7fb169600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb164160be0 0x7fb1640c8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59604/0 (socket says 172.21.15.89:59604) 2024-04-02T09:14:32.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.017+0000 7fb169600700 1 -- 172.21.15.89:0/3139314697 learned_addr learned my addr 172.21.15.89:0/3139314697 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:32.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.018+0000 7fb163e00700 1 --2- 172.21.15.89:0/3139314697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1640ceca0 0x7fb1641f5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:32.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.018+0000 7fb168c00700 1 --2- 172.21.15.89:0/3139314697 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1640ce0e0 0x7fb1641f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:32.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.018+0000 7fb169600700 1 -- 172.21.15.89:0/3139314697 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1640ce0e0 msgr2=0x7fb1641f3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:32.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.018+0000 7fb169600700 1 --2- 172.21.15.89:0/3139314697 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1640ce0e0 0x7fb1641f3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.018+0000 7fb169600700 1 -- 172.21.15.89:0/3139314697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1640ceca0 msgr2=0x7fb1641f5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:32.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.018+0000 7fb169600700 1 --2- 172.21.15.89:0/3139314697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1640ceca0 0x7fb1641f5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.018+0000 7fb169600700 1 -- 172.21.15.89:0/3139314697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb158009550 con 0x7fb164160be0 2024-04-02T09:14:32.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.018+0000 7fb169600700 1 --2- 172.21.15.89:0/3139314697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb164160be0 0x7fb1640c8200 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7fb15c00ce80 tx=0x7fb15c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:32.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.018+0000 7fb160c00700 1 -- 172.21.15.89:0/3139314697 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c00ad40 con 0x7fb164160be0 2024-04-02T09:14:32.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.018+0000 7fb160c00700 1 -- 172.21.15.89:0/3139314697 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c015440 con 0x7fb164160be0 2024-04-02T09:14:32.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.018+0000 7fb160c00700 1 -- 172.21.15.89:0/3139314697 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c00aea0 con 0x7fb164160be0 2024-04-02T09:14:32.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.018+0000 7fb16b600700 1 -- 172.21.15.89:0/3139314697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1640c8cf0 con 0x7fb164160be0 2024-04-02T09:14:32.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.019+0000 7fb16b600700 1 -- 172.21.15.89:0/3139314697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1640c92b0 con 0x7fb164160be0 2024-04-02T09:14:32.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.019+0000 7fb16b600700 1 -- 172.21.15.89:0/3139314697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1641f3ec0 con 0x7fb164160be0 2024-04-02T09:14:32.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.020+0000 7fb160c00700 1 -- 172.21.15.89:0/3139314697 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fb15c00d430 con 0x7fb164160be0 2024-04-02T09:14:32.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.021+0000 7fb160c00700 1 --2- 172.21.15.89:0/3139314697 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb14406aed0 0x7fb14406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:32.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.021+0000 7fb160c00700 1 -- 172.21.15.89:0/3139314697 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fb15c089a00 con 0x7fb164160be0 2024-04-02T09:14:32.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.021+0000 7fb168c00700 1 --2- 172.21.15.89:0/3139314697 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb14406aed0 0x7fb14406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:32.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.022+0000 7fb168c00700 1 --2- 172.21.15.89:0/3139314697 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb14406aed0 0x7fb14406d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb158014040 tx=0x7fb158008030 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:32.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.026+0000 7fb160c00700 1 -- 172.21.15.89:0/3139314697 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb15c054a00 con 0x7fb164160be0 2024-04-02T09:14:32.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.229+0000 7fb16b600700 1 -- 172.21.15.89:0/3139314697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb164067720 con 0x7fb164160be0 2024-04-02T09:14:32.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.230+0000 7fb160c00700 1 -- 172.21.15.89:0/3139314697 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v261) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb15c01a020 con 0x7fb164160be0 2024-04-02T09:14:32.228 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:32.228 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T09:14:32.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:32.229 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T09:14:32.229 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T09:14:32.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:32.229 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T09:14:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.232+0000 7fb16b600700 1 -- 172.21.15.89:0/3139314697 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb14406aed0 msgr2=0x7fb14406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.232+0000 7fb16b600700 1 --2- 172.21.15.89:0/3139314697 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb14406aed0 0x7fb14406d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb158014040 tx=0x7fb158008030 comp rx=0 tx=0).stop 2024-04-02T09:14:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.232+0000 7fb16b600700 1 -- 172.21.15.89:0/3139314697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb164160be0 msgr2=0x7fb1640c8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.232+0000 7fb16b600700 1 --2- 172.21.15.89:0/3139314697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb164160be0 0x7fb1640c8200 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7fb15c00ce80 tx=0x7fb15c00ceb0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.232+0000 7fb16b600700 1 -- 172.21.15.89:0/3139314697 shutdown_connections 2024-04-02T09:14:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.232+0000 7fb16b600700 1 --2- 172.21.15.89:0/3139314697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb164160be0 0x7fb1640c8200 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.232+0000 7fb16b600700 1 --2- 172.21.15.89:0/3139314697 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb1640ce0e0 0x7fb1641f3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.232+0000 7fb16b600700 1 --2- 172.21.15.89:0/3139314697 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb14406aed0 0x7fb14406d390 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.232+0000 7fb16b600700 1 --2- 172.21.15.89:0/3139314697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1640ceca0 0x7fb1641f5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.232+0000 7fb16b600700 1 -- 172.21.15.89:0/3139314697 >> 172.21.15.89:0/3139314697 conn(0x7fb16408aff0 msgr2=0x7fb1640d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.232+0000 7fb16b600700 1 -- 172.21.15.89:0/3139314697 shutdown_connections 2024-04-02T09:14:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.232+0000 7fb16b600700 1 -- 172.21.15.89:0/3139314697 wait complete. 2024-04-02T09:14:32.249 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T09:14:32.249 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T09:14:32.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.445+0000 7fec24e00700 1 Processor -- start 2024-04-02T09:14:32.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.446+0000 7fec24e00700 1 -- start start 2024-04-02T09:14:32.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.447+0000 7fec24e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2015ddf0 0x7fec200ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:32.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.447+0000 7fec24e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec200ce8a0 con 0x7fec2015e1d0 2024-04-02T09:14:32.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.447+0000 7fec24e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec200cdc90 con 0x7fec200d28e0 2024-04-02T09:14:32.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.447+0000 7fec24e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2015ec60 con 0x7fec2015ddf0 2024-04-02T09:14:32.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.448+0000 7fec17e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fec2015e1d0 0x7fec200d21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50214/0 (socket says 172.21.15.89:50214) 2024-04-02T09:14:32.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.448+0000 7fec17e00700 1 -- 172.21.15.89:0/3926275758 learned_addr learned my addr 172.21.15.89:0/3926275758 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:32.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.448+0000 7fec1d400700 1 -- 172.21.15.89:0/3926275758 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728286912 0 0) 0x7fec200ce8a0 con 0x7fec2015e1d0 2024-04-02T09:14:32.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.448+0000 7fec1de00700 1 --2- 172.21.15.89:0/3926275758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2015ddf0 0x7fec200ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:32.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.448+0000 7fec1d400700 1 -- 172.21.15.89:0/3926275758 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec040035b0 con 0x7fec2015e1d0 2024-04-02T09:14:32.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.448+0000 7fec1d400700 1 -- 172.21.15.89:0/3926275758 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3807209578 0 0) 0x7fec040035b0 con 0x7fec2015e1d0 2024-04-02T09:14:32.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.448+0000 7fec1d400700 1 -- 172.21.15.89:0/3926275758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2015ddf0 msgr2=0x7fec200ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:32.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.448+0000 7fec1d400700 1 --2- 172.21.15.89:0/3926275758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2015ddf0 0x7fec200ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.448+0000 7fec1d400700 1 -- 172.21.15.89:0/3926275758 >> v1:172.21.15.113:6789/0 conn(0x7fec200d28e0 legacy=0x7fec2015f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:32.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.448+0000 7fec1d400700 1 -- 172.21.15.89:0/3926275758 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec200cdf70 con 0x7fec2015e1d0 2024-04-02T09:14:32.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.448+0000 7fec1d400700 1 -- 172.21.15.89:0/3926275758 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fec08001670 con 0x7fec2015e1d0 2024-04-02T09:14:32.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.448+0000 7fec1d400700 1 -- 172.21.15.89:0/3926275758 >> v1:172.21.15.89:6789/0 conn(0x7fec2015e1d0 legacy=0x7fec200d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:32.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec1d400700 1 --2- 172.21.15.89:0/3926275758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec04003a30 0x7fec04003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:32.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec1d400700 1 --2- 172.21.15.89:0/3926275758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2015ddf0 0x7fec04004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:32.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec1d400700 1 --2- 172.21.15.89:0/3926275758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec04004d90 0x7fec04005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:32.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec1d400700 1 -- 172.21.15.89:0/3926275758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2015ec60 con 0x7fec04003a30 2024-04-02T09:14:32.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec1d400700 1 -- 172.21.15.89:0/3926275758 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0400bac0 con 0x7fec04004d90 2024-04-02T09:14:32.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec1d400700 1 -- 172.21.15.89:0/3926275758 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0400bc80 con 0x7fec2015ddf0 2024-04-02T09:14:32.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec1de00700 1 --2- 172.21.15.89:0/3926275758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec04003a30 0x7fec04003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:32.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec1e800700 1 --2- 172.21.15.89:0/3926275758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec04004d90 0x7fec04005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:32.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec17e00700 1 --2- 172.21.15.89:0/3926275758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2015ddf0 0x7fec04004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:32.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec1de00700 1 -- 172.21.15.89:0/3926275758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2015ddf0 msgr2=0x7fec04004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:32.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec1de00700 1 --2- 172.21.15.89:0/3926275758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec2015ddf0 0x7fec04004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec1de00700 1 -- 172.21.15.89:0/3926275758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec04004d90 msgr2=0x7fec04005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:32.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec1de00700 1 --2- 172.21.15.89:0/3926275758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec04004d90 0x7fec04005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec1de00700 1 -- 172.21.15.89:0/3926275758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec040035b0 con 0x7fec04003a30 2024-04-02T09:14:32.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.449+0000 7fec1de00700 1 --2- 172.21.15.89:0/3926275758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec04003a30 0x7fec04003e10 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7fec10009b60 tx=0x7fec04005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10293213c0890803 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:32.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.450+0000 7fec1d400700 1 -- 172.21.15.89:0/3926275758 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1000e6c0 con 0x7fec04003a30 2024-04-02T09:14:32.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.450+0000 7fec1d400700 1 -- 172.21.15.89:0/3926275758 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1000e820 con 0x7fec04003a30 2024-04-02T09:14:32.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.450+0000 7fec1d400700 1 -- 172.21.15.89:0/3926275758 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1000eb90 con 0x7fec04003a30 2024-04-02T09:14:32.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.450+0000 7fec24e00700 1 -- 172.21.15.89:0/3926275758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec04003a30 msgr2=0x7fec04003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:32.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.450+0000 7fec24e00700 1 --2- 172.21.15.89:0/3926275758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec04003a30 0x7fec04003e10 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7fec10009b60 tx=0x7fec04005880 comp rx=0 tx=0).stop 2024-04-02T09:14:32.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.450+0000 7fec1e800700 1 -- 172.21.15.89:0/3926275758 reap_dead start 2024-04-02T09:14:32.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.450+0000 7fec24e00700 1 -- 172.21.15.89:0/3926275758 shutdown_connections 2024-04-02T09:14:32.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.450+0000 7fec24e00700 1 -- 172.21.15.89:0/3926275758 >> 172.21.15.89:0/3926275758 conn(0x7fec2008aff0 msgr2=0x7fec200d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:32.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.450+0000 7fec24e00700 1 -- 172.21.15.89:0/3926275758 shutdown_connections 2024-04-02T09:14:32.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.450+0000 7fec24e00700 1 -- 172.21.15.89:0/3926275758 wait complete. 2024-04-02T09:14:32.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.452+0000 7fec24e00700 1 Processor -- start 2024-04-02T09:14:32.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.452+0000 7fec24e00700 1 -- start start 2024-04-02T09:14:32.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.453+0000 7fec24e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec04003a30 0x7fec20155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:32.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.453+0000 7fec24e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec201568f0 0x7fec200c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:32.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.453+0000 7fec24e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec201563a0 0x7fec200c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.453+0000 7fec24e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec201e8aa0 con 0x7fec04003a30 2024-04-02T09:14:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.453+0000 7fec24e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec201e7b40 con 0x7fec201563a0 2024-04-02T09:14:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.453+0000 7fec24e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec201e7e20 con 0x7fec201568f0 2024-04-02T09:14:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.453+0000 7fec1de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec04003a30 0x7fec20155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.453+0000 7fec1de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec04003a30 0x7fec20155ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59618/0 (socket says 172.21.15.89:59618) 2024-04-02T09:14:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.453+0000 7fec1de00700 1 -- 172.21.15.89:0/2277305818 learned_addr learned my addr 172.21.15.89:0/2277305818 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.453+0000 7fec17e00700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec201568f0 0x7fec200c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.453+0000 7fec1e800700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec201563a0 0x7fec200c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.454+0000 7fec1de00700 1 -- 172.21.15.89:0/2277305818 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec201568f0 msgr2=0x7fec200c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.454+0000 7fec1de00700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec201568f0 0x7fec200c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.454+0000 7fec1de00700 1 -- 172.21.15.89:0/2277305818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec201563a0 msgr2=0x7fec200c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.454+0000 7fec1de00700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec201563a0 0x7fec200c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.454+0000 7fec1de00700 1 -- 172.21.15.89:0/2277305818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec201e8100 con 0x7fec04003a30 2024-04-02T09:14:32.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.454+0000 7fec1e800700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec201563a0 0x7fec200c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:32.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.454+0000 7fec1de00700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec04003a30 0x7fec20155ad0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7fec040047b0 tx=0x7fec100047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:32.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.454+0000 7fec16000700 1 -- 172.21.15.89:0/2277305818 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec10021070 con 0x7fec04003a30 2024-04-02T09:14:32.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.454+0000 7fec16000700 1 -- 172.21.15.89:0/2277305818 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec10004520 con 0x7fec04003a30 2024-04-02T09:14:32.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.454+0000 7fec24e00700 1 -- 172.21.15.89:0/2277305818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec201e83e0 con 0x7fec04003a30 2024-04-02T09:14:32.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.454+0000 7fec16000700 1 -- 172.21.15.89:0/2277305818 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1000e6c0 con 0x7fec04003a30 2024-04-02T09:14:32.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.454+0000 7fec24e00700 1 -- 172.21.15.89:0/2277305818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec20156010 con 0x7fec04003a30 2024-04-02T09:14:32.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.455+0000 7fec24e00700 1 -- 172.21.15.89:0/2277305818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec201561a0 con 0x7fec04003a30 2024-04-02T09:14:32.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.456+0000 7fec16000700 1 -- 172.21.15.89:0/2277305818 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fec10025070 con 0x7fec04003a30 2024-04-02T09:14:32.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.457+0000 7fec16000700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7febfc06aed0 0x7febfc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:32.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.457+0000 7fec16000700 1 -- 172.21.15.89:0/2277305818 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fec1008a630 con 0x7fec04003a30 2024-04-02T09:14:32.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.457+0000 7fec17e00700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7febfc06aed0 0x7febfc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:32.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.458+0000 7fec17e00700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7febfc06aed0 0x7febfc06d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fec08001cb0 tx=0x7fec08007040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:32.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.461+0000 7fec16000700 1 -- 172.21.15.89:0/2277305818 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fec100556d0 con 0x7fec04003a30 2024-04-02T09:14:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.660+0000 7fec24e00700 1 -- 172.21.15.89:0/2277305818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fec200b3500 con 0x7fec04003a30 2024-04-02T09:14:32.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.845+0000 7fec16000700 1 -- 172.21.15.89:0/2277305818 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v262) v1 ==== 138+0+0 (secure 0 0 0) 0x7fec10058cf0 con 0x7fec04003a30 2024-04-02T09:14:32.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.849+0000 7fec24e00700 1 -- 172.21.15.89:0/2277305818 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7febfc06aed0 msgr2=0x7febfc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:32.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.849+0000 7fec24e00700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7febfc06aed0 0x7febfc06d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fec08001cb0 tx=0x7fec08007040 comp rx=0 tx=0).stop 2024-04-02T09:14:32.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.849+0000 7fec24e00700 1 -- 172.21.15.89:0/2277305818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec04003a30 msgr2=0x7fec20155ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:32.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.849+0000 7fec24e00700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec04003a30 0x7fec20155ad0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7fec040047b0 tx=0x7fec100047c0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.849+0000 7fec24e00700 1 -- 172.21.15.89:0/2277305818 shutdown_connections 2024-04-02T09:14:32.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.849+0000 7fec24e00700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7febfc06aed0 0x7febfc06d390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.849+0000 7fec24e00700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec201563a0 0x7fec200c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.849+0000 7fec24e00700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec04003a30 0x7fec20155ad0 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.849+0000 7fec24e00700 1 --2- 172.21.15.89:0/2277305818 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fec201568f0 0x7fec200c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:32.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.849+0000 7fec24e00700 1 -- 172.21.15.89:0/2277305818 >> 172.21.15.89:0/2277305818 conn(0x7fec2008aff0 msgr2=0x7fec200b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:32.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.850+0000 7fec24e00700 1 -- 172.21.15.89:0/2277305818 shutdown_connections 2024-04-02T09:14:32.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:32.850+0000 7fec24e00700 1 -- 172.21.15.89:0/2277305818 wait complete. 2024-04-02T09:14:32.849 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T09:14:32.867 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T09:14:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.069+0000 7f8f85200700 1 Processor -- start 2024-04-02T09:14:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.069+0000 7f8f85200700 1 -- start start 2024-04-02T09:14:33.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.070+0000 7f8f85200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f8015d9d0 0x7f8f800d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:33.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.070+0000 7f8f85200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8f800d16a0 0x7f8f800d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:33.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.070+0000 7f8f85200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8f800d2370 0x7f8f8015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:33.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.070+0000 7f8f85200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f800d2820 con 0x7f8f800d16a0 2024-04-02T09:14:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.070+0000 7f8f85200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f80161420 con 0x7f8f800d2370 2024-04-02T09:14:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.070+0000 7f8f85200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f80161700 con 0x7f8f8015d9d0 2024-04-02T09:14:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.070+0000 7f8f77e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8f800d16a0 0x7f8f800d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.070+0000 7f8f77e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8f800d16a0 0x7f8f800d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59624/0 (socket says 172.21.15.89:59624) 2024-04-02T09:14:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.070+0000 7f8f77e00700 1 -- 172.21.15.89:0/3276912667 learned_addr learned my addr 172.21.15.89:0/3276912667 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.070+0000 7f8f7de00700 1 --2- 172.21.15.89:0/3276912667 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f8015d9d0 0x7f8f800d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.070+0000 7f8f7e800700 1 --2- 172.21.15.89:0/3276912667 >> v2:172.21.15.113:3300/0 conn(0x7f8f800d2370 0x7f8f8015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.071+0000 7f8f77e00700 1 -- 172.21.15.89:0/3276912667 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f8015d9d0 msgr2=0x7f8f800d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.071+0000 7f8f77e00700 1 --2- 172.21.15.89:0/3276912667 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f8015d9d0 0x7f8f800d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.071+0000 7f8f77e00700 1 -- 172.21.15.89:0/3276912667 >> v2:172.21.15.113:3300/0 conn(0x7f8f800d2370 msgr2=0x7f8f8015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.071+0000 7f8f77e00700 1 --2- 172.21.15.89:0/3276912667 >> v2:172.21.15.113:3300/0 conn(0x7f8f800d2370 0x7f8f8015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.071+0000 7f8f77e00700 1 -- 172.21.15.89:0/3276912667 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f801619e0 con 0x7f8f800d16a0 2024-04-02T09:14:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.071+0000 7f8f77e00700 1 --2- 172.21.15.89:0/3276912667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f800d16a0 0x7f8f800d1e30 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f8f800cdc00 tx=0x7f8f68009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cbc967bcf876029 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:33.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.071+0000 7f8f7d400700 1 -- 172.21.15.89:0/3276912667 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f68005e30 con 0x7f8f800d16a0 2024-04-02T09:14:33.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.071+0000 7f8f7d400700 1 -- 172.21.15.89:0/3276912667 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f68004030 con 0x7f8f800d16a0 2024-04-02T09:14:33.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.071+0000 7f8f7d400700 1 -- 172.21.15.89:0/3276912667 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f680043e0 con 0x7f8f800d16a0 2024-04-02T09:14:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.072+0000 7f8f85200700 1 -- 172.21.15.89:0/3276912667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f800d16a0 msgr2=0x7f8f800d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.072+0000 7f8f85200700 1 --2- 172.21.15.89:0/3276912667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f800d16a0 0x7f8f800d1e30 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f8f800cdc00 tx=0x7f8f68009bd0 comp rx=0 tx=0).stop 2024-04-02T09:14:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.072+0000 7f8f85200700 1 -- 172.21.15.89:0/3276912667 shutdown_connections 2024-04-02T09:14:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.072+0000 7f8f85200700 1 --2- 172.21.15.89:0/3276912667 >> v2:172.21.15.113:3300/0 conn(0x7f8f800d2370 0x7f8f8015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.072+0000 7f8f85200700 1 --2- 172.21.15.89:0/3276912667 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f8015d9d0 0x7f8f800d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.072+0000 7f8f85200700 1 --2- 172.21.15.89:0/3276912667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f800d16a0 0x7f8f800d1e30 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.072+0000 7f8f85200700 1 -- 172.21.15.89:0/3276912667 >> 172.21.15.89:0/3276912667 conn(0x7f8f8008aff0 msgr2=0x7f8f800d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.072+0000 7f8f85200700 1 -- 172.21.15.89:0/3276912667 shutdown_connections 2024-04-02T09:14:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.072+0000 7f8f85200700 1 -- 172.21.15.89:0/3276912667 wait complete. 2024-04-02T09:14:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.074+0000 7f8f85200700 1 Processor -- start 2024-04-02T09:14:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.074+0000 7f8f85200700 1 -- start start 2024-04-02T09:14:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f85200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f800d16a0 0x7f8f80156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f85200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f800d2370 0x7f8f800b8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f85200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f8015d9d0 0x7f8f800b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f85200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f80159300 con 0x7f8f8015d9d0 2024-04-02T09:14:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f85200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f80156fa0 con 0x7f8f800d16a0 2024-04-02T09:14:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f85200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f801586f0 con 0x7f8f800d2370 2024-04-02T09:14:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f7e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f8015d9d0 0x7f8f800b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f7e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f8015d9d0 0x7f8f800b1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59632/0 (socket says 172.21.15.89:59632) 2024-04-02T09:14:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f77e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f800d2370 0x7f8f800b8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f7e800700 1 -- 172.21.15.89:0/2666878525 learned_addr learned my addr 172.21.15.89:0/2666878525 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f7de00700 1 --2- 172.21.15.89:0/2666878525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f800d16a0 0x7f8f80156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f7e800700 1 -- 172.21.15.89:0/2666878525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f800d2370 msgr2=0x7f8f800b8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f7e800700 1 --2- 172.21.15.89:0/2666878525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f800d2370 0x7f8f800b8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f7e800700 1 -- 172.21.15.89:0/2666878525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f800d16a0 msgr2=0x7f8f80156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f7e800700 1 --2- 172.21.15.89:0/2666878525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f800d16a0 0x7f8f80156a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.075+0000 7f8f7e800700 1 -- 172.21.15.89:0/2666878525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f68009500 con 0x7f8f8015d9d0 2024-04-02T09:14:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.076+0000 7f8f7e800700 1 --2- 172.21.15.89:0/2666878525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f8015d9d0 0x7f8f800b1350 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800eed0 tx=0x7f8f7800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.076+0000 7f8f76000700 1 -- 172.21.15.89:0/2666878525 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f78010040 con 0x7f8f8015d9d0 2024-04-02T09:14:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.076+0000 7f8f76000700 1 -- 172.21.15.89:0/2666878525 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7800bc80 con 0x7f8f8015d9d0 2024-04-02T09:14:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.076+0000 7f8f85200700 1 -- 172.21.15.89:0/2666878525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f801589d0 con 0x7f8f8015d9d0 2024-04-02T09:14:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.076+0000 7f8f76000700 1 -- 172.21.15.89:0/2666878525 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f78009e40 con 0x7f8f8015d9d0 2024-04-02T09:14:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.076+0000 7f8f85200700 1 -- 172.21.15.89:0/2666878525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f80158f90 con 0x7f8f8015d9d0 2024-04-02T09:14:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.077+0000 7f8f85200700 1 -- 172.21.15.89:0/2666878525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f800d1ca0 con 0x7f8f8015d9d0 2024-04-02T09:14:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.078+0000 7f8f76000700 1 -- 172.21.15.89:0/2666878525 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f8f78014400 con 0x7f8f8015d9d0 2024-04-02T09:14:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.079+0000 7f8f76000700 1 --2- 172.21.15.89:0/2666878525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f5c06aed0 0x7f8f5c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.079+0000 7f8f76000700 1 -- 172.21.15.89:0/2666878525 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f8f78089bf0 con 0x7f8f8015d9d0 2024-04-02T09:14:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.079+0000 7f8f7de00700 1 --2- 172.21.15.89:0/2666878525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f5c06aed0 0x7f8f5c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.080+0000 7f8f7de00700 1 --2- 172.21.15.89:0/2666878525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f5c06aed0 0x7f8f5c06d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009b40 tx=0x7f8f70009210 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:33.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.084+0000 7f8f76000700 1 -- 172.21.15.89:0/2666878525 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8f78054970 con 0x7f8f8015d9d0 2024-04-02T09:14:33.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.284+0000 7f8f85200700 1 -- 172.21.15.89:0/2666878525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8f80067720 con 0x7f8f8015d9d0 2024-04-02T09:14:33.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.855+0000 7f8f76000700 1 -- 172.21.15.89:0/2666878525 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v263) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8f78018020 con 0x7f8f8015d9d0 2024-04-02T09:14:33.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.863+0000 7f8f85200700 1 -- 172.21.15.89:0/2666878525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f5c06aed0 msgr2=0x7f8f5c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:33.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.863+0000 7f8f85200700 1 --2- 172.21.15.89:0/2666878525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f5c06aed0 0x7f8f5c06d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009b40 tx=0x7f8f70009210 comp rx=0 tx=0).stop 2024-04-02T09:14:33.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.863+0000 7f8f85200700 1 -- 172.21.15.89:0/2666878525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f8015d9d0 msgr2=0x7f8f800b1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:33.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.863+0000 7f8f85200700 1 --2- 172.21.15.89:0/2666878525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f8015d9d0 0x7f8f800b1350 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800eed0 tx=0x7f8f7800b390 comp rx=0 tx=0).stop 2024-04-02T09:14:33.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.864+0000 7f8f85200700 1 -- 172.21.15.89:0/2666878525 shutdown_connections 2024-04-02T09:14:33.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.864+0000 7f8f85200700 1 --2- 172.21.15.89:0/2666878525 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f5c06aed0 0x7f8f5c06d390 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:33.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.864+0000 7f8f85200700 1 --2- 172.21.15.89:0/2666878525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f800d16a0 0x7f8f80156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:33.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.864+0000 7f8f85200700 1 --2- 172.21.15.89:0/2666878525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f8015d9d0 0x7f8f800b1350 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:33.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.864+0000 7f8f85200700 1 --2- 172.21.15.89:0/2666878525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f800d2370 0x7f8f800b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:33.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.864+0000 7f8f85200700 1 -- 172.21.15.89:0/2666878525 >> 172.21.15.89:0/2666878525 conn(0x7f8f8008aff0 msgr2=0x7f8f800ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:33.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.864+0000 7f8f85200700 1 -- 172.21.15.89:0/2666878525 shutdown_connections 2024-04-02T09:14:33.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:33.865+0000 7f8f85200700 1 -- 172.21.15.89:0/2666878525 wait complete. 2024-04-02T09:14:33.867 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T09:14:33.889 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T09:14:34.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.086+0000 7fba79a00700 1 Processor -- start 2024-04-02T09:14:34.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.086+0000 7fba79a00700 1 -- start start 2024-04-02T09:14:34.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.087+0000 7fba79a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba740ced80 0x7fba740d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.088+0000 7fba79a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fba7415fcb0 0x7fba74160140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.088+0000 7fba79a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba740cf1e0 con 0x7fba7415dad0 2024-04-02T09:14:34.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.088+0000 7fba79a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba740cfcc0 con 0x7fba7415fcb0 2024-04-02T09:14:34.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.088+0000 7fba79a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba740cd9f0 con 0x7fba740ced80 2024-04-02T09:14:34.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.088+0000 7fba72a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fba7415dad0 0x7fba740d1630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50230/0 (socket says 172.21.15.89:50230) 2024-04-02T09:14:34.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.088+0000 7fba72a00700 1 -- 172.21.15.89:0/1365636646 learned_addr learned my addr 172.21.15.89:0/1365636646 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:34.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.088+0000 7fba73400700 1 --2- 172.21.15.89:0/1365636646 >> v2:172.21.15.113:3300/0 conn(0x7fba7415fcb0 0x7fba74160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.088+0000 7fba72000700 1 --2- 172.21.15.89:0/1365636646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba740ced80 0x7fba740d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.088+0000 7fba71600700 1 -- 172.21.15.89:0/1365636646 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047295044 0 0) 0x7fba64001110 con 0x7fba7415dad0 2024-04-02T09:14:34.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.088+0000 7fba71600700 1 -- 172.21.15.89:0/1365636646 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba580035b0 con 0x7fba7415dad0 2024-04-02T09:14:34.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.088+0000 7fba71600700 1 -- 172.21.15.89:0/1365636646 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2713994476 0 0) 0x7fba580035b0 con 0x7fba7415dad0 2024-04-02T09:14:34.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.088+0000 7fba71600700 1 -- 172.21.15.89:0/1365636646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba740ced80 msgr2=0x7fba740d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:34.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.088+0000 7fba71600700 1 --2- 172.21.15.89:0/1365636646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba740ced80 0x7fba740d1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba71600700 1 -- 172.21.15.89:0/1365636646 >> v2:172.21.15.113:3300/0 conn(0x7fba7415fcb0 msgr2=0x7fba74160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:34.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba71600700 1 --2- 172.21.15.89:0/1365636646 >> v2:172.21.15.113:3300/0 conn(0x7fba7415fcb0 0x7fba74160140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba71600700 1 -- 172.21.15.89:0/1365636646 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7415f820 con 0x7fba7415dad0 2024-04-02T09:14:34.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba71600700 1 -- 172.21.15.89:0/1365636646 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fba64001800 con 0x7fba7415dad0 2024-04-02T09:14:34.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba71600700 1 -- 172.21.15.89:0/1365636646 >> v1:172.21.15.89:6789/0 conn(0x7fba7415dad0 legacy=0x7fba740d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:34.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba71600700 1 --2- 172.21.15.89:0/1365636646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba740ced80 0x7fba58003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba71600700 1 --2- 172.21.15.89:0/1365636646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba58003f80 0x7fba58004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba71600700 1 --2- 172.21.15.89:0/1365636646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba58004d90 0x7fba58005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba71600700 1 -- 172.21.15.89:0/1365636646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba740cfcc0 con 0x7fba58003f80 2024-04-02T09:14:34.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba71600700 1 -- 172.21.15.89:0/1365636646 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba740cd9f0 con 0x7fba58004d90 2024-04-02T09:14:34.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba71600700 1 -- 172.21.15.89:0/1365636646 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5800ba10 con 0x7fba740ced80 2024-04-02T09:14:34.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba72000700 1 --2- 172.21.15.89:0/1365636646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba58003f80 0x7fba58004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba72a00700 1 --2- 172.21.15.89:0/1365636646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba740ced80 0x7fba58003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba73400700 1 --2- 172.21.15.89:0/1365636646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba58004d90 0x7fba58005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba72000700 1 -- 172.21.15.89:0/1365636646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba740ced80 msgr2=0x7fba58003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:34.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba72000700 1 --2- 172.21.15.89:0/1365636646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba740ced80 0x7fba58003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba72000700 1 -- 172.21.15.89:0/1365636646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba58004d90 msgr2=0x7fba58005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:34.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba72000700 1 --2- 172.21.15.89:0/1365636646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba58004d90 0x7fba58005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.089+0000 7fba72000700 1 -- 172.21.15.89:0/1365636646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba580035b0 con 0x7fba58003f80 2024-04-02T09:14:34.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.090+0000 7fba72000700 1 --2- 172.21.15.89:0/1365636646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba58003f80 0x7fba58004850 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7fba60009b60 tx=0x7fba58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e3d041656c64ad server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:34.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.090+0000 7fba71600700 1 -- 172.21.15.89:0/1365636646 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6000e660 con 0x7fba58003f80 2024-04-02T09:14:34.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.090+0000 7fba71600700 1 -- 172.21.15.89:0/1365636646 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6000e7c0 con 0x7fba58003f80 2024-04-02T09:14:34.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.090+0000 7fba71600700 1 -- 172.21.15.89:0/1365636646 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6000eb30 con 0x7fba58003f80 2024-04-02T09:14:34.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.090+0000 7fba79a00700 1 -- 172.21.15.89:0/1365636646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba58003f80 msgr2=0x7fba58004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:34.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.090+0000 7fba79a00700 1 --2- 172.21.15.89:0/1365636646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba58003f80 0x7fba58004850 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7fba60009b60 tx=0x7fba58005880 comp rx=0 tx=0).stop 2024-04-02T09:14:34.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.090+0000 7fba73400700 1 -- 172.21.15.89:0/1365636646 reap_dead start 2024-04-02T09:14:34.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.090+0000 7fba79a00700 1 -- 172.21.15.89:0/1365636646 shutdown_connections 2024-04-02T09:14:34.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.090+0000 7fba79a00700 1 -- 172.21.15.89:0/1365636646 >> 172.21.15.89:0/1365636646 conn(0x7fba7408aff0 msgr2=0x7fba740d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:34.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.090+0000 7fba79a00700 1 -- 172.21.15.89:0/1365636646 shutdown_connections 2024-04-02T09:14:34.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.091+0000 7fba79a00700 1 -- 172.21.15.89:0/1365636646 wait complete. 2024-04-02T09:14:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.092+0000 7fba79a00700 1 Processor -- start 2024-04-02T09:14:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.092+0000 7fba79a00700 1 -- start start 2024-04-02T09:14:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.093+0000 7fba79a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba58003f80 0x7fba740ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.093+0000 7fba79a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba740cbdf0 0x7fba740b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.093+0000 7fba79a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba74151550 0x7fba74158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.093+0000 7fba79a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba740cc620 con 0x7fba740cbdf0 2024-04-02T09:14:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.093+0000 7fba79a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba740b7570 con 0x7fba58003f80 2024-04-02T09:14:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.093+0000 7fba79a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba740cc340 con 0x7fba74151550 2024-04-02T09:14:34.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.093+0000 7fba72000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba740cbdf0 0x7fba740b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.094+0000 7fba72000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba740cbdf0 0x7fba740b7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59654/0 (socket says 172.21.15.89:59654) 2024-04-02T09:14:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.094+0000 7fba72000700 1 -- 172.21.15.89:0/3154373951 learned_addr learned my addr 172.21.15.89:0/3154373951 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.094+0000 7fba72a00700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba58003f80 0x7fba740ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.094+0000 7fba73400700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba74151550 0x7fba74158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.094+0000 7fba72000700 1 -- 172.21.15.89:0/3154373951 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba74151550 msgr2=0x7fba74158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.094+0000 7fba72000700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba74151550 0x7fba74158c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.094+0000 7fba72000700 1 -- 172.21.15.89:0/3154373951 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba58003f80 msgr2=0x7fba740ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.094+0000 7fba72000700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba58003f80 0x7fba740ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.094+0000 7fba72000700 1 -- 172.21.15.89:0/3154373951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba740b6f20 con 0x7fba740cbdf0 2024-04-02T09:14:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.094+0000 7fba73400700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba74151550 0x7fba74158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.094+0000 7fba72000700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba740cbdf0 0x7fba740b7a00 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7fba6000e4b0 tx=0x7fba600047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.094+0000 7fba6aa00700 1 -- 172.21.15.89:0/3154373951 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6000e660 con 0x7fba740cbdf0 2024-04-02T09:14:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.094+0000 7fba6aa00700 1 -- 172.21.15.89:0/3154373951 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba60003ab0 con 0x7fba740cbdf0 2024-04-02T09:14:34.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.095+0000 7fba79a00700 1 -- 172.21.15.89:0/3154373951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba740b7200 con 0x7fba740cbdf0 2024-04-02T09:14:34.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.095+0000 7fba6aa00700 1 -- 172.21.15.89:0/3154373951 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60003e60 con 0x7fba740cbdf0 2024-04-02T09:14:34.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.095+0000 7fba79a00700 1 -- 172.21.15.89:0/3154373951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba740b7f40 con 0x7fba740cbdf0 2024-04-02T09:14:34.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.096+0000 7fba79a00700 1 -- 172.21.15.89:0/3154373951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba740b6d80 con 0x7fba740cbdf0 2024-04-02T09:14:34.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.097+0000 7fba6aa00700 1 -- 172.21.15.89:0/3154373951 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fba60013470 con 0x7fba740cbdf0 2024-04-02T09:14:34.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.099+0000 7fba6aa00700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fba5406af00 0x7fba5406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.099+0000 7fba6aa00700 1 -- 172.21.15.89:0/3154373951 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fba6008b7f0 con 0x7fba740cbdf0 2024-04-02T09:14:34.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.099+0000 7fba72a00700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fba5406af00 0x7fba5406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.100+0000 7fba72a00700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fba5406af00 0x7fba5406d3c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fba64001050 tx=0x7fba64009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:34.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.103+0000 7fba6aa00700 1 -- 172.21.15.89:0/3154373951 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fba600567a0 con 0x7fba740cbdf0 2024-04-02T09:14:34.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.350+0000 7fba79a00700 1 -- 172.21.15.89:0/3154373951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fba740b6d80 con 0x7fba740cbdf0 2024-04-02T09:14:34.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.358+0000 7fba6aa00700 1 -- 172.21.15.89:0/3154373951 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 34 and data pool 35 v131) v1 ==== 174+0+0 (secure 0 0 0) 0x7fba740b6d80 con 0x7fba740cbdf0 2024-04-02T09:14:34.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.363+0000 7fba79a00700 1 -- 172.21.15.89:0/3154373951 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fba5406af00 msgr2=0x7fba5406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:34.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.363+0000 7fba79a00700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fba5406af00 0x7fba5406d3c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fba64001050 tx=0x7fba64009040 comp rx=0 tx=0).stop 2024-04-02T09:14:34.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.363+0000 7fba79a00700 1 -- 172.21.15.89:0/3154373951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba740cbdf0 msgr2=0x7fba740b7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:34.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.363+0000 7fba79a00700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba740cbdf0 0x7fba740b7a00 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7fba6000e4b0 tx=0x7fba600047c0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.363+0000 7fba79a00700 1 -- 172.21.15.89:0/3154373951 shutdown_connections 2024-04-02T09:14:34.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.363+0000 7fba79a00700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fba5406af00 0x7fba5406d3c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.363+0000 7fba79a00700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba58003f80 0x7fba740ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.363+0000 7fba79a00700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba74151550 0x7fba74158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.363+0000 7fba79a00700 1 --2- 172.21.15.89:0/3154373951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba740cbdf0 0x7fba740b7a00 secure :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7fba6000e4b0 tx=0x7fba600047c0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.363+0000 7fba79a00700 1 -- 172.21.15.89:0/3154373951 >> 172.21.15.89:0/3154373951 conn(0x7fba7408aff0 msgr2=0x7fba740d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:34.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.365+0000 7fba79a00700 1 -- 172.21.15.89:0/3154373951 shutdown_connections 2024-04-02T09:14:34.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.366+0000 7fba79a00700 1 -- 172.21.15.89:0/3154373951 wait complete. 2024-04-02T09:14:34.369 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 34 and data pool 35 2024-04-02T09:14:34.395 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:14:34.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.587+0000 7fcae2400700 1 Processor -- start 2024-04-02T09:14:34.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.588+0000 7fcae2400700 1 -- start start 2024-04-02T09:14:34.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.589+0000 7fcae2400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcadc15d920 0x7fcadc06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.589+0000 7fcae2400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcadc099a60 0x7fcadc099e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.589+0000 7fcae2400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcadc0d08d0 con 0x7fcadc09a910 2024-04-02T09:14:34.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.589+0000 7fcae2400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcadc06c300 con 0x7fcadc099a60 2024-04-02T09:14:34.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.589+0000 7fcae2400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcadc09a380 con 0x7fcadc15d920 2024-04-02T09:14:34.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.589+0000 7fcad3e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcadc09a910 0x7fcadc0cfff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50242/0 (socket says 172.21.15.89:50242) 2024-04-02T09:14:34.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.589+0000 7fcad3e00700 1 -- 172.21.15.89:0/210738209 learned_addr learned my addr 172.21.15.89:0/210738209 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:34.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.589+0000 7fcadbe00700 1 --2- 172.21.15.89:0/210738209 >> v2:172.21.15.89:3300/0 conn(0x7fcadc099a60 0x7fcadc099e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.589+0000 7fcadb400700 1 --2- 172.21.15.89:0/210738209 >> v2:172.21.15.113:3300/0 conn(0x7fcadc15d920 0x7fcadc06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.590+0000 7fcadbe00700 1 -- 172.21.15.89:0/210738209 >> v2:172.21.15.113:3300/0 conn(0x7fcadc15d920 msgr2=0x7fcadc06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:34.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.590+0000 7fcadbe00700 1 --2- 172.21.15.89:0/210738209 >> v2:172.21.15.113:3300/0 conn(0x7fcadc15d920 0x7fcadc06c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.590+0000 7fcadbe00700 1 -- 172.21.15.89:0/210738209 >> v1:172.21.15.89:6789/0 conn(0x7fcadc09a910 legacy=0x7fcadc0cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:34.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.590+0000 7fcadbe00700 1 -- 172.21.15.89:0/210738209 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcadc06b9d0 con 0x7fcadc099a60 2024-04-02T09:14:34.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.590+0000 7fcadbe00700 1 --2- 172.21.15.89:0/210738209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc099a60 0x7fcadc099e40 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7fcadc09a6b0 tx=0x7fcad400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f64c6a8c4eec44 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:34.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.590+0000 7fcadaa00700 1 -- 172.21.15.89:0/210738209 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad400e070 con 0x7fcadc099a60 2024-04-02T09:14:34.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.590+0000 7fcadaa00700 1 -- 172.21.15.89:0/210738209 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad400bb20 con 0x7fcadc099a60 2024-04-02T09:14:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.590+0000 7fcadaa00700 1 -- 172.21.15.89:0/210738209 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad4012430 con 0x7fcadc099a60 2024-04-02T09:14:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.591+0000 7fcae2400700 1 -- 172.21.15.89:0/210738209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc099a60 msgr2=0x7fcadc099e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.591+0000 7fcae2400700 1 --2- 172.21.15.89:0/210738209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc099a60 0x7fcadc099e40 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7fcadc09a6b0 tx=0x7fcad400b330 comp rx=0 tx=0).stop 2024-04-02T09:14:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.591+0000 7fcae2400700 1 -- 172.21.15.89:0/210738209 shutdown_connections 2024-04-02T09:14:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.591+0000 7fcae2400700 1 --2- 172.21.15.89:0/210738209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc099a60 0x7fcadc099e40 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.591+0000 7fcae2400700 1 --2- 172.21.15.89:0/210738209 >> v2:172.21.15.113:3300/0 conn(0x7fcadc15d920 0x7fcadc06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.591+0000 7fcae2400700 1 -- 172.21.15.89:0/210738209 >> 172.21.15.89:0/210738209 conn(0x7fcadc0d1350 msgr2=0x7fcadc0d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.591+0000 7fcae2400700 1 -- 172.21.15.89:0/210738209 shutdown_connections 2024-04-02T09:14:34.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.591+0000 7fcae2400700 1 -- 172.21.15.89:0/210738209 wait complete. 2024-04-02T09:14:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.593+0000 7fcae2400700 1 Processor -- start 2024-04-02T09:14:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.593+0000 7fcae2400700 1 -- start start 2024-04-02T09:14:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcae2400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc099a60 0x7fcadc14ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcae2400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcadc09a910 0x7fcadc15f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcae2400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcadc15d920 0x7fcadc15e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcae2400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc15dd00 con 0x7fcadc099a60 2024-04-02T09:14:34.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcae2400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc14efd0 con 0x7fcadc09a910 2024-04-02T09:14:34.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcae2400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc14f270 con 0x7fcadc15d920 2024-04-02T09:14:34.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcadb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc099a60 0x7fcadc14ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcadb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc099a60 0x7fcadc14ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59678/0 (socket says 172.21.15.89:59678) 2024-04-02T09:14:34.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcadb400700 1 -- 172.21.15.89:0/1571474302 learned_addr learned my addr 172.21.15.89:0/1571474302 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:34.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcad3e00700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcadc09a910 0x7fcadc15f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcadbe00700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcadc15d920 0x7fcadc15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcadb400700 1 -- 172.21.15.89:0/1571474302 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcadc15d920 msgr2=0x7fcadc15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:34.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcadb400700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcadc15d920 0x7fcadc15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcadb400700 1 -- 172.21.15.89:0/1571474302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcadc09a910 msgr2=0x7fcadc15f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:34.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcadb400700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcadc09a910 0x7fcadc15f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcadb400700 1 -- 172.21.15.89:0/1571474302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad4009a70 con 0x7fcadc099a60 2024-04-02T09:14:34.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcadbe00700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcadc15d920 0x7fcadc15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:34.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.594+0000 7fcadb400700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc099a60 0x7fcadc14ea90 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7fcacc00e870 tx=0x7fcacc00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:34.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.595+0000 7fcad3400700 1 -- 172.21.15.89:0/1571474302 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc009590 con 0x7fcadc099a60 2024-04-02T09:14:34.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.595+0000 7fcad3400700 1 -- 172.21.15.89:0/1571474302 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcacc00ee70 con 0x7fcadc099a60 2024-04-02T09:14:34.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.595+0000 7fcad3400700 1 -- 172.21.15.89:0/1571474302 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc00ca50 con 0x7fcadc099a60 2024-04-02T09:14:34.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.595+0000 7fcae2400700 1 -- 172.21.15.89:0/1571474302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcadc14f550 con 0x7fcadc099a60 2024-04-02T09:14:34.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.595+0000 7fcae2400700 1 -- 172.21.15.89:0/1571474302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcadc15df10 con 0x7fcadc099a60 2024-04-02T09:14:34.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.596+0000 7fcae2400700 1 -- 172.21.15.89:0/1571474302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcadc067720 con 0x7fcadc099a60 2024-04-02T09:14:34.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.597+0000 7fcad3400700 1 -- 172.21.15.89:0/1571474302 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fcacc00cbb0 con 0x7fcadc099a60 2024-04-02T09:14:34.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.598+0000 7fcad3400700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcab806aed0 0x7fcab806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:34.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.598+0000 7fcad3400700 1 -- 172.21.15.89:0/1571474302 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fcacc089970 con 0x7fcadc099a60 2024-04-02T09:14:34.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.598+0000 7fcad3e00700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcab806aed0 0x7fcab806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:34.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.599+0000 7fcad3e00700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcab806aed0 0x7fcab806d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcadc0d10b0 tx=0x7fcac4009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:34.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.603+0000 7fcad3400700 1 -- 172.21.15.89:0/1571474302 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcacc054590 con 0x7fcadc099a60 2024-04-02T09:14:34.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.807+0000 7fcae2400700 1 -- 172.21.15.89:0/1571474302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcadc15e280 con 0x7fcadc099a60 2024-04-02T09:14:34.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.809+0000 7fcad3400700 1 -- 172.21.15.89:0/1571474302 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v264) v1 ==== 74+0+16757 (secure 0 0 0) 0x7fcacc00ce60 con 0x7fcadc099a60 2024-04-02T09:14:34.807 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:34.807 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":264,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:14:34.351433+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:14:32.705979+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-04-02T09:14:33.330752+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:14:34.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.811+0000 7fcae2400700 1 -- 172.21.15.89:0/1571474302 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcab806aed0 msgr2=0x7fcab806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:34.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.811+0000 7fcae2400700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcab806aed0 0x7fcab806d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcadc0d10b0 tx=0x7fcac4009040 comp rx=0 tx=0).stop 2024-04-02T09:14:34.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.811+0000 7fcae2400700 1 -- 172.21.15.89:0/1571474302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc099a60 msgr2=0x7fcadc14ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:34.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.811+0000 7fcae2400700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc099a60 0x7fcadc14ea90 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7fcacc00e870 tx=0x7fcacc00ec30 comp rx=0 tx=0).stop 2024-04-02T09:14:34.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.811+0000 7fcae2400700 1 -- 172.21.15.89:0/1571474302 shutdown_connections 2024-04-02T09:14:34.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.811+0000 7fcae2400700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcadc15d920 0x7fcadc15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.811+0000 7fcae2400700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc099a60 0x7fcadc14ea90 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.811+0000 7fcae2400700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcab806aed0 0x7fcab806d390 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.811+0000 7fcae2400700 1 --2- 172.21.15.89:0/1571474302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcadc09a910 0x7fcadc15f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:34.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.811+0000 7fcae2400700 1 -- 172.21.15.89:0/1571474302 >> 172.21.15.89:0/1571474302 conn(0x7fcadc0d1350 msgr2=0x7fcadc06f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:34.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.811+0000 7fcae2400700 1 -- 172.21.15.89:0/1571474302 shutdown_connections 2024-04-02T09:14:34.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:34.811+0000 7fcae2400700 1 -- 172.21.15.89:0/1571474302 wait complete. 2024-04-02T09:14:34.828 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:14:35.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.022+0000 7fccba200700 1 Processor -- start 2024-04-02T09:14:35.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.022+0000 7fccba200700 1 -- start start 2024-04-02T09:14:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.023+0000 7fccba200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccb415da50 0x7fccb40ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.023+0000 7fccba200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fccb40d0cf0 0x7fccb40b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.023+0000 7fccba200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb40d24e0 con 0x7fccb40cec70 2024-04-02T09:14:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.023+0000 7fccba200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb40b95d0 con 0x7fccb40d0cf0 2024-04-02T09:14:35.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.023+0000 7fccba200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb415de30 con 0x7fccb415da50 2024-04-02T09:14:35.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.023+0000 7fccb3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fccb40d0cf0 0x7fccb40b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:35.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.023+0000 7fccb3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fccb40d0cf0 0x7fccb40b6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59690/0 (socket says 172.21.15.89:59690) 2024-04-02T09:14:35.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.023+0000 7fccb3e00700 1 -- 172.21.15.89:0/2460847646 learned_addr learned my addr 172.21.15.89:0/2460847646 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:35.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.023+0000 7fccb3400700 1 --2- 172.21.15.89:0/2460847646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccb415da50 0x7fccb40ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:35.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.024+0000 7fccb3e00700 1 -- 172.21.15.89:0/2460847646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccb415da50 msgr2=0x7fccb40ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:35.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.024+0000 7fccb3e00700 1 --2- 172.21.15.89:0/2460847646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccb415da50 0x7fccb40ce730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.024+0000 7fccb3e00700 1 -- 172.21.15.89:0/2460847646 >> v1:172.21.15.89:6789/0 conn(0x7fccb40cec70 legacy=0x7fccb40d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:35.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.024+0000 7fccb3e00700 1 -- 172.21.15.89:0/2460847646 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb40d27c0 con 0x7fccb40d0cf0 2024-04-02T09:14:35.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.024+0000 7fccb3e00700 1 --2- 172.21.15.89:0/2460847646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccb40d0cf0 0x7fccb40b6990 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7fccac009e50 tx=0x7fccac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f892b760ef0c41 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:35.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.024+0000 7fccb2000700 1 -- 172.21.15.89:0/2460847646 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac00e070 con 0x7fccb40d0cf0 2024-04-02T09:14:35.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.024+0000 7fccb2000700 1 -- 172.21.15.89:0/2460847646 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac00b910 con 0x7fccb40d0cf0 2024-04-02T09:14:35.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.024+0000 7fccb2000700 1 -- 172.21.15.89:0/2460847646 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac00bcc0 con 0x7fccb40d0cf0 2024-04-02T09:14:35.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.025+0000 7fccba200700 1 -- 172.21.15.89:0/2460847646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccb40d0cf0 msgr2=0x7fccb40b6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:35.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.025+0000 7fccba200700 1 --2- 172.21.15.89:0/2460847646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccb40d0cf0 0x7fccb40b6990 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7fccac009e50 tx=0x7fccac00b330 comp rx=0 tx=0).stop 2024-04-02T09:14:35.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.025+0000 7fccba200700 1 -- 172.21.15.89:0/2460847646 shutdown_connections 2024-04-02T09:14:35.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.025+0000 7fccba200700 1 --2- 172.21.15.89:0/2460847646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccb40d0cf0 0x7fccb40b6990 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.025+0000 7fccba200700 1 --2- 172.21.15.89:0/2460847646 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccb415da50 0x7fccb40ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.025+0000 7fccba200700 1 -- 172.21.15.89:0/2460847646 >> 172.21.15.89:0/2460847646 conn(0x7fccb40666a0 msgr2=0x7fccb4066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:35.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.025+0000 7fccba200700 1 -- 172.21.15.89:0/2460847646 shutdown_connections 2024-04-02T09:14:35.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.025+0000 7fccba200700 1 -- 172.21.15.89:0/2460847646 wait complete. 2024-04-02T09:14:35.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.027+0000 7fccba200700 1 Processor -- start 2024-04-02T09:14:35.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.027+0000 7fccba200700 1 -- start start 2024-04-02T09:14:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.028+0000 7fccba200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccb40cec70 0x7fccb40b5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.028+0000 7fccba200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb40d0cf0 0x7fccb4151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.028+0000 7fccba200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccb415da50 0x7fccb4147aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.028+0000 7fccba200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40b51f0 con 0x7fccb415da50 2024-04-02T09:14:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.028+0000 7fccba200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40b4ec0 con 0x7fccb40d0cf0 2024-04-02T09:14:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.028+0000 7fccba200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40cb090 con 0x7fccb40cec70 2024-04-02T09:14:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.028+0000 7fccb3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccb415da50 0x7fccb4147aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.028+0000 7fccb3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccb415da50 0x7fccb4147aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59700/0 (socket says 172.21.15.89:59700) 2024-04-02T09:14:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.028+0000 7fccb3e00700 1 -- 172.21.15.89:0/4021519292 learned_addr learned my addr 172.21.15.89:0/4021519292 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.028+0000 7fccb3400700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccb40cec70 0x7fccb40b5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.028+0000 7fccb2a00700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb40d0cf0 0x7fccb4151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.029+0000 7fccb3e00700 1 -- 172.21.15.89:0/4021519292 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccb40cec70 msgr2=0x7fccb40b5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.029+0000 7fccb3e00700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccb40cec70 0x7fccb40b5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.029+0000 7fccb3e00700 1 -- 172.21.15.89:0/4021519292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb40d0cf0 msgr2=0x7fccb4151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.029+0000 7fccb3e00700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb40d0cf0 0x7fccb4151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.029+0000 7fccb3e00700 1 -- 172.21.15.89:0/4021519292 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac009b30 con 0x7fccb415da50 2024-04-02T09:14:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.029+0000 7fccb2a00700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb40d0cf0 0x7fccb4151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.029+0000 7fccb3e00700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccb415da50 0x7fccb4147aa0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7fccb40741d0 tx=0x7fccac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.029+0000 7fccab400700 1 -- 172.21.15.89:0/4021519292 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac0048f0 con 0x7fccb415da50 2024-04-02T09:14:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.029+0000 7fccab400700 1 -- 172.21.15.89:0/4021519292 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac0099b0 con 0x7fccb415da50 2024-04-02T09:14:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.029+0000 7fccba200700 1 -- 172.21.15.89:0/4021519292 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb40caad0 con 0x7fccb415da50 2024-04-02T09:14:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.029+0000 7fccab400700 1 -- 172.21.15.89:0/4021519292 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac01c680 con 0x7fccb415da50 2024-04-02T09:14:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.029+0000 7fccba200700 1 -- 172.21.15.89:0/4021519292 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb40b5bc0 con 0x7fccb415da50 2024-04-02T09:14:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.030+0000 7fccba200700 1 -- 172.21.15.89:0/4021519292 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb40b4d20 con 0x7fccb415da50 2024-04-02T09:14:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.031+0000 7fccab400700 1 -- 172.21.15.89:0/4021519292 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fccac01c7e0 con 0x7fccb415da50 2024-04-02T09:14:35.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.032+0000 7fccab400700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcc9406aed0 0x7fcc9406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:35.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.032+0000 7fccab400700 1 -- 172.21.15.89:0/4021519292 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fccac08b660 con 0x7fccb415da50 2024-04-02T09:14:35.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.032+0000 7fccb3400700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcc9406aed0 0x7fcc9406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:35.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.033+0000 7fccb3400700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcc9406aed0 0x7fcc9406d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009b40 tx=0x7fcca4009210 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:35.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.036+0000 7fccab400700 1 -- 172.21.15.89:0/4021519292 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fccb40b4d20 con 0x7fccb415da50 2024-04-02T09:14:35.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.237+0000 7fccba200700 1 -- 172.21.15.89:0/4021519292 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccb40b4d20 con 0x7fccb415da50 2024-04-02T09:14:35.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.238+0000 7fccab400700 1 -- 172.21.15.89:0/4021519292 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v264) v1 ==== 74+0+16757 (secure 0 0 0) 0x7fccb40b4d20 con 0x7fccb415da50 2024-04-02T09:14:35.237 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:35.237 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":264,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:14:34.351433+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:14:32.705979+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-04-02T09:14:33.330752+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:14:35.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.241+0000 7fccba200700 1 -- 172.21.15.89:0/4021519292 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcc9406aed0 msgr2=0x7fcc9406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:35.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.241+0000 7fccba200700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcc9406aed0 0x7fcc9406d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009b40 tx=0x7fcca4009210 comp rx=0 tx=0).stop 2024-04-02T09:14:35.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.241+0000 7fccba200700 1 -- 172.21.15.89:0/4021519292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccb415da50 msgr2=0x7fccb4147aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:35.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.241+0000 7fccba200700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccb415da50 0x7fccb4147aa0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7fccb40741d0 tx=0x7fccac004ad0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.241+0000 7fccba200700 1 -- 172.21.15.89:0/4021519292 shutdown_connections 2024-04-02T09:14:35.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.241+0000 7fccba200700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccb415da50 0x7fccb4147aa0 unknown :-1 s=CLOSED pgs=2801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.241+0000 7fccba200700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fccb40cec70 0x7fccb40b5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.241+0000 7fccba200700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcc9406aed0 0x7fcc9406d390 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.241+0000 7fccba200700 1 --2- 172.21.15.89:0/4021519292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb40d0cf0 0x7fccb4151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.241+0000 7fccba200700 1 -- 172.21.15.89:0/4021519292 >> 172.21.15.89:0/4021519292 conn(0x7fccb40666a0 msgr2=0x7fccb40b7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:35.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.241+0000 7fccba200700 1 -- 172.21.15.89:0/4021519292 shutdown_connections 2024-04-02T09:14:35.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.241+0000 7fccba200700 1 -- 172.21.15.89:0/4021519292 wait complete. 2024-04-02T09:14:35.257 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T09:14:35.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.457+0000 7fafe6400700 1 Processor -- start 2024-04-02T09:14:35.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.457+0000 7fafe6400700 1 -- start start 2024-04-02T09:14:35.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.458+0000 7fafe6400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fafe015d9d0 0x7fafe00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:35.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.458+0000 7fafe6400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fafe00d16a0 0x7fafe00d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:35.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.459+0000 7fafe6400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe0161420 con 0x7fafe00d2370 2024-04-02T09:14:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.459+0000 7fafe6400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe00d2820 con 0x7fafe00d16a0 2024-04-02T09:14:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.459+0000 7fafe6400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe0161700 con 0x7fafe015d9d0 2024-04-02T09:14:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.459+0000 7fafdea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fafe00d16a0 0x7fafe00d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.459+0000 7fafdea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fafe00d16a0 0x7fafe00d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59708/0 (socket says 172.21.15.89:59708) 2024-04-02T09:14:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.459+0000 7fafdea00700 1 -- 172.21.15.89:0/3197640436 learned_addr learned my addr 172.21.15.89:0/3197640436 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.459+0000 7fafdf400700 1 --2- 172.21.15.89:0/3197640436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fafe015d9d0 0x7fafe00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.459+0000 7fafdea00700 1 -- 172.21.15.89:0/3197640436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fafe015d9d0 msgr2=0x7fafe00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.459+0000 7fafdea00700 1 --2- 172.21.15.89:0/3197640436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fafe015d9d0 0x7fafe00d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.459+0000 7fafdea00700 1 -- 172.21.15.89:0/3197640436 >> v1:172.21.15.113:6789/0 conn(0x7fafe00d2370 legacy=0x7fafe015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.459+0000 7fafdea00700 1 -- 172.21.15.89:0/3197640436 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe01619e0 con 0x7fafe00d16a0 2024-04-02T09:14:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.459+0000 7fafdea00700 1 --2- 172.21.15.89:0/3197640436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00d16a0 0x7fafe00d1e30 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7fafcc006fd0 tx=0x7fafcc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61bc0363f61691c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:35.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.460+0000 7fafde000700 1 -- 172.21.15.89:0/3197640436 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc005e30 con 0x7fafe00d16a0 2024-04-02T09:14:35.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.460+0000 7fafde000700 1 -- 172.21.15.89:0/3197640436 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafcc006030 con 0x7fafe00d16a0 2024-04-02T09:14:35.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.460+0000 7fafde000700 1 -- 172.21.15.89:0/3197640436 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc0063c0 con 0x7fafe00d16a0 2024-04-02T09:14:35.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.460+0000 7fafe6400700 1 -- 172.21.15.89:0/3197640436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00d16a0 msgr2=0x7fafe00d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:35.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.460+0000 7fafe6400700 1 --2- 172.21.15.89:0/3197640436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00d16a0 0x7fafe00d1e30 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7fafcc006fd0 tx=0x7fafcc009bd0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.460+0000 7fafe6400700 1 -- 172.21.15.89:0/3197640436 shutdown_connections 2024-04-02T09:14:35.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.460+0000 7fafe6400700 1 --2- 172.21.15.89:0/3197640436 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fafe015d9d0 0x7fafe00d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.460+0000 7fafe6400700 1 --2- 172.21.15.89:0/3197640436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00d16a0 0x7fafe00d1e30 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.460+0000 7fafe6400700 1 -- 172.21.15.89:0/3197640436 >> 172.21.15.89:0/3197640436 conn(0x7fafe008aff0 msgr2=0x7fafe00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:35.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.460+0000 7fafe6400700 1 -- 172.21.15.89:0/3197640436 shutdown_connections 2024-04-02T09:14:35.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.460+0000 7fafe6400700 1 -- 172.21.15.89:0/3197640436 wait complete. 2024-04-02T09:14:35.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.463+0000 7fafe6400700 1 Processor -- start 2024-04-02T09:14:35.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.463+0000 7fafe6400700 1 -- start start 2024-04-02T09:14:35.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.464+0000 7fafe6400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00d16a0 0x7fafe01585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:35.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.464+0000 7fafe6400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fafe00d2370 0x7fafe015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:35.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.464+0000 7fafe6400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafe015d9d0 0x7fafe015e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.464+0000 7fafe6400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe015f530 con 0x7fafe00d16a0 2024-04-02T09:14:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.464+0000 7fafe6400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe01e7590 con 0x7fafe015d9d0 2024-04-02T09:14:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.464+0000 7fafe6400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe015e920 con 0x7fafe00d2370 2024-04-02T09:14:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.464+0000 7fafdf400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00d16a0 0x7fafe01585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.464+0000 7fafdf400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00d16a0 0x7fafe01585c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59718/0 (socket says 172.21.15.89:59718) 2024-04-02T09:14:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.464+0000 7fafdf400700 1 -- 172.21.15.89:0/3867378617 learned_addr learned my addr 172.21.15.89:0/3867378617 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.464+0000 7fafdea00700 1 --2- 172.21.15.89:0/3867378617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fafe00d2370 0x7fafe015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.464+0000 7fafdfe00700 1 --2- 172.21.15.89:0/3867378617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafe015d9d0 0x7fafe015e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.465+0000 7fafdf400700 1 -- 172.21.15.89:0/3867378617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fafe00d2370 msgr2=0x7fafe015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.465+0000 7fafdf400700 1 --2- 172.21.15.89:0/3867378617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fafe00d2370 0x7fafe015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.465+0000 7fafdf400700 1 -- 172.21.15.89:0/3867378617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafe015d9d0 msgr2=0x7fafe015e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.465+0000 7fafdf400700 1 --2- 172.21.15.89:0/3867378617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafe015d9d0 0x7fafe015e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.465+0000 7fafdf400700 1 -- 172.21.15.89:0/3867378617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc009500 con 0x7fafe00d16a0 2024-04-02T09:14:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.465+0000 7fafdf400700 1 --2- 172.21.15.89:0/3867378617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00d16a0 0x7fafe01585c0 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7fafd000d800 tx=0x7fafd000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:35.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.465+0000 7fafd7400700 1 -- 172.21.15.89:0/3867378617 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd00125f0 con 0x7fafe00d16a0 2024-04-02T09:14:35.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.465+0000 7fafd7400700 1 -- 172.21.15.89:0/3867378617 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd0012750 con 0x7fafe00d16a0 2024-04-02T09:14:35.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.465+0000 7fafe6400700 1 -- 172.21.15.89:0/3867378617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe015ec00 con 0x7fafe00d16a0 2024-04-02T09:14:35.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.465+0000 7fafd7400700 1 -- 172.21.15.89:0/3867378617 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0012b00 con 0x7fafe00d16a0 2024-04-02T09:14:35.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.465+0000 7fafe6400700 1 -- 172.21.15.89:0/3867378617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe015f1c0 con 0x7fafe00d16a0 2024-04-02T09:14:35.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.466+0000 7fafe6400700 1 -- 172.21.15.89:0/3867378617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe015f810 con 0x7fafe00d16a0 2024-04-02T09:14:35.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.467+0000 7fafd7400700 1 -- 172.21.15.89:0/3867378617 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fafd0012c60 con 0x7fafe00d16a0 2024-04-02T09:14:35.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.468+0000 7fafd7400700 1 --2- 172.21.15.89:0/3867378617 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fafc006aed0 0x7fafc006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:35.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.469+0000 7fafd7400700 1 -- 172.21.15.89:0/3867378617 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fafd0089c70 con 0x7fafe00d16a0 2024-04-02T09:14:35.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.469+0000 7fafdea00700 1 --2- 172.21.15.89:0/3867378617 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fafc006aed0 0x7fafc006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:35.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.469+0000 7fafdea00700 1 --2- 172.21.15.89:0/3867378617 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fafc006aed0 0x7fafc006d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fafe00d1ae0 tx=0x7fafcc008070 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.473+0000 7fafd7400700 1 -- 172.21.15.89:0/3867378617 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fafd0054930 con 0x7fafe00d16a0 2024-04-02T09:14:35.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:35.719+0000 7fafe6400700 1 -- 172.21.15.89:0/3867378617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fafe00d1ca0 con 0x7fafe00d16a0 2024-04-02T09:14:36.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.363+0000 7fafd7400700 1 -- 172.21.15.89:0/3867378617 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v134) v1 ==== 118+0+0 (secure 0 0 0) 0x7fafd0057f50 con 0x7fafe00d16a0 2024-04-02T09:14:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.366+0000 7fafe6400700 1 -- 172.21.15.89:0/3867378617 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fafc006aed0 msgr2=0x7fafc006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.366+0000 7fafe6400700 1 --2- 172.21.15.89:0/3867378617 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fafc006aed0 0x7fafc006d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fafe00d1ae0 tx=0x7fafcc008070 comp rx=0 tx=0).stop 2024-04-02T09:14:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.366+0000 7fafe6400700 1 -- 172.21.15.89:0/3867378617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00d16a0 msgr2=0x7fafe01585c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.366+0000 7fafe6400700 1 --2- 172.21.15.89:0/3867378617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00d16a0 0x7fafe01585c0 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7fafd000d800 tx=0x7fafd000db10 comp rx=0 tx=0).stop 2024-04-02T09:14:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.366+0000 7fafe6400700 1 -- 172.21.15.89:0/3867378617 shutdown_connections 2024-04-02T09:14:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.366+0000 7fafe6400700 1 --2- 172.21.15.89:0/3867378617 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fafc006aed0 0x7fafc006d390 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.366+0000 7fafe6400700 1 --2- 172.21.15.89:0/3867378617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafe015d9d0 0x7fafe015e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.366+0000 7fafe6400700 1 --2- 172.21.15.89:0/3867378617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00d16a0 0x7fafe01585c0 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.366+0000 7fafe6400700 1 --2- 172.21.15.89:0/3867378617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fafe00d2370 0x7fafe015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.366+0000 7fafe6400700 1 -- 172.21.15.89:0/3867378617 >> 172.21.15.89:0/3867378617 conn(0x7fafe008aff0 msgr2=0x7fafe00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.366+0000 7fafe6400700 1 -- 172.21.15.89:0/3867378617 shutdown_connections 2024-04-02T09:14:36.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.366+0000 7fafe6400700 1 -- 172.21.15.89:0/3867378617 wait complete. 2024-04-02T09:14:36.384 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T09:14:36.385 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T09:14:36.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.583+0000 7eff0e800700 1 Processor -- start 2024-04-02T09:14:36.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.583+0000 7eff0e800700 1 -- start start 2024-04-02T09:14:36.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.585+0000 7eff0e800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7eff08160e50 0x7eff080b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:36.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.585+0000 7eff0e800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7eff080b3f80 0x7eff080b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:36.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.585+0000 7eff0e800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff080cef00 con 0x7eff080d3f20 2024-04-02T09:14:36.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.585+0000 7eff0e800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff08161230 con 0x7eff080b3f80 2024-04-02T09:14:36.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.585+0000 7eff0e800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff080d2a80 con 0x7eff08160e50 2024-04-02T09:14:36.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.585+0000 7eff07e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7eff080b3f80 0x7eff080b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:36.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.585+0000 7eff07e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7eff080b3f80 0x7eff080b53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59732/0 (socket says 172.21.15.89:59732) 2024-04-02T09:14:36.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.585+0000 7eff07e00700 1 -- 172.21.15.89:0/3598625069 learned_addr learned my addr 172.21.15.89:0/3598625069 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:36.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.585+0000 7eff07400700 1 --2- 172.21.15.89:0/3598625069 >> v2:172.21.15.113:3300/0 conn(0x7eff08160e50 0x7eff080b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:36.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.585+0000 7eff07e00700 1 -- 172.21.15.89:0/3598625069 >> v2:172.21.15.113:3300/0 conn(0x7eff08160e50 msgr2=0x7eff080b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:36.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.585+0000 7eff07e00700 1 --2- 172.21.15.89:0/3598625069 >> v2:172.21.15.113:3300/0 conn(0x7eff08160e50 0x7eff080b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:36.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.585+0000 7eff07e00700 1 -- 172.21.15.89:0/3598625069 >> v1:172.21.15.89:6789/0 conn(0x7eff080d3f20 legacy=0x7eff080b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:36.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.585+0000 7eff07e00700 1 -- 172.21.15.89:0/3598625069 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff080d43b0 con 0x7eff080b3f80 2024-04-02T09:14:36.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.585+0000 7eff07400700 1 --2- 172.21.15.89:0/3598625069 >> v2:172.21.15.113:3300/0 conn(0x7eff08160e50 0x7eff080b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:36.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.586+0000 7eff07e00700 1 --2- 172.21.15.89:0/3598625069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff080b3f80 0x7eff080b53d0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7eff080cff10 tx=0x7eff0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2502362d545d5944 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:36.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.586+0000 7eff06000700 1 -- 172.21.15.89:0/3598625069 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0000e070 con 0x7eff080b3f80 2024-04-02T09:14:36.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.586+0000 7eff06000700 1 -- 172.21.15.89:0/3598625069 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0000b9c0 con 0x7eff080b3f80 2024-04-02T09:14:36.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.586+0000 7eff06000700 1 -- 172.21.15.89:0/3598625069 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0000bd70 con 0x7eff080b3f80 2024-04-02T09:14:36.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.586+0000 7eff0e800700 1 -- 172.21.15.89:0/3598625069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff080b3f80 msgr2=0x7eff080b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:36.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.586+0000 7eff0e800700 1 --2- 172.21.15.89:0/3598625069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff080b3f80 0x7eff080b53d0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7eff080cff10 tx=0x7eff0000b330 comp rx=0 tx=0).stop 2024-04-02T09:14:36.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.586+0000 7eff0e800700 1 -- 172.21.15.89:0/3598625069 shutdown_connections 2024-04-02T09:14:36.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.586+0000 7eff0e800700 1 --2- 172.21.15.89:0/3598625069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff080b3f80 0x7eff080b53d0 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:36.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.586+0000 7eff0e800700 1 --2- 172.21.15.89:0/3598625069 >> v2:172.21.15.113:3300/0 conn(0x7eff08160e50 0x7eff080b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:36.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.586+0000 7eff0e800700 1 -- 172.21.15.89:0/3598625069 >> 172.21.15.89:0/3598625069 conn(0x7eff0808aff0 msgr2=0x7eff080d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:36.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.587+0000 7eff0e800700 1 -- 172.21.15.89:0/3598625069 shutdown_connections 2024-04-02T09:14:36.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.587+0000 7eff0e800700 1 -- 172.21.15.89:0/3598625069 wait complete. 2024-04-02T09:14:36.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.588+0000 7eff0e800700 1 Processor -- start 2024-04-02T09:14:36.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.588+0000 7eff0e800700 1 -- start start 2024-04-02T09:14:36.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.589+0000 7eff0e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff080b3f80 0x7eff0814e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:36.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.589+0000 7eff0e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff08160e50 0x7eff080b6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:36.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.589+0000 7eff0e800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff080cca70 0x7eff080b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:36.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.589+0000 7eff0e800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff080b7f30 con 0x7eff080b3f80 2024-04-02T09:14:36.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.589+0000 7eff0e800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0814eec0 con 0x7eff080cca70 2024-04-02T09:14:36.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.589+0000 7eff0e800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff080b72b0 con 0x7eff08160e50 2024-04-02T09:14:36.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.589+0000 7eff07400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff080b3f80 0x7eff0814e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:36.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7eff07400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff080b3f80 0x7eff0814e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59734/0 (socket says 172.21.15.89:59734) 2024-04-02T09:14:36.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7eff07400700 1 -- 172.21.15.89:0/975070784 learned_addr learned my addr 172.21.15.89:0/975070784 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:36.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7eff07e00700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff080cca70 0x7eff080b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:36.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7eff06a00700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff08160e50 0x7eff080b6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:36.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7eff07400700 1 -- 172.21.15.89:0/975070784 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff08160e50 msgr2=0x7eff080b6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:36.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7eff07400700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff08160e50 0x7eff080b6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:36.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7eff07400700 1 -- 172.21.15.89:0/975070784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff080cca70 msgr2=0x7eff080b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:36.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7eff07400700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff080cca70 0x7eff080b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:36.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7eff07400700 1 -- 172.21.15.89:0/975070784 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff00009b30 con 0x7eff080b3f80 2024-04-02T09:14:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7eff07e00700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff080cca70 0x7eff080b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7eff06a00700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff08160e50 0x7eff080b6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7eff07400700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff080b3f80 0x7eff0814e980 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7efef800c750 tx=0x7efef800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7efeff400700 1 -- 172.21.15.89:0/975070784 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef8011800 con 0x7eff080b3f80 2024-04-02T09:14:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7efeff400700 1 -- 172.21.15.89:0/975070784 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef8011960 con 0x7eff080b3f80 2024-04-02T09:14:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7efeff400700 1 -- 172.21.15.89:0/975070784 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef8011cd0 con 0x7eff080b3f80 2024-04-02T09:14:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.590+0000 7eff0e800700 1 -- 172.21.15.89:0/975070784 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff080b7590 con 0x7eff080b3f80 2024-04-02T09:14:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.591+0000 7eff0e800700 1 -- 172.21.15.89:0/975070784 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff080b7b50 con 0x7eff080b3f80 2024-04-02T09:14:36.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.592+0000 7eff0e800700 1 -- 172.21.15.89:0/975070784 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff08067720 con 0x7eff080b3f80 2024-04-02T09:14:36.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.593+0000 7efeff400700 1 -- 172.21.15.89:0/975070784 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7efef8021350 con 0x7eff080b3f80 2024-04-02T09:14:36.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.594+0000 7efeff400700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efee806aed0 0x7efee806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:36.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.594+0000 7efeff400700 1 -- 172.21.15.89:0/975070784 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7efef8089a30 con 0x7eff080b3f80 2024-04-02T09:14:36.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.594+0000 7eff06a00700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efee806aed0 0x7efee806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:36.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.594+0000 7eff06a00700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efee806aed0 0x7efee806d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efef4000f60 tx=0x7efef4009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:36.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.598+0000 7efeff400700 1 -- 172.21.15.89:0/975070784 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efef80545e0 con 0x7eff080b3f80 2024-04-02T09:14:36.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:36.848+0000 7eff0e800700 1 -- 172.21.15.89:0/975070784 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7eff081f32c0 con 0x7eff080b3f80 2024-04-02T09:14:37.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.365+0000 7efeff400700 1 -- 172.21.15.89:0/975070784 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v135) v1 ==== 122+0+0 (secure 0 0 0) 0x7efef801e0c0 con 0x7eff080b3f80 2024-04-02T09:14:37.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.368+0000 7eff0e800700 1 -- 172.21.15.89:0/975070784 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efee806aed0 msgr2=0x7efee806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:37.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.368+0000 7eff0e800700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efee806aed0 0x7efee806d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efef4000f60 tx=0x7efef4009040 comp rx=0 tx=0).stop 2024-04-02T09:14:37.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.368+0000 7eff0e800700 1 -- 172.21.15.89:0/975070784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff080b3f80 msgr2=0x7eff0814e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:37.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.368+0000 7eff0e800700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff080b3f80 0x7eff0814e980 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7efef800c750 tx=0x7efef800cb10 comp rx=0 tx=0).stop 2024-04-02T09:14:37.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.368+0000 7eff0e800700 1 -- 172.21.15.89:0/975070784 shutdown_connections 2024-04-02T09:14:37.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.368+0000 7eff0e800700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efee806aed0 0x7efee806d390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:37.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.368+0000 7eff0e800700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff080cca70 0x7eff080b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:37.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.368+0000 7eff0e800700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff080b3f80 0x7eff0814e980 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:37.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.368+0000 7eff0e800700 1 --2- 172.21.15.89:0/975070784 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7eff08160e50 0x7eff080b6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:37.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.368+0000 7eff0e800700 1 -- 172.21.15.89:0/975070784 >> 172.21.15.89:0/975070784 conn(0x7eff0808aff0 msgr2=0x7eff0806b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:37.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.368+0000 7eff0e800700 1 -- 172.21.15.89:0/975070784 shutdown_connections 2024-04-02T09:14:37.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.368+0000 7eff0e800700 1 -- 172.21.15.89:0/975070784 wait complete. 2024-04-02T09:14:37.386 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T09:14:37.387 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:14:37.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.580+0000 7fdb8f600700 1 Processor -- start 2024-04-02T09:14:37.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.581+0000 7fdb8f600700 1 -- start start 2024-04-02T09:14:37.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.582+0000 7fdb8f600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdb8815d640 0x7fdb880d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:37.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.582+0000 7fdb8f600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdb881619d0 0x7fdb880b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:37.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.582+0000 7fdb8f600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb880d30d0 con 0x7fdb880ceec0 2024-04-02T09:14:37.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.582+0000 7fdb8f600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb880b6840 con 0x7fdb881619d0 2024-04-02T09:14:37.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.582+0000 7fdb8f600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb880d2880 con 0x7fdb8815d640 2024-04-02T09:14:37.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.582+0000 7fdb8d600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdb881619d0 0x7fdb880b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:37.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.582+0000 7fdb8d600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdb881619d0 0x7fdb880b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59750/0 (socket says 172.21.15.89:59750) 2024-04-02T09:14:37.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.582+0000 7fdb8d600700 1 -- 172.21.15.89:0/2307942774 learned_addr learned my addr 172.21.15.89:0/2307942774 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:37.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.582+0000 7fdb8cc00700 1 --2- 172.21.15.89:0/2307942774 >> v2:172.21.15.113:3300/0 conn(0x7fdb8815d640 0x7fdb880d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:37.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.582+0000 7fdb8d600700 1 -- 172.21.15.89:0/2307942774 >> v2:172.21.15.113:3300/0 conn(0x7fdb8815d640 msgr2=0x7fdb880d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:37.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.582+0000 7fdb8d600700 1 --2- 172.21.15.89:0/2307942774 >> v2:172.21.15.113:3300/0 conn(0x7fdb8815d640 0x7fdb880d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:37.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.582+0000 7fdb8d600700 1 -- 172.21.15.89:0/2307942774 >> v1:172.21.15.89:6789/0 conn(0x7fdb880ceec0 legacy=0x7fdb881612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:37.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.582+0000 7fdb8d600700 1 -- 172.21.15.89:0/2307942774 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb880d2b30 con 0x7fdb881619d0 2024-04-02T09:14:37.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.582+0000 7fdb8cc00700 1 --2- 172.21.15.89:0/2307942774 >> v2:172.21.15.113:3300/0 conn(0x7fdb8815d640 0x7fdb880d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:37.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.583+0000 7fdb8d600700 1 --2- 172.21.15.89:0/2307942774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb881619d0 0x7fdb880b6cd0 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7fdb880b5c20 tx=0x7fdb8000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18791c600c87ae81 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:37.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.583+0000 7fdb87400700 1 -- 172.21.15.89:0/2307942774 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8000e070 con 0x7fdb881619d0 2024-04-02T09:14:37.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.583+0000 7fdb87400700 1 -- 172.21.15.89:0/2307942774 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8000b9c0 con 0x7fdb881619d0 2024-04-02T09:14:37.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.583+0000 7fdb87400700 1 -- 172.21.15.89:0/2307942774 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8000bd70 con 0x7fdb881619d0 2024-04-02T09:14:37.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.583+0000 7fdb8f600700 1 -- 172.21.15.89:0/2307942774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb881619d0 msgr2=0x7fdb880b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:37.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.583+0000 7fdb8f600700 1 --2- 172.21.15.89:0/2307942774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb881619d0 0x7fdb880b6cd0 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7fdb880b5c20 tx=0x7fdb8000b330 comp rx=0 tx=0).stop 2024-04-02T09:14:37.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.584+0000 7fdb8f600700 1 -- 172.21.15.89:0/2307942774 shutdown_connections 2024-04-02T09:14:37.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.584+0000 7fdb8f600700 1 --2- 172.21.15.89:0/2307942774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb881619d0 0x7fdb880b6cd0 unknown :-1 s=CLOSED pgs=2808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:37.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.584+0000 7fdb8f600700 1 --2- 172.21.15.89:0/2307942774 >> v2:172.21.15.113:3300/0 conn(0x7fdb8815d640 0x7fdb880d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:37.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.584+0000 7fdb8f600700 1 -- 172.21.15.89:0/2307942774 >> 172.21.15.89:0/2307942774 conn(0x7fdb8808aff0 msgr2=0x7fdb880d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:37.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.584+0000 7fdb8f600700 1 -- 172.21.15.89:0/2307942774 shutdown_connections 2024-04-02T09:14:37.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.584+0000 7fdb8f600700 1 -- 172.21.15.89:0/2307942774 wait complete. 2024-04-02T09:14:37.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.585+0000 7fdb8f600700 1 Processor -- start 2024-04-02T09:14:37.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.585+0000 7fdb8f600700 1 -- start start 2024-04-02T09:14:37.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.586+0000 7fdb8f600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb880ceec0 0x7fdb880c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:37.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.586+0000 7fdb8f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb8815d640 0x7fdb881f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:37.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.586+0000 7fdb8f600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb881619d0 0x7fdb88156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:37.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.586+0000 7fdb8f600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8815e730 con 0x7fdb880ceec0 2024-04-02T09:14:37.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.586+0000 7fdb8f600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb881f2ee0 con 0x7fdb881619d0 2024-04-02T09:14:37.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.586+0000 7fdb8f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8815e400 con 0x7fdb8815d640 2024-04-02T09:14:37.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.587+0000 7fdb8cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb880ceec0 0x7fdb880c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:37.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.587+0000 7fdb8cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb880ceec0 0x7fdb880c99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59766/0 (socket says 172.21.15.89:59766) 2024-04-02T09:14:37.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.587+0000 7fdb8cc00700 1 -- 172.21.15.89:0/191454162 learned_addr learned my addr 172.21.15.89:0/191454162 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:37.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.587+0000 7fdb8cc00700 1 -- 172.21.15.89:0/191454162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb8815d640 msgr2=0x7fdb881f29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:37.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.587+0000 7fdb8d600700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb881619d0 0x7fdb88156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:37.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.587+0000 7fdb87e00700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb8815d640 0x7fdb881f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:37.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.587+0000 7fdb8cc00700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb8815d640 0x7fdb881f29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:37.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.587+0000 7fdb8cc00700 1 -- 172.21.15.89:0/191454162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb881619d0 msgr2=0x7fdb88156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:37.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.587+0000 7fdb8cc00700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb881619d0 0x7fdb88156e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:37.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.587+0000 7fdb8cc00700 1 -- 172.21.15.89:0/191454162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb80009b30 con 0x7fdb880ceec0 2024-04-02T09:14:37.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.587+0000 7fdb87e00700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb8815d640 0x7fdb881f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:37.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.587+0000 7fdb8d600700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb881619d0 0x7fdb88156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:37.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.587+0000 7fdb8cc00700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb880ceec0 0x7fdb880c99d0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00c7f0 tx=0x7fdb7c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:37.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.587+0000 7fdb84c00700 1 -- 172.21.15.89:0/191454162 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c011800 con 0x7fdb880ceec0 2024-04-02T09:14:37.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.588+0000 7fdb84c00700 1 -- 172.21.15.89:0/191454162 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7c011960 con 0x7fdb880ceec0 2024-04-02T09:14:37.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.588+0000 7fdb84c00700 1 -- 172.21.15.89:0/191454162 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c011d10 con 0x7fdb880ceec0 2024-04-02T09:14:37.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.588+0000 7fdb8f600700 1 -- 172.21.15.89:0/191454162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb88157690 con 0x7fdb880ceec0 2024-04-02T09:14:37.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.588+0000 7fdb8f600700 1 -- 172.21.15.89:0/191454162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb88157c50 con 0x7fdb880ceec0 2024-04-02T09:14:37.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.589+0000 7fdb8f600700 1 -- 172.21.15.89:0/191454162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb88067720 con 0x7fdb880ceec0 2024-04-02T09:14:37.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.590+0000 7fdb84c00700 1 -- 172.21.15.89:0/191454162 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fdb7c018360 con 0x7fdb880ceec0 2024-04-02T09:14:37.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.590+0000 7fdb84c00700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb6806aed0 0x7fdb6806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:37.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.590+0000 7fdb84c00700 1 -- 172.21.15.89:0/191454162 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fdb7c089a30 con 0x7fdb880ceec0 2024-04-02T09:14:37.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.591+0000 7fdb87e00700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb6806aed0 0x7fdb6806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:37.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.591+0000 7fdb87e00700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb6806aed0 0x7fdb6806d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdb880b52b0 tx=0x7fdb74009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:37.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.595+0000 7fdb84c00700 1 -- 172.21.15.89:0/191454162 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdb7c0545e0 con 0x7fdb880ceec0 2024-04-02T09:14:37.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.839+0000 7fdb8f600700 1 -- 172.21.15.89:0/191454162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb8815bea0 con 0x7fdb880ceec0 2024-04-02T09:14:37.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.839+0000 7fdb84c00700 1 -- 172.21.15.89:0/191454162 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+4756 (secure 0 0 0) 0x7fdb7c01f090 con 0x7fdb880ceec0 2024-04-02T09:14:37.838 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:37.838 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":135,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10031,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1597421311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1597421311},{"type":"v1","addr":"172.21.15.113:6834","nonce":1597421311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":10034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3719211992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":3719211992},{"type":"v1","addr":"172.21.15.113:6835","nonce":3719211992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":10037,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3947526387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3947526387},{"type":"v1","addr":"172.21.15.89:6835","nonce":3947526387}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:14:34.351811+0000","modified":"2024-04-02T09:14:37.361983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12762":{"gid":12762,"name":"c","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2124958719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2124958719},{"type":"v1","addr":"172.21.15.89:6837","nonce":2124958719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-04-02T09:14:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.842+0000 7fdb8f600700 1 -- 172.21.15.89:0/191454162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb6806aed0 msgr2=0x7fdb6806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.842+0000 7fdb8f600700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb6806aed0 0x7fdb6806d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdb880b52b0 tx=0x7fdb74009040 comp rx=0 tx=0).stop 2024-04-02T09:14:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.842+0000 7fdb8f600700 1 -- 172.21.15.89:0/191454162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb880ceec0 msgr2=0x7fdb880c99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.842+0000 7fdb8f600700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb880ceec0 0x7fdb880c99d0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00c7f0 tx=0x7fdb7c00cbb0 comp rx=0 tx=0).stop 2024-04-02T09:14:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.842+0000 7fdb8f600700 1 -- 172.21.15.89:0/191454162 shutdown_connections 2024-04-02T09:14:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.842+0000 7fdb8f600700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fdb6806aed0 0x7fdb6806d390 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.842+0000 7fdb8f600700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb881619d0 0x7fdb88156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.842+0000 7fdb8f600700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb880ceec0 0x7fdb880c99d0 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.842+0000 7fdb8f600700 1 --2- 172.21.15.89:0/191454162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb8815d640 0x7fdb881f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.842+0000 7fdb8f600700 1 -- 172.21.15.89:0/191454162 >> 172.21.15.89:0/191454162 conn(0x7fdb8808aff0 msgr2=0x7fdb8815b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.842+0000 7fdb8f600700 1 -- 172.21.15.89:0/191454162 shutdown_connections 2024-04-02T09:14:37.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:37.842+0000 7fdb8f600700 1 -- 172.21.15.89:0/191454162 wait complete. 2024-04-02T09:14:37.842 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 135 2024-04-02T09:14:37.862 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:14:38.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.055+0000 7fcbd6e00700 1 Processor -- start 2024-04-02T09:14:38.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.055+0000 7fcbd6e00700 1 -- start start 2024-04-02T09:14:38.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.056+0000 7fcbd6e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcbd0072580 0x7fcbd00b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.057+0000 7fcbd6e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd00b2180 con 0x7fcbd00b2e50 2024-04-02T09:14:38.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.057+0000 7fcbd6e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd015b100 con 0x7fcbd00b3af0 2024-04-02T09:14:38.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.057+0000 7fcbd6e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd015b3e0 con 0x7fcbd0072580 2024-04-02T09:14:38.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.057+0000 7fcbd4e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcbd00b2e50 0x7fcbd00709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50278/0 (socket says 172.21.15.89:50278) 2024-04-02T09:14:38.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.057+0000 7fcbd4e00700 1 -- 172.21.15.89:0/315723930 learned_addr learned my addr 172.21.15.89:0/315723930 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:38.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.057+0000 7fcbcf400700 1 --2- 172.21.15.89:0/315723930 >> v2:172.21.15.113:3300/0 conn(0x7fcbd0072580 0x7fcbd00b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.057+0000 7fcbcea00700 1 -- 172.21.15.89:0/315723930 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649122377 0 0) 0x7fcbd00b2180 con 0x7fcbd00b2e50 2024-04-02T09:14:38.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.057+0000 7fcbcea00700 1 -- 172.21.15.89:0/315723930 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb40035b0 con 0x7fcbd00b2e50 2024-04-02T09:14:38.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.057+0000 7fcbcea00700 1 -- 172.21.15.89:0/315723930 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618048084 0 0) 0x7fcbd015b100 con 0x7fcbd00b3af0 2024-04-02T09:14:38.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.057+0000 7fcbcea00700 1 -- 172.21.15.89:0/315723930 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd00b2180 con 0x7fcbd00b3af0 2024-04-02T09:14:38.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.057+0000 7fcbcf400700 1 -- 172.21.15.89:0/315723930 >> v1:172.21.15.113:6789/0 conn(0x7fcbd00b3af0 legacy=0x7fcbd015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.057+0000 7fcbcf400700 1 -- 172.21.15.89:0/315723930 >> v1:172.21.15.89:6789/0 conn(0x7fcbd00b2e50 legacy=0x7fcbd00709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.057+0000 7fcbcea00700 1 -- 172.21.15.89:0/315723930 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 81534854 0 0) 0x7fcbb40035b0 con 0x7fcbd00b2e50 2024-04-02T09:14:38.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.058+0000 7fcbcf400700 1 -- 172.21.15.89:0/315723930 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd00b23f0 con 0x7fcbd0072580 2024-04-02T09:14:38.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.058+0000 7fcbcf400700 1 --2- 172.21.15.89:0/315723930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbd0072580 0x7fcbd00b2910 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009840 tx=0x7fcbbc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7fe24971c329228 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:38.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.058+0000 7fcbcea00700 1 -- 172.21.15.89:0/315723930 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc011830 con 0x7fcbd0072580 2024-04-02T09:14:38.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.058+0000 7fcbcea00700 1 -- 172.21.15.89:0/315723930 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbbc011990 con 0x7fcbd0072580 2024-04-02T09:14:38.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.058+0000 7fcbcea00700 1 -- 172.21.15.89:0/315723930 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc011d20 con 0x7fcbd0072580 2024-04-02T09:14:38.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.059+0000 7fcbd6e00700 1 -- 172.21.15.89:0/315723930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbd0072580 msgr2=0x7fcbd00b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.059+0000 7fcbd6e00700 1 --2- 172.21.15.89:0/315723930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbd0072580 0x7fcbd00b2910 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009840 tx=0x7fcbbc009b50 comp rx=0 tx=0).stop 2024-04-02T09:14:38.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.059+0000 7fcbd6e00700 1 -- 172.21.15.89:0/315723930 shutdown_connections 2024-04-02T09:14:38.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.059+0000 7fcbd6e00700 1 --2- 172.21.15.89:0/315723930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbd0072580 0x7fcbd00b2910 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.059+0000 7fcbd6e00700 1 -- 172.21.15.89:0/315723930 >> 172.21.15.89:0/315723930 conn(0x7fcbd00cf190 msgr2=0x7fcbd00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:38.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.059+0000 7fcbd6e00700 1 -- 172.21.15.89:0/315723930 shutdown_connections 2024-04-02T09:14:38.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.059+0000 7fcbd6e00700 1 -- 172.21.15.89:0/315723930 wait complete. 2024-04-02T09:14:38.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.061+0000 7fcbd6e00700 1 Processor -- start 2024-04-02T09:14:38.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.061+0000 7fcbd6e00700 1 -- start start 2024-04-02T09:14:38.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.062+0000 7fcbd6e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbd0072580 0x7fcbd01583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.062+0000 7fcbd6e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbd00b2e50 0x7fcbd00cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbd6e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd00b3af0 0x7fcbd00cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbd6e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd0150c00 con 0x7fcbd00b3af0 2024-04-02T09:14:38.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbd6e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd01f49a0 con 0x7fcbd00b2e50 2024-04-02T09:14:38.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbd6e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd01f4c80 con 0x7fcbd0072580 2024-04-02T09:14:38.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbd4e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd00b3af0 0x7fcbd00cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbd4e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd00b3af0 0x7fcbd00cd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59768/0 (socket says 172.21.15.89:59768) 2024-04-02T09:14:38.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbd4e00700 1 -- 172.21.15.89:0/63271162 learned_addr learned my addr 172.21.15.89:0/63271162 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:38.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbcf400700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbd00b2e50 0x7fcbd00cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbcfe00700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbd0072580 0x7fcbd01583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbd4e00700 1 -- 172.21.15.89:0/63271162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbd0072580 msgr2=0x7fcbd01583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbd4e00700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbd0072580 0x7fcbd01583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbd4e00700 1 -- 172.21.15.89:0/63271162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbd00b2e50 msgr2=0x7fcbd00cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbd4e00700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbd00b2e50 0x7fcbd00cc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbd4e00700 1 -- 172.21.15.89:0/63271162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc0094a0 con 0x7fcbd00b3af0 2024-04-02T09:14:38.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.063+0000 7fcbcfe00700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbd0072580 0x7fcbd01583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:38.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.064+0000 7fcbd4e00700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd00b3af0 0x7fcbd00cd180 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800cb70 tx=0x7fcbc800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:38.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.064+0000 7fcbc7e00700 1 -- 172.21.15.89:0/63271162 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc800ab50 con 0x7fcbd00b3af0 2024-04-02T09:14:38.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.064+0000 7fcbc7e00700 1 -- 172.21.15.89:0/63271162 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc800acb0 con 0x7fcbd00b3af0 2024-04-02T09:14:38.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.064+0000 7fcbd6e00700 1 -- 172.21.15.89:0/63271162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd01f4f60 con 0x7fcbd00b3af0 2024-04-02T09:14:38.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.064+0000 7fcbc7e00700 1 -- 172.21.15.89:0/63271162 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8012640 con 0x7fcbd00b3af0 2024-04-02T09:14:38.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.064+0000 7fcbd6e00700 1 -- 172.21.15.89:0/63271162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd01f5520 con 0x7fcbd00b3af0 2024-04-02T09:14:38.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.065+0000 7fcbd6e00700 1 -- 172.21.15.89:0/63271162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd015eb10 con 0x7fcbd00b3af0 2024-04-02T09:14:38.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.066+0000 7fcbc7e00700 1 -- 172.21.15.89:0/63271162 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fcbc80127a0 con 0x7fcbd00b3af0 2024-04-02T09:14:38.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.067+0000 7fcbc7e00700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcbb006aed0 0x7fcbb006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.067+0000 7fcbc7e00700 1 -- 172.21.15.89:0/63271162 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fcbc8089d60 con 0x7fcbd00b3af0 2024-04-02T09:14:38.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.067+0000 7fcbcfe00700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcbb006aed0 0x7fcbb006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.067+0000 7fcbcfe00700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcbb006aed0 0x7fcbb006d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00011e0 tx=0x7fcbc0009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:38.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.071+0000 7fcbc7e00700 1 -- 172.21.15.89:0/63271162 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcbc80549c0 con 0x7fcbd00b3af0 2024-04-02T09:14:38.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.272+0000 7fcbd6e00700 1 -- 172.21.15.89:0/63271162 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbd014fdf0 con 0x7fcbd00b3af0 2024-04-02T09:14:38.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.274+0000 7fcbc7e00700 1 -- 172.21.15.89:0/63271162 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v265) v1 ==== 74+0+16744 (secure 0 0 0) 0x7fcbc8057fe0 con 0x7fcbd00b3af0 2024-04-02T09:14:38.272 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:38.272 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":265,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:14:35.354254+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:14:32.705979+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-04-02T09:14:33.330752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:14:38.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.276+0000 7fcbd6e00700 1 -- 172.21.15.89:0/63271162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcbb006aed0 msgr2=0x7fcbb006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.276+0000 7fcbd6e00700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcbb006aed0 0x7fcbb006d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00011e0 tx=0x7fcbc0009040 comp rx=0 tx=0).stop 2024-04-02T09:14:38.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.276+0000 7fcbd6e00700 1 -- 172.21.15.89:0/63271162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd00b3af0 msgr2=0x7fcbd00cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.276+0000 7fcbd6e00700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd00b3af0 0x7fcbd00cd180 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800cb70 tx=0x7fcbc800cf30 comp rx=0 tx=0).stop 2024-04-02T09:14:38.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.276+0000 7fcbd6e00700 1 -- 172.21.15.89:0/63271162 shutdown_connections 2024-04-02T09:14:38.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.276+0000 7fcbd6e00700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd00b3af0 0x7fcbd00cd180 unknown :-1 s=CLOSED pgs=2811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.276+0000 7fcbd6e00700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcbd0072580 0x7fcbd01583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.276+0000 7fcbd6e00700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcbb006aed0 0x7fcbb006d390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.276+0000 7fcbd6e00700 1 --2- 172.21.15.89:0/63271162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcbd00b2e50 0x7fcbd00cc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.276+0000 7fcbd6e00700 1 -- 172.21.15.89:0/63271162 >> 172.21.15.89:0/63271162 conn(0x7fcbd00cf190 msgr2=0x7fcbd00aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:38.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.276+0000 7fcbd6e00700 1 -- 172.21.15.89:0/63271162 shutdown_connections 2024-04-02T09:14:38.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.276+0000 7fcbd6e00700 1 -- 172.21.15.89:0/63271162 wait complete. 2024-04-02T09:14:38.293 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T09:14:38.293 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:14:38.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.488+0000 7fd6a7000700 1 Processor -- start 2024-04-02T09:14:38.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.489+0000 7fd6a7000700 1 -- start start 2024-04-02T09:14:38.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.490+0000 7fd6a7000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6a01619d0 0x7fd6a01685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.490+0000 7fd6a7000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a00d30d0 con 0x7fd6a015d640 2024-04-02T09:14:38.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.490+0000 7fd6a7000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a00b6840 con 0x7fd6a00ceec0 2024-04-02T09:14:38.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.490+0000 7fd6a7000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a00d2b30 con 0x7fd6a01619d0 2024-04-02T09:14:38.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.490+0000 7fd6a5000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6a01619d0 0x7fd6a01685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.490+0000 7fd6a5000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6a01619d0 0x7fd6a01685f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59774/0 (socket says 172.21.15.89:59774) 2024-04-02T09:14:38.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.490+0000 7fd6a5000700 1 -- 172.21.15.89:0/3428104238 learned_addr learned my addr 172.21.15.89:0/3428104238 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.490+0000 7fd6a5000700 1 -- 172.21.15.89:0/3428104238 >> v1:172.21.15.113:6789/0 conn(0x7fd6a00ceec0 legacy=0x7fd6a01612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.490+0000 7fd6a5000700 1 -- 172.21.15.89:0/3428104238 >> v1:172.21.15.89:6789/0 conn(0x7fd6a015d640 legacy=0x7fd6a00d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.490+0000 7fd6a5000700 1 -- 172.21.15.89:0/3428104238 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a00d2df0 con 0x7fd6a01619d0 2024-04-02T09:14:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.491+0000 7fd6a5000700 1 --2- 172.21.15.89:0/3428104238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6a01619d0 0x7fd6a01685f0 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7fd698009e50 tx=0x7fd69800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476b37ab027be849 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.491+0000 7fd69ea00700 1 -- 172.21.15.89:0/3428104238 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69800e070 con 0x7fd6a01619d0 2024-04-02T09:14:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.491+0000 7fd69ea00700 1 -- 172.21.15.89:0/3428104238 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd698003a20 con 0x7fd6a01619d0 2024-04-02T09:14:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.491+0000 7fd69ea00700 1 -- 172.21.15.89:0/3428104238 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd698003dd0 con 0x7fd6a01619d0 2024-04-02T09:14:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.491+0000 7fd6a7000700 1 -- 172.21.15.89:0/3428104238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6a01619d0 msgr2=0x7fd6a01685f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.491+0000 7fd6a7000700 1 --2- 172.21.15.89:0/3428104238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6a01619d0 0x7fd6a01685f0 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7fd698009e50 tx=0x7fd69800b330 comp rx=0 tx=0).stop 2024-04-02T09:14:38.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.491+0000 7fd6a7000700 1 -- 172.21.15.89:0/3428104238 shutdown_connections 2024-04-02T09:14:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.491+0000 7fd6a7000700 1 --2- 172.21.15.89:0/3428104238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6a01619d0 0x7fd6a01685f0 unknown :-1 s=CLOSED pgs=2813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.491+0000 7fd6a7000700 1 -- 172.21.15.89:0/3428104238 >> 172.21.15.89:0/3428104238 conn(0x7fd6a008aff0 msgr2=0x7fd6a00d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.492+0000 7fd6a7000700 1 -- 172.21.15.89:0/3428104238 shutdown_connections 2024-04-02T09:14:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.492+0000 7fd6a7000700 1 -- 172.21.15.89:0/3428104238 wait complete. 2024-04-02T09:14:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.493+0000 7fd6a7000700 1 Processor -- start 2024-04-02T09:14:38.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.493+0000 7fd6a7000700 1 -- start start 2024-04-02T09:14:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.494+0000 7fd6a7000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6a00ceec0 0x7fd6a015c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.494+0000 7fd6a7000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6a015d640 0x7fd6a00cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.494+0000 7fd6a7000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6a01619d0 0x7fd6a01f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.494+0000 7fd6a7000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a01f6460 con 0x7fd6a015d640 2024-04-02T09:14:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.494+0000 7fd6a7000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a00cb570 con 0x7fd6a01619d0 2024-04-02T09:14:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.494+0000 7fd6a7000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a00cb810 con 0x7fd6a00ceec0 2024-04-02T09:14:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.495+0000 7fd69f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6a015d640 0x7fd6a00cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.495+0000 7fd69f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6a015d640 0x7fd6a00cb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59788/0 (socket says 172.21.15.89:59788) 2024-04-02T09:14:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.495+0000 7fd69f400700 1 -- 172.21.15.89:0/963577847 learned_addr learned my addr 172.21.15.89:0/963577847 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.495+0000 7fd69fe00700 1 --2- 172.21.15.89:0/963577847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6a00ceec0 0x7fd6a015c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.495+0000 7fd6a5000700 1 --2- 172.21.15.89:0/963577847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6a01619d0 0x7fd6a01f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.495+0000 7fd69f400700 1 -- 172.21.15.89:0/963577847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6a00ceec0 msgr2=0x7fd6a015c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.495+0000 7fd69f400700 1 --2- 172.21.15.89:0/963577847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6a00ceec0 0x7fd6a015c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.495+0000 7fd69f400700 1 -- 172.21.15.89:0/963577847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6a01619d0 msgr2=0x7fd6a01f30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.495+0000 7fd69f400700 1 --2- 172.21.15.89:0/963577847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6a01619d0 0x7fd6a01f30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.495+0000 7fd69f400700 1 -- 172.21.15.89:0/963577847 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd698009b30 con 0x7fd6a015d640 2024-04-02T09:14:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.495+0000 7fd69f400700 1 --2- 172.21.15.89:0/963577847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6a015d640 0x7fd6a00cb030 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00c4d0 tx=0x7fd68c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.496+0000 7fd697e00700 1 -- 172.21.15.89:0/963577847 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c011790 con 0x7fd6a015d640 2024-04-02T09:14:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.496+0000 7fd697e00700 1 -- 172.21.15.89:0/963577847 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68c0118f0 con 0x7fd6a015d640 2024-04-02T09:14:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.496+0000 7fd697e00700 1 -- 172.21.15.89:0/963577847 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c011ca0 con 0x7fd6a015d640 2024-04-02T09:14:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.496+0000 7fd6a7000700 1 -- 172.21.15.89:0/963577847 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a00cbaf0 con 0x7fd6a015d640 2024-04-02T09:14:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.496+0000 7fd6a7000700 1 -- 172.21.15.89:0/963577847 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a00cc0b0 con 0x7fd6a015d640 2024-04-02T09:14:38.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.497+0000 7fd6a7000700 1 -- 172.21.15.89:0/963577847 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a0067720 con 0x7fd6a015d640 2024-04-02T09:14:38.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.498+0000 7fd697e00700 1 -- 172.21.15.89:0/963577847 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fd68c01c460 con 0x7fd6a015d640 2024-04-02T09:14:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.498+0000 7fd697e00700 1 --2- 172.21.15.89:0/963577847 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd68006aed0 0x7fd68006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.499+0000 7fd697e00700 1 -- 172.21.15.89:0/963577847 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fd68c0894f0 con 0x7fd6a015d640 2024-04-02T09:14:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.499+0000 7fd69fe00700 1 --2- 172.21.15.89:0/963577847 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd68006aed0 0x7fd68006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.499+0000 7fd69fe00700 1 --2- 172.21.15.89:0/963577847 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd68006aed0 0x7fd68006d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd6900018a0 tx=0x7fd690009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:38.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.503+0000 7fd697e00700 1 -- 172.21.15.89:0/963577847 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd68c054270 con 0x7fd6a015d640 2024-04-02T09:14:38.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.701+0000 7fd6a7000700 1 -- 172.21.15.89:0/963577847 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd6a00b7ce0 con 0x7fd68006aed0 2024-04-02T09:14:38.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.711+0000 7fd697e00700 1 -- 172.21.15.89:0/963577847 <== mgr.11320 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+274194 (secure 0 0 0) 0x7fd690001540 con 0x7fd68006aed0 2024-04-02T09:14:38.713 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:38.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.715+0000 7fd6a7000700 1 -- 172.21.15.89:0/963577847 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd68006aed0 msgr2=0x7fd68006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.715+0000 7fd6a7000700 1 --2- 172.21.15.89:0/963577847 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd68006aed0 0x7fd68006d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd6900018a0 tx=0x7fd690009040 comp rx=0 tx=0).stop 2024-04-02T09:14:38.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.715+0000 7fd6a7000700 1 -- 172.21.15.89:0/963577847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6a015d640 msgr2=0x7fd6a00cb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.715+0000 7fd6a7000700 1 --2- 172.21.15.89:0/963577847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6a015d640 0x7fd6a00cb030 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00c4d0 tx=0x7fd68c00c7e0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.715+0000 7fd6a7000700 1 -- 172.21.15.89:0/963577847 shutdown_connections 2024-04-02T09:14:38.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.715+0000 7fd6a7000700 1 --2- 172.21.15.89:0/963577847 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd68006aed0 0x7fd68006d390 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.715+0000 7fd6a7000700 1 --2- 172.21.15.89:0/963577847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6a01619d0 0x7fd6a01f30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.715+0000 7fd6a7000700 1 --2- 172.21.15.89:0/963577847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6a00ceec0 0x7fd6a015c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.715+0000 7fd6a7000700 1 --2- 172.21.15.89:0/963577847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6a015d640 0x7fd6a00cb030 unknown :-1 s=CLOSED pgs=2814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.715+0000 7fd6a7000700 1 -- 172.21.15.89:0/963577847 >> 172.21.15.89:0/963577847 conn(0x7fd6a008aff0 msgr2=0x7fd6a006c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:38.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.715+0000 7fd6a7000700 1 -- 172.21.15.89:0/963577847 shutdown_connections 2024-04-02T09:14:38.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.715+0000 7fd6a7000700 1 -- 172.21.15.89:0/963577847 wait complete. 2024-04-02T09:14:38.716 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:14:38.731 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-04-02T09:14:37.258226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":418600,"kb_used_data":26912,"kb_used_omap":12,"kb_used_meta":391539,"kb_avail":754556120,"statfs":{"total":773094113280,"available":772665466880,"internally_reserved":0,"allocated":27557888,"data_stored":14733514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":400936504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.413332"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"250'381","reported_seq":1361,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-04-02T09:14:29.329113+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:14:29.329113+0000","last_peered":"2024-04-02T09:14:29.329113+0000","last_clean":"2024-04-02T09:14:29.329113+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:14:29.329113+0000","last_undegraded":"2024-04-02T09:14:29.329113+0000","last_fullsized":"2024-04-02T09:14:29.329113+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60576,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311264,"statfs":{"total":96636764160,"available":96574734336,"internally_reserved":0,"allocated":3297280,"data_stored":1694352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.70899999999999996,"15min":0.71799999999999997},"min":{"1min":0.34599999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.80000000000000004,"5min":8.016,"15min":8.016},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.69899999999999995,"15min":0.71899999999999997},"min":{"1min":0.379,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.83299999999999996,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.48899999999999999}]},{"osd":1,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.74299999999999999,"15min":0.74199999999999999},"min":{"1min":0.46300000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91000000000000003,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.42399999999999999},"max":{"1min":0.73499999999999999,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.72999999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.77900000000000003,"15min":0.76900000000000002},"min":{"1min":0.46999999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86199999999999999,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.78400000000000003,"15min":0.80400000000000005},"min":{"1min":0.53800000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.872,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.871}]},{"osd":3,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.751,"15min":0.749},"min":{"1min":0.49199999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86299999999999999,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.79000000000000004,"15min":0.77200000000000002},"min":{"1min":0.51900000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":5.093,"15min":5.093},"last":0.64800000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.66400000000000003,"15min":0.624},"min":{"1min":0.39200000000000002,"5min":0.38300000000000001,"15min":0.36499999999999999},"max":{"1min":0.83699999999999997,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.746},"min":{"1min":0.58499999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90100000000000002,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":1.052}]},{"osd":5,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.73199999999999998,"15min":0.69399999999999995},"min":{"1min":0.436,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":0.88600000000000001,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.77000000000000002,"15min":0.73399999999999999},"min":{"1min":0.54000000000000004,"5min":0.42599999999999999,"15min":0.41099999999999998},"max":{"1min":0.878,"5min":3.492,"15min":3.492},"last":0.89100000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.72099999999999997,"15min":0.69599999999999995},"min":{"1min":0.51300000000000001,"5min":0.48299999999999998,"15min":0.47099999999999997},"max":{"1min":0.90700000000000003,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.77600000000000002,"15min":0.71699999999999997},"min":{"1min":0.42199999999999999,"5min":0.41599999999999998,"15min":0.41099999999999998},"max":{"1min":0.91600000000000004,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.91100000000000003}]}]},{"osd":6,"up_from":13,"seq":55834576006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":42528,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329312,"statfs":{"total":96636764160,"available":96593215488,"internally_reserved":0,"allocated":3297280,"data_stored":1694352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68200000000000005,"15min":0.72099999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93999999999999995,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.625},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.73399999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.0600000000000001},"last":0.745}]},{"osd":1,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.67900000000000005,"15min":0.71299999999999997},"min":{"1min":0.46000000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.999,"5min":1.044,"15min":1.212},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.70799999999999996,"15min":0.747},"min":{"1min":0.505,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.0589999999999999,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.65400000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.69799999999999995,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94799999999999995,"5min":1.046,"15min":1.2789999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70099999999999996,"15min":0.73299999999999998},"min":{"1min":0.47499999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.226},"last":0.66300000000000003}]},{"osd":3,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.71999999999999997,"15min":0.75600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.026,"5min":1.141,"15min":1.2549999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.04,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.71899999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.628,"15min":0.626},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65800000000000003,"15min":0.66200000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.41999999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1879999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.64600000000000002,"15min":0.65200000000000002},"min":{"1min":0.433,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88800000000000001,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.70699999999999996,"15min":0.72499999999999998},"min":{"1min":0.501,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.73499999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73899999999999999},"min":{"1min":0.48499999999999999,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.0129999999999999,"5min":1.21,"15min":1.2370000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.72899999999999998,"15min":0.73899999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.49399999999999999},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.69399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834576007,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57060,"kb_used_data":3800,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314780,"statfs":{"total":96636764160,"available":96578334720,"internally_reserved":0,"allocated":3891200,"data_stored":2284720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42899999999999999,"15min":0.44800000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82499999999999996},"last":0.443},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48299999999999998,"15min":0.503},"min":{"1min":0.31900000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.79800000000000004},"last":0.52300000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.33600000000000002,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.59599999999999997,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.497},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58699999999999997},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.54300000000000004}]},{"osd":3,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54100000000000004,"15min":0.55600000000000005},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.47599999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.76000000000000001,"15min":0.78100000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.433},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.753,"15min":0.77700000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.41799999999999998},"max":{"1min":0.98299999999999998,"5min":1.05,"15min":1.25},"last":0.63800000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.753,"15min":0.78200000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.93400000000000005,"5min":1.03,"15min":1.321},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78400000000000003,"15min":0.80100000000000005},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.115,"5min":1.115,"15min":1.173},"last":0.67800000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.78300000000000003,"15min":0.80700000000000005},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.48099999999999998},"max":{"1min":1.087,"5min":1.087,"15min":1.2210000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.79100000000000004,"15min":0.80600000000000005},"min":{"1min":0.52900000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.024,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.85299999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.78900000000000003,"15min":0.81499999999999995},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0740000000000001,"5min":1.1080000000000001,"15min":1.236},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81200000000000006,"15min":0.83099999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.129,"5min":1.129,"15min":1.2789999999999999},"last":0.66600000000000004}]}]},{"osd":0,"up_from":13,"seq":55834576007,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52388,"kb_used_data":3216,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319452,"statfs":{"total":96636764160,"available":96583118848,"internally_reserved":0,"allocated":3293184,"data_stored":1693333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.42699999999999999,"15min":0.45000000000000001},"min":{"1min":0.26700000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.58999999999999997,"5min":0.69899999999999995,"15min":0.86699999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.48099999999999998,"15min":0.499},"min":{"1min":0.29799999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66300000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.46899999999999997}]},{"osd":2,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.48699999999999999,"15min":0.50600000000000001},"min":{"1min":0.35699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.621,"5min":0.69399999999999995,"15min":0.82699999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52100000000000002,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.63300000000000001,"5min":0.999,"15min":0.999},"last":0.51200000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53100000000000003,"15min":0.56100000000000005},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.63500000000000001,"5min":0.79800000000000004,"15min":0.85699999999999998},"last":0.437},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.63},"min":{"1min":0.47599999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.66000000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.498}]},{"osd":4,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.82199999999999995},"min":{"1min":0.622,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":0.86499999999999999,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.76000000000000001,"15min":0.81599999999999995},"min":{"1min":0.55100000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.90200000000000002,"5min":1.169,"15min":1.169},"last":0.80100000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.76500000000000001,"15min":0.81699999999999995},"min":{"1min":0.58299999999999996,"5min":0.497,"15min":0.497},"max":{"1min":0.95199999999999996,"5min":1.145,"15min":1.232},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.78500000000000003,"15min":0.82499999999999996},"min":{"1min":0.53600000000000003,"5min":0.497,"15min":0.497},"max":{"1min":0.94399999999999995,"5min":1.21,"15min":1.21},"last":0.86599999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.75,"15min":0.79100000000000004},"min":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.88200000000000001,"5min":1.1830000000000001,"15min":1.196},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.77200000000000002,"15min":0.80400000000000005},"min":{"1min":0.54200000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96799999999999997,"5min":1.246,"15min":1.246},"last":0.93200000000000005}]},{"osd":7,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.80700000000000005,"15min":0.84799999999999998},"min":{"1min":0.60999999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.98699999999999999,"5min":1.292,"15min":1.292},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.79800000000000004,"15min":0.83999999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.95799999999999996,"5min":1.1719999999999999,"15min":1.2130000000000001},"last":0.83299999999999996}]}]},{"osd":2,"up_from":13,"seq":55834576006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52392,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319448,"statfs":{"total":96636764160,"available":96583114752,"internally_reserved":0,"allocated":3297280,"data_stored":1694352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.45100000000000001,"15min":0.443},"min":{"1min":0.28899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.316},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.31,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":0.46100000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.496,"15min":0.503},"min":{"1min":0.34000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53600000000000003,"15min":0.55400000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.42399999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.71199999999999997,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.626},"min":{"1min":0.45400000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.44}]},{"osd":4,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.75900000000000001,"15min":0.77300000000000002},"min":{"1min":0.56699999999999995,"5min":0.495,"15min":0.495},"max":{"1min":2.3999999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.77500000000000002,"15min":0.78900000000000003},"min":{"1min":0.54600000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.52700000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.77600000000000002,"15min":0.78600000000000003},"min":{"1min":0.57799999999999996,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1379999999999999,"5min":1.99,"15min":1.99},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.78000000000000003,"15min":0.80100000000000005},"min":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.51900000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79100000000000004,"15min":0.80500000000000005},"min":{"1min":0.59299999999999997,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77300000000000002,"15min":0.80100000000000005},"min":{"1min":0.55900000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.103,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.60799999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.80000000000000004,"15min":0.82399999999999995},"min":{"1min":0.56999999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.82199999999999995},"min":{"1min":0.59099999999999997,"5min":0.497,"15min":0.497},"max":{"1min":0.97499999999999998,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.54900000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52388,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319452,"statfs":{"total":96636764160,"available":96583118848,"internally_reserved":0,"allocated":3297280,"data_stored":1694352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.41699999999999998,"15min":0.443},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64000000000000001,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.312},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.58699999999999997,"5min":0.71199999999999997,"15min":0.876},"last":0.35199999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47499999999999998,"15min":0.498},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.90300000000000002},"last":0.5},{"interface":"front","average":{"1min":0.496,"5min":0.53900000000000003,"15min":0.56799999999999995},"min":{"1min":0.38500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.66100000000000003,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.58099999999999996,"15min":0.61699999999999999},"min":{"1min":0.41299999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71299999999999997,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56799999999999995,"15min":0.60999999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":1.0840000000000001},"last":0.46200000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72099999999999997,"15min":0.751},"min":{"1min":0.438,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.73499999999999999,"15min":0.76600000000000001},"min":{"1min":0.49299999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.97099999999999997,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.57099999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.74299999999999999,"15min":0.77400000000000002},"min":{"1min":0.59399999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.75700000000000001,"15min":0.78000000000000003},"min":{"1min":0.51100000000000001,"5min":0.505,"15min":0.505},"max":{"1min":0.98299999999999998,"5min":1.266,"15min":1.266},"last":0.62}]},{"osd":6,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.96799999999999997,"5min":1.284,"15min":1.284},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76500000000000001,"15min":0.80300000000000005},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.495},"max":{"1min":1.0389999999999999,"5min":1.2,"15min":1.2},"last":0.67200000000000004}]},{"osd":7,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.76600000000000001,"15min":0.80400000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.125,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.78900000000000003,"15min":0.81699999999999995},"min":{"1min":0.57599999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.087,"5min":1.248,"15min":1.248},"last":0.68500000000000005}]}]},{"osd":4,"up_from":13,"seq":55834576006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48296,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323544,"statfs":{"total":96636764160,"available":96587309056,"internally_reserved":0,"allocated":3297280,"data_stored":1694352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64500000000000002,"15min":0.67700000000000005},"min":{"1min":0.38100000000000001,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":0.94399999999999995,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.63600000000000001,"15min":0.67100000000000004},"min":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.075},"last":0.496}]},{"osd":1,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.70499999999999996,"15min":0.746},"min":{"1min":0.51700000000000002,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":1.014,"5min":1.0549999999999999,"15min":1.069},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72199999999999998,"15min":0.75700000000000001},"min":{"1min":0.49099999999999999,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.57399999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66200000000000003,"15min":0.69999999999999996},"min":{"1min":0.439,"5min":0.438,"15min":0.438},"max":{"1min":0.95799999999999996,"5min":1.077,"15min":1.077},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68700000000000006,"15min":0.71899999999999997},"min":{"1min":0.47799999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.032,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.45300000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.71399999999999997,"15min":0.753},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.47399999999999998},"max":{"1min":0.93999999999999995,"5min":0.98099999999999998,"15min":1.018},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70999999999999996,"15min":0.74199999999999999},"min":{"1min":0.52100000000000002,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.55800000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.41299999999999998,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.93700000000000006,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.70299999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.64200000000000002,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91800000000000004,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.746,"5min":0.69199999999999995,"15min":0.69599999999999995},"min":{"1min":0.46899999999999997,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":1.0149999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.64700000000000002}]},{"osd":7,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.68500000000000005,"15min":0.70499999999999996},"min":{"1min":0.45500000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.0329999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76500000000000001,"15min":0.78000000000000003},"min":{"1min":0.624,"5min":0.5,"15min":0.5},"max":{"1min":1.0700000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.80500000000000005}]}]},{"osd":5,"up_from":13,"seq":55834576009,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52972,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318868,"statfs":{"total":96636764160,"available":96582520832,"internally_reserved":0,"allocated":3887104,"data_stored":2283701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.82699999999999996,"5min":0.98699999999999999,"15min":1.004},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.68999999999999995,"15min":0.71999999999999997},"min":{"1min":0.48399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.028,"5min":1.085,"15min":1.085},"last":0.5}]},{"osd":1,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.68100000000000005,"15min":0.71499999999999997},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84399999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70999999999999996,"15min":0.72499999999999998},"min":{"1min":0.52300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88500000000000001,"5min":0.95299999999999996,"15min":1.052},"last":0.53900000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.71599999999999997,"15min":0.73799999999999999},"min":{"1min":0.47199999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97299999999999998,"5min":1.004,"15min":1.014},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.69499999999999995,"15min":0.73699999999999999},"min":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.81799999999999995,"5min":1.018,"15min":1.0680000000000001},"last":0.52700000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.747,"15min":0.77700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.88500000000000001,"5min":0.98899999999999999,"15min":1.083},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77000000000000002,"15min":0.79000000000000004},"min":{"1min":0.58399999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1020000000000001},"last":0.50600000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.40699999999999997,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.054},"last":0.441},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":0.98799999999999999,"5min":1.145,"15min":1.145},"last":0.46000000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67900000000000005,"15min":0.67300000000000004},"min":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96799999999999997,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71399999999999997,"15min":0.71099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.94899999999999995,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.48599999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73999999999999999,"15min":0.73799999999999999},"min":{"1min":0.46500000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.0820000000000001,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76300000000000001,"15min":0.76600000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1519999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:14:38.736 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:14:38.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.966+0000 7fcb92200700 1 Processor -- start 2024-04-02T09:14:38.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.966+0000 7fcb92200700 1 -- start start 2024-04-02T09:14:38.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.967+0000 7fcb92200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb8c15d640 0x7fcb8c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.968+0000 7fcb92200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb8c0ceec0 0x7fcb8c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.968+0000 7fcb92200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8c0d30d0 con 0x7fcb8c161800 2024-04-02T09:14:38.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.968+0000 7fcb92200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8c161c90 con 0x7fcb8c0ceec0 2024-04-02T09:14:38.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.968+0000 7fcb92200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0b6840 con 0x7fcb8c15d640 2024-04-02T09:14:38.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.968+0000 7fcb8aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb8c0ceec0 0x7fcb8c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.968+0000 7fcb8aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb8c0ceec0 0x7fcb8c1612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35066/0 (socket says 172.21.15.89:35066) 2024-04-02T09:14:38.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.968+0000 7fcb8aa00700 1 -- 172.21.15.89:0/1203999417 learned_addr learned my addr 172.21.15.89:0/1203999417 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:38.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.968+0000 7fcb8b400700 1 --2- 172.21.15.89:0/1203999417 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb8c15d640 0x7fcb8c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.968+0000 7fcb8aa00700 1 -- 172.21.15.89:0/1203999417 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb8c15d640 msgr2=0x7fcb8c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:38.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.968+0000 7fcb8aa00700 1 --2- 172.21.15.89:0/1203999417 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb8c15d640 0x7fcb8c0d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.968+0000 7fcb8aa00700 1 -- 172.21.15.89:0/1203999417 >> v1:172.21.15.113:6789/0 conn(0x7fcb8c161800 legacy=0x7fcb8c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:38.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.968+0000 7fcb8aa00700 1 -- 172.21.15.89:0/1203999417 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c0d2880 con 0x7fcb8c0ceec0 2024-04-02T09:14:38.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.969+0000 7fcb8aa00700 1 --2- 172.21.15.89:0/1203999417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb8c0ceec0 0x7fcb8c1612c0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7fcb780098f0 tx=0x7fcb78009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c4844c3c832195 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:38.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.969+0000 7fcb8a000700 1 -- 172.21.15.89:0/1203999417 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78004230 con 0x7fcb8c0ceec0 2024-04-02T09:14:38.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.969+0000 7fcb8a000700 1 -- 172.21.15.89:0/1203999417 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb78004390 con 0x7fcb8c0ceec0 2024-04-02T09:14:38.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.969+0000 7fcb8a000700 1 -- 172.21.15.89:0/1203999417 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb780036a0 con 0x7fcb8c0ceec0 2024-04-02T09:14:38.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.969+0000 7fcb92200700 1 -- 172.21.15.89:0/1203999417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb8c0ceec0 msgr2=0x7fcb8c1612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.969+0000 7fcb92200700 1 --2- 172.21.15.89:0/1203999417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb8c0ceec0 0x7fcb8c1612c0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7fcb780098f0 tx=0x7fcb78009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:38.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.969+0000 7fcb92200700 1 -- 172.21.15.89:0/1203999417 shutdown_connections 2024-04-02T09:14:38.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.969+0000 7fcb92200700 1 --2- 172.21.15.89:0/1203999417 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb8c15d640 0x7fcb8c0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.969+0000 7fcb92200700 1 --2- 172.21.15.89:0/1203999417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb8c0ceec0 0x7fcb8c1612c0 unknown :-1 s=CLOSED pgs=2815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.969+0000 7fcb92200700 1 -- 172.21.15.89:0/1203999417 >> 172.21.15.89:0/1203999417 conn(0x7fcb8c08aff0 msgr2=0x7fcb8c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:38.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.970+0000 7fcb92200700 1 -- 172.21.15.89:0/1203999417 shutdown_connections 2024-04-02T09:14:38.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.970+0000 7fcb92200700 1 -- 172.21.15.89:0/1203999417 wait complete. 2024-04-02T09:14:38.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.971+0000 7fcb92200700 1 Processor -- start 2024-04-02T09:14:38.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.971+0000 7fcb92200700 1 -- start start 2024-04-02T09:14:38.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.972+0000 7fcb92200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb8c0ceec0 0x7fcb8c158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.972+0000 7fcb92200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb8c15d640 0x7fcb8c1f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.972+0000 7fcb92200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb8c161800 0x7fcb8c1f6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.972+0000 7fcb92200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0c79e0 con 0x7fcb8c15d640 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.972+0000 7fcb92200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0c7080 con 0x7fcb8c161800 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.972+0000 7fcb92200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c1f7450 con 0x7fcb8c0ceec0 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.973+0000 7fcb8aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb8c15d640 0x7fcb8c1f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.973+0000 7fcb8aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb8c15d640 0x7fcb8c1f28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35080/0 (socket says 172.21.15.89:35080) 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.973+0000 7fcb8aa00700 1 -- 172.21.15.89:0/1919220154 learned_addr learned my addr 172.21.15.89:0/1919220154 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.973+0000 7fcb8b400700 1 --2- 172.21.15.89:0/1919220154 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb8c0ceec0 0x7fcb8c158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.973+0000 7fcb8be00700 1 --2- 172.21.15.89:0/1919220154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb8c161800 0x7fcb8c1f6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.973+0000 7fcb8aa00700 1 -- 172.21.15.89:0/1919220154 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb8c0ceec0 msgr2=0x7fcb8c158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.973+0000 7fcb8aa00700 1 --2- 172.21.15.89:0/1919220154 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb8c0ceec0 0x7fcb8c158710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.973+0000 7fcb8aa00700 1 -- 172.21.15.89:0/1919220154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb8c161800 msgr2=0x7fcb8c1f6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.973+0000 7fcb8aa00700 1 --2- 172.21.15.89:0/1919220154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb8c161800 0x7fcb8c1f6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.973+0000 7fcb8aa00700 1 -- 172.21.15.89:0/1919220154 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb78009550 con 0x7fcb8c15d640 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.973+0000 7fcb8aa00700 1 --2- 172.21.15.89:0/1919220154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb8c15d640 0x7fcb8c1f28a0 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009130 tx=0x7fcb78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.974+0000 7fcb83400700 1 -- 172.21.15.89:0/1919220154 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78003cb0 con 0x7fcb8c15d640 2024-04-02T09:14:38.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.974+0000 7fcb83400700 1 -- 172.21.15.89:0/1919220154 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb78003e10 con 0x7fcb8c15d640 2024-04-02T09:14:38.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.974+0000 7fcb83400700 1 -- 172.21.15.89:0/1919220154 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7800b8a0 con 0x7fcb8c15d640 2024-04-02T09:14:38.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.974+0000 7fcb92200700 1 -- 172.21.15.89:0/1919220154 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8c0c7320 con 0x7fcb8c15d640 2024-04-02T09:14:38.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.974+0000 7fcb92200700 1 -- 172.21.15.89:0/1919220154 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8c1f2de0 con 0x7fcb8c15d640 2024-04-02T09:14:38.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.975+0000 7fcb92200700 1 -- 172.21.15.89:0/1919220154 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8c067720 con 0x7fcb8c15d640 2024-04-02T09:14:38.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.976+0000 7fcb83400700 1 -- 172.21.15.89:0/1919220154 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fcb7800ba00 con 0x7fcb8c15d640 2024-04-02T09:14:38.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.976+0000 7fcb83400700 1 --2- 172.21.15.89:0/1919220154 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcb6c06aed0 0x7fcb6c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:38.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.976+0000 7fcb83400700 1 -- 172.21.15.89:0/1919220154 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fcb7808adc0 con 0x7fcb8c15d640 2024-04-02T09:14:38.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.977+0000 7fcb8b400700 1 --2- 172.21.15.89:0/1919220154 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcb6c06aed0 0x7fcb6c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:38.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.977+0000 7fcb8b400700 1 --2- 172.21.15.89:0/1919220154 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcb6c06aed0 0x7fcb6c06d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c006b20 tx=0x7fcb7c008040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:38.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:38.981+0000 7fcb83400700 1 -- 172.21.15.89:0/1919220154 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcb78055b40 con 0x7fcb8c15d640 2024-04-02T09:14:39.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.177+0000 7fcb92200700 1 -- 172.21.15.89:0/1919220154 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcb8c0b64d0 con 0x7fcb6c06aed0 2024-04-02T09:14:39.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.185+0000 7fcb83400700 1 -- 172.21.15.89:0/1919220154 <== mgr.11320 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+274194 (secure 0 0 0) 0x7fcb7c009e40 con 0x7fcb6c06aed0 2024-04-02T09:14:39.187 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.190+0000 7fcb92200700 1 -- 172.21.15.89:0/1919220154 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcb6c06aed0 msgr2=0x7fcb6c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.190+0000 7fcb92200700 1 --2- 172.21.15.89:0/1919220154 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcb6c06aed0 0x7fcb6c06d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c006b20 tx=0x7fcb7c008040 comp rx=0 tx=0).stop 2024-04-02T09:14:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.190+0000 7fcb92200700 1 -- 172.21.15.89:0/1919220154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb8c15d640 msgr2=0x7fcb8c1f28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.190+0000 7fcb92200700 1 --2- 172.21.15.89:0/1919220154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb8c15d640 0x7fcb8c1f28a0 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009130 tx=0x7fcb78004750 comp rx=0 tx=0).stop 2024-04-02T09:14:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.190+0000 7fcb92200700 1 -- 172.21.15.89:0/1919220154 shutdown_connections 2024-04-02T09:14:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.190+0000 7fcb92200700 1 --2- 172.21.15.89:0/1919220154 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcb6c06aed0 0x7fcb6c06d390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.190+0000 7fcb92200700 1 --2- 172.21.15.89:0/1919220154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb8c161800 0x7fcb8c1f6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.190+0000 7fcb92200700 1 --2- 172.21.15.89:0/1919220154 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb8c0ceec0 0x7fcb8c158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.190+0000 7fcb92200700 1 --2- 172.21.15.89:0/1919220154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb8c15d640 0x7fcb8c1f28a0 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.190+0000 7fcb92200700 1 -- 172.21.15.89:0/1919220154 >> 172.21.15.89:0/1919220154 conn(0x7fcb8c08aff0 msgr2=0x7fcb8c06d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.190+0000 7fcb92200700 1 -- 172.21.15.89:0/1919220154 shutdown_connections 2024-04-02T09:14:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.190+0000 7fcb92200700 1 -- 172.21.15.89:0/1919220154 wait complete. 2024-04-02T09:14:39.190 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:14:39.206 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-04-02T09:14:37.258226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":418600,"kb_used_data":26912,"kb_used_omap":12,"kb_used_meta":391539,"kb_avail":754556120,"statfs":{"total":773094113280,"available":772665466880,"internally_reserved":0,"allocated":27557888,"data_stored":14733514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":400936504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.413332"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:32.836277+0000","last_change":"2024-04-02T09:14:32.836277+0000","last_active":"2024-04-02T09:14:32.836277+0000","last_peered":"2024-04-02T09:14:32.836277+0000","last_clean":"2024-04-02T09:14:32.836277+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:32.836277+0000","last_undegraded":"2024-04-02T09:14:32.836277+0000","last_fullsized":"2024-04-02T09:14:32.836277+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-02T09:14:33.844811+0000","last_change":"2024-04-02T09:14:33.844811+0000","last_active":"2024-04-02T09:14:33.844811+0000","last_peered":"2024-04-02T09:14:33.844811+0000","last_clean":"2024-04-02T09:14:33.844811+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-02T09:14:33.844811+0000","last_undegraded":"2024-04-02T09:14:33.844811+0000","last_fullsized":"2024-04-02T09:14:33.844811+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"250'381","reported_seq":1361,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-04-02T09:14:29.329113+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:14:29.329113+0000","last_peered":"2024-04-02T09:14:29.329113+0000","last_clean":"2024-04-02T09:14:29.329113+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:14:29.329113+0000","last_undegraded":"2024-04-02T09:14:29.329113+0000","last_fullsized":"2024-04-02T09:14:29.329113+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60576,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311264,"statfs":{"total":96636764160,"available":96574734336,"internally_reserved":0,"allocated":3297280,"data_stored":1694352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.70899999999999996,"15min":0.71799999999999997},"min":{"1min":0.34599999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.80000000000000004,"5min":8.016,"15min":8.016},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.69899999999999995,"15min":0.71899999999999997},"min":{"1min":0.379,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.83299999999999996,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.48899999999999999}]},{"osd":1,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.74299999999999999,"15min":0.74199999999999999},"min":{"1min":0.46300000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91000000000000003,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.42399999999999999},"max":{"1min":0.73499999999999999,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.72999999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.77900000000000003,"15min":0.76900000000000002},"min":{"1min":0.46999999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86199999999999999,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.78400000000000003,"15min":0.80400000000000005},"min":{"1min":0.53800000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.872,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.871}]},{"osd":3,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.751,"15min":0.749},"min":{"1min":0.49199999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86299999999999999,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.79000000000000004,"15min":0.77200000000000002},"min":{"1min":0.51900000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":5.093,"15min":5.093},"last":0.64800000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.66400000000000003,"15min":0.624},"min":{"1min":0.39200000000000002,"5min":0.38300000000000001,"15min":0.36499999999999999},"max":{"1min":0.83699999999999997,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.746},"min":{"1min":0.58499999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90100000000000002,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":1.052}]},{"osd":5,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.73199999999999998,"15min":0.69399999999999995},"min":{"1min":0.436,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":0.88600000000000001,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.77000000000000002,"15min":0.73399999999999999},"min":{"1min":0.54000000000000004,"5min":0.42599999999999999,"15min":0.41099999999999998},"max":{"1min":0.878,"5min":3.492,"15min":3.492},"last":0.89100000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.72099999999999997,"15min":0.69599999999999995},"min":{"1min":0.51300000000000001,"5min":0.48299999999999998,"15min":0.47099999999999997},"max":{"1min":0.90700000000000003,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.77600000000000002,"15min":0.71699999999999997},"min":{"1min":0.42199999999999999,"5min":0.41599999999999998,"15min":0.41099999999999998},"max":{"1min":0.91600000000000004,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.91100000000000003}]}]},{"osd":6,"up_from":13,"seq":55834576006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":42528,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329312,"statfs":{"total":96636764160,"available":96593215488,"internally_reserved":0,"allocated":3297280,"data_stored":1694352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68200000000000005,"15min":0.72099999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93999999999999995,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.625},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.73399999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.0600000000000001},"last":0.745}]},{"osd":1,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.67900000000000005,"15min":0.71299999999999997},"min":{"1min":0.46000000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.999,"5min":1.044,"15min":1.212},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.70799999999999996,"15min":0.747},"min":{"1min":0.505,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.0589999999999999,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.65400000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.69799999999999995,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94799999999999995,"5min":1.046,"15min":1.2789999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70099999999999996,"15min":0.73299999999999998},"min":{"1min":0.47499999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.226},"last":0.66300000000000003}]},{"osd":3,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.71999999999999997,"15min":0.75600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.026,"5min":1.141,"15min":1.2549999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.04,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.71899999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.628,"15min":0.626},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65800000000000003,"15min":0.66200000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.41999999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1879999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.64600000000000002,"15min":0.65200000000000002},"min":{"1min":0.433,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88800000000000001,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.70699999999999996,"15min":0.72499999999999998},"min":{"1min":0.501,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.73499999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73899999999999999},"min":{"1min":0.48499999999999999,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.0129999999999999,"5min":1.21,"15min":1.2370000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.72899999999999998,"15min":0.73899999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.49399999999999999},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.69399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834576007,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57060,"kb_used_data":3800,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314780,"statfs":{"total":96636764160,"available":96578334720,"internally_reserved":0,"allocated":3891200,"data_stored":2284720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42899999999999999,"15min":0.44800000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82499999999999996},"last":0.443},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48299999999999998,"15min":0.503},"min":{"1min":0.31900000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.79800000000000004},"last":0.52300000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.33600000000000002,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.59599999999999997,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.497},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58699999999999997},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.54300000000000004}]},{"osd":3,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54100000000000004,"15min":0.55600000000000005},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.47599999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.76000000000000001,"15min":0.78100000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.433},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.753,"15min":0.77700000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.41799999999999998},"max":{"1min":0.98299999999999998,"5min":1.05,"15min":1.25},"last":0.63800000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.753,"15min":0.78200000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.93400000000000005,"5min":1.03,"15min":1.321},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78400000000000003,"15min":0.80100000000000005},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.115,"5min":1.115,"15min":1.173},"last":0.67800000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.78300000000000003,"15min":0.80700000000000005},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.48099999999999998},"max":{"1min":1.087,"5min":1.087,"15min":1.2210000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.79100000000000004,"15min":0.80600000000000005},"min":{"1min":0.52900000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.024,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.85299999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.78900000000000003,"15min":0.81499999999999995},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0740000000000001,"5min":1.1080000000000001,"15min":1.236},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81200000000000006,"15min":0.83099999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.129,"5min":1.129,"15min":1.2789999999999999},"last":0.66600000000000004}]}]},{"osd":0,"up_from":13,"seq":55834576007,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52388,"kb_used_data":3216,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319452,"statfs":{"total":96636764160,"available":96583118848,"internally_reserved":0,"allocated":3293184,"data_stored":1693333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.42699999999999999,"15min":0.45000000000000001},"min":{"1min":0.26700000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.58999999999999997,"5min":0.69899999999999995,"15min":0.86699999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.48099999999999998,"15min":0.499},"min":{"1min":0.29799999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66300000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.46899999999999997}]},{"osd":2,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.48699999999999999,"15min":0.50600000000000001},"min":{"1min":0.35699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.621,"5min":0.69399999999999995,"15min":0.82699999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52100000000000002,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.63300000000000001,"5min":0.999,"15min":0.999},"last":0.51200000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53100000000000003,"15min":0.56100000000000005},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.63500000000000001,"5min":0.79800000000000004,"15min":0.85699999999999998},"last":0.437},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.63},"min":{"1min":0.47599999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.66000000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.498}]},{"osd":4,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.82199999999999995},"min":{"1min":0.622,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":0.86499999999999999,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.76000000000000001,"15min":0.81599999999999995},"min":{"1min":0.55100000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.90200000000000002,"5min":1.169,"15min":1.169},"last":0.80100000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.76500000000000001,"15min":0.81699999999999995},"min":{"1min":0.58299999999999996,"5min":0.497,"15min":0.497},"max":{"1min":0.95199999999999996,"5min":1.145,"15min":1.232},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.78500000000000003,"15min":0.82499999999999996},"min":{"1min":0.53600000000000003,"5min":0.497,"15min":0.497},"max":{"1min":0.94399999999999995,"5min":1.21,"15min":1.21},"last":0.86599999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.75,"15min":0.79100000000000004},"min":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.88200000000000001,"5min":1.1830000000000001,"15min":1.196},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.77200000000000002,"15min":0.80400000000000005},"min":{"1min":0.54200000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96799999999999997,"5min":1.246,"15min":1.246},"last":0.93200000000000005}]},{"osd":7,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.80700000000000005,"15min":0.84799999999999998},"min":{"1min":0.60999999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.98699999999999999,"5min":1.292,"15min":1.292},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.79800000000000004,"15min":0.83999999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.95799999999999996,"5min":1.1719999999999999,"15min":1.2130000000000001},"last":0.83299999999999996}]}]},{"osd":2,"up_from":13,"seq":55834576006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52392,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319448,"statfs":{"total":96636764160,"available":96583114752,"internally_reserved":0,"allocated":3297280,"data_stored":1694352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.45100000000000001,"15min":0.443},"min":{"1min":0.28899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.316},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.31,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":0.46100000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.496,"15min":0.503},"min":{"1min":0.34000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53600000000000003,"15min":0.55400000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.42399999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.71199999999999997,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.626},"min":{"1min":0.45400000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.44}]},{"osd":4,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.75900000000000001,"15min":0.77300000000000002},"min":{"1min":0.56699999999999995,"5min":0.495,"15min":0.495},"max":{"1min":2.3999999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.77500000000000002,"15min":0.78900000000000003},"min":{"1min":0.54600000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.52700000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.77600000000000002,"15min":0.78600000000000003},"min":{"1min":0.57799999999999996,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1379999999999999,"5min":1.99,"15min":1.99},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.78000000000000003,"15min":0.80100000000000005},"min":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.51900000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79100000000000004,"15min":0.80500000000000005},"min":{"1min":0.59299999999999997,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77300000000000002,"15min":0.80100000000000005},"min":{"1min":0.55900000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.103,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.60799999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.80000000000000004,"15min":0.82399999999999995},"min":{"1min":0.56999999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.82199999999999995},"min":{"1min":0.59099999999999997,"5min":0.497,"15min":0.497},"max":{"1min":0.97499999999999998,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.54900000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52388,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319452,"statfs":{"total":96636764160,"available":96583118848,"internally_reserved":0,"allocated":3297280,"data_stored":1694352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.41699999999999998,"15min":0.443},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64000000000000001,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.312},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.58699999999999997,"5min":0.71199999999999997,"15min":0.876},"last":0.35199999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47499999999999998,"15min":0.498},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.90300000000000002},"last":0.5},{"interface":"front","average":{"1min":0.496,"5min":0.53900000000000003,"15min":0.56799999999999995},"min":{"1min":0.38500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.66100000000000003,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.58099999999999996,"15min":0.61699999999999999},"min":{"1min":0.41299999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71299999999999997,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56799999999999995,"15min":0.60999999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":1.0840000000000001},"last":0.46200000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72099999999999997,"15min":0.751},"min":{"1min":0.438,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.73499999999999999,"15min":0.76600000000000001},"min":{"1min":0.49299999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.97099999999999997,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.57099999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.74299999999999999,"15min":0.77400000000000002},"min":{"1min":0.59399999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.75700000000000001,"15min":0.78000000000000003},"min":{"1min":0.51100000000000001,"5min":0.505,"15min":0.505},"max":{"1min":0.98299999999999998,"5min":1.266,"15min":1.266},"last":0.62}]},{"osd":6,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.96799999999999997,"5min":1.284,"15min":1.284},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76500000000000001,"15min":0.80300000000000005},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.495},"max":{"1min":1.0389999999999999,"5min":1.2,"15min":1.2},"last":0.67200000000000004}]},{"osd":7,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.76600000000000001,"15min":0.80400000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.125,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.78900000000000003,"15min":0.81699999999999995},"min":{"1min":0.57599999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.087,"5min":1.248,"15min":1.248},"last":0.68500000000000005}]}]},{"osd":4,"up_from":13,"seq":55834576006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48296,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323544,"statfs":{"total":96636764160,"available":96587309056,"internally_reserved":0,"allocated":3297280,"data_stored":1694352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64500000000000002,"15min":0.67700000000000005},"min":{"1min":0.38100000000000001,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":0.94399999999999995,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.63600000000000001,"15min":0.67100000000000004},"min":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.075},"last":0.496}]},{"osd":1,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.70499999999999996,"15min":0.746},"min":{"1min":0.51700000000000002,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":1.014,"5min":1.0549999999999999,"15min":1.069},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72199999999999998,"15min":0.75700000000000001},"min":{"1min":0.49099999999999999,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.57399999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66200000000000003,"15min":0.69999999999999996},"min":{"1min":0.439,"5min":0.438,"15min":0.438},"max":{"1min":0.95799999999999996,"5min":1.077,"15min":1.077},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68700000000000006,"15min":0.71899999999999997},"min":{"1min":0.47799999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.032,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.45300000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.71399999999999997,"15min":0.753},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.47399999999999998},"max":{"1min":0.93999999999999995,"5min":0.98099999999999998,"15min":1.018},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70999999999999996,"15min":0.74199999999999999},"min":{"1min":0.52100000000000002,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.55800000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.41299999999999998,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.93700000000000006,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.70299999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.64200000000000002,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91800000000000004,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.746,"5min":0.69199999999999995,"15min":0.69599999999999995},"min":{"1min":0.46899999999999997,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":1.0149999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.64700000000000002}]},{"osd":7,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.68500000000000005,"15min":0.70499999999999996},"min":{"1min":0.45500000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.0329999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76500000000000001,"15min":0.78000000000000003},"min":{"1min":0.624,"5min":0.5,"15min":0.5},"max":{"1min":1.0700000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.80500000000000005}]}]},{"osd":5,"up_from":13,"seq":55834576009,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52972,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318868,"statfs":{"total":96636764160,"available":96582520832,"internally_reserved":0,"allocated":3887104,"data_stored":2283701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.82699999999999996,"5min":0.98699999999999999,"15min":1.004},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.68999999999999995,"15min":0.71999999999999997},"min":{"1min":0.48399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.028,"5min":1.085,"15min":1.085},"last":0.5}]},{"osd":1,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.68100000000000005,"15min":0.71499999999999997},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84399999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70999999999999996,"15min":0.72499999999999998},"min":{"1min":0.52300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88500000000000001,"5min":0.95299999999999996,"15min":1.052},"last":0.53900000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.71599999999999997,"15min":0.73799999999999999},"min":{"1min":0.47199999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97299999999999998,"5min":1.004,"15min":1.014},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.69499999999999995,"15min":0.73699999999999999},"min":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.81799999999999995,"5min":1.018,"15min":1.0680000000000001},"last":0.52700000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.747,"15min":0.77700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.88500000000000001,"5min":0.98899999999999999,"15min":1.083},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77000000000000002,"15min":0.79000000000000004},"min":{"1min":0.58399999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1020000000000001},"last":0.50600000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.40699999999999997,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.054},"last":0.441},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":0.98799999999999999,"5min":1.145,"15min":1.145},"last":0.46000000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67900000000000005,"15min":0.67300000000000004},"min":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96799999999999997,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71399999999999997,"15min":0.71099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.94899999999999995,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.48599999999999999}]},{"osd":7,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73999999999999999,"15min":0.73799999999999999},"min":{"1min":0.46500000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.0820000000000001,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76300000000000001,"15min":0.76600000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1519999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:14:39.210 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:14:39.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.442+0000 7f2594800700 1 Processor -- start 2024-04-02T09:14:39.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.442+0000 7f2594800700 1 -- start start 2024-04-02T09:14:39.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.443+0000 7f2594800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f258c161ca0 0x7f258c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:39.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.443+0000 7f2594800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f258c0ceda0 con 0x7f258c0d18b0 2024-04-02T09:14:39.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.443+0000 7f2594800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f258c1612f0 con 0x7f258c160f10 2024-04-02T09:14:39.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.443+0000 7f2594800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258c0d1d30 con 0x7f258c161ca0 2024-04-02T09:14:39.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.444+0000 7f2591400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f258c0d18b0 0x7f258c161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38660/0 (socket says 172.21.15.89:38660) 2024-04-02T09:14:39.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.444+0000 7f2591400700 1 -- 172.21.15.89:0/370175171 learned_addr learned my addr 172.21.15.89:0/370175171 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:39.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.444+0000 7f2592800700 1 --2- 172.21.15.89:0/370175171 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f258c161ca0 0x7f258c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:39.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.444+0000 7f2590a00700 1 -- 172.21.15.89:0/370175171 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466755129 0 0) 0x7f258c0ceda0 con 0x7f258c0d18b0 2024-04-02T09:14:39.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.444+0000 7f2590a00700 1 -- 172.21.15.89:0/370175171 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25740035b0 con 0x7f258c0d18b0 2024-04-02T09:14:39.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.444+0000 7f2590a00700 1 -- 172.21.15.89:0/370175171 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1286462441 0 0) 0x7f25740035b0 con 0x7f258c0d18b0 2024-04-02T09:14:39.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.444+0000 7f2590a00700 1 -- 172.21.15.89:0/370175171 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f258c161ca0 msgr2=0x7f258c0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:39.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.444+0000 7f2590a00700 1 --2- 172.21.15.89:0/370175171 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f258c161ca0 0x7f258c0cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.444+0000 7f2590a00700 1 -- 172.21.15.89:0/370175171 >> v1:172.21.15.113:6789/0 conn(0x7f258c160f10 legacy=0x7f258c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:39.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.444+0000 7f2590a00700 1 -- 172.21.15.89:0/370175171 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258c0ce470 con 0x7f258c0d18b0 2024-04-02T09:14:39.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.444+0000 7f2590a00700 1 -- 172.21.15.89:0/370175171 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f257c001670 con 0x7f258c0d18b0 2024-04-02T09:14:39.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.444+0000 7f2590a00700 1 -- 172.21.15.89:0/370175171 >> v1:172.21.15.89:6789/0 conn(0x7f258c0d18b0 legacy=0x7f258c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:39.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.445+0000 7f2590a00700 1 --2- 172.21.15.89:0/370175171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2574003a30 0x7f2574003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:39.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.445+0000 7f2590a00700 1 --2- 172.21.15.89:0/370175171 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f258c161ca0 0x7f2574004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:39.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.445+0000 7f2590a00700 1 --2- 172.21.15.89:0/370175171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2574004d90 0x7f2574005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:39.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.445+0000 7f2590a00700 1 -- 172.21.15.89:0/370175171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258c0d1d30 con 0x7f2574004d90 2024-04-02T09:14:39.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.445+0000 7f2590a00700 1 -- 172.21.15.89:0/370175171 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257400bb20 con 0x7f2574003a30 2024-04-02T09:14:39.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.445+0000 7f2590a00700 1 -- 172.21.15.89:0/370175171 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257400bce0 con 0x7f258c161ca0 2024-04-02T09:14:39.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.445+0000 7f2591400700 1 --2- 172.21.15.89:0/370175171 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f258c161ca0 0x7f2574004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:39.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.445+0000 7f2592800700 1 --2- 172.21.15.89:0/370175171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2574004d90 0x7f2574005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:39.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.445+0000 7f2591e00700 1 --2- 172.21.15.89:0/370175171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2574003a30 0x7f2574003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:39.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.445+0000 7f2592800700 1 -- 172.21.15.89:0/370175171 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f258c161ca0 msgr2=0x7f2574004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:39.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.445+0000 7f2592800700 1 --2- 172.21.15.89:0/370175171 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f258c161ca0 0x7f2574004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.445+0000 7f2592800700 1 -- 172.21.15.89:0/370175171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2574003a30 msgr2=0x7f2574003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:39.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.445+0000 7f2592800700 1 --2- 172.21.15.89:0/370175171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2574003a30 0x7f2574003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.445+0000 7f2592800700 1 -- 172.21.15.89:0/370175171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25740035b0 con 0x7f2574004d90 2024-04-02T09:14:39.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.446+0000 7f2592800700 1 --2- 172.21.15.89:0/370175171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2574004d90 0x7f2574005210 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7f2588009fc0 tx=0x7f2574005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb37628775897ed server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:39.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.446+0000 7f2590a00700 1 -- 172.21.15.89:0/370175171 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258800ec30 con 0x7f2574004d90 2024-04-02T09:14:39.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.446+0000 7f2590a00700 1 -- 172.21.15.89:0/370175171 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258800f040 con 0x7f2574004d90 2024-04-02T09:14:39.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.446+0000 7f2590a00700 1 -- 172.21.15.89:0/370175171 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2588014480 con 0x7f2574004d90 2024-04-02T09:14:39.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.446+0000 7f2594800700 1 -- 172.21.15.89:0/370175171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2574004d90 msgr2=0x7f2574005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:39.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.446+0000 7f2594800700 1 --2- 172.21.15.89:0/370175171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2574004d90 0x7f2574005210 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7f2588009fc0 tx=0x7f2574005880 comp rx=0 tx=0).stop 2024-04-02T09:14:39.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.446+0000 7f2592800700 1 -- 172.21.15.89:0/370175171 reap_dead start 2024-04-02T09:14:39.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.446+0000 7f2594800700 1 -- 172.21.15.89:0/370175171 shutdown_connections 2024-04-02T09:14:39.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.446+0000 7f2594800700 1 -- 172.21.15.89:0/370175171 >> 172.21.15.89:0/370175171 conn(0x7f258c08aff0 msgr2=0x7f258c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:39.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.446+0000 7f2594800700 1 -- 172.21.15.89:0/370175171 shutdown_connections 2024-04-02T09:14:39.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.446+0000 7f2594800700 1 -- 172.21.15.89:0/370175171 wait complete. 2024-04-02T09:14:39.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.448+0000 7f2594800700 1 Processor -- start 2024-04-02T09:14:39.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.448+0000 7f2594800700 1 -- start start 2024-04-02T09:14:39.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.449+0000 7f2594800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2574004d90 0x7f258c15ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:39.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.449+0000 7f2594800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f258c0b4d80 0x7f258c0c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:39.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.449+0000 7f2594800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f258c15f8f0 0x7f258c0ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:39.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.449+0000 7f2594800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258c148d90 con 0x7f258c15f8f0 2024-04-02T09:14:39.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.449+0000 7f2594800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258c148180 con 0x7f2574004d90 2024-04-02T09:14:39.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.449+0000 7f2594800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258c148460 con 0x7f258c0b4d80 2024-04-02T09:14:39.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.449+0000 7f2592800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f258c15f8f0 0x7f258c0ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:39.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.449+0000 7f2592800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f258c15f8f0 0x7f258c0ccac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35100/0 (socket says 172.21.15.89:35100) 2024-04-02T09:14:39.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.449+0000 7f2592800700 1 -- 172.21.15.89:0/3332054141 learned_addr learned my addr 172.21.15.89:0/3332054141 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:39.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.449+0000 7f2591e00700 1 --2- 172.21.15.89:0/3332054141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2574004d90 0x7f258c15ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:39.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.449+0000 7f2591400700 1 --2- 172.21.15.89:0/3332054141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f258c0b4d80 0x7f258c0c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.450+0000 7f2592800700 1 -- 172.21.15.89:0/3332054141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f258c0b4d80 msgr2=0x7f258c0c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.450+0000 7f2592800700 1 --2- 172.21.15.89:0/3332054141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f258c0b4d80 0x7f258c0c91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.450+0000 7f2592800700 1 -- 172.21.15.89:0/3332054141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2574004d90 msgr2=0x7f258c15ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.450+0000 7f2592800700 1 --2- 172.21.15.89:0/3332054141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2574004d90 0x7f258c15ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.450+0000 7f2592800700 1 -- 172.21.15.89:0/3332054141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258c148740 con 0x7f258c15f8f0 2024-04-02T09:14:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.450+0000 7f2592800700 1 --2- 172.21.15.89:0/3332054141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f258c15f8f0 0x7f258c0ccac0 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7f2588017040 tx=0x7f2588009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.450+0000 7f2586000700 1 -- 172.21.15.89:0/3332054141 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2588007000 con 0x7f258c15f8f0 2024-04-02T09:14:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.450+0000 7f2586000700 1 -- 172.21.15.89:0/3332054141 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2588003e50 con 0x7f258c15f8f0 2024-04-02T09:14:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.450+0000 7f2594800700 1 -- 172.21.15.89:0/3332054141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258c148a20 con 0x7f258c15f8f0 2024-04-02T09:14:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.450+0000 7f2586000700 1 -- 172.21.15.89:0/3332054141 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25880149f0 con 0x7f258c15f8f0 2024-04-02T09:14:39.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.450+0000 7f2594800700 1 -- 172.21.15.89:0/3332054141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f258c15ff50 con 0x7f258c15f8f0 2024-04-02T09:14:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.451+0000 7f2594800700 1 -- 172.21.15.89:0/3332054141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f258c0b5160 con 0x7f258c15f8f0 2024-04-02T09:14:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.452+0000 7f2586000700 1 -- 172.21.15.89:0/3332054141 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f2588023020 con 0x7f258c15f8f0 2024-04-02T09:14:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.453+0000 7f2586000700 1 --2- 172.21.15.89:0/3332054141 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f257006aed0 0x7f257006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:39.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.453+0000 7f2586000700 1 -- 172.21.15.89:0/3332054141 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f258808ad30 con 0x7f258c15f8f0 2024-04-02T09:14:39.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.453+0000 7f2591e00700 1 --2- 172.21.15.89:0/3332054141 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f257006aed0 0x7f257006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:39.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.454+0000 7f2591e00700 1 --2- 172.21.15.89:0/3332054141 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f257006aed0 0x7f257006d390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2580000c00 tx=0x7f2580009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.458+0000 7f2586000700 1 -- 172.21.15.89:0/3332054141 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f258c0b5160 con 0x7f258c15f8f0 2024-04-02T09:14:39.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.656+0000 7f2594800700 1 -- 172.21.15.89:0/3332054141 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f258c0ce120 con 0x7f257006aed0 2024-04-02T09:14:39.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.665+0000 7f2586000700 1 -- 172.21.15.89:0/3332054141 <== mgr.11320 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289387 (secure 0 0 0) 0x7f2580006860 con 0x7f257006aed0 2024-04-02T09:14:39.667 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.669+0000 7f2594800700 1 -- 172.21.15.89:0/3332054141 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f257006aed0 msgr2=0x7f257006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.669+0000 7f2594800700 1 --2- 172.21.15.89:0/3332054141 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f257006aed0 0x7f257006d390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2580000c00 tx=0x7f2580009040 comp rx=0 tx=0).stop 2024-04-02T09:14:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.669+0000 7f2594800700 1 -- 172.21.15.89:0/3332054141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f258c15f8f0 msgr2=0x7f258c0ccac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.669+0000 7f2594800700 1 --2- 172.21.15.89:0/3332054141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f258c15f8f0 0x7f258c0ccac0 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7f2588017040 tx=0x7f2588009750 comp rx=0 tx=0).stop 2024-04-02T09:14:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.669+0000 7f2594800700 1 -- 172.21.15.89:0/3332054141 shutdown_connections 2024-04-02T09:14:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.670+0000 7f2594800700 1 --2- 172.21.15.89:0/3332054141 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f257006aed0 0x7f257006d390 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.670+0000 7f2594800700 1 --2- 172.21.15.89:0/3332054141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2574004d90 0x7f258c15ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.670+0000 7f2594800700 1 --2- 172.21.15.89:0/3332054141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f258c15f8f0 0x7f258c0ccac0 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.670+0000 7f2594800700 1 --2- 172.21.15.89:0/3332054141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f258c0b4d80 0x7f258c0c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.670+0000 7f2594800700 1 -- 172.21.15.89:0/3332054141 >> 172.21.15.89:0/3332054141 conn(0x7f258c08aff0 msgr2=0x7f258c0cae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:39.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.670+0000 7f2594800700 1 -- 172.21.15.89:0/3332054141 shutdown_connections 2024-04-02T09:14:39.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.670+0000 7f2594800700 1 -- 172.21.15.89:0/3332054141 wait complete. 2024-04-02T09:14:39.670 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:14:39.686 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-04-02T09:14:39.258766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":683,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":419072,"kb_used_data":27360,"kb_used_omap":12,"kb_used_meta":391539,"kb_avail":754555648,"statfs":{"total":773094113280,"available":772664983552,"internally_reserved":0,"allocated":28016640,"data_stored":14953910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":400936504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001174"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523216+0000","last_change":"2024-04-02T09:14:34.377961+0000","last_active":"2024-04-02T09:14:35.523216+0000","last_peered":"2024-04-02T09:14:35.523216+0000","last_clean":"2024-04-02T09:14:35.523216+0000","last_became_active":"2024-04-02T09:14:34.377267+0000","last_became_peered":"2024-04-02T09:14:34.377267+0000","last_unstale":"2024-04-02T09:14:35.523216+0000","last_undegraded":"2024-04-02T09:14:35.523216+0000","last_fullsized":"2024-04-02T09:14:35.523216+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:44:06.694478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364561+0000","last_change":"2024-04-02T09:14:34.373205+0000","last_active":"2024-04-02T09:14:35.364561+0000","last_peered":"2024-04-02T09:14:35.364561+0000","last_clean":"2024-04-02T09:14:35.364561+0000","last_became_active":"2024-04-02T09:14:34.370571+0000","last_became_peered":"2024-04-02T09:14:34.370571+0000","last_unstale":"2024-04-02T09:14:35.364561+0000","last_undegraded":"2024-04-02T09:14:35.364561+0000","last_fullsized":"2024-04-02T09:14:35.364561+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:55:48.430034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363862+0000","last_change":"2024-04-02T09:14:34.369016+0000","last_active":"2024-04-02T09:14:35.363862+0000","last_peered":"2024-04-02T09:14:35.363862+0000","last_clean":"2024-04-02T09:14:35.363862+0000","last_became_active":"2024-04-02T09:14:34.368553+0000","last_became_peered":"2024-04-02T09:14:34.368553+0000","last_unstale":"2024-04-02T09:14:35.363862+0000","last_undegraded":"2024-04-02T09:14:35.363862+0000","last_fullsized":"2024-04-02T09:14:35.363862+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:12.477806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522578+0000","last_change":"2024-04-02T09:14:34.383881+0000","last_active":"2024-04-02T09:14:35.522578+0000","last_peered":"2024-04-02T09:14:35.522578+0000","last_clean":"2024-04-02T09:14:35.522578+0000","last_became_active":"2024-04-02T09:14:34.383189+0000","last_became_peered":"2024-04-02T09:14:34.383189+0000","last_unstale":"2024-04-02T09:14:35.522578+0000","last_undegraded":"2024-04-02T09:14:35.522578+0000","last_fullsized":"2024-04-02T09:14:35.522578+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:49:07.747576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522857+0000","last_change":"2024-04-02T09:14:34.384650+0000","last_active":"2024-04-02T09:14:35.522857+0000","last_peered":"2024-04-02T09:14:35.522857+0000","last_clean":"2024-04-02T09:14:35.522857+0000","last_became_active":"2024-04-02T09:14:34.384353+0000","last_became_peered":"2024-04-02T09:14:34.384353+0000","last_unstale":"2024-04-02T09:14:35.522857+0000","last_undegraded":"2024-04-02T09:14:35.522857+0000","last_fullsized":"2024-04-02T09:14:35.522857+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:36.082626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364110+0000","last_change":"2024-04-02T09:14:34.367970+0000","last_active":"2024-04-02T09:14:35.364110+0000","last_peered":"2024-04-02T09:14:35.364110+0000","last_clean":"2024-04-02T09:14:35.364110+0000","last_became_active":"2024-04-02T09:14:34.365750+0000","last_became_peered":"2024-04-02T09:14:34.365750+0000","last_unstale":"2024-04-02T09:14:35.364110+0000","last_undegraded":"2024-04-02T09:14:35.364110+0000","last_fullsized":"2024-04-02T09:14:35.364110+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:32.796452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362566+0000","last_change":"2024-04-02T09:14:34.381480+0000","last_active":"2024-04-02T09:14:35.362566+0000","last_peered":"2024-04-02T09:14:35.362566+0000","last_clean":"2024-04-02T09:14:35.362566+0000","last_became_active":"2024-04-02T09:14:34.381243+0000","last_became_peered":"2024-04-02T09:14:34.381243+0000","last_unstale":"2024-04-02T09:14:35.362566+0000","last_undegraded":"2024-04-02T09:14:35.362566+0000","last_fullsized":"2024-04-02T09:14:35.362566+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:43:52.860015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362432+0000","last_change":"2024-04-02T09:14:34.390416+0000","last_active":"2024-04-02T09:14:35.362432+0000","last_peered":"2024-04-02T09:14:35.362432+0000","last_clean":"2024-04-02T09:14:35.362432+0000","last_became_active":"2024-04-02T09:14:34.390126+0000","last_became_peered":"2024-04-02T09:14:34.390126+0000","last_unstale":"2024-04-02T09:14:35.362432+0000","last_undegraded":"2024-04-02T09:14:35.362432+0000","last_fullsized":"2024-04-02T09:14:35.362432+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:05:04.134051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364695+0000","last_change":"2024-04-02T09:14:34.362861+0000","last_active":"2024-04-02T09:14:35.364695+0000","last_peered":"2024-04-02T09:14:35.364695+0000","last_clean":"2024-04-02T09:14:35.364695+0000","last_became_active":"2024-04-02T09:14:34.362580+0000","last_became_peered":"2024-04-02T09:14:34.362580+0000","last_unstale":"2024-04-02T09:14:35.364695+0000","last_undegraded":"2024-04-02T09:14:35.364695+0000","last_fullsized":"2024-04-02T09:14:35.364695+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:12.477806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362736+0000","last_change":"2024-04-02T09:14:34.380960+0000","last_active":"2024-04-02T09:14:35.362736+0000","last_peered":"2024-04-02T09:14:35.362736+0000","last_clean":"2024-04-02T09:14:35.362736+0000","last_became_active":"2024-04-02T09:14:34.380726+0000","last_became_peered":"2024-04-02T09:14:34.380726+0000","last_unstale":"2024-04-02T09:14:35.362736+0000","last_undegraded":"2024-04-02T09:14:35.362736+0000","last_fullsized":"2024-04-02T09:14:35.362736+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:51:23.449925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522728+0000","last_change":"2024-04-02T09:14:34.382824+0000","last_active":"2024-04-02T09:14:35.522728+0000","last_peered":"2024-04-02T09:14:35.522728+0000","last_clean":"2024-04-02T09:14:35.522728+0000","last_became_active":"2024-04-02T09:14:34.382044+0000","last_became_peered":"2024-04-02T09:14:34.382044+0000","last_unstale":"2024-04-02T09:14:35.522728+0000","last_undegraded":"2024-04-02T09:14:35.522728+0000","last_fullsized":"2024-04-02T09:14:35.522728+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:56.492180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364322+0000","last_change":"2024-04-02T09:14:34.366396+0000","last_active":"2024-04-02T09:14:35.364322+0000","last_peered":"2024-04-02T09:14:35.364322+0000","last_clean":"2024-04-02T09:14:35.364322+0000","last_became_active":"2024-04-02T09:14:34.366115+0000","last_became_peered":"2024-04-02T09:14:34.366115+0000","last_unstale":"2024-04-02T09:14:35.364322+0000","last_undegraded":"2024-04-02T09:14:35.364322+0000","last_fullsized":"2024-04-02T09:14:35.364322+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:56:03.446485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523710+0000","last_change":"2024-04-02T09:14:34.374183+0000","last_active":"2024-04-02T09:14:35.523710+0000","last_peered":"2024-04-02T09:14:35.523710+0000","last_clean":"2024-04-02T09:14:35.523710+0000","last_became_active":"2024-04-02T09:14:34.371570+0000","last_became_peered":"2024-04-02T09:14:34.371570+0000","last_unstale":"2024-04-02T09:14:35.523710+0000","last_undegraded":"2024-04-02T09:14:35.523710+0000","last_fullsized":"2024-04-02T09:14:35.523710+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:06:08.871157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522853+0000","last_change":"2024-04-02T09:14:34.385243+0000","last_active":"2024-04-02T09:14:35.522853+0000","last_peered":"2024-04-02T09:14:35.522853+0000","last_clean":"2024-04-02T09:14:35.522853+0000","last_became_active":"2024-04-02T09:14:34.384989+0000","last_became_peered":"2024-04-02T09:14:34.384989+0000","last_unstale":"2024-04-02T09:14:35.522853+0000","last_undegraded":"2024-04-02T09:14:35.522853+0000","last_fullsized":"2024-04-02T09:14:35.522853+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:28.009617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362467+0000","last_change":"2024-04-02T09:14:34.379614+0000","last_active":"2024-04-02T09:14:35.362467+0000","last_peered":"2024-04-02T09:14:35.362467+0000","last_clean":"2024-04-02T09:14:35.362467+0000","last_became_active":"2024-04-02T09:14:34.378063+0000","last_became_peered":"2024-04-02T09:14:34.378063+0000","last_unstale":"2024-04-02T09:14:35.362467+0000","last_undegraded":"2024-04-02T09:14:35.362467+0000","last_fullsized":"2024-04-02T09:14:35.362467+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:07.213744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522170+0000","last_change":"2024-04-02T09:14:34.373018+0000","last_active":"2024-04-02T09:14:35.522170+0000","last_peered":"2024-04-02T09:14:35.522170+0000","last_clean":"2024-04-02T09:14:35.522170+0000","last_became_active":"2024-04-02T09:14:34.372010+0000","last_became_peered":"2024-04-02T09:14:34.372010+0000","last_unstale":"2024-04-02T09:14:35.522170+0000","last_undegraded":"2024-04-02T09:14:35.522170+0000","last_fullsized":"2024-04-02T09:14:35.522170+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:58.362648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523162+0000","last_change":"2024-04-02T09:14:34.391051+0000","last_active":"2024-04-02T09:14:35.523162+0000","last_peered":"2024-04-02T09:14:35.523162+0000","last_clean":"2024-04-02T09:14:35.523162+0000","last_became_active":"2024-04-02T09:14:34.390721+0000","last_became_peered":"2024-04-02T09:14:34.390721+0000","last_unstale":"2024-04-02T09:14:35.523162+0000","last_undegraded":"2024-04-02T09:14:35.523162+0000","last_fullsized":"2024-04-02T09:14:35.523162+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:53:01.672940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362877+0000","last_change":"2024-04-02T09:14:34.383335+0000","last_active":"2024-04-02T09:14:35.362877+0000","last_peered":"2024-04-02T09:14:35.362877+0000","last_clean":"2024-04-02T09:14:35.362877+0000","last_became_active":"2024-04-02T09:14:34.383105+0000","last_became_peered":"2024-04-02T09:14:34.383105+0000","last_unstale":"2024-04-02T09:14:35.362877+0000","last_undegraded":"2024-04-02T09:14:35.362877+0000","last_fullsized":"2024-04-02T09:14:35.362877+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:08:25.093735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523794+0000","last_change":"2024-04-02T09:14:34.385798+0000","last_active":"2024-04-02T09:14:35.523794+0000","last_peered":"2024-04-02T09:14:35.523794+0000","last_clean":"2024-04-02T09:14:35.523794+0000","last_became_active":"2024-04-02T09:14:34.385264+0000","last_became_peered":"2024-04-02T09:14:34.385264+0000","last_unstale":"2024-04-02T09:14:35.523794+0000","last_undegraded":"2024-04-02T09:14:35.523794+0000","last_fullsized":"2024-04-02T09:14:35.523794+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:41.722545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362562+0000","last_change":"2024-04-02T09:14:34.384422+0000","last_active":"2024-04-02T09:14:35.362562+0000","last_peered":"2024-04-02T09:14:35.362562+0000","last_clean":"2024-04-02T09:14:35.362562+0000","last_became_active":"2024-04-02T09:14:34.384187+0000","last_became_peered":"2024-04-02T09:14:34.384187+0000","last_unstale":"2024-04-02T09:14:35.362562+0000","last_undegraded":"2024-04-02T09:14:35.362562+0000","last_fullsized":"2024-04-02T09:14:35.362562+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:45.536966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523625+0000","last_change":"2024-04-02T09:14:34.375508+0000","last_active":"2024-04-02T09:14:35.523625+0000","last_peered":"2024-04-02T09:14:35.523625+0000","last_clean":"2024-04-02T09:14:35.523625+0000","last_became_active":"2024-04-02T09:14:34.374458+0000","last_became_peered":"2024-04-02T09:14:34.374458+0000","last_unstale":"2024-04-02T09:14:35.523625+0000","last_undegraded":"2024-04-02T09:14:35.523625+0000","last_fullsized":"2024-04-02T09:14:35.523625+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:23.023963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522793+0000","last_change":"2024-04-02T09:14:34.371948+0000","last_active":"2024-04-02T09:14:35.522793+0000","last_peered":"2024-04-02T09:14:35.522793+0000","last_clean":"2024-04-02T09:14:35.522793+0000","last_became_active":"2024-04-02T09:14:34.371586+0000","last_became_peered":"2024-04-02T09:14:34.371586+0000","last_unstale":"2024-04-02T09:14:35.522793+0000","last_undegraded":"2024-04-02T09:14:35.522793+0000","last_fullsized":"2024-04-02T09:14:35.522793+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:27.813013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522861+0000","last_change":"2024-04-02T09:14:34.372317+0000","last_active":"2024-04-02T09:14:35.522861+0000","last_peered":"2024-04-02T09:14:35.522861+0000","last_clean":"2024-04-02T09:14:35.522861+0000","last_became_active":"2024-04-02T09:14:34.371999+0000","last_became_peered":"2024-04-02T09:14:34.371999+0000","last_unstale":"2024-04-02T09:14:35.522861+0000","last_undegraded":"2024-04-02T09:14:35.522861+0000","last_fullsized":"2024-04-02T09:14:35.522861+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:49.154841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363323+0000","last_change":"2024-04-02T09:14:34.369575+0000","last_active":"2024-04-02T09:14:35.363323+0000","last_peered":"2024-04-02T09:14:35.363323+0000","last_clean":"2024-04-02T09:14:35.363323+0000","last_became_active":"2024-04-02T09:14:34.369273+0000","last_became_peered":"2024-04-02T09:14:34.369273+0000","last_unstale":"2024-04-02T09:14:35.363323+0000","last_undegraded":"2024-04-02T09:14:35.363323+0000","last_fullsized":"2024-04-02T09:14:35.363323+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:12:47.137280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523287+0000","last_change":"2024-04-02T09:14:34.372802+0000","last_active":"2024-04-02T09:14:35.523287+0000","last_peered":"2024-04-02T09:14:35.523287+0000","last_clean":"2024-04-02T09:14:35.523287+0000","last_became_active":"2024-04-02T09:14:34.372511+0000","last_became_peered":"2024-04-02T09:14:34.372511+0000","last_unstale":"2024-04-02T09:14:35.523287+0000","last_undegraded":"2024-04-02T09:14:35.523287+0000","last_fullsized":"2024-04-02T09:14:35.523287+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:46.642400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363775+0000","last_change":"2024-04-02T09:14:34.370273+0000","last_active":"2024-04-02T09:14:35.363775+0000","last_peered":"2024-04-02T09:14:35.363775+0000","last_clean":"2024-04-02T09:14:35.363775+0000","last_became_active":"2024-04-02T09:14:34.369221+0000","last_became_peered":"2024-04-02T09:14:34.369221+0000","last_unstale":"2024-04-02T09:14:35.363775+0000","last_undegraded":"2024-04-02T09:14:35.363775+0000","last_fullsized":"2024-04-02T09:14:35.363775+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:32.796452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523112+0000","last_change":"2024-04-02T09:14:34.381402+0000","last_active":"2024-04-02T09:14:35.523112+0000","last_peered":"2024-04-02T09:14:35.523112+0000","last_clean":"2024-04-02T09:14:35.523112+0000","last_became_active":"2024-04-02T09:14:34.381063+0000","last_became_peered":"2024-04-02T09:14:34.381063+0000","last_unstale":"2024-04-02T09:14:35.523112+0000","last_undegraded":"2024-04-02T09:14:35.523112+0000","last_fullsized":"2024-04-02T09:14:35.523112+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:24:11.707496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522665+0000","last_change":"2024-04-02T09:14:34.383686+0000","last_active":"2024-04-02T09:14:35.522665+0000","last_peered":"2024-04-02T09:14:35.522665+0000","last_clean":"2024-04-02T09:14:35.522665+0000","last_became_active":"2024-04-02T09:14:34.382500+0000","last_became_peered":"2024-04-02T09:14:34.382500+0000","last_unstale":"2024-04-02T09:14:35.522665+0000","last_undegraded":"2024-04-02T09:14:35.522665+0000","last_fullsized":"2024-04-02T09:14:35.522665+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:35:17.449940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362773+0000","last_change":"2024-04-02T09:14:34.387782+0000","last_active":"2024-04-02T09:14:35.362773+0000","last_peered":"2024-04-02T09:14:35.362773+0000","last_clean":"2024-04-02T09:14:35.362773+0000","last_became_active":"2024-04-02T09:14:34.383376+0000","last_became_peered":"2024-04-02T09:14:34.383376+0000","last_unstale":"2024-04-02T09:14:35.362773+0000","last_undegraded":"2024-04-02T09:14:35.362773+0000","last_fullsized":"2024-04-02T09:14:35.362773+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:28:13.971403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522423+0000","last_change":"2024-04-02T09:14:34.374368+0000","last_active":"2024-04-02T09:14:35.522423+0000","last_peered":"2024-04-02T09:14:35.522423+0000","last_clean":"2024-04-02T09:14:35.522423+0000","last_became_active":"2024-04-02T09:14:34.374046+0000","last_became_peered":"2024-04-02T09:14:34.374046+0000","last_unstale":"2024-04-02T09:14:35.522423+0000","last_undegraded":"2024-04-02T09:14:35.522423+0000","last_fullsized":"2024-04-02T09:14:35.522423+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:37:45.647925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363599+0000","last_change":"2024-04-02T09:14:34.384135+0000","last_active":"2024-04-02T09:14:35.363599+0000","last_peered":"2024-04-02T09:14:35.363599+0000","last_clean":"2024-04-02T09:14:35.363599+0000","last_became_active":"2024-04-02T09:14:34.383849+0000","last_became_peered":"2024-04-02T09:14:34.383849+0000","last_unstale":"2024-04-02T09:14:35.363599+0000","last_undegraded":"2024-04-02T09:14:35.363599+0000","last_fullsized":"2024-04-02T09:14:35.363599+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:23:36.207832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363879+0000","last_change":"2024-04-02T09:14:34.362600+0000","last_active":"2024-04-02T09:14:35.363879+0000","last_peered":"2024-04-02T09:14:35.363879+0000","last_clean":"2024-04-02T09:14:35.363879+0000","last_became_active":"2024-04-02T09:14:34.362080+0000","last_became_peered":"2024-04-02T09:14:34.362080+0000","last_unstale":"2024-04-02T09:14:35.363879+0000","last_undegraded":"2024-04-02T09:14:35.363879+0000","last_fullsized":"2024-04-02T09:14:35.363879+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:41.728873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"264'4","reported_seq":15,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523394+0000","last_change":"2024-04-02T09:14:33.875502+0000","last_active":"2024-04-02T09:14:35.523394+0000","last_peered":"2024-04-02T09:14:35.523394+0000","last_clean":"2024-04-02T09:14:35.523394+0000","last_became_active":"2024-04-02T09:14:33.875191+0000","last_became_peered":"2024-04-02T09:14:33.875191+0000","last_unstale":"2024-04-02T09:14:35.523394+0000","last_undegraded":"2024-04-02T09:14:35.523394+0000","last_fullsized":"2024-04-02T09:14:35.523394+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:54.914511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523839+0000","last_change":"2024-04-02T09:14:34.372870+0000","last_active":"2024-04-02T09:14:35.523839+0000","last_peered":"2024-04-02T09:14:35.523839+0000","last_clean":"2024-04-02T09:14:35.523839+0000","last_became_active":"2024-04-02T09:14:34.372591+0000","last_became_peered":"2024-04-02T09:14:34.372591+0000","last_unstale":"2024-04-02T09:14:35.523839+0000","last_undegraded":"2024-04-02T09:14:35.523839+0000","last_fullsized":"2024-04-02T09:14:35.523839+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:21.044232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1e","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364686+0000","last_change":"2024-04-02T09:14:33.879018+0000","last_active":"2024-04-02T09:14:35.364686+0000","last_peered":"2024-04-02T09:14:35.364686+0000","last_clean":"2024-04-02T09:14:35.364686+0000","last_became_active":"2024-04-02T09:14:33.877939+0000","last_became_peered":"2024-04-02T09:14:33.877939+0000","last_unstale":"2024-04-02T09:14:35.364686+0000","last_undegraded":"2024-04-02T09:14:35.364686+0000","last_fullsized":"2024-04-02T09:14:35.364686+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:21:10.605008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363660+0000","last_change":"2024-04-02T09:14:34.381630+0000","last_active":"2024-04-02T09:14:35.363660+0000","last_peered":"2024-04-02T09:14:35.363660+0000","last_clean":"2024-04-02T09:14:35.363660+0000","last_became_active":"2024-04-02T09:14:34.381362+0000","last_became_peered":"2024-04-02T09:14:34.381362+0000","last_unstale":"2024-04-02T09:14:35.363660+0000","last_undegraded":"2024-04-02T09:14:35.363660+0000","last_fullsized":"2024-04-02T09:14:35.363660+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:17:46.076532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1d","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364496+0000","last_change":"2024-04-02T09:14:33.877689+0000","last_active":"2024-04-02T09:14:35.364496+0000","last_peered":"2024-04-02T09:14:35.364496+0000","last_clean":"2024-04-02T09:14:35.364496+0000","last_became_active":"2024-04-02T09:14:33.875355+0000","last_became_peered":"2024-04-02T09:14:33.875355+0000","last_unstale":"2024-04-02T09:14:35.364496+0000","last_undegraded":"2024-04-02T09:14:35.364496+0000","last_fullsized":"2024-04-02T09:14:35.364496+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:53:35.396685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364415+0000","last_change":"2024-04-02T09:14:34.384893+0000","last_active":"2024-04-02T09:14:35.364415+0000","last_peered":"2024-04-02T09:14:35.364415+0000","last_clean":"2024-04-02T09:14:35.364415+0000","last_became_active":"2024-04-02T09:14:34.384622+0000","last_became_peered":"2024-04-02T09:14:34.384622+0000","last_unstale":"2024-04-02T09:14:35.364415+0000","last_undegraded":"2024-04-02T09:14:35.364415+0000","last_fullsized":"2024-04-02T09:14:35.364415+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:13:28.694192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1c","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523485+0000","last_change":"2024-04-02T09:14:33.860753+0000","last_active":"2024-04-02T09:14:35.523485+0000","last_peered":"2024-04-02T09:14:35.523485+0000","last_clean":"2024-04-02T09:14:35.523485+0000","last_became_active":"2024-04-02T09:14:33.860419+0000","last_became_peered":"2024-04-02T09:14:33.860419+0000","last_unstale":"2024-04-02T09:14:35.523485+0000","last_undegraded":"2024-04-02T09:14:35.523485+0000","last_fullsized":"2024-04-02T09:14:35.523485+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:54.064730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364273+0000","last_change":"2024-04-02T09:14:34.373334+0000","last_active":"2024-04-02T09:14:35.364273+0000","last_peered":"2024-04-02T09:14:35.364273+0000","last_clean":"2024-04-02T09:14:35.364273+0000","last_became_active":"2024-04-02T09:14:34.372987+0000","last_became_peered":"2024-04-02T09:14:34.372987+0000","last_unstale":"2024-04-02T09:14:35.364273+0000","last_undegraded":"2024-04-02T09:14:35.364273+0000","last_fullsized":"2024-04-02T09:14:35.364273+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:42.139739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1b","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523580+0000","last_change":"2024-04-02T09:14:33.874800+0000","last_active":"2024-04-02T09:14:35.523580+0000","last_peered":"2024-04-02T09:14:35.523580+0000","last_clean":"2024-04-02T09:14:35.523580+0000","last_became_active":"2024-04-02T09:14:33.874504+0000","last_became_peered":"2024-04-02T09:14:33.874504+0000","last_unstale":"2024-04-02T09:14:35.523580+0000","last_undegraded":"2024-04-02T09:14:35.523580+0000","last_fullsized":"2024-04-02T09:14:35.523580+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:11.515065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523030+0000","last_change":"2024-04-02T09:14:34.362124+0000","last_active":"2024-04-02T09:14:35.523030+0000","last_peered":"2024-04-02T09:14:35.523030+0000","last_clean":"2024-04-02T09:14:35.523030+0000","last_became_active":"2024-04-02T09:14:34.361786+0000","last_became_peered":"2024-04-02T09:14:34.361786+0000","last_unstale":"2024-04-02T09:14:35.523030+0000","last_undegraded":"2024-04-02T09:14:35.523030+0000","last_fullsized":"2024-04-02T09:14:35.523030+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:42:12.604946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1a","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362815+0000","last_change":"2024-04-02T09:14:33.877648+0000","last_active":"2024-04-02T09:14:35.362815+0000","last_peered":"2024-04-02T09:14:35.362815+0000","last_clean":"2024-04-02T09:14:35.362815+0000","last_became_active":"2024-04-02T09:14:33.877308+0000","last_became_peered":"2024-04-02T09:14:33.877308+0000","last_unstale":"2024-04-02T09:14:35.362815+0000","last_undegraded":"2024-04-02T09:14:35.362815+0000","last_fullsized":"2024-04-02T09:14:35.362815+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:31.787918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363331+0000","last_change":"2024-04-02T09:14:34.378176+0000","last_active":"2024-04-02T09:14:35.363331+0000","last_peered":"2024-04-02T09:14:35.363331+0000","last_clean":"2024-04-02T09:14:35.363331+0000","last_became_active":"2024-04-02T09:14:34.376201+0000","last_became_peered":"2024-04-02T09:14:34.376201+0000","last_unstale":"2024-04-02T09:14:35.363331+0000","last_undegraded":"2024-04-02T09:14:35.363331+0000","last_fullsized":"2024-04-02T09:14:35.363331+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:55.923045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522590+0000","last_change":"2024-04-02T09:14:33.869026+0000","last_active":"2024-04-02T09:14:35.522590+0000","last_peered":"2024-04-02T09:14:35.522590+0000","last_clean":"2024-04-02T09:14:35.522590+0000","last_became_active":"2024-04-02T09:14:33.868743+0000","last_became_peered":"2024-04-02T09:14:33.868743+0000","last_unstale":"2024-04-02T09:14:35.522590+0000","last_undegraded":"2024-04-02T09:14:35.522590+0000","last_fullsized":"2024-04-02T09:14:35.522590+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:31.787918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522526+0000","last_change":"2024-04-02T09:14:34.376819+0000","last_active":"2024-04-02T09:14:35.522526+0000","last_peered":"2024-04-02T09:14:35.522526+0000","last_clean":"2024-04-02T09:14:35.522526+0000","last_became_active":"2024-04-02T09:14:34.376555+0000","last_became_peered":"2024-04-02T09:14:34.376555+0000","last_unstale":"2024-04-02T09:14:35.522526+0000","last_undegraded":"2024-04-02T09:14:35.522526+0000","last_fullsized":"2024-04-02T09:14:35.522526+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:36:25.999949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363666+0000","last_change":"2024-04-02T09:14:33.879857+0000","last_active":"2024-04-02T09:14:35.363666+0000","last_peered":"2024-04-02T09:14:35.363666+0000","last_clean":"2024-04-02T09:14:35.363666+0000","last_became_active":"2024-04-02T09:14:33.879161+0000","last_became_peered":"2024-04-02T09:14:33.879161+0000","last_unstale":"2024-04-02T09:14:35.363666+0000","last_undegraded":"2024-04-02T09:14:35.363666+0000","last_fullsized":"2024-04-02T09:14:35.363666+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:28:14.207222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362885+0000","last_change":"2024-04-02T09:14:34.389470+0000","last_active":"2024-04-02T09:14:35.362885+0000","last_peered":"2024-04-02T09:14:35.362885+0000","last_clean":"2024-04-02T09:14:35.362885+0000","last_became_active":"2024-04-02T09:14:34.388852+0000","last_became_peered":"2024-04-02T09:14:34.388852+0000","last_unstale":"2024-04-02T09:14:35.362885+0000","last_undegraded":"2024-04-02T09:14:35.362885+0000","last_fullsized":"2024-04-02T09:14:35.362885+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:15.565863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523327+0000","last_change":"2024-04-02T09:14:33.869704+0000","last_active":"2024-04-02T09:14:35.523327+0000","last_peered":"2024-04-02T09:14:35.523327+0000","last_clean":"2024-04-02T09:14:35.523327+0000","last_became_active":"2024-04-02T09:14:33.868809+0000","last_became_peered":"2024-04-02T09:14:33.868809+0000","last_unstale":"2024-04-02T09:14:35.523327+0000","last_undegraded":"2024-04-02T09:14:35.523327+0000","last_fullsized":"2024-04-02T09:14:35.523327+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:12:46.128746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523864+0000","last_change":"2024-04-02T09:14:34.379506+0000","last_active":"2024-04-02T09:14:35.523864+0000","last_peered":"2024-04-02T09:14:35.523864+0000","last_clean":"2024-04-02T09:14:35.523864+0000","last_became_active":"2024-04-02T09:14:34.378787+0000","last_became_peered":"2024-04-02T09:14:34.378787+0000","last_unstale":"2024-04-02T09:14:35.523864+0000","last_undegraded":"2024-04-02T09:14:35.523864+0000","last_fullsized":"2024-04-02T09:14:35.523864+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:13:47.277991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523512+0000","last_change":"2024-04-02T09:14:33.874472+0000","last_active":"2024-04-02T09:14:35.523512+0000","last_peered":"2024-04-02T09:14:35.523512+0000","last_clean":"2024-04-02T09:14:35.523512+0000","last_became_active":"2024-04-02T09:14:33.874190+0000","last_became_peered":"2024-04-02T09:14:33.874190+0000","last_unstale":"2024-04-02T09:14:35.523512+0000","last_undegraded":"2024-04-02T09:14:35.523512+0000","last_fullsized":"2024-04-02T09:14:35.523512+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:06:40.311963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522771+0000","last_change":"2024-04-02T09:14:34.382684+0000","last_active":"2024-04-02T09:14:35.522771+0000","last_peered":"2024-04-02T09:14:35.522771+0000","last_clean":"2024-04-02T09:14:35.522771+0000","last_became_active":"2024-04-02T09:14:34.380586+0000","last_became_peered":"2024-04-02T09:14:34.380586+0000","last_unstale":"2024-04-02T09:14:35.522771+0000","last_undegraded":"2024-04-02T09:14:35.522771+0000","last_fullsized":"2024-04-02T09:14:35.522771+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:56:58.056857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.15","version":"264'3","reported_seq":14,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364540+0000","last_change":"2024-04-02T09:14:33.878269+0000","last_active":"2024-04-02T09:14:35.364540+0000","last_peered":"2024-04-02T09:14:35.364540+0000","last_clean":"2024-04-02T09:14:35.364540+0000","last_became_active":"2024-04-02T09:14:33.877496+0000","last_became_peered":"2024-04-02T09:14:33.877496+0000","last_unstale":"2024-04-02T09:14:35.364540+0000","last_undegraded":"2024-04-02T09:14:35.364540+0000","last_fullsized":"2024-04-02T09:14:35.364540+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:11:32.742344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523334+0000","last_change":"2024-04-02T09:14:34.372160+0000","last_active":"2024-04-02T09:14:35.523334+0000","last_peered":"2024-04-02T09:14:35.523334+0000","last_clean":"2024-04-02T09:14:35.523334+0000","last_became_active":"2024-04-02T09:14:34.371325+0000","last_became_peered":"2024-04-02T09:14:34.371325+0000","last_unstale":"2024-04-02T09:14:35.523334+0000","last_undegraded":"2024-04-02T09:14:35.523334+0000","last_fullsized":"2024-04-02T09:14:35.523334+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:36.082626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"264'6","reported_seq":17,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523450+0000","last_change":"2024-04-02T09:14:33.875419+0000","last_active":"2024-04-02T09:14:35.523450+0000","last_peered":"2024-04-02T09:14:35.523450+0000","last_clean":"2024-04-02T09:14:35.523450+0000","last_became_active":"2024-04-02T09:14:33.874433+0000","last_became_peered":"2024-04-02T09:14:33.874433+0000","last_unstale":"2024-04-02T09:14:35.523450+0000","last_undegraded":"2024-04-02T09:14:35.523450+0000","last_fullsized":"2024-04-02T09:14:35.523450+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:07:50.116863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522576+0000","last_change":"2024-04-02T09:14:34.374573+0000","last_active":"2024-04-02T09:14:35.522576+0000","last_peered":"2024-04-02T09:14:35.522576+0000","last_clean":"2024-04-02T09:14:35.522576+0000","last_became_active":"2024-04-02T09:14:34.374327+0000","last_became_peered":"2024-04-02T09:14:34.374327+0000","last_unstale":"2024-04-02T09:14:35.522576+0000","last_undegraded":"2024-04-02T09:14:35.522576+0000","last_fullsized":"2024-04-02T09:14:35.522576+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:31:00.788937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.13","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523486+0000","last_change":"2024-04-02T09:14:33.877528+0000","last_active":"2024-04-02T09:14:35.523486+0000","last_peered":"2024-04-02T09:14:35.523486+0000","last_clean":"2024-04-02T09:14:35.523486+0000","last_became_active":"2024-04-02T09:14:33.877262+0000","last_became_peered":"2024-04-02T09:14:33.877262+0000","last_unstale":"2024-04-02T09:14:35.523486+0000","last_undegraded":"2024-04-02T09:14:35.523486+0000","last_fullsized":"2024-04-02T09:14:35.523486+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:06:40.311963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522781+0000","last_change":"2024-04-02T09:14:34.380073+0000","last_active":"2024-04-02T09:14:35.522781+0000","last_peered":"2024-04-02T09:14:35.522781+0000","last_clean":"2024-04-02T09:14:35.522781+0000","last_became_active":"2024-04-02T09:14:34.379603+0000","last_became_peered":"2024-04-02T09:14:34.379603+0000","last_unstale":"2024-04-02T09:14:35.522781+0000","last_undegraded":"2024-04-02T09:14:35.522781+0000","last_fullsized":"2024-04-02T09:14:35.522781+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:45.536966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364291+0000","last_change":"2024-04-02T09:14:33.876709+0000","last_active":"2024-04-02T09:14:35.364291+0000","last_peered":"2024-04-02T09:14:35.364291+0000","last_clean":"2024-04-02T09:14:35.364291+0000","last_became_active":"2024-04-02T09:14:33.876371+0000","last_became_peered":"2024-04-02T09:14:33.876371+0000","last_unstale":"2024-04-02T09:14:35.364291+0000","last_undegraded":"2024-04-02T09:14:35.364291+0000","last_fullsized":"2024-04-02T09:14:35.364291+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:32:30.574385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522551+0000","last_change":"2024-04-02T09:14:34.381595+0000","last_active":"2024-04-02T09:14:35.522551+0000","last_peered":"2024-04-02T09:14:35.522551+0000","last_clean":"2024-04-02T09:14:35.522551+0000","last_became_active":"2024-04-02T09:14:34.381321+0000","last_became_peered":"2024-04-02T09:14:34.381321+0000","last_unstale":"2024-04-02T09:14:35.522551+0000","last_undegraded":"2024-04-02T09:14:35.522551+0000","last_fullsized":"2024-04-02T09:14:35.522551+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:27.813013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522721+0000","last_change":"2024-04-02T09:14:33.869087+0000","last_active":"2024-04-02T09:14:35.522721+0000","last_peered":"2024-04-02T09:14:35.522721+0000","last_clean":"2024-04-02T09:14:35.522721+0000","last_became_active":"2024-04-02T09:14:33.868778+0000","last_became_peered":"2024-04-02T09:14:33.868778+0000","last_unstale":"2024-04-02T09:14:35.522721+0000","last_undegraded":"2024-04-02T09:14:35.522721+0000","last_fullsized":"2024-04-02T09:14:35.522721+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:53:25.730047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522621+0000","last_change":"2024-04-02T09:14:34.382115+0000","last_active":"2024-04-02T09:14:35.522621+0000","last_peered":"2024-04-02T09:14:35.522621+0000","last_clean":"2024-04-02T09:14:35.522621+0000","last_became_active":"2024-04-02T09:14:34.381857+0000","last_became_peered":"2024-04-02T09:14:34.381857+0000","last_unstale":"2024-04-02T09:14:35.522621+0000","last_undegraded":"2024-04-02T09:14:35.522621+0000","last_fullsized":"2024-04-02T09:14:35.522621+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:43:47.084525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.10","version":"264'8","reported_seq":19,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522720+0000","last_change":"2024-04-02T09:14:33.869954+0000","last_active":"2024-04-02T09:14:35.522720+0000","last_peered":"2024-04-02T09:14:35.522720+0000","last_clean":"2024-04-02T09:14:35.522720+0000","last_became_active":"2024-04-02T09:14:33.867537+0000","last_became_peered":"2024-04-02T09:14:33.867537+0000","last_unstale":"2024-04-02T09:14:35.522720+0000","last_undegraded":"2024-04-02T09:14:35.522720+0000","last_fullsized":"2024-04-02T09:14:35.522720+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:36.927662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364624+0000","last_change":"2024-04-02T09:14:34.373375+0000","last_active":"2024-04-02T09:14:35.364624+0000","last_peered":"2024-04-02T09:14:35.364624+0000","last_clean":"2024-04-02T09:14:35.364624+0000","last_became_active":"2024-04-02T09:14:34.370488+0000","last_became_peered":"2024-04-02T09:14:34.370488+0000","last_unstale":"2024-04-02T09:14:35.364624+0000","last_undegraded":"2024-04-02T09:14:35.364624+0000","last_fullsized":"2024-04-02T09:14:35.364624+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:32:00.070021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.f","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523408+0000","last_change":"2024-04-02T09:14:33.869073+0000","last_active":"2024-04-02T09:14:35.523408+0000","last_peered":"2024-04-02T09:14:35.523408+0000","last_clean":"2024-04-02T09:14:35.523408+0000","last_became_active":"2024-04-02T09:14:33.868799+0000","last_became_peered":"2024-04-02T09:14:33.868799+0000","last_unstale":"2024-04-02T09:14:35.523408+0000","last_undegraded":"2024-04-02T09:14:35.523408+0000","last_fullsized":"2024-04-02T09:14:35.523408+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:56:02.437951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523338+0000","last_change":"2024-04-02T09:14:34.381984+0000","last_active":"2024-04-02T09:14:35.523338+0000","last_peered":"2024-04-02T09:14:35.523338+0000","last_clean":"2024-04-02T09:14:35.523338+0000","last_became_active":"2024-04-02T09:14:34.381724+0000","last_became_peered":"2024-04-02T09:14:34.381724+0000","last_unstale":"2024-04-02T09:14:35.523338+0000","last_undegraded":"2024-04-02T09:14:35.523338+0000","last_fullsized":"2024-04-02T09:14:35.523338+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:10:43.121011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.e","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523146+0000","last_change":"2024-04-02T09:14:33.877468+0000","last_active":"2024-04-02T09:14:35.523146+0000","last_peered":"2024-04-02T09:14:35.523146+0000","last_clean":"2024-04-02T09:14:35.523146+0000","last_became_active":"2024-04-02T09:14:33.877199+0000","last_became_peered":"2024-04-02T09:14:33.877199+0000","last_unstale":"2024-04-02T09:14:35.523146+0000","last_undegraded":"2024-04-02T09:14:35.523146+0000","last_fullsized":"2024-04-02T09:14:35.523146+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:11.469272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523352+0000","last_change":"2024-04-02T09:14:34.373693+0000","last_active":"2024-04-02T09:14:35.523352+0000","last_peered":"2024-04-02T09:14:35.523352+0000","last_clean":"2024-04-02T09:14:35.523352+0000","last_became_active":"2024-04-02T09:14:34.372531+0000","last_became_peered":"2024-04-02T09:14:34.372531+0000","last_unstale":"2024-04-02T09:14:35.523352+0000","last_undegraded":"2024-04-02T09:14:35.523352+0000","last_fullsized":"2024-04-02T09:14:35.523352+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:24:29.851257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.d","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523408+0000","last_change":"2024-04-02T09:14:33.874303+0000","last_active":"2024-04-02T09:14:35.523408+0000","last_peered":"2024-04-02T09:14:35.523408+0000","last_clean":"2024-04-02T09:14:35.523408+0000","last_became_active":"2024-04-02T09:14:33.874024+0000","last_became_peered":"2024-04-02T09:14:33.874024+0000","last_unstale":"2024-04-02T09:14:35.523408+0000","last_undegraded":"2024-04-02T09:14:35.523408+0000","last_fullsized":"2024-04-02T09:14:35.523408+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:23:35.199298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523471+0000","last_change":"2024-04-02T09:14:34.373452+0000","last_active":"2024-04-02T09:14:35.523471+0000","last_peered":"2024-04-02T09:14:35.523471+0000","last_clean":"2024-04-02T09:14:35.523471+0000","last_became_active":"2024-04-02T09:14:34.372772+0000","last_became_peered":"2024-04-02T09:14:34.372772+0000","last_unstale":"2024-04-02T09:14:35.523471+0000","last_undegraded":"2024-04-02T09:14:35.523471+0000","last_fullsized":"2024-04-02T09:14:35.523471+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:41.722545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523664+0000","last_change":"2024-04-02T09:14:33.863647+0000","last_active":"2024-04-02T09:14:35.523664+0000","last_peered":"2024-04-02T09:14:35.523664+0000","last_clean":"2024-04-02T09:14:35.523664+0000","last_became_active":"2024-04-02T09:14:33.863385+0000","last_became_peered":"2024-04-02T09:14:33.863385+0000","last_unstale":"2024-04-02T09:14:35.523664+0000","last_undegraded":"2024-04-02T09:14:35.523664+0000","last_fullsized":"2024-04-02T09:14:35.523664+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:57:55.628012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523120+0000","last_change":"2024-04-02T09:14:34.390484+0000","last_active":"2024-04-02T09:14:35.523120+0000","last_peered":"2024-04-02T09:14:35.523120+0000","last_clean":"2024-04-02T09:14:35.523120+0000","last_became_active":"2024-04-02T09:14:34.390145+0000","last_became_peered":"2024-04-02T09:14:34.390145+0000","last_unstale":"2024-04-02T09:14:35.523120+0000","last_undegraded":"2024-04-02T09:14:35.523120+0000","last_fullsized":"2024-04-02T09:14:35.523120+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:43:52.860015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.b","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364214+0000","last_change":"2024-04-02T09:14:33.876355+0000","last_active":"2024-04-02T09:14:35.364214+0000","last_peered":"2024-04-02T09:14:35.364214+0000","last_clean":"2024-04-02T09:14:35.364214+0000","last_became_active":"2024-04-02T09:14:33.876041+0000","last_became_peered":"2024-04-02T09:14:33.876041+0000","last_unstale":"2024-04-02T09:14:35.364214+0000","last_undegraded":"2024-04-02T09:14:35.364214+0000","last_fullsized":"2024-04-02T09:14:35.364214+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:10:38.419248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362885+0000","last_change":"2024-04-02T09:14:34.388100+0000","last_active":"2024-04-02T09:14:35.362885+0000","last_peered":"2024-04-02T09:14:35.362885+0000","last_clean":"2024-04-02T09:14:35.362885+0000","last_became_active":"2024-04-02T09:14:34.383116+0000","last_became_peered":"2024-04-02T09:14:34.383116+0000","last_unstale":"2024-04-02T09:14:35.362885+0000","last_undegraded":"2024-04-02T09:14:35.362885+0000","last_fullsized":"2024-04-02T09:14:35.362885+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:51.893190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364368+0000","last_change":"2024-04-02T09:14:33.878926+0000","last_active":"2024-04-02T09:14:35.364368+0000","last_peered":"2024-04-02T09:14:35.364368+0000","last_clean":"2024-04-02T09:14:35.364368+0000","last_became_active":"2024-04-02T09:14:33.878596+0000","last_became_peered":"2024-04-02T09:14:33.878596+0000","last_unstale":"2024-04-02T09:14:35.364368+0000","last_undegraded":"2024-04-02T09:14:35.364368+0000","last_fullsized":"2024-04-02T09:14:35.364368+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:14:19.001834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523585+0000","last_change":"2024-04-02T09:14:34.374525+0000","last_active":"2024-04-02T09:14:35.523585+0000","last_peered":"2024-04-02T09:14:35.523585+0000","last_clean":"2024-04-02T09:14:35.523585+0000","last_became_active":"2024-04-02T09:14:34.374214+0000","last_became_peered":"2024-04-02T09:14:34.374214+0000","last_unstale":"2024-04-02T09:14:35.523585+0000","last_undegraded":"2024-04-02T09:14:35.523585+0000","last_fullsized":"2024-04-02T09:14:35.523585+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:20:30.074905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362662+0000","last_change":"2024-04-02T09:14:33.872263+0000","last_active":"2024-04-02T09:14:35.362662+0000","last_peered":"2024-04-02T09:14:35.362662+0000","last_clean":"2024-04-02T09:14:35.362662+0000","last_became_active":"2024-04-02T09:14:33.871830+0000","last_became_peered":"2024-04-02T09:14:33.871830+0000","last_unstale":"2024-04-02T09:14:35.362662+0000","last_undegraded":"2024-04-02T09:14:35.362662+0000","last_fullsized":"2024-04-02T09:14:35.362662+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:11.469272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522704+0000","last_change":"2024-04-02T09:14:34.374102+0000","last_active":"2024-04-02T09:14:35.522704+0000","last_peered":"2024-04-02T09:14:35.522704+0000","last_clean":"2024-04-02T09:14:35.522704+0000","last_became_active":"2024-04-02T09:14:34.373810+0000","last_became_peered":"2024-04-02T09:14:34.373810+0000","last_unstale":"2024-04-02T09:14:35.522704+0000","last_undegraded":"2024-04-02T09:14:35.522704+0000","last_fullsized":"2024-04-02T09:14:35.522704+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:20:30.074905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363497+0000","last_change":"2024-04-02T09:14:33.867489+0000","last_active":"2024-04-02T09:14:35.363497+0000","last_peered":"2024-04-02T09:14:35.363497+0000","last_clean":"2024-04-02T09:14:35.363497+0000","last_became_active":"2024-04-02T09:14:33.867206+0000","last_became_peered":"2024-04-02T09:14:33.867206+0000","last_unstale":"2024-04-02T09:14:35.363497+0000","last_undegraded":"2024-04-02T09:14:35.363497+0000","last_fullsized":"2024-04-02T09:14:35.363497+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:55:44.507753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362972+0000","last_change":"2024-04-02T09:14:34.380478+0000","last_active":"2024-04-02T09:14:35.362972+0000","last_peered":"2024-04-02T09:14:35.362972+0000","last_clean":"2024-04-02T09:14:35.362972+0000","last_became_active":"2024-04-02T09:14:34.380228+0000","last_became_peered":"2024-04-02T09:14:34.380228+0000","last_unstale":"2024-04-02T09:14:35.362972+0000","last_undegraded":"2024-04-02T09:14:35.362972+0000","last_fullsized":"2024-04-02T09:14:35.362972+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:28.009617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.7","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362982+0000","last_change":"2024-04-02T09:14:33.878819+0000","last_active":"2024-04-02T09:14:35.362982+0000","last_peered":"2024-04-02T09:14:35.362982+0000","last_clean":"2024-04-02T09:14:35.362982+0000","last_became_active":"2024-04-02T09:14:33.878105+0000","last_became_peered":"2024-04-02T09:14:33.878105+0000","last_unstale":"2024-04-02T09:14:35.362982+0000","last_undegraded":"2024-04-02T09:14:35.362982+0000","last_fullsized":"2024-04-02T09:14:35.362982+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:17:45.067998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523932+0000","last_change":"2024-04-02T09:14:34.372991+0000","last_active":"2024-04-02T09:14:35.523932+0000","last_peered":"2024-04-02T09:14:35.523932+0000","last_clean":"2024-04-02T09:14:35.523932+0000","last_became_active":"2024-04-02T09:14:34.372739+0000","last_became_peered":"2024-04-02T09:14:34.372739+0000","last_unstale":"2024-04-02T09:14:35.523932+0000","last_undegraded":"2024-04-02T09:14:35.523932+0000","last_fullsized":"2024-04-02T09:14:35.523932+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:26.825924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523722+0000","last_change":"2024-04-02T09:14:33.871269+0000","last_active":"2024-04-02T09:14:35.523722+0000","last_peered":"2024-04-02T09:14:35.523722+0000","last_clean":"2024-04-02T09:14:35.523722+0000","last_became_active":"2024-04-02T09:14:33.870767+0000","last_became_peered":"2024-04-02T09:14:33.870767+0000","last_unstale":"2024-04-02T09:14:35.523722+0000","last_undegraded":"2024-04-02T09:14:35.523722+0000","last_fullsized":"2024-04-02T09:14:35.523722+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:55:47.421500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362682+0000","last_change":"2024-04-02T09:14:34.382888+0000","last_active":"2024-04-02T09:14:35.362682+0000","last_peered":"2024-04-02T09:14:35.362682+0000","last_clean":"2024-04-02T09:14:35.362682+0000","last_became_active":"2024-04-02T09:14:34.382551+0000","last_became_peered":"2024-04-02T09:14:34.382551+0000","last_unstale":"2024-04-02T09:14:35.362682+0000","last_undegraded":"2024-04-02T09:14:35.362682+0000","last_fullsized":"2024-04-02T09:14:35.362682+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:36.082626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362917+0000","last_change":"2024-04-02T09:14:33.876070+0000","last_active":"2024-04-02T09:14:35.362917+0000","last_peered":"2024-04-02T09:14:35.362917+0000","last_clean":"2024-04-02T09:14:35.362917+0000","last_became_active":"2024-04-02T09:14:33.875825+0000","last_became_peered":"2024-04-02T09:14:33.875825+0000","last_unstale":"2024-04-02T09:14:35.362917+0000","last_undegraded":"2024-04-02T09:14:35.362917+0000","last_fullsized":"2024-04-02T09:14:35.362917+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:23:35.199298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363906+0000","last_change":"2024-04-02T09:14:34.362399+0000","last_active":"2024-04-02T09:14:35.363906+0000","last_peered":"2024-04-02T09:14:35.363906+0000","last_clean":"2024-04-02T09:14:35.363906+0000","last_became_active":"2024-04-02T09:14:34.361978+0000","last_became_peered":"2024-04-02T09:14:34.361978+0000","last_unstale":"2024-04-02T09:14:35.363906+0000","last_undegraded":"2024-04-02T09:14:35.363906+0000","last_fullsized":"2024-04-02T09:14:35.363906+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:15.964840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523592+0000","last_change":"2024-04-02T09:14:33.866042+0000","last_active":"2024-04-02T09:14:35.523592+0000","last_peered":"2024-04-02T09:14:35.523592+0000","last_clean":"2024-04-02T09:14:35.523592+0000","last_became_active":"2024-04-02T09:14:33.865757+0000","last_became_peered":"2024-04-02T09:14:33.865757+0000","last_unstale":"2024-04-02T09:14:35.523592+0000","last_undegraded":"2024-04-02T09:14:35.523592+0000","last_fullsized":"2024-04-02T09:14:35.523592+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:24.796757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522937+0000","last_change":"2024-04-02T09:14:34.375746+0000","last_active":"2024-04-02T09:14:35.522937+0000","last_peered":"2024-04-02T09:14:35.522937+0000","last_clean":"2024-04-02T09:14:35.522937+0000","last_became_active":"2024-04-02T09:14:34.375131+0000","last_became_peered":"2024-04-02T09:14:34.375131+0000","last_unstale":"2024-04-02T09:14:35.522937+0000","last_undegraded":"2024-04-02T09:14:35.522937+0000","last_fullsized":"2024-04-02T09:14:35.522937+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:03:24.411151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523040+0000","last_change":"2024-04-02T09:14:33.868955+0000","last_active":"2024-04-02T09:14:35.523040+0000","last_peered":"2024-04-02T09:14:35.523040+0000","last_clean":"2024-04-02T09:14:35.523040+0000","last_became_active":"2024-04-02T09:14:33.868639+0000","last_became_peered":"2024-04-02T09:14:33.868639+0000","last_unstale":"2024-04-02T09:14:35.523040+0000","last_undegraded":"2024-04-02T09:14:35.523040+0000","last_fullsized":"2024-04-02T09:14:35.523040+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:39:56.716397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522916+0000","last_change":"2024-04-02T09:14:34.379918+0000","last_active":"2024-04-02T09:14:35.522916+0000","last_peered":"2024-04-02T09:14:35.522916+0000","last_clean":"2024-04-02T09:14:35.522916+0000","last_became_active":"2024-04-02T09:14:34.379284+0000","last_became_peered":"2024-04-02T09:14:34.379284+0000","last_unstale":"2024-04-02T09:14:35.522916+0000","last_undegraded":"2024-04-02T09:14:35.522916+0000","last_fullsized":"2024-04-02T09:14:35.522916+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:53:01.672940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522856+0000","last_change":"2024-04-02T09:14:33.870645+0000","last_active":"2024-04-02T09:14:35.522856+0000","last_peered":"2024-04-02T09:14:35.522856+0000","last_clean":"2024-04-02T09:14:35.522856+0000","last_became_active":"2024-04-02T09:14:33.870355+0000","last_became_peered":"2024-04-02T09:14:33.870355+0000","last_unstale":"2024-04-02T09:14:35.522856+0000","last_undegraded":"2024-04-02T09:14:35.522856+0000","last_fullsized":"2024-04-02T09:14:35.522856+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:53:25.730047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523681+0000","last_change":"2024-04-02T09:14:34.370359+0000","last_active":"2024-04-02T09:14:35.523681+0000","last_peered":"2024-04-02T09:14:35.523681+0000","last_clean":"2024-04-02T09:14:35.523681+0000","last_became_active":"2024-04-02T09:14:34.370115+0000","last_became_peered":"2024-04-02T09:14:34.370115+0000","last_unstale":"2024-04-02T09:14:35.523681+0000","last_undegraded":"2024-04-02T09:14:35.523681+0000","last_fullsized":"2024-04-02T09:14:35.523681+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:43:52.860015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522693+0000","last_change":"2024-04-02T09:14:33.865894+0000","last_active":"2024-04-02T09:14:35.522693+0000","last_peered":"2024-04-02T09:14:35.522693+0000","last_clean":"2024-04-02T09:14:35.522693+0000","last_became_active":"2024-04-02T09:14:33.865610+0000","last_became_peered":"2024-04-02T09:14:33.865610+0000","last_unstale":"2024-04-02T09:14:35.522693+0000","last_undegraded":"2024-04-02T09:14:35.522693+0000","last_fullsized":"2024-04-02T09:14:35.522693+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:41.131205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522925+0000","last_change":"2024-04-02T09:14:34.372238+0000","last_active":"2024-04-02T09:14:35.522925+0000","last_peered":"2024-04-02T09:14:35.522925+0000","last_clean":"2024-04-02T09:14:35.522925+0000","last_became_active":"2024-04-02T09:14:34.371553+0000","last_became_peered":"2024-04-02T09:14:34.371553+0000","last_unstale":"2024-04-02T09:14:35.522925+0000","last_undegraded":"2024-04-02T09:14:35.522925+0000","last_fullsized":"2024-04-02T09:14:35.522925+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:03:24.411151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522812+0000","last_change":"2024-04-02T09:14:33.874635+0000","last_active":"2024-04-02T09:14:35.522812+0000","last_peered":"2024-04-02T09:14:35.522812+0000","last_clean":"2024-04-02T09:14:35.522812+0000","last_became_active":"2024-04-02T09:14:33.874348+0000","last_became_peered":"2024-04-02T09:14:33.874348+0000","last_unstale":"2024-04-02T09:14:35.522812+0000","last_undegraded":"2024-04-02T09:14:35.522812+0000","last_fullsized":"2024-04-02T09:14:35.522812+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:18:07.343969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362709+0000","last_change":"2024-04-02T09:14:34.387134+0000","last_active":"2024-04-02T09:14:35.362709+0000","last_peered":"2024-04-02T09:14:35.362709+0000","last_clean":"2024-04-02T09:14:35.362709+0000","last_became_active":"2024-04-02T09:14:34.386882+0000","last_became_peered":"2024-04-02T09:14:34.386882+0000","last_unstale":"2024-04-02T09:14:35.362709+0000","last_undegraded":"2024-04-02T09:14:35.362709+0000","last_fullsized":"2024-04-02T09:14:35.362709+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:44:06.694478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"250'381","reported_seq":1365,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-04-02T09:14:34.367154+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:14:34.367154+0000","last_peered":"2024-04-02T09:14:34.367154+0000","last_clean":"2024-04-02T09:14:34.367154+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:14:34.367154+0000","last_undegraded":"2024-04-02T09:14:34.367154+0000","last_fullsized":"2024-04-02T09:14:34.367154+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576009,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60644,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311196,"statfs":{"total":96636764160,"available":96574664704,"internally_reserved":0,"allocated":3358720,"data_stored":1721457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.70899999999999996,"15min":0.71799999999999997},"min":{"1min":0.34599999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.80000000000000004,"5min":8.016,"15min":8.016},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.69899999999999995,"15min":0.71899999999999997},"min":{"1min":0.379,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.83299999999999996,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.754}]},{"osd":1,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.74299999999999999,"15min":0.74199999999999999},"min":{"1min":0.46300000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91000000000000003,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.42399999999999999},"max":{"1min":0.73499999999999999,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.57499999999999996}]},{"osd":2,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.77900000000000003,"15min":0.76900000000000002},"min":{"1min":0.46999999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86199999999999999,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.78400000000000003,"15min":0.80400000000000005},"min":{"1min":0.53800000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.872,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.68999999999999995}]},{"osd":3,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.751,"15min":0.749},"min":{"1min":0.49199999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86299999999999999,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.79000000000000004,"15min":0.77200000000000002},"min":{"1min":0.51900000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":5.093,"15min":5.093},"last":0.78700000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.66400000000000003,"15min":0.624},"min":{"1min":0.39200000000000002,"5min":0.38300000000000001,"15min":0.36499999999999999},"max":{"1min":0.83699999999999997,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.746},"min":{"1min":0.58499999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90100000000000002,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.76800000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.73199999999999998,"15min":0.69399999999999995},"min":{"1min":0.436,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":0.88600000000000001,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.621},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.77000000000000002,"15min":0.73399999999999999},"min":{"1min":0.54000000000000004,"5min":0.42599999999999999,"15min":0.41099999999999998},"max":{"1min":0.878,"5min":3.492,"15min":3.492},"last":0.73999999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.72099999999999997,"15min":0.69599999999999995},"min":{"1min":0.51300000000000001,"5min":0.48299999999999998,"15min":0.47099999999999997},"max":{"1min":0.90700000000000003,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.77600000000000002,"15min":0.71699999999999997},"min":{"1min":0.42199999999999999,"5min":0.41599999999999998,"15min":0.41099999999999998},"max":{"1min":0.91600000000000004,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.72299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834576007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42600,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329240,"statfs":{"total":96636764160,"available":96593141760,"internally_reserved":0,"allocated":3362816,"data_stored":1722909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68200000000000005,"15min":0.72099999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93999999999999995,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.73399999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.0600000000000001},"last":0.79300000000000004}]},{"osd":1,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.67900000000000005,"15min":0.71299999999999997},"min":{"1min":0.46000000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.999,"5min":1.044,"15min":1.212},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.70799999999999996,"15min":0.747},"min":{"1min":0.505,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.0589999999999999,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.76600000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.69799999999999995,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94799999999999995,"5min":1.046,"15min":1.2789999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70099999999999996,"15min":0.73299999999999998},"min":{"1min":0.47499999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.226},"last":0.81200000000000006}]},{"osd":3,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.71999999999999997,"15min":0.75600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.026,"5min":1.141,"15min":1.2549999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.04,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.82199999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.628,"15min":0.626},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65800000000000003,"15min":0.66200000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.41999999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1879999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.64600000000000002,"15min":0.65200000000000002},"min":{"1min":0.433,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88800000000000001,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.70699999999999996,"15min":0.72499999999999998},"min":{"1min":0.501,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.60599999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73899999999999999},"min":{"1min":0.48499999999999999,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.0129999999999999,"5min":1.21,"15min":1.2370000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.72899999999999998,"15min":0.73899999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.49399999999999999},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.72099999999999997}]}]},{"osd":1,"up_from":13,"seq":55834576008,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57120,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314720,"statfs":{"total":96636764160,"available":96578273280,"internally_reserved":0,"allocated":3940352,"data_stored":2311679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42899999999999999,"15min":0.44800000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82499999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48299999999999998,"15min":0.503},"min":{"1min":0.31900000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.79800000000000004},"last":0.46400000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.33600000000000002,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.59599999999999997,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58699999999999997},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.443}]},{"osd":3,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54100000000000004,"15min":0.55600000000000005},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.52300000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.76000000000000001,"15min":0.78100000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.433},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.753,"15min":0.77700000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.41799999999999998},"max":{"1min":0.98299999999999998,"5min":1.05,"15min":1.25},"last":0.57699999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.753,"15min":0.78200000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.93400000000000005,"5min":1.03,"15min":1.321},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78400000000000003,"15min":0.80100000000000005},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.115,"5min":1.115,"15min":1.173},"last":0.55100000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.78300000000000003,"15min":0.80700000000000005},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.48099999999999998},"max":{"1min":1.087,"5min":1.087,"15min":1.2210000000000001},"last":0.752},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.79100000000000004,"15min":0.80600000000000005},"min":{"1min":0.52900000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.024,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.60599999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.78900000000000003,"15min":0.81499999999999995},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0740000000000001,"5min":1.1080000000000001,"15min":1.236},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81200000000000006,"15min":0.83099999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.129,"5min":1.129,"15min":1.2789999999999999},"last":0.80700000000000005}]}]},{"osd":0,"up_from":13,"seq":55834576008,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52448,"kb_used_data":3272,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319392,"statfs":{"total":96636764160,"available":96583057408,"internally_reserved":0,"allocated":3350528,"data_stored":1720944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.42699999999999999,"15min":0.45000000000000001},"min":{"1min":0.26700000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.58999999999999997,"5min":0.69899999999999995,"15min":0.86699999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.48099999999999998,"15min":0.499},"min":{"1min":0.29799999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66300000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.437}]},{"osd":2,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.48699999999999999,"15min":0.50600000000000001},"min":{"1min":0.35699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.621,"5min":0.69399999999999995,"15min":0.82699999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52100000000000002,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.63300000000000001,"5min":0.999,"15min":0.999},"last":0.38200000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53100000000000003,"15min":0.56100000000000005},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.63500000000000001,"5min":0.79800000000000004,"15min":0.85699999999999998},"last":0.505},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.63},"min":{"1min":0.47599999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.66000000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.54800000000000004}]},{"osd":4,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.82199999999999995},"min":{"1min":0.622,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":0.86499999999999999,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.76000000000000001,"15min":0.81599999999999995},"min":{"1min":0.55100000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.90200000000000002,"5min":1.169,"15min":1.169},"last":0.66200000000000003}]},{"osd":5,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.76500000000000001,"15min":0.81699999999999995},"min":{"1min":0.58299999999999996,"5min":0.497,"15min":0.497},"max":{"1min":0.95199999999999996,"5min":1.145,"15min":1.232},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.78500000000000003,"15min":0.82499999999999996},"min":{"1min":0.53600000000000003,"5min":0.497,"15min":0.497},"max":{"1min":0.94399999999999995,"5min":1.21,"15min":1.21},"last":0.70299999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.75,"15min":0.79100000000000004},"min":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.88200000000000001,"5min":1.1830000000000001,"15min":1.196},"last":0.747},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.77200000000000002,"15min":0.80400000000000005},"min":{"1min":0.54200000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96799999999999997,"5min":1.246,"15min":1.246},"last":0.56299999999999994}]},{"osd":7,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.80700000000000005,"15min":0.84799999999999998},"min":{"1min":0.60999999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.98699999999999999,"5min":1.292,"15min":1.292},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.79800000000000004,"15min":0.83999999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.95799999999999996,"5min":1.1719999999999999,"15min":1.2130000000000001},"last":0.69099999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576007,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52440,"kb_used_data":3268,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319400,"statfs":{"total":96636764160,"available":96583065600,"internally_reserved":0,"allocated":3346432,"data_stored":1721311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.45100000000000001,"15min":0.443},"min":{"1min":0.28899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.316},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.31,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":0.46100000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.496,"15min":0.503},"min":{"1min":0.34000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53600000000000003,"15min":0.55400000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.42399999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.71199999999999997,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.626},"min":{"1min":0.45400000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.44}]},{"osd":4,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.75900000000000001,"15min":0.77300000000000002},"min":{"1min":0.56699999999999995,"5min":0.495,"15min":0.495},"max":{"1min":2.3999999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.77500000000000002,"15min":0.78900000000000003},"min":{"1min":0.54600000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.52700000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.77600000000000002,"15min":0.78600000000000003},"min":{"1min":0.57799999999999996,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1379999999999999,"5min":1.99,"15min":1.99},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.78000000000000003,"15min":0.80100000000000005},"min":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.51900000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79100000000000004,"15min":0.80500000000000005},"min":{"1min":0.59299999999999997,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77300000000000002,"15min":0.80100000000000005},"min":{"1min":0.55900000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.103,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.60799999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.80000000000000004,"15min":0.82399999999999995},"min":{"1min":0.56999999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.82199999999999995},"min":{"1min":0.59099999999999997,"5min":0.497,"15min":0.497},"max":{"1min":0.97499999999999998,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.54900000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576007,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52464,"kb_used_data":3292,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319376,"statfs":{"total":96636764160,"available":96583041024,"internally_reserved":0,"allocated":3371008,"data_stored":1723057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.41699999999999998,"15min":0.443},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64000000000000001,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.435},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.58699999999999997,"5min":0.71199999999999997,"15min":0.876},"last":0.41299999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47499999999999998,"15min":0.498},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.90300000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.53900000000000003,"15min":0.56799999999999995},"min":{"1min":0.38500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.66100000000000003,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.48999999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.58099999999999996,"15min":0.61699999999999999},"min":{"1min":0.41299999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71299999999999997,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56799999999999995,"15min":0.60999999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":1.0840000000000001},"last":0.45800000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72099999999999997,"15min":0.751},"min":{"1min":0.438,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.73499999999999999,"15min":0.76600000000000001},"min":{"1min":0.49299999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.97099999999999997,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.57799999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.74299999999999999,"15min":0.77400000000000002},"min":{"1min":0.59399999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.63},{"interface":"front","average":{"1min":0.751,"5min":0.75700000000000001,"15min":0.78000000000000003},"min":{"1min":0.51100000000000001,"5min":0.505,"15min":0.505},"max":{"1min":0.98299999999999998,"5min":1.266,"15min":1.266},"last":0.65000000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.96799999999999997,"5min":1.284,"15min":1.284},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76500000000000001,"15min":0.80300000000000005},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.495},"max":{"1min":1.0389999999999999,"5min":1.2,"15min":1.2},"last":0.72699999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.76600000000000001,"15min":0.80400000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.125,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.78900000000000003,"15min":0.81699999999999995},"min":{"1min":0.57599999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.087,"5min":1.248,"15min":1.248},"last":0.61699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576007,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48344,"kb_used_data":3268,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323496,"statfs":{"total":96636764160,"available":96587259904,"internally_reserved":0,"allocated":3346432,"data_stored":1721311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64500000000000002,"15min":0.67700000000000005},"min":{"1min":0.38100000000000001,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":0.94399999999999995,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.44},{"interface":"front","average":{"1min":0.62,"5min":0.63600000000000001,"15min":0.67100000000000004},"min":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.075},"last":0.66200000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.70499999999999996,"15min":0.746},"min":{"1min":0.51700000000000002,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":1.014,"5min":1.0549999999999999,"15min":1.069},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72199999999999998,"15min":0.75700000000000001},"min":{"1min":0.49099999999999999,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.64000000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66200000000000003,"15min":0.69999999999999996},"min":{"1min":0.439,"5min":0.438,"15min":0.438},"max":{"1min":0.95799999999999996,"5min":1.077,"15min":1.077},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68700000000000006,"15min":0.71899999999999997},"min":{"1min":0.47799999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.032,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.50600000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.71399999999999997,"15min":0.753},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.47399999999999998},"max":{"1min":0.93999999999999995,"5min":0.98099999999999998,"15min":1.018},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70999999999999996,"15min":0.74199999999999999},"min":{"1min":0.52100000000000002,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.41299999999999998,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.93700000000000006,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.755}]},{"osd":6,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.64200000000000002,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91800000000000004,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.69199999999999995,"15min":0.69599999999999995},"min":{"1min":0.46899999999999997,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":1.0149999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.70399999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.68500000000000005,"15min":0.70499999999999996},"min":{"1min":0.45500000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.0329999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76500000000000001,"15min":0.78000000000000003},"min":{"1min":0.624,"5min":0.5,"15min":0.5},"max":{"1min":1.0700000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.81299999999999994}]}]},{"osd":5,"up_from":13,"seq":55834576010,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53012,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318828,"statfs":{"total":96636764160,"available":96582479872,"internally_reserved":0,"allocated":3940352,"data_stored":2311242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.82699999999999996,"5min":0.98699999999999999,"15min":1.004},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.68999999999999995,"15min":0.71999999999999997},"min":{"1min":0.48399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.028,"5min":1.085,"15min":1.085},"last":0.83599999999999997}]},{"osd":1,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.68100000000000005,"15min":0.71499999999999997},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84399999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70999999999999996,"15min":0.72499999999999998},"min":{"1min":0.52300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88500000000000001,"5min":0.95299999999999996,"15min":1.052},"last":0.81799999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.71599999999999997,"15min":0.73799999999999999},"min":{"1min":0.47199999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97299999999999998,"5min":1.004,"15min":1.014},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.69499999999999995,"15min":0.73699999999999999},"min":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.81799999999999995,"5min":1.018,"15min":1.0680000000000001},"last":0.59199999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.747,"15min":0.77700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.88500000000000001,"5min":0.98899999999999999,"15min":1.083},"last":0.871},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77000000000000002,"15min":0.79000000000000004},"min":{"1min":0.58399999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1020000000000001},"last":0.754}]},{"osd":4,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.40699999999999997,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.054},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":0.98799999999999999,"5min":1.145,"15min":1.145},"last":0.68000000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67900000000000005,"15min":0.67300000000000004},"min":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96799999999999997,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71399999999999997,"15min":0.71099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.94899999999999995,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.69799999999999995}]},{"osd":7,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73999999999999999,"15min":0.73799999999999999},"min":{"1min":0.46500000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.0820000000000001,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76300000000000001,"15min":0.76600000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1519999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.71399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:14:39.690 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-02T09:14:39.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.918+0000 7f3251800700 1 Processor -- start 2024-04-02T09:14:39.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.918+0000 7f3251800700 1 -- start start 2024-04-02T09:14:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.919+0000 7f3251800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f324c15d640 0x7f324c0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.919+0000 7f3251800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f324c0d30d0 con 0x7f324c0ceec0 2024-04-02T09:14:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.919+0000 7f3251800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f324c0b6840 con 0x7f324c1619d0 2024-04-02T09:14:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.919+0000 7f3251800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324c0d2880 con 0x7f324c15d640 2024-04-02T09:14:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.920+0000 7f324a000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f324c0ceec0 0x7f324c1612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38676/0 (socket says 172.21.15.89:38676) 2024-04-02T09:14:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.920+0000 7f324a000700 1 -- 172.21.15.89:0/693483865 learned_addr learned my addr 172.21.15.89:0/693483865 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:39.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.920+0000 7f324aa00700 1 --2- 172.21.15.89:0/693483865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f324c15d640 0x7f324c0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:39.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.920+0000 7f3249600700 1 -- 172.21.15.89:0/693483865 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432123039 0 0) 0x7f324c0d30d0 con 0x7f324c0ceec0 2024-04-02T09:14:39.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.920+0000 7f3249600700 1 -- 172.21.15.89:0/693483865 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32300035b0 con 0x7f324c0ceec0 2024-04-02T09:14:39.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.920+0000 7f3249600700 1 -- 172.21.15.89:0/693483865 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874492724 0 0) 0x7f324c0b6840 con 0x7f324c1619d0 2024-04-02T09:14:39.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.920+0000 7f3249600700 1 -- 172.21.15.89:0/693483865 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f324c0d30d0 con 0x7f324c1619d0 2024-04-02T09:14:39.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.920+0000 7f3249600700 1 -- 172.21.15.89:0/693483865 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3254620302 0 0) 0x7f32300035b0 con 0x7f324c0ceec0 2024-04-02T09:14:39.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.920+0000 7f324aa00700 1 -- 172.21.15.89:0/693483865 >> v1:172.21.15.113:6789/0 conn(0x7f324c1619d0 legacy=0x7f324c0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:39.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.920+0000 7f324aa00700 1 -- 172.21.15.89:0/693483865 >> v1:172.21.15.89:6789/0 conn(0x7f324c0ceec0 legacy=0x7f324c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:39.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.920+0000 7f324aa00700 1 -- 172.21.15.89:0/693483865 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f324c0d2b30 con 0x7f324c15d640 2024-04-02T09:14:39.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.921+0000 7f324aa00700 1 --2- 172.21.15.89:0/693483865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f324c15d640 0x7f324c0d2340 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f323c009870 tx=0x7f323c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86dc5839bcdaccf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:39.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.921+0000 7f3249600700 1 -- 172.21.15.89:0/693483865 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3238001670 con 0x7f324c15d640 2024-04-02T09:14:39.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.921+0000 7f3249600700 1 -- 172.21.15.89:0/693483865 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323c004030 con 0x7f324c15d640 2024-04-02T09:14:39.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.921+0000 7f3249600700 1 -- 172.21.15.89:0/693483865 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c0043a0 con 0x7f324c15d640 2024-04-02T09:14:39.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.921+0000 7f3251800700 1 -- 172.21.15.89:0/693483865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f324c15d640 msgr2=0x7f324c0d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:39.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.921+0000 7f3251800700 1 --2- 172.21.15.89:0/693483865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f324c15d640 0x7f324c0d2340 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f323c009870 tx=0x7f323c009b80 comp rx=0 tx=0).stop 2024-04-02T09:14:39.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.921+0000 7f3251800700 1 -- 172.21.15.89:0/693483865 shutdown_connections 2024-04-02T09:14:39.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.921+0000 7f3251800700 1 --2- 172.21.15.89:0/693483865 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f324c15d640 0x7f324c0d2340 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.921+0000 7f3251800700 1 -- 172.21.15.89:0/693483865 >> 172.21.15.89:0/693483865 conn(0x7f324c08aff0 msgr2=0x7f324c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:39.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.922+0000 7f3251800700 1 -- 172.21.15.89:0/693483865 shutdown_connections 2024-04-02T09:14:39.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.922+0000 7f3251800700 1 -- 172.21.15.89:0/693483865 wait complete. 2024-04-02T09:14:39.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.923+0000 7f3251800700 1 Processor -- start 2024-04-02T09:14:39.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.923+0000 7f3251800700 1 -- start start 2024-04-02T09:14:39.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.924+0000 7f3251800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f324c0ceec0 0x7f324c0a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:39.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.924+0000 7f3251800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f324c15d640 0x7f324c1f6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.924+0000 7f3251800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f324c1619d0 0x7f324c1f6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.924+0000 7f3251800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324c0cc320 con 0x7f324c0ceec0 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.924+0000 7f3251800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324c0cb710 con 0x7f324c15d640 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.924+0000 7f3251800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324c0cb9f0 con 0x7f324c1619d0 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.924+0000 7f324aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f324c0ceec0 0x7f324c0a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.924+0000 7f324aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f324c0ceec0 0x7f324c0a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35102/0 (socket says 172.21.15.89:35102) 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.924+0000 7f324aa00700 1 -- 172.21.15.89:0/4163376770 learned_addr learned my addr 172.21.15.89:0/4163376770 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.925+0000 7f324a000700 1 --2- 172.21.15.89:0/4163376770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f324c15d640 0x7f324c1f6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.925+0000 7f324aa00700 1 -- 172.21.15.89:0/4163376770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f324c1619d0 msgr2=0x7f324c1f6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.925+0000 7f324aa00700 1 --2- 172.21.15.89:0/4163376770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f324c1619d0 0x7f324c1f6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.925+0000 7f324aa00700 1 -- 172.21.15.89:0/4163376770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f324c15d640 msgr2=0x7f324c1f6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.925+0000 7f324aa00700 1 --2- 172.21.15.89:0/4163376770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f324c15d640 0x7f324c1f6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.925+0000 7f324aa00700 1 -- 172.21.15.89:0/4163376770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c0094a0 con 0x7f324c0ceec0 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.925+0000 7f324aa00700 1 --2- 172.21.15.89:0/4163376770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f324c0ceec0 0x7f324c0a0bc0 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f323c000c00 tx=0x7f323c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.925+0000 7f3242a00700 1 -- 172.21.15.89:0/4163376770 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c003e40 con 0x7f324c0ceec0 2024-04-02T09:14:39.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.925+0000 7f3242a00700 1 -- 172.21.15.89:0/4163376770 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323c01f070 con 0x7f324c0ceec0 2024-04-02T09:14:39.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.925+0000 7f3251800700 1 -- 172.21.15.89:0/4163376770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f324c0cbcd0 con 0x7f324c0ceec0 2024-04-02T09:14:39.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.925+0000 7f3242a00700 1 -- 172.21.15.89:0/4163376770 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c011870 con 0x7f324c0ceec0 2024-04-02T09:14:39.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.926+0000 7f3251800700 1 -- 172.21.15.89:0/4163376770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f324c1575e0 con 0x7f324c0ceec0 2024-04-02T09:14:39.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.926+0000 7f3251800700 1 -- 172.21.15.89:0/4163376770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f324c067720 con 0x7f324c0ceec0 2024-04-02T09:14:39.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.927+0000 7f3242a00700 1 -- 172.21.15.89:0/4163376770 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f323c0119d0 con 0x7f324c0ceec0 2024-04-02T09:14:39.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.928+0000 7f3242a00700 1 --2- 172.21.15.89:0/4163376770 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f322c06aed0 0x7f322c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:39.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.928+0000 7f3242a00700 1 -- 172.21.15.89:0/4163376770 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f323c021070 con 0x7f324c0ceec0 2024-04-02T09:14:39.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.928+0000 7f324a000700 1 --2- 172.21.15.89:0/4163376770 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f322c06aed0 0x7f322c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:39.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.929+0000 7f324a000700 1 --2- 172.21.15.89:0/4163376770 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f322c06aed0 0x7f322c06d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f324c1f6260 tx=0x7f3238009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:39.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:39.932+0000 7f3242a00700 1 -- 172.21.15.89:0/4163376770 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f323c055fa0 con 0x7f324c0ceec0 2024-04-02T09:14:40.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.216+0000 7f3251800700 1 -- 172.21.15.89:0/4163376770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f324c15c210 con 0x7f324c0ceec0 2024-04-02T09:14:40.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.216+0000 7f3242a00700 1 -- 172.21.15.89:0/4163376770 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1029 (secure 0 0 0) 0x7f323c0595c0 con 0x7f324c0ceec0 2024-04-02T09:14:40.214 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:40.215 INFO:teuthology.orchestra.run.smithi089.stdout:{"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5952,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":265,"num_osds":8,"num_up_osds":8,"osd_up_since":1712043340,"num_in_osds":8,"osd_in_since":1712043332,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":96},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":428646400,"bytes_avail":772665466880,"bytes_total":773094113280,"unknown_pgs_ratio":0.98969072103500366},"fsmap":{"epoch":135,"id":17,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":17,"rank":0,"name":"c","status":"up:active","gid":12762}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":23,"modified":"2024-04-02T09:11:01.562287+0000","services":{}},"progress_events":{}} 2024-04-02T09:14:40.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.219+0000 7f3251800700 1 -- 172.21.15.89:0/4163376770 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f322c06aed0 msgr2=0x7f322c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:40.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.219+0000 7f3251800700 1 --2- 172.21.15.89:0/4163376770 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f322c06aed0 0x7f322c06d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f324c1f6260 tx=0x7f3238009040 comp rx=0 tx=0).stop 2024-04-02T09:14:40.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.219+0000 7f3251800700 1 -- 172.21.15.89:0/4163376770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f324c0ceec0 msgr2=0x7f324c0a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:40.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.219+0000 7f3251800700 1 --2- 172.21.15.89:0/4163376770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f324c0ceec0 0x7f324c0a0bc0 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f323c000c00 tx=0x7f323c004750 comp rx=0 tx=0).stop 2024-04-02T09:14:40.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.219+0000 7f3251800700 1 -- 172.21.15.89:0/4163376770 shutdown_connections 2024-04-02T09:14:40.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.219+0000 7f3251800700 1 --2- 172.21.15.89:0/4163376770 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f324c1619d0 0x7f324c1f6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.219+0000 7f3251800700 1 --2- 172.21.15.89:0/4163376770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f324c0ceec0 0x7f324c0a0bc0 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.219+0000 7f3251800700 1 --2- 172.21.15.89:0/4163376770 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f322c06aed0 0x7f322c06d390 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.219+0000 7f3251800700 1 --2- 172.21.15.89:0/4163376770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f324c15d640 0x7f324c1f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.219+0000 7f3251800700 1 -- 172.21.15.89:0/4163376770 >> 172.21.15.89:0/4163376770 conn(0x7f324c08aff0 msgr2=0x7f324c0d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:40.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.219+0000 7f3251800700 1 -- 172.21.15.89:0/4163376770 shutdown_connections 2024-04-02T09:14:40.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.219+0000 7f3251800700 1 -- 172.21.15.89:0/4163376770 wait complete. 2024-04-02T09:14:40.235 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-02T09:14:40.235 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:14:40.255 INFO:tasks.ceph.mds.c.smithi089.stderr:2024-04-02T09:14:40.257+0000 7f9757a00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T09:14:40.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.431+0000 7f5c56a00700 1 Processor -- start 2024-04-02T09:14:40.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.432+0000 7f5c56a00700 1 -- start start 2024-04-02T09:14:40.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.433+0000 7f5c56a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c5015d640 0x7f5c500d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:40.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.433+0000 7f5c56a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5c500ceec0 0x7f5c501612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:40.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.433+0000 7f5c56a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5c50161800 0x7f5c500cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:40.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.433+0000 7f5c56a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c50161c90 con 0x7f5c500ceec0 2024-04-02T09:14:40.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.433+0000 7f5c56a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c500d30d0 con 0x7f5c50161800 2024-04-02T09:14:40.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.433+0000 7f5c56a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c500b6840 con 0x7f5c5015d640 2024-04-02T09:14:40.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.433+0000 7f5c4f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5c500ceec0 0x7f5c501612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:40.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.433+0000 7f5c4f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5c500ceec0 0x7f5c501612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35104/0 (socket says 172.21.15.89:35104) 2024-04-02T09:14:40.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.433+0000 7f5c4f400700 1 -- 172.21.15.89:0/3025405287 learned_addr learned my addr 172.21.15.89:0/3025405287 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:40.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.433+0000 7f5c4fe00700 1 --2- 172.21.15.89:0/3025405287 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c5015d640 0x7f5c500d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:40.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.433+0000 7f5c54a00700 1 --2- 172.21.15.89:0/3025405287 >> v2:172.21.15.113:3300/0 conn(0x7f5c50161800 0x7f5c500cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:40.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.434+0000 7f5c4f400700 1 -- 172.21.15.89:0/3025405287 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c5015d640 msgr2=0x7f5c500d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:40.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.434+0000 7f5c4f400700 1 --2- 172.21.15.89:0/3025405287 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c5015d640 0x7f5c500d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.434+0000 7f5c4f400700 1 -- 172.21.15.89:0/3025405287 >> v2:172.21.15.113:3300/0 conn(0x7f5c50161800 msgr2=0x7f5c500cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:40.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.434+0000 7f5c4f400700 1 --2- 172.21.15.89:0/3025405287 >> v2:172.21.15.113:3300/0 conn(0x7f5c50161800 0x7f5c500cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.434+0000 7f5c4f400700 1 -- 172.21.15.89:0/3025405287 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c500d2880 con 0x7f5c500ceec0 2024-04-02T09:14:40.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.434+0000 7f5c54a00700 1 --2- 172.21.15.89:0/3025405287 >> v2:172.21.15.113:3300/0 conn(0x7f5c50161800 0x7f5c500cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:40.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.434+0000 7f5c4f400700 1 --2- 172.21.15.89:0/3025405287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c500ceec0 0x7f5c501612c0 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f5c500c7050 tx=0x7f5c3c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b974ac92500ba545 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:40.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.434+0000 7f5c4ea00700 1 -- 172.21.15.89:0/3025405287 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c00b870 con 0x7f5c500ceec0 2024-04-02T09:14:40.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.434+0000 7f5c4ea00700 1 -- 172.21.15.89:0/3025405287 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c00b9d0 con 0x7f5c500ceec0 2024-04-02T09:14:40.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.434+0000 7f5c4ea00700 1 -- 172.21.15.89:0/3025405287 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c00bd60 con 0x7f5c500ceec0 2024-04-02T09:14:40.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.435+0000 7f5c56a00700 1 -- 172.21.15.89:0/3025405287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c500ceec0 msgr2=0x7f5c501612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:40.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.435+0000 7f5c56a00700 1 --2- 172.21.15.89:0/3025405287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c500ceec0 0x7f5c501612c0 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f5c500c7050 tx=0x7f5c3c009bd0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.435+0000 7f5c56a00700 1 -- 172.21.15.89:0/3025405287 shutdown_connections 2024-04-02T09:14:40.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.435+0000 7f5c56a00700 1 --2- 172.21.15.89:0/3025405287 >> v2:172.21.15.113:3300/0 conn(0x7f5c50161800 0x7f5c500cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.435+0000 7f5c56a00700 1 --2- 172.21.15.89:0/3025405287 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c5015d640 0x7f5c500d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.435+0000 7f5c56a00700 1 --2- 172.21.15.89:0/3025405287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c500ceec0 0x7f5c501612c0 unknown :-1 s=CLOSED pgs=2822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.435+0000 7f5c56a00700 1 -- 172.21.15.89:0/3025405287 >> 172.21.15.89:0/3025405287 conn(0x7f5c5008aff0 msgr2=0x7f5c500d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:40.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.435+0000 7f5c56a00700 1 -- 172.21.15.89:0/3025405287 shutdown_connections 2024-04-02T09:14:40.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.435+0000 7f5c56a00700 1 -- 172.21.15.89:0/3025405287 wait complete. 2024-04-02T09:14:40.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.436+0000 7f5c56a00700 1 Processor -- start 2024-04-02T09:14:40.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.437+0000 7f5c56a00700 1 -- start start 2024-04-02T09:14:40.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.437+0000 7f5c56a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c500ceec0 0x7f5c500ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:40.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.437+0000 7f5c56a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c5015d640 0x7f5c5014fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:40.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.437+0000 7f5c56a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c50161800 0x7f5c50150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:40.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.437+0000 7f5c56a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5014f770 con 0x7f5c50161800 2024-04-02T09:14:40.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.437+0000 7f5c56a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c500ce8d0 con 0x7f5c5015d640 2024-04-02T09:14:40.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.437+0000 7f5c56a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5014eb60 con 0x7f5c500ceec0 2024-04-02T09:14:40.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.438+0000 7f5c54a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c50161800 0x7f5c50150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:40.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.438+0000 7f5c54a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c50161800 0x7f5c50150140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35110/0 (socket says 172.21.15.89:35110) 2024-04-02T09:14:40.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.438+0000 7f5c54a00700 1 -- 172.21.15.89:0/210751368 learned_addr learned my addr 172.21.15.89:0/210751368 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:40.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.438+0000 7f5c4fe00700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c500ceec0 0x7f5c500ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:40.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.438+0000 7f5c4f400700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c5015d640 0x7f5c5014fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:40.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.438+0000 7f5c54a00700 1 -- 172.21.15.89:0/210751368 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c500ceec0 msgr2=0x7f5c500ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:40.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.438+0000 7f5c54a00700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c500ceec0 0x7f5c500ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.438+0000 7f5c54a00700 1 -- 172.21.15.89:0/210751368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c5015d640 msgr2=0x7f5c5014fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:40.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.438+0000 7f5c54a00700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c5015d640 0x7f5c5014fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.438+0000 7f5c54a00700 1 -- 172.21.15.89:0/210751368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3c009550 con 0x7f5c50161800 2024-04-02T09:14:40.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.438+0000 7f5c4fe00700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c500ceec0 0x7f5c500ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:40.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.438+0000 7f5c4f400700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c5015d640 0x7f5c5014fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:40.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.438+0000 7f5c54a00700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c50161800 0x7f5c50150140 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f5c48009fa0 tx=0x7f5c4800d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:40.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.439+0000 7f5c47e00700 1 -- 172.21.15.89:0/210751368 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c48003a20 con 0x7f5c50161800 2024-04-02T09:14:40.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.439+0000 7f5c47e00700 1 -- 172.21.15.89:0/210751368 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4800f040 con 0x7f5c50161800 2024-04-02T09:14:40.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.439+0000 7f5c47e00700 1 -- 172.21.15.89:0/210751368 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c48003de0 con 0x7f5c50161800 2024-04-02T09:14:40.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.439+0000 7f5c56a00700 1 -- 172.21.15.89:0/210751368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5014ee40 con 0x7f5c50161800 2024-04-02T09:14:40.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.439+0000 7f5c56a00700 1 -- 172.21.15.89:0/210751368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5014f400 con 0x7f5c50161800 2024-04-02T09:14:40.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.440+0000 7f5c56a00700 1 -- 172.21.15.89:0/210751368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c50067720 con 0x7f5c50161800 2024-04-02T09:14:40.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.441+0000 7f5c47e00700 1 -- 172.21.15.89:0/210751368 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5c48004030 con 0x7f5c50161800 2024-04-02T09:14:40.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.442+0000 7f5c47e00700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c3006aed0 0x7f5c3006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:40.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.442+0000 7f5c47e00700 1 -- 172.21.15.89:0/210751368 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f5c48089b30 con 0x7f5c50161800 2024-04-02T09:14:40.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.442+0000 7f5c4fe00700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c3006aed0 0x7f5c3006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:40.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.443+0000 7f5c4fe00700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c3006aed0 0x7f5c3006d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5c400099e0 tx=0x7f5c40008040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:40.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.446+0000 7f5c47e00700 1 -- 172.21.15.89:0/210751368 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c48054770 con 0x7f5c50161800 2024-04-02T09:14:40.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.646+0000 7f5c56a00700 1 -- 172.21.15.89:0/210751368 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5c500b64d0 con 0x7f5c3006aed0 2024-04-02T09:14:40.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.654+0000 7f5c47e00700 1 -- 172.21.15.89:0/210751368 <== mgr.11320 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289387 (secure 0 0 0) 0x7f5c40009e50 con 0x7f5c3006aed0 2024-04-02T09:14:40.658 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:40.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.659+0000 7f5c56a00700 1 -- 172.21.15.89:0/210751368 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c3006aed0 msgr2=0x7f5c3006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:40.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.659+0000 7f5c56a00700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c3006aed0 0x7f5c3006d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5c400099e0 tx=0x7f5c40008040 comp rx=0 tx=0).stop 2024-04-02T09:14:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.659+0000 7f5c56a00700 1 -- 172.21.15.89:0/210751368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c50161800 msgr2=0x7f5c50150140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.659+0000 7f5c56a00700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c50161800 0x7f5c50150140 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f5c48009fa0 tx=0x7f5c4800d460 comp rx=0 tx=0).stop 2024-04-02T09:14:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.659+0000 7f5c56a00700 1 -- 172.21.15.89:0/210751368 shutdown_connections 2024-04-02T09:14:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.659+0000 7f5c56a00700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c50161800 0x7f5c50150140 unknown :-1 s=CLOSED pgs=2823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.659+0000 7f5c56a00700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c500ceec0 0x7f5c500ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.659+0000 7f5c56a00700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c3006aed0 0x7f5c3006d390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.659+0000 7f5c56a00700 1 --2- 172.21.15.89:0/210751368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c5015d640 0x7f5c5014fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.659+0000 7f5c56a00700 1 -- 172.21.15.89:0/210751368 >> 172.21.15.89:0/210751368 conn(0x7f5c5008aff0 msgr2=0x7f5c500b7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.659+0000 7f5c56a00700 1 -- 172.21.15.89:0/210751368 shutdown_connections 2024-04-02T09:14:40.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:40.659+0000 7f5c56a00700 1 -- 172.21.15.89:0/210751368 wait complete. 2024-04-02T09:14:40.662 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:14:40.675 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-04-02T09:14:39.258766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":683,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":419072,"kb_used_data":27360,"kb_used_omap":12,"kb_used_meta":391539,"kb_avail":754555648,"statfs":{"total":773094113280,"available":772664983552,"internally_reserved":0,"allocated":28016640,"data_stored":14953910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":400936504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001174"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523216+0000","last_change":"2024-04-02T09:14:34.377961+0000","last_active":"2024-04-02T09:14:35.523216+0000","last_peered":"2024-04-02T09:14:35.523216+0000","last_clean":"2024-04-02T09:14:35.523216+0000","last_became_active":"2024-04-02T09:14:34.377267+0000","last_became_peered":"2024-04-02T09:14:34.377267+0000","last_unstale":"2024-04-02T09:14:35.523216+0000","last_undegraded":"2024-04-02T09:14:35.523216+0000","last_fullsized":"2024-04-02T09:14:35.523216+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:44:06.694478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364561+0000","last_change":"2024-04-02T09:14:34.373205+0000","last_active":"2024-04-02T09:14:35.364561+0000","last_peered":"2024-04-02T09:14:35.364561+0000","last_clean":"2024-04-02T09:14:35.364561+0000","last_became_active":"2024-04-02T09:14:34.370571+0000","last_became_peered":"2024-04-02T09:14:34.370571+0000","last_unstale":"2024-04-02T09:14:35.364561+0000","last_undegraded":"2024-04-02T09:14:35.364561+0000","last_fullsized":"2024-04-02T09:14:35.364561+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:55:48.430034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363862+0000","last_change":"2024-04-02T09:14:34.369016+0000","last_active":"2024-04-02T09:14:35.363862+0000","last_peered":"2024-04-02T09:14:35.363862+0000","last_clean":"2024-04-02T09:14:35.363862+0000","last_became_active":"2024-04-02T09:14:34.368553+0000","last_became_peered":"2024-04-02T09:14:34.368553+0000","last_unstale":"2024-04-02T09:14:35.363862+0000","last_undegraded":"2024-04-02T09:14:35.363862+0000","last_fullsized":"2024-04-02T09:14:35.363862+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:12.477806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522578+0000","last_change":"2024-04-02T09:14:34.383881+0000","last_active":"2024-04-02T09:14:35.522578+0000","last_peered":"2024-04-02T09:14:35.522578+0000","last_clean":"2024-04-02T09:14:35.522578+0000","last_became_active":"2024-04-02T09:14:34.383189+0000","last_became_peered":"2024-04-02T09:14:34.383189+0000","last_unstale":"2024-04-02T09:14:35.522578+0000","last_undegraded":"2024-04-02T09:14:35.522578+0000","last_fullsized":"2024-04-02T09:14:35.522578+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:49:07.747576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522857+0000","last_change":"2024-04-02T09:14:34.384650+0000","last_active":"2024-04-02T09:14:35.522857+0000","last_peered":"2024-04-02T09:14:35.522857+0000","last_clean":"2024-04-02T09:14:35.522857+0000","last_became_active":"2024-04-02T09:14:34.384353+0000","last_became_peered":"2024-04-02T09:14:34.384353+0000","last_unstale":"2024-04-02T09:14:35.522857+0000","last_undegraded":"2024-04-02T09:14:35.522857+0000","last_fullsized":"2024-04-02T09:14:35.522857+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:36.082626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364110+0000","last_change":"2024-04-02T09:14:34.367970+0000","last_active":"2024-04-02T09:14:35.364110+0000","last_peered":"2024-04-02T09:14:35.364110+0000","last_clean":"2024-04-02T09:14:35.364110+0000","last_became_active":"2024-04-02T09:14:34.365750+0000","last_became_peered":"2024-04-02T09:14:34.365750+0000","last_unstale":"2024-04-02T09:14:35.364110+0000","last_undegraded":"2024-04-02T09:14:35.364110+0000","last_fullsized":"2024-04-02T09:14:35.364110+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:32.796452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362566+0000","last_change":"2024-04-02T09:14:34.381480+0000","last_active":"2024-04-02T09:14:35.362566+0000","last_peered":"2024-04-02T09:14:35.362566+0000","last_clean":"2024-04-02T09:14:35.362566+0000","last_became_active":"2024-04-02T09:14:34.381243+0000","last_became_peered":"2024-04-02T09:14:34.381243+0000","last_unstale":"2024-04-02T09:14:35.362566+0000","last_undegraded":"2024-04-02T09:14:35.362566+0000","last_fullsized":"2024-04-02T09:14:35.362566+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:43:52.860015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362432+0000","last_change":"2024-04-02T09:14:34.390416+0000","last_active":"2024-04-02T09:14:35.362432+0000","last_peered":"2024-04-02T09:14:35.362432+0000","last_clean":"2024-04-02T09:14:35.362432+0000","last_became_active":"2024-04-02T09:14:34.390126+0000","last_became_peered":"2024-04-02T09:14:34.390126+0000","last_unstale":"2024-04-02T09:14:35.362432+0000","last_undegraded":"2024-04-02T09:14:35.362432+0000","last_fullsized":"2024-04-02T09:14:35.362432+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:05:04.134051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364695+0000","last_change":"2024-04-02T09:14:34.362861+0000","last_active":"2024-04-02T09:14:35.364695+0000","last_peered":"2024-04-02T09:14:35.364695+0000","last_clean":"2024-04-02T09:14:35.364695+0000","last_became_active":"2024-04-02T09:14:34.362580+0000","last_became_peered":"2024-04-02T09:14:34.362580+0000","last_unstale":"2024-04-02T09:14:35.364695+0000","last_undegraded":"2024-04-02T09:14:35.364695+0000","last_fullsized":"2024-04-02T09:14:35.364695+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:12.477806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362736+0000","last_change":"2024-04-02T09:14:34.380960+0000","last_active":"2024-04-02T09:14:35.362736+0000","last_peered":"2024-04-02T09:14:35.362736+0000","last_clean":"2024-04-02T09:14:35.362736+0000","last_became_active":"2024-04-02T09:14:34.380726+0000","last_became_peered":"2024-04-02T09:14:34.380726+0000","last_unstale":"2024-04-02T09:14:35.362736+0000","last_undegraded":"2024-04-02T09:14:35.362736+0000","last_fullsized":"2024-04-02T09:14:35.362736+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:51:23.449925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522728+0000","last_change":"2024-04-02T09:14:34.382824+0000","last_active":"2024-04-02T09:14:35.522728+0000","last_peered":"2024-04-02T09:14:35.522728+0000","last_clean":"2024-04-02T09:14:35.522728+0000","last_became_active":"2024-04-02T09:14:34.382044+0000","last_became_peered":"2024-04-02T09:14:34.382044+0000","last_unstale":"2024-04-02T09:14:35.522728+0000","last_undegraded":"2024-04-02T09:14:35.522728+0000","last_fullsized":"2024-04-02T09:14:35.522728+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:56.492180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364322+0000","last_change":"2024-04-02T09:14:34.366396+0000","last_active":"2024-04-02T09:14:35.364322+0000","last_peered":"2024-04-02T09:14:35.364322+0000","last_clean":"2024-04-02T09:14:35.364322+0000","last_became_active":"2024-04-02T09:14:34.366115+0000","last_became_peered":"2024-04-02T09:14:34.366115+0000","last_unstale":"2024-04-02T09:14:35.364322+0000","last_undegraded":"2024-04-02T09:14:35.364322+0000","last_fullsized":"2024-04-02T09:14:35.364322+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:56:03.446485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523710+0000","last_change":"2024-04-02T09:14:34.374183+0000","last_active":"2024-04-02T09:14:35.523710+0000","last_peered":"2024-04-02T09:14:35.523710+0000","last_clean":"2024-04-02T09:14:35.523710+0000","last_became_active":"2024-04-02T09:14:34.371570+0000","last_became_peered":"2024-04-02T09:14:34.371570+0000","last_unstale":"2024-04-02T09:14:35.523710+0000","last_undegraded":"2024-04-02T09:14:35.523710+0000","last_fullsized":"2024-04-02T09:14:35.523710+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:06:08.871157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522853+0000","last_change":"2024-04-02T09:14:34.385243+0000","last_active":"2024-04-02T09:14:35.522853+0000","last_peered":"2024-04-02T09:14:35.522853+0000","last_clean":"2024-04-02T09:14:35.522853+0000","last_became_active":"2024-04-02T09:14:34.384989+0000","last_became_peered":"2024-04-02T09:14:34.384989+0000","last_unstale":"2024-04-02T09:14:35.522853+0000","last_undegraded":"2024-04-02T09:14:35.522853+0000","last_fullsized":"2024-04-02T09:14:35.522853+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:28.009617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362467+0000","last_change":"2024-04-02T09:14:34.379614+0000","last_active":"2024-04-02T09:14:35.362467+0000","last_peered":"2024-04-02T09:14:35.362467+0000","last_clean":"2024-04-02T09:14:35.362467+0000","last_became_active":"2024-04-02T09:14:34.378063+0000","last_became_peered":"2024-04-02T09:14:34.378063+0000","last_unstale":"2024-04-02T09:14:35.362467+0000","last_undegraded":"2024-04-02T09:14:35.362467+0000","last_fullsized":"2024-04-02T09:14:35.362467+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:07.213744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522170+0000","last_change":"2024-04-02T09:14:34.373018+0000","last_active":"2024-04-02T09:14:35.522170+0000","last_peered":"2024-04-02T09:14:35.522170+0000","last_clean":"2024-04-02T09:14:35.522170+0000","last_became_active":"2024-04-02T09:14:34.372010+0000","last_became_peered":"2024-04-02T09:14:34.372010+0000","last_unstale":"2024-04-02T09:14:35.522170+0000","last_undegraded":"2024-04-02T09:14:35.522170+0000","last_fullsized":"2024-04-02T09:14:35.522170+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:58.362648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523162+0000","last_change":"2024-04-02T09:14:34.391051+0000","last_active":"2024-04-02T09:14:35.523162+0000","last_peered":"2024-04-02T09:14:35.523162+0000","last_clean":"2024-04-02T09:14:35.523162+0000","last_became_active":"2024-04-02T09:14:34.390721+0000","last_became_peered":"2024-04-02T09:14:34.390721+0000","last_unstale":"2024-04-02T09:14:35.523162+0000","last_undegraded":"2024-04-02T09:14:35.523162+0000","last_fullsized":"2024-04-02T09:14:35.523162+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:53:01.672940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362877+0000","last_change":"2024-04-02T09:14:34.383335+0000","last_active":"2024-04-02T09:14:35.362877+0000","last_peered":"2024-04-02T09:14:35.362877+0000","last_clean":"2024-04-02T09:14:35.362877+0000","last_became_active":"2024-04-02T09:14:34.383105+0000","last_became_peered":"2024-04-02T09:14:34.383105+0000","last_unstale":"2024-04-02T09:14:35.362877+0000","last_undegraded":"2024-04-02T09:14:35.362877+0000","last_fullsized":"2024-04-02T09:14:35.362877+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:08:25.093735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523794+0000","last_change":"2024-04-02T09:14:34.385798+0000","last_active":"2024-04-02T09:14:35.523794+0000","last_peered":"2024-04-02T09:14:35.523794+0000","last_clean":"2024-04-02T09:14:35.523794+0000","last_became_active":"2024-04-02T09:14:34.385264+0000","last_became_peered":"2024-04-02T09:14:34.385264+0000","last_unstale":"2024-04-02T09:14:35.523794+0000","last_undegraded":"2024-04-02T09:14:35.523794+0000","last_fullsized":"2024-04-02T09:14:35.523794+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:41.722545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362562+0000","last_change":"2024-04-02T09:14:34.384422+0000","last_active":"2024-04-02T09:14:35.362562+0000","last_peered":"2024-04-02T09:14:35.362562+0000","last_clean":"2024-04-02T09:14:35.362562+0000","last_became_active":"2024-04-02T09:14:34.384187+0000","last_became_peered":"2024-04-02T09:14:34.384187+0000","last_unstale":"2024-04-02T09:14:35.362562+0000","last_undegraded":"2024-04-02T09:14:35.362562+0000","last_fullsized":"2024-04-02T09:14:35.362562+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:45.536966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523625+0000","last_change":"2024-04-02T09:14:34.375508+0000","last_active":"2024-04-02T09:14:35.523625+0000","last_peered":"2024-04-02T09:14:35.523625+0000","last_clean":"2024-04-02T09:14:35.523625+0000","last_became_active":"2024-04-02T09:14:34.374458+0000","last_became_peered":"2024-04-02T09:14:34.374458+0000","last_unstale":"2024-04-02T09:14:35.523625+0000","last_undegraded":"2024-04-02T09:14:35.523625+0000","last_fullsized":"2024-04-02T09:14:35.523625+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:23.023963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522793+0000","last_change":"2024-04-02T09:14:34.371948+0000","last_active":"2024-04-02T09:14:35.522793+0000","last_peered":"2024-04-02T09:14:35.522793+0000","last_clean":"2024-04-02T09:14:35.522793+0000","last_became_active":"2024-04-02T09:14:34.371586+0000","last_became_peered":"2024-04-02T09:14:34.371586+0000","last_unstale":"2024-04-02T09:14:35.522793+0000","last_undegraded":"2024-04-02T09:14:35.522793+0000","last_fullsized":"2024-04-02T09:14:35.522793+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:27.813013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522861+0000","last_change":"2024-04-02T09:14:34.372317+0000","last_active":"2024-04-02T09:14:35.522861+0000","last_peered":"2024-04-02T09:14:35.522861+0000","last_clean":"2024-04-02T09:14:35.522861+0000","last_became_active":"2024-04-02T09:14:34.371999+0000","last_became_peered":"2024-04-02T09:14:34.371999+0000","last_unstale":"2024-04-02T09:14:35.522861+0000","last_undegraded":"2024-04-02T09:14:35.522861+0000","last_fullsized":"2024-04-02T09:14:35.522861+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:49.154841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363323+0000","last_change":"2024-04-02T09:14:34.369575+0000","last_active":"2024-04-02T09:14:35.363323+0000","last_peered":"2024-04-02T09:14:35.363323+0000","last_clean":"2024-04-02T09:14:35.363323+0000","last_became_active":"2024-04-02T09:14:34.369273+0000","last_became_peered":"2024-04-02T09:14:34.369273+0000","last_unstale":"2024-04-02T09:14:35.363323+0000","last_undegraded":"2024-04-02T09:14:35.363323+0000","last_fullsized":"2024-04-02T09:14:35.363323+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:12:47.137280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523287+0000","last_change":"2024-04-02T09:14:34.372802+0000","last_active":"2024-04-02T09:14:35.523287+0000","last_peered":"2024-04-02T09:14:35.523287+0000","last_clean":"2024-04-02T09:14:35.523287+0000","last_became_active":"2024-04-02T09:14:34.372511+0000","last_became_peered":"2024-04-02T09:14:34.372511+0000","last_unstale":"2024-04-02T09:14:35.523287+0000","last_undegraded":"2024-04-02T09:14:35.523287+0000","last_fullsized":"2024-04-02T09:14:35.523287+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:46.642400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363775+0000","last_change":"2024-04-02T09:14:34.370273+0000","last_active":"2024-04-02T09:14:35.363775+0000","last_peered":"2024-04-02T09:14:35.363775+0000","last_clean":"2024-04-02T09:14:35.363775+0000","last_became_active":"2024-04-02T09:14:34.369221+0000","last_became_peered":"2024-04-02T09:14:34.369221+0000","last_unstale":"2024-04-02T09:14:35.363775+0000","last_undegraded":"2024-04-02T09:14:35.363775+0000","last_fullsized":"2024-04-02T09:14:35.363775+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:32.796452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523112+0000","last_change":"2024-04-02T09:14:34.381402+0000","last_active":"2024-04-02T09:14:35.523112+0000","last_peered":"2024-04-02T09:14:35.523112+0000","last_clean":"2024-04-02T09:14:35.523112+0000","last_became_active":"2024-04-02T09:14:34.381063+0000","last_became_peered":"2024-04-02T09:14:34.381063+0000","last_unstale":"2024-04-02T09:14:35.523112+0000","last_undegraded":"2024-04-02T09:14:35.523112+0000","last_fullsized":"2024-04-02T09:14:35.523112+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:24:11.707496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522665+0000","last_change":"2024-04-02T09:14:34.383686+0000","last_active":"2024-04-02T09:14:35.522665+0000","last_peered":"2024-04-02T09:14:35.522665+0000","last_clean":"2024-04-02T09:14:35.522665+0000","last_became_active":"2024-04-02T09:14:34.382500+0000","last_became_peered":"2024-04-02T09:14:34.382500+0000","last_unstale":"2024-04-02T09:14:35.522665+0000","last_undegraded":"2024-04-02T09:14:35.522665+0000","last_fullsized":"2024-04-02T09:14:35.522665+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:35:17.449940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362773+0000","last_change":"2024-04-02T09:14:34.387782+0000","last_active":"2024-04-02T09:14:35.362773+0000","last_peered":"2024-04-02T09:14:35.362773+0000","last_clean":"2024-04-02T09:14:35.362773+0000","last_became_active":"2024-04-02T09:14:34.383376+0000","last_became_peered":"2024-04-02T09:14:34.383376+0000","last_unstale":"2024-04-02T09:14:35.362773+0000","last_undegraded":"2024-04-02T09:14:35.362773+0000","last_fullsized":"2024-04-02T09:14:35.362773+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:28:13.971403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522423+0000","last_change":"2024-04-02T09:14:34.374368+0000","last_active":"2024-04-02T09:14:35.522423+0000","last_peered":"2024-04-02T09:14:35.522423+0000","last_clean":"2024-04-02T09:14:35.522423+0000","last_became_active":"2024-04-02T09:14:34.374046+0000","last_became_peered":"2024-04-02T09:14:34.374046+0000","last_unstale":"2024-04-02T09:14:35.522423+0000","last_undegraded":"2024-04-02T09:14:35.522423+0000","last_fullsized":"2024-04-02T09:14:35.522423+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:37:45.647925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363599+0000","last_change":"2024-04-02T09:14:34.384135+0000","last_active":"2024-04-02T09:14:35.363599+0000","last_peered":"2024-04-02T09:14:35.363599+0000","last_clean":"2024-04-02T09:14:35.363599+0000","last_became_active":"2024-04-02T09:14:34.383849+0000","last_became_peered":"2024-04-02T09:14:34.383849+0000","last_unstale":"2024-04-02T09:14:35.363599+0000","last_undegraded":"2024-04-02T09:14:35.363599+0000","last_fullsized":"2024-04-02T09:14:35.363599+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:23:36.207832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363879+0000","last_change":"2024-04-02T09:14:34.362600+0000","last_active":"2024-04-02T09:14:35.363879+0000","last_peered":"2024-04-02T09:14:35.363879+0000","last_clean":"2024-04-02T09:14:35.363879+0000","last_became_active":"2024-04-02T09:14:34.362080+0000","last_became_peered":"2024-04-02T09:14:34.362080+0000","last_unstale":"2024-04-02T09:14:35.363879+0000","last_undegraded":"2024-04-02T09:14:35.363879+0000","last_fullsized":"2024-04-02T09:14:35.363879+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:41.728873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"264'4","reported_seq":15,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523394+0000","last_change":"2024-04-02T09:14:33.875502+0000","last_active":"2024-04-02T09:14:35.523394+0000","last_peered":"2024-04-02T09:14:35.523394+0000","last_clean":"2024-04-02T09:14:35.523394+0000","last_became_active":"2024-04-02T09:14:33.875191+0000","last_became_peered":"2024-04-02T09:14:33.875191+0000","last_unstale":"2024-04-02T09:14:35.523394+0000","last_undegraded":"2024-04-02T09:14:35.523394+0000","last_fullsized":"2024-04-02T09:14:35.523394+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:54.914511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523839+0000","last_change":"2024-04-02T09:14:34.372870+0000","last_active":"2024-04-02T09:14:35.523839+0000","last_peered":"2024-04-02T09:14:35.523839+0000","last_clean":"2024-04-02T09:14:35.523839+0000","last_became_active":"2024-04-02T09:14:34.372591+0000","last_became_peered":"2024-04-02T09:14:34.372591+0000","last_unstale":"2024-04-02T09:14:35.523839+0000","last_undegraded":"2024-04-02T09:14:35.523839+0000","last_fullsized":"2024-04-02T09:14:35.523839+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:21.044232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1e","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364686+0000","last_change":"2024-04-02T09:14:33.879018+0000","last_active":"2024-04-02T09:14:35.364686+0000","last_peered":"2024-04-02T09:14:35.364686+0000","last_clean":"2024-04-02T09:14:35.364686+0000","last_became_active":"2024-04-02T09:14:33.877939+0000","last_became_peered":"2024-04-02T09:14:33.877939+0000","last_unstale":"2024-04-02T09:14:35.364686+0000","last_undegraded":"2024-04-02T09:14:35.364686+0000","last_fullsized":"2024-04-02T09:14:35.364686+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:21:10.605008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363660+0000","last_change":"2024-04-02T09:14:34.381630+0000","last_active":"2024-04-02T09:14:35.363660+0000","last_peered":"2024-04-02T09:14:35.363660+0000","last_clean":"2024-04-02T09:14:35.363660+0000","last_became_active":"2024-04-02T09:14:34.381362+0000","last_became_peered":"2024-04-02T09:14:34.381362+0000","last_unstale":"2024-04-02T09:14:35.363660+0000","last_undegraded":"2024-04-02T09:14:35.363660+0000","last_fullsized":"2024-04-02T09:14:35.363660+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:17:46.076532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1d","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364496+0000","last_change":"2024-04-02T09:14:33.877689+0000","last_active":"2024-04-02T09:14:35.364496+0000","last_peered":"2024-04-02T09:14:35.364496+0000","last_clean":"2024-04-02T09:14:35.364496+0000","last_became_active":"2024-04-02T09:14:33.875355+0000","last_became_peered":"2024-04-02T09:14:33.875355+0000","last_unstale":"2024-04-02T09:14:35.364496+0000","last_undegraded":"2024-04-02T09:14:35.364496+0000","last_fullsized":"2024-04-02T09:14:35.364496+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:53:35.396685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364415+0000","last_change":"2024-04-02T09:14:34.384893+0000","last_active":"2024-04-02T09:14:35.364415+0000","last_peered":"2024-04-02T09:14:35.364415+0000","last_clean":"2024-04-02T09:14:35.364415+0000","last_became_active":"2024-04-02T09:14:34.384622+0000","last_became_peered":"2024-04-02T09:14:34.384622+0000","last_unstale":"2024-04-02T09:14:35.364415+0000","last_undegraded":"2024-04-02T09:14:35.364415+0000","last_fullsized":"2024-04-02T09:14:35.364415+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:13:28.694192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1c","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523485+0000","last_change":"2024-04-02T09:14:33.860753+0000","last_active":"2024-04-02T09:14:35.523485+0000","last_peered":"2024-04-02T09:14:35.523485+0000","last_clean":"2024-04-02T09:14:35.523485+0000","last_became_active":"2024-04-02T09:14:33.860419+0000","last_became_peered":"2024-04-02T09:14:33.860419+0000","last_unstale":"2024-04-02T09:14:35.523485+0000","last_undegraded":"2024-04-02T09:14:35.523485+0000","last_fullsized":"2024-04-02T09:14:35.523485+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:54.064730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364273+0000","last_change":"2024-04-02T09:14:34.373334+0000","last_active":"2024-04-02T09:14:35.364273+0000","last_peered":"2024-04-02T09:14:35.364273+0000","last_clean":"2024-04-02T09:14:35.364273+0000","last_became_active":"2024-04-02T09:14:34.372987+0000","last_became_peered":"2024-04-02T09:14:34.372987+0000","last_unstale":"2024-04-02T09:14:35.364273+0000","last_undegraded":"2024-04-02T09:14:35.364273+0000","last_fullsized":"2024-04-02T09:14:35.364273+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:42.139739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1b","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523580+0000","last_change":"2024-04-02T09:14:33.874800+0000","last_active":"2024-04-02T09:14:35.523580+0000","last_peered":"2024-04-02T09:14:35.523580+0000","last_clean":"2024-04-02T09:14:35.523580+0000","last_became_active":"2024-04-02T09:14:33.874504+0000","last_became_peered":"2024-04-02T09:14:33.874504+0000","last_unstale":"2024-04-02T09:14:35.523580+0000","last_undegraded":"2024-04-02T09:14:35.523580+0000","last_fullsized":"2024-04-02T09:14:35.523580+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:11.515065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523030+0000","last_change":"2024-04-02T09:14:34.362124+0000","last_active":"2024-04-02T09:14:35.523030+0000","last_peered":"2024-04-02T09:14:35.523030+0000","last_clean":"2024-04-02T09:14:35.523030+0000","last_became_active":"2024-04-02T09:14:34.361786+0000","last_became_peered":"2024-04-02T09:14:34.361786+0000","last_unstale":"2024-04-02T09:14:35.523030+0000","last_undegraded":"2024-04-02T09:14:35.523030+0000","last_fullsized":"2024-04-02T09:14:35.523030+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:42:12.604946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1a","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362815+0000","last_change":"2024-04-02T09:14:33.877648+0000","last_active":"2024-04-02T09:14:35.362815+0000","last_peered":"2024-04-02T09:14:35.362815+0000","last_clean":"2024-04-02T09:14:35.362815+0000","last_became_active":"2024-04-02T09:14:33.877308+0000","last_became_peered":"2024-04-02T09:14:33.877308+0000","last_unstale":"2024-04-02T09:14:35.362815+0000","last_undegraded":"2024-04-02T09:14:35.362815+0000","last_fullsized":"2024-04-02T09:14:35.362815+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:31.787918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363331+0000","last_change":"2024-04-02T09:14:34.378176+0000","last_active":"2024-04-02T09:14:35.363331+0000","last_peered":"2024-04-02T09:14:35.363331+0000","last_clean":"2024-04-02T09:14:35.363331+0000","last_became_active":"2024-04-02T09:14:34.376201+0000","last_became_peered":"2024-04-02T09:14:34.376201+0000","last_unstale":"2024-04-02T09:14:35.363331+0000","last_undegraded":"2024-04-02T09:14:35.363331+0000","last_fullsized":"2024-04-02T09:14:35.363331+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:55.923045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522590+0000","last_change":"2024-04-02T09:14:33.869026+0000","last_active":"2024-04-02T09:14:35.522590+0000","last_peered":"2024-04-02T09:14:35.522590+0000","last_clean":"2024-04-02T09:14:35.522590+0000","last_became_active":"2024-04-02T09:14:33.868743+0000","last_became_peered":"2024-04-02T09:14:33.868743+0000","last_unstale":"2024-04-02T09:14:35.522590+0000","last_undegraded":"2024-04-02T09:14:35.522590+0000","last_fullsized":"2024-04-02T09:14:35.522590+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:31.787918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522526+0000","last_change":"2024-04-02T09:14:34.376819+0000","last_active":"2024-04-02T09:14:35.522526+0000","last_peered":"2024-04-02T09:14:35.522526+0000","last_clean":"2024-04-02T09:14:35.522526+0000","last_became_active":"2024-04-02T09:14:34.376555+0000","last_became_peered":"2024-04-02T09:14:34.376555+0000","last_unstale":"2024-04-02T09:14:35.522526+0000","last_undegraded":"2024-04-02T09:14:35.522526+0000","last_fullsized":"2024-04-02T09:14:35.522526+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:36:25.999949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363666+0000","last_change":"2024-04-02T09:14:33.879857+0000","last_active":"2024-04-02T09:14:35.363666+0000","last_peered":"2024-04-02T09:14:35.363666+0000","last_clean":"2024-04-02T09:14:35.363666+0000","last_became_active":"2024-04-02T09:14:33.879161+0000","last_became_peered":"2024-04-02T09:14:33.879161+0000","last_unstale":"2024-04-02T09:14:35.363666+0000","last_undegraded":"2024-04-02T09:14:35.363666+0000","last_fullsized":"2024-04-02T09:14:35.363666+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:28:14.207222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362885+0000","last_change":"2024-04-02T09:14:34.389470+0000","last_active":"2024-04-02T09:14:35.362885+0000","last_peered":"2024-04-02T09:14:35.362885+0000","last_clean":"2024-04-02T09:14:35.362885+0000","last_became_active":"2024-04-02T09:14:34.388852+0000","last_became_peered":"2024-04-02T09:14:34.388852+0000","last_unstale":"2024-04-02T09:14:35.362885+0000","last_undegraded":"2024-04-02T09:14:35.362885+0000","last_fullsized":"2024-04-02T09:14:35.362885+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:15.565863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523327+0000","last_change":"2024-04-02T09:14:33.869704+0000","last_active":"2024-04-02T09:14:35.523327+0000","last_peered":"2024-04-02T09:14:35.523327+0000","last_clean":"2024-04-02T09:14:35.523327+0000","last_became_active":"2024-04-02T09:14:33.868809+0000","last_became_peered":"2024-04-02T09:14:33.868809+0000","last_unstale":"2024-04-02T09:14:35.523327+0000","last_undegraded":"2024-04-02T09:14:35.523327+0000","last_fullsized":"2024-04-02T09:14:35.523327+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:12:46.128746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523864+0000","last_change":"2024-04-02T09:14:34.379506+0000","last_active":"2024-04-02T09:14:35.523864+0000","last_peered":"2024-04-02T09:14:35.523864+0000","last_clean":"2024-04-02T09:14:35.523864+0000","last_became_active":"2024-04-02T09:14:34.378787+0000","last_became_peered":"2024-04-02T09:14:34.378787+0000","last_unstale":"2024-04-02T09:14:35.523864+0000","last_undegraded":"2024-04-02T09:14:35.523864+0000","last_fullsized":"2024-04-02T09:14:35.523864+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:13:47.277991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523512+0000","last_change":"2024-04-02T09:14:33.874472+0000","last_active":"2024-04-02T09:14:35.523512+0000","last_peered":"2024-04-02T09:14:35.523512+0000","last_clean":"2024-04-02T09:14:35.523512+0000","last_became_active":"2024-04-02T09:14:33.874190+0000","last_became_peered":"2024-04-02T09:14:33.874190+0000","last_unstale":"2024-04-02T09:14:35.523512+0000","last_undegraded":"2024-04-02T09:14:35.523512+0000","last_fullsized":"2024-04-02T09:14:35.523512+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:06:40.311963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522771+0000","last_change":"2024-04-02T09:14:34.382684+0000","last_active":"2024-04-02T09:14:35.522771+0000","last_peered":"2024-04-02T09:14:35.522771+0000","last_clean":"2024-04-02T09:14:35.522771+0000","last_became_active":"2024-04-02T09:14:34.380586+0000","last_became_peered":"2024-04-02T09:14:34.380586+0000","last_unstale":"2024-04-02T09:14:35.522771+0000","last_undegraded":"2024-04-02T09:14:35.522771+0000","last_fullsized":"2024-04-02T09:14:35.522771+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:56:58.056857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.15","version":"264'3","reported_seq":14,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364540+0000","last_change":"2024-04-02T09:14:33.878269+0000","last_active":"2024-04-02T09:14:35.364540+0000","last_peered":"2024-04-02T09:14:35.364540+0000","last_clean":"2024-04-02T09:14:35.364540+0000","last_became_active":"2024-04-02T09:14:33.877496+0000","last_became_peered":"2024-04-02T09:14:33.877496+0000","last_unstale":"2024-04-02T09:14:35.364540+0000","last_undegraded":"2024-04-02T09:14:35.364540+0000","last_fullsized":"2024-04-02T09:14:35.364540+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:11:32.742344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523334+0000","last_change":"2024-04-02T09:14:34.372160+0000","last_active":"2024-04-02T09:14:35.523334+0000","last_peered":"2024-04-02T09:14:35.523334+0000","last_clean":"2024-04-02T09:14:35.523334+0000","last_became_active":"2024-04-02T09:14:34.371325+0000","last_became_peered":"2024-04-02T09:14:34.371325+0000","last_unstale":"2024-04-02T09:14:35.523334+0000","last_undegraded":"2024-04-02T09:14:35.523334+0000","last_fullsized":"2024-04-02T09:14:35.523334+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:36.082626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"264'6","reported_seq":17,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523450+0000","last_change":"2024-04-02T09:14:33.875419+0000","last_active":"2024-04-02T09:14:35.523450+0000","last_peered":"2024-04-02T09:14:35.523450+0000","last_clean":"2024-04-02T09:14:35.523450+0000","last_became_active":"2024-04-02T09:14:33.874433+0000","last_became_peered":"2024-04-02T09:14:33.874433+0000","last_unstale":"2024-04-02T09:14:35.523450+0000","last_undegraded":"2024-04-02T09:14:35.523450+0000","last_fullsized":"2024-04-02T09:14:35.523450+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:07:50.116863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522576+0000","last_change":"2024-04-02T09:14:34.374573+0000","last_active":"2024-04-02T09:14:35.522576+0000","last_peered":"2024-04-02T09:14:35.522576+0000","last_clean":"2024-04-02T09:14:35.522576+0000","last_became_active":"2024-04-02T09:14:34.374327+0000","last_became_peered":"2024-04-02T09:14:34.374327+0000","last_unstale":"2024-04-02T09:14:35.522576+0000","last_undegraded":"2024-04-02T09:14:35.522576+0000","last_fullsized":"2024-04-02T09:14:35.522576+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:31:00.788937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.13","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523486+0000","last_change":"2024-04-02T09:14:33.877528+0000","last_active":"2024-04-02T09:14:35.523486+0000","last_peered":"2024-04-02T09:14:35.523486+0000","last_clean":"2024-04-02T09:14:35.523486+0000","last_became_active":"2024-04-02T09:14:33.877262+0000","last_became_peered":"2024-04-02T09:14:33.877262+0000","last_unstale":"2024-04-02T09:14:35.523486+0000","last_undegraded":"2024-04-02T09:14:35.523486+0000","last_fullsized":"2024-04-02T09:14:35.523486+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:06:40.311963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522781+0000","last_change":"2024-04-02T09:14:34.380073+0000","last_active":"2024-04-02T09:14:35.522781+0000","last_peered":"2024-04-02T09:14:35.522781+0000","last_clean":"2024-04-02T09:14:35.522781+0000","last_became_active":"2024-04-02T09:14:34.379603+0000","last_became_peered":"2024-04-02T09:14:34.379603+0000","last_unstale":"2024-04-02T09:14:35.522781+0000","last_undegraded":"2024-04-02T09:14:35.522781+0000","last_fullsized":"2024-04-02T09:14:35.522781+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:45.536966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364291+0000","last_change":"2024-04-02T09:14:33.876709+0000","last_active":"2024-04-02T09:14:35.364291+0000","last_peered":"2024-04-02T09:14:35.364291+0000","last_clean":"2024-04-02T09:14:35.364291+0000","last_became_active":"2024-04-02T09:14:33.876371+0000","last_became_peered":"2024-04-02T09:14:33.876371+0000","last_unstale":"2024-04-02T09:14:35.364291+0000","last_undegraded":"2024-04-02T09:14:35.364291+0000","last_fullsized":"2024-04-02T09:14:35.364291+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:32:30.574385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522551+0000","last_change":"2024-04-02T09:14:34.381595+0000","last_active":"2024-04-02T09:14:35.522551+0000","last_peered":"2024-04-02T09:14:35.522551+0000","last_clean":"2024-04-02T09:14:35.522551+0000","last_became_active":"2024-04-02T09:14:34.381321+0000","last_became_peered":"2024-04-02T09:14:34.381321+0000","last_unstale":"2024-04-02T09:14:35.522551+0000","last_undegraded":"2024-04-02T09:14:35.522551+0000","last_fullsized":"2024-04-02T09:14:35.522551+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:27.813013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522721+0000","last_change":"2024-04-02T09:14:33.869087+0000","last_active":"2024-04-02T09:14:35.522721+0000","last_peered":"2024-04-02T09:14:35.522721+0000","last_clean":"2024-04-02T09:14:35.522721+0000","last_became_active":"2024-04-02T09:14:33.868778+0000","last_became_peered":"2024-04-02T09:14:33.868778+0000","last_unstale":"2024-04-02T09:14:35.522721+0000","last_undegraded":"2024-04-02T09:14:35.522721+0000","last_fullsized":"2024-04-02T09:14:35.522721+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:53:25.730047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522621+0000","last_change":"2024-04-02T09:14:34.382115+0000","last_active":"2024-04-02T09:14:35.522621+0000","last_peered":"2024-04-02T09:14:35.522621+0000","last_clean":"2024-04-02T09:14:35.522621+0000","last_became_active":"2024-04-02T09:14:34.381857+0000","last_became_peered":"2024-04-02T09:14:34.381857+0000","last_unstale":"2024-04-02T09:14:35.522621+0000","last_undegraded":"2024-04-02T09:14:35.522621+0000","last_fullsized":"2024-04-02T09:14:35.522621+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:43:47.084525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.10","version":"264'8","reported_seq":19,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522720+0000","last_change":"2024-04-02T09:14:33.869954+0000","last_active":"2024-04-02T09:14:35.522720+0000","last_peered":"2024-04-02T09:14:35.522720+0000","last_clean":"2024-04-02T09:14:35.522720+0000","last_became_active":"2024-04-02T09:14:33.867537+0000","last_became_peered":"2024-04-02T09:14:33.867537+0000","last_unstale":"2024-04-02T09:14:35.522720+0000","last_undegraded":"2024-04-02T09:14:35.522720+0000","last_fullsized":"2024-04-02T09:14:35.522720+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:36.927662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364624+0000","last_change":"2024-04-02T09:14:34.373375+0000","last_active":"2024-04-02T09:14:35.364624+0000","last_peered":"2024-04-02T09:14:35.364624+0000","last_clean":"2024-04-02T09:14:35.364624+0000","last_became_active":"2024-04-02T09:14:34.370488+0000","last_became_peered":"2024-04-02T09:14:34.370488+0000","last_unstale":"2024-04-02T09:14:35.364624+0000","last_undegraded":"2024-04-02T09:14:35.364624+0000","last_fullsized":"2024-04-02T09:14:35.364624+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:32:00.070021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.f","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523408+0000","last_change":"2024-04-02T09:14:33.869073+0000","last_active":"2024-04-02T09:14:35.523408+0000","last_peered":"2024-04-02T09:14:35.523408+0000","last_clean":"2024-04-02T09:14:35.523408+0000","last_became_active":"2024-04-02T09:14:33.868799+0000","last_became_peered":"2024-04-02T09:14:33.868799+0000","last_unstale":"2024-04-02T09:14:35.523408+0000","last_undegraded":"2024-04-02T09:14:35.523408+0000","last_fullsized":"2024-04-02T09:14:35.523408+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:56:02.437951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523338+0000","last_change":"2024-04-02T09:14:34.381984+0000","last_active":"2024-04-02T09:14:35.523338+0000","last_peered":"2024-04-02T09:14:35.523338+0000","last_clean":"2024-04-02T09:14:35.523338+0000","last_became_active":"2024-04-02T09:14:34.381724+0000","last_became_peered":"2024-04-02T09:14:34.381724+0000","last_unstale":"2024-04-02T09:14:35.523338+0000","last_undegraded":"2024-04-02T09:14:35.523338+0000","last_fullsized":"2024-04-02T09:14:35.523338+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:10:43.121011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.e","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523146+0000","last_change":"2024-04-02T09:14:33.877468+0000","last_active":"2024-04-02T09:14:35.523146+0000","last_peered":"2024-04-02T09:14:35.523146+0000","last_clean":"2024-04-02T09:14:35.523146+0000","last_became_active":"2024-04-02T09:14:33.877199+0000","last_became_peered":"2024-04-02T09:14:33.877199+0000","last_unstale":"2024-04-02T09:14:35.523146+0000","last_undegraded":"2024-04-02T09:14:35.523146+0000","last_fullsized":"2024-04-02T09:14:35.523146+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:11.469272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523352+0000","last_change":"2024-04-02T09:14:34.373693+0000","last_active":"2024-04-02T09:14:35.523352+0000","last_peered":"2024-04-02T09:14:35.523352+0000","last_clean":"2024-04-02T09:14:35.523352+0000","last_became_active":"2024-04-02T09:14:34.372531+0000","last_became_peered":"2024-04-02T09:14:34.372531+0000","last_unstale":"2024-04-02T09:14:35.523352+0000","last_undegraded":"2024-04-02T09:14:35.523352+0000","last_fullsized":"2024-04-02T09:14:35.523352+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:24:29.851257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.d","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523408+0000","last_change":"2024-04-02T09:14:33.874303+0000","last_active":"2024-04-02T09:14:35.523408+0000","last_peered":"2024-04-02T09:14:35.523408+0000","last_clean":"2024-04-02T09:14:35.523408+0000","last_became_active":"2024-04-02T09:14:33.874024+0000","last_became_peered":"2024-04-02T09:14:33.874024+0000","last_unstale":"2024-04-02T09:14:35.523408+0000","last_undegraded":"2024-04-02T09:14:35.523408+0000","last_fullsized":"2024-04-02T09:14:35.523408+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:23:35.199298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523471+0000","last_change":"2024-04-02T09:14:34.373452+0000","last_active":"2024-04-02T09:14:35.523471+0000","last_peered":"2024-04-02T09:14:35.523471+0000","last_clean":"2024-04-02T09:14:35.523471+0000","last_became_active":"2024-04-02T09:14:34.372772+0000","last_became_peered":"2024-04-02T09:14:34.372772+0000","last_unstale":"2024-04-02T09:14:35.523471+0000","last_undegraded":"2024-04-02T09:14:35.523471+0000","last_fullsized":"2024-04-02T09:14:35.523471+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:41.722545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523664+0000","last_change":"2024-04-02T09:14:33.863647+0000","last_active":"2024-04-02T09:14:35.523664+0000","last_peered":"2024-04-02T09:14:35.523664+0000","last_clean":"2024-04-02T09:14:35.523664+0000","last_became_active":"2024-04-02T09:14:33.863385+0000","last_became_peered":"2024-04-02T09:14:33.863385+0000","last_unstale":"2024-04-02T09:14:35.523664+0000","last_undegraded":"2024-04-02T09:14:35.523664+0000","last_fullsized":"2024-04-02T09:14:35.523664+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:57:55.628012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523120+0000","last_change":"2024-04-02T09:14:34.390484+0000","last_active":"2024-04-02T09:14:35.523120+0000","last_peered":"2024-04-02T09:14:35.523120+0000","last_clean":"2024-04-02T09:14:35.523120+0000","last_became_active":"2024-04-02T09:14:34.390145+0000","last_became_peered":"2024-04-02T09:14:34.390145+0000","last_unstale":"2024-04-02T09:14:35.523120+0000","last_undegraded":"2024-04-02T09:14:35.523120+0000","last_fullsized":"2024-04-02T09:14:35.523120+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:43:52.860015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.b","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364214+0000","last_change":"2024-04-02T09:14:33.876355+0000","last_active":"2024-04-02T09:14:35.364214+0000","last_peered":"2024-04-02T09:14:35.364214+0000","last_clean":"2024-04-02T09:14:35.364214+0000","last_became_active":"2024-04-02T09:14:33.876041+0000","last_became_peered":"2024-04-02T09:14:33.876041+0000","last_unstale":"2024-04-02T09:14:35.364214+0000","last_undegraded":"2024-04-02T09:14:35.364214+0000","last_fullsized":"2024-04-02T09:14:35.364214+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:10:38.419248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362885+0000","last_change":"2024-04-02T09:14:34.388100+0000","last_active":"2024-04-02T09:14:35.362885+0000","last_peered":"2024-04-02T09:14:35.362885+0000","last_clean":"2024-04-02T09:14:35.362885+0000","last_became_active":"2024-04-02T09:14:34.383116+0000","last_became_peered":"2024-04-02T09:14:34.383116+0000","last_unstale":"2024-04-02T09:14:35.362885+0000","last_undegraded":"2024-04-02T09:14:35.362885+0000","last_fullsized":"2024-04-02T09:14:35.362885+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:51.893190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364368+0000","last_change":"2024-04-02T09:14:33.878926+0000","last_active":"2024-04-02T09:14:35.364368+0000","last_peered":"2024-04-02T09:14:35.364368+0000","last_clean":"2024-04-02T09:14:35.364368+0000","last_became_active":"2024-04-02T09:14:33.878596+0000","last_became_peered":"2024-04-02T09:14:33.878596+0000","last_unstale":"2024-04-02T09:14:35.364368+0000","last_undegraded":"2024-04-02T09:14:35.364368+0000","last_fullsized":"2024-04-02T09:14:35.364368+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:14:19.001834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523585+0000","last_change":"2024-04-02T09:14:34.374525+0000","last_active":"2024-04-02T09:14:35.523585+0000","last_peered":"2024-04-02T09:14:35.523585+0000","last_clean":"2024-04-02T09:14:35.523585+0000","last_became_active":"2024-04-02T09:14:34.374214+0000","last_became_peered":"2024-04-02T09:14:34.374214+0000","last_unstale":"2024-04-02T09:14:35.523585+0000","last_undegraded":"2024-04-02T09:14:35.523585+0000","last_fullsized":"2024-04-02T09:14:35.523585+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:20:30.074905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362662+0000","last_change":"2024-04-02T09:14:33.872263+0000","last_active":"2024-04-02T09:14:35.362662+0000","last_peered":"2024-04-02T09:14:35.362662+0000","last_clean":"2024-04-02T09:14:35.362662+0000","last_became_active":"2024-04-02T09:14:33.871830+0000","last_became_peered":"2024-04-02T09:14:33.871830+0000","last_unstale":"2024-04-02T09:14:35.362662+0000","last_undegraded":"2024-04-02T09:14:35.362662+0000","last_fullsized":"2024-04-02T09:14:35.362662+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:11.469272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522704+0000","last_change":"2024-04-02T09:14:34.374102+0000","last_active":"2024-04-02T09:14:35.522704+0000","last_peered":"2024-04-02T09:14:35.522704+0000","last_clean":"2024-04-02T09:14:35.522704+0000","last_became_active":"2024-04-02T09:14:34.373810+0000","last_became_peered":"2024-04-02T09:14:34.373810+0000","last_unstale":"2024-04-02T09:14:35.522704+0000","last_undegraded":"2024-04-02T09:14:35.522704+0000","last_fullsized":"2024-04-02T09:14:35.522704+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:20:30.074905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363497+0000","last_change":"2024-04-02T09:14:33.867489+0000","last_active":"2024-04-02T09:14:35.363497+0000","last_peered":"2024-04-02T09:14:35.363497+0000","last_clean":"2024-04-02T09:14:35.363497+0000","last_became_active":"2024-04-02T09:14:33.867206+0000","last_became_peered":"2024-04-02T09:14:33.867206+0000","last_unstale":"2024-04-02T09:14:35.363497+0000","last_undegraded":"2024-04-02T09:14:35.363497+0000","last_fullsized":"2024-04-02T09:14:35.363497+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:55:44.507753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362972+0000","last_change":"2024-04-02T09:14:34.380478+0000","last_active":"2024-04-02T09:14:35.362972+0000","last_peered":"2024-04-02T09:14:35.362972+0000","last_clean":"2024-04-02T09:14:35.362972+0000","last_became_active":"2024-04-02T09:14:34.380228+0000","last_became_peered":"2024-04-02T09:14:34.380228+0000","last_unstale":"2024-04-02T09:14:35.362972+0000","last_undegraded":"2024-04-02T09:14:35.362972+0000","last_fullsized":"2024-04-02T09:14:35.362972+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:28.009617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.7","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362982+0000","last_change":"2024-04-02T09:14:33.878819+0000","last_active":"2024-04-02T09:14:35.362982+0000","last_peered":"2024-04-02T09:14:35.362982+0000","last_clean":"2024-04-02T09:14:35.362982+0000","last_became_active":"2024-04-02T09:14:33.878105+0000","last_became_peered":"2024-04-02T09:14:33.878105+0000","last_unstale":"2024-04-02T09:14:35.362982+0000","last_undegraded":"2024-04-02T09:14:35.362982+0000","last_fullsized":"2024-04-02T09:14:35.362982+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:17:45.067998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523932+0000","last_change":"2024-04-02T09:14:34.372991+0000","last_active":"2024-04-02T09:14:35.523932+0000","last_peered":"2024-04-02T09:14:35.523932+0000","last_clean":"2024-04-02T09:14:35.523932+0000","last_became_active":"2024-04-02T09:14:34.372739+0000","last_became_peered":"2024-04-02T09:14:34.372739+0000","last_unstale":"2024-04-02T09:14:35.523932+0000","last_undegraded":"2024-04-02T09:14:35.523932+0000","last_fullsized":"2024-04-02T09:14:35.523932+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:26.825924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523722+0000","last_change":"2024-04-02T09:14:33.871269+0000","last_active":"2024-04-02T09:14:35.523722+0000","last_peered":"2024-04-02T09:14:35.523722+0000","last_clean":"2024-04-02T09:14:35.523722+0000","last_became_active":"2024-04-02T09:14:33.870767+0000","last_became_peered":"2024-04-02T09:14:33.870767+0000","last_unstale":"2024-04-02T09:14:35.523722+0000","last_undegraded":"2024-04-02T09:14:35.523722+0000","last_fullsized":"2024-04-02T09:14:35.523722+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:55:47.421500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362682+0000","last_change":"2024-04-02T09:14:34.382888+0000","last_active":"2024-04-02T09:14:35.362682+0000","last_peered":"2024-04-02T09:14:35.362682+0000","last_clean":"2024-04-02T09:14:35.362682+0000","last_became_active":"2024-04-02T09:14:34.382551+0000","last_became_peered":"2024-04-02T09:14:34.382551+0000","last_unstale":"2024-04-02T09:14:35.362682+0000","last_undegraded":"2024-04-02T09:14:35.362682+0000","last_fullsized":"2024-04-02T09:14:35.362682+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:36.082626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362917+0000","last_change":"2024-04-02T09:14:33.876070+0000","last_active":"2024-04-02T09:14:35.362917+0000","last_peered":"2024-04-02T09:14:35.362917+0000","last_clean":"2024-04-02T09:14:35.362917+0000","last_became_active":"2024-04-02T09:14:33.875825+0000","last_became_peered":"2024-04-02T09:14:33.875825+0000","last_unstale":"2024-04-02T09:14:35.362917+0000","last_undegraded":"2024-04-02T09:14:35.362917+0000","last_fullsized":"2024-04-02T09:14:35.362917+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:23:35.199298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363906+0000","last_change":"2024-04-02T09:14:34.362399+0000","last_active":"2024-04-02T09:14:35.363906+0000","last_peered":"2024-04-02T09:14:35.363906+0000","last_clean":"2024-04-02T09:14:35.363906+0000","last_became_active":"2024-04-02T09:14:34.361978+0000","last_became_peered":"2024-04-02T09:14:34.361978+0000","last_unstale":"2024-04-02T09:14:35.363906+0000","last_undegraded":"2024-04-02T09:14:35.363906+0000","last_fullsized":"2024-04-02T09:14:35.363906+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:15.964840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523592+0000","last_change":"2024-04-02T09:14:33.866042+0000","last_active":"2024-04-02T09:14:35.523592+0000","last_peered":"2024-04-02T09:14:35.523592+0000","last_clean":"2024-04-02T09:14:35.523592+0000","last_became_active":"2024-04-02T09:14:33.865757+0000","last_became_peered":"2024-04-02T09:14:33.865757+0000","last_unstale":"2024-04-02T09:14:35.523592+0000","last_undegraded":"2024-04-02T09:14:35.523592+0000","last_fullsized":"2024-04-02T09:14:35.523592+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:24.796757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522937+0000","last_change":"2024-04-02T09:14:34.375746+0000","last_active":"2024-04-02T09:14:35.522937+0000","last_peered":"2024-04-02T09:14:35.522937+0000","last_clean":"2024-04-02T09:14:35.522937+0000","last_became_active":"2024-04-02T09:14:34.375131+0000","last_became_peered":"2024-04-02T09:14:34.375131+0000","last_unstale":"2024-04-02T09:14:35.522937+0000","last_undegraded":"2024-04-02T09:14:35.522937+0000","last_fullsized":"2024-04-02T09:14:35.522937+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:03:24.411151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523040+0000","last_change":"2024-04-02T09:14:33.868955+0000","last_active":"2024-04-02T09:14:35.523040+0000","last_peered":"2024-04-02T09:14:35.523040+0000","last_clean":"2024-04-02T09:14:35.523040+0000","last_became_active":"2024-04-02T09:14:33.868639+0000","last_became_peered":"2024-04-02T09:14:33.868639+0000","last_unstale":"2024-04-02T09:14:35.523040+0000","last_undegraded":"2024-04-02T09:14:35.523040+0000","last_fullsized":"2024-04-02T09:14:35.523040+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:39:56.716397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522916+0000","last_change":"2024-04-02T09:14:34.379918+0000","last_active":"2024-04-02T09:14:35.522916+0000","last_peered":"2024-04-02T09:14:35.522916+0000","last_clean":"2024-04-02T09:14:35.522916+0000","last_became_active":"2024-04-02T09:14:34.379284+0000","last_became_peered":"2024-04-02T09:14:34.379284+0000","last_unstale":"2024-04-02T09:14:35.522916+0000","last_undegraded":"2024-04-02T09:14:35.522916+0000","last_fullsized":"2024-04-02T09:14:35.522916+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:53:01.672940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522856+0000","last_change":"2024-04-02T09:14:33.870645+0000","last_active":"2024-04-02T09:14:35.522856+0000","last_peered":"2024-04-02T09:14:35.522856+0000","last_clean":"2024-04-02T09:14:35.522856+0000","last_became_active":"2024-04-02T09:14:33.870355+0000","last_became_peered":"2024-04-02T09:14:33.870355+0000","last_unstale":"2024-04-02T09:14:35.522856+0000","last_undegraded":"2024-04-02T09:14:35.522856+0000","last_fullsized":"2024-04-02T09:14:35.522856+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:53:25.730047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523681+0000","last_change":"2024-04-02T09:14:34.370359+0000","last_active":"2024-04-02T09:14:35.523681+0000","last_peered":"2024-04-02T09:14:35.523681+0000","last_clean":"2024-04-02T09:14:35.523681+0000","last_became_active":"2024-04-02T09:14:34.370115+0000","last_became_peered":"2024-04-02T09:14:34.370115+0000","last_unstale":"2024-04-02T09:14:35.523681+0000","last_undegraded":"2024-04-02T09:14:35.523681+0000","last_fullsized":"2024-04-02T09:14:35.523681+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:43:52.860015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522693+0000","last_change":"2024-04-02T09:14:33.865894+0000","last_active":"2024-04-02T09:14:35.522693+0000","last_peered":"2024-04-02T09:14:35.522693+0000","last_clean":"2024-04-02T09:14:35.522693+0000","last_became_active":"2024-04-02T09:14:33.865610+0000","last_became_peered":"2024-04-02T09:14:33.865610+0000","last_unstale":"2024-04-02T09:14:35.522693+0000","last_undegraded":"2024-04-02T09:14:35.522693+0000","last_fullsized":"2024-04-02T09:14:35.522693+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:41.131205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522925+0000","last_change":"2024-04-02T09:14:34.372238+0000","last_active":"2024-04-02T09:14:35.522925+0000","last_peered":"2024-04-02T09:14:35.522925+0000","last_clean":"2024-04-02T09:14:35.522925+0000","last_became_active":"2024-04-02T09:14:34.371553+0000","last_became_peered":"2024-04-02T09:14:34.371553+0000","last_unstale":"2024-04-02T09:14:35.522925+0000","last_undegraded":"2024-04-02T09:14:35.522925+0000","last_fullsized":"2024-04-02T09:14:35.522925+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:03:24.411151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522812+0000","last_change":"2024-04-02T09:14:33.874635+0000","last_active":"2024-04-02T09:14:35.522812+0000","last_peered":"2024-04-02T09:14:35.522812+0000","last_clean":"2024-04-02T09:14:35.522812+0000","last_became_active":"2024-04-02T09:14:33.874348+0000","last_became_peered":"2024-04-02T09:14:33.874348+0000","last_unstale":"2024-04-02T09:14:35.522812+0000","last_undegraded":"2024-04-02T09:14:35.522812+0000","last_fullsized":"2024-04-02T09:14:35.522812+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:18:07.343969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362709+0000","last_change":"2024-04-02T09:14:34.387134+0000","last_active":"2024-04-02T09:14:35.362709+0000","last_peered":"2024-04-02T09:14:35.362709+0000","last_clean":"2024-04-02T09:14:35.362709+0000","last_became_active":"2024-04-02T09:14:34.386882+0000","last_became_peered":"2024-04-02T09:14:34.386882+0000","last_unstale":"2024-04-02T09:14:35.362709+0000","last_undegraded":"2024-04-02T09:14:35.362709+0000","last_fullsized":"2024-04-02T09:14:35.362709+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:44:06.694478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"250'381","reported_seq":1365,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-04-02T09:14:34.367154+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:14:34.367154+0000","last_peered":"2024-04-02T09:14:34.367154+0000","last_clean":"2024-04-02T09:14:34.367154+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:14:34.367154+0000","last_undegraded":"2024-04-02T09:14:34.367154+0000","last_fullsized":"2024-04-02T09:14:34.367154+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576009,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60644,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311196,"statfs":{"total":96636764160,"available":96574664704,"internally_reserved":0,"allocated":3358720,"data_stored":1721457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.70899999999999996,"15min":0.71799999999999997},"min":{"1min":0.34599999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.80000000000000004,"5min":8.016,"15min":8.016},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.69899999999999995,"15min":0.71899999999999997},"min":{"1min":0.379,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.83299999999999996,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.754}]},{"osd":1,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.74299999999999999,"15min":0.74199999999999999},"min":{"1min":0.46300000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91000000000000003,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.42399999999999999},"max":{"1min":0.73499999999999999,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.57499999999999996}]},{"osd":2,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.77900000000000003,"15min":0.76900000000000002},"min":{"1min":0.46999999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86199999999999999,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.78400000000000003,"15min":0.80400000000000005},"min":{"1min":0.53800000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.872,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.68999999999999995}]},{"osd":3,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.751,"15min":0.749},"min":{"1min":0.49199999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86299999999999999,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.79000000000000004,"15min":0.77200000000000002},"min":{"1min":0.51900000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":5.093,"15min":5.093},"last":0.78700000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.66400000000000003,"15min":0.624},"min":{"1min":0.39200000000000002,"5min":0.38300000000000001,"15min":0.36499999999999999},"max":{"1min":0.83699999999999997,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.746},"min":{"1min":0.58499999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90100000000000002,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.76800000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.73199999999999998,"15min":0.69399999999999995},"min":{"1min":0.436,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":0.88600000000000001,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.621},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.77000000000000002,"15min":0.73399999999999999},"min":{"1min":0.54000000000000004,"5min":0.42599999999999999,"15min":0.41099999999999998},"max":{"1min":0.878,"5min":3.492,"15min":3.492},"last":0.73999999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.72099999999999997,"15min":0.69599999999999995},"min":{"1min":0.51300000000000001,"5min":0.48299999999999998,"15min":0.47099999999999997},"max":{"1min":0.90700000000000003,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.77600000000000002,"15min":0.71699999999999997},"min":{"1min":0.42199999999999999,"5min":0.41599999999999998,"15min":0.41099999999999998},"max":{"1min":0.91600000000000004,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.72299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834576007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42600,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329240,"statfs":{"total":96636764160,"available":96593141760,"internally_reserved":0,"allocated":3362816,"data_stored":1722909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68200000000000005,"15min":0.72099999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93999999999999995,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.73399999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.0600000000000001},"last":0.79300000000000004}]},{"osd":1,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.67900000000000005,"15min":0.71299999999999997},"min":{"1min":0.46000000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.999,"5min":1.044,"15min":1.212},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.70799999999999996,"15min":0.747},"min":{"1min":0.505,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.0589999999999999,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.76600000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.69799999999999995,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94799999999999995,"5min":1.046,"15min":1.2789999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70099999999999996,"15min":0.73299999999999998},"min":{"1min":0.47499999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.226},"last":0.81200000000000006}]},{"osd":3,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.71999999999999997,"15min":0.75600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.026,"5min":1.141,"15min":1.2549999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.04,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.82199999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.628,"15min":0.626},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65800000000000003,"15min":0.66200000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.41999999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1879999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.64600000000000002,"15min":0.65200000000000002},"min":{"1min":0.433,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88800000000000001,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.70699999999999996,"15min":0.72499999999999998},"min":{"1min":0.501,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.60599999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73899999999999999},"min":{"1min":0.48499999999999999,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.0129999999999999,"5min":1.21,"15min":1.2370000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.72899999999999998,"15min":0.73899999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.49399999999999999},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.72099999999999997}]}]},{"osd":1,"up_from":13,"seq":55834576008,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57120,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314720,"statfs":{"total":96636764160,"available":96578273280,"internally_reserved":0,"allocated":3940352,"data_stored":2311679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42899999999999999,"15min":0.44800000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82499999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48299999999999998,"15min":0.503},"min":{"1min":0.31900000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.79800000000000004},"last":0.46400000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.33600000000000002,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.59599999999999997,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58699999999999997},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.443}]},{"osd":3,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54100000000000004,"15min":0.55600000000000005},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.52300000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.76000000000000001,"15min":0.78100000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.433},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.753,"15min":0.77700000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.41799999999999998},"max":{"1min":0.98299999999999998,"5min":1.05,"15min":1.25},"last":0.57699999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.753,"15min":0.78200000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.93400000000000005,"5min":1.03,"15min":1.321},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78400000000000003,"15min":0.80100000000000005},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.115,"5min":1.115,"15min":1.173},"last":0.55100000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.78300000000000003,"15min":0.80700000000000005},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.48099999999999998},"max":{"1min":1.087,"5min":1.087,"15min":1.2210000000000001},"last":0.752},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.79100000000000004,"15min":0.80600000000000005},"min":{"1min":0.52900000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.024,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.60599999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.78900000000000003,"15min":0.81499999999999995},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0740000000000001,"5min":1.1080000000000001,"15min":1.236},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81200000000000006,"15min":0.83099999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.129,"5min":1.129,"15min":1.2789999999999999},"last":0.80700000000000005}]}]},{"osd":0,"up_from":13,"seq":55834576008,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52448,"kb_used_data":3272,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319392,"statfs":{"total":96636764160,"available":96583057408,"internally_reserved":0,"allocated":3350528,"data_stored":1720944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.42699999999999999,"15min":0.45000000000000001},"min":{"1min":0.26700000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.58999999999999997,"5min":0.69899999999999995,"15min":0.86699999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.48099999999999998,"15min":0.499},"min":{"1min":0.29799999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66300000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.437}]},{"osd":2,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.48699999999999999,"15min":0.50600000000000001},"min":{"1min":0.35699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.621,"5min":0.69399999999999995,"15min":0.82699999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52100000000000002,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.63300000000000001,"5min":0.999,"15min":0.999},"last":0.38200000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53100000000000003,"15min":0.56100000000000005},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.63500000000000001,"5min":0.79800000000000004,"15min":0.85699999999999998},"last":0.505},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.63},"min":{"1min":0.47599999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.66000000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.54800000000000004}]},{"osd":4,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.82199999999999995},"min":{"1min":0.622,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":0.86499999999999999,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.76000000000000001,"15min":0.81599999999999995},"min":{"1min":0.55100000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.90200000000000002,"5min":1.169,"15min":1.169},"last":0.66200000000000003}]},{"osd":5,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.76500000000000001,"15min":0.81699999999999995},"min":{"1min":0.58299999999999996,"5min":0.497,"15min":0.497},"max":{"1min":0.95199999999999996,"5min":1.145,"15min":1.232},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.78500000000000003,"15min":0.82499999999999996},"min":{"1min":0.53600000000000003,"5min":0.497,"15min":0.497},"max":{"1min":0.94399999999999995,"5min":1.21,"15min":1.21},"last":0.70299999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.75,"15min":0.79100000000000004},"min":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.88200000000000001,"5min":1.1830000000000001,"15min":1.196},"last":0.747},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.77200000000000002,"15min":0.80400000000000005},"min":{"1min":0.54200000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96799999999999997,"5min":1.246,"15min":1.246},"last":0.56299999999999994}]},{"osd":7,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.80700000000000005,"15min":0.84799999999999998},"min":{"1min":0.60999999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.98699999999999999,"5min":1.292,"15min":1.292},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.79800000000000004,"15min":0.83999999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.95799999999999996,"5min":1.1719999999999999,"15min":1.2130000000000001},"last":0.69099999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576007,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52440,"kb_used_data":3268,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319400,"statfs":{"total":96636764160,"available":96583065600,"internally_reserved":0,"allocated":3346432,"data_stored":1721311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.45100000000000001,"15min":0.443},"min":{"1min":0.28899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.316},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.31,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":0.46100000000000002}]},{"osd":1,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.496,"15min":0.503},"min":{"1min":0.34000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53600000000000003,"15min":0.55400000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.42399999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.71199999999999997,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.626},"min":{"1min":0.45400000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.44}]},{"osd":4,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.75900000000000001,"15min":0.77300000000000002},"min":{"1min":0.56699999999999995,"5min":0.495,"15min":0.495},"max":{"1min":2.3999999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.77500000000000002,"15min":0.78900000000000003},"min":{"1min":0.54600000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.52700000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.77600000000000002,"15min":0.78600000000000003},"min":{"1min":0.57799999999999996,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1379999999999999,"5min":1.99,"15min":1.99},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.78000000000000003,"15min":0.80100000000000005},"min":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.51900000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79100000000000004,"15min":0.80500000000000005},"min":{"1min":0.59299999999999997,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77300000000000002,"15min":0.80100000000000005},"min":{"1min":0.55900000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.103,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.60799999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.80000000000000004,"15min":0.82399999999999995},"min":{"1min":0.56999999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.82199999999999995},"min":{"1min":0.59099999999999997,"5min":0.497,"15min":0.497},"max":{"1min":0.97499999999999998,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.54900000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576007,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52464,"kb_used_data":3292,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319376,"statfs":{"total":96636764160,"available":96583041024,"internally_reserved":0,"allocated":3371008,"data_stored":1723057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.41699999999999998,"15min":0.443},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64000000000000001,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.435},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.58699999999999997,"5min":0.71199999999999997,"15min":0.876},"last":0.41299999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47499999999999998,"15min":0.498},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.90300000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.53900000000000003,"15min":0.56799999999999995},"min":{"1min":0.38500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.66100000000000003,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.48999999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.58099999999999996,"15min":0.61699999999999999},"min":{"1min":0.41299999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71299999999999997,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56799999999999995,"15min":0.60999999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":1.0840000000000001},"last":0.45800000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72099999999999997,"15min":0.751},"min":{"1min":0.438,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.73499999999999999,"15min":0.76600000000000001},"min":{"1min":0.49299999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.97099999999999997,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.57799999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.74299999999999999,"15min":0.77400000000000002},"min":{"1min":0.59399999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.63},{"interface":"front","average":{"1min":0.751,"5min":0.75700000000000001,"15min":0.78000000000000003},"min":{"1min":0.51100000000000001,"5min":0.505,"15min":0.505},"max":{"1min":0.98299999999999998,"5min":1.266,"15min":1.266},"last":0.65000000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.96799999999999997,"5min":1.284,"15min":1.284},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76500000000000001,"15min":0.80300000000000005},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.495},"max":{"1min":1.0389999999999999,"5min":1.2,"15min":1.2},"last":0.72699999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.76600000000000001,"15min":0.80400000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.125,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.78900000000000003,"15min":0.81699999999999995},"min":{"1min":0.57599999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.087,"5min":1.248,"15min":1.248},"last":0.61699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576007,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48344,"kb_used_data":3268,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323496,"statfs":{"total":96636764160,"available":96587259904,"internally_reserved":0,"allocated":3346432,"data_stored":1721311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64500000000000002,"15min":0.67700000000000005},"min":{"1min":0.38100000000000001,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":0.94399999999999995,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.44},{"interface":"front","average":{"1min":0.62,"5min":0.63600000000000001,"15min":0.67100000000000004},"min":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.075},"last":0.66200000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.70499999999999996,"15min":0.746},"min":{"1min":0.51700000000000002,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":1.014,"5min":1.0549999999999999,"15min":1.069},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72199999999999998,"15min":0.75700000000000001},"min":{"1min":0.49099999999999999,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.64000000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66200000000000003,"15min":0.69999999999999996},"min":{"1min":0.439,"5min":0.438,"15min":0.438},"max":{"1min":0.95799999999999996,"5min":1.077,"15min":1.077},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68700000000000006,"15min":0.71899999999999997},"min":{"1min":0.47799999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.032,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.50600000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.71399999999999997,"15min":0.753},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.47399999999999998},"max":{"1min":0.93999999999999995,"5min":0.98099999999999998,"15min":1.018},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70999999999999996,"15min":0.74199999999999999},"min":{"1min":0.52100000000000002,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.41299999999999998,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.93700000000000006,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.755}]},{"osd":6,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.64200000000000002,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91800000000000004,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.69199999999999995,"15min":0.69599999999999995},"min":{"1min":0.46899999999999997,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":1.0149999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.70399999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.68500000000000005,"15min":0.70499999999999996},"min":{"1min":0.45500000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.0329999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76500000000000001,"15min":0.78000000000000003},"min":{"1min":0.624,"5min":0.5,"15min":0.5},"max":{"1min":1.0700000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.81299999999999994}]}]},{"osd":5,"up_from":13,"seq":55834576010,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53012,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318828,"statfs":{"total":96636764160,"available":96582479872,"internally_reserved":0,"allocated":3940352,"data_stored":2311242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.82699999999999996,"5min":0.98699999999999999,"15min":1.004},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.68999999999999995,"15min":0.71999999999999997},"min":{"1min":0.48399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.028,"5min":1.085,"15min":1.085},"last":0.83599999999999997}]},{"osd":1,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.68100000000000005,"15min":0.71499999999999997},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84399999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70999999999999996,"15min":0.72499999999999998},"min":{"1min":0.52300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88500000000000001,"5min":0.95299999999999996,"15min":1.052},"last":0.81799999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.71599999999999997,"15min":0.73799999999999999},"min":{"1min":0.47199999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97299999999999998,"5min":1.004,"15min":1.014},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.69499999999999995,"15min":0.73699999999999999},"min":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.81799999999999995,"5min":1.018,"15min":1.0680000000000001},"last":0.59199999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.747,"15min":0.77700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.88500000000000001,"5min":0.98899999999999999,"15min":1.083},"last":0.871},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77000000000000002,"15min":0.79000000000000004},"min":{"1min":0.58399999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1020000000000001},"last":0.754}]},{"osd":4,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.40699999999999997,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.054},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":0.98799999999999999,"5min":1.145,"15min":1.145},"last":0.68000000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67900000000000005,"15min":0.67300000000000004},"min":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96799999999999997,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71399999999999997,"15min":0.71099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.94899999999999995,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.69799999999999995}]},{"osd":7,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73999999999999999,"15min":0.73799999999999999},"min":{"1min":0.46500000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.0820000000000001,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76300000000000001,"15min":0.76600000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1519999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.71399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:14:43.408 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:14:43.414 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:14:43.680 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:14:43.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.875+0000 7f083d400700 1 Processor -- start 2024-04-02T09:14:43.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.876+0000 7f083d400700 1 -- start start 2024-04-02T09:14:43.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.877+0000 7f083d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f083815d640 0x7f08380d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:43.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.877+0000 7f083d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08381619d0 0x7f08380b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:43.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.877+0000 7f083d400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08380d30d0 con 0x7f08380ceec0 2024-04-02T09:14:43.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.877+0000 7f083d400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08380b6840 con 0x7f083815d640 2024-04-02T09:14:43.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.877+0000 7f083d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08380d2880 con 0x7f08381619d0 2024-04-02T09:14:43.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.877+0000 7f0836800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f083815d640 0x7f08380d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:43.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.877+0000 7f0836800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f083815d640 0x7f08380d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35112/0 (socket says 172.21.15.89:35112) 2024-04-02T09:14:43.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.877+0000 7f0836800700 1 -- 172.21.15.89:0/3351523195 learned_addr learned my addr 172.21.15.89:0/3351523195 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:43.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.877+0000 7f0837200700 1 --2- 172.21.15.89:0/3351523195 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08381619d0 0x7f08380b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:43.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.877+0000 7f0836800700 1 -- 172.21.15.89:0/3351523195 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08381619d0 msgr2=0x7f08380b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:43.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.877+0000 7f0836800700 1 --2- 172.21.15.89:0/3351523195 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08381619d0 0x7f08380b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:43.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.877+0000 7f0836800700 1 -- 172.21.15.89:0/3351523195 >> v1:172.21.15.89:6789/0 conn(0x7f08380ceec0 legacy=0x7f08381612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:43.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.877+0000 7f0836800700 1 -- 172.21.15.89:0/3351523195 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08380d2b30 con 0x7f083815d640 2024-04-02T09:14:43.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.878+0000 7f0837200700 1 --2- 172.21.15.89:0/3351523195 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08381619d0 0x7f08380b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:43.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.878+0000 7f0836800700 1 --2- 172.21.15.89:0/3351523195 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f083815d640 0x7f08380d2340 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f08280098f0 tx=0x7f0828009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77446471792cdfe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:43.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.878+0000 7f0835400700 1 -- 172.21.15.89:0/3351523195 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082800b8a0 con 0x7f083815d640 2024-04-02T09:14:43.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.878+0000 7f0835400700 1 -- 172.21.15.89:0/3351523195 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082800ba00 con 0x7f083815d640 2024-04-02T09:14:43.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.878+0000 7f0835400700 1 -- 172.21.15.89:0/3351523195 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082800bd90 con 0x7f083815d640 2024-04-02T09:14:43.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.878+0000 7f083d400700 1 -- 172.21.15.89:0/3351523195 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f083815d640 msgr2=0x7f08380d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:43.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.878+0000 7f083d400700 1 --2- 172.21.15.89:0/3351523195 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f083815d640 0x7f08380d2340 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f08280098f0 tx=0x7f0828009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:43.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.879+0000 7f083d400700 1 -- 172.21.15.89:0/3351523195 shutdown_connections 2024-04-02T09:14:43.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.879+0000 7f083d400700 1 --2- 172.21.15.89:0/3351523195 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f08381619d0 0x7f08380b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:43.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.879+0000 7f083d400700 1 --2- 172.21.15.89:0/3351523195 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f083815d640 0x7f08380d2340 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:43.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.879+0000 7f083d400700 1 -- 172.21.15.89:0/3351523195 >> 172.21.15.89:0/3351523195 conn(0x7f083808aff0 msgr2=0x7f08380d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:43.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.879+0000 7f083d400700 1 -- 172.21.15.89:0/3351523195 shutdown_connections 2024-04-02T09:14:43.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.879+0000 7f083d400700 1 -- 172.21.15.89:0/3351523195 wait complete. 2024-04-02T09:14:43.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.881+0000 7f083d400700 1 Processor -- start 2024-04-02T09:14:43.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.881+0000 7f083d400700 1 -- start start 2024-04-02T09:14:43.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.882+0000 7f083d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08380ceec0 0x7f083815a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:43.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.882+0000 7f083d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f083815d640 0x7f0838158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:43.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.882+0000 7f083d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08381619d0 0x7f0838158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:43.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.882+0000 7f083d400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083809fc90 con 0x7f08380ceec0 2024-04-02T09:14:43.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.882+0000 7f083d400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08380c7120 con 0x7f08381619d0 2024-04-02T09:14:43.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.882+0000 7f083d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08380c73c0 con 0x7f083815d640 2024-04-02T09:14:43.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.883+0000 7f0836800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08380ceec0 0x7f083815a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:43.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.883+0000 7f0836800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08380ceec0 0x7f083815a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35126/0 (socket says 172.21.15.89:35126) 2024-04-02T09:14:43.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.883+0000 7f0836800700 1 -- 172.21.15.89:0/4281193892 learned_addr learned my addr 172.21.15.89:0/4281193892 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.883+0000 7f0837200700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08381619d0 0x7f0838158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.883+0000 7f0835e00700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f083815d640 0x7f0838158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.883+0000 7f0836800700 1 -- 172.21.15.89:0/4281193892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f083815d640 msgr2=0x7f0838158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.883+0000 7f0836800700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f083815d640 0x7f0838158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.883+0000 7f0836800700 1 -- 172.21.15.89:0/4281193892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08381619d0 msgr2=0x7f0838158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.883+0000 7f0836800700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08381619d0 0x7f0838158bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.883+0000 7f0836800700 1 -- 172.21.15.89:0/4281193892 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0828009550 con 0x7f08380ceec0 2024-04-02T09:14:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.883+0000 7f0837200700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08381619d0 0x7f0838158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.883+0000 7f0836800700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08380ceec0 0x7f083815a800 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f0828009130 tx=0x7f0828004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.884+0000 7f082ea00700 1 -- 172.21.15.89:0/4281193892 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08280036a0 con 0x7f08380ceec0 2024-04-02T09:14:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.884+0000 7f082ea00700 1 -- 172.21.15.89:0/4281193892 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082801e070 con 0x7f08380ceec0 2024-04-02T09:14:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.884+0000 7f083d400700 1 -- 172.21.15.89:0/4281193892 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08380c76a0 con 0x7f08380ceec0 2024-04-02T09:14:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.884+0000 7f082ea00700 1 -- 172.21.15.89:0/4281193892 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0828003a60 con 0x7f08380ceec0 2024-04-02T09:14:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.884+0000 7f083d400700 1 -- 172.21.15.89:0/4281193892 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08380c7c60 con 0x7f08380ceec0 2024-04-02T09:14:43.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.885+0000 7f083d400700 1 -- 172.21.15.89:0/4281193892 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0838067720 con 0x7f08380ceec0 2024-04-02T09:14:43.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.886+0000 7f082ea00700 1 -- 172.21.15.89:0/4281193892 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0828003bc0 con 0x7f08380ceec0 2024-04-02T09:14:43.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.887+0000 7f082ea00700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f081806aed0 0x7f081806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:43.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.887+0000 7f082ea00700 1 -- 172.21.15.89:0/4281193892 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f0828020070 con 0x7f08380ceec0 2024-04-02T09:14:43.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.887+0000 7f0835e00700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f081806aed0 0x7f081806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:43.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.888+0000 7f0835e00700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f081806aed0 0x7f081806d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f083815c7f0 tx=0x7f0824009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:43.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:43.892+0000 7f082ea00700 1 -- 172.21.15.89:0/4281193892 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0828055c00 con 0x7f08380ceec0 2024-04-02T09:14:44.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.089+0000 7f083d400700 1 -- 172.21.15.89:0/4281193892 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f08380b64d0 con 0x7f081806aed0 2024-04-02T09:14:44.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.097+0000 7f082ea00700 1 -- 172.21.15.89:0/4281193892 <== mgr.11320 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289305 (secure 0 0 0) 0x7f08240013f0 con 0x7f081806aed0 2024-04-02T09:14:44.099 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:44.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.101+0000 7f083d400700 1 -- 172.21.15.89:0/4281193892 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f081806aed0 msgr2=0x7f081806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:44.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.101+0000 7f083d400700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f081806aed0 0x7f081806d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f083815c7f0 tx=0x7f0824009040 comp rx=0 tx=0).stop 2024-04-02T09:14:44.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.101+0000 7f083d400700 1 -- 172.21.15.89:0/4281193892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08380ceec0 msgr2=0x7f083815a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:44.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.101+0000 7f083d400700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08380ceec0 0x7f083815a800 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f0828009130 tx=0x7f0828004750 comp rx=0 tx=0).stop 2024-04-02T09:14:44.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.101+0000 7f083d400700 1 -- 172.21.15.89:0/4281193892 shutdown_connections 2024-04-02T09:14:44.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.101+0000 7f083d400700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f081806aed0 0x7f081806d390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.101+0000 7f083d400700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08381619d0 0x7f0838158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.101+0000 7f083d400700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08380ceec0 0x7f083815a800 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.101+0000 7f083d400700 1 --2- 172.21.15.89:0/4281193892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f083815d640 0x7f0838158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.101+0000 7f083d400700 1 -- 172.21.15.89:0/4281193892 >> 172.21.15.89:0/4281193892 conn(0x7f083808aff0 msgr2=0x7f08380b6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:44.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.102+0000 7f083d400700 1 -- 172.21.15.89:0/4281193892 shutdown_connections 2024-04-02T09:14:44.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.102+0000 7f083d400700 1 -- 172.21.15.89:0/4281193892 wait complete. 2024-04-02T09:14:44.102 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:14:44.118 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-04-02T09:14:43.259519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":684,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":420,"ondisk_log_size":420,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":419072,"kb_used_data":27360,"kb_used_omap":12,"kb_used_meta":391539,"kb_avail":754555648,"statfs":{"total":773094113280,"available":772664983552,"internally_reserved":0,"allocated":28016640,"data_stored":14953910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":400936504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.905001"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523216+0000","last_change":"2024-04-02T09:14:34.377961+0000","last_active":"2024-04-02T09:14:35.523216+0000","last_peered":"2024-04-02T09:14:35.523216+0000","last_clean":"2024-04-02T09:14:35.523216+0000","last_became_active":"2024-04-02T09:14:34.377267+0000","last_became_peered":"2024-04-02T09:14:34.377267+0000","last_unstale":"2024-04-02T09:14:35.523216+0000","last_undegraded":"2024-04-02T09:14:35.523216+0000","last_fullsized":"2024-04-02T09:14:35.523216+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:44:06.694478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364561+0000","last_change":"2024-04-02T09:14:34.373205+0000","last_active":"2024-04-02T09:14:35.364561+0000","last_peered":"2024-04-02T09:14:35.364561+0000","last_clean":"2024-04-02T09:14:35.364561+0000","last_became_active":"2024-04-02T09:14:34.370571+0000","last_became_peered":"2024-04-02T09:14:34.370571+0000","last_unstale":"2024-04-02T09:14:35.364561+0000","last_undegraded":"2024-04-02T09:14:35.364561+0000","last_fullsized":"2024-04-02T09:14:35.364561+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:55:48.430034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363862+0000","last_change":"2024-04-02T09:14:34.369016+0000","last_active":"2024-04-02T09:14:35.363862+0000","last_peered":"2024-04-02T09:14:35.363862+0000","last_clean":"2024-04-02T09:14:35.363862+0000","last_became_active":"2024-04-02T09:14:34.368553+0000","last_became_peered":"2024-04-02T09:14:34.368553+0000","last_unstale":"2024-04-02T09:14:35.363862+0000","last_undegraded":"2024-04-02T09:14:35.363862+0000","last_fullsized":"2024-04-02T09:14:35.363862+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:12.477806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522578+0000","last_change":"2024-04-02T09:14:34.383881+0000","last_active":"2024-04-02T09:14:35.522578+0000","last_peered":"2024-04-02T09:14:35.522578+0000","last_clean":"2024-04-02T09:14:35.522578+0000","last_became_active":"2024-04-02T09:14:34.383189+0000","last_became_peered":"2024-04-02T09:14:34.383189+0000","last_unstale":"2024-04-02T09:14:35.522578+0000","last_undegraded":"2024-04-02T09:14:35.522578+0000","last_fullsized":"2024-04-02T09:14:35.522578+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:49:07.747576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522857+0000","last_change":"2024-04-02T09:14:34.384650+0000","last_active":"2024-04-02T09:14:35.522857+0000","last_peered":"2024-04-02T09:14:35.522857+0000","last_clean":"2024-04-02T09:14:35.522857+0000","last_became_active":"2024-04-02T09:14:34.384353+0000","last_became_peered":"2024-04-02T09:14:34.384353+0000","last_unstale":"2024-04-02T09:14:35.522857+0000","last_undegraded":"2024-04-02T09:14:35.522857+0000","last_fullsized":"2024-04-02T09:14:35.522857+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:36.082626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364110+0000","last_change":"2024-04-02T09:14:34.367970+0000","last_active":"2024-04-02T09:14:35.364110+0000","last_peered":"2024-04-02T09:14:35.364110+0000","last_clean":"2024-04-02T09:14:35.364110+0000","last_became_active":"2024-04-02T09:14:34.365750+0000","last_became_peered":"2024-04-02T09:14:34.365750+0000","last_unstale":"2024-04-02T09:14:35.364110+0000","last_undegraded":"2024-04-02T09:14:35.364110+0000","last_fullsized":"2024-04-02T09:14:35.364110+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:32.796452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362566+0000","last_change":"2024-04-02T09:14:34.381480+0000","last_active":"2024-04-02T09:14:35.362566+0000","last_peered":"2024-04-02T09:14:35.362566+0000","last_clean":"2024-04-02T09:14:35.362566+0000","last_became_active":"2024-04-02T09:14:34.381243+0000","last_became_peered":"2024-04-02T09:14:34.381243+0000","last_unstale":"2024-04-02T09:14:35.362566+0000","last_undegraded":"2024-04-02T09:14:35.362566+0000","last_fullsized":"2024-04-02T09:14:35.362566+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:43:52.860015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362432+0000","last_change":"2024-04-02T09:14:34.390416+0000","last_active":"2024-04-02T09:14:35.362432+0000","last_peered":"2024-04-02T09:14:35.362432+0000","last_clean":"2024-04-02T09:14:35.362432+0000","last_became_active":"2024-04-02T09:14:34.390126+0000","last_became_peered":"2024-04-02T09:14:34.390126+0000","last_unstale":"2024-04-02T09:14:35.362432+0000","last_undegraded":"2024-04-02T09:14:35.362432+0000","last_fullsized":"2024-04-02T09:14:35.362432+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:05:04.134051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364695+0000","last_change":"2024-04-02T09:14:34.362861+0000","last_active":"2024-04-02T09:14:35.364695+0000","last_peered":"2024-04-02T09:14:35.364695+0000","last_clean":"2024-04-02T09:14:35.364695+0000","last_became_active":"2024-04-02T09:14:34.362580+0000","last_became_peered":"2024-04-02T09:14:34.362580+0000","last_unstale":"2024-04-02T09:14:35.364695+0000","last_undegraded":"2024-04-02T09:14:35.364695+0000","last_fullsized":"2024-04-02T09:14:35.364695+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:12.477806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362736+0000","last_change":"2024-04-02T09:14:34.380960+0000","last_active":"2024-04-02T09:14:35.362736+0000","last_peered":"2024-04-02T09:14:35.362736+0000","last_clean":"2024-04-02T09:14:35.362736+0000","last_became_active":"2024-04-02T09:14:34.380726+0000","last_became_peered":"2024-04-02T09:14:34.380726+0000","last_unstale":"2024-04-02T09:14:35.362736+0000","last_undegraded":"2024-04-02T09:14:35.362736+0000","last_fullsized":"2024-04-02T09:14:35.362736+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:51:23.449925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522728+0000","last_change":"2024-04-02T09:14:34.382824+0000","last_active":"2024-04-02T09:14:35.522728+0000","last_peered":"2024-04-02T09:14:35.522728+0000","last_clean":"2024-04-02T09:14:35.522728+0000","last_became_active":"2024-04-02T09:14:34.382044+0000","last_became_peered":"2024-04-02T09:14:34.382044+0000","last_unstale":"2024-04-02T09:14:35.522728+0000","last_undegraded":"2024-04-02T09:14:35.522728+0000","last_fullsized":"2024-04-02T09:14:35.522728+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:23:56.492180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364322+0000","last_change":"2024-04-02T09:14:34.366396+0000","last_active":"2024-04-02T09:14:35.364322+0000","last_peered":"2024-04-02T09:14:35.364322+0000","last_clean":"2024-04-02T09:14:35.364322+0000","last_became_active":"2024-04-02T09:14:34.366115+0000","last_became_peered":"2024-04-02T09:14:34.366115+0000","last_unstale":"2024-04-02T09:14:35.364322+0000","last_undegraded":"2024-04-02T09:14:35.364322+0000","last_fullsized":"2024-04-02T09:14:35.364322+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:56:03.446485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523710+0000","last_change":"2024-04-02T09:14:34.374183+0000","last_active":"2024-04-02T09:14:35.523710+0000","last_peered":"2024-04-02T09:14:35.523710+0000","last_clean":"2024-04-02T09:14:35.523710+0000","last_became_active":"2024-04-02T09:14:34.371570+0000","last_became_peered":"2024-04-02T09:14:34.371570+0000","last_unstale":"2024-04-02T09:14:35.523710+0000","last_undegraded":"2024-04-02T09:14:35.523710+0000","last_fullsized":"2024-04-02T09:14:35.523710+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:06:08.871157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522853+0000","last_change":"2024-04-02T09:14:34.385243+0000","last_active":"2024-04-02T09:14:35.522853+0000","last_peered":"2024-04-02T09:14:35.522853+0000","last_clean":"2024-04-02T09:14:35.522853+0000","last_became_active":"2024-04-02T09:14:34.384989+0000","last_became_peered":"2024-04-02T09:14:34.384989+0000","last_unstale":"2024-04-02T09:14:35.522853+0000","last_undegraded":"2024-04-02T09:14:35.522853+0000","last_fullsized":"2024-04-02T09:14:35.522853+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:28.009617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362467+0000","last_change":"2024-04-02T09:14:34.379614+0000","last_active":"2024-04-02T09:14:35.362467+0000","last_peered":"2024-04-02T09:14:35.362467+0000","last_clean":"2024-04-02T09:14:35.362467+0000","last_became_active":"2024-04-02T09:14:34.378063+0000","last_became_peered":"2024-04-02T09:14:34.378063+0000","last_unstale":"2024-04-02T09:14:35.362467+0000","last_undegraded":"2024-04-02T09:14:35.362467+0000","last_fullsized":"2024-04-02T09:14:35.362467+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:07.213744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522170+0000","last_change":"2024-04-02T09:14:34.373018+0000","last_active":"2024-04-02T09:14:35.522170+0000","last_peered":"2024-04-02T09:14:35.522170+0000","last_clean":"2024-04-02T09:14:35.522170+0000","last_became_active":"2024-04-02T09:14:34.372010+0000","last_became_peered":"2024-04-02T09:14:34.372010+0000","last_unstale":"2024-04-02T09:14:35.522170+0000","last_undegraded":"2024-04-02T09:14:35.522170+0000","last_fullsized":"2024-04-02T09:14:35.522170+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:58.362648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523162+0000","last_change":"2024-04-02T09:14:34.391051+0000","last_active":"2024-04-02T09:14:35.523162+0000","last_peered":"2024-04-02T09:14:35.523162+0000","last_clean":"2024-04-02T09:14:35.523162+0000","last_became_active":"2024-04-02T09:14:34.390721+0000","last_became_peered":"2024-04-02T09:14:34.390721+0000","last_unstale":"2024-04-02T09:14:35.523162+0000","last_undegraded":"2024-04-02T09:14:35.523162+0000","last_fullsized":"2024-04-02T09:14:35.523162+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:53:01.672940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362877+0000","last_change":"2024-04-02T09:14:34.383335+0000","last_active":"2024-04-02T09:14:35.362877+0000","last_peered":"2024-04-02T09:14:35.362877+0000","last_clean":"2024-04-02T09:14:35.362877+0000","last_became_active":"2024-04-02T09:14:34.383105+0000","last_became_peered":"2024-04-02T09:14:34.383105+0000","last_unstale":"2024-04-02T09:14:35.362877+0000","last_undegraded":"2024-04-02T09:14:35.362877+0000","last_fullsized":"2024-04-02T09:14:35.362877+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:08:25.093735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523794+0000","last_change":"2024-04-02T09:14:34.385798+0000","last_active":"2024-04-02T09:14:35.523794+0000","last_peered":"2024-04-02T09:14:35.523794+0000","last_clean":"2024-04-02T09:14:35.523794+0000","last_became_active":"2024-04-02T09:14:34.385264+0000","last_became_peered":"2024-04-02T09:14:34.385264+0000","last_unstale":"2024-04-02T09:14:35.523794+0000","last_undegraded":"2024-04-02T09:14:35.523794+0000","last_fullsized":"2024-04-02T09:14:35.523794+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:41.722545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362562+0000","last_change":"2024-04-02T09:14:34.384422+0000","last_active":"2024-04-02T09:14:35.362562+0000","last_peered":"2024-04-02T09:14:35.362562+0000","last_clean":"2024-04-02T09:14:35.362562+0000","last_became_active":"2024-04-02T09:14:34.384187+0000","last_became_peered":"2024-04-02T09:14:34.384187+0000","last_unstale":"2024-04-02T09:14:35.362562+0000","last_undegraded":"2024-04-02T09:14:35.362562+0000","last_fullsized":"2024-04-02T09:14:35.362562+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:45.536966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523625+0000","last_change":"2024-04-02T09:14:34.375508+0000","last_active":"2024-04-02T09:14:35.523625+0000","last_peered":"2024-04-02T09:14:35.523625+0000","last_clean":"2024-04-02T09:14:35.523625+0000","last_became_active":"2024-04-02T09:14:34.374458+0000","last_became_peered":"2024-04-02T09:14:34.374458+0000","last_unstale":"2024-04-02T09:14:35.523625+0000","last_undegraded":"2024-04-02T09:14:35.523625+0000","last_fullsized":"2024-04-02T09:14:35.523625+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:23.023963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522793+0000","last_change":"2024-04-02T09:14:34.371948+0000","last_active":"2024-04-02T09:14:35.522793+0000","last_peered":"2024-04-02T09:14:35.522793+0000","last_clean":"2024-04-02T09:14:35.522793+0000","last_became_active":"2024-04-02T09:14:34.371586+0000","last_became_peered":"2024-04-02T09:14:34.371586+0000","last_unstale":"2024-04-02T09:14:35.522793+0000","last_undegraded":"2024-04-02T09:14:35.522793+0000","last_fullsized":"2024-04-02T09:14:35.522793+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:27.813013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522861+0000","last_change":"2024-04-02T09:14:34.372317+0000","last_active":"2024-04-02T09:14:35.522861+0000","last_peered":"2024-04-02T09:14:35.522861+0000","last_clean":"2024-04-02T09:14:35.522861+0000","last_became_active":"2024-04-02T09:14:34.371999+0000","last_became_peered":"2024-04-02T09:14:34.371999+0000","last_unstale":"2024-04-02T09:14:35.522861+0000","last_undegraded":"2024-04-02T09:14:35.522861+0000","last_fullsized":"2024-04-02T09:14:35.522861+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:01:49.154841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363323+0000","last_change":"2024-04-02T09:14:34.369575+0000","last_active":"2024-04-02T09:14:35.363323+0000","last_peered":"2024-04-02T09:14:35.363323+0000","last_clean":"2024-04-02T09:14:35.363323+0000","last_became_active":"2024-04-02T09:14:34.369273+0000","last_became_peered":"2024-04-02T09:14:34.369273+0000","last_unstale":"2024-04-02T09:14:35.363323+0000","last_undegraded":"2024-04-02T09:14:35.363323+0000","last_fullsized":"2024-04-02T09:14:35.363323+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:12:47.137280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523287+0000","last_change":"2024-04-02T09:14:34.372802+0000","last_active":"2024-04-02T09:14:35.523287+0000","last_peered":"2024-04-02T09:14:35.523287+0000","last_clean":"2024-04-02T09:14:35.523287+0000","last_became_active":"2024-04-02T09:14:34.372511+0000","last_became_peered":"2024-04-02T09:14:34.372511+0000","last_unstale":"2024-04-02T09:14:35.523287+0000","last_undegraded":"2024-04-02T09:14:35.523287+0000","last_fullsized":"2024-04-02T09:14:35.523287+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:46.642400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363775+0000","last_change":"2024-04-02T09:14:34.370273+0000","last_active":"2024-04-02T09:14:35.363775+0000","last_peered":"2024-04-02T09:14:35.363775+0000","last_clean":"2024-04-02T09:14:35.363775+0000","last_became_active":"2024-04-02T09:14:34.369221+0000","last_became_peered":"2024-04-02T09:14:34.369221+0000","last_unstale":"2024-04-02T09:14:35.363775+0000","last_undegraded":"2024-04-02T09:14:35.363775+0000","last_fullsized":"2024-04-02T09:14:35.363775+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:32.796452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523112+0000","last_change":"2024-04-02T09:14:34.381402+0000","last_active":"2024-04-02T09:14:35.523112+0000","last_peered":"2024-04-02T09:14:35.523112+0000","last_clean":"2024-04-02T09:14:35.523112+0000","last_became_active":"2024-04-02T09:14:34.381063+0000","last_became_peered":"2024-04-02T09:14:34.381063+0000","last_unstale":"2024-04-02T09:14:35.523112+0000","last_undegraded":"2024-04-02T09:14:35.523112+0000","last_fullsized":"2024-04-02T09:14:35.523112+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:24:11.707496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522665+0000","last_change":"2024-04-02T09:14:34.383686+0000","last_active":"2024-04-02T09:14:35.522665+0000","last_peered":"2024-04-02T09:14:35.522665+0000","last_clean":"2024-04-02T09:14:35.522665+0000","last_became_active":"2024-04-02T09:14:34.382500+0000","last_became_peered":"2024-04-02T09:14:34.382500+0000","last_unstale":"2024-04-02T09:14:35.522665+0000","last_undegraded":"2024-04-02T09:14:35.522665+0000","last_fullsized":"2024-04-02T09:14:35.522665+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:35:17.449940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362773+0000","last_change":"2024-04-02T09:14:34.387782+0000","last_active":"2024-04-02T09:14:35.362773+0000","last_peered":"2024-04-02T09:14:35.362773+0000","last_clean":"2024-04-02T09:14:35.362773+0000","last_became_active":"2024-04-02T09:14:34.383376+0000","last_became_peered":"2024-04-02T09:14:34.383376+0000","last_unstale":"2024-04-02T09:14:35.362773+0000","last_undegraded":"2024-04-02T09:14:35.362773+0000","last_fullsized":"2024-04-02T09:14:35.362773+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:28:13.971403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522423+0000","last_change":"2024-04-02T09:14:34.374368+0000","last_active":"2024-04-02T09:14:35.522423+0000","last_peered":"2024-04-02T09:14:35.522423+0000","last_clean":"2024-04-02T09:14:35.522423+0000","last_became_active":"2024-04-02T09:14:34.374046+0000","last_became_peered":"2024-04-02T09:14:34.374046+0000","last_unstale":"2024-04-02T09:14:35.522423+0000","last_undegraded":"2024-04-02T09:14:35.522423+0000","last_fullsized":"2024-04-02T09:14:35.522423+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:37:45.647925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363599+0000","last_change":"2024-04-02T09:14:34.384135+0000","last_active":"2024-04-02T09:14:35.363599+0000","last_peered":"2024-04-02T09:14:35.363599+0000","last_clean":"2024-04-02T09:14:35.363599+0000","last_became_active":"2024-04-02T09:14:34.383849+0000","last_became_peered":"2024-04-02T09:14:34.383849+0000","last_unstale":"2024-04-02T09:14:35.363599+0000","last_undegraded":"2024-04-02T09:14:35.363599+0000","last_fullsized":"2024-04-02T09:14:35.363599+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:23:36.207832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363879+0000","last_change":"2024-04-02T09:14:34.362600+0000","last_active":"2024-04-02T09:14:35.363879+0000","last_peered":"2024-04-02T09:14:35.363879+0000","last_clean":"2024-04-02T09:14:35.363879+0000","last_became_active":"2024-04-02T09:14:34.362080+0000","last_became_peered":"2024-04-02T09:14:34.362080+0000","last_unstale":"2024-04-02T09:14:35.363879+0000","last_undegraded":"2024-04-02T09:14:35.363879+0000","last_fullsized":"2024-04-02T09:14:35.363879+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:15:41.728873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"264'4","reported_seq":15,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523394+0000","last_change":"2024-04-02T09:14:33.875502+0000","last_active":"2024-04-02T09:14:35.523394+0000","last_peered":"2024-04-02T09:14:35.523394+0000","last_clean":"2024-04-02T09:14:35.523394+0000","last_became_active":"2024-04-02T09:14:33.875191+0000","last_became_peered":"2024-04-02T09:14:33.875191+0000","last_unstale":"2024-04-02T09:14:35.523394+0000","last_undegraded":"2024-04-02T09:14:35.523394+0000","last_fullsized":"2024-04-02T09:14:35.523394+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:54.914511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523839+0000","last_change":"2024-04-02T09:14:34.372870+0000","last_active":"2024-04-02T09:14:35.523839+0000","last_peered":"2024-04-02T09:14:35.523839+0000","last_clean":"2024-04-02T09:14:35.523839+0000","last_became_active":"2024-04-02T09:14:34.372591+0000","last_became_peered":"2024-04-02T09:14:34.372591+0000","last_unstale":"2024-04-02T09:14:35.523839+0000","last_undegraded":"2024-04-02T09:14:35.523839+0000","last_fullsized":"2024-04-02T09:14:35.523839+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:21.044232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1e","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364686+0000","last_change":"2024-04-02T09:14:33.879018+0000","last_active":"2024-04-02T09:14:35.364686+0000","last_peered":"2024-04-02T09:14:35.364686+0000","last_clean":"2024-04-02T09:14:35.364686+0000","last_became_active":"2024-04-02T09:14:33.877939+0000","last_became_peered":"2024-04-02T09:14:33.877939+0000","last_unstale":"2024-04-02T09:14:35.364686+0000","last_undegraded":"2024-04-02T09:14:35.364686+0000","last_fullsized":"2024-04-02T09:14:35.364686+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:21:10.605008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363660+0000","last_change":"2024-04-02T09:14:34.381630+0000","last_active":"2024-04-02T09:14:35.363660+0000","last_peered":"2024-04-02T09:14:35.363660+0000","last_clean":"2024-04-02T09:14:35.363660+0000","last_became_active":"2024-04-02T09:14:34.381362+0000","last_became_peered":"2024-04-02T09:14:34.381362+0000","last_unstale":"2024-04-02T09:14:35.363660+0000","last_undegraded":"2024-04-02T09:14:35.363660+0000","last_fullsized":"2024-04-02T09:14:35.363660+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:17:46.076532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1d","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364496+0000","last_change":"2024-04-02T09:14:33.877689+0000","last_active":"2024-04-02T09:14:35.364496+0000","last_peered":"2024-04-02T09:14:35.364496+0000","last_clean":"2024-04-02T09:14:35.364496+0000","last_became_active":"2024-04-02T09:14:33.875355+0000","last_became_peered":"2024-04-02T09:14:33.875355+0000","last_unstale":"2024-04-02T09:14:35.364496+0000","last_undegraded":"2024-04-02T09:14:35.364496+0000","last_fullsized":"2024-04-02T09:14:35.364496+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:53:35.396685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364415+0000","last_change":"2024-04-02T09:14:34.384893+0000","last_active":"2024-04-02T09:14:35.364415+0000","last_peered":"2024-04-02T09:14:35.364415+0000","last_clean":"2024-04-02T09:14:35.364415+0000","last_became_active":"2024-04-02T09:14:34.384622+0000","last_became_peered":"2024-04-02T09:14:34.384622+0000","last_unstale":"2024-04-02T09:14:35.364415+0000","last_undegraded":"2024-04-02T09:14:35.364415+0000","last_fullsized":"2024-04-02T09:14:35.364415+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:13:28.694192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1c","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523485+0000","last_change":"2024-04-02T09:14:33.860753+0000","last_active":"2024-04-02T09:14:35.523485+0000","last_peered":"2024-04-02T09:14:35.523485+0000","last_clean":"2024-04-02T09:14:35.523485+0000","last_became_active":"2024-04-02T09:14:33.860419+0000","last_became_peered":"2024-04-02T09:14:33.860419+0000","last_unstale":"2024-04-02T09:14:35.523485+0000","last_undegraded":"2024-04-02T09:14:35.523485+0000","last_fullsized":"2024-04-02T09:14:35.523485+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:54.064730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364273+0000","last_change":"2024-04-02T09:14:34.373334+0000","last_active":"2024-04-02T09:14:35.364273+0000","last_peered":"2024-04-02T09:14:35.364273+0000","last_clean":"2024-04-02T09:14:35.364273+0000","last_became_active":"2024-04-02T09:14:34.372987+0000","last_became_peered":"2024-04-02T09:14:34.372987+0000","last_unstale":"2024-04-02T09:14:35.364273+0000","last_undegraded":"2024-04-02T09:14:35.364273+0000","last_fullsized":"2024-04-02T09:14:35.364273+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:42.139739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1b","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523580+0000","last_change":"2024-04-02T09:14:33.874800+0000","last_active":"2024-04-02T09:14:35.523580+0000","last_peered":"2024-04-02T09:14:35.523580+0000","last_clean":"2024-04-02T09:14:35.523580+0000","last_became_active":"2024-04-02T09:14:33.874504+0000","last_became_peered":"2024-04-02T09:14:33.874504+0000","last_unstale":"2024-04-02T09:14:35.523580+0000","last_undegraded":"2024-04-02T09:14:35.523580+0000","last_fullsized":"2024-04-02T09:14:35.523580+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:12:11.515065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523030+0000","last_change":"2024-04-02T09:14:34.362124+0000","last_active":"2024-04-02T09:14:35.523030+0000","last_peered":"2024-04-02T09:14:35.523030+0000","last_clean":"2024-04-02T09:14:35.523030+0000","last_became_active":"2024-04-02T09:14:34.361786+0000","last_became_peered":"2024-04-02T09:14:34.361786+0000","last_unstale":"2024-04-02T09:14:35.523030+0000","last_undegraded":"2024-04-02T09:14:35.523030+0000","last_fullsized":"2024-04-02T09:14:35.523030+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:42:12.604946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1a","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362815+0000","last_change":"2024-04-02T09:14:33.877648+0000","last_active":"2024-04-02T09:14:35.362815+0000","last_peered":"2024-04-02T09:14:35.362815+0000","last_clean":"2024-04-02T09:14:35.362815+0000","last_became_active":"2024-04-02T09:14:33.877308+0000","last_became_peered":"2024-04-02T09:14:33.877308+0000","last_unstale":"2024-04-02T09:14:35.362815+0000","last_undegraded":"2024-04-02T09:14:35.362815+0000","last_fullsized":"2024-04-02T09:14:35.362815+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:31.787918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363331+0000","last_change":"2024-04-02T09:14:34.378176+0000","last_active":"2024-04-02T09:14:35.363331+0000","last_peered":"2024-04-02T09:14:35.363331+0000","last_clean":"2024-04-02T09:14:35.363331+0000","last_became_active":"2024-04-02T09:14:34.376201+0000","last_became_peered":"2024-04-02T09:14:34.376201+0000","last_unstale":"2024-04-02T09:14:35.363331+0000","last_undegraded":"2024-04-02T09:14:35.363331+0000","last_fullsized":"2024-04-02T09:14:35.363331+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:30:55.923045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522590+0000","last_change":"2024-04-02T09:14:33.869026+0000","last_active":"2024-04-02T09:14:35.522590+0000","last_peered":"2024-04-02T09:14:35.522590+0000","last_clean":"2024-04-02T09:14:35.522590+0000","last_became_active":"2024-04-02T09:14:33.868743+0000","last_became_peered":"2024-04-02T09:14:33.868743+0000","last_unstale":"2024-04-02T09:14:35.522590+0000","last_undegraded":"2024-04-02T09:14:35.522590+0000","last_fullsized":"2024-04-02T09:14:35.522590+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:31.787918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522526+0000","last_change":"2024-04-02T09:14:34.376819+0000","last_active":"2024-04-02T09:14:35.522526+0000","last_peered":"2024-04-02T09:14:35.522526+0000","last_clean":"2024-04-02T09:14:35.522526+0000","last_became_active":"2024-04-02T09:14:34.376555+0000","last_became_peered":"2024-04-02T09:14:34.376555+0000","last_unstale":"2024-04-02T09:14:35.522526+0000","last_undegraded":"2024-04-02T09:14:35.522526+0000","last_fullsized":"2024-04-02T09:14:35.522526+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:36:25.999949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363666+0000","last_change":"2024-04-02T09:14:33.879857+0000","last_active":"2024-04-02T09:14:35.363666+0000","last_peered":"2024-04-02T09:14:35.363666+0000","last_clean":"2024-04-02T09:14:35.363666+0000","last_became_active":"2024-04-02T09:14:33.879161+0000","last_became_peered":"2024-04-02T09:14:33.879161+0000","last_unstale":"2024-04-02T09:14:35.363666+0000","last_undegraded":"2024-04-02T09:14:35.363666+0000","last_fullsized":"2024-04-02T09:14:35.363666+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:28:14.207222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362885+0000","last_change":"2024-04-02T09:14:34.389470+0000","last_active":"2024-04-02T09:14:35.362885+0000","last_peered":"2024-04-02T09:14:35.362885+0000","last_clean":"2024-04-02T09:14:35.362885+0000","last_became_active":"2024-04-02T09:14:34.388852+0000","last_became_peered":"2024-04-02T09:14:34.388852+0000","last_unstale":"2024-04-02T09:14:35.362885+0000","last_undegraded":"2024-04-02T09:14:35.362885+0000","last_fullsized":"2024-04-02T09:14:35.362885+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:50:15.565863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523327+0000","last_change":"2024-04-02T09:14:33.869704+0000","last_active":"2024-04-02T09:14:35.523327+0000","last_peered":"2024-04-02T09:14:35.523327+0000","last_clean":"2024-04-02T09:14:35.523327+0000","last_became_active":"2024-04-02T09:14:33.868809+0000","last_became_peered":"2024-04-02T09:14:33.868809+0000","last_unstale":"2024-04-02T09:14:35.523327+0000","last_undegraded":"2024-04-02T09:14:35.523327+0000","last_fullsized":"2024-04-02T09:14:35.523327+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:12:46.128746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523864+0000","last_change":"2024-04-02T09:14:34.379506+0000","last_active":"2024-04-02T09:14:35.523864+0000","last_peered":"2024-04-02T09:14:35.523864+0000","last_clean":"2024-04-02T09:14:35.523864+0000","last_became_active":"2024-04-02T09:14:34.378787+0000","last_became_peered":"2024-04-02T09:14:34.378787+0000","last_unstale":"2024-04-02T09:14:35.523864+0000","last_undegraded":"2024-04-02T09:14:35.523864+0000","last_fullsized":"2024-04-02T09:14:35.523864+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:13:47.277991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523512+0000","last_change":"2024-04-02T09:14:33.874472+0000","last_active":"2024-04-02T09:14:35.523512+0000","last_peered":"2024-04-02T09:14:35.523512+0000","last_clean":"2024-04-02T09:14:35.523512+0000","last_became_active":"2024-04-02T09:14:33.874190+0000","last_became_peered":"2024-04-02T09:14:33.874190+0000","last_unstale":"2024-04-02T09:14:35.523512+0000","last_undegraded":"2024-04-02T09:14:35.523512+0000","last_fullsized":"2024-04-02T09:14:35.523512+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:06:40.311963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522771+0000","last_change":"2024-04-02T09:14:34.382684+0000","last_active":"2024-04-02T09:14:35.522771+0000","last_peered":"2024-04-02T09:14:35.522771+0000","last_clean":"2024-04-02T09:14:35.522771+0000","last_became_active":"2024-04-02T09:14:34.380586+0000","last_became_peered":"2024-04-02T09:14:34.380586+0000","last_unstale":"2024-04-02T09:14:35.522771+0000","last_undegraded":"2024-04-02T09:14:35.522771+0000","last_fullsized":"2024-04-02T09:14:35.522771+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:56:58.056857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.15","version":"264'3","reported_seq":14,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364540+0000","last_change":"2024-04-02T09:14:33.878269+0000","last_active":"2024-04-02T09:14:35.364540+0000","last_peered":"2024-04-02T09:14:35.364540+0000","last_clean":"2024-04-02T09:14:35.364540+0000","last_became_active":"2024-04-02T09:14:33.877496+0000","last_became_peered":"2024-04-02T09:14:33.877496+0000","last_unstale":"2024-04-02T09:14:35.364540+0000","last_undegraded":"2024-04-02T09:14:35.364540+0000","last_fullsized":"2024-04-02T09:14:35.364540+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:11:32.742344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523334+0000","last_change":"2024-04-02T09:14:34.372160+0000","last_active":"2024-04-02T09:14:35.523334+0000","last_peered":"2024-04-02T09:14:35.523334+0000","last_clean":"2024-04-02T09:14:35.523334+0000","last_became_active":"2024-04-02T09:14:34.371325+0000","last_became_peered":"2024-04-02T09:14:34.371325+0000","last_unstale":"2024-04-02T09:14:35.523334+0000","last_undegraded":"2024-04-02T09:14:35.523334+0000","last_fullsized":"2024-04-02T09:14:35.523334+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:36.082626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"264'6","reported_seq":17,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523450+0000","last_change":"2024-04-02T09:14:33.875419+0000","last_active":"2024-04-02T09:14:35.523450+0000","last_peered":"2024-04-02T09:14:35.523450+0000","last_clean":"2024-04-02T09:14:35.523450+0000","last_became_active":"2024-04-02T09:14:33.874433+0000","last_became_peered":"2024-04-02T09:14:33.874433+0000","last_unstale":"2024-04-02T09:14:35.523450+0000","last_undegraded":"2024-04-02T09:14:35.523450+0000","last_fullsized":"2024-04-02T09:14:35.523450+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:07:50.116863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522576+0000","last_change":"2024-04-02T09:14:34.374573+0000","last_active":"2024-04-02T09:14:35.522576+0000","last_peered":"2024-04-02T09:14:35.522576+0000","last_clean":"2024-04-02T09:14:35.522576+0000","last_became_active":"2024-04-02T09:14:34.374327+0000","last_became_peered":"2024-04-02T09:14:34.374327+0000","last_unstale":"2024-04-02T09:14:35.522576+0000","last_undegraded":"2024-04-02T09:14:35.522576+0000","last_fullsized":"2024-04-02T09:14:35.522576+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:31:00.788937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.13","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523486+0000","last_change":"2024-04-02T09:14:33.877528+0000","last_active":"2024-04-02T09:14:35.523486+0000","last_peered":"2024-04-02T09:14:35.523486+0000","last_clean":"2024-04-02T09:14:35.523486+0000","last_became_active":"2024-04-02T09:14:33.877262+0000","last_became_peered":"2024-04-02T09:14:33.877262+0000","last_unstale":"2024-04-02T09:14:35.523486+0000","last_undegraded":"2024-04-02T09:14:35.523486+0000","last_fullsized":"2024-04-02T09:14:35.523486+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:06:40.311963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522781+0000","last_change":"2024-04-02T09:14:34.380073+0000","last_active":"2024-04-02T09:14:35.522781+0000","last_peered":"2024-04-02T09:14:35.522781+0000","last_clean":"2024-04-02T09:14:35.522781+0000","last_became_active":"2024-04-02T09:14:34.379603+0000","last_became_peered":"2024-04-02T09:14:34.379603+0000","last_unstale":"2024-04-02T09:14:35.522781+0000","last_undegraded":"2024-04-02T09:14:35.522781+0000","last_fullsized":"2024-04-02T09:14:35.522781+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:25:45.536966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364291+0000","last_change":"2024-04-02T09:14:33.876709+0000","last_active":"2024-04-02T09:14:35.364291+0000","last_peered":"2024-04-02T09:14:35.364291+0000","last_clean":"2024-04-02T09:14:35.364291+0000","last_became_active":"2024-04-02T09:14:33.876371+0000","last_became_peered":"2024-04-02T09:14:33.876371+0000","last_unstale":"2024-04-02T09:14:35.364291+0000","last_undegraded":"2024-04-02T09:14:35.364291+0000","last_fullsized":"2024-04-02T09:14:35.364291+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:32:30.574385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522551+0000","last_change":"2024-04-02T09:14:34.381595+0000","last_active":"2024-04-02T09:14:35.522551+0000","last_peered":"2024-04-02T09:14:35.522551+0000","last_clean":"2024-04-02T09:14:35.522551+0000","last_became_active":"2024-04-02T09:14:34.381321+0000","last_became_peered":"2024-04-02T09:14:34.381321+0000","last_unstale":"2024-04-02T09:14:35.522551+0000","last_undegraded":"2024-04-02T09:14:35.522551+0000","last_fullsized":"2024-04-02T09:14:35.522551+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:38:27.813013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522721+0000","last_change":"2024-04-02T09:14:33.869087+0000","last_active":"2024-04-02T09:14:35.522721+0000","last_peered":"2024-04-02T09:14:35.522721+0000","last_clean":"2024-04-02T09:14:35.522721+0000","last_became_active":"2024-04-02T09:14:33.868778+0000","last_became_peered":"2024-04-02T09:14:33.868778+0000","last_unstale":"2024-04-02T09:14:35.522721+0000","last_undegraded":"2024-04-02T09:14:35.522721+0000","last_fullsized":"2024-04-02T09:14:35.522721+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:53:25.730047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522621+0000","last_change":"2024-04-02T09:14:34.382115+0000","last_active":"2024-04-02T09:14:35.522621+0000","last_peered":"2024-04-02T09:14:35.522621+0000","last_clean":"2024-04-02T09:14:35.522621+0000","last_became_active":"2024-04-02T09:14:34.381857+0000","last_became_peered":"2024-04-02T09:14:34.381857+0000","last_unstale":"2024-04-02T09:14:35.522621+0000","last_undegraded":"2024-04-02T09:14:35.522621+0000","last_fullsized":"2024-04-02T09:14:35.522621+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:43:47.084525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.10","version":"264'8","reported_seq":19,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522720+0000","last_change":"2024-04-02T09:14:33.869954+0000","last_active":"2024-04-02T09:14:35.522720+0000","last_peered":"2024-04-02T09:14:35.522720+0000","last_clean":"2024-04-02T09:14:35.522720+0000","last_became_active":"2024-04-02T09:14:33.867537+0000","last_became_peered":"2024-04-02T09:14:33.867537+0000","last_unstale":"2024-04-02T09:14:35.522720+0000","last_undegraded":"2024-04-02T09:14:35.522720+0000","last_fullsized":"2024-04-02T09:14:35.522720+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:36.927662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364624+0000","last_change":"2024-04-02T09:14:34.373375+0000","last_active":"2024-04-02T09:14:35.364624+0000","last_peered":"2024-04-02T09:14:35.364624+0000","last_clean":"2024-04-02T09:14:35.364624+0000","last_became_active":"2024-04-02T09:14:34.370488+0000","last_became_peered":"2024-04-02T09:14:34.370488+0000","last_unstale":"2024-04-02T09:14:35.364624+0000","last_undegraded":"2024-04-02T09:14:35.364624+0000","last_fullsized":"2024-04-02T09:14:35.364624+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:32:00.070021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.f","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523408+0000","last_change":"2024-04-02T09:14:33.869073+0000","last_active":"2024-04-02T09:14:35.523408+0000","last_peered":"2024-04-02T09:14:35.523408+0000","last_clean":"2024-04-02T09:14:35.523408+0000","last_became_active":"2024-04-02T09:14:33.868799+0000","last_became_peered":"2024-04-02T09:14:33.868799+0000","last_unstale":"2024-04-02T09:14:35.523408+0000","last_undegraded":"2024-04-02T09:14:35.523408+0000","last_fullsized":"2024-04-02T09:14:35.523408+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:56:02.437951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523338+0000","last_change":"2024-04-02T09:14:34.381984+0000","last_active":"2024-04-02T09:14:35.523338+0000","last_peered":"2024-04-02T09:14:35.523338+0000","last_clean":"2024-04-02T09:14:35.523338+0000","last_became_active":"2024-04-02T09:14:34.381724+0000","last_became_peered":"2024-04-02T09:14:34.381724+0000","last_unstale":"2024-04-02T09:14:35.523338+0000","last_undegraded":"2024-04-02T09:14:35.523338+0000","last_fullsized":"2024-04-02T09:14:35.523338+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:10:43.121011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.e","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523146+0000","last_change":"2024-04-02T09:14:33.877468+0000","last_active":"2024-04-02T09:14:35.523146+0000","last_peered":"2024-04-02T09:14:35.523146+0000","last_clean":"2024-04-02T09:14:35.523146+0000","last_became_active":"2024-04-02T09:14:33.877199+0000","last_became_peered":"2024-04-02T09:14:33.877199+0000","last_unstale":"2024-04-02T09:14:35.523146+0000","last_undegraded":"2024-04-02T09:14:35.523146+0000","last_fullsized":"2024-04-02T09:14:35.523146+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:11.469272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523352+0000","last_change":"2024-04-02T09:14:34.373693+0000","last_active":"2024-04-02T09:14:35.523352+0000","last_peered":"2024-04-02T09:14:35.523352+0000","last_clean":"2024-04-02T09:14:35.523352+0000","last_became_active":"2024-04-02T09:14:34.372531+0000","last_became_peered":"2024-04-02T09:14:34.372531+0000","last_unstale":"2024-04-02T09:14:35.523352+0000","last_undegraded":"2024-04-02T09:14:35.523352+0000","last_fullsized":"2024-04-02T09:14:35.523352+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:24:29.851257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.d","version":"264'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523408+0000","last_change":"2024-04-02T09:14:33.874303+0000","last_active":"2024-04-02T09:14:35.523408+0000","last_peered":"2024-04-02T09:14:35.523408+0000","last_clean":"2024-04-02T09:14:35.523408+0000","last_became_active":"2024-04-02T09:14:33.874024+0000","last_became_peered":"2024-04-02T09:14:33.874024+0000","last_unstale":"2024-04-02T09:14:35.523408+0000","last_undegraded":"2024-04-02T09:14:35.523408+0000","last_fullsized":"2024-04-02T09:14:35.523408+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:23:35.199298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523471+0000","last_change":"2024-04-02T09:14:34.373452+0000","last_active":"2024-04-02T09:14:35.523471+0000","last_peered":"2024-04-02T09:14:35.523471+0000","last_clean":"2024-04-02T09:14:35.523471+0000","last_became_active":"2024-04-02T09:14:34.372772+0000","last_became_peered":"2024-04-02T09:14:34.372772+0000","last_unstale":"2024-04-02T09:14:35.523471+0000","last_undegraded":"2024-04-02T09:14:35.523471+0000","last_fullsized":"2024-04-02T09:14:35.523471+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:53:41.722545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523664+0000","last_change":"2024-04-02T09:14:33.863647+0000","last_active":"2024-04-02T09:14:35.523664+0000","last_peered":"2024-04-02T09:14:35.523664+0000","last_clean":"2024-04-02T09:14:35.523664+0000","last_became_active":"2024-04-02T09:14:33.863385+0000","last_became_peered":"2024-04-02T09:14:33.863385+0000","last_unstale":"2024-04-02T09:14:35.523664+0000","last_undegraded":"2024-04-02T09:14:35.523664+0000","last_fullsized":"2024-04-02T09:14:35.523664+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:57:55.628012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523120+0000","last_change":"2024-04-02T09:14:34.390484+0000","last_active":"2024-04-02T09:14:35.523120+0000","last_peered":"2024-04-02T09:14:35.523120+0000","last_clean":"2024-04-02T09:14:35.523120+0000","last_became_active":"2024-04-02T09:14:34.390145+0000","last_became_peered":"2024-04-02T09:14:34.390145+0000","last_unstale":"2024-04-02T09:14:35.523120+0000","last_undegraded":"2024-04-02T09:14:35.523120+0000","last_fullsized":"2024-04-02T09:14:35.523120+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:43:52.860015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.b","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364214+0000","last_change":"2024-04-02T09:14:33.876355+0000","last_active":"2024-04-02T09:14:35.364214+0000","last_peered":"2024-04-02T09:14:35.364214+0000","last_clean":"2024-04-02T09:14:35.364214+0000","last_became_active":"2024-04-02T09:14:33.876041+0000","last_became_peered":"2024-04-02T09:14:33.876041+0000","last_unstale":"2024-04-02T09:14:35.364214+0000","last_undegraded":"2024-04-02T09:14:35.364214+0000","last_fullsized":"2024-04-02T09:14:35.364214+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:10:38.419248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362885+0000","last_change":"2024-04-02T09:14:34.388100+0000","last_active":"2024-04-02T09:14:35.362885+0000","last_peered":"2024-04-02T09:14:35.362885+0000","last_clean":"2024-04-02T09:14:35.362885+0000","last_became_active":"2024-04-02T09:14:34.383116+0000","last_became_peered":"2024-04-02T09:14:34.383116+0000","last_unstale":"2024-04-02T09:14:35.362885+0000","last_undegraded":"2024-04-02T09:14:35.362885+0000","last_fullsized":"2024-04-02T09:14:35.362885+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:11:51.893190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.364368+0000","last_change":"2024-04-02T09:14:33.878926+0000","last_active":"2024-04-02T09:14:35.364368+0000","last_peered":"2024-04-02T09:14:35.364368+0000","last_clean":"2024-04-02T09:14:35.364368+0000","last_became_active":"2024-04-02T09:14:33.878596+0000","last_became_peered":"2024-04-02T09:14:33.878596+0000","last_unstale":"2024-04-02T09:14:35.364368+0000","last_undegraded":"2024-04-02T09:14:35.364368+0000","last_fullsized":"2024-04-02T09:14:35.364368+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:14:19.001834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523585+0000","last_change":"2024-04-02T09:14:34.374525+0000","last_active":"2024-04-02T09:14:35.523585+0000","last_peered":"2024-04-02T09:14:35.523585+0000","last_clean":"2024-04-02T09:14:35.523585+0000","last_became_active":"2024-04-02T09:14:34.374214+0000","last_became_peered":"2024-04-02T09:14:34.374214+0000","last_unstale":"2024-04-02T09:14:35.523585+0000","last_undegraded":"2024-04-02T09:14:35.523585+0000","last_fullsized":"2024-04-02T09:14:35.523585+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:20:30.074905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362662+0000","last_change":"2024-04-02T09:14:33.872263+0000","last_active":"2024-04-02T09:14:35.362662+0000","last_peered":"2024-04-02T09:14:35.362662+0000","last_clean":"2024-04-02T09:14:35.362662+0000","last_became_active":"2024-04-02T09:14:33.871830+0000","last_became_peered":"2024-04-02T09:14:33.871830+0000","last_unstale":"2024-04-02T09:14:35.362662+0000","last_undegraded":"2024-04-02T09:14:35.362662+0000","last_fullsized":"2024-04-02T09:14:35.362662+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:31:11.469272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522704+0000","last_change":"2024-04-02T09:14:34.374102+0000","last_active":"2024-04-02T09:14:35.522704+0000","last_peered":"2024-04-02T09:14:35.522704+0000","last_clean":"2024-04-02T09:14:35.522704+0000","last_became_active":"2024-04-02T09:14:34.373810+0000","last_became_peered":"2024-04-02T09:14:34.373810+0000","last_unstale":"2024-04-02T09:14:35.522704+0000","last_undegraded":"2024-04-02T09:14:35.522704+0000","last_fullsized":"2024-04-02T09:14:35.522704+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:20:30.074905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363497+0000","last_change":"2024-04-02T09:14:33.867489+0000","last_active":"2024-04-02T09:14:35.363497+0000","last_peered":"2024-04-02T09:14:35.363497+0000","last_clean":"2024-04-02T09:14:35.363497+0000","last_became_active":"2024-04-02T09:14:33.867206+0000","last_became_peered":"2024-04-02T09:14:33.867206+0000","last_unstale":"2024-04-02T09:14:35.363497+0000","last_undegraded":"2024-04-02T09:14:35.363497+0000","last_fullsized":"2024-04-02T09:14:35.363497+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:55:44.507753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362972+0000","last_change":"2024-04-02T09:14:34.380478+0000","last_active":"2024-04-02T09:14:35.362972+0000","last_peered":"2024-04-02T09:14:35.362972+0000","last_clean":"2024-04-02T09:14:35.362972+0000","last_became_active":"2024-04-02T09:14:34.380228+0000","last_became_peered":"2024-04-02T09:14:34.380228+0000","last_unstale":"2024-04-02T09:14:35.362972+0000","last_undegraded":"2024-04-02T09:14:35.362972+0000","last_fullsized":"2024-04-02T09:14:35.362972+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:56:28.009617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.7","version":"264'2","reported_seq":13,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362982+0000","last_change":"2024-04-02T09:14:33.878819+0000","last_active":"2024-04-02T09:14:35.362982+0000","last_peered":"2024-04-02T09:14:35.362982+0000","last_clean":"2024-04-02T09:14:35.362982+0000","last_became_active":"2024-04-02T09:14:33.878105+0000","last_became_peered":"2024-04-02T09:14:33.878105+0000","last_unstale":"2024-04-02T09:14:35.362982+0000","last_undegraded":"2024-04-02T09:14:35.362982+0000","last_fullsized":"2024-04-02T09:14:35.362982+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:17:45.067998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523932+0000","last_change":"2024-04-02T09:14:34.372991+0000","last_active":"2024-04-02T09:14:35.523932+0000","last_peered":"2024-04-02T09:14:35.523932+0000","last_clean":"2024-04-02T09:14:35.523932+0000","last_became_active":"2024-04-02T09:14:34.372739+0000","last_became_peered":"2024-04-02T09:14:34.372739+0000","last_unstale":"2024-04-02T09:14:35.523932+0000","last_undegraded":"2024-04-02T09:14:35.523932+0000","last_fullsized":"2024-04-02T09:14:35.523932+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:26.825924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"265'1","reported_seq":12,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:37.266978+0000","last_change":"2024-04-02T09:14:33.871269+0000","last_active":"2024-04-02T09:14:37.266978+0000","last_peered":"2024-04-02T09:14:37.266978+0000","last_clean":"2024-04-02T09:14:37.266978+0000","last_became_active":"2024-04-02T09:14:33.870767+0000","last_became_peered":"2024-04-02T09:14:33.870767+0000","last_unstale":"2024-04-02T09:14:37.266978+0000","last_undegraded":"2024-04-02T09:14:37.266978+0000","last_fullsized":"2024-04-02T09:14:37.266978+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:55:47.421500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362682+0000","last_change":"2024-04-02T09:14:34.382888+0000","last_active":"2024-04-02T09:14:35.362682+0000","last_peered":"2024-04-02T09:14:35.362682+0000","last_clean":"2024-04-02T09:14:35.362682+0000","last_became_active":"2024-04-02T09:14:34.382551+0000","last_became_peered":"2024-04-02T09:14:34.382551+0000","last_unstale":"2024-04-02T09:14:35.362682+0000","last_undegraded":"2024-04-02T09:14:35.362682+0000","last_fullsized":"2024-04-02T09:14:35.362682+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:44:36.082626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362917+0000","last_change":"2024-04-02T09:14:33.876070+0000","last_active":"2024-04-02T09:14:35.362917+0000","last_peered":"2024-04-02T09:14:35.362917+0000","last_clean":"2024-04-02T09:14:35.362917+0000","last_became_active":"2024-04-02T09:14:33.875825+0000","last_became_peered":"2024-04-02T09:14:33.875825+0000","last_unstale":"2024-04-02T09:14:35.362917+0000","last_undegraded":"2024-04-02T09:14:35.362917+0000","last_fullsized":"2024-04-02T09:14:35.362917+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:23:35.199298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.363906+0000","last_change":"2024-04-02T09:14:34.362399+0000","last_active":"2024-04-02T09:14:35.363906+0000","last_peered":"2024-04-02T09:14:35.363906+0000","last_clean":"2024-04-02T09:14:35.363906+0000","last_became_active":"2024-04-02T09:14:34.361978+0000","last_became_peered":"2024-04-02T09:14:34.361978+0000","last_unstale":"2024-04-02T09:14:35.363906+0000","last_undegraded":"2024-04-02T09:14:35.363906+0000","last_fullsized":"2024-04-02T09:14:35.363906+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:15.964840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523592+0000","last_change":"2024-04-02T09:14:33.866042+0000","last_active":"2024-04-02T09:14:35.523592+0000","last_peered":"2024-04-02T09:14:35.523592+0000","last_clean":"2024-04-02T09:14:35.523592+0000","last_became_active":"2024-04-02T09:14:33.865757+0000","last_became_peered":"2024-04-02T09:14:33.865757+0000","last_unstale":"2024-04-02T09:14:35.523592+0000","last_undegraded":"2024-04-02T09:14:35.523592+0000","last_fullsized":"2024-04-02T09:14:35.523592+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:24.796757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522937+0000","last_change":"2024-04-02T09:14:34.375746+0000","last_active":"2024-04-02T09:14:35.522937+0000","last_peered":"2024-04-02T09:14:35.522937+0000","last_clean":"2024-04-02T09:14:35.522937+0000","last_became_active":"2024-04-02T09:14:34.375131+0000","last_became_peered":"2024-04-02T09:14:34.375131+0000","last_unstale":"2024-04-02T09:14:35.522937+0000","last_undegraded":"2024-04-02T09:14:35.522937+0000","last_fullsized":"2024-04-02T09:14:35.522937+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:03:24.411151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523040+0000","last_change":"2024-04-02T09:14:33.868955+0000","last_active":"2024-04-02T09:14:35.523040+0000","last_peered":"2024-04-02T09:14:35.523040+0000","last_clean":"2024-04-02T09:14:35.523040+0000","last_became_active":"2024-04-02T09:14:33.868639+0000","last_became_peered":"2024-04-02T09:14:33.868639+0000","last_unstale":"2024-04-02T09:14:35.523040+0000","last_undegraded":"2024-04-02T09:14:35.523040+0000","last_fullsized":"2024-04-02T09:14:35.523040+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:39:56.716397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522916+0000","last_change":"2024-04-02T09:14:34.379918+0000","last_active":"2024-04-02T09:14:35.522916+0000","last_peered":"2024-04-02T09:14:35.522916+0000","last_clean":"2024-04-02T09:14:35.522916+0000","last_became_active":"2024-04-02T09:14:34.379284+0000","last_became_peered":"2024-04-02T09:14:34.379284+0000","last_unstale":"2024-04-02T09:14:35.522916+0000","last_undegraded":"2024-04-02T09:14:35.522916+0000","last_fullsized":"2024-04-02T09:14:35.522916+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:53:01.672940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522856+0000","last_change":"2024-04-02T09:14:33.870645+0000","last_active":"2024-04-02T09:14:35.522856+0000","last_peered":"2024-04-02T09:14:35.522856+0000","last_clean":"2024-04-02T09:14:35.522856+0000","last_became_active":"2024-04-02T09:14:33.870355+0000","last_became_peered":"2024-04-02T09:14:33.870355+0000","last_unstale":"2024-04-02T09:14:35.522856+0000","last_undegraded":"2024-04-02T09:14:35.522856+0000","last_fullsized":"2024-04-02T09:14:35.522856+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:53:25.730047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.523681+0000","last_change":"2024-04-02T09:14:34.370359+0000","last_active":"2024-04-02T09:14:35.523681+0000","last_peered":"2024-04-02T09:14:35.523681+0000","last_clean":"2024-04-02T09:14:35.523681+0000","last_became_active":"2024-04-02T09:14:34.370115+0000","last_became_peered":"2024-04-02T09:14:34.370115+0000","last_unstale":"2024-04-02T09:14:35.523681+0000","last_undegraded":"2024-04-02T09:14:35.523681+0000","last_fullsized":"2024-04-02T09:14:35.523681+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:43:52.860015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522693+0000","last_change":"2024-04-02T09:14:33.865894+0000","last_active":"2024-04-02T09:14:35.522693+0000","last_peered":"2024-04-02T09:14:35.522693+0000","last_clean":"2024-04-02T09:14:35.522693+0000","last_became_active":"2024-04-02T09:14:33.865610+0000","last_became_peered":"2024-04-02T09:14:33.865610+0000","last_unstale":"2024-04-02T09:14:35.522693+0000","last_undegraded":"2024-04-02T09:14:35.522693+0000","last_fullsized":"2024-04-02T09:14:35.522693+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:41.131205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522925+0000","last_change":"2024-04-02T09:14:34.372238+0000","last_active":"2024-04-02T09:14:35.522925+0000","last_peered":"2024-04-02T09:14:35.522925+0000","last_clean":"2024-04-02T09:14:35.522925+0000","last_became_active":"2024-04-02T09:14:34.371553+0000","last_became_peered":"2024-04-02T09:14:34.371553+0000","last_unstale":"2024-04-02T09:14:35.522925+0000","last_undegraded":"2024-04-02T09:14:35.522925+0000","last_fullsized":"2024-04-02T09:14:35.522925+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:03:24.411151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.522812+0000","last_change":"2024-04-02T09:14:33.874635+0000","last_active":"2024-04-02T09:14:35.522812+0000","last_peered":"2024-04-02T09:14:35.522812+0000","last_clean":"2024-04-02T09:14:35.522812+0000","last_became_active":"2024-04-02T09:14:33.874348+0000","last_became_peered":"2024-04-02T09:14:33.874348+0000","last_unstale":"2024-04-02T09:14:35.522812+0000","last_undegraded":"2024-04-02T09:14:35.522812+0000","last_fullsized":"2024-04-02T09:14:35.522812+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:32.836277+0000","last_clean_scrub_stamp":"2024-04-02T09:14:32.836277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:18:07.343969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":10,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-02T09:14:35.362709+0000","last_change":"2024-04-02T09:14:34.387134+0000","last_active":"2024-04-02T09:14:35.362709+0000","last_peered":"2024-04-02T09:14:35.362709+0000","last_clean":"2024-04-02T09:14:35.362709+0000","last_became_active":"2024-04-02T09:14:34.386882+0000","last_became_peered":"2024-04-02T09:14:34.386882+0000","last_unstale":"2024-04-02T09:14:35.362709+0000","last_undegraded":"2024-04-02T09:14:35.362709+0000","last_fullsized":"2024-04-02T09:14:35.362709+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:14:33.844811+0000","last_clean_scrub_stamp":"2024-04-02T09:14:33.844811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:44:06.694478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"250'381","reported_seq":1365,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-04-02T09:14:34.367154+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:14:34.367154+0000","last_peered":"2024-04-02T09:14:34.367154+0000","last_clean":"2024-04-02T09:14:34.367154+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:14:34.367154+0000","last_undegraded":"2024-04-02T09:14:34.367154+0000","last_fullsized":"2024-04-02T09:14:34.367154+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576010,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60644,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311196,"statfs":{"total":96636764160,"available":96574664704,"internally_reserved":0,"allocated":3358720,"data_stored":1721457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.70899999999999996,"15min":0.71799999999999997},"min":{"1min":0.34599999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.80000000000000004,"5min":8.016,"15min":8.016},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.69899999999999995,"15min":0.71899999999999997},"min":{"1min":0.379,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.83299999999999996,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.754}]},{"osd":1,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.74299999999999999,"15min":0.74199999999999999},"min":{"1min":0.46300000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91000000000000003,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.42399999999999999},"max":{"1min":0.73499999999999999,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.57499999999999996}]},{"osd":2,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.77900000000000003,"15min":0.76900000000000002},"min":{"1min":0.46999999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86199999999999999,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.78400000000000003,"15min":0.80400000000000005},"min":{"1min":0.53800000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.872,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.68999999999999995}]},{"osd":3,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.751,"15min":0.749},"min":{"1min":0.49199999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86299999999999999,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.79000000000000004,"15min":0.77200000000000002},"min":{"1min":0.51900000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":5.093,"15min":5.093},"last":0.78700000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.66400000000000003,"15min":0.624},"min":{"1min":0.39200000000000002,"5min":0.38300000000000001,"15min":0.36499999999999999},"max":{"1min":0.83699999999999997,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.746},"min":{"1min":0.58499999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90100000000000002,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.76800000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.73199999999999998,"15min":0.69399999999999995},"min":{"1min":0.436,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":0.88600000000000001,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.621},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.77000000000000002,"15min":0.73399999999999999},"min":{"1min":0.54000000000000004,"5min":0.42599999999999999,"15min":0.41099999999999998},"max":{"1min":0.878,"5min":3.492,"15min":3.492},"last":0.73999999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.72099999999999997,"15min":0.69599999999999995},"min":{"1min":0.51300000000000001,"5min":0.48299999999999998,"15min":0.47099999999999997},"max":{"1min":0.90700000000000003,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.77600000000000002,"15min":0.71699999999999997},"min":{"1min":0.42199999999999999,"5min":0.41599999999999998,"15min":0.41099999999999998},"max":{"1min":0.91600000000000004,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.72299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834576008,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42600,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329240,"statfs":{"total":96636764160,"available":96593141760,"internally_reserved":0,"allocated":3362816,"data_stored":1722909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68200000000000005,"15min":0.72099999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93999999999999995,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.73399999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.0600000000000001},"last":0.87}]},{"osd":1,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.67900000000000005,"15min":0.71299999999999997},"min":{"1min":0.46000000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.999,"5min":1.044,"15min":1.212},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.70799999999999996,"15min":0.747},"min":{"1min":0.505,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.0589999999999999,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.88400000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.69799999999999995,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94799999999999995,"5min":1.046,"15min":1.2789999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70099999999999996,"15min":0.73299999999999998},"min":{"1min":0.47499999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.226},"last":0.90000000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.71999999999999997,"15min":0.75600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.026,"5min":1.141,"15min":1.2549999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.04,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.96499999999999997}]},{"osd":4,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.628,"15min":0.626},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65800000000000003,"15min":0.66200000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.41999999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1879999999999999},"last":0.754}]},{"osd":5,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.64600000000000002,"15min":0.65200000000000002},"min":{"1min":0.433,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88800000000000001,"5min":0.96599999999999997,"15min":1.2669999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.70699999999999996,"15min":0.72499999999999998},"min":{"1min":0.501,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.93500000000000005}]},{"osd":7,"last update":"Tue Apr 2 09:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73899999999999999},"min":{"1min":0.48499999999999999,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":1.0129999999999999,"5min":1.21,"15min":1.2370000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.72899999999999998,"15min":0.73899999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.49399999999999999},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.94999999999999996}]}]},{"osd":1,"up_from":13,"seq":55834576009,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57120,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314720,"statfs":{"total":96636764160,"available":96578273280,"internally_reserved":0,"allocated":3940352,"data_stored":2311679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42899999999999999,"15min":0.44800000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82499999999999996},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48299999999999998,"15min":0.503},"min":{"1min":0.31900000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.79800000000000004},"last":0.60399999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.33600000000000002,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.59599999999999997,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58699999999999997},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.82599999999999996}]},{"osd":3,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54100000000000004,"15min":0.55600000000000005},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.69099999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.76000000000000001,"15min":0.78100000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.433},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.876},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.753,"15min":0.77700000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.41799999999999998},"max":{"1min":0.98299999999999998,"5min":1.05,"15min":1.25},"last":1.0009999999999999}]},{"osd":5,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.753,"15min":0.78200000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.93400000000000005,"5min":1.03,"15min":1.321},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78400000000000003,"15min":0.80100000000000005},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.115,"5min":1.115,"15min":1.173},"last":0.98199999999999998}]},{"osd":6,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.78300000000000003,"15min":0.80700000000000005},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.48099999999999998},"max":{"1min":1.087,"5min":1.087,"15min":1.2210000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.79100000000000004,"15min":0.80600000000000005},"min":{"1min":0.52900000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.024,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.94999999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.78900000000000003,"15min":0.81499999999999995},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0740000000000001,"5min":1.1080000000000001,"15min":1.236},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81200000000000006,"15min":0.83099999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.129,"5min":1.129,"15min":1.2789999999999999},"last":0.80500000000000005}]}]},{"osd":0,"up_from":13,"seq":55834576009,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52448,"kb_used_data":3272,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319392,"statfs":{"total":96636764160,"available":96583057408,"internally_reserved":0,"allocated":3350528,"data_stored":1720944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.42699999999999999,"15min":0.45000000000000001},"min":{"1min":0.26700000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.58999999999999997,"5min":0.69899999999999995,"15min":0.86699999999999999},"last":0.432},{"interface":"front","average":{"1min":0.438,"5min":0.48099999999999998,"15min":0.499},"min":{"1min":0.29799999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66300000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.52200000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.48699999999999999,"15min":0.50600000000000001},"min":{"1min":0.35699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.621,"5min":0.69399999999999995,"15min":0.82699999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52100000000000002,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.63300000000000001,"5min":0.999,"15min":0.999},"last":0.46999999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.53100000000000003,"15min":0.56100000000000005},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.63500000000000001,"5min":0.79800000000000004,"15min":0.85699999999999998},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.63},"min":{"1min":0.47599999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.66000000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58099999999999996}]},{"osd":4,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.82199999999999995},"min":{"1min":0.622,"5min":0.54600000000000004,"15min":0.45600000000000002},"max":{"1min":0.86499999999999999,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.76000000000000001,"15min":0.81599999999999995},"min":{"1min":0.55100000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.90200000000000002,"5min":1.169,"15min":1.169},"last":0.496}]},{"osd":5,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.76500000000000001,"15min":0.81699999999999995},"min":{"1min":0.58299999999999996,"5min":0.497,"15min":0.497},"max":{"1min":0.95199999999999996,"5min":1.145,"15min":1.232},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.78500000000000003,"15min":0.82499999999999996},"min":{"1min":0.53600000000000003,"5min":0.497,"15min":0.497},"max":{"1min":0.94399999999999995,"5min":1.21,"15min":1.21},"last":0.623}]},{"osd":6,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.75,"15min":0.79100000000000004},"min":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.88200000000000001,"5min":1.1830000000000001,"15min":1.196},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.77200000000000002,"15min":0.80400000000000005},"min":{"1min":0.54200000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96799999999999997,"5min":1.246,"15min":1.246},"last":0.55700000000000005}]},{"osd":7,"last update":"Tue Apr 2 09:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.80700000000000005,"15min":0.84799999999999998},"min":{"1min":0.60999999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.98699999999999999,"5min":1.292,"15min":1.292},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.79800000000000004,"15min":0.83999999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.95799999999999996,"5min":1.1719999999999999,"15min":1.2130000000000001},"last":0.70999999999999996}]}]},{"osd":2,"up_from":13,"seq":55834576008,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52440,"kb_used_data":3268,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319400,"statfs":{"total":96636764160,"available":96583065600,"internally_reserved":0,"allocated":3346432,"data_stored":1721311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.45100000000000001,"15min":0.443},"min":{"1min":0.28899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.31,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":0.54900000000000004}]},{"osd":1,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.496,"15min":0.503},"min":{"1min":0.34000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53600000000000003,"15min":0.55400000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.41699999999999998}]},{"osd":3,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.71199999999999997,"5min":0.85099999999999998,"15min":1.2749999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.626},"min":{"1min":0.45400000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.48499999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.75900000000000001,"15min":0.77300000000000002},"min":{"1min":0.56699999999999995,"5min":0.495,"15min":0.495},"max":{"1min":2.3999999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.77500000000000002,"15min":0.78900000000000003},"min":{"1min":0.54600000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.74399999999999999}]},{"osd":5,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.77600000000000002,"15min":0.78600000000000003},"min":{"1min":0.57799999999999996,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1379999999999999,"5min":1.99,"15min":1.99},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.78000000000000003,"15min":0.80100000000000005},"min":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.70399999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79100000000000004,"15min":0.80500000000000005},"min":{"1min":0.59299999999999997,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77300000000000002,"15min":0.80100000000000005},"min":{"1min":0.55900000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.103,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.67900000000000005}]},{"osd":7,"last update":"Tue Apr 2 09:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.80000000000000004,"15min":0.82399999999999995},"min":{"1min":0.56999999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.82199999999999995},"min":{"1min":0.59099999999999997,"5min":0.497,"15min":0.497},"max":{"1min":0.97499999999999998,"5min":1.1040000000000001,"15min":1.3460000000000001},"last":0.64600000000000002}]}]},{"osd":3,"up_from":13,"seq":55834576008,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52464,"kb_used_data":3292,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319376,"statfs":{"total":96636764160,"available":96583041024,"internally_reserved":0,"allocated":3371008,"data_stored":1723057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.41699999999999998,"15min":0.443},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64000000000000001,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.58699999999999997,"5min":0.71199999999999997,"15min":0.876},"last":0.44800000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47499999999999998,"15min":0.498},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.90300000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.53900000000000003,"15min":0.56799999999999995},"min":{"1min":0.38500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.66100000000000003,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.46100000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.58099999999999996,"15min":0.61699999999999999},"min":{"1min":0.41299999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71299999999999997,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56799999999999995,"15min":0.60999999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":1.0840000000000001},"last":0.42599999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72099999999999997,"15min":0.751},"min":{"1min":0.438,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.73499999999999999,"15min":0.76600000000000001},"min":{"1min":0.49299999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.97099999999999997,"5min":1.0569999999999999,"15min":1.1619999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.74299999999999999,"15min":0.77400000000000002},"min":{"1min":0.59399999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.75700000000000001,"15min":0.78000000000000003},"min":{"1min":0.51100000000000001,"5min":0.505,"15min":0.505},"max":{"1min":0.98299999999999998,"5min":1.266,"15min":1.266},"last":0.64200000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.96799999999999997,"5min":1.284,"15min":1.284},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76500000000000001,"15min":0.80300000000000005},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.495},"max":{"1min":1.0389999999999999,"5min":1.2,"15min":1.2},"last":0.751}]},{"osd":7,"last update":"Tue Apr 2 09:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.76600000000000001,"15min":0.80400000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.125,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.78900000000000003,"15min":0.81699999999999995},"min":{"1min":0.57599999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.087,"5min":1.248,"15min":1.248},"last":0.73899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576008,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48344,"kb_used_data":3268,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323496,"statfs":{"total":96636764160,"available":96587259904,"internally_reserved":0,"allocated":3346432,"data_stored":1721311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64500000000000002,"15min":0.67700000000000005},"min":{"1min":0.38100000000000001,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":0.94399999999999995,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.63600000000000001,"15min":0.67100000000000004},"min":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.075},"last":0.56100000000000005}]},{"osd":1,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.70499999999999996,"15min":0.746},"min":{"1min":0.51700000000000002,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":1.014,"5min":1.0549999999999999,"15min":1.069},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72199999999999998,"15min":0.75700000000000001},"min":{"1min":0.49099999999999999,"5min":0.42699999999999999,"15min":0.40799999999999997},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.68600000000000005}]},{"osd":2,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66200000000000003,"15min":0.69999999999999996},"min":{"1min":0.439,"5min":0.438,"15min":0.438},"max":{"1min":0.95799999999999996,"5min":1.077,"15min":1.077},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68700000000000006,"15min":0.71899999999999997},"min":{"1min":0.47799999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.032,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.63700000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.71399999999999997,"15min":0.753},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.47399999999999998},"max":{"1min":0.93999999999999995,"5min":0.98099999999999998,"15min":1.018},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70999999999999996,"15min":0.74199999999999999},"min":{"1min":0.52100000000000002,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.752}]},{"osd":5,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.60899999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.41299999999999998,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.93700000000000006,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.70199999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.64200000000000002,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.91800000000000004,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.746,"5min":0.69199999999999995,"15min":0.69599999999999995},"min":{"1min":0.46899999999999997,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":1.0149999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.76600000000000001}]},{"osd":7,"last update":"Tue Apr 2 09:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.68500000000000005,"15min":0.70499999999999996},"min":{"1min":0.45500000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.0329999999999999},"last":0.88},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76500000000000001,"15min":0.78000000000000003},"min":{"1min":0.624,"5min":0.5,"15min":0.5},"max":{"1min":1.0700000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.78100000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576011,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53012,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318828,"statfs":{"total":96636764160,"available":96582479872,"internally_reserved":0,"allocated":3940352,"data_stored":2311242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68700000000000006,"15min":0.70299999999999996},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.82699999999999996,"5min":0.98699999999999999,"15min":1.004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.68999999999999995,"15min":0.71999999999999997},"min":{"1min":0.48399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.028,"5min":1.085,"15min":1.085},"last":0.72899999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.68100000000000005,"15min":0.71499999999999997},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84399999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70999999999999996,"15min":0.72499999999999998},"min":{"1min":0.52300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88500000000000001,"5min":0.95299999999999996,"15min":1.052},"last":0.65600000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.71599999999999997,"15min":0.73799999999999999},"min":{"1min":0.47199999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97299999999999998,"5min":1.004,"15min":1.014},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.69499999999999995,"15min":0.73699999999999999},"min":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.81799999999999995,"5min":1.018,"15min":1.0680000000000001},"last":0.747}]},{"osd":3,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.747,"15min":0.77700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.88500000000000001,"5min":0.98899999999999999,"15min":1.083},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.77000000000000002,"15min":0.79000000000000004},"min":{"1min":0.58399999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.1020000000000001},"last":0.67700000000000005}]},{"osd":4,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.40699999999999997,"5min":0.38300000000000001,"15min":0.373},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":1.054},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":0.98799999999999999,"5min":1.145,"15min":1.145},"last":0.58799999999999997}]},{"osd":6,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67900000000000005,"15min":0.67300000000000004},"min":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96799999999999997,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71399999999999997,"15min":0.71099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.94899999999999995,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.502}]},{"osd":7,"last update":"Tue Apr 2 09:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73999999999999999,"15min":0.73799999999999999},"min":{"1min":0.46500000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.0820000000000001,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76300000000000001,"15min":0.76600000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1519999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.63200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:14:44.122 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T09:14:44.123 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T09:14:44.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.360+0000 7fab79800700 1 Processor -- start 2024-04-02T09:14:44.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.360+0000 7fab79800700 1 -- start start 2024-04-02T09:14:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.362+0000 7fab79800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fab740cbb30 0x7fab740cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.362+0000 7fab79800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab740cd480 0x7fab7406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.362+0000 7fab79800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab740cc450 con 0x7fab740cd020 2024-04-02T09:14:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.362+0000 7fab79800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab740cdae0 con 0x7fab740cbb30 2024-04-02T09:14:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.362+0000 7fab79800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab740cddc0 con 0x7fab740cd480 2024-04-02T09:14:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.362+0000 7fab72a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fab740cbb30 0x7fab740cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.362+0000 7fab72a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fab740cbb30 0x7fab740cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35136/0 (socket says 172.21.15.89:35136) 2024-04-02T09:14:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.362+0000 7fab72a00700 1 -- 172.21.15.89:0/674382115 learned_addr learned my addr 172.21.15.89:0/674382115 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.362+0000 7fab72a00700 1 -- 172.21.15.89:0/674382115 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab740cd480 msgr2=0x7fab7406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.362+0000 7fab73400700 1 --2- 172.21.15.89:0/674382115 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab740cd480 0x7fab7406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.362+0000 7fab72a00700 1 --2- 172.21.15.89:0/674382115 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab740cd480 0x7fab7406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.362+0000 7fab72a00700 1 -- 172.21.15.89:0/674382115 >> v1:172.21.15.113:6789/0 conn(0x7fab740cd020 legacy=0x7fab740b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.362+0000 7fab72a00700 1 -- 172.21.15.89:0/674382115 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab740ce0a0 con 0x7fab740cbb30 2024-04-02T09:14:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.362+0000 7fab73400700 1 --2- 172.21.15.89:0/674382115 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab740cd480 0x7fab7406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.363+0000 7fab72a00700 1 --2- 172.21.15.89:0/674382115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab740cbb30 0x7fab740cbf10 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7fab640098f0 tx=0x7fab64009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7b82671b706e4d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.363+0000 7fab71600700 1 -- 172.21.15.89:0/674382115 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab64004230 con 0x7fab740cbb30 2024-04-02T09:14:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.363+0000 7fab71600700 1 -- 172.21.15.89:0/674382115 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab64004390 con 0x7fab740cbb30 2024-04-02T09:14:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.363+0000 7fab71600700 1 -- 172.21.15.89:0/674382115 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab640036a0 con 0x7fab740cbb30 2024-04-02T09:14:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.363+0000 7fab79800700 1 -- 172.21.15.89:0/674382115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab740cbb30 msgr2=0x7fab740cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.363+0000 7fab79800700 1 --2- 172.21.15.89:0/674382115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab740cbb30 0x7fab740cbf10 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7fab640098f0 tx=0x7fab64009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.363+0000 7fab79800700 1 -- 172.21.15.89:0/674382115 shutdown_connections 2024-04-02T09:14:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.363+0000 7fab79800700 1 --2- 172.21.15.89:0/674382115 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab740cd480 0x7fab7406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.363+0000 7fab79800700 1 --2- 172.21.15.89:0/674382115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab740cbb30 0x7fab740cbf10 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.363+0000 7fab79800700 1 -- 172.21.15.89:0/674382115 >> 172.21.15.89:0/674382115 conn(0x7fab740b0d70 msgr2=0x7fab740b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.363+0000 7fab79800700 1 -- 172.21.15.89:0/674382115 shutdown_connections 2024-04-02T09:14:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.364+0000 7fab79800700 1 -- 172.21.15.89:0/674382115 wait complete. 2024-04-02T09:14:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.365+0000 7fab79800700 1 Processor -- start 2024-04-02T09:14:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.365+0000 7fab79800700 1 -- start start 2024-04-02T09:14:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.366+0000 7fab79800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab740cbb30 0x7fab7409cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.366+0000 7fab79800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab740cd020 0x7fab740af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.366+0000 7fab79800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab740cd480 0x7fab741f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.366+0000 7fab79800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab74154fb0 con 0x7fab740cd480 2024-04-02T09:14:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.366+0000 7fab79800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab74155290 con 0x7fab740cbb30 2024-04-02T09:14:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.366+0000 7fab79800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab74155570 con 0x7fab740cd020 2024-04-02T09:14:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.367+0000 7fab73400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab740cd480 0x7fab741f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.367+0000 7fab72a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab740cbb30 0x7fab7409cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.367+0000 7fab73400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab740cd480 0x7fab741f34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35152/0 (socket says 172.21.15.89:35152) 2024-04-02T09:14:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.367+0000 7fab73400700 1 -- 172.21.15.89:0/88161583 learned_addr learned my addr 172.21.15.89:0/88161583 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.367+0000 7fab72000700 1 --2- 172.21.15.89:0/88161583 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab740cd020 0x7fab740af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.367+0000 7fab73400700 1 -- 172.21.15.89:0/88161583 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab740cd020 msgr2=0x7fab740af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.367+0000 7fab73400700 1 --2- 172.21.15.89:0/88161583 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab740cd020 0x7fab740af400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.367+0000 7fab73400700 1 -- 172.21.15.89:0/88161583 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab740cbb30 msgr2=0x7fab7409cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.367+0000 7fab73400700 1 --2- 172.21.15.89:0/88161583 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab740cbb30 0x7fab7409cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.367+0000 7fab73400700 1 -- 172.21.15.89:0/88161583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab64009550 con 0x7fab740cd480 2024-04-02T09:14:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.367+0000 7fab73400700 1 --2- 172.21.15.89:0/88161583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab740cd480 0x7fab741f34d0 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00cef0 tx=0x7fab6c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.368+0000 7fab6aa00700 1 -- 172.21.15.89:0/88161583 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c00ad00 con 0x7fab740cd480 2024-04-02T09:14:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.368+0000 7fab6aa00700 1 -- 172.21.15.89:0/88161583 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c00ae60 con 0x7fab740cd480 2024-04-02T09:14:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.368+0000 7fab79800700 1 -- 172.21.15.89:0/88161583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab74155850 con 0x7fab740cd480 2024-04-02T09:14:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.368+0000 7fab6aa00700 1 -- 172.21.15.89:0/88161583 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c015680 con 0x7fab740cd480 2024-04-02T09:14:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.368+0000 7fab79800700 1 -- 172.21.15.89:0/88161583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7409d0c0 con 0x7fab740cd480 2024-04-02T09:14:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.369+0000 7fab79800700 1 -- 172.21.15.89:0/88161583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab74155d70 con 0x7fab740cd480 2024-04-02T09:14:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.370+0000 7fab6aa00700 1 -- 172.21.15.89:0/88161583 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fab6c00d430 con 0x7fab740cd480 2024-04-02T09:14:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.370+0000 7fab6aa00700 1 --2- 172.21.15.89:0/88161583 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fab5406aed0 0x7fab5406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.371+0000 7fab6aa00700 1 -- 172.21.15.89:0/88161583 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fab6c089e50 con 0x7fab740cd480 2024-04-02T09:14:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.371+0000 7fab72a00700 1 --2- 172.21.15.89:0/88161583 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fab5406aed0 0x7fab5406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.371+0000 7fab72a00700 1 --2- 172.21.15.89:0/88161583 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fab5406aed0 0x7fab5406d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fab64009130 tx=0x7fab64008030 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:44.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.376+0000 7fab6aa00700 1 -- 172.21.15.89:0/88161583 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fab74155d70 con 0x7fab740cd480 2024-04-02T09:14:44.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.657+0000 7fab79800700 1 -- 172.21.15.89:0/88161583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fab74155d70 con 0x7fab740cd480 2024-04-02T09:14:44.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.661+0000 7fab6aa00700 1 -- 172.21.15.89:0/88161583 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v57) v1 ==== 190+0+0 (secure 0 0 0) 0x7fab74155d70 con 0x7fab740cd480 2024-04-02T09:14:44.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.664+0000 7fab79800700 1 -- 172.21.15.89:0/88161583 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fab5406aed0 msgr2=0x7fab5406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:44.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.664+0000 7fab79800700 1 --2- 172.21.15.89:0/88161583 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fab5406aed0 0x7fab5406d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fab64009130 tx=0x7fab64008030 comp rx=0 tx=0).stop 2024-04-02T09:14:44.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.664+0000 7fab79800700 1 -- 172.21.15.89:0/88161583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab740cd480 msgr2=0x7fab741f34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:44.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.664+0000 7fab79800700 1 --2- 172.21.15.89:0/88161583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab740cd480 0x7fab741f34d0 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00cef0 tx=0x7fab6c00cf20 comp rx=0 tx=0).stop 2024-04-02T09:14:44.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.664+0000 7fab79800700 1 -- 172.21.15.89:0/88161583 shutdown_connections 2024-04-02T09:14:44.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.664+0000 7fab79800700 1 --2- 172.21.15.89:0/88161583 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fab5406aed0 0x7fab5406d390 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.664+0000 7fab79800700 1 --2- 172.21.15.89:0/88161583 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab740cbb30 0x7fab7409cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.664+0000 7fab79800700 1 --2- 172.21.15.89:0/88161583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab740cd480 0x7fab741f34d0 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.664+0000 7fab79800700 1 --2- 172.21.15.89:0/88161583 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab740cd020 0x7fab740af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.664+0000 7fab79800700 1 -- 172.21.15.89:0/88161583 >> 172.21.15.89:0/88161583 conn(0x7fab740b0d70 msgr2=0x7fab7406e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:44.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.664+0000 7fab79800700 1 -- 172.21.15.89:0/88161583 shutdown_connections 2024-04-02T09:14:44.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.664+0000 7fab79800700 1 -- 172.21.15.89:0/88161583 wait complete. 2024-04-02T09:14:44.663 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T09:14:44.681 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:14:44.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.874+0000 7f8fa5400700 1 Processor -- start 2024-04-02T09:14:44.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.874+0000 7f8fa5400700 1 -- start start 2024-04-02T09:14:44.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.875+0000 7f8fa5400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa0160f10 0x7f8fa00d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:44.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.875+0000 7f8fa5400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8fa00cf230 0x7f8fa00d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:44.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.875+0000 7f8fa5400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00ceda0 con 0x7f8fa0161590 2024-04-02T09:14:44.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.875+0000 7f8fa5400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa00d23d0 con 0x7f8fa00cf230 2024-04-02T09:14:44.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.875+0000 7f8fa5400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa01612f0 con 0x7f8fa0160f10 2024-04-02T09:14:44.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.876+0000 7f8f9de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8fa00cf230 0x7f8fa00d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:44.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.876+0000 7f8f9de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8fa00cf230 0x7f8fa00d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35164/0 (socket says 172.21.15.89:35164) 2024-04-02T09:14:44.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.876+0000 7f8f9de00700 1 -- 172.21.15.89:0/2464596392 learned_addr learned my addr 172.21.15.89:0/2464596392 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:44.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.876+0000 7f8f9e800700 1 --2- 172.21.15.89:0/2464596392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa0160f10 0x7f8fa00d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:44.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.876+0000 7f8f9de00700 1 -- 172.21.15.89:0/2464596392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa0160f10 msgr2=0x7f8fa00d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:44.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.876+0000 7f8f9de00700 1 --2- 172.21.15.89:0/2464596392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa0160f10 0x7f8fa00d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.876+0000 7f8f9de00700 1 -- 172.21.15.89:0/2464596392 >> v1:172.21.15.113:6789/0 conn(0x7f8fa0161590 legacy=0x7f8fa00b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:44.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.876+0000 7f8f9de00700 1 -- 172.21.15.89:0/2464596392 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa00ce470 con 0x7f8fa00cf230 2024-04-02T09:14:44.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.876+0000 7f8f9de00700 1 --2- 172.21.15.89:0/2464596392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00cf230 0x7f8fa00d1e90 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c0098f0 tx=0x7f8f8c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d3519813d6b2e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:44.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.876+0000 7f8f9d400700 1 -- 172.21.15.89:0/2464596392 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c011830 con 0x7f8fa00cf230 2024-04-02T09:14:44.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.876+0000 7f8f9d400700 1 -- 172.21.15.89:0/2464596392 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8c011990 con 0x7f8fa00cf230 2024-04-02T09:14:44.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.876+0000 7f8f9d400700 1 -- 172.21.15.89:0/2464596392 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c011d20 con 0x7f8fa00cf230 2024-04-02T09:14:44.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.877+0000 7f8fa5400700 1 -- 172.21.15.89:0/2464596392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00cf230 msgr2=0x7f8fa00d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:44.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.877+0000 7f8fa5400700 1 --2- 172.21.15.89:0/2464596392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00cf230 0x7f8fa00d1e90 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c0098f0 tx=0x7f8f8c009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:44.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.877+0000 7f8fa5400700 1 -- 172.21.15.89:0/2464596392 shutdown_connections 2024-04-02T09:14:44.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.877+0000 7f8fa5400700 1 --2- 172.21.15.89:0/2464596392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa0160f10 0x7f8fa00d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.877+0000 7f8fa5400700 1 --2- 172.21.15.89:0/2464596392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00cf230 0x7f8fa00d1e90 unknown :-1 s=CLOSED pgs=2829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.877+0000 7f8fa5400700 1 -- 172.21.15.89:0/2464596392 >> 172.21.15.89:0/2464596392 conn(0x7f8fa008aff0 msgr2=0x7f8fa00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.877+0000 7f8fa5400700 1 -- 172.21.15.89:0/2464596392 shutdown_connections 2024-04-02T09:14:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.877+0000 7f8fa5400700 1 -- 172.21.15.89:0/2464596392 wait complete. 2024-04-02T09:14:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.879+0000 7f8fa5400700 1 Processor -- start 2024-04-02T09:14:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.879+0000 7f8fa5400700 1 -- start start 2024-04-02T09:14:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8fa5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00cf230 0x7f8fa00b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8fa5400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa0160f10 0x7f8fa01f7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8fa5400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fa0161590 0x7f8fa015f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8fa5400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa00b8d00 con 0x7f8fa00cf230 2024-04-02T09:14:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8fa5400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa00b6450 con 0x7f8fa0161590 2024-04-02T09:14:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8fa5400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa01f7890 con 0x7f8fa0160f10 2024-04-02T09:14:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8f9e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00cf230 0x7f8fa00b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8f9e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00cf230 0x7f8fa00b7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35174/0 (socket says 172.21.15.89:35174) 2024-04-02T09:14:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8f9e800700 1 -- 172.21.15.89:0/744459080 learned_addr learned my addr 172.21.15.89:0/744459080 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8f9f200700 1 --2- 172.21.15.89:0/744459080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fa0161590 0x7f8fa015f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8f9de00700 1 --2- 172.21.15.89:0/744459080 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa0160f10 0x7f8fa01f7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8f9e800700 1 -- 172.21.15.89:0/744459080 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa0160f10 msgr2=0x7f8fa01f7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:44.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8f9e800700 1 --2- 172.21.15.89:0/744459080 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa0160f10 0x7f8fa01f7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8f9e800700 1 -- 172.21.15.89:0/744459080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fa0161590 msgr2=0x7f8fa015f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:44.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8f9e800700 1 --2- 172.21.15.89:0/744459080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fa0161590 0x7f8fa015f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:44.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.880+0000 7f8f9e800700 1 -- 172.21.15.89:0/744459080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c009550 con 0x7f8fa00cf230 2024-04-02T09:14:44.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.881+0000 7f8f9e800700 1 --2- 172.21.15.89:0/744459080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00cf230 0x7f8fa00b7a80 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f8f9000e870 tx=0x7f8f9000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:44.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.881+0000 7f8f96a00700 1 -- 172.21.15.89:0/744459080 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f90009590 con 0x7f8fa00cf230 2024-04-02T09:14:44.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.881+0000 7f8f96a00700 1 -- 172.21.15.89:0/744459080 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9000ee70 con 0x7f8fa00cf230 2024-04-02T09:14:44.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.881+0000 7f8fa5400700 1 -- 172.21.15.89:0/744459080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa00b8080 con 0x7f8fa00cf230 2024-04-02T09:14:44.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.881+0000 7f8f96a00700 1 -- 172.21.15.89:0/744459080 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9000ca50 con 0x7f8fa00cf230 2024-04-02T09:14:44.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.881+0000 7f8fa5400700 1 -- 172.21.15.89:0/744459080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa00b8640 con 0x7f8fa00cf230 2024-04-02T09:14:44.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.882+0000 7f8fa5400700 1 -- 172.21.15.89:0/744459080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa0067720 con 0x7f8fa00cf230 2024-04-02T09:14:44.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.883+0000 7f8f96a00700 1 -- 172.21.15.89:0/744459080 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f8f9000cbb0 con 0x7f8fa00cf230 2024-04-02T09:14:44.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.884+0000 7f8f96a00700 1 --2- 172.21.15.89:0/744459080 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f8006aed0 0x7f8f8006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:44.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.884+0000 7f8f96a00700 1 -- 172.21.15.89:0/744459080 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f8f900899b0 con 0x7f8fa00cf230 2024-04-02T09:14:44.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.884+0000 7f8f9de00700 1 --2- 172.21.15.89:0/744459080 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f8006aed0 0x7f8f8006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:44.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.885+0000 7f8f9de00700 1 --2- 172.21.15.89:0/744459080 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f8006aed0 0x7f8f8006d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c014040 tx=0x7f8f8c008030 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:44.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:44.888+0000 7f8f96a00700 1 -- 172.21.15.89:0/744459080 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8f900545d0 con 0x7f8fa00cf230 2024-04-02T09:14:45.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.131+0000 7f8fa5400700 1 -- 172.21.15.89:0/744459080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fa006bdf0 con 0x7f8fa00cf230 2024-04-02T09:14:45.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.132+0000 7f8f96a00700 1 -- 172.21.15.89:0/744459080 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+4756 (secure 0 0 0) 0x7f8f9000ce60 con 0x7f8fa00cf230 2024-04-02T09:14:45.130 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:45.130 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":135,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10031,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1597421311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1597421311},{"type":"v1","addr":"172.21.15.113:6834","nonce":1597421311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":10034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3719211992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":3719211992},{"type":"v1","addr":"172.21.15.113:6835","nonce":3719211992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":10037,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3947526387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3947526387},{"type":"v1","addr":"172.21.15.89:6835","nonce":3947526387}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:14:34.351811+0000","modified":"2024-04-02T09:14:37.361983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12762":{"gid":12762,"name":"c","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2124958719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2124958719},{"type":"v1","addr":"172.21.15.89:6837","nonce":2124958719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-04-02T09:14:45.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.134+0000 7f8fa5400700 1 -- 172.21.15.89:0/744459080 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f8006aed0 msgr2=0x7f8f8006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.134+0000 7f8fa5400700 1 --2- 172.21.15.89:0/744459080 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f8006aed0 0x7f8f8006d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c014040 tx=0x7f8f8c008030 comp rx=0 tx=0).stop 2024-04-02T09:14:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.134+0000 7f8fa5400700 1 -- 172.21.15.89:0/744459080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00cf230 msgr2=0x7f8fa00b7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.134+0000 7f8fa5400700 1 --2- 172.21.15.89:0/744459080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00cf230 0x7f8fa00b7a80 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f8f9000e870 tx=0x7f8f9000ec30 comp rx=0 tx=0).stop 2024-04-02T09:14:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.134+0000 7f8fa5400700 1 -- 172.21.15.89:0/744459080 shutdown_connections 2024-04-02T09:14:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.134+0000 7f8fa5400700 1 --2- 172.21.15.89:0/744459080 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8f8006aed0 0x7f8f8006d390 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.134+0000 7f8fa5400700 1 --2- 172.21.15.89:0/744459080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fa0161590 0x7f8fa015f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.134+0000 7f8fa5400700 1 --2- 172.21.15.89:0/744459080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fa00cf230 0x7f8fa00b7a80 unknown :-1 s=CLOSED pgs=2830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.134+0000 7f8fa5400700 1 --2- 172.21.15.89:0/744459080 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8fa0160f10 0x7f8fa01f7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.134+0000 7f8fa5400700 1 -- 172.21.15.89:0/744459080 >> 172.21.15.89:0/744459080 conn(0x7f8fa008aff0 msgr2=0x7f8fa0160170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.134+0000 7f8fa5400700 1 -- 172.21.15.89:0/744459080 shutdown_connections 2024-04-02T09:14:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.134+0000 7f8fa5400700 1 -- 172.21.15.89:0/744459080 wait complete. 2024-04-02T09:14:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 135 2024-04-02T09:14:45.553 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 135, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T09:14:34.351811+0000', 'modified': '2024-04-02T09:14:37.361983+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12762}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12762': {'gid': 12762, 'name': 'c', 'rank': 0, 'incarnation': 131, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6837/2124958719', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6836', 'nonce': 2124958719}, {'type': 'v1', 'addr': '172.21.15.89:6837', 'nonce': 2124958719}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T09:14:45.553 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T09:14:45.553 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-02T09:14:45.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.750+0000 7f0b66400700 1 Processor -- start 2024-04-02T09:14:45.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.750+0000 7f0b66400700 1 -- start start 2024-04-02T09:14:45.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.752+0000 7f0b66400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b600d2540 0x7f0b601687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.752+0000 7f0b66400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b60161420 con 0x7f0b600d16a0 2024-04-02T09:14:45.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.752+0000 7f0b66400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b60161700 con 0x7f0b6015d9d0 2024-04-02T09:14:45.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.752+0000 7f0b66400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b600d4340 con 0x7f0b600d2540 2024-04-02T09:14:45.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.752+0000 7f0b5fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b600d2540 0x7f0b601687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.752+0000 7f0b5fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0b600d2540 0x7f0b601687b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35180/0 (socket says 172.21.15.89:35180) 2024-04-02T09:14:45.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.752+0000 7f0b5fe00700 1 -- 172.21.15.89:0/3367953665 learned_addr learned my addr 172.21.15.89:0/3367953665 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:45.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.752+0000 7f0b5ea00700 1 --1- 172.21.15.89:0/3367953665 >> v1:172.21.15.89:6789/0 conn(0x7f0b600d16a0 0x7f0b600d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38698/0 (socket says 172.21.15.89:38698) 2024-04-02T09:14:45.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.752+0000 7f0b5fe00700 1 -- 172.21.15.89:0/3367953665 >> v1:172.21.15.113:6789/0 conn(0x7f0b6015d9d0 legacy=0x7f0b600d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:45.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.752+0000 7f0b5fe00700 1 -- 172.21.15.89:0/3367953665 >> v1:172.21.15.89:6789/0 conn(0x7f0b600d16a0 legacy=0x7f0b600d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:45.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.752+0000 7f0b5fe00700 1 -- 172.21.15.89:0/3367953665 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b601619e0 con 0x7f0b600d2540 2024-04-02T09:14:45.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.753+0000 7f0b5fe00700 1 --2- 172.21.15.89:0/3367953665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b600d2540 0x7f0b601687b0 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7f0b58009e50 tx=0x7f0b5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8be62ff53ce4f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:45.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.753+0000 7f0b5e000700 1 -- 172.21.15.89:0/3367953665 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5800e070 con 0x7f0b600d2540 2024-04-02T09:14:45.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.753+0000 7f0b5e000700 1 -- 172.21.15.89:0/3367953665 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b58003a20 con 0x7f0b600d2540 2024-04-02T09:14:45.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.753+0000 7f0b5e000700 1 -- 172.21.15.89:0/3367953665 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58003dd0 con 0x7f0b600d2540 2024-04-02T09:14:45.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.753+0000 7f0b66400700 1 -- 172.21.15.89:0/3367953665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b600d2540 msgr2=0x7f0b601687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:45.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.753+0000 7f0b66400700 1 --2- 172.21.15.89:0/3367953665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b600d2540 0x7f0b601687b0 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7f0b58009e50 tx=0x7f0b5800b330 comp rx=0 tx=0).stop 2024-04-02T09:14:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.753+0000 7f0b66400700 1 -- 172.21.15.89:0/3367953665 shutdown_connections 2024-04-02T09:14:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.753+0000 7f0b66400700 1 --2- 172.21.15.89:0/3367953665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b600d2540 0x7f0b601687b0 unknown :-1 s=CLOSED pgs=2832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.753+0000 7f0b66400700 1 -- 172.21.15.89:0/3367953665 >> 172.21.15.89:0/3367953665 conn(0x7f0b6008aff0 msgr2=0x7f0b600d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.753+0000 7f0b66400700 1 -- 172.21.15.89:0/3367953665 shutdown_connections 2024-04-02T09:14:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.754+0000 7f0b66400700 1 -- 172.21.15.89:0/3367953665 wait complete. 2024-04-02T09:14:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.755+0000 7f0b66400700 1 Processor -- start 2024-04-02T09:14:45.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.755+0000 7f0b66400700 1 -- start start 2024-04-02T09:14:45.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.756+0000 7f0b66400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b600d16a0 0x7f0b6014f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.756+0000 7f0b66400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b600d2540 0x7f0b600cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.756+0000 7f0b66400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b6015d9d0 0x7f0b600cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.756+0000 7f0b66400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b601e7dc0 con 0x7f0b600d16a0 2024-04-02T09:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.756+0000 7f0b66400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b600cc450 con 0x7f0b600d2540 2024-04-02T09:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.756+0000 7f0b66400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b601e7420 con 0x7f0b6015d9d0 2024-04-02T09:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.756+0000 7f0b5f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b600d16a0 0x7f0b6014f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.756+0000 7f0b5f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b600d16a0 0x7f0b6014f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35190/0 (socket says 172.21.15.89:35190) 2024-04-02T09:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.756+0000 7f0b5f400700 1 -- 172.21.15.89:0/1817175262 learned_addr learned my addr 172.21.15.89:0/1817175262 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b5ea00700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b600d2540 0x7f0b600cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b5fe00700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b6015d9d0 0x7f0b600cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b5f400700 1 -- 172.21.15.89:0/1817175262 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b6015d9d0 msgr2=0x7f0b600cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b5f400700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b6015d9d0 0x7f0b600cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b5f400700 1 -- 172.21.15.89:0/1817175262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b600d2540 msgr2=0x7f0b600cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b5f400700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b600d2540 0x7f0b600cb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b5f400700 1 -- 172.21.15.89:0/1817175262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b58009b30 con 0x7f0b600d16a0 2024-04-02T09:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b5fe00700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b6015d9d0 0x7f0b600cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b5ea00700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b600d2540 0x7f0b600cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b5f400700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b600d16a0 0x7f0b6014f910 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000b4a0 tx=0x7f0b5000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b57400700 1 -- 172.21.15.89:0/1817175262 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50011830 con 0x7f0b600d16a0 2024-04-02T09:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b57400700 1 -- 172.21.15.89:0/1817175262 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b50011990 con 0x7f0b600d16a0 2024-04-02T09:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b57400700 1 -- 172.21.15.89:0/1817175262 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50011d40 con 0x7f0b600d16a0 2024-04-02T09:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b66400700 1 -- 172.21.15.89:0/1817175262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b601e7700 con 0x7f0b600d16a0 2024-04-02T09:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.757+0000 7f0b66400700 1 -- 172.21.15.89:0/1817175262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b6014fe50 con 0x7f0b600d16a0 2024-04-02T09:14:45.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.760+0000 7f0b57400700 1 -- 172.21.15.89:0/1817175262 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0b5001a400 con 0x7f0b600d16a0 2024-04-02T09:14:45.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.761+0000 7f0b57400700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b4006aed0 0x7f0b4006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.761+0000 7f0b57400700 1 -- 172.21.15.89:0/1817175262 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f0b50089660 con 0x7f0b600d16a0 2024-04-02T09:14:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.761+0000 7f0b5ea00700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b4006aed0 0x7f0b4006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.762+0000 7f0b5ea00700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b4006aed0 0x7f0b4006d390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0b600ca8b0 tx=0x7f0b4c009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:45.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.763+0000 7f0b79ec2b80 1 Processor -- start 2024-04-02T09:14:45.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.763+0000 7f0b79ec2b80 1 -- start start 2024-04-02T09:14:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b79ec2b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b8545cd830 0x55b8545cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b79ec2b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545a0be0 0x55b8545a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b79ec2b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b85459eb60 0x55b8545caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b79ec2b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b854518650 con 0x55b8545a0be0 2024-04-02T09:14:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b79ec2b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8544f9c30 con 0x55b8545cd830 2024-04-02T09:14:45.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b79ec2b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b8544f7430 con 0x55b85459eb60 2024-04-02T09:14:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b5ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545a0be0 0x55b8545a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b5ea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545a0be0 0x55b8545a3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35202/0 (socket says 172.21.15.89:35202) 2024-04-02T09:14:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b5ea00700 1 -- 172.21.15.89:0/2760772273 learned_addr learned my addr 172.21.15.89:0/2760772273 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b5f400700 1 --2- 172.21.15.89:0/2760772273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b8545cd830 0x55b8545cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b5fe00700 1 --2- 172.21.15.89:0/2760772273 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b85459eb60 0x55b8545caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b5ea00700 1 -- 172.21.15.89:0/2760772273 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b85459eb60 msgr2=0x55b8545caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b5ea00700 1 --2- 172.21.15.89:0/2760772273 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b85459eb60 0x55b8545caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b5ea00700 1 -- 172.21.15.89:0/2760772273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b8545cd830 msgr2=0x55b8545cdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b5ea00700 1 --2- 172.21.15.89:0/2760772273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b8545cd830 0x55b8545cdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b5ea00700 1 -- 172.21.15.89:0/2760772273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8542e6790 con 0x55b8545a0be0 2024-04-02T09:14:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.764+0000 7f0b5f400700 1 --2- 172.21.15.89:0/2760772273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b8545cd830 0x55b8545cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.765+0000 7f0b5ea00700 1 --2- 172.21.15.89:0/2760772273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545a0be0 0x55b8545a3a10 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c007300 tx=0x7f0b4c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c764c09231adbd38 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:45.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.765+0000 7f0b4be00700 1 -- 172.21.15.89:0/2760772273 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c0076c0 con 0x55b8545a0be0 2024-04-02T09:14:45.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.765+0000 7f0b4be00700 1 -- 172.21.15.89:0/2760772273 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4c021450 con 0x55b8545a0be0 2024-04-02T09:14:45.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.765+0000 7f0b4be00700 1 -- 172.21.15.89:0/2760772273 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c0217c0 con 0x55b8545a0be0 2024-04-02T09:14:45.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.765+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/2760772273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545a0be0 msgr2=0x55b8545a3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:45.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.765+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/2760772273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545a0be0 0x55b8545a3a10 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c007300 tx=0x7f0b4c011000 comp rx=0 tx=0).stop 2024-04-02T09:14:45.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.766+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/2760772273 shutdown_connections 2024-04-02T09:14:45.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.766+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/2760772273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b8545cd830 0x55b8545cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.766+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/2760772273 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b85459eb60 0x55b8545caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.766+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/2760772273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545a0be0 0x55b8545a3a10 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.766+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/2760772273 >> 172.21.15.89:0/2760772273 conn(0x55b8545016a0 msgr2=0x55b8545e8640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:45.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.766+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/2760772273 shutdown_connections 2024-04-02T09:14:45.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.766+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/2760772273 wait complete. 2024-04-02T09:14:45.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.766+0000 7f0b79ec2b80 1 -- start start 2024-04-02T09:14:45.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.767+0000 7f0b79ec2b80 1 Processor -- start 2024-04-02T09:14:45.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b79ec2b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b85459eb60 0x55b8545f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b79ec2b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545a0be0 0x55b8545cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b79ec2b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545cd830 0x55b8545efdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b79ec2b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8544f2d20 con 0x55b8545cd830 2024-04-02T09:14:45.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b79ec2b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b85448e770 con 0x55b85459eb60 2024-04-02T09:14:45.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b79ec2b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b8544b16f0 con 0x55b8545a0be0 2024-04-02T09:14:45.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b79ec2b80 10 client.0 _ll_register_callbacks cb 0x55b8544985e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T09:14:45.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b5fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545cd830 0x55b8545efdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b5fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545cd830 0x55b8545efdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35214/0 (socket says 172.21.15.89:35214) 2024-04-02T09:14:45.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b5fe00700 1 -- 172.21.15.89:0/2724737396 learned_addr learned my addr 172.21.15.89:0/2724737396 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:45.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b5ea00700 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545a0be0 0x55b8545cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b5fe00700 1 -- 172.21.15.89:0/2724737396 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545a0be0 msgr2=0x55b8545cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:45.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b5fe00700 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545a0be0 0x55b8545cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b5f400700 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b85459eb60 0x55b8545f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b5fe00700 1 -- 172.21.15.89:0/2724737396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b85459eb60 msgr2=0x55b8545f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:45.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b5fe00700 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b85459eb60 0x55b8545f7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b5fe00700 1 -- 172.21.15.89:0/2724737396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b4c00f040 con 0x55b8545cd830 2024-04-02T09:14:45.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b5ea00700 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545a0be0 0x55b8545cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:45.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.768+0000 7f0b5f400700 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b85459eb60 0x55b8545f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:45.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b5fe00700 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545cd830 0x55b8545efdb0 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7f0b58015040 tx=0x7f0b58004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f475f1db08eeb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:45.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b48c00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T09:14:45.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b48c00700 1 -- 172.21.15.89:0/2724737396 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58004440 con 0x55b8545cd830 2024-04-02T09:14:45.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b48c00700 1 -- 172.21.15.89:0/2724737396 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f0b580045a0 con 0x55b8545cd830 2024-04-02T09:14:45.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/2724737396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b854595920 con 0x55b8545cd830 2024-04-02T09:14:45.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b48c00700 1 -- 172.21.15.89:0/2724737396 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b5800be90 con 0x55b8545cd830 2024-04-02T09:14:45.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b79ec2b80 10 client.12930 fetch_fsmap learned FSMap version 135 2024-04-02T09:14:45.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/2724737396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=2+}) v3 -- 0x55b8543d30a0 con 0x55b8545cd830 2024-04-02T09:14:45.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b48c00700 1 -- 172.21.15.89:0/2724737396 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b854595920 con 0x55b8545cd830 2024-04-02T09:14:45.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b48c00700 1 -- 172.21.15.89:0/2724737396 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 135) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0b58023020 con 0x55b8545cd830 2024-04-02T09:14:45.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b79ec2b80 10 client.12930 fetch_fsmap finished waiting for FSMap version 135 2024-04-02T09:14:45.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b79ec2b80 10 client.12930 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T09:14:45.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b79ec2b80 20 client.12930 populate_metadata read hostname 'smithi089' 2024-04-02T09:14:45.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.89:6836/2124958719,v1:172.21.15.89:6837/2124958719] conn(0x55b8545d0c30 0x55b8545d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b79ec2b80 4 client.12930 mds_command: new command op to 12762 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T09:14:45.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.769+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/2724737396 --> [v2:172.21.15.89:6836/2124958719,v1:172.21.15.89:6837/2124958719] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b8545726f0 con 0x55b8545d0c30 2024-04-02T09:14:45.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.770+0000 7f0b5f400700 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.89:6836/2124958719,v1:172.21.15.89:6837/2124958719] conn(0x55b8545d0c30 0x55b8545d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.770+0000 7f0b5f400700 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.89:6836/2124958719,v1:172.21.15.89:6837/2124958719] conn(0x55b8545d0c30 0x55b8545d1010 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eb2626e47f0b93c server_cookie=c2ae2cf8675f3fde in_seq=0 out_seq=0 2024-04-02T09:14:45.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.770+0000 7f0b48c00700 10 client.12930 ms_handle_connect on v2:172.21.15.89:6836/2124958719 2024-04-02T09:14:45.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.771+0000 7f0b48c00700 1 -- 172.21.15.89:0/2724737396 <== mds.0 v2:172.21.15.89:6836/2124958719 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b8545726f0 con 0x55b8545d0c30 2024-04-02T09:14:45.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.771+0000 7f0b48c00700 10 client.12930 handle_command_reply: tid=0 2024-04-02T09:14:45.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.771+0000 7f0b79ec2b80 1 client.12930 shutdown 2024-04-02T09:14:45.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.771+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/2724737396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545cd830 msgr2=0x55b8545efdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:45.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.771+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545cd830 0x55b8545efdb0 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7f0b58015040 tx=0x7f0b58004b40 comp rx=0 tx=0).stop 2024-04-02T09:14:45.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.771+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/2724737396 shutdown_connections 2024-04-02T09:14:45.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.771+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.89:6836/2124958719,v1:172.21.15.89:6837/2124958719] conn(0x55b8545d0c30 0x55b8545d1010 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.771+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b85459eb60 0x55b8545f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.771+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545cd830 0x55b8545efdb0 unknown :-1 s=CLOSED pgs=2835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.771+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/2724737396 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545a0be0 0x55b8545cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.771+0000 7f0b48c00700 0 client.12930 ms_handle_reset on v2:172.21.15.89:6836/2124958719 2024-04-02T09:14:45.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.771+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/2724737396 >> 172.21.15.89:0/2724737396 conn(0x55b8545016a0 msgr2=0x55b8545a2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:45.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.771+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/2724737396 shutdown_connections 2024-04-02T09:14:45.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.771+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/2724737396 wait complete. 2024-04-02T09:14:45.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.772+0000 7f0b79ec2b80 20 client.12930 trim_cache size 0 max 16384 2024-04-02T09:14:45.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.790+0000 7f0b79ec2b80 1 Processor -- start 2024-04-02T09:14:45.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.790+0000 7f0b79ec2b80 1 -- start start 2024-04-02T09:14:45.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.790+0000 7f0b79ec2b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b85459eb60 0x55b8545cc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.790+0000 7f0b79ec2b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b8545a0be0 0x55b854648c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b79ec2b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545cd830 0x55b854649150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b79ec2b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8542e7390 con 0x55b85459eb60 2024-04-02T09:14:45.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b79ec2b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b854649690 con 0x55b8545a0be0 2024-04-02T09:14:45.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b79ec2b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b8545f7060 con 0x55b8545cd830 2024-04-02T09:14:45.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b5f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b85459eb60 0x55b8545cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b5f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b85459eb60 0x55b8545cc480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35230/0 (socket says 172.21.15.89:35230) 2024-04-02T09:14:45.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b5f400700 1 -- 172.21.15.89:0/3152084892 learned_addr learned my addr 172.21.15.89:0/3152084892 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:45.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b5ea00700 1 --2- 172.21.15.89:0/3152084892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b8545a0be0 0x55b854648c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b5fe00700 1 --2- 172.21.15.89:0/3152084892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545cd830 0x55b854649150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b5f400700 1 -- 172.21.15.89:0/3152084892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545cd830 msgr2=0x55b854649150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:45.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b5f400700 1 --2- 172.21.15.89:0/3152084892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545cd830 0x55b854649150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b5f400700 1 -- 172.21.15.89:0/3152084892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b8545a0be0 msgr2=0x55b854648c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:45.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b5f400700 1 --2- 172.21.15.89:0/3152084892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b8545a0be0 0x55b854648c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b5f400700 1 -- 172.21.15.89:0/3152084892 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8544b9010 con 0x55b85459eb60 2024-04-02T09:14:45.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b5fe00700 1 --2- 172.21.15.89:0/3152084892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545cd830 0x55b854649150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:14:45.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b5f400700 1 --2- 172.21.15.89:0/3152084892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b85459eb60 0x55b8545cc480 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f0b500544e0 tx=0x7f0b5000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6614155f48bf7b69 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:45.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.791+0000 7f0b54c00700 1 -- 172.21.15.89:0/3152084892 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50085070 con 0x55b85459eb60 2024-04-02T09:14:45.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.792+0000 7f0b54c00700 1 -- 172.21.15.89:0/3152084892 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b500603d0 con 0x55b85459eb60 2024-04-02T09:14:45.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.792+0000 7f0b54c00700 1 -- 172.21.15.89:0/3152084892 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5005e0b0 con 0x55b85459eb60 2024-04-02T09:14:45.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.792+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/3152084892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b85459eb60 msgr2=0x55b8545cc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:45.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.792+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/3152084892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b85459eb60 0x55b8545cc480 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f0b500544e0 tx=0x7f0b5000bf90 comp rx=0 tx=0).stop 2024-04-02T09:14:45.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.792+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/3152084892 shutdown_connections 2024-04-02T09:14:45.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.792+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/3152084892 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545cd830 0x55b854649150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.792+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/3152084892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b85459eb60 0x55b8545cc480 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.792+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/3152084892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b8545a0be0 0x55b854648c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.792+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/3152084892 >> 172.21.15.89:0/3152084892 conn(0x55b8545016a0 msgr2=0x55b85460b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:45.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.792+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/3152084892 shutdown_connections 2024-04-02T09:14:45.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.792+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/3152084892 wait complete. 2024-04-02T09:14:45.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.793+0000 7f0b79ec2b80 1 -- start start 2024-04-02T09:14:45.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.793+0000 7f0b79ec2b80 1 Processor -- start 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.794+0000 7f0b79ec2b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b85459eb60 0x55b8545e9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.794+0000 7f0b79ec2b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545a0be0 0x55b854644a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.794+0000 7f0b79ec2b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545cd830 0x55b8545ecc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.794+0000 7f0b79ec2b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8545c9310 con 0x55b8545cd830 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.794+0000 7f0b79ec2b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8545c9000 con 0x55b85459eb60 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.794+0000 7f0b79ec2b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b8545a2a60 con 0x55b8545a0be0 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.794+0000 7f0b79ec2b80 10 client.0 _ll_register_callbacks cb 0x7f0b58015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.794+0000 7f0b5fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545cd830 0x55b8545ecc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.794+0000 7f0b5ea00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545a0be0 0x55b854644a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.794+0000 7f0b5fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545cd830 0x55b8545ecc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35238/0 (socket says 172.21.15.89:35238) 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.794+0000 7f0b5fe00700 1 -- 172.21.15.89:0/1872873279 learned_addr learned my addr 172.21.15.89:0/1872873279 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.794+0000 7f0b5f400700 1 --2- 172.21.15.89:0/1872873279 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b85459eb60 0x55b8545e9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.795+0000 7f0b5fe00700 1 -- 172.21.15.89:0/1872873279 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545a0be0 msgr2=0x55b854644a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.795+0000 7f0b5fe00700 1 --2- 172.21.15.89:0/1872873279 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545a0be0 0x55b854644a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.795+0000 7f0b5fe00700 1 -- 172.21.15.89:0/1872873279 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b85459eb60 msgr2=0x55b8545e9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:45.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.795+0000 7f0b5fe00700 1 --2- 172.21.15.89:0/1872873279 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b85459eb60 0x55b8545e9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.795+0000 7f0b5fe00700 1 -- 172.21.15.89:0/1872873279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b6014fe50 con 0x55b8545cd830 2024-04-02T09:14:45.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.795+0000 7f0b5fe00700 1 --2- 172.21.15.89:0/1872873279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545cd830 0x55b8545ecc20 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f0b58015040 tx=0x7f0b58009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb1d3314e7984660 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:45.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.795+0000 7f0b48c00700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T09:14:45.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.795+0000 7f0b48c00700 1 -- 172.21.15.89:0/1872873279 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5800e040 con 0x55b8545cd830 2024-04-02T09:14:45.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.795+0000 7f0b48c00700 1 -- 172.21.15.89:0/1872873279 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f0b58032ca0 con 0x55b8545cd830 2024-04-02T09:14:45.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.795+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/1872873279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0b600ccf00 con 0x55b8545cd830 2024-04-02T09:14:45.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.795+0000 7f0b48c00700 1 -- 172.21.15.89:0/1872873279 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b580048d0 con 0x55b8545cd830 2024-04-02T09:14:45.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.795+0000 7f0b79ec2b80 10 client.12936 fetch_fsmap learned FSMap version 135 2024-04-02T09:14:45.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.795+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/1872873279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=2+}) v3 -- 0x55b8543471d0 con 0x55b8545cd830 2024-04-02T09:14:45.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.796+0000 7f0b48c00700 1 -- 172.21.15.89:0/1872873279 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b600ccf00 con 0x55b8545cd830 2024-04-02T09:14:45.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.796+0000 7f0b48c00700 1 -- 172.21.15.89:0/1872873279 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 135) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0b58023070 con 0x55b8545cd830 2024-04-02T09:14:45.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.796+0000 7f0b79ec2b80 10 client.12936 fetch_fsmap finished waiting for FSMap version 135 2024-04-02T09:14:45.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.796+0000 7f0b79ec2b80 10 client.12936 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T09:14:45.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.796+0000 7f0b79ec2b80 20 client.12936 populate_metadata read hostname 'smithi089' 2024-04-02T09:14:45.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.796+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/1872873279 >> [v2:172.21.15.89:6836/2124958719,v1:172.21.15.89:6837/2124958719] conn(0x55b8546489e0 0x55b854648dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:45.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.796+0000 7f0b79ec2b80 4 client.12936 mds_command: new command op to 12762 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T09:14:45.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.796+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/1872873279 --> [v2:172.21.15.89:6836/2124958719,v1:172.21.15.89:6837/2124958719] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f0b4c021450 con 0x55b8546489e0 2024-04-02T09:14:45.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.796+0000 7f0b5f400700 1 --2- 172.21.15.89:0/1872873279 >> [v2:172.21.15.89:6836/2124958719,v1:172.21.15.89:6837/2124958719] conn(0x55b8546489e0 0x55b854648dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:45.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b5f400700 1 --2- 172.21.15.89:0/1872873279 >> [v2:172.21.15.89:6836/2124958719,v1:172.21.15.89:6837/2124958719] conn(0x55b8546489e0 0x55b854648dc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a88973f90e3a0e9 server_cookie=27d31c1b0a689cf4 in_seq=0 out_seq=0 2024-04-02T09:14:45.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b48c00700 10 client.12936 ms_handle_connect on v2:172.21.15.89:6836/2124958719 2024-04-02T09:14:45.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b48c00700 1 -- 172.21.15.89:0/1872873279 <== mds.0 v2:172.21.15.89:6836/2124958719 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x7f0b4c021450 con 0x55b8546489e0 2024-04-02T09:14:45.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b48c00700 10 client.12936 handle_command_reply: tid=0 2024-04-02T09:14:45.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b79ec2b80 1 client.12936 shutdown 2024-04-02T09:14:45.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/1872873279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545cd830 msgr2=0x55b8545ecc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:45.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/1872873279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545cd830 0x55b8545ecc20 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f0b58015040 tx=0x7f0b58009810 comp rx=0 tx=0).stop 2024-04-02T09:14:45.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/1872873279 shutdown_connections 2024-04-02T09:14:45.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/1872873279 >> [v2:172.21.15.89:6836/2124958719,v1:172.21.15.89:6837/2124958719] conn(0x55b8546489e0 0x55b854648dc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/1872873279 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b85459eb60 0x55b8545e9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/1872873279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8545cd830 0x55b8545ecc20 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b48c00700 0 client.12936 ms_handle_reset on v2:172.21.15.89:6836/2124958719 2024-04-02T09:14:45.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b79ec2b80 1 --2- 172.21.15.89:0/1872873279 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55b8545a0be0 0x55b854644a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/1872873279 >> 172.21.15.89:0/1872873279 conn(0x55b8545016a0 msgr2=0x55b854599880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:45.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/1872873279 shutdown_connections 2024-04-02T09:14:45.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b79ec2b80 1 -- 172.21.15.89:0/1872873279 wait complete. 2024-04-02T09:14:45.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.797+0000 7f0b79ec2b80 20 client.12936 trim_cache size 0 max 16384 2024-04-02T09:14:45.803 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:14:45.803 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T09:14:45.803 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T09:14:45.803 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 12762, 2024-04-02T09:14:45.804 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T09:14:45.804 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T09:14:45.804 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T09:14:45.804 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 11.442999006000001, 2024-04-02T09:14:45.804 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 135, 2024-04-02T09:14:45.804 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 265, 2024-04-02T09:14:45.804 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 265, 2024-04-02T09:14:45.805 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 16.470901227999999 2024-04-02T09:14:45.805 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:14:45.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.798+0000 7f0b66400700 1 -- 172.21.15.89:0/1817175262 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b4006aed0 msgr2=0x7f0b4006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:45.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.798+0000 7f0b66400700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b4006aed0 0x7f0b4006d390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0b600ca8b0 tx=0x7f0b4c009040 comp rx=0 tx=0).stop 2024-04-02T09:14:45.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.798+0000 7f0b66400700 1 -- 172.21.15.89:0/1817175262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b600d16a0 msgr2=0x7f0b6014f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:45.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.798+0000 7f0b66400700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b600d16a0 0x7f0b6014f910 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000b4a0 tx=0x7f0b5000b860 comp rx=0 tx=0).stop 2024-04-02T09:14:45.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.798+0000 7f0b66400700 1 -- 172.21.15.89:0/1817175262 shutdown_connections 2024-04-02T09:14:45.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.798+0000 7f0b66400700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0b6015d9d0 0x7f0b600cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.798+0000 7f0b66400700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b600d16a0 0x7f0b6014f910 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.798+0000 7f0b66400700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f0b4006aed0 0x7f0b4006d390 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.798+0000 7f0b66400700 1 --2- 172.21.15.89:0/1817175262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b600d2540 0x7f0b600cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:45.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.798+0000 7f0b66400700 1 -- 172.21.15.89:0/1817175262 >> 172.21.15.89:0/1817175262 conn(0x7f0b6008aff0 msgr2=0x7f0b600b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:45.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.798+0000 7f0b66400700 1 -- 172.21.15.89:0/1817175262 shutdown_connections 2024-04-02T09:14:45.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:45.798+0000 7f0b66400700 1 -- 172.21.15.89:0/1817175262 wait complete. 2024-04-02T09:14:45.814 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T09:14:46.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.006+0000 7f3305400700 1 Processor -- start 2024-04-02T09:14:46.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.007+0000 7f3305400700 1 -- start start 2024-04-02T09:14:46.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.008+0000 7f3305400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f330015db40 0x7f33000d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:46.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.008+0000 7f3305400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33000d1330 con 0x7f330015d760 2024-04-02T09:14:46.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.008+0000 7f3305400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33000d1610 con 0x7f33000cf3c0 2024-04-02T09:14:46.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.008+0000 7f3305400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33000d1fb0 con 0x7f330015db40 2024-04-02T09:14:46.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.008+0000 7f32fe800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f330015d760 0x7f33000cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38708/0 (socket says 172.21.15.89:38708) 2024-04-02T09:14:46.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.008+0000 7f32fe800700 1 -- 172.21.15.89:0/3447524786 learned_addr learned my addr 172.21.15.89:0/3447524786 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:46.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.008+0000 7f32ff200700 1 --2- 172.21.15.89:0/3447524786 >> v2:172.21.15.113:3300/0 conn(0x7f330015db40 0x7f33000d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:46.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.008+0000 7f32fd400700 1 -- 172.21.15.89:0/3447524786 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778556690 0 0) 0x7f33000d1330 con 0x7f330015d760 2024-04-02T09:14:46.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.008+0000 7f32fd400700 1 -- 172.21.15.89:0/3447524786 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e40035b0 con 0x7f330015d760 2024-04-02T09:14:46.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fd400700 1 -- 172.21.15.89:0/3447524786 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 577665144 0 0) 0x7f32e40035b0 con 0x7f330015d760 2024-04-02T09:14:46.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fd400700 1 -- 172.21.15.89:0/3447524786 >> v2:172.21.15.113:3300/0 conn(0x7f330015db40 msgr2=0x7f33000d23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:46.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fd400700 1 --2- 172.21.15.89:0/3447524786 >> v2:172.21.15.113:3300/0 conn(0x7f330015db40 0x7f33000d23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fd400700 1 -- 172.21.15.89:0/3447524786 >> v1:172.21.15.113:6789/0 conn(0x7f33000cf3c0 legacy=0x7f33000cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fd400700 1 -- 172.21.15.89:0/3447524786 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33000d18f0 con 0x7f330015d760 2024-04-02T09:14:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fd400700 1 -- 172.21.15.89:0/3447524786 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f32f0001670 con 0x7f330015d760 2024-04-02T09:14:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fd400700 1 -- 172.21.15.89:0/3447524786 >> v1:172.21.15.89:6789/0 conn(0x7f330015d760 legacy=0x7f33000cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fd400700 1 --2- 172.21.15.89:0/3447524786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e4003a00 0x7f32e4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fd400700 1 --2- 172.21.15.89:0/3447524786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e4004850 0x7f32e4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:46.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fd400700 1 --2- 172.21.15.89:0/3447524786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32e4005210 0x7f32e400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:46.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fd400700 1 -- 172.21.15.89:0/3447524786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33000d1fb0 con 0x7f32e4003a00 2024-04-02T09:14:46.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fd400700 1 -- 172.21.15.89:0/3447524786 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e400c1b0 con 0x7f32e4004850 2024-04-02T09:14:46.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fd400700 1 -- 172.21.15.89:0/3447524786 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e400c370 con 0x7f32e4005210 2024-04-02T09:14:46.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fe800700 1 --2- 172.21.15.89:0/3447524786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e4003a00 0x7f32e4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:46.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fde00700 1 --2- 172.21.15.89:0/3447524786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e4004850 0x7f32e4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:46.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32ff200700 1 --2- 172.21.15.89:0/3447524786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32e4005210 0x7f32e400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:46.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fe800700 1 -- 172.21.15.89:0/3447524786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32e4005210 msgr2=0x7f32e400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:46.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fe800700 1 --2- 172.21.15.89:0/3447524786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32e4005210 0x7f32e400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:46.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fe800700 1 -- 172.21.15.89:0/3447524786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e4004850 msgr2=0x7f32e4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:46.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fe800700 1 --2- 172.21.15.89:0/3447524786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e4004850 0x7f32e4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:46.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32fe800700 1 -- 172.21.15.89:0/3447524786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e40035b0 con 0x7f32e4003a00 2024-04-02T09:14:46.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.009+0000 7f32ff200700 1 -- 172.21.15.89:0/3447524786 reap_dead start 2024-04-02T09:14:46.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.010+0000 7f32fe800700 1 --2- 172.21.15.89:0/3447524786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e4003a00 0x7f32e4003e20 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f32f000c240 tx=0x7f32e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f88d1cae0e950be server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:46.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.010+0000 7f32fd400700 1 -- 172.21.15.89:0/3447524786 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f0007c10 con 0x7f32e4003a00 2024-04-02T09:14:46.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.010+0000 7f32fd400700 1 -- 172.21.15.89:0/3447524786 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f000ce00 con 0x7f32e4003a00 2024-04-02T09:14:46.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.010+0000 7f32fd400700 1 -- 172.21.15.89:0/3447524786 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f0005ba0 con 0x7f32e4003a00 2024-04-02T09:14:46.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.010+0000 7f3305400700 1 -- 172.21.15.89:0/3447524786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e4003a00 msgr2=0x7f32e4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:46.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.010+0000 7f3305400700 1 --2- 172.21.15.89:0/3447524786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e4003a00 0x7f32e4003e20 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f32f000c240 tx=0x7f32e4005880 comp rx=0 tx=0).stop 2024-04-02T09:14:46.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.010+0000 7f3305400700 1 -- 172.21.15.89:0/3447524786 shutdown_connections 2024-04-02T09:14:46.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.010+0000 7f3305400700 1 --2- 172.21.15.89:0/3447524786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e4003a00 0x7f32e4003e20 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:46.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.010+0000 7f3305400700 1 -- 172.21.15.89:0/3447524786 >> 172.21.15.89:0/3447524786 conn(0x7f330008aff0 msgr2=0x7f33000d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:46.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.010+0000 7f3305400700 1 -- 172.21.15.89:0/3447524786 shutdown_connections 2024-04-02T09:14:46.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.011+0000 7f3305400700 1 -- 172.21.15.89:0/3447524786 wait complete. 2024-04-02T09:14:46.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.012+0000 7f3305400700 1 Processor -- start 2024-04-02T09:14:46.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.012+0000 7f3305400700 1 -- start start 2024-04-02T09:14:46.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.013+0000 7f3305400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e4003a00 0x7f33001f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:46.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.013+0000 7f3305400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33001f7880 0x7f33000c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:46.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.013+0000 7f3305400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33001f6b90 0x7f33001f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:46.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.013+0000 7f3305400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33001f7580 con 0x7f32e4003a00 2024-04-02T09:14:46.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.013+0000 7f3305400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33001f69a0 con 0x7f33001f6b90 2024-04-02T09:14:46.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.013+0000 7f3305400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33000c9560 con 0x7f33001f7880 2024-04-02T09:14:46.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.013+0000 7f32fe800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e4003a00 0x7f33001f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:46.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.014+0000 7f32fe800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e4003a00 0x7f33001f6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35246/0 (socket says 172.21.15.89:35246) 2024-04-02T09:14:46.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.014+0000 7f32fe800700 1 -- 172.21.15.89:0/1312558816 learned_addr learned my addr 172.21.15.89:0/1312558816 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:46.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.014+0000 7f32fde00700 1 --2- 172.21.15.89:0/1312558816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33001f7880 0x7f33000c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:46.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.014+0000 7f32ff200700 1 --2- 172.21.15.89:0/1312558816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33001f6b90 0x7f33001f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:46.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.014+0000 7f32fe800700 1 -- 172.21.15.89:0/1312558816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33001f7880 msgr2=0x7f33000c6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:46.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.014+0000 7f32fe800700 1 --2- 172.21.15.89:0/1312558816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33001f7880 0x7f33000c6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:46.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.014+0000 7f32fe800700 1 -- 172.21.15.89:0/1312558816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33001f6b90 msgr2=0x7f33001f7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:46.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.014+0000 7f32fe800700 1 --2- 172.21.15.89:0/1312558816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33001f6b90 0x7f33001f7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:46.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.014+0000 7f32fe800700 1 -- 172.21.15.89:0/1312558816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33000c9840 con 0x7f32e4003a00 2024-04-02T09:14:46.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.014+0000 7f32fe800700 1 --2- 172.21.15.89:0/1312558816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e4003a00 0x7f33001f6460 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f32e400c180 tx=0x7f32f0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:46.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.014+0000 7f32f6a00700 1 -- 172.21.15.89:0/1312558816 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f0004c80 con 0x7f32e4003a00 2024-04-02T09:14:46.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.014+0000 7f32f6a00700 1 -- 172.21.15.89:0/1312558816 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f0004de0 con 0x7f32e4003a00 2024-04-02T09:14:46.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.014+0000 7f32f6a00700 1 -- 172.21.15.89:0/1312558816 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f00072f0 con 0x7f32e4003a00 2024-04-02T09:14:46.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.015+0000 7f3305400700 1 -- 172.21.15.89:0/1312558816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33000c9b20 con 0x7f32e4003a00 2024-04-02T09:14:46.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.015+0000 7f3305400700 1 -- 172.21.15.89:0/1312558816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33000c7300 con 0x7f32e4003a00 2024-04-02T09:14:46.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.016+0000 7f3305400700 1 -- 172.21.15.89:0/1312558816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33000c7490 con 0x7f32e4003a00 2024-04-02T09:14:46.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.017+0000 7f32f6a00700 1 -- 172.21.15.89:0/1312558816 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f32f0007450 con 0x7f32e4003a00 2024-04-02T09:14:46.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.017+0000 7f32f6a00700 1 --2- 172.21.15.89:0/1312558816 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f32e006aed0 0x7f32e006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:46.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.018+0000 7f32f6a00700 1 -- 172.21.15.89:0/1312558816 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f32f008a240 con 0x7f32e4003a00 2024-04-02T09:14:46.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.018+0000 7f32fde00700 1 --2- 172.21.15.89:0/1312558816 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f32e006aed0 0x7f32e006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:46.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.018+0000 7f32fde00700 1 --2- 172.21.15.89:0/1312558816 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f32e006aed0 0x7f32e006d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f33000ceaf0 tx=0x7f32ec009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:46.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.022+0000 7f32f6a00700 1 -- 172.21.15.89:0/1312558816 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f33000c7490 con 0x7f32e4003a00 2024-04-02T09:14:46.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.270+0000 7f3305400700 1 -- 172.21.15.89:0/1312558816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f33000c7490 con 0x7f32e4003a00 2024-04-02T09:14:46.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.270+0000 7f32f6a00700 1 -- 172.21.15.89:0/1312558816 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v135) v1 ==== 53+0+73 (secure 0 0 0) 0x7f33000c7490 con 0x7f32e4003a00 2024-04-02T09:14:46.269 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T09:14:46.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.273+0000 7f3305400700 1 -- 172.21.15.89:0/1312558816 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f32e006aed0 msgr2=0x7f32e006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:46.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.273+0000 7f3305400700 1 --2- 172.21.15.89:0/1312558816 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f32e006aed0 0x7f32e006d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f33000ceaf0 tx=0x7f32ec009040 comp rx=0 tx=0).stop 2024-04-02T09:14:46.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.273+0000 7f3305400700 1 -- 172.21.15.89:0/1312558816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e4003a00 msgr2=0x7f33001f6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:46.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.273+0000 7f3305400700 1 --2- 172.21.15.89:0/1312558816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e4003a00 0x7f33001f6460 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f32e400c180 tx=0x7f32f0001a00 comp rx=0 tx=0).stop 2024-04-02T09:14:46.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.273+0000 7f3305400700 1 -- 172.21.15.89:0/1312558816 shutdown_connections 2024-04-02T09:14:46.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.273+0000 7f3305400700 1 --2- 172.21.15.89:0/1312558816 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f32e006aed0 0x7f32e006d390 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:46.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.273+0000 7f3305400700 1 --2- 172.21.15.89:0/1312558816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33001f6b90 0x7f33001f7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:46.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.273+0000 7f3305400700 1 --2- 172.21.15.89:0/1312558816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32e4003a00 0x7f33001f6460 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:46.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.273+0000 7f3305400700 1 --2- 172.21.15.89:0/1312558816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f33001f7880 0x7f33000c6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:46.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.273+0000 7f3305400700 1 -- 172.21.15.89:0/1312558816 >> 172.21.15.89:0/1312558816 conn(0x7f330008aff0 msgr2=0x7f33000b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:46.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.273+0000 7f3305400700 1 -- 172.21.15.89:0/1312558816 shutdown_connections 2024-04-02T09:14:46.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:46.273+0000 7f3305400700 1 -- 172.21.15.89:0/1312558816 wait complete. 2024-04-02T09:14:46.289 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T09:14:46.289 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T09:14:46.289 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T09:14:46.289 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T09:14:46.289 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T09:14:46.289 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T09:14:46.289 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T09:14:46.289 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T09:14:46.289 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:14:46.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:46.290 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T09:14:46.335 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T09:14:46.335 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T09:14:46.335 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T09:14:46.335 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:14:46.335 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T09:14:46.335 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:14:46.335 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:14:46.336 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:14:46.336 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T09:14:46.336 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:14:46.336 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:14:46.336 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T09:14:46.336 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:14:46.336 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:14:46.336 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T09:14:46.336 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:14:46.336 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T09:14:46.336 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 36542sec preferred_lft 36542sec 2024-04-02T09:14:46.336 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T09:14:46.336 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:14:46.336 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T09:14:46.337 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:14:46.337 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T09:14:46.337 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:14:46.337 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T09:14:46.337 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:14:46.337 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T09:14:46.337 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T09:14:46.337 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T09:14:46.337 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:14:46.337 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T09:14:46.337 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T09:14:46.337 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T09:14:46.337 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:14:46.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:46.338 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T09:14:46.439 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T09:14:46.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:46.442 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T09:14:46.487 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T09:14:46.487 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T09:14:46.487 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T09:14:46.487 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T09:14:46.488 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T09:14:46.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:46.489 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T09:14:46.489 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T09:14:46.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T09:14:46.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T09:14:46.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T09:14:46.489 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T09:14:46.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:46.689 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T09:14:46.734 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T09:14:46.734 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T09:14:46.734 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:14:46.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:46.734 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T09:14:46.841 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T09:14:46.842 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T09:14:46.886 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:14:46.886 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T09:14:46.887 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T09:14:46.970 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T09:14:46.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:46.970 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:14:47.110 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:14:47.110 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T09:14:47.137 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:14:47.137 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T09:14:47.137 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T09:14:47.137 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T09:14:47.137 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T09:14:47.143 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T09:14:47.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:47.143 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T09:14:47.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:47.209 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T09:14:47.309 INFO:teuthology.orchestra.run.smithi089.stdout:client12945 2024-04-02T09:14:47.309 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:14:47.309 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:14:47.353 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T09:14:47.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.589+0000 7f8af2600700 1 Processor -- start 2024-04-02T09:14:47.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.589+0000 7f8af2600700 1 -- start start 2024-04-02T09:14:47.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.590+0000 7f8af2600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8aec160e50 0x7f8aec0b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:47.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.590+0000 7f8af2600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8aec0b3f80 0x7f8aec0b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:47.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.590+0000 7f8af2600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aec0cef00 con 0x7f8aec0d3f20 2024-04-02T09:14:47.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.590+0000 7f8af2600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aec161230 con 0x7f8aec160e50 2024-04-02T09:14:47.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.590+0000 7f8af2600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0d2a80 con 0x7f8aec0b3f80 2024-04-02T09:14:47.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.590+0000 7f8aeb400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8aec160e50 0x7f8aec0b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:47.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.590+0000 7f8aeb400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8aec160e50 0x7f8aec0b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35254/0 (socket says 172.21.15.89:35254) 2024-04-02T09:14:47.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.590+0000 7f8aeb400700 1 -- 172.21.15.89:0/3337220323 learned_addr learned my addr 172.21.15.89:0/3337220323 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:47.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.591+0000 7f8aebe00700 1 --2- 172.21.15.89:0/3337220323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8aec0b3f80 0x7f8aec0b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:47.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.591+0000 7f8aeb400700 1 -- 172.21.15.89:0/3337220323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8aec0b3f80 msgr2=0x7f8aec0b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:47.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.591+0000 7f8aeb400700 1 --2- 172.21.15.89:0/3337220323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8aec0b3f80 0x7f8aec0b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:47.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.591+0000 7f8aeb400700 1 -- 172.21.15.89:0/3337220323 >> v1:172.21.15.113:6789/0 conn(0x7f8aec0d3f20 legacy=0x7f8aec0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:47.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.591+0000 7f8aeb400700 1 -- 172.21.15.89:0/3337220323 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec0d43b0 con 0x7f8aec160e50 2024-04-02T09:14:47.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.591+0000 7f8aeb400700 1 --2- 172.21.15.89:0/3337220323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aec160e50 0x7f8aec0b4e90 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7f8adc0098f0 tx=0x7f8adc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df725cb11c811580 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:47.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.591+0000 7f8aea000700 1 -- 172.21.15.89:0/3337220323 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc011830 con 0x7f8aec160e50 2024-04-02T09:14:47.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.591+0000 7f8aea000700 1 -- 172.21.15.89:0/3337220323 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc011990 con 0x7f8aec160e50 2024-04-02T09:14:47.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.591+0000 7f8aea000700 1 -- 172.21.15.89:0/3337220323 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc011d20 con 0x7f8aec160e50 2024-04-02T09:14:47.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.592+0000 7f8af2600700 1 -- 172.21.15.89:0/3337220323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aec160e50 msgr2=0x7f8aec0b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:47.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.592+0000 7f8af2600700 1 --2- 172.21.15.89:0/3337220323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aec160e50 0x7f8aec0b4e90 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7f8adc0098f0 tx=0x7f8adc009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:47.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.592+0000 7f8af2600700 1 -- 172.21.15.89:0/3337220323 shutdown_connections 2024-04-02T09:14:47.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.592+0000 7f8af2600700 1 --2- 172.21.15.89:0/3337220323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8aec0b3f80 0x7f8aec0b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:47.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.592+0000 7f8af2600700 1 --2- 172.21.15.89:0/3337220323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aec160e50 0x7f8aec0b4e90 unknown :-1 s=CLOSED pgs=2842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:47.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.592+0000 7f8af2600700 1 -- 172.21.15.89:0/3337220323 >> 172.21.15.89:0/3337220323 conn(0x7f8aec08aff0 msgr2=0x7f8aec0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:47.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.592+0000 7f8af2600700 1 -- 172.21.15.89:0/3337220323 shutdown_connections 2024-04-02T09:14:47.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.592+0000 7f8af2600700 1 -- 172.21.15.89:0/3337220323 wait complete. 2024-04-02T09:14:47.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.594+0000 7f8af2600700 1 Processor -- start 2024-04-02T09:14:47.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.594+0000 7f8af2600700 1 -- start start 2024-04-02T09:14:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8af2600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8aec0b3f80 0x7f8aec0b6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8af2600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8aec160e50 0x7f8aec0b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8af2600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aec157a10 0x7f8aec0b07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8af2600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0b91b0 con 0x7f8aec157a10 2024-04-02T09:14:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8af2600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0b8110 con 0x7f8aec0b3f80 2024-04-02T09:14:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8af2600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0b83f0 con 0x7f8aec160e50 2024-04-02T09:14:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8aebe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aec157a10 0x7f8aec0b07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8aebe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aec157a10 0x7f8aec0b07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35258/0 (socket says 172.21.15.89:35258) 2024-04-02T09:14:47.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8aebe00700 1 -- 172.21.15.89:0/4048043639 learned_addr learned my addr 172.21.15.89:0/4048043639 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8aeb400700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8aec0b3f80 0x7f8aec0b6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8aeaa00700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8aec160e50 0x7f8aec0b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8aebe00700 1 -- 172.21.15.89:0/4048043639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8aec160e50 msgr2=0x7f8aec0b78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8aebe00700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8aec160e50 0x7f8aec0b78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8aebe00700 1 -- 172.21.15.89:0/4048043639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8aec0b3f80 msgr2=0x7f8aec0b6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.595+0000 7f8aebe00700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8aec0b3f80 0x7f8aec0b6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.596+0000 7f8aebe00700 1 -- 172.21.15.89:0/4048043639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc009550 con 0x7f8aec157a10 2024-04-02T09:14:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.596+0000 7f8aeaa00700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8aec160e50 0x7f8aec0b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:14:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.596+0000 7f8aebe00700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aec157a10 0x7f8aec0b07a0 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400efc0 tx=0x7f8ae400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.596+0000 7f8ae3400700 1 -- 172.21.15.89:0/4048043639 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae400cd40 con 0x7f8aec157a10 2024-04-02T09:14:47.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.596+0000 7f8ae3400700 1 -- 172.21.15.89:0/4048043639 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae400f040 con 0x7f8aec157a10 2024-04-02T09:14:47.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.596+0000 7f8ae3400700 1 -- 172.21.15.89:0/4048043639 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae40146c0 con 0x7f8aec157a10 2024-04-02T09:14:47.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.596+0000 7f8af2600700 1 -- 172.21.15.89:0/4048043639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aec0b86d0 con 0x7f8aec157a10 2024-04-02T09:14:47.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.596+0000 7f8af2600700 1 -- 172.21.15.89:0/4048043639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aec0b8c90 con 0x7f8aec157a10 2024-04-02T09:14:47.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.597+0000 7f8af2600700 1 -- 172.21.15.89:0/4048043639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aec0b8eb0 con 0x7f8aec157a10 2024-04-02T09:14:47.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.598+0000 7f8ae3400700 1 -- 172.21.15.89:0/4048043639 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f8ae4014820 con 0x7f8aec157a10 2024-04-02T09:14:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.599+0000 7f8ae3400700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8acc06aed0 0x7f8acc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.599+0000 7f8ae3400700 1 -- 172.21.15.89:0/4048043639 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f8ae408a660 con 0x7f8aec157a10 2024-04-02T09:14:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.599+0000 7f8aeb400700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8acc06aed0 0x7f8acc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:47.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.600+0000 7f8aeb400700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8acc06aed0 0x7f8acc06d390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8adc014040 tx=0x7f8adc008030 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:47.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.603+0000 7f8ae3400700 1 -- 172.21.15.89:0/4048043639 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ae4055200 con 0x7f8aec157a10 2024-04-02T09:14:47.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.832+0000 7f8af2600700 1 -- 172.21.15.89:0/4048043639 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8aec0d27a0 con 0x7f8acc06aed0 2024-04-02T09:14:47.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.834+0000 7f8ae3400700 1 -- 172.21.15.89:0/4048043639 <== mgr.11320 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f8adc011b80 con 0x7f8acc06aed0 2024-04-02T09:14:47.832 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T09:14:47.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:14:47.832 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T09:14:47.832 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:14:47.832 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T09:14:47.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.836+0000 7f8af2600700 1 -- 172.21.15.89:0/4048043639 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8acc06aed0 msgr2=0x7f8acc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:47.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.836+0000 7f8af2600700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8acc06aed0 0x7f8acc06d390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8adc014040 tx=0x7f8adc008030 comp rx=0 tx=0).stop 2024-04-02T09:14:47.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.836+0000 7f8af2600700 1 -- 172.21.15.89:0/4048043639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aec157a10 msgr2=0x7f8aec0b07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:47.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.836+0000 7f8af2600700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aec157a10 0x7f8aec0b07a0 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400efc0 tx=0x7f8ae400c3a0 comp rx=0 tx=0).stop 2024-04-02T09:14:47.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.836+0000 7f8af2600700 1 -- 172.21.15.89:0/4048043639 shutdown_connections 2024-04-02T09:14:47.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.836+0000 7f8af2600700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8acc06aed0 0x7f8acc06d390 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:47.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.836+0000 7f8af2600700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8aec0b3f80 0x7f8aec0b6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:47.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.836+0000 7f8af2600700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8aec157a10 0x7f8aec0b07a0 unknown :-1 s=CLOSED pgs=2843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:47.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.836+0000 7f8af2600700 1 --2- 172.21.15.89:0/4048043639 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8aec160e50 0x7f8aec0b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:47.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.836+0000 7f8af2600700 1 -- 172.21.15.89:0/4048043639 >> 172.21.15.89:0/4048043639 conn(0x7f8aec08aff0 msgr2=0x7f8aec0d0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:47.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.836+0000 7f8af2600700 1 -- 172.21.15.89:0/4048043639 shutdown_connections 2024-04-02T09:14:47.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:47.836+0000 7f8af2600700 1 -- 172.21.15.89:0/4048043639 wait complete. 2024-04-02T09:14:47.853 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T09:14:48.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.046+0000 7f3ecf800700 1 Processor -- start 2024-04-02T09:14:48.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.046+0000 7f3ecf800700 1 -- start start 2024-04-02T09:14:48.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.047+0000 7f3ecf800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ec80d21f0 0x7f3ec80ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:48.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.047+0000 7f3ecf800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec80ce270 con 0x7f3ec80cec40 2024-04-02T09:14:48.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.047+0000 7f3ecf800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec815b600 con 0x7f3ec81611b0 2024-04-02T09:14:48.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.047+0000 7f3ecf800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80cf0b0 con 0x7f3ec80d21f0 2024-04-02T09:14:48.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.048+0000 7f3ecd800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3ec80cec40 0x7f3ec8161590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38716/0 (socket says 172.21.15.89:38716) 2024-04-02T09:14:48.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.048+0000 7f3ecd800700 1 -- 172.21.15.89:0/2099891711 learned_addr learned my addr 172.21.15.89:0/2099891711 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:48.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.048+0000 7f3ec7e00700 1 --2- 172.21.15.89:0/2099891711 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ec80d21f0 0x7f3ec80ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:48.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.048+0000 7f3ec7400700 1 -- 172.21.15.89:0/2099891711 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261459507 0 0) 0x7f3ec80ce270 con 0x7f3ec80cec40 2024-04-02T09:14:48.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.048+0000 7f3ec7400700 1 -- 172.21.15.89:0/2099891711 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb00035b0 con 0x7f3ec80cec40 2024-04-02T09:14:48.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.048+0000 7f3ec7e00700 1 -- 172.21.15.89:0/2099891711 >> v1:172.21.15.113:6789/0 conn(0x7f3ec81611b0 legacy=0x7f3ec815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:48.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.048+0000 7f3ec7400700 1 -- 172.21.15.89:0/2099891711 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678399025 0 0) 0x7f3ec815b600 con 0x7f3ec81611b0 2024-04-02T09:14:48.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.048+0000 7f3ec7e00700 1 -- 172.21.15.89:0/2099891711 >> v1:172.21.15.89:6789/0 conn(0x7f3ec80cec40 legacy=0x7f3ec8161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:48.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.048+0000 7f3ec7e00700 1 -- 172.21.15.89:0/2099891711 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec80d2760 con 0x7f3ec80d21f0 2024-04-02T09:14:48.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.049+0000 7f3ec7e00700 1 --2- 172.21.15.89:0/2099891711 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ec80d21f0 0x7f3ec80ce700 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4009870 tx=0x7f3eb4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da1bc8a943ba2db6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:48.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.049+0000 7f3ec7400700 1 -- 172.21.15.89:0/2099891711 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4011830 con 0x7f3ec80d21f0 2024-04-02T09:14:48.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.049+0000 7f3ec7400700 1 -- 172.21.15.89:0/2099891711 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb4011990 con 0x7f3ec80d21f0 2024-04-02T09:14:48.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.049+0000 7f3ec7400700 1 -- 172.21.15.89:0/2099891711 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4011d20 con 0x7f3ec80d21f0 2024-04-02T09:14:48.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.049+0000 7f3ecf800700 1 -- 172.21.15.89:0/2099891711 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ec80d21f0 msgr2=0x7f3ec80ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:48.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.049+0000 7f3ecf800700 1 --2- 172.21.15.89:0/2099891711 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ec80d21f0 0x7f3ec80ce700 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4009870 tx=0x7f3eb4009b80 comp rx=0 tx=0).stop 2024-04-02T09:14:48.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.049+0000 7f3ecf800700 1 -- 172.21.15.89:0/2099891711 shutdown_connections 2024-04-02T09:14:48.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.049+0000 7f3ecf800700 1 --2- 172.21.15.89:0/2099891711 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ec80d21f0 0x7f3ec80ce700 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:48.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.049+0000 7f3ecf800700 1 -- 172.21.15.89:0/2099891711 >> 172.21.15.89:0/2099891711 conn(0x7f3ec808aff0 msgr2=0x7f3ec80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:48.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.049+0000 7f3ecf800700 1 -- 172.21.15.89:0/2099891711 shutdown_connections 2024-04-02T09:14:48.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.050+0000 7f3ecf800700 1 -- 172.21.15.89:0/2099891711 wait complete. 2024-04-02T09:14:48.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.051+0000 7f3ecf800700 1 Processor -- start 2024-04-02T09:14:48.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.051+0000 7f3ecf800700 1 -- start start 2024-04-02T09:14:48.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.052+0000 7f3ecf800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ec80cec40 0x7f3ec80c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:48.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.052+0000 7f3ecf800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ec80d21f0 0x7f3ec806c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:48.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.052+0000 7f3ecf800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ec81611b0 0x7f3ec8156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:48.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.052+0000 7f3ecf800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec8154f80 con 0x7f3ec80d21f0 2024-04-02T09:14:48.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.052+0000 7f3ecf800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec8155260 con 0x7f3ec81611b0 2024-04-02T09:14:48.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.052+0000 7f3ecf800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec8155540 con 0x7f3ec80cec40 2024-04-02T09:14:48.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.052+0000 7f3ec7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ec80d21f0 0x7f3ec806c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:48.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.052+0000 7f3ec7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ec80d21f0 0x7f3ec806c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35270/0 (socket says 172.21.15.89:35270) 2024-04-02T09:14:48.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.052+0000 7f3ec7e00700 1 -- 172.21.15.89:0/291803353 learned_addr learned my addr 172.21.15.89:0/291803353 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:48.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.052+0000 7f3ecce00700 1 --2- 172.21.15.89:0/291803353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ec80cec40 0x7f3ec80c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:48.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.053+0000 7f3ecd800700 1 --2- 172.21.15.89:0/291803353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ec81611b0 0x7f3ec8156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:48.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.053+0000 7f3ec7e00700 1 -- 172.21.15.89:0/291803353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ec80cec40 msgr2=0x7f3ec80c7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:48.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.053+0000 7f3ec7e00700 1 --2- 172.21.15.89:0/291803353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ec80cec40 0x7f3ec80c7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:48.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.053+0000 7f3ec7e00700 1 -- 172.21.15.89:0/291803353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ec81611b0 msgr2=0x7f3ec8156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:48.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.053+0000 7f3ec7e00700 1 --2- 172.21.15.89:0/291803353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ec81611b0 0x7f3ec8156300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:48.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.053+0000 7f3ec7e00700 1 -- 172.21.15.89:0/291803353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb40094a0 con 0x7f3ec80d21f0 2024-04-02T09:14:48.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.053+0000 7f3ec7e00700 1 --2- 172.21.15.89:0/291803353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ec80d21f0 0x7f3ec806c380 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7f3eb40090b0 tx=0x7f3eb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:48.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.053+0000 7f3ec4c00700 1 -- 172.21.15.89:0/291803353 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb401a7e0 con 0x7f3ec80d21f0 2024-04-02T09:14:48.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.053+0000 7f3ec4c00700 1 -- 172.21.15.89:0/291803353 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb401a940 con 0x7f3ec80d21f0 2024-04-02T09:14:48.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.053+0000 7f3ec4c00700 1 -- 172.21.15.89:0/291803353 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb401acd0 con 0x7f3ec80d21f0 2024-04-02T09:14:48.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.053+0000 7f3ecf800700 1 -- 172.21.15.89:0/291803353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec8155820 con 0x7f3ec80d21f0 2024-04-02T09:14:48.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.054+0000 7f3ecf800700 1 -- 172.21.15.89:0/291803353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec8155de0 con 0x7f3ec80d21f0 2024-04-02T09:14:48.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.055+0000 7f3ecf800700 1 -- 172.21.15.89:0/291803353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec8067720 con 0x7f3ec80d21f0 2024-04-02T09:14:48.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.056+0000 7f3ec4c00700 1 -- 172.21.15.89:0/291803353 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 167) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f3eb4022070 con 0x7f3ec80d21f0 2024-04-02T09:14:48.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.056+0000 7f3ec4c00700 1 --2- 172.21.15.89:0/291803353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ea806aed0 0x7f3ea806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:48.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.056+0000 7f3ec4c00700 1 -- 172.21.15.89:0/291803353 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f3eb408a5a0 con 0x7f3ec80d21f0 2024-04-02T09:14:48.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.057+0000 7f3ecce00700 1 --2- 172.21.15.89:0/291803353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ea806aed0 0x7f3ea806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:48.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.057+0000 7f3ecce00700 1 --2- 172.21.15.89:0/291803353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ea806aed0 0x7f3ea806d390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc000f60 tx=0x7f3ebc009040 comp rx=0 tx=0).ready entity=mgr.11320 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:48.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.061+0000 7f3ec4c00700 1 -- 172.21.15.89:0/291803353 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3eb40552f0 con 0x7f3ec80d21f0 2024-04-02T09:14:48.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.329+0000 7f3ecf800700 1 -- 172.21.15.89:0/291803353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f3ec80b5d00 con 0x7f3ec80d21f0 2024-04-02T09:14:48.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.679+0000 7f3ec4c00700 1 -- 172.21.15.89:0/291803353 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 168) v1 ==== 89358+0+0 (secure 0 0 0) 0x7f3eb40556c0 con 0x7f3ec80d21f0 2024-04-02T09:14:48.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.680+0000 7f3ec4c00700 1 -- 172.21.15.89:0/291803353 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v168) v1 ==== 92+0+0 (secure 0 0 0) 0x7f3eb40552f0 con 0x7f3ec80d21f0 2024-04-02T09:14:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.683+0000 7f3ecf800700 1 -- 172.21.15.89:0/291803353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ea806aed0 msgr2=0x7f3ea806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.683+0000 7f3ecf800700 1 --2- 172.21.15.89:0/291803353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ea806aed0 0x7f3ea806d390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc000f60 tx=0x7f3ebc009040 comp rx=0 tx=0).stop 2024-04-02T09:14:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.683+0000 7f3ecf800700 1 -- 172.21.15.89:0/291803353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ec80d21f0 msgr2=0x7f3ec806c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.683+0000 7f3ecf800700 1 --2- 172.21.15.89:0/291803353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ec80d21f0 0x7f3ec806c380 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7f3eb40090b0 tx=0x7f3eb4004750 comp rx=0 tx=0).stop 2024-04-02T09:14:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.683+0000 7f3ecf800700 1 -- 172.21.15.89:0/291803353 shutdown_connections 2024-04-02T09:14:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.683+0000 7f3ecf800700 1 --2- 172.21.15.89:0/291803353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ea806aed0 0x7f3ea806d390 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.683+0000 7f3ecf800700 1 --2- 172.21.15.89:0/291803353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ec81611b0 0x7f3ec8156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.683+0000 7f3ecf800700 1 --2- 172.21.15.89:0/291803353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ec80cec40 0x7f3ec80c7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.683+0000 7f3ecf800700 1 --2- 172.21.15.89:0/291803353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ec80d21f0 0x7f3ec806c380 unknown :-1 s=CLOSED pgs=2845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.683+0000 7f3ecf800700 1 -- 172.21.15.89:0/291803353 >> 172.21.15.89:0/291803353 conn(0x7f3ec808aff0 msgr2=0x7f3ec80c9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.683+0000 7f3ecf800700 1 -- 172.21.15.89:0/291803353 shutdown_connections 2024-04-02T09:14:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.683+0000 7f3ecf800700 1 -- 172.21.15.89:0/291803353 wait complete. 2024-04-02T09:14:48.702 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T09:14:48.893 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:48.895+0000 7fd99d702240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:14:48.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.906+0000 7fc455c00700 1 Processor -- start 2024-04-02T09:14:48.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.906+0000 7fc455c00700 1 -- start start 2024-04-02T09:14:48.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.907+0000 7fc455c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc4500cee10 0x7fc450161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:48.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.907+0000 7fc455c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4500ce270 con 0x7fc4501611b0 2024-04-02T09:14:48.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.907+0000 7fc455c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45015b600 con 0x7fc4500d21f0 2024-04-02T09:14:48.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.907+0000 7fc455c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4500d2760 con 0x7fc4500cee10 2024-04-02T09:14:48.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.907+0000 7fc44ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc4501611b0 0x7fc45015ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39032/0 (socket says 172.21.15.89:39032) 2024-04-02T09:14:48.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.907+0000 7fc44ea00700 1 -- 172.21.15.89:0/4292538884 learned_addr learned my addr 172.21.15.89:0/4292538884 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:48.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.907+0000 7fc44f400700 1 --2- 172.21.15.89:0/4292538884 >> v2:172.21.15.113:3300/0 conn(0x7fc4500cee10 0x7fc450161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:48.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.908+0000 7fc44e000700 1 -- 172.21.15.89:0/4292538884 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908201835 0 0) 0x7fc45015b600 con 0x7fc4500d21f0 2024-04-02T09:14:48.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.908+0000 7fc44e000700 1 -- 172.21.15.89:0/4292538884 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4340035b0 con 0x7fc4500d21f0 2024-04-02T09:14:48.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.908+0000 7fc44e000700 1 -- 172.21.15.89:0/4292538884 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352550168 0 0) 0x7fc4500ce270 con 0x7fc4501611b0 2024-04-02T09:14:48.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.908+0000 7fc44e000700 1 -- 172.21.15.89:0/4292538884 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc45015b600 con 0x7fc4501611b0 2024-04-02T09:14:48.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.908+0000 7fc44f400700 1 -- 172.21.15.89:0/4292538884 >> v1:172.21.15.113:6789/0 conn(0x7fc4500d21f0 legacy=0x7fc4500ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:48.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.908+0000 7fc44f400700 1 -- 172.21.15.89:0/4292538884 >> v1:172.21.15.89:6789/0 conn(0x7fc4501611b0 legacy=0x7fc45015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:48.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.908+0000 7fc44f400700 1 -- 172.21.15.89:0/4292538884 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4500d2a40 con 0x7fc4500cee10 2024-04-02T09:14:48.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.908+0000 7fc44f400700 1 --2- 172.21.15.89:0/4292538884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4500cee10 0x7fc450161590 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fc448009d90 tx=0x7fc44800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80857b3cf4a7e45c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:48.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.908+0000 7fc44e000700 1 -- 172.21.15.89:0/4292538884 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44800e070 con 0x7fc4500cee10 2024-04-02T09:14:48.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.908+0000 7fc44e000700 1 -- 172.21.15.89:0/4292538884 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44800bb20 con 0x7fc4500cee10 2024-04-02T09:14:48.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.908+0000 7fc44e000700 1 -- 172.21.15.89:0/4292538884 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448012430 con 0x7fc4500cee10 2024-04-02T09:14:48.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.909+0000 7fc455c00700 1 -- 172.21.15.89:0/4292538884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4500cee10 msgr2=0x7fc450161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:48.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.909+0000 7fc455c00700 1 --2- 172.21.15.89:0/4292538884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4500cee10 0x7fc450161590 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fc448009d90 tx=0x7fc44800b330 comp rx=0 tx=0).stop 2024-04-02T09:14:48.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.909+0000 7fc455c00700 1 -- 172.21.15.89:0/4292538884 shutdown_connections 2024-04-02T09:14:48.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.909+0000 7fc455c00700 1 --2- 172.21.15.89:0/4292538884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4500cee10 0x7fc450161590 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:48.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.909+0000 7fc455c00700 1 -- 172.21.15.89:0/4292538884 >> 172.21.15.89:0/4292538884 conn(0x7fc45008aff0 msgr2=0x7fc4500d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:48.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.909+0000 7fc455c00700 1 -- 172.21.15.89:0/4292538884 shutdown_connections 2024-04-02T09:14:48.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.909+0000 7fc455c00700 1 -- 172.21.15.89:0/4292538884 wait complete. 2024-04-02T09:14:48.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.911+0000 7fc455c00700 1 Processor -- start 2024-04-02T09:14:48.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.911+0000 7fc455c00700 1 -- start start 2024-04-02T09:14:48.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.912+0000 7fc455c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4500cee10 0x7fc4500b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:48.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.912+0000 7fc455c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4500d21f0 0x7fc450154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:48.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.912+0000 7fc455c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4501611b0 0x7fc4500ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:48.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.912+0000 7fc455c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4500c7850 con 0x7fc4500d21f0 2024-04-02T09:14:48.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.912+0000 7fc455c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4500ccbf0 con 0x7fc4501611b0 2024-04-02T09:14:48.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.912+0000 7fc455c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4500cc910 con 0x7fc4500cee10 2024-04-02T09:14:48.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.912+0000 7fc447e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4500d21f0 0x7fc450154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:48.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.912+0000 7fc447e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4500d21f0 0x7fc450154ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42712/0 (socket says 172.21.15.89:42712) 2024-04-02T09:14:48.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.912+0000 7fc447e00700 1 -- 172.21.15.89:0/2112319159 learned_addr learned my addr 172.21.15.89:0/2112319159 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:48.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.912+0000 7fc44f400700 1 --2- 172.21.15.89:0/2112319159 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4501611b0 0x7fc4500ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:48.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.912+0000 7fc44ea00700 1 --2- 172.21.15.89:0/2112319159 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4500cee10 0x7fc4500b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:48.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.913+0000 7fc447e00700 1 -- 172.21.15.89:0/2112319159 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4500cee10 msgr2=0x7fc4500b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:48.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.913+0000 7fc447e00700 1 --2- 172.21.15.89:0/2112319159 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4500cee10 0x7fc4500b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:48.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.913+0000 7fc447e00700 1 -- 172.21.15.89:0/2112319159 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4501611b0 msgr2=0x7fc4500ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:48.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.913+0000 7fc447e00700 1 --2- 172.21.15.89:0/2112319159 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4501611b0 0x7fc4500ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:48.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.913+0000 7fc447e00700 1 -- 172.21.15.89:0/2112319159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc448009a70 con 0x7fc4500d21f0 2024-04-02T09:14:48.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.913+0000 7fc447e00700 1 --2- 172.21.15.89:0/2112319159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4500d21f0 0x7fc450154ae0 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c520 tx=0x7fc43800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:48.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.913+0000 7fc446a00700 1 -- 172.21.15.89:0/2112319159 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4380052d0 con 0x7fc4500d21f0 2024-04-02T09:14:48.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.913+0000 7fc446a00700 1 -- 172.21.15.89:0/2112319159 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc438005450 con 0x7fc4500d21f0 2024-04-02T09:14:48.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.913+0000 7fc446a00700 1 -- 172.21.15.89:0/2112319159 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4380038c0 con 0x7fc4500d21f0 2024-04-02T09:14:48.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.913+0000 7fc455c00700 1 -- 172.21.15.89:0/2112319159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4500c7200 con 0x7fc4500d21f0 2024-04-02T09:14:48.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.913+0000 7fc455c00700 1 -- 172.21.15.89:0/2112319159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc450155020 con 0x7fc4500d21f0 2024-04-02T09:14:48.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.914+0000 7fc455c00700 1 -- 172.21.15.89:0/2112319159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4500c7060 con 0x7fc4500d21f0 2024-04-02T09:14:48.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.915+0000 7fc446a00700 1 -- 172.21.15.89:0/2112319159 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 168) v1 ==== 89358+0+0 (secure 0 0 0) 0x7fc438003a20 con 0x7fc4500d21f0 2024-04-02T09:14:48.914 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:48.916+0000 7fa4fc34e240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:14:48.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.916+0000 7fc446a00700 1 --2- 172.21.15.89:0/2112319159 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc42c06af20 0x7fc42c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:48.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.916+0000 7fc446a00700 1 -- 172.21.15.89:0/2112319159 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fc438088e90 con 0x7fc4500d21f0 2024-04-02T09:14:48.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.916+0000 7fc44ea00700 1 -- 172.21.15.89:0/2112319159 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc42c06af20 msgr2=0x7fc42c06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:48.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.916+0000 7fc44ea00700 1 --2- 172.21.15.89:0/2112319159 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc42c06af20 0x7fc42c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:48.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:48.925+0000 7fc446a00700 1 -- 172.21.15.89:0/2112319159 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc4500c7060 con 0x7fc4500d21f0 2024-04-02T09:14:49.079 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:49.082+0000 7fd99d702240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:14:49.107 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:49.109+0000 7fa4fc34e240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:14:49.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.117+0000 7fc44ea00700 1 -- 172.21.15.89:0/2112319159 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc42c06af20 msgr2=0x7fc42c06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:49.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.117+0000 7fc44ea00700 1 --2- 172.21.15.89:0/2112319159 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc42c06af20 0x7fc42c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:49.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.172+0000 7fc455c00700 1 -- 172.21.15.89:0/2112319159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fc4500c7060 con 0x7fc4500d21f0 2024-04-02T09:14:49.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.176+0000 7fc446a00700 1 -- 172.21.15.89:0/2112319159 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v89)=0 v89) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc4500c7060 con 0x7fc4500d21f0 2024-04-02T09:14:49.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.179+0000 7fc455c00700 1 -- 172.21.15.89:0/2112319159 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc42c06af20 msgr2=0x7fc42c06d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:49.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.179+0000 7fc455c00700 1 --2- 172.21.15.89:0/2112319159 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc42c06af20 0x7fc42c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.179+0000 7fc455c00700 1 -- 172.21.15.89:0/2112319159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4500d21f0 msgr2=0x7fc450154ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:49.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.179+0000 7fc455c00700 1 --2- 172.21.15.89:0/2112319159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4500d21f0 0x7fc450154ae0 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c520 tx=0x7fc43800c830 comp rx=0 tx=0).stop 2024-04-02T09:14:49.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.179+0000 7fc455c00700 1 -- 172.21.15.89:0/2112319159 shutdown_connections 2024-04-02T09:14:49.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.179+0000 7fc455c00700 1 --2- 172.21.15.89:0/2112319159 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc42c06af20 0x7fc42c06d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.179+0000 7fc455c00700 1 --2- 172.21.15.89:0/2112319159 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4501611b0 0x7fc4500ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.179+0000 7fc455c00700 1 --2- 172.21.15.89:0/2112319159 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4500cee10 0x7fc4500b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.179+0000 7fc455c00700 1 --2- 172.21.15.89:0/2112319159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4500d21f0 0x7fc450154ae0 unknown :-1 s=CLOSED pgs=2849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.179+0000 7fc455c00700 1 -- 172.21.15.89:0/2112319159 >> 172.21.15.89:0/2112319159 conn(0x7fc45008aff0 msgr2=0x7fc4500d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:49.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.179+0000 7fc455c00700 1 -- 172.21.15.89:0/2112319159 shutdown_connections 2024-04-02T09:14:49.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.179+0000 7fc455c00700 1 -- 172.21.15.89:0/2112319159 wait complete. 2024-04-02T09:14:49.197 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T09:14:49.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.400+0000 7f6feba00700 1 Processor -- start 2024-04-02T09:14:49.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.400+0000 7f6feba00700 1 -- start start 2024-04-02T09:14:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.402+0000 7f6feba00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6fe40cbb30 0x7f6fe40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.402+0000 7f6feba00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe40cc450 con 0x7f6fe40cd020 2024-04-02T09:14:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.402+0000 7f6feba00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe40cdae0 con 0x7f6fe40cd480 2024-04-02T09:14:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.402+0000 7f6feba00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe40cd8f0 con 0x7f6fe40cbb30 2024-04-02T09:14:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.402+0000 7f6fe3e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6fe40cd020 0x7f6fe40b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39040/0 (socket says 172.21.15.89:39040) 2024-04-02T09:14:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.402+0000 7f6fe3e00700 1 -- 172.21.15.89:0/2822983021 learned_addr learned my addr 172.21.15.89:0/2822983021 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.402+0000 7f6fe9000700 1 --2- 172.21.15.89:0/2822983021 >> v2:172.21.15.113:3300/0 conn(0x7f6fe40cbb30 0x7f6fe40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.402+0000 7f6fe3400700 1 -- 172.21.15.89:0/2822983021 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924587365 0 0) 0x7f6fe40cc450 con 0x7f6fe40cd020 2024-04-02T09:14:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.402+0000 7f6fe3400700 1 -- 172.21.15.89:0/2822983021 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fcc0035b0 con 0x7f6fe40cd020 2024-04-02T09:14:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.402+0000 7f6fe3400700 1 -- 172.21.15.89:0/2822983021 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036530917 0 0) 0x7f6fe40cdae0 con 0x7f6fe40cd480 2024-04-02T09:14:49.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.403+0000 7f6fe3400700 1 -- 172.21.15.89:0/2822983021 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe40cc450 con 0x7f6fe40cd480 2024-04-02T09:14:49.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.403+0000 7f6fe9000700 1 -- 172.21.15.89:0/2822983021 >> v1:172.21.15.113:6789/0 conn(0x7f6fe40cd480 legacy=0x7f6fe406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:49.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.403+0000 7f6fe9000700 1 -- 172.21.15.89:0/2822983021 >> v1:172.21.15.89:6789/0 conn(0x7f6fe40cd020 legacy=0x7f6fe40b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:49.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.403+0000 7f6fe9000700 1 -- 172.21.15.89:0/2822983021 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe40cddc0 con 0x7f6fe40cbb30 2024-04-02T09:14:49.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.403+0000 7f6fe9000700 1 --2- 172.21.15.89:0/2822983021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe40cbb30 0x7f6fe40cbf10 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009840 tx=0x7f6fd8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2afe47330bf05d24 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:49.402 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:49.405+0000 7fd99d702240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:14:49.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.403+0000 7f6fe3400700 1 -- 172.21.15.89:0/2822983021 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8011830 con 0x7f6fe40cbb30 2024-04-02T09:14:49.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.403+0000 7f6fe3400700 1 -- 172.21.15.89:0/2822983021 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd8011990 con 0x7f6fe40cbb30 2024-04-02T09:14:49.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.403+0000 7f6fe3400700 1 -- 172.21.15.89:0/2822983021 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8011d20 con 0x7f6fe40cbb30 2024-04-02T09:14:49.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.404+0000 7f6feba00700 1 -- 172.21.15.89:0/2822983021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe40cbb30 msgr2=0x7f6fe40cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:49.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.404+0000 7f6feba00700 1 --2- 172.21.15.89:0/2822983021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe40cbb30 0x7f6fe40cbf10 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009840 tx=0x7f6fd8009b50 comp rx=0 tx=0).stop 2024-04-02T09:14:49.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.404+0000 7f6feba00700 1 -- 172.21.15.89:0/2822983021 shutdown_connections 2024-04-02T09:14:49.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.404+0000 7f6feba00700 1 --2- 172.21.15.89:0/2822983021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe40cbb30 0x7f6fe40cbf10 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.404+0000 7f6feba00700 1 -- 172.21.15.89:0/2822983021 >> 172.21.15.89:0/2822983021 conn(0x7f6fe40b0d70 msgr2=0x7f6fe40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:49.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.404+0000 7f6feba00700 1 -- 172.21.15.89:0/2822983021 shutdown_connections 2024-04-02T09:14:49.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.404+0000 7f6feba00700 1 -- 172.21.15.89:0/2822983021 wait complete. 2024-04-02T09:14:49.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.407+0000 7f6feba00700 1 Processor -- start 2024-04-02T09:14:49.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.407+0000 7f6feba00700 1 -- start start 2024-04-02T09:14:49.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.408+0000 7f6feba00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe40cbb30 0x7f6fe409cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:49.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6feba00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe40cd020 0x7f6fe40991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:49.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6feba00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40cd480 0x7f6fe41f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:49.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6feba00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe409d180 con 0x7f6fe40cbb30 2024-04-02T09:14:49.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6feba00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40c9af0 con 0x7f6fe40cd020 2024-04-02T09:14:49.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6feba00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40c9dd0 con 0x7f6fe40cd480 2024-04-02T09:14:49.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6fe9000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe40cbb30 0x7f6fe409cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:49.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6fe9000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe40cbb30 0x7f6fe409cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42720/0 (socket says 172.21.15.89:42720) 2024-04-02T09:14:49.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6fe9000700 1 -- 172.21.15.89:0/790534103 learned_addr learned my addr 172.21.15.89:0/790534103 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:49.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6fe3e00700 1 --2- 172.21.15.89:0/790534103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe40cd020 0x7f6fe40991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:49.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6fe9a00700 1 --2- 172.21.15.89:0/790534103 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40cd480 0x7f6fe41f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:49.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6fe9000700 1 -- 172.21.15.89:0/790534103 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40cd480 msgr2=0x7f6fe41f37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:49.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6fe9000700 1 --2- 172.21.15.89:0/790534103 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40cd480 0x7f6fe41f37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6fe9000700 1 -- 172.21.15.89:0/790534103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe40cd020 msgr2=0x7f6fe40991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:49.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6fe9000700 1 --2- 172.21.15.89:0/790534103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe40cd020 0x7f6fe40991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.409+0000 7f6fe9000700 1 -- 172.21.15.89:0/790534103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd80094a0 con 0x7f6fe40cbb30 2024-04-02T09:14:49.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.410+0000 7f6fe9000700 1 --2- 172.21.15.89:0/790534103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe40cbb30 0x7f6fe409cc40 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8014040 tx=0x7f6fd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:49.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.410+0000 7f6fe0c00700 1 -- 172.21.15.89:0/790534103 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd801a6e0 con 0x7f6fe40cbb30 2024-04-02T09:14:49.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.410+0000 7f6fe0c00700 1 -- 172.21.15.89:0/790534103 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd801a840 con 0x7f6fe40cbb30 2024-04-02T09:14:49.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.410+0000 7f6feba00700 1 -- 172.21.15.89:0/790534103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe4099710 con 0x7f6fe40cbb30 2024-04-02T09:14:49.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.410+0000 7f6fe0c00700 1 -- 172.21.15.89:0/790534103 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd801abb0 con 0x7f6fe40cbb30 2024-04-02T09:14:49.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.410+0000 7f6feba00700 1 -- 172.21.15.89:0/790534103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe41f4510 con 0x7f6fe40cbb30 2024-04-02T09:14:49.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.411+0000 7f6feba00700 1 -- 172.21.15.89:0/790534103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe4067720 con 0x7f6fe40cbb30 2024-04-02T09:14:49.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.412+0000 7f6fe0c00700 1 -- 172.21.15.89:0/790534103 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 168) v1 ==== 89358+0+0 (secure 0 0 0) 0x7f6fd8022070 con 0x7f6fe40cbb30 2024-04-02T09:14:49.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.413+0000 7f6fe0c00700 1 --2- 172.21.15.89:0/790534103 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6fc406af20 0x7f6fc406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:49.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.414+0000 7f6fe3e00700 1 -- 172.21.15.89:0/790534103 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6fc406af20 msgr2=0x7f6fc406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:49.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.414+0000 7f6fe0c00700 1 -- 172.21.15.89:0/790534103 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f6fd808a860 con 0x7f6fe40cbb30 2024-04-02T09:14:49.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.414+0000 7f6fe3e00700 1 --2- 172.21.15.89:0/790534103 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6fc406af20 0x7f6fc406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:49.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.418+0000 7f6fe0c00700 1 -- 172.21.15.89:0/790534103 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6fd8055630 con 0x7f6fe40cbb30 2024-04-02T09:14:49.451 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:49.453+0000 7fa4fc34e240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:14:49.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.615+0000 7f6fe3e00700 1 -- 172.21.15.89:0/790534103 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6fc406af20 msgr2=0x7f6fc406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:49.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.615+0000 7f6fe3e00700 1 --2- 172.21.15.89:0/790534103 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6fc406af20 0x7f6fc406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:49.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.629+0000 7f6feba00700 1 -- 172.21.15.89:0/790534103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f6fe40ce2e0 con 0x7f6fe40cbb30 2024-04-02T09:14:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.633+0000 7f6fe0c00700 1 -- 172.21.15.89:0/790534103 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v90)=0 v90) v1 ==== 133+0+0 (secure 0 0 0) 0x7f6fd80290c0 con 0x7f6fe40cbb30 2024-04-02T09:14:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.635+0000 7f6feba00700 1 -- 172.21.15.89:0/790534103 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6fc406af20 msgr2=0x7f6fc406d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.635+0000 7f6feba00700 1 --2- 172.21.15.89:0/790534103 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6fc406af20 0x7f6fc406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.635+0000 7f6feba00700 1 -- 172.21.15.89:0/790534103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe40cbb30 msgr2=0x7f6fe409cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.635+0000 7f6feba00700 1 --2- 172.21.15.89:0/790534103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe40cbb30 0x7f6fe409cc40 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8014040 tx=0x7f6fd8004750 comp rx=0 tx=0).stop 2024-04-02T09:14:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.635+0000 7f6feba00700 1 -- 172.21.15.89:0/790534103 shutdown_connections 2024-04-02T09:14:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.635+0000 7f6feba00700 1 --2- 172.21.15.89:0/790534103 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40cd480 0x7f6fe41f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.635+0000 7f6feba00700 1 --2- 172.21.15.89:0/790534103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe40cbb30 0x7f6fe409cc40 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.635+0000 7f6feba00700 1 --2- 172.21.15.89:0/790534103 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6fc406af20 0x7f6fc406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.635+0000 7f6feba00700 1 --2- 172.21.15.89:0/790534103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe40cd020 0x7f6fe40991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.635+0000 7f6feba00700 1 -- 172.21.15.89:0/790534103 >> 172.21.15.89:0/790534103 conn(0x7f6fe40b0d70 msgr2=0x7f6fe415bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.635+0000 7f6feba00700 1 -- 172.21.15.89:0/790534103 shutdown_connections 2024-04-02T09:14:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.635+0000 7f6feba00700 1 -- 172.21.15.89:0/790534103 wait complete. 2024-04-02T09:14:49.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:49.653 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-02T09:14:49.702 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-04-02T09:14:49.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.905+0000 7f9efee00700 1 Processor -- start 2024-04-02T09:14:49.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.905+0000 7f9efee00700 1 -- start start 2024-04-02T09:14:49.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.906+0000 7f9efee00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ef8160d70 0x7f9ef80b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:49.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.906+0000 7f9efee00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9ef815ef70 0x7f9ef815e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:49.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.906+0000 7f9efee00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef8161150 con 0x7f9ef806a790 2024-04-02T09:14:49.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.906+0000 7f9efee00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ef80b4cd0 con 0x7f9ef815ef70 2024-04-02T09:14:49.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.906+0000 7f9efee00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef806dba0 con 0x7f9ef8160d70 2024-04-02T09:14:49.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.907+0000 7f9ef7e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ef8160d70 0x7f9ef80b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:49.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.907+0000 7f9ef7400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9ef815ef70 0x7f9ef815e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:49.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.907+0000 7f9ef7400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9ef815ef70 0x7f9ef815e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:48006/0 (socket says 172.21.15.89:48006) 2024-04-02T09:14:49.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.907+0000 7f9efce00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f9ef806a790 0x7f9ef81613a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:53794/0 (socket says 172.21.15.89:53794) 2024-04-02T09:14:49.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.907+0000 7f9ef7400700 1 -- 172.21.15.89:0/2649118626 learned_addr learned my addr 172.21.15.89:0/2649118626 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:49.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.907+0000 7f9ef7e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ef8160d70 0x7f9ef80b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:50980/0 (socket says 172.21.15.89:50980) 2024-04-02T09:14:49.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.907+0000 7f9ef6a00700 1 -- 172.21.15.89:0/2649118626 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433305199 0 0) 0x7f9ef8161150 con 0x7f9ef806a790 2024-04-02T09:14:49.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.907+0000 7f9ef6a00700 1 -- 172.21.15.89:0/2649118626 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9edc0035b0 con 0x7f9ef806a790 2024-04-02T09:14:49.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.907+0000 7f9ef7e00700 1 -- 172.21.15.89:0/2649118626 >> v2:172.21.15.113:3300/0 conn(0x7f9ef815ef70 msgr2=0x7f9ef815e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:49.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.907+0000 7f9ef7e00700 1 --2- 172.21.15.89:0/2649118626 >> v2:172.21.15.113:3300/0 conn(0x7f9ef815ef70 0x7f9ef815e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.907+0000 7f9ef7e00700 1 -- 172.21.15.89:0/2649118626 >> v1:172.21.15.113:6789/0 conn(0x7f9ef806a790 legacy=0x7f9ef81613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:49.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.907+0000 7f9ef7e00700 1 -- 172.21.15.89:0/2649118626 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef815ed30 con 0x7f9ef8160d70 2024-04-02T09:14:49.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.907+0000 7f9ef7400700 1 --2- 172.21.15.89:0/2649118626 >> v2:172.21.15.113:3300/0 conn(0x7f9ef815ef70 0x7f9ef815e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:14:49.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.907+0000 7f9ef7e00700 1 --2- 172.21.15.89:0/2649118626 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ef8160d70 0x7f9ef80b3200 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009840 tx=0x7f9ee8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f0622c29b924408 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:49.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.908+0000 7f9ef6a00700 1 -- 172.21.15.89:0/2649118626 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8011830 con 0x7f9ef8160d70 2024-04-02T09:14:49.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.908+0000 7f9ef6a00700 1 -- 172.21.15.89:0/2649118626 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee8011990 con 0x7f9ef8160d70 2024-04-02T09:14:49.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.908+0000 7f9ef6a00700 1 -- 172.21.15.89:0/2649118626 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8011d20 con 0x7f9ef8160d70 2024-04-02T09:14:49.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.908+0000 7f9efee00700 1 -- 172.21.15.89:0/2649118626 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ef8160d70 msgr2=0x7f9ef80b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:49.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.908+0000 7f9efee00700 1 --2- 172.21.15.89:0/2649118626 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ef8160d70 0x7f9ef80b3200 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009840 tx=0x7f9ee8009b50 comp rx=0 tx=0).stop 2024-04-02T09:14:49.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.908+0000 7f9efee00700 1 -- 172.21.15.89:0/2649118626 shutdown_connections 2024-04-02T09:14:49.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.908+0000 7f9efee00700 1 --2- 172.21.15.89:0/2649118626 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ef8160d70 0x7f9ef80b3200 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.908+0000 7f9efee00700 1 --2- 172.21.15.89:0/2649118626 >> v2:172.21.15.113:3300/0 conn(0x7f9ef815ef70 0x7f9ef815e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.908+0000 7f9efee00700 1 -- 172.21.15.89:0/2649118626 >> 172.21.15.89:0/2649118626 conn(0x7f9ef80b5720 msgr2=0x7f9ef80b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:49.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.908+0000 7f9efee00700 1 -- 172.21.15.89:0/2649118626 shutdown_connections 2024-04-02T09:14:49.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.908+0000 7f9efee00700 1 -- 172.21.15.89:0/2649118626 wait complete. 2024-04-02T09:14:49.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.910+0000 7f9efee00700 1 Processor -- start 2024-04-02T09:14:49.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.910+0000 7f9efee00700 1 -- start start 2024-04-02T09:14:49.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.911+0000 7f9efee00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ef806a790 0x7f9ef81379d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:49.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.911+0000 7f9efee00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef815ef70 0x7f9ef8161360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:49.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.911+0000 7f9efee00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ef8160d70 0x7f9ef81618a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:49.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.911+0000 7f9efee00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef8137f10 con 0x7f9ef806a790 2024-04-02T09:14:49.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.911+0000 7f9efee00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef81329e0 con 0x7f9ef815ef70 2024-04-02T09:14:49.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.911+0000 7f9efee00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef8132cc0 con 0x7f9ef8160d70 2024-04-02T09:14:49.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.911+0000 7f9ef7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ef806a790 0x7f9ef81379d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:49.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.911+0000 7f9ef7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ef806a790 0x7f9ef81379d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42736/0 (socket says 172.21.15.89:42736) 2024-04-02T09:14:49.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.912+0000 7f9ef7e00700 1 -- 172.21.15.89:0/1379450032 learned_addr learned my addr 172.21.15.89:0/1379450032 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:49.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.912+0000 7f9ef7400700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef815ef70 0x7f9ef8161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:49.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.912+0000 7f9efce00700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ef8160d70 0x7f9ef81618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:49.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.912+0000 7f9ef7e00700 1 -- 172.21.15.89:0/1379450032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ef8160d70 msgr2=0x7f9ef81618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:49.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.912+0000 7f9ef7e00700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ef8160d70 0x7f9ef81618a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.912+0000 7f9ef7e00700 1 -- 172.21.15.89:0/1379450032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef815ef70 msgr2=0x7f9ef8161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:49.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.912+0000 7f9ef7e00700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef815ef70 0x7f9ef8161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:49.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.912+0000 7f9ef7e00700 1 -- 172.21.15.89:0/1379450032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee4009550 con 0x7f9ef806a790 2024-04-02T09:14:49.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.912+0000 7f9ef7e00700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ef806a790 0x7f9ef81379d0 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80090b0 tx=0x7f9ee8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:49.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.912+0000 7f9eefe00700 1 -- 172.21.15.89:0/1379450032 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee80036a0 con 0x7f9ef806a790 2024-04-02T09:14:49.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.913+0000 7f9eefe00700 1 -- 172.21.15.89:0/1379450032 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee801e070 con 0x7f9ef806a790 2024-04-02T09:14:49.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.913+0000 7f9eefe00700 1 -- 172.21.15.89:0/1379450032 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8003a40 con 0x7f9ef806a790 2024-04-02T09:14:49.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.913+0000 7f9efee00700 1 -- 172.21.15.89:0/1379450032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee80094a0 con 0x7f9ef806a790 2024-04-02T09:14:49.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.913+0000 7f9efee00700 1 -- 172.21.15.89:0/1379450032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef8133280 con 0x7f9ef806a790 2024-04-02T09:14:49.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.914+0000 7f9efee00700 1 -- 172.21.15.89:0/1379450032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef809b270 con 0x7f9ef806a790 2024-04-02T09:14:49.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.915+0000 7f9eefe00700 1 -- 172.21.15.89:0/1379450032 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 168) v1 ==== 89358+0+0 (secure 0 0 0) 0x7f9ee8003ba0 con 0x7f9ef806a790 2024-04-02T09:14:49.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.915+0000 7f9eefe00700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 0x7f9ed806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:49.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.916+0000 7f9eefe00700 1 -- 172.21.15.89:0/1379450032 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f9ee808a740 con 0x7f9ef806a790 2024-04-02T09:14:49.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.916+0000 7f9ef7400700 1 -- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 msgr2=0x7f9ed806d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:49.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.916+0000 7f9ef7400700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 0x7f9ed806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:14:49.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:49.920+0000 7f9eefe00700 1 -- 172.21.15.89:0/1379450032 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9ee8055490 con 0x7f9ef806a790 2024-04-02T09:14:49.934 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:49.936+0000 7fd99d702240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:14:50.010 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:50.011+0000 7fa4fc34e240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:14:50.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:50.117+0000 7f9ef7400700 1 -- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 msgr2=0x7f9ed806d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:50.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:50.117+0000 7f9ef7400700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 0x7f9ed806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:14:50.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:50.150+0000 7f9efee00700 1 -- 172.21.15.89:0/1379450032 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ef80b54a0 con 0x7f9ed806af20 2024-04-02T09:14:50.191 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:50.193+0000 7fd99d702240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:14:50.269 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:50.270+0000 7fa4fc34e240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:14:50.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:50.518+0000 7f9ef7400700 1 -- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 msgr2=0x7f9ed806d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:50.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:50.518+0000 7f9ef7400700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 0x7f9ed806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T09:14:50.655 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:50.658+0000 7fd99d702240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:14:50.734 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:50.735+0000 7fa4fc34e240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:14:51.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:51.319+0000 7f9ef7400700 1 -- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 msgr2=0x7f9ed806d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:51.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:51.319+0000 7f9ef7400700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 0x7f9ed806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T09:14:51.361 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:51.363+0000 7fd99d702240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:14:51.446 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:51.446+0000 7fa4fc34e240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:14:51.453 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:51.454+0000 7fd99d702240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:14:51.537 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:51.538+0000 7fa4fc34e240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:14:51.610 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:51.611+0000 7fd99d702240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:14:51.697 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:51.698+0000 7fa4fc34e240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:14:51.824 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:51.824+0000 7fd99d702240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:14:51.912 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:51.912+0000 7fa4fc34e240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:14:51.915 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:51.916+0000 7fd99d702240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:14:52.004 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:52.005+0000 7fa4fc34e240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:14:52.013 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:52.015+0000 7fd99d702240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:14:52.104 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:52.105+0000 7fa4fc34e240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:14:52.111 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:52.113+0000 7fd99d702240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:14:52.204 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:52.204+0000 7fa4fc34e240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:14:52.209 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:52.210+0000 7fd99d702240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:14:52.302 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:52.303+0000 7fa4fc34e240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:14:52.422 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:52.423+0000 7fd99d702240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:14:52.517 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:52.518+0000 7fa4fc34e240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:14:52.677 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:52.678+0000 7fd99d702240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:14:52.775 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:52.775+0000 7fa4fc34e240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:14:52.868 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:52.869+0000 7fd99d702240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:14:52.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:52.920+0000 7f9ef7400700 1 -- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 msgr2=0x7f9ed806d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:14:52.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:52.920+0000 7f9ef7400700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 0x7f9ed806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T09:14:52.964 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:52.965+0000 7fd99d702240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:14:52.968 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:52.969+0000 7fa4fc34e240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:14:53.061 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:53.061+0000 7fa4fc34e240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:14:53.692 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:53.693+0000 7fd99d702240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:14:53.798 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:53.799+0000 7fa4fc34e240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:14:53.905 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:53.906+0000 7fd99d702240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:14:54.010 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:54.011+0000 7fa4fc34e240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:14:54.112 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:54.113+0000 7fd99d702240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:14:54.218 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:54.218+0000 7fa4fc34e240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:14:54.590 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:54.592+0000 7fd99d702240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:14:54.698 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:54.699+0000 7fa4fc34e240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:14:54.797 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:54.799+0000 7fd99d702240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:14:54.908 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:54.909+0000 7fa4fc34e240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:14:55.232 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:55.234+0000 7fd99d702240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:14:55.343 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:55.344+0000 7fd99d702240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:14:55.350 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:55.351+0000 7fa4fc34e240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:14:55.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:55.404+0000 7f9eefe00700 1 -- 172.21.15.89:0/1379450032 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 169) v1 ==== 89358+0+0 (secure 0 0 0) 0x7f9ee8055860 con 0x7f9ef806a790 2024-04-02T09:14:55.469 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:55.470+0000 7fa4fc34e240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:14:55.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:55.479+0000 7f9eefe00700 1 -- 172.21.15.89:0/1379450032 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 170) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f9ee8074260 con 0x7f9ef806a790 2024-04-02T09:14:55.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:55.479+0000 7f9eefe00700 1 -- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 msgr2=0x7f9ed806d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:14:55.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:55.479+0000 7f9eefe00700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 0x7f9ed806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:55.640 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:55.640+0000 7fa464a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:55.640 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:55.640+0000 7fa464a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:55.640 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:55.640+0000 7fa464a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:55.640 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:55.640+0000 7fa464a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:55.640 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:55.640+0000 7fa464a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:55.640 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:55.641+0000 7fa468600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:55.641 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:55.641+0000 7fa468600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:55.641 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:55.641+0000 7fa468600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:55.641 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:55.641+0000 7fa468600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:55.641 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:55.641+0000 7fa468600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:14:56.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.483+0000 7f9eefe00700 1 -- 172.21.15.89:0/1379450032 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mgrmap(e 171) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f9ee808c830 con 0x7f9ef806a790 2024-04-02T09:14:56.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.483+0000 7f9eefe00700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 0x7f9ed806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:56.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.483+0000 7f9eefe00700 1 -- 172.21.15.89:0/1379450032 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ed801cdf0 con 0x7f9ed806af20 2024-04-02T09:14:56.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.487+0000 7f9ef7400700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 0x7f9ed806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:56.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.491+0000 7f9ef7400700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 0x7f9ed806d3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9ef80ae1a0 tx=0x7f9ee40092c0 comp rx=0 tx=0).ready entity=mgr.12969 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:56.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.564+0000 7f9eefe00700 1 -- 172.21.15.89:0/1379450032 <== mgr.12969 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9ed801cdf0 con 0x7f9ed806af20 2024-04-02T09:14:56.564 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path snap_test_dir1 2024-04-02T09:14:56.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.567+0000 7f9efee00700 1 -- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 msgr2=0x7f9ed806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:56.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.567+0000 7f9efee00700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 0x7f9ed806d3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9ef80ae1a0 tx=0x7f9ee40092c0 comp rx=0 tx=0).stop 2024-04-02T09:14:56.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.567+0000 7f9efee00700 1 -- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ef806a790 msgr2=0x7f9ef81379d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:56.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.567+0000 7f9efee00700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ef806a790 0x7f9ef81379d0 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80090b0 tx=0x7f9ee8004750 comp rx=0 tx=0).stop 2024-04-02T09:14:56.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.567+0000 7f9efee00700 1 -- 172.21.15.89:0/1379450032 shutdown_connections 2024-04-02T09:14:56.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.567+0000 7f9efee00700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ef8160d70 0x7f9ef81618a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:56.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.567+0000 7f9efee00700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ef806a790 0x7f9ef81379d0 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:56.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.567+0000 7f9efee00700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9ed806af20 0x7f9ed806d3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:56.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.567+0000 7f9efee00700 1 --2- 172.21.15.89:0/1379450032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef815ef70 0x7f9ef8161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:56.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.567+0000 7f9efee00700 1 -- 172.21.15.89:0/1379450032 >> 172.21.15.89:0/1379450032 conn(0x7f9ef80b5720 msgr2=0x7f9ef809ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:56.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.567+0000 7f9efee00700 1 -- 172.21.15.89:0/1379450032 shutdown_connections 2024-04-02T09:14:56.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.567+0000 7f9efee00700 1 -- 172.21.15.89:0/1379450032 wait complete. 2024-04-02T09:14:56.585 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-02T09:14:56.585 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-04-02T09:14:56.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.782+0000 7fd26aa00700 1 Processor -- start 2024-04-02T09:14:56.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.782+0000 7fd26aa00700 1 -- start start 2024-04-02T09:14:56.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.783+0000 7fd26aa00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2640d22a0 0x7fd2640cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:56.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.783+0000 7fd26aa00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26415dd10 con 0x7fd264162030 2024-04-02T09:14:56.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.783+0000 7fd26aa00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26415dfc0 con 0x7fd26415d930 2024-04-02T09:14:56.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.783+0000 7fd26aa00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd26415ec40 con 0x7fd2640d22a0 2024-04-02T09:14:56.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.783+0000 7fd25be00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd264162030 0x7fd2640d1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39046/0 (socket says 172.21.15.89:39046) 2024-04-02T09:14:56.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.783+0000 7fd25be00700 1 -- 172.21.15.89:0/2868488298 learned_addr learned my addr 172.21.15.89:0/2868488298 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:14:56.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.783+0000 7fd268a00700 1 --2- 172.21.15.89:0/2868488298 >> v2:172.21.15.89:3300/0 conn(0x7fd2640d22a0 0x7fd2640cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:56.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.784+0000 7fd263400700 1 -- 172.21.15.89:0/2868488298 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088586389 0 0) 0x7fd26415dd10 con 0x7fd264162030 2024-04-02T09:14:56.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.784+0000 7fd263400700 1 -- 172.21.15.89:0/2868488298 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2480035b0 con 0x7fd264162030 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.784+0000 7fd263400700 1 -- 172.21.15.89:0/2868488298 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000481587 0 0) 0x7fd26415dfc0 con 0x7fd26415d930 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.784+0000 7fd268a00700 1 -- 172.21.15.89:0/2868488298 >> v1:172.21.15.113:6789/0 conn(0x7fd26415d930 legacy=0x7fd2640d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.784+0000 7fd268a00700 1 -- 172.21.15.89:0/2868488298 >> v1:172.21.15.89:6789/0 conn(0x7fd264162030 legacy=0x7fd2640d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.784+0000 7fd268a00700 1 -- 172.21.15.89:0/2868488298 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26415e2a0 con 0x7fd2640d22a0 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.784+0000 7fd268a00700 1 --2- 172.21.15.89:0/2868488298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2640d22a0 0x7fd2640cf310 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009d90 tx=0x7fd25c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad9c2acd19347cd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.784+0000 7fd263400700 1 -- 172.21.15.89:0/2868488298 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25c00e070 con 0x7fd2640d22a0 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.784+0000 7fd263400700 1 -- 172.21.15.89:0/2868488298 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25c00bb20 con 0x7fd2640d22a0 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.784+0000 7fd263400700 1 -- 172.21.15.89:0/2868488298 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25c00bed0 con 0x7fd2640d22a0 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.785+0000 7fd26aa00700 1 -- 172.21.15.89:0/2868488298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2640d22a0 msgr2=0x7fd2640cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.785+0000 7fd26aa00700 1 --2- 172.21.15.89:0/2868488298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2640d22a0 0x7fd2640cf310 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009d90 tx=0x7fd25c00b330 comp rx=0 tx=0).stop 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.785+0000 7fd26aa00700 1 -- 172.21.15.89:0/2868488298 shutdown_connections 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.785+0000 7fd26aa00700 1 --2- 172.21.15.89:0/2868488298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2640d22a0 0x7fd2640cf310 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.785+0000 7fd26aa00700 1 -- 172.21.15.89:0/2868488298 >> 172.21.15.89:0/2868488298 conn(0x7fd26408aff0 msgr2=0x7fd2640d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.785+0000 7fd26aa00700 1 -- 172.21.15.89:0/2868488298 shutdown_connections 2024-04-02T09:14:56.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.785+0000 7fd26aa00700 1 -- 172.21.15.89:0/2868488298 wait complete. 2024-04-02T09:14:56.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.787+0000 7fd26aa00700 1 Processor -- start 2024-04-02T09:14:56.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.787+0000 7fd26aa00700 1 -- start start 2024-04-02T09:14:56.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd26aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2640d22a0 0x7fd264159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:56.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd26aa00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd26415d930 0x7fd2641529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:56.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd26aa00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd264162030 0x7fd2640b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:56.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd26aa00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26415ad10 con 0x7fd2640d22a0 2024-04-02T09:14:56.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd26aa00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd264159940 con 0x7fd26415d930 2024-04-02T09:14:56.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd26aa00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd264152ef0 con 0x7fd264162030 2024-04-02T09:14:56.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd263e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2640d22a0 0x7fd264159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:56.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd263e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2640d22a0 0x7fd264159400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42878/0 (socket says 172.21.15.89:42878) 2024-04-02T09:14:56.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd263e00700 1 -- 172.21.15.89:0/2767875909 learned_addr learned my addr 172.21.15.89:0/2767875909 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:56.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd263e00700 1 -- 172.21.15.89:0/2767875909 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd264162030 msgr2=0x7fd2640b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:56.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd263e00700 1 --2- 172.21.15.89:0/2767875909 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd264162030 0x7fd2640b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:56.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd263e00700 1 -- 172.21.15.89:0/2767875909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd26415d930 msgr2=0x7fd2641529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:56.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd263e00700 1 --2- 172.21.15.89:0/2767875909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd26415d930 0x7fd2641529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:56.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd263e00700 1 -- 172.21.15.89:0/2767875909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c009a70 con 0x7fd2640d22a0 2024-04-02T09:14:56.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.789+0000 7fd263e00700 1 --2- 172.21.15.89:0/2767875909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2640d22a0 0x7fd264159400 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7fd25400c4a0 tx=0x7fd25400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:56.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.790+0000 7fd260c00700 1 -- 172.21.15.89:0/2767875909 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2540054e0 con 0x7fd2640d22a0 2024-04-02T09:14:56.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.790+0000 7fd260c00700 1 -- 172.21.15.89:0/2767875909 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd254003680 con 0x7fd2640d22a0 2024-04-02T09:14:56.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.790+0000 7fd260c00700 1 -- 172.21.15.89:0/2767875909 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254003a70 con 0x7fd2640d22a0 2024-04-02T09:14:56.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.790+0000 7fd26aa00700 1 -- 172.21.15.89:0/2767875909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26415ce10 con 0x7fd2640d22a0 2024-04-02T09:14:56.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.790+0000 7fd26aa00700 1 -- 172.21.15.89:0/2767875909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26415b290 con 0x7fd2640d22a0 2024-04-02T09:14:56.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.791+0000 7fd26aa00700 1 -- 172.21.15.89:0/2767875909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2640b2a30 con 0x7fd2640d22a0 2024-04-02T09:14:56.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.792+0000 7fd260c00700 1 -- 172.21.15.89:0/2767875909 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 171) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fd254003bd0 con 0x7fd2640d22a0 2024-04-02T09:14:56.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.793+0000 7fd260c00700 1 --2- 172.21.15.89:0/2767875909 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd24006ab30 0x7fd24006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:56.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.793+0000 7fd260c00700 1 -- 172.21.15.89:0/2767875909 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7fd254088920 con 0x7fd2640d22a0 2024-04-02T09:14:56.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.793+0000 7fd25be00700 1 --2- 172.21.15.89:0/2767875909 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd24006ab30 0x7fd24006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:56.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.793+0000 7fd25be00700 1 --2- 172.21.15.89:0/2767875909 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd24006ab30 0x7fd24006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd24c001610 tx=0x7fd24c0079c0 comp rx=0 tx=0).ready entity=mgr.12969 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:56.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:56.798+0000 7fd260c00700 1 -- 172.21.15.89:0/2767875909 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd254053a00 con 0x7fd2640d22a0 2024-04-02T09:14:57.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.025+0000 7fd26aa00700 1 -- 172.21.15.89:0/2767875909 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd26415e860 con 0x7fd24006ab30 2024-04-02T09:14:57.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.088+0000 7fd260c00700 1 -- 172.21.15.89:0/2767875909 <== mgr.12969 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7fd24c005940 con 0x7fd24006ab30 2024-04-02T09:14:57.087 INFO:teuthology.orchestra.run.smithi089.stdout:Retention added to path snap_test_dir1 2024-04-02T09:14:57.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.090+0000 7fd26aa00700 1 -- 172.21.15.89:0/2767875909 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd24006ab30 msgr2=0x7fd24006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:57.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.090+0000 7fd26aa00700 1 --2- 172.21.15.89:0/2767875909 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd24006ab30 0x7fd24006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd24c001610 tx=0x7fd24c0079c0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.090+0000 7fd26aa00700 1 -- 172.21.15.89:0/2767875909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2640d22a0 msgr2=0x7fd264159400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:57.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.090+0000 7fd26aa00700 1 --2- 172.21.15.89:0/2767875909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2640d22a0 0x7fd264159400 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7fd25400c4a0 tx=0x7fd25400c860 comp rx=0 tx=0).stop 2024-04-02T09:14:57.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.091+0000 7fd26aa00700 1 -- 172.21.15.89:0/2767875909 shutdown_connections 2024-04-02T09:14:57.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.091+0000 7fd26aa00700 1 --2- 172.21.15.89:0/2767875909 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd264162030 0x7fd2640b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.091+0000 7fd26aa00700 1 --2- 172.21.15.89:0/2767875909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2640d22a0 0x7fd264159400 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.091+0000 7fd26aa00700 1 --2- 172.21.15.89:0/2767875909 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd24006ab30 0x7fd24006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.091+0000 7fd26aa00700 1 --2- 172.21.15.89:0/2767875909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd26415d930 0x7fd2641529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.091+0000 7fd26aa00700 1 -- 172.21.15.89:0/2767875909 >> 172.21.15.89:0/2767875909 conn(0x7fd26408aff0 msgr2=0x7fd26406caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:57.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.091+0000 7fd26aa00700 1 -- 172.21.15.89:0/2767875909 shutdown_connections 2024-04-02T09:14:57.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.091+0000 7fd26aa00700 1 -- 172.21.15.89:0/2767875909 wait complete. 2024-04-02T09:14:57.108 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-04-02T09:14:57.129 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-02-09_15_00 in ~3s... 2024-04-02T09:14:57.129 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-04-02T09:14:57.129 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-04-02T09:14:57.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.322+0000 7f5417200700 1 Processor -- start 2024-04-02T09:14:57.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.322+0000 7f5417200700 1 -- start start 2024-04-02T09:14:57.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f5417200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f54100d21f0 0x7f54100ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:57.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f5417200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54100cec40 0x7f5410161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:57.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f5417200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54100ce270 con 0x7f54101611b0 2024-04-02T09:14:57.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f5417200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f541015b600 con 0x7f54100d21f0 2024-04-02T09:14:57.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f5417200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54100d2760 con 0x7f54100cec40 2024-04-02T09:14:57.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f540f400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f54100d21f0 0x7f54100ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:57.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f540f400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f54100d21f0 0x7f54100ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:48130/0 (socket says 172.21.15.89:48130) 2024-04-02T09:14:57.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f540f400700 1 -- 172.21.15.89:0/1660580878 learned_addr learned my addr 172.21.15.89:0/1660580878 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:57.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f540fe00700 1 --1- 172.21.15.89:0/1660580878 >> v1:172.21.15.113:6789/0 conn(0x7f54101611b0 0x7f541015ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:53804/0 (socket says 172.21.15.89:53804) 2024-04-02T09:14:57.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f5415200700 1 --2- 172.21.15.89:0/1660580878 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54100cec40 0x7f5410161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:57.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f540ea00700 1 -- 172.21.15.89:0/1660580878 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017738302 0 0) 0x7f54100ce270 con 0x7f54101611b0 2024-04-02T09:14:57.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f540f400700 1 -- 172.21.15.89:0/1660580878 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54100cec40 msgr2=0x7f5410161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:57.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f540f400700 1 --2- 172.21.15.89:0/1660580878 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54100cec40 0x7f5410161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f540f400700 1 -- 172.21.15.89:0/1660580878 >> v1:172.21.15.113:6789/0 conn(0x7f54101611b0 legacy=0x7f541015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:57.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f540f400700 1 -- 172.21.15.89:0/1660580878 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54100d2a40 con 0x7f54100d21f0 2024-04-02T09:14:57.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.324+0000 7f5415200700 1 --2- 172.21.15.89:0/1660580878 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54100cec40 0x7f5410161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:14:57.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.325+0000 7f540f400700 1 --2- 172.21.15.89:0/1660580878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54100d21f0 0x7f54100ce700 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0098f0 tx=0x7f53fc009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25faf737670d4527 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:57.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.325+0000 7f540ea00700 1 -- 172.21.15.89:0/1660580878 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc011830 con 0x7f54100d21f0 2024-04-02T09:14:57.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.325+0000 7f540ea00700 1 -- 172.21.15.89:0/1660580878 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc011990 con 0x7f54100d21f0 2024-04-02T09:14:57.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.325+0000 7f540ea00700 1 -- 172.21.15.89:0/1660580878 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc011d20 con 0x7f54100d21f0 2024-04-02T09:14:57.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.326+0000 7f5417200700 1 -- 172.21.15.89:0/1660580878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54100d21f0 msgr2=0x7f54100ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:57.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.326+0000 7f5417200700 1 --2- 172.21.15.89:0/1660580878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54100d21f0 0x7f54100ce700 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0098f0 tx=0x7f53fc009c00 comp rx=0 tx=0).stop 2024-04-02T09:14:57.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.326+0000 7f5417200700 1 -- 172.21.15.89:0/1660580878 shutdown_connections 2024-04-02T09:14:57.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.326+0000 7f5417200700 1 --2- 172.21.15.89:0/1660580878 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54100cec40 0x7f5410161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.326+0000 7f5417200700 1 --2- 172.21.15.89:0/1660580878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54100d21f0 0x7f54100ce700 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.326+0000 7f5417200700 1 -- 172.21.15.89:0/1660580878 >> 172.21.15.89:0/1660580878 conn(0x7f541008aff0 msgr2=0x7f54100d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:57.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.326+0000 7f5417200700 1 -- 172.21.15.89:0/1660580878 shutdown_connections 2024-04-02T09:14:57.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.326+0000 7f5417200700 1 -- 172.21.15.89:0/1660580878 wait complete. 2024-04-02T09:14:57.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.327+0000 7f5417200700 1 Processor -- start 2024-04-02T09:14:57.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.328+0000 7f5417200700 1 -- start start 2024-04-02T09:14:57.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.328+0000 7f5417200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54100cec40 0x7f5410158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.328+0000 7f5417200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54100d21f0 0x7f54101f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.329+0000 7f5417200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54101611b0 0x7f54100b6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.329+0000 7f5417200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54100b6150 con 0x7f54100d21f0 2024-04-02T09:14:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.329+0000 7f5417200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5410158fd0 con 0x7f54100cec40 2024-04-02T09:14:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.329+0000 7f5417200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54100cc380 con 0x7f54101611b0 2024-04-02T09:14:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.329+0000 7f540f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54100d21f0 0x7f54101f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.329+0000 7f540f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54100d21f0 0x7f54101f4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42890/0 (socket says 172.21.15.89:42890) 2024-04-02T09:14:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.329+0000 7f540f400700 1 -- 172.21.15.89:0/2452777979 learned_addr learned my addr 172.21.15.89:0/2452777979 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.329+0000 7f540fe00700 1 --2- 172.21.15.89:0/2452777979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54100cec40 0x7f5410158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.329+0000 7f5415200700 1 --2- 172.21.15.89:0/2452777979 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54101611b0 0x7f54100b6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.329+0000 7f540f400700 1 -- 172.21.15.89:0/2452777979 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54101611b0 msgr2=0x7f54100b6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.329+0000 7f540f400700 1 --2- 172.21.15.89:0/2452777979 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54101611b0 0x7f54100b6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.329+0000 7f540f400700 1 -- 172.21.15.89:0/2452777979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54100cec40 msgr2=0x7f5410158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:57.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.329+0000 7f540f400700 1 --2- 172.21.15.89:0/2452777979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54100cec40 0x7f5410158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.329+0000 7f540f400700 1 -- 172.21.15.89:0/2452777979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5408009a70 con 0x7f54100d21f0 2024-04-02T09:14:57.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.330+0000 7f540f400700 1 --2- 172.21.15.89:0/2452777979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54100d21f0 0x7f54101f4210 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7f53fc009c40 tx=0x7f53fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:57.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.330+0000 7f5407e00700 1 -- 172.21.15.89:0/2452777979 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc0045d0 con 0x7f54100d21f0 2024-04-02T09:14:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.330+0000 7f5407e00700 1 -- 172.21.15.89:0/2452777979 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc01f070 con 0x7f54100d21f0 2024-04-02T09:14:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.330+0000 7f5417200700 1 -- 172.21.15.89:0/2452777979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53fc009550 con 0x7f54100d21f0 2024-04-02T09:14:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.330+0000 7f5407e00700 1 -- 172.21.15.89:0/2452777979 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc0038e0 con 0x7f54100d21f0 2024-04-02T09:14:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.330+0000 7f5417200700 1 -- 172.21.15.89:0/2452777979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54100b63a0 con 0x7f54100d21f0 2024-04-02T09:14:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.331+0000 7f5417200700 1 -- 172.21.15.89:0/2452777979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5410067720 con 0x7f54100d21f0 2024-04-02T09:14:57.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.332+0000 7f5407e00700 1 -- 172.21.15.89:0/2452777979 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 171) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f53fc003a40 con 0x7f54100d21f0 2024-04-02T09:14:57.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.333+0000 7f5407e00700 1 --2- 172.21.15.89:0/2452777979 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f53f006ab30 0x7f53f006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:57.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.333+0000 7f5407e00700 1 -- 172.21.15.89:0/2452777979 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f53fc021070 con 0x7f54100d21f0 2024-04-02T09:14:57.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.333+0000 7f540fe00700 1 --2- 172.21.15.89:0/2452777979 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f53f006ab30 0x7f53f006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:57.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.334+0000 7f540fe00700 1 --2- 172.21.15.89:0/2452777979 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f53f006ab30 0x7f53f006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5400000f60 tx=0x7f5400007e70 comp rx=0 tx=0).ready entity=mgr.12969 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:57.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.338+0000 7f5407e00700 1 -- 172.21.15.89:0/2452777979 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f53fc055ce0 con 0x7f54100d21f0 2024-04-02T09:14:57.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.492+0000 7f5407e00700 1 -- 172.21.15.89:0/2452777979 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 172) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f53fc003cf0 con 0x7f54100d21f0 2024-04-02T09:14:57.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.579+0000 7f5417200700 1 -- 172.21.15.89:0/2452777979 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f541015f6a0 con 0x7f53f006ab30 2024-04-02T09:14:57.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.586+0000 7f5407e00700 1 -- 172.21.15.89:0/2452777979 <== mgr.12969 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f5400004ab0 con 0x7f53f006ab30 2024-04-02T09:14:57.586 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:14:57.586 INFO:teuthology.orchestra.run.smithi089.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-04-02T09:14:57.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.589+0000 7f5417200700 1 -- 172.21.15.89:0/2452777979 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f53f006ab30 msgr2=0x7f53f006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:57.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.589+0000 7f5417200700 1 --2- 172.21.15.89:0/2452777979 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f53f006ab30 0x7f53f006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5400000f60 tx=0x7f5400007e70 comp rx=0 tx=0).stop 2024-04-02T09:14:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.589+0000 7f5417200700 1 -- 172.21.15.89:0/2452777979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54100d21f0 msgr2=0x7f54101f4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.589+0000 7f5417200700 1 --2- 172.21.15.89:0/2452777979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54100d21f0 0x7f54101f4210 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7f53fc009c40 tx=0x7f53fc004750 comp rx=0 tx=0).stop 2024-04-02T09:14:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.589+0000 7f5417200700 1 -- 172.21.15.89:0/2452777979 shutdown_connections 2024-04-02T09:14:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.589+0000 7f5417200700 1 --2- 172.21.15.89:0/2452777979 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f53f006ab30 0x7f53f006cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.589+0000 7f5417200700 1 --2- 172.21.15.89:0/2452777979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54100cec40 0x7f5410158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.589+0000 7f5417200700 1 --2- 172.21.15.89:0/2452777979 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54101611b0 0x7f54100b6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.589+0000 7f5417200700 1 --2- 172.21.15.89:0/2452777979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54100d21f0 0x7f54101f4210 unknown :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.589+0000 7f5417200700 1 -- 172.21.15.89:0/2452777979 >> 172.21.15.89:0/2452777979 conn(0x7f541008aff0 msgr2=0x7f54101598c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.589+0000 7f5417200700 1 -- 172.21.15.89:0/2452777979 shutdown_connections 2024-04-02T09:14:57.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.589+0000 7f5417200700 1 -- 172.21.15.89:0/2452777979 wait complete. 2024-04-02T09:14:57.606 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-04-02T09:14:57.606 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-04-02T09:14:57.606 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-02T09:14:57.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.801+0000 7f6491000700 1 Processor -- start 2024-04-02T09:14:57.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.801+0000 7f6491000700 1 -- start start 2024-04-02T09:14:57.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.802+0000 7f6491000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f648c1611b0 0x7f648c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:57.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.802+0000 7f6491000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f648c0cee10 0x7f648c161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:57.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.802+0000 7f6491000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f648c0ce270 con 0x7f648c15f4d0 2024-04-02T09:14:57.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.802+0000 7f6491000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f648c15b600 con 0x7f648c0cee10 2024-04-02T09:14:57.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.802+0000 7f6491000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648c0d2760 con 0x7f648c1611b0 2024-04-02T09:14:57.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.803+0000 7f648a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f648c0cee10 0x7f648c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:57.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.803+0000 7f648a800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f648c0cee10 0x7f648c161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42926/0 (socket says 172.21.15.89:42926) 2024-04-02T09:14:57.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.803+0000 7f648a800700 1 -- 172.21.15.89:0/3713676003 learned_addr learned my addr 172.21.15.89:0/3713676003 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:57.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.803+0000 7f6489e00700 1 --2- 172.21.15.89:0/3713676003 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f648c1611b0 0x7f648c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:57.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.803+0000 7f648a800700 1 -- 172.21.15.89:0/3713676003 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f648c1611b0 msgr2=0x7f648c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:57.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.803+0000 7f648a800700 1 --2- 172.21.15.89:0/3713676003 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f648c1611b0 0x7f648c15ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.803+0000 7f648a800700 1 -- 172.21.15.89:0/3713676003 >> v1:172.21.15.113:6789/0 conn(0x7f648c15f4d0 legacy=0x7f648c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.803+0000 7f648a800700 1 -- 172.21.15.89:0/3713676003 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648c0d2a40 con 0x7f648c0cee10 2024-04-02T09:14:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.803+0000 7f648a800700 1 --2- 172.21.15.89:0/3713676003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c0cee10 0x7f648c161590 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7f6484009e50 tx=0x7f648400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce25da4ff5ab6f12 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.803+0000 7f6488a00700 1 -- 172.21.15.89:0/3713676003 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648400e070 con 0x7f648c0cee10 2024-04-02T09:14:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.804+0000 7f6488a00700 1 -- 172.21.15.89:0/3713676003 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6484003a20 con 0x7f648c0cee10 2024-04-02T09:14:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.804+0000 7f6488a00700 1 -- 172.21.15.89:0/3713676003 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484003dd0 con 0x7f648c0cee10 2024-04-02T09:14:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.804+0000 7f6491000700 1 -- 172.21.15.89:0/3713676003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c0cee10 msgr2=0x7f648c161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.804+0000 7f6491000700 1 --2- 172.21.15.89:0/3713676003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c0cee10 0x7f648c161590 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7f6484009e50 tx=0x7f648400b330 comp rx=0 tx=0).stop 2024-04-02T09:14:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.804+0000 7f6491000700 1 -- 172.21.15.89:0/3713676003 shutdown_connections 2024-04-02T09:14:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.804+0000 7f6491000700 1 --2- 172.21.15.89:0/3713676003 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f648c1611b0 0x7f648c15ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.804+0000 7f6491000700 1 --2- 172.21.15.89:0/3713676003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c0cee10 0x7f648c161590 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.804+0000 7f6491000700 1 -- 172.21.15.89:0/3713676003 >> 172.21.15.89:0/3713676003 conn(0x7f648c08aff0 msgr2=0x7f648c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.804+0000 7f6491000700 1 -- 172.21.15.89:0/3713676003 shutdown_connections 2024-04-02T09:14:57.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.804+0000 7f6491000700 1 -- 172.21.15.89:0/3713676003 wait complete. 2024-04-02T09:14:57.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.806+0000 7f6491000700 1 Processor -- start 2024-04-02T09:14:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.806+0000 7f6491000700 1 -- start start 2024-04-02T09:14:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.807+0000 7f6491000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f648c0cee10 0x7f648c15aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.807+0000 7f6491000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c15f4d0 0x7f648c09d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.807+0000 7f6491000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f648c1611b0 0x7f648c09d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.807+0000 7f6491000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c09dca0 con 0x7f648c15f4d0 2024-04-02T09:14:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.807+0000 7f6491000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c15afd0 con 0x7f648c1611b0 2024-04-02T09:14:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.807+0000 7f6491000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648c09dee0 con 0x7f648c0cee10 2024-04-02T09:14:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.807+0000 7f6489400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c15f4d0 0x7f648c09d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.807+0000 7f6489400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c15f4d0 0x7f648c09d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42932/0 (socket says 172.21.15.89:42932) 2024-04-02T09:14:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.807+0000 7f6489400700 1 -- 172.21.15.89:0/3763157786 learned_addr learned my addr 172.21.15.89:0/3763157786 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.807+0000 7f6489e00700 1 --2- 172.21.15.89:0/3763157786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f648c0cee10 0x7f648c15aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.807+0000 7f648a800700 1 --2- 172.21.15.89:0/3763157786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f648c1611b0 0x7f648c09d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.807+0000 7f6489400700 1 -- 172.21.15.89:0/3763157786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f648c0cee10 msgr2=0x7f648c15aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.808+0000 7f6489400700 1 --2- 172.21.15.89:0/3763157786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f648c0cee10 0x7f648c15aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.808+0000 7f6489400700 1 -- 172.21.15.89:0/3763157786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f648c1611b0 msgr2=0x7f648c09d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:57.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.808+0000 7f6489400700 1 --2- 172.21.15.89:0/3763157786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f648c1611b0 0x7f648c09d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:57.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.808+0000 7f6489400700 1 -- 172.21.15.89:0/3763157786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6484009b30 con 0x7f648c15f4d0 2024-04-02T09:14:57.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.808+0000 7f6489400700 1 --2- 172.21.15.89:0/3763157786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c15f4d0 0x7f648c09d220 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f647800c4a0 tx=0x7f647800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:57.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.808+0000 7f6482000700 1 -- 172.21.15.89:0/3763157786 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6478011790 con 0x7f648c15f4d0 2024-04-02T09:14:57.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.808+0000 7f6482000700 1 -- 172.21.15.89:0/3763157786 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64780118f0 con 0x7f648c15f4d0 2024-04-02T09:14:57.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.808+0000 7f6491000700 1 -- 172.21.15.89:0/3763157786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648c09e1c0 con 0x7f648c15f4d0 2024-04-02T09:14:57.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.808+0000 7f6482000700 1 -- 172.21.15.89:0/3763157786 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6478011ca0 con 0x7f648c15f4d0 2024-04-02T09:14:57.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.808+0000 7f6491000700 1 -- 172.21.15.89:0/3763157786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648c0b65d0 con 0x7f648c15f4d0 2024-04-02T09:14:57.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.809+0000 7f6491000700 1 -- 172.21.15.89:0/3763157786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648c0b6150 con 0x7f648c15f4d0 2024-04-02T09:14:57.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.810+0000 7f6482000700 1 -- 172.21.15.89:0/3763157786 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 172) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f647801c460 con 0x7f648c15f4d0 2024-04-02T09:14:57.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.811+0000 7f6482000700 1 --2- 172.21.15.89:0/3763157786 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f646c06ae00 0x7f646c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:57.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.811+0000 7f6482000700 1 -- 172.21.15.89:0/3763157786 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f6478089530 con 0x7f648c15f4d0 2024-04-02T09:14:57.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.811+0000 7f6489e00700 1 --2- 172.21.15.89:0/3763157786 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f646c06ae00 0x7f646c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:57.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.812+0000 7f6489e00700 1 --2- 172.21.15.89:0/3763157786 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f646c06ae00 0x7f646c06d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f647c006b20 tx=0x7f647c008040 comp rx=0 tx=0).ready entity=mgr.12969 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:57.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:57.816+0000 7f6482000700 1 -- 172.21.15.89:0/3763157786 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f648c0b6150 con 0x7f648c15f4d0 2024-04-02T09:14:58.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.046+0000 7f6491000700 1 -- 172.21.15.89:0/3763157786 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f648c0cdf00 con 0x7f646c06ae00 2024-04-02T09:14:58.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.116+0000 7f6482000700 1 -- 172.21.15.89:0/3763157786 <== mgr.12969 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f647c0076b0 con 0x7f646c06ae00 2024-04-02T09:14:58.115 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule removed for path snap_test_dir1 2024-04-02T09:14:58.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.118+0000 7f6491000700 1 -- 172.21.15.89:0/3763157786 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f646c06ae00 msgr2=0x7f646c06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:58.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.118+0000 7f6491000700 1 --2- 172.21.15.89:0/3763157786 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f646c06ae00 0x7f646c06d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f647c006b20 tx=0x7f647c008040 comp rx=0 tx=0).stop 2024-04-02T09:14:58.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.118+0000 7f6491000700 1 -- 172.21.15.89:0/3763157786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c15f4d0 msgr2=0x7f648c09d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:58.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.118+0000 7f6491000700 1 --2- 172.21.15.89:0/3763157786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c15f4d0 0x7f648c09d220 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f647800c4a0 tx=0x7f647800c7b0 comp rx=0 tx=0).stop 2024-04-02T09:14:58.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.118+0000 7f6491000700 1 -- 172.21.15.89:0/3763157786 shutdown_connections 2024-04-02T09:14:58.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.118+0000 7f6491000700 1 --2- 172.21.15.89:0/3763157786 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f646c06ae00 0x7f646c06d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:58.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.118+0000 7f6491000700 1 --2- 172.21.15.89:0/3763157786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f648c1611b0 0x7f648c09d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:58.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.118+0000 7f6491000700 1 --2- 172.21.15.89:0/3763157786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f648c0cee10 0x7f648c15aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:58.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.118+0000 7f6491000700 1 --2- 172.21.15.89:0/3763157786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c15f4d0 0x7f648c09d220 unknown :-1 s=CLOSED pgs=2870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:58.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.118+0000 7f6491000700 1 -- 172.21.15.89:0/3763157786 >> 172.21.15.89:0/3763157786 conn(0x7f648c08aff0 msgr2=0x7f648c06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:58.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.119+0000 7f6491000700 1 -- 172.21.15.89:0/3763157786 shutdown_connections 2024-04-02T09:14:58.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.119+0000 7f6491000700 1 -- 172.21.15.89:0/3763157786 wait complete. 2024-04-02T09:14:58.135 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-02T09:14:58.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:58.135 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-02T09:14:58.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:14:58.182 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-02T09:14:58.268 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T09:14:58.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.510+0000 7f519d600700 1 Processor -- start 2024-04-02T09:14:58.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.510+0000 7f519d600700 1 -- start start 2024-04-02T09:14:58.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.511+0000 7f519d600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51980cee10 0x7f5198161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:58.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.511+0000 7f519d600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51980ce270 con 0x7f51980d21f0 2024-04-02T09:14:58.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.511+0000 7f519d600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519815b600 con 0x7f51981611b0 2024-04-02T09:14:58.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.511+0000 7f519d600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51980d2760 con 0x7f51980cee10 2024-04-02T09:14:58.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.511+0000 7f5197400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51980cee10 0x7f5198161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:58.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.511+0000 7f5197400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51980cee10 0x7f5198161590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42944/0 (socket says 172.21.15.89:42944) 2024-04-02T09:14:58.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.511+0000 7f5196000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f51980d21f0 0x7f51980ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39048/0 (socket says 172.21.15.89:39048) 2024-04-02T09:14:58.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.511+0000 7f5197400700 1 -- 172.21.15.89:0/1202138995 learned_addr learned my addr 172.21.15.89:0/1202138995 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:58.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.512+0000 7f5197400700 1 -- 172.21.15.89:0/1202138995 >> v1:172.21.15.113:6789/0 conn(0x7f51981611b0 legacy=0x7f519815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:14:58.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.512+0000 7f5197400700 1 -- 172.21.15.89:0/1202138995 >> v1:172.21.15.89:6789/0 conn(0x7f51980d21f0 legacy=0x7f51980ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:58.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.512+0000 7f5197400700 1 -- 172.21.15.89:0/1202138995 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51980d2a40 con 0x7f51980cee10 2024-04-02T09:14:58.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.512+0000 7f5197400700 1 --2- 172.21.15.89:0/1202138995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51980cee10 0x7f5198161590 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f5190009e50 tx=0x7f519000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b980728059bba61d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:58.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.512+0000 7f5195600700 1 -- 172.21.15.89:0/1202138995 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519000e070 con 0x7f51980cee10 2024-04-02T09:14:58.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.512+0000 7f5195600700 1 -- 172.21.15.89:0/1202138995 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5190003a20 con 0x7f51980cee10 2024-04-02T09:14:58.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.512+0000 7f5195600700 1 -- 172.21.15.89:0/1202138995 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5190003dd0 con 0x7f51980cee10 2024-04-02T09:14:58.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.513+0000 7f519d600700 1 -- 172.21.15.89:0/1202138995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51980cee10 msgr2=0x7f5198161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:58.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.513+0000 7f519d600700 1 --2- 172.21.15.89:0/1202138995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51980cee10 0x7f5198161590 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f5190009e50 tx=0x7f519000b330 comp rx=0 tx=0).stop 2024-04-02T09:14:58.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.513+0000 7f519d600700 1 -- 172.21.15.89:0/1202138995 shutdown_connections 2024-04-02T09:14:58.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.513+0000 7f519d600700 1 --2- 172.21.15.89:0/1202138995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51980cee10 0x7f5198161590 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:58.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.513+0000 7f519d600700 1 -- 172.21.15.89:0/1202138995 >> 172.21.15.89:0/1202138995 conn(0x7f519808aff0 msgr2=0x7f51980d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:58.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.513+0000 7f519d600700 1 -- 172.21.15.89:0/1202138995 shutdown_connections 2024-04-02T09:14:58.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.513+0000 7f519d600700 1 -- 172.21.15.89:0/1202138995 wait complete. 2024-04-02T09:14:58.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.515+0000 7f519d600700 1 Processor -- start 2024-04-02T09:14:58.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.515+0000 7f519d600700 1 -- start start 2024-04-02T09:14:58.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.516+0000 7f519d600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51980cee10 0x7f51981585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:58.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.516+0000 7f519d600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51980d21f0 0x7f51980b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:58.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.516+0000 7f519d600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51981611b0 0x7f51980b6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:58.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.516+0000 7f519d600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5198158b10 con 0x7f51981611b0 2024-04-02T09:14:58.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.516+0000 7f519d600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51980c8f60 con 0x7f51980d21f0 2024-04-02T09:14:58.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.516+0000 7f519d600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5198159010 con 0x7f51980cee10 2024-04-02T09:14:58.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.516+0000 7f5197400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51981611b0 0x7f51980b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:58.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.516+0000 7f5196a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51980cee10 0x7f51981585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:58.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.516+0000 7f5197400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51981611b0 0x7f51980b6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42950/0 (socket says 172.21.15.89:42950) 2024-04-02T09:14:58.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.516+0000 7f5196000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51980d21f0 0x7f51980b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:58.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.516+0000 7f5197400700 1 -- 172.21.15.89:0/228828353 learned_addr learned my addr 172.21.15.89:0/228828353 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:14:58.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.517+0000 7f5197400700 1 -- 172.21.15.89:0/228828353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51980cee10 msgr2=0x7f51981585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:58.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.517+0000 7f5197400700 1 --2- 172.21.15.89:0/228828353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51980cee10 0x7f51981585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:58.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.517+0000 7f5197400700 1 -- 172.21.15.89:0/228828353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51980d21f0 msgr2=0x7f51980b6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:58.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.517+0000 7f5197400700 1 --2- 172.21.15.89:0/228828353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51980d21f0 0x7f51980b6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:58.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.517+0000 7f5197400700 1 -- 172.21.15.89:0/228828353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5190009b30 con 0x7f51981611b0 2024-04-02T09:14:58.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.517+0000 7f5197400700 1 --2- 172.21.15.89:0/228828353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51981611b0 0x7f51980b6690 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f519808ce10 tx=0x7f5190004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:58.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.517+0000 7f518ea00700 1 -- 172.21.15.89:0/228828353 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5190004440 con 0x7f51981611b0 2024-04-02T09:14:58.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.517+0000 7f518ea00700 1 -- 172.21.15.89:0/228828353 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51900045a0 con 0x7f51981611b0 2024-04-02T09:14:58.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.517+0000 7f519d600700 1 -- 172.21.15.89:0/228828353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51980c8350 con 0x7f51981611b0 2024-04-02T09:14:58.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.517+0000 7f518ea00700 1 -- 172.21.15.89:0/228828353 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5190004950 con 0x7f51981611b0 2024-04-02T09:14:58.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.517+0000 7f519d600700 1 -- 172.21.15.89:0/228828353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51980c8910 con 0x7f51981611b0 2024-04-02T09:14:58.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.518+0000 7f519d600700 1 -- 172.21.15.89:0/228828353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5198067720 con 0x7f51981611b0 2024-04-02T09:14:58.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.519+0000 7f518ea00700 1 -- 172.21.15.89:0/228828353 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 172) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f5190012430 con 0x7f51981611b0 2024-04-02T09:14:58.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.520+0000 7f518ea00700 1 --2- 172.21.15.89:0/228828353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f517806ae00 0x7f517806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:14:58.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.521+0000 7f518ea00700 1 -- 172.21.15.89:0/228828353 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f519008ade0 con 0x7f51981611b0 2024-04-02T09:14:58.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.521+0000 7f5196a00700 1 --2- 172.21.15.89:0/228828353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f517806ae00 0x7f517806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:14:58.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.521+0000 7f5196a00700 1 --2- 172.21.15.89:0/228828353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f517806ae00 0x7f517806d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f51980c7700 tx=0x7f51880090e0 comp rx=0 tx=0).ready entity=mgr.12969 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:14:58.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.525+0000 7f518ea00700 1 -- 172.21.15.89:0/228828353 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5190055a20 con 0x7f51981611b0 2024-04-02T09:14:58.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:58.784+0000 7f519d600700 1 -- 172.21.15.89:0/228828353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f51980b5d00 con 0x7f51981611b0 2024-04-02T09:14:59.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:59.504+0000 7f518ea00700 1 -- 172.21.15.89:0/228828353 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f5190055df0 con 0x7f51981611b0 2024-04-02T09:14:59.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:59.505+0000 7f518ea00700 1 -- 172.21.15.89:0/228828353 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v173) v1 ==== 93+0+0 (secure 0 0 0) 0x7f5190052390 con 0x7f51981611b0 2024-04-02T09:14:59.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:59.508+0000 7f519d600700 1 -- 172.21.15.89:0/228828353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f517806ae00 msgr2=0x7f517806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:59.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:59.508+0000 7f519d600700 1 --2- 172.21.15.89:0/228828353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f517806ae00 0x7f517806d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f51980c7700 tx=0x7f51880090e0 comp rx=0 tx=0).stop 2024-04-02T09:14:59.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:59.508+0000 7f519d600700 1 -- 172.21.15.89:0/228828353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51981611b0 msgr2=0x7f51980b6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:14:59.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:59.508+0000 7f519d600700 1 --2- 172.21.15.89:0/228828353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51981611b0 0x7f51980b6690 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f519808ce10 tx=0x7f5190004ad0 comp rx=0 tx=0).stop 2024-04-02T09:14:59.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:59.508+0000 7f519d600700 1 -- 172.21.15.89:0/228828353 shutdown_connections 2024-04-02T09:14:59.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:59.508+0000 7f519d600700 1 --2- 172.21.15.89:0/228828353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51981611b0 0x7f51980b6690 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:59.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:59.508+0000 7f519d600700 1 --2- 172.21.15.89:0/228828353 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51980cee10 0x7f51981585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:59.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:59.508+0000 7f519d600700 1 --2- 172.21.15.89:0/228828353 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f517806ae00 0x7f517806d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:59.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:59.508+0000 7f519d600700 1 --2- 172.21.15.89:0/228828353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51980d21f0 0x7f51980b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:14:59.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:59.508+0000 7f519d600700 1 -- 172.21.15.89:0/228828353 >> 172.21.15.89:0/228828353 conn(0x7f519808aff0 msgr2=0x7f519815a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:14:59.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:59.508+0000 7f519d600700 1 -- 172.21.15.89:0/228828353 shutdown_connections 2024-04-02T09:14:59.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:14:59.508+0000 7f519d600700 1 -- 172.21.15.89:0/228828353 wait complete. 2024-04-02T09:14:59.527 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T09:14:59.529 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:14:59.618 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:14:59.715 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T09:14:59.716 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:14:59.716 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:14:59.730 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:59.731+0000 7f4bc690b240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:14:59.734 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:59.735+0000 7f92319a2240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:14:59.760 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:14:59.760 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:14:59.844 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:14:59.915 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:14:59.917+0000 7f4bc690b240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:14:59.925 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:14:59.926+0000 7f92319a2240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:15:00.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.100+0000 7fb5cfe00700 1 Processor -- start 2024-04-02T09:15:00.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.100+0000 7fb5cfe00700 1 -- start start 2024-04-02T09:15:00.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.101+0000 7fb5cfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb5c81619d0 0x7fb5c81685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:00.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.101+0000 7fb5cfe00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c80d30d0 con 0x7fb5c815d640 2024-04-02T09:15:00.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.101+0000 7fb5cfe00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c80b6840 con 0x7fb5c80ceec0 2024-04-02T09:15:00.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.101+0000 7fb5cfe00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c80d2b30 con 0x7fb5c81619d0 2024-04-02T09:15:00.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.101+0000 7fb5cde00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb5c81619d0 0x7fb5c81685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:00.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.101+0000 7fb5cde00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb5c81619d0 0x7fb5c81685f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54382/0 (socket says 172.21.15.89:54382) 2024-04-02T09:15:00.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.101+0000 7fb5cd400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb5c815d640 0x7fb5c80d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59912/0 (socket says 172.21.15.89:59912) 2024-04-02T09:15:00.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.101+0000 7fb5cde00700 1 -- 172.21.15.89:0/3267009992 learned_addr learned my addr 172.21.15.89:0/3267009992 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.102+0000 7fb5cde00700 1 -- 172.21.15.89:0/3267009992 >> v1:172.21.15.113:6789/0 conn(0x7fb5c80ceec0 legacy=0x7fb5c81612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.102+0000 7fb5cde00700 1 -- 172.21.15.89:0/3267009992 >> v1:172.21.15.89:6789/0 conn(0x7fb5c815d640 legacy=0x7fb5c80d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.102+0000 7fb5cde00700 1 -- 172.21.15.89:0/3267009992 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c80d2df0 con 0x7fb5c81619d0 2024-04-02T09:15:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.102+0000 7fb5c7e00700 1 -- 172.21.15.89:0/3267009992 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724980393 0 0) 0x7fb5c80d30d0 con 0x7fb5c815d640 2024-04-02T09:15:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.102+0000 7fb5cde00700 1 --2- 172.21.15.89:0/3267009992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5c81619d0 0x7fb5c81685f0 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0009e50 tx=0x7fb5c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b747ab22274ce8aa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.102+0000 7fb5c7e00700 1 -- 172.21.15.89:0/3267009992 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c000e070 con 0x7fb5c81619d0 2024-04-02T09:15:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.102+0000 7fb5c7e00700 1 -- 172.21.15.89:0/3267009992 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c000b9c0 con 0x7fb5c81619d0 2024-04-02T09:15:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.102+0000 7fb5c7e00700 1 -- 172.21.15.89:0/3267009992 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c000bd70 con 0x7fb5c81619d0 2024-04-02T09:15:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.103+0000 7fb5cfe00700 1 -- 172.21.15.89:0/3267009992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5c81619d0 msgr2=0x7fb5c81685f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.103+0000 7fb5cfe00700 1 --2- 172.21.15.89:0/3267009992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5c81619d0 0x7fb5c81685f0 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0009e50 tx=0x7fb5c000b330 comp rx=0 tx=0).stop 2024-04-02T09:15:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.103+0000 7fb5cfe00700 1 -- 172.21.15.89:0/3267009992 shutdown_connections 2024-04-02T09:15:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.103+0000 7fb5cfe00700 1 --2- 172.21.15.89:0/3267009992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5c81619d0 0x7fb5c81685f0 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.103+0000 7fb5cfe00700 1 -- 172.21.15.89:0/3267009992 >> 172.21.15.89:0/3267009992 conn(0x7fb5c808aff0 msgr2=0x7fb5c80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:00.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.103+0000 7fb5cfe00700 1 -- 172.21.15.89:0/3267009992 shutdown_connections 2024-04-02T09:15:00.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.103+0000 7fb5cfe00700 1 -- 172.21.15.89:0/3267009992 wait complete. 2024-04-02T09:15:00.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.105+0000 7fb5cfe00700 1 Processor -- start 2024-04-02T09:15:00.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.105+0000 7fb5cfe00700 1 -- start start 2024-04-02T09:15:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cfe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5c80ceec0 0x7fb5c81f6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cfe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5c815d640 0x7fb5c81f6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cfe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5c81619d0 0x7fb5c8149060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cfe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80c7b30 con 0x7fb5c81619d0 2024-04-02T09:15:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cfe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80c7e10 con 0x7fb5c80ceec0 2024-04-02T09:15:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cfe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80c80f0 con 0x7fb5c815d640 2024-04-02T09:15:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cd400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5c80ceec0 0x7fb5c81f6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cd400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5c80ceec0 0x7fb5c81f6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:50530/0 (socket says 172.21.15.89:50530) 2024-04-02T09:15:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cd400700 1 -- 172.21.15.89:0/757496853 learned_addr learned my addr 172.21.15.89:0/757496853 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cde00700 1 --2- 172.21.15.89:0/757496853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5c81619d0 0x7fb5c8149060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cca00700 1 --2- 172.21.15.89:0/757496853 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5c815d640 0x7fb5c81f6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cd400700 1 -- 172.21.15.89:0/757496853 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5c815d640 msgr2=0x7fb5c81f6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cd400700 1 --2- 172.21.15.89:0/757496853 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5c815d640 0x7fb5c81f6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cd400700 1 -- 172.21.15.89:0/757496853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5c81619d0 msgr2=0x7fb5c8149060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.106+0000 7fb5cd400700 1 --2- 172.21.15.89:0/757496853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5c81619d0 0x7fb5c8149060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.107+0000 7fb5cd400700 1 -- 172.21.15.89:0/757496853 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c0009b30 con 0x7fb5c80ceec0 2024-04-02T09:15:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.107+0000 7fb5cd400700 1 --2- 172.21.15.89:0/757496853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5c80ceec0 0x7fb5c81f6650 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00c520 tx=0x7fb5bc00c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:00.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.107+0000 7fb5c5600700 1 -- 172.21.15.89:0/757496853 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc004c80 con 0x7fb5c80ceec0 2024-04-02T09:15:00.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.107+0000 7fb5c5600700 1 -- 172.21.15.89:0/757496853 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5bc004e00 con 0x7fb5c80ceec0 2024-04-02T09:15:00.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.107+0000 7fb5cfe00700 1 -- 172.21.15.89:0/757496853 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c80c83d0 con 0x7fb5c80ceec0 2024-04-02T09:15:00.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.107+0000 7fb5c5600700 1 -- 172.21.15.89:0/757496853 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc007310 con 0x7fb5c80ceec0 2024-04-02T09:15:00.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.107+0000 7fb5cfe00700 1 -- 172.21.15.89:0/757496853 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c8156a30 con 0x7fb5c80ceec0 2024-04-02T09:15:00.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.108+0000 7fb5cfe00700 1 -- 172.21.15.89:0/757496853 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c80c7f50 con 0x7fb5c80ceec0 2024-04-02T09:15:00.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.109+0000 7fb5c5600700 1 -- 172.21.15.89:0/757496853 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fb5bc007470 con 0x7fb5c80ceec0 2024-04-02T09:15:00.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.110+0000 7fb5c5600700 1 --2- 172.21.15.89:0/757496853 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb5a806ae40 0x7fb5a806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:00.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.110+0000 7fb5cca00700 1 -- 172.21.15.89:0/757496853 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb5a806ae40 msgr2=0x7fb5a806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:00.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.110+0000 7fb5c5600700 1 -- 172.21.15.89:0/757496853 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7fb5bc089630 con 0x7fb5c80ceec0 2024-04-02T09:15:00.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.110+0000 7fb5cca00700 1 --2- 172.21.15.89:0/757496853 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb5a806ae40 0x7fb5a806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:00.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.114+0000 7fb5c5600700 1 -- 172.21.15.89:0/757496853 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb5c80c7f50 con 0x7fb5c80ceec0 2024-04-02T09:15:00.234 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:00.235+0000 7f4bc690b240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:15:00.261 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:00.262+0000 7f92319a2240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:15:00.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.311+0000 7fb5cca00700 1 -- 172.21.15.89:0/757496853 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb5a806ae40 msgr2=0x7fb5a806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:00.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.311+0000 7fb5cca00700 1 --2- 172.21.15.89:0/757496853 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb5a806ae40 0x7fb5a806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.326+0000 7fb5cfe00700 1 -- 172.21.15.89:0/757496853 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5c80c7f50 con 0x7fb5c80ceec0 2024-04-02T09:15:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.327+0000 7fb5c5600700 1 -- 172.21.15.89:0/757496853 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v266) v1 ==== 74+0+17360 (secure 0 0 0) 0x7fb5c80c7f50 con 0x7fb5c80ceec0 2024-04-02T09:15:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:00.327 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":266,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:14:55.473418+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:14:32.705979+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-04-02T09:14:33.330752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/422001866":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/3534660524":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/2610551691":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/1604459128":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/3010462327":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/3737296061":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/1785135719":"2024-04-03T09:14:55.473381+0000","172.21.15.89:6833/108855":"2024-04-03T09:14:55.473381+0000","172.21.15.89:6832/108855":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/1017424418":"2024-04-03T09:14:55.473381+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:15:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.329+0000 7fb5cfe00700 1 -- 172.21.15.89:0/757496853 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb5a806ae40 msgr2=0x7fb5a806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.329+0000 7fb5cfe00700 1 --2- 172.21.15.89:0/757496853 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb5a806ae40 0x7fb5a806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.329+0000 7fb5cfe00700 1 -- 172.21.15.89:0/757496853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5c80ceec0 msgr2=0x7fb5c81f6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.329+0000 7fb5cfe00700 1 --2- 172.21.15.89:0/757496853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5c80ceec0 0x7fb5c81f6650 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00c520 tx=0x7fb5bc00c830 comp rx=0 tx=0).stop 2024-04-02T09:15:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.330+0000 7fb5cfe00700 1 -- 172.21.15.89:0/757496853 shutdown_connections 2024-04-02T09:15:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.330+0000 7fb5cfe00700 1 --2- 172.21.15.89:0/757496853 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb5a806ae40 0x7fb5a806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.330+0000 7fb5cfe00700 1 --2- 172.21.15.89:0/757496853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5c80ceec0 0x7fb5c81f6650 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.330+0000 7fb5cfe00700 1 --2- 172.21.15.89:0/757496853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5c81619d0 0x7fb5c8149060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.330+0000 7fb5cfe00700 1 --2- 172.21.15.89:0/757496853 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb5c815d640 0x7fb5c81f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.330+0000 7fb5cfe00700 1 -- 172.21.15.89:0/757496853 >> 172.21.15.89:0/757496853 conn(0x7fb5c808aff0 msgr2=0x7fb5c806c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.330+0000 7fb5cfe00700 1 -- 172.21.15.89:0/757496853 shutdown_connections 2024-04-02T09:15:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.330+0000 7fb5cfe00700 1 -- 172.21.15.89:0/757496853 wait complete. 2024-04-02T09:15:00.347 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T09:14:32.705979+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T09:14:33.330752+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '265', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:15:00.348 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:15:00.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.591+0000 7fd990400700 1 Processor -- start 2024-04-02T09:15:00.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.591+0000 7fd990400700 1 -- start start 2024-04-02T09:15:00.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.592+0000 7fd990400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd9880cbb30 0x7fd9880cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:00.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.592+0000 7fd990400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd9880cd020 0x7fd9880cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:00.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.592+0000 7fd990400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd9880b2070 0x7fd9880b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:00.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.592+0000 7fd990400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9880cc450 con 0x7fd9880cd020 2024-04-02T09:15:00.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.592+0000 7fd990400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9880cdae0 con 0x7fd9880b2070 2024-04-02T09:15:00.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.592+0000 7fd990400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880cddc0 con 0x7fd9880cbb30 2024-04-02T09:15:00.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.592+0000 7fd98d000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd9880cd020 0x7fd9880cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:00.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.593+0000 7fd98d000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd9880cd020 0x7fd9880cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54404/0 (socket says 172.21.15.89:54404) 2024-04-02T09:15:00.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.593+0000 7fd98d000700 1 -- 172.21.15.89:0/1526025113 learned_addr learned my addr 172.21.15.89:0/1526025113 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:00.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.593+0000 7fd98da00700 1 --2- 172.21.15.89:0/1526025113 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd9880cbb30 0x7fd9880cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:00.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.593+0000 7fd98e400700 1 --2- 172.21.15.89:0/1526025113 >> v2:172.21.15.113:3300/0 conn(0x7fd9880b2070 0x7fd9880b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:00.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.593+0000 7fd98d000700 1 -- 172.21.15.89:0/1526025113 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd9880cbb30 msgr2=0x7fd9880cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:00.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.593+0000 7fd98d000700 1 --2- 172.21.15.89:0/1526025113 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd9880cbb30 0x7fd9880cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.593+0000 7fd98d000700 1 -- 172.21.15.89:0/1526025113 >> v2:172.21.15.113:3300/0 conn(0x7fd9880b2070 msgr2=0x7fd9880b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:00.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.593+0000 7fd98d000700 1 --2- 172.21.15.89:0/1526025113 >> v2:172.21.15.113:3300/0 conn(0x7fd9880b2070 0x7fd9880b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.593+0000 7fd98d000700 1 -- 172.21.15.89:0/1526025113 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9880ce0a0 con 0x7fd9880cd020 2024-04-02T09:15:00.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.593+0000 7fd98d000700 1 --2- 172.21.15.89:0/1526025113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9880cd020 0x7fd9880cd4a0 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7fd98806c9f0 tx=0x7fd974009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46db8974d2cda874 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:00.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.594+0000 7fd987e00700 1 -- 172.21.15.89:0/1526025113 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974011830 con 0x7fd9880cd020 2024-04-02T09:15:00.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.594+0000 7fd987e00700 1 -- 172.21.15.89:0/1526025113 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd974011990 con 0x7fd9880cd020 2024-04-02T09:15:00.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.594+0000 7fd987e00700 1 -- 172.21.15.89:0/1526025113 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974011d20 con 0x7fd9880cd020 2024-04-02T09:15:00.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.594+0000 7fd990400700 1 -- 172.21.15.89:0/1526025113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9880cd020 msgr2=0x7fd9880cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:00.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.594+0000 7fd990400700 1 --2- 172.21.15.89:0/1526025113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9880cd020 0x7fd9880cd4a0 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7fd98806c9f0 tx=0x7fd974009bd0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.594+0000 7fd990400700 1 -- 172.21.15.89:0/1526025113 shutdown_connections 2024-04-02T09:15:00.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.594+0000 7fd990400700 1 --2- 172.21.15.89:0/1526025113 >> v2:172.21.15.113:3300/0 conn(0x7fd9880b2070 0x7fd9880b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.594+0000 7fd990400700 1 --2- 172.21.15.89:0/1526025113 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd9880cbb30 0x7fd9880cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.594+0000 7fd990400700 1 --2- 172.21.15.89:0/1526025113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9880cd020 0x7fd9880cd4a0 unknown :-1 s=CLOSED pgs=2878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.594+0000 7fd990400700 1 -- 172.21.15.89:0/1526025113 >> 172.21.15.89:0/1526025113 conn(0x7fd9880b0d70 msgr2=0x7fd9880b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:00.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.595+0000 7fd990400700 1 -- 172.21.15.89:0/1526025113 shutdown_connections 2024-04-02T09:15:00.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.595+0000 7fd990400700 1 -- 172.21.15.89:0/1526025113 wait complete. 2024-04-02T09:15:00.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.596+0000 7fd990400700 1 Processor -- start 2024-04-02T09:15:00.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.596+0000 7fd990400700 1 -- start start 2024-04-02T09:15:00.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.597+0000 7fd990400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9880b2070 0x7fd988148420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:00.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.597+0000 7fd990400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd9880cbb30 0x7fd98815a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.597+0000 7fd990400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9880cd020 0x7fd98815abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.597+0000 7fd990400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880ac050 con 0x7fd9880cd020 2024-04-02T09:15:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.597+0000 7fd990400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880ad340 con 0x7fd9880b2070 2024-04-02T09:15:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.597+0000 7fd990400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880ac2a0 con 0x7fd9880cbb30 2024-04-02T09:15:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.598+0000 7fd98e400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9880cd020 0x7fd98815abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.598+0000 7fd98d000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd9880cbb30 0x7fd98815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.598+0000 7fd98e400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9880cd020 0x7fd98815abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54408/0 (socket says 172.21.15.89:54408) 2024-04-02T09:15:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.598+0000 7fd98da00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9880b2070 0x7fd988148420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.598+0000 7fd98e400700 1 -- 172.21.15.89:0/3071914513 learned_addr learned my addr 172.21.15.89:0/3071914513 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.598+0000 7fd98d000700 1 -- 172.21.15.89:0/3071914513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9880b2070 msgr2=0x7fd988148420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.598+0000 7fd98d000700 1 --2- 172.21.15.89:0/3071914513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9880b2070 0x7fd988148420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.598+0000 7fd98d000700 1 -- 172.21.15.89:0/3071914513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9880cd020 msgr2=0x7fd98815abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:00.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.598+0000 7fd98d000700 1 --2- 172.21.15.89:0/3071914513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9880cd020 0x7fd98815abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.598+0000 7fd98d000700 1 -- 172.21.15.89:0/3071914513 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd974009550 con 0x7fd9880cbb30 2024-04-02T09:15:00.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.598+0000 7fd98e400700 1 --2- 172.21.15.89:0/3071914513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9880cd020 0x7fd98815abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:15:00.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.598+0000 7fd98d000700 1 --2- 172.21.15.89:0/3071914513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd9880cbb30 0x7fd98815a670 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd974016000 tx=0x7fd974004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:00.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.599+0000 7fd985600700 1 -- 172.21.15.89:0/3071914513 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974004240 con 0x7fd9880cbb30 2024-04-02T09:15:00.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.599+0000 7fd985600700 1 -- 172.21.15.89:0/3071914513 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97401f070 con 0x7fd9880cbb30 2024-04-02T09:15:00.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.599+0000 7fd990400700 1 -- 172.21.15.89:0/3071914513 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9880ac580 con 0x7fd9880cbb30 2024-04-02T09:15:00.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.599+0000 7fd985600700 1 -- 172.21.15.89:0/3071914513 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9740036a0 con 0x7fd9880cbb30 2024-04-02T09:15:00.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.599+0000 7fd990400700 1 -- 172.21.15.89:0/3071914513 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9880acb40 con 0x7fd9880cbb30 2024-04-02T09:15:00.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.600+0000 7fd990400700 1 -- 172.21.15.89:0/3071914513 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9880ca7d0 con 0x7fd9880cbb30 2024-04-02T09:15:00.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.601+0000 7fd985600700 1 -- 172.21.15.89:0/3071914513 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fd974021030 con 0x7fd9880cbb30 2024-04-02T09:15:00.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.602+0000 7fd985600700 1 --2- 172.21.15.89:0/3071914513 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd96806ae40 0x7fd96806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:00.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.602+0000 7fd985600700 1 -- 172.21.15.89:0/3071914513 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7fd97408aaa0 con 0x7fd9880cbb30 2024-04-02T09:15:00.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.602+0000 7fd98da00700 1 -- 172.21.15.89:0/3071914513 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd96806ae40 msgr2=0x7fd96806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:00.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.602+0000 7fd98da00700 1 --2- 172.21.15.89:0/3071914513 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd96806ae40 0x7fd96806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:00.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.606+0000 7fd985600700 1 -- 172.21.15.89:0/3071914513 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd974055680 con 0x7fd9880cbb30 2024-04-02T09:15:00.763 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:00.765+0000 7f4bc690b240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:15:00.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.803+0000 7fd98da00700 1 -- 172.21.15.89:0/3071914513 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd96806ae40 msgr2=0x7fd96806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:00.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.803+0000 7fd98da00700 1 --2- 172.21.15.89:0/3071914513 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd96806ae40 0x7fd96806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:00.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.813+0000 7fd990400700 1 -- 172.21.15.89:0/3071914513 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd98815bda0 con 0x7fd9880cbb30 2024-04-02T09:15:00.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.813+0000 7fd985600700 1 -- 172.21.15.89:0/3071914513 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v266) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd974058ca0 con 0x7fd9880cbb30 2024-04-02T09:15:00.813 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:15:00.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.817+0000 7fd990400700 1 -- 172.21.15.89:0/3071914513 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd96806ae40 msgr2=0x7fd96806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:00.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.817+0000 7fd990400700 1 --2- 172.21.15.89:0/3071914513 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd96806ae40 0x7fd96806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.817+0000 7fd990400700 1 -- 172.21.15.89:0/3071914513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd9880cbb30 msgr2=0x7fd98815a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:00.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.817+0000 7fd990400700 1 --2- 172.21.15.89:0/3071914513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd9880cbb30 0x7fd98815a670 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd974016000 tx=0x7fd974004750 comp rx=0 tx=0).stop 2024-04-02T09:15:00.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.817+0000 7fd990400700 1 -- 172.21.15.89:0/3071914513 shutdown_connections 2024-04-02T09:15:00.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.817+0000 7fd990400700 1 --2- 172.21.15.89:0/3071914513 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd96806ae40 0x7fd96806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.817+0000 7fd990400700 1 --2- 172.21.15.89:0/3071914513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9880b2070 0x7fd988148420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.817+0000 7fd990400700 1 --2- 172.21.15.89:0/3071914513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9880cd020 0x7fd98815abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.817+0000 7fd990400700 1 --2- 172.21.15.89:0/3071914513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd9880cbb30 0x7fd98815a670 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:00.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.818+0000 7fd990400700 1 -- 172.21.15.89:0/3071914513 >> 172.21.15.89:0/3071914513 conn(0x7fd9880b0d70 msgr2=0x7fd9880b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:00.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.818+0000 7fd990400700 1 -- 172.21.15.89:0/3071914513 shutdown_connections 2024-04-02T09:15:00.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:00.818+0000 7fd990400700 1 -- 172.21.15.89:0/3071914513 wait complete. 2024-04-02T09:15:00.819 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:00.819+0000 7f92319a2240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:15:00.843 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T09:15:01.019 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:01.020+0000 7f4bc690b240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:15:01.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.043+0000 7fe19ac00700 1 Processor -- start 2024-04-02T09:15:01.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.043+0000 7fe19ac00700 1 -- start start 2024-04-02T09:15:01.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.044+0000 7fe19ac00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe1940cbb30 0x7fe1940cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.044+0000 7fe19ac00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe1940cd020 0x7fe1940cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.045+0000 7fe19ac00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1940cc450 con 0x7fe1940b2070 2024-04-02T09:15:01.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.045+0000 7fe19ac00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1940cdae0 con 0x7fe1940cd020 2024-04-02T09:15:01.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.045+0000 7fe19ac00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1940cddc0 con 0x7fe1940cbb30 2024-04-02T09:15:01.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.045+0000 7fe193400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe1940cd020 0x7fe1940cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:01.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.045+0000 7fe193400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe1940cd020 0x7fe1940cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54416/0 (socket says 172.21.15.89:54416) 2024-04-02T09:15:01.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.045+0000 7fe193400700 1 -- 172.21.15.89:0/2170436389 learned_addr learned my addr 172.21.15.89:0/2170436389 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:01.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.045+0000 7fe193e00700 1 --2- 172.21.15.89:0/2170436389 >> v2:172.21.15.113:3300/0 conn(0x7fe1940cbb30 0x7fe1940cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:01.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.045+0000 7fe193400700 1 -- 172.21.15.89:0/2170436389 >> v2:172.21.15.113:3300/0 conn(0x7fe1940cbb30 msgr2=0x7fe1940cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:01.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.045+0000 7fe193400700 1 --2- 172.21.15.89:0/2170436389 >> v2:172.21.15.113:3300/0 conn(0x7fe1940cbb30 0x7fe1940cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.045+0000 7fe193400700 1 -- 172.21.15.89:0/2170436389 >> v1:172.21.15.89:6789/0 conn(0x7fe1940b2070 legacy=0x7fe19406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.045+0000 7fe193400700 1 -- 172.21.15.89:0/2170436389 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1940ce0a0 con 0x7fe1940cd020 2024-04-02T09:15:01.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.045+0000 7fe192a00700 1 -- 172.21.15.89:0/2170436389 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194638631 0 0) 0x7fe1940cc450 con 0x7fe1940b2070 2024-04-02T09:15:01.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.046+0000 7fe193400700 1 --2- 172.21.15.89:0/2170436389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1940cd020 0x7fe1940cd480 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7fe19406fd00 tx=0x7fe180009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a17041731c0463e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:01.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.046+0000 7fe192a00700 1 -- 172.21.15.89:0/2170436389 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe180011830 con 0x7fe1940cd020 2024-04-02T09:15:01.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.046+0000 7fe192a00700 1 -- 172.21.15.89:0/2170436389 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe180011990 con 0x7fe1940cd020 2024-04-02T09:15:01.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.046+0000 7fe192a00700 1 -- 172.21.15.89:0/2170436389 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe180011d20 con 0x7fe1940cd020 2024-04-02T09:15:01.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.047+0000 7fe19ac00700 1 -- 172.21.15.89:0/2170436389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1940cd020 msgr2=0x7fe1940cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.047+0000 7fe19ac00700 1 --2- 172.21.15.89:0/2170436389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1940cd020 0x7fe1940cd480 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7fe19406fd00 tx=0x7fe180009bd0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.047+0000 7fe19ac00700 1 -- 172.21.15.89:0/2170436389 shutdown_connections 2024-04-02T09:15:01.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.047+0000 7fe19ac00700 1 --2- 172.21.15.89:0/2170436389 >> v2:172.21.15.113:3300/0 conn(0x7fe1940cbb30 0x7fe1940cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.047+0000 7fe19ac00700 1 --2- 172.21.15.89:0/2170436389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1940cd020 0x7fe1940cd480 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.047+0000 7fe19ac00700 1 -- 172.21.15.89:0/2170436389 >> 172.21.15.89:0/2170436389 conn(0x7fe1940b0d70 msgr2=0x7fe1940b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:01.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.047+0000 7fe19ac00700 1 -- 172.21.15.89:0/2170436389 shutdown_connections 2024-04-02T09:15:01.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.047+0000 7fe19ac00700 1 -- 172.21.15.89:0/2170436389 wait complete. 2024-04-02T09:15:01.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.049+0000 7fe19ac00700 1 Processor -- start 2024-04-02T09:15:01.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.049+0000 7fe19ac00700 1 -- start start 2024-04-02T09:15:01.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.050+0000 7fe19ac00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1940b2070 0x7fe1941f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.050+0000 7fe19ac00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1940cbb30 0x7fe19415a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.050+0000 7fe19ac00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1940cd020 0x7fe19415d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.050+0000 7fe19ac00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1941f35d0 con 0x7fe1940b2070 2024-04-02T09:15:01.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.050+0000 7fe19ac00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe194155340 con 0x7fe1940cd020 2024-04-02T09:15:01.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.050+0000 7fe19ac00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe194155620 con 0x7fe1940cbb30 2024-04-02T09:15:01.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.050+0000 7fe193e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1940b2070 0x7fe1941f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:01.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.050+0000 7fe193e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1940b2070 0x7fe1941f3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54426/0 (socket says 172.21.15.89:54426) 2024-04-02T09:15:01.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.050+0000 7fe193e00700 1 -- 172.21.15.89:0/3607965039 learned_addr learned my addr 172.21.15.89:0/3607965039 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:01.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.050+0000 7fe193400700 1 --2- 172.21.15.89:0/3607965039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1940cbb30 0x7fe19415a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:01.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.050+0000 7fe198c00700 1 --2- 172.21.15.89:0/3607965039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1940cd020 0x7fe19415d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:01.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.051+0000 7fe193e00700 1 -- 172.21.15.89:0/3607965039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1940cbb30 msgr2=0x7fe19415a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.051+0000 7fe193e00700 1 --2- 172.21.15.89:0/3607965039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1940cbb30 0x7fe19415a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.051+0000 7fe193e00700 1 -- 172.21.15.89:0/3607965039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1940cd020 msgr2=0x7fe19415d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.051+0000 7fe193e00700 1 --2- 172.21.15.89:0/3607965039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1940cd020 0x7fe19415d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.051+0000 7fe193e00700 1 -- 172.21.15.89:0/3607965039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe180009550 con 0x7fe1940b2070 2024-04-02T09:15:01.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.051+0000 7fe193e00700 1 --2- 172.21.15.89:0/3607965039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1940b2070 0x7fe1941f3090 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7fe18400e870 tx=0x7fe18400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:01.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.051+0000 7fe18be00700 1 -- 172.21.15.89:0/3607965039 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184009590 con 0x7fe1940b2070 2024-04-02T09:15:01.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.051+0000 7fe18be00700 1 -- 172.21.15.89:0/3607965039 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe184013680 con 0x7fe1940b2070 2024-04-02T09:15:01.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.051+0000 7fe19ac00700 1 -- 172.21.15.89:0/3607965039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1940ac050 con 0x7fe1940b2070 2024-04-02T09:15:01.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.051+0000 7fe18be00700 1 -- 172.21.15.89:0/3607965039 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1840139f0 con 0x7fe1940b2070 2024-04-02T09:15:01.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.051+0000 7fe19ac00700 1 -- 172.21.15.89:0/3607965039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1940ac610 con 0x7fe1940b2070 2024-04-02T09:15:01.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.052+0000 7fe19ac00700 1 -- 172.21.15.89:0/3607965039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe194067720 con 0x7fe1940b2070 2024-04-02T09:15:01.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.053+0000 7fe18be00700 1 -- 172.21.15.89:0/3607965039 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fe184013b50 con 0x7fe1940b2070 2024-04-02T09:15:01.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.054+0000 7fe18be00700 1 --2- 172.21.15.89:0/3607965039 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe17406ae40 0x7fe17406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.054+0000 7fe18be00700 1 -- 172.21.15.89:0/3607965039 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7fe184089870 con 0x7fe1940b2070 2024-04-02T09:15:01.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.054+0000 7fe193400700 1 -- 172.21.15.89:0/3607965039 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe17406ae40 msgr2=0x7fe17406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:01.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.054+0000 7fe193400700 1 --2- 172.21.15.89:0/3607965039 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe17406ae40 0x7fe17406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:01.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.059+0000 7fe18be00700 1 -- 172.21.15.89:0/3607965039 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe184054460 con 0x7fe1940b2070 2024-04-02T09:15:01.095 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:01.096+0000 7f92319a2240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:15:01.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.255+0000 7fe193400700 1 -- 172.21.15.89:0/3607965039 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe17406ae40 msgr2=0x7fe17406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:01.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.255+0000 7fe193400700 1 --2- 172.21.15.89:0/3607965039 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe17406ae40 0x7fe17406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:01.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.267+0000 7fe19ac00700 1 -- 172.21.15.89:0/3607965039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe1940ce2e0 con 0x7fe1940b2070 2024-04-02T09:15:01.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.267+0000 7fe18be00700 1 -- 172.21.15.89:0/3607965039 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v266) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe184057a80 con 0x7fe1940b2070 2024-04-02T09:15:01.267 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:15:01.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.270+0000 7fe19ac00700 1 -- 172.21.15.89:0/3607965039 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe17406ae40 msgr2=0x7fe17406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:01.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.270+0000 7fe19ac00700 1 --2- 172.21.15.89:0/3607965039 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe17406ae40 0x7fe17406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.270+0000 7fe19ac00700 1 -- 172.21.15.89:0/3607965039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1940b2070 msgr2=0x7fe1941f3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.270+0000 7fe19ac00700 1 --2- 172.21.15.89:0/3607965039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1940b2070 0x7fe1941f3090 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7fe18400e870 tx=0x7fe18400eb80 comp rx=0 tx=0).stop 2024-04-02T09:15:01.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.270+0000 7fe19ac00700 1 -- 172.21.15.89:0/3607965039 shutdown_connections 2024-04-02T09:15:01.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.270+0000 7fe19ac00700 1 --2- 172.21.15.89:0/3607965039 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe17406ae40 0x7fe17406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.270+0000 7fe19ac00700 1 --2- 172.21.15.89:0/3607965039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1940cd020 0x7fe19415d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.270+0000 7fe19ac00700 1 --2- 172.21.15.89:0/3607965039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1940b2070 0x7fe1941f3090 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.270+0000 7fe19ac00700 1 --2- 172.21.15.89:0/3607965039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe1940cbb30 0x7fe19415a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.270+0000 7fe19ac00700 1 -- 172.21.15.89:0/3607965039 >> 172.21.15.89:0/3607965039 conn(0x7fe1940b0d70 msgr2=0x7fe1940aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:01.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.270+0000 7fe19ac00700 1 -- 172.21.15.89:0/3607965039 shutdown_connections 2024-04-02T09:15:01.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.270+0000 7fe19ac00700 1 -- 172.21.15.89:0/3607965039 wait complete. 2024-04-02T09:15:01.288 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T09:15:01.476 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:01.477+0000 7f4bc690b240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:15:01.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.486+0000 7f5af7e00700 1 Processor -- start 2024-04-02T09:15:01.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.486+0000 7f5af7e00700 1 -- start start 2024-04-02T09:15:01.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.488+0000 7f5af7e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5af0160d70 0x7f5af00b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.488+0000 7f5af7e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af0161150 con 0x7f5af015ef70 2024-04-02T09:15:01.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.488+0000 7f5af7e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af006dba0 con 0x7f5af006a790 2024-04-02T09:15:01.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.488+0000 7f5af7e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af00b4cd0 con 0x7f5af0160d70 2024-04-02T09:15:01.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.488+0000 7f5af4a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5af015ef70 0x7f5af015e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59940/0 (socket says 172.21.15.89:59940) 2024-04-02T09:15:01.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.488+0000 7f5af4a00700 1 -- 172.21.15.89:0/1304550263 learned_addr learned my addr 172.21.15.89:0/1304550263 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:01.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.488+0000 7f5af5400700 1 --2- 172.21.15.89:0/1304550263 >> v2:172.21.15.113:3300/0 conn(0x7f5af0160d70 0x7f5af00b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:01.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.488+0000 7f5aefe00700 1 -- 172.21.15.89:0/1304550263 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204298668 0 0) 0x7f5af0161150 con 0x7f5af015ef70 2024-04-02T09:15:01.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.488+0000 7f5aefe00700 1 -- 172.21.15.89:0/1304550263 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad80035b0 con 0x7f5af015ef70 2024-04-02T09:15:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.489+0000 7f5af5400700 1 -- 172.21.15.89:0/1304550263 >> v1:172.21.15.113:6789/0 conn(0x7f5af006a790 legacy=0x7f5af006e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.489+0000 7f5aefe00700 1 -- 172.21.15.89:0/1304550263 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482696770 0 0) 0x7f5af006dba0 con 0x7f5af006a790 2024-04-02T09:15:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.489+0000 7f5af5400700 1 -- 172.21.15.89:0/1304550263 >> v1:172.21.15.89:6789/0 conn(0x7f5af015ef70 legacy=0x7f5af015e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.489+0000 7f5af5400700 1 -- 172.21.15.89:0/1304550263 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af00b4ee0 con 0x7f5af0160d70 2024-04-02T09:15:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.489+0000 7f5af5400700 1 --2- 172.21.15.89:0/1304550263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5af0160d70 0x7f5af00b3200 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009840 tx=0x7f5ae4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b77508d5eb5cd93b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.489+0000 7f5aefe00700 1 -- 172.21.15.89:0/1304550263 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4011830 con 0x7f5af0160d70 2024-04-02T09:15:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.489+0000 7f5aefe00700 1 -- 172.21.15.89:0/1304550263 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae4011990 con 0x7f5af0160d70 2024-04-02T09:15:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.489+0000 7f5aefe00700 1 -- 172.21.15.89:0/1304550263 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4011d00 con 0x7f5af0160d70 2024-04-02T09:15:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.490+0000 7f5af7e00700 1 -- 172.21.15.89:0/1304550263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5af0160d70 msgr2=0x7f5af00b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.490+0000 7f5af7e00700 1 --2- 172.21.15.89:0/1304550263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5af0160d70 0x7f5af00b3200 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009840 tx=0x7f5ae4009b50 comp rx=0 tx=0).stop 2024-04-02T09:15:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.490+0000 7f5af7e00700 1 -- 172.21.15.89:0/1304550263 shutdown_connections 2024-04-02T09:15:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.490+0000 7f5af7e00700 1 --2- 172.21.15.89:0/1304550263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5af0160d70 0x7f5af00b3200 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.490+0000 7f5af7e00700 1 -- 172.21.15.89:0/1304550263 >> 172.21.15.89:0/1304550263 conn(0x7f5af00b5720 msgr2=0x7f5af00b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.490+0000 7f5af7e00700 1 -- 172.21.15.89:0/1304550263 shutdown_connections 2024-04-02T09:15:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.490+0000 7f5af7e00700 1 -- 172.21.15.89:0/1304550263 wait complete. 2024-04-02T09:15:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.492+0000 7f5af7e00700 1 Processor -- start 2024-04-02T09:15:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.492+0000 7f5af7e00700 1 -- start start 2024-04-02T09:15:01.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.493+0000 7f5af7e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5af006a790 0x7f5af0158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.493+0000 7f5af7e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5af015ef70 0x7f5af0152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.493+0000 7f5af7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5af0160d70 0x7f5af0155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.493+0000 7f5af7e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af0152890 con 0x7f5af0160d70 2024-04-02T09:15:01.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.493+0000 7f5af7e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00ad280 con 0x7f5af015ef70 2024-04-02T09:15:01.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.493+0000 7f5af7e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af0157570 con 0x7f5af006a790 2024-04-02T09:15:01.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.493+0000 7f5af5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5af0160d70 0x7f5af0155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:01.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.493+0000 7f5af4a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5af015ef70 0x7f5af0152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:01.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.493+0000 7f5af5400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5af006a790 0x7f5af0158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:01.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.493+0000 7f5af5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5af0160d70 0x7f5af0155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54428/0 (socket says 172.21.15.89:54428) 2024-04-02T09:15:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.493+0000 7f5af5e00700 1 -- 172.21.15.89:0/987091936 learned_addr learned my addr 172.21.15.89:0/987091936 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.494+0000 7f5af4a00700 1 -- 172.21.15.89:0/987091936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5af006a790 msgr2=0x7f5af0158610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.494+0000 7f5af4a00700 1 --2- 172.21.15.89:0/987091936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5af006a790 0x7f5af0158610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.494+0000 7f5af4a00700 1 -- 172.21.15.89:0/987091936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5af0160d70 msgr2=0x7f5af0155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.494+0000 7f5af4a00700 1 --2- 172.21.15.89:0/987091936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5af0160d70 0x7f5af0155510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.494+0000 7f5af4a00700 1 -- 172.21.15.89:0/987091936 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae40094a0 con 0x7f5af015ef70 2024-04-02T09:15:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.494+0000 7f5af5400700 1 --2- 172.21.15.89:0/987091936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5af006a790 0x7f5af0158610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:15:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.494+0000 7f5af4a00700 1 --2- 172.21.15.89:0/987091936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5af015ef70 0x7f5af0152350 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f5af015f3b0 tx=0x7f5adc00c700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:01.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.494+0000 7f5aed600700 1 -- 172.21.15.89:0/987091936 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc005b70 con 0x7f5af015ef70 2024-04-02T09:15:01.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.494+0000 7f5aed600700 1 -- 172.21.15.89:0/987091936 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc005cd0 con 0x7f5af015ef70 2024-04-02T09:15:01.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.494+0000 7f5af7e00700 1 -- 172.21.15.89:0/987091936 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af0157850 con 0x7f5af015ef70 2024-04-02T09:15:01.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.494+0000 7f5aed600700 1 -- 172.21.15.89:0/987091936 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc0072f0 con 0x7f5af015ef70 2024-04-02T09:15:01.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.494+0000 7f5af7e00700 1 -- 172.21.15.89:0/987091936 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af0157e10 con 0x7f5af015ef70 2024-04-02T09:15:01.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.495+0000 7f5af7e00700 1 -- 172.21.15.89:0/987091936 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af0067720 con 0x7f5af015ef70 2024-04-02T09:15:01.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.496+0000 7f5aed600700 1 -- 172.21.15.89:0/987091936 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f5adc007450 con 0x7f5af015ef70 2024-04-02T09:15:01.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.497+0000 7f5aed600700 1 --2- 172.21.15.89:0/987091936 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ad006ae40 0x7f5ad006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.497+0000 7f5aed600700 1 -- 172.21.15.89:0/987091936 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f5adc088d90 con 0x7f5af015ef70 2024-04-02T09:15:01.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.497+0000 7f5af5400700 1 -- 172.21.15.89:0/987091936 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ad006ae40 msgr2=0x7f5ad006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:01.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.497+0000 7f5af5400700 1 --2- 172.21.15.89:0/987091936 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ad006ae40 0x7f5ad006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:01.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.506+0000 7f5aed600700 1 -- 172.21.15.89:0/987091936 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5adc053850 con 0x7f5af015ef70 2024-04-02T09:15:01.574 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:01.575+0000 7f92319a2240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:15:01.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.698+0000 7f5af5400700 1 -- 172.21.15.89:0/987091936 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ad006ae40 msgr2=0x7f5ad006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:01.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.698+0000 7f5af5400700 1 --2- 172.21.15.89:0/987091936 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ad006ae40 0x7f5ad006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:01.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.711+0000 7f5af7e00700 1 -- 172.21.15.89:0/987091936 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5af00d10c0 con 0x7f5af015ef70 2024-04-02T09:15:01.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.712+0000 7f5aed600700 1 -- 172.21.15.89:0/987091936 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v266) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5adc017030 con 0x7f5af015ef70 2024-04-02T09:15:01.711 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:15:01.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.714+0000 7f5af7e00700 1 -- 172.21.15.89:0/987091936 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ad006ae40 msgr2=0x7f5ad006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:01.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.714+0000 7f5af7e00700 1 --2- 172.21.15.89:0/987091936 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ad006ae40 0x7f5ad006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.714+0000 7f5af7e00700 1 -- 172.21.15.89:0/987091936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5af015ef70 msgr2=0x7f5af0152350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.714+0000 7f5af7e00700 1 --2- 172.21.15.89:0/987091936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5af015ef70 0x7f5af0152350 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f5af015f3b0 tx=0x7f5adc00c700 comp rx=0 tx=0).stop 2024-04-02T09:15:01.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.714+0000 7f5af7e00700 1 -- 172.21.15.89:0/987091936 shutdown_connections 2024-04-02T09:15:01.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.714+0000 7f5af7e00700 1 --2- 172.21.15.89:0/987091936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5af0160d70 0x7f5af0155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.714+0000 7f5af7e00700 1 --2- 172.21.15.89:0/987091936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5af006a790 0x7f5af0158610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.714+0000 7f5af7e00700 1 --2- 172.21.15.89:0/987091936 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5ad006ae40 0x7f5ad006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.714+0000 7f5af7e00700 1 --2- 172.21.15.89:0/987091936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5af015ef70 0x7f5af0152350 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.714+0000 7f5af7e00700 1 -- 172.21.15.89:0/987091936 >> 172.21.15.89:0/987091936 conn(0x7f5af00b5720 msgr2=0x7f5af0152b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:01.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.714+0000 7f5af7e00700 1 -- 172.21.15.89:0/987091936 shutdown_connections 2024-04-02T09:15:01.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.714+0000 7f5af7e00700 1 -- 172.21.15.89:0/987091936 wait complete. 2024-04-02T09:15:01.732 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T09:15:01.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.928+0000 7fc2a4a00700 1 Processor -- start 2024-04-02T09:15:01.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.929+0000 7fc2a4a00700 1 -- start start 2024-04-02T09:15:01.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.930+0000 7fc2a4a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc2a015dad0 0x7fc2a00ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.930+0000 7fc2a4a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc2a00d1630 0x7fc2a00d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.930+0000 7fc2a4a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a015f820 con 0x7fc2a00d2000 2024-04-02T09:15:01.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.930+0000 7fc2a4a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a00cfcc0 con 0x7fc2a00d1630 2024-04-02T09:15:01.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.930+0000 7fc2a4a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a00cd9f0 con 0x7fc2a015dad0 2024-04-02T09:15:01.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.930+0000 7fc29d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc2a00d1630 0x7fc2a00d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:01.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.930+0000 7fc29d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc2a00d1630 0x7fc2a00d1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54430/0 (socket says 172.21.15.89:54430) 2024-04-02T09:15:01.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.930+0000 7fc29d400700 1 -- 172.21.15.89:0/1872236404 learned_addr learned my addr 172.21.15.89:0/1872236404 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:01.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.930+0000 7fc29de00700 1 --2- 172.21.15.89:0/1872236404 >> v2:172.21.15.113:3300/0 conn(0x7fc2a015dad0 0x7fc2a00ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:01.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.930+0000 7fc29d400700 1 -- 172.21.15.89:0/1872236404 >> v2:172.21.15.113:3300/0 conn(0x7fc2a015dad0 msgr2=0x7fc2a00ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:01.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.930+0000 7fc29d400700 1 --2- 172.21.15.89:0/1872236404 >> v2:172.21.15.113:3300/0 conn(0x7fc2a015dad0 0x7fc2a00ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.930+0000 7fc29d400700 1 -- 172.21.15.89:0/1872236404 >> v1:172.21.15.113:6789/0 conn(0x7fc2a00d2000 legacy=0x7fc2a015deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.930+0000 7fc29d400700 1 -- 172.21.15.89:0/1872236404 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a00d2430 con 0x7fc2a00d1630 2024-04-02T09:15:01.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.930+0000 7fc29de00700 1 --2- 172.21.15.89:0/1872236404 >> v2:172.21.15.113:3300/0 conn(0x7fc2a015dad0 0x7fc2a00ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:01.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.931+0000 7fc29d400700 1 --2- 172.21.15.89:0/1872236404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a00d1630 0x7fc2a00d1ac0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7fc2a008d380 tx=0x7fc28c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a5570e9557a7ca9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:01.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.931+0000 7fc29ca00700 1 -- 172.21.15.89:0/1872236404 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c00b870 con 0x7fc2a00d1630 2024-04-02T09:15:01.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.931+0000 7fc29ca00700 1 -- 172.21.15.89:0/1872236404 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28c00b9d0 con 0x7fc2a00d1630 2024-04-02T09:15:01.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.931+0000 7fc29ca00700 1 -- 172.21.15.89:0/1872236404 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c00bd60 con 0x7fc2a00d1630 2024-04-02T09:15:01.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.931+0000 7fc2a4a00700 1 -- 172.21.15.89:0/1872236404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a00d1630 msgr2=0x7fc2a00d1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.931+0000 7fc2a4a00700 1 --2- 172.21.15.89:0/1872236404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a00d1630 0x7fc2a00d1ac0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7fc2a008d380 tx=0x7fc28c009bd0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.932+0000 7fc2a4a00700 1 -- 172.21.15.89:0/1872236404 shutdown_connections 2024-04-02T09:15:01.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.932+0000 7fc2a4a00700 1 --2- 172.21.15.89:0/1872236404 >> v2:172.21.15.113:3300/0 conn(0x7fc2a015dad0 0x7fc2a00ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.932+0000 7fc2a4a00700 1 --2- 172.21.15.89:0/1872236404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a00d1630 0x7fc2a00d1ac0 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.932+0000 7fc2a4a00700 1 -- 172.21.15.89:0/1872236404 >> 172.21.15.89:0/1872236404 conn(0x7fc2a008aff0 msgr2=0x7fc2a00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:01.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.932+0000 7fc2a4a00700 1 -- 172.21.15.89:0/1872236404 shutdown_connections 2024-04-02T09:15:01.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.932+0000 7fc2a4a00700 1 -- 172.21.15.89:0/1872236404 wait complete. 2024-04-02T09:15:01.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.933+0000 7fc2a4a00700 1 Processor -- start 2024-04-02T09:15:01.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.933+0000 7fc2a4a00700 1 -- start start 2024-04-02T09:15:01.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.934+0000 7fc2a4a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a00d1630 0x7fc2a01e9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.934+0000 7fc2a4a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a00d2000 0x7fc2a00b7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.934+0000 7fc2a4a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2a015dad0 0x7fc2a01505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.934+0000 7fc2a4a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a00ca960 con 0x7fc2a00d1630 2024-04-02T09:15:01.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.934+0000 7fc2a4a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a00cac40 con 0x7fc2a015dad0 2024-04-02T09:15:01.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.934+0000 7fc2a4a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a00caf20 con 0x7fc2a00d2000 2024-04-02T09:15:01.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.935+0000 7fc29de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a00d1630 0x7fc2a01e9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:01.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.935+0000 7fc29d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a00d2000 0x7fc2a00b7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.935+0000 7fc29de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a00d1630 0x7fc2a01e9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54438/0 (socket says 172.21.15.89:54438) 2024-04-02T09:15:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.935+0000 7fc29de00700 1 -- 172.21.15.89:0/2105507096 learned_addr learned my addr 172.21.15.89:0/2105507096 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.935+0000 7fc29e800700 1 --2- 172.21.15.89:0/2105507096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2a015dad0 0x7fc2a01505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.935+0000 7fc29de00700 1 -- 172.21.15.89:0/2105507096 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a00d2000 msgr2=0x7fc2a00b7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.935+0000 7fc29de00700 1 --2- 172.21.15.89:0/2105507096 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a00d2000 0x7fc2a00b7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.935+0000 7fc29de00700 1 -- 172.21.15.89:0/2105507096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2a015dad0 msgr2=0x7fc2a01505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:01.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.935+0000 7fc29de00700 1 --2- 172.21.15.89:0/2105507096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2a015dad0 0x7fc2a01505f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:01.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.935+0000 7fc29de00700 1 -- 172.21.15.89:0/2105507096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c009550 con 0x7fc2a00d1630 2024-04-02T09:15:01.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.935+0000 7fc29d400700 1 --2- 172.21.15.89:0/2105507096 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a00d2000 0x7fc2a00b7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:15:01.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.935+0000 7fc29de00700 1 --2- 172.21.15.89:0/2105507096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a00d1630 0x7fc2a01e9140 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7fc29000e810 tx=0x7fc29000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:01.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.936+0000 7fc296000700 1 -- 172.21.15.89:0/2105507096 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29000c020 con 0x7fc2a00d1630 2024-04-02T09:15:01.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.936+0000 7fc296000700 1 -- 172.21.15.89:0/2105507096 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc290004250 con 0x7fc2a00d1630 2024-04-02T09:15:01.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.936+0000 7fc296000700 1 -- 172.21.15.89:0/2105507096 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2900036a0 con 0x7fc2a00d1630 2024-04-02T09:15:01.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.936+0000 7fc2a4a00700 1 -- 172.21.15.89:0/2105507096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a00cb200 con 0x7fc2a00d1630 2024-04-02T09:15:01.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.936+0000 7fc2a4a00700 1 -- 172.21.15.89:0/2105507096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a00ca5f0 con 0x7fc2a00d1630 2024-04-02T09:15:01.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.937+0000 7fc2a4a00700 1 -- 172.21.15.89:0/2105507096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a00cb850 con 0x7fc2a00d1630 2024-04-02T09:15:01.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.938+0000 7fc296000700 1 -- 172.21.15.89:0/2105507096 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fc290003800 con 0x7fc2a00d1630 2024-04-02T09:15:01.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.939+0000 7fc296000700 1 --2- 172.21.15.89:0/2105507096 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc28006ae40 0x7fc28006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:01.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.939+0000 7fc296000700 1 -- 172.21.15.89:0/2105507096 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7fc290089470 con 0x7fc2a00d1630 2024-04-02T09:15:01.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.939+0000 7fc29d400700 1 -- 172.21.15.89:0/2105507096 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc28006ae40 msgr2=0x7fc28006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:01.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.939+0000 7fc29d400700 1 --2- 172.21.15.89:0/2105507096 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc28006ae40 0x7fc28006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:01.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:01.943+0000 7fc296000700 1 -- 172.21.15.89:0/2105507096 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc290053eb0 con 0x7fc2a00d1630 2024-04-02T09:15:02.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.140+0000 7fc29d400700 1 -- 172.21.15.89:0/2105507096 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc28006ae40 msgr2=0x7fc28006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:02.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.140+0000 7fc29d400700 1 --2- 172.21.15.89:0/2105507096 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc28006ae40 0x7fc28006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:02.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.167+0000 7fc2a4a00700 1 -- 172.21.15.89:0/2105507096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc2a0067720 con 0x7fc2a00d1630 2024-04-02T09:15:02.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.167+0000 7fc296000700 1 -- 172.21.15.89:0/2105507096 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v266) v1 ==== 81+0+577 (secure 0 0 0) 0x7fc2900574d0 con 0x7fc2a00d1630 2024-04-02T09:15:02.167 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/422001866 2024-04-03T09:14:55.473381+0000 2024-04-02T09:15:02.167 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3534660524 2024-04-03T09:14:55.473381+0000 2024-04-02T09:15:02.167 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2610551691 2024-04-03T09:14:55.473381+0000 2024-04-02T09:15:02.167 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1604459128 2024-04-03T09:14:55.473381+0000 2024-04-02T09:15:02.167 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3010462327 2024-04-03T09:14:55.473381+0000 2024-04-02T09:15:02.167 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3737296061 2024-04-03T09:14:55.473381+0000 2024-04-02T09:15:02.167 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1785135719 2024-04-03T09:14:55.473381+0000 2024-04-02T09:15:02.167 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/108855 2024-04-03T09:14:55.473381+0000 2024-04-02T09:15:02.167 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/108855 2024-04-03T09:14:55.473381+0000 2024-04-02T09:15:02.167 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1017424418 2024-04-03T09:14:55.473381+0000 2024-04-02T09:15:02.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.169+0000 7fc2a4a00700 1 -- 172.21.15.89:0/2105507096 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc28006ae40 msgr2=0x7fc28006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:02.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.169+0000 7fc2a4a00700 1 --2- 172.21.15.89:0/2105507096 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc28006ae40 0x7fc28006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:02.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.169+0000 7fc2a4a00700 1 -- 172.21.15.89:0/2105507096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a00d1630 msgr2=0x7fc2a01e9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:02.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.169+0000 7fc2a4a00700 1 --2- 172.21.15.89:0/2105507096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a00d1630 0x7fc2a01e9140 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7fc29000e810 tx=0x7fc29000eb20 comp rx=0 tx=0).stop 2024-04-02T09:15:02.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.169+0000 7fc2a4a00700 1 -- 172.21.15.89:0/2105507096 shutdown_connections 2024-04-02T09:15:02.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.169+0000 7fc2a4a00700 1 --2- 172.21.15.89:0/2105507096 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc28006ae40 0x7fc28006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:02.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.169+0000 7fc2a4a00700 1 --2- 172.21.15.89:0/2105507096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2a015dad0 0x7fc2a01505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:02.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.169+0000 7fc2a4a00700 1 --2- 172.21.15.89:0/2105507096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2a00d1630 0x7fc2a01e9140 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:02.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.169+0000 7fc2a4a00700 1 --2- 172.21.15.89:0/2105507096 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc2a00d2000 0x7fc2a00b7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:02.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.169+0000 7fc2a4a00700 1 -- 172.21.15.89:0/2105507096 >> 172.21.15.89:0/2105507096 conn(0x7fc2a008aff0 msgr2=0x7fc2a006c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:02.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.170+0000 7fc2a4a00700 1 -- 172.21.15.89:0/2105507096 shutdown_connections 2024-04-02T09:15:02.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.170+0000 7fc2a4a00700 1 -- 172.21.15.89:0/2105507096 wait complete. 2024-04-02T09:15:02.170 INFO:teuthology.orchestra.run.smithi089.stderr:listed 10 entries 2024-04-02T09:15:02.187 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T09:15:02.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:15:02.187 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T09:15:02.190 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:02.191+0000 7f4bc690b240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:15:02.260 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T09:15:02.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:15:02.260 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T09:15:02.282 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:02.284+0000 7f4bc690b240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:15:02.305 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:15:02.305 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:15:02.324 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:02.325+0000 7f92319a2240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:15:02.355 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:15:02.355 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:15:02.355 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:15:02.400 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:15:02.400 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:15:02.400 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:15:02.422 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:02.423+0000 7f92319a2240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:15:02.448 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:15:02.448 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:15:02.462 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:02.464+0000 7f4bc690b240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:15:02.592 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:02.593+0000 7f92319a2240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:15:02.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.646+0000 7f1aa1200700 1 Processor -- start 2024-04-02T09:15:02.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.646+0000 7f1aa1200700 1 -- start start 2024-04-02T09:15:02.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.647+0000 7f1aa1200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1a9c15d710 0x7f1a9c0d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:02.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.647+0000 7f1aa1200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a9c06bb40 con 0x7f1a9c06c650 2024-04-02T09:15:02.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.647+0000 7f1aa1200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a9c099b00 con 0x7f1a9c09aba0 2024-04-02T09:15:02.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.647+0000 7f1aa1200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a9c099de0 con 0x7f1a9c15d710 2024-04-02T09:15:02.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.647+0000 7f1a9a800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1a9c06c650 0x7f1a9c0ccbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59948/0 (socket says 172.21.15.89:59948) 2024-04-02T09:15:02.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.647+0000 7f1a9a800700 1 -- 172.21.15.89:0/1837599500 learned_addr learned my addr 172.21.15.89:0/1837599500 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:02.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.647+0000 7f1a99e00700 1 --2- 172.21.15.89:0/1837599500 >> v2:172.21.15.113:3300/0 conn(0x7f1a9c15d710 0x7f1a9c0d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:02.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.648+0000 7f1a98a00700 1 -- 172.21.15.89:0/1837599500 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242744199 0 0) 0x7f1a9c06bb40 con 0x7f1a9c06c650 2024-04-02T09:15:02.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.648+0000 7f1a98a00700 1 -- 172.21.15.89:0/1837599500 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a800035b0 con 0x7f1a9c06c650 2024-04-02T09:15:02.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.648+0000 7f1a98a00700 1 -- 172.21.15.89:0/1837599500 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279083608 0 0) 0x7f1a9c099b00 con 0x7f1a9c09aba0 2024-04-02T09:15:02.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.648+0000 7f1a98a00700 1 -- 172.21.15.89:0/1837599500 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a9c06bb40 con 0x7f1a9c09aba0 2024-04-02T09:15:02.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.648+0000 7f1a99e00700 1 -- 172.21.15.89:0/1837599500 >> v1:172.21.15.113:6789/0 conn(0x7f1a9c09aba0 legacy=0x7f1a9c06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:02.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.648+0000 7f1a99e00700 1 -- 172.21.15.89:0/1837599500 >> v1:172.21.15.89:6789/0 conn(0x7f1a9c06c650 legacy=0x7f1a9c0ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:02.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.648+0000 7f1a99e00700 1 -- 172.21.15.89:0/1837599500 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c09a0c0 con 0x7f1a9c15d710 2024-04-02T09:15:02.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.648+0000 7f1a99e00700 1 --2- 172.21.15.89:0/1837599500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a9c15d710 0x7f1a9c0d0130 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009840 tx=0x7f1a8c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3e2a56dda2e960a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:02.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.648+0000 7f1a98a00700 1 -- 172.21.15.89:0/1837599500 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c011830 con 0x7f1a9c15d710 2024-04-02T09:15:02.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.648+0000 7f1a98a00700 1 -- 172.21.15.89:0/1837599500 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8c011990 con 0x7f1a9c15d710 2024-04-02T09:15:02.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.648+0000 7f1a98a00700 1 -- 172.21.15.89:0/1837599500 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c011d20 con 0x7f1a9c15d710 2024-04-02T09:15:02.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.649+0000 7f1aa1200700 1 -- 172.21.15.89:0/1837599500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a9c15d710 msgr2=0x7f1a9c0d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:02.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.649+0000 7f1aa1200700 1 --2- 172.21.15.89:0/1837599500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a9c15d710 0x7f1a9c0d0130 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009840 tx=0x7f1a8c009b50 comp rx=0 tx=0).stop 2024-04-02T09:15:02.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.649+0000 7f1aa1200700 1 -- 172.21.15.89:0/1837599500 shutdown_connections 2024-04-02T09:15:02.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.649+0000 7f1aa1200700 1 --2- 172.21.15.89:0/1837599500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a9c15d710 0x7f1a9c0d0130 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:02.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.649+0000 7f1aa1200700 1 -- 172.21.15.89:0/1837599500 >> 172.21.15.89:0/1837599500 conn(0x7f1a9c0d13e0 msgr2=0x7f1a9c0d17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:02.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.649+0000 7f1aa1200700 1 -- 172.21.15.89:0/1837599500 shutdown_connections 2024-04-02T09:15:02.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.649+0000 7f1aa1200700 1 -- 172.21.15.89:0/1837599500 wait complete. 2024-04-02T09:15:02.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.652+0000 7f1aa1200700 1 Processor -- start 2024-04-02T09:15:02.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.652+0000 7f1aa1200700 1 -- start start 2024-04-02T09:15:02.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1aa1200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a9c06c650 0x7f1a9c0c4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:02.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1aa1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a9c09aba0 0x7f1a9c1506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:02.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1aa1200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a9c15d710 0x7f1a9c0b1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:02.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1aa1200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c0b52f0 con 0x7f1a9c09aba0 2024-04-02T09:15:02.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1aa1200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c0c5450 con 0x7f1a9c15d710 2024-04-02T09:15:02.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1aa1200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c0b5540 con 0x7f1a9c06c650 2024-04-02T09:15:02.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1a99400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a9c09aba0 0x7f1a9c1506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:02.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1a99400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a9c09aba0 0x7f1a9c1506b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54452/0 (socket says 172.21.15.89:54452) 2024-04-02T09:15:02.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1a99400700 1 -- 172.21.15.89:0/4238477119 learned_addr learned my addr 172.21.15.89:0/4238477119 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:02.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1a99e00700 1 --2- 172.21.15.89:0/4238477119 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a9c06c650 0x7f1a9c0c4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:02.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1a9a800700 1 --2- 172.21.15.89:0/4238477119 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a9c15d710 0x7f1a9c0b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:02.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1a99400700 1 -- 172.21.15.89:0/4238477119 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a9c06c650 msgr2=0x7f1a9c0c4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:02.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1a99400700 1 --2- 172.21.15.89:0/4238477119 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a9c06c650 0x7f1a9c0c4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:02.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1a99400700 1 -- 172.21.15.89:0/4238477119 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a9c15d710 msgr2=0x7f1a9c0b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:02.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1a99400700 1 --2- 172.21.15.89:0/4238477119 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a9c15d710 0x7f1a9c0b1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:02.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.654+0000 7f1a99400700 1 -- 172.21.15.89:0/4238477119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c0094a0 con 0x7f1a9c09aba0 2024-04-02T09:15:02.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.655+0000 7f1a99400700 1 --2- 172.21.15.89:0/4238477119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a9c09aba0 0x7f1a9c1506b0 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800c4f0 tx=0x7f1a8800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:02.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.655+0000 7f1a92000700 1 -- 172.21.15.89:0/4238477119 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a880052d0 con 0x7f1a9c09aba0 2024-04-02T09:15:02.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.655+0000 7f1a92000700 1 -- 172.21.15.89:0/4238477119 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a88005430 con 0x7f1a9c09aba0 2024-04-02T09:15:02.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.655+0000 7f1aa1200700 1 -- 172.21.15.89:0/4238477119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a9c0b5820 con 0x7f1a9c09aba0 2024-04-02T09:15:02.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.655+0000 7f1a92000700 1 -- 172.21.15.89:0/4238477119 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a88003880 con 0x7f1a9c09aba0 2024-04-02T09:15:02.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.655+0000 7f1aa1200700 1 -- 172.21.15.89:0/4238477119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a9c0b5de0 con 0x7f1a9c09aba0 2024-04-02T09:15:02.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.656+0000 7f1aa1200700 1 -- 172.21.15.89:0/4238477119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a9c0b6000 con 0x7f1a9c09aba0 2024-04-02T09:15:02.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.657+0000 7f1a92000700 1 -- 172.21.15.89:0/4238477119 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f1a880039e0 con 0x7f1a9c09aba0 2024-04-02T09:15:02.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.658+0000 7f1a92000700 1 --2- 172.21.15.89:0/4238477119 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1a7c06ae40 0x7f1a7c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:02.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.658+0000 7f1a92000700 1 -- 172.21.15.89:0/4238477119 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f1a88088cd0 con 0x7f1a9c09aba0 2024-04-02T09:15:02.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.659+0000 7f1a99e00700 1 -- 172.21.15.89:0/4238477119 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1a7c06ae40 msgr2=0x7f1a7c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:02.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.659+0000 7f1a99e00700 1 --2- 172.21.15.89:0/4238477119 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1a7c06ae40 0x7f1a7c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:02.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.662+0000 7f1a92000700 1 -- 172.21.15.89:0/4238477119 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1a880538d0 con 0x7f1a9c09aba0 2024-04-02T09:15:02.680 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:02.682+0000 7f4bc690b240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:15:02.772 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:02.773+0000 7f4bc690b240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:15:02.815 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:02.816+0000 7f92319a2240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:15:02.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.860+0000 7f1a99e00700 1 -- 172.21.15.89:0/4238477119 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1a7c06ae40 msgr2=0x7f1a7c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:02.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.860+0000 7f1a99e00700 1 --2- 172.21.15.89:0/4238477119 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1a7c06ae40 0x7f1a7c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:02.871 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:02.872+0000 7f4bc690b240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:15:02.909 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:02.910+0000 7f92319a2240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:15:02.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.914+0000 7f1aa1200700 1 -- 172.21.15.89:0/4238477119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a9c067720 con 0x7f1a9c09aba0 2024-04-02T09:15:02.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.915+0000 7f1a92000700 1 -- 172.21.15.89:0/4238477119 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+4756 (secure 0 0 0) 0x7f1a88017090 con 0x7f1a9c09aba0 2024-04-02T09:15:02.914 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:02.914 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":135,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10031,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1597421311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1597421311},{"type":"v1","addr":"172.21.15.113:6834","nonce":1597421311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":10034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3719211992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":3719211992},{"type":"v1","addr":"172.21.15.113:6835","nonce":3719211992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":10037,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3947526387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3947526387},{"type":"v1","addr":"172.21.15.89:6835","nonce":3947526387}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:14:34.351811+0000","modified":"2024-04-02T09:14:37.361983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12762":{"gid":12762,"name":"c","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2124958719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2124958719},{"type":"v1","addr":"172.21.15.89:6837","nonce":2124958719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-04-02T09:15:02.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.917+0000 7f1aa1200700 1 -- 172.21.15.89:0/4238477119 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1a7c06ae40 msgr2=0x7f1a7c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:02.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.917+0000 7f1aa1200700 1 --2- 172.21.15.89:0/4238477119 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1a7c06ae40 0x7f1a7c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:02.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.917+0000 7f1aa1200700 1 -- 172.21.15.89:0/4238477119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a9c09aba0 msgr2=0x7f1a9c1506b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:02.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.917+0000 7f1aa1200700 1 --2- 172.21.15.89:0/4238477119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a9c09aba0 0x7f1a9c1506b0 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800c4f0 tx=0x7f1a8800c800 comp rx=0 tx=0).stop 2024-04-02T09:15:02.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.917+0000 7f1aa1200700 1 -- 172.21.15.89:0/4238477119 shutdown_connections 2024-04-02T09:15:02.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.917+0000 7f1aa1200700 1 --2- 172.21.15.89:0/4238477119 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1a7c06ae40 0x7f1a7c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:02.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.917+0000 7f1aa1200700 1 --2- 172.21.15.89:0/4238477119 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a9c15d710 0x7f1a9c0b1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:02.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.917+0000 7f1aa1200700 1 --2- 172.21.15.89:0/4238477119 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a9c06c650 0x7f1a9c0c4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:02.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.917+0000 7f1aa1200700 1 --2- 172.21.15.89:0/4238477119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a9c09aba0 0x7f1a9c1506b0 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:02.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.917+0000 7f1aa1200700 1 -- 172.21.15.89:0/4238477119 >> 172.21.15.89:0/4238477119 conn(0x7f1a9c0d13e0 msgr2=0x7f1a9c0cde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:02.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.917+0000 7f1aa1200700 1 -- 172.21.15.89:0/4238477119 shutdown_connections 2024-04-02T09:15:02.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:02.917+0000 7f1aa1200700 1 -- 172.21.15.89:0/4238477119 wait complete. 2024-04-02T09:15:02.918 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 135 2024-04-02T09:15:02.936 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:15:02.969 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:02.970+0000 7f4bc690b240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:15:03.022 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:03.023+0000 7f92319a2240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:15:03.066 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:03.068+0000 7f4bc690b240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:15:03.126 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:03.127+0000 7f92319a2240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:15:03.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.135+0000 7fa9fcc00700 1 Processor -- start 2024-04-02T09:15:03.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.136+0000 7fa9fcc00700 1 -- start start 2024-04-02T09:15:03.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.137+0000 7fa9fcc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa9f8160c90 0x7fa9f80b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:03.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.137+0000 7fa9fcc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80ad970 0x7fa9f80b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:03.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.137+0000 7fa9fcc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f80ac4d0 con 0x7fa9f80718e0 2024-04-02T09:15:03.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.137+0000 7fa9fcc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f8071e50 con 0x7fa9f8160c90 2024-04-02T09:15:03.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.137+0000 7fa9fcc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f8072130 con 0x7fa9f80ad970 2024-04-02T09:15:03.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.137+0000 7fa9f5e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa9f8160c90 0x7fa9f80b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:03.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.137+0000 7fa9f5e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa9f8160c90 0x7fa9f80b20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54454/0 (socket says 172.21.15.89:54454) 2024-04-02T09:15:03.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.137+0000 7fa9f5e00700 1 -- 172.21.15.89:0/3332469936 learned_addr learned my addr 172.21.15.89:0/3332469936 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:03.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.137+0000 7fa9f6800700 1 --2- 172.21.15.89:0/3332469936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80ad970 0x7fa9f80b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:03.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.138+0000 7fa9f5e00700 1 -- 172.21.15.89:0/3332469936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80ad970 msgr2=0x7fa9f80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.138+0000 7fa9f5e00700 1 --2- 172.21.15.89:0/3332469936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80ad970 0x7fa9f80b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.138+0000 7fa9f5e00700 1 -- 172.21.15.89:0/3332469936 >> v1:172.21.15.113:6789/0 conn(0x7fa9f80718e0 legacy=0x7fa9f806cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.138+0000 7fa9f5e00700 1 -- 172.21.15.89:0/3332469936 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f8072410 con 0x7fa9f8160c90 2024-04-02T09:15:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.138+0000 7fa9f4a00700 1 -- 172.21.15.89:0/3332469936 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810986091 0 0) 0x7fa9f80ac4d0 con 0x7fa9f80718e0 2024-04-02T09:15:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.138+0000 7fa9f5e00700 1 --2- 172.21.15.89:0/3332469936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f8160c90 0x7fa9f80b20e0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80098f0 tx=0x7fa9e8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25a5de42c2e8a39 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.138+0000 7fa9f4a00700 1 -- 172.21.15.89:0/3332469936 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8011830 con 0x7fa9f8160c90 2024-04-02T09:15:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.138+0000 7fa9f4a00700 1 -- 172.21.15.89:0/3332469936 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e8011990 con 0x7fa9f8160c90 2024-04-02T09:15:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.138+0000 7fa9f4a00700 1 -- 172.21.15.89:0/3332469936 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8011d20 con 0x7fa9f8160c90 2024-04-02T09:15:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.139+0000 7fa9fcc00700 1 -- 172.21.15.89:0/3332469936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f8160c90 msgr2=0x7fa9f80b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.139+0000 7fa9fcc00700 1 --2- 172.21.15.89:0/3332469936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f8160c90 0x7fa9f80b20e0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80098f0 tx=0x7fa9e8009c00 comp rx=0 tx=0).stop 2024-04-02T09:15:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.139+0000 7fa9fcc00700 1 -- 172.21.15.89:0/3332469936 shutdown_connections 2024-04-02T09:15:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.139+0000 7fa9fcc00700 1 --2- 172.21.15.89:0/3332469936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80ad970 0x7fa9f80b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.139+0000 7fa9fcc00700 1 --2- 172.21.15.89:0/3332469936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f8160c90 0x7fa9f80b20e0 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.139+0000 7fa9fcc00700 1 -- 172.21.15.89:0/3332469936 >> 172.21.15.89:0/3332469936 conn(0x7fa9f80cf190 msgr2=0x7fa9f80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.139+0000 7fa9fcc00700 1 -- 172.21.15.89:0/3332469936 shutdown_connections 2024-04-02T09:15:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.139+0000 7fa9fcc00700 1 -- 172.21.15.89:0/3332469936 wait complete. 2024-04-02T09:15:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.141+0000 7fa9fcc00700 1 Processor -- start 2024-04-02T09:15:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.141+0000 7fa9fcc00700 1 -- start start 2024-04-02T09:15:03.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9fcc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80718e0 0x7fa9f8154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:03.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9fcc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f80ad970 0x7fa9f8154790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:03.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9fcc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f8160c90 0x7fa9f81f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:03.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9fcc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f8155ed0 con 0x7fa9f80ad970 2024-04-02T09:15:03.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9fcc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f8156560 con 0x7fa9f8160c90 2024-04-02T09:15:03.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9fcc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f8153cc0 con 0x7fa9f80718e0 2024-04-02T09:15:03.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9f5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f80ad970 0x7fa9f8154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9f5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f80ad970 0x7fa9f8154790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54470/0 (socket says 172.21.15.89:54470) 2024-04-02T09:15:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9f5400700 1 -- 172.21.15.89:0/874679764 learned_addr learned my addr 172.21.15.89:0/874679764 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9f5e00700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80718e0 0x7fa9f8154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9f6800700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f8160c90 0x7fa9f81f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9f5400700 1 -- 172.21.15.89:0/874679764 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80718e0 msgr2=0x7fa9f8154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9f5400700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80718e0 0x7fa9f8154250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9f5400700 1 -- 172.21.15.89:0/874679764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f8160c90 msgr2=0x7fa9f81f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9f5400700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f8160c90 0x7fa9f81f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9f5400700 1 -- 172.21.15.89:0/874679764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8009550 con 0x7fa9f80ad970 2024-04-02T09:15:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.142+0000 7fa9f5e00700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80718e0 0x7fa9f8154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:15:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.143+0000 7fa9f5400700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f80ad970 0x7fa9f8154790 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40011e0 tx=0x7fa9e400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.143+0000 7fa9ee000700 1 -- 172.21.15.89:0/874679764 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e40054e0 con 0x7fa9f80ad970 2024-04-02T09:15:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.143+0000 7fa9ee000700 1 -- 172.21.15.89:0/874679764 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e4003680 con 0x7fa9f80ad970 2024-04-02T09:15:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.143+0000 7fa9ee000700 1 -- 172.21.15.89:0/874679764 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4003a30 con 0x7fa9f80ad970 2024-04-02T09:15:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.143+0000 7fa9fcc00700 1 -- 172.21.15.89:0/874679764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f8154e30 con 0x7fa9f80ad970 2024-04-02T09:15:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.143+0000 7fa9fcc00700 1 -- 172.21.15.89:0/874679764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f81553f0 con 0x7fa9f80ad970 2024-04-02T09:15:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.144+0000 7fa9fcc00700 1 -- 172.21.15.89:0/874679764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f8067720 con 0x7fa9f80ad970 2024-04-02T09:15:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.145+0000 7fa9ee000700 1 -- 172.21.15.89:0/874679764 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fa9e4003b90 con 0x7fa9f80ad970 2024-04-02T09:15:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.146+0000 7fa9ee000700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa9d806ae40 0x7fa9d806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.146+0000 7fa9f5e00700 1 -- 172.21.15.89:0/874679764 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa9d806ae40 msgr2=0x7fa9d806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.146+0000 7fa9ee000700 1 -- 172.21.15.89:0/874679764 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7fa9e4088dc0 con 0x7fa9f80ad970 2024-04-02T09:15:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.146+0000 7fa9f5e00700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa9d806ae40 0x7fa9d806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:03.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.151+0000 7fa9ee000700 1 -- 172.21.15.89:0/874679764 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa9e40539c0 con 0x7fa9f80ad970 2024-04-02T09:15:03.229 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:03.230+0000 7f92319a2240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:15:03.280 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:03.281+0000 7f4bc690b240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:15:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.347+0000 7fa9f5e00700 1 -- 172.21.15.89:0/874679764 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa9d806ae40 msgr2=0x7fa9d806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.347+0000 7fa9f5e00700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa9d806ae40 0x7fa9d806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:03.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.357+0000 7fa9fcc00700 1 -- 172.21.15.89:0/874679764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9f80aefb0 con 0x7fa9f80ad970 2024-04-02T09:15:03.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.359+0000 7fa9ee000700 1 -- 172.21.15.89:0/874679764 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v266) v1 ==== 74+0+17360 (secure 0 0 0) 0x7fa9e4056fe0 con 0x7fa9f80ad970 2024-04-02T09:15:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:03.358 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":266,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:14:55.473418+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:14:32.705979+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-04-02T09:14:33.330752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/422001866":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/3534660524":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/2610551691":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/1604459128":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/3010462327":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/3737296061":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/1785135719":"2024-04-03T09:14:55.473381+0000","172.21.15.89:6833/108855":"2024-04-03T09:14:55.473381+0000","172.21.15.89:6832/108855":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/1017424418":"2024-04-03T09:14:55.473381+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:15:03.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.361+0000 7fa9fcc00700 1 -- 172.21.15.89:0/874679764 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa9d806ae40 msgr2=0x7fa9d806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:03.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.361+0000 7fa9fcc00700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa9d806ae40 0x7fa9d806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.361+0000 7fa9fcc00700 1 -- 172.21.15.89:0/874679764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f80ad970 msgr2=0x7fa9f8154790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:03.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.361+0000 7fa9fcc00700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f80ad970 0x7fa9f8154790 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40011e0 tx=0x7fa9e400ca30 comp rx=0 tx=0).stop 2024-04-02T09:15:03.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.361+0000 7fa9fcc00700 1 -- 172.21.15.89:0/874679764 shutdown_connections 2024-04-02T09:15:03.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.361+0000 7fa9fcc00700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa9d806ae40 0x7fa9d806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.361+0000 7fa9fcc00700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f8160c90 0x7fa9f81f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.361+0000 7fa9fcc00700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9f80718e0 0x7fa9f8154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.361+0000 7fa9fcc00700 1 --2- 172.21.15.89:0/874679764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9f80ad970 0x7fa9f8154790 unknown :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.361+0000 7fa9fcc00700 1 -- 172.21.15.89:0/874679764 >> 172.21.15.89:0/874679764 conn(0x7fa9f80cf190 msgr2=0x7fa9f806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:03.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.361+0000 7fa9fcc00700 1 -- 172.21.15.89:0/874679764 shutdown_connections 2024-04-02T09:15:03.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.361+0000 7fa9fcc00700 1 -- 172.21.15.89:0/874679764 wait complete. 2024-04-02T09:15:03.380 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T09:14:32.705979+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T09:14:33.330752+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '265', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:15:03.380 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:15:03.459 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:03.460+0000 7f92319a2240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:15:03.538 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:03.539+0000 7f4bc690b240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:15:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.587+0000 7fb08d400700 1 Processor -- start 2024-04-02T09:15:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.587+0000 7fb08d400700 1 -- start start 2024-04-02T09:15:03.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.588+0000 7fb08d400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb08809a750 0x7fb0880d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.588+0000 7fb08d400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0880d0970 con 0x7fb088099a60 2024-04-02T09:15:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.588+0000 7fb08d400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb08806c020 con 0x7fb08815d990 2024-04-02T09:15:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.588+0000 7fb08d400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0880c81a0 con 0x7fb08809a750 2024-04-02T09:15:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.588+0000 7fb086800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb088099a60 0x7fb088099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59958/0 (socket says 172.21.15.89:59958) 2024-04-02T09:15:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.588+0000 7fb086800700 1 -- 172.21.15.89:0/2559030887 learned_addr learned my addr 172.21.15.89:0/2559030887 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.589+0000 7fb087200700 1 --2- 172.21.15.89:0/2559030887 >> v2:172.21.15.113:3300/0 conn(0x7fb08809a750 0x7fb0880d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.589+0000 7fb085400700 1 -- 172.21.15.89:0/2559030887 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413442428 0 0) 0x7fb0880d0970 con 0x7fb088099a60 2024-04-02T09:15:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.589+0000 7fb085400700 1 -- 172.21.15.89:0/2559030887 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb06c0035b0 con 0x7fb088099a60 2024-04-02T09:15:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.589+0000 7fb085400700 1 -- 172.21.15.89:0/2559030887 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431103666 0 0) 0x7fb08806c020 con 0x7fb08815d990 2024-04-02T09:15:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.589+0000 7fb085400700 1 -- 172.21.15.89:0/2559030887 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0880d0970 con 0x7fb08815d990 2024-04-02T09:15:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.589+0000 7fb087200700 1 -- 172.21.15.89:0/2559030887 >> v1:172.21.15.113:6789/0 conn(0x7fb08815d990 legacy=0x7fb08806c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.589+0000 7fb087200700 1 -- 172.21.15.89:0/2559030887 >> v1:172.21.15.89:6789/0 conn(0x7fb088099a60 legacy=0x7fb088099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.589+0000 7fb087200700 1 -- 172.21.15.89:0/2559030887 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0880d0690 con 0x7fb08809a750 2024-04-02T09:15:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.589+0000 7fb087200700 1 --2- 172.21.15.89:0/2559030887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08809a750 0x7fb0880d0060 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb080009d90 tx=0x7fb08000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a754de18cc33410 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.590+0000 7fb085400700 1 -- 172.21.15.89:0/2559030887 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08000e070 con 0x7fb08809a750 2024-04-02T09:15:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.590+0000 7fb085400700 1 -- 172.21.15.89:0/2559030887 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08000bb20 con 0x7fb08809a750 2024-04-02T09:15:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.590+0000 7fb085400700 1 -- 172.21.15.89:0/2559030887 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080012430 con 0x7fb08809a750 2024-04-02T09:15:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.590+0000 7fb08d400700 1 -- 172.21.15.89:0/2559030887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08809a750 msgr2=0x7fb0880d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:03.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.590+0000 7fb08d400700 1 --2- 172.21.15.89:0/2559030887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08809a750 0x7fb0880d0060 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb080009d90 tx=0x7fb08000b330 comp rx=0 tx=0).stop 2024-04-02T09:15:03.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.590+0000 7fb08d400700 1 -- 172.21.15.89:0/2559030887 shutdown_connections 2024-04-02T09:15:03.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.590+0000 7fb08d400700 1 --2- 172.21.15.89:0/2559030887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08809a750 0x7fb0880d0060 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.590+0000 7fb08d400700 1 -- 172.21.15.89:0/2559030887 >> 172.21.15.89:0/2559030887 conn(0x7fb0880b2e20 msgr2=0x7fb0880b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:03.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.591+0000 7fb08d400700 1 -- 172.21.15.89:0/2559030887 shutdown_connections 2024-04-02T09:15:03.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.591+0000 7fb08d400700 1 -- 172.21.15.89:0/2559030887 wait complete. 2024-04-02T09:15:03.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.592+0000 7fb08d400700 1 Processor -- start 2024-04-02T09:15:03.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.592+0000 7fb08d400700 1 -- start start 2024-04-02T09:15:03.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.593+0000 7fb08d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb088099a60 0x7fb0880c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:03.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.593+0000 7fb08d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08809a750 0x7fb0880c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:03.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.593+0000 7fb08d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb08815d990 0x7fb08815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:03.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.593+0000 7fb08d400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880b4aa0 con 0x7fb088099a60 2024-04-02T09:15:03.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.593+0000 7fb08d400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08815b810 con 0x7fb08809a750 2024-04-02T09:15:03.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.593+0000 7fb08d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb08815baf0 con 0x7fb08815d990 2024-04-02T09:15:03.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.594+0000 7fb086800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb088099a60 0x7fb0880c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:03.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.594+0000 7fb086800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb088099a60 0x7fb0880c4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54472/0 (socket says 172.21.15.89:54472) 2024-04-02T09:15:03.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.594+0000 7fb086800700 1 -- 172.21.15.89:0/3938128721 learned_addr learned my addr 172.21.15.89:0/3938128721 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.594+0000 7fb087200700 1 --2- 172.21.15.89:0/3938128721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb08815d990 0x7fb08815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.594+0000 7fb085e00700 1 --2- 172.21.15.89:0/3938128721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08809a750 0x7fb0880c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.594+0000 7fb086800700 1 -- 172.21.15.89:0/3938128721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb08815d990 msgr2=0x7fb08815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.594+0000 7fb086800700 1 --2- 172.21.15.89:0/3938128721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb08815d990 0x7fb08815ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.594+0000 7fb086800700 1 -- 172.21.15.89:0/3938128721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08809a750 msgr2=0x7fb0880c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.594+0000 7fb086800700 1 --2- 172.21.15.89:0/3938128721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08809a750 0x7fb0880c52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.594+0000 7fb086800700 1 -- 172.21.15.89:0/3938128721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb080009a70 con 0x7fb088099a60 2024-04-02T09:15:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.594+0000 7fb086800700 1 --2- 172.21.15.89:0/3938128721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb088099a60 0x7fb0880c4d90 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7fb07800c700 tx=0x7fb07800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.595+0000 7fb07ea00700 1 -- 172.21.15.89:0/3938128721 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb078005d70 con 0x7fb088099a60 2024-04-02T09:15:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.595+0000 7fb07ea00700 1 -- 172.21.15.89:0/3938128721 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0780070d0 con 0x7fb088099a60 2024-04-02T09:15:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.595+0000 7fb07ea00700 1 -- 172.21.15.89:0/3938128721 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb078007440 con 0x7fb088099a60 2024-04-02T09:15:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.595+0000 7fb08d400700 1 -- 172.21.15.89:0/3938128721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0880b5100 con 0x7fb088099a60 2024-04-02T09:15:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.595+0000 7fb08d400700 1 -- 172.21.15.89:0/3938128721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0880b56c0 con 0x7fb088099a60 2024-04-02T09:15:03.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.596+0000 7fb08d400700 1 -- 172.21.15.89:0/3938128721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0880b58b0 con 0x7fb088099a60 2024-04-02T09:15:03.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.597+0000 7fb07ea00700 1 -- 172.21.15.89:0/3938128721 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fb0780075a0 con 0x7fb088099a60 2024-04-02T09:15:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.598+0000 7fb07ea00700 1 --2- 172.21.15.89:0/3938128721 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb06806ae40 0x7fb06806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.598+0000 7fb085e00700 1 -- 172.21.15.89:0/3938128721 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb06806ae40 msgr2=0x7fb06806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.598+0000 7fb07ea00700 1 -- 172.21.15.89:0/3938128721 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7fb078088d20 con 0x7fb088099a60 2024-04-02T09:15:03.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.598+0000 7fb085e00700 1 --2- 172.21.15.89:0/3938128721 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb06806ae40 0x7fb06806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:03.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.602+0000 7fb07ea00700 1 -- 172.21.15.89:0/3938128721 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb078053920 con 0x7fb088099a60 2024-04-02T09:15:03.726 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:03.727+0000 7f92319a2240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:15:03.730 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:03.731+0000 7f4bc690b240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:15:03.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.799+0000 7fb085e00700 1 -- 172.21.15.89:0/3938128721 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb06806ae40 msgr2=0x7fb06806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:03.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.799+0000 7fb085e00700 1 --2- 172.21.15.89:0/3938128721 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb06806ae40 0x7fb06806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:03.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.813+0000 7fb08d400700 1 -- 172.21.15.89:0/3938128721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb088067720 con 0x7fb088099a60 2024-04-02T09:15:03.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.813+0000 7fb07ea00700 1 -- 172.21.15.89:0/3938128721 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v266) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb078017020 con 0x7fb088099a60 2024-04-02T09:15:03.812 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:15:03.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.815+0000 7fb08d400700 1 -- 172.21.15.89:0/3938128721 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb06806ae40 msgr2=0x7fb06806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:03.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.815+0000 7fb08d400700 1 --2- 172.21.15.89:0/3938128721 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb06806ae40 0x7fb06806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.815+0000 7fb08d400700 1 -- 172.21.15.89:0/3938128721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb088099a60 msgr2=0x7fb0880c4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:03.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.815+0000 7fb08d400700 1 --2- 172.21.15.89:0/3938128721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb088099a60 0x7fb0880c4d90 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7fb07800c700 tx=0x7fb07800ca10 comp rx=0 tx=0).stop 2024-04-02T09:15:03.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.815+0000 7fb08d400700 1 -- 172.21.15.89:0/3938128721 shutdown_connections 2024-04-02T09:15:03.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.816+0000 7fb08d400700 1 --2- 172.21.15.89:0/3938128721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb08815d990 0x7fb08815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.816+0000 7fb08d400700 1 --2- 172.21.15.89:0/3938128721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb088099a60 0x7fb0880c4d90 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.816+0000 7fb08d400700 1 --2- 172.21.15.89:0/3938128721 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb06806ae40 0x7fb06806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.816+0000 7fb08d400700 1 --2- 172.21.15.89:0/3938128721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08809a750 0x7fb0880c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:03.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.816+0000 7fb08d400700 1 -- 172.21.15.89:0/3938128721 >> 172.21.15.89:0/3938128721 conn(0x7fb0880b2e20 msgr2=0x7fb0880cf510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:03.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.816+0000 7fb08d400700 1 -- 172.21.15.89:0/3938128721 shutdown_connections 2024-04-02T09:15:03.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:03.816+0000 7fb08d400700 1 -- 172.21.15.89:0/3938128721 wait complete. 2024-04-02T09:15:03.825 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:03.825+0000 7f4bc690b240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:15:03.833 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T09:15:03.930 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:03.932+0000 7f92319a2240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:15:04.028 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:04.029+0000 7f92319a2240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:15:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.030+0000 7f86ae400700 1 Processor -- start 2024-04-02T09:15:04.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.031+0000 7f86ae400700 1 -- start start 2024-04-02T09:15:04.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.032+0000 7f86ae400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f86a80d21f0 0x7f86a80ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.032+0000 7f86ae400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f86a80cec40 0x7f86a8161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.032+0000 7f86ae400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a80ce270 con 0x7f86a81611b0 2024-04-02T09:15:04.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.032+0000 7f86ae400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a815b600 con 0x7f86a80d21f0 2024-04-02T09:15:04.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.032+0000 7f86ae400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a80d2760 con 0x7f86a80cec40 2024-04-02T09:15:04.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.032+0000 7f86a6a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f86a80d21f0 0x7f86a80ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.032+0000 7f86a6a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f86a80d21f0 0x7f86a80ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54478/0 (socket says 172.21.15.89:54478) 2024-04-02T09:15:04.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.032+0000 7f86a6a00700 1 -- 172.21.15.89:0/3077702764 learned_addr learned my addr 172.21.15.89:0/3077702764 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:04.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.032+0000 7f86a7e00700 1 --2- 172.21.15.89:0/3077702764 >> v2:172.21.15.113:3300/0 conn(0x7f86a80cec40 0x7f86a8161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.033+0000 7f86a6a00700 1 -- 172.21.15.89:0/3077702764 >> v2:172.21.15.113:3300/0 conn(0x7f86a80cec40 msgr2=0x7f86a8161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:04.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.033+0000 7f86a6a00700 1 --2- 172.21.15.89:0/3077702764 >> v2:172.21.15.113:3300/0 conn(0x7f86a80cec40 0x7f86a8161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.033+0000 7f86a6a00700 1 -- 172.21.15.89:0/3077702764 >> v1:172.21.15.113:6789/0 conn(0x7f86a81611b0 legacy=0x7f86a815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:04.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.033+0000 7f86a6a00700 1 -- 172.21.15.89:0/3077702764 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a80d2a40 con 0x7f86a80d21f0 2024-04-02T09:15:04.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.033+0000 7f86a6a00700 1 --2- 172.21.15.89:0/3077702764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86a80d21f0 0x7f86a80ce700 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f86a80b5cb0 tx=0x7f8694009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a69dedb4fa4d0d6b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:04.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.033+0000 7f86a6000700 1 -- 172.21.15.89:0/3077702764 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8694011830 con 0x7f86a80d21f0 2024-04-02T09:15:04.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.033+0000 7f86a6000700 1 -- 172.21.15.89:0/3077702764 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8694011990 con 0x7f86a80d21f0 2024-04-02T09:15:04.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.033+0000 7f86a6000700 1 -- 172.21.15.89:0/3077702764 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8694011d20 con 0x7f86a80d21f0 2024-04-02T09:15:04.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.034+0000 7f86ae400700 1 -- 172.21.15.89:0/3077702764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86a80d21f0 msgr2=0x7f86a80ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:04.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.034+0000 7f86ae400700 1 --2- 172.21.15.89:0/3077702764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86a80d21f0 0x7f86a80ce700 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f86a80b5cb0 tx=0x7f8694009bd0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.034+0000 7f86ae400700 1 -- 172.21.15.89:0/3077702764 shutdown_connections 2024-04-02T09:15:04.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.034+0000 7f86ae400700 1 --2- 172.21.15.89:0/3077702764 >> v2:172.21.15.113:3300/0 conn(0x7f86a80cec40 0x7f86a8161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.034+0000 7f86ae400700 1 --2- 172.21.15.89:0/3077702764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86a80d21f0 0x7f86a80ce700 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.034+0000 7f86ae400700 1 -- 172.21.15.89:0/3077702764 >> 172.21.15.89:0/3077702764 conn(0x7f86a808aff0 msgr2=0x7f86a80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.034+0000 7f86ae400700 1 -- 172.21.15.89:0/3077702764 shutdown_connections 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.034+0000 7f86ae400700 1 -- 172.21.15.89:0/3077702764 wait complete. 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.036+0000 7f86ae400700 1 Processor -- start 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.036+0000 7f86ae400700 1 -- start start 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86ae400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86a80cec40 0x7f86a80b8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86ae400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86a80d21f0 0x7f86a8155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86ae400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86a81611b0 0x7f86a815da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86ae400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80ca870 con 0x7f86a80d21f0 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86ae400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a8156fb0 con 0x7f86a81611b0 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86ae400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80b8d60 con 0x7f86a80cec40 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86a7e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86a81611b0 0x7f86a815da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86a6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86a80d21f0 0x7f86a8155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86a7e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86a81611b0 0x7f86a815da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:50674/0 (socket says 172.21.15.89:50674) 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86a7e00700 1 -- 172.21.15.89:0/2773771082 learned_addr learned my addr 172.21.15.89:0/2773771082 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:04.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86a7400700 1 --2- 172.21.15.89:0/2773771082 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86a80cec40 0x7f86a80b8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86a7e00700 1 -- 172.21.15.89:0/2773771082 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86a80cec40 msgr2=0x7f86a80b8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:04.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86a7e00700 1 --2- 172.21.15.89:0/2773771082 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86a80cec40 0x7f86a80b8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86a7e00700 1 -- 172.21.15.89:0/2773771082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86a80d21f0 msgr2=0x7f86a8155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:04.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86a7e00700 1 --2- 172.21.15.89:0/2773771082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86a80d21f0 0x7f86a8155ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86a7e00700 1 -- 172.21.15.89:0/2773771082 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8694009550 con 0x7f86a81611b0 2024-04-02T09:15:04.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.037+0000 7f86a6a00700 1 --2- 172.21.15.89:0/2773771082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86a80d21f0 0x7f86a8155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:15:04.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.038+0000 7f86a7e00700 1 --2- 172.21.15.89:0/2773771082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86a81611b0 0x7f86a815da30 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f86a000efc0 tx=0x7f86a000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:04.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.038+0000 7f869f400700 1 -- 172.21.15.89:0/2773771082 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a0009ea0 con 0x7f86a81611b0 2024-04-02T09:15:04.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.038+0000 7f869f400700 1 -- 172.21.15.89:0/2773771082 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a000f040 con 0x7f86a81611b0 2024-04-02T09:15:04.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.038+0000 7f869f400700 1 -- 172.21.15.89:0/2773771082 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a000ced0 con 0x7f86a81611b0 2024-04-02T09:15:04.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.038+0000 7f86ae400700 1 -- 172.21.15.89:0/2773771082 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a80b9040 con 0x7f86a81611b0 2024-04-02T09:15:04.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.038+0000 7f86ae400700 1 -- 172.21.15.89:0/2773771082 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a80c73e0 con 0x7f86a81611b0 2024-04-02T09:15:04.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.039+0000 7f86ae400700 1 -- 172.21.15.89:0/2773771082 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a8067720 con 0x7f86a81611b0 2024-04-02T09:15:04.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.040+0000 7f869f400700 1 -- 172.21.15.89:0/2773771082 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f86a0014480 con 0x7f86a81611b0 2024-04-02T09:15:04.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.041+0000 7f869f400700 1 --2- 172.21.15.89:0/2773771082 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f868806ae40 0x7f868806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.041+0000 7f869f400700 1 -- 172.21.15.89:0/2773771082 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f86a008a340 con 0x7f86a81611b0 2024-04-02T09:15:04.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.041+0000 7f86a7400700 1 -- 172.21.15.89:0/2773771082 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f868806ae40 msgr2=0x7f868806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:04.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.041+0000 7f86a7400700 1 --2- 172.21.15.89:0/2773771082 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f868806ae40 0x7f868806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:04.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.046+0000 7f869f400700 1 -- 172.21.15.89:0/2773771082 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f86a0054d80 con 0x7f86a81611b0 2024-04-02T09:15:04.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.243+0000 7f86a7400700 1 -- 172.21.15.89:0/2773771082 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f868806ae40 msgr2=0x7f868806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:04.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.243+0000 7f86a7400700 1 --2- 172.21.15.89:0/2773771082 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f868806ae40 0x7f868806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:04.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.248+0000 7f86ae400700 1 -- 172.21.15.89:0/2773771082 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f86a80b5d00 con 0x7f86a81611b0 2024-04-02T09:15:04.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.249+0000 7f869f400700 1 -- 172.21.15.89:0/2773771082 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v266) v1 ==== 104+0+11 (secure 0 0 0) 0x7f86a00583a0 con 0x7f86a81611b0 2024-04-02T09:15:04.248 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:15:04.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.252+0000 7f86ae400700 1 -- 172.21.15.89:0/2773771082 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f868806ae40 msgr2=0x7f868806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:04.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.252+0000 7f86ae400700 1 --2- 172.21.15.89:0/2773771082 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f868806ae40 0x7f868806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.252+0000 7f86ae400700 1 -- 172.21.15.89:0/2773771082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86a81611b0 msgr2=0x7f86a815da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:04.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.252+0000 7f86ae400700 1 --2- 172.21.15.89:0/2773771082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86a81611b0 0x7f86a815da30 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f86a000efc0 tx=0x7f86a000c3a0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.252+0000 7f86ae400700 1 -- 172.21.15.89:0/2773771082 shutdown_connections 2024-04-02T09:15:04.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.252+0000 7f86ae400700 1 --2- 172.21.15.89:0/2773771082 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f868806ae40 0x7f868806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.252+0000 7f86ae400700 1 --2- 172.21.15.89:0/2773771082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86a81611b0 0x7f86a815da30 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.252+0000 7f86ae400700 1 --2- 172.21.15.89:0/2773771082 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86a80cec40 0x7f86a80b8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.252+0000 7f86ae400700 1 --2- 172.21.15.89:0/2773771082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86a80d21f0 0x7f86a8155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.252+0000 7f86ae400700 1 -- 172.21.15.89:0/2773771082 >> 172.21.15.89:0/2773771082 conn(0x7f86a808aff0 msgr2=0x7f86a806c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:04.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.252+0000 7f86ae400700 1 -- 172.21.15.89:0/2773771082 shutdown_connections 2024-04-02T09:15:04.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.252+0000 7f86ae400700 1 -- 172.21.15.89:0/2773771082 wait complete. 2024-04-02T09:15:04.269 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T09:15:04.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.470+0000 7f8e58000700 1 Processor -- start 2024-04-02T09:15:04.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.470+0000 7f8e58000700 1 -- start start 2024-04-02T09:15:04.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.471+0000 7f8e58000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8e50157f90 0x7f8e500b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.471+0000 7f8e58000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e500b3b30 0x7f8e50161890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.471+0000 7f8e58000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e500b3070 con 0x7f8e501613e0 2024-04-02T09:15:04.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.471+0000 7f8e58000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e50158370 con 0x7f8e50157f90 2024-04-02T09:15:04.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.471+0000 7f8e58000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e500b2710 con 0x7f8e500b3b30 2024-04-02T09:15:04.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.471+0000 7f8e54c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8e501613e0 0x7f8e500b3420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59968/0 (socket says 172.21.15.89:59968) 2024-04-02T09:15:04.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.471+0000 7f8e54c00700 1 -- 172.21.15.89:0/1535191120 learned_addr learned my addr 172.21.15.89:0/1535191120 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:04.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.471+0000 7f8e55600700 1 --2- 172.21.15.89:0/1535191120 >> v2:172.21.15.113:3300/0 conn(0x7f8e50157f90 0x7f8e500b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.471+0000 7f8e56000700 1 --2- 172.21.15.89:0/1535191120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e500b3b30 0x7f8e50161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.472+0000 7f8e4fe00700 1 -- 172.21.15.89:0/1535191120 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984484259 0 0) 0x7f8e3c001110 con 0x7f8e501613e0 2024-04-02T09:15:04.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.472+0000 7f8e4fe00700 1 -- 172.21.15.89:0/1535191120 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e380035b0 con 0x7f8e501613e0 2024-04-02T09:15:04.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.472+0000 7f8e56000700 1 -- 172.21.15.89:0/1535191120 >> v2:172.21.15.113:3300/0 conn(0x7f8e50157f90 msgr2=0x7f8e500b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:04.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.472+0000 7f8e56000700 1 --2- 172.21.15.89:0/1535191120 >> v2:172.21.15.113:3300/0 conn(0x7f8e50157f90 0x7f8e500b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.472+0000 7f8e56000700 1 -- 172.21.15.89:0/1535191120 >> v1:172.21.15.89:6789/0 conn(0x7f8e501613e0 legacy=0x7f8e500b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:04.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.472+0000 7f8e56000700 1 -- 172.21.15.89:0/1535191120 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e500b2070 con 0x7f8e500b3b30 2024-04-02T09:15:04.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.472+0000 7f8e55600700 1 --2- 172.21.15.89:0/1535191120 >> v2:172.21.15.113:3300/0 conn(0x7f8e50157f90 0x7f8e500b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:15:04.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.472+0000 7f8e56000700 1 --2- 172.21.15.89:0/1535191120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e500b3b30 0x7f8e50161890 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8e48009d90 tx=0x7f8e4800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4159778fac5351 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:04.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.473+0000 7f8e4fe00700 1 -- 172.21.15.89:0/1535191120 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4800e070 con 0x7f8e500b3b30 2024-04-02T09:15:04.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.473+0000 7f8e4fe00700 1 -- 172.21.15.89:0/1535191120 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4800bb20 con 0x7f8e500b3b30 2024-04-02T09:15:04.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.473+0000 7f8e4fe00700 1 -- 172.21.15.89:0/1535191120 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48012430 con 0x7f8e500b3b30 2024-04-02T09:15:04.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.473+0000 7f8e58000700 1 -- 172.21.15.89:0/1535191120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e500b3b30 msgr2=0x7f8e50161890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:04.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.473+0000 7f8e58000700 1 --2- 172.21.15.89:0/1535191120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e500b3b30 0x7f8e50161890 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8e48009d90 tx=0x7f8e4800b330 comp rx=0 tx=0).stop 2024-04-02T09:15:04.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.473+0000 7f8e58000700 1 -- 172.21.15.89:0/1535191120 shutdown_connections 2024-04-02T09:15:04.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.473+0000 7f8e58000700 1 --2- 172.21.15.89:0/1535191120 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e500b3b30 0x7f8e50161890 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.473+0000 7f8e58000700 1 --2- 172.21.15.89:0/1535191120 >> v2:172.21.15.113:3300/0 conn(0x7f8e50157f90 0x7f8e500b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.473+0000 7f8e58000700 1 -- 172.21.15.89:0/1535191120 >> 172.21.15.89:0/1535191120 conn(0x7f8e500b0d70 msgr2=0x7f8e500b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:04.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.473+0000 7f8e58000700 1 -- 172.21.15.89:0/1535191120 shutdown_connections 2024-04-02T09:15:04.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.473+0000 7f8e58000700 1 -- 172.21.15.89:0/1535191120 wait complete. 2024-04-02T09:15:04.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.475+0000 7f8e58000700 1 Processor -- start 2024-04-02T09:15:04.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.475+0000 7f8e58000700 1 -- start start 2024-04-02T09:15:04.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.476+0000 7f8e58000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e500b3b30 0x7f8e500afa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.476+0000 7f8e58000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e50157f90 0x7f8e500affc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.476+0000 7f8e58000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e501613e0 0x7f8e5015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.476+0000 7f8e58000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e500ac300 con 0x7f8e50157f90 2024-04-02T09:15:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.476+0000 7f8e58000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e50145810 con 0x7f8e501613e0 2024-04-02T09:15:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.476+0000 7f8e58000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e50145af0 con 0x7f8e500b3b30 2024-04-02T09:15:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.476+0000 7f8e54c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e50157f90 0x7f8e500affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.476+0000 7f8e54c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e50157f90 0x7f8e500affc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54490/0 (socket says 172.21.15.89:54490) 2024-04-02T09:15:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.476+0000 7f8e54c00700 1 -- 172.21.15.89:0/2845248303 learned_addr learned my addr 172.21.15.89:0/2845248303 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.476+0000 7f8e56000700 1 --2- 172.21.15.89:0/2845248303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e501613e0 0x7f8e5015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.476+0000 7f8e55600700 1 --2- 172.21.15.89:0/2845248303 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e500b3b30 0x7f8e500afa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.477+0000 7f8e54c00700 1 -- 172.21.15.89:0/2845248303 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e500b3b30 msgr2=0x7f8e500afa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.477+0000 7f8e54c00700 1 --2- 172.21.15.89:0/2845248303 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e500b3b30 0x7f8e500afa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.477+0000 7f8e54c00700 1 -- 172.21.15.89:0/2845248303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e501613e0 msgr2=0x7f8e5015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.477+0000 7f8e54c00700 1 --2- 172.21.15.89:0/2845248303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e501613e0 0x7f8e5015ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.477+0000 7f8e54c00700 1 -- 172.21.15.89:0/2845248303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e440094a0 con 0x7f8e50157f90 2024-04-02T09:15:04.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.477+0000 7f8e54c00700 1 --2- 172.21.15.89:0/2845248303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e50157f90 0x7f8e500affc0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c007fd0 tx=0x7f8e3c00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:04.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.477+0000 7f8e4d600700 1 -- 172.21.15.89:0/2845248303 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c005d70 con 0x7f8e50157f90 2024-04-02T09:15:04.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.477+0000 7f8e4d600700 1 -- 172.21.15.89:0/2845248303 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3c0070d0 con 0x7f8e50157f90 2024-04-02T09:15:04.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.477+0000 7f8e58000700 1 -- 172.21.15.89:0/2845248303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e48009a70 con 0x7f8e50157f90 2024-04-02T09:15:04.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.477+0000 7f8e4d600700 1 -- 172.21.15.89:0/2845248303 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c007460 con 0x7f8e50157f90 2024-04-02T09:15:04.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.477+0000 7f8e58000700 1 -- 172.21.15.89:0/2845248303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5006cc20 con 0x7f8e50157f90 2024-04-02T09:15:04.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.479+0000 7f8e58000700 1 -- 172.21.15.89:0/2845248303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e50067720 con 0x7f8e50157f90 2024-04-02T09:15:04.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.480+0000 7f8e4d600700 1 -- 172.21.15.89:0/2845248303 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f8e3c0075c0 con 0x7f8e50157f90 2024-04-02T09:15:04.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.480+0000 7f8e4d600700 1 --2- 172.21.15.89:0/2845248303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8e3006ae70 0x7f8e3006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.480+0000 7f8e55600700 1 -- 172.21.15.89:0/2845248303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8e3006ae70 msgr2=0x7f8e3006d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:04.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.480+0000 7f8e4d600700 1 -- 172.21.15.89:0/2845248303 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f8e3c088d10 con 0x7f8e50157f90 2024-04-02T09:15:04.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.481+0000 7f8e55600700 1 --2- 172.21.15.89:0/2845248303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8e3006ae70 0x7f8e3006d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.485+0000 7f8e4d600700 1 -- 172.21.15.89:0/2845248303 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8e3c053880 con 0x7f8e50157f90 2024-04-02T09:15:04.555 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:04.556+0000 7f4bc690b240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:15:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.682+0000 7f8e55600700 1 -- 172.21.15.89:0/2845248303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8e3006ae70 msgr2=0x7f8e3006d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.682+0000 7f8e55600700 1 --2- 172.21.15.89:0/2845248303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8e3006ae70 0x7f8e3006d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.693+0000 7f8e58000700 1 -- 172.21.15.89:0/2845248303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8e500b3070 con 0x7f8e50157f90 2024-04-02T09:15:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.693+0000 7f8e4d600700 1 -- 172.21.15.89:0/2845248303 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v266) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8e500b3070 con 0x7f8e50157f90 2024-04-02T09:15:04.692 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:15:04.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.696+0000 7f8e58000700 1 -- 172.21.15.89:0/2845248303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8e3006ae70 msgr2=0x7f8e3006d330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.696+0000 7f8e58000700 1 --2- 172.21.15.89:0/2845248303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8e3006ae70 0x7f8e3006d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.696+0000 7f8e58000700 1 -- 172.21.15.89:0/2845248303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e50157f90 msgr2=0x7f8e500affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.696+0000 7f8e58000700 1 --2- 172.21.15.89:0/2845248303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e50157f90 0x7f8e500affc0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c007fd0 tx=0x7f8e3c00c9e0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.696+0000 7f8e58000700 1 -- 172.21.15.89:0/2845248303 shutdown_connections 2024-04-02T09:15:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.696+0000 7f8e58000700 1 --2- 172.21.15.89:0/2845248303 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f8e3006ae70 0x7f8e3006d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.696+0000 7f8e58000700 1 --2- 172.21.15.89:0/2845248303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e501613e0 0x7f8e5015ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.696+0000 7f8e58000700 1 --2- 172.21.15.89:0/2845248303 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8e500b3b30 0x7f8e500afa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.696+0000 7f8e58000700 1 --2- 172.21.15.89:0/2845248303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e50157f90 0x7f8e500affc0 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.696+0000 7f8e58000700 1 -- 172.21.15.89:0/2845248303 >> 172.21.15.89:0/2845248303 conn(0x7f8e500b0d70 msgr2=0x7f8e5015e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.696+0000 7f8e58000700 1 -- 172.21.15.89:0/2845248303 shutdown_connections 2024-04-02T09:15:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.696+0000 7f8e58000700 1 -- 172.21.15.89:0/2845248303 wait complete. 2024-04-02T09:15:04.713 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:15:04.766 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:04.768+0000 7f4bc690b240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:15:04.800 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:04.802+0000 7f92319a2240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:15:04.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.909+0000 7f9835000700 1 Processor -- start 2024-04-02T09:15:04.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.909+0000 7f9835000700 1 -- start start 2024-04-02T09:15:04.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.910+0000 7f9835000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9830072580 0x7f98300b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.910+0000 7f9835000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98300b2e50 0x7f98300709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.910+0000 7f9835000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98300b2180 con 0x7f98300b3af0 2024-04-02T09:15:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.910+0000 7f9835000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f983015b100 con 0x7f9830072580 2024-04-02T09:15:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.910+0000 7f9835000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983015b3e0 con 0x7f98300b2e50 2024-04-02T09:15:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.911+0000 7f9827e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9830072580 0x7f98300b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.911+0000 7f982e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98300b2e50 0x7f98300709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.911+0000 7f982de00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f98300b3af0 0x7f983015a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:48816/0 (socket says 172.21.15.89:48816) 2024-04-02T09:15:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.911+0000 7f982de00700 1 -- 172.21.15.89:0/129061010 learned_addr learned my addr 172.21.15.89:0/129061010 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.911+0000 7f982d400700 1 -- 172.21.15.89:0/129061010 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494125360 0 0) 0x7f98300b2180 con 0x7f98300b3af0 2024-04-02T09:15:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.911+0000 7f982d400700 1 -- 172.21.15.89:0/129061010 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98140035b0 con 0x7f98300b3af0 2024-04-02T09:15:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.911+0000 7f9827e00700 1 -- 172.21.15.89:0/129061010 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98300b2e50 msgr2=0x7f98300709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.911+0000 7f9827e00700 1 --2- 172.21.15.89:0/129061010 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98300b2e50 0x7f98300709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.911+0000 7f9827e00700 1 -- 172.21.15.89:0/129061010 >> v1:172.21.15.113:6789/0 conn(0x7f98300b3af0 legacy=0x7f983015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:04.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.911+0000 7f9827e00700 1 -- 172.21.15.89:0/129061010 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98300b23f0 con 0x7f9830072580 2024-04-02T09:15:04.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.911+0000 7f982e800700 1 --2- 172.21.15.89:0/129061010 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98300b2e50 0x7f98300709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:15:04.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.912+0000 7f9827e00700 1 --2- 172.21.15.89:0/129061010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9830072580 0x7f98300b2910 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f9818009840 tx=0x7f9818009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdab995a23177a00 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:04.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.912+0000 7f982d400700 1 -- 172.21.15.89:0/129061010 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818011830 con 0x7f9830072580 2024-04-02T09:15:04.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.912+0000 7f982d400700 1 -- 172.21.15.89:0/129061010 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9818011990 con 0x7f9830072580 2024-04-02T09:15:04.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.912+0000 7f982d400700 1 -- 172.21.15.89:0/129061010 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818011d20 con 0x7f9830072580 2024-04-02T09:15:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.913+0000 7f9835000700 1 -- 172.21.15.89:0/129061010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9830072580 msgr2=0x7f98300b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.913+0000 7f9835000700 1 --2- 172.21.15.89:0/129061010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9830072580 0x7f98300b2910 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f9818009840 tx=0x7f9818009b50 comp rx=0 tx=0).stop 2024-04-02T09:15:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.913+0000 7f9835000700 1 -- 172.21.15.89:0/129061010 shutdown_connections 2024-04-02T09:15:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.913+0000 7f9835000700 1 --2- 172.21.15.89:0/129061010 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98300b2e50 0x7f98300709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.913+0000 7f9835000700 1 --2- 172.21.15.89:0/129061010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9830072580 0x7f98300b2910 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.913+0000 7f9835000700 1 -- 172.21.15.89:0/129061010 >> 172.21.15.89:0/129061010 conn(0x7f98300cf190 msgr2=0x7f98300cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:04.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.913+0000 7f9835000700 1 -- 172.21.15.89:0/129061010 shutdown_connections 2024-04-02T09:15:04.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.913+0000 7f9835000700 1 -- 172.21.15.89:0/129061010 wait complete. 2024-04-02T09:15:04.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.914+0000 7f9835000700 1 Processor -- start 2024-04-02T09:15:04.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.915+0000 7f9835000700 1 -- start start 2024-04-02T09:15:04.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f9835000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9830072580 0x7f9830155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f9835000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98300b2e50 0x7f9830150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f9835000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98300b3af0 0x7f9830153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f9835000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9830155ca0 con 0x7f98300b3af0 2024-04-02T09:15:04.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f9835000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9830155eb0 con 0x7f98300b2e50 2024-04-02T09:15:04.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f9835000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9830152360 con 0x7f9830072580 2024-04-02T09:15:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f982e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98300b3af0 0x7f9830153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f982de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9830072580 0x7f9830155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f9827e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98300b2e50 0x7f9830150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f982e800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98300b3af0 0x7f9830153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54502/0 (socket says 172.21.15.89:54502) 2024-04-02T09:15:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f982de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9830072580 0x7f9830155760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:58198/0 (socket says 172.21.15.89:58198) 2024-04-02T09:15:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f982e800700 1 -- 172.21.15.89:0/918375631 learned_addr learned my addr 172.21.15.89:0/918375631 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f982e800700 1 -- 172.21.15.89:0/918375631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9830072580 msgr2=0x7f9830155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f982e800700 1 --2- 172.21.15.89:0/918375631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9830072580 0x7f9830155760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f982e800700 1 -- 172.21.15.89:0/918375631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98300b2e50 msgr2=0x7f9830150a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f982e800700 1 --2- 172.21.15.89:0/918375631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98300b2e50 0x7f9830150a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.916+0000 7f982e800700 1 -- 172.21.15.89:0/918375631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9828009a70 con 0x7f98300b3af0 2024-04-02T09:15:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.917+0000 7f982e800700 1 --2- 172.21.15.89:0/918375631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98300b3af0 0x7f9830153120 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f982800efc0 tx=0x7f982800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:04.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.917+0000 7f9826000700 1 -- 172.21.15.89:0/918375631 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982800ca90 con 0x7f98300b3af0 2024-04-02T09:15:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.917+0000 7f9826000700 1 -- 172.21.15.89:0/918375631 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982800f040 con 0x7f98300b3af0 2024-04-02T09:15:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.917+0000 7f9835000700 1 -- 172.21.15.89:0/918375631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98180094a0 con 0x7f98300b3af0 2024-04-02T09:15:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.917+0000 7f9826000700 1 -- 172.21.15.89:0/918375631 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982800ce10 con 0x7f98300b3af0 2024-04-02T09:15:04.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.917+0000 7f9835000700 1 -- 172.21.15.89:0/918375631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9830152920 con 0x7f98300b3af0 2024-04-02T09:15:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.919+0000 7f9835000700 1 -- 172.21.15.89:0/918375631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9830067720 con 0x7f98300b3af0 2024-04-02T09:15:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.919+0000 7f9826000700 1 -- 172.21.15.89:0/918375631 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f9828014480 con 0x7f98300b3af0 2024-04-02T09:15:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.920+0000 7f9826000700 1 --2- 172.21.15.89:0/918375631 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f980c06ae40 0x7f980c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.920+0000 7f9826000700 1 -- 172.21.15.89:0/918375631 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f982808a1d0 con 0x7f98300b3af0 2024-04-02T09:15:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.920+0000 7f982de00700 1 -- 172.21.15.89:0/918375631 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f980c06ae40 msgr2=0x7f980c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:04.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.920+0000 7f982de00700 1 --2- 172.21.15.89:0/918375631 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f980c06ae40 0x7f980c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:04.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:04.924+0000 7f9826000700 1 -- 172.21.15.89:0/918375631 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9828054c10 con 0x7f98300b3af0 2024-04-02T09:15:04.975 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:04.977+0000 7f4bc690b240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:15:05.041 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:05.043+0000 7f92319a2240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:15:05.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.121+0000 7f982de00700 1 -- 172.21.15.89:0/918375631 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f980c06ae40 msgr2=0x7f980c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:05.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.121+0000 7f982de00700 1 --2- 172.21.15.89:0/918375631 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f980c06ae40 0x7f980c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:05.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.194+0000 7f9835000700 1 -- 172.21.15.89:0/918375631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9830070ed0 con 0x7f98300b3af0 2024-04-02T09:15:05.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.194+0000 7f9826000700 1 -- 172.21.15.89:0/918375631 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+4756 (secure 0 0 0) 0x7f9828009d70 con 0x7f98300b3af0 2024-04-02T09:15:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:05.193 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":135,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10031,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1597421311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1597421311},{"type":"v1","addr":"172.21.15.113:6834","nonce":1597421311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":10034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3719211992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":3719211992},{"type":"v1","addr":"172.21.15.113:6835","nonce":3719211992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":10037,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3947526387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3947526387},{"type":"v1","addr":"172.21.15.89:6835","nonce":3947526387}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:14:34.351811+0000","modified":"2024-04-02T09:14:37.361983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12762":{"gid":12762,"name":"c","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2124958719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2124958719},{"type":"v1","addr":"172.21.15.89:6837","nonce":2124958719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-04-02T09:15:05.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.197+0000 7f9835000700 1 -- 172.21.15.89:0/918375631 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f980c06ae40 msgr2=0x7f980c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:05.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.197+0000 7f9835000700 1 --2- 172.21.15.89:0/918375631 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f980c06ae40 0x7f980c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.197+0000 7f9835000700 1 -- 172.21.15.89:0/918375631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98300b3af0 msgr2=0x7f9830153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:05.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.197+0000 7f9835000700 1 --2- 172.21.15.89:0/918375631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98300b3af0 0x7f9830153120 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f982800efc0 tx=0x7f982800c3a0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.197+0000 7f9835000700 1 -- 172.21.15.89:0/918375631 shutdown_connections 2024-04-02T09:15:05.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.197+0000 7f9835000700 1 --2- 172.21.15.89:0/918375631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98300b3af0 0x7f9830153120 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.197+0000 7f9835000700 1 --2- 172.21.15.89:0/918375631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9830072580 0x7f9830155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.197+0000 7f9835000700 1 --2- 172.21.15.89:0/918375631 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f980c06ae40 0x7f980c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.197+0000 7f9835000700 1 --2- 172.21.15.89:0/918375631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98300b2e50 0x7f9830150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.197+0000 7f9835000700 1 -- 172.21.15.89:0/918375631 >> 172.21.15.89:0/918375631 conn(0x7f98300cf190 msgr2=0x7f983006d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:05.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.197+0000 7f9835000700 1 -- 172.21.15.89:0/918375631 shutdown_connections 2024-04-02T09:15:05.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.197+0000 7f9835000700 1 -- 172.21.15.89:0/918375631 wait complete. 2024-04-02T09:15:05.197 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 135 2024-04-02T09:15:05.214 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:15:05.259 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:05.260+0000 7f92319a2240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:15:05.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.412+0000 7f2f38000700 1 Processor -- start 2024-04-02T09:15:05.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.412+0000 7f2f38000700 1 -- start start 2024-04-02T09:15:05.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.414+0000 7f2f38000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2f300acda0 0x7f2f3015e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:05.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.414+0000 7f2f38000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3015eb80 con 0x7f2f300ad180 2024-04-02T09:15:05.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.414+0000 7f2f38000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f300ae7b0 con 0x7f2f300b4860 2024-04-02T09:15:05.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.414+0000 7f2f38000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f3006eef0 con 0x7f2f300acda0 2024-04-02T09:15:05.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.414+0000 7f2f35600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2f300acda0 0x7f2f3015e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:05.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.414+0000 7f2f35600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2f300acda0 0x7f2f3015e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54504/0 (socket says 172.21.15.89:54504) 2024-04-02T09:15:05.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.414+0000 7f2f35600700 1 -- 172.21.15.89:0/3985973641 learned_addr learned my addr 172.21.15.89:0/3985973641 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:05.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.414+0000 7f2f34c00700 1 --1- 172.21.15.89:0/3985973641 >> v1:172.21.15.89:6789/0 conn(0x7f2f300ad180 0x7f2f300b4150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59982/0 (socket says 172.21.15.89:59982) 2024-04-02T09:15:05.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.414+0000 7f2f35600700 1 -- 172.21.15.89:0/3985973641 >> v1:172.21.15.113:6789/0 conn(0x7f2f300b4860 legacy=0x7f2f300b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:05.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.414+0000 7f2f35600700 1 -- 172.21.15.89:0/3985973641 >> v1:172.21.15.89:6789/0 conn(0x7f2f300ad180 legacy=0x7f2f300b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:05.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.414+0000 7f2f35600700 1 -- 172.21.15.89:0/3985973641 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f300aea90 con 0x7f2f300acda0 2024-04-02T09:15:05.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.414+0000 7f2f2fe00700 1 -- 172.21.15.89:0/3985973641 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384723882 0 0) 0x7f2f3015eb80 con 0x7f2f300ad180 2024-04-02T09:15:05.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.415+0000 7f2f35600700 1 --2- 172.21.15.89:0/3985973641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f300acda0 0x7f2f3015e640 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f2f240098f0 tx=0x7f2f24009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443f79bf35a73ad0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:05.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.415+0000 7f2f2fe00700 1 -- 172.21.15.89:0/3985973641 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2400b8a0 con 0x7f2f300acda0 2024-04-02T09:15:05.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.415+0000 7f2f2fe00700 1 -- 172.21.15.89:0/3985973641 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2400ba00 con 0x7f2f300acda0 2024-04-02T09:15:05.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.415+0000 7f2f2fe00700 1 -- 172.21.15.89:0/3985973641 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2400bd90 con 0x7f2f300acda0 2024-04-02T09:15:05.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.416+0000 7f2f38000700 1 -- 172.21.15.89:0/3985973641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f300acda0 msgr2=0x7f2f3015e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:05.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.416+0000 7f2f38000700 1 --2- 172.21.15.89:0/3985973641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f300acda0 0x7f2f3015e640 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f2f240098f0 tx=0x7f2f24009c00 comp rx=0 tx=0).stop 2024-04-02T09:15:05.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.416+0000 7f2f38000700 1 -- 172.21.15.89:0/3985973641 shutdown_connections 2024-04-02T09:15:05.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.416+0000 7f2f38000700 1 --2- 172.21.15.89:0/3985973641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f300acda0 0x7f2f3015e640 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.416+0000 7f2f38000700 1 -- 172.21.15.89:0/3985973641 >> 172.21.15.89:0/3985973641 conn(0x7f2f30072170 msgr2=0x7f2f3006a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:05.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.416+0000 7f2f38000700 1 -- 172.21.15.89:0/3985973641 shutdown_connections 2024-04-02T09:15:05.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.416+0000 7f2f38000700 1 -- 172.21.15.89:0/3985973641 wait complete. 2024-04-02T09:15:05.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.417+0000 7f2f38000700 1 Processor -- start 2024-04-02T09:15:05.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.418+0000 7f2f38000700 1 -- start start 2024-04-02T09:15:05.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.418+0000 7f2f38000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f300acda0 0x7f2f30159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:05.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.418+0000 7f2f38000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f300ad180 0x7f2f300ce600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:05.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.418+0000 7f2f38000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f300b4860 0x7f2f300ceb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:05.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.418+0000 7f2f38000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f300cf080 con 0x7f2f300acda0 2024-04-02T09:15:05.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.418+0000 7f2f38000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3015a9e0 con 0x7f2f300b4860 2024-04-02T09:15:05.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.418+0000 7f2f38000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f301f67e0 con 0x7f2f300ad180 2024-04-02T09:15:05.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.419+0000 7f2f35600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f300acda0 0x7f2f30159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:05.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.419+0000 7f2f35600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f300acda0 0x7f2f30159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54516/0 (socket says 172.21.15.89:54516) 2024-04-02T09:15:05.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.419+0000 7f2f35600700 1 -- 172.21.15.89:0/1105940890 learned_addr learned my addr 172.21.15.89:0/1105940890 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:05.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.419+0000 7f2f36000700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f300b4860 0x7f2f300ceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:05.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.419+0000 7f2f34c00700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f300ad180 0x7f2f300ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:05.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.419+0000 7f2f35600700 1 -- 172.21.15.89:0/1105940890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f300ad180 msgr2=0x7f2f300ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:05.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.419+0000 7f2f35600700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f300ad180 0x7f2f300ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.419+0000 7f2f35600700 1 -- 172.21.15.89:0/1105940890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f300b4860 msgr2=0x7f2f300ceb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:05.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.419+0000 7f2f35600700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f300b4860 0x7f2f300ceb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.419+0000 7f2f35600700 1 -- 172.21.15.89:0/1105940890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f24009550 con 0x7f2f300acda0 2024-04-02T09:15:05.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.419+0000 7f2f34c00700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f300ad180 0x7f2f300ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:05.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.419+0000 7f2f35600700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f300acda0 0x7f2f30159c50 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f2f24014040 tx=0x7f2f24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:05.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.420+0000 7f2f2d600700 1 -- 172.21.15.89:0/1105940890 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f24011970 con 0x7f2f300acda0 2024-04-02T09:15:05.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.420+0000 7f2f2d600700 1 -- 172.21.15.89:0/1105940890 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2401e070 con 0x7f2f300acda0 2024-04-02T09:15:05.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.420+0000 7f2f38000700 1 -- 172.21.15.89:0/1105940890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f301f6a80 con 0x7f2f300acda0 2024-04-02T09:15:05.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.420+0000 7f2f2d600700 1 -- 172.21.15.89:0/1105940890 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f24011cd0 con 0x7f2f300acda0 2024-04-02T09:15:05.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.420+0000 7f2f38000700 1 -- 172.21.15.89:0/1105940890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f301f7040 con 0x7f2f300acda0 2024-04-02T09:15:05.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.421+0000 7f2f38000700 1 -- 172.21.15.89:0/1105940890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f30067720 con 0x7f2f300acda0 2024-04-02T09:15:05.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.423+0000 7f2f2d600700 1 -- 172.21.15.89:0/1105940890 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f2f2402d450 con 0x7f2f300acda0 2024-04-02T09:15:05.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.423+0000 7f2f2d600700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2f1006ae40 0x7f2f1006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:05.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.423+0000 7f2f2d600700 1 -- 172.21.15.89:0/1105940890 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f2f2408a8f0 con 0x7f2f300acda0 2024-04-02T09:15:05.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.424+0000 7f2f34c00700 1 -- 172.21.15.89:0/1105940890 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2f1006ae40 msgr2=0x7f2f1006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:05.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.424+0000 7f2f34c00700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2f1006ae40 0x7f2f1006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:05.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.428+0000 7f2f2d600700 1 -- 172.21.15.89:0/1105940890 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2f24055570 con 0x7f2f300acda0 2024-04-02T09:15:05.449 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:05.450+0000 7f4bc690b240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:15:05.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.625+0000 7f2f34c00700 1 -- 172.21.15.89:0/1105940890 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2f1006ae40 msgr2=0x7f2f1006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:05.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.625+0000 7f2f34c00700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2f1006ae40 0x7f2f1006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:05.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.636+0000 7f2f38000700 1 -- 172.21.15.89:0/1105940890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f300adf40 con 0x7f2f300acda0 2024-04-02T09:15:05.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.637+0000 7f2f2d600700 1 -- 172.21.15.89:0/1105940890 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v266) v1 ==== 74+0+17360 (secure 0 0 0) 0x7f2f24058b90 con 0x7f2f300acda0 2024-04-02T09:15:05.636 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:05.636 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":266,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:14:55.473418+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:14:32.705979+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-04-02T09:14:33.330752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/422001866":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/3534660524":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/2610551691":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/1604459128":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/3010462327":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/3737296061":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/1785135719":"2024-04-03T09:14:55.473381+0000","172.21.15.89:6833/108855":"2024-04-03T09:14:55.473381+0000","172.21.15.89:6832/108855":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/1017424418":"2024-04-03T09:14:55.473381+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:15:05.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.640+0000 7f2f38000700 1 -- 172.21.15.89:0/1105940890 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2f1006ae40 msgr2=0x7f2f1006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:05.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.640+0000 7f2f38000700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2f1006ae40 0x7f2f1006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.640+0000 7f2f38000700 1 -- 172.21.15.89:0/1105940890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f300acda0 msgr2=0x7f2f30159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:05.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.640+0000 7f2f38000700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f300acda0 0x7f2f30159c50 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f2f24014040 tx=0x7f2f24004750 comp rx=0 tx=0).stop 2024-04-02T09:15:05.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.640+0000 7f2f38000700 1 -- 172.21.15.89:0/1105940890 shutdown_connections 2024-04-02T09:15:05.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.640+0000 7f2f38000700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2f1006ae40 0x7f2f1006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.640+0000 7f2f38000700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f300b4860 0x7f2f300ceb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.640+0000 7f2f38000700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f300acda0 0x7f2f30159c50 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.640+0000 7f2f38000700 1 --2- 172.21.15.89:0/1105940890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f300ad180 0x7f2f300ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.640+0000 7f2f38000700 1 -- 172.21.15.89:0/1105940890 >> 172.21.15.89:0/1105940890 conn(0x7f2f30072170 msgr2=0x7f2f300af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:05.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.640+0000 7f2f38000700 1 -- 172.21.15.89:0/1105940890 shutdown_connections 2024-04-02T09:15:05.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.640+0000 7f2f38000700 1 -- 172.21.15.89:0/1105940890 wait complete. 2024-04-02T09:15:05.657 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T09:15:05.658 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:15:05.658 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:05.659+0000 7f4bc690b240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:15:05.771 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:05.773+0000 7f92319a2240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:15:05.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.901+0000 7f6556e00700 1 Processor -- start 2024-04-02T09:15:05.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.901+0000 7f6556e00700 1 -- start start 2024-04-02T09:15:05.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.902+0000 7f6556e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f655015b5d0 0x7f65500ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:05.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.902+0000 7f6556e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f65500cb3f0 0x7f65500ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:05.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.902+0000 7f6556e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f655015fd50 con 0x7f65500ae930 2024-04-02T09:15:05.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.902+0000 7f6556e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f655015b9b0 con 0x7f65500cb3f0 2024-04-02T09:15:05.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.902+0000 7f6556e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655015f140 con 0x7f655015b5d0 2024-04-02T09:15:05.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.903+0000 7f6554e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f65500ae930 0x7f6550140dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59992/0 (socket says 172.21.15.89:59992) 2024-04-02T09:15:05.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.903+0000 7f6554e00700 1 -- 172.21.15.89:0/339185998 learned_addr learned my addr 172.21.15.89:0/339185998 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:05.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.903+0000 7f654f400700 1 --2- 172.21.15.89:0/339185998 >> v2:172.21.15.113:3300/0 conn(0x7f65500cb3f0 0x7f65500ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:05.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.903+0000 7f654fe00700 1 --2- 172.21.15.89:0/339185998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f655015b5d0 0x7f65500ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:05.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.903+0000 7f654ea00700 1 -- 172.21.15.89:0/339185998 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221297980 0 0) 0x7f655015fd50 con 0x7f65500ae930 2024-04-02T09:15:05.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.903+0000 7f654ea00700 1 -- 172.21.15.89:0/339185998 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65340035b0 con 0x7f65500ae930 2024-04-02T09:15:05.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.903+0000 7f654fe00700 1 -- 172.21.15.89:0/339185998 >> v2:172.21.15.113:3300/0 conn(0x7f65500cb3f0 msgr2=0x7f65500ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:05.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.903+0000 7f654fe00700 1 --2- 172.21.15.89:0/339185998 >> v2:172.21.15.113:3300/0 conn(0x7f65500cb3f0 0x7f65500ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.903+0000 7f654fe00700 1 -- 172.21.15.89:0/339185998 >> v1:172.21.15.89:6789/0 conn(0x7f65500ae930 legacy=0x7f6550140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:05.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.903+0000 7f654fe00700 1 -- 172.21.15.89:0/339185998 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655015f420 con 0x7f655015b5d0 2024-04-02T09:15:05.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.903+0000 7f654f400700 1 --2- 172.21.15.89:0/339185998 >> v2:172.21.15.113:3300/0 conn(0x7f65500cb3f0 0x7f65500ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:15:05.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.904+0000 7f654fe00700 1 --2- 172.21.15.89:0/339185998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f655015b5d0 0x7f65500ae3f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6540009840 tx=0x7f6540009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=813c3d87aee86fc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:05.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.904+0000 7f654ea00700 1 -- 172.21.15.89:0/339185998 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540011830 con 0x7f655015b5d0 2024-04-02T09:15:05.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.904+0000 7f654ea00700 1 -- 172.21.15.89:0/339185998 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6540011990 con 0x7f655015b5d0 2024-04-02T09:15:05.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.904+0000 7f654ea00700 1 -- 172.21.15.89:0/339185998 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540011d20 con 0x7f655015b5d0 2024-04-02T09:15:05.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.905+0000 7f6556e00700 1 -- 172.21.15.89:0/339185998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f655015b5d0 msgr2=0x7f65500ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:05.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.905+0000 7f6556e00700 1 --2- 172.21.15.89:0/339185998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f655015b5d0 0x7f65500ae3f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6540009840 tx=0x7f6540009b50 comp rx=0 tx=0).stop 2024-04-02T09:15:05.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.905+0000 7f6556e00700 1 -- 172.21.15.89:0/339185998 shutdown_connections 2024-04-02T09:15:05.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.905+0000 7f6556e00700 1 --2- 172.21.15.89:0/339185998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f655015b5d0 0x7f65500ae3f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.905+0000 7f6556e00700 1 --2- 172.21.15.89:0/339185998 >> v2:172.21.15.113:3300/0 conn(0x7f65500cb3f0 0x7f65500ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.905+0000 7f6556e00700 1 -- 172.21.15.89:0/339185998 >> 172.21.15.89:0/339185998 conn(0x7f65500cf190 msgr2=0x7f65500cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:05.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.905+0000 7f6556e00700 1 -- 172.21.15.89:0/339185998 shutdown_connections 2024-04-02T09:15:05.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.905+0000 7f6556e00700 1 -- 172.21.15.89:0/339185998 wait complete. 2024-04-02T09:15:05.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.907+0000 7f6556e00700 1 Processor -- start 2024-04-02T09:15:05.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.907+0000 7f6556e00700 1 -- start start 2024-04-02T09:15:05.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.908+0000 7f6556e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65500ae930 0x7f65500713d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:05.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.908+0000 7f6556e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f655015b5d0 0x7f655015a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:05.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.909+0000 7f6556e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655014f1e0 0x7f6550155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:05.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.909+0000 7f6556e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655014fcc0 con 0x7f655014f1e0 2024-04-02T09:15:05.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.909+0000 7f6556e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6550071a60 con 0x7f655015b5d0 2024-04-02T09:15:05.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.909+0000 7f6556e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6550071d40 con 0x7f65500ae930 2024-04-02T09:15:05.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.909+0000 7f6554e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655014f1e0 0x7f6550155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:05.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.909+0000 7f654fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65500ae930 0x7f65500713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:05.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.909+0000 7f6554e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655014f1e0 0x7f6550155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54526/0 (socket says 172.21.15.89:54526) 2024-04-02T09:15:05.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.909+0000 7f6554e00700 1 -- 172.21.15.89:0/1175164427 learned_addr learned my addr 172.21.15.89:0/1175164427 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:05.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.909+0000 7f654f400700 1 --2- 172.21.15.89:0/1175164427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f655015b5d0 0x7f655015a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:05.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.909+0000 7f6554e00700 1 -- 172.21.15.89:0/1175164427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65500ae930 msgr2=0x7f65500713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:05.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.909+0000 7f6554e00700 1 --2- 172.21.15.89:0/1175164427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65500ae930 0x7f65500713d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.909+0000 7f6554e00700 1 -- 172.21.15.89:0/1175164427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f655015b5d0 msgr2=0x7f655015a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:05.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.909+0000 7f6554e00700 1 --2- 172.21.15.89:0/1175164427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f655015b5d0 0x7f655015a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:05.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.909+0000 7f6554e00700 1 -- 172.21.15.89:0/1175164427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c0094a0 con 0x7f655014f1e0 2024-04-02T09:15:05.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.910+0000 7f6554e00700 1 --2- 172.21.15.89:0/1175164427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655014f1e0 0x7f6550155960 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7f654800caa0 tx=0x7f654800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:05.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.910+0000 7f6547e00700 1 -- 172.21.15.89:0/1175164427 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654800a980 con 0x7f655014f1e0 2024-04-02T09:15:05.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.910+0000 7f6547e00700 1 -- 172.21.15.89:0/1175164427 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654800aae0 con 0x7f655014f1e0 2024-04-02T09:15:05.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.910+0000 7f6556e00700 1 -- 172.21.15.89:0/1175164427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65400094a0 con 0x7f655014f1e0 2024-04-02T09:15:05.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.910+0000 7f6547e00700 1 -- 172.21.15.89:0/1175164427 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654800ae90 con 0x7f655014f1e0 2024-04-02T09:15:05.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.910+0000 7f6556e00700 1 -- 172.21.15.89:0/1175164427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6550072300 con 0x7f655014f1e0 2024-04-02T09:15:05.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.911+0000 7f6556e00700 1 -- 172.21.15.89:0/1175164427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65500c8460 con 0x7f655014f1e0 2024-04-02T09:15:05.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.912+0000 7f6547e00700 1 -- 172.21.15.89:0/1175164427 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f654801b3b0 con 0x7f655014f1e0 2024-04-02T09:15:05.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.913+0000 7f6547e00700 1 --2- 172.21.15.89:0/1175164427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f653006ae40 0x7f653006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:05.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.913+0000 7f654fe00700 1 -- 172.21.15.89:0/1175164427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f653006ae40 msgr2=0x7f653006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:05.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.913+0000 7f6547e00700 1 -- 172.21.15.89:0/1175164427 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f6548089550 con 0x7f655014f1e0 2024-04-02T09:15:05.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.913+0000 7f654fe00700 1 --2- 172.21.15.89:0/1175164427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f653006ae40 0x7f653006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:05.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:05.917+0000 7f6547e00700 1 -- 172.21.15.89:0/1175164427 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f65480541f0 con 0x7f655014f1e0 2024-04-02T09:15:05.996 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:05.998+0000 7f92319a2240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:15:06.099 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:06.100+0000 7f4bc690b240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:15:06.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.114+0000 7f654fe00700 1 -- 172.21.15.89:0/1175164427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f653006ae40 msgr2=0x7f653006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:06.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.114+0000 7f654fe00700 1 --2- 172.21.15.89:0/1175164427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f653006ae40 0x7f653006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:06.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.173+0000 7f6556e00700 1 -- 172.21.15.89:0/1175164427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6550067720 con 0x7f655014f1e0 2024-04-02T09:15:06.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.174+0000 7f6547e00700 1 -- 172.21.15.89:0/1175164427 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+4756 (secure 0 0 0) 0x7f6548057810 con 0x7f655014f1e0 2024-04-02T09:15:06.172 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:06.172 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":135,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10031,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1597421311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1597421311},{"type":"v1","addr":"172.21.15.113:6834","nonce":1597421311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":10034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3719211992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":3719211992},{"type":"v1","addr":"172.21.15.113:6835","nonce":3719211992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":10037,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3947526387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3947526387},{"type":"v1","addr":"172.21.15.89:6835","nonce":3947526387}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:14:34.351811+0000","modified":"2024-04-02T09:14:37.361983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12762":{"gid":12762,"name":"c","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2124958719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2124958719},{"type":"v1","addr":"172.21.15.89:6837","nonce":2124958719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-04-02T09:15:06.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.177+0000 7f6556e00700 1 -- 172.21.15.89:0/1175164427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f653006ae40 msgr2=0x7f653006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.177+0000 7f6556e00700 1 --2- 172.21.15.89:0/1175164427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f653006ae40 0x7f653006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.177+0000 7f6556e00700 1 -- 172.21.15.89:0/1175164427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655014f1e0 msgr2=0x7f6550155960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.177+0000 7f6556e00700 1 --2- 172.21.15.89:0/1175164427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655014f1e0 0x7f6550155960 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7f654800caa0 tx=0x7f654800cdb0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.177+0000 7f6556e00700 1 -- 172.21.15.89:0/1175164427 shutdown_connections 2024-04-02T09:15:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.177+0000 7f6556e00700 1 --2- 172.21.15.89:0/1175164427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655014f1e0 0x7f6550155960 unknown :-1 s=CLOSED pgs=2899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.177+0000 7f6556e00700 1 --2- 172.21.15.89:0/1175164427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65500ae930 0x7f65500713d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.177+0000 7f6556e00700 1 --2- 172.21.15.89:0/1175164427 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f653006ae40 0x7f653006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.177+0000 7f6556e00700 1 --2- 172.21.15.89:0/1175164427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f655015b5d0 0x7f655015a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.177+0000 7f6556e00700 1 -- 172.21.15.89:0/1175164427 >> 172.21.15.89:0/1175164427 conn(0x7f65500cf190 msgr2=0x7f65500aed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.177+0000 7f6556e00700 1 -- 172.21.15.89:0/1175164427 shutdown_connections 2024-04-02T09:15:06.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.177+0000 7f6556e00700 1 -- 172.21.15.89:0/1175164427 wait complete. 2024-04-02T09:15:06.177 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 135 2024-04-02T09:15:06.195 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:15:06.210 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:06.211+0000 7f4bc690b240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:15:06.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.401+0000 7f6edde00700 1 Processor -- start 2024-04-02T09:15:06.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.401+0000 7f6edde00700 1 -- start start 2024-04-02T09:15:06.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.402+0000 7f6edde00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ed80cb3f0 0x7f6ed80ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:06.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.402+0000 7f6edde00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6ed80ac600 0x7f6ed806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:06.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.402+0000 7f6edde00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed815fd50 con 0x7f6ed815b5d0 2024-04-02T09:15:06.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.402+0000 7f6edde00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed815b9b0 con 0x7f6ed80ac600 2024-04-02T09:15:06.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.402+0000 7f6edde00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed80aca70 con 0x7f6ed80cb3f0 2024-04-02T09:15:06.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.402+0000 7f6ed6a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6ed815b5d0 0x7f6ed80ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59994/0 (socket says 172.21.15.89:59994) 2024-04-02T09:15:06.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.402+0000 7f6ed6a00700 1 -- 172.21.15.89:0/2292585051 learned_addr learned my addr 172.21.15.89:0/2292585051 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:06.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.403+0000 7f6ed6000700 1 --2- 172.21.15.89:0/2292585051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ed80cb3f0 0x7f6ed80ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:06.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.403+0000 7f6ed7400700 1 --2- 172.21.15.89:0/2292585051 >> v2:172.21.15.113:3300/0 conn(0x7f6ed80ac600 0x7f6ed806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:06.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.403+0000 7f6ed5600700 1 -- 172.21.15.89:0/2292585051 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219500160 0 0) 0x7f6ed815fd50 con 0x7f6ed815b5d0 2024-04-02T09:15:06.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.403+0000 7f6ed5600700 1 -- 172.21.15.89:0/2292585051 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ebc0035b0 con 0x7f6ed815b5d0 2024-04-02T09:15:06.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.403+0000 7f6ed6000700 1 -- 172.21.15.89:0/2292585051 >> v2:172.21.15.113:3300/0 conn(0x7f6ed80ac600 msgr2=0x7f6ed806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:06.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.403+0000 7f6ed6000700 1 --2- 172.21.15.89:0/2292585051 >> v2:172.21.15.113:3300/0 conn(0x7f6ed80ac600 0x7f6ed806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.403+0000 7f6ed6000700 1 -- 172.21.15.89:0/2292585051 >> v1:172.21.15.89:6789/0 conn(0x7f6ed815b5d0 legacy=0x7f6ed80ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:06.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.403+0000 7f6ed6000700 1 -- 172.21.15.89:0/2292585051 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed815f140 con 0x7f6ed80cb3f0 2024-04-02T09:15:06.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.403+0000 7f6ed7400700 1 --2- 172.21.15.89:0/2292585051 >> v2:172.21.15.113:3300/0 conn(0x7f6ed80ac600 0x7f6ed806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:15:06.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.403+0000 7f6ed6000700 1 --2- 172.21.15.89:0/2292585051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ed80cb3f0 0x7f6ed80ac0c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009840 tx=0x7f6ec4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=218b4a8f796b600a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:06.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.404+0000 7f6ed5600700 1 -- 172.21.15.89:0/2292585051 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4011830 con 0x7f6ed80cb3f0 2024-04-02T09:15:06.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.404+0000 7f6ed5600700 1 -- 172.21.15.89:0/2292585051 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec4011990 con 0x7f6ed80cb3f0 2024-04-02T09:15:06.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.404+0000 7f6ed5600700 1 -- 172.21.15.89:0/2292585051 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4011d20 con 0x7f6ed80cb3f0 2024-04-02T09:15:06.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.404+0000 7f6edde00700 1 -- 172.21.15.89:0/2292585051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ed80cb3f0 msgr2=0x7f6ed80ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:06.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.404+0000 7f6edde00700 1 --2- 172.21.15.89:0/2292585051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ed80cb3f0 0x7f6ed80ac0c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009840 tx=0x7f6ec4009b50 comp rx=0 tx=0).stop 2024-04-02T09:15:06.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.404+0000 7f6edde00700 1 -- 172.21.15.89:0/2292585051 shutdown_connections 2024-04-02T09:15:06.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.404+0000 7f6edde00700 1 --2- 172.21.15.89:0/2292585051 >> v2:172.21.15.113:3300/0 conn(0x7f6ed80ac600 0x7f6ed806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.404+0000 7f6edde00700 1 --2- 172.21.15.89:0/2292585051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ed80cb3f0 0x7f6ed80ac0c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.404+0000 7f6edde00700 1 -- 172.21.15.89:0/2292585051 >> 172.21.15.89:0/2292585051 conn(0x7f6ed80cf190 msgr2=0x7f6ed80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:06.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.404+0000 7f6edde00700 1 -- 172.21.15.89:0/2292585051 shutdown_connections 2024-04-02T09:15:06.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.404+0000 7f6edde00700 1 -- 172.21.15.89:0/2292585051 wait complete. 2024-04-02T09:15:06.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.406+0000 7f6edde00700 1 Processor -- start 2024-04-02T09:15:06.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.406+0000 7f6edde00700 1 -- start start 2024-04-02T09:15:06.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.407+0000 7f6edde00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ed80ac600 0x7f6ed80ce120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:06.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.407+0000 7f6edde00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ed815b5d0 0x7f6ed80b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:06.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.407+0000 7f6edde00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ed81f2d30 0x7f6ed81f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:06.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.407+0000 7f6edde00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed81450f0 con 0x7f6ed80ac600 2024-04-02T09:15:06.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.407+0000 7f6edde00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed81f3d00 con 0x7f6ed815b5d0 2024-04-02T09:15:06.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.407+0000 7f6edde00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed81f31b0 con 0x7f6ed81f2d30 2024-04-02T09:15:06.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.407+0000 7f6ed6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ed80ac600 0x7f6ed80ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:06.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.408+0000 7f6ed6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ed80ac600 0x7f6ed80ce120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54540/0 (socket says 172.21.15.89:54540) 2024-04-02T09:15:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.408+0000 7f6ed6a00700 1 -- 172.21.15.89:0/3359348325 learned_addr learned my addr 172.21.15.89:0/3359348325 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.408+0000 7f6ed6000700 1 --2- 172.21.15.89:0/3359348325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ed815b5d0 0x7f6ed80b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.408+0000 7f6ed7400700 1 --2- 172.21.15.89:0/3359348325 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ed81f2d30 0x7f6ed81f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.408+0000 7f6ed6a00700 1 -- 172.21.15.89:0/3359348325 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ed81f2d30 msgr2=0x7f6ed81f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.408+0000 7f6ed6a00700 1 --2- 172.21.15.89:0/3359348325 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ed81f2d30 0x7f6ed81f3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.408+0000 7f6ed6a00700 1 -- 172.21.15.89:0/3359348325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ed815b5d0 msgr2=0x7f6ed80b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.408+0000 7f6ed6a00700 1 --2- 172.21.15.89:0/3359348325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ed815b5d0 0x7f6ed80b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.408+0000 7f6ed6a00700 1 -- 172.21.15.89:0/3359348325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed0009a70 con 0x7f6ed80ac600 2024-04-02T09:15:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.408+0000 7f6ed6a00700 1 --2- 172.21.15.89:0/3359348325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ed80ac600 0x7f6ed80ce120 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8001eb0 tx=0x7f6ec800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.408+0000 7f6ecf400700 1 -- 172.21.15.89:0/3359348325 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec800cea0 con 0x7f6ed80ac600 2024-04-02T09:15:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.408+0000 7f6ecf400700 1 -- 172.21.15.89:0/3359348325 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec8005d70 con 0x7f6ed80ac600 2024-04-02T09:15:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.408+0000 7f6ecf400700 1 -- 172.21.15.89:0/3359348325 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec80072f0 con 0x7f6ed80ac600 2024-04-02T09:15:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.409+0000 7f6edde00700 1 -- 172.21.15.89:0/3359348325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec40094a0 con 0x7f6ed80ac600 2024-04-02T09:15:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.409+0000 7f6edde00700 1 -- 172.21.15.89:0/3359348325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed81f3740 con 0x7f6ed80ac600 2024-04-02T09:15:06.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.410+0000 7f6edde00700 1 -- 172.21.15.89:0/3359348325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed8067720 con 0x7f6ed80ac600 2024-04-02T09:15:06.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.411+0000 7f6ecf400700 1 -- 172.21.15.89:0/3359348325 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 173) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f6ec8007450 con 0x7f6ed80ac600 2024-04-02T09:15:06.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.412+0000 7f6ecf400700 1 --2- 172.21.15.89:0/3359348325 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6eb806ae40 0x7f6eb806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:06.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.412+0000 7f6ecf400700 1 -- 172.21.15.89:0/3359348325 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f6ec8088d30 con 0x7f6ed80ac600 2024-04-02T09:15:06.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.412+0000 7f6ed6000700 1 -- 172.21.15.89:0/3359348325 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6eb806ae40 msgr2=0x7f6eb806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:06.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.412+0000 7f6ed6000700 1 --2- 172.21.15.89:0/3359348325 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6eb806ae40 0x7f6eb806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:06.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.416+0000 7f6ecf400700 1 -- 172.21.15.89:0/3359348325 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ec8053920 con 0x7f6ed80ac600 2024-04-02T09:15:06.468 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:06.470+0000 7f92319a2240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:15:06.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.526+0000 7f6ecf400700 1 -- 172.21.15.89:0/3359348325 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 174) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f6ec8053cf0 con 0x7f6ed80ac600 2024-04-02T09:15:06.585 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:06.586+0000 7f92319a2240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:15:06.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.597+0000 7f6ecf400700 1 -- 172.21.15.89:0/3359348325 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 175) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f6ec8086e50 con 0x7f6ed80ac600 2024-04-02T09:15:06.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.597+0000 7f6ecf400700 1 -- 172.21.15.89:0/3359348325 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6eb806ae40 msgr2=0x7f6eb806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:06.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.597+0000 7f6ecf400700 1 --2- 172.21.15.89:0/3359348325 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6eb806ae40 0x7f6eb806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.666 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:06.667+0000 7f91a4a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:06.666 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:06.667+0000 7f91a4a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:06.666 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:06.667+0000 7f91a4a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:06.666 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:06.667+0000 7f91a4a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:06.666 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:06.667+0000 7f91a4a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:06.667 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:06.668+0000 7f919fa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:06.667 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:06.668+0000 7f919fa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:06.667 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:06.668+0000 7f919fa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:06.667 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:06.668+0000 7f919fa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:06.667 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:06.668+0000 7f919fa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:06.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.693+0000 7f6edde00700 1 -- 172.21.15.89:0/3359348325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ed806ea60 con 0x7f6ed80ac600 2024-04-02T09:15:06.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.693+0000 7f6ecf400700 1 -- 172.21.15.89:0/3359348325 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+4756 (secure 0 0 0) 0x7f6ec8054930 con 0x7f6ed80ac600 2024-04-02T09:15:06.692 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:06.692 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":135,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10031,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1597421311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1597421311},{"type":"v1","addr":"172.21.15.113:6834","nonce":1597421311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":10034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3719211992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":3719211992},{"type":"v1","addr":"172.21.15.113:6835","nonce":3719211992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":10037,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3947526387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3947526387},{"type":"v1","addr":"172.21.15.89:6835","nonce":3947526387}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:14:34.351811+0000","modified":"2024-04-02T09:14:37.361983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12762":{"gid":12762,"name":"c","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/2124958719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2124958719},{"type":"v1","addr":"172.21.15.89:6837","nonce":2124958719}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-04-02T09:15:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.695+0000 7f6edde00700 1 -- 172.21.15.89:0/3359348325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ed80ac600 msgr2=0x7f6ed80ce120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.695+0000 7f6edde00700 1 --2- 172.21.15.89:0/3359348325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ed80ac600 0x7f6ed80ce120 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8001eb0 tx=0x7f6ec800c7b0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.695+0000 7f6edde00700 1 -- 172.21.15.89:0/3359348325 shutdown_connections 2024-04-02T09:15:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.695+0000 7f6edde00700 1 --2- 172.21.15.89:0/3359348325 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ed81f2d30 0x7f6ed81f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.695+0000 7f6edde00700 1 --2- 172.21.15.89:0/3359348325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ed80ac600 0x7f6ed80ce120 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.695+0000 7f6edde00700 1 --2- 172.21.15.89:0/3359348325 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6eb806ae40 0x7f6eb806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.695+0000 7f6edde00700 1 --2- 172.21.15.89:0/3359348325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ed815b5d0 0x7f6ed80b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.695+0000 7f6edde00700 1 -- 172.21.15.89:0/3359348325 >> 172.21.15.89:0/3359348325 conn(0x7f6ed80cf190 msgr2=0x7f6ed815e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:06.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.696+0000 7f6edde00700 1 -- 172.21.15.89:0/3359348325 shutdown_connections 2024-04-02T09:15:06.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.696+0000 7f6edde00700 1 -- 172.21.15.89:0/3359348325 wait complete. 2024-04-02T09:15:06.695 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 135 2024-04-02T09:15:06.711 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:15:06.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.903+0000 7f6998600700 1 Processor -- start 2024-04-02T09:15:06.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.903+0000 7f6998600700 1 -- start start 2024-04-02T09:15:06.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.904+0000 7f6998600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f699015b5d0 0x7f69900ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:06.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.904+0000 7f6998600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69900ac7d0 0x7f699006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:06.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.904+0000 7f6998600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f699015fd50 con 0x7f69900cb3f0 2024-04-02T09:15:06.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.904+0000 7f6998600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f699015b9b0 con 0x7f699015b5d0 2024-04-02T09:15:06.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.904+0000 7f6998600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699015f140 con 0x7f69900ac7d0 2024-04-02T09:15:06.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.905+0000 7f6995c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f699015b5d0 0x7f69900ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:06.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.905+0000 7f6995200700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f69900cb3f0 0x7f69900ac0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:48840/0 (socket says 172.21.15.89:48840) 2024-04-02T09:15:06.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.905+0000 7f6995200700 1 -- 172.21.15.89:0/1133336392 learned_addr learned my addr 172.21.15.89:0/1133336392 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:06.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.905+0000 7f6996600700 1 --2- 172.21.15.89:0/1133336392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69900ac7d0 0x7f699006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:06.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.905+0000 7f6996600700 1 -- 172.21.15.89:0/1133336392 >> v2:172.21.15.113:3300/0 conn(0x7f699015b5d0 msgr2=0x7f69900ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:06.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.905+0000 7f6996600700 1 --2- 172.21.15.89:0/1133336392 >> v2:172.21.15.113:3300/0 conn(0x7f699015b5d0 0x7f69900ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.905+0000 7f6996600700 1 -- 172.21.15.89:0/1133336392 >> v1:172.21.15.113:6789/0 conn(0x7f69900cb3f0 legacy=0x7f69900ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:06.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.905+0000 7f6996600700 1 -- 172.21.15.89:0/1133336392 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699015f420 con 0x7f69900ac7d0 2024-04-02T09:15:06.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.906+0000 7f6996600700 1 --2- 172.21.15.89:0/1133336392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69900ac7d0 0x7f699006c9d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6988009d90 tx=0x7f698800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c78b77b60cb97d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:06.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.906+0000 7f698fe00700 1 -- 172.21.15.89:0/1133336392 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698800e070 con 0x7f69900ac7d0 2024-04-02T09:15:06.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.906+0000 7f698fe00700 1 -- 172.21.15.89:0/1133336392 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698800bb20 con 0x7f69900ac7d0 2024-04-02T09:15:06.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.906+0000 7f698fe00700 1 -- 172.21.15.89:0/1133336392 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6988012430 con 0x7f69900ac7d0 2024-04-02T09:15:06.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.907+0000 7f6998600700 1 -- 172.21.15.89:0/1133336392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69900ac7d0 msgr2=0x7f699006c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:06.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.907+0000 7f6998600700 1 --2- 172.21.15.89:0/1133336392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69900ac7d0 0x7f699006c9d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6988009d90 tx=0x7f698800b330 comp rx=0 tx=0).stop 2024-04-02T09:15:06.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.907+0000 7f6998600700 1 -- 172.21.15.89:0/1133336392 shutdown_connections 2024-04-02T09:15:06.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.907+0000 7f6998600700 1 --2- 172.21.15.89:0/1133336392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69900ac7d0 0x7f699006c9d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.907+0000 7f6998600700 1 --2- 172.21.15.89:0/1133336392 >> v2:172.21.15.113:3300/0 conn(0x7f699015b5d0 0x7f69900ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.907+0000 7f6998600700 1 -- 172.21.15.89:0/1133336392 >> 172.21.15.89:0/1133336392 conn(0x7f69900cf190 msgr2=0x7f69900cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:06.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.907+0000 7f6998600700 1 -- 172.21.15.89:0/1133336392 shutdown_connections 2024-04-02T09:15:06.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.907+0000 7f6998600700 1 -- 172.21.15.89:0/1133336392 wait complete. 2024-04-02T09:15:06.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.909+0000 7f6998600700 1 Processor -- start 2024-04-02T09:15:06.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.909+0000 7f6998600700 1 -- start start 2024-04-02T09:15:06.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6998600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69900ac7d0 0x7f69901569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:06.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6998600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f699015b5d0 0x7f69900cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:06.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6998600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69900cc0e0 0x7f6990159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:06.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6998600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69900ca360 con 0x7f69900ac7d0 2024-04-02T09:15:06.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6998600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69900cd2e0 con 0x7f69900cc0e0 2024-04-02T09:15:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6998600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69900cc9b0 con 0x7f699015b5d0 2024-04-02T09:15:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6995c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69900ac7d0 0x7f69901569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6995c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69900ac7d0 0x7f69901569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54632/0 (socket says 172.21.15.89:54632) 2024-04-02T09:15:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6995c00700 1 -- 172.21.15.89:0/116054246 learned_addr learned my addr 172.21.15.89:0/116054246 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6996600700 1 --2- 172.21.15.89:0/116054246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69900cc0e0 0x7f6990159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6995200700 1 --2- 172.21.15.89:0/116054246 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f699015b5d0 0x7f69900cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6995c00700 1 -- 172.21.15.89:0/116054246 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f699015b5d0 msgr2=0x7f69900cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6995c00700 1 --2- 172.21.15.89:0/116054246 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f699015b5d0 0x7f69900cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6995c00700 1 -- 172.21.15.89:0/116054246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69900cc0e0 msgr2=0x7f6990159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6995c00700 1 --2- 172.21.15.89:0/116054246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69900cc0e0 0x7f6990159050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.910+0000 7f6995c00700 1 -- 172.21.15.89:0/116054246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6988009a70 con 0x7f69900ac7d0 2024-04-02T09:15:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.911+0000 7f6995c00700 1 --2- 172.21.15.89:0/116054246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69900ac7d0 0x7f69901569d0 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f698400e870 tx=0x7f698400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.911+0000 7f698d600700 1 -- 172.21.15.89:0/116054246 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6984009590 con 0x7f69900ac7d0 2024-04-02T09:15:06.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.911+0000 7f698d600700 1 -- 172.21.15.89:0/116054246 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6984013680 con 0x7f69900ac7d0 2024-04-02T09:15:06.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.911+0000 7f6998600700 1 -- 172.21.15.89:0/116054246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69900ccc90 con 0x7f69900ac7d0 2024-04-02T09:15:06.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.911+0000 7f698d600700 1 -- 172.21.15.89:0/116054246 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69840139f0 con 0x7f69900ac7d0 2024-04-02T09:15:06.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.911+0000 7f6998600700 1 -- 172.21.15.89:0/116054246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6990156f10 con 0x7f69900ac7d0 2024-04-02T09:15:06.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.912+0000 7f6998600700 1 -- 172.21.15.89:0/116054246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6990067720 con 0x7f69900ac7d0 2024-04-02T09:15:06.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.913+0000 7f698d600700 1 -- 172.21.15.89:0/116054246 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 175) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f6984013b50 con 0x7f69900ac7d0 2024-04-02T09:15:06.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.914+0000 7f698d600700 1 -- 172.21.15.89:0/116054246 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6888+0+0 (secure 0 0 0) 0x7f6984089160 con 0x7f69900ac7d0 2024-04-02T09:15:06.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:06.918+0000 7f698d600700 1 -- 172.21.15.89:0/116054246 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6984054200 con 0x7f69900ac7d0 2024-04-02T09:15:07.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.121+0000 7f6998600700 1 -- 172.21.15.89:0/116054246 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6990158c70 con 0x7f69900ac7d0 2024-04-02T09:15:07.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.122+0000 7f698d600700 1 -- 172.21.15.89:0/116054246 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v267) v1 ==== 74+0+17792 (secure 0 0 0) 0x7f6984057820 con 0x7f69900ac7d0 2024-04-02T09:15:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:07.121 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":267,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:15:06.589532+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:14:32.705979+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-04-02T09:14:33.330752+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/2246067199":"2024-04-03T09:15:06.589496+0000","172.21.15.89:0/2914640881":"2024-04-03T09:15:06.589496+0000","172.21.15.89:0/585665000":"2024-04-03T09:15:06.589496+0000","172.21.15.89:0/3573435543":"2024-04-03T09:15:06.589496+0000","172.21.15.89:0/1380935513":"2024-04-03T09:15:06.589496+0000","172.21.15.89:0/422001866":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/3534660524":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/1376894516":"2024-04-03T09:15:06.589496+0000","172.21.15.89:0/2610551691":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/1604459128":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/3010462327":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/3737296061":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/1785135719":"2024-04-03T09:14:55.473381+0000","172.21.15.89:6833/108855":"2024-04-03T09:14:55.473381+0000","172.21.15.89:6832/108855":"2024-04-03T09:14:55.473381+0000","172.21.15.89:0/740368670":"2024-04-03T09:15:06.589496+0000","172.21.15.89:0/1017424418":"2024-04-03T09:14:55.473381+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:15:07.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.125+0000 7f6998600700 1 -- 172.21.15.89:0/116054246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69900ac7d0 msgr2=0x7f69901569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:07.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.125+0000 7f6998600700 1 --2- 172.21.15.89:0/116054246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69900ac7d0 0x7f69901569d0 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f698400e870 tx=0x7f698400eb80 comp rx=0 tx=0).stop 2024-04-02T09:15:07.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.125+0000 7f6998600700 1 -- 172.21.15.89:0/116054246 shutdown_connections 2024-04-02T09:15:07.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.125+0000 7f6998600700 1 --2- 172.21.15.89:0/116054246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69900cc0e0 0x7f6990159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.125+0000 7f6998600700 1 --2- 172.21.15.89:0/116054246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69900ac7d0 0x7f69901569d0 unknown :-1 s=CLOSED pgs=2910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.125+0000 7f6998600700 1 --2- 172.21.15.89:0/116054246 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f699015b5d0 0x7f69900cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.125+0000 7f6998600700 1 -- 172.21.15.89:0/116054246 >> 172.21.15.89:0/116054246 conn(0x7f69900cf190 msgr2=0x7f69900aee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:07.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.125+0000 7f6998600700 1 -- 172.21.15.89:0/116054246 shutdown_connections 2024-04-02T09:15:07.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.125+0000 7f6998600700 1 -- 172.21.15.89:0/116054246 wait complete. 2024-04-02T09:15:07.141 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T09:15:07.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.334+0000 7f5c27c00700 1 Processor -- start 2024-04-02T09:15:07.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.335+0000 7f5c27c00700 1 -- start start 2024-04-02T09:15:07.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.336+0000 7f5c27c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5c200cd480 0x7f5c2006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:07.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.336+0000 7f5c27c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c200cdae0 con 0x7f5c200cbb30 2024-04-02T09:15:07.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.336+0000 7f5c27c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c200cddc0 con 0x7f5c200cd020 2024-04-02T09:15:07.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.336+0000 7f5c27c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c200ce0a0 con 0x7f5c200cd480 2024-04-02T09:15:07.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.336+0000 7f5c25c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5c200cd480 0x7f5c2006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:07.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.336+0000 7f5c25200700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5c200cbb30 0x7f5c200cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60000/0 (socket says 172.21.15.89:60000) 2024-04-02T09:15:07.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.336+0000 7f5c25200700 1 -- 172.21.15.89:0/3485639214 learned_addr learned my addr 172.21.15.89:0/3485639214 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:07.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.336+0000 7f5c25c00700 1 -- 172.21.15.89:0/3485639214 >> v1:172.21.15.113:6789/0 conn(0x7f5c200cd020 legacy=0x7f5c200b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:07.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.336+0000 7f5c25c00700 1 -- 172.21.15.89:0/3485639214 >> v1:172.21.15.89:6789/0 conn(0x7f5c200cbb30 legacy=0x7f5c200cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:07.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.336+0000 7f5c25c00700 1 -- 172.21.15.89:0/3485639214 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2006dc20 con 0x7f5c200cd480 2024-04-02T09:15:07.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.336+0000 7f5c1f400700 1 -- 172.21.15.89:0/3485639214 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360930301 0 0) 0x7f5c200cdae0 con 0x7f5c200cbb30 2024-04-02T09:15:07.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.337+0000 7f5c25c00700 1 --2- 172.21.15.89:0/3485639214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c200cd480 0x7f5c2006e420 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f5c18009d90 tx=0x7f5c1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65b6aaebe09dd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:07.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.337+0000 7f5c1f400700 1 -- 172.21.15.89:0/3485639214 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1800e070 con 0x7f5c200cd480 2024-04-02T09:15:07.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.337+0000 7f5c1f400700 1 -- 172.21.15.89:0/3485639214 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1800bbd0 con 0x7f5c200cd480 2024-04-02T09:15:07.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.337+0000 7f5c1f400700 1 -- 172.21.15.89:0/3485639214 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18012430 con 0x7f5c200cd480 2024-04-02T09:15:07.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.337+0000 7f5c27c00700 1 -- 172.21.15.89:0/3485639214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c200cd480 msgr2=0x7f5c2006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:07.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.337+0000 7f5c27c00700 1 --2- 172.21.15.89:0/3485639214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c200cd480 0x7f5c2006e420 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f5c18009d90 tx=0x7f5c1800b330 comp rx=0 tx=0).stop 2024-04-02T09:15:07.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.338+0000 7f5c27c00700 1 -- 172.21.15.89:0/3485639214 shutdown_connections 2024-04-02T09:15:07.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.338+0000 7f5c27c00700 1 --2- 172.21.15.89:0/3485639214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c200cd480 0x7f5c2006e420 unknown :-1 s=CLOSED pgs=2912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.338+0000 7f5c27c00700 1 -- 172.21.15.89:0/3485639214 >> 172.21.15.89:0/3485639214 conn(0x7f5c200b0d70 msgr2=0x7f5c200b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:07.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.338+0000 7f5c27c00700 1 -- 172.21.15.89:0/3485639214 shutdown_connections 2024-04-02T09:15:07.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.338+0000 7f5c27c00700 1 -- 172.21.15.89:0/3485639214 wait complete. 2024-04-02T09:15:07.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.339+0000 7f5c27c00700 1 Processor -- start 2024-04-02T09:15:07.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.339+0000 7f5c27c00700 1 -- start start 2024-04-02T09:15:07.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.340+0000 7f5c27c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c200cbb30 0x7f5c200cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:07.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.340+0000 7f5c27c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c200cd020 0x7f5c201f60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:07.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.340+0000 7f5c27c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c200cd480 0x7f5c200ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:07.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.340+0000 7f5c27c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c201f6600 con 0x7f5c200cd020 2024-04-02T09:15:07.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.340+0000 7f5c27c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2015ab70 con 0x7f5c200cbb30 2024-04-02T09:15:07.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.340+0000 7f5c27c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2015ae50 con 0x7f5c200cd480 2024-04-02T09:15:07.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.340+0000 7f5c1fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c200cd020 0x7f5c201f60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:07.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.341+0000 7f5c1fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c200cd020 0x7f5c201f60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54644/0 (socket says 172.21.15.89:54644) 2024-04-02T09:15:07.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.341+0000 7f5c1fe00700 1 -- 172.21.15.89:0/2181913267 learned_addr learned my addr 172.21.15.89:0/2181913267 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:07.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.341+0000 7f5c25200700 1 --2- 172.21.15.89:0/2181913267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c200cbb30 0x7f5c200cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:07.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.341+0000 7f5c25c00700 1 --2- 172.21.15.89:0/2181913267 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c200cd480 0x7f5c200ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:07.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.341+0000 7f5c1fe00700 1 -- 172.21.15.89:0/2181913267 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c200cd480 msgr2=0x7f5c200ad0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:07.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.341+0000 7f5c1fe00700 1 --2- 172.21.15.89:0/2181913267 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c200cd480 0x7f5c200ad0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.341+0000 7f5c1fe00700 1 -- 172.21.15.89:0/2181913267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c200cbb30 msgr2=0x7f5c200cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:07.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.341+0000 7f5c1fe00700 1 --2- 172.21.15.89:0/2181913267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c200cbb30 0x7f5c200cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.341+0000 7f5c1fe00700 1 -- 172.21.15.89:0/2181913267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c18009a70 con 0x7f5c200cd020 2024-04-02T09:15:07.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.341+0000 7f5c1fe00700 1 --2- 172.21.15.89:0/2181913267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c200cd020 0x7f5c201f60c0 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c00c450 tx=0x7f5c0c00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:07.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.341+0000 7f5c1cc00700 1 -- 172.21.15.89:0/2181913267 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c0052d0 con 0x7f5c200cd020 2024-04-02T09:15:07.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.341+0000 7f5c1cc00700 1 -- 172.21.15.89:0/2181913267 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0c005430 con 0x7f5c200cd020 2024-04-02T09:15:07.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.341+0000 7f5c1cc00700 1 -- 172.21.15.89:0/2181913267 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c0038c0 con 0x7f5c200cd020 2024-04-02T09:15:07.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.342+0000 7f5c27c00700 1 -- 172.21.15.89:0/2181913267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c2015b130 con 0x7f5c200cd020 2024-04-02T09:15:07.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.342+0000 7f5c27c00700 1 -- 172.21.15.89:0/2181913267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c200c8770 con 0x7f5c200cd020 2024-04-02T09:15:07.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.343+0000 7f5c27c00700 1 -- 172.21.15.89:0/2181913267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c20067720 con 0x7f5c200cd020 2024-04-02T09:15:07.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.344+0000 7f5c1cc00700 1 -- 172.21.15.89:0/2181913267 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 175) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f5c0c003a20 con 0x7f5c200cd020 2024-04-02T09:15:07.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.344+0000 7f5c1cc00700 1 -- 172.21.15.89:0/2181913267 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6888+0+0 (secure 0 0 0) 0x7f5c0c088650 con 0x7f5c200cd020 2024-04-02T09:15:07.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.349+0000 7f5c1cc00700 1 -- 172.21.15.89:0/2181913267 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c0c0536f0 con 0x7f5c200cd020 2024-04-02T09:15:07.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.593+0000 7f5c27c00700 1 -- 172.21.15.89:0/2181913267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5c200ce2e0 con 0x7f5c200cd020 2024-04-02T09:15:07.591 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T09:15:07.593+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T09:15:07.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.600+0000 7f5c1cc00700 1 -- 172.21.15.89:0/2181913267 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 176) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f5c0c0531a0 con 0x7f5c200cd020 2024-04-02T09:15:07.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.601+0000 7f5c1cc00700 1 --2- 172.21.15.89:0/2181913267 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c00070920 0x7f5c00072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:07.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.602+0000 7f5c25200700 1 --2- 172.21.15.89:0/2181913267 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c00070920 0x7f5c00072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:07.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.605+0000 7f5c25200700 1 --2- 172.21.15.89:0/2181913267 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c00070920 0x7f5c00072d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5c14000f60 tx=0x7f5c14004a50 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:07.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.612+0000 7f5c1cc00700 1 -- 172.21.15.89:0/2181913267 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v136) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5c0c072d30 con 0x7f5c200cd020 2024-04-02T09:15:07.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.615+0000 7f5c27c00700 1 -- 172.21.15.89:0/2181913267 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c00070920 msgr2=0x7f5c00072d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:07.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.615+0000 7f5c27c00700 1 --2- 172.21.15.89:0/2181913267 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c00070920 0x7f5c00072d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5c14000f60 tx=0x7f5c14004a50 comp rx=0 tx=0).stop 2024-04-02T09:15:07.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.615+0000 7f5c27c00700 1 -- 172.21.15.89:0/2181913267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c200cd020 msgr2=0x7f5c201f60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:07.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.615+0000 7f5c27c00700 1 --2- 172.21.15.89:0/2181913267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c200cd020 0x7f5c201f60c0 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c00c450 tx=0x7f5c0c00c760 comp rx=0 tx=0).stop 2024-04-02T09:15:07.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.615+0000 7f5c27c00700 1 -- 172.21.15.89:0/2181913267 shutdown_connections 2024-04-02T09:15:07.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.615+0000 7f5c27c00700 1 --2- 172.21.15.89:0/2181913267 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c00070920 0x7f5c00072d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.615+0000 7f5c27c00700 1 --2- 172.21.15.89:0/2181913267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c200cbb30 0x7f5c200cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.615+0000 7f5c27c00700 1 --2- 172.21.15.89:0/2181913267 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c200cd480 0x7f5c200ad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.615+0000 7f5c27c00700 1 --2- 172.21.15.89:0/2181913267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c200cd020 0x7f5c201f60c0 unknown :-1 s=CLOSED pgs=2913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.615+0000 7f5c27c00700 1 -- 172.21.15.89:0/2181913267 >> 172.21.15.89:0/2181913267 conn(0x7f5c200b0d70 msgr2=0x7f5c2006e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:07.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.615+0000 7f5c27c00700 1 -- 172.21.15.89:0/2181913267 shutdown_connections 2024-04-02T09:15:07.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.615+0000 7f5c27c00700 1 -- 172.21.15.89:0/2181913267 wait complete. 2024-04-02T09:15:07.615 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T09:15:07.619 INFO:tasks.ceph.mds.c.smithi089.stderr: -5244> 2024-04-02T09:14:40.257+0000 7f9757a00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T09:15:07.632 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T09:15:07.674 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T09:15:07.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.833+0000 7f41c0400700 1 Processor -- start 2024-04-02T09:15:07.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.834+0000 7f41c0400700 1 -- start start 2024-04-02T09:15:07.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.835+0000 7f41c0400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f41b81611b0 0x7f41b815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:07.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.835+0000 7f41c0400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f41b80cee10 0x7f41b8161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:07.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.835+0000 7f41c0400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b80ce270 con 0x7f41b815f4d0 2024-04-02T09:15:07.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.835+0000 7f41c0400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b815b600 con 0x7f41b81611b0 2024-04-02T09:15:07.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.835+0000 7f41c0400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b80d2760 con 0x7f41b80cee10 2024-04-02T09:15:07.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.835+0000 7f41bda00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f41b81611b0 0x7f41b815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:07.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.835+0000 7f41bda00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f41b81611b0 0x7f41b815ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54658/0 (socket says 172.21.15.89:54658) 2024-04-02T09:15:07.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.835+0000 7f41bda00700 1 -- 172.21.15.89:0/4187692262 learned_addr learned my addr 172.21.15.89:0/4187692262 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:07.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.835+0000 7f41be400700 1 --2- 172.21.15.89:0/4187692262 >> v2:172.21.15.113:3300/0 conn(0x7f41b80cee10 0x7f41b8161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:07.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.836+0000 7f41bda00700 1 -- 172.21.15.89:0/4187692262 >> v2:172.21.15.113:3300/0 conn(0x7f41b80cee10 msgr2=0x7f41b8161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:07.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.836+0000 7f41bda00700 1 --2- 172.21.15.89:0/4187692262 >> v2:172.21.15.113:3300/0 conn(0x7f41b80cee10 0x7f41b8161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.836+0000 7f41bda00700 1 -- 172.21.15.89:0/4187692262 >> v1:172.21.15.113:6789/0 conn(0x7f41b815f4d0 legacy=0x7f41b80ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:07.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.836+0000 7f41bda00700 1 -- 172.21.15.89:0/4187692262 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b80d2a40 con 0x7f41b81611b0 2024-04-02T09:15:07.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.836+0000 7f41bda00700 1 --2- 172.21.15.89:0/4187692262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b81611b0 0x7f41b815ef90 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7f41b80d2f60 tx=0x7f41ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=208e890b5954e2df server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:07.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.836+0000 7f41b7e00700 1 -- 172.21.15.89:0/4187692262 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac011830 con 0x7f41b81611b0 2024-04-02T09:15:07.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.836+0000 7f41b7e00700 1 -- 172.21.15.89:0/4187692262 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ac011990 con 0x7f41b81611b0 2024-04-02T09:15:07.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.836+0000 7f41b7e00700 1 -- 172.21.15.89:0/4187692262 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac011d20 con 0x7f41b81611b0 2024-04-02T09:15:07.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.837+0000 7f41c0400700 1 -- 172.21.15.89:0/4187692262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b81611b0 msgr2=0x7f41b815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:07.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.837+0000 7f41c0400700 1 --2- 172.21.15.89:0/4187692262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b81611b0 0x7f41b815ef90 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7f41b80d2f60 tx=0x7f41ac009bd0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.837+0000 7f41c0400700 1 -- 172.21.15.89:0/4187692262 shutdown_connections 2024-04-02T09:15:07.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.837+0000 7f41c0400700 1 --2- 172.21.15.89:0/4187692262 >> v2:172.21.15.113:3300/0 conn(0x7f41b80cee10 0x7f41b8161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.837+0000 7f41c0400700 1 --2- 172.21.15.89:0/4187692262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b81611b0 0x7f41b815ef90 unknown :-1 s=CLOSED pgs=2917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.837+0000 7f41c0400700 1 -- 172.21.15.89:0/4187692262 >> 172.21.15.89:0/4187692262 conn(0x7f41b808aff0 msgr2=0x7f41b80d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:07.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.837+0000 7f41c0400700 1 -- 172.21.15.89:0/4187692262 shutdown_connections 2024-04-02T09:15:07.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.837+0000 7f41c0400700 1 -- 172.21.15.89:0/4187692262 wait complete. 2024-04-02T09:15:07.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.839+0000 7f41c0400700 1 Processor -- start 2024-04-02T09:15:07.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.839+0000 7f41c0400700 1 -- start start 2024-04-02T09:15:07.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.840+0000 7f41c0400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41b80cee10 0x7f41b8152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:07.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41c0400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41b815f4d0 0x7f41b80cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:07.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41c0400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b81611b0 0x7f41b80c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:07.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41c0400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b80cc5f0 con 0x7f41b81611b0 2024-04-02T09:15:07.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41c0400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b80cc030 con 0x7f41b815f4d0 2024-04-02T09:15:07.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41c0400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b80cc310 con 0x7f41b80cee10 2024-04-02T09:15:07.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41be400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b81611b0 0x7f41b80c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:07.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41bd000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41b815f4d0 0x7f41b80cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:07.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41be400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b81611b0 0x7f41b80c93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54674/0 (socket says 172.21.15.89:54674) 2024-04-02T09:15:07.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41be400700 1 -- 172.21.15.89:0/1798120119 learned_addr learned my addr 172.21.15.89:0/1798120119 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:07.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41bda00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41b80cee10 0x7f41b8152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:07.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41be400700 1 -- 172.21.15.89:0/1798120119 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41b80cee10 msgr2=0x7f41b8152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:07.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41be400700 1 --2- 172.21.15.89:0/1798120119 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41b80cee10 0x7f41b8152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41be400700 1 -- 172.21.15.89:0/1798120119 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41b815f4d0 msgr2=0x7f41b80cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:07.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41be400700 1 --2- 172.21.15.89:0/1798120119 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41b815f4d0 0x7f41b80cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:07.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.841+0000 7f41be400700 1 -- 172.21.15.89:0/1798120119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac009550 con 0x7f41b81611b0 2024-04-02T09:15:07.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.842+0000 7f41be400700 1 --2- 172.21.15.89:0/1798120119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b81611b0 0x7f41b80c93c0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f41b000efc0 tx=0x7f41b000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:07.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.842+0000 7f41b5600700 1 -- 172.21.15.89:0/1798120119 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b0009ea0 con 0x7f41b81611b0 2024-04-02T09:15:07.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.842+0000 7f41b5600700 1 -- 172.21.15.89:0/1798120119 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b000f040 con 0x7f41b81611b0 2024-04-02T09:15:07.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.842+0000 7f41b5600700 1 -- 172.21.15.89:0/1798120119 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b000ced0 con 0x7f41b81611b0 2024-04-02T09:15:07.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.842+0000 7f41c0400700 1 -- 172.21.15.89:0/1798120119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b80c7100 con 0x7f41b81611b0 2024-04-02T09:15:07.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.842+0000 7f41c0400700 1 -- 172.21.15.89:0/1798120119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b81529b0 con 0x7f41b81611b0 2024-04-02T09:15:07.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.843+0000 7f41c0400700 1 -- 172.21.15.89:0/1798120119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b8152bd0 con 0x7f41b81611b0 2024-04-02T09:15:07.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.844+0000 7f41b5600700 1 -- 172.21.15.89:0/1798120119 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 176) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f41b0014480 con 0x7f41b81611b0 2024-04-02T09:15:07.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.845+0000 7f41b5600700 1 --2- 172.21.15.89:0/1798120119 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f419806aae0 0x7f419806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:07.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.845+0000 7f41b5600700 1 -- 172.21.15.89:0/1798120119 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6974+0+0 (secure 0 0 0) 0x7f41b0089cf0 con 0x7f41b81611b0 2024-04-02T09:15:07.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.845+0000 7f41bda00700 1 --2- 172.21.15.89:0/1798120119 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f419806aae0 0x7f419806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:07.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.846+0000 7f41bda00700 1 --2- 172.21.15.89:0/1798120119 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f419806aae0 0x7f419806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f41ac014040 tx=0x7f41ac008030 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:07.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:07.850+0000 7f41b5600700 1 -- 172.21.15.89:0/1798120119 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f41b8152bd0 con 0x7f41b81611b0 2024-04-02T09:15:08.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.096+0000 7f41c0400700 1 -- 172.21.15.89:0/1798120119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f41b80c7620 con 0x7f41b81611b0 2024-04-02T09:15:08.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.611+0000 7f41b5600700 1 -- 172.21.15.89:0/1798120119 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v137) v1 ==== 104+0+0 (secure 0 0 0) 0x7f41b80c7620 con 0x7f41b81611b0 2024-04-02T09:15:08.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.614+0000 7f41c0400700 1 -- 172.21.15.89:0/1798120119 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f419806aae0 msgr2=0x7f419806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:08.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.614+0000 7f41c0400700 1 --2- 172.21.15.89:0/1798120119 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f419806aae0 0x7f419806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f41ac014040 tx=0x7f41ac008030 comp rx=0 tx=0).stop 2024-04-02T09:15:08.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.614+0000 7f41c0400700 1 -- 172.21.15.89:0/1798120119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b81611b0 msgr2=0x7f41b80c93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:08.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.614+0000 7f41c0400700 1 --2- 172.21.15.89:0/1798120119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b81611b0 0x7f41b80c93c0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f41b000efc0 tx=0x7f41b000c3a0 comp rx=0 tx=0).stop 2024-04-02T09:15:08.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.614+0000 7f41c0400700 1 -- 172.21.15.89:0/1798120119 shutdown_connections 2024-04-02T09:15:08.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.614+0000 7f41c0400700 1 --2- 172.21.15.89:0/1798120119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b81611b0 0x7f41b80c93c0 unknown :-1 s=CLOSED pgs=2918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:08.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.614+0000 7f41c0400700 1 --2- 172.21.15.89:0/1798120119 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41b80cee10 0x7f41b8152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:08.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.614+0000 7f41c0400700 1 --2- 172.21.15.89:0/1798120119 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f419806aae0 0x7f419806cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:08.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.614+0000 7f41c0400700 1 --2- 172.21.15.89:0/1798120119 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41b815f4d0 0x7f41b80cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:08.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.614+0000 7f41c0400700 1 -- 172.21.15.89:0/1798120119 >> 172.21.15.89:0/1798120119 conn(0x7f41b808aff0 msgr2=0x7f41b806c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:08.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.615+0000 7f41c0400700 1 -- 172.21.15.89:0/1798120119 shutdown_connections 2024-04-02T09:15:08.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.615+0000 7f41c0400700 1 -- 172.21.15.89:0/1798120119 wait complete. 2024-04-02T09:15:08.632 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T09:15:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.868+0000 7f481e200700 1 Processor -- start 2024-04-02T09:15:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.868+0000 7f481e200700 1 -- start start 2024-04-02T09:15:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.869+0000 7f481e200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4818160e10 0x7f4818099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.869+0000 7f481e200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f48180b2380 0x7f4818099fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.869+0000 7f481e200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481809ada0 con 0x7f481809a4e0 2024-04-02T09:15:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.869+0000 7f481e200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48181611f0 con 0x7f4818160e10 2024-04-02T09:15:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.869+0000 7f481e200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481806cc80 con 0x7f48180b2380 2024-04-02T09:15:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.869+0000 7f4817400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4818160e10 0x7f4818099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.870+0000 7f4817400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4818160e10 0x7f4818099a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45598/0 (socket says 172.21.15.89:45598) 2024-04-02T09:15:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.870+0000 7f4817400700 1 -- 172.21.15.89:0/2509783510 learned_addr learned my addr 172.21.15.89:0/2509783510 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.870+0000 7f4817e00700 1 --1- 172.21.15.89:0/2509783510 >> v1:172.21.15.89:6789/0 conn(0x7f481809a4e0 0x7f48180cf530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41830/0 (socket says 172.21.15.89:41830) 2024-04-02T09:15:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.870+0000 7f4816a00700 1 --2- 172.21.15.89:0/2509783510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f48180b2380 0x7f4818099fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.870+0000 7f4817400700 1 -- 172.21.15.89:0/2509783510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f48180b2380 msgr2=0x7f4818099fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.870+0000 7f4817400700 1 --2- 172.21.15.89:0/2509783510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f48180b2380 0x7f4818099fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.870+0000 7f4817400700 1 -- 172.21.15.89:0/2509783510 >> v1:172.21.15.89:6789/0 conn(0x7f481809a4e0 legacy=0x7f48180cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.870+0000 7f4817400700 1 -- 172.21.15.89:0/2509783510 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48180d0840 con 0x7f4818160e10 2024-04-02T09:15:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.870+0000 7f4816000700 1 -- 172.21.15.89:0/2509783510 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606883172 0 0) 0x7f481809ada0 con 0x7f481809a4e0 2024-04-02T09:15:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.870+0000 7f4817400700 1 --2- 172.21.15.89:0/2509783510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4818160e10 0x7f4818099a60 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f48080098f0 tx=0x7f4808009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13985a00201859c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.870+0000 7f4816000700 1 -- 172.21.15.89:0/2509783510 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480800b8a0 con 0x7f4818160e10 2024-04-02T09:15:08.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.871+0000 7f4816000700 1 -- 172.21.15.89:0/2509783510 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480800ba00 con 0x7f4818160e10 2024-04-02T09:15:08.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.871+0000 7f4816000700 1 -- 172.21.15.89:0/2509783510 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480800bd90 con 0x7f4818160e10 2024-04-02T09:15:08.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.871+0000 7f481e200700 1 -- 172.21.15.89:0/2509783510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4818160e10 msgr2=0x7f4818099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:08.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.871+0000 7f481e200700 1 --2- 172.21.15.89:0/2509783510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4818160e10 0x7f4818099a60 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f48080098f0 tx=0x7f4808009c00 comp rx=0 tx=0).stop 2024-04-02T09:15:08.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.871+0000 7f481e200700 1 -- 172.21.15.89:0/2509783510 shutdown_connections 2024-04-02T09:15:08.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.871+0000 7f481e200700 1 --2- 172.21.15.89:0/2509783510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4818160e10 0x7f4818099a60 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:08.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.871+0000 7f481e200700 1 --2- 172.21.15.89:0/2509783510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f48180b2380 0x7f4818099fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:08.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.871+0000 7f481e200700 1 -- 172.21.15.89:0/2509783510 >> 172.21.15.89:0/2509783510 conn(0x7f48180b2e20 msgr2=0x7f48180b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:08.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.871+0000 7f481e200700 1 -- 172.21.15.89:0/2509783510 shutdown_connections 2024-04-02T09:15:08.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.871+0000 7f481e200700 1 -- 172.21.15.89:0/2509783510 wait complete. 2024-04-02T09:15:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.873+0000 7f481e200700 1 Processor -- start 2024-04-02T09:15:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.873+0000 7f481e200700 1 -- start start 2024-04-02T09:15:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.874+0000 7f481e200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f481809a4e0 0x7f4818148380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.874+0000 7f481e200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48180b2380 0x7f4818148ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.874+0000 7f481e200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4818160e10 0x7f4818149000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.874+0000 7f481e200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48180c7a40 con 0x7f48180b2380 2024-04-02T09:15:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.874+0000 7f481e200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481815edf0 con 0x7f4818160e10 2024-04-02T09:15:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.874+0000 7f481e200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481815f0d0 con 0x7f481809a4e0 2024-04-02T09:15:08.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.874+0000 7f4816a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48180b2380 0x7f4818148ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.874+0000 7f4816a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48180b2380 0x7f4818148ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45602/0 (socket says 172.21.15.89:45602) 2024-04-02T09:15:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.874+0000 7f4816a00700 1 -- 172.21.15.89:0/287352258 learned_addr learned my addr 172.21.15.89:0/287352258 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.874+0000 7f4817e00700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4818160e10 0x7f4818149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.874+0000 7f4817400700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f481809a4e0 0x7f4818148380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.875+0000 7f4816a00700 1 -- 172.21.15.89:0/287352258 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f481809a4e0 msgr2=0x7f4818148380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.875+0000 7f4816a00700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f481809a4e0 0x7f4818148380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.875+0000 7f4816a00700 1 -- 172.21.15.89:0/287352258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4818160e10 msgr2=0x7f4818149000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.875+0000 7f4816a00700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4818160e10 0x7f4818149000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.875+0000 7f4816a00700 1 -- 172.21.15.89:0/287352258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4808009550 con 0x7f48180b2380 2024-04-02T09:15:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.875+0000 7f4817e00700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4818160e10 0x7f4818149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.875+0000 7f4816a00700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48180b2380 0x7f4818148ac0 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7f480400e810 tx=0x7f480400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.875+0000 7f480f400700 1 -- 172.21.15.89:0/287352258 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4804013610 con 0x7f48180b2380 2024-04-02T09:15:08.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.875+0000 7f480f400700 1 -- 172.21.15.89:0/287352258 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4804013770 con 0x7f48180b2380 2024-04-02T09:15:08.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.875+0000 7f480f400700 1 -- 172.21.15.89:0/287352258 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4804013b20 con 0x7f48180b2380 2024-04-02T09:15:08.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.875+0000 7f481e200700 1 -- 172.21.15.89:0/287352258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481815f3b0 con 0x7f48180b2380 2024-04-02T09:15:08.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.875+0000 7f481e200700 1 -- 172.21.15.89:0/287352258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f481815ae40 con 0x7f48180b2380 2024-04-02T09:15:08.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.876+0000 7f481e200700 1 -- 172.21.15.89:0/287352258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481815ec50 con 0x7f48180b2380 2024-04-02T09:15:08.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.877+0000 7f480f400700 1 -- 172.21.15.89:0/287352258 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 176) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f4804013c80 con 0x7f48180b2380 2024-04-02T09:15:08.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.878+0000 7f480f400700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f47f806aae0 0x7f47f806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:08.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.878+0000 7f480f400700 1 -- 172.21.15.89:0/287352258 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6974+0+0 (secure 0 0 0) 0x7f4804089150 con 0x7f48180b2380 2024-04-02T09:15:08.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.878+0000 7f4817400700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f47f806aae0 0x7f47f806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:08.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.879+0000 7f4817400700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f47f806aae0 0x7f47f806cfa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4808014040 tx=0x7f4808008030 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:08.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:08.882+0000 7f480f400700 1 -- 172.21.15.89:0/287352258 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f481815ec50 con 0x7f48180b2380 2024-04-02T09:15:09.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.083+0000 7f481e200700 1 -- 172.21.15.89:0/287352258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f481815ec50 con 0x7f48180b2380 2024-04-02T09:15:09.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.539+0000 7f480f400700 1 -- 172.21.15.89:0/287352258 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 177) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f48040544d0 con 0x7f48180b2380 2024-04-02T09:15:09.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.626+0000 7f480f400700 1 -- 172.21.15.89:0/287352258 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v269) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4804057b90 con 0x7f48180b2380 2024-04-02T09:15:09.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.630+0000 7f481e200700 1 -- 172.21.15.89:0/287352258 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f47f806aae0 msgr2=0x7f47f806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:09.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.630+0000 7f481e200700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f47f806aae0 0x7f47f806cfa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4808014040 tx=0x7f4808008030 comp rx=0 tx=0).stop 2024-04-02T09:15:09.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.630+0000 7f481e200700 1 -- 172.21.15.89:0/287352258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48180b2380 msgr2=0x7f4818148ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:09.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.630+0000 7f481e200700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48180b2380 0x7f4818148ac0 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7f480400e810 tx=0x7f480400ebd0 comp rx=0 tx=0).stop 2024-04-02T09:15:09.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.630+0000 7f481e200700 1 -- 172.21.15.89:0/287352258 shutdown_connections 2024-04-02T09:15:09.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.630+0000 7f481e200700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f47f806aae0 0x7f47f806cfa0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:09.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.630+0000 7f481e200700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4818160e10 0x7f4818149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:09.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.630+0000 7f481e200700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f481809a4e0 0x7f4818148380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:09.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.630+0000 7f481e200700 1 --2- 172.21.15.89:0/287352258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48180b2380 0x7f4818148ac0 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:09.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.630+0000 7f481e200700 1 -- 172.21.15.89:0/287352258 >> 172.21.15.89:0/287352258 conn(0x7f48180b2e20 msgr2=0x7f481815c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:09.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.631+0000 7f481e200700 1 -- 172.21.15.89:0/287352258 shutdown_connections 2024-04-02T09:15:09.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.631+0000 7f481e200700 1 -- 172.21.15.89:0/287352258 wait complete. 2024-04-02T09:15:09.634 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T09:15:09.656 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T09:15:09.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.854+0000 7f14fbc00700 1 Processor -- start 2024-04-02T09:15:09.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.854+0000 7f14fbc00700 1 -- start start 2024-04-02T09:15:09.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.856+0000 7f14fbc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f14f415ddf0 0x7f14f40ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:09.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.856+0000 7f14fbc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f14f415e1d0 0x7f14f40d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:09.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.856+0000 7f14fbc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f40d2b40 con 0x7f14f40d2710 2024-04-02T09:15:09.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.856+0000 7f14fbc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f415ec60 con 0x7f14f415e1d0 2024-04-02T09:15:09.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.856+0000 7f14fbc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f40ce8a0 con 0x7f14f415ddf0 2024-04-02T09:15:09.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.856+0000 7f14f3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f14f415e1d0 0x7f14f40d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:09.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.856+0000 7f14f3e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f14f415e1d0 0x7f14f40d21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45616/0 (socket says 172.21.15.89:45616) 2024-04-02T09:15:09.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.856+0000 7f14f3e00700 1 -- 172.21.15.89:0/105930409 learned_addr learned my addr 172.21.15.89:0/105930409 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:09.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.856+0000 7f14f9200700 1 --2- 172.21.15.89:0/105930409 >> v2:172.21.15.113:3300/0 conn(0x7f14f415ddf0 0x7f14f40ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:09.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.856+0000 7f14f3e00700 1 -- 172.21.15.89:0/105930409 >> v2:172.21.15.113:3300/0 conn(0x7f14f415ddf0 msgr2=0x7f14f40ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:09.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.856+0000 7f14f3e00700 1 --2- 172.21.15.89:0/105930409 >> v2:172.21.15.113:3300/0 conn(0x7f14f415ddf0 0x7f14f40ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:09.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.856+0000 7f14f3e00700 1 -- 172.21.15.89:0/105930409 >> v1:172.21.15.113:6789/0 conn(0x7f14f40d2710 legacy=0x7f14f41612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:09.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.856+0000 7f14f3e00700 1 -- 172.21.15.89:0/105930409 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f40cdc90 con 0x7f14f415e1d0 2024-04-02T09:15:09.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.857+0000 7f14f3e00700 1 --2- 172.21.15.89:0/105930409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14f415e1d0 0x7f14f40d21d0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f14f406b3e0 tx=0x7f14e0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c967d473d9bde2f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:09.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.857+0000 7f14f3400700 1 -- 172.21.15.89:0/105930409 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e0004230 con 0x7f14f415e1d0 2024-04-02T09:15:09.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.857+0000 7f14f3400700 1 -- 172.21.15.89:0/105930409 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e0004390 con 0x7f14f415e1d0 2024-04-02T09:15:09.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.857+0000 7f14f3400700 1 -- 172.21.15.89:0/105930409 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e00036a0 con 0x7f14f415e1d0 2024-04-02T09:15:09.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.857+0000 7f14fbc00700 1 -- 172.21.15.89:0/105930409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14f415e1d0 msgr2=0x7f14f40d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:09.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.857+0000 7f14fbc00700 1 --2- 172.21.15.89:0/105930409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14f415e1d0 0x7f14f40d21d0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f14f406b3e0 tx=0x7f14e0009bd0 comp rx=0 tx=0).stop 2024-04-02T09:15:09.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.858+0000 7f14fbc00700 1 -- 172.21.15.89:0/105930409 shutdown_connections 2024-04-02T09:15:09.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.858+0000 7f14fbc00700 1 --2- 172.21.15.89:0/105930409 >> v2:172.21.15.113:3300/0 conn(0x7f14f415ddf0 0x7f14f40ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:09.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.858+0000 7f14fbc00700 1 --2- 172.21.15.89:0/105930409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14f415e1d0 0x7f14f40d21d0 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:09.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.858+0000 7f14fbc00700 1 -- 172.21.15.89:0/105930409 >> 172.21.15.89:0/105930409 conn(0x7f14f408aff0 msgr2=0x7f14f40d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:09.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.858+0000 7f14fbc00700 1 -- 172.21.15.89:0/105930409 shutdown_connections 2024-04-02T09:15:09.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.858+0000 7f14fbc00700 1 -- 172.21.15.89:0/105930409 wait complete. 2024-04-02T09:15:09.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.859+0000 7f14fbc00700 1 Processor -- start 2024-04-02T09:15:09.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.859+0000 7f14fbc00700 1 -- start start 2024-04-02T09:15:09.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.860+0000 7f14fbc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14f40d2710 0x7f14f4156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:09.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.860+0000 7f14fbc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14f415ddf0 0x7f14f40c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:09.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.860+0000 7f14fbc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14f415e1d0 0x7f14f40c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:09.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.860+0000 7f14fbc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40cc300 con 0x7f14f415ddf0 2024-04-02T09:15:09.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.860+0000 7f14fbc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40cb980 con 0x7f14f40d2710 2024-04-02T09:15:09.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.860+0000 7f14fbc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40cbc40 con 0x7f14f415e1d0 2024-04-02T09:15:09.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.860+0000 7f14f3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14f415ddf0 0x7f14f40c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:09.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14f3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14f415ddf0 0x7f14f40c7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45630/0 (socket says 172.21.15.89:45630) 2024-04-02T09:15:09.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14f3e00700 1 -- 172.21.15.89:0/2291561177 learned_addr learned my addr 172.21.15.89:0/2291561177 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:09.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14f9200700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14f40d2710 0x7f14f4156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:09.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14f9c00700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14f415e1d0 0x7f14f40c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:09.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14f3e00700 1 -- 172.21.15.89:0/2291561177 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14f415e1d0 msgr2=0x7f14f40c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:09.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14f3e00700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14f415e1d0 0x7f14f40c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:09.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14f3e00700 1 -- 172.21.15.89:0/2291561177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14f40d2710 msgr2=0x7f14f4156140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:09.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14f3e00700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14f40d2710 0x7f14f4156140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:09.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14f3e00700 1 -- 172.21.15.89:0/2291561177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e0009550 con 0x7f14f415ddf0 2024-04-02T09:15:09.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14f9c00700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14f415e1d0 0x7f14f40c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:09.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14f3e00700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14f415ddf0 0x7f14f40c7020 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f14e0009130 tx=0x7f14e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:09.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14f0c00700 1 -- 172.21.15.89:0/2291561177 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e0003e20 con 0x7f14f415ddf0 2024-04-02T09:15:09.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14f0c00700 1 -- 172.21.15.89:0/2291561177 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e0011630 con 0x7f14f415ddf0 2024-04-02T09:15:09.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14f0c00700 1 -- 172.21.15.89:0/2291561177 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e00119e0 con 0x7f14f415ddf0 2024-04-02T09:15:09.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.861+0000 7f14fbc00700 1 -- 172.21.15.89:0/2291561177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f40cbf20 con 0x7f14f415ddf0 2024-04-02T09:15:09.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.862+0000 7f14fbc00700 1 -- 172.21.15.89:0/2291561177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f40cc720 con 0x7f14f415ddf0 2024-04-02T09:15:09.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.862+0000 7f14fbc00700 1 -- 172.21.15.89:0/2291561177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f4067720 con 0x7f14f415ddf0 2024-04-02T09:15:09.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.863+0000 7f14f0c00700 1 -- 172.21.15.89:0/2291561177 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 177) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f14e0011b40 con 0x7f14f415ddf0 2024-04-02T09:15:09.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.864+0000 7f14f0c00700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f14d406ad20 0x7f14d406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:09.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.864+0000 7f14f0c00700 1 -- 172.21.15.89:0/2291561177 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6536+0+0 (secure 0 0 0) 0x7f14e008ab40 con 0x7f14f415ddf0 2024-04-02T09:15:09.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.864+0000 7f14f9200700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f14d406ad20 0x7f14d406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:09.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.865+0000 7f14f9200700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f14d406ad20 0x7f14d406d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f14e8006b20 tx=0x7f14e8008040 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:09.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:09.868+0000 7f14f0c00700 1 -- 172.21.15.89:0/2291561177 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f14e0055a00 con 0x7f14f415ddf0 2024-04-02T09:15:10.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.070+0000 7f14fbc00700 1 -- 172.21.15.89:0/2291561177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f14f40d3050 con 0x7f14f415ddf0 2024-04-02T09:15:10.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.641+0000 7f14f0c00700 1 -- 172.21.15.89:0/2291561177 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v270) v1 ==== 169+0+0 (secure 0 0 0) 0x7f14e0059020 con 0x7f14f415ddf0 2024-04-02T09:15:10.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.644+0000 7f14fbc00700 1 -- 172.21.15.89:0/2291561177 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f14d406ad20 msgr2=0x7f14d406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:10.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.644+0000 7f14fbc00700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f14d406ad20 0x7f14d406d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f14e8006b20 tx=0x7f14e8008040 comp rx=0 tx=0).stop 2024-04-02T09:15:10.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.644+0000 7f14fbc00700 1 -- 172.21.15.89:0/2291561177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14f415ddf0 msgr2=0x7f14f40c7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:10.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.644+0000 7f14fbc00700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14f415ddf0 0x7f14f40c7020 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f14e0009130 tx=0x7f14e0004750 comp rx=0 tx=0).stop 2024-04-02T09:15:10.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.644+0000 7f14fbc00700 1 -- 172.21.15.89:0/2291561177 shutdown_connections 2024-04-02T09:15:10.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.644+0000 7f14fbc00700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f14d406ad20 0x7f14d406d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:10.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.644+0000 7f14fbc00700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14f40d2710 0x7f14f4156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:10.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.644+0000 7f14fbc00700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f14f415e1d0 0x7f14f40c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:10.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.644+0000 7f14fbc00700 1 --2- 172.21.15.89:0/2291561177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14f415ddf0 0x7f14f40c7020 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:10.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.644+0000 7f14fbc00700 1 -- 172.21.15.89:0/2291561177 >> 172.21.15.89:0/2291561177 conn(0x7f14f408aff0 msgr2=0x7f14f40b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:10.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.644+0000 7f14fbc00700 1 -- 172.21.15.89:0/2291561177 shutdown_connections 2024-04-02T09:15:10.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.644+0000 7f14fbc00700 1 -- 172.21.15.89:0/2291561177 wait complete. 2024-04-02T09:15:10.644 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T09:15:10.663 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T09:15:10.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.863+0000 7f1f3c600700 1 Processor -- start 2024-04-02T09:15:10.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.863+0000 7f1f3c600700 1 -- start start 2024-04-02T09:15:10.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.864+0000 7f1f3c600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f3415d9d0 0x7f1f340d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:10.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.864+0000 7f1f3c600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1f340d16a0 0x7f1f340d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:10.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.864+0000 7f1f3c600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f34161420 con 0x7f1f340d2370 2024-04-02T09:15:10.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.864+0000 7f1f3c600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f340d2820 con 0x7f1f340d16a0 2024-04-02T09:15:10.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.864+0000 7f1f3c600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f34161700 con 0x7f1f3415d9d0 2024-04-02T09:15:10.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.864+0000 7f1f39200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1f340d16a0 0x7f1f340d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:10.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.864+0000 7f1f39200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1f340d16a0 0x7f1f340d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45638/0 (socket says 172.21.15.89:45638) 2024-04-02T09:15:10.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.864+0000 7f1f39200700 1 -- 172.21.15.89:0/4039865649 learned_addr learned my addr 172.21.15.89:0/4039865649 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:10.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.865+0000 7f1f39200700 1 -- 172.21.15.89:0/4039865649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f3415d9d0 msgr2=0x7f1f340d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:10.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.865+0000 7f1f39c00700 1 --2- 172.21.15.89:0/4039865649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f3415d9d0 0x7f1f340d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:10.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.865+0000 7f1f39200700 1 --2- 172.21.15.89:0/4039865649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f3415d9d0 0x7f1f340d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:10.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.865+0000 7f1f39200700 1 -- 172.21.15.89:0/4039865649 >> v1:172.21.15.113:6789/0 conn(0x7f1f340d2370 legacy=0x7f1f3415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:10.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.865+0000 7f1f39200700 1 -- 172.21.15.89:0/4039865649 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f341619e0 con 0x7f1f340d16a0 2024-04-02T09:15:10.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.865+0000 7f1f39c00700 1 --2- 172.21.15.89:0/4039865649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f3415d9d0 0x7f1f340d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:15:10.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.865+0000 7f1f39200700 1 --2- 172.21.15.89:0/4039865649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f340d16a0 0x7f1f340d1e30 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f1f20006fd0 tx=0x7f1f20009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab5a58dac3bef49 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:10.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.865+0000 7f1f33e00700 1 -- 172.21.15.89:0/4039865649 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20005e30 con 0x7f1f340d16a0 2024-04-02T09:15:10.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.865+0000 7f1f33e00700 1 -- 172.21.15.89:0/4039865649 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f20006030 con 0x7f1f340d16a0 2024-04-02T09:15:10.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.865+0000 7f1f33e00700 1 -- 172.21.15.89:0/4039865649 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f200063c0 con 0x7f1f340d16a0 2024-04-02T09:15:10.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.866+0000 7f1f3c600700 1 -- 172.21.15.89:0/4039865649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f340d16a0 msgr2=0x7f1f340d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:10.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.866+0000 7f1f3c600700 1 --2- 172.21.15.89:0/4039865649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f340d16a0 0x7f1f340d1e30 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f1f20006fd0 tx=0x7f1f20009bd0 comp rx=0 tx=0).stop 2024-04-02T09:15:10.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.866+0000 7f1f3c600700 1 -- 172.21.15.89:0/4039865649 shutdown_connections 2024-04-02T09:15:10.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.866+0000 7f1f3c600700 1 --2- 172.21.15.89:0/4039865649 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f3415d9d0 0x7f1f340d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:10.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.866+0000 7f1f3c600700 1 --2- 172.21.15.89:0/4039865649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f340d16a0 0x7f1f340d1e30 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:10.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.866+0000 7f1f3c600700 1 -- 172.21.15.89:0/4039865649 >> 172.21.15.89:0/4039865649 conn(0x7f1f3408aff0 msgr2=0x7f1f340d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:10.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.866+0000 7f1f3c600700 1 -- 172.21.15.89:0/4039865649 shutdown_connections 2024-04-02T09:15:10.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.866+0000 7f1f3c600700 1 -- 172.21.15.89:0/4039865649 wait complete. 2024-04-02T09:15:10.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.868+0000 7f1f3c600700 1 Processor -- start 2024-04-02T09:15:10.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.868+0000 7f1f3c600700 1 -- start start 2024-04-02T09:15:10.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.868+0000 7f1f3c600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f340d16a0 0x7f1f34160700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:10.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f3c600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f340d2370 0x7f1f34158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:10.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f3c600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f3415d9d0 0x7f1f340b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:10.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f3c600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f341e84e0 con 0x7f1f340d2370 2024-04-02T09:15:10.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f3c600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f341e8780 con 0x7f1f340d16a0 2024-04-02T09:15:10.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f3c600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f341e8a60 con 0x7f1f3415d9d0 2024-04-02T09:15:10.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f39200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f340d2370 0x7f1f34158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:10.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f39200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f340d2370 0x7f1f34158900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45650/0 (socket says 172.21.15.89:45650) 2024-04-02T09:15:10.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f39200700 1 -- 172.21.15.89:0/2823663393 learned_addr learned my addr 172.21.15.89:0/2823663393 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:10.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f3a600700 1 --2- 172.21.15.89:0/2823663393 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f3415d9d0 0x7f1f340b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:10.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f39c00700 1 --2- 172.21.15.89:0/2823663393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f340d16a0 0x7f1f34160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:10.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f39200700 1 -- 172.21.15.89:0/2823663393 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f3415d9d0 msgr2=0x7f1f340b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:10.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f39200700 1 --2- 172.21.15.89:0/2823663393 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f3415d9d0 0x7f1f340b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:10.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f39200700 1 -- 172.21.15.89:0/2823663393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f340d16a0 msgr2=0x7f1f34160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:10.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f39200700 1 --2- 172.21.15.89:0/2823663393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f340d16a0 0x7f1f34160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:10.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f39200700 1 -- 172.21.15.89:0/2823663393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f20009500 con 0x7f1f340d2370 2024-04-02T09:15:10.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.869+0000 7f1f39200700 1 --2- 172.21.15.89:0/2823663393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f340d2370 0x7f1f34158900 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009c10 tx=0x7f1f20004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:10.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.870+0000 7f1f31600700 1 -- 172.21.15.89:0/2823663393 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20025480 con 0x7f1f340d2370 2024-04-02T09:15:10.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.870+0000 7f1f31600700 1 -- 172.21.15.89:0/2823663393 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f20020070 con 0x7f1f340d2370 2024-04-02T09:15:10.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.870+0000 7f1f3c600700 1 -- 172.21.15.89:0/2823663393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f341e8d40 con 0x7f1f340d2370 2024-04-02T09:15:10.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.870+0000 7f1f31600700 1 -- 172.21.15.89:0/2823663393 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20025820 con 0x7f1f340d2370 2024-04-02T09:15:10.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.870+0000 7f1f3c600700 1 -- 172.21.15.89:0/2823663393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f34160c40 con 0x7f1f340d2370 2024-04-02T09:15:10.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.871+0000 7f1f3c600700 1 -- 172.21.15.89:0/2823663393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f340d1ca0 con 0x7f1f340d2370 2024-04-02T09:15:10.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.872+0000 7f1f31600700 1 -- 172.21.15.89:0/2823663393 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 177) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f1f20022030 con 0x7f1f340d2370 2024-04-02T09:15:10.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.873+0000 7f1f31600700 1 --2- 172.21.15.89:0/2823663393 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1f1406ad20 0x7f1f1406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:10.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.873+0000 7f1f31600700 1 -- 172.21.15.89:0/2823663393 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f1f2008abb0 con 0x7f1f340d2370 2024-04-02T09:15:10.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.873+0000 7f1f39c00700 1 --2- 172.21.15.89:0/2823663393 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1f1406ad20 0x7f1f1406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:10.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.873+0000 7f1f39c00700 1 --2- 172.21.15.89:0/2823663393 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1f1406ad20 0x7f1f1406d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1f28006fd0 tx=0x7f1f28008040 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:10.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:10.877+0000 7f1f31600700 1 -- 172.21.15.89:0/2823663393 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1f20055b90 con 0x7f1f340d2370 2024-04-02T09:15:11.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.076+0000 7f1f3c600700 1 -- 172.21.15.89:0/2823663393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f1f341e9260 con 0x7f1f340d2370 2024-04-02T09:15:11.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.080+0000 7f1f31600700 1 -- 172.21.15.89:0/2823663393 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v91) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1f341e9260 con 0x7f1f340d2370 2024-04-02T09:15:11.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.083+0000 7f1f3c600700 1 -- 172.21.15.89:0/2823663393 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1f1406ad20 msgr2=0x7f1f1406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:11.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.083+0000 7f1f3c600700 1 --2- 172.21.15.89:0/2823663393 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1f1406ad20 0x7f1f1406d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1f28006fd0 tx=0x7f1f28008040 comp rx=0 tx=0).stop 2024-04-02T09:15:11.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.083+0000 7f1f3c600700 1 -- 172.21.15.89:0/2823663393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f340d2370 msgr2=0x7f1f34158900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:11.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.083+0000 7f1f3c600700 1 --2- 172.21.15.89:0/2823663393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f340d2370 0x7f1f34158900 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009c10 tx=0x7f1f20004750 comp rx=0 tx=0).stop 2024-04-02T09:15:11.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.083+0000 7f1f3c600700 1 -- 172.21.15.89:0/2823663393 shutdown_connections 2024-04-02T09:15:11.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.083+0000 7f1f3c600700 1 --2- 172.21.15.89:0/2823663393 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1f1406ad20 0x7f1f1406d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.083+0000 7f1f3c600700 1 --2- 172.21.15.89:0/2823663393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f340d16a0 0x7f1f34160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.083+0000 7f1f3c600700 1 --2- 172.21.15.89:0/2823663393 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f3415d9d0 0x7f1f340b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.083+0000 7f1f3c600700 1 --2- 172.21.15.89:0/2823663393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f340d2370 0x7f1f34158900 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.083+0000 7f1f3c600700 1 -- 172.21.15.89:0/2823663393 >> 172.21.15.89:0/2823663393 conn(0x7f1f3408aff0 msgr2=0x7f1f3415bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:11.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.084+0000 7f1f3c600700 1 -- 172.21.15.89:0/2823663393 shutdown_connections 2024-04-02T09:15:11.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.084+0000 7f1f3c600700 1 -- 172.21.15.89:0/2823663393 wait complete. 2024-04-02T09:15:11.100 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T09:15:11.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.293+0000 7ffadb400700 1 Processor -- start 2024-04-02T09:15:11.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.293+0000 7ffadb400700 1 -- start start 2024-04-02T09:15:11.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.295+0000 7ffadb400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffad415d9d0 0x7ffad40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:11.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.295+0000 7ffadb400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ffad40d16a0 0x7ffad40d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:11.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.295+0000 7ffadb400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad4161420 con 0x7ffad40d2370 2024-04-02T09:15:11.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.295+0000 7ffadb400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad40d27e0 con 0x7ffad415d9d0 2024-04-02T09:15:11.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.295+0000 7ffadb400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad4161700 con 0x7ffad40d16a0 2024-04-02T09:15:11.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.295+0000 7ffad8a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffad415d9d0 0x7ffad40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.295+0000 7ffad8a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffad415d9d0 0x7ffad40d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45664/0 (socket says 172.21.15.89:45664) 2024-04-02T09:15:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.295+0000 7ffad8a00700 1 -- 172.21.15.89:0/1465907687 learned_addr learned my addr 172.21.15.89:0/1465907687 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.295+0000 7ffad3e00700 1 --2- 172.21.15.89:0/1465907687 >> v2:172.21.15.113:3300/0 conn(0x7ffad40d16a0 0x7ffad40d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.295+0000 7ffad8a00700 1 -- 172.21.15.89:0/1465907687 >> v2:172.21.15.113:3300/0 conn(0x7ffad40d16a0 msgr2=0x7ffad40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.295+0000 7ffad8a00700 1 --2- 172.21.15.89:0/1465907687 >> v2:172.21.15.113:3300/0 conn(0x7ffad40d16a0 0x7ffad40d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.295+0000 7ffad8a00700 1 -- 172.21.15.89:0/1465907687 >> v1:172.21.15.113:6789/0 conn(0x7ffad40d2370 legacy=0x7ffad415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.295+0000 7ffad8a00700 1 -- 172.21.15.89:0/1465907687 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad41619e0 con 0x7ffad415d9d0 2024-04-02T09:15:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.295+0000 7ffad3e00700 1 --2- 172.21.15.89:0/1465907687 >> v2:172.21.15.113:3300/0 conn(0x7ffad40d16a0 0x7ffad40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.296+0000 7ffad8a00700 1 --2- 172.21.15.89:0/1465907687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad415d9d0 0x7ffad40d3c30 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7ffad40d12f0 tx=0x7ffac8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d864afe21340422 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.296+0000 7ffad3400700 1 -- 172.21.15.89:0/1465907687 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac800b870 con 0x7ffad415d9d0 2024-04-02T09:15:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.296+0000 7ffad3400700 1 -- 172.21.15.89:0/1465907687 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac800b9d0 con 0x7ffad415d9d0 2024-04-02T09:15:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.296+0000 7ffad3400700 1 -- 172.21.15.89:0/1465907687 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac800bd60 con 0x7ffad415d9d0 2024-04-02T09:15:11.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.296+0000 7ffadb400700 1 -- 172.21.15.89:0/1465907687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad415d9d0 msgr2=0x7ffad40d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:11.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.296+0000 7ffadb400700 1 --2- 172.21.15.89:0/1465907687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad415d9d0 0x7ffad40d3c30 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7ffad40d12f0 tx=0x7ffac8009bd0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.296+0000 7ffadb400700 1 -- 172.21.15.89:0/1465907687 shutdown_connections 2024-04-02T09:15:11.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.296+0000 7ffadb400700 1 --2- 172.21.15.89:0/1465907687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad415d9d0 0x7ffad40d3c30 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.296+0000 7ffadb400700 1 --2- 172.21.15.89:0/1465907687 >> v2:172.21.15.113:3300/0 conn(0x7ffad40d16a0 0x7ffad40d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.296+0000 7ffadb400700 1 -- 172.21.15.89:0/1465907687 >> 172.21.15.89:0/1465907687 conn(0x7ffad408aff0 msgr2=0x7ffad40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:11.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.297+0000 7ffadb400700 1 -- 172.21.15.89:0/1465907687 shutdown_connections 2024-04-02T09:15:11.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.297+0000 7ffadb400700 1 -- 172.21.15.89:0/1465907687 wait complete. 2024-04-02T09:15:11.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.298+0000 7ffadb400700 1 Processor -- start 2024-04-02T09:15:11.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.298+0000 7ffadb400700 1 -- start start 2024-04-02T09:15:11.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.299+0000 7ffadb400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffad40d16a0 0x7ffad40b7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:11.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.299+0000 7ffadb400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad40d2370 0x7ffad41580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:11.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.299+0000 7ffadb400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad415d9d0 0x7ffad40b8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:11.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.299+0000 7ffadb400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad40b6fd0 con 0x7ffad415d9d0 2024-04-02T09:15:11.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.299+0000 7ffadb400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad40b79a0 con 0x7ffad40d16a0 2024-04-02T09:15:11.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.299+0000 7ffadb400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad40b63c0 con 0x7ffad40d2370 2024-04-02T09:15:11.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.299+0000 7ffad9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad415d9d0 0x7ffad40b8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:11.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.300+0000 7ffad9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad415d9d0 0x7ffad40b8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45670/0 (socket says 172.21.15.89:45670) 2024-04-02T09:15:11.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.300+0000 7ffad9400700 1 -- 172.21.15.89:0/3457833055 learned_addr learned my addr 172.21.15.89:0/3457833055 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.300+0000 7ffad8a00700 1 --2- 172.21.15.89:0/3457833055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffad40d16a0 0x7ffad40b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.300+0000 7ffad3e00700 1 --2- 172.21.15.89:0/3457833055 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad40d2370 0x7ffad41580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.300+0000 7ffad9400700 1 -- 172.21.15.89:0/3457833055 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad40d2370 msgr2=0x7ffad41580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.300+0000 7ffad9400700 1 --2- 172.21.15.89:0/3457833055 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad40d2370 0x7ffad41580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.300+0000 7ffad9400700 1 -- 172.21.15.89:0/3457833055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffad40d16a0 msgr2=0x7ffad40b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.300+0000 7ffad9400700 1 --2- 172.21.15.89:0/3457833055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffad40d16a0 0x7ffad40b7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.300+0000 7ffad9400700 1 -- 172.21.15.89:0/3457833055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac8009550 con 0x7ffad415d9d0 2024-04-02T09:15:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.300+0000 7ffad9400700 1 --2- 172.21.15.89:0/3457833055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad415d9d0 0x7ffad40b8250 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00cc50 tx=0x7ffacc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.300+0000 7ffad0c00700 1 -- 172.21.15.89:0/3457833055 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc003c00 con 0x7ffad415d9d0 2024-04-02T09:15:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.300+0000 7ffad0c00700 1 -- 172.21.15.89:0/3457833055 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffacc003d60 con 0x7ffad415d9d0 2024-04-02T09:15:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.301+0000 7ffadb400700 1 -- 172.21.15.89:0/3457833055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad40b66a0 con 0x7ffad415d9d0 2024-04-02T09:15:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.301+0000 7ffad0c00700 1 -- 172.21.15.89:0/3457833055 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc00aab0 con 0x7ffad415d9d0 2024-04-02T09:15:11.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.301+0000 7ffadb400700 1 -- 172.21.15.89:0/3457833055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad40b6c60 con 0x7ffad415d9d0 2024-04-02T09:15:11.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.302+0000 7ffadb400700 1 -- 172.21.15.89:0/3457833055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad40d1ca0 con 0x7ffad415d9d0 2024-04-02T09:15:11.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.303+0000 7ffad0c00700 1 -- 172.21.15.89:0/3457833055 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 177) v1 ==== 89183+0+0 (secure 0 0 0) 0x7ffacc00ac10 con 0x7ffad415d9d0 2024-04-02T09:15:11.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.303+0000 7ffad0c00700 1 --2- 172.21.15.89:0/3457833055 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffab406ad20 0x7ffab406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:11.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.303+0000 7ffad0c00700 1 -- 172.21.15.89:0/3457833055 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6122+0+0 (secure 0 0 0) 0x7ffacc089540 con 0x7ffad415d9d0 2024-04-02T09:15:11.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.304+0000 7ffad8a00700 1 --2- 172.21.15.89:0/3457833055 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffab406ad20 0x7ffab406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:11.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.304+0000 7ffad8a00700 1 --2- 172.21.15.89:0/3457833055 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffab406ad20 0x7ffab406d1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffac8014040 tx=0x7ffac8008030 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:11.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.308+0000 7ffad0c00700 1 -- 172.21.15.89:0/3457833055 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffacc054490 con 0x7ffad415d9d0 2024-04-02T09:15:11.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.522+0000 7ffadb400700 1 -- 172.21.15.89:0/3457833055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7ffad406b9b0 con 0x7ffad415d9d0 2024-04-02T09:15:11.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.527+0000 7ffad0c00700 1 -- 172.21.15.89:0/3457833055 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v92) v1 ==== 120+0+0 (secure 0 0 0) 0x7ffacc057ab0 con 0x7ffad415d9d0 2024-04-02T09:15:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.529+0000 7ffadb400700 1 -- 172.21.15.89:0/3457833055 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffab406ad20 msgr2=0x7ffab406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.529+0000 7ffadb400700 1 --2- 172.21.15.89:0/3457833055 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffab406ad20 0x7ffab406d1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffac8014040 tx=0x7ffac8008030 comp rx=0 tx=0).stop 2024-04-02T09:15:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.529+0000 7ffadb400700 1 -- 172.21.15.89:0/3457833055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad415d9d0 msgr2=0x7ffad40b8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.529+0000 7ffadb400700 1 --2- 172.21.15.89:0/3457833055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad415d9d0 0x7ffad40b8250 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00cc50 tx=0x7ffacc00cf60 comp rx=0 tx=0).stop 2024-04-02T09:15:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.529+0000 7ffadb400700 1 -- 172.21.15.89:0/3457833055 shutdown_connections 2024-04-02T09:15:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.529+0000 7ffadb400700 1 --2- 172.21.15.89:0/3457833055 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ffab406ad20 0x7ffab406d1e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.529+0000 7ffadb400700 1 --2- 172.21.15.89:0/3457833055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffad40d16a0 0x7ffad40b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.529+0000 7ffadb400700 1 --2- 172.21.15.89:0/3457833055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffad415d9d0 0x7ffad40b8250 unknown :-1 s=CLOSED pgs=2929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.529+0000 7ffadb400700 1 --2- 172.21.15.89:0/3457833055 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffad40d2370 0x7ffad41580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.529+0000 7ffadb400700 1 -- 172.21.15.89:0/3457833055 >> 172.21.15.89:0/3457833055 conn(0x7ffad408aff0 msgr2=0x7ffad415bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:11.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.529+0000 7ffadb400700 1 -- 172.21.15.89:0/3457833055 shutdown_connections 2024-04-02T09:15:11.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.529+0000 7ffadb400700 1 -- 172.21.15.89:0/3457833055 wait complete. 2024-04-02T09:15:11.548 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T09:15:11.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.743+0000 7f3b2be00700 1 Processor -- start 2024-04-02T09:15:11.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.744+0000 7f3b2be00700 1 -- start start 2024-04-02T09:15:11.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.745+0000 7f3b2be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3b240cbb30 0x7f3b240cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:11.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.745+0000 7f3b2be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b240cd020 0x7f3b240cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:11.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.745+0000 7f3b2be00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b240cc450 con 0x7f3b240b2070 2024-04-02T09:15:11.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.745+0000 7f3b2be00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b240cdae0 con 0x7f3b240cbb30 2024-04-02T09:15:11.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.745+0000 7f3b2be00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240cddc0 con 0x7f3b240cd020 2024-04-02T09:15:11.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.745+0000 7f3b29400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3b240cbb30 0x7f3b240cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:11.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.745+0000 7f3b29400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3b240cbb30 0x7f3b240cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45676/0 (socket says 172.21.15.89:45676) 2024-04-02T09:15:11.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.745+0000 7f3b29400700 1 -- 172.21.15.89:0/701912149 learned_addr learned my addr 172.21.15.89:0/701912149 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:11.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.745+0000 7f3b28a00700 1 --2- 172.21.15.89:0/701912149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b240cd020 0x7f3b240cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:11.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.745+0000 7f3b29400700 1 -- 172.21.15.89:0/701912149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b240cd020 msgr2=0x7f3b240cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.745+0000 7f3b29400700 1 --2- 172.21.15.89:0/701912149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b240cd020 0x7f3b240cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.745+0000 7f3b29400700 1 -- 172.21.15.89:0/701912149 >> v1:172.21.15.113:6789/0 conn(0x7f3b240b2070 legacy=0x7f3b2406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.745+0000 7f3b29400700 1 -- 172.21.15.89:0/701912149 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b240ce0a0 con 0x7f3b240cbb30 2024-04-02T09:15:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.746+0000 7f3b29400700 1 --2- 172.21.15.89:0/701912149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b240cbb30 0x7f3b240cbf10 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f3b180098f0 tx=0x7f3b18009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d86ebc1b3a4dc6e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.746+0000 7f3b23e00700 1 -- 172.21.15.89:0/701912149 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1800b8a0 con 0x7f3b240cbb30 2024-04-02T09:15:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.746+0000 7f3b23e00700 1 -- 172.21.15.89:0/701912149 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1800ba00 con 0x7f3b240cbb30 2024-04-02T09:15:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.746+0000 7f3b23e00700 1 -- 172.21.15.89:0/701912149 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1800bd90 con 0x7f3b240cbb30 2024-04-02T09:15:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.746+0000 7f3b2be00700 1 -- 172.21.15.89:0/701912149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b240cbb30 msgr2=0x7f3b240cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.746+0000 7f3b2be00700 1 --2- 172.21.15.89:0/701912149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b240cbb30 0x7f3b240cbf10 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f3b180098f0 tx=0x7f3b18009c00 comp rx=0 tx=0).stop 2024-04-02T09:15:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.747+0000 7f3b2be00700 1 -- 172.21.15.89:0/701912149 shutdown_connections 2024-04-02T09:15:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.747+0000 7f3b2be00700 1 --2- 172.21.15.89:0/701912149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b240cbb30 0x7f3b240cbf10 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.747+0000 7f3b2be00700 1 --2- 172.21.15.89:0/701912149 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b240cd020 0x7f3b240cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.747+0000 7f3b2be00700 1 -- 172.21.15.89:0/701912149 >> 172.21.15.89:0/701912149 conn(0x7f3b240b0d70 msgr2=0x7f3b240b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.747+0000 7f3b2be00700 1 -- 172.21.15.89:0/701912149 shutdown_connections 2024-04-02T09:15:11.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.747+0000 7f3b2be00700 1 -- 172.21.15.89:0/701912149 wait complete. 2024-04-02T09:15:11.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.748+0000 7f3b2be00700 1 Processor -- start 2024-04-02T09:15:11.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.748+0000 7f3b2be00700 1 -- start start 2024-04-02T09:15:11.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.749+0000 7f3b2be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b240b2070 0x7f3b24072370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:11.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.749+0000 7f3b2be00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b240cbb30 0x7f3b240ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:11.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.749+0000 7f3b2be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b240cd020 0x7f3b2415a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:11.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.749+0000 7f3b2be00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240c9410 con 0x7f3b240cd020 2024-04-02T09:15:11.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.749+0000 7f3b2be00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240728b0 con 0x7f3b240cbb30 2024-04-02T09:15:11.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.749+0000 7f3b2be00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240ca8d0 con 0x7f3b240b2070 2024-04-02T09:15:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.750+0000 7f3b29e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b240cd020 0x7f3b2415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.750+0000 7f3b29e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b240cd020 0x7f3b2415a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45692/0 (socket says 172.21.15.89:45692) 2024-04-02T09:15:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.750+0000 7f3b29e00700 1 -- 172.21.15.89:0/2070434595 learned_addr learned my addr 172.21.15.89:0/2070434595 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.750+0000 7f3b29400700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b240b2070 0x7f3b24072370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.750+0000 7f3b28a00700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b240cbb30 0x7f3b240ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.750+0000 7f3b29e00700 1 -- 172.21.15.89:0/2070434595 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b240b2070 msgr2=0x7f3b24072370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.750+0000 7f3b29e00700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b240b2070 0x7f3b24072370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.750+0000 7f3b29e00700 1 -- 172.21.15.89:0/2070434595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b240cbb30 msgr2=0x7f3b240ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.750+0000 7f3b29e00700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b240cbb30 0x7f3b240ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.750+0000 7f3b29e00700 1 -- 172.21.15.89:0/2070434595 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b18009550 con 0x7f3b240cd020 2024-04-02T09:15:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.750+0000 7f3b28a00700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b240cbb30 0x7f3b240ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:15:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.750+0000 7f3b29e00700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b240cd020 0x7f3b2415a670 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00cb50 tx=0x7f3b1c00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.750+0000 7f3b21600700 1 -- 172.21.15.89:0/2070434595 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c003c00 con 0x7f3b240cd020 2024-04-02T09:15:11.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.751+0000 7f3b21600700 1 -- 172.21.15.89:0/2070434595 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c003d60 con 0x7f3b240cd020 2024-04-02T09:15:11.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.751+0000 7f3b2be00700 1 -- 172.21.15.89:0/2070434595 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b240c8790 con 0x7f3b240cd020 2024-04-02T09:15:11.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.751+0000 7f3b21600700 1 -- 172.21.15.89:0/2070434595 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c00ab70 con 0x7f3b240cd020 2024-04-02T09:15:11.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.751+0000 7f3b2be00700 1 -- 172.21.15.89:0/2070434595 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b240c8d50 con 0x7f3b240cd020 2024-04-02T09:15:11.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.752+0000 7f3b2be00700 1 -- 172.21.15.89:0/2070434595 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b241f3380 con 0x7f3b240cd020 2024-04-02T09:15:11.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.753+0000 7f3b21600700 1 -- 172.21.15.89:0/2070434595 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3b1c00acd0 con 0x7f3b240cd020 2024-04-02T09:15:11.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.753+0000 7f3b21600700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3b0406adb0 0x7f3b0406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:11.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.753+0000 7f3b21600700 1 -- 172.21.15.89:0/2070434595 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f3b1c0896a0 con 0x7f3b240cd020 2024-04-02T09:15:11.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.754+0000 7f3b29400700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3b0406adb0 0x7f3b0406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:11.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.754+0000 7f3b29400700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3b0406adb0 0x7f3b0406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3b18014040 tx=0x7f3b18008030 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:11.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.758+0000 7f3b21600700 1 -- 172.21.15.89:0/2070434595 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3b1c0545c0 con 0x7f3b240cd020 2024-04-02T09:15:11.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.960+0000 7f3b2be00700 1 -- 172.21.15.89:0/2070434595 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3b2415bda0 con 0x7f3b240cd020 2024-04-02T09:15:11.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.967+0000 7f3b21600700 1 -- 172.21.15.89:0/2070434595 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v93) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3b1c057be0 con 0x7f3b240cd020 2024-04-02T09:15:11.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.969+0000 7f3b2be00700 1 -- 172.21.15.89:0/2070434595 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3b0406adb0 msgr2=0x7f3b0406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:11.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.969+0000 7f3b2be00700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3b0406adb0 0x7f3b0406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3b18014040 tx=0x7f3b18008030 comp rx=0 tx=0).stop 2024-04-02T09:15:11.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.969+0000 7f3b2be00700 1 -- 172.21.15.89:0/2070434595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b240cd020 msgr2=0x7f3b2415a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:11.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.969+0000 7f3b2be00700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b240cd020 0x7f3b2415a670 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00cb50 tx=0x7f3b1c00cf10 comp rx=0 tx=0).stop 2024-04-02T09:15:11.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.969+0000 7f3b2be00700 1 -- 172.21.15.89:0/2070434595 shutdown_connections 2024-04-02T09:15:11.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.969+0000 7f3b2be00700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b240cd020 0x7f3b2415a670 unknown :-1 s=CLOSED pgs=2931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.969+0000 7f3b2be00700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b240b2070 0x7f3b24072370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.969+0000 7f3b2be00700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3b0406adb0 0x7f3b0406d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.969+0000 7f3b2be00700 1 --2- 172.21.15.89:0/2070434595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b240cbb30 0x7f3b240ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:11.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.969+0000 7f3b2be00700 1 -- 172.21.15.89:0/2070434595 >> 172.21.15.89:0/2070434595 conn(0x7f3b240b0d70 msgr2=0x7f3b2415c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:11.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.969+0000 7f3b2be00700 1 -- 172.21.15.89:0/2070434595 shutdown_connections 2024-04-02T09:15:11.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:11.970+0000 7f3b2be00700 1 -- 172.21.15.89:0/2070434595 wait complete. 2024-04-02T09:15:11.985 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-04-02T09:15:12.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.177+0000 7f7b10800700 1 Processor -- start 2024-04-02T09:15:12.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.178+0000 7f7b10800700 1 -- start start 2024-04-02T09:15:12.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.179+0000 7f7b10800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b0806a790 0x7f7b0806ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:12.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.179+0000 7f7b10800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7b080cebd0 0x7f7b080cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:12.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.179+0000 7f7b10800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b0806f850 con 0x7f7b080afbe0 2024-04-02T09:15:12.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.179+0000 7f7b10800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b0806e3f0 con 0x7f7b0806a790 2024-04-02T09:15:12.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.179+0000 7f7b10800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b0806b0b0 con 0x7f7b080cebd0 2024-04-02T09:15:12.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.179+0000 7f7b0de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b0806a790 0x7f7b0806ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:12.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.179+0000 7f7b0de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7b0806a790 0x7f7b0806ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45700/0 (socket says 172.21.15.89:45700) 2024-04-02T09:15:12.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.179+0000 7f7b0de00700 1 -- 172.21.15.89:0/4046263547 learned_addr learned my addr 172.21.15.89:0/4046263547 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:12.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.180+0000 7f7b0e800700 1 --2- 172.21.15.89:0/4046263547 >> v2:172.21.15.113:3300/0 conn(0x7f7b080cebd0 0x7f7b080cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:12.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.180+0000 7f7b0de00700 1 -- 172.21.15.89:0/4046263547 >> v2:172.21.15.113:3300/0 conn(0x7f7b080cebd0 msgr2=0x7f7b080cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:12.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.180+0000 7f7b0de00700 1 --2- 172.21.15.89:0/4046263547 >> v2:172.21.15.113:3300/0 conn(0x7f7b080cebd0 0x7f7b080cefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:12.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.180+0000 7f7b0de00700 1 -- 172.21.15.89:0/4046263547 >> v1:172.21.15.113:6789/0 conn(0x7f7b080afbe0 legacy=0x7f7b0806dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:12.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.180+0000 7f7b0de00700 1 -- 172.21.15.89:0/4046263547 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0806fad0 con 0x7f7b0806a790 2024-04-02T09:15:12.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.180+0000 7f7b0de00700 1 --2- 172.21.15.89:0/4046263547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b0806a790 0x7f7b0806ab70 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7f7b08072db0 tx=0x7f7afc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1fa9c29d05db7f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:12.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.180+0000 7f7b0ca00700 1 -- 172.21.15.89:0/4046263547 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc004230 con 0x7f7b0806a790 2024-04-02T09:15:12.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.180+0000 7f7b0ca00700 1 -- 172.21.15.89:0/4046263547 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7afc004390 con 0x7f7b0806a790 2024-04-02T09:15:12.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.180+0000 7f7b0ca00700 1 -- 172.21.15.89:0/4046263547 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc0036a0 con 0x7f7b0806a790 2024-04-02T09:15:12.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.181+0000 7f7b10800700 1 -- 172.21.15.89:0/4046263547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b0806a790 msgr2=0x7f7b0806ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:12.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.181+0000 7f7b10800700 1 --2- 172.21.15.89:0/4046263547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b0806a790 0x7f7b0806ab70 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7f7b08072db0 tx=0x7f7afc009bd0 comp rx=0 tx=0).stop 2024-04-02T09:15:12.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.181+0000 7f7b10800700 1 -- 172.21.15.89:0/4046263547 shutdown_connections 2024-04-02T09:15:12.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.181+0000 7f7b10800700 1 --2- 172.21.15.89:0/4046263547 >> v2:172.21.15.113:3300/0 conn(0x7f7b080cebd0 0x7f7b080cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:12.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.181+0000 7f7b10800700 1 --2- 172.21.15.89:0/4046263547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b0806a790 0x7f7b0806ab70 unknown :-1 s=CLOSED pgs=2932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:12.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.181+0000 7f7b10800700 1 -- 172.21.15.89:0/4046263547 >> 172.21.15.89:0/4046263547 conn(0x7f7b080b5ab0 msgr2=0x7f7b080b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:12.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.181+0000 7f7b10800700 1 -- 172.21.15.89:0/4046263547 shutdown_connections 2024-04-02T09:15:12.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.181+0000 7f7b10800700 1 -- 172.21.15.89:0/4046263547 wait complete. 2024-04-02T09:15:12.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.183+0000 7f7b10800700 1 Processor -- start 2024-04-02T09:15:12.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.183+0000 7f7b10800700 1 -- start start 2024-04-02T09:15:12.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b10800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b0806a790 0x7f7b080cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:12.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b10800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b080afbe0 0x7f7b080cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:12.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b10800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b080cebd0 0x7f7b0809dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:12.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b10800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080c9b80 con 0x7f7b080afbe0 2024-04-02T09:15:12.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b10800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080c9e60 con 0x7f7b080cebd0 2024-04-02T09:15:12.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b10800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080ca140 con 0x7f7b0806a790 2024-04-02T09:15:12.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b0d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b080afbe0 0x7f7b080cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:12.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b0d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b080afbe0 0x7f7b080cccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45710/0 (socket says 172.21.15.89:45710) 2024-04-02T09:15:12.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b0d400700 1 -- 172.21.15.89:0/330260592 learned_addr learned my addr 172.21.15.89:0/330260592 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:12.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b0e800700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b080cebd0 0x7f7b0809dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:12.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b0de00700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b0806a790 0x7f7b080cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:12.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b0d400700 1 -- 172.21.15.89:0/330260592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b0806a790 msgr2=0x7f7b080cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:12.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b0d400700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b0806a790 0x7f7b080cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:12.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b0d400700 1 -- 172.21.15.89:0/330260592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b080cebd0 msgr2=0x7f7b0809dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:12.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b0d400700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b080cebd0 0x7f7b0809dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:12.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b0d400700 1 -- 172.21.15.89:0/330260592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc009550 con 0x7f7b080afbe0 2024-04-02T09:15:12.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b0e800700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b080cebd0 0x7f7b0809dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:15:12.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.184+0000 7f7b0de00700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b0806a790 0x7f7b080cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:15:12.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.185+0000 7f7b0d400700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b080afbe0 0x7f7b080cccc0 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f7af800b4a0 tx=0x7f7af800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:12.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.185+0000 7f7b02000700 1 -- 172.21.15.89:0/330260592 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8011830 con 0x7f7b080afbe0 2024-04-02T09:15:12.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.185+0000 7f7b02000700 1 -- 172.21.15.89:0/330260592 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af8011990 con 0x7f7b080afbe0 2024-04-02T09:15:12.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.185+0000 7f7b10800700 1 -- 172.21.15.89:0/330260592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b080ca420 con 0x7f7b080afbe0 2024-04-02T09:15:12.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.185+0000 7f7b02000700 1 -- 172.21.15.89:0/330260592 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8011d40 con 0x7f7b080afbe0 2024-04-02T09:15:12.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.185+0000 7f7b10800700 1 -- 172.21.15.89:0/330260592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b080cd200 con 0x7f7b080afbe0 2024-04-02T09:15:12.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.186+0000 7f7b10800700 1 -- 172.21.15.89:0/330260592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b080ca940 con 0x7f7b080afbe0 2024-04-02T09:15:12.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.187+0000 7f7b02000700 1 -- 172.21.15.89:0/330260592 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7af801a400 con 0x7f7b080afbe0 2024-04-02T09:15:12.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.188+0000 7f7b02000700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7aec06adb0 0x7f7aec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:12.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.188+0000 7f7b02000700 1 -- 172.21.15.89:0/330260592 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f7af8089340 con 0x7f7b080afbe0 2024-04-02T09:15:12.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.188+0000 7f7b0de00700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7aec06adb0 0x7f7aec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:12.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.189+0000 7f7b0de00700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7aec06adb0 0x7f7aec06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009c10 tx=0x7f7afc008030 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:12.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.192+0000 7f7b02000700 1 -- 172.21.15.89:0/330260592 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7b080ca940 con 0x7f7b080afbe0 2024-04-02T09:15:12.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.476+0000 7f7b10800700 1 -- 172.21.15.89:0/330260592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f7b080ca940 con 0x7f7b080afbe0 2024-04-02T09:15:12.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.550+0000 7f7b02000700 1 -- 172.21.15.89:0/330260592 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3275) v1 ==== 155+0+0 (secure 0 0 0) 0x7f7b080ca940 con 0x7f7b080afbe0 2024-04-02T09:15:12.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.552+0000 7f7b10800700 1 -- 172.21.15.89:0/330260592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7aec06adb0 msgr2=0x7f7aec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:12.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.552+0000 7f7b10800700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7aec06adb0 0x7f7aec06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009c10 tx=0x7f7afc008030 comp rx=0 tx=0).stop 2024-04-02T09:15:12.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.552+0000 7f7b10800700 1 -- 172.21.15.89:0/330260592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b080afbe0 msgr2=0x7f7b080cccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:12.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.552+0000 7f7b10800700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b080afbe0 0x7f7b080cccc0 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f7af800b4a0 tx=0x7f7af800b860 comp rx=0 tx=0).stop 2024-04-02T09:15:12.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.553+0000 7f7b10800700 1 -- 172.21.15.89:0/330260592 shutdown_connections 2024-04-02T09:15:12.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.553+0000 7f7b10800700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7aec06adb0 0x7f7aec06d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:12.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.553+0000 7f7b10800700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b080cebd0 0x7f7b0809dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:12.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.553+0000 7f7b10800700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b0806a790 0x7f7b080cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:12.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.553+0000 7f7b10800700 1 --2- 172.21.15.89:0/330260592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b080afbe0 0x7f7b080cccc0 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:12.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.553+0000 7f7b10800700 1 -- 172.21.15.89:0/330260592 >> 172.21.15.89:0/330260592 conn(0x7f7b080b5ab0 msgr2=0x7f7b080b5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:12.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.553+0000 7f7b10800700 1 -- 172.21.15.89:0/330260592 shutdown_connections 2024-04-02T09:15:12.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.553+0000 7f7b10800700 1 -- 172.21.15.89:0/330260592 wait complete. 2024-04-02T09:15:12.569 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2024-04-02T09:15:12.569 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) Test subvol schedule creation succeeds for default subvolgroup. 2024-04-02T09:15:12.570 INFO:tasks.cephfs_test_runner:test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-02T09:15:12.570 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01' 2024-04-02T09:15:12.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.764+0000 7f9e69000700 1 Processor -- start 2024-04-02T09:15:12.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.764+0000 7f9e69000700 1 -- start start 2024-04-02T09:15:12.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.765+0000 7f9e69000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e6415d640 0x7f9e640d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:12.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.765+0000 7f9e69000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e640d30d0 con 0x7f9e640ceec0 2024-04-02T09:15:12.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.765+0000 7f9e69000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e640b6840 con 0x7f9e641619d0 2024-04-02T09:15:12.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.765+0000 7f9e69000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e640d2880 con 0x7f9e6415d640 2024-04-02T09:15:12.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.765+0000 7f9e61e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e6415d640 0x7f9e640d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:12.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.766+0000 7f9e61e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e6415d640 0x7f9e640d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45720/0 (socket says 172.21.15.89:45720) 2024-04-02T09:15:12.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.766+0000 7f9e61e00700 1 -- 172.21.15.89:0/714851886 learned_addr learned my addr 172.21.15.89:0/714851886 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:12.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.766+0000 7f9e61400700 1 --1- 172.21.15.89:0/714851886 >> v1:172.21.15.89:6789/0 conn(0x7f9e640ceec0 0x7f9e641612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41840/0 (socket says 172.21.15.89:41840) 2024-04-02T09:15:12.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.766+0000 7f9e61e00700 1 -- 172.21.15.89:0/714851886 >> v1:172.21.15.113:6789/0 conn(0x7f9e641619d0 legacy=0x7f9e640b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:12.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.766+0000 7f9e61e00700 1 -- 172.21.15.89:0/714851886 >> v1:172.21.15.89:6789/0 conn(0x7f9e640ceec0 legacy=0x7f9e641612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:12.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.766+0000 7f9e61e00700 1 -- 172.21.15.89:0/714851886 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e640d2b30 con 0x7f9e6415d640 2024-04-02T09:15:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.766+0000 7f9e61e00700 1 --2- 172.21.15.89:0/714851886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e6415d640 0x7f9e640d2340 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7f9e540098f0 tx=0x7f9e54009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1727c4740b3fae9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.766+0000 7f9e60a00700 1 -- 172.21.15.89:0/714851886 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54004230 con 0x7f9e6415d640 2024-04-02T09:15:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.766+0000 7f9e60a00700 1 -- 172.21.15.89:0/714851886 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e54004390 con 0x7f9e6415d640 2024-04-02T09:15:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.766+0000 7f9e60a00700 1 -- 172.21.15.89:0/714851886 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e540036a0 con 0x7f9e6415d640 2024-04-02T09:15:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.767+0000 7f9e69000700 1 -- 172.21.15.89:0/714851886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e6415d640 msgr2=0x7f9e640d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.767+0000 7f9e69000700 1 --2- 172.21.15.89:0/714851886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e6415d640 0x7f9e640d2340 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7f9e540098f0 tx=0x7f9e54009c00 comp rx=0 tx=0).stop 2024-04-02T09:15:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.767+0000 7f9e69000700 1 -- 172.21.15.89:0/714851886 shutdown_connections 2024-04-02T09:15:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.767+0000 7f9e69000700 1 --2- 172.21.15.89:0/714851886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e6415d640 0x7f9e640d2340 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.767+0000 7f9e69000700 1 -- 172.21.15.89:0/714851886 >> 172.21.15.89:0/714851886 conn(0x7f9e6408aff0 msgr2=0x7f9e640d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.767+0000 7f9e69000700 1 -- 172.21.15.89:0/714851886 shutdown_connections 2024-04-02T09:15:12.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.767+0000 7f9e69000700 1 -- 172.21.15.89:0/714851886 wait complete. 2024-04-02T09:15:12.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.769+0000 7f9e69000700 1 Processor -- start 2024-04-02T09:15:12.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.769+0000 7f9e69000700 1 -- start start 2024-04-02T09:15:12.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e69000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e640ceec0 0x7f9e640c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:12.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e69000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6415d640 0x7f9e6414a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:12.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e69000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e641619d0 0x7f9e640c7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:12.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e69000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e640c8d30 con 0x7f9e641619d0 2024-04-02T09:15:12.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e69000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e640ca1b0 con 0x7f9e6415d640 2024-04-02T09:15:12.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e69000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64158190 con 0x7f9e640ceec0 2024-04-02T09:15:12.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e62800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e641619d0 0x7f9e640c7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:12.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e62800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e641619d0 0x7f9e640c7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45732/0 (socket says 172.21.15.89:45732) 2024-04-02T09:15:12.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e62800700 1 -- 172.21.15.89:0/2285529592 learned_addr learned my addr 172.21.15.89:0/2285529592 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:12.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e61400700 1 --2- 172.21.15.89:0/2285529592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6415d640 0x7f9e6414a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:12.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e61e00700 1 --2- 172.21.15.89:0/2285529592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e640ceec0 0x7f9e640c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:12.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e62800700 1 -- 172.21.15.89:0/2285529592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e640ceec0 msgr2=0x7f9e640c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:12.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e62800700 1 --2- 172.21.15.89:0/2285529592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e640ceec0 0x7f9e640c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:12.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e62800700 1 -- 172.21.15.89:0/2285529592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6415d640 msgr2=0x7f9e6414a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:12.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e62800700 1 --2- 172.21.15.89:0/2285529592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6415d640 0x7f9e6414a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:12.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.770+0000 7f9e62800700 1 -- 172.21.15.89:0/2285529592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54009550 con 0x7f9e641619d0 2024-04-02T09:15:12.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.771+0000 7f9e62800700 1 --2- 172.21.15.89:0/2285529592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e641619d0 0x7f9e640c7130 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00b970 tx=0x7f9e5c00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:12.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.771+0000 7f9e5a000700 1 -- 172.21.15.89:0/2285529592 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c00c9b0 con 0x7f9e641619d0 2024-04-02T09:15:12.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.771+0000 7f9e5a000700 1 -- 172.21.15.89:0/2285529592 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5c00cb10 con 0x7f9e641619d0 2024-04-02T09:15:12.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.771+0000 7f9e5a000700 1 -- 172.21.15.89:0/2285529592 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c014440 con 0x7f9e641619d0 2024-04-02T09:15:12.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.771+0000 7f9e69000700 1 -- 172.21.15.89:0/2285529592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6414aea0 con 0x7f9e641619d0 2024-04-02T09:15:12.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.771+0000 7f9e69000700 1 -- 172.21.15.89:0/2285529592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e640c9ed0 con 0x7f9e641619d0 2024-04-02T09:15:12.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.772+0000 7f9e69000700 1 -- 172.21.15.89:0/2285529592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e640c84d0 con 0x7f9e641619d0 2024-04-02T09:15:12.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.773+0000 7f9e5a000700 1 -- 172.21.15.89:0/2285529592 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9e5c01d3b0 con 0x7f9e641619d0 2024-04-02T09:15:12.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.774+0000 7f9e5a000700 1 --2- 172.21.15.89:0/2285529592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e4406adb0 0x7f9e4406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:12.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.774+0000 7f9e5a000700 1 -- 172.21.15.89:0/2285529592 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f9e5c089380 con 0x7f9e641619d0 2024-04-02T09:15:12.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.775+0000 7f9e61e00700 1 --2- 172.21.15.89:0/2285529592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e4406adb0 0x7f9e4406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:12.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.775+0000 7f9e61e00700 1 --2- 172.21.15.89:0/2285529592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e4406adb0 0x7f9e4406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009130 tx=0x7f9e54008030 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:12.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:12.779+0000 7f9e5a000700 1 -- 172.21.15.89:0/2285529592 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e640c84d0 con 0x7f9e641619d0 2024-04-02T09:15:13.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.058+0000 7f9e69000700 1 -- 172.21.15.89:0/2285529592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]} v 0) v1 -- 0x7f9e640c84d0 con 0x7f9e641619d0 2024-04-02T09:15:13.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.555+0000 7f9e5a000700 1 -- 172.21.15.89:0/2285529592 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]}]=0 v3276) v1 ==== 204+0+0 (secure 0 0 0) 0x7f9e640c84d0 con 0x7f9e641619d0 2024-04-02T09:15:13.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.557+0000 7f9e69000700 1 -- 172.21.15.89:0/2285529592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e4406adb0 msgr2=0x7f9e4406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:13.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.557+0000 7f9e69000700 1 --2- 172.21.15.89:0/2285529592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e4406adb0 0x7f9e4406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009130 tx=0x7f9e54008030 comp rx=0 tx=0).stop 2024-04-02T09:15:13.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.557+0000 7f9e69000700 1 -- 172.21.15.89:0/2285529592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e641619d0 msgr2=0x7f9e640c7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:13.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.557+0000 7f9e69000700 1 --2- 172.21.15.89:0/2285529592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e641619d0 0x7f9e640c7130 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00b970 tx=0x7f9e5c00bd30 comp rx=0 tx=0).stop 2024-04-02T09:15:13.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.557+0000 7f9e69000700 1 -- 172.21.15.89:0/2285529592 shutdown_connections 2024-04-02T09:15:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.558+0000 7f9e69000700 1 --2- 172.21.15.89:0/2285529592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e641619d0 0x7f9e640c7130 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.558+0000 7f9e69000700 1 --2- 172.21.15.89:0/2285529592 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e640ceec0 0x7f9e640c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.558+0000 7f9e69000700 1 --2- 172.21.15.89:0/2285529592 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e4406adb0 0x7f9e4406d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.558+0000 7f9e69000700 1 --2- 172.21.15.89:0/2285529592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6415d640 0x7f9e6414a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.558+0000 7f9e69000700 1 -- 172.21.15.89:0/2285529592 >> 172.21.15.89:0/2285529592 conn(0x7f9e6408aff0 msgr2=0x7f9e6415b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.558+0000 7f9e69000700 1 -- 172.21.15.89:0/2285529592 shutdown_connections 2024-04-02T09:15:13.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.558+0000 7f9e69000700 1 -- 172.21.15.89:0/2285529592 wait complete. 2024-04-02T09:15:13.575 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-02T09:15:13.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.771+0000 7f2f34600700 1 Processor -- start 2024-04-02T09:15:13.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.771+0000 7f2f34600700 1 -- start start 2024-04-02T09:15:13.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.773+0000 7f2f34600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2f2c15d9d0 0x7f2f2c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:13.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.773+0000 7f2f34600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2f2c0d16a0 0x7f2f2c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:13.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.773+0000 7f2f34600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2c161420 con 0x7f2f2c0d2370 2024-04-02T09:15:13.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.773+0000 7f2f34600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2c0d27e0 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.773+0000 7f2f34600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2c161700 con 0x7f2f2c15d9d0 2024-04-02T09:15:13.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.773+0000 7f2f31200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2f2c0d16a0 0x7f2f2c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:13.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.773+0000 7f2f31200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2f2c0d16a0 0x7f2f2c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45746/0 (socket says 172.21.15.89:45746) 2024-04-02T09:15:13.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.773+0000 7f2f31200700 1 -- 172.21.15.89:0/106816541 learned_addr learned my addr 172.21.15.89:0/106816541 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:13.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.773+0000 7f2f31c00700 1 --2- 172.21.15.89:0/106816541 >> v2:172.21.15.113:3300/0 conn(0x7f2f2c15d9d0 0x7f2f2c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:13.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.773+0000 7f2f31200700 1 -- 172.21.15.89:0/106816541 >> v2:172.21.15.113:3300/0 conn(0x7f2f2c15d9d0 msgr2=0x7f2f2c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:13.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.773+0000 7f2f31200700 1 --2- 172.21.15.89:0/106816541 >> v2:172.21.15.113:3300/0 conn(0x7f2f2c15d9d0 0x7f2f2c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:13.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.773+0000 7f2f31200700 1 -- 172.21.15.89:0/106816541 >> v1:172.21.15.113:6789/0 conn(0x7f2f2c0d2370 legacy=0x7f2f2c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:13.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.773+0000 7f2f31200700 1 -- 172.21.15.89:0/106816541 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2c1619e0 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.774+0000 7f2f31200700 1 --2- 172.21.15.89:0/106816541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c0d16a0 0x7f2f2c0d1e30 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0d0ff0 tx=0x7f2f18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b8e25e4b498feb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:13.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.774+0000 7f2f2be00700 1 -- 172.21.15.89:0/106816541 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f18005e30 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.774+0000 7f2f2be00700 1 -- 172.21.15.89:0/106816541 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f18004030 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.774+0000 7f2f2be00700 1 -- 172.21.15.89:0/106816541 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f180043e0 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.774+0000 7f2f34600700 1 -- 172.21.15.89:0/106816541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c0d16a0 msgr2=0x7f2f2c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:13.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.774+0000 7f2f34600700 1 --2- 172.21.15.89:0/106816541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c0d16a0 0x7f2f2c0d1e30 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0d0ff0 tx=0x7f2f18009bd0 comp rx=0 tx=0).stop 2024-04-02T09:15:13.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.775+0000 7f2f34600700 1 -- 172.21.15.89:0/106816541 shutdown_connections 2024-04-02T09:15:13.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.775+0000 7f2f34600700 1 --2- 172.21.15.89:0/106816541 >> v2:172.21.15.113:3300/0 conn(0x7f2f2c15d9d0 0x7f2f2c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:13.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.775+0000 7f2f34600700 1 --2- 172.21.15.89:0/106816541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c0d16a0 0x7f2f2c0d1e30 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:13.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.775+0000 7f2f34600700 1 -- 172.21.15.89:0/106816541 >> 172.21.15.89:0/106816541 conn(0x7f2f2c08aff0 msgr2=0x7f2f2c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:13.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.775+0000 7f2f34600700 1 -- 172.21.15.89:0/106816541 shutdown_connections 2024-04-02T09:15:13.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.775+0000 7f2f34600700 1 -- 172.21.15.89:0/106816541 wait complete. 2024-04-02T09:15:13.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.776+0000 7f2f34600700 1 Processor -- start 2024-04-02T09:15:13.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.776+0000 7f2f34600700 1 -- start start 2024-04-02T09:15:13.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.777+0000 7f2f34600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c0d16a0 0x7f2f2c155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:13.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.777+0000 7f2f34600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f2c0d2370 0x7f2f2c156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:13.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.777+0000 7f2f34600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2c15d9d0 0x7f2f2c15fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:13.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.777+0000 7f2f34600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c0b6020 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.777+0000 7f2f34600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c1591f0 con 0x7f2f2c0d2370 2024-04-02T09:15:13.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.777+0000 7f2f34600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c1594d0 con 0x7f2f2c15d9d0 2024-04-02T09:15:13.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.778+0000 7f2f31c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c0d16a0 0x7f2f2c155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:13.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.778+0000 7f2f31c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c0d16a0 0x7f2f2c155320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45750/0 (socket says 172.21.15.89:45750) 2024-04-02T09:15:13.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.778+0000 7f2f31c00700 1 -- 172.21.15.89:0/262977972 learned_addr learned my addr 172.21.15.89:0/262977972 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:13.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.778+0000 7f2f31200700 1 --2- 172.21.15.89:0/262977972 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f2c0d2370 0x7f2f2c156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:13.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.778+0000 7f2f32600700 1 --2- 172.21.15.89:0/262977972 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2c15d9d0 0x7f2f2c15fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:13.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.778+0000 7f2f31c00700 1 -- 172.21.15.89:0/262977972 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2c15d9d0 msgr2=0x7f2f2c15fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:13.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.778+0000 7f2f31c00700 1 --2- 172.21.15.89:0/262977972 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2c15d9d0 0x7f2f2c15fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:13.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.778+0000 7f2f31c00700 1 -- 172.21.15.89:0/262977972 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f2c0d2370 msgr2=0x7f2f2c156690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:13.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.778+0000 7f2f31c00700 1 --2- 172.21.15.89:0/262977972 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f2c0d2370 0x7f2f2c156690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:13.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.778+0000 7f2f31c00700 1 -- 172.21.15.89:0/262977972 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f18009500 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.778+0000 7f2f31c00700 1 --2- 172.21.15.89:0/262977972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c0d16a0 0x7f2f2c155320 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000d800 tx=0x7f2f2000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:13.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.779+0000 7f2f29600700 1 -- 172.21.15.89:0/262977972 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f200125f0 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.779+0000 7f2f29600700 1 -- 172.21.15.89:0/262977972 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f20012750 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.779+0000 7f2f34600700 1 -- 172.21.15.89:0/262977972 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f2c0b6230 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.779+0000 7f2f29600700 1 -- 172.21.15.89:0/262977972 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f20012b00 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.779+0000 7f2f34600700 1 -- 172.21.15.89:0/262977972 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2c158e80 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.780+0000 7f2f34600700 1 -- 172.21.15.89:0/262977972 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2c0d1ca0 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.781+0000 7f2f29600700 1 -- 172.21.15.89:0/262977972 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2f20012c60 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.781+0000 7f2f29600700 1 --2- 172.21.15.89:0/262977972 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2f0c06adb0 0x7f2f0c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:13.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.782+0000 7f2f29600700 1 -- 172.21.15.89:0/262977972 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f2f200899d0 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.782+0000 7f2f31200700 1 --2- 172.21.15.89:0/262977972 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2f0c06adb0 0x7f2f0c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:13.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.782+0000 7f2f31200700 1 --2- 172.21.15.89:0/262977972 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2f0c06adb0 0x7f2f0c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0d0ff0 tx=0x7f2f18008070 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:13.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.786+0000 7f2f29600700 1 -- 172.21.15.89:0/262977972 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2f20054860 con 0x7f2f2c0d16a0 2024-04-02T09:15:13.795 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:15:13.800 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:15:13.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:13.996+0000 7f2f34600700 1 -- 172.21.15.89:0/262977972 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2f2c159640 con 0x7f2f2c0d16a0 2024-04-02T09:15:14.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.002+0000 7f2f29600700 1 -- 172.21.15.89:0/262977972 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v94)=0 v94) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2f20057e80 con 0x7f2f2c0d16a0 2024-04-02T09:15:14.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.004+0000 7f2f34600700 1 -- 172.21.15.89:0/262977972 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2f0c06adb0 msgr2=0x7f2f0c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:14.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.004+0000 7f2f34600700 1 --2- 172.21.15.89:0/262977972 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2f0c06adb0 0x7f2f0c06d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0d0ff0 tx=0x7f2f18008070 comp rx=0 tx=0).stop 2024-04-02T09:15:14.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.004+0000 7f2f34600700 1 -- 172.21.15.89:0/262977972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c0d16a0 msgr2=0x7f2f2c155320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:14.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.004+0000 7f2f34600700 1 --2- 172.21.15.89:0/262977972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c0d16a0 0x7f2f2c155320 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000d800 tx=0x7f2f2000db10 comp rx=0 tx=0).stop 2024-04-02T09:15:14.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.004+0000 7f2f34600700 1 -- 172.21.15.89:0/262977972 shutdown_connections 2024-04-02T09:15:14.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.004+0000 7f2f34600700 1 --2- 172.21.15.89:0/262977972 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2c15d9d0 0x7f2f2c15fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.004+0000 7f2f34600700 1 --2- 172.21.15.89:0/262977972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f2c0d16a0 0x7f2f2c155320 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.004+0000 7f2f34600700 1 --2- 172.21.15.89:0/262977972 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2f0c06adb0 0x7f2f0c06d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.004+0000 7f2f34600700 1 --2- 172.21.15.89:0/262977972 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f2c0d2370 0x7f2f2c156690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.004+0000 7f2f34600700 1 -- 172.21.15.89:0/262977972 >> 172.21.15.89:0/262977972 conn(0x7f2f2c08aff0 msgr2=0x7f2f2c0b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:14.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.004+0000 7f2f34600700 1 -- 172.21.15.89:0/262977972 shutdown_connections 2024-04-02T09:15:14.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.004+0000 7f2f34600700 1 -- 172.21.15.89:0/262977972 wait complete. 2024-04-02T09:15:14.020 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T09:15:14.020 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:15:14.113 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:15:14.206 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T09:15:14.206 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:15:14.206 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:15:14.250 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:15:14.251 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:15:14.251 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:15:14.298 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:15:14.299 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:15:14.299 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:15:14.343 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:15:14.343 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:15:14.343 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:15:14.388 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:15:14.388 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-02T09:15:14.389 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:15:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.583+0000 7f36da800700 1 Processor -- start 2024-04-02T09:15:14.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.584+0000 7f36da800700 1 -- start start 2024-04-02T09:15:14.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.585+0000 7f36da800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36d415d640 0x7f36d40d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:14.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.585+0000 7f36da800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f36d41619d0 0x7f36d40b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:14.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.585+0000 7f36da800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d40d30d0 con 0x7f36d40ceec0 2024-04-02T09:15:14.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.585+0000 7f36da800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d40b6840 con 0x7f36d41619d0 2024-04-02T09:15:14.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.585+0000 7f36da800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d40d2880 con 0x7f36d415d640 2024-04-02T09:15:14.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.586+0000 7f36d3e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f36d41619d0 0x7f36d40b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:14.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.586+0000 7f36d2a00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f36d40ceec0 0x7f36d41612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:37120/0 (socket says 172.21.15.89:37120) 2024-04-02T09:15:14.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.586+0000 7f36d2a00700 1 -- 172.21.15.89:0/44359817 learned_addr learned my addr 172.21.15.89:0/44359817 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:14.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.586+0000 7f36d3400700 1 --2- 172.21.15.89:0/44359817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36d415d640 0x7f36d40d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:14.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.586+0000 7f36d2000700 1 -- 172.21.15.89:0/44359817 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644366296 0 0) 0x7f36d40d30d0 con 0x7f36d40ceec0 2024-04-02T09:15:14.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.586+0000 7f36d2000700 1 -- 172.21.15.89:0/44359817 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b80035b0 con 0x7f36d40ceec0 2024-04-02T09:15:14.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.586+0000 7f36d3e00700 1 -- 172.21.15.89:0/44359817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36d415d640 msgr2=0x7f36d40d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:14.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.586+0000 7f36d3e00700 1 --2- 172.21.15.89:0/44359817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36d415d640 0x7f36d40d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.586+0000 7f36d3e00700 1 -- 172.21.15.89:0/44359817 >> v1:172.21.15.113:6789/0 conn(0x7f36d40ceec0 legacy=0x7f36d41612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:14.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.586+0000 7f36d3e00700 1 -- 172.21.15.89:0/44359817 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d40d2b30 con 0x7f36d41619d0 2024-04-02T09:15:14.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.587+0000 7f36d3400700 1 --2- 172.21.15.89:0/44359817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36d415d640 0x7f36d40d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:15:14.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.587+0000 7f36d3e00700 1 --2- 172.21.15.89:0/44359817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36d41619d0 0x7f36d40b6cd0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009d90 tx=0x7f36cc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=172cf1e1b2c4eac1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:14.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.587+0000 7f36d2000700 1 -- 172.21.15.89:0/44359817 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc00e070 con 0x7f36d41619d0 2024-04-02T09:15:14.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.587+0000 7f36d2000700 1 -- 172.21.15.89:0/44359817 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc00bb20 con 0x7f36d41619d0 2024-04-02T09:15:14.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.587+0000 7f36d2000700 1 -- 172.21.15.89:0/44359817 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc012430 con 0x7f36d41619d0 2024-04-02T09:15:14.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.587+0000 7f36da800700 1 -- 172.21.15.89:0/44359817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36d41619d0 msgr2=0x7f36d40b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:14.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.587+0000 7f36da800700 1 --2- 172.21.15.89:0/44359817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36d41619d0 0x7f36d40b6cd0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009d90 tx=0x7f36cc00b330 comp rx=0 tx=0).stop 2024-04-02T09:15:14.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.588+0000 7f36da800700 1 -- 172.21.15.89:0/44359817 shutdown_connections 2024-04-02T09:15:14.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.588+0000 7f36da800700 1 --2- 172.21.15.89:0/44359817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36d41619d0 0x7f36d40b6cd0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.588+0000 7f36da800700 1 --2- 172.21.15.89:0/44359817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36d415d640 0x7f36d40d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.588+0000 7f36da800700 1 -- 172.21.15.89:0/44359817 >> 172.21.15.89:0/44359817 conn(0x7f36d408aff0 msgr2=0x7f36d40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:14.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.588+0000 7f36da800700 1 -- 172.21.15.89:0/44359817 shutdown_connections 2024-04-02T09:15:14.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.588+0000 7f36da800700 1 -- 172.21.15.89:0/44359817 wait complete. 2024-04-02T09:15:14.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.590+0000 7f36da800700 1 Processor -- start 2024-04-02T09:15:14.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.590+0000 7f36da800700 1 -- start start 2024-04-02T09:15:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.591+0000 7f36da800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36d40ceec0 0x7f36d40c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.591+0000 7f36da800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36d415d640 0x7f36d415fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.591+0000 7f36da800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36d41619d0 0x7f36d4158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.591+0000 7f36da800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d40c9310 con 0x7f36d40ceec0 2024-04-02T09:15:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.591+0000 7f36da800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d4160530 con 0x7f36d415d640 2024-04-02T09:15:14.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.591+0000 7f36da800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d4153550 con 0x7f36d41619d0 2024-04-02T09:15:14.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.592+0000 7f36d3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36d40ceec0 0x7f36d40c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:14.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.592+0000 7f36d3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36d40ceec0 0x7f36d40c8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45760/0 (socket says 172.21.15.89:45760) 2024-04-02T09:15:14.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.592+0000 7f36d3400700 1 -- 172.21.15.89:0/2707428178 learned_addr learned my addr 172.21.15.89:0/2707428178 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:14.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.592+0000 7f36d3e00700 1 --2- 172.21.15.89:0/2707428178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36d41619d0 0x7f36d4158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:14.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.592+0000 7f36d2a00700 1 --2- 172.21.15.89:0/2707428178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36d415d640 0x7f36d415fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:14.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.592+0000 7f36d3400700 1 -- 172.21.15.89:0/2707428178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36d41619d0 msgr2=0x7f36d4158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:14.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.592+0000 7f36d3400700 1 --2- 172.21.15.89:0/2707428178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36d41619d0 0x7f36d4158b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.592+0000 7f36d3400700 1 -- 172.21.15.89:0/2707428178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36d415d640 msgr2=0x7f36d415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:14.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.592+0000 7f36d3400700 1 --2- 172.21.15.89:0/2707428178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36d415d640 0x7f36d415fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.592+0000 7f36d3400700 1 -- 172.21.15.89:0/2707428178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c40094a0 con 0x7f36d40ceec0 2024-04-02T09:15:14.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.592+0000 7f36d3400700 1 --2- 172.21.15.89:0/2707428178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36d40ceec0 0x7f36d40c8050 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f36c400ebe0 tx=0x7f36c400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:14.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.593+0000 7f36cb400700 1 -- 172.21.15.89:0/2707428178 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c4009760 con 0x7f36d40ceec0 2024-04-02T09:15:14.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.593+0000 7f36cb400700 1 -- 172.21.15.89:0/2707428178 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c400c850 con 0x7f36d40ceec0 2024-04-02T09:15:14.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.593+0000 7f36cb400700 1 -- 172.21.15.89:0/2707428178 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c400cbe0 con 0x7f36d40ceec0 2024-04-02T09:15:14.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.593+0000 7f36da800700 1 -- 172.21.15.89:0/2707428178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36cc009a70 con 0x7f36d40ceec0 2024-04-02T09:15:14.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.593+0000 7f36da800700 1 -- 172.21.15.89:0/2707428178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d40c7570 con 0x7f36d40ceec0 2024-04-02T09:15:14.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.594+0000 7f36da800700 1 -- 172.21.15.89:0/2707428178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d40cead0 con 0x7f36d40ceec0 2024-04-02T09:15:14.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.595+0000 7f36cb400700 1 -- 172.21.15.89:0/2707428178 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f36c400cd40 con 0x7f36d40ceec0 2024-04-02T09:15:14.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.595+0000 7f36cb400700 1 --2- 172.21.15.89:0/2707428178 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f36b406adb0 0x7f36b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:14.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.596+0000 7f36cb400700 1 -- 172.21.15.89:0/2707428178 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f36c4089760 con 0x7f36d40ceec0 2024-04-02T09:15:14.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.596+0000 7f36d2a00700 1 --2- 172.21.15.89:0/2707428178 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f36b406adb0 0x7f36b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:14.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.596+0000 7f36d2a00700 1 --2- 172.21.15.89:0/2707428178 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f36b406adb0 0x7f36b406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f36d40d4220 tx=0x7f36c0007dc0 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.600+0000 7f36cb400700 1 -- 172.21.15.89:0/2707428178 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f36c4054560 con 0x7f36d40ceec0 2024-04-02T09:15:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.844+0000 7f36da800700 1 -- 172.21.15.89:0/2707428178 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36d4067720 con 0x7f36d40ceec0 2024-04-02T09:15:14.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.844+0000 7f36cb400700 1 -- 172.21.15.89:0/2707428178 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+3541 (secure 0 0 0) 0x7f36c4017df0 con 0x7f36d40ceec0 2024-04-02T09:15:14.843 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:14.843 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":137,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10031,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/1597421311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1597421311},{"type":"v1","addr":"172.21.15.113:6834","nonce":1597421311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":10034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/3719211992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":3719211992},{"type":"v1","addr":"172.21.15.113:6835","nonce":3719211992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":10037,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3947526387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3947526387},{"type":"v1","addr":"172.21.15.89:6835","nonce":3947526387}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":13206,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/503499774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":503499774},{"type":"v1","addr":"172.21.15.89:6837","nonce":503499774}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137}],"filesystems":[]} 2024-04-02T09:15:14.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.847+0000 7f36da800700 1 -- 172.21.15.89:0/2707428178 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f36b406adb0 msgr2=0x7f36b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:14.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.847+0000 7f36da800700 1 --2- 172.21.15.89:0/2707428178 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f36b406adb0 0x7f36b406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f36d40d4220 tx=0x7f36c0007dc0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.847+0000 7f36da800700 1 -- 172.21.15.89:0/2707428178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36d40ceec0 msgr2=0x7f36d40c8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:14.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.847+0000 7f36da800700 1 --2- 172.21.15.89:0/2707428178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36d40ceec0 0x7f36d40c8050 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f36c400ebe0 tx=0x7f36c400eef0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.847+0000 7f36da800700 1 -- 172.21.15.89:0/2707428178 shutdown_connections 2024-04-02T09:15:14.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.847+0000 7f36da800700 1 --2- 172.21.15.89:0/2707428178 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f36d41619d0 0x7f36d4158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.847+0000 7f36da800700 1 --2- 172.21.15.89:0/2707428178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36d40ceec0 0x7f36d40c8050 unknown :-1 s=CLOSED pgs=2939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.847+0000 7f36da800700 1 --2- 172.21.15.89:0/2707428178 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f36b406adb0 0x7f36b406d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.847+0000 7f36da800700 1 --2- 172.21.15.89:0/2707428178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36d415d640 0x7f36d415fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:14.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.847+0000 7f36da800700 1 -- 172.21.15.89:0/2707428178 >> 172.21.15.89:0/2707428178 conn(0x7f36d408aff0 msgr2=0x7f36d40d3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:14.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.847+0000 7f36da800700 1 -- 172.21.15.89:0/2707428178 shutdown_connections 2024-04-02T09:15:14.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:14.847+0000 7f36da800700 1 -- 172.21.15.89:0/2707428178 wait complete. 2024-04-02T09:15:14.847 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 137 2024-04-02T09:15:14.863 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-02T09:15:14.864 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-02T09:15:14.864 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T09:15:14.864 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:15:14.864 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-02T09:15:14.864 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-02T09:15:14.864 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T09:15:14.864 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:15:14.865 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-02T09:15:14.865 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-02T09:15:14.865 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T09:15:14.865 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:15:14.865 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-02T09:15:14.865 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-02T09:15:14.865 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T09:15:14.866 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:15:14.878 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T09:15:14.878 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-02T09:15:14.879 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T09:15:14.880 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-02T09:15:14.881 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T09:15:14.881 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-02T09:15:14.923 INFO:tasks.ceph.mds.c:Started 2024-04-02T09:15:14.923 DEBUG:teuthology.parallel:result is None 2024-04-02T09:15:14.925 INFO:tasks.ceph.mds.b:Started 2024-04-02T09:15:14.925 DEBUG:teuthology.parallel:result is None 2024-04-02T09:15:14.929 INFO:tasks.ceph.mds.d:Started 2024-04-02T09:15:14.930 DEBUG:teuthology.parallel:result is None 2024-04-02T09:15:15.061 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2024-04-02T09:15:15.069 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2024-04-02T09:15:15.069 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2024-04-02T09:15:20.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-04-02T09:15:20.878 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T09:15:20.879 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-02T09:15:20.884 INFO:tasks.ceph.mds.a:Started 2024-04-02T09:15:20.884 DEBUG:teuthology.parallel:result is None 2024-04-02T09:15:20.884 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-02T09:15:21.036 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T09:15:21.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.093+0000 7f5c95000700 1 Processor -- start 2024-04-02T09:15:21.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.093+0000 7f5c95000700 1 -- start start 2024-04-02T09:15:21.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.094+0000 7f5c95000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c9015a5c0 0x7f5c900709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:21.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.094+0000 7f5c95000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c900cb310 con 0x7f5c900ce840 2024-04-02T09:15:21.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.094+0000 7f5c95000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c90070f00 con 0x7f5c9015dfe0 2024-04-02T09:15:21.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.095+0000 7f5c95000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c900711a0 con 0x7f5c9015a5c0 2024-04-02T09:15:21.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.095+0000 7f5c8e800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5c900ce840 0x7f5c900cba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50654/0 (socket says 172.21.15.89:50654) 2024-04-02T09:15:21.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.095+0000 7f5c8e800700 1 -- 172.21.15.89:0/759076537 learned_addr learned my addr 172.21.15.89:0/759076537 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.095+0000 7f5c8de00700 1 --2- 172.21.15.89:0/759076537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c9015a5c0 0x7f5c900709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.095+0000 7f5c8ca00700 1 -- 172.21.15.89:0/759076537 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453504804 0 0) 0x7f5c900cb310 con 0x7f5c900ce840 2024-04-02T09:15:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.095+0000 7f5c8ca00700 1 -- 172.21.15.89:0/759076537 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c740035b0 con 0x7f5c900ce840 2024-04-02T09:15:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.095+0000 7f5c8ca00700 1 -- 172.21.15.89:0/759076537 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2349468293 0 0) 0x7f5c740035b0 con 0x7f5c900ce840 2024-04-02T09:15:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.095+0000 7f5c8ca00700 1 -- 172.21.15.89:0/759076537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c9015a5c0 msgr2=0x7f5c900709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.095+0000 7f5c8ca00700 1 --2- 172.21.15.89:0/759076537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c9015a5c0 0x7f5c900709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.095+0000 7f5c8ca00700 1 -- 172.21.15.89:0/759076537 >> v1:172.21.15.113:6789/0 conn(0x7f5c9015dfe0 legacy=0x7f5c900ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8ca00700 1 -- 172.21.15.89:0/759076537 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c90071480 con 0x7f5c900ce840 2024-04-02T09:15:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8ca00700 1 -- 172.21.15.89:0/759076537 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f5c880019f0 con 0x7f5c900ce840 2024-04-02T09:15:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8ca00700 1 -- 172.21.15.89:0/759076537 >> v1:172.21.15.89:6789/0 conn(0x7f5c900ce840 legacy=0x7f5c900cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8ca00700 1 --2- 172.21.15.89:0/759076537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c9015a5c0 0x7f5c74003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8ca00700 1 --2- 172.21.15.89:0/759076537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c74003f70 0x7f5c74004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8ca00700 1 --2- 172.21.15.89:0/759076537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c74004d90 0x7f5c74005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8ca00700 1 -- 172.21.15.89:0/759076537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c900711a0 con 0x7f5c74003f70 2024-04-02T09:15:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8ca00700 1 -- 172.21.15.89:0/759076537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7400bb50 con 0x7f5c74004d90 2024-04-02T09:15:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8ca00700 1 -- 172.21.15.89:0/759076537 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7400bd10 con 0x7f5c9015a5c0 2024-04-02T09:15:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8d400700 1 --2- 172.21.15.89:0/759076537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c74003f70 0x7f5c74004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8de00700 1 --2- 172.21.15.89:0/759076537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c9015a5c0 0x7f5c74003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8d400700 1 -- 172.21.15.89:0/759076537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c9015a5c0 msgr2=0x7f5c74003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8d400700 1 --2- 172.21.15.89:0/759076537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c9015a5c0 0x7f5c74003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8d400700 1 -- 172.21.15.89:0/759076537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c74004d90 msgr2=0x7f5c74005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8d400700 1 --2- 172.21.15.89:0/759076537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c74004d90 0x7f5c74005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.096+0000 7f5c8d400700 1 -- 172.21.15.89:0/759076537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c740035b0 con 0x7f5c74003f70 2024-04-02T09:15:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.097+0000 7f5c8d400700 1 --2- 172.21.15.89:0/759076537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c74003f70 0x7f5c74004850 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c0018d0 tx=0x7f5c74005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49072a90c6db90c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.097+0000 7f5c8ca00700 1 -- 172.21.15.89:0/759076537 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c00c690 con 0x7f5c74003f70 2024-04-02T09:15:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.097+0000 7f5c8ca00700 1 -- 172.21.15.89:0/759076537 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7c00c7f0 con 0x7f5c74003f70 2024-04-02T09:15:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.097+0000 7f5c8ca00700 1 -- 172.21.15.89:0/759076537 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c00cba0 con 0x7f5c74003f70 2024-04-02T09:15:21.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.097+0000 7f5c95000700 1 -- 172.21.15.89:0/759076537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c74003f70 msgr2=0x7f5c74004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:21.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.097+0000 7f5c95000700 1 --2- 172.21.15.89:0/759076537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c74003f70 0x7f5c74004850 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c0018d0 tx=0x7f5c74005880 comp rx=0 tx=0).stop 2024-04-02T09:15:21.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.097+0000 7f5c8e800700 1 -- 172.21.15.89:0/759076537 reap_dead start 2024-04-02T09:15:21.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.097+0000 7f5c95000700 1 -- 172.21.15.89:0/759076537 shutdown_connections 2024-04-02T09:15:21.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.097+0000 7f5c95000700 1 -- 172.21.15.89:0/759076537 >> 172.21.15.89:0/759076537 conn(0x7f5c900b0d70 msgr2=0x7f5c900b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:21.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.098+0000 7f5c95000700 1 -- 172.21.15.89:0/759076537 shutdown_connections 2024-04-02T09:15:21.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.098+0000 7f5c95000700 1 -- 172.21.15.89:0/759076537 wait complete. 2024-04-02T09:15:21.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.099+0000 7f5c95000700 1 Processor -- start 2024-04-02T09:15:21.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.099+0000 7f5c95000700 1 -- start start 2024-04-02T09:15:21.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.100+0000 7f5c95000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c74003f70 0x7f5c9014c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:21.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.100+0000 7f5c95000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c9015d290 0x7f5c9015ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:21.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.100+0000 7f5c95000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c90156da0 0x7f5c90157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:21.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.100+0000 7f5c95000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9014cc40 con 0x7f5c74003f70 2024-04-02T09:15:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.100+0000 7f5c95000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c90158fa0 con 0x7f5c90156da0 2024-04-02T09:15:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.100+0000 7f5c95000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c901565c0 con 0x7f5c9015d290 2024-04-02T09:15:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.101+0000 7f5c8de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c74003f70 0x7f5c9014c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.101+0000 7f5c8de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c74003f70 0x7f5c9014c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58984/0 (socket says 172.21.15.89:58984) 2024-04-02T09:15:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.101+0000 7f5c8de00700 1 -- 172.21.15.89:0/1716622022 learned_addr learned my addr 172.21.15.89:0/1716622022 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.101+0000 7f5c8d400700 1 --2- 172.21.15.89:0/1716622022 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c9015d290 0x7f5c9015ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.101+0000 7f5c8e800700 1 --2- 172.21.15.89:0/1716622022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c90156da0 0x7f5c90157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.101+0000 7f5c8de00700 1 -- 172.21.15.89:0/1716622022 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c9015d290 msgr2=0x7f5c9015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.101+0000 7f5c8de00700 1 --2- 172.21.15.89:0/1716622022 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c9015d290 0x7f5c9015ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.101+0000 7f5c8de00700 1 -- 172.21.15.89:0/1716622022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c90156da0 msgr2=0x7f5c90157230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.101+0000 7f5c8de00700 1 --2- 172.21.15.89:0/1716622022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c90156da0 0x7f5c90157230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.101+0000 7f5c8de00700 1 -- 172.21.15.89:0/1716622022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9015c290 con 0x7f5c74003f70 2024-04-02T09:15:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.101+0000 7f5c8de00700 1 --2- 172.21.15.89:0/1716622022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c74003f70 0x7f5c9014c700 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f5c80011ad0 tx=0x7f5c80011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.101+0000 7f5c86000700 1 -- 172.21.15.89:0/1716622022 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8000f940 con 0x7f5c74003f70 2024-04-02T09:15:21.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.101+0000 7f5c86000700 1 -- 172.21.15.89:0/1716622022 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8000faa0 con 0x7f5c74003f70 2024-04-02T09:15:21.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.101+0000 7f5c95000700 1 -- 172.21.15.89:0/1716622022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c9015c570 con 0x7f5c74003f70 2024-04-02T09:15:21.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.102+0000 7f5c86000700 1 -- 172.21.15.89:0/1716622022 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8000fe50 con 0x7f5c74003f70 2024-04-02T09:15:21.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.102+0000 7f5c95000700 1 -- 172.21.15.89:0/1716622022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c90156880 con 0x7f5c74003f70 2024-04-02T09:15:21.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.103+0000 7f5c95000700 1 -- 172.21.15.89:0/1716622022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c90067720 con 0x7f5c74003f70 2024-04-02T09:15:21.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.104+0000 7f5c86000700 1 -- 172.21.15.89:0/1716622022 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5c80014440 con 0x7f5c74003f70 2024-04-02T09:15:21.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.105+0000 7f5c86000700 1 --2- 172.21.15.89:0/1716622022 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c7006adb0 0x7f5c7006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:21.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.105+0000 7f5c86000700 1 -- 172.21.15.89:0/1716622022 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f5c80089ce0 con 0x7f5c74003f70 2024-04-02T09:15:21.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.105+0000 7f5c8d400700 1 --2- 172.21.15.89:0/1716622022 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c7006adb0 0x7f5c7006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:21.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.105+0000 7f5c8d400700 1 --2- 172.21.15.89:0/1716622022 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c7006adb0 0x7f5c7006d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c000c00 tx=0x7f5c7c005040 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:21.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.109+0000 7f5c86000700 1 -- 172.21.15.89:0/1716622022 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c80054cc0 con 0x7f5c74003f70 2024-04-02T09:15:21.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:21.326+0000 7f5c95000700 1 -- 172.21.15.89:0/1716622022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5c900b1ad0 con 0x7f5c74003f70 2024-04-02T09:15:22.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.062+0000 7f5c86000700 1 -- 172.21.15.89:0/1716622022 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v275) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5c80022020 con 0x7f5c74003f70 2024-04-02T09:15:22.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.065+0000 7f5c95000700 1 -- 172.21.15.89:0/1716622022 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c7006adb0 msgr2=0x7f5c7006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:22.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.065+0000 7f5c95000700 1 --2- 172.21.15.89:0/1716622022 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c7006adb0 0x7f5c7006d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c000c00 tx=0x7f5c7c005040 comp rx=0 tx=0).stop 2024-04-02T09:15:22.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.065+0000 7f5c95000700 1 -- 172.21.15.89:0/1716622022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c74003f70 msgr2=0x7f5c9014c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.065+0000 7f5c95000700 1 --2- 172.21.15.89:0/1716622022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c74003f70 0x7f5c9014c700 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f5c80011ad0 tx=0x7f5c80011de0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.065+0000 7f5c95000700 1 -- 172.21.15.89:0/1716622022 shutdown_connections 2024-04-02T09:15:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.065+0000 7f5c95000700 1 --2- 172.21.15.89:0/1716622022 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5c7006adb0 0x7f5c7006d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.065+0000 7f5c95000700 1 --2- 172.21.15.89:0/1716622022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c90156da0 0x7f5c90157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.065+0000 7f5c95000700 1 --2- 172.21.15.89:0/1716622022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c74003f70 0x7f5c9014c700 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.065+0000 7f5c95000700 1 --2- 172.21.15.89:0/1716622022 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5c9015d290 0x7f5c9015ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.065+0000 7f5c95000700 1 -- 172.21.15.89:0/1716622022 >> 172.21.15.89:0/1716622022 conn(0x7f5c900b0d70 msgr2=0x7f5c9006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.065+0000 7f5c95000700 1 -- 172.21.15.89:0/1716622022 shutdown_connections 2024-04-02T09:15:22.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.065+0000 7f5c95000700 1 -- 172.21.15.89:0/1716622022 wait complete. 2024-04-02T09:15:22.064 INFO:teuthology.orchestra.run.smithi089.stderr: removed all blocklist entries 2024-04-02T09:15:22.084 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-02T09:15:22.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.279+0000 7f58bfe00700 1 Processor -- start 2024-04-02T09:15:22.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.279+0000 7f58bfe00700 1 -- start start 2024-04-02T09:15:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.280+0000 7f58bfe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58b80d18b0 0x7f58b80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.280+0000 7f58bfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f58b8161590 0x7f58b8161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.280+0000 7f58bfe00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b80ceda0 con 0x7f58b8160f10 2024-04-02T09:15:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.280+0000 7f58bfe00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b80d1d30 con 0x7f58b8161590 2024-04-02T09:15:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.280+0000 7f58bfe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b8161f60 con 0x7f58b80d18b0 2024-04-02T09:15:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.280+0000 7f58bde00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f58b8161590 0x7f58b8161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.281+0000 7f58bde00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f58b8161590 0x7f58b8161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58986/0 (socket says 172.21.15.89:58986) 2024-04-02T09:15:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.281+0000 7f58bde00700 1 -- 172.21.15.89:0/2114531126 learned_addr learned my addr 172.21.15.89:0/2114531126 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.281+0000 7f58bca00700 1 --2- 172.21.15.89:0/2114531126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58b80d18b0 0x7f58b80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.281+0000 7f58bde00700 1 -- 172.21.15.89:0/2114531126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58b80d18b0 msgr2=0x7f58b80cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.281+0000 7f58bde00700 1 --2- 172.21.15.89:0/2114531126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58b80d18b0 0x7f58b80cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.281+0000 7f58bde00700 1 -- 172.21.15.89:0/2114531126 >> v1:172.21.15.113:6789/0 conn(0x7f58b8160f10 legacy=0x7f58b80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.281+0000 7f58bde00700 1 -- 172.21.15.89:0/2114531126 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b81612f0 con 0x7f58b8161590 2024-04-02T09:15:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.281+0000 7f58bde00700 1 --2- 172.21.15.89:0/2114531126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b8161590 0x7f58b8161a20 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7f58b0009e80 tx=0x7f58b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396bed4fd3a34753 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.281+0000 7f58b7e00700 1 -- 172.21.15.89:0/2114531126 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b000e070 con 0x7f58b8161590 2024-04-02T09:15:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.281+0000 7f58b7e00700 1 -- 172.21.15.89:0/2114531126 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b000b980 con 0x7f58b8161590 2024-04-02T09:15:22.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.281+0000 7f58b7e00700 1 -- 172.21.15.89:0/2114531126 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b000bd30 con 0x7f58b8161590 2024-04-02T09:15:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.282+0000 7f58bfe00700 1 -- 172.21.15.89:0/2114531126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b8161590 msgr2=0x7f58b8161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.282+0000 7f58bfe00700 1 --2- 172.21.15.89:0/2114531126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b8161590 0x7f58b8161a20 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7f58b0009e80 tx=0x7f58b000b330 comp rx=0 tx=0).stop 2024-04-02T09:15:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.282+0000 7f58bfe00700 1 -- 172.21.15.89:0/2114531126 shutdown_connections 2024-04-02T09:15:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.282+0000 7f58bfe00700 1 --2- 172.21.15.89:0/2114531126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b8161590 0x7f58b8161a20 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.282+0000 7f58bfe00700 1 --2- 172.21.15.89:0/2114531126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58b80d18b0 0x7f58b80cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.282+0000 7f58bfe00700 1 -- 172.21.15.89:0/2114531126 >> 172.21.15.89:0/2114531126 conn(0x7f58b808aff0 msgr2=0x7f58b80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.282+0000 7f58bfe00700 1 -- 172.21.15.89:0/2114531126 shutdown_connections 2024-04-02T09:15:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.282+0000 7f58bfe00700 1 -- 172.21.15.89:0/2114531126 wait complete. 2024-04-02T09:15:22.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.284+0000 7f58bfe00700 1 Processor -- start 2024-04-02T09:15:22.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.284+0000 7f58bfe00700 1 -- start start 2024-04-02T09:15:22.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.285+0000 7f58bfe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58b80d18b0 0x7f58b8158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.285+0000 7f58bfe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b8160f10 0x7f58b80b75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.285+0000 7f58bfe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58b8161590 0x7f58b8156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.285+0000 7f58bfe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b80b85b0 con 0x7f58b8160f10 2024-04-02T09:15:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.285+0000 7f58bfe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b80b7ff0 con 0x7f58b8161590 2024-04-02T09:15:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.285+0000 7f58bfe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b80b82d0 con 0x7f58b80d18b0 2024-04-02T09:15:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.285+0000 7f58bca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b8160f10 0x7f58b80b75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.285+0000 7f58bca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b8160f10 0x7f58b80b75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59002/0 (socket says 172.21.15.89:59002) 2024-04-02T09:15:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.285+0000 7f58bca00700 1 -- 172.21.15.89:0/3046369998 learned_addr learned my addr 172.21.15.89:0/3046369998 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.285+0000 7f58bde00700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58b8161590 0x7f58b8156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.285+0000 7f58bd400700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58b80d18b0 0x7f58b8158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.286+0000 7f58bca00700 1 -- 172.21.15.89:0/3046369998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58b80d18b0 msgr2=0x7f58b8158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.286+0000 7f58bca00700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58b80d18b0 0x7f58b8158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.286+0000 7f58bca00700 1 -- 172.21.15.89:0/3046369998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58b8161590 msgr2=0x7f58b8156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.286+0000 7f58bca00700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58b8161590 0x7f58b8156f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.286+0000 7f58bca00700 1 -- 172.21.15.89:0/3046369998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b0009b30 con 0x7f58b8160f10 2024-04-02T09:15:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.286+0000 7f58bd400700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58b80d18b0 0x7f58b8158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.286+0000 7f58bca00700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b8160f10 0x7f58b80b75a0 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f58a400d800 tx=0x7f58a400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.286+0000 7f58b5600700 1 -- 172.21.15.89:0/3046369998 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a40125f0 con 0x7f58b8160f10 2024-04-02T09:15:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.286+0000 7f58b5600700 1 -- 172.21.15.89:0/3046369998 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a4012750 con 0x7f58b8160f10 2024-04-02T09:15:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.286+0000 7f58b5600700 1 -- 172.21.15.89:0/3046369998 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4012b00 con 0x7f58b8160f10 2024-04-02T09:15:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.286+0000 7f58bfe00700 1 -- 172.21.15.89:0/3046369998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b8156710 con 0x7f58b8160f10 2024-04-02T09:15:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.286+0000 7f58bfe00700 1 -- 172.21.15.89:0/3046369998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b815d1f0 con 0x7f58b8160f10 2024-04-02T09:15:22.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.287+0000 7f58bfe00700 1 -- 172.21.15.89:0/3046369998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b80729f0 con 0x7f58b8160f10 2024-04-02T09:15:22.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.288+0000 7f58b5600700 1 -- 172.21.15.89:0/3046369998 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f58a4012c60 con 0x7f58b8160f10 2024-04-02T09:15:22.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.289+0000 7f58b5600700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f589806adb0 0x7f589806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:22.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.289+0000 7f58b5600700 1 -- 172.21.15.89:0/3046369998 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f58a40896a0 con 0x7f58b8160f10 2024-04-02T09:15:22.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.289+0000 7f58bd400700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f589806adb0 0x7f589806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:22.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.290+0000 7f58bd400700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f589806adb0 0x7f589806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f58ac000f60 tx=0x7f58ac009040 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:22.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.294+0000 7f58b5600700 1 -- 172.21.15.89:0/3046369998 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f58a4054860 con 0x7f58b8160f10 2024-04-02T09:15:22.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.573+0000 7f58bfe00700 1 -- 172.21.15.89:0/3046369998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f58b815d380 con 0x7f58b8160f10 2024-04-02T09:15:22.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.573+0000 7f58b5600700 1 -- 172.21.15.89:0/3046369998 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v57) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f58a4057e80 con 0x7f58b8160f10 2024-04-02T09:15:22.572 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:22.572 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:15:22.572 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_dump": [ 2024-04-02T09:15:22.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.572 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.a", 2024-04-02T09:15:22.572 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtm25KfGBAANzNne7IfZLHC5njMAwSv3Q==", 2024-04-02T09:15:22.572 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.572 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:15:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:15:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:15:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.b", 2024-04-02T09:15:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmW1xOLxAAtyH4+jNVZEW0CskIUbORPw==", 2024-04-02T09:15:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:15:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:15:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:15:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.574 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.c", 2024-04-02T09:15:22.574 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmlfG6IxAAkUvnNoyhkzpjJr8wx6kd8g==", 2024-04-02T09:15:22.574 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.574 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:15:22.574 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:15:22.574 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:15:22.574 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.574 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mds.d", 2024-04-02T09:15:22.574 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmHZFxOhAAAURLxOWY/ZICyUa4nm3NrA==", 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.0", 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAatQtmsXk8ChAAxpwe6PmbVCuKkAN0riE1gg==", 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.575 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.576 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.1", 2024-04-02T09:15:22.576 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAdtQtmtHpuDhAASNLmi2obsmBGptrWF3M69w==", 2024-04-02T09:15:22.576 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.576 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:15:22.576 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:15:22.576 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.576 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.576 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.2", 2024-04-02T09:15:22.576 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAgtQtmeOvFERAAK+o4h0V2SGgUmGiLBd9OXA==", 2024-04-02T09:15:22.576 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.577 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:15:22.577 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:15:22.577 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.577 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.577 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.3", 2024-04-02T09:15:22.577 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAjtQtmUBFHFRAAiPA1Ff6Qnkq4Emf4QmXdhg==", 2024-04-02T09:15:22.577 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.577 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:15:22.577 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:15:22.577 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.577 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.578 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.4", 2024-04-02T09:15:22.578 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAptQtmoBARMhAA155GM1jOMAEqTP16cJmLVg==", 2024-04-02T09:15:22.578 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.578 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:15:22.578 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:15:22.578 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.578 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.578 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.5", 2024-04-02T09:15:22.578 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAstQtmwJBpNhAAeMMDN+foe4f6yRCFNDtT2Q==", 2024-04-02T09:15:22.578 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.579 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:15:22.579 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:15:22.579 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.579 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.579 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.6", 2024-04-02T09:15:22.579 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAvtQtmpIHZORAAImJOn4tsP/cLcJJeMS9naw==", 2024-04-02T09:15:22.579 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.579 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:15:22.579 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:15:22.579 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.579 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "osd.7", 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAztQtmB5VGARAAaoWxHGYJB3cfUHH9dAMI3Q==", 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow profile osd", 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile osd", 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.0", 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmfeyfCBAAGhyv/vojdF069MufnXygPQ==", 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.580 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:15:22.581 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow r", 2024-04-02T09:15:22.581 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-02T09:15:22.581 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.581 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.1", 2024-04-02T09:15:22.581 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmZ0dYFxAAGfcA31KvG+u6dgsFAHjc+A==", 2024-04-02T09:15:22.581 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.581 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:15:22.581 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:15:22.581 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:15:22.581 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:15:22.581 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.2", 2024-04-02T09:15:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtm3UnWDxAA3l7RB1Ld6upM6imNFRwrAg==", 2024-04-02T09:15:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:15:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:15:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:15:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:15:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.3", 2024-04-02T09:15:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAWtQtmBGybHhAACj3naCYb6W3s5oCZe7rrAQ==", 2024-04-02T09:15:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.583 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow", 2024-04-02T09:15:22.583 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow r", 2024-04-02T09:15:22.583 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow rw", 2024-04-02T09:15:22.583 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow rwx" 2024-04-02T09:15:22.583 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.583 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "client.admin", 2024-04-02T09:15:22.583 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmccFhABAAh/ZYeHsmQnz60gIc+fauZw==", 2024-04-02T09:15:22.584 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.584 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:15:22.584 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": "allow *", 2024-04-02T09:15:22.584 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow *", 2024-04-02T09:15:22.584 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.584 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.584 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.x", 2024-04-02T09:15:22.584 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAVtQtmKUojERAAFL8wwITMbnpxCwDxsKjBog==", 2024-04-02T09:15:22.584 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.584 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:15:22.584 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T09:15:22.584 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.585 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-02T09:15:22.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:22.585 INFO:teuthology.orchestra.run.smithi089.stdout: "entity": "mgr.y", 2024-04-02T09:15:22.585 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "AQAUtQtmWXdyJxAApbJg6cbCk8XI5DmaVkLiwA==", 2024-04-02T09:15:22.585 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": { 2024-04-02T09:15:22.585 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": "allow *", 2024-04-02T09:15:22.585 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": "allow profile mgr", 2024-04-02T09:15:22.585 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": "allow *" 2024-04-02T09:15:22.585 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.585 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:22.585 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-02T09:15:22.585 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:15:22.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.576+0000 7f58bfe00700 1 -- 172.21.15.89:0/3046369998 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f589806adb0 msgr2=0x7f589806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:22.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.576+0000 7f58bfe00700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f589806adb0 0x7f589806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f58ac000f60 tx=0x7f58ac009040 comp rx=0 tx=0).stop 2024-04-02T09:15:22.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.576+0000 7f58bfe00700 1 -- 172.21.15.89:0/3046369998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b8160f10 msgr2=0x7f58b80b75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.576+0000 7f58bfe00700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b8160f10 0x7f58b80b75a0 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f58a400d800 tx=0x7f58a400db10 comp rx=0 tx=0).stop 2024-04-02T09:15:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.576+0000 7f58bfe00700 1 -- 172.21.15.89:0/3046369998 shutdown_connections 2024-04-02T09:15:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.576+0000 7f58bfe00700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f589806adb0 0x7f589806d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.576+0000 7f58bfe00700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58b8161590 0x7f58b8156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.576+0000 7f58bfe00700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58b80d18b0 0x7f58b8158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.576+0000 7f58bfe00700 1 --2- 172.21.15.89:0/3046369998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b8160f10 0x7f58b80b75a0 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.576+0000 7f58bfe00700 1 -- 172.21.15.89:0/3046369998 >> 172.21.15.89:0/3046369998 conn(0x7f58b808aff0 msgr2=0x7f58b806bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.576+0000 7f58bfe00700 1 -- 172.21.15.89:0/3046369998 shutdown_connections 2024-04-02T09:15:22.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.576+0000 7f58bfe00700 1 -- 172.21.15.89:0/3046369998 wait complete. 2024-04-02T09:15:22.593 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:15:22.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.789+0000 7f8032e00700 1 Processor -- start 2024-04-02T09:15:22.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.789+0000 7f8032e00700 1 -- start start 2024-04-02T09:15:22.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.790+0000 7f8032e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c160f10 0x7f802c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:22.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.790+0000 7f8032e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f802c0cf230 0x7f802c0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:22.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.790+0000 7f8032e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f802c161590 0x7f802c161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:22.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.790+0000 7f8032e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f802c161f60 con 0x7f802c0cf230 2024-04-02T09:15:22.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.790+0000 7f8032e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f802c0ceda0 con 0x7f802c161590 2024-04-02T09:15:22.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.790+0000 7f8032e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802c0d23d0 con 0x7f802c160f10 2024-04-02T09:15:22.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.790+0000 7f8023e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f802c0cf230 0x7f802c0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:22.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.790+0000 7f8023e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f802c0cf230 0x7f802c0d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59004/0 (socket says 172.21.15.89:59004) 2024-04-02T09:15:22.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.790+0000 7f8023e00700 1 -- 172.21.15.89:0/24745293 learned_addr learned my addr 172.21.15.89:0/24745293 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:22.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.790+0000 7f802be00700 1 --2- 172.21.15.89:0/24745293 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c160f10 0x7f802c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:22.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.790+0000 7f8030e00700 1 --2- 172.21.15.89:0/24745293 >> v2:172.21.15.113:3300/0 conn(0x7f802c161590 0x7f802c161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:22.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.791+0000 7f8023e00700 1 -- 172.21.15.89:0/24745293 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c160f10 msgr2=0x7f802c0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:22.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.791+0000 7f8023e00700 1 --2- 172.21.15.89:0/24745293 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c160f10 0x7f802c0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.791+0000 7f8023e00700 1 -- 172.21.15.89:0/24745293 >> v2:172.21.15.113:3300/0 conn(0x7f802c161590 msgr2=0x7f802c161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:22.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.791+0000 7f8023e00700 1 --2- 172.21.15.89:0/24745293 >> v2:172.21.15.113:3300/0 conn(0x7f802c161590 0x7f802c161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.791+0000 7f8023e00700 1 -- 172.21.15.89:0/24745293 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c1612f0 con 0x7f802c0cf230 2024-04-02T09:15:22.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.791+0000 7f8023e00700 1 --2- 172.21.15.89:0/24745293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cf230 0x7f802c0d1e90 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7f802c0cfad0 tx=0x7f8014009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02cc7fff74ce3f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.791+0000 7f802b400700 1 -- 172.21.15.89:0/24745293 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014011830 con 0x7f802c0cf230 2024-04-02T09:15:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.791+0000 7f802b400700 1 -- 172.21.15.89:0/24745293 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8014011990 con 0x7f802c0cf230 2024-04-02T09:15:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.791+0000 7f802b400700 1 -- 172.21.15.89:0/24745293 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014011d20 con 0x7f802c0cf230 2024-04-02T09:15:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.792+0000 7f8032e00700 1 -- 172.21.15.89:0/24745293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cf230 msgr2=0x7f802c0d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.792+0000 7f8032e00700 1 --2- 172.21.15.89:0/24745293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cf230 0x7f802c0d1e90 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7f802c0cfad0 tx=0x7f8014009bd0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.792+0000 7f8032e00700 1 -- 172.21.15.89:0/24745293 shutdown_connections 2024-04-02T09:15:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.792+0000 7f8032e00700 1 --2- 172.21.15.89:0/24745293 >> v2:172.21.15.113:3300/0 conn(0x7f802c161590 0x7f802c161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.792+0000 7f8032e00700 1 --2- 172.21.15.89:0/24745293 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c160f10 0x7f802c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.792+0000 7f8032e00700 1 --2- 172.21.15.89:0/24745293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cf230 0x7f802c0d1e90 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.792+0000 7f8032e00700 1 -- 172.21.15.89:0/24745293 >> 172.21.15.89:0/24745293 conn(0x7f802c08aff0 msgr2=0x7f802c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.792+0000 7f8032e00700 1 -- 172.21.15.89:0/24745293 shutdown_connections 2024-04-02T09:15:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.792+0000 7f8032e00700 1 -- 172.21.15.89:0/24745293 wait complete. 2024-04-02T09:15:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.794+0000 7f8032e00700 1 Processor -- start 2024-04-02T09:15:22.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.795+0000 7f8032e00700 1 -- start start 2024-04-02T09:15:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f8032e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cf230 0x7f802c1555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f8032e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c160f10 0x7f802c1f6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f8032e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f802c161590 0x7f802c1f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f8032e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c09e540 con 0x7f802c0cf230 2024-04-02T09:15:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f8032e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c09f5c0 con 0x7f802c161590 2024-04-02T09:15:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f8032e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802c0b4820 con 0x7f802c160f10 2024-04-02T09:15:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f802be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cf230 0x7f802c1555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f802be00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cf230 0x7f802c1555f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59014/0 (socket says 172.21.15.89:59014) 2024-04-02T09:15:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f802be00700 1 -- 172.21.15.89:0/621970545 learned_addr learned my addr 172.21.15.89:0/621970545 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f8023e00700 1 --2- 172.21.15.89:0/621970545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c160f10 0x7f802c1f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f8030e00700 1 --2- 172.21.15.89:0/621970545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f802c161590 0x7f802c1f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f802be00700 1 -- 172.21.15.89:0/621970545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c160f10 msgr2=0x7f802c1f6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f802be00700 1 --2- 172.21.15.89:0/621970545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c160f10 0x7f802c1f6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f802be00700 1 -- 172.21.15.89:0/621970545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f802c161590 msgr2=0x7f802c1f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f802be00700 1 --2- 172.21.15.89:0/621970545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f802c161590 0x7f802c1f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.796+0000 7f802be00700 1 -- 172.21.15.89:0/621970545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8014009550 con 0x7f802c0cf230 2024-04-02T09:15:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.797+0000 7f802be00700 1 --2- 172.21.15.89:0/621970545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cf230 0x7f802c1555f0 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7f801c00e8a0 tx=0x7f801c00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.797+0000 7f8028c00700 1 -- 172.21.15.89:0/621970545 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c009590 con 0x7f802c0cf230 2024-04-02T09:15:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.797+0000 7f8028c00700 1 -- 172.21.15.89:0/621970545 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801c00eea0 con 0x7f802c0cf230 2024-04-02T09:15:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.797+0000 7f8028c00700 1 -- 172.21.15.89:0/621970545 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c00ca50 con 0x7f802c0cf230 2024-04-02T09:15:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.797+0000 7f8032e00700 1 -- 172.21.15.89:0/621970545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802c09e790 con 0x7f802c0cf230 2024-04-02T09:15:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.797+0000 7f8032e00700 1 -- 172.21.15.89:0/621970545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802c09ef70 con 0x7f802c0cf230 2024-04-02T09:15:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.798+0000 7f8032e00700 1 -- 172.21.15.89:0/621970545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802c067720 con 0x7f802c0cf230 2024-04-02T09:15:22.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.799+0000 7f8028c00700 1 -- 172.21.15.89:0/621970545 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f801c00cbb0 con 0x7f802c0cf230 2024-04-02T09:15:22.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.800+0000 7f8028c00700 1 --2- 172.21.15.89:0/621970545 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f800806adb0 0x7f800806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:22.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.800+0000 7f8028c00700 1 -- 172.21.15.89:0/621970545 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f801c089350 con 0x7f802c0cf230 2024-04-02T09:15:22.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.800+0000 7f8023e00700 1 --2- 172.21.15.89:0/621970545 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f800806adb0 0x7f800806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:22.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.801+0000 7f8023e00700 1 --2- 172.21.15.89:0/621970545 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f800806adb0 0x7f800806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8014014040 tx=0x7f8014008030 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:22.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:22.804+0000 7f8028c00700 1 -- 172.21.15.89:0/621970545 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f801c054510 con 0x7f802c0cf230 2024-04-02T09:15:23.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.004+0000 7f8032e00700 1 -- 172.21.15.89:0/621970545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f802c0d0ad0 con 0x7f802c0cf230 2024-04-02T09:15:23.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.005+0000 7f8028c00700 1 -- 172.21.15.89:0/621970545 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v275) v1 ==== 74+0+13088 (secure 0 0 0) 0x7f801c057b30 con 0x7f802c0cf230 2024-04-02T09:15:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:23.003 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":275,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:15:22.054710+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:15:23.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.007+0000 7f8032e00700 1 -- 172.21.15.89:0/621970545 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f800806adb0 msgr2=0x7f800806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.007+0000 7f8032e00700 1 --2- 172.21.15.89:0/621970545 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f800806adb0 0x7f800806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8014014040 tx=0x7f8014008030 comp rx=0 tx=0).stop 2024-04-02T09:15:23.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.007+0000 7f8032e00700 1 -- 172.21.15.89:0/621970545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cf230 msgr2=0x7f802c1555f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.007+0000 7f8032e00700 1 --2- 172.21.15.89:0/621970545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cf230 0x7f802c1555f0 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7f801c00e8a0 tx=0x7f801c00ec60 comp rx=0 tx=0).stop 2024-04-02T09:15:23.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.007+0000 7f8032e00700 1 -- 172.21.15.89:0/621970545 shutdown_connections 2024-04-02T09:15:23.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.007+0000 7f8032e00700 1 --2- 172.21.15.89:0/621970545 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f800806adb0 0x7f800806d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.007+0000 7f8032e00700 1 --2- 172.21.15.89:0/621970545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f802c161590 0x7f802c1f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.007+0000 7f8032e00700 1 --2- 172.21.15.89:0/621970545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f802c0cf230 0x7f802c1555f0 unknown :-1 s=CLOSED pgs=2954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.007+0000 7f8032e00700 1 --2- 172.21.15.89:0/621970545 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f802c160f10 0x7f802c1f6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.007+0000 7f8032e00700 1 -- 172.21.15.89:0/621970545 >> 172.21.15.89:0/621970545 conn(0x7f802c08aff0 msgr2=0x7f802c0d1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:23.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.007+0000 7f8032e00700 1 -- 172.21.15.89:0/621970545 shutdown_connections 2024-04-02T09:15:23.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.007+0000 7f8032e00700 1 -- 172.21.15.89:0/621970545 wait complete. 2024-04-02T09:15:23.025 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:15:23.025 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:15:23.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.220+0000 7f71b9400700 1 Processor -- start 2024-04-02T09:15:23.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.221+0000 7f71b9400700 1 -- start start 2024-04-02T09:15:23.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.222+0000 7f71b9400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f71b40cee10 0x7f71b4161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:23.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.222+0000 7f71b9400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b40ce270 con 0x7f71b41611b0 2024-04-02T09:15:23.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.222+0000 7f71b9400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b415b600 con 0x7f71b40d21f0 2024-04-02T09:15:23.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.222+0000 7f71b9400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71b40d2760 con 0x7f71b40cee10 2024-04-02T09:15:23.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.222+0000 7f71b3200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f71b40cee10 0x7f71b4161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:23.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.222+0000 7f71b2800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f71b41611b0 0x7f71b415ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50670/0 (socket says 172.21.15.89:50670) 2024-04-02T09:15:23.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.222+0000 7f71b2800700 1 -- 172.21.15.89:0/1791160303 learned_addr learned my addr 172.21.15.89:0/1791160303 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:23.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.222+0000 7f71b3200700 1 -- 172.21.15.89:0/1791160303 >> v1:172.21.15.113:6789/0 conn(0x7f71b40d21f0 legacy=0x7f71b40ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:23.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.222+0000 7f71b1400700 1 -- 172.21.15.89:0/1791160303 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651821696 0 0) 0x7f71b40ce270 con 0x7f71b41611b0 2024-04-02T09:15:23.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.222+0000 7f71b3200700 1 -- 172.21.15.89:0/1791160303 >> v1:172.21.15.89:6789/0 conn(0x7f71b41611b0 legacy=0x7f71b415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.222+0000 7f71b3200700 1 -- 172.21.15.89:0/1791160303 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b40d2a40 con 0x7f71b40cee10 2024-04-02T09:15:23.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.223+0000 7f71b3200700 1 --2- 172.21.15.89:0/1791160303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71b40cee10 0x7f71b4161590 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009d90 tx=0x7f71ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea0e0dfad981c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:23.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.223+0000 7f71b1400700 1 -- 172.21.15.89:0/1791160303 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac00e070 con 0x7f71b40cee10 2024-04-02T09:15:23.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.223+0000 7f71b1400700 1 -- 172.21.15.89:0/1791160303 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ac00bbd0 con 0x7f71b40cee10 2024-04-02T09:15:23.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.223+0000 7f71b1400700 1 -- 172.21.15.89:0/1791160303 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac003a20 con 0x7f71b40cee10 2024-04-02T09:15:23.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.223+0000 7f71b9400700 1 -- 172.21.15.89:0/1791160303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71b40cee10 msgr2=0x7f71b4161590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.223+0000 7f71b9400700 1 --2- 172.21.15.89:0/1791160303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71b40cee10 0x7f71b4161590 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009d90 tx=0x7f71ac00b330 comp rx=0 tx=0).stop 2024-04-02T09:15:23.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.223+0000 7f71b9400700 1 -- 172.21.15.89:0/1791160303 shutdown_connections 2024-04-02T09:15:23.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.223+0000 7f71b9400700 1 --2- 172.21.15.89:0/1791160303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71b40cee10 0x7f71b4161590 unknown :-1 s=CLOSED pgs=2956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.223+0000 7f71b9400700 1 -- 172.21.15.89:0/1791160303 >> 172.21.15.89:0/1791160303 conn(0x7f71b408aff0 msgr2=0x7f71b40d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:23.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.224+0000 7f71b9400700 1 -- 172.21.15.89:0/1791160303 shutdown_connections 2024-04-02T09:15:23.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.224+0000 7f71b9400700 1 -- 172.21.15.89:0/1791160303 wait complete. 2024-04-02T09:15:23.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.225+0000 7f71b9400700 1 Processor -- start 2024-04-02T09:15:23.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.225+0000 7f71b9400700 1 -- start start 2024-04-02T09:15:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.226+0000 7f71b9400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71b40cee10 0x7f71b415a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.226+0000 7f71b9400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71b40d21f0 0x7f71b40c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.226+0000 7f71b9400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71b41611b0 0x7f71b41e8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.226+0000 7f71b9400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b40c7100 con 0x7f71b40d21f0 2024-04-02T09:15:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.226+0000 7f71b9400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b40c73e0 con 0x7f71b40cee10 2024-04-02T09:15:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.226+0000 7f71b9400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b40c76c0 con 0x7f71b41611b0 2024-04-02T09:15:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.226+0000 7f71b1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71b40d21f0 0x7f71b40c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71b1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71b40d21f0 0x7f71b40c81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59032/0 (socket says 172.21.15.89:59032) 2024-04-02T09:15:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71b1e00700 1 -- 172.21.15.89:0/804518370 learned_addr learned my addr 172.21.15.89:0/804518370 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71b2800700 1 --2- 172.21.15.89:0/804518370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71b40cee10 0x7f71b415a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71b3200700 1 --2- 172.21.15.89:0/804518370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71b41611b0 0x7f71b41e8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:23.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71b1e00700 1 -- 172.21.15.89:0/804518370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71b41611b0 msgr2=0x7f71b41e8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71b1e00700 1 --2- 172.21.15.89:0/804518370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71b41611b0 0x7f71b41e8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71b1e00700 1 -- 172.21.15.89:0/804518370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71b40cee10 msgr2=0x7f71b415a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71b1e00700 1 --2- 172.21.15.89:0/804518370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71b40cee10 0x7f71b415a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71b1e00700 1 -- 172.21.15.89:0/804518370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac009a70 con 0x7f71b40d21f0 2024-04-02T09:15:23.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71b1e00700 1 --2- 172.21.15.89:0/804518370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71b40d21f0 0x7f71b40c81a0 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7f71a000c4d0 tx=0x7f71a000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:23.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71aaa00700 1 -- 172.21.15.89:0/804518370 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a0011790 con 0x7f71b40d21f0 2024-04-02T09:15:23.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71aaa00700 1 -- 172.21.15.89:0/804518370 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a00118f0 con 0x7f71b40d21f0 2024-04-02T09:15:23.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71aaa00700 1 -- 172.21.15.89:0/804518370 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a0011ca0 con 0x7f71b40d21f0 2024-04-02T09:15:23.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71b9400700 1 -- 172.21.15.89:0/804518370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b40c79a0 con 0x7f71b40d21f0 2024-04-02T09:15:23.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.227+0000 7f71b9400700 1 -- 172.21.15.89:0/804518370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b415a7e0 con 0x7f71b40d21f0 2024-04-02T09:15:23.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.228+0000 7f71b9400700 1 -- 172.21.15.89:0/804518370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b40c7ec0 con 0x7f71b40d21f0 2024-04-02T09:15:23.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.229+0000 7f71aaa00700 1 -- 172.21.15.89:0/804518370 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f71a001c460 con 0x7f71b40d21f0 2024-04-02T09:15:23.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.230+0000 7f71aaa00700 1 --2- 172.21.15.89:0/804518370 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f719406adb0 0x7f719406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:23.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.230+0000 7f71aaa00700 1 -- 172.21.15.89:0/804518370 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f71a0089180 con 0x7f71b40d21f0 2024-04-02T09:15:23.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.230+0000 7f71b2800700 1 --2- 172.21.15.89:0/804518370 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f719406adb0 0x7f719406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:23.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.231+0000 7f71b2800700 1 --2- 172.21.15.89:0/804518370 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f719406adb0 0x7f719406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f71a4000f60 tx=0x7f71a4007c90 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:23.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.234+0000 7f71aaa00700 1 -- 172.21.15.89:0/804518370 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f71b40c7ec0 con 0x7f71b40d21f0 2024-04-02T09:15:23.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.434+0000 7f71b9400700 1 -- 172.21.15.89:0/804518370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f71b40c7ec0 con 0x7f71b40d21f0 2024-04-02T09:15:23.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.435+0000 7f71aaa00700 1 -- 172.21.15.89:0/804518370 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v275) v1 ==== 93+0+10 (secure 0 0 0) 0x7f71b40c7ec0 con 0x7f71b40d21f0 2024-04-02T09:15:23.433 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:15:23.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.437+0000 7f71b9400700 1 -- 172.21.15.89:0/804518370 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f719406adb0 msgr2=0x7f719406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.437+0000 7f71b9400700 1 --2- 172.21.15.89:0/804518370 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f719406adb0 0x7f719406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f71a4000f60 tx=0x7f71a4007c90 comp rx=0 tx=0).stop 2024-04-02T09:15:23.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.437+0000 7f71b9400700 1 -- 172.21.15.89:0/804518370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71b40d21f0 msgr2=0x7f71b40c81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.437+0000 7f71b9400700 1 --2- 172.21.15.89:0/804518370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71b40d21f0 0x7f71b40c81a0 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7f71a000c4d0 tx=0x7f71a000c890 comp rx=0 tx=0).stop 2024-04-02T09:15:23.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.437+0000 7f71b9400700 1 -- 172.21.15.89:0/804518370 shutdown_connections 2024-04-02T09:15:23.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.437+0000 7f71b9400700 1 --2- 172.21.15.89:0/804518370 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f719406adb0 0x7f719406d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.437+0000 7f71b9400700 1 --2- 172.21.15.89:0/804518370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71b40cee10 0x7f71b415a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.437+0000 7f71b9400700 1 --2- 172.21.15.89:0/804518370 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71b41611b0 0x7f71b41e8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.438+0000 7f71b9400700 1 --2- 172.21.15.89:0/804518370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71b40d21f0 0x7f71b40c81a0 unknown :-1 s=CLOSED pgs=2957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.438+0000 7f71b9400700 1 -- 172.21.15.89:0/804518370 >> 172.21.15.89:0/804518370 conn(0x7f71b408aff0 msgr2=0x7f71b4161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:23.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.438+0000 7f71b9400700 1 -- 172.21.15.89:0/804518370 shutdown_connections 2024-04-02T09:15:23.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.438+0000 7f71b9400700 1 -- 172.21.15.89:0/804518370 wait complete. 2024-04-02T09:15:23.454 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T09:15:23.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.645+0000 7f4d20a00700 1 Processor -- start 2024-04-02T09:15:23.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.646+0000 7f4d20a00700 1 -- start start 2024-04-02T09:15:23.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.647+0000 7f4d20a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d1c15db40 0x7f4d1c0d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:23.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.647+0000 7f4d20a00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1c0d1330 con 0x7f4d1c15d760 2024-04-02T09:15:23.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.647+0000 7f4d20a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1c0d1610 con 0x7f4d1c0cf3c0 2024-04-02T09:15:23.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.647+0000 7f4d20a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c0d1fb0 con 0x7f4d1c15db40 2024-04-02T09:15:23.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.647+0000 7f4d19e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4d1c15d760 0x7f4d1c0cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50684/0 (socket says 172.21.15.89:50684) 2024-04-02T09:15:23.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.647+0000 7f4d19e00700 1 -- 172.21.15.89:0/3123043152 learned_addr learned my addr 172.21.15.89:0/3123043152 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:23.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.647+0000 7f4d1a800700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d1c15db40 0x7f4d1c0d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:23.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.647+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385813461 0 0) 0x7f4d1c0d1330 con 0x7f4d1c15d760 2024-04-02T09:15:23.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.647+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d000035b0 con 0x7f4d1c15d760 2024-04-02T09:15:23.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254680684 0 0) 0x7f4d1c0d1610 con 0x7f4d1c0cf3c0 2024-04-02T09:15:23.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d1c0d1330 con 0x7f4d1c0cf3c0 2024-04-02T09:15:23.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1500406673 0 0) 0x7f4d000035b0 con 0x7f4d1c15d760 2024-04-02T09:15:23.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d1c15db40 msgr2=0x7f4d1c0d23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:23.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d1c15db40 0x7f4d1c0d23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 >> v1:172.21.15.113:6789/0 conn(0x7f4d1c0cf3c0 legacy=0x7f4d1c0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c0d18f0 con 0x7f4d1c15d760 2024-04-02T09:15:23.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f4d0c001670 con 0x7f4d1c15d760 2024-04-02T09:15:23.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d1a800700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d1c15db40 0x7f4d1c0d23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:15:23.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 >> v1:172.21.15.89:6789/0 conn(0x7f4d1c15d760 legacy=0x7f4d1c0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d00003bc0 0x7f4d00003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:23.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d1c15db40 0x7f4d00004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:23.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d00004d90 0x7f4d00005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:23.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c0d1fb0 con 0x7f4d00003bc0 2024-04-02T09:15:23.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0000bb20 con 0x7f4d00004d90 2024-04-02T09:15:23.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0000bce0 con 0x7f4d1c15db40 2024-04-02T09:15:23.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d19e00700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d00003bc0 0x7f4d00003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:23.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d1a800700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d00004d90 0x7f4d00005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:23.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d19400700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d1c15db40 0x7f4d00004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:23.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d19e00700 1 -- 172.21.15.89:0/3123043152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d1c15db40 msgr2=0x7f4d00004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:23.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d19e00700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d1c15db40 0x7f4d00004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d19e00700 1 -- 172.21.15.89:0/3123043152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d00004d90 msgr2=0x7f4d00005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:23.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d19e00700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d00004d90 0x7f4d00005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d19e00700 1 -- 172.21.15.89:0/3123043152 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d14009a70 con 0x7f4d00003bc0 2024-04-02T09:15:23.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d19400700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d1c15db40 0x7f4d00004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:15:23.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.648+0000 7f4d1a800700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d00004d90 0x7f4d00005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:23.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.649+0000 7f4d19e00700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d00003bc0 0x7f4d00003fc0 secure :-1 s=READY pgs=2959 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00c210 tx=0x7f4d00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91fe3a684b62bd01 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:23.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.649+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c00cdb0 con 0x7f4d00003bc0 2024-04-02T09:15:23.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.649+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0c005960 con 0x7f4d00003bc0 2024-04-02T09:15:23.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.649+0000 7f4d18a00700 1 -- 172.21.15.89:0/3123043152 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c005cf0 con 0x7f4d00003bc0 2024-04-02T09:15:23.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.649+0000 7f4d20a00700 1 -- 172.21.15.89:0/3123043152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d00003bc0 msgr2=0x7f4d00003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.649+0000 7f4d20a00700 1 --2- 172.21.15.89:0/3123043152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d00003bc0 0x7f4d00003fc0 secure :-1 s=READY pgs=2959 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00c210 tx=0x7f4d00005880 comp rx=0 tx=0).stop 2024-04-02T09:15:23.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.649+0000 7f4d1a800700 1 -- 172.21.15.89:0/3123043152 reap_dead start 2024-04-02T09:15:23.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.649+0000 7f4d20a00700 1 -- 172.21.15.89:0/3123043152 shutdown_connections 2024-04-02T09:15:23.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.649+0000 7f4d20a00700 1 -- 172.21.15.89:0/3123043152 >> 172.21.15.89:0/3123043152 conn(0x7f4d1c08aff0 msgr2=0x7f4d1c0d35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:23.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.649+0000 7f4d20a00700 1 -- 172.21.15.89:0/3123043152 shutdown_connections 2024-04-02T09:15:23.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.649+0000 7f4d20a00700 1 -- 172.21.15.89:0/3123043152 wait complete. 2024-04-02T09:15:23.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.652+0000 7f4d20a00700 1 Processor -- start 2024-04-02T09:15:23.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.652+0000 7f4d20a00700 1 -- start start 2024-04-02T09:15:23.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.653+0000 7f4d20a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d00003bc0 0x7f4d1c1f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:23.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.653+0000 7f4d20a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c0b88d0 0x7f4d1c0c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:23.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.653+0000 7f4d20a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d1c0c6e20 0x7f4d1c148940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:23.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.653+0000 7f4d20a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c1f3510 con 0x7f4d1c0b88d0 2024-04-02T09:15:23.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.653+0000 7f4d20a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c0b8440 con 0x7f4d1c0c6e20 2024-04-02T09:15:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.653+0000 7f4d20a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c0c76c0 con 0x7f4d00003bc0 2024-04-02T09:15:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.653+0000 7f4d19400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c0b88d0 0x7f4d1c0c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.653+0000 7f4d19400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c0b88d0 0x7f4d1c0c9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59052/0 (socket says 172.21.15.89:59052) 2024-04-02T09:15:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.653+0000 7f4d19400700 1 -- 172.21.15.89:0/1780556356 learned_addr learned my addr 172.21.15.89:0/1780556356 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.654+0000 7f4d19e00700 1 --2- 172.21.15.89:0/1780556356 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d00003bc0 0x7f4d1c1f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.654+0000 7f4d1a800700 1 --2- 172.21.15.89:0/1780556356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d1c0c6e20 0x7f4d1c148940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.654+0000 7f4d19400700 1 -- 172.21.15.89:0/1780556356 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d00003bc0 msgr2=0x7f4d1c1f2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.654+0000 7f4d19400700 1 --2- 172.21.15.89:0/1780556356 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d00003bc0 0x7f4d1c1f2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.654+0000 7f4d19400700 1 -- 172.21.15.89:0/1780556356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d1c0c6e20 msgr2=0x7f4d1c148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.654+0000 7f4d19400700 1 --2- 172.21.15.89:0/1780556356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d1c0c6e20 0x7f4d1c148940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.654+0000 7f4d19400700 1 -- 172.21.15.89:0/1780556356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c0c73e0 con 0x7f4d1c0b88d0 2024-04-02T09:15:23.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.654+0000 7f4d19400700 1 --2- 172.21.15.89:0/1780556356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c0b88d0 0x7f4d1c0c9da0 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800d7a0 tx=0x7f4d0800db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:23.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.654+0000 7f4d12000700 1 -- 172.21.15.89:0/1780556356 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d08006a60 con 0x7f4d1c0b88d0 2024-04-02T09:15:23.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.654+0000 7f4d12000700 1 -- 172.21.15.89:0/1780556356 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d08006be0 con 0x7f4d1c0b88d0 2024-04-02T09:15:23.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.654+0000 7f4d12000700 1 -- 172.21.15.89:0/1780556356 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d08007030 con 0x7f4d1c0b88d0 2024-04-02T09:15:23.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.654+0000 7f4d20a00700 1 -- 172.21.15.89:0/1780556356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1c0b7df0 con 0x7f4d1c0b88d0 2024-04-02T09:15:23.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.655+0000 7f4d20a00700 1 -- 172.21.15.89:0/1780556356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1c0ca2e0 con 0x7f4d1c0b88d0 2024-04-02T09:15:23.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.656+0000 7f4d20a00700 1 -- 172.21.15.89:0/1780556356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d1c0b85f0 con 0x7f4d1c0b88d0 2024-04-02T09:15:23.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.656+0000 7f4d12000700 1 -- 172.21.15.89:0/1780556356 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4d08007190 con 0x7f4d1c0b88d0 2024-04-02T09:15:23.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.657+0000 7f4d12000700 1 --2- 172.21.15.89:0/1780556356 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4cfc06adb0 0x7f4cfc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:23.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.657+0000 7f4d12000700 1 -- 172.21.15.89:0/1780556356 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f4d080892c0 con 0x7f4d1c0b88d0 2024-04-02T09:15:23.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.657+0000 7f4d19e00700 1 --2- 172.21.15.89:0/1780556356 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4cfc06adb0 0x7f4cfc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:23.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.658+0000 7f4d19e00700 1 --2- 172.21.15.89:0/1780556356 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4cfc06adb0 0x7f4cfc06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c146780 tx=0x7f4d0c00a000 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:23.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.662+0000 7f4d12000700 1 -- 172.21.15.89:0/1780556356 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4d1c0b85f0 con 0x7f4d1c0b88d0 2024-04-02T09:15:23.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.867+0000 7f4d20a00700 1 -- 172.21.15.89:0/1780556356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4d1c0b7c30 con 0x7f4d1c0b88d0 2024-04-02T09:15:23.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.867+0000 7f4d12000700 1 -- 172.21.15.89:0/1780556356 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v275) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4d08054370 con 0x7f4d1c0b88d0 2024-04-02T09:15:23.865 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:23.866 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T09:15:23.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:23.866 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-04-02T09:15:23.866 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-04-02T09:15:23.866 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:23.866 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T09:15:23.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.869+0000 7f4d20a00700 1 -- 172.21.15.89:0/1780556356 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4cfc06adb0 msgr2=0x7f4cfc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.869+0000 7f4d20a00700 1 --2- 172.21.15.89:0/1780556356 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4cfc06adb0 0x7f4cfc06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c146780 tx=0x7f4d0c00a000 comp rx=0 tx=0).stop 2024-04-02T09:15:23.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.869+0000 7f4d20a00700 1 -- 172.21.15.89:0/1780556356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c0b88d0 msgr2=0x7f4d1c0c9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:23.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.869+0000 7f4d20a00700 1 --2- 172.21.15.89:0/1780556356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c0b88d0 0x7f4d1c0c9da0 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800d7a0 tx=0x7f4d0800db60 comp rx=0 tx=0).stop 2024-04-02T09:15:23.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.869+0000 7f4d20a00700 1 -- 172.21.15.89:0/1780556356 shutdown_connections 2024-04-02T09:15:23.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.869+0000 7f4d20a00700 1 --2- 172.21.15.89:0/1780556356 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4cfc06adb0 0x7f4cfc06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.869+0000 7f4d20a00700 1 --2- 172.21.15.89:0/1780556356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d1c0c6e20 0x7f4d1c148940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.869+0000 7f4d20a00700 1 --2- 172.21.15.89:0/1780556356 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4d00003bc0 0x7f4d1c1f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.869+0000 7f4d20a00700 1 --2- 172.21.15.89:0/1780556356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c0b88d0 0x7f4d1c0c9da0 unknown :-1 s=CLOSED pgs=2960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:23.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.869+0000 7f4d20a00700 1 -- 172.21.15.89:0/1780556356 >> 172.21.15.89:0/1780556356 conn(0x7f4d1c08aff0 msgr2=0x7f4d1c0b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:23.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.870+0000 7f4d20a00700 1 -- 172.21.15.89:0/1780556356 shutdown_connections 2024-04-02T09:15:23.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:23.870+0000 7f4d20a00700 1 -- 172.21.15.89:0/1780556356 wait complete. 2024-04-02T09:15:23.887 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T09:15:23.887 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T09:15:24.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.084+0000 7f2245e00700 1 Processor -- start 2024-04-02T09:15:24.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.085+0000 7f2245e00700 1 -- start start 2024-04-02T09:15:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.086+0000 7f2245e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f22400ced30 0x7f22400d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.086+0000 7f2245e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22400ce8a0 con 0x7f22400d2710 2024-04-02T09:15:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.086+0000 7f2245e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22400cdc90 con 0x7f224015ddf0 2024-04-02T09:15:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.086+0000 7f2245e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224015ec60 con 0x7f22400ced30 2024-04-02T09:15:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.086+0000 7f223f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f22400d2710 0x7f224015f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50690/0 (socket says 172.21.15.89:50690) 2024-04-02T09:15:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.086+0000 7f223f400700 1 -- 172.21.15.89:0/3472535011 learned_addr learned my addr 172.21.15.89:0/3472535011 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.086+0000 7f223e000700 1 --2- 172.21.15.89:0/3472535011 >> v2:172.21.15.113:3300/0 conn(0x7f22400ced30 0x7f22400d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.086+0000 7f223d600700 1 -- 172.21.15.89:0/3472535011 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391605319 0 0) 0x7f22400ce8a0 con 0x7f22400d2710 2024-04-02T09:15:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.086+0000 7f223d600700 1 -- 172.21.15.89:0/3472535011 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22240035b0 con 0x7f22400d2710 2024-04-02T09:15:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.087+0000 7f223e000700 1 -- 172.21.15.89:0/3472535011 >> v1:172.21.15.113:6789/0 conn(0x7f224015ddf0 legacy=0x7f224015e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.087+0000 7f223e000700 1 -- 172.21.15.89:0/3472535011 >> v1:172.21.15.89:6789/0 conn(0x7f22400d2710 legacy=0x7f224015f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:24.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.087+0000 7f223e000700 1 -- 172.21.15.89:0/3472535011 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22400cdf70 con 0x7f22400ced30 2024-04-02T09:15:24.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.087+0000 7f223e000700 1 --2- 172.21.15.89:0/3472535011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400ced30 0x7f22400d21d0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f222c009840 tx=0x7f222c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcfadeadc07adb75 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:24.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.087+0000 7f223d600700 1 -- 172.21.15.89:0/3472535011 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c011830 con 0x7f22400ced30 2024-04-02T09:15:24.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.087+0000 7f223d600700 1 -- 172.21.15.89:0/3472535011 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222c011990 con 0x7f22400ced30 2024-04-02T09:15:24.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.087+0000 7f223d600700 1 -- 172.21.15.89:0/3472535011 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c011d20 con 0x7f22400ced30 2024-04-02T09:15:24.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.088+0000 7f2245e00700 1 -- 172.21.15.89:0/3472535011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400ced30 msgr2=0x7f22400d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:24.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.088+0000 7f2245e00700 1 --2- 172.21.15.89:0/3472535011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400ced30 0x7f22400d21d0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f222c009840 tx=0x7f222c009b50 comp rx=0 tx=0).stop 2024-04-02T09:15:24.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.088+0000 7f2245e00700 1 -- 172.21.15.89:0/3472535011 shutdown_connections 2024-04-02T09:15:24.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.088+0000 7f2245e00700 1 --2- 172.21.15.89:0/3472535011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400ced30 0x7f22400d21d0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:24.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.088+0000 7f2245e00700 1 -- 172.21.15.89:0/3472535011 >> 172.21.15.89:0/3472535011 conn(0x7f224008aff0 msgr2=0x7f22400d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:24.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.088+0000 7f2245e00700 1 -- 172.21.15.89:0/3472535011 shutdown_connections 2024-04-02T09:15:24.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.088+0000 7f2245e00700 1 -- 172.21.15.89:0/3472535011 wait complete. 2024-04-02T09:15:24.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.090+0000 7f2245e00700 1 Processor -- start 2024-04-02T09:15:24.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.090+0000 7f2245e00700 1 -- start start 2024-04-02T09:15:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f2245e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22400ced30 0x7f2240156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f2245e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400d2710 0x7f224015edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f2245e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f224015ddf0 0x7f224015f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f2245e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400b3910 con 0x7f224015ddf0 2024-04-02T09:15:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f2245e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2240160760 con 0x7f22400d2710 2024-04-02T09:15:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f2245e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224015f940 con 0x7f22400ced30 2024-04-02T09:15:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f223f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f224015ddf0 0x7f224015f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f223f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f224015ddf0 0x7f224015f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59062/0 (socket says 172.21.15.89:59062) 2024-04-02T09:15:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f223f400700 1 -- 172.21.15.89:0/1568743467 learned_addr learned my addr 172.21.15.89:0/1568743467 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f223e000700 1 --2- 172.21.15.89:0/1568743467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400d2710 0x7f224015edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f223ea00700 1 --2- 172.21.15.89:0/1568743467 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22400ced30 0x7f2240156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f223f400700 1 -- 172.21.15.89:0/1568743467 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22400ced30 msgr2=0x7f2240156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f223f400700 1 --2- 172.21.15.89:0/1568743467 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22400ced30 0x7f2240156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f223f400700 1 -- 172.21.15.89:0/1568743467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400d2710 msgr2=0x7f224015edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f223f400700 1 --2- 172.21.15.89:0/1568743467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400d2710 0x7f224015edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.091+0000 7f223f400700 1 -- 172.21.15.89:0/1568743467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c0094a0 con 0x7f224015ddf0 2024-04-02T09:15:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.092+0000 7f223f400700 1 --2- 172.21.15.89:0/1568743467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f224015ddf0 0x7f224015f310 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 crypto rx=0x7f223800cb40 tx=0x7f223800cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.092+0000 7f2237400700 1 -- 172.21.15.89:0/1568743467 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223800ab50 con 0x7f224015ddf0 2024-04-02T09:15:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.092+0000 7f2237400700 1 -- 172.21.15.89:0/1568743467 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223800acb0 con 0x7f224015ddf0 2024-04-02T09:15:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.092+0000 7f2237400700 1 -- 172.21.15.89:0/1568743467 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2238012620 con 0x7f224015ddf0 2024-04-02T09:15:24.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.092+0000 7f2245e00700 1 -- 172.21.15.89:0/1568743467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224015fc20 con 0x7f224015ddf0 2024-04-02T09:15:24.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.092+0000 7f2245e00700 1 -- 172.21.15.89:0/1568743467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2240157090 con 0x7f224015ddf0 2024-04-02T09:15:24.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.093+0000 7f2245e00700 1 -- 172.21.15.89:0/1568743467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22400b3a80 con 0x7f224015ddf0 2024-04-02T09:15:24.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.094+0000 7f2237400700 1 -- 172.21.15.89:0/1568743467 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f22380113a0 con 0x7f224015ddf0 2024-04-02T09:15:24.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.095+0000 7f2237400700 1 --2- 172.21.15.89:0/1568743467 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f222006adb0 0x7f222006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:24.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.095+0000 7f2237400700 1 -- 172.21.15.89:0/1568743467 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f22380891a0 con 0x7f224015ddf0 2024-04-02T09:15:24.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.095+0000 7f223ea00700 1 --2- 172.21.15.89:0/1568743467 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f222006adb0 0x7f222006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.096+0000 7f223ea00700 1 --2- 172.21.15.89:0/1568743467 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f222006adb0 0x7f222006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2230000f60 tx=0x7f2230009040 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:24.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.099+0000 7f2237400700 1 -- 172.21.15.89:0/1568743467 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2238054360 con 0x7f224015ddf0 2024-04-02T09:15:24.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:24.299+0000 7f2245e00700 1 -- 172.21.15.89:0/1568743467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2240067720 con 0x7f224015ddf0 2024-04-02T09:15:25.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.263+0000 7f2237400700 1 -- 172.21.15.89:0/1568743467 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v276) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2238057980 con 0x7f224015ddf0 2024-04-02T09:15:25.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.268+0000 7f2245e00700 1 -- 172.21.15.89:0/1568743467 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f222006adb0 msgr2=0x7f222006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:25.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.268+0000 7f2245e00700 1 --2- 172.21.15.89:0/1568743467 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f222006adb0 0x7f222006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2230000f60 tx=0x7f2230009040 comp rx=0 tx=0).stop 2024-04-02T09:15:25.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.268+0000 7f2245e00700 1 -- 172.21.15.89:0/1568743467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f224015ddf0 msgr2=0x7f224015f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:25.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.268+0000 7f2245e00700 1 --2- 172.21.15.89:0/1568743467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f224015ddf0 0x7f224015f310 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 crypto rx=0x7f223800cb40 tx=0x7f223800cf00 comp rx=0 tx=0).stop 2024-04-02T09:15:25.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.268+0000 7f2245e00700 1 -- 172.21.15.89:0/1568743467 shutdown_connections 2024-04-02T09:15:25.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.268+0000 7f2245e00700 1 --2- 172.21.15.89:0/1568743467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f224015ddf0 0x7f224015f310 unknown :-1 s=CLOSED pgs=2962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:25.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.268+0000 7f2245e00700 1 --2- 172.21.15.89:0/1568743467 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22400ced30 0x7f2240156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:25.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.268+0000 7f2245e00700 1 --2- 172.21.15.89:0/1568743467 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f222006adb0 0x7f222006d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:25.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.268+0000 7f2245e00700 1 --2- 172.21.15.89:0/1568743467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22400d2710 0x7f224015edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:25.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.268+0000 7f2245e00700 1 -- 172.21.15.89:0/1568743467 >> 172.21.15.89:0/1568743467 conn(0x7f224008aff0 msgr2=0x7f22400b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:25.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.270+0000 7f2245e00700 1 -- 172.21.15.89:0/1568743467 shutdown_connections 2024-04-02T09:15:25.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.270+0000 7f2245e00700 1 -- 172.21.15.89:0/1568743467 wait complete. 2024-04-02T09:15:25.270 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-04-02T09:15:25.290 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T09:15:25.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.491+0000 7fc975800700 1 Processor -- start 2024-04-02T09:15:25.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.491+0000 7fc975800700 1 -- start start 2024-04-02T09:15:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.492+0000 7fc975800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc97015d9d0 0x7fc9700d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.492+0000 7fc975800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9700d2540 0x7fc9700ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.493+0000 7fc975800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc970161420 con 0x7fc9700d16a0 2024-04-02T09:15:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.493+0000 7fc975800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc97015edc0 con 0x7fc97015d9d0 2024-04-02T09:15:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.493+0000 7fc975800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc970161700 con 0x7fc9700d2540 2024-04-02T09:15:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.493+0000 7fc96e000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc9700d16a0 0x7fc9700d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50704/0 (socket says 172.21.15.89:50704) 2024-04-02T09:15:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.493+0000 7fc96e000700 1 -- 172.21.15.89:0/687130330 learned_addr learned my addr 172.21.15.89:0/687130330 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.493+0000 7fc96ea00700 1 --2- 172.21.15.89:0/687130330 >> v2:172.21.15.89:3300/0 conn(0x7fc97015d9d0 0x7fc9700d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.493+0000 7fc96f400700 1 --2- 172.21.15.89:0/687130330 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9700d2540 0x7fc9700ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.493+0000 7fc96d600700 1 -- 172.21.15.89:0/687130330 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250180572 0 0) 0x7fc970161420 con 0x7fc9700d16a0 2024-04-02T09:15:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.493+0000 7fc96d600700 1 -- 172.21.15.89:0/687130330 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9540035b0 con 0x7fc9700d16a0 2024-04-02T09:15:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.493+0000 7fc96ea00700 1 -- 172.21.15.89:0/687130330 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9700d2540 msgr2=0x7fc9700ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.493+0000 7fc96ea00700 1 --2- 172.21.15.89:0/687130330 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9700d2540 0x7fc9700ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.493+0000 7fc96ea00700 1 -- 172.21.15.89:0/687130330 >> v1:172.21.15.89:6789/0 conn(0x7fc9700d16a0 legacy=0x7fc9700d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.493+0000 7fc96ea00700 1 -- 172.21.15.89:0/687130330 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9701619e0 con 0x7fc97015d9d0 2024-04-02T09:15:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.494+0000 7fc96ea00700 1 --2- 172.21.15.89:0/687130330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc97015d9d0 0x7fc9700d3c30 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7fc960009840 tx=0x7fc960009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6ed234d79b9ad1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.494+0000 7fc96d600700 1 -- 172.21.15.89:0/687130330 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc960011830 con 0x7fc97015d9d0 2024-04-02T09:15:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.494+0000 7fc96d600700 1 -- 172.21.15.89:0/687130330 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc960011990 con 0x7fc97015d9d0 2024-04-02T09:15:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.494+0000 7fc96d600700 1 -- 172.21.15.89:0/687130330 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc960011d20 con 0x7fc97015d9d0 2024-04-02T09:15:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.494+0000 7fc975800700 1 -- 172.21.15.89:0/687130330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc97015d9d0 msgr2=0x7fc9700d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.494+0000 7fc975800700 1 --2- 172.21.15.89:0/687130330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc97015d9d0 0x7fc9700d3c30 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7fc960009840 tx=0x7fc960009b50 comp rx=0 tx=0).stop 2024-04-02T09:15:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.495+0000 7fc975800700 1 -- 172.21.15.89:0/687130330 shutdown_connections 2024-04-02T09:15:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.495+0000 7fc975800700 1 --2- 172.21.15.89:0/687130330 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9700d2540 0x7fc9700ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.495+0000 7fc975800700 1 --2- 172.21.15.89:0/687130330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc97015d9d0 0x7fc9700d3c30 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.495+0000 7fc975800700 1 -- 172.21.15.89:0/687130330 >> 172.21.15.89:0/687130330 conn(0x7fc97008aff0 msgr2=0x7fc9700d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.495+0000 7fc975800700 1 -- 172.21.15.89:0/687130330 shutdown_connections 2024-04-02T09:15:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.495+0000 7fc975800700 1 -- 172.21.15.89:0/687130330 wait complete. 2024-04-02T09:15:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.496+0000 7fc975800700 1 Processor -- start 2024-04-02T09:15:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.496+0000 7fc975800700 1 -- start start 2024-04-02T09:15:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.497+0000 7fc975800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9700d16a0 0x7fc97015ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.497+0000 7fc975800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9700d2540 0x7fc9700caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.497+0000 7fc975800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc97015d9d0 0x7fc9700b0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.497+0000 7fc975800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9700cc700 con 0x7fc97015d9d0 2024-04-02T09:15:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.497+0000 7fc975800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9700cba80 con 0x7fc9700d16a0 2024-04-02T09:15:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.497+0000 7fc975800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9700cbd60 con 0x7fc9700d2540 2024-04-02T09:15:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.498+0000 7fc96f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc97015d9d0 0x7fc9700b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.498+0000 7fc96f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc97015d9d0 0x7fc9700b0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59076/0 (socket says 172.21.15.89:59076) 2024-04-02T09:15:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.498+0000 7fc96f400700 1 -- 172.21.15.89:0/1886390146 learned_addr learned my addr 172.21.15.89:0/1886390146 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.498+0000 7fc96ea00700 1 --2- 172.21.15.89:0/1886390146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9700d16a0 0x7fc97015ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.498+0000 7fc96e000700 1 --2- 172.21.15.89:0/1886390146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9700d2540 0x7fc9700caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.498+0000 7fc96f400700 1 -- 172.21.15.89:0/1886390146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9700d2540 msgr2=0x7fc9700caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.498+0000 7fc96f400700 1 --2- 172.21.15.89:0/1886390146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9700d2540 0x7fc9700caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.498+0000 7fc96f400700 1 -- 172.21.15.89:0/1886390146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9700d16a0 msgr2=0x7fc97015ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.498+0000 7fc96f400700 1 --2- 172.21.15.89:0/1886390146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9700d16a0 0x7fc97015ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.498+0000 7fc96f400700 1 -- 172.21.15.89:0/1886390146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9600094a0 con 0x7fc97015d9d0 2024-04-02T09:15:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.498+0000 7fc96f400700 1 --2- 172.21.15.89:0/1886390146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc97015d9d0 0x7fc9700b0800 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7fc96800efc0 tx=0x7fc96800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.499+0000 7fc966a00700 1 -- 172.21.15.89:0/1886390146 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc968009ea0 con 0x7fc97015d9d0 2024-04-02T09:15:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.499+0000 7fc966a00700 1 -- 172.21.15.89:0/1886390146 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96800f040 con 0x7fc97015d9d0 2024-04-02T09:15:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.499+0000 7fc966a00700 1 -- 172.21.15.89:0/1886390146 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc968014480 con 0x7fc97015d9d0 2024-04-02T09:15:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.499+0000 7fc975800700 1 -- 172.21.15.89:0/1886390146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9700cc040 con 0x7fc97015d9d0 2024-04-02T09:15:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.499+0000 7fc975800700 1 -- 172.21.15.89:0/1886390146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97015cf90 con 0x7fc97015d9d0 2024-04-02T09:15:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.500+0000 7fc975800700 1 -- 172.21.15.89:0/1886390146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9700d1ca0 con 0x7fc97015d9d0 2024-04-02T09:15:25.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.501+0000 7fc966a00700 1 -- 172.21.15.89:0/1886390146 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc9680145e0 con 0x7fc97015d9d0 2024-04-02T09:15:25.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.501+0000 7fc966a00700 1 --2- 172.21.15.89:0/1886390146 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc95006adb0 0x7fc95006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:25.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.502+0000 7fc966a00700 1 -- 172.21.15.89:0/1886390146 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fc968089ff0 con 0x7fc97015d9d0 2024-04-02T09:15:25.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.502+0000 7fc96ea00700 1 --2- 172.21.15.89:0/1886390146 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc95006adb0 0x7fc95006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:25.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.502+0000 7fc96ea00700 1 --2- 172.21.15.89:0/1886390146 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc95006adb0 0x7fc95006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc960000c00 tx=0x7fc96001c040 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:25.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.507+0000 7fc966a00700 1 -- 172.21.15.89:0/1886390146 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc968055020 con 0x7fc97015d9d0 2024-04-02T09:15:25.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:25.711+0000 7fc975800700 1 -- 172.21.15.89:0/1886390146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc9700cc840 con 0x7fc97015d9d0 2024-04-02T09:15:26.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.270+0000 7fc966a00700 1 -- 172.21.15.89:0/1886390146 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v277) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc9700cc840 con 0x7fc97015d9d0 2024-04-02T09:15:26.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.279+0000 7fc975800700 1 -- 172.21.15.89:0/1886390146 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc95006adb0 msgr2=0x7fc95006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:26.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.279+0000 7fc975800700 1 --2- 172.21.15.89:0/1886390146 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc95006adb0 0x7fc95006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc960000c00 tx=0x7fc96001c040 comp rx=0 tx=0).stop 2024-04-02T09:15:26.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.279+0000 7fc975800700 1 -- 172.21.15.89:0/1886390146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc97015d9d0 msgr2=0x7fc9700b0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:26.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.279+0000 7fc975800700 1 --2- 172.21.15.89:0/1886390146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc97015d9d0 0x7fc9700b0800 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7fc96800efc0 tx=0x7fc96800c3a0 comp rx=0 tx=0).stop 2024-04-02T09:15:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.280+0000 7fc975800700 1 -- 172.21.15.89:0/1886390146 shutdown_connections 2024-04-02T09:15:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.280+0000 7fc975800700 1 --2- 172.21.15.89:0/1886390146 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fc95006adb0 0x7fc95006d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.280+0000 7fc975800700 1 --2- 172.21.15.89:0/1886390146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9700d16a0 0x7fc97015ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.280+0000 7fc975800700 1 --2- 172.21.15.89:0/1886390146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc97015d9d0 0x7fc9700b0800 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.280+0000 7fc975800700 1 --2- 172.21.15.89:0/1886390146 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9700d2540 0x7fc9700caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.280+0000 7fc975800700 1 -- 172.21.15.89:0/1886390146 >> 172.21.15.89:0/1886390146 conn(0x7fc97008aff0 msgr2=0x7fc9700d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:26.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.282+0000 7fc975800700 1 -- 172.21.15.89:0/1886390146 shutdown_connections 2024-04-02T09:15:26.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.284+0000 7fc975800700 1 -- 172.21.15.89:0/1886390146 wait complete. 2024-04-02T09:15:26.286 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-04-02T09:15:26.308 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T09:15:26.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.506+0000 7f8180c00700 1 Processor -- start 2024-04-02T09:15:26.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.507+0000 7f8180c00700 1 -- start start 2024-04-02T09:15:26.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.508+0000 7f8180c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f817c1611b0 0x7f817c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:26.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.508+0000 7f8180c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f817c0cee10 0x7f817c161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:26.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.508+0000 7f8180c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c0ce270 con 0x7f817c15f4d0 2024-04-02T09:15:26.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.508+0000 7f8180c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f817c15b600 con 0x7f817c0cee10 2024-04-02T09:15:26.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.508+0000 7f8180c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817c0d2760 con 0x7f817c1611b0 2024-04-02T09:15:26.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.509+0000 7f8173e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f817c15f4d0 0x7f817c0ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50720/0 (socket says 172.21.15.89:50720) 2024-04-02T09:15:26.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.509+0000 7f8173e00700 1 -- 172.21.15.89:0/998996058 learned_addr learned my addr 172.21.15.89:0/998996058 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:26.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.509+0000 7f8179e00700 1 --2- 172.21.15.89:0/998996058 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f817c1611b0 0x7f817c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:26.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.509+0000 7f817a800700 1 --2- 172.21.15.89:0/998996058 >> v2:172.21.15.113:3300/0 conn(0x7f817c0cee10 0x7f817c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:26.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.509+0000 7f8179400700 1 -- 172.21.15.89:0/998996058 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089645519 0 0) 0x7f817c0ce270 con 0x7f817c15f4d0 2024-04-02T09:15:26.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.509+0000 7f8179400700 1 -- 172.21.15.89:0/998996058 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81600035b0 con 0x7f817c15f4d0 2024-04-02T09:15:26.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.509+0000 7f8179e00700 1 -- 172.21.15.89:0/998996058 >> v2:172.21.15.113:3300/0 conn(0x7f817c0cee10 msgr2=0x7f817c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:26.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.509+0000 7f8179e00700 1 --2- 172.21.15.89:0/998996058 >> v2:172.21.15.113:3300/0 conn(0x7f817c0cee10 0x7f817c161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:26.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.509+0000 7f8179e00700 1 -- 172.21.15.89:0/998996058 >> v1:172.21.15.89:6789/0 conn(0x7f817c15f4d0 legacy=0x7f817c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:26.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.509+0000 7f8179e00700 1 -- 172.21.15.89:0/998996058 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c0d2a40 con 0x7f817c1611b0 2024-04-02T09:15:26.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.510+0000 7f8179e00700 1 --2- 172.21.15.89:0/998996058 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f817c1611b0 0x7f817c15ef90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f816c009840 tx=0x7f816c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df1c78e9101de558 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:26.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.510+0000 7f8179400700 1 -- 172.21.15.89:0/998996058 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c011830 con 0x7f817c1611b0 2024-04-02T09:15:26.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.510+0000 7f8179400700 1 -- 172.21.15.89:0/998996058 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816c011990 con 0x7f817c1611b0 2024-04-02T09:15:26.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.510+0000 7f8179400700 1 -- 172.21.15.89:0/998996058 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c011d20 con 0x7f817c1611b0 2024-04-02T09:15:26.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.510+0000 7f8180c00700 1 -- 172.21.15.89:0/998996058 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f817c1611b0 msgr2=0x7f817c15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:26.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.510+0000 7f8180c00700 1 --2- 172.21.15.89:0/998996058 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f817c1611b0 0x7f817c15ef90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f816c009840 tx=0x7f816c009b50 comp rx=0 tx=0).stop 2024-04-02T09:15:26.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.510+0000 7f8180c00700 1 -- 172.21.15.89:0/998996058 shutdown_connections 2024-04-02T09:15:26.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.510+0000 7f8180c00700 1 --2- 172.21.15.89:0/998996058 >> v2:172.21.15.113:3300/0 conn(0x7f817c0cee10 0x7f817c161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:26.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.510+0000 7f8180c00700 1 --2- 172.21.15.89:0/998996058 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f817c1611b0 0x7f817c15ef90 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:26.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.510+0000 7f8180c00700 1 -- 172.21.15.89:0/998996058 >> 172.21.15.89:0/998996058 conn(0x7f817c08aff0 msgr2=0x7f817c0d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:26.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.510+0000 7f8180c00700 1 -- 172.21.15.89:0/998996058 shutdown_connections 2024-04-02T09:15:26.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.511+0000 7f8180c00700 1 -- 172.21.15.89:0/998996058 wait complete. 2024-04-02T09:15:26.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.512+0000 7f8180c00700 1 Processor -- start 2024-04-02T09:15:26.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.512+0000 7f8180c00700 1 -- start start 2024-04-02T09:15:26.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.513+0000 7f8180c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f817c0cee10 0x7f817c0b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:26.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.513+0000 7f8180c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f817c15f4d0 0x7f817c0b77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:26.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.513+0000 7f8180c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f817c1611b0 0x7f817c158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:26.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.513+0000 7f8180c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c0b8ee0 con 0x7f817c0cee10 2024-04-02T09:15:26.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.513+0000 7f8180c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c0b7430 con 0x7f817c1611b0 2024-04-02T09:15:26.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.513+0000 7f8180c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817c0b8260 con 0x7f817c15f4d0 2024-04-02T09:15:26.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.513+0000 7f8179e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f817c0cee10 0x7f817c0b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:26.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.513+0000 7f8179e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f817c0cee10 0x7f817c0b6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59092/0 (socket says 172.21.15.89:59092) 2024-04-02T09:15:26.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.513+0000 7f8179e00700 1 -- 172.21.15.89:0/2020293994 learned_addr learned my addr 172.21.15.89:0/2020293994 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:26.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.514+0000 7f817a800700 1 --2- 172.21.15.89:0/2020293994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f817c1611b0 0x7f817c158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:26.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.514+0000 7f8173e00700 1 --2- 172.21.15.89:0/2020293994 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f817c15f4d0 0x7f817c0b77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:26.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.514+0000 7f8179e00700 1 -- 172.21.15.89:0/2020293994 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f817c15f4d0 msgr2=0x7f817c0b77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:26.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.514+0000 7f8179e00700 1 --2- 172.21.15.89:0/2020293994 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f817c15f4d0 0x7f817c0b77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:26.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.514+0000 7f8179e00700 1 -- 172.21.15.89:0/2020293994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f817c1611b0 msgr2=0x7f817c158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:26.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.514+0000 7f8179e00700 1 --2- 172.21.15.89:0/2020293994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f817c1611b0 0x7f817c158bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:26.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.514+0000 7f8179e00700 1 -- 172.21.15.89:0/2020293994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c0094a0 con 0x7f817c0cee10 2024-04-02T09:15:26.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.514+0000 7f8179e00700 1 --2- 172.21.15.89:0/2020293994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f817c0cee10 0x7f817c0b6150 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7f816c009fd0 tx=0x7f816c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:26.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.514+0000 7f8172000700 1 -- 172.21.15.89:0/2020293994 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c0036a0 con 0x7f817c0cee10 2024-04-02T09:15:26.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.514+0000 7f8172000700 1 -- 172.21.15.89:0/2020293994 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816c01e070 con 0x7f817c0cee10 2024-04-02T09:15:26.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.514+0000 7f8180c00700 1 -- 172.21.15.89:0/2020293994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817c0b8540 con 0x7f817c0cee10 2024-04-02T09:15:26.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.514+0000 7f8172000700 1 -- 172.21.15.89:0/2020293994 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c003a60 con 0x7f817c0cee10 2024-04-02T09:15:26.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.515+0000 7f8180c00700 1 -- 172.21.15.89:0/2020293994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f817c0b8b00 con 0x7f817c0cee10 2024-04-02T09:15:26.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.515+0000 7f8180c00700 1 -- 172.21.15.89:0/2020293994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817c067720 con 0x7f817c0cee10 2024-04-02T09:15:26.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.517+0000 7f8172000700 1 -- 172.21.15.89:0/2020293994 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f816c003bc0 con 0x7f817c0cee10 2024-04-02T09:15:26.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.517+0000 7f8172000700 1 --2- 172.21.15.89:0/2020293994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f815806adb0 0x7f815806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:26.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.518+0000 7f8172000700 1 -- 172.21.15.89:0/2020293994 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(277..277 src has 1..277) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f816c08a540 con 0x7f817c0cee10 2024-04-02T09:15:26.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.518+0000 7f8173e00700 1 --2- 172.21.15.89:0/2020293994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f815806adb0 0x7f815806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:26.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.518+0000 7f8173e00700 1 --2- 172.21.15.89:0/2020293994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f815806adb0 0x7f815806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f817c1f6660 tx=0x7f8164009040 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.523+0000 7f8172000700 1 -- 172.21.15.89:0/2020293994 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f816c0554a0 con 0x7f817c0cee10 2024-04-02T09:15:26.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.770+0000 7f8180c00700 1 -- 172.21.15.89:0/2020293994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f817c0b5d00 con 0x7f817c0cee10 2024-04-02T09:15:26.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.778+0000 7f8172000700 1 -- 172.21.15.89:0/2020293994 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 36 and data pool 37 v140) v1 ==== 174+0+0 (secure 0 0 0) 0x7f816c058ac0 con 0x7f817c0cee10 2024-04-02T09:15:26.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.783+0000 7f8180c00700 1 -- 172.21.15.89:0/2020293994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f815806adb0 msgr2=0x7f815806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:26.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.783+0000 7f8180c00700 1 --2- 172.21.15.89:0/2020293994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f815806adb0 0x7f815806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f817c1f6660 tx=0x7f8164009040 comp rx=0 tx=0).stop 2024-04-02T09:15:26.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.783+0000 7f8180c00700 1 -- 172.21.15.89:0/2020293994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f817c0cee10 msgr2=0x7f817c0b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:26.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.783+0000 7f8180c00700 1 --2- 172.21.15.89:0/2020293994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f817c0cee10 0x7f817c0b6150 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7f816c009fd0 tx=0x7f816c004750 comp rx=0 tx=0).stop 2024-04-02T09:15:26.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.784+0000 7f8180c00700 1 -- 172.21.15.89:0/2020293994 shutdown_connections 2024-04-02T09:15:26.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.784+0000 7f8180c00700 1 --2- 172.21.15.89:0/2020293994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f815806adb0 0x7f815806d270 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f817c1f6660 tx=0x7f8164009040 comp rx=0 tx=0).stop 2024-04-02T09:15:26.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.784+0000 7f8180c00700 1 --2- 172.21.15.89:0/2020293994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f817c1611b0 0x7f817c158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:26.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.784+0000 7f8180c00700 1 --2- 172.21.15.89:0/2020293994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f817c0cee10 0x7f817c0b6150 unknown :-1 s=CLOSED pgs=2967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:26.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.784+0000 7f8180c00700 1 --2- 172.21.15.89:0/2020293994 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f817c15f4d0 0x7f817c0b77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:26.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.784+0000 7f8180c00700 1 -- 172.21.15.89:0/2020293994 >> 172.21.15.89:0/2020293994 conn(0x7f817c08aff0 msgr2=0x7f817c15da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:26.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.785+0000 7f8180c00700 1 -- 172.21.15.89:0/2020293994 shutdown_connections 2024-04-02T09:15:26.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:26.787+0000 7f8180c00700 1 -- 172.21.15.89:0/2020293994 wait complete. 2024-04-02T09:15:26.795 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 36 and data pool 37 2024-04-02T09:15:26.817 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:15:27.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.010+0000 7f86f3400700 1 Processor -- start 2024-04-02T09:15:27.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.010+0000 7f86f3400700 1 -- start start 2024-04-02T09:15:27.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.011+0000 7f86f3400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f86ec1619d0 0x7f86ec1685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.011+0000 7f86f3400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86ec0d30d0 con 0x7f86ec15d640 2024-04-02T09:15:27.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.011+0000 7f86f3400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86ec0b6840 con 0x7f86ec0ceec0 2024-04-02T09:15:27.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.011+0000 7f86f3400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86ec0d2b30 con 0x7f86ec1619d0 2024-04-02T09:15:27.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.011+0000 7f86f0a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f86ec15d640 0x7f86ec0d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50722/0 (socket says 172.21.15.89:50722) 2024-04-02T09:15:27.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.011+0000 7f86f0a00700 1 -- 172.21.15.89:0/811396776 learned_addr learned my addr 172.21.15.89:0/811396776 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:27.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86f1400700 1 --2- 172.21.15.89:0/811396776 >> v2:172.21.15.113:3300/0 conn(0x7f86ec1619d0 0x7f86ec1685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 -- 172.21.15.89:0/811396776 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994108086 0 0) 0x7f86ec0d30d0 con 0x7f86ec15d640 2024-04-02T09:15:27.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 -- 172.21.15.89:0/811396776 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d40035b0 con 0x7f86ec15d640 2024-04-02T09:15:27.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 -- 172.21.15.89:0/811396776 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1866456705 0 0) 0x7f86d40035b0 con 0x7f86ec15d640 2024-04-02T09:15:27.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 -- 172.21.15.89:0/811396776 >> v2:172.21.15.113:3300/0 conn(0x7f86ec1619d0 msgr2=0x7f86ec1685f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:27.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 --2- 172.21.15.89:0/811396776 >> v2:172.21.15.113:3300/0 conn(0x7f86ec1619d0 0x7f86ec1685f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 -- 172.21.15.89:0/811396776 >> v1:172.21.15.113:6789/0 conn(0x7f86ec0ceec0 legacy=0x7f86ec1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 -- 172.21.15.89:0/811396776 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ec0d2df0 con 0x7f86ec15d640 2024-04-02T09:15:27.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 -- 172.21.15.89:0/811396776 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f86e0001670 con 0x7f86ec15d640 2024-04-02T09:15:27.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 -- 172.21.15.89:0/811396776 >> v1:172.21.15.89:6789/0 conn(0x7f86ec15d640 legacy=0x7f86ec0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 --2- 172.21.15.89:0/811396776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d4003a00 0x7f86d4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 --2- 172.21.15.89:0/811396776 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d4004850 0x7f86d4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 --2- 172.21.15.89:0/811396776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86d4005210 0x7f86d400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 -- 172.21.15.89:0/811396776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec0d2b30 con 0x7f86d4003a00 2024-04-02T09:15:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 -- 172.21.15.89:0/811396776 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d400c1b0 con 0x7f86d4005210 2024-04-02T09:15:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86eb400700 1 -- 172.21.15.89:0/811396776 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d400c370 con 0x7f86d4004850 2024-04-02T09:15:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.012+0000 7f86f0a00700 1 --2- 172.21.15.89:0/811396776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d4003a00 0x7f86d4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.013+0000 7f86ebe00700 1 --2- 172.21.15.89:0/811396776 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d4004850 0x7f86d4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.013+0000 7f86f1400700 1 --2- 172.21.15.89:0/811396776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86d4005210 0x7f86d400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.013+0000 7f86f0a00700 1 -- 172.21.15.89:0/811396776 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d4004850 msgr2=0x7f86d4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.013+0000 7f86f0a00700 1 --2- 172.21.15.89:0/811396776 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d4004850 0x7f86d4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.013+0000 7f86f0a00700 1 -- 172.21.15.89:0/811396776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86d4005210 msgr2=0x7f86d400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:27.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.013+0000 7f86f0a00700 1 --2- 172.21.15.89:0/811396776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86d4005210 0x7f86d400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.013+0000 7f86f0a00700 1 -- 172.21.15.89:0/811396776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d40035b0 con 0x7f86d4003a00 2024-04-02T09:15:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.013+0000 7f86f1400700 1 --2- 172.21.15.89:0/811396776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86d4005210 0x7f86d400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.013+0000 7f86f1400700 1 -- 172.21.15.89:0/811396776 reap_dead start 2024-04-02T09:15:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.013+0000 7f86f0a00700 1 --2- 172.21.15.89:0/811396776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d4003a00 0x7f86d4003e20 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f86e000c240 tx=0x7f86d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766d14940d117080 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.013+0000 7f86eb400700 1 -- 172.21.15.89:0/811396776 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e0007c10 con 0x7f86d4003a00 2024-04-02T09:15:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.013+0000 7f86eb400700 1 -- 172.21.15.89:0/811396776 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e000ce00 con 0x7f86d4003a00 2024-04-02T09:15:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.013+0000 7f86eb400700 1 -- 172.21.15.89:0/811396776 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e0005ba0 con 0x7f86d4003a00 2024-04-02T09:15:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.014+0000 7f86f3400700 1 -- 172.21.15.89:0/811396776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d4003a00 msgr2=0x7f86d4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.014+0000 7f86f3400700 1 --2- 172.21.15.89:0/811396776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d4003a00 0x7f86d4003e20 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f86e000c240 tx=0x7f86d4005880 comp rx=0 tx=0).stop 2024-04-02T09:15:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.014+0000 7f86f3400700 1 -- 172.21.15.89:0/811396776 shutdown_connections 2024-04-02T09:15:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.014+0000 7f86f3400700 1 --2- 172.21.15.89:0/811396776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86d4003a00 0x7f86d4003e20 unknown :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.014+0000 7f86f3400700 1 -- 172.21.15.89:0/811396776 >> 172.21.15.89:0/811396776 conn(0x7f86ec08aff0 msgr2=0x7f86ec0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.014+0000 7f86f3400700 1 -- 172.21.15.89:0/811396776 shutdown_connections 2024-04-02T09:15:27.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.014+0000 7f86f3400700 1 -- 172.21.15.89:0/811396776 wait complete. 2024-04-02T09:15:27.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.016+0000 7f86f3400700 1 Processor -- start 2024-04-02T09:15:27.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.016+0000 7f86f3400700 1 -- start start 2024-04-02T09:15:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86f3400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d4003a00 0x7f86ec1440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86f3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ec141530 0x7f86ec1587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86f3400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86ec143380 0x7f86ec155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86f3400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec158d20 con 0x7f86ec141530 2024-04-02T09:15:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86f3400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec158fa0 con 0x7f86ec143380 2024-04-02T09:15:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86f3400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec15ed70 con 0x7f86d4003a00 2024-04-02T09:15:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86ebe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ec141530 0x7f86ec1587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86ebe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ec141530 0x7f86ec1587e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59124/0 (socket says 172.21.15.89:59124) 2024-04-02T09:15:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86ebe00700 1 -- 172.21.15.89:0/3343725036 learned_addr learned my addr 172.21.15.89:0/3343725036 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86f0a00700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d4003a00 0x7f86ec1440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86f1400700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86ec143380 0x7f86ec155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86ebe00700 1 -- 172.21.15.89:0/3343725036 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d4003a00 msgr2=0x7f86ec1440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86ebe00700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d4003a00 0x7f86ec1440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86ebe00700 1 -- 172.21.15.89:0/3343725036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86ec143380 msgr2=0x7f86ec155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86ebe00700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86ec143380 0x7f86ec155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86ebe00700 1 -- 172.21.15.89:0/3343725036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ec15f050 con 0x7f86ec141530 2024-04-02T09:15:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.017+0000 7f86f1400700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86ec143380 0x7f86ec155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.018+0000 7f86ebe00700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ec141530 0x7f86ec1587e0 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7f86d800e550 tx=0x7f86d800e860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.018+0000 7f86e8c00700 1 -- 172.21.15.89:0/3343725036 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d80052e0 con 0x7f86ec141530 2024-04-02T09:15:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.018+0000 7f86e8c00700 1 -- 172.21.15.89:0/3343725036 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d8005460 con 0x7f86ec141530 2024-04-02T09:15:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.018+0000 7f86e8c00700 1 -- 172.21.15.89:0/3343725036 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d80038c0 con 0x7f86ec141530 2024-04-02T09:15:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.018+0000 7f86f3400700 1 -- 172.21.15.89:0/3343725036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ec15f330 con 0x7f86ec141530 2024-04-02T09:15:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.018+0000 7f86f3400700 1 -- 172.21.15.89:0/3343725036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ec1419b0 con 0x7f86ec141530 2024-04-02T09:15:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.019+0000 7f86f3400700 1 -- 172.21.15.89:0/3343725036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ec15f980 con 0x7f86ec141530 2024-04-02T09:15:27.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.020+0000 7f86e8c00700 1 -- 172.21.15.89:0/3343725036 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f86d8003a20 con 0x7f86ec141530 2024-04-02T09:15:27.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.021+0000 7f86e8c00700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f86cc06adb0 0x7f86cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.021+0000 7f86e8c00700 1 -- 172.21.15.89:0/3343725036 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f86d8088d30 con 0x7f86ec141530 2024-04-02T09:15:27.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.021+0000 7f86f0a00700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f86cc06adb0 0x7f86cc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.022+0000 7f86f0a00700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f86cc06adb0 0x7f86cc06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f86e0000c00 tx=0x7f86e0007000 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:27.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.025+0000 7f86e8c00700 1 -- 172.21.15.89:0/3343725036 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f86d8053c50 con 0x7f86ec141530 2024-04-02T09:15:27.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.226+0000 7f86f3400700 1 -- 172.21.15.89:0/3343725036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86ec067720 con 0x7f86ec141530 2024-04-02T09:15:27.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.227+0000 7f86e8c00700 1 -- 172.21.15.89:0/3343725036 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v278) v1 ==== 74+0+16749 (secure 0 0 0) 0x7f86d8019090 con 0x7f86ec141530 2024-04-02T09:15:27.226 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:27.226 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":278,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:15:26.771880+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:15:24.345253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-04-02T09:15:25.756381+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:15:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.230+0000 7f86f3400700 1 -- 172.21.15.89:0/3343725036 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f86cc06adb0 msgr2=0x7f86cc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.230+0000 7f86f3400700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f86cc06adb0 0x7f86cc06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f86e0000c00 tx=0x7f86e0007000 comp rx=0 tx=0).stop 2024-04-02T09:15:27.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.230+0000 7f86f3400700 1 -- 172.21.15.89:0/3343725036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ec141530 msgr2=0x7f86ec1587e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.230+0000 7f86f3400700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ec141530 0x7f86ec1587e0 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7f86d800e550 tx=0x7f86d800e860 comp rx=0 tx=0).stop 2024-04-02T09:15:27.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.230+0000 7f86f3400700 1 -- 172.21.15.89:0/3343725036 shutdown_connections 2024-04-02T09:15:27.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.230+0000 7f86f3400700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f86cc06adb0 0x7f86cc06d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.230+0000 7f86f3400700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86ec143380 0x7f86ec155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.230+0000 7f86f3400700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f86d4003a00 0x7f86ec1440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.230+0000 7f86f3400700 1 --2- 172.21.15.89:0/3343725036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86ec141530 0x7f86ec1587e0 unknown :-1 s=CLOSED pgs=2970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.230+0000 7f86f3400700 1 -- 172.21.15.89:0/3343725036 >> 172.21.15.89:0/3343725036 conn(0x7f86ec08aff0 msgr2=0x7f86ec0cffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:27.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.230+0000 7f86f3400700 1 -- 172.21.15.89:0/3343725036 shutdown_connections 2024-04-02T09:15:27.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.230+0000 7f86f3400700 1 -- 172.21.15.89:0/3343725036 wait complete. 2024-04-02T09:15:27.247 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:15:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.440+0000 7f7dbee00700 1 Processor -- start 2024-04-02T09:15:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.440+0000 7f7dbee00700 1 -- start start 2024-04-02T09:15:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.441+0000 7f7dbee00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7db80b3af0 0x7f7db815a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.442+0000 7f7dbee00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db80b2180 con 0x7f7db80709c0 2024-04-02T09:15:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.442+0000 7f7dbee00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db815b100 con 0x7f7db8072580 2024-04-02T09:15:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.442+0000 7f7dbee00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db815b3e0 con 0x7f7db80b3af0 2024-04-02T09:15:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.442+0000 7f7dbce00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7db80709c0 0x7f7db806e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50724/0 (socket says 172.21.15.89:50724) 2024-04-02T09:15:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.442+0000 7f7dbce00700 1 -- 172.21.15.89:0/2019572500 learned_addr learned my addr 172.21.15.89:0/2019572500 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:27.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.442+0000 7f7db7e00700 1 --2- 172.21.15.89:0/2019572500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7db80b3af0 0x7f7db815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.442+0000 7f7db6a00700 1 -- 172.21.15.89:0/2019572500 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628347756 0 0) 0x7f7db80b2180 con 0x7f7db80709c0 2024-04-02T09:15:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.442+0000 7f7db6a00700 1 -- 172.21.15.89:0/2019572500 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d9c0035b0 con 0x7f7db80709c0 2024-04-02T09:15:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.442+0000 7f7db6a00700 1 -- 172.21.15.89:0/2019572500 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034360895 0 0) 0x7f7db815b100 con 0x7f7db8072580 2024-04-02T09:15:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.442+0000 7f7db6a00700 1 -- 172.21.15.89:0/2019572500 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7db80b2180 con 0x7f7db8072580 2024-04-02T09:15:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.443+0000 7f7db7e00700 1 -- 172.21.15.89:0/2019572500 >> v1:172.21.15.113:6789/0 conn(0x7f7db8072580 legacy=0x7f7db80b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.443+0000 7f7db7e00700 1 -- 172.21.15.89:0/2019572500 >> v1:172.21.15.89:6789/0 conn(0x7f7db80709c0 legacy=0x7f7db806e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.443+0000 7f7db7e00700 1 -- 172.21.15.89:0/2019572500 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db80b23f0 con 0x7f7db80b3af0 2024-04-02T09:15:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.443+0000 7f7db7e00700 1 --2- 172.21.15.89:0/2019572500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7db80b3af0 0x7f7db815a860 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7da8009870 tx=0x7f7da8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc86652fe4a43228 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.443+0000 7f7db6a00700 1 -- 172.21.15.89:0/2019572500 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da8011830 con 0x7f7db80b3af0 2024-04-02T09:15:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.443+0000 7f7db6a00700 1 -- 172.21.15.89:0/2019572500 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da8011990 con 0x7f7db80b3af0 2024-04-02T09:15:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.443+0000 7f7db6a00700 1 -- 172.21.15.89:0/2019572500 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da8011d20 con 0x7f7db80b3af0 2024-04-02T09:15:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.444+0000 7f7dbee00700 1 -- 172.21.15.89:0/2019572500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7db80b3af0 msgr2=0x7f7db815a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.444+0000 7f7dbee00700 1 --2- 172.21.15.89:0/2019572500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7db80b3af0 0x7f7db815a860 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7da8009870 tx=0x7f7da8009b80 comp rx=0 tx=0).stop 2024-04-02T09:15:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.444+0000 7f7dbee00700 1 -- 172.21.15.89:0/2019572500 shutdown_connections 2024-04-02T09:15:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.444+0000 7f7dbee00700 1 --2- 172.21.15.89:0/2019572500 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7db80b3af0 0x7f7db815a860 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.444+0000 7f7dbee00700 1 -- 172.21.15.89:0/2019572500 >> 172.21.15.89:0/2019572500 conn(0x7f7db80cf190 msgr2=0x7f7db80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.444+0000 7f7dbee00700 1 -- 172.21.15.89:0/2019572500 shutdown_connections 2024-04-02T09:15:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.444+0000 7f7dbee00700 1 -- 172.21.15.89:0/2019572500 wait complete. 2024-04-02T09:15:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.446+0000 7f7dbee00700 1 Processor -- start 2024-04-02T09:15:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.446+0000 7f7dbee00700 1 -- start start 2024-04-02T09:15:27.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7dbee00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7db80709c0 0x7f7db8156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7dbee00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7db8072580 0x7f7db81f5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7dbee00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7db80b3af0 0x7f7db8159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7dbee00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db81ed610 con 0x7f7db80709c0 2024-04-02T09:15:27.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7dbee00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db8152730 con 0x7f7db8072580 2024-04-02T09:15:27.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7dbee00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db8152a10 con 0x7f7db80b3af0 2024-04-02T09:15:27.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7db7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7db80709c0 0x7f7db8156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7db7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7db80709c0 0x7f7db8156e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59130/0 (socket says 172.21.15.89:59130) 2024-04-02T09:15:27.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7db7e00700 1 -- 172.21.15.89:0/2249527992 learned_addr learned my addr 172.21.15.89:0/2249527992 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:27.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7db7400700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7db8072580 0x7f7db81f5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7dbce00700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7db80b3af0 0x7f7db8159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7db7e00700 1 -- 172.21.15.89:0/2249527992 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7db80b3af0 msgr2=0x7f7db8159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7db7e00700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7db80b3af0 0x7f7db8159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7db7e00700 1 -- 172.21.15.89:0/2249527992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7db8072580 msgr2=0x7f7db81f5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7db7e00700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7db8072580 0x7f7db81f5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7db7e00700 1 -- 172.21.15.89:0/2249527992 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da80094a0 con 0x7f7db80709c0 2024-04-02T09:15:27.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7dbce00700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7db80b3af0 0x7f7db8159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:27.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.447+0000 7f7db7400700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7db8072580 0x7f7db81f5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:27.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.448+0000 7f7db7e00700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7db80709c0 0x7f7db8156e50 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7f7da80097c0 tx=0x7f7da8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:27.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.448+0000 7f7dafe00700 1 -- 172.21.15.89:0/2249527992 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da801b870 con 0x7f7db80709c0 2024-04-02T09:15:27.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.448+0000 7f7dafe00700 1 -- 172.21.15.89:0/2249527992 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da801b9d0 con 0x7f7db80709c0 2024-04-02T09:15:27.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.448+0000 7f7dafe00700 1 -- 172.21.15.89:0/2249527992 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da801bd80 con 0x7f7db80709c0 2024-04-02T09:15:27.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.448+0000 7f7dbee00700 1 -- 172.21.15.89:0/2249527992 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db8152cf0 con 0x7f7db80709c0 2024-04-02T09:15:27.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.448+0000 7f7dbee00700 1 -- 172.21.15.89:0/2249527992 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db8157390 con 0x7f7db80709c0 2024-04-02T09:15:27.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.449+0000 7f7dbee00700 1 -- 172.21.15.89:0/2249527992 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db8067720 con 0x7f7db80709c0 2024-04-02T09:15:27.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.450+0000 7f7dafe00700 1 -- 172.21.15.89:0/2249527992 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7da8023070 con 0x7f7db80709c0 2024-04-02T09:15:27.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.451+0000 7f7dafe00700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7d9806adb0 0x7f7d9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.451+0000 7f7dafe00700 1 -- 172.21.15.89:0/2249527992 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f7da808a3c0 con 0x7f7db80709c0 2024-04-02T09:15:27.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.451+0000 7f7db7400700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7d9806adb0 0x7f7d9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.452+0000 7f7db7400700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7d9806adb0 0x7f7d9806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7db815a440 tx=0x7f7da4009040 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.455+0000 7f7dafe00700 1 -- 172.21.15.89:0/2249527992 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7da8055480 con 0x7f7db80709c0 2024-04-02T09:15:27.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-04-02T09:15:27.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.658+0000 7f7dbee00700 1 -- 172.21.15.89:0/2249527992 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7db81f42a0 con 0x7f7db80709c0 2024-04-02T09:15:27.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.659+0000 7f7dafe00700 1 -- 172.21.15.89:0/2249527992 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v278) v1 ==== 74+0+16749 (secure 0 0 0) 0x7f7da8058aa0 con 0x7f7db80709c0 2024-04-02T09:15:27.658 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:27.659 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":278,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:15:26.771880+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:15:24.345253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-04-02T09:15:25.756381+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:15:27.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.662+0000 7f7dbee00700 1 -- 172.21.15.89:0/2249527992 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7d9806adb0 msgr2=0x7f7d9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.662+0000 7f7dbee00700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7d9806adb0 0x7f7d9806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7db815a440 tx=0x7f7da4009040 comp rx=0 tx=0).stop 2024-04-02T09:15:27.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.662+0000 7f7dbee00700 1 -- 172.21.15.89:0/2249527992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7db80709c0 msgr2=0x7f7db8156e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.662+0000 7f7dbee00700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7db80709c0 0x7f7db8156e50 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7f7da80097c0 tx=0x7f7da8004750 comp rx=0 tx=0).stop 2024-04-02T09:15:27.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.662+0000 7f7dbee00700 1 -- 172.21.15.89:0/2249527992 shutdown_connections 2024-04-02T09:15:27.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.662+0000 7f7dbee00700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7db80b3af0 0x7f7db8159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.662+0000 7f7dbee00700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7db80709c0 0x7f7db8156e50 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.662+0000 7f7dbee00700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7d9806adb0 0x7f7d9806d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.662+0000 7f7dbee00700 1 --2- 172.21.15.89:0/2249527992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7db8072580 0x7f7db81f5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.662+0000 7f7dbee00700 1 -- 172.21.15.89:0/2249527992 >> 172.21.15.89:0/2249527992 conn(0x7f7db80cf190 msgr2=0x7f7db815dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:27.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.662+0000 7f7dbee00700 1 -- 172.21.15.89:0/2249527992 shutdown_connections 2024-04-02T09:15:27.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.662+0000 7f7dbee00700 1 -- 172.21.15.89:0/2249527992 wait complete. 2024-04-02T09:15:27.679 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T09:15:27.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.878+0000 7f5edb800700 1 Processor -- start 2024-04-02T09:15:27.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.878+0000 7f5edb800700 1 -- start start 2024-04-02T09:15:27.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.879+0000 7f5edb800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ed40d2540 0x7f5ed41687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.879+0000 7f5edb800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed4161420 con 0x7f5ed40d16a0 2024-04-02T09:15:27.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.879+0000 7f5edb800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed4161700 con 0x7f5ed415d9d0 2024-04-02T09:15:27.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.879+0000 7f5edb800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed40d4340 con 0x7f5ed40d2540 2024-04-02T09:15:27.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.879+0000 7f5ed3e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5ed40d16a0 0x7f5ed40d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50732/0 (socket says 172.21.15.89:50732) 2024-04-02T09:15:27.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.879+0000 7f5ed3e00700 1 -- 172.21.15.89:0/2478101373 learned_addr learned my addr 172.21.15.89:0/2478101373 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:27.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed9800700 1 --2- 172.21.15.89:0/2478101373 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ed40d2540 0x7f5ed41687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630792371 0 0) 0x7f5ed4161420 con 0x7f5ed40d16a0 2024-04-02T09:15:27.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ebc0035b0 con 0x7f5ed40d16a0 2024-04-02T09:15:27.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496401265 0 0) 0x7f5ed4161700 con 0x7f5ed415d9d0 2024-04-02T09:15:27.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed4161420 con 0x7f5ed415d9d0 2024-04-02T09:15:27.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 710151933 0 0) 0x7f5ebc0035b0 con 0x7f5ed40d16a0 2024-04-02T09:15:27.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ed40d2540 msgr2=0x7f5ed41687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:27.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 --2- 172.21.15.89:0/2478101373 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ed40d2540 0x7f5ed41687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 >> v1:172.21.15.113:6789/0 conn(0x7f5ed415d9d0 legacy=0x7f5ed40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed41619e0 con 0x7f5ed40d16a0 2024-04-02T09:15:27.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f5ec00017f0 con 0x7f5ed40d16a0 2024-04-02T09:15:27.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 >> v1:172.21.15.89:6789/0 conn(0x7f5ed40d16a0 legacy=0x7f5ed40d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 --2- 172.21.15.89:0/2478101373 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ed40d2540 0x7f5ebc003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 --2- 172.21.15.89:0/2478101373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ebc004170 0x7f5ebc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 --2- 172.21.15.89:0/2478101373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ebc004d90 0x7f5ebc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc00bb20 con 0x7f5ebc004170 2024-04-02T09:15:27.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc00bc90 con 0x7f5ebc004d90 2024-04-02T09:15:27.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc00be50 con 0x7f5ed40d2540 2024-04-02T09:15:27.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.880+0000 7f5ed3e00700 1 --2- 172.21.15.89:0/2478101373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ebc004170 0x7f5ebc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.881+0000 7f5ed8e00700 1 --2- 172.21.15.89:0/2478101373 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ed40d2540 0x7f5ebc003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.881+0000 7f5ed9800700 1 --2- 172.21.15.89:0/2478101373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ebc004d90 0x7f5ebc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.881+0000 7f5ed3e00700 1 -- 172.21.15.89:0/2478101373 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ed40d2540 msgr2=0x7f5ebc003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:27.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.881+0000 7f5ed3e00700 1 --2- 172.21.15.89:0/2478101373 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ed40d2540 0x7f5ebc003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.881+0000 7f5ed3e00700 1 -- 172.21.15.89:0/2478101373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ebc004d90 msgr2=0x7f5ebc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:27.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.881+0000 7f5ed3e00700 1 --2- 172.21.15.89:0/2478101373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ebc004d90 0x7f5ebc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.881+0000 7f5ed3e00700 1 -- 172.21.15.89:0/2478101373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc0035b0 con 0x7f5ebc004170 2024-04-02T09:15:27.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.881+0000 7f5ed9800700 1 --2- 172.21.15.89:0/2478101373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ebc004d90 0x7f5ebc005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:27.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.881+0000 7f5ed3e00700 1 --2- 172.21.15.89:0/2478101373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ebc004170 0x7f5ebc004850 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0007fd0 tx=0x7f5ebc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e778f7cdcfcc1a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:27.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.881+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0007ac0 con 0x7f5ebc004170 2024-04-02T09:15:27.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.881+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec000cb90 con 0x7f5ebc004170 2024-04-02T09:15:27.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.881+0000 7f5ed3400700 1 -- 172.21.15.89:0/2478101373 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec00070d0 con 0x7f5ebc004170 2024-04-02T09:15:27.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.882+0000 7f5edb800700 1 -- 172.21.15.89:0/2478101373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ebc004170 msgr2=0x7f5ebc004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.882+0000 7f5edb800700 1 --2- 172.21.15.89:0/2478101373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ebc004170 0x7f5ebc004850 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0007fd0 tx=0x7f5ebc005880 comp rx=0 tx=0).stop 2024-04-02T09:15:27.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.882+0000 7f5ed9800700 1 -- 172.21.15.89:0/2478101373 reap_dead start 2024-04-02T09:15:27.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.882+0000 7f5edb800700 1 -- 172.21.15.89:0/2478101373 shutdown_connections 2024-04-02T09:15:27.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.882+0000 7f5edb800700 1 -- 172.21.15.89:0/2478101373 >> 172.21.15.89:0/2478101373 conn(0x7f5ed408aff0 msgr2=0x7f5ed40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:27.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.882+0000 7f5edb800700 1 -- 172.21.15.89:0/2478101373 shutdown_connections 2024-04-02T09:15:27.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.882+0000 7f5edb800700 1 -- 172.21.15.89:0/2478101373 wait complete. 2024-04-02T09:15:27.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.884+0000 7f5edb800700 1 Processor -- start 2024-04-02T09:15:27.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.884+0000 7f5edb800700 1 -- start start 2024-04-02T09:15:27.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5edb800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ebc004170 0x7f5ed40b8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5edb800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ed40b45e0 0x7f5ed41f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5edb800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ed40b4a60 0x7f5ed40c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5edb800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed415f130 con 0x7f5ed40b45e0 2024-04-02T09:15:27.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5edb800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed415f410 con 0x7f5ebc004170 2024-04-02T09:15:27.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5edb800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed415f6f0 con 0x7f5ed40b4a60 2024-04-02T09:15:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5ed3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ed40b45e0 0x7f5ed41f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5ed3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ed40b45e0 0x7f5ed41f3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59142/0 (socket says 172.21.15.89:59142) 2024-04-02T09:15:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5ed3e00700 1 -- 172.21.15.89:0/2289133122 learned_addr learned my addr 172.21.15.89:0/2289133122 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5ed9800700 1 --2- 172.21.15.89:0/2289133122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ed40b4a60 0x7f5ed40c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5ed8e00700 1 --2- 172.21.15.89:0/2289133122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ebc004170 0x7f5ed40b8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5ed3e00700 1 -- 172.21.15.89:0/2289133122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ed40b4a60 msgr2=0x7f5ed40c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5ed3e00700 1 --2- 172.21.15.89:0/2289133122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ed40b4a60 0x7f5ed40c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5ed3e00700 1 -- 172.21.15.89:0/2289133122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ebc004170 msgr2=0x7f5ed40b8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5ed3e00700 1 --2- 172.21.15.89:0/2289133122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ebc004170 0x7f5ed40b8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.886+0000 7f5ed3e00700 1 -- 172.21.15.89:0/2289133122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed415f9d0 con 0x7f5ed40b45e0 2024-04-02T09:15:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.887+0000 7f5ed3e00700 1 --2- 172.21.15.89:0/2289133122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ed40b45e0 0x7f5ed41f3ac0 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0000ef0 tx=0x7f5ed41615f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.887+0000 7f5ed0c00700 1 -- 172.21.15.89:0/2289133122 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec00072e0 con 0x7f5ed40b45e0 2024-04-02T09:15:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.887+0000 7f5ed0c00700 1 -- 172.21.15.89:0/2289133122 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec0007460 con 0x7f5ed40b45e0 2024-04-02T09:15:27.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.887+0000 7f5edb800700 1 -- 172.21.15.89:0/2289133122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed415fcb0 con 0x7f5ed40b45e0 2024-04-02T09:15:27.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.887+0000 7f5ed0c00700 1 -- 172.21.15.89:0/2289133122 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0005830 con 0x7f5ed40b45e0 2024-04-02T09:15:27.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.887+0000 7f5edb800700 1 -- 172.21.15.89:0/2289133122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed4158a80 con 0x7f5ed40b45e0 2024-04-02T09:15:27.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.888+0000 7f5edb800700 1 -- 172.21.15.89:0/2289133122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed40d1ca0 con 0x7f5ed40b45e0 2024-04-02T09:15:27.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.889+0000 7f5ed0c00700 1 -- 172.21.15.89:0/2289133122 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5ec0019030 con 0x7f5ed40b45e0 2024-04-02T09:15:27.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.890+0000 7f5ed0c00700 1 --2- 172.21.15.89:0/2289133122 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5eb406adb0 0x7f5eb406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:27.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.891+0000 7f5ed0c00700 1 -- 172.21.15.89:0/2289133122 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f5ec008a430 con 0x7f5ed40b45e0 2024-04-02T09:15:27.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.891+0000 7f5ed8e00700 1 --2- 172.21.15.89:0/2289133122 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5eb406adb0 0x7f5eb406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:27.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.891+0000 7f5ed8e00700 1 --2- 172.21.15.89:0/2289133122 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5eb406adb0 0x7f5eb406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8007fd0 tx=0x7f5ec8009040 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:27.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:27.895+0000 7f5ed0c00700 1 -- 172.21.15.89:0/2289133122 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5ec0055330 con 0x7f5ed40b45e0 2024-04-02T09:15:28.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:28.142+0000 7f5edb800700 1 -- 172.21.15.89:0/2289133122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5ed4067720 con 0x7f5ed40b45e0 2024-04-02T09:15:28.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:28.786+0000 7f5ed0c00700 1 -- 172.21.15.89:0/2289133122 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v143) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5ec0058950 con 0x7f5ed40b45e0 2024-04-02T09:15:28.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:28.789+0000 7f5edb800700 1 -- 172.21.15.89:0/2289133122 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5eb406adb0 msgr2=0x7f5eb406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:28.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:28.789+0000 7f5edb800700 1 --2- 172.21.15.89:0/2289133122 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5eb406adb0 0x7f5eb406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8007fd0 tx=0x7f5ec8009040 comp rx=0 tx=0).stop 2024-04-02T09:15:28.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:28.789+0000 7f5edb800700 1 -- 172.21.15.89:0/2289133122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ed40b45e0 msgr2=0x7f5ed41f3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:28.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:28.789+0000 7f5edb800700 1 --2- 172.21.15.89:0/2289133122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ed40b45e0 0x7f5ed41f3ac0 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0000ef0 tx=0x7f5ed41615f0 comp rx=0 tx=0).stop 2024-04-02T09:15:28.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:28.789+0000 7f5edb800700 1 -- 172.21.15.89:0/2289133122 shutdown_connections 2024-04-02T09:15:28.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:28.789+0000 7f5edb800700 1 --2- 172.21.15.89:0/2289133122 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5eb406adb0 0x7f5eb406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:28.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:28.789+0000 7f5edb800700 1 --2- 172.21.15.89:0/2289133122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ebc004170 0x7f5ed40b8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:28.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:28.789+0000 7f5edb800700 1 --2- 172.21.15.89:0/2289133122 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5ed40b4a60 0x7f5ed40c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:28.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:28.789+0000 7f5edb800700 1 --2- 172.21.15.89:0/2289133122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ed40b45e0 0x7f5ed41f3ac0 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:28.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:28.789+0000 7f5edb800700 1 -- 172.21.15.89:0/2289133122 >> 172.21.15.89:0/2289133122 conn(0x7f5ed408aff0 msgr2=0x7f5ed415b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:28.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:28.790+0000 7f5edb800700 1 -- 172.21.15.89:0/2289133122 shutdown_connections 2024-04-02T09:15:28.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:28.790+0000 7f5edb800700 1 -- 172.21.15.89:0/2289133122 wait complete. 2024-04-02T09:15:28.809 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-02T09:15:28.809 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-02T09:15:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.006+0000 7f17ccc00700 1 Processor -- start 2024-04-02T09:15:29.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.006+0000 7f17ccc00700 1 -- start start 2024-04-02T09:15:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.008+0000 7f17ccc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f17c815dfe0 0x7f17c80ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.008+0000 7f17ccc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c80ce670 0x7f17c80cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.008+0000 7f17ccc00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c80cb310 con 0x7f17c815a5c0 2024-04-02T09:15:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.008+0000 7f17ccc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c80ceb00 con 0x7f17c815dfe0 2024-04-02T09:15:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.008+0000 7f17ccc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c80711a0 con 0x7f17c80ce670 2024-04-02T09:15:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.008+0000 7f17c5400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f17c815dfe0 0x7f17c80ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.008+0000 7f17c5400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f17c815dfe0 0x7f17c80ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52482/0 (socket says 172.21.15.89:52482) 2024-04-02T09:15:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.008+0000 7f17c5400700 1 -- 172.21.15.89:0/4520765 learned_addr learned my addr 172.21.15.89:0/4520765 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.008+0000 7f17c6800700 1 --2- 172.21.15.89:0/4520765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c80ce670 0x7f17c80cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.008+0000 7f17c5400700 1 -- 172.21.15.89:0/4520765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c80ce670 msgr2=0x7f17c80cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:29.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.008+0000 7f17c5400700 1 --2- 172.21.15.89:0/4520765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c80ce670 0x7f17c80cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:29.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.008+0000 7f17c5400700 1 -- 172.21.15.89:0/4520765 >> v1:172.21.15.113:6789/0 conn(0x7f17c815a5c0 legacy=0x7f17c80709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:29.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.008+0000 7f17c5400700 1 -- 172.21.15.89:0/4520765 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c8071480 con 0x7f17c815dfe0 2024-04-02T09:15:29.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.008+0000 7f17c6800700 1 --2- 172.21.15.89:0/4520765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c80ce670 0x7f17c80cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:15:29.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.009+0000 7f17c5400700 1 --2- 172.21.15.89:0/4520765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c815dfe0 0x7f17c80ce130 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f17b40098f0 tx=0x7f17b4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=affd079bda0bc8ce server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:29.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.009+0000 7f17c4a00700 1 -- 172.21.15.89:0/4520765 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b4004230 con 0x7f17c815dfe0 2024-04-02T09:15:29.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.009+0000 7f17c4a00700 1 -- 172.21.15.89:0/4520765 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b4004390 con 0x7f17c815dfe0 2024-04-02T09:15:29.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.009+0000 7f17c4a00700 1 -- 172.21.15.89:0/4520765 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b40036a0 con 0x7f17c815dfe0 2024-04-02T09:15:29.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.009+0000 7f17ccc00700 1 -- 172.21.15.89:0/4520765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c815dfe0 msgr2=0x7f17c80ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:29.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.009+0000 7f17ccc00700 1 --2- 172.21.15.89:0/4520765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c815dfe0 0x7f17c80ce130 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f17b40098f0 tx=0x7f17b4009c00 comp rx=0 tx=0).stop 2024-04-02T09:15:29.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.009+0000 7f17ccc00700 1 -- 172.21.15.89:0/4520765 shutdown_connections 2024-04-02T09:15:29.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.009+0000 7f17ccc00700 1 --2- 172.21.15.89:0/4520765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c80ce670 0x7f17c80cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:29.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.009+0000 7f17ccc00700 1 --2- 172.21.15.89:0/4520765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c815dfe0 0x7f17c80ce130 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:29.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.009+0000 7f17ccc00700 1 -- 172.21.15.89:0/4520765 >> 172.21.15.89:0/4520765 conn(0x7f17c80b0d70 msgr2=0x7f17c80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:29.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.010+0000 7f17ccc00700 1 -- 172.21.15.89:0/4520765 shutdown_connections 2024-04-02T09:15:29.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.010+0000 7f17ccc00700 1 -- 172.21.15.89:0/4520765 wait complete. 2024-04-02T09:15:29.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.011+0000 7f17ccc00700 1 Processor -- start 2024-04-02T09:15:29.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.011+0000 7f17ccc00700 1 -- start start 2024-04-02T09:15:29.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.012+0000 7f17ccc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c80ce670 0x7f17c81558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:29.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.012+0000 7f17ccc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c815a5c0 0x7f17c81f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:29.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.012+0000 7f17ccc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17c815dfe0 0x7f17c806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:29.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.012+0000 7f17ccc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c814ced0 con 0x7f17c80ce670 2024-04-02T09:15:29.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.012+0000 7f17ccc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81548d0 con 0x7f17c815dfe0 2024-04-02T09:15:29.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.012+0000 7f17ccc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c8155450 con 0x7f17c815a5c0 2024-04-02T09:15:29.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.012+0000 7f17c5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c80ce670 0x7f17c81558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:29.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.013+0000 7f17c5e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c80ce670 0x7f17c81558e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52490/0 (socket says 172.21.15.89:52490) 2024-04-02T09:15:29.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.013+0000 7f17c5e00700 1 -- 172.21.15.89:0/1156075943 learned_addr learned my addr 172.21.15.89:0/1156075943 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:29.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.013+0000 7f17c6800700 1 --2- 172.21.15.89:0/1156075943 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17c815dfe0 0x7f17c806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:29.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.013+0000 7f17c5400700 1 --2- 172.21.15.89:0/1156075943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c815a5c0 0x7f17c81f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:29.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.013+0000 7f17c5e00700 1 -- 172.21.15.89:0/1156075943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c815a5c0 msgr2=0x7f17c81f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:29.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.013+0000 7f17c5e00700 1 --2- 172.21.15.89:0/1156075943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c815a5c0 0x7f17c81f3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:29.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.013+0000 7f17c5e00700 1 -- 172.21.15.89:0/1156075943 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17c815dfe0 msgr2=0x7f17c806d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:29.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.013+0000 7f17c5e00700 1 --2- 172.21.15.89:0/1156075943 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17c815dfe0 0x7f17c806d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:29.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.013+0000 7f17c5e00700 1 -- 172.21.15.89:0/1156075943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b4009550 con 0x7f17c80ce670 2024-04-02T09:15:29.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.013+0000 7f17c5e00700 1 --2- 172.21.15.89:0/1156075943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c80ce670 0x7f17c81558e0 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7f17b800c4d0 tx=0x7f17b800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:29.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.014+0000 7f17be000700 1 -- 172.21.15.89:0/1156075943 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8011790 con 0x7f17c80ce670 2024-04-02T09:15:29.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.014+0000 7f17be000700 1 -- 172.21.15.89:0/1156075943 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b80118f0 con 0x7f17c80ce670 2024-04-02T09:15:29.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.014+0000 7f17ccc00700 1 -- 172.21.15.89:0/1156075943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c8154b20 con 0x7f17c80ce670 2024-04-02T09:15:29.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.014+0000 7f17be000700 1 -- 172.21.15.89:0/1156075943 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8011ca0 con 0x7f17c80ce670 2024-04-02T09:15:29.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.014+0000 7f17ccc00700 1 -- 172.21.15.89:0/1156075943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c81550e0 con 0x7f17c80ce670 2024-04-02T09:15:29.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.015+0000 7f17ccc00700 1 -- 172.21.15.89:0/1156075943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c8155730 con 0x7f17c80ce670 2024-04-02T09:15:29.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.016+0000 7f17be000700 1 -- 172.21.15.89:0/1156075943 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f17b801c460 con 0x7f17c80ce670 2024-04-02T09:15:29.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.017+0000 7f17be000700 1 --2- 172.21.15.89:0/1156075943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f17a806adb0 0x7f17a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:29.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.017+0000 7f17be000700 1 -- 172.21.15.89:0/1156075943 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f17b80892b0 con 0x7f17c80ce670 2024-04-02T09:15:29.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.017+0000 7f17c5400700 1 --2- 172.21.15.89:0/1156075943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f17a806adb0 0x7f17a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:29.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.018+0000 7f17c5400700 1 --2- 172.21.15.89:0/1156075943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f17a806adb0 0x7f17a806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009130 tx=0x7f17b4008030 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:29.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.021+0000 7f17be000700 1 -- 172.21.15.89:0/1156075943 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f17b8054250 con 0x7f17c80ce670 2024-04-02T09:15:29.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:29.274+0000 7f17ccc00700 1 -- 172.21.15.89:0/1156075943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f17c8155270 con 0x7f17c80ce670 2024-04-02T09:15:30.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.262+0000 7f17be000700 1 -- 172.21.15.89:0/1156075943 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v144) v1 ==== 122+0+0 (secure 0 0 0) 0x7f17b8057870 con 0x7f17c80ce670 2024-04-02T09:15:30.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.265+0000 7f17ccc00700 1 -- 172.21.15.89:0/1156075943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f17a806adb0 msgr2=0x7f17a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:30.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.266+0000 7f17ccc00700 1 --2- 172.21.15.89:0/1156075943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f17a806adb0 0x7f17a806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009130 tx=0x7f17b4008030 comp rx=0 tx=0).stop 2024-04-02T09:15:30.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.266+0000 7f17ccc00700 1 -- 172.21.15.89:0/1156075943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c80ce670 msgr2=0x7f17c81558e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:30.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.266+0000 7f17ccc00700 1 --2- 172.21.15.89:0/1156075943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c80ce670 0x7f17c81558e0 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7f17b800c4d0 tx=0x7f17b800c7e0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.266+0000 7f17ccc00700 1 -- 172.21.15.89:0/1156075943 shutdown_connections 2024-04-02T09:15:30.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.266+0000 7f17ccc00700 1 --2- 172.21.15.89:0/1156075943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f17a806adb0 0x7f17a806d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.266+0000 7f17ccc00700 1 --2- 172.21.15.89:0/1156075943 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17c815dfe0 0x7f17c806d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.266+0000 7f17ccc00700 1 --2- 172.21.15.89:0/1156075943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c80ce670 0x7f17c81558e0 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.266+0000 7f17ccc00700 1 --2- 172.21.15.89:0/1156075943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f17c815a5c0 0x7f17c81f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.266+0000 7f17ccc00700 1 -- 172.21.15.89:0/1156075943 >> 172.21.15.89:0/1156075943 conn(0x7f17c80b0d70 msgr2=0x7f17c80cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:30.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.266+0000 7f17ccc00700 1 -- 172.21.15.89:0/1156075943 shutdown_connections 2024-04-02T09:15:30.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.266+0000 7f17ccc00700 1 -- 172.21.15.89:0/1156075943 wait complete. 2024-04-02T09:15:30.286 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-02T09:15:30.286 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:15:30.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.482+0000 7fba94000700 1 Processor -- start 2024-04-02T09:15:30.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.482+0000 7fba94000700 1 -- start start 2024-04-02T09:15:30.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.484+0000 7fba94000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fba8c0b5160 0x7fba8c0ad180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:30.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.484+0000 7fba94000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8c1651d0 0x7fba8c1675c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:30.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.484+0000 7fba94000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8c0ad6c0 con 0x7fba8c0acda0 2024-04-02T09:15:30.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.484+0000 7fba94000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8c0b55e0 con 0x7fba8c0b5160 2024-04-02T09:15:30.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.484+0000 7fba94000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c06eef0 con 0x7fba8c1651d0 2024-04-02T09:15:30.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.484+0000 7fba91600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fba8c0acda0 0x7fba8c15e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48336/0 (socket says 172.21.15.89:48336) 2024-04-02T09:15:30.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.484+0000 7fba91600700 1 -- 172.21.15.89:0/2301232768 learned_addr learned my addr 172.21.15.89:0/2301232768 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:30.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.484+0000 7fba90c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fba8c0b5160 0x7fba8c0ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:30.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.484+0000 7fba8be00700 1 -- 172.21.15.89:0/2301232768 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794675301 0 0) 0x7fba8c0ad6c0 con 0x7fba8c0acda0 2024-04-02T09:15:30.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.484+0000 7fba8be00700 1 -- 172.21.15.89:0/2301232768 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba740035b0 con 0x7fba8c0acda0 2024-04-02T09:15:30.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.484+0000 7fba92000700 1 --2- 172.21.15.89:0/2301232768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8c1651d0 0x7fba8c1675c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:30.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.485+0000 7fba90c00700 1 -- 172.21.15.89:0/2301232768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8c1651d0 msgr2=0x7fba8c1675c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:30.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.485+0000 7fba90c00700 1 --2- 172.21.15.89:0/2301232768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8c1651d0 0x7fba8c1675c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.485+0000 7fba90c00700 1 -- 172.21.15.89:0/2301232768 >> v1:172.21.15.89:6789/0 conn(0x7fba8c0acda0 legacy=0x7fba8c15e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:30.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.485+0000 7fba90c00700 1 -- 172.21.15.89:0/2301232768 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c0ae7b0 con 0x7fba8c0b5160 2024-04-02T09:15:30.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.485+0000 7fba92000700 1 --2- 172.21.15.89:0/2301232768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8c1651d0 0x7fba8c1675c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:15:30.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.485+0000 7fba90c00700 1 --2- 172.21.15.89:0/2301232768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0b5160 0x7fba8c0ad180 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7fba78009840 tx=0x7fba78009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11842d93b3bb448 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:30.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.485+0000 7fba8be00700 1 -- 172.21.15.89:0/2301232768 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78011830 con 0x7fba8c0b5160 2024-04-02T09:15:30.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.485+0000 7fba8be00700 1 -- 172.21.15.89:0/2301232768 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba78011990 con 0x7fba8c0b5160 2024-04-02T09:15:30.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.485+0000 7fba8be00700 1 -- 172.21.15.89:0/2301232768 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78011d20 con 0x7fba8c0b5160 2024-04-02T09:15:30.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.485+0000 7fba94000700 1 -- 172.21.15.89:0/2301232768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0b5160 msgr2=0x7fba8c0ad180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.486+0000 7fba94000700 1 --2- 172.21.15.89:0/2301232768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0b5160 0x7fba8c0ad180 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7fba78009840 tx=0x7fba78009b50 comp rx=0 tx=0).stop 2024-04-02T09:15:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.486+0000 7fba94000700 1 -- 172.21.15.89:0/2301232768 shutdown_connections 2024-04-02T09:15:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.486+0000 7fba94000700 1 --2- 172.21.15.89:0/2301232768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8c1651d0 0x7fba8c1675c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.486+0000 7fba94000700 1 --2- 172.21.15.89:0/2301232768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0b5160 0x7fba8c0ad180 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.486+0000 7fba94000700 1 -- 172.21.15.89:0/2301232768 >> 172.21.15.89:0/2301232768 conn(0x7fba8c072170 msgr2=0x7fba8c06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.486+0000 7fba94000700 1 -- 172.21.15.89:0/2301232768 shutdown_connections 2024-04-02T09:15:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.486+0000 7fba94000700 1 -- 172.21.15.89:0/2301232768 wait complete. 2024-04-02T09:15:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.487+0000 7fba94000700 1 Processor -- start 2024-04-02T09:15:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.487+0000 7fba94000700 1 -- start start 2024-04-02T09:15:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.488+0000 7fba94000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8c0acda0 0x7fba8c13cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.488+0000 7fba94000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0b5160 0x7fba8c151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.488+0000 7fba94000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba8c1651d0 0x7fba8c15f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.488+0000 7fba94000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c15e5b0 con 0x7fba8c0b5160 2024-04-02T09:15:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.488+0000 7fba94000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c15e850 con 0x7fba8c1651d0 2024-04-02T09:15:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.488+0000 7fba94000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c15eb30 con 0x7fba8c0acda0 2024-04-02T09:15:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.489+0000 7fba90c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0b5160 0x7fba8c151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.489+0000 7fba90c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0b5160 0x7fba8c151c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52510/0 (socket says 172.21.15.89:52510) 2024-04-02T09:15:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.489+0000 7fba90c00700 1 -- 172.21.15.89:0/2240135300 learned_addr learned my addr 172.21.15.89:0/2240135300 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.489+0000 7fba92000700 1 --2- 172.21.15.89:0/2240135300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba8c1651d0 0x7fba8c15f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.489+0000 7fba91600700 1 --2- 172.21.15.89:0/2240135300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8c0acda0 0x7fba8c13cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.489+0000 7fba90c00700 1 -- 172.21.15.89:0/2240135300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8c0acda0 msgr2=0x7fba8c13cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.489+0000 7fba90c00700 1 --2- 172.21.15.89:0/2240135300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8c0acda0 0x7fba8c13cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.489+0000 7fba90c00700 1 -- 172.21.15.89:0/2240135300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba8c1651d0 msgr2=0x7fba8c15f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.489+0000 7fba90c00700 1 --2- 172.21.15.89:0/2240135300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba8c1651d0 0x7fba8c15f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.489+0000 7fba90c00700 1 -- 172.21.15.89:0/2240135300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba780094a0 con 0x7fba8c0b5160 2024-04-02T09:15:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.489+0000 7fba90c00700 1 --2- 172.21.15.89:0/2240135300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0b5160 0x7fba8c151c80 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7fba78009fd0 tx=0x7fba78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.489+0000 7fba89600700 1 -- 172.21.15.89:0/2240135300 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba780036a0 con 0x7fba8c0b5160 2024-04-02T09:15:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.490+0000 7fba89600700 1 -- 172.21.15.89:0/2240135300 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7801f070 con 0x7fba8c0b5160 2024-04-02T09:15:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.490+0000 7fba94000700 1 -- 172.21.15.89:0/2240135300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba8c15ee10 con 0x7fba8c0b5160 2024-04-02T09:15:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.490+0000 7fba89600700 1 -- 172.21.15.89:0/2240135300 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78003a60 con 0x7fba8c0b5160 2024-04-02T09:15:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.490+0000 7fba94000700 1 -- 172.21.15.89:0/2240135300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba8c15f3d0 con 0x7fba8c0b5160 2024-04-02T09:15:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.491+0000 7fba94000700 1 -- 172.21.15.89:0/2240135300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba8c066e90 con 0x7fba8c0b5160 2024-04-02T09:15:30.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.492+0000 7fba89600700 1 -- 172.21.15.89:0/2240135300 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fba78003bc0 con 0x7fba8c0b5160 2024-04-02T09:15:30.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.492+0000 7fba89600700 1 --2- 172.21.15.89:0/2240135300 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fba6c06adb0 0x7fba6c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:30.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.493+0000 7fba89600700 1 -- 172.21.15.89:0/2240135300 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fba78021070 con 0x7fba8c0b5160 2024-04-02T09:15:30.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.493+0000 7fba91600700 1 --2- 172.21.15.89:0/2240135300 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fba6c06adb0 0x7fba6c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:30.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.493+0000 7fba91600700 1 --2- 172.21.15.89:0/2240135300 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fba6c06adb0 0x7fba6c06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fba800011e0 tx=0x7fba80009040 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:30.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.497+0000 7fba89600700 1 -- 172.21.15.89:0/2240135300 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fba7808e030 con 0x7fba8c0b5160 2024-04-02T09:15:30.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.740+0000 7fba94000700 1 -- 172.21.15.89:0/2240135300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba8c15f5c0 con 0x7fba8c0b5160 2024-04-02T09:15:30.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.740+0000 7fba89600700 1 -- 172.21.15.89:0/2240135300 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4753 (secure 0 0 0) 0x7fba7802b090 con 0x7fba8c0b5160 2024-04-02T09:15:30.739 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:30.739 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10376,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2447701494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2447701494},{"type":"v1","addr":"172.21.15.113:6834","nonce":2447701494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":12103,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/361614750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":361614750},{"type":"v1","addr":"172.21.15.113:6835","nonce":361614750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":13281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2871180587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2871180587},{"type":"v1","addr":"172.21.15.89:6835","nonce":2871180587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:15:26.772241+0000","modified":"2024-04-02T09:15:30.258749+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13308":{"gid":13308,"name":"a","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3853118738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3853118738},{"type":"v1","addr":"172.21.15.89:6837","nonce":3853118738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-04-02T09:15:30.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.743+0000 7fba94000700 1 -- 172.21.15.89:0/2240135300 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fba6c06adb0 msgr2=0x7fba6c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:30.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.743+0000 7fba94000700 1 --2- 172.21.15.89:0/2240135300 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fba6c06adb0 0x7fba6c06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fba800011e0 tx=0x7fba80009040 comp rx=0 tx=0).stop 2024-04-02T09:15:30.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.743+0000 7fba94000700 1 -- 172.21.15.89:0/2240135300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0b5160 msgr2=0x7fba8c151c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:30.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.743+0000 7fba94000700 1 --2- 172.21.15.89:0/2240135300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0b5160 0x7fba8c151c80 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7fba78009fd0 tx=0x7fba78004750 comp rx=0 tx=0).stop 2024-04-02T09:15:30.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.743+0000 7fba94000700 1 -- 172.21.15.89:0/2240135300 shutdown_connections 2024-04-02T09:15:30.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.743+0000 7fba94000700 1 --2- 172.21.15.89:0/2240135300 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fba6c06adb0 0x7fba6c06d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.743+0000 7fba94000700 1 --2- 172.21.15.89:0/2240135300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba8c1651d0 0x7fba8c15f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.743+0000 7fba94000700 1 --2- 172.21.15.89:0/2240135300 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba8c0acda0 0x7fba8c13cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.743+0000 7fba94000700 1 --2- 172.21.15.89:0/2240135300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba8c0b5160 0x7fba8c151c80 unknown :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.743+0000 7fba94000700 1 -- 172.21.15.89:0/2240135300 >> 172.21.15.89:0/2240135300 conn(0x7fba8c072170 msgr2=0x7fba8c0b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:30.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.743+0000 7fba94000700 1 -- 172.21.15.89:0/2240135300 shutdown_connections 2024-04-02T09:15:30.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.743+0000 7fba94000700 1 -- 172.21.15.89:0/2240135300 wait complete. 2024-04-02T09:15:30.743 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 144 2024-04-02T09:15:30.760 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:15:30.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.953+0000 7f057e400700 1 Processor -- start 2024-04-02T09:15:30.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.953+0000 7f057e400700 1 -- start start 2024-04-02T09:15:30.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.954+0000 7f057e400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0578160f10 0x7f05780d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:30.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.954+0000 7f057e400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05780ceda0 con 0x7f0578161590 2024-04-02T09:15:30.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.954+0000 7f057e400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05781612f0 con 0x7f05780cf230 2024-04-02T09:15:30.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.954+0000 7f057e400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05780ce470 con 0x7f0578160f10 2024-04-02T09:15:30.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.955+0000 7f0577e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0578161590 0x7f05780b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48340/0 (socket says 172.21.15.89:48340) 2024-04-02T09:15:30.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.955+0000 7f0577e00700 1 -- 172.21.15.89:0/2084991654 learned_addr learned my addr 172.21.15.89:0/2084991654 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:30.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.955+0000 7f0576000700 1 -- 172.21.15.89:0/2084991654 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202846582 0 0) 0x7f05780ceda0 con 0x7f0578161590 2024-04-02T09:15:30.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.955+0000 7f0576000700 1 -- 172.21.15.89:0/2084991654 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f055c0035b0 con 0x7f0578161590 2024-04-02T09:15:30.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.955+0000 7f0577400700 1 --2- 172.21.15.89:0/2084991654 >> v2:172.21.15.113:3300/0 conn(0x7f0578160f10 0x7f05780d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:30.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.955+0000 7f0576000700 1 -- 172.21.15.89:0/2084991654 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2301551450 0 0) 0x7f055c0035b0 con 0x7f0578161590 2024-04-02T09:15:30.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.955+0000 7f0576000700 1 -- 172.21.15.89:0/2084991654 >> v2:172.21.15.113:3300/0 conn(0x7f0578160f10 msgr2=0x7f05780d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:30.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.955+0000 7f0576000700 1 --2- 172.21.15.89:0/2084991654 >> v2:172.21.15.113:3300/0 conn(0x7f0578160f10 0x7f05780d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.955+0000 7f0576000700 1 -- 172.21.15.89:0/2084991654 >> v1:172.21.15.113:6789/0 conn(0x7f05780cf230 legacy=0x7f05780d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:30.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.955+0000 7f0576000700 1 -- 172.21.15.89:0/2084991654 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05780ce750 con 0x7f0578161590 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.955+0000 7f0576000700 1 -- 172.21.15.89:0/2084991654 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f05700019f0 con 0x7f0578161590 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0576000700 1 -- 172.21.15.89:0/2084991654 >> v1:172.21.15.89:6789/0 conn(0x7f0578161590 legacy=0x7f05780b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0576000700 1 --2- 172.21.15.89:0/2084991654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f055c003a00 0x7f055c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0576000700 1 --2- 172.21.15.89:0/2084991654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f055c004850 0x7f055c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0576000700 1 --2- 172.21.15.89:0/2084991654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f055c005210 0x7f055c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0576000700 1 -- 172.21.15.89:0/2084991654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05780ce470 con 0x7f055c005210 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0576000700 1 -- 172.21.15.89:0/2084991654 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055c00c1b0 con 0x7f055c003a00 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0576000700 1 -- 172.21.15.89:0/2084991654 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f055c00c370 con 0x7f055c004850 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0577e00700 1 --2- 172.21.15.89:0/2084991654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f055c005210 0x7f055c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0577400700 1 --2- 172.21.15.89:0/2084991654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f055c003a00 0x7f055c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0576a00700 1 --2- 172.21.15.89:0/2084991654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f055c004850 0x7f055c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0577e00700 1 -- 172.21.15.89:0/2084991654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f055c004850 msgr2=0x7f055c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0577e00700 1 --2- 172.21.15.89:0/2084991654 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f055c004850 0x7f055c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0577e00700 1 -- 172.21.15.89:0/2084991654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f055c003a00 msgr2=0x7f055c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0577e00700 1 --2- 172.21.15.89:0/2084991654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f055c003a00 0x7f055c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0577e00700 1 -- 172.21.15.89:0/2084991654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055c0035b0 con 0x7f055c005210 2024-04-02T09:15:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.956+0000 7f0577e00700 1 -- 172.21.15.89:0/2084991654 reap_dead start 2024-04-02T09:15:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.957+0000 7f0577e00700 1 --2- 172.21.15.89:0/2084991654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f055c005210 0x7f055c00ba10 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7f057000a830 tx=0x7f055c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2b22e19351bd8e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.957+0000 7f0576000700 1 -- 172.21.15.89:0/2084991654 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057000a2f0 con 0x7f055c005210 2024-04-02T09:15:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.957+0000 7f0576000700 1 -- 172.21.15.89:0/2084991654 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0570012620 con 0x7f055c005210 2024-04-02T09:15:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.957+0000 7f0576000700 1 -- 172.21.15.89:0/2084991654 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05700129d0 con 0x7f055c005210 2024-04-02T09:15:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.957+0000 7f057e400700 1 -- 172.21.15.89:0/2084991654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f055c005210 msgr2=0x7f055c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.957+0000 7f057e400700 1 --2- 172.21.15.89:0/2084991654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f055c005210 0x7f055c00ba10 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7f057000a830 tx=0x7f055c005880 comp rx=0 tx=0).stop 2024-04-02T09:15:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.957+0000 7f057e400700 1 -- 172.21.15.89:0/2084991654 shutdown_connections 2024-04-02T09:15:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.957+0000 7f057e400700 1 --2- 172.21.15.89:0/2084991654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f055c005210 0x7f055c00ba10 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.957+0000 7f057e400700 1 -- 172.21.15.89:0/2084991654 >> 172.21.15.89:0/2084991654 conn(0x7f057808aff0 msgr2=0x7f05780d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.957+0000 7f057e400700 1 -- 172.21.15.89:0/2084991654 shutdown_connections 2024-04-02T09:15:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.958+0000 7f057e400700 1 -- 172.21.15.89:0/2084991654 wait complete. 2024-04-02T09:15:30.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.959+0000 7f057e400700 1 Processor -- start 2024-04-02T09:15:30.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.959+0000 7f057e400700 1 -- start start 2024-04-02T09:15:30.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.960+0000 7f057e400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f055c005210 0x7f05781f7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:30.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.960+0000 7f057e400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f057815e9d0 0x7f0578159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:30.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.960+0000 7f057e400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0578151110 0x7f0578151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:30.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.960+0000 7f057e400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0578151ad0 con 0x7f057815e9d0 2024-04-02T09:15:30.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.960+0000 7f057e400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057815f1c0 con 0x7f0578151110 2024-04-02T09:15:30.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.960+0000 7f057e400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057815f4a0 con 0x7f055c005210 2024-04-02T09:15:30.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.960+0000 7f0576a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f057815e9d0 0x7f0578159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:30.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.960+0000 7f0576a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f057815e9d0 0x7f0578159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52516/0 (socket says 172.21.15.89:52516) 2024-04-02T09:15:30.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.960+0000 7f0576a00700 1 -- 172.21.15.89:0/441067338 learned_addr learned my addr 172.21.15.89:0/441067338 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:30.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.961+0000 7f0577400700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f055c005210 0x7f05781f7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:30.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.961+0000 7f0577e00700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0578151110 0x7f0578151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.961+0000 7f0576a00700 1 -- 172.21.15.89:0/441067338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f055c005210 msgr2=0x7f05781f7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.961+0000 7f0576a00700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f055c005210 0x7f05781f7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.961+0000 7f0576a00700 1 -- 172.21.15.89:0/441067338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0578151110 msgr2=0x7f0578151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.961+0000 7f0576a00700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0578151110 0x7f0578151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.961+0000 7f0576a00700 1 -- 172.21.15.89:0/441067338 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057815f780 con 0x7f057815e9d0 2024-04-02T09:15:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.961+0000 7f0577e00700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0578151110 0x7f0578151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.961+0000 7f0576a00700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f057815e9d0 0x7f0578159080 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7f056400e490 tx=0x7f056400e850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.961+0000 7f056f400700 1 -- 172.21.15.89:0/441067338 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05640052e0 con 0x7f057815e9d0 2024-04-02T09:15:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.961+0000 7f056f400700 1 -- 172.21.15.89:0/441067338 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0564005440 con 0x7f057815e9d0 2024-04-02T09:15:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.961+0000 7f057e400700 1 -- 172.21.15.89:0/441067338 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05781567c0 con 0x7f057815e9d0 2024-04-02T09:15:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.961+0000 7f056f400700 1 -- 172.21.15.89:0/441067338 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05640038c0 con 0x7f057815e9d0 2024-04-02T09:15:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.962+0000 7f057e400700 1 -- 172.21.15.89:0/441067338 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057815ee30 con 0x7f057815e9d0 2024-04-02T09:15:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.962+0000 7f057e400700 1 -- 172.21.15.89:0/441067338 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05780d4520 con 0x7f057815e9d0 2024-04-02T09:15:30.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.963+0000 7f056f400700 1 -- 172.21.15.89:0/441067338 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0564003a20 con 0x7f057815e9d0 2024-04-02T09:15:30.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.964+0000 7f056f400700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f055806adb0 0x7f055806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:30.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.964+0000 7f056f400700 1 -- 172.21.15.89:0/441067338 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f0564088c40 con 0x7f057815e9d0 2024-04-02T09:15:30.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.964+0000 7f0577400700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f055806adb0 0x7f055806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:30.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.965+0000 7f0577400700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f055806adb0 0x7f055806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0568009be0 tx=0x7f0568009210 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:30.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:30.969+0000 7f056f400700 1 -- 172.21.15.89:0/441067338 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0564053b60 con 0x7f057815e9d0 2024-04-02T09:15:31.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.174+0000 7f057e400700 1 -- 172.21.15.89:0/441067338 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0578067720 con 0x7f057815e9d0 2024-04-02T09:15:31.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.175+0000 7f056f400700 1 -- 172.21.15.89:0/441067338 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v279) v1 ==== 74+0+16736 (secure 0 0 0) 0x7f0564019020 con 0x7f057815e9d0 2024-04-02T09:15:31.174 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:31.174 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":279,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:15:27.774751+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:15:24.345253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-04-02T09:15:25.756381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"279","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:15:31.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.177+0000 7f057e400700 1 -- 172.21.15.89:0/441067338 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f055806adb0 msgr2=0x7f055806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:31.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.178+0000 7f057e400700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f055806adb0 0x7f055806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0568009be0 tx=0x7f0568009210 comp rx=0 tx=0).stop 2024-04-02T09:15:31.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.178+0000 7f057e400700 1 -- 172.21.15.89:0/441067338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f057815e9d0 msgr2=0x7f0578159080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:31.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.178+0000 7f057e400700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f057815e9d0 0x7f0578159080 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7f056400e490 tx=0x7f056400e850 comp rx=0 tx=0).stop 2024-04-02T09:15:31.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.178+0000 7f057e400700 1 -- 172.21.15.89:0/441067338 shutdown_connections 2024-04-02T09:15:31.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.178+0000 7f057e400700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f055806adb0 0x7f055806d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.178+0000 7f057e400700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0578151110 0x7f0578151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.178+0000 7f057e400700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f055c005210 0x7f05781f7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.178+0000 7f057e400700 1 --2- 172.21.15.89:0/441067338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f057815e9d0 0x7f0578159080 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.178+0000 7f057e400700 1 -- 172.21.15.89:0/441067338 >> 172.21.15.89:0/441067338 conn(0x7f057808aff0 msgr2=0x7f05780ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:31.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.178+0000 7f057e400700 1 -- 172.21.15.89:0/441067338 shutdown_connections 2024-04-02T09:15:31.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.178+0000 7f057e400700 1 -- 172.21.15.89:0/441067338 wait complete. 2024-04-02T09:15:31.194 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T09:15:31.194 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:15:31.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.389+0000 7f1f4f600700 1 Processor -- start 2024-04-02T09:15:31.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.389+0000 7f1f4f600700 1 -- start start 2024-04-02T09:15:31.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.390+0000 7f1f4f600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1f480d18b0 0x7f1f480cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:31.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.390+0000 7f1f4f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f48161590 0x7f1f48161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:31.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.391+0000 7f1f4f600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f480ceda0 con 0x7f1f48160f10 2024-04-02T09:15:31.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.391+0000 7f1f4f600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f480d1d30 con 0x7f1f480d18b0 2024-04-02T09:15:31.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.391+0000 7f1f4f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f48161f60 con 0x7f1f48161590 2024-04-02T09:15:31.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.391+0000 7f1f47e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1f480d18b0 0x7f1f480cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.391+0000 7f1f4d600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f48161590 0x7f1f48161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.391+0000 7f1f47e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1f480d18b0 0x7f1f480cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:59342/0 (socket says 172.21.15.89:59342) 2024-04-02T09:15:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.391+0000 7f1f4d600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f48161590 0x7f1f48161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:40544/0 (socket says 172.21.15.89:40544) 2024-04-02T09:15:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.391+0000 7f1f47e00700 1 -- 172.21.15.89:0/2656129703 learned_addr learned my addr 172.21.15.89:0/2656129703 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.391+0000 7f1f4cc00700 1 --1- 172.21.15.89:0/2656129703 >> v1:172.21.15.113:6789/0 conn(0x7f1f48160f10 0x7f1f480d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:43496/0 (socket says 172.21.15.89:43496) 2024-04-02T09:15:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.392+0000 7f1f4d600700 1 -- 172.21.15.89:0/2656129703 >> v2:172.21.15.113:3300/0 conn(0x7f1f480d18b0 msgr2=0x7f1f480cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.392+0000 7f1f4d600700 1 --2- 172.21.15.89:0/2656129703 >> v2:172.21.15.113:3300/0 conn(0x7f1f480d18b0 0x7f1f480cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.392+0000 7f1f4d600700 1 -- 172.21.15.89:0/2656129703 >> v1:172.21.15.113:6789/0 conn(0x7f1f48160f10 legacy=0x7f1f480d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.392+0000 7f1f4d600700 1 -- 172.21.15.89:0/2656129703 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f481612f0 con 0x7f1f48161590 2024-04-02T09:15:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.392+0000 7f1f47e00700 1 --2- 172.21.15.89:0/2656129703 >> v2:172.21.15.113:3300/0 conn(0x7f1f480d18b0 0x7f1f480cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:15:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.392+0000 7f1f47400700 1 -- 172.21.15.89:0/2656129703 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739659378 0 0) 0x7f1f480ceda0 con 0x7f1f48160f10 2024-04-02T09:15:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.392+0000 7f1f4d600700 1 --2- 172.21.15.89:0/2656129703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f48161590 0x7f1f48161a20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009d90 tx=0x7f1f4000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=316392f8bafe9cf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.392+0000 7f1f47400700 1 -- 172.21.15.89:0/2656129703 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4000e070 con 0x7f1f48161590 2024-04-02T09:15:31.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.392+0000 7f1f47400700 1 -- 172.21.15.89:0/2656129703 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4000bb20 con 0x7f1f48161590 2024-04-02T09:15:31.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.392+0000 7f1f47400700 1 -- 172.21.15.89:0/2656129703 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40012430 con 0x7f1f48161590 2024-04-02T09:15:31.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.393+0000 7f1f4f600700 1 -- 172.21.15.89:0/2656129703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f48161590 msgr2=0x7f1f48161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:31.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.393+0000 7f1f4f600700 1 --2- 172.21.15.89:0/2656129703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f48161590 0x7f1f48161a20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009d90 tx=0x7f1f4000b330 comp rx=0 tx=0).stop 2024-04-02T09:15:31.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.393+0000 7f1f4f600700 1 -- 172.21.15.89:0/2656129703 shutdown_connections 2024-04-02T09:15:31.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.393+0000 7f1f4f600700 1 --2- 172.21.15.89:0/2656129703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f48161590 0x7f1f48161a20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.393+0000 7f1f4f600700 1 --2- 172.21.15.89:0/2656129703 >> v2:172.21.15.113:3300/0 conn(0x7f1f480d18b0 0x7f1f480cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.393+0000 7f1f4f600700 1 -- 172.21.15.89:0/2656129703 >> 172.21.15.89:0/2656129703 conn(0x7f1f4808aff0 msgr2=0x7f1f480d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:31.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.393+0000 7f1f4f600700 1 -- 172.21.15.89:0/2656129703 shutdown_connections 2024-04-02T09:15:31.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.393+0000 7f1f4f600700 1 -- 172.21.15.89:0/2656129703 wait complete. 2024-04-02T09:15:31.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.395+0000 7f1f4f600700 1 Processor -- start 2024-04-02T09:15:31.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.395+0000 7f1f4f600700 1 -- start start 2024-04-02T09:15:31.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4f600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f480d18b0 0x7f1f48158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:31.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4f600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f48160f10 0x7f1f480b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:31.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4f600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f48161590 0x7f1f480b8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4f600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f481591d0 con 0x7f1f480d18b0 2024-04-02T09:15:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4f600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4809e9d0 con 0x7f1f48160f10 2024-04-02T09:15:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4f600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4809ecb0 con 0x7f1f48161590 2024-04-02T09:15:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f480d18b0 0x7f1f48158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f480d18b0 0x7f1f48158c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52526/0 (socket says 172.21.15.89:52526) 2024-04-02T09:15:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4cc00700 1 -- 172.21.15.89:0/1025238356 learned_addr learned my addr 172.21.15.89:0/1025238356 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f47e00700 1 --2- 172.21.15.89:0/1025238356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f48160f10 0x7f1f480b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4d600700 1 --2- 172.21.15.89:0/1025238356 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f48161590 0x7f1f480b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4cc00700 1 -- 172.21.15.89:0/1025238356 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f48161590 msgr2=0x7f1f480b8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4cc00700 1 --2- 172.21.15.89:0/1025238356 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f48161590 0x7f1f480b8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4cc00700 1 -- 172.21.15.89:0/1025238356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f48160f10 msgr2=0x7f1f480b8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4cc00700 1 --2- 172.21.15.89:0/1025238356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f48160f10 0x7f1f480b8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.396+0000 7f1f4cc00700 1 -- 172.21.15.89:0/1025238356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f34009550 con 0x7f1f480d18b0 2024-04-02T09:15:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.397+0000 7f1f4cc00700 1 --2- 172.21.15.89:0/1025238356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f480d18b0 0x7f1f48158c90 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c550 tx=0x7f1f3c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:31.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.397+0000 7f1f44c00700 1 -- 172.21.15.89:0/1025238356 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c004c80 con 0x7f1f480d18b0 2024-04-02T09:15:31.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.397+0000 7f1f44c00700 1 -- 172.21.15.89:0/1025238356 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3c004e00 con 0x7f1f480d18b0 2024-04-02T09:15:31.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.397+0000 7f1f44c00700 1 -- 172.21.15.89:0/1025238356 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c007310 con 0x7f1f480d18b0 2024-04-02T09:15:31.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.397+0000 7f1f4f600700 1 -- 172.21.15.89:0/1025238356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f40009a70 con 0x7f1f480d18b0 2024-04-02T09:15:31.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.397+0000 7f1f4f600700 1 -- 172.21.15.89:0/1025238356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4809f270 con 0x7f1f480d18b0 2024-04-02T09:15:31.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.398+0000 7f1f4f600700 1 -- 172.21.15.89:0/1025238356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f48067720 con 0x7f1f480d18b0 2024-04-02T09:15:31.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.399+0000 7f1f44c00700 1 -- 172.21.15.89:0/1025238356 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1f3c007470 con 0x7f1f480d18b0 2024-04-02T09:15:31.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.400+0000 7f1f44c00700 1 --2- 172.21.15.89:0/1025238356 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1f2806adb0 0x7f1f2806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:31.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.400+0000 7f1f44c00700 1 -- 172.21.15.89:0/1025238356 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f1f3c089250 con 0x7f1f480d18b0 2024-04-02T09:15:31.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.400+0000 7f1f47e00700 1 --2- 172.21.15.89:0/1025238356 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1f2806adb0 0x7f1f2806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:31.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.401+0000 7f1f47e00700 1 --2- 172.21.15.89:0/1025238356 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1f2806adb0 0x7f1f2806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1f480b95c0 tx=0x7f1f340092c0 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:31.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.404+0000 7f1f44c00700 1 -- 172.21.15.89:0/1025238356 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1f3c054160 con 0x7f1f480d18b0 2024-04-02T09:15:31.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.600+0000 7f1f4f600700 1 -- 172.21.15.89:0/1025238356 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1f4809ee20 con 0x7f1f2806adb0 2024-04-02T09:15:31.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.611+0000 7f1f44c00700 1 -- 172.21.15.89:0/1025238356 <== mgr.13065 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289394 (secure 0 0 0) 0x7f1f4809ee20 con 0x7f1f2806adb0 2024-04-02T09:15:31.613 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:31.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.615+0000 7f1f4f600700 1 -- 172.21.15.89:0/1025238356 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1f2806adb0 msgr2=0x7f1f2806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:31.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.615+0000 7f1f4f600700 1 --2- 172.21.15.89:0/1025238356 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1f2806adb0 0x7f1f2806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1f480b95c0 tx=0x7f1f340092c0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.615+0000 7f1f4f600700 1 -- 172.21.15.89:0/1025238356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f480d18b0 msgr2=0x7f1f48158c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:31.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.615+0000 7f1f4f600700 1 --2- 172.21.15.89:0/1025238356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f480d18b0 0x7f1f48158c90 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c550 tx=0x7f1f3c00c860 comp rx=0 tx=0).stop 2024-04-02T09:15:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.615+0000 7f1f4f600700 1 -- 172.21.15.89:0/1025238356 shutdown_connections 2024-04-02T09:15:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.615+0000 7f1f4f600700 1 --2- 172.21.15.89:0/1025238356 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1f48161590 0x7f1f480b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.615+0000 7f1f4f600700 1 --2- 172.21.15.89:0/1025238356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f480d18b0 0x7f1f48158c90 unknown :-1 s=CLOSED pgs=2984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.615+0000 7f1f4f600700 1 --2- 172.21.15.89:0/1025238356 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1f2806adb0 0x7f1f2806d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.615+0000 7f1f4f600700 1 --2- 172.21.15.89:0/1025238356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1f48160f10 0x7f1f480b8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.615+0000 7f1f4f600700 1 -- 172.21.15.89:0/1025238356 >> 172.21.15.89:0/1025238356 conn(0x7f1f4808aff0 msgr2=0x7f1f480ca650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.615+0000 7f1f4f600700 1 -- 172.21.15.89:0/1025238356 shutdown_connections 2024-04-02T09:15:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.615+0000 7f1f4f600700 1 -- 172.21.15.89:0/1025238356 wait complete. 2024-04-02T09:15:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:15:31.632 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-02T09:15:30.613268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":723,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":441,"ondisk_log_size":441,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":424428,"kb_used_data":28640,"kb_used_omap":12,"kb_used_meta":395635,"kb_avail":754550292,"statfs":{"total":773094113280,"available":772659499008,"internally_reserved":0,"allocated":29327360,"data_stored":15645387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":405130808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.350279"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.804800+0000","last_change":"2024-04-02T09:15:26.802021+0000","last_active":"2024-04-02T09:15:26.804800+0000","last_peered":"2024-04-02T09:15:26.804800+0000","last_clean":"2024-04-02T09:15:26.804800+0000","last_became_active":"2024-04-02T09:15:26.801766+0000","last_became_peered":"2024-04-02T09:15:26.801766+0000","last_unstale":"2024-04-02T09:15:26.804800+0000","last_undegraded":"2024-04-02T09:15:26.804800+0000","last_fullsized":"2024-04-02T09:15:26.804800+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:10:44.608892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.792895+0000","last_change":"2024-04-02T09:15:26.785747+0000","last_active":"2024-04-02T09:15:26.792895+0000","last_peered":"2024-04-02T09:15:26.792895+0000","last_clean":"2024-04-02T09:15:26.792895+0000","last_became_active":"2024-04-02T09:15:26.784926+0000","last_became_peered":"2024-04-02T09:15:26.784926+0000","last_unstale":"2024-04-02T09:15:26.792895+0000","last_undegraded":"2024-04-02T09:15:26.792895+0000","last_fullsized":"2024-04-02T09:15:26.792895+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:21:39.563539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.803157+0000","last_change":"2024-04-02T09:15:26.796747+0000","last_active":"2024-04-02T09:15:26.803157+0000","last_peered":"2024-04-02T09:15:26.803157+0000","last_clean":"2024-04-02T09:15:26.803157+0000","last_became_active":"2024-04-02T09:15:26.795819+0000","last_became_peered":"2024-04-02T09:15:26.795819+0000","last_unstale":"2024-04-02T09:15:26.803157+0000","last_undegraded":"2024-04-02T09:15:26.803157+0000","last_fullsized":"2024-04-02T09:15:26.803157+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:21:39.563539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812398+0000","last_change":"2024-04-02T09:15:26.809799+0000","last_active":"2024-04-02T09:15:26.812398+0000","last_peered":"2024-04-02T09:15:26.812398+0000","last_clean":"2024-04-02T09:15:26.812398+0000","last_became_active":"2024-04-02T09:15:26.808211+0000","last_became_peered":"2024-04-02T09:15:26.808211+0000","last_unstale":"2024-04-02T09:15:26.812398+0000","last_undegraded":"2024-04-02T09:15:26.812398+0000","last_fullsized":"2024-04-02T09:15:26.812398+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:18:39.670018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.809486+0000","last_change":"2024-04-02T09:15:26.807096+0000","last_active":"2024-04-02T09:15:26.809486+0000","last_peered":"2024-04-02T09:15:26.809486+0000","last_clean":"2024-04-02T09:15:26.809486+0000","last_became_active":"2024-04-02T09:15:26.806445+0000","last_became_peered":"2024-04-02T09:15:26.806445+0000","last_unstale":"2024-04-02T09:15:26.809486+0000","last_undegraded":"2024-04-02T09:15:26.809486+0000","last_fullsized":"2024-04-02T09:15:26.809486+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:44:27.767171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.794631+0000","last_change":"2024-04-02T09:15:26.789801+0000","last_active":"2024-04-02T09:15:26.794631+0000","last_peered":"2024-04-02T09:15:26.794631+0000","last_clean":"2024-04-02T09:15:26.794631+0000","last_became_active":"2024-04-02T09:15:26.789427+0000","last_became_peered":"2024-04-02T09:15:26.789427+0000","last_unstale":"2024-04-02T09:15:26.794631+0000","last_undegraded":"2024-04-02T09:15:26.794631+0000","last_fullsized":"2024-04-02T09:15:26.794631+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:04:05.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811923+0000","last_change":"2024-04-02T09:15:26.810612+0000","last_active":"2024-04-02T09:15:26.811923+0000","last_peered":"2024-04-02T09:15:26.811923+0000","last_clean":"2024-04-02T09:15:26.811923+0000","last_became_active":"2024-04-02T09:15:26.810338+0000","last_became_peered":"2024-04-02T09:15:26.810338+0000","last_unstale":"2024-04-02T09:15:26.811923+0000","last_undegraded":"2024-04-02T09:15:26.811923+0000","last_fullsized":"2024-04-02T09:15:26.811923+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:08:06.148499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811505+0000","last_change":"2024-04-02T09:15:26.805407+0000","last_active":"2024-04-02T09:15:26.811505+0000","last_peered":"2024-04-02T09:15:26.811505+0000","last_clean":"2024-04-02T09:15:26.811505+0000","last_became_active":"2024-04-02T09:15:26.804705+0000","last_became_peered":"2024-04-02T09:15:26.804705+0000","last_unstale":"2024-04-02T09:15:26.811505+0000","last_undegraded":"2024-04-02T09:15:26.811505+0000","last_fullsized":"2024-04-02T09:15:26.811505+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:15:58.057433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.809707+0000","last_change":"2024-04-02T09:15:26.796245+0000","last_active":"2024-04-02T09:15:26.809707+0000","last_peered":"2024-04-02T09:15:26.809707+0000","last_clean":"2024-04-02T09:15:26.809707+0000","last_became_active":"2024-04-02T09:15:26.795144+0000","last_became_peered":"2024-04-02T09:15:26.795144+0000","last_unstale":"2024-04-02T09:15:26.809707+0000","last_undegraded":"2024-04-02T09:15:26.809707+0000","last_fullsized":"2024-04-02T09:15:26.809707+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:40:11.841909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.789737+0000","last_change":"2024-04-02T09:15:26.786693+0000","last_active":"2024-04-02T09:15:26.789737+0000","last_peered":"2024-04-02T09:15:26.789737+0000","last_clean":"2024-04-02T09:15:26.789737+0000","last_became_active":"2024-04-02T09:15:26.786405+0000","last_became_peered":"2024-04-02T09:15:26.786405+0000","last_unstale":"2024-04-02T09:15:26.789737+0000","last_undegraded":"2024-04-02T09:15:26.789737+0000","last_fullsized":"2024-04-02T09:15:26.789737+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:06.126308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812261+0000","last_change":"2024-04-02T09:15:26.807980+0000","last_active":"2024-04-02T09:15:26.812261+0000","last_peered":"2024-04-02T09:15:26.812261+0000","last_clean":"2024-04-02T09:15:26.812261+0000","last_became_active":"2024-04-02T09:15:26.802932+0000","last_became_peered":"2024-04-02T09:15:26.802932+0000","last_unstale":"2024-04-02T09:15:26.812261+0000","last_undegraded":"2024-04-02T09:15:26.812261+0000","last_fullsized":"2024-04-02T09:15:26.812261+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:10:52.826706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.805240+0000","last_change":"2024-04-02T09:15:26.803334+0000","last_active":"2024-04-02T09:15:26.805240+0000","last_peered":"2024-04-02T09:15:26.805240+0000","last_clean":"2024-04-02T09:15:26.805240+0000","last_became_active":"2024-04-02T09:15:26.803072+0000","last_became_peered":"2024-04-02T09:15:26.803072+0000","last_unstale":"2024-04-02T09:15:26.805240+0000","last_undegraded":"2024-04-02T09:15:26.805240+0000","last_fullsized":"2024-04-02T09:15:26.805240+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:50:49.687799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.797587+0000","last_change":"2024-04-02T09:15:26.796160+0000","last_active":"2024-04-02T09:15:26.797587+0000","last_peered":"2024-04-02T09:15:26.797587+0000","last_clean":"2024-04-02T09:15:26.797587+0000","last_became_active":"2024-04-02T09:15:26.795912+0000","last_became_peered":"2024-04-02T09:15:26.795912+0000","last_unstale":"2024-04-02T09:15:26.797587+0000","last_undegraded":"2024-04-02T09:15:26.797587+0000","last_fullsized":"2024-04-02T09:15:26.797587+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:11:49.759531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.793498+0000","last_change":"2024-04-02T09:15:26.788996+0000","last_active":"2024-04-02T09:15:26.793498+0000","last_peered":"2024-04-02T09:15:26.793498+0000","last_clean":"2024-04-02T09:15:26.793498+0000","last_became_active":"2024-04-02T09:15:26.787458+0000","last_became_peered":"2024-04-02T09:15:26.787458+0000","last_unstale":"2024-04-02T09:15:26.793498+0000","last_undegraded":"2024-04-02T09:15:26.793498+0000","last_fullsized":"2024-04-02T09:15:26.793498+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:16:10.155874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.793516+0000","last_change":"2024-04-02T09:15:26.789348+0000","last_active":"2024-04-02T09:15:26.793516+0000","last_peered":"2024-04-02T09:15:26.793516+0000","last_clean":"2024-04-02T09:15:26.793516+0000","last_became_active":"2024-04-02T09:15:26.789042+0000","last_became_peered":"2024-04-02T09:15:26.789042+0000","last_unstale":"2024-04-02T09:15:26.793516+0000","last_undegraded":"2024-04-02T09:15:26.793516+0000","last_fullsized":"2024-04-02T09:15:26.793516+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:15.459115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.807571+0000","last_change":"2024-04-02T09:15:26.802826+0000","last_active":"2024-04-02T09:15:26.807571+0000","last_peered":"2024-04-02T09:15:26.807571+0000","last_clean":"2024-04-02T09:15:26.807571+0000","last_became_active":"2024-04-02T09:15:26.800730+0000","last_became_peered":"2024-04-02T09:15:26.800730+0000","last_unstale":"2024-04-02T09:15:26.807571+0000","last_undegraded":"2024-04-02T09:15:26.807571+0000","last_fullsized":"2024-04-02T09:15:26.807571+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:00:04.287854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.809934+0000","last_change":"2024-04-02T09:15:26.806992+0000","last_active":"2024-04-02T09:15:26.809934+0000","last_peered":"2024-04-02T09:15:26.809934+0000","last_clean":"2024-04-02T09:15:26.809934+0000","last_became_active":"2024-04-02T09:15:26.805424+0000","last_became_peered":"2024-04-02T09:15:26.805424+0000","last_unstale":"2024-04-02T09:15:26.809934+0000","last_undegraded":"2024-04-02T09:15:26.809934+0000","last_fullsized":"2024-04-02T09:15:26.809934+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:15.900655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.796160+0000","last_change":"2024-04-02T09:15:26.788936+0000","last_active":"2024-04-02T09:15:26.796160+0000","last_peered":"2024-04-02T09:15:26.796160+0000","last_clean":"2024-04-02T09:15:26.796160+0000","last_became_active":"2024-04-02T09:15:26.788699+0000","last_became_peered":"2024-04-02T09:15:26.788699+0000","last_unstale":"2024-04-02T09:15:26.796160+0000","last_undegraded":"2024-04-02T09:15:26.796160+0000","last_fullsized":"2024-04-02T09:15:26.796160+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:41:59.952558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.793942+0000","last_change":"2024-04-02T09:15:26.788194+0000","last_active":"2024-04-02T09:15:26.793942+0000","last_peered":"2024-04-02T09:15:26.793942+0000","last_clean":"2024-04-02T09:15:26.793942+0000","last_became_active":"2024-04-02T09:15:26.787942+0000","last_became_peered":"2024-04-02T09:15:26.787942+0000","last_unstale":"2024-04-02T09:15:26.793942+0000","last_undegraded":"2024-04-02T09:15:26.793942+0000","last_fullsized":"2024-04-02T09:15:26.793942+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:07.926116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.803258+0000","last_change":"2024-04-02T09:15:26.799129+0000","last_active":"2024-04-02T09:15:26.803258+0000","last_peered":"2024-04-02T09:15:26.803258+0000","last_clean":"2024-04-02T09:15:26.803258+0000","last_became_active":"2024-04-02T09:15:26.798383+0000","last_became_peered":"2024-04-02T09:15:26.798383+0000","last_unstale":"2024-04-02T09:15:26.803258+0000","last_undegraded":"2024-04-02T09:15:26.803258+0000","last_fullsized":"2024-04-02T09:15:26.803258+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:04:05.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.798878+0000","last_change":"2024-04-02T09:15:26.788186+0000","last_active":"2024-04-02T09:15:26.798878+0000","last_peered":"2024-04-02T09:15:26.798878+0000","last_clean":"2024-04-02T09:15:26.798878+0000","last_became_active":"2024-04-02T09:15:26.787463+0000","last_became_peered":"2024-04-02T09:15:26.787463+0000","last_unstale":"2024-04-02T09:15:26.798878+0000","last_undegraded":"2024-04-02T09:15:26.798878+0000","last_fullsized":"2024-04-02T09:15:26.798878+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:15:38.697300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811499+0000","last_change":"2024-04-02T09:15:26.806340+0000","last_active":"2024-04-02T09:15:26.811499+0000","last_peered":"2024-04-02T09:15:26.811499+0000","last_clean":"2024-04-02T09:15:26.811499+0000","last_became_active":"2024-04-02T09:15:26.806058+0000","last_became_peered":"2024-04-02T09:15:26.806058+0000","last_unstale":"2024-04-02T09:15:26.811499+0000","last_undegraded":"2024-04-02T09:15:26.811499+0000","last_fullsized":"2024-04-02T09:15:26.811499+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:16.336401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.810518+0000","last_change":"2024-04-02T09:15:26.805203+0000","last_active":"2024-04-02T09:15:26.810518+0000","last_peered":"2024-04-02T09:15:26.810518+0000","last_clean":"2024-04-02T09:15:26.810518+0000","last_became_active":"2024-04-02T09:15:26.804903+0000","last_became_peered":"2024-04-02T09:15:26.804903+0000","last_unstale":"2024-04-02T09:15:26.810518+0000","last_undegraded":"2024-04-02T09:15:26.810518+0000","last_fullsized":"2024-04-02T09:15:26.810518+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:12:30.918942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811328+0000","last_change":"2024-04-02T09:15:26.806543+0000","last_active":"2024-04-02T09:15:26.811328+0000","last_peered":"2024-04-02T09:15:26.811328+0000","last_clean":"2024-04-02T09:15:26.811328+0000","last_became_active":"2024-04-02T09:15:26.802757+0000","last_became_peered":"2024-04-02T09:15:26.802757+0000","last_unstale":"2024-04-02T09:15:26.811328+0000","last_undegraded":"2024-04-02T09:15:26.811328+0000","last_fullsized":"2024-04-02T09:15:26.811328+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:15:58.057433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.802727+0000","last_change":"2024-04-02T09:15:26.801740+0000","last_active":"2024-04-02T09:15:26.802727+0000","last_peered":"2024-04-02T09:15:26.802727+0000","last_clean":"2024-04-02T09:15:26.802727+0000","last_became_active":"2024-04-02T09:15:26.801102+0000","last_became_peered":"2024-04-02T09:15:26.801102+0000","last_unstale":"2024-04-02T09:15:26.802727+0000","last_undegraded":"2024-04-02T09:15:26.802727+0000","last_fullsized":"2024-04-02T09:15:26.802727+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:17:59.811955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.803023+0000","last_change":"2024-04-02T09:15:26.800643+0000","last_active":"2024-04-02T09:15:26.803023+0000","last_peered":"2024-04-02T09:15:26.803023+0000","last_clean":"2024-04-02T09:15:26.803023+0000","last_became_active":"2024-04-02T09:15:26.800383+0000","last_became_peered":"2024-04-02T09:15:26.800383+0000","last_unstale":"2024-04-02T09:15:26.803023+0000","last_undegraded":"2024-04-02T09:15:26.803023+0000","last_fullsized":"2024-04-02T09:15:26.803023+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:30.395362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.795777+0000","last_change":"2024-04-02T09:15:26.791824+0000","last_active":"2024-04-02T09:15:26.795777+0000","last_peered":"2024-04-02T09:15:26.795777+0000","last_clean":"2024-04-02T09:15:26.795777+0000","last_became_active":"2024-04-02T09:15:26.788155+0000","last_became_peered":"2024-04-02T09:15:26.788155+0000","last_unstale":"2024-04-02T09:15:26.795777+0000","last_undegraded":"2024-04-02T09:15:26.795777+0000","last_fullsized":"2024-04-02T09:15:26.795777+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:46.150941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.798900+0000","last_change":"2024-04-02T09:15:26.788373+0000","last_active":"2024-04-02T09:15:26.798900+0000","last_peered":"2024-04-02T09:15:26.798900+0000","last_clean":"2024-04-02T09:15:26.798900+0000","last_became_active":"2024-04-02T09:15:26.787892+0000","last_became_peered":"2024-04-02T09:15:26.787892+0000","last_unstale":"2024-04-02T09:15:26.798900+0000","last_undegraded":"2024-04-02T09:15:26.798900+0000","last_fullsized":"2024-04-02T09:15:26.798900+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:24.606743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.792571+0000","last_change":"2024-04-02T09:15:26.787474+0000","last_active":"2024-04-02T09:15:26.792571+0000","last_peered":"2024-04-02T09:15:26.792571+0000","last_clean":"2024-04-02T09:15:26.792571+0000","last_became_active":"2024-04-02T09:15:26.787219+0000","last_became_peered":"2024-04-02T09:15:26.787219+0000","last_unstale":"2024-04-02T09:15:26.792571+0000","last_undegraded":"2024-04-02T09:15:26.792571+0000","last_fullsized":"2024-04-02T09:15:26.792571+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:25:19.925731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.801005+0000","last_change":"2024-04-02T09:15:26.799255+0000","last_active":"2024-04-02T09:15:26.801005+0000","last_peered":"2024-04-02T09:15:26.801005+0000","last_clean":"2024-04-02T09:15:26.801005+0000","last_became_active":"2024-04-02T09:15:26.798619+0000","last_became_peered":"2024-04-02T09:15:26.798619+0000","last_unstale":"2024-04-02T09:15:26.801005+0000","last_undegraded":"2024-04-02T09:15:26.801005+0000","last_fullsized":"2024-04-02T09:15:26.801005+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:26:39.847255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811983+0000","last_change":"2024-04-02T09:15:26.807667+0000","last_active":"2024-04-02T09:15:26.811983+0000","last_peered":"2024-04-02T09:15:26.811983+0000","last_clean":"2024-04-02T09:15:26.811983+0000","last_became_active":"2024-04-02T09:15:26.806815+0000","last_became_peered":"2024-04-02T09:15:26.806815+0000","last_unstale":"2024-04-02T09:15:26.811983+0000","last_undegraded":"2024-04-02T09:15:26.811983+0000","last_fullsized":"2024-04-02T09:15:26.811983+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:46:34.730490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.804790+0000","last_change":"2024-04-02T09:15:26.802754+0000","last_active":"2024-04-02T09:15:26.804790+0000","last_peered":"2024-04-02T09:15:26.804790+0000","last_clean":"2024-04-02T09:15:26.804790+0000","last_became_active":"2024-04-02T09:15:26.802265+0000","last_became_peered":"2024-04-02T09:15:26.802265+0000","last_unstale":"2024-04-02T09:15:26.804790+0000","last_undegraded":"2024-04-02T09:15:26.804790+0000","last_fullsized":"2024-04-02T09:15:26.804790+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:53.826558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1f","version":"278'4","reported_seq":14,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811089+0000","last_change":"2024-04-02T09:15:26.284648+0000","last_active":"2024-04-02T09:15:26.811089+0000","last_peered":"2024-04-02T09:15:26.811089+0000","last_clean":"2024-04-02T09:15:26.811089+0000","last_became_active":"2024-04-02T09:15:26.284387+0000","last_became_peered":"2024-04-02T09:15:26.284387+0000","last_unstale":"2024-04-02T09:15:26.811089+0000","last_undegraded":"2024-04-02T09:15:26.811089+0000","last_fullsized":"2024-04-02T09:15:26.811089+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:21:21.487014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.796793+0000","last_change":"2024-04-02T09:15:26.795544+0000","last_active":"2024-04-02T09:15:26.796793+0000","last_peered":"2024-04-02T09:15:26.796793+0000","last_clean":"2024-04-02T09:15:26.796793+0000","last_became_active":"2024-04-02T09:15:26.794302+0000","last_became_peered":"2024-04-02T09:15:26.794302+0000","last_unstale":"2024-04-02T09:15:26.796793+0000","last_undegraded":"2024-04-02T09:15:26.796793+0000","last_fullsized":"2024-04-02T09:15:26.796793+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:30:49.865298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1e","version":"278'1","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812290+0000","last_change":"2024-04-02T09:15:26.285252+0000","last_active":"2024-04-02T09:15:26.812290+0000","last_peered":"2024-04-02T09:15:26.812290+0000","last_clean":"2024-04-02T09:15:26.812290+0000","last_became_active":"2024-04-02T09:15:26.284893+0000","last_became_peered":"2024-04-02T09:15:26.284893+0000","last_unstale":"2024-04-02T09:15:26.812290+0000","last_undegraded":"2024-04-02T09:15:26.812290+0000","last_fullsized":"2024-04-02T09:15:26.812290+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:51:06.977972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811676+0000","last_change":"2024-04-02T09:15:26.802849+0000","last_active":"2024-04-02T09:15:26.811676+0000","last_peered":"2024-04-02T09:15:26.811676+0000","last_clean":"2024-04-02T09:15:26.811676+0000","last_became_active":"2024-04-02T09:15:26.801678+0000","last_became_peered":"2024-04-02T09:15:26.801678+0000","last_unstale":"2024-04-02T09:15:26.811676+0000","last_undegraded":"2024-04-02T09:15:26.811676+0000","last_fullsized":"2024-04-02T09:15:26.811676+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:22:40.762308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"278'2","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.810022+0000","last_change":"2024-04-02T09:15:26.285953+0000","last_active":"2024-04-02T09:15:26.810022+0000","last_peered":"2024-04-02T09:15:26.810022+0000","last_clean":"2024-04-02T09:15:26.810022+0000","last_became_active":"2024-04-02T09:15:26.285628+0000","last_became_peered":"2024-04-02T09:15:26.285628+0000","last_unstale":"2024-04-02T09:15:26.810022+0000","last_undegraded":"2024-04-02T09:15:26.810022+0000","last_fullsized":"2024-04-02T09:15:26.810022+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:53.495029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.797688+0000","last_change":"2024-04-02T09:15:26.791532+0000","last_active":"2024-04-02T09:15:26.797688+0000","last_peered":"2024-04-02T09:15:26.797688+0000","last_clean":"2024-04-02T09:15:26.797688+0000","last_became_active":"2024-04-02T09:15:26.791239+0000","last_became_peered":"2024-04-02T09:15:26.791239+0000","last_unstale":"2024-04-02T09:15:26.797688+0000","last_undegraded":"2024-04-02T09:15:26.797688+0000","last_fullsized":"2024-04-02T09:15:26.797688+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:06.126308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1c","version":"278'2","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.806677+0000","last_change":"2024-04-02T09:15:26.283742+0000","last_active":"2024-04-02T09:15:26.806677+0000","last_peered":"2024-04-02T09:15:26.806677+0000","last_clean":"2024-04-02T09:15:26.806677+0000","last_became_active":"2024-04-02T09:15:26.283332+0000","last_became_peered":"2024-04-02T09:15:26.283332+0000","last_unstale":"2024-04-02T09:15:26.806677+0000","last_undegraded":"2024-04-02T09:15:26.806677+0000","last_fullsized":"2024-04-02T09:15:26.806677+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:14:41.653521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.802995+0000","last_change":"2024-04-02T09:15:26.798223+0000","last_active":"2024-04-02T09:15:26.802995+0000","last_peered":"2024-04-02T09:15:26.802995+0000","last_clean":"2024-04-02T09:15:26.802995+0000","last_became_active":"2024-04-02T09:15:26.797936+0000","last_became_peered":"2024-04-02T09:15:26.797936+0000","last_unstale":"2024-04-02T09:15:26.802995+0000","last_undegraded":"2024-04-02T09:15:26.802995+0000","last_fullsized":"2024-04-02T09:15:26.802995+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:15:38.697300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1b","version":"278'1","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.806850+0000","last_change":"2024-04-02T09:15:26.291916+0000","last_active":"2024-04-02T09:15:26.806850+0000","last_peered":"2024-04-02T09:15:26.806850+0000","last_clean":"2024-04-02T09:15:26.806850+0000","last_became_active":"2024-04-02T09:15:26.291463+0000","last_became_peered":"2024-04-02T09:15:26.291463+0000","last_unstale":"2024-04-02T09:15:26.806850+0000","last_undegraded":"2024-04-02T09:15:26.806850+0000","last_fullsized":"2024-04-02T09:15:26.806850+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:16:09.150518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.809505+0000","last_change":"2024-04-02T09:15:26.807350+0000","last_active":"2024-04-02T09:15:26.809505+0000","last_peered":"2024-04-02T09:15:26.809505+0000","last_clean":"2024-04-02T09:15:26.809505+0000","last_became_active":"2024-04-02T09:15:26.806848+0000","last_became_peered":"2024-04-02T09:15:26.806848+0000","last_unstale":"2024-04-02T09:15:26.809505+0000","last_undegraded":"2024-04-02T09:15:26.809505+0000","last_fullsized":"2024-04-02T09:15:26.809505+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:24.606743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1a","version":"278'1","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.793785+0000","last_change":"2024-04-02T09:15:26.290746+0000","last_active":"2024-04-02T09:15:26.793785+0000","last_peered":"2024-04-02T09:15:26.793785+0000","last_clean":"2024-04-02T09:15:26.793785+0000","last_became_active":"2024-04-02T09:15:26.286464+0000","last_became_peered":"2024-04-02T09:15:26.286464+0000","last_unstale":"2024-04-02T09:15:26.793785+0000","last_undegraded":"2024-04-02T09:15:26.793785+0000","last_fullsized":"2024-04-02T09:15:26.793785+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:02:00.543263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811348+0000","last_change":"2024-04-02T09:15:26.807360+0000","last_active":"2024-04-02T09:15:26.811348+0000","last_peered":"2024-04-02T09:15:26.811348+0000","last_clean":"2024-04-02T09:15:26.811348+0000","last_became_active":"2024-04-02T09:15:26.805796+0000","last_became_peered":"2024-04-02T09:15:26.805796+0000","last_unstale":"2024-04-02T09:15:26.811348+0000","last_undegraded":"2024-04-02T09:15:26.811348+0000","last_fullsized":"2024-04-02T09:15:26.811348+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:18:39.670018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.791937+0000","last_change":"2024-04-02T09:15:26.292532+0000","last_active":"2024-04-02T09:15:26.791937+0000","last_peered":"2024-04-02T09:15:26.791937+0000","last_clean":"2024-04-02T09:15:26.791937+0000","last_became_active":"2024-04-02T09:15:26.292118+0000","last_became_peered":"2024-04-02T09:15:26.292118+0000","last_unstale":"2024-04-02T09:15:26.791937+0000","last_undegraded":"2024-04-02T09:15:26.791937+0000","last_fullsized":"2024-04-02T09:15:26.791937+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:45.145585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812280+0000","last_change":"2024-04-02T09:15:26.803471+0000","last_active":"2024-04-02T09:15:26.812280+0000","last_peered":"2024-04-02T09:15:26.812280+0000","last_clean":"2024-04-02T09:15:26.812280+0000","last_became_active":"2024-04-02T09:15:26.801873+0000","last_became_peered":"2024-04-02T09:15:26.801873+0000","last_unstale":"2024-04-02T09:15:26.812280+0000","last_undegraded":"2024-04-02T09:15:26.812280+0000","last_fullsized":"2024-04-02T09:15:26.812280+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:12:30.918942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.780729+0000","last_change":"2024-04-02T09:15:26.287003+0000","last_active":"2024-04-02T09:15:26.780729+0000","last_peered":"2024-04-02T09:15:26.780729+0000","last_clean":"2024-04-02T09:15:26.780729+0000","last_became_active":"2024-04-02T09:15:26.286719+0000","last_became_peered":"2024-04-02T09:15:26.286719+0000","last_unstale":"2024-04-02T09:15:26.780729+0000","last_undegraded":"2024-04-02T09:15:26.780729+0000","last_fullsized":"2024-04-02T09:15:26.780729+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:32:07.104940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.803539+0000","last_change":"2024-04-02T09:15:26.799013+0000","last_active":"2024-04-02T09:15:26.803539+0000","last_peered":"2024-04-02T09:15:26.803539+0000","last_clean":"2024-04-02T09:15:26.803539+0000","last_became_active":"2024-04-02T09:15:26.798762+0000","last_became_peered":"2024-04-02T09:15:26.798762+0000","last_unstale":"2024-04-02T09:15:26.803539+0000","last_undegraded":"2024-04-02T09:15:26.803539+0000","last_fullsized":"2024-04-02T09:15:26.803539+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:40:17.423928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.783700+0000","last_change":"2024-04-02T09:15:26.278451+0000","last_active":"2024-04-02T09:15:26.783700+0000","last_peered":"2024-04-02T09:15:26.783700+0000","last_clean":"2024-04-02T09:15:26.783700+0000","last_became_active":"2024-04-02T09:15:26.278150+0000","last_became_peered":"2024-04-02T09:15:26.278150+0000","last_unstale":"2024-04-02T09:15:26.783700+0000","last_undegraded":"2024-04-02T09:15:26.783700+0000","last_fullsized":"2024-04-02T09:15:26.783700+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:29:33.555261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.792390+0000","last_change":"2024-04-02T09:15:26.786896+0000","last_active":"2024-04-02T09:15:26.792390+0000","last_peered":"2024-04-02T09:15:26.792390+0000","last_clean":"2024-04-02T09:15:26.792390+0000","last_became_active":"2024-04-02T09:15:26.786634+0000","last_became_peered":"2024-04-02T09:15:26.786634+0000","last_unstale":"2024-04-02T09:15:26.792390+0000","last_undegraded":"2024-04-02T09:15:26.792390+0000","last_fullsized":"2024-04-02T09:15:26.792390+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:44:05.340509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.776745+0000","last_change":"2024-04-02T09:15:26.288958+0000","last_active":"2024-04-02T09:15:26.776745+0000","last_peered":"2024-04-02T09:15:26.776745+0000","last_clean":"2024-04-02T09:15:26.776745+0000","last_became_active":"2024-04-02T09:15:26.288713+0000","last_became_peered":"2024-04-02T09:15:26.288713+0000","last_unstale":"2024-04-02T09:15:26.776745+0000","last_undegraded":"2024-04-02T09:15:26.776745+0000","last_fullsized":"2024-04-02T09:15:26.776745+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:04:15.823260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.798491+0000","last_change":"2024-04-02T09:15:26.793442+0000","last_active":"2024-04-02T09:15:26.798491+0000","last_peered":"2024-04-02T09:15:26.798491+0000","last_clean":"2024-04-02T09:15:26.798491+0000","last_became_active":"2024-04-02T09:15:26.792538+0000","last_became_peered":"2024-04-02T09:15:26.792538+0000","last_unstale":"2024-04-02T09:15:26.798491+0000","last_undegraded":"2024-04-02T09:15:26.798491+0000","last_fullsized":"2024-04-02T09:15:26.798491+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:03:29.044095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.15","version":"278'3","reported_seq":13,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.808899+0000","last_change":"2024-04-02T09:15:26.281980+0000","last_active":"2024-04-02T09:15:26.808899+0000","last_peered":"2024-04-02T09:15:26.808899+0000","last_clean":"2024-04-02T09:15:26.808899+0000","last_became_active":"2024-04-02T09:15:26.281710+0000","last_became_peered":"2024-04-02T09:15:26.281710+0000","last_unstale":"2024-04-02T09:15:26.808899+0000","last_undegraded":"2024-04-02T09:15:26.808899+0000","last_fullsized":"2024-04-02T09:15:26.808899+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:18:21.078476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.803906+0000","last_change":"2024-04-02T09:15:26.795494+0000","last_active":"2024-04-02T09:15:26.803906+0000","last_peered":"2024-04-02T09:15:26.803906+0000","last_clean":"2024-04-02T09:15:26.803906+0000","last_became_active":"2024-04-02T09:15:26.795201+0000","last_became_peered":"2024-04-02T09:15:26.795201+0000","last_unstale":"2024-04-02T09:15:26.803906+0000","last_undegraded":"2024-04-02T09:15:26.803906+0000","last_fullsized":"2024-04-02T09:15:26.803906+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:45.732192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.14","version":"278'6","reported_seq":16,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.813574+0000","last_change":"2024-04-02T09:15:26.292478+0000","last_active":"2024-04-02T09:15:26.813574+0000","last_peered":"2024-04-02T09:15:26.813574+0000","last_clean":"2024-04-02T09:15:26.813574+0000","last_became_active":"2024-04-02T09:15:26.292060+0000","last_became_peered":"2024-04-02T09:15:26.292060+0000","last_unstale":"2024-04-02T09:15:26.813574+0000","last_undegraded":"2024-04-02T09:15:26.813574+0000","last_fullsized":"2024-04-02T09:15:26.813574+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:11:48.754175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.796939+0000","last_change":"2024-04-02T09:15:26.789695+0000","last_active":"2024-04-02T09:15:26.796939+0000","last_peered":"2024-04-02T09:15:26.796939+0000","last_clean":"2024-04-02T09:15:26.796939+0000","last_became_active":"2024-04-02T09:15:26.789419+0000","last_became_peered":"2024-04-02T09:15:26.789419+0000","last_unstale":"2024-04-02T09:15:26.796939+0000","last_undegraded":"2024-04-02T09:15:26.796939+0000","last_fullsized":"2024-04-02T09:15:26.796939+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:41:06.039590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.13","version":"278'2","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812576+0000","last_change":"2024-04-02T09:15:26.286966+0000","last_active":"2024-04-02T09:15:26.812576+0000","last_peered":"2024-04-02T09:15:26.812576+0000","last_clean":"2024-04-02T09:15:26.812576+0000","last_became_active":"2024-04-02T09:15:26.286699+0000","last_became_peered":"2024-04-02T09:15:26.286699+0000","last_unstale":"2024-04-02T09:15:26.812576+0000","last_undegraded":"2024-04-02T09:15:26.812576+0000","last_fullsized":"2024-04-02T09:15:26.812576+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:29:05.383512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.793196+0000","last_change":"2024-04-02T09:15:26.787881+0000","last_active":"2024-04-02T09:15:26.793196+0000","last_peered":"2024-04-02T09:15:26.793196+0000","last_clean":"2024-04-02T09:15:26.793196+0000","last_became_active":"2024-04-02T09:15:26.787595+0000","last_became_peered":"2024-04-02T09:15:26.787595+0000","last_unstale":"2024-04-02T09:15:26.793196+0000","last_undegraded":"2024-04-02T09:15:26.793196+0000","last_fullsized":"2024-04-02T09:15:26.793196+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:09:17.511200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.794749+0000","last_change":"2024-04-02T09:15:26.288973+0000","last_active":"2024-04-02T09:15:26.794749+0000","last_peered":"2024-04-02T09:15:26.794749+0000","last_clean":"2024-04-02T09:15:26.794749+0000","last_became_active":"2024-04-02T09:15:26.288716+0000","last_became_peered":"2024-04-02T09:15:26.288716+0000","last_unstale":"2024-04-02T09:15:26.794749+0000","last_undegraded":"2024-04-02T09:15:26.794749+0000","last_fullsized":"2024-04-02T09:15:26.794749+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:06.920760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811363+0000","last_change":"2024-04-02T09:15:26.807483+0000","last_active":"2024-04-02T09:15:26.811363+0000","last_peered":"2024-04-02T09:15:26.811363+0000","last_clean":"2024-04-02T09:15:26.811363+0000","last_became_active":"2024-04-02T09:15:26.807121+0000","last_became_peered":"2024-04-02T09:15:26.807121+0000","last_unstale":"2024-04-02T09:15:26.811363+0000","last_undegraded":"2024-04-02T09:15:26.811363+0000","last_fullsized":"2024-04-02T09:15:26.811363+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:52:47.662567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.781582+0000","last_change":"2024-04-02T09:15:26.290229+0000","last_active":"2024-04-02T09:15:26.781582+0000","last_peered":"2024-04-02T09:15:26.781582+0000","last_clean":"2024-04-02T09:15:26.781582+0000","last_became_active":"2024-04-02T09:15:26.289273+0000","last_became_peered":"2024-04-02T09:15:26.289273+0000","last_unstale":"2024-04-02T09:15:26.781582+0000","last_undegraded":"2024-04-02T09:15:26.781582+0000","last_fullsized":"2024-04-02T09:15:26.781582+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:40:16.418572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.810538+0000","last_change":"2024-04-02T09:15:26.808563+0000","last_active":"2024-04-02T09:15:26.810538+0000","last_peered":"2024-04-02T09:15:26.810538+0000","last_clean":"2024-04-02T09:15:26.810538+0000","last_became_active":"2024-04-02T09:15:26.808283+0000","last_became_peered":"2024-04-02T09:15:26.808283+0000","last_unstale":"2024-04-02T09:15:26.810538+0000","last_undegraded":"2024-04-02T09:15:26.810538+0000","last_fullsized":"2024-04-02T09:15:26.810538+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:46:34.730490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.10","version":"278'8","reported_seq":18,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812527+0000","last_change":"2024-04-02T09:15:26.289161+0000","last_active":"2024-04-02T09:15:26.812527+0000","last_peered":"2024-04-02T09:15:26.812527+0000","last_clean":"2024-04-02T09:15:26.812527+0000","last_became_active":"2024-04-02T09:15:26.288231+0000","last_became_peered":"2024-04-02T09:15:26.288231+0000","last_unstale":"2024-04-02T09:15:26.812527+0000","last_undegraded":"2024-04-02T09:15:26.812527+0000","last_fullsized":"2024-04-02T09:15:26.812527+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:49:29.546689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.799109+0000","last_change":"2024-04-02T09:15:26.795008+0000","last_active":"2024-04-02T09:15:26.799109+0000","last_peered":"2024-04-02T09:15:26.799109+0000","last_clean":"2024-04-02T09:15:26.799109+0000","last_became_active":"2024-04-02T09:15:26.794739+0000","last_became_peered":"2024-04-02T09:15:26.794739+0000","last_unstale":"2024-04-02T09:15:26.799109+0000","last_undegraded":"2024-04-02T09:15:26.799109+0000","last_fullsized":"2024-04-02T09:15:26.799109+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:45.732192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.f","version":"278'1","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.808833+0000","last_change":"2024-04-02T09:15:26.282846+0000","last_active":"2024-04-02T09:15:26.808833+0000","last_peered":"2024-04-02T09:15:26.808833+0000","last_clean":"2024-04-02T09:15:26.808833+0000","last_became_active":"2024-04-02T09:15:26.281867+0000","last_became_peered":"2024-04-02T09:15:26.281867+0000","last_unstale":"2024-04-02T09:15:26.808833+0000","last_undegraded":"2024-04-02T09:15:26.808833+0000","last_fullsized":"2024-04-02T09:15:26.808833+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:49:51.579415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811777+0000","last_change":"2024-04-02T09:15:26.810130+0000","last_active":"2024-04-02T09:15:26.811777+0000","last_peered":"2024-04-02T09:15:26.811777+0000","last_clean":"2024-04-02T09:15:26.811777+0000","last_became_active":"2024-04-02T09:15:26.809019+0000","last_became_peered":"2024-04-02T09:15:26.809019+0000","last_unstale":"2024-04-02T09:15:26.811777+0000","last_undegraded":"2024-04-02T09:15:26.811777+0000","last_fullsized":"2024-04-02T09:15:26.811777+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:23:03.344490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.e","version":"278'2","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.808322+0000","last_change":"2024-04-02T09:15:26.291584+0000","last_active":"2024-04-02T09:15:26.808322+0000","last_peered":"2024-04-02T09:15:26.808322+0000","last_clean":"2024-04-02T09:15:26.808322+0000","last_became_active":"2024-04-02T09:15:26.290637+0000","last_became_peered":"2024-04-02T09:15:26.290637+0000","last_unstale":"2024-04-02T09:15:26.808322+0000","last_undegraded":"2024-04-02T09:15:26.808322+0000","last_fullsized":"2024-04-02T09:15:26.808322+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:17:55.858050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812232+0000","last_change":"2024-04-02T09:15:26.805111+0000","last_active":"2024-04-02T09:15:26.812232+0000","last_peered":"2024-04-02T09:15:26.812232+0000","last_clean":"2024-04-02T09:15:26.812232+0000","last_became_active":"2024-04-02T09:15:26.803334+0000","last_became_peered":"2024-04-02T09:15:26.803334+0000","last_unstale":"2024-04-02T09:15:26.812232+0000","last_undegraded":"2024-04-02T09:15:26.812232+0000","last_fullsized":"2024-04-02T09:15:26.812232+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:59:02.406432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.d","version":"278'1","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.809946+0000","last_change":"2024-04-02T09:15:26.284703+0000","last_active":"2024-04-02T09:15:26.809946+0000","last_peered":"2024-04-02T09:15:26.809946+0000","last_clean":"2024-04-02T09:15:26.809946+0000","last_became_active":"2024-04-02T09:15:26.284431+0000","last_became_peered":"2024-04-02T09:15:26.284431+0000","last_unstale":"2024-04-02T09:15:26.809946+0000","last_undegraded":"2024-04-02T09:15:26.809946+0000","last_fullsized":"2024-04-02T09:15:26.809946+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:58.625853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.810493+0000","last_change":"2024-04-02T09:15:26.808276+0000","last_active":"2024-04-02T09:15:26.810493+0000","last_peered":"2024-04-02T09:15:26.810493+0000","last_clean":"2024-04-02T09:15:26.810493+0000","last_became_active":"2024-04-02T09:15:26.807091+0000","last_became_peered":"2024-04-02T09:15:26.807091+0000","last_unstale":"2024-04-02T09:15:26.810493+0000","last_undegraded":"2024-04-02T09:15:26.810493+0000","last_fullsized":"2024-04-02T09:15:26.810493+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:45.951714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.786858+0000","last_change":"2024-04-02T09:15:26.288044+0000","last_active":"2024-04-02T09:15:26.786858+0000","last_peered":"2024-04-02T09:15:26.786858+0000","last_clean":"2024-04-02T09:15:26.786858+0000","last_became_active":"2024-04-02T09:15:26.287498+0000","last_became_peered":"2024-04-02T09:15:26.287498+0000","last_unstale":"2024-04-02T09:15:26.786858+0000","last_undegraded":"2024-04-02T09:15:26.786858+0000","last_fullsized":"2024-04-02T09:15:26.786858+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:02:32.179689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.793760+0000","last_change":"2024-04-02T09:15:26.786372+0000","last_active":"2024-04-02T09:15:26.793760+0000","last_peered":"2024-04-02T09:15:26.793760+0000","last_clean":"2024-04-02T09:15:26.793760+0000","last_became_active":"2024-04-02T09:15:26.786126+0000","last_became_peered":"2024-04-02T09:15:26.786126+0000","last_unstale":"2024-04-02T09:15:26.793760+0000","last_undegraded":"2024-04-02T09:15:26.793760+0000","last_fullsized":"2024-04-02T09:15:26.793760+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:07:01.749821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.b","version":"278'2","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.810397+0000","last_change":"2024-04-02T09:15:26.288142+0000","last_active":"2024-04-02T09:15:26.810397+0000","last_peered":"2024-04-02T09:15:26.810397+0000","last_clean":"2024-04-02T09:15:26.810397+0000","last_became_active":"2024-04-02T09:15:26.287881+0000","last_became_peered":"2024-04-02T09:15:26.287881+0000","last_unstale":"2024-04-02T09:15:26.810397+0000","last_undegraded":"2024-04-02T09:15:26.810397+0000","last_fullsized":"2024-04-02T09:15:26.810397+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:12:43.305299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812300+0000","last_change":"2024-04-02T09:15:26.801996+0000","last_active":"2024-04-02T09:15:26.812300+0000","last_peered":"2024-04-02T09:15:26.812300+0000","last_clean":"2024-04-02T09:15:26.812300+0000","last_became_active":"2024-04-02T09:15:26.801743+0000","last_became_peered":"2024-04-02T09:15:26.801743+0000","last_unstale":"2024-04-02T09:15:26.812300+0000","last_undegraded":"2024-04-02T09:15:26.812300+0000","last_fullsized":"2024-04-02T09:15:26.812300+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:28:28.895276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.786769+0000","last_change":"2024-04-02T09:15:26.284022+0000","last_active":"2024-04-02T09:15:26.786769+0000","last_peered":"2024-04-02T09:15:26.786769+0000","last_clean":"2024-04-02T09:15:26.786769+0000","last_became_active":"2024-04-02T09:15:26.283734+0000","last_became_peered":"2024-04-02T09:15:26.283734+0000","last_unstale":"2024-04-02T09:15:26.786769+0000","last_undegraded":"2024-04-02T09:15:26.786769+0000","last_fullsized":"2024-04-02T09:15:26.786769+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:11:16.738763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.806228+0000","last_change":"2024-04-02T09:15:26.804380+0000","last_active":"2024-04-02T09:15:26.806228+0000","last_peered":"2024-04-02T09:15:26.806228+0000","last_clean":"2024-04-02T09:15:26.806228+0000","last_became_active":"2024-04-02T09:15:26.804080+0000","last_became_peered":"2024-04-02T09:15:26.804080+0000","last_unstale":"2024-04-02T09:15:26.806228+0000","last_undegraded":"2024-04-02T09:15:26.806228+0000","last_fullsized":"2024-04-02T09:15:26.806228+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:32:08.110296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.785833+0000","last_change":"2024-04-02T09:15:26.282673+0000","last_active":"2024-04-02T09:15:26.785833+0000","last_peered":"2024-04-02T09:15:26.785833+0000","last_clean":"2024-04-02T09:15:26.785833+0000","last_became_active":"2024-04-02T09:15:26.282410+0000","last_became_peered":"2024-04-02T09:15:26.282410+0000","last_unstale":"2024-04-02T09:15:26.785833+0000","last_undegraded":"2024-04-02T09:15:26.785833+0000","last_fullsized":"2024-04-02T09:15:26.785833+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:15:13.448698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.813838+0000","last_change":"2024-04-02T09:15:26.810951+0000","last_active":"2024-04-02T09:15:26.813838+0000","last_peered":"2024-04-02T09:15:26.813838+0000","last_clean":"2024-04-02T09:15:26.813838+0000","last_became_active":"2024-04-02T09:15:26.810327+0000","last_became_peered":"2024-04-02T09:15:26.810327+0000","last_unstale":"2024-04-02T09:15:26.813838+0000","last_undegraded":"2024-04-02T09:15:26.813838+0000","last_fullsized":"2024-04-02T09:15:26.813838+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:04.644247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.781029+0000","last_change":"2024-04-02T09:15:26.286166+0000","last_active":"2024-04-02T09:15:26.781029+0000","last_peered":"2024-04-02T09:15:26.781029+0000","last_clean":"2024-04-02T09:15:26.781029+0000","last_became_active":"2024-04-02T09:15:26.285895+0000","last_became_peered":"2024-04-02T09:15:26.285895+0000","last_unstale":"2024-04-02T09:15:26.781029+0000","last_undegraded":"2024-04-02T09:15:26.781029+0000","last_fullsized":"2024-04-02T09:15:26.781029+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:03:28.038739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812391+0000","last_change":"2024-04-02T09:15:26.809682+0000","last_active":"2024-04-02T09:15:26.812391+0000","last_peered":"2024-04-02T09:15:26.812391+0000","last_clean":"2024-04-02T09:15:26.812391+0000","last_became_active":"2024-04-02T09:15:26.805991+0000","last_became_peered":"2024-04-02T09:15:26.805991+0000","last_unstale":"2024-04-02T09:15:26.812391+0000","last_undegraded":"2024-04-02T09:15:26.812391+0000","last_fullsized":"2024-04-02T09:15:26.812391+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:06:53.885091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.7","version":"278'2","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811891+0000","last_change":"2024-04-02T09:15:26.285684+0000","last_active":"2024-04-02T09:15:26.811891+0000","last_peered":"2024-04-02T09:15:26.811891+0000","last_clean":"2024-04-02T09:15:26.811891+0000","last_became_active":"2024-04-02T09:15:26.284959+0000","last_became_peered":"2024-04-02T09:15:26.284959+0000","last_unstale":"2024-04-02T09:15:26.811891+0000","last_undegraded":"2024-04-02T09:15:26.811891+0000","last_fullsized":"2024-04-02T09:15:26.811891+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:57:19.421726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.792372+0000","last_change":"2024-04-02T09:15:26.785967+0000","last_active":"2024-04-02T09:15:26.792372+0000","last_peered":"2024-04-02T09:15:26.792372+0000","last_clean":"2024-04-02T09:15:26.792372+0000","last_became_active":"2024-04-02T09:15:26.785713+0000","last_became_peered":"2024-04-02T09:15:26.785713+0000","last_unstale":"2024-04-02T09:15:26.792372+0000","last_undegraded":"2024-04-02T09:15:26.792372+0000","last_fullsized":"2024-04-02T09:15:26.792372+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:20.157837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.783421+0000","last_change":"2024-04-02T09:15:26.289438+0000","last_active":"2024-04-02T09:15:26.783421+0000","last_peered":"2024-04-02T09:15:26.783421+0000","last_clean":"2024-04-02T09:15:26.783421+0000","last_became_active":"2024-04-02T09:15:26.289147+0000","last_became_peered":"2024-04-02T09:15:26.289147+0000","last_unstale":"2024-04-02T09:15:26.783421+0000","last_undegraded":"2024-04-02T09:15:26.783421+0000","last_fullsized":"2024-04-02T09:15:26.783421+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:12.082097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.809186+0000","last_change":"2024-04-02T09:15:26.806637+0000","last_active":"2024-04-02T09:15:26.809186+0000","last_peered":"2024-04-02T09:15:26.809186+0000","last_clean":"2024-04-02T09:15:26.809186+0000","last_became_active":"2024-04-02T09:15:26.804177+0000","last_became_peered":"2024-04-02T09:15:26.804177+0000","last_unstale":"2024-04-02T09:15:26.809186+0000","last_undegraded":"2024-04-02T09:15:26.809186+0000","last_fullsized":"2024-04-02T09:15:26.809186+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:48:46.501785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.781162+0000","last_change":"2024-04-02T09:15:26.290231+0000","last_active":"2024-04-02T09:15:26.781162+0000","last_peered":"2024-04-02T09:15:26.781162+0000","last_clean":"2024-04-02T09:15:26.781162+0000","last_became_active":"2024-04-02T09:15:26.289007+0000","last_became_peered":"2024-04-02T09:15:26.289007+0000","last_unstale":"2024-04-02T09:15:26.781162+0000","last_undegraded":"2024-04-02T09:15:26.781162+0000","last_fullsized":"2024-04-02T09:15:26.781162+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:44.726836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.796255+0000","last_change":"2024-04-02T09:15:26.793838+0000","last_active":"2024-04-02T09:15:26.796255+0000","last_peered":"2024-04-02T09:15:26.796255+0000","last_clean":"2024-04-02T09:15:26.796255+0000","last_became_active":"2024-04-02T09:15:26.791718+0000","last_became_peered":"2024-04-02T09:15:26.791718+0000","last_unstale":"2024-04-02T09:15:26.796255+0000","last_undegraded":"2024-04-02T09:15:26.796255+0000","last_fullsized":"2024-04-02T09:15:26.796255+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:17:56.863406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.780926+0000","last_change":"2024-04-02T09:15:26.281559+0000","last_active":"2024-04-02T09:15:26.780926+0000","last_peered":"2024-04-02T09:15:26.780926+0000","last_clean":"2024-04-02T09:15:26.780926+0000","last_became_active":"2024-04-02T09:15:26.280091+0000","last_became_peered":"2024-04-02T09:15:26.280091+0000","last_unstale":"2024-04-02T09:15:26.780926+0000","last_undegraded":"2024-04-02T09:15:26.780926+0000","last_fullsized":"2024-04-02T09:15:26.780926+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:00.549782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.803695+0000","last_change":"2024-04-02T09:15:26.795720+0000","last_active":"2024-04-02T09:15:26.803695+0000","last_peered":"2024-04-02T09:15:26.803695+0000","last_clean":"2024-04-02T09:15:26.803695+0000","last_became_active":"2024-04-02T09:15:26.795426+0000","last_became_peered":"2024-04-02T09:15:26.795426+0000","last_unstale":"2024-04-02T09:15:26.803695+0000","last_undegraded":"2024-04-02T09:15:26.803695+0000","last_fullsized":"2024-04-02T09:15:26.803695+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:52:35.854324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.790787+0000","last_change":"2024-04-02T09:15:26.287978+0000","last_active":"2024-04-02T09:15:26.790787+0000","last_peered":"2024-04-02T09:15:26.790787+0000","last_clean":"2024-04-02T09:15:26.790787+0000","last_became_active":"2024-04-02T09:15:26.287720+0000","last_became_peered":"2024-04-02T09:15:26.287720+0000","last_unstale":"2024-04-02T09:15:26.790787+0000","last_undegraded":"2024-04-02T09:15:26.790787+0000","last_fullsized":"2024-04-02T09:15:26.790787+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:19.152481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.805618+0000","last_change":"2024-04-02T09:15:26.802648+0000","last_active":"2024-04-02T09:15:26.805618+0000","last_peered":"2024-04-02T09:15:26.805618+0000","last_clean":"2024-04-02T09:15:26.805618+0000","last_became_active":"2024-04-02T09:15:26.802312+0000","last_became_peered":"2024-04-02T09:15:26.802312+0000","last_unstale":"2024-04-02T09:15:26.805618+0000","last_undegraded":"2024-04-02T09:15:26.805618+0000","last_fullsized":"2024-04-02T09:15:26.805618+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:29:34.560617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.794740+0000","last_change":"2024-04-02T09:15:26.292409+0000","last_active":"2024-04-02T09:15:26.794740+0000","last_peered":"2024-04-02T09:15:26.794740+0000","last_clean":"2024-04-02T09:15:26.794740+0000","last_became_active":"2024-04-02T09:15:26.292132+0000","last_became_peered":"2024-04-02T09:15:26.292132+0000","last_unstale":"2024-04-02T09:15:26.794740+0000","last_undegraded":"2024-04-02T09:15:26.794740+0000","last_fullsized":"2024-04-02T09:15:26.794740+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:52:14.862034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.803049+0000","last_change":"2024-04-02T09:15:26.797299+0000","last_active":"2024-04-02T09:15:26.803049+0000","last_peered":"2024-04-02T09:15:26.803049+0000","last_clean":"2024-04-02T09:15:26.803049+0000","last_became_active":"2024-04-02T09:15:26.797032+0000","last_became_peered":"2024-04-02T09:15:26.797032+0000","last_unstale":"2024-04-02T09:15:26.803049+0000","last_undegraded":"2024-04-02T09:15:26.803049+0000","last_fullsized":"2024-04-02T09:15:26.803049+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:01.555138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.792039+0000","last_change":"2024-04-02T09:15:26.289014+0000","last_active":"2024-04-02T09:15:26.792039+0000","last_peered":"2024-04-02T09:15:26.792039+0000","last_clean":"2024-04-02T09:15:26.792039+0000","last_became_active":"2024-04-02T09:15:26.288766+0000","last_became_peered":"2024-04-02T09:15:26.288766+0000","last_unstale":"2024-04-02T09:15:26.792039+0000","last_undegraded":"2024-04-02T09:15:26.792039+0000","last_fullsized":"2024-04-02T09:15:26.792039+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:25:18.920375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.793487+0000","last_change":"2024-04-02T09:15:26.786850+0000","last_active":"2024-04-02T09:15:26.793487+0000","last_peered":"2024-04-02T09:15:26.793487+0000","last_clean":"2024-04-02T09:15:26.793487+0000","last_became_active":"2024-04-02T09:15:26.786590+0000","last_became_peered":"2024-04-02T09:15:26.786590+0000","last_unstale":"2024-04-02T09:15:26.793487+0000","last_undegraded":"2024-04-02T09:15:26.793487+0000","last_fullsized":"2024-04-02T09:15:26.793487+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:46:26.535123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.782841+0000","last_change":"2024-04-02T09:15:26.282287+0000","last_active":"2024-04-02T09:15:26.782841+0000","last_peered":"2024-04-02T09:15:26.782841+0000","last_clean":"2024-04-02T09:15:26.782841+0000","last_became_active":"2024-04-02T09:15:26.281984+0000","last_became_peered":"2024-04-02T09:15:26.281984+0000","last_unstale":"2024-04-02T09:15:26.782841+0000","last_undegraded":"2024-04-02T09:15:26.782841+0000","last_fullsized":"2024-04-02T09:15:26.782841+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:10:58.674355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.804744+0000","last_change":"2024-04-02T09:15:26.803112+0000","last_active":"2024-04-02T09:15:26.804744+0000","last_peered":"2024-04-02T09:15:26.804744+0000","last_clean":"2024-04-02T09:15:26.804744+0000","last_became_active":"2024-04-02T09:15:26.799381+0000","last_became_peered":"2024-04-02T09:15:26.799381+0000","last_unstale":"2024-04-02T09:15:26.804744+0000","last_undegraded":"2024-04-02T09:15:26.804744+0000","last_fullsized":"2024-04-02T09:15:26.804744+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:46:26.535123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"268'403","reported_seq":1454,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.289732+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:15:26.289732+0000","last_peered":"2024-04-02T09:15:26.289732+0000","last_clean":"2024-04-02T09:15:26.289732+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:15:26.289732+0000","last_undegraded":"2024-04-02T09:15:26.289732+0000","last_fullsized":"2024-04-02T09:15:26.289732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60796,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311044,"statfs":{"total":96636764160,"available":96574509056,"internally_reserved":0,"allocated":3514368,"data_stored":1808858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.71899999999999997,"15min":0.71499999999999997},"min":{"1min":0.40500000000000003,"5min":0.309,"15min":0.309},"max":{"1min":1.115,"5min":8.016,"15min":8.016},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.70199999999999996,"15min":0.71299999999999997},"min":{"1min":0.48899999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.044,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.57299999999999995}]},{"osd":1,"last update":"Tue Apr 2 09:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.746,"15min":0.73699999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.38200000000000001},"max":{"1min":0.81299999999999994,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.68500000000000005,"15min":0.70399999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.84499999999999997,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.68999999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.76000000000000001,"15min":0.76100000000000001},"min":{"1min":0.44600000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.202,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.79300000000000004,"15min":0.80200000000000005},"min":{"1min":0.53100000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2470000000000001,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.77300000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.747,"15min":0.74399999999999999},"min":{"1min":0.49099999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0229999999999999,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.78800000000000003,"15min":0.76800000000000002},"min":{"1min":0.48199999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.91100000000000003,"5min":5.093,"15min":5.093},"last":0.50700000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67100000000000004,"15min":0.628},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86299999999999999,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.79900000000000004,"15min":0.749},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.052,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.75900000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.73099999999999998,"15min":0.69499999999999995},"min":{"1min":0.46400000000000002,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":1.036,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.76500000000000001,"15min":0.73599999999999999},"min":{"1min":0.436,"5min":0.42599999999999999,"15min":0.41099999999999998},"max":{"1min":0.95299999999999996,"5min":3.492,"15min":3.492},"last":0.74199999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.72099999999999997,"15min":0.69799999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.85499999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.77900000000000003,"15min":0.72199999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93200000000000005,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.79500000000000004}]}]},{"osd":6,"up_from":13,"seq":55834576015,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42760,"kb_used_data":3444,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329080,"statfs":{"total":96636764160,"available":96592977920,"internally_reserved":0,"allocated":3526656,"data_stored":1809544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.67200000000000004,"15min":0.70899999999999996},"min":{"1min":0.432,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81399999999999995,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.68000000000000005,"15min":0.72299999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.87,"5min":0.98399999999999999,"15min":1.0600000000000001},"last":0.66700000000000004}]},{"osd":1,"last update":"Tue Apr 2 09:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.67900000000000005,"15min":0.70599999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0149999999999999,"5min":1.044,"15min":1.212},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.69899999999999995,"15min":0.73599999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.88400000000000001,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.73199999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.69799999999999995,"15min":0.73499999999999999},"min":{"1min":0.45200000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.85299999999999998,"5min":1.046,"15min":1.2789999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.68600000000000005,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.90000000000000002,"5min":0.99299999999999999,"15min":1.226},"last":0.69899999999999995}]},{"osd":3,"last update":"Tue Apr 2 09:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70899999999999996,"15min":0.745},"min":{"1min":0.52200000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.84399999999999997,"5min":1.141,"15min":1.2549999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.73399999999999999,"15min":0.76300000000000001},"min":{"1min":0.56399999999999995,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.96499999999999997,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.81899999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.629,"15min":0.623},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86199999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.65900000000000003,"15min":0.66100000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.84899999999999998,"5min":1.1100000000000001,"15min":1.1879999999999999},"last":0.76500000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.65500000000000003,"15min":0.65300000000000002},"min":{"1min":0.48099999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.2669999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.51000000000000001,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":0.93500000000000005,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.877}]},{"osd":7,"last update":"Tue Apr 2 09:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.55300000000000005,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":0.92200000000000004,"5min":1.21,"15min":1.2370000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72999999999999998,"15min":0.73499999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94999999999999996,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.90600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576015,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56704,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315136,"statfs":{"total":96636764160,"available":96578699264,"internally_reserved":0,"allocated":3518464,"data_stored":1808928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.42099999999999999,"15min":0.443},"min":{"1min":0.28299999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73599999999999999,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.45100000000000001,"15min":0.47799999999999998},"min":{"1min":0.34300000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.61199999999999999,"5min":0.71199999999999997,"15min":0.876},"last":0.36099999999999999}]},{"osd":1,"last update":"Tue Apr 2 09:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46899999999999997,"15min":0.495},"min":{"1min":0.30499999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.61099999999999999,"5min":0.81699999999999995,"15min":0.90300000000000002},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53400000000000003,"15min":0.56399999999999995},"min":{"1min":0.41699999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.34300000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.57799999999999996,"15min":0.61199999999999999},"min":{"1min":0.41799999999999998,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.70299999999999996,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55800000000000005,"15min":0.60299999999999998},"min":{"1min":0.38700000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67900000000000005,"5min":0.85099999999999998,"15min":1.0840000000000001},"last":0.495}]},{"osd":4,"last update":"Tue Apr 2 09:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71899999999999997,"15min":0.748},"min":{"1min":0.53000000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83099999999999996,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72799999999999998,"15min":0.76000000000000001},"min":{"1min":0.505,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.89500000000000002,"5min":1.018,"15min":1.1619999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Tue Apr 2 09:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73999999999999999,"15min":0.76900000000000002},"min":{"1min":0.52200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92800000000000005,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.74099999999999999,"15min":0.77500000000000002},"min":{"1min":0.61199999999999999,"5min":0.505,"15min":0.505},"max":{"1min":0.877,"5min":0.99099999999999999,"15min":1.266},"last":0.58999999999999997}]},{"osd":6,"last update":"Tue Apr 2 09:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.747,"15min":0.77800000000000002},"min":{"1min":0.54400000000000004,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.996,"5min":1.0489999999999999,"15min":1.284},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76700000000000002,"15min":0.80200000000000005},"min":{"1min":0.625,"5min":0.495,"15min":0.495},"max":{"1min":0.98099999999999998,"5min":1.0840000000000001,"15min":1.2},"last":0.59599999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.753,"15min":0.79700000000000004},"min":{"1min":0.51500000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.95399999999999996,"5min":1.125,"15min":1.3260000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.754,"5min":0.78000000000000003,"15min":0.81399999999999995},"min":{"1min":0.61399999999999999,"5min":0.495,"15min":0.495},"max":{"1min":0.94499999999999995,"5min":1.103,"15min":1.248},"last":0.60299999999999998}]}]},{"osd":0,"up_from":13,"seq":55834576016,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52580,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319260,"statfs":{"total":96636764160,"available":96582922240,"internally_reserved":0,"allocated":3497984,"data_stored":1800616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42899999999999999,"15min":0.44600000000000001},"min":{"1min":0.28199999999999997,"5min":0.26700000000000002,"15min":0.254},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.86699999999999999},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.46600000000000003,"15min":0.49199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.52200000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.29299999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49299999999999999,"15min":0.50600000000000001},"min":{"1min":0.36899999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.307},{"interface":"front","average":{"1min":0.443,"5min":0.51100000000000001,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.52800000000000002,"5min":0.746,"15min":0.999},"last":0.45000000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.52200000000000002,"15min":0.55100000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.312},"max":{"1min":0.75600000000000001,"5min":0.79800000000000004,"15min":0.85699999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58899999999999997,"15min":0.624},"min":{"1min":0.45200000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.81100000000000005,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.46100000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.45600000000000002},"max":{"1min":1.0169999999999999,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.77700000000000002,"15min":0.80900000000000005},"min":{"1min":0.496,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.012,"5min":1.169,"15min":1.169},"last":0.70999999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.78000000000000003,"15min":0.81000000000000005},"min":{"1min":0.56899999999999995,"5min":0.54000000000000004,"15min":0.497},"max":{"1min":1.0840000000000001,"5min":1.145,"15min":1.232},"last":0.628},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.78400000000000003,"15min":0.81799999999999995},"min":{"1min":0.52600000000000002,"5min":0.497,"15min":0.497},"max":{"1min":0.98099999999999998,"5min":1.21,"15min":1.21},"last":0.58499999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.75800000000000001,"15min":0.78900000000000003},"min":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91600000000000004,"5min":1.1830000000000001,"15min":1.196},"last":0.497},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":0.79700000000000004},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.52900000000000003},"max":{"1min":0.95699999999999996,"5min":1.246,"15min":1.246},"last":0.55000000000000004}]},{"osd":7,"last update":"Tue Apr 2 09:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.81399999999999995,"15min":0.84099999999999997},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.99399999999999999,"5min":1.292,"15min":1.292},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.79600000000000004,"15min":0.83299999999999996},"min":{"1min":0.54100000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.046,"5min":1.1719999999999999,"15min":1.2130000000000001},"last":0.67600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834576016,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57284,"kb_used_data":4024,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314556,"statfs":{"total":96636764160,"available":96578105344,"internally_reserved":0,"allocated":4120576,"data_stored":2400790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.433,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66800000000000004,"5min":0.82199999999999995,"15min":0.82499999999999996},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48299999999999998,"15min":0.502},"min":{"1min":0.35699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.623,"5min":0.78800000000000003,"15min":0.79800000000000004},"last":0.46300000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.48699999999999999,"15min":0.496},"min":{"1min":0.34200000000000003,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.63900000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55700000000000005,"15min":0.58499999999999996},"min":{"1min":0.41099999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.82599999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47899999999999998}]},{"osd":3,"last update":"Tue Apr 2 09:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.55100000000000005,"15min":0.56799999999999995},"min":{"1min":0.42599999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77000000000000002,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53700000000000003,"15min":0.55500000000000005},"min":{"1min":0.35699999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75600000000000001,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.49399999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.748,"15min":0.77600000000000002},"min":{"1min":0.58299999999999996,"5min":0.44500000000000001,"15min":0.433},"max":{"1min":0.94599999999999995,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.74199999999999999,"15min":0.77000000000000002},"min":{"1min":0.56899999999999995,"5min":0.48999999999999999,"15min":0.41799999999999998},"max":{"1min":1.0009999999999999,"5min":1.05,"15min":1.25},"last":0.94099999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.746,"15min":0.77700000000000002},"min":{"1min":0.54200000000000004,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.321},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.77100000000000002,"15min":0.79800000000000004},"min":{"1min":0.55100000000000005,"5min":0.501,"15min":0.501},"max":{"1min":0.98199999999999998,"5min":1.115,"15min":1.173},"last":0.90900000000000003}]},{"osd":6,"last update":"Tue Apr 2 09:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.77200000000000002,"15min":0.80000000000000004},"min":{"1min":0.53600000000000003,"5min":0.51600000000000001,"15min":0.48099999999999998},"max":{"1min":1.0009999999999999,"5min":1.087,"15min":1.2210000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.78400000000000003,"15min":0.80500000000000005},"min":{"1min":0.59899999999999998,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.014,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.89300000000000002}]},{"osd":7,"last update":"Tue Apr 2 09:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.78200000000000003,"15min":0.81100000000000005},"min":{"1min":0.52300000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0389999999999999,"5min":1.1080000000000001,"15min":1.236},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.80900000000000005,"15min":0.82399999999999995},"min":{"1min":0.56000000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.0369999999999999,"5min":1.129,"15min":1.2789999999999999},"last":0.79100000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576015,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52608,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319232,"statfs":{"total":96636764160,"available":96582893568,"internally_reserved":0,"allocated":3518464,"data_stored":1809440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.441,"15min":0.44},"min":{"1min":0.315,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.54700000000000004,"5min":1.681,"15min":1.681},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.497,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":1.091,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":0.378}]},{"osd":1,"last update":"Tue Apr 2 09:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.496,"15min":0.499},"min":{"1min":0.36399999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.60199999999999998,"5min":1.72,"15min":1.72},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53800000000000003,"15min":0.55000000000000004},"min":{"1min":0.40699999999999997,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.84499999999999997,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.54500000000000004},"min":{"1min":0.35999999999999999,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.72099999999999997,"5min":0.80700000000000005,"15min":1.2749999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58999999999999997,"15min":0.621},"min":{"1min":0.44,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.86799999999999999,"5min":1.754,"15min":1.754},"last":0.54100000000000004}]},{"osd":4,"last update":"Tue Apr 2 09:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76400000000000001,"15min":0.77100000000000002},"min":{"1min":0.58399999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.266,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.76900000000000002,"15min":0.78600000000000003},"min":{"1min":0.502,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0429999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.624}]},{"osd":5,"last update":"Tue Apr 2 09:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.75800000000000001,"15min":0.78100000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.96499999999999997,"5min":1.1379999999999999,"15min":1.99},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.77800000000000002,"15min":0.79500000000000004},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1120000000000001,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.69499999999999995}]},{"osd":6,"last update":"Tue Apr 2 09:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.78200000000000003,"15min":0.80200000000000005},"min":{"1min":0.56399999999999995,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.498,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.755,"5min":0.77100000000000002,"15min":0.79700000000000004},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.54100000000000004},"max":{"1min":1.0209999999999999,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.71199999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.80200000000000005,"15min":0.82199999999999995},"min":{"1min":0.624,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.776,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79300000000000004,"15min":0.81899999999999995},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.497},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.61099999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576015,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48520,"kb_used_data":3444,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323320,"statfs":{"total":96636764160,"available":96587079680,"internally_reserved":0,"allocated":3526656,"data_stored":1809004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64000000000000001,"15min":0.67000000000000004},"min":{"1min":0.38600000000000001,"5min":0.38100000000000001,"15min":0.29299999999999998},"max":{"1min":0.86499999999999999,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.63600000000000001,"15min":0.66800000000000004},"min":{"1min":0.48399999999999999,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.81100000000000005,"5min":0.97099999999999997,"15min":1.075},"last":0.73199999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.74199999999999999},"min":{"1min":0.54400000000000004,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":0.88500000000000001,"5min":1.0329999999999999,"15min":1.069},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.72399999999999998,"15min":0.752},"min":{"1min":0.54000000000000004,"5min":0.46100000000000002,"15min":0.40799999999999997},"max":{"1min":0.97999999999999998,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.74299999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.66700000000000004,"15min":0.69799999999999995},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.077,"15min":1.077},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68100000000000005,"15min":0.71399999999999997},"min":{"1min":0.45300000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93300000000000005,"5min":1.032,"15min":1.0740000000000001},"last":0.69199999999999995}]},{"osd":3,"last update":"Tue Apr 2 09:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.71899999999999997,"15min":0.749},"min":{"1min":0.499,"5min":0.48799999999999999,"15min":0.47399999999999998},"max":{"1min":0.89900000000000002,"5min":0.98099999999999998,"15min":1.018},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70499999999999996,"15min":0.74399999999999999},"min":{"1min":0.55800000000000005,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":0.86399999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.67400000000000004}]},{"osd":5,"last update":"Tue Apr 2 09:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.60799999999999998,"15min":0.61099999999999999},"min":{"1min":0.502,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.90800000000000003,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.622,"15min":0.624},"min":{"1min":0.46300000000000002,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.88900000000000001,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.495}]},{"osd":6,"last update":"Tue Apr 2 09:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.42799999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.93899999999999995,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68899999999999995,"15min":0.69699999999999995},"min":{"1min":0.46999999999999997,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":0.95199999999999996,"5min":1.0149999999999999,"15min":1.0269999999999999},"last":0.62}]},{"osd":7,"last update":"Tue Apr 2 09:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.69899999999999995,"15min":0.70999999999999996},"min":{"1min":0.56899999999999995,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.94199999999999995,"5min":1.002,"15min":1.0329999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.76500000000000001,"15min":0.78000000000000003},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.056,"5min":1.0960000000000001,"15min":1.0980000000000001},"last":0.65500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576018,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53176,"kb_used_data":4008,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318664,"statfs":{"total":96636764160,"available":96582311936,"internally_reserved":0,"allocated":4104192,"data_stored":2398207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.67300000000000004,"15min":0.69499999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.89000000000000001,"5min":0.98699999999999999,"15min":1.004},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68799999999999994,"15min":0.71399999999999997},"min":{"1min":0.47299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.871,"5min":1.085,"15min":1.085},"last":0.68100000000000005}]},{"osd":1,"last update":"Tue Apr 2 09:14:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.67200000000000004,"15min":0.70699999999999996},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77300000000000002,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.69399999999999995,"15min":0.71899999999999997},"min":{"1min":0.47399999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.89900000000000002,"5min":0.95299999999999996,"15min":1.052},"last":0.65200000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:14:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.70199999999999996,"15min":0.73199999999999998},"min":{"1min":0.52300000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.83599999999999997,"5min":1.004,"15min":1.014},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68700000000000006,"15min":0.72999999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.82899999999999996,"5min":1.018,"15min":1.0680000000000001},"last":0.88500000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:14:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.73599999999999999,"15min":0.77100000000000002},"min":{"1min":0.55000000000000004,"5min":0.496,"15min":0.496},"max":{"1min":0.92300000000000004,"5min":0.98899999999999999,"15min":1.083},"last":0.753},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.76000000000000001,"15min":0.78300000000000003},"min":{"1min":0.50600000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88700000000000001,"5min":1.0109999999999999,"15min":1.1020000000000001},"last":0.98099999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:14:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.441,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.88700000000000001,"5min":0.98399999999999999,"15min":1.054},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.68400000000000005,"15min":0.67600000000000005},"min":{"1min":0.46000000000000002,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":0.91000000000000003,"5min":1.145,"15min":1.145},"last":0.83299999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:14:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.68400000000000005,"15min":0.67700000000000005},"min":{"1min":0.441,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93799999999999994,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":1.006},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.48599999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.95599999999999996,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.81299999999999994}]},{"osd":7,"last update":"Tue Apr 2 09:14:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.73499999999999999,"15min":0.73599999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.97099999999999997,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76200000000000001,"15min":0.76600000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.98699999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":1.03}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:15:31.638 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:15:31.684 INFO:tasks.ceph.mds.a.smithi089.stderr:2024-04-02T09:15:31.686+0000 7f7e1a400700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T09:15:31.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.865+0000 7f55ad800700 1 Processor -- start 2024-04-02T09:15:31.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.865+0000 7f55ad800700 1 -- start start 2024-04-02T09:15:31.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.866+0000 7f55ad800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f55a815d640 0x7f55a80d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:31.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.866+0000 7f55ad800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a81619d0 0x7f55a80b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:31.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.866+0000 7f55ad800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a80d30d0 con 0x7f55a80ceec0 2024-04-02T09:15:31.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.866+0000 7f55ad800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a80b6840 con 0x7f55a815d640 2024-04-02T09:15:31.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.866+0000 7f55ad800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a80d2880 con 0x7f55a81619d0 2024-04-02T09:15:31.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.867+0000 7f55a6a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f55a815d640 0x7f55a80d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:31.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.867+0000 7f55a6a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f55a815d640 0x7f55a80d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52534/0 (socket says 172.21.15.89:52534) 2024-04-02T09:15:31.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.867+0000 7f55a6a00700 1 -- 172.21.15.89:0/3181237069 learned_addr learned my addr 172.21.15.89:0/3181237069 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:31.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.867+0000 7f55a7400700 1 --2- 172.21.15.89:0/3181237069 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a81619d0 0x7f55a80b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:31.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.867+0000 7f55a6a00700 1 -- 172.21.15.89:0/3181237069 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a81619d0 msgr2=0x7f55a80b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:31.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.867+0000 7f55a6a00700 1 --2- 172.21.15.89:0/3181237069 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a81619d0 0x7f55a80b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.867+0000 7f55a6a00700 1 -- 172.21.15.89:0/3181237069 >> v1:172.21.15.113:6789/0 conn(0x7f55a80ceec0 legacy=0x7f55a81612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:31.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.867+0000 7f55a6a00700 1 -- 172.21.15.89:0/3181237069 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a80d2b30 con 0x7f55a815d640 2024-04-02T09:15:31.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.867+0000 7f55a6a00700 1 --2- 172.21.15.89:0/3181237069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a815d640 0x7f55a80d2340 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f55980098f0 tx=0x7f5598009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5813c7399694ef server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:31.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.868+0000 7f55a6000700 1 -- 172.21.15.89:0/3181237069 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559800b8a0 con 0x7f55a815d640 2024-04-02T09:15:31.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.868+0000 7f55a6000700 1 -- 172.21.15.89:0/3181237069 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559800ba00 con 0x7f55a815d640 2024-04-02T09:15:31.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.868+0000 7f55a6000700 1 -- 172.21.15.89:0/3181237069 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559800bd90 con 0x7f55a815d640 2024-04-02T09:15:31.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.868+0000 7f55ad800700 1 -- 172.21.15.89:0/3181237069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a815d640 msgr2=0x7f55a80d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:31.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.868+0000 7f55ad800700 1 --2- 172.21.15.89:0/3181237069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a815d640 0x7f55a80d2340 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f55980098f0 tx=0x7f5598009c00 comp rx=0 tx=0).stop 2024-04-02T09:15:31.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.868+0000 7f55ad800700 1 -- 172.21.15.89:0/3181237069 shutdown_connections 2024-04-02T09:15:31.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.868+0000 7f55ad800700 1 --2- 172.21.15.89:0/3181237069 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a81619d0 0x7f55a80b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.868+0000 7f55ad800700 1 --2- 172.21.15.89:0/3181237069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a815d640 0x7f55a80d2340 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.868+0000 7f55ad800700 1 -- 172.21.15.89:0/3181237069 >> 172.21.15.89:0/3181237069 conn(0x7f55a808aff0 msgr2=0x7f55a80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:31.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.868+0000 7f55ad800700 1 -- 172.21.15.89:0/3181237069 shutdown_connections 2024-04-02T09:15:31.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.868+0000 7f55ad800700 1 -- 172.21.15.89:0/3181237069 wait complete. 2024-04-02T09:15:31.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.870+0000 7f55ad800700 1 Processor -- start 2024-04-02T09:15:31.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.870+0000 7f55ad800700 1 -- start start 2024-04-02T09:15:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.871+0000 7f55ad800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a80ceec0 0x7f55a8158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.871+0000 7f55ad800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a815d640 0x7f55a80c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.871+0000 7f55ad800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55a81619d0 0x7f55a815f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.871+0000 7f55ad800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a81f3020 con 0x7f55a815d640 2024-04-02T09:15:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.871+0000 7f55ad800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a809db40 con 0x7f55a81619d0 2024-04-02T09:15:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.871+0000 7f55ad800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a809de20 con 0x7f55a80ceec0 2024-04-02T09:15:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.871+0000 7f559e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a815d640 0x7f55a80c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.871+0000 7f55a6a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a80ceec0 0x7f55a8158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.871+0000 7f559e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a815d640 0x7f55a80c9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52538/0 (socket says 172.21.15.89:52538) 2024-04-02T09:15:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.871+0000 7f559e000700 1 -- 172.21.15.89:0/2076393823 learned_addr learned my addr 172.21.15.89:0/2076393823 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.871+0000 7f55a7400700 1 --2- 172.21.15.89:0/2076393823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55a81619d0 0x7f55a815f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.872+0000 7f559e000700 1 -- 172.21.15.89:0/2076393823 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a80ceec0 msgr2=0x7f55a8158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.872+0000 7f559e000700 1 --2- 172.21.15.89:0/2076393823 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a80ceec0 0x7f55a8158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.872+0000 7f559e000700 1 -- 172.21.15.89:0/2076393823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55a81619d0 msgr2=0x7f55a815f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.872+0000 7f559e000700 1 --2- 172.21.15.89:0/2076393823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55a81619d0 0x7f55a815f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.872+0000 7f559e000700 1 -- 172.21.15.89:0/2076393823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5598009550 con 0x7f55a815d640 2024-04-02T09:15:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.872+0000 7f559e000700 1 --2- 172.21.15.89:0/2076393823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a815d640 0x7f55a80c9230 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7f559000c450 tx=0x7f559000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.872+0000 7f559f400700 1 -- 172.21.15.89:0/2076393823 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55900052d0 con 0x7f55a815d640 2024-04-02T09:15:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.872+0000 7f559f400700 1 -- 172.21.15.89:0/2076393823 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5590005430 con 0x7f55a815d640 2024-04-02T09:15:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.872+0000 7f55ad800700 1 -- 172.21.15.89:0/2076393823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a809e100 con 0x7f55a815d640 2024-04-02T09:15:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.872+0000 7f559f400700 1 -- 172.21.15.89:0/2076393823 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55900038c0 con 0x7f55a815d640 2024-04-02T09:15:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.872+0000 7f55ad800700 1 -- 172.21.15.89:0/2076393823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a809e3e0 con 0x7f55a815d640 2024-04-02T09:15:31.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.873+0000 7f55ad800700 1 -- 172.21.15.89:0/2076393823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a809f920 con 0x7f55a815d640 2024-04-02T09:15:31.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.874+0000 7f559f400700 1 -- 172.21.15.89:0/2076393823 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5590003a20 con 0x7f55a815d640 2024-04-02T09:15:31.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.875+0000 7f559f400700 1 --2- 172.21.15.89:0/2076393823 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f558406adb0 0x7f558406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:31.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.875+0000 7f559f400700 1 -- 172.21.15.89:0/2076393823 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f5590088920 con 0x7f55a815d640 2024-04-02T09:15:31.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.875+0000 7f55a6a00700 1 --2- 172.21.15.89:0/2076393823 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f558406adb0 0x7f558406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.876+0000 7f55a6a00700 1 --2- 172.21.15.89:0/2076393823 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f558406adb0 0x7f558406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5598009c40 tx=0x7f5598008030 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:31.879+0000 7f559f400700 1 -- 172.21.15.89:0/2076393823 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f55a809f920 con 0x7f55a815d640 2024-04-02T09:15:32.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.074+0000 7f55ad800700 1 -- 172.21.15.89:0/2076393823 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f55a809df90 con 0x7f558406adb0 2024-04-02T09:15:32.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.082+0000 7f559f400700 1 -- 172.21.15.89:0/2076393823 <== mgr.13065 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289394 (secure 0 0 0) 0x7f55a809df90 con 0x7f558406adb0 2024-04-02T09:15:32.086 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:32.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.087+0000 7f55ad800700 1 -- 172.21.15.89:0/2076393823 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f558406adb0 msgr2=0x7f558406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:32.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.087+0000 7f55ad800700 1 --2- 172.21.15.89:0/2076393823 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f558406adb0 0x7f558406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5598009c40 tx=0x7f5598008030 comp rx=0 tx=0).stop 2024-04-02T09:15:32.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.087+0000 7f55ad800700 1 -- 172.21.15.89:0/2076393823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a815d640 msgr2=0x7f55a80c9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:32.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.087+0000 7f55ad800700 1 --2- 172.21.15.89:0/2076393823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a815d640 0x7f55a80c9230 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7f559000c450 tx=0x7f559000c760 comp rx=0 tx=0).stop 2024-04-02T09:15:32.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.087+0000 7f55ad800700 1 -- 172.21.15.89:0/2076393823 shutdown_connections 2024-04-02T09:15:32.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.087+0000 7f55ad800700 1 --2- 172.21.15.89:0/2076393823 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f558406adb0 0x7f558406d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.087+0000 7f55ad800700 1 --2- 172.21.15.89:0/2076393823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55a81619d0 0x7f55a815f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.087+0000 7f55ad800700 1 --2- 172.21.15.89:0/2076393823 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55a80ceec0 0x7f55a8158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.087+0000 7f55ad800700 1 --2- 172.21.15.89:0/2076393823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55a815d640 0x7f55a80c9230 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.087+0000 7f55ad800700 1 -- 172.21.15.89:0/2076393823 >> 172.21.15.89:0/2076393823 conn(0x7f55a808aff0 msgr2=0x7f55a80b7c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:32.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.087+0000 7f55ad800700 1 -- 172.21.15.89:0/2076393823 shutdown_connections 2024-04-02T09:15:32.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.087+0000 7f55ad800700 1 -- 172.21.15.89:0/2076393823 wait complete. 2024-04-02T09:15:32.092 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:15:32.104 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-02T09:15:30.613268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":723,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":441,"ondisk_log_size":441,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":424428,"kb_used_data":28640,"kb_used_omap":12,"kb_used_meta":395635,"kb_avail":754550292,"statfs":{"total":773094113280,"available":772659499008,"internally_reserved":0,"allocated":29327360,"data_stored":15645387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":405130808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.350279"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.804800+0000","last_change":"2024-04-02T09:15:26.802021+0000","last_active":"2024-04-02T09:15:26.804800+0000","last_peered":"2024-04-02T09:15:26.804800+0000","last_clean":"2024-04-02T09:15:26.804800+0000","last_became_active":"2024-04-02T09:15:26.801766+0000","last_became_peered":"2024-04-02T09:15:26.801766+0000","last_unstale":"2024-04-02T09:15:26.804800+0000","last_undegraded":"2024-04-02T09:15:26.804800+0000","last_fullsized":"2024-04-02T09:15:26.804800+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:10:44.608892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.792895+0000","last_change":"2024-04-02T09:15:26.785747+0000","last_active":"2024-04-02T09:15:26.792895+0000","last_peered":"2024-04-02T09:15:26.792895+0000","last_clean":"2024-04-02T09:15:26.792895+0000","last_became_active":"2024-04-02T09:15:26.784926+0000","last_became_peered":"2024-04-02T09:15:26.784926+0000","last_unstale":"2024-04-02T09:15:26.792895+0000","last_undegraded":"2024-04-02T09:15:26.792895+0000","last_fullsized":"2024-04-02T09:15:26.792895+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:21:39.563539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.803157+0000","last_change":"2024-04-02T09:15:26.796747+0000","last_active":"2024-04-02T09:15:26.803157+0000","last_peered":"2024-04-02T09:15:26.803157+0000","last_clean":"2024-04-02T09:15:26.803157+0000","last_became_active":"2024-04-02T09:15:26.795819+0000","last_became_peered":"2024-04-02T09:15:26.795819+0000","last_unstale":"2024-04-02T09:15:26.803157+0000","last_undegraded":"2024-04-02T09:15:26.803157+0000","last_fullsized":"2024-04-02T09:15:26.803157+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:21:39.563539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812398+0000","last_change":"2024-04-02T09:15:26.809799+0000","last_active":"2024-04-02T09:15:26.812398+0000","last_peered":"2024-04-02T09:15:26.812398+0000","last_clean":"2024-04-02T09:15:26.812398+0000","last_became_active":"2024-04-02T09:15:26.808211+0000","last_became_peered":"2024-04-02T09:15:26.808211+0000","last_unstale":"2024-04-02T09:15:26.812398+0000","last_undegraded":"2024-04-02T09:15:26.812398+0000","last_fullsized":"2024-04-02T09:15:26.812398+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:18:39.670018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.809486+0000","last_change":"2024-04-02T09:15:26.807096+0000","last_active":"2024-04-02T09:15:26.809486+0000","last_peered":"2024-04-02T09:15:26.809486+0000","last_clean":"2024-04-02T09:15:26.809486+0000","last_became_active":"2024-04-02T09:15:26.806445+0000","last_became_peered":"2024-04-02T09:15:26.806445+0000","last_unstale":"2024-04-02T09:15:26.809486+0000","last_undegraded":"2024-04-02T09:15:26.809486+0000","last_fullsized":"2024-04-02T09:15:26.809486+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:44:27.767171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.794631+0000","last_change":"2024-04-02T09:15:26.789801+0000","last_active":"2024-04-02T09:15:26.794631+0000","last_peered":"2024-04-02T09:15:26.794631+0000","last_clean":"2024-04-02T09:15:26.794631+0000","last_became_active":"2024-04-02T09:15:26.789427+0000","last_became_peered":"2024-04-02T09:15:26.789427+0000","last_unstale":"2024-04-02T09:15:26.794631+0000","last_undegraded":"2024-04-02T09:15:26.794631+0000","last_fullsized":"2024-04-02T09:15:26.794631+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:04:05.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811923+0000","last_change":"2024-04-02T09:15:26.810612+0000","last_active":"2024-04-02T09:15:26.811923+0000","last_peered":"2024-04-02T09:15:26.811923+0000","last_clean":"2024-04-02T09:15:26.811923+0000","last_became_active":"2024-04-02T09:15:26.810338+0000","last_became_peered":"2024-04-02T09:15:26.810338+0000","last_unstale":"2024-04-02T09:15:26.811923+0000","last_undegraded":"2024-04-02T09:15:26.811923+0000","last_fullsized":"2024-04-02T09:15:26.811923+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:08:06.148499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811505+0000","last_change":"2024-04-02T09:15:26.805407+0000","last_active":"2024-04-02T09:15:26.811505+0000","last_peered":"2024-04-02T09:15:26.811505+0000","last_clean":"2024-04-02T09:15:26.811505+0000","last_became_active":"2024-04-02T09:15:26.804705+0000","last_became_peered":"2024-04-02T09:15:26.804705+0000","last_unstale":"2024-04-02T09:15:26.811505+0000","last_undegraded":"2024-04-02T09:15:26.811505+0000","last_fullsized":"2024-04-02T09:15:26.811505+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:15:58.057433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.809707+0000","last_change":"2024-04-02T09:15:26.796245+0000","last_active":"2024-04-02T09:15:26.809707+0000","last_peered":"2024-04-02T09:15:26.809707+0000","last_clean":"2024-04-02T09:15:26.809707+0000","last_became_active":"2024-04-02T09:15:26.795144+0000","last_became_peered":"2024-04-02T09:15:26.795144+0000","last_unstale":"2024-04-02T09:15:26.809707+0000","last_undegraded":"2024-04-02T09:15:26.809707+0000","last_fullsized":"2024-04-02T09:15:26.809707+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:40:11.841909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.789737+0000","last_change":"2024-04-02T09:15:26.786693+0000","last_active":"2024-04-02T09:15:26.789737+0000","last_peered":"2024-04-02T09:15:26.789737+0000","last_clean":"2024-04-02T09:15:26.789737+0000","last_became_active":"2024-04-02T09:15:26.786405+0000","last_became_peered":"2024-04-02T09:15:26.786405+0000","last_unstale":"2024-04-02T09:15:26.789737+0000","last_undegraded":"2024-04-02T09:15:26.789737+0000","last_fullsized":"2024-04-02T09:15:26.789737+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:06.126308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812261+0000","last_change":"2024-04-02T09:15:26.807980+0000","last_active":"2024-04-02T09:15:26.812261+0000","last_peered":"2024-04-02T09:15:26.812261+0000","last_clean":"2024-04-02T09:15:26.812261+0000","last_became_active":"2024-04-02T09:15:26.802932+0000","last_became_peered":"2024-04-02T09:15:26.802932+0000","last_unstale":"2024-04-02T09:15:26.812261+0000","last_undegraded":"2024-04-02T09:15:26.812261+0000","last_fullsized":"2024-04-02T09:15:26.812261+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:10:52.826706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.805240+0000","last_change":"2024-04-02T09:15:26.803334+0000","last_active":"2024-04-02T09:15:26.805240+0000","last_peered":"2024-04-02T09:15:26.805240+0000","last_clean":"2024-04-02T09:15:26.805240+0000","last_became_active":"2024-04-02T09:15:26.803072+0000","last_became_peered":"2024-04-02T09:15:26.803072+0000","last_unstale":"2024-04-02T09:15:26.805240+0000","last_undegraded":"2024-04-02T09:15:26.805240+0000","last_fullsized":"2024-04-02T09:15:26.805240+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:50:49.687799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.797587+0000","last_change":"2024-04-02T09:15:26.796160+0000","last_active":"2024-04-02T09:15:26.797587+0000","last_peered":"2024-04-02T09:15:26.797587+0000","last_clean":"2024-04-02T09:15:26.797587+0000","last_became_active":"2024-04-02T09:15:26.795912+0000","last_became_peered":"2024-04-02T09:15:26.795912+0000","last_unstale":"2024-04-02T09:15:26.797587+0000","last_undegraded":"2024-04-02T09:15:26.797587+0000","last_fullsized":"2024-04-02T09:15:26.797587+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:11:49.759531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.793498+0000","last_change":"2024-04-02T09:15:26.788996+0000","last_active":"2024-04-02T09:15:26.793498+0000","last_peered":"2024-04-02T09:15:26.793498+0000","last_clean":"2024-04-02T09:15:26.793498+0000","last_became_active":"2024-04-02T09:15:26.787458+0000","last_became_peered":"2024-04-02T09:15:26.787458+0000","last_unstale":"2024-04-02T09:15:26.793498+0000","last_undegraded":"2024-04-02T09:15:26.793498+0000","last_fullsized":"2024-04-02T09:15:26.793498+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:16:10.155874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.793516+0000","last_change":"2024-04-02T09:15:26.789348+0000","last_active":"2024-04-02T09:15:26.793516+0000","last_peered":"2024-04-02T09:15:26.793516+0000","last_clean":"2024-04-02T09:15:26.793516+0000","last_became_active":"2024-04-02T09:15:26.789042+0000","last_became_peered":"2024-04-02T09:15:26.789042+0000","last_unstale":"2024-04-02T09:15:26.793516+0000","last_undegraded":"2024-04-02T09:15:26.793516+0000","last_fullsized":"2024-04-02T09:15:26.793516+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:52:15.459115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.807571+0000","last_change":"2024-04-02T09:15:26.802826+0000","last_active":"2024-04-02T09:15:26.807571+0000","last_peered":"2024-04-02T09:15:26.807571+0000","last_clean":"2024-04-02T09:15:26.807571+0000","last_became_active":"2024-04-02T09:15:26.800730+0000","last_became_peered":"2024-04-02T09:15:26.800730+0000","last_unstale":"2024-04-02T09:15:26.807571+0000","last_undegraded":"2024-04-02T09:15:26.807571+0000","last_fullsized":"2024-04-02T09:15:26.807571+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:00:04.287854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.809934+0000","last_change":"2024-04-02T09:15:26.806992+0000","last_active":"2024-04-02T09:15:26.809934+0000","last_peered":"2024-04-02T09:15:26.809934+0000","last_clean":"2024-04-02T09:15:26.809934+0000","last_became_active":"2024-04-02T09:15:26.805424+0000","last_became_peered":"2024-04-02T09:15:26.805424+0000","last_unstale":"2024-04-02T09:15:26.809934+0000","last_undegraded":"2024-04-02T09:15:26.809934+0000","last_fullsized":"2024-04-02T09:15:26.809934+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:46:15.900655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.796160+0000","last_change":"2024-04-02T09:15:26.788936+0000","last_active":"2024-04-02T09:15:26.796160+0000","last_peered":"2024-04-02T09:15:26.796160+0000","last_clean":"2024-04-02T09:15:26.796160+0000","last_became_active":"2024-04-02T09:15:26.788699+0000","last_became_peered":"2024-04-02T09:15:26.788699+0000","last_unstale":"2024-04-02T09:15:26.796160+0000","last_undegraded":"2024-04-02T09:15:26.796160+0000","last_fullsized":"2024-04-02T09:15:26.796160+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:41:59.952558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.793942+0000","last_change":"2024-04-02T09:15:26.788194+0000","last_active":"2024-04-02T09:15:26.793942+0000","last_peered":"2024-04-02T09:15:26.793942+0000","last_clean":"2024-04-02T09:15:26.793942+0000","last_became_active":"2024-04-02T09:15:26.787942+0000","last_became_peered":"2024-04-02T09:15:26.787942+0000","last_unstale":"2024-04-02T09:15:26.793942+0000","last_undegraded":"2024-04-02T09:15:26.793942+0000","last_fullsized":"2024-04-02T09:15:26.793942+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:07.926116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.803258+0000","last_change":"2024-04-02T09:15:26.799129+0000","last_active":"2024-04-02T09:15:26.803258+0000","last_peered":"2024-04-02T09:15:26.803258+0000","last_clean":"2024-04-02T09:15:26.803258+0000","last_became_active":"2024-04-02T09:15:26.798383+0000","last_became_peered":"2024-04-02T09:15:26.798383+0000","last_unstale":"2024-04-02T09:15:26.803258+0000","last_undegraded":"2024-04-02T09:15:26.803258+0000","last_fullsized":"2024-04-02T09:15:26.803258+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:04:05.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.798878+0000","last_change":"2024-04-02T09:15:26.788186+0000","last_active":"2024-04-02T09:15:26.798878+0000","last_peered":"2024-04-02T09:15:26.798878+0000","last_clean":"2024-04-02T09:15:26.798878+0000","last_became_active":"2024-04-02T09:15:26.787463+0000","last_became_peered":"2024-04-02T09:15:26.787463+0000","last_unstale":"2024-04-02T09:15:26.798878+0000","last_undegraded":"2024-04-02T09:15:26.798878+0000","last_fullsized":"2024-04-02T09:15:26.798878+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:15:38.697300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811499+0000","last_change":"2024-04-02T09:15:26.806340+0000","last_active":"2024-04-02T09:15:26.811499+0000","last_peered":"2024-04-02T09:15:26.811499+0000","last_clean":"2024-04-02T09:15:26.811499+0000","last_became_active":"2024-04-02T09:15:26.806058+0000","last_became_peered":"2024-04-02T09:15:26.806058+0000","last_unstale":"2024-04-02T09:15:26.811499+0000","last_undegraded":"2024-04-02T09:15:26.811499+0000","last_fullsized":"2024-04-02T09:15:26.811499+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:40:16.336401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.810518+0000","last_change":"2024-04-02T09:15:26.805203+0000","last_active":"2024-04-02T09:15:26.810518+0000","last_peered":"2024-04-02T09:15:26.810518+0000","last_clean":"2024-04-02T09:15:26.810518+0000","last_became_active":"2024-04-02T09:15:26.804903+0000","last_became_peered":"2024-04-02T09:15:26.804903+0000","last_unstale":"2024-04-02T09:15:26.810518+0000","last_undegraded":"2024-04-02T09:15:26.810518+0000","last_fullsized":"2024-04-02T09:15:26.810518+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:12:30.918942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811328+0000","last_change":"2024-04-02T09:15:26.806543+0000","last_active":"2024-04-02T09:15:26.811328+0000","last_peered":"2024-04-02T09:15:26.811328+0000","last_clean":"2024-04-02T09:15:26.811328+0000","last_became_active":"2024-04-02T09:15:26.802757+0000","last_became_peered":"2024-04-02T09:15:26.802757+0000","last_unstale":"2024-04-02T09:15:26.811328+0000","last_undegraded":"2024-04-02T09:15:26.811328+0000","last_fullsized":"2024-04-02T09:15:26.811328+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:15:58.057433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.802727+0000","last_change":"2024-04-02T09:15:26.801740+0000","last_active":"2024-04-02T09:15:26.802727+0000","last_peered":"2024-04-02T09:15:26.802727+0000","last_clean":"2024-04-02T09:15:26.802727+0000","last_became_active":"2024-04-02T09:15:26.801102+0000","last_became_peered":"2024-04-02T09:15:26.801102+0000","last_unstale":"2024-04-02T09:15:26.802727+0000","last_undegraded":"2024-04-02T09:15:26.802727+0000","last_fullsized":"2024-04-02T09:15:26.802727+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:17:59.811955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.803023+0000","last_change":"2024-04-02T09:15:26.800643+0000","last_active":"2024-04-02T09:15:26.803023+0000","last_peered":"2024-04-02T09:15:26.803023+0000","last_clean":"2024-04-02T09:15:26.803023+0000","last_became_active":"2024-04-02T09:15:26.800383+0000","last_became_peered":"2024-04-02T09:15:26.800383+0000","last_unstale":"2024-04-02T09:15:26.803023+0000","last_undegraded":"2024-04-02T09:15:26.803023+0000","last_fullsized":"2024-04-02T09:15:26.803023+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:49:30.395362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.795777+0000","last_change":"2024-04-02T09:15:26.791824+0000","last_active":"2024-04-02T09:15:26.795777+0000","last_peered":"2024-04-02T09:15:26.795777+0000","last_clean":"2024-04-02T09:15:26.795777+0000","last_became_active":"2024-04-02T09:15:26.788155+0000","last_became_peered":"2024-04-02T09:15:26.788155+0000","last_unstale":"2024-04-02T09:15:26.795777+0000","last_undegraded":"2024-04-02T09:15:26.795777+0000","last_fullsized":"2024-04-02T09:15:26.795777+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:46.150941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.798900+0000","last_change":"2024-04-02T09:15:26.788373+0000","last_active":"2024-04-02T09:15:26.798900+0000","last_peered":"2024-04-02T09:15:26.798900+0000","last_clean":"2024-04-02T09:15:26.798900+0000","last_became_active":"2024-04-02T09:15:26.787892+0000","last_became_peered":"2024-04-02T09:15:26.787892+0000","last_unstale":"2024-04-02T09:15:26.798900+0000","last_undegraded":"2024-04-02T09:15:26.798900+0000","last_fullsized":"2024-04-02T09:15:26.798900+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:24.606743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.792571+0000","last_change":"2024-04-02T09:15:26.787474+0000","last_active":"2024-04-02T09:15:26.792571+0000","last_peered":"2024-04-02T09:15:26.792571+0000","last_clean":"2024-04-02T09:15:26.792571+0000","last_became_active":"2024-04-02T09:15:26.787219+0000","last_became_peered":"2024-04-02T09:15:26.787219+0000","last_unstale":"2024-04-02T09:15:26.792571+0000","last_undegraded":"2024-04-02T09:15:26.792571+0000","last_fullsized":"2024-04-02T09:15:26.792571+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:25:19.925731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.801005+0000","last_change":"2024-04-02T09:15:26.799255+0000","last_active":"2024-04-02T09:15:26.801005+0000","last_peered":"2024-04-02T09:15:26.801005+0000","last_clean":"2024-04-02T09:15:26.801005+0000","last_became_active":"2024-04-02T09:15:26.798619+0000","last_became_peered":"2024-04-02T09:15:26.798619+0000","last_unstale":"2024-04-02T09:15:26.801005+0000","last_undegraded":"2024-04-02T09:15:26.801005+0000","last_fullsized":"2024-04-02T09:15:26.801005+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:26:39.847255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811983+0000","last_change":"2024-04-02T09:15:26.807667+0000","last_active":"2024-04-02T09:15:26.811983+0000","last_peered":"2024-04-02T09:15:26.811983+0000","last_clean":"2024-04-02T09:15:26.811983+0000","last_became_active":"2024-04-02T09:15:26.806815+0000","last_became_peered":"2024-04-02T09:15:26.806815+0000","last_unstale":"2024-04-02T09:15:26.811983+0000","last_undegraded":"2024-04-02T09:15:26.811983+0000","last_fullsized":"2024-04-02T09:15:26.811983+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:46:34.730490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.804790+0000","last_change":"2024-04-02T09:15:26.802754+0000","last_active":"2024-04-02T09:15:26.804790+0000","last_peered":"2024-04-02T09:15:26.804790+0000","last_clean":"2024-04-02T09:15:26.804790+0000","last_became_active":"2024-04-02T09:15:26.802265+0000","last_became_peered":"2024-04-02T09:15:26.802265+0000","last_unstale":"2024-04-02T09:15:26.804790+0000","last_undegraded":"2024-04-02T09:15:26.804790+0000","last_fullsized":"2024-04-02T09:15:26.804790+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:16:53.826558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1f","version":"278'4","reported_seq":14,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811089+0000","last_change":"2024-04-02T09:15:26.284648+0000","last_active":"2024-04-02T09:15:26.811089+0000","last_peered":"2024-04-02T09:15:26.811089+0000","last_clean":"2024-04-02T09:15:26.811089+0000","last_became_active":"2024-04-02T09:15:26.284387+0000","last_became_peered":"2024-04-02T09:15:26.284387+0000","last_unstale":"2024-04-02T09:15:26.811089+0000","last_undegraded":"2024-04-02T09:15:26.811089+0000","last_fullsized":"2024-04-02T09:15:26.811089+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:21:21.487014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.796793+0000","last_change":"2024-04-02T09:15:26.795544+0000","last_active":"2024-04-02T09:15:26.796793+0000","last_peered":"2024-04-02T09:15:26.796793+0000","last_clean":"2024-04-02T09:15:26.796793+0000","last_became_active":"2024-04-02T09:15:26.794302+0000","last_became_peered":"2024-04-02T09:15:26.794302+0000","last_unstale":"2024-04-02T09:15:26.796793+0000","last_undegraded":"2024-04-02T09:15:26.796793+0000","last_fullsized":"2024-04-02T09:15:26.796793+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:30:49.865298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1e","version":"278'1","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812290+0000","last_change":"2024-04-02T09:15:26.285252+0000","last_active":"2024-04-02T09:15:26.812290+0000","last_peered":"2024-04-02T09:15:26.812290+0000","last_clean":"2024-04-02T09:15:26.812290+0000","last_became_active":"2024-04-02T09:15:26.284893+0000","last_became_peered":"2024-04-02T09:15:26.284893+0000","last_unstale":"2024-04-02T09:15:26.812290+0000","last_undegraded":"2024-04-02T09:15:26.812290+0000","last_fullsized":"2024-04-02T09:15:26.812290+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:51:06.977972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811676+0000","last_change":"2024-04-02T09:15:26.802849+0000","last_active":"2024-04-02T09:15:26.811676+0000","last_peered":"2024-04-02T09:15:26.811676+0000","last_clean":"2024-04-02T09:15:26.811676+0000","last_became_active":"2024-04-02T09:15:26.801678+0000","last_became_peered":"2024-04-02T09:15:26.801678+0000","last_unstale":"2024-04-02T09:15:26.811676+0000","last_undegraded":"2024-04-02T09:15:26.811676+0000","last_fullsized":"2024-04-02T09:15:26.811676+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:22:40.762308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"278'2","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.810022+0000","last_change":"2024-04-02T09:15:26.285953+0000","last_active":"2024-04-02T09:15:26.810022+0000","last_peered":"2024-04-02T09:15:26.810022+0000","last_clean":"2024-04-02T09:15:26.810022+0000","last_became_active":"2024-04-02T09:15:26.285628+0000","last_became_peered":"2024-04-02T09:15:26.285628+0000","last_unstale":"2024-04-02T09:15:26.810022+0000","last_undegraded":"2024-04-02T09:15:26.810022+0000","last_fullsized":"2024-04-02T09:15:26.810022+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:53.495029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.797688+0000","last_change":"2024-04-02T09:15:26.791532+0000","last_active":"2024-04-02T09:15:26.797688+0000","last_peered":"2024-04-02T09:15:26.797688+0000","last_clean":"2024-04-02T09:15:26.797688+0000","last_became_active":"2024-04-02T09:15:26.791239+0000","last_became_peered":"2024-04-02T09:15:26.791239+0000","last_unstale":"2024-04-02T09:15:26.797688+0000","last_undegraded":"2024-04-02T09:15:26.797688+0000","last_fullsized":"2024-04-02T09:15:26.797688+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:06.126308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1c","version":"278'2","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.806677+0000","last_change":"2024-04-02T09:15:26.283742+0000","last_active":"2024-04-02T09:15:26.806677+0000","last_peered":"2024-04-02T09:15:26.806677+0000","last_clean":"2024-04-02T09:15:26.806677+0000","last_became_active":"2024-04-02T09:15:26.283332+0000","last_became_peered":"2024-04-02T09:15:26.283332+0000","last_unstale":"2024-04-02T09:15:26.806677+0000","last_undegraded":"2024-04-02T09:15:26.806677+0000","last_fullsized":"2024-04-02T09:15:26.806677+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:14:41.653521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.802995+0000","last_change":"2024-04-02T09:15:26.798223+0000","last_active":"2024-04-02T09:15:26.802995+0000","last_peered":"2024-04-02T09:15:26.802995+0000","last_clean":"2024-04-02T09:15:26.802995+0000","last_became_active":"2024-04-02T09:15:26.797936+0000","last_became_peered":"2024-04-02T09:15:26.797936+0000","last_unstale":"2024-04-02T09:15:26.802995+0000","last_undegraded":"2024-04-02T09:15:26.802995+0000","last_fullsized":"2024-04-02T09:15:26.802995+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:15:38.697300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1b","version":"278'1","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.806850+0000","last_change":"2024-04-02T09:15:26.291916+0000","last_active":"2024-04-02T09:15:26.806850+0000","last_peered":"2024-04-02T09:15:26.806850+0000","last_clean":"2024-04-02T09:15:26.806850+0000","last_became_active":"2024-04-02T09:15:26.291463+0000","last_became_peered":"2024-04-02T09:15:26.291463+0000","last_unstale":"2024-04-02T09:15:26.806850+0000","last_undegraded":"2024-04-02T09:15:26.806850+0000","last_fullsized":"2024-04-02T09:15:26.806850+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:16:09.150518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.809505+0000","last_change":"2024-04-02T09:15:26.807350+0000","last_active":"2024-04-02T09:15:26.809505+0000","last_peered":"2024-04-02T09:15:26.809505+0000","last_clean":"2024-04-02T09:15:26.809505+0000","last_became_active":"2024-04-02T09:15:26.806848+0000","last_became_peered":"2024-04-02T09:15:26.806848+0000","last_unstale":"2024-04-02T09:15:26.809505+0000","last_undegraded":"2024-04-02T09:15:26.809505+0000","last_fullsized":"2024-04-02T09:15:26.809505+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:08:24.606743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1a","version":"278'1","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.793785+0000","last_change":"2024-04-02T09:15:26.290746+0000","last_active":"2024-04-02T09:15:26.793785+0000","last_peered":"2024-04-02T09:15:26.793785+0000","last_clean":"2024-04-02T09:15:26.793785+0000","last_became_active":"2024-04-02T09:15:26.286464+0000","last_became_peered":"2024-04-02T09:15:26.286464+0000","last_unstale":"2024-04-02T09:15:26.793785+0000","last_undegraded":"2024-04-02T09:15:26.793785+0000","last_fullsized":"2024-04-02T09:15:26.793785+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:02:00.543263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811348+0000","last_change":"2024-04-02T09:15:26.807360+0000","last_active":"2024-04-02T09:15:26.811348+0000","last_peered":"2024-04-02T09:15:26.811348+0000","last_clean":"2024-04-02T09:15:26.811348+0000","last_became_active":"2024-04-02T09:15:26.805796+0000","last_became_peered":"2024-04-02T09:15:26.805796+0000","last_unstale":"2024-04-02T09:15:26.811348+0000","last_undegraded":"2024-04-02T09:15:26.811348+0000","last_fullsized":"2024-04-02T09:15:26.811348+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:18:39.670018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.791937+0000","last_change":"2024-04-02T09:15:26.292532+0000","last_active":"2024-04-02T09:15:26.791937+0000","last_peered":"2024-04-02T09:15:26.791937+0000","last_clean":"2024-04-02T09:15:26.791937+0000","last_became_active":"2024-04-02T09:15:26.292118+0000","last_became_peered":"2024-04-02T09:15:26.292118+0000","last_unstale":"2024-04-02T09:15:26.791937+0000","last_undegraded":"2024-04-02T09:15:26.791937+0000","last_fullsized":"2024-04-02T09:15:26.791937+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:42:45.145585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812280+0000","last_change":"2024-04-02T09:15:26.803471+0000","last_active":"2024-04-02T09:15:26.812280+0000","last_peered":"2024-04-02T09:15:26.812280+0000","last_clean":"2024-04-02T09:15:26.812280+0000","last_became_active":"2024-04-02T09:15:26.801873+0000","last_became_peered":"2024-04-02T09:15:26.801873+0000","last_unstale":"2024-04-02T09:15:26.812280+0000","last_undegraded":"2024-04-02T09:15:26.812280+0000","last_fullsized":"2024-04-02T09:15:26.812280+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:12:30.918942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.780729+0000","last_change":"2024-04-02T09:15:26.287003+0000","last_active":"2024-04-02T09:15:26.780729+0000","last_peered":"2024-04-02T09:15:26.780729+0000","last_clean":"2024-04-02T09:15:26.780729+0000","last_became_active":"2024-04-02T09:15:26.286719+0000","last_became_peered":"2024-04-02T09:15:26.286719+0000","last_unstale":"2024-04-02T09:15:26.780729+0000","last_undegraded":"2024-04-02T09:15:26.780729+0000","last_fullsized":"2024-04-02T09:15:26.780729+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:32:07.104940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.803539+0000","last_change":"2024-04-02T09:15:26.799013+0000","last_active":"2024-04-02T09:15:26.803539+0000","last_peered":"2024-04-02T09:15:26.803539+0000","last_clean":"2024-04-02T09:15:26.803539+0000","last_became_active":"2024-04-02T09:15:26.798762+0000","last_became_peered":"2024-04-02T09:15:26.798762+0000","last_unstale":"2024-04-02T09:15:26.803539+0000","last_undegraded":"2024-04-02T09:15:26.803539+0000","last_fullsized":"2024-04-02T09:15:26.803539+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:40:17.423928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.783700+0000","last_change":"2024-04-02T09:15:26.278451+0000","last_active":"2024-04-02T09:15:26.783700+0000","last_peered":"2024-04-02T09:15:26.783700+0000","last_clean":"2024-04-02T09:15:26.783700+0000","last_became_active":"2024-04-02T09:15:26.278150+0000","last_became_peered":"2024-04-02T09:15:26.278150+0000","last_unstale":"2024-04-02T09:15:26.783700+0000","last_undegraded":"2024-04-02T09:15:26.783700+0000","last_fullsized":"2024-04-02T09:15:26.783700+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:29:33.555261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.792390+0000","last_change":"2024-04-02T09:15:26.786896+0000","last_active":"2024-04-02T09:15:26.792390+0000","last_peered":"2024-04-02T09:15:26.792390+0000","last_clean":"2024-04-02T09:15:26.792390+0000","last_became_active":"2024-04-02T09:15:26.786634+0000","last_became_peered":"2024-04-02T09:15:26.786634+0000","last_unstale":"2024-04-02T09:15:26.792390+0000","last_undegraded":"2024-04-02T09:15:26.792390+0000","last_fullsized":"2024-04-02T09:15:26.792390+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:44:05.340509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.776745+0000","last_change":"2024-04-02T09:15:26.288958+0000","last_active":"2024-04-02T09:15:26.776745+0000","last_peered":"2024-04-02T09:15:26.776745+0000","last_clean":"2024-04-02T09:15:26.776745+0000","last_became_active":"2024-04-02T09:15:26.288713+0000","last_became_peered":"2024-04-02T09:15:26.288713+0000","last_unstale":"2024-04-02T09:15:26.776745+0000","last_undegraded":"2024-04-02T09:15:26.776745+0000","last_fullsized":"2024-04-02T09:15:26.776745+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:04:15.823260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.798491+0000","last_change":"2024-04-02T09:15:26.793442+0000","last_active":"2024-04-02T09:15:26.798491+0000","last_peered":"2024-04-02T09:15:26.798491+0000","last_clean":"2024-04-02T09:15:26.798491+0000","last_became_active":"2024-04-02T09:15:26.792538+0000","last_became_peered":"2024-04-02T09:15:26.792538+0000","last_unstale":"2024-04-02T09:15:26.798491+0000","last_undegraded":"2024-04-02T09:15:26.798491+0000","last_fullsized":"2024-04-02T09:15:26.798491+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:03:29.044095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.15","version":"278'3","reported_seq":13,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.808899+0000","last_change":"2024-04-02T09:15:26.281980+0000","last_active":"2024-04-02T09:15:26.808899+0000","last_peered":"2024-04-02T09:15:26.808899+0000","last_clean":"2024-04-02T09:15:26.808899+0000","last_became_active":"2024-04-02T09:15:26.281710+0000","last_became_peered":"2024-04-02T09:15:26.281710+0000","last_unstale":"2024-04-02T09:15:26.808899+0000","last_undegraded":"2024-04-02T09:15:26.808899+0000","last_fullsized":"2024-04-02T09:15:26.808899+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:18:21.078476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.803906+0000","last_change":"2024-04-02T09:15:26.795494+0000","last_active":"2024-04-02T09:15:26.803906+0000","last_peered":"2024-04-02T09:15:26.803906+0000","last_clean":"2024-04-02T09:15:26.803906+0000","last_became_active":"2024-04-02T09:15:26.795201+0000","last_became_peered":"2024-04-02T09:15:26.795201+0000","last_unstale":"2024-04-02T09:15:26.803906+0000","last_undegraded":"2024-04-02T09:15:26.803906+0000","last_fullsized":"2024-04-02T09:15:26.803906+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:45.732192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.14","version":"278'6","reported_seq":16,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.813574+0000","last_change":"2024-04-02T09:15:26.292478+0000","last_active":"2024-04-02T09:15:26.813574+0000","last_peered":"2024-04-02T09:15:26.813574+0000","last_clean":"2024-04-02T09:15:26.813574+0000","last_became_active":"2024-04-02T09:15:26.292060+0000","last_became_peered":"2024-04-02T09:15:26.292060+0000","last_unstale":"2024-04-02T09:15:26.813574+0000","last_undegraded":"2024-04-02T09:15:26.813574+0000","last_fullsized":"2024-04-02T09:15:26.813574+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:11:48.754175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.796939+0000","last_change":"2024-04-02T09:15:26.789695+0000","last_active":"2024-04-02T09:15:26.796939+0000","last_peered":"2024-04-02T09:15:26.796939+0000","last_clean":"2024-04-02T09:15:26.796939+0000","last_became_active":"2024-04-02T09:15:26.789419+0000","last_became_peered":"2024-04-02T09:15:26.789419+0000","last_unstale":"2024-04-02T09:15:26.796939+0000","last_undegraded":"2024-04-02T09:15:26.796939+0000","last_fullsized":"2024-04-02T09:15:26.796939+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:41:06.039590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.13","version":"278'2","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812576+0000","last_change":"2024-04-02T09:15:26.286966+0000","last_active":"2024-04-02T09:15:26.812576+0000","last_peered":"2024-04-02T09:15:26.812576+0000","last_clean":"2024-04-02T09:15:26.812576+0000","last_became_active":"2024-04-02T09:15:26.286699+0000","last_became_peered":"2024-04-02T09:15:26.286699+0000","last_unstale":"2024-04-02T09:15:26.812576+0000","last_undegraded":"2024-04-02T09:15:26.812576+0000","last_fullsized":"2024-04-02T09:15:26.812576+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:29:05.383512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.793196+0000","last_change":"2024-04-02T09:15:26.787881+0000","last_active":"2024-04-02T09:15:26.793196+0000","last_peered":"2024-04-02T09:15:26.793196+0000","last_clean":"2024-04-02T09:15:26.793196+0000","last_became_active":"2024-04-02T09:15:26.787595+0000","last_became_peered":"2024-04-02T09:15:26.787595+0000","last_unstale":"2024-04-02T09:15:26.793196+0000","last_undegraded":"2024-04-02T09:15:26.793196+0000","last_fullsized":"2024-04-02T09:15:26.793196+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:09:17.511200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.794749+0000","last_change":"2024-04-02T09:15:26.288973+0000","last_active":"2024-04-02T09:15:26.794749+0000","last_peered":"2024-04-02T09:15:26.794749+0000","last_clean":"2024-04-02T09:15:26.794749+0000","last_became_active":"2024-04-02T09:15:26.288716+0000","last_became_peered":"2024-04-02T09:15:26.288716+0000","last_unstale":"2024-04-02T09:15:26.794749+0000","last_undegraded":"2024-04-02T09:15:26.794749+0000","last_fullsized":"2024-04-02T09:15:26.794749+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:27:06.920760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811363+0000","last_change":"2024-04-02T09:15:26.807483+0000","last_active":"2024-04-02T09:15:26.811363+0000","last_peered":"2024-04-02T09:15:26.811363+0000","last_clean":"2024-04-02T09:15:26.811363+0000","last_became_active":"2024-04-02T09:15:26.807121+0000","last_became_peered":"2024-04-02T09:15:26.807121+0000","last_unstale":"2024-04-02T09:15:26.811363+0000","last_undegraded":"2024-04-02T09:15:26.811363+0000","last_fullsized":"2024-04-02T09:15:26.811363+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:52:47.662567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.781582+0000","last_change":"2024-04-02T09:15:26.290229+0000","last_active":"2024-04-02T09:15:26.781582+0000","last_peered":"2024-04-02T09:15:26.781582+0000","last_clean":"2024-04-02T09:15:26.781582+0000","last_became_active":"2024-04-02T09:15:26.289273+0000","last_became_peered":"2024-04-02T09:15:26.289273+0000","last_unstale":"2024-04-02T09:15:26.781582+0000","last_undegraded":"2024-04-02T09:15:26.781582+0000","last_fullsized":"2024-04-02T09:15:26.781582+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:40:16.418572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.810538+0000","last_change":"2024-04-02T09:15:26.808563+0000","last_active":"2024-04-02T09:15:26.810538+0000","last_peered":"2024-04-02T09:15:26.810538+0000","last_clean":"2024-04-02T09:15:26.810538+0000","last_became_active":"2024-04-02T09:15:26.808283+0000","last_became_peered":"2024-04-02T09:15:26.808283+0000","last_unstale":"2024-04-02T09:15:26.810538+0000","last_undegraded":"2024-04-02T09:15:26.810538+0000","last_fullsized":"2024-04-02T09:15:26.810538+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T13:46:34.730490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.10","version":"278'8","reported_seq":18,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812527+0000","last_change":"2024-04-02T09:15:26.289161+0000","last_active":"2024-04-02T09:15:26.812527+0000","last_peered":"2024-04-02T09:15:26.812527+0000","last_clean":"2024-04-02T09:15:26.812527+0000","last_became_active":"2024-04-02T09:15:26.288231+0000","last_became_peered":"2024-04-02T09:15:26.288231+0000","last_unstale":"2024-04-02T09:15:26.812527+0000","last_undegraded":"2024-04-02T09:15:26.812527+0000","last_fullsized":"2024-04-02T09:15:26.812527+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:49:29.546689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.799109+0000","last_change":"2024-04-02T09:15:26.795008+0000","last_active":"2024-04-02T09:15:26.799109+0000","last_peered":"2024-04-02T09:15:26.799109+0000","last_clean":"2024-04-02T09:15:26.799109+0000","last_became_active":"2024-04-02T09:15:26.794739+0000","last_became_peered":"2024-04-02T09:15:26.794739+0000","last_unstale":"2024-04-02T09:15:26.799109+0000","last_undegraded":"2024-04-02T09:15:26.799109+0000","last_fullsized":"2024-04-02T09:15:26.799109+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:45.732192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.f","version":"278'1","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.808833+0000","last_change":"2024-04-02T09:15:26.282846+0000","last_active":"2024-04-02T09:15:26.808833+0000","last_peered":"2024-04-02T09:15:26.808833+0000","last_clean":"2024-04-02T09:15:26.808833+0000","last_became_active":"2024-04-02T09:15:26.281867+0000","last_became_peered":"2024-04-02T09:15:26.281867+0000","last_unstale":"2024-04-02T09:15:26.808833+0000","last_undegraded":"2024-04-02T09:15:26.808833+0000","last_fullsized":"2024-04-02T09:15:26.808833+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:49:51.579415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811777+0000","last_change":"2024-04-02T09:15:26.810130+0000","last_active":"2024-04-02T09:15:26.811777+0000","last_peered":"2024-04-02T09:15:26.811777+0000","last_clean":"2024-04-02T09:15:26.811777+0000","last_became_active":"2024-04-02T09:15:26.809019+0000","last_became_peered":"2024-04-02T09:15:26.809019+0000","last_unstale":"2024-04-02T09:15:26.811777+0000","last_undegraded":"2024-04-02T09:15:26.811777+0000","last_fullsized":"2024-04-02T09:15:26.811777+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:23:03.344490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.e","version":"278'2","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.808322+0000","last_change":"2024-04-02T09:15:26.291584+0000","last_active":"2024-04-02T09:15:26.808322+0000","last_peered":"2024-04-02T09:15:26.808322+0000","last_clean":"2024-04-02T09:15:26.808322+0000","last_became_active":"2024-04-02T09:15:26.290637+0000","last_became_peered":"2024-04-02T09:15:26.290637+0000","last_unstale":"2024-04-02T09:15:26.808322+0000","last_undegraded":"2024-04-02T09:15:26.808322+0000","last_fullsized":"2024-04-02T09:15:26.808322+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:17:55.858050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812232+0000","last_change":"2024-04-02T09:15:26.805111+0000","last_active":"2024-04-02T09:15:26.812232+0000","last_peered":"2024-04-02T09:15:26.812232+0000","last_clean":"2024-04-02T09:15:26.812232+0000","last_became_active":"2024-04-02T09:15:26.803334+0000","last_became_peered":"2024-04-02T09:15:26.803334+0000","last_unstale":"2024-04-02T09:15:26.812232+0000","last_undegraded":"2024-04-02T09:15:26.812232+0000","last_fullsized":"2024-04-02T09:15:26.812232+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:59:02.406432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.d","version":"278'1","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.809946+0000","last_change":"2024-04-02T09:15:26.284703+0000","last_active":"2024-04-02T09:15:26.809946+0000","last_peered":"2024-04-02T09:15:26.809946+0000","last_clean":"2024-04-02T09:15:26.809946+0000","last_became_active":"2024-04-02T09:15:26.284431+0000","last_became_peered":"2024-04-02T09:15:26.284431+0000","last_unstale":"2024-04-02T09:15:26.809946+0000","last_undegraded":"2024-04-02T09:15:26.809946+0000","last_fullsized":"2024-04-02T09:15:26.809946+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:02:58.625853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.810493+0000","last_change":"2024-04-02T09:15:26.808276+0000","last_active":"2024-04-02T09:15:26.810493+0000","last_peered":"2024-04-02T09:15:26.810493+0000","last_clean":"2024-04-02T09:15:26.810493+0000","last_became_active":"2024-04-02T09:15:26.807091+0000","last_became_peered":"2024-04-02T09:15:26.807091+0000","last_unstale":"2024-04-02T09:15:26.810493+0000","last_undegraded":"2024-04-02T09:15:26.810493+0000","last_fullsized":"2024-04-02T09:15:26.810493+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:11:45.951714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.786858+0000","last_change":"2024-04-02T09:15:26.288044+0000","last_active":"2024-04-02T09:15:26.786858+0000","last_peered":"2024-04-02T09:15:26.786858+0000","last_clean":"2024-04-02T09:15:26.786858+0000","last_became_active":"2024-04-02T09:15:26.287498+0000","last_became_peered":"2024-04-02T09:15:26.287498+0000","last_unstale":"2024-04-02T09:15:26.786858+0000","last_undegraded":"2024-04-02T09:15:26.786858+0000","last_fullsized":"2024-04-02T09:15:26.786858+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:02:32.179689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.793760+0000","last_change":"2024-04-02T09:15:26.786372+0000","last_active":"2024-04-02T09:15:26.793760+0000","last_peered":"2024-04-02T09:15:26.793760+0000","last_clean":"2024-04-02T09:15:26.793760+0000","last_became_active":"2024-04-02T09:15:26.786126+0000","last_became_peered":"2024-04-02T09:15:26.786126+0000","last_unstale":"2024-04-02T09:15:26.793760+0000","last_undegraded":"2024-04-02T09:15:26.793760+0000","last_fullsized":"2024-04-02T09:15:26.793760+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:07:01.749821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.b","version":"278'2","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.810397+0000","last_change":"2024-04-02T09:15:26.288142+0000","last_active":"2024-04-02T09:15:26.810397+0000","last_peered":"2024-04-02T09:15:26.810397+0000","last_clean":"2024-04-02T09:15:26.810397+0000","last_became_active":"2024-04-02T09:15:26.287881+0000","last_became_peered":"2024-04-02T09:15:26.287881+0000","last_unstale":"2024-04-02T09:15:26.810397+0000","last_undegraded":"2024-04-02T09:15:26.810397+0000","last_fullsized":"2024-04-02T09:15:26.810397+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:12:43.305299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812300+0000","last_change":"2024-04-02T09:15:26.801996+0000","last_active":"2024-04-02T09:15:26.812300+0000","last_peered":"2024-04-02T09:15:26.812300+0000","last_clean":"2024-04-02T09:15:26.812300+0000","last_became_active":"2024-04-02T09:15:26.801743+0000","last_became_peered":"2024-04-02T09:15:26.801743+0000","last_unstale":"2024-04-02T09:15:26.812300+0000","last_undegraded":"2024-04-02T09:15:26.812300+0000","last_fullsized":"2024-04-02T09:15:26.812300+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:28:28.895276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.786769+0000","last_change":"2024-04-02T09:15:26.284022+0000","last_active":"2024-04-02T09:15:26.786769+0000","last_peered":"2024-04-02T09:15:26.786769+0000","last_clean":"2024-04-02T09:15:26.786769+0000","last_became_active":"2024-04-02T09:15:26.283734+0000","last_became_peered":"2024-04-02T09:15:26.283734+0000","last_unstale":"2024-04-02T09:15:26.786769+0000","last_undegraded":"2024-04-02T09:15:26.786769+0000","last_fullsized":"2024-04-02T09:15:26.786769+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:11:16.738763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.806228+0000","last_change":"2024-04-02T09:15:26.804380+0000","last_active":"2024-04-02T09:15:26.806228+0000","last_peered":"2024-04-02T09:15:26.806228+0000","last_clean":"2024-04-02T09:15:26.806228+0000","last_became_active":"2024-04-02T09:15:26.804080+0000","last_became_peered":"2024-04-02T09:15:26.804080+0000","last_unstale":"2024-04-02T09:15:26.806228+0000","last_undegraded":"2024-04-02T09:15:26.806228+0000","last_fullsized":"2024-04-02T09:15:26.806228+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:32:08.110296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.785833+0000","last_change":"2024-04-02T09:15:26.282673+0000","last_active":"2024-04-02T09:15:26.785833+0000","last_peered":"2024-04-02T09:15:26.785833+0000","last_clean":"2024-04-02T09:15:26.785833+0000","last_became_active":"2024-04-02T09:15:26.282410+0000","last_became_peered":"2024-04-02T09:15:26.282410+0000","last_unstale":"2024-04-02T09:15:26.785833+0000","last_undegraded":"2024-04-02T09:15:26.785833+0000","last_fullsized":"2024-04-02T09:15:26.785833+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:15:13.448698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.813838+0000","last_change":"2024-04-02T09:15:26.810951+0000","last_active":"2024-04-02T09:15:26.813838+0000","last_peered":"2024-04-02T09:15:26.813838+0000","last_clean":"2024-04-02T09:15:26.813838+0000","last_became_active":"2024-04-02T09:15:26.810327+0000","last_became_peered":"2024-04-02T09:15:26.810327+0000","last_unstale":"2024-04-02T09:15:26.813838+0000","last_undegraded":"2024-04-02T09:15:26.813838+0000","last_fullsized":"2024-04-02T09:15:26.813838+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:29:04.644247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.781029+0000","last_change":"2024-04-02T09:15:26.286166+0000","last_active":"2024-04-02T09:15:26.781029+0000","last_peered":"2024-04-02T09:15:26.781029+0000","last_clean":"2024-04-02T09:15:26.781029+0000","last_became_active":"2024-04-02T09:15:26.285895+0000","last_became_peered":"2024-04-02T09:15:26.285895+0000","last_unstale":"2024-04-02T09:15:26.781029+0000","last_undegraded":"2024-04-02T09:15:26.781029+0000","last_fullsized":"2024-04-02T09:15:26.781029+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:03:28.038739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.812391+0000","last_change":"2024-04-02T09:15:26.809682+0000","last_active":"2024-04-02T09:15:26.812391+0000","last_peered":"2024-04-02T09:15:26.812391+0000","last_clean":"2024-04-02T09:15:26.812391+0000","last_became_active":"2024-04-02T09:15:26.805991+0000","last_became_peered":"2024-04-02T09:15:26.805991+0000","last_unstale":"2024-04-02T09:15:26.812391+0000","last_undegraded":"2024-04-02T09:15:26.812391+0000","last_fullsized":"2024-04-02T09:15:26.812391+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:06:53.885091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.7","version":"278'2","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.811891+0000","last_change":"2024-04-02T09:15:26.285684+0000","last_active":"2024-04-02T09:15:26.811891+0000","last_peered":"2024-04-02T09:15:26.811891+0000","last_clean":"2024-04-02T09:15:26.811891+0000","last_became_active":"2024-04-02T09:15:26.284959+0000","last_became_peered":"2024-04-02T09:15:26.284959+0000","last_unstale":"2024-04-02T09:15:26.811891+0000","last_undegraded":"2024-04-02T09:15:26.811891+0000","last_fullsized":"2024-04-02T09:15:26.811891+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:57:19.421726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.792372+0000","last_change":"2024-04-02T09:15:26.785967+0000","last_active":"2024-04-02T09:15:26.792372+0000","last_peered":"2024-04-02T09:15:26.792372+0000","last_clean":"2024-04-02T09:15:26.792372+0000","last_became_active":"2024-04-02T09:15:26.785713+0000","last_became_peered":"2024-04-02T09:15:26.785713+0000","last_unstale":"2024-04-02T09:15:26.792372+0000","last_undegraded":"2024-04-02T09:15:26.792372+0000","last_fullsized":"2024-04-02T09:15:26.792372+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:20.157837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.783421+0000","last_change":"2024-04-02T09:15:26.289438+0000","last_active":"2024-04-02T09:15:26.783421+0000","last_peered":"2024-04-02T09:15:26.783421+0000","last_clean":"2024-04-02T09:15:26.783421+0000","last_became_active":"2024-04-02T09:15:26.289147+0000","last_became_peered":"2024-04-02T09:15:26.289147+0000","last_unstale":"2024-04-02T09:15:26.783421+0000","last_undegraded":"2024-04-02T09:15:26.783421+0000","last_fullsized":"2024-04-02T09:15:26.783421+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:14:12.082097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.809186+0000","last_change":"2024-04-02T09:15:26.806637+0000","last_active":"2024-04-02T09:15:26.809186+0000","last_peered":"2024-04-02T09:15:26.809186+0000","last_clean":"2024-04-02T09:15:26.809186+0000","last_became_active":"2024-04-02T09:15:26.804177+0000","last_became_peered":"2024-04-02T09:15:26.804177+0000","last_unstale":"2024-04-02T09:15:26.809186+0000","last_undegraded":"2024-04-02T09:15:26.809186+0000","last_fullsized":"2024-04-02T09:15:26.809186+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:48:46.501785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.781162+0000","last_change":"2024-04-02T09:15:26.290231+0000","last_active":"2024-04-02T09:15:26.781162+0000","last_peered":"2024-04-02T09:15:26.781162+0000","last_clean":"2024-04-02T09:15:26.781162+0000","last_became_active":"2024-04-02T09:15:26.289007+0000","last_became_peered":"2024-04-02T09:15:26.289007+0000","last_unstale":"2024-04-02T09:15:26.781162+0000","last_undegraded":"2024-04-02T09:15:26.781162+0000","last_fullsized":"2024-04-02T09:15:26.781162+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T09:56:44.726836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.796255+0000","last_change":"2024-04-02T09:15:26.793838+0000","last_active":"2024-04-02T09:15:26.796255+0000","last_peered":"2024-04-02T09:15:26.796255+0000","last_clean":"2024-04-02T09:15:26.796255+0000","last_became_active":"2024-04-02T09:15:26.791718+0000","last_became_peered":"2024-04-02T09:15:26.791718+0000","last_unstale":"2024-04-02T09:15:26.796255+0000","last_undegraded":"2024-04-02T09:15:26.796255+0000","last_fullsized":"2024-04-02T09:15:26.796255+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:17:56.863406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.780926+0000","last_change":"2024-04-02T09:15:26.281559+0000","last_active":"2024-04-02T09:15:26.780926+0000","last_peered":"2024-04-02T09:15:26.780926+0000","last_clean":"2024-04-02T09:15:26.780926+0000","last_became_active":"2024-04-02T09:15:26.280091+0000","last_became_peered":"2024-04-02T09:15:26.280091+0000","last_unstale":"2024-04-02T09:15:26.780926+0000","last_undegraded":"2024-04-02T09:15:26.780926+0000","last_fullsized":"2024-04-02T09:15:26.780926+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:00.549782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.803695+0000","last_change":"2024-04-02T09:15:26.795720+0000","last_active":"2024-04-02T09:15:26.803695+0000","last_peered":"2024-04-02T09:15:26.803695+0000","last_clean":"2024-04-02T09:15:26.803695+0000","last_became_active":"2024-04-02T09:15:26.795426+0000","last_became_peered":"2024-04-02T09:15:26.795426+0000","last_unstale":"2024-04-02T09:15:26.803695+0000","last_undegraded":"2024-04-02T09:15:26.803695+0000","last_fullsized":"2024-04-02T09:15:26.803695+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:52:35.854324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.790787+0000","last_change":"2024-04-02T09:15:26.287978+0000","last_active":"2024-04-02T09:15:26.790787+0000","last_peered":"2024-04-02T09:15:26.790787+0000","last_clean":"2024-04-02T09:15:26.790787+0000","last_became_active":"2024-04-02T09:15:26.287720+0000","last_became_peered":"2024-04-02T09:15:26.287720+0000","last_unstale":"2024-04-02T09:15:26.790787+0000","last_undegraded":"2024-04-02T09:15:26.790787+0000","last_fullsized":"2024-04-02T09:15:26.790787+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:30:19.152481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.805618+0000","last_change":"2024-04-02T09:15:26.802648+0000","last_active":"2024-04-02T09:15:26.805618+0000","last_peered":"2024-04-02T09:15:26.805618+0000","last_clean":"2024-04-02T09:15:26.805618+0000","last_became_active":"2024-04-02T09:15:26.802312+0000","last_became_peered":"2024-04-02T09:15:26.802312+0000","last_unstale":"2024-04-02T09:15:26.805618+0000","last_undegraded":"2024-04-02T09:15:26.805618+0000","last_fullsized":"2024-04-02T09:15:26.805618+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T12:29:34.560617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.794740+0000","last_change":"2024-04-02T09:15:26.292409+0000","last_active":"2024-04-02T09:15:26.794740+0000","last_peered":"2024-04-02T09:15:26.794740+0000","last_clean":"2024-04-02T09:15:26.794740+0000","last_became_active":"2024-04-02T09:15:26.292132+0000","last_became_peered":"2024-04-02T09:15:26.292132+0000","last_unstale":"2024-04-02T09:15:26.794740+0000","last_undegraded":"2024-04-02T09:15:26.794740+0000","last_fullsized":"2024-04-02T09:15:26.794740+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:52:14.862034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.803049+0000","last_change":"2024-04-02T09:15:26.797299+0000","last_active":"2024-04-02T09:15:26.803049+0000","last_peered":"2024-04-02T09:15:26.803049+0000","last_clean":"2024-04-02T09:15:26.803049+0000","last_became_active":"2024-04-02T09:15:26.797032+0000","last_became_peered":"2024-04-02T09:15:26.797032+0000","last_unstale":"2024-04-02T09:15:26.803049+0000","last_undegraded":"2024-04-02T09:15:26.803049+0000","last_fullsized":"2024-04-02T09:15:26.803049+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:01.555138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.792039+0000","last_change":"2024-04-02T09:15:26.289014+0000","last_active":"2024-04-02T09:15:26.792039+0000","last_peered":"2024-04-02T09:15:26.792039+0000","last_clean":"2024-04-02T09:15:26.792039+0000","last_became_active":"2024-04-02T09:15:26.288766+0000","last_became_peered":"2024-04-02T09:15:26.288766+0000","last_unstale":"2024-04-02T09:15:26.792039+0000","last_undegraded":"2024-04-02T09:15:26.792039+0000","last_fullsized":"2024-04-02T09:15:26.792039+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T10:25:18.920375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.793487+0000","last_change":"2024-04-02T09:15:26.786850+0000","last_active":"2024-04-02T09:15:26.793487+0000","last_peered":"2024-04-02T09:15:26.793487+0000","last_clean":"2024-04-02T09:15:26.793487+0000","last_became_active":"2024-04-02T09:15:26.786590+0000","last_became_peered":"2024-04-02T09:15:26.786590+0000","last_unstale":"2024-04-02T09:15:26.793487+0000","last_undegraded":"2024-04-02T09:15:26.793487+0000","last_fullsized":"2024-04-02T09:15:26.793487+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:46:26.535123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":10,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.782841+0000","last_change":"2024-04-02T09:15:26.282287+0000","last_active":"2024-04-02T09:15:26.782841+0000","last_peered":"2024-04-02T09:15:26.782841+0000","last_clean":"2024-04-02T09:15:26.782841+0000","last_became_active":"2024-04-02T09:15:26.281984+0000","last_became_peered":"2024-04-02T09:15:26.281984+0000","last_unstale":"2024-04-02T09:15:26.782841+0000","last_undegraded":"2024-04-02T09:15:26.782841+0000","last_fullsized":"2024-04-02T09:15:26.782841+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:25.256920+0000","last_clean_scrub_stamp":"2024-04-02T09:15:25.256920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T11:10:58.674355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":9,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.804744+0000","last_change":"2024-04-02T09:15:26.803112+0000","last_active":"2024-04-02T09:15:26.804744+0000","last_peered":"2024-04-02T09:15:26.804744+0000","last_clean":"2024-04-02T09:15:26.804744+0000","last_became_active":"2024-04-02T09:15:26.799381+0000","last_became_peered":"2024-04-02T09:15:26.799381+0000","last_unstale":"2024-04-02T09:15:26.804744+0000","last_undegraded":"2024-04-02T09:15:26.804744+0000","last_fullsized":"2024-04-02T09:15:26.804744+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T09:15:26.262276+0000","last_clean_scrub_stamp":"2024-04-02T09:15:26.262276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:46:26.535123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"268'403","reported_seq":1454,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-02T09:15:26.289732+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:15:26.289732+0000","last_peered":"2024-04-02T09:15:26.289732+0000","last_clean":"2024-04-02T09:15:26.289732+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:15:26.289732+0000","last_undegraded":"2024-04-02T09:15:26.289732+0000","last_fullsized":"2024-04-02T09:15:26.289732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60796,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311044,"statfs":{"total":96636764160,"available":96574509056,"internally_reserved":0,"allocated":3514368,"data_stored":1808858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.71899999999999997,"15min":0.71499999999999997},"min":{"1min":0.40500000000000003,"5min":0.309,"15min":0.309},"max":{"1min":1.115,"5min":8.016,"15min":8.016},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.70199999999999996,"15min":0.71299999999999997},"min":{"1min":0.48899999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.044,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.57299999999999995}]},{"osd":1,"last update":"Tue Apr 2 09:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.746,"15min":0.73699999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.38200000000000001},"max":{"1min":0.81299999999999994,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.68500000000000005,"15min":0.70399999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.84499999999999997,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.68999999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.76000000000000001,"15min":0.76100000000000001},"min":{"1min":0.44600000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.202,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.79300000000000004,"15min":0.80200000000000005},"min":{"1min":0.53100000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2470000000000001,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.77300000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.747,"15min":0.74399999999999999},"min":{"1min":0.49099999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0229999999999999,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.78800000000000003,"15min":0.76800000000000002},"min":{"1min":0.48199999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.91100000000000003,"5min":5.093,"15min":5.093},"last":0.50700000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67100000000000004,"15min":0.628},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86299999999999999,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.79900000000000004,"15min":0.749},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.052,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.75900000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.73099999999999998,"15min":0.69499999999999995},"min":{"1min":0.46400000000000002,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":1.036,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.76500000000000001,"15min":0.73599999999999999},"min":{"1min":0.436,"5min":0.42599999999999999,"15min":0.41099999999999998},"max":{"1min":0.95299999999999996,"5min":3.492,"15min":3.492},"last":0.74199999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.72099999999999997,"15min":0.69799999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.85499999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.77900000000000003,"15min":0.72199999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93200000000000005,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.79500000000000004}]}]},{"osd":6,"up_from":13,"seq":55834576015,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42760,"kb_used_data":3444,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329080,"statfs":{"total":96636764160,"available":96592977920,"internally_reserved":0,"allocated":3526656,"data_stored":1809544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.67200000000000004,"15min":0.70899999999999996},"min":{"1min":0.432,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81399999999999995,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.68000000000000005,"15min":0.72299999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.87,"5min":0.98399999999999999,"15min":1.0600000000000001},"last":0.66700000000000004}]},{"osd":1,"last update":"Tue Apr 2 09:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.67900000000000005,"15min":0.70599999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0149999999999999,"5min":1.044,"15min":1.212},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.69899999999999995,"15min":0.73599999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.88400000000000001,"5min":1.0860000000000001,"15min":1.1759999999999999},"last":0.73199999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.69799999999999995,"15min":0.73499999999999999},"min":{"1min":0.45200000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.85299999999999998,"5min":1.046,"15min":1.2789999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.68600000000000005,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.90000000000000002,"5min":0.99299999999999999,"15min":1.226},"last":0.69899999999999995}]},{"osd":3,"last update":"Tue Apr 2 09:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70899999999999996,"15min":0.745},"min":{"1min":0.52200000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.84399999999999997,"5min":1.141,"15min":1.2549999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.73399999999999999,"15min":0.76300000000000001},"min":{"1min":0.56399999999999995,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.96499999999999997,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.81899999999999995}]},{"osd":4,"last update":"Tue Apr 2 09:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.629,"15min":0.623},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86199999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.65900000000000003,"15min":0.66100000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.84899999999999998,"5min":1.1100000000000001,"15min":1.1879999999999999},"last":0.76500000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.65500000000000003,"15min":0.65300000000000002},"min":{"1min":0.48099999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.2669999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.51000000000000001,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":0.93500000000000005,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.877}]},{"osd":7,"last update":"Tue Apr 2 09:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.55300000000000005,"5min":0.48299999999999998,"15min":0.45400000000000001},"max":{"1min":0.92200000000000004,"5min":1.21,"15min":1.2370000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72999999999999998,"15min":0.73499999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94999999999999996,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.90600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576015,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56704,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315136,"statfs":{"total":96636764160,"available":96578699264,"internally_reserved":0,"allocated":3518464,"data_stored":1808928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.42099999999999999,"15min":0.443},"min":{"1min":0.28299999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73599999999999999,"5min":0.74199999999999999,"15min":0.76000000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.45100000000000001,"15min":0.47799999999999998},"min":{"1min":0.34300000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.61199999999999999,"5min":0.71199999999999997,"15min":0.876},"last":0.36099999999999999}]},{"osd":1,"last update":"Tue Apr 2 09:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46899999999999997,"15min":0.495},"min":{"1min":0.30499999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.61099999999999999,"5min":0.81699999999999995,"15min":0.90300000000000002},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53400000000000003,"15min":0.56399999999999995},"min":{"1min":0.41699999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.98499999999999999},"last":0.34300000000000003}]},{"osd":2,"last update":"Tue Apr 2 09:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.57799999999999996,"15min":0.61199999999999999},"min":{"1min":0.41799999999999998,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.70299999999999996,"5min":0.91600000000000004,"15min":0.93100000000000005},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55800000000000005,"15min":0.60299999999999998},"min":{"1min":0.38700000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67900000000000005,"5min":0.85099999999999998,"15min":1.0840000000000001},"last":0.495}]},{"osd":4,"last update":"Tue Apr 2 09:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71899999999999997,"15min":0.748},"min":{"1min":0.53000000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83099999999999996,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72799999999999998,"15min":0.76000000000000001},"min":{"1min":0.505,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.89500000000000002,"5min":1.018,"15min":1.1619999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Tue Apr 2 09:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73999999999999999,"15min":0.76900000000000002},"min":{"1min":0.52200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92800000000000005,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.74099999999999999,"15min":0.77500000000000002},"min":{"1min":0.61199999999999999,"5min":0.505,"15min":0.505},"max":{"1min":0.877,"5min":0.99099999999999999,"15min":1.266},"last":0.58999999999999997}]},{"osd":6,"last update":"Tue Apr 2 09:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.747,"15min":0.77800000000000002},"min":{"1min":0.54400000000000004,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.996,"5min":1.0489999999999999,"15min":1.284},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76700000000000002,"15min":0.80200000000000005},"min":{"1min":0.625,"5min":0.495,"15min":0.495},"max":{"1min":0.98099999999999998,"5min":1.0840000000000001,"15min":1.2},"last":0.59599999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.753,"15min":0.79700000000000004},"min":{"1min":0.51500000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.95399999999999996,"5min":1.125,"15min":1.3260000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.754,"5min":0.78000000000000003,"15min":0.81399999999999995},"min":{"1min":0.61399999999999999,"5min":0.495,"15min":0.495},"max":{"1min":0.94499999999999995,"5min":1.103,"15min":1.248},"last":0.60299999999999998}]}]},{"osd":0,"up_from":13,"seq":55834576016,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52580,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319260,"statfs":{"total":96636764160,"available":96582922240,"internally_reserved":0,"allocated":3497984,"data_stored":1800616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42899999999999999,"15min":0.44600000000000001},"min":{"1min":0.28199999999999997,"5min":0.26700000000000002,"15min":0.254},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.86699999999999999},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.46600000000000003,"15min":0.49199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.52200000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.29299999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49299999999999999,"15min":0.50600000000000001},"min":{"1min":0.36899999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.307},{"interface":"front","average":{"1min":0.443,"5min":0.51100000000000001,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.52800000000000002,"5min":0.746,"15min":0.999},"last":0.45000000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.52200000000000002,"15min":0.55100000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.312},"max":{"1min":0.75600000000000001,"5min":0.79800000000000004,"15min":0.85699999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58899999999999997,"15min":0.624},"min":{"1min":0.45200000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.81100000000000005,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.46100000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.78000000000000003,"15min":0.81699999999999995},"min":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.45600000000000002},"max":{"1min":1.0169999999999999,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.77700000000000002,"15min":0.80900000000000005},"min":{"1min":0.496,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.012,"5min":1.169,"15min":1.169},"last":0.70999999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.78000000000000003,"15min":0.81000000000000005},"min":{"1min":0.56899999999999995,"5min":0.54000000000000004,"15min":0.497},"max":{"1min":1.0840000000000001,"5min":1.145,"15min":1.232},"last":0.628},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.78400000000000003,"15min":0.81799999999999995},"min":{"1min":0.52600000000000002,"5min":0.497,"15min":0.497},"max":{"1min":0.98099999999999998,"5min":1.21,"15min":1.21},"last":0.58499999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.75800000000000001,"15min":0.78900000000000003},"min":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91600000000000004,"5min":1.1830000000000001,"15min":1.196},"last":0.497},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":0.79700000000000004},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.52900000000000003},"max":{"1min":0.95699999999999996,"5min":1.246,"15min":1.246},"last":0.55000000000000004}]},{"osd":7,"last update":"Tue Apr 2 09:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.81399999999999995,"15min":0.84099999999999997},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.99399999999999999,"5min":1.292,"15min":1.292},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.79600000000000004,"15min":0.83299999999999996},"min":{"1min":0.54100000000000004,"5min":0.499,"15min":0.499},"max":{"1min":1.046,"5min":1.1719999999999999,"15min":1.2130000000000001},"last":0.67600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834576016,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57284,"kb_used_data":4024,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314556,"statfs":{"total":96636764160,"available":96578105344,"internally_reserved":0,"allocated":4120576,"data_stored":2400790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.433,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66800000000000004,"5min":0.82199999999999995,"15min":0.82499999999999996},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48299999999999998,"15min":0.502},"min":{"1min":0.35699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.623,"5min":0.78800000000000003,"15min":0.79800000000000004},"last":0.46300000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.48699999999999999,"15min":0.496},"min":{"1min":0.34200000000000003,"5min":0.33000000000000002,"15min":0.32400000000000001},"max":{"1min":0.63900000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55700000000000005,"15min":0.58499999999999996},"min":{"1min":0.41099999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.82599999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47899999999999998}]},{"osd":3,"last update":"Tue Apr 2 09:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.55100000000000005,"15min":0.56799999999999995},"min":{"1min":0.42599999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77000000000000002,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53700000000000003,"15min":0.55500000000000005},"min":{"1min":0.35699999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75600000000000001,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.49399999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.748,"15min":0.77600000000000002},"min":{"1min":0.58299999999999996,"5min":0.44500000000000001,"15min":0.433},"max":{"1min":0.94599999999999995,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.74199999999999999,"15min":0.77000000000000002},"min":{"1min":0.56899999999999995,"5min":0.48999999999999999,"15min":0.41799999999999998},"max":{"1min":1.0009999999999999,"5min":1.05,"15min":1.25},"last":0.94099999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.746,"15min":0.77700000000000002},"min":{"1min":0.54200000000000004,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.321},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.77100000000000002,"15min":0.79800000000000004},"min":{"1min":0.55100000000000005,"5min":0.501,"15min":0.501},"max":{"1min":0.98199999999999998,"5min":1.115,"15min":1.173},"last":0.90900000000000003}]},{"osd":6,"last update":"Tue Apr 2 09:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.77200000000000002,"15min":0.80000000000000004},"min":{"1min":0.53600000000000003,"5min":0.51600000000000001,"15min":0.48099999999999998},"max":{"1min":1.0009999999999999,"5min":1.087,"15min":1.2210000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.78400000000000003,"15min":0.80500000000000005},"min":{"1min":0.59899999999999998,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.014,"5min":1.0880000000000001,"15min":1.2529999999999999},"last":0.89300000000000002}]},{"osd":7,"last update":"Tue Apr 2 09:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.78200000000000003,"15min":0.81100000000000005},"min":{"1min":0.52300000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0389999999999999,"5min":1.1080000000000001,"15min":1.236},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.80900000000000005,"15min":0.82399999999999995},"min":{"1min":0.56000000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.0369999999999999,"5min":1.129,"15min":1.2789999999999999},"last":0.79100000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576015,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52608,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319232,"statfs":{"total":96636764160,"available":96582893568,"internally_reserved":0,"allocated":3518464,"data_stored":1809440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.441,"15min":0.44},"min":{"1min":0.315,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.54700000000000004,"5min":1.681,"15min":1.681},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.497,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":1.091,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":0.378}]},{"osd":1,"last update":"Tue Apr 2 09:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.496,"15min":0.499},"min":{"1min":0.36399999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.60199999999999998,"5min":1.72,"15min":1.72},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53800000000000003,"15min":0.55000000000000004},"min":{"1min":0.40699999999999997,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.84499999999999997,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Tue Apr 2 09:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.54500000000000004},"min":{"1min":0.35999999999999999,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.72099999999999997,"5min":0.80700000000000005,"15min":1.2749999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58999999999999997,"15min":0.621},"min":{"1min":0.44,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.86799999999999999,"5min":1.754,"15min":1.754},"last":0.54100000000000004}]},{"osd":4,"last update":"Tue Apr 2 09:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76400000000000001,"15min":0.77100000000000002},"min":{"1min":0.58399999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.266,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.76900000000000002,"15min":0.78600000000000003},"min":{"1min":0.502,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0429999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.624}]},{"osd":5,"last update":"Tue Apr 2 09:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.75800000000000001,"15min":0.78100000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.96499999999999997,"5min":1.1379999999999999,"15min":1.99},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.77800000000000002,"15min":0.79500000000000004},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1120000000000001,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.69499999999999995}]},{"osd":6,"last update":"Tue Apr 2 09:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.78200000000000003,"15min":0.80200000000000005},"min":{"1min":0.56399999999999995,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.498,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.755,"5min":0.77100000000000002,"15min":0.79700000000000004},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.54100000000000004},"max":{"1min":1.0209999999999999,"5min":1.1619999999999999,"15min":1.3819999999999999},"last":0.71199999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.80200000000000005,"15min":0.82199999999999995},"min":{"1min":0.624,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.776,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79300000000000004,"15min":0.81899999999999995},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.497},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.61099999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576015,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48520,"kb_used_data":3444,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323320,"statfs":{"total":96636764160,"available":96587079680,"internally_reserved":0,"allocated":3526656,"data_stored":1809004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64000000000000001,"15min":0.67000000000000004},"min":{"1min":0.38600000000000001,"5min":0.38100000000000001,"15min":0.29299999999999998},"max":{"1min":0.86499999999999999,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.63600000000000001,"15min":0.66800000000000004},"min":{"1min":0.48399999999999999,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.81100000000000005,"5min":0.97099999999999997,"15min":1.075},"last":0.73199999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.74199999999999999},"min":{"1min":0.54400000000000004,"5min":0.47099999999999997,"15min":0.434},"max":{"1min":0.88500000000000001,"5min":1.0329999999999999,"15min":1.069},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.72399999999999998,"15min":0.752},"min":{"1min":0.54000000000000004,"5min":0.46100000000000002,"15min":0.40799999999999997},"max":{"1min":0.97999999999999998,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.74299999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.66700000000000004,"15min":0.69799999999999995},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.077,"15min":1.077},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68100000000000005,"15min":0.71399999999999997},"min":{"1min":0.45300000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93300000000000005,"5min":1.032,"15min":1.0740000000000001},"last":0.69199999999999995}]},{"osd":3,"last update":"Tue Apr 2 09:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.71899999999999997,"15min":0.749},"min":{"1min":0.499,"5min":0.48799999999999999,"15min":0.47399999999999998},"max":{"1min":0.89900000000000002,"5min":0.98099999999999998,"15min":1.018},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70499999999999996,"15min":0.74399999999999999},"min":{"1min":0.55800000000000005,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":0.86399999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.67400000000000004}]},{"osd":5,"last update":"Tue Apr 2 09:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.60799999999999998,"15min":0.61099999999999999},"min":{"1min":0.502,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.90800000000000003,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.622,"15min":0.624},"min":{"1min":0.46300000000000002,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.88900000000000001,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.495}]},{"osd":6,"last update":"Tue Apr 2 09:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.42799999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.93899999999999995,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68899999999999995,"15min":0.69699999999999995},"min":{"1min":0.46999999999999997,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":0.95199999999999996,"5min":1.0149999999999999,"15min":1.0269999999999999},"last":0.62}]},{"osd":7,"last update":"Tue Apr 2 09:15:25 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.69899999999999995,"15min":0.70999999999999996},"min":{"1min":0.56899999999999995,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.94199999999999995,"5min":1.002,"15min":1.0329999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.76500000000000001,"15min":0.78000000000000003},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.056,"5min":1.0960000000000001,"15min":1.0980000000000001},"last":0.65500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576018,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53176,"kb_used_data":4008,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318664,"statfs":{"total":96636764160,"available":96582311936,"internally_reserved":0,"allocated":4104192,"data_stored":2398207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:14:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.67300000000000004,"15min":0.69499999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.39600000000000002},"max":{"1min":0.89000000000000001,"5min":0.98699999999999999,"15min":1.004},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68799999999999994,"15min":0.71399999999999997},"min":{"1min":0.47299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.871,"5min":1.085,"15min":1.085},"last":0.68100000000000005}]},{"osd":1,"last update":"Tue Apr 2 09:14:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.67200000000000004,"15min":0.70699999999999996},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77300000000000002,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.69399999999999995,"15min":0.71899999999999997},"min":{"1min":0.47399999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.89900000000000002,"5min":0.95299999999999996,"15min":1.052},"last":0.65200000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:14:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.70199999999999996,"15min":0.73199999999999998},"min":{"1min":0.52300000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.83599999999999997,"5min":1.004,"15min":1.014},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68700000000000006,"15min":0.72999999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.82899999999999996,"5min":1.018,"15min":1.0680000000000001},"last":0.88500000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:14:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.73599999999999999,"15min":0.77100000000000002},"min":{"1min":0.55000000000000004,"5min":0.496,"15min":0.496},"max":{"1min":0.92300000000000004,"5min":0.98899999999999999,"15min":1.083},"last":0.753},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.76000000000000001,"15min":0.78300000000000003},"min":{"1min":0.50600000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88700000000000001,"5min":1.0109999999999999,"15min":1.1020000000000001},"last":0.98099999999999998}]},{"osd":4,"last update":"Tue Apr 2 09:14:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.441,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.88700000000000001,"5min":0.98399999999999999,"15min":1.054},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.68400000000000005,"15min":0.67600000000000005},"min":{"1min":0.46000000000000002,"5min":0.40400000000000003,"15min":0.39900000000000002},"max":{"1min":0.91000000000000003,"5min":1.145,"15min":1.145},"last":0.83299999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:14:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.68400000000000005,"15min":0.67700000000000005},"min":{"1min":0.441,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93799999999999994,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":1.006},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.48599999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.95599999999999996,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.81299999999999994}]},{"osd":7,"last update":"Tue Apr 2 09:14:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.73499999999999999,"15min":0.73599999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.97099999999999997,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76200000000000001,"15min":0.76600000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.98699999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":1.03}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:15:32.111 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T09:15:32.111 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-02T09:15:32.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.346+0000 7fc016600700 1 Processor -- start 2024-04-02T09:15:32.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.346+0000 7fc016600700 1 -- start start 2024-04-02T09:15:32.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc016600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc01015d9d0 0x7fc0100d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc016600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc0100d16a0 0x7fc0100d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc016600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0100d2370 0x7fc01015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc016600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0100d2820 con 0x7fc01015d9d0 2024-04-02T09:15:32.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc016600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc010161420 con 0x7fc0100d16a0 2024-04-02T09:15:32.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc016600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc010161700 con 0x7fc0100d2370 2024-04-02T09:15:32.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc00f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc01015d9d0 0x7fc0100d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc00f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc01015d9d0 0x7fc0100d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52554/0 (socket says 172.21.15.89:52554) 2024-04-02T09:15:32.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc00f400700 1 -- 172.21.15.89:0/4202269586 learned_addr learned my addr 172.21.15.89:0/4202269586 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:32.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc00fe00700 1 --2- 172.21.15.89:0/4202269586 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0100d2370 0x7fc01015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc007e00700 1 --2- 172.21.15.89:0/4202269586 >> v2:172.21.15.113:3300/0 conn(0x7fc0100d16a0 0x7fc0100d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc00f400700 1 -- 172.21.15.89:0/4202269586 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0100d2370 msgr2=0x7fc01015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:32.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc00f400700 1 --2- 172.21.15.89:0/4202269586 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0100d2370 0x7fc01015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc00f400700 1 -- 172.21.15.89:0/4202269586 >> v2:172.21.15.113:3300/0 conn(0x7fc0100d16a0 msgr2=0x7fc0100d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:32.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc00f400700 1 --2- 172.21.15.89:0/4202269586 >> v2:172.21.15.113:3300/0 conn(0x7fc0100d16a0 0x7fc0100d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.347+0000 7fc00f400700 1 -- 172.21.15.89:0/4202269586 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0101619e0 con 0x7fc01015d9d0 2024-04-02T09:15:32.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.348+0000 7fc00f400700 1 --2- 172.21.15.89:0/4202269586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc01015d9d0 0x7fc0100d3c30 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7fc0100d12f0 tx=0x7fc000009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6840dcd553ff4e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:32.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.348+0000 7fc00ea00700 1 -- 172.21.15.89:0/4202269586 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00000b870 con 0x7fc01015d9d0 2024-04-02T09:15:32.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.348+0000 7fc00ea00700 1 -- 172.21.15.89:0/4202269586 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00000b9d0 con 0x7fc01015d9d0 2024-04-02T09:15:32.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.348+0000 7fc00ea00700 1 -- 172.21.15.89:0/4202269586 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00000bd60 con 0x7fc01015d9d0 2024-04-02T09:15:32.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.349+0000 7fc016600700 1 -- 172.21.15.89:0/4202269586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc01015d9d0 msgr2=0x7fc0100d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:32.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.349+0000 7fc016600700 1 --2- 172.21.15.89:0/4202269586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc01015d9d0 0x7fc0100d3c30 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7fc0100d12f0 tx=0x7fc000009bd0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.349+0000 7fc016600700 1 -- 172.21.15.89:0/4202269586 shutdown_connections 2024-04-02T09:15:32.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.349+0000 7fc016600700 1 --2- 172.21.15.89:0/4202269586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc01015d9d0 0x7fc0100d3c30 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.349+0000 7fc016600700 1 --2- 172.21.15.89:0/4202269586 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0100d2370 0x7fc01015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.349+0000 7fc016600700 1 --2- 172.21.15.89:0/4202269586 >> v2:172.21.15.113:3300/0 conn(0x7fc0100d16a0 0x7fc0100d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.349+0000 7fc016600700 1 -- 172.21.15.89:0/4202269586 >> 172.21.15.89:0/4202269586 conn(0x7fc01008aff0 msgr2=0x7fc0100d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:32.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.349+0000 7fc016600700 1 -- 172.21.15.89:0/4202269586 shutdown_connections 2024-04-02T09:15:32.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.349+0000 7fc016600700 1 -- 172.21.15.89:0/4202269586 wait complete. 2024-04-02T09:15:32.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.350+0000 7fc016600700 1 Processor -- start 2024-04-02T09:15:32.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.351+0000 7fc016600700 1 -- start start 2024-04-02T09:15:32.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.351+0000 7fc016600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0100d16a0 0x7fc0100ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.351+0000 7fc016600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100d2370 0x7fc01014a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.351+0000 7fc016600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc01015d9d0 0x7fc01014ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.351+0000 7fc016600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0100cac80 con 0x7fc0100d2370 2024-04-02T09:15:32.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.351+0000 7fc016600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0100cd390 con 0x7fc01015d9d0 2024-04-02T09:15:32.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.351+0000 7fc016600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0100cd670 con 0x7fc0100d16a0 2024-04-02T09:15:32.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.352+0000 7fc007e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100d2370 0x7fc01014a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.352+0000 7fc007e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100d2370 0x7fc01014a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52560/0 (socket says 172.21.15.89:52560) 2024-04-02T09:15:32.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.352+0000 7fc007e00700 1 -- 172.21.15.89:0/3783959937 learned_addr learned my addr 172.21.15.89:0/3783959937 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:32.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.352+0000 7fc00f400700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0100d16a0 0x7fc0100ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.352+0000 7fc00fe00700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc01015d9d0 0x7fc01014ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.352+0000 7fc007e00700 1 -- 172.21.15.89:0/3783959937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0100d16a0 msgr2=0x7fc0100ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:32.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.352+0000 7fc007e00700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0100d16a0 0x7fc0100ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.352+0000 7fc007e00700 1 -- 172.21.15.89:0/3783959937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc01015d9d0 msgr2=0x7fc01014ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:32.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.352+0000 7fc007e00700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc01015d9d0 0x7fc01014ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.352+0000 7fc007e00700 1 -- 172.21.15.89:0/3783959937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc000009550 con 0x7fc0100d2370 2024-04-02T09:15:32.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.352+0000 7fc00fe00700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc01015d9d0 0x7fc01014ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:32.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.352+0000 7fc007e00700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100d2370 0x7fc01014a2b0 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7fbff800e810 tx=0x7fbff800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:32.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.353+0000 7fc007400700 1 -- 172.21.15.89:0/3783959937 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff800c070 con 0x7fc0100d2370 2024-04-02T09:15:32.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.353+0000 7fc007400700 1 -- 172.21.15.89:0/3783959937 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff8013610 con 0x7fc0100d2370 2024-04-02T09:15:32.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.353+0000 7fc007400700 1 -- 172.21.15.89:0/3783959937 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff80139a0 con 0x7fc0100d2370 2024-04-02T09:15:32.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.353+0000 7fc016600700 1 -- 172.21.15.89:0/3783959937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0100cd950 con 0x7fc0100d2370 2024-04-02T09:15:32.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.353+0000 7fc016600700 1 -- 172.21.15.89:0/3783959937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01014a7f0 con 0x7fc0100d2370 2024-04-02T09:15:32.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.354+0000 7fc016600700 1 -- 172.21.15.89:0/3783959937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0100d1ca0 con 0x7fc0100d2370 2024-04-02T09:15:32.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.355+0000 7fc007400700 1 -- 172.21.15.89:0/3783959937 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbff8013b00 con 0x7fc0100d2370 2024-04-02T09:15:32.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.356+0000 7fc007400700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbfec06adb0 0x7fbfec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.356+0000 7fc007400700 1 -- 172.21.15.89:0/3783959937 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fbff8089500 con 0x7fc0100d2370 2024-04-02T09:15:32.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.356+0000 7fc00f400700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbfec06adb0 0x7fbfec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.356+0000 7fc00f400700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbfec06adb0 0x7fbfec06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc000014040 tx=0x7fc000008030 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:32.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.360+0000 7fc007400700 1 -- 172.21.15.89:0/3783959937 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbff8054300 con 0x7fc0100d2370 2024-04-02T09:15:32.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.641+0000 7fc016600700 1 -- 172.21.15.89:0/3783959937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc0100d3310 con 0x7fc0100d2370 2024-04-02T09:15:32.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.646+0000 7fc007400700 1 -- 172.21.15.89:0/3783959937 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v58) v1 ==== 190+0+0 (secure 0 0 0) 0x7fbff8013db0 con 0x7fc0100d2370 2024-04-02T09:15:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.648+0000 7fc016600700 1 -- 172.21.15.89:0/3783959937 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbfec06adb0 msgr2=0x7fbfec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.648+0000 7fc016600700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbfec06adb0 0x7fbfec06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc000014040 tx=0x7fc000008030 comp rx=0 tx=0).stop 2024-04-02T09:15:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.648+0000 7fc016600700 1 -- 172.21.15.89:0/3783959937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100d2370 msgr2=0x7fc01014a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.648+0000 7fc016600700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100d2370 0x7fc01014a2b0 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7fbff800e810 tx=0x7fbff800eb20 comp rx=0 tx=0).stop 2024-04-02T09:15:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.648+0000 7fc016600700 1 -- 172.21.15.89:0/3783959937 shutdown_connections 2024-04-02T09:15:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.648+0000 7fc016600700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbfec06adb0 0x7fbfec06d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.648+0000 7fc016600700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc01015d9d0 0x7fc01014ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.648+0000 7fc016600700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc0100d16a0 0x7fc0100ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.648+0000 7fc016600700 1 --2- 172.21.15.89:0/3783959937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100d2370 0x7fc01014a2b0 unknown :-1 s=CLOSED pgs=2988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.648+0000 7fc016600700 1 -- 172.21.15.89:0/3783959937 >> 172.21.15.89:0/3783959937 conn(0x7fc01008aff0 msgr2=0x7fc0100ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.648+0000 7fc016600700 1 -- 172.21.15.89:0/3783959937 shutdown_connections 2024-04-02T09:15:32.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.648+0000 7fc016600700 1 -- 172.21.15.89:0/3783959937 wait complete. 2024-04-02T09:15:32.648 INFO:teuthology.orchestra.run.smithi089.stderr:updated caps for client.0 2024-04-02T09:15:32.665 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:15:32.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.860+0000 7f49fee00700 1 Processor -- start 2024-04-02T09:15:32.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.860+0000 7f49fee00700 1 -- start start 2024-04-02T09:15:32.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.861+0000 7f49fee00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49f80ac7d0 0x7f49f8166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.861+0000 7f49fee00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f815fd50 con 0x7f49f815b5d0 2024-04-02T09:15:32.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.861+0000 7f49fee00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f815f140 con 0x7f49f80cb3f0 2024-04-02T09:15:32.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.861+0000 7f49fee00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f815b9b0 con 0x7f49f80ac7d0 2024-04-02T09:15:32.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.861+0000 7f49f7e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f49f815b5d0 0x7f49f80ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48344/0 (socket says 172.21.15.89:48344) 2024-04-02T09:15:32.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.861+0000 7f49f7e00700 1 -- 172.21.15.89:0/536358683 learned_addr learned my addr 172.21.15.89:0/536358683 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:32.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49fce00700 1 --2- 172.21.15.89:0/536358683 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49f80ac7d0 0x7f49f8166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096899197 0 0) 0x7f49f815fd50 con 0x7f49f815b5d0 2024-04-02T09:15:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49dc0035b0 con 0x7f49f815b5d0 2024-04-02T09:15:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039615197 0 0) 0x7f49f815f140 con 0x7f49f80cb3f0 2024-04-02T09:15:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49f815fd50 con 0x7f49f80cb3f0 2024-04-02T09:15:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4225564429 0 0) 0x7f49dc0035b0 con 0x7f49f815b5d0 2024-04-02T09:15:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49f80ac7d0 msgr2=0x7f49f8166430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 --2- 172.21.15.89:0/536358683 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49f80ac7d0 0x7f49f8166430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 >> v1:172.21.15.113:6789/0 conn(0x7f49f80cb3f0 legacy=0x7f49f80ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f815f420 con 0x7f49f815b5d0 2024-04-02T09:15:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f49e8001670 con 0x7f49f815b5d0 2024-04-02T09:15:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 >> v1:172.21.15.89:6789/0 conn(0x7f49f815b5d0 legacy=0x7f49f80ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 --2- 172.21.15.89:0/536358683 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49f80ac7d0 0x7f49dc003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 --2- 172.21.15.89:0/536358683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49dc004170 0x7f49dc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 --2- 172.21.15.89:0/536358683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49dc004d90 0x7f49dc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f815b9b0 con 0x7f49dc004170 2024-04-02T09:15:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc00bbd0 con 0x7f49dc004d90 2024-04-02T09:15:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc00bd90 con 0x7f49f80ac7d0 2024-04-02T09:15:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.862+0000 7f49f7400700 1 --2- 172.21.15.89:0/536358683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49dc004170 0x7f49dc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.863+0000 7f49fce00700 1 --2- 172.21.15.89:0/536358683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49dc004d90 0x7f49dc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.863+0000 7f49f7e00700 1 --2- 172.21.15.89:0/536358683 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49f80ac7d0 0x7f49dc003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.863+0000 7f49f7400700 1 -- 172.21.15.89:0/536358683 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49f80ac7d0 msgr2=0x7f49dc003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.863+0000 7f49f7400700 1 --2- 172.21.15.89:0/536358683 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49f80ac7d0 0x7f49dc003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.863+0000 7f49f7400700 1 -- 172.21.15.89:0/536358683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49dc004d90 msgr2=0x7f49dc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.863+0000 7f49f7400700 1 --2- 172.21.15.89:0/536358683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49dc004d90 0x7f49dc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.863+0000 7f49f7400700 1 -- 172.21.15.89:0/536358683 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49dc0035b0 con 0x7f49dc004170 2024-04-02T09:15:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.863+0000 7f49f7400700 1 --2- 172.21.15.89:0/536358683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49dc004170 0x7f49dc004850 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f49e4001940 tx=0x7f49dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fff03477e6654b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.863+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e4001430 con 0x7f49dc004170 2024-04-02T09:15:32.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.863+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e4001d70 con 0x7f49dc004170 2024-04-02T09:15:32.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.863+0000 7f49f6a00700 1 -- 172.21.15.89:0/536358683 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e4005510 con 0x7f49dc004170 2024-04-02T09:15:32.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.864+0000 7f49fee00700 1 -- 172.21.15.89:0/536358683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49dc004170 msgr2=0x7f49dc004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:32.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.864+0000 7f49fee00700 1 --2- 172.21.15.89:0/536358683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49dc004170 0x7f49dc004850 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f49e4001940 tx=0x7f49dc005880 comp rx=0 tx=0).stop 2024-04-02T09:15:32.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.864+0000 7f49fce00700 1 -- 172.21.15.89:0/536358683 reap_dead start 2024-04-02T09:15:32.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.864+0000 7f49fee00700 1 -- 172.21.15.89:0/536358683 shutdown_connections 2024-04-02T09:15:32.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.864+0000 7f49fee00700 1 -- 172.21.15.89:0/536358683 >> 172.21.15.89:0/536358683 conn(0x7f49f80cf190 msgr2=0x7f49f80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:32.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.864+0000 7f49fee00700 1 -- 172.21.15.89:0/536358683 shutdown_connections 2024-04-02T09:15:32.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.864+0000 7f49fee00700 1 -- 172.21.15.89:0/536358683 wait complete. 2024-04-02T09:15:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.866+0000 7f49fee00700 1 Processor -- start 2024-04-02T09:15:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.866+0000 7f49fee00700 1 -- start start 2024-04-02T09:15:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49fee00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49dc004170 0x7f49f80b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49fee00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49f81f2e00 0x7f49f80cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49fee00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49f80ccb60 0x7f49f80ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49fee00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f80cd830 con 0x7f49dc004170 2024-04-02T09:15:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49fee00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f8071c20 con 0x7f49f81f2e00 2024-04-02T09:15:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49fee00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f8071f00 con 0x7f49f80ccb60 2024-04-02T09:15:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49f7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49dc004170 0x7f49f80b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49f7e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49dc004170 0x7f49f80b4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52582/0 (socket says 172.21.15.89:52582) 2024-04-02T09:15:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49f7e00700 1 -- 172.21.15.89:0/2081143689 learned_addr learned my addr 172.21.15.89:0/2081143689 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49fce00700 1 --2- 172.21.15.89:0/2081143689 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49f80ccb60 0x7f49f80ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49f7400700 1 --2- 172.21.15.89:0/2081143689 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49f81f2e00 0x7f49f80cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49f7e00700 1 -- 172.21.15.89:0/2081143689 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49f80ccb60 msgr2=0x7f49f80ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49f7e00700 1 --2- 172.21.15.89:0/2081143689 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49f80ccb60 0x7f49f80ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49f7e00700 1 -- 172.21.15.89:0/2081143689 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49f81f2e00 msgr2=0x7f49f80cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49f7e00700 1 --2- 172.21.15.89:0/2081143689 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49f81f2e00 0x7f49f80cc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.867+0000 7f49f7e00700 1 -- 172.21.15.89:0/2081143689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f80721e0 con 0x7f49dc004170 2024-04-02T09:15:32.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.868+0000 7f49f7e00700 1 --2- 172.21.15.89:0/2081143689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49dc004170 0x7f49f80b4140 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7f49e8004f80 tx=0x7f49e80073f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.868+0000 7f49efe00700 1 -- 172.21.15.89:0/2081143689 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8011580 con 0x7f49dc004170 2024-04-02T09:15:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.868+0000 7f49efe00700 1 -- 172.21.15.89:0/2081143689 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e8011700 con 0x7f49dc004170 2024-04-02T09:15:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.868+0000 7f49fee00700 1 -- 172.21.15.89:0/2081143689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f81561a0 con 0x7f49dc004170 2024-04-02T09:15:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.868+0000 7f49efe00700 1 -- 172.21.15.89:0/2081143689 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8011ab0 con 0x7f49dc004170 2024-04-02T09:15:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.868+0000 7f49fee00700 1 -- 172.21.15.89:0/2081143689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f81f3280 con 0x7f49dc004170 2024-04-02T09:15:32.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.869+0000 7f49fee00700 1 -- 172.21.15.89:0/2081143689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f815f9a0 con 0x7f49dc004170 2024-04-02T09:15:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.870+0000 7f49efe00700 1 -- 172.21.15.89:0/2081143689 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f49e8011c10 con 0x7f49dc004170 2024-04-02T09:15:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.871+0000 7f49efe00700 1 --2- 172.21.15.89:0/2081143689 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f49d806adb0 0x7f49d806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.871+0000 7f49efe00700 1 -- 172.21.15.89:0/2081143689 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f49e80896b0 con 0x7f49dc004170 2024-04-02T09:15:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.871+0000 7f49f7400700 1 --2- 172.21.15.89:0/2081143689 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f49d806adb0 0x7f49d806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.872+0000 7f49f7400700 1 --2- 172.21.15.89:0/2081143689 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f49d806adb0 0x7f49d806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f49f81538a0 tx=0x7f49e400a000 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:32.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:32.876+0000 7f49efe00700 1 -- 172.21.15.89:0/2081143689 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f49e8054520 con 0x7f49dc004170 2024-04-02T09:15:33.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.119+0000 7f49fee00700 1 -- 172.21.15.89:0/2081143689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49f8072040 con 0x7f49dc004170 2024-04-02T09:15:33.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.119+0000 7f49efe00700 1 -- 172.21.15.89:0/2081143689 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4753 (secure 0 0 0) 0x7f49f8072040 con 0x7f49dc004170 2024-04-02T09:15:33.118 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:15:33.118 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10376,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2447701494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2447701494},{"type":"v1","addr":"172.21.15.113:6834","nonce":2447701494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":12103,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/361614750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":361614750},{"type":"v1","addr":"172.21.15.113:6835","nonce":361614750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":13281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2871180587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2871180587},{"type":"v1","addr":"172.21.15.89:6835","nonce":2871180587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:15:26.772241+0000","modified":"2024-04-02T09:15:30.258749+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13308":{"gid":13308,"name":"a","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3853118738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3853118738},{"type":"v1","addr":"172.21.15.89:6837","nonce":3853118738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-04-02T09:15:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.122+0000 7f49fee00700 1 -- 172.21.15.89:0/2081143689 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f49d806adb0 msgr2=0x7f49d806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.122+0000 7f49fee00700 1 --2- 172.21.15.89:0/2081143689 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f49d806adb0 0x7f49d806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f49f81538a0 tx=0x7f49e400a000 comp rx=0 tx=0).stop 2024-04-02T09:15:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.122+0000 7f49fee00700 1 -- 172.21.15.89:0/2081143689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49dc004170 msgr2=0x7f49f80b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.122+0000 7f49fee00700 1 --2- 172.21.15.89:0/2081143689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49dc004170 0x7f49f80b4140 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7f49e8004f80 tx=0x7f49e80073f0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.122+0000 7f49fee00700 1 -- 172.21.15.89:0/2081143689 shutdown_connections 2024-04-02T09:15:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.122+0000 7f49fee00700 1 --2- 172.21.15.89:0/2081143689 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49f80ccb60 0x7f49f80ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.122+0000 7f49fee00700 1 --2- 172.21.15.89:0/2081143689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49dc004170 0x7f49f80b4140 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.122+0000 7f49fee00700 1 --2- 172.21.15.89:0/2081143689 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f49d806adb0 0x7f49d806d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.122+0000 7f49fee00700 1 --2- 172.21.15.89:0/2081143689 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49f81f2e00 0x7f49f80cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.122+0000 7f49fee00700 1 -- 172.21.15.89:0/2081143689 >> 172.21.15.89:0/2081143689 conn(0x7f49f80cf190 msgr2=0x7f49f80aed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.122+0000 7f49fee00700 1 -- 172.21.15.89:0/2081143689 shutdown_connections 2024-04-02T09:15:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.122+0000 7f49fee00700 1 -- 172.21.15.89:0/2081143689 wait complete. 2024-04-02T09:15:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 144 2024-04-02T09:15:33.542 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 144, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-02T09:15:26.772241+0000', 'modified': '2024-04-02T09:15:30.258749+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 13308}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_13308': {'gid': 13308, 'name': 'a', 'rank': 0, 'incarnation': 140, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6837/3853118738', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6836', 'nonce': 3853118738}, {'type': 'v1', 'addr': '172.21.15.89:6837', 'nonce': 3853118738}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-02T09:15:33.542 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-02T09:15:33.542 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-02T09:15:33.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.736+0000 7f3ee7600700 1 Processor -- start 2024-04-02T09:15:33.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.737+0000 7f3ee7600700 1 -- start start 2024-04-02T09:15:33.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.738+0000 7f3ee7600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3ee015a5c0 0x7f3ee00709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.738+0000 7f3ee7600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3ee015dfe0 0x7f3ee00ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.738+0000 7f3ee7600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee00cb310 con 0x7f3ee00ce670 2024-04-02T09:15:33.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.738+0000 7f3ee7600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee00ceae0 con 0x7f3ee015dfe0 2024-04-02T09:15:33.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.738+0000 7f3ee7600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee0070f00 con 0x7f3ee015a5c0 2024-04-02T09:15:33.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.738+0000 7f3edfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3ee015dfe0 0x7f3ee00ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.738+0000 7f3edfe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3ee015dfe0 0x7f3ee00ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52588/0 (socket says 172.21.15.89:52588) 2024-04-02T09:15:33.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.738+0000 7f3edfe00700 1 -- 172.21.15.89:0/442454162 learned_addr learned my addr 172.21.15.89:0/442454162 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:33.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.738+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/442454162 >> v2:172.21.15.113:3300/0 conn(0x7f3ee015a5c0 0x7f3ee00709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.738+0000 7f3edfe00700 1 -- 172.21.15.89:0/442454162 >> v2:172.21.15.113:3300/0 conn(0x7f3ee015a5c0 msgr2=0x7f3ee00709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:33.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.738+0000 7f3edfe00700 1 --2- 172.21.15.89:0/442454162 >> v2:172.21.15.113:3300/0 conn(0x7f3ee015a5c0 0x7f3ee00709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.738+0000 7f3edfe00700 1 -- 172.21.15.89:0/442454162 >> v1:172.21.15.113:6789/0 conn(0x7f3ee00ce670 legacy=0x7f3ee00b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:33.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.738+0000 7f3edfe00700 1 -- 172.21.15.89:0/442454162 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee00cca90 con 0x7f3ee015dfe0 2024-04-02T09:15:33.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.739+0000 7f3edfe00700 1 --2- 172.21.15.89:0/442454162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee015dfe0 0x7f3ee00ce130 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0072e70 tx=0x7f3ecc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25081085d4fbc1e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:33.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.739+0000 7f3edf400700 1 -- 172.21.15.89:0/442454162 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc004230 con 0x7f3ee015dfe0 2024-04-02T09:15:33.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.739+0000 7f3edf400700 1 -- 172.21.15.89:0/442454162 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ecc004390 con 0x7f3ee015dfe0 2024-04-02T09:15:33.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.739+0000 7f3edf400700 1 -- 172.21.15.89:0/442454162 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc0036a0 con 0x7f3ee015dfe0 2024-04-02T09:15:33.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.739+0000 7f3ee7600700 1 -- 172.21.15.89:0/442454162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee015dfe0 msgr2=0x7f3ee00ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:33.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.739+0000 7f3ee7600700 1 --2- 172.21.15.89:0/442454162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee015dfe0 0x7f3ee00ce130 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0072e70 tx=0x7f3ecc009bd0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.740+0000 7f3ee7600700 1 -- 172.21.15.89:0/442454162 shutdown_connections 2024-04-02T09:15:33.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.740+0000 7f3ee7600700 1 --2- 172.21.15.89:0/442454162 >> v2:172.21.15.113:3300/0 conn(0x7f3ee015a5c0 0x7f3ee00709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.740+0000 7f3ee7600700 1 --2- 172.21.15.89:0/442454162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee015dfe0 0x7f3ee00ce130 unknown :-1 s=CLOSED pgs=2992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.740+0000 7f3ee7600700 1 -- 172.21.15.89:0/442454162 >> 172.21.15.89:0/442454162 conn(0x7f3ee00b0d70 msgr2=0x7f3ee00b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:33.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.740+0000 7f3ee7600700 1 -- 172.21.15.89:0/442454162 shutdown_connections 2024-04-02T09:15:33.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.740+0000 7f3ee7600700 1 -- 172.21.15.89:0/442454162 wait complete. 2024-04-02T09:15:33.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.741+0000 7f3ee7600700 1 Processor -- start 2024-04-02T09:15:33.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.741+0000 7f3ee7600700 1 -- start start 2024-04-02T09:15:33.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.742+0000 7f3ee7600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee00ce670 0x7f3ee01ed400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.742+0000 7f3ee7600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ee015a5c0 0x7f3ee00adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.742+0000 7f3ee7600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ee015dfe0 0x7f3ee0158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.742+0000 7f3ee7600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee01ed940 con 0x7f3ee00ce670 2024-04-02T09:15:33.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.742+0000 7f3ee7600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee01f3400 con 0x7f3ee015a5c0 2024-04-02T09:15:33.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.742+0000 7f3ee7600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee01f3680 con 0x7f3ee015dfe0 2024-04-02T09:15:33.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.743+0000 7f3ee4c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee00ce670 0x7f3ee01ed400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.743+0000 7f3ee4c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee00ce670 0x7f3ee01ed400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52602/0 (socket says 172.21.15.89:52602) 2024-04-02T09:15:33.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.743+0000 7f3ee4c00700 1 -- 172.21.15.89:0/1839355858 learned_addr learned my addr 172.21.15.89:0/1839355858 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:33.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.743+0000 7f3edfe00700 1 --2- 172.21.15.89:0/1839355858 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ee015a5c0 0x7f3ee00adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.743+0000 7f3ee5600700 1 --2- 172.21.15.89:0/1839355858 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ee015dfe0 0x7f3ee0158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.743+0000 7f3ee4c00700 1 -- 172.21.15.89:0/1839355858 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ee015dfe0 msgr2=0x7f3ee0158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:33.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.743+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/1839355858 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ee015dfe0 0x7f3ee0158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.743+0000 7f3ee4c00700 1 -- 172.21.15.89:0/1839355858 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ee015a5c0 msgr2=0x7f3ee00adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:33.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.743+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/1839355858 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ee015a5c0 0x7f3ee00adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.743+0000 7f3ee4c00700 1 -- 172.21.15.89:0/1839355858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ecc009550 con 0x7f3ee00ce670 2024-04-02T09:15:33.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.743+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/1839355858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee00ce670 0x7f3ee01ed400 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400e810 tx=0x7f3ed400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:33.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.744+0000 7f3edcc00700 1 -- 172.21.15.89:0/1839355858 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4013610 con 0x7f3ee00ce670 2024-04-02T09:15:33.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.744+0000 7f3edcc00700 1 -- 172.21.15.89:0/1839355858 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed4013770 con 0x7f3ee00ce670 2024-04-02T09:15:33.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.744+0000 7f3edcc00700 1 -- 172.21.15.89:0/1839355858 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4013b20 con 0x7f3ee00ce670 2024-04-02T09:15:33.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.744+0000 7f3ee7600700 1 -- 172.21.15.89:0/1839355858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee01f3960 con 0x7f3ee00ce670 2024-04-02T09:15:33.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.744+0000 7f3ee7600700 1 -- 172.21.15.89:0/1839355858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee01f3f20 con 0x7f3ee00ce670 2024-04-02T09:15:33.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.746+0000 7f3edcc00700 1 -- 172.21.15.89:0/1839355858 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3ed4013c80 con 0x7f3ee00ce670 2024-04-02T09:15:33.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.746+0000 7f3edcc00700 1 --2- 172.21.15.89:0/1839355858 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ec006adb0 0x7f3ec006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.747+0000 7f3edcc00700 1 -- 172.21.15.89:0/1839355858 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f3ed40893e0 con 0x7f3ee00ce670 2024-04-02T09:15:33.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.747+0000 7f3edfe00700 1 --2- 172.21.15.89:0/1839355858 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ec006adb0 0x7f3ec006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.747+0000 7f3edfe00700 1 --2- 172.21.15.89:0/1839355858 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ec006adb0 0x7f3ec006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009130 tx=0x7f3ecc008030 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:33.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.749+0000 7f3efb120b80 1 Processor -- start 2024-04-02T09:15:33.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.749+0000 7f3efb120b80 1 -- start start 2024-04-02T09:15:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3efb120b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f684830 0x55db4f684c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3efb120b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f657be0 0x55db4f65aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3efb120b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f655b60 0x55db4f681a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3efb120b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db4f5cf650 con 0x55db4f684830 2024-04-02T09:15:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3efb120b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db4f5b0c30 con 0x55db4f655b60 2024-04-02T09:15:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3efb120b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db4f5ae430 con 0x55db4f657be0 2024-04-02T09:15:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3ee4c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f684830 0x55db4f684c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3ee4c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f684830 0x55db4f684c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52614/0 (socket says 172.21.15.89:52614) 2024-04-02T09:15:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3ee4c00700 1 -- 172.21.15.89:0/832878197 learned_addr learned my addr 172.21.15.89:0/832878197 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3ee4c00700 1 -- 172.21.15.89:0/832878197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f657be0 msgr2=0x55db4f65aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3edfe00700 1 --2- 172.21.15.89:0/832878197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f657be0 0x55db4f65aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3ee5600700 1 --2- 172.21.15.89:0/832878197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f655b60 0x55db4f681a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/832878197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f657be0 0x55db4f65aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3ee4c00700 1 -- 172.21.15.89:0/832878197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f655b60 msgr2=0x55db4f681a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/832878197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f655b60 0x55db4f681a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3ee4c00700 1 -- 172.21.15.89:0/832878197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db4f39d790 con 0x55db4f684830 2024-04-02T09:15:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3edfe00700 1 --2- 172.21.15.89:0/832878197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f657be0 0x55db4f65aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:15:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3ee5600700 1 --2- 172.21.15.89:0/832878197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f655b60 0x55db4f681a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:15:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.750+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/832878197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f684830 0x55db4f684c10 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7f3ed405fcd0 tx=0x7f3ed40585e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c837f85a044735e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.751+0000 7f3ed1200700 1 -- 172.21.15.89:0/832878197 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4060200 con 0x55db4f684830 2024-04-02T09:15:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.751+0000 7f3ed1200700 1 -- 172.21.15.89:0/832878197 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed4089070 con 0x55db4f684830 2024-04-02T09:15:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.751+0000 7f3ed1200700 1 -- 172.21.15.89:0/832878197 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4058270 con 0x55db4f684830 2024-04-02T09:15:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.751+0000 7f3efb120b80 1 -- 172.21.15.89:0/832878197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f684830 msgr2=0x55db4f684c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.751+0000 7f3efb120b80 1 --2- 172.21.15.89:0/832878197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f684830 0x55db4f684c10 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7f3ed405fcd0 tx=0x7f3ed40585e0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.751+0000 7f3efb120b80 1 -- 172.21.15.89:0/832878197 shutdown_connections 2024-04-02T09:15:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.751+0000 7f3efb120b80 1 --2- 172.21.15.89:0/832878197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f655b60 0x55db4f681a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.751+0000 7f3efb120b80 1 --2- 172.21.15.89:0/832878197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f684830 0x55db4f684c10 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.751+0000 7f3efb120b80 1 --2- 172.21.15.89:0/832878197 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f657be0 0x55db4f65aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.751+0000 7f3efb120b80 1 -- 172.21.15.89:0/832878197 >> 172.21.15.89:0/832878197 conn(0x55db4f5b86a0 msgr2=0x55db4f69f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.752+0000 7f3efb120b80 1 -- 172.21.15.89:0/832878197 shutdown_connections 2024-04-02T09:15:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.752+0000 7f3efb120b80 1 -- 172.21.15.89:0/832878197 wait complete. 2024-04-02T09:15:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.752+0000 7f3efb120b80 1 -- start start 2024-04-02T09:15:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.752+0000 7f3efb120b80 1 Processor -- start 2024-04-02T09:15:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.753+0000 7f3efb120b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 0x55db4f680bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.753+0000 7f3efb120b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f657be0 0x55db4f6857e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.753+0000 7f3efb120b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f684830 0x55db4f6a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.754+0000 7f3efb120b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db4f5a9d20 con 0x55db4f657be0 2024-04-02T09:15:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.754+0000 7f3efb120b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db4f545770 con 0x55db4f684830 2024-04-02T09:15:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.754+0000 7f3efb120b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db4f5686f0 con 0x55db4f655b60 2024-04-02T09:15:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.754+0000 7f3efb120b80 10 client.0 _ll_register_callbacks cb 0x55db4f54f5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T09:15:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.754+0000 7f3edfe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f657be0 0x55db4f6857e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.754+0000 7f3edfe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f657be0 0x55db4f6857e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52620/0 (socket says 172.21.15.89:52620) 2024-04-02T09:15:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.754+0000 7f3edfe00700 1 -- 172.21.15.89:0/223695000 learned_addr learned my addr 172.21.15.89:0/223695000 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.754+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/223695000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 0x55db4f680bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.754+0000 7f3ee5600700 1 --2- 172.21.15.89:0/223695000 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f684830 0x55db4f6a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.754+0000 7f3edfe00700 1 -- 172.21.15.89:0/223695000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 msgr2=0x55db4f680bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.754+0000 7f3edfe00700 1 --2- 172.21.15.89:0/223695000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 0x55db4f680bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.754+0000 7f3edfe00700 1 -- 172.21.15.89:0/223695000 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f684830 msgr2=0x55db4f6a2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.754+0000 7f3edfe00700 1 --2- 172.21.15.89:0/223695000 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f684830 0x55db4f6a2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.754+0000 7f3edfe00700 1 -- 172.21.15.89:0/223695000 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee01f3f20 con 0x55db4f657be0 2024-04-02T09:15:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.755+0000 7f3edfe00700 1 --2- 172.21.15.89:0/223695000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f657be0 0x55db4f6857e0 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009fd0 tx=0x7f3ecc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=924a729ce22dd9fc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.755+0000 7f3ec5600700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T09:15:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.755+0000 7f3ec5600700 1 -- 172.21.15.89:0/223695000 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc012c10 con 0x55db4f657be0 2024-04-02T09:15:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.755+0000 7f3ec5600700 1 -- 172.21.15.89:0/223695000 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f3ecc012d70 con 0x55db4f657be0 2024-04-02T09:15:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.755+0000 7f3efb120b80 1 -- 172.21.15.89:0/223695000 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55db4f64c920 con 0x55db4f657be0 2024-04-02T09:15:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.755+0000 7f3ec5600700 1 -- 172.21.15.89:0/223695000 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ecc026450 con 0x55db4f657be0 2024-04-02T09:15:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.755+0000 7f3efb120b80 10 client.13434 fetch_fsmap learned FSMap version 144 2024-04-02T09:15:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.755+0000 7f3efb120b80 1 -- 172.21.15.89:0/223695000 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=2+}) v3 -- 0x55db4f48a0a0 con 0x55db4f657be0 2024-04-02T09:15:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.755+0000 7f3ec5600700 1 -- 172.21.15.89:0/223695000 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55db4f64c920 con 0x55db4f657be0 2024-04-02T09:15:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.755+0000 7f3ec5600700 1 -- 172.21.15.89:0/223695000 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 144) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3ecc0265b0 con 0x55db4f657be0 2024-04-02T09:15:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.755+0000 7f3efb120b80 10 client.13434 fetch_fsmap finished waiting for FSMap version 144 2024-04-02T09:15:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.755+0000 7f3efb120b80 10 client.13434 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T09:15:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.755+0000 7f3efb120b80 20 client.13434 populate_metadata read hostname 'smithi089' 2024-04-02T09:15:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.755+0000 7f3efb120b80 1 --2- 172.21.15.89:0/223695000 >> [v2:172.21.15.89:6836/3853118738,v1:172.21.15.89:6837/3853118738] conn(0x55db4f687e90 0x55db4f688270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.756+0000 7f3efb120b80 4 client.13434 mds_command: new command op to 13308 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T09:15:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.756+0000 7f3efb120b80 1 -- 172.21.15.89:0/223695000 --> [v2:172.21.15.89:6836/3853118738,v1:172.21.15.89:6837/3853118738] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db4f6296f0 con 0x55db4f687e90 2024-04-02T09:15:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.756+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/223695000 >> [v2:172.21.15.89:6836/3853118738,v1:172.21.15.89:6837/3853118738] conn(0x55db4f687e90 0x55db4f688270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.756+0000 7f3ec5600700 10 client.13434 ms_handle_connect on v2:172.21.15.89:6836/3853118738 2024-04-02T09:15:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.756+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/223695000 >> [v2:172.21.15.89:6836/3853118738,v1:172.21.15.89:6837/3853118738] conn(0x55db4f687e90 0x55db4f688270 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5583a5309596efd5 server_cookie=368297ef157ff34c in_seq=0 out_seq=0 2024-04-02T09:15:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.757+0000 7f3ec5600700 1 -- 172.21.15.89:0/223695000 <== mds.0 v2:172.21.15.89:6836/3853118738 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55db4f6296f0 con 0x55db4f687e90 2024-04-02T09:15:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.757+0000 7f3ec5600700 10 client.13434 handle_command_reply: tid=0 2024-04-02T09:15:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.757+0000 7f3efb120b80 1 client.13434 shutdown 2024-04-02T09:15:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.757+0000 7f3efb120b80 1 -- 172.21.15.89:0/223695000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f657be0 msgr2=0x55db4f6857e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.757+0000 7f3efb120b80 1 --2- 172.21.15.89:0/223695000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f657be0 0x55db4f6857e0 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009fd0 tx=0x7f3ecc0047c0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.757+0000 7f3efb120b80 1 -- 172.21.15.89:0/223695000 shutdown_connections 2024-04-02T09:15:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.757+0000 7f3efb120b80 1 --2- 172.21.15.89:0/223695000 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f684830 0x55db4f6a2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.757+0000 7f3efb120b80 1 --2- 172.21.15.89:0/223695000 >> [v2:172.21.15.89:6836/3853118738,v1:172.21.15.89:6837/3853118738] conn(0x55db4f687e90 0x55db4f688270 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.757+0000 7f3efb120b80 1 --2- 172.21.15.89:0/223695000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 0x55db4f680bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.757+0000 7f3efb120b80 1 --2- 172.21.15.89:0/223695000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f657be0 0x55db4f6857e0 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.757+0000 7f3efb120b80 1 -- 172.21.15.89:0/223695000 >> 172.21.15.89:0/223695000 conn(0x55db4f5b86a0 msgr2=0x55db4f659e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.757+0000 7f3ec5600700 0 client.13434 ms_handle_reset on v2:172.21.15.89:6836/3853118738 2024-04-02T09:15:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.757+0000 7f3efb120b80 1 -- 172.21.15.89:0/223695000 shutdown_connections 2024-04-02T09:15:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.757+0000 7f3efb120b80 1 -- 172.21.15.89:0/223695000 wait complete. 2024-04-02T09:15:33.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.758+0000 7f3efb120b80 20 client.13434 trim_cache size 0 max 16384 2024-04-02T09:15:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.777+0000 7f3efb120b80 1 Processor -- start 2024-04-02T09:15:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.777+0000 7f3efb120b80 1 -- start start 2024-04-02T09:15:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3efb120b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 0x55db4f69f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3efb120b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f657be0 0x55db4f6535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3efb120b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f684830 0x55db4f653ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3efb120b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db4f6802d0 con 0x55db4f657be0 2024-04-02T09:15:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3efb120b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db4f39e390 con 0x55db4f684830 2024-04-02T09:15:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3efb120b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db4f6556d0 con 0x55db4f655b60 2024-04-02T09:15:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3edfe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f657be0 0x55db4f6535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3edfe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f657be0 0x55db4f6535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52626/0 (socket says 172.21.15.89:52626) 2024-04-02T09:15:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3edfe00700 1 -- 172.21.15.89:0/3850035828 learned_addr learned my addr 172.21.15.89:0/3850035828 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/3850035828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 0x55db4f69f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3ee5600700 1 --2- 172.21.15.89:0/3850035828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f684830 0x55db4f653ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3edfe00700 1 -- 172.21.15.89:0/3850035828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 msgr2=0x55db4f69f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3edfe00700 1 --2- 172.21.15.89:0/3850035828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 0x55db4f69f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3edfe00700 1 -- 172.21.15.89:0/3850035828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f684830 msgr2=0x55db4f653ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3edfe00700 1 --2- 172.21.15.89:0/3850035828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f684830 0x55db4f653ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.778+0000 7f3edfe00700 1 -- 172.21.15.89:0/3850035828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db4f570010 con 0x55db4f657be0 2024-04-02T09:15:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.779+0000 7f3edfe00700 1 --2- 172.21.15.89:0/3850035828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f657be0 0x55db4f6535a0 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc0040b0 tx=0x7f3ecc029040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9158fb4c8ec7f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.779+0000 7f3ed1c00700 1 -- 172.21.15.89:0/3850035828 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc026b10 con 0x55db4f657be0 2024-04-02T09:15:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.779+0000 7f3ed1c00700 1 -- 172.21.15.89:0/3850035828 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ecc026450 con 0x55db4f657be0 2024-04-02T09:15:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.779+0000 7f3ed1c00700 1 -- 172.21.15.89:0/3850035828 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc012a10 con 0x55db4f657be0 2024-04-02T09:15:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.780+0000 7f3efb120b80 1 -- 172.21.15.89:0/3850035828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f657be0 msgr2=0x55db4f6535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.780+0000 7f3efb120b80 1 --2- 172.21.15.89:0/3850035828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f657be0 0x55db4f6535a0 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc0040b0 tx=0x7f3ecc029040 comp rx=0 tx=0).stop 2024-04-02T09:15:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.780+0000 7f3efb120b80 1 -- 172.21.15.89:0/3850035828 shutdown_connections 2024-04-02T09:15:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.780+0000 7f3efb120b80 1 --2- 172.21.15.89:0/3850035828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f684830 0x55db4f653ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.780+0000 7f3efb120b80 1 --2- 172.21.15.89:0/3850035828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 0x55db4f69f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.780+0000 7f3efb120b80 1 --2- 172.21.15.89:0/3850035828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f657be0 0x55db4f6535a0 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.780+0000 7f3efb120b80 1 -- 172.21.15.89:0/3850035828 >> 172.21.15.89:0/3850035828 conn(0x55db4f5b86a0 msgr2=0x55db4f650880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.780+0000 7f3efb120b80 1 -- 172.21.15.89:0/3850035828 shutdown_connections 2024-04-02T09:15:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.780+0000 7f3efb120b80 1 -- 172.21.15.89:0/3850035828 wait complete. 2024-04-02T09:15:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.780+0000 7f3efb120b80 1 -- start start 2024-04-02T09:15:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.781+0000 7f3efb120b80 1 Processor -- start 2024-04-02T09:15:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3efb120b80 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 0x55db4f6a2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3efb120b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f657be0 0x55db4f706120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3efb120b80 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f684830 0x55db4f684c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3efb120b80 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db4f688bd0 con 0x55db4f684830 2024-04-02T09:15:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3efb120b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db4f706970 con 0x55db4f657be0 2024-04-02T09:15:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3efb120b80 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db4f6805c0 con 0x55db4f655b60 2024-04-02T09:15:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3efb120b80 10 client.0 _ll_register_callbacks cb 0x7f3ecc004260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T09:15:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3ee5600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f684830 0x55db4f684c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3ee5600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f684830 0x55db4f684c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52632/0 (socket says 172.21.15.89:52632) 2024-04-02T09:15:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3ee5600700 1 -- 172.21.15.89:0/4223462504 learned_addr learned my addr 172.21.15.89:0/4223462504 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3edfe00700 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f657be0 0x55db4f706120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3ee5600700 1 -- 172.21.15.89:0/4223462504 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 msgr2=0x55db4f6a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 0x55db4f6a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3ee5600700 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 0x55db4f6a2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3ee5600700 1 -- 172.21.15.89:0/4223462504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f657be0 msgr2=0x55db4f706120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:33.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3ee5600700 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f657be0 0x55db4f706120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3ee5600700 1 -- 172.21.15.89:0/4223462504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db4f48a0a0 con 0x55db4f684830 2024-04-02T09:15:33.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3edfe00700 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f657be0 0x55db4f706120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:15:33.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.782+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 0x55db4f6a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:15:33.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3ee5600700 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f684830 0x55db4f684c10 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x55db4f683d00 tx=0x7f3ed8013c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64313b94a7b9e571 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:33.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3ec5600700 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-04-02T09:15:33.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3ec5600700 1 -- 172.21.15.89:0/4223462504 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8011af0 con 0x55db4f684830 2024-04-02T09:15:33.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3ec5600700 1 -- 172.21.15.89:0/4223462504 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f3ed8011c50 con 0x55db4f684830 2024-04-02T09:15:33.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3efb120b80 1 -- 172.21.15.89:0/4223462504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ee00ac2f0 con 0x55db4f684830 2024-04-02T09:15:33.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3ec5600700 1 -- 172.21.15.89:0/4223462504 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ed8005420 con 0x55db4f684830 2024-04-02T09:15:33.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3efb120b80 10 client.13440 fetch_fsmap learned FSMap version 144 2024-04-02T09:15:33.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3efb120b80 1 -- 172.21.15.89:0/4223462504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=2+}) v3 -- 0x55db4f3fe1d0 con 0x55db4f684830 2024-04-02T09:15:33.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3ec5600700 1 -- 172.21.15.89:0/4223462504 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee00ac2f0 con 0x55db4f684830 2024-04-02T09:15:33.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3ec5600700 1 -- 172.21.15.89:0/4223462504 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 144) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3ed8021440 con 0x55db4f684830 2024-04-02T09:15:33.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3efb120b80 10 client.13440 fetch_fsmap finished waiting for FSMap version 144 2024-04-02T09:15:33.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3efb120b80 10 client.13440 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-02T09:15:33.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3efb120b80 20 client.13440 populate_metadata read hostname 'smithi089' 2024-04-02T09:15:33.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3efb120b80 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.89:6836/3853118738,v1:172.21.15.89:6837/3853118738] conn(0x55db4f654ca0 0x55db4f655080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3efb120b80 4 client.13440 mds_command: new command op to 13308 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T09:15:33.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.783+0000 7f3efb120b80 1 -- 172.21.15.89:0/4223462504 --> [v2:172.21.15.89:6836/3853118738,v1:172.21.15.89:6837/3853118738] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f3ed4089070 con 0x55db4f654ca0 2024-04-02T09:15:33.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.784+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.89:6836/3853118738,v1:172.21.15.89:6837/3853118738] conn(0x55db4f654ca0 0x55db4f655080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.784+0000 7f3ee4c00700 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.89:6836/3853118738,v1:172.21.15.89:6837/3853118738] conn(0x55db4f654ca0 0x55db4f655080 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b74be75f53598950 server_cookie=f88834ae0a58155d in_seq=0 out_seq=0 2024-04-02T09:15:33.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.784+0000 7f3ec5600700 10 client.13440 ms_handle_connect on v2:172.21.15.89:6836/3853118738 2024-04-02T09:15:33.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.784+0000 7f3ec5600700 1 -- 172.21.15.89:0/4223462504 <== mds.0 v2:172.21.15.89:6836/3853118738 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x7f3ed4089070 con 0x55db4f654ca0 2024-04-02T09:15:33.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.784+0000 7f3ec5600700 10 client.13440 handle_command_reply: tid=0 2024-04-02T09:15:33.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.784+0000 7f3efb120b80 1 client.13440 shutdown 2024-04-02T09:15:33.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.784+0000 7f3efb120b80 1 -- 172.21.15.89:0/4223462504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f684830 msgr2=0x55db4f684c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:33.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.784+0000 7f3efb120b80 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f684830 0x55db4f684c10 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x55db4f683d00 tx=0x7f3ed8013c70 comp rx=0 tx=0).stop 2024-04-02T09:15:33.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.784+0000 7f3efb120b80 1 -- 172.21.15.89:0/4223462504 shutdown_connections 2024-04-02T09:15:33.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.784+0000 7f3efb120b80 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.89:6836/3853118738,v1:172.21.15.89:6837/3853118738] conn(0x55db4f654ca0 0x55db4f655080 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.784+0000 7f3efb120b80 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db4f684830 0x55db4f684c10 unknown :-1 s=CLOSED pgs=2997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.785+0000 7f3ec5600700 0 client.13440 ms_handle_reset on v2:172.21.15.89:6836/3853118738 2024-04-02T09:15:33.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.785+0000 7f3efb120b80 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55db4f655b60 0x55db4f6a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.785+0000 7f3efb120b80 1 --2- 172.21.15.89:0/4223462504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db4f657be0 0x55db4f706120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.785+0000 7f3efb120b80 1 -- 172.21.15.89:0/4223462504 >> 172.21.15.89:0/4223462504 conn(0x55db4f5b86a0 msgr2=0x55db4f6a2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:33.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.785+0000 7f3efb120b80 1 -- 172.21.15.89:0/4223462504 shutdown_connections 2024-04-02T09:15:33.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.785+0000 7f3efb120b80 1 -- 172.21.15.89:0/4223462504 wait complete. 2024-04-02T09:15:33.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.785+0000 7f3efb120b80 20 client.13440 trim_cache size 0 max 16384 2024-04-02T09:15:33.789 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:15:33.789 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "ddf7474f-4630-4a1e-ada2-3982714d822b", 2024-04-02T09:15:33.789 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-04-02T09:15:33.789 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 13308, 2024-04-02T09:15:33.789 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-04-02T09:15:33.789 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-04-02T09:15:33.789 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-04-02T09:15:33.789 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 7.0112837360000002, 2024-04-02T09:15:33.789 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 144, 2024-04-02T09:15:33.789 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 279, 2024-04-02T09:15:33.789 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 279, 2024-04-02T09:15:33.789 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 12.743827402000001 2024-04-02T09:15:33.789 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:15:33.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.785+0000 7f3ee7600700 1 -- 172.21.15.89:0/1839355858 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ec006adb0 msgr2=0x7f3ec006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:33.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.785+0000 7f3ee7600700 1 --2- 172.21.15.89:0/1839355858 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ec006adb0 0x7f3ec006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009130 tx=0x7f3ecc008030 comp rx=0 tx=0).stop 2024-04-02T09:15:33.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.785+0000 7f3ee7600700 1 -- 172.21.15.89:0/1839355858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee00ce670 msgr2=0x7f3ee01ed400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:33.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.785+0000 7f3ee7600700 1 --2- 172.21.15.89:0/1839355858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee00ce670 0x7f3ee01ed400 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400e810 tx=0x7f3ed400eb20 comp rx=0 tx=0).stop 2024-04-02T09:15:33.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.786+0000 7f3ee7600700 1 -- 172.21.15.89:0/1839355858 shutdown_connections 2024-04-02T09:15:33.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.786+0000 7f3ee7600700 1 --2- 172.21.15.89:0/1839355858 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ee015dfe0 0x7f3ee0158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.786+0000 7f3ee7600700 1 --2- 172.21.15.89:0/1839355858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee00ce670 0x7f3ee01ed400 unknown :-1 s=CLOSED pgs=2993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.786+0000 7f3ee7600700 1 --2- 172.21.15.89:0/1839355858 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ec006adb0 0x7f3ec006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.786+0000 7f3ee7600700 1 --2- 172.21.15.89:0/1839355858 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ee015a5c0 0x7f3ee00adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.786+0000 7f3ee7600700 1 -- 172.21.15.89:0/1839355858 >> 172.21.15.89:0/1839355858 conn(0x7f3ee00b0d70 msgr2=0x7f3ee00cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:33.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.786+0000 7f3ee7600700 1 -- 172.21.15.89:0/1839355858 shutdown_connections 2024-04-02T09:15:33.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.786+0000 7f3ee7600700 1 -- 172.21.15.89:0/1839355858 wait complete. 2024-04-02T09:15:33.802 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-02T09:15:33.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.995+0000 7f3bcb000700 1 Processor -- start 2024-04-02T09:15:33.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.995+0000 7f3bcb000700 1 -- start start 2024-04-02T09:15:33.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.996+0000 7f3bcb000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bc4161ca0 0x7f3bc40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:33.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.996+0000 7f3bcb000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bc40ceda0 con 0x7f3bc4160f10 2024-04-02T09:15:33.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.996+0000 7f3bcb000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bc41612f0 con 0x7f3bc40d18b0 2024-04-02T09:15:33.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.996+0000 7f3bcb000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc40d1d30 con 0x7f3bc4161ca0 2024-04-02T09:15:33.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.997+0000 7f3bc3e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3bc4160f10 0x7f3bc40d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48352/0 (socket says 172.21.15.89:48352) 2024-04-02T09:15:33.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.997+0000 7f3bc3e00700 1 -- 172.21.15.89:0/2033846532 learned_addr learned my addr 172.21.15.89:0/2033846532 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:33.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.997+0000 7f3bc9000700 1 --2- 172.21.15.89:0/2033846532 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bc4161ca0 0x7f3bc40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:33.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.997+0000 7f3bc2a00700 1 -- 172.21.15.89:0/2033846532 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928532785 0 0) 0x7f3bc40ceda0 con 0x7f3bc4160f10 2024-04-02T09:15:33.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.997+0000 7f3bc2a00700 1 -- 172.21.15.89:0/2033846532 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ba80035b0 con 0x7f3bc4160f10 2024-04-02T09:15:33.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.997+0000 7f3bc2a00700 1 -- 172.21.15.89:0/2033846532 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350595100 0 0) 0x7f3bc41612f0 con 0x7f3bc40d18b0 2024-04-02T09:15:33.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.997+0000 7f3bc2a00700 1 -- 172.21.15.89:0/2033846532 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bc40ceda0 con 0x7f3bc40d18b0 2024-04-02T09:15:33.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.997+0000 7f3bc2a00700 1 -- 172.21.15.89:0/2033846532 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3854809076 0 0) 0x7f3ba80035b0 con 0x7f3bc4160f10 2024-04-02T09:15:33.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.997+0000 7f3bc9000700 1 -- 172.21.15.89:0/2033846532 >> v1:172.21.15.113:6789/0 conn(0x7f3bc40d18b0 legacy=0x7f3bc4161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:33.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.997+0000 7f3bc9000700 1 -- 172.21.15.89:0/2033846532 >> v1:172.21.15.89:6789/0 conn(0x7f3bc4160f10 legacy=0x7f3bc40d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:33.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.997+0000 7f3bc9000700 1 -- 172.21.15.89:0/2033846532 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc40ce470 con 0x7f3bc4161ca0 2024-04-02T09:15:33.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.998+0000 7f3bc9000700 1 --2- 172.21.15.89:0/2033846532 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bc4161ca0 0x7f3bc40cf230 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009dc0 tx=0x7f3bbc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe0b5d89402a3cf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:33.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.998+0000 7f3bc2a00700 1 -- 172.21.15.89:0/2033846532 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc00e070 con 0x7f3bc4161ca0 2024-04-02T09:15:33.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.998+0000 7f3bc2a00700 1 -- 172.21.15.89:0/2033846532 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bbc003a20 con 0x7f3bc4161ca0 2024-04-02T09:15:33.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.998+0000 7f3bc2a00700 1 -- 172.21.15.89:0/2033846532 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4001670 con 0x7f3bc4161ca0 2024-04-02T09:15:33.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.998+0000 7f3bcb000700 1 -- 172.21.15.89:0/2033846532 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bc4161ca0 msgr2=0x7f3bc40cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:33.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.998+0000 7f3bcb000700 1 --2- 172.21.15.89:0/2033846532 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bc4161ca0 0x7f3bc40cf230 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009dc0 tx=0x7f3bbc00b330 comp rx=0 tx=0).stop 2024-04-02T09:15:33.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.999+0000 7f3bcb000700 1 -- 172.21.15.89:0/2033846532 shutdown_connections 2024-04-02T09:15:33.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.999+0000 7f3bcb000700 1 --2- 172.21.15.89:0/2033846532 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bc4161ca0 0x7f3bc40cf230 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:33.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.999+0000 7f3bcb000700 1 -- 172.21.15.89:0/2033846532 >> 172.21.15.89:0/2033846532 conn(0x7f3bc408aff0 msgr2=0x7f3bc40d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:34.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.999+0000 7f3bcb000700 1 -- 172.21.15.89:0/2033846532 shutdown_connections 2024-04-02T09:15:34.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:33.999+0000 7f3bcb000700 1 -- 172.21.15.89:0/2033846532 wait complete. 2024-04-02T09:15:34.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.000+0000 7f3bcb000700 1 Processor -- start 2024-04-02T09:15:34.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.000+0000 7f3bcb000700 1 -- start start 2024-04-02T09:15:34.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.001+0000 7f3bcb000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bc40d18b0 0x7f3bc41f63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:34.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.001+0000 7f3bcb000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bc4160f10 0x7f3bc41f6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:34.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.001+0000 7f3bcb000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bc4161ca0 0x7f3bc41f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:34.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.001+0000 7f3bcb000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc41f68e0 con 0x7f3bc40d18b0 2024-04-02T09:15:34.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.001+0000 7f3bcb000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc415a220 con 0x7f3bc4160f10 2024-04-02T09:15:34.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.001+0000 7f3bcb000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc415a500 con 0x7f3bc4161ca0 2024-04-02T09:15:34.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.002+0000 7f3bc3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bc40d18b0 0x7f3bc41f63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:34.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.002+0000 7f3bc3e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bc40d18b0 0x7f3bc41f63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52642/0 (socket says 172.21.15.89:52642) 2024-04-02T09:15:34.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.002+0000 7f3bc3e00700 1 -- 172.21.15.89:0/3379020478 learned_addr learned my addr 172.21.15.89:0/3379020478 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:34.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.002+0000 7f3bc9000700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bc4161ca0 0x7f3bc41f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:34.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.002+0000 7f3bc3400700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bc4160f10 0x7f3bc41f6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:34.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.002+0000 7f3bc3e00700 1 -- 172.21.15.89:0/3379020478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bc4161ca0 msgr2=0x7f3bc41f70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:34.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.002+0000 7f3bc3e00700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bc4161ca0 0x7f3bc41f70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:34.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.002+0000 7f3bc3e00700 1 -- 172.21.15.89:0/3379020478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bc4160f10 msgr2=0x7f3bc41f6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:34.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.002+0000 7f3bc3e00700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bc4160f10 0x7f3bc41f6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:34.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.002+0000 7f3bc3e00700 1 -- 172.21.15.89:0/3379020478 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bbc009a70 con 0x7f3bc40d18b0 2024-04-02T09:15:34.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.002+0000 7f3bc9000700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bc4161ca0 0x7f3bc41f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:34.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.002+0000 7f3bc3e00700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bc40d18b0 0x7f3bc41f63a0 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400c520 tx=0x7f3bb400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:34.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.002+0000 7f3bbbe00700 1 -- 172.21.15.89:0/3379020478 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4004c80 con 0x7f3bc40d18b0 2024-04-02T09:15:34.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.003+0000 7f3bbbe00700 1 -- 172.21.15.89:0/3379020478 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb4004de0 con 0x7f3bc40d18b0 2024-04-02T09:15:34.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.003+0000 7f3bcb000700 1 -- 172.21.15.89:0/3379020478 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc415a7e0 con 0x7f3bc40d18b0 2024-04-02T09:15:34.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.003+0000 7f3bbbe00700 1 -- 172.21.15.89:0/3379020478 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4007310 con 0x7f3bc40d18b0 2024-04-02T09:15:34.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.003+0000 7f3bcb000700 1 -- 172.21.15.89:0/3379020478 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc415ada0 con 0x7f3bc40d18b0 2024-04-02T09:15:34.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.004+0000 7f3bcb000700 1 -- 172.21.15.89:0/3379020478 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc415afc0 con 0x7f3bc40d18b0 2024-04-02T09:15:34.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.005+0000 7f3bbbe00700 1 -- 172.21.15.89:0/3379020478 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3bb4007470 con 0x7f3bc40d18b0 2024-04-02T09:15:34.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.005+0000 7f3bbbe00700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ba406adb0 0x7f3ba406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:34.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.005+0000 7f3bbbe00700 1 -- 172.21.15.89:0/3379020478 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f3bb4089360 con 0x7f3bc40d18b0 2024-04-02T09:15:34.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.006+0000 7f3bc3400700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ba406adb0 0x7f3ba406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:34.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.006+0000 7f3bc3400700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ba406adb0 0x7f3ba406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3bc40cf6e0 tx=0x7f3bb0009040 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:34.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.010+0000 7f3bbbe00700 1 -- 172.21.15.89:0/3379020478 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3bb4054300 con 0x7f3bc40d18b0 2024-04-02T09:15:34.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.255+0000 7f3bcb000700 1 -- 172.21.15.89:0/3379020478 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3bc4067720 con 0x7f3bc40d18b0 2024-04-02T09:15:34.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.255+0000 7f3bbbe00700 1 -- 172.21.15.89:0/3379020478 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v144) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3bb4057920 con 0x7f3bc40d18b0 2024-04-02T09:15:34.254 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T09:15:34.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.257+0000 7f3bcb000700 1 -- 172.21.15.89:0/3379020478 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ba406adb0 msgr2=0x7f3ba406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:34.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.257+0000 7f3bcb000700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ba406adb0 0x7f3ba406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3bc40cf6e0 tx=0x7f3bb0009040 comp rx=0 tx=0).stop 2024-04-02T09:15:34.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.257+0000 7f3bcb000700 1 -- 172.21.15.89:0/3379020478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bc40d18b0 msgr2=0x7f3bc41f63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:34.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.257+0000 7f3bcb000700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bc40d18b0 0x7f3bc41f63a0 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400c520 tx=0x7f3bb400c8e0 comp rx=0 tx=0).stop 2024-04-02T09:15:34.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.258+0000 7f3bcb000700 1 -- 172.21.15.89:0/3379020478 shutdown_connections 2024-04-02T09:15:34.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.258+0000 7f3bcb000700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3bc4161ca0 0x7f3bc41f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:34.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.258+0000 7f3bcb000700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bc40d18b0 0x7f3bc41f63a0 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:34.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.258+0000 7f3bcb000700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f3ba406adb0 0x7f3ba406d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:34.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.258+0000 7f3bcb000700 1 --2- 172.21.15.89:0/3379020478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3bc4160f10 0x7f3bc41f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:34.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.258+0000 7f3bcb000700 1 -- 172.21.15.89:0/3379020478 >> 172.21.15.89:0/3379020478 conn(0x7f3bc408aff0 msgr2=0x7f3bc40cadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:34.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.258+0000 7f3bcb000700 1 -- 172.21.15.89:0/3379020478 shutdown_connections 2024-04-02T09:15:34.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:34.258+0000 7f3bcb000700 1 -- 172.21.15.89:0/3379020478 wait complete. 2024-04-02T09:15:34.277 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-02T09:15:34.277 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T09:15:34.277 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-04-02T09:15:34.277 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T09:15:34.277 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-02T09:15:34.277 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-02T09:15:34.277 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-02T09:15:34.277 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T09:15:34.277 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:15:34.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:15:34.278 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-04-02T09:15:34.324 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T09:15:34.324 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T09:15:34.324 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T09:15:34.324 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:15:34.325 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-04-02T09:15:34.325 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:15:34.325 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:15:34.325 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:15:34.325 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-04-02T09:15:34.325 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:15:34.325 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:15:34.325 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-04-02T09:15:34.325 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T09:15:34.325 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:15:34.325 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T09:15:34.325 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:15:34.325 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T09:15:34.326 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 36494sec preferred_lft 36494sec 2024-04-02T09:15:34.326 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-04-02T09:15:34.326 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:15:34.326 INFO:teuthology.orchestra.run.smithi089.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-02T09:15:34.326 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff 2024-04-02T09:15:34.326 INFO:teuthology.orchestra.run.smithi089.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-02T09:15:34.326 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:15:34.326 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::a83b:3eff:fe5c:a7eb/64 scope link 2024-04-02T09:15:34.326 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:15:34.326 INFO:teuthology.orchestra.run.smithi089.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T09:15:34.326 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether ba:3e:75:e5:86:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T09:15:34.326 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::b83e:75ff:fee5:8649/64 scope link 2024-04-02T09:15:34.326 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:15:34.327 INFO:teuthology.orchestra.run.smithi089.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-02T09:15:34.327 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 06:c1:f4:47:98:85 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T09:15:34.327 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::4c1:f4ff:fe47:9885/64 scope link 2024-04-02T09:15:34.327 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-04-02T09:15:34.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:15:34.327 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T09:15:34.429 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-04-02T09:15:34.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:15:34.432 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-04-02T09:15:34.478 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-04-02T09:15:34.478 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T09:15:34.479 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T09:15:34.479 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T09:15:34.479 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T09:15:34.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:15:34.480 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-04-02T09:15:34.480 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-04-02T09:15:34.480 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T09:15:34.480 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T09:15:34.480 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T09:15:34.480 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-04-02T09:15:34.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:15:34.641 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T09:15:34.686 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T09:15:34.686 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T09:15:34.686 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:15:34.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:15:34.686 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-04-02T09:15:34.795 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T09:15:34.795 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-02T09:15:34.840 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-02T09:15:34.841 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T09:15:34.841 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-02T09:15:34.925 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-02T09:15:34.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:15:34.925 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:15:35.064 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:15:35.065 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T09:15:35.091 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-02T09:15:35.091 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-02T09:15:35.091 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.113:3300,172.21.15.113:3301" 2024-04-02T09:15:35.091 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@ddf7474f-4630-4a1e-ada2-3982714d822b.cephfs=/ 2024-04-02T09:15:35.091 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.89:3300/172.21.15.113:3300/172.21.15.113:3301" will pass to kernel 2024-04-02T09:15:35.095 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T09:15:35.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:15:35.096 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-02T09:15:35.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:15:35.163 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T09:15:35.264 INFO:teuthology.orchestra.run.smithi089.stdout:client12232 2024-04-02T09:15:35.264 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:15:35.264 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:15:35.308 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-02T09:15:35.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.540+0000 7efc05800700 1 Processor -- start 2024-04-02T09:15:35.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.541+0000 7efc05800700 1 -- start start 2024-04-02T09:15:35.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.542+0000 7efc05800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efc00160d70 0x7efc000b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:35.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.542+0000 7efc05800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efc0015ef70 0x7efc0015e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:35.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.542+0000 7efc05800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc00161150 con 0x7efc0006a790 2024-04-02T09:15:35.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.542+0000 7efc05800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc000b4cd0 con 0x7efc0015ef70 2024-04-02T09:15:35.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.542+0000 7efc05800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc0006dba0 con 0x7efc00160d70 2024-04-02T09:15:35.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.542+0000 7efbff400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efc0006a790 0x7efc001613a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48356/0 (socket says 172.21.15.89:48356) 2024-04-02T09:15:35.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.542+0000 7efbfe000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efc0015ef70 0x7efc0015e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:35.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.542+0000 7efbff400700 1 -- 172.21.15.89:0/1384655486 learned_addr learned my addr 172.21.15.89:0/1384655486 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:35.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.543+0000 7efbfea00700 1 --2- 172.21.15.89:0/1384655486 >> v2:172.21.15.113:3300/0 conn(0x7efc00160d70 0x7efc000b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:35.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.543+0000 7efbfe000700 1 -- 172.21.15.89:0/1384655486 >> v2:172.21.15.113:3300/0 conn(0x7efc00160d70 msgr2=0x7efc000b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:35.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.543+0000 7efbfe000700 1 --2- 172.21.15.89:0/1384655486 >> v2:172.21.15.113:3300/0 conn(0x7efc00160d70 0x7efc000b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:35.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.543+0000 7efbfe000700 1 -- 172.21.15.89:0/1384655486 >> v1:172.21.15.89:6789/0 conn(0x7efc0006a790 legacy=0x7efc001613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:35.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.543+0000 7efbfe000700 1 -- 172.21.15.89:0/1384655486 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0015ed30 con 0x7efc0015ef70 2024-04-02T09:15:35.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.543+0000 7efbfea00700 1 --2- 172.21.15.89:0/1384655486 >> v2:172.21.15.113:3300/0 conn(0x7efc00160d70 0x7efc000b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:35.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.543+0000 7efbfd600700 1 -- 172.21.15.89:0/1384655486 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879896910 0 0) 0x7efc00161150 con 0x7efc0006a790 2024-04-02T09:15:35.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.543+0000 7efbfe000700 1 --2- 172.21.15.89:0/1384655486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc0015ef70 0x7efc0015e7f0 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7efc0006c880 tx=0x7efbec009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf96c01d25e5c4ce server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:35.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.543+0000 7efbfd600700 1 -- 172.21.15.89:0/1384655486 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbec011830 con 0x7efc0015ef70 2024-04-02T09:15:35.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.543+0000 7efbfd600700 1 -- 172.21.15.89:0/1384655486 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbec011990 con 0x7efc0015ef70 2024-04-02T09:15:35.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.543+0000 7efbfd600700 1 -- 172.21.15.89:0/1384655486 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbec011d20 con 0x7efc0015ef70 2024-04-02T09:15:35.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.544+0000 7efc05800700 1 -- 172.21.15.89:0/1384655486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc0015ef70 msgr2=0x7efc0015e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:35.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.544+0000 7efc05800700 1 --2- 172.21.15.89:0/1384655486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc0015ef70 0x7efc0015e7f0 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7efc0006c880 tx=0x7efbec009b20 comp rx=0 tx=0).stop 2024-04-02T09:15:35.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.544+0000 7efc05800700 1 -- 172.21.15.89:0/1384655486 shutdown_connections 2024-04-02T09:15:35.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.544+0000 7efc05800700 1 --2- 172.21.15.89:0/1384655486 >> v2:172.21.15.113:3300/0 conn(0x7efc00160d70 0x7efc000b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:35.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.544+0000 7efc05800700 1 --2- 172.21.15.89:0/1384655486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc0015ef70 0x7efc0015e7f0 unknown :-1 s=CLOSED pgs=3001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:35.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.544+0000 7efc05800700 1 -- 172.21.15.89:0/1384655486 >> 172.21.15.89:0/1384655486 conn(0x7efc000b5720 msgr2=0x7efc000b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:35.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.544+0000 7efc05800700 1 -- 172.21.15.89:0/1384655486 shutdown_connections 2024-04-02T09:15:35.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.544+0000 7efc05800700 1 -- 172.21.15.89:0/1384655486 wait complete. 2024-04-02T09:15:35.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.546+0000 7efc05800700 1 Processor -- start 2024-04-02T09:15:35.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.546+0000 7efc05800700 1 -- start start 2024-04-02T09:15:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.547+0000 7efc05800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efc0006a790 0x7efc00146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.547+0000 7efc05800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0015ef70 0x7efc000b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.547+0000 7efc05800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc00160d70 0x7efc000b1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.547+0000 7efc05800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc001491c0 con 0x7efc00160d70 2024-04-02T09:15:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.547+0000 7efc05800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc001494a0 con 0x7efc0015ef70 2024-04-02T09:15:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.547+0000 7efc05800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc00149780 con 0x7efc0006a790 2024-04-02T09:15:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.547+0000 7efbff400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc00160d70 0x7efc000b1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.547+0000 7efbff400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc00160d70 0x7efc000b1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52654/0 (socket says 172.21.15.89:52654) 2024-04-02T09:15:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.547+0000 7efbff400700 1 -- 172.21.15.89:0/106677994 learned_addr learned my addr 172.21.15.89:0/106677994 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.547+0000 7efbfea00700 1 --2- 172.21.15.89:0/106677994 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efc0006a790 0x7efc00146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.548+0000 7efbfe000700 1 --2- 172.21.15.89:0/106677994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0015ef70 0x7efc000b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.548+0000 7efbff400700 1 -- 172.21.15.89:0/106677994 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efc0006a790 msgr2=0x7efc00146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.548+0000 7efbff400700 1 --2- 172.21.15.89:0/106677994 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efc0006a790 0x7efc00146c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:35.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.548+0000 7efbff400700 1 -- 172.21.15.89:0/106677994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0015ef70 msgr2=0x7efc000b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:35.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.548+0000 7efbff400700 1 --2- 172.21.15.89:0/106677994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0015ef70 0x7efc000b13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:35.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.548+0000 7efbff400700 1 -- 172.21.15.89:0/106677994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbec0094a0 con 0x7efc00160d70 2024-04-02T09:15:35.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.548+0000 7efbff400700 1 --2- 172.21.15.89:0/106677994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc00160d70 0x7efc000b1920 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7efbf8002230 tx=0x7efbf800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:35.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.548+0000 7efbf6a00700 1 -- 172.21.15.89:0/106677994 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf800aac0 con 0x7efc00160d70 2024-04-02T09:15:35.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.548+0000 7efbf6a00700 1 -- 172.21.15.89:0/106677994 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf800ac40 con 0x7efc00160d70 2024-04-02T09:15:35.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.548+0000 7efc05800700 1 -- 172.21.15.89:0/106677994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc00149a60 con 0x7efc00160d70 2024-04-02T09:15:35.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.548+0000 7efbf6a00700 1 -- 172.21.15.89:0/106677994 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf80070d0 con 0x7efc00160d70 2024-04-02T09:15:35.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.549+0000 7efc05800700 1 -- 172.21.15.89:0/106677994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc00147160 con 0x7efc00160d70 2024-04-02T09:15:35.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.549+0000 7efc05800700 1 -- 172.21.15.89:0/106677994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc00149f80 con 0x7efc00160d70 2024-04-02T09:15:35.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.551+0000 7efbf6a00700 1 -- 172.21.15.89:0/106677994 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7efbf8007250 con 0x7efc00160d70 2024-04-02T09:15:35.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.552+0000 7efbf6a00700 1 --2- 172.21.15.89:0/106677994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efbe006adb0 0x7efbe006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:35.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.552+0000 7efbf6a00700 1 -- 172.21.15.89:0/106677994 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7efbf8089150 con 0x7efc00160d70 2024-04-02T09:15:35.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.552+0000 7efbfea00700 1 --2- 172.21.15.89:0/106677994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efbe006adb0 0x7efbe006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:35.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.553+0000 7efbfea00700 1 --2- 172.21.15.89:0/106677994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efbe006adb0 0x7efbe006d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efbf000c940 tx=0x7efbf0008040 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:35.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.557+0000 7efbf6a00700 1 -- 172.21.15.89:0/106677994 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efc00149f80 con 0x7efc00160d70 2024-04-02T09:15:35.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.787+0000 7efc05800700 1 -- 172.21.15.89:0/106677994 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7efc000b54a0 con 0x7efbe006adb0 2024-04-02T09:15:35.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.789+0000 7efbf6a00700 1 -- 172.21.15.89:0/106677994 <== mgr.13065 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7efbf000cdb0 con 0x7efbe006adb0 2024-04-02T09:15:35.788 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-04-02T09:15:35.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-02T09:15:35.788 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephfs" 2024-04-02T09:15:35.788 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-02T09:15:35.788 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-04-02T09:15:35.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.791+0000 7efc05800700 1 -- 172.21.15.89:0/106677994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efbe006adb0 msgr2=0x7efbe006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:35.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.791+0000 7efc05800700 1 --2- 172.21.15.89:0/106677994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efbe006adb0 0x7efbe006d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efbf000c940 tx=0x7efbf0008040 comp rx=0 tx=0).stop 2024-04-02T09:15:35.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.791+0000 7efc05800700 1 -- 172.21.15.89:0/106677994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc00160d70 msgr2=0x7efc000b1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:35.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.791+0000 7efc05800700 1 --2- 172.21.15.89:0/106677994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc00160d70 0x7efc000b1920 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7efbf8002230 tx=0x7efbf800cf50 comp rx=0 tx=0).stop 2024-04-02T09:15:35.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.792+0000 7efc05800700 1 -- 172.21.15.89:0/106677994 shutdown_connections 2024-04-02T09:15:35.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.792+0000 7efc05800700 1 --2- 172.21.15.89:0/106677994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc00160d70 0x7efc000b1920 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:35.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.792+0000 7efc05800700 1 --2- 172.21.15.89:0/106677994 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efc0006a790 0x7efc00146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:35.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.792+0000 7efc05800700 1 --2- 172.21.15.89:0/106677994 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7efbe006adb0 0x7efbe006d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:35.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.792+0000 7efc05800700 1 --2- 172.21.15.89:0/106677994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0015ef70 0x7efc000b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:35.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.792+0000 7efc05800700 1 -- 172.21.15.89:0/106677994 >> 172.21.15.89:0/106677994 conn(0x7efc000b5720 msgr2=0x7efc00159020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:35.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.792+0000 7efc05800700 1 -- 172.21.15.89:0/106677994 shutdown_connections 2024-04-02T09:15:35.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:35.792+0000 7efc05800700 1 -- 172.21.15.89:0/106677994 wait complete. 2024-04-02T09:15:35.809 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-02T09:15:36.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.002+0000 7fa45e000700 1 Processor -- start 2024-04-02T09:15:36.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.003+0000 7fa45e000700 1 -- start start 2024-04-02T09:15:36.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.004+0000 7fa45e000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa4580cd480 0x7fa45806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:36.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.004+0000 7fa45e000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4580cdae0 con 0x7fa4580cd020 2024-04-02T09:15:36.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.004+0000 7fa45e000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4580cddc0 con 0x7fa4580cbb30 2024-04-02T09:15:36.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.004+0000 7fa45e000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4580ce0a0 con 0x7fa4580cd480 2024-04-02T09:15:36.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.004+0000 7fa44fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa4580cd020 0x7fa4580b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48366/0 (socket says 172.21.15.89:48366) 2024-04-02T09:15:36.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.004+0000 7fa44fe00700 1 -- 172.21.15.89:0/1230552558 learned_addr learned my addr 172.21.15.89:0/1230552558 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:36.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.004+0000 7fa457e00700 1 --2- 172.21.15.89:0/1230552558 >> v2:172.21.15.113:3300/0 conn(0x7fa4580cd480 0x7fa45806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:36.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.004+0000 7fa456a00700 1 -- 172.21.15.89:0/1230552558 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764006164 0 0) 0x7fa4580cdae0 con 0x7fa4580cd020 2024-04-02T09:15:36.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.004+0000 7fa456a00700 1 -- 172.21.15.89:0/1230552558 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa43c0035b0 con 0x7fa4580cd020 2024-04-02T09:15:36.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa456a00700 1 -- 172.21.15.89:0/1230552558 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3790813971 0 0) 0x7fa43c0035b0 con 0x7fa4580cd020 2024-04-02T09:15:36.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa456a00700 1 -- 172.21.15.89:0/1230552558 >> v2:172.21.15.113:3300/0 conn(0x7fa4580cd480 msgr2=0x7fa45806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:36.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa456a00700 1 --2- 172.21.15.89:0/1230552558 >> v2:172.21.15.113:3300/0 conn(0x7fa4580cd480 0x7fa45806e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa456a00700 1 -- 172.21.15.89:0/1230552558 >> v1:172.21.15.113:6789/0 conn(0x7fa4580cbb30 legacy=0x7fa4580cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:36.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa456a00700 1 -- 172.21.15.89:0/1230552558 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45806dc20 con 0x7fa4580cd020 2024-04-02T09:15:36.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa456a00700 1 -- 172.21.15.89:0/1230552558 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fa440001670 con 0x7fa4580cd020 2024-04-02T09:15:36.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa456a00700 1 -- 172.21.15.89:0/1230552558 >> v1:172.21.15.89:6789/0 conn(0x7fa4580cd020 legacy=0x7fa4580b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:36.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa456a00700 1 --2- 172.21.15.89:0/1230552558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa43c003a00 0x7fa43c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:36.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa456a00700 1 --2- 172.21.15.89:0/1230552558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa43c004850 0x7fa43c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:36.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa456a00700 1 --2- 172.21.15.89:0/1230552558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c005210 0x7fa43c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:36.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa456a00700 1 -- 172.21.15.89:0/1230552558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580ce0a0 con 0x7fa43c005210 2024-04-02T09:15:36.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa456a00700 1 -- 172.21.15.89:0/1230552558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c00c0b0 con 0x7fa43c003a00 2024-04-02T09:15:36.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa456a00700 1 -- 172.21.15.89:0/1230552558 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c00c270 con 0x7fa43c004850 2024-04-02T09:15:36.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa457e00700 1 --2- 172.21.15.89:0/1230552558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c005210 0x7fa43c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:36.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa457400700 1 --2- 172.21.15.89:0/1230552558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa43c003a00 0x7fa43c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:36.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.005+0000 7fa44fe00700 1 --2- 172.21.15.89:0/1230552558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa43c004850 0x7fa43c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:36.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.006+0000 7fa457e00700 1 -- 172.21.15.89:0/1230552558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa43c004850 msgr2=0x7fa43c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:36.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.006+0000 7fa457e00700 1 --2- 172.21.15.89:0/1230552558 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa43c004850 0x7fa43c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.006+0000 7fa457e00700 1 -- 172.21.15.89:0/1230552558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa43c003a00 msgr2=0x7fa43c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:36.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.006+0000 7fa457e00700 1 --2- 172.21.15.89:0/1230552558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa43c003a00 0x7fa43c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.006+0000 7fa457e00700 1 -- 172.21.15.89:0/1230552558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43c0035b0 con 0x7fa43c005210 2024-04-02T09:15:36.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.006+0000 7fa457e00700 1 -- 172.21.15.89:0/1230552558 reap_dead start 2024-04-02T09:15:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.006+0000 7fa457e00700 1 --2- 172.21.15.89:0/1230552558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c005210 0x7fa43c00ba10 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7fa450009fc0 tx=0x7fa43c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f3c399178f2080 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.006+0000 7fa456a00700 1 -- 172.21.15.89:0/1230552558 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa450003a20 con 0x7fa43c005210 2024-04-02T09:15:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.006+0000 7fa456a00700 1 -- 172.21.15.89:0/1230552558 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45000f040 con 0x7fa43c005210 2024-04-02T09:15:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.006+0000 7fa456a00700 1 -- 172.21.15.89:0/1230552558 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa450003da0 con 0x7fa43c005210 2024-04-02T09:15:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.007+0000 7fa45e000700 1 -- 172.21.15.89:0/1230552558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c005210 msgr2=0x7fa43c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.007+0000 7fa45e000700 1 --2- 172.21.15.89:0/1230552558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c005210 0x7fa43c00ba10 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7fa450009fc0 tx=0x7fa43c005880 comp rx=0 tx=0).stop 2024-04-02T09:15:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.007+0000 7fa45e000700 1 -- 172.21.15.89:0/1230552558 shutdown_connections 2024-04-02T09:15:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.007+0000 7fa45e000700 1 --2- 172.21.15.89:0/1230552558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c005210 0x7fa43c00ba10 unknown :-1 s=CLOSED pgs=3004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.007+0000 7fa45e000700 1 -- 172.21.15.89:0/1230552558 >> 172.21.15.89:0/1230552558 conn(0x7fa4580b0d70 msgr2=0x7fa4580b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.007+0000 7fa45e000700 1 -- 172.21.15.89:0/1230552558 shutdown_connections 2024-04-02T09:15:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.007+0000 7fa45e000700 1 -- 172.21.15.89:0/1230552558 wait complete. 2024-04-02T09:15:36.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.008+0000 7fa45e000700 1 Processor -- start 2024-04-02T09:15:36.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.009+0000 7fa45e000700 1 -- start start 2024-04-02T09:15:36.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa45e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c005210 0x7fa458157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:36.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa45e000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4580ca0a0 0x7fa4581f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:36.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa45e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4581f3960 0x7fa4580b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:36.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa45e000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580993e0 con 0x7fa43c005210 2024-04-02T09:15:36.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa45e000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580c83f0 con 0x7fa4580ca0a0 2024-04-02T09:15:36.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa45e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4581f6ed0 con 0x7fa4581f3960 2024-04-02T09:15:36.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa457400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c005210 0x7fa458157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:36.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa457400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c005210 0x7fa458157930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52672/0 (socket says 172.21.15.89:52672) 2024-04-02T09:15:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa457400700 1 -- 172.21.15.89:0/1142204418 learned_addr learned my addr 172.21.15.89:0/1142204418 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa44fe00700 1 --2- 172.21.15.89:0/1142204418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4580ca0a0 0x7fa4581f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa457e00700 1 --2- 172.21.15.89:0/1142204418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4581f3960 0x7fa4580b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa457400700 1 -- 172.21.15.89:0/1142204418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4581f3960 msgr2=0x7fa4580b2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa457400700 1 --2- 172.21.15.89:0/1142204418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4581f3960 0x7fa4580b2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa457400700 1 -- 172.21.15.89:0/1142204418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4580ca0a0 msgr2=0x7fa4581f3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa457400700 1 --2- 172.21.15.89:0/1142204418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4580ca0a0 0x7fa4581f3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa457400700 1 -- 172.21.15.89:0/1142204418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4581f71b0 con 0x7fa43c005210 2024-04-02T09:15:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.010+0000 7fa457400700 1 --2- 172.21.15.89:0/1142204418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c005210 0x7fa458157930 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7fa44800f820 tx=0x7fa44800fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.011+0000 7fa44f400700 1 -- 172.21.15.89:0/1142204418 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4480052e0 con 0x7fa43c005210 2024-04-02T09:15:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.011+0000 7fa44f400700 1 -- 172.21.15.89:0/1142204418 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa448005460 con 0x7fa43c005210 2024-04-02T09:15:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.011+0000 7fa44f400700 1 -- 172.21.15.89:0/1142204418 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4480038c0 con 0x7fa43c005210 2024-04-02T09:15:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.011+0000 7fa45e000700 1 -- 172.21.15.89:0/1142204418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4581f7490 con 0x7fa43c005210 2024-04-02T09:15:36.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.011+0000 7fa45e000700 1 -- 172.21.15.89:0/1142204418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4581f69f0 con 0x7fa43c005210 2024-04-02T09:15:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.012+0000 7fa45e000700 1 -- 172.21.15.89:0/1142204418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa458067720 con 0x7fa43c005210 2024-04-02T09:15:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.013+0000 7fa44f400700 1 -- 172.21.15.89:0/1142204418 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 178) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa448003a20 con 0x7fa43c005210 2024-04-02T09:15:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.013+0000 7fa44f400700 1 --2- 172.21.15.89:0/1142204418 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa43406adb0 0x7fa43406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.014+0000 7fa44f400700 1 -- 172.21.15.89:0/1142204418 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fa4480893e0 con 0x7fa43c005210 2024-04-02T09:15:36.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.014+0000 7fa44fe00700 1 --2- 172.21.15.89:0/1142204418 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa43406adb0 0x7fa43406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:36.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.014+0000 7fa44fe00700 1 --2- 172.21.15.89:0/1142204418 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa43406adb0 0x7fa43406d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa4580ca480 tx=0x7fa440009040 comp rx=0 tx=0).ready entity=mgr.13065 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:36.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.018+0000 7fa44f400700 1 -- 172.21.15.89:0/1142204418 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa4480542f0 con 0x7fa43c005210 2024-04-02T09:15:36.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.276+0000 7fa45e000700 1 -- 172.21.15.89:0/1142204418 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fa4580c8530 con 0x7fa43c005210 2024-04-02T09:15:36.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.322+0000 7fa44f400700 1 -- 172.21.15.89:0/1142204418 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 179) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fa448087740 con 0x7fa43c005210 2024-04-02T09:15:36.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.324+0000 7fa44f400700 1 -- 172.21.15.89:0/1142204418 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v179) v1 ==== 92+0+0 (secure 0 0 0) 0x7fa448086d50 con 0x7fa43c005210 2024-04-02T09:15:36.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.326+0000 7fa45e000700 1 -- 172.21.15.89:0/1142204418 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa43406adb0 msgr2=0x7fa43406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:36.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.326+0000 7fa45e000700 1 --2- 172.21.15.89:0/1142204418 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa43406adb0 0x7fa43406d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa4580ca480 tx=0x7fa440009040 comp rx=0 tx=0).stop 2024-04-02T09:15:36.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.326+0000 7fa45e000700 1 -- 172.21.15.89:0/1142204418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c005210 msgr2=0x7fa458157930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:36.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.326+0000 7fa45e000700 1 --2- 172.21.15.89:0/1142204418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c005210 0x7fa458157930 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7fa44800f820 tx=0x7fa44800fbe0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.326+0000 7fa45e000700 1 -- 172.21.15.89:0/1142204418 shutdown_connections 2024-04-02T09:15:36.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.326+0000 7fa45e000700 1 --2- 172.21.15.89:0/1142204418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4581f3960 0x7fa4580b2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.326+0000 7fa45e000700 1 --2- 172.21.15.89:0/1142204418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa43c005210 0x7fa458157930 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.326+0000 7fa45e000700 1 --2- 172.21.15.89:0/1142204418 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa43406adb0 0x7fa43406d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.326+0000 7fa45e000700 1 --2- 172.21.15.89:0/1142204418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4580ca0a0 0x7fa4581f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.326+0000 7fa45e000700 1 -- 172.21.15.89:0/1142204418 >> 172.21.15.89:0/1142204418 conn(0x7fa4580b0d70 msgr2=0x7fa4580ae7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:36.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.327+0000 7fa45e000700 1 -- 172.21.15.89:0/1142204418 shutdown_connections 2024-04-02T09:15:36.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.327+0000 7fa45e000700 1 -- 172.21.15.89:0/1142204418 wait complete. 2024-04-02T09:15:36.345 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-02T09:15:36.541 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:36.543+0000 7f4ab9478240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:15:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.548+0000 7f06fc800700 1 Processor -- start 2024-04-02T09:15:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.548+0000 7f06fc800700 1 -- start start 2024-04-02T09:15:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.549+0000 7f06fc800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06f415dad0 0x7f06f40ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.549+0000 7f06fc800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f06f40d1630 0x7f06f40d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.550+0000 7f06fc800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f415f820 con 0x7f06f40d1fd0 2024-04-02T09:15:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.550+0000 7f06fc800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f40cfcc0 con 0x7f06f40d1630 2024-04-02T09:15:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.550+0000 7f06fc800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f40cd9f0 con 0x7f06f415dad0 2024-04-02T09:15:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.550+0000 7f06f9400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f06f40d1630 0x7f06f40d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.550+0000 7f06f9400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f06f40d1630 0x7f06f40d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52690/0 (socket says 172.21.15.89:52690) 2024-04-02T09:15:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.550+0000 7f06f9400700 1 -- 172.21.15.89:0/3356390702 learned_addr learned my addr 172.21.15.89:0/3356390702 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.550+0000 7f06f9e00700 1 --2- 172.21.15.89:0/3356390702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06f415dad0 0x7f06f40ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.550+0000 7f06f9400700 1 -- 172.21.15.89:0/3356390702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06f415dad0 msgr2=0x7f06f40ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.550+0000 7f06f9400700 1 --2- 172.21.15.89:0/3356390702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06f415dad0 0x7f06f40ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.550+0000 7f06f9400700 1 -- 172.21.15.89:0/3356390702 >> v1:172.21.15.113:6789/0 conn(0x7f06f40d1fd0 legacy=0x7f06f415deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.550+0000 7f06f9400700 1 -- 172.21.15.89:0/3356390702 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f40d2460 con 0x7f06f40d1630 2024-04-02T09:15:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.551+0000 7f06f9400700 1 --2- 172.21.15.89:0/3356390702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06f40d1630 0x7f06f40d1a90 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7f06e40098f0 tx=0x7f06e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3584e116f7988fd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.551+0000 7f06f8a00700 1 -- 172.21.15.89:0/3356390702 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e4011830 con 0x7f06f40d1630 2024-04-02T09:15:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.551+0000 7f06f8a00700 1 -- 172.21.15.89:0/3356390702 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e4011990 con 0x7f06f40d1630 2024-04-02T09:15:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.551+0000 7f06f8a00700 1 -- 172.21.15.89:0/3356390702 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e4011d20 con 0x7f06f40d1630 2024-04-02T09:15:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.551+0000 7f06fc800700 1 -- 172.21.15.89:0/3356390702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06f40d1630 msgr2=0x7f06f40d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.551+0000 7f06fc800700 1 --2- 172.21.15.89:0/3356390702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06f40d1630 0x7f06f40d1a90 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7f06e40098f0 tx=0x7f06e4009c00 comp rx=0 tx=0).stop 2024-04-02T09:15:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.551+0000 7f06fc800700 1 -- 172.21.15.89:0/3356390702 shutdown_connections 2024-04-02T09:15:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.551+0000 7f06fc800700 1 --2- 172.21.15.89:0/3356390702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06f415dad0 0x7f06f40ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.551+0000 7f06fc800700 1 --2- 172.21.15.89:0/3356390702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06f40d1630 0x7f06f40d1a90 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.551+0000 7f06fc800700 1 -- 172.21.15.89:0/3356390702 >> 172.21.15.89:0/3356390702 conn(0x7f06f408aff0 msgr2=0x7f06f40d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.552+0000 7f06fc800700 1 -- 172.21.15.89:0/3356390702 shutdown_connections 2024-04-02T09:15:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.552+0000 7f06fc800700 1 -- 172.21.15.89:0/3356390702 wait complete. 2024-04-02T09:15:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.553+0000 7f06fc800700 1 Processor -- start 2024-04-02T09:15:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.553+0000 7f06fc800700 1 -- start start 2024-04-02T09:15:36.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.554+0000 7f06fc800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06f40d1630 0x7f06f41440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:36.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.554+0000 7f06fc800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06f40d1fd0 0x7f06f40c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:36.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.554+0000 7f06fc800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06f415dad0 0x7f06f40c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.554+0000 7f06fc800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f40ca320 con 0x7f06f415dad0 2024-04-02T09:15:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.554+0000 7f06fc800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f4144600 con 0x7f06f40d1630 2024-04-02T09:15:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.554+0000 7f06fc800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f4147950 con 0x7f06f40d1fd0 2024-04-02T09:15:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.555+0000 7f06f9e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06f40d1630 0x7f06f41440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.555+0000 7f06fa800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06f415dad0 0x7f06f40c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.555+0000 7f06f9e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06f40d1630 0x7f06f41440c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:59540/0 (socket says 172.21.15.89:59540) 2024-04-02T09:15:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.555+0000 7f06f9e00700 1 -- 172.21.15.89:0/2123695442 learned_addr learned my addr 172.21.15.89:0/2123695442 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.555+0000 7f06f9400700 1 --2- 172.21.15.89:0/2123695442 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06f40d1fd0 0x7f06f40c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.555+0000 7f06fa800700 1 -- 172.21.15.89:0/2123695442 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06f40d1fd0 msgr2=0x7f06f40c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.555+0000 7f06fa800700 1 --2- 172.21.15.89:0/2123695442 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06f40d1fd0 0x7f06f40c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.555+0000 7f06fa800700 1 -- 172.21.15.89:0/2123695442 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06f40d1630 msgr2=0x7f06f41440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.555+0000 7f06fa800700 1 --2- 172.21.15.89:0/2123695442 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06f40d1630 0x7f06f41440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.555+0000 7f06fa800700 1 -- 172.21.15.89:0/2123695442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e4009550 con 0x7f06f415dad0 2024-04-02T09:15:36.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.555+0000 7f06fa800700 1 --2- 172.21.15.89:0/2123695442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06f415dad0 0x7f06f40c7590 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f06f000cb90 tx=0x7f06f000cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:36.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.556+0000 7f06ee000700 1 -- 172.21.15.89:0/2123695442 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f0003c00 con 0x7f06f415dad0 2024-04-02T09:15:36.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.556+0000 7f06ee000700 1 -- 172.21.15.89:0/2123695442 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f0003d60 con 0x7f06f415dad0 2024-04-02T09:15:36.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.556+0000 7f06fc800700 1 -- 172.21.15.89:0/2123695442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f40c9d60 con 0x7f06f415dad0 2024-04-02T09:15:36.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.556+0000 7f06ee000700 1 -- 172.21.15.89:0/2123695442 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f000aa80 con 0x7f06f415dad0 2024-04-02T09:15:36.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.556+0000 7f06fc800700 1 -- 172.21.15.89:0/2123695442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f40cb250 con 0x7f06f415dad0 2024-04-02T09:15:36.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.557+0000 7f06fc800700 1 -- 172.21.15.89:0/2123695442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f40cb0b0 con 0x7f06f415dad0 2024-04-02T09:15:36.557 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:36.558+0000 7fab2be90240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:15:36.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.558+0000 7f06ee000700 1 -- 172.21.15.89:0/2123695442 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 179) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f06f000abe0 con 0x7f06f415dad0 2024-04-02T09:15:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.559+0000 7f06ee000700 1 --2- 172.21.15.89:0/2123695442 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f06d806ae00 0x7f06d806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.559+0000 7f06ee000700 1 -- 172.21.15.89:0/2123695442 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f06f0089630 con 0x7f06f415dad0 2024-04-02T09:15:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.559+0000 7f06f9e00700 1 -- 172.21.15.89:0/2123695442 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f06d806ae00 msgr2=0x7f06d806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.559+0000 7f06f9e00700 1 --2- 172.21.15.89:0/2123695442 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f06d806ae00 0x7f06d806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:36.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.563+0000 7f06ee000700 1 -- 172.21.15.89:0/2123695442 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f06f40cb0b0 con 0x7f06f415dad0 2024-04-02T09:15:36.727 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:36.729+0000 7f4ab9478240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:15:36.748 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:36.749+0000 7fab2be90240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:15:36.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.760+0000 7f06f9e00700 1 -- 172.21.15.89:0/2123695442 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f06d806ae00 msgr2=0x7f06d806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:36.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.760+0000 7f06f9e00700 1 --2- 172.21.15.89:0/2123695442 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f06d806ae00 0x7f06d806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:36.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.810+0000 7f06fc800700 1 -- 172.21.15.89:0/2123695442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f06f40cb0b0 con 0x7f06f415dad0 2024-04-02T09:15:36.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.814+0000 7f06ee000700 1 -- 172.21.15.89:0/2123695442 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v95)=0 v95) v1 ==== 138+0+0 (secure 0 0 0) 0x7f06f40cb0b0 con 0x7f06f415dad0 2024-04-02T09:15:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.816+0000 7f06fc800700 1 -- 172.21.15.89:0/2123695442 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f06d806ae00 msgr2=0x7f06d806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.816+0000 7f06fc800700 1 --2- 172.21.15.89:0/2123695442 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f06d806ae00 0x7f06d806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.816+0000 7f06fc800700 1 -- 172.21.15.89:0/2123695442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06f415dad0 msgr2=0x7f06f40c7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.816+0000 7f06fc800700 1 --2- 172.21.15.89:0/2123695442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06f415dad0 0x7f06f40c7590 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f06f000cb90 tx=0x7f06f000cea0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.817+0000 7f06fc800700 1 -- 172.21.15.89:0/2123695442 shutdown_connections 2024-04-02T09:15:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.817+0000 7f06fc800700 1 --2- 172.21.15.89:0/2123695442 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f06d806ae00 0x7f06d806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.817+0000 7f06fc800700 1 --2- 172.21.15.89:0/2123695442 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06f40d1630 0x7f06f41440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.817+0000 7f06fc800700 1 --2- 172.21.15.89:0/2123695442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06f415dad0 0x7f06f40c7590 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.817+0000 7f06fc800700 1 --2- 172.21.15.89:0/2123695442 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06f40d1fd0 0x7f06f40c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.817+0000 7f06fc800700 1 -- 172.21.15.89:0/2123695442 >> 172.21.15.89:0/2123695442 conn(0x7f06f408aff0 msgr2=0x7f06f415ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.817+0000 7f06fc800700 1 -- 172.21.15.89:0/2123695442 shutdown_connections 2024-04-02T09:15:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:36.817+0000 7f06fc800700 1 -- 172.21.15.89:0/2123695442 wait complete. 2024-04-02T09:15:36.835 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-02T09:15:37.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.033+0000 7f2d66600700 1 Processor -- start 2024-04-02T09:15:37.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.033+0000 7f2d66600700 1 -- start start 2024-04-02T09:15:37.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.034+0000 7f2d66600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6015dad0 0x7f2d600ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:37.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.034+0000 7f2d66600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6015f820 con 0x7f2d600d1630 2024-04-02T09:15:37.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.034+0000 7f2d66600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d600cf420 con 0x7f2d600d21d0 2024-04-02T09:15:37.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.034+0000 7f2d66600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d600cfcc0 con 0x7f2d6015dad0 2024-04-02T09:15:37.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.035+0000 7f2d5ea00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2d600d1630 0x7f2d600d1ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48380/0 (socket says 172.21.15.89:48380) 2024-04-02T09:15:37.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.035+0000 7f2d5ea00700 1 -- 172.21.15.89:0/240044250 learned_addr learned my addr 172.21.15.89:0/240044250 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:37.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.035+0000 7f2d5f400700 1 --2- 172.21.15.89:0/240044250 >> v2:172.21.15.113:3300/0 conn(0x7f2d6015dad0 0x7f2d600ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.035+0000 7f2d5e000700 1 -- 172.21.15.89:0/240044250 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296971776 0 0) 0x7f2d6015f820 con 0x7f2d600d1630 2024-04-02T09:15:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.035+0000 7f2d5f400700 1 -- 172.21.15.89:0/240044250 >> v1:172.21.15.113:6789/0 conn(0x7f2d600d21d0 legacy=0x7f2d6015fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.035+0000 7f2d5f400700 1 -- 172.21.15.89:0/240044250 >> v1:172.21.15.89:6789/0 conn(0x7f2d600d1630 legacy=0x7f2d600d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.035+0000 7f2d5f400700 1 -- 172.21.15.89:0/240044250 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d600cf700 con 0x7f2d6015dad0 2024-04-02T09:15:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.036+0000 7f2d5f400700 1 --2- 172.21.15.89:0/240044250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d6015dad0 0x7f2d600ced80 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f2d50009840 tx=0x7f2d50009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe5ab5c87141fc7f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.036+0000 7f2d5e000700 1 -- 172.21.15.89:0/240044250 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d50011830 con 0x7f2d6015dad0 2024-04-02T09:15:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.036+0000 7f2d5e000700 1 -- 172.21.15.89:0/240044250 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d50011990 con 0x7f2d6015dad0 2024-04-02T09:15:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.036+0000 7f2d5e000700 1 -- 172.21.15.89:0/240044250 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d50011d20 con 0x7f2d6015dad0 2024-04-02T09:15:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.036+0000 7f2d66600700 1 -- 172.21.15.89:0/240044250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d6015dad0 msgr2=0x7f2d600ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.036+0000 7f2d66600700 1 --2- 172.21.15.89:0/240044250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d6015dad0 0x7f2d600ced80 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f2d50009840 tx=0x7f2d50009b50 comp rx=0 tx=0).stop 2024-04-02T09:15:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.036+0000 7f2d66600700 1 -- 172.21.15.89:0/240044250 shutdown_connections 2024-04-02T09:15:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.036+0000 7f2d66600700 1 --2- 172.21.15.89:0/240044250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d6015dad0 0x7f2d600ced80 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.036+0000 7f2d66600700 1 -- 172.21.15.89:0/240044250 >> 172.21.15.89:0/240044250 conn(0x7f2d6008aff0 msgr2=0x7f2d600d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:37.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.037+0000 7f2d66600700 1 -- 172.21.15.89:0/240044250 shutdown_connections 2024-04-02T09:15:37.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.037+0000 7f2d66600700 1 -- 172.21.15.89:0/240044250 wait complete. 2024-04-02T09:15:37.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.038+0000 7f2d66600700 1 Processor -- start 2024-04-02T09:15:37.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.038+0000 7f2d66600700 1 -- start start 2024-04-02T09:15:37.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.039+0000 7f2d66600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d600d1630 0x7f2d601f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:37.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.039+0000 7f2d66600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d600d21d0 0x7f2d600c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:37.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.039+0000 7f2d66600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d6015dad0 0x7f2d6009ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:37.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.039+0000 7f2d66600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d601592c0 con 0x7f2d6015dad0 2024-04-02T09:15:37.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.039+0000 7f2d66600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d600c82d0 con 0x7f2d600d1630 2024-04-02T09:15:37.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.039+0000 7f2d66600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d600c76c0 con 0x7f2d600d21d0 2024-04-02T09:15:37.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.040+0000 7f2d5f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d600d1630 0x7f2d601f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:37.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.040+0000 7f2d5fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d6015dad0 0x7f2d6009ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:37.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.040+0000 7f2d5f400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d600d1630 0x7f2d601f2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:59556/0 (socket says 172.21.15.89:59556) 2024-04-02T09:15:37.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.040+0000 7f2d5fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d6015dad0 0x7f2d6009ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52714/0 (socket says 172.21.15.89:52714) 2024-04-02T09:15:37.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.040+0000 7f2d5f400700 1 -- 172.21.15.89:0/312025091 learned_addr learned my addr 172.21.15.89:0/312025091 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:37.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.040+0000 7f2d5ea00700 1 --2- 172.21.15.89:0/312025091 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d600d21d0 0x7f2d600c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:37.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.040+0000 7f2d5f400700 1 -- 172.21.15.89:0/312025091 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d600d21d0 msgr2=0x7f2d600c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:37.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.040+0000 7f2d5f400700 1 --2- 172.21.15.89:0/312025091 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d600d21d0 0x7f2d600c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:37.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.040+0000 7f2d5f400700 1 -- 172.21.15.89:0/312025091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d6015dad0 msgr2=0x7f2d6009ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:37.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.040+0000 7f2d5f400700 1 --2- 172.21.15.89:0/312025091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d6015dad0 0x7f2d6009ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:37.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.040+0000 7f2d5f400700 1 -- 172.21.15.89:0/312025091 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d500094a0 con 0x7f2d600d1630 2024-04-02T09:15:37.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.040+0000 7f2d5fe00700 1 --2- 172.21.15.89:0/312025091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d6015dad0 0x7f2d6009ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:15:37.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.040+0000 7f2d5f400700 1 --2- 172.21.15.89:0/312025091 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d600d1630 0x7f2d601f2dc0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f2d50014040 tx=0x7f2d50004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:37.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.041+0000 7f2d57400700 1 -- 172.21.15.89:0/312025091 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5001a7e0 con 0x7f2d600d1630 2024-04-02T09:15:37.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.041+0000 7f2d57400700 1 -- 172.21.15.89:0/312025091 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5001a940 con 0x7f2d600d1630 2024-04-02T09:15:37.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.041+0000 7f2d57400700 1 -- 172.21.15.89:0/312025091 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5001acb0 con 0x7f2d600d1630 2024-04-02T09:15:37.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.041+0000 7f2d66600700 1 -- 172.21.15.89:0/312025091 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d600c79a0 con 0x7f2d600d1630 2024-04-02T09:15:37.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.041+0000 7f2d66600700 1 -- 172.21.15.89:0/312025091 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d600c7f60 con 0x7f2d600d1630 2024-04-02T09:15:37.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.042+0000 7f2d66600700 1 -- 172.21.15.89:0/312025091 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d60067720 con 0x7f2d600d1630 2024-04-02T09:15:37.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.043+0000 7f2d57400700 1 -- 172.21.15.89:0/312025091 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 179) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f2d50022070 con 0x7f2d600d1630 2024-04-02T09:15:37.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.044+0000 7f2d57400700 1 --2- 172.21.15.89:0/312025091 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2d4006ae00 0x7f2d4006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:37.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.044+0000 7f2d5ea00700 1 -- 172.21.15.89:0/312025091 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2d4006ae00 msgr2=0x7f2d4006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:37.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.044+0000 7f2d57400700 1 -- 172.21.15.89:0/312025091 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f2d5008a3d0 con 0x7f2d600d1630 2024-04-02T09:15:37.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.044+0000 7f2d5ea00700 1 --2- 172.21.15.89:0/312025091 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2d4006ae00 0x7f2d4006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:37.046 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:37.049+0000 7f4ab9478240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:15:37.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.049+0000 7f2d57400700 1 -- 172.21.15.89:0/312025091 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2d50055380 con 0x7f2d600d1630 2024-04-02T09:15:37.083 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:37.084+0000 7fab2be90240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:15:37.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.245+0000 7f2d5ea00700 1 -- 172.21.15.89:0/312025091 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2d4006ae00 msgr2=0x7f2d4006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:37.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.245+0000 7f2d5ea00700 1 --2- 172.21.15.89:0/312025091 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2d4006ae00 0x7f2d4006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:37.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.256+0000 7f2d66600700 1 -- 172.21.15.89:0/312025091 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f2d601f3300 con 0x7f2d600d1630 2024-04-02T09:15:37.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.261+0000 7f2d57400700 1 -- 172.21.15.89:0/312025091 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v96)=0 v96) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2d500589a0 con 0x7f2d600d1630 2024-04-02T09:15:37.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.264+0000 7f2d66600700 1 -- 172.21.15.89:0/312025091 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2d4006ae00 msgr2=0x7f2d4006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:37.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.264+0000 7f2d66600700 1 --2- 172.21.15.89:0/312025091 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2d4006ae00 0x7f2d4006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:37.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.264+0000 7f2d66600700 1 -- 172.21.15.89:0/312025091 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d600d1630 msgr2=0x7f2d601f2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:37.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.264+0000 7f2d66600700 1 --2- 172.21.15.89:0/312025091 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d600d1630 0x7f2d601f2dc0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f2d50014040 tx=0x7f2d50004750 comp rx=0 tx=0).stop 2024-04-02T09:15:37.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.264+0000 7f2d66600700 1 -- 172.21.15.89:0/312025091 shutdown_connections 2024-04-02T09:15:37.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.264+0000 7f2d66600700 1 --2- 172.21.15.89:0/312025091 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2d4006ae00 0x7f2d4006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:37.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.264+0000 7f2d66600700 1 --2- 172.21.15.89:0/312025091 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d600d1630 0x7f2d601f2dc0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:37.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.264+0000 7f2d66600700 1 --2- 172.21.15.89:0/312025091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d6015dad0 0x7f2d6009ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:37.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.264+0000 7f2d66600700 1 --2- 172.21.15.89:0/312025091 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d600d21d0 0x7f2d600c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:37.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.264+0000 7f2d66600700 1 -- 172.21.15.89:0/312025091 >> 172.21.15.89:0/312025091 conn(0x7f2d6008aff0 msgr2=0x7f2d6015fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:37.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.264+0000 7f2d66600700 1 -- 172.21.15.89:0/312025091 shutdown_connections 2024-04-02T09:15:37.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.264+0000 7f2d66600700 1 -- 172.21.15.89:0/312025091 wait complete. 2024-04-02T09:15:37.282 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv01 2024-04-02T09:15:37.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.483+0000 7f1113600700 1 Processor -- start 2024-04-02T09:15:37.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.484+0000 7f1113600700 1 -- start start 2024-04-02T09:15:37.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.485+0000 7f1113600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f110c0cd020 0x7f110c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:37.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.485+0000 7f1113600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110c0cdae0 con 0x7f110c0b2070 2024-04-02T09:15:37.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.485+0000 7f1113600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110c0cddc0 con 0x7f110c0cbb30 2024-04-02T09:15:37.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.485+0000 7f1113600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f110c0ce0a0 con 0x7f110c0cd020 2024-04-02T09:15:37.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.485+0000 7f1111600700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f110c0b2070 0x7f110c06e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48396/0 (socket says 172.21.15.89:48396) 2024-04-02T09:15:37.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.485+0000 7f1111600700 1 -- 172.21.15.89:0/2373077679 learned_addr learned my addr 172.21.15.89:0/2373077679 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:15:37.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.485+0000 7f110be00700 1 --2- 172.21.15.89:0/2373077679 >> v2:172.21.15.89:3300/0 conn(0x7f110c0cd020 0x7f110c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:37.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.486+0000 7f110be00700 1 -- 172.21.15.89:0/2373077679 >> v1:172.21.15.113:6789/0 conn(0x7f110c0cbb30 legacy=0x7f110c0cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:37.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.486+0000 7f110be00700 1 -- 172.21.15.89:0/2373077679 >> v1:172.21.15.89:6789/0 conn(0x7f110c0b2070 legacy=0x7f110c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:37.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.486+0000 7f110be00700 1 -- 172.21.15.89:0/2373077679 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110c06dc20 con 0x7f110c0cd020 2024-04-02T09:15:37.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.486+0000 7f110b400700 1 -- 172.21.15.89:0/2373077679 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697085006 0 0) 0x7f110c0cddc0 con 0x7f110c0cbb30 2024-04-02T09:15:37.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.486+0000 7f110be00700 1 --2- 172.21.15.89:0/2373077679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f110c0cd020 0x7f110c0cd480 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009840 tx=0x7f10f8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee09c999d13ea18 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:37.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.486+0000 7f110b400700 1 -- 172.21.15.89:0/2373077679 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8011830 con 0x7f110c0cd020 2024-04-02T09:15:37.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.486+0000 7f110b400700 1 -- 172.21.15.89:0/2373077679 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f8011990 con 0x7f110c0cd020 2024-04-02T09:15:37.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.486+0000 7f110b400700 1 -- 172.21.15.89:0/2373077679 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8011d20 con 0x7f110c0cd020 2024-04-02T09:15:37.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.487+0000 7f1113600700 1 -- 172.21.15.89:0/2373077679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f110c0cd020 msgr2=0x7f110c0cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:37.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.487+0000 7f1113600700 1 --2- 172.21.15.89:0/2373077679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f110c0cd020 0x7f110c0cd480 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009840 tx=0x7f10f8009b50 comp rx=0 tx=0).stop 2024-04-02T09:15:37.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.487+0000 7f1113600700 1 -- 172.21.15.89:0/2373077679 shutdown_connections 2024-04-02T09:15:37.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.487+0000 7f1113600700 1 --2- 172.21.15.89:0/2373077679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f110c0cd020 0x7f110c0cd480 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:37.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.487+0000 7f1113600700 1 -- 172.21.15.89:0/2373077679 >> 172.21.15.89:0/2373077679 conn(0x7f110c0b0d70 msgr2=0x7f110c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:37.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.487+0000 7f1113600700 1 -- 172.21.15.89:0/2373077679 shutdown_connections 2024-04-02T09:15:37.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.487+0000 7f1113600700 1 -- 172.21.15.89:0/2373077679 wait complete. 2024-04-02T09:15:37.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.490+0000 7f1113600700 1 Processor -- start 2024-04-02T09:15:37.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.490+0000 7f1113600700 1 -- start start 2024-04-02T09:15:37.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.491+0000 7f1113600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f110c0b2070 0x7f110c154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:37.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f1113600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f110c0cbb30 0x7f110c1548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:37.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f1113600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f110c0cd020 0x7f110c15e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:37.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f1113600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110c154e30 con 0x7f110c0cd020 2024-04-02T09:15:37.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f1113600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110c155820 con 0x7f110c0cbb30 2024-04-02T09:15:37.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f1113600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110c15e8e0 con 0x7f110c0b2070 2024-04-02T09:15:37.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f1111600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f110c0cd020 0x7f110c15e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:37.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f1111600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f110c0cd020 0x7f110c15e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52722/0 (socket says 172.21.15.89:52722) 2024-04-02T09:15:37.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f1111600700 1 -- 172.21.15.89:0/1933931015 learned_addr learned my addr 172.21.15.89:0/1933931015 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:37.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f110be00700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f110c0cbb30 0x7f110c1548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:37.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f1110c00700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f110c0b2070 0x7f110c154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:37.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f1111600700 1 -- 172.21.15.89:0/1933931015 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f110c0b2070 msgr2=0x7f110c154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:37.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f1111600700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f110c0b2070 0x7f110c154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:37.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f1111600700 1 -- 172.21.15.89:0/1933931015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f110c0cbb30 msgr2=0x7f110c1548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:37.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f1111600700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f110c0cbb30 0x7f110c1548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:37.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f1111600700 1 -- 172.21.15.89:0/1933931015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f80094a0 con 0x7f110c0cd020 2024-04-02T09:15:37.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.492+0000 7f110be00700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f110c0cbb30 0x7f110c1548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:15:37.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.493+0000 7f1111600700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f110c0cd020 0x7f110c15e3a0 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7f110400cc20 tx=0x7f110400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:37.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.493+0000 7f1108c00700 1 -- 172.21.15.89:0/1933931015 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110400a9c0 con 0x7f110c0cd020 2024-04-02T09:15:37.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.493+0000 7f1108c00700 1 -- 172.21.15.89:0/1933931015 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110400ab20 con 0x7f110c0cd020 2024-04-02T09:15:37.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.493+0000 7f1113600700 1 -- 172.21.15.89:0/1933931015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f110c1e4bb0 con 0x7f110c0cd020 2024-04-02T09:15:37.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.493+0000 7f1108c00700 1 -- 172.21.15.89:0/1933931015 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110400ae90 con 0x7f110c0cd020 2024-04-02T09:15:37.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.493+0000 7f1113600700 1 -- 172.21.15.89:0/1933931015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f110c1e5170 con 0x7f110c0cd020 2024-04-02T09:15:37.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.495+0000 7f1113600700 1 -- 172.21.15.89:0/1933931015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f110c067720 con 0x7f110c0cd020 2024-04-02T09:15:37.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.495+0000 7f1108c00700 1 -- 172.21.15.89:0/1933931015 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 179) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f110401b3b0 con 0x7f110c0cd020 2024-04-02T09:15:37.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.496+0000 7f1108c00700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 0x7f10ec06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:37.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.496+0000 7f1108c00700 1 -- 172.21.15.89:0/1933931015 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f11040891e0 con 0x7f110c0cd020 2024-04-02T09:15:37.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.496+0000 7f1110c00700 1 -- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 msgr2=0x7f10ec06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:37.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.496+0000 7f1110c00700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 0x7f10ec06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:15:37.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.500+0000 7f1108c00700 1 -- 172.21.15.89:0/1933931015 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1104054100 con 0x7f110c0cd020 2024-04-02T09:15:37.573 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:37.575+0000 7f4ab9478240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:15:37.685 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:37.685+0000 7fab2be90240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:15:37.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.697+0000 7f1110c00700 1 -- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 msgr2=0x7f10ec06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:37.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.697+0000 7f1110c00700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 0x7f10ec06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:15:37.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:37.803+0000 7f1113600700 1 -- 172.21.15.89:0/1933931015 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) v1 -- 0x7f110c15bdd0 con 0x7f10ec06ae00 2024-04-02T09:15:37.827 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:37.829+0000 7f4ab9478240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:15:37.945 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:37.945+0000 7fab2be90240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:15:38.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:38.098+0000 7f1110c00700 1 -- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 msgr2=0x7f10ec06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:38.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:38.098+0000 7f1110c00700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 0x7f10ec06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T09:15:38.281 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:38.282+0000 7f4ab9478240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:15:38.402 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:38.403+0000 7fab2be90240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:15:38.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:38.899+0000 7f1110c00700 1 -- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 msgr2=0x7f10ec06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:38.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:38.899+0000 7f1110c00700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 0x7f10ec06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T09:15:38.986 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:38.987+0000 7f4ab9478240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:15:39.077 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:39.078+0000 7f4ab9478240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:15:39.108 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:39.109+0000 7fab2be90240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:15:39.200 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:39.201+0000 7fab2be90240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:15:39.236 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:39.237+0000 7f4ab9478240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:15:39.361 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:39.362+0000 7fab2be90240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:15:39.450 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:39.452+0000 7f4ab9478240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:15:39.542 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:39.543+0000 7f4ab9478240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:15:39.575 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:39.576+0000 7fab2be90240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:15:39.639 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:39.641+0000 7f4ab9478240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:15:39.668 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:39.669+0000 7fab2be90240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:15:39.737 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:39.738+0000 7f4ab9478240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:15:39.768 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:39.769+0000 7fab2be90240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:15:39.835 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:39.836+0000 7f4ab9478240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:15:39.868 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:39.869+0000 7fab2be90240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:15:39.965 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:39.966+0000 7fab2be90240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:15:40.046 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:40.047+0000 7f4ab9478240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:15:40.177 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:40.177+0000 7fab2be90240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:15:40.301 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:40.302+0000 7f4ab9478240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:15:40.433 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:40.434+0000 7fab2be90240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:15:40.492 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:40.493+0000 7f4ab9478240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:15:40.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:40.500+0000 7f1110c00700 1 -- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 msgr2=0x7f10ec06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:15:40.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:40.500+0000 7f1110c00700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 0x7f10ec06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T09:15:40.584 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:40.585+0000 7f4ab9478240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:15:40.625 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:40.626+0000 7fab2be90240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:15:40.719 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:40.719+0000 7fab2be90240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:15:41.312 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:41.313+0000 7f4ab9478240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:15:41.448 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:41.449+0000 7fab2be90240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:15:41.524 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:41.525+0000 7f4ab9478240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:15:41.663 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:41.664+0000 7fab2be90240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:15:41.736 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:41.737+0000 7f4ab9478240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:15:41.872 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:41.873+0000 7fab2be90240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:15:42.243 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:42.244+0000 7f4ab9478240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:15:42.353 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:42.354+0000 7fab2be90240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:15:42.453 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:42.454+0000 7f4ab9478240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:15:42.565 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:42.565+0000 7fab2be90240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:15:42.896 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:42.897+0000 7f4ab9478240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:15:43.007 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:15:43.009+0000 7f4ab9478240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:15:43.009 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:43.010+0000 7fab2be90240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:15:43.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:43.069+0000 7f1108c00700 1 -- 172.21.15.89:0/1933931015 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 180) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f11040544d0 con 0x7f110c0cd020 2024-04-02T09:15:43.124 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:43.125+0000 7fab2be90240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:15:43.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:43.135+0000 7f1108c00700 1 -- 172.21.15.89:0/1933931015 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 181) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f11040874a0 con 0x7f110c0cd020 2024-04-02T09:15:43.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:43.135+0000 7f1108c00700 1 -- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 msgr2=0x7f10ec06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:15:43.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:43.135+0000 7f1108c00700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 0x7f10ec06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:43.305 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:43.306+0000 7faa98200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:43.305 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:43.306+0000 7faa94600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:43.306 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:43.306+0000 7faa94600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:43.306 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:43.306+0000 7faa94600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:43.306 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:43.306+0000 7faa94600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:43.306 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:43.306+0000 7faa94600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:43.306 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:43.306+0000 7faa98200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:43.306 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:43.306+0000 7faa98200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:43.306 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:43.306+0000 7faa98200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:43.306 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:43.306+0000 7faa98200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:44.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.139+0000 7f1108c00700 1 -- 172.21.15.89:0/1933931015 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mgrmap(e 182) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f1104054da0 con 0x7f110c0cd020 2024-04-02T09:15:44.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.140+0000 7f1108c00700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 0x7f10ec06d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:44.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.140+0000 7f1108c00700 1 -- 172.21.15.89:0/1933931015 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) v1 -- 0x7f10ec0283c0 con 0x7f10ec06ae00 2024-04-02T09:15:44.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.141+0000 7f1110c00700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 0x7f10ec06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:44.143 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:15:44.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.145+0000 7f1110c00700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 0x7f10ec06d240 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1100007fd0 tx=0x7f1100009040 comp rx=0 tx=0).ready entity=mgr.13473 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:44.149 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:15:44.158 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:44.154+0000 7faa90a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:44.158 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:44.154+0000 7faa90a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:44.158 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:44.154+0000 7faa90a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:44.158 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:44.154+0000 7faa90a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:44.158 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:15:44.154+0000 7faa90a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:15:44.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.190+0000 7f1108c00700 1 -- 172.21.15.89:0/1933931015 <== mgr.13473 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f10ec0283c0 con 0x7f10ec06ae00 2024-04-02T09:15:44.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.192+0000 7f1113600700 1 -- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 msgr2=0x7f10ec06d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:44.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.192+0000 7f1113600700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 0x7f10ec06d240 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1100007fd0 tx=0x7f1100009040 comp rx=0 tx=0).stop 2024-04-02T09:15:44.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.192+0000 7f1113600700 1 -- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f110c0cd020 msgr2=0x7f110c15e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:44.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.192+0000 7f1113600700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f110c0cd020 0x7f110c15e3a0 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7f110400cc20 tx=0x7f110400cf30 comp rx=0 tx=0).stop 2024-04-02T09:15:44.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.192+0000 7f1113600700 1 -- 172.21.15.89:0/1933931015 shutdown_connections 2024-04-02T09:15:44.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.192+0000 7f1113600700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f110c0cd020 0x7f110c15e3a0 unknown :-1 s=CLOSED pgs=3014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:44.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.192+0000 7f1113600700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f110c0b2070 0x7f110c154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:44.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.192+0000 7f1113600700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f10ec06ae00 0x7f10ec06d240 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:44.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.192+0000 7f1113600700 1 --2- 172.21.15.89:0/1933931015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f110c0cbb30 0x7f110c1548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:44.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.192+0000 7f1113600700 1 -- 172.21.15.89:0/1933931015 >> 172.21.15.89:0/1933931015 conn(0x7f110c0b0d70 msgr2=0x7f110c0af2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:44.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.192+0000 7f1113600700 1 -- 172.21.15.89:0/1933931015 shutdown_connections 2024-04-02T09:15:44.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.192+0000 7f1113600700 1 -- 172.21.15.89:0/1933931015 wait complete. 2024-04-02T09:15:44.212 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --subvol sv01 . 1m 2024-04-02T09:15:44.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.452+0000 7f9b94200700 1 Processor -- start 2024-04-02T09:15:44.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.452+0000 7f9b94200700 1 -- start start 2024-04-02T09:15:44.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.453+0000 7f9b94200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9b8c15d930 0x7f9b8c0d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:44.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.453+0000 7f9b94200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9b8c0d22a0 0x7f9b8c0ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:44.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.453+0000 7f9b94200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b8c15dd10 con 0x7f9b8c162030 2024-04-02T09:15:44.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.453+0000 7f9b94200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b8c15ec40 con 0x7f9b8c0d22a0 2024-04-02T09:15:44.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.453+0000 7f9b94200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b8c15dfc0 con 0x7f9b8c15d930 2024-04-02T09:15:44.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.453+0000 7f9b92200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9b8c0d22a0 0x7f9b8c0ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:44.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.453+0000 7f9b92200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9b8c0d22a0 0x7f9b8c0ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34032/0 (socket says 172.21.15.89:34032) 2024-04-02T09:15:44.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.453+0000 7f9b92200700 1 -- 172.21.15.89:0/3457465900 learned_addr learned my addr 172.21.15.89:0/3457465900 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:44.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.453+0000 7f9b91800700 1 --2- 172.21.15.89:0/3457465900 >> v2:172.21.15.113:3300/0 conn(0x7f9b8c15d930 0x7f9b8c0d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:44.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.454+0000 7f9b92200700 1 -- 172.21.15.89:0/3457465900 >> v2:172.21.15.113:3300/0 conn(0x7f9b8c15d930 msgr2=0x7f9b8c0d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:15:44.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.454+0000 7f9b92200700 1 --2- 172.21.15.89:0/3457465900 >> v2:172.21.15.113:3300/0 conn(0x7f9b8c15d930 0x7f9b8c0d3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:44.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.454+0000 7f9b92200700 1 -- 172.21.15.89:0/3457465900 >> v1:172.21.15.113:6789/0 conn(0x7f9b8c162030 legacy=0x7f9b8c0d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:44.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.454+0000 7f9b92200700 1 -- 172.21.15.89:0/3457465900 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c15e2a0 con 0x7f9b8c0d22a0 2024-04-02T09:15:44.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.454+0000 7f9b92200700 1 --2- 172.21.15.89:0/3457465900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b8c0d22a0 0x7f9b8c0ce300 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0a1430 tx=0x7f9b8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e3889d64af71a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:44.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.454+0000 7f9b8be00700 1 -- 172.21.15.89:0/3457465900 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8400e070 con 0x7f9b8c0d22a0 2024-04-02T09:15:44.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.454+0000 7f9b8be00700 1 -- 172.21.15.89:0/3457465900 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8400b910 con 0x7f9b8c0d22a0 2024-04-02T09:15:44.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.454+0000 7f9b8be00700 1 -- 172.21.15.89:0/3457465900 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8400bcc0 con 0x7f9b8c0d22a0 2024-04-02T09:15:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.455+0000 7f9b94200700 1 -- 172.21.15.89:0/3457465900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b8c0d22a0 msgr2=0x7f9b8c0ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.455+0000 7f9b94200700 1 --2- 172.21.15.89:0/3457465900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b8c0d22a0 0x7f9b8c0ce300 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0a1430 tx=0x7f9b8400b330 comp rx=0 tx=0).stop 2024-04-02T09:15:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.455+0000 7f9b94200700 1 -- 172.21.15.89:0/3457465900 shutdown_connections 2024-04-02T09:15:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.455+0000 7f9b94200700 1 --2- 172.21.15.89:0/3457465900 >> v2:172.21.15.113:3300/0 conn(0x7f9b8c15d930 0x7f9b8c0d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.455+0000 7f9b94200700 1 --2- 172.21.15.89:0/3457465900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b8c0d22a0 0x7f9b8c0ce300 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.455+0000 7f9b94200700 1 -- 172.21.15.89:0/3457465900 >> 172.21.15.89:0/3457465900 conn(0x7f9b8c08aff0 msgr2=0x7f9b8c0d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.455+0000 7f9b94200700 1 -- 172.21.15.89:0/3457465900 shutdown_connections 2024-04-02T09:15:44.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.455+0000 7f9b94200700 1 -- 172.21.15.89:0/3457465900 wait complete. 2024-04-02T09:15:44.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.457+0000 7f9b94200700 1 Processor -- start 2024-04-02T09:15:44.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.457+0000 7f9b94200700 1 -- start start 2024-04-02T09:15:44.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b94200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b8c0d22a0 0x7f9b8c0b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:44.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b94200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b8c15d930 0x7f9b8c0b6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b94200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b8c162030 0x7f9b8c14e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b94200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c0b1c00 con 0x7f9b8c15d930 2024-04-02T09:15:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b94200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c06c750 con 0x7f9b8c0d22a0 2024-04-02T09:15:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b94200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c0b0ff0 con 0x7f9b8c162030 2024-04-02T09:15:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b90e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b8c15d930 0x7f9b8c0b6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b90e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b8c15d930 0x7f9b8c0b6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34038/0 (socket says 172.21.15.89:34038) 2024-04-02T09:15:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b90e00700 1 -- 172.21.15.89:0/2988175425 learned_addr learned my addr 172.21.15.89:0/2988175425 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:15:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b91800700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b8c0d22a0 0x7f9b8c0b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b92200700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b8c162030 0x7f9b8c14e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b90e00700 1 -- 172.21.15.89:0/2988175425 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b8c162030 msgr2=0x7f9b8c14e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b90e00700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b8c162030 0x7f9b8c14e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b90e00700 1 -- 172.21.15.89:0/2988175425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b8c0d22a0 msgr2=0x7f9b8c0b8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b90e00700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b8c0d22a0 0x7f9b8c0b8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b90e00700 1 -- 172.21.15.89:0/2988175425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b84009b30 con 0x7f9b8c15d930 2024-04-02T09:15:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b92200700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b8c162030 0x7f9b8c14e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.458+0000 7f9b91800700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b8c0d22a0 0x7f9b8c0b8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:15:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.459+0000 7f9b90e00700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b8c15d930 0x7f9b8c0b6da0 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800c450 tx=0x7f9b7800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.459+0000 7f9b89600700 1 -- 172.21.15.89:0/2988175425 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b780052d0 con 0x7f9b8c15d930 2024-04-02T09:15:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.459+0000 7f9b89600700 1 -- 172.21.15.89:0/2988175425 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b78005430 con 0x7f9b8c15d930 2024-04-02T09:15:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.459+0000 7f9b89600700 1 -- 172.21.15.89:0/2988175425 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b780038c0 con 0x7f9b8c15d930 2024-04-02T09:15:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.459+0000 7f9b94200700 1 -- 172.21.15.89:0/2988175425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8c0b12d0 con 0x7f9b8c15d930 2024-04-02T09:15:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.459+0000 7f9b94200700 1 -- 172.21.15.89:0/2988175425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b8c0b1890 con 0x7f9b8c15d930 2024-04-02T09:15:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.460+0000 7f9b94200700 1 -- 172.21.15.89:0/2988175425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b8c067720 con 0x7f9b8c15d930 2024-04-02T09:15:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.461+0000 7f9b89600700 1 -- 172.21.15.89:0/2988175425 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 182) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f9b78003a20 con 0x7f9b8c15d930 2024-04-02T09:15:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.462+0000 7f9b89600700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9b6c06ab30 0x7f9b6c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:15:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.462+0000 7f9b89600700 1 -- 172.21.15.89:0/2988175425 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f9b78088690 con 0x7f9b8c15d930 2024-04-02T09:15:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.462+0000 7f9b91800700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9b6c06ab30 0x7f9b6c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:15:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.463+0000 7f9b91800700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9b6c06ab30 0x7f9b6c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009c30 tx=0x7f9b80008040 comp rx=0 tx=0).ready entity=mgr.13473 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:15:44.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.467+0000 7f9b89600700 1 -- 172.21.15.89:0/2988175425 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9b780537c0 con 0x7f9b8c15d930 2024-04-02T09:15:44.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.693+0000 7f9b94200700 1 -- 172.21.15.89:0/2988175425 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "subvol": "sv01", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b8c0d4410 con 0x7f9b6c06ab30 2024-04-02T09:15:44.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.785+0000 7f9b89600700 1 -- 172.21.15.89:0/2988175425 <== mgr.13473 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+85 (secure 0 0 0) 0x7f9b80009960 con 0x7f9b6c06ab30 2024-04-02T09:15:44.785 INFO:teuthology.orchestra.run.smithi089.stdout:Schedule set for path /volumes/_nogroup/sv01/84da8938-de5d-4ee3-8c84-009fb5854a78/.. 2024-04-02T09:15:44.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.788+0000 7f9b94200700 1 -- 172.21.15.89:0/2988175425 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9b6c06ab30 msgr2=0x7f9b6c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:44.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.788+0000 7f9b94200700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9b6c06ab30 0x7f9b6c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009c30 tx=0x7f9b80008040 comp rx=0 tx=0).stop 2024-04-02T09:15:44.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.788+0000 7f9b94200700 1 -- 172.21.15.89:0/2988175425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b8c15d930 msgr2=0x7f9b8c0b6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:15:44.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.788+0000 7f9b94200700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b8c15d930 0x7f9b8c0b6da0 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800c450 tx=0x7f9b7800c810 comp rx=0 tx=0).stop 2024-04-02T09:15:44.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.788+0000 7f9b94200700 1 -- 172.21.15.89:0/2988175425 shutdown_connections 2024-04-02T09:15:44.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.788+0000 7f9b94200700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9b6c06ab30 0x7f9b6c06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:44.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.788+0000 7f9b94200700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b8c0d22a0 0x7f9b8c0b8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:44.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.788+0000 7f9b94200700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9b8c162030 0x7f9b8c14e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:44.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.788+0000 7f9b94200700 1 --2- 172.21.15.89:0/2988175425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b8c15d930 0x7f9b8c0b6da0 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:15:44.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.788+0000 7f9b94200700 1 -- 172.21.15.89:0/2988175425 >> 172.21.15.89:0/2988175425 conn(0x7f9b8c08aff0 msgr2=0x7f9b8c0cebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:15:44.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.788+0000 7f9b94200700 1 -- 172.21.15.89:0/2988175425 shutdown_connections 2024-04-02T09:15:44.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:15:44.788+0000 7f9b94200700 1 -- 172.21.15.89:0/2988175425 wait complete. 2024-04-02T09:15:44.805 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /volumes/_nogroup/sv01/84da8938-de5d-4ee3-8c84-009fb5854a78/.. 2024-04-02T09:16:00.799 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:16:00.801+0000 7faa7de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:16:00.800 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:16:00.801+0000 7faa7de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:16:00.800 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:16:00.801+0000 7faa7de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:16:00.800 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:16:00.801+0000 7faa7de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:16:00.800 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:16:00.801+0000 7faa7de00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:16:14.573 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:16:14.578 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:16:44.896 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:16:44.902 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:16:59.807 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-02T09:17:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.012+0000 7fddf4800700 1 Processor -- start 2024-04-02T09:17:00.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.012+0000 7fddf4800700 1 -- start start 2024-04-02T09:17:00.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.013+0000 7fddf4800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fddec06da80 0x7fddec15b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:00.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.013+0000 7fddf4800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddec06eae0 con 0x7fddec0b4a50 2024-04-02T09:17:00.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.013+0000 7fddf4800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddec06f660 con 0x7fddec15af30 2024-04-02T09:17:00.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.013+0000 7fddf4800700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddec06faf0 con 0x7fddec06da80 2024-04-02T09:17:00.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.014+0000 7fddf2800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fddec0b4a50 0x7fddec0b2c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50960/0 (socket says 172.21.15.89:50960) 2024-04-02T09:17:00.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.014+0000 7fddf2800700 1 -- 172.21.15.89:0/1669050859 learned_addr learned my addr 172.21.15.89:0/1669050859 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:00.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.014+0000 7fddf1400700 1 --2- 172.21.15.89:0/1669050859 >> v2:172.21.15.113:3300/0 conn(0x7fddec06da80 0x7fddec15b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:00.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.014+0000 7fddf0a00700 1 -- 172.21.15.89:0/1669050859 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469648046 0 0) 0x7fddec06eae0 con 0x7fddec0b4a50 2024-04-02T09:17:00.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.014+0000 7fddf0a00700 1 -- 172.21.15.89:0/1669050859 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd40035b0 con 0x7fddec0b4a50 2024-04-02T09:17:00.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.014+0000 7fddf0a00700 1 -- 172.21.15.89:0/1669050859 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948626817 0 0) 0x7fddec06f660 con 0x7fddec15af30 2024-04-02T09:17:00.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.014+0000 7fddf0a00700 1 -- 172.21.15.89:0/1669050859 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddec06eae0 con 0x7fddec15af30 2024-04-02T09:17:00.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.014+0000 7fddf0a00700 1 -- 172.21.15.89:0/1669050859 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3936190507 0 0) 0x7fddd40035b0 con 0x7fddec0b4a50 2024-04-02T09:17:00.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.014+0000 7fddf1400700 1 -- 172.21.15.89:0/1669050859 >> v1:172.21.15.113:6789/0 conn(0x7fddec15af30 legacy=0x7fddec06d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:00.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.014+0000 7fddf1400700 1 -- 172.21.15.89:0/1669050859 >> v1:172.21.15.89:6789/0 conn(0x7fddec0b4a50 legacy=0x7fddec0b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:00.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.014+0000 7fddf1400700 1 -- 172.21.15.89:0/1669050859 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec06e150 con 0x7fddec06da80 2024-04-02T09:17:00.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.015+0000 7fddf1400700 1 --2- 172.21.15.89:0/1669050859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddec06da80 0x7fddec15b310 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fdddc009840 tx=0x7fdddc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8cfba095841c012 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:00.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.015+0000 7fddf0a00700 1 -- 172.21.15.89:0/1669050859 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde80019f0 con 0x7fddec06da80 2024-04-02T09:17:00.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.015+0000 7fddf0a00700 1 -- 172.21.15.89:0/1669050859 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdddc004030 con 0x7fddec06da80 2024-04-02T09:17:00.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.015+0000 7fddf0a00700 1 -- 172.21.15.89:0/1669050859 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc0043c0 con 0x7fddec06da80 2024-04-02T09:17:00.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.015+0000 7fddf4800700 1 -- 172.21.15.89:0/1669050859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddec06da80 msgr2=0x7fddec15b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:00.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.015+0000 7fddf4800700 1 --2- 172.21.15.89:0/1669050859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddec06da80 0x7fddec15b310 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fdddc009840 tx=0x7fdddc009b50 comp rx=0 tx=0).stop 2024-04-02T09:17:00.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.016+0000 7fddf4800700 1 -- 172.21.15.89:0/1669050859 shutdown_connections 2024-04-02T09:17:00.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.016+0000 7fddf4800700 1 --2- 172.21.15.89:0/1669050859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddec06da80 0x7fddec15b310 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:00.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.016+0000 7fddf4800700 1 -- 172.21.15.89:0/1669050859 >> 172.21.15.89:0/1669050859 conn(0x7fddec071fb0 msgr2=0x7fddec06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:00.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.016+0000 7fddf4800700 1 -- 172.21.15.89:0/1669050859 shutdown_connections 2024-04-02T09:17:00.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.016+0000 7fddf4800700 1 -- 172.21.15.89:0/1669050859 wait complete. 2024-04-02T09:17:00.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.017+0000 7fddf4800700 1 Processor -- start 2024-04-02T09:17:00.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.017+0000 7fddf4800700 1 -- start start 2024-04-02T09:17:00.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.018+0000 7fddf4800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddec06da80 0x7fddec14b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:00.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.018+0000 7fddf4800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddec0b4a50 0x7fddec0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:00.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.018+0000 7fddf4800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec15af30 0x7fddec1f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:00.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.018+0000 7fddf4800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddec157a00 con 0x7fddec06da80 2024-04-02T09:17:00.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.018+0000 7fddf4800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddec1f3bf0 con 0x7fddec0b4a50 2024-04-02T09:17:00.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.018+0000 7fddf4800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddec156df0 con 0x7fddec15af30 2024-04-02T09:17:00.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.019+0000 7fddf1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddec06da80 0x7fddec14b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:00.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.019+0000 7fddf1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddec06da80 0x7fddec14b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56048/0 (socket says 172.21.15.89:56048) 2024-04-02T09:17:00.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.019+0000 7fddf1e00700 1 -- 172.21.15.89:0/2736203569 learned_addr learned my addr 172.21.15.89:0/2736203569 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:00.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.019+0000 7fddf1e00700 1 -- 172.21.15.89:0/2736203569 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec15af30 msgr2=0x7fddec1f36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:00.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.019+0000 7fddf1e00700 1 --2- 172.21.15.89:0/2736203569 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec15af30 0x7fddec1f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:00.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.019+0000 7fddf1e00700 1 -- 172.21.15.89:0/2736203569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddec0b4a50 msgr2=0x7fddec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:00.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.019+0000 7fddf1e00700 1 --2- 172.21.15.89:0/2736203569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddec0b4a50 0x7fddec0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:00.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.019+0000 7fddf1e00700 1 -- 172.21.15.89:0/2736203569 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc0094a0 con 0x7fddec06da80 2024-04-02T09:17:00.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.019+0000 7fddf1e00700 1 --2- 172.21.15.89:0/2736203569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddec06da80 0x7fddec14b5f0 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7fdde000c500 tx=0x7fdde000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:00.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.019+0000 7fdde6000700 1 -- 172.21.15.89:0/2736203569 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde00052d0 con 0x7fddec06da80 2024-04-02T09:17:00.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.019+0000 7fdde6000700 1 -- 172.21.15.89:0/2736203569 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde0005430 con 0x7fddec06da80 2024-04-02T09:17:00.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.019+0000 7fdde6000700 1 -- 172.21.15.89:0/2736203569 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde00038c0 con 0x7fddec06da80 2024-04-02T09:17:00.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.019+0000 7fddf4800700 1 -- 172.21.15.89:0/2736203569 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddec1570d0 con 0x7fddec06da80 2024-04-02T09:17:00.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.020+0000 7fddf4800700 1 -- 172.21.15.89:0/2736203569 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddec157690 con 0x7fddec06da80 2024-04-02T09:17:00.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.021+0000 7fddf4800700 1 -- 172.21.15.89:0/2736203569 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddec157820 con 0x7fddec06da80 2024-04-02T09:17:00.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.021+0000 7fdde6000700 1 -- 172.21.15.89:0/2736203569 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 185) v1 ==== 89421+0+0 (secure 0 0 0) 0x7fdde0003a20 con 0x7fddec06da80 2024-04-02T09:17:00.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.022+0000 7fdde6000700 1 --2- 172.21.15.89:0/2736203569 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fddd006afb0 0x7fddd006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:00.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.022+0000 7fdde6000700 1 -- 172.21.15.89:0/2736203569 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fdde0089190 con 0x7fddec06da80 2024-04-02T09:17:00.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.023+0000 7fddf1400700 1 --2- 172.21.15.89:0/2736203569 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fddd006afb0 0x7fddd006d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:00.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.023+0000 7fddf1400700 1 --2- 172.21.15.89:0/2736203569 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fddd006afb0 0x7fddd006d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fddec0b4100 tx=0x7fdddc010000 comp rx=0 tx=0).ready entity=mgr.13473 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:00.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.027+0000 7fdde6000700 1 -- 172.21.15.89:0/2736203569 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdde0053be0 con 0x7fddec06da80 2024-04-02T09:17:00.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.287+0000 7fddf4800700 1 -- 172.21.15.89:0/2736203569 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fddec15e670 con 0x7fddec06da80 2024-04-02T09:17:00.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.428+0000 7fdde6000700 1 -- 172.21.15.89:0/2736203569 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fdde0053fb0 con 0x7fddec06da80 2024-04-02T09:17:00.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.430+0000 7fdde6000700 1 -- 172.21.15.89:0/2736203569 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v186) v1 ==== 93+0+0 (secure 0 0 0) 0x7fdde0071f00 con 0x7fddec06da80 2024-04-02T09:17:00.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.432+0000 7fddf4800700 1 -- 172.21.15.89:0/2736203569 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fddd006afb0 msgr2=0x7fddd006d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:00.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.432+0000 7fddf4800700 1 --2- 172.21.15.89:0/2736203569 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fddd006afb0 0x7fddd006d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fddec0b4100 tx=0x7fdddc010000 comp rx=0 tx=0).stop 2024-04-02T09:17:00.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.432+0000 7fddf4800700 1 -- 172.21.15.89:0/2736203569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddec06da80 msgr2=0x7fddec14b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:00.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.432+0000 7fddf4800700 1 --2- 172.21.15.89:0/2736203569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddec06da80 0x7fddec14b5f0 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7fdde000c500 tx=0x7fdde000c8c0 comp rx=0 tx=0).stop 2024-04-02T09:17:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.432+0000 7fddf4800700 1 -- 172.21.15.89:0/2736203569 shutdown_connections 2024-04-02T09:17:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.432+0000 7fddf4800700 1 --2- 172.21.15.89:0/2736203569 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec15af30 0x7fddec1f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.432+0000 7fddf4800700 1 --2- 172.21.15.89:0/2736203569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddec06da80 0x7fddec14b5f0 unknown :-1 s=CLOSED pgs=3032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.432+0000 7fddf4800700 1 --2- 172.21.15.89:0/2736203569 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fddd006afb0 0x7fddd006d470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.432+0000 7fddf4800700 1 --2- 172.21.15.89:0/2736203569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddec0b4a50 0x7fddec0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.432+0000 7fddf4800700 1 -- 172.21.15.89:0/2736203569 >> 172.21.15.89:0/2736203569 conn(0x7fddec071fb0 msgr2=0x7fddec06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.432+0000 7fddf4800700 1 -- 172.21.15.89:0/2736203569 shutdown_connections 2024-04-02T09:17:00.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:00.432+0000 7fddf4800700 1 -- 172.21.15.89:0/2736203569 wait complete. 2024-04-02T09:17:00.451 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-02T09:17:00.451 DEBUG:teuthology.orchestra.run.smithi089:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:17:00.544 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-02T09:17:00.632 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-02T09:17:00.633 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:17:00.633 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:17:00.652 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:00.654+0000 7f0061d0f240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:17:00.658 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:00.660+0000 7fdd2c137240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T09:17:00.677 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:17:00.677 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:17:00.763 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:17:00.838 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:00.839+0000 7f0061d0f240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:17:00.851 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:00.853+0000 7fdd2c137240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T09:17:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.023+0000 7f8505800700 1 Processor -- start 2024-04-02T09:17:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.023+0000 7f8505800700 1 -- start start 2024-04-02T09:17:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.024+0000 7f8505800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85000acda0 0x7f850015e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.024+0000 7f8505800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85000ad180 0x7f85000b4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.024+0000 7f8505800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850015eb80 con 0x7f85000b4690 2024-04-02T09:17:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.024+0000 7f8505800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850006eef0 con 0x7f85000ad180 2024-04-02T09:17:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.024+0000 7f8505800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85000ae7b0 con 0x7f85000acda0 2024-04-02T09:17:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.024+0000 7f84ff400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f85000b4690 0x7f85000b4b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50974/0 (socket says 172.21.15.89:50974) 2024-04-02T09:17:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.025+0000 7f84ff400700 1 -- 172.21.15.89:0/1846082283 learned_addr learned my addr 172.21.15.89:0/1846082283 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.025+0000 7f84fe000700 1 --2- 172.21.15.89:0/1846082283 >> v2:172.21.15.89:3300/0 conn(0x7f85000ad180 0x7f85000b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.025+0000 7f84fea00700 1 --2- 172.21.15.89:0/1846082283 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85000acda0 0x7f850015e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.025+0000 7f84fe000700 1 -- 172.21.15.89:0/1846082283 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85000acda0 msgr2=0x7f850015e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.025+0000 7f84fe000700 1 --2- 172.21.15.89:0/1846082283 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85000acda0 0x7f850015e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.025+0000 7f84fd600700 1 -- 172.21.15.89:0/1846082283 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756036729 0 0) 0x7f850015eb80 con 0x7f85000b4690 2024-04-02T09:17:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.025+0000 7f84fe000700 1 -- 172.21.15.89:0/1846082283 >> v1:172.21.15.89:6789/0 conn(0x7f85000b4690 legacy=0x7f85000b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.025+0000 7f84fe000700 1 -- 172.21.15.89:0/1846082283 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85000aea90 con 0x7f85000ad180 2024-04-02T09:17:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.025+0000 7f84fe000700 1 --2- 172.21.15.89:0/1846082283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85000ad180 0x7f85000b4150 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7f84ec009840 tx=0x7f84ec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f722bf2a37e1d16c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.026+0000 7f84fd600700 1 -- 172.21.15.89:0/1846082283 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec011830 con 0x7f85000ad180 2024-04-02T09:17:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.026+0000 7f84fd600700 1 -- 172.21.15.89:0/1846082283 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ec011990 con 0x7f85000ad180 2024-04-02T09:17:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.026+0000 7f84fd600700 1 -- 172.21.15.89:0/1846082283 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec011d20 con 0x7f85000ad180 2024-04-02T09:17:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.026+0000 7f8505800700 1 -- 172.21.15.89:0/1846082283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85000ad180 msgr2=0x7f85000b4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.026+0000 7f8505800700 1 --2- 172.21.15.89:0/1846082283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85000ad180 0x7f85000b4150 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7f84ec009840 tx=0x7f84ec009b50 comp rx=0 tx=0).stop 2024-04-02T09:17:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.026+0000 7f8505800700 1 -- 172.21.15.89:0/1846082283 shutdown_connections 2024-04-02T09:17:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.026+0000 7f8505800700 1 --2- 172.21.15.89:0/1846082283 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85000acda0 0x7f850015e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.026+0000 7f8505800700 1 --2- 172.21.15.89:0/1846082283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85000ad180 0x7f85000b4150 unknown :-1 s=CLOSED pgs=3037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.026+0000 7f8505800700 1 -- 172.21.15.89:0/1846082283 >> 172.21.15.89:0/1846082283 conn(0x7f8500072170 msgr2=0x7f850006a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.026+0000 7f8505800700 1 -- 172.21.15.89:0/1846082283 shutdown_connections 2024-04-02T09:17:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.026+0000 7f8505800700 1 -- 172.21.15.89:0/1846082283 wait complete. 2024-04-02T09:17:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.029+0000 7f8505800700 1 Processor -- start 2024-04-02T09:17:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.029+0000 7f8505800700 1 -- start start 2024-04-02T09:17:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.030+0000 7f8505800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85000acda0 0x7f85000ce420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.030+0000 7f8505800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85000ad180 0x7f85000ce960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.030+0000 7f8505800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85000b4690 0x7f85000ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.030+0000 7f8505800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85000cc620 con 0x7f85000acda0 2024-04-02T09:17:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.030+0000 7f8505800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8500153f50 con 0x7f85000b4690 2024-04-02T09:17:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.030+0000 7f8505800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8500154230 con 0x7f85000ad180 2024-04-02T09:17:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.030+0000 7f84fea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85000acda0 0x7f85000ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.030+0000 7f84fea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85000acda0 0x7f85000ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56076/0 (socket says 172.21.15.89:56076) 2024-04-02T09:17:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.030+0000 7f84fea00700 1 -- 172.21.15.89:0/2051964134 learned_addr learned my addr 172.21.15.89:0/2051964134 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.030+0000 7f84fe000700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85000ad180 0x7f85000ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.030+0000 7f84ff400700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85000b4690 0x7f85000ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.031+0000 7f84fe000700 1 -- 172.21.15.89:0/2051964134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85000b4690 msgr2=0x7f85000ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.031+0000 7f84fe000700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85000b4690 0x7f85000ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.031+0000 7f84fe000700 1 -- 172.21.15.89:0/2051964134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85000acda0 msgr2=0x7f85000ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.031+0000 7f84fe000700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85000acda0 0x7f85000ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.031+0000 7f84fe000700 1 -- 172.21.15.89:0/2051964134 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ec0094a0 con 0x7f85000ad180 2024-04-02T09:17:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.031+0000 7f84fea00700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85000acda0 0x7f85000ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:17:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.031+0000 7f84fe000700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85000ad180 0x7f85000ce960 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f84ec016000 tx=0x7f84ec004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.031+0000 7f84f6a00700 1 -- 172.21.15.89:0/2051964134 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec0036a0 con 0x7f85000ad180 2024-04-02T09:17:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.031+0000 7f84f6a00700 1 -- 172.21.15.89:0/2051964134 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ec01f070 con 0x7f85000ad180 2024-04-02T09:17:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.031+0000 7f84f6a00700 1 -- 172.21.15.89:0/2051964134 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec003a60 con 0x7f85000ad180 2024-04-02T09:17:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.031+0000 7f8505800700 1 -- 172.21.15.89:0/2051964134 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8500154510 con 0x7f85000ad180 2024-04-02T09:17:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.032+0000 7f8505800700 1 -- 172.21.15.89:0/2051964134 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8500152dd0 con 0x7f85000ad180 2024-04-02T09:17:01.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.033+0000 7f8505800700 1 -- 172.21.15.89:0/2051964134 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8500066e90 con 0x7f85000ad180 2024-04-02T09:17:01.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.033+0000 7f84f6a00700 1 -- 172.21.15.89:0/2051964134 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f84ec003bc0 con 0x7f85000ad180 2024-04-02T09:17:01.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.034+0000 7f84f6a00700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f84e006af60 0x7f84e006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.034+0000 7f84fea00700 1 -- 172.21.15.89:0/2051964134 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f84e006af60 msgr2=0x7f84e006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:01.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.034+0000 7f84f6a00700 1 -- 172.21.15.89:0/2051964134 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f84ec021070 con 0x7f85000ad180 2024-04-02T09:17:01.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.034+0000 7f84fea00700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f84e006af60 0x7f84e006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:17:01.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.039+0000 7f84f6a00700 1 -- 172.21.15.89:0/2051964134 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f84ec055e20 con 0x7f85000ad180 2024-04-02T09:17:01.157 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:01.158+0000 7f0061d0f240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:17:01.182 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:01.184+0000 7fdd2c137240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T09:17:01.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.235+0000 7f84fea00700 1 -- 172.21.15.89:0/2051964134 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f84e006af60 msgr2=0x7f84e006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:01.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.235+0000 7f84fea00700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f84e006af60 0x7f84e006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:17:01.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.248+0000 7f8505800700 1 -- 172.21.15.89:0/2051964134 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8500154370 con 0x7f85000ad180 2024-04-02T09:17:01.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.250+0000 7f84f6a00700 1 -- 172.21.15.89:0/2051964134 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v280) v1 ==== 74+0+17226 (secure 0 0 0) 0x7f8500154370 con 0x7f85000ad180 2024-04-02T09:17:01.248 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:17:01.249 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":280,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:15:43.128368+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:15:24.345253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-04-02T09:15:25.756381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"279","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/2320969126":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/2687295125":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/693914703":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/1405191830":"2024-04-03T09:15:43.128314+0000","172.21.15.89:6833/108855":"2024-04-03T09:15:43.128314+0000","172.21.15.89:6832/108855":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/701778740":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/799882283":"2024-04-03T09:15:43.128314+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:17:01.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.252+0000 7f8505800700 1 -- 172.21.15.89:0/2051964134 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f84e006af60 msgr2=0x7f84e006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:17:01.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.252+0000 7f8505800700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f84e006af60 0x7f84e006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.252+0000 7f8505800700 1 -- 172.21.15.89:0/2051964134 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85000ad180 msgr2=0x7f85000ce960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:01.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.252+0000 7f8505800700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85000ad180 0x7f85000ce960 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f84ec016000 tx=0x7f84ec004750 comp rx=0 tx=0).stop 2024-04-02T09:17:01.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.252+0000 7f8505800700 1 -- 172.21.15.89:0/2051964134 shutdown_connections 2024-04-02T09:17:01.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.252+0000 7f8505800700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f84e006af60 0x7f84e006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.252+0000 7f8505800700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85000b4690 0x7f85000ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.252+0000 7f8505800700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85000acda0 0x7f85000ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.252+0000 7f8505800700 1 --2- 172.21.15.89:0/2051964134 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f85000ad180 0x7f85000ce960 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.252+0000 7f8505800700 1 -- 172.21.15.89:0/2051964134 >> 172.21.15.89:0/2051964134 conn(0x7f8500072170 msgr2=0x7f85000af9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:01.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.252+0000 7f8505800700 1 -- 172.21.15.89:0/2051964134 shutdown_connections 2024-04-02T09:17:01.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.252+0000 7f8505800700 1 -- 172.21.15.89:0/2051964134 wait complete. 2024-04-02T09:17:01.269 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T09:15:24.345253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '278', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T09:15:25.756381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '279', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:17:01.270 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:17:01.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.510+0000 7fa744c00700 1 Processor -- start 2024-04-02T09:17:01.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.510+0000 7fa744c00700 1 -- start start 2024-04-02T09:17:01.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.511+0000 7fa744c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa7400cd020 0x7fa7400cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.511+0000 7fa744c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7400b2070 0x7fa7400b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.511+0000 7fa744c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7400cdae0 con 0x7fa7400cbb30 2024-04-02T09:17:01.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.511+0000 7fa744c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7400cddc0 con 0x7fa7400cd020 2024-04-02T09:17:01.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.511+0000 7fa744c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400ce0a0 con 0x7fa7400b2070 2024-04-02T09:17:01.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.512+0000 7fa73e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7400b2070 0x7fa7400b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.512+0000 7fa73d400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa7400cd020 0x7fa7400cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.512+0000 7fa73e800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7400b2070 0x7fa7400b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:58222/0 (socket says 172.21.15.89:58222) 2024-04-02T09:17:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.512+0000 7fa73d400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa7400cd020 0x7fa7400cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:48534/0 (socket says 172.21.15.89:48534) 2024-04-02T09:17:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.512+0000 7fa73e800700 1 -- 172.21.15.89:0/1171778251 learned_addr learned my addr 172.21.15.89:0/1171778251 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.512+0000 7fa73de00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fa7400cbb30 0x7fa7400cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:54896/0 (socket says 172.21.15.89:54896) 2024-04-02T09:17:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.512+0000 7fa73ca00700 1 -- 172.21.15.89:0/1171778251 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441667114 0 0) 0x7fa7400cdae0 con 0x7fa7400cbb30 2024-04-02T09:17:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.512+0000 7fa73d400700 1 -- 172.21.15.89:0/1171778251 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7400b2070 msgr2=0x7fa7400b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.512+0000 7fa73d400700 1 --2- 172.21.15.89:0/1171778251 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7400b2070 0x7fa7400b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.512+0000 7fa73d400700 1 -- 172.21.15.89:0/1171778251 >> v1:172.21.15.113:6789/0 conn(0x7fa7400cbb30 legacy=0x7fa7400cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.512+0000 7fa73d400700 1 -- 172.21.15.89:0/1171778251 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74006dc20 con 0x7fa7400cd020 2024-04-02T09:17:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.512+0000 7fa73e800700 1 --2- 172.21.15.89:0/1171778251 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7400b2070 0x7fa7400b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:17:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.513+0000 7fa73d400700 1 --2- 172.21.15.89:0/1171778251 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7400cd020 0x7fa7400cd480 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009840 tx=0x7fa72c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=643e742f6f83934d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.513+0000 7fa73ca00700 1 -- 172.21.15.89:0/1171778251 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c011830 con 0x7fa7400cd020 2024-04-02T09:17:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.513+0000 7fa73ca00700 1 -- 172.21.15.89:0/1171778251 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c011990 con 0x7fa7400cd020 2024-04-02T09:17:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.513+0000 7fa73ca00700 1 -- 172.21.15.89:0/1171778251 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c011d20 con 0x7fa7400cd020 2024-04-02T09:17:01.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.514+0000 7fa744c00700 1 -- 172.21.15.89:0/1171778251 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7400cd020 msgr2=0x7fa7400cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:01.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.514+0000 7fa744c00700 1 --2- 172.21.15.89:0/1171778251 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7400cd020 0x7fa7400cd480 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009840 tx=0x7fa72c009b50 comp rx=0 tx=0).stop 2024-04-02T09:17:01.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.514+0000 7fa744c00700 1 -- 172.21.15.89:0/1171778251 shutdown_connections 2024-04-02T09:17:01.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.514+0000 7fa744c00700 1 --2- 172.21.15.89:0/1171778251 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7400b2070 0x7fa7400b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.514+0000 7fa744c00700 1 --2- 172.21.15.89:0/1171778251 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7400cd020 0x7fa7400cd480 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.514+0000 7fa744c00700 1 -- 172.21.15.89:0/1171778251 >> 172.21.15.89:0/1171778251 conn(0x7fa7400b0d70 msgr2=0x7fa7400b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:01.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.514+0000 7fa744c00700 1 -- 172.21.15.89:0/1171778251 shutdown_connections 2024-04-02T09:17:01.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.514+0000 7fa744c00700 1 -- 172.21.15.89:0/1171778251 wait complete. 2024-04-02T09:17:01.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.516+0000 7fa744c00700 1 Processor -- start 2024-04-02T09:17:01.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.516+0000 7fa744c00700 1 -- start start 2024-04-02T09:17:01.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.517+0000 7fa744c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7400b2070 0x7fa740157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.517+0000 7fa744c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7400cbb30 0x7fa74015dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.517+0000 7fa744c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7400cd020 0x7fa74015e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.517+0000 7fa744c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400cac00 con 0x7fa7400b2070 2024-04-02T09:17:01.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.517+0000 7fa744c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa740157c60 con 0x7fa7400cd020 2024-04-02T09:17:01.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.517+0000 7fa744c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400c9250 con 0x7fa7400cbb30 2024-04-02T09:17:01.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.517+0000 7fa73de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7400b2070 0x7fa740157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.517+0000 7fa73de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7400b2070 0x7fa740157720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56088/0 (socket says 172.21.15.89:56088) 2024-04-02T09:17:01.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.517+0000 7fa73de00700 1 -- 172.21.15.89:0/3391733404 learned_addr learned my addr 172.21.15.89:0/3391733404 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:01.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.517+0000 7fa73d400700 1 --2- 172.21.15.89:0/3391733404 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7400cbb30 0x7fa74015dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.517+0000 7fa73e800700 1 --2- 172.21.15.89:0/3391733404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7400cd020 0x7fa74015e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.518+0000 7fa73de00700 1 -- 172.21.15.89:0/3391733404 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7400cbb30 msgr2=0x7fa74015dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:01.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.518+0000 7fa73de00700 1 --2- 172.21.15.89:0/3391733404 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7400cbb30 0x7fa74015dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.518+0000 7fa73de00700 1 -- 172.21.15.89:0/3391733404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7400cd020 msgr2=0x7fa74015e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:01.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.518+0000 7fa73de00700 1 --2- 172.21.15.89:0/3391733404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7400cd020 0x7fa74015e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.518+0000 7fa73de00700 1 -- 172.21.15.89:0/3391733404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa738009b30 con 0x7fa7400b2070 2024-04-02T09:17:01.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.518+0000 7fa73de00700 1 --2- 172.21.15.89:0/3391733404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7400b2070 0x7fa740157720 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7fa73000c750 tx=0x7fa73000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:01.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.518+0000 7fa736000700 1 -- 172.21.15.89:0/3391733404 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa730004c80 con 0x7fa7400b2070 2024-04-02T09:17:01.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.518+0000 7fa736000700 1 -- 172.21.15.89:0/3391733404 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa730004e00 con 0x7fa7400b2070 2024-04-02T09:17:01.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.518+0000 7fa744c00700 1 -- 172.21.15.89:0/3391733404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa72c0094a0 con 0x7fa7400b2070 2024-04-02T09:17:01.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.518+0000 7fa736000700 1 -- 172.21.15.89:0/3391733404 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa730007310 con 0x7fa7400b2070 2024-04-02T09:17:01.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.518+0000 7fa744c00700 1 -- 172.21.15.89:0/3391733404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7400addb0 con 0x7fa7400b2070 2024-04-02T09:17:01.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.520+0000 7fa744c00700 1 -- 172.21.15.89:0/3391733404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa740067720 con 0x7fa7400b2070 2024-04-02T09:17:01.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.521+0000 7fa736000700 1 -- 172.21.15.89:0/3391733404 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fa730007470 con 0x7fa7400b2070 2024-04-02T09:17:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.523+0000 7fa736000700 1 --2- 172.21.15.89:0/3391733404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa72006af60 0x7fa72006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.523+0000 7fa73d400700 1 -- 172.21.15.89:0/3391733404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa72006af60 msgr2=0x7fa72006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.523+0000 7fa736000700 1 -- 172.21.15.89:0/3391733404 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fa7300896d0 con 0x7fa7400b2070 2024-04-02T09:17:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.523+0000 7fa73d400700 1 --2- 172.21.15.89:0/3391733404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa72006af60 0x7fa72006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:17:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.527+0000 7fa736000700 1 -- 172.21.15.89:0/3391733404 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa730054240 con 0x7fa7400b2070 2024-04-02T09:17:01.685 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:01.687+0000 7f0061d0f240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:17:01.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.723+0000 7fa73d400700 1 -- 172.21.15.89:0/3391733404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa72006af60 msgr2=0x7fa72006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:01.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.723+0000 7fa73d400700 1 --2- 172.21.15.89:0/3391733404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa72006af60 0x7fa72006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:17:01.734 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:01.734+0000 7fdd2c137240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T09:17:01.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.736+0000 7fa744c00700 1 -- 172.21.15.89:0/3391733404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa7400ce2e0 con 0x7fa7400b2070 2024-04-02T09:17:01.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.736+0000 7fa736000700 1 -- 172.21.15.89:0/3391733404 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v280) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa730057860 con 0x7fa7400b2070 2024-04-02T09:17:01.736 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:17:01.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.738+0000 7fa744c00700 1 -- 172.21.15.89:0/3391733404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa72006af60 msgr2=0x7fa72006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:17:01.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.738+0000 7fa744c00700 1 --2- 172.21.15.89:0/3391733404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa72006af60 0x7fa72006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.738+0000 7fa744c00700 1 -- 172.21.15.89:0/3391733404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7400b2070 msgr2=0x7fa740157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:01.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.738+0000 7fa744c00700 1 --2- 172.21.15.89:0/3391733404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7400b2070 0x7fa740157720 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7fa73000c750 tx=0x7fa73000ca60 comp rx=0 tx=0).stop 2024-04-02T09:17:01.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.739+0000 7fa744c00700 1 -- 172.21.15.89:0/3391733404 shutdown_connections 2024-04-02T09:17:01.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.739+0000 7fa744c00700 1 --2- 172.21.15.89:0/3391733404 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa72006af60 0x7fa72006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.739+0000 7fa744c00700 1 --2- 172.21.15.89:0/3391733404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7400cd020 0x7fa74015e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.739+0000 7fa744c00700 1 --2- 172.21.15.89:0/3391733404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7400b2070 0x7fa740157720 unknown :-1 s=CLOSED pgs=3038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.739+0000 7fa744c00700 1 --2- 172.21.15.89:0/3391733404 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa7400cbb30 0x7fa74015dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.739+0000 7fa744c00700 1 -- 172.21.15.89:0/3391733404 >> 172.21.15.89:0/3391733404 conn(0x7fa7400b0d70 msgr2=0x7fa7400afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:01.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.739+0000 7fa744c00700 1 -- 172.21.15.89:0/3391733404 shutdown_connections 2024-04-02T09:17:01.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.739+0000 7fa744c00700 1 -- 172.21.15.89:0/3391733404 wait complete. 2024-04-02T09:17:01.756 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T09:17:01.940 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:01.941+0000 7f0061d0f240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:17:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.956+0000 7f72f3c00700 1 Processor -- start 2024-04-02T09:17:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.957+0000 7f72f3c00700 1 -- start start 2024-04-02T09:17:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.958+0000 7f72f3c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72ec15d9d0 0x7f72ec0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.958+0000 7f72f3c00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f72ec0d16a0 0x7f72ec0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.958+0000 7f72f3c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f72ec0d2370 0x7f72ec15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.958+0000 7f72f3c00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec0d2820 con 0x7f72ec0d16a0 2024-04-02T09:17:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.958+0000 7f72f3c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec161420 con 0x7f72ec0d2370 2024-04-02T09:17:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.958+0000 7f72f3c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec161700 con 0x7f72ec15d9d0 2024-04-02T09:17:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.958+0000 7f72ebe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f72ec0d16a0 0x7f72ec0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.958+0000 7f72ebe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f72ec0d16a0 0x7f72ec0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56104/0 (socket says 172.21.15.89:56104) 2024-04-02T09:17:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.958+0000 7f72ebe00700 1 -- 172.21.15.89:0/3150504576 learned_addr learned my addr 172.21.15.89:0/3150504576 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.958+0000 7f72f1c00700 1 --2- 172.21.15.89:0/3150504576 >> v2:172.21.15.113:3300/0 conn(0x7f72ec0d2370 0x7f72ec15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.958+0000 7f72f1200700 1 --2- 172.21.15.89:0/3150504576 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72ec15d9d0 0x7f72ec0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.959+0000 7f72ebe00700 1 -- 172.21.15.89:0/3150504576 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72ec15d9d0 msgr2=0x7f72ec0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.959+0000 7f72ebe00700 1 --2- 172.21.15.89:0/3150504576 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72ec15d9d0 0x7f72ec0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.959+0000 7f72ebe00700 1 -- 172.21.15.89:0/3150504576 >> v2:172.21.15.113:3300/0 conn(0x7f72ec0d2370 msgr2=0x7f72ec15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.959+0000 7f72ebe00700 1 --2- 172.21.15.89:0/3150504576 >> v2:172.21.15.113:3300/0 conn(0x7f72ec0d2370 0x7f72ec15ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.959+0000 7f72ebe00700 1 -- 172.21.15.89:0/3150504576 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec1619e0 con 0x7f72ec0d16a0 2024-04-02T09:17:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.959+0000 7f72ebe00700 1 --2- 172.21.15.89:0/3150504576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72ec0d16a0 0x7f72ec0d1e30 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0cdc00 tx=0x7f72d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378774d6ddf28768 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.959+0000 7f72eb400700 1 -- 172.21.15.89:0/3150504576 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d8005e30 con 0x7f72ec0d16a0 2024-04-02T09:17:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.959+0000 7f72eb400700 1 -- 172.21.15.89:0/3150504576 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d8004030 con 0x7f72ec0d16a0 2024-04-02T09:17:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.959+0000 7f72eb400700 1 -- 172.21.15.89:0/3150504576 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d80043e0 con 0x7f72ec0d16a0 2024-04-02T09:17:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.960+0000 7f72f3c00700 1 -- 172.21.15.89:0/3150504576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72ec0d16a0 msgr2=0x7f72ec0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.960+0000 7f72f3c00700 1 --2- 172.21.15.89:0/3150504576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72ec0d16a0 0x7f72ec0d1e30 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0cdc00 tx=0x7f72d8009bd0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.960+0000 7f72f3c00700 1 -- 172.21.15.89:0/3150504576 shutdown_connections 2024-04-02T09:17:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.960+0000 7f72f3c00700 1 --2- 172.21.15.89:0/3150504576 >> v2:172.21.15.113:3300/0 conn(0x7f72ec0d2370 0x7f72ec15ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.960+0000 7f72f3c00700 1 --2- 172.21.15.89:0/3150504576 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72ec15d9d0 0x7f72ec0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.960+0000 7f72f3c00700 1 --2- 172.21.15.89:0/3150504576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72ec0d16a0 0x7f72ec0d1e30 unknown :-1 s=CLOSED pgs=3039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.960+0000 7f72f3c00700 1 -- 172.21.15.89:0/3150504576 >> 172.21.15.89:0/3150504576 conn(0x7f72ec08aff0 msgr2=0x7f72ec0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.960+0000 7f72f3c00700 1 -- 172.21.15.89:0/3150504576 shutdown_connections 2024-04-02T09:17:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.960+0000 7f72f3c00700 1 -- 172.21.15.89:0/3150504576 wait complete. 2024-04-02T09:17:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.962+0000 7f72f3c00700 1 Processor -- start 2024-04-02T09:17:01.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.962+0000 7f72f3c00700 1 -- start start 2024-04-02T09:17:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.963+0000 7f72f3c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72ec0d16a0 0x7f72ec15a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.963+0000 7f72f3c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72ec0d2370 0x7f72ec0b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.963+0000 7f72f3c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72ec15d9d0 0x7f72ec0b7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.963+0000 7f72f3c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0cb9f0 con 0x7f72ec0d16a0 2024-04-02T09:17:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.963+0000 7f72f3c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0b8e80 con 0x7f72ec15d9d0 2024-04-02T09:17:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.963+0000 7f72f3c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0b85e0 con 0x7f72ec0d2370 2024-04-02T09:17:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.963+0000 7f72f1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72ec0d16a0 0x7f72ec15a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.963+0000 7f72f1200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72ec0d16a0 0x7f72ec15a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56118/0 (socket says 172.21.15.89:56118) 2024-04-02T09:17:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.963+0000 7f72f1200700 1 -- 172.21.15.89:0/2010408247 learned_addr learned my addr 172.21.15.89:0/2010408247 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.963+0000 7f72ebe00700 1 --2- 172.21.15.89:0/2010408247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72ec0d2370 0x7f72ec0b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.964+0000 7f72f1c00700 1 --2- 172.21.15.89:0/2010408247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72ec15d9d0 0x7f72ec0b7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.964+0000 7f72f1200700 1 -- 172.21.15.89:0/2010408247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72ec0d2370 msgr2=0x7f72ec0b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.964+0000 7f72f1200700 1 --2- 172.21.15.89:0/2010408247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72ec0d2370 0x7f72ec0b7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.964+0000 7f72f1200700 1 -- 172.21.15.89:0/2010408247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72ec15d9d0 msgr2=0x7f72ec0b7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.964+0000 7f72f1200700 1 --2- 172.21.15.89:0/2010408247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72ec15d9d0 0x7f72ec0b7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.964+0000 7f72f1200700 1 -- 172.21.15.89:0/2010408247 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d8009500 con 0x7f72ec0d16a0 2024-04-02T09:17:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.964+0000 7f72f1200700 1 --2- 172.21.15.89:0/2010408247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72ec0d16a0 0x7f72ec15a0e0 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7f72e000e8a0 tx=0x7f72e000ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.964+0000 7f72e8c00700 1 -- 172.21.15.89:0/2010408247 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0009590 con 0x7f72ec0d16a0 2024-04-02T09:17:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.964+0000 7f72e8c00700 1 -- 172.21.15.89:0/2010408247 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e0013680 con 0x7f72ec0d16a0 2024-04-02T09:17:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.964+0000 7f72e8c00700 1 -- 172.21.15.89:0/2010408247 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e00139f0 con 0x7f72ec0d16a0 2024-04-02T09:17:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.964+0000 7f72f3c00700 1 -- 172.21.15.89:0/2010408247 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ec0b88c0 con 0x7f72ec0d16a0 2024-04-02T09:17:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.965+0000 7f72f3c00700 1 -- 172.21.15.89:0/2010408247 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ec0cbc00 con 0x7f72ec0d16a0 2024-04-02T09:17:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.966+0000 7f72f3c00700 1 -- 172.21.15.89:0/2010408247 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ec0d1ca0 con 0x7f72ec0d16a0 2024-04-02T09:17:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.966+0000 7f72e8c00700 1 -- 172.21.15.89:0/2010408247 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f72e0013b50 con 0x7f72ec0d16a0 2024-04-02T09:17:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.967+0000 7f72e8c00700 1 --2- 172.21.15.89:0/2010408247 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72cc06af60 0x7f72cc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.967+0000 7f72e8c00700 1 -- 172.21.15.89:0/2010408247 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f72e00898f0 con 0x7f72ec0d16a0 2024-04-02T09:17:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.967+0000 7f72ebe00700 1 -- 172.21.15.89:0/2010408247 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72cc06af60 msgr2=0x7f72cc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.967+0000 7f72ebe00700 1 --2- 172.21.15.89:0/2010408247 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72cc06af60 0x7f72cc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:17:01.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:01.972+0000 7f72e8c00700 1 -- 172.21.15.89:0/2010408247 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f72e0054460 con 0x7f72ec0d16a0 2024-04-02T09:17:02.008 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:02.009+0000 7fdd2c137240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T09:17:02.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.168+0000 7f72ebe00700 1 -- 172.21.15.89:0/2010408247 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72cc06af60 msgr2=0x7f72cc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:02.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.168+0000 7f72ebe00700 1 --2- 172.21.15.89:0/2010408247 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72cc06af60 0x7f72cc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:17:02.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.184+0000 7f72f3c00700 1 -- 172.21.15.89:0/2010408247 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f72ec067720 con 0x7f72ec0d16a0 2024-04-02T09:17:02.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.184+0000 7f72e8c00700 1 -- 172.21.15.89:0/2010408247 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v280) v1 ==== 104+0+11 (secure 0 0 0) 0x7f72e0057a80 con 0x7f72ec0d16a0 2024-04-02T09:17:02.184 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:17:02.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.187+0000 7f72f3c00700 1 -- 172.21.15.89:0/2010408247 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72cc06af60 msgr2=0x7f72cc06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:17:02.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.187+0000 7f72f3c00700 1 --2- 172.21.15.89:0/2010408247 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72cc06af60 0x7f72cc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.187+0000 7f72f3c00700 1 -- 172.21.15.89:0/2010408247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72ec0d16a0 msgr2=0x7f72ec15a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:02.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.187+0000 7f72f3c00700 1 --2- 172.21.15.89:0/2010408247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72ec0d16a0 0x7f72ec15a0e0 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7f72e000e8a0 tx=0x7f72e000ebb0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.187+0000 7f72f3c00700 1 -- 172.21.15.89:0/2010408247 shutdown_connections 2024-04-02T09:17:02.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.187+0000 7f72f3c00700 1 --2- 172.21.15.89:0/2010408247 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f72cc06af60 0x7f72cc06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.187+0000 7f72f3c00700 1 --2- 172.21.15.89:0/2010408247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72ec15d9d0 0x7f72ec0b7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.187+0000 7f72f3c00700 1 --2- 172.21.15.89:0/2010408247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72ec0d16a0 0x7f72ec15a0e0 unknown :-1 s=CLOSED pgs=3040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.187+0000 7f72f3c00700 1 --2- 172.21.15.89:0/2010408247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f72ec0d2370 0x7f72ec0b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.187+0000 7f72f3c00700 1 -- 172.21.15.89:0/2010408247 >> 172.21.15.89:0/2010408247 conn(0x7f72ec08aff0 msgr2=0x7f72ec0ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:02.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.187+0000 7f72f3c00700 1 -- 172.21.15.89:0/2010408247 shutdown_connections 2024-04-02T09:17:02.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.187+0000 7f72f3c00700 1 -- 172.21.15.89:0/2010408247 wait complete. 2024-04-02T09:17:02.205 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T09:17:02.400 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:02.401+0000 7f0061d0f240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:17:02.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.407+0000 7fecfa000700 1 Processor -- start 2024-04-02T09:17:02.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.408+0000 7fecfa000700 1 -- start start 2024-04-02T09:17:02.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.409+0000 7fecfa000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fecf4157f90 0x7fecf40b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:02.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.409+0000 7fecfa000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fecf41613e0 0x7fecf4161860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:02.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.409+0000 7fecfa000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf40b3070 con 0x7fecf40b3420 2024-04-02T09:17:02.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.409+0000 7fecfa000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf4158370 con 0x7fecf4157f90 2024-04-02T09:17:02.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.409+0000 7fecfa000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40b2710 con 0x7fecf41613e0 2024-04-02T09:17:02.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.409+0000 7fecf3400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fecf4157f90 0x7fecf40b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:02.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.409+0000 7fecebe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fecf41613e0 0x7fecf4161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:02.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.409+0000 7fecf3400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fecf4157f90 0x7fecf40b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:48568/0 (socket says 172.21.15.89:48568) 2024-04-02T09:17:02.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.409+0000 7fecf3400700 1 -- 172.21.15.89:0/2599139234 learned_addr learned my addr 172.21.15.89:0/2599139234 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:02.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.410+0000 7fecf3400700 1 -- 172.21.15.89:0/2599139234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fecf41613e0 msgr2=0x7fecf4161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:02.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.410+0000 7fecf3400700 1 --2- 172.21.15.89:0/2599139234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fecf41613e0 0x7fecf4161860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.410+0000 7fecf3400700 1 -- 172.21.15.89:0/2599139234 >> v1:172.21.15.113:6789/0 conn(0x7fecf40b3420 legacy=0x7fecf40cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:02.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.410+0000 7fecf3400700 1 -- 172.21.15.89:0/2599139234 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf40b2070 con 0x7fecf4157f90 2024-04-02T09:17:02.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.410+0000 7fecebe00700 1 --2- 172.21.15.89:0/2599139234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fecf41613e0 0x7fecf4161860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:17:02.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.410+0000 7fecf3400700 1 --2- 172.21.15.89:0/2599139234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecf4157f90 0x7fecf40b2b30 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fece40098f0 tx=0x7fece4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb07ab1c1cca2ad7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:02.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.410+0000 7fecf2a00700 1 -- 172.21.15.89:0/2599139234 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4011830 con 0x7fecf4157f90 2024-04-02T09:17:02.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.410+0000 7fecf2a00700 1 -- 172.21.15.89:0/2599139234 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece4011990 con 0x7fecf4157f90 2024-04-02T09:17:02.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.410+0000 7fecf2a00700 1 -- 172.21.15.89:0/2599139234 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4011d20 con 0x7fecf4157f90 2024-04-02T09:17:02.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.411+0000 7fecfa000700 1 -- 172.21.15.89:0/2599139234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecf4157f90 msgr2=0x7fecf40b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:02.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.411+0000 7fecfa000700 1 --2- 172.21.15.89:0/2599139234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecf4157f90 0x7fecf40b2b30 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fece40098f0 tx=0x7fece4009c00 comp rx=0 tx=0).stop 2024-04-02T09:17:02.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.411+0000 7fecfa000700 1 -- 172.21.15.89:0/2599139234 shutdown_connections 2024-04-02T09:17:02.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.411+0000 7fecfa000700 1 --2- 172.21.15.89:0/2599139234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecf4157f90 0x7fecf40b2b30 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.411+0000 7fecfa000700 1 --2- 172.21.15.89:0/2599139234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fecf41613e0 0x7fecf4161860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.411+0000 7fecfa000700 1 -- 172.21.15.89:0/2599139234 >> 172.21.15.89:0/2599139234 conn(0x7fecf40b0d70 msgr2=0x7fecf40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:02.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.411+0000 7fecfa000700 1 -- 172.21.15.89:0/2599139234 shutdown_connections 2024-04-02T09:17:02.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.411+0000 7fecfa000700 1 -- 172.21.15.89:0/2599139234 wait complete. 2024-04-02T09:17:02.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.414+0000 7fecfa000700 1 Processor -- start 2024-04-02T09:17:02.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.414+0000 7fecfa000700 1 -- start start 2024-04-02T09:17:02.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.415+0000 7fecfa000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fecf40b3420 0x7fecf409a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:02.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.415+0000 7fecfa000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecf4157f90 0x7fecf40991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:02.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.415+0000 7fecfa000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecf41613e0 0x7fecf406d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:02.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.415+0000 7fecfa000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf41f8180 con 0x7fecf4157f90 2024-04-02T09:17:02.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.415+0000 7fecfa000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf4149bc0 con 0x7fecf41613e0 2024-04-02T09:17:02.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.415+0000 7fecfa000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf41f7220 con 0x7fecf40b3420 2024-04-02T09:17:02.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.416+0000 7fecebe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecf4157f90 0x7fecf40991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:02.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.416+0000 7fecf3400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fecf40b3420 0x7fecf409a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:02.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.416+0000 7fecf3400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fecf40b3420 0x7fecf409a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:58272/0 (socket says 172.21.15.89:58272) 2024-04-02T09:17:02.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.416+0000 7fecebe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecf4157f90 0x7fecf40991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56124/0 (socket says 172.21.15.89:56124) 2024-04-02T09:17:02.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.416+0000 7fecf3400700 1 -- 172.21.15.89:0/4814723 learned_addr learned my addr 172.21.15.89:0/4814723 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:02.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.416+0000 7fecf3e00700 1 --2- 172.21.15.89:0/4814723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecf41613e0 0x7fecf406d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:02.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.416+0000 7fecf3400700 1 -- 172.21.15.89:0/4814723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecf41613e0 msgr2=0x7fecf406d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:02.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.416+0000 7fecf3400700 1 --2- 172.21.15.89:0/4814723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecf41613e0 0x7fecf406d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.416+0000 7fecf3400700 1 -- 172.21.15.89:0/4814723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecf4157f90 msgr2=0x7fecf40991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:02.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.416+0000 7fecf3400700 1 --2- 172.21.15.89:0/4814723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecf4157f90 0x7fecf40991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.416+0000 7fecf3400700 1 -- 172.21.15.89:0/4814723 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecdc0094a0 con 0x7fecf40b3420 2024-04-02T09:17:02.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.416+0000 7fecf3400700 1 --2- 172.21.15.89:0/4814723 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fecf40b3420 0x7fecf409a5b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fece4000c00 tx=0x7fece4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:02.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.417+0000 7feceb400700 1 -- 172.21.15.89:0/4814723 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4004400 con 0x7fecf40b3420 2024-04-02T09:17:02.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.417+0000 7feceb400700 1 -- 172.21.15.89:0/4814723 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece401f070 con 0x7fecf40b3420 2024-04-02T09:17:02.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.417+0000 7feceb400700 1 -- 172.21.15.89:0/4814723 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece40038e0 con 0x7fecf40b3420 2024-04-02T09:17:02.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.417+0000 7fecfa000700 1 -- 172.21.15.89:0/4814723 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece4009550 con 0x7fecf40b3420 2024-04-02T09:17:02.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.417+0000 7fecfa000700 1 -- 172.21.15.89:0/4814723 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf41f77e0 con 0x7fecf40b3420 2024-04-02T09:17:02.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.418+0000 7fecfa000700 1 -- 172.21.15.89:0/4814723 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf406f2d0 con 0x7fecf40b3420 2024-04-02T09:17:02.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.419+0000 7feceb400700 1 -- 172.21.15.89:0/4814723 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fece4021030 con 0x7fecf40b3420 2024-04-02T09:17:02.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.420+0000 7feceb400700 1 --2- 172.21.15.89:0/4814723 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fecd006af60 0x7fecd006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:02.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.420+0000 7feceb400700 1 -- 172.21.15.89:0/4814723 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fece408ae70 con 0x7fecf40b3420 2024-04-02T09:17:02.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.420+0000 7fecebe00700 1 -- 172.21.15.89:0/4814723 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fecd006af60 msgr2=0x7fecd006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:02.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.420+0000 7fecebe00700 1 --2- 172.21.15.89:0/4814723 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fecd006af60 0x7fecd006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:17:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.425+0000 7feceb400700 1 -- 172.21.15.89:0/4814723 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fece4055ad0 con 0x7fecf40b3420 2024-04-02T09:17:02.495 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:02.496+0000 7fdd2c137240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T09:17:02.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.621+0000 7fecebe00700 1 -- 172.21.15.89:0/4814723 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fecd006af60 msgr2=0x7fecd006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:02.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.621+0000 7fecebe00700 1 --2- 172.21.15.89:0/4814723 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fecd006af60 0x7fecd006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:17:02.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.635+0000 7fecfa000700 1 -- 172.21.15.89:0/4814723 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fecf40cc8b0 con 0x7fecf40b3420 2024-04-02T09:17:02.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.636+0000 7feceb400700 1 -- 172.21.15.89:0/4814723 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v280) v1 ==== 100+0+11 (secure 0 0 0) 0x7fece4015020 con 0x7fecf40b3420 2024-04-02T09:17:02.635 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:17:02.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.638+0000 7fecfa000700 1 -- 172.21.15.89:0/4814723 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fecd006af60 msgr2=0x7fecd006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:17:02.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.638+0000 7fecfa000700 1 --2- 172.21.15.89:0/4814723 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fecd006af60 0x7fecd006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.638+0000 7fecfa000700 1 -- 172.21.15.89:0/4814723 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fecf40b3420 msgr2=0x7fecf409a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:02.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.638+0000 7fecfa000700 1 --2- 172.21.15.89:0/4814723 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fecf40b3420 0x7fecf409a5b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fece4000c00 tx=0x7fece4004750 comp rx=0 tx=0).stop 2024-04-02T09:17:02.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.638+0000 7fecfa000700 1 -- 172.21.15.89:0/4814723 shutdown_connections 2024-04-02T09:17:02.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.638+0000 7fecfa000700 1 --2- 172.21.15.89:0/4814723 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fecd006af60 0x7fecd006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.638+0000 7fecfa000700 1 --2- 172.21.15.89:0/4814723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecf41613e0 0x7fecf406d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.638+0000 7fecfa000700 1 --2- 172.21.15.89:0/4814723 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fecf40b3420 0x7fecf409a5b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.638+0000 7fecfa000700 1 --2- 172.21.15.89:0/4814723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecf4157f90 0x7fecf40991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.638+0000 7fecfa000700 1 -- 172.21.15.89:0/4814723 >> 172.21.15.89:0/4814723 conn(0x7fecf40b0d70 msgr2=0x7fecf40b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:02.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.638+0000 7fecfa000700 1 -- 172.21.15.89:0/4814723 shutdown_connections 2024-04-02T09:17:02.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.638+0000 7fecfa000700 1 -- 172.21.15.89:0/4814723 wait complete. 2024-04-02T09:17:02.656 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T09:17:02.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.856+0000 7f23e1e00700 1 Processor -- start 2024-04-02T09:17:02.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.856+0000 7f23e1e00700 1 -- start start 2024-04-02T09:17:02.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.858+0000 7f23e1e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23dc160d70 0x7f23dc0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.858+0000 7f23e1e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23dc06a790 0x7f23dc06ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.858+0000 7f23e1e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23dc161150 con 0x7f23dc15ef70 2024-04-02T09:17:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.858+0000 7f23e1e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23dc06b180 con 0x7f23dc06a790 2024-04-02T09:17:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.858+0000 7f23e1e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc0b4cd0 con 0x7f23dc160d70 2024-04-02T09:17:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.858+0000 7f23db400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23dc06a790 0x7f23dc06ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.858+0000 7f23db400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23dc06a790 0x7f23dc06ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56130/0 (socket says 172.21.15.89:56130) 2024-04-02T09:17:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.858+0000 7f23db400700 1 -- 172.21.15.89:0/1077051174 learned_addr learned my addr 172.21.15.89:0/1077051174 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.858+0000 7f23daa00700 1 --2- 172.21.15.89:0/1077051174 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23dc160d70 0x7f23dc0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:02.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.859+0000 7f23db400700 1 -- 172.21.15.89:0/1077051174 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23dc160d70 msgr2=0x7f23dc0b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:02.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.859+0000 7f23db400700 1 --2- 172.21.15.89:0/1077051174 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23dc160d70 0x7f23dc0b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.859+0000 7f23db400700 1 -- 172.21.15.89:0/1077051174 >> v1:172.21.15.113:6789/0 conn(0x7f23dc15ef70 legacy=0x7f23dc15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:02.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.859+0000 7f23db400700 1 -- 172.21.15.89:0/1077051174 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23dc06dba0 con 0x7f23dc06a790 2024-04-02T09:17:02.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.859+0000 7f23d9600700 1 -- 172.21.15.89:0/1077051174 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422862888 0 0) 0x7f23dc161150 con 0x7f23dc15ef70 2024-04-02T09:17:02.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.859+0000 7f23db400700 1 --2- 172.21.15.89:0/1077051174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23dc06a790 0x7f23dc06ac40 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009e50 tx=0x7f23d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952ccb8f8b1faa8a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:02.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.859+0000 7f23d9600700 1 -- 172.21.15.89:0/1077051174 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d400e070 con 0x7f23dc06a790 2024-04-02T09:17:02.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.859+0000 7f23d9600700 1 -- 172.21.15.89:0/1077051174 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d400b910 con 0x7f23dc06a790 2024-04-02T09:17:02.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.859+0000 7f23d9600700 1 -- 172.21.15.89:0/1077051174 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d400bcc0 con 0x7f23dc06a790 2024-04-02T09:17:02.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.860+0000 7f23e1e00700 1 -- 172.21.15.89:0/1077051174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23dc06a790 msgr2=0x7f23dc06ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:02.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.860+0000 7f23e1e00700 1 --2- 172.21.15.89:0/1077051174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23dc06a790 0x7f23dc06ac40 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009e50 tx=0x7f23d400b330 comp rx=0 tx=0).stop 2024-04-02T09:17:02.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.860+0000 7f23e1e00700 1 -- 172.21.15.89:0/1077051174 shutdown_connections 2024-04-02T09:17:02.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.860+0000 7f23e1e00700 1 --2- 172.21.15.89:0/1077051174 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23dc160d70 0x7f23dc0b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.860+0000 7f23e1e00700 1 --2- 172.21.15.89:0/1077051174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23dc06a790 0x7f23dc06ac40 unknown :-1 s=CLOSED pgs=3041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.860+0000 7f23e1e00700 1 -- 172.21.15.89:0/1077051174 >> 172.21.15.89:0/1077051174 conn(0x7f23dc0b5720 msgr2=0x7f23dc0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:02.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.860+0000 7f23e1e00700 1 -- 172.21.15.89:0/1077051174 shutdown_connections 2024-04-02T09:17:02.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.860+0000 7f23e1e00700 1 -- 172.21.15.89:0/1077051174 wait complete. 2024-04-02T09:17:02.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.862+0000 7f23e1e00700 1 Processor -- start 2024-04-02T09:17:02.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.862+0000 7f23e1e00700 1 -- start start 2024-04-02T09:17:02.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.863+0000 7f23e1e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23dc06a790 0x7f23dc13cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:02.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.863+0000 7f23e1e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23dc15ef70 0x7f23dc15cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:02.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.863+0000 7f23e1e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23dc160d70 0x7f23dc13e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:02.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.863+0000 7f23e1e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc0ad1e0 con 0x7f23dc06a790 2024-04-02T09:17:02.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.863+0000 7f23e1e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc13d490 con 0x7f23dc160d70 2024-04-02T09:17:02.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.863+0000 7f23e1e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc13d770 con 0x7f23dc15ef70 2024-04-02T09:17:02.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.863+0000 7f23daa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23dc06a790 0x7f23dc13cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:02.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.863+0000 7f23daa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23dc06a790 0x7f23dc13cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56146/0 (socket says 172.21.15.89:56146) 2024-04-02T09:17:02.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.863+0000 7f23daa00700 1 -- 172.21.15.89:0/983502643 learned_addr learned my addr 172.21.15.89:0/983502643 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:02.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.863+0000 7f23da000700 1 --2- 172.21.15.89:0/983502643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23dc15ef70 0x7f23dc15cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:02.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.863+0000 7f23db400700 1 --2- 172.21.15.89:0/983502643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23dc160d70 0x7f23dc13e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:02.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.864+0000 7f23daa00700 1 -- 172.21.15.89:0/983502643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23dc15ef70 msgr2=0x7f23dc15cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:02.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.864+0000 7f23daa00700 1 --2- 172.21.15.89:0/983502643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23dc15ef70 0x7f23dc15cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.864+0000 7f23daa00700 1 -- 172.21.15.89:0/983502643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23dc160d70 msgr2=0x7f23dc13e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:02.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.864+0000 7f23daa00700 1 --2- 172.21.15.89:0/983502643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23dc160d70 0x7f23dc13e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:02.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.864+0000 7f23daa00700 1 -- 172.21.15.89:0/983502643 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d4009b30 con 0x7f23dc06a790 2024-04-02T09:17:02.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.864+0000 7f23daa00700 1 --2- 172.21.15.89:0/983502643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23dc06a790 0x7f23dc13cea0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7f23cc00e870 tx=0x7f23cc00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:02.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.864+0000 7f23d3400700 1 -- 172.21.15.89:0/983502643 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23cc009590 con 0x7f23dc06a790 2024-04-02T09:17:02.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.864+0000 7f23d3400700 1 -- 172.21.15.89:0/983502643 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23cc013680 con 0x7f23dc06a790 2024-04-02T09:17:02.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.864+0000 7f23e1e00700 1 -- 172.21.15.89:0/983502643 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23dc13da50 con 0x7f23dc06a790 2024-04-02T09:17:02.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.864+0000 7f23d3400700 1 -- 172.21.15.89:0/983502643 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23cc0139f0 con 0x7f23dc06a790 2024-04-02T09:17:02.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.864+0000 7f23e1e00700 1 -- 172.21.15.89:0/983502643 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23dc15d950 con 0x7f23dc06a790 2024-04-02T09:17:02.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.866+0000 7f23e1e00700 1 -- 172.21.15.89:0/983502643 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23dc067720 con 0x7f23dc06a790 2024-04-02T09:17:02.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.866+0000 7f23d3400700 1 -- 172.21.15.89:0/983502643 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f23cc013b50 con 0x7f23dc06a790 2024-04-02T09:17:02.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.867+0000 7f23d3400700 1 --2- 172.21.15.89:0/983502643 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23bc06af60 0x7f23bc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:02.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.867+0000 7f23d3400700 1 -- 172.21.15.89:0/983502643 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f23cc0898a0 con 0x7f23dc06a790 2024-04-02T09:17:02.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.867+0000 7f23da000700 1 -- 172.21.15.89:0/983502643 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23bc06af60 msgr2=0x7f23bc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:02.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.867+0000 7f23da000700 1 --2- 172.21.15.89:0/983502643 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23bc06af60 0x7f23bc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:17:02.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:02.876+0000 7f23d3400700 1 -- 172.21.15.89:0/983502643 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f23cc054460 con 0x7f23dc06a790 2024-04-02T09:17:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.068+0000 7f23da000700 1 -- 172.21.15.89:0/983502643 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23bc06af60 msgr2=0x7f23bc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.068+0000 7f23da000700 1 --2- 172.21.15.89:0/983502643 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23bc06af60 0x7f23bc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:17:03.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.102+0000 7f23e1e00700 1 -- 172.21.15.89:0/983502643 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f23dc06ec30 con 0x7f23dc06a790 2024-04-02T09:17:03.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.103+0000 7f23d3400700 1 -- 172.21.15.89:0/983502643 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v280) v1 ==== 80+0+459 (secure 0 0 0) 0x7f23cc057a80 con 0x7f23dc06a790 2024-04-02T09:17:03.102 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2320969126 2024-04-03T09:15:43.128314+0000 2024-04-02T09:17:03.102 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/2687295125 2024-04-03T09:15:43.128314+0000 2024-04-02T09:17:03.102 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/693914703 2024-04-03T09:15:43.128314+0000 2024-04-02T09:17:03.102 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1405191830 2024-04-03T09:15:43.128314+0000 2024-04-02T09:17:03.103 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6833/108855 2024-04-03T09:15:43.128314+0000 2024-04-02T09:17:03.103 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6832/108855 2024-04-03T09:15:43.128314+0000 2024-04-02T09:17:03.103 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/701778740 2024-04-03T09:15:43.128314+0000 2024-04-02T09:17:03.103 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/799882283 2024-04-03T09:15:43.128314+0000 2024-04-02T09:17:03.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.105+0000 7f23e1e00700 1 -- 172.21.15.89:0/983502643 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23bc06af60 msgr2=0x7f23bc06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:17:03.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.105+0000 7f23e1e00700 1 --2- 172.21.15.89:0/983502643 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23bc06af60 0x7f23bc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.105+0000 7f23e1e00700 1 -- 172.21.15.89:0/983502643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23dc06a790 msgr2=0x7f23dc13cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:03.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.105+0000 7f23e1e00700 1 --2- 172.21.15.89:0/983502643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23dc06a790 0x7f23dc13cea0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7f23cc00e870 tx=0x7f23cc00eb80 comp rx=0 tx=0).stop 2024-04-02T09:17:03.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.106+0000 7f23e1e00700 1 -- 172.21.15.89:0/983502643 shutdown_connections 2024-04-02T09:17:03.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.106+0000 7f23e1e00700 1 --2- 172.21.15.89:0/983502643 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f23bc06af60 0x7f23bc06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.106+0000 7f23e1e00700 1 --2- 172.21.15.89:0/983502643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23dc160d70 0x7f23dc13e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.106+0000 7f23e1e00700 1 --2- 172.21.15.89:0/983502643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23dc06a790 0x7f23dc13cea0 unknown :-1 s=CLOSED pgs=3042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.106+0000 7f23e1e00700 1 --2- 172.21.15.89:0/983502643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f23dc15ef70 0x7f23dc15cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.106+0000 7f23e1e00700 1 -- 172.21.15.89:0/983502643 >> 172.21.15.89:0/983502643 conn(0x7f23dc0b5720 msgr2=0x7f23dc15dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:03.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.106+0000 7f23e1e00700 1 -- 172.21.15.89:0/983502643 shutdown_connections 2024-04-02T09:17:03.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.106+0000 7f23e1e00700 1 -- 172.21.15.89:0/983502643 wait complete. 2024-04-02T09:17:03.107 INFO:teuthology.orchestra.run.smithi089.stderr:listed 8 entries 2024-04-02T09:17:03.118 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:03.120+0000 7f0061d0f240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:17:03.123 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T09:17:03.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:17:03.123 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-02T09:17:03.197 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T09:17:03.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:17:03.198 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-02T09:17:03.209 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:03.211+0000 7f0061d0f240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:17:03.243 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:17:03.243 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:17:03.253 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:03.254+0000 7fdd2c137240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T09:17:03.291 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:17:03.291 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:17:03.291 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:17:03.336 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:17:03.337 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:17:03.337 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:17:03.350 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:03.351+0000 7fdd2c137240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T09:17:03.387 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:17:03.387 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:17:03.387 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:03.388+0000 7f0061d0f240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:17:03.520 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:03.520+0000 7fdd2c137240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T09:17:03.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.584+0000 7f7abae00700 1 Processor -- start 2024-04-02T09:17:03.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.584+0000 7f7abae00700 1 -- start start 2024-04-02T09:17:03.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.585+0000 7f7abae00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7ab40b3af0 0x7f7ab415a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:03.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.585+0000 7f7abae00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7ab40709c0 0x7f7ab4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:03.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.585+0000 7f7abae00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ab40b2180 con 0x7f7ab4072580 2024-04-02T09:17:03.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.585+0000 7f7abae00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ab415b100 con 0x7f7ab40709c0 2024-04-02T09:17:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.585+0000 7f7abae00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ab415b3e0 con 0x7f7ab40b3af0 2024-04-02T09:17:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.585+0000 7f7ab3400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7ab4072580 0x7f7ab40b2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50982/0 (socket says 172.21.15.89:50982) 2024-04-02T09:17:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.585+0000 7f7ab3400700 1 -- 172.21.15.89:0/1735630026 learned_addr learned my addr 172.21.15.89:0/1735630026 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.585+0000 7f7ab3e00700 1 --2- 172.21.15.89:0/1735630026 >> v2:172.21.15.113:3300/0 conn(0x7f7ab40b3af0 0x7f7ab415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.585+0000 7f7ab8e00700 1 --2- 172.21.15.89:0/1735630026 >> v2:172.21.15.89:3300/0 conn(0x7f7ab40709c0 0x7f7ab4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.586+0000 7f7ab2a00700 1 -- 172.21.15.89:0/1735630026 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333771838 0 0) 0x7f7ab40b2180 con 0x7f7ab4072580 2024-04-02T09:17:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.586+0000 7f7ab2a00700 1 -- 172.21.15.89:0/1735630026 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a980035b0 con 0x7f7ab4072580 2024-04-02T09:17:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.586+0000 7f7ab8e00700 1 -- 172.21.15.89:0/1735630026 >> v2:172.21.15.113:3300/0 conn(0x7f7ab40b3af0 msgr2=0x7f7ab415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.586+0000 7f7ab8e00700 1 --2- 172.21.15.89:0/1735630026 >> v2:172.21.15.113:3300/0 conn(0x7f7ab40b3af0 0x7f7ab415a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.586+0000 7f7ab8e00700 1 -- 172.21.15.89:0/1735630026 >> v1:172.21.15.89:6789/0 conn(0x7f7ab4072580 legacy=0x7f7ab40b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:03.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.586+0000 7f7ab8e00700 1 -- 172.21.15.89:0/1735630026 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab40b23f0 con 0x7f7ab40709c0 2024-04-02T09:17:03.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.586+0000 7f7ab3e00700 1 --2- 172.21.15.89:0/1735630026 >> v2:172.21.15.113:3300/0 conn(0x7f7ab40b3af0 0x7f7ab415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:17:03.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.586+0000 7f7ab8e00700 1 --2- 172.21.15.89:0/1735630026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab40709c0 0x7f7ab4070e00 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7f7ab406b860 tx=0x7f7aac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cde0bf7f850c38b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.586+0000 7f7ab2a00700 1 -- 172.21.15.89:0/1735630026 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aac00e070 con 0x7f7ab40709c0 2024-04-02T09:17:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.586+0000 7f7ab2a00700 1 -- 172.21.15.89:0/1735630026 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aac00bb20 con 0x7f7ab40709c0 2024-04-02T09:17:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.587+0000 7f7ab2a00700 1 -- 172.21.15.89:0/1735630026 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aac012430 con 0x7f7ab40709c0 2024-04-02T09:17:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.587+0000 7f7abae00700 1 -- 172.21.15.89:0/1735630026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab40709c0 msgr2=0x7f7ab4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.587+0000 7f7abae00700 1 --2- 172.21.15.89:0/1735630026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab40709c0 0x7f7ab4070e00 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7f7ab406b860 tx=0x7f7aac00b330 comp rx=0 tx=0).stop 2024-04-02T09:17:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.587+0000 7f7abae00700 1 -- 172.21.15.89:0/1735630026 shutdown_connections 2024-04-02T09:17:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.587+0000 7f7abae00700 1 --2- 172.21.15.89:0/1735630026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab40709c0 0x7f7ab4070e00 unknown :-1 s=CLOSED pgs=3044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.587+0000 7f7abae00700 1 --2- 172.21.15.89:0/1735630026 >> v2:172.21.15.113:3300/0 conn(0x7f7ab40b3af0 0x7f7ab415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.587+0000 7f7abae00700 1 -- 172.21.15.89:0/1735630026 >> 172.21.15.89:0/1735630026 conn(0x7f7ab40cf190 msgr2=0x7f7ab40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:03.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.587+0000 7f7abae00700 1 -- 172.21.15.89:0/1735630026 shutdown_connections 2024-04-02T09:17:03.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.587+0000 7f7abae00700 1 -- 172.21.15.89:0/1735630026 wait complete. 2024-04-02T09:17:03.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.590+0000 7f7abae00700 1 Processor -- start 2024-04-02T09:17:03.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.590+0000 7f7abae00700 1 -- start start 2024-04-02T09:17:03.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.591+0000 7f7abae00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7ab40709c0 0x7f7ab4156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:03.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.591+0000 7f7abae00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4072580 0x7f7ab4157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:03.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.591+0000 7f7abae00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab40b3af0 0x7f7ab414e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:03.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.591+0000 7f7abae00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab41576a0 con 0x7f7ab4072580 2024-04-02T09:17:03.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.591+0000 7f7abae00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab4153420 con 0x7f7ab40b3af0 2024-04-02T09:17:03.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.591+0000 7f7abae00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab4158200 con 0x7f7ab40709c0 2024-04-02T09:17:03.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.591+0000 7f7ab3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4072580 0x7f7ab4157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:03.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.591+0000 7f7ab3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4072580 0x7f7ab4157040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56174/0 (socket says 172.21.15.89:56174) 2024-04-02T09:17:03.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.591+0000 7f7ab3400700 1 -- 172.21.15.89:0/3164256949 learned_addr learned my addr 172.21.15.89:0/3164256949 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:03.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.591+0000 7f7ab3e00700 1 --2- 172.21.15.89:0/3164256949 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7ab40709c0 0x7f7ab4156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:03.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.591+0000 7f7ab8e00700 1 --2- 172.21.15.89:0/3164256949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab40b3af0 0x7f7ab414e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:03.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.592+0000 7f7ab3400700 1 -- 172.21.15.89:0/3164256949 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7ab40709c0 msgr2=0x7f7ab4156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:03.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.592+0000 7f7ab3400700 1 --2- 172.21.15.89:0/3164256949 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7ab40709c0 0x7f7ab4156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.592+0000 7f7ab3400700 1 -- 172.21.15.89:0/3164256949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab40b3af0 msgr2=0x7f7ab414e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:03.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.592+0000 7f7ab3400700 1 --2- 172.21.15.89:0/3164256949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab40b3af0 0x7f7ab414e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.592+0000 7f7ab3400700 1 -- 172.21.15.89:0/3164256949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa40094a0 con 0x7f7ab4072580 2024-04-02T09:17:03.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.592+0000 7f7ab3400700 1 --2- 172.21.15.89:0/3164256949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4072580 0x7f7ab4157040 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0005910 tx=0x7f7aa000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:03.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.592+0000 7f7aabe00700 1 -- 172.21.15.89:0/3164256949 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0005d70 con 0x7f7ab4072580 2024-04-02T09:17:03.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.592+0000 7f7aabe00700 1 -- 172.21.15.89:0/3164256949 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa00070d0 con 0x7f7ab4072580 2024-04-02T09:17:03.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.592+0000 7f7abae00700 1 -- 172.21.15.89:0/3164256949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aac009a70 con 0x7f7ab4072580 2024-04-02T09:17:03.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.592+0000 7f7aabe00700 1 -- 172.21.15.89:0/3164256949 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0007460 con 0x7f7ab4072580 2024-04-02T09:17:03.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.592+0000 7f7abae00700 1 -- 172.21.15.89:0/3164256949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab41587c0 con 0x7f7ab4072580 2024-04-02T09:17:03.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.593+0000 7f7abae00700 1 -- 172.21.15.89:0/3164256949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab4067720 con 0x7f7ab4072580 2024-04-02T09:17:03.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.594+0000 7f7aabe00700 1 -- 172.21.15.89:0/3164256949 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f7aa00075c0 con 0x7f7ab4072580 2024-04-02T09:17:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.595+0000 7f7aabe00700 1 --2- 172.21.15.89:0/3164256949 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7a9406af60 0x7f7a9406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.595+0000 7f7aabe00700 1 -- 172.21.15.89:0/3164256949 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f7aa0088ef0 con 0x7f7ab4072580 2024-04-02T09:17:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.595+0000 7f7ab3e00700 1 -- 172.21.15.89:0/3164256949 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7a9406af60 msgr2=0x7f7a9406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:03.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.595+0000 7f7ab3e00700 1 --2- 172.21.15.89:0/3164256949 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7a9406af60 0x7f7a9406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:17:03.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.599+0000 7f7aabe00700 1 -- 172.21.15.89:0/3164256949 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7aa0053ab0 con 0x7f7ab4072580 2024-04-02T09:17:03.602 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:03.604+0000 7f0061d0f240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:17:03.694 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:03.696+0000 7f0061d0f240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:17:03.736 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:03.737+0000 7fdd2c137240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T09:17:03.793 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:03.794+0000 7f0061d0f240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:17:03.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.796+0000 7f7ab3e00700 1 -- 172.21.15.89:0/3164256949 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7a9406af60 msgr2=0x7f7a9406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:03.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.796+0000 7f7ab3e00700 1 --2- 172.21.15.89:0/3164256949 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7a9406af60 0x7f7a9406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:17:03.830 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:03.831+0000 7fdd2c137240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T09:17:03.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.851+0000 7f7abae00700 1 -- 172.21.15.89:0/3164256949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ab40cef70 con 0x7f7ab4072580 2024-04-02T09:17:03.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.852+0000 7f7aabe00700 1 -- 172.21.15.89:0/3164256949 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4753 (secure 0 0 0) 0x7f7aa0017090 con 0x7f7ab4072580 2024-04-02T09:17:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:17:03.852 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10376,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2447701494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2447701494},{"type":"v1","addr":"172.21.15.113:6834","nonce":2447701494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":12103,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/361614750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":361614750},{"type":"v1","addr":"172.21.15.113:6835","nonce":361614750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":13281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2871180587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2871180587},{"type":"v1","addr":"172.21.15.89:6835","nonce":2871180587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:15:26.772241+0000","modified":"2024-04-02T09:15:30.258749+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13308":{"gid":13308,"name":"a","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3853118738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3853118738},{"type":"v1","addr":"172.21.15.89:6837","nonce":3853118738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-04-02T09:17:03.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.855+0000 7f7abae00700 1 -- 172.21.15.89:0/3164256949 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7a9406af60 msgr2=0x7f7a9406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:17:03.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.855+0000 7f7abae00700 1 --2- 172.21.15.89:0/3164256949 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7a9406af60 0x7f7a9406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.855+0000 7f7abae00700 1 -- 172.21.15.89:0/3164256949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4072580 msgr2=0x7f7ab4157040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:03.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.855+0000 7f7abae00700 1 --2- 172.21.15.89:0/3164256949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4072580 0x7f7ab4157040 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0005910 tx=0x7f7aa000c9e0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.855+0000 7f7abae00700 1 -- 172.21.15.89:0/3164256949 shutdown_connections 2024-04-02T09:17:03.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.855+0000 7f7abae00700 1 --2- 172.21.15.89:0/3164256949 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f7a9406af60 0x7f7a9406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.855+0000 7f7abae00700 1 --2- 172.21.15.89:0/3164256949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab40b3af0 0x7f7ab414e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.855+0000 7f7abae00700 1 --2- 172.21.15.89:0/3164256949 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7ab40709c0 0x7f7ab4156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.855+0000 7f7abae00700 1 --2- 172.21.15.89:0/3164256949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4072580 0x7f7ab4157040 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:03.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.855+0000 7f7abae00700 1 -- 172.21.15.89:0/3164256949 >> 172.21.15.89:0/3164256949 conn(0x7f7ab40cf190 msgr2=0x7f7ab40aceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:03.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.855+0000 7f7abae00700 1 -- 172.21.15.89:0/3164256949 shutdown_connections 2024-04-02T09:17:03.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:03.855+0000 7f7abae00700 1 -- 172.21.15.89:0/3164256949 wait complete. 2024-04-02T09:17:03.856 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 144 2024-04-02T09:17:03.873 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:17:03.891 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:03.893+0000 7f0061d0f240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:17:03.937 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:03.938+0000 7fdd2c137240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T09:17:03.989 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:03.990+0000 7f0061d0f240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:17:04.041 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:04.042+0000 7fdd2c137240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T09:17:04.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.069+0000 7f619e600700 1 Processor -- start 2024-04-02T09:17:04.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.069+0000 7f619e600700 1 -- start start 2024-04-02T09:17:04.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.070+0000 7f619e600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f619815da50 0x7f61980ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.070+0000 7f619e600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61980d24e0 con 0x7f61980d0cf0 2024-04-02T09:17:04.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.070+0000 7f619e600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61980d27c0 con 0x7f61980cec70 2024-04-02T09:17:04.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.070+0000 7f619e600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61980b95d0 con 0x7f619815da50 2024-04-02T09:17:04.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.071+0000 7f6197400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f619815da50 0x7f61980ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:04.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.071+0000 7f6197e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f61980d0cf0 0x7f61980b6990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50988/0 (socket says 172.21.15.89:50988) 2024-04-02T09:17:04.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.071+0000 7f6197e00700 1 -- 172.21.15.89:0/1874496859 learned_addr learned my addr 172.21.15.89:0/1874496859 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:04.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.071+0000 7f6197400700 1 --2- 172.21.15.89:0/1874496859 >> v2:172.21.15.89:3300/0 conn(0x7f619815da50 0x7f61980ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56184/0 (socket says 172.21.15.89:56184) 2024-04-02T09:17:04.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.071+0000 7f6196a00700 1 --1- 172.21.15.89:0/1874496859 >> v1:172.21.15.113:6789/0 conn(0x7f61980cec70 0x7f61980d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:54932/0 (socket says 172.21.15.89:54932) 2024-04-02T09:17:04.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.071+0000 7f6197400700 1 -- 172.21.15.89:0/1874496859 >> v1:172.21.15.113:6789/0 conn(0x7f61980cec70 legacy=0x7f61980d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.071+0000 7f6197400700 1 -- 172.21.15.89:0/1874496859 >> v1:172.21.15.89:6789/0 conn(0x7f61980d0cf0 legacy=0x7f61980b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.071+0000 7f6197400700 1 -- 172.21.15.89:0/1874496859 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61980d2aa0 con 0x7f619815da50 2024-04-02T09:17:04.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.071+0000 7f6196000700 1 -- 172.21.15.89:0/1874496859 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861807117 0 0) 0x7f61980d24e0 con 0x7f61980d0cf0 2024-04-02T09:17:04.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.071+0000 7f6197400700 1 --2- 172.21.15.89:0/1874496859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f619815da50 0x7f61980ce730 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7f6188009840 tx=0x7f6188009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cbe69b13f20c0f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:04.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.072+0000 7f6196000700 1 -- 172.21.15.89:0/1874496859 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188011830 con 0x7f619815da50 2024-04-02T09:17:04.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.072+0000 7f6196000700 1 -- 172.21.15.89:0/1874496859 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6188011990 con 0x7f619815da50 2024-04-02T09:17:04.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.072+0000 7f6196000700 1 -- 172.21.15.89:0/1874496859 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188011d20 con 0x7f619815da50 2024-04-02T09:17:04.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.072+0000 7f619e600700 1 -- 172.21.15.89:0/1874496859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f619815da50 msgr2=0x7f61980ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.072+0000 7f619e600700 1 --2- 172.21.15.89:0/1874496859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f619815da50 0x7f61980ce730 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7f6188009840 tx=0x7f6188009b50 comp rx=0 tx=0).stop 2024-04-02T09:17:04.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.072+0000 7f619e600700 1 -- 172.21.15.89:0/1874496859 shutdown_connections 2024-04-02T09:17:04.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.072+0000 7f619e600700 1 --2- 172.21.15.89:0/1874496859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f619815da50 0x7f61980ce730 unknown :-1 s=CLOSED pgs=3047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.072+0000 7f619e600700 1 -- 172.21.15.89:0/1874496859 >> 172.21.15.89:0/1874496859 conn(0x7f61980666a0 msgr2=0x7f6198066ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:04.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.072+0000 7f619e600700 1 -- 172.21.15.89:0/1874496859 shutdown_connections 2024-04-02T09:17:04.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.072+0000 7f619e600700 1 -- 172.21.15.89:0/1874496859 wait complete. 2024-04-02T09:17:04.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.074+0000 7f619e600700 1 Processor -- start 2024-04-02T09:17:04.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.074+0000 7f619e600700 1 -- start start 2024-04-02T09:17:04.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.075+0000 7f619e600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61980cec70 0x7f61980cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.075+0000 7f619e600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61980d0cf0 0x7f619815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.075+0000 7f619e600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f619815da50 0x7f61980ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.075+0000 7f619e600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61980cad10 con 0x7f61980d0cf0 2024-04-02T09:17:04.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.075+0000 7f619e600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61980caff0 con 0x7f61980cec70 2024-04-02T09:17:04.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.075+0000 7f619e600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61980cb2d0 con 0x7f619815da50 2024-04-02T09:17:04.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.075+0000 7f6196a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61980d0cf0 0x7f619815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:04.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.075+0000 7f6197400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61980cec70 0x7f61980cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:04.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.075+0000 7f6197400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61980cec70 0x7f61980cbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:48618/0 (socket says 172.21.15.89:48618) 2024-04-02T09:17:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.075+0000 7f6196a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61980d0cf0 0x7f619815fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56194/0 (socket says 172.21.15.89:56194) 2024-04-02T09:17:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.075+0000 7f6197400700 1 -- 172.21.15.89:0/4103014617 learned_addr learned my addr 172.21.15.89:0/4103014617 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.075+0000 7f6197e00700 1 --2- 172.21.15.89:0/4103014617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f619815da50 0x7f61980ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.076+0000 7f6196a00700 1 -- 172.21.15.89:0/4103014617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f619815da50 msgr2=0x7f61980ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.076+0000 7f6196a00700 1 --2- 172.21.15.89:0/4103014617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f619815da50 0x7f61980ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.076+0000 7f6196a00700 1 -- 172.21.15.89:0/4103014617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61980cec70 msgr2=0x7f61980cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.076+0000 7f6196a00700 1 --2- 172.21.15.89:0/4103014617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61980cec70 0x7f61980cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.076+0000 7f6196a00700 1 -- 172.21.15.89:0/4103014617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61880094a0 con 0x7f61980d0cf0 2024-04-02T09:17:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.076+0000 7f6197400700 1 --2- 172.21.15.89:0/4103014617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61980cec70 0x7f61980cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:17:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.076+0000 7f6196a00700 1 --2- 172.21.15.89:0/4103014617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61980d0cf0 0x7f619815fe30 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f618400c420 tx=0x7f618400c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.076+0000 7f618f400700 1 -- 172.21.15.89:0/4103014617 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184005cb0 con 0x7f61980d0cf0 2024-04-02T09:17:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.077+0000 7f618f400700 1 -- 172.21.15.89:0/4103014617 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6184005e10 con 0x7f61980d0cf0 2024-04-02T09:17:04.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.077+0000 7f618f400700 1 -- 172.21.15.89:0/4103014617 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61840072d0 con 0x7f61980d0cf0 2024-04-02T09:17:04.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.077+0000 7f619e600700 1 -- 172.21.15.89:0/4103014617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61980cb5b0 con 0x7f61980d0cf0 2024-04-02T09:17:04.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.077+0000 7f619e600700 1 -- 172.21.15.89:0/4103014617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6198149440 con 0x7f61980d0cf0 2024-04-02T09:17:04.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.078+0000 7f619e600700 1 -- 172.21.15.89:0/4103014617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61980cbad0 con 0x7f61980d0cf0 2024-04-02T09:17:04.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.079+0000 7f618f400700 1 -- 172.21.15.89:0/4103014617 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f6184007430 con 0x7f61980d0cf0 2024-04-02T09:17:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.080+0000 7f618f400700 1 --2- 172.21.15.89:0/4103014617 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f617806af60 0x7f617806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.080+0000 7f618f400700 1 -- 172.21.15.89:0/4103014617 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f6184088c20 con 0x7f61980d0cf0 2024-04-02T09:17:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.080+0000 7f6197400700 1 -- 172.21.15.89:0/4103014617 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f617806af60 msgr2=0x7f617806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:04.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.080+0000 7f6197400700 1 --2- 172.21.15.89:0/4103014617 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f617806af60 0x7f617806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:17:04.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.085+0000 7f618f400700 1 -- 172.21.15.89:0/4103014617 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6184053840 con 0x7f61980d0cf0 2024-04-02T09:17:04.147 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:04.148+0000 7fdd2c137240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T09:17:04.200 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:04.202+0000 7f0061d0f240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:17:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.281+0000 7f6197400700 1 -- 172.21.15.89:0/4103014617 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f617806af60 msgr2=0x7f617806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:04.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.281+0000 7f6197400700 1 --2- 172.21.15.89:0/4103014617 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f617806af60 0x7f617806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:17:04.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.296+0000 7f619e600700 1 -- 172.21.15.89:0/4103014617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6198067720 con 0x7f61980d0cf0 2024-04-02T09:17:04.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.298+0000 7f618f400700 1 -- 172.21.15.89:0/4103014617 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v280) v1 ==== 74+0+17226 (secure 0 0 0) 0x7f6184017020 con 0x7f61980d0cf0 2024-04-02T09:17:04.297 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:17:04.297 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":280,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:15:43.128368+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:15:24.345253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-04-02T09:15:25.756381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"279","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/2320969126":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/2687295125":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/693914703":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/1405191830":"2024-04-03T09:15:43.128314+0000","172.21.15.89:6833/108855":"2024-04-03T09:15:43.128314+0000","172.21.15.89:6832/108855":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/701778740":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/799882283":"2024-04-03T09:15:43.128314+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:17:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.300+0000 7f619e600700 1 -- 172.21.15.89:0/4103014617 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f617806af60 msgr2=0x7f617806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:17:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.300+0000 7f619e600700 1 --2- 172.21.15.89:0/4103014617 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f617806af60 0x7f617806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.300+0000 7f619e600700 1 -- 172.21.15.89:0/4103014617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61980d0cf0 msgr2=0x7f619815fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.300+0000 7f619e600700 1 --2- 172.21.15.89:0/4103014617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61980d0cf0 0x7f619815fe30 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f618400c420 tx=0x7f618400c730 comp rx=0 tx=0).stop 2024-04-02T09:17:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.300+0000 7f619e600700 1 -- 172.21.15.89:0/4103014617 shutdown_connections 2024-04-02T09:17:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.300+0000 7f619e600700 1 --2- 172.21.15.89:0/4103014617 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f617806af60 0x7f617806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.300+0000 7f619e600700 1 --2- 172.21.15.89:0/4103014617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61980cec70 0x7f61980cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.300+0000 7f619e600700 1 --2- 172.21.15.89:0/4103014617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f619815da50 0x7f61980ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.300+0000 7f619e600700 1 --2- 172.21.15.89:0/4103014617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61980d0cf0 0x7f619815fe30 unknown :-1 s=CLOSED pgs=3048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.300+0000 7f619e600700 1 -- 172.21.15.89:0/4103014617 >> 172.21.15.89:0/4103014617 conn(0x7f61980666a0 msgr2=0x7f61980b52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.300+0000 7f619e600700 1 -- 172.21.15.89:0/4103014617 shutdown_connections 2024-04-02T09:17:04.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.300+0000 7f619e600700 1 -- 172.21.15.89:0/4103014617 wait complete. 2024-04-02T09:17:04.319 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T07:35:41.237970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T09:15:24.345253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '278', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T09:15:25.756381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '279', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T09:17:04.320 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T09:17:04.373 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:04.374+0000 7fdd2c137240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T09:17:04.458 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:04.460+0000 7f0061d0f240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:17:04.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.523+0000 7f124fe00700 1 Processor -- start 2024-04-02T09:17:04.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.523+0000 7f124fe00700 1 -- start start 2024-04-02T09:17:04.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.524+0000 7f124fe00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f124815dad0 0x7f12480ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.524+0000 7f124fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12480d1630 0x7f12480d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.524+0000 7f124fe00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124815f820 con 0x7f12480d1fd0 2024-04-02T09:17:04.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.524+0000 7f124fe00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12480cfcc0 con 0x7f124815dad0 2024-04-02T09:17:04.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.524+0000 7f124fe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12480cd9f0 con 0x7f12480d1630 2024-04-02T09:17:04.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.525+0000 7f124de00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f12480d1fd0 0x7f124815deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50994/0 (socket says 172.21.15.89:50994) 2024-04-02T09:17:04.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.525+0000 7f124de00700 1 -- 172.21.15.89:0/3965756513 learned_addr learned my addr 172.21.15.89:0/3965756513 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:04.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.525+0000 7f124ca00700 1 --2- 172.21.15.89:0/3965756513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12480d1630 0x7f12480d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:04.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.525+0000 7f124d400700 1 --2- 172.21.15.89:0/3965756513 >> v2:172.21.15.113:3300/0 conn(0x7f124815dad0 0x7f12480ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:04.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.525+0000 7f1247e00700 1 -- 172.21.15.89:0/3965756513 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352551077 0 0) 0x7f124815f820 con 0x7f12480d1fd0 2024-04-02T09:17:04.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.525+0000 7f1247e00700 1 -- 172.21.15.89:0/3965756513 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12300035b0 con 0x7f12480d1fd0 2024-04-02T09:17:04.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.525+0000 7f124ca00700 1 -- 172.21.15.89:0/3965756513 >> v2:172.21.15.113:3300/0 conn(0x7f124815dad0 msgr2=0x7f12480ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.525+0000 7f124ca00700 1 --2- 172.21.15.89:0/3965756513 >> v2:172.21.15.113:3300/0 conn(0x7f124815dad0 0x7f12480ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.525+0000 7f124ca00700 1 -- 172.21.15.89:0/3965756513 >> v1:172.21.15.89:6789/0 conn(0x7f12480d1fd0 legacy=0x7f124815deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.525+0000 7f124ca00700 1 -- 172.21.15.89:0/3965756513 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12480d2460 con 0x7f12480d1630 2024-04-02T09:17:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.525+0000 7f124d400700 1 --2- 172.21.15.89:0/3965756513 >> v2:172.21.15.113:3300/0 conn(0x7f124815dad0 0x7f12480ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:17:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.526+0000 7f124ca00700 1 --2- 172.21.15.89:0/3965756513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12480d1630 0x7f12480d1a90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1234009840 tx=0x7f1234009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f652aa97813c569 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.526+0000 7f1247e00700 1 -- 172.21.15.89:0/3965756513 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234011830 con 0x7f12480d1630 2024-04-02T09:17:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.526+0000 7f1247e00700 1 -- 172.21.15.89:0/3965756513 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1234011990 con 0x7f12480d1630 2024-04-02T09:17:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.526+0000 7f1247e00700 1 -- 172.21.15.89:0/3965756513 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234011d20 con 0x7f12480d1630 2024-04-02T09:17:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.526+0000 7f124fe00700 1 -- 172.21.15.89:0/3965756513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12480d1630 msgr2=0x7f12480d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.526+0000 7f124fe00700 1 --2- 172.21.15.89:0/3965756513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12480d1630 0x7f12480d1a90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1234009840 tx=0x7f1234009b50 comp rx=0 tx=0).stop 2024-04-02T09:17:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.527+0000 7f124fe00700 1 -- 172.21.15.89:0/3965756513 shutdown_connections 2024-04-02T09:17:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.527+0000 7f124fe00700 1 --2- 172.21.15.89:0/3965756513 >> v2:172.21.15.113:3300/0 conn(0x7f124815dad0 0x7f12480ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.527+0000 7f124fe00700 1 --2- 172.21.15.89:0/3965756513 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12480d1630 0x7f12480d1a90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.527+0000 7f124fe00700 1 -- 172.21.15.89:0/3965756513 >> 172.21.15.89:0/3965756513 conn(0x7f124808aff0 msgr2=0x7f12480d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:04.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.527+0000 7f124fe00700 1 -- 172.21.15.89:0/3965756513 shutdown_connections 2024-04-02T09:17:04.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.527+0000 7f124fe00700 1 -- 172.21.15.89:0/3965756513 wait complete. 2024-04-02T09:17:04.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.529+0000 7f124fe00700 1 Processor -- start 2024-04-02T09:17:04.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.529+0000 7f124fe00700 1 -- start start 2024-04-02T09:17:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.530+0000 7f124fe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480d1630 0x7f12481f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.530+0000 7f124fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12480d1fd0 0x7f12480c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f124815dad0 0x7f12480c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124fe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12480cb8c0 con 0x7f12480d1fd0 2024-04-02T09:17:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124fe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12480c9b50 con 0x7f12480d1630 2024-04-02T09:17:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124fe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12480c9e30 con 0x7f124815dad0 2024-04-02T09:17:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12480d1fd0 0x7f12480c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12480d1fd0 0x7f12480c8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56200/0 (socket says 172.21.15.89:56200) 2024-04-02T09:17:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124ca00700 1 -- 172.21.15.89:0/3626710740 learned_addr learned my addr 172.21.15.89:0/3626710740 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480d1630 0x7f12481f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124de00700 1 --2- 172.21.15.89:0/3626710740 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f124815dad0 0x7f12480c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124ca00700 1 -- 172.21.15.89:0/3626710740 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f124815dad0 msgr2=0x7f12480c94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124ca00700 1 --2- 172.21.15.89:0/3626710740 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f124815dad0 0x7f12480c94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124ca00700 1 -- 172.21.15.89:0/3626710740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480d1630 msgr2=0x7f12481f2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124ca00700 1 --2- 172.21.15.89:0/3626710740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480d1630 0x7f12481f2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124ca00700 1 -- 172.21.15.89:0/3626710740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123c0094a0 con 0x7f12480d1fd0 2024-04-02T09:17:04.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.531+0000 7f124d400700 1 --2- 172.21.15.89:0/3626710740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480d1630 0x7f12481f2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:17:04.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.532+0000 7f124ca00700 1 --2- 172.21.15.89:0/3626710740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12480d1fd0 0x7f12480c8f90 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7f12340090b0 tx=0x7f1234004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:04.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.532+0000 7f1245600700 1 -- 172.21.15.89:0/3626710740 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234004590 con 0x7f12480d1fd0 2024-04-02T09:17:04.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.532+0000 7f1245600700 1 -- 172.21.15.89:0/3626710740 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123401f070 con 0x7f12480d1fd0 2024-04-02T09:17:04.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.532+0000 7f124fe00700 1 -- 172.21.15.89:0/3626710740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12340094a0 con 0x7f12480d1fd0 2024-04-02T09:17:04.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.532+0000 7f1245600700 1 -- 172.21.15.89:0/3626710740 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12340038e0 con 0x7f12480d1fd0 2024-04-02T09:17:04.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.532+0000 7f124fe00700 1 -- 172.21.15.89:0/3626710740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1248150e30 con 0x7f12480d1fd0 2024-04-02T09:17:04.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.533+0000 7f124fe00700 1 -- 172.21.15.89:0/3626710740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12480ca350 con 0x7f12480d1fd0 2024-04-02T09:17:04.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.535+0000 7f1245600700 1 -- 172.21.15.89:0/3626710740 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f1234003a40 con 0x7f12480d1fd0 2024-04-02T09:17:04.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.536+0000 7f1245600700 1 --2- 172.21.15.89:0/3626710740 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f122806af60 0x7f122806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.536+0000 7f1245600700 1 -- 172.21.15.89:0/3626710740 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f1234021070 con 0x7f12480d1fd0 2024-04-02T09:17:04.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.536+0000 7f124d400700 1 -- 172.21.15.89:0/3626710740 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f122806af60 msgr2=0x7f122806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:04.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.536+0000 7f124d400700 1 --2- 172.21.15.89:0/3626710740 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f122806af60 0x7f122806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:17:04.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.541+0000 7f1245600700 1 -- 172.21.15.89:0/3626710740 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f12480ca350 con 0x7f12480d1fd0 2024-04-02T09:17:04.641 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:04.641+0000 7fdd2c137240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T09:17:04.651 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:04.653+0000 7f0061d0f240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:17:04.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.737+0000 7f124d400700 1 -- 172.21.15.89:0/3626710740 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f122806af60 msgr2=0x7f122806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:04.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.737+0000 7f124d400700 1 --2- 172.21.15.89:0/3626710740 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f122806af60 0x7f122806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:17:04.744 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:04.745+0000 7f0061d0f240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:17:04.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.746+0000 7f124fe00700 1 -- 172.21.15.89:0/3626710740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f12480cba00 con 0x7f12480d1fd0 2024-04-02T09:17:04.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.747+0000 7f1245600700 1 -- 172.21.15.89:0/3626710740 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v280) v1 ==== 93+0+10 (secure 0 0 0) 0x7f12480cba00 con 0x7f12480d1fd0 2024-04-02T09:17:04.746 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-04-02T09:17:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.749+0000 7f124fe00700 1 -- 172.21.15.89:0/3626710740 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f122806af60 msgr2=0x7f122806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:17:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.749+0000 7f124fe00700 1 --2- 172.21.15.89:0/3626710740 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f122806af60 0x7f122806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.749+0000 7f124fe00700 1 -- 172.21.15.89:0/3626710740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12480d1fd0 msgr2=0x7f12480c8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.749+0000 7f124fe00700 1 --2- 172.21.15.89:0/3626710740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12480d1fd0 0x7f12480c8f90 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7f12340090b0 tx=0x7f1234004750 comp rx=0 tx=0).stop 2024-04-02T09:17:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.749+0000 7f124fe00700 1 -- 172.21.15.89:0/3626710740 shutdown_connections 2024-04-02T09:17:04.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.749+0000 7f124fe00700 1 --2- 172.21.15.89:0/3626710740 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f122806af60 0x7f122806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.749+0000 7f124fe00700 1 --2- 172.21.15.89:0/3626710740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480d1630 0x7f12481f2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.749+0000 7f124fe00700 1 --2- 172.21.15.89:0/3626710740 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f124815dad0 0x7f12480c94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.749+0000 7f124fe00700 1 --2- 172.21.15.89:0/3626710740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12480d1fd0 0x7f12480c8f90 unknown :-1 s=CLOSED pgs=3050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.749+0000 7f124fe00700 1 -- 172.21.15.89:0/3626710740 >> 172.21.15.89:0/3626710740 conn(0x7f124808aff0 msgr2=0x7f124806cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:04.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.750+0000 7f124fe00700 1 -- 172.21.15.89:0/3626710740 shutdown_connections 2024-04-02T09:17:04.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.750+0000 7f124fe00700 1 -- 172.21.15.89:0/3626710740 wait complete. 2024-04-02T09:17:04.767 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T09:17:04.842 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:04.842+0000 7fdd2c137240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T09:17:04.942 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:04.942+0000 7fdd2c137240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T09:17:04.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.966+0000 7f5466200700 1 Processor -- start 2024-04-02T09:17:04.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.967+0000 7f5466200700 1 -- start start 2024-04-02T09:17:04.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.968+0000 7f5466200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f546015dad0 0x7f54600ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.968+0000 7f5466200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54600d21d0 0x7f546015fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.968+0000 7f5466200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f546015f820 con 0x7f54600d1630 2024-04-02T09:17:04.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.968+0000 7f5466200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54600cfcc0 con 0x7f546015dad0 2024-04-02T09:17:04.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.968+0000 7f5466200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54600cd9f0 con 0x7f54600d21d0 2024-04-02T09:17:04.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.968+0000 7f545f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f546015dad0 0x7f54600ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:04.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.968+0000 7f545f400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f546015dad0 0x7f54600ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56210/0 (socket says 172.21.15.89:56210) 2024-04-02T09:17:04.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.968+0000 7f545f400700 1 -- 172.21.15.89:0/3818268776 learned_addr learned my addr 172.21.15.89:0/3818268776 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:04.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.968+0000 7f545fe00700 1 --2- 172.21.15.89:0/3818268776 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54600d21d0 0x7f546015fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.969+0000 7f545f400700 1 -- 172.21.15.89:0/3818268776 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54600d21d0 msgr2=0x7f546015fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.969+0000 7f545f400700 1 --2- 172.21.15.89:0/3818268776 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54600d21d0 0x7f546015fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.969+0000 7f545f400700 1 -- 172.21.15.89:0/3818268776 >> v1:172.21.15.113:6789/0 conn(0x7f54600d1630 legacy=0x7f54600d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.969+0000 7f545f400700 1 -- 172.21.15.89:0/3818268776 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54600cf420 con 0x7f546015dad0 2024-04-02T09:17:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.969+0000 7f545ea00700 1 -- 172.21.15.89:0/3818268776 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283564171 0 0) 0x7f546015f820 con 0x7f54600d1630 2024-04-02T09:17:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.969+0000 7f545f400700 1 --2- 172.21.15.89:0/3818268776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f546015dad0 0x7f54600ced80 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7f54500098f0 tx=0x7f5450009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6686e464aa10e3d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.969+0000 7f545ea00700 1 -- 172.21.15.89:0/3818268776 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5450011830 con 0x7f546015dad0 2024-04-02T09:17:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.969+0000 7f545ea00700 1 -- 172.21.15.89:0/3818268776 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5450011990 con 0x7f546015dad0 2024-04-02T09:17:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.969+0000 7f545ea00700 1 -- 172.21.15.89:0/3818268776 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5450011d20 con 0x7f546015dad0 2024-04-02T09:17:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.970+0000 7f5466200700 1 -- 172.21.15.89:0/3818268776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f546015dad0 msgr2=0x7f54600ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.970+0000 7f5466200700 1 --2- 172.21.15.89:0/3818268776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f546015dad0 0x7f54600ced80 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7f54500098f0 tx=0x7f5450009c00 comp rx=0 tx=0).stop 2024-04-02T09:17:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.970+0000 7f5466200700 1 -- 172.21.15.89:0/3818268776 shutdown_connections 2024-04-02T09:17:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.970+0000 7f5466200700 1 --2- 172.21.15.89:0/3818268776 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f54600d21d0 0x7f546015fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.970+0000 7f5466200700 1 --2- 172.21.15.89:0/3818268776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f546015dad0 0x7f54600ced80 unknown :-1 s=CLOSED pgs=3051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.970+0000 7f5466200700 1 -- 172.21.15.89:0/3818268776 >> 172.21.15.89:0/3818268776 conn(0x7f546008aff0 msgr2=0x7f54600d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:04.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.970+0000 7f5466200700 1 -- 172.21.15.89:0/3818268776 shutdown_connections 2024-04-02T09:17:04.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.970+0000 7f5466200700 1 -- 172.21.15.89:0/3818268776 wait complete. 2024-04-02T09:17:04.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.973+0000 7f5466200700 1 Processor -- start 2024-04-02T09:17:04.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.973+0000 7f5466200700 1 -- start start 2024-04-02T09:17:04.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f5466200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54600d1630 0x7f54601f7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f5466200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54600d21d0 0x7f5460141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f5466200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f546015dad0 0x7f546006ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f5466200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5460141f70 con 0x7f54600d21d0 2024-04-02T09:17:04.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f5466200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54600b8e20 con 0x7f54600d1630 2024-04-02T09:17:04.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f5466200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54600b9100 con 0x7f546015dad0 2024-04-02T09:17:04.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f5457e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54600d21d0 0x7f5460141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:04.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f5457e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54600d21d0 0x7f5460141a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56224/0 (socket says 172.21.15.89:56224) 2024-04-02T09:17:04.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f5457e00700 1 -- 172.21.15.89:0/3590319025 learned_addr learned my addr 172.21.15.89:0/3590319025 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:04.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f545f400700 1 --2- 172.21.15.89:0/3590319025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54600d1630 0x7f54601f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:04.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f545fe00700 1 --2- 172.21.15.89:0/3590319025 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f546015dad0 0x7f546006ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:04.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f5457e00700 1 -- 172.21.15.89:0/3590319025 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f546015dad0 msgr2=0x7f546006ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f5457e00700 1 --2- 172.21.15.89:0/3590319025 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f546015dad0 0x7f546006ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f5457e00700 1 -- 172.21.15.89:0/3590319025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54600d1630 msgr2=0x7f54601f7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:04.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f5457e00700 1 --2- 172.21.15.89:0/3590319025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54600d1630 0x7f54601f7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:04.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.975+0000 7f5457e00700 1 -- 172.21.15.89:0/3590319025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5450009550 con 0x7f54600d21d0 2024-04-02T09:17:04.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.976+0000 7f5457e00700 1 --2- 172.21.15.89:0/3590319025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54600d21d0 0x7f5460141a30 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f54480011e0 tx=0x7f544800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:04.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.976+0000 7f5457400700 1 -- 172.21.15.89:0/3590319025 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54480054e0 con 0x7f54600d21d0 2024-04-02T09:17:04.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.976+0000 7f5457400700 1 -- 172.21.15.89:0/3590319025 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5448003680 con 0x7f54600d21d0 2024-04-02T09:17:04.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.976+0000 7f5457400700 1 -- 172.21.15.89:0/3590319025 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5448003a30 con 0x7f54600d21d0 2024-04-02T09:17:04.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.976+0000 7f5466200700 1 -- 172.21.15.89:0/3590319025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54601f6c60 con 0x7f54600d21d0 2024-04-02T09:17:04.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.976+0000 7f5466200700 1 -- 172.21.15.89:0/3590319025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54601f7220 con 0x7f54600d21d0 2024-04-02T09:17:04.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.977+0000 7f5466200700 1 -- 172.21.15.89:0/3590319025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54601f7430 con 0x7f54600d21d0 2024-04-02T09:17:04.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.978+0000 7f5457400700 1 -- 172.21.15.89:0/3590319025 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f5448003b90 con 0x7f54600d21d0 2024-04-02T09:17:04.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.979+0000 7f5457400700 1 --2- 172.21.15.89:0/3590319025 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f543c06af60 0x7f543c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:04.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.979+0000 7f545f400700 1 -- 172.21.15.89:0/3590319025 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f543c06af60 msgr2=0x7f543c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:04.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.980+0000 7f5457400700 1 -- 172.21.15.89:0/3590319025 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f5448088f10 con 0x7f54600d21d0 2024-04-02T09:17:04.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.980+0000 7f545f400700 1 --2- 172.21.15.89:0/3590319025 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f543c06af60 0x7f543c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:17:04.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:04.983+0000 7f5457400700 1 -- 172.21.15.89:0/3590319025 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5448053a40 con 0x7f54600d21d0 2024-04-02T09:17:05.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.181+0000 7f545f400700 1 -- 172.21.15.89:0/3590319025 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f543c06af60 msgr2=0x7f543c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:05.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.181+0000 7f545f400700 1 --2- 172.21.15.89:0/3590319025 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f543c06af60 0x7f543c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:17:05.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.215+0000 7f5466200700 1 -- 172.21.15.89:0/3590319025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5460067720 con 0x7f54600d21d0 2024-04-02T09:17:05.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.215+0000 7f5457400700 1 -- 172.21.15.89:0/3590319025 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v280) v1 ==== 104+0+11 (secure 0 0 0) 0x7f54480170a0 con 0x7f54600d21d0 2024-04-02T09:17:05.215 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-04-02T09:17:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.218+0000 7f5466200700 1 -- 172.21.15.89:0/3590319025 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f543c06af60 msgr2=0x7f543c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:17:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.218+0000 7f5466200700 1 --2- 172.21.15.89:0/3590319025 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f543c06af60 0x7f543c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.218+0000 7f5466200700 1 -- 172.21.15.89:0/3590319025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54600d21d0 msgr2=0x7f5460141a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.218+0000 7f5466200700 1 --2- 172.21.15.89:0/3590319025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54600d21d0 0x7f5460141a30 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f54480011e0 tx=0x7f544800ca30 comp rx=0 tx=0).stop 2024-04-02T09:17:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.218+0000 7f5466200700 1 -- 172.21.15.89:0/3590319025 shutdown_connections 2024-04-02T09:17:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.218+0000 7f5466200700 1 --2- 172.21.15.89:0/3590319025 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f543c06af60 0x7f543c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.218+0000 7f5466200700 1 --2- 172.21.15.89:0/3590319025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54600d1630 0x7f54601f7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.218+0000 7f5466200700 1 --2- 172.21.15.89:0/3590319025 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f546015dad0 0x7f546006ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.218+0000 7f5466200700 1 --2- 172.21.15.89:0/3590319025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54600d21d0 0x7f5460141a30 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.218+0000 7f5466200700 1 -- 172.21.15.89:0/3590319025 >> 172.21.15.89:0/3590319025 conn(0x7f546008aff0 msgr2=0x7f54600cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.218+0000 7f5466200700 1 -- 172.21.15.89:0/3590319025 shutdown_connections 2024-04-02T09:17:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.218+0000 7f5466200700 1 -- 172.21.15.89:0/3590319025 wait complete. 2024-04-02T09:17:05.235 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T09:17:05.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.435+0000 7fbe8a600700 1 Processor -- start 2024-04-02T09:17:05.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.435+0000 7fbe8a600700 1 -- start start 2024-04-02T09:17:05.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.436+0000 7fbe8a600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbe840d16a0 0x7fbe840d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:05.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.436+0000 7fbe8a600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe840d2370 0x7fbe840ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:05.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.436+0000 7fbe8a600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe84161420 con 0x7fbe8415d9d0 2024-04-02T09:17:05.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.436+0000 7fbe8a600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe840d2820 con 0x7fbe840d16a0 2024-04-02T09:17:05.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.436+0000 7fbe8a600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe840d4340 con 0x7fbe840d2370 2024-04-02T09:17:05.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.437+0000 7fbe82a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbe840d16a0 0x7fbe840d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:05.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.437+0000 7fbe83400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbe8415d9d0 0x7fbe840d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51002/0 (socket says 172.21.15.89:51002) 2024-04-02T09:17:05.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.437+0000 7fbe83400700 1 -- 172.21.15.89:0/555719544 learned_addr learned my addr 172.21.15.89:0/555719544 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:05.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.437+0000 7fbe83e00700 1 --2- 172.21.15.89:0/555719544 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe840d2370 0x7fbe840ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:05.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.437+0000 7fbe82a00700 1 -- 172.21.15.89:0/555719544 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe840d2370 msgr2=0x7fbe840ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:05.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.437+0000 7fbe82a00700 1 --2- 172.21.15.89:0/555719544 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe840d2370 0x7fbe840ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.437+0000 7fbe82a00700 1 -- 172.21.15.89:0/555719544 >> v1:172.21.15.89:6789/0 conn(0x7fbe8415d9d0 legacy=0x7fbe840d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:05.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.437+0000 7fbe82a00700 1 -- 172.21.15.89:0/555719544 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe84161700 con 0x7fbe840d16a0 2024-04-02T09:17:05.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.437+0000 7fbe82000700 1 -- 172.21.15.89:0/555719544 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937679547 0 0) 0x7fbe84161420 con 0x7fbe8415d9d0 2024-04-02T09:17:05.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.437+0000 7fbe82a00700 1 --2- 172.21.15.89:0/555719544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe840d16a0 0x7fbe840d1e30 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009820 tx=0x7fbe70009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f0e1790305ea7d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:05.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.438+0000 7fbe82000700 1 -- 172.21.15.89:0/555719544 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70005e30 con 0x7fbe840d16a0 2024-04-02T09:17:05.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.438+0000 7fbe82000700 1 -- 172.21.15.89:0/555719544 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe70006030 con 0x7fbe840d16a0 2024-04-02T09:17:05.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.438+0000 7fbe82000700 1 -- 172.21.15.89:0/555719544 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe700063c0 con 0x7fbe840d16a0 2024-04-02T09:17:05.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.438+0000 7fbe8a600700 1 -- 172.21.15.89:0/555719544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe840d16a0 msgr2=0x7fbe840d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:05.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.438+0000 7fbe8a600700 1 --2- 172.21.15.89:0/555719544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe840d16a0 0x7fbe840d1e30 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009820 tx=0x7fbe70009b30 comp rx=0 tx=0).stop 2024-04-02T09:17:05.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.438+0000 7fbe8a600700 1 -- 172.21.15.89:0/555719544 shutdown_connections 2024-04-02T09:17:05.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.438+0000 7fbe8a600700 1 --2- 172.21.15.89:0/555719544 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe840d2370 0x7fbe840ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.438+0000 7fbe8a600700 1 --2- 172.21.15.89:0/555719544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe840d16a0 0x7fbe840d1e30 unknown :-1 s=CLOSED pgs=3054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.438+0000 7fbe8a600700 1 -- 172.21.15.89:0/555719544 >> 172.21.15.89:0/555719544 conn(0x7fbe8408aff0 msgr2=0x7fbe840d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:05.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.438+0000 7fbe8a600700 1 -- 172.21.15.89:0/555719544 shutdown_connections 2024-04-02T09:17:05.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.438+0000 7fbe8a600700 1 -- 172.21.15.89:0/555719544 wait complete. 2024-04-02T09:17:05.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.441+0000 7fbe8a600700 1 Processor -- start 2024-04-02T09:17:05.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.441+0000 7fbe8a600700 1 -- start start 2024-04-02T09:17:05.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.442+0000 7fbe8a600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe840d16a0 0x7fbe841559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:05.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.442+0000 7fbe8a600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe840d2370 0x7fbe840b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:05.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.442+0000 7fbe8a600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe8415d9d0 0x7fbe840b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:05.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.442+0000 7fbe8a600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe840b8690 con 0x7fbe840d2370 2024-04-02T09:17:05.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.442+0000 7fbe8a600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe840b8b20 con 0x7fbe8415d9d0 2024-04-02T09:17:05.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.442+0000 7fbe8a600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe840b7a80 con 0x7fbe840d16a0 2024-04-02T09:17:05.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.442+0000 7fbe82a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe840d2370 0x7fbe840b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:05.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.442+0000 7fbe83400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe840d16a0 0x7fbe841559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:05.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.442+0000 7fbe82a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe840d2370 0x7fbe840b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56254/0 (socket says 172.21.15.89:56254) 2024-04-02T09:17:05.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.442+0000 7fbe82a00700 1 -- 172.21.15.89:0/727119973 learned_addr learned my addr 172.21.15.89:0/727119973 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:05.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.442+0000 7fbe83e00700 1 --2- 172.21.15.89:0/727119973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe8415d9d0 0x7fbe840b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:05.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.443+0000 7fbe82a00700 1 -- 172.21.15.89:0/727119973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe840d16a0 msgr2=0x7fbe841559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:05.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.443+0000 7fbe82a00700 1 --2- 172.21.15.89:0/727119973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe840d16a0 0x7fbe841559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.443+0000 7fbe82a00700 1 -- 172.21.15.89:0/727119973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe8415d9d0 msgr2=0x7fbe840b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:05.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.443+0000 7fbe82a00700 1 --2- 172.21.15.89:0/727119973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe8415d9d0 0x7fbe840b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.443+0000 7fbe82a00700 1 -- 172.21.15.89:0/727119973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe70009430 con 0x7fbe840d2370 2024-04-02T09:17:05.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.443+0000 7fbe82a00700 1 --2- 172.21.15.89:0/727119973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe840d2370 0x7fbe840b42e0 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 crypto rx=0x7fbe70000c00 tx=0x7fbe70004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:05.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.443+0000 7fbe7b400700 1 -- 172.21.15.89:0/727119973 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe700061a0 con 0x7fbe840d2370 2024-04-02T09:17:05.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.443+0000 7fbe7b400700 1 -- 172.21.15.89:0/727119973 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe70020070 con 0x7fbe840d2370 2024-04-02T09:17:05.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.443+0000 7fbe8a600700 1 -- 172.21.15.89:0/727119973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe840b7d60 con 0x7fbe840d2370 2024-04-02T09:17:05.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.443+0000 7fbe7b400700 1 -- 172.21.15.89:0/727119973 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70006540 con 0x7fbe840d2370 2024-04-02T09:17:05.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.444+0000 7fbe8a600700 1 -- 172.21.15.89:0/727119973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe840b8320 con 0x7fbe840d2370 2024-04-02T09:17:05.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.445+0000 7fbe8a600700 1 -- 172.21.15.89:0/727119973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe840d1ca0 con 0x7fbe840d2370 2024-04-02T09:17:05.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.446+0000 7fbe7b400700 1 -- 172.21.15.89:0/727119973 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fbe70022030 con 0x7fbe840d2370 2024-04-02T09:17:05.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.446+0000 7fbe7b400700 1 --2- 172.21.15.89:0/727119973 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe6406af60 0x7fbe6406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:05.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.446+0000 7fbe83400700 1 -- 172.21.15.89:0/727119973 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe6406af60 msgr2=0x7fbe6406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:05.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.446+0000 7fbe7b400700 1 -- 172.21.15.89:0/727119973 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fbe7008ae40 con 0x7fbe840d2370 2024-04-02T09:17:05.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.446+0000 7fbe83400700 1 --2- 172.21.15.89:0/727119973 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe6406af60 0x7fbe6406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:17:05.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.450+0000 7fbe7b400700 1 -- 172.21.15.89:0/727119973 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbe70055a40 con 0x7fbe840d2370 2024-04-02T09:17:05.475 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:05.477+0000 7f0061d0f240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:17:05.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.647+0000 7fbe83400700 1 -- 172.21.15.89:0/727119973 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe6406af60 msgr2=0x7fbe6406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:05.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.647+0000 7fbe83400700 1 --2- 172.21.15.89:0/727119973 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe6406af60 0x7fbe6406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:17:05.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.659+0000 7fbe8a600700 1 -- 172.21.15.89:0/727119973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbe84067720 con 0x7fbe840d2370 2024-04-02T09:17:05.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.659+0000 7fbe7b400700 1 -- 172.21.15.89:0/727119973 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v280) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbe70059060 con 0x7fbe840d2370 2024-04-02T09:17:05.658 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-04-02T09:17:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.662+0000 7fbe8a600700 1 -- 172.21.15.89:0/727119973 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe6406af60 msgr2=0x7fbe6406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:17:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.662+0000 7fbe8a600700 1 --2- 172.21.15.89:0/727119973 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe6406af60 0x7fbe6406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.662+0000 7fbe8a600700 1 -- 172.21.15.89:0/727119973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe840d2370 msgr2=0x7fbe840b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.662+0000 7fbe8a600700 1 --2- 172.21.15.89:0/727119973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe840d2370 0x7fbe840b42e0 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 crypto rx=0x7fbe70000c00 tx=0x7fbe70004750 comp rx=0 tx=0).stop 2024-04-02T09:17:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.662+0000 7fbe8a600700 1 -- 172.21.15.89:0/727119973 shutdown_connections 2024-04-02T09:17:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.662+0000 7fbe8a600700 1 --2- 172.21.15.89:0/727119973 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fbe6406af60 0x7fbe6406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.662+0000 7fbe8a600700 1 --2- 172.21.15.89:0/727119973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe8415d9d0 0x7fbe840b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.662+0000 7fbe8a600700 1 --2- 172.21.15.89:0/727119973 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe840d16a0 0x7fbe841559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.662+0000 7fbe8a600700 1 --2- 172.21.15.89:0/727119973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe840d2370 0x7fbe840b42e0 unknown :-1 s=CLOSED pgs=3055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.662+0000 7fbe8a600700 1 -- 172.21.15.89:0/727119973 >> 172.21.15.89:0/727119973 conn(0x7fbe8408aff0 msgr2=0x7fbe840b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.662+0000 7fbe8a600700 1 -- 172.21.15.89:0/727119973 shutdown_connections 2024-04-02T09:17:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.662+0000 7fbe8a600700 1 -- 172.21.15.89:0/727119973 wait complete. 2024-04-02T09:17:05.681 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:17:05.687 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:05.690+0000 7f0061d0f240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:17:05.746 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:05.747+0000 7fdd2c137240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T09:17:05.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.877+0000 7f874fe00700 1 Processor -- start 2024-04-02T09:17:05.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.878+0000 7f874fe00700 1 -- start start 2024-04-02T09:17:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.879+0000 7f874fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8748072580 0x7f87480b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.879+0000 7f874fe00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f87480b2e50 0x7f87480709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.879+0000 7f874fe00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87480b2180 con 0x7f87480b3af0 2024-04-02T09:17:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.879+0000 7f874fe00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874815b100 con 0x7f87480b2e50 2024-04-02T09:17:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.879+0000 7f874fe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874815b3e0 con 0x7f8748072580 2024-04-02T09:17:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.879+0000 7f874de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f87480b2e50 0x7f87480709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.879+0000 7f874de00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f87480b2e50 0x7f87480709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56268/0 (socket says 172.21.15.89:56268) 2024-04-02T09:17:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.879+0000 7f874de00700 1 -- 172.21.15.89:0/2476015642 learned_addr learned my addr 172.21.15.89:0/2476015642 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.879+0000 7f874ca00700 1 --2- 172.21.15.89:0/2476015642 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8748072580 0x7f87480b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.880+0000 7f874de00700 1 -- 172.21.15.89:0/2476015642 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8748072580 msgr2=0x7f87480b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.880+0000 7f874de00700 1 --2- 172.21.15.89:0/2476015642 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8748072580 0x7f87480b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.880+0000 7f874de00700 1 -- 172.21.15.89:0/2476015642 >> v1:172.21.15.113:6789/0 conn(0x7f87480b3af0 legacy=0x7f874815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.880+0000 7f874de00700 1 -- 172.21.15.89:0/2476015642 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87480b23f0 con 0x7f87480b2e50 2024-04-02T09:17:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.880+0000 7f8747e00700 1 -- 172.21.15.89:0/2476015642 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190132278 0 0) 0x7f87480b2180 con 0x7f87480b3af0 2024-04-02T09:17:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.880+0000 7f874de00700 1 --2- 172.21.15.89:0/2476015642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87480b2e50 0x7f87480709c0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f8740009e50 tx=0x7f874000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d04b6adccdf244 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.880+0000 7f8747e00700 1 -- 172.21.15.89:0/2476015642 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874000e070 con 0x7f87480b2e50 2024-04-02T09:17:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.880+0000 7f8747e00700 1 -- 172.21.15.89:0/2476015642 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874000b9c0 con 0x7f87480b2e50 2024-04-02T09:17:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.880+0000 7f8747e00700 1 -- 172.21.15.89:0/2476015642 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874000bd70 con 0x7f87480b2e50 2024-04-02T09:17:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.881+0000 7f874fe00700 1 -- 172.21.15.89:0/2476015642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87480b2e50 msgr2=0x7f87480709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.881+0000 7f874fe00700 1 --2- 172.21.15.89:0/2476015642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87480b2e50 0x7f87480709c0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f8740009e50 tx=0x7f874000b330 comp rx=0 tx=0).stop 2024-04-02T09:17:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.881+0000 7f874fe00700 1 -- 172.21.15.89:0/2476015642 shutdown_connections 2024-04-02T09:17:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.881+0000 7f874fe00700 1 --2- 172.21.15.89:0/2476015642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87480b2e50 0x7f87480709c0 unknown :-1 s=CLOSED pgs=3056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.881+0000 7f874fe00700 1 --2- 172.21.15.89:0/2476015642 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8748072580 0x7f87480b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.881+0000 7f874fe00700 1 -- 172.21.15.89:0/2476015642 >> 172.21.15.89:0/2476015642 conn(0x7f87480cf190 msgr2=0x7f87480cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.881+0000 7f874fe00700 1 -- 172.21.15.89:0/2476015642 shutdown_connections 2024-04-02T09:17:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.881+0000 7f874fe00700 1 -- 172.21.15.89:0/2476015642 wait complete. 2024-04-02T09:17:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.883+0000 7f874fe00700 1 Processor -- start 2024-04-02T09:17:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.883+0000 7f874fe00700 1 -- start start 2024-04-02T09:17:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874fe00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8748072580 0x7f87480c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87480b2e50 0x7f87480cc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874fe00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87480b3af0 0x7f8748146730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874fe00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87480ce220 con 0x7f87480b2e50 2024-04-02T09:17:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874fe00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87480ccaa0 con 0x7f8748072580 2024-04-02T09:17:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874fe00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87480ce450 con 0x7f87480b3af0 2024-04-02T09:17:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87480b2e50 0x7f87480cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874ca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87480b2e50 0x7f87480cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56284/0 (socket says 172.21.15.89:56284) 2024-04-02T09:17:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8748072580 0x7f87480c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874ca00700 1 -- 172.21.15.89:0/1383225989 learned_addr learned my addr 172.21.15.89:0/1383225989 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874de00700 1 --2- 172.21.15.89:0/1383225989 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87480b3af0 0x7f8748146730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874ca00700 1 -- 172.21.15.89:0/1383225989 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87480b3af0 msgr2=0x7f8748146730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874ca00700 1 --2- 172.21.15.89:0/1383225989 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87480b3af0 0x7f8748146730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874ca00700 1 -- 172.21.15.89:0/1383225989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8748072580 msgr2=0x7f87480c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874ca00700 1 --2- 172.21.15.89:0/1383225989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8748072580 0x7f87480c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.884+0000 7f874ca00700 1 -- 172.21.15.89:0/1383225989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8740009b30 con 0x7f87480b2e50 2024-04-02T09:17:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.885+0000 7f874ca00700 1 --2- 172.21.15.89:0/1383225989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87480b2e50 0x7f87480cc560 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f873400e810 tx=0x7f873400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.885+0000 7f8745600700 1 -- 172.21.15.89:0/1383225989 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873400c070 con 0x7f87480b2e50 2024-04-02T09:17:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.885+0000 7f8745600700 1 -- 172.21.15.89:0/1383225989 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8734013610 con 0x7f87480b2e50 2024-04-02T09:17:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.885+0000 7f874fe00700 1 -- 172.21.15.89:0/1383225989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87480ce730 con 0x7f87480b2e50 2024-04-02T09:17:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.885+0000 7f8745600700 1 -- 172.21.15.89:0/1383225989 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87340139c0 con 0x7f87480b2e50 2024-04-02T09:17:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.885+0000 7f874fe00700 1 -- 172.21.15.89:0/1383225989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8748156af0 con 0x7f87480b2e50 2024-04-02T09:17:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.886+0000 7f874fe00700 1 -- 172.21.15.89:0/1383225989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874806b620 con 0x7f87480b2e50 2024-04-02T09:17:05.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.888+0000 7f8745600700 1 -- 172.21.15.89:0/1383225989 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f8734013b20 con 0x7f87480b2e50 2024-04-02T09:17:05.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.889+0000 7f8745600700 1 --2- 172.21.15.89:0/1383225989 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f872806af60 0x7f872806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:05.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.889+0000 7f874d400700 1 -- 172.21.15.89:0/1383225989 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f872806af60 msgr2=0x7f872806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:05.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.889+0000 7f8745600700 1 -- 172.21.15.89:0/1383225989 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f87340899a0 con 0x7f87480b2e50 2024-04-02T09:17:05.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.889+0000 7f874d400700 1 --2- 172.21.15.89:0/1383225989 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f872806af60 0x7f872806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:17:05.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:05.894+0000 7f8745600700 1 -- 172.21.15.89:0/1383225989 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8734054380 con 0x7f87480b2e50 2024-04-02T09:17:05.896 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:05.899+0000 7f0061d0f240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:17:05.975 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:05.976+0000 7fdd2c137240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T09:17:06.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.091+0000 7f874d400700 1 -- 172.21.15.89:0/1383225989 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f872806af60 msgr2=0x7f872806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:06.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.091+0000 7f874d400700 1 --2- 172.21.15.89:0/1383225989 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f872806af60 0x7f872806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:17:06.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.146+0000 7f874fe00700 1 -- 172.21.15.89:0/1383225989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8748156d10 con 0x7f87480b2e50 2024-04-02T09:17:06.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.146+0000 7f8745600700 1 -- 172.21.15.89:0/1383225989 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4753 (secure 0 0 0) 0x7f87340579a0 con 0x7f87480b2e50 2024-04-02T09:17:06.146 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:17:06.146 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10376,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2447701494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2447701494},{"type":"v1","addr":"172.21.15.113:6834","nonce":2447701494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":12103,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/361614750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":361614750},{"type":"v1","addr":"172.21.15.113:6835","nonce":361614750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":13281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2871180587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2871180587},{"type":"v1","addr":"172.21.15.89:6835","nonce":2871180587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:15:26.772241+0000","modified":"2024-04-02T09:15:30.258749+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13308":{"gid":13308,"name":"a","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3853118738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3853118738},{"type":"v1","addr":"172.21.15.89:6837","nonce":3853118738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-04-02T09:17:06.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.149+0000 7f874fe00700 1 -- 172.21.15.89:0/1383225989 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f872806af60 msgr2=0x7f872806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:17:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.149+0000 7f874fe00700 1 --2- 172.21.15.89:0/1383225989 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f872806af60 0x7f872806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.149+0000 7f874fe00700 1 -- 172.21.15.89:0/1383225989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87480b2e50 msgr2=0x7f87480cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.149+0000 7f874fe00700 1 --2- 172.21.15.89:0/1383225989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87480b2e50 0x7f87480cc560 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f873400e810 tx=0x7f873400eb20 comp rx=0 tx=0).stop 2024-04-02T09:17:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.149+0000 7f874fe00700 1 -- 172.21.15.89:0/1383225989 shutdown_connections 2024-04-02T09:17:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.149+0000 7f874fe00700 1 --2- 172.21.15.89:0/1383225989 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f872806af60 0x7f872806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.149+0000 7f874fe00700 1 --2- 172.21.15.89:0/1383225989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8748072580 0x7f87480c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.149+0000 7f874fe00700 1 --2- 172.21.15.89:0/1383225989 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87480b3af0 0x7f8748146730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.149+0000 7f874fe00700 1 --2- 172.21.15.89:0/1383225989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87480b2e50 0x7f87480cc560 unknown :-1 s=CLOSED pgs=3057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.149+0000 7f874fe00700 1 -- 172.21.15.89:0/1383225989 >> 172.21.15.89:0/1383225989 conn(0x7f87480cf190 msgr2=0x7f874815dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.149+0000 7f874fe00700 1 -- 172.21.15.89:0/1383225989 shutdown_connections 2024-04-02T09:17:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.149+0000 7f874fe00700 1 -- 172.21.15.89:0/1383225989 wait complete. 2024-04-02T09:17:06.150 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 144 2024-04-02T09:17:06.167 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:17:06.195 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:06.196+0000 7fdd2c137240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:17:06.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.370+0000 7fb6a6c00700 1 Processor -- start 2024-04-02T09:17:06.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.370+0000 7fb6a6c00700 1 -- start start 2024-04-02T09:17:06.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.371+0000 7fb6a6c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb6a00cb3f0 0x7fb6a00ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:06.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.371+0000 7fb6a6c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a015fd50 con 0x7fb6a015b5d0 2024-04-02T09:17:06.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.371+0000 7fb6a6c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a00aca10 con 0x7fb6a00ac600 2024-04-02T09:17:06.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.371+0000 7fb6a6c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a015b9b0 con 0x7fb6a00cb3f0 2024-04-02T09:17:06.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.371+0000 7fb69fe00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb6a015b5d0 0x7fb6a00ae3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51012/0 (socket says 172.21.15.89:51012) 2024-04-02T09:17:06.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.371+0000 7fb69fe00700 1 -- 172.21.15.89:0/2564682747 learned_addr learned my addr 172.21.15.89:0/2564682747 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:06.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.371+0000 7fb69f400700 1 --2- 172.21.15.89:0/2564682747 >> v2:172.21.15.113:3300/0 conn(0x7fb6a00cb3f0 0x7fb6a00ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:06.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.372+0000 7fb69ea00700 1 -- 172.21.15.89:0/2564682747 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481499420 0 0) 0x7fb6a015fd50 con 0x7fb6a015b5d0 2024-04-02T09:17:06.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.372+0000 7fb69ea00700 1 -- 172.21.15.89:0/2564682747 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6840035b0 con 0x7fb6a015b5d0 2024-04-02T09:17:06.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.372+0000 7fb69ea00700 1 -- 172.21.15.89:0/2564682747 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253060204 0 0) 0x7fb6a00aca10 con 0x7fb6a00ac600 2024-04-02T09:17:06.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.372+0000 7fb69ea00700 1 -- 172.21.15.89:0/2564682747 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a015fd50 con 0x7fb6a00ac600 2024-04-02T09:17:06.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.372+0000 7fb69f400700 1 -- 172.21.15.89:0/2564682747 >> v1:172.21.15.113:6789/0 conn(0x7fb6a00ac600 legacy=0x7fb6a006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:06.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.372+0000 7fb69f400700 1 -- 172.21.15.89:0/2564682747 >> v1:172.21.15.89:6789/0 conn(0x7fb6a015b5d0 legacy=0x7fb6a00ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:06.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.372+0000 7fb69f400700 1 -- 172.21.15.89:0/2564682747 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a015f140 con 0x7fb6a00cb3f0 2024-04-02T09:17:06.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.372+0000 7fb69f400700 1 --2- 172.21.15.89:0/2564682747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6a00cb3f0 0x7fb6a00ac0c0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009840 tx=0x7fb68c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adcff48ba843140 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:06.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.372+0000 7fb69ea00700 1 -- 172.21.15.89:0/2564682747 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c011830 con 0x7fb6a00cb3f0 2024-04-02T09:17:06.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.373+0000 7fb69ea00700 1 -- 172.21.15.89:0/2564682747 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68c011990 con 0x7fb6a00cb3f0 2024-04-02T09:17:06.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.373+0000 7fb69ea00700 1 -- 172.21.15.89:0/2564682747 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c011d20 con 0x7fb6a00cb3f0 2024-04-02T09:17:06.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.373+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2564682747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6a00cb3f0 msgr2=0x7fb6a00ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:06.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.373+0000 7fb6a6c00700 1 --2- 172.21.15.89:0/2564682747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6a00cb3f0 0x7fb6a00ac0c0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009840 tx=0x7fb68c009b50 comp rx=0 tx=0).stop 2024-04-02T09:17:06.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.373+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2564682747 shutdown_connections 2024-04-02T09:17:06.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.373+0000 7fb6a6c00700 1 --2- 172.21.15.89:0/2564682747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6a00cb3f0 0x7fb6a00ac0c0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.373+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2564682747 >> 172.21.15.89:0/2564682747 conn(0x7fb6a00cf190 msgr2=0x7fb6a00cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:06.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.373+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2564682747 shutdown_connections 2024-04-02T09:17:06.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.373+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2564682747 wait complete. 2024-04-02T09:17:06.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.375+0000 7fb6a6c00700 1 Processor -- start 2024-04-02T09:17:06.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.375+0000 7fb6a6c00700 1 -- start start 2024-04-02T09:17:06.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.376+0000 7fb6a6c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6a00ac600 0x7fb6a01f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:06.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.376+0000 7fb6a6c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6a015b5d0 0x7fb6a00c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:06.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.376+0000 7fb6a6c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6a01443e0 0x7fb6a00c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:06.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.376+0000 7fb6a6c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a01f6a50 con 0x7fb6a00ac600 2024-04-02T09:17:06.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.376+0000 7fb6a6c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a00cea80 con 0x7fb6a015b5d0 2024-04-02T09:17:06.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.376+0000 7fb6a6c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a00cde40 con 0x7fb6a01443e0 2024-04-02T09:17:06.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.376+0000 7fb69fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6a00ac600 0x7fb6a01f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:06.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.376+0000 7fb69fe00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6a00ac600 0x7fb6a01f6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56294/0 (socket says 172.21.15.89:56294) 2024-04-02T09:17:06.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.377+0000 7fb69fe00700 1 -- 172.21.15.89:0/2331610261 learned_addr learned my addr 172.21.15.89:0/2331610261 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:06.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.377+0000 7fb69f400700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6a015b5d0 0x7fb6a00c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:06.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.377+0000 7fb6a4c00700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6a01443e0 0x7fb6a00c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:06.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.377+0000 7fb69fe00700 1 -- 172.21.15.89:0/2331610261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6a01443e0 msgr2=0x7fb6a00c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:06.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.377+0000 7fb69fe00700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6a01443e0 0x7fb6a00c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.377+0000 7fb69fe00700 1 -- 172.21.15.89:0/2331610261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6a015b5d0 msgr2=0x7fb6a00c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:06.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.377+0000 7fb69fe00700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6a015b5d0 0x7fb6a00c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.377+0000 7fb69fe00700 1 -- 172.21.15.89:0/2331610261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c0094a0 con 0x7fb6a00ac600 2024-04-02T09:17:06.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.377+0000 7fb6a4c00700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6a01443e0 0x7fb6a00c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:17:06.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.377+0000 7fb69fe00700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6a00ac600 0x7fb6a01f6510 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7fb69000c730 tx=0x7fb69000ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:06.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.377+0000 7fb697e00700 1 -- 172.21.15.89:0/2331610261 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690005d70 con 0x7fb6a00ac600 2024-04-02T09:17:06.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.377+0000 7fb697e00700 1 -- 172.21.15.89:0/2331610261 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6900070d0 con 0x7fb6a00ac600 2024-04-02T09:17:06.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.377+0000 7fb697e00700 1 -- 172.21.15.89:0/2331610261 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690007460 con 0x7fb6a00ac600 2024-04-02T09:17:06.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.377+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2331610261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a00aefb0 con 0x7fb6a00ac600 2024-04-02T09:17:06.379 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:06.380+0000 7f0061d0f240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:17:06.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.378+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2331610261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a00c9790 con 0x7fb6a00ac600 2024-04-02T09:17:06.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.379+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2331610261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a0067720 con 0x7fb6a00ac600 2024-04-02T09:17:06.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.380+0000 7fb697e00700 1 -- 172.21.15.89:0/2331610261 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fb6900075c0 con 0x7fb6a00ac600 2024-04-02T09:17:06.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.381+0000 7fb697e00700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb68006af60 0x7fb68006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:06.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.381+0000 7fb69f400700 1 -- 172.21.15.89:0/2331610261 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb68006af60 msgr2=0x7fb68006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:06.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.381+0000 7fb697e00700 1 -- 172.21.15.89:0/2331610261 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fb690088f90 con 0x7fb6a00ac600 2024-04-02T09:17:06.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.381+0000 7fb69f400700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb68006af60 0x7fb68006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:17:06.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.385+0000 7fb697e00700 1 -- 172.21.15.89:0/2331610261 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb690053b50 con 0x7fb6a00ac600 2024-04-02T09:17:06.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.582+0000 7fb69f400700 1 -- 172.21.15.89:0/2331610261 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb68006af60 msgr2=0x7fb68006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:06.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.582+0000 7fb69f400700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb68006af60 0x7fb68006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:17:06.587 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:06.590+0000 7f0061d0f240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:17:06.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.599+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2331610261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6a006e560 con 0x7fb6a00ac600 2024-04-02T09:17:06.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.600+0000 7fb697e00700 1 -- 172.21.15.89:0/2331610261 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v280) v1 ==== 74+0+17226 (secure 0 0 0) 0x7fb690017090 con 0x7fb6a00ac600 2024-04-02T09:17:06.600 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:17:06.600 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":280,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:15:43.128368+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:15:24.345253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-04-02T09:15:25.756381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"279","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/2320969126":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/2687295125":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/693914703":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/1405191830":"2024-04-03T09:15:43.128314+0000","172.21.15.89:6833/108855":"2024-04-03T09:15:43.128314+0000","172.21.15.89:6832/108855":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/701778740":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/799882283":"2024-04-03T09:15:43.128314+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:17:06.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.603+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2331610261 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb68006af60 msgr2=0x7fb68006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:17:06.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.603+0000 7fb6a6c00700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb68006af60 0x7fb68006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.603+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2331610261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6a00ac600 msgr2=0x7fb6a01f6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:06.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.603+0000 7fb6a6c00700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6a00ac600 0x7fb6a01f6510 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7fb69000c730 tx=0x7fb69000ca40 comp rx=0 tx=0).stop 2024-04-02T09:17:06.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.603+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2331610261 shutdown_connections 2024-04-02T09:17:06.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.603+0000 7fb6a6c00700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6a01443e0 0x7fb6a00c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.603+0000 7fb6a6c00700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6a00ac600 0x7fb6a01f6510 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.603+0000 7fb6a6c00700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb68006af60 0x7fb68006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.603+0000 7fb6a6c00700 1 --2- 172.21.15.89:0/2331610261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6a015b5d0 0x7fb6a00c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.603+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2331610261 >> 172.21.15.89:0/2331610261 conn(0x7fb6a00cf190 msgr2=0x7fb6a00b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:06.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.603+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2331610261 shutdown_connections 2024-04-02T09:17:06.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.603+0000 7fb6a6c00700 1 -- 172.21.15.89:0/2331610261 wait complete. 2024-04-02T09:17:06.621 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T09:17:06.621 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:17:06.707 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:06.707+0000 7fdd2c137240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T09:17:06.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.857+0000 7fcddda00700 1 Processor -- start 2024-04-02T09:17:06.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.857+0000 7fcddda00700 1 -- start start 2024-04-02T09:17:06.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.858+0000 7fcddda00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcdd80b3af0 0x7fcdd815a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.858+0000 7fcddda00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcdd8072580 0x7fcdd80b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.859+0000 7fcddda00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd80b2180 con 0x7fcdd80b2e50 2024-04-02T09:17:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.859+0000 7fcddda00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd815b100 con 0x7fcdd8072580 2024-04-02T09:17:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.859+0000 7fcddda00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd815b3e0 con 0x7fcdd80b3af0 2024-04-02T09:17:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.859+0000 7fcdd6000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcdd8072580 0x7fcdd80b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.859+0000 7fcdd7400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcdd80b2e50 0x7fcdd80709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51028/0 (socket says 172.21.15.89:51028) 2024-04-02T09:17:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.859+0000 7fcdd7400700 1 -- 172.21.15.89:0/967268375 learned_addr learned my addr 172.21.15.89:0/967268375 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.859+0000 7fcdd6a00700 1 --2- 172.21.15.89:0/967268375 >> v2:172.21.15.113:3300/0 conn(0x7fcdd80b3af0 0x7fcdd815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.859+0000 7fcdd6000700 1 -- 172.21.15.89:0/967268375 >> v2:172.21.15.113:3300/0 conn(0x7fcdd80b3af0 msgr2=0x7fcdd815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.859+0000 7fcdd6000700 1 --2- 172.21.15.89:0/967268375 >> v2:172.21.15.113:3300/0 conn(0x7fcdd80b3af0 0x7fcdd815a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.859+0000 7fcdd6000700 1 -- 172.21.15.89:0/967268375 >> v1:172.21.15.89:6789/0 conn(0x7fcdd80b2e50 legacy=0x7fcdd80709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.859+0000 7fcdd6000700 1 -- 172.21.15.89:0/967268375 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd80b23f0 con 0x7fcdd8072580 2024-04-02T09:17:06.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.860+0000 7fcdd6000700 1 --2- 172.21.15.89:0/967268375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd8072580 0x7fcdd80b2910 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7fcdd806b860 tx=0x7fcdc400bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3bdc617513e7e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:06.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.860+0000 7fcdd5600700 1 -- 172.21.15.89:0/967268375 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc400dc70 con 0x7fcdd8072580 2024-04-02T09:17:06.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.860+0000 7fcdd5600700 1 -- 172.21.15.89:0/967268375 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc400ddd0 con 0x7fcdd8072580 2024-04-02T09:17:06.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.860+0000 7fcdd5600700 1 -- 172.21.15.89:0/967268375 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4010690 con 0x7fcdd8072580 2024-04-02T09:17:06.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.861+0000 7fcddda00700 1 -- 172.21.15.89:0/967268375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd8072580 msgr2=0x7fcdd80b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:06.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.861+0000 7fcddda00700 1 --2- 172.21.15.89:0/967268375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd8072580 0x7fcdd80b2910 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7fcdd806b860 tx=0x7fcdc400bdc0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.861+0000 7fcddda00700 1 -- 172.21.15.89:0/967268375 shutdown_connections 2024-04-02T09:17:06.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.861+0000 7fcddda00700 1 --2- 172.21.15.89:0/967268375 >> v2:172.21.15.113:3300/0 conn(0x7fcdd80b3af0 0x7fcdd815a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.861+0000 7fcddda00700 1 --2- 172.21.15.89:0/967268375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd8072580 0x7fcdd80b2910 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.861+0000 7fcddda00700 1 -- 172.21.15.89:0/967268375 >> 172.21.15.89:0/967268375 conn(0x7fcdd80cf190 msgr2=0x7fcdd80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:06.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.861+0000 7fcddda00700 1 -- 172.21.15.89:0/967268375 shutdown_connections 2024-04-02T09:17:06.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.861+0000 7fcddda00700 1 -- 172.21.15.89:0/967268375 wait complete. 2024-04-02T09:17:06.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.863+0000 7fcddda00700 1 Processor -- start 2024-04-02T09:17:06.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.863+0000 7fcddda00700 1 -- start start 2024-04-02T09:17:06.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.864+0000 7fcddda00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd8072580 0x7fcdd80cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:06.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.864+0000 7fcddda00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd80b2e50 0x7fcdd81583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:06.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.864+0000 7fcddda00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdd80b3af0 0x7fcdd8151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.864+0000 7fcddda00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80b0070 con 0x7fcdd8072580 2024-04-02T09:17:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.864+0000 7fcddda00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80b0330 con 0x7fcdd80b2e50 2024-04-02T09:17:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.864+0000 7fcddda00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80b0610 con 0x7fcdd80b3af0 2024-04-02T09:17:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.865+0000 7fcdd6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd8072580 0x7fcdd80cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.865+0000 7fcdd6a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd8072580 0x7fcdd80cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56316/0 (socket says 172.21.15.89:56316) 2024-04-02T09:17:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.865+0000 7fcdd6a00700 1 -- 172.21.15.89:0/1974339898 learned_addr learned my addr 172.21.15.89:0/1974339898 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.865+0000 7fcdd7400700 1 --2- 172.21.15.89:0/1974339898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdd80b3af0 0x7fcdd8151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.865+0000 7fcdd6000700 1 --2- 172.21.15.89:0/1974339898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd80b2e50 0x7fcdd81583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.865+0000 7fcdd6a00700 1 -- 172.21.15.89:0/1974339898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdd80b3af0 msgr2=0x7fcdd8151cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.865+0000 7fcdd6a00700 1 --2- 172.21.15.89:0/1974339898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdd80b3af0 0x7fcdd8151cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.865+0000 7fcdd6a00700 1 -- 172.21.15.89:0/1974339898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd80b2e50 msgr2=0x7fcdd81583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.865+0000 7fcdd6a00700 1 --2- 172.21.15.89:0/1974339898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd80b2e50 0x7fcdd81583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.865+0000 7fcdd6a00700 1 -- 172.21.15.89:0/1974339898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc400b740 con 0x7fcdd8072580 2024-04-02T09:17:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.865+0000 7fcdd6a00700 1 --2- 172.21.15.89:0/1974339898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd8072580 0x7fcdd80cbba0 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800e870 tx=0x7fcdc800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:06.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.866+0000 7fcdcea00700 1 -- 172.21.15.89:0/1974339898 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8009590 con 0x7fcdd8072580 2024-04-02T09:17:06.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.866+0000 7fcdcea00700 1 -- 172.21.15.89:0/1974339898 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc800ee70 con 0x7fcdd8072580 2024-04-02T09:17:06.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.866+0000 7fcddda00700 1 -- 172.21.15.89:0/1974339898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd80b08f0 con 0x7fcdd8072580 2024-04-02T09:17:06.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.866+0000 7fcdcea00700 1 -- 172.21.15.89:0/1974339898 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc800ca50 con 0x7fcdd8072580 2024-04-02T09:17:06.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.866+0000 7fcddda00700 1 -- 172.21.15.89:0/1974339898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd8146550 con 0x7fcdd8072580 2024-04-02T09:17:06.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.867+0000 7fcddda00700 1 -- 172.21.15.89:0/1974339898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd8066ff0 con 0x7fcdd8072580 2024-04-02T09:17:06.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.868+0000 7fcdcea00700 1 -- 172.21.15.89:0/1974339898 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fcdc800cbb0 con 0x7fcdd8072580 2024-04-02T09:17:06.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.869+0000 7fcdcea00700 1 --2- 172.21.15.89:0/1974339898 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcdb806af90 0x7fcdb806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:06.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.869+0000 7fcdcea00700 1 -- 172.21.15.89:0/1974339898 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fcdc8089a40 con 0x7fcdd8072580 2024-04-02T09:17:06.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.869+0000 7fcdd6000700 1 -- 172.21.15.89:0/1974339898 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcdb806af90 msgr2=0x7fcdb806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:06.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.869+0000 7fcdd6000700 1 --2- 172.21.15.89:0/1974339898 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcdb806af90 0x7fcdb806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:17:06.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:06.877+0000 7fcdcea00700 1 -- 172.21.15.89:0/1974339898 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcdc8054570 con 0x7fcdd8072580 2024-04-02T09:17:06.934 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:06.935+0000 7fdd2c137240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T09:17:07.032 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:07.034+0000 7f0061d0f240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:17:07.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.070+0000 7fcdd6000700 1 -- 172.21.15.89:0/1974339898 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcdb806af90 msgr2=0x7fcdb806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:07.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.070+0000 7fcdd6000700 1 --2- 172.21.15.89:0/1974339898 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcdb806af90 0x7fcdb806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T09:17:07.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.134+0000 7fcddda00700 1 -- 172.21.15.89:0/1974339898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdd8067720 con 0x7fcdd8072580 2024-04-02T09:17:07.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.135+0000 7fcdcea00700 1 -- 172.21.15.89:0/1974339898 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4753 (secure 0 0 0) 0x7fcdc800ce60 con 0x7fcdd8072580 2024-04-02T09:17:07.134 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:17:07.134 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10376,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2447701494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2447701494},{"type":"v1","addr":"172.21.15.113:6834","nonce":2447701494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":12103,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/361614750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":361614750},{"type":"v1","addr":"172.21.15.113:6835","nonce":361614750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":13281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2871180587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2871180587},{"type":"v1","addr":"172.21.15.89:6835","nonce":2871180587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:15:26.772241+0000","modified":"2024-04-02T09:15:30.258749+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13308":{"gid":13308,"name":"a","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3853118738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3853118738},{"type":"v1","addr":"172.21.15.89:6837","nonce":3853118738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-04-02T09:17:07.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.137+0000 7fcddda00700 1 -- 172.21.15.89:0/1974339898 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcdb806af90 msgr2=0x7fcdb806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:17:07.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.137+0000 7fcddda00700 1 --2- 172.21.15.89:0/1974339898 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcdb806af90 0x7fcdb806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.137+0000 7fcddda00700 1 -- 172.21.15.89:0/1974339898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd8072580 msgr2=0x7fcdd80cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:07.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.137+0000 7fcddda00700 1 --2- 172.21.15.89:0/1974339898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd8072580 0x7fcdd80cbba0 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800e870 tx=0x7fcdc800ec30 comp rx=0 tx=0).stop 2024-04-02T09:17:07.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.137+0000 7fcddda00700 1 -- 172.21.15.89:0/1974339898 shutdown_connections 2024-04-02T09:17:07.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.137+0000 7fcddda00700 1 --2- 172.21.15.89:0/1974339898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdd80b3af0 0x7fcdd8151cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.137+0000 7fcddda00700 1 --2- 172.21.15.89:0/1974339898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd8072580 0x7fcdd80cbba0 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.137+0000 7fcddda00700 1 --2- 172.21.15.89:0/1974339898 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fcdb806af90 0x7fcdb806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.137+0000 7fcddda00700 1 --2- 172.21.15.89:0/1974339898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd80b2e50 0x7fcdd81583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.137+0000 7fcddda00700 1 -- 172.21.15.89:0/1974339898 >> 172.21.15.89:0/1974339898 conn(0x7fcdd80cf190 msgr2=0x7fcdd806aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:07.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.137+0000 7fcddda00700 1 -- 172.21.15.89:0/1974339898 shutdown_connections 2024-04-02T09:17:07.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.137+0000 7fcddda00700 1 -- 172.21.15.89:0/1974339898 wait complete. 2024-04-02T09:17:07.138 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 144 2024-04-02T09:17:07.143 INFO:tasks.ceph.mgr.x.smithi113.stderr:2024-04-02T09:17:07.145+0000 7f0061d0f240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:17:07.157 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:17:07.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.355+0000 7f6dfca00700 1 Processor -- start 2024-04-02T09:17:07.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.355+0000 7f6dfca00700 1 -- start start 2024-04-02T09:17:07.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.356+0000 7f6dfca00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6df80d18b0 0x7f6df80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:07.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.356+0000 7f6dfca00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df80ceda0 con 0x7f6df8161590 2024-04-02T09:17:07.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.356+0000 7f6dfca00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df81612f0 con 0x7f6df8160f10 2024-04-02T09:17:07.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.356+0000 7f6dfca00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6df80d1d30 con 0x7f6df80d18b0 2024-04-02T09:17:07.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.356+0000 7f6df6800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6df8161590 0x7f6df80b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51034/0 (socket says 172.21.15.89:51034) 2024-04-02T09:17:07.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.356+0000 7f6df5400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6df80d18b0 0x7f6df80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:07.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.356+0000 7f6df6800700 1 -- 172.21.15.89:0/247650082 learned_addr learned my addr 172.21.15.89:0/247650082 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:07.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.357+0000 7f6df4a00700 1 -- 172.21.15.89:0/247650082 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017075709 0 0) 0x7f6df81612f0 con 0x7f6df8160f10 2024-04-02T09:17:07.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.357+0000 7f6df5400700 1 -- 172.21.15.89:0/247650082 >> v1:172.21.15.113:6789/0 conn(0x7f6df8160f10 legacy=0x7f6df80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:07.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.357+0000 7f6df5400700 1 -- 172.21.15.89:0/247650082 >> v1:172.21.15.89:6789/0 conn(0x7f6df8161590 legacy=0x7f6df80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:07.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.357+0000 7f6df5400700 1 -- 172.21.15.89:0/247650082 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df80ce470 con 0x7f6df80d18b0 2024-04-02T09:17:07.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.357+0000 7f6df5400700 1 --2- 172.21.15.89:0/247650082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df80d18b0 0x7f6df80cf230 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009840 tx=0x7f6de4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3243aaef2839a98 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:07.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.357+0000 7f6df4a00700 1 -- 172.21.15.89:0/247650082 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de4011830 con 0x7f6df80d18b0 2024-04-02T09:17:07.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.357+0000 7f6df4a00700 1 -- 172.21.15.89:0/247650082 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de4011990 con 0x7f6df80d18b0 2024-04-02T09:17:07.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.357+0000 7f6df4a00700 1 -- 172.21.15.89:0/247650082 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de4011d20 con 0x7f6df80d18b0 2024-04-02T09:17:07.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.358+0000 7f6dfca00700 1 -- 172.21.15.89:0/247650082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df80d18b0 msgr2=0x7f6df80cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:07.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.358+0000 7f6dfca00700 1 --2- 172.21.15.89:0/247650082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df80d18b0 0x7f6df80cf230 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009840 tx=0x7f6de4009b50 comp rx=0 tx=0).stop 2024-04-02T09:17:07.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.358+0000 7f6dfca00700 1 -- 172.21.15.89:0/247650082 shutdown_connections 2024-04-02T09:17:07.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.358+0000 7f6dfca00700 1 --2- 172.21.15.89:0/247650082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df80d18b0 0x7f6df80cf230 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.358+0000 7f6dfca00700 1 -- 172.21.15.89:0/247650082 >> 172.21.15.89:0/247650082 conn(0x7f6df808aff0 msgr2=0x7f6df80d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:07.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.358+0000 7f6dfca00700 1 -- 172.21.15.89:0/247650082 shutdown_connections 2024-04-02T09:17:07.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.358+0000 7f6dfca00700 1 -- 172.21.15.89:0/247650082 wait complete. 2024-04-02T09:17:07.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.360+0000 7f6dfca00700 1 Processor -- start 2024-04-02T09:17:07.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.360+0000 7f6dfca00700 1 -- start start 2024-04-02T09:17:07.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.361+0000 7f6dfca00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6df80d18b0 0x7f6df8155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:07.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.361+0000 7f6dfca00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8160f10 0x7f6df80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:07.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.361+0000 7f6dfca00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6df8161590 0x7f6df80d0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:07.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.361+0000 7f6dfca00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df81f3e80 con 0x7f6df8160f10 2024-04-02T09:17:07.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.361+0000 7f6dfca00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df80b72b0 con 0x7f6df80d18b0 2024-04-02T09:17:07.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.361+0000 7f6dfca00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df80b7590 con 0x7f6df8161590 2024-04-02T09:17:07.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.361+0000 7f6df5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8160f10 0x7f6df80cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:07.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.361+0000 7f6df5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8160f10 0x7f6df80cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56332/0 (socket says 172.21.15.89:56332) 2024-04-02T09:17:07.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.361+0000 7f6df5400700 1 -- 172.21.15.89:0/956023597 learned_addr learned my addr 172.21.15.89:0/956023597 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:07.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.361+0000 7f6df5e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6df80d18b0 0x7f6df8155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:07.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.361+0000 7f6df6800700 1 --2- 172.21.15.89:0/956023597 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6df8161590 0x7f6df80d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:07.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.362+0000 7f6df5400700 1 -- 172.21.15.89:0/956023597 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6df8161590 msgr2=0x7f6df80d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:07.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.362+0000 7f6df5400700 1 --2- 172.21.15.89:0/956023597 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6df8161590 0x7f6df80d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.362+0000 7f6df5400700 1 -- 172.21.15.89:0/956023597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6df80d18b0 msgr2=0x7f6df8155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:07.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.362+0000 7f6df5400700 1 --2- 172.21.15.89:0/956023597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6df80d18b0 0x7f6df8155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.362+0000 7f6df5400700 1 -- 172.21.15.89:0/956023597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de40094a0 con 0x7f6df8160f10 2024-04-02T09:17:07.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.362+0000 7f6df5400700 1 --2- 172.21.15.89:0/956023597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8160f10 0x7f6df80cfb20 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f6de4014040 tx=0x7f6de4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:07.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.362+0000 7f6dee000700 1 -- 172.21.15.89:0/956023597 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de401a860 con 0x7f6df8160f10 2024-04-02T09:17:07.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.362+0000 7f6dee000700 1 -- 172.21.15.89:0/956023597 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de401a9c0 con 0x7f6df8160f10 2024-04-02T09:17:07.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.362+0000 7f6dfca00700 1 -- 172.21.15.89:0/956023597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df80b7870 con 0x7f6df8160f10 2024-04-02T09:17:07.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.362+0000 7f6dee000700 1 -- 172.21.15.89:0/956023597 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de401ad50 con 0x7f6df8160f10 2024-04-02T09:17:07.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.362+0000 7f6dfca00700 1 -- 172.21.15.89:0/956023597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df81560d0 con 0x7f6df8160f10 2024-04-02T09:17:07.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.363+0000 7f6dfca00700 1 -- 172.21.15.89:0/956023597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df80b7d90 con 0x7f6df8160f10 2024-04-02T09:17:07.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.364+0000 7f6dee000700 1 -- 172.21.15.89:0/956023597 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 186) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f6de4022070 con 0x7f6df8160f10 2024-04-02T09:17:07.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.366+0000 7f6dee000700 1 --2- 172.21.15.89:0/956023597 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6dd806af60 0x7f6dd806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:07.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.366+0000 7f6dee000700 1 -- 172.21.15.89:0/956023597 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f6de408abd0 con 0x7f6df8160f10 2024-04-02T09:17:07.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.366+0000 7f6df5e00700 1 -- 172.21.15.89:0/956023597 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6dd806af60 msgr2=0x7f6dd806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6832/108855 2024-04-02T09:17:07.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.366+0000 7f6df5e00700 1 --2- 172.21.15.89:0/956023597 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6dd806af60 0x7f6dd806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T09:17:07.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.370+0000 7f6dee000700 1 -- 172.21.15.89:0/956023597 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6df80b7d90 con 0x7f6df8160f10 2024-04-02T09:17:07.395 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:07.396+0000 7fdd2c137240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T09:17:07.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.444+0000 7f6dee000700 1 -- 172.21.15.89:0/956023597 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 187) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f6de4055b60 con 0x7f6df8160f10 2024-04-02T09:17:07.518 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:07.519+0000 7fdd2c137240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T09:17:07.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.535+0000 7f6dee000700 1 -- 172.21.15.89:0/956023597 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 188) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f6de4058940 con 0x7f6df8160f10 2024-04-02T09:17:07.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.535+0000 7f6dee000700 1 -- 172.21.15.89:0/956023597 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6dd806af60 msgr2=0x7f6dd806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T09:17:07.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.535+0000 7f6dee000700 1 --2- 172.21.15.89:0/956023597 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6dd806af60 0x7f6dd806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.644 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:07.645+0000 7fdc9f200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:17:07.644 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:07.645+0000 7fdc9f200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:17:07.645 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:07.645+0000 7fdc9f200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:17:07.645 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:07.645+0000 7fdc9f200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:17:07.645 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:07.645+0000 7fdc9f200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:17:07.645 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:07.646+0000 7fdc9b600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:17:07.645 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:07.646+0000 7fdc9b600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:17:07.645 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:07.646+0000 7fdc9b600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:17:07.646 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:07.646+0000 7fdc9b600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:17:07.646 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-02T09:17:07.646+0000 7fdc9b600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T09:17:07.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.673+0000 7f6dfca00700 1 -- 172.21.15.89:0/956023597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6df80b7d90 con 0x7f6df8160f10 2024-04-02T09:17:07.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.674+0000 7f6dee000700 1 -- 172.21.15.89:0/956023597 <== mon.0 v2:172.21.15.89:3300/0 9 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4753 (secure 0 0 0) 0x7f6df80b7d90 con 0x7f6df8160f10 2024-04-02T09:17:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:17:07.673 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10376,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2447701494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2447701494},{"type":"v1","addr":"172.21.15.113:6834","nonce":2447701494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":12103,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/361614750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":361614750},{"type":"v1","addr":"172.21.15.113:6835","nonce":361614750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":13281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2871180587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2871180587},{"type":"v1","addr":"172.21.15.89:6835","nonce":2871180587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-02T09:15:26.772241+0000","modified":"2024-04-02T09:15:30.258749+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13308":{"gid":13308,"name":"a","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6837/3853118738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3853118738},{"type":"v1","addr":"172.21.15.89:6837","nonce":3853118738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-04-02T09:17:07.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.676+0000 7f6dfca00700 1 -- 172.21.15.89:0/956023597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8160f10 msgr2=0x7f6df80cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:07.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.676+0000 7f6dfca00700 1 --2- 172.21.15.89:0/956023597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8160f10 0x7f6df80cfb20 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f6de4014040 tx=0x7f6de4004750 comp rx=0 tx=0).stop 2024-04-02T09:17:07.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.676+0000 7f6dfca00700 1 -- 172.21.15.89:0/956023597 shutdown_connections 2024-04-02T09:17:07.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.676+0000 7f6dfca00700 1 --2- 172.21.15.89:0/956023597 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f6dd806af60 0x7f6dd806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.676+0000 7f6dfca00700 1 --2- 172.21.15.89:0/956023597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6df80d18b0 0x7f6df8155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.676+0000 7f6dfca00700 1 --2- 172.21.15.89:0/956023597 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6df8161590 0x7f6df80d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.676+0000 7f6dfca00700 1 --2- 172.21.15.89:0/956023597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6df8160f10 0x7f6df80cfb20 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.676+0000 7f6dfca00700 1 -- 172.21.15.89:0/956023597 >> 172.21.15.89:0/956023597 conn(0x7f6df808aff0 msgr2=0x7f6df80cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:07.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.676+0000 7f6dfca00700 1 -- 172.21.15.89:0/956023597 shutdown_connections 2024-04-02T09:17:07.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.676+0000 7f6dfca00700 1 -- 172.21.15.89:0/956023597 wait complete. 2024-04-02T09:17:07.677 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 144 2024-04-02T09:17:07.694 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:17:07.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.884+0000 7f9959000700 1 Processor -- start 2024-04-02T09:17:07.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.885+0000 7f9959000700 1 -- start start 2024-04-02T09:17:07.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.886+0000 7f9959000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f995415d640 0x7f99540d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:07.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.886+0000 7f9959000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99540ceec0 0x7f99541612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:07.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.886+0000 7f9959000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99540d30d0 con 0x7f9954161800 2024-04-02T09:17:07.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.886+0000 7f9959000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9954161c90 con 0x7f995415d640 2024-04-02T09:17:07.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.886+0000 7f9959000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99540b6840 con 0x7f99540ceec0 2024-04-02T09:17:07.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.886+0000 7f9952800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9954161800 0x7f99540cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51048/0 (socket says 172.21.15.89:51048) 2024-04-02T09:17:07.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.886+0000 7f9952800700 1 -- 172.21.15.89:0/1909146648 learned_addr learned my addr 172.21.15.89:0/1909146648 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:07.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.886+0000 7f9951e00700 1 --2- 172.21.15.89:0/1909146648 >> v2:172.21.15.113:3300/0 conn(0x7f995415d640 0x7f99540d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:07.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.886+0000 7f9951400700 1 --2- 172.21.15.89:0/1909146648 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99540ceec0 0x7f99541612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:07.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.886+0000 7f9950a00700 1 -- 172.21.15.89:0/1909146648 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903947813 0 0) 0x7f99540d30d0 con 0x7f9954161800 2024-04-02T09:17:07.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.886+0000 7f9950a00700 1 -- 172.21.15.89:0/1909146648 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99380035b0 con 0x7f9954161800 2024-04-02T09:17:07.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.887+0000 7f9951e00700 1 -- 172.21.15.89:0/1909146648 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99540ceec0 msgr2=0x7f99541612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:07.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.887+0000 7f9951e00700 1 --2- 172.21.15.89:0/1909146648 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99540ceec0 0x7f99541612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.887+0000 7f9951e00700 1 -- 172.21.15.89:0/1909146648 >> v1:172.21.15.89:6789/0 conn(0x7f9954161800 legacy=0x7f99540cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:07.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.887+0000 7f9951e00700 1 -- 172.21.15.89:0/1909146648 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99540d2880 con 0x7f995415d640 2024-04-02T09:17:07.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.887+0000 7f9951400700 1 --2- 172.21.15.89:0/1909146648 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99540ceec0 0x7f99541612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:17:07.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.887+0000 7f9951e00700 1 --2- 172.21.15.89:0/1909146648 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f995415d640 0x7f99540d2340 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f9944009840 tx=0x7f9944009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d30a54854a4400d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:07.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.887+0000 7f9950a00700 1 -- 172.21.15.89:0/1909146648 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9944011830 con 0x7f995415d640 2024-04-02T09:17:07.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.887+0000 7f9950a00700 1 -- 172.21.15.89:0/1909146648 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9944011990 con 0x7f995415d640 2024-04-02T09:17:07.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.887+0000 7f9950a00700 1 -- 172.21.15.89:0/1909146648 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9944011d20 con 0x7f995415d640 2024-04-02T09:17:07.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.888+0000 7f9959000700 1 -- 172.21.15.89:0/1909146648 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f995415d640 msgr2=0x7f99540d2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:07.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.888+0000 7f9959000700 1 --2- 172.21.15.89:0/1909146648 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f995415d640 0x7f99540d2340 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f9944009840 tx=0x7f9944009b50 comp rx=0 tx=0).stop 2024-04-02T09:17:07.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.888+0000 7f9959000700 1 -- 172.21.15.89:0/1909146648 shutdown_connections 2024-04-02T09:17:07.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.888+0000 7f9959000700 1 --2- 172.21.15.89:0/1909146648 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f995415d640 0x7f99540d2340 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.888+0000 7f9959000700 1 --2- 172.21.15.89:0/1909146648 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99540ceec0 0x7f99541612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.888+0000 7f9959000700 1 -- 172.21.15.89:0/1909146648 >> 172.21.15.89:0/1909146648 conn(0x7f995408aff0 msgr2=0x7f99540d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:07.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.888+0000 7f9959000700 1 -- 172.21.15.89:0/1909146648 shutdown_connections 2024-04-02T09:17:07.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.888+0000 7f9959000700 1 -- 172.21.15.89:0/1909146648 wait complete. 2024-04-02T09:17:07.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.890+0000 7f9959000700 1 Processor -- start 2024-04-02T09:17:07.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.890+0000 7f9959000700 1 -- start start 2024-04-02T09:17:07.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9959000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99540ceec0 0x7f995406cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:07.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9959000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f995415d640 0x7f9954160180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:07.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9959000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9954161800 0x7f9954152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:07.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9959000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99541558b0 con 0x7f995415d640 2024-04-02T09:17:07.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9959000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9954154ca0 con 0x7f9954161800 2024-04-02T09:17:07.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9959000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9954154f80 con 0x7f99540ceec0 2024-04-02T09:17:07.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9951400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f995415d640 0x7f9954160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:07.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9951400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f995415d640 0x7f9954160180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56440/0 (socket says 172.21.15.89:56440) 2024-04-02T09:17:07.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9951400700 1 -- 172.21.15.89:0/2170937907 learned_addr learned my addr 172.21.15.89:0/2170937907 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:07.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9951e00700 1 --2- 172.21.15.89:0/2170937907 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99540ceec0 0x7f995406cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:07.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9952800700 1 --2- 172.21.15.89:0/2170937907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9954161800 0x7f9954152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:07.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9951400700 1 -- 172.21.15.89:0/2170937907 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99540ceec0 msgr2=0x7f995406cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:07.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9951400700 1 --2- 172.21.15.89:0/2170937907 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99540ceec0 0x7f995406cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9951400700 1 -- 172.21.15.89:0/2170937907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9954161800 msgr2=0x7f9954152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:07.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9951400700 1 --2- 172.21.15.89:0/2170937907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9954161800 0x7f9954152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:07.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9951400700 1 -- 172.21.15.89:0/2170937907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99400094a0 con 0x7f995415d640 2024-04-02T09:17:07.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9951e00700 1 --2- 172.21.15.89:0/2170937907 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99540ceec0 0x7f995406cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:17:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9952800700 1 --2- 172.21.15.89:0/2170937907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9954161800 0x7f9954152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:17:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.891+0000 7f9951400700 1 --2- 172.21.15.89:0/2170937907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f995415d640 0x7f9954160180 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7f994000ebe0 tx=0x7f9940009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.892+0000 7f994a000700 1 -- 172.21.15.89:0/2170937907 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994000c8b0 con 0x7f995415d640 2024-04-02T09:17:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.892+0000 7f994a000700 1 -- 172.21.15.89:0/2170937907 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994000ca10 con 0x7f995415d640 2024-04-02T09:17:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.892+0000 7f994a000700 1 -- 172.21.15.89:0/2170937907 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994000cda0 con 0x7f995415d640 2024-04-02T09:17:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.892+0000 7f9959000700 1 -- 172.21.15.89:0/2170937907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99440094a0 con 0x7f995415d640 2024-04-02T09:17:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.892+0000 7f9959000700 1 -- 172.21.15.89:0/2170937907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9954155540 con 0x7f995415d640 2024-04-02T09:17:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.893+0000 7f9959000700 1 -- 172.21.15.89:0/2170937907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9954067720 con 0x7f995415d640 2024-04-02T09:17:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.894+0000 7f994a000700 1 -- 172.21.15.89:0/2170937907 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 188) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f9940013470 con 0x7f995415d640 2024-04-02T09:17:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.895+0000 7f994a000700 1 -- 172.21.15.89:0/2170937907 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(281..281 src has 1..281) v4 ==== 6888+0+0 (secure 0 0 0) 0x7f9940089490 con 0x7f995415d640 2024-04-02T09:17:07.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:07.900+0000 7f994a000700 1 -- 172.21.15.89:0/2170937907 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9940054530 con 0x7f995415d640 2024-04-02T09:17:08.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.102+0000 7f9959000700 1 -- 172.21.15.89:0/2170937907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99540b5c70 con 0x7f995415d640 2024-04-02T09:17:08.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.103+0000 7f994a000700 1 -- 172.21.15.89:0/2170937907 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v281) v1 ==== 74+0+17780 (secure 0 0 0) 0x7f9940057b50 con 0x7f995415d640 2024-04-02T09:17:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:17:08.103 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":281,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:17:07.522522+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-04-02T09:15:24.345253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-04-02T09:15:25.756381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"279","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:0/3185841611":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/1687104676":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/2207153077":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/171309686":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/3381360183":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/573568808":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/876295205":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/1137235236":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/2320969126":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/2687295125":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/693914703":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/1405191830":"2024-04-03T09:15:43.128314+0000","172.21.15.89:6833/108855":"2024-04-03T09:15:43.128314+0000","172.21.15.89:6832/108855":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/701778740":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/445136997":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/799882283":"2024-04-03T09:15:43.128314+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:17:08.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.106+0000 7f9959000700 1 -- 172.21.15.89:0/2170937907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f995415d640 msgr2=0x7f9954160180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:08.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.106+0000 7f9959000700 1 --2- 172.21.15.89:0/2170937907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f995415d640 0x7f9954160180 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7f994000ebe0 tx=0x7f9940009e20 comp rx=0 tx=0).stop 2024-04-02T09:17:08.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.106+0000 7f9959000700 1 -- 172.21.15.89:0/2170937907 shutdown_connections 2024-04-02T09:17:08.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.106+0000 7f9959000700 1 --2- 172.21.15.89:0/2170937907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9954161800 0x7f9954152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.106+0000 7f9959000700 1 --2- 172.21.15.89:0/2170937907 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99540ceec0 0x7f995406cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.106+0000 7f9959000700 1 --2- 172.21.15.89:0/2170937907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f995415d640 0x7f9954160180 unknown :-1 s=CLOSED pgs=3071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.106+0000 7f9959000700 1 -- 172.21.15.89:0/2170937907 >> 172.21.15.89:0/2170937907 conn(0x7f995408aff0 msgr2=0x7f99540d17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:08.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.106+0000 7f9959000700 1 -- 172.21.15.89:0/2170937907 shutdown_connections 2024-04-02T09:17:08.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.106+0000 7f9959000700 1 -- 172.21.15.89:0/2170937907 wait complete. 2024-04-02T09:17:08.123 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T09:17:08.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.314+0000 7f4bd5600700 1 Processor -- start 2024-04-02T09:17:08.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.315+0000 7f4bd5600700 1 -- start start 2024-04-02T09:17:08.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.316+0000 7f4bd5600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4bd015d9d0 0x7f4bd00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.316+0000 7f4bd5600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4bd00d2540 0x7f4bd00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.316+0000 7f4bd5600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd0161420 con 0x7f4bd00d16a0 2024-04-02T09:17:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.316+0000 7f4bd5600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd015edc0 con 0x7f4bd015d9d0 2024-04-02T09:17:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.316+0000 7f4bd5600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd0161700 con 0x7f4bd00d2540 2024-04-02T09:17:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.316+0000 7f4bcea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4bd015d9d0 0x7f4bd00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.316+0000 7f4bcea00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4bd015d9d0 0x7f4bd00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56448/0 (socket says 172.21.15.89:56448) 2024-04-02T09:17:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.316+0000 7f4bcea00700 1 -- 172.21.15.89:0/2439496150 learned_addr learned my addr 172.21.15.89:0/2439496150 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.316+0000 7f4bcf400700 1 --2- 172.21.15.89:0/2439496150 >> v2:172.21.15.113:3300/0 conn(0x7f4bd00d2540 0x7f4bd00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.317+0000 7f4bcea00700 1 -- 172.21.15.89:0/2439496150 >> v2:172.21.15.113:3300/0 conn(0x7f4bd00d2540 msgr2=0x7f4bd00ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.317+0000 7f4bcea00700 1 --2- 172.21.15.89:0/2439496150 >> v2:172.21.15.113:3300/0 conn(0x7f4bd00d2540 0x7f4bd00ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.317+0000 7f4bcea00700 1 -- 172.21.15.89:0/2439496150 >> v1:172.21.15.89:6789/0 conn(0x7f4bd00d16a0 legacy=0x7f4bd00d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.317+0000 7f4bcea00700 1 -- 172.21.15.89:0/2439496150 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd01619e0 con 0x7f4bd015d9d0 2024-04-02T09:17:08.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.317+0000 7f4bcea00700 1 --2- 172.21.15.89:0/2439496150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd015d9d0 0x7f4bd00d3c30 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f4bd008ca10 tx=0x7f4bc0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba047fe312c0ec9d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.317+0000 7f4bcd600700 1 -- 172.21.15.89:0/2439496150 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc000b870 con 0x7f4bd015d9d0 2024-04-02T09:17:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.317+0000 7f4bcd600700 1 -- 172.21.15.89:0/2439496150 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc000b9d0 con 0x7f4bd015d9d0 2024-04-02T09:17:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.317+0000 7f4bcd600700 1 -- 172.21.15.89:0/2439496150 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc000bd60 con 0x7f4bd015d9d0 2024-04-02T09:17:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.318+0000 7f4bd5600700 1 -- 172.21.15.89:0/2439496150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd015d9d0 msgr2=0x7f4bd00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.318+0000 7f4bd5600700 1 --2- 172.21.15.89:0/2439496150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd015d9d0 0x7f4bd00d3c30 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f4bd008ca10 tx=0x7f4bc0009bd0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.318+0000 7f4bd5600700 1 -- 172.21.15.89:0/2439496150 shutdown_connections 2024-04-02T09:17:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.318+0000 7f4bd5600700 1 --2- 172.21.15.89:0/2439496150 >> v2:172.21.15.113:3300/0 conn(0x7f4bd00d2540 0x7f4bd00ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.318+0000 7f4bd5600700 1 --2- 172.21.15.89:0/2439496150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd015d9d0 0x7f4bd00d3c30 unknown :-1 s=CLOSED pgs=3073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.318+0000 7f4bd5600700 1 -- 172.21.15.89:0/2439496150 >> 172.21.15.89:0/2439496150 conn(0x7f4bd008aff0 msgr2=0x7f4bd00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.318+0000 7f4bd5600700 1 -- 172.21.15.89:0/2439496150 shutdown_connections 2024-04-02T09:17:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.318+0000 7f4bd5600700 1 -- 172.21.15.89:0/2439496150 wait complete. 2024-04-02T09:17:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.320+0000 7f4bd5600700 1 Processor -- start 2024-04-02T09:17:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.320+0000 7f4bd5600700 1 -- start start 2024-04-02T09:17:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.321+0000 7f4bd5600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd00d16a0 0x7f4bd00b7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.321+0000 7f4bd5600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd00d2540 0x7f4bd01e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.321+0000 7f4bd5600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bd015d9d0 0x7f4bd014f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.321+0000 7f4bd5600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd01e8730 con 0x7f4bd00d16a0 2024-04-02T09:17:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.321+0000 7f4bd5600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd01e8a10 con 0x7f4bd00d2540 2024-04-02T09:17:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.321+0000 7f4bd5600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd00cb380 con 0x7f4bd015d9d0 2024-04-02T09:17:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.322+0000 7f4bcea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd00d16a0 0x7f4bd00b7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.322+0000 7f4bcea00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd00d16a0 0x7f4bd00b7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56458/0 (socket says 172.21.15.89:56458) 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.322+0000 7f4bcea00700 1 -- 172.21.15.89:0/268555406 learned_addr learned my addr 172.21.15.89:0/268555406 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.322+0000 7f4bce000700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd00d2540 0x7f4bd01e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.322+0000 7f4bcf400700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bd015d9d0 0x7f4bd014f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.322+0000 7f4bcea00700 1 -- 172.21.15.89:0/268555406 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bd015d9d0 msgr2=0x7f4bd014f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.322+0000 7f4bcea00700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bd015d9d0 0x7f4bd014f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.322+0000 7f4bcea00700 1 -- 172.21.15.89:0/268555406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd00d2540 msgr2=0x7f4bd01e7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.322+0000 7f4bcea00700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd00d2540 0x7f4bd01e7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.322+0000 7f4bcea00700 1 -- 172.21.15.89:0/268555406 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc0009550 con 0x7f4bd00d16a0 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.322+0000 7f4bce000700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd00d2540 0x7f4bd01e7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.322+0000 7f4bcea00700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd00d16a0 0x7f4bd00b7e50 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0000c00 tx=0x7f4bc0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.322+0000 7f4bc6a00700 1 -- 172.21.15.89:0/268555406 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0004520 con 0x7f4bd00d16a0 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.323+0000 7f4bc6a00700 1 -- 172.21.15.89:0/268555406 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc001e070 con 0x7f4bd00d16a0 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.323+0000 7f4bd5600700 1 -- 172.21.15.89:0/268555406 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd00cb660 con 0x7f4bd00d16a0 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.323+0000 7f4bc6a00700 1 -- 172.21.15.89:0/268555406 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc00038a0 con 0x7f4bd00d16a0 2024-04-02T09:17:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.323+0000 7f4bd5600700 1 -- 172.21.15.89:0/268555406 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd015ca60 con 0x7f4bd00d16a0 2024-04-02T09:17:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.324+0000 7f4bd5600700 1 -- 172.21.15.89:0/268555406 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd00d1ca0 con 0x7f4bd00d16a0 2024-04-02T09:17:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.324+0000 7f4bc6a00700 1 -- 172.21.15.89:0/268555406 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 188) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f4bc0003a00 con 0x7f4bd00d16a0 2024-04-02T09:17:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.325+0000 7f4bc6a00700 1 -- 172.21.15.89:0/268555406 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(281..281 src has 1..281) v4 ==== 6888+0+0 (secure 0 0 0) 0x7f4bc0020070 con 0x7f4bd00d16a0 2024-04-02T09:17:08.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.330+0000 7f4bc6a00700 1 -- 172.21.15.89:0/268555406 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4bc00558d0 con 0x7f4bd00d16a0 2024-04-02T09:17:08.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.533+0000 7f4bc6a00700 1 -- 172.21.15.89:0/268555406 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 189) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f4bc0055380 con 0x7f4bd00d16a0 2024-04-02T09:17:08.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.534+0000 7f4bc6a00700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4bb0036150 0x7f4bb0072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:08.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.536+0000 7f4bce000700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4bb0036150 0x7f4bb0072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:08.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.539+0000 7f4bce000700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4bb0036150 0x7f4bb0072900 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4bd00cdbd0 tx=0x7f4bbc009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:08.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.589+0000 7f4bd5600700 1 -- 172.21.15.89:0/268555406 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4bd01e8bd0 con 0x7f4bd00d16a0 2024-04-02T09:17:08.589 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-02T09:17:08.590+0000 7f9430a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T09:17:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.602+0000 7f4bc6a00700 1 -- 172.21.15.89:0/268555406 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v145) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4bc005ce70 con 0x7f4bd00d16a0 2024-04-02T09:17:08.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.605+0000 7f4bd5600700 1 -- 172.21.15.89:0/268555406 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4bb0036150 msgr2=0x7f4bb0072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:08.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.605+0000 7f4bd5600700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4bb0036150 0x7f4bb0072900 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4bd00cdbd0 tx=0x7f4bbc009040 comp rx=0 tx=0).stop 2024-04-02T09:17:08.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.605+0000 7f4bd5600700 1 -- 172.21.15.89:0/268555406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd00d16a0 msgr2=0x7f4bd00b7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:08.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.605+0000 7f4bd5600700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd00d16a0 0x7f4bd00b7e50 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0000c00 tx=0x7f4bc0004750 comp rx=0 tx=0).stop 2024-04-02T09:17:08.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.605+0000 7f4bd5600700 1 -- 172.21.15.89:0/268555406 shutdown_connections 2024-04-02T09:17:08.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.605+0000 7f4bd5600700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4bd015d9d0 0x7f4bd014f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.605+0000 7f4bd5600700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd00d16a0 0x7f4bd00b7e50 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.605+0000 7f4bd5600700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4bb0036150 0x7f4bb0072900 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.605+0000 7f4bd5600700 1 --2- 172.21.15.89:0/268555406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd00d2540 0x7f4bd01e7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.605+0000 7f4bd5600700 1 -- 172.21.15.89:0/268555406 >> 172.21.15.89:0/268555406 conn(0x7f4bd008aff0 msgr2=0x7f4bd00d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:08.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.605+0000 7f4bd5600700 1 -- 172.21.15.89:0/268555406 shutdown_connections 2024-04-02T09:17:08.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.605+0000 7f4bd5600700 1 -- 172.21.15.89:0/268555406 wait complete. 2024-04-02T09:17:08.605 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T09:17:08.630 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T09:17:08.679 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2024-04-02T09:17:08.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.829+0000 7f65ad600700 1 Processor -- start 2024-04-02T09:17:08.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.829+0000 7f65ad600700 1 -- start start 2024-04-02T09:17:08.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.830+0000 7f65ad600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65a8160d70 0x7f65a80b3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:08.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.830+0000 7f65ad600700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f65a815ef70 0x7f65a815e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:08.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.830+0000 7f65ad600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65a8161150 con 0x7f65a806a790 2024-04-02T09:17:08.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.830+0000 7f65ad600700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65a80b4cd0 con 0x7f65a815ef70 2024-04-02T09:17:08.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.830+0000 7f65ad600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a806dba0 con 0x7f65a8160d70 2024-04-02T09:17:08.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.831+0000 7f65a6000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f65a815ef70 0x7f65a815e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:08.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.831+0000 7f65a7400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f65a806a790 0x7f65a81613a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59562/0 (socket says 172.21.15.89:59562) 2024-04-02T09:17:08.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.831+0000 7f65a7400700 1 -- 172.21.15.89:0/3035328102 learned_addr learned my addr 172.21.15.89:0/3035328102 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:08.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.831+0000 7f65a6a00700 1 --2- 172.21.15.89:0/3035328102 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65a8160d70 0x7f65a80b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:08.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.831+0000 7f65a6000700 1 -- 172.21.15.89:0/3035328102 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65a8160d70 msgr2=0x7f65a80b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:08.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.831+0000 7f65a6000700 1 --2- 172.21.15.89:0/3035328102 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65a8160d70 0x7f65a80b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.831+0000 7f65a6000700 1 -- 172.21.15.89:0/3035328102 >> v1:172.21.15.89:6789/0 conn(0x7f65a806a790 legacy=0x7f65a81613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:08.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.831+0000 7f65a6000700 1 -- 172.21.15.89:0/3035328102 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a815ed30 con 0x7f65a815ef70 2024-04-02T09:17:08.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.831+0000 7f65a5600700 1 -- 172.21.15.89:0/3035328102 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240650020 0 0) 0x7f65a8161150 con 0x7f65a806a790 2024-04-02T09:17:08.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.831+0000 7f65a6000700 1 --2- 172.21.15.89:0/3035328102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65a815ef70 0x7f65a815e7f0 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7f6594009840 tx=0x7f6594009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfa59efe6668f06 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:08.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.832+0000 7f65a5600700 1 -- 172.21.15.89:0/3035328102 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594011830 con 0x7f65a815ef70 2024-04-02T09:17:08.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.832+0000 7f65a5600700 1 -- 172.21.15.89:0/3035328102 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6594011990 con 0x7f65a815ef70 2024-04-02T09:17:08.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.832+0000 7f65a5600700 1 -- 172.21.15.89:0/3035328102 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594011d20 con 0x7f65a815ef70 2024-04-02T09:17:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.832+0000 7f65ad600700 1 -- 172.21.15.89:0/3035328102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65a815ef70 msgr2=0x7f65a815e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.832+0000 7f65ad600700 1 --2- 172.21.15.89:0/3035328102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65a815ef70 0x7f65a815e7f0 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7f6594009840 tx=0x7f6594009b50 comp rx=0 tx=0).stop 2024-04-02T09:17:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.832+0000 7f65ad600700 1 -- 172.21.15.89:0/3035328102 shutdown_connections 2024-04-02T09:17:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.832+0000 7f65ad600700 1 --2- 172.21.15.89:0/3035328102 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65a8160d70 0x7f65a80b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.832+0000 7f65ad600700 1 --2- 172.21.15.89:0/3035328102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65a815ef70 0x7f65a815e7f0 unknown :-1 s=CLOSED pgs=3078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.832+0000 7f65ad600700 1 -- 172.21.15.89:0/3035328102 >> 172.21.15.89:0/3035328102 conn(0x7f65a80b5720 msgr2=0x7f65a80b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.832+0000 7f65ad600700 1 -- 172.21.15.89:0/3035328102 shutdown_connections 2024-04-02T09:17:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.832+0000 7f65ad600700 1 -- 172.21.15.89:0/3035328102 wait complete. 2024-04-02T09:17:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.834+0000 7f65ad600700 1 Processor -- start 2024-04-02T09:17:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.834+0000 7f65ad600700 1 -- start start 2024-04-02T09:17:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.835+0000 7f65ad600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65a806a790 0x7f65a80b1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.835+0000 7f65ad600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65a815ef70 0x7f65a809d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.835+0000 7f65ad600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65a8160d70 0x7f65a809dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.835+0000 7f65ad600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a80c98b0 con 0x7f65a8160d70 2024-04-02T09:17:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.835+0000 7f65ad600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a81f3850 con 0x7f65a815ef70 2024-04-02T09:17:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.835+0000 7f65ad600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a81f3b30 con 0x7f65a806a790 2024-04-02T09:17:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.835+0000 7f65a7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65a8160d70 0x7f65a809dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.835+0000 7f65a7400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65a8160d70 0x7f65a809dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56762/0 (socket says 172.21.15.89:56762) 2024-04-02T09:17:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.835+0000 7f65a7400700 1 -- 172.21.15.89:0/2984279707 learned_addr learned my addr 172.21.15.89:0/2984279707 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.835+0000 7f65a6a00700 1 --2- 172.21.15.89:0/2984279707 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65a806a790 0x7f65a80b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.835+0000 7f65a6000700 1 --2- 172.21.15.89:0/2984279707 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65a815ef70 0x7f65a809d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.836+0000 7f65a7400700 1 -- 172.21.15.89:0/2984279707 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65a806a790 msgr2=0x7f65a80b1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.836+0000 7f65a7400700 1 --2- 172.21.15.89:0/2984279707 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65a806a790 0x7f65a80b1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.836+0000 7f65a7400700 1 -- 172.21.15.89:0/2984279707 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65a815ef70 msgr2=0x7f65a809d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.836+0000 7f65a7400700 1 --2- 172.21.15.89:0/2984279707 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65a815ef70 0x7f65a809d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.836+0000 7f65a7400700 1 -- 172.21.15.89:0/2984279707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65940094a0 con 0x7f65a8160d70 2024-04-02T09:17:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.836+0000 7f65a7400700 1 --2- 172.21.15.89:0/2984279707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65a8160d70 0x7f65a809dad0 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7f65a000cca0 tx=0x7f65a000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.836+0000 7f659ea00700 1 -- 172.21.15.89:0/2984279707 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a000ab70 con 0x7f65a8160d70 2024-04-02T09:17:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.836+0000 7f659ea00700 1 -- 172.21.15.89:0/2984279707 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a000acd0 con 0x7f65a8160d70 2024-04-02T09:17:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.836+0000 7f659ea00700 1 -- 172.21.15.89:0/2984279707 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a0007310 con 0x7f65a8160d70 2024-04-02T09:17:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.836+0000 7f65ad600700 1 -- 172.21.15.89:0/2984279707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a81f3e60 con 0x7f65a8160d70 2024-04-02T09:17:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.836+0000 7f65ad600700 1 -- 172.21.15.89:0/2984279707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a81f4420 con 0x7f65a8160d70 2024-04-02T09:17:08.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.837+0000 7f65ad600700 1 -- 172.21.15.89:0/2984279707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a81f4640 con 0x7f65a8160d70 2024-04-02T09:17:08.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.838+0000 7f659ea00700 1 -- 172.21.15.89:0/2984279707 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 189) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f65a0007470 con 0x7f65a8160d70 2024-04-02T09:17:08.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.839+0000 7f659ea00700 1 --2- 172.21.15.89:0/2984279707 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f658806aae0 0x7f658806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:08.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.839+0000 7f659ea00700 1 -- 172.21.15.89:0/2984279707 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(282..282 src has 1..282) v4 ==== 6974+0+0 (secure 0 0 0) 0x7f65a0089050 con 0x7f65a8160d70 2024-04-02T09:17:08.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.839+0000 7f65a6a00700 1 --2- 172.21.15.89:0/2984279707 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f658806aae0 0x7f658806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:08.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.840+0000 7f65a6a00700 1 --2- 172.21.15.89:0/2984279707 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f658806aae0 0x7f658806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f65a80b3cf0 tx=0x7f6598008040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:08.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:08.843+0000 7f659ea00700 1 -- 172.21.15.89:0/2984279707 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f65a00bc580 con 0x7f65a8160d70 2024-04-02T09:17:09.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.090+0000 7f65ad600700 1 -- 172.21.15.89:0/2984279707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f65a8067720 con 0x7f65a8160d70 2024-04-02T09:17:09.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.604+0000 7f659ea00700 1 -- 172.21.15.89:0/2984279707 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v146) v1 ==== 104+0+0 (secure 0 0 0) 0x7f65a0018020 con 0x7f65a8160d70 2024-04-02T09:17:09.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.607+0000 7f65ad600700 1 -- 172.21.15.89:0/2984279707 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f658806aae0 msgr2=0x7f658806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:09.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.607+0000 7f65ad600700 1 --2- 172.21.15.89:0/2984279707 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f658806aae0 0x7f658806cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f65a80b3cf0 tx=0x7f6598008040 comp rx=0 tx=0).stop 2024-04-02T09:17:09.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.607+0000 7f65ad600700 1 -- 172.21.15.89:0/2984279707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65a8160d70 msgr2=0x7f65a809dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:09.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.607+0000 7f65ad600700 1 --2- 172.21.15.89:0/2984279707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65a8160d70 0x7f65a809dad0 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7f65a000cca0 tx=0x7f65a000a060 comp rx=0 tx=0).stop 2024-04-02T09:17:09.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.607+0000 7f65ad600700 1 -- 172.21.15.89:0/2984279707 shutdown_connections 2024-04-02T09:17:09.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.607+0000 7f65ad600700 1 --2- 172.21.15.89:0/2984279707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65a8160d70 0x7f65a809dad0 unknown :-1 s=CLOSED pgs=3079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:09.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.607+0000 7f65ad600700 1 --2- 172.21.15.89:0/2984279707 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65a806a790 0x7f65a80b1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:09.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.607+0000 7f65ad600700 1 --2- 172.21.15.89:0/2984279707 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f658806aae0 0x7f658806cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:09.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.607+0000 7f65ad600700 1 --2- 172.21.15.89:0/2984279707 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65a815ef70 0x7f65a809d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:09.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.607+0000 7f65ad600700 1 -- 172.21.15.89:0/2984279707 >> 172.21.15.89:0/2984279707 conn(0x7f65a80b5720 msgr2=0x7f65a809c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:09.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.607+0000 7f65ad600700 1 -- 172.21.15.89:0/2984279707 shutdown_connections 2024-04-02T09:17:09.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.608+0000 7f65ad600700 1 -- 172.21.15.89:0/2984279707 wait complete. 2024-04-02T09:17:09.628 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T09:17:09.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.826+0000 7f505e000700 1 Processor -- start 2024-04-02T09:17:09.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.826+0000 7f505e000700 1 -- start start 2024-04-02T09:17:09.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.827+0000 7f505e000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5058155ac0 0x7f505815c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:09.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.827+0000 7f505e000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f505806c910 0x7f50580adf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:09.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.827+0000 7f505e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f50580ae4a0 0x7f505815f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:09.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.827+0000 7f505e000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f505806dd00 con 0x7f5058155ac0 2024-04-02T09:17:09.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.827+0000 7f505e000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f505806d0f0 con 0x7f505806c910 2024-04-02T09:17:09.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.827+0000 7f505e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f505806d3d0 con 0x7f50580ae4a0 2024-04-02T09:17:09.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5057400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5058155ac0 0x7f505815c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:09.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5057400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5058155ac0 0x7f505815c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56790/0 (socket says 172.21.15.89:56790) 2024-04-02T09:17:09.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5057400700 1 -- 172.21.15.89:0/556441763 learned_addr learned my addr 172.21.15.89:0/556441763 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:09.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5056a00700 1 --2- 172.21.15.89:0/556441763 >> v2:172.21.15.113:3300/0 conn(0x7f505806c910 0x7f50580adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:09.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5057e00700 1 --2- 172.21.15.89:0/556441763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f50580ae4a0 0x7f505815f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:09.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5057400700 1 -- 172.21.15.89:0/556441763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f50580ae4a0 msgr2=0x7f505815f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:09.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5057400700 1 --2- 172.21.15.89:0/556441763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f50580ae4a0 0x7f505815f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:09.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5057400700 1 -- 172.21.15.89:0/556441763 >> v2:172.21.15.113:3300/0 conn(0x7f505806c910 msgr2=0x7f50580adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:09.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5057400700 1 --2- 172.21.15.89:0/556441763 >> v2:172.21.15.113:3300/0 conn(0x7f505806c910 0x7f50580adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:09.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5057400700 1 -- 172.21.15.89:0/556441763 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505806d6b0 con 0x7f5058155ac0 2024-04-02T09:17:09.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5056a00700 1 --2- 172.21.15.89:0/556441763 >> v2:172.21.15.113:3300/0 conn(0x7f505806c910 0x7f50580adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:17:09.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5057e00700 1 --2- 172.21.15.89:0/556441763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f50580ae4a0 0x7f505815f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:17:09.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5057400700 1 --2- 172.21.15.89:0/556441763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5058155ac0 0x7f505815c700 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7f50580b46d0 tx=0x7f5048009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d941869b5895c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:09.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5056000700 1 -- 172.21.15.89:0/556441763 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504800b870 con 0x7f5058155ac0 2024-04-02T09:17:09.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.828+0000 7f5056000700 1 -- 172.21.15.89:0/556441763 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504800b9d0 con 0x7f5058155ac0 2024-04-02T09:17:09.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.829+0000 7f5056000700 1 -- 172.21.15.89:0/556441763 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504800bd60 con 0x7f5058155ac0 2024-04-02T09:17:09.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.829+0000 7f505e000700 1 -- 172.21.15.89:0/556441763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5058155ac0 msgr2=0x7f505815c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:09.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.829+0000 7f505e000700 1 --2- 172.21.15.89:0/556441763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5058155ac0 0x7f505815c700 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7f50580b46d0 tx=0x7f5048009bd0 comp rx=0 tx=0).stop 2024-04-02T09:17:09.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.829+0000 7f505e000700 1 -- 172.21.15.89:0/556441763 shutdown_connections 2024-04-02T09:17:09.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.829+0000 7f505e000700 1 --2- 172.21.15.89:0/556441763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5058155ac0 0x7f505815c700 unknown :-1 s=CLOSED pgs=3082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:09.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.829+0000 7f505e000700 1 --2- 172.21.15.89:0/556441763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f50580ae4a0 0x7f505815f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:09.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.829+0000 7f505e000700 1 --2- 172.21.15.89:0/556441763 >> v2:172.21.15.113:3300/0 conn(0x7f505806c910 0x7f50580adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:09.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.829+0000 7f505e000700 1 -- 172.21.15.89:0/556441763 >> 172.21.15.89:0/556441763 conn(0x7f50580cf210 msgr2=0x7f50580cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:09.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.829+0000 7f505e000700 1 -- 172.21.15.89:0/556441763 shutdown_connections 2024-04-02T09:17:09.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.829+0000 7f505e000700 1 -- 172.21.15.89:0/556441763 wait complete. 2024-04-02T09:17:09.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.831+0000 7f505e000700 1 Processor -- start 2024-04-02T09:17:09.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.831+0000 7f505e000700 1 -- start start 2024-04-02T09:17:09.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f505e000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f505806c910 0x7f50581f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:09.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f505e000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50580ae4a0 0x7f5058152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:09.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f505e000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5058155ac0 0x7f505814f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:09.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f505e000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50581527f0 con 0x7f5058155ac0 2024-04-02T09:17:09.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f505e000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5058151be0 con 0x7f50580ae4a0 2024-04-02T09:17:09.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f505e000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5058151ec0 con 0x7f505806c910 2024-04-02T09:17:09.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f5057e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5058155ac0 0x7f505814f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:09.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f5057e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5058155ac0 0x7f505814f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56792/0 (socket says 172.21.15.89:56792) 2024-04-02T09:17:09.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f5057e00700 1 -- 172.21.15.89:0/1130853355 learned_addr learned my addr 172.21.15.89:0/1130853355 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:09.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f5056a00700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50580ae4a0 0x7f5058152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:09.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f5057400700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f505806c910 0x7f50581f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:09.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f5057e00700 1 -- 172.21.15.89:0/1130853355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f505806c910 msgr2=0x7f50581f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:09.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f5057e00700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f505806c910 0x7f50581f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:09.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f5057e00700 1 -- 172.21.15.89:0/1130853355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50580ae4a0 msgr2=0x7f5058152c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:09.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f5057e00700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50580ae4a0 0x7f5058152c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:09.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.832+0000 7f5057e00700 1 -- 172.21.15.89:0/1130853355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5048009550 con 0x7f5058155ac0 2024-04-02T09:17:09.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.833+0000 7f5057400700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f505806c910 0x7f50581f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:17:09.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.833+0000 7f5056a00700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50580ae4a0 0x7f5058152c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:17:09.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.833+0000 7f5057e00700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5058155ac0 0x7f505814f620 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7f5050007ef0 tx=0x7f505000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:09.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.833+0000 7f504f400700 1 -- 172.21.15.89:0/1130853355 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505000de60 con 0x7f5058155ac0 2024-04-02T09:17:09.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.833+0000 7f504f400700 1 -- 172.21.15.89:0/1130853355 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505000f040 con 0x7f5058155ac0 2024-04-02T09:17:09.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.833+0000 7f505e000700 1 -- 172.21.15.89:0/1130853355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50581521a0 con 0x7f5058155ac0 2024-04-02T09:17:09.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.833+0000 7f504f400700 1 -- 172.21.15.89:0/1130853355 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5050003c60 con 0x7f5058155ac0 2024-04-02T09:17:09.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.833+0000 7f505e000700 1 -- 172.21.15.89:0/1130853355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50581f2c80 con 0x7f5058155ac0 2024-04-02T09:17:09.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.834+0000 7f505e000700 1 -- 172.21.15.89:0/1130853355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5058152960 con 0x7f5058155ac0 2024-04-02T09:17:09.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.835+0000 7f504f400700 1 -- 172.21.15.89:0/1130853355 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 189) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f5050004030 con 0x7f5058155ac0 2024-04-02T09:17:09.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.836+0000 7f504f400700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f503806aae0 0x7f503806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:09.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.837+0000 7f504f400700 1 -- 172.21.15.89:0/1130853355 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(282..282 src has 1..282) v4 ==== 6974+0+0 (secure 0 0 0) 0x7f50500896d0 con 0x7f5058155ac0 2024-04-02T09:17:09.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.837+0000 7f5057400700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f503806aae0 0x7f503806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:09.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.837+0000 7f5057400700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f503806aae0 0x7f503806cfa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5048009130 tx=0x7f5048008030 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:09.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:09.841+0000 7f504f400700 1 -- 172.21.15.89:0/1130853355 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5050054630 con 0x7f5058155ac0 2024-04-02T09:17:10.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.042+0000 7f505e000700 1 -- 172.21.15.89:0/1130853355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5058067720 con 0x7f5058155ac0 2024-04-02T09:17:10.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.292+0000 7f504f400700 1 -- 172.21.15.89:0/1130853355 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 190) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f5050054a00 con 0x7f5058155ac0 2024-04-02T09:17:10.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.605+0000 7f504f400700 1 -- 172.21.15.89:0/1130853355 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v283) v1 ==== 181+0+0 (secure 0 0 0) 0x7f50500715b0 con 0x7f5058155ac0 2024-04-02T09:17:10.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.609+0000 7f505e000700 1 -- 172.21.15.89:0/1130853355 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f503806aae0 msgr2=0x7f503806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:10.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.609+0000 7f505e000700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f503806aae0 0x7f503806cfa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5048009130 tx=0x7f5048008030 comp rx=0 tx=0).stop 2024-04-02T09:17:10.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.609+0000 7f505e000700 1 -- 172.21.15.89:0/1130853355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5058155ac0 msgr2=0x7f505814f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:10.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.609+0000 7f505e000700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5058155ac0 0x7f505814f620 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7f5050007ef0 tx=0x7f505000d3b0 comp rx=0 tx=0).stop 2024-04-02T09:17:10.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.609+0000 7f505e000700 1 -- 172.21.15.89:0/1130853355 shutdown_connections 2024-04-02T09:17:10.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.609+0000 7f505e000700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5058155ac0 0x7f505814f620 unknown :-1 s=CLOSED pgs=3083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:10.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.609+0000 7f505e000700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f505806c910 0x7f50581f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:10.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.609+0000 7f505e000700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f503806aae0 0x7f503806cfa0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:10.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.609+0000 7f505e000700 1 --2- 172.21.15.89:0/1130853355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50580ae4a0 0x7f5058152c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:10.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.609+0000 7f505e000700 1 -- 172.21.15.89:0/1130853355 >> 172.21.15.89:0/1130853355 conn(0x7f50580cf210 msgr2=0x7f50580b2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:10.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.610+0000 7f505e000700 1 -- 172.21.15.89:0/1130853355 shutdown_connections 2024-04-02T09:17:10.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.610+0000 7f505e000700 1 -- 172.21.15.89:0/1130853355 wait complete. 2024-04-02T09:17:10.610 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-04-02T09:17:10.628 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T09:17:10.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.827+0000 7f91c9000700 1 Processor -- start 2024-04-02T09:17:10.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.827+0000 7f91c9000700 1 -- start start 2024-04-02T09:17:10.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.828+0000 7f91c9000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f91c415d930 0x7f91c40d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:10.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.828+0000 7f91c9000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91c4162030 0x7f91c40d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:10.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.828+0000 7f91c9000700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c415dd10 con 0x7f91c40d20d0 2024-04-02T09:17:10.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.828+0000 7f91c9000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91c415ec40 con 0x7f91c415d930 2024-04-02T09:17:10.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.828+0000 7f91c9000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c40d2560 con 0x7f91c4162030 2024-04-02T09:17:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.829+0000 7f91c1e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f91c415d930 0x7f91c40d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.829+0000 7f91c1400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91c4162030 0x7f91c40d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.829+0000 7f91c2800700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f91c40d20d0 0x7f91c40cfc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:38308/0 (socket says 172.21.15.89:38308) 2024-04-02T09:17:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.829+0000 7f91c1400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91c4162030 0x7f91c40d1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:54206/0 (socket says 172.21.15.89:54206) 2024-04-02T09:17:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.829+0000 7f91c2800700 1 -- 172.21.15.89:0/571920752 learned_addr learned my addr 172.21.15.89:0/571920752 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.829+0000 7f91c1400700 1 -- 172.21.15.89:0/571920752 >> v2:172.21.15.113:3300/0 conn(0x7f91c415d930 msgr2=0x7f91c40d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:10.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.829+0000 7f91c1400700 1 --2- 172.21.15.89:0/571920752 >> v2:172.21.15.113:3300/0 conn(0x7f91c415d930 0x7f91c40d3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:10.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.829+0000 7f91c1400700 1 -- 172.21.15.89:0/571920752 >> v1:172.21.15.113:6789/0 conn(0x7f91c40d20d0 legacy=0x7f91c40cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:10.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.829+0000 7f91c1400700 1 -- 172.21.15.89:0/571920752 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c415dfc0 con 0x7f91c4162030 2024-04-02T09:17:10.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.829+0000 7f91c1e00700 1 --2- 172.21.15.89:0/571920752 >> v2:172.21.15.113:3300/0 conn(0x7f91c415d930 0x7f91c40d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:17:10.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.830+0000 7f91c1400700 1 --2- 172.21.15.89:0/571920752 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91c4162030 0x7f91c40d1b90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009840 tx=0x7f91b0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2b8223c82c23336 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:10.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.830+0000 7f91c0a00700 1 -- 172.21.15.89:0/571920752 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b0011830 con 0x7f91c4162030 2024-04-02T09:17:10.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.830+0000 7f91c0a00700 1 -- 172.21.15.89:0/571920752 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b0011990 con 0x7f91c4162030 2024-04-02T09:17:10.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.830+0000 7f91c0a00700 1 -- 172.21.15.89:0/571920752 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b0011d20 con 0x7f91c4162030 2024-04-02T09:17:10.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.830+0000 7f91c9000700 1 -- 172.21.15.89:0/571920752 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91c4162030 msgr2=0x7f91c40d1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:10.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.830+0000 7f91c9000700 1 --2- 172.21.15.89:0/571920752 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91c4162030 0x7f91c40d1b90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009840 tx=0x7f91b0009b50 comp rx=0 tx=0).stop 2024-04-02T09:17:10.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.831+0000 7f91c9000700 1 -- 172.21.15.89:0/571920752 shutdown_connections 2024-04-02T09:17:10.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.831+0000 7f91c9000700 1 --2- 172.21.15.89:0/571920752 >> v2:172.21.15.113:3300/0 conn(0x7f91c415d930 0x7f91c40d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:10.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.831+0000 7f91c9000700 1 --2- 172.21.15.89:0/571920752 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91c4162030 0x7f91c40d1b90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:10.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.831+0000 7f91c9000700 1 -- 172.21.15.89:0/571920752 >> 172.21.15.89:0/571920752 conn(0x7f91c408aff0 msgr2=0x7f91c40d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:10.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.831+0000 7f91c9000700 1 -- 172.21.15.89:0/571920752 shutdown_connections 2024-04-02T09:17:10.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.831+0000 7f91c9000700 1 -- 172.21.15.89:0/571920752 wait complete. 2024-04-02T09:17:10.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.832+0000 7f91c9000700 1 Processor -- start 2024-04-02T09:17:10.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.832+0000 7f91c9000700 1 -- start start 2024-04-02T09:17:10.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.833+0000 7f91c9000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91c40d20d0 0x7f91c40ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:10.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.833+0000 7f91c9000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c415d930 0x7f91c40cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:10.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.833+0000 7f91c9000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91c4162030 0x7f91c41f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:10.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.833+0000 7f91c9000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c40cb6e0 con 0x7f91c415d930 2024-04-02T09:17:10.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.833+0000 7f91c9000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c40caed0 con 0x7f91c40d20d0 2024-04-02T09:17:10.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.833+0000 7f91c9000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c4161000 con 0x7f91c4162030 2024-04-02T09:17:10.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.834+0000 7f91c1400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c415d930 0x7f91c40cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:10.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.834+0000 7f91c1400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c415d930 0x7f91c40cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56802/0 (socket says 172.21.15.89:56802) 2024-04-02T09:17:10.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.834+0000 7f91c1400700 1 -- 172.21.15.89:0/1660707410 learned_addr learned my addr 172.21.15.89:0/1660707410 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:10.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.834+0000 7f91c1e00700 1 --2- 172.21.15.89:0/1660707410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91c40d20d0 0x7f91c40ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:10.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.834+0000 7f91c2800700 1 --2- 172.21.15.89:0/1660707410 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91c4162030 0x7f91c41f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:10.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.834+0000 7f91c1400700 1 -- 172.21.15.89:0/1660707410 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91c4162030 msgr2=0x7f91c41f6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:10.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.834+0000 7f91c1400700 1 --2- 172.21.15.89:0/1660707410 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91c4162030 0x7f91c41f6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:10.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.834+0000 7f91c1400700 1 -- 172.21.15.89:0/1660707410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91c40d20d0 msgr2=0x7f91c40ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:10.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.834+0000 7f91c1400700 1 --2- 172.21.15.89:0/1660707410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91c40d20d0 0x7f91c40ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:10.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.834+0000 7f91c1400700 1 -- 172.21.15.89:0/1660707410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b40094a0 con 0x7f91c415d930 2024-04-02T09:17:10.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.834+0000 7f91c1400700 1 --2- 172.21.15.89:0/1660707410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c415d930 0x7f91c40cce00 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7f91b00090b0 tx=0x7f91b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:10.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.835+0000 7f91ba000700 1 -- 172.21.15.89:0/1660707410 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b00036a0 con 0x7f91c415d930 2024-04-02T09:17:10.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.835+0000 7f91ba000700 1 -- 172.21.15.89:0/1660707410 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b001e070 con 0x7f91c415d930 2024-04-02T09:17:10.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.835+0000 7f91ba000700 1 -- 172.21.15.89:0/1660707410 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b0003a80 con 0x7f91c415d930 2024-04-02T09:17:10.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.835+0000 7f91c9000700 1 -- 172.21.15.89:0/1660707410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b00094a0 con 0x7f91c415d930 2024-04-02T09:17:10.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.835+0000 7f91c9000700 1 -- 172.21.15.89:0/1660707410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c40cb400 con 0x7f91c415d930 2024-04-02T09:17:10.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.836+0000 7f91c9000700 1 -- 172.21.15.89:0/1660707410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c4161190 con 0x7f91c415d930 2024-04-02T09:17:10.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.837+0000 7f91ba000700 1 -- 172.21.15.89:0/1660707410 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 190) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f91b0003be0 con 0x7f91c415d930 2024-04-02T09:17:10.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.838+0000 7f91ba000700 1 --2- 172.21.15.89:0/1660707410 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f91a406ad20 0x7f91a406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:10.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.838+0000 7f91ba000700 1 -- 172.21.15.89:0/1660707410 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6536+0+0 (secure 0 0 0) 0x7f91b008a4d0 con 0x7f91c415d930 2024-04-02T09:17:10.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.838+0000 7f91c1e00700 1 --2- 172.21.15.89:0/1660707410 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f91a406ad20 0x7f91a406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:10.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.838+0000 7f91c1e00700 1 --2- 172.21.15.89:0/1660707410 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f91a406ad20 0x7f91a406d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f91b4009d70 tx=0x7f91b4009210 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:10.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:10.842+0000 7f91ba000700 1 -- 172.21.15.89:0/1660707410 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f91b0055410 con 0x7f91c415d930 2024-04-02T09:17:11.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.049+0000 7f91c9000700 1 -- 172.21.15.89:0/1660707410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f91c4067720 con 0x7f91c415d930 2024-04-02T09:17:11.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.612+0000 7f91ba000700 1 -- 172.21.15.89:0/1660707410 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f91b005f080 con 0x7f91c415d930 2024-04-02T09:17:11.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.620+0000 7f91ba000700 1 -- 172.21.15.89:0/1660707410 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v284) v1 ==== 169+0+0 (secure 0 0 0) 0x7f91b006f100 con 0x7f91c415d930 2024-04-02T09:17:11.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.624+0000 7f91c9000700 1 -- 172.21.15.89:0/1660707410 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f91a406ad20 msgr2=0x7f91a406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:11.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.624+0000 7f91c9000700 1 --2- 172.21.15.89:0/1660707410 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f91a406ad20 0x7f91a406d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f91b4009d70 tx=0x7f91b4009210 comp rx=0 tx=0).stop 2024-04-02T09:17:11.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.624+0000 7f91c9000700 1 -- 172.21.15.89:0/1660707410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c415d930 msgr2=0x7f91c40cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:11.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.624+0000 7f91c9000700 1 --2- 172.21.15.89:0/1660707410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c415d930 0x7f91c40cce00 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7f91b00090b0 tx=0x7f91b0004750 comp rx=0 tx=0).stop 2024-04-02T09:17:11.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.624+0000 7f91c9000700 1 -- 172.21.15.89:0/1660707410 shutdown_connections 2024-04-02T09:17:11.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.624+0000 7f91c9000700 1 --2- 172.21.15.89:0/1660707410 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f91a406ad20 0x7f91a406d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:11.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.624+0000 7f91c9000700 1 --2- 172.21.15.89:0/1660707410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91c40d20d0 0x7f91c40ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:11.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.624+0000 7f91c9000700 1 --2- 172.21.15.89:0/1660707410 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91c4162030 0x7f91c41f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:11.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.624+0000 7f91c9000700 1 --2- 172.21.15.89:0/1660707410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91c415d930 0x7f91c40cce00 unknown :-1 s=CLOSED pgs=3084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:11.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.624+0000 7f91c9000700 1 -- 172.21.15.89:0/1660707410 >> 172.21.15.89:0/1660707410 conn(0x7f91c408aff0 msgr2=0x7f91c40b12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:11.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.624+0000 7f91c9000700 1 -- 172.21.15.89:0/1660707410 shutdown_connections 2024-04-02T09:17:11.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.624+0000 7f91c9000700 1 -- 172.21.15.89:0/1660707410 wait complete. 2024-04-02T09:17:11.625 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-04-02T09:17:11.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-02T09:17:11.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.836+0000 7f9531e00700 1 Processor -- start 2024-04-02T09:17:11.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.837+0000 7f9531e00700 1 -- start start 2024-04-02T09:17:11.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.838+0000 7f9531e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f952c0d16a0 0x7f952c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:11.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.838+0000 7f9531e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f952c0d2370 0x7f952c0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:11.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.838+0000 7f9531e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952c161420 con 0x7f952c15d9d0 2024-04-02T09:17:11.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.838+0000 7f9531e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952c0d2820 con 0x7f952c0d16a0 2024-04-02T09:17:11.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.838+0000 7f9531e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c0d4340 con 0x7f952c0d2370 2024-04-02T09:17:11.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.838+0000 7f952a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f952c0d16a0 0x7f952c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:11.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.838+0000 7f952a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f952c0d16a0 0x7f952c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56806/0 (socket says 172.21.15.89:56806) 2024-04-02T09:17:11.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.838+0000 7f952a000700 1 -- 172.21.15.89:0/104307083 learned_addr learned my addr 172.21.15.89:0/104307083 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:11.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.838+0000 7f952a000700 1 -- 172.21.15.89:0/104307083 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f952c0d2370 msgr2=0x7f952c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:11.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.838+0000 7f952b400700 1 --2- 172.21.15.89:0/104307083 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f952c0d2370 0x7f952c0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:11.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.838+0000 7f952a000700 1 --2- 172.21.15.89:0/104307083 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f952c0d2370 0x7f952c0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:11.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.839+0000 7f952a000700 1 -- 172.21.15.89:0/104307083 >> v1:172.21.15.113:6789/0 conn(0x7f952c15d9d0 legacy=0x7f952c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:11.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.839+0000 7f952a000700 1 -- 172.21.15.89:0/104307083 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c161700 con 0x7f952c0d16a0 2024-04-02T09:17:11.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.839+0000 7f952b400700 1 --2- 172.21.15.89:0/104307083 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f952c0d2370 0x7f952c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:17:11.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.839+0000 7f952a000700 1 --2- 172.21.15.89:0/104307083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f952c0d16a0 0x7f952c0d1e30 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7f9518006fd0 tx=0x7f9518009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5361855f019c184 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:11.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.839+0000 7f9529600700 1 -- 172.21.15.89:0/104307083 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9518005e30 con 0x7f952c0d16a0 2024-04-02T09:17:11.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.839+0000 7f9529600700 1 -- 172.21.15.89:0/104307083 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9518004030 con 0x7f952c0d16a0 2024-04-02T09:17:11.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.839+0000 7f9529600700 1 -- 172.21.15.89:0/104307083 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95180043e0 con 0x7f952c0d16a0 2024-04-02T09:17:11.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.840+0000 7f9531e00700 1 -- 172.21.15.89:0/104307083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f952c0d16a0 msgr2=0x7f952c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:11.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.840+0000 7f9531e00700 1 --2- 172.21.15.89:0/104307083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f952c0d16a0 0x7f952c0d1e30 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7f9518006fd0 tx=0x7f9518009bd0 comp rx=0 tx=0).stop 2024-04-02T09:17:11.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.840+0000 7f9531e00700 1 -- 172.21.15.89:0/104307083 shutdown_connections 2024-04-02T09:17:11.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.840+0000 7f9531e00700 1 --2- 172.21.15.89:0/104307083 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f952c0d2370 0x7f952c0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:11.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.840+0000 7f9531e00700 1 --2- 172.21.15.89:0/104307083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f952c0d16a0 0x7f952c0d1e30 unknown :-1 s=CLOSED pgs=3085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:11.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.840+0000 7f9531e00700 1 -- 172.21.15.89:0/104307083 >> 172.21.15.89:0/104307083 conn(0x7f952c08aff0 msgr2=0x7f952c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:11.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.840+0000 7f9531e00700 1 -- 172.21.15.89:0/104307083 shutdown_connections 2024-04-02T09:17:11.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.840+0000 7f9531e00700 1 -- 172.21.15.89:0/104307083 wait complete. 2024-04-02T09:17:11.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.841+0000 7f9531e00700 1 Processor -- start 2024-04-02T09:17:11.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.842+0000 7f9531e00700 1 -- start start 2024-04-02T09:17:11.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.842+0000 7f9531e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f952c0d16a0 0x7f952c0c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:11.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.842+0000 7f9531e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f952c0d2370 0x7f952c0cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:11.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f9531e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f952c15d9d0 0x7f952c0cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:11.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f9531e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c0cd370 con 0x7f952c15d9d0 2024-04-02T09:17:11.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f9531e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c0cb400 con 0x7f952c0d2370 2024-04-02T09:17:11.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f9531e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c0cab60 con 0x7f952c0d16a0 2024-04-02T09:17:11.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f952b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f952c15d9d0 0x7f952c0cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:11.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f952b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f952c15d9d0 0x7f952c0cce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56822/0 (socket says 172.21.15.89:56822) 2024-04-02T09:17:11.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f952b400700 1 -- 172.21.15.89:0/3176052223 learned_addr learned my addr 172.21.15.89:0/3176052223 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:11.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f952aa00700 1 --2- 172.21.15.89:0/3176052223 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f952c0d16a0 0x7f952c0c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:11.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f952a000700 1 --2- 172.21.15.89:0/3176052223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f952c0d2370 0x7f952c0cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:11.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f952b400700 1 -- 172.21.15.89:0/3176052223 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f952c0d16a0 msgr2=0x7f952c0c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:11.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f952b400700 1 --2- 172.21.15.89:0/3176052223 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f952c0d16a0 0x7f952c0c98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:11.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f952b400700 1 -- 172.21.15.89:0/3176052223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f952c0d2370 msgr2=0x7f952c0cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:11.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f952b400700 1 --2- 172.21.15.89:0/3176052223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f952c0d2370 0x7f952c0cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:11.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f952b400700 1 -- 172.21.15.89:0/3176052223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9518009500 con 0x7f952c15d9d0 2024-04-02T09:17:11.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.843+0000 7f952b400700 1 --2- 172.21.15.89:0/3176052223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f952c15d9d0 0x7f952c0cce30 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f952400cef0 tx=0x7f952400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:11.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.844+0000 7f9523400700 1 -- 172.21.15.89:0/3176052223 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952400ae50 con 0x7f952c15d9d0 2024-04-02T09:17:11.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.844+0000 7f9523400700 1 -- 172.21.15.89:0/3176052223 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9524015440 con 0x7f952c15d9d0 2024-04-02T09:17:11.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.844+0000 7f9523400700 1 -- 172.21.15.89:0/3176052223 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95240157f0 con 0x7f952c15d9d0 2024-04-02T09:17:11.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.844+0000 7f9531e00700 1 -- 172.21.15.89:0/3176052223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f952c0cae40 con 0x7f952c15d9d0 2024-04-02T09:17:11.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.844+0000 7f9531e00700 1 -- 172.21.15.89:0/3176052223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952c0b6cc0 con 0x7f952c15d9d0 2024-04-02T09:17:11.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.845+0000 7f9531e00700 1 -- 172.21.15.89:0/3176052223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952c0d1ca0 con 0x7f952c15d9d0 2024-04-02T09:17:11.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.846+0000 7f9523400700 1 -- 172.21.15.89:0/3176052223 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f952400d430 con 0x7f952c15d9d0 2024-04-02T09:17:11.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.847+0000 7f9523400700 1 --2- 172.21.15.89:0/3176052223 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f950c06adb0 0x7f950c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:11.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.847+0000 7f9523400700 1 -- 172.21.15.89:0/3176052223 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f9524089c10 con 0x7f952c15d9d0 2024-04-02T09:17:11.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.847+0000 7f952aa00700 1 --2- 172.21.15.89:0/3176052223 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f950c06adb0 0x7f950c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:11.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.848+0000 7f952aa00700 1 --2- 172.21.15.89:0/3176052223 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f950c06adb0 0x7f950c06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f951c000f60 tx=0x7f951c009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:11.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:11.852+0000 7f9523400700 1 -- 172.21.15.89:0/3176052223 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9524054b40 con 0x7f952c15d9d0 2024-04-02T09:17:12.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.058+0000 7f9531e00700 1 -- 172.21.15.89:0/3176052223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f952c0ce880 con 0x7f952c15d9d0 2024-04-02T09:17:12.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.062+0000 7f9523400700 1 -- 172.21.15.89:0/3176052223 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v97) v1 ==== 115+0+0 (secure 0 0 0) 0x7f952c0ce880 con 0x7f952c15d9d0 2024-04-02T09:17:12.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.065+0000 7f9531e00700 1 -- 172.21.15.89:0/3176052223 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f950c06adb0 msgr2=0x7f950c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.065+0000 7f9531e00700 1 --2- 172.21.15.89:0/3176052223 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f950c06adb0 0x7f950c06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f951c000f60 tx=0x7f951c009040 comp rx=0 tx=0).stop 2024-04-02T09:17:12.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.065+0000 7f9531e00700 1 -- 172.21.15.89:0/3176052223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f952c15d9d0 msgr2=0x7f952c0cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.065+0000 7f9531e00700 1 --2- 172.21.15.89:0/3176052223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f952c15d9d0 0x7f952c0cce30 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f952400cef0 tx=0x7f952400a340 comp rx=0 tx=0).stop 2024-04-02T09:17:12.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.065+0000 7f9531e00700 1 -- 172.21.15.89:0/3176052223 shutdown_connections 2024-04-02T09:17:12.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.065+0000 7f9531e00700 1 --2- 172.21.15.89:0/3176052223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f952c15d9d0 0x7f952c0cce30 unknown :-1 s=CLOSED pgs=3086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.065+0000 7f9531e00700 1 --2- 172.21.15.89:0/3176052223 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f952c0d16a0 0x7f952c0c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.065+0000 7f9531e00700 1 --2- 172.21.15.89:0/3176052223 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f950c06adb0 0x7f950c06d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.065+0000 7f9531e00700 1 --2- 172.21.15.89:0/3176052223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f952c0d2370 0x7f952c0cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.065+0000 7f9531e00700 1 -- 172.21.15.89:0/3176052223 >> 172.21.15.89:0/3176052223 conn(0x7f952c08aff0 msgr2=0x7f952c0b12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:12.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.065+0000 7f9531e00700 1 -- 172.21.15.89:0/3176052223 shutdown_connections 2024-04-02T09:17:12.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.065+0000 7f9531e00700 1 -- 172.21.15.89:0/3176052223 wait complete. 2024-04-02T09:17:12.083 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-02T09:17:12.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.282+0000 7f3264600700 1 Processor -- start 2024-04-02T09:17:12.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.282+0000 7f3264600700 1 -- start start 2024-04-02T09:17:12.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.283+0000 7f3264600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325c0d16a0 0x7f325c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.283+0000 7f3264600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325c161420 con 0x7f325c15d9d0 2024-04-02T09:17:12.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.283+0000 7f3264600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325c161700 con 0x7f325c0d2370 2024-04-02T09:17:12.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.283+0000 7f3264600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0d2820 con 0x7f325c0d16a0 2024-04-02T09:17:12.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.284+0000 7f3261c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f325c15d9d0 0x7f325c0d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59570/0 (socket says 172.21.15.89:59570) 2024-04-02T09:17:12.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.284+0000 7f3261c00700 1 -- 172.21.15.89:0/1001423474 learned_addr learned my addr 172.21.15.89:0/1001423474 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:12.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.284+0000 7f3261200700 1 --2- 172.21.15.89:0/1001423474 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325c0d16a0 0x7f325c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.284+0000 7f325be00700 1 -- 172.21.15.89:0/1001423474 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162906507 0 0) 0x7f325c161420 con 0x7f325c15d9d0 2024-04-02T09:17:12.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.284+0000 7f325be00700 1 -- 172.21.15.89:0/1001423474 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32440035b0 con 0x7f325c15d9d0 2024-04-02T09:17:12.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.284+0000 7f325be00700 1 -- 172.21.15.89:0/1001423474 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2206969406 0 0) 0x7f32440035b0 con 0x7f325c15d9d0 2024-04-02T09:17:12.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.284+0000 7f325be00700 1 -- 172.21.15.89:0/1001423474 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325c0d16a0 msgr2=0x7f325c0d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:12.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.284+0000 7f325be00700 1 --2- 172.21.15.89:0/1001423474 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325c0d16a0 0x7f325c0d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.284+0000 7f325be00700 1 -- 172.21.15.89:0/1001423474 >> v1:172.21.15.113:6789/0 conn(0x7f325c0d2370 legacy=0x7f325c0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.284+0000 7f325be00700 1 -- 172.21.15.89:0/1001423474 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c1619e0 con 0x7f325c15d9d0 2024-04-02T09:17:12.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.284+0000 7f325be00700 1 -- 172.21.15.89:0/1001423474 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f3250001670 con 0x7f325c15d9d0 2024-04-02T09:17:12.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.284+0000 7f325be00700 1 -- 172.21.15.89:0/1001423474 >> v1:172.21.15.89:6789/0 conn(0x7f325c15d9d0 legacy=0x7f325c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f325be00700 1 --2- 172.21.15.89:0/1001423474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244003a30 0x7f3244003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f325be00700 1 --2- 172.21.15.89:0/1001423474 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325c0d16a0 0x7f3244004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f325be00700 1 --2- 172.21.15.89:0/1001423474 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3244004d90 0x7f3244005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f325be00700 1 -- 172.21.15.89:0/1001423474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0d2820 con 0x7f3244003a30 2024-04-02T09:17:12.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f325be00700 1 -- 172.21.15.89:0/1001423474 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324400bac0 con 0x7f3244004d90 2024-04-02T09:17:12.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f325be00700 1 -- 172.21.15.89:0/1001423474 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324400bc80 con 0x7f325c0d16a0 2024-04-02T09:17:12.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f3261c00700 1 --2- 172.21.15.89:0/1001423474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244003a30 0x7f3244003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f3261200700 1 --2- 172.21.15.89:0/1001423474 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325c0d16a0 0x7f3244004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f3262600700 1 --2- 172.21.15.89:0/1001423474 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3244004d90 0x7f3244005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f3261c00700 1 -- 172.21.15.89:0/1001423474 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325c0d16a0 msgr2=0x7f3244004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:12.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f3261c00700 1 --2- 172.21.15.89:0/1001423474 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325c0d16a0 0x7f3244004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f3261c00700 1 -- 172.21.15.89:0/1001423474 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3244004d90 msgr2=0x7f3244005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:12.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f3261c00700 1 --2- 172.21.15.89:0/1001423474 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3244004d90 0x7f3244005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f3261c00700 1 -- 172.21.15.89:0/1001423474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32440035b0 con 0x7f3244003a30 2024-04-02T09:17:12.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.285+0000 7f3261c00700 1 --2- 172.21.15.89:0/1001423474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244003a30 0x7f3244003e50 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7f325000c210 tx=0x7f3244005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f4049198e3505 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:12.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.286+0000 7f325be00700 1 -- 172.21.15.89:0/1001423474 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250007c10 con 0x7f3244003a30 2024-04-02T09:17:12.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.286+0000 7f325be00700 1 -- 172.21.15.89:0/1001423474 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325000cdd0 con 0x7f3244003a30 2024-04-02T09:17:12.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.286+0000 7f325be00700 1 -- 172.21.15.89:0/1001423474 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250005ba0 con 0x7f3244003a30 2024-04-02T09:17:12.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.286+0000 7f3264600700 1 -- 172.21.15.89:0/1001423474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244003a30 msgr2=0x7f3244003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.286+0000 7f3264600700 1 --2- 172.21.15.89:0/1001423474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244003a30 0x7f3244003e50 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7f325000c210 tx=0x7f3244005880 comp rx=0 tx=0).stop 2024-04-02T09:17:12.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.286+0000 7f3262600700 1 -- 172.21.15.89:0/1001423474 reap_dead start 2024-04-02T09:17:12.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.286+0000 7f3264600700 1 -- 172.21.15.89:0/1001423474 shutdown_connections 2024-04-02T09:17:12.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.286+0000 7f3264600700 1 -- 172.21.15.89:0/1001423474 >> 172.21.15.89:0/1001423474 conn(0x7f325c08aff0 msgr2=0x7f325c0d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:12.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.286+0000 7f3264600700 1 -- 172.21.15.89:0/1001423474 shutdown_connections 2024-04-02T09:17:12.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.286+0000 7f3264600700 1 -- 172.21.15.89:0/1001423474 wait complete. 2024-04-02T09:17:12.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.288+0000 7f3264600700 1 Processor -- start 2024-04-02T09:17:12.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.288+0000 7f3264600700 1 -- start start 2024-04-02T09:17:12.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.289+0000 7f3264600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244003a30 0x7f325c0c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.289+0000 7f3264600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325c15a550 0x7f325c0cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.289+0000 7f3264600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f325c0cc340 0x7f325c0cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.289+0000 7f3264600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0cb420 con 0x7f3244003a30 2024-04-02T09:17:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.289+0000 7f3264600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0ca1c0 con 0x7f325c0cc340 2024-04-02T09:17:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.289+0000 7f3264600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0b7300 con 0x7f325c15a550 2024-04-02T09:17:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.289+0000 7f3261c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244003a30 0x7f325c0c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.289+0000 7f3261c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244003a30 0x7f325c0c9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56844/0 (socket says 172.21.15.89:56844) 2024-04-02T09:17:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.289+0000 7f3261c00700 1 -- 172.21.15.89:0/1938254943 learned_addr learned my addr 172.21.15.89:0/1938254943 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.290+0000 7f3261c00700 1 -- 172.21.15.89:0/1938254943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325c15a550 msgr2=0x7f325c0cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.290+0000 7f3261200700 1 --2- 172.21.15.89:0/1938254943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325c15a550 0x7f325c0cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.290+0000 7f3261c00700 1 --2- 172.21.15.89:0/1938254943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325c15a550 0x7f325c0cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.290+0000 7f3261c00700 1 -- 172.21.15.89:0/1938254943 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f325c0cc340 msgr2=0x7f325c0cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.290+0000 7f3261c00700 1 --2- 172.21.15.89:0/1938254943 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f325c0cc340 0x7f325c0cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.290+0000 7f3261c00700 1 -- 172.21.15.89:0/1938254943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c0b6d40 con 0x7f3244003a30 2024-04-02T09:17:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.290+0000 7f3261c00700 1 --2- 172.21.15.89:0/1938254943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244003a30 0x7f325c0c9c80 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7f32440047b0 tx=0x7f3250001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.290+0000 7f3259600700 1 -- 172.21.15.89:0/1938254943 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250001630 con 0x7f3244003a30 2024-04-02T09:17:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.290+0000 7f3259600700 1 -- 172.21.15.89:0/1938254943 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32500070d0 con 0x7f3244003a30 2024-04-02T09:17:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.290+0000 7f3264600700 1 -- 172.21.15.89:0/1938254943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c0b7020 con 0x7f3244003a30 2024-04-02T09:17:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.290+0000 7f3259600700 1 -- 172.21.15.89:0/1938254943 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32500074a0 con 0x7f3244003a30 2024-04-02T09:17:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.290+0000 7f3264600700 1 -- 172.21.15.89:0/1938254943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325c0cb8e0 con 0x7f3244003a30 2024-04-02T09:17:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.291+0000 7f3264600700 1 -- 172.21.15.89:0/1938254943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325c0cbb00 con 0x7f3244003a30 2024-04-02T09:17:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.292+0000 7f3259600700 1 -- 172.21.15.89:0/1938254943 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3250019030 con 0x7f3244003a30 2024-04-02T09:17:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.293+0000 7f3259600700 1 --2- 172.21.15.89:0/1938254943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f323c06adb0 0x7f323c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.293+0000 7f3259600700 1 -- 172.21.15.89:0/1938254943 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f325008a750 con 0x7f3244003a30 2024-04-02T09:17:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.293+0000 7f3261200700 1 --2- 172.21.15.89:0/1938254943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f323c06adb0 0x7f323c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.294+0000 7f3261200700 1 --2- 172.21.15.89:0/1938254943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f323c06adb0 0x7f323c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f325c0b6720 tx=0x7f3248008040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:12.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.297+0000 7f3259600700 1 -- 172.21.15.89:0/1938254943 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3250055710 con 0x7f3244003a30 2024-04-02T09:17:12.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.501+0000 7f3264600700 1 -- 172.21.15.89:0/1938254943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f325c0cb420 con 0x7f3244003a30 2024-04-02T09:17:12.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.506+0000 7f3259600700 1 -- 172.21.15.89:0/1938254943 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v98) v1 ==== 120+0+0 (secure 0 0 0) 0x7f3250058d30 con 0x7f3244003a30 2024-04-02T09:17:12.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.508+0000 7f3264600700 1 -- 172.21.15.89:0/1938254943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f323c06adb0 msgr2=0x7f323c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.508+0000 7f3264600700 1 --2- 172.21.15.89:0/1938254943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f323c06adb0 0x7f323c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f325c0b6720 tx=0x7f3248008040 comp rx=0 tx=0).stop 2024-04-02T09:17:12.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.508+0000 7f3264600700 1 -- 172.21.15.89:0/1938254943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244003a30 msgr2=0x7f325c0c9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.508+0000 7f3264600700 1 --2- 172.21.15.89:0/1938254943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244003a30 0x7f325c0c9c80 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7f32440047b0 tx=0x7f3250001a00 comp rx=0 tx=0).stop 2024-04-02T09:17:12.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.508+0000 7f3264600700 1 -- 172.21.15.89:0/1938254943 shutdown_connections 2024-04-02T09:17:12.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.508+0000 7f3264600700 1 --2- 172.21.15.89:0/1938254943 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f323c06adb0 0x7f323c06d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.508+0000 7f3264600700 1 --2- 172.21.15.89:0/1938254943 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f325c0cc340 0x7f325c0cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.508+0000 7f3264600700 1 --2- 172.21.15.89:0/1938254943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244003a30 0x7f325c0c9c80 unknown :-1 s=CLOSED pgs=3089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.508+0000 7f3264600700 1 --2- 172.21.15.89:0/1938254943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325c15a550 0x7f325c0cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.509+0000 7f3264600700 1 -- 172.21.15.89:0/1938254943 >> 172.21.15.89:0/1938254943 conn(0x7f325c08aff0 msgr2=0x7f325c0ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:12.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.509+0000 7f3264600700 1 -- 172.21.15.89:0/1938254943 shutdown_connections 2024-04-02T09:17:12.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.509+0000 7f3264600700 1 -- 172.21.15.89:0/1938254943 wait complete. 2024-04-02T09:17:12.526 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-02T09:17:12.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.721+0000 7f4b8ba00700 1 Processor -- start 2024-04-02T09:17:12.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.721+0000 7f4b8ba00700 1 -- start start 2024-04-02T09:17:12.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.722+0000 7f4b8ba00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4b840d16a0 0x7f4b840d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.722+0000 7f4b8ba00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b84161420 con 0x7f4b840d2370 2024-04-02T09:17:12.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.722+0000 7f4b8ba00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b84161700 con 0x7f4b8415d9d0 2024-04-02T09:17:12.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.722+0000 7f4b8ba00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b840d4340 con 0x7f4b840d16a0 2024-04-02T09:17:12.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.722+0000 7f4b89a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4b840d2370 0x7f4b840ce360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59578/0 (socket says 172.21.15.89:59578) 2024-04-02T09:17:12.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.722+0000 7f4b89a00700 1 -- 172.21.15.89:0/3512461291 learned_addr learned my addr 172.21.15.89:0/3512461291 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83e00700 1 --2- 172.21.15.89:0/3512461291 >> v2:172.21.15.113:3300/0 conn(0x7f4b840d16a0 0x7f4b840d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 -- 172.21.15.89:0/3512461291 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472570731 0 0) 0x7f4b84161420 con 0x7f4b840d2370 2024-04-02T09:17:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 -- 172.21.15.89:0/3512461291 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b6c0035b0 con 0x7f4b840d2370 2024-04-02T09:17:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 -- 172.21.15.89:0/3512461291 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2677830972 0 0) 0x7f4b6c0035b0 con 0x7f4b840d2370 2024-04-02T09:17:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 -- 172.21.15.89:0/3512461291 >> v2:172.21.15.113:3300/0 conn(0x7f4b840d16a0 msgr2=0x7f4b840d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 --2- 172.21.15.89:0/3512461291 >> v2:172.21.15.113:3300/0 conn(0x7f4b840d16a0 0x7f4b840d1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 -- 172.21.15.89:0/3512461291 >> v1:172.21.15.113:6789/0 conn(0x7f4b8415d9d0 legacy=0x7f4b840d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 -- 172.21.15.89:0/3512461291 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b841619e0 con 0x7f4b840d2370 2024-04-02T09:17:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 -- 172.21.15.89:0/3512461291 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f4b7c0019f0 con 0x7f4b840d2370 2024-04-02T09:17:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 -- 172.21.15.89:0/3512461291 >> v1:172.21.15.89:6789/0 conn(0x7f4b840d2370 legacy=0x7f4b840ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 --2- 172.21.15.89:0/3512461291 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b6c003a50 0x7f4b6c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 --2- 172.21.15.89:0/3512461291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b6c004850 0x7f4b6c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 --2- 172.21.15.89:0/3512461291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b6c005210 0x7f4b6c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 -- 172.21.15.89:0/3512461291 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c00c0e0 con 0x7f4b6c005210 2024-04-02T09:17:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 -- 172.21.15.89:0/3512461291 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c00c250 con 0x7f4b6c004850 2024-04-02T09:17:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b83400700 1 -- 172.21.15.89:0/3512461291 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c00c410 con 0x7f4b6c003a50 2024-04-02T09:17:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.723+0000 7f4b89a00700 1 --2- 172.21.15.89:0/3512461291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b6c005210 0x7f4b6c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.724+0000 7f4b83e00700 1 --2- 172.21.15.89:0/3512461291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b6c004850 0x7f4b6c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.724+0000 7f4b89000700 1 --2- 172.21.15.89:0/3512461291 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b6c003a50 0x7f4b6c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.724+0000 7f4b89a00700 1 -- 172.21.15.89:0/3512461291 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b6c003a50 msgr2=0x7f4b6c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.724+0000 7f4b89a00700 1 --2- 172.21.15.89:0/3512461291 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b6c003a50 0x7f4b6c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.724+0000 7f4b89a00700 1 -- 172.21.15.89:0/3512461291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b6c004850 msgr2=0x7f4b6c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.724+0000 7f4b89a00700 1 --2- 172.21.15.89:0/3512461291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b6c004850 0x7f4b6c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.724+0000 7f4b89a00700 1 -- 172.21.15.89:0/3512461291 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c0035b0 con 0x7f4b6c005210 2024-04-02T09:17:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.724+0000 7f4b89a00700 1 -- 172.21.15.89:0/3512461291 reap_dead start 2024-04-02T09:17:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.724+0000 7f4b89a00700 1 --2- 172.21.15.89:0/3512461291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b6c005210 0x7f4b6c00ba10 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00a750 tx=0x7f4b6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdadb9705a1f8f7c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.724+0000 7f4b83400700 1 -- 172.21.15.89:0/3512461291 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c00a210 con 0x7f4b6c005210 2024-04-02T09:17:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.724+0000 7f4b83400700 1 -- 172.21.15.89:0/3512461291 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7c007310 con 0x7f4b6c005210 2024-04-02T09:17:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.724+0000 7f4b83400700 1 -- 172.21.15.89:0/3512461291 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0076c0 con 0x7f4b6c005210 2024-04-02T09:17:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.725+0000 7f4b8ba00700 1 -- 172.21.15.89:0/3512461291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b6c005210 msgr2=0x7f4b6c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.725+0000 7f4b8ba00700 1 --2- 172.21.15.89:0/3512461291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b6c005210 0x7f4b6c00ba10 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00a750 tx=0x7f4b6c005880 comp rx=0 tx=0).stop 2024-04-02T09:17:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.725+0000 7f4b8ba00700 1 -- 172.21.15.89:0/3512461291 shutdown_connections 2024-04-02T09:17:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.725+0000 7f4b8ba00700 1 --2- 172.21.15.89:0/3512461291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b6c005210 0x7f4b6c00ba10 unknown :-1 s=CLOSED pgs=3091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.725+0000 7f4b8ba00700 1 -- 172.21.15.89:0/3512461291 >> 172.21.15.89:0/3512461291 conn(0x7f4b8408aff0 msgr2=0x7f4b840d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.725+0000 7f4b8ba00700 1 -- 172.21.15.89:0/3512461291 shutdown_connections 2024-04-02T09:17:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.725+0000 7f4b8ba00700 1 -- 172.21.15.89:0/3512461291 wait complete. 2024-04-02T09:17:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.727+0000 7f4b8ba00700 1 Processor -- start 2024-04-02T09:17:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.727+0000 7f4b8ba00700 1 -- start start 2024-04-02T09:17:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b8ba00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b6c005210 0x7f4b840b7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b8ba00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b840b8f80 0x7f4b841f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b8ba00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b841f36a0 0x7f4b841f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b8ba00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8415ddb0 con 0x7f4b841f36a0 2024-04-02T09:17:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b8ba00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b841f4f50 con 0x7f4b840b8f80 2024-04-02T09:17:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b8ba00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b841f4060 con 0x7f4b6c005210 2024-04-02T09:17:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b89a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b841f36a0 0x7f4b841f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b89a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b841f36a0 0x7f4b841f3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56860/0 (socket says 172.21.15.89:56860) 2024-04-02T09:17:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b89a00700 1 -- 172.21.15.89:0/894155277 learned_addr learned my addr 172.21.15.89:0/894155277 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b83e00700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b840b8f80 0x7f4b841f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b89000700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b6c005210 0x7f4b840b7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b89a00700 1 -- 172.21.15.89:0/894155277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b6c005210 msgr2=0x7f4b840b7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b89a00700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b6c005210 0x7f4b840b7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b89a00700 1 -- 172.21.15.89:0/894155277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b840b8f80 msgr2=0x7f4b841f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b89a00700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b840b8f80 0x7f4b841f3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b89a00700 1 -- 172.21.15.89:0/894155277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b840b77d0 con 0x7f4b841f36a0 2024-04-02T09:17:12.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.728+0000 7f4b83e00700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b840b8f80 0x7f4b841f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:17:12.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.729+0000 7f4b89a00700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b841f36a0 0x7f4b841f3b20 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 crypto rx=0x7f4b840cdc30 tx=0x7f4b841615f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:12.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.729+0000 7f4b80c00700 1 -- 172.21.15.89:0/894155277 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c001250 con 0x7f4b841f36a0 2024-04-02T09:17:12.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.729+0000 7f4b80c00700 1 -- 172.21.15.89:0/894155277 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7c01d410 con 0x7f4b841f36a0 2024-04-02T09:17:12.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.729+0000 7f4b80c00700 1 -- 172.21.15.89:0/894155277 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c01d780 con 0x7f4b841f36a0 2024-04-02T09:17:12.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.729+0000 7f4b8ba00700 1 -- 172.21.15.89:0/894155277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b841f4360 con 0x7f4b841f36a0 2024-04-02T09:17:12.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.729+0000 7f4b8ba00700 1 -- 172.21.15.89:0/894155277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b841f4900 con 0x7f4b841f36a0 2024-04-02T09:17:12.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.730+0000 7f4b8ba00700 1 -- 172.21.15.89:0/894155277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b841f50c0 con 0x7f4b841f36a0 2024-04-02T09:17:12.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.731+0000 7f4b80c00700 1 -- 172.21.15.89:0/894155277 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4b7c01a030 con 0x7f4b841f36a0 2024-04-02T09:17:12.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.732+0000 7f4b80c00700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4b6406adb0 0x7f4b6406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:12.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.732+0000 7f4b80c00700 1 -- 172.21.15.89:0/894155277 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f4b7c08ad00 con 0x7f4b841f36a0 2024-04-02T09:17:12.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.732+0000 7f4b89000700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4b6406adb0 0x7f4b6406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:12.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.733+0000 7f4b89000700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4b6406adb0 0x7f4b6406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4b78007fd0 tx=0x7f4b78009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.736+0000 7f4b80c00700 1 -- 172.21.15.89:0/894155277 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4b7c055cc0 con 0x7f4b841f36a0 2024-04-02T09:17:12.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.936+0000 7f4b8ba00700 1 -- 172.21.15.89:0/894155277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4b84067720 con 0x7f4b841f36a0 2024-04-02T09:17:12.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.940+0000 7f4b80c00700 1 -- 172.21.15.89:0/894155277 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v99) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4b7c0180a0 con 0x7f4b841f36a0 2024-04-02T09:17:12.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.943+0000 7f4b8ba00700 1 -- 172.21.15.89:0/894155277 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4b6406adb0 msgr2=0x7f4b6406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.943+0000 7f4b8ba00700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4b6406adb0 0x7f4b6406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4b78007fd0 tx=0x7f4b78009040 comp rx=0 tx=0).stop 2024-04-02T09:17:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.943+0000 7f4b8ba00700 1 -- 172.21.15.89:0/894155277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b841f36a0 msgr2=0x7f4b841f3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.943+0000 7f4b8ba00700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b841f36a0 0x7f4b841f3b20 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 crypto rx=0x7f4b840cdc30 tx=0x7f4b841615f0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.943+0000 7f4b8ba00700 1 -- 172.21.15.89:0/894155277 shutdown_connections 2024-04-02T09:17:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.943+0000 7f4b8ba00700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b841f36a0 0x7f4b841f3b20 unknown :-1 s=CLOSED pgs=3092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.943+0000 7f4b8ba00700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b6c005210 0x7f4b840b7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.943+0000 7f4b8ba00700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f4b6406adb0 0x7f4b6406d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.943+0000 7f4b8ba00700 1 --2- 172.21.15.89:0/894155277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b840b8f80 0x7f4b841f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.943+0000 7f4b8ba00700 1 -- 172.21.15.89:0/894155277 >> 172.21.15.89:0/894155277 conn(0x7f4b8408aff0 msgr2=0x7f4b840b14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.943+0000 7f4b8ba00700 1 -- 172.21.15.89:0/894155277 shutdown_connections 2024-04-02T09:17:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:12.943+0000 7f4b8ba00700 1 -- 172.21.15.89:0/894155277 wait complete. 2024-04-02T09:17:12.960 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01' 2024-04-02T09:17:13.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.152+0000 7ff28ec00700 1 Processor -- start 2024-04-02T09:17:13.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.152+0000 7ff28ec00700 1 -- start start 2024-04-02T09:17:13.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.153+0000 7ff28ec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815b5d0 0x7ff2880ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:13.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.153+0000 7ff28ec00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff2880ac7d0 0x7ff28806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:13.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.154+0000 7ff28ec00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28815fd50 con 0x7ff2880cb3f0 2024-04-02T09:17:13.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.154+0000 7ff28ec00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff28815b9b0 con 0x7ff2880ac7d0 2024-04-02T09:17:13.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.154+0000 7ff28ec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28815f140 con 0x7ff28815b5d0 2024-04-02T09:17:13.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.154+0000 7ff28cc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff2880ac7d0 0x7ff28806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:13.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.154+0000 7ff28cc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff2880ac7d0 0x7ff28806c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56874/0 (socket says 172.21.15.89:56874) 2024-04-02T09:17:13.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.154+0000 7ff28cc00700 1 -- 172.21.15.89:0/3434744161 learned_addr learned my addr 172.21.15.89:0/3434744161 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:13.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.154+0000 7ff287e00700 1 --2- 172.21.15.89:0/3434744161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815b5d0 0x7ff2880ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:13.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.154+0000 7ff28cc00700 1 -- 172.21.15.89:0/3434744161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815b5d0 msgr2=0x7ff2880ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:13.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.154+0000 7ff28cc00700 1 --2- 172.21.15.89:0/3434744161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815b5d0 0x7ff2880ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:13.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.154+0000 7ff28cc00700 1 -- 172.21.15.89:0/3434744161 >> v1:172.21.15.113:6789/0 conn(0x7ff2880cb3f0 legacy=0x7ff2880ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:13.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.154+0000 7ff28cc00700 1 -- 172.21.15.89:0/3434744161 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28815f420 con 0x7ff2880ac7d0 2024-04-02T09:17:13.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.154+0000 7ff28cc00700 1 --2- 172.21.15.89:0/3434744161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2880ac7d0 0x7ff28806c9d0 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 crypto rx=0x7ff280009e50 tx=0x7ff28000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9bb4375e3be215 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:13.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.155+0000 7ff286a00700 1 -- 172.21.15.89:0/3434744161 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28000e070 con 0x7ff2880ac7d0 2024-04-02T09:17:13.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.155+0000 7ff286a00700 1 -- 172.21.15.89:0/3434744161 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28000b9c0 con 0x7ff2880ac7d0 2024-04-02T09:17:13.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.155+0000 7ff286a00700 1 -- 172.21.15.89:0/3434744161 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28000bd70 con 0x7ff2880ac7d0 2024-04-02T09:17:13.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.155+0000 7ff28ec00700 1 -- 172.21.15.89:0/3434744161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2880ac7d0 msgr2=0x7ff28806c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:13.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.155+0000 7ff28ec00700 1 --2- 172.21.15.89:0/3434744161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2880ac7d0 0x7ff28806c9d0 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 crypto rx=0x7ff280009e50 tx=0x7ff28000b330 comp rx=0 tx=0).stop 2024-04-02T09:17:13.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.155+0000 7ff28ec00700 1 -- 172.21.15.89:0/3434744161 shutdown_connections 2024-04-02T09:17:13.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.155+0000 7ff28ec00700 1 --2- 172.21.15.89:0/3434744161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff28815b5d0 0x7ff2880ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:13.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.155+0000 7ff28ec00700 1 --2- 172.21.15.89:0/3434744161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2880ac7d0 0x7ff28806c9d0 unknown :-1 s=CLOSED pgs=3093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:13.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.155+0000 7ff28ec00700 1 -- 172.21.15.89:0/3434744161 >> 172.21.15.89:0/3434744161 conn(0x7ff2880cf190 msgr2=0x7ff2880cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:13.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.155+0000 7ff28ec00700 1 -- 172.21.15.89:0/3434744161 shutdown_connections 2024-04-02T09:17:13.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.155+0000 7ff28ec00700 1 -- 172.21.15.89:0/3434744161 wait complete. 2024-04-02T09:17:13.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.157+0000 7ff28ec00700 1 Processor -- start 2024-04-02T09:17:13.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.157+0000 7ff28ec00700 1 -- start start 2024-04-02T09:17:13.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.158+0000 7ff28ec00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2880ac7d0 0x7ff288072540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:13.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.158+0000 7ff28ec00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28815b5d0 0x7ff288071d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:13.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.158+0000 7ff28ec00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff28815a690 0x7ff2880ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:13.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.158+0000 7ff28ec00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2880ce890 con 0x7ff28815a690 2024-04-02T09:17:13.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.158+0000 7ff28ec00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28815aba0 con 0x7ff28815b5d0 2024-04-02T09:17:13.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.158+0000 7ff28ec00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2880722d0 con 0x7ff2880ac7d0 2024-04-02T09:17:13.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.158+0000 7ff28cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff28815a690 0x7ff2880ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:13.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.158+0000 7ff28cc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff28815a690 0x7ff2880ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56884/0 (socket says 172.21.15.89:56884) 2024-04-02T09:17:13.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.158+0000 7ff28cc00700 1 -- 172.21.15.89:0/2974418414 learned_addr learned my addr 172.21.15.89:0/2974418414 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:13.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.158+0000 7ff287400700 1 --2- 172.21.15.89:0/2974418414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28815b5d0 0x7ff288071d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:13.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.159+0000 7ff287e00700 1 --2- 172.21.15.89:0/2974418414 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2880ac7d0 0x7ff288072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:13.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.159+0000 7ff28cc00700 1 -- 172.21.15.89:0/2974418414 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2880ac7d0 msgr2=0x7ff288072540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:13.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.159+0000 7ff28cc00700 1 --2- 172.21.15.89:0/2974418414 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2880ac7d0 0x7ff288072540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:13.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.159+0000 7ff28cc00700 1 -- 172.21.15.89:0/2974418414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28815b5d0 msgr2=0x7ff288071d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:13.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.159+0000 7ff28cc00700 1 --2- 172.21.15.89:0/2974418414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28815b5d0 0x7ff288071d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:13.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.159+0000 7ff28cc00700 1 -- 172.21.15.89:0/2974418414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff280009b30 con 0x7ff28815a690 2024-04-02T09:17:13.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.159+0000 7ff28cc00700 1 --2- 172.21.15.89:0/2974418414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff28815a690 0x7ff2880ccd00 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7ff280012000 tx=0x7ff280004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:13.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.159+0000 7ff27fe00700 1 -- 172.21.15.89:0/2974418414 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff280004960 con 0x7ff28815a690 2024-04-02T09:17:13.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.159+0000 7ff27fe00700 1 -- 172.21.15.89:0/2974418414 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2800099b0 con 0x7ff28815a690 2024-04-02T09:17:13.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.159+0000 7ff27fe00700 1 -- 172.21.15.89:0/2974418414 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28001c680 con 0x7ff28815a690 2024-04-02T09:17:13.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.159+0000 7ff28ec00700 1 -- 172.21.15.89:0/2974418414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff288152990 con 0x7ff28815a690 2024-04-02T09:17:13.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.159+0000 7ff28ec00700 1 -- 172.21.15.89:0/2974418414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff288152f50 con 0x7ff28815a690 2024-04-02T09:17:13.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.160+0000 7ff28ec00700 1 -- 172.21.15.89:0/2974418414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff288067720 con 0x7ff28815a690 2024-04-02T09:17:13.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.161+0000 7ff27fe00700 1 -- 172.21.15.89:0/2974418414 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff28001c7e0 con 0x7ff28815a690 2024-04-02T09:17:13.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.162+0000 7ff27fe00700 1 --2- 172.21.15.89:0/2974418414 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff26806ade0 0x7ff26806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:13.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.163+0000 7ff27fe00700 1 -- 172.21.15.89:0/2974418414 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7ff28008b140 con 0x7ff28815a690 2024-04-02T09:17:13.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.163+0000 7ff287e00700 1 --2- 172.21.15.89:0/2974418414 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff26806ade0 0x7ff26806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:13.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.163+0000 7ff287e00700 1 --2- 172.21.15.89:0/2974418414 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff26806ade0 0x7ff26806d2a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff27800b9b0 tx=0x7ff278008040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:13.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.167+0000 7ff27fe00700 1 -- 172.21.15.89:0/2974418414 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff280055f50 con 0x7ff28815a690 2024-04-02T09:17:13.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.455+0000 7ff28ec00700 1 -- 172.21.15.89:0/2974418414 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]} v 0) v1 -- 0x7ff28806ea60 con 0x7ff28815a690 2024-04-02T09:17:13.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.613+0000 7ff27fe00700 1 -- 172.21.15.89:0/2974418414 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]}]=0 v3352) v1 ==== 201+0+0 (secure 0 0 0) 0x7ff280059570 con 0x7ff28815a690 2024-04-02T09:17:13.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.616+0000 7ff28ec00700 1 -- 172.21.15.89:0/2974418414 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff26806ade0 msgr2=0x7ff26806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:13.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.616+0000 7ff28ec00700 1 --2- 172.21.15.89:0/2974418414 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff26806ade0 0x7ff26806d2a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff27800b9b0 tx=0x7ff278008040 comp rx=0 tx=0).stop 2024-04-02T09:17:13.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.616+0000 7ff28ec00700 1 -- 172.21.15.89:0/2974418414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff28815a690 msgr2=0x7ff2880ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:13.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.616+0000 7ff28ec00700 1 --2- 172.21.15.89:0/2974418414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff28815a690 0x7ff2880ccd00 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7ff280012000 tx=0x7ff280004ad0 comp rx=0 tx=0).stop 2024-04-02T09:17:13.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.616+0000 7ff28ec00700 1 -- 172.21.15.89:0/2974418414 shutdown_connections 2024-04-02T09:17:13.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.616+0000 7ff28ec00700 1 --2- 172.21.15.89:0/2974418414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff28815a690 0x7ff2880ccd00 unknown :-1 s=CLOSED pgs=3094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:13.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.616+0000 7ff28ec00700 1 --2- 172.21.15.89:0/2974418414 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2880ac7d0 0x7ff288072540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:13.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.616+0000 7ff28ec00700 1 --2- 172.21.15.89:0/2974418414 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff26806ade0 0x7ff26806d2a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:13.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.616+0000 7ff28ec00700 1 --2- 172.21.15.89:0/2974418414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28815b5d0 0x7ff288071d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:13.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.616+0000 7ff28ec00700 1 -- 172.21.15.89:0/2974418414 >> 172.21.15.89:0/2974418414 conn(0x7ff2880cf190 msgr2=0x7ff28815cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:13.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.616+0000 7ff28ec00700 1 -- 172.21.15.89:0/2974418414 shutdown_connections 2024-04-02T09:17:13.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:13.616+0000 7ff28ec00700 1 -- 172.21.15.89:0/2974418414 wait complete. 2024-04-02T09:17:13.747 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. ... ERROR 2024-04-02T09:17:13.748 INFO:tasks.cephfs_test_runner: 2024-04-02T09:17:13.748 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-02T09:17:13.748 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-02T09:17:13.748 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. 2024-04-02T09:17:13.748 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-02T09:17:13.748 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-02T09:17:13.748 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_vshankar_ceph_d46289a69430c98a5ee030c7aba50d0c7af4ad4b/qa/tasks/cephfs/test_snap_schedules.py", line 678, in test_snap_schedule_subvol_and_group_arguments_01 2024-04-02T09:17:13.748 INFO:tasks.cephfs_test_runner: self._verify_snap_schedule(self.CREATE_VERSION, 'sv01', None) 2024-04-02T09:17:13.748 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_vshankar_ceph_d46289a69430c98a5ee030c7aba50d0c7af4ad4b/qa/tasks/cephfs/test_snap_schedules.py", line 660, in _verify_snap_schedule 2024-04-02T09:17:13.748 INFO:tasks.cephfs_test_runner: path = self._get_subvol_snapdir_path(version, subvol, group) 2024-04-02T09:17:13.748 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_vshankar_ceph_d46289a69430c98a5ee030c7aba50d0c7af4ad4b/qa/tasks/cephfs/test_snap_schedules.py", line 653, in _get_subvol_snapdir_path 2024-04-02T09:17:13.748 INFO:tasks.cephfs_test_runner: path = self.get_ceph_cmd_stdout("fs", *args).rstrip() 2024-04-02T09:17:13.749 INFO:tasks.cephfs_test_runner:AttributeError: 'TestSnapSchedulesSubvolAndGroupArguments' object has no attribute 'get_ceph_cmd_stdout' 2024-04-02T09:17:13.749 INFO:tasks.cephfs_test_runner: 2024-04-02T09:17:13.749 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-02T09:17:13.749 INFO:tasks.cephfs_test_runner:Ran 15 tests in 6042.811s 2024-04-02T09:17:13.749 INFO:tasks.cephfs_test_runner: 2024-04-02T09:17:13.749 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-04-02T09:17:13.749 INFO:tasks.cephfs_test_runner: 2024-04-02T09:17:13.749 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-02T09:17:13.749 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-02T09:17:13.749 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. 2024-04-02T09:17:13.749 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-02T09:17:13.749 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-02T09:17:13.749 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_vshankar_ceph_d46289a69430c98a5ee030c7aba50d0c7af4ad4b/qa/tasks/cephfs/test_snap_schedules.py", line 678, in test_snap_schedule_subvol_and_group_arguments_01 2024-04-02T09:17:13.749 INFO:tasks.cephfs_test_runner: self._verify_snap_schedule(self.CREATE_VERSION, 'sv01', None) 2024-04-02T09:17:13.750 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_vshankar_ceph_d46289a69430c98a5ee030c7aba50d0c7af4ad4b/qa/tasks/cephfs/test_snap_schedules.py", line 660, in _verify_snap_schedule 2024-04-02T09:17:13.750 INFO:tasks.cephfs_test_runner: path = self._get_subvol_snapdir_path(version, subvol, group) 2024-04-02T09:17:13.750 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_vshankar_ceph_d46289a69430c98a5ee030c7aba50d0c7af4ad4b/qa/tasks/cephfs/test_snap_schedules.py", line 653, in _get_subvol_snapdir_path 2024-04-02T09:17:13.750 INFO:tasks.cephfs_test_runner: path = self.get_ceph_cmd_stdout("fs", *args).rstrip() 2024-04-02T09:17:13.750 INFO:tasks.cephfs_test_runner:AttributeError: 'TestSnapSchedulesSubvolAndGroupArguments' object has no attribute 'get_ceph_cmd_stdout' 2024-04-02T09:17:13.750 INFO:tasks.cephfs_test_runner: 2024-04-02T09:17:13.750 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_d46289a69430c98a5ee030c7aba50d0c7af4ad4b/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-02T09:17:13.969 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b74153b402544243954c3e270c08762c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_d46289a69430c98a5ee030c7aba50d0c7af4ad4b/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-02T09:17:13.971 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-02T09:17:13.992 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-02T09:17:14.000 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-02T09:17:14.000 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:17:14.000 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:17:14.049 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:17:14.050 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-02T09:17:14.050 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:17:14.134 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345468k,nr_inodes=4086367,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275356k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-04-02T09:17:14.134 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:17:14.134 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:17:14.184 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:17:14.184 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-02T09:17:14.184 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T09:17:14.229 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345592k,nr_inodes=4086398,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275380k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-04-02T09:17:14.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:17:14.230 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T09:17:14.275 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-02T09:17:14.275 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-02T09:17:14.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:17:14.276 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-02T09:17:14.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:17:14.384 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-02T09:17:14.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:17:14.452 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2024-04-02T09:17:14.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:17:14.572 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-04-02T09:17:14.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:17:14.619 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2024-04-02T09:17:14.692 INFO:teuthology.orchestra.run.smithi089.stderr:Cannot find device "ceph-brx" 2024-04-02T09:17:14.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T09:17:14.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:17:14.695 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-04-02T09:17:14.745 INFO:teuthology.orchestra.run.smithi113.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-02T09:17:14.746 INFO:teuthology.orchestra.run.smithi113.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-02T09:17:14.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:17:14.747 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-02T09:17:14.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:17:14.818 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-02T09:17:14.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:17:14.887 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link delete ceph-brx 2024-04-02T09:17:15.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:17:15.003 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-04-02T09:17:15.009 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:17:15.013 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:17:15.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T09:17:15.049 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link delete ceph-brx 2024-04-02T09:17:15.123 INFO:teuthology.orchestra.run.smithi113.stderr:Cannot find device "ceph-brx" 2024-04-02T09:17:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T09:17:15.126 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-02T09:17:15.136 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-02T09:17:15.136 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:17:15.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.329+0000 7f5e08c00700 1 Processor -- start 2024-04-02T09:17:15.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.330+0000 7f5e08c00700 1 -- start start 2024-04-02T09:17:15.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.331+0000 7f5e08c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5e040b3af0 0x7f5e0415a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:15.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.331+0000 7f5e08c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5e040709c0 0x7f5e04070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:15.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.331+0000 7f5e08c00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e040b2180 con 0x7f5e04072580 2024-04-02T09:17:15.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.331+0000 7f5e08c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e0415b100 con 0x7f5e040b3af0 2024-04-02T09:17:15.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.331+0000 7f5e08c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0415b3e0 con 0x7f5e040709c0 2024-04-02T09:17:15.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.331+0000 7f5e01400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5e04072580 0x7f5e040b2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59580/0 (socket says 172.21.15.89:59580) 2024-04-02T09:17:15.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.331+0000 7f5e01400700 1 -- 172.21.15.89:0/3743121779 learned_addr learned my addr 172.21.15.89:0/3743121779 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:15.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.331+0000 7f5e02800700 1 --2- 172.21.15.89:0/3743121779 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5e040709c0 0x7f5e04070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:15.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.331+0000 7f5e01e00700 1 --2- 172.21.15.89:0/3743121779 >> v2:172.21.15.113:3300/0 conn(0x7f5e040b3af0 0x7f5e0415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:15.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.331+0000 7f5e00a00700 1 -- 172.21.15.89:0/3743121779 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307106262 0 0) 0x7f5e040b2180 con 0x7f5e04072580 2024-04-02T09:17:15.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.331+0000 7f5e00a00700 1 -- 172.21.15.89:0/3743121779 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de80035b0 con 0x7f5e04072580 2024-04-02T09:17:15.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.331+0000 7f5e02800700 1 -- 172.21.15.89:0/3743121779 >> v2:172.21.15.113:3300/0 conn(0x7f5e040b3af0 msgr2=0x7f5e0415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:15.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.332+0000 7f5e02800700 1 --2- 172.21.15.89:0/3743121779 >> v2:172.21.15.113:3300/0 conn(0x7f5e040b3af0 0x7f5e0415a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.332+0000 7f5e02800700 1 -- 172.21.15.89:0/3743121779 >> v1:172.21.15.89:6789/0 conn(0x7f5e04072580 legacy=0x7f5e040b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:15.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.332+0000 7f5e02800700 1 -- 172.21.15.89:0/3743121779 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e040b23f0 con 0x7f5e040709c0 2024-04-02T09:17:15.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.332+0000 7f5e01e00700 1 --2- 172.21.15.89:0/3743121779 >> v2:172.21.15.113:3300/0 conn(0x7f5e040b3af0 0x7f5e0415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:17:15.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.332+0000 7f5e02800700 1 --2- 172.21.15.89:0/3743121779 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5e040709c0 0x7f5e04070e50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc009d90 tx=0x7f5dfc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8627ec8935bf95e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:15.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.332+0000 7f5e00a00700 1 -- 172.21.15.89:0/3743121779 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc00bd20 con 0x7f5e040709c0 2024-04-02T09:17:15.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.332+0000 7f5e00a00700 1 -- 172.21.15.89:0/3743121779 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dfc00be80 con 0x7f5e040709c0 2024-04-02T09:17:15.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.332+0000 7f5e00a00700 1 -- 172.21.15.89:0/3743121779 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc0116a0 con 0x7f5e040709c0 2024-04-02T09:17:15.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.333+0000 7f5e08c00700 1 -- 172.21.15.89:0/3743121779 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5e040709c0 msgr2=0x7f5e04070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:15.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.333+0000 7f5e08c00700 1 --2- 172.21.15.89:0/3743121779 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5e040709c0 0x7f5e04070e50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc009d90 tx=0x7f5dfc00b330 comp rx=0 tx=0).stop 2024-04-02T09:17:15.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.333+0000 7f5e08c00700 1 -- 172.21.15.89:0/3743121779 shutdown_connections 2024-04-02T09:17:15.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.333+0000 7f5e08c00700 1 --2- 172.21.15.89:0/3743121779 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5e040709c0 0x7f5e04070e50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.333+0000 7f5e08c00700 1 --2- 172.21.15.89:0/3743121779 >> v2:172.21.15.113:3300/0 conn(0x7f5e040b3af0 0x7f5e0415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.333+0000 7f5e08c00700 1 -- 172.21.15.89:0/3743121779 >> 172.21.15.89:0/3743121779 conn(0x7f5e040cf190 msgr2=0x7f5e040cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:15.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.333+0000 7f5e08c00700 1 -- 172.21.15.89:0/3743121779 shutdown_connections 2024-04-02T09:17:15.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.333+0000 7f5e08c00700 1 -- 172.21.15.89:0/3743121779 wait complete. 2024-04-02T09:17:15.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.334+0000 7f5e08c00700 1 Processor -- start 2024-04-02T09:17:15.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.335+0000 7f5e08c00700 1 -- start start 2024-04-02T09:17:15.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.335+0000 7f5e08c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e040709c0 0x7f5e04155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:15.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.335+0000 7f5e08c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e04072580 0x7f5e04157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:15.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.335+0000 7f5e08c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5e040b3af0 0x7f5e04148f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:15.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.335+0000 7f5e08c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0415bce0 con 0x7f5e040709c0 2024-04-02T09:17:15.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.335+0000 7f5e08c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e040c8460 con 0x7f5e04072580 2024-04-02T09:17:15.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.335+0000 7f5e08c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e04151650 con 0x7f5e040b3af0 2024-04-02T09:17:15.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.336+0000 7f5e01e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e040709c0 0x7f5e04155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:15.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.336+0000 7f5e01e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e040709c0 0x7f5e04155880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56890/0 (socket says 172.21.15.89:56890) 2024-04-02T09:17:15.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.336+0000 7f5e01e00700 1 -- 172.21.15.89:0/1175325763 learned_addr learned my addr 172.21.15.89:0/1175325763 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:15.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.336+0000 7f5e02800700 1 --2- 172.21.15.89:0/1175325763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5e040b3af0 0x7f5e04148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:15.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.336+0000 7f5e01e00700 1 -- 172.21.15.89:0/1175325763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5e040b3af0 msgr2=0x7f5e04148f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:15.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.336+0000 7f5e01e00700 1 --2- 172.21.15.89:0/1175325763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5e040b3af0 0x7f5e04148f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.336+0000 7f5e01e00700 1 -- 172.21.15.89:0/1175325763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e04072580 msgr2=0x7f5e04157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:15.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.336+0000 7f5e01e00700 1 --2- 172.21.15.89:0/1175325763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e04072580 0x7f5e04157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.336+0000 7f5e01e00700 1 -- 172.21.15.89:0/1175325763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df40094a0 con 0x7f5e040709c0 2024-04-02T09:17:15.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.336+0000 7f5e02800700 1 --2- 172.21.15.89:0/1175325763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5e040b3af0 0x7f5e04148f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:17:15.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.336+0000 7f5e01e00700 1 --2- 172.21.15.89:0/1175325763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e040709c0 0x7f5e04155880 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f5df400ebe0 tx=0x7f5df4009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:15.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.337+0000 7f5dfa000700 1 -- 172.21.15.89:0/1175325763 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df400c910 con 0x7f5e040709c0 2024-04-02T09:17:15.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.337+0000 7f5dfa000700 1 -- 172.21.15.89:0/1175325763 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df400ca70 con 0x7f5e040709c0 2024-04-02T09:17:15.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.337+0000 7f5e08c00700 1 -- 172.21.15.89:0/1175325763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dfc009a70 con 0x7f5e040709c0 2024-04-02T09:17:15.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.337+0000 7f5dfa000700 1 -- 172.21.15.89:0/1175325763 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df400cde0 con 0x7f5e040709c0 2024-04-02T09:17:15.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.337+0000 7f5e08c00700 1 -- 172.21.15.89:0/1175325763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e04151370 con 0x7f5e040709c0 2024-04-02T09:17:15.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.338+0000 7f5e08c00700 1 -- 172.21.15.89:0/1175325763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e04067720 con 0x7f5e040709c0 2024-04-02T09:17:15.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.339+0000 7f5dfa000700 1 -- 172.21.15.89:0/1175325763 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5df4013470 con 0x7f5e040709c0 2024-04-02T09:17:15.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.341+0000 7f5dfa000700 1 --2- 172.21.15.89:0/1175325763 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5de406adb0 0x7f5de406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:15.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.341+0000 7f5dfa000700 1 -- 172.21.15.89:0/1175325763 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f5df40899a0 con 0x7f5e040709c0 2024-04-02T09:17:15.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.341+0000 7f5e01400700 1 --2- 172.21.15.89:0/1175325763 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5de406adb0 0x7f5de406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:15.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.341+0000 7f5e01400700 1 --2- 172.21.15.89:0/1175325763 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5de406adb0 0x7f5de406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5e041540e0 tx=0x7f5df00079c0 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:15.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.345+0000 7f5dfa000700 1 -- 172.21.15.89:0/1175325763 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5df40547b0 con 0x7f5e040709c0 2024-04-02T09:17:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.544+0000 7f5e08c00700 1 -- 172.21.15.89:0/1175325763 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5e0406dea0 con 0x7f5de406adb0 2024-04-02T09:17:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.547+0000 7f5dfa000700 1 -- 172.21.15.89:0/1175325763 <== mgr.13575 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50437 (secure 0 0 0) 0x7f5e0406dea0 con 0x7f5de406adb0 2024-04-02T09:17:15.547 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:17:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.550+0000 7f5e08c00700 1 -- 172.21.15.89:0/1175325763 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5de406adb0 msgr2=0x7f5de406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.550+0000 7f5e08c00700 1 --2- 172.21.15.89:0/1175325763 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5de406adb0 0x7f5de406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5e041540e0 tx=0x7f5df00079c0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.550+0000 7f5e08c00700 1 -- 172.21.15.89:0/1175325763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e040709c0 msgr2=0x7f5e04155880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.550+0000 7f5e08c00700 1 --2- 172.21.15.89:0/1175325763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e040709c0 0x7f5e04155880 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f5df400ebe0 tx=0x7f5df4009ec0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.550+0000 7f5e08c00700 1 -- 172.21.15.89:0/1175325763 shutdown_connections 2024-04-02T09:17:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.550+0000 7f5e08c00700 1 --2- 172.21.15.89:0/1175325763 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5e040b3af0 0x7f5e04148f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.550+0000 7f5e08c00700 1 --2- 172.21.15.89:0/1175325763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e040709c0 0x7f5e04155880 unknown :-1 s=CLOSED pgs=3096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.550+0000 7f5e08c00700 1 --2- 172.21.15.89:0/1175325763 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5de406adb0 0x7f5de406d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.550+0000 7f5e08c00700 1 --2- 172.21.15.89:0/1175325763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e04072580 0x7f5e04157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.550+0000 7f5e08c00700 1 -- 172.21.15.89:0/1175325763 >> 172.21.15.89:0/1175325763 conn(0x7f5e040cf190 msgr2=0x7f5e0406ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:15.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.550+0000 7f5e08c00700 1 -- 172.21.15.89:0/1175325763 shutdown_connections 2024-04-02T09:17:15.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.550+0000 7f5e08c00700 1 -- 172.21.15.89:0/1175325763 wait complete. 2024-04-02T09:17:15.551 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:17:15.567 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-02T09:17:13.568501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":447176,"kb_used_data":29248,"kb_used_omap":12,"kb_used_meta":417779,"kb_avail":754527544,"statfs":{"total":773094113280,"available":772636205056,"internally_reserved":0,"allocated":29949952,"data_stored":15989882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":427806264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-766068,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-200,"num_read_kb":-56,"num_write":-289,"num_write_kb":-2297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.021711"},"pg_stats":[{"pgid":"1.0","version":"282'425","reported_seq":1539,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-02T09:17:10.886632+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:17:10.886632+0000","last_peered":"2024-04-02T09:17:10.886632+0000","last_clean":"2024-04-02T09:17:10.886632+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:17:10.886632+0000","last_undegraded":"2024-04-02T09:17:10.886632+0000","last_fullsized":"2024-04-02T09:17:10.886632+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60876,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310964,"statfs":{"total":96636764160,"available":96574427136,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.70499999999999996,"15min":0.70999999999999996},"min":{"1min":0.437,"5min":0.309,"15min":0.309},"max":{"1min":0.96699999999999997,"5min":8.016,"15min":8.016},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.71399999999999997},"min":{"1min":0.57299999999999995,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.081,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.59999999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.74299999999999999,"15min":0.73599999999999999},"min":{"1min":0.46800000000000003,"5min":0.42199999999999999,"15min":0.38200000000000001},"max":{"1min":1.0069999999999999,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67500000000000004,"15min":0.70099999999999996},"min":{"1min":0.52300000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82899999999999996,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.58099999999999996}]},{"osd":2,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.748,"15min":0.75900000000000001},"min":{"1min":0.48299999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91600000000000004,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79500000000000004,"15min":0.80200000000000005},"min":{"1min":0.60199999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.1140000000000001,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.68300000000000005}]},{"osd":3,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.747,"15min":0.74399999999999999},"min":{"1min":0.56799999999999995,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.94899999999999995,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.77200000000000002,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.91700000000000004,"5min":5.093,"15min":5.093},"last":0.73799999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.65900000000000003,"15min":0.63100000000000001},"min":{"1min":0.45800000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.99099999999999999,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.79300000000000004,"15min":0.75},"min":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.03,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.77600000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.72799999999999998,"15min":0.69799999999999995},"min":{"1min":0.45900000000000002,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":0.94699999999999995,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.754,"15min":0.73699999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.93300000000000005,"5min":3.492,"15min":3.492},"last":0.76000000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.70899999999999996,"15min":0.69999999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97499999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.76900000000000002,"15min":0.72499999999999998},"min":{"1min":0.42299999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.96099999999999997,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.82499999999999996}]}]},{"osd":6,"up_from":13,"seq":55834576034,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64972,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306868,"statfs":{"total":96636764160,"available":96570232832,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.65900000000000003,"15min":0.70799999999999996},"min":{"1min":0.52000000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.94199999999999995,"5min":0.94899999999999995,"15min":1.1879999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.68700000000000006,"15min":0.72199999999999998},"min":{"1min":0.52100000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83999999999999997,"5min":0.98399999999999999,"15min":1.0600000000000001},"last":0.72099999999999997}]},{"osd":1,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.59499999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.95099999999999996,"5min":1.0149999999999999,"15min":1.212},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.755,"5min":0.69799999999999995,"15min":0.73199999999999998},"min":{"1min":0.48499999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.1759999999999999},"last":1.0700000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68500000000000005,"15min":0.72999999999999998},"min":{"1min":0.47399999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.98099999999999998,"5min":1.046,"15min":1.2789999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.68100000000000005,"15min":0.71899999999999997},"min":{"1min":0.57199999999999995,"5min":0.44,"15min":0.439},"max":{"1min":0.89300000000000002,"5min":0.99299999999999999,"15min":1.226},"last":0.747}]},{"osd":3,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.69399999999999995,"15min":0.73399999999999999},"min":{"1min":0.501,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.98099999999999998,"5min":1.026,"15min":1.2549999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.73599999999999999,"15min":0.76300000000000001},"min":{"1min":0.60499999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.95799999999999996,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.65800000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64900000000000002,"15min":0.625},"min":{"1min":0.439,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0329999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68400000000000005,"15min":0.66800000000000004},"min":{"1min":0.46800000000000003,"5min":0.373,"15min":0.373},"max":{"1min":0.96599999999999997,"5min":1.1439999999999999,"15min":1.1879999999999999},"last":0.96599999999999997}]},{"osd":5,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66000000000000003,"15min":0.65500000000000003},"min":{"1min":0.48399999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.92300000000000004,"5min":0.98699999999999999,"15min":1.2669999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.53800000000000003,"5min":0.436,"15min":0.436},"max":{"1min":1.095,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":1.095}]},{"osd":7,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.58599999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.999,"5min":1.131,"15min":1.2370000000000001},"last":0.871},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73199999999999998,"15min":0.73499999999999999},"min":{"1min":0.54200000000000004,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.01,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":1.01}]}]},{"osd":1,"up_from":13,"seq":55834576035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57356,"kb_used_data":4092,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314484,"statfs":{"total":96636764160,"available":96578031616,"internally_reserved":0,"allocated":4190208,"data_stored":2441766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.437,"15min":0.44600000000000001},"min":{"1min":0.33900000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.73699999999999999,"5min":1.76,"15min":1.76},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.48299999999999998,"15min":0.503},"min":{"1min":0.33700000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.39700000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.48399999999999999,"15min":0.497},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.70599999999999996,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.45100000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.84199999999999997,"5min":1.859,"15min":1.859},"last":0.54700000000000004}]},{"osd":3,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.55300000000000005,"15min":0.56799999999999995},"min":{"1min":0.46899999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82299999999999995,"5min":1.8169999999999999,"15min":1.8169999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.52900000000000003,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.73099999999999998,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.45200000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.755,"15min":0.77600000000000002},"min":{"1min":0.65300000000000002,"5min":0.44500000000000001,"15min":0.433},"max":{"1min":1.0720000000000001,"5min":1.948,"15min":1.948},"last":0.622},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.72899999999999998,"15min":0.76200000000000001},"min":{"1min":0.53600000000000003,"5min":0.48999999999999999,"15min":0.41799999999999998},"max":{"1min":1.018,"5min":1.1060000000000001,"15min":1.25},"last":0.73799999999999999}]},{"osd":5,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76700000000000002,"15min":0.77900000000000003},"min":{"1min":0.59699999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.90700000000000003,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.79900000000000004,"15min":0.80500000000000005},"min":{"1min":0.64400000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.0069999999999999,"5min":4.5229999999999997,"15min":4.5229999999999997},"last":0.63}]},{"osd":6,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.79200000000000004,"15min":0.80400000000000005},"min":{"1min":0.58199999999999996,"5min":0.501,"15min":0.48099999999999998},"max":{"1min":0.997,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.77400000000000002,"15min":0.80000000000000004},"min":{"1min":0.59999999999999998,"5min":0.52900000000000003,"15min":0.49099999999999999},"max":{"1min":0.97899999999999998,"5min":1.2290000000000001,"15min":1.2529999999999999},"last":0.68799999999999994}]},{"osd":7,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.78600000000000003,"15min":0.81000000000000005},"min":{"1min":0.67400000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0589999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.80300000000000005,"15min":0.82399999999999995},"min":{"1min":0.64800000000000002,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0920000000000001,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.70199999999999996}]}]},{"osd":0,"up_from":13,"seq":55834576035,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52672,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319168,"statfs":{"total":96636764160,"available":96582828032,"internally_reserved":0,"allocated":3592192,"data_stored":1850379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.42799999999999999,"15min":0.44500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.628,"5min":0.77100000000000002,"15min":0.78100000000000003},"last":0.317},{"interface":"front","average":{"1min":0.5,"5min":0.46600000000000003,"15min":0.49099999999999999},"min":{"1min":0.29299999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65200000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.44800000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.50700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70099999999999996,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.50800000000000001,"15min":0.53800000000000003},"min":{"1min":0.41299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66000000000000003,"5min":0.72699999999999998,"15min":0.999},"last":0.54600000000000004}]},{"osd":3,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.54900000000000004},"min":{"1min":0.376,"5min":0.33400000000000002,"15min":0.312},"max":{"1min":0.67400000000000004,"5min":0.77800000000000002,"15min":0.85699999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58499999999999996,"15min":0.623},"min":{"1min":0.46100000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.73599999999999999,"5min":0.82799999999999996,"15min":0.92000000000000004},"last":0.754}]},{"osd":4,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.77200000000000002,"15min":0.81399999999999995},"min":{"1min":0.61199999999999999,"5min":0.55000000000000004,"15min":0.45600000000000002},"max":{"1min":1.129,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.628,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0369999999999999,"5min":1.169,"15min":1.169},"last":0.85299999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.77000000000000002,"15min":0.80400000000000005},"min":{"1min":0.56599999999999995,"5min":0.54000000000000004,"15min":0.497},"max":{"1min":1.0509999999999999,"5min":1.1419999999999999,"15min":1.232},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77000000000000002,"15min":0.81399999999999995},"min":{"1min":0.53200000000000003,"5min":0.497,"15min":0.497},"max":{"1min":1.0669999999999999,"5min":1.21,"15min":1.21},"last":0.79800000000000004}]},{"osd":6,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74399999999999999,"15min":0.78300000000000003},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.02,"5min":1.1830000000000001,"15min":1.196},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.76600000000000001,"15min":0.79600000000000004},"min":{"1min":0.55000000000000004,"5min":0.53700000000000003,"15min":0.52900000000000003},"max":{"1min":1.073,"5min":1.246,"15min":1.246},"last":0.93200000000000005}]},{"osd":7,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.81899999999999995,"15min":0.84499999999999997},"min":{"1min":0.64900000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.879},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.80200000000000005,"15min":0.83299999999999996},"min":{"1min":0.60799999999999998,"5min":0.499,"15min":0.499},"max":{"1min":2.3889999999999998,"5min":2.3889999999999998,"15min":2.3889999999999998},"last":0.94499999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576034,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.435,"15min":0.438},"min":{"1min":0.29699999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56699999999999995,"5min":1.681,"15min":1.681},"last":0.435},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":0.82899999999999996,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":0.35899999999999999}]},{"osd":1,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.5},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65800000000000003,"5min":1.72,"15min":1.72},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53100000000000003,"15min":0.54800000000000004},"min":{"1min":0.38200000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92900000000000005,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.51900000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53000000000000003,"15min":0.54200000000000004},"min":{"1min":0.38300000000000001,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.871,"5min":0.871,"15min":1.2749999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58699999999999997,"15min":0.61799999999999999},"min":{"1min":0.46600000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82499999999999996,"5min":1.754,"15min":1.754},"last":0.5}]},{"osd":4,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.75600000000000001,"15min":0.76800000000000002},"min":{"1min":0.53300000000000003,"5min":0.495,"15min":0.495},"max":{"1min":1.0940000000000001,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.48999999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.02,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.97999999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.75600000000000001,"15min":0.77400000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.177,"5min":1.177,"15min":1.99},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.748,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.081,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":1.1379999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76800000000000002,"15min":0.79700000000000004},"min":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1399999999999999,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.198},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.76900000000000002,"15min":0.79200000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.014,"5min":1.103,"15min":1.3819999999999999},"last":1.0720000000000001}]},{"osd":7,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.78500000000000003,"15min":0.81499999999999995},"min":{"1min":0.55300000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.196,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.81200000000000006},"min":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.497},"max":{"1min":1.1200000000000001,"5min":1.603,"15min":1.603},"last":1.014}]}]},{"osd":3,"up_from":13,"seq":55834576034,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56780,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315060,"statfs":{"total":96636764160,"available":96578621440,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41599999999999998,"15min":0.437},"min":{"1min":0.32500000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68000000000000005,"5min":0.73599999999999999,"15min":0.76000000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.436,"15min":0.47399999999999998},"min":{"1min":0.34100000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57199999999999995,"5min":0.68300000000000005,"15min":0.876},"last":0.38600000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.47799999999999998,"15min":0.49399999999999999},"min":{"1min":0.35199999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.77600000000000002,"5min":0.81699999999999995,"15min":0.90300000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52400000000000002,"15min":0.55900000000000005},"min":{"1min":0.41499999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.754,"5min":0.754,"15min":0.98499999999999999},"last":0.48599999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55200000000000005,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.93100000000000005},"last":0.44},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55200000000000005,"15min":0.59699999999999998},"min":{"1min":0.38,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.877,"5min":0.877,"15min":1.0840000000000001},"last":0.46100000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70599999999999996,"15min":0.73999999999999999},"min":{"1min":0.498,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86199999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.73099999999999998,"15min":0.75800000000000001},"min":{"1min":0.57299999999999995,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.91700000000000004,"5min":1.0760000000000001,"15min":1.1619999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72999999999999998,"15min":0.76500000000000001},"min":{"1min":0.54400000000000004,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.874,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.73299999999999998,"15min":0.76800000000000002},"min":{"1min":0.55000000000000004,"5min":0.51100000000000001,"15min":0.505},"max":{"1min":0.97199999999999998,"5min":1.0780000000000001,"15min":1.266},"last":0.56100000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.748,"15min":0.77400000000000002},"min":{"1min":0.57199999999999995,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.056,"5min":1.0900000000000001,"15min":1.284},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77200000000000002,"15min":0.79900000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.495},"max":{"1min":0.93799999999999994,"5min":1.093,"15min":1.2},"last":0.627}]},{"osd":7,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.745,"15min":0.79100000000000004},"min":{"1min":0.52700000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0840000000000001,"5min":1.125,"15min":1.3260000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.78000000000000003,"15min":0.81000000000000005},"min":{"1min":0.65500000000000003,"5min":0.495,"15min":0.495},"max":{"1min":1.0620000000000001,"5min":1.087,"15min":1.248},"last":0.69999999999999996}]}]},{"osd":4,"up_from":13,"seq":55834576034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48584,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323256,"statfs":{"total":96636764160,"available":96587014144,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63300000000000001,"15min":0.67200000000000004},"min":{"1min":0.49099999999999999,"5min":0.38100000000000001,"15min":0.29299999999999998},"max":{"1min":0.86799999999999999,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.63400000000000001,"15min":0.66900000000000004},"min":{"1min":0.505,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.92800000000000005,"5min":0.97099999999999997,"15min":1.075},"last":0.61899999999999999}]},{"osd":1,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.70699999999999996,"15min":0.74099999999999999},"min":{"1min":0.53200000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.996,"5min":1.0329999999999999,"15min":1.069},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.71999999999999997,"15min":0.751},"min":{"1min":0.48099999999999998,"5min":0.46100000000000002,"15min":0.40799999999999997},"max":{"1min":0.89500000000000002,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.75600000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65600000000000003,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.95399999999999996,"5min":1.077,"15min":1.077},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67600000000000005,"15min":0.71199999999999997},"min":{"1min":0.50900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95999999999999996,"5min":1.032,"15min":1.0740000000000001},"last":0.48399999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.71599999999999997,"15min":0.749},"min":{"1min":0.53000000000000003,"5min":0.48799999999999999,"15min":0.47399999999999998},"max":{"1min":0.94599999999999995,"5min":0.98099999999999998,"15min":1.018},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.69899999999999995,"15min":0.73999999999999999},"min":{"1min":0.59699999999999998,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":0.93799999999999994,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.72499999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.47899999999999998,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.88200000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.629,"15min":0.625},"min":{"1min":0.46000000000000002,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.82699999999999996,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.77400000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.50800000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81599999999999995,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.68799999999999994,"15min":0.69399999999999995},"min":{"1min":0.49299999999999999,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":0.74399999999999999,"5min":1.0149999999999999,"15min":1.0269999999999999},"last":0.85499999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.70299999999999996,"15min":0.70599999999999996},"min":{"1min":0.48399999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.90200000000000002,"5min":1.002,"15min":1.002},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76900000000000002,"15min":0.77700000000000002},"min":{"1min":0.56399999999999995,"5min":0.499,"15min":0.499},"max":{"1min":0.88900000000000001,"5min":1.0960000000000001,"15min":1.0980000000000001},"last":0.96199999999999997}]}]},{"osd":5,"up_from":13,"seq":55834576037,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53252,"kb_used_data":4088,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318588,"statfs":{"total":96636764160,"available":96582234112,"internally_reserved":0,"allocated":4186112,"data_stored":2440747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.66800000000000004,"15min":0.68999999999999995},"min":{"1min":0.54300000000000004,"5min":0.42199999999999999,"15min":0.39600000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":1.004},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.52800000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.90800000000000003,"5min":1.085,"15min":1.085},"last":0.65400000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.66400000000000003,"15min":0.70399999999999996},"min":{"1min":0.55700000000000005,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89000000000000001,"5min":0.94499999999999995,"15min":1.0189999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.69899999999999995,"15min":0.71299999999999997},"min":{"1min":0.60399999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.89400000000000002,"5min":0.96799999999999997,"15min":1.052},"last":0.72199999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.69499999999999995,"15min":0.72799999999999998},"min":{"1min":0.57899999999999996,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92400000000000004,"5min":0.97299999999999998,"15min":1.014},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.69299999999999995,"15min":0.72499999999999998},"min":{"1min":0.622,"5min":0.436,"15min":0.436},"max":{"1min":0.96199999999999997,"5min":1.018,"15min":1.0680000000000001},"last":0.49399999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.71399999999999997,"15min":0.76000000000000001},"min":{"1min":0.59399999999999997,"5min":0.496,"15min":0.496},"max":{"1min":0.92700000000000005,"5min":0.93600000000000005,"15min":1.083},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.76900000000000002,"15min":0.78100000000000003},"min":{"1min":0.71799999999999997,"5min":0.50600000000000001,"15min":0.47399999999999998},"max":{"1min":0.92600000000000005,"5min":1.0109999999999999,"15min":1.1020000000000001},"last":0.63300000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65100000000000002,"15min":0.63100000000000001},"min":{"1min":0.45000000000000001,"5min":0.40699999999999997,"15min":0.38300000000000001},"max":{"1min":0.91300000000000003,"5min":0.98399999999999999,"15min":1.054},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.68700000000000006,"15min":0.66900000000000004},"min":{"1min":0.46300000000000002,"5min":0.45900000000000002,"15min":0.40400000000000003},"max":{"1min":0.93999999999999995,"5min":1.145,"15min":1.145},"last":0.748}]},{"osd":6,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.67700000000000005,"15min":0.67400000000000004},"min":{"1min":0.497,"5min":0.434,"15min":0.41099999999999998},"max":{"1min":0.79000000000000004,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71599999999999997,"15min":0.70999999999999996},"min":{"1min":0.50600000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.96399999999999997,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.62}]},{"osd":7,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.73899999999999999,"15min":0.73199999999999998},"min":{"1min":0.51900000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82699999999999996,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.75,"5min":0.77000000000000002,"15min":0.76300000000000001},"min":{"1min":0.52700000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95999999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.75900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:17:15.570 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:17:15.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.763+0000 7f1acd200700 1 Processor -- start 2024-04-02T09:17:15.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.763+0000 7f1acd200700 1 -- start start 2024-04-02T09:17:15.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.763+0000 7f1acd200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ac815d640 0x7f1ac80d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:15.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.763+0000 7f1acd200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1ac81619d0 0x7f1ac80b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:15.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.763+0000 7f1acd200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac80d30d0 con 0x7f1ac80ceec0 2024-04-02T09:17:15.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.763+0000 7f1acd200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac80b6840 con 0x7f1ac81619d0 2024-04-02T09:17:15.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.763+0000 7f1acd200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac80d2880 con 0x7f1ac815d640 2024-04-02T09:17:15.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.764+0000 7f1ac5e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ac815d640 0x7f1ac80d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:15.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.764+0000 7f1ac6800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1ac81619d0 0x7f1ac80b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:15.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.764+0000 7f1ac5400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f1ac80ceec0 0x7f1ac81612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:38352/0 (socket says 172.21.15.89:38352) 2024-04-02T09:17:15.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.764+0000 7f1ac5e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ac815d640 0x7f1ac80d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:54326/0 (socket says 172.21.15.89:54326) 2024-04-02T09:17:15.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.764+0000 7f1ac5400700 1 -- 172.21.15.89:0/4063876056 learned_addr learned my addr 172.21.15.89:0/4063876056 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:15.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.764+0000 7f1ac6800700 1 -- 172.21.15.89:0/4063876056 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ac815d640 msgr2=0x7f1ac80d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:15.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.764+0000 7f1ac6800700 1 --2- 172.21.15.89:0/4063876056 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ac815d640 0x7f1ac80d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.764+0000 7f1ac6800700 1 -- 172.21.15.89:0/4063876056 >> v1:172.21.15.113:6789/0 conn(0x7f1ac80ceec0 legacy=0x7f1ac81612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:15.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.764+0000 7f1ac6800700 1 -- 172.21.15.89:0/4063876056 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac80d2b30 con 0x7f1ac81619d0 2024-04-02T09:17:15.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.764+0000 7f1ac6800700 1 --2- 172.21.15.89:0/4063876056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ac81619d0 0x7f1ac80b6cd0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009d90 tx=0x7f1ac000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9954839d91ca0cd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:15.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.765+0000 7f1ac4a00700 1 -- 172.21.15.89:0/4063876056 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac000e070 con 0x7f1ac81619d0 2024-04-02T09:17:15.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.765+0000 7f1ac4a00700 1 -- 172.21.15.89:0/4063876056 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac000bb20 con 0x7f1ac81619d0 2024-04-02T09:17:15.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.765+0000 7f1ac4a00700 1 -- 172.21.15.89:0/4063876056 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0012430 con 0x7f1ac81619d0 2024-04-02T09:17:15.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.765+0000 7f1acd200700 1 -- 172.21.15.89:0/4063876056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ac81619d0 msgr2=0x7f1ac80b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:15.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.765+0000 7f1acd200700 1 --2- 172.21.15.89:0/4063876056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ac81619d0 0x7f1ac80b6cd0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009d90 tx=0x7f1ac000b330 comp rx=0 tx=0).stop 2024-04-02T09:17:15.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.765+0000 7f1acd200700 1 -- 172.21.15.89:0/4063876056 shutdown_connections 2024-04-02T09:17:15.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.765+0000 7f1acd200700 1 --2- 172.21.15.89:0/4063876056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ac81619d0 0x7f1ac80b6cd0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.765+0000 7f1acd200700 1 --2- 172.21.15.89:0/4063876056 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ac815d640 0x7f1ac80d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.765+0000 7f1acd200700 1 -- 172.21.15.89:0/4063876056 >> 172.21.15.89:0/4063876056 conn(0x7f1ac808aff0 msgr2=0x7f1ac80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:15.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.765+0000 7f1acd200700 1 -- 172.21.15.89:0/4063876056 shutdown_connections 2024-04-02T09:17:15.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.765+0000 7f1acd200700 1 -- 172.21.15.89:0/4063876056 wait complete. 2024-04-02T09:17:15.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.767+0000 7f1acd200700 1 Processor -- start 2024-04-02T09:17:15.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.767+0000 7f1acd200700 1 -- start start 2024-04-02T09:17:15.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1acd200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ac80ceec0 0x7f1ac8159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:15.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1acd200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ac815d640 0x7f1ac8155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:15.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1acd200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ac81619d0 0x7f1ac8158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:15.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1acd200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac81f3050 con 0x7f1ac815d640 2024-04-02T09:17:15.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1acd200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac8155fe0 con 0x7f1ac80ceec0 2024-04-02T09:17:15.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1acd200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac8159e20 con 0x7f1ac81619d0 2024-04-02T09:17:15.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1ac5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ac815d640 0x7f1ac8155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:15.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1ac5e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ac80ceec0 0x7f1ac8159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:15.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1ac5400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ac815d640 0x7f1ac8155aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56896/0 (socket says 172.21.15.89:56896) 2024-04-02T09:17:15.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1ac5400700 1 -- 172.21.15.89:0/3456843768 learned_addr learned my addr 172.21.15.89:0/3456843768 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:15.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1ac6800700 1 --2- 172.21.15.89:0/3456843768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ac81619d0 0x7f1ac8158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:15.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1ac5e00700 1 -- 172.21.15.89:0/3456843768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ac81619d0 msgr2=0x7f1ac8158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:15.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1ac5e00700 1 --2- 172.21.15.89:0/3456843768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ac81619d0 0x7f1ac8158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1ac5e00700 1 -- 172.21.15.89:0/3456843768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ac815d640 msgr2=0x7f1ac8155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:15.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1ac5e00700 1 --2- 172.21.15.89:0/3456843768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ac815d640 0x7f1ac8155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1ac5e00700 1 -- 172.21.15.89:0/3456843768 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac0009a70 con 0x7f1ac80ceec0 2024-04-02T09:17:15.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1ac5e00700 1 --2- 172.21.15.89:0/3456843768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ac80ceec0 0x7f1ac8159840 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f1ab800e870 tx=0x7f1ab800eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:15.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1abe000700 1 -- 172.21.15.89:0/3456843768 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8013680 con 0x7f1ac80ceec0 2024-04-02T09:17:15.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1abe000700 1 -- 172.21.15.89:0/3456843768 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab80137e0 con 0x7f1ac80ceec0 2024-04-02T09:17:15.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1acd200700 1 -- 172.21.15.89:0/3456843768 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac815a100 con 0x7f1ac80ceec0 2024-04-02T09:17:15.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.769+0000 7f1abe000700 1 -- 172.21.15.89:0/3456843768 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8013b90 con 0x7f1ac80ceec0 2024-04-02T09:17:15.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.770+0000 7f1acd200700 1 -- 172.21.15.89:0/3456843768 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac815a6c0 con 0x7f1ac80ceec0 2024-04-02T09:17:15.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.771+0000 7f1acd200700 1 -- 172.21.15.89:0/3456843768 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac81f32c0 con 0x7f1ac80ceec0 2024-04-02T09:17:15.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.771+0000 7f1abe000700 1 -- 172.21.15.89:0/3456843768 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1ab8013cf0 con 0x7f1ac80ceec0 2024-04-02T09:17:15.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.772+0000 7f1abe000700 1 --2- 172.21.15.89:0/3456843768 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1aa806adb0 0x7f1aa806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:15.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.772+0000 7f1abe000700 1 -- 172.21.15.89:0/3456843768 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f1ab80893f0 con 0x7f1ac80ceec0 2024-04-02T09:17:15.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.773+0000 7f1ac5400700 1 --2- 172.21.15.89:0/3456843768 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1aa806adb0 0x7f1aa806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:15.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.773+0000 7f1ac5400700 1 --2- 172.21.15.89:0/3456843768 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1aa806adb0 0x7f1aa806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80ceba0 tx=0x7f1ab4009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:15.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.778+0000 7f1abe000700 1 -- 172.21.15.89:0/3456843768 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ab8054310 con 0x7f1ac80ceec0 2024-04-02T09:17:15.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.973+0000 7f1acd200700 1 -- 172.21.15.89:0/3456843768 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1ac80b64d0 con 0x7f1aa806adb0 2024-04-02T09:17:15.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.976+0000 7f1abe000700 1 -- 172.21.15.89:0/3456843768 <== mgr.13575 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50438 (secure 0 0 0) 0x7f1ab40015a0 con 0x7f1aa806adb0 2024-04-02T09:17:15.976 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:17:15.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.979+0000 7f1acd200700 1 -- 172.21.15.89:0/3456843768 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1aa806adb0 msgr2=0x7f1aa806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:15.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.979+0000 7f1acd200700 1 --2- 172.21.15.89:0/3456843768 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1aa806adb0 0x7f1aa806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80ceba0 tx=0x7f1ab4009040 comp rx=0 tx=0).stop 2024-04-02T09:17:15.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.979+0000 7f1acd200700 1 -- 172.21.15.89:0/3456843768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ac80ceec0 msgr2=0x7f1ac8159840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:15.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.979+0000 7f1acd200700 1 --2- 172.21.15.89:0/3456843768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ac80ceec0 0x7f1ac8159840 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f1ab800e870 tx=0x7f1ab800eb80 comp rx=0 tx=0).stop 2024-04-02T09:17:15.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.979+0000 7f1acd200700 1 -- 172.21.15.89:0/3456843768 shutdown_connections 2024-04-02T09:17:15.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.979+0000 7f1acd200700 1 --2- 172.21.15.89:0/3456843768 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f1aa806adb0 0x7f1aa806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.979+0000 7f1acd200700 1 --2- 172.21.15.89:0/3456843768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ac80ceec0 0x7f1ac8159840 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.979+0000 7f1acd200700 1 --2- 172.21.15.89:0/3456843768 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1ac81619d0 0x7f1ac8158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.979+0000 7f1acd200700 1 --2- 172.21.15.89:0/3456843768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ac815d640 0x7f1ac8155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:15.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.979+0000 7f1acd200700 1 -- 172.21.15.89:0/3456843768 >> 172.21.15.89:0/3456843768 conn(0x7f1ac808aff0 msgr2=0x7f1ac806b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:15.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.979+0000 7f1acd200700 1 -- 172.21.15.89:0/3456843768 shutdown_connections 2024-04-02T09:17:15.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:15.979+0000 7f1acd200700 1 -- 172.21.15.89:0/3456843768 wait complete. 2024-04-02T09:17:15.979 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:17:15.995 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-02T09:17:15.568937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":447176,"kb_used_data":29248,"kb_used_omap":12,"kb_used_meta":417779,"kb_avail":754527544,"statfs":{"total":773094113280,"available":772636205056,"internally_reserved":0,"allocated":29949952,"data_stored":15989882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":427806264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-766068,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-200,"num_read_kb":-56,"num_write":-289,"num_write_kb":-2297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.975074"},"pg_stats":[{"pgid":"1.0","version":"282'425","reported_seq":1539,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-02T09:17:10.886632+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:17:10.886632+0000","last_peered":"2024-04-02T09:17:10.886632+0000","last_clean":"2024-04-02T09:17:10.886632+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:17:10.886632+0000","last_undegraded":"2024-04-02T09:17:10.886632+0000","last_fullsized":"2024-04-02T09:17:10.886632+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60876,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310964,"statfs":{"total":96636764160,"available":96574427136,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.70499999999999996,"15min":0.70999999999999996},"min":{"1min":0.437,"5min":0.309,"15min":0.309},"max":{"1min":0.96699999999999997,"5min":8.016,"15min":8.016},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.71399999999999997},"min":{"1min":0.57299999999999995,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.081,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.59999999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.74299999999999999,"15min":0.73599999999999999},"min":{"1min":0.46800000000000003,"5min":0.42199999999999999,"15min":0.38200000000000001},"max":{"1min":1.0069999999999999,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67500000000000004,"15min":0.70099999999999996},"min":{"1min":0.52300000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82899999999999996,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.58099999999999996}]},{"osd":2,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.748,"15min":0.75900000000000001},"min":{"1min":0.48299999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91600000000000004,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79500000000000004,"15min":0.80200000000000005},"min":{"1min":0.60199999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.1140000000000001,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.68300000000000005}]},{"osd":3,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.747,"15min":0.74399999999999999},"min":{"1min":0.56799999999999995,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.94899999999999995,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.77200000000000002,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.91700000000000004,"5min":5.093,"15min":5.093},"last":0.73799999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.65900000000000003,"15min":0.63100000000000001},"min":{"1min":0.45800000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.99099999999999999,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.79300000000000004,"15min":0.75},"min":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.03,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.77600000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.72799999999999998,"15min":0.69799999999999995},"min":{"1min":0.45900000000000002,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":0.94699999999999995,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.754,"15min":0.73699999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.93300000000000005,"5min":3.492,"15min":3.492},"last":0.76000000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.70899999999999996,"15min":0.69999999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97499999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.76900000000000002,"15min":0.72499999999999998},"min":{"1min":0.42299999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.96099999999999997,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.82499999999999996}]}]},{"osd":6,"up_from":13,"seq":55834576034,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64972,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306868,"statfs":{"total":96636764160,"available":96570232832,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.65900000000000003,"15min":0.70799999999999996},"min":{"1min":0.52000000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.94199999999999995,"5min":0.94899999999999995,"15min":1.1879999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.68700000000000006,"15min":0.72199999999999998},"min":{"1min":0.52100000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83999999999999997,"5min":0.98399999999999999,"15min":1.0600000000000001},"last":0.72099999999999997}]},{"osd":1,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.59499999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.95099999999999996,"5min":1.0149999999999999,"15min":1.212},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.755,"5min":0.69799999999999995,"15min":0.73199999999999998},"min":{"1min":0.48499999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.1759999999999999},"last":1.0700000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68500000000000005,"15min":0.72999999999999998},"min":{"1min":0.47399999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.98099999999999998,"5min":1.046,"15min":1.2789999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.68100000000000005,"15min":0.71899999999999997},"min":{"1min":0.57199999999999995,"5min":0.44,"15min":0.439},"max":{"1min":0.89300000000000002,"5min":0.99299999999999999,"15min":1.226},"last":0.747}]},{"osd":3,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.69399999999999995,"15min":0.73399999999999999},"min":{"1min":0.501,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.98099999999999998,"5min":1.026,"15min":1.2549999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.73599999999999999,"15min":0.76300000000000001},"min":{"1min":0.60499999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.95799999999999996,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.65800000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64900000000000002,"15min":0.625},"min":{"1min":0.439,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0329999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68400000000000005,"15min":0.66800000000000004},"min":{"1min":0.46800000000000003,"5min":0.373,"15min":0.373},"max":{"1min":0.96599999999999997,"5min":1.1439999999999999,"15min":1.1879999999999999},"last":0.96599999999999997}]},{"osd":5,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66000000000000003,"15min":0.65500000000000003},"min":{"1min":0.48399999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.92300000000000004,"5min":0.98699999999999999,"15min":1.2669999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.53800000000000003,"5min":0.436,"15min":0.436},"max":{"1min":1.095,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":1.095}]},{"osd":7,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.58599999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.999,"5min":1.131,"15min":1.2370000000000001},"last":0.871},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73199999999999998,"15min":0.73499999999999999},"min":{"1min":0.54200000000000004,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.01,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":1.01}]}]},{"osd":1,"up_from":13,"seq":55834576035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57356,"kb_used_data":4092,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314484,"statfs":{"total":96636764160,"available":96578031616,"internally_reserved":0,"allocated":4190208,"data_stored":2441766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.437,"15min":0.44600000000000001},"min":{"1min":0.33900000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.73699999999999999,"5min":1.76,"15min":1.76},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.48299999999999998,"15min":0.503},"min":{"1min":0.33700000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.39700000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.48399999999999999,"15min":0.497},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.70599999999999996,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.45100000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.84199999999999997,"5min":1.859,"15min":1.859},"last":0.54700000000000004}]},{"osd":3,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.55300000000000005,"15min":0.56799999999999995},"min":{"1min":0.46899999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82299999999999995,"5min":1.8169999999999999,"15min":1.8169999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.52900000000000003,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.73099999999999998,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.45200000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.755,"15min":0.77600000000000002},"min":{"1min":0.65300000000000002,"5min":0.44500000000000001,"15min":0.433},"max":{"1min":1.0720000000000001,"5min":1.948,"15min":1.948},"last":0.622},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.72899999999999998,"15min":0.76200000000000001},"min":{"1min":0.53600000000000003,"5min":0.48999999999999999,"15min":0.41799999999999998},"max":{"1min":1.018,"5min":1.1060000000000001,"15min":1.25},"last":0.73799999999999999}]},{"osd":5,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76700000000000002,"15min":0.77900000000000003},"min":{"1min":0.59699999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.90700000000000003,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.79900000000000004,"15min":0.80500000000000005},"min":{"1min":0.64400000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.0069999999999999,"5min":4.5229999999999997,"15min":4.5229999999999997},"last":0.63}]},{"osd":6,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.79200000000000004,"15min":0.80400000000000005},"min":{"1min":0.58199999999999996,"5min":0.501,"15min":0.48099999999999998},"max":{"1min":0.997,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.77400000000000002,"15min":0.80000000000000004},"min":{"1min":0.59999999999999998,"5min":0.52900000000000003,"15min":0.49099999999999999},"max":{"1min":0.97899999999999998,"5min":1.2290000000000001,"15min":1.2529999999999999},"last":0.68799999999999994}]},{"osd":7,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.78600000000000003,"15min":0.81000000000000005},"min":{"1min":0.67400000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0589999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.80300000000000005,"15min":0.82399999999999995},"min":{"1min":0.64800000000000002,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0920000000000001,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.70199999999999996}]}]},{"osd":0,"up_from":13,"seq":55834576035,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52672,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319168,"statfs":{"total":96636764160,"available":96582828032,"internally_reserved":0,"allocated":3592192,"data_stored":1850379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.42799999999999999,"15min":0.44500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.628,"5min":0.77100000000000002,"15min":0.78100000000000003},"last":0.317},{"interface":"front","average":{"1min":0.5,"5min":0.46600000000000003,"15min":0.49099999999999999},"min":{"1min":0.29299999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65200000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.44800000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.50700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70099999999999996,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.50800000000000001,"15min":0.53800000000000003},"min":{"1min":0.41299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66000000000000003,"5min":0.72699999999999998,"15min":0.999},"last":0.54600000000000004}]},{"osd":3,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.54900000000000004},"min":{"1min":0.376,"5min":0.33400000000000002,"15min":0.312},"max":{"1min":0.67400000000000004,"5min":0.77800000000000002,"15min":0.85699999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58499999999999996,"15min":0.623},"min":{"1min":0.46100000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.73599999999999999,"5min":0.82799999999999996,"15min":0.92000000000000004},"last":0.754}]},{"osd":4,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.77200000000000002,"15min":0.81399999999999995},"min":{"1min":0.61199999999999999,"5min":0.55000000000000004,"15min":0.45600000000000002},"max":{"1min":1.129,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.628,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0369999999999999,"5min":1.169,"15min":1.169},"last":0.85299999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.77000000000000002,"15min":0.80400000000000005},"min":{"1min":0.56599999999999995,"5min":0.54000000000000004,"15min":0.497},"max":{"1min":1.0509999999999999,"5min":1.1419999999999999,"15min":1.232},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77000000000000002,"15min":0.81399999999999995},"min":{"1min":0.53200000000000003,"5min":0.497,"15min":0.497},"max":{"1min":1.0669999999999999,"5min":1.21,"15min":1.21},"last":0.79800000000000004}]},{"osd":6,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74399999999999999,"15min":0.78300000000000003},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.02,"5min":1.1830000000000001,"15min":1.196},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.76600000000000001,"15min":0.79600000000000004},"min":{"1min":0.55000000000000004,"5min":0.53700000000000003,"15min":0.52900000000000003},"max":{"1min":1.073,"5min":1.246,"15min":1.246},"last":0.93200000000000005}]},{"osd":7,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.81899999999999995,"15min":0.84499999999999997},"min":{"1min":0.64900000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.879},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.80200000000000005,"15min":0.83299999999999996},"min":{"1min":0.60799999999999998,"5min":0.499,"15min":0.499},"max":{"1min":2.3889999999999998,"5min":2.3889999999999998,"15min":2.3889999999999998},"last":0.94499999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576034,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.435,"15min":0.438},"min":{"1min":0.29699999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56699999999999995,"5min":1.681,"15min":1.681},"last":0.435},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":0.82899999999999996,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":0.35899999999999999}]},{"osd":1,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.5},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65800000000000003,"5min":1.72,"15min":1.72},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53100000000000003,"15min":0.54800000000000004},"min":{"1min":0.38200000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92900000000000005,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.51900000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53000000000000003,"15min":0.54200000000000004},"min":{"1min":0.38300000000000001,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.871,"5min":0.871,"15min":1.2749999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58699999999999997,"15min":0.61799999999999999},"min":{"1min":0.46600000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82499999999999996,"5min":1.754,"15min":1.754},"last":0.5}]},{"osd":4,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.75600000000000001,"15min":0.76800000000000002},"min":{"1min":0.53300000000000003,"5min":0.495,"15min":0.495},"max":{"1min":1.0940000000000001,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.48999999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.02,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.97999999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.75600000000000001,"15min":0.77400000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.177,"5min":1.177,"15min":1.99},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.748,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.081,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":1.1379999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76800000000000002,"15min":0.79700000000000004},"min":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1399999999999999,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.198},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.76900000000000002,"15min":0.79200000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.014,"5min":1.103,"15min":1.3819999999999999},"last":1.0720000000000001}]},{"osd":7,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.78500000000000003,"15min":0.81499999999999995},"min":{"1min":0.55300000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.196,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.81200000000000006},"min":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.497},"max":{"1min":1.1200000000000001,"5min":1.603,"15min":1.603},"last":1.014}]}]},{"osd":3,"up_from":13,"seq":55834576034,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56780,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315060,"statfs":{"total":96636764160,"available":96578621440,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41599999999999998,"15min":0.437},"min":{"1min":0.32500000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68000000000000005,"5min":0.73599999999999999,"15min":0.76000000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.436,"15min":0.47399999999999998},"min":{"1min":0.34100000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57199999999999995,"5min":0.68300000000000005,"15min":0.876},"last":0.38600000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.47799999999999998,"15min":0.49399999999999999},"min":{"1min":0.35199999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.77600000000000002,"5min":0.81699999999999995,"15min":0.90300000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52400000000000002,"15min":0.55900000000000005},"min":{"1min":0.41499999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.754,"5min":0.754,"15min":0.98499999999999999},"last":0.48599999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55200000000000005,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.93100000000000005},"last":0.44},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55200000000000005,"15min":0.59699999999999998},"min":{"1min":0.38,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.877,"5min":0.877,"15min":1.0840000000000001},"last":0.46100000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70599999999999996,"15min":0.73999999999999999},"min":{"1min":0.498,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86199999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.73099999999999998,"15min":0.75800000000000001},"min":{"1min":0.57299999999999995,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.91700000000000004,"5min":1.0760000000000001,"15min":1.1619999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72999999999999998,"15min":0.76500000000000001},"min":{"1min":0.54400000000000004,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.874,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.73299999999999998,"15min":0.76800000000000002},"min":{"1min":0.55000000000000004,"5min":0.51100000000000001,"15min":0.505},"max":{"1min":0.97199999999999998,"5min":1.0780000000000001,"15min":1.266},"last":0.56100000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.748,"15min":0.77400000000000002},"min":{"1min":0.57199999999999995,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.056,"5min":1.0900000000000001,"15min":1.284},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77200000000000002,"15min":0.79900000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.495},"max":{"1min":0.93799999999999994,"5min":1.093,"15min":1.2},"last":0.627}]},{"osd":7,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.745,"15min":0.79100000000000004},"min":{"1min":0.52700000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0840000000000001,"5min":1.125,"15min":1.3260000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.78000000000000003,"15min":0.81000000000000005},"min":{"1min":0.65500000000000003,"5min":0.495,"15min":0.495},"max":{"1min":1.0620000000000001,"5min":1.087,"15min":1.248},"last":0.69999999999999996}]}]},{"osd":4,"up_from":13,"seq":55834576034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48584,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323256,"statfs":{"total":96636764160,"available":96587014144,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63300000000000001,"15min":0.67200000000000004},"min":{"1min":0.49099999999999999,"5min":0.38100000000000001,"15min":0.29299999999999998},"max":{"1min":0.86799999999999999,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.63400000000000001,"15min":0.66900000000000004},"min":{"1min":0.505,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.92800000000000005,"5min":0.97099999999999997,"15min":1.075},"last":0.61899999999999999}]},{"osd":1,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.70699999999999996,"15min":0.74099999999999999},"min":{"1min":0.53200000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.996,"5min":1.0329999999999999,"15min":1.069},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.71999999999999997,"15min":0.751},"min":{"1min":0.48099999999999998,"5min":0.46100000000000002,"15min":0.40799999999999997},"max":{"1min":0.89500000000000002,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.75600000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65600000000000003,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.95399999999999996,"5min":1.077,"15min":1.077},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67600000000000005,"15min":0.71199999999999997},"min":{"1min":0.50900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95999999999999996,"5min":1.032,"15min":1.0740000000000001},"last":0.48399999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.71599999999999997,"15min":0.749},"min":{"1min":0.53000000000000003,"5min":0.48799999999999999,"15min":0.47399999999999998},"max":{"1min":0.94599999999999995,"5min":0.98099999999999998,"15min":1.018},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.69899999999999995,"15min":0.73999999999999999},"min":{"1min":0.59699999999999998,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":0.93799999999999994,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.72499999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.47899999999999998,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.88200000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.629,"15min":0.625},"min":{"1min":0.46000000000000002,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.82699999999999996,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.77400000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.50800000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81599999999999995,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.68799999999999994,"15min":0.69399999999999995},"min":{"1min":0.49299999999999999,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":0.74399999999999999,"5min":1.0149999999999999,"15min":1.0269999999999999},"last":0.85499999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.70299999999999996,"15min":0.70599999999999996},"min":{"1min":0.48399999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.90200000000000002,"5min":1.002,"15min":1.002},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76900000000000002,"15min":0.77700000000000002},"min":{"1min":0.56399999999999995,"5min":0.499,"15min":0.499},"max":{"1min":0.88900000000000001,"5min":1.0960000000000001,"15min":1.0980000000000001},"last":0.96199999999999997}]}]},{"osd":5,"up_from":13,"seq":55834576037,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53252,"kb_used_data":4088,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318588,"statfs":{"total":96636764160,"available":96582234112,"internally_reserved":0,"allocated":4186112,"data_stored":2440747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.66800000000000004,"15min":0.68999999999999995},"min":{"1min":0.54300000000000004,"5min":0.42199999999999999,"15min":0.39600000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":1.004},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.52800000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.90800000000000003,"5min":1.085,"15min":1.085},"last":0.65400000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.66400000000000003,"15min":0.70399999999999996},"min":{"1min":0.55700000000000005,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89000000000000001,"5min":0.94499999999999995,"15min":1.0189999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.69899999999999995,"15min":0.71299999999999997},"min":{"1min":0.60399999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.89400000000000002,"5min":0.96799999999999997,"15min":1.052},"last":0.72199999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.69499999999999995,"15min":0.72799999999999998},"min":{"1min":0.57899999999999996,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92400000000000004,"5min":0.97299999999999998,"15min":1.014},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.69299999999999995,"15min":0.72499999999999998},"min":{"1min":0.622,"5min":0.436,"15min":0.436},"max":{"1min":0.96199999999999997,"5min":1.018,"15min":1.0680000000000001},"last":0.49399999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.71399999999999997,"15min":0.76000000000000001},"min":{"1min":0.59399999999999997,"5min":0.496,"15min":0.496},"max":{"1min":0.92700000000000005,"5min":0.93600000000000005,"15min":1.083},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.76900000000000002,"15min":0.78100000000000003},"min":{"1min":0.71799999999999997,"5min":0.50600000000000001,"15min":0.47399999999999998},"max":{"1min":0.92600000000000005,"5min":1.0109999999999999,"15min":1.1020000000000001},"last":0.63300000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65100000000000002,"15min":0.63100000000000001},"min":{"1min":0.45000000000000001,"5min":0.40699999999999997,"15min":0.38300000000000001},"max":{"1min":0.91300000000000003,"5min":0.98399999999999999,"15min":1.054},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.68700000000000006,"15min":0.66900000000000004},"min":{"1min":0.46300000000000002,"5min":0.45900000000000002,"15min":0.40400000000000003},"max":{"1min":0.93999999999999995,"5min":1.145,"15min":1.145},"last":0.748}]},{"osd":6,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.67700000000000005,"15min":0.67400000000000004},"min":{"1min":0.497,"5min":0.434,"15min":0.41099999999999998},"max":{"1min":0.79000000000000004,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71599999999999997,"15min":0.70999999999999996},"min":{"1min":0.50600000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.96399999999999997,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.62}]},{"osd":7,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.73899999999999999,"15min":0.73199999999999998},"min":{"1min":0.51900000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82699999999999996,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.75,"5min":0.77000000000000002,"15min":0.76300000000000001},"min":{"1min":0.52700000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95999999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.75900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:17:15.997 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-02T09:17:15.997 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:17:16.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.190+0000 7f28cc600700 1 Processor -- start 2024-04-02T09:17:16.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.190+0000 7f28cc600700 1 -- start start 2024-04-02T09:17:16.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.191+0000 7f28cc600700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f28c415d670 0x7f28c40ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:16.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.191+0000 7f28cc600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28c406dee0 0x7f28c406e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:16.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.191+0000 7f28cc600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c40b3890 con 0x7f28c415da50 2024-04-02T09:17:16.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.191+0000 7f28cc600700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c40acac0 con 0x7f28c415d670 2024-04-02T09:17:16.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.191+0000 7f28cc600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c4159f50 con 0x7f28c406dee0 2024-04-02T09:17:16.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.192+0000 7f28ca600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28c406dee0 0x7f28c406e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:16.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.192+0000 7f28c9c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f28c415d670 0x7f28c40ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:16.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.192+0000 7f28c9200700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f28c415da50 0x7f28c40b2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:38362/0 (socket says 172.21.15.89:38362) 2024-04-02T09:17:16.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.192+0000 7f28c9200700 1 -- 172.21.15.89:0/297626472 learned_addr learned my addr 172.21.15.89:0/297626472 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:16.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.193+0000 7f28c9c00700 1 -- 172.21.15.89:0/297626472 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28c406dee0 msgr2=0x7f28c406e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:16.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.193+0000 7f28c9c00700 1 --2- 172.21.15.89:0/297626472 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28c406dee0 0x7f28c406e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.193+0000 7f28c9c00700 1 -- 172.21.15.89:0/297626472 >> v1:172.21.15.113:6789/0 conn(0x7f28c415da50 legacy=0x7f28c40b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:16.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.193+0000 7f28c9c00700 1 -- 172.21.15.89:0/297626472 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c40b35b0 con 0x7f28c415d670 2024-04-02T09:17:16.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.193+0000 7f28ca600700 1 --2- 172.21.15.89:0/297626472 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28c406dee0 0x7f28c406e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:17:16.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.193+0000 7f28c3e00700 1 -- 172.21.15.89:0/297626472 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320852133 0 0) 0x7f28c40b3890 con 0x7f28c415da50 2024-04-02T09:17:16.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.193+0000 7f28c9c00700 1 --2- 172.21.15.89:0/297626472 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28c415d670 0x7f28c40ac580 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f28b8009840 tx=0x7f28b8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5941ae2de91f4573 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:16.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.193+0000 7f28c3e00700 1 -- 172.21.15.89:0/297626472 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8011830 con 0x7f28c415d670 2024-04-02T09:17:16.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.193+0000 7f28c3e00700 1 -- 172.21.15.89:0/297626472 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b8011990 con 0x7f28c415d670 2024-04-02T09:17:16.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.193+0000 7f28c3e00700 1 -- 172.21.15.89:0/297626472 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8011d20 con 0x7f28c415d670 2024-04-02T09:17:16.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.194+0000 7f28cc600700 1 -- 172.21.15.89:0/297626472 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28c415d670 msgr2=0x7f28c40ac580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:16.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.194+0000 7f28cc600700 1 --2- 172.21.15.89:0/297626472 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28c415d670 0x7f28c40ac580 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f28b8009840 tx=0x7f28b8009b50 comp rx=0 tx=0).stop 2024-04-02T09:17:16.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.194+0000 7f28cc600700 1 -- 172.21.15.89:0/297626472 shutdown_connections 2024-04-02T09:17:16.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.194+0000 7f28cc600700 1 --2- 172.21.15.89:0/297626472 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28c406dee0 0x7f28c406e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.194+0000 7f28cc600700 1 --2- 172.21.15.89:0/297626472 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28c415d670 0x7f28c40ac580 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.194+0000 7f28cc600700 1 -- 172.21.15.89:0/297626472 >> 172.21.15.89:0/297626472 conn(0x7f28c40cf190 msgr2=0x7f28c40cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:16.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.194+0000 7f28cc600700 1 -- 172.21.15.89:0/297626472 shutdown_connections 2024-04-02T09:17:16.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.194+0000 7f28cc600700 1 -- 172.21.15.89:0/297626472 wait complete. 2024-04-02T09:17:16.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.196+0000 7f28cc600700 1 Processor -- start 2024-04-02T09:17:16.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.196+0000 7f28cc600700 1 -- start start 2024-04-02T09:17:16.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28cc600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28c406dee0 0x7f28c40cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:16.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28cc600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28c415d670 0x7f28c41421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:16.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28cc600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28c415da50 0x7f28c4142700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:16.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28cc600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c4150850 con 0x7f28c406dee0 2024-04-02T09:17:16.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28cc600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c4097ea0 con 0x7f28c415d670 2024-04-02T09:17:16.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28cc600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c4098d00 con 0x7f28c415da50 2024-04-02T09:17:16.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28c9c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28c406dee0 0x7f28c40cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:16.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28c9c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28c406dee0 0x7f28c40cc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56906/0 (socket says 172.21.15.89:56906) 2024-04-02T09:17:16.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28c9c00700 1 -- 172.21.15.89:0/3775534805 learned_addr learned my addr 172.21.15.89:0/3775534805 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:16.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28ca600700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28c415da50 0x7f28c4142700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:16.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28c9200700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28c415d670 0x7f28c41421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:16.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28c9c00700 1 -- 172.21.15.89:0/3775534805 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28c415da50 msgr2=0x7f28c4142700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:16.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28c9c00700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28c415da50 0x7f28c4142700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28c9c00700 1 -- 172.21.15.89:0/3775534805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28c415d670 msgr2=0x7f28c41421c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:16.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28c9c00700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28c415d670 0x7f28c41421c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28c9c00700 1 -- 172.21.15.89:0/3775534805 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc009a70 con 0x7f28c406dee0 2024-04-02T09:17:16.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.197+0000 7f28c9200700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28c415d670 0x7f28c41421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:17:16.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.198+0000 7f28c9c00700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28c406dee0 0x7f28c40cc3e0 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f28b80090b0 tx=0x7f28b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:16.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.198+0000 7f28c1600700 1 -- 172.21.15.89:0/3775534805 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b80036a0 con 0x7f28c406dee0 2024-04-02T09:17:16.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.198+0000 7f28c1600700 1 -- 172.21.15.89:0/3775534805 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b801f070 con 0x7f28c406dee0 2024-04-02T09:17:16.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.198+0000 7f28c1600700 1 -- 172.21.15.89:0/3775534805 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8003a20 con 0x7f28c406dee0 2024-04-02T09:17:16.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.198+0000 7f28cc600700 1 -- 172.21.15.89:0/3775534805 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b80094a0 con 0x7f28c406dee0 2024-04-02T09:17:16.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.198+0000 7f28cc600700 1 -- 172.21.15.89:0/3775534805 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c40983d0 con 0x7f28c406dee0 2024-04-02T09:17:16.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.199+0000 7f28cc600700 1 -- 172.21.15.89:0/3775534805 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c40ae7c0 con 0x7f28c406dee0 2024-04-02T09:17:16.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.200+0000 7f28c1600700 1 -- 172.21.15.89:0/3775534805 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f28b8003b80 con 0x7f28c406dee0 2024-04-02T09:17:16.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.201+0000 7f28c1600700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f28a406adb0 0x7f28a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:16.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.201+0000 7f28c1600700 1 -- 172.21.15.89:0/3775534805 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f28b8021070 con 0x7f28c406dee0 2024-04-02T09:17:16.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.201+0000 7f28c9200700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f28a406adb0 0x7f28a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:16.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.202+0000 7f28c9200700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f28a406adb0 0x7f28a406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f28b0007fd0 tx=0x7f28b0009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:16.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.206+0000 7f28c1600700 1 -- 172.21.15.89:0/3775534805 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f28b8055f10 con 0x7f28c406dee0 2024-04-02T09:17:16.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.402+0000 7f28cc600700 1 -- 172.21.15.89:0/3775534805 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f28c406d9c0 con 0x7f28a406adb0 2024-04-02T09:17:16.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.405+0000 7f28c1600700 1 -- 172.21.15.89:0/3775534805 <== mgr.13575 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50438 (secure 0 0 0) 0x7f28b0001420 con 0x7f28a406adb0 2024-04-02T09:17:16.404 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:17:16.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.407+0000 7f28cc600700 1 -- 172.21.15.89:0/3775534805 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f28a406adb0 msgr2=0x7f28a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:16.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.407+0000 7f28cc600700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f28a406adb0 0x7f28a406d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f28b0007fd0 tx=0x7f28b0009040 comp rx=0 tx=0).stop 2024-04-02T09:17:16.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.407+0000 7f28cc600700 1 -- 172.21.15.89:0/3775534805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28c406dee0 msgr2=0x7f28c40cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:16.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.407+0000 7f28cc600700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28c406dee0 0x7f28c40cc3e0 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f28b80090b0 tx=0x7f28b8004750 comp rx=0 tx=0).stop 2024-04-02T09:17:16.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.407+0000 7f28cc600700 1 -- 172.21.15.89:0/3775534805 shutdown_connections 2024-04-02T09:17:16.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.407+0000 7f28cc600700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28c415da50 0x7f28c4142700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.407+0000 7f28cc600700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28c406dee0 0x7f28c40cc3e0 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.407+0000 7f28cc600700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f28a406adb0 0x7f28a406d270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.407+0000 7f28cc600700 1 --2- 172.21.15.89:0/3775534805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28c415d670 0x7f28c41421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.407+0000 7f28cc600700 1 -- 172.21.15.89:0/3775534805 >> 172.21.15.89:0/3775534805 conn(0x7f28c40cf190 msgr2=0x7f28c40acc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:16.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.407+0000 7f28cc600700 1 -- 172.21.15.89:0/3775534805 shutdown_connections 2024-04-02T09:17:16.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.407+0000 7f28cc600700 1 -- 172.21.15.89:0/3775534805 wait complete. 2024-04-02T09:17:16.408 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:17:16.423 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-02T09:17:15.568937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":447176,"kb_used_data":29248,"kb_used_omap":12,"kb_used_meta":417779,"kb_avail":754527544,"statfs":{"total":773094113280,"available":772636205056,"internally_reserved":0,"allocated":29949952,"data_stored":15989882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":427806264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-766068,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-200,"num_read_kb":-56,"num_write":-289,"num_write_kb":-2297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.975074"},"pg_stats":[{"pgid":"1.0","version":"282'425","reported_seq":1539,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-04-02T09:17:10.886632+0000","last_change":"2024-04-02T07:35:43.317365+0000","last_active":"2024-04-02T09:17:10.886632+0000","last_peered":"2024-04-02T09:17:10.886632+0000","last_clean":"2024-04-02T09:17:10.886632+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:17:10.886632+0000","last_undegraded":"2024-04-02T09:17:10.886632+0000","last_fullsized":"2024-04-02T09:17:10.886632+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T07:35:41.934453+0000","last_clean_scrub_stamp":"2024-04-02T07:35:41.934453+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:28:40.577341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60876,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310964,"statfs":{"total":96636764160,"available":96574427136,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.70499999999999996,"15min":0.70999999999999996},"min":{"1min":0.437,"5min":0.309,"15min":0.309},"max":{"1min":0.96699999999999997,"5min":8.016,"15min":8.016},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.71399999999999997},"min":{"1min":0.57299999999999995,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.081,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.59999999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.74299999999999999,"15min":0.73599999999999999},"min":{"1min":0.46800000000000003,"5min":0.42199999999999999,"15min":0.38200000000000001},"max":{"1min":1.0069999999999999,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67500000000000004,"15min":0.70099999999999996},"min":{"1min":0.52300000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82899999999999996,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.58099999999999996}]},{"osd":2,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.748,"15min":0.75900000000000001},"min":{"1min":0.48299999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91600000000000004,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79500000000000004,"15min":0.80200000000000005},"min":{"1min":0.60199999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.1140000000000001,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.68300000000000005}]},{"osd":3,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.747,"15min":0.74399999999999999},"min":{"1min":0.56799999999999995,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.94899999999999995,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.77200000000000002,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.91700000000000004,"5min":5.093,"15min":5.093},"last":0.73799999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.65900000000000003,"15min":0.63100000000000001},"min":{"1min":0.45800000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.99099999999999999,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.79300000000000004,"15min":0.75},"min":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.03,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.77600000000000002}]},{"osd":5,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.72799999999999998,"15min":0.69799999999999995},"min":{"1min":0.45900000000000002,"5min":0.41599999999999998,"15min":0.41199999999999998},"max":{"1min":0.94699999999999995,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.754,"15min":0.73699999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.93300000000000005,"5min":3.492,"15min":3.492},"last":0.76000000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.70899999999999996,"15min":0.69999999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97499999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.76900000000000002,"15min":0.72499999999999998},"min":{"1min":0.42299999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.96099999999999997,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.82499999999999996}]}]},{"osd":6,"up_from":13,"seq":55834576034,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64972,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306868,"statfs":{"total":96636764160,"available":96570232832,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.65900000000000003,"15min":0.70799999999999996},"min":{"1min":0.52000000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.94199999999999995,"5min":0.94899999999999995,"15min":1.1879999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.68700000000000006,"15min":0.72199999999999998},"min":{"1min":0.52100000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83999999999999997,"5min":0.98399999999999999,"15min":1.0600000000000001},"last":0.72099999999999997}]},{"osd":1,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.59499999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.95099999999999996,"5min":1.0149999999999999,"15min":1.212},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.755,"5min":0.69799999999999995,"15min":0.73199999999999998},"min":{"1min":0.48499999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.1759999999999999},"last":1.0700000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68500000000000005,"15min":0.72999999999999998},"min":{"1min":0.47399999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.98099999999999998,"5min":1.046,"15min":1.2789999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.68100000000000005,"15min":0.71899999999999997},"min":{"1min":0.57199999999999995,"5min":0.44,"15min":0.439},"max":{"1min":0.89300000000000002,"5min":0.99299999999999999,"15min":1.226},"last":0.747}]},{"osd":3,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.69399999999999995,"15min":0.73399999999999999},"min":{"1min":0.501,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.98099999999999998,"5min":1.026,"15min":1.2549999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.73599999999999999,"15min":0.76300000000000001},"min":{"1min":0.60499999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.95799999999999996,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.65800000000000003}]},{"osd":4,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64900000000000002,"15min":0.625},"min":{"1min":0.439,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0329999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68400000000000005,"15min":0.66800000000000004},"min":{"1min":0.46800000000000003,"5min":0.373,"15min":0.373},"max":{"1min":0.96599999999999997,"5min":1.1439999999999999,"15min":1.1879999999999999},"last":0.96599999999999997}]},{"osd":5,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66000000000000003,"15min":0.65500000000000003},"min":{"1min":0.48399999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.92300000000000004,"5min":0.98699999999999999,"15min":1.2669999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.53800000000000003,"5min":0.436,"15min":0.436},"max":{"1min":1.095,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":1.095}]},{"osd":7,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.58599999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.999,"5min":1.131,"15min":1.2370000000000001},"last":0.871},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73199999999999998,"15min":0.73499999999999999},"min":{"1min":0.54200000000000004,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.01,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":1.01}]}]},{"osd":1,"up_from":13,"seq":55834576035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57356,"kb_used_data":4092,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314484,"statfs":{"total":96636764160,"available":96578031616,"internally_reserved":0,"allocated":4190208,"data_stored":2441766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.437,"15min":0.44600000000000001},"min":{"1min":0.33900000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.73699999999999999,"5min":1.76,"15min":1.76},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.48299999999999998,"15min":0.503},"min":{"1min":0.33700000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.39700000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.48399999999999999,"15min":0.497},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.70599999999999996,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.45100000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.84199999999999997,"5min":1.859,"15min":1.859},"last":0.54700000000000004}]},{"osd":3,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.55300000000000005,"15min":0.56799999999999995},"min":{"1min":0.46899999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82299999999999995,"5min":1.8169999999999999,"15min":1.8169999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.52900000000000003,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.73099999999999998,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.45200000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.755,"15min":0.77600000000000002},"min":{"1min":0.65300000000000002,"5min":0.44500000000000001,"15min":0.433},"max":{"1min":1.0720000000000001,"5min":1.948,"15min":1.948},"last":0.622},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.72899999999999998,"15min":0.76200000000000001},"min":{"1min":0.53600000000000003,"5min":0.48999999999999999,"15min":0.41799999999999998},"max":{"1min":1.018,"5min":1.1060000000000001,"15min":1.25},"last":0.73799999999999999}]},{"osd":5,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76700000000000002,"15min":0.77900000000000003},"min":{"1min":0.59699999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.90700000000000003,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.79900000000000004,"15min":0.80500000000000005},"min":{"1min":0.64400000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.0069999999999999,"5min":4.5229999999999997,"15min":4.5229999999999997},"last":0.63}]},{"osd":6,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.79200000000000004,"15min":0.80400000000000005},"min":{"1min":0.58199999999999996,"5min":0.501,"15min":0.48099999999999998},"max":{"1min":0.997,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.77400000000000002,"15min":0.80000000000000004},"min":{"1min":0.59999999999999998,"5min":0.52900000000000003,"15min":0.49099999999999999},"max":{"1min":0.97899999999999998,"5min":1.2290000000000001,"15min":1.2529999999999999},"last":0.68799999999999994}]},{"osd":7,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.78600000000000003,"15min":0.81000000000000005},"min":{"1min":0.67400000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0589999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.80300000000000005,"15min":0.82399999999999995},"min":{"1min":0.64800000000000002,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0920000000000001,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.70199999999999996}]}]},{"osd":0,"up_from":13,"seq":55834576035,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52672,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319168,"statfs":{"total":96636764160,"available":96582828032,"internally_reserved":0,"allocated":3592192,"data_stored":1850379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.42799999999999999,"15min":0.44500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.628,"5min":0.77100000000000002,"15min":0.78100000000000003},"last":0.317},{"interface":"front","average":{"1min":0.5,"5min":0.46600000000000003,"15min":0.49099999999999999},"min":{"1min":0.29299999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65200000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.44800000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.50700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70099999999999996,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.50800000000000001,"15min":0.53800000000000003},"min":{"1min":0.41299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66000000000000003,"5min":0.72699999999999998,"15min":0.999},"last":0.54600000000000004}]},{"osd":3,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.54900000000000004},"min":{"1min":0.376,"5min":0.33400000000000002,"15min":0.312},"max":{"1min":0.67400000000000004,"5min":0.77800000000000002,"15min":0.85699999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58499999999999996,"15min":0.623},"min":{"1min":0.46100000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.73599999999999999,"5min":0.82799999999999996,"15min":0.92000000000000004},"last":0.754}]},{"osd":4,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.77200000000000002,"15min":0.81399999999999995},"min":{"1min":0.61199999999999999,"5min":0.55000000000000004,"15min":0.45600000000000002},"max":{"1min":1.129,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.628,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0369999999999999,"5min":1.169,"15min":1.169},"last":0.85299999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.77000000000000002,"15min":0.80400000000000005},"min":{"1min":0.56599999999999995,"5min":0.54000000000000004,"15min":0.497},"max":{"1min":1.0509999999999999,"5min":1.1419999999999999,"15min":1.232},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77000000000000002,"15min":0.81399999999999995},"min":{"1min":0.53200000000000003,"5min":0.497,"15min":0.497},"max":{"1min":1.0669999999999999,"5min":1.21,"15min":1.21},"last":0.79800000000000004}]},{"osd":6,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74399999999999999,"15min":0.78300000000000003},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.02,"5min":1.1830000000000001,"15min":1.196},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.76600000000000001,"15min":0.79600000000000004},"min":{"1min":0.55000000000000004,"5min":0.53700000000000003,"15min":0.52900000000000003},"max":{"1min":1.073,"5min":1.246,"15min":1.246},"last":0.93200000000000005}]},{"osd":7,"last update":"Tue Apr 2 09:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.81899999999999995,"15min":0.84499999999999997},"min":{"1min":0.64900000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.879},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.80200000000000005,"15min":0.83299999999999996},"min":{"1min":0.60799999999999998,"5min":0.499,"15min":0.499},"max":{"1min":2.3889999999999998,"5min":2.3889999999999998,"15min":2.3889999999999998},"last":0.94499999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576034,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.435,"15min":0.438},"min":{"1min":0.29699999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56699999999999995,"5min":1.681,"15min":1.681},"last":0.435},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":0.82899999999999996,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":0.35899999999999999}]},{"osd":1,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.5},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65800000000000003,"5min":1.72,"15min":1.72},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53100000000000003,"15min":0.54800000000000004},"min":{"1min":0.38200000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92900000000000005,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.51900000000000002}]},{"osd":3,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53000000000000003,"15min":0.54200000000000004},"min":{"1min":0.38300000000000001,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.871,"5min":0.871,"15min":1.2749999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58699999999999997,"15min":0.61799999999999999},"min":{"1min":0.46600000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82499999999999996,"5min":1.754,"15min":1.754},"last":0.5}]},{"osd":4,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.75600000000000001,"15min":0.76800000000000002},"min":{"1min":0.53300000000000003,"5min":0.495,"15min":0.495},"max":{"1min":1.0940000000000001,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.75700000000000001,"15min":0.78200000000000003},"min":{"1min":0.48999999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.02,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.97999999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.75600000000000001,"15min":0.77400000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.177,"5min":1.177,"15min":1.99},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.748,"5min":0.76500000000000001,"15min":0.79000000000000004},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.081,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":1.1379999999999999}]},{"osd":6,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76800000000000002,"15min":0.79700000000000004},"min":{"1min":0.51900000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1399999999999999,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.198},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.76900000000000002,"15min":0.79200000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.014,"5min":1.103,"15min":1.3819999999999999},"last":1.0720000000000001}]},{"osd":7,"last update":"Tue Apr 2 09:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.78500000000000003,"15min":0.81499999999999995},"min":{"1min":0.55300000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.196,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.81200000000000006},"min":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.497},"max":{"1min":1.1200000000000001,"5min":1.603,"15min":1.603},"last":1.014}]}]},{"osd":3,"up_from":13,"seq":55834576034,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56780,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315060,"statfs":{"total":96636764160,"available":96578621440,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41599999999999998,"15min":0.437},"min":{"1min":0.32500000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68000000000000005,"5min":0.73599999999999999,"15min":0.76000000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.436,"15min":0.47399999999999998},"min":{"1min":0.34100000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57199999999999995,"5min":0.68300000000000005,"15min":0.876},"last":0.38600000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.47799999999999998,"15min":0.49399999999999999},"min":{"1min":0.35199999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.77600000000000002,"5min":0.81699999999999995,"15min":0.90300000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52400000000000002,"15min":0.55900000000000005},"min":{"1min":0.41499999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.754,"5min":0.754,"15min":0.98499999999999999},"last":0.48599999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55200000000000005,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.93100000000000005},"last":0.44},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55200000000000005,"15min":0.59699999999999998},"min":{"1min":0.38,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.877,"5min":0.877,"15min":1.0840000000000001},"last":0.46100000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70599999999999996,"15min":0.73999999999999999},"min":{"1min":0.498,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86199999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.73099999999999998,"15min":0.75800000000000001},"min":{"1min":0.57299999999999995,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.91700000000000004,"5min":1.0760000000000001,"15min":1.1619999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72999999999999998,"15min":0.76500000000000001},"min":{"1min":0.54400000000000004,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.874,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.73299999999999998,"15min":0.76800000000000002},"min":{"1min":0.55000000000000004,"5min":0.51100000000000001,"15min":0.505},"max":{"1min":0.97199999999999998,"5min":1.0780000000000001,"15min":1.266},"last":0.56100000000000005}]},{"osd":6,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.748,"15min":0.77400000000000002},"min":{"1min":0.57199999999999995,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.056,"5min":1.0900000000000001,"15min":1.284},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77200000000000002,"15min":0.79900000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.495},"max":{"1min":0.93799999999999994,"5min":1.093,"15min":1.2},"last":0.627}]},{"osd":7,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.745,"15min":0.79100000000000004},"min":{"1min":0.52700000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0840000000000001,"5min":1.125,"15min":1.3260000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.78000000000000003,"15min":0.81000000000000005},"min":{"1min":0.65500000000000003,"5min":0.495,"15min":0.495},"max":{"1min":1.0620000000000001,"5min":1.087,"15min":1.248},"last":0.69999999999999996}]}]},{"osd":4,"up_from":13,"seq":55834576034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48584,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323256,"statfs":{"total":96636764160,"available":96587014144,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63300000000000001,"15min":0.67200000000000004},"min":{"1min":0.49099999999999999,"5min":0.38100000000000001,"15min":0.29299999999999998},"max":{"1min":0.86799999999999999,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.63400000000000001,"15min":0.66900000000000004},"min":{"1min":0.505,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.92800000000000005,"5min":0.97099999999999997,"15min":1.075},"last":0.61899999999999999}]},{"osd":1,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.70699999999999996,"15min":0.74099999999999999},"min":{"1min":0.53200000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.996,"5min":1.0329999999999999,"15min":1.069},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.71999999999999997,"15min":0.751},"min":{"1min":0.48099999999999998,"5min":0.46100000000000002,"15min":0.40799999999999997},"max":{"1min":0.89500000000000002,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.75600000000000001}]},{"osd":2,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65600000000000003,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.95399999999999996,"5min":1.077,"15min":1.077},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67600000000000005,"15min":0.71199999999999997},"min":{"1min":0.50900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95999999999999996,"5min":1.032,"15min":1.0740000000000001},"last":0.48399999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.71599999999999997,"15min":0.749},"min":{"1min":0.53000000000000003,"5min":0.48799999999999999,"15min":0.47399999999999998},"max":{"1min":0.94599999999999995,"5min":0.98099999999999998,"15min":1.018},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.69899999999999995,"15min":0.73999999999999999},"min":{"1min":0.59699999999999998,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":0.93799999999999994,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.72499999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.47899999999999998,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.88200000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.629,"15min":0.625},"min":{"1min":0.46000000000000002,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.82699999999999996,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.77400000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.50800000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81599999999999995,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.68799999999999994,"15min":0.69399999999999995},"min":{"1min":0.49299999999999999,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":0.74399999999999999,"5min":1.0149999999999999,"15min":1.0269999999999999},"last":0.85499999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.70299999999999996,"15min":0.70599999999999996},"min":{"1min":0.48399999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.90200000000000002,"5min":1.002,"15min":1.002},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76900000000000002,"15min":0.77700000000000002},"min":{"1min":0.56399999999999995,"5min":0.499,"15min":0.499},"max":{"1min":0.88900000000000001,"5min":1.0960000000000001,"15min":1.0980000000000001},"last":0.96199999999999997}]}]},{"osd":5,"up_from":13,"seq":55834576037,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53252,"kb_used_data":4088,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318588,"statfs":{"total":96636764160,"available":96582234112,"internally_reserved":0,"allocated":4186112,"data_stored":2440747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.66800000000000004,"15min":0.68999999999999995},"min":{"1min":0.54300000000000004,"5min":0.42199999999999999,"15min":0.39600000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":1.004},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.52800000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.90800000000000003,"5min":1.085,"15min":1.085},"last":0.65400000000000003}]},{"osd":1,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.66400000000000003,"15min":0.70399999999999996},"min":{"1min":0.55700000000000005,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89000000000000001,"5min":0.94499999999999995,"15min":1.0189999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.69899999999999995,"15min":0.71299999999999997},"min":{"1min":0.60399999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.89400000000000002,"5min":0.96799999999999997,"15min":1.052},"last":0.72199999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.69499999999999995,"15min":0.72799999999999998},"min":{"1min":0.57899999999999996,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92400000000000004,"5min":0.97299999999999998,"15min":1.014},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.69299999999999995,"15min":0.72499999999999998},"min":{"1min":0.622,"5min":0.436,"15min":0.436},"max":{"1min":0.96199999999999997,"5min":1.018,"15min":1.0680000000000001},"last":0.49399999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.71399999999999997,"15min":0.76000000000000001},"min":{"1min":0.59399999999999997,"5min":0.496,"15min":0.496},"max":{"1min":0.92700000000000005,"5min":0.93600000000000005,"15min":1.083},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.76900000000000002,"15min":0.78100000000000003},"min":{"1min":0.71799999999999997,"5min":0.50600000000000001,"15min":0.47399999999999998},"max":{"1min":0.92600000000000005,"5min":1.0109999999999999,"15min":1.1020000000000001},"last":0.63300000000000001}]},{"osd":4,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65100000000000002,"15min":0.63100000000000001},"min":{"1min":0.45000000000000001,"5min":0.40699999999999997,"15min":0.38300000000000001},"max":{"1min":0.91300000000000003,"5min":0.98399999999999999,"15min":1.054},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.68700000000000006,"15min":0.66900000000000004},"min":{"1min":0.46300000000000002,"5min":0.45900000000000002,"15min":0.40400000000000003},"max":{"1min":0.93999999999999995,"5min":1.145,"15min":1.145},"last":0.748}]},{"osd":6,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.67700000000000005,"15min":0.67400000000000004},"min":{"1min":0.497,"5min":0.434,"15min":0.41099999999999998},"max":{"1min":0.79000000000000004,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71599999999999997,"15min":0.70999999999999996},"min":{"1min":0.50600000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.96399999999999997,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.62}]},{"osd":7,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.73899999999999999,"15min":0.73199999999999998},"min":{"1min":0.51900000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82699999999999996,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.75,"5min":0.77000000000000002,"15min":0.76300000000000001},"min":{"1min":0.52700000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95999999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.75900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:17:16.425 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T09:17:16.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.616+0000 7f9e81a00700 1 Processor -- start 2024-04-02T09:17:16.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.616+0000 7f9e81a00700 1 -- start start 2024-04-02T09:17:16.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.617+0000 7f9e81a00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9e7c160f10 0x7f9e7c0d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:16.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.617+0000 7f9e81a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e7c161590 0x7f9e7c161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.617+0000 7f9e81a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c0ceda0 con 0x7f9e7c0cf230 2024-04-02T09:17:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.617+0000 7f9e81a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e7c161f80 con 0x7f9e7c161590 2024-04-02T09:17:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.617+0000 7f9e81a00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e7c1612f0 con 0x7f9e7c160f10 2024-04-02T09:17:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.618+0000 7f9e7b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e7c161590 0x7f9e7c161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.618+0000 7f9e7aa00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9e7c160f10 0x7f9e7c0d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.618+0000 7f9e7b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e7c161590 0x7f9e7c161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56908/0 (socket says 172.21.15.89:56908) 2024-04-02T09:17:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.618+0000 7f9e7b400700 1 -- 172.21.15.89:0/3351193990 learned_addr learned my addr 172.21.15.89:0/3351193990 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:16.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.618+0000 7f9e7b400700 1 -- 172.21.15.89:0/3351193990 >> v2:172.21.15.113:3300/0 conn(0x7f9e7c160f10 msgr2=0x7f9e7c0d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:16.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.618+0000 7f9e7b400700 1 --2- 172.21.15.89:0/3351193990 >> v2:172.21.15.113:3300/0 conn(0x7f9e7c160f10 0x7f9e7c0d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.618+0000 7f9e7b400700 1 -- 172.21.15.89:0/3351193990 >> v1:172.21.15.113:6789/0 conn(0x7f9e7c0cf230 legacy=0x7f9e7c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:16.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.618+0000 7f9e7b400700 1 -- 172.21.15.89:0/3351193990 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c0ce470 con 0x7f9e7c161590 2024-04-02T09:17:16.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.618+0000 7f9e7b400700 1 --2- 172.21.15.89:0/3351193990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c161590 0x7f9e7c161a40 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c0d1080 tx=0x7f9e7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ba668aa25d369e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:16.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.619+0000 7f9e79600700 1 -- 172.21.15.89:0/3351193990 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7400e070 con 0x7f9e7c161590 2024-04-02T09:17:16.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.619+0000 7f9e79600700 1 -- 172.21.15.89:0/3351193990 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7400b910 con 0x7f9e7c161590 2024-04-02T09:17:16.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.619+0000 7f9e79600700 1 -- 172.21.15.89:0/3351193990 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7400bcc0 con 0x7f9e7c161590 2024-04-02T09:17:16.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.620+0000 7f9e81a00700 1 -- 172.21.15.89:0/3351193990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c161590 msgr2=0x7f9e7c161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.620+0000 7f9e81a00700 1 --2- 172.21.15.89:0/3351193990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c161590 0x7f9e7c161a40 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c0d1080 tx=0x7f9e7400b330 comp rx=0 tx=0).stop 2024-04-02T09:17:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.620+0000 7f9e81a00700 1 -- 172.21.15.89:0/3351193990 shutdown_connections 2024-04-02T09:17:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.620+0000 7f9e81a00700 1 --2- 172.21.15.89:0/3351193990 >> v2:172.21.15.113:3300/0 conn(0x7f9e7c160f10 0x7f9e7c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.620+0000 7f9e81a00700 1 --2- 172.21.15.89:0/3351193990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c161590 0x7f9e7c161a40 unknown :-1 s=CLOSED pgs=3098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.620+0000 7f9e81a00700 1 -- 172.21.15.89:0/3351193990 >> 172.21.15.89:0/3351193990 conn(0x7f9e7c08aff0 msgr2=0x7f9e7c0d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.620+0000 7f9e81a00700 1 -- 172.21.15.89:0/3351193990 shutdown_connections 2024-04-02T09:17:16.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.620+0000 7f9e81a00700 1 -- 172.21.15.89:0/3351193990 wait complete. 2024-04-02T09:17:16.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.621+0000 7f9e81a00700 1 Processor -- start 2024-04-02T09:17:16.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.622+0000 7f9e81a00700 1 -- start start 2024-04-02T09:17:16.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.622+0000 7f9e81a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c0cf230 0x7f9e7c157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:16.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.622+0000 7f9e81a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e7c160f10 0x7f9e7c15a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:16.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.622+0000 7f9e81a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e7c161590 0x7f9e7c15af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:16.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.622+0000 7f9e81a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c158250 con 0x7f9e7c0cf230 2024-04-02T09:17:16.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.622+0000 7f9e81a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c15f360 con 0x7f9e7c161590 2024-04-02T09:17:16.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.622+0000 7f9e81a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c15f640 con 0x7f9e7c160f10 2024-04-02T09:17:16.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.623+0000 7f9e7aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c0cf230 0x7f9e7c157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:16.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.623+0000 7f9e7aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c0cf230 0x7f9e7c157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56916/0 (socket says 172.21.15.89:56916) 2024-04-02T09:17:16.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.623+0000 7f9e7aa00700 1 -- 172.21.15.89:0/564297950 learned_addr learned my addr 172.21.15.89:0/564297950 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:16.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.623+0000 7f9e7b400700 1 --2- 172.21.15.89:0/564297950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e7c161590 0x7f9e7c15af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:16.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.623+0000 7f9e7a000700 1 --2- 172.21.15.89:0/564297950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e7c160f10 0x7f9e7c15a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:16.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.623+0000 7f9e7aa00700 1 -- 172.21.15.89:0/564297950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e7c160f10 msgr2=0x7f9e7c15a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:16.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.623+0000 7f9e7aa00700 1 --2- 172.21.15.89:0/564297950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e7c160f10 0x7f9e7c15a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.623+0000 7f9e7aa00700 1 -- 172.21.15.89:0/564297950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e7c161590 msgr2=0x7f9e7c15af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:16.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.623+0000 7f9e7aa00700 1 --2- 172.21.15.89:0/564297950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e7c161590 0x7f9e7c15af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.623+0000 7f9e7aa00700 1 -- 172.21.15.89:0/564297950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e74009b30 con 0x7f9e7c0cf230 2024-04-02T09:17:16.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.623+0000 7f9e7aa00700 1 --2- 172.21.15.89:0/564297950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c0cf230 0x7f9e7c157d10 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c00e870 tx=0x7f9e6c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:16.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.624+0000 7f9e72a00700 1 -- 172.21.15.89:0/564297950 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c009590 con 0x7f9e7c0cf230 2024-04-02T09:17:16.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.624+0000 7f9e72a00700 1 -- 172.21.15.89:0/564297950 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c013680 con 0x7f9e7c0cf230 2024-04-02T09:17:16.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.624+0000 7f9e81a00700 1 -- 172.21.15.89:0/564297950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7c15f920 con 0x7f9e7c0cf230 2024-04-02T09:17:16.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.624+0000 7f9e72a00700 1 -- 172.21.15.89:0/564297950 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c0139f0 con 0x7f9e7c0cf230 2024-04-02T09:17:16.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.624+0000 7f9e81a00700 1 -- 172.21.15.89:0/564297950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7c15fee0 con 0x7f9e7c0cf230 2024-04-02T09:17:16.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.625+0000 7f9e81a00700 1 -- 172.21.15.89:0/564297950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7c15d0b0 con 0x7f9e7c0cf230 2024-04-02T09:17:16.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.626+0000 7f9e72a00700 1 -- 172.21.15.89:0/564297950 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9e6c013b50 con 0x7f9e7c0cf230 2024-04-02T09:17:16.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.626+0000 7f9e72a00700 1 --2- 172.21.15.89:0/564297950 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e5c06adb0 0x7f9e5c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:16.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.627+0000 7f9e72a00700 1 -- 172.21.15.89:0/564297950 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f9e6c089400 con 0x7f9e7c0cf230 2024-04-02T09:17:16.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.627+0000 7f9e7a000700 1 --2- 172.21.15.89:0/564297950 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e5c06adb0 0x7f9e5c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:16.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.627+0000 7f9e7a000700 1 --2- 172.21.15.89:0/564297950 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e5c06adb0 0x7f9e5c06d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c0c9f70 tx=0x7f9e68009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:16.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.631+0000 7f9e72a00700 1 -- 172.21.15.89:0/564297950 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e6c0543c0 con 0x7f9e7c0cf230 2024-04-02T09:17:16.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.833+0000 7f9e81a00700 1 -- 172.21.15.89:0/564297950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e7c067720 con 0x7f9e7c0cf230 2024-04-02T09:17:16.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.834+0000 7f9e72a00700 1 -- 172.21.15.89:0/564297950 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v284) v1 ==== 74+0+14262 (secure 0 0 0) 0x7f9e6c0579e0 con 0x7f9e7c0cf230 2024-04-02T09:17:16.833 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:17:16.833 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":284,"fsid":"ddf7474f-4630-4a1e-ada2-3982714d822b","created":"2024-04-02T07:35:27.940354+0000","modified":"2024-04-02T09:17:11.606247+0000","last_up_change":"2024-04-02T07:35:40.932499+0000","last_in_change":"2024-04-02T07:35:32.121142+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T07:35:41.237970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c3756e3-ac90-4364-90bf-287137eff24e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":43701},{"type":"v1","addr":"172.21.15.89:6801","nonce":43701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":43701},{"type":"v1","addr":"172.21.15.89:6803","nonce":43701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":43701},{"type":"v1","addr":"172.21.15.89:6807","nonce":43701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":43701},{"type":"v1","addr":"172.21.15.89:6805","nonce":43701}]},"public_addr":"172.21.15.89:6801/43701","cluster_addr":"172.21.15.89:6803/43701","heartbeat_back_addr":"172.21.15.89:6807/43701","heartbeat_front_addr":"172.21.15.89:6805/43701","state":["exists","up"]},{"osd":1,"uuid":"ecb9cf24-f3c6-4d9d-9fdb-d862e50b60fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":43703},{"type":"v1","addr":"172.21.15.89:6809","nonce":43703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":43703},{"type":"v1","addr":"172.21.15.89:6811","nonce":43703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":43703},{"type":"v1","addr":"172.21.15.89:6815","nonce":43703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":43703},{"type":"v1","addr":"172.21.15.89:6813","nonce":43703}]},"public_addr":"172.21.15.89:6809/43703","cluster_addr":"172.21.15.89:6811/43703","heartbeat_back_addr":"172.21.15.89:6815/43703","heartbeat_front_addr":"172.21.15.89:6813/43703","state":["exists","up"]},{"osd":2,"uuid":"5dcd8a58-660d-4cfd-9d05-6109d2d0c378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":43713},{"type":"v1","addr":"172.21.15.89:6825","nonce":43713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":43713},{"type":"v1","addr":"172.21.15.89:6827","nonce":43713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":43713},{"type":"v1","addr":"172.21.15.89:6831","nonce":43713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":43713},{"type":"v1","addr":"172.21.15.89:6829","nonce":43713}]},"public_addr":"172.21.15.89:6825/43713","cluster_addr":"172.21.15.89:6827/43713","heartbeat_back_addr":"172.21.15.89:6831/43713","heartbeat_front_addr":"172.21.15.89:6829/43713","state":["exists","up"]},{"osd":3,"uuid":"e8e4e352-db2d-4b96-9282-b2a4883c8f66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":43716},{"type":"v1","addr":"172.21.15.89:6817","nonce":43716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":43716},{"type":"v1","addr":"172.21.15.89:6819","nonce":43716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":43716},{"type":"v1","addr":"172.21.15.89:6823","nonce":43716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":43716},{"type":"v1","addr":"172.21.15.89:6821","nonce":43716}]},"public_addr":"172.21.15.89:6817/43716","cluster_addr":"172.21.15.89:6819/43716","heartbeat_back_addr":"172.21.15.89:6823/43716","heartbeat_front_addr":"172.21.15.89:6821/43716","state":["exists","up"]},{"osd":4,"uuid":"cb9001a3-5ee2-4c09-afcf-881626477a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":43940},{"type":"v1","addr":"172.21.15.113:6809","nonce":43940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":43940},{"type":"v1","addr":"172.21.15.113:6811","nonce":43940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":43940},{"type":"v1","addr":"172.21.15.113:6815","nonce":43940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":43940},{"type":"v1","addr":"172.21.15.113:6813","nonce":43940}]},"public_addr":"172.21.15.113:6809/43940","cluster_addr":"172.21.15.113:6811/43940","heartbeat_back_addr":"172.21.15.113:6815/43940","heartbeat_front_addr":"172.21.15.113:6813/43940","state":["exists","up"]},{"osd":5,"uuid":"9678135c-a55d-4a00-b68f-4605313c9756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":43939},{"type":"v1","addr":"172.21.15.113:6801","nonce":43939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":43939},{"type":"v1","addr":"172.21.15.113:6803","nonce":43939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":43939},{"type":"v1","addr":"172.21.15.113:6807","nonce":43939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":43939},{"type":"v1","addr":"172.21.15.113:6805","nonce":43939}]},"public_addr":"172.21.15.113:6801/43939","cluster_addr":"172.21.15.113:6803/43939","heartbeat_back_addr":"172.21.15.113:6807/43939","heartbeat_front_addr":"172.21.15.113:6805/43939","state":["exists","up"]},{"osd":6,"uuid":"6c67a32e-252d-4449-b039-35b67c16f38e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":43958},{"type":"v1","addr":"172.21.15.113:6825","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":43958},{"type":"v1","addr":"172.21.15.113:6827","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":43958},{"type":"v1","addr":"172.21.15.113:6831","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":43958},{"type":"v1","addr":"172.21.15.113:6829","nonce":43958}]},"public_addr":"172.21.15.113:6825/43958","cluster_addr":"172.21.15.113:6827/43958","heartbeat_back_addr":"172.21.15.113:6831/43958","heartbeat_front_addr":"172.21.15.113:6829/43958","state":["exists","up"]},{"osd":7,"uuid":"c4564c2a-1b9d-4ed2-a54b-19158f3dc757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":43941},{"type":"v1","addr":"172.21.15.113:6817","nonce":43941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":43941},{"type":"v1","addr":"172.21.15.113:6819","nonce":43941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":43941},{"type":"v1","addr":"172.21.15.113:6823","nonce":43941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":43941},{"type":"v1","addr":"172.21.15.113:6821","nonce":43941}]},"public_addr":"172.21.15.113:6817/43941","cluster_addr":"172.21.15.113:6819/43941","heartbeat_back_addr":"172.21.15.113:6823/43941","heartbeat_front_addr":"172.21.15.113:6821/43941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.331203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.387214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.403591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.484712+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.445745+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.389010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.487783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T07:35:37.459656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6836/3853118738":"2024-04-03T09:17:08.591874+0000","172.21.15.89:0/3185841611":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/1687104676":"2024-04-03T09:17:07.522481+0000","172.21.15.89:6837/3853118738":"2024-04-03T09:17:08.591874+0000","172.21.15.89:0/2207153077":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/171309686":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/3381360183":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/573568808":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/876295205":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/1137235236":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/2320969126":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/2687295125":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/693914703":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/1405191830":"2024-04-03T09:15:43.128314+0000","172.21.15.89:6833/108855":"2024-04-03T09:15:43.128314+0000","172.21.15.89:6832/108855":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/701778740":"2024-04-03T09:15:43.128314+0000","172.21.15.89:0/445136997":"2024-04-03T09:17:07.522481+0000","172.21.15.89:0/799882283":"2024-04-03T09:15:43.128314+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":15,"length":2},{"begin":21,"length":2},{"begin":26,"length":2},{"begin":33,"length":1},{"begin":37,"length":2},{"begin":44,"length":1},{"begin":48,"length":2},{"begin":55,"length":1},{"begin":59,"length":2},{"begin":66,"length":1},{"begin":70,"length":3},{"begin":77,"length":1},{"begin":81,"length":3},{"begin":88,"length":1},{"begin":93,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":109,"length":2},{"begin":188,"length":34}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T09:17:16.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.836+0000 7f9e81a00700 1 -- 172.21.15.89:0/564297950 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e5c06adb0 msgr2=0x7f9e5c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:16.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.836+0000 7f9e81a00700 1 --2- 172.21.15.89:0/564297950 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e5c06adb0 0x7f9e5c06d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c0c9f70 tx=0x7f9e68009040 comp rx=0 tx=0).stop 2024-04-02T09:17:16.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.836+0000 7f9e81a00700 1 -- 172.21.15.89:0/564297950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c0cf230 msgr2=0x7f9e7c157d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:16.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.836+0000 7f9e81a00700 1 --2- 172.21.15.89:0/564297950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c0cf230 0x7f9e7c157d10 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c00e870 tx=0x7f9e6c00eb80 comp rx=0 tx=0).stop 2024-04-02T09:17:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.837+0000 7f9e81a00700 1 -- 172.21.15.89:0/564297950 shutdown_connections 2024-04-02T09:17:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.837+0000 7f9e81a00700 1 --2- 172.21.15.89:0/564297950 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f9e5c06adb0 0x7f9e5c06d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.837+0000 7f9e81a00700 1 --2- 172.21.15.89:0/564297950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e7c161590 0x7f9e7c15af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.837+0000 7f9e81a00700 1 --2- 172.21.15.89:0/564297950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c0cf230 0x7f9e7c157d10 unknown :-1 s=CLOSED pgs=3099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.837+0000 7f9e81a00700 1 --2- 172.21.15.89:0/564297950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e7c160f10 0x7f9e7c15a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.837+0000 7f9e81a00700 1 -- 172.21.15.89:0/564297950 >> 172.21.15.89:0/564297950 conn(0x7f9e7c08aff0 msgr2=0x7f9e7c0cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:16.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.837+0000 7f9e81a00700 1 -- 172.21.15.89:0/564297950 shutdown_connections 2024-04-02T09:17:16.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:16.837+0000 7f9e81a00700 1 -- 172.21.15.89:0/564297950 wait complete. 2024-04-02T09:17:17.854 INFO:tasks.ceph:Scrubbing osd.0 2024-04-02T09:17:17.854 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-02T09:17:18.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.062+0000 7fb24c000700 1 Processor -- start 2024-04-02T09:17:18.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.063+0000 7fb24c000700 1 -- start start 2024-04-02T09:17:18.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.064+0000 7fb24c000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2441580c0 0x7fb24406e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.064+0000 7fb24c000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb24415b330 0x7fb2440b2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.064+0000 7fb24c000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb244070980 con 0x7fb24406e7f0 2024-04-02T09:17:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.064+0000 7fb24c000700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb244071570 con 0x7fb24415b330 2024-04-02T09:17:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.064+0000 7fb24c000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb244071990 con 0x7fb2441580c0 2024-04-02T09:17:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.064+0000 7fb24a000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb24415b330 0x7fb2440b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.064+0000 7fb248c00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb24406e7f0 0x7fb24415ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59590/0 (socket says 172.21.15.89:59590) 2024-04-02T09:17:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.064+0000 7fb248c00700 1 -- 172.21.15.89:0/2063203475 learned_addr learned my addr 172.21.15.89:0/2063203475 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.064+0000 7fb249600700 1 --2- 172.21.15.89:0/2063203475 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2441580c0 0x7fb24406e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.065+0000 7fb24a000700 1 -- 172.21.15.89:0/2063203475 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2441580c0 msgr2=0x7fb24406e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.065+0000 7fb24a000700 1 --2- 172.21.15.89:0/2063203475 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2441580c0 0x7fb24406e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.065+0000 7fb24a000700 1 -- 172.21.15.89:0/2063203475 >> v1:172.21.15.89:6789/0 conn(0x7fb24406e7f0 legacy=0x7fb24415ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.065+0000 7fb24a000700 1 -- 172.21.15.89:0/2063203475 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24415b760 con 0x7fb24415b330 2024-04-02T09:17:18.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.065+0000 7fb24a000700 1 --2- 172.21.15.89:0/2063203475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb24415b330 0x7fb2440b2180 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009d90 tx=0x7fb23c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55485b192d89bff3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:18.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.065+0000 7fb243e00700 1 -- 172.21.15.89:0/2063203475 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c00e070 con 0x7fb24415b330 2024-04-02T09:17:18.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.065+0000 7fb243e00700 1 -- 172.21.15.89:0/2063203475 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23c00bbd0 con 0x7fb24415b330 2024-04-02T09:17:18.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.065+0000 7fb243e00700 1 -- 172.21.15.89:0/2063203475 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c012430 con 0x7fb24415b330 2024-04-02T09:17:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.066+0000 7fb24c000700 1 -- 172.21.15.89:0/2063203475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb24415b330 msgr2=0x7fb2440b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.066+0000 7fb24c000700 1 --2- 172.21.15.89:0/2063203475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb24415b330 0x7fb2440b2180 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009d90 tx=0x7fb23c00b330 comp rx=0 tx=0).stop 2024-04-02T09:17:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.066+0000 7fb24c000700 1 -- 172.21.15.89:0/2063203475 shutdown_connections 2024-04-02T09:17:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.066+0000 7fb24c000700 1 --2- 172.21.15.89:0/2063203475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb24415b330 0x7fb2440b2180 unknown :-1 s=CLOSED pgs=3101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.066+0000 7fb24c000700 1 --2- 172.21.15.89:0/2063203475 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2441580c0 0x7fb24406e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.066+0000 7fb24c000700 1 -- 172.21.15.89:0/2063203475 >> 172.21.15.89:0/2063203475 conn(0x7fb2440cf210 msgr2=0x7fb2440cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.066+0000 7fb24c000700 1 -- 172.21.15.89:0/2063203475 shutdown_connections 2024-04-02T09:17:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.066+0000 7fb24c000700 1 -- 172.21.15.89:0/2063203475 wait complete. 2024-04-02T09:17:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.068+0000 7fb24c000700 1 Processor -- start 2024-04-02T09:17:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.068+0000 7fb24c000700 1 -- start start 2024-04-02T09:17:18.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb24c000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb24406e7f0 0x7fb244158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb24c000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2441580c0 0x7fb2441538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb24c000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb24415b330 0x7fb244153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb24c000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb244099820 con 0x7fb2441580c0 2024-04-02T09:17:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb24c000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb244154d00 con 0x7fb24406e7f0 2024-04-02T09:17:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb24c000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb244099460 con 0x7fb24415b330 2024-04-02T09:17:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb248c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2441580c0 0x7fb2441538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb248c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2441580c0 0x7fb2441538a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56926/0 (socket says 172.21.15.89:56926) 2024-04-02T09:17:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb248c00700 1 -- 172.21.15.89:0/2511758385 learned_addr learned my addr 172.21.15.89:0/2511758385 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:18.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb249600700 1 --2- 172.21.15.89:0/2511758385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb24406e7f0 0x7fb244158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb24a000700 1 --2- 172.21.15.89:0/2511758385 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb24415b330 0x7fb244153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb248c00700 1 -- 172.21.15.89:0/2511758385 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb24415b330 msgr2=0x7fb244153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb248c00700 1 --2- 172.21.15.89:0/2511758385 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb24415b330 0x7fb244153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb248c00700 1 -- 172.21.15.89:0/2511758385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb24406e7f0 msgr2=0x7fb244158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb248c00700 1 --2- 172.21.15.89:0/2511758385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb24406e7f0 0x7fb244158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.069+0000 7fb248c00700 1 -- 172.21.15.89:0/2511758385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c009a70 con 0x7fb2441580c0 2024-04-02T09:17:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.070+0000 7fb248c00700 1 --2- 172.21.15.89:0/2511758385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2441580c0 0x7fb2441538a0 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7fb23000c550 tx=0x7fb23000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.070+0000 7fb241600700 1 -- 172.21.15.89:0/2511758385 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb230005d70 con 0x7fb2441580c0 2024-04-02T09:17:18.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.070+0000 7fb241600700 1 -- 172.21.15.89:0/2511758385 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2300070d0 con 0x7fb2441580c0 2024-04-02T09:17:18.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.070+0000 7fb241600700 1 -- 172.21.15.89:0/2511758385 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb230007440 con 0x7fb2441580c0 2024-04-02T09:17:18.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.070+0000 7fb24c000700 1 -- 172.21.15.89:0/2511758385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb244156cb0 con 0x7fb2441580c0 2024-04-02T09:17:18.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.070+0000 7fb24c000700 1 -- 172.21.15.89:0/2511758385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2441591d0 con 0x7fb2441580c0 2024-04-02T09:17:18.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.071+0000 7fb24c000700 1 -- 172.21.15.89:0/2511758385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb2440992e0 con 0x7fb2441580c0 2024-04-02T09:17:18.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.072+0000 7fb241600700 1 -- 172.21.15.89:0/2511758385 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb2300075a0 con 0x7fb2441580c0 2024-04-02T09:17:18.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.073+0000 7fb241600700 1 --2- 172.21.15.89:0/2511758385 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb22406adb0 0x7fb22406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.073+0000 7fb241600700 1 -- 172.21.15.89:0/2511758385 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7fb230088900 con 0x7fb2441580c0 2024-04-02T09:17:18.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.073+0000 7fb249600700 1 --2- 172.21.15.89:0/2511758385 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb22406adb0 0x7fb22406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.073+0000 7fb241600700 1 --2- 172.21.15.89:0/2511758385 >> [v2:172.21.15.89:6800/43701,v1:172.21.15.89:6801/43701] conn(0x7fb224071680 0x7fb224073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.073+0000 7fb241600700 1 -- 172.21.15.89:0/2511758385 --> [v2:172.21.15.89:6800/43701,v1:172.21.15.89:6801/43701] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2240740f0 con 0x7fb224071680 2024-04-02T09:17:18.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.073+0000 7fb241600700 1 -- 172.21.15.89:0/2511758385 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=284) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb23008d2a0 con 0x7fb2441580c0 2024-04-02T09:17:18.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.073+0000 7fb24a000700 1 --2- 172.21.15.89:0/2511758385 >> [v2:172.21.15.89:6800/43701,v1:172.21.15.89:6801/43701] conn(0x7fb224071680 0x7fb224073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.074+0000 7fb249600700 1 --2- 172.21.15.89:0/2511758385 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb22406adb0 0x7fb22406d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb238006b20 tx=0x7fb238008040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:18.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.074+0000 7fb24a000700 1 --2- 172.21.15.89:0/2511758385 >> [v2:172.21.15.89:6800/43701,v1:172.21.15.89:6801/43701] conn(0x7fb224071680 0x7fb224073ac0 crc :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:18.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.075+0000 7fb241600700 1 -- 172.21.15.89:0/2511758385 <== osd.0 v2:172.21.15.89:6800/43701 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb2240740f0 con 0x7fb224071680 2024-04-02T09:17:18.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.089+0000 7fb24c000700 1 -- 172.21.15.89:0/2511758385 --> [v2:172.21.15.89:6800/43701,v1:172.21.15.89:6801/43701] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb244071190 con 0x7fb224071680 2024-04-02T09:17:18.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.090+0000 7fb241600700 1 -- 172.21.15.89:0/2511758385 <== osd.0 v2:172.21.15.89:6800/43701 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb23c0125e0 con 0x7fb224071680 2024-04-02T09:17:18.088 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:17:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T09:17:18.089 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:17:18.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.090+0000 7fb24c000700 1 -- 172.21.15.89:0/2511758385 >> [v2:172.21.15.89:6800/43701,v1:172.21.15.89:6801/43701] conn(0x7fb224071680 msgr2=0x7fb224073ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.090+0000 7fb24c000700 1 --2- 172.21.15.89:0/2511758385 >> [v2:172.21.15.89:6800/43701,v1:172.21.15.89:6801/43701] conn(0x7fb224071680 0x7fb224073ac0 crc :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.090+0000 7fb24c000700 1 -- 172.21.15.89:0/2511758385 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb22406adb0 msgr2=0x7fb22406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.090+0000 7fb24c000700 1 --2- 172.21.15.89:0/2511758385 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb22406adb0 0x7fb22406d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb238006b20 tx=0x7fb238008040 comp rx=0 tx=0).stop 2024-04-02T09:17:18.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.090+0000 7fb24c000700 1 -- 172.21.15.89:0/2511758385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2441580c0 msgr2=0x7fb2441538a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.090+0000 7fb24c000700 1 --2- 172.21.15.89:0/2511758385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2441580c0 0x7fb2441538a0 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7fb23000c550 tx=0x7fb23000c860 comp rx=0 tx=0).stop 2024-04-02T09:17:18.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.090+0000 7fb24a000700 1 -- 172.21.15.89:0/2511758385 reap_dead start 2024-04-02T09:17:18.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.090+0000 7fb24c000700 1 -- 172.21.15.89:0/2511758385 shutdown_connections 2024-04-02T09:17:18.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.090+0000 7fb24c000700 1 -- 172.21.15.89:0/2511758385 >> 172.21.15.89:0/2511758385 conn(0x7fb2440cf210 msgr2=0x7fb24415d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:18.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.090+0000 7fb24c000700 1 -- 172.21.15.89:0/2511758385 shutdown_connections 2024-04-02T09:17:18.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.090+0000 7fb24c000700 1 -- 172.21.15.89:0/2511758385 wait complete. 2024-04-02T09:17:18.106 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-02T09:17:18.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.300+0000 7f74bf800700 1 Processor -- start 2024-04-02T09:17:18.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.300+0000 7f74bf800700 1 -- start start 2024-04-02T09:17:18.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.301+0000 7f74bf800700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f74b815d9d0 0x7f74b80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.301+0000 7f74bf800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f74b80d2540 0x7f74b80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.301+0000 7f74bf800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b8161420 con 0x7f74b80d16a0 2024-04-02T09:17:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.301+0000 7f74bf800700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b815edc0 con 0x7f74b815d9d0 2024-04-02T09:17:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.301+0000 7f74bf800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b8161700 con 0x7f74b80d2540 2024-04-02T09:17:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.301+0000 7f74bce00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f74b815d9d0 0x7f74b80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.301+0000 7f74bce00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f74b815d9d0 0x7f74b80d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56928/0 (socket says 172.21.15.89:56928) 2024-04-02T09:17:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.301+0000 7f74bce00700 1 -- 172.21.15.89:0/3847359535 learned_addr learned my addr 172.21.15.89:0/3847359535 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.301+0000 7f74bd800700 1 --2- 172.21.15.89:0/3847359535 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f74b80d2540 0x7f74b80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.302+0000 7f74bce00700 1 -- 172.21.15.89:0/3847359535 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f74b80d2540 msgr2=0x7f74b80ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.302+0000 7f74bce00700 1 --2- 172.21.15.89:0/3847359535 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f74b80d2540 0x7f74b80ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.302+0000 7f74bce00700 1 -- 172.21.15.89:0/3847359535 >> v1:172.21.15.113:6789/0 conn(0x7f74b80d16a0 legacy=0x7f74b80d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.302+0000 7f74bce00700 1 -- 172.21.15.89:0/3847359535 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b81619e0 con 0x7f74b815d9d0 2024-04-02T09:17:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.302+0000 7f74bce00700 1 --2- 172.21.15.89:0/3847359535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b815d9d0 0x7f74b80d3c30 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0098f0 tx=0x7f74ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e269da799d80226 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.302+0000 7f74b7400700 1 -- 172.21.15.89:0/3847359535 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac011830 con 0x7f74b815d9d0 2024-04-02T09:17:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.302+0000 7f74b7400700 1 -- 172.21.15.89:0/3847359535 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac011990 con 0x7f74b815d9d0 2024-04-02T09:17:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.302+0000 7f74b7400700 1 -- 172.21.15.89:0/3847359535 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac011d20 con 0x7f74b815d9d0 2024-04-02T09:17:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.303+0000 7f74bf800700 1 -- 172.21.15.89:0/3847359535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b815d9d0 msgr2=0x7f74b80d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.303+0000 7f74bf800700 1 --2- 172.21.15.89:0/3847359535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b815d9d0 0x7f74b80d3c30 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0098f0 tx=0x7f74ac009c00 comp rx=0 tx=0).stop 2024-04-02T09:17:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.303+0000 7f74bf800700 1 -- 172.21.15.89:0/3847359535 shutdown_connections 2024-04-02T09:17:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.303+0000 7f74bf800700 1 --2- 172.21.15.89:0/3847359535 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f74b80d2540 0x7f74b80ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.303+0000 7f74bf800700 1 --2- 172.21.15.89:0/3847359535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b815d9d0 0x7f74b80d3c30 unknown :-1 s=CLOSED pgs=3103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.303+0000 7f74bf800700 1 -- 172.21.15.89:0/3847359535 >> 172.21.15.89:0/3847359535 conn(0x7f74b808aff0 msgr2=0x7f74b80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.303+0000 7f74bf800700 1 -- 172.21.15.89:0/3847359535 shutdown_connections 2024-04-02T09:17:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.303+0000 7f74bf800700 1 -- 172.21.15.89:0/3847359535 wait complete. 2024-04-02T09:17:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.305+0000 7f74bf800700 1 Processor -- start 2024-04-02T09:17:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.305+0000 7f74bf800700 1 -- start start 2024-04-02T09:17:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bf800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b80d16a0 0x7f74b81e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bf800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f74b80d2540 0x7f74b814f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bf800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74b815d9d0 0x7f74b80b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bf800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b80cb5a0 con 0x7f74b80d16a0 2024-04-02T09:17:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bf800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b81e82e0 con 0x7f74b815d9d0 2024-04-02T09:17:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bf800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b81e85c0 con 0x7f74b80d2540 2024-04-02T09:17:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bce00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b80d16a0 0x7f74b81e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bce00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b80d16a0 0x7f74b81e7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56938/0 (socket says 172.21.15.89:56938) 2024-04-02T09:17:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bce00700 1 -- 172.21.15.89:0/2725247333 learned_addr learned my addr 172.21.15.89:0/2725247333 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74b7e00700 1 --2- 172.21.15.89:0/2725247333 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f74b80d2540 0x7f74b814f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bd800700 1 --2- 172.21.15.89:0/2725247333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74b815d9d0 0x7f74b80b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bce00700 1 -- 172.21.15.89:0/2725247333 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f74b80d2540 msgr2=0x7f74b814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bce00700 1 --2- 172.21.15.89:0/2725247333 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f74b80d2540 0x7f74b814f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bce00700 1 -- 172.21.15.89:0/2725247333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74b815d9d0 msgr2=0x7f74b80b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bce00700 1 --2- 172.21.15.89:0/2725247333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74b815d9d0 0x7f74b80b7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.306+0000 7f74bce00700 1 -- 172.21.15.89:0/2725247333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac009550 con 0x7f74b80d16a0 2024-04-02T09:17:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.307+0000 7f74bce00700 1 --2- 172.21.15.89:0/2725247333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b80d16a0 0x7f74b81e7c90 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009c40 tx=0x7f74ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.307+0000 7f74b4c00700 1 -- 172.21.15.89:0/2725247333 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac0036a0 con 0x7f74b80d16a0 2024-04-02T09:17:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.307+0000 7f74b4c00700 1 -- 172.21.15.89:0/2725247333 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac01e070 con 0x7f74b80d16a0 2024-04-02T09:17:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.307+0000 7f74b4c00700 1 -- 172.21.15.89:0/2725247333 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac003a60 con 0x7f74b80d16a0 2024-04-02T09:17:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.307+0000 7f74bf800700 1 -- 172.21.15.89:0/2725247333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b81e88a0 con 0x7f74b80d16a0 2024-04-02T09:17:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.307+0000 7f74bf800700 1 -- 172.21.15.89:0/2725247333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b814fd90 con 0x7f74b80d16a0 2024-04-02T09:17:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.308+0000 7f74bf800700 1 -- 172.21.15.89:0/2725247333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b80d1ca0 con 0x7f74b80d16a0 2024-04-02T09:17:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.309+0000 7f74b4c00700 1 -- 172.21.15.89:0/2725247333 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f74ac003bc0 con 0x7f74b80d16a0 2024-04-02T09:17:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.310+0000 7f74b4c00700 1 --2- 172.21.15.89:0/2725247333 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f749806adb0 0x7f749806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.310+0000 7f74b4c00700 1 -- 172.21.15.89:0/2725247333 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f74ac08a550 con 0x7f74b80d16a0 2024-04-02T09:17:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.310+0000 7f74b7e00700 1 --2- 172.21.15.89:0/2725247333 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f749806adb0 0x7f749806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.311+0000 7f74b7e00700 1 --2- 172.21.15.89:0/2725247333 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f749806adb0 0x7f749806d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f74b815c930 tx=0x7f74a4009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.314+0000 7f74b4c00700 1 -- 172.21.15.89:0/2725247333 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f74ac055510 con 0x7f74b80d16a0 2024-04-02T09:17:18.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.514+0000 7f74bf800700 1 -- 172.21.15.89:0/2725247333 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f74b8161fa0 con 0x7f749806adb0 2024-04-02T09:17:18.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.515+0000 7f74b4c00700 1 -- 172.21.15.89:0/2725247333 <== mgr.13575 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f74a40050d0 con 0x7f749806adb0 2024-04-02T09:17:18.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.517+0000 7f74bf800700 1 -- 172.21.15.89:0/2725247333 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f749806adb0 msgr2=0x7f749806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.517+0000 7f74bf800700 1 --2- 172.21.15.89:0/2725247333 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f749806adb0 0x7f749806d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f74b815c930 tx=0x7f74a4009040 comp rx=0 tx=0).stop 2024-04-02T09:17:18.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.517+0000 7f74bf800700 1 -- 172.21.15.89:0/2725247333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b80d16a0 msgr2=0x7f74b81e7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.517+0000 7f74bf800700 1 --2- 172.21.15.89:0/2725247333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b80d16a0 0x7f74b81e7c90 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009c40 tx=0x7f74ac004750 comp rx=0 tx=0).stop 2024-04-02T09:17:18.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.517+0000 7f74bf800700 1 -- 172.21.15.89:0/2725247333 shutdown_connections 2024-04-02T09:17:18.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.517+0000 7f74bf800700 1 --2- 172.21.15.89:0/2725247333 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f749806adb0 0x7f749806d270 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.517+0000 7f74bf800700 1 --2- 172.21.15.89:0/2725247333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74b815d9d0 0x7f74b80b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.517+0000 7f74bf800700 1 --2- 172.21.15.89:0/2725247333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74b80d16a0 0x7f74b81e7c90 unknown :-1 s=CLOSED pgs=3104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.517+0000 7f74bf800700 1 --2- 172.21.15.89:0/2725247333 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f74b80d2540 0x7f74b814f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.517+0000 7f74bf800700 1 -- 172.21.15.89:0/2725247333 >> 172.21.15.89:0/2725247333 conn(0x7f74b808aff0 msgr2=0x7f74b80b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:18.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.517+0000 7f74bf800700 1 -- 172.21.15.89:0/2725247333 shutdown_connections 2024-04-02T09:17:18.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.517+0000 7f74bf800700 1 -- 172.21.15.89:0/2725247333 wait complete. 2024-04-02T09:17:18.517 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 0 to deep-scrub 2024-04-02T09:17:18.533 INFO:tasks.ceph:Scrubbing osd.1 2024-04-02T09:17:18.534 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-02T09:17:18.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.732+0000 7fd60c000700 1 Processor -- start 2024-04-02T09:17:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.732+0000 7fd60c000700 1 -- start start 2024-04-02T09:17:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.733+0000 7fd60c000700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd60415d930 0x7fd6040d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.733+0000 7fd60c000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd604162030 0x7fd6040d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.733+0000 7fd60c000700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60415dd10 con 0x7fd6040d20d0 2024-04-02T09:17:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.733+0000 7fd60c000700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd60415ec40 con 0x7fd60415d930 2024-04-02T09:17:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.733+0000 7fd60c000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6040d2560 con 0x7fd604162030 2024-04-02T09:17:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.733+0000 7fd60a000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd6040d20d0 0x7fd6040cfc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59594/0 (socket says 172.21.15.89:59594) 2024-04-02T09:17:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.733+0000 7fd60a000700 1 -- 172.21.15.89:0/2187385893 learned_addr learned my addr 172.21.15.89:0/2187385893 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd609600700 1 --2- 172.21.15.89:0/2187385893 >> v2:172.21.15.113:3300/0 conn(0x7fd60415d930 0x7fd6040d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd608c00700 1 --2- 172.21.15.89:0/2187385893 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd604162030 0x7fd6040d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 -- 172.21.15.89:0/2187385893 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206772537 0 0) 0x7fd60415dd10 con 0x7fd6040d20d0 2024-04-02T09:17:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 -- 172.21.15.89:0/2187385893 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5ec0035b0 con 0x7fd6040d20d0 2024-04-02T09:17:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 -- 172.21.15.89:0/2187385893 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 907951710 0 0) 0x7fd5ec0035b0 con 0x7fd6040d20d0 2024-04-02T09:17:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 -- 172.21.15.89:0/2187385893 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd604162030 msgr2=0x7fd6040d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 --2- 172.21.15.89:0/2187385893 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd604162030 0x7fd6040d1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 -- 172.21.15.89:0/2187385893 >> v2:172.21.15.113:3300/0 conn(0x7fd60415d930 msgr2=0x7fd6040d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 --2- 172.21.15.89:0/2187385893 >> v2:172.21.15.113:3300/0 conn(0x7fd60415d930 0x7fd6040d3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 -- 172.21.15.89:0/2187385893 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60415dfc0 con 0x7fd6040d20d0 2024-04-02T09:17:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 -- 172.21.15.89:0/2187385893 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7fd5fc0019f0 con 0x7fd6040d20d0 2024-04-02T09:17:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 -- 172.21.15.89:0/2187385893 >> v1:172.21.15.89:6789/0 conn(0x7fd6040d20d0 legacy=0x7fd6040cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 --2- 172.21.15.89:0/2187385893 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd604162030 0x7fd5ec003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 --2- 172.21.15.89:0/2187385893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5ec003f80 0x7fd5ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 --2- 172.21.15.89:0/2187385893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5ec004d90 0x7fd5ec005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 -- 172.21.15.89:0/2187385893 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60415ec40 con 0x7fd5ec004d90 2024-04-02T09:17:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 -- 172.21.15.89:0/2187385893 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6040d2560 con 0x7fd5ec003f80 2024-04-02T09:17:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.734+0000 7fd603e00700 1 -- 172.21.15.89:0/2187385893 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec00ba10 con 0x7fd604162030 2024-04-02T09:17:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.735+0000 7fd60a000700 1 --2- 172.21.15.89:0/2187385893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5ec004d90 0x7fd5ec005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.735+0000 7fd608c00700 1 --2- 172.21.15.89:0/2187385893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5ec003f80 0x7fd5ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.735+0000 7fd609600700 1 --2- 172.21.15.89:0/2187385893 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd604162030 0x7fd5ec003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.735+0000 7fd60a000700 1 -- 172.21.15.89:0/2187385893 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd604162030 msgr2=0x7fd5ec003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.735+0000 7fd60a000700 1 --2- 172.21.15.89:0/2187385893 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd604162030 0x7fd5ec003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.735+0000 7fd60a000700 1 -- 172.21.15.89:0/2187385893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5ec003f80 msgr2=0x7fd5ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.735+0000 7fd60a000700 1 --2- 172.21.15.89:0/2187385893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5ec003f80 0x7fd5ec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.735+0000 7fd60a000700 1 -- 172.21.15.89:0/2187385893 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec0035b0 con 0x7fd5ec004d90 2024-04-02T09:17:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.735+0000 7fd60a000700 1 --2- 172.21.15.89:0/2187385893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5ec004d90 0x7fd5ec005210 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00a860 tx=0x7fd5ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11171cdcc473c211 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.735+0000 7fd603e00700 1 -- 172.21.15.89:0/2187385893 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc012600 con 0x7fd5ec004d90 2024-04-02T09:17:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.735+0000 7fd603e00700 1 -- 172.21.15.89:0/2187385893 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5fc012780 con 0x7fd5ec004d90 2024-04-02T09:17:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.735+0000 7fd603e00700 1 -- 172.21.15.89:0/2187385893 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc012b30 con 0x7fd5ec004d90 2024-04-02T09:17:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.736+0000 7fd60c000700 1 -- 172.21.15.89:0/2187385893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5ec004d90 msgr2=0x7fd5ec005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.736+0000 7fd60c000700 1 --2- 172.21.15.89:0/2187385893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5ec004d90 0x7fd5ec005210 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00a860 tx=0x7fd5ec005880 comp rx=0 tx=0).stop 2024-04-02T09:17:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.736+0000 7fd60a000700 1 -- 172.21.15.89:0/2187385893 reap_dead start 2024-04-02T09:17:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.736+0000 7fd60c000700 1 -- 172.21.15.89:0/2187385893 shutdown_connections 2024-04-02T09:17:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.736+0000 7fd60c000700 1 -- 172.21.15.89:0/2187385893 >> 172.21.15.89:0/2187385893 conn(0x7fd60408aff0 msgr2=0x7fd6040d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.736+0000 7fd60c000700 1 -- 172.21.15.89:0/2187385893 shutdown_connections 2024-04-02T09:17:18.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.736+0000 7fd60c000700 1 -- 172.21.15.89:0/2187385893 wait complete. 2024-04-02T09:17:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.738+0000 7fd60c000700 1 Processor -- start 2024-04-02T09:17:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.739+0000 7fd60c000700 1 -- start start 2024-04-02T09:17:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.740+0000 7fd60c000700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5ec004d90 0x7fd6040c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.740+0000 7fd60c000700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6041614b0 0x7fd6040b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.740+0000 7fd60c000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd604143500 0x7fd604143980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.740+0000 7fd60c000700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd604143ec0 con 0x7fd604143500 2024-04-02T09:17:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.740+0000 7fd60c000700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd604141530 con 0x7fd5ec004d90 2024-04-02T09:17:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.740+0000 7fd60c000700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd604161930 con 0x7fd6041614b0 2024-04-02T09:17:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.740+0000 7fd60a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd604143500 0x7fd604143980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.740+0000 7fd60a000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd604143500 0x7fd604143980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56954/0 (socket says 172.21.15.89:56954) 2024-04-02T09:17:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.740+0000 7fd60a000700 1 -- 172.21.15.89:0/1957438967 learned_addr learned my addr 172.21.15.89:0/1957438967 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.740+0000 7fd608c00700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6041614b0 0x7fd6040b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.740+0000 7fd609600700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5ec004d90 0x7fd6040c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.741+0000 7fd60a000700 1 -- 172.21.15.89:0/1957438967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6041614b0 msgr2=0x7fd6040b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.741+0000 7fd60a000700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6041614b0 0x7fd6040b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.741+0000 7fd60a000700 1 -- 172.21.15.89:0/1957438967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5ec004d90 msgr2=0x7fd6040c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.741+0000 7fd60a000700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5ec004d90 0x7fd6040c8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.741+0000 7fd60a000700 1 -- 172.21.15.89:0/1957438967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6040b2580 con 0x7fd604143500 2024-04-02T09:17:18.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.741+0000 7fd608c00700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6041614b0 0x7fd6040b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:17:18.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.741+0000 7fd60a000700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd604143500 0x7fd604143980 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc012000 tx=0x7fd5fc001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:18.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.741+0000 7fd601600700 1 -- 172.21.15.89:0/1957438967 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc01e600 con 0x7fd604143500 2024-04-02T09:17:18.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.741+0000 7fd601600700 1 -- 172.21.15.89:0/1957438967 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5fc01e780 con 0x7fd604143500 2024-04-02T09:17:18.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.741+0000 7fd601600700 1 -- 172.21.15.89:0/1957438967 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc01eb30 con 0x7fd604143500 2024-04-02T09:17:18.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.741+0000 7fd60c000700 1 -- 172.21.15.89:0/1957438967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6040c9120 con 0x7fd604143500 2024-04-02T09:17:18.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.741+0000 7fd60c000700 1 -- 172.21.15.89:0/1957438967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6040c96e0 con 0x7fd604143500 2024-04-02T09:17:18.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.742+0000 7fd60c000700 1 -- 172.21.15.89:0/1957438967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd60415e860 con 0x7fd604143500 2024-04-02T09:17:18.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.743+0000 7fd601600700 1 -- 172.21.15.89:0/1957438967 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd5fc019030 con 0x7fd604143500 2024-04-02T09:17:18.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.744+0000 7fd601600700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd5e406adb0 0x7fd5e406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.744+0000 7fd601600700 1 -- 172.21.15.89:0/1957438967 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7fd5fc08b530 con 0x7fd604143500 2024-04-02T09:17:18.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.744+0000 7fd609600700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd5e406adb0 0x7fd5e406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.744+0000 7fd601600700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.89:6808/43703,v1:172.21.15.89:6809/43703] conn(0x7fd5e4071680 0x7fd5e4073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.744+0000 7fd601600700 1 -- 172.21.15.89:0/1957438967 --> [v2:172.21.15.89:6808/43703,v1:172.21.15.89:6809/43703] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5e40740f0 con 0x7fd5e4071680 2024-04-02T09:17:18.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.744+0000 7fd601600700 1 -- 172.21.15.89:0/1957438967 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=284) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5fc08db10 con 0x7fd604143500 2024-04-02T09:17:18.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.744+0000 7fd608c00700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.89:6808/43703,v1:172.21.15.89:6809/43703] conn(0x7fd5e4071680 0x7fd5e4073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.745+0000 7fd609600700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd5e406adb0 0x7fd5e406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009c20 tx=0x7fd5f8008040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:18.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.745+0000 7fd608c00700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.89:6808/43703,v1:172.21.15.89:6809/43703] conn(0x7fd5e4071680 0x7fd5e4073ac0 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:18.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.746+0000 7fd601600700 1 -- 172.21.15.89:0/1957438967 <== osd.1 v2:172.21.15.89:6808/43703 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd5e40740f0 con 0x7fd5e4071680 2024-04-02T09:17:18.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.761+0000 7fd60c000700 1 -- 172.21.15.89:0/1957438967 --> [v2:172.21.15.89:6808/43703,v1:172.21.15.89:6809/43703] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd60415e580 con 0x7fd5e4071680 2024-04-02T09:17:18.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.761+0000 7fd601600700 1 -- 172.21.15.89:0/1957438967 <== osd.1 v2:172.21.15.89:6808/43703 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd5f0009c70 con 0x7fd5e4071680 2024-04-02T09:17:18.759 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:17:18.760 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T09:17:18.760 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:17:18.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.761+0000 7fd60c000700 1 -- 172.21.15.89:0/1957438967 >> [v2:172.21.15.89:6808/43703,v1:172.21.15.89:6809/43703] conn(0x7fd5e4071680 msgr2=0x7fd5e4073ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.761+0000 7fd60c000700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.89:6808/43703,v1:172.21.15.89:6809/43703] conn(0x7fd5e4071680 0x7fd5e4073ac0 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.761+0000 7fd60c000700 1 -- 172.21.15.89:0/1957438967 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd5e406adb0 msgr2=0x7fd5e406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.761+0000 7fd60c000700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd5e406adb0 0x7fd5e406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009c20 tx=0x7fd5f8008040 comp rx=0 tx=0).stop 2024-04-02T09:17:18.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.761+0000 7fd60c000700 1 -- 172.21.15.89:0/1957438967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd604143500 msgr2=0x7fd604143980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.761+0000 7fd60c000700 1 --2- 172.21.15.89:0/1957438967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd604143500 0x7fd604143980 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc012000 tx=0x7fd5fc001d80 comp rx=0 tx=0).stop 2024-04-02T09:17:18.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.761+0000 7fd60a000700 1 -- 172.21.15.89:0/1957438967 reap_dead start 2024-04-02T09:17:18.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.761+0000 7fd60c000700 1 -- 172.21.15.89:0/1957438967 shutdown_connections 2024-04-02T09:17:18.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.761+0000 7fd60c000700 1 -- 172.21.15.89:0/1957438967 >> 172.21.15.89:0/1957438967 conn(0x7fd60408aff0 msgr2=0x7fd60415b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:18.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.762+0000 7fd60c000700 1 -- 172.21.15.89:0/1957438967 shutdown_connections 2024-04-02T09:17:18.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.762+0000 7fd60c000700 1 -- 172.21.15.89:0/1957438967 wait complete. 2024-04-02T09:17:18.777 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-02T09:17:18.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.972+0000 7f2a6d400700 1 Processor -- start 2024-04-02T09:17:18.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.972+0000 7f2a6d400700 1 -- start start 2024-04-02T09:17:18.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a6d400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a680d16a0 0x7f2a680d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a6d400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2a680d2370 0x7f2a680ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a6d400700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a68161420 con 0x7f2a6815d9d0 2024-04-02T09:17:18.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a6d400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a680d4340 con 0x7f2a680d16a0 2024-04-02T09:17:18.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a6d400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a6815edc0 con 0x7f2a680d2370 2024-04-02T09:17:18.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a65e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a680d16a0 0x7f2a680d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a65e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a680d16a0 0x7f2a680d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46348/0 (socket says 172.21.15.89:46348) 2024-04-02T09:17:18.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a66800700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2a6815d9d0 0x7f2a680d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57908/0 (socket says 172.21.15.89:57908) 2024-04-02T09:17:18.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a65e00700 1 -- 172.21.15.89:0/2093582010 learned_addr learned my addr 172.21.15.89:0/2093582010 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:18.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a67200700 1 --2- 172.21.15.89:0/2093582010 >> v2:172.21.15.113:3300/0 conn(0x7f2a680d2370 0x7f2a680ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a65e00700 1 -- 172.21.15.89:0/2093582010 >> v2:172.21.15.113:3300/0 conn(0x7f2a680d2370 msgr2=0x7f2a680ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:18.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a65e00700 1 --2- 172.21.15.89:0/2093582010 >> v2:172.21.15.113:3300/0 conn(0x7f2a680d2370 0x7f2a680ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a65e00700 1 -- 172.21.15.89:0/2093582010 >> v1:172.21.15.89:6789/0 conn(0x7f2a6815d9d0 legacy=0x7f2a680d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a65e00700 1 -- 172.21.15.89:0/2093582010 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a68161700 con 0x7f2a680d16a0 2024-04-02T09:17:18.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.974+0000 7f2a65400700 1 -- 172.21.15.89:0/2093582010 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830008630 0 0) 0x7f2a68161420 con 0x7f2a6815d9d0 2024-04-02T09:17:18.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.975+0000 7f2a65e00700 1 --2- 172.21.15.89:0/2093582010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680d16a0 0x7f2a680d1e30 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7f2a6808ca10 tx=0x7f2a54009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ef3b548a872605 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:18.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.975+0000 7f2a65400700 1 -- 172.21.15.89:0/2093582010 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54005e30 con 0x7f2a680d16a0 2024-04-02T09:17:18.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.975+0000 7f2a65400700 1 -- 172.21.15.89:0/2093582010 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a54004030 con 0x7f2a680d16a0 2024-04-02T09:17:18.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.975+0000 7f2a65400700 1 -- 172.21.15.89:0/2093582010 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a540043e0 con 0x7f2a680d16a0 2024-04-02T09:17:18.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.975+0000 7f2a6d400700 1 -- 172.21.15.89:0/2093582010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680d16a0 msgr2=0x7f2a680d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.975+0000 7f2a6d400700 1 --2- 172.21.15.89:0/2093582010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680d16a0 0x7f2a680d1e30 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7f2a6808ca10 tx=0x7f2a54009bd0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.975+0000 7f2a6d400700 1 -- 172.21.15.89:0/2093582010 shutdown_connections 2024-04-02T09:17:18.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.975+0000 7f2a6d400700 1 --2- 172.21.15.89:0/2093582010 >> v2:172.21.15.113:3300/0 conn(0x7f2a680d2370 0x7f2a680ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.975+0000 7f2a6d400700 1 --2- 172.21.15.89:0/2093582010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680d16a0 0x7f2a680d1e30 unknown :-1 s=CLOSED pgs=3109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.975+0000 7f2a6d400700 1 -- 172.21.15.89:0/2093582010 >> 172.21.15.89:0/2093582010 conn(0x7f2a6808aff0 msgr2=0x7f2a680d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:18.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.975+0000 7f2a6d400700 1 -- 172.21.15.89:0/2093582010 shutdown_connections 2024-04-02T09:17:18.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.976+0000 7f2a6d400700 1 -- 172.21.15.89:0/2093582010 wait complete. 2024-04-02T09:17:18.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.977+0000 7f2a6d400700 1 Processor -- start 2024-04-02T09:17:18.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.977+0000 7f2a6d400700 1 -- start start 2024-04-02T09:17:18.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.978+0000 7f2a6d400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a680d16a0 0x7f2a680b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.978+0000 7f2a6d400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a680d2370 0x7f2a681f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.978+0000 7f2a6d400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6815d9d0 0x7f2a681f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.978+0000 7f2a6d400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a680b75a0 con 0x7f2a6815d9d0 2024-04-02T09:17:18.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.978+0000 7f2a6d400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a680b6920 con 0x7f2a680d16a0 2024-04-02T09:17:18.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.978+0000 7f2a6d400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a680b6c00 con 0x7f2a680d2370 2024-04-02T09:17:18.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a67200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6815d9d0 0x7f2a681f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a67200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6815d9d0 0x7f2a681f36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46354/0 (socket says 172.21.15.89:46354) 2024-04-02T09:17:18.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a67200700 1 -- 172.21.15.89:0/4071879678 learned_addr learned my addr 172.21.15.89:0/4071879678 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:18.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a66800700 1 --2- 172.21.15.89:0/4071879678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a680d16a0 0x7f2a680b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a65e00700 1 --2- 172.21.15.89:0/4071879678 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a680d2370 0x7f2a681f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a67200700 1 -- 172.21.15.89:0/4071879678 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a680d2370 msgr2=0x7f2a681f3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a67200700 1 --2- 172.21.15.89:0/4071879678 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a680d2370 0x7f2a681f3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a67200700 1 -- 172.21.15.89:0/4071879678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a680d16a0 msgr2=0x7f2a680b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:18.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a67200700 1 --2- 172.21.15.89:0/4071879678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a680d16a0 0x7f2a680b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:18.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a67200700 1 -- 172.21.15.89:0/4071879678 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a54009500 con 0x7f2a6815d9d0 2024-04-02T09:17:18.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a67200700 1 --2- 172.21.15.89:0/4071879678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6815d9d0 0x7f2a681f36b0 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7f2a60007e40 tx=0x7f2a6000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:18.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a5ea00700 1 -- 172.21.15.89:0/4071879678 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6000ed30 con 0x7f2a6815d9d0 2024-04-02T09:17:18.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a5ea00700 1 -- 172.21.15.89:0/4071879678 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a60014420 con 0x7f2a6815d9d0 2024-04-02T09:17:18.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a6d400700 1 -- 172.21.15.89:0/4071879678 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a680b6ee0 con 0x7f2a6815d9d0 2024-04-02T09:17:18.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.979+0000 7f2a5ea00700 1 -- 172.21.15.89:0/4071879678 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a600147d0 con 0x7f2a6815d9d0 2024-04-02T09:17:18.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.980+0000 7f2a6d400700 1 -- 172.21.15.89:0/4071879678 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a680b6560 con 0x7f2a6815d9d0 2024-04-02T09:17:18.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.981+0000 7f2a6d400700 1 -- 172.21.15.89:0/4071879678 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a680d1ca0 con 0x7f2a6815d9d0 2024-04-02T09:17:18.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.981+0000 7f2a5ea00700 1 -- 172.21.15.89:0/4071879678 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2a60014930 con 0x7f2a6815d9d0 2024-04-02T09:17:18.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.982+0000 7f2a5ea00700 1 --2- 172.21.15.89:0/4071879678 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2a4806adb0 0x7f2a4806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:18.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.982+0000 7f2a5ea00700 1 -- 172.21.15.89:0/4071879678 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f2a60089d10 con 0x7f2a6815d9d0 2024-04-02T09:17:18.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.983+0000 7f2a66800700 1 --2- 172.21.15.89:0/4071879678 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2a4806adb0 0x7f2a4806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:18.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.983+0000 7f2a66800700 1 --2- 172.21.15.89:0/4071879678 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2a4806adb0 0x7f2a4806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2a58000f60 tx=0x7f2a58007e70 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:18.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:18.987+0000 7f2a5ea00700 1 -- 172.21.15.89:0/4071879678 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2a60054c40 con 0x7f2a6815d9d0 2024-04-02T09:17:19.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.191+0000 7f2a6d400700 1 -- 172.21.15.89:0/4071879678 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a68161fa0 con 0x7f2a4806adb0 2024-04-02T09:17:19.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.192+0000 7f2a5ea00700 1 -- 172.21.15.89:0/4071879678 <== mgr.13575 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2a580133f0 con 0x7f2a4806adb0 2024-04-02T09:17:19.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.194+0000 7f2a6d400700 1 -- 172.21.15.89:0/4071879678 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2a4806adb0 msgr2=0x7f2a4806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.194+0000 7f2a6d400700 1 --2- 172.21.15.89:0/4071879678 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2a4806adb0 0x7f2a4806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2a58000f60 tx=0x7f2a58007e70 comp rx=0 tx=0).stop 2024-04-02T09:17:19.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.194+0000 7f2a6d400700 1 -- 172.21.15.89:0/4071879678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6815d9d0 msgr2=0x7f2a681f36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.194+0000 7f2a6d400700 1 --2- 172.21.15.89:0/4071879678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6815d9d0 0x7f2a681f36b0 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7f2a60007e40 tx=0x7f2a6000e3c0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.194+0000 7f2a6d400700 1 -- 172.21.15.89:0/4071879678 shutdown_connections 2024-04-02T09:17:19.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.194+0000 7f2a6d400700 1 --2- 172.21.15.89:0/4071879678 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f2a4806adb0 0x7f2a4806d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.194+0000 7f2a6d400700 1 --2- 172.21.15.89:0/4071879678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a680d16a0 0x7f2a680b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.194+0000 7f2a6d400700 1 --2- 172.21.15.89:0/4071879678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a6815d9d0 0x7f2a681f36b0 unknown :-1 s=CLOSED pgs=3110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.194+0000 7f2a6d400700 1 --2- 172.21.15.89:0/4071879678 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2a680d2370 0x7f2a681f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.194+0000 7f2a6d400700 1 -- 172.21.15.89:0/4071879678 >> 172.21.15.89:0/4071879678 conn(0x7f2a6808aff0 msgr2=0x7f2a680d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:19.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.194+0000 7f2a6d400700 1 -- 172.21.15.89:0/4071879678 shutdown_connections 2024-04-02T09:17:19.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.195+0000 7f2a6d400700 1 -- 172.21.15.89:0/4071879678 wait complete. 2024-04-02T09:17:19.195 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 1 to deep-scrub 2024-04-02T09:17:19.212 INFO:tasks.ceph:Scrubbing osd.2 2024-04-02T09:17:19.212 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-02T09:17:19.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.410+0000 7fa590c00700 1 Processor -- start 2024-04-02T09:17:19.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.410+0000 7fa590c00700 1 -- start start 2024-04-02T09:17:19.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.411+0000 7fa590c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa58c0b56e0 0x7fa58c0d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.411+0000 7fa590c00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa58c0d1770 0x7fa58c06ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.411+0000 7fa590c00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58c0ceb70 con 0x7fa58c15d7f0 2024-04-02T09:17:19.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.411+0000 7fa590c00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa58c15dbd0 con 0x7fa58c0d1770 2024-04-02T09:17:19.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.411+0000 7fa590c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c0cf000 con 0x7fa58c0b56e0 2024-04-02T09:17:19.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.411+0000 7fa589400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa58c0b56e0 0x7fa58c0d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:19.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.411+0000 7fa58a800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa58c0d1770 0x7fa58c06ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:19.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.412+0000 7fa58a800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa58c0d1770 0x7fa58c06ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:59358/0 (socket says 172.21.15.89:59358) 2024-04-02T09:17:19.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.412+0000 7fa589400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa58c0b56e0 0x7fa58c0d25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:38900/0 (socket says 172.21.15.89:38900) 2024-04-02T09:17:19.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.412+0000 7fa58a800700 1 -- 172.21.15.89:0/2945026481 learned_addr learned my addr 172.21.15.89:0/2945026481 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:19.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.412+0000 7fa589e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fa58c15d7f0 0x7fa58c0b4fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:44200/0 (socket says 172.21.15.89:44200) 2024-04-02T09:17:19.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.412+0000 7fa58a800700 1 -- 172.21.15.89:0/2945026481 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa58c0b56e0 msgr2=0x7fa58c0d25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:19.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.412+0000 7fa58a800700 1 --2- 172.21.15.89:0/2945026481 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa58c0b56e0 0x7fa58c0d25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.412+0000 7fa58a800700 1 -- 172.21.15.89:0/2945026481 >> v1:172.21.15.113:6789/0 conn(0x7fa58c15d7f0 legacy=0x7fa58c0b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.412+0000 7fa58a800700 1 -- 172.21.15.89:0/2945026481 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c0cdf60 con 0x7fa58c0d1770 2024-04-02T09:17:19.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.412+0000 7fa589400700 1 --2- 172.21.15.89:0/2945026481 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa58c0b56e0 0x7fa58c0d25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:17:19.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.412+0000 7fa58a800700 1 --2- 172.21.15.89:0/2945026481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa58c0d1770 0x7fa58c06ceb0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fa584009e50 tx=0x7fa58400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0710b8099937af1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:19.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.413+0000 7fa588a00700 1 -- 172.21.15.89:0/2945026481 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58400e070 con 0x7fa58c0d1770 2024-04-02T09:17:19.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.413+0000 7fa588a00700 1 -- 172.21.15.89:0/2945026481 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58400b910 con 0x7fa58c0d1770 2024-04-02T09:17:19.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.413+0000 7fa588a00700 1 -- 172.21.15.89:0/2945026481 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58400bcc0 con 0x7fa58c0d1770 2024-04-02T09:17:19.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.413+0000 7fa590c00700 1 -- 172.21.15.89:0/2945026481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa58c0d1770 msgr2=0x7fa58c06ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.413+0000 7fa590c00700 1 --2- 172.21.15.89:0/2945026481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa58c0d1770 0x7fa58c06ceb0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fa584009e50 tx=0x7fa58400b330 comp rx=0 tx=0).stop 2024-04-02T09:17:19.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.413+0000 7fa590c00700 1 -- 172.21.15.89:0/2945026481 shutdown_connections 2024-04-02T09:17:19.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.413+0000 7fa590c00700 1 --2- 172.21.15.89:0/2945026481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa58c0d1770 0x7fa58c06ceb0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.413+0000 7fa590c00700 1 --2- 172.21.15.89:0/2945026481 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa58c0b56e0 0x7fa58c0d25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.413+0000 7fa590c00700 1 -- 172.21.15.89:0/2945026481 >> 172.21.15.89:0/2945026481 conn(0x7fa58c08aff0 msgr2=0x7fa58c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:19.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.413+0000 7fa590c00700 1 -- 172.21.15.89:0/2945026481 shutdown_connections 2024-04-02T09:17:19.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.413+0000 7fa590c00700 1 -- 172.21.15.89:0/2945026481 wait complete. 2024-04-02T09:17:19.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.415+0000 7fa590c00700 1 Processor -- start 2024-04-02T09:17:19.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.415+0000 7fa590c00700 1 -- start start 2024-04-02T09:17:19.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.416+0000 7fa590c00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa58c0b56e0 0x7fa58c0c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.416+0000 7fa590c00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa58c0d1770 0x7fa58c0c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.416+0000 7fa590c00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa58c15d7f0 0x7fa58c06b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.416+0000 7fa590c00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c06c530 con 0x7fa58c0d1770 2024-04-02T09:17:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.416+0000 7fa590c00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c06c810 con 0x7fa58c15d7f0 2024-04-02T09:17:19.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.416+0000 7fa590c00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c06caf0 con 0x7fa58c0b56e0 2024-04-02T09:17:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.416+0000 7fa589400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa58c0d1770 0x7fa58c0c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.416+0000 7fa589400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa58c0d1770 0x7fa58c0c75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46362/0 (socket says 172.21.15.89:46362) 2024-04-02T09:17:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.416+0000 7fa589400700 1 -- 172.21.15.89:0/2156218160 learned_addr learned my addr 172.21.15.89:0/2156218160 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:19.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.416+0000 7fa589e00700 1 --2- 172.21.15.89:0/2156218160 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa58c0b56e0 0x7fa58c0c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:19.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.416+0000 7fa58a800700 1 --2- 172.21.15.89:0/2156218160 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa58c15d7f0 0x7fa58c06b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:19.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.417+0000 7fa589400700 1 -- 172.21.15.89:0/2156218160 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa58c0b56e0 msgr2=0x7fa58c0c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.417+0000 7fa589400700 1 --2- 172.21.15.89:0/2156218160 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa58c0b56e0 0x7fa58c0c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.417+0000 7fa589400700 1 -- 172.21.15.89:0/2156218160 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa58c15d7f0 msgr2=0x7fa58c06b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.417+0000 7fa589400700 1 --2- 172.21.15.89:0/2156218160 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa58c15d7f0 0x7fa58c06b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.417+0000 7fa589400700 1 -- 172.21.15.89:0/2156218160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5780094a0 con 0x7fa58c0d1770 2024-04-02T09:17:19.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.417+0000 7fa589400700 1 --2- 172.21.15.89:0/2156218160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa58c0d1770 0x7fa58c0c75c0 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 crypto rx=0x7fa57800ebe0 tx=0x7fa578009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:19.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.417+0000 7fa582000700 1 -- 172.21.15.89:0/2156218160 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57800c8b0 con 0x7fa58c0d1770 2024-04-02T09:17:19.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.417+0000 7fa582000700 1 -- 172.21.15.89:0/2156218160 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57800ca10 con 0x7fa58c0d1770 2024-04-02T09:17:19.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.417+0000 7fa590c00700 1 -- 172.21.15.89:0/2156218160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa584009b30 con 0x7fa58c0d1770 2024-04-02T09:17:19.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.417+0000 7fa582000700 1 -- 172.21.15.89:0/2156218160 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57800cda0 con 0x7fa58c0d1770 2024-04-02T09:17:19.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.417+0000 7fa590c00700 1 -- 172.21.15.89:0/2156218160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa58c06d0b0 con 0x7fa58c0d1770 2024-04-02T09:17:19.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.418+0000 7fa590c00700 1 -- 172.21.15.89:0/2156218160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa58c0ce800 con 0x7fa58c0d1770 2024-04-02T09:17:19.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.419+0000 7fa582000700 1 -- 172.21.15.89:0/2156218160 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa578013470 con 0x7fa58c0d1770 2024-04-02T09:17:19.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.420+0000 7fa582000700 1 --2- 172.21.15.89:0/2156218160 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa56c06adb0 0x7fa56c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.420+0000 7fa582000700 1 -- 172.21.15.89:0/2156218160 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7fa578089b60 con 0x7fa58c0d1770 2024-04-02T09:17:19.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.421+0000 7fa589e00700 1 --2- 172.21.15.89:0/2156218160 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa56c06adb0 0x7fa56c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:19.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.421+0000 7fa582000700 1 --2- 172.21.15.89:0/2156218160 >> [v2:172.21.15.89:6824/43713,v1:172.21.15.89:6825/43713] conn(0x7fa56c071680 0x7fa56c073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.421+0000 7fa582000700 1 -- 172.21.15.89:0/2156218160 --> [v2:172.21.15.89:6824/43713,v1:172.21.15.89:6825/43713] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa56c0740f0 con 0x7fa56c071680 2024-04-02T09:17:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.421+0000 7fa582000700 1 -- 172.21.15.89:0/2156218160 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=284) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa578089ed0 con 0x7fa58c0d1770 2024-04-02T09:17:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.421+0000 7fa58a800700 1 --2- 172.21.15.89:0/2156218160 >> [v2:172.21.15.89:6824/43713,v1:172.21.15.89:6825/43713] conn(0x7fa56c071680 0x7fa56c073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.421+0000 7fa589e00700 1 --2- 172.21.15.89:0/2156218160 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa56c06adb0 0x7fa56c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa57c000f60 tx=0x7fa57c007d10 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:19.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.421+0000 7fa58a800700 1 --2- 172.21.15.89:0/2156218160 >> [v2:172.21.15.89:6824/43713,v1:172.21.15.89:6825/43713] conn(0x7fa56c071680 0x7fa56c073ac0 crc :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:19.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.422+0000 7fa582000700 1 -- 172.21.15.89:0/2156218160 <== osd.2 v2:172.21.15.89:6824/43713 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa56c0740f0 con 0x7fa56c071680 2024-04-02T09:17:19.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.437+0000 7fa590c00700 1 -- 172.21.15.89:0/2156218160 --> [v2:172.21.15.89:6824/43713,v1:172.21.15.89:6825/43713] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa58c0ce520 con 0x7fa56c071680 2024-04-02T09:17:19.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.437+0000 7fa582000700 1 -- 172.21.15.89:0/2156218160 <== osd.2 v2:172.21.15.89:6824/43713 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fa584003a20 con 0x7fa56c071680 2024-04-02T09:17:19.436 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:17:19.437 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T09:17:19.437 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:17:19.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.438+0000 7fa590c00700 1 -- 172.21.15.89:0/2156218160 >> [v2:172.21.15.89:6824/43713,v1:172.21.15.89:6825/43713] conn(0x7fa56c071680 msgr2=0x7fa56c073ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.438+0000 7fa590c00700 1 --2- 172.21.15.89:0/2156218160 >> [v2:172.21.15.89:6824/43713,v1:172.21.15.89:6825/43713] conn(0x7fa56c071680 0x7fa56c073ac0 crc :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.438+0000 7fa590c00700 1 -- 172.21.15.89:0/2156218160 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa56c06adb0 msgr2=0x7fa56c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.438+0000 7fa590c00700 1 --2- 172.21.15.89:0/2156218160 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa56c06adb0 0x7fa56c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa57c000f60 tx=0x7fa57c007d10 comp rx=0 tx=0).stop 2024-04-02T09:17:19.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.438+0000 7fa590c00700 1 -- 172.21.15.89:0/2156218160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa58c0d1770 msgr2=0x7fa58c0c75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.438+0000 7fa590c00700 1 --2- 172.21.15.89:0/2156218160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa58c0d1770 0x7fa58c0c75c0 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 crypto rx=0x7fa57800ebe0 tx=0x7fa578009e20 comp rx=0 tx=0).stop 2024-04-02T09:17:19.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.438+0000 7fa58a800700 1 -- 172.21.15.89:0/2156218160 reap_dead start 2024-04-02T09:17:19.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.438+0000 7fa590c00700 1 -- 172.21.15.89:0/2156218160 shutdown_connections 2024-04-02T09:17:19.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.438+0000 7fa590c00700 1 -- 172.21.15.89:0/2156218160 >> 172.21.15.89:0/2156218160 conn(0x7fa58c08aff0 msgr2=0x7fa58c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:19.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.438+0000 7fa590c00700 1 -- 172.21.15.89:0/2156218160 shutdown_connections 2024-04-02T09:17:19.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.438+0000 7fa590c00700 1 -- 172.21.15.89:0/2156218160 wait complete. 2024-04-02T09:17:19.453 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-02T09:17:19.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.647+0000 7f5166e00700 1 Processor -- start 2024-04-02T09:17:19.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.648+0000 7f5166e00700 1 -- start start 2024-04-02T09:17:19.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.649+0000 7f5166e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51600cd480 0x7f516006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.649+0000 7f5166e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51600cdae0 con 0x7f51600cd020 2024-04-02T09:17:19.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.649+0000 7f5166e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51600cddc0 con 0x7f51600cbb30 2024-04-02T09:17:19.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.649+0000 7f5166e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51600ce0a0 con 0x7f51600cd480 2024-04-02T09:17:19.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.649+0000 7f515f400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f51600cd020 0x7f51600b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57916/0 (socket says 172.21.15.89:57916) 2024-04-02T09:17:19.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.649+0000 7f515f400700 1 -- 172.21.15.89:0/3708517391 learned_addr learned my addr 172.21.15.89:0/3708517391 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:19.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.649+0000 7f5164e00700 1 --2- 172.21.15.89:0/3708517391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51600cd480 0x7f516006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:19.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.649+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550837828 0 0) 0x7f51600cdae0 con 0x7f51600cd020 2024-04-02T09:17:19.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.649+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51440035b0 con 0x7f51600cd020 2024-04-02T09:17:19.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010918504 0 0) 0x7f51600cddc0 con 0x7f51600cbb30 2024-04-02T09:17:19.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51600cdae0 con 0x7f51600cbb30 2024-04-02T09:17:19.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 177680709 0 0) 0x7f51440035b0 con 0x7f51600cd020 2024-04-02T09:17:19.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51600cd480 msgr2=0x7f516006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:19.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 --2- 172.21.15.89:0/3708517391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51600cd480 0x7f516006e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 >> v1:172.21.15.113:6789/0 conn(0x7f51600cbb30 legacy=0x7f51600cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516006dc20 con 0x7f51600cd020 2024-04-02T09:17:19.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f514c001670 con 0x7f51600cd020 2024-04-02T09:17:19.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 >> v1:172.21.15.89:6789/0 conn(0x7f51600cd020 legacy=0x7f51600b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 --2- 172.21.15.89:0/3708517391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51600cd480 0x7f5144003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 --2- 172.21.15.89:0/3708517391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5144004120 0x7f5144004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 --2- 172.21.15.89:0/3708517391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5144004d90 0x7f5144005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51600ce0a0 con 0x7f5144004d90 2024-04-02T09:17:19.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514400baf0 con 0x7f5144004120 2024-04-02T09:17:19.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514400bcb0 con 0x7f51600cd480 2024-04-02T09:17:19.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f5164e00700 1 --2- 172.21.15.89:0/3708517391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5144004d90 0x7f5144005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:19.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515f400700 1 --2- 172.21.15.89:0/3708517391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5144004120 0x7f5144004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:19.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f515fe00700 1 --2- 172.21.15.89:0/3708517391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51600cd480 0x7f5144003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:19.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f5164e00700 1 -- 172.21.15.89:0/3708517391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51600cd480 msgr2=0x7f5144003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:19.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f5164e00700 1 --2- 172.21.15.89:0/3708517391 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f51600cd480 0x7f5144003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f5164e00700 1 -- 172.21.15.89:0/3708517391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5144004120 msgr2=0x7f5144004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:19.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.650+0000 7f5164e00700 1 --2- 172.21.15.89:0/3708517391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5144004120 0x7f5144004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.651+0000 7f5164e00700 1 -- 172.21.15.89:0/3708517391 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51440035b0 con 0x7f5144004d90 2024-04-02T09:17:19.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.651+0000 7f515f400700 1 --2- 172.21.15.89:0/3708517391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5144004120 0x7f5144004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:17:19.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.651+0000 7f5164e00700 1 --2- 172.21.15.89:0/3708517391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5144004d90 0x7f5144005210 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7f5158009fc0 tx=0x7f5144005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9636d67f7bff8e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:19.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.651+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515800ec30 con 0x7f5144004d90 2024-04-02T09:17:19.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.651+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515800f040 con 0x7f5144004d90 2024-04-02T09:17:19.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.651+0000 7f515ea00700 1 -- 172.21.15.89:0/3708517391 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5158014480 con 0x7f5144004d90 2024-04-02T09:17:19.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.651+0000 7f5166e00700 1 -- 172.21.15.89:0/3708517391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5144004d90 msgr2=0x7f5144005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.651+0000 7f5166e00700 1 --2- 172.21.15.89:0/3708517391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5144004d90 0x7f5144005210 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7f5158009fc0 tx=0x7f5144005880 comp rx=0 tx=0).stop 2024-04-02T09:17:19.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.651+0000 7f5164e00700 1 -- 172.21.15.89:0/3708517391 reap_dead start 2024-04-02T09:17:19.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.652+0000 7f5166e00700 1 -- 172.21.15.89:0/3708517391 shutdown_connections 2024-04-02T09:17:19.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.652+0000 7f5166e00700 1 -- 172.21.15.89:0/3708517391 >> 172.21.15.89:0/3708517391 conn(0x7f51600b0d70 msgr2=0x7f51600b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:19.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.652+0000 7f5166e00700 1 -- 172.21.15.89:0/3708517391 shutdown_connections 2024-04-02T09:17:19.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.652+0000 7f5166e00700 1 -- 172.21.15.89:0/3708517391 wait complete. 2024-04-02T09:17:19.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.653+0000 7f5166e00700 1 Processor -- start 2024-04-02T09:17:19.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.654+0000 7f5166e00700 1 -- start start 2024-04-02T09:17:19.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.654+0000 7f5166e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5144004d90 0x7f516015a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.654+0000 7f5166e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f516009ce40 0x7f516015abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.655+0000 7f5166e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51600c8480 0x7f51600c8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.655+0000 7f5166e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51600c9520 con 0x7f516009ce40 2024-04-02T09:17:19.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.655+0000 7f5166e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51600ae840 con 0x7f51600c8480 2024-04-02T09:17:19.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.655+0000 7f5166e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51600c9240 con 0x7f5144004d90 2024-04-02T09:17:19.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.655+0000 7f515f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f516009ce40 0x7f516015abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:19.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.655+0000 7f515f400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f516009ce40 0x7f516015abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46372/0 (socket says 172.21.15.89:46372) 2024-04-02T09:17:19.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.655+0000 7f515f400700 1 -- 172.21.15.89:0/3920495650 learned_addr learned my addr 172.21.15.89:0/3920495650 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:19.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.655+0000 7f515f400700 1 -- 172.21.15.89:0/3920495650 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5144004d90 msgr2=0x7f516015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.655+0000 7f515f400700 1 --2- 172.21.15.89:0/3920495650 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5144004d90 0x7f516015a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.655+0000 7f515f400700 1 -- 172.21.15.89:0/3920495650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51600c8480 msgr2=0x7f51600c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.655+0000 7f515f400700 1 --2- 172.21.15.89:0/3920495650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51600c8480 0x7f51600c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.655+0000 7f515f400700 1 -- 172.21.15.89:0/3920495650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51601446d0 con 0x7f516009ce40 2024-04-02T09:17:19.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.655+0000 7f515f400700 1 --2- 172.21.15.89:0/3920495650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f516009ce40 0x7f516015abb0 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 crypto rx=0x7f514c00e8a0 tx=0x7f514c00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:19.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.655+0000 7f5157e00700 1 -- 172.21.15.89:0/3920495650 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c00eea0 con 0x7f516009ce40 2024-04-02T09:17:19.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.656+0000 7f5157e00700 1 -- 172.21.15.89:0/3920495650 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514c00c700 con 0x7f516009ce40 2024-04-02T09:17:19.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.656+0000 7f5166e00700 1 -- 172.21.15.89:0/3920495650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51601449b0 con 0x7f516009ce40 2024-04-02T09:17:19.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.656+0000 7f5157e00700 1 -- 172.21.15.89:0/3920495650 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c00ca70 con 0x7f516009ce40 2024-04-02T09:17:19.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.656+0000 7f5166e00700 1 -- 172.21.15.89:0/3920495650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51600c8e40 con 0x7f516009ce40 2024-04-02T09:17:19.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.657+0000 7f5166e00700 1 -- 172.21.15.89:0/3920495650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5160144530 con 0x7f516009ce40 2024-04-02T09:17:19.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.657+0000 7f5157e00700 1 -- 172.21.15.89:0/3920495650 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f514c00cbd0 con 0x7f516009ce40 2024-04-02T09:17:19.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.658+0000 7f5157e00700 1 --2- 172.21.15.89:0/3920495650 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f514006adb0 0x7f514006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:19.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.658+0000 7f5157e00700 1 -- 172.21.15.89:0/3920495650 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f514c0894f0 con 0x7f516009ce40 2024-04-02T09:17:19.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.659+0000 7f515fe00700 1 --2- 172.21.15.89:0/3920495650 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f514006adb0 0x7f514006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:19.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.659+0000 7f515fe00700 1 --2- 172.21.15.89:0/3920495650 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f514006adb0 0x7f514006d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5160144400 tx=0x7f5150009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:19.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.663+0000 7f5157e00700 1 -- 172.21.15.89:0/3920495650 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5160144530 con 0x7f516009ce40 2024-04-02T09:17:19.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.863+0000 7f5166e00700 1 -- 172.21.15.89:0/3920495650 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f51601594a0 con 0x7f514006adb0 2024-04-02T09:17:19.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.863+0000 7f5157e00700 1 -- 172.21.15.89:0/3920495650 <== mgr.13575 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f51500050d0 con 0x7f514006adb0 2024-04-02T09:17:19.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.865+0000 7f5166e00700 1 -- 172.21.15.89:0/3920495650 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f514006adb0 msgr2=0x7f514006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.865+0000 7f5166e00700 1 --2- 172.21.15.89:0/3920495650 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f514006adb0 0x7f514006d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5160144400 tx=0x7f5150009040 comp rx=0 tx=0).stop 2024-04-02T09:17:19.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.865+0000 7f5166e00700 1 -- 172.21.15.89:0/3920495650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f516009ce40 msgr2=0x7f516015abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:19.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.865+0000 7f5166e00700 1 --2- 172.21.15.89:0/3920495650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f516009ce40 0x7f516015abb0 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 crypto rx=0x7f514c00e8a0 tx=0x7f514c00ec60 comp rx=0 tx=0).stop 2024-04-02T09:17:19.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.865+0000 7f5166e00700 1 -- 172.21.15.89:0/3920495650 shutdown_connections 2024-04-02T09:17:19.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.865+0000 7f5166e00700 1 --2- 172.21.15.89:0/3920495650 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f514006adb0 0x7f514006d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.865+0000 7f5166e00700 1 --2- 172.21.15.89:0/3920495650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51600c8480 0x7f51600c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.865+0000 7f5166e00700 1 --2- 172.21.15.89:0/3920495650 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5144004d90 0x7f516015a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.865+0000 7f5166e00700 1 --2- 172.21.15.89:0/3920495650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f516009ce40 0x7f516015abb0 unknown :-1 s=CLOSED pgs=3114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:19.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.865+0000 7f5166e00700 1 -- 172.21.15.89:0/3920495650 >> 172.21.15.89:0/3920495650 conn(0x7f51600b0d70 msgr2=0x7f51600ce540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:19.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.865+0000 7f5166e00700 1 -- 172.21.15.89:0/3920495650 shutdown_connections 2024-04-02T09:17:19.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:19.866+0000 7f5166e00700 1 -- 172.21.15.89:0/3920495650 wait complete. 2024-04-02T09:17:19.866 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 2 to deep-scrub 2024-04-02T09:17:19.882 INFO:tasks.ceph:Scrubbing osd.3 2024-04-02T09:17:19.882 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-02T09:17:20.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.079+0000 7fd399a00700 1 Processor -- start 2024-04-02T09:17:20.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.079+0000 7fd399a00700 1 -- start start 2024-04-02T09:17:20.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.080+0000 7fd399a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd39415ddf0 0x7fd3940ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.080+0000 7fd399a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd39415e1d0 0x7fd3940d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.080+0000 7fd399a00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3940ce8a0 con 0x7fd3940d2710 2024-04-02T09:17:20.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.080+0000 7fd399a00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3940d2b80 con 0x7fd39415e1d0 2024-04-02T09:17:20.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.080+0000 7fd399a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39415ec60 con 0x7fd39415ddf0 2024-04-02T09:17:20.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.081+0000 7fd38be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd39415e1d0 0x7fd3940d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.081+0000 7fd38be00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd39415e1d0 0x7fd3940d21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46376/0 (socket says 172.21.15.89:46376) 2024-04-02T09:17:20.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.081+0000 7fd38be00700 1 -- 172.21.15.89:0/461129453 learned_addr learned my addr 172.21.15.89:0/461129453 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:20.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.081+0000 7fd392a00700 1 --2- 172.21.15.89:0/461129453 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd39415ddf0 0x7fd3940ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.081+0000 7fd38be00700 1 -- 172.21.15.89:0/461129453 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd39415ddf0 msgr2=0x7fd3940ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:20.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.081+0000 7fd38be00700 1 --2- 172.21.15.89:0/461129453 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd39415ddf0 0x7fd3940ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.081+0000 7fd38be00700 1 -- 172.21.15.89:0/461129453 >> v1:172.21.15.113:6789/0 conn(0x7fd3940d2710 legacy=0x7fd3941612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:20.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.081+0000 7fd38be00700 1 -- 172.21.15.89:0/461129453 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3940cdc90 con 0x7fd39415e1d0 2024-04-02T09:17:20.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.081+0000 7fd392a00700 1 --2- 172.21.15.89:0/461129453 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd39415ddf0 0x7fd3940ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:17:20.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.081+0000 7fd38be00700 1 --2- 172.21.15.89:0/461129453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd39415e1d0 0x7fd3940d21d0 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7fd37c0098f0 tx=0x7fd37c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ea7e196252e6a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:20.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.082+0000 7fd392000700 1 -- 172.21.15.89:0/461129453 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37c004230 con 0x7fd39415e1d0 2024-04-02T09:17:20.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.082+0000 7fd392000700 1 -- 172.21.15.89:0/461129453 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37c004390 con 0x7fd39415e1d0 2024-04-02T09:17:20.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.082+0000 7fd392000700 1 -- 172.21.15.89:0/461129453 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37c0036a0 con 0x7fd39415e1d0 2024-04-02T09:17:20.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.082+0000 7fd399a00700 1 -- 172.21.15.89:0/461129453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd39415e1d0 msgr2=0x7fd3940d21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.082+0000 7fd399a00700 1 --2- 172.21.15.89:0/461129453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd39415e1d0 0x7fd3940d21d0 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7fd37c0098f0 tx=0x7fd37c009c00 comp rx=0 tx=0).stop 2024-04-02T09:17:20.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.082+0000 7fd399a00700 1 -- 172.21.15.89:0/461129453 shutdown_connections 2024-04-02T09:17:20.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.082+0000 7fd399a00700 1 --2- 172.21.15.89:0/461129453 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd39415ddf0 0x7fd3940ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.082+0000 7fd399a00700 1 --2- 172.21.15.89:0/461129453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd39415e1d0 0x7fd3940d21d0 unknown :-1 s=CLOSED pgs=3115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.082+0000 7fd399a00700 1 -- 172.21.15.89:0/461129453 >> 172.21.15.89:0/461129453 conn(0x7fd39408aff0 msgr2=0x7fd3940d3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:20.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.082+0000 7fd399a00700 1 -- 172.21.15.89:0/461129453 shutdown_connections 2024-04-02T09:17:20.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.082+0000 7fd399a00700 1 -- 172.21.15.89:0/461129453 wait complete. 2024-04-02T09:17:20.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.084+0000 7fd399a00700 1 Processor -- start 2024-04-02T09:17:20.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.084+0000 7fd399a00700 1 -- start start 2024-04-02T09:17:20.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.085+0000 7fd399a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3940d2710 0x7fd394156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.085+0000 7fd399a00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd39415ddf0 0x7fd394159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.085+0000 7fd399a00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd39415e1d0 0x7fd3941e9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.085+0000 7fd399a00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3940cd0a0 con 0x7fd3940d2710 2024-04-02T09:17:20.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.085+0000 7fd399a00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd394160210 con 0x7fd39415e1d0 2024-04-02T09:17:20.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.085+0000 7fd399a00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3941604f0 con 0x7fd39415ddf0 2024-04-02T09:17:20.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.085+0000 7fd392a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3940d2710 0x7fd394156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.085+0000 7fd38be00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd39415ddf0 0x7fd394159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.085+0000 7fd392a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3940d2710 0x7fd394156360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46386/0 (socket says 172.21.15.89:46386) 2024-04-02T09:17:20.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.085+0000 7fd392a00700 1 -- 172.21.15.89:0/2998904635 learned_addr learned my addr 172.21.15.89:0/2998904635 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:20.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.085+0000 7fd393400700 1 --2- 172.21.15.89:0/2998904635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd39415e1d0 0x7fd3941e9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.086+0000 7fd392a00700 1 -- 172.21.15.89:0/2998904635 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd39415ddf0 msgr2=0x7fd394159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.086+0000 7fd392a00700 1 --2- 172.21.15.89:0/2998904635 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd39415ddf0 0x7fd394159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.086+0000 7fd392a00700 1 -- 172.21.15.89:0/2998904635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd39415e1d0 msgr2=0x7fd3941e9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.086+0000 7fd392a00700 1 --2- 172.21.15.89:0/2998904635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd39415e1d0 0x7fd3941e9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.086+0000 7fd392a00700 1 -- 172.21.15.89:0/2998904635 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37c009550 con 0x7fd3940d2710 2024-04-02T09:17:20.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.086+0000 7fd392a00700 1 --2- 172.21.15.89:0/2998904635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3940d2710 0x7fd394156360 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7fd38400c750 tx=0x7fd38400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:20.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.086+0000 7fd38aa00700 1 -- 172.21.15.89:0/2998904635 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd384011800 con 0x7fd3940d2710 2024-04-02T09:17:20.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.086+0000 7fd38aa00700 1 -- 172.21.15.89:0/2998904635 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd384011960 con 0x7fd3940d2710 2024-04-02T09:17:20.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.086+0000 7fd38aa00700 1 -- 172.21.15.89:0/2998904635 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd384011cd0 con 0x7fd3940d2710 2024-04-02T09:17:20.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.086+0000 7fd399a00700 1 -- 172.21.15.89:0/2998904635 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39415a940 con 0x7fd3940d2710 2024-04-02T09:17:20.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.086+0000 7fd399a00700 1 -- 172.21.15.89:0/2998904635 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39415af00 con 0x7fd3940d2710 2024-04-02T09:17:20.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.087+0000 7fd399a00700 1 -- 172.21.15.89:0/2998904635 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd3940ce530 con 0x7fd3940d2710 2024-04-02T09:17:20.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.088+0000 7fd38aa00700 1 -- 172.21.15.89:0/2998904635 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd384021350 con 0x7fd3940d2710 2024-04-02T09:17:20.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.089+0000 7fd38aa00700 1 --2- 172.21.15.89:0/2998904635 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd37006adb0 0x7fd37006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.089+0000 7fd38aa00700 1 -- 172.21.15.89:0/2998904635 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7fd384089610 con 0x7fd3940d2710 2024-04-02T09:17:20.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.089+0000 7fd38be00700 1 --2- 172.21.15.89:0/2998904635 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd37006adb0 0x7fd37006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.089+0000 7fd38aa00700 1 --2- 172.21.15.89:0/2998904635 >> [v2:172.21.15.89:6816/43716,v1:172.21.15.89:6817/43716] conn(0x7fd370071680 0x7fd370073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.090+0000 7fd38aa00700 1 -- 172.21.15.89:0/2998904635 --> [v2:172.21.15.89:6816/43716,v1:172.21.15.89:6817/43716] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3700740f0 con 0x7fd370071680 2024-04-02T09:17:20.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.090+0000 7fd38aa00700 1 -- 172.21.15.89:0/2998904635 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=284) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd38408bcd0 con 0x7fd3940d2710 2024-04-02T09:17:20.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.090+0000 7fd393400700 1 --2- 172.21.15.89:0/2998904635 >> [v2:172.21.15.89:6816/43716,v1:172.21.15.89:6817/43716] conn(0x7fd370071680 0x7fd370073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.090+0000 7fd38be00700 1 --2- 172.21.15.89:0/2998904635 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd37006adb0 0x7fd37006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd37c009130 tx=0x7fd37c008030 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:20.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.090+0000 7fd393400700 1 --2- 172.21.15.89:0/2998904635 >> [v2:172.21.15.89:6816/43716,v1:172.21.15.89:6817/43716] conn(0x7fd370071680 0x7fd370073ac0 crc :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:20.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.091+0000 7fd38aa00700 1 -- 172.21.15.89:0/2998904635 <== osd.3 v2:172.21.15.89:6816/43716 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd3700740f0 con 0x7fd370071680 2024-04-02T09:17:20.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.106+0000 7fd399a00700 1 -- 172.21.15.89:0/2998904635 --> [v2:172.21.15.89:6816/43716,v1:172.21.15.89:6817/43716] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd3940ce250 con 0x7fd370071680 2024-04-02T09:17:20.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.106+0000 7fd38aa00700 1 -- 172.21.15.89:0/2998904635 <== osd.3 v2:172.21.15.89:6816/43716 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd38c005040 con 0x7fd370071680 2024-04-02T09:17:20.104 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:17:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T09:17:20.105 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:17:20.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.106+0000 7fd399a00700 1 -- 172.21.15.89:0/2998904635 >> [v2:172.21.15.89:6816/43716,v1:172.21.15.89:6817/43716] conn(0x7fd370071680 msgr2=0x7fd370073ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.106+0000 7fd399a00700 1 --2- 172.21.15.89:0/2998904635 >> [v2:172.21.15.89:6816/43716,v1:172.21.15.89:6817/43716] conn(0x7fd370071680 0x7fd370073ac0 crc :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.106+0000 7fd399a00700 1 -- 172.21.15.89:0/2998904635 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd37006adb0 msgr2=0x7fd37006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.106+0000 7fd399a00700 1 --2- 172.21.15.89:0/2998904635 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd37006adb0 0x7fd37006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd37c009130 tx=0x7fd37c008030 comp rx=0 tx=0).stop 2024-04-02T09:17:20.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.106+0000 7fd399a00700 1 -- 172.21.15.89:0/2998904635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3940d2710 msgr2=0x7fd394156360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.106+0000 7fd399a00700 1 --2- 172.21.15.89:0/2998904635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3940d2710 0x7fd394156360 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7fd38400c750 tx=0x7fd38400ca60 comp rx=0 tx=0).stop 2024-04-02T09:17:20.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.106+0000 7fd393400700 1 -- 172.21.15.89:0/2998904635 reap_dead start 2024-04-02T09:17:20.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.107+0000 7fd399a00700 1 -- 172.21.15.89:0/2998904635 shutdown_connections 2024-04-02T09:17:20.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.107+0000 7fd399a00700 1 -- 172.21.15.89:0/2998904635 >> 172.21.15.89:0/2998904635 conn(0x7fd39408aff0 msgr2=0x7fd3940d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:20.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.107+0000 7fd399a00700 1 -- 172.21.15.89:0/2998904635 shutdown_connections 2024-04-02T09:17:20.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.107+0000 7fd399a00700 1 -- 172.21.15.89:0/2998904635 wait complete. 2024-04-02T09:17:20.122 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-02T09:17:20.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.318+0000 7f4128200700 1 Processor -- start 2024-04-02T09:17:20.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.319+0000 7f4128200700 1 -- start start 2024-04-02T09:17:20.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.320+0000 7f4128200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41200cee10 0x7f4120161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.320+0000 7f4128200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41200ce270 con 0x7f41200d21f0 2024-04-02T09:17:20.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.320+0000 7f4128200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412015b600 con 0x7f41201611b0 2024-04-02T09:17:20.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.320+0000 7f4128200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41200d2760 con 0x7f41200cee10 2024-04-02T09:17:20.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.320+0000 7f4124e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f41200d21f0 0x7f41200ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57924/0 (socket says 172.21.15.89:57924) 2024-04-02T09:17:20.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.320+0000 7f4124e00700 1 -- 172.21.15.89:0/2542806434 learned_addr learned my addr 172.21.15.89:0/2542806434 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:20.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.320+0000 7f4126200700 1 --2- 172.21.15.89:0/2542806434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41200cee10 0x7f4120161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.320+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955763638 0 0) 0x7f41200ce270 con 0x7f41200d21f0 2024-04-02T09:17:20.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41080035b0 con 0x7f41200d21f0 2024-04-02T09:17:20.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552782359 0 0) 0x7f412015b600 con 0x7f41201611b0 2024-04-02T09:17:20.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41200ce270 con 0x7f41201611b0 2024-04-02T09:17:20.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 837065031 0 0) 0x7f41080035b0 con 0x7f41200d21f0 2024-04-02T09:17:20.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41200cee10 msgr2=0x7f4120161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:20.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 --2- 172.21.15.89:0/2542806434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41200cee10 0x7f4120161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 >> v1:172.21.15.113:6789/0 conn(0x7f41201611b0 legacy=0x7f412015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41200d2a40 con 0x7f41200d21f0 2024-04-02T09:17:20.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f410c001670 con 0x7f41200d21f0 2024-04-02T09:17:20.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f4126200700 1 --2- 172.21.15.89:0/2542806434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41200cee10 0x7f4120161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:17:20.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 >> v1:172.21.15.89:6789/0 conn(0x7f41200d21f0 legacy=0x7f41200ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 --2- 172.21.15.89:0/2542806434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108003bc0 0x7f4108003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 --2- 172.21.15.89:0/2542806434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41200cee10 0x7f4108004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 --2- 172.21.15.89:0/2542806434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108004d90 0x7f4108005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41200d2760 con 0x7f4108004d90 2024-04-02T09:17:20.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410800bac0 con 0x7f4108003bc0 2024-04-02T09:17:20.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410800bc80 con 0x7f41200cee10 2024-04-02T09:17:20.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f4126200700 1 --2- 172.21.15.89:0/2542806434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108004d90 0x7f4108005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f4124e00700 1 --2- 172.21.15.89:0/2542806434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41200cee10 0x7f4108004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.321+0000 7f4125800700 1 --2- 172.21.15.89:0/2542806434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108003bc0 0x7f4108003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.322+0000 7f4126200700 1 -- 172.21.15.89:0/2542806434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41200cee10 msgr2=0x7f4108004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:20.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.322+0000 7f4126200700 1 --2- 172.21.15.89:0/2542806434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41200cee10 0x7f4108004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.322+0000 7f4126200700 1 -- 172.21.15.89:0/2542806434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108003bc0 msgr2=0x7f4108003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:20.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.322+0000 7f4126200700 1 --2- 172.21.15.89:0/2542806434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108003bc0 0x7f4108003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.322+0000 7f4126200700 1 -- 172.21.15.89:0/2542806434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4118009a70 con 0x7f4108004d90 2024-04-02T09:17:20.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.322+0000 7f4126200700 1 --2- 172.21.15.89:0/2542806434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108004d90 0x7f4108005210 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7f411800e350 tx=0x7f4108005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae86e3b7798779c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:20.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.322+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411800edf0 con 0x7f4108004d90 2024-04-02T09:17:20.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.322+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411800f040 con 0x7f4108004d90 2024-04-02T09:17:20.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.322+0000 7f411fe00700 1 -- 172.21.15.89:0/2542806434 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41180146a0 con 0x7f4108004d90 2024-04-02T09:17:20.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.323+0000 7f4128200700 1 -- 172.21.15.89:0/2542806434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108004d90 msgr2=0x7f4108005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.323+0000 7f4128200700 1 --2- 172.21.15.89:0/2542806434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108004d90 0x7f4108005210 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7f411800e350 tx=0x7f4108005880 comp rx=0 tx=0).stop 2024-04-02T09:17:20.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.323+0000 7f4126200700 1 -- 172.21.15.89:0/2542806434 reap_dead start 2024-04-02T09:17:20.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.323+0000 7f4128200700 1 -- 172.21.15.89:0/2542806434 shutdown_connections 2024-04-02T09:17:20.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.323+0000 7f4128200700 1 -- 172.21.15.89:0/2542806434 >> 172.21.15.89:0/2542806434 conn(0x7f412008aff0 msgr2=0x7f41200d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:20.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.323+0000 7f4128200700 1 -- 172.21.15.89:0/2542806434 shutdown_connections 2024-04-02T09:17:20.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.323+0000 7f4128200700 1 -- 172.21.15.89:0/2542806434 wait complete. 2024-04-02T09:17:20.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.325+0000 7f4128200700 1 Processor -- start 2024-04-02T09:17:20.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.325+0000 7f4128200700 1 -- start start 2024-04-02T09:17:20.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4128200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108004d90 0x7f4120150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4128200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41200c7340 0x7f41200cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4128200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41200cb340 0x7f4120155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4128200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41200b61f0 con 0x7f41200cb340 2024-04-02T09:17:20.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4128200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41200b64d0 con 0x7f4108004d90 2024-04-02T09:17:20.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4128200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41200b67b0 con 0x7f41200c7340 2024-04-02T09:17:20.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4126200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41200cb340 0x7f4120155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4126200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41200cb340 0x7f4120155d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46400/0 (socket says 172.21.15.89:46400) 2024-04-02T09:17:20.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4126200700 1 -- 172.21.15.89:0/1712255103 learned_addr learned my addr 172.21.15.89:0/1712255103 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:20.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4125800700 1 --2- 172.21.15.89:0/1712255103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108004d90 0x7f4120150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4124e00700 1 --2- 172.21.15.89:0/1712255103 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41200c7340 0x7f41200cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4126200700 1 -- 172.21.15.89:0/1712255103 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41200c7340 msgr2=0x7f41200cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4126200700 1 --2- 172.21.15.89:0/1712255103 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41200c7340 0x7f41200cc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4126200700 1 -- 172.21.15.89:0/1712255103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108004d90 msgr2=0x7f4120150d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4126200700 1 --2- 172.21.15.89:0/1712255103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108004d90 0x7f4120150d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4126200700 1 -- 172.21.15.89:0/1712255103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41200b6a90 con 0x7f41200cb340 2024-04-02T09:17:20.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.327+0000 7f4126200700 1 --2- 172.21.15.89:0/1712255103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41200cb340 0x7f4120155d80 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7f411800e2f0 tx=0x7f4118009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:20.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.328+0000 7f411d600700 1 -- 172.21.15.89:0/1712255103 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118009490 con 0x7f41200cb340 2024-04-02T09:17:20.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.328+0000 7f411d600700 1 -- 172.21.15.89:0/1712255103 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4118004250 con 0x7f41200cb340 2024-04-02T09:17:20.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.328+0000 7f4128200700 1 -- 172.21.15.89:0/1712255103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41200b6d70 con 0x7f41200cb340 2024-04-02T09:17:20.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.328+0000 7f411d600700 1 -- 172.21.15.89:0/1712255103 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118004600 con 0x7f41200cb340 2024-04-02T09:17:20.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.328+0000 7f4128200700 1 -- 172.21.15.89:0/1712255103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41200cc820 con 0x7f41200cb340 2024-04-02T09:17:20.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.329+0000 7f4128200700 1 -- 172.21.15.89:0/1712255103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41200b6f90 con 0x7f41200cb340 2024-04-02T09:17:20.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.330+0000 7f411d600700 1 -- 172.21.15.89:0/1712255103 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4118004760 con 0x7f41200cb340 2024-04-02T09:17:20.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.331+0000 7f411d600700 1 --2- 172.21.15.89:0/1712255103 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f410006adb0 0x7f410006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.331+0000 7f411d600700 1 -- 172.21.15.89:0/1712255103 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f411808aa70 con 0x7f41200cb340 2024-04-02T09:17:20.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.331+0000 7f4125800700 1 --2- 172.21.15.89:0/1712255103 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f410006adb0 0x7f410006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.332+0000 7f4125800700 1 --2- 172.21.15.89:0/1712255103 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f410006adb0 0x7f410006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4114000c00 tx=0x7f4114009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:20.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.335+0000 7f411d600700 1 -- 172.21.15.89:0/1712255103 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4118055a30 con 0x7f41200cb340 2024-04-02T09:17:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.535+0000 7f4128200700 1 -- 172.21.15.89:0/1712255103 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f41200b6640 con 0x7f410006adb0 2024-04-02T09:17:20.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.535+0000 7f411d600700 1 -- 172.21.15.89:0/1712255103 <== mgr.13575 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f41200b6640 con 0x7f410006adb0 2024-04-02T09:17:20.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.538+0000 7f4128200700 1 -- 172.21.15.89:0/1712255103 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f410006adb0 msgr2=0x7f410006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.538+0000 7f4128200700 1 --2- 172.21.15.89:0/1712255103 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f410006adb0 0x7f410006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4114000c00 tx=0x7f4114009040 comp rx=0 tx=0).stop 2024-04-02T09:17:20.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.538+0000 7f4128200700 1 -- 172.21.15.89:0/1712255103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41200cb340 msgr2=0x7f4120155d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.538+0000 7f4128200700 1 --2- 172.21.15.89:0/1712255103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41200cb340 0x7f4120155d80 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7f411800e2f0 tx=0x7f4118009750 comp rx=0 tx=0).stop 2024-04-02T09:17:20.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.538+0000 7f4128200700 1 -- 172.21.15.89:0/1712255103 shutdown_connections 2024-04-02T09:17:20.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.538+0000 7f4128200700 1 --2- 172.21.15.89:0/1712255103 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f410006adb0 0x7f410006d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.538+0000 7f4128200700 1 --2- 172.21.15.89:0/1712255103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108004d90 0x7f4120150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.538+0000 7f4128200700 1 --2- 172.21.15.89:0/1712255103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41200cb340 0x7f4120155d80 unknown :-1 s=CLOSED pgs=3119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.538+0000 7f4128200700 1 --2- 172.21.15.89:0/1712255103 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41200c7340 0x7f41200cc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.538+0000 7f4128200700 1 -- 172.21.15.89:0/1712255103 >> 172.21.15.89:0/1712255103 conn(0x7f412008aff0 msgr2=0x7f41200d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:20.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.538+0000 7f4128200700 1 -- 172.21.15.89:0/1712255103 shutdown_connections 2024-04-02T09:17:20.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.538+0000 7f4128200700 1 -- 172.21.15.89:0/1712255103 wait complete. 2024-04-02T09:17:20.538 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 3 to deep-scrub 2024-04-02T09:17:20.554 INFO:tasks.ceph:Scrubbing osd.4 2024-04-02T09:17:20.555 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-02T09:17:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.753+0000 7f6131600700 1 Processor -- start 2024-04-02T09:17:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.753+0000 7f6131600700 1 -- start start 2024-04-02T09:17:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.754+0000 7f6131600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f612c0d1770 0x7f612c0d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.754+0000 7f6131600700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c0ceb70 con 0x7f612c0b56e0 2024-04-02T09:17:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.754+0000 7f6131600700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c0cf000 con 0x7f612c15d7f0 2024-04-02T09:17:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.754+0000 7f6131600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612c15dbd0 con 0x7f612c0d1770 2024-04-02T09:17:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.754+0000 7f612a000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f612c0b56e0 0x7f612c06ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57926/0 (socket says 172.21.15.89:57926) 2024-04-02T09:17:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.754+0000 7f612a000700 1 -- 172.21.15.89:0/150362497 learned_addr learned my addr 172.21.15.89:0/150362497 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:20.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.754+0000 7f612b400700 1 --2- 172.21.15.89:0/150362497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f612c0d1770 0x7f612c0d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.754+0000 7f6129600700 1 -- 172.21.15.89:0/150362497 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454229806 0 0) 0x7f612c0ceb70 con 0x7f612c0b56e0 2024-04-02T09:17:20.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.755+0000 7f6129600700 1 -- 172.21.15.89:0/150362497 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61100035b0 con 0x7f612c0b56e0 2024-04-02T09:17:20.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.755+0000 7f6129600700 1 -- 172.21.15.89:0/150362497 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173117718 0 0) 0x7f612c0cf000 con 0x7f612c15d7f0 2024-04-02T09:17:20.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.755+0000 7f6129600700 1 -- 172.21.15.89:0/150362497 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f612c0ceb70 con 0x7f612c15d7f0 2024-04-02T09:17:20.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.755+0000 7f612b400700 1 -- 172.21.15.89:0/150362497 >> v1:172.21.15.113:6789/0 conn(0x7f612c15d7f0 legacy=0x7f612c0b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.755+0000 7f612b400700 1 -- 172.21.15.89:0/150362497 >> v1:172.21.15.89:6789/0 conn(0x7f612c0b56e0 legacy=0x7f612c06ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.755+0000 7f6129600700 1 -- 172.21.15.89:0/150362497 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1594668569 0 0) 0x7f61100035b0 con 0x7f612c0b56e0 2024-04-02T09:17:20.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.755+0000 7f612b400700 1 -- 172.21.15.89:0/150362497 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c0cdf60 con 0x7f612c0d1770 2024-04-02T09:17:20.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.755+0000 7f612b400700 1 --2- 172.21.15.89:0/150362497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f612c0d1770 0x7f612c0d25d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6124009dc0 tx=0x7f612400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e1323117c7a780f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:20.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.755+0000 7f6129600700 1 -- 172.21.15.89:0/150362497 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612400e070 con 0x7f612c0d1770 2024-04-02T09:17:20.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.756+0000 7f6129600700 1 -- 172.21.15.89:0/150362497 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6124003a20 con 0x7f612c0d1770 2024-04-02T09:17:20.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.756+0000 7f6129600700 1 -- 172.21.15.89:0/150362497 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6118001670 con 0x7f612c0d1770 2024-04-02T09:17:20.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.756+0000 7f6131600700 1 -- 172.21.15.89:0/150362497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f612c0d1770 msgr2=0x7f612c0d25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.756+0000 7f6131600700 1 --2- 172.21.15.89:0/150362497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f612c0d1770 0x7f612c0d25d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6124009dc0 tx=0x7f612400b330 comp rx=0 tx=0).stop 2024-04-02T09:17:20.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.756+0000 7f6131600700 1 -- 172.21.15.89:0/150362497 shutdown_connections 2024-04-02T09:17:20.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.756+0000 7f6131600700 1 --2- 172.21.15.89:0/150362497 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f612c0d1770 0x7f612c0d25d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.756+0000 7f6131600700 1 -- 172.21.15.89:0/150362497 >> 172.21.15.89:0/150362497 conn(0x7f612c08aff0 msgr2=0x7f612c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:20.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.756+0000 7f6131600700 1 -- 172.21.15.89:0/150362497 shutdown_connections 2024-04-02T09:17:20.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.756+0000 7f6131600700 1 -- 172.21.15.89:0/150362497 wait complete. 2024-04-02T09:17:20.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.758+0000 7f6131600700 1 Processor -- start 2024-04-02T09:17:20.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.758+0000 7f6131600700 1 -- start start 2024-04-02T09:17:20.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.759+0000 7f6131600700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f612c0b56e0 0x7f612c151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.759+0000 7f6131600700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f612c0d1770 0x7f612c0c8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.759+0000 7f6131600700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f612c15d7f0 0x7f612c0b8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.759+0000 7f6131600700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0c9c90 con 0x7f612c0b56e0 2024-04-02T09:17:20.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.759+0000 7f6131600700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0c8d30 con 0x7f612c0d1770 2024-04-02T09:17:20.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.759+0000 7f6131600700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0c9010 con 0x7f612c15d7f0 2024-04-02T09:17:20.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.759+0000 7f612aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f612c0b56e0 0x7f612c151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.759+0000 7f612aa00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f612c0b56e0 0x7f612c151e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46404/0 (socket says 172.21.15.89:46404) 2024-04-02T09:17:20.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.759+0000 7f612aa00700 1 -- 172.21.15.89:0/2654567663 learned_addr learned my addr 172.21.15.89:0/2654567663 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:20.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f612a000700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f612c0d1770 0x7f612c0c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f612b400700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f612c15d7f0 0x7f612c0b8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f612aa00700 1 -- 172.21.15.89:0/2654567663 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f612c15d7f0 msgr2=0x7f612c0b8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f612aa00700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f612c15d7f0 0x7f612c0b8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f612aa00700 1 -- 172.21.15.89:0/2654567663 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f612c0d1770 msgr2=0x7f612c0c8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f612aa00700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f612c0d1770 0x7f612c0c8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f612aa00700 1 -- 172.21.15.89:0/2654567663 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6124009a70 con 0x7f612c0b56e0 2024-04-02T09:17:20.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f612a000700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f612c0d1770 0x7f612c0c8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:17:20.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f612b400700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f612c15d7f0 0x7f612c0b8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:17:20.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f612aa00700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f612c0b56e0 0x7f612c151e90 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7f611c00c520 tx=0x7f611c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:20.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f6122a00700 1 -- 172.21.15.89:0/2654567663 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c0052d0 con 0x7f612c0b56e0 2024-04-02T09:17:20.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f6122a00700 1 -- 172.21.15.89:0/2654567663 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611c005430 con 0x7f612c0b56e0 2024-04-02T09:17:20.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f6122a00700 1 -- 172.21.15.89:0/2654567663 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c0038c0 con 0x7f612c0b56e0 2024-04-02T09:17:20.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f6131600700 1 -- 172.21.15.89:0/2654567663 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612c0c92f0 con 0x7f612c0b56e0 2024-04-02T09:17:20.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.760+0000 7f6131600700 1 -- 172.21.15.89:0/2654567663 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612c0c98b0 con 0x7f612c0b56e0 2024-04-02T09:17:20.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.761+0000 7f6131600700 1 -- 172.21.15.89:0/2654567663 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f612c0ce800 con 0x7f612c0b56e0 2024-04-02T09:17:20.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.762+0000 7f6122a00700 1 -- 172.21.15.89:0/2654567663 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f611c003a20 con 0x7f612c0b56e0 2024-04-02T09:17:20.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.763+0000 7f6122a00700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f610c06adb0 0x7f610c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.763+0000 7f6122a00700 1 -- 172.21.15.89:0/2654567663 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f611c088b20 con 0x7f612c0b56e0 2024-04-02T09:17:20.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.763+0000 7f612a000700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f610c06adb0 0x7f610c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.763+0000 7f6122a00700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.113:6808/43940,v1:172.21.15.113:6809/43940] conn(0x7f610c071680 0x7f610c073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.763+0000 7f6122a00700 1 -- 172.21.15.89:0/2654567663 --> [v2:172.21.15.113:6808/43940,v1:172.21.15.113:6809/43940] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f610c0740f0 con 0x7f610c071680 2024-04-02T09:17:20.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.763+0000 7f6122a00700 1 -- 172.21.15.89:0/2654567663 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=284) v2 ==== 24+0+0 (secure 0 0 0) 0x7f611c08d2c0 con 0x7f612c0b56e0 2024-04-02T09:17:20.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.764+0000 7f612a000700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f610c06adb0 0x7f610c06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f612c160920 tx=0x7f6118009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:20.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.764+0000 7f612b400700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.113:6808/43940,v1:172.21.15.113:6809/43940] conn(0x7f610c071680 0x7f610c073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.765+0000 7f612b400700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.113:6808/43940,v1:172.21.15.113:6809/43940] conn(0x7f610c071680 0x7f610c073ac0 crc :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:20.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.766+0000 7f6122a00700 1 -- 172.21.15.89:0/2654567663 <== osd.4 v2:172.21.15.113:6808/43940 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f610c0740f0 con 0x7f610c071680 2024-04-02T09:17:20.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.780+0000 7f6131600700 1 -- 172.21.15.89:0/2654567663 --> [v2:172.21.15.113:6808/43940,v1:172.21.15.113:6809/43940] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f612c0ce520 con 0x7f610c071680 2024-04-02T09:17:20.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.781+0000 7f6122a00700 1 -- 172.21.15.89:0/2654567663 <== osd.4 v2:172.21.15.113:6808/43940 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6124009450 con 0x7f610c071680 2024-04-02T09:17:20.780 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:17:20.780 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T09:17:20.780 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:17:20.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.782+0000 7f6131600700 1 -- 172.21.15.89:0/2654567663 >> [v2:172.21.15.113:6808/43940,v1:172.21.15.113:6809/43940] conn(0x7f610c071680 msgr2=0x7f610c073ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.782+0000 7f6131600700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.113:6808/43940,v1:172.21.15.113:6809/43940] conn(0x7f610c071680 0x7f610c073ac0 crc :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.782+0000 7f6131600700 1 -- 172.21.15.89:0/2654567663 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f610c06adb0 msgr2=0x7f610c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.782+0000 7f6131600700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f610c06adb0 0x7f610c06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f612c160920 tx=0x7f6118009040 comp rx=0 tx=0).stop 2024-04-02T09:17:20.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.782+0000 7f6131600700 1 -- 172.21.15.89:0/2654567663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f612c0b56e0 msgr2=0x7f612c151e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.782+0000 7f6131600700 1 --2- 172.21.15.89:0/2654567663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f612c0b56e0 0x7f612c151e90 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7f611c00c520 tx=0x7f611c00c8e0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.782+0000 7f612b400700 1 -- 172.21.15.89:0/2654567663 reap_dead start 2024-04-02T09:17:20.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.782+0000 7f6131600700 1 -- 172.21.15.89:0/2654567663 shutdown_connections 2024-04-02T09:17:20.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.782+0000 7f6131600700 1 -- 172.21.15.89:0/2654567663 >> 172.21.15.89:0/2654567663 conn(0x7f612c08aff0 msgr2=0x7f612c0d34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:20.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.782+0000 7f6131600700 1 -- 172.21.15.89:0/2654567663 shutdown_connections 2024-04-02T09:17:20.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.782+0000 7f6131600700 1 -- 172.21.15.89:0/2654567663 wait complete. 2024-04-02T09:17:20.797 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-02T09:17:20.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.992+0000 7f09da800700 1 Processor -- start 2024-04-02T09:17:20.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.992+0000 7f09da800700 1 -- start start 2024-04-02T09:17:20.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.993+0000 7f09da800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09d40cd5c0 0x7f09d415e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.994+0000 7f09da800700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d415eda0 con 0x7f09d415cfe0 2024-04-02T09:17:20.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.994+0000 7f09da800700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d4099dd0 con 0x7f09d406f310 2024-04-02T09:17:20.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.994+0000 7f09da800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d409a0b0 con 0x7f09d40cd5c0 2024-04-02T09:17:20.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.994+0000 7f09d3e00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f09d415cfe0 0x7f09d415d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57934/0 (socket says 172.21.15.89:57934) 2024-04-02T09:17:20.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.994+0000 7f09d3e00700 1 -- 172.21.15.89:0/313351222 learned_addr learned my addr 172.21.15.89:0/313351222 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:20.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.994+0000 7f09d3400700 1 --2- 172.21.15.89:0/313351222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09d40cd5c0 0x7f09d415e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:20.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.994+0000 7f09d2000700 1 -- 172.21.15.89:0/313351222 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305165784 0 0) 0x7f09cc001490 con 0x7f09d415cfe0 2024-04-02T09:17:20.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.994+0000 7f09d2000700 1 -- 172.21.15.89:0/313351222 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b80035b0 con 0x7f09d415cfe0 2024-04-02T09:17:20.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.994+0000 7f09d3400700 1 -- 172.21.15.89:0/313351222 >> v1:172.21.15.113:6789/0 conn(0x7f09d406f310 legacy=0x7f09d406f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.994+0000 7f09d2000700 1 -- 172.21.15.89:0/313351222 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203361815 0 0) 0x7f09d4099dd0 con 0x7f09d406f310 2024-04-02T09:17:20.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.994+0000 7f09d3400700 1 -- 172.21.15.89:0/313351222 >> v1:172.21.15.89:6789/0 conn(0x7f09d415cfe0 legacy=0x7f09d415d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.994+0000 7f09d3400700 1 -- 172.21.15.89:0/313351222 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d409a390 con 0x7f09d40cd5c0 2024-04-02T09:17:20.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.995+0000 7f09d3400700 1 --2- 172.21.15.89:0/313351222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09d40cd5c0 0x7f09d415e860 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f09c4009870 tx=0x7f09c4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f851e385af36078 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:20.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.995+0000 7f09d2000700 1 -- 172.21.15.89:0/313351222 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c4011830 con 0x7f09d40cd5c0 2024-04-02T09:17:20.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.995+0000 7f09d2000700 1 -- 172.21.15.89:0/313351222 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c4011990 con 0x7f09d40cd5c0 2024-04-02T09:17:20.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.995+0000 7f09d2000700 1 -- 172.21.15.89:0/313351222 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c4011d20 con 0x7f09d40cd5c0 2024-04-02T09:17:20.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.996+0000 7f09da800700 1 -- 172.21.15.89:0/313351222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09d40cd5c0 msgr2=0x7f09d415e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:20.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.996+0000 7f09da800700 1 --2- 172.21.15.89:0/313351222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09d40cd5c0 0x7f09d415e860 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f09c4009870 tx=0x7f09c4009b80 comp rx=0 tx=0).stop 2024-04-02T09:17:20.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.996+0000 7f09da800700 1 -- 172.21.15.89:0/313351222 shutdown_connections 2024-04-02T09:17:20.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.996+0000 7f09da800700 1 --2- 172.21.15.89:0/313351222 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09d40cd5c0 0x7f09d415e860 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:20.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.996+0000 7f09da800700 1 -- 172.21.15.89:0/313351222 >> 172.21.15.89:0/313351222 conn(0x7f09d40d1370 msgr2=0x7f09d40d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:20.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.996+0000 7f09da800700 1 -- 172.21.15.89:0/313351222 shutdown_connections 2024-04-02T09:17:20.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.996+0000 7f09da800700 1 -- 172.21.15.89:0/313351222 wait complete. 2024-04-02T09:17:20.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.997+0000 7f09da800700 1 Processor -- start 2024-04-02T09:17:20.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.998+0000 7f09da800700 1 -- start start 2024-04-02T09:17:20.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.998+0000 7f09da800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09d406f310 0x7f09d40c6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09da800700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d40cd5c0 0x7f09d4156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09da800700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09d415cfe0 0x7f09d40b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:20.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09da800700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40c6910 con 0x7f09d406f310 2024-04-02T09:17:20.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09da800700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40c6650 con 0x7f09d40cd5c0 2024-04-02T09:17:20.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09da800700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40b12f0 con 0x7f09d415cfe0 2024-04-02T09:17:20.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09d3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09d406f310 0x7f09d40c6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09d3400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09d406f310 0x7f09d40c6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46406/0 (socket says 172.21.15.89:46406) 2024-04-02T09:17:21.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09d3400700 1 -- 172.21.15.89:0/3180515358 learned_addr learned my addr 172.21.15.89:0/3180515358 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:21.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09d3e00700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09d415cfe0 0x7f09d40b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09d2a00700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d40cd5c0 0x7f09d4156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09d3400700 1 -- 172.21.15.89:0/3180515358 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09d415cfe0 msgr2=0x7f09d40b60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09d3400700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09d415cfe0 0x7f09d40b60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09d3400700 1 -- 172.21.15.89:0/3180515358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d40cd5c0 msgr2=0x7f09d4156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09d3400700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d40cd5c0 0x7f09d4156f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09d3400700 1 -- 172.21.15.89:0/3180515358 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c40094a0 con 0x7f09d406f310 2024-04-02T09:17:21.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09d2a00700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d40cd5c0 0x7f09d4156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:17:21.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:20.999+0000 7f09d3400700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09d406f310 0x7f09d40c6110 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7f09c4009bc0 tx=0x7f09c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:21.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.000+0000 7f09cb400700 1 -- 172.21.15.89:0/3180515358 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c401a910 con 0x7f09d406f310 2024-04-02T09:17:21.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.000+0000 7f09cb400700 1 -- 172.21.15.89:0/3180515358 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c401aa70 con 0x7f09d406f310 2024-04-02T09:17:21.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.000+0000 7f09da800700 1 -- 172.21.15.89:0/3180515358 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d40b5300 con 0x7f09d406f310 2024-04-02T09:17:21.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.000+0000 7f09cb400700 1 -- 172.21.15.89:0/3180515358 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c401ae00 con 0x7f09d406f310 2024-04-02T09:17:21.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.000+0000 7f09da800700 1 -- 172.21.15.89:0/3180515358 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d40b58c0 con 0x7f09d406f310 2024-04-02T09:17:21.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.001+0000 7f09da800700 1 -- 172.21.15.89:0/3180515358 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d4067720 con 0x7f09d406f310 2024-04-02T09:17:21.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.002+0000 7f09cb400700 1 -- 172.21.15.89:0/3180515358 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f09c4022070 con 0x7f09d406f310 2024-04-02T09:17:21.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.002+0000 7f09cb400700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f09b406adb0 0x7f09b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.003+0000 7f09cb400700 1 -- 172.21.15.89:0/3180515358 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f09c408a7b0 con 0x7f09d406f310 2024-04-02T09:17:21.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.003+0000 7f09d2a00700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f09b406adb0 0x7f09b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.003+0000 7f09d2a00700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f09b406adb0 0x7f09b406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f09d4157e20 tx=0x7f09c0009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:21.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.007+0000 7f09cb400700 1 -- 172.21.15.89:0/3180515358 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09c40556e0 con 0x7f09d406f310 2024-04-02T09:17:21.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.205+0000 7f09da800700 1 -- 172.21.15.89:0/3180515358 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f09d4099a60 con 0x7f09b406adb0 2024-04-02T09:17:21.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.205+0000 7f09cb400700 1 -- 172.21.15.89:0/3180515358 <== mgr.13575 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f09c00014d0 con 0x7f09b406adb0 2024-04-02T09:17:21.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.208+0000 7f09da800700 1 -- 172.21.15.89:0/3180515358 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f09b406adb0 msgr2=0x7f09b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.208+0000 7f09da800700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f09b406adb0 0x7f09b406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f09d4157e20 tx=0x7f09c0009040 comp rx=0 tx=0).stop 2024-04-02T09:17:21.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.208+0000 7f09da800700 1 -- 172.21.15.89:0/3180515358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09d406f310 msgr2=0x7f09d40c6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.208+0000 7f09da800700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09d406f310 0x7f09d40c6110 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7f09c4009bc0 tx=0x7f09c4004750 comp rx=0 tx=0).stop 2024-04-02T09:17:21.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.208+0000 7f09da800700 1 -- 172.21.15.89:0/3180515358 shutdown_connections 2024-04-02T09:17:21.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.208+0000 7f09da800700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09d415cfe0 0x7f09d40b60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.208+0000 7f09da800700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09d406f310 0x7f09d40c6110 unknown :-1 s=CLOSED pgs=3123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.208+0000 7f09da800700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f09b406adb0 0x7f09b406d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.208+0000 7f09da800700 1 --2- 172.21.15.89:0/3180515358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d40cd5c0 0x7f09d4156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.208+0000 7f09da800700 1 -- 172.21.15.89:0/3180515358 >> 172.21.15.89:0/3180515358 conn(0x7f09d40d1370 msgr2=0x7f09d406cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:21.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.208+0000 7f09da800700 1 -- 172.21.15.89:0/3180515358 shutdown_connections 2024-04-02T09:17:21.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.208+0000 7f09da800700 1 -- 172.21.15.89:0/3180515358 wait complete. 2024-04-02T09:17:21.208 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 4 to deep-scrub 2024-04-02T09:17:21.226 INFO:tasks.ceph:Scrubbing osd.5 2024-04-02T09:17:21.226 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-02T09:17:21.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.423+0000 7fd59dc00700 1 Processor -- start 2024-04-02T09:17:21.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.423+0000 7fd59dc00700 1 -- start start 2024-04-02T09:17:21.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.424+0000 7fd59dc00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd5980b5580 0x7fd59815e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.424+0000 7fd59dc00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd59815f260 0x7fd5980d1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.424+0000 7fd59dc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd59815f6c0 0x7fd5980d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.425+0000 7fd59dc00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd59815ee70 con 0x7fd5980b5580 2024-04-02T09:17:21.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.425+0000 7fd59dc00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5980ce430 con 0x7fd59815f260 2024-04-02T09:17:21.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.425+0000 7fd59dc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5980ce710 con 0x7fd59815f6c0 2024-04-02T09:17:21.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.425+0000 7fd596a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd5980b5580 0x7fd59815e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.425+0000 7fd596a00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd5980b5580 0x7fd59815e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46420/0 (socket says 172.21.15.89:46420) 2024-04-02T09:17:21.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.425+0000 7fd596a00700 1 -- 172.21.15.89:0/1406086996 learned_addr learned my addr 172.21.15.89:0/1406086996 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:21.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.425+0000 7fd596000700 1 --2- 172.21.15.89:0/1406086996 >> v2:172.21.15.113:3300/0 conn(0x7fd59815f260 0x7fd5980d1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.425+0000 7fd597400700 1 --2- 172.21.15.89:0/1406086996 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd59815f6c0 0x7fd5980d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.425+0000 7fd596a00700 1 -- 172.21.15.89:0/1406086996 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd59815f6c0 msgr2=0x7fd5980d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.425+0000 7fd596a00700 1 --2- 172.21.15.89:0/1406086996 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd59815f6c0 0x7fd5980d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.425+0000 7fd596a00700 1 -- 172.21.15.89:0/1406086996 >> v2:172.21.15.113:3300/0 conn(0x7fd59815f260 msgr2=0x7fd5980d1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.425+0000 7fd596a00700 1 --2- 172.21.15.89:0/1406086996 >> v2:172.21.15.113:3300/0 conn(0x7fd59815f260 0x7fd5980d1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.425+0000 7fd596a00700 1 -- 172.21.15.89:0/1406086996 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5980ce9f0 con 0x7fd5980b5580 2024-04-02T09:17:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.425+0000 7fd596a00700 1 --2- 172.21.15.89:0/1406086996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5980b5580 0x7fd59815e930 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 crypto rx=0x7fd59808d1e0 tx=0x7fd588009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6219185732ebcd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:21.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.426+0000 7fd595600700 1 -- 172.21.15.89:0/1406086996 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58800b870 con 0x7fd5980b5580 2024-04-02T09:17:21.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.426+0000 7fd595600700 1 -- 172.21.15.89:0/1406086996 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58800b9d0 con 0x7fd5980b5580 2024-04-02T09:17:21.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.426+0000 7fd595600700 1 -- 172.21.15.89:0/1406086996 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58800bd60 con 0x7fd5980b5580 2024-04-02T09:17:21.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.426+0000 7fd59dc00700 1 -- 172.21.15.89:0/1406086996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5980b5580 msgr2=0x7fd59815e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.426+0000 7fd59dc00700 1 --2- 172.21.15.89:0/1406086996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5980b5580 0x7fd59815e930 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 crypto rx=0x7fd59808d1e0 tx=0x7fd588009bd0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.426+0000 7fd59dc00700 1 -- 172.21.15.89:0/1406086996 shutdown_connections 2024-04-02T09:17:21.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.426+0000 7fd59dc00700 1 --2- 172.21.15.89:0/1406086996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5980b5580 0x7fd59815e930 unknown :-1 s=CLOSED pgs=3124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.426+0000 7fd59dc00700 1 --2- 172.21.15.89:0/1406086996 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd59815f6c0 0x7fd5980d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.426+0000 7fd59dc00700 1 --2- 172.21.15.89:0/1406086996 >> v2:172.21.15.113:3300/0 conn(0x7fd59815f260 0x7fd5980d1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.426+0000 7fd59dc00700 1 -- 172.21.15.89:0/1406086996 >> 172.21.15.89:0/1406086996 conn(0x7fd59808aff0 msgr2=0x7fd5980d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:21.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.426+0000 7fd59dc00700 1 -- 172.21.15.89:0/1406086996 shutdown_connections 2024-04-02T09:17:21.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.426+0000 7fd59dc00700 1 -- 172.21.15.89:0/1406086996 wait complete. 2024-04-02T09:17:21.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.429+0000 7fd59dc00700 1 Processor -- start 2024-04-02T09:17:21.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.429+0000 7fd59dc00700 1 -- start start 2024-04-02T09:17:21.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.430+0000 7fd59dc00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5980b5580 0x7fd59815dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.430+0000 7fd59dc00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd59815f260 0x7fd5980d1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.430+0000 7fd59dc00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd59815f6c0 0x7fd5980d0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.430+0000 7fd59dc00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5980c7590 con 0x7fd59815f260 2024-04-02T09:17:21.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.430+0000 7fd59dc00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5980cd690 con 0x7fd5980b5580 2024-04-02T09:17:21.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.430+0000 7fd59dc00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5980c71f0 con 0x7fd59815f6c0 2024-04-02T09:17:21.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.430+0000 7fd596000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd59815f260 0x7fd5980d1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.430+0000 7fd596000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd59815f260 0x7fd5980d1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46426/0 (socket says 172.21.15.89:46426) 2024-04-02T09:17:21.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.430+0000 7fd596000700 1 -- 172.21.15.89:0/243083264 learned_addr learned my addr 172.21.15.89:0/243083264 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:21.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.430+0000 7fd596a00700 1 --2- 172.21.15.89:0/243083264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5980b5580 0x7fd59815dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.430+0000 7fd597400700 1 --2- 172.21.15.89:0/243083264 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd59815f6c0 0x7fd5980d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.431+0000 7fd596000700 1 -- 172.21.15.89:0/243083264 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd59815f6c0 msgr2=0x7fd5980d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.431+0000 7fd596000700 1 --2- 172.21.15.89:0/243083264 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd59815f6c0 0x7fd5980d0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.431+0000 7fd596000700 1 -- 172.21.15.89:0/243083264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5980b5580 msgr2=0x7fd59815dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.431+0000 7fd596000700 1 --2- 172.21.15.89:0/243083264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5980b5580 0x7fd59815dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.431+0000 7fd596000700 1 -- 172.21.15.89:0/243083264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd588009550 con 0x7fd59815f260 2024-04-02T09:17:21.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.431+0000 7fd596000700 1 --2- 172.21.15.89:0/243083264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd59815f260 0x7fd5980d1a70 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7fd58400e840 tx=0x7fd58400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:21.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.431+0000 7fd58ea00700 1 -- 172.21.15.89:0/243083264 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58400c070 con 0x7fd59815f260 2024-04-02T09:17:21.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.431+0000 7fd58ea00700 1 -- 172.21.15.89:0/243083264 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd584013610 con 0x7fd59815f260 2024-04-02T09:17:21.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.431+0000 7fd58ea00700 1 -- 172.21.15.89:0/243083264 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5840139a0 con 0x7fd59815f260 2024-04-02T09:17:21.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.431+0000 7fd59dc00700 1 -- 172.21.15.89:0/243083264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5980d1fb0 con 0x7fd59815f260 2024-04-02T09:17:21.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.431+0000 7fd59dc00700 1 -- 172.21.15.89:0/243083264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5980cd3b0 con 0x7fd59815f260 2024-04-02T09:17:21.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.432+0000 7fd59dc00700 1 -- 172.21.15.89:0/243083264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd5980b5960 con 0x7fd59815f260 2024-04-02T09:17:21.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.433+0000 7fd58ea00700 1 -- 172.21.15.89:0/243083264 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd584013b00 con 0x7fd59815f260 2024-04-02T09:17:21.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.434+0000 7fd58ea00700 1 --2- 172.21.15.89:0/243083264 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd57806adb0 0x7fd57806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.434+0000 7fd58ea00700 1 -- 172.21.15.89:0/243083264 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7fd5840893a0 con 0x7fd59815f260 2024-04-02T09:17:21.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.434+0000 7fd596a00700 1 --2- 172.21.15.89:0/243083264 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd57806adb0 0x7fd57806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.434+0000 7fd58ea00700 1 --2- 172.21.15.89:0/243083264 >> [v2:172.21.15.113:6800/43939,v1:172.21.15.113:6801/43939] conn(0x7fd578071680 0x7fd578073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.434+0000 7fd58ea00700 1 -- 172.21.15.89:0/243083264 --> [v2:172.21.15.113:6800/43939,v1:172.21.15.113:6801/43939] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5780740f0 con 0x7fd578071680 2024-04-02T09:17:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.434+0000 7fd58ea00700 1 -- 172.21.15.89:0/243083264 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=284) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd58408ba60 con 0x7fd59815f260 2024-04-02T09:17:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.435+0000 7fd597400700 1 --2- 172.21.15.89:0/243083264 >> [v2:172.21.15.113:6800/43939,v1:172.21.15.113:6801/43939] conn(0x7fd578071680 0x7fd578073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.435+0000 7fd596a00700 1 --2- 172.21.15.89:0/243083264 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd57806adb0 0x7fd57806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd588000c00 tx=0x7fd588008030 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.435+0000 7fd597400700 1 --2- 172.21.15.89:0/243083264 >> [v2:172.21.15.113:6800/43939,v1:172.21.15.113:6801/43939] conn(0x7fd578071680 0x7fd578073ac0 crc :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:21.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.436+0000 7fd58ea00700 1 -- 172.21.15.89:0/243083264 <== osd.5 v2:172.21.15.113:6800/43939 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd5780740f0 con 0x7fd578071680 2024-04-02T09:17:21.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.451+0000 7fd59dc00700 1 -- 172.21.15.89:0/243083264 --> [v2:172.21.15.113:6800/43939,v1:172.21.15.113:6801/43939] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd5980cefb0 con 0x7fd578071680 2024-04-02T09:17:21.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.452+0000 7fd58ea00700 1 -- 172.21.15.89:0/243083264 <== osd.5 v2:172.21.15.113:6800/43939 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd5900083a0 con 0x7fd578071680 2024-04-02T09:17:21.450 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:17:21.450 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T09:17:21.450 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:17:21.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.452+0000 7fd59dc00700 1 -- 172.21.15.89:0/243083264 >> [v2:172.21.15.113:6800/43939,v1:172.21.15.113:6801/43939] conn(0x7fd578071680 msgr2=0x7fd578073ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.452+0000 7fd59dc00700 1 --2- 172.21.15.89:0/243083264 >> [v2:172.21.15.113:6800/43939,v1:172.21.15.113:6801/43939] conn(0x7fd578071680 0x7fd578073ac0 crc :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.452+0000 7fd59dc00700 1 -- 172.21.15.89:0/243083264 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd57806adb0 msgr2=0x7fd57806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.452+0000 7fd59dc00700 1 --2- 172.21.15.89:0/243083264 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd57806adb0 0x7fd57806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd588000c00 tx=0x7fd588008030 comp rx=0 tx=0).stop 2024-04-02T09:17:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.452+0000 7fd59dc00700 1 -- 172.21.15.89:0/243083264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd59815f260 msgr2=0x7fd5980d1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.452+0000 7fd59dc00700 1 --2- 172.21.15.89:0/243083264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd59815f260 0x7fd5980d1a70 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7fd58400e840 tx=0x7fd58400eb50 comp rx=0 tx=0).stop 2024-04-02T09:17:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.452+0000 7fd597400700 1 -- 172.21.15.89:0/243083264 reap_dead start 2024-04-02T09:17:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.453+0000 7fd59dc00700 1 -- 172.21.15.89:0/243083264 shutdown_connections 2024-04-02T09:17:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.453+0000 7fd59dc00700 1 -- 172.21.15.89:0/243083264 >> 172.21.15.89:0/243083264 conn(0x7fd59808aff0 msgr2=0x7fd5980b2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.453+0000 7fd59dc00700 1 -- 172.21.15.89:0/243083264 shutdown_connections 2024-04-02T09:17:21.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.453+0000 7fd59dc00700 1 -- 172.21.15.89:0/243083264 wait complete. 2024-04-02T09:17:21.468 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-02T09:17:21.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.662+0000 7fe508400700 1 Processor -- start 2024-04-02T09:17:21.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.663+0000 7fe508400700 1 -- start start 2024-04-02T09:17:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.664+0000 7fe508400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe5000cd020 0x7fe5000cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.664+0000 7fe508400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe5000b2070 0x7fe5000b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.664+0000 7fe508400700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5000cdae0 con 0x7fe5000cbb30 2024-04-02T09:17:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.664+0000 7fe508400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5000cddc0 con 0x7fe5000b2070 2024-04-02T09:17:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.664+0000 7fe508400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5000ce0a0 con 0x7fe5000cd020 2024-04-02T09:17:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.664+0000 7fe506400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe5000b2070 0x7fe5000b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.664+0000 7fe506400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe5000b2070 0x7fe5000b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46442/0 (socket says 172.21.15.89:46442) 2024-04-02T09:17:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.664+0000 7fe506400700 1 -- 172.21.15.89:0/2270913318 learned_addr learned my addr 172.21.15.89:0/2270913318 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.664+0000 7fe505000700 1 --2- 172.21.15.89:0/2270913318 >> v2:172.21.15.113:3300/0 conn(0x7fe5000cd020 0x7fe5000cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.664+0000 7fe506400700 1 -- 172.21.15.89:0/2270913318 >> v2:172.21.15.113:3300/0 conn(0x7fe5000cd020 msgr2=0x7fe5000cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.664+0000 7fe506400700 1 --2- 172.21.15.89:0/2270913318 >> v2:172.21.15.113:3300/0 conn(0x7fe5000cd020 0x7fe5000cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.664+0000 7fe506400700 1 -- 172.21.15.89:0/2270913318 >> v1:172.21.15.113:6789/0 conn(0x7fe5000cbb30 legacy=0x7fe5000cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.664+0000 7fe506400700 1 -- 172.21.15.89:0/2270913318 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50006dc20 con 0x7fe5000b2070 2024-04-02T09:17:21.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.664+0000 7fe505000700 1 --2- 172.21.15.89:0/2270913318 >> v2:172.21.15.113:3300/0 conn(0x7fe5000cd020 0x7fe5000cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:17:21.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.665+0000 7fe506400700 1 --2- 172.21.15.89:0/2270913318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5000b2070 0x7fe5000b2500 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7fe5000b1e30 tx=0x7fe4f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1337da91270e16 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:21.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.665+0000 7fe4ffe00700 1 -- 172.21.15.89:0/2270913318 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f800e070 con 0x7fe5000b2070 2024-04-02T09:17:21.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.665+0000 7fe4ffe00700 1 -- 172.21.15.89:0/2270913318 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f800b9c0 con 0x7fe5000b2070 2024-04-02T09:17:21.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.665+0000 7fe4ffe00700 1 -- 172.21.15.89:0/2270913318 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f800bd70 con 0x7fe5000b2070 2024-04-02T09:17:21.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.665+0000 7fe508400700 1 -- 172.21.15.89:0/2270913318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5000b2070 msgr2=0x7fe5000b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.665+0000 7fe508400700 1 --2- 172.21.15.89:0/2270913318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5000b2070 0x7fe5000b2500 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7fe5000b1e30 tx=0x7fe4f800b330 comp rx=0 tx=0).stop 2024-04-02T09:17:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.666+0000 7fe508400700 1 -- 172.21.15.89:0/2270913318 shutdown_connections 2024-04-02T09:17:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.666+0000 7fe508400700 1 --2- 172.21.15.89:0/2270913318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5000b2070 0x7fe5000b2500 unknown :-1 s=CLOSED pgs=3126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.666+0000 7fe508400700 1 --2- 172.21.15.89:0/2270913318 >> v2:172.21.15.113:3300/0 conn(0x7fe5000cd020 0x7fe5000cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.666+0000 7fe508400700 1 -- 172.21.15.89:0/2270913318 >> 172.21.15.89:0/2270913318 conn(0x7fe5000b0d70 msgr2=0x7fe5000b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.666+0000 7fe508400700 1 -- 172.21.15.89:0/2270913318 shutdown_connections 2024-04-02T09:17:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.666+0000 7fe508400700 1 -- 172.21.15.89:0/2270913318 wait complete. 2024-04-02T09:17:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.667+0000 7fe508400700 1 Processor -- start 2024-04-02T09:17:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.668+0000 7fe508400700 1 -- start start 2024-04-02T09:17:21.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.668+0000 7fe508400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe5000b2070 0x7fe500159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.668+0000 7fe508400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5000cbb30 0x7fe50014c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.668+0000 7fe508400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5000cd020 0x7fe5000cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.668+0000 7fe508400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe500153a60 con 0x7fe5000cbb30 2024-04-02T09:17:21.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.668+0000 7fe508400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50015a930 con 0x7fe5000cd020 2024-04-02T09:17:21.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.668+0000 7fe508400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe50014cb60 con 0x7fe5000b2070 2024-04-02T09:17:21.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.669+0000 7fe505000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5000cbb30 0x7fe50014c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.669+0000 7fe505000700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5000cbb30 0x7fe50014c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46446/0 (socket says 172.21.15.89:46446) 2024-04-02T09:17:21.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.669+0000 7fe505000700 1 -- 172.21.15.89:0/3514914305 learned_addr learned my addr 172.21.15.89:0/3514914305 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:21.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.669+0000 7fe505a00700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe5000b2070 0x7fe500159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.669+0000 7fe506400700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5000cd020 0x7fe5000cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.669+0000 7fe505000700 1 -- 172.21.15.89:0/3514914305 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe5000b2070 msgr2=0x7fe500159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.669+0000 7fe505000700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe5000b2070 0x7fe500159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.669+0000 7fe505000700 1 -- 172.21.15.89:0/3514914305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5000cd020 msgr2=0x7fe5000cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.669+0000 7fe505000700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5000cd020 0x7fe5000cacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.669+0000 7fe505000700 1 -- 172.21.15.89:0/3514914305 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f8009b30 con 0x7fe5000cbb30 2024-04-02T09:17:21.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.669+0000 7fe506400700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5000cd020 0x7fe5000cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:17:21.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.669+0000 7fe505000700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5000cbb30 0x7fe50014c620 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00e790 tx=0x7fe4ec00eaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:21.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.669+0000 7fe4fd600700 1 -- 172.21.15.89:0/3514914305 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec013610 con 0x7fe5000cbb30 2024-04-02T09:17:21.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.670+0000 7fe4fd600700 1 -- 172.21.15.89:0/3514914305 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec013770 con 0x7fe5000cbb30 2024-04-02T09:17:21.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.670+0000 7fe4fd600700 1 -- 172.21.15.89:0/3514914305 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec013b20 con 0x7fe5000cbb30 2024-04-02T09:17:21.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.670+0000 7fe508400700 1 -- 172.21.15.89:0/3514914305 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5000cb210 con 0x7fe5000cbb30 2024-04-02T09:17:21.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.670+0000 7fe508400700 1 -- 172.21.15.89:0/3514914305 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe50015a000 con 0x7fe5000cbb30 2024-04-02T09:17:21.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.671+0000 7fe508400700 1 -- 172.21.15.89:0/3514914305 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe500067720 con 0x7fe5000cbb30 2024-04-02T09:17:21.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.672+0000 7fe4fd600700 1 -- 172.21.15.89:0/3514914305 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe4ec013c80 con 0x7fe5000cbb30 2024-04-02T09:17:21.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.672+0000 7fe4fd600700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe4e006ade0 0x7fe4e006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:21.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.672+0000 7fe4fd600700 1 -- 172.21.15.89:0/3514914305 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7fe4ec0892e0 con 0x7fe5000cbb30 2024-04-02T09:17:21.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.673+0000 7fe505a00700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe4e006ade0 0x7fe4e006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:21.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.673+0000 7fe505a00700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe4e006ade0 0x7fe4e006d2a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4007fd0 tx=0x7fe4f4009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:21.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.677+0000 7fe4fd600700 1 -- 172.21.15.89:0/3514914305 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe4ec054060 con 0x7fe5000cbb30 2024-04-02T09:17:21.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.880+0000 7fe508400700 1 -- 172.21.15.89:0/3514914305 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fe50015bdd0 con 0x7fe4e006ade0 2024-04-02T09:17:21.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.881+0000 7fe4fd600700 1 -- 172.21.15.89:0/3514914305 <== mgr.13575 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe4f4001560 con 0x7fe4e006ade0 2024-04-02T09:17:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.883+0000 7fe508400700 1 -- 172.21.15.89:0/3514914305 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe4e006ade0 msgr2=0x7fe4e006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.883+0000 7fe508400700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe4e006ade0 0x7fe4e006d2a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4007fd0 tx=0x7fe4f4009040 comp rx=0 tx=0).stop 2024-04-02T09:17:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.883+0000 7fe508400700 1 -- 172.21.15.89:0/3514914305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5000cbb30 msgr2=0x7fe50014c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.883+0000 7fe508400700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5000cbb30 0x7fe50014c620 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00e790 tx=0x7fe4ec00eaa0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.883+0000 7fe508400700 1 -- 172.21.15.89:0/3514914305 shutdown_connections 2024-04-02T09:17:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.883+0000 7fe508400700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fe4e006ade0 0x7fe4e006d2a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.883+0000 7fe508400700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5000cd020 0x7fe5000cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.883+0000 7fe508400700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe5000b2070 0x7fe500159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.883+0000 7fe508400700 1 --2- 172.21.15.89:0/3514914305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5000cbb30 0x7fe50014c620 unknown :-1 s=CLOSED pgs=3127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.883+0000 7fe508400700 1 -- 172.21.15.89:0/3514914305 >> 172.21.15.89:0/3514914305 conn(0x7fe5000b0d70 msgr2=0x7fe5000afdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.883+0000 7fe508400700 1 -- 172.21.15.89:0/3514914305 shutdown_connections 2024-04-02T09:17:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:21.883+0000 7fe508400700 1 -- 172.21.15.89:0/3514914305 wait complete. 2024-04-02T09:17:21.884 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 5 to deep-scrub 2024-04-02T09:17:21.899 INFO:tasks.ceph:Scrubbing osd.6 2024-04-02T09:17:21.899 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-02T09:17:22.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.095+0000 7fd26b400700 1 Processor -- start 2024-04-02T09:17:22.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.095+0000 7fd26b400700 1 -- start start 2024-04-02T09:17:22.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.096+0000 7fd26b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd264071340 0x7fd2640b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.096+0000 7fd26b400700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd264159d10 0x7fd264071720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.096+0000 7fd26b400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd264070980 0x7fd26406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.096+0000 7fd26b400700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2640b2ff0 con 0x7fd264070980 2024-04-02T09:17:22.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.096+0000 7fd26b400700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2640b3e50 con 0x7fd264159d10 2024-04-02T09:17:22.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.096+0000 7fd26b400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd264071c60 con 0x7fd264071340 2024-04-02T09:17:22.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.096+0000 7fd269400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd264070980 0x7fd26406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.096+0000 7fd269400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd264070980 0x7fd26406d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46460/0 (socket says 172.21.15.89:46460) 2024-04-02T09:17:22.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.096+0000 7fd269400700 1 -- 172.21.15.89:0/1167108697 learned_addr learned my addr 172.21.15.89:0/1167108697 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:22.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.097+0000 7fd268a00700 1 --2- 172.21.15.89:0/1167108697 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd264071340 0x7fd2640b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.097+0000 7fd263e00700 1 --2- 172.21.15.89:0/1167108697 >> v2:172.21.15.113:3300/0 conn(0x7fd264159d10 0x7fd264071720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.097+0000 7fd269400700 1 -- 172.21.15.89:0/1167108697 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd264071340 msgr2=0x7fd2640b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:22.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.097+0000 7fd269400700 1 --2- 172.21.15.89:0/1167108697 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd264071340 0x7fd2640b42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.097+0000 7fd269400700 1 -- 172.21.15.89:0/1167108697 >> v2:172.21.15.113:3300/0 conn(0x7fd264159d10 msgr2=0x7fd264071720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:22.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.097+0000 7fd269400700 1 --2- 172.21.15.89:0/1167108697 >> v2:172.21.15.113:3300/0 conn(0x7fd264159d10 0x7fd264071720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.097+0000 7fd269400700 1 -- 172.21.15.89:0/1167108697 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2640b3240 con 0x7fd264070980 2024-04-02T09:17:22.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.097+0000 7fd269400700 1 --2- 172.21.15.89:0/1167108697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd264070980 0x7fd26406d1a0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7fd26406ed30 tx=0x7fd25c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3480cc11185abf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:22.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.097+0000 7fd263400700 1 -- 172.21.15.89:0/1167108697 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25c00bbc0 con 0x7fd264070980 2024-04-02T09:17:22.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.097+0000 7fd263400700 1 -- 172.21.15.89:0/1167108697 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25c00bd20 con 0x7fd264070980 2024-04-02T09:17:22.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.097+0000 7fd263400700 1 -- 172.21.15.89:0/1167108697 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25c0106c0 con 0x7fd264070980 2024-04-02T09:17:22.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.098+0000 7fd26b400700 1 -- 172.21.15.89:0/1167108697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd264070980 msgr2=0x7fd26406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.098+0000 7fd26b400700 1 --2- 172.21.15.89:0/1167108697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd264070980 0x7fd26406d1a0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7fd26406ed30 tx=0x7fd25c00b330 comp rx=0 tx=0).stop 2024-04-02T09:17:22.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.098+0000 7fd26b400700 1 -- 172.21.15.89:0/1167108697 shutdown_connections 2024-04-02T09:17:22.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.098+0000 7fd26b400700 1 --2- 172.21.15.89:0/1167108697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd264070980 0x7fd26406d1a0 unknown :-1 s=CLOSED pgs=3128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.098+0000 7fd26b400700 1 --2- 172.21.15.89:0/1167108697 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd264071340 0x7fd2640b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.098+0000 7fd26b400700 1 --2- 172.21.15.89:0/1167108697 >> v2:172.21.15.113:3300/0 conn(0x7fd264159d10 0x7fd264071720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.098+0000 7fd26b400700 1 -- 172.21.15.89:0/1167108697 >> 172.21.15.89:0/1167108697 conn(0x7fd2640cf210 msgr2=0x7fd2640cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:22.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.098+0000 7fd26b400700 1 -- 172.21.15.89:0/1167108697 shutdown_connections 2024-04-02T09:17:22.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.098+0000 7fd26b400700 1 -- 172.21.15.89:0/1167108697 wait complete. 2024-04-02T09:17:22.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.100+0000 7fd26b400700 1 Processor -- start 2024-04-02T09:17:22.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.100+0000 7fd26b400700 1 -- start start 2024-04-02T09:17:22.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd26b400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd264070980 0x7fd2640ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd26b400700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd264071340 0x7fd2641423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd26b400700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd264159d10 0x7fd264142910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd26b400700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2640cdd90 con 0x7fd264070980 2024-04-02T09:17:22.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd26b400700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2640cd110 con 0x7fd264071340 2024-04-02T09:17:22.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd26b400700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2640cd3f0 con 0x7fd264159d10 2024-04-02T09:17:22.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd268a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd264070980 0x7fd2640ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd268a00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd264070980 0x7fd2640ce1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46464/0 (socket says 172.21.15.89:46464) 2024-04-02T09:17:22.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd268a00700 1 -- 172.21.15.89:0/3710077018 learned_addr learned my addr 172.21.15.89:0/3710077018 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:22.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd263e00700 1 --2- 172.21.15.89:0/3710077018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd264071340 0x7fd2641423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd269400700 1 --2- 172.21.15.89:0/3710077018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd264159d10 0x7fd264142910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd268a00700 1 -- 172.21.15.89:0/3710077018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd264159d10 msgr2=0x7fd264142910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd268a00700 1 --2- 172.21.15.89:0/3710077018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd264159d10 0x7fd264142910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd268a00700 1 -- 172.21.15.89:0/3710077018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd264071340 msgr2=0x7fd2641423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd268a00700 1 --2- 172.21.15.89:0/3710077018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd264071340 0x7fd2641423d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.101+0000 7fd268a00700 1 -- 172.21.15.89:0/3710077018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c009b30 con 0x7fd264070980 2024-04-02T09:17:22.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.102+0000 7fd268a00700 1 --2- 172.21.15.89:0/3710077018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd264070980 0x7fd2640ce1b0 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 crypto rx=0x7fd25800e810 tx=0x7fd25800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:22.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.102+0000 7fd260c00700 1 -- 172.21.15.89:0/3710077018 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258013610 con 0x7fd264070980 2024-04-02T09:17:22.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.102+0000 7fd260c00700 1 -- 172.21.15.89:0/3710077018 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd258013770 con 0x7fd264070980 2024-04-02T09:17:22.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.102+0000 7fd26b400700 1 -- 172.21.15.89:0/3710077018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2640cd6d0 con 0x7fd264070980 2024-04-02T09:17:22.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.102+0000 7fd260c00700 1 -- 172.21.15.89:0/3710077018 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258013b20 con 0x7fd264070980 2024-04-02T09:17:22.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.102+0000 7fd26b400700 1 -- 172.21.15.89:0/3710077018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd264155cb0 con 0x7fd264070980 2024-04-02T09:17:22.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.103+0000 7fd26b400700 1 -- 172.21.15.89:0/3710077018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2640b3ae0 con 0x7fd264070980 2024-04-02T09:17:22.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.104+0000 7fd260c00700 1 -- 172.21.15.89:0/3710077018 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd258013c80 con 0x7fd264070980 2024-04-02T09:17:22.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.105+0000 7fd260c00700 1 --2- 172.21.15.89:0/3710077018 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd24406adb0 0x7fd24406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.105+0000 7fd260c00700 1 -- 172.21.15.89:0/3710077018 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7fd2580893c0 con 0x7fd264070980 2024-04-02T09:17:22.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.105+0000 7fd263e00700 1 --2- 172.21.15.89:0/3710077018 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd24406adb0 0x7fd24406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.105+0000 7fd260c00700 1 --2- 172.21.15.89:0/3710077018 >> [v2:172.21.15.113:6824/43958,v1:172.21.15.113:6825/43958] conn(0x7fd244071680 0x7fd244073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.105+0000 7fd260c00700 1 -- 172.21.15.89:0/3710077018 --> [v2:172.21.15.113:6824/43958,v1:172.21.15.113:6825/43958] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2440740f0 con 0x7fd244071680 2024-04-02T09:17:22.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.105+0000 7fd260c00700 1 -- 172.21.15.89:0/3710077018 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=284) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd25808ba80 con 0x7fd264070980 2024-04-02T09:17:22.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.105+0000 7fd263e00700 1 --2- 172.21.15.89:0/3710077018 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd24406adb0 0x7fd24406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd264159500 tx=0x7fd250008040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:22.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.105+0000 7fd269400700 1 --2- 172.21.15.89:0/3710077018 >> [v2:172.21.15.113:6824/43958,v1:172.21.15.113:6825/43958] conn(0x7fd244071680 0x7fd244073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.106+0000 7fd269400700 1 --2- 172.21.15.89:0/3710077018 >> [v2:172.21.15.113:6824/43958,v1:172.21.15.113:6825/43958] conn(0x7fd244071680 0x7fd244073ac0 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:22.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.107+0000 7fd260c00700 1 -- 172.21.15.89:0/3710077018 <== osd.6 v2:172.21.15.113:6824/43958 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd2440740f0 con 0x7fd244071680 2024-04-02T09:17:22.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.121+0000 7fd26b400700 1 -- 172.21.15.89:0/3710077018 --> [v2:172.21.15.113:6824/43958,v1:172.21.15.113:6825/43958] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd2640cd560 con 0x7fd244071680 2024-04-02T09:17:22.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.122+0000 7fd260c00700 1 -- 172.21.15.89:0/3710077018 <== osd.6 v2:172.21.15.113:6824/43958 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd2640cd560 con 0x7fd244071680 2024-04-02T09:17:22.120 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:17:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T09:17:22.121 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:17:22.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.122+0000 7fd26b400700 1 -- 172.21.15.89:0/3710077018 >> [v2:172.21.15.113:6824/43958,v1:172.21.15.113:6825/43958] conn(0x7fd244071680 msgr2=0x7fd244073ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.122+0000 7fd26b400700 1 --2- 172.21.15.89:0/3710077018 >> [v2:172.21.15.113:6824/43958,v1:172.21.15.113:6825/43958] conn(0x7fd244071680 0x7fd244073ac0 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.122+0000 7fd26b400700 1 -- 172.21.15.89:0/3710077018 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd24406adb0 msgr2=0x7fd24406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.122+0000 7fd26b400700 1 --2- 172.21.15.89:0/3710077018 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fd24406adb0 0x7fd24406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd264159500 tx=0x7fd250008040 comp rx=0 tx=0).stop 2024-04-02T09:17:22.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.122+0000 7fd26b400700 1 -- 172.21.15.89:0/3710077018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd264070980 msgr2=0x7fd2640ce1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.122+0000 7fd26b400700 1 --2- 172.21.15.89:0/3710077018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd264070980 0x7fd2640ce1b0 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 crypto rx=0x7fd25800e810 tx=0x7fd25800ebd0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.122+0000 7fd269400700 1 -- 172.21.15.89:0/3710077018 reap_dead start 2024-04-02T09:17:22.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.123+0000 7fd26b400700 1 -- 172.21.15.89:0/3710077018 shutdown_connections 2024-04-02T09:17:22.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.123+0000 7fd26b400700 1 -- 172.21.15.89:0/3710077018 >> 172.21.15.89:0/3710077018 conn(0x7fd2640cf210 msgr2=0x7fd2640ca7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:22.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.123+0000 7fd26b400700 1 -- 172.21.15.89:0/3710077018 shutdown_connections 2024-04-02T09:17:22.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.123+0000 7fd26b400700 1 -- 172.21.15.89:0/3710077018 wait complete. 2024-04-02T09:17:22.138 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-02T09:17:22.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.332+0000 7f291de00700 1 Processor -- start 2024-04-02T09:17:22.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.333+0000 7f291de00700 1 -- start start 2024-04-02T09:17:22.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.334+0000 7f291de00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f29180cee10 0x7f2918161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.334+0000 7f291de00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29180ce270 con 0x7f29180d21f0 2024-04-02T09:17:22.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.334+0000 7f291de00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291815b600 con 0x7f29181611b0 2024-04-02T09:17:22.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.334+0000 7f291de00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29180d2760 con 0x7f29180cee10 2024-04-02T09:17:22.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.334+0000 7f2916000700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f29180d21f0 0x7f29180ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57936/0 (socket says 172.21.15.89:57936) 2024-04-02T09:17:22.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.334+0000 7f2916000700 1 -- 172.21.15.89:0/1923998534 learned_addr learned my addr 172.21.15.89:0/1923998534 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:22.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.334+0000 7f2917400700 1 --2- 172.21.15.89:0/1923998534 >> v2:172.21.15.113:3300/0 conn(0x7f29180cee10 0x7f2918161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.334+0000 7f2915600700 1 -- 172.21.15.89:0/1923998534 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650698124 0 0) 0x7f29180ce270 con 0x7f29180d21f0 2024-04-02T09:17:22.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.334+0000 7f2915600700 1 -- 172.21.15.89:0/1923998534 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28fc0035b0 con 0x7f29180d21f0 2024-04-02T09:17:22.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2915600700 1 -- 172.21.15.89:0/1923998534 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3895757256 0 0) 0x7f28fc0035b0 con 0x7f29180d21f0 2024-04-02T09:17:22.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2915600700 1 -- 172.21.15.89:0/1923998534 >> v2:172.21.15.113:3300/0 conn(0x7f29180cee10 msgr2=0x7f2918161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:22.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2915600700 1 --2- 172.21.15.89:0/1923998534 >> v2:172.21.15.113:3300/0 conn(0x7f29180cee10 0x7f2918161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2915600700 1 -- 172.21.15.89:0/1923998534 >> v1:172.21.15.113:6789/0 conn(0x7f29181611b0 legacy=0x7f291815ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2915600700 1 -- 172.21.15.89:0/1923998534 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29180d2a40 con 0x7f29180d21f0 2024-04-02T09:17:22.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2915600700 1 -- 172.21.15.89:0/1923998534 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2043584179 0 0) 0x7f2904001670 con 0x7f29180d21f0 2024-04-02T09:17:22.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2915600700 1 -- 172.21.15.89:0/1923998534 >> v1:172.21.15.89:6789/0 conn(0x7f29180d21f0 legacy=0x7f29180ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2915600700 1 --2- 172.21.15.89:0/1923998534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28fc003a00 0x7f28fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2915600700 1 --2- 172.21.15.89:0/1923998534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28fc004850 0x7f28fc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2915600700 1 --2- 172.21.15.89:0/1923998534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28fc005210 0x7f28fc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2915600700 1 -- 172.21.15.89:0/1923998534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29180d2760 con 0x7f28fc004850 2024-04-02T09:17:22.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2915600700 1 -- 172.21.15.89:0/1923998534 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc00c060 con 0x7f28fc003a00 2024-04-02T09:17:22.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2915600700 1 -- 172.21.15.89:0/1923998534 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc00c220 con 0x7f28fc005210 2024-04-02T09:17:22.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2916000700 1 --2- 172.21.15.89:0/1923998534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28fc004850 0x7f28fc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2916a00700 1 --2- 172.21.15.89:0/1923998534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28fc003a00 0x7f28fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.335+0000 7f2917400700 1 --2- 172.21.15.89:0/1923998534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28fc005210 0x7f28fc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.336+0000 7f2916000700 1 -- 172.21.15.89:0/1923998534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28fc005210 msgr2=0x7f28fc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:22.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.336+0000 7f2916000700 1 --2- 172.21.15.89:0/1923998534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28fc005210 0x7f28fc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.336+0000 7f2916000700 1 -- 172.21.15.89:0/1923998534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28fc003a00 msgr2=0x7f28fc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:22.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.336+0000 7f2916000700 1 --2- 172.21.15.89:0/1923998534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28fc003a00 0x7f28fc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.336+0000 7f2916000700 1 -- 172.21.15.89:0/1923998534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc0035b0 con 0x7f28fc004850 2024-04-02T09:17:22.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.336+0000 7f2917400700 1 -- 172.21.15.89:0/1923998534 reap_dead start 2024-04-02T09:17:22.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.336+0000 7f2916000700 1 --2- 172.21.15.89:0/1923998534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28fc004850 0x7f28fc004cd0 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7f290400c210 tx=0x7f28fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70a91873189e526 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:22.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.336+0000 7f2915600700 1 -- 172.21.15.89:0/1923998534 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2904007b70 con 0x7f28fc004850 2024-04-02T09:17:22.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.336+0000 7f2915600700 1 -- 172.21.15.89:0/1923998534 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290400cdb0 con 0x7f28fc004850 2024-04-02T09:17:22.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.336+0000 7f2915600700 1 -- 172.21.15.89:0/1923998534 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2904005ba0 con 0x7f28fc004850 2024-04-02T09:17:22.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.337+0000 7f291de00700 1 -- 172.21.15.89:0/1923998534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28fc004850 msgr2=0x7f28fc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.337+0000 7f291de00700 1 --2- 172.21.15.89:0/1923998534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28fc004850 0x7f28fc004cd0 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7f290400c210 tx=0x7f28fc005880 comp rx=0 tx=0).stop 2024-04-02T09:17:22.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.337+0000 7f291de00700 1 -- 172.21.15.89:0/1923998534 shutdown_connections 2024-04-02T09:17:22.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.337+0000 7f291de00700 1 --2- 172.21.15.89:0/1923998534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28fc004850 0x7f28fc004cd0 unknown :-1 s=CLOSED pgs=3131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.337+0000 7f291de00700 1 -- 172.21.15.89:0/1923998534 >> 172.21.15.89:0/1923998534 conn(0x7f291808aff0 msgr2=0x7f29180d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:22.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.337+0000 7f291de00700 1 -- 172.21.15.89:0/1923998534 shutdown_connections 2024-04-02T09:17:22.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.337+0000 7f291de00700 1 -- 172.21.15.89:0/1923998534 wait complete. 2024-04-02T09:17:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.338+0000 7f291de00700 1 Processor -- start 2024-04-02T09:17:22.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.339+0000 7f291de00700 1 -- start start 2024-04-02T09:17:22.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.339+0000 7f291de00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28fc004850 0x7f2918153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.339+0000 7f291de00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29181f41d0 0x7f291815da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.339+0000 7f291de00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f291815df70 0x7f29181f69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.339+0000 7f291de00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291815e420 con 0x7f291815df70 2024-04-02T09:17:22.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.339+0000 7f291de00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29180c9e40 con 0x7f29181f41d0 2024-04-02T09:17:22.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.339+0000 7f291de00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29181f7730 con 0x7f28fc004850 2024-04-02T09:17:22.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.340+0000 7f2917400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f291815df70 0x7f29181f69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.340+0000 7f2917400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f291815df70 0x7f29181f69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46472/0 (socket says 172.21.15.89:46472) 2024-04-02T09:17:22.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.340+0000 7f2917400700 1 -- 172.21.15.89:0/4121076155 learned_addr learned my addr 172.21.15.89:0/4121076155 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:22.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.340+0000 7f2916a00700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28fc004850 0x7f2918153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.340+0000 7f2916000700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29181f41d0 0x7f291815da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.340+0000 7f2917400700 1 -- 172.21.15.89:0/4121076155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28fc004850 msgr2=0x7f2918153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.340+0000 7f2917400700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28fc004850 0x7f2918153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.340+0000 7f2917400700 1 -- 172.21.15.89:0/4121076155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29181f41d0 msgr2=0x7f291815da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.340+0000 7f2917400700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29181f41d0 0x7f291815da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.340+0000 7f2917400700 1 -- 172.21.15.89:0/4121076155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29181f3410 con 0x7f291815df70 2024-04-02T09:17:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.340+0000 7f2916000700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29181f41d0 0x7f291815da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:17:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.340+0000 7f2916a00700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28fc004850 0x7f2918153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:17:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.340+0000 7f2917400700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f291815df70 0x7f29181f69e0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7f29181610f0 tx=0x7f2910011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.341+0000 7f290f400700 1 -- 172.21.15.89:0/4121076155 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291000e040 con 0x7f291815df70 2024-04-02T09:17:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.341+0000 7f290f400700 1 -- 172.21.15.89:0/4121076155 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291000f040 con 0x7f291815df70 2024-04-02T09:17:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.341+0000 7f290f400700 1 -- 172.21.15.89:0/4121076155 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291000fd80 con 0x7f291815df70 2024-04-02T09:17:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.341+0000 7f291de00700 1 -- 172.21.15.89:0/4121076155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29181f36f0 con 0x7f291815df70 2024-04-02T09:17:22.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.341+0000 7f291de00700 1 -- 172.21.15.89:0/4121076155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29181f3cb0 con 0x7f291815df70 2024-04-02T09:17:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.342+0000 7f291de00700 1 -- 172.21.15.89:0/4121076155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29181f3ed0 con 0x7f291815df70 2024-04-02T09:17:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.343+0000 7f290f400700 1 -- 172.21.15.89:0/4121076155 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2910022460 con 0x7f291815df70 2024-04-02T09:17:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.344+0000 7f290f400700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f28f806adb0 0x7f28f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.344+0000 7f290f400700 1 -- 172.21.15.89:0/4121076155 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f2910089df0 con 0x7f291815df70 2024-04-02T09:17:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.344+0000 7f2916a00700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f28f806adb0 0x7f28f806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.345+0000 7f2916a00700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f28f806adb0 0x7f28f806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f29180684a0 tx=0x7f2908009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:22.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.349+0000 7f290f400700 1 -- 172.21.15.89:0/4121076155 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2910054b70 con 0x7f291815df70 2024-04-02T09:17:22.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.549+0000 7f291de00700 1 -- 172.21.15.89:0/4121076155 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f29180cdf00 con 0x7f28f806adb0 2024-04-02T09:17:22.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.549+0000 7f290f400700 1 -- 172.21.15.89:0/4121076155 <== mgr.13575 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2908001370 con 0x7f28f806adb0 2024-04-02T09:17:22.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.551+0000 7f291de00700 1 -- 172.21.15.89:0/4121076155 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f28f806adb0 msgr2=0x7f28f806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.551+0000 7f291de00700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f28f806adb0 0x7f28f806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f29180684a0 tx=0x7f2908009040 comp rx=0 tx=0).stop 2024-04-02T09:17:22.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.551+0000 7f291de00700 1 -- 172.21.15.89:0/4121076155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f291815df70 msgr2=0x7f29181f69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.551+0000 7f291de00700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f291815df70 0x7f29181f69e0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7f29181610f0 tx=0x7f2910011f80 comp rx=0 tx=0).stop 2024-04-02T09:17:22.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.551+0000 7f291de00700 1 -- 172.21.15.89:0/4121076155 shutdown_connections 2024-04-02T09:17:22.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.551+0000 7f291de00700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f291815df70 0x7f29181f69e0 unknown :-1 s=CLOSED pgs=3132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.551+0000 7f291de00700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28fc004850 0x7f2918153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.551+0000 7f291de00700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f28f806adb0 0x7f28f806d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.551+0000 7f291de00700 1 --2- 172.21.15.89:0/4121076155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29181f41d0 0x7f291815da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.551+0000 7f291de00700 1 -- 172.21.15.89:0/4121076155 >> 172.21.15.89:0/4121076155 conn(0x7f291808aff0 msgr2=0x7f291815d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:22.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.551+0000 7f291de00700 1 -- 172.21.15.89:0/4121076155 shutdown_connections 2024-04-02T09:17:22.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.551+0000 7f291de00700 1 -- 172.21.15.89:0/4121076155 wait complete. 2024-04-02T09:17:22.552 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 6 to deep-scrub 2024-04-02T09:17:22.568 INFO:tasks.ceph:Scrubbing osd.7 2024-04-02T09:17:22.569 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-02T09:17:22.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.766+0000 7f5db5200700 1 Processor -- start 2024-04-02T09:17:22.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.767+0000 7f5db5200700 1 -- start start 2024-04-02T09:17:22.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.768+0000 7f5db5200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db0155ac0 0x7f5db015c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.768+0000 7f5db5200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5db00ae670 0x7f5db015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.768+0000 7f5db5200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db006dd00 con 0x7f5db006c910 2024-04-02T09:17:22.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.768+0000 7f5db5200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5db006d0f0 con 0x7f5db00ae670 2024-04-02T09:17:22.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.768+0000 7f5db5200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db006d3d0 con 0x7f5db0155ac0 2024-04-02T09:17:22.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.768+0000 7f5dade00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db0155ac0 0x7f5db015c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.768+0000 7f5dae800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5db00ae670 0x7f5db015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.768+0000 7f5dae800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5db00ae670 0x7f5db015ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.89:59508/0 (socket says 172.21.15.89:59508) 2024-04-02T09:17:22.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.768+0000 7f5da7e00700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f5db006c910 0x7f5db00adf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:44290/0 (socket says 172.21.15.89:44290) 2024-04-02T09:17:22.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.768+0000 7f5dae800700 1 -- 172.21.15.89:0/192831598 learned_addr learned my addr 172.21.15.89:0/192831598 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:22.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.769+0000 7f5dae800700 1 -- 172.21.15.89:0/192831598 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db0155ac0 msgr2=0x7f5db015c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:22.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.769+0000 7f5dae800700 1 --2- 172.21.15.89:0/192831598 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db0155ac0 0x7f5db015c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.769+0000 7f5dae800700 1 -- 172.21.15.89:0/192831598 >> v1:172.21.15.113:6789/0 conn(0x7f5db006c910 legacy=0x7f5db00adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.769+0000 7f5dae800700 1 -- 172.21.15.89:0/192831598 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db006d6b0 con 0x7f5db00ae670 2024-04-02T09:17:22.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.769+0000 7f5dad400700 1 -- 172.21.15.89:0/192831598 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634101132 0 0) 0x7f5db006dd00 con 0x7f5db006c910 2024-04-02T09:17:22.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.769+0000 7f5dae800700 1 --2- 172.21.15.89:0/192831598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db00ae670 0x7f5db015ccb0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009e50 tx=0x7f5da800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9703b3228bf1191 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:22.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.769+0000 7f5dad400700 1 -- 172.21.15.89:0/192831598 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da800e070 con 0x7f5db00ae670 2024-04-02T09:17:22.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.769+0000 7f5dad400700 1 -- 172.21.15.89:0/192831598 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da800b910 con 0x7f5db00ae670 2024-04-02T09:17:22.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.769+0000 7f5dad400700 1 -- 172.21.15.89:0/192831598 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da800bcc0 con 0x7f5db00ae670 2024-04-02T09:17:22.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.770+0000 7f5db5200700 1 -- 172.21.15.89:0/192831598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db00ae670 msgr2=0x7f5db015ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.770+0000 7f5db5200700 1 --2- 172.21.15.89:0/192831598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db00ae670 0x7f5db015ccb0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009e50 tx=0x7f5da800b330 comp rx=0 tx=0).stop 2024-04-02T09:17:22.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.770+0000 7f5db5200700 1 -- 172.21.15.89:0/192831598 shutdown_connections 2024-04-02T09:17:22.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.770+0000 7f5db5200700 1 --2- 172.21.15.89:0/192831598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db00ae670 0x7f5db015ccb0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.770+0000 7f5db5200700 1 --2- 172.21.15.89:0/192831598 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db0155ac0 0x7f5db015c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.770+0000 7f5db5200700 1 -- 172.21.15.89:0/192831598 >> 172.21.15.89:0/192831598 conn(0x7f5db00cf210 msgr2=0x7f5db00cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:22.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.770+0000 7f5db5200700 1 -- 172.21.15.89:0/192831598 shutdown_connections 2024-04-02T09:17:22.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.770+0000 7f5db5200700 1 -- 172.21.15.89:0/192831598 wait complete. 2024-04-02T09:17:22.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.772+0000 7f5db5200700 1 Processor -- start 2024-04-02T09:17:22.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.772+0000 7f5db5200700 1 -- start start 2024-04-02T09:17:22.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5db5200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5db006c910 0x7f5db0149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5db5200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db00ae670 0x7f5db014c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5db5200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db0155ac0 0x7f5db00afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5db5200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db0149b90 con 0x7f5db006c910 2024-04-02T09:17:22.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5db5200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db014c740 con 0x7f5db00ae670 2024-04-02T09:17:22.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5db5200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db015a760 con 0x7f5db0155ac0 2024-04-02T09:17:22.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5dade00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5db006c910 0x7f5db0149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5dade00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5db006c910 0x7f5db0149650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46482/0 (socket says 172.21.15.89:46482) 2024-04-02T09:17:22.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5dade00700 1 -- 172.21.15.89:0/2486138844 learned_addr learned my addr 172.21.15.89:0/2486138844 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:22.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5dae800700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db0155ac0 0x7f5db00afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5da7e00700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db00ae670 0x7f5db014c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5dade00700 1 -- 172.21.15.89:0/2486138844 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db0155ac0 msgr2=0x7f5db00afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5dade00700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5db0155ac0 0x7f5db00afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5dade00700 1 -- 172.21.15.89:0/2486138844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db00ae670 msgr2=0x7f5db014c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5dade00700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db00ae670 0x7f5db014c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5dade00700 1 -- 172.21.15.89:0/2486138844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da8009b30 con 0x7f5db006c910 2024-04-02T09:17:22.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.773+0000 7f5da7e00700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db00ae670 0x7f5db014c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:17:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.774+0000 7f5dade00700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5db006c910 0x7f5db0149650 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7f5da000e870 tx=0x7f5da000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.774+0000 7f5da6000700 1 -- 172.21.15.89:0/2486138844 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da0009590 con 0x7f5db006c910 2024-04-02T09:17:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.774+0000 7f5da6000700 1 -- 172.21.15.89:0/2486138844 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da0013680 con 0x7f5db006c910 2024-04-02T09:17:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.774+0000 7f5db5200700 1 -- 172.21.15.89:0/2486138844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db0156020 con 0x7f5db006c910 2024-04-02T09:17:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.774+0000 7f5da6000700 1 -- 172.21.15.89:0/2486138844 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da00139f0 con 0x7f5db006c910 2024-04-02T09:17:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.774+0000 7f5db5200700 1 -- 172.21.15.89:0/2486138844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db015ad20 con 0x7f5db006c910 2024-04-02T09:17:22.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.775+0000 7f5db5200700 1 -- 172.21.15.89:0/2486138844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5db0155ea0 con 0x7f5db006c910 2024-04-02T09:17:22.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.776+0000 7f5da6000700 1 -- 172.21.15.89:0/2486138844 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5da0013b50 con 0x7f5db006c910 2024-04-02T09:17:22.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.777+0000 7f5da6000700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5d8c06adb0 0x7f5d8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.777+0000 7f5da6000700 1 -- 172.21.15.89:0/2486138844 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f5da0089d70 con 0x7f5db006c910 2024-04-02T09:17:22.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.777+0000 7f5da7e00700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5d8c06adb0 0x7f5d8c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.777+0000 7f5da6000700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.113:6816/43941,v1:172.21.15.113:6817/43941] conn(0x7f5d8c071680 0x7f5d8c073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:22.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.777+0000 7f5da6000700 1 -- 172.21.15.89:0/2486138844 --> [v2:172.21.15.113:6816/43941,v1:172.21.15.113:6817/43941] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d8c0740f0 con 0x7f5d8c071680 2024-04-02T09:17:22.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.777+0000 7f5da6000700 1 -- 172.21.15.89:0/2486138844 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=284) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5da008a1c0 con 0x7f5db006c910 2024-04-02T09:17:22.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.778+0000 7f5da7e00700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5d8c06adb0 0x7f5d8c06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5db00ce4b0 tx=0x7f5d98007ca0 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:22.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.778+0000 7f5dae800700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.113:6816/43941,v1:172.21.15.113:6817/43941] conn(0x7f5d8c071680 0x7f5d8c073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:22.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.778+0000 7f5dae800700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.113:6816/43941,v1:172.21.15.113:6817/43941] conn(0x7f5d8c071680 0x7f5d8c073ac0 crc :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:22.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.779+0000 7f5da6000700 1 -- 172.21.15.89:0/2486138844 <== osd.7 v2:172.21.15.113:6816/43941 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5d8c0740f0 con 0x7f5d8c071680 2024-04-02T09:17:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.794+0000 7f5db5200700 1 -- 172.21.15.89:0/2486138844 --> [v2:172.21.15.113:6816/43941,v1:172.21.15.113:6817/43941] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5db006cd90 con 0x7f5d8c071680 2024-04-02T09:17:22.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.794+0000 7f5da6000700 1 -- 172.21.15.89:0/2486138844 <== osd.7 v2:172.21.15.113:6816/43941 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5da80092e0 con 0x7f5d8c071680 2024-04-02T09:17:22.793 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-02T09:17:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-02T09:17:22.793 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-02T09:17:22.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.795+0000 7f5db5200700 1 -- 172.21.15.89:0/2486138844 >> [v2:172.21.15.113:6816/43941,v1:172.21.15.113:6817/43941] conn(0x7f5d8c071680 msgr2=0x7f5d8c073ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.795+0000 7f5db5200700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.113:6816/43941,v1:172.21.15.113:6817/43941] conn(0x7f5d8c071680 0x7f5d8c073ac0 crc :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.795+0000 7f5db5200700 1 -- 172.21.15.89:0/2486138844 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5d8c06adb0 msgr2=0x7f5d8c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.795+0000 7f5db5200700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f5d8c06adb0 0x7f5d8c06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5db00ce4b0 tx=0x7f5d98007ca0 comp rx=0 tx=0).stop 2024-04-02T09:17:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.795+0000 7f5db5200700 1 -- 172.21.15.89:0/2486138844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5db006c910 msgr2=0x7f5db0149650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.795+0000 7f5db5200700 1 --2- 172.21.15.89:0/2486138844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5db006c910 0x7f5db0149650 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7f5da000e870 tx=0x7f5da000eb80 comp rx=0 tx=0).stop 2024-04-02T09:17:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.795+0000 7f5dae800700 1 -- 172.21.15.89:0/2486138844 reap_dead start 2024-04-02T09:17:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.795+0000 7f5db5200700 1 -- 172.21.15.89:0/2486138844 shutdown_connections 2024-04-02T09:17:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.795+0000 7f5db5200700 1 -- 172.21.15.89:0/2486138844 >> 172.21.15.89:0/2486138844 conn(0x7f5db00cf210 msgr2=0x7f5db0070980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:22.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.795+0000 7f5db5200700 1 -- 172.21.15.89:0/2486138844 shutdown_connections 2024-04-02T09:17:22.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:22.795+0000 7f5db5200700 1 -- 172.21.15.89:0/2486138844 wait complete. 2024-04-02T09:17:22.810 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-02T09:17:23.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.005+0000 7f9204e00700 1 Processor -- start 2024-04-02T09:17:23.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.006+0000 7f9204e00700 1 -- start start 2024-04-02T09:17:23.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.007+0000 7f9204e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f920015d9d0 0x7f92000d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.007+0000 7f9204e00700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f92000d2540 0x7f92000ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.007+0000 7f9204e00700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9200161420 con 0x7f92000d16a0 2024-04-02T09:17:23.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.007+0000 7f9204e00700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f920015edc0 con 0x7f92000d2540 2024-04-02T09:17:23.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.007+0000 7f9204e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9200161700 con 0x7f920015d9d0 2024-04-02T09:17:23.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.008+0000 7f91fde00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f920015d9d0 0x7f92000d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.008+0000 7f91fe800700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f92000d2540 0x7f92000ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.008+0000 7f91fde00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f920015d9d0 0x7f92000d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.89:39134/0 (socket says 172.21.15.89:39134) 2024-04-02T09:17:23.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.008+0000 7f91fde00700 1 -- 172.21.15.89:0/3165612131 learned_addr learned my addr 172.21.15.89:0/3165612131 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:23.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.008+0000 7f91fd400700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f92000d16a0 0x7f92000d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.89:44298/0 (socket says 172.21.15.89:44298) 2024-04-02T09:17:23.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.008+0000 7f91fe800700 1 -- 172.21.15.89:0/3165612131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f920015d9d0 msgr2=0x7f92000d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:23.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.008+0000 7f91fe800700 1 --2- 172.21.15.89:0/3165612131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f920015d9d0 0x7f92000d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.008+0000 7f91fe800700 1 -- 172.21.15.89:0/3165612131 >> v1:172.21.15.113:6789/0 conn(0x7f92000d16a0 legacy=0x7f92000d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.008+0000 7f91fe800700 1 -- 172.21.15.89:0/3165612131 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92001619e0 con 0x7f92000d2540 2024-04-02T09:17:23.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.008+0000 7f91fde00700 1 --2- 172.21.15.89:0/3165612131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f920015d9d0 0x7f92000d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T09:17:23.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.008+0000 7f91fca00700 1 -- 172.21.15.89:0/3165612131 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398618792 0 0) 0x7f9200161420 con 0x7f92000d16a0 2024-04-02T09:17:23.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.009+0000 7f91fe800700 1 --2- 172.21.15.89:0/3165612131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92000d2540 0x7f92000ce360 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009dc0 tx=0x7f91f800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=514e7583a6d19c82 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:23.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.009+0000 7f91fca00700 1 -- 172.21.15.89:0/3165612131 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f800e070 con 0x7f92000d2540 2024-04-02T09:17:23.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.009+0000 7f91fca00700 1 -- 172.21.15.89:0/3165612131 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f800bae0 con 0x7f92000d2540 2024-04-02T09:17:23.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.009+0000 7f91fca00700 1 -- 172.21.15.89:0/3165612131 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f800be90 con 0x7f92000d2540 2024-04-02T09:17:23.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.009+0000 7f9204e00700 1 -- 172.21.15.89:0/3165612131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92000d2540 msgr2=0x7f92000ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.009+0000 7f9204e00700 1 --2- 172.21.15.89:0/3165612131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92000d2540 0x7f92000ce360 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009dc0 tx=0x7f91f800b330 comp rx=0 tx=0).stop 2024-04-02T09:17:23.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.010+0000 7f9204e00700 1 -- 172.21.15.89:0/3165612131 shutdown_connections 2024-04-02T09:17:23.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.010+0000 7f9204e00700 1 --2- 172.21.15.89:0/3165612131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92000d2540 0x7f92000ce360 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.010+0000 7f9204e00700 1 --2- 172.21.15.89:0/3165612131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f920015d9d0 0x7f92000d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.010+0000 7f9204e00700 1 -- 172.21.15.89:0/3165612131 >> 172.21.15.89:0/3165612131 conn(0x7f920008aff0 msgr2=0x7f92000d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:23.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.010+0000 7f9204e00700 1 -- 172.21.15.89:0/3165612131 shutdown_connections 2024-04-02T09:17:23.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.010+0000 7f9204e00700 1 -- 172.21.15.89:0/3165612131 wait complete. 2024-04-02T09:17:23.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.011+0000 7f9204e00700 1 Processor -- start 2024-04-02T09:17:23.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.011+0000 7f9204e00700 1 -- start start 2024-04-02T09:17:23.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.012+0000 7f9204e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92000d16a0 0x7f92000b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.012+0000 7f9204e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92000d2540 0x7f92001601d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.012+0000 7f9204e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f920015d9d0 0x7f9200158530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.012+0000 7f9204e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92000b7080 con 0x7f920015d9d0 2024-04-02T09:17:23.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.012+0000 7f9204e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92000b7770 con 0x7f92000d16a0 2024-04-02T09:17:23.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.012+0000 7f9204e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92000cd3d0 con 0x7f92000d2540 2024-04-02T09:17:23.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.013+0000 7f91fe800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f920015d9d0 0x7f9200158530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.013+0000 7f91fe800700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f920015d9d0 0x7f9200158530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46490/0 (socket says 172.21.15.89:46490) 2024-04-02T09:17:23.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.013+0000 7f91fe800700 1 -- 172.21.15.89:0/1835413260 learned_addr learned my addr 172.21.15.89:0/1835413260 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:23.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.013+0000 7f91fd400700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92000d2540 0x7f92001601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.013+0000 7f91fde00700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92000d16a0 0x7f92000b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.013+0000 7f91fe800700 1 -- 172.21.15.89:0/1835413260 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92000d2540 msgr2=0x7f92001601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.013+0000 7f91fe800700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92000d2540 0x7f92001601d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.013+0000 7f91fe800700 1 -- 172.21.15.89:0/1835413260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92000d16a0 msgr2=0x7f92000b7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.013+0000 7f91fe800700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92000d16a0 0x7f92000b7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.013+0000 7f91fe800700 1 -- 172.21.15.89:0/1835413260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f0009550 con 0x7f920015d9d0 2024-04-02T09:17:23.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.013+0000 7f91fde00700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92000d16a0 0x7f92000b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T09:17:23.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.013+0000 7f91fe800700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f920015d9d0 0x7f9200158530 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7f91f8014040 tx=0x7f91f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:23.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.014+0000 7f91f6000700 1 -- 172.21.15.89:0/1835413260 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f80045d0 con 0x7f920015d9d0 2024-04-02T09:17:23.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.014+0000 7f91f6000700 1 -- 172.21.15.89:0/1835413260 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f8020070 con 0x7f920015d9d0 2024-04-02T09:17:23.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.014+0000 7f9204e00700 1 -- 172.21.15.89:0/1835413260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f8009a70 con 0x7f920015d9d0 2024-04-02T09:17:23.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.014+0000 7f91f6000700 1 -- 172.21.15.89:0/1835413260 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f8004970 con 0x7f920015d9d0 2024-04-02T09:17:23.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.014+0000 7f9204e00700 1 -- 172.21.15.89:0/1835413260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92000b67e0 con 0x7f920015d9d0 2024-04-02T09:17:23.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.015+0000 7f9204e00700 1 -- 172.21.15.89:0/1835413260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92000d1ca0 con 0x7f920015d9d0 2024-04-02T09:17:23.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.016+0000 7f91f6000700 1 -- 172.21.15.89:0/1835413260 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f91f8012430 con 0x7f920015d9d0 2024-04-02T09:17:23.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.017+0000 7f91f6000700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f91e006adb0 0x7f91e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.017+0000 7f91f6000700 1 -- 172.21.15.89:0/1835413260 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7f91f8022070 con 0x7f920015d9d0 2024-04-02T09:17:23.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.017+0000 7f91fde00700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f91e006adb0 0x7f91e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.018+0000 7f91fde00700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f91e006adb0 0x7f91e006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009d10 tx=0x7f91f0008040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:23.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.021+0000 7f91f6000700 1 -- 172.21.15.89:0/1835413260 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f91f805a0f0 con 0x7f920015d9d0 2024-04-02T09:17:23.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.223+0000 7f9204e00700 1 -- 172.21.15.89:0/1835413260 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f9200161fa0 con 0x7f91e006adb0 2024-04-02T09:17:23.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.224+0000 7f91f6000700 1 -- 172.21.15.89:0/1835413260 <== mgr.13575 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f91f0009960 con 0x7f91e006adb0 2024-04-02T09:17:23.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.226+0000 7f9204e00700 1 -- 172.21.15.89:0/1835413260 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f91e006adb0 msgr2=0x7f91e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.226+0000 7f9204e00700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f91e006adb0 0x7f91e006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009d10 tx=0x7f91f0008040 comp rx=0 tx=0).stop 2024-04-02T09:17:23.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.226+0000 7f9204e00700 1 -- 172.21.15.89:0/1835413260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f920015d9d0 msgr2=0x7f9200158530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.226+0000 7f9204e00700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f920015d9d0 0x7f9200158530 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7f91f8014040 tx=0x7f91f8004ad0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.226+0000 7f9204e00700 1 -- 172.21.15.89:0/1835413260 shutdown_connections 2024-04-02T09:17:23.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.226+0000 7f9204e00700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7f91e006adb0 0x7f91e006d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.226+0000 7f9204e00700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92000d16a0 0x7f92000b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.226+0000 7f9204e00700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f920015d9d0 0x7f9200158530 unknown :-1 s=CLOSED pgs=3134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.226+0000 7f9204e00700 1 --2- 172.21.15.89:0/1835413260 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f92000d2540 0x7f92001601d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.226+0000 7f9204e00700 1 -- 172.21.15.89:0/1835413260 >> 172.21.15.89:0/1835413260 conn(0x7f920008aff0 msgr2=0x7f920015ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.226+0000 7f9204e00700 1 -- 172.21.15.89:0/1835413260 shutdown_connections 2024-04-02T09:17:23.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.226+0000 7f9204e00700 1 -- 172.21.15.89:0/1835413260 wait complete. 2024-04-02T09:17:23.227 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 7 to deep-scrub 2024-04-02T09:17:23.243 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T09:17:23.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.436+0000 7fa0e5200700 1 Processor -- start 2024-04-02T09:17:23.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.436+0000 7fa0e5200700 1 -- start start 2024-04-02T09:17:23.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.438+0000 7fa0e5200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa0e00d18b0 0x7fa0e00cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.438+0000 7fa0e5200700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa0e0161590 0x7fa0e0161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.438+0000 7fa0e5200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e00ceda0 con 0x7fa0e0160f10 2024-04-02T09:17:23.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.438+0000 7fa0e5200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e00d1d30 con 0x7fa0e00d18b0 2024-04-02T09:17:23.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.438+0000 7fa0e5200700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e0161f60 con 0x7fa0e0161590 2024-04-02T09:17:23.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.438+0000 7fa0dd400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa0e00d18b0 0x7fa0e00cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.438+0000 7fa0dd400700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa0e00d18b0 0x7fa0e00cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46504/0 (socket says 172.21.15.89:46504) 2024-04-02T09:17:23.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.438+0000 7fa0dd400700 1 -- 172.21.15.89:0/3780574961 learned_addr learned my addr 172.21.15.89:0/3780574961 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:23.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.438+0000 7fa0de800700 1 --2- 172.21.15.89:0/3780574961 >> v2:172.21.15.113:3300/0 conn(0x7fa0e0161590 0x7fa0e0161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.438+0000 7fa0dd400700 1 -- 172.21.15.89:0/3780574961 >> v2:172.21.15.113:3300/0 conn(0x7fa0e0161590 msgr2=0x7fa0e0161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:23.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.438+0000 7fa0dd400700 1 --2- 172.21.15.89:0/3780574961 >> v2:172.21.15.113:3300/0 conn(0x7fa0e0161590 0x7fa0e0161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.438+0000 7fa0dd400700 1 -- 172.21.15.89:0/3780574961 >> v1:172.21.15.113:6789/0 conn(0x7fa0e0160f10 legacy=0x7fa0e00d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.438+0000 7fa0dd400700 1 -- 172.21.15.89:0/3780574961 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e01612f0 con 0x7fa0e00d18b0 2024-04-02T09:17:23.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.439+0000 7fa0dd400700 1 --2- 172.21.15.89:0/3780574961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e00d18b0 0x7fa0e00cf230 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00d1030 tx=0x7fa0cc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ccbe0b326ed3ac4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:23.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.439+0000 7fa0dca00700 1 -- 172.21.15.89:0/3780574961 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc00b870 con 0x7fa0e00d18b0 2024-04-02T09:17:23.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.439+0000 7fa0dca00700 1 -- 172.21.15.89:0/3780574961 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0cc00b9d0 con 0x7fa0e00d18b0 2024-04-02T09:17:23.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.439+0000 7fa0dca00700 1 -- 172.21.15.89:0/3780574961 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc00bd60 con 0x7fa0e00d18b0 2024-04-02T09:17:23.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.439+0000 7fa0e5200700 1 -- 172.21.15.89:0/3780574961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e00d18b0 msgr2=0x7fa0e00cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.439+0000 7fa0e5200700 1 --2- 172.21.15.89:0/3780574961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e00d18b0 0x7fa0e00cf230 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00d1030 tx=0x7fa0cc009bd0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.439+0000 7fa0e5200700 1 -- 172.21.15.89:0/3780574961 shutdown_connections 2024-04-02T09:17:23.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.439+0000 7fa0e5200700 1 --2- 172.21.15.89:0/3780574961 >> v2:172.21.15.113:3300/0 conn(0x7fa0e0161590 0x7fa0e0161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.439+0000 7fa0e5200700 1 --2- 172.21.15.89:0/3780574961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e00d18b0 0x7fa0e00cf230 unknown :-1 s=CLOSED pgs=3135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.439+0000 7fa0e5200700 1 -- 172.21.15.89:0/3780574961 >> 172.21.15.89:0/3780574961 conn(0x7fa0e008aff0 msgr2=0x7fa0e00d3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:23.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.440+0000 7fa0e5200700 1 -- 172.21.15.89:0/3780574961 shutdown_connections 2024-04-02T09:17:23.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.440+0000 7fa0e5200700 1 -- 172.21.15.89:0/3780574961 wait complete. 2024-04-02T09:17:23.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.441+0000 7fa0e5200700 1 Processor -- start 2024-04-02T09:17:23.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.441+0000 7fa0e5200700 1 -- start start 2024-04-02T09:17:23.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.442+0000 7fa0e5200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0e00d18b0 0x7fa0e009f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.442+0000 7fa0e5200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e0160f10 0x7fa0e015bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.442+0000 7fa0e5200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0e0161590 0x7fa0e0160130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.442+0000 7fa0e5200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e00a0790 con 0x7fa0e0160f10 2024-04-02T09:17:23.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.442+0000 7fa0e5200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e015c070 con 0x7fa0e0161590 2024-04-02T09:17:23.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.442+0000 7fa0e5200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e009fb10 con 0x7fa0e00d18b0 2024-04-02T09:17:23.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.443+0000 7fa0dd400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e0160f10 0x7fa0e015bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.443+0000 7fa0dd400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e0160f10 0x7fa0e015bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46508/0 (socket says 172.21.15.89:46508) 2024-04-02T09:17:23.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.443+0000 7fa0dd400700 1 -- 172.21.15.89:0/2883744282 learned_addr learned my addr 172.21.15.89:0/2883744282 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:23.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.443+0000 7fa0de800700 1 --2- 172.21.15.89:0/2883744282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0e0161590 0x7fa0e0160130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.443+0000 7fa0dde00700 1 --2- 172.21.15.89:0/2883744282 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0e00d18b0 0x7fa0e009f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.443+0000 7fa0dd400700 1 -- 172.21.15.89:0/2883744282 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0e00d18b0 msgr2=0x7fa0e009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.443+0000 7fa0dd400700 1 --2- 172.21.15.89:0/2883744282 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0e00d18b0 0x7fa0e009f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.443+0000 7fa0dd400700 1 -- 172.21.15.89:0/2883744282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0e0161590 msgr2=0x7fa0e0160130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.443+0000 7fa0dd400700 1 --2- 172.21.15.89:0/2883744282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0e0161590 0x7fa0e0160130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.443+0000 7fa0dd400700 1 -- 172.21.15.89:0/2883744282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0cc009550 con 0x7fa0e0160f10 2024-04-02T09:17:23.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.443+0000 7fa0dd400700 1 --2- 172.21.15.89:0/2883744282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e0160f10 0x7fa0e015bb30 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc000c00 tx=0x7fa0cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:23.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.443+0000 7fa0d6000700 1 -- 172.21.15.89:0/2883744282 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc004520 con 0x7fa0e0160f10 2024-04-02T09:17:23.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.444+0000 7fa0d6000700 1 -- 172.21.15.89:0/2883744282 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0cc01e070 con 0x7fa0e0160f10 2024-04-02T09:17:23.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.444+0000 7fa0e5200700 1 -- 172.21.15.89:0/2883744282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e009fdf0 con 0x7fa0e0160f10 2024-04-02T09:17:23.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.444+0000 7fa0d6000700 1 -- 172.21.15.89:0/2883744282 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc0038a0 con 0x7fa0e0160f10 2024-04-02T09:17:23.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.444+0000 7fa0e5200700 1 -- 172.21.15.89:0/2883744282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e00a03b0 con 0x7fa0e0160f10 2024-04-02T09:17:23.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.445+0000 7fa0e5200700 1 -- 172.21.15.89:0/2883744282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e00a0540 con 0x7fa0e0160f10 2024-04-02T09:17:23.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.446+0000 7fa0d6000700 1 -- 172.21.15.89:0/2883744282 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa0cc003a00 con 0x7fa0e0160f10 2024-04-02T09:17:23.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.447+0000 7fa0d6000700 1 --2- 172.21.15.89:0/2883744282 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa0c006adb0 0x7fa0c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.447+0000 7fa0d6000700 1 -- 172.21.15.89:0/2883744282 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7fa0cc020070 con 0x7fa0e0160f10 2024-04-02T09:17:23.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.447+0000 7fa0dde00700 1 --2- 172.21.15.89:0/2883744282 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa0c006adb0 0x7fa0c006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.448+0000 7fa0dde00700 1 --2- 172.21.15.89:0/2883744282 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa0c006adb0 0x7fa0c006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00ca330 tx=0x7fa0d0009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:23.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.451+0000 7fa0d6000700 1 -- 172.21.15.89:0/2883744282 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa0cc055c50 con 0x7fa0e0160f10 2024-04-02T09:17:23.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.652+0000 7fa0e5200700 1 -- 172.21.15.89:0/2883744282 --> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa0e00ce750 con 0x7fa0c006adb0 2024-04-02T09:17:23.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.654+0000 7fa0d6000700 1 -- 172.21.15.89:0/2883744282 <== mgr.13575 v2:172.21.15.89:6832/108855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50475 (secure 0 0 0) 0x7fa0d00050d0 con 0x7fa0c006adb0 2024-04-02T09:17:23.654 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:17:23.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.657+0000 7fa0e5200700 1 -- 172.21.15.89:0/2883744282 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa0c006adb0 msgr2=0x7fa0c006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.657+0000 7fa0e5200700 1 --2- 172.21.15.89:0/2883744282 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa0c006adb0 0x7fa0c006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00ca330 tx=0x7fa0d0009040 comp rx=0 tx=0).stop 2024-04-02T09:17:23.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.657+0000 7fa0e5200700 1 -- 172.21.15.89:0/2883744282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e0160f10 msgr2=0x7fa0e015bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.657+0000 7fa0e5200700 1 --2- 172.21.15.89:0/2883744282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e0160f10 0x7fa0e015bb30 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc000c00 tx=0x7fa0cc004750 comp rx=0 tx=0).stop 2024-04-02T09:17:23.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.657+0000 7fa0e5200700 1 -- 172.21.15.89:0/2883744282 shutdown_connections 2024-04-02T09:17:23.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.657+0000 7fa0e5200700 1 --2- 172.21.15.89:0/2883744282 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fa0c006adb0 0x7fa0c006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.657+0000 7fa0e5200700 1 --2- 172.21.15.89:0/2883744282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0e0161590 0x7fa0e0160130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.657+0000 7fa0e5200700 1 --2- 172.21.15.89:0/2883744282 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0e00d18b0 0x7fa0e009f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.657+0000 7fa0e5200700 1 --2- 172.21.15.89:0/2883744282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e0160f10 0x7fa0e015bb30 unknown :-1 s=CLOSED pgs=3136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.657+0000 7fa0e5200700 1 -- 172.21.15.89:0/2883744282 >> 172.21.15.89:0/2883744282 conn(0x7fa0e008aff0 msgr2=0x7fa0e00cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:23.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.657+0000 7fa0e5200700 1 -- 172.21.15.89:0/2883744282 shutdown_connections 2024-04-02T09:17:23.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.657+0000 7fa0e5200700 1 -- 172.21.15.89:0/2883744282 wait complete. 2024-04-02T09:17:23.660 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-02T09:17:23.673 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-02T09:17:23.570449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":447164,"kb_used_data":29248,"kb_used_omap":12,"kb_used_meta":417779,"kb_avail":754527556,"statfs":{"total":773094113280,"available":772636217344,"internally_reserved":0,"allocated":29949952,"data_stored":15989882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":427806264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.950699"},"pg_stats":[{"pgid":"1.0","version":"284'426","reported_seq":1552,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-04-02T09:17:20.854295+0000","last_change":"2024-04-02T09:17:20.854372+0000","last_active":"2024-04-02T09:17:20.854295+0000","last_peered":"2024-04-02T09:17:20.854295+0000","last_clean":"2024-04-02T09:17:20.854295+0000","last_became_active":"2024-04-02T07:35:43.316208+0000","last_became_peered":"2024-04-02T07:35:43.316208+0000","last_unstale":"2024-04-02T09:17:20.854295+0000","last_undegraded":"2024-04-02T09:17:20.854295+0000","last_fullsized":"2024-04-02T09:17:20.854295+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'426","last_scrub_stamp":"2024-04-02T09:17:20.854275+0000","last_deep_scrub":"284'426","last_deep_scrub_stamp":"2024-04-02T09:17:20.854275+0000","last_clean_scrub_stamp":"2024-04-02T09:17:20.854275+0000","objects_scrubbed":2,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:57:51.650731+0000","scrub_duration":0.0060138759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576038,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60876,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94310964,"statfs":{"total":96636764160,"available":96574427136,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72199999999999998,"15min":0.70699999999999996},"min":{"1min":0.44600000000000001,"5min":0.34599999999999997,"15min":0.309},"max":{"1min":0.95499999999999996,"5min":8.016,"15min":8.016},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.72099999999999997,"15min":0.71399999999999997},"min":{"1min":0.57799999999999996,"5min":0.379,"15min":0.34100000000000003},"max":{"1min":1.0329999999999999,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":0.67400000000000004}]},{"osd":1,"last update":"Tue Apr 2 09:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.747,"15min":0.73199999999999998},"min":{"1min":0.52300000000000002,"5min":0.42199999999999999,"15min":0.38200000000000001},"max":{"1min":0.93700000000000006,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.69199999999999995,"15min":0.70099999999999996},"min":{"1min":0.58099999999999996,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.98899999999999999,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.624}]},{"osd":2,"last update":"Tue Apr 2 09:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76900000000000002,"15min":0.75800000000000001},"min":{"1min":0.64500000000000002,"5min":0.437,"15min":0.40000000000000002},"max":{"1min":0.98499999999999999,"5min":7.9790000000000001,"15min":7.9790000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.79800000000000004,"15min":0.80100000000000005},"min":{"1min":0.63300000000000001,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0029999999999999,"5min":4.8819999999999997,"15min":4.8819999999999997},"last":0.85299999999999998}]},{"osd":3,"last update":"Tue Apr 2 09:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.76200000000000001,"15min":0.74199999999999999},"min":{"1min":0.54900000000000004,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93700000000000006,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.77900000000000003,"15min":0.76700000000000002},"min":{"1min":0.67000000000000004,"5min":0.48199999999999998,"15min":0.45600000000000002},"max":{"1min":0.97299999999999998,"5min":5.093,"15min":5.093},"last":0.74299999999999999}]},{"osd":4,"last update":"Tue Apr 2 09:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.66600000000000004,"15min":0.63400000000000001},"min":{"1min":0.45300000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83999999999999997,"5min":5.4589999999999996,"15min":5.4589999999999996},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.79700000000000004,"15min":0.752},"min":{"1min":0.56299999999999994,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.01,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.76100000000000001}]},{"osd":5,"last update":"Tue Apr 2 09:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72999999999999998,"15min":0.69899999999999995},"min":{"1min":0.46999999999999997,"5min":0.42499999999999999,"15min":0.41299999999999998},"max":{"1min":0.97299999999999998,"5min":5.1340000000000003,"15min":5.1340000000000003},"last":0.88},{"interface":"front","average":{"1min":0.752,"5min":0.75800000000000001,"15min":0.73799999999999999},"min":{"1min":0.59299999999999997,"5min":0.372,"15min":0.372},"max":{"1min":0.96099999999999997,"5min":3.492,"15min":3.492},"last":0.84399999999999997}]},{"osd":6,"last update":"Tue Apr 2 09:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.70399999999999996,"15min":0.70199999999999996},"min":{"1min":0.497,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.92300000000000004,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.76800000000000002,"15min":0.72799999999999998},"min":{"1min":0.46200000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.90400000000000003,"5min":7.8899999999999997,"15min":7.8899999999999997},"last":0.83099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834576036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64972,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306868,"statfs":{"total":96636764160,"available":96570232832,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.65900000000000003,"15min":0.70799999999999996},"min":{"1min":0.52000000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.94199999999999995,"5min":0.94899999999999995,"15min":1.1879999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.68700000000000006,"15min":0.72199999999999998},"min":{"1min":0.52100000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83999999999999997,"5min":0.98399999999999999,"15min":1.0600000000000001},"last":0.755}]},{"osd":1,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.59499999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.95099999999999996,"5min":1.0149999999999999,"15min":1.212},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.755,"5min":0.69799999999999995,"15min":0.73199999999999998},"min":{"1min":0.48499999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.1759999999999999},"last":0.57099999999999995}]},{"osd":2,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68500000000000005,"15min":0.72999999999999998},"min":{"1min":0.47399999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.98099999999999998,"5min":1.046,"15min":1.2789999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.68100000000000005,"15min":0.71899999999999997},"min":{"1min":0.57199999999999995,"5min":0.44,"15min":0.439},"max":{"1min":0.89300000000000002,"5min":0.99299999999999999,"15min":1.226},"last":0.61199999999999999}]},{"osd":3,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.69399999999999995,"15min":0.73399999999999999},"min":{"1min":0.501,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.98099999999999998,"5min":1.026,"15min":1.2549999999999999},"last":0.629},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.73599999999999999,"15min":0.76300000000000001},"min":{"1min":0.60499999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.95799999999999996,"5min":1.1899999999999999,"15min":1.3200000000000001},"last":0.79700000000000004}]},{"osd":4,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64900000000000002,"15min":0.625},"min":{"1min":0.439,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0329999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68400000000000005,"15min":0.66800000000000004},"min":{"1min":0.46800000000000003,"5min":0.373,"15min":0.373},"max":{"1min":0.96599999999999997,"5min":1.1439999999999999,"15min":1.1879999999999999},"last":0.85399999999999998}]},{"osd":5,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66000000000000003,"15min":0.65500000000000003},"min":{"1min":0.48399999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.92300000000000004,"5min":0.98699999999999999,"15min":1.2669999999999999},"last":0.497},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.53800000000000003,"5min":0.436,"15min":0.436},"max":{"1min":1.095,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.66900000000000004}]},{"osd":7,"last update":"Tue Apr 2 09:17:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.58599999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.999,"5min":1.131,"15min":1.2370000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73199999999999998,"15min":0.73499999999999999},"min":{"1min":0.54200000000000004,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.01,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.65000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834576037,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57348,"kb_used_data":4092,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314492,"statfs":{"total":96636764160,"available":96578039808,"internally_reserved":0,"allocated":4190208,"data_stored":2441766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.437,"15min":0.44600000000000001},"min":{"1min":0.33900000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.73699999999999999,"5min":1.76,"15min":1.76},"last":0.498},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.48299999999999998,"15min":0.503},"min":{"1min":0.33700000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.35799999999999998}]},{"osd":2,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.48399999999999999,"15min":0.497},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.70599999999999996,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.57199999999999995,"15min":0.58499999999999996},"min":{"1min":0.45100000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.84199999999999997,"5min":1.859,"15min":1.859},"last":0.437}]},{"osd":3,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.55300000000000005,"15min":0.56799999999999995},"min":{"1min":0.46899999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82299999999999995,"5min":1.8169999999999999,"15min":1.8169999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.52900000000000003,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.73099999999999998,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.46400000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.755,"15min":0.77600000000000002},"min":{"1min":0.65300000000000002,"5min":0.44500000000000001,"15min":0.433},"max":{"1min":1.0720000000000001,"5min":1.948,"15min":1.948},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.72899999999999998,"15min":0.76200000000000001},"min":{"1min":0.53600000000000003,"5min":0.48999999999999999,"15min":0.41799999999999998},"max":{"1min":1.018,"5min":1.1060000000000001,"15min":1.25},"last":0.69199999999999995}]},{"osd":5,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76700000000000002,"15min":0.77900000000000003},"min":{"1min":0.59699999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.90700000000000003,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.79900000000000004,"15min":0.80500000000000005},"min":{"1min":0.64400000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.0069999999999999,"5min":4.5229999999999997,"15min":4.5229999999999997},"last":0.67300000000000004}]},{"osd":6,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.79200000000000004,"15min":0.80400000000000005},"min":{"1min":0.58199999999999996,"5min":0.501,"15min":0.48099999999999998},"max":{"1min":0.997,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.77400000000000002,"15min":0.80000000000000004},"min":{"1min":0.59999999999999998,"5min":0.52900000000000003,"15min":0.49099999999999999},"max":{"1min":0.97899999999999998,"5min":1.2290000000000001,"15min":1.2529999999999999},"last":0.72099999999999997}]},{"osd":7,"last update":"Tue Apr 2 09:17:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.78600000000000003,"15min":0.81000000000000005},"min":{"1min":0.67400000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0589999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.80300000000000005,"15min":0.82399999999999995},"min":{"1min":0.64800000000000002,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0920000000000001,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.71399999999999997}]}]},{"osd":0,"up_from":13,"seq":55834576037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52672,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319168,"statfs":{"total":96636764160,"available":96582828032,"internally_reserved":0,"allocated":3592192,"data_stored":1850379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 09:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42199999999999999,"15min":0.44400000000000001},"min":{"1min":0.317,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.61899999999999999,"5min":0.77100000000000002,"15min":0.78100000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.46000000000000002,"15min":0.48799999999999999},"min":{"1min":0.29599999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.629,"5min":0.66300000000000003,"15min":0.91600000000000004},"last":0.43099999999999999}]},{"osd":2,"last update":"Tue Apr 2 09:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48999999999999999,"15min":0.50600000000000001},"min":{"1min":0.40200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.67400000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.501},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.50800000000000001,"15min":0.53600000000000003},"min":{"1min":0.38700000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71099999999999997,"5min":0.72699999999999998,"15min":0.999},"last":0.38700000000000001}]},{"osd":3,"last update":"Tue Apr 2 09:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.503,"15min":0.54500000000000004},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.312},"max":{"1min":0.73299999999999998,"5min":0.75600000000000001,"15min":0.85699999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.57699999999999996,"15min":0.62},"min":{"1min":0.47799999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.79900000000000004,"5min":0.81100000000000005,"15min":0.92000000000000004},"last":0.54400000000000004}]},{"osd":4,"last update":"Tue Apr 2 09:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.76000000000000001,"15min":0.80800000000000005},"min":{"1min":0.72999999999999998,"5min":0.55000000000000004,"15min":0.45600000000000002},"max":{"1min":0.90100000000000002,"5min":1.129,"15min":1.1950000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.76400000000000001,"15min":0.80300000000000005},"min":{"1min":0.60499999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.89300000000000002,"5min":1.0369999999999999,"15min":1.169},"last":0.86099999999999999}]},{"osd":5,"last update":"Tue Apr 2 09:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.752,"15min":0.79900000000000004},"min":{"1min":0.629,"5min":0.54000000000000004,"15min":0.497},"max":{"1min":0.96399999999999997,"5min":1.0840000000000001,"15min":1.232},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.76100000000000001,"15min":0.81100000000000005},"min":{"1min":0.67900000000000005,"5min":0.497,"15min":0.497},"max":{"1min":0.90200000000000002,"5min":1.0669999999999999,"15min":1.21},"last":0.89500000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73099999999999998,"15min":0.77700000000000002},"min":{"1min":0.59099999999999997,"5min":0.497,"15min":0.497},"max":{"1min":0.93300000000000005,"5min":1.02,"15min":1.196},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.747,"15min":0.79100000000000004},"min":{"1min":0.59399999999999997,"5min":0.53700000000000003,"15min":0.52900000000000003},"max":{"1min":0.95799999999999996,"5min":1.073,"15min":1.246},"last":0.95799999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.81200000000000006,"15min":0.84399999999999997},"min":{"1min":0.71999999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.93500000000000005,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.79200000000000004,"15min":0.82799999999999996},"min":{"1min":0.60399999999999998,"5min":0.499,"15min":0.499},"max":{"1min":1.0169999999999999,"5min":2.3889999999999998,"15min":2.3889999999999998},"last":0.92000000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:17:23 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.43099999999999999,"15min":0.44},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70299999999999996,"5min":1.681,"15min":1.681},"last":0.311},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49099999999999999,"15min":0.503},"min":{"1min":0.29599999999999999,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":0.71299999999999997,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":0.44400000000000001}]},{"osd":1,"last update":"Tue Apr 2 09:17:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.49099999999999999,"15min":0.503},"min":{"1min":0.38,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67800000000000005,"5min":1.72,"15min":1.72},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.53400000000000003,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.753,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.56999999999999995}]},{"osd":3,"last update":"Tue Apr 2 09:17:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52400000000000002,"15min":0.54100000000000004},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.34599999999999997},"max":{"1min":0.65700000000000003,"5min":0.871,"15min":1.2749999999999999},"last":0.433},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57999999999999996,"15min":0.61599999999999999},"min":{"1min":0.46000000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.73499999999999999,"5min":1.754,"15min":1.754},"last":0.46000000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:17:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.76200000000000001,"15min":0.77100000000000002},"min":{"1min":0.55900000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.0409999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.749,"15min":0.78200000000000003},"min":{"1min":0.55400000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.97999999999999998,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.79600000000000004}]},{"osd":5,"last update":"Tue Apr 2 09:17:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.746,"15min":0.77400000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.96099999999999997,"5min":1.177,"15min":1.99},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.76700000000000002,"15min":0.79100000000000004},"min":{"1min":0.52300000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1379999999999999,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.70199999999999996}]},{"osd":6,"last update":"Tue Apr 2 09:17:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.77700000000000002,"15min":0.80200000000000005},"min":{"1min":0.52000000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.198,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.77500000000000002,"15min":0.79500000000000004},"min":{"1min":0.54000000000000004,"5min":0.504,"15min":0.504},"max":{"1min":1.0720000000000001,"5min":1.103,"15min":1.3819999999999999},"last":0.88600000000000001}]},{"osd":7,"last update":"Tue Apr 2 09:17:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.79100000000000004,"15min":0.81699999999999995},"min":{"1min":0.58799999999999997,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0980000000000001,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78800000000000003,"15min":0.81399999999999995},"min":{"1min":0.61299999999999999,"5min":0.53000000000000003,"15min":0.497},"max":{"1min":1.026,"5min":1.603,"15min":1.603},"last":0.90800000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56780,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315060,"statfs":{"total":96636764160,"available":96578621440,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41599999999999998,"15min":0.437},"min":{"1min":0.32500000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68000000000000005,"5min":0.73599999999999999,"15min":0.76000000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.436,"15min":0.47399999999999998},"min":{"1min":0.34100000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57199999999999995,"5min":0.68300000000000005,"15min":0.876},"last":0.35599999999999998}]},{"osd":1,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.47799999999999998,"15min":0.49399999999999999},"min":{"1min":0.35199999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.77600000000000002,"5min":0.81699999999999995,"15min":0.90300000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52400000000000002,"15min":0.55900000000000005},"min":{"1min":0.41499999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.754,"5min":0.754,"15min":0.98499999999999999},"last":0.57799999999999996}]},{"osd":2,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55200000000000005,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.40100000000000002,"15min":0.38200000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.93100000000000005},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55200000000000005,"15min":0.59699999999999998},"min":{"1min":0.38,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.877,"5min":0.877,"15min":1.0840000000000001},"last":0.39800000000000002}]},{"osd":4,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70599999999999996,"15min":0.73999999999999999},"min":{"1min":0.498,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86199999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.752,"5min":0.73099999999999998,"15min":0.75800000000000001},"min":{"1min":0.57299999999999995,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.91700000000000004,"5min":1.0760000000000001,"15min":1.1619999999999999},"last":0.79100000000000004}]},{"osd":5,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72999999999999998,"15min":0.76500000000000001},"min":{"1min":0.54400000000000004,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.874,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.73299999999999998,"15min":0.76800000000000002},"min":{"1min":0.55000000000000004,"5min":0.51100000000000001,"15min":0.505},"max":{"1min":0.97199999999999998,"5min":1.0780000000000001,"15min":1.266},"last":0.628}]},{"osd":6,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.748,"15min":0.77400000000000002},"min":{"1min":0.57199999999999995,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.056,"5min":1.0900000000000001,"15min":1.284},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77200000000000002,"15min":0.79900000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.495},"max":{"1min":0.93799999999999994,"5min":1.093,"15min":1.2},"last":0.85799999999999998}]},{"osd":7,"last update":"Tue Apr 2 09:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.745,"15min":0.79100000000000004},"min":{"1min":0.52700000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0840000000000001,"5min":1.125,"15min":1.3260000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.78000000000000003,"15min":0.81000000000000005},"min":{"1min":0.65500000000000003,"5min":0.495,"15min":0.495},"max":{"1min":1.0620000000000001,"5min":1.087,"15min":1.248},"last":0.752}]}]},{"osd":4,"up_from":13,"seq":55834576036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48584,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323256,"statfs":{"total":96636764160,"available":96587014144,"internally_reserved":0,"allocated":3596288,"data_stored":1851398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63300000000000001,"15min":0.67200000000000004},"min":{"1min":0.49099999999999999,"5min":0.38100000000000001,"15min":0.29299999999999998},"max":{"1min":0.86799999999999999,"5min":1.0189999999999999,"15min":1.0389999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.63400000000000001,"15min":0.66900000000000004},"min":{"1min":0.505,"5min":0.40999999999999998,"15min":0.39600000000000002},"max":{"1min":0.92800000000000005,"5min":0.97099999999999997,"15min":1.075},"last":0.625}]},{"osd":1,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.70699999999999996,"15min":0.74099999999999999},"min":{"1min":0.53200000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.996,"5min":1.0329999999999999,"15min":1.069},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.71999999999999997,"15min":0.751},"min":{"1min":0.48099999999999998,"5min":0.46100000000000002,"15min":0.40799999999999997},"max":{"1min":0.89500000000000002,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.65300000000000002}]},{"osd":2,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65600000000000003,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.95399999999999996,"5min":1.077,"15min":1.077},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67600000000000005,"15min":0.71199999999999997},"min":{"1min":0.50900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95999999999999996,"5min":1.032,"15min":1.0740000000000001},"last":0.60599999999999998}]},{"osd":3,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.71599999999999997,"15min":0.749},"min":{"1min":0.53000000000000003,"5min":0.48799999999999999,"15min":0.47399999999999998},"max":{"1min":0.94599999999999995,"5min":0.98099999999999998,"15min":1.018},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.69899999999999995,"15min":0.73999999999999999},"min":{"1min":0.59699999999999998,"5min":0.51200000000000001,"15min":0.505},"max":{"1min":0.93799999999999994,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.57499999999999996}]},{"osd":5,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.47899999999999998,"5min":0.35299999999999998,"15min":0.34000000000000002},"max":{"1min":0.88200000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.629,"15min":0.625},"min":{"1min":0.46000000000000002,"5min":0.371,"15min":0.35099999999999998},"max":{"1min":0.82699999999999996,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.76700000000000002}]},{"osd":6,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.50800000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81599999999999995,"5min":0.96899999999999997,"15min":1.0189999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.68799999999999994,"15min":0.69399999999999995},"min":{"1min":0.49299999999999999,"5min":0.40100000000000002,"15min":0.38400000000000001},"max":{"1min":0.74399999999999999,"5min":1.0149999999999999,"15min":1.0269999999999999},"last":0.69199999999999995}]},{"osd":7,"last update":"Tue Apr 2 09:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.70299999999999996,"15min":0.70599999999999996},"min":{"1min":0.48399999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.90200000000000002,"5min":1.002,"15min":1.002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76900000000000002,"15min":0.77700000000000002},"min":{"1min":0.56399999999999995,"5min":0.499,"15min":0.499},"max":{"1min":0.88900000000000001,"5min":1.0960000000000001,"15min":1.0980000000000001},"last":0.74399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576039,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53248,"kb_used_data":4088,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318592,"statfs":{"total":96636764160,"available":96582238208,"internally_reserved":0,"allocated":4186112,"data_stored":2440747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.66800000000000004,"15min":0.68999999999999995},"min":{"1min":0.54300000000000004,"5min":0.42199999999999999,"15min":0.39600000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":1.004},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.52800000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.90800000000000003,"5min":1.085,"15min":1.085},"last":0.72099999999999997}]},{"osd":1,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.66400000000000003,"15min":0.70399999999999996},"min":{"1min":0.55700000000000005,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89000000000000001,"5min":0.94499999999999995,"15min":1.0189999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.69899999999999995,"15min":0.71299999999999997},"min":{"1min":0.60399999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.89400000000000002,"5min":0.96799999999999997,"15min":1.052},"last":0.55600000000000005}]},{"osd":2,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.69499999999999995,"15min":0.72799999999999998},"min":{"1min":0.57899999999999996,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92400000000000004,"5min":0.97299999999999998,"15min":1.014},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.69299999999999995,"15min":0.72499999999999998},"min":{"1min":0.622,"5min":0.436,"15min":0.436},"max":{"1min":0.96199999999999997,"5min":1.018,"15min":1.0680000000000001},"last":0.60399999999999998}]},{"osd":3,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.71399999999999997,"15min":0.76000000000000001},"min":{"1min":0.59399999999999997,"5min":0.496,"15min":0.496},"max":{"1min":0.92700000000000005,"5min":0.93600000000000005,"15min":1.083},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.76900000000000002,"15min":0.78100000000000003},"min":{"1min":0.71799999999999997,"5min":0.50600000000000001,"15min":0.47399999999999998},"max":{"1min":0.92600000000000005,"5min":1.0109999999999999,"15min":1.1020000000000001},"last":0.67000000000000004}]},{"osd":4,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65100000000000002,"15min":0.63100000000000001},"min":{"1min":0.45000000000000001,"5min":0.40699999999999997,"15min":0.38300000000000001},"max":{"1min":0.91300000000000003,"5min":0.98399999999999999,"15min":1.054},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.68700000000000006,"15min":0.66900000000000004},"min":{"1min":0.46300000000000002,"5min":0.45900000000000002,"15min":0.40400000000000003},"max":{"1min":0.93999999999999995,"5min":1.145,"15min":1.145},"last":0.76100000000000001}]},{"osd":6,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.67700000000000005,"15min":0.67400000000000004},"min":{"1min":0.497,"5min":0.434,"15min":0.41099999999999998},"max":{"1min":0.79000000000000004,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71599999999999997,"15min":0.70999999999999996},"min":{"1min":0.50600000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.96399999999999997,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.70299999999999996}]},{"osd":7,"last update":"Tue Apr 2 09:16:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.73899999999999999,"15min":0.73199999999999998},"min":{"1min":0.51900000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82699999999999996,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.75,"5min":0.77000000000000002,"15min":0.76300000000000001},"min":{"1min":0.52700000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95999999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.77700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T09:17:23.683 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-02T09:17:23.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.871+0000 7ff699e00700 1 Processor -- start 2024-04-02T09:17:23.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.871+0000 7ff699e00700 1 -- start start 2024-04-02T09:17:23.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.872+0000 7ff699e00700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff6940d21f0 0x7ff6940ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.872+0000 7ff699e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6940cec40 0x7ff694161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.872+0000 7ff699e00700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6940ce270 con 0x7ff6941611b0 2024-04-02T09:17:23.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.872+0000 7ff699e00700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69415b600 con 0x7ff6940d21f0 2024-04-02T09:17:23.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.872+0000 7ff699e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6940d2760 con 0x7ff6940cec40 2024-04-02T09:17:23.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.873+0000 7ff692000700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff6940d21f0 0x7ff6940ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.873+0000 7ff692a00700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff6941611b0 0x7ff69415ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57950/0 (socket says 172.21.15.89:57950) 2024-04-02T09:17:23.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.873+0000 7ff692a00700 1 -- 172.21.15.89:0/529748548 learned_addr learned my addr 172.21.15.89:0/529748548 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:23.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.873+0000 7ff693400700 1 --2- 172.21.15.89:0/529748548 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6940cec40 0x7ff694161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.873+0000 7ff692000700 1 -- 172.21.15.89:0/529748548 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6940cec40 msgr2=0x7ff694161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T09:17:23.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.873+0000 7ff692000700 1 --2- 172.21.15.89:0/529748548 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6940cec40 0x7ff694161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.873+0000 7ff692000700 1 -- 172.21.15.89:0/529748548 >> v1:172.21.15.89:6789/0 conn(0x7ff6941611b0 legacy=0x7ff69415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.873+0000 7ff692000700 1 -- 172.21.15.89:0/529748548 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6940d2a40 con 0x7ff6940d21f0 2024-04-02T09:17:23.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.873+0000 7ff693400700 1 --2- 172.21.15.89:0/529748548 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6940cec40 0x7ff694161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T09:17:23.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.873+0000 7ff692000700 1 --2- 172.21.15.89:0/529748548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6940d21f0 0x7ff6940ce700 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7ff680009840 tx=0x7ff680009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29b73df6d3e45a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:23.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.874+0000 7ff691600700 1 -- 172.21.15.89:0/529748548 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff680011830 con 0x7ff6940d21f0 2024-04-02T09:17:23.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.874+0000 7ff691600700 1 -- 172.21.15.89:0/529748548 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff680011990 con 0x7ff6940d21f0 2024-04-02T09:17:23.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.874+0000 7ff691600700 1 -- 172.21.15.89:0/529748548 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff680011d20 con 0x7ff6940d21f0 2024-04-02T09:17:23.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.874+0000 7ff699e00700 1 -- 172.21.15.89:0/529748548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6940d21f0 msgr2=0x7ff6940ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.874+0000 7ff699e00700 1 --2- 172.21.15.89:0/529748548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6940d21f0 0x7ff6940ce700 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7ff680009840 tx=0x7ff680009b50 comp rx=0 tx=0).stop 2024-04-02T09:17:23.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.874+0000 7ff699e00700 1 -- 172.21.15.89:0/529748548 shutdown_connections 2024-04-02T09:17:23.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.874+0000 7ff699e00700 1 --2- 172.21.15.89:0/529748548 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6940cec40 0x7ff694161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.874+0000 7ff699e00700 1 --2- 172.21.15.89:0/529748548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6940d21f0 0x7ff6940ce700 unknown :-1 s=CLOSED pgs=3138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.874+0000 7ff699e00700 1 -- 172.21.15.89:0/529748548 >> 172.21.15.89:0/529748548 conn(0x7ff69408aff0 msgr2=0x7ff6940d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:23.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.874+0000 7ff699e00700 1 -- 172.21.15.89:0/529748548 shutdown_connections 2024-04-02T09:17:23.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.874+0000 7ff699e00700 1 -- 172.21.15.89:0/529748548 wait complete. 2024-04-02T09:17:23.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.876+0000 7ff699e00700 1 Processor -- start 2024-04-02T09:17:23.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.876+0000 7ff699e00700 1 -- start start 2024-04-02T09:17:23.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.877+0000 7ff699e00700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6940cec40 0x7ff69415ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.877+0000 7ff699e00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6940d21f0 0x7ff6940cc980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.877+0000 7ff699e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6941611b0 0x7ff6940ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.877+0000 7ff699e00700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69409de10 con 0x7ff6941611b0 2024-04-02T09:17:23.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.877+0000 7ff699e00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6940cc040 con 0x7ff6940d21f0 2024-04-02T09:17:23.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.877+0000 7ff699e00700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6940cb4c0 con 0x7ff6940cec40 2024-04-02T09:17:23.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.877+0000 7ff693400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6941611b0 0x7ff6940ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.877+0000 7ff693400700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6941611b0 0x7ff6940ccec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46526/0 (socket says 172.21.15.89:46526) 2024-04-02T09:17:23.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.877+0000 7ff693400700 1 -- 172.21.15.89:0/1155959758 learned_addr learned my addr 172.21.15.89:0/1155959758 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:23.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.877+0000 7ff692a00700 1 --2- 172.21.15.89:0/1155959758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6940cec40 0x7ff69415ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.877+0000 7ff692000700 1 --2- 172.21.15.89:0/1155959758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6940d21f0 0x7ff6940cc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.878+0000 7ff693400700 1 -- 172.21.15.89:0/1155959758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6940cec40 msgr2=0x7ff69415ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.878+0000 7ff693400700 1 --2- 172.21.15.89:0/1155959758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6940cec40 0x7ff69415ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.878+0000 7ff693400700 1 -- 172.21.15.89:0/1155959758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6940d21f0 msgr2=0x7ff6940cc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:23.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.878+0000 7ff693400700 1 --2- 172.21.15.89:0/1155959758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6940d21f0 0x7ff6940cc980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:23.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.878+0000 7ff693400700 1 -- 172.21.15.89:0/1155959758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6800094a0 con 0x7ff6941611b0 2024-04-02T09:17:23.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.878+0000 7ff693400700 1 --2- 172.21.15.89:0/1155959758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6941611b0 0x7ff6940ccec0 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00cef0 tx=0x7ff68c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:23.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.878+0000 7ff68b400700 1 -- 172.21.15.89:0/1155959758 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c00ad00 con 0x7ff6941611b0 2024-04-02T09:17:23.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.878+0000 7ff68b400700 1 -- 172.21.15.89:0/1155959758 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68c00ae60 con 0x7ff6941611b0 2024-04-02T09:17:23.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.878+0000 7ff68b400700 1 -- 172.21.15.89:0/1155959758 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c015680 con 0x7ff6941611b0 2024-04-02T09:17:23.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.878+0000 7ff699e00700 1 -- 172.21.15.89:0/1155959758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6940cb7a0 con 0x7ff6941611b0 2024-04-02T09:17:23.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.878+0000 7ff699e00700 1 -- 172.21.15.89:0/1155959758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6940cbd60 con 0x7ff6941611b0 2024-04-02T09:17:23.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.879+0000 7ff699e00700 1 -- 172.21.15.89:0/1155959758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69409df80 con 0x7ff6941611b0 2024-04-02T09:17:23.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.881+0000 7ff68b400700 1 -- 172.21.15.89:0/1155959758 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff68c00d430 con 0x7ff6941611b0 2024-04-02T09:17:23.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.882+0000 7ff68b400700 1 --2- 172.21.15.89:0/1155959758 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff67406adb0 0x7ff67406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:23.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.882+0000 7ff68b400700 1 -- 172.21.15.89:0/1155959758 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7ff68c089ab0 con 0x7ff6941611b0 2024-04-02T09:17:23.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.882+0000 7ff692a00700 1 --2- 172.21.15.89:0/1155959758 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff67406adb0 0x7ff67406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:23.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.882+0000 7ff692a00700 1 --2- 172.21.15.89:0/1155959758 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff67406adb0 0x7ff67406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff684005910 tx=0x7ff684009040 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:23.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:23.886+0000 7ff68b400700 1 -- 172.21.15.89:0/1155959758 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff68c054a50 con 0x7ff6941611b0 2024-04-02T09:17:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.087+0000 7ff699e00700 1 -- 172.21.15.89:0/1155959758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7ff694067720 con 0x7ff6941611b0 2024-04-02T09:17:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.092+0000 7ff68b400700 1 -- 172.21.15.89:0/1155959758 <== mon.0 v2:172.21.15.89:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7ff68c054a50 con 0x7ff6941611b0 2024-04-02T09:17:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.092+0000 7ff68b400700 1 -- 172.21.15.89:0/1155959758 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v100)=0 v100) v1 ==== 123+0+0 (secure 0 0 0) 0x7ff68c058070 con 0x7ff6941611b0 2024-04-02T09:17:24.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.095+0000 7ff699e00700 1 -- 172.21.15.89:0/1155959758 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff67406adb0 msgr2=0x7ff67406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:24.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.095+0000 7ff699e00700 1 --2- 172.21.15.89:0/1155959758 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff67406adb0 0x7ff67406d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff684005910 tx=0x7ff684009040 comp rx=0 tx=0).stop 2024-04-02T09:17:24.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.096+0000 7ff699e00700 1 -- 172.21.15.89:0/1155959758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6941611b0 msgr2=0x7ff6940ccec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.096+0000 7ff699e00700 1 --2- 172.21.15.89:0/1155959758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6941611b0 0x7ff6940ccec0 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00cef0 tx=0x7ff68c00cf20 comp rx=0 tx=0).stop 2024-04-02T09:17:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.096+0000 7ff699e00700 1 -- 172.21.15.89:0/1155959758 shutdown_connections 2024-04-02T09:17:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.096+0000 7ff699e00700 1 --2- 172.21.15.89:0/1155959758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6941611b0 0x7ff6940ccec0 unknown :-1 s=CLOSED pgs=3139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.096+0000 7ff699e00700 1 --2- 172.21.15.89:0/1155959758 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6940cec40 0x7ff69415ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.096+0000 7ff699e00700 1 --2- 172.21.15.89:0/1155959758 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7ff67406adb0 0x7ff67406d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.096+0000 7ff699e00700 1 --2- 172.21.15.89:0/1155959758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6940d21f0 0x7ff6940cc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.096+0000 7ff699e00700 1 -- 172.21.15.89:0/1155959758 >> 172.21.15.89:0/1155959758 conn(0x7ff69408aff0 msgr2=0x7ff69415ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.096+0000 7ff699e00700 1 -- 172.21.15.89:0/1155959758 shutdown_connections 2024-04-02T09:17:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.096+0000 7ff699e00700 1 -- 172.21.15.89:0/1155959758 wait complete. 2024-04-02T09:17:24.112 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T09:17:24.113 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T09:17:24.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.306+0000 7fb421200700 1 Processor -- start 2024-04-02T09:17:24.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.306+0000 7fb421200700 1 -- start start 2024-04-02T09:17:24.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.308+0000 7fb421200700 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb41c0b7a30 0x7fb41c1685d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:24.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.308+0000 7fb421200700 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41c160fc0 con 0x7fb41c0ce590 2024-04-02T09:17:24.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.308+0000 7fb421200700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41c0d21e0 con 0x7fb41c160be0 2024-04-02T09:17:24.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.308+0000 7fb421200700 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb41c0b65d0 con 0x7fb41c0b7a30 2024-04-02T09:17:24.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.308+0000 7fb419400700 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb41c0ce590 0x7fb41c0ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57954/0 (socket says 172.21.15.89:57954) 2024-04-02T09:17:24.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.308+0000 7fb419400700 1 -- 172.21.15.89:0/1668291637 learned_addr learned my addr 172.21.15.89:0/1668291637 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-04-02T09:17:24.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.308+0000 7fb41a800700 1 --2- 172.21.15.89:0/1668291637 >> v2:172.21.15.89:3300/0 conn(0x7fb41c0b7a30 0x7fb41c1685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:24.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.308+0000 7fb418a00700 1 -- 172.21.15.89:0/1668291637 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162711195 0 0) 0x7fb41c160fc0 con 0x7fb41c0ce590 2024-04-02T09:17:24.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.308+0000 7fb41a800700 1 -- 172.21.15.89:0/1668291637 >> v1:172.21.15.113:6789/0 conn(0x7fb41c160be0 legacy=0x7fb41c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:24.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.308+0000 7fb41a800700 1 -- 172.21.15.89:0/1668291637 >> v1:172.21.15.89:6789/0 conn(0x7fb41c0ce590 legacy=0x7fb41c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:24.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.308+0000 7fb41a800700 1 -- 172.21.15.89:0/1668291637 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c0d24c0 con 0x7fb41c0b7a30 2024-04-02T09:17:24.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.309+0000 7fb41a800700 1 --2- 172.21.15.89:0/1668291637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb41c0b7a30 0x7fb41c1685d0 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7fb414009d90 tx=0x7fb41400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86de8b1c5d85dffa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:24.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.309+0000 7fb418a00700 1 -- 172.21.15.89:0/1668291637 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41400e070 con 0x7fb41c0b7a30 2024-04-02T09:17:24.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.309+0000 7fb418a00700 1 -- 172.21.15.89:0/1668291637 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fb41400bbd0 con 0x7fb41c0b7a30 2024-04-02T09:17:24.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.309+0000 7fb418a00700 1 -- 172.21.15.89:0/1668291637 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb414012430 con 0x7fb41c0b7a30 2024-04-02T09:17:24.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.309+0000 7fb421200700 1 -- 172.21.15.89:0/1668291637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb41c0b7a30 msgr2=0x7fb41c1685d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:24.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.309+0000 7fb421200700 1 --2- 172.21.15.89:0/1668291637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb41c0b7a30 0x7fb41c1685d0 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7fb414009d90 tx=0x7fb41400b330 comp rx=0 tx=0).stop 2024-04-02T09:17:24.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.309+0000 7fb421200700 1 -- 172.21.15.89:0/1668291637 shutdown_connections 2024-04-02T09:17:24.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.309+0000 7fb421200700 1 --2- 172.21.15.89:0/1668291637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb41c0b7a30 0x7fb41c1685d0 unknown :-1 s=CLOSED pgs=3141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:24.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.309+0000 7fb421200700 1 -- 172.21.15.89:0/1668291637 >> 172.21.15.89:0/1668291637 conn(0x7fb41c08aff0 msgr2=0x7fb41c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:24.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.309+0000 7fb421200700 1 -- 172.21.15.89:0/1668291637 shutdown_connections 2024-04-02T09:17:24.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.310+0000 7fb421200700 1 -- 172.21.15.89:0/1668291637 wait complete. 2024-04-02T09:17:24.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.311+0000 7fb421200700 1 Processor -- start 2024-04-02T09:17:24.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.311+0000 7fb421200700 1 -- start start 2024-04-02T09:17:24.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.312+0000 7fb421200700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb41c0b7a30 0x7fb41c161420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.312+0000 7fb421200700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb41c0ce590 0x7fb41c09da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.312+0000 7fb421200700 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb41c160be0 0x7fb41c1524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.312+0000 7fb421200700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c0c7e10 con 0x7fb41c0b7a30 2024-04-02T09:17:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.312+0000 7fb421200700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c0b93d0 con 0x7fb41c0ce590 2024-04-02T09:17:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.312+0000 7fb421200700 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c0c7200 con 0x7fb41c160be0 2024-04-02T09:17:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.312+0000 7fb419e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb41c0b7a30 0x7fb41c161420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:24.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.312+0000 7fb419e00700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb41c0b7a30 0x7fb41c161420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46544/0 (socket says 172.21.15.89:46544) 2024-04-02T09:17:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.312+0000 7fb419e00700 1 -- 172.21.15.89:0/2389371501 learned_addr learned my addr 172.21.15.89:0/2389371501 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-04-02T09:17:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.312+0000 7fb419400700 1 --2- 172.21.15.89:0/2389371501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb41c0ce590 0x7fb41c09da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.313+0000 7fb41a800700 1 --2- 172.21.15.89:0/2389371501 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb41c160be0 0x7fb41c1524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.313+0000 7fb419e00700 1 -- 172.21.15.89:0/2389371501 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb41c160be0 msgr2=0x7fb41c1524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.313+0000 7fb419e00700 1 --2- 172.21.15.89:0/2389371501 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb41c160be0 0x7fb41c1524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.313+0000 7fb419e00700 1 -- 172.21.15.89:0/2389371501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb41c0ce590 msgr2=0x7fb41c09da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.313+0000 7fb419e00700 1 --2- 172.21.15.89:0/2389371501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb41c0ce590 0x7fb41c09da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.313+0000 7fb419e00700 1 -- 172.21.15.89:0/2389371501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb414009a70 con 0x7fb41c0b7a30 2024-04-02T09:17:24.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.313+0000 7fb419e00700 1 --2- 172.21.15.89:0/2389371501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb41c0b7a30 0x7fb41c161420 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00c4e0 tx=0x7fb40c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:24.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.313+0000 7fb412000700 1 -- 172.21.15.89:0/2389371501 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c0052d0 con 0x7fb41c0b7a30 2024-04-02T09:17:24.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.313+0000 7fb412000700 1 -- 172.21.15.89:0/2389371501 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fb40c005430 con 0x7fb41c0b7a30 2024-04-02T09:17:24.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.313+0000 7fb421200700 1 -- 172.21.15.89:0/2389371501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41c0c74e0 con 0x7fb41c0b7a30 2024-04-02T09:17:24.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.313+0000 7fb412000700 1 -- 172.21.15.89:0/2389371501 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c0038e0 con 0x7fb41c0b7a30 2024-04-02T09:17:24.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.313+0000 7fb421200700 1 -- 172.21.15.89:0/2389371501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41c0c7aa0 con 0x7fb41c0b7a30 2024-04-02T09:17:24.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.314+0000 7fb421200700 1 -- 172.21.15.89:0/2389371501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb41c067720 con 0x7fb41c0b7a30 2024-04-02T09:17:24.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.315+0000 7fb412000700 1 -- 172.21.15.89:0/2389371501 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 191) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb40c003a40 con 0x7fb41c0b7a30 2024-04-02T09:17:24.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.317+0000 7fb412000700 1 --2- 172.21.15.89:0/2389371501 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb3fc06adb0 0x7fb3fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T09:17:24.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.317+0000 7fb412000700 1 -- 172.21.15.89:0/2389371501 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 6122+0+0 (secure 0 0 0) 0x7fb40c088ac0 con 0x7fb41c0b7a30 2024-04-02T09:17:24.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.317+0000 7fb419400700 1 --2- 172.21.15.89:0/2389371501 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb3fc06adb0 0x7fb3fc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T09:17:24.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.317+0000 7fb419400700 1 --2- 172.21.15.89:0/2389371501 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb3fc06adb0 0x7fb3fc06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb408007b10 tx=0x7fb408007a60 comp rx=0 tx=0).ready entity=mgr.13575 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T09:17:24.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.321+0000 7fb412000700 1 -- 172.21.15.89:0/2389371501 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb40c053950 con 0x7fb41c0b7a30 2024-04-02T09:17:24.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.564+0000 7fb421200700 1 -- 172.21.15.89:0/2389371501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb41c160020 con 0x7fb41c0b7a30 2024-04-02T09:17:24.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.564+0000 7fb412000700 1 -- 172.21.15.89:0/2389371501 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+3541 (secure 0 0 0) 0x7fb40c017090 con 0x7fb41c0b7a30 2024-04-02T09:17:24.563 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-02T09:17:24.563 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":146,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10376,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2447701494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2447701494},{"type":"v1","addr":"172.21.15.113:6834","nonce":2447701494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":12103,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/361614750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":361614750},{"type":"v1","addr":"172.21.15.113:6835","nonce":361614750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":13281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2871180587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2871180587},{"type":"v1","addr":"172.21.15.89:6835","nonce":2871180587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138},{"gid":13719,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2493300658","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2493300658},{"type":"v1","addr":"172.21.15.89:6837","nonce":2493300658}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146}],"filesystems":[]} 2024-04-02T09:17:24.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.566+0000 7fb421200700 1 -- 172.21.15.89:0/2389371501 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb3fc06adb0 msgr2=0x7fb3fc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:24.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.566+0000 7fb421200700 1 --2- 172.21.15.89:0/2389371501 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb3fc06adb0 0x7fb3fc06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb408007b10 tx=0x7fb408007a60 comp rx=0 tx=0).stop 2024-04-02T09:17:24.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.566+0000 7fb421200700 1 -- 172.21.15.89:0/2389371501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb41c0b7a30 msgr2=0x7fb41c161420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T09:17:24.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.566+0000 7fb421200700 1 --2- 172.21.15.89:0/2389371501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb41c0b7a30 0x7fb41c161420 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00c4e0 tx=0x7fb40c00c7f0 comp rx=0 tx=0).stop 2024-04-02T09:17:24.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.566+0000 7fb421200700 1 -- 172.21.15.89:0/2389371501 shutdown_connections 2024-04-02T09:17:24.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.566+0000 7fb421200700 1 --2- 172.21.15.89:0/2389371501 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb41c160be0 0x7fb41c1524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:24.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.566+0000 7fb421200700 1 --2- 172.21.15.89:0/2389371501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb41c0b7a30 0x7fb41c161420 unknown :-1 s=CLOSED pgs=3142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:24.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.566+0000 7fb421200700 1 --2- 172.21.15.89:0/2389371501 >> [v2:172.21.15.89:6832/108855,v1:172.21.15.89:6833/108855] conn(0x7fb3fc06adb0 0x7fb3fc06d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:24.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.566+0000 7fb421200700 1 --2- 172.21.15.89:0/2389371501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb41c0ce590 0x7fb41c09da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T09:17:24.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.567+0000 7fb421200700 1 -- 172.21.15.89:0/2389371501 >> 172.21.15.89:0/2389371501 conn(0x7fb41c08aff0 msgr2=0x7fb41c06c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T09:17:24.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.567+0000 7fb421200700 1 -- 172.21.15.89:0/2389371501 shutdown_connections 2024-04-02T09:17:24.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-02T09:17:24.567+0000 7fb421200700 1 -- 172.21.15.89:0/2389371501 wait complete. 2024-04-02T09:17:24.567 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 146 2024-04-02T09:17:24.583 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-02T09:17:24.584 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-02T09:17:24.584 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-02T09:17:24.584 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:24.597 INFO:tasks.ceph.mds.a:Stopped 2024-04-02T09:17:24.597 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-02T09:17:24.597 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:24.652 INFO:tasks.ceph.mds.c:Stopped 2024-04-02T09:17:24.652 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-02T09:17:24.652 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:24.669 INFO:tasks.ceph.mds.b:Stopped 2024-04-02T09:17:24.669 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-02T09:17:24.669 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:24.726 INFO:tasks.ceph.mds.d:Stopped 2024-04-02T09:17:24.726 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-02T09:17:24.726 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-02T09:17:24.726 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:24.775 INFO:tasks.ceph.osd.0:Stopped 2024-04-02T09:17:24.775 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-02T09:17:24.776 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:24.836 INFO:tasks.ceph.osd.1:Stopped 2024-04-02T09:17:24.837 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-02T09:17:24.837 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:24.892 INFO:tasks.ceph.osd.2:Stopped 2024-04-02T09:17:24.892 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-02T09:17:24.892 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:24.945 INFO:tasks.ceph.osd.3:Stopped 2024-04-02T09:17:24.945 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-02T09:17:24.945 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:24.995 INFO:tasks.ceph.osd.4:Stopped 2024-04-02T09:17:24.996 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-02T09:17:24.996 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:25.059 INFO:tasks.ceph.osd.5:Stopped 2024-04-02T09:17:25.059 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-02T09:17:25.059 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:25.118 INFO:tasks.ceph.osd.6:Stopped 2024-04-02T09:17:25.119 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-02T09:17:25.119 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:25.171 INFO:tasks.ceph.osd.7:Stopped 2024-04-02T09:17:25.171 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-02T09:17:25.171 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-02T09:17:25.172 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:25.244 INFO:tasks.ceph.mgr.y:Stopped 2024-04-02T09:17:25.245 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-02T09:17:25.245 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:25.310 INFO:tasks.ceph.mgr.x:Stopped 2024-04-02T09:17:25.310 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-02T09:17:25.310 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-02T09:17:25.311 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:25.345 INFO:tasks.ceph.mon.a:Stopped 2024-04-02T09:17:25.345 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-02T09:17:25.346 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:25.372 INFO:tasks.ceph.mon.b:Stopped 2024-04-02T09:17:25.372 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-02T09:17:25.372 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T09:17:25.439 INFO:tasks.ceph.mon.c:Stopped 2024-04-02T09:17:25.439 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-02T09:17:25.440 DEBUG:teuthology.orchestra.run.smithi089:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-04-02T09:17:25.526 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T09:17:25.527 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-02T09:17:25.770 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T09:17:25.771 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-02T09:17:25.910 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T09:17:25.910 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-02T09:17:26.024 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T09:17:26.024 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-02T09:17:26.161 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi113.front.sepia.ceph.com 2024-04-02T09:17:26.162 DEBUG:teuthology.orchestra.run.smithi113:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-02T09:17:26.430 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi113.front.sepia.ceph.com 2024-04-02T09:17:26.431 DEBUG:teuthology.orchestra.run.smithi113:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-02T09:17:26.549 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi113.front.sepia.ceph.com 2024-04-02T09:17:26.549 DEBUG:teuthology.orchestra.run.smithi113:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-02T09:17:26.713 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi113.front.sepia.ceph.com 2024-04-02T09:17:26.713 DEBUG:teuthology.orchestra.run.smithi113:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-02T09:17:26.849 INFO:tasks.ceph:Archiving mon data... 2024-04-02T09:17:26.850 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459/data/mon.a.tgz 2024-04-02T09:17:26.850 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-04-02T09:17:26.896 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.cvEZV2fb32 2024-04-02T09:17:26.897 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.cvEZV2fb32 2024-04-02T09:17:27.725 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.cvEZV2fb32 2024-04-02T09:17:27.838 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.cvEZV2fb32 is 5MB 2024-04-02T09:17:28.026 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.cvEZV2fb32 2024-04-02T09:17:28.071 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459/data/mon.b.tgz 2024-04-02T09:17:28.072 DEBUG:teuthology.orchestra.run.smithi113:> mktemp 2024-04-02T09:17:28.121 INFO:teuthology.orchestra.run.smithi113.stdout:/tmp/tmp.qsFe5AqeAr 2024-04-02T09:17:28.122 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.qsFe5AqeAr 2024-04-02T09:17:28.941 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0666 /tmp/tmp.qsFe5AqeAr 2024-04-02T09:17:29.053 DEBUG:teuthology.orchestra.remote:smithi113:/tmp/tmp.qsFe5AqeAr is 5MB 2024-04-02T09:17:29.244 DEBUG:teuthology.orchestra.run.smithi113:> rm -fr /tmp/tmp.qsFe5AqeAr 2024-04-02T09:17:29.289 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459/data/mon.c.tgz 2024-04-02T09:17:29.290 DEBUG:teuthology.orchestra.run.smithi113:> mktemp 2024-04-02T09:17:29.373 INFO:teuthology.orchestra.run.smithi113.stdout:/tmp/tmp.3g0kKcFzJl 2024-04-02T09:17:29.373 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.3g0kKcFzJl 2024-04-02T09:17:30.189 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0666 /tmp/tmp.3g0kKcFzJl 2024-04-02T09:17:30.301 DEBUG:teuthology.orchestra.remote:smithi113:/tmp/tmp.3g0kKcFzJl is 5MB 2024-04-02T09:17:30.498 DEBUG:teuthology.orchestra.run.smithi113:> rm -fr /tmp/tmp.3g0kKcFzJl 2024-04-02T09:17:30.543 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-02T09:17:30.544 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-02T09:17:30.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-02T09:17:30.649 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-02T09:17:30.650 DEBUG:teuthology.orchestra.run.smithi089:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-02T09:17:30.657 DEBUG:teuthology.orchestra.run.smithi113:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-02T09:17:30.734 INFO:teuthology.orchestra.run.smithi089.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-02T09:17:30.755 INFO:teuthology.orchestra.run.smithi113.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-02T09:17:30.772 INFO:tasks.ceph:Archiving crash dumps... 2024-04-02T09:17:30.774 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459/remote/smithi089/crash 2024-04-02T09:17:30.775 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-02T09:17:30.841 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459/remote/smithi113/crash 2024-04-02T09:17:30.842 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-02T09:17:30.907 INFO:tasks.ceph:Shutting down logrotate 2024-04-02T09:17:30.907 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:17:30.911 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-02T09:17:31.003 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-02T09:17:31.068 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-02T09:17:31.134 INFO:tasks.ceph:Compressing logs... 2024-04-02T09:17:31.135 DEBUG:teuthology.orchestra.run.smithi089:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T09:17:31.139 DEBUG:teuthology.orchestra.run.smithi113:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T09:17:53.091 INFO:tasks.ceph:Archiving logs... 2024-04-02T09:17:53.091 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459/remote/smithi089/log 2024-04-02T09:17:53.092 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-02T09:17:57.143 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459/remote/smithi113/log 2024-04-02T09:17:57.144 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-02T09:17:59.268 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-02T09:17:59.279 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-02T09:17:59.279 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-02T09:17:59.284 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-02T09:17:59.412 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-02T09:17:59.421 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-02T09:17:59.421 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T09:17:59.426 DEBUG:teuthology.orchestra.run.smithi113:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T09:17:59.466 INFO:teuthology.orchestra.run.smithi089.stderr:bash: ntpq: command not found 2024-04-02T09:17:59.468 INFO:teuthology.orchestra.run.smithi113.stderr:bash: ntpq: command not found 2024-04-02T09:17:59.474 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T09:17:59.474 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-04-02T09:17:59.474 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 13 +133us[ +133us] +/- 74ms 2024-04-02T09:17:59.474 INFO:teuthology.orchestra.run.smithi089.stdout:^* hv02.front.sepia.ceph.com 3 8 377 203 +275us[ +259us] +/- 56ms 2024-04-02T09:17:59.475 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 -502us[ -502us] +/- 66ms 2024-04-02T09:17:59.475 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T09:17:59.475 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T09:17:59.475 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2024-04-02T09:17:59.476 INFO:teuthology.orchestra.run.smithi113.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 10 +140us[ +140us] +/- 74ms 2024-04-02T09:17:59.476 INFO:teuthology.orchestra.run.smithi113.stdout:^* hv02.front.sepia.ceph.com 3 6 377 10 +214us[ +209us] +/- 59ms 2024-04-02T09:17:59.476 INFO:teuthology.orchestra.run.smithi113.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 76 -438us[ -446us] +/- 66ms 2024-04-02T09:17:59.476 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T09:17:59.477 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-02T09:17:59.487 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-02T09:17:59.488 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-02T09:17:59.497 DEBUG:teuthology.orchestra.run.smithi089:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-02T09:17:59.685 DEBUG:teuthology.orchestra.run.smithi113:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-02T09:17:59.817 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-02T09:17:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T09:17:59.891 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-02T09:17:59.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T09:17:59.962 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-02T09:17:59.972 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-02T09:17:59.989 INFO:teuthology.task.internal:Duration was 6783.109233 seconds 2024-04-02T09:17:59.989 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-02T09:18:00.001 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-02T09:18:00.001 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-02T09:18:00.005 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-02T09:18:00.112 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T09:18:00.116 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T09:18:00.414 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-02T09:18:00.414 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com 2024-04-02T09:18:00.414 DEBUG:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-02T09:18:00.473 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi113.front.sepia.ceph.com 2024-04-02T09:18:00.473 DEBUG:teuthology.orchestra.run.smithi113:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-02T09:18:00.533 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-02T09:18:00.533 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T09:18:00.538 DEBUG:teuthology.orchestra.run.smithi113:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T09:18:00.626 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-02T09:18:00.627 DEBUG:teuthology.orchestra.run.smithi089:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-02T09:18:00.631 DEBUG:teuthology.orchestra.run.smithi113:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-02T09:18:00.774 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-02T09:18:00.786 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-02T09:18:00.786 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-02T09:18:00.853 DEBUG:teuthology.orchestra.run.smithi113:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-02T09:18:00.920 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-02T09:18:00.930 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-02T09:18:00.934 DEBUG:teuthology.orchestra.run.smithi113:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-02T09:18:00.996 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2024-04-02T09:18:01.000 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = core 2024-04-02T09:18:01.032 DEBUG:teuthology.orchestra.run.smithi089:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-02T09:18:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T09:18:01.075 DEBUG:teuthology.orchestra.run.smithi113:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-02T09:18:01.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T09:18:01.118 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-02T09:18:01.129 INFO:teuthology.task.internal:Transferring archived files... 2024-04-02T09:18:01.130 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459/remote/smithi089 2024-04-02T09:18:01.130 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-02T09:18:01.212 DEBUG:teuthology.misc:Transferring archived files from smithi113:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459/remote/smithi113 2024-04-02T09:18:01.212 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-02T09:18:01.291 INFO:teuthology.task.internal:Removing archive directory... 2024-04-02T09:18:01.292 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-02T09:18:01.296 DEBUG:teuthology.orchestra.run.smithi113:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-02T09:18:01.340 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-02T09:18:01.353 INFO:teuthology.task.internal:Not uploading archives. 2024-04-02T09:18:01.353 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-02T09:18:01.366 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-02T09:18:01.367 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-02T09:18:01.380 DEBUG:teuthology.orchestra.run.smithi113:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-02T09:18:01.421 INFO:teuthology.orchestra.run.smithi089.stdout: 524293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 2 09:18 /home/ubuntu/cephtest 2024-04-02T09:18:01.424 INFO:teuthology.orchestra.run.smithi113.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 2 09:18 /home/ubuntu/cephtest 2024-04-02T09:18:01.428 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-02T09:18:01.438 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-02T09:18:01.461 INFO:teuthology.nuke:Checking targets against current locks 2024-04-02T09:18:01.488 DEBUG:teuthology.nuke:shortname: smithi089 2024-04-02T09:18:01.489 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T09:18:01.516 DEBUG:teuthology.nuke:shortname: smithi113 2024-04-02T09:18:01.517 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T09:18:01.520 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 07:13:45.433852', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGtwTkQ7YIBikeL384i+WuovQQ9VOrg7Jr01f4YTXbCtcED02BXdNA6Vxc692Plr1mrtlShqlFiikUvYOzF/3U='} 2024-04-02T09:18:01.543 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-02_06:51:51-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7635459', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 07:13:45.431954', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNNQGxHVHDjN0qBn7n38dK/3CdgI7Sj/i3cmkwTSGB4mRRhXktG01i0oglnwBx99f4xOW6VGFSlX6tXmYuVKlWE='} 2024-04-02T09:18:01.573 INFO:teuthology.orchestra.console.smithi089:Power off 2024-04-02T09:18:01.574 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-02T09:18:01.600 DEBUG:teuthology.orchestra.console.smithi089:power off output: Chassis Power Control: Down/Off 2024-04-02T09:18:01.600 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T09:18:01.619 INFO:teuthology.orchestra.console.smithi113:Power off 2024-04-02T09:18:01.620 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-02T09:18:01.631 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-04-02T09:18:01.643 DEBUG:teuthology.orchestra.console.smithi113:power off output: Chassis Power Control: Down/Off 2024-04-02T09:18:01.643 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T09:18:01.667 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-04-02T09:18:05.633 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T09:18:05.669 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T09:18:05.745 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-04-02T09:18:05.780 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-04-02T09:18:09.746 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T09:18:09.781 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T09:18:09.860 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is off 2024-04-02T09:18:09.894 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is off 2024-04-02T09:18:09.961 INFO:teuthology.orchestra.console.smithi089:Power off completed 2024-04-02T09:18:09.995 INFO:teuthology.orchestra.console.smithi113:Power off completed 2024-04-02T09:18:10.103 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 client.1-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} duration: 6783.109233379364 failure_reason: 'Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments)' flavor: default owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b74153b402544243954c3e270c08762c status: fail success: false 2024-04-02T09:18:10.103 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T09:18:10.188 INFO:teuthology.run:FAIL